diff --git a/.github/CLAUDE.md b/.github/CLAUDE.md new file mode 100644 index 0000000000..e1901075ac --- /dev/null +++ b/.github/CLAUDE.md @@ -0,0 +1,69 @@ +# .github/CLAUDE.md + +Workflow-authoring rules for GitHub Actions. + +## Triggers: restrict `push` to long-lived branches + +The CI trigger events: + +- `pull_request`: a commit is pushed to a branch with an open PR. Runs CI on the + PR. +- `merge_group`: the merge queue builds a temporary `gh-readonly-queue/*` branch + and tests the change as it would land on `main`. +- `push`: a branch ref is updated. We only want this for the long-lived branches + (`main` and `release/**`), i.e. the post-merge run once a PR lands. + +We want CI on `pull_request`, on `merge_group`, and on `push` to a long-lived +branch: one run per stage. A bare `push` trigger instead duplicates two of these: + +- `push` + `pull_request`: a branch pushed directly to this repo (a maintainer + branch, not a fork) fires both. Fork PRs avoid this because the `push` lands on + the fork, not on this repo. +- `push` + `merge_group`: the merge queue's push to the `gh-readonly-queue/*` + branch fires both. + +Restricting `push` to long-lived branches drops the redundant `push` in each +case: + +```yaml +on: + push: + branches: [main, "release/**"] + pull_request: + merge_group: +``` + +## Never `paths`/`paths-ignore` a workflow whose jobs are required checks + +Do NOT add `paths` or `paths-ignore` filters to a workflow whose jobs are required +status checks (most of `build.yml`). When a path filter skips the whole workflow, +the required checks never report, and the PR stays blocked on "Expected, waiting +for status" indefinitely. A skipped workflow is not the same as a passing one. + +## Supply chain security: `--locked` + +Every `cargo` command in CI that resolves dependencies MUST use `--locked` to +enforce the committed `Cargo.lock`. This prevents CI from silently picking up a +newer (potentially compromised) transitive dependency. If `Cargo.lock` is out of +sync with `Cargo.toml`, the build fails immediately, forcing dependency changes to +be explicit and reviewable. See the top-level comment in `build.yml` for full +rationale. Commands exempt from `--locked`: `cargo +nightly fmt` (no dep +resolution), `cargo msrv verify/show` (wrapper tool), `cargo miri setup` (tooling +setup). + +## Action safety + +When writing any github action, consider safety including mitigating common attack +vectors such as expression injection and pull request target attacks. + +```yaml +# The code below is vulnerable to expression injection +run: | + echo "Comment: ${{ github.event.comment.body }}" + +# To mitigate instead use environment variables +env: + COMMENT_BODY: ${{ github.event.comment.body }} +run: | + echo "Comment: $COMMENT_BODY" +``` diff --git a/.github/SECURITY_MODEL.md b/.github/SECURITY_MODEL.md new file mode 100644 index 0000000000..c2de9dd883 --- /dev/null +++ b/.github/SECURITY_MODEL.md @@ -0,0 +1,61 @@ +# Benchmark workflow security model + +Scope: [`benchmark.yml`](workflows/benchmark.yml) (runs bench on the PR merge +commit -- PR head merged into base) and +[`benchmark-post-comment.yml`](workflows/benchmark-post-comment.yml) (posts the +result comment in base-branch context). Other workflows in this repo may follow +different patterns -- this document is specific to the benchmark pair. + +## Running untrusted PR code + +The `run-benchmark` job in `benchmark.yml` checks out the PR merge commit (PR +head merged into base) and runs its build + bench harness. To bound what that +code can do: + +- Workflow-level `permissions: {}` is the default; each job opts in to the + minimum scope it needs. `run-benchmark` declares only `contents: read`, + so any leaked GITHUB_TOKEN has no write power. Fork PRs additionally get + a read-only token from GitHub by default under the `pull_request` event. +- `actions/checkout` uses `persist-credentials: false` so the token isn't + left in the local git config after checkout. +- `cargo install critcmp` runs *before* the PR merge-commit checkout, so the + PR's `.cargo/config.toml` cannot redirect the registry for that install. + +Do not add other secrets or grant write permissions to `run-benchmark`. + +## Rust-cache poisoning + +`Swatinem/rust-cache` is configured with +`save-if: github.event_name == 'push' && github.ref == 'refs/heads/main'`. +PR runs read-restore from main's cache; they never save. The +`github.event_name == 'push'` clause is the load-bearing half -- the +benchmark workflow has no `push` trigger outside of `push: main` for the +cache-warming job, so PR runs cannot match this condition and therefore +cannot write the cache. + +Do not relax the `github.event_name == 'push'` clause. If you do, a PR-head +run can land compile artifacts in main's cache scope (poisoning subsequent +runs) or evict main's entries through the shared cache budget. + +## Cross-PR comment forgery + +`benchmark-post-comment.yml` runs in base-branch context with +`pull-requests: write`. It must derive the target PR number from a source +the fork cannot forge. The choice depends on which event triggered the +upstream bench workflow: + +- **`pull_request` upstream:** use `gh pr view --repo "$REPO" "owner:branch"`, + where `owner:branch` is built from `workflow_run.head_repository.owner.login` + and `workflow_run.head_branch`. Both fields are populated by GitHub from the + trusted workflow_run event payload, not from anything the fork wrote. +- **`issue_comment` upstream:** trust the `bench-pr-number` artifact. For + `issue_comment` events GitHub serves `benchmark.yml` from the default + branch (not the PR head), so the "Stash PR number" + "Upload PR number" + steps run trusted code. The upload happens *before* `actions/checkout` + brings PR code onto the runner, so once the artifact is on GitHub's + storage it is immutable for the rest of the run -- the bench script + cannot alter it, and `contents: read` denies any API-level mutation. + +`workflow_run` itself runs the post-comment workflow from the default +branch (a GitHub guarantee), so a fork PR cannot modify the resolver to +trust the wrong source. diff --git a/.github/actions/install-and-cache/action.yml b/.github/actions/install-and-cache/action.yml new file mode 100644 index 0000000000..8da88c8a41 --- /dev/null +++ b/.github/actions/install-and-cache/action.yml @@ -0,0 +1,101 @@ +# This is copied from https://github.com/tecolicom/actions-install-and-cache +# which is Copyright 2022 Office TECOLI, LLC + +name: install-and-cache generic backend +description: 'GitHub Action to run installer and cache the result' +branding: + color: orange + icon: type + +inputs: + run: { required: true, type: string } + path: { required: true, type: string } + cache: { required: false, type: string, default: yes } + key: { required: false, type: string } + sudo: { required: false, type: string } + verbose: { required: false, type: string, default: false } + +outputs: + cache-hit: + value: ${{ steps.cache.outputs.cache-hit }} + +runs: + using: composite + steps: + + - id: setup + shell: bash + run: | + : setup install-and-cache + define() { IFS='\n' read -r -d '' ${1} || true ; } + define script <<'EOS_cad8_c24e_' + ${{ inputs.run }} + EOS_cad8_c24e_ + directory="${{ inputs.path }}" + given_key="${{ inputs.key }}" + archive= key= + case "${{ inputs.cache }}" in + yes|workflow) + cache="${{ inputs.cache }}" + uname -mrs + hash=$( (uname -mrs ; cat <<< "$script" ; echo $directory) | \ + (md5sum||md5) | awk '{print $1}' ) + key="${hash}${given_key:+-$given_key}" + [ "$cache" == 'workflow' ] && \ + key+="-${{ github.run_id }}-${{ github.run_attempt }}" + archive=$HOME/archive-$hash.tz + ;; + *) + cache=no + ;; + esac + # use "--recursive-unlink" option if GNU tar is found + if tar --version | grep GNU > /dev/null + then + tar="tar --recursive-unlink" + elif gtar --version | grep GNU > /dev/null + then + tar="gtar --recursive-unlink" + else + tar=tar + fi + sed 's/^ *//' << END >> $GITHUB_OUTPUT + cache=$cache + archive=$archive + key=$key + tar=$tar + END + + - id: cache + if: steps.setup.outputs.cache != 'no' + uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: ${{ steps.setup.outputs.archive }} + key: ${{ steps.setup.outputs.key }} + + - id: extract + if: steps.setup.outputs.cache != 'no' && steps.cache.outputs.cache-hit == 'true' + shell: bash + run: | + : extract + archive="${{ steps.setup.outputs.archive }}" + verbose="${{ inputs.verbose }}" + tar="${{ steps.setup.outputs.tar }}" + ls -l $archive + if [ -s $archive ] + then + opt=-Pxz + [[ $verbose == yes || $verbose == true ]] && opt+=v + sudo $tar -C / $opt -f $archive + else + echo "$archive is empty" + fi + + - id: install-and-archive + if: steps.cache.outputs.cache-hit != 'true' + uses: tecolicom/actions-install-and-archive@9d5afb27f9900f2df47fe40de58fbd837032bddf # v1.3 + with: + run: ${{ inputs.run }} + archive: ${{ steps.setup.outputs.archive }} + path: ${{ inputs.path }} + sudo: ${{ inputs.sudo }} diff --git a/.github/actions/use-homebrew-tools/action.yml b/.github/actions/use-homebrew-tools/action.yml new file mode 100644 index 0000000000..bf8f65f1dd --- /dev/null +++ b/.github/actions/use-homebrew-tools/action.yml @@ -0,0 +1,47 @@ +# This is copied from https://github.com/tecolicom/actions-use-homebrew-tools/ +# which is Copyright 2022 Office TECOLI, LLC + +name: install-and-cache homebrew tools +description: 'GitHub Action to install and cache homebrew tools' +branding: + color: orange + icon: type + +inputs: + tools: { required: false, type: string } + key: { required: false, type: string } + path: { required: false, type: string } + cache: { required: false, type: string, default: yes } + verbose: { required: false, type: boolean, default: false } + +outputs: + cache-hit: + value: ${{ steps.update.outputs.cache-hit }} + +runs: + using: composite + steps: + + - id: setup + shell: bash + run: | + : setup use-homebrew-tools + given_key="${{ inputs.key }}" + brew_version="$(brew --version)" + echo "$brew_version" + version_key="$( echo "$brew_version" | (md5sum||md5) | awk '{print $1}' )" + key="${given_key:+$given_key-}${version_key}" + sed 's/^ *//' << END >> $GITHUB_OUTPUT + command=brew install + prefix=$(brew --prefix) + key=$key + END + + - id: update + uses: ./.github/actions/install-and-cache + with: + run: ${{ steps.setup.outputs.command }} ${{ inputs.tools }} + path: ${{ steps.setup.outputs.prefix }} ${{ inputs.path }} + key: ${{ steps.setup.outputs.key }} + cache: ${{ inputs.cache }} + verbose: ${{ inputs.verbose }} diff --git a/.github/scripts/seed_uc_table.sh b/.github/scripts/seed_uc_table.sh new file mode 100755 index 0000000000..408aaf7f3a --- /dev/null +++ b/.github/scripts/seed_uc_table.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Seed a catalog-managed Delta table in the running UC server via its `bin/uc` CLI, +# so the gated `load_table` live test has a table to read. +# +# Usage: seed_uc_table.sh +# Overridable via env: UC_DIR. +set -euo pipefail + +UC_DIR="${UC_DIR:-$HOME/unitycatalog}" +catalog="${1:?catalog required}" +schema="${2:?schema required}" +table="${3:?table required}" + +cd "$UC_DIR" +bin/uc table create \ + --full_name "$catalog.$schema.$table" \ + --columns "id INT, name STRING" \ + --table_type MANAGED diff --git a/.github/scripts/setup_unitycatalog.sh b/.github/scripts/setup_unitycatalog.sh new file mode 100755 index 0000000000..37c4611eab --- /dev/null +++ b/.github/scripts/setup_unitycatalog.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +# Clone and build the Unity Catalog server at a pinned commit so the gated +# unity-catalog-delta-client-default live integration tests can run against a real server in CI. +# +# Pinned to a commit that serves the Delta-Tables API the kernel UC crate targets. The published +# `:latest` artifacts/images predate these endpoints, so building from this commit is required. +# Bump UC_COMMIT when the server contract advances. +# +# Overridable via env: UC_REPO, UC_COMMIT, UC_DIR. +set -euo pipefail + +UC_REPO="${UC_REPO:-https://github.com/unitycatalog/unitycatalog.git}" +UC_COMMIT="${UC_COMMIT:-0f1445227bd251b386420c90136515daefa4e03d}" +UC_DIR="${UC_DIR:-$HOME/unitycatalog}" + +if [[ ! -d "$UC_DIR/.git" ]]; then + git init -q "$UC_DIR" + git -C "$UC_DIR" remote add origin "$UC_REPO" +fi + +git -C "$UC_DIR" fetch --depth 1 origin "$UC_COMMIT" +git -C "$UC_DIR" checkout -q --force FETCH_HEAD + +cd "$UC_DIR" +# Build the server jar up front so the launch step starts fast and a cache can capture it. +# `bin/start-uc-server` will reuse this jar instead of rebuilding when present. +build/sbt -info package diff --git a/.github/scripts/uc_server.sh b/.github/scripts/uc_server.sh new file mode 100755 index 0000000000..fc2981e335 --- /dev/null +++ b/.github/scripts/uc_server.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# Start or stop the Unity Catalog server built by setup_unitycatalog.sh, for the gated +# unity-catalog-delta-client-default live integration tests. +# +# Usage: uc_oss_server.sh {start|stop} +# Overridable via env: UC_DIR, UC_HOST, UC_PID_FILE, UC_LOG_FILE. +set -euo pipefail + +UC_DIR="${UC_DIR:-$HOME/unitycatalog}" +UC_HOST="${UC_HOST:-http://localhost:8080}" +PID_FILE="${UC_PID_FILE:-$UC_DIR/uc-server.pid}" +LOG_FILE="${UC_LOG_FILE:-$UC_DIR/uc-server.log}" +# Used to poll the UC server to see when it is ready to accept requests. +HEALTH_URL="$UC_HOST/api/2.1/unity-catalog/catalogs" + +start() { + cd "$UC_DIR" + + # Set a storage root so the server can allocate locations for UC catalog-managed tables; a fresh + # server has none, so the staging-tables create endpoint 400s. Point it at a local dir for CI, + # kept outside UC_DIR so created tables don't end up in the build cache. + local managed_dir="${UC_MANAGED_DIR:-${TMPDIR:-/tmp}/uc-managed-tables}" + local props="$UC_DIR/etc/conf/server.properties" + mkdir -p "$managed_dir" + sed -i '/^storage-root\.tables=/d' "$props" + echo "storage-root.tables=file://$managed_dir" >>"$props" + + echo "Starting Unity Catalog server (logs: $LOG_FILE)" + # Launch in a new session so stop() can kill the whole process group (the forked JVM child). + setsid bash -c 'echo $$ >"'"$PID_FILE"'"; exec bin/start-uc-server >"'"$LOG_FILE"'" 2>&1' & + + # Wait up to ~5 minutes: covers an incremental sbt build on first boot plus JVM startup. + for i in $(seq 1 150); do + if curl -fsS "$HEALTH_URL" >/dev/null 2>&1; then + echo "Unity Catalog server is up (after $((i * 2))s)." + return 0 + fi + if [[ -f "$PID_FILE" ]] && ! kill -0 "$(cat "$PID_FILE")" 2>/dev/null; then + echo "Server process exited before becoming healthy. Logs:" >&2 + cat "$LOG_FILE" >&2 + return 1 + fi + sleep 2 + done + + echo "Server did not become healthy within the timeout. Logs:" >&2 + cat "$LOG_FILE" >&2 + return 1 +} + +stop() { + if [[ -f "$PID_FILE" ]]; then + local pid + pid="$(cat "$PID_FILE")" + # Kill the whole process group (negative pid) to take down the forked JVM child. + kill -- -"$pid" 2>/dev/null || kill "$pid" 2>/dev/null || true + rm -f "$PID_FILE" + fi +} + +case "${1:-}" in + start) start ;; + stop) stop ;; + *) + echo "usage: $0 {start|stop}" >&2 + exit 2 + ;; +esac diff --git a/.github/workflows/auto-assign-pr.yml b/.github/workflows/auto-assign-pr.yml index 30aa86703e..0ef4f28db6 100644 --- a/.github/workflows/auto-assign-pr.yml +++ b/.github/workflows/auto-assign-pr.yml @@ -12,4 +12,4 @@ jobs: assign-author: runs-on: ubuntu-latest steps: - - uses: toshimaru/auto-author-assign@v2.1.1 + - uses: toshimaru/auto-author-assign@16f0022cf3d7970c106d8d1105f75a1165edb516 # v2.1.1 diff --git a/.github/workflows/benchmark-post-comment.yml b/.github/workflows/benchmark-post-comment.yml new file mode 100644 index 0000000000..fef3d71adf --- /dev/null +++ b/.github/workflows/benchmark-post-comment.yml @@ -0,0 +1,294 @@ +# IMPORTANT: this workflow posts a github-actions[bot] comment on PRs based +# on data produced by benchmark.yml (which runs untrusted PR code). The +# trust boundaries are non-obvious; see ../SECURITY_MODEL.md. Anyone +# editing this file -- or reviewing changes to it -- MUST read that +# document first. In particular, the `case "$EVENT"` resolver in each job +# is load-bearing: trusting the wrong source per upstream event would let +# a fork forge cross-PR comments. +# +# Companion to .github/workflows/benchmark.yml. The bench workflow runs under +# `pull_request` / `push` / `merge_group` / `issue_comment` with `contents: read` +# only -- it has no write scope on PR-triggered runs from forks (read-only +# GITHUB_TOKEN per GitHub's default), so it cannot post a comment itself. +# Instead, run-benchmarks.sh writes the comment body to /tmp/bench-comment.md, +# benchmark.yml uploads that file as the `bench-comment` artifact, and this +# workflow (running in base-branch context, which does have write scope) +# downloads it and posts/updates the PR comment. +# +# Two jobs, routed by whether the `bench-comment` artifact exists: +# +# bench succeeds: `bench-comment` uploaded +# -> post-comment posts the table; post-failure-comment skips +# bench fails +# ├─ regression gate tripped (a benchmark >=15% slower): benchmark.yml +# │ uploads `bench-comment` before the gate runs, so it exists despite +# │ the failed job +# │ -> post-comment posts the table; post-failure-comment skips +# └─ broke before writing /tmp/bench-comment.md (e.g. compile error) +# -> no artifact; post-failure-comment posts a static failure note +# +# Pattern is identical to pr-validator.yml + comment-on-title-failure.yml. +# +# CAVEAT: workflow_run only fires for workflow definitions that live on the +# default branch. A PR that modifies either this workflow or benchmark.yml +# won't trigger this post workflow until the changes are merged to main. Test +# end-to-end on a fork/staging repo, or accept a one-PR window of no comments +# right after merge. + +name: Post benchmark results + +on: + workflow_run: + workflows: ["Benchmarking PR performance"] + types: [completed] + +# Empty default; each job opts into the minimum scopes it needs. +permissions: {} + +# Two rapid runs of workflow A (e.g., a quick push followed by a /bench +# comment) can both complete near-simultaneously; without concurrency, both +# instances of this workflow race on find-comment / create-or-update-comment +# and may double-post. Key on the PR identity (head repo + head branch), +# not workflow_run.id, since the latter is unique per upstream run and would +# put every instance in its own group. For issue_comment upstream events +# head_branch is the default branch, which over-serializes /bench comments +# across different PRs through one group -- acceptable since posting is +# fast and we don't cancel-in-progress here. +concurrency: + group: bench-comment-${{ github.event.workflow_run.head_repository.full_name }}-${{ github.event.workflow_run.head_branch }} + cancel-in-progress: false + +jobs: + post-comment: + name: Post benchmark results + # Post when the upstream succeeded OR failed (a failure is usually the + # regression gate tripping, and the result table still exists), and the + # upstream event was a PR event with a conversation to post to. + # `merge_group` and `push: main` runs of workflow A have no PR to comment on. + if: > + contains(fromJSON('["success","failure","timed_out"]'), github.event.workflow_run.conclusion) + && (github.event.workflow_run.event == 'pull_request' + || github.event.workflow_run.event == 'issue_comment') + runs-on: ubuntu-latest + permissions: + pull-requests: write + # download-artifact across workflows needs actions:read. + actions: read + steps: + - name: Download bench-pr-number (issue_comment upstream only) + # See SECURITY MODEL at the top of this file -- this artifact is + # trusted only on the issue_comment path. continue-on-error so an + # absent artifact (e.g., a non-/bench issue_comment that skipped + # run-benchmark) flows through to the resolve step rather than + # failing the whole workflow. + id: download-num + if: github.event.workflow_run.event == 'issue_comment' + continue-on-error: true + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 + with: + name: bench-pr-number + path: /tmp/ + run-id: ${{ github.event.workflow_run.id }} + github-token: ${{ github.token }} + - name: Resolve PR number + id: pr + env: + EVENT: ${{ github.event.workflow_run.event }} + GH_TOKEN: ${{ github.token }} + REPO: ${{ github.repository }} + # Conditional `owner:branch` for fork PRs vs same-repo (copied from + # comment-on-title-failure.yml's resolver). + PR_BRANCH: |- + ${{ + (github.event.workflow_run.head_repository.owner.login != github.event.workflow_run.repository.owner.login) + && format('{0}:{1}', github.event.workflow_run.head_repository.owner.login, github.event.workflow_run.head_branch) + || github.event.workflow_run.head_branch + }} + run: | + set -u + # Step 1: pick the trusted source for the PR number based on which + # upstream event fired. See ../SECURITY_MODEL.md for why each + # branch is safe -- they trust different things for different reasons. + case "$EVENT" in + pull_request) + # Look up the PR by its head branch. head_branch and + # head_repository come from the workflow_run payload (set by + # GitHub, not the fork), so this lookup is trusted. + NUM=$(gh pr view --repo "$REPO" "$PR_BRANCH" --json number --jq .number || true) + ;; + issue_comment) + # Read the PR number from the bench-pr-number artifact. The + # upstream YAML uploaded this before PR-controlled code ran on + # the runner (for issue_comment events GitHub serves + # benchmark.yml from the default branch -- trusted). + if [[ ! -f /tmp/pr-number.txt ]]; then + echo "bench-pr-number artifact missing (likely a non-/bench comment that skipped run-benchmark); nothing to post." >&2 + echo "skip=true" >> "$GITHUB_OUTPUT" + exit 0 + fi + NUM=$(tr -d '[:space:]' < /tmp/pr-number.txt) + ;; + *) + echo "Unexpected upstream event: $EVENT" >&2 + echo "skip=true" >> "$GITHUB_OUTPUT" + exit 0 + ;; + esac + # Step 2: validate before emitting -- catches gh-lookup failures + # (empty NUM) and any malformed artifact contents. + if [[ -z "$NUM" || ! "$NUM" =~ ^[1-9][0-9]*$ ]]; then + echo "Could not resolve PR number ('$NUM'); skipping." >&2 + echo "skip=true" >> "$GITHUB_OUTPUT" + else + printf 'number=%s\n' "$NUM" >> "$GITHUB_OUTPUT" + fi + - name: Download bench comment + # continue-on-error: a successful upstream run may still have skipped + # run-benchmark (e.g., a draft PR push satisfies the `pull_request` + # trigger but the job's `if:` excludes drafts). In that case there is + # no body artifact, and we should exit cleanly without touching the + # existing PR comment. + id: download-body + if: steps.pr.outputs.skip != 'true' + continue-on-error: true + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 + with: + name: bench-comment + path: /tmp/ + run-id: ${{ github.event.workflow_run.id }} + github-token: ${{ github.token }} + - name: Find existing bench comment + id: find + if: steps.pr.outputs.skip != 'true' && steps.download-body.outcome == 'success' + uses: peter-evans/find-comment@3eae4d37986fb5a8592848f6a574fdf654e61f9e # v3.1.0 + with: + issue-number: ${{ steps.pr.outputs.number }} + comment-author: 'github-actions[bot]' + body-includes: '' + - name: Post or update PR comment + # When `comment-id` is empty (no prior comment) this creates one; + # otherwise it replaces the existing body. Either way only one bench + # comment exists on the PR. + if: steps.pr.outputs.skip != 'true' && steps.download-body.outcome == 'success' + uses: peter-evans/create-or-update-comment@71345be0265236311c031f5c7866368bd1eff043 # v4.0.0 + with: + comment-id: ${{ steps.find.outputs.comment-id }} + issue-number: ${{ steps.pr.outputs.number }} + edit-mode: replace + body-path: /tmp/bench-comment.md + + post-failure-comment: + name: Post benchmark failure note + # Replace any existing success comment with a failure note so reviewers + # don't see stale-green after a push that breaks the bench itself. + # `timed_out` counts as failure for our purposes; we skip `cancelled` (a + # superseding run will post its own update) and `skipped` (nothing + # meaningful happened, e.g., draft). + if: > + contains(fromJSON('["failure","timed_out"]'), github.event.workflow_run.conclusion) + && (github.event.workflow_run.event == 'pull_request' + || github.event.workflow_run.event == 'issue_comment') + runs-on: ubuntu-latest + permissions: + pull-requests: write + actions: read + steps: + - name: Download bench-pr-number (issue_comment upstream only) + # Same security rationale as the post-comment job. Trusted only on + # the issue_comment path. On the pull_request path we use gh pr view + # against the trusted workflow_run.head_branch instead. + id: download-num + if: github.event.workflow_run.event == 'issue_comment' + continue-on-error: true + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 + with: + name: bench-pr-number + path: /tmp/ + run-id: ${{ github.event.workflow_run.id }} + github-token: ${{ github.token }} + - name: Resolve PR number + id: pr + env: + EVENT: ${{ github.event.workflow_run.event }} + GH_TOKEN: ${{ github.token }} + REPO: ${{ github.repository }} + PR_BRANCH: |- + ${{ + (github.event.workflow_run.head_repository.owner.login != github.event.workflow_run.repository.owner.login) + && format('{0}:{1}', github.event.workflow_run.head_repository.owner.login, github.event.workflow_run.head_branch) + || github.event.workflow_run.head_branch + }} + run: | + set -u + # Step 1: pick the trusted source for the PR number based on which + # upstream event fired. See ../SECURITY_MODEL.md for why each + # branch is safe -- they trust different things for different reasons. + case "$EVENT" in + pull_request) + # Look up the PR by its head branch. head_branch and + # head_repository come from the workflow_run payload (set by + # GitHub, not the fork), so this lookup is trusted. + NUM=$(gh pr view --repo "$REPO" "$PR_BRANCH" --json number --jq .number || true) + ;; + issue_comment) + # Read the PR number from the bench-pr-number artifact. The + # upstream YAML uploaded this before PR-controlled code ran on + # the runner (for issue_comment events GitHub serves + # benchmark.yml from the default branch -- trusted). + if [[ ! -f /tmp/pr-number.txt ]]; then + echo "bench-pr-number artifact missing; cannot post failure note." >&2 + echo "skip=true" >> "$GITHUB_OUTPUT" + exit 0 + fi + NUM=$(tr -d '[:space:]' < /tmp/pr-number.txt) + ;; + *) + echo "Unexpected upstream event: $EVENT" >&2 + echo "skip=true" >> "$GITHUB_OUTPUT" + exit 0 + ;; + esac + # Step 2: validate before emitting -- catches gh-lookup failures + # (empty NUM) and any malformed artifact contents. + if [[ -z "$NUM" || ! "$NUM" =~ ^[1-9][0-9]*$ ]]; then + echo "Could not resolve PR number ('$NUM'); skipping failure comment." >&2 + echo "skip=true" >> "$GITHUB_OUTPUT" + else + printf 'number=%s\n' "$NUM" >> "$GITHUB_OUTPUT" + fi + - name: Probe for bench comment artifact + # Detect whether the bench produced a comment artifact; continue-on-error + # so "absent" is a clean outcome rather than a job failure. The find/post + # steps below run only when it's absent: when present, the failure was the + # gate tripping and post-comment owns the comment (see the header routing). + id: probe-body + if: steps.pr.outputs.skip != 'true' + continue-on-error: true + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 + with: + name: bench-comment + path: /tmp/ + run-id: ${{ github.event.workflow_run.id }} + github-token: ${{ github.token }} + - name: Find existing bench comment + id: find + if: steps.pr.outputs.skip != 'true' && steps.probe-body.outcome != 'success' + uses: peter-evans/find-comment@3eae4d37986fb5a8592848f6a574fdf654e61f9e # v3.1.0 + with: + issue-number: ${{ steps.pr.outputs.number }} + comment-author: 'github-actions[bot]' + body-includes: '' + - name: Post or update failure note + if: steps.pr.outputs.skip != 'true' && steps.probe-body.outcome != 'success' + uses: peter-evans/create-or-update-comment@71345be0265236311c031f5c7866368bd1eff043 # v4.0.0 + env: + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.event.workflow_run.id }} + with: + comment-id: ${{ steps.find.outputs.comment-id }} + issue-number: ${{ steps.pr.outputs.number }} + edit-mode: replace + body: | + + ## Benchmark results: ❌ Fail + + Benchmark run failed before producing results. [See workflow run](${{ env.RUN_URL }}) for details. diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml index 32e8140537..2a2757d74c 100644 --- a/.github/workflows/benchmark.yml +++ b/.github/workflows/benchmark.yml @@ -1,27 +1,253 @@ -on: [pull_request_target] +# IMPORTANT: this workflow runs untrusted PR code and has a non-obvious +# security model documented at ../SECURITY_MODEL.md. Anyone editing this +# file -- or reviewing changes to it -- MUST read that document first. +# Several constraints below (permissions blocks, save-if expressions, the +# step ordering around actions/checkout, the bench-pr-number upload) look +# like style choices but are load-bearing. +# +# Benchmarks run on four triggers: +# - pull_request: every push to a non-draft PR auto-runs with BENCH_TAGS=base +# - push (main only): warms the rust-cache so PR runs restore a warm target/ +# - issue_comment: a /bench [--tags ...] [--filter ...] comment runs with +# the supplied flags (including on draft PRs) +# - merge_group: declared so this workflow can be added to branch protection +# as a required status check later. Currently a no-op. +# Comment posting lives in a separate workflow_run-triggered workflow at +# .github/workflows/benchmark-post-comment.yml (mirroring the pr-validator / +# comment-on-title-failure split). This workflow uploads /tmp/bench-comment.md +# as an artifact; that workflow downloads it and posts/updates the PR comment +# in base-branch context (which has the write scope this workflow lacks under +# pull_request from a fork). +# +# The run-benchmark job fails if any benchmark regresses past the fail +# threshold in benchmarks/ci/parse_critcmp.py, unless the PR carries the +# ignore-benchmark-failure label. The comment artifact is uploaded before that +# gate runs, so the result table still gets posted on a gate failure. +on: + pull_request: + types: [opened, synchronize, reopened, ready_for_review] + push: + branches: [main] + merge_group: + issue_comment: + types: [created, edited] + name: Benchmarking PR performance + +# Start from an empty default; each job opts into the minimum scopes it needs. +# Adding a new job without thinking about permissions inherits zero, not full. +permissions: {} + jobs: - runBenchmark: + warm-bench-cache: + # On every push to main, compile the bench harness (no measurement) so + # rust-cache saves a warm target/ under main's scope. PR runs then restore + # this entry instead of compiling cold every time. Rapid main pushes + # dedupe via the concurrency group. + name: Warm bench cache + if: github.event_name == 'push' && github.ref == 'refs/heads/main' + concurrency: + group: warm-bench-cache + cancel-in-progress: true + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + with: + persist-credentials: false + - uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 + with: + cache: false + # shared-key keeps this and `run-benchmark` in the same cache namespace + # so PR runs can restore what we save here. + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + cache-bin: false + shared-key: bench + save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} + - name: Compile bench harness + run: (cd benchmarks && cargo bench --locked --bench workload_bench --no-run) + + bench-merge-queue-noop: + # Placeholder job for the merge_group trigger so the workflow can be wired + # up as a required status check later without restructuring. Does nothing + # meaningful today (see top-level comment for rationale). + name: Bench (merge queue no-op) + if: github.event_name == 'merge_group' + runs-on: ubuntu-latest + permissions: {} + steps: + - run: echo "Benchmark workflow is a no-op on merge queue events." + + run-benchmark: name: Run benchmarks + # Auto-trigger: PR push events, skipping drafts. Comment-trigger: a + # literal /bench, optionally followed by args (e.g. /bench --tags base). + if: > + (github.event_name == 'pull_request' && github.event.pull_request.draft == false) + || (github.event_name == 'issue_comment' && github.event.issue.pull_request && + (github.event.comment.body == '/bench' || startsWith(github.event.comment.body, '/bench '))) + # Job-level (not workflow-level) so unrelated PR comments don't join the + # group and cancel an in-flight bench via cancel-in-progress. + concurrency: + group: bench-${{ github.event.pull_request.number || github.event.issue.number }} + cancel-in-progress: true runs-on: ubuntu-latest permissions: - pull-requests: write + contents: read + outputs: + pr_number: ${{ steps.pr.outputs.pr_number }} steps: - - uses: actions/checkout@v4 + - name: Get PR metadata + id: pr + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + REPO: ${{ github.repository }} + EVENT_NAME: ${{ github.event_name }} + ISSUE_NUMBER: ${{ github.event.issue.number }} + PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }} + PR_BASE_REF: ${{ github.event.pull_request.base.ref }} + PR_NUMBER: ${{ github.event.pull_request.number }} + run: | + # GitHub Actions runs bash with -eo pipefail by default; -u catches + # typos / unset variables. All four fields below are then validated + # against strict allowlists before being used or emitted -- the PR + # head sha and base ref are attacker-controllable (fork PR author), + # and the issue/PR number is concatenated into a gh api URL. + set -u + if [[ "$EVENT_NAME" == "issue_comment" ]]; then + # /bench path: the issue_comment payload doesn't include the PR + # head/base, so look it up via the API. Validate the number first + # so a malformed value cannot inject path segments into the URL. + [[ "$ISSUE_NUMBER" =~ ^[1-9][0-9]*$ ]] \ + || { echo "Invalid ISSUE_NUMBER: $ISSUE_NUMBER" >&2; exit 1; } + PR_DATA=$(gh api "repos/$REPO/pulls/$ISSUE_NUMBER") + HEAD_SHA=$(echo "$PR_DATA" | jq -r .head.sha) + BASE_REF=$(echo "$PR_DATA" | jq -r .base.ref) + NUM="$ISSUE_NUMBER" + else + # Auto-trigger path: pull_request payload has everything. + HEAD_SHA="$PR_HEAD_SHA" + BASE_REF="$PR_BASE_REF" + NUM="$PR_NUMBER" + [[ "$NUM" =~ ^[1-9][0-9]*$ ]] \ + || { echo "Invalid PR_NUMBER: $NUM" >&2; exit 1; } + fi + # HEAD_SHA must be a hex Git SHA (40 char SHA-1 or 64 char SHA-256). + # The allowlist also catches the empty-string case if jq somehow + # returned no value. + [[ "$HEAD_SHA" =~ ^[a-f0-9]{40,64}$ ]] \ + || { echo "Invalid HEAD_SHA: $HEAD_SHA" >&2; exit 1; } + # BASE_REF is a Git ref name; standard ref-name allowlist (letters, + # digits, slash, dot, dash, underscore). Both regexes exclude + # newlines, so no separate newline check is needed. + [[ "$BASE_REF" =~ ^[a-zA-Z0-9/_.-]+$ ]] \ + || { echo "Invalid BASE_REF: $BASE_REF" >&2; exit 1; } + printf 'head_sha=%s\n' "$HEAD_SHA" >> "$GITHUB_OUTPUT" + printf 'base_ref=%s\n' "$BASE_REF" >> "$GITHUB_OUTPUT" + printf 'pr_number=%s\n' "$NUM" >> "$GITHUB_OUTPUT" + # Read labels here, in trusted pre-checkout context, so the + # regression gate's override decision can't be influenced by PR code + # that runs later. Capture into a variable before matching: + # `gh ... | grep -q` would let grep close the pipe early and trip + # pipefail. On any read error LABELS is empty and the gate stays + # enforced. + LABELS=$(gh pr view "$NUM" --repo "$REPO" --json labels --jq '.labels[].name' || true) + if grep -qx 'ignore-benchmark-failure' <<< "$LABELS"; then + printf 'ignore_failure=true\n' >> "$GITHUB_OUTPUT" + else + printf 'ignore_failure=false\n' >> "$GITHUB_OUTPUT" + fi + - name: Stash PR number for the post-comment workflow + # Written to /tmp/pr-number.txt and uploaded as its own artifact in + # the next step. The two-step shape (write then upload) keeps the + # value out of /tmp by the time PR-controlled code runs below; see + # the upload step for the security rationale. + env: + PR_NUMBER: ${{ steps.pr.outputs.pr_number }} + run: printf '%s\n' "$PR_NUMBER" > /tmp/pr-number.txt + - name: Upload PR number (trusted, pre-checkout) + # Uploaded BEFORE checking out the PR head so the artifact's contents + # cannot be tampered with by PR-controlled code that runs in later + # steps. Once stored on GitHub Actions storage the artifact is + # immutable, and this job's contents:read token has no actions:write + # scope, so the bench script cannot delete or replace it. + # + # The post-comment workflow trusts this artifact only for the + # `issue_comment` upstream path, where benchmark.yml itself is + # served from the default branch (trusted) -- the fork cannot + # influence what gets written. For `pull_request` upstream the + # workflow YAML is fork-controlled, so the post-comment workflow + # ignores this artifact and derives the PR number from the trusted + # workflow_run.head_branch instead. + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: bench-pr-number + path: /tmp/pr-number.txt + - name: Install critcmp + # Installed before checkout so the PR's .cargo/config.toml cannot + # redirect the registry to a malicious source. The runner's + # pre-installed Rust is sufficient -- no toolchain setup needed here. + # --locked is omitted for cargo install (same exemption as cargo miri + # setup); --version pins the top-level crate. + run: cargo install critcmp --version 0.1.8 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 with: - ref: ${{ github.event.pull_request.head.sha }} - - uses: actions-rust-lang/setup-rust-toolchain@v1 + # Check out the integration commit (PR head merged into base) that + # GitHub maintains at refs/pull//merge -- the same ref build.yml + # and the test jobs resolve to via their bare checkout. Benchmarking + # this measures the PR as it would land, not the raw head. An + # unmergeable PR has no maintained merge ref, so the checkout fails + # and the job stops (parity with build/test; see SECURITY_MODEL.md). + ref: refs/pull/${{ steps.pr.outputs.pr_number }}/merge + persist-credentials: false + - uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 with: cache: false - # See build.yml top-level comment for why save-if is restricted to main. - - uses: Swatinem/rust-cache@v2 + # See build.yml top-level comment for why save-if is restricted to main + # and why cache-bin is disabled. `shared-key: bench` keeps this aligned + # with `warm-bench-cache` so PR runs restore the entry that main pushes + # save -- without the shared-key, each job's cache lives in a separate + # namespace and the warming would not benefit PRs. + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: + cache-bin: false + shared-key: bench save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} - - uses: boa-dev/criterion-compare-action@v3 + - name: Run benchmarks + # The benchmark-post-comment workflow picks up the uploaded artifact + # and posts the PR comment in base-branch context (which has the + # write scope this workflow lacks on fork PRs). + # COMMENT is only meaningful under issue_comment; under pull_request + # the expression resolves to "" and the script treats that as bare /bench. env: - BENCH_TAGS: base + COMMENT: ${{ github.event.comment.body }} + TRIGGER: ${{ github.event_name == 'issue_comment' && '/bench' || 'auto-push' }} + BASE_REF: ${{ steps.pr.outputs.base_ref }} + HEAD_SHA: ${{ steps.pr.outputs.head_sha }} + BENCH_IGNORE_FAILURE: ${{ steps.pr.outputs.ignore_failure }} + run: bash benchmarks/ci/run-benchmarks.sh + - name: Upload bench comment + # Default `if: success()` -- a bench failure leaves no body to post. + # Runs before the regression gate below so the artifact exists even + # when the gate fails the job. + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: - token: ${{ secrets.GITHUB_TOKEN }} - branchName: ${{ github.base_ref }} - cwd: benchmarks - benchName: workload_bench + name: bench-comment + path: /tmp/bench-comment.md + - name: Enforce benchmark regression gate + # parse_critcmp.py wrote "true" to this file iff some benchmark crossed + # its fail threshold. Fail the job on a regression unless the PR opted + # out with the ignore-benchmark-failure label (resolved pre-checkout in + # the trusted metadata step). + env: + IGNORE_FAILURE: ${{ steps.pr.outputs.ignore_failure }} + run: | + set -u + REGRESSED=$(tr -d '[:space:]' < /tmp/bench-regression.txt 2>/dev/null || echo false) + if [[ "$REGRESSED" == "true" && "$IGNORE_FAILURE" != "true" ]]; then + echo "::error::A benchmark regressed by at least 15%. Add the 'ignore-benchmark-failure' label to override this gate." >&2 + exit 1 + fi + echo "Regression gate passed (regressed=$REGRESSED, ignore_failure=$IGNORE_FAILURE)." diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index e1815594f7..731688f1f5 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -1,11 +1,33 @@ name: build -on: [push, pull_request, merge_group] +# Restrict push to long-lived branches to avoid duplicate CI runs. See .github/CLAUDE.md. +on: + push: + branches: [main, "release/**"] + pull_request: + merge_group: + +concurrency: + # A new push to a PR cancels that PR's in-flight run. On push to main/release, head_ref + # is empty, so run_id (always unique) is used and those runs never cancel each other. + group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} + cancel-in-progress: true env: CARGO_TERM_COLOR: always RUST_BACKTRACE: 1 + # Strip debuginfo (debugger source-line/variable maps, type layouts) in CI: no debugger + # attaches here, so it is pure compile overhead. Test output, logs, and panics are unaffected. + CARGO_PROFILE_DEV_DEBUG: "0" + CARGO_PROFILE_TEST_DEBUG: "0" +# Supply chain security: all cargo commands that resolve dependencies use --locked to +# enforce the committed Cargo.lock. This prevents CI from silently resolving a newer +# (potentially compromised) dependency version. If Cargo.lock is out of sync with +# Cargo.toml, the build fails immediately. Any dependency change must be an explicit, +# reviewable update to Cargo.lock in the PR. Commands that skip --locked: cargo +nightly fmt +# (no dep resolution), cargo msrv verify/show (wrapper tool), cargo miri setup (tooling). +# # Swatinem/rust-cache caches the cargo registry and target directory (~450MB per job). # save-if restricts cache writes to main pushes only. PRs read from main's cache but # never write their own entries. @@ -17,43 +39,56 @@ env: # # Without this, GHA's ref-scoped caching works against us: each PR writes ~6.3GB of # cache entries (14 jobs x ~450MB) that only that PR can read. A handful of active PRs -# fills the 10GB cache budget, LRU evicts main's shared entries, and every subsequent -# PR compiles from scratch. +# fills the 50GB cache budget (as of 2026-06), LRU evicts main's shared entries, and +# every subsequent PR compiles from scratch. # # The save-if condition checks both event_name == 'push' and ref == main because -# pull_request_target events (used by benchmark.yml, semver-checks.yml) set github.ref -# to the base branch (main), not the PR branch. Without the event_name check, those -# workflows would write cache entries on every PR. +# pull_request_target events set github.ref to the base branch (main), not the PR +# branch. Without the event_name check, those workflows would write cache entries on +# every PR. # # Note: actions-rust-lang/setup-rust-toolchain has built-in Swatinem/rust-cache that # writes on every run with no save-if support. We disable it with cache: false and # manage caching explicitly via the Swatinem/rust-cache steps below. +# +# cache-bin: false disables caching of ~/.cargo/bin to work around +# https://github.com/Swatinem/rust-cache/issues/341, where the cached rustup binary +# breaks cargo on the new macOS runner image. Drop the cache-bin: false lines once +# that issue is resolved and we've bumped to a fixed version. Skipping this cache is +# essentially free for us: all cargo tools (cargo-nextest, cargo-msrv, cargo-llvm-cov) +# are installed via taiki-e/install-action, which fetches prebuilt binaries. jobs: format: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 - - name: Install minimal stable with rustfmt - uses: actions-rust-lang/setup-rust-toolchain@v1 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + - name: Install nightly with rustfmt + uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 with: cache: false + toolchain: nightly components: rustfmt - name: format - run: cargo fmt -- --check + run: cargo +nightly fmt -- --check + # Separate workspace; the root fmt above does not reach it. See datafusion-executor/Cargo.toml. + - name: format datafusion-executor + working-directory: datafusion-executor + run: cargo +nightly fmt -- --check msrv: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 - name: Install minimal stable and cargo msrv - uses: actions-rust-lang/setup-rust-toolchain@v1 + uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 with: cache: false - - uses: Swatinem/rust-cache@v2 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: + cache-bin: false save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} - - uses: taiki-e/install-action@v2 + - uses: taiki-e/install-action@7bc99eee1f1b8902a125006cf790a1f4c8461e63 # v2.69.8 with: tool: cargo-msrv - name: verify-msrv @@ -65,23 +100,24 @@ jobs: msrv-run-tests: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 - name: Install minimal stable and cargo msrv - uses: actions-rust-lang/setup-rust-toolchain@v1 + uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 with: cache: false - - uses: Swatinem/rust-cache@v2 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: + cache-bin: false save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} - - uses: taiki-e/install-action@v2 + - uses: taiki-e/install-action@7bc99eee1f1b8902a125006cf790a1f4c8461e63 # v2.69.8 with: tool: cargo-msrv - - uses: taiki-e/install-action@nextest + - uses: taiki-e/install-action@98ec31d284eb962f41c14065e9391a955aa810cf # nextest - name: Get rust-version from Cargo.toml id: rust-version run: echo "RUST_VERSION=$(cargo msrv show --path kernel/ --output-format minimal)" >> $GITHUB_ENV - name: Install specified rust version - uses: actions-rust-lang/setup-rust-toolchain@v1 + uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 with: cache: false toolchain: ${{ env.RUST_VERSION }} @@ -89,22 +125,24 @@ jobs: run: | pushd kernel echo "Testing with $(cargo msrv show --output-format minimal)" - cargo +$(cargo msrv show --output-format minimal) nextest run + cargo +$(cargo msrv show --output-format minimal) nextest run --locked + cargo +$(cargo msrv show --output-format minimal) test --doc docs: runs-on: ubuntu-latest env: RUSTDOCFLAGS: -D warnings steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 - name: Install minimal stable - uses: actions-rust-lang/setup-rust-toolchain@v1 + uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 with: cache: false - - uses: Swatinem/rust-cache@v2 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: + cache-bin: false save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} - name: build docs - run: cargo doc --workspace --all-features --no-deps + run: cargo doc --locked --workspace --all-features --no-deps # When we run cargo { build, clippy } --no-default-features, we want to build/lint the kernel to # ensure that we can build the kernel without any features enabled. Unfortunately, due to how @@ -116,6 +154,7 @@ jobs: # 1. build/clippy all packages that depend on the kernel with some features enabled: # - acceptance # - test_utils + # - delta_kernel_workloads # - feature_tests # (and examples) # - inspect-table @@ -136,26 +175,42 @@ jobs: - ubuntu-latest - windows-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 - name: Install minimal stable with clippy - uses: actions-rust-lang/setup-rust-toolchain@v1 + uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 with: cache: false components: clippy - - uses: Swatinem/rust-cache@v2 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: + cache-bin: false save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} - name: build and lint with clippy - run: cargo clippy --benches --tests --all-features -- -D warnings + run: cargo clippy --locked --benches --tests --all-features -- -D warnings - name: lint without default features - packages which depend on kernel with features enabled - run: cargo clippy --workspace --no-default-features --exclude delta_kernel --exclude delta_kernel_ffi --exclude delta_kernel_derive --exclude delta_kernel_ffi_macros -- -D warnings + run: cargo clippy --locked --workspace --no-default-features --exclude delta_kernel --exclude delta_kernel_ffi --exclude delta_kernel_derive --exclude delta_kernel_ffi_macros -- -D warnings - name: lint without default features - packages which don't depend on kernel with features enabled - run: cargo clippy --no-default-features --package delta_kernel --package delta_kernel_ffi --package delta_kernel_derive --package delta_kernel_ffi_macros -- -D warnings + run: cargo clippy --locked --no-default-features --package delta_kernel --package delta_kernel_ffi --package delta_kernel_derive --package delta_kernel_ffi_macros -- -D warnings + - name: check kernel declarative-plans feature-only build + run: cargo check --locked -p delta_kernel --no-default-features --features declarative-plans,vendored-protoc + - name: lint kernel declarative-plans feature-only build + run: cargo clippy --locked -p delta_kernel --no-default-features --features declarative-plans,vendored-protoc --tests -- -D warnings - name: check kernel builds with default-engine-native-tls - run: cargo build -p feature_tests --features default-engine-native-tls + run: cargo build --locked -p feature_tests --features default-engine-native-tls + - name: test native-tls backend has no crypto provider conflict + run: cargo test --locked -p feature_tests --features default-engine-native-tls - name: check kernel builds with default-engine-rustls - run: cargo build -p feature_tests --features default-engine-rustls + run: cargo build --locked -p feature_tests --features default-engine-rustls + - name: test rustls TLS backend feature-tests + run: cargo test --locked -p feature_tests --features default-engine-rustls + # Runs the test suite under two feature configurations, one per matrix leg, so they build and + # run in parallel: `all-features` is the full workspace suite; `no-declarative-plans` runs + # kernel + acceptance with every kernel feature except declarative-plans, to exercise the + # cfg(not(feature = "declarative-plans")) paths and the feature-off build. The job `name` + # keeps the all-features leg reporting as `test ()` so it stays the same required status + # check, and the cache `key` keeps each configuration's target dir in its own cache entry. test: + name: test (${{ matrix.os }}${{ matrix.config == 'no-declarative-plans' && ', no-decl' || '' }}) runs-on: ${{ matrix.os }} strategy: matrix: @@ -163,9 +218,12 @@ jobs: - macOS-latest - ubuntu-latest - windows-latest + config: + - all-features + - no-declarative-plans steps: - - uses: actions/checkout@v4 - - uses: dorny/paths-filter@v3 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + - uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2 id: filter with: filters: | @@ -173,18 +231,58 @@ jobs: - 'ffi/src/handle.rs' - 'ffi-proc-macros/**' - name: Install minimal stable with clippy and rustfmt - uses: actions-rust-lang/setup-rust-toolchain@v1 + uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 with: cache: false - - uses: Swatinem/rust-cache@v2 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: + cache-bin: false + key: ${{ matrix.config }} + save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} + - uses: taiki-e/install-action@98ec31d284eb962f41c14065e9391a955aa810cf # nextest + - name: all-features tests + if: matrix.config == 'all-features' + shell: bash + env: + FFI_CHANGED: ${{ steps.filter.outputs.ffi }} + run: | + cargo nextest run --locked --workspace --all-features -E 'not test(read_table_version_hdfs) and not test(invalid_handle_code)' + cargo test --locked --workspace --all-features --doc + if [ "$FFI_CHANGED" = "true" ]; then + cargo test --locked --package delta_kernel_ffi --features internal-api -- invalid_handle_code + fi + output=$(cargo run --profile test --example metrics-tester) + grep -q 'invalid_field' <<< "$output" + grep -q 'Invalid uuid' <<< "$output" + - name: no-declarative-plans tests + if: matrix.config == 'no-declarative-plans' + shell: bash + run: | + cargo nextest run --locked -p delta_kernel -p acceptance --no-default-features --features internal-api,prettyprint,test-utils,integration-test,arrow,arrow-conversion,arrow-expression,schema-diff,check-constraints-in-dev,adaptive-metadata-in-dev,default-engine-base -E 'not test(read_table_version_hdfs) and not test(invalid_handle_code)' + cargo test --locked -p delta_kernel -p acceptance --no-default-features --features internal-api,prettyprint,test-utils,integration-test,arrow,arrow-conversion,arrow-expression,schema-diff,check-constraints-in-dev,adaptive-metadata-in-dev,default-engine-base --doc + + # Fails red on trybuild drift, but kept out of required checks so it never blocks a merge. + invalid-handle-code: + if: github.event_name != 'merge_group' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + - name: Install minimal stable + uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 + with: + cache: false + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + cache-bin: false save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} - - uses: taiki-e/install-action@nextest - - name: test - run: cargo nextest run --workspace --all-features -E 'not test(read_table_version_hdfs) and not test(invalid_handle_code)' - name: trybuild tests - if: steps.filter.outputs.ffi == 'true' - run: cargo test --package delta_kernel_ffi --features --internal-api invalid_handle_code + run: cargo test --locked --package delta_kernel_ffi --features internal-api -- invalid_handle_code + - name: notify on failure + if: failure() + run: | + echo "::warning title=invalid_handle_code trybuild tests failed::Check whether the compile_fail expectations need updating for the current rustc, or whether handle.rs/ffi-proc-macros regressed." + echo "## invalid_handle_code trybuild tests failed" >> "$GITHUB_STEP_SUMMARY" + echo "Check whether the compile_fail expectations need updating for the current rustc, or whether \`ffi/src/handle.rs\` / \`ffi-proc-macros\` regressed." >> "$GITHUB_STEP_SUMMARY" ffi_test: runs-on: ${{ matrix.os }} @@ -194,9 +292,9 @@ jobs: - macOS-latest - ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 - name: Setup cmake - uses: jwlawson/actions-setup-cmake@v2 + uses: jwlawson/actions-setup-cmake@0d6a7d60b009d01c9e7523be22153ff8f19460d3 # v2.2.0 with: cmake-version: "3.30.x" - name: Install arrow-glib-linux @@ -213,27 +311,30 @@ jobs: sudo apt install -y -V libarrow-glib-dev # For GLib (C) sudo apt install -y -V valgrind # For memory leak test fi + # TODO (#2707): This step's cache misses 100% of runs, wasting ~4.5 min per macOS run on a + # fresh `brew install`. See the issue for root cause and fix options. - name: Install arrow-glib-macOS if: runner.os == 'macOS' - uses: tecolicom/actions-use-homebrew-tools@v1 + uses: ./.github/actions/use-homebrew-tools with: tools: "apache-arrow apache-arrow-glib" - name: Install minimal stable - uses: actions-rust-lang/setup-rust-toolchain@v1 + uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 with: cache: false - - uses: Swatinem/rust-cache@v2 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: + cache-bin: false save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} - name: Set output on fail run: echo "CTEST_OUTPUT_ON_FAILURE=1" >> "$GITHUB_ENV" - name: Build kernel run: | pushd acceptance - cargo build + cargo build --locked popd pushd ffi - cargo b --features default-engine-rustls,test-ffi,tracing,uc-catalog + cargo build --locked --features default-engine-rustls,test-ffi,tracing,delta-kernel-unity-catalog popd - name: build and run read-table test run: | @@ -251,9 +352,51 @@ jobs: cmake .. make make test - - name: build and run uc-catalog-ffi test + - name: build and run delta-kernel-unity-catalog-ffi test + run: | + pushd ffi/examples/delta-kernel-unity-catalog-example + mkdir build + pushd build + cmake .. + make + make test + - name: build and run create-table test + run: | + pushd ffi/examples/create-table + mkdir build + pushd build + cmake .. + make + make test + # NOTE: write-table's ctest seeds its target table by invoking the create-table + # binary, so create-table must be built first (its build/ dir is preserved by the + # preceding step and write-table's CMakeLists references it via a relative path). + - name: build and run write-table test + run: | + pushd ffi/examples/write-table + mkdir build + pushd build + cmake .. + make + make test + - name: build update-dv example run: | - pushd ffi/examples/uc-catalog-example + pushd ffi/examples/update-dv + mkdir build + pushd build + cmake .. + make + - name: build and run read-table-changes test + run: | + pushd ffi/examples/read-table-changes + mkdir build + pushd build + cmake .. + make + make test + - name: build and run checkpoint-table test + run: | + pushd ffi/examples/checkpoint-table mkdir build pushd build cmake .. @@ -266,42 +409,78 @@ jobs: matrix: partition: [1, 2, 3] steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 - name: Install Miri run: | rustup toolchain install nightly --component miri rustup override set nightly cargo miri setup - - uses: Swatinem/rust-cache@v2 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: + cache-bin: false save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} - - uses: taiki-e/install-action@nextest + - uses: taiki-e/install-action@98ec31d284eb962f41c14065e9391a955aa810cf # nextest - name: Test with Miri + # -Zmiri-provenance-gc=1000000 runs Miri's pointer-provenance garbage collector every 1M + # basic blocks instead of the default 10k, reducing GC pauses. It only changes GC + # frequency, so it does not weaken undefined-behavior or data-race detection. run: | pushd ffi - MIRIFLAGS=-Zmiri-disable-isolation cargo miri nextest run --features default-engine-rustls,uc-catalog --partition slice:${{ matrix.partition }}/3 + MIRIFLAGS="-Zmiri-disable-isolation -Zmiri-provenance-gc=1000000" cargo miri nextest run --locked --features default-engine-rustls,delta-kernel-unity-catalog,declarative-plans,vendored-protoc --partition slice:${{ matrix.partition }}/3 coverage: runs-on: ubuntu-latest env: CARGO_TERM_COLOR: always steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 - name: Install rust - uses: actions-rust-lang/setup-rust-toolchain@v1 + uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 with: cache: false - - uses: Swatinem/rust-cache@v2 + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: + cache-bin: false save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} - name: Install cargo-llvm-cov - uses: taiki-e/install-action@cargo-llvm-cov + uses: taiki-e/install-action@2d15d02e710b40b6332201aba6af30d595b5cd96 # cargo-llvm-cov - name: Generate code coverage - run: cargo llvm-cov --all-features --workspace --codecov --output-path codecov.json -- --skip read_table_version_hdfs + run: cargo llvm-cov --locked --all-features --workspace --codecov --output-path codecov.json -- --skip read_table_version_hdfs --skip handle::tests::invalid_handle_code - name: Upload coverage to Codecov - uses: codecov/codecov-action@v5 + uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0 with: files: codecov.json fail_ci_if_error: true env: CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} + + # Separate workspace; the --workspace jobs above never touch it. See datafusion-executor/Cargo.toml. + datafusion-executor: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + - name: Install minimal stable with clippy + uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 + with: + cache: false + components: clippy + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + workspaces: datafusion-executor + cache-bin: false + save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} + - uses: taiki-e/install-action@98ec31d284eb962f41c14065e9391a955aa810cf # nextest + - name: build and lint with clippy + working-directory: datafusion-executor + run: cargo clippy --locked --benches --tests --all-features -- -D warnings + - name: build docs + working-directory: datafusion-executor + env: + RUSTDOCFLAGS: -D warnings + run: cargo doc --locked --all-features --no-deps + - name: test + working-directory: datafusion-executor + run: cargo nextest run --locked --all-features + - name: doc tests + working-directory: datafusion-executor + run: cargo test --locked --all-features --doc diff --git a/.github/workflows/comment-on-title-failure.yml b/.github/workflows/comment-on-title-failure.yml index 34136abe9e..7f5b5a3091 100644 --- a/.github/workflows/comment-on-title-failure.yml +++ b/.github/workflows/comment-on-title-failure.yml @@ -11,25 +11,42 @@ jobs: permissions: pull-requests: write steps: - - name: Find associated PR - id: pr + # Step taken from: https://github.com/orgs/community/discussions/25220#discussioncomment-11316244 + - name: Find PR info + id: pr-context env: GH_TOKEN: ${{ github.token }} - HEAD_SHA: ${{ github.event.workflow_run.head_sha }} + PR_TARGET_REPO: ${{ github.repository }} + # If the PR is from a fork, prefix it with `:`, otherwise only the PR branch name is relevant: + PR_BRANCH: |- + ${{ + (github.event.workflow_run.head_repository.owner.login != github.event.workflow_run.repository.owner.login) + && format('{0}:{1}', github.event.workflow_run.head_repository.owner.login, github.event.workflow_run.head_branch) + || github.event.workflow_run.head_branch + }} + # Query the PR number by repo + branch, then assign to step output: run: | - pr_json=$(gh api --paginate repos/${{ github.repository }}/pulls \ - --jq ".[] | select(.head.sha == \"$HEAD_SHA\")") - echo "number=$(echo "$pr_json" | jq -r '.number')" >> "$GITHUB_OUTPUT" - echo "title=$(echo "$pr_json" | jq -r '.title')" >> "$GITHUB_OUTPUT" + gh pr view --repo "${PR_TARGET_REPO}" "${PR_BRANCH}" \ + --json 'number,title' --jq '"number=\(.number)\ntitle=\(.title)"' \ + >> "${GITHUB_OUTPUT}" + + - name: Find existing comment + id: find + uses: peter-evans/find-comment@3eae4d37986fb5a8592848f6a574fdf654e61f9e # v3.1.0 + with: + issue-number: ${{ steps.pr-context.outputs.number }} + comment-author: 'github-actions[bot]' + body-includes: PR title does not match the required pattern - name: Post or update failure comment if: ${{ github.event.workflow_run.conclusion == 'failure' }} - uses: peter-evans/create-or-update-comment@v4 + uses: peter-evans/create-or-update-comment@71345be0265236311c031f5c7866368bd1eff043 # v4.0.0 env: - PR_TITLE: ${{ steps.pr.outputs.title }} + PR_TITLE: ${{ steps.pr-context.outputs.title }} with: - issue-number: ${{ steps.pr.outputs.number }} - comment-tag: pr-title-validation + comment-id: ${{ steps.find.outputs.comment-id }} + issue-number: ${{ steps.pr-context.outputs.number }} + edit-mode: replace body: | PR title does not match the required pattern. Please ensure you follow the [conventional commits](https://www.conventionalcommits.org/) spec. @@ -37,16 +54,8 @@ jobs: **Title:** `${{ env.PR_TITLE }}` - - name: Find existing comment - if: ${{ github.event.workflow_run.conclusion == 'success' }} - id: find - uses: peter-evans/find-comment@v3 - with: - issue-number: ${{ steps.pr.outputs.number }} - comment-tag: pr-title-validation - - name: Delete comment on success - if: ${{ steps.find.outputs.comment-id != '' }} + if: ${{ github.event.workflow_run.conclusion == 'success' && steps.find.outputs.comment-id != '' }} env: GH_TOKEN: ${{ github.token }} run: | diff --git a/.github/workflows/pr-body-validator.yml b/.github/workflows/pr-body-validator.yml new file mode 100644 index 0000000000..48cfe74c21 --- /dev/null +++ b/.github/workflows/pr-body-validator.yml @@ -0,0 +1,47 @@ +name: Validate PR Body + +on: + pull_request: + # synchronize is required: validate-body is a required check, and required checks must + # re-report on every new head SHA or the PR blocks on "Expected, waiting for status". + types: [opened, edited, reopened, synchronize] + merge_group: + +jobs: + validate-body: + runs-on: ubuntu-latest + steps: + - name: Validate PR Body + shell: bash + env: + GH_TOKEN: ${{ github.token }} + REPO: ${{ github.repository }} + EVENT_NAME: ${{ github.event_name }} + PR_NUMBER: ${{ github.event.pull_request.number }} + PR_BODY: ${{ github.event.pull_request.body }} + MERGE_GROUP_REF: ${{ github.event.merge_group.head_ref }} + run: | + set -euo pipefail + + # merge_group runs don't have github.event.pull_request, so fetch the body from the PR + # referenced by the merge queue ref (refs/heads/gh-readonly-queue//pr--). + if [[ "$EVENT_NAME" == "merge_group" ]]; then + PR_NUMBER=$(sed -nE 's/.*\/pr-([0-9]+)-.*/\1/p' <<< "$MERGE_GROUP_REF") + if [[ -z "$PR_NUMBER" ]]; then + echo "::error::could not parse a PR number from merge_group ref '$MERGE_GROUP_REF'" + exit 1 + fi + PR_BODY=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json body --jq '.body // ""') + fi + + echo "Validating body of PR #${PR_NUMBER} (event: ${EVENT_NAME})" + echo "::group::PR body being validated" + printf '%s\n' "$PR_BODY" + echo "::endgroup::" + + if LC_ALL=C grep -q '[^[:print:][:space:]]' <<< "$PR_BODY"; then + echo "PR body contains non-ascii characters. Please remove them." + exit 1 + else + echo "PR body contains ascii characters only" + fi diff --git a/.github/workflows/pr-title-validator.yml b/.github/workflows/pr-title-validator.yml new file mode 100644 index 0000000000..85519a108b --- /dev/null +++ b/.github/workflows/pr-title-validator.yml @@ -0,0 +1,107 @@ +name: Validate PR Title + +on: + pull_request: + types: [opened, edited, reopened, synchronize, labeled, unlabeled] + merge_group: + workflow_run: + workflows: ["semver-label"] # we need this since auto-labels from jobs don't trigger a workflow + types: [completed] + +jobs: + validate-title: + runs-on: ubuntu-latest + permissions: + contents: read + actions: read # cross-workflow artifact download (workflow_run path) + steps: + # for workflow triggered there's no tie back because we're triggered via another workflow + # (semvar-label) so we have to resort to having had semvar-label upload an artifact for us to + # pull + - name: Download PR number artifact + if: github.event_name == 'workflow_run' + continue-on-error: true + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: pr-number + github-token: ${{ github.token }} + run-id: ${{ github.event.workflow_run.id }} + + - name: Resolve PR metadata + id: pr + env: + GH_TOKEN: ${{ github.token }} + EVENT_NAME: ${{ github.event_name }} + REPO: ${{ github.repository }} + # Captured as env vars to prevent expression injection into the shell command. + PR_TITLE: ${{ github.event.pull_request.title }} + PR_NUMBER: ${{ github.event.pull_request.number }} + PR_LABELS_JSON: ${{ toJson(github.event.pull_request.labels.*.name) }} + MERGE_GROUP_REF: ${{ github.event.merge_group.head_ref }} + # We have to deal with three different triggers, and get the title + labels in a different + # way for each: + # pr: just use the $ {{ github.* }} stuff + # merge_group: we have a head_ref which includes the pr number, so sed it out of that and + # then fetch the title and labels via gh + # workflow: see comment for "Download PR number artifact" comment + run: | + set -euo pipefail + + emit_title() { + # Multiline delimiter syntax so a title containing newlines cannot inject + # additional key=value pairs into GITHUB_OUTPUT. + { + echo 'title<> "$GITHUB_OUTPUT" + } + + # Fetch the current title + labels for a known PR number via gh + resolve_from_pr() { + local json + json=$(gh pr view "$1" --repo "$REPO" --json number,title,labels) + echo "number=$1" >> "$GITHUB_OUTPUT" + emit_title "$(jq -r '.title' <<< "$json")" + echo "labels=$(jq -c '[.labels[].name]' <<< "$json")" >> "$GITHUB_OUTPUT" + } + + case "$EVENT_NAME" in + pull_request) + echo "number=$PR_NUMBER" >> "$GITHUB_OUTPUT" + emit_title "$PR_TITLE" + echo "labels=$(jq -c '.' <<< "$PR_LABELS_JSON")" >> "$GITHUB_OUTPUT" + ;; + merge_group) + # head_ref looks like refs/heads/gh-readonly-queue//pr--. + num=$(sed -nE 's/.*\/pr-([0-9]+)-.*/\1/p' <<< "$MERGE_GROUP_REF") + if [[ -z "$num" ]]; then + echo "::error::could not parse a PR number from merge_group ref '$MERGE_GROUP_REF'" + exit 1 + fi + resolve_from_pr "$num" + ;; + workflow_run) + if [[ ! -f pr-number.txt ]]; then + echo "No pr-number artifact (semver-label was skipped); nothing to validate." + echo "skip=true" >> "$GITHUB_OUTPUT" + exit 0 + fi + resolve_from_pr "$(cat pr-number.txt)" + ;; + *) + echo "::error::unsupported event '$EVENT_NAME'" + exit 1 + ;; + esac + + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + if: steps.pr.outputs.skip != 'true' + + - uses: ./.github/actions/pr-title-validator + if: steps.pr.outputs.skip != 'true' + with: + regex: '^(feat|fix|chore|docs|perf|refactor|test|ci)!?(\(.+\))?: .{1,72}$' + breaking-change-regex: '^(feat|fix|chore|docs|perf|refactor|test|ci)!(\(.+\))?: .{1,72}$' + labels: ${{ steps.pr.outputs.labels }} + title: ${{ steps.pr.outputs.title }} diff --git a/.github/workflows/pr-validator.yml b/.github/workflows/pr-validator.yml deleted file mode 100644 index 2dcd13afd9..0000000000 --- a/.github/workflows/pr-validator.yml +++ /dev/null @@ -1,38 +0,0 @@ -name: Validate PR Title - -on: - pull_request: - types: [opened, edited, reopened, synchronize, labeled, unlabeled] - workflow_run: - workflows: ["semver-checks"] # we need this since auto-labels from jobs don't trigger a workflow - types: [completed] - -jobs: - validate-title: - runs-on: ubuntu-latest - steps: - - name: Resolve PR metadata - id: pr - env: - GH_TOKEN: ${{ github.token }} - run: | - if [[ "${{ github.event_name }}" == "workflow_run" ]]; then - pr_json=$(gh api --paginate repos/${{ github.repository }}/pulls \ - --jq ".[] | select(.head.sha == \"${{ github.event.workflow_run.head_sha }}\")") - echo "number=$(echo "$pr_json" | jq -r '.number')" >> "$GITHUB_OUTPUT" - echo "title=$(echo "$pr_json" | jq -r '.title')" >> "$GITHUB_OUTPUT" - echo "labels=$(echo "$pr_json" | jq -c '[.labels[].name]')" >> "$GITHUB_OUTPUT" - else - echo "number=${{ github.event.pull_request.number }}" >> "$GITHUB_OUTPUT" - echo "title=${{ github.event.pull_request.title }}" >> "$GITHUB_OUTPUT" - echo "labels=$(echo '${{ toJson(github.event.pull_request.labels.*.name) }}' | jq -c '.')" >> "$GITHUB_OUTPUT" - fi - - - uses: actions/checkout@v4 - - - uses: ./.github/actions/pr-title-validator - with: - regex: '^(feat|fix|chore|docs|perf|refactor|test|ci)!?(\(.+\))?: .{1,72}$' - breaking-change-regex: '^(feat|fix|chore|docs|perf|refactor|test|ci)!(\(.+\))?: .{1,72}$' - labels: ${{ steps.pr.outputs.labels }} - title: ${{ steps.pr.outputs.title }} diff --git a/.github/workflows/run-examples.yml b/.github/workflows/run-examples.yml index 7f7babe93b..a1da3237d6 100644 --- a/.github/workflows/run-examples.yml +++ b/.github/workflows/run-examples.yml @@ -1,6 +1,17 @@ name: run-examples -on: [push, pull_request, merge_group] +# Restrict push to long-lived branches to avoid duplicate CI runs. See .github/CLAUDE.md. +on: + push: + branches: [main, "release/**"] + pull_request: + merge_group: + +concurrency: + # A new push to a PR cancels that PR's in-flight run. Non-PR events (push, merge_group) have + # no head_ref, so run_id (always unique) is used and those runs never cancel each other. + group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} + cancel-in-progress: true env: CARGO_TERM_COLOR: always @@ -10,14 +21,16 @@ jobs: run-examples: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 - name: Install minimal stable - uses: actions-rust-lang/setup-rust-toolchain@v1 + uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 with: cache: false - # See build.yml top-level comment for why save-if is restricted to main. - - uses: Swatinem/rust-cache@v2 + # See build.yml top-level comment for why save-if is restricted to main and why + # cache-bin is disabled. + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: + cache-bin: false save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} - name: Run all examples @@ -43,23 +56,23 @@ jobs: # Special case for write-table: it needs a temp directory if [ "$example_dir" = "write-table" ]; then tmp_dir=$(mktemp -d) - cargo run --manifest-path "$example_dir/Cargo.toml" --release -- "$tmp_dir" + cargo run --locked --manifest-path "$example_dir/Cargo.toml" --release -- "$tmp_dir" rm -r "$tmp_dir" # Special case for inspect-table: it needs an operation/subcommand, run each one elif [ "$example_dir" = "inspect-table" ]; then for operation in table-version metadata schema scan-metadata actions; do echo " Running inspect-table with operation: $operation" - cargo run --manifest-path "$example_dir/Cargo.toml" --release -- ../tests/data/table-without-dv-small $operation + cargo run --locked --manifest-path "$example_dir/Cargo.toml" --release -- ../tests/data/table-without-dv-small $operation done # Special case for read-table-changes: skip running it in CI as it needs a specific CDF-enabled table # but still verify it compiles # TODO: Add a suitable test table for CDF elif [ "$example_dir" = "read-table-changes" ]; then echo "Building read-table-changes (skipping run - requires CDF-enabled table)" - cargo build --manifest-path "$example_dir/Cargo.toml" --release + cargo build --locked --manifest-path "$example_dir/Cargo.toml" --release else # All other examples run with the test table path - cargo run --manifest-path "$example_dir/Cargo.toml" --release -- ../tests/data/table-without-dv-small + cargo run --locked --manifest-path "$example_dir/Cargo.toml" --release -- ../tests/data/table-without-dv-small fi echo "" diff --git a/.github/workflows/run_integration_test.yml b/.github/workflows/run_integration_test.yml index 65c073d98c..8d23a5ca45 100644 --- a/.github/workflows/run_integration_test.yml +++ b/.github/workflows/run_integration_test.yml @@ -1,36 +1,39 @@ -name: Run tests to ensure we can compile across arrow versions +# TODO: Disabled. The test script runs cargo update which resolves fresh dependencies, +# bypassing the Cargo.lock supply chain policy (see build.yml top-level comment). -on: [workflow_dispatch, push, pull_request, merge_group] - -jobs: - arrow_integration_test: - runs-on: ${{ matrix.os }} - timeout-minutes: 20 - strategy: - fail-fast: false - matrix: - include: - - os: macOS-latest - - os: ubuntu-latest - - os: windows-latest - skip: ${{ github.event_name == 'pull_request' || github.event_name == 'merge_group' }} # skip running windows tests on PRs and merge queue since they are slow - steps: - - name: Skip job for pull requests on Windows - if: ${{ matrix.skip }} - run: echo "Skipping job for pull requests on Windows." - - uses: actions/checkout@v4 - if: ${{ !matrix.skip }} - - name: Setup rust toolchain - if: ${{ !matrix.skip }} - uses: actions-rust-lang/setup-rust-toolchain@v1 - with: - cache: false - # See build.yml top-level comment for why save-if is restricted to main. - - uses: Swatinem/rust-cache@v2 - if: ${{ !matrix.skip }} - with: - save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} - - name: Run integration tests - if: ${{ !matrix.skip }} - shell: bash - run: pushd integration-tests && ./test-all-arrow-versions.sh +# name: Run tests to ensure we can compile across arrow versions +# +# on: [workflow_dispatch, push, pull_request, merge_group] +# +# jobs: +# arrow_integration_test: +# runs-on: ${{ matrix.os }} +# timeout-minutes: 20 +# strategy: +# fail-fast: false +# matrix: +# include: +# - os: macOS-latest +# - os: ubuntu-latest +# - os: windows-latest +# skip: ${{ github.event_name == 'pull_request' || github.event_name == 'merge_group' }} # skip running windows tests on PRs and merge queue since they are slow +# steps: +# - name: Skip job for pull requests on Windows +# if: ${{ matrix.skip }} +# run: echo "Skipping job for pull requests on Windows." +# - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 +# if: ${{ !matrix.skip }} +# - name: Setup rust toolchain +# if: ${{ !matrix.skip }} +# uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 +# with: +# cache: false +# # See build.yml top-level comment for why save-if is restricted to main. +# - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 +# if: ${{ !matrix.skip }} +# with: +# save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} +# - name: Run integration tests +# if: ${{ !matrix.skip }} +# shell: bash +# run: pushd integration-tests && ./test-all-arrow-versions.sh diff --git a/.github/workflows/semver-checks.yml b/.github/workflows/semver-checks.yml index 5392aaf5a8..01e8bef4fe 100644 --- a/.github/workflows/semver-checks.yml +++ b/.github/workflows/semver-checks.yml @@ -1,14 +1,22 @@ name: semver-checks -# Trigger when a PR is opened or changed +# Trigger when a PR is opened or changed. This runs with `pull_request` trigger, which means it has +# only read perms. The adding of the label happens in semver-label.yml via workflow_run which will +# will look at the status of this job, and always runs in the base-repo context. on: - pull_request_target: + pull_request: types: - opened - synchronize - reopened merge_group: +concurrency: + # A new push to a PR cancels that PR's in-flight run. Non-PR events (merge_group) have no + # head_ref, so run_id (always unique) is used and those runs never cancel each other. + group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} + cancel-in-progress: true + env: CARGO_TERM_COLOR: always RUST_BACKTRACE: 1 @@ -17,11 +25,9 @@ jobs: check_if_pr_breaks_semver: runs-on: ubuntu-latest permissions: - # this job runs with read because it checks out the PR head which could contain malicious code contents: read steps: - - uses: actions/checkout@v4 - name: checkout full rep + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 with: fetch-depth: 0 ref: >- @@ -29,77 +35,62 @@ jobs: && github.event.merge_group.head_sha || github.event.pull_request.head.sha }} - name: Install minimal stable - uses: actions-rust-lang/setup-rust-toolchain@v1 + uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 with: cache: false - # See build.yml top-level comment for why save-if is restricted to main. - - uses: Swatinem/rust-cache@v2 + # See build.yml top-level comment for why save-if is restricted to main and why + # cache-bin is disabled. + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: + cache-bin: false save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} - name: Install cargo-semver-checks - shell: bash - run: | - cargo install cargo-semver-checks --locked + uses: taiki-e/install-action@7bc99eee1f1b8902a125006cf790a1f4c8461e63 # v2.69.8 + with: + tool: cargo-semver-checks - name: Compute baseline revision id: baseline shell: bash + env: + MERGE_GROUP_BASE_SHA: ${{ github.event.merge_group.base_sha }} + PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }} + PR_BASE_SHA: ${{ github.event.pull_request.base.sha }} run: | if [ "${{ github.event_name }}" = "merge_group" ]; then - echo "rev=${{ github.event.merge_group.base_sha }}" >> "$GITHUB_OUTPUT" + echo "rev=${MERGE_GROUP_BASE_SHA}" >> "$GITHUB_OUTPUT" else # Use the merge-base instead of the PR base SHA. The base SHA is the tip of # the target branch when the webhook fires, which can differ from where the PR # actually diverged. Using merge-base avoids false positives when the PR branch # is behind the target branch. - MERGE_BASE=$(git merge-base ${{ github.event.pull_request.head.sha }} ${{ github.event.pull_request.base.sha }}) + MERGE_BASE=$(git merge-base "$PR_HEAD_SHA" "$PR_BASE_SHA") echo "rev=${MERGE_BASE}" >> "$GITHUB_OUTPUT" fi - - name: Run check + - name: Run semver check id: check continue-on-error: true shell: bash + env: + BASELINE_REV: ${{ steps.baseline.outputs.rev }} # only check semver on released crates (delta_kernel and delta_kernel_ffi). # note that this won't run on proc macro/derive crates, so don't need to include # delta_kernel_derive etc. run: | cargo semver-checks -p delta_kernel -p delta_kernel_ffi --all-features \ - --baseline-rev ${{ steps.baseline.outputs.rev }} - - name: On Failure - id: set_failure - if: ${{ steps.check.outcome == 'failure' }} - run: | - echo "Checks failed" - echo "check_status=failure" >> $GITHUB_OUTPUT - - name: On Success - id: set_success - if: ${{ steps.check.outcome == 'success' }} - run: | - echo "Checks succeed" - echo "check_status=success" >> $GITHUB_OUTPUT - outputs: - check_status: ${{ steps.set_failure.outputs.check_status || steps.set_success.outputs.check_status }} - update_label_if_needed: - needs: check_if_pr_breaks_semver - runs-on: ubuntu-latest - permissions: - # this job only looks at previous output and then sets a label, so malicious code in the PR - # isn't a concern - pull-requests: write - steps: - - name: On Failure - if: github.event_name == 'pull_request_target' && needs.check_if_pr_breaks_semver.outputs.check_status == 'failure' - uses: actions-ecosystem/action-add-labels@v1 + --baseline-rev "$BASELINE_REV" + # Upload the step outcome as an artifact so semver-label.yml can read it via workflow_run. + # steps.check.outcome is the raw result *before* continue-on-error converts it to "success", + # so it correctly reflects whether a breaking change was detected. + # Only upload for pull_request events; merge_group runs have no PR to label. + - name: Save semver outcome + if: github.event_name == 'pull_request' + env: + SEMVER_OUTCOME: ${{ steps.check.outcome }} + run: echo "$SEMVER_OUTCOME" > semver-outcome.txt + - name: Upload semver outcome + if: github.event_name == 'pull_request' + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: - labels: breaking-change - - name: Remove breaking-change label - if: github.event_name == 'pull_request_target' && needs.check_if_pr_breaks_semver.outputs.check_status == 'success' && contains(github.event.pull_request.labels.*.name, 'breaking-change') - uses: actions-ecosystem/action-remove-labels@v1 - with: - labels: breaking-change - - name: On Success - if: needs.check_if_pr_breaks_semver.outputs.check_status == 'success' - run: | - echo "Checks succeed" - - name: Fail On Incorrect Previous Output - if: needs.check_if_pr_breaks_semver.outputs.check_status != 'success' && needs.check_if_pr_breaks_semver.outputs.check_status != 'failure' - run: exit 1 + name: semver-outcome + path: semver-outcome.txt + retention-days: 1 diff --git a/.github/workflows/semver-label.yml b/.github/workflows/semver-label.yml new file mode 100644 index 0000000000..196a3ca0f3 --- /dev/null +++ b/.github/workflows/semver-label.yml @@ -0,0 +1,94 @@ +name: semver-label + +# Apply or remove the breaking-change label based on the outcome of the semver-checks workflow. +# This must be a separate workflow from semver-checks.yml: label writes require pull-requests:write, +# which is unavailable in pull_request workflows triggered by fork PRs. workflow_run always runs +# in the base-repo context with full write permissions, and never executes PR code. +on: + workflow_run: + workflows: ["semver-checks"] + types: [completed] + +jobs: + update_label_if_needed: + runs-on: ubuntu-latest + permissions: + pull-requests: write + actions: read + # Label updates only apply to PRs; merge_group runs have no associated PR to label. + if: github.event.workflow_run.event == 'pull_request' + steps: + # Resolve PR number from the triggering workflow run's branch. For fork PRs the branch + # must be prefixed with `:` so gh pr view can locate it. + # Pattern from: https://github.com/orgs/community/discussions/25220#discussioncomment-11316244 + - name: Find PR number + id: pr-context + env: + GH_TOKEN: ${{ github.token }} + PR_TARGET_REPO: ${{ github.repository }} + PR_BRANCH: |- + ${{ + (github.event.workflow_run.head_repository.owner.login != github.event.workflow_run.repository.owner.login) + && format('{0}:{1}', github.event.workflow_run.head_repository.owner.login, github.event.workflow_run.head_branch) + || github.event.workflow_run.head_branch + }} + run: | + echo "Looking up PR for branch '${PR_BRANCH}' in repo '${PR_TARGET_REPO}'" + gh pr view --repo "${PR_TARGET_REPO}" "${PR_BRANCH}" \ + --json 'number' --jq '"number=\(.number)"' \ + >> "${GITHUB_OUTPUT}" + echo "PR lookup complete: $(cat "${GITHUB_OUTPUT}")" + + # Download the semver outcome artifact written by semver-checks.yml. + # steps.check.outcome in that workflow is the raw result before continue-on-error + # converts it to "success", so it correctly reflects whether a breaking change was found. + - name: Download semver outcome + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: semver-outcome + github-token: ${{ github.token }} + run-id: ${{ github.event.workflow_run.id }} + + - name: Update breaking-change label + if: steps.pr-context.outputs.number != '' + env: + GH_TOKEN: ${{ github.token }} + PR_NUMBER: ${{ steps.pr-context.outputs.number }} + run: | + STEP_OUTCOME=$(cat semver-outcome.txt) + echo "Semver check outcome: '${STEP_OUTCOME}' for PR #${PR_NUMBER}" + + if [[ "$STEP_OUTCOME" == "failure" ]]; then + echo "Breaking change detected -- adding 'breaking-change' label to PR #$PR_NUMBER" + gh pr edit "$PR_NUMBER" --repo "$GITHUB_REPOSITORY" --add-label "breaking-change" + elif [[ "$STEP_OUTCOME" == "success" ]]; then + # Remove the label only if it is currently present; gh pr edit fails on absent labels. + CURRENT_LABELS=$(gh pr view "$PR_NUMBER" --repo "$GITHUB_REPOSITORY" --json labels --jq '[.labels[].name]') + echo "Current PR labels: $CURRENT_LABELS" + if echo "$CURRENT_LABELS" | jq -e '.[] | select(. == "breaking-change")' > /dev/null 2>&1; then + echo "Semver check passed -- removing 'breaking-change' label from PR #$PR_NUMBER" + gh pr edit "$PR_NUMBER" --repo "$GITHUB_REPOSITORY" --remove-label "breaking-change" + else + echo "Semver check passed -- 'breaking-change' label not present, nothing to do" + fi + else + echo "ERROR: unexpected semver outcome '${STEP_OUTCOME}' in semver-outcome.txt" + exit 1 + fi + + # Create an artifact for pr-title-validator, which is triggered by this workflow via + # workflow_run. Due to how it's triggered, that run will have no idea what PR it's running + # against, so we output it here so it can do its job + - name: Record PR number + if: steps.pr-context.outputs.number != '' + env: + PR_NUMBER: ${{ steps.pr-context.outputs.number }} + run: echo "$PR_NUMBER" > pr-number.txt + + - name: Upload PR number for title validator + if: steps.pr-context.outputs.number != '' + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: pr-number + path: pr-number.txt + if-no-files-found: error diff --git a/.github/workflows/unitycatalog_test.yml b/.github/workflows/unitycatalog_test.yml new file mode 100644 index 0000000000..2f61686f2d --- /dev/null +++ b/.github/workflows/unitycatalog_test.yml @@ -0,0 +1,105 @@ +name: unity-catalog-test + +# Runs the gated `unity-catalog-delta-rest-client` and `delta-kernel-unity-catalog` live +# integration tests against a real Unity Catalog server (built from a pinned commit). Exercises the +# read-only `get_config` handshake (needs no table), `load_table` against a table seeded via the +# Delta-Commits (legacy) API before the test step, and the mutating create tests against the +# throwaway server. +on: + push: + branches: [main, "release/**"] + paths: + - "delta-kernel-unity-catalog/**" + - "unity-catalog-delta-client-api/**" + - "unity-catalog-delta-rest-client/**" + - ".github/workflows/unitycatalog_test.yml" + - ".github/scripts/setup_unitycatalog.sh" + - ".github/scripts/uc_server.sh" + - ".github/scripts/seed_uc_table.sh" + pull_request: + paths: + - "delta-kernel-unity-catalog/**" + - "unity-catalog-delta-client-api/**" + - "unity-catalog-delta-rest-client/**" + - ".github/workflows/unitycatalog_test.yml" + - ".github/scripts/setup_unitycatalog.sh" + - ".github/scripts/uc_server.sh" + - ".github/scripts/seed_uc_table.sh" + workflow_dispatch: + +concurrency: + group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} + cancel-in-progress: true + +env: + CARGO_TERM_COLOR: always + RUST_BACKTRACE: 1 + UC_COMMIT: 0f1445227bd251b386420c90136515daefa4e03d + UC_DIR: ${{ github.workspace }}/.uc-server + +jobs: + uc-test: + name: UC server integration test + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Install Java 17 + uses: actions/setup-java@17f84c3641ba7b8f6deff6309fc4c864478f5d62 # v3.14.1 + with: + distribution: zulu + java-version: "17" + + - name: Install Rust + uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 + with: + cache: false + + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + cache-bin: false + save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} + + - uses: taiki-e/install-action@98ec31d284eb962f41c14065e9391a955aa810cf # nextest + + - name: Cache Unity Catalog build + uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: | + ${{ env.UC_DIR }} + ~/.ivy2 + ~/.sbt + ~/.cache/coursier + key: uc-${{ runner.os }}-${{ env.UC_COMMIT }}-${{ hashFiles('.github/scripts/setup_unitycatalog.sh') }} + + - name: Build Unity Catalog server + run: bash .github/scripts/setup_unitycatalog.sh + + - name: Start Unity Catalog server + run: bash .github/scripts/uc_server.sh start + + # Seed a catalog-managed (MANAGED) Delta table so the load_table live test has a table to read. + - name: Seed a managed table + run: bash .github/scripts/seed_uc_table.sh unity default smoke_test_table + + - name: Run UC live integration tests + env: + UC_SERVER_URL: http://localhost:8080 + UC_TOKEN: not-used + # Fresh server seeds the `unity` catalog with a `default` schema. + UC_TEST_CATALOG: unity + UC_TEST_SCHEMA: default + UC_TEST_TABLE: smoke_test_table + # Live tests are `#[ignore]` so a normal `cargo nextest` reports them as skipped rather than + # passing. `--run-ignored only -E 'test(live_)'` runs exactly the ignored live tests here, + # where a UC server is available. + run: cargo nextest run --locked -p unity-catalog-delta-rest-client -p delta-kernel-unity-catalog --features integration-test --run-ignored only -E 'test(live_)' + + - name: Stop Unity Catalog server + if: always() + run: bash .github/scripts/uc_server.sh stop + + - name: Dump server log on failure + if: failure() + run: cat "$UC_DIR/uc-server.log" || true diff --git a/.github/workflows/user-guide.yml b/.github/workflows/user-guide.yml new file mode 100644 index 0000000000..cc6d132eca --- /dev/null +++ b/.github/workflows/user-guide.yml @@ -0,0 +1,56 @@ +name: user-guide + +# Restrict push to long-lived branches to avoid duplicate CI runs. See .github/CLAUDE.md. +on: + push: + branches: [main, "release/**"] + pull_request: + merge_group: + +concurrency: + # A new push to a PR cancels that PR's in-flight run. Non-PR events (push, merge_group) have + # no head_ref, so run_id (always unique) is used and those runs never cancel each other. + group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} + cancel-in-progress: true + +env: + CARGO_TERM_COLOR: always + RUST_BACKTRACE: 1 + +jobs: + user-guide: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + - name: Install minimal stable + uses: actions-rust-lang/setup-rust-toolchain@150fca883cd4034361b621bd4e6a9d34e5143606 # v1.15.4 + with: + cache: false + # See build.yml top-level comment for why save-if is restricted to main and why + # cache-bin is disabled. + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + cache-bin: false + save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} + # mdbook: builds and serves the book. + # mdbook-mermaid: preprocessor that renders ```mermaid blocks as diagrams. + - uses: taiki-e/install-action@7bc99eee1f1b8902a125006cf790a1f4c8461e63 # v2.69.8 + with: + tool: mdbook,mdbook-mermaid + # mdbook-linkcheck2: backend that fails the build on broken internal + # `[link](../foo.md)` references and missing #section anchors. It's a + # maintained fork of mdbook-linkcheck; the original is incompatible + # with mdbook 0.5+. Not in taiki-e/install-action's catalog, so we + # cargo install it directly. + - name: Install mdbook-linkcheck2 + run: cargo install --locked mdbook-linkcheck2 + # Compiles every Rust code block in the guide via mdbook test, using the + # kernel deps that the Makefile builds first. + - name: Test that all code examples compile + working-directory: docs/user-guide + run: make test + # Renders the book to HTML; with [output.linkcheck2] in book.toml this + # also runs mdbook-linkcheck2, so broken internal links fail CI here. + - name: Build the book + working-directory: docs/user-guide + run: mdbook build diff --git a/.gitignore b/.gitignore index 1df9bfb99c..f25f5e81e1 100644 --- a/.gitignore +++ b/.gitignore @@ -2,7 +2,11 @@ .DS_Store # IDE -.claude/ +# .claude — keep private by default, allowlist published slash commands +.claude/* +!.claude/commands/ +.claude/commands/* +!.claude/commands/update-user-guide.md .cursor/ .dir-locals.el .idea/ @@ -16,15 +20,18 @@ # Rust .cargo-home target/ -/Cargo.lock integration-tests/Cargo.lock +# Python +__pycache__ + # Project acceptance/tests/dat/ +acceptance/workloads/ ffi/examples/read-table/build ffi/examples/visit-expression/build /build /kernel/target /target -/benchmarks/data/workloads/ +/benchmarks/workloads/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 4405169ae2..ec58b517fb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,1422 @@ # Changelog + +## [v0.27.1](https://github.com/delta-io/delta-kernel-rs/tree/v0.27.1/) (2026-08-14) + +[Full Changelog](https://github.com/delta-io/delta-kernel-rs/compare/v0.26.0...v0.27.1) + + +### 🏗️ Breaking changes + +1. Preserve typed schema-field metadata across FFI ([#3086]) + - `EngineSchemaVisitor`'s `visit_*` callbacks now take `&CMetadataMap` instead of `&CStringMap`, so + metadata values keep their kernel type; read it from the `CMetadataValueKind` tag instead of a key + allow-list. Adds `visit_metadata_map` and `get_from_metadata_map`, removes + `StructField::metadata_with_string_values()`. Partition values still use `CStringMap`. +2. Expose clustering column infos read path via FFI ([#2573]) + - New FFI symbol `visit_clustering_columns` (with a `ClusteringColumnVisitor` callback) reports one + descriptor per clustering column: logical and physical path, plus a type tag. The Rust API only + gains a symbol, but the FFI ABI changes, so implement the visitor if you read clustering metadata. +3. Make the `col!` macro a preferred alias of `column_expr!` ([#3071]) + - `col!` is now an alias of `column_expr!`: string literals always split on `.`, constants are allowed + only without dots or special chars, and runtime values no longer compile. Use `Expression::column` + for runtime column names. `column_expr!` still works but is now `doc(hidden)`. +4. Require acknowledging column defaults before writing ([#3036]) + - On tables with column defaults, `partitioned_write_context` and `unpartitioned_write_context` now + fail unless `Transaction::ack_column_defaults()` was called first (the connector takes responsibility + for applying the defaults). +5. Validate add file partition columns at commit time ([#2939]) + - Add-file partition values are now checked against the table's partition columns at commit; missing, + extra, or mismatched keys that used to slip through now fail the commit. +6. Deterministically select among complete checkpoints at one version ([#3046]) + - Read path only. When a version has more than one complete checkpoint, kernel picks one + deterministically (by naming scheme, then part count, then file name) instead of whichever it + happened to hit, and tables that used to fail loading on a `_last_checkpoint` part-count mismatch now + load. Nothing to change on your end. +7. Add Cast expression node ([#2962]) + - The public `Expression` enum isn't `#[non_exhaustive]`, and it gains a `Cast(CastExpression)` + variant. Add an arm to any exhaustive match on `Expression`. Compile break only. +8. Add interval `Scalar` variants ([#2854]) + - `Scalar` (also not `#[non_exhaustive]`) gains `IntervalYearMonth(i32)` and `IntervalDayTime(i64)`, + the value-side companions to the interval `PrimitiveType`s from v0.26.0. Add arms to any exhaustive + match on `Scalar`. Interval support is still limited, so this is mostly a compile break. +9. Add interval support to FFI visitors ([#2903]) + - Another FFI ABI break: this adds callback fields to the `#[repr(C)]` `EngineSchemaVisitor` and + `EngineExpressionVisitor` structs. Rebuild against the regenerated header and handle the new interval + callbacks. +10. Label snapshot-load metric events with source and load type ([#2915], [#2916]) + - The public metric structs `LogSegmentLoadSuccess`, `ProtocolMetadataLoadSuccess`, and + `SnapshotBuildSuccess` gain fields (a fresh-vs-incremental load type, and a `source` on + protocol/metadata loads), and `crc_versions_behind` goes from a bool to a count. None are + `#[non_exhaustive]`, so code that constructs or exhaustively matches them needs updating. + `LogSegmentLoad` and `ProtocolMetadataLoad` now fire on incremental loads too, not just fresh. +11. Derive `Scalar` from POD structs; infallible `Vec`/`HashMap` conversions ([#3095]) + - Converting a `Vec` or `HashMap` into `Scalar::Array` / `Scalar::Map` is now infallible (it used to + return a `Result`), so drop the `?`/`unwrap` at those call sites. Also adds a proc macro that + derives `From for Scalar`. +12. Derive macro for populating a struct from a `Scalar` ([#3101]) + - `StructData::into_values` is replaced by `StructData::into_parts`, which also returns field info, so + update callers to `into_parts`. Adds a derive macro for `TryFrom`, round-tripping a struct + through `Scalar`. + +### 🚀 Features / new APIs + +1. Add predicate pushdown to incremental scan ([#2860]) +2. Add method to calculate base stats field IDs ([#2923]) +3. Strip newly-introduced column mapping metadata on none-mode writes ([#2897]) +4. Support a fallback on PlanBasedEngine ([#2924]) +5. Add UC Delta-Tables API wire models and UpdateTableClient trait ([#2906]) +6. Send User-Agent header on UC Delta-Tables API requests ([#2909]) +7. Add UC OSS-server live integration test CI for get_config and load_table ([#2925]) +8. Read interval types in default engine ([#2769]) +9. Register intervalType reader-writer table feature ([#2880]) +10. Expose several internal apis for connectors to do log cleanup ([#2926]) +11. Add UC Delta-Tables credential-vending wire types and client method ([#2934]) +12. Add single-comparison parser for CHECK constraint predicates ([#2883]) +13. Expose incremental scan via FFI ([#2864]) +14. Column defaults flag removal ([#2922]) +15. Re-introduce CRC caching ([#2908]) +16. Added write support with nullCount stats ([#2845]) +17. Add benchmark registry ([#2918]) +18. Add UC Delta-Tables create-table wire types and staging E2E test ([#2941]) +19. Declarative P&M query ([#2874]) +20. Add geometry and geography schema types ([#2930]) +21. Add CDF file listing implementation for read time CDF ([#2803]) +22. Enhance ToSchema macro with additional field tags ([#2932]) +23. *(ffi)* Expose with_stats with_partition_values scan builder APIs ([#2988]) +24. Enable checkpoint row group skipping for partition columns ([#2978]) +25. Add base Structs for modelling AMT ([#2933]) +26. *(ffi)* Expose snapshot file stats from CRC ([#2977]) +27. Add cooperative cancellation to scan_metadata ([#2937]) +28. Use stale CRC in DomainMetadata queries ([#2961]) +29. Add checkpoint structure/schema for adaptive metadata tree. ([#2851]) +30. Validation for required addFile fields at commit time ([#2856]) +31. Add retry when data skipping before dedup fails ([#2944]) +32. *(ffi)* Expose snapshot_publish_with_committer for catalog-managed tables ([#2899]) +33. Enable reading tables with geospatial columns ([#2931]) +34. Push checkpoint partition casts to parquet handlers ([#2983]) +35. Add declarative metadata scan plan ([#2887]) +36. Expose parquet stats collection ([#3022]) +37. Migrate UC integration to Delta Tables LoadTable and UpdateTable APIs ([#2855]) +38. *(ffi)* Expose declarative metadata scan plan ([#3023]) +39. Create Unity Catalog managed tables via the Delta Tables API ([#2826]) +40. SetTxn queries: fix retention bug, root in CRC ([#3001]) +41. *(ffi)* Track peak native memory via optional tracking allocator ([#2973]) +42. Add DataFusionExecutor struct ([#2929]) +43. Kernel to datafusion scalar conversion ([#2957]) +44. Add UC Delta-Tables reportMetrics wire types and client method ([#2956]) +45. Add FileStatsAccumulator to merge per-row-group stats ([#3010]) +46. Respect scan metadata output options in declarative plans ([#3015]) +47. Expression conversion ([#2958]) +48. Tag UC client User Agent and rename unity-catalog-delta-rest-client ([#3033]) +49. Add create_staging_table and create_table to UCClient ([#3034]) +50. Utilities for converting between AMT and Delta DV structs ([#3021]) +51. Define and use Predicate TRUE/FALSE/NULL constants ([#3065]) +52. Make the PlanBasedEngine fallback optional ([#3069]) +53. Gate vendored protoc behind its own feature ([#3054]) +54. Rename Load to DynamicScan and require file metadata ([#3024]) +55. Refine agg definitions and sync engine impl ([#3081]) +56. 3-arg Min/MaxNotNullBy ([#3082]) +57. Define and implement SUM/COUNT aggregates ([#3083]) +58. Initial predicate conversion ([#2971]) +59. Typed Struct and StructPatch expression conversion ([#2972]) +60. Convert MapToStruct expressions to DataFusion ([#2996]) +61. Lower ParseJson to a kernel-delegating scalar UDF ([#3056]) +62. Validate required fields for dv update ([#3026]) + +### 🐛 Bug Fixes + +1. Change PlanBasedEngine FFI to not free fallback engine ([#2943]) +2. Clarify schema JSON recursion limit errors ([#2940]) +3. Bump parquet to 58.1 and preserve exact zero null counts ([#2952]) +4. Stop tracing events from warning on metrics ([#2799]) +5. Source checkpoint partition columns from physical schema ([#2976]) +6. Interval table feature cleanup ([#2959]) +7. Make generic errors more specific in snapshot creation ([#2987]) +8. Reclassify MalformedJson error during schema parsing to Schema error ([#2999]) +9. Normalize decimal scalar parsing failures to ParseError ([#2954]) +10. Accept `timestampWithoutTimezone` feature alias for legacy tables ([#2842]) +11. Include offending feature in InvalidProtocol feature-validation errors ([#3008]) +12. Three plan/executor improvements ([#3029]) +13. Correctly generate `extendedFileMetadata` for removeFile ([#3006]) +14. Dv update respect shorten selection vector ([#3051]) +15. Remove `interval-type-in-dev` feature flag ([#3041]) +16. Truncate timestamp stats to milliseconds in JSON encoding ([#3073]) +17. Dv resolution when multiple batches are produced ([#3077]) +18. Block dataChange removeFile for appendOnly tables ([#3042]) +19. Remove version from kernel in datafusion executor ([#3108]) +20. Logical comparison for NULL instead of physical comparison ([#3085]) +21. Read tables with orphaned columnMapping reader feature ([#3097]) +22. Spurious delta acceptance failure due to network connection issues ([#3109]) +23. Fix default-feature build (import CollectInto from crate::utils) ([#3123]) + +### 📚 Documentation + +1. Document plan expression semantics for executor authors ([#3017]) +2. Add crate metadata and READMEs for the UC crates ([#3061]) +3. Explain why timestamp stats are floored, not rounded or ceiled ([#3090]) +4. Explain why kernel fails on a dangling _last_checkpoint hint ([#3084]) + +### ⚡ Performance + +1. Skip non-add checkpoint row groups during scans ([#2991]) +2. Omit raw checkpoint stats when structured stats are compatible ([#2998]) +3. Speed up the Miri CI job and widen its coverage ([#3035]) + +### 🚜 Refactor + +1. Extract log_tail_from_commits helper in UC kernel client ([#2927]) +2. Rename NullGuarded to Checkpoint data skipping creator ([#2960]) +3. Extract StatColumns and emit prefixes from the checkpoint creator ([#2963]) +4. Expose treemap to bool functions from internal api ([#2986]) +5. Consolidate test engine wrappers behind DelegatingEngine ([#3031]) +6. Extract unit test utils from `kernel/src/utils.rs` ([#3018]) +7. Move plan construction to `Scan` ([#3039]) + +### 🧪 Testing + +1. Strengthen scan metadata struct columns coverage ([#3053]) + +### ⚙️ Chores/CI + +1. Split no-declarative-plans tests into a parallel job ([#3038]) +2. Scaffold datafusion-executor crate ([#2928]) +3. Version the UC crates independently at 0.1.0 ([#3062]) +4. Use col! instead of column_expr! ([#3074]) +5. Use col and col_name macros everywhere they make sense ([#3121]) + + +[#2860]: https://github.com/delta-io/delta-kernel-rs/pull/2860 +[#2923]: https://github.com/delta-io/delta-kernel-rs/pull/2923 +[#2897]: https://github.com/delta-io/delta-kernel-rs/pull/2897 +[#2924]: https://github.com/delta-io/delta-kernel-rs/pull/2924 +[#2906]: https://github.com/delta-io/delta-kernel-rs/pull/2906 +[#2909]: https://github.com/delta-io/delta-kernel-rs/pull/2909 +[#2925]: https://github.com/delta-io/delta-kernel-rs/pull/2925 +[#2769]: https://github.com/delta-io/delta-kernel-rs/pull/2769 +[#2880]: https://github.com/delta-io/delta-kernel-rs/pull/2880 +[#2926]: https://github.com/delta-io/delta-kernel-rs/pull/2926 +[#2927]: https://github.com/delta-io/delta-kernel-rs/pull/2927 +[#2934]: https://github.com/delta-io/delta-kernel-rs/pull/2934 +[#2943]: https://github.com/delta-io/delta-kernel-rs/pull/2943 +[#2883]: https://github.com/delta-io/delta-kernel-rs/pull/2883 +[#2864]: https://github.com/delta-io/delta-kernel-rs/pull/2864 +[#2922]: https://github.com/delta-io/delta-kernel-rs/pull/2922 +[#2908]: https://github.com/delta-io/delta-kernel-rs/pull/2908 +[#2915]: https://github.com/delta-io/delta-kernel-rs/pull/2915 +[#2845]: https://github.com/delta-io/delta-kernel-rs/pull/2845 +[#2940]: https://github.com/delta-io/delta-kernel-rs/pull/2940 +[#2918]: https://github.com/delta-io/delta-kernel-rs/pull/2918 +[#2952]: https://github.com/delta-io/delta-kernel-rs/pull/2952 +[#2941]: https://github.com/delta-io/delta-kernel-rs/pull/2941 +[#2874]: https://github.com/delta-io/delta-kernel-rs/pull/2874 +[#2960]: https://github.com/delta-io/delta-kernel-rs/pull/2960 +[#2963]: https://github.com/delta-io/delta-kernel-rs/pull/2963 +[#2799]: https://github.com/delta-io/delta-kernel-rs/pull/2799 +[#2962]: https://github.com/delta-io/delta-kernel-rs/pull/2962 +[#2930]: https://github.com/delta-io/delta-kernel-rs/pull/2930 +[#2803]: https://github.com/delta-io/delta-kernel-rs/pull/2803 +[#2976]: https://github.com/delta-io/delta-kernel-rs/pull/2976 +[#2959]: https://github.com/delta-io/delta-kernel-rs/pull/2959 +[#2986]: https://github.com/delta-io/delta-kernel-rs/pull/2986 +[#2932]: https://github.com/delta-io/delta-kernel-rs/pull/2932 +[#2991]: https://github.com/delta-io/delta-kernel-rs/pull/2991 +[#2988]: https://github.com/delta-io/delta-kernel-rs/pull/2988 +[#2978]: https://github.com/delta-io/delta-kernel-rs/pull/2978 +[#2933]: https://github.com/delta-io/delta-kernel-rs/pull/2933 +[#2977]: https://github.com/delta-io/delta-kernel-rs/pull/2977 +[#2937]: https://github.com/delta-io/delta-kernel-rs/pull/2937 +[#2961]: https://github.com/delta-io/delta-kernel-rs/pull/2961 +[#2987]: https://github.com/delta-io/delta-kernel-rs/pull/2987 +[#2851]: https://github.com/delta-io/delta-kernel-rs/pull/2851 +[#2856]: https://github.com/delta-io/delta-kernel-rs/pull/2856 +[#2944]: https://github.com/delta-io/delta-kernel-rs/pull/2944 +[#2999]: https://github.com/delta-io/delta-kernel-rs/pull/2999 +[#2899]: https://github.com/delta-io/delta-kernel-rs/pull/2899 +[#2931]: https://github.com/delta-io/delta-kernel-rs/pull/2931 +[#2916]: https://github.com/delta-io/delta-kernel-rs/pull/2916 +[#2954]: https://github.com/delta-io/delta-kernel-rs/pull/2954 +[#2998]: https://github.com/delta-io/delta-kernel-rs/pull/2998 +[#2842]: https://github.com/delta-io/delta-kernel-rs/pull/2842 +[#2854]: https://github.com/delta-io/delta-kernel-rs/pull/2854 +[#2903]: https://github.com/delta-io/delta-kernel-rs/pull/2903 +[#2983]: https://github.com/delta-io/delta-kernel-rs/pull/2983 +[#3008]: https://github.com/delta-io/delta-kernel-rs/pull/3008 +[#2887]: https://github.com/delta-io/delta-kernel-rs/pull/2887 +[#3022]: https://github.com/delta-io/delta-kernel-rs/pull/3022 +[#3017]: https://github.com/delta-io/delta-kernel-rs/pull/3017 +[#3029]: https://github.com/delta-io/delta-kernel-rs/pull/3029 +[#2855]: https://github.com/delta-io/delta-kernel-rs/pull/2855 +[#3023]: https://github.com/delta-io/delta-kernel-rs/pull/3023 +[#2826]: https://github.com/delta-io/delta-kernel-rs/pull/2826 +[#3006]: https://github.com/delta-io/delta-kernel-rs/pull/3006 +[#3031]: https://github.com/delta-io/delta-kernel-rs/pull/3031 +[#3038]: https://github.com/delta-io/delta-kernel-rs/pull/3038 +[#3001]: https://github.com/delta-io/delta-kernel-rs/pull/3001 +[#2928]: https://github.com/delta-io/delta-kernel-rs/pull/2928 +[#2973]: https://github.com/delta-io/delta-kernel-rs/pull/2973 +[#2929]: https://github.com/delta-io/delta-kernel-rs/pull/2929 +[#2957]: https://github.com/delta-io/delta-kernel-rs/pull/2957 +[#2939]: https://github.com/delta-io/delta-kernel-rs/pull/2939 +[#2956]: https://github.com/delta-io/delta-kernel-rs/pull/2956 +[#3010]: https://github.com/delta-io/delta-kernel-rs/pull/3010 +[#3015]: https://github.com/delta-io/delta-kernel-rs/pull/3015 +[#3018]: https://github.com/delta-io/delta-kernel-rs/pull/3018 +[#3036]: https://github.com/delta-io/delta-kernel-rs/pull/3036 +[#2958]: https://github.com/delta-io/delta-kernel-rs/pull/2958 +[#3051]: https://github.com/delta-io/delta-kernel-rs/pull/3051 +[#3033]: https://github.com/delta-io/delta-kernel-rs/pull/3033 +[#3034]: https://github.com/delta-io/delta-kernel-rs/pull/3034 +[#2573]: https://github.com/delta-io/delta-kernel-rs/pull/2573 +[#3046]: https://github.com/delta-io/delta-kernel-rs/pull/3046 +[#3041]: https://github.com/delta-io/delta-kernel-rs/pull/3041 +[#3073]: https://github.com/delta-io/delta-kernel-rs/pull/3073 +[#3061]: https://github.com/delta-io/delta-kernel-rs/pull/3061 +[#3062]: https://github.com/delta-io/delta-kernel-rs/pull/3062 +[#3021]: https://github.com/delta-io/delta-kernel-rs/pull/3021 +[#3065]: https://github.com/delta-io/delta-kernel-rs/pull/3065 +[#3069]: https://github.com/delta-io/delta-kernel-rs/pull/3069 +[#3077]: https://github.com/delta-io/delta-kernel-rs/pull/3077 +[#3054]: https://github.com/delta-io/delta-kernel-rs/pull/3054 +[#3090]: https://github.com/delta-io/delta-kernel-rs/pull/3090 +[#3024]: https://github.com/delta-io/delta-kernel-rs/pull/3024 +[#3035]: https://github.com/delta-io/delta-kernel-rs/pull/3035 +[#3039]: https://github.com/delta-io/delta-kernel-rs/pull/3039 +[#3042]: https://github.com/delta-io/delta-kernel-rs/pull/3042 +[#3086]: https://github.com/delta-io/delta-kernel-rs/pull/3086 +[#3081]: https://github.com/delta-io/delta-kernel-rs/pull/3081 +[#3084]: https://github.com/delta-io/delta-kernel-rs/pull/3084 +[#3071]: https://github.com/delta-io/delta-kernel-rs/pull/3071 +[#3082]: https://github.com/delta-io/delta-kernel-rs/pull/3082 +[#3074]: https://github.com/delta-io/delta-kernel-rs/pull/3074 +[#3083]: https://github.com/delta-io/delta-kernel-rs/pull/3083 +[#2971]: https://github.com/delta-io/delta-kernel-rs/pull/2971 +[#3108]: https://github.com/delta-io/delta-kernel-rs/pull/3108 +[#3085]: https://github.com/delta-io/delta-kernel-rs/pull/3085 +[#3097]: https://github.com/delta-io/delta-kernel-rs/pull/3097 +[#3095]: https://github.com/delta-io/delta-kernel-rs/pull/3095 +[#3101]: https://github.com/delta-io/delta-kernel-rs/pull/3101 +[#2972]: https://github.com/delta-io/delta-kernel-rs/pull/2972 +[#2996]: https://github.com/delta-io/delta-kernel-rs/pull/2996 +[#3056]: https://github.com/delta-io/delta-kernel-rs/pull/3056 +[#3026]: https://github.com/delta-io/delta-kernel-rs/pull/3026 +[#3109]: https://github.com/delta-io/delta-kernel-rs/pull/3109 +[#3053]: https://github.com/delta-io/delta-kernel-rs/pull/3053 +[#3121]: https://github.com/delta-io/delta-kernel-rs/pull/3121 +[#3123]: https://github.com/delta-io/delta-kernel-rs/pull/3123 + + +## [v0.26.0](https://github.com/delta-io/delta-kernel-rs/tree/v0.26.0/) (2026-07-13) + +[Full Changelog](https://github.com/delta-io/delta-kernel-rs/compare/v0.25.0...v0.26.0) + + +### 🏗️ Breaking changes + +1. Read interval types in core kernel ([#2768]) + - `PrimitiveType` gains `IntervalYearMonth` / `IntervalDayTime` variants (with + `DataType::INTERVAL_YEAR_MONTH` / `INTERVAL_DAY_TIME` constants). Add arms to any exhaustive + match on `PrimitiveType` or `DataType`. Interval reads are not yet functional, so this is + only a compile break. +2. Expose checkpoint writes via checkpoint_snapshot + FfiCheckpointSpec ([#2722]) + - FFI only. `checkpoint_snapshot` gains a `spec: Option<&FfiCheckpointSpec>` argument and + returns `FfiCheckpointWriteResult` instead of `bool`. Pass `NULL` for the old auto-V1/V2 + behavior, switch on the result tag, and `free_snapshot` the handle it carries. +3. Carry scan sub-step latencies as nanoseconds ([#2850]) + - `ScanMetadataCompleted`'s `dedup_visitor_time_ms` / `predicate_eval_time_ms` become + `Duration` fields `dedup_visitor_time` / `predicate_eval_time`; call `.as_millis()` for the + old value. The FFI struct instead renames them to `*_time_ns` (nanoseconds). +4. Add arrow-59, drop arrow-57 ([#2847]) + - The `arrow-57` cargo feature is removed and `arrow-59` added; the default `arrow` feature now + resolves to `arrow-59` (was `arrow-58`). Move any explicit `arrow-57` pin to `arrow-58` or + `arrow-59`; transitive Arrow 59 changes may surface. +5. Added delete functionality to StorageHandler ([#2823]) + - The `StorageHandler` trait gains a required `fn delete(&self, path: &Url) -> DeltaResult<()>` + (no default impl). Implement it on your storage handler. +6. Make default engine read I/O concurrency configurable ([#2813]) + - `DefaultParquetHandler::with_readahead` is removed; `with_buffer_size` / `with_batch_size` + (both handlers) now take `NonZero`. Wrap existing size args in json handler with `NonZero::new`. +7. Improve the column_name! macro and friends ([#2891]) + - The internal re-exported proc macro `delta_kernel_derive::parse_column_name` is renamed to + `column_name_segments`. Additionally, `column_name!` now requires at least one argument + and rejects empty path segments. Previously accepted empty/zero-segment invocations now + fail to compile. `column_name!` also gains multi-argument / string-constant support. + +### 🚀 Features / new APIs + +1. Accept millisecond-precision timestamps in arrow-to-kernel conversion ([#2805]) +2. Expose the FFI function for CommitRange builder ([#2775]) +3. Add java_package option to declarative-plan protos ([#2817]) +4. Add column defaults struct ([#2797]) +5. Allow maintenance properties during CREATE TABLE ([#2816]) +6. Add without_row_transforms scan opt-out ([#2756]) +7. Emit partitionValues_parsed via PartitionValuesOptions in scan output ([#2755]) +8. Schema! macro for easier schema creation ([#2795]) +9. Add with_correlation_id to create-table and alter-table builders ([#2834]) +10. Add schema and transaction methods for column defaults ([#2808]) +11. Add commit range iterator FFI bindings ([#2776]) +12. Expose create-table data layout/partitioned write context via FFI ([#2774]) +13. Parse the full v2Checkpoint object from _last_checkpoint ([#2777]) +14. Support more cases for `snap.write_checksum` ([#2853]) +15. Add checkpoint-shape resolution via PlanExecutor ([#2779]) +16. Add generic REST/HTTP ObjectStore for the default engine ([#2661]) +17. Add SQL predicate tokenizer for CHECK constraints ([#2882]) +18. Expose if a snapshot is built as latest from intent level ([#2905]) +19. Add proper declarative plan agg node ([#2790]) +20. Conversion utilities for plan IR to proto bytes ([#2802]) +21. Introduce fluent PlanBuilder to easily construct Plans ([#2772]) +22. Simplify agg alias handling ([#2868]) +23. Embed proto serialized schema inside of CParquetFooter ([#2865]) +24. Add reader/writer features for AMT ([#2871]) + +### 🐛 Bug Fixes + +1. Update use statements in default engine doc strings ([#2801]) +2. Update snapshot builder info to not include fields ([#2807]) +3. Accept LargeBinary and BinaryView for variant fields ([#2737]) +4. Pair struct fields by name instead of by position ([#2822]) +5. Parse empty-string partition values with spark cast semantics ([#2859]) +6. Tolerate stale column mapping annotations when disabled ([#2886]) +7. Rename IntoArray::into_array to avoid std name collision on Rust 1.97 ([#2901]) + +### 📚 Documentation + +1. Minor test comment update ([#2838]) +2. Add incremental scan user-guide page ([#2863]) +3. Recommend TestTableBuilder for integration test setup ([#2873]) + +### 🚜 Refactor + +1. Extract shared table-feature promotion helpers ([#2844]) +2. Define and use FoldWithOption extension trait ([#2878]) +3. Identify plan nodes by index, drop RefId and output ([#2829]) + +### 🧪 Testing + +1. Add VersionTarget::AtTimestamp and IncrementalFrom to TestTableBuilder ([#2652]) +2. Assert row_id is unique for row tracking enabled feature sets ([#2794]) +3. Extend TableConfig with numIndexedCols and dataSkippingStatsColumns ([#2653]) +4. Cover all-null file pruning across operators and log-replay sources ([#2830]) +5. Verify all primitive values in parsed_stats test ([#2835]) +6. Ensuring dv works with row tracking feature ([#2806]) + +### ⚙️ Chores/CI + +1. Upgrade to strum-28 ([#2849]) +2. Define XXX_FIELD to go with XXX_NAME ([#2870]) + + +[#2790]: https://github.com/delta-io/delta-kernel-rs/pull/2790 +[#2801]: https://github.com/delta-io/delta-kernel-rs/pull/2801 +[#2807]: https://github.com/delta-io/delta-kernel-rs/pull/2807 +[#2652]: https://github.com/delta-io/delta-kernel-rs/pull/2652 +[#2794]: https://github.com/delta-io/delta-kernel-rs/pull/2794 +[#2805]: https://github.com/delta-io/delta-kernel-rs/pull/2805 +[#2802]: https://github.com/delta-io/delta-kernel-rs/pull/2802 +[#2775]: https://github.com/delta-io/delta-kernel-rs/pull/2775 +[#2768]: https://github.com/delta-io/delta-kernel-rs/pull/2768 +[#2817]: https://github.com/delta-io/delta-kernel-rs/pull/2817 +[#2797]: https://github.com/delta-io/delta-kernel-rs/pull/2797 +[#2653]: https://github.com/delta-io/delta-kernel-rs/pull/2653 +[#2816]: https://github.com/delta-io/delta-kernel-rs/pull/2816 +[#2829]: https://github.com/delta-io/delta-kernel-rs/pull/2829 +[#2830]: https://github.com/delta-io/delta-kernel-rs/pull/2830 +[#2756]: https://github.com/delta-io/delta-kernel-rs/pull/2756 +[#2755]: https://github.com/delta-io/delta-kernel-rs/pull/2755 +[#2838]: https://github.com/delta-io/delta-kernel-rs/pull/2838 +[#2795]: https://github.com/delta-io/delta-kernel-rs/pull/2795 +[#2834]: https://github.com/delta-io/delta-kernel-rs/pull/2834 +[#2808]: https://github.com/delta-io/delta-kernel-rs/pull/2808 +[#2722]: https://github.com/delta-io/delta-kernel-rs/pull/2722 +[#2849]: https://github.com/delta-io/delta-kernel-rs/pull/2849 +[#2844]: https://github.com/delta-io/delta-kernel-rs/pull/2844 +[#2835]: https://github.com/delta-io/delta-kernel-rs/pull/2835 +[#2737]: https://github.com/delta-io/delta-kernel-rs/pull/2737 +[#2850]: https://github.com/delta-io/delta-kernel-rs/pull/2850 +[#2806]: https://github.com/delta-io/delta-kernel-rs/pull/2806 +[#2847]: https://github.com/delta-io/delta-kernel-rs/pull/2847 +[#2772]: https://github.com/delta-io/delta-kernel-rs/pull/2772 +[#2776]: https://github.com/delta-io/delta-kernel-rs/pull/2776 +[#2822]: https://github.com/delta-io/delta-kernel-rs/pull/2822 +[#2813]: https://github.com/delta-io/delta-kernel-rs/pull/2813 +[#2859]: https://github.com/delta-io/delta-kernel-rs/pull/2859 +[#2823]: https://github.com/delta-io/delta-kernel-rs/pull/2823 +[#2868]: https://github.com/delta-io/delta-kernel-rs/pull/2868 +[#2870]: https://github.com/delta-io/delta-kernel-rs/pull/2870 +[#2774]: https://github.com/delta-io/delta-kernel-rs/pull/2774 +[#2865]: https://github.com/delta-io/delta-kernel-rs/pull/2865 +[#2878]: https://github.com/delta-io/delta-kernel-rs/pull/2878 +[#2777]: https://github.com/delta-io/delta-kernel-rs/pull/2777 +[#2853]: https://github.com/delta-io/delta-kernel-rs/pull/2853 +[#2863]: https://github.com/delta-io/delta-kernel-rs/pull/2863 +[#2886]: https://github.com/delta-io/delta-kernel-rs/pull/2886 +[#2901]: https://github.com/delta-io/delta-kernel-rs/pull/2901 +[#2871]: https://github.com/delta-io/delta-kernel-rs/pull/2871 +[#2779]: https://github.com/delta-io/delta-kernel-rs/pull/2779 +[#2891]: https://github.com/delta-io/delta-kernel-rs/pull/2891 +[#2661]: https://github.com/delta-io/delta-kernel-rs/pull/2661 +[#2882]: https://github.com/delta-io/delta-kernel-rs/pull/2882 +[#2873]: https://github.com/delta-io/delta-kernel-rs/pull/2873 +[#2905]: https://github.com/delta-io/delta-kernel-rs/pull/2905 + + +## [v0.25.0](https://github.com/delta-io/delta-kernel-rs/tree/v0.25.0/) (2026-06-17) + +[Full Changelog](https://github.com/delta-io/delta-kernel-rs/compare/v0.24.0...v0.25.0) + +> [!IMPORTANT] +> **The default engine now lives in its own `delta_kernel_default_engine` crate ([#2397]).** +> Update your `Cargo.toml`: +> +> ```toml +> # before (v0.24.0) +> delta_kernel = { version = "0.24.0", features = [/* any features you choose */] } +> +> # after (v0.25.0) +> delta_kernel = "0.25.0" +> delta_kernel_default_engine = { version = "0.25.0", features = [/* any features you choose */] } +> ``` +> +> Update your `use` statements: +> +> ```rust +> // before (v0.24.0) +> use delta_kernel::engine::default::DefaultEngine; +> use delta_kernel::engine::default::executor::tokio::TokioBackgroundExecutor; +> use delta_kernel::engine::default::storage::store_from_url; +> +> // after (v0.25.0) +> use delta_kernel_default_engine::DefaultEngine; +> use delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor; +> use delta_kernel_default_engine::storage::store_from_url; +> ``` + + +### 🏗️ Breaking changes +1. Support void primitive type in schema ([#1858]) + - Adds a `Void` variant to the `PrimitiveType` enum (and a `DataType::VOID` constant). + Exhaustive matches on `PrimitiveType` must handle the new variant. +2. Add commit timestamp to the history_manager search result ([#2663]) + - `latest_version_as_of` and `first_version_after` now return a `CommitAt` instead of a + bare `Version`. +3. Define and use CollectInto trait ([#2699]) + - `ColumnName::new`/`new_nested`, `Expression::column`, and `Predicate::column` now take + `impl CollectInto<..>` instead of `impl IntoIterator`. Existing iterator + arguments still work; turbofish-annotated calls must drop the explicit type parameter. +4. Use type constructors to avoid Box::new and DataType::from boilerplate ([#2700]) + - `ArrayType::new` now takes `element_type: impl Into` instead of `DataType`. + A bare `DataType` still works; callers relying on type inference may need an annotation. +5. Replace ScanBuilder stats setters with StatsOptions struct ([#2666]) + - `ScanBuilder::with_stats_columns` is removed; pass a `StatsOptions` value to + `ScanBuilder::with_stats` instead. +6. Extract default engine into separate crate ([#2397]) + - The default engine moved to a new `delta_kernel_default_engine` crate. Replace + `delta_kernel::engine::default::{DefaultEngine, executor::*}` imports with + `delta_kernel_default_engine::*`. +7. Add failure metric events ([#2702]) + - `MetricEvent` variants were renamed and split into success/failure pairs: + `LogSegmentLoaded` -> `LogSegmentLoad{Success,Failure}`, + `ProtocolMetadataLoaded` -> `ProtocolMetadataLoad{Success,Failure}`, + `SnapshotCompleted` -> `SnapshotBuild{Success,Failure}`, `CrcReadCompleted` -> + `CrcRead{Success,Failure}`. Success variants' duration field is now `duration` (was + `total_duration`). +8. History_manager: add get_earliest_recreatable version method ([#2675]) + - `LogHistoryError` gains a `NoRecreatableCommit { log_root: Url }` variant. Callers + exhaustively matching `LogHistoryError` must add an arm for it (or `..`). +9. Add `Transaction::commit` metric events ([#2719]) + - `MetricEvent` gains `TransactionCommitSuccess` and `TransactionCommitFailure` variants. + Exhaustive matches on `MetricEvent` must handle the new variants. +10. Introduce struct patch builder ([#2665]) + - The FFI `EngineExpressionVisitor` struct-patch callbacks changed: `visit_struct_patch_expr` + now takes `prepended_field_list_id`, `field_patch_list_id`, and `appended_field_list_id`, + and `visit_field_patch` takes `field_name` by value plus `insertion_expr_list_id`, + `keep_input: bool`, and `optional: bool` (replacing `is_replace`). Engines implementing + these callbacks must update their signatures. +11. Introduce schema struct patching ([#2672]) + - `StructType::with_field_{inserted_after,inserted_before,removed,replaced}` are removed; + use `SchemaStructPatchBuilder` instead. `ExpressionStructPatchBuilder` methods are renamed + (`with_dropped_field` -> `drop`, `with_replaced_field` -> `replace`, + `with_inserted_field_after` -> `insert_after`, `with_prepended_field` -> `prepend`, + `with_appended_field` -> `append`), and `Expression::struct_patch` is now fallible. +12. Checkpoint writer eagerly creates output schema ([#2726]) + - `Snapshot::create_checkpoint_writer` now takes an `engine: &dyn Engine` argument. Pass + the engine you already used for checkpointing. +13. Improve plan execution FFI and error handling ([#2729]) + - Under the `declarative-plans` feature, the `CExecuteOpFn` FFI callback (and the plan-result + iterator `next` callbacks) no longer return a `CPlanResultWrapper`; they take an `out` + pointer to write the result into and return `()`. `CPlanResultWrapper`, `PlanResultCleanup`, + and `engine_error_to_kernel` are removed. +14. Add recreatable-commit resolution to history manager ([#2714]) + - `latest_version_as_of` and `first_version_after` gain a fourth + `resolved_commit_type: HistoryCommitType` argument. Pass `HistoryCommitType::Recreatable` + to restrict results to loadable versions, or `HistoryCommitType::Published` to allow any + version present in the log. +15. Add `table_type` to snapshot-load and scan metric events ([#2721]) + - The metric-event structs (`LogSegmentLoad*`, `ProtocolMetadataLoad*`, `SnapshotBuild*`, + `TransactionCommit*`, `ScanMetadataCompleted`) gain a `table_type: TableType` field. + Callers constructing or exhaustively matching these must account for the new field. +16. Add caller-supplied correlation id to operation metric events ([#2734]) + - Those same metric-event structs additionally gain a `correlation_id: Option>` + field, and `SnapshotLoadMetricContext` is no longer `Copy`. Callers constructing or + exhaustively matching these events must account for the new field. +17. Strum-ify metric label enums (and fix missing dv metric) ([#2732]) + - `TransactionCommitSuccess` gains a `num_dv_updates: u64` field; callers constructing or + exhaustively matching it must account for the new field. `ScanType` gains + `EnumString`/`IntoStaticStr` (plus `Display`/`AsRefStr`, replacing its manual `Display` + impl) and `CommitFailureReason` gains `IntoStaticStr`. + +### 🚀 Features / new APIs + +1. Add MeteredDeltaEngine wrapping engines for storage metrics ([#2623]) +2. Support void primitive type in schema ([#1858]) +3. Add col and lit expression builders ([#2684]) +4. Added get_earliest_delta_file function ([#2582]) +5. Add commit timestamp to the history_manager search result ([#2663]) +6. Define and use CollectInto trait ([#2699]) +7. Add basic plan IR behind declarative-plans feature ([#2613]) +8. Replace ScanBuilder stats setters with StatsOptions struct ([#2666]) +9. Introduce FFI types for plan execution ([#2680]) +10. Add MeteredJsonHandler and MeteredParquetHandler ([#2694]) +11. Use incremental CRC replay in Snapshot construction ([#2678]) +12. Add FFI-backed PlanExecutor impl ([#2685]) +13. Add failure metric events ([#2702]) +14. Support incremental CRC reconstruction on snapshot `builder_from` API ([#2712]) +15. Support local DAT corpus override for acceptance workloads ([#2705]) +16. Add proto wire format for declarative-plans IR ([#2629]) +17. History_manager: add get_earliest_recreatable version method ([#2675]) +18. Add a public history_manager::get_earliest_commit ([#2713]) +19. Add `Transaction::commit` metric events ([#2719]) +20. Introduce struct patch builder ([#2665]) +21. Introduce schema struct patching ([#2672]) +22. Add recreatable-commit resolution to history manager ([#2714]) +23. Add `table_type` to snapshot-load and scan metric events ([#2721]) +24. Define and use a new projection struct patch ([#2730]) +25. Add caller-supplied correlation id to operation metric events ([#2734]) +26. Automatically insert partition columns when needed ([#2715]) +27. Add internal kernel sql parser for literals ([#2670]) +28. Add CommitRange for iterating Delta log actions across a commit range ([#2514]) + +### 🐛 Bug Fixes + +1. Set stats.tightBound to false for DV updates ([#2681]) +2. Specify selection-vector contracts at the engine boundary ([#2728]) +3. Restore CRC-rooted P&M replay removed by #2678 ([#2741]) +4. Strum-ify metric label enums (and fix missing dv metric) ([#2732]) +5. Update benchmark ci to checkout merge head ([#2771]) + +### 📚 Documentation + +1. Add issue for heap size estimation optimize ([#2758]) + +### ⚡ Performance + +1. Incremental CRC speedup (1.4x speed, 2.15x memory) ([#2693]) +2. Terminate log listing before paging through staged commits ([#2738]) + +### 🚜 Refactor + +1. Recompute WriteState on demand instead of trying to memoize it ([#2682]) +2. Update benchmarking ci to run on every push and update output format ([#2691]) +3. Bridge Operation::QueryPlan onto the new IR ([#2637]) +4. Extract default engine into separate crate ([#2397]) +5. Update benchmark output format, update tables ([#2703]) +6. Unify column mapping validations to `MakePhysical` ([#2599]) +7. Project node takes a single expression instead of a list ([#2727]) +8. Checkpoint writer eagerly creates output schema ([#2726]) +9. Eagerly materialize the physical no-partition-schema ([#2743]) +10. Add and use Schema::field_at and friends ([#2744]) +11. Improve plan execution FFI and error handling ([#2729]) +12. Extract workloads crate ([#2761]) + +### 🧪 Testing + +1. Checkpoint should not contain tombstoned domain metadata ([#2365]) +2. Add CRC support to TestTableBuilder via Snapshot::write_checksum ([#2522]) +3. Parsed-stats integration test covering nested structs via table-builder ([#2649]) +4. Addressed failing invalid-handle-code ([#2706]) +5. Non-blocking invalid-handle-code in CI ([#2745]) +6. Inject sparse nulls into table-builder data generation ([#2657]) + +### ⚙️ Chores/CI + +1. Make title validator it run in merge queue ([#2688]) +2. Expose column mapping metadata generators via internal_api ([#2671]) +3. Use type constructors to avoid Box::new and DataType::from boilerplate ([#2700]) +4. Fixup title and body validator actions for merge/workflow ([#2697]) +5. Fix dupliate CI runs ([#2704]) +6. Various improvements ([#2708]) +7. Bump codecov-action to v7.0.0 to fix coverage uploads ([#2711]) +8. Fix flaky metrics deadlock test ([#2716]) +9. Make delta_kernel_default_engine publishable ([#2561]) +10. *(deps)* Bump rustls-webpki from 0.103.10 to 0.103.13 ([#2735]) +11. Run body validator on synchronize and reopened ([#2748]) +12. Fail bench job on 15% regressions and tighten the bench comment ([#2742]) + + +[#2688]: https://github.com/delta-io/delta-kernel-rs/pull/2688 +[#2682]: https://github.com/delta-io/delta-kernel-rs/pull/2682 +[#2693]: https://github.com/delta-io/delta-kernel-rs/pull/2693 +[#2623]: https://github.com/delta-io/delta-kernel-rs/pull/2623 +[#1858]: https://github.com/delta-io/delta-kernel-rs/pull/1858 +[#2691]: https://github.com/delta-io/delta-kernel-rs/pull/2691 +[#2365]: https://github.com/delta-io/delta-kernel-rs/pull/2365 +[#2671]: https://github.com/delta-io/delta-kernel-rs/pull/2671 +[#2684]: https://github.com/delta-io/delta-kernel-rs/pull/2684 +[#2582]: https://github.com/delta-io/delta-kernel-rs/pull/2582 +[#2663]: https://github.com/delta-io/delta-kernel-rs/pull/2663 +[#2522]: https://github.com/delta-io/delta-kernel-rs/pull/2522 +[#2699]: https://github.com/delta-io/delta-kernel-rs/pull/2699 +[#2700]: https://github.com/delta-io/delta-kernel-rs/pull/2700 +[#2697]: https://github.com/delta-io/delta-kernel-rs/pull/2697 +[#2613]: https://github.com/delta-io/delta-kernel-rs/pull/2613 +[#2666]: https://github.com/delta-io/delta-kernel-rs/pull/2666 +[#2680]: https://github.com/delta-io/delta-kernel-rs/pull/2680 +[#2694]: https://github.com/delta-io/delta-kernel-rs/pull/2694 +[#2637]: https://github.com/delta-io/delta-kernel-rs/pull/2637 +[#2704]: https://github.com/delta-io/delta-kernel-rs/pull/2704 +[#2397]: https://github.com/delta-io/delta-kernel-rs/pull/2397 +[#2681]: https://github.com/delta-io/delta-kernel-rs/pull/2681 +[#2708]: https://github.com/delta-io/delta-kernel-rs/pull/2708 +[#2711]: https://github.com/delta-io/delta-kernel-rs/pull/2711 +[#2678]: https://github.com/delta-io/delta-kernel-rs/pull/2678 +[#2649]: https://github.com/delta-io/delta-kernel-rs/pull/2649 +[#2685]: https://github.com/delta-io/delta-kernel-rs/pull/2685 +[#2702]: https://github.com/delta-io/delta-kernel-rs/pull/2702 +[#2712]: https://github.com/delta-io/delta-kernel-rs/pull/2712 +[#2716]: https://github.com/delta-io/delta-kernel-rs/pull/2716 +[#2705]: https://github.com/delta-io/delta-kernel-rs/pull/2705 +[#2629]: https://github.com/delta-io/delta-kernel-rs/pull/2629 +[#2703]: https://github.com/delta-io/delta-kernel-rs/pull/2703 +[#2675]: https://github.com/delta-io/delta-kernel-rs/pull/2675 +[#2713]: https://github.com/delta-io/delta-kernel-rs/pull/2713 +[#2719]: https://github.com/delta-io/delta-kernel-rs/pull/2719 +[#2599]: https://github.com/delta-io/delta-kernel-rs/pull/2599 +[#2665]: https://github.com/delta-io/delta-kernel-rs/pull/2665 +[#2727]: https://github.com/delta-io/delta-kernel-rs/pull/2727 +[#2728]: https://github.com/delta-io/delta-kernel-rs/pull/2728 +[#2672]: https://github.com/delta-io/delta-kernel-rs/pull/2672 +[#2726]: https://github.com/delta-io/delta-kernel-rs/pull/2726 +[#2706]: https://github.com/delta-io/delta-kernel-rs/pull/2706 +[#2741]: https://github.com/delta-io/delta-kernel-rs/pull/2741 +[#2561]: https://github.com/delta-io/delta-kernel-rs/pull/2561 +[#2735]: https://github.com/delta-io/delta-kernel-rs/pull/2735 +[#2743]: https://github.com/delta-io/delta-kernel-rs/pull/2743 +[#2744]: https://github.com/delta-io/delta-kernel-rs/pull/2744 +[#2729]: https://github.com/delta-io/delta-kernel-rs/pull/2729 +[#2714]: https://github.com/delta-io/delta-kernel-rs/pull/2714 +[#2721]: https://github.com/delta-io/delta-kernel-rs/pull/2721 +[#2730]: https://github.com/delta-io/delta-kernel-rs/pull/2730 +[#2748]: https://github.com/delta-io/delta-kernel-rs/pull/2748 +[#2738]: https://github.com/delta-io/delta-kernel-rs/pull/2738 +[#2734]: https://github.com/delta-io/delta-kernel-rs/pull/2734 +[#2742]: https://github.com/delta-io/delta-kernel-rs/pull/2742 +[#2715]: https://github.com/delta-io/delta-kernel-rs/pull/2715 +[#2761]: https://github.com/delta-io/delta-kernel-rs/pull/2761 +[#2745]: https://github.com/delta-io/delta-kernel-rs/pull/2745 +[#2670]: https://github.com/delta-io/delta-kernel-rs/pull/2670 +[#2732]: https://github.com/delta-io/delta-kernel-rs/pull/2732 +[#2657]: https://github.com/delta-io/delta-kernel-rs/pull/2657 +[#2758]: https://github.com/delta-io/delta-kernel-rs/pull/2758 +[#2514]: https://github.com/delta-io/delta-kernel-rs/pull/2514 +[#2771]: https://github.com/delta-io/delta-kernel-rs/pull/2771 + + +## [v0.24.0](https://github.com/delta-io/delta-kernel-rs/tree/v0.24.0/) (2026-05-29) + +[Full Changelog](https://github.com/delta-io/delta-kernel-rs/compare/v0.23.0...v0.24.0) + + +### 🏗️ Breaking changes + +1. Reorganize `metrics` module for better cohesion ([#2604]) + - `MetricEvent` variants are tuple variants wrapping a per-event struct (e.g. + `MetricEvent::ScanMetadataCompleted(ScanMetadataCompleted { .. })`). Callers matching on + `MetricEvent` must match the wrapped struct instead of inline named fields. +2. Add CRC read metrics via tracing instrumentation ([#2540]) + - Adds the `MetricEvent::CrcReadCompleted` event and a `has_latest_crc_file` field on + `LogSegmentLoaded`. Exhaustive matches on `MetricEvent` must handle the new variant. +3. Add file size metrics for scans ([#2585]) + - `ScanMetadataCompleted` gains an `active_add_files_bytes: u64` field (total size of add + files surviving log replay). Callers constructing or exhaustively matching it must account + for the new field. +4. Re-add `nearest_timestamp` to `LogHistoryError` ([#2600]) + - The `LogHistoryError::TimestampOutOfRange` variant gains a `nearest_timestamp` field. + Callers matching this variant with named fields must add `nearest_timestamp` (or `..`). +5. Rename `Plan` enum to `Operation` ([#2635]) + - Rename `Plan` to `Operation` at all call sites (exported from `delta_kernel::plans` under + the `declarative-plans` feature). Variant names are unchanged. +6. Introduce `DomainMetadataState` typed enum ([#2567]) + - The `Crc` field `domain_metadata: Option>` becomes + `domain_metadata_state: DomainMetadataState` (`Complete` / `Partial`). Affects consumers of + the `crc` module, which is public only under the `test-utils` feature. +7. Introduce `SetTransactionState` typed enum ([#2570]) + - The `Crc` field `set_transactions: Option>` becomes + `set_transaction_state: SetTransactionState` (`Complete` / `Partial`). Affects consumers of + the `crc` module, which is public only under the `test-utils` feature. +8. Support `variantShredding` as a feature ([#2594]) + - Kernel now supports the `variantShredding` table feature (previously only + `variantShredding-preview`). Tables enabling it are read and written instead of rejected as + an unsupported feature. + +### 🚀 Features / new APIs + +1. *(ffi)* Expose transaction deletion vector updates ([#2495]) +2. Allow fields with same physical name and different physical path ([#2576]) +3. Enforce CRC ICT consistency on write ([#2584]) +4. Support creating Delta tables with empty schema ([#2545]) +5. Two tiny CRC refactors ([#2597]) +6. Block read and blind-append on empty-schema Delta tables ([#2546]) +7. Introduce PlanExecutor trait + basic plan IR ([#2590]) +8. Reverse incremental CRC replay accumulator + visitor ([#2602]) +9. Introduce PlanBasedEngine and a SyncPlanExecutor ([#2621]) + +### 🐛 Bug Fixes + +1. Block removeFile for some row tracking table ([#2539]) +2. Coerce field names during parquet field-id matching ([#2558]) +3. Surface numRecords from struct-stats-only checkpoints via ScanFile.stats ([#2542]) +4. Drop predicate refs to non-stats columns during rewrite ([#2575]) + +### 📚 Documentation + +1. Update list_from doc ([#2581]) +2. Minor user guide README update ([#2601]) + +### 🚜 Refactor + +1. Move apply_schema out of arrow_expression ([#2580]) +2. Create stat field name constants ([#2586]) +3. Reshape CrcDelta, get it ready for incremental visitor and accumulator ([#2583]) +4. CRC test infra / setup ([#2616]) +5. Use DeltaResultIterator where possible ([#2622]) +6. Rename ScopedDeltaResultIterator ([#2632]) +7. Add snapshot/incremental.rs; snapshot section headers ([#2647]) + +### 🧪 Testing + +1. Support nested structs in table-builder data generation ([#2648]) + + +[#2567]: https://github.com/delta-io/delta-kernel-rs/pull/2567 +[#2495]: https://github.com/delta-io/delta-kernel-rs/pull/2495 +[#2539]: https://github.com/delta-io/delta-kernel-rs/pull/2539 +[#2570]: https://github.com/delta-io/delta-kernel-rs/pull/2570 +[#2558]: https://github.com/delta-io/delta-kernel-rs/pull/2558 +[#2580]: https://github.com/delta-io/delta-kernel-rs/pull/2580 +[#2581]: https://github.com/delta-io/delta-kernel-rs/pull/2581 +[#2540]: https://github.com/delta-io/delta-kernel-rs/pull/2540 +[#2542]: https://github.com/delta-io/delta-kernel-rs/pull/2542 +[#2576]: https://github.com/delta-io/delta-kernel-rs/pull/2576 +[#2584]: https://github.com/delta-io/delta-kernel-rs/pull/2584 +[#2586]: https://github.com/delta-io/delta-kernel-rs/pull/2586 +[#2583]: https://github.com/delta-io/delta-kernel-rs/pull/2583 +[#2575]: https://github.com/delta-io/delta-kernel-rs/pull/2575 +[#2545]: https://github.com/delta-io/delta-kernel-rs/pull/2545 +[#2597]: https://github.com/delta-io/delta-kernel-rs/pull/2597 +[#2546]: https://github.com/delta-io/delta-kernel-rs/pull/2546 +[#2594]: https://github.com/delta-io/delta-kernel-rs/pull/2594 +[#2601]: https://github.com/delta-io/delta-kernel-rs/pull/2601 +[#2590]: https://github.com/delta-io/delta-kernel-rs/pull/2590 +[#2585]: https://github.com/delta-io/delta-kernel-rs/pull/2585 +[#2616]: https://github.com/delta-io/delta-kernel-rs/pull/2616 +[#2622]: https://github.com/delta-io/delta-kernel-rs/pull/2622 +[#2602]: https://github.com/delta-io/delta-kernel-rs/pull/2602 +[#2621]: https://github.com/delta-io/delta-kernel-rs/pull/2621 +[#2632]: https://github.com/delta-io/delta-kernel-rs/pull/2632 +[#2635]: https://github.com/delta-io/delta-kernel-rs/pull/2635 +[#2604]: https://github.com/delta-io/delta-kernel-rs/pull/2604 +[#2600]: https://github.com/delta-io/delta-kernel-rs/pull/2600 +[#2647]: https://github.com/delta-io/delta-kernel-rs/pull/2647 +[#2648]: https://github.com/delta-io/delta-kernel-rs/pull/2648 + + +## [v0.23.0](https://github.com/delta-io/delta-kernel-rs/tree/v0.23.0/) (2026-05-15) + +[Full Changelog](https://github.com/delta-io/delta-kernel-rs/compare/v0.22.1...v0.23.0) + + +### 🏗️ Breaking changes + +1. Add `delta.parquet.compression.codec` table property ([#2523]) + - Adds `parquet_compression_codec: Option` field to + `TableProperties`. Callers using exhaustive struct construction must add + `parquet_compression_codec: None`. Parsing is case-insensitive and accepts `none` as an + alias for `uncompressed`, matching the protocol. +2. Schema and expression transforms use generic carriers ([#2151]) + - `ExpressionTransform` and `SchemaTransform` traits now use a generic `Carrier` type + instead of being hard-wired to `Option>`. Implementations must declare the new + associated types -- use the `transform_output_type!` macro and pick a carrier that + matches the transform's behavior (`Option` for filtering, `Cow` for non-filtering, + `()` for infallible visitors, `Result<(), ()>` for short-circuiting visitors, + `DeltaResult<()>` for fallible validators, etc.). Existing transforms can usually be + simplified once ported to the appropriate carrier. +### 🚀 Features / new APIs + +1. Materialize partition values when `icebergCompatV3` enabled ([#2504]) +2. Enforce and test non-null column invariant on writes ([#2483]) +3. Block alter table when `icebergCompatV3` enabled ([#2505]) +4. Write parquet id for nested items in map/array according to metadata ([#2508]) +5. Validate `stats.numRecords` on writes for icebergCompatV3 tables ([#2512]) +6. Expose LastCheckpointHint's path() under internal-api ([#2487]) +7. Add with_operation in ffi ([#2532]) +8. Support `icebergCompatV3` for create table ([#2517]) +9. Support `icebergCompatV3` ([#2518]) +10. Respect randomizeFilePrefixes and randomPrefixLength in write_dir ([#2513]) +11. Lazily create default executor in DefaultEngineBuilder ([#2536]) +12. Expose CommittedTransaction with post-commit snapshot ([#2488]) +13. Expose ffi accessors for column mapping write path ([#2509]) +14. Preserve pre-existing column mapping metadata on writes ([#2500]) +15. Add IncrementalScanBuilder (raw streaming) ([#2519]) +16. Classify metadata-only re-adds on incremental scan ([#2520]) + +### 🐛 Bug Fixes + +1. Optimize incremental snapshot updates on new checkpoint discovery ([#2351]) +2. Handle `LargeStringArray` in `parse_json` ([#1938]) +3. Accumulate warnings so reporter no longer has deadlock risk ([#2550]) +4. Put correct msrv badge in readme ([#2556]) +5. Share without_partition physical schema across TableConfiguration clones ([#2560]) + +### 📚 Documentation + +1. Add kernel rust user guide ([#2479]) + +### 🚜 Refactor + +1. Enhance SyncEngine with ObjectStore support ([#2396]) +2. Cleanup `crc/mod.rs` ([#2559]) +3. Introduce `FileStatsState` typed enum ([#2562]) + +### 🧪 Testing + +1. Cover non-catalog-managed UC tables in read/write examples ([#2463]) +2. Cover partitionValues_parsed in partitioned write tests ([#2467]) +3. Fix flaky metrics tests caused by tracing callsite cache poisoning ([#2484]) +4. Add multi-checkpoint support to TestTableBuilder ([#2284]) +5. Consolidate some test lines + script to check ([#2348]) +6. Add post-cleanup for TestTableBuilder ([#2543]) +7. Add LastCheckpointHintState (Present/Missing/Stale) for TestTableBuilder ([#2521]) + +### ⚙️ Chores/CI + +1. Clean full target before user-guide doc tests ([#2548]) +2. Bump object_store 0.13.1 -> 0.13.2 ([#2366]) +3. Bring changelog up to date for 0.22.1 ([#2554]) +4. Disable rust-cache cache-bin to fix macOS runner ([#2566]) +5. *(deps)* Bump openssl from 0.10.76 to 0.10.79 ([#2563]) + + +[#2479]: https://github.com/delta-io/delta-kernel-rs/pull/2479 +[#2151]: https://github.com/delta-io/delta-kernel-rs/pull/2151 +[#2504]: https://github.com/delta-io/delta-kernel-rs/pull/2504 +[#2483]: https://github.com/delta-io/delta-kernel-rs/pull/2483 +[#2505]: https://github.com/delta-io/delta-kernel-rs/pull/2505 +[#2508]: https://github.com/delta-io/delta-kernel-rs/pull/2508 +[#2512]: https://github.com/delta-io/delta-kernel-rs/pull/2512 +[#2463]: https://github.com/delta-io/delta-kernel-rs/pull/2463 +[#2467]: https://github.com/delta-io/delta-kernel-rs/pull/2467 +[#2484]: https://github.com/delta-io/delta-kernel-rs/pull/2484 +[#2487]: https://github.com/delta-io/delta-kernel-rs/pull/2487 +[#2351]: https://github.com/delta-io/delta-kernel-rs/pull/2351 +[#2532]: https://github.com/delta-io/delta-kernel-rs/pull/2532 +[#2517]: https://github.com/delta-io/delta-kernel-rs/pull/2517 +[#2523]: https://github.com/delta-io/delta-kernel-rs/pull/2523 +[#2518]: https://github.com/delta-io/delta-kernel-rs/pull/2518 +[#2513]: https://github.com/delta-io/delta-kernel-rs/pull/2513 +[#1938]: https://github.com/delta-io/delta-kernel-rs/pull/1938 +[#2536]: https://github.com/delta-io/delta-kernel-rs/pull/2536 +[#2284]: https://github.com/delta-io/delta-kernel-rs/pull/2284 +[#2348]: https://github.com/delta-io/delta-kernel-rs/pull/2348 +[#2488]: https://github.com/delta-io/delta-kernel-rs/pull/2488 +[#2548]: https://github.com/delta-io/delta-kernel-rs/pull/2548 +[#2509]: https://github.com/delta-io/delta-kernel-rs/pull/2509 +[#2366]: https://github.com/delta-io/delta-kernel-rs/pull/2366 +[#2550]: https://github.com/delta-io/delta-kernel-rs/pull/2550 +[#2396]: https://github.com/delta-io/delta-kernel-rs/pull/2396 +[#2554]: https://github.com/delta-io/delta-kernel-rs/pull/2554 +[#2500]: https://github.com/delta-io/delta-kernel-rs/pull/2500 +[#2556]: https://github.com/delta-io/delta-kernel-rs/pull/2556 +[#2543]: https://github.com/delta-io/delta-kernel-rs/pull/2543 +[#2566]: https://github.com/delta-io/delta-kernel-rs/pull/2566 +[#2562]: https://github.com/delta-io/delta-kernel-rs/pull/2562 +[#2519]: https://github.com/delta-io/delta-kernel-rs/pull/2519 +[#2560]: https://github.com/delta-io/delta-kernel-rs/pull/2560 +[#2520]: https://github.com/delta-io/delta-kernel-rs/pull/2520 +[#2563]: https://github.com/delta-io/delta-kernel-rs/pull/2563 +[#2559]: https://github.com/delta-io/delta-kernel-rs/pull/2559 +[#2521]: https://github.com/delta-io/delta-kernel-rs/pull/2521 + + +## [v0.22.1](https://github.com/delta-io/delta-kernel-rs/tree/v0.22.1/) (2026-01-20) + +[Full Changelog](https://github.com/delta-io/delta-kernel-rs/compare/v0.22.0...v0.22.1) + +### 🐛 Bug Fixes + +1. fix: accumulate warnings so reporter no longer has deadlock risk. Previous the `warn!` in some + reporter code could cause deadlocks. (#2550) + +[#2550]: https://github.com/delta-io/delta-kernel-rs/pull/2550 + +## [v0.22.0](https://github.com/delta-io/delta-kernel-rs/tree/v0.22.0/) (2026-04-29) + +[Full Changelog](https://github.com/delta-io/delta-kernel-rs/compare/v0.21.0...v0.22.0) + + +### 🏗️ Breaking changes + +1. Add delta.parquet.format.version table property ([#2369]) + - Adds `parquet_format_version: Option` field to `TableProperties`. Callers using + exhaustive struct construction must add `parquet_format_version: None`. +2. Robust partitioned-write APIs ([#2356]) + - Replaces the old `WriteContext` API with partition-aware variants: use + `txn.partitioned_write_context(partition_values)` or `txn.unpartitioned_write_context()`. + Partition values are now passed as `Map` (kernel handles serialization per + the Delta spec) instead of `Map`. + `DefaultParquetHandler::write_parquet_file` signature changed from + `(path, data, partition_values, stats_columns)` to `(data, &WriteContext)`. See + `kernel/tests/integration/write/partitioned.rs` and the updated + `kernel/examples/write-table` for migration examples. +3. Add typed null literal support to FFI expression visitors ([#2375]) + - FFI `visit_expression_literal_null` and `visit_literal_null` now accept `type_tag` (plus + `precision`/`scale` for decimal). FFI engines must provide type information when emitting + null literals; see the new `NullTypeTag` enum for the contract. +4. Change get_create_table_builder to accept EngineSchema visitor ([#2378]) + - FFI `get_create_table_builder` now takes `&EngineSchema` instead of `Handle`. + Engines must implement the visitor pattern (`visit_field_*` callbacks) to populate the + schema, matching the existing `scan_builder_with_schema` pattern. See the + `ffi/examples/create-table` example for a reference implementation. +5. Transform stats parsed for remove actions ([#2061]) + - Adds `fn has_field(&self, name: &ColumnName) -> bool` to the `EngineData` trait. Custom + `EngineData` implementors must add this method. +6. Default to writing relative paths in `add.path` ([#2410]) + - Kernel now writes **relative** paths in `add.path` (e.g. `abc.parquet`) instead of absolute + URLs (e.g. `s3://bucket/table/abc.parquet`), matching Delta Spark. +7. Add tests for histograms and expose stats and histogram ([#2373]) + - New public types `FileStats` and `FileSizeHistogram`, plus `Snapshot::get_or_load_file_stats` + and `Snapshot::get_file_stats_if_loaded` accessors. Connectors can read CRC file-stats + histogram data through these public APIs. +8. Separate read state from effective state in Transaction ([#2385]) + - Internal `Transaction` refactor: splits the held snapshot into + `read_snapshot_opt: Option` (pre-commit state) and + `effective_table_config: TableConfiguration` (state this commit will produce). +9. Make scan_table_changes_next return *mut ArrowFFIData ([#2430]) + - The Ok variant of FFI `scan_table_changes_next`'s `ExternResult` is now a heap-allocated + `*mut ArrowFFIData` instead of an inline `ArrowFFIData` value. FFI consumers must switch + from value-style access (`res.ok`) to pointer-style and call the new `free_arrow_ffi_data` + on non-null results. +10. Update `CheckpointWriter::finalize` to accept `LastCheckpointHintStats` ([#2400]) + - `CheckpointWriter::finalize` now takes a `LastCheckpointHintStats` struct instead of + `(FileMeta, ActionReconciliationIteratorState)`. Construct the new struct to correctly + populate `_last_checkpoint` (including `sizeInBytes` and `size`) for V2 checkpoints with + sidecars. +11. Replace existing metrics reporting with tracing ([#1822]) + - `MetricsReporter` is no longer part of the `Engine` APIs. Instead you can register a + reporter as a tracing layer. See `kernel/examples/inspect-table` for an example of how + to migrate. +12. Add CheckpointRowGroupFilter for checkpoint data skipping ([#1893]) + - `ParquetStatsProvider::get_parquet_rowcount_stat` return type changed from `i64` to + `Option`. Engines implementing `ParquetStatsProvider` must wrap their existing return + in `Some(num_rows)`. + +### 🚀 Features / new APIs + +1. Add infrastructures for sidecar splitting support ([#2271]) +2. Add `extract_primitive_scalar` utility for Arrow-to-Scalar conversion ([#2368]) +3. `ParquetHandler` auto-creates the target directory if it does not exist ([#2287]) +4. Add schema validation for CREATE TABLE ([#2309]) +5. Add row tracking support for create table ([#2317]) +6. *(tests)* Add read-path integration tests for row tracking ([#2316]) +7. Add Arrow batch-mode scan metadata FFI ([#2395]) +8. Reject non-null columns in CREATE TABLE unconditionally ([#2404]) +9. Auto-enable invariants writer feature for non-null columns in CREATE TABLE ([#2418]) +10. Add ffi examples for cdf, create-table, write-table ([#2431]) +11. Add high level api for timestamp conversion ([#900]) +12. Collect nullCount statistics for array, map, and variant columns ([#2442]) +13. Add AlterTable framework with add_column support ([#2387]) +14. Add set_nullable support for ALTER TABLE ([#2388]) +15. Allow materializePartitionColumns feature signal in CREATE TABLE ([#2481]) + +### 🐛 Bug Fixes + +1. Acceptance test framework should reject negative snapshot version ([#2364]) +2. Exclude partition columns from write-path stats collection ([#2362]) +3. Add BYTE/SHORT support to stats verifier and GetData trait ([#2382]) +4. Support presigned DB URLs ([#2398]) +5. Ensure doctests are run in GitHub Actions ([#2412]) +6. Missing PR link in the v0.21.0 CHANGELOG ([#2428]) +7. Drop partitionValues_parsed in build_remove_transform ([#2429]) +8. URI-encode Hive partition path for partitioned writes ([#2424]) +9. Propagate null bitmap in evaluate_map_to_struct ([#2419]) +10. Correct inaccuracies in ffi examples ([#2432]) +11. Restore the Snapshot::new internal API ([#2425]) +12. Clear stale CRC file in LogSegment::try_new_with_checkpoint ([#2457]) +13. Keep name-based validation for column expressions with struct ([#2440]) +14. Reuse LazyCrc in checkpoint early-return during incremental update ([#2329]) +15. Add deserialization alias for file histogram ([#2489]) + +### 📚 Documentation + +1. Remove references to default-engine feature ([#2417]) + +### 🚜 Refactor + +1. Extract prerequisite schema constructions for `CheckpointWriter` ([#2313]) +2. Change inconsistent kernel modules to use mod convention ([#2408]) +3. Enforce line width and import ordering with nightly rustfmt ([#2383]) +4. Add more test setup utils for partitioned write tests ([#2422]) +5. Move data file methods behind SupportsDataFiles trait ([#2386]) + +### 🧪 Testing + +1. Add CountingReporter integration tests across different scenarios ([#2194]) +2. Predicate parser on acceptance workload harness ([#2215]) +3. Add partition support for TestTableBuilder ([#2321]) +4. Add FeatureSet methods for table builder ([#2283]) +5. Split write.rs into topic-focused files ([#2460]) +6. Consolidate write tests into single [[test]] binary ([#2472]) +7. Collapse integration tests into a single binary ([#2477]) +8. Migrate a test to use the create table builder ([#2482]) + +### ⚙️ Chores/CI + +1. Don't generate unused Arrow schema. ([#2107]) +2. Validate ascii only in PR body via CI job ([#2405]) +3. Skip invalid handle code tests for coverage ([#2414]) +4. PathMode ([#2360], [#2410]) and `WriteContext::partition_group_key` ([#2392], [#2403]) were + added and reverted within this release. No net user-visible change. + + +[#2369]: https://github.com/delta-io/delta-kernel-rs/pull/2369 +[#2364]: https://github.com/delta-io/delta-kernel-rs/pull/2364 +[#2271]: https://github.com/delta-io/delta-kernel-rs/pull/2271 +[#2194]: https://github.com/delta-io/delta-kernel-rs/pull/2194 +[#2368]: https://github.com/delta-io/delta-kernel-rs/pull/2368 +[#2362]: https://github.com/delta-io/delta-kernel-rs/pull/2362 +[#2356]: https://github.com/delta-io/delta-kernel-rs/pull/2356 +[#2287]: https://github.com/delta-io/delta-kernel-rs/pull/2287 +[#2215]: https://github.com/delta-io/delta-kernel-rs/pull/2215 +[#2382]: https://github.com/delta-io/delta-kernel-rs/pull/2382 +[#2375]: https://github.com/delta-io/delta-kernel-rs/pull/2375 +[#2309]: https://github.com/delta-io/delta-kernel-rs/pull/2309 +[#2392]: https://github.com/delta-io/delta-kernel-rs/pull/2392 +[#2317]: https://github.com/delta-io/delta-kernel-rs/pull/2317 +[#2378]: https://github.com/delta-io/delta-kernel-rs/pull/2378 +[#2398]: https://github.com/delta-io/delta-kernel-rs/pull/2398 +[#2316]: https://github.com/delta-io/delta-kernel-rs/pull/2316 +[#2403]: https://github.com/delta-io/delta-kernel-rs/pull/2403 +[#2107]: https://github.com/delta-io/delta-kernel-rs/pull/2107 +[#2061]: https://github.com/delta-io/delta-kernel-rs/pull/2061 +[#2395]: https://github.com/delta-io/delta-kernel-rs/pull/2395 +[#2360]: https://github.com/delta-io/delta-kernel-rs/pull/2360 +[#2373]: https://github.com/delta-io/delta-kernel-rs/pull/2373 +[#2313]: https://github.com/delta-io/delta-kernel-rs/pull/2313 +[#2408]: https://github.com/delta-io/delta-kernel-rs/pull/2408 +[#2410]: https://github.com/delta-io/delta-kernel-rs/pull/2410 +[#2383]: https://github.com/delta-io/delta-kernel-rs/pull/2383 +[#2412]: https://github.com/delta-io/delta-kernel-rs/pull/2412 +[#2405]: https://github.com/delta-io/delta-kernel-rs/pull/2405 +[#2404]: https://github.com/delta-io/delta-kernel-rs/pull/2404 +[#2414]: https://github.com/delta-io/delta-kernel-rs/pull/2414 +[#2385]: https://github.com/delta-io/delta-kernel-rs/pull/2385 +[#2422]: https://github.com/delta-io/delta-kernel-rs/pull/2422 +[#2417]: https://github.com/delta-io/delta-kernel-rs/pull/2417 +[#2428]: https://github.com/delta-io/delta-kernel-rs/pull/2428 +[#2429]: https://github.com/delta-io/delta-kernel-rs/pull/2429 +[#2321]: https://github.com/delta-io/delta-kernel-rs/pull/2321 +[#2424]: https://github.com/delta-io/delta-kernel-rs/pull/2424 +[#2418]: https://github.com/delta-io/delta-kernel-rs/pull/2418 +[#2419]: https://github.com/delta-io/delta-kernel-rs/pull/2419 +[#2432]: https://github.com/delta-io/delta-kernel-rs/pull/2432 +[#2430]: https://github.com/delta-io/delta-kernel-rs/pull/2430 +[#2400]: https://github.com/delta-io/delta-kernel-rs/pull/2400 +[#2425]: https://github.com/delta-io/delta-kernel-rs/pull/2425 +[#2431]: https://github.com/delta-io/delta-kernel-rs/pull/2431 +[#1822]: https://github.com/delta-io/delta-kernel-rs/pull/1822 +[#900]: https://github.com/delta-io/delta-kernel-rs/pull/900 +[#2442]: https://github.com/delta-io/delta-kernel-rs/pull/2442 +[#2386]: https://github.com/delta-io/delta-kernel-rs/pull/2386 +[#2283]: https://github.com/delta-io/delta-kernel-rs/pull/2283 +[#2457]: https://github.com/delta-io/delta-kernel-rs/pull/2457 +[#1893]: https://github.com/delta-io/delta-kernel-rs/pull/1893 +[#2440]: https://github.com/delta-io/delta-kernel-rs/pull/2440 +[#2460]: https://github.com/delta-io/delta-kernel-rs/pull/2460 +[#2387]: https://github.com/delta-io/delta-kernel-rs/pull/2387 +[#2472]: https://github.com/delta-io/delta-kernel-rs/pull/2472 +[#2388]: https://github.com/delta-io/delta-kernel-rs/pull/2388 +[#2481]: https://github.com/delta-io/delta-kernel-rs/pull/2481 +[#2477]: https://github.com/delta-io/delta-kernel-rs/pull/2477 +[#2329]: https://github.com/delta-io/delta-kernel-rs/pull/2329 +[#2482]: https://github.com/delta-io/delta-kernel-rs/pull/2482 +[#2489]: https://github.com/delta-io/delta-kernel-rs/pull/2489 + + +## [v0.21.0](https://github.com/delta-io/delta-kernel-rs/tree/v0.21.0/) (2026-04-10) + +[Full Changelog](https://github.com/delta-io/delta-kernel-rs/compare/v0.20.0...v0.21.0) + + +### 🏗️ Breaking changes + +1. Add partitioned variant to DataLayout enum ([#2145]) + - Adds `Partitioned` variant to `DataLayout` enum. Update match statements to handle the new variant. +2. Add create many API to engine ([#2070]) + - Adds `create_many` method to `ParquetHandler` trait. Implementors must add this method. See the trait rustdocs for details. +3. Rename uc-catalog and uc-client crates ([#2136]) + - `delta-kernel-uc-catalog` renamed to `delta-kernel-unity-catalog`. `delta-kernel-uc-client` renamed to `unity-catalog-delta-rest-client`. Update `Cargo.toml` dependencies accordingly. +4. Checksum and checkpoint APIs return updated Snapshot ([#2182]) + - `Snapshot::checkpoint()` and checksum APIs now return the updated `Snapshot`. Callers must handle the returned value. +5. Add P&M to CommitMetadata and enforce committer/table type matching ([#2250]) + - Enforces that committer type matches table type (catalog-managed vs path-based). Use appropriate committer for your table type. +6. Add UCCommitter validation for catalog-managed tables ([#2254]) + - `UCCommitter` now rejects commits to non-catalog-managed tables. Use `FileSystemCommitter` for path-based tables. +7. Refactor snapshot FFI to use builder pattern and enable snapshot reuse ([#2255]) + - FFI snapshot creation now uses builder pattern. Update FFI callers to use the new builder APIs. +8. Make tags and remove partition values allow null values in map ([#2281]) + - `tags` and `partitionValues` map values are now nullable. Update code that assumes non-null values. +9. Better naming style for column mapping related functions/variables ([#2290]) + - Renamed: `make_physical` to `to_physical_name`, `make_physical_struct` to `to_physical_schema`, `transform_struct_for_projection` to `projection_transform`. Update call sites. +10. Remove the catalog-managed feature flag ([#2310]) + - The `catalog-managed` feature flag is removed. Catalog-managed table support is now always available. +11. Update snapshot.checkpoint API to return a CheckpointResult ([#2314]) + - `Snapshot::checkpoint()` now returns `CheckpointResult` instead of `Snapshot`. Access the snapshot via `CheckpointResult::snapshot`. +12. Remove old non-builder snapshot FFI functions ([#2318]) + - Removed legacy FFI snapshot functions. Use the new builder-pattern FFI functions instead. +13. Support version 0 (table creation) commits in UCCommitter ([#2247]) + - Connectors using `UCCommitter` for table creation must now handle post-commit finalization via the UC create table API. +14. Pass computed ICT to CommitMetadata instead of wall-clock time ([#2319]) + - `CommitMetadata` now uses computed in-commit timestamp instead of wall-clock time. Callers relying on wall-clock timing should update accordingly. +15. Upgrade to arrow-58 and object_store-13, drop arrow-56 support ([#2116]) + - Minimum supported Arrow version is now arrow-57. Update your `Cargo.toml` if using `arrow-56` feature. +16. Crc File Histogram Read and Write Support ([#2235]) + - Adds `AddedHistogram` and `RemovedHistogram` fields to `FileStatsDelta` struct. +17. Add ScanMetadataCompleted metric event ([#2236]) + - Adds `ScanMetadataCompleted` variant to `MetricEvent` enum. Update metric reporters to handle the new variant. +18. Instrument JSON and Parquet handler reads with MetricsReporter ([#2169]) + - Adds `JsonReadCompleted` and `ParquetReadCompleted` variants to `MetricEvent` enum. Update metric reporters to handle new variants. +19. New transform helpers for unary and binary children ([#2150]) + - Removes public `CowExt` trait. Remove any usages of this trait. +20. New mod transforms for expression and schema transforms ([#2077]) + - Moves `SchemaTransform` and `ExpressionTransform` to new `transforms` module. Update import paths. +21. Introduce object_store compat shim ([#2111]) + - Renames `object_store` dependency to `object_store_12`. Update any direct references. +22. Consolidate domain metadata reads through Snapshot ([#2065]) + - Domain metadata reads now go through `Snapshot` methods. Update callers using old free functions. +23. Don't read or write arrow schema in parquet files ([#2025]) + - Parquet files no longer include arrow schema metadata. Code relying on this metadata must be updated. +24. Rename include_stats_columns to include_all_stats_columns ([#1996]) + - Renames `ScanBuilder::include_stats_columns()` to `ScanBuilder::include_all_stats_columns()`. Update call sites. + +### 🚀 Features / new APIs + +1. Add SQL -> Kernel predicate parser to benchmark framework ([#2099]) +2. Add observability metrics for scan log replay ([#1866]) +3. Filtered engine data visitor ([#1942]) +4. Trigger benchmarking with comments ([#2089]) +5. Unify data stats and partition values in DataSkippingFilter ([#1948]) +6. Download benchmark workloads from DAT release ([#2163]) +7. Add partitioned variant to DataLayout enum ([#2145]) +8. Expose table_properties in FFI via visit_table_properties ([#2196]) +9. Allow checkpoint stats properties in CREATE TABLE ([#2210]) +10. Add crc file histogram initial struct and methods ([#2212]) +11. BinaryPredicate evaluate expression with ArrowViewType. ([#2052]) +12. Add acceptance workloads testing harness ([#2092]) +13. Enable DeletionVectors table feature in CREATE TABLE ([#2245]) +14. Checksum and checkpoint APIs return updated Snapshot ([#2182]) +15. Adding ScanBuilder FFI functions for Scans ([#2237]) +16. Add CountingReporter and fix metrics forwarding ([#2166]) +17. Instrument JSON and Parquet handler reads with MetricsReporter ([#2169]) +18. Wire CountingReporter into workload benchmarks ([#2171]) +19. Add create many API to engine ([#2070]) +20. Add ScanMetadataCompleted metric event ([#2236]) +21. Allow AppendOnly, ChangeDataFeed, and TypeWidening in CREATE TABLE ([#2279]) +22. Support max timestamp stats for data skipping ([#2249]) +23. Add list with backward checkpoint scan ([#2174]) +24. Add Snapshot::get_timestamp ([#2266]) +25. Make tags and remove partition values allow null values in map ([#2281]) +26. Support UC credential vending and S3 benchmarks ([#2109]) +27. Add catalogManaged to allowed features in CREATE TABLE ([#2293]) +28. Add catalog-managed table creation utilities ([#2203]) +29. Support version 0 (table creation) commits in UCCommitter ([#2247]) +30. Update snapshot.checkpoint API to return a CheckpointResult ([#2314]) +31. Cached checkpoint output schema ([#2270]) +32. Refactor snapshot FFI to use builder pattern and enable snapshot reuse ([#2255]) +33. Add P&M to CommitMetadata and enforce committer/table type matching ([#2250]) +34. Add UCCommitter validation for catalog-managed tables ([#2254]) +35. Crc File Histogram Read and Write Support ([#2235]) +36. Add FFI function to expose snapshot's timestamp ([#2274]) +37. Add FFI create table DDL functions ([#2296]) +38. Add FFI remove files DML functions ([#2297]) +39. Expose Protocol and Metadata as opaque FFI handle types ([#2260]) +40. Add FFI bindings for domain metadata write operations ([#2327]) + +### 🐛 Bug Fixes + +1. Treat null literal as unknown in meta-predicate evaluation ([#2097]) +2. Update TokioBackgroundExecutor to join thread instead of detaching ([#2126]) +3. Use thread pools and multi-thread tokio executor in read metadata benchmark runner ([#2044]) +4. Emit null stats for all-null columns instead of omitting them ([#2187]) +5. Allow Date/Timestamp casting for stats_parsed compatibility ([#2074]) +6. Filter evaluator input schema ([#2195]) +7. SnapshotCompleted.total_duration now includes log segment loading ([#2183]) +8. Avoid creating empty stats schemas ([#2199]) +9. Prevent dual TLS crypto backends from reqwest default features ([#2178]) +10. Vendor and pin homebrew actions ([#2243]) +11. Validate min_reader/writer_version are at least 1 ([#2202]) +12. Preserve loaded LazyCrc during incremental snapshot updates ([#2211]) +13. Detect stats_parsed in multi-part V1 checkpoints ([#2214]) +14. Downgrade per-batch data skipping log from info to debug ([#2219]) +15. Unknown table features in feature list are "supported" ([#2159]) +16. Remove debug_assert_eq before require in scan evaluator row count checks ([#2262]) +17. Adopt checkpoint written later for same-version snapshot refresh ([#2143]) +18. Return error when parquet handler returns empty data for scan files ([#2261]) +19. Refactor benchmarking workflow to not require criterion compare action ([#2264]) +20. Skip name-based validation for struct columns in expression evaluator ([#2160]) +21. Handle missing leaf columns in nested struct during parquet projection ([#2170]) +22. Pass computed ICT to CommitMetadata instead of wall-clock time ([#2319]) +23. Detect and handle empty (0-byte) log files during listing ([#2336]) + +### 📚 Documentation + +1. Update claude readme to include github actions safety note ([#2190]) +2. Add line width and comment divider style rules to CLAUDE.md ([#2277]) +3. Add documentation for current tags ([#2234]) +4. Document benchmarking in CI accuracy ([#2302]) + +### ⚡ Performance + +1. Pre-size dedup HashSet in ScanLogReplayProcessor ([#2186]) +2. Pre-size HashMap in ArrowEngineData::visit_rows ([#2185]) +3. Remove dead schema conversions in expression evaluators ([#2184]) + +### 🚜 Refactor + +1. Finalized benchmark table names and added new tables ([#2072]) +2. New transform helpers for unary and binary children ([#2150]) +3. Remove legacy row-level partition filter path ([#2158]) +4. Restructured list log files function ([#2173]) +5. Consolidate and add testing for set transaction expiration ([#2176]) +6. Rename uc-catalog and uc-client crates ([#2136]) +7. Better naming style for column mapping related functions/variables ([#2290]) +8. Centralize computation for physical schema without partition columns ([#2142]) +9. Consolidate FFI test setup helpers into ffi_test_utils ([#2307]) +10. *(action_reconciliation)* Combine getter index and field name constants ([#1717]) ([#1774]) +11. Extract shared stat helpers from RowGroupFilter ([#2324]) +12. Extract WriteContext to its own file ([#2349]) + +### ⚙️ Chores/CI + +1. Clean up arrow deps in cargo files ([#2115]) +2. Commit Cargo.lock and enforce --locked in all CI workflows ([#2240]) +3. Harden pr-title-validator a bit ([#2246]) +4. Renable semver ([#2248]) +5. Attempt fixup of semver-label job ([#2253]) +6. Use artifacts for semver label ([#2258]) +7. Remove old non-builder snapshot FFI functions ([#2318]) +8. Remove the catalog-managed feature flag ([#2310]) +9. Upgrade to arrow-58 and object_store-13, drop arrow-56 support ([#2116]) + +### Other + +[#2097]: https://github.com/delta-io/delta-kernel-rs/pull/2097 +[#2099]: https://github.com/delta-io/delta-kernel-rs/pull/2099 +[#2126]: https://github.com/delta-io/delta-kernel-rs/pull/2126 +[#2115]: https://github.com/delta-io/delta-kernel-rs/pull/2115 +[#1866]: https://github.com/delta-io/delta-kernel-rs/pull/1866 +[#2044]: https://github.com/delta-io/delta-kernel-rs/pull/2044 +[#1942]: https://github.com/delta-io/delta-kernel-rs/pull/1942 +[#2072]: https://github.com/delta-io/delta-kernel-rs/pull/2072 +[#2089]: https://github.com/delta-io/delta-kernel-rs/pull/2089 +[#2187]: https://github.com/delta-io/delta-kernel-rs/pull/2187 +[#2190]: https://github.com/delta-io/delta-kernel-rs/pull/2190 +[#1948]: https://github.com/delta-io/delta-kernel-rs/pull/1948 +[#2150]: https://github.com/delta-io/delta-kernel-rs/pull/2150 +[#2074]: https://github.com/delta-io/delta-kernel-rs/pull/2074 +[#2195]: https://github.com/delta-io/delta-kernel-rs/pull/2195 +[#2158]: https://github.com/delta-io/delta-kernel-rs/pull/2158 +[#2186]: https://github.com/delta-io/delta-kernel-rs/pull/2186 +[#2185]: https://github.com/delta-io/delta-kernel-rs/pull/2185 +[#2173]: https://github.com/delta-io/delta-kernel-rs/pull/2173 +[#2163]: https://github.com/delta-io/delta-kernel-rs/pull/2163 +[#2145]: https://github.com/delta-io/delta-kernel-rs/pull/2145 +[#2184]: https://github.com/delta-io/delta-kernel-rs/pull/2184 +[#2183]: https://github.com/delta-io/delta-kernel-rs/pull/2183 +[#2199]: https://github.com/delta-io/delta-kernel-rs/pull/2199 +[#2196]: https://github.com/delta-io/delta-kernel-rs/pull/2196 +[#2210]: https://github.com/delta-io/delta-kernel-rs/pull/2210 +[#2178]: https://github.com/delta-io/delta-kernel-rs/pull/2178 +[#2240]: https://github.com/delta-io/delta-kernel-rs/pull/2240 +[#2243]: https://github.com/delta-io/delta-kernel-rs/pull/2243 +[#2202]: https://github.com/delta-io/delta-kernel-rs/pull/2202 +[#2211]: https://github.com/delta-io/delta-kernel-rs/pull/2211 +[#2214]: https://github.com/delta-io/delta-kernel-rs/pull/2214 +[#2246]: https://github.com/delta-io/delta-kernel-rs/pull/2246 +[#2219]: https://github.com/delta-io/delta-kernel-rs/pull/2219 +[#2212]: https://github.com/delta-io/delta-kernel-rs/pull/2212 +[#2176]: https://github.com/delta-io/delta-kernel-rs/pull/2176 +[#2159]: https://github.com/delta-io/delta-kernel-rs/pull/2159 +[#2248]: https://github.com/delta-io/delta-kernel-rs/pull/2248 +[#2253]: https://github.com/delta-io/delta-kernel-rs/pull/2253 +[#2052]: https://github.com/delta-io/delta-kernel-rs/pull/2052 +[#2092]: https://github.com/delta-io/delta-kernel-rs/pull/2092 +[#2258]: https://github.com/delta-io/delta-kernel-rs/pull/2258 +[#2136]: https://github.com/delta-io/delta-kernel-rs/pull/2136 +[#2245]: https://github.com/delta-io/delta-kernel-rs/pull/2245 +[#2182]: https://github.com/delta-io/delta-kernel-rs/pull/2182 +[#2262]: https://github.com/delta-io/delta-kernel-rs/pull/2262 +[#2237]: https://github.com/delta-io/delta-kernel-rs/pull/2237 +[#2166]: https://github.com/delta-io/delta-kernel-rs/pull/2166 +[#2169]: https://github.com/delta-io/delta-kernel-rs/pull/2169 +[#2171]: https://github.com/delta-io/delta-kernel-rs/pull/2171 +[#2143]: https://github.com/delta-io/delta-kernel-rs/pull/2143 +[#2070]: https://github.com/delta-io/delta-kernel-rs/pull/2070 +[#2261]: https://github.com/delta-io/delta-kernel-rs/pull/2261 +[#2277]: https://github.com/delta-io/delta-kernel-rs/pull/2277 +[#2236]: https://github.com/delta-io/delta-kernel-rs/pull/2236 +[#2279]: https://github.com/delta-io/delta-kernel-rs/pull/2279 +[#2249]: https://github.com/delta-io/delta-kernel-rs/pull/2249 +[#2290]: https://github.com/delta-io/delta-kernel-rs/pull/2290 +[#2174]: https://github.com/delta-io/delta-kernel-rs/pull/2174 +[#2264]: https://github.com/delta-io/delta-kernel-rs/pull/2264 +[#2234]: https://github.com/delta-io/delta-kernel-rs/pull/2234 +[#2302]: https://github.com/delta-io/delta-kernel-rs/pull/2302 +[#2142]: https://github.com/delta-io/delta-kernel-rs/pull/2142 +[#2266]: https://github.com/delta-io/delta-kernel-rs/pull/2266 +[#2281]: https://github.com/delta-io/delta-kernel-rs/pull/2281 +[#2109]: https://github.com/delta-io/delta-kernel-rs/pull/2109 +[#2293]: https://github.com/delta-io/delta-kernel-rs/pull/2293 +[#2203]: https://github.com/delta-io/delta-kernel-rs/pull/2203 +[#2247]: https://github.com/delta-io/delta-kernel-rs/pull/2247 +[#2160]: https://github.com/delta-io/delta-kernel-rs/pull/2160 +[#2314]: https://github.com/delta-io/delta-kernel-rs/pull/2314 +[#2270]: https://github.com/delta-io/delta-kernel-rs/pull/2270 +[#2255]: https://github.com/delta-io/delta-kernel-rs/pull/2255 +[#2250]: https://github.com/delta-io/delta-kernel-rs/pull/2250 +[#2254]: https://github.com/delta-io/delta-kernel-rs/pull/2254 +[#2307]: https://github.com/delta-io/delta-kernel-rs/pull/2307 +[#2170]: https://github.com/delta-io/delta-kernel-rs/pull/2170 +[#2235]: https://github.com/delta-io/delta-kernel-rs/pull/2235 +[#2274]: https://github.com/delta-io/delta-kernel-rs/pull/2274 +[#1774]: https://github.com/delta-io/delta-kernel-rs/pull/1774 +[#2296]: https://github.com/delta-io/delta-kernel-rs/pull/2296 +[#2318]: https://github.com/delta-io/delta-kernel-rs/pull/2318 +[#2310]: https://github.com/delta-io/delta-kernel-rs/pull/2310 +[#2297]: https://github.com/delta-io/delta-kernel-rs/pull/2297 +[#2324]: https://github.com/delta-io/delta-kernel-rs/pull/2324 +[#2260]: https://github.com/delta-io/delta-kernel-rs/pull/2260 +[#2327]: https://github.com/delta-io/delta-kernel-rs/pull/2327 +[#2319]: https://github.com/delta-io/delta-kernel-rs/pull/2319 +[#2116]: https://github.com/delta-io/delta-kernel-rs/pull/2116 +[#2349]: https://github.com/delta-io/delta-kernel-rs/pull/2349 +[#2336]: https://github.com/delta-io/delta-kernel-rs/pull/2336 +[#2077]: https://github.com/delta-io/delta-kernel-rs/pull/2077 +[#2111]: https://github.com/delta-io/delta-kernel-rs/pull/2111 +[#2065]: https://github.com/delta-io/delta-kernel-rs/pull/2065 +[#2025]: https://github.com/delta-io/delta-kernel-rs/pull/2025 +[#1996]: https://github.com/delta-io/delta-kernel-rs/pull/1996 +[#1717]: https://github.com/delta-io/delta-kernel-rs/pull/1717 +[#1922]: https://github.com/delta-io/delta-kernel-rs/pull/1922 + ## [v0.20.0](https://github.com/delta-io/delta-kernel-rs/tree/v0.20.0/) (2026-02-26) [Full Changelog](https://github.com/delta-io/delta-kernel-rs/compare/v0.19.2...v0.20.0) @@ -56,7 +1473,7 @@ 22. Implement schema diffing for flat schemas (2/5]) ([#1478]) 23. Add API on Scan to perform 2-phase log replay ([#1547]) 24. Enable distributed log replay serde serialization for serializable scan state ([#1549]) -25. Add InCommitTimestamp support to ChangeDataFeed ([#1670]) +25. Add InCommitTimestamp support to ChangeDataFeed ([#1670]) 26. Add include_stats_columns API and output_stats_schema field ([#1728]) 27. Add write support for clustered tables behind feature flag ([#1704]) 28. Add snapshot load instrumentation ([#1750]) diff --git a/CLAUDE.md b/CLAUDE.md index 8084538afc..c2664c4dcf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,11 +8,15 @@ internals. Kernel never does I/O directly -- it defines _what_ to do via its API (`Snapshot`, `Scan`, `Transaction`) and delegates _how_ to the `Engine` trait. Current capabilities: table reads with predicates, data skipping, deletion vectors, change -data feed, checkpoints (V1 & V2), log compaction, blind append writes, table creation -(including clustered tables), and catalog-managed table support. +data feed, checkpoints (V1 & V2), log compaction (disabled, #2337), blind append writes, table creation +(including clustered tables), catalog-managed table support, and incremental scan over a +version range (`incremental_scan`). ## Build & Test Commands +> **`datafusion_executor` is a separate workspace,** so the `--workspace` commands below do NOT +> touch it. Build/test it on its own -- see `datafusion-executor/CLAUDE.md`. + ```bash # Build cargo build --workspace --all-features @@ -30,12 +34,16 @@ cargo nextest run -p delta_kernel --lib --all-features test_name_here cargo nextest run --workspace --all-features test_name_here # Format, lint, and doc check (always run after code changes) -cargo fmt \ +cargo +nightly fmt \ && cargo clippy --workspace --benches --tests --all-features -- -D warnings \ && cargo doc --workspace --all-features --no-deps +# Workspace no-default-features lint for crates that depend on kernel's Arrow APIs +cargo clippy --workspace --no-default-features --features arrow \ + --exclude delta_kernel --exclude delta_kernel_ffi --exclude delta_kernel_derive --exclude delta_kernel_ffi_macros -- -D warnings + # Quick pre-push check (mimics CI) -cargo fmt \ +cargo +nightly fmt \ && cargo clippy --workspace --benches --tests --all-features -- -D warnings \ && cargo doc --workspace --all-features --no-deps \ && cargo nextest run --workspace --all-features @@ -43,30 +51,45 @@ cargo fmt \ ### Crate Names for `-p` Flag -| Crate | Directory | Description | -|------------------------|-----------------|------------------------------------------------| -| `delta_kernel` | `kernel/` | Core library | -| `delta_kernel_ffi` | `ffi/` | C/C++ FFI bindings | -| `delta_kernel_derive` | `derive-macros/`| Proc macros | -| `acceptance` | `acceptance/` | Acceptance tests (DAT) | -| `test_utils` | `test-utils/` | Shared test utilities | -| `feature_tests` | `feature-tests/`| Feature flag tests | -| `uc-catalog` | `uc-catalog/` | Unity Catalog integration (UCCatalog, UCCommitter) | -| `uc-client` | `uc-client/` | Unity Catalog REST client | +| Crate | Directory | Description | +|--------------------------------------|---------------------------------------|--------------------------------------------------------------------------| +| `delta_kernel` | `kernel/` | Core library | +| `delta_kernel_default_engine` | `default-engine/` | Default Arrow/Tokio `Engine` implementation | +| `delta_kernel_ffi` | `ffi/` | C/C++ FFI bindings | +| `delta_kernel_derive` | `derive-macros/` | Proc macros | +| `acceptance` | `acceptance/` | Acceptance tests (DAT) | +| `test_utils` | `test-utils/` | Shared test utilities | +| `delta_kernel_workloads` | `workloads/` | Shared workload spec types + SQL predicate parser | +| `feature_tests` | `feature-tests/` | Feature flag tests | +| `delta-kernel-unity-catalog` | `delta-kernel-unity-catalog/` | Unity Catalog integration (UCCommitter, snapshot + create-table helpers) | +| `unity-catalog-delta-client-api` | `unity-catalog-delta-client-api/` | Transport-agnostic UC client traits + wire models | +| `unity-catalog-delta-rest-client` | `unity-catalog-delta-rest-client/` | REST/HTTP client for the Unity Catalog Delta Tables API | ### Feature Flags -- `default-engine` / `default-engine-rustls` / `default-engine-native-tls` -- async - Arrow/Tokio engine (pick one TLS backend) +Some noteworthy ones (see `[features]` in `kernel/Cargo.toml` for the full list): + +- TLS backend selection (`rustls` / `native-tls`) lives on the `delta_kernel_default_engine` + crate, not on kernel itself. - `arrow`, `arrow-XX`, `arrow-YY` -- Arrow version selection (kernel tracks the latest two major Arrow releases; `arrow` defaults to latest). Kernel itself does not depend on Arrow, - but default-engine does. -- `arrow-conversion`, `arrow-expression` -- Arrow interop (auto-enabled by default engine) -- `catalog-managed` -- catalog-managed table support (experimental) + but the default engine does. +- `arrow-conversion`, `arrow-expression` -- Arrow interop (auto-enabled by `default-engine-base`) +- `prettyprint` -- enables Arrow pretty-print helpers (primarily test/example oriented) - `clustered-table` -- clustered table write support (experimental) +- `adaptive-metadata-in-dev` -- adaptiveMetadata (Iceberg V4 adaptive metadata tree) support + (experimental, in development). Gates `KernelSupport::Supported` for the + `adaptiveMetadata-preview` reader+writer feature (reads/writes to tables listing it are blocked + with the cargo feature off). +- `geo-type-in-dev` -- geospatial type support (geometry and geography columns) (experimental, + in development). Gates `KernelSupport` for the `geospatial` reader+writer feature: with the + cargo feature off, any table listing it is rejected; with it on, scans and CDF are supported + but writes are still blocked. - `internal-api` -- unstable APIs like `parallel_scan_metadata`. Items are marked with the `#[internal_api]` proc macro attribute. -- `test-utils`, `integration-test` -- development only +- `declarative-plans` -- experimental declarative-plan IR (`kernel/src/plans/`) and the prost + proto wire format mirroring it (`kernel/proto/`). Auto-enables `internal-api` and `arrow`. +- `test-utils`, `integration-test` -- development only (`test-utils` enables `prettyprint`) ## Architecture at a Glance @@ -77,23 +100,40 @@ version. From it you build a `Scan` (reads) or `Transaction` (writes). `execute()` (simple), `scan_metadata()` (advanced/distributed), `parallel_scan_metadata()` (two-phase distributed log replay). -**Write path:** `Snapshot` -> `Transaction` -> `commit()`. Kernel provides `WriteContext`, -assembles commit actions, enforces protocol compliance, delegates atomic commit to a -`Committer`. +**Write path:** `Snapshot` -> `Transaction` -> `commit()`. Kernel provides `WriteContext` +(via `partitioned_write_context` or `unpartitioned_write_context`), assembles commit +actions, enforces protocol compliance, delegates atomic commit to a `Committer`. **Engine trait:** five handlers (`StorageHandler`, `JsonHandler`, `ParquetHandler`, `EvaluationHandler`, optional `MetricsReporter`). `DefaultEngine` lives in `kernel/src/engine/default/`. -**EngineData:** opaque columnar data interface. IMPORTANT: never access `EngineData` columns -directly -- always use the visitor pattern (`visit_rows` with typed `GetData` accessors). +**EngineData:** opaque columnar data interface. NEVER access `EngineData` columns +directly -- ALWAYS use the visitor pattern (`visit_rows` with typed `GetData` accessors). ## Testing - **Unit tests** test internal APIs and module internals. It is fine to use public APIs like `create_table` in a unit test as setup (e.g. to create a table for testing reads, writes, or state loading). -- **Integration tests** exercise only public APIs end-to-end. +- **Integration tests** exercise only public APIs end-to-end. See `kernel/tests/README.md` + for a catalog of available test tables (schema, protocol, features, and which tests use + them). Consult it before creating new test data to avoid duplication. +- **Consider `TestTableBuilder` (`test_utils::table_builder`) to build the table under test.** + Unlike `create_table`, which produces an empty table (just `00.json`) with a given set of + features and data layout, the builder *builds up* a multi-version table: data files written + across many commits, plus checkpoints, CRC files, a stale/missing `_last_checkpoint` hint, or + post-cleanup logs. So when a test needs a populated table history in a specific state, the + builder is often a good fit, composing `LogState`, `FeatureSet`, `DataLayoutConfig`, and + `TableConfig` through the real kernel write path so the table is protocol-correct by + construction. Load a snapshot at any `VersionTarget` with the `build_snapshot!` macro. For + coverage across many table states, the `default_sweep` cross-product template + (`LogState x FeatureSet x (DataLayoutConfig, TableConfig) x VersionTarget` -- data layout and + table config are bundled into one axis to avoid a cartesian explosion), or a per-axis template + with your own `#[values]`, can help; see + `kernel/tests/integration/cross_product/mod.rs`. Drop to lower-level setup like + `test_table_setup` (or hand-rolled `add_commit` / `LocalMockTable`) only when necessary -- + e.g. for states the builder cannot express, such as corrupt or malformed logs. - Consider how the feature interacts with Delta table features (see Protocol TLDR below). - Consider write paths: normal commits, checkpointing, CRC files, log compaction files. - Consider read paths: loading a snapshot from scratch at latest version, at a specific @@ -107,7 +147,23 @@ directly -- always use the visitor pattern (`visit_rows` with typed `GetData` ac or inputs. Prefer `#[case]` over duplicating test functions. When parameters are independent and form a cartesian product, prefer `#[values]` over enumerating every combination with `#[case]`. -- Reuse helpers from `test_utils` instead of writing custom ones when possible. +- Actively look for rstest consolidation opportunities: when writing multiple tests + that share the same setup/flow and differ only in configuration and expected + outcome, write one parameterized rstest instead of separate functions. Also check + whether a new test duplicates the flow of an existing nearby test and should be + merged into it as a new `#[case]`. A common pattern is toggling a feature (e.g. + column mapping on/off) and asserting success vs. error. +- Reuse helpers from `test_utils` and the integration-test fixtures instead of writing + custom ones when possible. See **Common test helpers** below for a curated starter list. +- **Committing in tests:** Use `txn.commit(engine)?.unwrap_committed()` to assert a + successful commit and get the `CommittedTransaction`. When you only need the resulting + snapshot, use `txn.commit(engine)?.unwrap_post_commit_snapshot()` to get the + `SnapshotRef` directly. Do NOT use `match` + `panic!` for either -- they provide a clear + error message on failure. Available under `#[cfg(test)]` and the `test-utils` feature. +- **Prefer snapshot/public API assertions over reading raw commit JSON.** Only read raw + commit JSON when the data is inaccessible via public API (e.g., system domain metadata + is blocked by `get_domain_metadata`). For commit JSON reads, use `read_actions_from_commit` + from `test_utils` -- do NOT write local helpers that duplicate this. - **`add_commit` and table setup in tests:** `add_commit` takes a `table_root` string and resolves it to an absolute object-store path. The `table_root` must be a proper URL string with a trailing slash (e.g. `"memory:///"`, `"file:///tmp/my_table/"`). Avoid using the @@ -116,9 +172,75 @@ directly -- always use the visitor pattern (`visit_rows` with typed `GetData` ac (`LocalFileSystem::new()`) with a `file:///` URL string. Do NOT use `LocalFileSystem::new_with_prefix()` with `add_commit` -- the prefix causes double-nesting because `add_commit` already resolves the full path from the URL. For in-memory tests, use - `InMemory::new()` with `"memory:///"`. Always use the same `table_root` URL string for + `InMemory::new()` with `"memory:///"`. ALWAYS use the same `table_root` URL string for both `add_commit` (writing log files) and `snapshot`/`Snapshot::try_new` (reading the - table). Always include a trailing slash in directory URLs to ensure correct path joining. + table). ALWAYS include a trailing slash in directory URLs to ensure correct path joining. + +### Common test helpers + +Before writing a custom helper, check this curated list and the locations below. +This list is non-exhaustive -- when in doubt, browse the source files directly +(`test-utils/src/lib.rs`, `kernel/tests/integration/common/`, +`kernel/tests/integration//mod.rs`). + +**Arrow construction (from `delta_kernel::arrow`)** + +- `arrow::array::new_null_array(&arrow_type, n)` -- Arrow array of `n` nulls of any Arrow + type. Prefer this over per-type `Int32Array::from(vec![None as Option])` builders. +- `engine::arrow_conversion::TryIntoArrow`: + `(&kernel_data_type).try_into_arrow()` for `DataType`, + `(&kernel_struct_type).try_into_arrow()` for `StructType` -> Arrow `Schema`. + +**Engine + table setup (from `test_utils`)** + +- `test_table_setup()` / `test_table_setup_mt()` -- engine + temp table path. Use the `_mt` + variant under `#[tokio::test(flavor = "multi_thread")]`. Required whenever a test calls + `snapshot.checkpoint()`: it issues nested `block_on` calls that deadlock on a single-threaded + runtime / `TokioBackgroundExecutor`. +- `engine_store_setup(name, opts)` -- returns `(store, engine, table_location)` when a test + needs direct object-store access. +- `setup_test_tables(...)` -- multiple pre-built tables for read/scan tests. + +**Table creation in tests** + +- `test_utils::table_builder::TestTableBuilder` (and the `test_table(...)` shorthand) -- worth + considering for a table in a specific state: composes `LogState`, `FeatureSet`, + `DataLayoutConfig`, and `TableConfig` through the real write path. Pair with the + `build_snapshot!` macro and, for broad coverage, the `default_sweep` template. See the Testing + section above. +- Prefer the kernel `create_table` builder + (`delta_kernel::transaction::create_table::create_table`) when you need a single bespoke + table rather than the builder's composed states. It exercises the same path connectors use + and auto-derives the protocol from the schema and feature flags. +- `test_utils::create_table` (a JSON helper that hand-rolls protocol + metadata) is older + but still needed when the kernel builder cannot enable a particular feature combination. + +**Schema fixtures** + +- `test_utils`: `nested_schema`, `schema_with_type`, `nested_schema_with_type`, + `multi_schema_with_type`, `top_level_ntz_schema` / `nested_ntz_schema` / + `multiple_ntz_schema`, `top_level_variant_schema` / `nested_variant_schema` / + `multiple_variant_schema`. +- `kernel/tests/integration/create_table/mod.rs`: `simple_schema`, `partition_test_schema`. + +**Commit + read helpers (from `test_utils`)** + +- `add_commit`, `add_staged_commit` -- write a JSON commit at a given version. +- `read_actions_from_commit` -- read raw JSON actions from a specific commit. Use this + instead of hand-rolled `serde_json` parsing. +- `test_read` -- full-scan read of a table; use for round-trip assertions. +- `into_record_batch` -- convert `Box` to Arrow `RecordBatch`. + +**Assertion helpers (from `test_utils`)** + +- `assert_schema_has_field(schema, &["a", "b"])` -- assert a (possibly nested) field path. +- `assert_result_error_with_message(result, "needle")` -- assert an error contains a + substring. + +**If a name here doesn't match what's in code:** the list may have drifted from a rename. +Run `rg '^pub (fn|async fn)' test-utils/src/lib.rs` to discover the current public surface, +and update this section in your PR. The same pattern works for +`kernel/tests/integration/common/write_utils.rs`. ## Protocol TLDR @@ -142,44 +264,106 @@ is the source of truth. Key concepts: **Table features**: -- Writer: `appendOnly`, `invariants`, `checkConstraints`, `generatedColumns`, - `allowColumnDefaults`, `changeDataFeed`, `identityColumns`, `rowTracking`, - `domainMetadata`, `icebergCompatV1`, `icebergCompatV2`, `clustering`, - `inCommitTimestamp` -- Reader + writer: `columnMapping`, `deletionVectors`, `timestampNtz`, - `v2Checkpoint`, `vacuumProtocolCheck`, `variantType`, `variantType-preview`, - `typeWidening` +- Writer: `allowColumnDefaults`, `appendOnly`, `changeDataFeed`, `checkConstraints`, + `clustering`, `domainMetadata`, `generatedColumns`, `icebergCompatV1`, `icebergCompatV2`, + `icebergCompatV3`, `identityColumns`, `inCommitTimestamp`, `invariants`, `rowTracking` +- Reader + writer: `adaptiveMetadata-preview`, `catalogManaged`, `catalogOwned-preview`, + `columnMapping`, `deletionVectors`, `geospatial`, `timestampNtz`, + `typeWidening`, `v2Checkpoint`, `vacuumProtocolCheck`, `variantShredding`, + `variantShredding-preview`, `variantType`, `variantType-preview` Keep this list updated when new protocol features are added to kernel. ## Common Gotchas -- **EngineData is opaque:** Never downcast to `ArrowEngineData` or any concrete type - in production code (ok in tests). Never assume one batch per file -- always iterate. -- **Column mapping:** Physical column names can differ from logical names. Always use +- **EngineData is opaque:** NEVER downcast to `ArrowEngineData` or any concrete type + in production code (ok in tests). NEVER assume one batch per file -- ALWAYS iterate. +- **Column mapping:** Physical column names can differ from logical names. ALWAYS use the schema from `Snapshot::schema()` for user data columns. Metadata/system schema column names (defined by the protocol) are not subject to column mapping. - **Transforms:** Generic recursive schema and expression transform traits and helpers are in `kernel/src/transforms/`. +- **Tracing layer callbacks must not emit tracing events directly:** Calling `warn!()` or + any tracing macro inside a `tracing_subscriber::Layer` callback (`on_event`, `on_record`, + `on_close`) while holding a span's `extensions_mut()` write lock will re-enter the layer + and deadlock on the same lock. In `on_new_span`, no extension lock is held during + `attrs.record()`, so direct `warn!()` is safe there. In `on_record`, store warnings in a + `pending_warnings: Vec` field on the visitor, take them out after the extensions + block closes, and emit via `warn!()` only then. (`on_event`'s visitor does no + warning-eligible work, so it may run under the lock directly.) See + `kernel/src/metrics/reporter.rs` for the canonical pattern. + +## Code Style + +- Line width is 100 characters. Wrap comments and string literals at 100, not 80. +- Place `use` imports at the top of the file (for non-test code) or at the top of the + `mod tests` block (for test code) -- never inside function bodies. +- Prefer `==` over `matches!` for simple single-variant enum comparisons. `matches!` is + for patterns with bindings or guards. For example: `self == Variant` not + `matches!(self, Variant)`. +- Prefer `StructField::nullable` / `StructField::not_null` over + `StructField::new(name, type, bool)` when nullability is known at compile time. + Reserve `StructField::new` for cases where nullability is a runtime value. +- Leverage `impl Into` to avoid `DataType::Struct/Array/Map(Box::new(...))` + boilerplate. `StructType`, `ArrayType`, and `MapType` all implement `Into`, + and constructors like `StructField::new`/`nullable`/`not_null`, `ArrayType::new`, and + `MapType::new` accept `impl Into`. So: + - When passing to a parameter that accepts `impl Into`, pass the container + type directly: `StructField::nullable("a", ArrayType::new(DataType::INTEGER, true))` + -- do NOT wrap in `DataType::from(...)` or `.into()` (redundant at best, and an + ambiguous-type compile error at worst). + - When a concrete `DataType` value is actually required (e.g. a `DataType`-typed + binding/field, a `[DataType]`/`Vec` element, or a `&DataType` argument), + prefer `DataType::from(ArrayType::new(...))` over + `DataType::Array(Box::new(ArrayType::new(...)))`. +- Prefer the `DeltaResultIterator<'a, T>` / `DeltaResultIteratorStatic` aliases over + hand-rolled `Box> + Send (+ 'a)>`. +- Prefer the `col!` macro and `lit(value)` constructor over `Expression::column(...)` / + `Expression::literal(...)` when building expressions inline. `col!` uses the same + compile-time segment rules as `column_name!` (string literals split on `.`; constants are + single simple segments). Use `Expression::column([...])` for runtime or non-simple names. + (`column_expr!` is a doc-hidden compatibility alias of `col!`.) +- Prefer the `schema!` / `schema_ref!` macros for inline declarative schema literals, + `lazy_schema_ref!` for `LazyLock` statics, and `try_schema!` when names of + interpolated fields might collide. For Delta log action schemas, reuse the canonical + `*_FIELD` and `LOG_*_SCHEMA` statics from `actions` instead of re-declaring + `StructField::nullable(ACTION_NAME, Action::to_schema())` or projecting from + `get_commit_schema()`. Prefer `StructType::try_new` or schema builder/patch APIs for complex + data-dependent schema manipulation. +- NEVER panic in production code -- use errors instead. Panicking + (including `unwrap()`, `expect()`, `panic!()`, `unreachable!()`, etc) is acceptable in test code only. +- Order a file so the most important APIs and impls come first; put private helper functions + toward the bottom. Within that, order by visibility: `pub` first, then `pub(crate)`, then + private. A reader scanning top to bottom should hit the public surface before the private + plumbing. (Order-sensitive items like `macro_rules!` used within the file are exempt -- they + must precede their use.) -## Code Style / Documentation +## Comment & Doc Style - MUST include doc comments for all public functions, structs, enums, and methods. - MUST document function parameters, return values, and errors. +- Doc comments focus on "what" (contract with caller) more than "how" (implementation), + unless the "how" meaningfully impacts the "what". +- Code comments state intent and explain "why" -- don't restate what the code self-documents. +- Be succinct. No verbose AI-slop comments. With well-written and well-named code, + verbose comments are worse than none. +- Say each thing once, in the right place -- don't repeat the same idea across + doc comment and inline comment. +- Comments earn their place only for hidden invariants, real-bug workarounds, or + constraints the reader can't see from the code itself. +- Don't enumerate what grep can answer. Lists like `// Used by a, b, c` rot the + moment `d` lands. Describe the shape; let the reader grep. +- No stale-prone anchors in durable docs or source comments: counts ("the 10 + variants", "5-arm match"), line numbers, or enumeration lists. Describe the + shape; let the reader grep. +- Comments MUST NOT include temporal references -- only refer to current code and + design, not past iterations. - Keep comments up-to-date with code changes. - Include examples in doc comments for complex functions only. +- Use `==` as a visual section divider in comments (e.g. `// === Helpers ===` or + `// ============`). - NEVER use emoji or unicode in comments that emulates emoji (e.g. special arrows, checkmarks). Use ASCII equivalents (`->`, `=>`, etc.) instead. -- Comments should be concise and non-repetitive -- find the right place to say it once. -- Comments should never include temporal references -- only refer to current code and - design, not past iterations. -- Doc comments focus on "what" (contract with caller) more than "how" (implementation), - unless the "how" meaningfully impacts the "what". -- Code comments state intent and explain "why" -- don't restate what the code self-documents. -- Place `use` imports at the top of the file (for non-test code) or at the top of the - `mod tests` block (for test code) -- never inside function bodies. -- NEVER panic in production code -- use errors instead. Panicking - (including `unwrap()`, `expect()`, `panic!()`, `unreachable!()`, etc) is acceptable in test code only. ## Pull Requests @@ -200,10 +384,12 @@ and data flow. Keep it concise. Read these when relevant to the task at hand: - `CLAUDE/architecture.md` -- kernel architecture: snapshot loading, read/write paths, engine trait system, EngineData, key modules, catalog-managed tables +- `docs/user-guide/CLAUDE.md` -- writing standards for the mdBook user guide - Always cross-check protocol behavior against the [Delta protocol spec](https://raw.githubusercontent.com/delta-io/delta/master/PROTOCOL.md) **Keeping docs current:** If you notice anything inaccurate in these docs -- renamed structs, traits, functions, modules, crates, APIs, stale data flows, wrong file paths -- inform the user so they can be updated. After major changes, update this file, -`CLAUDE/architecture.md`, `ffi/CLAUDE.md`, and any relevant `/CLAUDE.md` files. +`CLAUDE/architecture.md`, `ffi/CLAUDE.md`, `.github/CLAUDE.md`, and any relevant +`/CLAUDE.md` files. diff --git a/CLAUDE/architecture.md b/CLAUDE/architecture.md index 864c22b0a7..a8fb27ed7e 100644 --- a/CLAUDE/architecture.md +++ b/CLAUDE/architecture.md @@ -28,7 +28,8 @@ properties, and version number. Built via `Snapshot::builder_for(url).build(engine)` (latest version) or `.at_version(v).build(engine)` (specific version). For catalog-managed tables, -`.with_log_tail(commits)` supplies recent unpublished commits from the catalog. +`.with_log_tail(commits)` supplies recent unpublished commits from the catalog and +`.with_max_catalog_version(v)` caps the snapshot at the latest catalog-ratified version. **Snapshot loading internals:** 1. **LogSegment** (`kernel/src/log_segment/`) -- discovers commits + checkpoints for the @@ -45,12 +46,12 @@ start a `Transaction` to write data, or create a checkpoint. `Snapshot` -> `ScanBuilder` -> `Scan` -> data -The scan pipeline: log replay (build active file list) -> data skipping (prune files via stats) --> file reading -> physical-to-logical transform (partition values, column mapping, schema -evolution) -> deletion vector filtering. +The scan pipeline: log replay (build active file list) -> data skipping (prune files via stats +and partition values) -> file reading -> physical-to-logical transform (partition values, +column mapping, schema evolution) -> deletion vector filtering. **Key modules** (`kernel/src/scan/`): `log_replay.rs` (reconcile Add/Remove into active file -set), `data_skipping.rs` (rewrite predicates against min/max/nullCount stats). +set), `data_skipping.rs` (rewrite predicates against min/max/nullCount stats and partition values). **Execution paths:** - `scan.execute(engine)` -- kernel handles everything end-to-end, returns `EngineData` @@ -59,18 +60,23 @@ set), `data_skipping.rs` (rewrite predicates against min/max/nullCount stats). - `scan.parallel_scan_metadata(engine)` -- two-phase distributed log replay (`pub(crate)`, requires `internal-api` feature) +**Incremental read:** `Snapshot::incremental_scan_builder(base_version)` streams the file-action +diff over `(base_version, target_version]` -- live Adds as a `FilteredEngineData` iterator +plus a terminal summary of live Add and Remove file keys. Use this to advance a cached +file listing without re-scanning the table. + ## Write Path `Snapshot` -> `Transaction` -> commit -The kernel coordinates the write transaction: it provides the write context (target directory, -physical schema, stats columns), assembles commit actions (CommitInfo, Add files), enforces -protocol compliance (table features, schema validation), and delegates the atomic commit to a -`Committer`. +The kernel coordinates the write transaction: it provides the write context (validated partition +values, recommended write directory, physical schema, stats columns), assembles commit +actions (CommitInfo, Add files), enforces protocol compliance (table features, schema validation), +and delegates the atomic commit to a `Committer`. **Steps:** 1. Create `Transaction` from a snapshot with a `Committer` (e.g. `FileSystemCommitter`) -2. Get `WriteContext` for target dir, physical schema, and stats columns +2. Get `WriteContext` via `partitioned_write_context(values)` or `unpartitioned_write_context()` 3. Write Parquet files (via engine), collect file metadata 4. Register files via `txn.add_files(metadata)` 5. Commit: returns `CommittedTransaction`, `ConflictedTransaction`, or `RetryableTransaction` @@ -117,7 +123,11 @@ all returned batches -- the engine may split a single file across multiple batch - `kernel/src/snapshot/` -- `Snapshot`, `SnapshotBuilder`, entry point for reads/writes - `kernel/src/scan/` -- `Scan`, `ScanBuilder`, log replay, data skipping +- `kernel/src/incremental_scan/` -- `IncrementalScanBuilder`, streaming file-action diff + between two versions - `kernel/src/transaction/` -- `Transaction`, `WriteContext`, `create_table` builder +- `kernel/src/partition/` -- partition value validation, serialization, Hive-style path + encoding, URI encoding for `add.path` - `kernel/src/committer/` -- `Committer` trait, `FileSystemCommitter` - `kernel/src/log_segment/` -- log file discovery, Protocol/Metadata replay - `kernel/src/log_replay.rs` -- file-action deduplication, `LogReplayProcessor` trait @@ -126,10 +136,10 @@ all returned batches -- the engine may split a single file across multiple batch SetTransaction, DomainMetadata, Sidecar, CheckpointMetadata) - `kernel/src/schema/` -- `StructType`/`StructField`/`DataType`, projections - `kernel/src/expressions/` -- expression AST (`Expression`, `Predicate`, `Scalar`), - `column_expr!` macro + `col!` macro - `kernel/src/transforms/` -- generic recursive transforms (`ExpressionTransform`, `SchemaTransform`) -- `kernel/src/checkpoint/` -- checkpoint writing (V1 and V2 single-file classic-named) +- `kernel/src/checkpoint/` -- classic-named checkpoint writing (V1; V2 with or without sidecars) - `kernel/src/table_configuration.rs` -- table metadata, properties, feature management - `kernel/src/table_features/` -- protocol feature definitions, `TableFeature` enum - `kernel/src/table_properties.rs` -- table property parsing (delta.appendOnly, etc.) @@ -140,10 +150,10 @@ all returned batches -- the engine may split a single file across multiple batch Tables whose commits go through a catalog (e.g. Unity Catalog) instead of direct filesystem writes. Kernel doesn't know about catalogs -- the catalog client provides a log tail via -`SnapshotBuilder::with_log_tail()` and a custom `Committer` for staging/ratifying/publishing -commits. Requires `catalog-managed` feature flag. +`SnapshotBuilder::with_log_tail()`, caps the version via `with_max_catalog_version()`, and +uses a custom `Committer` for staging/ratifying/publishing commits. -The `UCCommitter` (in the `uc-catalog` crate) is the reference implementation of a catalog +The `UCCommitter` (in the `delta-kernel-unity-catalog` crate) is the reference implementation of a catalog committer for Unity Catalog. It stages commits to `_staged_commits/`, calls the UC commit API to ratify them, and publishes by copying to `_delta_log/`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e781798435..680f7088c5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -93,7 +93,7 @@ Our trunk branch is named `main`. Here's the typical workflow: # build docs cargo doc --workspace --all-features # highly recommend editor that automatically formats, but in case you need to: - cargo fmt + cargo +nightly fmt # run more tests cargo test --workspace --all-features -- --skip read_table_version_hdfs @@ -116,7 +116,7 @@ Our trunk branch is named `main`. Here's the typical workflow: #### General Tips 1. When making your first PR, please read our contributor guidelines: https://github.com/delta-incubator/delta-kernel-rs/blob/main/CONTRIBUTING.md -2. Run `cargo t --all-features --all-targets` to get started testing, and run `cargo fmt`. +2. Run `cargo t --all-features --all-targets` to get started testing, and run `cargo +nightly fmt`. 3. Ensure you have added or run the appropriate tests for your PR. 4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 5. Be sure to keep the PR description updated to reflect all changes. diff --git a/Cargo.lock b/Cargo.lock new file mode 100644 index 0000000000..9239cae0a1 --- /dev/null +++ b/Cargo.lock @@ -0,0 +1,5786 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "acceptance" +version = "0.27.1" +dependencies = [ + "datatest-stable", + "delta_kernel", + "delta_kernel_default_engine", + "delta_kernel_workloads", + "flate2", + "futures", + "itertools 0.14.0", + "serde", + "serde_json", + "sha2", + "tar", + "tempfile", + "test-case", + "test-log", + "test_utils", + "thiserror 2.0.18", + "tokio", + "tracing", + "tracing-subscriber", + "ureq", + "url", +] + +[[package]] +name = "addr2line" +version = "0.25.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b5d307320b3181d6d7954e663bd7c774a838b8220fe0593c86d9fb09f498b4b" +dependencies = [ + "gimli", +] + +[[package]] +name = "adler2" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" + +[[package]] +name = "aes" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0" +dependencies = [ + "cfg-if", + "cipher", + "cpufeatures 0.2.17", +] + +[[package]] +name = "ahash" +version = "0.8.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" +dependencies = [ + "cfg-if", + "const-random", + "getrandom 0.3.4", + "once_cell", + "version_check", + "zerocopy", +] + +[[package]] +name = "aho-corasick" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" +dependencies = [ + "memchr", +] + +[[package]] +name = "alloc-no-stdlib" +version = "2.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc7bb162ec39d46ab1ca8c77bf72e890535becd1751bb45f64c597edb4c8c6b3" + +[[package]] +name = "alloc-stdlib" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94fb8275041c72129eb51b7d0322c29b8387a0386127718b096429201a5d6ece" +dependencies = [ + "alloc-no-stdlib", +] + +[[package]] +name = "android_system_properties" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" +dependencies = [ + "libc", +] + +[[package]] +name = "anes" +version = "0.1.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b46cbb362ab8752921c97e041f5e366ee6297bd428a31275b9fcf1e380f7299" + +[[package]] +name = "anstream" +version = "0.6.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "43d5b281e737544384e969a5ccad3f1cdd24b48086a0fc1b2a5262a26b8f4f4a" +dependencies = [ + "anstyle", + "anstyle-parse 0.2.7", + "anstyle-query", + "anstyle-wincon", + "colorchoice", + "is_terminal_polyfill", + "utf8parse", +] + +[[package]] +name = "anstream" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" +dependencies = [ + "anstyle", + "anstyle-parse 1.0.0", + "anstyle-query", + "anstyle-wincon", + "colorchoice", + "is_terminal_polyfill", + "utf8parse", +] + +[[package]] +name = "anstyle" +version = "1.0.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" + +[[package]] +name = "anstyle-parse" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4e7644824f0aa2c7b9384579234ef10eb7efb6a0deb83f9630a49594dd9c15c2" +dependencies = [ + "utf8parse", +] + +[[package]] +name = "anstyle-parse" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e" +dependencies = [ + "utf8parse", +] + +[[package]] +name = "anstyle-query" +version = "1.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "anstyle-wincon" +version = "3.0.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" +dependencies = [ + "anstyle", + "once_cell_polyfill", + "windows-sys 0.61.2", +] + +[[package]] +name = "anyhow" +version = "1.0.102" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" + +[[package]] +name = "ar_archive_writer" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7eb93bbb63b9c227414f6eb3a0adfddca591a8ce1e9b60661bb08969b87e340b" +dependencies = [ + "object", +] + +[[package]] +name = "arrow" +version = "58.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d441fdda254b65f3e9025910eb2c2066b6295d9c8ed409522b8d2ace1ff8574c" +dependencies = [ + "arrow-arith 58.1.0", + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-cast 58.1.0", + "arrow-csv 58.1.0", + "arrow-data 58.1.0", + "arrow-ipc 58.1.0", + "arrow-json 58.1.0", + "arrow-ord 58.1.0", + "arrow-row 58.1.0", + "arrow-schema 58.1.0", + "arrow-select 58.1.0", + "arrow-string 58.1.0", +] + +[[package]] +name = "arrow" +version = "59.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ffaaa3e009861fd829d0a24dd6f115aa8e4634324bb092147d43baafe69ca4a7" +dependencies = [ + "arrow-arith 59.0.0", + "arrow-array 59.0.0", + "arrow-buffer 59.0.0", + "arrow-cast 59.0.0", + "arrow-csv 59.0.0", + "arrow-data 59.0.0", + "arrow-ipc 59.0.0", + "arrow-json 59.0.0", + "arrow-ord 59.0.0", + "arrow-row 59.0.0", + "arrow-schema 59.0.0", + "arrow-select 59.0.0", + "arrow-string 59.0.0", +] + +[[package]] +name = "arrow-arith" +version = "58.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ced5406f8b720cc0bc3aa9cf5758f93e8593cda5490677aa194e4b4b383f9a59" +dependencies = [ + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-schema 58.1.0", + "chrono", + "num-traits", +] + +[[package]] +name = "arrow-arith" +version = "59.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ac95125e1d71c4a252b5a9c729aef111e80418f08aaa6dbabd1ba66918247fc" +dependencies = [ + "arrow-array 59.0.0", + "arrow-buffer 59.0.0", + "arrow-data 59.0.0", + "arrow-schema 59.0.0", + "chrono", + "num-traits", +] + +[[package]] +name = "arrow-array" +version = "58.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "772bd34cacdda8baec9418d80d23d0fb4d50ef0735685bd45158b83dfeb6e62d" +dependencies = [ + "ahash", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-schema 58.1.0", + "chrono", + "chrono-tz", + "half", + "hashbrown 0.16.1", + "num-complex", + "num-integer", + "num-traits", +] + +[[package]] +name = "arrow-array" +version = "59.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c60c79628e9a97cb90d7a0dc3e944f216a902f837d4ecabc14d524bddbbc137" +dependencies = [ + "ahash", + "arrow-buffer 59.0.0", + "arrow-data 59.0.0", + "arrow-schema 59.0.0", + "chrono", + "chrono-tz", + "half", + "hashbrown 0.17.1", + "num-complex", + "num-integer", + "num-traits", +] + +[[package]] +name = "arrow-buffer" +version = "58.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "898f4cf1e9598fdb77f356fdf2134feedfd0ee8d5a4e0a5f573e7d0aec16baa4" +dependencies = [ + "bytes", + "half", + "num-bigint", + "num-traits", +] + +[[package]] +name = "arrow-buffer" +version = "59.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6026f638c400e9878c1b1cc05c3cfd46fbf381285916ab408678701c1df46c1a" +dependencies = [ + "bytes", + "half", + "num-bigint", + "num-traits", +] + +[[package]] +name = "arrow-cast" +version = "58.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b0127816c96533d20fc938729f48c52d3e48f99717e7a0b5ade77d742510736d" +dependencies = [ + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-ord 58.1.0", + "arrow-schema 58.1.0", + "arrow-select 58.1.0", + "atoi", + "base64", + "chrono", + "comfy-table", + "half", + "lexical-core", + "num-traits", + "ryu", +] + +[[package]] +name = "arrow-cast" +version = "59.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c82c236c3caf8df5664284f3f1fbe89938852163998c3fdbf37e84ac220445e9" +dependencies = [ + "arrow-array 59.0.0", + "arrow-buffer 59.0.0", + "arrow-data 59.0.0", + "arrow-ord 59.0.0", + "arrow-schema 59.0.0", + "arrow-select 59.0.0", + "atoi", + "base64", + "chrono", + "comfy-table", + "half", + "lexical-core", + "num-traits", + "ryu", +] + +[[package]] +name = "arrow-csv" +version = "58.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ca025bd0f38eeecb57c2153c0123b960494138e6a957bbda10da2b25415209fe" +dependencies = [ + "arrow-array 58.1.0", + "arrow-cast 58.1.0", + "arrow-schema 58.1.0", + "chrono", + "csv", + "csv-core", + "regex", +] + +[[package]] +name = "arrow-csv" +version = "59.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "12714e5fb7954159af1e26d4e0d37108bcf1a2ad5ee5c5bf02a944d564d588b7" +dependencies = [ + "arrow-array 59.0.0", + "arrow-cast 59.0.0", + "arrow-schema 59.0.0", + "chrono", + "csv", + "csv-core", + "regex", +] + +[[package]] +name = "arrow-data" +version = "58.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "42d10beeab2b1c3bb0b53a00f7c944a178b622173a5c7bcabc3cb45d90238df4" +dependencies = [ + "arrow-buffer 58.1.0", + "arrow-schema 58.1.0", + "half", + "num-integer", + "num-traits", +] + +[[package]] +name = "arrow-data" +version = "59.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7bd568aa70c4ec5947027b0d5caee94877433b661a0bb9e8ddceeeb5f0c9b1ab" +dependencies = [ + "arrow-buffer 59.0.0", + "arrow-schema 59.0.0", + "half", + "num-integer", + "num-traits", +] + +[[package]] +name = "arrow-ipc" +version = "58.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "609a441080e338147a84e8e6904b6da482cefb957c5cdc0f3398872f69a315d0" +dependencies = [ + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-schema 58.1.0", + "arrow-select 58.1.0", + "flatbuffers", +] + +[[package]] +name = "arrow-ipc" +version = "59.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e57ee4d470eab1a021bc4b63fa2b2c15d572892bf227b0a982d3b755a6c662b5" +dependencies = [ + "arrow-array 59.0.0", + "arrow-buffer 59.0.0", + "arrow-data 59.0.0", + "arrow-schema 59.0.0", + "arrow-select 59.0.0", + "flatbuffers", +] + +[[package]] +name = "arrow-json" +version = "58.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ead0914e4861a531be48fe05858265cf854a4880b9ed12618b1d08cba9bebc8" +dependencies = [ + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-cast 58.1.0", + "arrow-data 58.1.0", + "arrow-schema 58.1.0", + "chrono", + "half", + "indexmap", + "itoa", + "lexical-core", + "memchr", + "num-traits", + "ryu", + "serde_core", + "serde_json", + "simdutf8", +] + +[[package]] +name = "arrow-json" +version = "59.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38f47e0e7a284e1f3707a780dc8cd5451b1614e9e398ea2d9ca03c7a2fe9a9ed" +dependencies = [ + "arrow-array 59.0.0", + "arrow-buffer 59.0.0", + "arrow-cast 59.0.0", + "arrow-ord 59.0.0", + "arrow-schema 59.0.0", + "arrow-select 59.0.0", + "chrono", + "half", + "indexmap", + "itoa", + "lexical-core", + "memchr", + "num-traits", + "ryu", + "serde_core", + "serde_json", + "simdutf8", +] + +[[package]] +name = "arrow-ord" +version = "58.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "763a7ba279b20b52dad300e68cfc37c17efa65e68623169076855b3a9e941ca5" +dependencies = [ + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-schema 58.1.0", + "arrow-select 58.1.0", +] + +[[package]] +name = "arrow-ord" +version = "59.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a79cf73ad2eba8686ec2aa9bbf8671208e509025f166afc040cedbd94ffe4983" +dependencies = [ + "arrow-array 59.0.0", + "arrow-buffer 59.0.0", + "arrow-data 59.0.0", + "arrow-schema 59.0.0", + "arrow-select 59.0.0", +] + +[[package]] +name = "arrow-row" +version = "58.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e14fe367802f16d7668163ff647830258e6e0aeea9a4d79aaedf273af3bdcd3e" +dependencies = [ + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-schema 58.1.0", + "half", +] + +[[package]] +name = "arrow-row" +version = "59.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cea0f7d8ed6182f14952761e2c0f989852d5aa334fcbc49f73a9f2247c25b879" +dependencies = [ + "arrow-array 59.0.0", + "arrow-buffer 59.0.0", + "arrow-data 59.0.0", + "arrow-schema 59.0.0", + "half", +] + +[[package]] +name = "arrow-schema" +version = "58.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c30a1365d7a7dc50cc847e54154e6af49e4c4b0fddc9f607b687f29212082743" +dependencies = [ + "bitflags", +] + +[[package]] +name = "arrow-schema" +version = "59.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "80b3e786a0dd9103acd583a6fb486dbf2f3268466cc0bd571dcf34cef231c1f1" +dependencies = [ + "bitflags", +] + +[[package]] +name = "arrow-select" +version = "58.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78694888660a9e8ac949853db393af2a8b8fc82c19ce333132dfa2e72cc1a7fe" +dependencies = [ + "ahash", + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-schema 58.1.0", + "num-traits", +] + +[[package]] +name = "arrow-select" +version = "59.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "067a67e0361f6c31f4a7248759f36ca4ca71b187a941ed4d49da1c7d3d4db624" +dependencies = [ + "ahash", + "arrow-array 59.0.0", + "arrow-buffer 59.0.0", + "arrow-data 59.0.0", + "arrow-schema 59.0.0", + "num-traits", +] + +[[package]] +name = "arrow-string" +version = "58.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "61e04a01f8bb73ce54437514c5fd3ee2aa3e8abe4c777ee5cc55853b1652f79e" +dependencies = [ + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-schema 58.1.0", + "arrow-select 58.1.0", + "memchr", + "num-traits", + "regex", + "regex-syntax", +] + +[[package]] +name = "arrow-string" +version = "59.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "99bc95847f3ff62a2b03d6f8ce2e3e78f01362060549a2a311898dd442f6256d" +dependencies = [ + "arrow-array 59.0.0", + "arrow-buffer 59.0.0", + "arrow-data 59.0.0", + "arrow-schema 59.0.0", + "arrow-select 59.0.0", + "memchr", + "num-traits", + "regex", + "regex-syntax", +] + +[[package]] +name = "assert-json-diff" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47e4f2b81832e72834d7518d8487a0396a28cc408186a2e8854c0f98011faf12" +dependencies = [ + "serde", + "serde_json", +] + +[[package]] +name = "async-stream" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b5a71a6f37880a80d1d7f19efd781e4b5de42c88f0722cc13bcb6cc2cfe8476" +dependencies = [ + "async-stream-impl", + "futures-core", + "pin-project-lite", +] + +[[package]] +name = "async-stream-impl" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c7c24de15d275a1ecfd47a380fb4d5ec9bfe0933f309ed5e705b775596a3574d" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "async-trait" +version = "0.1.89" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "atoi" +version = "2.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f28d99ec8bfea296261ca1af174f24225171fea9664ba9003cbebee704810528" +dependencies = [ + "num-traits", +] + +[[package]] +name = "atomic-waker" +version = "1.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" + +[[package]] +name = "autocfg" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" + +[[package]] +name = "aws-lc-rs" +version = "1.16.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a054912289d18629dc78375ba2c3726a3afe3ff71b4edba9dedfca0e3446d1fc" +dependencies = [ + "aws-lc-sys", + "zeroize", +] + +[[package]] +name = "aws-lc-sys" +version = "0.39.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fa7e52a4c5c547c741610a2c6f123f3881e409b714cd27e6798ef020c514f0a" +dependencies = [ + "cc", + "cmake", + "dunce", + "fs_extra", +] + +[[package]] +name = "backtrace" +version = "0.3.76" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb531853791a215d7c62a30daf0dde835f381ab5de4589cfe7c649d2cbe92bd6" +dependencies = [ + "addr2line", + "cfg-if", + "libc", + "miniz_oxide", + "object", + "rustc-demangle", + "windows-link", +] + +[[package]] +name = "base64" +version = "0.22.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" + +[[package]] +name = "base64ct" +version = "1.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" + +[[package]] +name = "bit-set" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08807e080ed7f9d5433fa9b275196cfc35414f66a0c79d864dc51a0d825231a3" +dependencies = [ + "bit-vec", +] + +[[package]] +name = "bit-vec" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5e764a1d40d510daf35e07be9eb06e75770908c27d411ee6c92109c9840eaaf7" + +[[package]] +name = "bitflags" +version = "2.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "843867be96c8daad0d758b57df9392b6d8d271134fce549de6ce169ff98a92af" + +[[package]] +name = "block-buffer" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" +dependencies = [ + "generic-array", +] + +[[package]] +name = "block-padding" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a8894febbff9f758034a5b8e12d87918f56dfc64a8e1fe757d65e29041538d93" +dependencies = [ + "generic-array", +] + +[[package]] +name = "brotli" +version = "8.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4bd8b9603c7aa97359dbd97ecf258968c95f3adddd6db2f7e7a5bef101c84560" +dependencies = [ + "alloc-no-stdlib", + "alloc-stdlib", + "brotli-decompressor", +] + +[[package]] +name = "brotli-decompressor" +version = "5.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "874bb8112abecc98cbd6d81ea4fa7e94fb9449648c93cc89aa40c81c24d7de03" +dependencies = [ + "alloc-no-stdlib", + "alloc-stdlib", +] + +[[package]] +name = "bumpalo" +version = "3.20.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5d20789868f4b01b2f2caec9f5c4e0213b41e3e5702a50157d699ae31ced2fcb" + +[[package]] +name = "bytemuck" +version = "1.25.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8efb64bd706a16a1bdde310ae86b351e4d21550d98d056f22f8a7f7a2183fec" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "bytes" +version = "1.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" + +[[package]] +name = "camino" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e629a66d692cb9ff1a1c664e41771b3dcaf961985a9774c0eb0bd1b51cf60a48" + +[[package]] +name = "cast" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "37b2a672a2cb129a2e41c10b1224bb368f9f37a2b16b612598138befd7b37eb5" + +[[package]] +name = "cbc" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "26b52a9543ae338f279b96b0b9fed9c8093744685043739079ce85cd58f289a6" +dependencies = [ + "cipher", +] + +[[package]] +name = "cbindgen" +version = "0.29.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "befbfd072a8e81c02f8c507aefce431fe5e7d051f83d48a23ffc9b9fe5a11799" +dependencies = [ + "clap", + "heck", + "indexmap", + "log", + "proc-macro2", + "quote", + "serde", + "serde_json", + "syn", + "tempfile", + "toml 0.9.12+spec-1.1.0", +] + +[[package]] +name = "cc" +version = "1.2.57" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7a0dd1ca384932ff3641c8718a02769f1698e7563dc6974ffd03346116310423" +dependencies = [ + "find-msvc-tools", + "jobserver", + "libc", + "shlex", +] + +[[package]] +name = "cesu8" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6d43a04d8753f35258c91f8ec639f792891f748a1edbd759cf1dcea3382ad83c" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "cfg_aliases" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" + +[[package]] +name = "chacha20" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "rand_core 0.10.0", +] + +[[package]] +name = "checkpoint-table" +version = "0.1.0" +dependencies = [ + "bytes", + "chrono", + "clap", + "common", + "delta_kernel", + "delta_kernel_default_engine", + "env_logger", + "futures", + "tokio", + "url", +] + +[[package]] +name = "chrono" +version = "0.4.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c673075a2e0e5f4a1dde27ce9dee1ea4558c7ffe648f576438a20ca1d2acc4b0" +dependencies = [ + "iana-time-zone", + "js-sys", + "num-traits", + "serde", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "chrono-tz" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6139a8597ed92cf816dfb33f5dd6cf0bb93a6adc938f11039f371bc5bcd26c3" +dependencies = [ + "chrono", + "phf", +] + +[[package]] +name = "ciborium" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "42e69ffd6f0917f5c029256a24d0161db17cea3997d185db0d35926308770f0e" +dependencies = [ + "ciborium-io", + "ciborium-ll", + "serde", +] + +[[package]] +name = "ciborium-io" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "05afea1e0a06c9be33d539b876f1ce3692f4afea2cb41f740e7743225ed1c757" + +[[package]] +name = "ciborium-ll" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57663b653d948a338bfb3eeba9bb2fd5fcfaecb9e199e87e1eda4d9e8b240fd9" +dependencies = [ + "ciborium-io", + "half", +] + +[[package]] +name = "cipher" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" +dependencies = [ + "crypto-common", + "inout", +] + +[[package]] +name = "clap" +version = "4.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b193af5b67834b676abd72466a96c1024e6a6ad978a1f484bd90b85c94041351" +dependencies = [ + "clap_builder", + "clap_derive", +] + +[[package]] +name = "clap_builder" +version = "4.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "714a53001bf66416adb0e2ef5ac857140e7dc3a0c48fb28b2f10762fc4b5069f" +dependencies = [ + "anstream 1.0.0", + "anstyle", + "clap_lex", + "strsim", +] + +[[package]] +name = "clap_derive" +version = "4.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1110bd8a634a1ab8cb04345d8d878267d57c3cf1b38d91b71af6686408bbca6a" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "clap_lex" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" + +[[package]] +name = "cmake" +version = "0.1.57" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "75443c44cd6b379beb8c5b45d85d0773baf31cce901fe7bb252f4eff3008ef7d" +dependencies = [ + "cc", +] + +[[package]] +name = "colorchoice" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" + +[[package]] +name = "combine" +version = "4.6.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd" +dependencies = [ + "bytes", + "memchr", +] + +[[package]] +name = "comfy-table" +version = "7.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e0d05af1e006a2407bedef5af410552494ce5be9090444dbbcb57258c1af3d56" +dependencies = [ + "strum 0.26.3", + "strum_macros 0.26.4", + "unicode-width", +] + +[[package]] +name = "common" +version = "0.27.1" +dependencies = [ + "clap", + "delta_kernel", + "delta_kernel_default_engine", + "url", +] + +[[package]] +name = "const-random" +version = "0.1.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "87e00182fe74b066627d63b85fd550ac2998d4b0bd86bfed477a0ae4c7c71359" +dependencies = [ + "const-random-macro", +] + +[[package]] +name = "const-random-macro" +version = "0.1.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9d839f2a20b0aee515dc581a6172f2321f96cab76c1a38a4c584a194955390e" +dependencies = [ + "getrandom 0.2.17", + "once_cell", + "tiny-keccak", +] + +[[package]] +name = "core-foundation" +version = "0.9.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91e195e091a93c46f7102ec7818a2aa394e1e1771c3ab4825963fa03e45afb8f" +dependencies = [ + "core-foundation-sys", + "libc", +] + +[[package]] +name = "core-foundation" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b2a6cd9ae233e7f62ba4e9353e81a88df7fc8a5987b8d445b4d90c879bd156f6" +dependencies = [ + "core-foundation-sys", + "libc", +] + +[[package]] +name = "core-foundation-sys" +version = "0.8.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" + +[[package]] +name = "cpufeatures" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" +dependencies = [ + "libc", +] + +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + +[[package]] +name = "crc" +version = "3.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5eb8a2a1cd12ab0d987a5d5e825195d372001a4094a0376319d5a0ad71c1ba0d" +dependencies = [ + "crc-catalog", +] + +[[package]] +name = "crc-catalog" +version = "2.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "19d374276b40fb8bbdee95aef7c7fa6b5316ec764510eb64b8dd0e2ed0d7e7f5" + +[[package]] +name = "crc32fast" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "criterion" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2b12d017a929603d80db1831cd3a24082f8137ce19c69e6447f54f5fc8d692f" +dependencies = [ + "anes", + "cast", + "ciborium", + "clap", + "criterion-plot", + "is-terminal", + "itertools 0.10.5", + "num-traits", + "once_cell", + "oorandom", + "plotters", + "rayon", + "regex", + "serde", + "serde_derive", + "serde_json", + "tinytemplate", + "walkdir", +] + +[[package]] +name = "criterion-plot" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b50826342786a51a89e2da3a28f1c32b06e387201bc2d19791f622c673706b1" +dependencies = [ + "cast", + "itertools 0.10.5", +] + +[[package]] +name = "crossbeam-deque" +version = "0.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9dd111b7b7f7d55b72c0a6ae361660ee5853c9af73f70c3c2ef6858b950e2e51" +dependencies = [ + "crossbeam-epoch", + "crossbeam-utils", +] + +[[package]] +name = "crossbeam-epoch" +version = "0.9.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e" +dependencies = [ + "crossbeam-utils", +] + +[[package]] +name = "crossbeam-utils" +version = "0.8.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" + +[[package]] +name = "crunchy" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + +[[package]] +name = "csv" +version = "1.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52cd9d68cf7efc6ddfaaee42e7288d3a99d613d4b50f76ce9827ae0c6e14f938" +dependencies = [ + "csv-core", + "itoa", + "ryu", + "serde_core", +] + +[[package]] +name = "csv-core" +version = "0.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "704a3c26996a80471189265814dbc2c257598b96b8a7feae2d31ace646bb9782" +dependencies = [ + "memchr", +] + +[[package]] +name = "ctr" +version = "0.9.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0369ee1ad671834580515889b80f2ea915f23b8be8d0daa4bbaf2ac5c7590835" +dependencies = [ + "cipher", +] + +[[package]] +name = "datatest-stable" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a867d7322eb69cf3a68a5426387a25b45cb3b9c5ee41023ee6cea92e2afadd82" +dependencies = [ + "camino", + "fancy-regex", + "libtest-mimic", + "walkdir", +] + +[[package]] +name = "deadpool" +version = "0.12.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0be2b1d1d6ec8d846f05e137292d0b89133caf95ef33695424c09568bdd39b1b" +dependencies = [ + "deadpool-runtime", + "lazy_static", + "num_cpus", + "tokio", +] + +[[package]] +name = "deadpool-runtime" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "092966b41edc516079bdf31ec78a2e0588d1d0c08f78b91d8307215928642b2b" + +[[package]] +name = "delta-kernel-unity-catalog" +version = "0.1.0" +dependencies = [ + "delta_kernel", + "delta_kernel_default_engine", + "reqwest 0.13.2", + "rstest", + "serde_json", + "tempfile", + "test_utils", + "tokio", + "tracing", + "unity-catalog-delta-client-api", + "unity-catalog-delta-rest-client", + "url", + "uuid", +] + +[[package]] +name = "delta_kernel" +version = "0.27.1" +dependencies = [ + "arrow 58.1.0", + "arrow 59.0.0", + "async-trait", + "bytes", + "chrono", + "chrono-tz", + "crc", + "criterion", + "delta_kernel", + "delta_kernel_derive", + "futures", + "hdfs-native", + "hdfs-native-object-store", + "indexmap", + "itertools 0.14.0", + "object_store 0.13.2", + "parquet 58.1.0", + "parquet 59.0.0", + "paste", + "percent-encoding", + "prost 0.13.5", + "prost-build", + "protoc-bin-vendored", + "rand 0.9.2", + "reqwest 0.13.2", + "roaring", + "rstest", + "rstest_reuse", + "rustc_version", + "serde", + "serde_json", + "strum 0.28.0", + "tempfile", + "test-log", + "test_utils", + "thiserror 2.0.18", + "tokio", + "tracing", + "tracing-subscriber", + "url", + "uuid", + "walkdir", + "z85", +] + +[[package]] +name = "delta_kernel_benchmarks" +version = "0.27.1" +dependencies = [ + "criterion", + "delta-kernel-unity-catalog", + "delta_kernel", + "delta_kernel_default_engine", + "delta_kernel_workloads", + "flate2", + "object_store 0.12.5", + "rayon", + "serde", + "serde_json", + "sha2", + "tar", + "tempfile", + "test_utils", + "tokio", + "unity-catalog-delta-client-api", + "unity-catalog-delta-rest-client", + "ureq", + "url", +] + +[[package]] +name = "delta_kernel_default_engine" +version = "0.27.1" +dependencies = [ + "async-stream", + "async-trait", + "bytes", + "chrono", + "delta_kernel", + "delta_kernel_default_engine_test_utils", + "futures", + "hdfs-native-object-store", + "itertools 0.14.0", + "reqwest 0.13.2", + "rstest", + "serde", + "serde_json", + "tempfile", + "test_utils", + "tokio", + "tracing", + "tracing-subscriber", + "url", + "uuid", + "wiremock", +] + +[[package]] +name = "delta_kernel_default_engine_test_utils" +version = "0.27.1" +dependencies = [ + "delta_kernel", +] + +[[package]] +name = "delta_kernel_derive" +version = "0.27.1" +dependencies = [ + "proc-macro2", + "quote", + "rstest", + "syn", +] + +[[package]] +name = "delta_kernel_ffi" +version = "0.27.1" +dependencies = [ + "bytes", + "cbindgen", + "delta-kernel-unity-catalog", + "delta_kernel", + "delta_kernel_default_engine", + "delta_kernel_ffi_macros", + "itertools 0.14.0", + "libc", + "paste", + "peak_alloc", + "prost 0.13.5", + "rand 0.9.2", + "rstest", + "serde", + "serde_json", + "tempfile", + "test_utils", + "tokio", + "tracing", + "tracing-core", + "tracing-subscriber", + "trybuild", + "unity-catalog-delta-client-api", + "url", +] + +[[package]] +name = "delta_kernel_ffi_macros" +version = "0.27.1" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "delta_kernel_workloads" +version = "0.27.1" +dependencies = [ + "delta_kernel", + "itertools 0.14.0", + "rstest", + "serde", + "serde_json", + "sqlparser", + "tempfile", + "url", +] + +[[package]] +name = "der" +version = "0.7.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb" +dependencies = [ + "pem-rfc7468", + "zeroize", +] + +[[package]] +name = "des" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ffdd80ce8ce993de27e9f063a444a4d53ce8e8db4c1f00cc03af5ad5a9867a1e" +dependencies = [ + "cipher", +] + +[[package]] +name = "dhat" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "98cd11d84628e233de0ce467de10b8633f4ddaecafadefc86e13b84b8739b827" +dependencies = [ + "backtrace", + "lazy_static", + "mintex", + "parking_lot", + "rustc-hash 1.1.0", + "serde", + "serde_json", + "thousands", +] + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "block-buffer", + "crypto-common", + "subtle", +] + +[[package]] +name = "displaydoc" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "97369cbbc041bc366949bc74d34658d6cda5621039731c6310521892a3a20ae0" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "dns-lookup" +version = "3.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e39034cee21a2f5bbb66ba0e3689819c4bb5d00382a282006e802a7ffa6c41d" +dependencies = [ + "cfg-if", + "libc", + "socket2", + "windows-sys 0.60.2", +] + +[[package]] +name = "dunce" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813" + +[[package]] +name = "either" +version = "1.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719" + +[[package]] +name = "encoding_rs" +version = "0.8.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "75030f3c4f45dafd7586dd6780965a8c7e8e285a5ecb86713e63a79c5b2766f3" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "env_filter" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7a1c3cc8e57274ec99de65301228b537f1e4eedc1b8e0f9411c6caac8ae7308f" +dependencies = [ + "log", + "regex", +] + +[[package]] +name = "env_logger" +version = "0.11.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b2daee4ea451f429a58296525ddf28b45a3b64f1acf6587e2067437bb11e218d" +dependencies = [ + "anstream 0.6.21", + "anstyle", + "env_filter", + "jiff", + "log", +] + +[[package]] +name = "equivalent" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + +[[package]] +name = "escape8259" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5692dd7b5a1978a5aeb0ce83b7655c58ca8efdcb79d21036ea249da95afec2c6" + +[[package]] +name = "fancy-regex" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e24cb5a94bcae1e5408b0effca5cd7172ea3c5755049c5f3af4cd283a165298" +dependencies = [ + "bit-set", + "regex-automata", + "regex-syntax", +] + +[[package]] +name = "fastrand" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "37909eebbb50d72f9059c3b6d82c0463f2ff062c9e95845c43a6c9c0355411be" + +[[package]] +name = "feature_tests" +version = "0.27.1" +dependencies = [ + "delta_kernel", + "delta_kernel_default_engine", + "reqwest 0.13.2", + "rustls", +] + +[[package]] +name = "filetime" +version = "0.2.27" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f98844151eee8917efc50bd9e8318cb963ae8b297431495d3f758616ea5c57db" +dependencies = [ + "cfg-if", + "libc", + "libredox", +] + +[[package]] +name = "find-msvc-tools" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" + +[[package]] +name = "fixedbitset" +version = "0.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99" + +[[package]] +name = "flatbuffers" +version = "25.12.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "35f6839d7b3b98adde531effaf34f0c2badc6f4735d26fe74709d8e513a96ef3" +dependencies = [ + "bitflags", + "rustc_version", +] + +[[package]] +name = "flate2" +version = "1.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" +dependencies = [ + "crc32fast", + "miniz_oxide", + "zlib-rs", +] + +[[package]] +name = "fnv" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" + +[[package]] +name = "foldhash" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" + +[[package]] +name = "foreign-types" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f6f339eb8adc052cd2ca78910fda869aefa38d22d5cb648e6485e4d3fc06f3b1" +dependencies = [ + "foreign-types-shared", +] + +[[package]] +name = "foreign-types-shared" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "00b0228411908ca8685dba7fc2cdd70ec9990a6e753e89b6ac91a84c40fbaf4b" + +[[package]] +name = "form_urlencoded" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf" +dependencies = [ + "percent-encoding", +] + +[[package]] +name = "fs_extra" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c" + +[[package]] +name = "futures" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b147ee9d1f6d097cef9ce628cd2ee62288d963e16fb287bd9286455b241382d" +dependencies = [ + "futures-channel", + "futures-core", + "futures-executor", + "futures-io", + "futures-sink", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-channel" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "07bbe89c50d7a535e539b8c17bc0b49bdb77747034daa8087407d655f3f7cc1d" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" + +[[package]] +name = "futures-executor" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "baf29c38818342a3b26b5b923639e7b1f4a61fc5e76102d4b1981c6dc7a7579d" +dependencies = [ + "futures-core", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-io" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cecba35d7ad927e23624b22ad55235f2239cfa44fd10428eecbeba6d6a717718" + +[[package]] +name = "futures-macro" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e835b70203e41293343137df5c0664546da5745f82ec9b84d40be8336958447b" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "futures-sink" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c39754e157331b013978ec91992bde1ac089843443c49cbc7f46150b0fad0893" + +[[package]] +name = "futures-task" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" + +[[package]] +name = "futures-timer" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f288b0a4f20f9a56b5d1da57e2227c661b7b16168e2f72365f57b63326e29b24" + +[[package]] +name = "futures-util" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" +dependencies = [ + "futures-channel", + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "slab", +] + +[[package]] +name = "g2gen" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c5a7e0eb46f83a20260b850117d204366674e85d3a908d90865c78df9a6b1dfc" +dependencies = [ + "g2poly", + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "g2p" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "539e2644c030d3bf4cd208cb842d2ce2f80e82e6e8472390bcef83ceba0d80ad" +dependencies = [ + "g2gen", + "g2poly", +] + +[[package]] +name = "g2poly" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "312d2295c7302019c395cfb90dacd00a82a2eabd700429bba9c7a3f38dbbe11b" + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +dependencies = [ + "cfg-if", + "js-sys", + "libc", + "wasi", + "wasm-bindgen", +] + +[[package]] +name = "getrandom" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" +dependencies = [ + "cfg-if", + "js-sys", + "libc", + "r-efi 5.3.0", + "wasip2", + "wasm-bindgen", +] + +[[package]] +name = "getrandom" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555" +dependencies = [ + "cfg-if", + "libc", + "r-efi 6.0.0", + "rand_core 0.10.0", + "wasip2", + "wasip3", +] + +[[package]] +name = "gimli" +version = "0.32.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e629b9b98ef3dd8afe6ca2bd0f89306cec16d43d907889945bc5d6687f2f13c7" + +[[package]] +name = "glob" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280" + +[[package]] +name = "h2" +version = "0.4.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2f44da3a8150a6703ed5d34e164b875fd14c2cdab9af1252a9a1020bde2bdc54" +dependencies = [ + "atomic-waker", + "bytes", + "fnv", + "futures-core", + "futures-sink", + "http", + "indexmap", + "slab", + "tokio", + "tokio-util", + "tracing", +] + +[[package]] +name = "half" +version = "2.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b" +dependencies = [ + "cfg-if", + "crunchy", + "num-traits", + "zerocopy", +] + +[[package]] +name = "hashbrown" +version = "0.15.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" +dependencies = [ + "foldhash", +] + +[[package]] +name = "hashbrown" +version = "0.16.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" + +[[package]] +name = "hashbrown" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" + +[[package]] +name = "hdfs-native" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "51610510377a0847d53b78b53f9c6c9b7df3ffb300d1181b2e04f68bba363734" +dependencies = [ + "aes", + "base64", + "bitflags", + "bumpalo", + "bytes", + "cbc", + "chrono", + "cipher", + "crc", + "ctr", + "des", + "dns-lookup", + "futures", + "g2p", + "hex", + "hmac", + "libc", + "libloading", + "log", + "md-5", + "num-traits", + "once_cell", + "prost 0.14.3", + "prost-types 0.14.3", + "rand 0.9.2", + "regex", + "roxmltree", + "socket2", + "thiserror 2.0.18", + "tokio", + "url", + "uuid", + "which", + "whoami", +] + +[[package]] +name = "hdfs-native-object-store" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a8fd74d01f5831e0a81581e252831c70572b7a9145b813fad99323460f2f69" +dependencies = [ + "async-trait", + "bytes", + "chrono", + "futures", + "hdfs-native", + "object_store 0.13.2", + "thiserror 2.0.18", + "tokio", +] + +[[package]] +name = "heck" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" + +[[package]] +name = "hermit-abi" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c" + +[[package]] +name = "hex" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" + +[[package]] +name = "hmac" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" +dependencies = [ + "digest", +] + +[[package]] +name = "http" +version = "1.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3ba2a386d7f85a81f119ad7498ebe444d2e22c2af0b86b069416ace48b3311a" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "http-body" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1efedce1fb8e6913f23e0c92de8e62cd5b772a67e7b3946df930a62566c93184" +dependencies = [ + "bytes", + "http", +] + +[[package]] +name = "http-body-util" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b021d93e26becf5dc7e1b75b1bed1fd93124b374ceb73f43d4d4eafec896a64a" +dependencies = [ + "bytes", + "futures-core", + "http", + "http-body", + "pin-project-lite", +] + +[[package]] +name = "httparse" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" + +[[package]] +name = "httpdate" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9" + +[[package]] +name = "humantime" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "135b12329e5e3ce057a9f972339ea52bc954fe1e9358ef27f95e89716fbc5424" + +[[package]] +name = "hyper" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2ab2d4f250c3d7b1c9fcdff1cece94ea4e2dfbec68614f7b87cb205f24ca9d11" +dependencies = [ + "atomic-waker", + "bytes", + "futures-channel", + "futures-core", + "h2", + "http", + "http-body", + "httparse", + "httpdate", + "itoa", + "pin-project-lite", + "pin-utils", + "smallvec", + "tokio", + "want", +] + +[[package]] +name = "hyper-rustls" +version = "0.27.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3c93eb611681b207e1fe55d5a71ecf91572ec8a6705cdb6857f7d8d5242cf58" +dependencies = [ + "http", + "hyper", + "hyper-util", + "rustls", + "rustls-native-certs", + "rustls-pki-types", + "tokio", + "tokio-rustls", + "tower-service", +] + +[[package]] +name = "hyper-tls" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "70206fc6890eaca9fde8a0bf71caa2ddfc9fe045ac9e5c70df101a7dbde866e0" +dependencies = [ + "bytes", + "http-body-util", + "hyper", + "hyper-util", + "native-tls", + "tokio", + "tokio-native-tls", + "tower-service", +] + +[[package]] +name = "hyper-util" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" +dependencies = [ + "base64", + "bytes", + "futures-channel", + "futures-util", + "http", + "http-body", + "hyper", + "ipnet", + "libc", + "percent-encoding", + "pin-project-lite", + "socket2", + "system-configuration", + "tokio", + "tower-service", + "tracing", + "windows-registry", +] + +[[package]] +name = "iana-time-zone" +version = "0.1.65" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" +dependencies = [ + "android_system_properties", + "core-foundation-sys", + "iana-time-zone-haiku", + "js-sys", + "log", + "wasm-bindgen", + "windows-core", +] + +[[package]] +name = "iana-time-zone-haiku" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" +dependencies = [ + "cc", +] + +[[package]] +name = "icu_collections" +version = "2.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4c6b649701667bbe825c3b7e6388cb521c23d88644678e83c0c4d0a621a34b43" +dependencies = [ + "displaydoc", + "potential_utf", + "yoke", + "zerofrom", + "zerovec", +] + +[[package]] +name = "icu_locale_core" +version = "2.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "edba7861004dd3714265b4db54a3c390e880ab658fec5f7db895fae2046b5bb6" +dependencies = [ + "displaydoc", + "litemap", + "tinystr", + "writeable", + "zerovec", +] + +[[package]] +name = "icu_normalizer" +version = "2.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5f6c8828b67bf8908d82127b2054ea1b4427ff0230ee9141c54251934ab1b599" +dependencies = [ + "icu_collections", + "icu_normalizer_data", + "icu_properties", + "icu_provider", + "smallvec", + "zerovec", +] + +[[package]] +name = "icu_normalizer_data" +version = "2.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7aedcccd01fc5fe81e6b489c15b247b8b0690feb23304303a9e560f37efc560a" + +[[package]] +name = "icu_properties" +version = "2.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "020bfc02fe870ec3a66d93e677ccca0562506e5872c650f893269e08615d74ec" +dependencies = [ + "icu_collections", + "icu_locale_core", + "icu_properties_data", + "icu_provider", + "zerotrie", + "zerovec", +] + +[[package]] +name = "icu_properties_data" +version = "2.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "616c294cf8d725c6afcd8f55abc17c56464ef6211f9ed59cccffe534129c77af" + +[[package]] +name = "icu_provider" +version = "2.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85962cf0ce02e1e0a629cc34e7ca3e373ce20dda4c4d7294bbd0bf1fdb59e614" +dependencies = [ + "displaydoc", + "icu_locale_core", + "writeable", + "yoke", + "zerofrom", + "zerotrie", + "zerovec", +] + +[[package]] +name = "id-arena" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954" + +[[package]] +name = "idna" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de" +dependencies = [ + "idna_adapter", + "smallvec", + "utf8_iter", +] + +[[package]] +name = "idna_adapter" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3acae9609540aa318d1bc588455225fb2085b9ed0c4f6bd0d9d5bcd86f1a0344" +dependencies = [ + "icu_normalizer", + "icu_properties", +] + +[[package]] +name = "indexmap" +version = "2.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7714e70437a7dc3ac8eb7e6f8df75fd8eb422675fc7678aff7364301092b1017" +dependencies = [ + "equivalent", + "hashbrown 0.16.1", + "serde", + "serde_core", +] + +[[package]] +name = "inout" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01" +dependencies = [ + "block-padding", + "generic-array", +] + +[[package]] +name = "inspect-table" +version = "0.1.0" +dependencies = [ + "chrono", + "clap", + "common", + "delta_kernel", + "delta_kernel_default_engine", + "env_logger", + "tracing", + "tracing-subscriber", +] + +[[package]] +name = "integer-encoding" +version = "3.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8bb03732005da905c88227371639bf1ad885cc712789c011c31c5fb3ab3ccf02" + +[[package]] +name = "ipnet" +version = "2.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2" + +[[package]] +name = "iri-string" +version = "0.7.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c91338f0783edbd6195decb37bae672fd3b165faffb89bf7b9e6942f8b1a731a" +dependencies = [ + "memchr", + "serde", +] + +[[package]] +name = "is-terminal" +version = "0.4.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3640c1c38b8e4e43584d8df18be5fc6b0aa314ce6ebf51b53313d4306cca8e46" +dependencies = [ + "hermit-abi", + "libc", + "windows-sys 0.61.2", +] + +[[package]] +name = "is_terminal_polyfill" +version = "1.70.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" + +[[package]] +name = "itertools" +version = "0.10.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b0fd2260e829bddf4cb6ea802289de2f86d6a7a690192fbe91b3f46e0f2c8473" +dependencies = [ + "either", +] + +[[package]] +name = "itertools" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" +dependencies = [ + "either", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "jiff" +version = "0.2.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a3546dc96b6d42c5f24902af9e2538e82e39ad350b0c766eb3fbf2d8f3d8359" +dependencies = [ + "jiff-static", + "log", + "portable-atomic", + "portable-atomic-util", + "serde_core", +] + +[[package]] +name = "jiff-static" +version = "0.2.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2a8c8b344124222efd714b73bb41f8b5120b27a7cc1c75593a6ff768d9d05aa4" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "jni" +version = "0.21.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a87aa2bb7d2af34197c04845522473242e1aa17c12f4935d5856491a7fb8c97" +dependencies = [ + "cesu8", + "cfg-if", + "combine", + "jni-sys", + "log", + "thiserror 1.0.69", + "walkdir", + "windows-sys 0.45.0", +] + +[[package]] +name = "jni-sys" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8eaf4bc02d17cbdd7ff4c7438cafcdf7fb9a4613313ad11b4f8fefe7d3fa0130" + +[[package]] +name = "jobserver" +version = "0.1.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33" +dependencies = [ + "getrandom 0.3.4", + "libc", +] + +[[package]] +name = "js-sys" +version = "0.3.91" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b49715b7073f385ba4bc528e5747d02e66cb39c6146efb66b781f131f0fb399c" +dependencies = [ + "once_cell", + "wasm-bindgen", +] + +[[package]] +name = "lazy_static" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" + +[[package]] +name = "leb128fmt" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" + +[[package]] +name = "lexical-core" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d8d125a277f807e55a77304455eb7b1cb52f2b18c143b60e766c120bd64a594" +dependencies = [ + "lexical-parse-float", + "lexical-parse-integer", + "lexical-util", + "lexical-write-float", + "lexical-write-integer", +] + +[[package]] +name = "lexical-parse-float" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52a9f232fbd6f550bc0137dcb5f99ab674071ac2d690ac69704593cb4abbea56" +dependencies = [ + "lexical-parse-integer", + "lexical-util", +] + +[[package]] +name = "lexical-parse-integer" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a7a039f8fb9c19c996cd7b2fcce303c1b2874fe1aca544edc85c4a5f8489b34" +dependencies = [ + "lexical-util", +] + +[[package]] +name = "lexical-util" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2604dd126bb14f13fb5d1bd6a66155079cb9fa655b37f875b3a742c705dbed17" + +[[package]] +name = "lexical-write-float" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "50c438c87c013188d415fbabbb1dceb44249ab81664efbd31b14ae55dabb6361" +dependencies = [ + "lexical-util", + "lexical-write-integer", +] + +[[package]] +name = "lexical-write-integer" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "409851a618475d2d5796377cad353802345cba92c867d9fbcde9cf4eac4e14df" +dependencies = [ + "lexical-util", +] + +[[package]] +name = "libc" +version = "0.2.183" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b5b646652bf6661599e1da8901b3b9522896f01e736bad5f723fe7a3a27f899d" + +[[package]] +name = "libloading" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "754ca22de805bb5744484a5b151a9e1a8e837d5dc232c2d7d8c2e3492edc8b60" +dependencies = [ + "cfg-if", + "windows-link", +] + +[[package]] +name = "libm" +version = "0.2.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" + +[[package]] +name = "libredox" +version = "0.1.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1744e39d1d6a9948f4f388969627434e31128196de472883b39f148769bfe30a" +dependencies = [ + "bitflags", + "libc", + "plain", + "redox_syscall 0.7.3", +] + +[[package]] +name = "libtest-mimic" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "14e6ba06f0ade6e504aff834d7c34298e5155c6baca353cc6a4aaff2f9fd7f33" +dependencies = [ + "anstream 1.0.0", + "anstyle", + "clap", + "escape8259", +] + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "litemap" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6373607a59f0be73a39b6fe456b8192fcc3585f602af20751600e974dd455e77" + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "log" +version = "0.4.29" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897" + +[[package]] +name = "lru-slab" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154" + +[[package]] +name = "lz4_flex" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "db9a0d582c2874f68138a16ce1867e0ffde6c0bb0a0df85e1f36d04146db488a" +dependencies = [ + "twox-hash", +] + +[[package]] +name = "matchers" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d1525a2a28c7f4fa0fc98bb91ae755d1e2d1505079e05539e35bc876b5d65ae9" +dependencies = [ + "regex-automata", +] + +[[package]] +name = "md-5" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf" +dependencies = [ + "cfg-if", + "digest", +] + +[[package]] +name = "mem-test" +version = "0.27.1" +dependencies = [ + "delta_kernel", + "delta_kernel_default_engine", + "dhat", + "rayon", + "serde_json", + "tempfile", + "tracing", + "url", +] + +[[package]] +name = "memchr" +version = "2.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79" + +[[package]] +name = "mime" +version = "0.3.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a" + +[[package]] +name = "miniz_oxide" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" +dependencies = [ + "adler2", + "simd-adler32", +] + +[[package]] +name = "mintex" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c505b3e17ed6b70a7ed2e67fbb2c560ee327353556120d6e72f5232b6880d536" + +[[package]] +name = "mio" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a69bcab0ad47271a0234d9422b131806bf3968021e5dc9328caf2d4cd58557fc" +dependencies = [ + "libc", + "wasi", + "windows-sys 0.61.2", +] + +[[package]] +name = "multimap" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d87ecb2933e8aeadb3e3a02b828fed80a7528047e68b4f424523a0981a3a084" + +[[package]] +name = "native-tls" +version = "0.2.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "465500e14ea162429d264d44189adc38b199b62b1c21eea9f69e4b73cb03bbf2" +dependencies = [ + "libc", + "log", + "openssl", + "openssl-probe", + "openssl-sys", + "schannel", + "security-framework", + "security-framework-sys", + "tempfile", +] + +[[package]] +name = "nu-ansi-term" +version = "0.50.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "num-bigint" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a5e44f723f1133c9deac646763579fdb3ac745e418f2a7af9cd0c431da1f20b9" +dependencies = [ + "num-integer", + "num-traits", +] + +[[package]] +name = "num-complex" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-integer" +version = "0.1.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", + "libm", +] + +[[package]] +name = "num_cpus" +version = "1.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91df4bbde75afed763b708b7eee1e8e7651e02d97f6d5dd763e89367e957b23b" +dependencies = [ + "hermit-abi", + "libc", +] + +[[package]] +name = "object" +version = "0.37.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff76201f031d8863c38aa7f905eca4f53abbfa15f609db4277d44cd8938f33fe" +dependencies = [ + "memchr", +] + +[[package]] +name = "object_store" +version = "0.12.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fbfbfff40aeccab00ec8a910b57ca8ecf4319b335c542f2edcd19dd25a1e2a00" +dependencies = [ + "async-trait", + "base64", + "bytes", + "chrono", + "form_urlencoded", + "futures", + "http", + "http-body-util", + "humantime", + "hyper", + "itertools 0.14.0", + "md-5", + "parking_lot", + "percent-encoding", + "quick-xml 0.38.4", + "rand 0.9.2", + "reqwest 0.12.28", + "ring", + "serde", + "serde_json", + "serde_urlencoded", + "thiserror 2.0.18", + "tokio", + "tracing", + "url", + "walkdir", + "wasm-bindgen-futures", + "web-time", +] + +[[package]] +name = "object_store" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "622acbc9100d3c10e2ee15804b0caa40e55c933d5aa53814cd520805b7958a49" +dependencies = [ + "async-trait", + "base64", + "bytes", + "chrono", + "form_urlencoded", + "futures-channel", + "futures-core", + "futures-util", + "http", + "http-body-util", + "httparse", + "humantime", + "hyper", + "itertools 0.14.0", + "md-5", + "parking_lot", + "percent-encoding", + "quick-xml 0.39.2", + "rand 0.10.0", + "reqwest 0.12.28", + "ring", + "rustls-pki-types", + "serde", + "serde_json", + "serde_urlencoded", + "thiserror 2.0.18", + "tokio", + "tracing", + "url", + "walkdir", + "wasm-bindgen-futures", + "web-time", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "once_cell_polyfill" +version = "1.70.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" + +[[package]] +name = "oorandom" +version = "11.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6790f58c7ff633d8771f42965289203411a5e5c68388703c06e14f24770b41e" + +[[package]] +name = "openssl" +version = "0.10.79" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf0b434746ee2832f4f0baf10137e1cabb18cbe6912c69e2e33263c45250f542" +dependencies = [ + "bitflags", + "cfg-if", + "foreign-types", + "libc", + "openssl-macros", + "openssl-sys", +] + +[[package]] +name = "openssl-macros" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a948666b637a0f465e8564c73e89d4dde00d72d4d473cc972f390fc3dcee7d9c" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "openssl-probe" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe" + +[[package]] +name = "openssl-sys" +version = "0.9.115" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "158fe5b292746440aa6e7a7e690e55aeb72d41505e2804c23c6973ad0e9c9781" +dependencies = [ + "cc", + "libc", + "pkg-config", + "vcpkg", +] + +[[package]] +name = "ordered-float" +version = "2.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68f19d67e5a2795c94e73e0bb1cc1a7edeb2e28efd39e2e1c9b7a40c1108b11c" +dependencies = [ + "num-traits", +] + +[[package]] +name = "parking_lot" +version = "0.12.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a" +dependencies = [ + "lock_api", + "parking_lot_core", +] + +[[package]] +name = "parking_lot_core" +version = "0.9.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" +dependencies = [ + "cfg-if", + "libc", + "redox_syscall 0.5.18", + "smallvec", + "windows-link", +] + +[[package]] +name = "parquet" +version = "58.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d3f9f2205199603564127932b89695f52b62322f541d0fc7179d57c2e1c9877" +dependencies = [ + "ahash", + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-ipc 58.1.0", + "arrow-schema 58.1.0", + "arrow-select 58.1.0", + "base64", + "brotli", + "bytes", + "chrono", + "flate2", + "futures", + "half", + "hashbrown 0.16.1", + "lz4_flex", + "num-bigint", + "num-integer", + "num-traits", + "object_store 0.13.2", + "paste", + "seq-macro", + "simdutf8", + "snap", + "thrift", + "tokio", + "twox-hash", + "zstd", +] + +[[package]] +name = "parquet" +version = "59.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "970dff83e97d953c827ae8176f6bf4e9f77bf62daacc01ec5df348ec5eacd913" +dependencies = [ + "ahash", + "arrow-array 59.0.0", + "arrow-buffer 59.0.0", + "arrow-data 59.0.0", + "arrow-ipc 59.0.0", + "arrow-schema 59.0.0", + "arrow-select 59.0.0", + "base64", + "brotli", + "bytes", + "chrono", + "flate2", + "futures", + "half", + "hashbrown 0.17.1", + "lz4_flex", + "num-bigint", + "num-integer", + "num-traits", + "object_store 0.13.2", + "paste", + "seq-macro", + "simdutf8", + "snap", + "tokio", + "twox-hash", + "zstd", +] + +[[package]] +name = "paste" +version = "1.0.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" + +[[package]] +name = "peak_alloc" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccc90935a8dd139fdf341762773687a1e361d3f54b396a55d9dd1f7001e484bb" + +[[package]] +name = "pem-rfc7468" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88b39c9bfcfc231068454382784bb460aae594343fb030d46e9f50a645418412" +dependencies = [ + "base64ct", +] + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "petgraph" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3672b37090dbd86368a4145bc067582552b29c27377cad4e0a306c97f9bd7772" +dependencies = [ + "fixedbitset", + "indexmap", +] + +[[package]] +name = "phf" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "913273894cec178f401a31ec4b656318d95473527be05c0752cc41cdc32be8b7" +dependencies = [ + "phf_shared", +] + +[[package]] +name = "phf_shared" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06005508882fb681fd97892ecff4b7fd0fee13ef1aa569f8695dae7ab9099981" +dependencies = [ + "siphasher", +] + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "pin-utils" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184" + +[[package]] +name = "pkg-config" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7edddbd0b52d732b21ad9a5fab5c704c14cd949e5e9a1ec5929a24fded1b904c" + +[[package]] +name = "plain" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4596b6d070b27117e987119b4dac604f3c58cfb0b191112e24771b2faeac1a6" + +[[package]] +name = "plotters" +version = "0.3.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5aeb6f403d7a4911efb1e33402027fc44f29b5bf6def3effcc22d7bb75f2b747" +dependencies = [ + "num-traits", + "plotters-backend", + "plotters-svg", + "wasm-bindgen", + "web-sys", +] + +[[package]] +name = "plotters-backend" +version = "0.3.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df42e13c12958a16b3f7f4386b9ab1f3e7933914ecea48da7139435263a4172a" + +[[package]] +name = "plotters-svg" +version = "0.3.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "51bae2ac328883f7acdfea3d66a7c35751187f870bc81f94563733a154d7a670" +dependencies = [ + "plotters-backend", +] + +[[package]] +name = "portable-atomic" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" + +[[package]] +name = "portable-atomic-util" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "091397be61a01d4be58e7841595bd4bfedb15f1cd54977d79b8271e94ed799a3" +dependencies = [ + "portable-atomic", +] + +[[package]] +name = "potential_utf" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b73949432f5e2a09657003c25bca5e19a0e9c84f8058ca374f49e0ebe605af77" +dependencies = [ + "zerovec", +] + +[[package]] +name = "ppv-lite86" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" +dependencies = [ + "zerocopy", +] + +[[package]] +name = "prettyplease" +version = "0.2.37" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" +dependencies = [ + "proc-macro2", + "syn", +] + +[[package]] +name = "proc-macro-crate" +version = "3.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e67ba7e9b2b56446f1d419b1d807906278ffa1a658a8a5d8a39dcb1f5a78614f" +dependencies = [ + "toml_edit", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "prost" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2796faa41db3ec313a31f7624d9286acf277b52de526150b7e69f3debf891ee5" +dependencies = [ + "bytes", + "prost-derive 0.13.5", +] + +[[package]] +name = "prost" +version = "0.14.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2ea70524a2f82d518bce41317d0fae74151505651af45faf1ffbd6fd33f0568" +dependencies = [ + "bytes", + "prost-derive 0.14.3", +] + +[[package]] +name = "prost-build" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf" +dependencies = [ + "heck", + "itertools 0.14.0", + "log", + "multimap", + "once_cell", + "petgraph", + "prettyplease", + "prost 0.13.5", + "prost-types 0.13.5", + "regex", + "syn", + "tempfile", +] + +[[package]] +name = "prost-derive" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d" +dependencies = [ + "anyhow", + "itertools 0.14.0", + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "prost-derive" +version = "0.14.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "27c6023962132f4b30eb4c172c91ce92d933da334c59c23cddee82358ddafb0b" +dependencies = [ + "anyhow", + "itertools 0.14.0", + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "prost-types" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52c2c1bf36ddb1a1c396b3601a3cec27c2462e45f07c386894ec3ccf5332bd16" +dependencies = [ + "prost 0.13.5", +] + +[[package]] +name = "prost-types" +version = "0.14.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8991c4cbdb8bc5b11f0b074ffe286c30e523de90fee5ba8132f1399f23cb3dd7" +dependencies = [ + "prost 0.14.3", +] + +[[package]] +name = "protoc-bin-vendored" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d1c381df33c98266b5f08186583660090a4ffa0889e76c7e9a5e175f645a67fa" +dependencies = [ + "protoc-bin-vendored-linux-aarch_64", + "protoc-bin-vendored-linux-ppcle_64", + "protoc-bin-vendored-linux-s390_64", + "protoc-bin-vendored-linux-x86_32", + "protoc-bin-vendored-linux-x86_64", + "protoc-bin-vendored-macos-aarch_64", + "protoc-bin-vendored-macos-x86_64", + "protoc-bin-vendored-win32", +] + +[[package]] +name = "protoc-bin-vendored-linux-aarch_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c350df4d49b5b9e3ca79f7e646fde2377b199e13cfa87320308397e1f37e1a4c" + +[[package]] +name = "protoc-bin-vendored-linux-ppcle_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a55a63e6c7244f19b5c6393f025017eb5d793fd5467823a099740a7a4222440c" + +[[package]] +name = "protoc-bin-vendored-linux-s390_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1dba5565db4288e935d5330a07c264a4ee8e4a5b4a4e6f4e83fad824cc32f3b0" + +[[package]] +name = "protoc-bin-vendored-linux-x86_32" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8854774b24ee28b7868cd71dccaae8e02a2365e67a4a87a6cd11ee6cdbdf9cf5" + +[[package]] +name = "protoc-bin-vendored-linux-x86_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b38b07546580df720fa464ce124c4b03630a6fb83e05c336fea2a241df7e5d78" + +[[package]] +name = "protoc-bin-vendored-macos-aarch_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "89278a9926ce312e51f1d999fee8825d324d603213344a9a706daa009f1d8092" + +[[package]] +name = "protoc-bin-vendored-macos-x86_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "81745feda7ccfb9471d7a4de888f0652e806d5795b61480605d4943176299756" + +[[package]] +name = "protoc-bin-vendored-win32" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "95067976aca6421a523e491fce939a3e65249bac4b977adee0ee9771568e8aa3" + +[[package]] +name = "psm" +version = "0.1.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3852766467df634d74f0b2d7819bf8dc483a0eb2e3b0f50f756f9cfe8b0d18d8" +dependencies = [ + "ar_archive_writer", + "cc", +] + +[[package]] +name = "quick-xml" +version = "0.38.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b66c2058c55a409d601666cffe35f04333cf1013010882cec174a7467cd4e21c" +dependencies = [ + "memchr", + "serde", +] + +[[package]] +name = "quick-xml" +version = "0.39.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "958f21e8e7ceb5a1aa7fa87fab28e7c75976e0bfe7e23ff069e0a260f894067d" +dependencies = [ + "memchr", + "serde", +] + +[[package]] +name = "quinn" +version = "0.11.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20" +dependencies = [ + "bytes", + "cfg_aliases", + "pin-project-lite", + "quinn-proto", + "quinn-udp", + "rustc-hash 2.1.1", + "rustls", + "socket2", + "thiserror 2.0.18", + "tokio", + "tracing", + "web-time", +] + +[[package]] +name = "quinn-proto" +version = "0.11.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098" +dependencies = [ + "aws-lc-rs", + "bytes", + "getrandom 0.3.4", + "lru-slab", + "rand 0.9.2", + "ring", + "rustc-hash 2.1.1", + "rustls", + "rustls-pki-types", + "slab", + "thiserror 2.0.18", + "tinyvec", + "tracing", + "web-time", +] + +[[package]] +name = "quinn-udp" +version = "0.5.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "addec6a0dcad8a8d96a771f815f0eaf55f9d1805756410b39f5fa81332574cbd" +dependencies = [ + "cfg_aliases", + "libc", + "once_cell", + "socket2", + "tracing", + "windows-sys 0.60.2", +] + +[[package]] +name = "quote" +version = "1.0.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "rand" +version = "0.8.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "34af8d1a0e25924bc5b7c43c079c942339d8f0a8b57c39049bef581b46327404" +dependencies = [ + "libc", + "rand_chacha 0.3.1", + "rand_core 0.6.4", +] + +[[package]] +name = "rand" +version = "0.9.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6db2770f06117d490610c7488547d543617b21bfa07796d7a12f6f1bd53850d1" +dependencies = [ + "rand_chacha 0.9.0", + "rand_core 0.9.5", +] + +[[package]] +name = "rand" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bc266eb313df6c5c09c1c7b1fbe2510961e5bcd3add930c1e31f7ed9da0feff8" +dependencies = [ + "chacha20", + "getrandom 0.4.2", + "rand_core 0.10.0", +] + +[[package]] +name = "rand_chacha" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" +dependencies = [ + "ppv-lite86", + "rand_core 0.6.4", +] + +[[package]] +name = "rand_chacha" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" +dependencies = [ + "ppv-lite86", + "rand_core 0.9.5", +] + +[[package]] +name = "rand_core" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" +dependencies = [ + "getrandom 0.2.17", +] + +[[package]] +name = "rand_core" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" +dependencies = [ + "getrandom 0.3.4", +] + +[[package]] +name = "rand_core" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c8d0fd677905edcbeedbf2edb6494d676f0e98d54d5cf9bda0b061cb8fb8aba" + +[[package]] +name = "rayon" +version = "1.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "368f01d005bf8fd9b1206fb6fa653e6c4a81ceb1466406b81792d87c5677a58f" +dependencies = [ + "either", + "rayon-core", +] + +[[package]] +name = "rayon-core" +version = "1.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "22e18b0f0062d30d4230b2e85ff77fdfe4326feb054b9783a3460d8435c8ab91" +dependencies = [ + "crossbeam-deque", + "crossbeam-utils", +] + +[[package]] +name = "read-table-changes" +version = "0.1.0" +dependencies = [ + "clap", + "common", + "delta_kernel", + "delta_kernel_default_engine", + "itertools 0.14.0", +] + +[[package]] +name = "read-table-multi-threaded" +version = "0.1.0" +dependencies = [ + "clap", + "common", + "delta_kernel", + "delta_kernel_default_engine", + "env_logger", + "spmc", + "url", +] + +[[package]] +name = "read-table-single-threaded" +version = "0.1.0" +dependencies = [ + "clap", + "common", + "delta_kernel", + "delta_kernel_default_engine", + "env_logger", + "itertools 0.14.0", +] + +[[package]] +name = "recursive" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0786a43debb760f491b1bc0269fe5e84155353c67482b9e60d0cfb596054b43e" +dependencies = [ + "recursive-proc-macro-impl", + "stacker", +] + +[[package]] +name = "recursive-proc-macro-impl" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76009fbe0614077fc1a2ce255e3a1881a2e3a3527097d5dc6d8212c585e7e38b" +dependencies = [ + "quote", + "syn", +] + +[[package]] +name = "redox_syscall" +version = "0.5.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" +dependencies = [ + "bitflags", +] + +[[package]] +name = "redox_syscall" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce70a74e890531977d37e532c34d45e9055d2409ed08ddba14529471ed0be16" +dependencies = [ + "bitflags", +] + +[[package]] +name = "regex" +version = "1.12.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e10754a14b9137dd7b1e3e5b0493cc9171fdd105e0ab477f51b72e7f3ac0e276" +dependencies = [ + "aho-corasick", + "memchr", + "regex-automata", + "regex-syntax", +] + +[[package]] +name = "regex-automata" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f" +dependencies = [ + "aho-corasick", + "memchr", + "regex-syntax", +] + +[[package]] +name = "regex-syntax" +version = "0.8.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a" + +[[package]] +name = "relative-path" +version = "1.9.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba39f3699c378cd8970968dcbff9c43159ea4cfbd88d43c00b22f2ef10a435d2" + +[[package]] +name = "reqwest" +version = "0.12.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147" +dependencies = [ + "base64", + "bytes", + "futures-core", + "futures-util", + "h2", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-rustls", + "hyper-util", + "js-sys", + "log", + "percent-encoding", + "pin-project-lite", + "quinn", + "rustls", + "rustls-native-certs", + "rustls-pki-types", + "serde", + "serde_json", + "serde_urlencoded", + "sync_wrapper", + "tokio", + "tokio-rustls", + "tokio-util", + "tower", + "tower-http", + "tower-service", + "url", + "wasm-bindgen", + "wasm-bindgen-futures", + "wasm-streams", + "web-sys", +] + +[[package]] +name = "reqwest" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ab3f43e3283ab1488b624b44b0e988d0acea0b3214e694730a055cb6b2efa801" +dependencies = [ + "base64", + "bytes", + "encoding_rs", + "futures-core", + "h2", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-rustls", + "hyper-tls", + "hyper-util", + "js-sys", + "log", + "mime", + "native-tls", + "percent-encoding", + "pin-project-lite", + "quinn", + "rustls", + "rustls-pki-types", + "rustls-platform-verifier", + "serde", + "serde_json", + "serde_urlencoded", + "sync_wrapper", + "tokio", + "tokio-native-tls", + "tokio-rustls", + "tower", + "tower-http", + "tower-service", + "url", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", +] + +[[package]] +name = "ring" +version = "0.17.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7" +dependencies = [ + "cc", + "cfg-if", + "getrandom 0.2.17", + "libc", + "untrusted", + "windows-sys 0.52.0", +] + +[[package]] +name = "roaring" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ba9ce64a8f45d7fc86358410bb1a82e8c987504c0d4900e9141d69a9f26c885" +dependencies = [ + "bytemuck", + "byteorder", +] + +[[package]] +name = "roxmltree" +version = "0.21.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1964b10c76125c36f8afe190065a4bf9a87bf324842c05701330bba9f1cacbb" +dependencies = [ + "memchr", +] + +[[package]] +name = "rstest" +version = "0.23.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0a2c585be59b6b5dd66a9d2084aa1d8bd52fbdb806eafdeffb52791147862035" +dependencies = [ + "futures", + "futures-timer", + "rstest_macros", + "rustc_version", +] + +[[package]] +name = "rstest_macros" +version = "0.23.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "825ea780781b15345a146be27eaefb05085e337e869bff01b4306a4fd4a9ad5a" +dependencies = [ + "cfg-if", + "glob", + "proc-macro-crate", + "proc-macro2", + "quote", + "regex", + "relative-path", + "rustc_version", + "syn", + "unicode-ident", +] + +[[package]] +name = "rstest_reuse" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b3a8fb4672e840a587a66fc577a5491375df51ddb88f2a2c2a792598c326fe14" +dependencies = [ + "quote", + "rand 0.8.5", + "syn", +] + +[[package]] +name = "rustc-demangle" +version = "0.1.27" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b50b8869d9fc858ce7266cce0194bd74df58b9d0e3f6df3a9fc8eb470d95c09d" + +[[package]] +name = "rustc-hash" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2" + +[[package]] +name = "rustc-hash" +version = "2.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "357703d41365b4b27c590e3ed91eabb1b663f07c4c084095e60cbed4362dff0d" + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "rustix" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" +dependencies = [ + "bitflags", + "errno", + "libc", + "linux-raw-sys", + "windows-sys 0.61.2", +] + +[[package]] +name = "rustls" +version = "0.23.37" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "758025cb5fccfd3bc2fd74708fd4682be41d99e5dff73c377c0646c6012c73a4" +dependencies = [ + "aws-lc-rs", + "once_cell", + "ring", + "rustls-pki-types", + "rustls-webpki", + "subtle", + "zeroize", +] + +[[package]] +name = "rustls-native-certs" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "612460d5f7bea540c490b2b6395d8e34a953e52b491accd6c86c8164c5932a63" +dependencies = [ + "openssl-probe", + "rustls-pki-types", + "schannel", + "security-framework", +] + +[[package]] +name = "rustls-pki-types" +version = "1.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "be040f8b0a225e40375822a563fa9524378b9d63112f53e19ffff34df5d33fdd" +dependencies = [ + "web-time", + "zeroize", +] + +[[package]] +name = "rustls-platform-verifier" +version = "0.6.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d99feebc72bae7ab76ba994bb5e121b8d83d910ca40b36e0921f53becc41784" +dependencies = [ + "core-foundation 0.10.1", + "core-foundation-sys", + "jni", + "log", + "once_cell", + "rustls", + "rustls-native-certs", + "rustls-platform-verifier-android", + "rustls-webpki", + "security-framework", + "security-framework-sys", + "webpki-root-certs", + "windows-sys 0.61.2", +] + +[[package]] +name = "rustls-platform-verifier-android" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f" + +[[package]] +name = "rustls-webpki" +version = "0.103.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" +dependencies = [ + "aws-lc-rs", + "ring", + "rustls-pki-types", + "untrusted", +] + +[[package]] +name = "rustversion" +version = "1.0.22" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" + +[[package]] +name = "ryu" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" + +[[package]] +name = "same-file" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" +dependencies = [ + "winapi-util", +] + +[[package]] +name = "schannel" +version = "0.1.29" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91c1b7e4904c873ef0710c1f407dde2e6287de2bebc1bbbf7d430bb7cbffd939" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "security-framework" +version = "3.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d" +dependencies = [ + "bitflags", + "core-foundation 0.10.1", + "core-foundation-sys", + "libc", + "security-framework-sys", +] + +[[package]] +name = "security-framework-sys" +version = "2.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2691df843ecc5d231c0b14ece2acc3efb62c0a398c7e1d875f3983ce020e3" +dependencies = [ + "core-foundation-sys", + "libc", +] + +[[package]] +name = "semver" +version = "1.0.27" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d767eb0aabc880b29956c35734170f26ed551a859dbd361d140cdbeca61ab1e2" + +[[package]] +name = "seq-macro" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "serde_json" +version = "1.0.149" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "serde_spanned" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8bbf91e5a4d6315eee45e704372590b30e260ee83af6639d64557f51b067776" +dependencies = [ + "serde_core", +] + +[[package]] +name = "serde_urlencoded" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" +dependencies = [ + "form_urlencoded", + "itoa", + "ryu", + "serde", +] + +[[package]] +name = "sha2" +version = "0.10.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "digest", +] + +[[package]] +name = "sharded-slab" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6" +dependencies = [ + "lazy_static", +] + +[[package]] +name = "shlex" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64" + +[[package]] +name = "signal-hook-registry" +version = "1.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" +dependencies = [ + "errno", + "libc", +] + +[[package]] +name = "simd-adler32" +version = "0.3.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e320a6c5ad31d271ad523dcf3ad13e2767ad8b1cb8f047f75a8aeaf8da139da2" + +[[package]] +name = "simdutf8" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e" + +[[package]] +name = "siphasher" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b2aa850e253778c88a04c3d7323b043aeda9d3e30d5971937c1855769763678e" + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "smallvec" +version = "1.15.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03" + +[[package]] +name = "snap" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b6b67fb9a61334225b5b790716f609cd58395f895b3fe8b328786812a40bc3b" + +[[package]] +name = "socket2" +version = "0.6.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a766e1110788c36f4fa1c2b71b387a7815aa65f88ce0229841826633d93723e" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + +[[package]] +name = "spmc" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "02a8428da277a8e3a15271d79943e80ccc2ef254e78813a166a08d65e4c3ece5" + +[[package]] +name = "sqlparser" +version = "0.55.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4521174166bac1ff04fe16ef4524c70144cd29682a45978978ca3d7f4e0be11" +dependencies = [ + "log", + "recursive", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "stacker" +version = "0.1.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08d74a23609d509411d10e2176dc2a4346e3b4aea2e7b1869f19fdedbc71c013" +dependencies = [ + "cc", + "cfg-if", + "libc", + "psm", + "windows-sys 0.59.0", +] + +[[package]] +name = "strsim" +version = "0.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" + +[[package]] +name = "strum" +version = "0.26.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fec0f0aef304996cf250b31b5a10dee7980c85da9d759361292b8bca5a18f06" + +[[package]] +name = "strum" +version = "0.28.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9628de9b8791db39ceda2b119bbe13134770b56c138ec1d3af810d045c04f9bd" +dependencies = [ + "strum_macros 0.28.0", +] + +[[package]] +name = "strum_macros" +version = "0.26.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4c6bee85a5a24955dc440386795aa378cd9cf82acd5f764469152d2270e581be" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "rustversion", + "syn", +] + +[[package]] +name = "strum_macros" +version = "0.28.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ab85eea0270ee17587ed4156089e10b9e6880ee688791d45a905f5b1ca36f664" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "subtle" +version = "2.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" + +[[package]] +name = "syn" +version = "2.0.117" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "sync_wrapper" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263" +dependencies = [ + "futures-core", +] + +[[package]] +name = "synstructure" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "system-configuration" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a13f3d0daba03132c0aa9767f98351b3488edc2c100cda2d2ec2b04f3d8d3c8b" +dependencies = [ + "bitflags", + "core-foundation 0.9.4", + "system-configuration-sys", +] + +[[package]] +name = "system-configuration-sys" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e1d1b10ced5ca923a1fcb8d03e96b8d3268065d724548c0211415ff6ac6bac4" +dependencies = [ + "core-foundation-sys", + "libc", +] + +[[package]] +name = "tar" +version = "0.4.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "22692a6476a21fa75fdfc11d452fda482af402c008cdbaf3476414e122040973" +dependencies = [ + "filetime", + "libc", + "xattr", +] + +[[package]] +name = "target-triple" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "591ef38edfb78ca4771ee32cf494cb8771944bee237a9b91fc9c1424ac4b777b" + +[[package]] +name = "tempfile" +version = "3.27.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" +dependencies = [ + "fastrand", + "getrandom 0.4.2", + "once_cell", + "rustix", + "windows-sys 0.61.2", +] + +[[package]] +name = "termcolor" +version = "1.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06794f8f6c5c898b3275aebefa6b8a1cb24cd2c6c79397ab15774837a0bc5755" +dependencies = [ + "winapi-util", +] + +[[package]] +name = "test-case" +version = "3.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eb2550dd13afcd286853192af8601920d959b14c401fcece38071d53bf0768a8" +dependencies = [ + "test-case-macros", +] + +[[package]] +name = "test-case-core" +version = "3.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "adcb7fd841cd518e279be3d5a3eb0636409487998a4aff22f3de87b81e88384f" +dependencies = [ + "cfg-if", + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "test-case-macros" +version = "3.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c89e72a01ed4c579669add59014b9a524d609c0c88c6a585ce37485879f6ffb" +dependencies = [ + "proc-macro2", + "quote", + "syn", + "test-case-core", +] + +[[package]] +name = "test-log" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "37d53ac171c92a39e4769491c4b4dde7022c60042254b5fc044ae409d34a24d4" +dependencies = [ + "test-log-macros", + "tracing-subscriber", +] + +[[package]] +name = "test-log-macros" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "be35209fd0781c5401458ab66e4f98accf63553e8fae7425503e92fdd319783b" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "test_utils" +version = "0.27.1" +dependencies = [ + "delta_kernel", + "delta_kernel_default_engine", + "itertools 0.14.0", + "rstest", + "rstest_reuse", + "serde_json", + "tar", + "tempfile", + "tokio", + "tracing", + "tracing-subscriber", + "url", + "uuid", + "zstd", +] + +[[package]] +name = "thiserror" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52" +dependencies = [ + "thiserror-impl 1.0.69", +] + +[[package]] +name = "thiserror" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" +dependencies = [ + "thiserror-impl 2.0.18", +] + +[[package]] +name = "thiserror-impl" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "thousands" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3bf63baf9f5039dadc247375c29eb13706706cfde997d0330d05aa63a77d8820" + +[[package]] +name = "thread_local" +version = "1.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f60246a4944f24f6e018aa17cdeffb7818b76356965d03b07d6a9886e8962185" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "thrift" +version = "0.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e54bc85fc7faa8bc175c4bab5b92ba8d9a3ce893d0e9f42cc455c8ab16a9e09" +dependencies = [ + "byteorder", + "integer-encoding", + "ordered-float", +] + +[[package]] +name = "tiny-keccak" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2c9d3793400a45f954c52e73d068316d76b6f4e36977e3fcebb13a2721e80237" +dependencies = [ + "crunchy", +] + +[[package]] +name = "tinystr" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "42d3e9c45c09de15d06dd8acf5f4e0e399e85927b7f00711024eb7ae10fa4869" +dependencies = [ + "displaydoc", + "zerovec", +] + +[[package]] +name = "tinytemplate" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "be4d6b5f19ff7664e8c98d03e2139cb510db9b0a60b55f8e8709b689d939b6bc" +dependencies = [ + "serde", + "serde_json", +] + +[[package]] +name = "tinyvec" +version = "1.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3e61e67053d25a4e82c844e8424039d9745781b3fc4f32b8d55ed50f5f667ef3" +dependencies = [ + "tinyvec_macros", +] + +[[package]] +name = "tinyvec_macros" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" + +[[package]] +name = "tokio" +version = "1.50.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "27ad5e34374e03cfffefc301becb44e9dc3c17584f414349ebe29ed26661822d" +dependencies = [ + "bytes", + "libc", + "mio", + "parking_lot", + "pin-project-lite", + "signal-hook-registry", + "socket2", + "tokio-macros", + "windows-sys 0.61.2", +] + +[[package]] +name = "tokio-macros" +version = "2.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c55a2eff8b69ce66c84f85e1da1c233edc36ceb85a2058d11b0d6a3c7e7569c" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "tokio-native-tls" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbae76ab933c85776efabc971569dd6119c580d8f5d448769dec1764bf796ef2" +dependencies = [ + "native-tls", + "tokio", +] + +[[package]] +name = "tokio-rustls" +version = "0.26.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" +dependencies = [ + "rustls", + "tokio", +] + +[[package]] +name = "tokio-util" +version = "0.7.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ae9cec805b01e8fc3fd2fe289f89149a9b66dd16786abd8b19cfa7b48cb0098" +dependencies = [ + "bytes", + "futures-core", + "futures-sink", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "toml" +version = "0.9.12+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf92845e79fc2e2def6a5d828f0801e29a2f8acc037becc5ab08595c7d5e9863" +dependencies = [ + "indexmap", + "serde_core", + "serde_spanned", + "toml_datetime 0.7.5+spec-1.1.0", + "toml_parser", + "toml_writer", + "winnow 0.7.15", +] + +[[package]] +name = "toml" +version = "1.0.7+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dd28d57d8a6f6e458bc0b8784f8fdcc4b99a437936056fa122cb234f18656a96" +dependencies = [ + "indexmap", + "serde_core", + "serde_spanned", + "toml_datetime 1.0.1+spec-1.1.0", + "toml_parser", + "toml_writer", + "winnow 1.0.0", +] + +[[package]] +name = "toml_datetime" +version = "0.7.5+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92e1cfed4a3038bc5a127e35a2d360f145e1f4b971b551a2ba5fd7aedf7e1347" +dependencies = [ + "serde_core", +] + +[[package]] +name = "toml_datetime" +version = "1.0.1+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b320e741db58cac564e26c607d3cc1fdc4a88fd36c879568c07856ed83ff3e9" +dependencies = [ + "serde_core", +] + +[[package]] +name = "toml_edit" +version = "0.25.5+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ca1a40644a28bce036923f6a431df0b34236949d111cc07cb6dca830c9ef2e1" +dependencies = [ + "indexmap", + "toml_datetime 1.0.1+spec-1.1.0", + "toml_parser", + "winnow 1.0.0", +] + +[[package]] +name = "toml_parser" +version = "1.0.10+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7df25b4befd31c4816df190124375d5a20c6b6921e2cad937316de3fccd63420" +dependencies = [ + "winnow 1.0.0", +] + +[[package]] +name = "toml_writer" +version = "1.0.7+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f17aaa1c6e3dc22b1da4b6bba97d066e354c7945cac2f7852d4e4e7ca7a6b56d" + +[[package]] +name = "tower" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4" +dependencies = [ + "futures-core", + "futures-util", + "pin-project-lite", + "sync_wrapper", + "tokio", + "tower-layer", + "tower-service", +] + +[[package]] +name = "tower-http" +version = "0.6.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d4e6559d53cc268e5031cd8429d05415bc4cb4aefc4aa5d6cc35fbf5b924a1f8" +dependencies = [ + "bitflags", + "bytes", + "futures-util", + "http", + "http-body", + "iri-string", + "pin-project-lite", + "tower", + "tower-layer", + "tower-service", +] + +[[package]] +name = "tower-layer" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e" + +[[package]] +name = "tower-service" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" + +[[package]] +name = "tracing" +version = "0.1.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100" +dependencies = [ + "log", + "pin-project-lite", + "tracing-attributes", + "tracing-core", +] + +[[package]] +name = "tracing-attributes" +version = "0.1.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "tracing-core" +version = "0.1.36" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a" +dependencies = [ + "once_cell", + "valuable", +] + +[[package]] +name = "tracing-log" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ee855f1f400bd0e5c02d150ae5de3840039a3f54b025156404e34c23c03f47c3" +dependencies = [ + "log", + "once_cell", + "tracing-core", +] + +[[package]] +name = "tracing-serde" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "704b1aeb7be0d0a84fc9828cae51dab5970fee5088f83d1dd7ee6f6246fc6ff1" +dependencies = [ + "serde", + "tracing-core", +] + +[[package]] +name = "tracing-subscriber" +version = "0.3.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb7f578e5945fb242538965c2d0b04418d38ec25c79d160cd279bf0731c8d319" +dependencies = [ + "matchers", + "nu-ansi-term", + "once_cell", + "regex-automata", + "serde", + "serde_json", + "sharded-slab", + "smallvec", + "thread_local", + "tracing", + "tracing-core", + "tracing-log", + "tracing-serde", +] + +[[package]] +name = "try-lock" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" + +[[package]] +name = "trybuild" +version = "1.0.116" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47c635f0191bd3a2941013e5062667100969f8c4e9cd787c14f977265d73616e" +dependencies = [ + "glob", + "serde", + "serde_derive", + "serde_json", + "target-triple", + "termcolor", + "toml 1.0.7+spec-1.1.0", +] + +[[package]] +name = "twox-hash" +version = "2.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c" + +[[package]] +name = "typenum" +version = "1.19.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "562d481066bde0658276a35467c4af00bdc6ee726305698a55b86e61d7ad82bb" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "unicode-width" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254" + +[[package]] +name = "unicode-xid" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" + +[[package]] +name = "unity-catalog-delta-client-api" +version = "0.1.0" +dependencies = [ + "serde", + "serde_json", + "thiserror 2.0.18", + "tokio", +] + +[[package]] +name = "unity-catalog-delta-rest-client" +version = "0.1.0" +dependencies = [ + "percent-encoding", + "reqwest 0.13.2", + "serde", + "serde_json", + "thiserror 2.0.18", + "tokio", + "tracing", + "unity-catalog-delta-client-api", + "url", +] + +[[package]] +name = "untrusted" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" + +[[package]] +name = "ureq" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fdc97a28575b85cfedf2a7e7d3cc64b3e11bd8ac766666318003abbacc7a21fc" +dependencies = [ + "base64", + "der", + "flate2", + "log", + "native-tls", + "percent-encoding", + "rustls-pki-types", + "ureq-proto", + "utf-8", + "webpki-root-certs", +] + +[[package]] +name = "ureq-proto" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d81f9efa9df032be5934a46a068815a10a042b494b6a58cb0a1a97bb5467ed6f" +dependencies = [ + "base64", + "http", + "httparse", + "log", +] + +[[package]] +name = "url" +version = "2.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed" +dependencies = [ + "form_urlencoded", + "idna", + "percent-encoding", + "serde", + "serde_derive", +] + +[[package]] +name = "utf-8" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09cc8ee72d2a9becf2f2febe0205bbed8fc6615b7cb429ad062dc7b7ddd036a9" + +[[package]] +name = "utf8_iter" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" + +[[package]] +name = "utf8parse" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" + +[[package]] +name = "uuid" +version = "1.22.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a68d3c8f01c0cfa54a75291d83601161799e4a89a39e0929f4b0354d88757a37" +dependencies = [ + "getrandom 0.4.2", + "js-sys", + "rand 0.10.0", + "wasm-bindgen", +] + +[[package]] +name = "valuable" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65" + +[[package]] +name = "vcpkg" +version = "0.2.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426" + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "walkdir" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" +dependencies = [ + "same-file", + "winapi-util", +] + +[[package]] +name = "want" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e" +dependencies = [ + "try-lock", +] + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "wasip2" +version = "1.0.2+wasi-0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9517f9239f02c069db75e65f174b3da828fe5f5b945c4dd26bd25d89c03ebcf5" +dependencies = [ + "wit-bindgen", +] + +[[package]] +name = "wasip3" +version = "0.4.0+wasi-0.3.0-rc-2026-01-06" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5" +dependencies = [ + "wit-bindgen", +] + +[[package]] +name = "wasite" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8dad83b4f25e74f184f64c43b150b91efe7647395b42289f38e50566d82855b" + +[[package]] +name = "wasm-bindgen" +version = "0.2.114" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6532f9a5c1ece3798cb1c2cfdba640b9b3ba884f5db45973a6f442510a87d38e" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-futures" +version = "0.4.64" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e9c5522b3a28661442748e09d40924dfb9ca614b21c00d3fd135720e48b67db8" +dependencies = [ + "cfg-if", + "futures-util", + "js-sys", + "once_cell", + "wasm-bindgen", + "web-sys", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.114" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "18a2d50fcf105fb33bb15f00e7a77b772945a2ee45dcf454961fd843e74c18e6" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.114" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "03ce4caeaac547cdf713d280eda22a730824dd11e6b8c3ca9e42247b25c631e3" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.114" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "75a326b8c223ee17883a4251907455a2431acc2791c98c26279376490c378c16" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "wasm-encoder" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "990065f2fe63003fe337b932cfb5e3b80e0b4d0f5ff650e6985b1048f62c8319" +dependencies = [ + "leb128fmt", + "wasmparser", +] + +[[package]] +name = "wasm-metadata" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909" +dependencies = [ + "anyhow", + "indexmap", + "wasm-encoder", + "wasmparser", +] + +[[package]] +name = "wasm-streams" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "15053d8d85c7eccdbefef60f06769760a563c7f0a9d6902a13d35c7800b0ad65" +dependencies = [ + "futures-util", + "js-sys", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", +] + +[[package]] +name = "wasmparser" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe" +dependencies = [ + "bitflags", + "hashbrown 0.15.5", + "indexmap", + "semver", +] + +[[package]] +name = "web-sys" +version = "0.3.91" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "854ba17bb104abfb26ba36da9729addc7ce7f06f5c0f90f3c391f8461cca21f9" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "web-time" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "webpki-root-certs" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "804f18a4ac2676ffb4e8b5b5fa9ae38af06df08162314f96a68d2a363e21a8ca" +dependencies = [ + "rustls-pki-types", +] + +[[package]] +name = "which" +version = "8.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "81995fafaaaf6ae47a7d0cc83c67caf92aeb7e5331650ae6ff856f7c0c60c459" +dependencies = [ + "libc", +] + +[[package]] +name = "whoami" +version = "1.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5d4a4db5077702ca3015d3d02d74974948aba2ad9e12ab7df718ee64ccd7e97d" +dependencies = [ + "libredox", + "wasite", + "web-sys", +] + +[[package]] +name = "winapi-util" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "windows-core" +version = "0.62.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" +dependencies = [ + "windows-implement", + "windows-interface", + "windows-link", + "windows-result", + "windows-strings", +] + +[[package]] +name = "windows-implement" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "windows-interface" +version = "0.59.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-registry" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "02752bf7fbdcce7f2a27a742f798510f3e5ad88dbe84871e5168e2120c3d5720" +dependencies = [ + "windows-link", + "windows-result", + "windows-strings", +] + +[[package]] +name = "windows-result" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-strings" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-sys" +version = "0.45.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "75283be5efb2831d37ea142365f009c02ec203cd29a3ebecbc093d52315b66d0" +dependencies = [ + "windows-targets 0.42.2", +] + +[[package]] +name = "windows-sys" +version = "0.52.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" +dependencies = [ + "windows-targets 0.52.6", +] + +[[package]] +name = "windows-sys" +version = "0.59.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b" +dependencies = [ + "windows-targets 0.52.6", +] + +[[package]] +name = "windows-sys" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb" +dependencies = [ + "windows-targets 0.53.5", +] + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-targets" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e5180c00cd44c9b1c88adb3693291f1cd93605ded80c250a75d472756b4d071" +dependencies = [ + "windows_aarch64_gnullvm 0.42.2", + "windows_aarch64_msvc 0.42.2", + "windows_i686_gnu 0.42.2", + "windows_i686_msvc 0.42.2", + "windows_x86_64_gnu 0.42.2", + "windows_x86_64_gnullvm 0.42.2", + "windows_x86_64_msvc 0.42.2", +] + +[[package]] +name = "windows-targets" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" +dependencies = [ + "windows_aarch64_gnullvm 0.52.6", + "windows_aarch64_msvc 0.52.6", + "windows_i686_gnu 0.52.6", + "windows_i686_gnullvm 0.52.6", + "windows_i686_msvc 0.52.6", + "windows_x86_64_gnu 0.52.6", + "windows_x86_64_gnullvm 0.52.6", + "windows_x86_64_msvc 0.52.6", +] + +[[package]] +name = "windows-targets" +version = "0.53.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3" +dependencies = [ + "windows-link", + "windows_aarch64_gnullvm 0.53.1", + "windows_aarch64_msvc 0.53.1", + "windows_i686_gnu 0.53.1", + "windows_i686_gnullvm 0.53.1", + "windows_i686_msvc 0.53.1", + "windows_x86_64_gnu 0.53.1", + "windows_x86_64_gnullvm 0.53.1", + "windows_x86_64_msvc 0.53.1", +] + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "597a5118570b68bc08d8d59125332c54f1ba9d9adeedeef5b99b02ba2b0698f8" + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e08e8864a60f06ef0d0ff4ba04124db8b0fb3be5776a5cd47641e942e58c4d43" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" + +[[package]] +name = "windows_i686_gnu" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c61d927d8da41da96a81f029489353e68739737d3beca43145c8afec9a31a84f" + +[[package]] +name = "windows_i686_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" + +[[package]] +name = "windows_i686_gnu" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" + +[[package]] +name = "windows_i686_msvc" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "44d840b6ec649f480a41c8d80f9c65108b92d89345dd94027bfe06ac444d1060" + +[[package]] +name = "windows_i686_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" + +[[package]] +name = "windows_i686_msvc" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8de912b8b8feb55c064867cf047dda097f92d51efad5b491dfb98f6bbb70cb36" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "26d41b46a36d453748aedef1486d5c7a85db22e56aff34643984ea85514e94a3" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" + +[[package]] +name = "winnow" +version = "0.7.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df79d97927682d2fd8adb29682d1140b343be4ac0f08fd68b7765d9c059d3945" + +[[package]] +name = "winnow" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a90e88e4667264a994d34e6d1ab2d26d398dcdca8b7f52bec8668957517fc7d8" +dependencies = [ + "memchr", +] + +[[package]] +name = "wiremock" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08db1edfb05d9b3c1542e521aea074442088292f00b5f28e435c714a98f85031" +dependencies = [ + "assert-json-diff", + "base64", + "deadpool", + "futures", + "http", + "http-body-util", + "hyper", + "hyper-util", + "log", + "once_cell", + "regex", + "serde", + "serde_json", + "tokio", + "url", +] + +[[package]] +name = "wit-bindgen" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5" +dependencies = [ + "wit-bindgen-rust-macro", +] + +[[package]] +name = "wit-bindgen-core" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ea61de684c3ea68cb082b7a88508a8b27fcc8b797d738bfc99a82facf1d752dc" +dependencies = [ + "anyhow", + "heck", + "wit-parser", +] + +[[package]] +name = "wit-bindgen-rust" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21" +dependencies = [ + "anyhow", + "heck", + "indexmap", + "prettyplease", + "syn", + "wasm-metadata", + "wit-bindgen-core", + "wit-component", +] + +[[package]] +name = "wit-bindgen-rust-macro" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c0f9bfd77e6a48eccf51359e3ae77140a7f50b1e2ebfe62422d8afdaffab17a" +dependencies = [ + "anyhow", + "prettyplease", + "proc-macro2", + "quote", + "syn", + "wit-bindgen-core", + "wit-bindgen-rust", +] + +[[package]] +name = "wit-component" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2" +dependencies = [ + "anyhow", + "bitflags", + "indexmap", + "log", + "serde", + "serde_derive", + "serde_json", + "wasm-encoder", + "wasm-metadata", + "wasmparser", + "wit-parser", +] + +[[package]] +name = "wit-parser" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736" +dependencies = [ + "anyhow", + "id-arena", + "indexmap", + "log", + "semver", + "serde", + "serde_derive", + "serde_json", + "unicode-xid", + "wasmparser", +] + +[[package]] +name = "write-table" +version = "0.1.0" +dependencies = [ + "chrono", + "clap", + "common", + "delta_kernel", + "delta_kernel_default_engine", + "env_logger", + "itertools 0.14.0", + "serde_json", + "tokio", + "url", + "uuid", +] + +[[package]] +name = "writeable" +version = "0.6.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9edde0db4769d2dc68579893f2306b26c6ecfbe0ef499b013d731b7b9247e0b9" + +[[package]] +name = "xattr" +version = "1.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32e45ad4206f6d2479085147f02bc2ef834ac85886624a23575ae137c8aa8156" +dependencies = [ + "libc", + "rustix", +] + +[[package]] +name = "yoke" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72d6e5c6afb84d73944e5cedb052c4680d5657337201555f9f2a16b7406d4954" +dependencies = [ + "stable_deref_trait", + "yoke-derive", + "zerofrom", +] + +[[package]] +name = "yoke-derive" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b659052874eb698efe5b9e8cf382204678a0086ebf46982b79d6ca3182927e5d" +dependencies = [ + "proc-macro2", + "quote", + "syn", + "synstructure", +] + +[[package]] +name = "z85" +version = "3.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6e61e59a957b7ccee15d2049f86e8bfd6f66968fcd88f018950662d9b86e675" + +[[package]] +name = "zerocopy" +version = "0.8.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "efbb2a062be311f2ba113ce66f697a4dc589f85e78a4aea276200804cea0ed87" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0e8bc7269b54418e7aeeef514aa68f8690b8c0489a06b0136e5f57c4c5ccab89" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "zerofrom" +version = "0.1.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "50cc42e0333e05660c3587f3bf9d0478688e15d870fab3346451ce7f8c9fbea5" +dependencies = [ + "zerofrom-derive", +] + +[[package]] +name = "zerofrom-derive" +version = "0.1.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d71e5d6e06ab090c67b5e44993ec16b72dcbaabc526db883a360057678b48502" +dependencies = [ + "proc-macro2", + "quote", + "syn", + "synstructure", +] + +[[package]] +name = "zeroize" +version = "1.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0" + +[[package]] +name = "zerotrie" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2a59c17a5562d507e4b54960e8569ebee33bee890c70aa3fe7b97e85a9fd7851" +dependencies = [ + "displaydoc", + "yoke", + "zerofrom", +] + +[[package]] +name = "zerovec" +version = "0.11.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c28719294829477f525be0186d13efa9a3c602f7ec202ca9e353d310fb9a002" +dependencies = [ + "yoke", + "zerofrom", + "zerovec-derive", +] + +[[package]] +name = "zerovec-derive" +version = "0.11.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eadce39539ca5cb3985590102671f2567e659fca9666581ad3411d59207951f3" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "zlib-rs" +version = "0.6.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3be3d40e40a133f9c916ee3f9f4fa2d9d63435b5fbe1bfc6d9dae0aa0ada1513" + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" + +[[package]] +name = "zstd" +version = "0.13.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a" +dependencies = [ + "zstd-safe", +] + +[[package]] +name = "zstd-safe" +version = "7.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d" +dependencies = [ + "zstd-sys", +] + +[[package]] +name = "zstd-sys" +version = "2.0.16+zstd.1.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" +dependencies = [ + "cc", + "pkg-config", +] diff --git a/Cargo.toml b/Cargo.toml index a829c2f7fe..e199a3d42e 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -2,15 +2,19 @@ members = [ "acceptance", "benchmarks", + "default-engine", + "default-engine/test-utils", "derive-macros", "ffi", "kernel", "kernel/examples/*", "test-utils", + "workloads", "feature-tests", "mem-test", - "uc-client", # WIP: this is an experimental UC client for catalog-managed table work - "uc-catalog", # WIP: this is an experimental UC catalog implementation + "unity-catalog-delta-rest-client", # WIP: this is an experimental UC client for catalog-managed table work + "delta-kernel-unity-catalog", # WIP: this is an experimental UC catalog implementation + "unity-catalog-delta-client-api", # WIP: transport-agnostic UC client API traits ] # note that in addition to the members above, the workspace includes examples: # - inspect-table @@ -18,7 +22,7 @@ members = [ # - read-table-multi-threaded # - read-table-single-threaded # - write-table -resolver = "2" +resolver = "3" [workspace.package] edition = "2021" @@ -27,5 +31,5 @@ keywords = ["deltalake", "delta", "datalake"] license = "Apache-2.0" repository = "https://github.com/delta-io/delta-kernel-rs" readme = "README.md" -rust-version = "1.85" -version = "0.20.0" +rust-version = "1.88" +version = "0.27.1" diff --git a/README.md b/README.md index e01cedb403..4a6fbc8e2c 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Delta Kernel (rust)   [![build-status]][actions] [![latest-version]][crates.io] [![docs]][docs.rs] [![rustc-version-1.85+]][rustc] +# Delta Kernel (rust)   [![build-status]][actions] [![latest-version]][crates.io] [![docs]][docs.rs] ![Crates.io MSRV](https://img.shields.io/crates/msrv/delta_kernel) [build-status]: https://img.shields.io/github/actions/workflow/status/delta-io/delta-kernel-rs/build.yml?branch=main [actions]: https://github.com/delta-io/delta-kernel-rs/actions/workflows/build.yml?query=branch%3Amain @@ -22,10 +22,17 @@ is the Rust/C equivalent of [Java Delta Kernel][java-kernel]. Delta-kernel-rs is split into a few different crates: - kernel: The actual core kernel crate +- default-engine: The default Arrow/Tokio-based `Engine` implementation, published as + `delta_kernel_default_engine` - acceptance: Acceptance tests that validate correctness via the [Delta Acceptance Tests][dat] - derive-macros: A crate for our [derive-macros] to live in - ffi: Functionality that enables delta-kernel-rs to be used from `C` or `C++` See the [ffi](ffi) directory for more information. +- unity-catalog-delta-client-api: Transport-agnostic client traits and wire models for the Unity + Catalog Delta Tables API +- unity-catalog-delta-rest-client: REST/HTTP client for the Unity Catalog Delta Tables API +- delta-kernel-unity-catalog: Unity Catalog integration for the kernel, providing a catalog + `Committer` and helpers for catalog-managed tables ## Building By default we build only the `kernel` and `acceptance` crates, which will also build `derive-macros` @@ -45,26 +52,34 @@ In general, you will want to depend on `delta-kernel-rs` by adding it as a depen module. The core kernel includes facilities for reading and writing delta tables, and allows the consumer to implement their own `Engine` trait in order to build engine-specific implementations of the various `Engine` APIs that the kernel relies on (e.g. implement an engine-specific -`read_json_files()` using the native engine JSON reader). If there is no need to implement the -consumer's own `Engine` trait, the kernel has a feature flag to enable a default, asynchronous -`Engine` implementation built with [Arrow] and [Tokio]. +`read_json_files()` using the native engine JSON reader). If you do not need a custom `Engine`, +add the `delta_kernel_default_engine` crate to get the default asynchronous `Engine` implementation +built with [Arrow] and [Tokio]. ```toml # fewer dependencies, requires consumer to implement Engine trait. # allows consumers to implement their own in-memory format -delta_kernel = "0.20.0" +delta_kernel = "0.27.1" -# or turn on the default engine, based on latest arrow -delta_kernel = { version = "0.20.0", features = ["default-engine", "arrow"] } +# or pull in the default Arrow/Tokio engine alongside the kernel +delta_kernel = "0.27.1" +delta_kernel_default_engine = { version = "0.27.1", features = ["rustls"] } ``` ### Feature flags -There are more feature flags in addition to the `default-engine` flag shown above. Relevant flags -include: +`delta_kernel_default_engine` exposes the following feature flags: + +| Feature flag | Description | +| ------------- | ------------- | +| `rustls` | Use the rustls TLS backend for HTTPS object stores | +| `native-tls` | Use the native-tls TLS backend for HTTPS object stores | +| `arrow-58` | Build against arrow 58 (see Arrow versioning below) | +| `arrow-59` | Build against arrow 59 (see Arrow versioning below) | + +The `delta_kernel` crate itself exposes a few additional flags: | Feature flag | Description | | ------------- | ------------- | -| `default-engine` | Turn on the 'default' engine: async, arrow-based `Engine` implementation | | `arrow-conversion` | Conversion utilities for arrow/kernel schema interoperation | | `arrow-expression` | Expression system implementation for arrow | @@ -74,8 +89,8 @@ are still unstable. We therefore may break APIs within minor releases (that is, we will not break APIs in patch releases (`0.1.0` -> `0.1.1`). ## Arrow versioning -If you enable the `default-engine` feature, you get an implementation of the `Engine` trait that -uses [Arrow] as its data format. +If you depend on `delta_kernel_default_engine` (with either the `rustls` or `native-tls` feature), +you get an implementation of the `Engine` trait that uses [Arrow] as its data format. The [`arrow crate`](https://docs.rs/arrow/latest/arrow/) tends to release new major versions rather frequently. To enable engines that already integrate arrow to also integrate kernel and not force @@ -85,12 +100,12 @@ arrow versions as we can. We allow selecting the version of arrow to use via feature flags. Currently we support the following flags: -- `arrow-56`: Use arrow version 56 -- `arrow-57`: Use arrow version 57 +- `arrow-58`: Use arrow version 58 +- `arrow-59`: Use arrow version 59 - `arrow`: Use the latest arrow version. Note that this is an _unstable_ flag: we will bump this to the latest arrow version at every arrow version release. Only removing old arrow versions will cause a breaking change for kernel. If you require a specific version N of arrow, you should - specify it directly with `arrow-N`, e.g. `arrow-56`. + specify it directly with `arrow-N`, e.g. `arrow-58`. Note that if more than one `arrow-x` feature is enabled, kernel will use the _highest_ (latest) specified flag. This also means that if you use `--all-features` you will get the latest version of @@ -169,4 +184,4 @@ Some design principles which should be considered: [cargo-llvm-cov]: https://github.com/taiki-e/cargo-llvm-cov [FFI]: ffi/ [Arrow]: https://arrow.apache.org/rust/arrow/index.html -[Tokio]: https://tokio.rs/ \ No newline at end of file +[Tokio]: https://tokio.rs/ diff --git a/acceptance/Cargo.toml b/acceptance/Cargo.toml index 940702e785..a3dd32da82 100644 --- a/acceptance/Cargo.toml +++ b/acceptance/Cargo.toml @@ -14,28 +14,41 @@ rust-version.workspace = true [package.metadata.release] release = false +[features] +default = ["arrow"] +arrow = ["arrow-59"] +arrow-59 = ["delta_kernel/arrow-59", "delta_kernel_workloads/arrow-59", "test_utils/arrow-59"] +arrow-58 = ["delta_kernel/arrow-58", "delta_kernel_workloads/arrow-58", "test_utils/arrow-58"] + [dependencies] delta_kernel = { path = "../kernel", features = [ - "default-engine-rustls", - "arrow", + "default-engine-base", "internal-api", + "prettyprint", ] } +delta_kernel_default_engine = { path = "../default-engine", features = ["rustls"] } +delta_kernel_workloads = { path = "../workloads" } futures = "0.3" itertools = "0.14" serde = { version = "1", features = ["derive"] } serde_json = "1" thiserror = "2" +tracing = { version = "0.1", default-features = false } url = "2" [build-dependencies] -ureq = "3.1" +# native-tls validates against the OS trust store (like curl), so build-script downloads work +# behind a sanctioned TLS-intercepting corporate proxy whose CA isn't in ureq's bundled +# webpki-roots. See build.rs `build_agent`. +ureq = { version = "3.1", default-features = false, features = ["native-tls", "gzip"] } flate2 = "1.1" +sha2 = "0.10" tar = "0.4" [dev-dependencies] datatest-stable = "0.3" test-log = { version = "0.2", default-features = false, features = ["trace"] } -test_utils = { path = "../test-utils" } +test_utils = { path = "../test-utils", default-features = false } tempfile = "3" test-case = { version = "3.3.1" } tokio = { version = "1.47" } @@ -47,3 +60,7 @@ tracing-subscriber = { version = "0.3", default-features = false, features = [ [[test]] name = "dat_reader" harness = false + +[[test]] +name = "acceptance_workloads_reader" +harness = false diff --git a/acceptance/README.md b/acceptance/README.md index 8f89790f13..fd9018f192 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -5,3 +5,45 @@ These tests have been implemented as a separate sub-project in the workspace to ensure that they are acting as a "connector" for the kernel APIs, thereby allowing us to test the exact same engine interfaces that any connector implemented on top of delta-kernel-rs would be using. + +## Authoring a DAT case locally + +CI downloads a published DAT corpus, pinned to a version in `build.rs`. To create +a *new* DAT case without bumping that version, generate a corpus locally and point +the build at it with `DELTA_ACCEPTANCE_WORKLOADS_PATH`. See +https://github.com/delta-incubator/dat/blob/main/workload-generator/README.md for +the full authoring guide. + +Add a case to a `*Suite.scala` in the DAT `workload-generator`. Each `test(...)` +block is one case: build a table with `sql`, register it, then declare the specs to +generate (`readSpec` for scans, `snapshotSpec` for protocol/metadata). + +```scala +// in src/test/scala/io/delta/workload/tables/ReadsSuite.scala +test("my_case") { + sql("CREATE TABLE tbl (id INT, price DECIMAL(10,2)) USING delta") + sql("INSERT INTO tbl VALUES (1, 9.99), (2, 100.00)") + val t = registerTable("tbl") + readSpec(t) // full scan + readSpec(t, predicate = "id > 1") // predicate pushdown + snapshotSpec(t) // protocol + metadata +} +``` + +Then generate just that case and run it. The `-z` filter generates one case instead +of the whole suite (each case is a few seconds of Spark). + +```bash +# 1. Generate the corpus locally (Spark needs JDK 17). +export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 # any JDK 17 +export PATH="$JAVA_HOME/bin:$PATH" +cd /workload-generator +WORKLOAD_OUTPUT_DIR=/tmp/wl WORKLOAD_FORCE=true sbt 'testOnly *ReadsSuite -- -z "my_case"' + +# 2. Point kernel-rs at the local corpus and iterate. +cd +DELTA_ACCEPTANCE_WORKLOADS_PATH=/tmp/wl cargo nextest run -p acceptance +``` + +Do **not** commit a generated corpus. Bump `ACCEPTANCE_WORKLOADS_VERSION` +in `build.rs` once the case lands in a DAT release. diff --git a/acceptance/build.rs b/acceptance/build.rs index 22bac27f3a..8fccdaa133 100644 --- a/acceptance/build.rs +++ b/acceptance/build.rs @@ -1,26 +1,46 @@ -//! Build script for DAT +//! Build script for DAT and acceptance workload specs use std::env; use std::fs::File; use std::io::{BufReader, BufWriter, Read, Write}; -use std::path::Path; +use std::path::{Path, PathBuf}; +use std::time::Duration; use flate2::read::GzDecoder; +use sha2::{Digest, Sha256}; use tar::Archive; +use ureq::tls::{RootCerts, TlsConfig, TlsProvider}; use ureq::{Agent, Proxy}; const DAT_EXISTS_FILE_CHECK: &str = "tests/dat/.done"; -const OUTPUT_FOLDER: &str = "tests/dat"; -const VERSION: &str = "0.0.3"; +const DAT_OUTPUT_FOLDER: &str = "tests/dat"; +const DAT_VERSION: &str = "0.0.3"; +const ACCEPTANCE_WORKLOADS_VERSION: &str = "0.0.4"; +const DOWNLOAD_ATTEMPTS: usize = 3; + +// SHA-256 of the release assets. Each download is otherwise trusted purely on TLS; verifying these +// digests before extraction stops a tampered or MITM'd tarball from being unpacked to disk. Update +// alongside the version constants above. +const DAT_CHECKSUM: &str = "19c045bc6f4e8531d1985d0f7bb156d788e65078b435d613c8e4a9c753b4a982"; +const WORKLOAD_CHECKSUM: &str = "c129283e152239c810a6cf2e35571268bd7452fb379f6294abcee340f2436312"; + +/// Workloads to skip on Windows due to invalid filename characters. +/// Windows does not support these characters in filenames: < > : " | ? * +/// Additionally, some percent-encoded characters cause issues. +#[cfg(windows)] +const WINDOWS_SKIP_WORKLOADS: &[&str] = &[ + // Contains files with #, %, and ? in filenames + "fpe_special_chars_path/", +]; fn main() { - if dat_exists() { - return; + if !dat_exists() { + let tarball_data = download_dat_files(); + extract_dat_tarball(tarball_data); + write_done_file(); } - let tarball_data = download_dat_files(); - extract_tarball(tarball_data); - write_done_file(); + extract_acceptance_workloads(); } fn dat_exists() -> bool { @@ -29,34 +49,75 @@ fn dat_exists() -> bool { fn download_dat_files() -> Vec { let tarball_url = format!( - "https://github.com/delta-incubator/dat/releases/download/v{VERSION}/deltalake-dat-v{VERSION}.tar.gz" + "https://github.com/delta-incubator/dat/releases/download/v{DAT_VERSION}/deltalake-dat-v{DAT_VERSION}.tar.gz" ); - let response = if let Ok(proxy_url) = env::var("HTTPS_PROXY") { - let proxy = Proxy::new(&proxy_url).unwrap(); - let config = Agent::config_builder().proxy(proxy.into()).build(); - let agent = Agent::new_with_config(config); - agent.get(&tarball_url).call().unwrap() - } else { - ureq::get(&tarball_url).call().unwrap() - }; + download_tarball(&tarball_url, DAT_CHECKSUM) +} + +fn download_tarball(url: &str, expected_checksum: &str) -> Vec { + let agent = build_agent(); + let mut attempt = 1; + + loop { + match download(&agent, url) { + Ok(tarball_data) => { + verify_checksum(&tarball_data, expected_checksum); + return tarball_data; + } + Err(error) if attempt < DOWNLOAD_ATTEMPTS => { + eprintln!("Download attempt {attempt} failed: {error}. Retrying..."); + std::thread::sleep(Duration::from_secs(1)); + attempt += 1; + } + Err(error) => panic!("Failed to download {url}: {error}"), + } + } +} +fn download(agent: &Agent, url: &str) -> Result, Box> { + let response = agent.get(url).call()?; let mut tarball_data: Vec = Vec::new(); response .into_body() .as_reader() - .read_to_end(&mut tarball_data) - .unwrap(); + .read_to_end(&mut tarball_data)?; + + Ok(tarball_data) +} - tarball_data +/// Panic unless the SHA-256 of `data` equals `expected` (lowercase hex). Called before any +/// extraction, so a download whose digest doesn't match the pinned value fails the build instead +/// of being unpacked to disk. +fn verify_checksum(data: &[u8], expected: &str) { + let actual: String = Sha256::digest(data) + .iter() + .map(|b| format!("{b:02x}")) + .collect(); + if actual != expected { + panic!("tarball checksum mismatch: expected {expected}, got {actual}"); + } } -fn extract_tarball(tarball_data: Vec) { +/// Build a `ureq` agent that validates TLS against the OS trust store (native-tls). +fn build_agent() -> Agent { + let tls_config = TlsConfig::builder() + .provider(TlsProvider::NativeTls) + .root_certs(RootCerts::PlatformVerifier) + .build(); + let config = Agent::config_builder() + .tls_config(tls_config) + .proxy(Proxy::try_from_env()) + .build(); + Agent::new_with_config(config) +} + +fn extract_dat_tarball(tarball_data: Vec) { let tarball = GzDecoder::new(BufReader::new(&tarball_data[..])); let mut archive = Archive::new(tarball); - std::fs::create_dir_all(OUTPUT_FOLDER).expect("Failed to create output directory"); + std::fs::create_dir_all(DAT_OUTPUT_FOLDER).expect("Failed to create output directory"); archive - .unpack(OUTPUT_FOLDER) + .unpack(DAT_OUTPUT_FOLDER) .expect("Failed to unpack tarball"); } @@ -65,3 +126,119 @@ fn write_done_file() { BufWriter::new(File::create(DAT_EXISTS_FILE_CHECK).expect("Failed to create .done file")); write!(done_file, "done").expect("Failed to write .done file"); } + +/// Download and extract acceptance workload specs if not already done. +/// Downloads from GitHub releases at `v{ACCEPTANCE_WORKLOADS_VERSION}_dat` tag. +/// Extracts to `acceptance/workloads/`. +fn extract_acceptance_workloads() { + let manifest_dir = env::var("CARGO_MANIFEST_DIR").expect("CARGO_MANIFEST_DIR not set"); + let dir = PathBuf::from(manifest_dir); + + let output_dir = dir.join("workloads"); + + // if DELTA_ACCEPTANCE_WORKLOADS_PATH is set, point `workloads/` at that locally-generated + // corpus instead of downloading the pinned release, so a dev can iterate against their own + // corpus. + println!("cargo::rerun-if-env-changed=DELTA_ACCEPTANCE_WORKLOADS_PATH"); + if let Ok(local) = env::var("DELTA_ACCEPTANCE_WORKLOADS_PATH") { + link_local_workloads(&local, &output_dir); + return; + } + // Drop a stale override symlink (from a prior run) so we download into a real directory. + if output_dir.is_symlink() { + std::fs::remove_file(&output_dir).expect("Failed to remove stale workloads symlink"); + } + + let done_marker = output_dir.join(".done"); + + // Tell Cargo to re-run if the done marker changes + println!("cargo::rerun-if-changed={}", done_marker.display()); + + if done_marker.exists() { + return; + } + + // Download from GitHub releases + let tarball_url = format!( + "https://github.com/delta-incubator/dat/releases/download/v0.04-preview/v{ACCEPTANCE_WORKLOADS_VERSION}_dat_workloads.tar.gz" + ); + + let tarball_data = download_tarball(&tarball_url, WORKLOAD_CHECKSUM); + + // Extract tarball, skipping files with invalid Windows filenames + let decoder = GzDecoder::new(BufReader::new(&tarball_data[..])); + let mut archive = Archive::new(decoder); + std::fs::create_dir_all(&output_dir).expect("Failed to create acceptance output directory"); + + // Extract workloads one-by-one to skip tests that contain delta logs with special characters + // on Windows machines. This is because Windows does not support such files in the filesystem. + for entry in archive.entries().expect("Failed to read tarball entries") { + let mut entry = entry.expect("Failed to read tarball entry"); + + // On Windows, skip workloads that contain files with invalid filename characters + #[cfg(windows)] + { + let path = entry.path().expect("Failed to get entry path"); + let path_str = path.to_string_lossy(); + + // Skip entire workloads known to have problematic filenames + let should_skip = WINDOWS_SKIP_WORKLOADS + .iter() + .any(|skip| path_str.contains(skip)); + if should_skip { + eprintln!("Skipping Windows-incompatible workload file: {}", path_str); + continue; + } + } + + entry + .unpack_in(&output_dir) + .expect("Failed to unpack entry"); + } + + // Write .done marker + let mut done_file = BufWriter::new( + File::create(&done_marker).expect("Failed to create acceptance workloads .done file"), + ); + write!(done_file, "done").expect("Failed to write acceptance workloads .done file"); +} + +/// Point `workloads/` at a locally-generated corpus (`DELTA_ACCEPTANCE_WORKLOADS_PATH`) via a +/// symlink, replacing any prior download. +fn link_local_workloads(local: &str, output_dir: &Path) { + let local_path = std::fs::canonicalize(local).unwrap_or_else(|e| { + panic!("DELTA_ACCEPTANCE_WORKLOADS_PATH '{local}' is not accessible: {e}") + }); + assert!( + local_path.is_dir(), + "DELTA_ACCEPTANCE_WORKLOADS_PATH '{}' is not a directory", + local_path.display() + ); + + // Replace whatever is at workloads/ (a prior download dir, or an old symlink). + if output_dir.is_symlink() || output_dir.is_file() { + std::fs::remove_file(output_dir).expect("Failed to remove existing workloads path"); + } else if output_dir.is_dir() { + std::fs::remove_dir_all(output_dir).expect("Failed to remove existing workloads dir"); + } + + #[cfg(unix)] + std::os::unix::fs::symlink(&local_path, output_dir).unwrap_or_else(|e| { + panic!( + "Failed to symlink workloads -> {}: {e}", + local_path.display() + ) + }); + #[cfg(windows)] + std::os::windows::fs::symlink_dir(&local_path, output_dir).unwrap_or_else(|e| { + panic!( + "Failed to symlink workloads -> {}: {e}", + local_path.display() + ) + }); + + println!( + "cargo::warning=acceptance: using local workloads from {}", + local_path.display() + ); +} diff --git a/acceptance/src/acceptance_workloads/mod.rs b/acceptance/src/acceptance_workloads/mod.rs new file mode 100644 index 0000000000..4fa0998e7d --- /dev/null +++ b/acceptance/src/acceptance_workloads/mod.rs @@ -0,0 +1,147 @@ +//! Delta workload specification framework. +//! +//! Provides shared types and loading logic for correctness testing and benchmarking +//! workloads. All workload specs follow a flat file layout: +//! +//! ```text +//! / +//! table_info.json # Table metadata (optional) +//! delta/ # The Delta table +//! _delta_log/ +//! specs/ # Workload specifications (flat files) +//! workload_a.json +//! workload_b.json +//! expected/ # Expected results +//! workload_a/ +//! expected_data/ # Expected output as Parquet files +//! part-00000.parquet +//! part-00001.parquet +//! expected_metadata/ # Expected add files after data skipping as Parquet files +//! part-aaaaa.parquet +//! part-bbbbb.parquet +//! ``` +//! +//! ## Expected Data Format +//! +//! The `expected_data/` directory contains one or more Parquet files representing the +//! expected table content after executing the workload. Files starting with `.` or `_` +//! are ignored (e.g., `_SUCCESS`, `.crc` files). The data is compared order-independently +//! against the actual scan output by sorting rows before comparison. +//! +//! ## Read Spec Format +//! +//! Read workloads execute a scan and compare results against expected data: +//! +//! ```json +//! { +//! "type": "read", +//! "version": 5, // optional: time travel to version +//! "predicate": "id > 10", // optional: filter predicate +//! "columns": ["id", "name"], // optional: column projection +//! "expected": { "rowCount": 100 } +//! } +//! ``` +//! +//! For error cases, use `"error"` instead of `"expected"`: +//! +//! ```json +//! { +//! "type": "read", +//! "error": { "errorCode": "TABLE_NOT_FOUND" } +//! } +//! ``` +//! +//! ## Snapshot Construction Spec Format +//! +//! Snapshot workloads verify that a snapshot can be constructed and its metadata matches: +//! +//! ```json +//! { +//! "type": "snapshotConstruction", +//! "version": 5, // optional: time travel to version +//! "expected": { +//! "protocol": { "minReaderVersion": 1, "minWriterVersion": 2 }, +//! "metadata": { "id": "...", "schemaString": "...", ... } +//! } +//! } +//! ``` +//! +//! For error cases: +//! +//! ```json +//! { +//! "type": "snapshotConstruction", +//! "error": { "errorCode": "INVALID_TABLE" } +//! } +//! ``` + +pub mod validation; +pub mod workload; + +use std::path::{Path, PathBuf}; + +use delta_kernel_workloads::models::{Spec, TableInfo}; +use url::Url; + +/// A fully resolved test case ready for execution. +#[derive(Debug)] +pub struct TestCase { + /// Table metadata (absent if `table_info.json` doesn't exist). This + /// occurs when the table is corrupt and used for error testing. + pub table_info: Option, + /// Root directory of the test case + pub root_dir: PathBuf, + /// The workload specification + pub spec: Spec, + /// Name of the workload (spec filename without extension) + pub workload_name: String, +} + +impl TestCase { + /// Create a test case from a spec file path. + /// + /// Given a path like `.../workloads//specs/.json`, + /// loads the spec and resolves the test case root directory. + pub fn from_spec_path(spec_path: impl AsRef) -> Self { + let spec_path = spec_path.as_ref(); + + let workload_name = spec_path + .file_stem() + .and_then(|s| s.to_str()) + .expect("Invalid spec path: missing filename") + .to_string(); + + let root_dir = spec_path + .parent() // specs/ + .and_then(|p| p.parent()) // test_case/ + .expect("Invalid spec path: must be /specs/.json") + .to_path_buf(); + + let content = std::fs::read_to_string(spec_path).expect("Failed to read spec file"); + let spec: Spec = serde_json::from_str(&content).expect("Failed to parse spec file"); + + Self { + table_info: None, + root_dir, + spec, + workload_name, + } + } + + /// URL to the Delta table directory. + /// + /// Uses `table_info.table_path` if available, otherwise defaults to + /// `/delta`. + pub fn table_root(&self) -> Result { + match self.table_info.as_ref() { + Some(table_info) => Ok(table_info.resolved_table_root()), + None => Url::from_directory_path(self.root_dir.join("delta")) + .map_err(|_| format!("Failed to construct table root for {:?}", self.root_dir)), + } + } + + /// Path to the expected results directory for this workload. + pub fn expected_dir(&self) -> PathBuf { + self.root_dir.join("expected").join(&self.workload_name) + } +} diff --git a/acceptance/src/acceptance_workloads/validation.rs b/acceptance/src/acceptance_workloads/validation.rs new file mode 100644 index 0000000000..72d1acd7a4 --- /dev/null +++ b/acceptance/src/acceptance_workloads/validation.rs @@ -0,0 +1,166 @@ +//! Result validation for acceptance workload test cases. +//! +//! Compares actual kernel results against expected outcomes from the spec. For read workloads, +//! expected data is loaded from Parquet files in `expected_data/` and compared order-independently. +//! For snapshot workloads, protocol and metadata are compared directly. + +use std::fs::{self, File}; +use std::path::Path; + +use delta_kernel::arrow::array::RecordBatch; +use delta_kernel::arrow::compute::concat_batches; +use delta_kernel::arrow::datatypes::Schema as ArrowSchema; +use delta_kernel::engine::arrow_conversion::TryFromKernel; +use delta_kernel::parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder; +use delta_kernel::DeltaResult; +use delta_kernel_workloads::models::{ReadExpected, SnapshotExpected}; +use itertools::Itertools; +use tracing::debug; + +use super::workload::{ReadResult, SnapshotResult}; +use crate::data::assert_data_matches; + +/// Read expected data from parquet files in expected_dir/expected_data/. +fn read_expected_data(expected_dir: &Path) -> Result { + let expected_data_dir = expected_dir.join("expected_data"); + if !expected_data_dir.exists() { + return Err(format!( + "Expected data directory not found: {}", + expected_data_dir.display() + )); + } + + let parquet_paths = fs::read_dir(&expected_data_dir) + .map_err(|e| format!("Failed to read expected_data dir: {e}"))? + .filter_map(|entry| { + let path = entry.ok()?.path(); + let filename = path.file_name()?.to_str()?; + + if filename.starts_with('.') || filename.starts_with('_') { + return None; + } + + if path.extension()?.to_str()? == "parquet" { + Some(path) + } else { + None + } + }) + .collect_vec(); + + let mut batches = vec![]; + let mut schema = None; + + for path in parquet_paths { + let file = File::open(&path) + .map_err(|e| format!("Failed to open parquet file {}: {e}", path.display()))?; + let builder = ParquetRecordBatchReaderBuilder::try_new(file) + .map_err(|e| format!("Failed to create parquet reader: {e}"))?; + + if schema.is_none() { + schema = Some(builder.schema().clone()); + } + + let reader = builder + .build() + .map_err(|e| format!("Failed to build parquet reader: {e}"))?; + + for batch in reader { + let batch = batch.map_err(|e| format!("Failed to read batch: {e}"))?; + batches.push(batch); + } + } + + let schema = schema + .ok_or_else(|| format!("No parquet files found in {}", expected_data_dir.display()))?; + let all_data = + concat_batches(&schema, &batches).map_err(|e| format!("Failed to concat batches: {e}"))?; + Ok(all_data) +} + +/// Validate read results against expected outcome. +pub fn validate_read_result( + result: DeltaResult, + expected_dir: &Path, + expected: &ReadExpected, +) -> Result<(), String> { + match (result, expected) { + (Ok(read_result), ReadExpected::Success { expected: exp }) => { + // TODO: Check file_count and files_skipped against scan metrics once available. + // Note: These would be informational only, not authoritative, since different + // data skipping implementations may produce different results. + let _ = (exp.file_count, exp.files_skipped); + + // Validate data content + let expected_data = read_expected_data(expected_dir)?; + + let schema = ArrowSchema::try_from_kernel(read_result.schema.as_ref()) + .map_err(|e| e.to_string())?; + let schema = std::sync::Arc::new(schema); + assert_data_matches(read_result.batches, &schema, expected_data) + .map_err(|e| e.to_string())?; + + // Validate row count against spec's expected row counts + if read_result.row_count != exp.row_count { + return Err(format!( + "Row count mismatch: expected {}, got {}", + exp.row_count, read_result.row_count + )); + } + + Ok(()) + } + (Err(kernel_err), ReadExpected::Error { error }) => { + debug!( + "Got expected error '{}' with message: {:?}\nKernel error: {}", + error.error_code, error.error_message, kernel_err + ); + Ok(()) + } + (Ok(_), ReadExpected::Error { error }) => Err(format!( + "Expected error '{}' but succeeded", + error.error_code + )), + (Err(e), ReadExpected::Success { .. }) => { + Err(format!("Expected success but got error: {}", e)) + } + } +} + +/// Validate snapshot result against expected outcome. +pub fn validate_snapshot( + result: DeltaResult, + expected: &SnapshotExpected, +) -> Result<(), String> { + match (result, expected) { + (Ok(snapshot_result), SnapshotExpected::Success { expected }) => { + if snapshot_result.protocol != *expected.protocol { + return Err(format!( + "Expected protocol to match:\n{:?}\n{:?}", + snapshot_result.protocol, expected.protocol + )); + } + if snapshot_result.metadata != *expected.metadata { + return Err(format!( + "Expected metadata to match:\n{:?}\n{:?}", + snapshot_result.metadata, expected.metadata + )); + } + Ok(()) + } + (Err(kernel_err), SnapshotExpected::Error { error }) => { + debug!( + "Got expected error '{}' with message: {:?}\nKernel error: {}", + error.error_code, error.error_message, kernel_err + ); + Ok(()) + } + (Ok(_), SnapshotExpected::Error { error }) => Err(format!( + "Expected error '{}' but succeeded", + error.error_code + )), + (Err(e), SnapshotExpected::Success { .. }) => { + Err(format!("Expected success but got error: {}", e)) + } + } +} diff --git a/acceptance/src/acceptance_workloads/workload.rs b/acceptance/src/acceptance_workloads/workload.rs new file mode 100644 index 0000000000..246fbace69 --- /dev/null +++ b/acceptance/src/acceptance_workloads/workload.rs @@ -0,0 +1,177 @@ +//! Workload execution logic for Delta workload specifications. + +use std::sync::Arc; + +use delta_kernel::actions::{Metadata, Protocol}; +use delta_kernel::arrow::array::RecordBatch; +use delta_kernel::arrow::compute::filter_record_batch; +use delta_kernel::engine::arrow_data::EngineDataArrowExt as _; +use delta_kernel::engine::arrow_expression::evaluate_expression::evaluate_predicate; +use delta_kernel::expressions::Predicate; +use delta_kernel::schema::Schema; +use delta_kernel::snapshot::Snapshot; +use delta_kernel::{DeltaResult, Engine, Error, Version}; +use delta_kernel_workloads::models::{ReadSpec, SnapshotConstructionSpec, Spec, TimeTravel}; +use delta_kernel_workloads::predicate_parser::parse_predicate; +use itertools::Itertools; +use url::Url; + +use super::validation::{validate_read_result, validate_snapshot}; + +/// Result of executing a read workload. +#[derive(Debug)] +pub struct ReadResult { + /// The record batches from the scan. + pub batches: Vec, + /// The kernel schema of the data. + pub schema: Arc, + /// Total number of rows in the result. + pub row_count: u64, +} + +/// Result of executing a snapshot workload. +#[derive(Debug)] +pub struct SnapshotResult { + /// The version of the snapshot. + pub version: Version, + /// The protocol at this version. + pub protocol: Protocol, + /// The table metadata at this version. + pub metadata: Metadata, +} + +/// Build a snapshot with optional time travel. +fn build_snapshot( + engine: &dyn Engine, + table_root: &Url, + time_travel: Option<&TimeTravel>, +) -> DeltaResult> { + let version = time_travel + .map(TimeTravel::as_version) + .transpose() + .map_err(Error::generic)?; + + let mut builder = Snapshot::builder_for(table_root.clone()); + if let Some(v) = version { + builder = builder.at_version(v); + } + builder.build(engine) +} + +/// Execute a read workload. +pub fn execute_read_workload( + engine: Arc, + table_root: &Url, + read_spec: &ReadSpec, +) -> DeltaResult { + let snapshot = build_snapshot(engine.as_ref(), table_root, read_spec.time_travel.as_ref())?; + + let table_schema = snapshot.schema(); + + // Build scan with optional predicate and column projection + let mut scan_builder = snapshot.scan_builder(); + + // Extract and parse the predicate if one is present + let predicate = if let Some(ref predicate_string) = read_spec.predicate { + let predicate = parse_predicate(predicate_string, &table_schema).map_err(Error::generic)?; + let predicate = Arc::new(predicate); + scan_builder = scan_builder.with_predicate(predicate.clone()); + Some(predicate) + } else { + None + }; + + if let Some(ref cols) = read_spec.columns { + let projected_schema = table_schema.project(cols)?; + scan_builder = scan_builder.with_schema(projected_schema); + } + let scan = scan_builder.build()?; + + let schema = scan.logical_schema(); + + // Execute scan and apply row-level filtering + let batches: Vec = scan + .execute(engine)? + .map(|data| data?.try_into_record_batch()) + .try_collect()?; + let batches = filter_batches_with_predicate(batches, predicate.as_deref())?; + + // Compute row count from filtered batches + let row_count: u64 = batches.iter().map(|b| b.num_rows() as u64).sum(); + + Ok(ReadResult { + batches, + schema: schema.clone(), + row_count, + }) +} + +/// Filter record batches using a predicate expression. +fn filter_batches_with_predicate( + batches: Vec, + predicate: Option<&Predicate>, +) -> DeltaResult> { + let Some(predicate) = predicate else { + return Ok(batches); + }; + + batches + .into_iter() + .map(|batch| { + // Evaluate predicate to get boolean selection array + let selection = evaluate_predicate(predicate, &batch, false)?; + // Filter the batch using the selection + let filtered = filter_record_batch(&batch, &selection)?; + Ok(filtered) + }) + .collect() +} + +/// Execute a snapshot workload (for metadata validation). +pub fn execute_snapshot_workload( + engine: Arc, + table_root: &Url, + snapshot_spec: &SnapshotConstructionSpec, +) -> DeltaResult { + let snapshot = build_snapshot( + engine.as_ref(), + table_root, + snapshot_spec.time_travel.as_ref(), + )?; + + let config = snapshot.table_configuration(); + + Ok(SnapshotResult { + version: snapshot.version(), + protocol: config.protocol().clone(), + metadata: config.metadata().clone(), + }) +} + +/// Execute a workload and validate results. +pub fn execute_and_validate_workload( + engine: Arc, + table_root: &Url, + spec: &Spec, + expected_dir: &std::path::Path, +) -> Result<(), Box> { + match spec { + Spec::Read(read_spec) => { + let expected = read_spec + .expected + .as_ref() + .ok_or("ReadSpec must have expected or error field")?; + let result = execute_read_workload(engine, table_root, read_spec); + validate_read_result(result, expected_dir, expected)?; + } + Spec::SnapshotConstruction(snapshot_spec) => { + let expected = snapshot_spec + .expected + .as_ref() + .ok_or("SnapshotSpec must have expected or error field")?; + let result = execute_snapshot_workload(engine, table_root, snapshot_spec.as_ref()); + validate_snapshot(result, expected)?; + } + } + Ok(()) +} diff --git a/acceptance/src/data.rs b/acceptance/src/data.rs index 113b649b3c..39f6082aee 100644 --- a/acceptance/src/data.rs +++ b/acceptance/src/data.rs @@ -1,17 +1,20 @@ -use std::{path::Path, sync::Arc}; - -use delta_kernel::arrow::array::{Array, RecordBatch}; -use delta_kernel::arrow::compute::{concat_batches, lexsort_to_indices, take, SortColumn}; -use delta_kernel::arrow::datatypes::{DataType, Schema}; +use std::path::Path; +use std::sync::Arc; +use delta_kernel::arrow::array::RecordBatch; +use delta_kernel::arrow::compute::concat_batches; +use delta_kernel::arrow::datatypes::{DataType, Field, Fields, Schema, SchemaRef}; +use delta_kernel::arrow::util::pretty::pretty_format_batches; use delta_kernel::engine::arrow_data::EngineDataArrowExt as _; -use delta_kernel::object_store::{local::LocalFileSystem, ObjectStore}; +use delta_kernel::object_store::local::LocalFileSystem; +use delta_kernel::object_store::ObjectStore; use delta_kernel::parquet::arrow::async_reader::{ ParquetObjectReader, ParquetRecordBatchStreamBuilder, }; use delta_kernel::snapshot::Snapshot; use delta_kernel::{DeltaResult, Engine, Error}; -use futures::{stream::TryStreamExt, StreamExt}; +use futures::stream::TryStreamExt; +use futures::StreamExt; use itertools::Itertools; use crate::{TestCaseInfo, TestResult}; @@ -41,72 +44,85 @@ pub async fn read_golden(path: &Path, _version: Option<&str>) -> DeltaResult DeltaResult { - // Sort by all columns - let mut sort_columns = vec![]; - for col in batch.columns() { - match col.data_type() { - DataType::Struct(_) | DataType::List(_) | DataType::Map(_, _) => { - // can't sort structs, lists, or maps - } - _ => sort_columns.push(SortColumn { - values: col.clone(), - options: None, - }), - } +fn assert_schema_fields_match(schema: &Schema, golden: &Schema) -> DeltaResult<()> { + let schema_stripped = strip_metadata(schema); + let golden_stripped = strip_metadata(golden); + if schema_stripped.fields() != golden_stripped.fields() { + return Err(Error::generic(format!( + "Schema mismatch:\nActual: {:?}\nExpected: {:?}", + schema_stripped.fields(), + golden_stripped.fields() + ))); } - let indices = lexsort_to_indices(&sort_columns, None)?; - let columns = batch - .columns() - .iter() - .map(|c| take(c, &indices, None).unwrap()) - .collect(); - Ok(RecordBatch::try_new(batch.schema(), columns)?) + Ok(()) } -// Ensure that two schema have the same field names, and dict_is_ordered -// We ignore: -// - data type: This is checked already in `assert_columns_match` -// - nullability: parquet marks many things as nullable that we don't in our schema -// - metadata: because that diverges from the real data to the golden tabled data -fn assert_schema_fields_match(schema: &Schema, golden: &Schema) { - for (schema_field, golden_field) in schema.fields.iter().zip(golden.fields.iter()) { - assert!( - schema_field.name() == golden_field.name(), - "Field names don't match" - ); - assert!( - schema_field.dict_is_ordered() == golden_field.dict_is_ordered(), - "Field dict_is_ordered doesn't match" - ); +/// Recursively strip metadata from schema and all nested fields. +fn strip_metadata(schema: &Schema) -> Schema { + fn strip_field(field: &Field) -> Field { + Field::new( + field.name(), + strip_type(field.data_type()), + field.is_nullable(), + ) } -} -// some things are equivalent, but don't show up as equivalent for `==`, so we normalize here -fn normalize_col(col: Arc) -> Arc { - if let DataType::Timestamp(unit, Some(zone)) = col.data_type() { - if **zone == *"+00:00" { - let data_type = DataType::Timestamp(*unit, Some("UTC".into())); - delta_kernel::arrow::compute::cast(&col, &data_type).expect("Could not cast to UTC") - } else { - col + fn strip_type(dt: &DataType) -> DataType { + match dt { + DataType::Struct(fields) => DataType::Struct(Fields::from( + fields.iter().map(|f| strip_field(f)).collect_vec(), + )), + DataType::List(f) => DataType::List(Arc::new(strip_field(f))), + DataType::LargeList(f) => DataType::LargeList(Arc::new(strip_field(f))), + DataType::FixedSizeList(f, n) => DataType::FixedSizeList(Arc::new(strip_field(f)), *n), + DataType::Map(f, sorted) => DataType::Map(Arc::new(strip_field(f)), *sorted), + other => other.clone(), } - } else { - col } + Schema::new(schema.fields().iter().map(|f| strip_field(f)).collect_vec()) } -fn assert_columns_match(actual: &[Arc], expected: &[Arc]) { - for (actual, expected) in actual.iter().zip(expected) { - let actual = normalize_col(actual.clone()); - let expected = normalize_col(expected.clone()); - // note that array equality includes data_type equality - // See: https://arrow.apache.org/rust/arrow_data/equal/fn.equal.html - assert_eq!( - &actual, &expected, - "Column data didn't match. Got {actual:?}, expected {expected:?}" - ); +pub fn assert_data_matches( + result: Vec, + result_schema: &SchemaRef, + expected: RecordBatch, +) -> DeltaResult<()> { + let all_data = concat_batches(result_schema, result.iter())?; + + // Validate schemas match + assert_schema_fields_match(all_data.schema().as_ref(), expected.schema().as_ref())?; + + // Format both batches as strings for order-independent comparison + let actual_str = pretty_format_batches(std::slice::from_ref(&all_data)) + .map_err(|e| Error::generic(format!("Failed to format actual: {}", e)))? + .to_string(); + let expected_str = pretty_format_batches(std::slice::from_ref(&expected)) + .map_err(|e| Error::generic(format!("Failed to format expected: {}", e)))? + .to_string(); + + let mut actual_lines: Vec<&str> = actual_str.trim().lines().collect(); + let mut expected_lines: Vec<&str> = expected_str.trim().lines().collect(); + + // Sort data lines (skip header at indices 0-1 and footer at last index) + let num_actual = actual_lines.len(); + let num_expected = expected_lines.len(); + if num_actual > 3 { + actual_lines[2..num_actual - 1].sort_unstable(); + } + if num_expected > 3 { + expected_lines[2..num_expected - 1].sort_unstable(); + } + + // Compare sorted lines + if actual_lines != expected_lines { + return Err(Error::generic(format!( + "Data mismatch:\nExpected:\n{}\nActual:\n{}", + expected_lines.join("\n"), + actual_lines.join("\n") + ))); } + + Ok(()) } pub async fn assert_scan_metadata( @@ -127,17 +143,8 @@ pub async fn assert_scan_metadata( Ok(record_batch) }) .try_collect()?; - let all_data = concat_batches(&schema.unwrap(), batches.iter()).map_err(Error::from)?; - let all_data = sort_record_batch(all_data)?; let golden = read_golden(test_case.root_dir(), None).await?; - let golden = sort_record_batch(golden)?; - - assert_columns_match(all_data.columns(), golden.columns()); - assert_schema_fields_match(all_data.schema().as_ref(), golden.schema().as_ref()); - assert!( - all_data.num_rows() == golden.num_rows(), - "Didn't have same number of rows" - ); + assert_data_matches(batches, &schema.unwrap(), golden)?; Ok(()) } diff --git a/acceptance/src/lib.rs b/acceptance/src/lib.rs index 72658cdb4b..dbd2c1aeed 100644 --- a/acceptance/src/lib.rs +++ b/acceptance/src/lib.rs @@ -1,5 +1,6 @@ //! Helpers to validate Engine implementations +pub mod acceptance_workloads; pub mod data; pub mod meta; pub use meta::*; diff --git a/acceptance/src/meta.rs b/acceptance/src/meta.rs index ed325a5a9f..46afcf42d9 100644 --- a/acceptance/src/meta.rs +++ b/acceptance/src/meta.rs @@ -3,13 +3,14 @@ use std::fs::File; use std::path::{Path, PathBuf}; use std::sync::Arc; -use delta_kernel::object_store::{local::LocalFileSystem, path::Path as ObjectPath, ObjectStore}; +use delta_kernel::object_store::local::LocalFileSystem; +use delta_kernel::object_store::path::Path as ObjectPath; +use delta_kernel::object_store::ObjectStore; +use delta_kernel::{Engine, Error, Snapshot, Version}; use futures::stream::TryStreamExt; use serde::{Deserialize, Serialize}; use url::Url; -use delta_kernel::{Engine, Error, Snapshot, Version}; - #[derive(Debug, thiserror::Error)] pub enum AssertionError { #[error("Invalid test case data")] diff --git a/acceptance/tests/acceptance_workloads_reader.rs b/acceptance/tests/acceptance_workloads_reader.rs new file mode 100644 index 0000000000..6d75666ee5 --- /dev/null +++ b/acceptance/tests/acceptance_workloads_reader.rs @@ -0,0 +1,631 @@ +//! Test harness for acceptance workloads. +//! +//! This test uses datatest-stable to discover and run all workload specs in the +//! acceptance_workloads directory. Each spec file becomes its own test. + +use std::path::Path; + +use acceptance::acceptance_workloads::workload::execute_and_validate_workload; +use acceptance::acceptance_workloads::TestCase; + +/// Tests that cannot be executed due to test harness limitations. +/// These fail at parse time or cause infrastructure issues (OOM, hang). +/// All other failures (bugs, divergences, missing features) go in EXPECTED_KERNEL_FAILURES. +const SKIP_LIST: &[(&str, &str)] = &[ + ("DV-017/", "Huge table (2B rows) causes OOM/hang"), + // The Spec enum only supports "read" and "snapshot" types. + // CDF, transaction, and domain metadata specs fail at parse time. + ("_cdf", "CDF (Change Data Feed) test type not yet supported"), + ("_txn", "Transaction test type not yet supported"), + ( + "_domain_metadata", + "Domain metadata test type not yet supported", + ), + // These have `"type": "cdf"` which the harness cannot parse. + ( + "cdc_err_", + "CDF test type not yet supported (fails at parse time)", + ), + // These tests panic in reorder_struct_array (arrow_utils.rs:1068) with "index out of bounds". + // Both have predicate `flag = true` on tables with 6 columns (including timestamp/decimal). + // Both tables were created via CLONE/RESTORE operations. Simpler boolean predicate tests pass. + // + // Root cause: When data skipping evaluates the predicate, kernel builds a stats schema. + // Stats have nested structure (minValues/maxValues/nullCount are nested structs), but + // boolean columns have no min/max stats - only nullCount. This causes a mismatch between + // the expected stats schema and actual stats, leading to an invalid array index access + // during column reordering. The bug is in kernel's data skipping stats handling, not + // predicate parsing. Tests without predicates on these same tables work fine. + ( + "cloneDeepMultiType_readFiltered", + "Kernel panic in reorder_struct_array (index out of bounds)", + ), + ( + "restoreCheckData_filterBoolean", + "Kernel panic in reorder_struct_array (index out of bounds)", + ), +]; + +fn should_skip_test(test_path: &str) -> Option<&'static str> { + for (pattern, reason) in SKIP_LIST { + if test_path.contains(pattern) { + return Some(reason); + } + } + None +} + +/// Tests that CAN be executed but are expected to fail (kernel bugs or divergences). +/// Unlike SKIP_LIST, these workloads run and we assert they produce wrong results or errors. +/// When a kernel fix lands, the test will pass and the entry should be removed. +const EXPECTED_KERNEL_FAILURES: &[(&str, &[&str])] = &[ + // Kernel reads timestamps as Timestamp(Microsecond, Some("UTC")), + // Spark writes expected data as Timestamp(Nanosecond, None). + // Same instant, different Arrow representation. + ( + "Timestamp type divergence (kernel uses Microsecond/UTC, Spark uses Nanosecond/None)", + &[ + "cloneDeepMultiType/specs/cloneDeepMultiType_readAll", + "cr_timestamp_boundaries/specs/cr_timestamp_boundaries_read_all", + "dcscStructWithSpecialTypes/specs/dcscStructWithSpecialTypes_read_high_amount", + "dcscStructWithSpecialTypes/specs/dcscStructWithSpecialTypes_read_by_date", + "dcscStructWithSpecialTypes/specs/dcscStructWithSpecialTypes_read_all", + "dpReadPartitionTimestamp/specs/dpReadPartitionTimestamp_readAll", + "dsReadMultipleTypes/specs/dsReadMultipleTypes_readAll", + "dsReadTimestampType/specs/dsReadTimestampType_readAll", + "ds_datetime/specs/ds_datetime_hit_dt_eq", + "ds_datetime/specs/ds_datetime_hit_dt_gte", + "ds_datetime/specs/ds_datetime_hit_dt_lt", + "ds_datetime/specs/ds_datetime_miss_dt_2023", + "ds_datetime/specs/ds_datetime_miss_dt_2025", + "ds_date_trunc_timestamp/specs/ds_date_trunc_timestamp_hit_trunc_month_june", + "ds_date_trunc_timestamp/specs/ds_date_trunc_timestamp_hit_trunc_month_march", + "ds_date_trunc_timestamp/specs/ds_date_trunc_timestamp_miss_trunc_month_jan", + "gc_datetime/specs/gc_datetime_filter_date", + "gc_datetime/specs/gc_datetime_read_all", + "ntz_mixed_tz_ntz/specs/ntz_mixed_tz_ntz_filter_ntz_col", + "ntz_mixed_tz_ntz/specs/ntz_mixed_tz_ntz_full_scan", + "pve_timestamp_partition/specs/pve_timestamp_partition_read_all", + "restoreCheckData/specs/restoreCheckData_filterDecimal", + "restoreCheckData/specs/restoreCheckData_readAll", + "st_datetime_stats/specs/st_datetime_stats_filter_date_eq", + "st_datetime_stats/specs/st_datetime_stats_filter_date_range", + "st_datetime_stats/specs/st_datetime_stats_full_scan", + "cdc_multiple_types/specs/cdc_multiple_types_read_all", + "cdc_timestamp_tz_handling/specs/cdc_timestamp_tz_handling_read_all", + "gc_append_data/specs/gc_append_data_read_all", + "gc_delete/specs/gc_delete_read_all", + "gc_insert_by_name/specs/gc_insert_by_name_read_all", + "gc_update_source/specs/gc_update_source_read_all", + ], + ), + // Kernel produces {metadata, value}, Spark produces {value, metadata}. + ( + "Variant field order divergence (kernel {metadata,value}, Spark {value,metadata})", + &[ + "var_001_basic/specs/var_001_basic_read_all", + "var_001_basic/specs/var_001_basic_select_variant_col", + "var_002_basic_stats/specs/var_002_basic_stats_read_all", + "var_003_nested_stats/specs/var_003_nested_stats_read_all", + "var_004_non_objects/specs/var_004_non_objects_filter_first_three", + "var_004_non_objects/specs/var_004_non_objects_read_all", + "var_005_null_counts/specs/var_005_null_counts_filter_non_null", + "var_005_null_counts/specs/var_005_null_counts_read_all", + "var_006_different_types/specs/var_006_different_types_filter_by_id", + "var_006_different_types/specs/var_006_different_types_read_all", + "var_007_partitions/specs/var_007_partitions_filter_partition", + "var_007_partitions/specs/var_007_partitions_read_all", + "var_008_many_fields/specs/var_008_many_fields_read_all", + "var_009_unusual_chars/specs/var_009_unusual_chars_read_all", + "var_010_nested_fields/specs/var_010_nested_fields_read_all", + "var_011_missing_values/specs/var_011_missing_values_read_all", + "var_012_mixed_types/specs/var_012_mixed_types_filter_half", + "var_012_mixed_types/specs/var_012_mixed_types_read_all", + "var_013_extreme_values/specs/var_013_extreme_values_read_all", + "var_014_variant_in_struct/specs/var_014_variant_in_struct_filter_label", + "var_014_variant_in_struct/specs/var_014_variant_in_struct_read_all", + "var_015_string_skipping/specs/var_015_string_skipping_filter_middle", + "var_015_string_skipping/specs/var_015_string_skipping_read_all", + "var_016_array_variant/specs/var_016_array_variant_filter_array_size", + "var_016_array_variant/specs/var_016_array_variant_read_all", + "var_017_map_variant/specs/var_017_map_variant_filter_by_id", + "var_017_map_variant/specs/var_017_map_variant_read_all", + "var_018_column_mapping/specs/var_018_column_mapping_filter_by_id", + "var_018_column_mapping/specs/var_018_column_mapping_read_all", + "var_019_schema_evolution/specs/var_019_schema_evolution_filter_new_column", + "var_019_schema_evolution/specs/var_019_schema_evolution_read_all", + "var_019_schema_evolution/specs/var_019_schema_evolution_read_v2_before_evolution", + "var_020_time_travel/specs/var_020_time_travel_read_latest", + "var_020_time_travel/specs/var_020_time_travel_read_v1", + "var_020_time_travel/specs/var_020_time_travel_read_v2", + "var_021_optimized/specs/var_021_optimized_filter_after_optimize", + "var_021_optimized/specs/var_021_optimized_read_all", + "var_022_stat_fields/specs/var_022_stat_fields_read_all", + "var_all_json_types_read_all", + "var_cdf_read/specs/var_cdf_read_read_all", + "var_deeply_nested/specs/var_deeply_nested_read_all", + "var_large_array/specs/var_large_array_read_all", + "var_null_top_level/specs/var_null_top_level_filter_not_null", + "var_null_top_level/specs/var_null_top_level_read_all", + "var_numeric_precision/specs/var_numeric_precision_read_all", + "var_predicate_non_variant/specs/var_predicate_non_variant_filter_category_A", + "var_predicate_non_variant/specs/var_predicate_non_variant_read_all", + "var_projection/specs/var_projection_project_id_data", + "var_projection/specs/var_projection_read_all", + "var_unicode_escapes/specs/var_unicode_escapes_read_all", + "ds_variant_null_stats/specs/ds_variant_null_stats_hit_null_v_is_null", + "ds_variant_null_stats/specs/ds_variant_null_stats_hit_null_v_struct_v_is_null", + "ds_variant_null_stats/specs/ds_variant_null_stats_hit_v_is_not_null", + "ds_variant_null_stats/specs/ds_variant_null_stats_hit_v_struct_v_not_null", + "ds_variant_null_stats/specs/ds_variant_null_stats_miss_null_v_is_not_null", + "ds_variant_null_stats/specs/ds_variant_null_stats_miss_null_v_struct_v_not_null", + "ds_variant_null_stats/specs/ds_variant_null_stats_miss_v_is_null", + "ds_variant_null_stats/specs/ds_variant_null_stats_miss_v_struct_v_is_null", + "var_null_top_level/specs/var_null_top_level_filter_null", + ], + ), + ( + "Cannot fall back to log replay when checkpoint files are missing or incomplete", + &[ + "corrupt_incomplete_multipart_checkpoint/", + "ckp_incomplete_multipart/", + "ckp_missing_checkpoint_file/", + ], + ), + ( + "Cannot cast list to non-list data types during type widening", + &[ + "tw_array_element/specs/tw_array_element_read_", + "tw_map_key_value_widening/specs/tw_map_key_value_widening_read_all", + ], + ), + ( + "Schema deserialization fails for TimestampNTZ type", + &["ds_multi_file_time/"], + ), + ( + "Column mapping id mode fails with None in final_fields_cols", + &[ + "cm_id_matching_swapped/specs/cm_id_matching_swapped_select_", + "cm_id_matching_nonexistent/specs/cm_id_matching_nonexistent_select_", + ], + ), + ( + "Absolute-path DV has invalid percent-encoded path", + &["dv_storage_type_p/specs/dv_storage_type_p_read_after_absolute_path_dv"], + ), + ( + "Fails on missing/empty delta log (no files in log segment)", + &[ + "ct_empty_delta_log/specs/ct_empty_delta_log_snapshot", + "ct_missing_delta_log/specs/ct_missing_delta_log_snapshot", + "dseReadNonDeltaPath/specs/dseReadNonDeltaPath_snapshot", + ], + ), + ( + "Accepts truncated log when initial commits are missing but CRC files exist", + &["prod_truncated_log/"], + ), + ( + "Accepts checkpoint-only tables (no commits)", + &["cp_checkpoint_only_table/specs/cp_checkpoint_only_table_error"], + ), + ( + "Projected column not found after column mapping/schema order change", + &[ + "ds_schema_order_mismatch/specs/ds_schema_order_mismatch_single_col_last", + "ds_with_dvs_edge/specs/ds_with_dvs_edge_proj_and_skip_with_dv", + "dv_projection_with_pred/specs/dv_projection_with_pred_proj_and_pred", + ], + ), + ( + "Does not reject unsupported column mapping mode", + &["cm_err_003_invalid_mode/specs/cm_err_003_invalid_mode_error"], + ), + ( + "Reads corrupt/invalid commit or checkpoint without error", + &[ + "corrupt_truncated_commit_json_error", + "cp_err_missing_protocol/specs/cp_err_missing_protocol_error", + "ct_corrupt_parquet/specs/ct_corrupt_parquet_error", + "ct_invalid_json_error", + "dsReadCorruptCheckpoint/specs/dsReadCorruptCheckpoint_error", + "dsReadCorruptJson_error", + "dsReadModifyCheckpoint/specs/dsReadModifyCheckpoint_error", + ], + ), + ( + "Does not reject duplicate actions in commit", + &[ + "ct_duplicate_metadata/specs/ct_duplicate_metadata_error", + "ct_duplicate_protocol/specs/ct_duplicate_protocol_error", + "err_duplicate_add_same_version/specs/err_duplicate_add_same_version_error", + ], + ), + ( + "Snapshot construction succeeds even when data files are missing", + &[ + "ct_missing_data_file/specs/ct_missing_data_file_error", + "dv_err_002_missing_file/specs/dv_err_002_missing_file_error", + ], + ), + ( + "Does not validate DV integrity", + &[ + "dv_err_001_checksum/specs/dv_err_001_checksum_error", + "dv_err_003_malformed_path/specs/dv_err_003_malformed_path_error", + "err_dv_invalid_storage_type/specs/err_dv_invalid_storage_type_error", + "err_add_and_remove_same_path_dv/specs/err_add_and_remove_same_path_dv_error", + ], + ), + ( + "Does not validate schema integrity", + &[ + "err_schema_empty/specs/err_schema_empty_error", + "err_schema_invalid_json_error", + ], + ), + ( + "Does not require version 0 to exist", + &["err_missing_version_0/specs/err_missing_version_0_error"], + ), + ( + "Does not reject unknown reader features", + &["ev_unknown_reader_feature/specs/ev_unknown_reader_feature_error"], + ), + ( + "Does not enforce time travel safety", + &[ + "tt_blocked_beyond_retention/specs/tt_blocked_beyond_retention_error", + "tt_after_vacuum/specs/tt_after_vacuum_error", + ], + ), + ( + "_metadata.file_path column projection not supported", + &["DV-003/specs/DV-003_metadata_file_path"], + ), + ( + "variantShredding feature not supported", + &["pv_002_upgrade_to_current/specs/pv_002_upgrade_to_current_read_latest"], + ), + // Predicate parser: LIKE operator not supported + ( + "Predicate parser: LIKE operator not supported", + &[ + "ds_and_two_fields_hit_and_a_b_like", + "ds_and_two_fields_miss_and_b_like_2016", + "ds_long_strings_max_hit_like_A", + "ds_long_strings_max_hit_like_C", + "ds_long_strings_min_hit_like_A", + "ds_or_two_fields_hit_or_a_b_like", + "ds_or_two_fields_hit_or_a_b_like2", + "ds_or_two_fields_miss_or_b_like_2016", + "ds_starts_with_hit_like_a", + "ds_starts_with_hit_like_all", + "ds_starts_with_hit_like_ap", + "ds_starts_with_hit_like_m", + "ds_starts_with_hit_like_mic", + "ds_starts_with_miss_like_xyz", + "ds_starts_with_nested_hit_like_a", + "ds_starts_with_nested_hit_like_all", + "ds_starts_with_nested_hit_like_ap", + "ds_starts_with_nested_hit_like_m", + "ds_starts_with_nested_hit_like_mic", + "ds_starts_with_nested_miss_like_xyz", + "ds_string_patterns_hit_like_a", + "ds_string_patterns_hit_like_all", + "ds_string_patterns_hit_like_b", + "ds_string_patterns_miss_like_z", + "dsReadLikePredicate/specs/dsReadLikePredicate_filterLikeAlp", + "se_rename_pred/specs/se_rename_pred_pred_full_name_like", + "st_string_stats/specs/st_string_stats_filter_string_like", + ], + ), + // Predicate parser: CAST function not supported + ( + "Predicate parser: CAST function not supported", + &[ + "dpReadPartitionTimestamp/specs/dpReadPartitionTimestamp_filterTsEq", + "dpReadPartitionTimestamp/specs/dpReadPartitionTimestamp_filterTsGt", + "pve_byte_partition/specs/pve_byte_partition_filter_max", + "pve_byte_partition/specs/pve_byte_partition_filter_min", + "pve_byte_partition/specs/pve_byte_partition_filter_positive", + "pve_float_partition/specs/pve_float_partition_filter_eq", + "pve_float_partition/specs/pve_float_partition_filter_positive", + "pve_short_partition/specs/pve_short_partition_filter_max", + "pve_short_partition/specs/pve_short_partition_filter_min", + "pve_short_partition/specs/pve_short_partition_filter_range", + ], + ), + // Predicate parser: date/time functions not supported (date_add, datediff, month, year, trunc) + ( + "Predicate parser: date/time functions not supported", + &[ + "ds_date_add_sub_hit_date_add_all", + "ds_date_add_sub_hit_date_add_gt", + "ds_date_add_sub_miss_date_add_future", + "ds_datediff_hit_datediff_gt_30", + "ds_datediff_hit_datediff_lt_20", + "ds_month_function_hit_month_1", + "ds_month_function_hit_month_6", + "ds_month_function_miss_month_12", + "ds_trunc_date_hit_trunc_year_2023", + "ds_trunc_date_hit_trunc_year_2024", + "ds_trunc_date_miss_trunc_year_2020", + "ds_year_function_hit_year_2023", + "ds_year_function_hit_year_2024", + "ds_year_function_miss_year_2020", + ], + ), + // Predicate parser: IS NULL only supported on column references, not expressions + ( + "Predicate parser: IS NULL only supported on column references", + &[ + "ds_nulls_mixed_hit_a_gt_0_is_null", + "ds_nulls_mixed_hit_a_lt_0_is_null", + "ds_nulls_mixed_hit_not_a_gt_0_is_null", + "ds_nulls_nonnulls_only_hit_not_isnull", + "ds_nulls_nonnulls_only_miss_and_isnull", + "ds_nulls_nonnulls_only_miss_gt0_isnull", + "ds_nulls_nonnulls_only_miss_lt0_isnull", + "ds_nulls_nonnulls_only_miss_or_isnull", + "ds_nulls_only_nonnull_miss_and_isnull", + "ds_nulls_only_nonnull_miss_or_isnull", + "ds_nulls_only_null_hit_a_gt_1_is_null", + "ds_nulls_only_null_miss_not_a_gt_1_is_null", + "ds_nulls_partial_stats_hit_b_and_a_isnull", + "ds_nulls_partial_stats_hit_b_gt0_isnull", + "ds_nulls_partial_stats_hit_b_lt0_isnull", + "ds_nulls_partial_stats_hit_b_or_a_isnull", + "ds_nulls_partial_stats_hit_not_a_isnull", + "ds_nulls_partial_stats_miss_a_and_isnull", + "ds_nulls_partial_stats_miss_a_gt0_isnull", + "ds_nulls_partial_stats_miss_a_lt0_isnull", + "ds_nulls_partial_stats_miss_a_or_isnull", + ], + ), + // Predicate parser: function calls not supported (length, size, HEX) + ( + "Predicate parser: function calls not supported (length, size, HEX)", + &[ + "cc_005_varchar_constraint/specs/cc_005_varchar_constraint_filter_short_string", + "cc_009_array_constraint/specs/cc_009_array_constraint_filter_array_size", + "ds_stats_after_rename/specs/ds_stats_after_rename_hit_cc8_hex_1111", + "ds_stats_after_rename/specs/ds_stats_after_rename_hit_cc8_hex_3333", + "ds_typed_stats/specs/ds_typed_stats_hit_c8_hex_1111", + "ds_typed_stats/specs/ds_typed_stats_hit_c8_hex_3333", + ], + ), + // Predicate parser: variant_get function not supported + ( + "Predicate parser: variant_get function not supported", + &["var_022_stat_fields/specs/var_022_stat_fields_filter_variant_field"], + ), + // Predicate: _metadata columns not supported + ( + "Predicate: _metadata columns not supported", + &["rt_filter_read/specs/rt_filter_read_read_filtered"], + ), + // Kernel evaluator doesn't support `Column IN (literal_array)` pattern yet. + // The predicate parser correctly generates this, but evaluate_expression only handles + // `Literal IN Column` (scalar in array column) and `Literal IN Literal(Array)`. + ( + "Kernel: Column IN (literal_array) evaluation not supported", + &[ + "DV-004/specs/DV-004_filter_300_787_239", + "cks_dv_in_crc/specs/cks_dv_in_crc_read_remaining", + "dpReadPartitionAfterAppend/specs/dpReadPartitionAfterAppend_filterPartInCD", + "dpReadPartitionIn/specs/dpReadPartitionIn_filterPartInAC", + "dpReadPartitionIn/specs/dpReadPartitionIn_filterPartInBDE", + "ds_in_list/specs/ds_in_list_in_list_single_file", + "ds_in_list/specs/ds_in_list_in_list", + "ds_in_nested/specs/ds_in_nested_hit_code_in_1_2", + "ds_in_nested/specs/ds_in_nested_miss_code_in_99", + "ds_in_set/specs/ds_in_set_hit_in_1", + "ds_in_set/specs/ds_in_set_hit_in_12", + "ds_in_set/specs/ds_in_set_hit_in_123", + "ds_in_set/specs/ds_in_set_miss_in_456", + "ds_in_with_nulls_mixed/specs/ds_in_with_nulls_mixed_hit_in_1_null", + "ds_in_with_nulls_mixed/specs/ds_in_with_nulls_mixed_miss_in_99_null", + "ds_in_with_nulls_only/specs/ds_in_with_nulls_only_hit_in_1_2", + "ds_in_with_nulls_only/specs/ds_in_with_nulls_only_hit_in_5", + "ds_in_with_thresholds/specs/ds_in_with_thresholds_hit_in_cross_files", + "ds_in_with_thresholds/specs/ds_in_with_thresholds_hit_in_small", + "ds_in_with_thresholds/specs/ds_in_with_thresholds_miss_in_no_match", + "ds_not_in/specs/ds_not_in_not_in_1_2", + "ds_not_in/specs/ds_not_in_not_in_3", + "ds_not_in/specs/ds_not_in_not_in_all", + "ds_not_in/specs/ds_not_in_not_in_outside", + "dsReadInPredicate/specs/dsReadInPredicate_readInSet", + "dv_partition_pruning/specs/dv_partition_pruning_prune_north_or_east", + "tt_partition_filter/specs/tt_partition_filter_v0_part_0_or_1", + ], + ), + // Tests with simple predicates that fail due to timestamp schema mismatch + // These have predicates like "c1 = 1" that work, but the result comparison fails + // because kernel uses Timestamp(Microsecond, UTC) while Spark uses Timestamp(Nanosecond, None) + ( + "Simple predicates but timestamp schema mismatch in result validation", + &[ + "cc_007_multiple_constraints/specs/cc_007_multiple_constraints_filter_amount", + "cc_020_decimal_constraint/specs/cc_020_decimal_constraint_filter_high_price", + "ds_stats_after_drop/specs/ds_stats_after_drop_hit_c1_eq_1", + "ds_stats_after_drop/specs/ds_stats_after_drop_hit_c10_gt_1_5", + "ds_stats_after_drop/specs/ds_stats_after_drop_hit_c3_lt_1_5", + "ds_stats_after_drop/specs/ds_stats_after_drop_hit_c4_gt_1_0", + "ds_stats_after_drop/specs/ds_stats_after_drop_hit_c5_gte_ts", + "ds_stats_after_drop/specs/ds_stats_after_drop_hit_c6_gte_ts_ntz", + "ds_stats_after_drop/specs/ds_stats_after_drop_hit_c7_eq_date", + "ds_stats_after_drop/specs/ds_stats_after_drop_miss_c1_eq_10", + "ds_stats_after_drop/specs/ds_stats_after_drop_miss_c10_gt_2_5", + "ds_stats_after_drop/specs/ds_stats_after_drop_miss_c3_lt_0_5", + "ds_stats_after_drop/specs/ds_stats_after_drop_miss_c4_gt_5_0", + "ds_stats_after_drop/specs/ds_stats_after_drop_miss_c5_gte_future", + "ds_stats_after_drop/specs/ds_stats_after_drop_miss_c6_gte_future_ntz", + "ds_stats_after_drop/specs/ds_stats_after_drop_miss_c7_eq_future", + "ds_stats_after_rename/specs/ds_stats_after_rename_hit_cc1_eq_1", + "ds_stats_after_rename/specs/ds_stats_after_rename_hit_cc10_gt_1_5", + "ds_stats_after_rename/specs/ds_stats_after_rename_hit_cc2_eq_2", + "ds_stats_after_rename/specs/ds_stats_after_rename_hit_cc3_lt_1_5", + "ds_stats_after_rename/specs/ds_stats_after_rename_hit_cc4_gt_1_0", + "ds_stats_after_rename/specs/ds_stats_after_rename_hit_cc5_gte_ts", + "ds_stats_after_rename/specs/ds_stats_after_rename_hit_cc6_gte_ts_ntz", + "ds_stats_after_rename/specs/ds_stats_after_rename_hit_cc7_eq_date", + "ds_stats_after_rename/specs/ds_stats_after_rename_hit_cc9_false", + "ds_stats_after_rename/specs/ds_stats_after_rename_hit_cc9_true", + "ds_stats_after_rename/specs/ds_stats_after_rename_miss_cc1_eq_10", + "ds_stats_after_rename/specs/ds_stats_after_rename_miss_cc10_gt_2_5", + "ds_stats_after_rename/specs/ds_stats_after_rename_miss_cc2_eq_4", + "ds_stats_after_rename/specs/ds_stats_after_rename_miss_cc3_lt_0_5", + "ds_stats_after_rename/specs/ds_stats_after_rename_miss_cc4_gt_5_0", + "ds_stats_after_rename/specs/ds_stats_after_rename_miss_cc5_gte_future", + "ds_stats_after_rename/specs/ds_stats_after_rename_miss_cc6_gte_future_ntz", + "ds_stats_after_rename/specs/ds_stats_after_rename_miss_cc7_eq_future", + "ds_timestamp_microsecond/specs/ds_timestamp_microsecond_filter_exact_microsecond", + "ds_timestamp_microsecond/specs/ds_timestamp_microsecond_filter_microsecond", + "ds_timestamp_microsecond/specs/ds_timestamp_microsecond_read_all", + "pve_timestamp_partition/specs/pve_timestamp_partition_filter_eq", + "pve_timestamp_partition/specs/pve_timestamp_partition_filter_range", + "ds_typed_stats/specs/ds_typed_stats_hit_c1_eq_1", + "ds_typed_stats/specs/ds_typed_stats_hit_c10_gt_1_5", + "ds_typed_stats/specs/ds_typed_stats_hit_c2_eq_2", + "ds_typed_stats/specs/ds_typed_stats_hit_c3_lt_1_5", + "ds_typed_stats/specs/ds_typed_stats_hit_c4_gt_1_0", + "ds_typed_stats/specs/ds_typed_stats_hit_c5_gte_ts", + "ds_typed_stats/specs/ds_typed_stats_hit_c6_gte_ts_ntz", + "ds_typed_stats/specs/ds_typed_stats_hit_c7_eq_date", + "ds_typed_stats/specs/ds_typed_stats_hit_c9_false", + "ds_typed_stats/specs/ds_typed_stats_hit_c9_true", + "ds_typed_stats/specs/ds_typed_stats_miss_c1_eq_10", + "ds_typed_stats/specs/ds_typed_stats_miss_c10_gt_2_5", + "ds_typed_stats/specs/ds_typed_stats_miss_c2_eq_4", + "ds_typed_stats/specs/ds_typed_stats_miss_c3_lt_0_5", + "ds_typed_stats/specs/ds_typed_stats_miss_c4_gt_5_0", + "ds_typed_stats/specs/ds_typed_stats_miss_c5_gte_future", + "ds_typed_stats/specs/ds_typed_stats_miss_c6_gte_future_ntz", + "ds_typed_stats/specs/ds_typed_stats_miss_c7_eq_future", + "dsReadDecimalType/specs/dsReadDecimalType_readExpensive", + "gc_append_data/specs/gc_append_data_filter_partition", + "gc_delete/specs/gc_delete_filter_remaining", + "gc_insert_by_name/specs/gc_insert_by_name_filter_c2_g", + "gc_update_source/specs/gc_update_source_filter_c2_g_updated", + "tw_decimal_precision_read_large_values", + ], + ), + ( + "Timestamp-based time travel not yet supported", + &[ + "ict_basic_read_v0_ts", + "ict_dml_read_v0_ts", + "ict_enable_later_read_v0_no_ict_ts", + "ict_enable_later_read_v1_no_ict_ts", + "ict_enable_later_read_v3_with_ict_ts", + "ict_enabled_mid_lifecycle_read_v0_no_ict_ts", + "ict_enabled_mid_lifecycle_read_v1_no_ict_ts", + "ict_from_crc_read_v0_ts", + "ict_from_crc_read_v1_ts", + "ict_multiple_commits_read_v0_ts", + "ict_multiple_commits_read_v1_ts", + "ict_multiple_commits_read_v2_ts", + "ict_time_travel_read_v0_ts", + "ict_time_travel_read_v1_ts", + "ict_time_travel_read_v2_ts", + "ict_timestamp_resolution_edges_read_ts_v2_exact", + "ict_timestamp_resolution_edges_read_ts_v2_minus_1ms", + "ict_timestamp_resolution_edges_read_ts_v2_plus_1ms", + "ict_timestamp_resolution_edges_read_ts_v3_exact", + "ict_timestamp_resolution_edges_read_ts_v3_minus_1ms", + "ict_with_checkpoint_read_v1_ts", + "ict_with_checkpoint_read_v3_ts", + "tt_at_syntax_timestamp_v0", + "tt_at_syntax_timestamp_v1", + "tt_column_defaults_timestamp_v0_empty", + "tt_column_defaults_timestamp_v1_null", + "tt_exact_timestamp_version_0_exact_ts", + "tt_exact_timestamp_version_1_exact_ts", + "tt_multi_version_scans_timestamp_v0", + "tt_multi_version_scans_timestamp_v1", + "tt_partition_evolution_timestamp_v0_old_partition", + "tt_partition_evolution_timestamp_v1_new_partition", + "tt_relation_caching_timestamp_v0", + "tt_relation_caching_timestamp_v1", + "tt_schema_evolution_timestamp_v0_old_schema", + "tt_schema_evolution_timestamp_v1_new_schema", + "tt_sql_syntax_timestamp_v0", + "tt_sql_syntax_timestamp_v1", + "tt_timestamp_between_commits_timestamp_v0", + "tt_timestamp_between_commits_timestamp_v1", + "tt_timestamp_between_commits_timestamp_v2", + "tt_version_read_timestamp_v0", + "tt_version_read_timestamp_v1", + "tt_version_read_timestamp_v2", + ], + ), + // IS NULL predicates on struct columns fail because kernel's GetReferencedFields only + // resolves primitive columns (which have stats). Struct columns don't have nullCount stats + // at the struct level - only their primitive children do. The error "Predicate references + // unknown column: " is misleading since the column exists, it's just a struct type. + ( + "IS NULL on struct columns: kernel only resolves primitive columns for data skipping", + &[ + "dcscStructWithNull/specs/dcscStructWithNull_read_null", + "dcscStructWithNull/specs/dcscStructWithNull_read_non_null", + "ddefReadDefaultNested/specs/ddefReadDefaultNested_readNonNull", + ], + ), +]; + +fn acceptance_workloads_test(spec_path: &Path) -> datatest_stable::Result<()> { + let spec_path_raw = format!( + "{}/{}", + env!["CARGO_MANIFEST_DIR"], + spec_path.to_str().unwrap() + ); + let spec_path_abs = std::fs::canonicalize(&spec_path_raw) + .unwrap_or_else(|_| std::path::PathBuf::from(&spec_path_raw)); + let spec_path_str = spec_path_abs.to_string_lossy().to_string(); + // Normalize Windows backslashes to forward slashes for pattern matching + #[cfg(windows)] + let spec_path_str = spec_path_str.replace('\\', "/"); + + // Check expected kernel failures FIRST (path matching only - these need to + // actually run to assert kernel still fails). Skip list checked second. + let expected_failure = EXPECTED_KERNEL_FAILURES + .iter() + .find(|(_, patterns)| patterns.iter().any(|p| spec_path_str.contains(p))); + + if expected_failure.is_none() && should_skip_test(&spec_path_str).is_some() { + return Ok(()); + } + + // Load and execute test case + let test_case = TestCase::from_spec_path(&spec_path_abs); + let table_root = test_case.table_root().expect("Failed to get table URL"); + let engine = test_utils::create_default_engine(&table_root).expect("Failed to create engine"); + let result = execute_and_validate_workload( + engine, + &table_root, + &test_case.spec, + &test_case.expected_dir(), + ); + + match (result, expected_failure) { + (Err(_), Some(_)) => {} // Expected to fail, did fail + (Ok(_), None) => {} // Expected to pass, did pass + (Ok(_), Some((reason, _))) => panic!( + "Workload '{}' was expected to fail but succeeded! \ + Reason: {reason}. Remove from EXPECTED_KERNEL_FAILURES!", + test_case.workload_name + ), + (Err(e), None) => panic!("Workload '{}' failed: {}", test_case.workload_name, e), + } + Ok(()) +} + +datatest_stable::harness! { + { + test = acceptance_workloads_test, + root = "workloads/", + pattern = r"specs/.*\.json$" + }, +} diff --git a/acceptance/tests/other.rs b/acceptance/tests/other.rs index 09d3163713..b48edd1661 100644 --- a/acceptance/tests/other.rs +++ b/acceptance/tests/other.rs @@ -3,6 +3,8 @@ /// Since each new `.rs` file in this directory results in increased build and link time, it is /// important to only add new files if absolutely necessary for code readability or test /// performance. +use std::path::Path; + use delta_kernel::last_checkpoint_hint::LastCheckpointHint; #[test] @@ -15,6 +17,19 @@ fn test_checkpoint_serde() { assert_eq!(cp.version, 2) } +/// Guards against mistaking local-override results for the pinned release. +#[test] +fn acceptance_workloads_is_not_a_local_override() { + let workloads = Path::new(env!("CARGO_MANIFEST_DIR")).join("workloads"); + let meta = std::fs::symlink_metadata(&workloads).expect("workloads/ should exist after build"); + assert!( + !meta.file_type().is_symlink(), + "acceptance/workloads is a symlink to a LOCAL corpus, so you are not testing the pinned \ + release. Unset DELTA_ACCEPTANCE_WORKLOADS_PATH and rebuild (or remove acceptance/workloads) \ + before trusting these results." + ); +} + /* #[tokio::test] async fn test_read_last_checkpoint() { diff --git a/benchmarks/Cargo.toml b/benchmarks/Cargo.toml index 48114d6522..1a820d3b40 100644 --- a/benchmarks/Cargo.toml +++ b/benchmarks/Cargo.toml @@ -8,17 +8,36 @@ repository.workspace = true version.workspace = true rust-version.workspace = true +[features] +default = ["arrow"] +arrow = ["arrow-59"] +arrow-59 = ["delta_kernel/arrow-59", "delta_kernel_workloads/arrow-59"] +arrow-58 = ["delta_kernel/arrow-58", "delta_kernel_workloads/arrow-58"] + +[build-dependencies] +flate2 = "1.1" +sha2 = "0.10" +tar = "0.4" +ureq = { version = "3.1", default-features = false, features = ["native-tls", "gzip"] } + [dependencies] -delta_kernel = { path = "../kernel", features = ["arrow", "default-engine-rustls", "internal-api"] } -flate2 = "1" +delta_kernel = { path = "../kernel", features = ["default-engine-base", "internal-api"] } +delta_kernel_default_engine = { path = "../default-engine", features = ["rustls"] } +delta_kernel_workloads = { path = "../workloads" } +delta-kernel-unity-catalog = { path = "../delta-kernel-unity-catalog" } +object_store = { version = "0.12.3", features = ["aws"] } +test_utils = { path = "../test-utils" } +rayon = "1" serde = { version = "1", features = ["derive"] } serde_json = "1" -tar = "0.4" +tokio = { version = "1", features = ["rt-multi-thread"] } +unity-catalog-delta-client-api = { path = "../unity-catalog-delta-client-api" } +unity-catalog-delta-rest-client = { path = "../unity-catalog-delta-rest-client" } url = { version = "2", features = ["serde"] } [dev-dependencies] criterion = "0.5" -rstest = "0.23" +tempfile = "3" [[bench]] name = "workload_bench" diff --git a/benchmarks/README.md b/benchmarks/README.md index 1c9d7df392..87edb49db0 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -23,54 +23,74 @@ samply record cargo bench -p delta_kernel_benchmarks --bench workload_bench "som #### By benchmark name -Benchmark names follow a hierarchical path structure assembled from the Criterion group name, the table name, the spec file name, the operation, and (for `Read` workloads) the read config name: +Benchmark names use the human-readable table name and spec file name. Read benchmarks also +include the harness config name: ``` -workload_benchmarks/{table_name}/{spec_file_name}/{operation}/{config_name} +{table_name}/{spec_file_name} # snapshot construction +{table_name}/{spec_file_name}/{config_name} # read ``` -- `workload_benchmarks` — the Criterion benchmark group (always this literal string) -- `{table_name}` — the `name` field from `table_info.json` +- `{table_name}` — the human-readable `name` from the table's `tableInfo.json` - `{spec_file_name}` — the spec filename without its `.json` extension (the `case_name`) -- `{operation}` — `snapshot_construction` or `read_metadata` -- `{config_name}` — only present for `Read` workloads; e.g. `serial`, `parallel_2`, `parallel_4` +- `{config_name}` — the read harness config from + [`bench-registry.json`](#registry-bench-registryjson), such as `serial` or `parallel2` Examples: ``` -workload_benchmarks/checkpoint_v9_1009_versions/snapshot_latest/snapshot_construction -workload_benchmarks/checkpoint_v9_1009_versions/snapshot_latest/read_metadata/serial -workload_benchmarks/checkpoint_v9_1009_versions/snapshot_latest/read_metadata/parallel_4 +crcVeryStale/snapshotLatest +v1Checkpoint/readMetadataLatest/serial +v2Checkpoint/readMetadataLatest/parallel2 ``` The filter argument is a regular expression, so you can create patterns to target the benchmarks that you want: ```bash # all benchmarks for a specific table name -cargo bench -p delta_kernel_benchmarks --bench workload_bench "checkpoint_v9_1009_versions" +cargo bench -p delta_kernel_benchmarks --bench workload_bench "crcVeryStale" # all benchmarks for either of two tables (| for OR) -cargo bench -p delta_kernel_benchmarks --bench workload_bench "checkpoint_v9_1009_versions|10_adds" +cargo bench -p delta_kernel_benchmarks --bench workload_bench "crcVeryStale|crcLatest" -# snapshot_construction workloads for a specific table (.* to AND two parts of the name) -cargo bench -p delta_kernel_benchmarks --bench workload_bench "checkpoint_v9_1009_versions.*snapshot_construction" +# all snapshot-construction benchmarks (the snapshotLatest spec) +cargo bench -p delta_kernel_benchmarks --bench workload_bench "snapshotLatest" + +# snapshot-construction workloads for a specific table (.* to AND two parts of the name) +cargo bench -p delta_kernel_benchmarks --bench workload_bench "crcVeryStale.*snapshotLatest" # profile a specific benchmark with samply -samply record cargo bench -p delta_kernel_benchmarks --bench workload_bench "workload_benchmarks/checkpoint_v9_1009_versions/snapshot_latest/snapshot_construction" +samply record cargo bench -p delta_kernel_benchmarks --bench workload_bench "crcVeryStale/snapshotLatest" ``` #### By tag (`BENCH_TAGS`) -Set the `BENCH_TAGS` environment variable to a comma-separated list of tags to run only tables whose `tags` field (in table_info.json) contains at least one matching tag. If `BENCH_TAGS` is unset or empty, all tables are loaded and benchmarked. +Set the `BENCH_TAGS` environment variable to a comma-separated list of tags to run only tables whose `tags` field (in `tableInfo.json`) contains at least one matching tag. If `BENCH_TAGS` is unset or empty, all tables are loaded and benchmarked. ```bash # run only tables tagged "base" BENCH_TAGS=base cargo bench -p delta_kernel_benchmarks ``` -Built-in tags: +Built-in tags (with current table assignments - for the most up-to-date table assignments, run benchmarks locally and inspect `benchmarks/workloads/benchmarks//tableInfo.json` to learn about the existing tables): - **`base`** — a base set of tables run in CI - -You can also add custom tags to any `table_info.json` to group tables relevant to your work, then pass that tag via `BENCH_TAGS` without modifying any code: + - Tables: `101kAdds1kCommitsSinceChkpt1Chkpt` +- **`commit-size-scaling`** — tables for comparing how log replay time scales with the number of actions in the log; all are single-commit tables with varying action counts (100, 1k, 10k, 100k, 1M) + - Tables: `100Adds0Chkpts`, `1kAdds0Chkpts`, `10kAdds0Chkpts`, `100kAdds0Chkpts`, `1MAddsNoData0Chkpts` +- **`checkpoint-reads-by-type`** — tables for comparing checkpoint reading performance for different kinds of checkpointing + - Tables: `10kAdds0CommitsSinceChkpt1Chkpt`, `10kAdds0CommitsSinceChkpt1V2Chkpt` +- **`v2-checkpoint`** — tables with v2 checkpoints + - Tables: `10kAdds0CommitsSinceChkpt1V2Chkpt` +- **`crc-optimization`** — tables for comparing how CRC files affect log replay timing; designed to isolate the effect of CRC files at different versions relative to the checkpoint and latest version + - Tables: `101kAdds1kCommitsSinceChkpt1Chkpt`, `20kAdds100CommitsSinceChkpt1Chkpt0CommitsSinceCrc`, `20kAdds100CommitsSinceChkpt1Chkpt50CommitsSinceCrc`, `20kAdds100CommitsSinceChkpt1ChkptNoCrc` +- **`time-travel-optimization`** — tables with multiple specs or specs not at the latest version, useful for benchmarking snapshot construction at historical versions + - Tables: `101kAdds1kCommitsSinceChkpt1Chkpt`, `200kAdds0CommitsSinceChkpt2Chkpts0CommitsSinceCrc` +- **`listing-optimization`** — table for benchmarking log listing efficiency (e.g. `list_from()` call patterns); useful for features that optimize how the delta log directory is scanned + - Tables: `200kAdds0CommitsSinceChkpt2Chkpts0CommitsSinceCrc` +- **`metadata-only`** — tables with no actual data files, useful for isolating log metadata processing overhead + - Tables: `1MAddsNoData0Chkpts` + + +You can also add custom tags to the `tags` field of any local `tableInfo.json` to group tables relevant to your work, then pass that tag via `BENCH_TAGS` without modifying any code: ```bash BENCH_TAGS=my-feature cargo bench -p delta_kernel_benchmarks @@ -79,16 +99,91 @@ BENCH_TAGS=my-feature cargo bench -p delta_kernel_benchmarks BENCH_TAGS=base,my-feature cargo bench -p delta_kernel_benchmarks ``` +### Running benchmarking on a PR + +Benchmarks run automatically on every push to a non-draft PR with `BENCH_TAGS=base`, no filter. +Results are posted as a single PR comment that updates in place on each push, so the latest +comment always reflects the latest commit. The comment is posted by a companion +`workflow_run`-triggered workflow ([benchmark-post-comment.yml](../.github/workflows/benchmark-post-comment.yml)), +so the bench job itself runs with a read-only token even on fork PRs. To bench a draft PR or +override the tags or filter, post a `/bench` comment as documented below -- it updates the +same comment that the auto-trigger uses. + +The bench job fails if any benchmark is at least 15% slower than the base branch. The result +comment is still posted so you can see which benchmark regressed. To merge anyway (for an +expected or noise-driven regression), add the `ignore-benchmark-failure` label to the PR. + +To trigger benchmarks on a pull request manually, post a comment using the following syntax: + +``` +/bench [--tags ] [--filter ] +``` + +- `--tags` sets `BENCH_TAGS` (comma-separated), controlling which table groupings run. +- `--filter` is a single-token Criterion regex matched against benchmark names. +- Both flags are optional and independent; they can be given in any order. +- When both are specified, they apply as AND: only benchmarks from tables that match the tag filter AND whose name matches the regex are run. +- Running just `/bench` (with no flags) defaults to `BENCH_TAGS=base`. If neither flag is parsed, the same default applies. + +Examples: +``` +/bench # BENCH_TAGS=base, all benchmark names +/bench --tags base,my-tag # BENCH_TAGS=base,my-tag, all benchmark names +/bench --filter snapshotLatest # no BENCH_TAGS set, only snapshot-construction benchmarks +/bench --tags base --filter crcVeryStale.*snapshotLatest # only snapshot-construction benchmarks from tables tagged "base" +/bench --filter crcVeryStale|crcLatest # no BENCH_TAGS set, OR two table names +``` + +See [By tag (`BENCH_TAGS`)](#by-tag-bench_tags) for how tags work and [By benchmark name](#by-benchmark-name) for regex pattern examples. Results are posted automatically as a PR comment, comparing the PR branch against the base branch. +CI timings are noisy and tend to run higher than on dedicated hardware, but proportional differences between branches are a rough signal for performance changes. + +## Registry (`bench-registry.json`) + +[`bench-registry.json`](bench-registry.json) maps read benchmarks to the harness configs they run. +Each registered read workload expands into one Criterion benchmark per config, and the config +`name` becomes the trailing path segment of the benchmark name (see +[By benchmark name](#by-benchmark-name)). Snapshot-construction workloads construct a new snapshot +each iteration and are not configured through the registry. + +The registry is a checked-in file under the crate root, separate from the spec files: the +workload archive (`benchmarks/workloads/`) is downloaded at build time and gitignored, so configs +that belong with the source live here and reference benchmarks by name. + +```json +{ + "10kAdds0CommitsSinceChkpt1V2Chkpt": { + "readMetadataLatest": [ + { "name": "serial", "parallelScan": "disabled" }, + { "name": "parallel2", "parallelScan": { "enabled": { "numThreads": 2 } } } + ] + } +} +``` + +- The file nests read config lists by table directory name then case name + (`{ table: { case: [configs] } }`). +- A read benchmark whose `(table, case)` key is absent from the registry falls back to the built-in + serial config. An explicit but empty config list is rejected at load rather than silently + dropping the benchmark. +- Config `name`s within a list must be unique and non-empty (they form the benchmark name's last + segment). + +Config-field value forms: + +- `parallelScan` (read configs): `"disabled"` or `{ "enabled": { "numThreads": } }` — serde + externally-tagged, so the fieldless variant is a bare string and the parameterized variant a + single-key object. + ## Workload data layout -Each table lives in its own subdirectory under `benchmarks/data/workloads/benchmarks/`: +Each table lives in its own subdirectory under `benchmarks/workloads/benchmarks/`: ``` -benchmarks/data/workloads/ +benchmarks/workloads/ ├── benchmarks/ │ └── / -│ ├── table_info.json # describes the table (name, path, etc.) -│ ├── delta/ # Delta table data (if no explicit table_path) +│ ├── tableInfo.json # describes the table (name, schema, protocol, etc.) +│ ├── delta/ # Delta table data (if no explicit tablePath) │ └── specs/ │ └── .json # one file per benchmark operation └── tests/ # reserved for future test workloads (currently empty) @@ -96,22 +191,28 @@ benchmarks/data/workloads/ ## Loading workloads -Workloads are loaded from `benchmarks/data/workloads.tar.gz`. On first run the tarball is extracted to `benchmarks/data/workloads/` and a `.done` file is written (to `benchmarks/data/workloads/`) to skip re-extraction on subsequent runs. To pick up changes to the tarball, delete the `.done` file. +Workloads are downloaded from the DAT GitHub release and extracted to `benchmarks/workloads/` automatically by `build.rs` when the crate is built. A `.done` marker file is written on success to skip re-downloading on subsequent builds. To force a fresh download, delete `benchmarks/workloads/.done`. + +Workloads are discovered automatically by path. `load_all_workloads()` scans every subdirectory of `benchmarks/workloads/benchmarks/`, loading `tableInfo.json` and every spec file under `specs/`. The subdirectory name becomes the registry table key, the human-readable `name` from `tableInfo.json` becomes the benchmark identifier's first segment, and the spec filename (without extension) becomes the `case_name`. -Workloads are discovered automatically by path. `load_all_workloads()` scans every subdirectory of `benchmarks/data/workloads/benchmarks/`, loading `table_info.json` and every spec file under `specs/`. The spec filename (without extension) becomes the `case_name`. +## Current benchmarking workloads -## Adding a new table +There is no single exhaustive list of all benchmark tables and their contents maintained in this README, as this can change over time. The [built-in tags](#by-tag-bench_tags) section includes a list of table names grouped by tag, but this is non-exhaustive and subject to change. To explore which tables exist and what each one contains, run benchmarks locally and inspect the `tableInfo.json` file in each table's directory under `benchmarks/workloads/benchmarks/`. -To benchmark against a custom Delta table: +## Adding a new table locally + +### Local tables + +To benchmark against a local Delta table: 1. Extract the workload archive if you haven't already — the simplest way is to run any benchmark once, which auto-extracts it: ```bash cargo bench -p delta_kernel_benchmarks --bench workload_bench ``` -2. Create a directory for the new table under `benchmarks/data/workloads/benchmarks/`: +2. Create a directory for the new table under `benchmarks/workloads/benchmarks/`: ``` - benchmarks/data/workloads/benchmarks// - ├── table_info.json # {"name": "", "description": "...", "tags": [...]} + benchmarks/data/workloads/benchmarks// + ├── tableInfo.json # see TableInfo section below for required fields ├── delta/ # Delta table files (_delta_log/, parquet data, etc.) └── specs/ └── .json # one or more spec files describing operations to benchmark @@ -121,23 +222,68 @@ To benchmark against a custom Delta table: cargo bench -p delta_kernel_benchmarks --bench workload_bench "" ``` -If you want to commit this change and add it to the `workloads.tar.gz` archive: +### Remote tables (S3 / UC) + +Remote tables are benchmarked via `KERNEL_BENCH_WORKLOAD_DIR`, which points to a directory of +table configs outside of the workload archive. Each subdirectory has the same layout as local +tables (`tableInfo.json` + `specs/`), but no `delta/` directory is needed. + +There are two types of remote tables, determined by the `tableInfo.json` fields: + +- **S3 tables** — set `tablePath` to the S3 URL (e.g. `s3://bucket/path`). Requires `AWS_*` + env vars for credentials. +- **UC tables** — set `catalogInfo` with a `tableName` field (e.g. `catalog.schema.table`). + Credentials are vended via UC at runtime (`UC_WORKSPACE` / `UC_TOKEN` env vars). The + benchmark harness automatically detects catalog-managed tables (via UC properties) and + uses the appropriate snapshot loading path. + +Example `tableInfo.json` for a UC table: +```json +{ + "name": "my_uc_table", + "description": "A UC-managed table", + "catalogInfo": {"tableName": "catalog.schema.table"}, + "schema": {"type": "struct", "fields": [...]}, + "protocol": {"minReaderVersion": 1, "minWriterVersion": 2}, + "logInfo": {"numAddFiles": 100, "numRemoveFiles": 0, "sizeInBytes": 0, "numCommits": 1, "numActions": 100}, + "properties": {}, + "dataLayout": {}, + "tags": [] +} +``` + +Example: ```bash -cd benchmarks/data/workloads -tar -czf ../workloads.tar.gz . +KERNEL_BENCH_WORKLOAD_DIR=/path/to/my/tables \ + UC_WORKSPACE=https://my-workspace.cloud.databricks.com UC_TOKEN=... AWS_REGION=us-west-2 \ + cargo bench --bench workload_bench ``` -Then commit the updated archive and delete the `.done` file so it is re-extracted on the next run. ## Entities ### `TableInfo` -Deserialized from a `table_info.json` file. Describes a Delta table and includes its name, a human-readable description, a list of tags for filtering, and either an explicit `table_path` (for remote tables) or a local path (`delta/` subdirectory at the same directory level as `table_info.json`). `name`, `description`, and `tags` are all required fields while `table_path` is optional. `tags` is used to filter which tables are loaded via the `BENCH_TAGS` environment variable — see [By tag (`BENCH_TAGS`)](#by-tag-bench_tags). Note that `table_path` is mainly intended for remote tables (e.g. S3), but support for remote tables is not yet implemented; all current workloads are under `delta/` as described. +Deserialized from `tableInfo.json`. Captures a human label (`name`) and `description`, Delta schema and protocol, log statistics (`logInfo`), physical data layout, table properties, and benchmark tags. The `name` field becomes the benchmark identifier's table segment, while the registry table key comes from the table's directory name. See [`workloads/src/models.rs`](../workloads/src/models.rs) for field-level documentation. + +#### Example ```json { - "name": "basic_append", + "name": "myTable", "description": "A basic table with two append writes.", + "schema": {"type": "struct", "fields": [ + {"name": "id", "type": "long", "nullable": true, "metadata": {}} + ]}, + "protocol": {"minReaderVersion": 3, "minWriterVersion": 7, "readerFeatures": [], "writerFeatures": []}, + "logInfo": { + "numAddFiles": 10, + "numRemoveFiles": 0, + "sizeInBytes": 4096, + "numCommits": 2, + "numActions": 12 + }, + "properties": {}, + "dataLayout": {}, "tags": ["base"] } ``` @@ -164,39 +310,59 @@ Or with a specific version: } ``` +With a predicate for data skipping (SQL WHERE clause syntax): + +```json +{ + "type": "read", + "predicate": "id < 500 AND value > 10" +} +``` + +The `predicate` field accepts a SQL WHERE clause expression that is parsed into a kernel `Predicate` and passed to the scan builder. See [`workloads/src/predicate_parser.rs`](../workloads/src/predicate_parser.rs) for the full list of supported SQL features. + Snapshot construction specs: ```json { - "type": "snapshot_construction" + "type": "snapshotConstruction" } ``` Or with a specific version: ```json { - "type": "snapshot_construction", + "type": "snapshotConstruction", "version": 0 } ``` ### `Workload` -The concrete unit of work that gets benchmarked. Assembled when loading workloads by pairing a `Spec` (the operation) with a `TableInfo` (the table) and a `case_name`. A `Spec` file on its own solely describes an operation without context of the table it is performed on; when combined with a table, it becomes a `Workload`. A single table therefore produces multiple workloads, one for each spec file in its `specs/` directory. +The concrete unit of work that gets benchmarked. Assembled when loading workloads by pairing a `Spec` (the operation) with a `TableInfo` (the table, whose directory determines `TableInfo::registry_table_key()`) and a `case_name`. A `Spec` file on its own solely describes an operation without context of the table it is performed on; when combined with a table, it becomes a `Workload`. A single table therefore produces multiple workloads, one for each spec file in its `specs/` directory. ### `ReadConfig` -Specifies runtime parameters for `Read` workloads that are not part of the spec JSON — currently whether to scan serially or in parallel, and how many threads to use. Multiple configs can be applied to the same workload to compare modes. By default all workloads run serial log replay; workloads with sidecar files additionally run parallel configs to benchmark parallel scanning. +Specifies runtime parameters for `Read` workloads that are not part of the spec JSON — currently whether to scan serially or in parallel, and how many threads to use. Multiple configs can be applied to the same workload to compare modes. Which configs run for a given benchmark is determined by the [registry](#registry-bench-registryjson). ### `WorkloadRunner` -Owns all pre-built state for a workload (e.g. a pre-constructed `Snapshot`) so that `execute()` measures only the target operation. Each runner corresponds to one `Workload` plus whatever additional configuration that workload type requires — `Read` workloads take a `ReadConfig`, while `SnapshotConstruction` workloads require no extra configuration. +Owns all pre-built state for a workload (e.g. a pre-constructed `Snapshot`) so that `execute()` measures only the target operation. Read runners also carry the `ReadConfig` selected by the registry. Snapshot-construction runners always construct a new snapshot each iteration. ## Source Layout +The workload spec data types (`TableInfo`, `Spec`, `Workload`, …) live in the shared +[`delta_kernel_workloads`](../workloads/) crate, since the spec types are also used by the +`acceptance` crate. The read harness config types (`ReadConfig`, `ParallelScan`) and the registry +that maps read benchmarks to them are benchmark-specific and live in this crate. + | File | Purpose | |------|---------| -| `src/models.rs` | Data types: `TableInfo`, `Spec`, `Workload`, `ReadConfig`, `ReadOperation` | +| `../workloads/src/models.rs` | Workload spec types: `TableInfo`, `Spec`, `Workload`, `ReadOperation` | +| `../workloads/src/predicate_parser.rs` | SQL WHERE clause to kernel `Predicate` parser | +| `src/registry.rs` | Read harness config types and registry loader: `ReadConfig`, `ParallelScan`, `BenchRegistry` | +| `bench-registry.json` | Checked-in registry mapping read benchmarks to their harness configs | | `src/runners.rs` | `WorkloadRunner` trait and implementations: `ReadMetadataRunner`, `SnapshotConstructionRunner` | -| `src/utils.rs` | Workload loading: extracts the tarball and deserializes all workloads | -| `benches/workload_bench.rs` | Criterion entry point — loads workloads, builds runners, drives benchmarks | +| `src/utils.rs` | Workload loading: deserializes workloads from the extracted data directory | +| `benches/workload_bench.rs` | Criterion entry point — loads workloads + registry, builds runners, drives benchmarks | +| `build.rs` | Downloads and extracts benchmark workloads from the DAT GitHub release at build time | diff --git a/benchmarks/bench-registry.json b/benchmarks/bench-registry.json new file mode 100644 index 0000000000..c13e29338a --- /dev/null +++ b/benchmarks/bench-registry.json @@ -0,0 +1,8 @@ +{ + "10kAdds0CommitsSinceChkpt1V2Chkpt": { + "readMetadataLatest": [ + { "name": "serial", "parallelScan": "disabled" }, + { "name": "parallel2", "parallelScan": { "enabled": { "numThreads": 2 } } } + ] + } +} diff --git a/benchmarks/benches/workload_bench.rs b/benchmarks/benches/workload_bench.rs index 01374a685f..e6f9936510 100644 --- a/benchmarks/benches/workload_bench.rs +++ b/benchmarks/benches/workload_bench.rs @@ -1,30 +1,26 @@ +use std::path::Path; +use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::Arc; -use criterion::measurement::WallTime; -use criterion::{criterion_group, criterion_main, BenchmarkGroup, Criterion}; - -use delta_kernel::engine::default::executor::tokio::TokioBackgroundExecutor; -use delta_kernel::engine::default::DefaultEngine; - -use delta_kernel_benchmarks::models::{ - default_read_configs, ParallelScan, ReadConfig, ReadOperation, Spec, -}; +use criterion::{criterion_group, criterion_main, Criterion}; +use delta_kernel_benchmarks::registry::BenchRegistry; use delta_kernel_benchmarks::runners::{ - create_read_runner, SnapshotConstructionRunner, WorkloadRunner, + benchmark_name, configured_benchmark_name, create_read_runner, SnapshotConstructionRunner, + WorkloadRunner, }; use delta_kernel_benchmarks::utils::load_all_workloads; +use delta_kernel_workloads::models::{ReadOperation, Spec}; +use test_utils::CountingReporter; -fn setup_engine() -> Arc> { - use delta_kernel::object_store::local::LocalFileSystem; - - let store = Arc::new(LocalFileSystem::new()); - let engine = DefaultEngine::builder(store).build(); +// Checked-in registry mapping each benchmark to its harness configs. Lives under the crate root +// (not the gitignored, downloaded `workloads/` dir), so it is loaded relative to +// CARGO_MANIFEST_DIR. +const REGISTRY_PATH: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/bench-registry.json"); - Arc::new(engine) -} - -// Loads all workloads and sets up a shared engine and benchmark group -// For each workload, builds a runner that encapsulates the state (table info, engine, config, etc.) and execution logic +// Loads all workloads and sets up a shared runtime, then registers each as a top-level benchmark. +// For each workload, builds a runner that encapsulates the state (table info, engine, config, etc.) +// and execution logic. After each Criterion timing pass, runs one IO-profiling iteration and +// prints per-call storage and log-replay counts. fn workload_benchmarks(c: &mut Criterion) { let workloads = match load_all_workloads() { Ok(workloads) if !workloads.is_empty() => workloads, @@ -32,61 +28,72 @@ fn workload_benchmarks(c: &mut Criterion) { Err(e) => panic!("Failed to load workloads: {e}"), }; - let engine = setup_engine(); - let mut group = c.benchmark_group("workload_benchmarks"); + let registry = BenchRegistry::load_from_path(Path::new(REGISTRY_PATH)) + .expect("Failed to load bench-registry.json"); + registry + .validate(&workloads) + .expect("bench-registry.json must match the loaded workload types"); + + let reporter = Arc::new(CountingReporter::new()); + let runtime = Arc::new(tokio::runtime::Runtime::new().expect("Failed to create tokio runtime")); for workload in &workloads { + let case_name = &workload.case_name; match &workload.spec { Spec::Read(read_spec) => { + let configs = registry + .read_configs(workload) + .expect("loaded workload must have a registry table key"); for operation in [ReadOperation::ReadMetadata] { - for config in build_read_configs(&workload.table_info.name) { - let runner = create_read_runner( + for config in &configs { + let name = configured_benchmark_name( &workload.table_info, - &workload.case_name, + case_name, + &config.name, + ); + let runner = create_read_runner( + name, read_spec, operation, - config, - engine.clone(), + config.clone(), + &workload.table_info, + runtime.clone(), ) .expect("Failed to create read runner"); - run_benchmark(&mut group, runner.as_ref()); + run_benchmark(c, runner.as_ref(), &reporter); } } } Spec::SnapshotConstruction(snapshot_construction_spec) => { + let name = benchmark_name(&workload.table_info, case_name); let runner = SnapshotConstructionRunner::setup( - &workload.table_info, - &workload.case_name, + name, snapshot_construction_spec, - engine.clone(), + &workload.table_info, + runtime.clone(), ) .expect("Failed to create snapshot construction runner"); - run_benchmark(&mut group, &runner); + run_benchmark(c, &runner, &reporter); } } } - - group.finish(); } -// Registers a workload with Criterion under the given group and benchmarks its `execute()` function -fn run_benchmark(group: &mut BenchmarkGroup, runner: &dyn WorkloadRunner) { - group.bench_function(runner.name(), |b| { +// Registers a workload with Criterion and benchmarks its `execute()` function. +// After timing completes, runs one IO-profiling iteration and prints per-call storage and +// log-replay counts. The IO profile is skipped entirely when Criterion filters out the benchmark, +// since Criterion never calls the closure for filtered benchmarks. +fn run_benchmark(c: &mut Criterion, runner: &dyn WorkloadRunner, reporter: &CountingReporter) { + let bench_ran = AtomicBool::new(false); + c.bench_function(runner.name(), |b| { + bench_ran.store(true, Ordering::Relaxed); b.iter(|| runner.execute().expect("Benchmark execution failed")) }); -} - -fn build_read_configs(table_name: &str) -> Vec { - // Choose which benchmark configurations to run for a given table - // TODO: This function will take in table info to choose the appropriate configs for a given table - let mut configs = default_read_configs(); - if table_name.contains("v2_checkpoint") { - configs.push(ReadConfig { - name: "parallel_2".into(), - parallel_scan: ParallelScan::Enabled { num_threads: 2 }, - }); + if bench_ran.load(Ordering::Relaxed) { + reporter.reset(); + runner.execute().expect("IO profiling iteration failed"); + reporter.print_summary(runner.name()); } - configs } criterion_group!(benches, workload_benchmarks); diff --git a/benchmarks/build.rs b/benchmarks/build.rs new file mode 100644 index 0000000000..08850177a1 --- /dev/null +++ b/benchmarks/build.rs @@ -0,0 +1,108 @@ +//! Build script for benchmark workloads + +use std::env; +use std::fs::File; +use std::io::{BufReader, BufWriter, Read, Write}; +use std::path::{Path, PathBuf}; +use std::time::Duration; + +use flate2::read::GzDecoder; +use sha2::{Digest, Sha256}; +use tar::Archive; +use ureq::tls::{RootCerts, TlsConfig, TlsProvider}; +use ureq::{Agent, Proxy}; + +const VERSION: &str = "0.0.5-preview"; // release tag +const WORKLOADS_VERSION: &str = "0.0.5"; // version in filename +const WORKLOAD_CHECKSUM: &str = "ddac5359eca42e7ec65b4a7cfc6f4bc1d629d9c204ba714ec15e4ae830c37ba2"; // benchmarks checksum +const DOWNLOAD_ATTEMPTS: usize = 3; + +fn main() { + let manifest_dir = env::var("CARGO_MANIFEST_DIR").expect("CARGO_MANIFEST_DIR not set"); + let output_dir = PathBuf::from(manifest_dir).join("workloads"); + let done_marker = output_dir.join(".done"); + + println!("cargo::rerun-if-changed={}", done_marker.display()); + + if done_marker.exists() { + return; + } + + let tarball_data = download_workloads(); + extract_tarball(tarball_data, &output_dir); + write_done_file(&done_marker); +} + +fn download_workloads() -> Vec { + let tarball_url = format!( + "https://github.com/delta-incubator/dat/releases/download/v{VERSION}/v{WORKLOADS_VERSION}_benchmark_workloads.tar.gz" + ); + download_tarball(&tarball_url, WORKLOAD_CHECKSUM) +} + +fn download_tarball(url: &str, expected_checksum: &str) -> Vec { + let agent = build_agent(); + let mut attempt = 1; + + loop { + match download(&agent, url) { + Ok(data) => { + verify_checksum(&data, expected_checksum); + return data; + } + Err(error) if attempt < DOWNLOAD_ATTEMPTS => { + eprintln!("Download attempt {attempt} failed: {error}. Retrying..."); + std::thread::sleep(Duration::from_secs(1)); + attempt += 1; + } + Err(error) => panic!("Failed to download {url}: {error}"), + } + } +} + +fn download(agent: &Agent, url: &str) -> Result, Box> { + let response = agent.get(url).call()?; + let mut data: Vec = Vec::new(); + response.into_body().as_reader().read_to_end(&mut data)?; + + Ok(data) +} + +/// Panic unless the SHA-256 of `data` equals `expected` (lowercase hex). Called before any +/// extraction, so a download whose digest doesn't match the pinned value fails the build instead +/// of being unpacked to disk. +fn verify_checksum(data: &[u8], expected: &str) { + let actual: String = Sha256::digest(data) + .iter() + .map(|b| format!("{b:02x}")) + .collect(); + if actual != expected { + panic!("tarball checksum mismatch: expected {expected}, got {actual}"); + } +} + +fn build_agent() -> Agent { + let tls_config = TlsConfig::builder() + .provider(TlsProvider::NativeTls) + .root_certs(RootCerts::PlatformVerifier) + .build(); + let config = Agent::config_builder() + .tls_config(tls_config) + .proxy(Proxy::try_from_env()); + Agent::new_with_config(config.build()) +} + +fn extract_tarball(tarball_data: Vec, output_dir: &Path) { + let decoder = GzDecoder::new(BufReader::new(&tarball_data[..])); + let mut archive = Archive::new(decoder); + std::fs::create_dir_all(output_dir).expect("Failed to create output directory"); + archive + .unpack(output_dir) + .expect("Failed to unpack tarball"); +} + +fn write_done_file(done_marker: &Path) { + let mut done_file = + BufWriter::new(File::create(done_marker).expect("Failed to create .done file")); + write!(done_file, "done").expect("Failed to write .done file"); +} diff --git a/benchmarks/ci/parse_critcmp.py b/benchmarks/ci/parse_critcmp.py new file mode 100644 index 0000000000..af5e4d685e --- /dev/null +++ b/benchmarks/ci/parse_critcmp.py @@ -0,0 +1,233 @@ +#!/usr/bin/env python3 +""" +Parse critcmp output and format it as a GitHub-flavoured Markdown comment. + +Reads the output of `critcmp base changes` from stdin and writes: + 1. The comment's header line, ending with a pass/fail verdict that mirrors + the regression-gate outcome (BENCH_IGNORE_FAILURE=true renders the + label-override variant). + 2. A one-line summary of how many benchmarks fell into each tier, ordered + fastest to slowest. + 3. A `
` block (closed by default) containing the full + per-benchmark table with columns: Test | Change | Base | PR. + 4. The tier legend as a small-text footer line. + +Each benchmark is bucketed by its displayed PR/base multiplier into a tier with +a marker emoji (see the threshold constants). When any benchmark lands in the +slowest tier, the run records a regression in the file named by +BENCH_REGRESSION_FILE (if set) so the workflow can fail the job. + +Usage: + critcmp base changes | python3 benchmarks/ci/parse_critcmp.py +""" +import os +import re +import sys + +# Tiers compare the 2-decimal multiplier the table displays ("1.15x slower", +# "1.15x faster"), so a row's marker always matches its Change cell. +ROCKET_THRESHOLD = 1.15 +GRAY_THRESHOLD = 1.03 +FAIL_THRESHOLD = 1.15 + +# |ratio - 1| within this renders as "1.00x" with no faster/slower suffix +# (half of the 2-decimal display step). +NEUTRAL_THRESHOLD = 0.005 + +# Tier markers for the Change cell, fastest to slowest. +ROCKET = '🚀' # at least 1.15x faster +GREEN_CHECK = '✅' # faster or unchanged +GRAY_CHECK = '☑️' # at most 1.03x slower +CONSTRUCTION = '🚧' # between 1.03x and 1.15x slower +RED_X = '❌' # at least 1.15x slower (fails the job) + +# Fastest to slowest; drives both the summary order and the legend. +TIERS = [ROCKET, GREEN_CHECK, GRAY_CHECK, CONSTRUCTION, RED_X] + +def to_ms(value, units): + """Convert a critcmp duration to milliseconds. + + Matches exactly the units critcmp can emit (see `time()` in critcmp's + output.rs): ns, µs (U+00B5), ms, s. An unrecognized unit means the critcmp + output format changed; raise so the run fails loudly instead of rendering a + silently wrong table. + """ + u = units.strip() + if u == 's': + return value * 1e3 + if u == 'ms': + return value + if u == 'µs': + return value / 1e3 + if u == 'ns': + return value / 1e6 + raise ValueError(f'unrecognized critcmp time unit: {units!r}') + +def parse_duration(s): + m = re.match(r'([0-9.]+)±([0-9.]+)(.+)', s.strip()) + if not m: + return None + return float(m.group(1)), float(m.group(2)), m.group(3).strip() + +def parse_rows(lines): + """Parse critcmp stdout into a list of row dicts. + + Each row dict contains: + name: sanitized benchmark name (no backticks/pipes), unwrapped + base_display: base duration string or 'N/A' + chg_display: changes duration string or 'N/A' + ratio: chg_ms / base_ms, or None if either side is missing/zero + """ + rows = [] + for line in lines[2:]: # skip critcmp header rows + if not line.strip(): + continue + # critcmp columns (split on 2+ spaces): + # with throughput: name, baseFactor, baseDuration, baseBandwidth, changesFactor, changesDuration, changesBandwidth + # without throughput: name, baseFactor, baseDuration, changesFactor, changesDuration + # Locate duration fields by the presence of "±" rather than hardcoding indices, + # so the script works correctly regardless of whether bandwidth columns are present. + fields = re.split(r' +', line) + name = fields[0].strip() if fields else '' + dur_fields = [f.strip() for f in fields[1:] if '±' in f] + base_dur_str = dur_fields[0] if len(dur_fields) > 0 else None + chg_dur_str = dur_fields[1] if len(dur_fields) > 1 else None + + if not name and not base_dur_str and not chg_dur_str: + continue + + # N/A when a benchmark only exists in one of the two runs (added or removed). + base_display = base_dur_str or 'N/A' + chg_display = chg_dur_str or 'N/A' + ratio = None + + if base_dur_str and chg_dur_str: + base_p = parse_duration(base_dur_str) + chg_p = parse_duration(chg_dur_str) + if base_p and chg_p: + base_ms = to_ms(base_p[0], base_p[2]) + chg_ms = to_ms(chg_p[0], chg_p[2]) + + # Float-equality on zero is safe here: to_ms only multiplies/divides + # by powers of ten, so a zero output strictly implies a zero input. + # Do NOT replace with an epsilon -- that would tag legitimately fast + # benches (sub-nanosecond rounding) as N/A. + if base_ms != 0 and chg_ms != 0: + ratio = chg_ms / base_ms + + rows.append({ + 'name': name, + 'base_display': base_display, + 'chg_display': chg_display, + 'ratio': ratio, + }) + return rows + +def format_difference(ratio): + """Render a ratio as e.g. '1.00x', '1.50x slower', or '2.00x faster'.""" + if ratio is None: + return 'N/A' + if abs(ratio - 1.0) < NEUTRAL_THRESHOLD: + return '1.00x' + if ratio > 1: + return f'{ratio:.2f}x slower' + return f'{1.0 / ratio:.2f}x faster' + +def change_emoji(ratio): + """Pick the tier marker for the Change cell from the displayed 2-decimal + multiplier. Empty string for N/A rows.""" + if ratio is None: + return '' + if ratio < 1.0: + return ROCKET if round(1.0 / ratio, 2) >= ROCKET_THRESHOLD else GREEN_CHECK + shown = round(ratio, 2) + if shown <= 1.0: + return GREEN_CHECK + if shown <= GRAY_THRESHOLD: + return GRAY_CHECK + if shown < FAIL_THRESHOLD: + return CONSTRUCTION + return RED_X + +def render_verdict(regressed, ignored): + """Render the pass/fail verdict shown in the comment's header line, + mirroring the job's regression-gate outcome.""" + mult = f'{FAIL_THRESHOLD:.2f}x' + if not regressed: + return '✅ Pass' + if ignored: + return f'⚠️ Pass (≥{mult} slowdown ignored)' + return f'❌ Fail (a benchmark is ≥{mult} slower)' + +def render_summary(rows): + """Render the per-tier counts on one line, fastest to slowest. N/A rows + (added/removed benchmarks) are appended only when present.""" + counts = {tier: 0 for tier in TIERS} + na = 0 + for r in rows: + emoji = change_emoji(r['ratio']) + if emoji: + counts[emoji] += 1 + else: + na += 1 + parts = [f"{tier} {counts[tier]}" for tier in TIERS] + if na: + parts.append(f"N/A {na}") + return "**Summary:** " + "  ·  ".join(parts) + +def render_legend(): + """Render the tier legend as a small-text line for the comment footer.""" + fail = f'{FAIL_THRESHOLD:.2f}x' + gray = f'{GRAY_THRESHOLD:.2f}x' + rocket = f'{ROCKET_THRESHOLD:.2f}x' + return ( + f"Legend: {ROCKET} ≥{rocket} faster  · " + f"{GREEN_CHECK} faster or unchanged  · " + f"{GRAY_CHECK} ≤{gray} slower  · " + f"{CONSTRUCTION} {gray}-{fail} slower  · " + f"{RED_X} ≥{fail} slower" + ) + +def render_table(rows): + """Render the per-benchmark table wrapped in a closed-by-default
block.""" + out = [] + out.append("
") + out.append(f"Per-benchmark results ({len(rows)} rows)") + out.append("") + out.append("| Test | Change | Base | PR |") + out.append("|------|--------|--------------|------------------|") + for r in rows: + name_cell = f"`{r['name']}`" if r['name'] else '' + difference = format_difference(r['ratio']) + emoji = change_emoji(r['ratio']) + # Non-breaking spaces keep the marker and ratio on one line so the + # Change cell renders without wrapping. + change_cell = f"{emoji} {difference}".strip().replace(" ", " ") + out.append(f"| {name_cell} | {change_cell} | {r['base_display']} | {r['chg_display']} |") + out.append("") + out.append("
") + return "\n".join(out) + +def main(): + lines = sys.stdin.read().splitlines() + rows = parse_rows(lines) + regressed = any(change_emoji(r['ratio']) == RED_X for r in rows) + ignored = os.environ.get('BENCH_IGNORE_FAILURE') == 'true' + + print(f'## Benchmark results: {render_verdict(regressed, ignored)}') + print("") + print(render_summary(rows)) + print("") + print(render_table(rows)) + print("") + print(render_legend()) + + # Record whether any benchmark crossed FAIL_THRESHOLD so the workflow can + # fail the job (unless overridden by the ignore-benchmark-failure label). + flag_path = os.environ.get('BENCH_REGRESSION_FILE') + if flag_path: + with open(flag_path, 'w') as f: + f.write('true' if regressed else 'false') + +if __name__ == "__main__": + main() diff --git a/benchmarks/ci/run-benchmarks.sh b/benchmarks/ci/run-benchmarks.sh new file mode 100644 index 0000000000..946e231b9d --- /dev/null +++ b/benchmarks/ci/run-benchmarks.sh @@ -0,0 +1,152 @@ +#!/usr/bin/env bash +# Benchmark comparison script for pull requests. +# +# Called by .github/workflows/benchmark.yml (run-benchmark job) after the repo +# has been checked out at the PR's merge commit (refs/pull//merge -- PR head +# merged into base). Writes the formatted markdown comparison +# to /tmp/bench-comment.md; the companion benchmark-post-comment.yml workflow +# downloads it as an artifact and posts the PR comment in base-branch context. +# +# Expects the following environment variables: +# +# BASE_REF - base branch ref (e.g. "main") +# HEAD_SHA - full SHA of the PR head commit +# COMMENT - (optional) /bench PR comment body. Unset under the +# pull_request auto-trigger path; set to the comment +# body under the issue_comment path. +# TRIGGER - (optional) human-readable label for what kicked off the run +# ("auto-push" or "/bench"). Used in the comment header. +# BENCH_IGNORE_FAILURE - (optional) "true" when the PR carries the +# ignore-benchmark-failure label. Passed through to +# parse_critcmp.py so the comment's verdict line says the +# regression gate is overridden. + +set -euo pipefail +shopt -s extglob + +# --------------------------------------------------------------------------- +# 1. Parse the /bench comment +# Syntax: /bench [--tags ] [--filter ] +# --tags sets BENCH_TAGS (comma-separated tag list); defaults to "base" +# when the comment is just /bench (or COMMENT is unset, i.e. +# the auto-trigger path) +# --filter Criterion name regex passed as a positional arg to cargo bench +# --------------------------------------------------------------------------- + +# Auto-trigger path leaves COMMENT unset; treat that the same as a bare /bench. +COMMENT="${COMMENT:-}" + +ARGS="${COMMENT#/bench}" +ARGS="${ARGS##+( )}" + +TAGS="" +FILTER="" + +if [[ -z "$ARGS" ]]; then + # Bare /bench with no args: default to the "base" tag + TAGS="base" +else + # Normalize: strip /bench prefix, collapse all whitespace (including newlines) + # to spaces, then strip to a safe allowlist before parsing + ARGS=$(printf '%s' "$ARGS" | tr '\n\r\t' ' ' | tr -s ' ' | tr -cd 'a-zA-Z0-9,_./|*+?()[]^$ -') + ARGS="${ARGS## }" # strip leading space + ARGS="${ARGS%% }" # strip trailing space + + read -ra TOKENS <<< "$ARGS" + i=0 + while [[ $i -lt ${#TOKENS[@]} ]]; do + case "${TOKENS[$i]}" in + --tags) i=$((i + 1)); TAGS="${TOKENS[$i]:-}" ;; + --filter) i=$((i + 1)); FILTER="${TOKENS[$i]:-}" ;; + *) echo "Unknown token: '${TOKENS[$i]}'" >&2; exit 1 ;; + esac + i=$((i + 1)) + done +fi + +# Default: if nothing was parsed, run with BENCH_TAGS=base +if [[ -z "$TAGS" && -z "$FILTER" ]]; then + TAGS="base" +fi + +echo "Parsed tags: ${TAGS:-}" +echo "Parsed filter: ${FILTER:-}" + +[[ -n "$TAGS" ]] && export BENCH_TAGS="$TAGS" + +# --------------------------------------------------------------------------- +# 2. Log the runner environment +# GitHub-hosted runners draw from a heterogeneous pool (varying CPU model, +# cache size, neighbor load). Logging hardware and load per run gives us the +# data to attribute run-to-run variance to the machine vs. the code. +# --------------------------------------------------------------------------- +echo "=== Runner environment ===" +lscpu | grep -E '^(Model name|CPU\(s\)|Thread|CPU( max| min)? MHz|L[123][a-z]* cache)' || true +echo "loadavg: $(cat /proc/loadavg)" +echo "mem: $(free -m | awk '/^Mem:/ {print $2 " MB total, " $7 " MB available"}')" +echo "===========================" + +# --------------------------------------------------------------------------- +# 3. Benchmark the integration commit (PR head merged into base) +# --------------------------------------------------------------------------- +# HEAD is the merge commit checked out by the workflow; capture it so step 5 can +# restore this tree after the base checkout below. +MERGE_SHA=$(git rev-parse HEAD) +(cd benchmarks && cargo bench --locked --bench workload_bench -- --save-baseline changes "$FILTER") + +# --------------------------------------------------------------------------- +# 4. Switch to the base branch and benchmark it +# The benchmarks/target/ directory is not tracked by git, so the +# "changes" baseline files are preserved across the branch switch. +# --------------------------------------------------------------------------- +git fetch origin -- "$BASE_REF" +git checkout FETCH_HEAD +(cd benchmarks && cargo bench --locked --bench workload_bench -- --save-baseline base "$FILTER") + +# --------------------------------------------------------------------------- +# 5. Compare baselines with critcmp and format as a markdown comment +# (per-tier summary line + collapsed per-benchmark table; see +# benchmarks/ci/parse_critcmp.py for the format and tier thresholds). +# - Parses actual duration values (not rank factors) to compute a ratio +# --------------------------------------------------------------------------- +# Step 4 left the working tree on the base branch, so parse_critcmp.py on disk +# is the base version. Restore the merge tree so the comment is formatted by the +# PR's own formatter (a formatter change is then exercised on the PR that +# introduces it). The raw PR-head SHA is not reachable in the shallow merge-ref +# checkout, so restore the merge commit captured in step 3. Only tracked sources +# move; the `base`/`changes` criterion baselines live in gitignored +# benchmarks/target/ and survive the checkout. +git checkout "$MERGE_SHA" +# Use `critcmp` to compare the criterion output for `base` and `changes`. We use `critcmp` instead of manually +# parsing criterion outputs because criterion may update its output format. By using `critcmp`, we inherit all +# updated criterion output parsing. +# +# parse_critcmp.py records whether any benchmark regressed past its fail +# threshold in this file; benchmark.yml's gate step reads it to fail the job. +export BENCH_REGRESSION_FILE=/tmp/bench-regression.txt +COMPARISON=$((cd benchmarks && critcmp base changes) | python3 benchmarks/ci/parse_critcmp.py) + +# --------------------------------------------------------------------------- +# 6. Write results to /tmp/bench-comment.md +# benchmark.yml uploads this as an artifact; benchmark-post-comment.yml +# downloads it and posts the PR comment in base-branch context. +# --------------------------------------------------------------------------- +SHORT_SHA="${HEAD_SHA:0:7}" + +# Metadata footer shows commit + what fired this run + the active tags/filter + +# when the comment was last refreshed, so a reviewer can tell at a glance which +# configuration produced the displayed numbers (the same comment is reused +# across auto-trigger and /bench runs). +SUMMARY="Commit: \`${SHORT_SHA}\` · Trigger: ${TRIGGER:-auto-push}" +[[ -n "$TAGS" ]] && SUMMARY+=" · Tags: \`${TAGS}\`" +[[ -n "$FILTER" ]] && SUMMARY+=" · Filter: \`${FILTER}\`" +SUMMARY+=" · Updated: $(TZ=America/Los_Angeles date '+%Y-%m-%d %H:%M %Z')" + +# Leading marker is an HTML comment, invisible to readers; the post-comment +# job uses it as a stable identifier for find-and-update so each push reuses +# the same comment instead of stacking new ones. +{ + echo "" + echo "$COMPARISON" + echo "${SUMMARY}" +} > /tmp/bench-comment.md diff --git a/benchmarks/data/workloads.tar.gz b/benchmarks/data/workloads.tar.gz deleted file mode 100644 index f3726009a6..0000000000 Binary files a/benchmarks/data/workloads.tar.gz and /dev/null differ diff --git a/benchmarks/src/lib.rs b/benchmarks/src/lib.rs index 0f06f88ded..9eda3a8a94 100644 --- a/benchmarks/src/lib.rs +++ b/benchmarks/src/lib.rs @@ -1,3 +1,3 @@ -pub mod models; +pub mod registry; pub mod runners; pub mod utils; diff --git a/benchmarks/src/models.rs b/benchmarks/src/models.rs deleted file mode 100644 index 544b89cc49..0000000000 --- a/benchmarks/src/models.rs +++ /dev/null @@ -1,230 +0,0 @@ -//! Data models for workload specifications - -use serde::Deserialize; -use url::Url; - -use std::path::{Path, PathBuf}; - -/// ReadConfig represents a specific configuration for a read operation -/// A config represents configurations for a specific benchmark that aren't specified in the spec JSON file -#[derive(Clone, Debug)] -pub struct ReadConfig { - pub name: String, - pub parallel_scan: ParallelScan, -} - -/// Provides a default set of read configs for a given table, read spec, and operation -pub fn default_read_configs() -> Vec { - vec![ReadConfig { - name: "serial".into(), - parallel_scan: ParallelScan::Disabled, - }] -} - -#[derive(Clone, Debug)] -pub enum ParallelScan { - Disabled, - Enabled { num_threads: usize }, -} - -/// Table info JSON files are located at the root of each table directory -#[derive(Clone, Debug, Deserialize)] -pub struct TableInfo { - pub name: String, // Table name used for identifying the table - pub description: String, // Human-readable description of the table - pub table_path: Option, // URL to the table (for remote tables); also used to override the default local table path - pub tags: Vec, // Tags for filtering workloads - #[serde(skip, default)] - pub table_info_dir: PathBuf, // Path to the directory containing the table info JSON file -} - -impl TableInfo { - pub fn resolved_table_root(&self) -> Url { - self.table_path.clone().unwrap_or_else(|| { - // If table path is not provided, assume that the Delta table is in a delta/ subdirectory at the same level as table_info.json - Url::from_file_path(self.table_info_dir.join("delta")) - .expect("table_info_dir must be an absolute path") - }) - } - - pub fn from_json_path>(path: P) -> Result { - let content = std::fs::read_to_string(path.as_ref()).map_err(serde_json::Error::io)?; - let mut table_info: TableInfo = serde_json::from_str(&content)?; - //Stores the parent directory of the table info JSON file - if let Some(parent) = path.as_ref().parent() { - table_info.table_info_dir = parent.to_path_buf(); - } - Ok(table_info) - } -} - -/// Spec defines the operation performed on a table - defines what operation at what version (e.g. read at version 0) -/// There will be multiple specs for a given table -#[derive(Clone, Debug, Deserialize)] -#[serde(tag = "type", rename_all = "snake_case")] -pub enum Spec { - Read(ReadSpec), - SnapshotConstruction(SnapshotConstructionSpec), -} - -#[derive(Clone, Debug, Deserialize)] -pub struct ReadSpec { - pub version: Option, // If version is None, read at latest version -} - -impl ReadSpec { - pub fn as_str(&self) -> &str { - "read" - } -} - -#[derive(Clone, Debug, Deserialize)] -pub struct SnapshotConstructionSpec { - pub version: Option, // If version is None, read at latest version -} - -impl SnapshotConstructionSpec { - pub fn as_str(&self) -> &str { - "snapshot_construction" - } -} - -impl Spec { - pub fn as_str(&self) -> &str { - match self { - Spec::Read(read_spec) => read_spec.as_str(), - Spec::SnapshotConstruction(snapshot_construction_spec) => { - snapshot_construction_spec.as_str() - } - } - } - - pub fn from_json_path>(path: P) -> Result { - let content = std::fs::read_to_string(path.as_ref()).map_err(serde_json::Error::io)?; - let spec: Spec = serde_json::from_str(&content)?; - Ok(spec) - } -} - -/// For Read specs, we will either run a read data operation or a read metadata operation -#[derive(Clone, Copy, Debug)] -pub enum ReadOperation { - ReadData, - ReadMetadata, -} - -impl ReadOperation { - pub fn as_str(&self) -> &str { - match self { - ReadOperation::ReadData => "read_data", - ReadOperation::ReadMetadata => "read_metadata", - } - } -} - -/// Partial workload specification loaded from JSON - table, case name, and spec only -#[derive(Clone, Debug)] -pub struct Workload { - pub table_info: TableInfo, - pub case_name: String, // Name of the spec JSON file - pub spec: Spec, -} - -#[cfg(test)] -mod tests { - use super::*; - use rstest::rstest; - - #[rstest] - #[case( - r#"{"name": "full_table", "description": "All fields specified", "table_path": "s3://bucket/full_table", "tags": ["base"]}"#, - "full_table", - "All fields specified", - Some(Url::parse("s3://bucket/full_table").unwrap()), - &["base"] - )] - #[case( - r#"{"name": "no_path_table", "description": "No path specified", "tags": ["base"]}"#, - "no_path_table", - "No path specified", - None, - &["base"] - )] - #[case( - r#"{"name": "extra_fields_table", "description": "Has extra fields", "extra_field": "should be ignored", "tags": ["base"]}"#, - "extra_fields_table", - "Has extra fields", - None, - &["base"] - )] - fn test_deserialize_table_info( - #[case] json: &str, - #[case] expected_name: &str, - #[case] expected_description: &str, - #[case] expected_table_path: Option, - #[case] expected_tags: &[&str], - ) { - let table_info: TableInfo = - serde_json::from_str(json).expect("Failed to deserialize table info"); - - assert_eq!(table_info.name, expected_name); - assert_eq!(table_info.description, expected_description); - assert_eq!(table_info.table_path, expected_table_path); - let expected_tags: Vec = expected_tags.iter().map(|s| s.to_string()).collect(); - assert_eq!(table_info.tags, expected_tags); - } - - #[rstest] - #[case( - r#"{"description": "A table missing the required name field"}"#, - "missing field" - )] - fn test_deserialize_table_info_errors(#[case] json: &str, #[case] expected_msg: &str) { - let error = serde_json::from_str::(json).unwrap_err(); - assert!(error.to_string().contains(expected_msg)); - } - - #[rstest] - #[case(r#"{"type": "read", "version": 5}"#, "read", Some(5))] - #[case(r#"{"type": "read"}"#, "read", None)] - #[case( - r#"{"type": "read", "version": 7, "extra_field": "should be ignored"}"#, - "read", - Some(7) - )] - #[case( - r#"{"type": "snapshot_construction", "version": 5}"#, - "snapshot_construction", - Some(5) - )] - #[case(r#"{"type": "snapshot_construction"}"#, "snapshot_construction", None)] - #[case( - r#"{"type": "snapshot_construction", "version": 7, "extra_field": "should be ignored"}"#, - "snapshot_construction", - Some(7) - )] - fn test_deserialize_spec( - #[case] json: &str, - #[case] expected_type: &str, - #[case] expected_version: Option, - ) { - let spec: Spec = serde_json::from_str(json).expect("Failed to deserialize spec"); - assert_eq!(spec.as_str(), expected_type); - let version = match &spec { - Spec::Read(read_spec) => read_spec.version, - Spec::SnapshotConstruction(snapshot_construction_spec) => { - snapshot_construction_spec.version - } - }; - - assert_eq!(version, expected_version); - } - - #[rstest] - #[case(r#"{"version": 10}"#, "missing field")] - #[case(r#"{"type": "write", "version": 3}"#, "unknown variant")] - fn test_deserialize_spec_errors(#[case] json: &str, #[case] expected_msg: &str) { - let error = serde_json::from_str::(json).unwrap_err(); - assert!(error.to_string().contains(expected_msg)); - } -} diff --git a/benchmarks/src/registry.rs b/benchmarks/src/registry.rs new file mode 100644 index 0000000000..6b4c4f8c30 --- /dev/null +++ b/benchmarks/src/registry.rs @@ -0,0 +1,314 @@ +//! Benchmark harness-config registry. +//! +//! Defines benchmark harness parameters and maps workloads to the configurations used to run them. +//! These parameters control how a benchmark runs rather than what operation it performs. + +use std::collections::{HashMap, HashSet}; +use std::path::Path; + +use delta_kernel_workloads::models::{Spec, Workload}; +use serde::Deserialize; + +// === Harness configs === + +/// A read-operation config: benchmark parameters not carried in the spec JSON. Deserialized +/// directly from a `bench-registry.json` entry. +#[derive(Clone, Debug, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +pub struct ReadConfig { + pub name: String, + pub parallel_scan: ParallelScan, +} + +/// The built-in default read configs (a single serial config). +pub fn default_read_configs() -> Vec { + vec![ReadConfig { + name: "serial".into(), + parallel_scan: ParallelScan::Disabled, + }] +} + +/// Scan parallelism configuration. +/// +/// In registry JSON, disabled scans use the string `"disabled"`. Enabled scans use an object that +/// specifies the thread count: `{ "enabled": { "numThreads": } }`. +#[derive(Clone, Debug, PartialEq, Eq, Deserialize)] +#[serde(rename_all = "camelCase", rename_all_fields = "camelCase")] +pub enum ParallelScan { + Disabled, + Enabled { num_threads: usize }, +} + +// === Registry === + +/// Error type for registry loading and lookup. +pub type RegistryResult = Result>; + +/// Read harness configs keyed by table directory name and workload spec filename. +/// +/// For example, this entry runs `readMetadataLatest` against the table in the +/// `10kAdds0CommitsSinceChkpt1V2Chkpt` directory once serially and once with two scan threads: +/// +/// ```json +/// { +/// "10kAdds0CommitsSinceChkpt1V2Chkpt": { +/// "readMetadataLatest": [ +/// { "name": "serial", "parallelScan": "disabled" }, +/// { "name": "parallel2", "parallelScan": { "enabled": { "numThreads": 2 } } } +/// ] +/// } +/// } +/// ``` +/// +/// Each configuration creates a separate Criterion benchmark whose final name segment is the +/// configuration `name`. Read workloads absent from the registry use the built-in serial config. +#[derive(Clone, Debug, Default, Deserialize)] +#[serde(transparent)] +pub struct BenchRegistry { + tables: HashMap>>, +} + +impl BenchRegistry { + /// Loads a registry from `path`. + /// + /// Returns an error when the file cannot be read or deserialized. Call [`Self::validate`] + /// before using the registry. + pub fn load_from_path(path: &Path) -> RegistryResult { + let content = std::fs::read_to_string(path) + .map_err(|e| format!("reading registry {}: {e}", path.display()))?; + serde_json::from_str(&content) + .map_err(|e| format!("parsing registry {}: {e}", path.display()).into()) + } + + /// Returns the registry configs for a read workload, or the built-in serial config when the + /// workload is not registered. + /// + /// Returns an error when the workload's table directory has no valid registry key. + pub fn read_configs(&self, workload: &Workload) -> RegistryResult> { + let table = workload.table_info.registry_table_key().ok_or_else(|| { + format!( + "could not derive a registry table key for workload '{}'", + workload.case_name + ) + })?; + Ok(self + .lookup(table, &workload.case_name) + .map_or_else(default_read_configs, ToOwned::to_owned)) + } + + /// Validates the registry structure and its relationship to `workloads`. + /// + /// Only registry entries matching a workload in `workloads` receive workload-type validation, + /// allowing callers to validate a tag-filtered workload set. Returns an error for empty config + /// lists, duplicate config names, invalid workload registry keys, or read configs targeting a + /// non-read workload. + pub fn validate(&self, workloads: &[Workload]) -> RegistryResult<()> { + for (table, cases) in &self.tables { + for (case, configs) in cases { + let key = format!("{table}/{case}"); + if configs.is_empty() { + return Err(format!("registry entry '{key}' has an empty config list").into()); + } + let names: HashSet<_> = configs.iter().map(|config| config.name.as_str()).collect(); + if names.len() != configs.len() { + return Err(format!("registry entry '{key}' has duplicate config names").into()); + } + } + } + + for workload in workloads { + let table = workload.table_info.registry_table_key().ok_or_else(|| { + format!( + "could not derive a registry table key for workload '{}'", + workload.case_name + ) + })?; + if self.lookup(table, &workload.case_name).is_some() { + if let Spec::SnapshotConstruction(_) = &workload.spec { + return Err(format!( + "registry entry '{table}/{}' provides read configs for a {} workload", + workload.case_name, + workload.spec.as_str() + ) + .into()); + } + } + } + Ok(()) + } + + /// All `(table, case)` keys in the registry. Lets the harness cross-check every entry against + /// the loaded workloads so a stale/typo'd key (which would otherwise silently fall back to + /// defaults) is caught. + pub fn keys(&self) -> impl Iterator { + self.tables.iter().flat_map(|(table, cases)| { + cases + .keys() + .map(move |case| (table.as_str(), case.as_str())) + }) + } + + /// Looks up the config list for `"{table}/{case}"`, if the registry lists it. + fn lookup(&self, table: &str, case: &str) -> Option<&[ReadConfig]> { + self.tables + .get(table) + .and_then(|cases| cases.get(case)) + .map(Vec::as_slice) + } +} + +#[cfg(test)] +mod tests { + use std::path::PathBuf; + + use delta_kernel_workloads::models::{Spec, TableInfo, Workload}; + + use super::*; + + fn registry_from_str(json: &str) -> BenchRegistry { + let registry: BenchRegistry = serde_json::from_str(json).expect("failed to parse registry"); + registry.validate(&[]).expect("registry should validate"); + registry + } + + fn read_workload(table: &str, case: &str) -> Workload { + let mut table_info: TableInfo = serde_json::from_str( + r#"{ + "name": "test", "description": "test", "tablePath": "s3://bucket/table", + "schema": {"type": "struct", "fields": []}, + "protocol": {"minReaderVersion": 1, "minWriterVersion": 2}, + "logInfo": {"numAddFiles": 0, "numRemoveFiles": 0, "sizeInBytes": 0, + "numCommits": 1, "numActions": 1}, + "properties": {}, "dataLayout": {}, "tags": [] + }"#, + ) + .unwrap(); + table_info.table_info_dir = PathBuf::from(table); + Workload { + table_info, + case_name: case.to_string(), + spec: serde_json::from_str::(r#"{"type": "read"}"#).unwrap(), + } + } + + const FULL_REGISTRY: &str = r#"{ + "v2": { + "readMetadataLatest": [ + { "name": "serial", "parallelScan": "disabled" }, + { "name": "parallel2", "parallelScan": { "enabled": { "numThreads": 2 } } } + ] + } + }"#; + + #[test] + fn parses_full_registry() { + let registry = registry_from_str(FULL_REGISTRY); + assert_eq!(registry.tables.len(), 1); + assert_eq!(registry.tables["v2"]["readMetadataLatest"].len(), 2); + } + + #[test] + fn read_configs_use_explicit_entry() { + let registry = registry_from_str(FULL_REGISTRY); + let configs = registry + .read_configs(&read_workload("v2", "readMetadataLatest")) + .unwrap(); + let names: Vec<_> = configs.iter().map(|c| c.name.as_str()).collect(); + assert_eq!(names, vec!["serial", "parallel2"]); + assert_eq!(configs[0].parallel_scan, ParallelScan::Disabled); + assert_eq!( + configs[1].parallel_scan, + ParallelScan::Enabled { num_threads: 2 } + ); + } + + #[test] + fn unlisted_read_benchmark_falls_back_to_serial() { + // An unlisted read benchmark in a populated registry and any lookup against an empty + // registry both fall back to the built-in serial config. + for registry in [registry_from_str(FULL_REGISTRY), registry_from_str("{}")] { + let read = registry + .read_configs(&read_workload("other", "readMetadataLatest")) + .unwrap(); + assert_eq!(read.len(), 1); + assert_eq!(read[0].name, "serial"); + } + } + + #[test] + fn read_configs_errors_without_registry_table_key() { + let registry = registry_from_str(FULL_REGISTRY); + let mut workload = read_workload("v2", "readMetadataLatest"); + workload.table_info.table_info_dir.clear(); + + let error = registry.read_configs(&workload).unwrap_err().to_string(); + assert!(error.contains("could not derive a registry table key")); + } + + #[test] + fn workload_validation_accepts_read_configs_for_read_workload() { + let registry = registry_from_str(FULL_REGISTRY); + registry + .validate(&[read_workload("v2", "readMetadataLatest")]) + .unwrap(); + } + + #[test] + fn workload_validation_rejects_read_configs_for_snapshot_workload() { + let registry = registry_from_str(FULL_REGISTRY); + let mut workload = read_workload("v2", "readMetadataLatest"); + workload.spec = serde_json::from_str(r#"{"type":"snapshotConstruction"}"#).unwrap(); + + let error = registry.validate(&[workload]).unwrap_err().to_string(); + assert!( + error.contains("provides read configs for a snapshotConstruction workload"), + "got: {error}" + ); + } + + #[test] + fn duplicate_config_names_error_at_validation() { + let json = r#"{ + "t": { "readMetadataLatest": [ + { "name": "dup", "parallelScan": "disabled" }, + { "name": "dup", "parallelScan": "disabled" } + ] } + }"#; + let registry: BenchRegistry = serde_json::from_str(json).unwrap(); + let err = registry.validate(&[]).unwrap_err().to_string(); + assert!(err.contains("duplicate config names"), "got: {err}"); + } + + #[test] + fn empty_config_list_errors_at_validation() { + // An explicit `[]` would expand to zero benchmarks, silently dropping the benchmark, so it + // is rejected at load rather than accepted like a missing key (which falls back to + // default). + let json = r#"{ "t": { "readMetadataLatest": [] } }"#; + let registry: BenchRegistry = serde_json::from_str(json).unwrap(); + let err = registry.validate(&[]).unwrap_err().to_string(); + assert!(err.contains("empty config list"), "got: {err}"); + } + + #[test] + fn malformed_entry_errors_at_load() { + // `parallelScn` is a typo, so the entry is not a valid read config. + let json = r#"{ + "t": { "c": [ { "name": "x", "parallelScn": "disabled" } ] } + }"#; + assert!(serde_json::from_str::(json).is_err()); + } + + #[test] + fn non_object_table_entry_errors() { + // Each table key must map to an object of case -> config list, not a scalar. + let result: Result = serde_json::from_str(r#"{ "t": 1 }"#); + assert!(result.is_err()); + } + + #[test] + fn load_from_path_missing_file_errors() { + let result = BenchRegistry::load_from_path(Path::new("/nonexistent/bench-registry.json")); + assert!(result.is_err()); + } +} diff --git a/benchmarks/src/runners.rs b/benchmarks/src/runners.rs index b83b452dd3..c65ae174fd 100644 --- a/benchmarks/src/runners.rs +++ b/benchmarks/src/runners.rs @@ -1,70 +1,274 @@ //! Benchmark runners for executing Delta table operations. //! -//! Each runner holds all the state required for its workload (e.g. read metadata needs pre-built snapshots and a config) -//! so that `execute` measures only the operation itself -//! Results are discarded for benchmarking purposes - -use crate::models::{ - ParallelScan, ReadConfig, ReadOperation, ReadSpec, SnapshotConstructionSpec, TableInfo, -}; -use delta_kernel::scan::{AfterSequentialScanMetadata, ParallelScanMetadata}; -use delta_kernel::Snapshot; -use delta_kernel::{Engine, Error}; +//! Each runner holds all the state required for its workload (e.g. read metadata needs +//! pre-built snapshots and a config) so that `execute` measures only the operation itself. +//! Results are discarded for benchmarking purposes. +//! +//! Engine and snapshot construction is handled based on `TableInfo`: +//! - S3 tables (`table_path` with s3://): credentials from `AWS_*` env vars +//! - Local tables: local filesystem engine use std::hint::black_box; use std::sync::Arc; -use std::thread; + +use delta_kernel::expressions::PredicateRef; +use delta_kernel::object_store::local::LocalFileSystem; +use delta_kernel::scan::{AfterSequentialScanMetadata, ParallelScanMetadata, StatsOptions}; +use delta_kernel::{Engine, Snapshot}; +use delta_kernel_default_engine::executor::tokio::TokioMultiThreadExecutor; +use delta_kernel_default_engine::DefaultEngine; +use delta_kernel_unity_catalog::snapshot_builder_from_load_table; +use delta_kernel_workloads::models::{ + ReadOperation, ReadSpec, SnapshotConstructionSpec, TableInfo, TimeTravel, +}; +use delta_kernel_workloads::predicate_parser::parse_predicate; +use unity_catalog_delta_client_api::Operation; +use unity_catalog_delta_rest_client::{ClientConfig, UCClient}; use url::Url; -/// Each runner holds all the state required for its workload (e.g. read metadata needs pre-built snapshots and a config) -/// so that `execute` measures only the operation itself +use crate::registry::{ParallelScan, ReadConfig}; + pub trait WorkloadRunner { fn execute(&self) -> Result<(), Box>; fn name(&self) -> &str; } +/// Builds `{table}/{case}` from `table_info.name` and `case_name`. +pub fn benchmark_name(table_info: &TableInfo, case_name: &str) -> String { + format!("{}/{case_name}", table_info.name) +} + +/// Builds `{table}/{case}/{config}` from `table_info.name`, `case_name`, and `config_name`. +pub fn configured_benchmark_name( + table_info: &TableInfo, + case_name: &str, + config_name: &str, +) -> String { + format!("{}/{case_name}/{config_name}", table_info.name) +} + +fn build_engine( + store: Arc, + runtime: Arc, +) -> Arc { + let executor = TokioMultiThreadExecutor::new(runtime.handle().clone()); + Arc::new( + DefaultEngine::builder(store) + .with_task_executor(Arc::new(executor)) + .build(), + ) +} + +/// Determines how a snapshot is loaded. Built once at setup via [`resolve_snapshot_strategy`], +/// then used by runners to construct snapshots. +enum SnapshotStrategy { + /// Standard snapshot builder (local, S3, or UC-managed non-catalog-managed tables). + Standard { url: Url }, + /// Catalog-managed table: snapshot loaded via the UC Delta-Tables API. The three-part table + /// name is resolved via `UCClient::load_table`, whose response is turned into a snapshot + /// builder by `snapshot_builder_from_load_table`. + CatalogManaged { + table_name: String, + client: Box, + }, +} + +impl SnapshotStrategy { + /// Builds a snapshot using this strategy. + fn load_snapshot( + &self, + engine: &dyn Engine, + runtime: &tokio::runtime::Runtime, + time_travel: Option<&TimeTravel>, + ) -> Result, Box> { + match self { + SnapshotStrategy::Standard { url } => { + let mut builder = Snapshot::builder_for(url.clone()); + if let Some(tt) = time_travel { + builder = builder.at_version(tt.as_version()?); + } + Ok(builder.build(engine)?) + } + SnapshotStrategy::CatalogManaged { table_name, client } => { + let (catalog, schema, table) = parse_three_part_name(table_name)?; + let resp = runtime + .block_on(client.load_table(catalog, schema, table)) + .map_err(|e| format!("Catalog load_table failed: {e}"))?; + let mut builder = snapshot_builder_from_load_table(&resp)?; + if let Some(tt) = time_travel { + builder = builder.at_version(tt.as_version()?); + } + Ok(builder.build(engine)?) + } + } + } +} + +/// Splits a `"catalog.schema.table"` name into its three parts. +fn parse_three_part_name(name: &str) -> Result<(&str, &str, &str), Box> { + match name.split('.').collect::>()[..] { + [catalog, schema, table] => Ok((catalog, schema, table)), + _ => Err( + format!("expected a three-part table name 'catalog.schema.table', got: {name}").into(), + ), + } +} + +/// Resolves the engine and snapshot strategy from a [`TableInfo`]. +/// +/// For catalog-managed tables (`catalog_info` is present), credentials are vended via `UCClient` +/// and the snapshot is loaded through the UC Delta-Tables API (`load_table`). For non-UC tables, +/// the engine is built from env vars (`AWS_*` for S3, local filesystem otherwise). +fn resolve_snapshot_strategy( + table_info: &TableInfo, + runtime: Arc, +) -> Result<(Arc, SnapshotStrategy), Box> { + let Some(cm) = &table_info.catalog_info else { + let url = table_info.resolved_table_root(); + let engine = resolve_engine_for_url(&url, runtime)?; + return Ok((engine, SnapshotStrategy::Standard { url })); + }; + + let endpoint = std::env::var("UC_WORKSPACE").map_err(|_| "UC_WORKSPACE required")?; + let token = std::env::var("UC_TOKEN").map_err(|_| "UC_TOKEN required")?; + let config = ClientConfig::build(&endpoint, &token).build()?; + let client = Box::new(UCClient::new(config)?); + + let (catalog, schema, table) = parse_three_part_name(&cm.table_name)?; + let creds = runtime + .block_on(client.get_table_credentials(catalog, schema, table, Operation::Read)) + .map_err(|e| format!("Credential vending failed: {e}"))?; + + let resp = runtime + .block_on(client.load_table(catalog, schema, table)) + .map_err(|e| format!("load_table failed: {e}"))?; + let table_url = Url::parse(&resp.metadata.location)?; + + let opts = object_store_options(&creds.storage_credentials); + let (store, _) = delta_kernel::object_store::parse_url_opts(&table_url, opts)?; + let engine = build_engine(store.into(), runtime); + + Ok(( + engine, + SnapshotStrategy::CatalogManaged { + table_name: cm.table_name.clone(), + client, + }, + )) +} + +/// Maps vended UC storage credentials into `object_store` option keys. +fn object_store_options( + creds: &[unity_catalog_delta_client_api::StorageCredential], +) -> Vec<(String, String)> { + let mut opts = Vec::new(); + for cred in creds { + for (key, value) in &cred.config { + let mapped = match key.as_str() { + "s3.access-key-id" => "access_key_id", + "s3.secret-access-key" => "secret_access_key", + "s3.session-token" => "session_token", + other => other, + }; + opts.push((mapped.to_string(), value.clone())); + } + } + if let Ok(region) = std::env::var("UC_AWS_REGION") { + opts.push(("region".to_string(), region)); + } + opts +} + +/// Builds an engine from the table URL scheme and env vars (S3 or local). +fn resolve_engine_for_url( + url: &Url, + runtime: Arc, +) -> Result, Box> { + match url.scheme() { + "s3" | "s3a" => { + let region = + std::env::var("AWS_REGION").map_err(|_| "AWS_REGION required for S3 tables")?; + let mut opts: Vec<(&str, String)> = vec![("region", region)]; + for (env_key, opt_key) in [ + ("AWS_ACCESS_KEY_ID", "access_key_id"), + ("AWS_SECRET_ACCESS_KEY", "secret_access_key"), + ("AWS_SESSION_TOKEN", "session_token"), + ] { + if let Ok(v) = std::env::var(env_key) { + opts.push((opt_key, v)); + } + } + let (store, _) = delta_kernel::object_store::parse_url_opts(url, opts)?; + Ok(build_engine(store.into(), runtime)) + } + "file" => Ok(build_engine(Arc::new(LocalFileSystem::new()), runtime)), + scheme => Err(format!( + "Unsupported scheme '{scheme}': only s3://, s3a://, and file:// are supported" + ) + .into()), + } +} + pub struct ReadMetadataRunner { snapshot: Arc, engine: Arc, name: String, config: ReadConfig, + thread_pool: Option, /* None for serial configuration, Some for parallel + * configuration */ + predicate: Option, } impl ReadMetadataRunner { pub fn setup( - table_info: &TableInfo, - case_name: &str, + name: String, read_spec: &ReadSpec, config: ReadConfig, - engine: Arc, + table_info: &TableInfo, + runtime: Arc, ) -> Result> { - let url = table_info.resolved_table_root(); - - let mut builder = Snapshot::builder_for(url); - if let Some(version) = read_spec.version { - builder = builder.at_version(version); - } - - let snapshot = builder.build(engine.as_ref())?; - - let name = format!( - "{}/{}/{}/{}", - table_info.name, - case_name, - ReadOperation::ReadMetadata.as_str(), - config.name, - ); + let (engine, strategy) = resolve_snapshot_strategy(table_info, runtime.clone())?; + let snapshot = + strategy.load_snapshot(engine.as_ref(), &runtime, read_spec.time_travel.as_ref())?; + + let predicate = read_spec + .predicate + .as_deref() + .map(|sql| parse_predicate(sql, &snapshot.schema())) + .transpose()? + .map(Arc::new); + + let thread_pool = match &config.parallel_scan { + ParallelScan::Enabled { num_threads } => { + if *num_threads == 0 { + return Err("num_threads in ReadConfig must be greater than 0".into()); + } + let thread_pool = rayon::ThreadPoolBuilder::new() + .num_threads(*num_threads) + .build()?; + Some(thread_pool) + } + ParallelScan::Disabled => None, + }; Ok(Self { snapshot, engine, name, config, + thread_pool, + predicate, }) } fn execute_serial(&self) -> Result<(), Box> { - let scan = self.snapshot.clone().scan_builder().build()?; + let scan = self + .snapshot + .clone() + .scan_builder() + .with_predicate(self.predicate.clone()) + .with_stats(StatsOptions::all_struct()) + .build()?; let metadata_iter = scan.scan_metadata(self.engine.as_ref())?; for result in metadata_iter { black_box(result?); @@ -72,8 +276,19 @@ impl ReadMetadataRunner { Ok(()) } - fn execute_parallel(&self, num_threads: usize) -> Result<(), Box> { - let scan = self.snapshot.clone().scan_builder().build()?; + fn execute_parallel(&self) -> Result<(), Box> { + let pool = self + .thread_pool + .as_ref() + .ok_or("thread_pool must be Some for parallel execution")?; + + let scan = self + .snapshot + .clone() + .scan_builder() + .with_predicate(self.predicate.clone()) + .with_stats(StatsOptions::all_struct()) + .build()?; let mut phase1 = scan.parallel_scan_metadata(self.engine.clone())?; for result in phase1.by_ref() { @@ -83,9 +298,7 @@ impl ReadMetadataRunner { match phase1.finish()? { AfterSequentialScanMetadata::Done => {} AfterSequentialScanMetadata::Parallel { state, files } => { - if num_threads == 0 { - return Err("num_threads in ReadConfig must be greater than 0".into()); - } + let num_threads = pool.current_num_threads(); let files_per_worker = files.len().div_ceil(num_threads); let partitions: Vec<_> = files @@ -95,33 +308,25 @@ impl ReadMetadataRunner { let state = Arc::new(*state); - let handles: Vec<_> = partitions - .into_iter() - .map(|partition_files| { + pool.scope(|s| { + for partition_files in partitions { let engine = self.engine.clone(); let state = state.clone(); - thread::spawn(move || -> Result<(), Error> { + s.spawn(move |_| { if partition_files.is_empty() { - return Ok(()); + return; } let parallel = - ParallelScanMetadata::try_new(engine, state, partition_files)?; + ParallelScanMetadata::try_new(engine, state, partition_files) + .expect("Failed to create ParallelScanMetadata"); for result in parallel { - black_box(result?); + black_box(result.expect("Parallel scan error")); } - - Ok(()) - }) - }) - .collect(); - - for handle in handles { - handle.join().map_err(|_| -> Box { - "Worker thread panicked".into() - })??; - } + }); + } + }); } } Ok(()) @@ -131,15 +336,9 @@ impl ReadMetadataRunner { impl WorkloadRunner for ReadMetadataRunner { fn execute(&self) -> Result<(), Box> { match &self.config.parallel_scan { - ParallelScan::Disabled => { - self.execute_serial()?; - } - ParallelScan::Enabled { num_threads } => { - self.execute_parallel(*num_threads)?; - } + ParallelScan::Disabled => self.execute_serial(), + ParallelScan::Enabled { .. } => self.execute_parallel(), } - - Ok(()) } fn name(&self) -> &str { @@ -147,50 +346,46 @@ impl WorkloadRunner for ReadMetadataRunner { } } -/// Factory function that creates the appropriate read runner for a given operation and config +/// Factory function that creates the appropriate read runner for a given operation and config. +/// `name` is the full benchmark identifier (`{table_name}/{case}/{config}`). pub fn create_read_runner( - table_info: &TableInfo, - case_name: &str, + name: String, read_spec: &ReadSpec, operation: ReadOperation, config: ReadConfig, - engine: Arc, + table_info: &TableInfo, + runtime: Arc, ) -> Result, Box> { match operation { ReadOperation::ReadMetadata => Ok(Box::new(ReadMetadataRunner::setup( - table_info, case_name, read_spec, config, engine, + name, read_spec, config, table_info, runtime, )?)), ReadOperation::ReadData => Err("ReadDataRunner not yet implemented".into()), } } pub struct SnapshotConstructionRunner { - url: Url, - version: Option, engine: Arc, + runtime: Arc, + snapshot_strategy: SnapshotStrategy, + time_travel: Option, name: String, } impl SnapshotConstructionRunner { pub fn setup( - table_info: &TableInfo, - case_name: &str, + name: String, snapshot_spec: &SnapshotConstructionSpec, - engine: Arc, + table_info: &TableInfo, + runtime: Arc, ) -> Result> { - let url = table_info.resolved_table_root(); - - let name = format!( - "{}/{}/{}", - table_info.name, - case_name, - snapshot_spec.as_str() - ); + let (engine, snapshot_strategy) = resolve_snapshot_strategy(table_info, runtime.clone())?; Ok(Self { - url, - version: snapshot_spec.version, engine, + runtime, + snapshot_strategy, + time_travel: snapshot_spec.time_travel.clone(), name, }) } @@ -198,12 +393,12 @@ impl SnapshotConstructionRunner { impl WorkloadRunner for SnapshotConstructionRunner { fn execute(&self) -> Result<(), Box> { - let mut builder = Snapshot::builder_for(self.url.clone()); - if let Some(version) = self.version { - builder = builder.at_version(version); - } - black_box(builder.build(self.engine.as_ref())?); - + let snapshot = self.snapshot_strategy.load_snapshot( + self.engine.as_ref(), + &self.runtime, + self.time_travel.as_ref(), + )?; + black_box(snapshot); Ok(()) } @@ -214,28 +409,66 @@ impl WorkloadRunner for SnapshotConstructionRunner { #[cfg(test)] mod tests { + use std::collections::HashMap; + use std::path::{Path, PathBuf}; + use std::slice; + use std::sync::LazyLock; + + use delta_kernel_workloads::models::{ReadSpec, Spec, TableInfo, TimeTravel, Workload}; + use super::*; - use crate::models::{ParallelScan, ReadConfig, ReadSpec, TableInfo}; - use delta_kernel::engine::default::DefaultEngine; - use delta_kernel::object_store::local::LocalFileSystem; - use std::path::PathBuf; + use crate::registry::BenchRegistry; + use crate::utils::load_all_workloads; + + fn test_runtime() -> Arc { + static RT: LazyLock> = LazyLock::new(|| { + Arc::new(tokio::runtime::Runtime::new().expect("failed to create runtime")) + }); + RT.clone() + } fn test_table_info() -> TableInfo { let path = format!( "{}/../kernel/tests/data/basic_partitioned", env!("CARGO_MANIFEST_DIR") ); - TableInfo { - name: "basic_partitioned".to_string(), - description: "basic partitioned table for testing".to_string(), - table_path: Some(Url::from_file_path(path).unwrap()), - tags: vec![], - table_info_dir: PathBuf::new(), - } + let json = format!( + r#"{{ + "name": "basic_partitioned", + "description": "basic partitioned table for testing", + "tablePath": "{}", + "schema": {{ + "type": "struct", + "fields": [ + {{"name": "letter", "type": "string", "nullable": true, "metadata": {{}}}}, + {{"name": "number", "type": "long", "nullable": true, "metadata": {{}}}}, + {{"name": "a_float", "type": "double", "nullable": true, "metadata": {{}}}} + ] + }}, + "protocol": {{"minReaderVersion": 1, "minWriterVersion": 2}}, + "logInfo": {{ + "numAddFiles": 6, + "numRemoveFiles": 0, + "sizeInBytes": 4505, + "numCommits": 2, + "numActions": 10 + }}, + "properties": {{}}, + "dataLayout": {{"numPartitionColumns": 1, "numDistinctPartitions": 5}}, + "tags": [] + }}"#, + Url::from_file_path(path).unwrap() + ); + serde_json::from_str(&json).expect("failed to build test TableInfo") } fn test_read_spec() -> ReadSpec { - ReadSpec { version: None } + ReadSpec { + time_travel: None, + predicate: None, + columns: None, + expected: None, + } } fn serial_config() -> ReadConfig { @@ -247,61 +480,64 @@ mod tests { fn parallel_config() -> ReadConfig { ReadConfig { - name: "parallel".to_string(), + name: "parallel2".to_string(), parallel_scan: ParallelScan::Enabled { num_threads: 2 }, } } - fn test_engine() -> Arc { - let store = Arc::new(LocalFileSystem::new()); - Arc::new(DefaultEngine::builder(store).build()) + #[test] + fn configured_benchmark_name_uses_human_readable_table_name() { + let mut table_info = test_table_info(); + table_info.name = "Human readable table".to_string(); + + assert_eq!( + configured_benchmark_name(&table_info, "testCase", "serial"), + "Human readable table/testCase/serial" + ); } #[test] fn test_read_metadata_runner_serial() { let runner = ReadMetadataRunner::setup( - &test_table_info(), - "test_case", + configured_benchmark_name(&test_table_info(), "testCase", "serial"), &test_read_spec(), serial_config(), - test_engine(), + &test_table_info(), + test_runtime(), ) .expect("setup should succeed"); - assert_eq!( - runner.name(), - "basic_partitioned/test_case/read_metadata/serial" - ); + assert_eq!(runner.name(), "basic_partitioned/testCase/serial"); assert!(runner.execute().is_ok()); } #[test] fn test_read_metadata_runner_parallel() { let runner = ReadMetadataRunner::setup( - &test_table_info(), - "test_case", + configured_benchmark_name(&test_table_info(), "testCase", "parallel2"), &test_read_spec(), parallel_config(), - test_engine(), + &test_table_info(), + test_runtime(), ) .expect("setup should succeed"); - assert_eq!( - runner.name(), - "basic_partitioned/test_case/read_metadata/parallel" - ); + assert_eq!(runner.name(), "basic_partitioned/testCase/parallel2"); assert!(runner.execute().is_ok()); } fn test_snapshot_spec() -> SnapshotConstructionSpec { - SnapshotConstructionSpec { version: None } + SnapshotConstructionSpec { + time_travel: None, + expected: None, + } } #[test] fn test_snapshot_construction_runner_setup() { let runner = SnapshotConstructionRunner::setup( - &test_table_info(), - "test_case", + benchmark_name(&test_table_info(), "testCase"), &test_snapshot_spec(), - test_engine(), + &test_table_info(), + test_runtime(), ); assert!(runner.is_ok()); } @@ -309,54 +545,164 @@ mod tests { #[test] fn test_snapshot_construction_runner_name() { let runner = SnapshotConstructionRunner::setup( - &test_table_info(), - "test_case", + benchmark_name(&test_table_info(), "testCase"), &test_snapshot_spec(), - test_engine(), + &test_table_info(), + test_runtime(), ) .expect("setup should succeed"); - assert_eq!( - runner.name(), - "basic_partitioned/test_case/snapshot_construction" - ); + assert_eq!(runner.name(), "basic_partitioned/testCase"); } #[test] fn test_snapshot_construction_runner_execute() { let runner = SnapshotConstructionRunner::setup( - &test_table_info(), - "test_case", + benchmark_name(&test_table_info(), "testCase"), &test_snapshot_spec(), - test_engine(), + &test_table_info(), + test_runtime(), ) .expect("setup should succeed"); assert!(runner.execute().is_ok()); } + /// Guards the checked-in `bench-registry.json` so a malformed edit or a duplicate config name + /// fails fast in `cargo test` rather than only when the harness runs. + #[test] + fn checked_in_registry_loads_and_validates() { + let path = format!("{}/bench-registry.json", env!("CARGO_MANIFEST_DIR")); + let registry = BenchRegistry::load_from_path(Path::new(&path)) + .expect("checked-in bench-registry.json must load"); + let mut table_info = test_table_info(); + table_info.table_info_dir = PathBuf::from("10kAdds0CommitsSinceChkpt1V2Chkpt"); + let workload = Workload { + table_info, + case_name: "readMetadataLatest".to_string(), + spec: Spec::Read(test_read_spec()), + }; + registry + .validate(slice::from_ref(&workload)) + .expect("checked-in bench-registry.json must validate"); + let read = registry.read_configs(&workload).unwrap(); + assert_eq!( + read.iter().map(|c| c.name.as_str()).collect::>(), + vec!["serial", "parallel2"] + ); + } + + /// Every registry key must name a real read workload. A key miss silently falls back to the + /// serial default, so a stale/typo'd key would otherwise be invisible until the harness runs. + /// Skips when the downloaded workload archive is absent (e.g. offline unit runs). + #[test] + fn checked_in_registry_keys_match_real_workloads() { + let Ok(workloads) = load_all_workloads() else { + return; // workload archive not downloaded -- nothing to cross-check + }; + let workload_by_key: HashMap<(&str, &str), &Workload> = workloads + .iter() + .map(|w| { + ( + ( + w.table_info + .registry_table_key() + .expect("loaded workloads must have a registry table key"), + w.case_name.as_str(), + ), + w, + ) + }) + .collect(); + + let path = format!("{}/bench-registry.json", env!("CARGO_MANIFEST_DIR")); + let registry = BenchRegistry::load_from_path(Path::new(&path)) + .expect("checked-in bench-registry.json must load"); + registry + .validate(&workloads) + .expect("registry configs must match workload types"); + + for (table, case) in registry.keys() { + workload_by_key.get(&(table, case)).unwrap_or_else(|| { + panic!("registry key '{table}/{case}' matches no workload (stale or typo'd?)") + }); + } + } + #[test] fn test_create_read_runner_read_metadata() { let runner = create_read_runner( - &test_table_info(), - "test_case", + configured_benchmark_name(&test_table_info(), "testCase", "serial"), &test_read_spec(), ReadOperation::ReadMetadata, serial_config(), - test_engine(), + &test_table_info(), + test_runtime(), ) .expect("create_read_runner should succeed"); assert!(runner.execute().is_ok()); } + #[test] + fn test_read_metadata_runner_with_valid_predicate() { + let mut spec = test_read_spec(); + spec.predicate = Some("letter = 'a'".to_string()); + let runner = ReadMetadataRunner::setup( + configured_benchmark_name(&test_table_info(), "test_case", "serial"), + &spec, + serial_config(), + &test_table_info(), + test_runtime(), + ) + .expect("setup should succeed"); + assert!(runner.execute().is_ok()); + } + + #[test] + fn test_read_metadata_runner_with_invalid_predicate() { + let mut spec = test_read_spec(); + spec.predicate = Some("a LIKE '%foo'".to_string()); + let result = ReadMetadataRunner::setup( + configured_benchmark_name(&test_table_info(), "test_case", "serial"), + &spec, + serial_config(), + &test_table_info(), + test_runtime(), + ); + assert!(result.is_err()); + } + #[test] fn test_create_read_runner_read_data_unimplemented() { let result = create_read_runner( - &test_table_info(), - "test_case", + configured_benchmark_name(&test_table_info(), "testCase", "serial"), &test_read_spec(), ReadOperation::ReadData, serial_config(), - test_engine(), + &test_table_info(), + test_runtime(), ); assert!(result.is_err()); } + + #[test] + fn test_resolve_engine_unsupported_scheme() { + let url = Url::parse("gs://bucket/table").unwrap(); + let result = resolve_engine_for_url(&url, test_runtime()); + assert!(result.is_err()); + } + + #[test] + fn test_snapshot_construction_with_time_travel() { + let spec = SnapshotConstructionSpec { + time_travel: Some(TimeTravel::Version { version: 0 }), + expected: None, + }; + let runner = SnapshotConstructionRunner::setup( + benchmark_name(&test_table_info(), "testCase"), + &spec, + &test_table_info(), + test_runtime(), + ) + .expect("setup should succeed"); + assert!(runner.execute().is_ok()); + } } diff --git a/benchmarks/src/utils.rs b/benchmarks/src/utils.rs index 3cf0bdb9a9..35390d6fdc 100644 --- a/benchmarks/src/utils.rs +++ b/benchmarks/src/utils.rs @@ -1,52 +1,50 @@ //! Utility functions for loading workload specifications -use crate::models::{Spec, TableInfo, Workload}; -use flate2::read::GzDecoder; -use std::io::{BufReader, Write}; use std::path::{Path, PathBuf}; -use tar::Archive; + +use delta_kernel_workloads::models::{Spec, TableInfo, Workload}; // Environment variable used to filter benchmarks by tag (e.g. `BENCH_TAGS=base,feature_x`). pub const BENCH_TAGS_ENV_VAR: &str = "BENCH_TAGS"; -// Workload extraction configuration -const WORKLOAD_TAR: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/data/workloads.tar.gz"); -const OUTPUT_FOLDER: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/data/workloads"); -const DONE_FILE: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/data/workloads/.done"); -const TABLE_INFO_FILE_NAME: &str = "table_info.json"; +const OUTPUT_FOLDER: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/workloads"); +const TABLE_INFO_FILE_NAME: &str = "tableInfo.json"; const SPECS_DIR_NAME: &str = "specs"; const BENCHMARKS_DIR_NAME: &str = "benchmarks"; const DELTA_DIR_NAME: &str = "delta"; -/// Loads all workload specifications from OUTPUT_FOLDER, optionally filtered by BENCH_TAGS +/// Loads all workload specifications from `OUTPUT_FOLDER`, optionally filtered by `BENCH_TAGS`. +/// +/// If `KERNEL_BENCH_WORKLOAD_DIR` is set, loads from that directory instead (for remote/S3 tables). /// -/// On first run, extracts from WORKLOAD_TAR if it exists. -/// Uses a .done file to avoid re-extracting on subsequent runs +/// Workloads are downloaded and extracted into `OUTPUT_FOLDER` at build time by `build.rs`. /// /// If the `BENCH_TAGS` environment variable is set (e.g. `BENCH_TAGS=base`), -/// only workloads whose `table_info.json` has at least one matching tag are returned -/// If `BENCH_TAGS` is unset or empty, all workloads are returned +/// only workloads whose `table_info.json` has at least one matching tag are returned. +/// If `BENCH_TAGS` is unset or empty, all workloads are returned. pub fn load_all_workloads() -> Result, Box> { - if !workload_specs_exist() { - extract_workload_specs()?; - } + // When KERNEL_BENCH_WORKLOAD_DIR is set, tag filtering is skipped -- remote workload + // directories are assumed to be curated, so all tables in the directory are benchmarked. + let (base_dir, required_tags) = if let Ok(dir) = std::env::var("KERNEL_BENCH_WORKLOAD_DIR") { + (PathBuf::from(dir), None) + } else { + let benchmarks_dir = PathBuf::from(OUTPUT_FOLDER).join(BENCHMARKS_DIR_NAME); + (benchmarks_dir, get_required_tags()) + }; - let spec_dir = PathBuf::from(OUTPUT_FOLDER); - let benchmarks_dir = spec_dir.join(BENCHMARKS_DIR_NAME); - let table_directories = find_table_directories(&benchmarks_dir)?; + let table_directories = find_table_directories(&base_dir)?; - let required_tags = get_required_tags(); let mut all_workloads = Vec::new(); for table_dir in table_directories { - all_workloads.extend(load_specs_from_table(&table_dir, required_tags.as_ref())?); + all_workloads.extend(load_specs_from_table(&table_dir, required_tags.as_deref())?); } Ok(all_workloads) } -/// Reads the BENCH_TAGS environment variable and returns the set of tags -/// Returns None if unset or empty (meaning we should run all workloads) +/// Reads the `BENCH_TAGS` environment variable and returns the set of tags +/// Returns `None` if unset or empty (meaning we should run all workloads) fn get_required_tags() -> Option> { std::env::var(BENCH_TAGS_ENV_VAR) .ok() @@ -54,64 +52,9 @@ fn get_required_tags() -> Option> { .map(|s| s.split(',').map(|t| t.trim().to_string()).collect()) } -/// Returns true if table_tags has ANY tag in common with required, where required is the set of tags specified with BENCH_TAGS -fn matches_tags(table_tags: &[String], required: &[String]) -> bool { - required.iter().any(|r| table_tags.contains(r)) -} - -/// Checks if workload specs have already been extracted by looking for the .done file -/// If the .done file exists, we don't need to extract the workload specs again -/// -/// TODO(#1939): the usage of this function is a naive check; -/// currently, the .done file must be manually deleted to force re-extraction of workload specs -fn workload_specs_exist() -> bool { - Path::new(DONE_FILE).exists() -} - -/// Extracts workload specs from WORKLOAD_TAR into OUTPUT_FOLDER and writes a .done file on success -fn extract_workload_specs() -> Result<(), Box> { - let tar_path = Path::new(WORKLOAD_TAR); - - if !tar_path.exists() { - return Err(format!("Workload tarball not found at {WORKLOAD_TAR}").into()); - } - - extract_tarball(tar_path)?; - write_done_file()?; - - Ok(()) -} - -/// Extracts a tarball at path into OUTPUT_FOLDER -/// This is used to extract WORKLOAD_TAR into OUTPUT_FOLDER -fn extract_tarball(path: &Path) -> Result<(), Box> { - let file = std::fs::File::open(path)?; - let tarball = GzDecoder::new(BufReader::new(file)); - let mut archive = Archive::new(tarball); - - std::fs::create_dir_all(OUTPUT_FOLDER) - .map_err(|e| format!("Failed to create output directory: {e}"))?; - - archive - .unpack(OUTPUT_FOLDER) - .map_err(|e| format!("Failed to unpack tarball: {e}"))?; - - Ok(()) -} - -/// Writes DONE_FILE to mark that workload specs have been successfully extracted -/// See TODO(#1939) for `workload_specs_exist` above; this file must be manually deleted to force re-extraction -fn write_done_file() -> Result<(), Box> { - let mut done_file = std::fs::File::create(DONE_FILE) - .map_err(|e| format!("Failed to create .done file: {e}"))?; - - write!(done_file, "done").map_err(|e| format!("Failed to write .done file: {e}"))?; - - Ok(()) -} - -/// Returns all subdirectories of `base_dir`. In practice this is called with base_dir = OUTPUT_FOLDER/BENCHMARKS_DIR_NAME, -/// Each subdirectory returned represents a table to be benchmarked and contains the table itself, specs, and table info +/// Returns all subdirectories of `base_dir`. In practice this is called with `base_dir` = +/// `OUTPUT_FOLDER`/`BENCHMARKS_DIR_NAME`, Each subdirectory returned represents a table to be +/// benchmarked and contains the table itself, specs, and table info fn find_table_directories(base_dir: &Path) -> Result, Box> { let entries = std::fs::read_dir(base_dir) .map_err(|e| format!("Cannot read directory {}: {}", base_dir.display(), e))?; @@ -132,12 +75,15 @@ fn find_table_directories(base_dir: &Path) -> Result, Box>, + required_tags: Option<&[String]>, ) -> Result, Box> { let specs_dir = table_dir.join(SPECS_DIR_NAME); @@ -156,24 +102,33 @@ fn load_specs_from_table( // Skip this table if BENCH_TAGS is set and none of its tags match if let Some(tags) = required_tags { - if !matches_tags(&table_info.tags, tags) { + if !table_info.matches_tags(tags) { return Ok(vec![]); } } - // If the table path is not provided, assume that the Delta table is in a DELTA_DIR_NAME/ subdirectory at the same level as table_info.json - if table_info.table_path.is_none() { + // Remote tables (table_path or catalog_info present) don't need local data. + // Local tables must have a delta/ subdirectory next to tableInfo.json. + let is_remote = table_info.table_path.is_some() || table_info.catalog_info.is_some(); + if !is_remote { let delta_dir = table_dir.join(DELTA_DIR_NAME); if !delta_dir.is_dir() { return Err(format!( - "Table data not found for '{}'. Expected a 'delta' directory in {}", - table_info.name, + "Table data not found. Expected a 'delta' directory in {}", table_dir.display() ) .into()); } } + if table_info.registry_table_key().is_none() { + return Err(format!( + "could not derive table name from directory {}", + table_dir.display() + ) + .into()); + } + let spec_files = find_spec_files(&specs_dir)?; let mut workloads = Vec::new(); @@ -229,66 +184,63 @@ mod tests { use super::*; #[test] - fn test_matches_tags_or_semantics() { - let table_tags = vec!["ci".to_string(), "checkpoints".to_string()]; - - assert!(matches_tags(&table_tags, &["ci".to_string()])); - assert!(matches_tags(&table_tags, &["checkpoints".to_string()])); - assert!(matches_tags( - &table_tags, - &["ci".to_string(), "large".to_string()] - )); - } - - #[test] - fn test_matches_tags_no_match() { - let table_tags = vec!["large".to_string()]; - assert!(!matches_tags( - &table_tags, - &["ci".to_string(), "checkpoints".to_string()] - )); - } - - #[test] - fn test_matches_tags_empty_table_tags() { - let table_tags: Vec = vec![]; - assert!(!matches_tags(&table_tags, &["ci".to_string()])); - } - - #[test] - fn test_get_required_tags_unset() { + fn test_get_required_tags() { + // These cases must run sequentially b/c env vars conflict when these tests are separate (as + // they run in parallel) std::env::remove_var(BENCH_TAGS_ENV_VAR); assert!(get_required_tags().is_none()); - } - #[test] - fn test_get_required_tags_empty() { std::env::set_var(BENCH_TAGS_ENV_VAR, ""); assert!(get_required_tags().is_none()); - std::env::remove_var(BENCH_TAGS_ENV_VAR); - } - #[test] - fn test_get_required_tags_single() { std::env::set_var(BENCH_TAGS_ENV_VAR, "ci"); - let tags = get_required_tags().unwrap(); - assert_eq!(tags, vec!["ci"]); - std::env::remove_var(BENCH_TAGS_ENV_VAR); - } + assert_eq!(get_required_tags().unwrap(), vec!["ci"]); - #[test] - fn test_get_required_tags_multiple() { std::env::set_var(BENCH_TAGS_ENV_VAR, "ci,checkpoints,v2"); - let tags = get_required_tags().unwrap(); - assert_eq!(tags, vec!["ci", "checkpoints", "v2"]); + assert_eq!( + get_required_tags().unwrap(), + vec!["ci", "checkpoints", "v2"] + ); + + std::env::set_var(BENCH_TAGS_ENV_VAR, " ci , checkpoints "); + assert_eq!(get_required_tags().unwrap(), vec!["ci", "checkpoints"]); + std::env::remove_var(BENCH_TAGS_ENV_VAR); } #[test] - fn test_get_required_tags_with_whitespace() { - std::env::set_var(BENCH_TAGS_ENV_VAR, " ci , checkpoints "); - let tags = get_required_tags().unwrap(); - assert_eq!(tags, vec!["ci", "checkpoints"]); - std::env::remove_var(BENCH_TAGS_ENV_VAR); + fn load_specs_from_table_uses_directory_basename_not_table_info_name() { + // A remote `tablePath` skips the local `delta/` requirement, so only `tableInfo.json` and + // `specs/` are needed. + let root = tempfile::tempdir().unwrap(); + let table_dir = root.path().join("dirBasename"); + std::fs::create_dir_all(table_dir.join(SPECS_DIR_NAME)).unwrap(); + std::fs::write( + table_dir.join(TABLE_INFO_FILE_NAME), + r#"{ + "name": "humanLabel", "description": "d", "tablePath": "s3://b/t", + "schema": {"type": "struct", "fields": []}, + "protocol": {"minReaderVersion": 1, "minWriterVersion": 2}, + "logInfo": {"numAddFiles": 0, "numRemoveFiles": 0, "sizeInBytes": 0, "numCommits": 1, "numActions": 1}, + "properties": {}, "dataLayout": {}, "tags": [] + }"#, + ) + .unwrap(); + std::fs::write( + table_dir + .join(SPECS_DIR_NAME) + .join("readMetadataLatest.json"), + r#"{"type": "read"}"#, + ) + .unwrap(); + + let workloads = load_specs_from_table(&table_dir, None).unwrap(); + assert_eq!(workloads.len(), 1); + assert_eq!( + workloads[0].table_info.registry_table_key(), + Some("dirBasename") + ); + assert_eq!(workloads[0].table_info.name, "humanLabel"); + assert_eq!(workloads[0].case_name, "readMetadataLatest"); } } diff --git a/datafusion-executor/CLAUDE.md b/datafusion-executor/CLAUDE.md new file mode 100644 index 0000000000..be855d17a0 --- /dev/null +++ b/datafusion-executor/CLAUDE.md @@ -0,0 +1,26 @@ +# CLAUDE.md -- datafusion-executor + +`datafusion_executor` is a DataFusion-based `PlanExecutor` for delta_kernel declarative plans. +Kernel emits executor-independent logical `Plan`s; this crate executes them by lowering each +plan to a DataFusion `LogicalPlan`, optimizing it, and running the resulting `ExecutionPlan`. + +## Separate workspace + +This crate has its own `[workspace]`, so it is NOT a member of the root workspace and the root's +`--workspace` commands do not touch it. The isolation exists to avoid an arrow version mismatch: +DataFusion pins a specific arrow major, so kernel must be built against that same major here. The +root workspace builds kernel against its own default arrow major, which need not match. As a root +member, Cargo feature unification would compile kernel against a single arrow major shared with +the rest of the workspace, so DataFusion's types would link against a kernel built on a different +arrow major -- a mismatch that breaks the build. A separate workspace resolves kernel's arrow +major in isolation, matched to DataFusion's. + +## Build & Test Commands + +cd into the crate (the subshell keeps that from leaking out); cargo picks up the nested workspace +automatically. + +```bash +(cd datafusion-executor && cargo build) +(cd datafusion-executor && cargo test) +``` diff --git a/datafusion-executor/Cargo.lock b/datafusion-executor/Cargo.lock new file mode 100644 index 0000000000..6eb7f9c5bf --- /dev/null +++ b/datafusion-executor/Cargo.lock @@ -0,0 +1,4199 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "adler2" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" + +[[package]] +name = "ahash" +version = "0.8.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" +dependencies = [ + "cfg-if", + "const-random", + "getrandom 0.3.4", + "once_cell", + "version_check", + "zerocopy", +] + +[[package]] +name = "aho-corasick" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" +dependencies = [ + "memchr", +] + +[[package]] +name = "alloc-no-stdlib" +version = "2.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc7bb162ec39d46ab1ca8c77bf72e890535becd1751bb45f64c597edb4c8c6b3" + +[[package]] +name = "alloc-stdlib" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0e76a019e91224d279006ff972f1e984179a6e9feb050adba6ce8274aef23195" +dependencies = [ + "alloc-no-stdlib", +] + +[[package]] +name = "allocator-api2" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923" + +[[package]] +name = "android_system_properties" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" +dependencies = [ + "libc", +] + +[[package]] +name = "anyhow" +version = "1.0.103" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2a4385e2e34eb35d6b3efe798b9eb88096925d87726c0798709bf56d9ed84af3" + +[[package]] +name = "ar_archive_writer" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4087686b4b0a3427190bae57a1d9a478dbb2d40c5dc1bd6e2b6d797913bdd348" +dependencies = [ + "object", +] + +[[package]] +name = "arrayref" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb" + +[[package]] +name = "arrayvec" +version = "0.7.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3fb67a6e08acf24fdeccbac2cb6ac4305825bd1f117462e0e6f2f193345ad56" + +[[package]] +name = "arrow" +version = "58.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "378530e55cd479eda3c14eb345310799717e6f76d0c332041e8487022166b471" +dependencies = [ + "arrow-arith", + "arrow-array", + "arrow-buffer", + "arrow-cast", + "arrow-csv", + "arrow-data", + "arrow-ipc", + "arrow-json", + "arrow-ord", + "arrow-row", + "arrow-schema", + "arrow-select", + "arrow-string", +] + +[[package]] +name = "arrow-arith" +version = "58.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a0ab212d2c1886e802f51c5212d78ebbcbb0bec980fff9dadc1eb8d45cd0b738" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "chrono", + "num-traits", +] + +[[package]] +name = "arrow-array" +version = "58.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfd33d3e92f207444098c75b42de99d329562be0cf686b307b097cc52b4e999e" +dependencies = [ + "ahash", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "chrono", + "chrono-tz", + "half", + "hashbrown 0.17.1", + "num-complex", + "num-integer", + "num-traits", +] + +[[package]] +name = "arrow-buffer" +version = "58.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c6cd424c2693bcdbc150d843dc9d4d137dd2de4782ce6df491ad11a3a0416c0" +dependencies = [ + "bytes", + "half", + "num-bigint", + "num-traits", +] + +[[package]] +name = "arrow-cast" +version = "58.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4c5aefb56a2c02e9e2b30746241058b85f8983f0fcff2ba0c6d09006e1cded7f" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-ord", + "arrow-schema", + "arrow-select", + "atoi", + "base64", + "chrono", + "comfy-table", + "half", + "lexical-core", + "num-traits", + "ryu", +] + +[[package]] +name = "arrow-csv" +version = "58.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e94e8cf7e517657a52b91ea1263acf38c4ca62a84655d72458a3359b12ab97de" +dependencies = [ + "arrow-array", + "arrow-cast", + "arrow-schema", + "chrono", + "csv", + "csv-core", + "regex", +] + +[[package]] +name = "arrow-data" +version = "58.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3c88210023a2bfee1896af366309a3028fc3bcbd6515fa29a7990ee1baa08ee0" +dependencies = [ + "arrow-buffer", + "arrow-schema", + "half", + "num-integer", + "num-traits", +] + +[[package]] +name = "arrow-ipc" +version = "58.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "238438f0834483703d88896db6fe5a7138b2230debc31b34c0336c2996e3c64f" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "arrow-select", + "flatbuffers", + "lz4_flex", + "zstd", +] + +[[package]] +name = "arrow-json" +version = "58.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "205ca2119e6d679d5c133c6f30e68f027738d95ed948cf77677ea69c7800036b" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-cast", + "arrow-ord", + "arrow-schema", + "arrow-select", + "chrono", + "half", + "indexmap", + "itoa", + "lexical-core", + "memchr", + "num-traits", + "ryu", + "serde_core", + "serde_json", + "simdutf8", +] + +[[package]] +name = "arrow-ord" +version = "58.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1bffd8fd2579286a5d63bac898159873e5094a79009940bcb42bbfce4f19f1d0" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "arrow-select", +] + +[[package]] +name = "arrow-row" +version = "58.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bab5994731204603c73ba69267616c50f80780774c6bb0476f1f830625115e0c" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "half", +] + +[[package]] +name = "arrow-schema" +version = "58.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f633dbfdf39c039ada1bf9e34c694816eb71fbb7dc78f613993b7245e078a1ed" +dependencies = [ + "bitflags", + "serde_core", + "serde_json", +] + +[[package]] +name = "arrow-select" +version = "58.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8cd065c54172ac787cf3f2f8d4107e0d3fdc26edba76fdf4f4cc170258942222" +dependencies = [ + "ahash", + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "num-traits", +] + +[[package]] +name = "arrow-string" +version = "58.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29dd7cda3ab9692f43a2e4acc444d760cc17b12bb6d8232ddf64e9bab7c06b42" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "arrow-select", + "memchr", + "num-traits", + "regex", + "regex-syntax", +] + +[[package]] +name = "async-compression" +version = "0.4.42" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e79b3f8a79cccc2898f31920fc69f304859b3bd567490f75ebf51ae1c792a9ac" +dependencies = [ + "compression-codecs", + "compression-core", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "async-trait" +version = "0.1.89" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "atoi" +version = "2.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f28d99ec8bfea296261ca1af174f24225171fea9664ba9003cbebee704810528" +dependencies = [ + "num-traits", +] + +[[package]] +name = "atomic-waker" +version = "1.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "base64" +version = "0.22.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" + +[[package]] +name = "bigdecimal" +version = "0.4.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4d6867f1565b3aad85681f1015055b087fcfd840d6aeee6eee7f2da317603695" +dependencies = [ + "autocfg", + "libm", + "num-bigint", + "num-integer", + "num-traits", +] + +[[package]] +name = "bitflags" +version = "2.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" + +[[package]] +name = "blake2" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "46502ad458c9a52b69d4d4d32775c788b7a1b85e8bc9d482d92250fc0e3f8efe" +dependencies = [ + "digest 0.10.7", +] + +[[package]] +name = "blake3" +version = "1.8.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0aa83c34e62843d924f905e0f5c866eb1dd6545fc4d719e803d9ba6030371fce" +dependencies = [ + "arrayref", + "arrayvec", + "cc", + "cfg-if", + "constant_time_eq", + "cpufeatures", +] + +[[package]] +name = "block-buffer" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" +dependencies = [ + "generic-array", +] + +[[package]] +name = "block-buffer" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "brotli" +version = "8.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5cc91aac060a7a1e25823bdccbfb6af1875b88f17c6daac97894eed8207166b3" +dependencies = [ + "alloc-no-stdlib", + "alloc-stdlib", + "brotli-decompressor", +] + +[[package]] +name = "brotli-decompressor" +version = "5.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a32acac15fe1967bc3986b2a6347dffc965602354ea6f450ad07e8bfd253583" +dependencies = [ + "alloc-no-stdlib", + "alloc-stdlib", +] + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "bytemuck" +version = "1.25.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6aedf8ae72766347502cf3cb4f41cf5e9cc37d28bee90f1fdaaae15f9cf9424" + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + +[[package]] +name = "bzip2" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3a53fac24f34a81bc9954b5d6cfce0c21e18ec6959f44f56e8e90e4bb7c346c" +dependencies = [ + "libbz2-rs-sys", +] + +[[package]] +name = "cc" +version = "1.2.67" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e17dd265a7d0f31ef544e1b20e03add05d3b45b491b633b10d67145d2acc1a38" +dependencies = [ + "find-msvc-tools", + "jobserver", + "libc", + "shlex", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "cfg_aliases" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" + +[[package]] +name = "chacha20" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81" +dependencies = [ + "cfg-if", + "cpufeatures", + "rand_core 0.10.1", +] + +[[package]] +name = "chrono" +version = "0.4.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327" +dependencies = [ + "iana-time-zone", + "js-sys", + "num-traits", + "serde", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "chrono-tz" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6139a8597ed92cf816dfb33f5dd6cf0bb93a6adc938f11039f371bc5bcd26c3" +dependencies = [ + "chrono", + "phf", +] + +[[package]] +name = "comfy-table" +version = "7.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "958c5d6ecf1f214b4c2bbbbf6ab9523a864bd136dcf71a7e8904799acfe1ad47" +dependencies = [ + "unicode-segmentation", + "unicode-width", +] + +[[package]] +name = "compression-codecs" +version = "0.4.38" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce2548391e9c1929c21bf6aa2680af86fe4c1b33e6cea9ac1cfeec0bd11218cf" +dependencies = [ + "bzip2", + "compression-core", + "flate2", + "liblzma", + "memchr", + "zstd", + "zstd-safe", +] + +[[package]] +name = "compression-core" +version = "0.4.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc14f565cf027a105f7a44ccf9e5b424348421a1d8952a8fc9d499d313107789" + +[[package]] +name = "const-oid" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" + +[[package]] +name = "const-random" +version = "0.1.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "87e00182fe74b066627d63b85fd550ac2998d4b0bd86bfed477a0ae4c7c71359" +dependencies = [ + "const-random-macro", +] + +[[package]] +name = "const-random-macro" +version = "0.1.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9d839f2a20b0aee515dc581a6172f2321f96cab76c1a38a4c584a194955390e" +dependencies = [ + "getrandom 0.2.17", + "once_cell", + "tiny-keccak", +] + +[[package]] +name = "constant_time_eq" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" + +[[package]] +name = "core-foundation" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b2a6cd9ae233e7f62ba4e9353e81a88df7fc8a5987b8d445b4d90c879bd156f6" +dependencies = [ + "core-foundation-sys", + "libc", +] + +[[package]] +name = "core-foundation-sys" +version = "0.8.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" + +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + +[[package]] +name = "crc" +version = "3.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5eb8a2a1cd12ab0d987a5d5e825195d372001a4094a0376319d5a0ad71c1ba0d" +dependencies = [ + "crc-catalog", +] + +[[package]] +name = "crc-catalog" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "217698eaf96b4a3f0bc4f3662aaa55bdf913cd54d7204591faa790070c6d0853" + +[[package]] +name = "crc32fast" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "crossbeam-utils" +version = "0.8.22" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "61803da095bee82a81bb1a452ecc25d3b2f1416d1897eb86430c6159ef717c17" + +[[package]] +name = "crunchy" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + +[[package]] +name = "crypto-common" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "csv" +version = "1.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52cd9d68cf7efc6ddfaaee42e7288d3a99d613d4b50f76ce9827ae0c6e14f938" +dependencies = [ + "csv-core", + "itoa", + "ryu", + "serde_core", +] + +[[package]] +name = "csv-core" +version = "0.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "704a3c26996a80471189265814dbc2c257598b96b8a7feae2d31ace646bb9782" +dependencies = [ + "memchr", +] + +[[package]] +name = "dashmap" +version = "6.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6361d5c062261c78a176addb82d4c821ae42bed6089de0e12603cd25de2059c" +dependencies = [ + "cfg-if", + "crossbeam-utils", + "hashbrown 0.14.5", + "lock_api", + "once_cell", + "parking_lot_core", +] + +[[package]] +name = "datafusion" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "997a31e15872606a49478e670c58302094c97cb96abb0a7d60720f8e92170040" +dependencies = [ + "arrow", + "arrow-schema", + "async-trait", + "bzip2", + "chrono", + "datafusion-catalog", + "datafusion-catalog-listing", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-datasource-arrow", + "datafusion-datasource-csv", + "datafusion-datasource-json", + "datafusion-datasource-parquet", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-functions", + "datafusion-functions-aggregate", + "datafusion-functions-nested", + "datafusion-functions-table", + "datafusion-functions-window", + "datafusion-optimizer", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-optimizer", + "datafusion-physical-plan", + "datafusion-session", + "datafusion-sql", + "flate2", + "futures", + "indexmap", + "itertools", + "liblzma", + "log", + "object_store", + "parking_lot", + "parquet", + "sqlparser", + "tempfile", + "tokio", + "url", + "uuid", + "zstd", +] + +[[package]] +name = "datafusion-catalog" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f7dd61161508f8f5fa1107774ea687bd753c22d83a32eebf963549f89de14139" +dependencies = [ + "arrow", + "async-trait", + "dashmap", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-plan", + "datafusion-session", + "futures", + "itertools", + "log", + "object_store", + "parking_lot", + "tokio", +] + +[[package]] +name = "datafusion-catalog-listing" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "897c70f871277f9ce99aa38347be0d679bbe3e617156c4d2a8378cec8a2a0891" +dependencies = [ + "arrow", + "async-trait", + "datafusion-catalog", + "datafusion-common", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "futures", + "itertools", + "log", + "object_store", +] + +[[package]] +name = "datafusion-common" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "121c9ded5d87d9172319e006f2afdb9928d72dbacd6a90a458d8acb1e3b43a65" +dependencies = [ + "arrow", + "arrow-ipc", + "arrow-schema", + "chrono", + "foldhash 0.2.0", + "half", + "hashbrown 0.17.1", + "indexmap", + "itertools", + "libc", + "log", + "object_store", + "parquet", + "recursive", + "sqlparser", + "tokio", + "uuid", + "web-time", +] + +[[package]] +name = "datafusion-common-runtime" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "981b9dae74f78ee3d9f714fb49b01919eab975461b56149510c3ba9ea11287d1" +dependencies = [ + "futures", + "log", + "tokio", +] + +[[package]] +name = "datafusion-datasource" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ffd7d295b2ec7c00d8a56562f41ed41062cf0af75549ed891c12a0a09eddfefe" +dependencies = [ + "arrow", + "async-compression", + "async-trait", + "bytes", + "bzip2", + "chrono", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", + "flate2", + "futures", + "glob", + "itertools", + "liblzma", + "log", + "object_store", + "parking_lot", + "rand 0.9.5", + "tokio", + "tokio-util", + "url", + "zstd", +] + +[[package]] +name = "datafusion-datasource-arrow" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "552b0b3f342f7ec41b3fbd70f6339dc82a30cfd0349e7f280e7852528085349f" +dependencies = [ + "arrow", + "arrow-ipc", + "async-trait", + "bytes", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", + "futures", + "itertools", + "object_store", + "tokio", +] + +[[package]] +name = "datafusion-datasource-csv" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68850aa426b897e879c8b87e512ea8124f1d0a2869a4e51808ddaaddf1bc0ada" +dependencies = [ + "arrow", + "async-trait", + "bytes", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", + "futures", + "object_store", + "regex", + "tokio", +] + +[[package]] +name = "datafusion-datasource-json" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "402f93242ae08ef99139ee2c528a49d087efe88d5c7b2c3ff5480855a40ce54f" +dependencies = [ + "arrow", + "async-trait", + "bytes", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", + "futures", + "object_store", + "tokio", + "tokio-stream", +] + +[[package]] +name = "datafusion-datasource-parquet" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ffd2499c1bee0eeccf6a57156105700eeeb17bc701899ac719183c4e74231450" +dependencies = [ + "arrow", + "async-trait", + "bytes", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions", + "datafusion-functions-aggregate-common", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-pruning", + "datafusion-session", + "futures", + "itertools", + "log", + "object_store", + "parking_lot", + "parquet", + "tokio", +] + +[[package]] +name = "datafusion-doc" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb9e7e5d11130c48c8bd4e80c79a9772dd28ce6dc330baca9246205d245b9e2e" + +[[package]] +name = "datafusion-execution" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "37a8643ab852eb68864e1b72ae789e8066282dce48eea6347ffb0aee33d1ccc0" +dependencies = [ + "arrow", + "arrow-buffer", + "async-trait", + "dashmap", + "datafusion-common", + "datafusion-expr", + "datafusion-physical-expr-common", + "futures", + "log", + "object_store", + "parking_lot", + "rand 0.9.5", + "tempfile", + "url", +] + +[[package]] +name = "datafusion-expr" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6932f4d71eed9c8d9341476a2b845aadfabde5495d08dbcd8fc23881f49fa7a0" +dependencies = [ + "arrow", + "arrow-schema", + "async-trait", + "chrono", + "datafusion-common", + "datafusion-doc", + "datafusion-expr-common", + "datafusion-functions-aggregate-common", + "datafusion-functions-window-common", + "datafusion-physical-expr-common", + "indexmap", + "itertools", + "recursive", + "serde_json", + "sqlparser", +] + +[[package]] +name = "datafusion-expr-common" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0225491839a31b1f7d2cb8092c2d50792e2fe1c1724e4e6d08e011f5feaf4ed2" +dependencies = [ + "arrow", + "datafusion-common", + "indexmap", + "itertools", +] + +[[package]] +name = "datafusion-functions" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "14872c47bfc3d21e53ec82f57074e6987a15941c1e2f43cde4ac6ae2746634e3" +dependencies = [ + "arrow", + "arrow-buffer", + "base64", + "blake2", + "blake3", + "chrono", + "chrono-tz", + "datafusion-common", + "datafusion-doc", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-macros", + "datafusion-physical-expr-common", + "hex", + "itertools", + "log", + "md-5 0.11.0", + "memchr", + "num-traits", + "rand 0.9.5", + "regex", + "sha2", + "uuid", +] + +[[package]] +name = "datafusion-functions-aggregate" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "75a2ca14e1b609be21e657e2d3130b2f446456b08393b377bb721a33952d2e09" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-doc", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions-aggregate-common", + "datafusion-macros", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "foldhash 0.2.0", + "half", + "log", + "num-traits", +] + +[[package]] +name = "datafusion-functions-aggregate-common" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ece74ba09092d2ef9c9b54a38445450aea292a1f8b04faf531936b723a24b3c" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-expr-common", + "datafusion-physical-expr-common", +] + +[[package]] +name = "datafusion-functions-nested" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f3e3f9ee8ca59bf70518802107de6f1b88a9509efdc629fadc5de9d6b2d5ef5" +dependencies = [ + "arrow", + "arrow-ord", + "datafusion-common", + "datafusion-doc", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-functions", + "datafusion-functions-aggregate", + "datafusion-functions-aggregate-common", + "datafusion-macros", + "datafusion-physical-expr-common", + "hashbrown 0.17.1", + "itertools", + "itoa", + "log", + "memchr", +] + +[[package]] +name = "datafusion-functions-table" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "89161dffc22cf2b50f9f4b1bee83b5221d3b4ed7c2e37fd7aa2b22a5297b3a26" +dependencies = [ + "arrow", + "async-trait", + "datafusion-catalog", + "datafusion-common", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-plan", + "parking_lot", +] + +[[package]] +name = "datafusion-functions-window" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7339345b226b3874037708bf5023ba1c2de705128f8457a095aae5ae9cb9c78" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-doc", + "datafusion-expr", + "datafusion-functions-window-common", + "datafusion-macros", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "log", +] + +[[package]] +name = "datafusion-functions-window-common" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fa84836dc2392df6f43d6a29d37fb56a8ebdc8b3f4e10ae8dc15861fd20278fb" +dependencies = [ + "datafusion-common", + "datafusion-physical-expr-common", +] + +[[package]] +name = "datafusion-macros" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "587164e03ad68732aa9e7bfe5686e3f25970d4c64fd4bd80790749840892dae5" +dependencies = [ + "datafusion-doc", + "quote", + "syn", +] + +[[package]] +name = "datafusion-optimizer" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77f20e8cf9e8654d92f4c16b24c487353ee5bf153ffc12d5772cd399ab8cd281" +dependencies = [ + "arrow", + "chrono", + "datafusion-common", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-physical-expr", + "indexmap", + "itertools", + "log", + "recursive", + "regex", + "regex-syntax", +] + +[[package]] +name = "datafusion-physical-expr" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f015a4a82f6f7ff7e1d8d4bf3870a936752fa38b17705dfcc14adef95aa8922c" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-functions-aggregate-common", + "datafusion-physical-expr-common", + "half", + "hashbrown 0.17.1", + "indexmap", + "itertools", + "parking_lot", + "petgraph 0.8.3", + "recursive", + "tokio", +] + +[[package]] +name = "datafusion-physical-expr-adapter" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "51e6ffff8acdfe54e0ea15ccf38115c4a9184433b0439f42907637928d00a235" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-expr", + "datafusion-functions", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "itertools", +] + +[[package]] +name = "datafusion-physical-expr-common" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7967a3e171c6a4bf09474b3f7a14f1a3db13ed1714ba12156f33fcce2bba54e8" +dependencies = [ + "arrow", + "chrono", + "datafusion-common", + "datafusion-expr-common", + "hashbrown 0.17.1", + "indexmap", + "itertools", + "parking_lot", + "pin-project", +] + +[[package]] +name = "datafusion-physical-optimizer" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59ff803e2a96054cb6d83f35f9e60fd4f42eac515e1932bd1b2dbc91d5fcbf36" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-pruning", + "itertools", + "recursive", +] + +[[package]] +name = "datafusion-physical-plan" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "776ee54d47d15bdb126452f9ca17b03761e3b004682914beaedd3f86eb507fbc" +dependencies = [ + "arrow", + "arrow-data", + "arrow-ipc", + "arrow-ord", + "arrow-schema", + "async-trait", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions", + "datafusion-functions-aggregate-common", + "datafusion-functions-window-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "futures", + "half", + "hashbrown 0.17.1", + "indexmap", + "itertools", + "log", + "num-traits", + "parking_lot", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "datafusion-pruning" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d5fb9e5774660aa69c3ba93c610f175f75b65cb8c3776edb3626de8f3a4f4ee3" +dependencies = [ + "arrow", + "datafusion-common", + "datafusion-datasource", + "datafusion-expr-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "log", +] + +[[package]] +name = "datafusion-session" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "15ce715fa2a61f4623cc234bcc14a3ef6a91f189128d5b14b468a6a17cdfc417" +dependencies = [ + "async-trait", + "datafusion-common", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-plan", + "parking_lot", +] + +[[package]] +name = "datafusion-sql" +version = "54.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6094ad36a3ed6d7ac87b20b479b2d0b118250f66cf997603829fdc65b44a7099" +dependencies = [ + "arrow", + "bigdecimal", + "chrono", + "datafusion-common", + "datafusion-expr", + "datafusion-functions-nested", + "indexmap", + "log", + "recursive", + "regex", + "sqlparser", +] + +[[package]] +name = "datafusion_executor" +version = "0.26.0" +dependencies = [ + "datafusion", + "delta_kernel", + "rstest", +] + +[[package]] +name = "delta_kernel" +version = "0.27.1" +dependencies = [ + "arrow", + "bytes", + "chrono", + "chrono-tz", + "crc", + "delta_kernel_derive", + "indexmap", + "itertools", + "object_store", + "parquet", + "percent-encoding", + "prost", + "prost-build", + "protoc-bin-vendored", + "rand 0.9.5", + "reqwest 0.13.4", + "roaring", + "rustc_version", + "serde", + "serde_json", + "strum", + "thiserror", + "tracing", + "tracing-subscriber", + "url", + "uuid", + "z85", +] + +[[package]] +name = "delta_kernel_derive" +version = "0.27.1" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "block-buffer 0.10.4", + "crypto-common 0.1.7", + "subtle", +] + +[[package]] +name = "digest" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" +dependencies = [ + "block-buffer 0.12.1", + "const-oid", + "crypto-common 0.2.2", +] + +[[package]] +name = "displaydoc" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ac70aa55017e108007fbaf5aa0f54b021c98f92ff8af59d42eda9da96e3dd4f" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "either" +version = "1.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e" + +[[package]] +name = "equivalent" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + +[[package]] +name = "fastrand" +version = "2.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f1f227452a390804cdb637b74a86990f2a7d7ba4b7d5693aac9b4dd6defd8d6" + +[[package]] +name = "find-msvc-tools" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" + +[[package]] +name = "fixedbitset" +version = "0.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99" + +[[package]] +name = "flatbuffers" +version = "25.12.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "35f6839d7b3b98adde531effaf34f0c2badc6f4735d26fe74709d8e513a96ef3" +dependencies = [ + "bitflags", + "rustc_version", +] + +[[package]] +name = "flate2" +version = "1.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" +dependencies = [ + "crc32fast", + "miniz_oxide", + "zlib-rs", +] + +[[package]] +name = "fnv" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" + +[[package]] +name = "foldhash" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" + +[[package]] +name = "foldhash" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" + +[[package]] +name = "form_urlencoded" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf" +dependencies = [ + "percent-encoding", +] + +[[package]] +name = "futures" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b147ee9d1f6d097cef9ce628cd2ee62288d963e16fb287bd9286455b241382d" +dependencies = [ + "futures-channel", + "futures-core", + "futures-executor", + "futures-io", + "futures-sink", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-channel" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "07bbe89c50d7a535e539b8c17bc0b49bdb77747034daa8087407d655f3f7cc1d" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" + +[[package]] +name = "futures-executor" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "baf29c38818342a3b26b5b923639e7b1f4a61fc5e76102d4b1981c6dc7a7579d" +dependencies = [ + "futures-core", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-io" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cecba35d7ad927e23624b22ad55235f2239cfa44fd10428eecbeba6d6a717718" + +[[package]] +name = "futures-macro" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e835b70203e41293343137df5c0664546da5745f82ec9b84d40be8336958447b" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "futures-sink" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c39754e157331b013978ec91992bde1ac089843443c49cbc7f46150b0fad0893" + +[[package]] +name = "futures-task" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" + +[[package]] +name = "futures-timer" +version = "3.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "af43fadb8a98512d547e37b4e92e0ced13e205c061b87b4623eff01d918d6968" + +[[package]] +name = "futures-util" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" +dependencies = [ + "futures-channel", + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "slab", +] + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +dependencies = [ + "cfg-if", + "js-sys", + "libc", + "wasi", + "wasm-bindgen", +] + +[[package]] +name = "getrandom" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" +dependencies = [ + "cfg-if", + "libc", + "r-efi 5.3.0", + "wasip2", +] + +[[package]] +name = "getrandom" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" +dependencies = [ + "cfg-if", + "js-sys", + "libc", + "r-efi 6.0.0", + "rand_core 0.10.1", + "wasm-bindgen", +] + +[[package]] +name = "glob" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280" + +[[package]] +name = "h2" +version = "0.4.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6cb093c84e8bd9b188d4c4a8cb6579fc016968d14c99882163cd3ff402a4f155" +dependencies = [ + "atomic-waker", + "bytes", + "fnv", + "futures-core", + "futures-sink", + "http", + "indexmap", + "slab", + "tokio", + "tokio-util", + "tracing", +] + +[[package]] +name = "half" +version = "2.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b" +dependencies = [ + "cfg-if", + "crunchy", + "num-traits", + "zerocopy", +] + +[[package]] +name = "hashbrown" +version = "0.14.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" + +[[package]] +name = "hashbrown" +version = "0.15.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" +dependencies = [ + "foldhash 0.1.5", +] + +[[package]] +name = "hashbrown" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" +dependencies = [ + "allocator-api2", + "equivalent", + "foldhash 0.2.0", +] + +[[package]] +name = "heck" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" + +[[package]] +name = "hex" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" + +[[package]] +name = "http" +version = "1.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6970f50e31d6fc17d3fa27329444bfa74e196cf62e95052a3f6fee181dba6425" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "http-body" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ca2a8f2913ee65f60facd6a5905613afaa448497a0230cc41ce022d93290bc2c" +dependencies = [ + "bytes", + "http", +] + +[[package]] +name = "http-body-util" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e9f41fd6a08e4d4ec69df65976da761afd5ad5e58a9d4acb46bd1c953a9e3ff2" +dependencies = [ + "bytes", + "futures-core", + "http", + "http-body", + "pin-project-lite", +] + +[[package]] +name = "httparse" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" + +[[package]] +name = "humantime" +version = "2.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "15cdd26707701c53297e2fa6afb323d55fbc1d0810c3aec078ae3ef0424c3c15" + +[[package]] +name = "hybrid-array" +version = "0.4.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "818356c5132c1fede50f837ca96afbe78ff42413047f4abb886217845e1b6c8c" +dependencies = [ + "typenum", +] + +[[package]] +name = "hyper" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "55281c53a1894c864990125767da440a4e630446785086f52523b20033b74498" +dependencies = [ + "atomic-waker", + "bytes", + "futures-channel", + "futures-core", + "h2", + "http", + "http-body", + "httparse", + "itoa", + "pin-project-lite", + "smallvec", + "tokio", + "want", +] + +[[package]] +name = "hyper-rustls" +version = "0.27.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f" +dependencies = [ + "http", + "hyper", + "hyper-util", + "rustls", + "rustls-native-certs", + "tokio", + "tokio-rustls", + "tower-service", +] + +[[package]] +name = "hyper-util" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" +dependencies = [ + "base64", + "bytes", + "futures-channel", + "futures-util", + "http", + "http-body", + "hyper", + "ipnet", + "libc", + "percent-encoding", + "pin-project-lite", + "socket2", + "tokio", + "tower-service", + "tracing", +] + +[[package]] +name = "iana-time-zone" +version = "0.1.65" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" +dependencies = [ + "android_system_properties", + "core-foundation-sys", + "iana-time-zone-haiku", + "js-sys", + "log", + "wasm-bindgen", + "windows-core", +] + +[[package]] +name = "iana-time-zone-haiku" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" +dependencies = [ + "cc", +] + +[[package]] +name = "icu_collections" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c" +dependencies = [ + "displaydoc", + "potential_utf", + "utf8_iter", + "yoke", + "zerofrom", + "zerovec", +] + +[[package]] +name = "icu_locale_core" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29" +dependencies = [ + "displaydoc", + "litemap", + "tinystr", + "writeable", + "zerovec", +] + +[[package]] +name = "icu_normalizer" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4" +dependencies = [ + "icu_collections", + "icu_normalizer_data", + "icu_properties", + "icu_provider", + "smallvec", + "zerovec", +] + +[[package]] +name = "icu_normalizer_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38" + +[[package]] +name = "icu_properties" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de" +dependencies = [ + "icu_collections", + "icu_locale_core", + "icu_properties_data", + "icu_provider", + "zerotrie", + "zerovec", +] + +[[package]] +name = "icu_properties_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14" + +[[package]] +name = "icu_provider" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421" +dependencies = [ + "displaydoc", + "icu_locale_core", + "writeable", + "yoke", + "zerofrom", + "zerotrie", + "zerovec", +] + +[[package]] +name = "idna" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de" +dependencies = [ + "idna_adapter", + "smallvec", + "utf8_iter", +] + +[[package]] +name = "idna_adapter" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714" +dependencies = [ + "icu_normalizer", + "icu_properties", +] + +[[package]] +name = "indexmap" +version = "2.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" +dependencies = [ + "equivalent", + "hashbrown 0.17.1", +] + +[[package]] +name = "integer-encoding" +version = "3.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8bb03732005da905c88227371639bf1ad885cc712789c011c31c5fb3ab3ccf02" + +[[package]] +name = "ipnet" +version = "2.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2" + +[[package]] +name = "itertools" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" +dependencies = [ + "either", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "jobserver" +version = "0.1.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3" +dependencies = [ + "getrandom 0.4.3", + "libc", +] + +[[package]] +name = "js-sys" +version = "0.3.103" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + +[[package]] +name = "lazy_static" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" + +[[package]] +name = "lexical-core" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d8d125a277f807e55a77304455eb7b1cb52f2b18c143b60e766c120bd64a594" +dependencies = [ + "lexical-parse-float", + "lexical-parse-integer", + "lexical-util", + "lexical-write-float", + "lexical-write-integer", +] + +[[package]] +name = "lexical-parse-float" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52a9f232fbd6f550bc0137dcb5f99ab674071ac2d690ac69704593cb4abbea56" +dependencies = [ + "lexical-parse-integer", + "lexical-util", +] + +[[package]] +name = "lexical-parse-integer" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a7a039f8fb9c19c996cd7b2fcce303c1b2874fe1aca544edc85c4a5f8489b34" +dependencies = [ + "lexical-util", +] + +[[package]] +name = "lexical-util" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2604dd126bb14f13fb5d1bd6a66155079cb9fa655b37f875b3a742c705dbed17" + +[[package]] +name = "lexical-write-float" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "50c438c87c013188d415fbabbb1dceb44249ab81664efbd31b14ae55dabb6361" +dependencies = [ + "lexical-util", + "lexical-write-integer", +] + +[[package]] +name = "lexical-write-integer" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "409851a618475d2d5796377cad353802345cba92c867d9fbcde9cf4eac4e14df" +dependencies = [ + "lexical-util", +] + +[[package]] +name = "libbz2-rs-sys" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "34b357333733e8260735ba5894eb928c02ecc69c78715f01a8019e7fa7f2db4c" + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "liblzma" +version = "0.4.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "45aec2360b3933207e27908049d8e4df4e476b58180afb1e56b2a4fb72efe4ba" +dependencies = [ + "liblzma-sys", +] + +[[package]] +name = "liblzma-sys" +version = "0.4.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a046c7f353ba30f810545151e04f63545833803f5b86ee3ddf1517247fe560a5" +dependencies = [ + "cc", + "libc", + "pkg-config", +] + +[[package]] +name = "libm" +version = "0.2.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "litemap" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0" + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "log" +version = "0.4.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" + +[[package]] +name = "lru-slab" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154" + +[[package]] +name = "lz4_flex" +version = "0.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ef0d4ed8669f8f8826eb00dc878084aa8f253506c4fd5e8f58f5bce72ddb97e" +dependencies = [ + "twox-hash", +] + +[[package]] +name = "md-5" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf" +dependencies = [ + "cfg-if", + "digest 0.10.7", +] + +[[package]] +name = "md-5" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98" +dependencies = [ + "cfg-if", + "digest 0.11.3", +] + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "miniz_oxide" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" +dependencies = [ + "adler2", + "simd-adler32", +] + +[[package]] +name = "mio" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "30d65c71f1ce40ab09135ce117d742b9f8a19ff91a41a8b57ed50bc2de59c427" +dependencies = [ + "libc", + "wasi", + "windows-sys 0.61.2", +] + +[[package]] +name = "multimap" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d87ecb2933e8aeadb3e3a02b828fed80a7528047e68b4f424523a0981a3a084" + +[[package]] +name = "nu-ansi-term" +version = "0.50.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "num-bigint" +version = "0.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c89e69e7e0f03bea5ef08013795c25018e101932225a656383bd384495ecc367" +dependencies = [ + "num-integer", + "num-traits", +] + +[[package]] +name = "num-complex" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-integer" +version = "0.1.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", + "libm", +] + +[[package]] +name = "object" +version = "0.37.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff76201f031d8863c38aa7f905eca4f53abbfa15f609db4277d44cd8938f33fe" +dependencies = [ + "memchr", +] + +[[package]] +name = "object_store" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "622acbc9100d3c10e2ee15804b0caa40e55c933d5aa53814cd520805b7958a49" +dependencies = [ + "async-trait", + "base64", + "bytes", + "chrono", + "form_urlencoded", + "futures-channel", + "futures-core", + "futures-util", + "http", + "http-body-util", + "httparse", + "humantime", + "hyper", + "itertools", + "md-5 0.10.6", + "parking_lot", + "percent-encoding", + "quick-xml", + "rand 0.10.2", + "reqwest 0.12.28", + "ring", + "rustls-pki-types", + "serde", + "serde_json", + "serde_urlencoded", + "thiserror", + "tokio", + "tracing", + "url", + "walkdir", + "wasm-bindgen-futures", + "web-time", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "openssl-probe" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe" + +[[package]] +name = "ordered-float" +version = "2.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68f19d67e5a2795c94e73e0bb1cc1a7edeb2e28efd39e2e1c9b7a40c1108b11c" +dependencies = [ + "num-traits", +] + +[[package]] +name = "parking_lot" +version = "0.12.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a" +dependencies = [ + "lock_api", + "parking_lot_core", +] + +[[package]] +name = "parking_lot_core" +version = "0.9.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" +dependencies = [ + "cfg-if", + "libc", + "redox_syscall", + "smallvec", + "windows-link", +] + +[[package]] +name = "parquet" +version = "58.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5dafa7d01085b62a47dd0c1829550a0a36710ea9c4fe358a05a85477cec8a908" +dependencies = [ + "ahash", + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-ipc", + "arrow-schema", + "arrow-select", + "base64", + "brotli", + "bytes", + "chrono", + "flate2", + "futures", + "half", + "hashbrown 0.17.1", + "lz4_flex", + "num-bigint", + "num-integer", + "num-traits", + "object_store", + "paste", + "seq-macro", + "simdutf8", + "snap", + "thrift", + "tokio", + "twox-hash", + "zstd", +] + +[[package]] +name = "paste" +version = "1.0.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "petgraph" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3672b37090dbd86368a4145bc067582552b29c27377cad4e0a306c97f9bd7772" +dependencies = [ + "fixedbitset", + "indexmap", +] + +[[package]] +name = "petgraph" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8701b58ea97060d5e5b155d383a69952a60943f0e6dfe30b04c287beb0b27455" +dependencies = [ + "fixedbitset", + "hashbrown 0.15.5", + "indexmap", + "serde", +] + +[[package]] +name = "phf" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "913273894cec178f401a31ec4b656318d95473527be05c0752cc41cdc32be8b7" +dependencies = [ + "phf_shared", +] + +[[package]] +name = "phf_shared" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06005508882fb681fd97892ecff4b7fd0fee13ef1aa569f8695dae7ab9099981" +dependencies = [ + "siphasher", +] + +[[package]] +name = "pin-project" +version = "1.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2466b2336ed02bcdca6b294417127b90ec92038d1d5c4fbeac971a922e0e0924" +dependencies = [ + "pin-project-internal", +] + +[[package]] +name = "pin-project-internal" +version = "1.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c96395f0a926bc13b1c17622aaddda1ecb55d49c8f1bf9777e4d877800a43f8b" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "pkg-config" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" + +[[package]] +name = "potential_utf" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564" +dependencies = [ + "zerovec", +] + +[[package]] +name = "ppv-lite86" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" +dependencies = [ + "zerocopy", +] + +[[package]] +name = "prettyplease" +version = "0.2.37" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" +dependencies = [ + "proc-macro2", + "syn", +] + +[[package]] +name = "proc-macro-crate" +version = "3.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e67ba7e9b2b56446f1d419b1d807906278ffa1a658a8a5d8a39dcb1f5a78614f" +dependencies = [ + "toml_edit", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "prost" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2796faa41db3ec313a31f7624d9286acf277b52de526150b7e69f3debf891ee5" +dependencies = [ + "bytes", + "prost-derive", +] + +[[package]] +name = "prost-build" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf" +dependencies = [ + "heck", + "itertools", + "log", + "multimap", + "once_cell", + "petgraph 0.7.1", + "prettyplease", + "prost", + "prost-types", + "regex", + "syn", + "tempfile", +] + +[[package]] +name = "prost-derive" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d" +dependencies = [ + "anyhow", + "itertools", + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "prost-types" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52c2c1bf36ddb1a1c396b3601a3cec27c2462e45f07c386894ec3ccf5332bd16" +dependencies = [ + "prost", +] + +[[package]] +name = "protoc-bin-vendored" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d1c381df33c98266b5f08186583660090a4ffa0889e76c7e9a5e175f645a67fa" +dependencies = [ + "protoc-bin-vendored-linux-aarch_64", + "protoc-bin-vendored-linux-ppcle_64", + "protoc-bin-vendored-linux-s390_64", + "protoc-bin-vendored-linux-x86_32", + "protoc-bin-vendored-linux-x86_64", + "protoc-bin-vendored-macos-aarch_64", + "protoc-bin-vendored-macos-x86_64", + "protoc-bin-vendored-win32", +] + +[[package]] +name = "protoc-bin-vendored-linux-aarch_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c350df4d49b5b9e3ca79f7e646fde2377b199e13cfa87320308397e1f37e1a4c" + +[[package]] +name = "protoc-bin-vendored-linux-ppcle_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a55a63e6c7244f19b5c6393f025017eb5d793fd5467823a099740a7a4222440c" + +[[package]] +name = "protoc-bin-vendored-linux-s390_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1dba5565db4288e935d5330a07c264a4ee8e4a5b4a4e6f4e83fad824cc32f3b0" + +[[package]] +name = "protoc-bin-vendored-linux-x86_32" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8854774b24ee28b7868cd71dccaae8e02a2365e67a4a87a6cd11ee6cdbdf9cf5" + +[[package]] +name = "protoc-bin-vendored-linux-x86_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b38b07546580df720fa464ce124c4b03630a6fb83e05c336fea2a241df7e5d78" + +[[package]] +name = "protoc-bin-vendored-macos-aarch_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "89278a9926ce312e51f1d999fee8825d324d603213344a9a706daa009f1d8092" + +[[package]] +name = "protoc-bin-vendored-macos-x86_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "81745feda7ccfb9471d7a4de888f0652e806d5795b61480605d4943176299756" + +[[package]] +name = "protoc-bin-vendored-win32" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "95067976aca6421a523e491fce939a3e65249bac4b977adee0ee9771568e8aa3" + +[[package]] +name = "psm" +version = "0.1.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "645dbe486e346d9b5de3ef16ede18c26e6c70ad97418f4874b8b1889d6e761ea" +dependencies = [ + "ar_archive_writer", + "cc", +] + +[[package]] +name = "quick-xml" +version = "0.39.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cdcc8dd4e2f670d309a5f0e83fe36dfdc05af317008fea29144da1a2ac858e5e" +dependencies = [ + "memchr", + "serde", +] + +[[package]] +name = "quinn" +version = "0.11.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c1a41e437b6bbd489372cd4971de128e85c855f56c57f283d20ff016cf7c0a8" +dependencies = [ + "bytes", + "cfg_aliases", + "pin-project-lite", + "quinn-proto", + "quinn-udp", + "rustc-hash", + "rustls", + "socket2", + "thiserror", + "tokio", + "tracing", + "web-time", +] + +[[package]] +name = "quinn-proto" +version = "0.11.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560" +dependencies = [ + "bytes", + "getrandom 0.4.3", + "lru-slab", + "rand 0.10.2", + "rand_pcg", + "ring", + "rustc-hash", + "rustls", + "rustls-pki-types", + "slab", + "thiserror", + "tinyvec", + "tracing", + "web-time", +] + +[[package]] +name = "quinn-udp" +version = "0.5.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "35a133f956daabe89a61a685c2649f13d82d5aa4bd5d12d1277e1072a21c0694" +dependencies = [ + "cfg_aliases", + "libc", + "once_cell", + "socket2", + "tracing", + "windows-sys 0.61.2", +] + +[[package]] +name = "quote" +version = "1.0.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "rand" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41" +dependencies = [ + "rand_chacha", + "rand_core 0.9.5", +] + +[[package]] +name = "rand" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80" +dependencies = [ + "chacha20", + "getrandom 0.4.3", + "rand_core 0.10.1", +] + +[[package]] +name = "rand_chacha" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" +dependencies = [ + "ppv-lite86", + "rand_core 0.9.5", +] + +[[package]] +name = "rand_core" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" +dependencies = [ + "getrandom 0.3.4", +] + +[[package]] +name = "rand_core" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" + +[[package]] +name = "rand_pcg" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a" +dependencies = [ + "rand_core 0.10.1", +] + +[[package]] +name = "recursive" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0786a43debb760f491b1bc0269fe5e84155353c67482b9e60d0cfb596054b43e" +dependencies = [ + "recursive-proc-macro-impl", + "stacker", +] + +[[package]] +name = "recursive-proc-macro-impl" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76009fbe0614077fc1a2ce255e3a1881a2e3a3527097d5dc6d8212c585e7e38b" +dependencies = [ + "quote", + "syn", +] + +[[package]] +name = "redox_syscall" +version = "0.5.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" +dependencies = [ + "bitflags", +] + +[[package]] +name = "regex" +version = "1.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2a0e75113e14dc5acb068cd0786884f214f1312650a3d36d269f5c4f3cdee8a2" +dependencies = [ + "aho-corasick", + "memchr", + "regex-automata", + "regex-syntax", +] + +[[package]] +name = "regex-automata" +version = "0.4.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1f388202e4b80542a0921078cc23b6333bcf1409c1e3f86404cae4766a6131db" +dependencies = [ + "aho-corasick", + "memchr", + "regex-syntax", +] + +[[package]] +name = "regex-syntax" +version = "0.8.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" + +[[package]] +name = "relative-path" +version = "1.9.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba39f3699c378cd8970968dcbff9c43159ea4cfbd88d43c00b22f2ef10a435d2" + +[[package]] +name = "reqwest" +version = "0.12.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147" +dependencies = [ + "base64", + "bytes", + "futures-core", + "futures-util", + "h2", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-rustls", + "hyper-util", + "js-sys", + "log", + "percent-encoding", + "pin-project-lite", + "quinn", + "rustls", + "rustls-native-certs", + "rustls-pki-types", + "serde", + "serde_json", + "serde_urlencoded", + "sync_wrapper", + "tokio", + "tokio-rustls", + "tokio-util", + "tower", + "tower-http", + "tower-service", + "url", + "wasm-bindgen", + "wasm-bindgen-futures", + "wasm-streams", + "web-sys", +] + +[[package]] +name = "reqwest" +version = "0.13.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "219c5811de6525e5416c7d5d53bb656d3afdbc6c5af816e0802bcfa42dbdc1c3" +dependencies = [ + "base64", + "bytes", + "futures-core", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-util", + "js-sys", + "log", + "percent-encoding", + "pin-project-lite", + "sync_wrapper", + "tokio", + "tower", + "tower-http", + "tower-service", + "url", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", +] + +[[package]] +name = "ring" +version = "0.17.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7" +dependencies = [ + "cc", + "cfg-if", + "getrandom 0.2.17", + "libc", + "untrusted", + "windows-sys 0.52.0", +] + +[[package]] +name = "roaring" +version = "0.11.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1dedc5658c6ecb3bdb5ef5f3295bb9253f42dcf3fd1402c03f6b1f7659c3c4a9" +dependencies = [ + "bytemuck", + "byteorder", +] + +[[package]] +name = "rstest" +version = "0.23.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0a2c585be59b6b5dd66a9d2084aa1d8bd52fbdb806eafdeffb52791147862035" +dependencies = [ + "futures", + "futures-timer", + "rstest_macros", + "rustc_version", +] + +[[package]] +name = "rstest_macros" +version = "0.23.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "825ea780781b15345a146be27eaefb05085e337e869bff01b4306a4fd4a9ad5a" +dependencies = [ + "cfg-if", + "glob", + "proc-macro-crate", + "proc-macro2", + "quote", + "regex", + "relative-path", + "rustc_version", + "syn", + "unicode-ident", +] + +[[package]] +name = "rustc-hash" +version = "2.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d" + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "rustix" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" +dependencies = [ + "bitflags", + "errno", + "libc", + "linux-raw-sys", + "windows-sys 0.61.2", +] + +[[package]] +name = "rustls" +version = "0.23.42" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3c54fcab019b409d04215d3a17cb438fd7fbf192ee61461f20f4fe18704bc138" +dependencies = [ + "once_cell", + "ring", + "rustls-pki-types", + "rustls-webpki", + "subtle", + "zeroize", +] + +[[package]] +name = "rustls-native-certs" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dab5152771c58876a2146916e53e35057e1a4dfa2b9df0f0305b07f611fdea4d" +dependencies = [ + "openssl-probe", + "rustls-pki-types", + "schannel", + "security-framework", +] + +[[package]] +name = "rustls-pki-types" +version = "1.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "764899a24af3980067ee14bc143654f297b22eaebfe3c7b6b211920a5a59b046" +dependencies = [ + "web-time", + "zeroize", +] + +[[package]] +name = "rustls-webpki" +version = "0.103.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" +dependencies = [ + "ring", + "rustls-pki-types", + "untrusted", +] + +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + +[[package]] +name = "ryu" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" + +[[package]] +name = "same-file" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" +dependencies = [ + "winapi-util", +] + +[[package]] +name = "schannel" +version = "0.1.29" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91c1b7e4904c873ef0710c1f407dde2e6287de2bebc1bbbf7d430bb7cbffd939" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "security-framework" +version = "3.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d" +dependencies = [ + "bitflags", + "core-foundation", + "core-foundation-sys", + "libc", + "security-framework-sys", +] + +[[package]] +name = "security-framework-sys" +version = "2.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2691df843ecc5d231c0b14ece2acc3efb62c0a398c7e1d875f3983ce020e3" +dependencies = [ + "core-foundation-sys", + "libc", +] + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "seq-macro" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "serde_json" +version = "1.0.150" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "serde_urlencoded" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" +dependencies = [ + "form_urlencoded", + "itoa", + "ryu", + "serde", +] + +[[package]] +name = "sha2" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest 0.11.3", +] + +[[package]] +name = "sharded-slab" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6" +dependencies = [ + "lazy_static", +] + +[[package]] +name = "shlex" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" + +[[package]] +name = "simd-adler32" +version = "0.3.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea" + +[[package]] +name = "simdutf8" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e" + +[[package]] +name = "siphasher" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649" + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "smallvec" +version = "1.15.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" + +[[package]] +name = "snap" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b6b67fb9a61334225b5b790716f609cd58395f895b3fe8b328786812a40bc3b" + +[[package]] +name = "socket2" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + +[[package]] +name = "sqlparser" +version = "0.62.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c6d1b651dc4edf07eead2a0c6c78016ce971bc2c10da5266861b13f25e7cec" +dependencies = [ + "log", + "recursive", + "sqlparser_derive", +] + +[[package]] +name = "sqlparser_derive" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6dd45d8fc1c79299bfbb7190e42ccbbdf6a5f52e4a6ad98d92357ea965bd289" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "stacker" +version = "0.1.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "640c8cdd92b6b12f5bcb1803ca3bbf5ab96e5e6b6b96b9ab77dabe9e880b3190" +dependencies = [ + "cc", + "cfg-if", + "libc", + "psm", + "windows-sys 0.61.2", +] + +[[package]] +name = "strum" +version = "0.28.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9628de9b8791db39ceda2b119bbe13134770b56c138ec1d3af810d045c04f9bd" +dependencies = [ + "strum_macros", +] + +[[package]] +name = "strum_macros" +version = "0.28.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ab85eea0270ee17587ed4156089e10b9e6880ee688791d45a905f5b1ca36f664" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "subtle" +version = "2.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "sync_wrapper" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263" +dependencies = [ + "futures-core", +] + +[[package]] +name = "synstructure" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "tempfile" +version = "3.27.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" +dependencies = [ + "fastrand", + "getrandom 0.4.3", + "once_cell", + "rustix", + "windows-sys 0.61.2", +] + +[[package]] +name = "thiserror" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "thread_local" +version = "1.1.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ad99c4c6d32803332c548b1af0540b357b3f5fc0be8f6c6bfe8b2e6ae784070" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "thrift" +version = "0.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e54bc85fc7faa8bc175c4bab5b92ba8d9a3ce893d0e9f42cc455c8ab16a9e09" +dependencies = [ + "byteorder", + "integer-encoding", + "ordered-float", +] + +[[package]] +name = "tiny-keccak" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2c9d3793400a45f954c52e73d068316d76b6f4e36977e3fcebb13a2721e80237" +dependencies = [ + "crunchy", +] + +[[package]] +name = "tinystr" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d" +dependencies = [ + "displaydoc", + "zerovec", +] + +[[package]] +name = "tinyvec" +version = "1.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb4ebadaa0af04fab11ae01eb5f9fdb5f9c5b875506e210e71c07873528baa7f" +dependencies = [ + "tinyvec_macros", +] + +[[package]] +name = "tinyvec_macros" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" + +[[package]] +name = "tokio" +version = "1.52.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fc7f01b389ac15039e4dc9531aa973a135d7a4135281b12d7c1bc79fd57fffe" +dependencies = [ + "bytes", + "libc", + "mio", + "pin-project-lite", + "socket2", + "tokio-macros", + "windows-sys 0.61.2", +] + +[[package]] +name = "tokio-macros" +version = "2.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "385a6cb71ab9ab790c5fe8d67f1645e6c450a7ce006a33de03daa956cf70a496" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "tokio-rustls" +version = "0.26.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" +dependencies = [ + "rustls", + "tokio", +] + +[[package]] +name = "tokio-stream" +version = "0.1.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32da49809aab5c3bc678af03902d4ccddea2a87d028d86392a4b1560c6906c70" +dependencies = [ + "futures-core", + "pin-project-lite", + "tokio", + "tokio-util", +] + +[[package]] +name = "tokio-util" +version = "0.7.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ae9cec805b01e8fc3fd2fe289f89149a9b66dd16786abd8b19cfa7b48cb0098" +dependencies = [ + "bytes", + "futures-core", + "futures-sink", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "toml_datetime" +version = "1.1.1+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3165f65f62e28e0115a00b2ebdd37eb6f3b641855f9d636d3cd4103767159ad7" +dependencies = [ + "serde_core", +] + +[[package]] +name = "toml_edit" +version = "0.25.13+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6975367e4d2ef766d86af01ffad14b622fecc8d4357a998fbc4deb6e9bacaf9b" +dependencies = [ + "indexmap", + "toml_datetime", + "toml_parser", + "winnow", +] + +[[package]] +name = "toml_parser" +version = "1.1.2+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a2abe9b86193656635d2411dc43050282ca48aa31c2451210f4202550afb7526" +dependencies = [ + "winnow", +] + +[[package]] +name = "tower" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4" +dependencies = [ + "futures-core", + "futures-util", + "pin-project-lite", + "sync_wrapper", + "tokio", + "tower-layer", + "tower-service", +] + +[[package]] +name = "tower-http" +version = "0.6.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840" +dependencies = [ + "bitflags", + "bytes", + "futures-util", + "http", + "http-body", + "pin-project-lite", + "tower", + "tower-layer", + "tower-service", + "url", +] + +[[package]] +name = "tower-layer" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e" + +[[package]] +name = "tower-service" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" + +[[package]] +name = "tracing" +version = "0.1.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100" +dependencies = [ + "log", + "pin-project-lite", + "tracing-attributes", + "tracing-core", +] + +[[package]] +name = "tracing-attributes" +version = "0.1.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "tracing-core" +version = "0.1.36" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a" +dependencies = [ + "once_cell", + "valuable", +] + +[[package]] +name = "tracing-log" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ee855f1f400bd0e5c02d150ae5de3840039a3f54b025156404e34c23c03f47c3" +dependencies = [ + "log", + "once_cell", + "tracing-core", +] + +[[package]] +name = "tracing-subscriber" +version = "0.3.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb7f578e5945fb242538965c2d0b04418d38ec25c79d160cd279bf0731c8d319" +dependencies = [ + "nu-ansi-term", + "sharded-slab", + "smallvec", + "thread_local", + "tracing-core", + "tracing-log", +] + +[[package]] +name = "try-lock" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" + +[[package]] +name = "twox-hash" +version = "2.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c" + +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "unicode-segmentation" +version = "1.13.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6f5d3c3b1bf09027a88a6bc961fc00497d651009560b5463668dc81b0fa87a8" + +[[package]] +name = "unicode-width" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254" + +[[package]] +name = "untrusted" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" + +[[package]] +name = "url" +version = "2.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed" +dependencies = [ + "form_urlencoded", + "idna", + "percent-encoding", + "serde", +] + +[[package]] +name = "utf8_iter" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" + +[[package]] +name = "uuid" +version = "1.23.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ea5fab0d6c3c01ae70085a09cb03d4c7a1d6314e2b3e075392783396d724ca0a" +dependencies = [ + "getrandom 0.4.3", + "js-sys", + "rand 0.10.2", + "wasm-bindgen", +] + +[[package]] +name = "valuable" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65" + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "walkdir" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" +dependencies = [ + "same-file", + "winapi-util", +] + +[[package]] +name = "want" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e" +dependencies = [ + "try-lock", +] + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "wasip2" +version = "1.0.4+wasi-0.2.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487" +dependencies = [ + "wit-bindgen", +] + +[[package]] +name = "wasm-bindgen" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-futures" +version = "0.4.76" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c62df1340f32221cb9c54d6a27b030e3dba64361d4a95bed55f9aacb44da291d" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "wasm-streams" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "15053d8d85c7eccdbefef60f06769760a563c7f0a9d6902a13d35c7800b0ad65" +dependencies = [ + "futures-util", + "js-sys", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", +] + +[[package]] +name = "web-sys" +version = "0.3.103" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8622dcb61c0bcc9fffa6938bed81210af2da9a7e4a1a834b2e37a59b6dfb6141" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "web-time" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "winapi-util" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "windows-core" +version = "0.62.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" +dependencies = [ + "windows-implement", + "windows-interface", + "windows-link", + "windows-result", + "windows-strings", +] + +[[package]] +name = "windows-implement" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "windows-interface" +version = "0.59.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-result" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-strings" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-sys" +version = "0.52.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" +dependencies = [ + "windows-targets", +] + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-targets" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" +dependencies = [ + "windows_aarch64_gnullvm", + "windows_aarch64_msvc", + "windows_i686_gnu", + "windows_i686_gnullvm", + "windows_i686_msvc", + "windows_x86_64_gnu", + "windows_x86_64_gnullvm", + "windows_x86_64_msvc", +] + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" + +[[package]] +name = "windows_i686_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" + +[[package]] +name = "windows_i686_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" + +[[package]] +name = "winnow" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23b97319f7b8343df12cc98938e5c3eb436064524c8d2b4e30a1d3a36eecdf81" +dependencies = [ + "memchr", +] + +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + +[[package]] +name = "writeable" +version = "0.6.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" + +[[package]] +name = "yoke" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "709fe23a0424b6a435d82152b1bd3fdfb0833487d5fa90d05d42762a9891fef5" +dependencies = [ + "stable_deref_trait", + "yoke-derive", + "zerofrom", +] + +[[package]] +name = "yoke-derive" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" +dependencies = [ + "proc-macro2", + "quote", + "syn", + "synstructure", +] + +[[package]] +name = "z85" +version = "3.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6e61e59a957b7ccee15d2049f86e8bfd6f66968fcd88f018950662d9b86e675" + +[[package]] +name = "zerocopy" +version = "0.8.54" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7cbbc0a705a0fd05cc3676525980d2bf5a9bc4adac6d6475209a7887cf59d19" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.54" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2e817b7b52d0c7358d3246da9d69935ebb18116b2b102b4230dac079b4862f5" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "zerofrom" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272" +dependencies = [ + "zerofrom-derive", +] + +[[package]] +name = "zerofrom-derive" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" +dependencies = [ + "proc-macro2", + "quote", + "syn", + "synstructure", +] + +[[package]] +name = "zeroize" +version = "1.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e" + +[[package]] +name = "zerotrie" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf" +dependencies = [ + "displaydoc", + "yoke", + "zerofrom", +] + +[[package]] +name = "zerovec" +version = "0.11.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239" +dependencies = [ + "yoke", + "zerofrom", + "zerovec-derive", +] + +[[package]] +name = "zerovec-derive" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "zlib-rs" +version = "0.6.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b142a20ec14a91d5bc708c1dc21b080c550113d8aa77afa29635673a65dd02c5" + +[[package]] +name = "zmij" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" + +[[package]] +name = "zstd" +version = "0.13.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a" +dependencies = [ + "zstd-safe", +] + +[[package]] +name = "zstd-safe" +version = "7.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d" +dependencies = [ + "zstd-sys", +] + +[[package]] +name = "zstd-sys" +version = "2.0.16+zstd.1.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" +dependencies = [ + "cc", + "pkg-config", +] diff --git a/datafusion-executor/Cargo.toml b/datafusion-executor/Cargo.toml new file mode 100644 index 0000000000..94bc989436 --- /dev/null +++ b/datafusion-executor/Cargo.toml @@ -0,0 +1,23 @@ +[package] +name = "datafusion_executor" +description = "DataFusion-based PlanExecutor for delta_kernel declarative plans." +edition = "2021" +homepage = "https://delta.io" +license = "Apache-2.0" +repository = "https://github.com/delta-io/delta-kernel-rs" +version = "0.26.0" +rust-version = "1.88" + +[workspace] + +[dependencies] +delta_kernel = { path = "../kernel", default-features = false, features = [ + "declarative-plans", + "vendored-protoc", + "arrow-58", + "default-engine-base", +] } +datafusion = "54" + +[dev-dependencies] +rstest = "0.23" diff --git a/datafusion-executor/src/expression.rs b/datafusion-executor/src/expression.rs new file mode 100644 index 0000000000..89b268165b --- /dev/null +++ b/datafusion-executor/src/expression.rs @@ -0,0 +1,1864 @@ +//! Conversion from a kernel [`Expression`](KernelExpression) to a DataFusion [`Expr`](DFExpr). + +use std::hash::Hash; +use std::sync::Arc; + +use datafusion::arrow::array::{new_null_array, ArrayRef, RecordBatch, StructArray}; +use datafusion::arrow::compute::can_cast_types; +use datafusion::arrow::datatypes::{DataType as ArrowDataType, Schema as ArrowSchema}; +use datafusion::common::utils::take_function_args; +use datafusion::common::{ + Column as DFColumn, DFSchema, DataFusionError, ScalarValue as DFScalarValue, +}; +use datafusion::functions::core::expr_fn::{ + coalesce, get_field, get_field_path, named_struct, nullif, +}; +use datafusion::functions_nested::expr_fn::make_array; +use datafusion::logical_expr::expr_fn::try_cast; +use datafusion::logical_expr::{ + binary_expr, cast, lit, Case, ColumnarValue, Expr as DFExpr, ExprSchemable, Operator, + ScalarFunctionArgs, ScalarUDF, ScalarUDFImpl, Signature, Volatility, +}; +use delta_kernel::engine::arrow_conversion::TryIntoArrow; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::engine::arrow_expression::evaluate_expression as kernel_expression; +use delta_kernel::engine::parse_json; +use delta_kernel::expressions::{ + BinaryExpression, BinaryExpressionOp, CastExpression, CastOptions, + ColumnName as KernelColumnName, ElementAtExpression, Expression as KernelExpression, + ExpressionRef, ExpressionStructPatch, MapToStructExpression, MapToStructOptions, + ParseJsonExpression, UnaryExpressionOp, VariadicExpression, VariadicExpressionOp, +}; +use delta_kernel::schema::{ + DataType as KernelDataType, PrimitiveType, SchemaRef as KernelSchemaRef, StructField, + StructType, +}; +use delta_kernel::{DeltaResult, EngineData, Error}; + +use crate::predicate::to_df_predicate_expr; +use crate::scalar::to_df_scalar; + +/// Converts `expr` into the equivalent DataFusion [`Expr`](DFExpr), resolving column references +/// against `input_schema`. +/// +/// `output_type` supplies result type information that the expression itself does not carry. +/// `Struct` and `StructPatch` require a struct type for output field names and computed child +/// types. `Array` validates that a supplied type is an array and forwards its element type, while +/// `Coalesce` forwards the result type unchanged to every branch. Callers pass `None` when the +/// result type is unknown or the expression does not need it. +/// +/// # Errors +/// Returns an error when a column cannot be resolved, a scalar cannot be converted, supplied type +/// information is incompatible with the expression, or a `StructPatch` is inconsistent with its +/// input or output schema. Returns [`Error::unsupported`] when no DataFusion equivalent is +/// implemented. +pub fn to_df_expr( + expr: &KernelExpression, + input_schema: &StructType, + output_type: Option<&KernelDataType>, +) -> DeltaResult { + match expr { + KernelExpression::Literal(scalar) => Ok(lit(to_df_scalar(scalar)?)), + KernelExpression::Column(name) => column_to_df_expr(name, input_schema), + KernelExpression::Binary(binary) => binary_expr_to_df_expr(binary, input_schema), + KernelExpression::Variadic(variadic) => { + variadic_to_df_expr(variadic, input_schema, output_type) + } + KernelExpression::Predicate(pred) => to_df_predicate_expr(pred, input_schema), + KernelExpression::Struct(fields, nullability) => { + struct_to_df_expr(fields, nullability.as_ref(), input_schema, output_type) + } + KernelExpression::StructPatch(patch) => { + struct_patch_to_df_expr(patch, input_schema, output_type) + } + KernelExpression::MapToStruct(map_to_struct) => { + map_to_struct_to_df_expr(map_to_struct, input_schema, output_type) + } + KernelExpression::ElementAt(element_at) => element_at_to_df_expr(element_at, input_schema), + KernelExpression::ParseJson(parse) => parse_json_to_df_expr(parse, input_schema), + + KernelExpression::Unary(u) => match u.op { + UnaryExpressionOp::ToJson => Err(Error::unsupported( + "converting the ToJson expression is not yet supported", + )), + }, + + KernelExpression::Cast(cast_expr) => cast_to_df_expr(cast_expr, input_schema), + + KernelExpression::Opaque(_) => Err(Error::unsupported( + "cannot convert an engine-defined Opaque expression", + )), + KernelExpression::Unknown(name) => Err(Error::unsupported(format!( + "cannot convert Unknown expression {name:?}" + ))), + } +} + +/// Lowers an `ElementAt` map lookup to a UDF backed by kernel's Arrow evaluator. +/// +/// # Errors +/// Returns an error when either child expression cannot be converted. +fn element_at_to_df_expr( + element_at: &ElementAtExpression, + input_schema: &StructType, +) -> DeltaResult { + let map = to_df_expr(&element_at.map_expr, input_schema, None)?; + let key = to_df_expr( + &element_at.key_expr, + input_schema, + Some(&KernelDataType::STRING), + )?; + Ok(ScalarUDF::new_from_impl(ElementAtUdf::new()).call(vec![map, key])) +} + +fn evaluate_kernel_udf( + num_rows: usize, + args: [(&'static str, ColumnarValue); N], + expression: KernelExpression, + result_type: &KernelDataType, +) -> Result { + let columns = args + .into_iter() + .map(|(name, value)| Ok((name, value.into_array(num_rows)?))) + .collect::, DataFusionError>>()?; + let batch = RecordBatch::try_from_iter(columns)?; + let result = kernel_expression::evaluate_expression(&expression, &batch, Some(result_type)) + .map_err(|error| DataFusionError::External(Box::new(error)))?; + Ok(ColumnarValue::Array(result)) +} + +/// A DataFusion scalar UDF that delegates `ElementAt` evaluation to kernel, preserving dynamic-key +/// and rightmost-duplicate semantics across executors. +#[derive(Debug, PartialEq, Eq, Hash)] +struct ElementAtUdf { + signature: Signature, +} + +impl ElementAtUdf { + fn new() -> Self { + Self { + signature: Signature::any(2, Volatility::Immutable), + } + } +} + +impl ScalarUDFImpl for ElementAtUdf { + fn name(&self) -> &str { + "kernel_element_at" + } + + fn signature(&self) -> &Signature { + &self.signature + } + + fn return_type(&self, _arg_types: &[ArrowDataType]) -> Result { + Ok(ArrowDataType::Utf8) + } + + fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result { + let num_rows = args.number_rows; + let [map, key] = take_function_args(self.name(), args.args)?; + let expression = KernelExpression::element_at( + KernelExpression::column(["map"]), + KernelExpression::column(["key"]), + ); + evaluate_kernel_udf( + num_rows, + [("map", map), ("key", key)], + expression, + &KernelDataType::STRING, + ) + } +} + +/// Lowers a `Cast` to DataFusion while retaining its optional timestamp timezone. +/// +/// # Errors +/// Returns an error when the child expression or either schema cannot be converted. +fn cast_to_df_expr(cast_expr: &CastExpression, input_schema: &StructType) -> DeltaResult { + let value = to_df_expr(&cast_expr.expr, input_schema, None)?; + let target = (&cast_expr.target).try_into_arrow()?; + let arrow_input_schema: ArrowSchema = input_schema.try_into_arrow()?; + let df_input_schema = DFSchema::try_from(arrow_input_schema).map_err(Error::generic_err)?; + let source = value + .get_type(&df_input_schema) + .map_err(Error::generic_err)?; + let timezone_cast = cast_expr.options.timestamp_timezone().is_some() + && cast_expr.target == KernelDataType::TIMESTAMP + && matches!( + source, + ArrowDataType::Utf8 | ArrowDataType::LargeUtf8 | ArrowDataType::Utf8View + ); + if timezone_cast { + let udf = KernelCastUdf::try_new(cast_expr.target.clone(), cast_expr.options.clone())?; + return Ok(ScalarUDF::new_from_impl(udf).call(vec![value])); + } + if can_cast_types(&source, &target) { + return Ok(try_cast(value, target)); + } + let udf = KernelCastUdf::try_new(cast_expr.target.clone(), cast_expr.options.clone())?; + Ok(ScalarUDF::new_from_impl(udf).call(vec![value])) +} + +/// A DataFusion scalar UDF that delegates casts requiring kernel semantics to its Arrow evaluator. +#[derive(Debug, PartialEq, Eq)] +struct KernelCastUdf { + target: KernelDataType, + options: CastOptions, + return_type: ArrowDataType, + signature: Signature, +} + +impl Hash for KernelCastUdf { + fn hash(&self, state: &mut H) { + self.return_type.hash(state); + self.options.hash(state); + } +} + +impl KernelCastUdf { + fn try_new(target: KernelDataType, options: CastOptions) -> DeltaResult { + let return_type = (&target).try_into_arrow()?; + Ok(Self { + target, + options, + return_type, + signature: Signature::any(1, Volatility::Immutable), + }) + } +} + +impl ScalarUDFImpl for KernelCastUdf { + fn name(&self) -> &str { + "kernel_cast" + } + + fn signature(&self) -> &Signature { + &self.signature + } + + fn return_type(&self, _arg_types: &[ArrowDataType]) -> Result { + Ok(self.return_type.clone()) + } + + fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result { + let num_rows = args.number_rows; + let [value] = take_function_args(self.name(), args.args)?; + let expression = KernelExpression::cast( + KernelExpression::column(["value"]), + self.target.clone(), + self.options.clone(), + ); + evaluate_kernel_udf(num_rows, [("value", value)], expression, &self.target) + } +} + +/// Lowers a column reference to a nested field access, e.g. `a.b.c` becomes a single +/// `get_field(col("a"), "b", "c")` call. The path is resolved against `input_schema` (via +/// [`StructType::field_at`]) to fail fast, but the resolved field is otherwise unused. +fn column_to_df_expr(name: &KernelColumnName, input_schema: &StructType) -> DeltaResult { + let _ = input_schema.field_at(name)?; + let mut path = name.iter(); + let Some(root) = path.next() else { + return Err(Error::generic("cannot convert an empty column reference")); + }; + let root = DFExpr::Column(DFColumn::new_unqualified(root)); + let field_names = Vec::from_iter(path.map(lit)); + // A bare column stays a bare column; only nested access wraps it in a `get_field` call. + if field_names.is_empty() { + Ok(root) + } else { + Ok(get_field_path(root, field_names)) + } +} + +/// Lowers an arithmetic binary expression (`Plus`/`Minus`/`Multiply`/`Divide`) to an +/// `Expr::BinaryExpr`. Comparison and `IN` operators are modeled as predicates, not expressions, +/// so they never reach this arm. +fn binary_expr_to_df_expr( + binary: &BinaryExpression, + input_schema: &StructType, +) -> DeltaResult { + let op = match binary.op { + BinaryExpressionOp::Plus => Operator::Plus, + BinaryExpressionOp::Minus => Operator::Minus, + BinaryExpressionOp::Multiply => Operator::Multiply, + BinaryExpressionOp::Divide => Operator::Divide, + }; + let left = to_df_expr(&binary.left, input_schema, None)?; + let right = to_df_expr(&binary.right, input_schema, None)?; + Ok(binary_expr(left, op, right)) +} + +/// Lowers a variadic expression: `Coalesce` to `coalesce(..)` and `Array` to `make_array(..)`, each +/// over the converted arguments. Coalesce is type-preserving, so it forwards `output_type` to each +/// argument (every branch produces the same type). Array is type-wrapping: a known `Array` +/// target is peeled to `E` and threaded to each element (so an array of structs still gets its +/// element schema); an unknown target leaves elements untyped. +fn variadic_to_df_expr( + variadic: &VariadicExpression, + input_schema: &StructType, + output_type: Option<&KernelDataType>, +) -> DeltaResult { + let arg_output_type = match variadic.op { + VariadicExpressionOp::Coalesce => output_type, + VariadicExpressionOp::Array => match output_type { + Some(KernelDataType::Array(arr)) => Some(arr.element_type()), + Some(other) => { + return Err(Error::unsupported(format!( + "converting an Array expression requires an array output type, got {other:?}" + ))) + } + None => None, + }, + }; + let args: DeltaResult> = variadic + .exprs + .iter() + .map(|e| to_df_expr(e, input_schema, arg_output_type)) + .collect(); + match variadic.op { + VariadicExpressionOp::Coalesce => Ok(coalesce(args?)), + VariadicExpressionOp::Array => Ok(make_array(args?)), + } +} + +/// Extracts the target struct type for a struct-shaped arm from the caller's `output_type`, +/// erroring if it is absent or not a [`KernelDataType::Struct`]. +fn require_struct_output<'a>( + output_type: Option<&'a KernelDataType>, + arm: &str, +) -> DeltaResult<&'a StructType> { + match output_type { + Some(KernelDataType::Struct(schema)) => Ok(schema), + Some(other) => Err(Error::unsupported(format!( + "converting a {arm} expression requires a struct output type, got {other:?}" + ))), + None => Err(Error::unsupported(format!( + "converting a {arm} expression requires a struct output type" + ))), + } +} + +/// `CASE WHEN guard THEN body ELSE NULL END`: nulls the whole struct where `guard` is not true, +/// matching kernel's row-level struct-null mask. The else is an untyped NULL so CASE coercion +/// promotes it to `body`'s (all-nullable) struct type rather than forcing a nullability match. +fn struct_null_when_not(guard: DFExpr, body: DFExpr) -> DFExpr { + DFExpr::Case(Case::new( + None, + vec![(Box::new(guard), Box::new(body))], + Some(Box::new(lit(DFScalarValue::Null))), + )) +} + +/// Lowers a struct constructor to `named_struct(..)`, taking field names and per-child target types +/// from `output_type`. An optional nullability predicate nulls the whole struct where it is not +/// true. +fn struct_to_df_expr( + fields: &[ExpressionRef], + nullability: Option<&ExpressionRef>, + input_schema: &StructType, + output_type: Option<&KernelDataType>, +) -> DeltaResult { + let target = require_struct_output(output_type, "Struct")?; + if fields.len() != target.num_fields() { + return Err(Error::generic(format!( + "Struct expression field count mismatch: {} fields in expression but {} in schema", + fields.len(), + target.num_fields() + ))); + } + // `named_struct` takes one flat arg list of alternating names and values: + // `[name1, value1, name2, value2, ...]`, hence two args per field. + let mut args = Vec::with_capacity(fields.len() * 2); + for (child, field) in fields.iter().zip(target.fields()) { + args.push(lit(field.name().to_string())); + args.push(to_df_expr(child, input_schema, Some(field.data_type()))?); + } + let body = named_struct(args); + let Some(pred) = nullability else { + return Ok(body); + }; + let guard = to_df_expr(pred, input_schema, None)?; + Ok(struct_null_when_not(guard, body)) +} + +/// Lowers a struct patch (a sparse edit of an input struct) to a `named_struct(..)` rebuild. Output +/// field names come positionally from `output_type`, whose corresponding field types are forwarded +/// when lowering computed values. Walks the evaluator's emission order: prepends, each input field +/// (passed through unless dropped/replaced, then its insertions), and appends. A nested patch +/// (`input_path` set) nulls the output where the source struct row is null, matching the +/// evaluator's preservation of the source struct's null buffer. +fn struct_patch_to_df_expr( + patch: &ExpressionStructPatch, + input_schema: &StructType, + output_type: Option<&KernelDataType>, +) -> DeltaResult { + let target = require_struct_output(output_type, "StructPatch")?; + + // A patch targets either the whole input struct (`input_path` is `None`), whose fields are the + // top-level columns, or the nested struct at that path, whose fields are reached through it. + let (mut source_struct, mut source_expr) = (input_schema, None); + if let Some(path) = patch.input_path() { + let KernelDataType::Struct(nested) = input_schema.field_at(path)?.data_type() else { + return Err(Error::generic(format!( + "StructPatch input_path '{path}' does not resolve to a struct" + ))); + }; + let source = column_to_df_expr(path, input_schema)?; + (source_struct, source_expr) = (nested.as_ref(), Some(source)); + } + + // Append `[name, value]` pairs in the evaluator's emission order, consuming one output field + // per appended value so each value is lowered against the type it lands in. + let mut output_fields = target.fields(); + let mut args = Vec::with_capacity(target.num_fields() * 2); + + // Both closures need the shared `output_fields` cursor, so it is threaded as a parameter rather + // than captured. + let append_field_with_converted_expr = + |args: &mut Vec, + output_fields: &mut dyn Iterator, + expr: &KernelExpression| + -> DeltaResult<()> { + let field = output_fields.next().ok_or_else(|| { + Error::generic("StructPatch produced more fields than the output schema has") + })?; + let value = to_df_expr(expr, input_schema, Some(field.data_type()))?; + args.push(lit(field.name().to_string())); + args.push(value); + Ok(()) + }; + let append_field_with_existing_col = |args: &mut Vec, + output_fields: &mut dyn Iterator, + name: &str| + -> DeltaResult<()> { + let field = output_fields.next().ok_or_else(|| { + Error::generic("StructPatch produced more fields than the output schema has") + })?; + let value = match &source_expr { + Some(base) => get_field(base.clone(), name.to_string()), + None => DFExpr::Column(DFColumn::new_unqualified(name)), + }; + args.push(lit(field.name().to_string())); + args.push(value); + Ok(()) + }; + + for expr in &patch.prepended_fields { + append_field_with_converted_expr(&mut args, &mut output_fields, expr)?; + } + + // Should only count required field patches (excluding optional) for missing input fields + // validation. An existing optional field can shadow a missing required field. + let mut used_required_field_patches = 0usize; + for input_field in source_struct.fields() { + let name = input_field.name(); + let field_patch = patch.field_patches.get(name); + + if field_patch.is_none_or(|fp| fp.keep_input) { + append_field_with_existing_col(&mut args, &mut output_fields, name)?; + } + + let Some(field_patch) = field_patch else { + continue; + }; + for expr in &field_patch.insertions { + append_field_with_converted_expr(&mut args, &mut output_fields, expr)?; + } + if !field_patch.optional { + used_required_field_patches += 1; + } + } + + let required = patch + .field_patches + .values() + .filter(|fp| !fp.optional) + .count(); + if used_required_field_patches < required { + return Err(Error::generic( + "StructPatch has non-optional field patches that reference missing input fields", + )); + } + + for expr in &patch.appended_fields { + append_field_with_converted_expr(&mut args, &mut output_fields, expr)?; + } + + if output_fields.next().is_some() { + return Err(Error::generic( + "StructPatch produced fewer fields than the output schema has", + )); + } + + let body = named_struct(args); + let Some(base) = source_expr else { + return Ok(body); + }; + Ok(struct_null_when_not(base.is_not_null(), body)) +} + +/// Lowers a `MapToStruct` (reshape a `Map` into a struct by parsing each value into +/// its target field type) to a DataFusion `named_struct(..)` rebuild. Field names and per-field +/// types come from `output_type`, which must be a struct holding only primitive fields (matching +/// the kernel evaluator, which supports only primitive targets). +/// +/// Each field extracts its value with `cast(get_field(map, name), T)`. For a numeric or temporal +/// type the raw value is first wrapped in `nullif(.., '')`, mapping an empty string to null before +/// the cast, so an empty string becomes null (kernel's `empty_string_partition_cast`) while an +/// unparseable value fails the cast (kernel's hard parse error). String and Binary keep the raw +/// value (empty is a valid empty string / empty bytes). A missing key or null value is already null +/// via [`get_field`]. The whole struct is nulled where the input map row is null, via ` IS NOT +/// NULL`. +/// +/// KNOWN DIVERGENCES from the kernel parser, all confined to malformed or non-spec-compliant input +/// (spec-compliant writers never emit any of these): +/// - Duplicate keys: `get_field` takes the leftmost entry, the kernel evaluator the rightmost. +/// - Boolean: arrow's cast also accepts `"yes"`/`"no"`/`"on"`/`"off"`/`"t"`/`"f"`/`"1"`/`"0"`, +/// while kernel accepts only `"true"`/`"false"`. +/// - Decimal: arrow's cast silently rescales/rounds to the target scale, while kernel requires the +/// value's scale to match the target's exactly (and hard-errors otherwise). +/// +/// # Errors +/// Returns an error when `output_type` is absent, not a struct, or has a non-primitive field, or +/// from lowering the map expression. +fn map_to_struct_to_df_expr( + map_to_struct: &MapToStructExpression, + input_schema: &StructType, + output_type: Option<&KernelDataType>, +) -> DeltaResult { + let target = require_struct_output(output_type, "MapToStruct")?; + let map = to_df_expr(&map_to_struct.map_expr, input_schema, None)?; + if map_to_struct.options.timestamp_timezone().is_some() { + let udf = + KernelMapToStructUdf::try_new(Arc::new(target.clone()), map_to_struct.options.clone())?; + return Ok(ScalarUDF::new_from_impl(udf).call(vec![map])); + } + + let mut args = Vec::with_capacity(target.num_fields() * 2); + for field in target.fields() { + let KernelDataType::Primitive(prim) = field.data_type() else { + return Err(Error::unsupported(format!( + "MapToStruct only supports primitive target types, but field '{}' is {:?}", + field.name(), + field.data_type() + ))); + }; + let raw = get_field(map.clone(), field.name().to_string()); + let value = match prim { + // An empty string is a value for these two (the empty string / empty bytes) and null + // for every other type based on kernel. + PrimitiveType::String | PrimitiveType::Binary => raw, + _ => nullif(raw, lit("")), + }; + let arrow_type = field + .data_type() + .try_into_arrow() + .map_err(Error::generic_err)?; + args.push(lit(field.name().to_string())); + args.push(cast(value, arrow_type)); + } + + Ok(struct_null_when_not(map.is_not_null(), named_struct(args))) +} + +/// A DataFusion scalar UDF that delegates timezone-aware partition parsing to kernel's Arrow +/// evaluator. +#[derive(Debug, PartialEq, Eq)] +struct KernelMapToStructUdf { + output_schema: KernelSchemaRef, + options: MapToStructOptions, + return_type: ArrowDataType, + signature: Signature, +} + +impl std::hash::Hash for KernelMapToStructUdf { + fn hash(&self, state: &mut H) { + for field in self.output_schema.fields() { + field.name().hash(state); + field.data_type().to_string().hash(state); + } + self.options.hash(state); + } +} + +impl KernelMapToStructUdf { + fn try_new(output_schema: KernelSchemaRef, options: MapToStructOptions) -> DeltaResult { + let arrow_schema: ArrowSchema = output_schema + .as_ref() + .try_into_arrow() + .map_err(Error::generic_err)?; + Ok(Self { + return_type: ArrowDataType::Struct(arrow_schema.fields().clone()), + output_schema, + options, + signature: Signature::any(1, Volatility::Immutable), + }) + } +} + +impl ScalarUDFImpl for KernelMapToStructUdf { + fn name(&self) -> &str { + "kernel_map_to_struct_with_timestamp_timezone" + } + + fn signature(&self) -> &Signature { + &self.signature + } + + fn return_type(&self, _arg_types: &[ArrowDataType]) -> Result { + Ok(self.return_type.clone()) + } + + fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result { + let num_rows = args.number_rows; + let [map] = take_function_args(self.name(), args.args)?; + let expression = KernelExpression::map_to_struct( + KernelExpression::column(["map"]), + self.options.clone(), + ); + let output_type = KernelDataType::from(self.output_schema.as_ref().clone()); + evaluate_kernel_udf(num_rows, [("map", map)], expression, &output_type) + } +} + +/// Lowers a `ParseJson` (parse a JSON-string column into a struct) to a call of the +/// [`ParseJsonUdf`] scalar UDF, which delegates to kernel's own JSON parser. Unlike the +/// struct-shaped arms, `ParseJson` is self-typed -- it carries its target `output_schema` -- so it +/// takes no `output_type` and lowers its string operand untyped. +fn parse_json_to_df_expr( + parse: &ParseJsonExpression, + input_schema: &StructType, +) -> DeltaResult { + let json = to_df_expr(&parse.json_expr, input_schema, None)?; + let udf = ScalarUDF::new_from_impl(ParseJsonUdf::try_new(parse.output_schema.clone())?); + Ok(udf.call(vec![json])) +} + +/// A DataFusion scalar UDF that parses a JSON-string column into a struct, delegating to kernel's +/// [`parse_json`] so the result is value-identical to the kernel evaluator by construction. Since a +/// [`ParseJsonExpression`] carries its own target schema, the schema is baked into the UDF instance +/// rather than passed as an argument. +/// +/// The UDF reproduces the coarse malformed-JSON backstop the evaluator applies around +/// `parse_json`: on a whole-batch parse error it returns an all-null struct rather than failing. +/// (The finer per-cell null for failure-prone leaves -- Timestamp/Date/Decimal -- already lives +/// inside `parse_json`.) +#[derive(Debug, PartialEq, Eq)] +struct ParseJsonUdf { + output_schema: KernelSchemaRef, + return_type: ArrowDataType, + signature: Signature, +} + +/// DataFusion requires scalar UDF implementations to support equality and hashing so UDF calls +/// can participate in expression comparison and optimizations such as common-subexpression +/// elimination. The output schema captures this UDF's schema-dependent behavior; its signature is +/// otherwise identical for every instance. +impl std::hash::Hash for ParseJsonUdf { + fn hash(&self, state: &mut H) { + for field in self.output_schema.fields() { + field.name().hash(state); + field.data_type().to_string().hash(state); + } + } +} + +impl ParseJsonUdf { + fn try_new(output_schema: KernelSchemaRef) -> DeltaResult { + let arrow_schema: ArrowSchema = output_schema + .as_ref() + .try_into_arrow() + .map_err(Error::generic_err)?; + Ok(Self { + return_type: ArrowDataType::Struct(arrow_schema.fields().clone()), + // Coerces Utf8 / LargeUtf8 / Utf8View, mirroring kernel's `parse_json_impl`. + signature: Signature::string(1, Volatility::Immutable), + output_schema, + }) + } +} + +impl ScalarUDFImpl for ParseJsonUdf { + fn name(&self) -> &str { + "kernel_parse_json" + } + + fn signature(&self) -> &Signature { + &self.signature + } + + fn return_type(&self, _arg_types: &[ArrowDataType]) -> Result { + Ok(self.return_type.clone()) + } + + fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result { + let num_rows = args.number_rows; + let [json] = take_function_args(self.name(), args.args)?; + let json = json.into_array(num_rows)?; + + // `parse_json` reads column 0 of an `EngineData`-wrapped batch; wrap the input to match. + let batch = RecordBatch::try_from_iter([("json", json)])?; + let input: Box = Box::new(ArrowEngineData::from(batch)); + + let parsed = match parse_json(input, self.output_schema.clone()) { + Ok(data) => { + let batch: RecordBatch = ArrowEngineData::try_from_engine_data(data) + .map_err(|e| DataFusionError::External(Box::new(e)))? + .into(); + Arc::new(StructArray::from(batch)) as ArrayRef + } + // Coarse malformed-JSON backstop, matching the evaluator's ParseJson arm. + Err(_) => new_null_array(&self.return_type, num_rows), + }; + Ok(ColumnarValue::Array(parsed)) + } +} + +#[cfg(test)] +mod tests { + use datafusion::arrow::array::{ + Array, AsArray, Int32Array, Int64Array, MapBuilder, StringArray, StringBuilder, + TimestampMicrosecondArray, + }; + use datafusion::arrow::datatypes::Field as ArrowField; + use datafusion::assert_batches_eq; + use datafusion::common::DFSchema; + use datafusion::physical_expr::create_physical_expr; + use datafusion::physical_expr::execution_props::ExecutionProps; + use delta_kernel::expressions::{ + col, lit, CastOptions, Expression as KernelExpr, ExpressionStructPatch, + ExpressionStructPatchBuilder, MapToStructOptions, + }; + use delta_kernel::schema::{ArrayType, DataType, MapType, StructField, StructType}; + use rstest::rstest; + + use super::*; + + /// Name-resolution scope for these tests: `a: { b: { c: long } }`, plus top-level `b` and `x`. + fn test_schema() -> StructType { + StructType::try_new([ + StructField::nullable( + "a", + StructType::try_new([StructField::nullable( + "b", + StructType::try_new([StructField::nullable("c", DataType::LONG)]).unwrap(), + )]) + .unwrap(), + ), + StructField::nullable("b", DataType::LONG), + StructField::nullable("x", DataType::LONG), + ]) + .unwrap() + } + + /// Lowers an expression against [`test_schema`] and renders it as a DataFusion `Display` + /// string. + fn lower(expr: KernelExpr) -> String { + to_df_expr(&expr, &test_schema(), None).unwrap().to_string() + } + + /// Lowers against [`test_schema`] targeting `output_type` and renders as a `Display` string. + fn lower_typed(expr: KernelExpr, output_type: DataType) -> String { + to_df_expr(&expr, &test_schema(), Some(&output_type)) + .unwrap() + .to_string() + } + + #[rstest] + #[case::i32(lit(7i32), "Int32(7)")] + #[case::i64(lit(42i64), "Int64(42)")] + #[case::string(lit("abc"), "Utf8(\"abc\")")] + #[case::boolean(lit(true), "Boolean(true)")] + #[case::null(KernelExpr::null_literal(DataType::LONG), "Int64(NULL)")] + fn literal_lowers_to_scalar(#[case] kernel: KernelExpr, #[case] expected: &str) { + assert_eq!(lower(kernel), expected); + } + + #[rstest] + #[case::single(col!("a"), "a")] + #[case::depth_2(col!("a.b"), "get_field(a, Utf8(\"b\"))")] + #[case::depth_3(col!("a.b.c"), "get_field(a, Utf8(\"b\"), Utf8(\"c\"))")] + fn column_lowers_to_nested_field_access(#[case] kernel: KernelExpr, #[case] expected: &str) { + assert_eq!(lower(kernel), expected); + } + + #[test] + fn cast_lowers_to_try_cast() { + let kernel = KernelExpr::cast(col!("x"), DataType::INTEGER, CastOptions::default()); + assert_eq!(lower(kernel), "TRY_CAST(x AS Int32)"); + } + + #[test] + fn cast_executes_with_safe_null_semantics() { + let input_schema = + StructType::try_new([StructField::nullable("s", DataType::STRING)]).unwrap(); + let arrow_schema: ArrowSchema = (&input_schema).try_into_arrow().unwrap(); + let batch = RecordBatch::try_new( + Arc::new(arrow_schema.clone()), + vec![Arc::new(StringArray::from(vec![ + Some("42"), + Some("bad"), + None, + ]))], + ) + .unwrap(); + let logical = to_df_expr( + &KernelExpr::cast(col!("s"), DataType::INTEGER, CastOptions::default()), + &input_schema, + Some(&DataType::INTEGER), + ) + .unwrap(); + let df_schema = DFSchema::try_from(arrow_schema).unwrap(); + let physical = create_physical_expr(&logical, &df_schema, &ExecutionProps::new()).unwrap(); + + let result = physical + .evaluate(&batch) + .unwrap() + .into_array(batch.num_rows()) + .unwrap(); + let values = result.as_any().downcast_ref::().unwrap(); + assert_eq!(values, &Int32Array::from(vec![Some(42), None, None])); + } + + #[test] + fn unsupported_cast_executes_as_typed_null() { + let input_schema = + StructType::try_new([StructField::nullable("x", DataType::LONG)]).unwrap(); + let target: DataType = + StructType::try_new([StructField::nullable("value", DataType::LONG)]) + .unwrap() + .into(); + let logical = to_df_expr( + &KernelExpr::cast(col!("x"), target.clone(), CastOptions::default()), + &input_schema, + Some(&target), + ) + .unwrap(); + + let arrow_schema: ArrowSchema = (&input_schema).try_into_arrow().unwrap(); + let batch = RecordBatch::try_new( + Arc::new(arrow_schema.clone()), + vec![Arc::new(Int64Array::from(vec![Some(1), None]))], + ) + .unwrap(); + let df_schema = DFSchema::try_from(arrow_schema).unwrap(); + let physical = create_physical_expr(&logical, &df_schema, &ExecutionProps::new()).unwrap(); + let result = physical + .evaluate(&batch) + .unwrap() + .into_array(batch.num_rows()) + .unwrap(); + let expected_type: ArrowDataType = (&target).try_into_arrow().unwrap(); + assert_eq!(result.data_type(), &expected_type); + assert_eq!(result.null_count(), batch.num_rows()); + } + + #[test] + fn unsupported_cast_evaluates_child_errors() { + let input_schema = + StructType::try_new([StructField::nullable("s", DataType::STRING)]).unwrap(); + let target: DataType = + StructType::try_new([StructField::nullable("value", DataType::LONG)]) + .unwrap() + .into(); + let child = KernelExpr::cast( + col!("s"), + DataType::TIMESTAMP, + CastOptions::default().with_timestamp_timezone("Not/AZone"), + ); + let logical = to_df_expr( + &KernelExpr::cast(child, target.clone(), CastOptions::default()), + &input_schema, + Some(&target), + ) + .unwrap(); + + let arrow_schema: ArrowSchema = (&input_schema).try_into_arrow().unwrap(); + let batch = RecordBatch::try_new( + Arc::new(arrow_schema.clone()), + vec![Arc::new(StringArray::from(vec![Some( + "2024-06-15 09:30:00", + )]))], + ) + .unwrap(); + let df_schema = DFSchema::try_from(arrow_schema).unwrap(); + let physical = create_physical_expr(&logical, &df_schema, &ExecutionProps::new()).unwrap(); + let error = physical.evaluate(&batch).unwrap_err(); + assert!(error.to_string().contains("Invalid timestamp timezone")); + } + + #[rstest] + #[case::plus(col!("a") + lit(1i64), "a + Int64(1)")] + #[case::minus(col!("a") - lit(1i64), "a - Int64(1)")] + #[case::multiply(col!("a") * lit(2i64), "a * Int64(2)")] + #[case::divide(col!("a") / lit(2i64), "a / Int64(2)")] + fn arithmetic_binary_lowers_to_binary_expr(#[case] kernel: KernelExpr, #[case] expected: &str) { + assert_eq!(lower(kernel), expected); + } + + /// Nested arithmetic lowers to the matching operator tree. + #[rstest] + #[case::precedence_pins_grouping( + (col!("x") + lit(1i64)) * (col!("b") - lit(2i64)), + "(x + Int64(1)) * (b - Int64(2))" + )] + #[case::nested_field_and_all_ops( + (col!("a.b.c") * lit(5i64) - (col!("b") + col!("x"))) / lit(20i64), + "(get_field(a, Utf8(\"b\"), Utf8(\"c\")) * Int64(5) - b + x) / Int64(20)" + )] + fn nested_arithmetic_lowers_to_operator_tree( + #[case] kernel: KernelExpr, + #[case] expected: &str, + ) { + assert_eq!(lower(kernel), expected); + } + + #[rstest] + #[case::coalesce( + KernelExpr::coalesce([col!("a"), col!("b"), lit(0i64)]), + "coalesce(a, b, Int64(0))" + )] + #[case::array( + KernelExpr::array([lit(1i64), lit(2i64)]), + "make_array(Int64(1), Int64(2))" + )] + #[case::nested_coalesce( + KernelExpr::coalesce([KernelExpr::coalesce([col!("a"), col!("b")]), col!("x")]), + "coalesce(coalesce(a, b), x)" + )] + #[case::nested_array( + KernelExpr::array([ + KernelExpr::array([lit(1i64), lit(2i64)]), + KernelExpr::array([lit(3i64), lit(4i64)]), + ]), + "make_array(make_array(Int64(1), Int64(2)), make_array(Int64(3), Int64(4)))" + )] + fn variadic_lowers_to_call(#[case] kernel: KernelExpr, #[case] expected: &str) { + assert_eq!(lower(kernel), expected); + } + + /// An array of structs peels the element type off the `Array` target and threads the + /// struct schema to each element, so the struct children get their field names. + #[test] + fn array_of_struct_threads_element_schema_to_each_element() { + let element = KernelExpr::struct_from([col!("b"), lit(1i64)]); + let kernel = KernelExpr::array([element]); + let target: DataType = ArrayType::new(pq_output_schema(), true).into(); + assert_eq!( + lower_typed(kernel, target), + "make_array(named_struct(Utf8(\"p\"), b, Utf8(\"q\"), Int64(1)))" + ); + } + + /// Nested `Array>`: the element type is peeled at each array level until the + /// struct schema reaches the leaf struct element. + #[test] + fn nested_array_of_array_peels_element_type_at_each_level() { + let inner = KernelExpr::array([KernelExpr::struct_from([col!("b")])]); + let kernel = KernelExpr::array([inner]); + let leaf = StructType::try_new([StructField::nullable("p", DataType::LONG)]).unwrap(); + let target: DataType = ArrayType::new(ArrayType::new(leaf, true), true).into(); + assert_eq!( + lower_typed(kernel, target), + "make_array(make_array(named_struct(Utf8(\"p\"), b)))" + ); + } + + /// An `Array` arm errors when it cannot resolve its element type: no target at all leaves a + /// struct element without field names (same as a bare `Struct`), and a non-array target has no + /// element type to peel. + #[rstest] + #[case::struct_element_without_target( + KernelExpr::array([KernelExpr::struct_from([col!("b")])]), + None + )] + #[case::non_array_target(KernelExpr::array([lit(1i64)]), Some(DataType::LONG))] + fn array_with_unresolvable_element_type_is_an_error( + #[case] kernel: KernelExpr, + #[case] output_type: Option, + ) { + to_df_expr(&kernel, &test_schema(), output_type.as_ref()).unwrap_err(); + } + + #[test] + fn embedded_predicate_delegates_to_predicate_converter() { + let kernel = KernelExpr::Predicate(Box::new(col!("b").is_null())); + assert_eq!(lower(kernel), "b IS NULL"); + } + + /// A column reference that does not resolve against the input schema fails at conversion time, + /// not later during DataFusion analysis. Covers each `field_at` failure mode. + #[rstest] + #[case::empty(KernelExpr::Column(KernelColumnName::default()))] + #[case::unknown_root(col!("nope"))] + #[case::unknown_nested(col!("a.b.missing"))] + #[case::descend_into_non_struct(col!("x.y"))] + fn unresolved_column_is_an_error(#[case] kernel: KernelExpr) { + to_df_expr(&kernel, &test_schema(), None).unwrap_err(); + } + + // === Struct === + + /// Output schema with names distinct from the input schema, proving names come from the target. + fn pq_output_schema() -> StructType { + StructType::try_new([ + StructField::nullable("p", DataType::LONG), + StructField::nullable("q", DataType::LONG), + ]) + .unwrap() + } + + #[test] + fn struct_lowers_to_named_struct_with_target_names() { + let kernel = KernelExpr::struct_from([col!("b"), lit(1i64)]); + assert_eq!( + lower_typed(kernel, pq_output_schema().into()), + "named_struct(Utf8(\"p\"), b, Utf8(\"q\"), Int64(1))" + ); + } + + #[test] + fn nested_struct_recurses_with_child_target_names() { + let inner = KernelExpr::struct_from([col!("b"), lit(1i64)]); + let kernel = KernelExpr::struct_from([inner]); + let target = + StructType::try_new([StructField::nullable("outer", pq_output_schema())]).unwrap(); + assert_eq!( + lower_typed(kernel, target.into()), + "named_struct(Utf8(\"outer\"), named_struct(Utf8(\"p\"), b, Utf8(\"q\"), Int64(1)))" + ); + } + + #[test] + fn struct_with_nullability_wraps_in_case() { + let kernel = KernelExpr::struct_with_nullability_from( + [col!("b"), lit(1i64)], + KernelExpr::Predicate(Box::new(col!("x").is_not_null())), + ); + // Kernel models IS NOT NULL as Not(IsNull), so the guard renders as "NOT x IS NULL". + let rendered = lower_typed(kernel, pq_output_schema().into()); + assert!( + rendered.starts_with("CASE WHEN NOT x IS NULL THEN named_struct("), + "{rendered}" + ); + assert!(rendered.ends_with("END"), "{rendered}"); + } + + #[test] + fn struct_without_target_is_unsupported() { + let kernel = KernelExpr::struct_from([col!("b")]); + to_df_expr(&kernel, &test_schema(), None).unwrap_err(); + } + + #[test] + fn struct_arity_mismatch_is_an_error() { + let kernel = KernelExpr::struct_from([col!("b"), lit(1i64)]); + let target: DataType = StructType::try_new([StructField::nullable("p", DataType::LONG)]) + .unwrap() + .into(); + to_df_expr(&kernel, &test_schema(), Some(&target)).unwrap_err(); + } + + // === Struct patch === + + /// Lowers a struct patch against `input`, targeting `output_schema`. + fn lower_patch( + patch: ExpressionStructPatch, + input: &StructType, + output_schema: &StructType, + ) -> String { + let expr = KernelExpr::struct_patch(patch).unwrap(); + let output_type: DataType = output_schema.clone().into(); + to_df_expr(&expr, input, Some(&output_type)) + .unwrap() + .to_string() + } + + /// Input struct `{ a: long, b: long }` for patch tests: the whole input schema for a top-level + /// patch, or the nested source struct for a nested one. + fn ab_schema() -> StructType { + StructType::try_new([ + StructField::nullable("a", DataType::LONG), + StructField::nullable("b", DataType::LONG), + ]) + .unwrap() + } + + /// Asserts `res` is an error whose message contains `message`. + #[track_caller] + fn assert_error_message(res: DeltaResult, message: &str) { + let error = res.err().expect("expected an error").to_string(); + assert!(error.contains(message), "{error}"); + } + + #[test] + fn empty_top_level_patch_passes_all_fields_through() { + let patch = ExpressionStructPatchBuilder::new().build().unwrap(); + assert_eq!( + lower_patch(patch, &ab_schema(), &pq_output_schema()), + "named_struct(Utf8(\"p\"), a, Utf8(\"q\"), b)" + ); + } + + #[test] + fn top_level_patch_replace_puts_expr_in_field_slot() { + let patch = ExpressionStructPatchBuilder::new() + .replace("a", lit(7i64)) + .build() + .unwrap(); + assert_eq!( + lower_patch(patch, &ab_schema(), &pq_output_schema()), + "named_struct(Utf8(\"p\"), Int64(7), Utf8(\"q\"), b)" + ); + } + + #[test] + fn top_level_patch_drop_removes_field() { + let patch = ExpressionStructPatchBuilder::new() + .drop("a") + .build() + .unwrap(); + let target = StructType::try_new([StructField::nullable("q", DataType::LONG)]).unwrap(); + assert_eq!( + lower_patch(patch, &ab_schema(), &target), + "named_struct(Utf8(\"q\"), b)" + ); + } + + #[test] + fn top_level_patch_prepend_and_append() { + let patch = ExpressionStructPatchBuilder::new() + .prepend(lit(0i64)) + .append(lit(9i64)) + .build() + .unwrap(); + let target = StructType::try_new([ + StructField::nullable("first", DataType::LONG), + StructField::nullable("a", DataType::LONG), + StructField::nullable("b", DataType::LONG), + StructField::nullable("last", DataType::LONG), + ]) + .unwrap(); + assert_eq!( + lower_patch(patch, &ab_schema(), &target), + "named_struct(Utf8(\"first\"), Int64(0), Utf8(\"a\"), a, Utf8(\"b\"), b, \ + Utf8(\"last\"), Int64(9))" + ); + } + + #[test] + fn top_level_patch_insert_after_field() { + let patch = ExpressionStructPatchBuilder::new() + .insert_after("a", lit(5i64)) + .build() + .unwrap(); + let target = StructType::try_new([ + StructField::nullable("a", DataType::LONG), + StructField::nullable("inserted", DataType::LONG), + StructField::nullable("b", DataType::LONG), + ]) + .unwrap(); + assert_eq!( + lower_patch(patch, &ab_schema(), &target), + "named_struct(Utf8(\"a\"), a, Utf8(\"inserted\"), Int64(5), Utf8(\"b\"), b)" + ); + } + + #[test] + fn nested_patch_wraps_in_null_guard_case() { + // Input schema: { s: { a: long, b: long } }. Patch replaces s.a with a literal. + let input = StructType::try_new([StructField::nullable("s", ab_schema())]).unwrap(); + let patch = ExpressionStructPatchBuilder::new_nested(["s"]) + .replace("a", lit(7i64)) + .build() + .unwrap(); + assert_eq!( + lower_patch(patch, &input, &pq_output_schema()), + "CASE WHEN s IS NOT NULL THEN named_struct(Utf8(\"p\"), Int64(7), Utf8(\"q\"), \ + get_field(s, Utf8(\"b\"))) ELSE NULL END" + ); + } + + #[test] + fn patch_too_many_output_fields_is_an_error() { + // Empty patch passes 2 fields; target declares 3. + let patch = ExpressionStructPatchBuilder::new().build().unwrap(); + let target: DataType = StructType::try_new([ + StructField::nullable("p", DataType::LONG), + StructField::nullable("q", DataType::LONG), + StructField::nullable("r", DataType::LONG), + ]) + .unwrap() + .into(); + let expr = KernelExpr::struct_patch(patch).unwrap(); + assert_error_message( + to_df_expr(&expr, &ab_schema(), Some(&target)), + "StructPatch produced fewer fields than the output schema has", + ); + } + + #[test] + fn patch_too_few_output_fields_is_an_error() { + // Empty patch passes 2 fields; target declares 1. + let patch = ExpressionStructPatchBuilder::new().build().unwrap(); + let target: DataType = StructType::try_new([StructField::nullable("p", DataType::LONG)]) + .unwrap() + .into(); + let expr = KernelExpr::struct_patch(patch).unwrap(); + assert_error_message( + to_df_expr(&expr, &ab_schema(), Some(&target)), + "StructPatch produced more fields than the output schema has", + ); + } + + #[test] + fn patch_without_target_is_unsupported() { + let patch = ExpressionStructPatchBuilder::new().build().unwrap(); + let expr = KernelExpr::struct_patch(patch).unwrap(); + assert_error_message( + to_df_expr(&expr, &ab_schema(), None), + "converting a StructPatch expression requires a struct output type", + ); + } + + #[rstest] + #[case::only_missing_required( + ExpressionStructPatchBuilder::new() + .replace("nonexistent", lit(1i64)) + .build() + .unwrap(), + pq_output_schema() + )] + #[case::matched_optional_does_not_mask_missing_required( + ExpressionStructPatchBuilder::new() + .drop_if_exists("b") + .replace("nonexistent", lit(1i64)) + .build() + .unwrap(), + StructType::try_new([StructField::nullable("p", DataType::LONG)]).unwrap() + )] + fn required_patch_on_missing_field_is_an_error( + #[case] patch: ExpressionStructPatch, + #[case] target: StructType, + ) { + let expr = KernelExpr::struct_patch(patch).unwrap(); + let target: DataType = target.into(); + assert_error_message( + to_df_expr(&expr, &ab_schema(), Some(&target)), + "StructPatch has non-optional field patches that reference missing input fields", + ); + } + + #[test] + fn optional_patch_on_missing_field_is_tolerated() { + // An optional drop on a missing field is silently ignored. + let patch = ExpressionStructPatchBuilder::new() + .drop_if_exists("nonexistent") + .build() + .unwrap(); + assert_eq!( + lower_patch(patch, &ab_schema(), &pq_output_schema()), + "named_struct(Utf8(\"p\"), a, Utf8(\"q\"), b)" + ); + } + + /// A struct target is re-derived and threaded at every nesting level: a `StructPatch` whose + /// appended field `g` is a `Struct` whose field `h` is a `Struct` whose `leaf` is a column. + /// Each level pulls its child's sub-schema from its own field type, so names land correctly all + /// the way down (`g` from the patch target, `h` from g's sub-schema, `leaf` from h's). + #[test] + fn nested_struct_targets_are_rederived_at_each_level() { + let deepest = KernelExpr::struct_from([col!("a")]); // { leaf: a } + let middle = KernelExpr::struct_from([deepest]); // { h: { leaf } } + let patch = ExpressionStructPatchBuilder::new() + .append(middle) + .build() + .unwrap(); + let target = StructType::try_new([ + StructField::nullable("a", DataType::LONG), + StructField::nullable("b", DataType::LONG), + StructField::nullable( + "g", + StructType::try_new([StructField::nullable( + "h", + StructType::try_new([StructField::nullable("leaf", DataType::LONG)]).unwrap(), + )]) + .unwrap(), + ), + ]) + .unwrap(); + assert_eq!( + lower_patch(patch, &ab_schema(), &target), + "named_struct(Utf8(\"a\"), a, Utf8(\"b\"), b, Utf8(\"g\"), \ + named_struct(Utf8(\"h\"), named_struct(Utf8(\"leaf\"), a)))" + ); + } + + // === MapToStruct === + + /// Input schema for map tests: `{ pv: map }`. + fn pv_map_schema() -> StructType { + StructType::try_new([StructField::nullable( + "pv", + MapType::new(DataType::STRING, DataType::STRING, true), + )]) + .unwrap() + } + + /// Lowers a `MapToStruct` over `pv` targeting `output_schema` and renders it as a `Display` + /// string. + fn lower_map_to_struct(output_schema: StructType) -> String { + let kernel = KernelExpr::map_to_struct(col!("pv"), MapToStructOptions::default()); + let target: DataType = output_schema.into(); + to_df_expr(&kernel, &pv_map_schema(), Some(&target)) + .unwrap() + .to_string() + } + + #[test] + fn element_at_lowers_to_kernel_udf() { + let kernel = KernelExpr::element_at(col!("pv"), lit("region")); + let rendered = to_df_expr(&kernel, &pv_map_schema(), Some(&DataType::STRING)) + .unwrap() + .to_string(); + assert_eq!(rendered, "kernel_element_at(pv, Utf8(\"region\"))"); + } + + #[test] + fn element_at_executes_dynamic_keys_and_rightmost_duplicates() { + let mut maps = MapBuilder::new(None, StringBuilder::new(), StringBuilder::new()); + maps.keys().append_value("region"); + maps.values().append_value("first"); + maps.keys().append_value("region"); + maps.values().append_value("last"); + maps.append(true).unwrap(); + maps.keys().append_value("a"); + maps.values().append_value("A"); + maps.keys().append_value("b"); + maps.values().append_value("B"); + maps.append(true).unwrap(); + maps.append(false).unwrap(); + maps.keys().append_value("a"); + maps.values().append_value("A"); + maps.append(true).unwrap(); + + let map = Arc::new(maps.finish()) as ArrayRef; + let key = Arc::new(StringArray::from(vec![ + Some("region"), + Some("b"), + Some("a"), + None, + ])) as ArrayRef; + let logical = to_df_expr( + &KernelExpr::element_at(col!("pv"), col!("key")), + &StructType::try_new([ + StructField::nullable("pv", MapType::new(DataType::STRING, DataType::STRING, true)), + StructField::nullable("key", DataType::STRING), + ]) + .unwrap(), + Some(&DataType::STRING), + ) + .unwrap(); + let input_schema = ArrowSchema::new(vec![ + ArrowField::new("pv", map.data_type().clone(), true), + ArrowField::new("key", ArrowDataType::Utf8, true), + ]); + let batch = RecordBatch::try_new(Arc::new(input_schema.clone()), vec![map, key]).unwrap(); + let df_schema = DFSchema::try_from(input_schema).unwrap(); + let physical = create_physical_expr(&logical, &df_schema, &ExecutionProps::new()).unwrap(); + let result = physical + .evaluate(&batch) + .unwrap() + .into_array(batch.num_rows()) + .unwrap(); + let result = result.as_string::(); + assert_eq!( + result, + &StringArray::from(vec![Some("last"), Some("B"), None, None]) + ); + } + + /// Each target field extracts its value with `cast(get_field(pv, name), T)`, and the whole + /// rebuild is wrapped in a null-map guard. Runtime cast/parse semantics (empty-string, + /// temporal, decimal, duplicate keys, null masking) are arrow's, verified end-to-end rather + /// than here. + #[test] + fn map_to_struct_lowers_to_named_struct_over_get_field() { + let target = StructType::try_new([ + StructField::nullable("region", DataType::STRING), + StructField::nullable("id", DataType::INTEGER), + ]) + .unwrap(); + let rendered = lower_map_to_struct(target); + assert_eq!( + rendered, + concat!( + r#"CASE WHEN pv IS NOT NULL THEN named_struct("#, + r#"Utf8("region"), CAST(get_field(pv, Utf8("region")) AS Utf8), "#, + r#"Utf8("id"), CAST(nullif(get_field(pv, Utf8("id")), Utf8("")) AS Int32)) "#, + r#"ELSE NULL END"#, + ) + ); + } + + /// String and Binary targets keep the raw value (empty string is a valid value), so they lower + /// to a bare `cast`; every other primitive first maps an empty string to null via `nullif`. + #[rstest] + #[case::string_bare_cast(DataType::STRING, "CAST(get_field(pv, Utf8(\"f\")) AS Utf8)")] + #[case::binary_bare_cast(DataType::BINARY, "CAST(get_field(pv, Utf8(\"f\")) AS Binary)")] + #[case::integer_wraps_nullif( + DataType::INTEGER, + "CAST(nullif(get_field(pv, Utf8(\"f\")), Utf8(\"\")) AS Int32)" + )] + fn map_to_struct_field_value_lowering( + #[case] field_type: DataType, + #[case] expected_value: &str, + ) { + let target = StructType::try_new([StructField::nullable("f", field_type)]).unwrap(); + let expected = + format!("CASE WHEN pv IS NOT NULL THEN named_struct(Utf8(\"f\"), {expected_value}) ELSE NULL END"); + assert_eq!(lower_map_to_struct(target), expected); + } + + /// The target must be a struct of primitive fields: an absent one leaves the rebuild without + /// field names, and a non-primitive field has no string-to-value cast. + #[rstest] + #[case::no_target(None, "MapToStruct expression requires a struct output type")] + #[case::non_primitive_field( + Some(DataType::from( + StructType::try_new([StructField::nullable("nested", pq_output_schema())]).unwrap() + )), + "MapToStruct only supports primitive target types, but field 'nested' is" + )] + fn map_to_struct_with_unsupported_target_is_an_error( + #[case] output_type: Option, + #[case] expected_message: &str, + ) { + let kernel = KernelExpr::map_to_struct(col!("pv"), MapToStructOptions::default()); + let err = to_df_expr(&kernel, &pv_map_schema(), output_type.as_ref()) + .unwrap_err() + .to_string(); + assert!(err.contains(expected_message), "{err}"); + } + + #[test] + fn map_to_struct_with_timezone_lowers_to_kernel_udf() { + let target = + StructType::try_new([StructField::nullable("ts", DataType::TIMESTAMP)]).unwrap(); + let map = KernelExpr::map_to_struct( + col!("pv"), + MapToStructOptions::default().with_timestamp_timezone("America/Los_Angeles"), + ); + assert_eq!( + to_df_expr(&map, &pv_map_schema(), Some(&DataType::from(target))) + .unwrap() + .to_string(), + "kernel_map_to_struct_with_timestamp_timezone(pv)" + ); + } + + #[test] + fn map_to_struct_with_timezone_executes_in_reader_timezone() { + let mut maps = MapBuilder::new(None, StringBuilder::new(), StringBuilder::new()); + maps.keys().append_value("ts"); + maps.values().append_value("2024-06-15 09:30:00"); + maps.append(true).unwrap(); + let map = Arc::new(maps.finish()) as ArrayRef; + let arrow_schema = + ArrowSchema::new(vec![ArrowField::new("pv", map.data_type().clone(), true)]); + let batch = RecordBatch::try_new(Arc::new(arrow_schema.clone()), vec![map]).unwrap(); + let target = + StructType::try_new([StructField::nullable("ts", DataType::TIMESTAMP)]).unwrap(); + let logical = to_df_expr( + &KernelExpr::map_to_struct( + col!("pv"), + MapToStructOptions::default().with_timestamp_timezone("America/Los_Angeles"), + ), + &pv_map_schema(), + Some(&DataType::from(target)), + ) + .unwrap(); + let df_schema = DFSchema::try_from(arrow_schema).unwrap(); + let physical = create_physical_expr(&logical, &df_schema, &ExecutionProps::new()).unwrap(); + let result = physical.evaluate(&batch).unwrap().into_array(1).unwrap(); + let timestamps = result + .as_struct() + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(timestamps.value(0), 1_718_469_000_000_000); + } + + #[test] + fn cast_with_timezone_lowers_to_kernel_udf() { + let input_schema = + StructType::try_new([StructField::nullable("s", DataType::STRING)]).unwrap(); + let cast = KernelExpr::cast( + col!("s"), + DataType::TIMESTAMP, + CastOptions::default().with_timestamp_timezone("America/Los_Angeles"), + ); + assert_eq!( + to_df_expr(&cast, &input_schema, Some(&DataType::TIMESTAMP)) + .unwrap() + .to_string(), + "kernel_cast(s)" + ); + } + + #[test] + fn cast_with_timezone_executes_in_reader_timezone() { + let input_schema = + StructType::try_new([StructField::nullable("s", DataType::STRING)]).unwrap(); + let arrow_schema: ArrowSchema = (&input_schema).try_into_arrow().unwrap(); + let batch = RecordBatch::try_new( + Arc::new(arrow_schema.clone()), + vec![Arc::new(StringArray::from(vec![Some( + "2024-06-15 09:30:00", + )]))], + ) + .unwrap(); + let logical = to_df_expr( + &KernelExpr::cast( + col!("s"), + DataType::TIMESTAMP, + CastOptions::default().with_timestamp_timezone("America/Los_Angeles"), + ), + &input_schema, + Some(&DataType::TIMESTAMP), + ) + .unwrap(); + let df_schema = DFSchema::try_from(arrow_schema).unwrap(); + let physical = create_physical_expr(&logical, &df_schema, &ExecutionProps::new()).unwrap(); + let result = physical.evaluate(&batch).unwrap().into_array(1).unwrap(); + let timestamps = result + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(timestamps.value(0), 1_718_469_000_000_000); + } + + // === ParseJson Shared Helpers === + + /// Input schema for JSON tests: `{ j: string }`. + fn json_input_schema() -> StructType { + StructType::try_new([StructField::nullable("j", DataType::STRING)]).unwrap() + } + + fn nested_parse_type() -> StructType { + StructType::try_new([ + StructField::nullable("n", DataType::LONG), + StructField::nullable("s", DataType::STRING), + ]) + .unwrap() + } + + /// Target parse schema `{ n: long, s: string }`. + fn parse_target() -> KernelSchemaRef { + Arc::new(nested_parse_type()) + } + + /// Lowers `parse_json(col("j"), schema)`, builds a physical expr over a one-column string batch + /// carrying `rows`, evaluates it, and returns the resulting struct column. + fn eval_parse_json(schema: KernelSchemaRef, rows: Vec>) -> StructArray { + let kernel = KernelExpr::parse_json(col!("j"), schema); + // Self-typed: no output_type is threaded in, yet lowering still succeeds. + let logical = to_df_expr(&kernel, &json_input_schema(), None).unwrap(); + + let arrow_schema: ArrowSchema = (&json_input_schema()).try_into_arrow().unwrap(); + let batch = RecordBatch::try_new( + Arc::new(arrow_schema.clone()), + vec![Arc::new(StringArray::from(rows)) as ArrayRef], + ) + .unwrap(); + + let df_schema = DFSchema::try_from(arrow_schema).unwrap(); + let physical = create_physical_expr(&logical, &df_schema, &ExecutionProps::new()).unwrap(); + physical + .evaluate(&batch) + .unwrap() + .into_array(batch.num_rows()) + .unwrap() + .as_struct() + .clone() + } + + /// [`eval_parse_json`] with the struct flattened to one column per parsed field. Panics on a + /// struct with a top-level null, so the malformed-backstop case must use [`eval_parse_json`]. + fn eval_parse_json_batch(schema: KernelSchemaRef, rows: Vec>) -> RecordBatch { + RecordBatch::from(eval_parse_json(schema, rows)) + } + + /// Asserts the result fields equal `target`'s arrow projection: the parse is typed to the + /// target schema, not merely compatible with it. + fn assert_matches_target(batch: &RecordBatch, target: &KernelSchemaRef) { + let target: ArrowSchema = target.as_ref().try_into_arrow().unwrap(); + assert_eq!(batch.schema().fields(), target.fields()); + } + + // === ParseJson Tests === + + #[test] + fn parse_json_lowers_to_udf_call() { + let kernel = KernelExpr::parse_json(col!("j"), parse_target()); + assert_eq!( + to_df_expr(&kernel, &json_input_schema(), None) + .unwrap() + .to_string(), + "kernel_parse_json(j)" + ); + } + + #[test] + fn parse_json_parses_fields_into_typed_struct() { + let batch = eval_parse_json_batch( + parse_target(), + vec![Some(r#"{"n": 1, "s": "a"}"#), Some(r#"{"n": 2, "s": "b"}"#)], + ); + assert_matches_target(&batch, &parse_target()); + assert_batches_eq!( + [ + "+---+---+", + "| n | s |", + "+---+---+", + "| 1 | a |", + "| 2 | b |", + "+---+---+", + ], + &[batch] + ); + } + + /// Every primitive `parse_json` can decode, in one struct: the integer/float/boolean families + /// decode directly, while the failure-prone leaves (date, both timestamps, decimal) route + /// through kernel's stringify-then-safe-cast path. Asserts they all land typed to the target. + #[test] + fn parse_json_decodes_all_supported_primitive_types() { + let target: KernelSchemaRef = Arc::new( + StructType::try_new([ + StructField::nullable("str", DataType::STRING), + StructField::nullable("long", DataType::LONG), + StructField::nullable("int", DataType::INTEGER), + StructField::nullable("short", DataType::SHORT), + StructField::nullable("byte", DataType::BYTE), + StructField::nullable("float", DataType::FLOAT), + StructField::nullable("double", DataType::DOUBLE), + StructField::nullable("bool", DataType::BOOLEAN), + StructField::nullable("date", DataType::DATE), + StructField::nullable("ts", DataType::TIMESTAMP), + StructField::nullable("ts_ntz", DataType::TIMESTAMP_NTZ), + StructField::nullable("dec", DataType::decimal(10, 2).unwrap()), + ]) + .unwrap(), + ); + let row = r#"{ + "str": "a", "long": 1, "int": 2, "short": 3, "byte": 4, + "float": 1.5, "double": 2.5, "bool": true, "date": "2024-01-02", + "ts": "2024-01-02T03:04:05Z", "ts_ntz": "2024-01-02T03:04:05", "dec": "12.34" + }"#; + let batch = eval_parse_json_batch(target.clone(), vec![Some(row)]); + assert_matches_target(&batch, &target); + assert_batches_eq!( + [ + "+-----+------+-----+-------+------+-------+--------+------+------------+----------------------+---------------------+-------+", + "| str | long | int | short | byte | float | double | bool | date | ts | ts_ntz | dec |", + "+-----+------+-----+-------+------+-------+--------+------+------------+----------------------+---------------------+-------+", + "| a | 1 | 2 | 3 | 4 | 1.5 | 2.5 | true | 2024-01-02 | 2024-01-02T03:04:05Z | 2024-01-02T03:04:05 | 12.34 |", + "+-----+------+-----+-------+------+-------+--------+------+------------+----------------------+---------------------+-------+", + ], + &[batch] + ); + } + + #[rstest] + #[case::array( + DataType::from(ArrayType::new(DataType::INTEGER, true)), + r#"[1, null, 3]"#, + "[1, , 3]" + )] + #[case::struct_( + DataType::from(nested_parse_type()), + r#"{"n": 1, "s": "a"}"#, + "{n: 1, s: a}" + )] + #[case::map( + DataType::from(MapType::new(DataType::STRING, DataType::LONG, true)), + r#"{"x": 1, "y": null}"#, + "{x: 1, y: }" + )] + #[case::array_of_structs( + DataType::from(ArrayType::new(nested_parse_type(), true)), + r#"[{"n": 1, "s": "a"}, null, {"n": 2, "s": "b"}]"#, + "[{n: 1, s: a}, , {n: 2, s: b}]" + )] + #[case::array_of_maps( + DataType::from(ArrayType::new( + MapType::new(DataType::STRING, DataType::LONG, true), + true, + )), + r#"[{"x": 1, "y": null}, {"z": 2}]"#, + "[{x: 1, y: }, {z: 2}]" + )] + #[case::array_of_arrays( + DataType::from(ArrayType::new(ArrayType::new(DataType::INTEGER, true), true)), + r#"[[1, null], [2, 3]]"#, + "[[1, ], [2, 3]]" + )] + #[case::struct_of_structs( + DataType::from( + StructType::try_new([StructField::nullable("inner", nested_parse_type())]).unwrap() + ), + r#"{"inner": {"n": 1, "s": "a"}}"#, + "{inner: {n: 1, s: a}}" + )] + #[case::struct_of_arrays( + DataType::from( + StructType::try_new([StructField::nullable( + "items", + ArrayType::new(DataType::INTEGER, true), + )]) + .unwrap() + ), + r#"{"items": [1, null, 3]}"#, + "{items: [1, , 3]}" + )] + #[case::struct_of_maps( + DataType::from( + StructType::try_new([StructField::nullable( + "items", + MapType::new(DataType::STRING, DataType::LONG, true), + )]) + .unwrap() + ), + r#"{"items": {"x": 1, "y": null}}"#, + "{items: {x: 1, y: }}" + )] + #[case::map_of_structs( + DataType::from(MapType::new(DataType::STRING, nested_parse_type(), true)), + r#"{"x": {"n": 1, "s": "a"}, "y": {"n": 2, "s": "b"}}"#, + "{x: {n: 1, s: a}, y: {n: 2, s: b}}" + )] + #[case::map_of_arrays( + DataType::from(MapType::new( + DataType::STRING, + ArrayType::new(DataType::INTEGER, true), + true, + )), + r#"{"x": [1, null], "y": [2, 3]}"#, + "{x: [1, ], y: [2, 3]}" + )] + #[case::map_of_maps( + DataType::from(MapType::new( + DataType::STRING, + MapType::new(DataType::STRING, DataType::LONG, true), + true, + )), + r#"{"x": {"a": 1}, "y": {"b": 2}}"#, + "{x: {a: 1}, y: {b: 2}}" + )] + fn parse_json_decodes_nested_container_field( + #[case] field_type: DataType, + #[case] json_value: &str, + #[case] expected_value: &str, + ) { + let target: KernelSchemaRef = + Arc::new(StructType::try_new([StructField::nullable("value", field_type)]).unwrap()); + let row = format!(r#"{{"value": {json_value}}}"#); + let batch = eval_parse_json_batch(target.clone(), vec![Some(row.as_str())]); + assert_matches_target(&batch, &target); + + let width = expected_value.len().max("value".len()); + let border = format!("+{}+", "-".repeat(width + 2)); + let header = format!("| {:width$} |", "value"); + let value = format!("| {expected_value:width$} |"); + let expected = [ + border.as_str(), + header.as_str(), + border.as_str(), + value.as_str(), + border.as_str(), + ]; + assert_batches_eq!(expected, &[batch]); + } + + /// `Binary` has no JSON decoder in arrow-json, so any row hits kernel's whole-batch parse error + /// and the coarse backstop nulls the struct. Documents that a `Binary` leaf is effectively + /// unsupported through this path rather than silently mis-decoding. + #[test] + fn parse_json_binary_leaf_is_unsupported_and_yields_all_null_struct() { + let target: KernelSchemaRef = + Arc::new(StructType::try_new([StructField::nullable("b", DataType::BINARY)]).unwrap()); + let out = eval_parse_json(target, vec![Some(r#"{"b": "aGk="}"#)]); + assert_eq!(out.len(), 1); + assert!(out.column(0).is_null(0)); + } + + /// A null input string decodes as `{}` (kernel's contract): the row stays present with all its + /// fields null, rather than nulling the whole struct row. + #[test] + fn parse_json_null_input_yields_present_row_with_null_fields() { + let batch = eval_parse_json_batch(parse_target(), vec![None]); + assert_batches_eq!( + [ + "+---+---+", + "| n | s |", + "+---+---+", + "| | |", + "+---+---+", + ], + &[batch] + ); + } + + /// A field absent from the JSON object parses to null. + #[test] + fn parse_json_missing_field_is_null() { + let batch = eval_parse_json_batch(parse_target(), vec![Some(r#"{"s": "only"}"#)]); + assert_batches_eq!( + [ + "+---+------+", + "| n | s |", + "+---+------+", + "| | only |", + "+---+------+", + ], + &[batch] + ); + } + + /// Genuinely malformed JSON hits the coarse backstop: the whole struct comes back all-null + /// (every field of every row null) rather than erroring the batch. + #[test] + fn parse_json_malformed_yields_all_null_struct() { + let out = eval_parse_json(parse_target(), vec![Some("{not json"), Some(r#"{"n": 5}"#)]); + assert_eq!(out.len(), 2); + assert!((0..2).all(|i| out.column(0).is_null(i) && out.column(1).is_null(i))); + } + + /// UDF identity must distinguish target schemas that share an arrow return type, or DataFusion + /// would treat the two calls as one common subexpression and parse both with one schema. + /// `integer` and `interval year to month` both map to arrow `Int32`. + #[rstest] + #[case(DataType::INTEGER, DataType::INTERVAL_YEAR_MONTH)] + #[case(DataType::LONG, DataType::INTERVAL_DAY_TIME)] + fn parse_json_udfs_with_same_return_type_but_different_schemas_are_not_equal( + #[case] left: DataType, + #[case] right: DataType, + ) { + let udf = |dt: DataType| { + ParseJsonUdf::try_new(Arc::new( + StructType::try_new([StructField::nullable("a", dt)]).unwrap(), + )) + .unwrap() + }; + let (left, right) = (udf(left), udf(right)); + assert_eq!( + left.return_type, right.return_type, + "precondition: arrow return types collide" + ); + assert_ne!(left, right); + } +} diff --git a/datafusion-executor/src/lib.rs b/datafusion-executor/src/lib.rs new file mode 100644 index 0000000000..b11ae474d5 --- /dev/null +++ b/datafusion-executor/src/lib.rs @@ -0,0 +1,51 @@ +//! A DataFusion-based [`PlanExecutor`](delta_kernel::PlanExecutor) for delta_kernel declarative +//! plans. +//! +//! Kernel emits executor-independent logical [`Plan`](delta_kernel::plans::ir::plan::Plan)s; this +//! crate executes them by lowering each plan to a DataFusion `LogicalPlan`, optimizing it, and +//! running the resulting `ExecutionPlan`. + +// TODO: remove once `session_ctx` and `storage_handler` are consumed by the query-execution path. +#![allow(dead_code)] + +use std::sync::Arc; + +use datafusion::execution::context::SessionContext; +use delta_kernel::StorageHandler; + +mod expression; +mod predicate; +mod scalar; + +pub use expression::to_df_expr; +pub use predicate::to_df_predicate_expr; +pub use scalar::to_df_scalar; + +/// Executes kernel declarative plans on DataFusion. +/// +/// Holds two handles, each owning a distinct part of the work: +/// - `session_ctx` -- *plan it, then run it*: DataFusion's `SessionContext` is the front door to +/// the query engine. It holds the session-scoped state needed to turn a query into something +/// runnable: configuration, registered tables/catalogs and functions, the logical/physical +/// optimizer rules, and a handle to the shared runtime environment (memory pool, object-store +/// registry). We use it to compile and optimize a kernel plan into a DataFusion `LogicalPlan`, +/// then lower it to a physical `ExecutionPlan`. It is heavyweight and meant to be long-lived and +/// shared. At execution time we derive a fresh per-run `TaskContext` from it via +/// `session_ctx.task_ctx()` and pass that to `ExecutionPlan::execute`. +/// - `storage_handler` -- *fetch the bytes the query engine can't*: a kernel [`StorageHandler`] for +/// the storage I/O DataFusion cannot do itself (deletion-vector resolution, footer reads, +/// listing). This is the file-system subset of a kernel [`Engine`](delta_kernel::Engine) -- the +/// executor needs nothing else from the engine, so it holds only this. +pub struct DataFusionExecutor { + session_ctx: SessionContext, + storage_handler: Arc, +} + +impl DataFusionExecutor { + pub fn new(storage_handler: Arc) -> Self { + Self { + session_ctx: SessionContext::new(), + storage_handler, + } + } +} diff --git a/datafusion-executor/src/predicate.rs b/datafusion-executor/src/predicate.rs new file mode 100644 index 0000000000..821e154487 --- /dev/null +++ b/datafusion-executor/src/predicate.rs @@ -0,0 +1,492 @@ +//! Conversion from a kernel [`Predicate`](KernelPredicate) to a boolean-valued DataFusion +//! [`Expr`](DFExpr). + +use datafusion::functions_nested::expr_fn::array_has; +use datafusion::logical_expr::expr::InList; +use datafusion::logical_expr::utils::{conjunction, disjunction}; +use datafusion::logical_expr::{binary_expr, lit, Expr as DFExpr, Operator}; +use delta_kernel::expressions::{ + ArrayData as KernelArrayData, BinaryPredicate as KernelBinaryPredicate, + BinaryPredicateOp as KernelBinaryPredicateOp, ColumnName as KernelColumnName, + Expression as KernelExpression, JunctionPredicate as KernelJunctionPredicate, + JunctionPredicateOp as KernelJunctionPredicateOp, Predicate as KernelPredicate, + Scalar as KernelScalar, UnaryPredicate as KernelUnaryPredicate, + UnaryPredicateOp as KernelUnaryPredicateOp, +}; +use delta_kernel::schema::{DataType, StructType}; +use delta_kernel::{DeltaResult, Error}; + +use crate::expression::to_df_expr; +use crate::scalar::to_df_scalar; + +/// Converts a kernel [`Predicate`](KernelPredicate) into a boolean-valued DataFusion +/// [`Expr`](DFExpr), checking column references against `input_schema`. +/// +/// # Errors +/// Returns [`Error::unsupported`] for engine-defined (`Opaque`) or `Unknown` predicates, and for an +/// `IN` whose right side is neither a literal array nor an array-typed column. Also propagates +/// errors from child expressions, such as an unresolved column or an interval literal (which has +/// no Arrow equivalent). +pub fn to_df_predicate_expr( + pred: &KernelPredicate, + input_schema: &StructType, +) -> DeltaResult { + match pred { + KernelPredicate::BooleanExpression(expr) => to_df_expr(expr, input_schema, None), + KernelPredicate::Not(inner) => { + let df_inner = to_df_predicate_expr(inner, input_schema)?; + Ok(DFExpr::Not(Box::new(df_inner))) + } + KernelPredicate::Unary(unary) => unary_to_df_predicate_expr(unary, input_schema), + KernelPredicate::Binary(binary) => binary_to_df_predicate_expr(binary, input_schema), + KernelPredicate::Junction(junction) => { + junction_to_df_predicate_expr(junction, input_schema) + } + KernelPredicate::Opaque(_) => Err(Error::unsupported( + "cannot convert an engine-defined Opaque predicate", + )), + KernelPredicate::Unknown(name) => Err(Error::unsupported(format!( + "cannot convert Unknown predicate {name:?}" + ))), + } +} + +/// Lowers a unary predicate. +fn unary_to_df_predicate_expr( + unary: &KernelUnaryPredicate, + input_schema: &StructType, +) -> DeltaResult { + let expr = to_df_expr(&unary.expr, input_schema, None)?; + match unary.op { + KernelUnaryPredicateOp::IsNull => Ok(DFExpr::IsNull(Box::new(expr))), + } +} + +/// Lowers a binary predicate. +fn binary_to_df_predicate_expr( + binary: &KernelBinaryPredicate, + input_schema: &StructType, +) -> DeltaResult { + let op = match binary.op { + KernelBinaryPredicateOp::In => { + return in_to_df_predicate_expr(&binary.left, &binary.right, input_schema) + } + KernelBinaryPredicateOp::Equal => Operator::Eq, + KernelBinaryPredicateOp::LessThan => Operator::Lt, + KernelBinaryPredicateOp::GreaterThan => Operator::Gt, + KernelBinaryPredicateOp::Distinct => Operator::IsDistinctFrom, + }; + let left = to_df_expr(&binary.left, input_schema, None)?; + let right = to_df_expr(&binary.right, input_schema, None)?; + Ok(binary_expr(left, op, right)) +} + +/// Lowers an `IN` predicate. Kernel models `x IN (..)` as `Binary(In, value, elements)`, where +/// `value` is a literal and `elements` is either a literal array (`1 IN (1, 2)`) or an array-typed +/// column (`1 IN col`). A literal array lowers to DataFusion's [`InList`], whose trailing flag +/// negates the test; a column lowers to `array_has(col, value)`. Kernel has no negated `IN` +/// (`NOT IN` arrives as `Not(Binary(In, ..))`, handled by the caller's `Not` arm), so the `InList` +/// flag is always `false` here. +/// +/// This accepts exactly the shapes kernel's Arrow evaluator accepts: both of its arms require a +/// literal left operand, so a column left operand is rejected here as it is there. +/// +/// Kernel and DataFusion agree once the result is forced non-null. Kernel compares elements with +/// logical (SQL) equality and reports membership as a plain boolean: a null matches nothing (not +/// even another null), so `NULL IN (1, 2)` and `NULL IN (1, NULL)` are both false, and the result +/// is never null. DataFusion's [`InList`] and `array_has` both keep SQL three-valued logic, so a +/// null on either side yields null. This lowering wraps the test in `IS TRUE`, collapsing that +/// null to false to match kernel. +/// +/// `IS TRUE` sits inside the predicate, not around it, because the caller's `Not` arm may negate +/// the result and a leftover null would change which rows a filter keeps. Kernel reads +/// `NOT NULL IN (1, 2)` as `NOT false` = true, but an unguarded DataFusion `NOT NULL` stays null +/// and drops the row. +/// +/// # Errors +/// Returns [`Error::unsupported`] if the left operand is not a literal, or the right operand is +/// neither a literal array nor an array-typed column. +fn in_to_df_predicate_expr( + value: &KernelExpression, + list: &KernelExpression, + input_schema: &StructType, +) -> DeltaResult { + // Kernel's Arrow evaluator accepts `IN` only with a literal left operand (a column left + // operand errors), so reject anything else to match it exactly. + let KernelExpression::Literal(_) = value else { + return Err(Error::unsupported( + "converting an IN predicate requires a literal left-hand side", + )); + }; + let value = to_df_expr(value, input_schema, None)?; + let membership = + match list { + KernelExpression::Literal(KernelScalar::Array(array)) => in_list_expr(value, array)?, + KernelExpression::Column(name) => array_has_expr(value, list, name, input_schema)?, + _ => return Err(Error::unsupported( + "converting an IN predicate requires a literal array or an array-typed column on \ + the right-hand side", + )), + }; + Ok(membership.is_true()) +} + +/// Builds `value IN ()` from a literal array. Null elements stay in the list: they can +/// never match under logical equality, and the caller's `IS TRUE` collapses the null they would +/// otherwise contribute down to false. +fn in_list_expr(value: DFExpr, array: &KernelArrayData) -> DeltaResult { + let elements: Vec = array + .array_elements() + .iter() + .map(|scalar| Ok(lit(to_df_scalar(scalar)?))) + .collect::>()?; + let in_expr = DFExpr::InList(InList::new(Box::new(value), elements, false)); + Ok(in_expr) +} + +/// Builds `array_has(, value)` for `value IN `, which kernel evaluates +/// element-wise. `column` is the kernel column expression and `name` its resolved name. Note the +/// flipped argument order: haystack first, needle second. +/// +/// # Errors +/// Returns [`Error::unsupported`] if `name` does not resolve to an array-typed column. +fn array_has_expr( + value: DFExpr, + column: &KernelExpression, + name: &KernelColumnName, + input_schema: &StructType, +) -> DeltaResult { + let DataType::Array(_) = input_schema.field_at(name)?.data_type else { + return Err(Error::unsupported( + "converting an IN predicate against a column requires an array-typed column", + )); + }; + Ok(array_has(to_df_expr(column, input_schema, None)?, value)) +} + +/// Lowers a junction (`And`/`Or`) by converting each child and combining them with DataFusion's +/// left-associative [`conjunction`]/[`disjunction`] helpers. +fn junction_to_df_predicate_expr( + junction: &KernelJunctionPredicate, + input_schema: &StructType, +) -> DeltaResult { + let preds: DeltaResult> = junction + .preds + .iter() + .map(|pred| to_df_predicate_expr(pred, input_schema)) + .collect(); + // An empty junction lowers `AND` to `true` and `OR` to `false`, keeping kernel semantics. + match junction.op { + KernelJunctionPredicateOp::And => Ok(conjunction(preds?).unwrap_or_else(|| lit(true))), + KernelJunctionPredicateOp::Or => Ok(disjunction(preds?).unwrap_or_else(|| lit(false))), + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use datafusion::arrow::array::{ + new_null_array, Array, AsArray, Int64Array, ListArray, RecordBatch, + }; + use datafusion::arrow::buffer::{OffsetBuffer, ScalarBuffer}; + use datafusion::arrow::datatypes::{DataType as ArrowDataType, Schema as ArrowSchema}; + use datafusion::common::DFSchema; + use datafusion::prelude::SessionContext; + use delta_kernel::engine::arrow_conversion::TryIntoArrow; + use delta_kernel::expressions::{ + col, lit, ArrayData as KernelArrayData, Expression as KernelExpr, Predicate as KernelPred, + }; + use delta_kernel::schema::{ArrayType, DataType, StructField}; + use rstest::rstest; + + use super::*; + + // === Shared helpers === + + /// Columns these tests resolve against: top-level `a`, `b`, `c` (all `long`) and `list` + /// (an array of nullable `long`, the right-hand side of a list-column `IN`). + fn test_schema() -> StructType { + StructType::try_new([ + StructField::nullable("a", DataType::LONG), + StructField::nullable("b", DataType::LONG), + StructField::nullable("c", DataType::LONG), + StructField::nullable("list", ArrayType::new(DataType::LONG, true)), + ]) + .unwrap() + } + + /// Lowers a predicate and returns its DataFusion `Display` string. + fn lower(pred: KernelPred) -> String { + to_df_predicate_expr(&pred, &test_schema()) + .unwrap() + .to_string() + } + + /// Lowers a predicate, runs it over one all-null row, and asserts the result is not null. + fn evaluate(pred: KernelPred) -> bool { + let df_expr = to_df_predicate_expr(&pred, &test_schema()).unwrap(); + let arrow_schema: ArrowSchema = (&test_schema()).try_into_arrow().unwrap(); + let arrow_schema = Arc::new(arrow_schema); + let batch = RecordBatch::try_new( + arrow_schema.clone(), + arrow_schema + .fields() + .iter() + .map(|field| new_null_array(field.data_type(), 1)) + .collect(), + ) + .unwrap(); + + let df_schema = DFSchema::try_from(arrow_schema).unwrap(); + let physical = SessionContext::new() + .create_physical_expr(df_expr, &df_schema) + .unwrap(); + let result = physical + .evaluate(&batch) + .unwrap() + .into_array(batch.num_rows()) + .unwrap(); + let result = result.as_boolean(); + + assert_eq!(result.null_count(), 0, "predicate evaluated to null"); + result.value(0) + } + + /// A literal `Scalar::Array` of longs. + fn long_array(values: impl IntoIterator) -> KernelExpr { + let elements: Vec = values.into_iter().map(KernelScalar::Long).collect(); + let array = + KernelArrayData::try_new(ArrayType::new(DataType::LONG, false), elements).unwrap(); + KernelExpr::literal(KernelScalar::Array(array)) + } + + /// A literal `Scalar::Array` of longs where `None` becomes a null element. + fn nullable_long_array(values: impl IntoIterator>) -> KernelExpr { + let elements: Vec = values + .into_iter() + .map(|v| match v { + Some(n) => KernelScalar::Long(n), + None => KernelScalar::Null(DataType::LONG), + }) + .collect(); + let array = + KernelArrayData::try_new(ArrayType::new(DataType::LONG, true), elements).unwrap(); + KernelExpr::literal(KernelScalar::Array(array)) + } + + // === Tests === + + #[rstest] + // Primitive comparisons lower to a native binary op. + #[case::eq(col!("a").eq(lit(1i64)), "a = Int64(1)")] + #[case::lt(col!("a").lt(lit(1i64)), "a < Int64(1)")] + #[case::gt(col!("a").gt(lit(1i64)), "a > Int64(1)")] + #[case::distinct( + col!("a").distinct(lit(1i64)), + "a IS DISTINCT FROM Int64(1)" + )] + // Kernel has no <=/>=/!= ops: each is `Not` of a comparison, so it renders negated. + #[case::ne(col!("a").ne(lit(1i64)), "NOT a = Int64(1)")] + #[case::le(col!("a").le(lit(1i64)), "NOT a > Int64(1)")] + #[case::ge(col!("a").ge(lit(1i64)), "NOT a < Int64(1)")] + // Unary. + #[case::is_null(col!("a").is_null(), "a IS NULL")] + #[case::is_not_null(col!("a").is_not_null(), "NOT a IS NULL")] + // IN / NOT IN. Kernel requires a literal needle. `IS TRUE` turns DataFusion's null into + // kernel's false, and sits inside the `Not` so a null value gives `NOT false` instead of + // `NOT NULL`. + #[case::in_list( + KernelPred::binary(KernelBinaryPredicateOp::In, lit(1i64), long_array([1, 2, 3])), + "Int64(1) IN ([Int64(1), Int64(2), Int64(3)]) IS TRUE" + )] + #[case::not_in( + KernelPred::not(KernelPred::binary(KernelBinaryPredicateOp::In, lit(1i64), long_array([1, 2]))), + "NOT Int64(1) IN ([Int64(1), Int64(2)]) IS TRUE" + )] + #[case::null_needle_in_list( + KernelPred::binary( + KernelBinaryPredicateOp::In, + KernelExpr::null_literal(DataType::LONG), + long_array([1, 2]), + ), + "Int64(NULL) IN ([Int64(1), Int64(2)]) IS TRUE" + )] + // A null element stays in the list; it can never match under logical equality, and `IS TRUE` + // collapses the null it contributes to false. + #[case::null_element_in_list( + KernelPred::binary( + KernelBinaryPredicateOp::In, + lit(1i64), + nullable_long_array([Some(1), None]), + ), + "Int64(1) IN ([Int64(1), Int64(NULL)]) IS TRUE" + )] + // A literal against a list column lowers to `array_has(list, value)` (haystack first). + #[case::in_list_column( + KernelPred::binary(KernelBinaryPredicateOp::In, lit(1i64), col!("list")), + "array_has(list, Int64(1)) IS TRUE" + )] + // Junctions fold left-associatively. + #[case::and( + KernelPred::and(col!("a").is_null(), col!("b").is_null()), + "a IS NULL AND b IS NULL" + )] + #[case::or( + KernelPred::or(col!("a").is_null(), col!("b").is_null()), + "a IS NULL OR b IS NULL" + )] + #[case::multi_and( + KernelPred::and_from([ + col!("a").is_null(), + col!("b").is_null(), + col!("c").is_null(), + ]), + "a IS NULL AND b IS NULL AND c IS NULL" + )] + // A bare boolean expression goes straight to the expression converter. + #[case::boolean_expression(KernelPred::from_expr(col!("a")), "a")] + // Nested predicates. DataFusion's Display omits parens around the junction under a `Not`, but + // the `Expr` tree is still `Not(And(..))`. + #[case::not_of_junction( + KernelPred::not(KernelPred::and(col!("a").is_null(), col!("b").is_null())), + "NOT a IS NULL AND b IS NULL" + )] + #[case::junction_of_junction( + KernelPred::or( + KernelPred::and(col!("a").is_null(), col!("b").is_null()), + col!("c").is_null(), + ), + "a IS NULL AND b IS NULL OR c IS NULL" + )] + #[case::and_of_comparisons( + KernelPred::and( + col!("a").eq(lit(1i64)), + col!("b").gt(lit(2i64)), + ), + "a = Int64(1) AND b > Int64(2)" + )] + fn predicate_lowers_to_expected(#[case] kernel: KernelPred, #[case] expected: &str) { + assert_eq!(lower(kernel), expected); + } + + #[rstest] + // Engine-defined and unknown predicates have no DataFusion equivalent. + #[case::unknown(KernelPred::Unknown("mystery".into()))] + // Kernel's evaluator rejects a column needle, so we do too, whether the elements are a literal + // array or a list column. + #[case::column_needle_in_literal_array( + KernelPred::binary(KernelBinaryPredicateOp::In, col!("a"), long_array([1, 2])) + )] + #[case::column_needle_in_list_column( + KernelPred::binary(KernelBinaryPredicateOp::In, col!("a"), col!("list")) + )] + // A literal needle against a non-array column (`b` is `long`) cannot be lowered. + #[case::in_non_array_column( + KernelPred::binary(KernelBinaryPredicateOp::In, lit(1i64), col!("b")) + )] + fn unsupported_predicate_is_an_error(#[case] pred: KernelPred) { + to_df_predicate_expr(&pred, &test_schema()).unwrap_err(); + } + + /// `IN` answers true or false but never null, so it still works under a `NOT`. Membership uses + /// logical (SQL) equality, so a null matches nothing -- not even another null -- matching + /// kernel's evaluator. + #[rstest] + #[case::present(Some(2), &[Some(1), Some(2)], true)] + #[case::absent(Some(9), &[Some(1), Some(2)], false)] + #[case::null_needle_without_null_element(None, &[Some(1), Some(2)], false)] + #[case::null_needle_with_null_element(None, &[Some(1), None], false)] + #[case::present_alongside_null_element(Some(1), &[Some(1), None], true)] + #[case::absent_alongside_null_element(Some(9), &[Some(1), None], false)] + #[case::only_null_element(None, &[None], false)] + fn in_predicate_matches_membership_and_never_nulls( + #[case] needle: Option, + #[case] elements: &[Option], + #[case] expected: bool, + ) { + let in_pred = KernelPred::binary( + KernelBinaryPredicateOp::In, + match needle { + Some(n) => KernelExpr::literal(n), + None => KernelExpr::null_literal(DataType::LONG), + }, + nullable_long_array(elements.to_vec()), + ); + + assert_eq!(evaluate(in_pred.clone()), expected); + assert_eq!(evaluate(KernelPred::not(in_pred)), !expected); + } + + /// A literal `IN ` lowers to `array_has` and follows the same never-null, + /// logical-equality rule as the literal-array form: a null needle or a null element matches + /// nothing, and the result is always a plain boolean, so `NOT IN` is its exact complement. + #[rstest] + #[case::present(Some(2), &[Some(1), Some(2)], true)] + #[case::absent(Some(9), &[Some(1), Some(2)], false)] + #[case::null_needle_without_null_element(None, &[Some(1), Some(2)], false)] + #[case::null_needle_with_null_element(None, &[Some(1), None], false)] + #[case::present_alongside_null_element(Some(1), &[Some(1), None], true)] + #[case::absent_alongside_null_element(Some(9), &[Some(1), None], false)] + #[case::only_null_element(None, &[None], false)] + fn in_list_column_matches_membership_and_never_nulls( + #[case] needle: Option, + #[case] elements: &[Option], + #[case] expected: bool, + ) { + let in_pred = KernelPred::binary( + KernelBinaryPredicateOp::In, + match needle { + Some(n) => KernelExpr::literal(n), + None => KernelExpr::null_literal(DataType::LONG), + }, + col!("list"), + ); + // Column `list` carries the elements; `a`/`b`/`c` are unused. + let batch = list_column_batch(elements); + + assert_eq!(evaluate_over(in_pred.clone(), &batch), expected); + assert_eq!(evaluate_over(KernelPred::not(in_pred), &batch), !expected); + } + + /// Lowers `pred` against [`test_schema`], evaluates it over the single-row `batch`, and asserts + /// the one result is not null. + fn evaluate_over(pred: KernelPred, batch: &RecordBatch) -> bool { + let df_expr = to_df_predicate_expr(&pred, &test_schema()).unwrap(); + let df_schema = DFSchema::try_from(batch.schema()).unwrap(); + let result = SessionContext::new() + .create_physical_expr(df_expr, &df_schema) + .unwrap() + .evaluate(batch) + .unwrap() + .into_array(batch.num_rows()) + .unwrap(); + let result = result.as_boolean(); + + assert_eq!(result.len(), 1, "expected a single-row result"); + assert_eq!(result.null_count(), 0, "predicate evaluated to null"); + result.value(0) + } + + /// A one-row batch whose single `list` column holds `elements`. + fn list_column_batch(elements: &[Option]) -> RecordBatch { + // Reuse the schema's own `list` field so the built array matches the name and metadata + // kernel synthesizes (an `element` child field, possibly with field-id metadata). + let arrow_schema: ArrowSchema = (&test_schema()).try_into_arrow().unwrap(); + let list_field = arrow_schema.field_with_name("list").unwrap().clone(); + let ArrowDataType::List(element_field) = list_field.data_type().clone() else { + unreachable!("`list` is declared as an array type"); + }; + let list = ListArray::new( + element_field, + OffsetBuffer::new(ScalarBuffer::from(vec![0i32, elements.len() as i32])), + Arc::new(Int64Array::from(elements.to_vec())), + None, + ); + RecordBatch::try_new( + Arc::new(ArrowSchema::new(vec![list_field])), + vec![Arc::new(list)], + ) + .unwrap() + } +} diff --git a/datafusion-executor/src/scalar.rs b/datafusion-executor/src/scalar.rs new file mode 100644 index 0000000000..7afeb6df3b --- /dev/null +++ b/datafusion-executor/src/scalar.rs @@ -0,0 +1,599 @@ +//! Conversion from a kernel [`Scalar`](KernelScalar) to a DataFusion +//! [`ScalarValue`](DFScalarValue). + +use std::sync::Arc; + +// Each foreign type is aliased with its source crate (`Kernel*`/`DF*`/`Arrow*`) so every use +// site reads unambiguously across the three crates this converter bridges. +use datafusion::arrow::array::{new_empty_array, ArrayRef, MapArray, StructArray}; +use datafusion::arrow::buffer::OffsetBuffer; +use datafusion::arrow::datatypes::{DataType as ArrowDataType, Field as ArrowField}; +use datafusion::common::scalar::ScalarStructBuilder; +use datafusion::common::utils::SingleRowListArrayBuilder; +use datafusion::common::ScalarValue as DFScalarValue; +use delta_kernel::engine::arrow_conversion::TryIntoArrow; +use delta_kernel::expressions::{ + ArrayData as KernelArrayData, MapData as KernelMapData, Scalar as KernelScalar, + StructData as KernelStructData, +}; +use delta_kernel::schema::DataType as KernelDataType; +use delta_kernel::{DeltaResult, Error}; + +/// Converts a kernel [`Scalar`](KernelScalar) into the equivalent DataFusion +/// [`ScalarValue`](DFScalarValue). +/// +/// # Errors +/// Returns an error for interval scalars, which are not yet supported; for a type with no Arrow +/// representation (e.g. a shredded variant); or if building the backing Arrow array for a nested +/// container otherwise fails. +pub fn to_df_scalar(scalar: &KernelScalar) -> DeltaResult { + Ok(match scalar { + KernelScalar::Integer(i) => DFScalarValue::Int32(Some(*i)), + KernelScalar::Long(i) => DFScalarValue::Int64(Some(*i)), + KernelScalar::Short(i) => DFScalarValue::Int16(Some(*i)), + KernelScalar::Byte(i) => DFScalarValue::Int8(Some(*i)), + KernelScalar::Float(f) => DFScalarValue::Float32(Some(*f)), + KernelScalar::Double(f) => DFScalarValue::Float64(Some(*f)), + KernelScalar::String(s) => DFScalarValue::Utf8(Some(s.clone())), + KernelScalar::Boolean(b) => DFScalarValue::Boolean(Some(*b)), + KernelScalar::Timestamp(v) => { + DFScalarValue::TimestampMicrosecond(Some(*v), Some("UTC".into())) + } + KernelScalar::TimestampNtz(v) => DFScalarValue::TimestampMicrosecond(Some(*v), None), + KernelScalar::Date(d) => DFScalarValue::Date32(Some(*d)), + KernelScalar::Binary(b) => DFScalarValue::Binary(Some(b.clone())), + // scale() is 0..=38, so the i8 cast never truncates. + KernelScalar::Decimal(d) => { + DFScalarValue::Decimal128(Some(d.bits()), d.precision(), d.scale() as i8) + } + KernelScalar::Struct(data) => struct_to_df_scalar(data)?, + KernelScalar::Array(data) => array_to_df_scalar(data)?, + KernelScalar::Map(data) => map_to_df_scalar(data)?, + KernelScalar::IntervalYearMonth(_) | KernelScalar::IntervalDayTime(_) => { + return Err(Error::unsupported( + "interval scalars are not supported in the DataFusion executor", + )) + } + KernelScalar::Null(data_type) => datatype_to_df_null_scalar(data_type)?, + }) +} + +/// Builds a typed-null `DFScalarValue` from a kernel type. +fn datatype_to_df_null_scalar(data_type: &KernelDataType) -> DeltaResult { + let arrow_type: ArrowDataType = data_type.try_into_arrow()?; + arrow_type.try_into().map_err(Error::generic_err) +} + +/// Builds a `DFScalarValue::List` holding a single list row of the converted elements. +fn array_to_df_scalar(data: &KernelArrayData) -> DeltaResult { + let elements: DeltaResult> = + data.array_elements().iter().map(to_df_scalar).collect(); + // Name the list's element field from kernel's own ArrayType->Arrow conversion + let element_field: ArrowField = data.array_type().try_into_arrow()?; + let element_array = df_scalars_to_arrow_array(elements?, element_field.data_type())?; + let list = SingleRowListArrayBuilder::new(element_array) + .with_field(&element_field) + .build_list_array(); + Ok(DFScalarValue::List(Arc::new(list))) +} + +/// Builds a `DFScalarValue::Struct` from the struct's fields and converted values. +fn struct_to_df_scalar(data: &KernelStructData) -> DeltaResult { + let mut builder = ScalarStructBuilder::new(); + for (field, value) in data.fields().iter().zip(data.values()) { + let arrow_field: ArrowField = field.try_into_arrow()?; + builder = builder.with_scalar(arrow_field, to_df_scalar(value)?); + } + builder.build().map_err(Error::generic_err) +} + +/// Builds a `DFScalarValue::Map` holding a single map row of the converted key/value pairs. +fn map_to_df_scalar(data: &KernelMapData) -> DeltaResult { + let map_type = data.map_type(); + let entries_field: ArrowField = map_type.try_into_arrow()?; + let ArrowDataType::Struct(kv_fields) = entries_field.data_type() else { + return Err(Error::generic("map entries type is not a struct")); + }; + let [key_field, value_field] = kv_fields.as_ref() else { + return Err(Error::generic( + "map entries struct must have exactly a key and value field", + )); + }; + + let pairs = data.pairs(); + let converted: DeltaResult<(Vec, Vec)> = pairs + .iter() + .map(|(key, value)| Ok((to_df_scalar(key)?, to_df_scalar(value)?))) + .collect(); + let (keys, values) = converted?; + let key_array = df_scalars_to_arrow_array(keys, key_field.data_type())?; + let value_array = df_scalars_to_arrow_array(values, value_field.data_type())?; + + let entries = StructArray::try_new(kv_fields.clone(), vec![key_array, value_array], None) + .map_err(Error::generic_err)?; + let offsets = OffsetBuffer::from_lengths([pairs.len()]); + let map_array = MapArray::try_new(Arc::new(entries_field), offsets, entries, None, false) + .map_err(Error::generic_err)?; + Ok(DFScalarValue::Map(Arc::new(map_array))) +} + +/// Collects converted scalars into a single Arrow column. [`DFScalarValue::iter_to_array`] infers +/// the type from the first element, so an empty column falls back to `arrow_type`. +fn df_scalars_to_arrow_array( + scalars: Vec, + arrow_type: &ArrowDataType, +) -> DeltaResult { + if scalars.is_empty() { + Ok(new_empty_array(arrow_type)) + } else { + DFScalarValue::iter_to_array(scalars).map_err(Error::generic_err) + } +} + +#[cfg(test)] +mod tests { + use datafusion::arrow::array::{Array, AsArray, Int32Array, ListArray}; + use datafusion::arrow::datatypes::Int32Type; + use datafusion::arrow::util::pretty::pretty_format_columns; + use delta_kernel::schema::{ + ArrayType, DataType as KernelDataType, MapType, StructField, StructType, + }; + use rstest::rstest; + + use super::*; + + // === Shared helpers === + + fn assert_rendered(value: &DFScalarValue, expected: &[&str]) { + let table = pretty_format_columns("c", &[value.to_array().unwrap()]) + .unwrap() + .to_string(); + let actual: Vec<&str> = table.lines().collect(); + assert_eq!( + actual, expected, + "\nexpected:\n{expected:#?}\nactual:\n{actual:#?}" + ); + } + + fn sample_struct_type() -> StructType { + StructType::try_new([ + StructField::not_null("a", KernelDataType::INTEGER), + StructField::nullable("b", KernelDataType::STRING), + ]) + .unwrap() + } + + fn sample_struct_scalar() -> KernelScalar { + KernelScalar::Struct( + KernelStructData::try_new( + sample_struct_type().fields().cloned().collect(), + vec![KernelScalar::Integer(1), KernelScalar::String("x".into())], + ) + .unwrap(), + ) + } + + fn sample_map_type() -> MapType { + MapType::new(KernelDataType::STRING, KernelDataType::INTEGER, false) + } + + fn sample_map_scalar() -> KernelScalar { + KernelScalar::Map( + KernelMapData::try_new( + sample_map_type(), + [(KernelScalar::String("k".into()), KernelScalar::Integer(1))], + ) + .unwrap(), + ) + } + + fn sample_int_array_scalar() -> KernelScalar { + KernelScalar::Array( + KernelArrayData::try_new( + ArrayType::new(KernelDataType::INTEGER, false), + [KernelScalar::Integer(1), KernelScalar::Integer(2)], + ) + .unwrap(), + ) + } + + // === Directly-converted primitive arms === + + #[rstest] + #[case::integer(KernelScalar::Integer(42), DFScalarValue::Int32(Some(42)))] + #[case::long( + KernelScalar::Long(9_876_543_210), + DFScalarValue::Int64(Some(9_876_543_210)) + )] + #[case::short(KernelScalar::Short(7), DFScalarValue::Int16(Some(7)))] + #[case::byte(KernelScalar::Byte(3), DFScalarValue::Int8(Some(3)))] + #[case::float(KernelScalar::Float(1.25), DFScalarValue::Float32(Some(1.25)))] + #[case::double(KernelScalar::Double(99.99), DFScalarValue::Float64(Some(99.99)))] + #[case::boolean(KernelScalar::Boolean(true), DFScalarValue::Boolean(Some(true)))] + #[case::string(KernelScalar::String("hi".into()), DFScalarValue::Utf8(Some("hi".into())))] + #[case::binary(KernelScalar::Binary(b"abc".to_vec()), DFScalarValue::Binary(Some(b"abc".to_vec())))] + #[case::date(KernelScalar::Date(20178), DFScalarValue::Date32(Some(20178)))] + #[case::timestamp( + KernelScalar::Timestamp(1_000_000), + DFScalarValue::TimestampMicrosecond(Some(1_000_000), Some("UTC".into())) + )] + #[case::timestamp_ntz( + KernelScalar::TimestampNtz(1_000_000), + DFScalarValue::TimestampMicrosecond(Some(1_000_000), None) + )] + #[case::decimal( + KernelScalar::decimal(12345, 10, 2).unwrap(), + DFScalarValue::Decimal128(Some(12345), 10, 2) + )] + fn primitive_scalar_converts_to_matching_value( + #[case] scalar: KernelScalar, + #[case] expected: DFScalarValue, + ) { + assert_eq!(to_df_scalar(&scalar).unwrap(), expected); + } + + #[test] + fn nan_and_infinity_are_preserved() { + match to_df_scalar(&KernelScalar::Double(f64::NAN)).unwrap() { + DFScalarValue::Float64(Some(v)) => assert!(v.is_nan()), + other => panic!("expected Float64 NaN, got {other:?}"), + } + assert_eq!( + to_df_scalar(&KernelScalar::Float(f32::INFINITY)).unwrap(), + DFScalarValue::Float32(Some(f32::INFINITY)) + ); + } + + // === Typed nulls: datatype_to_df_null_scalar === + + #[rstest] + #[case::integer(KernelDataType::INTEGER, DFScalarValue::Int32(None))] + #[case::long(KernelDataType::LONG, DFScalarValue::Int64(None))] + #[case::string(KernelDataType::STRING, DFScalarValue::Utf8(None))] + #[case::boolean(KernelDataType::BOOLEAN, DFScalarValue::Boolean(None))] + #[case::date(KernelDataType::DATE, DFScalarValue::Date32(None))] + #[case::timestamp( + KernelDataType::TIMESTAMP, + DFScalarValue::TimestampMicrosecond(None, Some("UTC".into())) + )] + #[case::timestamp_ntz( + KernelDataType::TIMESTAMP_NTZ, + DFScalarValue::TimestampMicrosecond(None, None) + )] + fn typed_null_scalar_converts_to_typed_null_value( + #[case] data_type: KernelDataType, + #[case] expected: DFScalarValue, + ) { + assert_eq!( + to_df_scalar(&KernelScalar::Null(data_type)).unwrap(), + expected + ); + } + + #[test] + fn null_struct_with_non_null_subfields_converts_to_null_struct() { + let struct_type = StructType::try_new([ + StructField::not_null("a", KernelDataType::INTEGER), + StructField::not_null("b", KernelDataType::STRING), + ]) + .unwrap(); + let value = to_df_scalar(&KernelScalar::Null(struct_type.into())).unwrap(); + assert!(matches!(value, DFScalarValue::Struct(_)), "got {value:?}"); + assert!(value.is_null(), "expected a null struct, got {value:?}"); + } + + // Container nulls take DataFusion's nested `try_new_null` paths (distinct from the + // primitive arms), so assert both the variant and that the value reads back as null. + #[rstest] + #[case::array(KernelDataType::from(ArrayType::new(KernelDataType::INTEGER, true)))] + #[case::map(KernelDataType::from(MapType::new( + KernelDataType::STRING, + KernelDataType::INTEGER, + false + )))] + fn null_container_converts_to_typed_null(#[case] data_type: KernelDataType) { + let value = to_df_scalar(&KernelScalar::Null(data_type)).unwrap(); + assert!(value.is_null(), "expected a null value, got {value:?}"); + } + + #[test] + fn null_decimal_converts_to_typed_null_decimal() { + let value = to_df_scalar(&KernelScalar::Null(KernelDataType::decimal(10, 2).unwrap())); + assert_eq!(value.unwrap(), DFScalarValue::Decimal128(None, 10, 2)); + } + + // A shredded (non-unshredded) variant has no Arrow representation in kernel's type + // conversion, so a typed null of that type surfaces an error. + #[test] + fn unrepresentable_type_returns_error() { + let shredded_variant = + KernelDataType::variant_type([StructField::not_null("x", KernelDataType::INTEGER)]) + .unwrap(); + to_df_scalar(&KernelScalar::Null(shredded_variant)).unwrap_err(); + } + + // === Arrays: array_to_df_scalar === + + // The list's element field is named "element" (kernel's LIST_ARRAY_ROOT), not DataFusion's + // default "item"; the expected value is built to match kernel's ArrayType->Arrow + // conversion. + #[test] + fn array_scalar_converts_to_list_with_matching_elements() { + let array = KernelArrayData::try_new( + ArrayType::new(KernelDataType::INTEGER, false), + [KernelScalar::Integer(1), KernelScalar::Integer(2)], + ) + .unwrap(); + let value = to_df_scalar(&KernelScalar::Array(array)).unwrap(); + let element_field = ArrowField::new("element", ArrowDataType::Int32, false); + let list = ListArray::new( + Arc::new(element_field), + OffsetBuffer::from_lengths([2]), + Arc::new(Int32Array::from(vec![1, 2])), + None, + ); + let expected = DFScalarValue::List(Arc::new(list)); + assert_eq!(value, expected); + } + + #[rstest] + #[case::array_of_structs( + ArrayType::new(sample_struct_type(), false), + vec![sample_struct_scalar()], + &[ + "+----------------+", + "| c |", + "+----------------+", + "| [{a: 1, b: x}] |", + "+----------------+", + ] + )] + #[case::array_of_maps( + ArrayType::new(sample_map_type(), false), + vec![sample_map_scalar()], + &[ + "+----------+", + "| c |", + "+----------+", + "| [{k: 1}] |", + "+----------+", + ] + )] + #[case::array_of_arrays( + ArrayType::new(ArrayType::new(KernelDataType::INTEGER, false), false), + vec![sample_int_array_scalar()], + &[ + "+----------+", + "| c |", + "+----------+", + "| [[1, 2]] |", + "+----------+", + ] + )] + fn nested_array_converts_to_list( + #[case] array_type: ArrayType, + #[case] elements: Vec, + #[case] expected: &[&str], + ) { + let data = KernelArrayData::try_new(array_type, elements).unwrap(); + let value = to_df_scalar(&KernelScalar::Array(data)).unwrap(); + assert!(matches!(value, DFScalarValue::List(_)), "got {value:?}"); + assert_rendered(&value, expected); + } + + // A nullable-element array carrying a null: the element field must stay nullable (so the + // null round-trips) rather than inheriting the values' non-null inference. + #[test] + fn array_with_null_element_keeps_nullable_element_field() { + let data = KernelArrayData::try_new( + ArrayType::new(KernelDataType::INTEGER, true), + [ + KernelScalar::Integer(1), + KernelScalar::Null(KernelDataType::INTEGER), + ], + ) + .unwrap(); + let value = to_df_scalar(&KernelScalar::Array(data)).unwrap(); + let DFScalarValue::List(list) = &value else { + panic!("expected List, got {value:?}"); + }; + let values = list.values(); + assert!( + values.is_null(1), + "second element should be null, got {values:?}" + ); + assert_rendered( + &value, + &[ + "+-------+", + "| c |", + "+-------+", + "| [1, ] |", + "+-------+", + ], + ); + } + + // The empty-column path in `df_scalars_to_arrow_array` (element type from the declared + // `ArrayType`, not from a first element) must still produce a typed empty list. + #[test] + fn empty_array_converts_to_empty_list() { + let empty: Vec = vec![]; + let data = KernelArrayData::try_new(ArrayType::new(KernelDataType::INTEGER, false), empty) + .unwrap(); + let value = to_df_scalar(&KernelScalar::Array(data)).unwrap(); + let DFScalarValue::List(list) = &value else { + panic!("expected List, got {value:?}"); + }; + assert_eq!( + list.values().len(), + 0, + "expected an empty list, got {list:?}" + ); + } + + // === Structs: struct_to_df_scalar === + + // Field names and nullability are part of struct equality, so asserting against a + // hand-built expected value pins them too. + #[test] + fn struct_scalar_converts_to_struct_with_matching_fields() { + let data = KernelStructData::try_new( + vec![ + StructField::not_null("a", KernelDataType::INTEGER), + StructField::nullable("b", KernelDataType::STRING), + ], + vec![KernelScalar::Integer(1), KernelScalar::String("x".into())], + ) + .unwrap(); + let value = to_df_scalar(&KernelScalar::Struct(data)).unwrap(); + let expected = ScalarStructBuilder::new() + .with_scalar( + ArrowField::new("a", ArrowDataType::Int32, false), + DFScalarValue::Int32(Some(1)), + ) + .with_scalar( + ArrowField::new("b", ArrowDataType::Utf8, true), + DFScalarValue::Utf8(Some("x".into())), + ) + .build() + .unwrap(); + assert_eq!(value, expected); + } + + // A struct field whose value is itself a nested container (struct or array). + #[rstest] + #[case::nested_struct( + StructField::not_null("inner", sample_struct_type()), + sample_struct_scalar(), + &[ + "+-----------------------+", + "| c |", + "+-----------------------+", + "| {inner: {a: 1, b: x}} |", + "+-----------------------+", + ] + )] + #[case::array_field( + StructField::not_null("arr", ArrayType::new(KernelDataType::INTEGER, false)), + sample_int_array_scalar(), + &[ + "+---------------+", + "| c |", + "+---------------+", + "| {arr: [1, 2]} |", + "+---------------+", + ] + )] + fn struct_with_container_field_converts_to_struct( + #[case] field: StructField, + #[case] value: KernelScalar, + #[case] expected: &[&str], + ) { + let data = KernelStructData::try_new(vec![field], vec![value]).unwrap(); + let value = to_df_scalar(&KernelScalar::Struct(data)).unwrap(); + assert!(matches!(value, DFScalarValue::Struct(_)), "got {value:?}"); + assert_rendered(&value, expected); + } + + // A present (non-null) struct that carries a null in a NULLABLE field. + #[test] + fn present_struct_with_null_nullable_subfield_converts() { + let data = KernelStructData::try_new( + vec![ + StructField::not_null("a", KernelDataType::INTEGER), + StructField::nullable("b", KernelDataType::STRING), + ], + vec![ + KernelScalar::Integer(1), + KernelScalar::Null(KernelDataType::STRING), + ], + ) + .unwrap(); + let value = to_df_scalar(&KernelScalar::Struct(data)).unwrap(); + let DFScalarValue::Struct(array) = &value else { + panic!("expected Struct, got {value:?}"); + }; + assert!(!value.is_null(), "struct itself is present, got {value:?}"); + // Subfield `b` must be an actual null + let b = array.column_by_name("b").unwrap(); + assert!(b.is_null(0), "subfield b should be null, got {b:?}"); + } + + // === Maps: map_to_df_scalar === + + // No symmetric DFScalarValue map constructor exists, so read the entries back directly + // rather than asserting against a hand-built expected value. + #[rstest] + #[case::single(vec![(KernelScalar::String("k".into()), KernelScalar::Integer(1))], vec![("k", 1)])] + #[case::empty(vec![], vec![])] + fn map_scalar_converts_to_map_with_matching_pairs( + #[case] pairs: Vec<(KernelScalar, KernelScalar)>, + #[case] expected: Vec<(&str, i32)>, + ) { + let data = KernelMapData::try_new( + MapType::new(KernelDataType::STRING, KernelDataType::INTEGER, false), + pairs, + ) + .unwrap(); + let value = to_df_scalar(&KernelScalar::Map(data)).unwrap(); + let DFScalarValue::Map(map) = &value else { + panic!("expected Map, got {value:?}"); + }; + let keys = map.keys().as_string::(); + let values = map.values().as_primitive::(); + let actual: Vec<(&str, i32)> = (0..keys.len()) + .map(|i| (keys.value(i), values.value(i))) + .collect(); + assert_eq!(actual, expected); + } + + #[rstest] + #[case::map_of_structs( + MapType::new(sample_struct_type(), sample_struct_type(), false), + vec![(sample_struct_scalar(), sample_struct_scalar())], + &[ + "+------------------------------+", + "| c |", + "+------------------------------+", + "| {{a: 1, b: x}: {a: 1, b: x}} |", + "+------------------------------+", + ] + )] + #[case::map_of_maps( + MapType::new(sample_map_type(), sample_map_type(), false), + vec![(sample_map_scalar(), sample_map_scalar())], + &[ + "+------------------+", + "| c |", + "+------------------+", + "| {{k: 1}: {k: 1}} |", + "+------------------+", + ] + )] + #[case::map_of_arrays( + MapType::new( + ArrayType::new(KernelDataType::INTEGER, false), + ArrayType::new(KernelDataType::INTEGER, false), + false, + ), + vec![(sample_int_array_scalar(), sample_int_array_scalar())], + &[ + "+------------------+", + "| c |", + "+------------------+", + "| {[1, 2]: [1, 2]} |", + "+------------------+", + ] + )] + fn nested_map_converts_to_map( + #[case] map_type: MapType, + #[case] pairs: Vec<(KernelScalar, KernelScalar)>, + #[case] expected: &[&str], + ) { + let data = KernelMapData::try_new(map_type, pairs).unwrap(); + let value = to_df_scalar(&KernelScalar::Map(data)).unwrap(); + assert!(matches!(value, DFScalarValue::Map(_)), "got {value:?}"); + assert_rendered(&value, expected); + } +} diff --git a/default-engine/Cargo.toml b/default-engine/Cargo.toml new file mode 100644 index 0000000000..7deb364d83 --- /dev/null +++ b/default-engine/Cargo.toml @@ -0,0 +1,53 @@ +[package] +name = "delta_kernel_default_engine" +description = "Default Arrow/Tokio-based engine implementation for delta_kernel." +edition.workspace = true +homepage.workspace = true +license.workspace = true +repository.workspace = true +readme.workspace = true +version.workspace = true +rust-version.workspace = true + +# for cargo-release +[package.metadata.release] +pre-release-replacements = [ + { file = "../README.md", search = "delta_kernel_default_engine = \\{ version = \"[a-z0-9\\.-]+\"", replace = "delta_kernel_default_engine = { version = \"{{version}}\"" }, +] + +[dependencies] +delta_kernel = { path = "../kernel", version = "0.27.1", features = [ + "default-engine-base", + "internal-api", +] } +async-stream = "0.3" +async-trait = "0.1" +bytes = "1.10" +chrono = "0.4.41" +futures = "0.3" +itertools = "0.14" +reqwest = { version = "0.13", default-features = false, features = ["charset", "http2", "query", "system-proxy"] } +serde_json = "1" +tokio = { version = "1.47", features = ["rt-multi-thread"] } +tracing = { version = "0.1", features = ["log"] } +url = "2" +uuid = { version = "1.18.0", features = ["v4"] } + +[features] +default = ["arrow", "rustls"] +arrow = ["arrow-59"] +arrow-59 = ["delta_kernel/arrow-59"] +arrow-58 = ["delta_kernel/arrow-58"] +native-tls = ["reqwest/native-tls"] +rustls = ["reqwest/rustls"] + +[dev-dependencies] +delta_kernel_default_engine_test_utils = { path = "./test-utils" } +rstest = "0.23" +serde = { version = "1", features = ["derive"] } +tempfile = "3" +test_utils = { path = "../test-utils", default-features = false } +tokio = { version = "1.47", features = ["rt-multi-thread", "macros"] } +tracing-subscriber = { version = "0.3", default-features = false, features = ["env-filter", "fmt"] } +wiremock = "0.6" +hdfs-native-object-store = { version = "0.16.0" } diff --git a/kernel/src/engine/default/executor.rs b/default-engine/src/executor.rs similarity index 89% rename from kernel/src/engine/default/executor.rs rename to default-engine/src/executor.rs index bcc8c1d583..641ef03741 100644 --- a/kernel/src/engine/default/executor.rs +++ b/default-engine/src/executor.rs @@ -10,9 +10,9 @@ //! A generic trait [TaskExecutor] can be implemented with your preferred async //! runtime. Behind the `tokio` feature flag, we provide a both a single-threaded //! and multi-threaded executor based on Tokio. -use futures::{future::BoxFuture, Future}; - -use crate::DeltaResult; +use delta_kernel::DeltaResult; +use futures::future::BoxFuture; +use futures::Future; /// An executor that can be used to run async tasks. This is used by IO functions /// within the `DefaultEngine`. @@ -49,23 +49,43 @@ pub trait TaskExecutor: Send + Sync + 'static { fn enter(&self) -> Self::Guard<'_>; } -#[cfg(any(feature = "tokio", test))] pub mod tokio { - use super::TaskExecutor; - use futures::TryFutureExt; - use futures::{future::BoxFuture, Future}; + use std::mem::ManuallyDrop; use std::sync::mpsc::channel; + + use delta_kernel::{DeltaResult, Error}; + use futures::future::BoxFuture; + use futures::{Future, TryFutureExt}; use tokio::runtime::{EnterGuard, Handle, RuntimeFlavor}; - use crate::{DeltaResult, Error}; + use crate::TaskExecutor; /// A [`TaskExecutor`] that uses the tokio single-threaded runtime in a /// background thread to service tasks. + /// + /// On drop, the background thread is joined to ensure the runtime is fully + /// shut down before the executor is destroyed. #[derive(Debug)] pub struct TokioBackgroundExecutor { - sender: tokio::sync::mpsc::Sender>, + sender: ManuallyDrop>>, handle: Handle, - _thread: std::thread::JoinHandle<()>, + /// `Option` because `join` takes ownership; we `take` it in `Drop` to move the + /// handle out. Never `None` outside of `Drop`. + thread: Option>, + } + + impl Drop for TokioBackgroundExecutor { + fn drop(&mut self) { + // SAFETY: The inner `Sender` has not been dropped yet because this is + // the only drop site and `Drop::drop` runs exactly once. + // Drop sender first to close the channel, signaling the background + // thread to exit its recv loop. + unsafe { ManuallyDrop::drop(&mut self.sender) }; + // Join the thread so that runtime shutdown completes before we return. + if let Some(thread) = self.thread.take() { + let _ = thread.join(); + } + } } impl Default for TokioBackgroundExecutor { @@ -93,9 +113,9 @@ pub mod tokio { }); let handle = handle_receiver.recv().unwrap(); Self { - sender, + sender: ManuallyDrop::new(sender), handle, - _thread: thread, + thread: Some(thread), } } } @@ -199,10 +219,10 @@ pub mod tokio { /// Create a new executor that owns its own multi-threaded Tokio runtime. /// /// # Parameters - /// - `worker_threads`: Number of worker threads. If `None`, uses Tokio's default. - /// See [`tokio::runtime::Builder::worker_threads`]. - /// - `max_blocking_threads`: Maximum number of threads for blocking operations. - /// If `None`, uses Tokio's default. See [`tokio::runtime::Builder::max_blocking_threads`]. + /// - `worker_threads`: Number of worker threads. If `None`, uses Tokio's default. See + /// [`tokio::runtime::Builder::worker_threads`]. + /// - `max_blocking_threads`: Maximum number of threads for blocking operations. If `None`, + /// uses Tokio's default. See [`tokio::runtime::Builder::max_blocking_threads`]. /// /// # Errors /// Returns an error if the runtime cannot be created. @@ -235,8 +255,8 @@ pub mod tokio { impl TaskExecutor for TokioMultiThreadExecutor { type Guard<'a> = EnterGuard<'a>; - // `block_on` uses `block_in_place`; If concurrent `block_on` calls exceed Tokio's `max_blocking_threads`, this can deadlock - // See: + // `block_on` uses `block_in_place`; If concurrent `block_on` calls exceed Tokio's + // `max_blocking_threads`, this can deadlock See: // https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.html#method.max_blocking_threads fn block_on(&self, task: T) -> T::Output where @@ -301,7 +321,10 @@ pub mod tokio { #[cfg(test)] mod test { - use super::*; + use std::sync::mpsc::channel; + + use super::{TokioBackgroundExecutor, TokioMultiThreadExecutor}; + use crate::*; async fn test_executor(executor: impl TaskExecutor) { // Can run a task @@ -421,7 +444,8 @@ pub mod tokio { tx.send(result).ok(); }); - // With 1 worker thread, 1 blocking thread and 4 nested block_on calls, this should deadlock + // With 1 worker thread, 1 blocking thread and 4 nested block_on calls, this should + // deadlock let timeout = Duration::from_millis(500); let result = rx.recv_timeout(timeout); diff --git a/kernel/src/engine/default/file_stream.rs b/default-engine/src/file_stream.rs similarity index 97% rename from kernel/src/engine/default/file_stream.rs rename to default-engine/src/file_stream.rs index ad93fcf087..c4c9f817a4 100644 --- a/kernel/src/engine/default/file_stream.rs +++ b/default-engine/src/file_stream.rs @@ -4,14 +4,13 @@ use std::ops::Range; use std::pin::Pin; use std::task::{ready, Context, Poll}; -use crate::arrow::array::RecordBatch; -use crate::arrow::datatypes::SchemaRef as ArrowSchemaRef; +use delta_kernel::arrow::array::RecordBatch; +use delta_kernel::arrow::datatypes::SchemaRef as ArrowSchemaRef; +use delta_kernel::{DeltaResult, FileMeta}; use futures::future::BoxFuture; use futures::stream::{BoxStream, Stream, StreamExt}; use futures::FutureExt; -use crate::{DeltaResult, FileMeta}; - /// A fallible future that resolves to a stream of [`RecordBatch`] /// cbindgen:ignore pub type FileOpenFuture = @@ -20,7 +19,7 @@ pub type FileOpenFuture = /// Generic API for opening a file using an [`ObjectStore`] and resolving to a /// stream of [`RecordBatch`] /// -/// [`ObjectStore`]: crate::object_store::ObjectStore +/// [`ObjectStore`]: delta_kernel::object_store::ObjectStore pub trait FileOpener: Send + Unpin { /// Asynchronously open the specified file and return a stream /// of [`RecordBatch`] @@ -179,7 +178,8 @@ impl FileStream { } Some(Err(err)) => { match self.on_error { - // If `OnError::Skip` we skip the file as soon as we hit the first error + // If `OnError::Skip` we skip the file as soon as we hit the first + // error OnError::Skip => match mem::take(next) { Some(future) => match future { NextOpen::Pending(future) => { diff --git a/kernel/src/engine/default/filesystem.rs b/default-engine/src/filesystem.rs similarity index 72% rename from kernel/src/engine/default/filesystem.rs rename to default-engine/src/filesystem.rs index 513ba6bb55..929697e042 100644 --- a/kernel/src/engine/default/filesystem.rs +++ b/default-engine/src/filesystem.rs @@ -1,135 +1,28 @@ -use std::pin::Pin; use std::sync::Arc; -use std::task::{Context, Poll}; -use std::time::{Duration, Instant}; use bytes::Bytes; -use delta_kernel_derive::internal_api; -use futures::stream::{self, BoxStream, Stream, StreamExt, TryStreamExt}; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::{self, DynObjectStore, ObjectStoreExt as _, PutMode}; +use delta_kernel::{DeltaResult, Error, FileMeta, FileSlice, StorageHandler}; +use futures::stream::{self, BoxStream, StreamExt, TryStreamExt}; use itertools::Itertools; use url::Url; -use super::UrlExt; -use crate::engine::default::executor::TaskExecutor; -use crate::metrics::{MetricEvent, MetricsReporter}; -use crate::object_store::path::Path; -use crate::object_store::{self, DynObjectStore, ObjectStore, PutMode}; -use crate::{DeltaResult, Error, FileMeta, FileSlice, StorageHandler}; - -/// Iterator wrapper that emits metrics when exhausted -/// -/// Generic over the inner iterator type and item type. -/// The `event_fn` receives (duration, num_files, bytes_read) to construct the appropriate MetricEvent. -/// Metrics are emitted either when the iterator is exhausted or when dropped. -struct MetricsIterator { - inner: I, - reporter: Option>, - start: Instant, - num_files: u64, - bytes_read: u64, - event_fn: fn(Duration, u64, u64) -> MetricEvent, - _phantom: std::marker::PhantomData, -} - -impl MetricsIterator { - fn new( - inner: I, - reporter: Option>, - start: Instant, - event_fn: fn(Duration, u64, u64) -> MetricEvent, - ) -> Self { - Self { - inner, - reporter, - start, - num_files: 0, - bytes_read: 0, - event_fn, - _phantom: std::marker::PhantomData, - } - } - - fn emit_metrics_once(&mut self) { - if let Some(r) = self.reporter.take() { - r.report((self.event_fn)( - self.start.elapsed(), - self.num_files, - self.bytes_read, - )); - } - } -} - -impl Drop for MetricsIterator { - fn drop(&mut self) { - self.emit_metrics_once(); - } -} - -impl Stream for MetricsIterator -where - I: Stream> + Unpin, -{ - type Item = I::Item; - - fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { - match futures::ready!(Pin::new(&mut self.inner).poll_next(cx)) { - Some(item) => { - if item.is_ok() { - self.num_files += 1; - } - Poll::Ready(Some(item)) - } - None => { - self.emit_metrics_once(); - Poll::Ready(None) - } - } - } -} - -impl Stream for MetricsIterator -where - I: Stream> + Unpin, -{ - type Item = I::Item; - - fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { - match futures::ready!(Pin::new(&mut self.inner).poll_next(cx)) { - Some(item) => { - if let Ok(ref bytes) = item { - self.num_files += 1; - self.bytes_read += bytes.len() as u64; - } - Poll::Ready(Some(item)) - } - None => { - self.emit_metrics_once(); - Poll::Ready(None) - } - } - } -} +use crate::executor::TaskExecutor; +use crate::UrlExt; #[derive(Debug)] pub struct ObjectStoreStorageHandler { inner: Arc, task_executor: Arc, - reporter: Option>, readahead: usize, } impl ObjectStoreStorageHandler { - #[internal_api] - pub(crate) fn new( - store: Arc, - task_executor: Arc, - reporter: Option>, - ) -> Self { + pub(crate) fn new(store: Arc, task_executor: Arc) -> Self { Self { inner: store, task_executor, - reporter, readahead: 10, } } @@ -141,17 +34,21 @@ impl ObjectStoreStorageHandler { } } -/// Native async implementation for list_from +/// Native async implementation for list_from. +/// +/// Storage metrics are emitted by the outer [`MeteredStorageHandler`] wrapping this +/// handler (e.g. inside `DefaultEngine`'s `storage_handler()`), so this function just +/// returns the raw stream. +/// +/// [`MeteredStorageHandler`]: delta_kernel::metrics::MeteredStorageHandler async fn list_from_impl( store: Arc, path: Url, - reporter: Option>, ) -> DeltaResult>> { - let start = Instant::now(); - - // The offset is used for list-after; the prefix is used to restrict the listing to a specific directory. - // Unfortunately, `Path` provides no easy way to check whether a name is directory-like, - // because it strips trailing /, so we're reduced to manually checking the original URL. + // The offset is used for list-after; the prefix is used to restrict the listing to a specific + // directory. Unfortunately, `Path` provides no easy way to check whether a name is + // directory-like, because it strips trailing /, so we're reduced to manually checking the + // original URL. let offset = Path::from_url_path(path.path())?; let prefix = if path.path().ends_with('/') { offset.clone() @@ -184,24 +81,10 @@ async fn list_from_impl( // Local filesystem doesn't return sorted list - need to collect and sort let mut items: Vec<_> = stream.try_collect().await?; items.sort_unstable(); - - if let Some(r) = reporter { - r.report(MetricEvent::StorageListCompleted { - duration: start.elapsed(), - num_files: items.len() as u64, - }); - } - Ok(Box::pin(stream::iter(items.into_iter().map(Ok)))) + Ok(Box::pin(stream::iter( + items.into_iter().map(Ok::), + ))) } else { - let stream = MetricsIterator::new( - stream, - reporter, - start, - |duration, num_files, _bytes_read| MetricEvent::StorageListCompleted { - duration, - num_files, - }, - ); Ok(Box::pin(stream)) } } @@ -211,9 +94,7 @@ async fn read_files_impl( store: Arc, files: Vec, readahead: usize, - reporter: Option>, ) -> DeltaResult>> { - let start = Instant::now(); let files = stream::iter(files).map(move |(url, range)| { let store = store.clone(); async move { @@ -244,16 +125,7 @@ async fn read_files_impl( // We allow executing up to `readahead` futures concurrently and // buffer the results. This allows us to achieve async concurrency. - Ok(Box::pin(MetricsIterator::new( - files.buffered(readahead), - reporter, - start, - |duration, num_files, bytes_read| MetricEvent::StorageReadCompleted { - duration, - num_files, - bytes_read, - }, - ))) + Ok(Box::pin(files.buffered(readahead))) } /// Native async implementation for copy_atomic @@ -261,28 +133,18 @@ async fn copy_atomic_impl( store: Arc, src_path: Path, dest_path: Path, - reporter: Option>, ) -> DeltaResult<()> { - let start = Instant::now(); - // Read source file then write atomically with PutMode::Create. Note that a GET/PUT is not // necessarily atomic, but since the source file is immutable, we aren't exposed to the // possibility of source file changing while we do the PUT. let data = store.get(&src_path).await?.bytes().await?; - let result = store + store .put_opts(&dest_path, data.into(), PutMode::Create.into()) - .await; - - if let Some(r) = reporter { - r.report(MetricEvent::StorageCopyCompleted { - duration: start.elapsed(), - }); - } - - result.map_err(|e| match e { - object_store::Error::AlreadyExists { .. } => Error::FileAlreadyExists(dest_path.into()), - e => e.into(), - })?; + .await + .map_err(|e| match e { + object_store::Error::AlreadyExists { .. } => Error::FileAlreadyExists(dest_path.into()), + e => e.into(), + })?; Ok(()) } @@ -306,6 +168,15 @@ async fn put_impl( Ok(()) } +/// Native async implementation for delete. +async fn delete_impl(store: Arc, path: Path) -> DeltaResult<()> { + match store.delete(&path).await { + Ok(()) => Ok(()), + Err(object_store::Error::NotFound { .. }) => Ok(()), + Err(e) => Err(e.into()), + } +} + /// Native async implementation for head async fn head_impl(store: Arc, url: Url) -> DeltaResult { let meta = store.head(&Path::from_url_path(url.path())?).await?; @@ -321,7 +192,7 @@ impl StorageHandler for ObjectStoreStorageHandler { &self, path: &Url, ) -> DeltaResult>>> { - let future = list_from_impl(self.inner.clone(), path.clone(), self.reporter.clone()); + let future = list_from_impl(self.inner.clone(), path.clone()); let iter = super::stream_future_to_iter(self.task_executor.clone(), future)?; Ok(iter) // type coercion drops the unneeded Send bound } @@ -336,12 +207,7 @@ impl StorageHandler for ObjectStoreStorageHandler { &self, files: Vec, ) -> DeltaResult>>> { - let future = read_files_impl( - self.inner.clone(), - files, - self.readahead, - self.reporter.clone(), - ); + let future = read_files_impl(self.inner.clone(), files, self.readahead); let iter = super::stream_future_to_iter(self.task_executor.clone(), future)?; Ok(iter) // type coercion drops the unneeded Send bound } @@ -355,12 +221,7 @@ impl StorageHandler for ObjectStoreStorageHandler { fn copy_atomic(&self, src: &Url, dest: &Url) -> DeltaResult<()> { let src_path = Path::from_url_path(src.path())?; let dest_path = Path::from_url_path(dest.path())?; - let future = copy_atomic_impl( - self.inner.clone(), - src_path, - dest_path, - self.reporter.clone(), - ); + let future = copy_atomic_impl(self.inner.clone(), src_path, dest_path); self.task_executor.block_on(future) } @@ -368,6 +229,12 @@ impl StorageHandler for ObjectStoreStorageHandler { let future = head_impl(self.inner.clone(), path.clone()); self.task_executor.block_on(future) } + + fn delete(&self, path: &Url) -> DeltaResult<()> { + let path = Path::from_url_path(path.path())?; + self.task_executor + .block_on(delete_impl(self.inner.clone(), path)) + } } /// Returns whether or not the [Url] can support ordered listing. @@ -380,19 +247,17 @@ impl StorageHandler for ObjectStoreStorageHandler { /// /// Although the `object_store` crate explicitly says it _does not_ return a sorted listing, in /// practice many implementations actually do: -/// - AWS: -/// [`ListObjectsV2`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) -/// states: "For general purpose buckets, ListObjectsV2 returns objects in lexicographical -/// order based on their key names." -/// - Azure: Docs state -/// [here](https://learn.microsoft.com/en-us/rest/api/storageservices/enumerating-blob-resources): -/// "A listing operation returns an XML response that contains all or part of the requested -/// list. The operation returns entities in alphabetical order." -/// - GCP: The [main](https://cloud.google.com/storage/docs/xml-api/get-bucket-list) doc -/// doesn't indicate order, but [this -/// page](https://cloud.google.com/storage/docs/xml-api/get-bucket-list) does say: "This page -/// shows you how to list the [objects](https://cloud.google.com/storage/docs/objects) stored -/// in your Cloud Storage buckets, which are ordered in the list lexicographically by name." +/// - AWS: [`ListObjectsV2`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) +/// states: "For general purpose buckets, ListObjectsV2 returns objects in lexicographical order +/// based on their key names." +/// - Azure: Docs state [here](https://learn.microsoft.com/en-us/rest/api/storageservices/enumerating-blob-resources): +/// "A listing operation returns an XML response that contains all or part of the requested list. +/// The operation returns entities in alphabetical order." +/// - GCP: The [main](https://cloud.google.com/storage/docs/xml-api/get-bucket-list) doc doesn't indicate +/// order, but [this page](https://cloud.google.com/storage/docs/xml-api/get-bucket-list) does say: +/// "This page shows you how to list the [objects](https://cloud.google.com/storage/docs/objects) +/// stored in your Cloud Storage buckets, which are ordered in the list lexicographically by +/// name." fn supports_ordered_listing(url: &Url) -> bool { !((url.scheme() == "file") // S3 Directory Buckets @@ -406,18 +271,16 @@ mod tests { use std::ops::Range; use std::time::Duration; + use delta_kernel::object_store::local::LocalFileSystem; + use delta_kernel::object_store::memory::InMemory; + use delta_kernel::Engine as _; + use delta_kernel_default_engine_test_utils::current_time_duration; use itertools::Itertools; - use test_utils::delta_path_for_version; - use crate::engine::default::executor::tokio::TokioBackgroundExecutor; - use crate::engine::default::DefaultEngineBuilder; - use crate::object_store::memory::InMemory; - use crate::object_store::{local::LocalFileSystem, ObjectStore}; - use crate::utils::current_time_duration; - use crate::Engine as _; - use super::*; + use crate::executor::tokio::TokioBackgroundExecutor; + use crate::DefaultEngineBuilder; fn setup_test() -> ( tempfile::TempDir, @@ -427,7 +290,7 @@ mod tests { let tmp = tempfile::tempdir().unwrap(); let store = Arc::new(LocalFileSystem::new()); let executor = Arc::new(TokioBackgroundExecutor::new()); - let handler = ObjectStoreStorageHandler::new(store.clone(), executor, None); + let handler = ObjectStoreStorageHandler::new(store.clone(), executor); (tmp, store, handler) } @@ -474,7 +337,7 @@ mod tests { let store = Arc::new(LocalFileSystem::new()); let executor = Arc::new(TokioBackgroundExecutor::new()); - let storage = ObjectStoreStorageHandler::new(store, executor, None); + let storage = ObjectStoreStorageHandler::new(store, executor); let mut slices: Vec = Vec::new(); @@ -666,4 +529,32 @@ mod tests { assert_eq!(read_back.len(), 1); assert_eq!(read_back[0], new_data); } + + #[test] + fn test_delete() { + let (tmp, _store, handler) = setup_test(); + + let data = Bytes::from("delete-test-data"); + let file_url = Url::from_file_path(tmp.path().join("delete.txt")).unwrap(); + handler.put(&file_url, data, false).unwrap(); + + handler.delete(&file_url).unwrap(); + + assert!(matches!( + handler.head(&file_url), + Err(Error::FileNotFound(_)) + )); + } + + #[test] + fn test_delete_nonexistent_is_ok() { + let (tmp, _store, handler) = setup_test(); + + let missing_url = Url::from_file_path(tmp.path().join("missing.txt")).unwrap(); + assert!(matches!( + handler.head(&missing_url), + Err(Error::FileNotFound(_)) + )); + handler.delete(&missing_url).unwrap(); + } } diff --git a/kernel/src/engine/default/json.rs b/default-engine/src/json.rs similarity index 87% rename from kernel/src/engine/default/json.rs rename to default-engine/src/json.rs index 5b293e534f..1e417ea8a2 100644 --- a/kernel/src/engine/default/json.rs +++ b/default-engine/src/json.rs @@ -1,29 +1,33 @@ //! Default Json handler implementation use std::io::BufReader; +use std::num::NonZero; use std::sync::Arc; use std::task::Poll; -use crate::arrow::datatypes::SchemaRef as ArrowSchemaRef; -use crate::arrow::json::ReaderBuilder; -use crate::arrow::record_batch::RecordBatch; -use crate::object_store::path::Path; -use crate::object_store::{self, DynObjectStore, GetResultPayload, PutMode}; use bytes::{Buf, Bytes}; -use futures::stream::{self, BoxStream}; -use futures::{ready, StreamExt, TryStreamExt}; -use url::Url; - -use super::executor::TaskExecutor; -use crate::engine::arrow_utils::{ +use delta_kernel::arrow::datatypes::SchemaRef as ArrowSchemaRef; +use delta_kernel::arrow::json::ReaderBuilder; +use delta_kernel::arrow::record_batch::RecordBatch; +use delta_kernel::engine::arrow_utils::{ build_json_reorder_indices, fixup_json_read, json_arrow_schema, parse_json as arrow_parse_json, to_json_bytes, }; -use crate::engine_data::FilteredEngineData; -use crate::schema::SchemaRef; -use crate::{ - DeltaResult, EngineData, Error, FileDataReadResultIterator, FileMeta, JsonHandler, PredicateRef, +use delta_kernel::engine_data::FilteredEngineData; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::{ + self, DynObjectStore, GetResultPayload, ObjectStoreExt as _, PutMode, }; +use delta_kernel::schema::SchemaRef; +use delta_kernel::{ + CancellationTokenRef, DeltaResult, DeltaResultIterator, EngineData, Error, + FileDataReadResultIterator, FileMeta, JsonHandler, PredicateRef, +}; +use futures::stream::{self, BoxStream}; +use futures::{ready, StreamExt, TryStreamExt}; +use url::Url; + +use crate::executor::TaskExecutor; #[derive(Debug)] pub struct DefaultJsonHandler { @@ -34,10 +38,10 @@ pub struct DefaultJsonHandler { /// The maximum number of read requests to buffer in memory at once. Note that this actually /// controls two things: the number of concurrent requests (done by `buffered`) and the size of /// the buffer (via our `sync_channel`). - buffer_size: usize, + buffer_size: NonZero, /// Limit the number of rows per batch. That is, for batch_size = N, then each RecordBatch /// yielded by the stream will have at most N rows. - batch_size: usize, + batch_size: NonZero, } impl DefaultJsonHandler { @@ -45,21 +49,21 @@ impl DefaultJsonHandler { Self { store, task_executor, - buffer_size: super::DEFAULT_BUFFER_SIZE, - batch_size: super::DEFAULT_BATCH_SIZE, + buffer_size: super::DEFAULT_READ_BUFFER_SIZE, + batch_size: super::DEFAULT_READ_BATCH_SIZE, } } /// Set the maximum number read requests to buffer in memory at once in /// [Self::read_json_files()]. /// - /// Defaults to 1000. + /// Defaults to `super::DEFAULT_READ_BUFFER_SIZE`. /// /// Memory constraints can be imposed by constraining the buffer size and batch size. Note that /// overall memory usage is proportional to the product of these two values. /// 1. Batch size governs the size of RecordBatches yielded in each iteration of the stream /// 2. Buffer size governs the number of concurrent tasks (which equals the size of the buffer - pub fn with_buffer_size(mut self, buffer_size: usize) -> Self { + pub fn with_buffer_size(mut self, buffer_size: NonZero) -> Self { self.buffer_size = buffer_size; self } @@ -67,13 +71,13 @@ impl DefaultJsonHandler { /// Limit the number of rows per batch. That is, for batch_size = N, then each RecordBatch /// yielded by the stream will have at most N rows. /// - /// Defaults to 1000 rows (json objects). + /// Defaults to `super::DEFAULT_READ_BATCH_SIZE` rows (json objects). /// /// See [Decoder::with_buffer_size] for details on constraining memory usage with buffer size /// and batch size. /// - /// [Decoder::with_buffer_size]: crate::arrow::json::reader::Decoder - pub fn with_batch_size(mut self, batch_size: usize) -> Self { + /// [Decoder::with_buffer_size]: delta_kernel::arrow::json::reader::Decoder + pub fn with_batch_size(mut self, batch_size: NonZero) -> Self { self.batch_size = batch_size; self } @@ -160,23 +164,37 @@ impl JsonHandler for DefaultJsonHandler { files: &[FileMeta], physical_schema: SchemaRef, predicate: Option, + ) -> DeltaResult { + self.read_json_files_with_cancellation(files, physical_schema, predicate, None) + } + + fn read_json_files_with_cancellation( + &self, + files: &[FileMeta], + physical_schema: SchemaRef, + predicate: Option, + cancellation_token: Option, ) -> DeltaResult { let future = read_json_files_impl( self.store.clone(), files.to_vec(), physical_schema, predicate, - self.batch_size, - self.buffer_size, + self.batch_size.get(), + self.buffer_size.get(), ); - super::stream_future_to_iter(self.task_executor.clone(), future) + super::stream_future_to_cancellable_iter( + self.task_executor.clone(), + future, + cancellation_token, + ) } // note: for now we just buffer all the data and write it out all at once fn write_json_file( &self, path: &Url, - data: Box> + Send + '_>, + data: DeltaResultIterator<'_, FilteredEngineData>, overwrite: bool, ) -> DeltaResult<()> { self.task_executor.block_on(write_json_file_impl( @@ -265,42 +283,32 @@ mod tests { use std::sync::{mpsc, Arc, Mutex}; use std::task::Waker; - use crate::actions::get_commit_schema; - use crate::arrow::array::{Array, AsArray, Int32Array, RecordBatch, StringArray}; - use crate::arrow::datatypes::{DataType, Field, Schema as ArrowSchema}; - use crate::engine::arrow_data::{ArrowEngineData, EngineDataArrowExt as _}; - use crate::engine::default::executor::tokio::{ - TokioBackgroundExecutor, TokioMultiThreadExecutor, - }; - use crate::object_store::local::LocalFileSystem; - use crate::object_store::memory::InMemory; - use crate::object_store::PutMultipartOptions; - use crate::object_store::{ - GetOptions, GetResult, ListResult, MultipartUpload, ObjectMeta, ObjectStore, PutOptions, - PutPayload, PutResult, Result, + use delta_kernel::actions::get_commit_schema; + use delta_kernel::arrow::array::{Array, AsArray, Int32Array, RecordBatch, StringArray}; + use delta_kernel::arrow::datatypes::{DataType, Field, Schema as ArrowSchema}; + use delta_kernel::engine::arrow_data::{ArrowEngineData, EngineDataArrowExt as _}; + use delta_kernel::object_store::local::LocalFileSystem; + use delta_kernel::object_store::memory::InMemory; + use delta_kernel::object_store::{ + CopyOptions, GetOptions, GetResult, ListResult, MultipartUpload, ObjectMeta, ObjectStore, + PutMultipartOptions, PutOptions, PutPayload, PutResult, Result, }; - use crate::schema::{DataType as DeltaDataType, Schema, StructField}; - use crate::utils::test_utils::string_array_to_engine_data; + use delta_kernel::schema::schema_ref; + use delta_kernel_default_engine_test_utils::{into_record_batch, string_array_to_engine_data}; use futures::future; use itertools::Itertools; use serde_json::json; + use test_utils::engine_contract::test_json_handler_file_path_contract; use tracing::info; - // TODO: should just use the one from test_utils, but running into dependency issues - fn into_record_batch(engine_data: Box) -> RecordBatch { - ArrowEngineData::try_from_engine_data(engine_data) - .unwrap() - .into() - } - use super::*; + use crate::executor::tokio::{TokioBackgroundExecutor, TokioMultiThreadExecutor}; /// Store wrapper that wraps an inner store to guarantee the ordering of GET requests. Note /// that since the keys are resolved in order, requests to subsequent keys in the order will /// block until the earlier keys are requested. /// /// WARN: Does not handle duplicate keys, and will fail on duplicate requests of the same key. - /// // TODO(zach): we can handle duplicate requests if we retain the ordering of the keys track // that all of the keys prior to the one requested have been resolved. #[derive(Debug)] @@ -350,11 +358,7 @@ mod tests { } #[async_trait::async_trait] - impl ObjectStore for OrderedGetStore { - async fn put(&self, location: &Path, payload: PutPayload) -> Result { - self.inner.put(location, payload).await - } - + impl delta_kernel::object_store::ObjectStore for OrderedGetStore { async fn put_opts( &self, location: &Path, @@ -364,10 +368,6 @@ mod tests { self.inner.put_opts(location, payload, opts).await } - async fn put_multipart(&self, location: &Path) -> Result> { - self.inner.put_multipart(location).await - } - async fn put_multipart_opts( &self, location: &Path, @@ -380,9 +380,17 @@ mod tests { // - if yes, remove the path from the queue and proceed with the GET request, then wake the // next path in order // - if no, register the waker and wait - async fn get(&self, location: &Path) -> Result { - // Do the actual GET request first, then introduce any artificial ordering delays as needed - let result = self.inner.get(location).await; + async fn get_opts(&self, location: &Path, options: GetOptions) -> Result { + // object_store 0.13 implements `head()` via `get_opts(..., head = true)`. + // The ordering queue is only meant to serialize content reads for the test, so + // skip queue accounting for HEAD probes used while constructing FileMeta. + if options.head { + return self.inner.get_opts(location, options).await; + } + + // Do the actual GET request first, then introduce any artificial ordering delays as + // needed + let result = self.inner.get_opts(location, options).await; // we implement a future which only resolves once the requested path is next in order future::poll_fn(move |cx| { @@ -438,24 +446,15 @@ mod tests { result } - async fn get_opts(&self, location: &Path, options: GetOptions) -> Result { - self.inner.get_opts(location, options).await - } - - async fn get_range(&self, location: &Path, range: Range) -> Result { - self.inner.get_range(location, range).await - } - async fn get_ranges(&self, location: &Path, ranges: &[Range]) -> Result> { self.inner.get_ranges(location, ranges).await } - async fn head(&self, location: &Path) -> Result { - self.inner.head(location).await - } - - async fn delete(&self, location: &Path) -> Result<()> { - self.inner.delete(location).await + fn delete_stream( + &self, + locations: BoxStream<'static, Result>, + ) -> BoxStream<'static, Result> { + self.inner.delete_stream(locations) } fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, Result> { @@ -474,20 +473,8 @@ mod tests { self.inner.list_with_delimiter(prefix).await } - async fn copy(&self, from: &Path, to: &Path) -> Result<()> { - self.inner.copy(from, to).await - } - - async fn rename(&self, from: &Path, to: &Path) -> Result<()> { - self.inner.rename(from, to).await - } - - async fn copy_if_not_exists(&self, from: &Path, to: &Path) -> Result<()> { - self.inner.copy_if_not_exists(from, to).await - } - - async fn rename_if_not_exists(&self, from: &Path, to: &Path) -> Result<()> { - self.inner.rename_if_not_exists(from, to).await + async fn copy_opts(&self, from: &Path, to: &Path, options: CopyOptions) -> Result<()> { + self.inner.copy_opts(from, to, options).await } } @@ -511,9 +498,10 @@ mod tests { } // Test that operationParameters with boolean/numeric primitives are coerced to strings. - // Some delta logs contain values like `"statsOnLoad": false` instead of `"statsOnLoad": "false"`. - // Without `with_coerce_primitive(true)`, this would fail with: - // "whilst decoding field 'commitInfo': whilst decoding field 'operationParameters': expected string got false" + // Some delta logs contain values like `"statsOnLoad": false` instead of `"statsOnLoad": + // "false"`. Without `with_coerce_primitive(true)`, this would fail with: + // "whilst decoding field 'commitInfo': whilst decoding field 'operationParameters': expected + // string got false" #[test] fn test_parse_json_coerce_operation_parameters() { let store = Arc::new(LocalFileSystem::new()); @@ -588,7 +576,7 @@ mod tests { let store = Arc::new(LocalFileSystem::new()); let path = std::fs::canonicalize(PathBuf::from( - "./tests/data/table-with-dv-small/_delta_log/00000000000000000000.json", + "../kernel/tests/data/table-with-dv-small/_delta_log/00000000000000000000.json", )) .unwrap(); let url = Url::from_file_path(path).unwrap(); @@ -613,7 +601,7 @@ mod tests { assert_eq!(data[0].num_rows(), 4); // limit batch size - let handler = handler.with_batch_size(2); + let handler = handler.with_batch_size(NonZero::new(2).unwrap()); let data: Vec = handler .read_json_files(files, get_commit_schema().clone(), None) .unwrap() @@ -684,12 +672,13 @@ mod tests { ); } - use crate::engine::default::DefaultEngineBuilder; - use crate::schema::StructType; - use crate::Engine; use std::io::Write; + + use delta_kernel::Engine; use tempfile::NamedTempFile; + use crate::DefaultEngineBuilder; + fn make_invalid_named_temp() -> (NamedTempFile, Url) { let mut temp_file = NamedTempFile::new().expect("Failed to create temp file"); write!(temp_file, r#"this is not valid json"#).expect("Failed to write to temp file"); @@ -705,8 +694,7 @@ mod tests { let _ = tracing_subscriber::fmt().try_init(); let (_temp_file1, file_url1) = make_invalid_named_temp(); let (_temp_file2, file_url2) = make_invalid_named_temp(); - let field = StructField::nullable("name", crate::schema::DataType::BOOLEAN); - let schema = Arc::new(StructType::try_new(vec![field]).unwrap()); + let schema = schema_ref! { nullable "name": BOOLEAN }; let default_engine = DefaultEngineBuilder::new(Arc::new(LocalFileSystem::new())).build(); // Helper to check that we get expected number of errors then stream ends @@ -823,11 +811,9 @@ mod tests { tokio::runtime::Handle::current(), )), ); - let handler = handler.with_buffer_size(*buffer_size); - let physical_schema = Arc::new(Schema::new_unchecked(vec![StructField::nullable( - "val", - DeltaDataType::INTEGER, - )])); + let handler = handler + .with_buffer_size(NonZero::new(*buffer_size).expect("buffer_size is non-zero")); + let physical_schema = schema_ref! { nullable "val": INTEGER }; let data: Vec = handler .read_json_files(&files, physical_schema, None) .unwrap() @@ -929,4 +915,18 @@ mod tests { Ok(()) } + + // === JsonHandler contract tests === + // + // Calls the shared contract helper in `engine::tests` against `DefaultJsonHandler` (the + // matching `SyncJsonHandler` invocation lives in `engine/sync/json.rs`). + + #[test] + fn json_handler_file_path_contract() { + let handler = DefaultJsonHandler::new( + Arc::new(LocalFileSystem::new()), + Arc::new(TokioBackgroundExecutor::new()), + ); + test_json_handler_file_path_contract(&handler); + } } diff --git a/default-engine/src/lib.rs b/default-engine/src/lib.rs new file mode 100644 index 0000000000..1ed387c1c4 --- /dev/null +++ b/default-engine/src/lib.rs @@ -0,0 +1,617 @@ +//! # The Default Engine +//! +//! The default implementation of [`Engine`] is [`DefaultEngine`]. +//! +//! The underlying implementations use asynchronous IO. Async tasks are run on +//! a separate thread pool, provided by the [`TaskExecutor`] trait. Read more in +//! the [executor] module. + +use std::future::Future; +use std::num::NonZero; +use std::sync::Arc; + +use delta_kernel::engine::arrow_conversion::TryFromArrow as _; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::engine::arrow_expression::ArrowEvaluationHandler; +use delta_kernel::metrics::{MeteredJsonHandler, MeteredParquetHandler, MeteredStorageHandler}; +use delta_kernel::object_store::DynObjectStore; +use delta_kernel::schema::Schema; +use delta_kernel::transaction::WriteContext; +use delta_kernel::{ + CancellationTokenRef, DeltaResult, Engine, EngineData, Error, EvaluationHandler, JsonHandler, + ParquetHandler, StorageHandler, +}; +use futures::future::{self, Either}; +use futures::stream::{BoxStream, StreamExt as _}; +use url::Url; + +use self::executor::TaskExecutor; +use self::filesystem::ObjectStoreStorageHandler; +use self::json::DefaultJsonHandler; +use self::parquet::DefaultParquetHandler; + +pub mod executor; +pub mod file_stream; +pub mod filesystem; +pub mod json; +pub mod parquet; +pub mod rest_store; +pub mod stats; +pub mod storage; + +/// Converts a Stream-producing future to a synchronous iterator. +/// +/// This method performs the initial blocking call to extract the stream from the future, and each +/// subsequent call to `next` on the iterator translates to a blocking `stream.next()` call, using +/// the provided `task_executor`. Buffered streams allow concurrency in the form of prefetching, +/// because that initial call will attempt to populate the N buffer slots; every call to +/// `stream.next()` leaves an empty slot (out of N buffer slots) that the stream immediately +/// attempts to fill by launching another future that can make progress in the background while we +/// block on and consume each of the N-1 entries that precede it. +/// +/// This is an internal utility for bridging object_store's async API to +/// Delta Kernel's synchronous handler traits. +pub(crate) fn stream_future_to_iter( + task_executor: Arc, + stream_future: impl Future>> + Send + 'static, +) -> DeltaResult + Send>> { + Ok(Box::new(BlockingStreamIterator { + stream: Some(task_executor.block_on(stream_future)?), + task_executor, + })) +} + +/// Like [`stream_future_to_iter`], but each blocking poll is raced against the cancellation +/// token. When the token fires, the iterator yields a single `Err(Error::Cancelled)` and then +/// ends, abandoning the in-flight read (dropping the stream releases its buffered work). +/// +/// Restricted to `DeltaResult` streams so cancellation can be surfaced as an item. With a `None` +/// token, behavior is identical to [`stream_future_to_iter`]. +pub(crate) fn stream_future_to_cancellable_iter( + task_executor: Arc, + stream_future: impl Future>>> + + Send + + 'static, + cancellation_token: Option, +) -> DeltaResult> + Send>> { + let Some(token) = cancellation_token else { + return stream_future_to_iter(task_executor, stream_future); + }; + // Race even the initial stream-producing future against cancellation. + let stream = match block_on_or_cancelled(&task_executor, token.clone(), stream_future) { + Some(result) => result?, + None => return Err(Error::Cancelled), + }; + Ok(Box::new(CancellableStreamIterator { + stream: Some(stream), + task_executor, + token, + })) +} + +/// Blocks on `future`, racing it against `token`'s cancellation. Returns `Some(output)` if the +/// future completed first, or `None` if cancellation won. Fast-paths an already-cancelled token. +pub(crate) fn block_on_or_cancelled( + task_executor: &Arc, + token: CancellationTokenRef, + future: impl Future + Send + 'static, +) -> Option +where + T: Send + 'static, +{ + if token.is_cancelled() { + return None; + } + task_executor.block_on(async move { + let cancelled = token.cancelled_future(); + futures::pin_mut!(cancelled); + match future::select(std::pin::pin!(future), cancelled).await { + Either::Left((output, _)) => Some(output), + Either::Right(((), _)) => None, + } + }) +} + +struct BlockingStreamIterator { + stream: Option>, + task_executor: Arc, +} + +impl Iterator for BlockingStreamIterator { + type Item = T; + + fn next(&mut self) -> Option { + // Move the stream into the future so we can block on it. + let mut stream = self.stream.take()?; + let (item, stream) = self + .task_executor + .block_on(async move { (stream.next().await, stream) }); + + // We must not poll an exhausted stream after it returned None. + if item.is_some() { + self.stream = Some(stream); + } + + item + } +} + +/// Cancellation-aware counterpart to [`BlockingStreamIterator`]: each `next()` races the blocking +/// `stream.next()` against the token and, once cancelled, drops the stream and yields exactly one +/// terminal `Err(Error::Cancelled)`. +struct CancellableStreamIterator { + stream: Option>>, + task_executor: Arc, + token: CancellationTokenRef, +} + +impl Iterator for CancellableStreamIterator { + type Item = DeltaResult; + + fn next(&mut self) -> Option { + let mut stream = self.stream.take()?; + match block_on_or_cancelled(&self.task_executor, self.token.clone(), async move { + let item = stream.next().await; + (item, stream) + }) { + Some((item, stream)) => { + // We must not poll an exhausted stream after it returned None. + if item.is_some() { + self.stream = Some(stream); + } + item + } + // Cancelled: `stream` was moved into the (now-dropped) future, releasing buffered + // work. Emit one terminal error; the taken `self.stream` stays `None`, fusing us. + None => Some(Err(Error::Cancelled)), + } + } +} + +const DEFAULT_BUFFER_SIZE: usize = 1000; +const DEFAULT_BATCH_SIZE: usize = 1000; + +/// Default file-level readahead depth for JSON and Parquet handlers. +pub(crate) const DEFAULT_READ_BUFFER_SIZE: NonZero = + NonZero::new(DEFAULT_BUFFER_SIZE).unwrap(); +/// Default row batch size for JSON and Parquet read streams. +pub(crate) const DEFAULT_READ_BATCH_SIZE: NonZero = + NonZero::new(DEFAULT_BATCH_SIZE).unwrap(); + +#[derive(Debug)] +pub struct DefaultEngine { + object_store: Arc, + task_executor: Arc, + storage: Arc, + json: Arc, + parquet: Arc, + /// Concrete parquet handler retained so [`Self::write_parquet`] and + /// [`Self::default_parquet_handler`] can reach the inherent `write_parquet_file` + /// helper, which the [`ParquetHandler`] trait surface doesn't expose. + raw_parquet: Arc>, + evaluation: Arc, +} + +/// Builder for creating [`DefaultEngine`] instances. +/// +/// # Example +/// +/// ```no_run +/// # use std::sync::Arc; +/// # use delta_kernel_default_engine::DefaultEngineBuilder; +/// # use delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor; +/// # use delta_kernel::object_store::local::LocalFileSystem; +/// // Build a DefaultEngine with default executor +/// let engine = DefaultEngineBuilder::new(Arc::new(LocalFileSystem::new())) +/// .build(); +/// +/// // Build with a custom executor +/// let engine = DefaultEngineBuilder::new(Arc::new(LocalFileSystem::new())) +/// .with_task_executor(Arc::new(TokioBackgroundExecutor::new())) +/// .build(); +/// ``` +#[derive(Debug)] +pub struct DefaultEngineBuilder { + object_store: Arc, + /// The state is either [`DefaultTaskExecutor`] or `Arc` with a custom task executor. + task_executor: E, + /// Read-path I/O concurrency config applied to the JSON and Parquet handlers. `None` fields + /// fall back to the handlers' defaults. + io_config: ReadIoConfig, +} + +/// Read-path I/O tuning for [`DefaultEngine`]'s JSON and Parquet handlers. +/// +/// Both knobs default to the handlers' built-in defaults when left unset. +#[derive(Debug, Default, Clone, Copy)] +struct ReadIoConfig { + /// Maximum number of files read concurrently (file-level readahead depth). See + /// [`DefaultEngineBuilder::with_buffer_size`]. + buffer_size: Option>, + /// Maximum number of rows per yielded batch. See [`DefaultEngineBuilder::with_batch_size`]. + batch_size: Option>, +} + +/// Represents the default [`TaskExecutor`]. The executor is created lazily to avoid unnecessary +/// instantiations. +pub struct DefaultTaskExecutor; + +impl DefaultEngineBuilder { + /// Create a new [`DefaultEngineBuilder`] instance with the default executor. + pub fn new(object_store: Arc) -> Self { + Self { + object_store, + task_executor: DefaultTaskExecutor, + io_config: ReadIoConfig::default(), + } + } + + /// Build the [`DefaultEngine`] instance. + pub fn build(self) -> DefaultEngine { + let task_executor = Arc::new(executor::tokio::TokioBackgroundExecutor::new()); + DefaultEngine::new_with_opts(self.object_store, task_executor, self.io_config) + } +} + +impl DefaultEngineBuilder { + /// Set a custom task executor for the engine. + /// + /// See [`executor::TaskExecutor`] for more details. + pub fn with_task_executor( + self, + task_executor: Arc, + ) -> DefaultEngineBuilder> { + DefaultEngineBuilder { + object_store: self.object_store, + task_executor, + io_config: self.io_config, + } + } + + /// Set the maximum number of files read concurrently by the JSON and Parquet handlers in their + /// `read_*_files` paths. This is the file-level I/O readahead depth: higher values overlap more + /// object-store requests to hide latency, at the cost of more in-flight memory. + /// + /// Defaults to the handlers' built-in value when unset. Ordering of returned data is preserved + /// regardless of this value. + pub fn with_buffer_size(mut self, buffer_size: NonZero) -> Self { + self.io_config.buffer_size = Some(buffer_size); + self + } + + /// Set the maximum number of rows per batch yielded by the JSON and Parquet handlers in their + /// `read_*_files` paths. + /// + /// Defaults to the handlers' built-in value when unset. Overall read memory usage is roughly + /// proportional to `buffer_size * batch_size`. + pub fn with_batch_size(mut self, batch_size: NonZero) -> Self { + self.io_config.batch_size = Some(batch_size); + self + } +} + +impl DefaultEngineBuilder> { + /// Build the [`DefaultEngine`] instance. + pub fn build(self) -> DefaultEngine { + DefaultEngine::new_with_opts(self.object_store, self.task_executor, self.io_config) + } +} + +impl DefaultEngine { + /// Create a [`DefaultEngineBuilder`] for constructing a [`DefaultEngine`] with custom options. + /// + /// # Parameters + /// + /// - `object_store`: The object store to use. + pub fn builder(object_store: Arc) -> DefaultEngineBuilder { + DefaultEngineBuilder::new(object_store) + } +} + +impl DefaultEngine { + fn new_with_opts( + object_store: Arc, + task_executor: Arc, + io_config: ReadIoConfig, + ) -> Self { + let raw_storage: Arc = Arc::new(ObjectStoreStorageHandler::new( + object_store.clone(), + task_executor.clone(), + )); + + let buffer_size = io_config.buffer_size.unwrap_or(DEFAULT_READ_BUFFER_SIZE); + let batch_size = io_config.batch_size.unwrap_or(DEFAULT_READ_BATCH_SIZE); + let json = DefaultJsonHandler::new(object_store.clone(), task_executor.clone()) + .with_buffer_size(buffer_size) + .with_batch_size(batch_size); + let parquet = DefaultParquetHandler::new(object_store.clone(), task_executor.clone()) + .with_buffer_size(buffer_size) + .with_batch_size(batch_size); + let raw_json: Arc = Arc::new(json); + let raw_parquet = Arc::new(parquet); + Self { + storage: Arc::new(MeteredStorageHandler::new(raw_storage)), + json: Arc::new(MeteredJsonHandler::new(raw_json)), + parquet: Arc::new(MeteredParquetHandler::new(raw_parquet.clone())), + raw_parquet, + object_store, + task_executor, + evaluation: Arc::new(ArrowEvaluationHandler {}), + } + } + + /// Enter the runtime context of the executor associated with this engine. + /// + /// # Panics + /// + /// When calling `enter` multiple times, the returned guards **must** be dropped in the reverse + /// order that they were acquired. Failure to do so will result in a panic and possible memory + /// leaks. + pub fn enter(&self) -> ::Guard<'_> { + self.task_executor.enter() + } + + pub fn get_object_store_for_url(&self, _url: &Url) -> Option> { + Some(self.object_store.clone()) + } + + /// Returns the concrete [`DefaultParquetHandler`] for callers that need the inherent + /// async `write_parquet_file` helper not exposed by the [`ParquetHandler`] trait. + /// For the metered trait surface used by reads, use [`Self::parquet_handler`]. + /// + /// TODO(#2701): lift the inherent helper onto [`DefaultEngine`] so this accessor + /// (and the `raw_parquet` field) can be removed. + pub fn default_parquet_handler(&self) -> Arc> { + self.raw_parquet.clone() + } + + /// Write `data` as a parquet file using the provided `write_context`. + /// + /// `data` must not contain partition columns. If the table materializes partition columns (e.g. + /// `materializePartitionColumns` or `icebergCompatV3`), this function automatically inserts + /// them into the data. + /// + /// The `write_context` must be created by [`Transaction::partitioned_write_context`] or + /// [`Transaction::unpartitioned_write_context`], which handle partition value validation, + /// serialization, and logical-to-physical key translation. + /// + /// [`Transaction::partitioned_write_context`]: delta_kernel::transaction::Transaction::partitioned_write_context + /// [`Transaction::unpartitioned_write_context`]: delta_kernel::transaction::Transaction::unpartitioned_write_context + pub async fn write_parquet( + &self, + data: &ArrowEngineData, + write_context: &WriteContext, + ) -> DeltaResult> { + let transform = write_context.logical_to_physical(); + let input_schema = Schema::try_from_arrow(data.record_batch().schema())?; + let output_schema = write_context.physical_schema(); + let logical_to_physical_expr = self.evaluation_handler().new_expression_evaluator( + input_schema.into(), + transform.clone(), + output_schema.clone().into(), + )?; + let physical_data = logical_to_physical_expr.evaluate(data)?; + self.raw_parquet + .write_parquet_file(physical_data, write_context) + .await + } +} + +/// Converts [`DataFileMetadata`] into Add action [`EngineData`] using the partition values and +/// table root from the provided [`WriteContext`]. +/// +/// Paths in the returned Add action metadata are stored relative to the table root. +/// +/// This is the public API for building Add action metadata from file write results. Custom +/// Arrow-based engines that write parquet files themselves (bypassing +/// [`DefaultEngine::write_parquet`]) should call this to produce the Add action metadata for +/// [`Transaction::add_files`]. +/// +/// [`DataFileMetadata`]: parquet::DataFileMetadata +/// [`Transaction::add_files`]: delta_kernel::transaction::Transaction::add_files +pub fn build_add_file_metadata( + file_metadata: parquet::DataFileMetadata, + write_context: &WriteContext, +) -> DeltaResult> { + let add_path = write_context.resolve_file_path(file_metadata.location())?; + file_metadata.as_record_batch(write_context.physical_partition_values(), &add_path) +} + +impl Engine for DefaultEngine { + fn evaluation_handler(&self) -> Arc { + self.evaluation.clone() + } + + fn storage_handler(&self) -> Arc { + self.storage.clone() + } + + fn json_handler(&self) -> Arc { + self.json.clone() + } + + fn parquet_handler(&self) -> Arc { + self.parquet.clone() + } +} + +trait UrlExt { + // Check if a given url is a presigned url and can be used + // to access the object store via simple http requests + fn is_presigned(&self) -> bool; +} + +impl UrlExt for Url { + fn is_presigned(&self) -> bool { + // We search a URL query string for these keys to see if we should consider it a presigned + // URL: + // - AWS: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html + // - Cloudflare R2: https://developers.cloudflare.com/r2/api/s3/presigned-urls/ + // - Azure Blob (SAS): https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#version-2020-12-06-and-later + // - Google Cloud Storage: https://cloud.google.com/storage/docs/authentication/signatures + // - Alibaba Cloud OSS: https://www.alibabacloud.com/help/en/oss/user-guide/upload-files-using-presigned-urls + // - Databricks presigned URLs + const PRESIGNED_KEYS: &[&str] = &[ + "X-Amz-Signature", + "sp", + "X-Goog-Credential", + "X-OSS-Credential", + "X-Databricks-Signature", + ]; + matches!(self.scheme(), "http" | "https") + && self + .query_pairs() + .any(|(k, _)| PRESIGNED_KEYS.iter().any(|p| k.eq_ignore_ascii_case(p))) + } +} + +#[cfg(test)] +mod tests { + use delta_kernel::object_store::local::LocalFileSystem; + use test_utils::engine_contract::test_arrow_engine; + + use super::*; + + #[test] + fn test_default_engine() { + let tmp = tempfile::tempdir().unwrap(); + let url = Url::from_directory_path(tmp.path()).unwrap(); + let object_store = Arc::new(LocalFileSystem::new()); + let engine = DefaultEngineBuilder::new(object_store).build(); + test_arrow_engine(&engine, &url); + } + + #[test] + fn test_default_engine_builder_new_and_build() { + let tmp = tempfile::tempdir().unwrap(); + let url = Url::from_directory_path(tmp.path()).unwrap(); + let object_store = Arc::new(LocalFileSystem::new()); + let engine = DefaultEngineBuilder::new(object_store).build(); + test_arrow_engine(&engine, &url); + } + + #[test] + fn test_default_engine_builder_with_custom_executor() { + let rt = tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build() + .unwrap(); + let tmp = tempfile::tempdir().unwrap(); + let url = Url::from_directory_path(tmp.path()).unwrap(); + let object_store = Arc::new(LocalFileSystem::new()); + let executor = Arc::new(executor::tokio::TokioMultiThreadExecutor::new( + rt.handle().clone(), + )); + let engine = DefaultEngineBuilder::new(object_store) + .with_task_executor(executor) + .build(); + test_arrow_engine(&engine, &url); + } + + #[test] + fn test_default_engine_builder_method() { + let tmp = tempfile::tempdir().unwrap(); + let url = Url::from_directory_path(tmp.path()).unwrap(); + let object_store = Arc::new(LocalFileSystem::new()); + let engine = DefaultEngine::builder(object_store).build(); + test_arrow_engine(&engine, &url); + } + + #[test] + fn test_default_engine_builder_all_options() { + let tmp = tempfile::tempdir().unwrap(); + let url = Url::from_directory_path(tmp.path()).unwrap(); + let object_store = Arc::new(LocalFileSystem::new()); + let executor = Arc::new(executor::tokio::TokioBackgroundExecutor::new()); + let engine = DefaultEngineBuilder::new(object_store) + .with_task_executor(executor) + .with_buffer_size(NonZero::new(4).unwrap()) + .with_batch_size(NonZero::new(8).unwrap()) + .build(); + test_arrow_engine(&engine, &url); + } + + #[test] + fn test_pre_signed_url() { + let url = Url::parse("https://example.com?X-Amz-Signature=foo").unwrap(); + assert!(url.is_presigned()); + + let url = Url::parse("https://example.com?sp=foo").unwrap(); + assert!(url.is_presigned()); + + let url = Url::parse("https://example.com?X-Goog-Credential=foo").unwrap(); + assert!(url.is_presigned()); + + let url = Url::parse("https://example.com?X-OSS-Credential=foo").unwrap(); + assert!(url.is_presigned()); + + let url = + Url::parse("https://example.com?X-Databricks-TTL=3599545&X-Databricks-Signature=bar") + .unwrap(); + assert!(url.is_presigned()); + + // assert that query keys are case insensitive + let url = Url::parse("https://example.com?x-gooG-credenTIAL=foo").unwrap(); + assert!(url.is_presigned()); + + let url = Url::parse("https://example.com?x-oss-CREDENTIAL=foo").unwrap(); + assert!(url.is_presigned()); + + let url = Url::parse("https://example.com").unwrap(); + assert!(!url.is_presigned()); + } + + // `block_on_or_cancelled` must return `None` (cancel won the race) when the future never + // completes on its own and the token fires while it is pending -- the `Either::Right` arm + // that the pre-cancelled fast-path never exercises. + #[test] + fn block_on_or_cancelled_none_when_cancel_wins_race() { + let executor = Arc::new(executor::tokio::TokioBackgroundExecutor::new()); + let token = Arc::new(test_utils::TestCancellationToken::default()); + let ct: CancellationTokenRef = token.clone(); + + // Fire cancellation shortly after the block begins; the only way out is the cancel arm. + let firing = token.clone(); + std::thread::spawn(move || { + std::thread::sleep(std::time::Duration::from_millis(50)); + firing.cancel(); + }); + + let out: Option = block_on_or_cancelled(&executor, ct, std::future::pending::()); + assert!(out.is_none(), "cancel must win the select and yield None"); + } + + // A stream that yields two items then blocks forever is cancelled mid-stream: the iterator + // yields the ready items, then exactly one `Err(Cancelled)`, then fuses. + #[test] + fn cancellable_stream_iterator_cancels_mid_stream_then_fuses() { + use futures::stream; + + let executor = Arc::new(executor::tokio::TokioBackgroundExecutor::new()); + let token = Arc::new(test_utils::TestCancellationToken::default()); + let ct: CancellationTokenRef = token.clone(); + + // 0, 1, then a pending tail that never resolves on its own. + let make_stream = async move { + let head = stream::iter(vec![Ok(0i32), Ok(1i32)]); + let tail = stream::once(std::future::pending::>()); + Ok(head.chain(tail).boxed()) + }; + let mut iter = stream_future_to_cancellable_iter(executor, make_stream, Some(ct)).unwrap(); + + assert!(matches!(iter.next(), Some(Ok(0)))); + assert!(matches!(iter.next(), Some(Ok(1)))); + + let firing = token.clone(); + std::thread::spawn(move || { + std::thread::sleep(std::time::Duration::from_millis(50)); + firing.cancel(); + }); + + assert!(matches!(iter.next(), Some(Err(Error::Cancelled)))); + assert!( + iter.next().is_none(), + "iterator must fuse after cancellation" + ); + } +} diff --git a/kernel/src/engine/default/parquet.rs b/default-engine/src/parquet.rs similarity index 63% rename from kernel/src/engine/default/parquet.rs rename to default-engine/src/parquet.rs index da427dda42..c200af7565 100644 --- a/kernel/src/engine/default/parquet.rs +++ b/default-engine/src/parquet.rs @@ -1,66 +1,56 @@ //! Default Parquet handler implementation use std::collections::HashMap; +use std::num::NonZero; use std::ops::Range; use std::sync::Arc; -use delta_kernel_derive::internal_api; - -use crate::arrow::array::builder::{MapBuilder, MapFieldNames, StringBuilder}; -use crate::arrow::array::{Array, Int64Array, RecordBatch, StringArray, StructArray}; -use crate::arrow::datatypes::{DataType, Field, Schema}; -use crate::object_store::path::Path; -use crate::object_store::{DynObjectStore, ObjectStore}; -use crate::parquet::arrow::arrow_reader::{ - ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder, -}; -use crate::parquet::arrow::arrow_writer::{ArrowWriter, ArrowWriterOptions}; -use crate::parquet::arrow::async_reader::{ParquetObjectReader, ParquetRecordBatchStreamBuilder}; -use crate::parquet::arrow::async_writer::AsyncArrowWriter; -use crate::parquet::arrow::async_writer::ParquetObjectWriter; -use futures::stream::{self, BoxStream}; -use futures::{StreamExt, TryStreamExt}; -use uuid::Uuid; - -use super::file_stream::{FileOpenFuture, FileOpener, FileStream}; -use super::stats::collect_stats; -use super::UrlExt; -use crate::engine::arrow_conversion::{TryFromArrow as _, TryIntoArrow as _}; -use crate::engine::arrow_data::ArrowEngineData; -use crate::engine::arrow_utils::{ +use delta_kernel::arrow::array::builder::{MapBuilder, MapFieldNames, StringBuilder}; +use delta_kernel::arrow::array::{Array, Int64Array, RecordBatch, StringArray, StructArray}; +use delta_kernel::arrow::datatypes::{DataType, Field, Schema}; +use delta_kernel::engine::arrow_conversion::{TryFromArrow as _, TryIntoArrow as _}; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::engine::arrow_utils::{ fixup_parquet_read, generate_mask, get_requested_indices, ordering_needs_row_indexes, RowIndexBuilder, }; -use crate::engine::default::executor::TaskExecutor; -use crate::engine::parquet_row_group_skipping::ParquetRowGroupSkipping; -use crate::expressions::ColumnName; -use crate::schema::{SchemaRef, StructType}; -use crate::{ - DeltaResult, EngineData, Error, FileDataReadResultIterator, FileMeta, ParquetFooter, - ParquetHandler, PredicateRef, +use delta_kernel::engine::parquet_row_group_skipping::ParquetRowGroupSkipping; +use delta_kernel::engine::{reader_options, writer_options}; +use delta_kernel::expressions::ColumnName; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::{DynObjectStore, ObjectStoreExt as _}; +use delta_kernel::parquet::arrow::arrow_reader::{ + ArrowReaderMetadata, ParquetRecordBatchReaderBuilder, }; +use delta_kernel::parquet::arrow::arrow_writer::ArrowWriter; +use delta_kernel::parquet::arrow::async_reader::{ + ParquetObjectReader, ParquetRecordBatchStreamBuilder, +}; +use delta_kernel::parquet::arrow::async_writer::{AsyncArrowWriter, ParquetObjectWriter}; +use delta_kernel::schema::{SchemaRef, StructType}; +use delta_kernel::transaction::WriteContext; +use delta_kernel::{ + CancellationTokenRef, DeltaResult, DeltaResultIteratorStatic, EngineData, Error, + FileDataReadResultIterator, FileMeta, ParquetFooter, ParquetHandler, PredicateRef, +}; +use futures::stream::{self, BoxStream}; +use futures::{StreamExt, TryStreamExt}; +use uuid::Uuid; -/// Returns the standard [`ArrowReaderOptions`] for all kernel parquet reads. -/// -/// Skipping the embedded Arrow IPC schema avoids dependence on Arrow-specific metadata and -/// ensures that type resolution is driven by the kernel schema rather than the file's schema. -pub(in crate::engine) fn reader_options() -> ArrowReaderOptions { - ArrowReaderOptions::new().with_skip_arrow_metadata(true) -} - -/// Returns the standard [`ArrowWriterOptions`] for all kernel parquet writes. -/// -/// Omitting the Arrow IPC schema from the file metadata keeps Delta files interoperable with -/// non-Arrow readers and avoids encoding Arrow-specific type information. -pub(in crate::engine) fn writer_options() -> ArrowWriterOptions { - ArrowWriterOptions::new().with_skip_arrow_metadata(true) -} +use crate::executor::TaskExecutor; +use crate::file_stream::{FileOpenFuture, FileOpener, FileStream}; +use crate::stats::collect_stats; +use crate::UrlExt; #[derive(Debug)] pub struct DefaultParquetHandler { store: Arc, task_executor: Arc, - readahead: usize, + /// The maximum number of files to read concurrently in [`Self::read_parquet_files()`]. This is + /// the number of futures buffered by `buffered`, i.e. the file-level I/O readahead depth. + buffer_size: NonZero, + /// The maximum number of rows per RecordBatch yielded by the read stream. + batch_size: NonZero, } /// Metadata of a data file (typically a parquet file). @@ -76,27 +66,28 @@ impl DataFileMetadata { Self { file_meta, stats } } - /// Convert DataFileMetadata into a record batch which matches the schema returned by - /// [`add_files_schema`]. + /// Returns the absolute URL of the written file. + pub fn location(&self) -> &url::Url { + &self.file_meta.location + } + + /// Converts this `DataFileMetadata` into an [`EngineData`] record batch matching the schema + /// returned by [`Transaction::add_files_schema`]. + /// + /// The `partition_values` map uses physical column names as keys and protocol-serialized + /// strings as values. `None` represents a null partition value. The serialization layer + /// converts nulls and empty strings to `None` before reaching this method, so `Some("")` + /// is not expected in normal usage. + /// + /// The `log_path` is the path string written to the Delta log's `add.path` field. /// - /// [`add_files_schema`]: crate::transaction::Transaction::add_files_schema - #[internal_api] + /// [`Transaction::add_files_schema`]: delta_kernel::transaction::Transaction::add_files_schema pub(crate) fn as_record_batch( &self, - partition_values: &HashMap, + partition_values: &HashMap>, + log_path: &str, ) -> DeltaResult> { - let DataFileMetadata { - file_meta: - FileMeta { - location, - last_modified, - size, - }, - stats, - .. - } = self; - // create the record batch of the write metadata - let path = Arc::new(StringArray::from(vec![location.to_string()])); + let path = Arc::new(StringArray::from(vec![log_path])); let key_builder = StringBuilder::new(); let val_builder = StringBuilder::new(); let names = MapFieldNames { @@ -107,25 +98,28 @@ impl DataFileMetadata { let mut builder = MapBuilder::new(Some(names), key_builder, val_builder); for (k, v) in partition_values { builder.keys().append_value(k); - if v.is_empty() { - // convert empty string to null as per the Delta Spec - builder.values().append_null(); - } else { - builder.values().append_value(v); + match v.as_deref() { + // The serialization layer already converts empty strings to None, so + // Some("") should not occur. The empty check is purely defensive. + Some(val) if !val.is_empty() => builder.values().append_value(val), + _ => builder.values().append_null(), } } builder.append(true)?; let partitions = Arc::new(builder.finish()); // this means max size we can write is i64::MAX (~8EB) - let size: i64 = (*size) + let size: i64 = self + .file_meta + .size .try_into() .map_err(|_| Error::generic("Failed to convert parquet metadata 'size' to i64"))?; let size = Arc::new(Int64Array::from(vec![size])); - let modification_time = Arc::new(Int64Array::from(vec![*last_modified])); + let modification_time = Arc::new(Int64Array::from(vec![self.file_meta.last_modified])); - let stats_array = Arc::new(stats.clone()); + let stats_array = Arc::new(self.stats.clone()); - // Build schema dynamically based on stats (stats schema varies based on collected statistics) + // Build schema dynamically based on stats (stats schema varies based on collected + // statistics) let key_value_struct = DataType::Struct( vec![ Field::new("key", DataType::Utf8, false), @@ -160,15 +154,34 @@ impl DefaultParquetHandler { Self { store, task_executor, - readahead: 10, + buffer_size: super::DEFAULT_READ_BUFFER_SIZE, + batch_size: super::DEFAULT_READ_BATCH_SIZE, } } - /// Max number of batches to read ahead while executing [Self::read_parquet_files()]. + /// Set the maximum number of files to read concurrently in [Self::read_parquet_files()]. + /// + /// Defaults to `super::DEFAULT_READ_BUFFER_SIZE`. /// - /// Defaults to 10. - pub fn with_readahead(mut self, readahead: usize) -> Self { - self.readahead = readahead; + /// This setting applies only to object-store reads. When every file in the batch uses a + /// presigned URL (`https://...`), reads bypass the object store and this value has no effect; + /// use [`Self::with_batch_size`] to tune RecordBatch chunking in that path. + /// + /// Memory constraints can be imposed by constraining the buffer size and batch size. Note that + /// overall memory usage is proportional to the product of these two values. + /// 1. Batch size governs the size of RecordBatches yielded in each iteration of the stream. + /// 2. Buffer size governs the number of concurrent file reads (which equals the size of the + /// readahead buffer). + pub fn with_buffer_size(mut self, buffer_size: NonZero) -> Self { + self.buffer_size = buffer_size; + self + } + + /// Set the maximum number of rows per RecordBatch yielded by [Self::read_parquet_files()]. + /// + /// Defaults to `super::DEFAULT_READ_BATCH_SIZE` rows. + pub fn with_batch_size(mut self, batch_size: NonZero) -> Self { + self.batch_size = batch_size; self } @@ -182,12 +195,13 @@ impl DefaultParquetHandler { path: &url::Url, data: Box, stats_columns: &[ColumnName], + physical_schema: &StructType, ) -> DeltaResult { let batch: Box<_> = ArrowEngineData::try_from_engine_data(data)?; let record_batch = batch.record_batch(); // Collect statistics before writing (includes numRecords) - let stats = collect_stats(record_batch, stats_columns)?; + let stats = collect_stats(record_batch, stats_columns, physical_schema)?; let mut buffer = vec![]; let mut writer = ArrowWriter::try_new_with_options( @@ -228,25 +242,28 @@ impl DefaultParquetHandler { Ok(DataFileMetadata::new(file_meta, stats)) } - /// Write `data` to `{path}/.parquet` as parquet using ArrowWriter and return the parquet - /// metadata as an EngineData batch which matches the [add file metadata] schema (where `` - /// is a generated UUIDv4). + /// Write `data` to a new parquet file under the [`WriteContext::write_dir`] and return + /// Add action metadata ready for [`Transaction::add_files`]. /// - /// Note that the schema does not contain the dataChange column. In order to set `data_change` flag, - /// use [`crate::transaction::Transaction::with_data_change`]. + /// Note that the schema does not contain the dataChange column. In order to set `data_change` + /// flag, use [`delta_kernel::transaction::Transaction::with_data_change`]. /// - /// [add file metadata]: crate::transaction::Transaction::add_files_schema + /// [`WriteContext::write_dir`]: delta_kernel::transaction::WriteContext::write_dir + /// [`Transaction::add_files`]: delta_kernel::transaction::Transaction::add_files pub async fn write_parquet_file( &self, - path: &url::Url, data: Box, - partition_values: HashMap, - stats_columns: Option<&[ColumnName]>, + write_context: &WriteContext, ) -> DeltaResult> { - let parquet_metadata = self - .write_parquet(path, data, stats_columns.unwrap_or(&[])) + let file_metadata = self + .write_parquet( + &write_context.write_dir(), + data, + write_context.stats_columns(), + write_context.physical_schema().as_ref(), + ) .await?; - parquet_metadata.as_record_batch(&partition_values) + super::build_add_file_metadata(file_metadata, write_context) } } @@ -256,6 +273,8 @@ async fn read_parquet_files_impl( files: Vec, physical_schema: SchemaRef, predicate: Option, + buffer_size: usize, + batch_size: usize, ) -> DeltaResult>>> { if files.is_empty() { return Ok(Box::pin(stream::empty())); @@ -273,7 +292,7 @@ async fn read_parquet_files_impl( // SAFETY: we did is_empty check above, this is ok. if files[0].location.is_presigned() { let file_opener = Box::new(PresignedUrlOpener::new( - 1024, + batch_size, physical_schema.clone(), predicate, )); @@ -288,21 +307,11 @@ async fn read_parquet_files_impl( let store = store.clone(); let schema = physical_schema.clone(); let predicate = predicate.clone(); - async move { - open_parquet_file( - store, - schema, - predicate, - None, - super::DEFAULT_BATCH_SIZE, - file, - ) - .await - } + async move { open_parquet_file(store, schema, predicate, None, batch_size, file).await } }); // create a stream from that iterator which buffers up to `buffer_size` futures at a time let result_stream = stream::iter(file_futures) - .buffered(super::DEFAULT_BUFFER_SIZE) + .buffered(buffer_size) .try_flatten() .map_ok(|record_batch| -> Box { Box::new(ArrowEngineData::new(record_batch)) @@ -317,14 +326,30 @@ impl ParquetHandler for DefaultParquetHandler { files: &[FileMeta], physical_schema: SchemaRef, predicate: Option, + ) -> DeltaResult { + self.read_parquet_files_with_cancellation(files, physical_schema, predicate, None) + } + + fn read_parquet_files_with_cancellation( + &self, + files: &[FileMeta], + physical_schema: SchemaRef, + predicate: Option, + cancellation_token: Option, ) -> DeltaResult { let future = read_parquet_files_impl( self.store.clone(), files.to_vec(), physical_schema, predicate, + self.buffer_size.get(), + self.batch_size.get(), ); - super::stream_future_to_iter(self.task_executor.clone(), future) + super::stream_future_to_cancellable_iter( + self.task_executor.clone(), + future, + cancellation_token, + ) } /// Writes engine data to a Parquet file at the specified location. @@ -334,8 +359,8 @@ impl ParquetHandler for DefaultParquetHandler { /// /// # Parameters /// - /// - `location` - The full URL path where the Parquet file should be written - /// (e.g., `s3://bucket/path/file.parquet`, `file:///path/to/file.parquet`). + /// - `location` - The full URL path where the Parquet file should be written (e.g., `s3://bucket/path/file.parquet`, + /// `file:///path/to/file.parquet`). /// - `data` - An iterator of engine data to be written to the Parquet file. /// /// # Returns @@ -344,7 +369,7 @@ impl ParquetHandler for DefaultParquetHandler { fn write_parquet_file( &self, location: url::Url, - mut data: Box>> + Send>, + mut data: DeltaResultIteratorStatic>, ) -> DeltaResult<()> { let store = self.store.clone(); @@ -381,11 +406,19 @@ impl ParquetHandler for DefaultParquetHandler { } fn read_parquet_footer(&self, file: &FileMeta) -> DeltaResult { + self.read_parquet_footer_with_cancellation(file, None) + } + + fn read_parquet_footer_with_cancellation( + &self, + file: &FileMeta, + cancellation_token: Option, + ) -> DeltaResult { let store = self.store.clone(); let location = file.location.clone(); let file_size = file.size; - self.task_executor.block_on(async move { + let footer_future = async move { let metadata = if location.is_presigned() { let client = reqwest::Client::new(); let response = @@ -403,11 +436,16 @@ impl ParquetHandler for DefaultParquetHandler { ArrowReaderMetadata::load_async(&mut reader, reader_options()).await? }; - let schema = StructType::try_from_arrow(metadata.schema().as_ref()) - .map(Arc::new) - .map_err(Error::Arrow)?; + let schema = Arc::new(StructType::try_from_arrow(metadata.schema().as_ref())?); Ok(ParquetFooter { schema }) - }) + }; + + // Race the footer read against cancellation so a cancelled request stops promptly. + match cancellation_token { + Some(token) => super::block_on_or_cancelled(&self.task_executor, token, footer_future) + .unwrap_or(Err(Error::Cancelled)), + None => self.task_executor.block_on(footer_future), + } } } @@ -424,7 +462,7 @@ async fn open_parquet_file( let path = Path::from_url_path(file_meta.location.path())?; let mut reader = { - use crate::object_store::ObjectStoreScheme; + use delta_kernel::object_store::ObjectStoreScheme; // HACK: unfortunately, `ParquetObjectReader` under the hood does a suffix range // request which isn't supported by Azure. For now we just detect if the URL is // pointing to azure and if so, do a HEAD request so we can pass in file size to the @@ -449,12 +487,10 @@ async fn open_parquet_file( } }; - let reader_options = reader_options(); - let metadata = ArrowReaderMetadata::load_async(&mut reader, reader_options.clone()).await?; + let metadata = ArrowReaderMetadata::load_async(&mut reader, reader_options()).await?; let parquet_schema = metadata.schema(); let (indices, requested_ordering) = get_requested_indices(&table_schema, parquet_schema)?; - let mut builder = - ParquetRecordBatchStreamBuilder::new_with_options(reader, reader_options).await?; + let mut builder = ParquetRecordBatchStreamBuilder::new_with_metadata(reader, metadata.clone()); if let Some(mask) = generate_mask( &table_schema, parquet_schema, @@ -479,14 +515,13 @@ async fn open_parquet_file( let mut row_indexes = row_indexes.map(|rb| rb.build()).transpose()?; let stream = builder.with_batch_size(batch_size).build()?; - let arrow_schema: Arc = Arc::new(table_schema.as_ref().try_into_arrow()?); let stream = stream.map(move |rbr| { fixup_parquet_read( rbr?, &requested_ordering, row_indexes.as_mut(), Some(&file_location), - Some(&arrow_schema), + Some(&table_schema), ) .map(Into::into) }); @@ -530,14 +565,13 @@ impl FileOpener for PresignedUrlOpener { Ok(Box::pin(async move { // fetch the file from the interweb let reader = client.get(&file_location).send().await?.bytes().await?; - let reader_options = reader_options(); - let metadata = ArrowReaderMetadata::load(&reader, reader_options.clone())?; + let metadata = ArrowReaderMetadata::load(&reader, reader_options())?; let parquet_schema = metadata.schema(); let (indices, requested_ordering) = get_requested_indices(&table_schema, parquet_schema)?; let mut builder = - ParquetRecordBatchReaderBuilder::try_new_with_options(reader, reader_options)?; + ParquetRecordBatchReaderBuilder::new_with_metadata(reader, metadata.clone()); if let Some(mask) = generate_mask( &table_schema, parquet_schema, @@ -562,7 +596,6 @@ impl FileOpener for PresignedUrlOpener { let reader = builder.with_batch_size(batch_size).build()?; let mut row_indexes = row_indexes.map(|rb| rb.build()).transpose()?; - let arrow_schema: Arc = Arc::new(table_schema.as_ref().try_into_arrow()?); let stream = futures::stream::iter(reader); let stream = stream.map(move |rbr| { fixup_parquet_read( @@ -570,7 +603,7 @@ impl FileOpener for PresignedUrlOpener { &requested_ordering, row_indexes.as_mut(), Some(&file_location), - Some(&arrow_schema), + Some(&table_schema), ) .map(Into::into) }); @@ -584,36 +617,131 @@ mod tests { use std::collections::HashMap; use std::path::PathBuf; use std::slice; + use std::sync::atomic::{AtomicUsize, Ordering}; - use crate::arrow::array::{ + use bytes::Bytes; + use delta_kernel::actions::{NUM_RECORDS, TIGHT_BOUNDS}; + use delta_kernel::arrow::array::{ Array, BinaryArray, BooleanArray, Date32Array, Decimal128Array, Float32Array, Float64Array, Int16Array, Int32Array, Int64Array, Int8Array, RecordBatch, StringArray, - TimestampMicrosecondArray, + TimestampMicrosecondArray, TimestampMillisecondArray, + }; + use delta_kernel::arrow::datatypes::{ + DataType as ArrowDataType, Field, Schema as ArrowSchema, TimeUnit, + }; + use delta_kernel::engine::arrow_conversion::TryIntoKernel as _; + use delta_kernel::engine::arrow_data::ArrowEngineData; + use delta_kernel::object_store::local::LocalFileSystem; + use delta_kernel::object_store::memory::InMemory; + use delta_kernel::object_store::{ + CopyOptions, GetOptions, GetResult, ListResult, MultipartUpload, ObjectMeta, ObjectStore, + PutMultipartOptions, PutOptions, PutPayload, PutResult, Result, + }; + use delta_kernel::parquet::arrow::{ARROW_SCHEMA_META_KEY, PARQUET_FIELD_ID_META_KEY}; + use delta_kernel::schema::{ + ColumnMetadataKey, DataType, MetadataValue, StructField, StructType, + }; + use delta_kernel::EngineData; + use delta_kernel_default_engine_test_utils::{ + assert_result_error_with_message, current_time_ms, + try_into_record_batch as into_record_batch, }; - use crate::arrow::datatypes::{DataType as ArrowDataType, Field, Schema as ArrowSchema}; - use crate::engine::arrow_conversion::TryIntoKernel as _; - use crate::engine::arrow_data::ArrowEngineData; - use crate::engine::default::executor::tokio::TokioBackgroundExecutor; - use crate::engine::default::DEFAULT_BATCH_SIZE; - use crate::object_store::{local::LocalFileSystem, memory::InMemory, ObjectStore}; - use crate::parquet::arrow::{ARROW_SCHEMA_META_KEY, PARQUET_FIELD_ID_META_KEY}; - use crate::schema::ColumnMetadataKey; - use crate::EngineData; - use itertools::Itertools; + use test_utils::engine_contract::{ + test_parquet_handler_footer_errors_on_missing_file, + test_parquet_handler_footer_preserves_field_ids, + test_parquet_handler_reads_file_with_arrow_schema, test_parquet_handler_reads_footer, + test_parquet_handler_write_always_overwrites, + test_parquet_handler_write_omits_arrow_schema, + }; use url::Url; - use crate::utils::current_time_ms; - use crate::utils::test_utils::assert_result_error_with_message; - use super::*; + use crate::executor::tokio::TokioBackgroundExecutor; + use crate::DEFAULT_BATCH_SIZE; - fn into_record_batch( - engine_data: DeltaResult>, - ) -> DeltaResult { - engine_data - .and_then(ArrowEngineData::try_from_engine_data) - .map(Into::into) + fn long_schema(name: &str) -> StructType { + StructType::new_unchecked([StructField::nullable(name, DataType::LONG)]) + } + + /// Test `ObjectStore` that counts footer fetches. `get_opts` (footer range GETs) is counted; + /// column-chunk data goes through `get_ranges` and is not counted, so the count isolates + /// footer reads. + #[derive(Debug)] + struct GetOptsCountingStore { + inner: T, + get_opts_count: AtomicUsize, + } + + impl GetOptsCountingStore { + fn new(inner: T) -> Self { + Self { + inner, + get_opts_count: AtomicUsize::new(0), + } + } + + fn get_opts_count(&self) -> usize { + self.get_opts_count.load(Ordering::SeqCst) + } + } + + impl std::fmt::Display for GetOptsCountingStore { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "GetOptsCountingStore({})", self.get_opts_count()) + } + } + + #[async_trait::async_trait] + impl delta_kernel::object_store::ObjectStore for GetOptsCountingStore { + // ===== The method we instrument: count footer-fetch GETs ===== + async fn get_opts(&self, location: &Path, options: GetOptions) -> Result { + self.get_opts_count.fetch_add(1, Ordering::SeqCst); + self.inner.get_opts(location, options).await + } + + // ===== Everything else: behavior unchanged, delegate to inner ===== + // Overridden (not inherited) so column-chunk data reads delegate straight to inner and + // stay off the get_opts counter. + async fn get_ranges(&self, location: &Path, ranges: &[Range]) -> Result> { + self.inner.get_ranges(location, ranges).await + } + + async fn put_opts( + &self, + location: &Path, + payload: PutPayload, + opts: PutOptions, + ) -> Result { + self.inner.put_opts(location, payload, opts).await + } + + async fn put_multipart_opts( + &self, + location: &Path, + opts: PutMultipartOptions, + ) -> Result> { + self.inner.put_multipart_opts(location, opts).await + } + + fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, Result> { + self.inner.list(prefix) + } + + async fn list_with_delimiter(&self, prefix: Option<&Path>) -> Result { + self.inner.list_with_delimiter(prefix).await + } + + fn delete_stream( + &self, + locations: BoxStream<'static, Result>, + ) -> BoxStream<'static, Result> { + self.inner.delete_stream(locations) + } + + async fn copy_opts(&self, from: &Path, to: &Path, options: CopyOptions) -> Result<()> { + self.inner.copy_opts(from, to, options).await + } } async fn read_all_rows_helper(file_meta: FileMeta) -> DeltaResult> { @@ -643,7 +771,7 @@ mod tests { #[tokio::test] async fn test_open_parquet_file_with_size() { let path = std::fs::canonicalize(PathBuf::from( - "./tests/data/table-with-dv-small/part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet" + "../kernel/tests/data/table-with-dv-small/part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet" )).unwrap(); let file_size = std::fs::metadata(&path).unwrap().len(); let url = Url::from_file_path(path).unwrap(); @@ -661,7 +789,7 @@ mod tests { #[tokio::test] async fn test_open_parquet_file_without_size() { let path = std::fs::canonicalize(PathBuf::from( - "./tests/data/table-with-dv-small/part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet" + "../kernel/tests/data/table-with-dv-small/part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet" )).unwrap(); let url = Url::from_file_path(path).unwrap(); let file_meta = FileMeta { @@ -675,12 +803,63 @@ mod tests { assert_eq!(data[0].num_rows(), 10); } + #[tokio::test] + async fn test_open_parquet_file_fetches_footer_once() { + let path = std::fs::canonicalize(PathBuf::from( + "../kernel/tests/data/table-with-dv-small/part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet" + )).unwrap(); + let file_size = std::fs::metadata(&path).unwrap().len(); + let url = Url::from_file_path(&path).unwrap(); + let location = Path::from_url_path(url.path()).unwrap(); + + // Baseline: how many `get_opts` calls a single footer load makes (one or more range GETs, + // depending on parquet version). The reader builder must not exceed this. + let baseline_store = Arc::new(GetOptsCountingStore::new(LocalFileSystem::new())); + let mut reader = + ParquetObjectReader::new(baseline_store.clone(), location).with_file_size(file_size); + let metadata = ArrowReaderMetadata::load_async(&mut reader, reader_options()) + .await + .unwrap(); + let footer_load_gets = baseline_store.get_opts_count(); + assert!( + footer_load_gets > 0, + "footer load should issue at least one GET" + ); + let physical_schema = Arc::new(metadata.schema().clone().try_into_kernel().unwrap()); + + // `open_parquet_file` must reuse the loaded footer rather than re-fetch it when building + // the reader, so its footer GETs equal a single load, not double. + let counting_store = Arc::new(GetOptsCountingStore::new(LocalFileSystem::new())); + let file_meta = FileMeta { + location: url, + last_modified: 0, + size: file_size, + }; + let stream = open_parquet_file( + counting_store.clone(), + physical_schema, + None, + None, + DEFAULT_BATCH_SIZE, + file_meta, + ) + .await + .unwrap(); + let _batches: Vec = stream.try_collect().await.unwrap(); + + assert_eq!( + counting_store.get_opts_count(), + footer_load_gets, + "footer should be fetched once, not re-fetched when constructing the reader" + ); + } + #[tokio::test] async fn test_read_parquet_files() { let store = Arc::new(LocalFileSystem::new()); let path = std::fs::canonicalize(PathBuf::from( - "./tests/data/table-with-dv-small/part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet" + "../kernel/tests/data/table-with-dv-small/part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet" )).unwrap(); let url = url::Url::from_file_path(path).unwrap(); let location = Path::from_url_path(url.path()).unwrap(); @@ -715,8 +894,100 @@ mod tests { assert_eq!(data[0].num_rows(), 10); } + // A Parquet file can physically store TIMESTAMP(MILLIS) (for example, checkpoint stats + // written by a non-kernel writer). End to end, the default engine must (a) convert the + // millisecond footer schema to the kernel's microsecond TIMESTAMP / TIMESTAMP_NTZ, and + // (b) rescale the values to microseconds (x1000) when reading them into that schema. + #[tokio::test] + async fn test_read_millisecond_timestamps() { + let temp_dir = tempfile::tempdir().unwrap(); + let file_path = temp_dir.path().join("ms_timestamps.parquet"); + + // 1700000000123 ms since epoch -> 1_700_000_000_123_000 us. + let ms: i64 = 1_700_000_000_123; + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new( + "ts_utc", + ArrowDataType::Timestamp(TimeUnit::Millisecond, Some("UTC".into())), + true, + ), + Field::new( + "ts_ntz", + ArrowDataType::Timestamp(TimeUnit::Millisecond, None), + true, + ), + ])); + let batch = RecordBatch::try_new( + arrow_schema.clone(), + vec![ + Arc::new(TimestampMillisecondArray::from(vec![ms]).with_timezone("UTC")) + as Arc, + Arc::new(TimestampMillisecondArray::from(vec![ms])) as Arc, + ], + ) + .unwrap(); + + let mut writer = ArrowWriter::try_new( + std::fs::File::create(&file_path).unwrap(), + arrow_schema, + None, + ) + .unwrap(); + writer.write(&batch).unwrap(); + writer.close().unwrap(); + + let file_size = std::fs::metadata(&file_path).unwrap().len(); + let file_meta = FileMeta { + location: Url::from_file_path(&file_path).unwrap(), + last_modified: 0, + size: file_size, + }; + + let handler = DefaultParquetHandler::new( + Arc::new(LocalFileSystem::new()), + Arc::new(TokioBackgroundExecutor::new()), + ); + + // (a) Footer schema: millisecond timestamps convert to the kernel's microsecond types. + let footer = handler.read_parquet_footer(&file_meta).unwrap(); + assert_eq!( + footer.schema.field("ts_utc").unwrap().data_type(), + &DataType::TIMESTAMP + ); + assert_eq!( + footer.schema.field("ts_ntz").unwrap().data_type(), + &DataType::TIMESTAMP_NTZ + ); + + // (b) Data path: values are rescaled ms -> us when read into the microsecond schema. + let data: Vec = handler + .read_parquet_files(slice::from_ref(&file_meta), footer.schema.clone(), None) + .unwrap() + .map(into_record_batch) + .try_collect() + .unwrap(); + + assert_eq!(data.len(), 1); + assert_eq!(data[0].num_rows(), 1); + let expected_us = ms * 1_000; + for col_idx in 0..2 { + let col = data[0] + .column(col_idx) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!( + col.value(0), + expected_us, + "column {col_idx} should be rescaled to us" + ); + } + } + #[rstest::rstest] - fn test_as_record_batch(#[values(true, false)] test_empty_str: bool) { + fn test_as_record_batch( + #[values(None, Some("a".to_string()))] partition_value: Option, + ) { let location = Url::parse("file:///test_url").unwrap(); let size = 1_000_000; let last_modified = 10000000000; @@ -724,8 +995,8 @@ mod tests { let file_metadata = FileMeta::new(location.clone(), last_modified, size); let stats = StructArray::try_new( vec![ - Field::new("numRecords", ArrowDataType::Int64, true), - Field::new("tightBounds", ArrowDataType::Boolean, true), + Field::new(NUM_RECORDS, ArrowDataType::Int64, true), + Field::new(TIGHT_BOUNDS, ArrowDataType::Boolean, true), ] .into(), vec![ @@ -736,14 +1007,9 @@ mod tests { ) .unwrap(); let data_file_metadata = DataFileMetadata::new(file_metadata, stats.clone()); - let partition_value = if test_empty_str { - "".to_string() - } else { - "a".to_string() - }; - let partition_values = HashMap::from([("partition1".to_string(), partition_value)]); + let partition_values = HashMap::from([("partition1".to_string(), partition_value.clone())]); let actual = data_file_metadata - .as_record_batch(&partition_values) + .as_record_batch(&partition_values, "test_url") .unwrap(); let actual = ArrowEngineData::try_from_engine_data(actual).unwrap(); @@ -758,17 +1024,16 @@ mod tests { ); partition_values_builder.keys().append_value("partition1"); - if test_empty_str { - partition_values_builder.values().append_null(); // empty string should go to null - } else { - partition_values_builder.values().append_value("a"); + match &partition_value { + None => partition_values_builder.values().append_null(), + Some(v) => partition_values_builder.values().append_value(v), } partition_values_builder.append(true).unwrap(); let partition_values = partition_values_builder.finish(); // Build expected schema dynamically based on stats let stats_field = Field::new("stats", stats.data_type().clone(), true); - let schema = Arc::new(crate::arrow::datatypes::Schema::new(vec![ + let schema = Arc::new(delta_kernel::arrow::datatypes::Schema::new(vec![ Field::new("path", ArrowDataType::Utf8, false), Field::new( "partitionValues", @@ -796,7 +1061,7 @@ mod tests { let expected = RecordBatch::try_new( schema, vec![ - Arc::new(StringArray::from(vec![location.to_string()])), + Arc::new(StringArray::from(vec!["test_url"])), Arc::new(partition_values), Arc::new(Int64Array::from(vec![size as i64])), Arc::new(Int64Array::from(vec![last_modified])), @@ -821,9 +1086,15 @@ mod tests { )]) .unwrap(), )); + let physical_schema = long_schema("a"); let write_metadata = parquet_handler - .write_parquet(&Url::parse("memory:///data/").unwrap(), data, &[]) + .write_parquet( + &Url::parse("memory:///data/").unwrap(), + data, + &[], + &physical_schema, + ) .await .unwrap(); @@ -855,7 +1126,7 @@ mod tests { // Check numRecords from stats let num_records = stats - .column_by_name("numRecords") + .column_by_name(NUM_RECORDS) .unwrap() .as_any() .downcast_ref::() @@ -900,10 +1171,16 @@ mod tests { )]) .unwrap(), )); + let physical_schema = long_schema("a"); assert_result_error_with_message( parquet_handler - .write_parquet(&Url::parse("memory:///data").unwrap(), data, &[]) + .write_parquet( + &Url::parse("memory:///data").unwrap(), + data, + &[], + &physical_schema, + ) .await, "Generic delta kernel error: Path must end with a trailing slash: memory:///data", ); @@ -932,7 +1209,7 @@ mod tests { )); // Create iterator with single batch - let data_iter: Box>> + Send> = + let data_iter: DeltaResultIteratorStatic> = Box::new(std::iter::once(Ok(engine_data))); // Test writing through the trait method @@ -973,6 +1250,59 @@ mod tests { assert_eq!(data[0].num_columns(), 2); } + #[tokio::test] + async fn test_read_parquet_files_respects_batch_size() { + let store = Arc::new(InMemory::new()); + let writer: Arc = Arc::new(DefaultParquetHandler::new( + store.clone(), + Arc::new(TokioBackgroundExecutor::new()), + )); + + let engine_data: Box = Box::new(ArrowEngineData::new( + RecordBatch::try_from_iter(vec![( + "x", + Arc::new(Int64Array::from((0..10).collect::>())) as Arc, + )]) + .unwrap(), + )); + let file_url = Url::parse("memory:///test/batch_size.parquet").unwrap(); + writer + .write_parquet_file(file_url.clone(), Box::new(std::iter::once(Ok(engine_data)))) + .unwrap(); + + let path = Path::from_url_path(file_url.path()).unwrap(); + let metadata = store.head(&path).await.unwrap(); + let reader = ParquetObjectReader::new(store.clone(), path); + let physical_schema = ParquetRecordBatchStreamBuilder::new(reader) + .await + .unwrap() + .schema() + .clone(); + let file_meta = FileMeta { + location: file_url, + last_modified: 0, + size: metadata.size, + }; + + // With a batch size of 4, the 10-row file should be split into batches of 4, 4, 2. + let handler = + DefaultParquetHandler::new(store.clone(), Arc::new(TokioBackgroundExecutor::new())) + .with_batch_size(NonZero::new(4).unwrap()); + let data: Vec = handler + .read_parquet_files( + slice::from_ref(&file_meta), + Arc::new(physical_schema.try_into_kernel().unwrap()), + None, + ) + .unwrap() + .map(into_record_batch) + .try_collect() + .unwrap(); + + let row_counts: Vec = data.iter().map(|b| b.num_rows()).collect(); + assert_eq!(row_counts, vec![4, 4, 2]); + } + #[tokio::test] async fn test_parquet_handler_trait_write_and_read_roundtrip() { let store = Arc::new(InMemory::new()); @@ -1081,7 +1411,7 @@ mod tests { )); // Create iterator with single batch - let data_iter: Box>> + Send> = + let data_iter: DeltaResultIteratorStatic> = Box::new(std::iter::once(Ok(engine_data))); // Write the data @@ -1255,8 +1585,8 @@ mod tests { /// Test that field IDs are accessible via ColumnMetadataKey::ParquetFieldId as documented. /// - /// Per trait definitions in lib.rs, field IDs should be accessible via StructField::get_config_value - /// with ColumnMetadataKey::ParquetFieldId. + /// Per trait definitions in lib.rs, field IDs should be accessible via + /// StructField::get_config_value with ColumnMetadataKey::ParquetFieldId. #[test] fn test_parquet_footer_read_with_field_id() { // Write parquet file with field ID @@ -1295,12 +1625,13 @@ mod tests { .find(|f| f.name() == "value") .unwrap(); - // Field ID is transformed to kernel key when reading + // Field ID is transformed to kernel key when reading. arrow->kernel parses the + // `PARQUET:field_id` string back into kernel's canonical `MetadataValue::Number(i64)`. assert_eq!( field .metadata() .get(ColumnMetadataKey::ParquetFieldId.as_ref()), - Some(&"42".into()) + Some(&MetadataValue::Number(42)) ); // Field ID should be accessible via documented API @@ -1308,8 +1639,8 @@ mod tests { .expect("Field ID should be accessible via ColumnMetadataKey::ParquetFieldId per lib.rs:836-837"); match field_id { - crate::schema::MetadataValue::String(id) => assert_eq!(id, "42"), - crate::schema::MetadataValue::Number(id) => assert_eq!(*id, 42), + delta_kernel::schema::MetadataValue::String(id) => assert_eq!(id, "42"), + delta_kernel::schema::MetadataValue::Number(id) => assert_eq!(*id, 42), other => panic!("Expected String or Number, got {other:?}"), } } @@ -1319,11 +1650,9 @@ mod tests { /// Per lib.rs:676-680, field IDs (via [`ColumnMetadataKey::ParquetFieldId`]) should take /// precedence over field names for column matching. /// - /// [`ColumnMetadataKey::ParquetFieldId`]: crate::schema::ColumnMetadataKey::ParquetFieldId + /// [`ColumnMetadataKey::ParquetFieldId`]: delta_kernel::schema::ColumnMetadataKey::ParquetFieldId #[test] fn test_read_parquet_with_field_id_matching() { - use crate::schema::{ColumnMetadataKey, MetadataValue, StructField, StructType}; - // Write parquet with field IDs using PARQUET_FIELD_ID_META_KEY (Parquet's native key) // The kernel will transform these to parquet.field.id when reading let fields = vec![ @@ -1357,13 +1686,12 @@ mod tests { // Create kernel schema with DIFFERENT names but SAME field IDs let kernel_schema = Arc::new( StructType::try_new(vec![ - StructField::new("user_id", crate::schema::DataType::LONG, false).with_metadata([ - ( + StructField::new("user_id", delta_kernel::schema::DataType::LONG, false) + .with_metadata([( ColumnMetadataKey::ParquetFieldId.as_ref(), MetadataValue::Number(1), - ), - ]), - StructField::new("user_name", crate::schema::DataType::STRING, false) + )]), + StructField::new("user_name", delta_kernel::schema::DataType::STRING, false) .with_metadata([( ColumnMetadataKey::ParquetFieldId.as_ref(), MetadataValue::Number(2), @@ -1393,6 +1721,12 @@ mod tests { assert_eq!(data.len(), 1); let batch = &data[0]; + // Verify columns were renamed to match the kernel schema (the names from the parquet + // file's schema are discarded; the matching agreed on field IDs only). + let schema = batch.schema(); + assert_eq!(schema.field(0).name(), "user_id"); + assert_eq!(schema.field(1).name(), "user_name"); + let id_col = batch .column(0) .as_any() @@ -1425,8 +1759,14 @@ mod tests { )]) .unwrap(), )); + let physical_schema = long_schema("a"); let metadata = parquet_handler - .write_parquet(&Url::parse("memory:///data/").unwrap(), data, &[]) + .write_parquet( + &Url::parse("memory:///data/").unwrap(), + data, + &[], + &physical_schema, + ) .await .unwrap(); @@ -1442,4 +1782,100 @@ mod tests { "Parquet file should not contain embedded Arrow schema metadata" ); } + + #[tokio::test] + async fn write_parquet_file_creates_parent_directories() { + // GIVEN a file path whose parent directories do not exist + let temp_dir = tempfile::tempdir().unwrap(); + let nested_path = temp_dir.path().join("a/b/c/output.parquet"); + assert!(!nested_path.parent().unwrap().exists()); + + let store = Arc::new(LocalFileSystem::new()); + let parquet_handler: Arc = Arc::new(DefaultParquetHandler::new( + store.clone(), + Arc::new(TokioBackgroundExecutor::new()), + )); + + let engine_data: Box = Box::new(ArrowEngineData::new( + RecordBatch::try_from_iter(vec![( + "x", + Arc::new(Int64Array::from(vec![1, 2, 3])) as Arc, + )]) + .unwrap(), + )); + let data_iter: DeltaResultIteratorStatic> = + Box::new(std::iter::once(Ok(engine_data))); + + // WHEN we write a parquet file to that path + let file_url = Url::from_file_path(&nested_path).unwrap(); + parquet_handler + .write_parquet_file(file_url.clone(), data_iter) + .unwrap(); + + // THEN the file is created and contains the expected data + assert!(nested_path.exists()); + + let path = Path::from_url_path(file_url.path()).unwrap(); + let reader = ParquetObjectReader::new(store.clone(), path); + let batches: Vec = ParquetRecordBatchStreamBuilder::new(reader) + .await + .unwrap() + .build() + .unwrap() + .try_collect() + .await + .unwrap(); + assert_eq!(batches.len(), 1); + let col = batches[0] + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(col.values(), &[1, 2, 3]); + } + + // === ParquetHandler contract tests === + // + // These call the shared contract helpers in `engine::tests` against `DefaultParquetHandler` + // (the matching `SyncParquetHandler` invocations live in `engine/sync/parquet.rs`). + + fn default_parquet_handler() -> DefaultParquetHandler { + DefaultParquetHandler::new( + Arc::new(LocalFileSystem::new()), + Arc::new(TokioBackgroundExecutor::new()), + ) + } + + #[test] + fn parquet_handler_reads_footer() { + let checkpoint = PathBuf::from( + "../kernel/tests/data/parsed-stats/_delta_log/00000000000000000003.checkpoint.parquet", + ); + test_parquet_handler_reads_footer(&default_parquet_handler(), &checkpoint); + } + + #[test] + fn parquet_handler_footer_errors_on_missing_file() { + test_parquet_handler_footer_errors_on_missing_file(&default_parquet_handler()); + } + + #[test] + fn parquet_handler_footer_preserves_field_ids() { + test_parquet_handler_footer_preserves_field_ids(&default_parquet_handler()); + } + + #[test] + fn parquet_handler_write_always_overwrites() { + test_parquet_handler_write_always_overwrites(&default_parquet_handler()); + } + + #[test] + fn parquet_handler_write_omits_arrow_schema() { + test_parquet_handler_write_omits_arrow_schema(&default_parquet_handler()); + } + + #[test] + fn parquet_handler_reads_file_with_arrow_schema() { + test_parquet_handler_reads_file_with_arrow_schema(&default_parquet_handler()); + } } diff --git a/default-engine/src/rest_store/auth.rs b/default-engine/src/rest_store/auth.rs new file mode 100644 index 0000000000..8a2284394d --- /dev/null +++ b/default-engine/src/rest_store/auth.rs @@ -0,0 +1,118 @@ +//! Auth header providers for [`RestObjectStore`](super::RestObjectStore): the headers attached to +//! every request, including refresh of short-lived credentials. + +use std::sync::PoisonError; +use std::time::Duration; + +use delta_kernel::object_store::Result as ObjectStoreResult; +use reqwest::header::{HeaderMap, HeaderName, HeaderValue}; + +use super::generic_error; + +/// Supplies the HTTP headers attached to every request a +/// [`RestObjectStore`](super::RestObjectStore) makes. +/// +/// Called once per request, so implementations own any caching and refresh of credentials. +/// A static set of headers can use [`StaticHeaderProvider`]; a provider backed by a +/// short-lived, refreshable token should return the current headers on each call. +pub trait AuthHeaderProvider: std::fmt::Debug + Send + Sync { + /// Return the headers to attach to the next request. + fn headers(&self) -> ObjectStoreResult; +} + +/// An [`AuthHeaderProvider`] that always returns the same fixed headers. +/// +/// Suitable when credentials do not expire within the store's lifetime, or when refresh is +/// handled out of band by swapping in a new store. +#[derive(Debug, Clone)] +pub struct StaticHeaderProvider { + headers: HeaderMap, +} + +impl StaticHeaderProvider { + /// Create a provider that returns `headers` on every request. + pub fn new(headers: HeaderMap) -> Self { + Self { headers } + } + + /// Build a provider from `(name, value)` header pairs. Errors if a name or value is not a + /// valid HTTP header. Lets a caller that can't construct a [`HeaderMap`] (e.g. across an FFI + /// boundary) supply headers as plain strings. + pub fn from_pairs( + pairs: impl IntoIterator, + ) -> ObjectStoreResult { + Ok(Self { + headers: headers_from_pairs(pairs)?, + }) + } +} + +impl AuthHeaderProvider for StaticHeaderProvider { + fn headers(&self) -> ObjectStoreResult { + Ok(self.headers.clone()) + } +} + +/// Build a [`HeaderMap`] from `(name, value)` pairs, erroring if a name or value is not a valid +/// HTTP header. +pub fn headers_from_pairs( + pairs: impl IntoIterator, +) -> ObjectStoreResult { + let mut headers = HeaderMap::new(); + for (name, value) in pairs { + let name = HeaderName::from_bytes(name.as_bytes()).map_err(generic_error)?; + let value = HeaderValue::from_str(&value).map_err(generic_error)?; + headers.insert(name, value); + } + Ok(headers) +} + +/// How far before a header set's TTL expires to refresh it, so a request never uses a credential +/// that expires in flight. +const HEADER_REFRESH_BUFFER: Duration = Duration::from_secs(30); + +/// An [`AuthHeaderProvider`] that produces headers via a closure returning `(headers, +/// Option)`. `Some(ttl)` caches the headers until a fixed safety margin before `ttl` +/// elapses, then produces fresh ones; `None` re-runs the closure on every request. +pub struct RefreshingHeaderProvider { + produce: Box ObjectStoreResult<(HeaderMap, Option)> + Send + Sync>, + cached: std::sync::Mutex>, +} + +impl std::fmt::Debug for RefreshingHeaderProvider { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("RefreshingHeaderProvider") + .finish_non_exhaustive() + } +} + +impl RefreshingHeaderProvider { + /// Create a provider that calls `produce` to obtain `(headers, ttl)`; see the type docs for how + /// the optional `ttl` drives caching. + pub fn new( + produce: impl Fn() -> ObjectStoreResult<(HeaderMap, Option)> + Send + Sync + 'static, + ) -> Self { + Self { + produce: Box::new(produce), + cached: std::sync::Mutex::new(None), + } + } +} + +impl AuthHeaderProvider for RefreshingHeaderProvider { + fn headers(&self) -> ObjectStoreResult { + // Hold the lock across the (rare) produce call so concurrent refreshes are single-flight. + let mut cached = self.cached.lock().unwrap_or_else(PoisonError::into_inner); + if let Some((headers, deadline)) = cached.as_ref() { + if deadline.saturating_duration_since(std::time::Instant::now()) > HEADER_REFRESH_BUFFER + { + return Ok(headers.clone()); + } + } + let (headers, ttl) = (self.produce)()?; + // Skip caching if an absurd TTL overflows `Instant`, rather than panicking. + *cached = ttl + .and_then(|ttl| Some((headers.clone(), std::time::Instant::now().checked_add(ttl)?))); + Ok(headers) + } +} diff --git a/default-engine/src/rest_store/client.rs b/default-engine/src/rest_store/client.rs new file mode 100644 index 0000000000..79fb835e7b --- /dev/null +++ b/default-engine/src/rest_store/client.rs @@ -0,0 +1,123 @@ +//! Builds the [`reqwest::Client`] backing a [`RestObjectStore`](super::RestObjectStore), including +//! optional TLS client auth, DNS overrides, and request timeouts. + +use std::net::SocketAddr; +use std::time::Duration; + +use delta_kernel::object_store::Result as ObjectStoreResult; +use reqwest::Client; +#[cfg(any(feature = "rustls", feature = "native-tls"))] +use reqwest::{Certificate, Identity}; + +use super::generic_error; + +/// Options for the [`reqwest::Client`] backing a [`RestObjectStore`](super::RestObjectStore). All +/// fields are optional; the default value yields a plain client. When `cert_path`, `key_path`, and +/// `ca_path` are all set, mutual TLS is enabled. +#[derive(Debug, Clone, Default)] +pub struct RestClientOptions { + /// PEM client-certificate path. mTLS is enabled only when cert, key, and CA are all set. + pub cert_path: Option, + /// PEM client-private-key path. + pub key_path: Option, + /// PEM CA-bundle path used to verify the server. + pub ca_path: Option, + /// DNS overrides as `host=ip:port` entries: connect to the given address for `host`, + /// bypassing the system resolver. TLS SNI/hostname verification is unchanged. + pub dns_overrides: Vec, + /// Request timeout in seconds; `None` uses reqwest's default. + pub timeout_secs: Option, +} + +// === mTLS configuration === +// +// reqwest's `Identity` is built differently per backend: rustls wants a single PEM bundle +// (cert followed by key), native-tls takes cert and key as separate PEM buffers. Each `cfg` +// variant of `build_identity` builds it the way its backend expects. + +#[cfg(feature = "rustls")] +fn build_identity(cert: &str, key: &str) -> ObjectStoreResult { + let mut bundle = std::fs::read(cert).map_err(generic_error)?; + bundle.extend_from_slice(&std::fs::read(key).map_err(generic_error)?); + Identity::from_pem(&bundle).map_err(generic_error) +} + +#[cfg(all(feature = "native-tls", not(feature = "rustls")))] +fn build_identity(cert: &str, key: &str) -> ObjectStoreResult { + let cert_pem = std::fs::read(cert).map_err(generic_error)?; + let key_pem = std::fs::read(key).map_err(generic_error)?; + Identity::from_pkcs8_pem(&cert_pem, &key_pem).map_err(generic_error) +} + +// CA-cert read and builder assembly are backend-agnostic, so they live once here. +#[cfg(any(feature = "rustls", feature = "native-tls"))] +fn configure_mtls( + builder: reqwest::ClientBuilder, + cert: &str, + key: &str, + ca: &str, +) -> ObjectStoreResult { + let identity = build_identity(cert, key)?; + let ca_cert = + Certificate::from_pem(&std::fs::read(ca).map_err(generic_error)?).map_err(generic_error)?; + Ok(builder.identity(identity).add_root_certificate(ca_cert)) +} + +#[cfg(not(any(feature = "rustls", feature = "native-tls")))] +fn configure_mtls( + _builder: reqwest::ClientBuilder, + _cert: &str, + _key: &str, + _ca: &str, +) -> ObjectStoreResult { + Err(generic_error( + "mTLS requires the `rustls` or `native-tls` feature to be enabled", + )) +} + +/// Build a [`reqwest::Client`] from [`RestClientOptions`]. Enables mTLS when `cert_path`, +/// `key_path`, and `ca_path` are all present. +pub fn build_rest_client(opts: &RestClientOptions) -> ObjectStoreResult { + let mut builder = Client::builder(); + // Pin the TLS backend when both are compiled in: prefer rustls (the crate's default + // feature), since reqwest otherwise defaults to native-tls and would mismatch the + // rustls-built identity. native-tls is used only when it is the sole backend. + #[cfg(feature = "rustls")] + { + builder = builder.use_rustls_tls(); + } + #[cfg(all(feature = "native-tls", not(feature = "rustls")))] + { + builder = builder.use_native_tls(); + } + if let Some(secs) = opts.timeout_secs { + builder = builder.timeout(Duration::from_secs(secs)); + } + match (&opts.cert_path, &opts.key_path, &opts.ca_path) { + (Some(cert), Some(key), Some(ca)) => { + builder = configure_mtls(builder, cert, key, ca)?; + } + (None, None, None) => {} + // A partial mTLS config is an error, not a silent fall-through to a plain client. + _ => { + return Err(generic_error( + "partial mTLS config: cert_path, key_path, and ca_path must be set \ + together (or all omitted)", + )); + } + } + for entry in &opts.dns_overrides { + let entry = entry.trim(); + if entry.is_empty() { + continue; + } + let (host, addr) = entry.split_once('=').ok_or_else(|| { + generic_error(format!( + "invalid DNS override (expected `host=ip:port`): {entry}" + )) + })?; + let addr: SocketAddr = addr.parse().map_err(generic_error)?; + builder = builder.resolve(host, addr); + } + builder.build().map_err(generic_error) +} diff --git a/default-engine/src/rest_store/config.rs b/default-engine/src/rest_store/config.rs new file mode 100644 index 0000000000..c05504a092 --- /dev/null +++ b/default-engine/src/rest_store/config.rs @@ -0,0 +1,194 @@ +//! REST endpoint configuration for [`RestObjectStore`](super::RestObjectStore): URL path prefixes +//! and JSON field names for list/put request and response shapes. + +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::{ObjectMeta, Result as ObjectStoreResult}; + +use super::generic_error; + +/// One page of a list response: the objects on this page plus an optional token for the next. +pub(crate) struct RestListPage { + /// The (file) objects on this page. Directories are filtered out, since + /// [`ObjectStore::list`](delta_kernel::object_store::ObjectStore::list) yields only objects. + pub(crate) objects: Vec, + /// Opaque token to fetch the next page, or `None` when this is the last page. + pub(crate) next_page_token: Option, +} + +/// Per-backend HTTP/JSON settings for a [`RestObjectStore`](super::RestObjectStore). +/// +/// Names URL path prefixes and list-response JSON fields so the store can talk to a REST file API +/// without service-specific code. Credentials come from +/// [`AuthHeaderProvider`](super::AuthHeaderProvider). +#[derive(Debug, Clone)] +pub struct RestEndpointConfig { + /// Path prefix for file (object) operations; the URL is `{base_url}/{files_prefix}/{path}`. + pub files_prefix: String, + /// Path prefix for directory (list) operations. + pub directories_prefix: String, + /// Query-param name carrying the pagination token. + pub page_token_param: String, + /// Query-param name carrying the list start offset. + pub start_from_param: String, + /// Query-param name (value `"true"`) requesting a recursive listing. + pub recursive_param: String, + /// Query-param name controlling overwrite-vs-create on `PUT` (value `"true"`/`"false"`). + pub overwrite_param: String, + /// List-response JSON field holding the array of entries. + pub contents_field: String, + /// List-response JSON field holding the next-page token. + pub next_page_token_field: String, + /// Per-entry JSON field holding the path. + pub entry_path_field: String, + /// Per-entry JSON field holding the size in bytes. + pub entry_size_field: String, + /// Per-entry JSON field holding the directory flag; truthy entries are skipped, since + /// [`ObjectStore::list`](delta_kernel::object_store::ObjectStore::list) yields only objects. + pub entry_is_directory_field: String, + /// Per-entry JSON field holding the last-modified time as epoch milliseconds. + pub entry_last_modified_field: String, + /// If set, every list-entry path must start with this prefix; it is stripped to yield the + /// store-relative path, and an entry outside the prefix is rejected (a scope check). Use when + /// the backend returns absolute keys but the store is rooted at a sub-path. + pub entry_strip_prefix: Option, +} + +/// Join `base_url`, a path `prefix`, and a store-relative `path` into a single URL, trimming +/// stray slashes at each seam. Empty `prefix` or `path` segments are omitted so an empty prefix +/// does not produce a double slash (`{base}//{path}`). +fn join_url(base_url: &str, prefix: &str, path: &str) -> String { + let mut segments = vec![base_url.trim_end_matches('/')]; + let prefix = prefix.trim_matches('/'); + if !prefix.is_empty() { + segments.push(prefix); + } + let path = path.trim_start_matches('/'); + if !path.is_empty() { + segments.push(path); + } + segments.join("/") +} + +impl RestEndpointConfig { + /// Build the URL for file (object) operations on `path`. + pub(crate) fn file_url(&self, base_url: &str, path: &str) -> String { + join_url(base_url, &self.files_prefix, path) + } + + /// Build the URL for directory (list) operations on `path`. + pub(crate) fn directory_url(&self, base_url: &str, path: &str) -> String { + join_url(base_url, &self.directories_prefix, path) + } + + /// Query parameters for a list request. `page_token` drives pagination; `start_from` is an + /// optional offset (see + /// [`ObjectStore::list_with_offset`](delta_kernel::object_store::ObjectStore::list_with_offset)); + /// `recursive` requests a recursive listing. + pub(crate) fn list_query( + &self, + page_token: Option<&str>, + start_from: Option<&str>, + recursive: bool, + ) -> Vec<(String, String)> { + let mut q = Vec::new(); + if let Some(t) = page_token { + q.push((self.page_token_param.clone(), t.to_string())); + } + if let Some(s) = start_from { + q.push((self.start_from_param.clone(), s.to_string())); + } + if recursive { + q.push((self.recursive_param.clone(), "true".to_string())); + } + q + } + + /// Parse a list-response body into a [`RestListPage`]. + /// + /// Entries **must** be ordered by ascending full path, and that order must hold *across* + /// pages (a page's first entry sorts after the previous page's last). + /// [`RestObjectStore`](super::RestObjectStore) relies on this for Delta log replay and does not + /// re-sort: it checks the order as it streams pages and fails the listing with an error if an + /// entry arrives out of order, rather than silently producing a wrong snapshot. + pub(crate) fn parse_list(&self, body: &[u8]) -> ObjectStoreResult { + let root: serde_json::Value = serde_json::from_slice(body).map_err(generic_error)?; + let mut objects = Vec::new(); + if let Some(entries) = root.get(&self.contents_field).and_then(|v| v.as_array()) { + for entry in entries { + let is_dir = entry + .get(&self.entry_is_directory_field) + .and_then(|v| v.as_bool()) + .unwrap_or(false); + if is_dir { + continue; + } + let Some(path) = entry.get(&self.entry_path_field).and_then(|v| v.as_str()) else { + return Err(generic_error(format!( + "list entry is missing the `{}` field", + self.entry_path_field + ))); + }; + let path = match &self.entry_strip_prefix { + Some(prefix) => match path.strip_prefix(prefix.as_str()) { + Some(rest) => rest.trim_start_matches('/'), + None => { + return Err(generic_error(format!( + "list entry `{path}` is outside the configured prefix `{prefix}`" + ))) + } + }, + None => path, + }; + let size = match entry.get(&self.entry_size_field) { + None => 0, + Some(v) => v.as_u64().ok_or_else(|| { + generic_error(format!( + "list entry field `{}` is not a non-negative integer", + self.entry_size_field + )) + })?, + }; + let last_modified = match entry.get(&self.entry_last_modified_field) { + None => chrono::DateTime::UNIX_EPOCH, + Some(v) => { + let ms = v.as_u64().ok_or_else(|| { + generic_error(format!( + "list entry field `{}` is not a non-negative integer", + self.entry_last_modified_field + )) + })?; + (std::time::SystemTime::UNIX_EPOCH + std::time::Duration::from_millis(ms)) + .into() + } + }; + // Fail loudly on an unparseable path rather than silently dropping it, so a + // misbehaving backend cannot corrupt log replay with a partial listing. + let location = Path::parse(path).map_err(|e| { + generic_error(format!("list entry path `{path}` is not a valid path: {e}")) + })?; + objects.push(ObjectMeta { + location, + last_modified, + size, + e_tag: None, + version: None, + }); + } + } + let next_page_token = root + .get(&self.next_page_token_field) + .and_then(|v| v.as_str()) + .filter(|s| !s.is_empty()) + .map(|s| s.to_string()); + Ok(RestListPage { + objects, + next_page_token, + }) + } + + /// Query parameters for a `PUT`. `overwrite` selects overwrite-vs-create semantics via the + /// configured [`overwrite_param`](Self::overwrite_param). + pub(crate) fn put_query(&self, overwrite: bool) -> Vec<(String, String)> { + vec![(self.overwrite_param.clone(), overwrite.to_string())] + } +} diff --git a/default-engine/src/rest_store/mod.rs b/default-engine/src/rest_store/mod.rs new file mode 100644 index 0000000000..6f0e745d9e --- /dev/null +++ b/default-engine/src/rest_store/mod.rs @@ -0,0 +1,46 @@ +//! A generic REST/HTTP-backed [`ObjectStore`](delta_kernel::object_store::ObjectStore). +//! +//! [`RestObjectStore`] implements [`ObjectStore`](delta_kernel::object_store::ObjectStore) over a +//! REST "file API" -- a service that exposes list/read/write of files behind authenticated HTTP +//! endpoints rather than as a plain blob store -- so the default engine can read and write Delta +//! tables through it with no service-specific logic in the kernel. +//! +//! The caller supplies everything service-specific through two pieces: +//! +//! - [`AuthHeaderProvider`] -- headers attached to every request (auth, identity). Consulted per +//! request, so an implementation can refresh short-lived credentials. +//! - [`RestEndpointConfig`] -- the REST dialect: path-to-URL mapping, list and write query +//! parameters, and list-response JSON field names. HTTP `404 -> NotFound` and `409 -> +//! AlreadyExists` are mapped in [`RestObjectStore`]. +//! +//! Only the operations kernel needs are implemented (read, head, list, write, delete); the rest +//! return [`ObjectStoreError::NotSupported`]. + +use delta_kernel::object_store::Error as ObjectStoreError; + +mod auth; +mod client; +mod config; +mod response; +mod store; +#[cfg(test)] +mod tests; + +pub use auth::{ + headers_from_pairs, AuthHeaderProvider, RefreshingHeaderProvider, StaticHeaderProvider, +}; +pub use client::{build_rest_client, RestClientOptions}; +pub use config::RestEndpointConfig; +// Re-exported so callers can name the header type in `AuthHeaderProvider`'s public API. +pub use reqwest::header::HeaderMap; +pub use store::RestObjectStore; + +/// Build a generic [`ObjectStoreError`] from a source error or message. +pub(crate) fn generic_error( + source: impl Into>, +) -> ObjectStoreError { + ObjectStoreError::Generic { + store: "RestObjectStore", + source: source.into(), + } +} diff --git a/default-engine/src/rest_store/response.rs b/default-engine/src/rest_store/response.rs new file mode 100644 index 0000000000..89c255374e --- /dev/null +++ b/default-engine/src/rest_store/response.rs @@ -0,0 +1,149 @@ +//! HTTP response status handling for [`RestObjectStore`](super::store::RestObjectStore). +//! +//! Three separate concerns show up across REST calls: +//! +//! 1. **Retry classification** ([`is_retryable_http_status`], [`is_transient`]) -- 5xx / dropped +//! connections may succeed on another attempt. These are *not* terminal errors yet. +//! 2. **Typed mapping** ([`typed_http_error`]) -- select status codes become specific +//! [`ObjectStoreError`] variants (`404 -> NotFound`, `409 -> AlreadyExists`). Returns `None` for +//! 2xx and for statuses with no typed mapping. +//! 3. **Generic non-2xx** ([`reject_remaining_non_success`]) -- reqwest's `error_for_status` turns +//! any remaining failure into [`ObjectStoreError::Generic`]. +//! +//! Most callers use [`ensure_success_response`], which applies (2) then (3). List and +//! verified-create encode extra rules in [`ensure_list_response`] and +//! [`classify_put_create_response`]. + +use delta_kernel::object_store::{Error as ObjectStoreError, Result as ObjectStoreResult}; + +use super::generic_error; + +/// The last failure that triggered a retry. +pub(crate) enum RetryFailure { + ServerError(reqwest::StatusCode), + Transport(String), +} + +impl std::fmt::Display for RetryFailure { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::ServerError(status) => write!(f, "HTTP {status}"), + Self::Transport(msg) => write!(f, "{msg}"), + } + } +} + +/// How verified-create PUT should treat an HTTP response. +pub(crate) enum PutCreateDisposition { + Success, + Terminal(ObjectStoreError), + Ambiguous { failure: RetryFailure }, +} + +/// HTTP 5xx may succeed on retry; callers classify it before mapping to a terminal error. +pub(crate) fn is_retryable_http_status(status: reqwest::StatusCode) -> bool { + status.is_server_error() +} + +/// Transport-level failures worth retrying for an idempotent request. +pub(crate) fn is_transient(err: &reqwest::Error) -> bool { + err.is_timeout() + || err.is_connect() + || ((err.is_request() || err.is_body()) && err.status().is_none()) +} + +/// If `status` maps to a typed [`ObjectStoreError`] (`404`, `409`, ...), return it. Returns `None` +/// for 2xx and for statuses with no typed mapping (see [`reject_remaining_non_success`]). +pub(crate) fn typed_http_error( + status: reqwest::StatusCode, + path: &str, +) -> Option { + match status { + reqwest::StatusCode::NOT_FOUND => Some(ObjectStoreError::NotFound { + path: path.to_string(), + source: "HTTP 404".into(), + }), + reqwest::StatusCode::CONFLICT => Some(ObjectStoreError::AlreadyExists { + path: path.to_string(), + source: "HTTP 409".into(), + }), + _ => None, + } +} + +/// Apply reqwest's catch-all check for any remaining non-2xx after typed mapping. +pub(crate) fn reject_remaining_non_success( + response: reqwest::Response, +) -> ObjectStoreResult { + response.error_for_status().map_err(generic_error) +} + +/// Typed status mapping, then generic non-2xx rejection. Returns the response on success. +pub(crate) fn ensure_success_response( + response: reqwest::Response, + path: &str, +) -> ObjectStoreResult { + if let Some(err) = typed_http_error(response.status(), path) { + return Err(err); + } + reject_remaining_non_success(response) +} + +/// Like [`ensure_success_response`], but a first-page `NotFound` yields `Ok(None)` so an absent +/// directory lists as empty. A `NotFound` mid-pagination is an error. +pub(crate) fn ensure_list_response( + response: reqwest::Response, + path: &str, + page_token: Option<&str>, +) -> ObjectStoreResult> { + if let Some(err) = typed_http_error(response.status(), path) { + if matches!(err, ObjectStoreError::NotFound { .. }) && page_token.is_none() { + return Ok(None); + } + return Err(err); + } + Ok(Some(reject_remaining_non_success(response)?)) +} + +/// Classify a verified-create PUT attempt: success, a terminal error, or an ambiguous outcome that +/// requires read-back reconciliation. +/// +/// - **Success** -- 2xx after [`reject_remaining_non_success`] validation. +/// - **Terminal** -- definite failure (`404`, first-attempt `409`, other non-retryable 4xx, or +/// non-transient transport error). +/// - **Ambiguous** -- retryable 5xx, `409` on a retry (may be our own landed write), or transient +/// transport error. +pub(crate) fn classify_put_create_response( + response: Result, + path: &str, + retries: u32, +) -> ObjectStoreResult { + let response = match response { + Ok(resp) => resp, + Err(e) if is_transient(&e) => { + return Ok(PutCreateDisposition::Ambiguous { + failure: RetryFailure::Transport(e.to_string()), + }); + } + Err(e) => return Err(generic_error(e)), + }; + let status = response.status(); + if status.is_success() { + reject_remaining_non_success(response)?; + return Ok(PutCreateDisposition::Success); + } + if let Some(err) = typed_http_error(status, path) { + if matches!(err, ObjectStoreError::AlreadyExists { .. }) && retries > 0 { + return Ok(PutCreateDisposition::Ambiguous { + failure: RetryFailure::ServerError(status), + }); + } + return Ok(PutCreateDisposition::Terminal(err)); + } + if is_retryable_http_status(status) { + return Ok(PutCreateDisposition::Ambiguous { + failure: RetryFailure::ServerError(status), + }); + } + reject_remaining_non_success(response).map(|_| PutCreateDisposition::Success) +} diff --git a/default-engine/src/rest_store/store.rs b/default-engine/src/rest_store/store.rs new file mode 100644 index 0000000000..07a9189771 --- /dev/null +++ b/default-engine/src/rest_store/store.rs @@ -0,0 +1,713 @@ +//! [`RestObjectStore`]: a generic REST/HTTP-backed +//! [`ObjectStore`](delta_kernel::object_store::ObjectStore). See the [module docs](super). + +use std::sync::Arc; +use std::time::Duration; + +use async_trait::async_trait; +use bytes::Bytes; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::{ + Attributes, CopyOptions, Error as ObjectStoreError, GetOptions, GetRange, GetResult, + GetResultPayload, ListResult, MultipartUpload, ObjectMeta, ObjectStore, PutMode, + PutMultipartOptions, PutOptions, PutPayload, PutResult, Result as ObjectStoreResult, +}; +use futures::stream::BoxStream; +use futures::StreamExt as _; +use reqwest::header::HeaderMap; +use reqwest::Client; +use tracing::{info, warn}; + +use super::auth::AuthHeaderProvider; +use super::config::RestEndpointConfig; +use super::generic_error; +use super::response::{ + classify_put_create_response, ensure_list_response, ensure_success_response, + is_retryable_http_status, is_transient, PutCreateDisposition, RetryFailure, +}; + +/// A generic REST/HTTP-backed [`ObjectStore`]. See the [module docs](super). +#[derive(Debug, Clone)] +pub struct RestObjectStore { + base_url: String, + client: Client, + auth: Arc, + config: Arc, + /// Retries (beyond the first attempt) for transient failures on idempotent requests. 0 + /// disables. + max_retries: u32, + /// Verify a `Create` put by reading it back when the outcome is ambiguous (5xx / dropped + /// connection), so a write that landed despite the error is not mistaken for a conflict. + verify_on_ambiguous: bool, +} + +impl std::fmt::Display for RestObjectStore { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "RestObjectStore({})", self.base_url) + } +} + +impl RestObjectStore { + /// Create a store targeting `base_url`, using `client` for transport, `auth` for per-request + /// headers, and `config` for the REST dialect. + pub fn new( + base_url: impl Into, + client: Client, + auth: Arc, + config: Arc, + ) -> Self { + Self { + base_url: base_url.into(), + client, + auth, + config, + max_retries: 0, + verify_on_ambiguous: false, + } + } + + /// Retry transient failures (5xx, connect/timeout) on idempotent requests up to `n` times. + pub fn with_max_retries(mut self, n: u32) -> Self { + self.max_retries = n; + self + } + + /// Verify a `Create` put by reading it back on an ambiguous outcome. + pub fn with_verify_on_ambiguous(mut self, verify: bool) -> Self { + self.verify_on_ambiguous = verify; + self + } + + /// Fetch the current auth headers from the provider. + fn headers(&self) -> ObjectStoreResult { + self.auth.headers() + } + + /// Send an idempotent request, retrying transient failures (5xx, connect/timeout) up to + /// [`Self::max_retries`]. Returns the response for the caller to map via + /// [`ensure_success_response`]. `target` labels retry-summary log lines (path or operation). + async fn send_idempotent( + &self, + target: &str, + make: impl Fn(&Client, HeaderMap) -> reqwest::RequestBuilder, + ) -> ObjectStoreResult { + let mut retries = 0u32; + let mut last_failure = None::; + loop { + let exhausted = retries >= self.max_retries; + // Fetch headers per attempt so a refreshable provider can produce a fresh token. + let headers = self.headers()?; + match make(&self.client, headers).send().await { + Ok(resp) if !exhausted && is_retryable_http_status(resp.status()) => { + last_failure = Some(RetryFailure::ServerError(resp.status())); + } + Ok(resp) => { + if let Some(failure) = last_failure.as_ref().filter(|_| retries > 0) { + log_retry_outcome(target, retries, failure, true); + } + return Ok(resp); + } + Err(e) if !exhausted && is_transient(&e) => { + last_failure = Some(RetryFailure::Transport(e.to_string())); + } + Err(e) => { + if retries > 0 { + let transport_failure = RetryFailure::Transport(e.to_string()); + let failure = last_failure.as_ref().unwrap_or(&transport_failure); + log_retry_outcome(target, retries, failure, false); + } + return Err(generic_error(e)); + } + } + retries += 1; + backoff(retries).await; + } + } + + /// PUT a `Create`, verifying the result on an ambiguous outcome (5xx or transient transport + /// error). Reads the object back to distinguish a write that landed (success) from a real + /// conflict, retrying only while the write is confirmed absent. + /// + /// Does not call [`ensure_success_response`] directly on every status: policy is encoded in + /// [`classify_put_create_response`]. HTTP 5xx and retry-time 409 are ambiguous (read-back + + /// retry), not terminal errors. + /// + /// A `409` (conflict) on the *first* attempt is a genuine pre-existing object and returns + /// `AlreadyExists` immediately. On a *retry*, a `409` is most likely the writer's own + /// already-landed write (the prior attempt succeeded despite an ambiguous error), so it is + /// routed through the read-back instead of being trusted verbatim. + /// + /// Assumes the backend writes objects verbatim and atomically, and that commit bodies carry + /// writer-unique content -- so a byte-for-byte match implies we wrote it, not a competitor. + async fn put_create_verified( + &self, + path: &str, + url: &str, + query: &[(String, String)], + body: Bytes, + ) -> ObjectStoreResult { + let mut retries = 0u32; + let mut last_failure = None::; + loop { + // Fetch headers per attempt so a refreshable provider can produce a fresh token. + let disposition = classify_put_create_response( + self.client + .put(url) + .query(query) + .headers(self.headers()?) + .body(body.clone()) + .send() + .await, + path, + retries, + )?; + + match disposition { + PutCreateDisposition::Success => { + if let Some(failure) = last_failure.as_ref().filter(|_| retries > 0) { + log_retry_outcome(path, retries, failure, true); + } + return Ok(put_result()); + } + PutCreateDisposition::Terminal(err) => return Err(err), + PutCreateDisposition::Ambiguous { failure } => last_failure = Some(failure), + } + + // Ambiguous outcome -- read back to tell a landed write from a real conflict. A + // transient read-back failure leaves the outcome ambiguous, so treat it like an absent + // write and consume a retry rather than making it terminal. + match self.read_back(path, &body).await { + Ok(WriteState::Matches) => { + if let Some(failure) = last_failure.as_ref().filter(|_| retries > 0) { + log_retry_outcome(path, retries, failure, true); + } + return Ok(put_result()); + } + Ok(WriteState::Differs) => { + return Err(ObjectStoreError::AlreadyExists { + path: path.to_string(), + source: "verified conflicting write".into(), + }) + } + Ok(WriteState::Absent) => {} + Err(e) if retries >= self.max_retries => { + let transport_failure = RetryFailure::Transport(e.to_string()); + let failure = last_failure.as_ref().unwrap_or(&transport_failure); + log_retry_outcome(path, retries, failure, false); + return Err(e); + } + Err(e) => { + last_failure = Some(RetryFailure::Transport(e.to_string())); + } + } + if retries >= self.max_retries { + if let Some(failure) = &last_failure { + log_retry_outcome(path, retries, failure, false); + } + return Err(generic_error(format!( + "exceeded max retries ({}) during put for `{path}` without confirming the write landed", + self.max_retries + ))); + } + retries += 1; + backoff(retries).await; + } + } + + /// Read `path` back and compare its bytes with `expected`. + async fn read_back(&self, path: &str, expected: &Bytes) -> ObjectStoreResult { + match self.get_file(path, None).await { + Ok((bytes, _, _)) if bytes == *expected => Ok(WriteState::Matches), + Ok(_) => Ok(WriteState::Differs), + Err(ObjectStoreError::NotFound { .. }) => Ok(WriteState::Absent), + Err(e) => Err(e), + } + } + + /// Delete a single object via HTTP `DELETE`. DELETE is idempotent, so transient failures are + /// retried via [`Self::send_idempotent`]. + async fn delete_one(&self, location: &Path) -> ObjectStoreResult<()> { + let path = location.as_ref(); + let url = self.config.file_url(&self.base_url, path); + let response = self + .send_idempotent(path, |c, h| c.delete(&url).headers(h)) + .await?; + ensure_success_response(response, path)?; + Ok(()) + } + + /// GET `path` (optionally with a `Range` header) and return the body, response headers, and + /// HTTP status. The status lets a ranged caller distinguish a partial (`206`) response from a + /// full-body `200`. + async fn get_file( + &self, + path: &str, + range_header: Option<&str>, + ) -> ObjectStoreResult<(Bytes, HeaderMap, reqwest::StatusCode)> { + let url = self.config.file_url(&self.base_url, path); + let range = range_header + .map(reqwest::header::HeaderValue::from_str) + .transpose() + .map_err(generic_error)?; + let response = self + .send_idempotent(path, |c, mut h| { + if let Some(v) = &range { + h.insert(reqwest::header::RANGE, v.clone()); + } + c.get(&url).headers(h) + }) + .await?; + let response = ensure_success_response(response, path)?; + let status = response.status(); + let resp_headers = response.headers().clone(); + let body = response.bytes().await.map_err(generic_error)?; + Ok((body, resp_headers, status)) + } + + /// Issue an HTTP `HEAD` and build [`ObjectMeta`] from the response headers, without + /// downloading the body. Used to serve `get_opts(head = true)` / `head()`. + async fn head_meta(&self, path: &str, location: &Path) -> ObjectStoreResult { + let url = self.config.file_url(&self.base_url, path); + let response = self + .send_idempotent(path, |c, h| c.head(&url).headers(h)) + .await?; + let response = ensure_success_response(response, path)?; + let headers = response.headers(); + let size = match headers.get(reqwest::header::CONTENT_LENGTH) { + None => { + return Err(generic_error(format!( + "HEAD for `{path}` is missing a Content-Length header" + ))); + } + Some(v) => { + let s = v + .to_str() + .map_err(|e| generic_error(format!("invalid Content-Length header: {e}")))?; + s.parse::().map_err(|e| { + generic_error(format!("invalid Content-Length header `{s}`: {e}")) + })? + } + }; + let last_modified = parse_last_modified(headers); + let e_tag = parse_etag(headers); + Ok(ObjectMeta { + location: location.clone(), + last_modified, + size, + e_tag, + version: None, + }) + } + + /// Stream a paginated listing of `prefix`. When `exclusive_offset` is set, entries at or + /// before it are dropped client-side so the [`ObjectStore::list_with_offset`] exclusive-offset + /// contract holds regardless of how the backend interprets its own offset parameter. + fn list_paginated( + &self, + prefix: String, + start_from: Option, + exclusive_offset: Option, + recursive: bool, + ) -> BoxStream<'static, ObjectStoreResult> { + let store = self.clone(); + let stream = async_stream::stream! { + let mut page_token: Option = None; + // start_from applies only to the first request; page_token drives later pages. + let mut start_from = start_from; + // Listings must be ascending by path across the whole stream; verified here so a + // misordered backend fails loudly instead of corrupting log replay. + let mut last_path: Option = None; + 'pages: loop { + let url = store.config.directory_url(&store.base_url, &prefix); + let query = store.config.list_query( + page_token.as_deref(), + start_from.as_deref(), + recursive, + ); + start_from = None; + let list_target = if prefix.is_empty() { + "list".to_string() + } else { + format!("list `{prefix}`") + }; + let response = match store + .send_idempotent(&list_target, |c, h| { + c.get(url.as_str()).query(&query).headers(h) + }) + .await + { + Ok(r) => r, + Err(e) => { yield Err(e); break; } + }; + let response = match ensure_list_response( + response, + &prefix, + page_token.as_deref(), + ) { + Ok(Some(r)) => r, + Ok(None) => break, + Err(e) => { yield Err(e); break; } + }; + let body = match response.bytes().await { + Ok(b) => b, + Err(e) => { yield Err(generic_error(e)); break; } + }; + let page = match store.config.parse_list(&body) { + Ok(p) => p, + Err(e) => { yield Err(e); break; } + }; + for meta in page.objects { + // Enforce the exclusive-offset contract client-side. + if let Some(off) = &exclusive_offset { + if meta.location <= *off { + continue; + } + } + if let Some(last) = &last_path { + if meta.location < *last { + yield Err(generic_error(format!( + "REST listing returned out-of-order entry `{}` after `{}`; \ + RestEndpointConfig must return lexicographically sorted paths", + meta.location, last + ))); + break 'pages; + } + } + last_path = Some(meta.location.clone()); + yield Ok(meta); + } + match page.next_page_token { + Some(token) => page_token = Some(token), + None => break, + } + } + }; + Box::pin(stream) + } +} + +/// Convert an HTTP `GetRange` into a `Range` header value. +fn get_range_to_header(range: &GetRange) -> String { + match range { + GetRange::Bounded(r) => format!("bytes={}-{}", r.start, r.end.saturating_sub(1)), + GetRange::Offset(n) => format!("bytes={}-", n), + GetRange::Suffix(n) => format!("bytes=-{}", n), + } +} + +/// Parse a `Content-Range: bytes start-end/total` header into `(range, total_size)`. +/// +/// Errors on a malformed header rather than guessing, matching `object_store`'s own HTTP client: +/// a server that sends a partial response with a bogus `Content-Range` should surface as an error, +/// not silently degrade the reported range/size. +fn parse_content_range(header: &str) -> ObjectStoreResult<(std::ops::Range, u64)> { + let invalid = || generic_error(format!("malformed Content-Range header: `{header}`")); + let (range_part, total_part) = header + .strip_prefix("bytes ") + .and_then(|inner| inner.split_once('/')) + .ok_or_else(invalid)?; + let total = total_part.parse::().map_err(|_| invalid())?; + let (start, end) = range_part.split_once('-').ok_or_else(invalid)?; + let start = start.parse::().map_err(|_| invalid())?; + let end = end.parse::().map_err(|_| invalid())?; + // Reject a reversed or out-of-bounds range: `end < start` would underflow in + // `GetResult::bytes()`, and `start > total` is nonsensical for a partial response. + if end < start || start > total { + return Err(invalid()); + } + Ok((start..end.saturating_add(1), total)) +} + +/// Build the [`ObjectStoreError::NotSupported`] returned for an operation Delta never issues. +fn not_supported(op: &str) -> ObjectStoreError { + ObjectStoreError::NotSupported { + source: format!("RestObjectStore does not support {op}").into(), + } +} + +/// A successful PUT result; this store surfaces no etag or version. +fn put_result() -> PutResult { + PutResult { + e_tag: None, + version: None, + } +} + +/// Outcome of reading a file back to compare against bytes we tried to write. +enum WriteState { + Matches, + Differs, + Absent, +} + +/// Log once when a request retried before completing. Per-attempt lines are omitted to avoid noise. +fn log_retry_outcome(target: &str, retries: u32, last_failure: &RetryFailure, succeeded: bool) { + if succeeded { + info!( + target, + retries, + last_failure = %last_failure, + "REST request succeeded after retries" + ); + } else { + warn!( + target, + retries, + last_failure = %last_failure, + "REST request failed after retries" + ); + } +} + +/// Exponential backoff for retry `n` (1-based): 100ms doubling, capped at 2s. `n.min(6)` bounds +/// the shift (`50 << 6` already exceeds the ceiling). +async fn backoff(n: u32) { + let ms = (50u64 << n.min(6)).min(2_000); + tokio::time::sleep(Duration::from_millis(ms)).await; +} + +/// Parse the `Last-Modified` response header (RFC 2822), defaulting to the Unix epoch when it is +/// absent or unparseable. +fn parse_last_modified(headers: &HeaderMap) -> chrono::DateTime { + headers + .get(reqwest::header::LAST_MODIFIED) + .and_then(|v| v.to_str().ok()) + .and_then(|s| chrono::DateTime::parse_from_rfc2822(s).ok()) + .map(|dt| dt.with_timezone(&chrono::Utc)) + .unwrap_or(chrono::DateTime::UNIX_EPOCH) +} + +/// Extract the `ETag` response header, if present and valid UTF-8. +fn parse_etag(headers: &HeaderMap) -> Option { + headers + .get(reqwest::header::ETAG) + .and_then(|v| v.to_str().ok()) + .map(|s| s.to_string()) +} + +#[async_trait] +impl ObjectStore for RestObjectStore { + async fn get_opts(&self, location: &Path, options: GetOptions) -> ObjectStoreResult { + let path_str = location.as_ref(); + + if let Some(range) = &options.range { + range.is_valid().map_err(generic_error)?; + } + + // A head-only request resolves metadata via HTTP HEAD and returns an empty body, so we + // don't download the object just to read its size/etag (e.g. parquet footer probes). + if options.head { + let meta = self.head_meta(path_str, location).await?; + options.check_preconditions(&meta)?; + let size = meta.size; + return Ok(GetResult { + payload: GetResultPayload::Stream(Box::pin(futures::stream::empty())), + range: 0..size, + meta, + attributes: Attributes::new(), + }); + } + + let range_header = options.range.as_ref().map(get_range_to_header); + let (content, headers, status) = self.get_file(path_str, range_header.as_deref()).await?; + + let (range, total_size) = if let Some(requested) = &options.range { + // A ranged request that comes back non-partial (a 200 with the full body) must not be + // silently treated as the requested slice. Mirror object_store's `NotPartial` behavior. + if status != reqwest::StatusCode::PARTIAL_CONTENT { + return Err(generic_error(format!( + "ranged GET for `{path_str}` returned a non-partial response (HTTP {status}); \ + expected 206 Partial Content" + ))); + } + let content_range = headers + .get(reqwest::header::CONTENT_RANGE) + .and_then(|v| v.to_str().ok()) + .ok_or_else(|| { + generic_error(format!( + "ranged GET for `{path_str}` is missing a Content-Range header" + )) + })?; + let (actual, total) = parse_content_range(content_range)?; + let expected = requested + .as_range(total) + .map_err(|e| generic_error(format!("invalid range for `{path_str}`: {e}")))?; + if actual != expected { + return Err(generic_error(format!( + "ranged GET for `{path_str}` returned unexpected Content-Range \ + `{content_range}`; expected bytes {expected:?}" + ))); + } + (actual, total) + } else { + // Derive byte range + total size from Content-Range (if present) or the body length. + match headers + .get(reqwest::header::CONTENT_RANGE) + .and_then(|v| v.to_str().ok()) + { + Some(cr) => parse_content_range(cr)?, + None => (0..content.len() as u64, content.len() as u64), + } + }; + + let last_modified = parse_last_modified(&headers); + let e_tag = parse_etag(&headers); + + let meta = ObjectMeta { + location: location.clone(), + last_modified, + size: total_size, + e_tag, + version: None, + }; + // Enforce client-side conditional preconditions (if_match / if_none_match / + // if_modified_since / if_unmodified_since) against the resolved metadata. + options.check_preconditions(&meta)?; + + let stream = Box::pin(futures::stream::once(futures::future::ready(Ok(content)))); + Ok(GetResult { + payload: GetResultPayload::Stream(stream), + meta, + range, + attributes: Attributes::new(), + }) + } + + async fn put_opts( + &self, + location: &Path, + payload: PutPayload, + opts: PutOptions, + ) -> ObjectStoreResult { + // Update (a conditional/compare-and-set put) is not supported: Delta's commit path never + // issues it, and the REST file API has no compare-and-set primitive to back it. + let overwrite = match opts.mode { + PutMode::Overwrite => true, + PutMode::Create => false, + PutMode::Update(_) => return Err(not_supported("PutMode::Update")), + }; + let path_str = location.as_ref(); + let url = self.config.file_url(&self.base_url, path_str); + let query = self.config.put_query(overwrite); + let body: Bytes = payload.into(); + // A non-idempotent PUT is only safe to retry when we can verify the write landed, so the + // verify path is gated on `Create` + verification enabled. + if !overwrite && self.verify_on_ambiguous { + return self.put_create_verified(path_str, &url, &query, body).await; + } + let response = self + .client + .put(&url) + .query(&query) + .headers(self.headers()?) + .body(body) + .send() + .await + .map_err(generic_error)?; + ensure_success_response(response, path_str)?; + Ok(put_result()) + } + + fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, ObjectStoreResult> { + let prefix = prefix.map(|p| p.as_ref().to_string()).unwrap_or_default(); + // Both `list` and `list_with_offset` recurse; only `list_with_delimiter` is non-recursive. + self.list_paginated(prefix, None, None, true) + } + + fn list_with_offset( + &self, + prefix: Option<&Path>, + offset: &Path, + ) -> BoxStream<'static, ObjectStoreResult> { + let prefix = prefix.map(|p| p.as_ref().to_string()).unwrap_or_default(); + // `offset` is a full path beginning with `prefix`; the REST list offset is relative to + // the directory being listed, so send only the leaf portion. The full `offset` is kept + // to enforce exclusivity client-side. + let offset_str = { + let raw = offset.as_ref(); + if !prefix.is_empty() && raw.starts_with(&prefix) { + raw[prefix.len()..].trim_start_matches('/').to_string() + } else { + raw.to_string() + } + }; + self.list_paginated(prefix, Some(offset_str), Some(offset.clone()), true) + } + + fn delete_stream( + &self, + locations: BoxStream<'static, ObjectStoreResult>, + ) -> BoxStream<'static, ObjectStoreResult> { + let store = self.clone(); + Box::pin(locations.then(move |location| { + let store = store.clone(); + async move { + let location = location?; + store.delete_one(&location).await?; + Ok(location) + } + })) + } + + // === Operations Delta never issues against a REST file store === + // These return NotSupported. + + async fn list_with_delimiter(&self, _prefix: Option<&Path>) -> ObjectStoreResult { + Err(not_supported("list_with_delimiter")) + } + + async fn put_multipart_opts( + &self, + _location: &Path, + _opts: PutMultipartOptions, + ) -> ObjectStoreResult> { + Err(not_supported("multipart upload")) + } + + async fn copy_opts( + &self, + _from: &Path, + _to: &Path, + _options: CopyOptions, + ) -> ObjectStoreResult<()> { + Err(not_supported("copy")) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn get_range_to_header_formats_bounded() { + assert_eq!(get_range_to_header(&GetRange::Bounded(2..6)), "bytes=2-5"); + } + + #[test] + fn get_range_to_header_formats_offset() { + assert_eq!(get_range_to_header(&GetRange::Offset(10)), "bytes=10-"); + } + + #[test] + fn get_range_to_header_formats_suffix() { + assert_eq!(get_range_to_header(&GetRange::Suffix(512)), "bytes=-512"); + } + + #[test] + fn parse_content_range_accepts_valid() { + let (range, total) = parse_content_range("bytes 2-5/10").unwrap(); + assert_eq!(range, 2..6); + assert_eq!(total, 10); + } + + #[test] + fn parse_content_range_rejects_reversed_range() { + // `end < start` would underflow in GetResult::bytes(); it must be an error. + assert!(parse_content_range("bytes 5-2/10").is_err()); + } + + #[test] + fn parse_content_range_rejects_start_past_total() { + assert!(parse_content_range("bytes 20-25/10").is_err()); + } +} diff --git a/default-engine/src/rest_store/tests.rs b/default-engine/src/rest_store/tests.rs new file mode 100644 index 0000000000..05f92a5348 --- /dev/null +++ b/default-engine/src/rest_store/tests.rs @@ -0,0 +1,1050 @@ +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::Arc; + +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::{ + Error as ObjectStoreError, GetOptions, GetRange, ObjectStore, ObjectStoreExt, PutMode, + PutOptions, PutPayload, Result as ObjectStoreResult, UpdateVersion, +}; +use futures::StreamExt as _; +use reqwest::header::HeaderMap; +use reqwest::Client; +use wiremock::matchers::{header, method, path}; +use wiremock::{Mock, MockServer, ResponseTemplate}; + +use super::*; + +fn token_producer( + calls: Arc, + ttl: Option, +) -> RefreshingHeaderProvider { + RefreshingHeaderProvider::new(move || { + let n = calls.fetch_add(1, Ordering::SeqCst); + let headers = + headers_from_pairs([("authorization".to_string(), format!("Bearer token-{n}"))])?; + Ok((headers, ttl)) + }) +} + +/// Without a TTL, [`RefreshingHeaderProvider`] produces on every call -- refreshed per request. +#[test] +fn refreshing_header_provider_without_ttl_produces_each_call() { + let calls = Arc::new(AtomicUsize::new(0)); + let provider = token_producer(calls.clone(), None); + let first = provider.headers().unwrap(); + let second = provider.headers().unwrap(); + assert_eq!(calls.load(Ordering::SeqCst), 2); + assert_eq!(first.get("authorization").unwrap(), "Bearer token-0"); + assert_eq!(second.get("authorization").unwrap(), "Bearer token-1"); +} + +/// With a TTL, headers are cached and the closure runs once until the TTL nears expiry. +#[test] +fn refreshing_header_provider_with_ttl_caches() { + let calls = Arc::new(AtomicUsize::new(0)); + let provider = token_producer(calls.clone(), Some(std::time::Duration::from_secs(3600))); + let first = provider.headers().unwrap(); + let second = provider.headers().unwrap(); + assert_eq!(calls.load(Ordering::SeqCst), 1); + assert_eq!(first.get("authorization").unwrap(), "Bearer token-0"); + assert_eq!(second.get("authorization").unwrap(), "Bearer token-0"); +} + +fn test_config() -> RestEndpointConfig { + RestEndpointConfig { + files_prefix: "files".into(), + directories_prefix: "dirs".into(), + page_token_param: "page_token".into(), + start_from_param: "start_from".into(), + recursive_param: "recursive".into(), + overwrite_param: "overwrite".into(), + contents_field: "contents".into(), + next_page_token_field: "next_page_token".into(), + entry_path_field: "path".into(), + entry_size_field: "size".into(), + entry_is_directory_field: "is_directory".into(), + entry_last_modified_field: "last_modified".into(), + entry_strip_prefix: None, + } +} + +#[test] +fn join_url_omits_empty_prefix_segment() { + let mut config = test_config(); + config.files_prefix.clear(); + config.directories_prefix.clear(); + assert_eq!( + config.file_url("http://localhost", "a.txt"), + "http://localhost/a.txt" + ); + assert_eq!( + config.directory_url("http://localhost", "d"), + "http://localhost/d" + ); +} + +#[test] +fn join_url_keeps_nonempty_prefix() { + let config = test_config(); + assert_eq!( + config.file_url("http://localhost/", "a.txt"), + "http://localhost/files/a.txt" + ); +} + +#[test] +fn parse_list_rejects_non_integer_last_modified() { + let config = test_config(); + let body = r#"{"contents":[{"path":"d/f1","size":1,"last_modified":"not-a-number"}]}"#; + assert!(matches!( + config.parse_list(body.as_bytes()), + Err(ObjectStoreError::Generic { .. }) + )); +} + +fn store_for(server: &MockServer, headers: HeaderMap) -> RestObjectStore { + RestObjectStore::new( + server.uri(), + Client::new(), + Arc::new(StaticHeaderProvider::new(headers)), + Arc::new(test_config()), + ) +} + +#[tokio::test] +async fn get_returns_body() { + let server = MockServer::start().await; + Mock::given(method("GET")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(200).set_body_bytes(b"hello".as_slice())) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()); + let bytes = store + .get(&Path::from("a.txt")) + .await + .unwrap() + .bytes() + .await + .unwrap(); + assert_eq!(&bytes[..], b"hello"); +} + +/// A malformed `Content-Range` surfaces as an error rather than silently degrading the +/// reported range/size (matches `object_store`'s own client). +#[tokio::test] +async fn get_malformed_content_range_yields_error() { + let server = MockServer::start().await; + Mock::given(method("GET")) + .and(path("/files/a.txt")) + .respond_with( + ResponseTemplate::new(200) + .insert_header("content-range", "bytes not-a-range") + .set_body_bytes(b"hello".as_slice()), + ) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()); + let err = store.get(&Path::from("a.txt")).await.unwrap_err(); + assert!( + matches!(err, ObjectStoreError::Generic { .. }), + "got {err:?}" + ); +} + +/// A ranged GET whose server ignores the range and returns a non-partial `200` with the full +/// body must error rather than silently treating the whole body as the requested slice (mirrors +/// object_store's `NotPartial` behavior). +#[tokio::test] +async fn ranged_get_non_partial_response_yields_error() { + let server = MockServer::start().await; + Mock::given(method("GET")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(200).set_body_bytes(b"full-body".as_slice())) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()); + let err = store + .get_opts( + &Path::from("a.txt"), + GetOptions { + range: Some((0..4).into()), + ..Default::default() + }, + ) + .await + .unwrap_err(); + assert!( + matches!(err, ObjectStoreError::Generic { .. }), + "got {err:?}" + ); +} + +#[tokio::test] +async fn ranged_get_unexpected_content_range_yields_error() { + let server = MockServer::start().await; + Mock::given(method("GET")) + .and(path("/files/a.txt")) + .respond_with( + ResponseTemplate::new(206) + .insert_header("content-range", "bytes 5-8/9") + .set_body_bytes(b"body".as_slice()), + ) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()); + let err = store + .get_opts( + &Path::from("a.txt"), + GetOptions { + range: Some((0..4).into()), + ..Default::default() + }, + ) + .await + .unwrap_err(); + assert!( + matches!(err, ObjectStoreError::Generic { .. }), + "got {err:?}" + ); +} + +#[tokio::test] +async fn head_uses_http_head_and_returns_meta_without_body() { + let server = MockServer::start().await; + Mock::given(method("HEAD")) + .and(path("/files/a.txt")) + .respond_with( + ResponseTemplate::new(200) + .insert_header("content-length", "42") + .insert_header("etag", "\"abc\""), + ) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()); + let res = store + .get_opts( + &Path::from("a.txt"), + GetOptions { + head: true, + ..Default::default() + }, + ) + .await + .unwrap(); + assert_eq!(res.meta.size, 42); + assert_eq!(res.meta.e_tag.as_deref(), Some("\"abc\"")); + // head must not stream a body. + let body = res.bytes().await.unwrap(); + assert!(body.is_empty()); +} + +#[tokio::test] +async fn head_missing_content_length_yields_error() { + let server = MockServer::start().await; + Mock::given(method("HEAD")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(200)) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()); + let err = store + .get_opts( + &Path::from("a.txt"), + GetOptions { + head: true, + ..Default::default() + }, + ) + .await + .unwrap_err(); + assert!( + matches!(err, ObjectStoreError::Generic { .. }), + "got {err:?}" + ); +} + +#[tokio::test] +async fn put_overwrite_succeeds() { + let server = MockServer::start().await; + Mock::given(method("PUT")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(204)) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()); + store + .put(&Path::from("a.txt"), PutPayload::from_static(b"hi")) + .await + .unwrap(); +} + +#[tokio::test] +async fn put_create_conflict_maps_to_already_exists() { + let server = MockServer::start().await; + Mock::given(method("PUT")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(409)) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()); + let err = store + .put_opts( + &Path::from("a.txt"), + PutPayload::from_static(b"hi"), + PutOptions { + mode: PutMode::Create, + ..Default::default() + }, + ) + .await + .unwrap_err(); + assert!(matches!(err, ObjectStoreError::AlreadyExists { .. })); +} + +#[tokio::test] +async fn put_update_is_not_supported() { + let server = MockServer::start().await; + let store = store_for(&server, HeaderMap::new()); + let err = store + .put_opts( + &Path::from("a.txt"), + PutPayload::from_static(b"hi"), + PutOptions { + mode: PutMode::Update(UpdateVersion { + e_tag: Some("v1".to_string()), + version: None, + }), + ..Default::default() + }, + ) + .await + .unwrap_err(); + assert!(matches!(err, ObjectStoreError::NotSupported { .. })); +} + +#[tokio::test] +async fn list_paginates_across_pages() { + let server = MockServer::start().await; + // First page (no page_token) returns one file + a token; second page returns two. + Mock::given(method("GET")) + .and(path("/dirs/d")) + .and(wiremock::matchers::query_param_is_missing("page_token")) + .respond_with( + ResponseTemplate::new(200).set_body_string( + r#"{"contents":[{"path":"d/1","size":1}],"next_page_token":"p2"}"#, + ), + ) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/dirs/d")) + .and(wiremock::matchers::query_param("page_token", "p2")) + .respond_with( + ResponseTemplate::new(200).set_body_string( + r#"{"contents":[{"path":"d/2","size":2},{"path":"d/3","size":3}]}"#, + ), + ) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()); + let metas: Vec<_> = store + .list(Some(&Path::from("d"))) + .collect::>() + .await + .into_iter() + .map(|r| r.unwrap()) + .collect(); + let paths: Vec<_> = metas + .iter() + .map(|m| m.location.as_ref().to_string()) + .collect(); + assert_eq!(paths, vec!["d/1", "d/2", "d/3"]); +} + +#[tokio::test] +async fn list_missing_directory_is_empty() { + let server = MockServer::start().await; + Mock::given(method("GET")) + .and(path("/dirs/missing")) + .respond_with(ResponseTemplate::new(404)) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()); + let metas: Vec<_> = store + .list(Some(&Path::from("missing"))) + .collect::>() + .await; + assert!(metas.is_empty()); +} + +/// A first-page 404 lists as empty, but a 404 *mid-pagination* (page_token set) means the +/// listing was truncated -- it must surface as an error, not a silent partial result. +#[tokio::test] +async fn list_not_found_mid_pagination_yields_error() { + let server = MockServer::start().await; + Mock::given(method("GET")) + .and(path("/dirs/d")) + .and(wiremock::matchers::query_param_is_missing("page_token")) + .respond_with( + ResponseTemplate::new(200).set_body_string( + r#"{"contents":[{"path":"d/1","size":1}],"next_page_token":"p2"}"#, + ), + ) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/dirs/d")) + .and(wiremock::matchers::query_param("page_token", "p2")) + .respond_with(ResponseTemplate::new(404)) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()); + let results: Vec<_> = store.list(Some(&Path::from("d"))).collect::>().await; + // First-page entry is yielded, then the mid-pagination 404 surfaces as an error. + assert_eq!(results.len(), 2); + assert!(results[0].is_ok()); + assert!( + matches!(results[1], Err(ObjectStoreError::NotFound { .. })), + "got {:?}", + results[1] + ); +} + +#[tokio::test] +async fn list_empty_next_page_token_ends_listing() { + let server = MockServer::start().await; + Mock::given(method("GET")) + .and(path("/dirs/d")) + .respond_with( + ResponseTemplate::new(200) + .set_body_string(r#"{"contents":[{"path":"d/1","size":1}],"next_page_token":""}"#), + ) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()); + let metas: Vec<_> = store + .list(Some(&Path::from("d"))) + .collect::>() + .await + .into_iter() + .map(|r| r.unwrap()) + .collect(); + assert_eq!(metas.len(), 1); + assert_eq!(metas[0].location.as_ref(), "d/1"); +} + +#[tokio::test] +async fn list_rejects_non_integer_size() { + let server = MockServer::start().await; + Mock::given(method("GET")) + .and(path("/dirs/d")) + .respond_with( + ResponseTemplate::new(200) + .set_body_string(r#"{"contents":[{"path":"d/1","size":"big"}]}"#), + ) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()); + let err = store + .list(Some(&Path::from("d"))) + .next() + .await + .unwrap() + .unwrap_err(); + assert!( + matches!(err, ObjectStoreError::Generic { .. }), + "got {err:?}" + ); +} + +#[tokio::test] +async fn get_empty_bounded_range_is_invalid() { + let server = MockServer::start().await; + let store = store_for(&server, HeaderMap::new()); + let err = store + .get_opts( + &Path::from("a.txt"), + GetOptions { + range: Some(GetRange::Bounded(0..0)), + ..Default::default() + }, + ) + .await + .unwrap_err(); + assert!( + matches!(err, ObjectStoreError::Generic { .. }), + "got {err:?}" + ); + assert!(err + .to_string() + .contains("Range started at 0 and ended at 0")); +} + +/// `list_with_offset` must exclude the offset entry itself even if the backend echoes it +/// back, so log replay never re-ingests the offset commit. +#[tokio::test] +async fn list_with_offset_excludes_offset_entry() { + let server = MockServer::start().await; + Mock::given(method("GET")) + .and(path("/dirs/d")) + .respond_with( + ResponseTemplate::new(200).set_body_string( + r#"{"contents":[{"path":"d/2","size":2},{"path":"d/3","size":3}]}"#, + ), + ) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()); + let paths: Vec<_> = store + .list_with_offset(Some(&Path::from("d")), &Path::from("d/2")) + .collect::>() + .await + .into_iter() + .map(|r| r.unwrap().location.as_ref().to_string()) + .collect(); + assert_eq!(paths, vec!["d/3"]); +} + +/// An out-of-order page violates the sorted-listing contract and must surface as an error +/// rather than feeding log replay a misordered listing. +#[tokio::test] +async fn list_out_of_order_entries_yields_error() { + let server = MockServer::start().await; + Mock::given(method("GET")) + .and(path("/dirs/d")) + .respond_with( + ResponseTemplate::new(200).set_body_string( + r#"{"contents":[{"path":"d/3","size":3},{"path":"d/1","size":1}]}"#, + ), + ) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()); + let results: Vec<_> = store.list(Some(&Path::from("d"))).collect::>().await; + assert_eq!(results.len(), 2); + assert_eq!(results[0].as_ref().unwrap().location.as_ref(), "d/3"); + assert!(matches!(results[1], Err(ObjectStoreError::Generic { .. }))); +} + +/// The sorted-listing contract holds *across* pages: a later page whose first entry sorts before +/// the previous page's last entry must surface as an error, not feed log replay a misordered list. +#[tokio::test] +async fn list_out_of_order_across_pages_yields_error() { + let server = MockServer::start().await; + Mock::given(method("GET")) + .and(path("/dirs/d")) + .and(wiremock::matchers::query_param_is_missing("page_token")) + .respond_with( + ResponseTemplate::new(200).set_body_string( + r#"{"contents":[{"path":"d/3","size":3}],"next_page_token":"p2"}"#, + ), + ) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/dirs/d")) + .and(wiremock::matchers::query_param("page_token", "p2")) + .respond_with( + ResponseTemplate::new(200).set_body_string(r#"{"contents":[{"path":"d/1","size":1}]}"#), + ) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()); + let results: Vec<_> = store.list(Some(&Path::from("d"))).collect::>().await; + assert_eq!(results.len(), 2); + assert_eq!(results[0].as_ref().unwrap().location.as_ref(), "d/3"); + assert!( + matches!(results[1], Err(ObjectStoreError::Generic { .. })), + "got {:?}", + results[1] + ); +} + +/// An unparseable entry path fails the whole page rather than being silently dropped, consistent +/// with the module's fail-loudly philosophy. +#[test] +fn parse_list_rejects_unparseable_entry_path() { + let config = test_config(); + // A `\0` byte in the path is not a valid object-store path. + let body = "{\"contents\":[{\"path\":\"d/\\u0000bad\",\"size\":1}]}"; + assert!(matches!( + config.parse_list(body.as_bytes()), + Err(ObjectStoreError::Generic { .. }) + )); +} + +#[tokio::test] +async fn auth_headers_are_sent() { + let server = MockServer::start().await; + // Only matches when the provider's header is present. + Mock::given(method("GET")) + .and(path("/files/a.txt")) + .and(header("x-test-auth", "secret")) + .respond_with(ResponseTemplate::new(200).set_body_bytes(b"ok".as_slice())) + .mount(&server) + .await; + let mut headers = HeaderMap::new(); + headers.insert( + "x-test-auth", + reqwest::header::HeaderValue::from_static("secret"), + ); + let store = store_for(&server, headers); + let bytes = store + .get(&Path::from("a.txt")) + .await + .unwrap() + .bytes() + .await + .unwrap(); + assert_eq!(&bytes[..], b"ok"); +} + +#[tokio::test] +async fn unsupported_operations_error_not_panic() { + let server = MockServer::start().await; + let store = store_for(&server, HeaderMap::new()); + assert!(matches!( + store + .copy(&Path::from("a"), &Path::from("b")) + .await + .unwrap_err(), + ObjectStoreError::NotSupported { .. } + )); + assert!(matches!( + store + .list_with_delimiter(Some(&Path::from("d"))) + .await + .unwrap_err(), + ObjectStoreError::NotSupported { .. } + )); +} + +#[tokio::test] +async fn config_driven_list_parses_and_reads() { + let server = MockServer::start().await; + let config = RestEndpointConfig { + files_prefix: "api/2.0/fs/files".into(), + directories_prefix: "api/2.0/fs/directories".into(), + page_token_param: "page_token".into(), + start_from_param: "start_from".into(), + recursive_param: "recursive".into(), + overwrite_param: "overwrite".into(), + contents_field: "contents".into(), + next_page_token_field: "next_page_token".into(), + entry_path_field: "path".into(), + entry_size_field: "file_size".into(), + entry_is_directory_field: "is_directory".into(), + entry_last_modified_field: "last_modified".into(), + entry_strip_prefix: None, + }; + // Listing with one file and one subdirectory; the directory entry must be filtered out. + Mock::given(method("GET")) + .and(path("/api/2.0/fs/directories/d")) + .respond_with(ResponseTemplate::new(200).set_body_string( + r#"{"contents":[{"path":"d/f1","file_size":7,"is_directory":false,"last_modified":1000},{"path":"d/sub","is_directory":true}]}"#, + )) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/api/2.0/fs/files/d/f1")) + .respond_with(ResponseTemplate::new(200).set_body_bytes(b"content".as_slice())) + .mount(&server) + .await; + let store = RestObjectStore::new( + server.uri(), + Client::new(), + Arc::new(StaticHeaderProvider::new(HeaderMap::new())), + Arc::new(config), + ); + let metas: Vec<_> = store + .list(Some(&Path::from("d"))) + .collect::>() + .await + .into_iter() + .map(|r| r.unwrap()) + .collect(); + assert_eq!(metas.len(), 1); + assert_eq!(metas[0].location.as_ref(), "d/f1"); + assert_eq!(metas[0].size, 7); + assert_eq!(metas[0].last_modified.timestamp_millis(), 1000); + let bytes = store + .get(&Path::from("d/f1")) + .await + .unwrap() + .bytes() + .await + .unwrap(); + assert_eq!(&bytes[..], b"content"); +} + +#[tokio::test] +async fn delete_issues_http_delete() { + let server = MockServer::start().await; + Mock::given(method("DELETE")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(204)) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()); + store.delete(&Path::from("a.txt")).await.unwrap(); +} + +fn store_with_strip_prefix(server: &MockServer, prefix: &str) -> RestObjectStore { + let config = RestEndpointConfig { + directories_prefix: "dirs".into(), + entry_strip_prefix: Some(prefix.into()), + entry_size_field: "file_size".into(), + ..test_config() + }; + RestObjectStore::new( + server.uri(), + Client::new(), + Arc::new(StaticHeaderProvider::new(HeaderMap::new())), + Arc::new(config), + ) +} + +#[tokio::test] +async fn list_strips_entry_prefix() { + let server = MockServer::start().await; + Mock::given(method("GET")) + .and(path("/dirs/d")) + .respond_with( + ResponseTemplate::new(200) + .set_body_string(r#"{"contents":[{"path":"/TablesById/u/d/f1","file_size":1}]}"#), + ) + .mount(&server) + .await; + let store = store_with_strip_prefix(&server, "/TablesById/u"); + let metas: Vec<_> = store + .list(Some(&Path::from("d"))) + .collect::>() + .await + .into_iter() + .map(|r| r.unwrap()) + .collect(); + assert_eq!(metas.len(), 1); + assert_eq!(metas[0].location.as_ref(), "d/f1"); +} + +#[tokio::test] +async fn list_rejects_entry_outside_strip_prefix() { + let server = MockServer::start().await; + Mock::given(method("GET")) + .and(path("/dirs/d")) + .respond_with( + ResponseTemplate::new(200).set_body_string(r#"{"contents":[{"path":"/other/x"}]}"#), + ) + .mount(&server) + .await; + let store = store_with_strip_prefix(&server, "/TablesById/u"); + let results: Vec<_> = store.list(Some(&Path::from("d"))).collect::>().await; + assert!(matches!( + results.as_slice(), + [Err(ObjectStoreError::Generic { .. })] + )); +} + +#[tokio::test] +async fn get_retries_transient_5xx() { + let server = MockServer::start().await; + Mock::given(method("GET")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(503)) + .up_to_n_times(1) + .with_priority(1) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(200).set_body_bytes(b"ok".as_slice())) + .with_priority(2) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()).with_max_retries(1); + let bytes = store + .get(&Path::from("a.txt")) + .await + .unwrap() + .bytes() + .await + .unwrap(); + assert_eq!(&bytes[..], b"ok"); +} + +#[tokio::test] +async fn put_create_verified_first_attempt_409_is_already_exists() { + let server = MockServer::start().await; + Mock::given(method("PUT")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(409)) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()).with_verify_on_ambiguous(true); + let err = store + .put_opts( + &Path::from("a.txt"), + PutPayload::from_static(b"hi"), + PutOptions { + mode: PutMode::Create, + ..Default::default() + }, + ) + .await + .unwrap_err(); + assert!(matches!(err, ObjectStoreError::AlreadyExists { .. })); +} + +/// A transient read-back failure during verified create propagates once retries are exhausted. +#[tokio::test] +async fn put_create_verified_read_back_error_propagates() { + let server = MockServer::start().await; + Mock::given(method("PUT")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(500)) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(503)) + .mount(&server) + .await; + let err = store_for(&server, HeaderMap::new()) + .with_verify_on_ambiguous(true) + .put_opts( + &Path::from("a.txt"), + PutPayload::from_static(b"hi"), + PutOptions { + mode: PutMode::Create, + ..Default::default() + }, + ) + .await + .unwrap_err(); + assert!(matches!(err, ObjectStoreError::Generic { .. })); +} + +/// PUT returns 500; the verified Create then reads back via `read_back`, with the read-back +/// response supplied by `get`. `max_retries` is 0, so an absent read-back fails immediately. +async fn put_create_with_readback( + server: &MockServer, + get: ResponseTemplate, +) -> ObjectStoreResult<()> { + Mock::given(method("PUT")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(500)) + .mount(server) + .await; + Mock::given(method("GET")) + .and(path("/files/a.txt")) + .respond_with(get) + .mount(server) + .await; + store_for(server, HeaderMap::new()) + .with_verify_on_ambiguous(true) + .put_opts( + &Path::from("a.txt"), + PutPayload::from_static(b"hi"), + PutOptions { + mode: PutMode::Create, + ..Default::default() + }, + ) + .await + .map(|_| ()) +} + +/// An ambiguous 5xx whose read-back matches what we wrote is treated as a success. +#[tokio::test] +async fn put_create_verified_matching_write_succeeds() { + let server = MockServer::start().await; + let get = ResponseTemplate::new(200).set_body_bytes(b"hi".as_slice()); + put_create_with_readback(&server, get).await.unwrap(); +} + +/// An ambiguous 5xx whose read-back differs is a real conflict. +#[tokio::test] +async fn put_create_verified_conflicting_write_is_already_exists() { + let server = MockServer::start().await; + let get = ResponseTemplate::new(200).set_body_bytes(b"other".as_slice()); + let err = put_create_with_readback(&server, get).await.unwrap_err(); + assert!(matches!(err, ObjectStoreError::AlreadyExists { .. })); +} + +/// An ambiguous 5xx whose read-back is absent (write never landed) fails to confirm. +#[tokio::test] +async fn put_create_verified_absent_write_fails_to_confirm() { + let server = MockServer::start().await; + let err = put_create_with_readback(&server, ResponseTemplate::new(404)) + .await + .unwrap_err(); + assert!(matches!(err, ObjectStoreError::Generic { .. })); +} + +/// Exhausting retries on persistent 5xx surfaces an error rather than looping or succeeding. +#[tokio::test] +async fn get_retry_exhaustion_surfaces_error() { + let server = MockServer::start().await; + Mock::given(method("GET")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(503)) + .mount(&server) + .await; + let store = store_for(&server, HeaderMap::new()).with_max_retries(1); + let err = store.get(&Path::from("a.txt")).await.unwrap_err(); + assert!(matches!(err, ObjectStoreError::Generic { .. })); +} + +/// A verified Create retries while the read-back is absent, then succeeds once it lands. +#[tokio::test] +async fn put_create_verified_retries_until_write_lands() { + let server = MockServer::start().await; + Mock::given(method("PUT")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(500)) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(404)) + .up_to_n_times(1) + .with_priority(1) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(200).set_body_bytes(b"hi".as_slice())) + .with_priority(2) + .mount(&server) + .await; + store_for(&server, HeaderMap::new()) + .with_verify_on_ambiguous(true) + .with_max_retries(1) + .put_opts( + &Path::from("a.txt"), + PutPayload::from_static(b"hi"), + PutOptions { + mode: PutMode::Create, + ..Default::default() + }, + ) + .await + .unwrap(); +} + +/// On a retry, a `409` is the writer's own already-landed write rather than a foreign conflict: +/// attempt1 PUT 500 with an absent read-back consumes a retry, then attempt2 PUT 409 reconciles +/// via read-back to matching bytes and succeeds (not `AlreadyExists`). +#[tokio::test] +async fn put_create_verified_retry_conflict_matches_own_write_succeeds() { + let server = MockServer::start().await; + // Attempt 1 PUT -> 500 (ambiguous); attempt 2 PUT -> 409 (our own landed write). + Mock::given(method("PUT")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(500)) + .up_to_n_times(1) + .with_priority(1) + .mount(&server) + .await; + Mock::given(method("PUT")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(409)) + .with_priority(2) + .mount(&server) + .await; + // Read-back after attempt 1 -> 404 (absent, consume a retry); after attempt 2 -> matching body. + Mock::given(method("GET")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(404)) + .up_to_n_times(1) + .with_priority(1) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(200).set_body_bytes(b"hi".as_slice())) + .with_priority(2) + .mount(&server) + .await; + store_for(&server, HeaderMap::new()) + .with_verify_on_ambiguous(true) + .with_max_retries(2) + .put_opts( + &Path::from("a.txt"), + PutPayload::from_static(b"hi"), + PutOptions { + mode: PutMode::Create, + ..Default::default() + }, + ) + .await + .unwrap(); +} + +#[tokio::test] +async fn delete_missing_maps_to_not_found() { + let server = MockServer::start().await; + Mock::given(method("DELETE")) + .and(path("/files/a.txt")) + .respond_with(ResponseTemplate::new(404)) + .mount(&server) + .await; + let err = store_for(&server, HeaderMap::new()) + .delete(&Path::from("a.txt")) + .await + .unwrap_err(); + assert!(matches!(err, ObjectStoreError::NotFound { .. })); +} + +#[tokio::test] +async fn delete_stream_deletes_each_path() { + let server = MockServer::start().await; + for p in ["a", "b"] { + Mock::given(method("DELETE")) + .and(path(format!("/files/{p}"))) + .respond_with(ResponseTemplate::new(204)) + .mount(&server) + .await; + } + let store = store_for(&server, HeaderMap::new()); + let paths = futures::stream::iter([Ok(Path::from("a")), Ok(Path::from("b"))]).boxed(); + let deleted: Vec<_> = store.delete_stream(paths).collect::>().await; + assert_eq!(deleted.len(), 2); + assert!(deleted.iter().all(|r| r.is_ok())); +} + +#[test] +fn build_rest_client_rejects_partial_mtls() { + let opts = RestClientOptions { + cert_path: Some("/x/cert.pem".into()), + ..Default::default() + }; + assert!(build_rest_client(&opts).is_err()); +} + +#[test] +fn build_rest_client_rejects_malformed_dns_override() { + let missing_eq = RestClientOptions { + dns_overrides: vec!["no-equals".into()], + ..Default::default() + }; + assert!(build_rest_client(&missing_eq).is_err()); + let bad_addr = RestClientOptions { + dns_overrides: vec!["host=not-an-addr".into()], + ..Default::default() + }; + assert!(build_rest_client(&bad_addr).is_err()); +} + +#[test] +fn build_rest_client_plain_and_valid_dns_override_succeed() { + assert!(build_rest_client(&RestClientOptions::default()).is_ok()); + let opts = RestClientOptions { + // Empty entries are skipped; a valid `host=ip:port` is accepted. + dns_overrides: vec!["".into(), "example.com=127.0.0.1:8443".into()], + timeout_secs: Some(5), + ..Default::default() + }; + assert!(build_rest_client(&opts).is_ok()); +} diff --git a/default-engine/src/stats.rs b/default-engine/src/stats.rs new file mode 100644 index 0000000000..3ebc0b13a8 --- /dev/null +++ b/default-engine/src/stats.rs @@ -0,0 +1,2855 @@ +//! Statistics collection for Delta Lake file writes: min, max, and null count per column. +//! +//! [`FileStatsAccumulator`] merges per-row-group statistics into one file-level statistics struct. + +use std::borrow::Cow; +use std::sync::Arc; + +use delta_kernel::actions::{MAX_VALUES, MIN_VALUES, NULL_COUNT, NUM_RECORDS, TIGHT_BOUNDS}; +use delta_kernel::arrow::array::{ + new_null_array, Array, ArrayRef, AsArray, BooleanArray, Decimal128Array, Int64Array, + LargeStringArray, PrimitiveArray, RecordBatch, StringArray, StringViewArray, StructArray, +}; +use delta_kernel::arrow::compute::concat; +use delta_kernel::arrow::compute::kernels::aggregate::{ + bool_and, max, max_string, min, min_string, sum_checked, +}; +use delta_kernel::arrow::datatypes::{ + ArrowPrimitiveType, DataType, Date32Type, Date64Type, Decimal128Type, Field, Float32Type, + Float64Type, Int16Type, Int32Type, Int64Type, Int8Type, SchemaRef as ArrowSchemaRef, TimeUnit, + TimestampMicrosecondType, TimestampMillisecondType, TimestampNanosecondType, + TimestampSecondType, UInt16Type, UInt32Type, UInt64Type, UInt8Type, +}; +use delta_kernel::column_trie::ColumnTrie; +use delta_kernel::engine::arrow_utils::fix_nested_null_masks; +use delta_kernel::expressions::ColumnName; +use delta_kernel::schema::{DataType as KernelDataType, StructType}; +use delta_kernel::{DeltaResult, Error}; + +/// Maximum prefix length for string statistics (Delta protocol requirement). +const STRING_PREFIX_LENGTH: usize = 32; + +/// Maximum expansion when searching for a valid max truncation point. +const STRING_EXPANSION_LIMIT: usize = STRING_PREFIX_LENGTH * 2; + +/// ASCII DEL character (0x7F) - used as tie-breaker for max values when truncated char is ASCII. +const ASCII_MAX_CHAR: char = '\x7F'; + +/// Maximum Unicode code point - used as tie-breaker for max values when truncated char is +/// non-ASCII. +const UTF8_MAX_CHAR: char = '\u{10FFFF}'; + +// ============================================================================ +// String truncation for Delta statistics +// ============================================================================ + +/// Truncate a string for min statistics. +/// +/// For min values, we simply truncate at the prefix length. The truncated value will always +/// be <= the original, which is correct for min statistics. +/// +/// Returns the original string if it's already within the limit. +fn truncate_min_string(s: &str) -> &str { + if s.len() <= STRING_PREFIX_LENGTH { + return s; + } + // Find char boundary at or before STRING_PREFIX_LENGTH + let end = s + .char_indices() + .take(STRING_PREFIX_LENGTH + 1) + .last() + .map(|(i, _)| i) + .unwrap_or(s.len()); + + // Take exactly STRING_PREFIX_LENGTH chars + let truncated_end = s + .char_indices() + .nth(STRING_PREFIX_LENGTH) + .map(|(i, _)| i) + .unwrap_or(end); + + &s[..truncated_end] +} + +/// Truncate a string for max statistics. +/// +/// For max values, we need to ensure the truncated value is >= all actual values in the column. +/// We do this by appending a "tie-breaker" character after truncation: +/// - ASCII_MAX_CHAR (0x7F) if the character at the truncation point is ASCII (< 0x7F) +/// - UTF8_MAX_CHAR (U+10FFFF) otherwise +/// +/// This ensures correct data skipping behavior: any string starting with the truncated prefix +/// will compare <= the truncated max + tie-breaker. +/// +/// Returns `Cow::Borrowed` if no truncation needed (avoiding allocation), `Cow::Owned` when +/// truncation is performed, or `None` if the string is too long to truncate safely. +fn truncate_max_string(s: &str) -> Option> { + if s.len() <= STRING_PREFIX_LENGTH { + return Some(Cow::Borrowed(s)); + } + + // Start at STRING_PREFIX_LENGTH chars + let char_indices: Vec<(usize, char)> = s.char_indices().collect(); + + // We can expand up to STRING_EXPANSION_LIMIT chars looking for a valid truncation point + let max_chars = char_indices.len().min(STRING_EXPANSION_LIMIT); + + // Start from STRING_PREFIX_LENGTH and look for a valid truncation point + for len in STRING_PREFIX_LENGTH..=max_chars { + if len >= char_indices.len() { + // Reached end of string - return original + return Some(Cow::Borrowed(s)); + } + + let (_, next_char) = char_indices[len]; + + // If the character being truncated is U+10FFFF (max Unicode code point), we cannot + // use this position. The tie-breaker must be >= the truncated char, but nothing is + // greater than U+10FFFF. Include it in the prefix and check the next character. + // (In Scala/Java this is a surrogate pair requiring substring check; in Rust it's one char) + if next_char == UTF8_MAX_CHAR { + continue; + } + + let truncation_byte_idx = char_indices[len].0; + let truncated = &s[..truncation_byte_idx]; + + // Choose tie-breaker based on the character being truncated + let tie_breaker = if next_char < ASCII_MAX_CHAR { + ASCII_MAX_CHAR + } else { + UTF8_MAX_CHAR + }; + + return Some(Cow::Owned(format!("{truncated}{tie_breaker}"))); + } + + // Could not find a valid truncation point within expansion limit + None +} + +// ============================================================================ +// Min/Max computation using Arrow compute kernels +// ============================================================================ + +/// Aggregation type selector. +#[derive(Clone, Copy)] +enum Agg { + Min, + Max, +} + +/// Compute aggregation for a primitive array. +fn agg_primitive(column: &ArrayRef, agg: Agg) -> DeltaResult> +where + T: ArrowPrimitiveType, + T::Native: PartialOrd, + PrimitiveArray: From>>, +{ + let array = column.as_primitive_opt::().ok_or_else(|| { + Error::generic(format!( + "Failed to downcast column to PrimitiveArray<{}>", + std::any::type_name::() + )) + })?; + let result = match agg { + Agg::Min => min(array), + Agg::Max => max(array), + }; + Ok(result.map(|v| Arc::new(PrimitiveArray::::from(vec![Some(v)])) as ArrayRef)) +} + +/// Compute aggregation for a timestamp array, preserving timezone. +fn agg_timestamp( + column: &ArrayRef, + tz: Option>, + agg: Agg, +) -> DeltaResult> +where + T: delta_kernel::arrow::datatypes::ArrowTimestampType, + PrimitiveArray: From>>, +{ + let array = column.as_primitive_opt::().ok_or_else(|| { + Error::generic(format!( + "Failed to downcast column to PrimitiveArray<{}>", + std::any::type_name::() + )) + })?; + let result = match agg { + Agg::Min => min(array), + Agg::Max => max(array), + }; + Ok(result.map(|v| { + Arc::new(PrimitiveArray::::from(vec![Some(v)]).with_timezone_opt(tz)) as ArrayRef + })) +} + +/// Compute aggregation for a decimal128 array, preserving precision and scale. +fn agg_decimal( + column: &ArrayRef, + precision: u8, + scale: i8, + agg: Agg, +) -> DeltaResult> { + let array = column + .as_primitive_opt::() + .ok_or_else(|| Error::generic("Failed to downcast column to Decimal128Array"))?; + let result = match agg { + Agg::Min => min(array), + Agg::Max => max(array), + }; + result + .map(|v| { + Decimal128Array::from(vec![Some(v)]) + .with_precision_and_scale(precision, scale) + .map(|arr| Arc::new(arr) as ArrayRef) + }) + .transpose() + .map_err(|e| Error::generic(format!("Invalid decimal precision/scale: {e}"))) +} + +/// Compute aggregation for a string array. +fn agg_string(column: &ArrayRef, agg: Agg) -> DeltaResult> { + let array = column + .as_string_opt::() + .ok_or_else(|| Error::generic("Failed to downcast column to StringArray"))?; + let result = match agg { + Agg::Min => min_string(array), + Agg::Max => max_string(array), + }; + Ok(result.map(|v| Arc::new(StringArray::from(vec![Some(v)])) as ArrayRef)) +} + +/// Compute aggregation for a large string array. +/// +/// Unlike StringArray, Arrow's compute kernels don't provide min/max for LargeStringArray, +/// so we iterate manually. `iter()` yields `Option<&str>` per element (None for nulls), +/// and `flatten()` filters out nulls so we only compare non-null values. +fn agg_large_string(column: &ArrayRef, agg: Agg) -> DeltaResult> { + let array = column + .as_string_opt::() + .ok_or_else(|| Error::generic("Failed to downcast column to LargeStringArray"))?; + let result = match agg { + Agg::Min => array.iter().flatten().min(), + Agg::Max => array.iter().flatten().max(), + }; + Ok(result.map(|v| Arc::new(LargeStringArray::from(vec![Some(v)])) as ArrayRef)) +} + +/// Compute aggregation for a string view array. +/// +/// Like LargeStringArray, Arrow's compute kernels don't provide min/max for StringViewArray. +/// See `agg_large_string` for explanation of `iter().flatten()`. +fn agg_string_view(column: &ArrayRef, agg: Agg) -> DeltaResult> { + let array = column + .as_string_view_opt() + .ok_or_else(|| Error::generic("Failed to downcast column to StringViewArray"))?; + let result: Option<&str> = match agg { + Agg::Min => array.iter().flatten().min(), + Agg::Max => array.iter().flatten().max(), + }; + Ok(result.map(|v| Arc::new(StringViewArray::from(vec![Some(v)])) as ArrayRef)) +} + +/// Compute min or max for a leaf column based on its data type. +/// +/// The result is the raw aggregate; [`truncate_stats_bound`] truncates string bounds. +fn compute_leaf_agg(column: &ArrayRef, agg: Agg) -> DeltaResult> { + match column.data_type() { + // Integer types + DataType::Int8 => agg_primitive::(column, agg), + DataType::Int16 => agg_primitive::(column, agg), + DataType::Int32 => agg_primitive::(column, agg), + DataType::Int64 => agg_primitive::(column, agg), + DataType::UInt8 => agg_primitive::(column, agg), + DataType::UInt16 => agg_primitive::(column, agg), + DataType::UInt32 => agg_primitive::(column, agg), + DataType::UInt64 => agg_primitive::(column, agg), + + // Float types + DataType::Float32 => agg_primitive::(column, agg), + DataType::Float64 => agg_primitive::(column, agg), + + // Date types + DataType::Date32 => agg_primitive::(column, agg), + DataType::Date64 => agg_primitive::(column, agg), + + // Timestamp types (preserve timezone) + DataType::Timestamp(TimeUnit::Second, tz) => { + agg_timestamp::(column, tz.clone(), agg) + } + DataType::Timestamp(TimeUnit::Millisecond, tz) => { + agg_timestamp::(column, tz.clone(), agg) + } + DataType::Timestamp(TimeUnit::Microsecond, tz) => { + agg_timestamp::(column, tz.clone(), agg) + } + DataType::Timestamp(TimeUnit::Nanosecond, tz) => { + agg_timestamp::(column, tz.clone(), agg) + } + + // Decimal type (preserve precision/scale) + DataType::Decimal128(p, s) => agg_decimal(column, *p, *s, agg), + + // String types + DataType::Utf8 => agg_string(column, agg), + DataType::LargeUtf8 => agg_large_string(column, agg), + DataType::Utf8View => agg_string_view(column, agg), + + // Unsupported types (structs handled separately, others return no min/max) + _ => Ok(None), + } +} + +// ============================================================================ +// Combined stats computation (single traversal) +// ============================================================================ + +/// Statistics computed for a column (leaf or nested struct). +#[derive(Default)] +struct ColumnStats { + null_count: Option, + min_value: Option, + max_value: Option, +} + +/// Compute all statistics for a column in a single traversal. +/// +/// Returns `ColumnStats` containing null_count, min, and max for this column. +/// For struct columns, these are nested StructArrays. For leaf columns, these are scalar arrays. +/// Complex types (Map, List, Variant) get nullCount only because they have no meaningful +/// ordering for min/max data skipping, but null counts are still useful for tracking nullability. +fn compute_column_stats( + column: &ArrayRef, + path: &mut Vec, + filter: &ColumnTrie<'_>, + null_count_only_filter: &ColumnTrie<'_>, +) -> DeltaResult { + match column.data_type() { + // A struct column that the filter marks as a terminal leaf (e.g. Variant, which is a + // struct at the Arrow level but a leaf for stats purposes) gets nullCount only, no + // recursion into sub-fields and no min/max. + DataType::Struct(_) if filter.is_terminal(path) => Ok(ColumnStats { + null_count: Some(Arc::new(Int64Array::from(vec![column.null_count() as i64]))), + min_value: None, + max_value: None, + }), + DataType::Struct(fields) => { + let struct_array = column + .as_struct_opt() + .ok_or_else(|| Error::generic("Failed to downcast column to StructArray"))?; + + // Propagate struct-level nulls to all descendants + let fixed_struct = fix_nested_null_masks(struct_array.clone()); + + // Accumulators for each stat type + let mut null_fields: Vec = Vec::new(); + let mut null_arrays: Vec = Vec::new(); + let mut min_fields: Vec = Vec::new(); + let mut min_arrays: Vec = Vec::new(); + let mut max_fields: Vec = Vec::new(); + let mut max_arrays: Vec = Vec::new(); + + for (i, field) in fields.iter().enumerate() { + path.push(field.name().to_string()); + + let child_stats = compute_column_stats( + fixed_struct.column(i), + path, + filter, + null_count_only_filter, + )?; + + if let Some(arr) = child_stats.null_count { + null_fields.push(Field::new(field.name(), arr.data_type().clone(), true)); + null_arrays.push(arr); + } + if let Some(arr) = child_stats.min_value { + min_fields.push(Field::new(field.name(), arr.data_type().clone(), true)); + min_arrays.push(arr); + } + if let Some(arr) = child_stats.max_value { + max_fields.push(Field::new(field.name(), arr.data_type().clone(), true)); + max_arrays.push(arr); + } + + path.pop(); + } + + // Build result structs (None if empty) + let build_struct = + |fields: Vec, arrays: Vec| -> DeltaResult> { + if fields.is_empty() { + Ok(None) + } else { + Ok(Some(Arc::new( + StructArray::try_new(fields.into(), arrays, None) + .map_err(|e| Error::generic(format!("stats struct: {e}")))?, + ) as ArrayRef)) + } + }; + + Ok(ColumnStats { + null_count: build_struct(null_fields, null_arrays)?, + min_value: build_struct(min_fields, min_arrays)?, + max_value: build_struct(max_fields, max_arrays)?, + }) + } + // Void columns (Arrow `Null` / kernel `VOID`): every value is null by definition, + // and the column has no parquet representation. We still need to publish nullCount + // for IS NULL / IS NOT NULL data skipping, so synthesize it from the array length. + // Use `column.len()` rather than `column.null_count()` because `NullArray` has no + // null buffer and the inherited `Array::null_count` default returns 0. + DataType::Null => { + if !filter.contains_prefix_of(path) { + return Ok(ColumnStats::default()); + } + Ok(ColumnStats { + null_count: Some(Arc::new(Int64Array::from(vec![column.len() as i64]))), + min_value: None, + max_value: None, + }) + } + _ => { + // Leaf: check filter, compute all stats together + if !filter.contains_prefix_of(path) { + return Ok(ColumnStats::default()); + } + + let null_count: Option = + Some(Arc::new(Int64Array::from(vec![column.null_count() as i64]))); + + let complex_type = matches!( + column.data_type(), + DataType::Map(_, _) + | DataType::List(_) + | DataType::LargeList(_) + | DataType::FixedSizeList(_, _) + | DataType::ListView(_) + | DataType::LargeListView(_) + ); + if complex_type || null_count_only_filter.contains_prefix_of(path) { + return Ok(ColumnStats { + null_count, + min_value: None, + max_value: None, + }); + } + + // When min/max is None (all nulls or unsupported type), emit a null-valued + // single-element array to keep the field present in the stats struct. This + // allows downstream consumers (like StatsColumnVerifier) to find the column and + // check nullCount == numRecords. The JSON serializer omits null fields, so + // the on-disk format still matches Spark's ignoreNullFields behavior. + let null_fallback = || -> ArrayRef { Arc::new(new_null_array(column.data_type(), 1)) }; + Ok(ColumnStats { + null_count, + min_value: Some(compute_leaf_agg(column, Agg::Min)?.unwrap_or_else(&null_fallback)), + max_value: Some(compute_leaf_agg(column, Agg::Max)?.unwrap_or_else(null_fallback)), + }) + } + } +} + +/// Accumulates (field_name, array) pairs for building a stats struct. +struct StatsAccumulator { + name: &'static str, + fields: Vec, + arrays: Vec, +} + +impl StatsAccumulator { + fn new(name: &'static str) -> Self { + Self { + name, + fields: Vec::new(), + arrays: Vec::new(), + } + } + + fn push(&mut self, field_name: &str, array: ArrayRef) { + self.fields + .push(Field::new(field_name, array.data_type().clone(), true)); + self.arrays.push(array); + } + + fn build(self) -> DeltaResult)>> { + if self.fields.is_empty() { + return Ok(None); + } + let struct_arr = StructArray::try_new(self.fields.into(), self.arrays, None) + .map_err(|e| Error::generic(format!("Failed to create {}: {e}", self.name)))?; + let field = Field::new(self.name, struct_arr.data_type().clone(), true); + Ok(Some((field, Arc::new(struct_arr) as Arc))) + } +} + +/// Collect statistics from a RecordBatch for Delta Lake file statistics. +/// +/// Returns a StructArray with the following fields: +/// - `numRecords`: total row count +/// - `nullCount`: nested struct with null counts per column +/// - `minValues`: nested struct with min values per column (null when all values are null) +/// - `maxValues`: nested struct with max values per column (null when all values are null) +/// - `tightBounds`: always true for new file writes +/// +/// String min/max values are truncated to a 32-character prefix with appropriate tie-breaker +/// characters for max values. See the `stats_schema` module documentation for the full stats +/// value rules. +/// +/// # Parameters +/// +/// - `batch`: The record batch to collect statistics from. +/// - `stats_columns`: The columns to include in `nullCount`, `minValues`, and `maxValues`. +/// - `physical_schema`: The kernel physical schema, used to preserve logical type distinctions that +/// are erased in Arrow arrays. +/// +/// # Returns +/// +/// A single-row struct array containing the collected file statistics. +/// +/// # Errors +/// +/// Returns an error if a column cannot be converted to its expected Arrow type or the output stats +/// array cannot be constructed. +pub fn collect_stats( + batch: &RecordBatch, + stats_columns: &[ColumnName], + physical_schema: &StructType, +) -> DeltaResult { + let null_count_only_columns = interval_column_names(physical_schema, stats_columns); + reduce_stats(&collect_stats_raw( + batch, + stats_columns, + &null_count_only_columns, + )?) +} + +#[cfg(test)] +pub(crate) fn collect_stats_for_test( + batch: &RecordBatch, + stats_columns: &[ColumnName], +) -> DeltaResult { + reduce_stats(&collect_stats_raw(batch, stats_columns, &[])?) +} + +/// Collect one batch's raw (untruncated) statistics as a single-row struct. [`reduce_stats`] turns +/// one or more such rows into the publishable statistics for a file. +fn collect_stats_raw( + batch: &RecordBatch, + stats_columns: &[ColumnName], + null_count_only_columns: &[ColumnName], +) -> DeltaResult { + let filter = ColumnTrie::from_columns(stats_columns); + let null_count_only_filter = ColumnTrie::from_columns(null_count_only_columns); + let schema = batch.schema(); + + // Collect all stats in a single traversal + let mut null_counts = StatsAccumulator::new(NULL_COUNT); + let mut min_values = StatsAccumulator::new(MIN_VALUES); + let mut max_values = StatsAccumulator::new(MAX_VALUES); + + for (col_idx, field) in schema.fields().iter().enumerate() { + let mut path = vec![field.name().to_string()]; + let column = batch.column(col_idx); + + // Single traversal computes all three stats + let stats = compute_column_stats(column, &mut path, &filter, &null_count_only_filter)?; + + if let Some(arr) = stats.null_count { + null_counts.push(field.name(), arr); + } + if let Some(arr) = stats.min_value { + min_values.push(field.name(), arr); + } + if let Some(arr) = stats.max_value { + max_values.push(field.name(), arr); + } + } + + // Build output struct + let mut fields = vec![Field::new(NUM_RECORDS, DataType::Int64, true)]; + let mut arrays: Vec> = + vec![Arc::new(Int64Array::from(vec![batch.num_rows() as i64]))]; + + for acc in [null_counts, min_values, max_values] { + if let Some((field, array)) = acc.build()? { + fields.push(field); + arrays.push(array); + } + } + + // tightBounds + fields.push(Field::new(TIGHT_BOUNDS, DataType::Boolean, true)); + arrays.push(Arc::new(BooleanArray::from(vec![true]))); + + StructArray::try_new(fields.into(), arrays, None) + .map_err(|e| Error::generic(format!("Failed to create stats struct: {e}"))) +} + +// ============================================================================ +// File-level stats aggregation +// ============================================================================ + +/// Accumulates per-row-group Delta statistics into a single file-level statistics struct. +/// +/// A connector that writes one file as multiple row groups feeds each row group's [`RecordBatch`] +/// to [`FileStatsAccumulator::merge`] as the row group closes, then calls +/// [`FileStatsAccumulator::finish`] once the file is complete to get the file-level statistics for +/// the Add action. Only one single-row statistics struct per row group is retained, never the row +/// groups themselves. +/// +/// Per-row-group statistics are accumulated raw and reduced once by +/// [`FileStatsAccumulator::finish`], which truncates string bounds as part of that reduction. Raw +/// bounds compose across row groups where truncated ones would not, so the file-level statistics +/// are byte-identical to collecting statistics over the whole file at once. +/// +/// Every batch must have the same Arrow schema, since the statistics shape derives from it: +/// [`FileStatsAccumulator::merge`] rejects a batch whose Arrow schema differs from that of the +/// first batch merged. A failed merge is terminal: every later [`merge`] and [`finish`] returns an +/// error, so a file whose row groups did not all merge cannot publish partial statistics. +/// +/// [`merge`]: FileStatsAccumulator::merge +/// [`finish`]: FileStatsAccumulator::finish +/// +/// # Example +/// +/// The merged statistics go into [`DataFileMetadata::new`](crate::parquet::DataFileMetadata::new) +/// to build the written file's Add action. +/// +/// ``` +/// # use std::sync::Arc; +/// # use delta_kernel::arrow::array::{Array, AsArray, Int64Array, RecordBatch, StructArray}; +/// # use delta_kernel::arrow::datatypes::{DataType, Field, Int64Type, Schema}; +/// # use delta_kernel::expressions::column_name; +/// # use delta_kernel::schema::{DataType as KernelDataType, StructField, StructType}; +/// # use delta_kernel_default_engine::stats::FileStatsAccumulator; +/// # fn main() -> delta_kernel::DeltaResult<()> { +/// let schema = Arc::new(Schema::new(vec![Field::new("id", DataType::Int64, false)])); +/// let row_group = |ids: Vec| { +/// RecordBatch::try_new(schema.clone(), vec![Arc::new(Int64Array::from(ids))]) +/// }; +/// let physical_schema = StructType::try_new([StructField::not_null("id", KernelDataType::LONG)])?; +/// +/// let mut acc = FileStatsAccumulator::new(&[column_name!("id")], &physical_schema); +/// acc.merge(&row_group(vec![1, 2])?)?; +/// acc.merge(&row_group(vec![3, 4])?)?; +/// let stats = acc.finish()?.expect("two row groups were merged"); +/// +/// let id_stat = |section: &str| -> i64 { +/// stats.column_by_name(section).unwrap().as_struct().column_by_name("id").unwrap() +/// .as_primitive::().value(0) +/// }; +/// assert_eq!( +/// stats.column_by_name("numRecords").unwrap().as_primitive::().value(0), +/// 4, +/// ); +/// assert_eq!(id_stat("minValues"), 1); // from the first row group +/// assert_eq!(id_stat("maxValues"), 4); // from the second +/// # Ok(()) +/// # } +/// ``` +#[derive(Debug)] +pub struct FileStatsAccumulator { + stats_columns: Vec, + null_count_only_columns: Vec, + batch_schema: Option, + state: AccumulatorState, +} + +/// Makes the terminal state representable, so a failed merge cannot be forgotten. +#[derive(Debug)] +enum AccumulatorState { + /// One raw statistics row per row group merged so far. + Open(Vec), + /// A merge failed, so the row groups merged so far can never be published. + Failed, +} + +const FAILED: &str = "statistics accumulator failed a previous merge"; + +impl FileStatsAccumulator { + /// Create an empty accumulator that collects statistics for `stats_columns` (the same + /// allowlist used for every row group of the file). `physical_schema` is the physical schema + /// of the data being written, used to distinguish interval columns (which get `nullCount` + /// only) from integer columns. + pub fn new(stats_columns: &[ColumnName], physical_schema: &StructType) -> Self { + Self { + null_count_only_columns: interval_column_names(physical_schema, stats_columns), + stats_columns: stats_columns.to_vec(), + batch_schema: None, + state: AccumulatorState::Open(Vec::new()), + } + } + + /// Collect one row group's raw statistics and add them to the accumulator. + /// + /// Returns an error if the accumulator already failed a merge, or if this one fails: `batch` + /// has a different Arrow schema than the first batch merged, it produces a different statistics + /// shape, or statistics collection fails. + pub fn merge(&mut self, batch: &RecordBatch) -> DeltaResult<()> { + // Catch every error path in one place, so no failure can leave publishable state behind. + self.try_merge(batch) + .inspect_err(|_| self.state = AccumulatorState::Failed) + } + + fn try_merge(&mut self, batch: &RecordBatch) -> DeltaResult<()> { + let AccumulatorState::Open(row_groups) = &mut self.state else { + return Err(Error::stats_validation(FAILED)); + }; + // Comparing derived stats shapes instead would collapse every nullCount-only leaf to the + // same Int64. + let first_schema = self.batch_schema.get_or_insert_with(|| batch.schema()); + if first_schema != &batch.schema() { + // `Debug`: schema equality covers the schema's metadata map, which `Display` omits. + return Err(Error::schema(format!( + "all row groups in a file must have the same schema; expected {:?} but got {:?}", + first_schema, + batch.schema(), + ))); + } + let batch_stats = + collect_stats_raw(batch, &self.stats_columns, &self.null_count_only_columns)?; + // Schema equality can ignore a struct column's nested field names, which statistics + // collection filters on, so equal schemas can still derive different stats shapes. + if let Some(first) = row_groups.first() { + if first.data_type() != batch_stats.data_type() { + return Err(Error::schema(format!( + "all row groups must produce the same statistics shape; batch schemas compare \ + equal but their nested field names differ: {} vs {}", + first.data_type(), + batch_stats.data_type(), + ))); + } + } + row_groups.push(batch_stats); + Ok(()) + } + + /// Consume the accumulator and return the file-level statistics, or `None` if no row group was + /// merged. A zero-row row group is still a merged row group and yields statistics with a + /// `numRecords` of 0. + /// + /// Publishing statistics for a file with no rows therefore requires merging a zero-row batch: + /// an accumulator that merged nothing yields `None`, and an Add action carrying no + /// `numRecords` is rejected by tables that require it (IcebergCompat tables, and any file + /// carrying a deletion vector). + /// + /// Returns `Err` if a merge failed, or if the accumulated statistics cannot be reduced into a + /// single row. + pub fn finish(self) -> DeltaResult> { + let AccumulatorState::Open(row_groups) = self.state else { + return Err(Error::stats_validation(FAILED)); + }; + if row_groups.is_empty() { + return Ok(None); + } + let rows: Vec<&dyn Array> = row_groups.iter().map(|s| s as &dyn Array).collect(); + let combined = concat(&rows) + .map_err(|e| Error::generic(format!("concat per-row-group stats: {e}")))?; + let combined = combined + .as_struct_opt() + .ok_or_else(|| Error::internal_error("concatenated stats are not a struct"))?; + reduce_stats(combined).map(Some) + } +} + +/// Reduce a raw statistics struct of one row per row group into the single publishable row for the +/// file: `numRecords` and `nullCount` summed, `minValues`/`maxValues` aggregated element-wise and +/// truncated per Delta's string rules, `tightBounds` AND-ed. +/// +/// Returns `Err` if `stats` is not shaped like the output of `collect_stats_raw`. +fn reduce_stats(stats: &StructArray) -> DeltaResult { + let fields = stats.fields().clone(); + let mut cols: Vec = Vec::with_capacity(stats.num_columns()); + for (field, col) in fields.iter().zip(stats.columns()) { + cols.push(match field.name().as_str() { + NUM_RECORDS => reduce_count_leaf(col)?, + NULL_COUNT => reduce_stats_children(col, &reduce_count_leaf)?, + MIN_VALUES => reduce_stats_children(col, &|c| reduce_minmax_leaf(c, Agg::Min))?, + MAX_VALUES => reduce_stats_children(col, &|c| reduce_minmax_leaf(c, Agg::Max))?, + TIGHT_BOUNDS => reduce_bool_and_leaf(col)?, + // Keep in sync with the sections `collect_stats_raw` produces. User columns live inside + // the sub-structs, so an unrecognized top-level name is a kernel bug, not bad input. + other => { + return Err(Error::internal_error(format!( + "cannot reduce unknown stats section: {other}" + ))) + } + }); + } + StructArray::try_new(fields, cols, None) + .map_err(|e| Error::generic(format!("rebuilding reduced stats struct: {e}"))) +} + +/// Reduce each child of a stats sub-struct (`nullCount`/`minValues`/`maxValues`) to one row, +/// recursing into nested structs so nested-column stats reduce correctly. `reduce` handles a leaf. +fn reduce_stats_children( + array: &ArrayRef, + reduce: &dyn Fn(&ArrayRef) -> DeltaResult, +) -> DeltaResult { + let struct_array = array + .as_struct_opt() + .ok_or_else(|| Error::internal_error("expected struct in stats sub-tree"))?; + let fields = struct_array.fields().clone(); + let cols = struct_array + .columns() + .iter() + .map(|col| match col.as_struct_opt() { + Some(_) => reduce_stats_children(col, reduce), + None => reduce(col), + }) + .collect::>>()?; + Ok(Arc::new(StructArray::try_new(fields, cols, None).map_err( + |e| Error::generic(format!("rebuilding reduced stats sub-struct: {e}")), + )?)) +} + +/// Sum an Int64 count leaf (`numRecords` or a `nullCount` leaf) to one row. +/// +/// A null count is unknown, not zero, and the protocol defines no default for it, so a null leaf is +/// an error rather than a silently invented count. +fn reduce_count_leaf(array: &ArrayRef) -> DeltaResult { + let arr = array + .as_primitive_opt::() + .ok_or_else(|| Error::internal_error("expected Int64 count leaf in stats"))?; + if arr.null_count() != 0 { + return Err(Error::internal_error("null count leaf in stats")); + } + let sum = sum_checked(arr) + .map_err(|e| Error::generic(format!("summing stats count leaf: {e}")))? + .unwrap_or(0); + Ok(Arc::new(Int64Array::from(vec![sum]))) +} + +/// AND a Boolean `tightBounds` leaf to one row: tight only if every row group was tight. +/// +/// A null is an error rather than an assumed bound tightness. +fn reduce_bool_and_leaf(array: &ArrayRef) -> DeltaResult { + let arr = array + .as_boolean_opt() + .ok_or_else(|| Error::internal_error("expected Boolean tightBounds leaf in stats"))?; + if arr.null_count() != 0 { + return Err(Error::internal_error("null tightBounds leaf in stats")); + } + let all = bool_and(arr).unwrap_or(true); + Ok(Arc::new(BooleanArray::from(vec![all]))) +} + +/// Reduce a min (or max) leaf to the one bound published for the file, null if none is +/// representable. +/// +/// Reusing `compute_leaf_agg` gives the bound the same ordering a single-pass collection uses (NaN, +/// decimal precision/scale, timestamp timezone, string byte order). +fn reduce_minmax_leaf(array: &ArrayRef, agg: Agg) -> DeltaResult { + let bound = match compute_leaf_agg(array, agg)? { + Some(bound) => truncate_stats_bound(&bound, agg)?, + None => None, + }; + Ok(bound.unwrap_or_else(|| new_null_array(array.data_type(), 1))) +} + +/// Truncate a single-row min/max bound per Delta's string rules, passing non-string bounds through. +/// +/// `None` when a max string has no representable truncated upper bound: no upper bound may be +/// published in that case. +fn truncate_stats_bound(bound: &ArrayRef, agg: Agg) -> DeltaResult> { + let s = match bound.data_type() { + DataType::Utf8 => bound.as_string_opt::().and_then(|a| a.iter().next()), + DataType::LargeUtf8 => bound.as_string_opt::().and_then(|a| a.iter().next()), + DataType::Utf8View => bound.as_string_view_opt().and_then(|a| a.iter().next()), + _ => return Ok(Some(bound.clone())), + }; + let s = s + .flatten() + .ok_or_else(|| Error::generic("expected a single non-null string stats bound"))?; + let Some(truncated) = (match agg { + Agg::Min => Some(Cow::Borrowed(truncate_min_string(s))), + Agg::Max => truncate_max_string(s), + }) else { + return Ok(None); + }; + let v = Some(&*truncated); + Ok(Some(match bound.data_type() { + DataType::LargeUtf8 => Arc::new(LargeStringArray::from(vec![v])) as ArrayRef, + DataType::Utf8View => Arc::new(StringViewArray::from(vec![v])) as ArrayRef, + _ => Arc::new(StringArray::from(vec![v])) as ArrayRef, + })) +} + +fn is_interval_type(data_type: &KernelDataType) -> bool { + let KernelDataType::Primitive(primitive_type) = data_type else { + return false; + }; + primitive_type.is_interval() +} + +fn interval_column_names(schema: &StructType, stats_columns: &[ColumnName]) -> Vec { + stats_columns + .iter() + .filter(|col| { + schema + .fields_of_path(col) + .ok() + .and_then(|fields| fields.last().map(|field| field.data_type())) + .is_some_and(is_interval_type) + }) + .cloned() + .collect() +} + +#[cfg(test)] +mod tests { + use delta_kernel::arrow::array::{ + Array, AsArray, BinaryArray, Decimal128Array, Float64Array, Int32Array, Int64Array, + ListArray, MapArray, NullArray, RecordBatchOptions, StringArray, + }; + use delta_kernel::arrow::buffer::{NullBuffer, OffsetBuffer}; + use delta_kernel::arrow::compute::concat_batches; + use delta_kernel::arrow::datatypes::{Fields, Int32Type, Int64Type, Schema}; + use delta_kernel::engine::arrow_conversion::TryFromArrow as _; + use delta_kernel::engine::arrow_expression::evaluate_expression::to_json; + use delta_kernel::expressions::column_name; + use delta_kernel::parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder; + use delta_kernel::schema::StructField; + use test_utils::assert_result_error_with_message; + + use super::*; + + fn collect_stats( + batch: &RecordBatch, + stats_columns: &[ColumnName], + ) -> DeltaResult { + super::collect_stats_for_test(batch, stats_columns) + } + + #[test] + fn test_collect_stats_single_batch() { + let schema = Arc::new(Schema::new(vec![Field::new("id", DataType::Int64, false)])); + + let batch = + RecordBatch::try_new(schema, vec![Arc::new(Int64Array::from(vec![1, 2, 3]))]).unwrap(); + + let stats = collect_stats(&batch, &[column_name!("id")]).unwrap(); + + assert_eq!(stats.len(), 1); + let num_records = stats + .column_by_name(NUM_RECORDS) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(num_records.value(0), 3); + } + + #[test] + fn test_collect_stats_null_counts() { + let schema = Arc::new(Schema::new(vec![ + Field::new("id", DataType::Int64, false), + Field::new("value", DataType::Utf8, true), + ])); + + let batch = RecordBatch::try_new( + schema, + vec![ + Arc::new(Int64Array::from(vec![1, 2, 3])), + Arc::new(StringArray::from(vec![Some("a"), None, Some("c")])), + ], + ) + .unwrap(); + + let stats = collect_stats(&batch, &[column_name!("id"), column_name!("value")]).unwrap(); + + // Check nullCount struct + let null_count = stats + .column_by_name(NULL_COUNT) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + // id has 0 nulls + let id_null_count = null_count + .column_by_name("id") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(id_null_count.value(0), 0); + + // value has 1 null + let value_null_count = null_count + .column_by_name("value") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(value_null_count.value(0), 1); + } + + #[test] + fn test_collect_stats_respects_stats_columns() { + let schema = Arc::new(Schema::new(vec![ + Field::new("id", DataType::Int64, false), + Field::new("value", DataType::Utf8, true), + ])); + + let batch = RecordBatch::try_new( + schema, + vec![ + Arc::new(Int64Array::from(vec![1, 2, 3])), + Arc::new(StringArray::from(vec![Some("a"), None, Some("c")])), + ], + ) + .unwrap(); + + // Only collect stats for "id", not "value" + let stats = collect_stats(&batch, &[column_name!("id")]).unwrap(); + + let null_count = stats + .column_by_name(NULL_COUNT) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + // Only id should be present + assert!(null_count.column_by_name("id").is_some()); + assert!(null_count.column_by_name("value").is_none()); + } + + #[test] + fn test_collect_stats_min_max() { + let schema = Arc::new(Schema::new(vec![ + Field::new("number", DataType::Int64, false), + Field::new("name", DataType::Utf8, true), + ])); + + let batch = RecordBatch::try_new( + schema, + vec![ + Arc::new(Int64Array::from(vec![5, 1, 9, 3])), + Arc::new(StringArray::from(vec![ + Some("banana"), + Some("apple"), + Some("cherry"), + None, + ])), + ], + ) + .unwrap(); + + let stats = collect_stats(&batch, &[column_name!("number"), column_name!("name")]).unwrap(); + + // Check minValues + let min_values = stats + .column_by_name(MIN_VALUES) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + let number_min = min_values + .column_by_name("number") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(number_min.value(0), 1); + + let name_min = min_values + .column_by_name("name") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(name_min.value(0), "apple"); + + // Check maxValues + let max_values = stats + .column_by_name(MAX_VALUES) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + let number_max = max_values + .column_by_name("number") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(number_max.value(0), 9); + + let name_max = max_values + .column_by_name("name") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(name_max.value(0), "cherry"); + } + + #[test] + fn test_collect_stats_all_nulls() { + let schema = Arc::new(Schema::new(vec![Field::new( + "value", + DataType::Int64, + true, + )])); + + let batch = RecordBatch::try_new( + schema, + vec![Arc::new(Int64Array::from(vec![ + None as Option, + None, + None, + ]))], + ) + .unwrap(); + + let stats = collect_stats(&batch, &[column_name!("value")]).unwrap(); + + // numRecords should be 3 + let num_records = stats + .column_by_name(NUM_RECORDS) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(num_records.value(0), 3); + + // nullCount should be 3 + let null_count = stats + .column_by_name(NULL_COUNT) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + let value_null_count = null_count + .column_by_name("value") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(value_null_count.value(0), 3); + + // All-null columns are present in minValues/maxValues but with null values. + // The field must exist so that StatsColumnVerifier can find it via visit_rows and + // check nullCount == numRecords. The JSON serializer omits null fields, so + // the on-disk format still matches Spark's ignoreNullFields behavior. + let min_values = stats + .column_by_name(MIN_VALUES) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + let min_col = min_values.column_by_name("value").unwrap(); + assert!(min_col.is_null(0)); + + let max_values = stats + .column_by_name(MAX_VALUES) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + let max_col = max_values.column_by_name("value").unwrap(); + assert!(max_col.is_null(0)); + } + + // A void column reaches stats only if a connector or direct caller bypasses the + // kernel-side physical write schema, which strips void columns. Even so, we must + // publish nullCount = numRecords rather than 0, because `NullArray` has no null + // buffer and the inherited `Array::null_count` default returns 0. Min/max are not + // meaningful for void. + #[rstest::rstest] + #[case::non_empty(5)] + #[case::empty(0)] + fn test_collect_stats_void_column_synthesizes_full_null_count(#[case] length: usize) { + let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Null, true)])); + let batch = RecordBatch::try_new(schema, vec![Arc::new(NullArray::new(length))]).unwrap(); + + let stats = collect_stats(&batch, &[column_name!("v")]).unwrap(); + + let null_count = stats + .column_by_name("nullCount") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + let v_null_count = null_count + .column_by_name("v") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(v_null_count.value(0), length as i64); + + // Void columns do not participate in min/max stats. With void as the only stats + // column, both struct accumulators stay empty and the fields are omitted entirely. + assert!(stats.column_by_name("minValues").is_none()); + assert!(stats.column_by_name("maxValues").is_none()); + } + + #[test] + fn test_collect_stats_void_column_nested_in_struct() { + // Worst realistic shape for the latent bug: void buried inside a struct alongside + // a non-void sibling. The recursion must reach the `DataType::Null` arm so that + // `s.v` records `nullCount = numRecords`, while `s.a` still gets full min/max. + let inner_fields = Fields::from(vec![ + Field::new("a", DataType::Int64, false), + Field::new("v", DataType::Null, true), + ]); + let schema = Arc::new(Schema::new(vec![Field::new( + "s", + DataType::Struct(inner_fields.clone()), + false, + )])); + let inner = StructArray::try_new( + inner_fields, + vec![ + Arc::new(Int64Array::from(vec![1, 2, 3, 4])) as ArrayRef, + Arc::new(NullArray::new(4)) as ArrayRef, + ], + None, + ) + .unwrap(); + let batch = RecordBatch::try_new(schema, vec![Arc::new(inner) as ArrayRef]).unwrap(); + + let stats = collect_stats(&batch, &[column_name!("s.a"), column_name!("s.v")]).unwrap(); + + let s_null_count = stats + .column_by_name("nullCount") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap() + .column_by_name("s") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + let v_null_count = s_null_count + .column_by_name("v") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(v_null_count.value(0), 4); + + // `s.a` keeps its min/max; `s.v` does not appear under min/max because void has no + // ordering. + let s_min = stats + .column_by_name("minValues") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap() + .column_by_name("s") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert!(s_min.column_by_name("a").is_some()); + assert!(s_min.column_by_name("v").is_none()); + } + + #[test] + fn test_collect_stats_empty_stats_columns() { + let schema = Arc::new(Schema::new(vec![Field::new("id", DataType::Int64, false)])); + + let batch = + RecordBatch::try_new(schema, vec![Arc::new(Int64Array::from(vec![1, 2, 3]))]).unwrap(); + + // No stats columns requested + let stats = collect_stats(&batch, &[]).unwrap(); + + // Should still have numRecords and tightBounds + assert!(stats.column_by_name(NUM_RECORDS).is_some()); + assert!(stats.column_by_name(TIGHT_BOUNDS).is_some()); + + // Should not have nullCount, minValues, maxValues + assert!(stats.column_by_name(NULL_COUNT).is_none()); + assert!(stats.column_by_name(MIN_VALUES).is_none()); + assert!(stats.column_by_name(MAX_VALUES).is_none()); + } + + #[test] + fn test_collect_stats_string_truncation_ascii() { + let schema = Arc::new(Schema::new(vec![Field::new("text", DataType::Utf8, false)])); + + // Create an ASCII string longer than 32 characters + let long_string = "a".repeat(50); + let batch = RecordBatch::try_new( + schema, + vec![Arc::new(StringArray::from(vec![long_string.as_str()]))], + ) + .unwrap(); + + let stats = collect_stats(&batch, &[column_name!("text")]).unwrap(); + + // Check minValues - should be truncated to exactly 32 chars + let min_values = stats + .column_by_name(MIN_VALUES) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + let text_min = min_values + .column_by_name("text") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + assert_eq!(text_min.value(0).len(), 32); + assert_eq!(text_min.value(0), "a".repeat(32)); + + // Check maxValues - should be 32 chars + 0x7F tie-breaker (since 'a' < 0x7F) + let max_values = stats + .column_by_name(MAX_VALUES) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + let text_max = max_values + .column_by_name("text") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + let expected_max = format!("{}\x7F", "a".repeat(32)); + assert_eq!(text_max.value(0), expected_max); + } + + #[test] + fn test_collect_stats_string_truncation_non_ascii() { + let schema = Arc::new(Schema::new(vec![Field::new("text", DataType::Utf8, false)])); + + // Create a string where the character BEING TRUNCATED (at position 32) is non-ASCII. + // The tie-breaker is chosen based on the first char being removed, not the last kept. + // 32 'a's followed by 'À' (>= 0x7F) followed by more chars + let long_string = format!("{}À{}", "a".repeat(32), "b".repeat(20)); + let batch = RecordBatch::try_new( + schema, + vec![Arc::new(StringArray::from(vec![long_string.as_str()]))], + ) + .unwrap(); + + let stats = collect_stats(&batch, &[column_name!("text")]).unwrap(); + + // Check maxValues - should use UTF8_MAX_CHAR since 'À' (the truncated char) >= 0x7F + let max_values = stats + .column_by_name(MAX_VALUES) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + let text_max = max_values + .column_by_name("text") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + // Should be 32 'a's + U+10FFFF (tie-breaker for non-ASCII truncated char) + let expected_max = format!("{}\u{10FFFF}", "a".repeat(32)); + assert_eq!(text_max.value(0), expected_max); + } + + #[test] + fn test_collect_stats_string_no_truncation_needed() { + let schema = Arc::new(Schema::new(vec![Field::new("text", DataType::Utf8, false)])); + + // String within 32 chars - should not be truncated + let short_string = "hello world"; + let batch = RecordBatch::try_new( + schema, + vec![Arc::new(StringArray::from(vec![short_string]))], + ) + .unwrap(); + + let stats = collect_stats(&batch, &[column_name!("text")]).unwrap(); + + let min_values = stats + .column_by_name(MIN_VALUES) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + let text_min = min_values + .column_by_name("text") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + assert_eq!(text_min.value(0), short_string); + + let max_values = stats + .column_by_name(MAX_VALUES) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + let text_max = max_values + .column_by_name("text") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + assert_eq!(text_max.value(0), short_string); + } + + #[test] + fn test_truncate_min_string() { + // Short string - no truncation + assert_eq!(truncate_min_string("hello"), "hello"); + + // Exactly 32 chars - no truncation + let s32 = "a".repeat(32); + assert_eq!(truncate_min_string(&s32), s32); + + // Long string - truncated to 32 chars + let s50 = "a".repeat(50); + assert_eq!(truncate_min_string(&s50), "a".repeat(32)); + + // Multi-byte characters + let multi = format!("{}À", "a".repeat(35)); // 'À' is 2 bytes in UTF-8 + assert_eq!(truncate_min_string(&multi).chars().count(), 32); + } + + #[test] + fn test_truncate_max_string() { + // Short string - no truncation, returns Cow::Borrowed + assert_eq!(truncate_max_string("hello").as_deref(), Some("hello")); + + // Exactly 32 chars - no truncation + let s32 = "a".repeat(32); + assert_eq!(truncate_max_string(&s32).as_deref(), Some(s32.as_str())); + + // Long ASCII string - truncated with 0x7F tie-breaker + // The 33rd char ('a') is < 0x7F, so we use 0x7F + let s50 = "a".repeat(50); + let expected = format!("{}\x7F", "a".repeat(32)); + assert_eq!( + truncate_max_string(&s50).as_deref(), + Some(expected.as_str()) + ); + + // Non-ASCII at truncation point - uses UTF8_MAX_CHAR + // 32 'a's then 'À' (which is >= 0x7F), so we use UTF8_MAX + let non_ascii = format!("{}À{}", "a".repeat(32), "b".repeat(20)); + let expected = format!("{}\u{10FFFF}", "a".repeat(32)); + assert_eq!( + truncate_max_string(&non_ascii).as_deref(), + Some(expected.as_str()) + ); + + // U+10FFFF at truncation point - must skip past it + // 32 'a's then U+10FFFF then 'b' - we can't truncate at U+10FFFF (no tie-breaker > it) + // so we include U+10FFFF in prefix and use 'b' to determine tie-breaker + let with_max_char = format!("{}\u{10FFFF}b{}", "a".repeat(32), "c".repeat(10)); + let expected = format!("{}\u{10FFFF}\x7F", "a".repeat(32)); // 'b' < 0x7F + assert_eq!( + truncate_max_string(&with_max_char).as_deref(), + Some(expected.as_str()) + ); + } + + #[test] + fn test_collect_stats_nested_struct() { + // Schema: { nested: { a: int64, b: string } } + let nested_fields = Fields::from(vec![ + Field::new("a", DataType::Int64, false), + Field::new("b", DataType::Utf8, true), + ]); + let schema = Arc::new(Schema::new(vec![Field::new( + "nested", + DataType::Struct(nested_fields.clone()), + false, + )])); + + // Build nested struct data + let a_array = Arc::new(Int64Array::from(vec![10, 5, 20])); + let b_array = Arc::new(StringArray::from(vec![Some("zebra"), Some("apple"), None])); + let nested_struct = StructArray::try_new( + nested_fields, + vec![a_array as ArrayRef, b_array as ArrayRef], + None, + ) + .unwrap(); + + let batch = + RecordBatch::try_new(schema, vec![Arc::new(nested_struct) as ArrayRef]).unwrap(); + + let stats = collect_stats( + &batch, + &[column_name!("nested.a"), column_name!("nested.b")], + ) + .unwrap(); + + // Check nullCount.nested.a = 0, nullCount.nested.b = 1 + let null_count = stats + .column_by_name(NULL_COUNT) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + let nested_null = null_count + .column_by_name("nested") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + let a_null = nested_null + .column_by_name("a") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(a_null.value(0), 0); + + let b_null = nested_null + .column_by_name("b") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(b_null.value(0), 1); + + // Check minValues.nested.a = 5, minValues.nested.b = "apple" + let min_values = stats + .column_by_name(MIN_VALUES) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + let nested_min = min_values + .column_by_name("nested") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + let a_min = nested_min + .column_by_name("a") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(a_min.value(0), 5); + + let b_min = nested_min + .column_by_name("b") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(b_min.value(0), "apple"); + + // Check maxValues.nested.a = 20, maxValues.nested.b = "zebra" + let max_values = stats + .column_by_name(MAX_VALUES) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + let nested_max = max_values + .column_by_name("nested") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + let a_max = nested_max + .column_by_name("a") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(a_max.value(0), 20); + + let b_max = nested_max + .column_by_name("b") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(b_max.value(0), "zebra"); + } + + #[test] + fn test_collect_stats_complex_types_null_count_only() { + // Schema with list column - should have nullCount but no min/max + let schema = Arc::new(Schema::new(vec![ + Field::new("id", DataType::Int64, false), + Field::new( + "list_col", + DataType::List(Arc::new(Field::new("item", DataType::Int64, true))), + true, + ), + ])); + + // Build list array: [[1, 2], null, [4, 5, 6]] + let values = Int64Array::from(vec![1, 2, 4, 5, 6]); + let offsets = OffsetBuffer::new(vec![0, 2, 2, 5].into()); + let list_array = ListArray::new( + Arc::new(Field::new("item", DataType::Int64, true)), + offsets, + Arc::new(values), + Some(vec![true, false, true].into()), // second element is null + ); + + let batch = RecordBatch::try_new( + schema, + vec![ + Arc::new(Int64Array::from(vec![1, 2, 3])), + Arc::new(list_array), + ], + ) + .unwrap(); + + // Request stats for both columns + let stats = collect_stats(&batch, &[column_name!("id"), column_name!("list_col")]).unwrap(); + + let null_count = stats + .column_by_name(NULL_COUNT) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + // id should have null count = 0 + let id_nulls = null_count + .column_by_name("id") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(id_nulls.value(0), 0); + + // list_col should have null count = 1 + let list_nulls = null_count + .column_by_name("list_col") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(list_nulls.value(0), 1); + + // minValues should have id but NOT list_col + let min_values = stats + .column_by_name(MIN_VALUES) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert!(min_values.column_by_name("id").is_some()); + assert!(min_values.column_by_name("list_col").is_none()); + + // maxValues should have id but NOT list_col + let max_values = stats + .column_by_name(MAX_VALUES) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert!(max_values.column_by_name("id").is_some()); + assert!(max_values.column_by_name("list_col").is_none()); + } + + #[test] + fn test_collect_stats_map_null_count_only() { + // === GIVEN: a batch with an id column and a map column (1 null out of 3 rows) === + let map_field = Field::new( + "entries", + DataType::Struct(Fields::from(vec![ + Field::new("key", DataType::Utf8, false), + Field::new("value", DataType::Int64, true), + ])), + false, + ); + let schema = Arc::new(Schema::new(vec![ + Field::new("id", DataType::Int64, false), + Field::new( + "map_col", + DataType::Map(Arc::new(map_field.clone()), false), + true, + ), + ])); + let keys = StringArray::from(vec!["a", "b"]); + let values = Int64Array::from(vec![1, 2]); + let entries = StructArray::new( + Fields::from(vec![ + Field::new("key", DataType::Utf8, false), + Field::new("value", DataType::Int64, true), + ]), + vec![Arc::new(keys) as ArrayRef, Arc::new(values) as ArrayRef], + None, + ); + let map_array = MapArray::new( + Arc::new(map_field), + OffsetBuffer::new(vec![0, 1, 1, 2].into()), + entries, + Some(vec![true, false, true].into()), // second element is null + false, + ); + let batch = RecordBatch::try_new( + schema, + vec![ + Arc::new(Int64Array::from(vec![1, 2, 3])), + Arc::new(map_array), + ], + ) + .unwrap(); + + // === WHEN: collecting stats for both columns === + let stats = collect_stats(&batch, &[column_name!("id"), column_name!("map_col")]).unwrap(); + + // === THEN: map_col has nullCount=1 but no min/max === + let null_count = child_struct(&stats, NULL_COUNT); + let map_nulls = null_count + .column_by_name("map_col") + .unwrap() + .as_primitive::(); + assert_eq!(map_nulls.value(0), 1); + + let min_values = child_struct(&stats, MIN_VALUES); + assert!(min_values.column_by_name("id").is_some()); + assert!(min_values.column_by_name("map_col").is_none()); + } + + #[test] + fn test_collect_stats_variant_terminal_struct_null_count_only() { + // Variant is Struct { metadata: Binary, value: Binary } at the Arrow level, but + // stats_column_names lists it as a terminal leaf ["v"]. The is_terminal guard in + // compute_column_stats must produce nullCount at the struct level without recursing + // into metadata/value sub-fields. + + // === GIVEN: a batch with an id column and a Variant column (1 null out of 3 rows) === + let variant_fields = Fields::from(vec![ + Field::new("metadata", DataType::Binary, false), + Field::new("value", DataType::Binary, false), + ]); + let schema = Arc::new(Schema::new(vec![ + Field::new("id", DataType::Int64, false), + Field::new("v", DataType::Struct(variant_fields.clone()), true), + ])); + let variant_array = StructArray::new( + variant_fields, + vec![ + Arc::new(BinaryArray::from(vec![ + Some([0x01, 0x00, 0x00].as_slice()), + None, + Some([0x01, 0x00, 0x00].as_slice()), + ])) as ArrayRef, + Arc::new(BinaryArray::from(vec![ + Some([0x0C].as_slice()), + None, + Some([0x0C].as_slice()), + ])) as ArrayRef, + ], + Some(NullBuffer::from_iter([true, false, true])), // row 1 is null + ); + let batch = RecordBatch::try_new( + schema, + vec![ + Arc::new(Int64Array::from(vec![1, 2, 3])), + Arc::new(variant_array), + ], + ) + .unwrap(); + + // === WHEN: collecting stats with "v" as a terminal column === + let stats = collect_stats(&batch, &[column_name!("id"), column_name!("v")]).unwrap(); + + // === THEN: v has nullCount=1 at the struct level, no recursion, no min/max === + let null_count = child_struct(&stats, NULL_COUNT); + let v_nulls = null_count + .column_by_name("v") + .unwrap() + .as_primitive::(); + assert_eq!(v_nulls.value(0), 1); + + // v must NOT be recursed into, no nested metadata/value in nullCount + assert!(null_count.column_by_name("metadata").is_none()); + assert!(null_count.column_by_name("value").is_none()); + + let min_values = child_struct(&stats, MIN_VALUES); + assert!(min_values.column_by_name("id").is_some()); + assert!(min_values.column_by_name("v").is_none()); + } + + #[test] + fn test_collect_stats_struct_with_nulls_at_struct_level() { + // Schema: { my_struct: { a: int32, b: int32 (nullable) } } + // Test both struct-level nulls and field-level nulls + let child_fields = Fields::from(vec![ + Field::new("a", DataType::Int32, false), + Field::new("b", DataType::Int32, true), + ]); + + let a_values = Int32Array::from(vec![1, 2, 3, 4]); + // b has field-level nulls at rows 0 and 2 + let b_values = Int32Array::from(vec![None, Some(20), None, Some(40)]); + + // Struct validity: [false, true, true, false] + // In Arrow: false = null, true = valid + // So rows 0 and 3 have null structs (entire struct is null) + let nulls = NullBuffer::from(vec![false, true, true, false]); + + let struct_array = StructArray::new( + child_fields.clone(), + vec![Arc::new(a_values), Arc::new(b_values)], + Some(nulls), + ); + + let schema = Schema::new(vec![Field::new( + "my_struct", + DataType::Struct(child_fields), + true, + )]); + + let batch = RecordBatch::try_new(Arc::new(schema), vec![Arc::new(struct_array)]).unwrap(); + + let stats = collect_stats( + &batch, + &[column_name!("my_struct.a"), column_name!("my_struct.b")], + ) + .unwrap(); + + // Visualizing the data: + // Row 0: struct=NULL, (a=1, b=None are "invisible") + // Row 1: struct=VALID, a=2, b=20 + // Row 2: struct=VALID, a=3, b=None + // Row 3: struct=NULL, (a=4, b=40 are "invisible") + // + // Expected behavior (struct nulls propagate to children): + // - a: visible values are [2, 3], nullCount = 2 (rows 0, 3 are struct-null) + // - b: visible values are [20, None], nullCount = 3 (rows 0, 3 struct-null + row 2 + // field-null) + // - a: min=2, max=3 + // - b: min=20, max=20 + + // nullCount includes struct-level nulls + assert_eq!( + get_stat::(&stats, NULL_COUNT, "my_struct", "a"), + 2 + ); + assert_eq!( + get_stat::(&stats, NULL_COUNT, "my_struct", "b"), + 3 + ); + + // minValues excludes values from null struct rows + assert_eq!( + get_stat::(&stats, MIN_VALUES, "my_struct", "a"), + 2 + ); + assert_eq!( + get_stat::(&stats, MIN_VALUES, "my_struct", "b"), + 20 + ); + + // maxValues excludes values from null struct rows + assert_eq!( + get_stat::(&stats, MAX_VALUES, "my_struct", "a"), + 3 + ); + assert_eq!( + get_stat::(&stats, MAX_VALUES, "my_struct", "b"), + 20 + ); + } + + #[test] + fn test_file_stats_accumulator_empty_returns_none() { + let physical_schema = StructType::try_from_arrow(&Schema::new(vec![Field::new( + "id", + DataType::Int64, + false, + )])) + .unwrap(); + let acc = FileStatsAccumulator::new(&[column_name!("id")], &physical_schema); + assert!(acc.finish().unwrap().is_none()); + } + + #[test] + fn test_file_stats_accumulator_single_batch_matches_collect_stats() { + let schema = Arc::new(Schema::new(vec![ + Field::new("id", DataType::Int64, false), + Field::new("value", DataType::Utf8, true), + ])); + let batch = RecordBatch::try_new( + schema.clone(), + vec![ + Arc::new(Int64Array::from(vec![1, 2, 3])), + Arc::new(StringArray::from(vec![Some("a"), None, Some("c")])), + ], + ) + .unwrap(); + let cols = vec![column_name!("id"), column_name!("value")]; + let physical_schema = StructType::try_from_arrow(schema.as_ref()).unwrap(); + + let single = collect_stats(&batch, &cols).unwrap(); + + let mut acc = FileStatsAccumulator::new(&cols, &physical_schema); + acc.merge(&batch).unwrap(); + let merged = acc.finish().unwrap().unwrap(); + + assert_eq!( + to_json(&merged).unwrap().as_string::().value(0), + to_json(&single).unwrap().as_string::().value(0), + ); + } + + /// Extracts a named child column from a StructArray, downcasting it to StructArray. + fn child_struct<'a>(parent: &'a StructArray, name: &str) -> &'a StructArray { + parent + .column_by_name(name) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap() + } + + /// Reads a primitive leaf of a nested column: `stats[stat_name][struct_name][field_name]`. + fn get_stat( + stats: &StructArray, + stat_name: &str, + struct_name: &str, + field_name: &str, + ) -> T::Native + where + T: delta_kernel::arrow::datatypes::ArrowPrimitiveType, + { + flat_stat::(child_struct(stats, stat_name), struct_name, field_name) + } + + /// Reads a primitive leaf of a top-level column: `stats[stat_name][field_name]`. + fn flat_stat(stats: &StructArray, stat_name: &str, field_name: &str) -> T::Native + where + T: delta_kernel::arrow::datatypes::ArrowPrimitiveType, + { + child_struct(stats, stat_name) + .column_by_name(field_name) + .unwrap() + .as_primitive::() + .value(0) + } + + /// Reads a string leaf directly under `struct_array`, `None` if the bound is null. + fn string_leaf<'a>(struct_array: &'a StructArray, field_name: &str) -> Option<&'a str> { + let leaf = struct_array + .column_by_name(field_name) + .unwrap() + .as_string_opt::() + .unwrap(); + leaf.is_valid(0).then(|| leaf.value(0)) + } + + #[test] + fn test_file_stats_accumulator_merges_two_batches() { + let schema = Arc::new(Schema::new(vec![ + Field::new("n", DataType::Int64, false), + Field::new("s", DataType::Utf8, true), + ])); + let physical_schema = StructType::try_from_arrow(schema.as_ref()).unwrap(); + let b1 = RecordBatch::try_new( + schema.clone(), + vec![ + Arc::new(Int64Array::from(vec![5, 1, 9])), + Arc::new(StringArray::from(vec![Some("banana"), Some("apple"), None])), + ], + ) + .unwrap(); + let b2 = RecordBatch::try_new( + schema, + vec![ + Arc::new(Int64Array::from(vec![3, 12, 0])), + Arc::new(StringArray::from(vec![Some("cherry"), None, Some("date")])), + ], + ) + .unwrap(); + let cols = vec![column_name!("n"), column_name!("s")]; + + let mut acc = FileStatsAccumulator::new(&cols, &physical_schema); + acc.merge(&b1).unwrap(); + acc.merge(&b2).unwrap(); + let merged = acc.finish().unwrap().unwrap(); + + assert_eq!( + merged + .column_by_name(NUM_RECORDS) + .unwrap() + .as_primitive::() + .value(0), + 6 + ); + assert_eq!(flat_stat::(&merged, NULL_COUNT, "s"), 2); + assert_eq!(flat_stat::(&merged, MIN_VALUES, "n"), 0); + assert_eq!(flat_stat::(&merged, MAX_VALUES, "n"), 12); + let leaf = |section| string_leaf(child_struct(&merged, section), "s"); + assert_eq!(leaf(MIN_VALUES), Some("apple")); + assert_eq!(leaf(MAX_VALUES), Some("date")); + assert!(merged + .column_by_name(TIGHT_BOUNDS) + .unwrap() + .as_boolean() + .value(0)); + } + + #[test] + fn test_file_stats_accumulator_min_max_skips_all_null_batch() { + let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Int64, true)])); + let physical_schema = StructType::try_from_arrow(schema.as_ref()).unwrap(); + let all_null = RecordBatch::try_new( + schema.clone(), + vec![Arc::new(Int64Array::from(vec![None as Option, None]))], + ) + .unwrap(); + let valued = RecordBatch::try_new( + schema, + vec![Arc::new(Int64Array::from(vec![Some(7), Some(4)]))], + ) + .unwrap(); + + let mut acc = FileStatsAccumulator::new(&[column_name!("v")], &physical_schema); + acc.merge(&all_null).unwrap(); + acc.merge(&valued).unwrap(); + let merged = acc.finish().unwrap().unwrap(); + + assert_eq!(flat_stat::(&merged, MIN_VALUES, "v"), 4); + assert_eq!(flat_stat::(&merged, MAX_VALUES, "v"), 7); + assert_eq!(flat_stat::(&merged, NULL_COUNT, "v"), 2); + } + + /// Recursively extracts leaf column names from an Arrow schema for stats collection. + fn extract_leaf_columns(fields: &Fields, prefix: &[String]) -> Vec { + let mut columns = Vec::new(); + for field in fields.iter() { + let mut path = prefix.to_vec(); + path.push(field.name().clone()); + match field.data_type() { + DataType::Struct(sub_fields) => { + columns.extend(extract_leaf_columns(sub_fields, &path)); + } + _ => { + columns.push(ColumnName::new(path)); + } + } + } + columns + } + + /// Recursively compares Spark stats JSON against kernel stats JSON. + /// Only checks keys present in `spark_val`; kernel may have extra keys. + fn assert_stats_match( + spark_val: &serde_json::Value, + kernel_val: &serde_json::Value, + path: &str, + ) { + match (spark_val, kernel_val) { + (serde_json::Value::Object(spark_map), serde_json::Value::Object(kernel_map)) => { + for (key, spark_child) in spark_map { + let child_path = if path.is_empty() { + key.clone() + } else { + format!("{path}.{key}") + }; + let kernel_child = kernel_map + .get(key) + .unwrap_or_else(|| panic!("Kernel stats missing key: {child_path}")); + assert_stats_match(spark_child, kernel_child, &child_path); + } + } + (serde_json::Value::Number(s), serde_json::Value::Number(k)) => { + let sv = s.as_f64().unwrap(); + let kv = k.as_f64().unwrap(); + assert!( + (sv - kv).abs() < 1e-6, + "Numeric mismatch at {path}: spark={sv}, kernel={kv}" + ); + } + (serde_json::Value::String(s), serde_json::Value::String(k)) => { + // Spark uses "Z" suffix for TZ timestamps and no suffix for NTZ. + let s_normalized = s.trim_end_matches('Z').trim_end_matches("+00:00"); + let k_normalized = k.trim_end_matches('Z').trim_end_matches("+00:00"); + + // Spark (Jackson) always includes fractional seconds (e.g., ".000") while Arrow's + // JSON encoder omits them when they are zero. Strip zero-only fractional parts + // so both formats compare equal. + if s_normalized.contains('T') && k_normalized.contains('T') { + let normalize_ts = |ts: &str| -> String { + if let Some(dot_pos) = ts.rfind('.') { + let frac = &ts[dot_pos + 1..]; + if frac.chars().all(|c| c == '0') { + return ts[..dot_pos].to_string(); + } + let trimmed = frac.trim_end_matches('0'); + return format!("{}.{trimmed}", &ts[..dot_pos]); + } + ts.to_string() + }; + let s_norm = normalize_ts(s_normalized); + let k_norm = normalize_ts(k_normalized); + assert_eq!( + s_norm, k_norm, + "Timestamp mismatch at {path}: spark={s}, kernel={k}" + ); + } else { + assert_eq!(s, k, "String mismatch at {path}: spark={s}, kernel={k}"); + } + } + _ => { + assert_eq!( + spark_val, kernel_val, + "Value mismatch at {path}: spark={spark_val}, kernel={kernel_val}" + ); + } + } + } + + // Verify that the `assert_stats_match` test helper correctly accepts equivalent values. + #[test] + fn test_assert_stats_match_accepts_equivalent_values() { + // Extra kernel keys are ignored + let spark = serde_json::json!({"a": 1, "b": "hello"}); + let kernel = serde_json::json!({"a": 1, "b": "hello", "extra": true}); + assert_stats_match(&spark, &kernel, ""); + + // Nested objects with extra kernel keys + let spark = serde_json::json!({"outer": {"inner": 42}}); + let kernel = serde_json::json!({"outer": {"inner": 42, "extra": 0}}); + assert_stats_match(&spark, &kernel, ""); + + // Timestamp with trailing ".000Z" vs no fractional part + let spark = serde_json::json!({"ts": "2023-06-15T12:30:00.000Z"}); + let kernel = serde_json::json!({"ts": "2023-06-15T12:30:00Z"}); + assert_stats_match(&spark, &kernel, ""); + + // Timestamp NTZ (no Z suffix) with trailing ".000" + let spark = serde_json::json!({"ts": "2023-06-15T12:30:00.000"}); + let kernel = serde_json::json!({"ts": "2023-06-15T12:30:00"}); + assert_stats_match(&spark, &kernel, ""); + + // Non-zero fractional seconds with different trailing zeros + let spark = serde_json::json!({"ts": "2023-06-15T12:30:00.500Z"}); + let kernel = serde_json::json!({"ts": "2023-06-15T12:30:00.5Z"}); + assert_stats_match(&spark, &kernel, ""); + } + + // Verify that the `assert_stats_match` test helper correctly rejects mismatched values. + #[test] + fn test_assert_stats_match_rejects_mismatches() { + let result = std::panic::catch_unwind(|| { + let spark = serde_json::json!({"a": 1}); + let kernel = serde_json::json!({"b": 1}); + assert_stats_match(&spark, &kernel, ""); + }); + assert!(result.is_err(), "should panic on missing key"); + + let result = std::panic::catch_unwind(|| { + let spark = serde_json::json!({"val": 1.0}); + let kernel = serde_json::json!({"val": 2.0}); + assert_stats_match(&spark, &kernel, ""); + }); + assert!(result.is_err(), "should panic on numeric mismatch"); + + let result = std::panic::catch_unwind(|| { + let spark = serde_json::json!({"s": "alpha"}); + let kernel = serde_json::json!({"s": "beta"}); + assert_stats_match(&spark, &kernel, ""); + }); + assert!(result.is_err(), "should panic on string mismatch"); + } + + /// Loads the PySpark-generated `stats-writing-all-types` fixture: the whole parquet file as one + /// batch, and Spark's own stats for it, taken from the `add` action of commit 1. The fixture + /// covers every supported stat type. + fn load_spark_all_types_fixture() -> (RecordBatch, serde_json::Value) { + let test_path = std::path::Path::new(env!("CARGO_MANIFEST_DIR")) + .join("../kernel/tests/data/stats-writing-all-types/delta"); + let commit_path = test_path + .join("_delta_log") + .join("00000000000000000001.json"); + let commit_data = std::fs::read_to_string(&commit_path).expect("read commit 1 json"); + + let add = commit_data + .lines() + .filter_map(|line| { + let action: serde_json::Value = + serde_json::from_str(line).expect("parse JSON line"); + action.get("add").cloned() + }) + .next() + .expect("commit 1 has an add action"); + let spark_stats: serde_json::Value = + serde_json::from_str(add["stats"].as_str().expect("stats str")) + .expect("parse Spark stats JSON"); + + let parquet_file_path = test_path.join(add["path"].as_str().expect("path str")); + let file = std::fs::File::open(&parquet_file_path).expect("open parquet file"); + let builder = + ParquetRecordBatchReaderBuilder::try_new(file).expect("parquet reader builder"); + let schema = builder.schema().clone(); + let batches: Vec = builder + .build() + .expect("build parquet reader") + .map(|b| b.expect("read batch")) + .collect(); + let whole = concat_batches(&schema, &batches).expect("concat batches"); + (whole, spark_stats) + } + + /// Asserts kernel's stats agree with Spark's on numRecords and every key Spark published. + fn assert_matches_spark_stats(kernel_stats: &StructArray, spark_stats: &serde_json::Value) { + let json_array = to_json(kernel_stats).expect("convert stats to JSON"); + let json_strings = json_array.as_string::(); + assert_eq!(json_strings.len(), 1, "should have exactly one stats row"); + let kernel_stats: serde_json::Value = + serde_json::from_str(json_strings.value(0)).expect("parse kernel stats JSON"); + + assert_eq!( + spark_stats[NUM_RECORDS], kernel_stats[NUM_RECORDS], + "numRecords mismatch" + ); + for section in &[NULL_COUNT, MIN_VALUES, MAX_VALUES] { + if let Some(spark_section) = spark_stats.get(*section) { + let kernel_section = kernel_stats + .get(*section) + .unwrap_or_else(|| panic!("Kernel stats missing {section}")); + assert_stats_match(spark_section, kernel_section, section); + } + } + } + + #[test] + fn test_collect_stats_matches_spark() { + let (whole, spark_stats) = load_spark_all_types_fixture(); + let stats_columns = extract_leaf_columns(whole.schema().fields(), &[]); + let stats = collect_stats(&whole, &stats_columns).expect("collect stats"); + assert_matches_spark_stats(&stats, &spark_stats); + } + + #[test] + fn test_file_stats_accumulator_nested_struct() { + let inner = Fields::from(vec![ + Field::new("a", DataType::Int64, false), + Field::new("b", DataType::Utf8, true), + ]); + let schema = Arc::new(Schema::new(vec![Field::new( + "s", + DataType::Struct(inner.clone()), + false, + )])); + let physical_schema = StructType::try_from_arrow(schema.as_ref()).unwrap(); + let mk = |a: Vec, b: Vec>| { + let st = StructArray::try_new( + inner.clone(), + vec![ + Arc::new(Int64Array::from(a)) as ArrayRef, + Arc::new(StringArray::from(b)) as ArrayRef, + ], + None, + ) + .unwrap(); + RecordBatch::try_new(schema.clone(), vec![Arc::new(st) as ArrayRef]).unwrap() + }; + let b1 = mk(vec![10, 5], vec![Some("m"), Some("z")]); + let b2 = mk(vec![2, 8], vec![Some("a"), None]); + + let mut acc = FileStatsAccumulator::new( + &[column_name!("s.a"), column_name!("s.b")], + &physical_schema, + ); + acc.merge(&b1).unwrap(); + acc.merge(&b2).unwrap(); + let merged = acc.finish().unwrap().unwrap(); + + assert_eq!(get_stat::(&merged, MIN_VALUES, "s", "a"), 2); + assert_eq!(get_stat::(&merged, MAX_VALUES, "s", "a"), 10); + assert_eq!(get_stat::(&merged, NULL_COUNT, "s", "b"), 1); + let nested_b = + |section| string_leaf(child_struct(child_struct(&merged, section), "s"), "b"); + assert_eq!(nested_b(MIN_VALUES), Some("a")); + assert_eq!(nested_b(MAX_VALUES), Some("z")); + } + + #[test] + fn test_file_stats_accumulator_preserves_decimal() { + let dt = DataType::Decimal128(10, 2); + let schema = Arc::new(Schema::new(vec![Field::new("d", dt.clone(), false)])); + let physical_schema = StructType::try_from_arrow(schema.as_ref()).unwrap(); + let mk = |vals: Vec| { + let arr = Decimal128Array::from(vals) + .with_precision_and_scale(10, 2) + .unwrap(); + RecordBatch::try_new(schema.clone(), vec![Arc::new(arr) as ArrayRef]).unwrap() + }; + let mut acc = FileStatsAccumulator::new(&[column_name!("d")], &physical_schema); + acc.merge(&mk(vec![500, 100])).unwrap(); + acc.merge(&mk(vec![50, 900])).unwrap(); + let merged = acc.finish().unwrap().unwrap(); + + assert_eq!( + child_struct(&merged, MIN_VALUES) + .column_by_name("d") + .unwrap() + .data_type(), + &dt + ); + assert_eq!(flat_stat::(&merged, MIN_VALUES, "d"), 50); + assert_eq!(flat_stat::(&merged, MAX_VALUES, "d"), 900); + } + + // NaN wins max and loses min, matching `compute_leaf_agg`. + #[test] + fn test_file_stats_accumulator_float_nan_ordering() { + let schema = Arc::new(Schema::new(vec![Field::new("f", DataType::Float64, true)])); + let physical_schema = StructType::try_from_arrow(schema.as_ref()).unwrap(); + let mk = |vals: Vec| { + RecordBatch::try_new( + schema.clone(), + vec![Arc::new(Float64Array::from(vals)) as ArrayRef], + ) + .unwrap() + }; + let mut acc = FileStatsAccumulator::new(&[column_name!("f")], &physical_schema); + acc.merge(&mk(vec![1.0, 2.0])).unwrap(); + acc.merge(&mk(vec![f64::NAN, 0.5])).unwrap(); + let merged = acc.finish().unwrap().unwrap(); + + assert_eq!(flat_stat::(&merged, MIN_VALUES, "f"), 0.5); + assert!(flat_stat::(&merged, MAX_VALUES, "f").is_nan()); + } + + #[rstest::rstest] + #[case::ascii( + "a".repeat(50), + "b".repeat(50), + Some("a".repeat(32)), + Some(format!("{}\x7F", "b".repeat(32))) + )] + // Over 32 bytes but under 32 chars: `truncate_max_string` gates on byte length but searches for + // the truncation point by char count, so the range is empty and no max is published. + #[case::no_max_bound_published( + "\u{6f22}".repeat(11), + "a".to_string(), + Some("a".to_string()), + None + )] + #[case::multibyte_min( + "a".repeat(32), + format!("{}\u{3042}", "a".repeat(31)), + Some("a".repeat(32)), + Some(format!("{}\u{3042}", "a".repeat(31))) + )] + fn test_file_stats_accumulator_string_stats_match_single_shot( + #[case] v1: String, + #[case] v2: String, + #[case] expected_min: Option, + #[case] expected_max: Option, + ) { + let schema = Arc::new(Schema::new(vec![Field::new("t", DataType::Utf8, false)])); + let physical_schema = StructType::try_from_arrow(schema.as_ref()).unwrap(); + let mk = |vals: Vec<&str>| { + RecordBatch::try_new( + schema.clone(), + vec![Arc::new(StringArray::from(vals)) as ArrayRef], + ) + .unwrap() + }; + let whole = mk(vec![v1.as_str(), v2.as_str()]); + let single = collect_stats(&whole, &[column_name!("t")]).unwrap(); + + let mut acc = FileStatsAccumulator::new(&[column_name!("t")], &physical_schema); + acc.merge(&mk(vec![v1.as_str()])).unwrap(); + acc.merge(&mk(vec![v2.as_str()])).unwrap(); + let merged = acc.finish().unwrap().unwrap(); + + let value = |st: &StructArray, sect: &str| -> Option { + string_leaf(child_struct(st, sect), "t").map(str::to_string) + }; + assert_eq!(value(&merged, MIN_VALUES), value(&single, MIN_VALUES)); + assert_eq!(value(&merged, MAX_VALUES), value(&single, MAX_VALUES)); + assert_eq!(value(&merged, MIN_VALUES), expected_min); + assert_eq!(value(&merged, MAX_VALUES), expected_max); + + if let Some(min) = value(&merged, MIN_VALUES) { + assert!(min.as_str() <= v1.as_str() && min.as_str() <= v2.as_str()); + } + if let Some(max) = value(&merged, MAX_VALUES) { + assert!(max.as_str() >= v1.as_str() && max.as_str() >= v2.as_str()); + } + } + + // Splits past the fixture's row count exercise merging empty row groups. + #[rstest::rstest] + fn test_file_stats_accumulator_matches_spark(#[values(1, 2, 3, 4, 6)] n_row_groups: usize) { + let (whole, spark_stats) = load_spark_all_types_fixture(); + let schema = whole.schema(); + let stats_columns = extract_leaf_columns(schema.fields(), &[]); + let physical_schema = StructType::try_from_arrow(schema.as_ref()).unwrap(); + + let mut acc = FileStatsAccumulator::new(&stats_columns, &physical_schema); + let rows_per_group = whole.num_rows().div_ceil(n_row_groups); + for i in 0..n_row_groups { + let offset = (i * rows_per_group).min(whole.num_rows()); + let len = rows_per_group.min(whole.num_rows() - offset); + acc.merge(&whole.slice(offset, len)) + .unwrap_or_else(|e| panic!("merge row group {i}: {e}")); + } + let merged = acc.finish().unwrap().expect("merged stats"); + + assert_matches_spark_stats(&merged, &spark_stats); + // Spark's JSON omits bounds it did not publish and never carries tightBounds, so only + // single-shot collection covers those. + let single = super::collect_stats(&whole, &stats_columns, &physical_schema).unwrap(); + assert_eq!( + to_json(&merged).unwrap().as_string::().value(0), + to_json(&single).unwrap().as_string::().value(0), + ); + } + + // A zero-row row group still counts as merged: it contributes numRecords and null bounds, so + // only an accumulator that merged nothing yields `None`. + #[rstest::rstest] + #[case::no_row_groups(vec![], None, None)] + #[case::one_row_group(vec![vec![1i64, 2, 3]], Some(3), Some((1, 3)))] + #[case::two_row_groups(vec![vec![1i64], vec![2i64]], Some(2), Some((1, 2)))] + #[case::only_zero_row_group(vec![vec![]], Some(0), None)] + #[case::zero_row_group_between(vec![vec![1i64], vec![], vec![2i64]], Some(2), Some((1, 2)))] + fn test_file_stats_accumulator_finish_counts_and_bounds( + #[case] row_groups: Vec>, + #[case] expected_num_records: Option, + #[case] expected_bounds: Option<(i64, i64)>, + ) { + let schema = Arc::new(Schema::new(vec![Field::new("n", DataType::Int64, false)])); + let physical_schema = StructType::try_from_arrow(schema.as_ref()).unwrap(); + let mut acc = FileStatsAccumulator::new(&[column_name!("n")], &physical_schema); + for rows in &row_groups { + let batch = RecordBatch::try_new( + schema.clone(), + vec![Arc::new(Int64Array::from(rows.clone()))], + ) + .unwrap(); + acc.merge(&batch).expect("merge must succeed"); + } + let merged = acc.finish().expect("finish must succeed"); + let num_records = merged.as_ref().map(|stats| { + stats + .column_by_name(NUM_RECORDS) + .unwrap() + .as_primitive::() + .value(0) + }); + assert_eq!(num_records, expected_num_records); + + let Some(stats) = merged else { return }; + let bound = |section: &str| { + let leaf = child_struct(&stats, section) + .column_by_name("n") + .unwrap() + .as_primitive::(); + leaf.is_valid(0).then(|| leaf.value(0)) + }; + assert_eq!(bound(MIN_VALUES), expected_bounds.map(|(min, _)| min)); + assert_eq!(bound(MAX_VALUES), expected_bounds.map(|(_, max)| max)); + assert_eq!(flat_stat::(&stats, NULL_COUNT, "n"), 0); + assert!(stats + .column_by_name(TIGHT_BOUNDS) + .unwrap() + .as_boolean() + .value(0)); + } + + // Every Arrow string type takes its own arm in `compute_leaf_agg`. + #[rstest::rstest] + fn test_file_stats_accumulator_string_types_match_single_shot( + #[values(DataType::Utf8, DataType::LargeUtf8, DataType::Utf8View)] string_type: DataType, + ) { + let schema = Arc::new(Schema::new(vec![Field::new( + "t", + string_type.clone(), + true, + )])); + let physical_schema = StructType::try_from_arrow(schema.as_ref()).unwrap(); + let mk = |vals: Vec>| { + let array: ArrayRef = match string_type { + DataType::Utf8 => Arc::new(StringArray::from(vals)), + DataType::LargeUtf8 => Arc::new(LargeStringArray::from(vals)), + DataType::Utf8View => Arc::new(StringViewArray::from(vals)), + ref other => panic!("unexpected string type {other}"), + }; + RecordBatch::try_new(schema.clone(), vec![array]).unwrap() + }; + // Long enough to be truncated, so the merge must compose raw values, not truncated ones. + let long_max = "z".repeat(40); + let rg1 = vec![Some("apple"), None]; + let rg2 = vec![Some(long_max.as_str()), Some("banana")]; + + let mut acc = FileStatsAccumulator::new(&[column_name!("t")], &physical_schema); + acc.merge(&mk(rg1.clone())).unwrap(); + acc.merge(&mk(rg2.clone())).unwrap(); + let merged = acc.finish().unwrap().unwrap(); + + let whole = mk([rg1, rg2].concat()); + let single = collect_stats(&whole, &[column_name!("t")]).unwrap(); + + assert_eq!( + to_json(&merged).unwrap().as_string::().value(0), + to_json(&single).unwrap().as_string::().value(0), + ); + } + + // A malformed stats tree is a kernel bug, not bad caller input, so the guards report + // `InternalError`. + fn assert_internal_error(result: DeltaResult, needle: &str) { + let err = result.expect_err("must be rejected"); + // `Error::internal_error` captures a backtrace, which wraps the variant in `Backtraced`. + let mut variant = &err; + while let Error::Backtraced { source, .. } = variant { + variant = source; + } + assert!( + matches!(variant, Error::InternalError(_)), + "expected an internal error, got: {err}" + ); + assert!( + err.to_string().contains(needle), + "error {err} does not mention {needle}" + ); + } + + #[test] + fn test_reduce_stats_rejects_unexpected_field() { + let stats = StructArray::try_new( + vec![Field::new("bogusSection", DataType::Int64, true)].into(), + vec![Arc::new(Int64Array::from(vec![1])) as ArrayRef], + None, + ) + .unwrap(); + assert_internal_error(reduce_stats(&stats), "unknown stats section"); + } + + #[test] + fn test_reduce_stats_children_rejects_non_struct() { + let leaf = Arc::new(Int64Array::from(vec![1])) as ArrayRef; + assert_internal_error( + reduce_stats_children(&leaf, &reduce_count_leaf), + "expected struct in stats sub-tree", + ); + } + + #[test] + fn test_leaf_reducers_reject_wrong_type() { + let strings = Arc::new(StringArray::from(vec!["a"])) as ArrayRef; + assert_internal_error(reduce_count_leaf(&strings), "expected Int64 count leaf"); + assert_internal_error( + reduce_bool_and_leaf(&strings), + "expected Boolean tightBounds leaf", + ); + } + + // Neither reducer may invent a value: a null count is unknown, not zero, and a null + // tightBounds must not be assumed tight. + #[test] + fn test_leaf_reducers_reject_null_leaves() { + let counts = Arc::new(Int64Array::from(vec![Some(1), None])) as ArrayRef; + assert_internal_error(reduce_count_leaf(&counts), "null count leaf"); + let tight = Arc::new(BooleanArray::from(vec![Some(true), None])) as ArrayRef; + assert_internal_error(reduce_bool_and_leaf(&tight), "null tightBounds leaf"); + } + + #[rstest::rstest] + #[case::all_tight(vec![true, true], true)] + #[case::one_wide(vec![true, false, true], false)] + #[case::all_wide(vec![false, false], false)] + fn test_reduce_bool_and_leaf_ands_across_row_groups( + #[case] row_groups: Vec, + #[case] expected: bool, + ) { + let leaf = Arc::new(BooleanArray::from(row_groups)) as ArrayRef; + let merged = reduce_bool_and_leaf(&leaf).unwrap(); + assert_eq!(merged.as_boolean().value(0), expected); + } + + fn batch_of(fields: Vec, columns: Vec) -> RecordBatch { + RecordBatch::try_new(Arc::new(Schema::new(fields)), columns).unwrap() + } + + fn i64s(values: &[i64]) -> ArrayRef { + Arc::new(Int64Array::from(values.to_vec())) + } + + // A one-row batch with one struct column `s` whose fields are named in the given order. + fn nested_batch(field_names: [&str; 2], values: [i64; 2]) -> RecordBatch { + let fields: Fields = field_names + .iter() + .map(|name| Field::new(*name, DataType::Int64, true)) + .collect(); + let columns = values.iter().map(|v| i64s(&[*v])).collect(); + let struct_array = Arc::new(StructArray::try_new(fields.clone(), columns, None).unwrap()); + batch_of( + vec![Field::new("s", DataType::Struct(fields), true)], + vec![struct_array], + ) + } + + fn i64_list_batch(values: &[i64]) -> RecordBatch { + let item = Arc::new(Field::new("item", DataType::Int64, true)); + let offsets = OffsetBuffer::new(vec![0, values.len() as i32].into()); + let list = ListArray::new(item.clone(), offsets, i64s(values), None); + batch_of( + vec![Field::new("c", DataType::List(item), true)], + vec![Arc::new(list)], + ) + } + + // Each case differs only in ways the derived stats shape would not catch. + #[rstest::rstest] + #[case::reordered_columns( + batch_of( + vec![Field::new("a", DataType::Int64, true), Field::new("b", DataType::Int64, true)], + vec![i64s(&[1, 2]), i64s(&[100, 200])], + ), + batch_of( + vec![Field::new("b", DataType::Int64, true), Field::new("a", DataType::Int64, true)], + vec![i64s(&[100, 200]), i64s(&[1, 2])], + ), + vec![column_name!("a"), column_name!("b")] + )] + #[case::reordered_nested_fields( + nested_batch(["x", "y"], [1, 2]), + nested_batch(["y", "x"], [3, 4]), + vec![column_name!("s", "x"), column_name!("s", "y")] + )] + #[case::extra_non_stats_column( + batch_of(vec![Field::new("a", DataType::Int64, true)], vec![i64s(&[1, 2])]), + batch_of( + vec![Field::new("a", DataType::Int64, true), Field::new("b", DataType::Utf8, true)], + vec![i64s(&[3, 4]), Arc::new(StringArray::from(vec!["x", "y"])) as ArrayRef], + ), + vec![column_name!("a")] + )] + #[case::nullability_only( + batch_of(vec![Field::new("a", DataType::Int64, true)], vec![i64s(&[1, 2])]), + batch_of(vec![Field::new("a", DataType::Int64, false)], vec![i64s(&[3, 4])]), + vec![column_name!("a")] + )] + // Both derive an Int64 nullCount leaf, but `Null` counts rows while a list counts nulls. + #[case::void_column_vs_list_column( + batch_of( + vec![Field::new("c", DataType::Null, true)], + vec![Arc::new(NullArray::new(3)) as ArrayRef], + ), + i64_list_batch(&[1, 2, 3]), + vec![column_name!("c")] + )] + fn test_file_stats_accumulator_rejects_differing_batch_schema( + #[case] first: RecordBatch, + #[case] second: RecordBatch, + #[case] stats_columns: Vec, + ) { + let physical_schema = StructType::try_from_arrow(first.schema().as_ref()).unwrap(); + let mut acc = FileStatsAccumulator::new(&stats_columns, &physical_schema); + acc.merge(&first).expect("first merge must succeed"); + let err = acc + .merge(&second) + .expect_err("a batch whose schema differs must be rejected"); + assert!( + matches!(err, Error::Schema(_)), + "expected a schema error, got: {err}" + ); + // A failed merge is terminal, so the first row group can never be published on its own. + assert_result_error_with_message(acc.merge(&first), FAILED); + let err = acc + .finish() + .expect_err("an accumulator that failed a merge must not publish statistics"); + // Its own variant, so callers can match the failure without matching on the message. + assert!( + matches!(&err, Error::StatsValidation(msg) if msg == FAILED), + "expected a stats validation error, got: {err}" + ); + } + + #[test] + fn test_file_stats_accumulator_rejects_differing_stats_shape() { + let declared = nested_batch(["x", "y"], [1, 2]); + // `with_match_field_names(false)` is the only way to get a schema-equal batch whose nested + // field names differ. + let renamed = RecordBatch::try_new_with_options( + declared.schema(), + nested_batch(["p", "q"], [3, 4]).columns().to_vec(), + &RecordBatchOptions::new().with_match_field_names(false), + ) + .unwrap(); + assert_eq!(declared.schema(), renamed.schema()); + + let physical_schema = StructType::try_from_arrow(declared.schema().as_ref()).unwrap(); + let mut acc = FileStatsAccumulator::new( + &[column_name!("s", "x"), column_name!("s", "y")], + &physical_schema, + ); + acc.merge(&declared).expect("first merge must succeed"); + let err = acc + .merge(&renamed) + .expect_err("a batch deriving a different stats shape must be rejected"); + assert!( + matches!(err, Error::Schema(_)), + "expected a schema error, got: {err}" + ); + assert_result_error_with_message(acc.finish(), FAILED); + } + + // Exists to compile: a connector holds the accumulator in its writer struct, so `merge` must be + // callable through `&mut self`. + #[test] + fn test_file_stats_accumulator_usable_as_struct_field() { + struct Writer { + acc: FileStatsAccumulator, + } + impl Writer { + fn write_row_group(&mut self, batch: &RecordBatch) -> DeltaResult<()> { + self.acc.merge(batch) + } + } + + let schema = Arc::new(Schema::new(vec![Field::new("n", DataType::Int64, false)])); + let physical_schema = StructType::try_from_arrow(schema.as_ref()).unwrap(); + let mut writer = Writer { + acc: FileStatsAccumulator::new(&[column_name!("n")], &physical_schema), + }; + let batch = RecordBatch::try_new(schema, vec![i64s(&[1, 2])]).unwrap(); + writer.write_row_group(&batch).unwrap(); + assert!(writer.acc.finish().unwrap().is_some()); + } + + // Arrow erases interval columns to plain integers, so only the kernel schema can classify them. + #[rstest::rstest] + fn test_file_stats_accumulator_interval_column_gets_null_count_only( + #[values(KernelDataType::INTERVAL_YEAR_MONTH, KernelDataType::INTERVAL_DAY_TIME)] + interval_type: KernelDataType, + ) { + // Year-month intervals are Int32 in Arrow, day-time intervals Int64. + let arrow_type = if interval_type == KernelDataType::INTERVAL_YEAR_MONTH { + DataType::Int32 + } else { + DataType::Int64 + }; + let schema = Arc::new(Schema::new(vec![ + Field::new("span", arrow_type.clone(), true), + Field::new("n", DataType::Int64, true), + ])); + let physical_schema = StructType::try_new([ + StructField::nullable("span", interval_type), + StructField::nullable("n", KernelDataType::LONG), + ]) + .unwrap(); + let mk = |spans: &[i64], ns: &[i64]| { + let span: ArrayRef = match arrow_type { + DataType::Int32 => Arc::new(Int32Array::from( + spans.iter().map(|v| *v as i32).collect::>(), + )), + _ => i64s(spans), + }; + RecordBatch::try_new(schema.clone(), vec![span, i64s(ns)]).unwrap() + }; + + let cols = vec![column_name!("span"), column_name!("n")]; + let mut acc = FileStatsAccumulator::new(&cols, &physical_schema); + acc.merge(&mk(&[5, 1], &[10, 20])).unwrap(); + acc.merge(&mk(&[9], &[30])).unwrap(); + let merged = acc.finish().unwrap().unwrap(); + + assert_eq!(flat_stat::(&merged, NULL_COUNT, "span"), 0); + for section in [MIN_VALUES, MAX_VALUES] { + assert!( + child_struct(&merged, section) + .column_by_name("span") + .is_none(), + "interval column must not appear in {section}" + ); + // The sibling column still gets bounds, so the exclusion is targeted. + assert!(child_struct(&merged, section).column_by_name("n").is_some()); + } + } + + #[test] + fn test_file_stats_accumulator_nested_string_truncates_and_handles_all_nulls() { + let inner = Fields::from(vec![ + Field::new("long", DataType::Utf8, true), + Field::new("empty", DataType::Utf8, true), + ]); + let schema = Arc::new(Schema::new(vec![Field::new( + "s", + DataType::Struct(inner.clone()), + true, + )])); + let physical_schema = StructType::try_from_arrow(schema.as_ref()).unwrap(); + let mk = |long: Vec<&str>| { + let nulls = vec![None as Option<&str>; long.len()]; + let struct_array = StructArray::try_new( + inner.clone(), + vec![ + Arc::new(StringArray::from(long)) as ArrayRef, + Arc::new(StringArray::from(nulls)) as ArrayRef, + ], + None, + ) + .unwrap(); + RecordBatch::try_new(schema.clone(), vec![Arc::new(struct_array) as ArrayRef]).unwrap() + }; + // Both exceed the 32-char prefix and the file-level max comes from the second row group, so + // truncating before reducing would give the wrong bound. + let short_prefix = format!("{}zzz", "a".repeat(40)); + let long_prefix = format!("{}aaa", "b".repeat(40)); + let cols = vec![column_name!("s", "long"), column_name!("s", "empty")]; + + let mut acc = FileStatsAccumulator::new(&cols, &physical_schema); + acc.merge(&mk(vec![short_prefix.as_str()])).unwrap(); + acc.merge(&mk(vec![long_prefix.as_str()])).unwrap(); + let merged = acc.finish().unwrap().unwrap(); + + let whole = mk(vec![short_prefix.as_str(), long_prefix.as_str()]); + let single = collect_stats(&whole, &cols).unwrap(); + let leaf = |stats: &StructArray, section, field| { + string_leaf(child_struct(child_struct(stats, section), "s"), field).map(str::to_string) + }; + + assert_eq!( + leaf(&merged, MIN_VALUES, "long").as_deref(), + Some("a".repeat(32).as_str()) + ); + assert_eq!( + leaf(&merged, MAX_VALUES, "long"), + Some(format!("{}\x7F", "b".repeat(32))) + ); + assert_eq!(leaf(&merged, MIN_VALUES, "empty"), None); + assert_eq!(leaf(&merged, MAX_VALUES, "empty"), None); + assert_eq!(get_stat::(&merged, NULL_COUNT, "s", "empty"), 2); + + for section in [MIN_VALUES, MAX_VALUES] { + for field in ["long", "empty"] { + assert_eq!( + leaf(&merged, section, field), + leaf(&single, section, field), + "{section}.s.{field} must match single-shot collection" + ); + } + } + } +} diff --git a/default-engine/src/stats_verifier_tests.rs b/default-engine/src/stats_verifier_tests.rs new file mode 100644 index 0000000000..cc518c7ec6 --- /dev/null +++ b/default-engine/src/stats_verifier_tests.rs @@ -0,0 +1,703 @@ +//! Round-trip tests that verify `collect_stats` output passes the kernel +//! `StatsColumnVerifier` for every supported logical type, plus pure verifier +//! coverage for the validator's edge cases. These tests live here because +//! `collect_stats` is a default-engine concern; kernel only owns the verifier +//! contract. + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use delta_kernel::arrow::array::types::{ + Date32Type, Decimal128Type, Float32Type, Float64Type, Int32Type, TimestampMicrosecondType, + }; + use delta_kernel::arrow::array::{ + Array, ArrayRef, BinaryArray, BooleanArray, Int16Array, Int64Array, Int8Array, + LargeStringArray, PrimitiveArray, RecordBatch, StringArray, StringViewArray, StructArray, + }; + use delta_kernel::arrow::datatypes::{ + DataType as ArrowDataType, Field as ArrowField, Fields, Schema as ArrowSchema, + }; + use delta_kernel::engine::arrow_data::ArrowEngineData; + use delta_kernel::expressions::column_name; + use delta_kernel::schema::DataType; + use delta_kernel::transaction::stats_verifier::{ + verify_num_records_present, StatsColumnVerifier, + }; + use delta_kernel::{EngineData, Error}; + use rstest::rstest; + + use crate::stats::collect_stats_for_test as collect_stats; + + /// Creates test add file data with stats.numRecords, stats.nullCount.col, + /// stats.minValues.col, and stats.maxValues.col — all of type LONG. + fn create_add_file_batch( + paths: Vec<&str>, + num_records: Vec>, + null_counts: Vec>, + min_values: Vec>, + max_values: Vec>, + ) -> Box { + assert_eq!(paths.len(), num_records.len()); + assert_eq!(paths.len(), null_counts.len()); + assert_eq!(paths.len(), min_values.len()); + assert_eq!(paths.len(), max_values.len()); + + let path_array = StringArray::from(paths.to_vec()); + let col_field = Arc::new(ArrowField::new("col", ArrowDataType::Int64, true)); + + let num_records_array = Int64Array::from(num_records); + let null_count_struct = StructArray::new( + Fields::from(vec![col_field.clone()]), + vec![Arc::new(Int64Array::from(null_counts)) as ArrayRef], + None, + ); + let min_values_struct = StructArray::new( + Fields::from(vec![col_field.clone()]), + vec![Arc::new(Int64Array::from(min_values)) as ArrayRef], + None, + ); + let max_values_struct = StructArray::new( + Fields::from(vec![col_field]), + vec![Arc::new(Int64Array::from(max_values)) as ArrayRef], + None, + ); + + let inner_struct_type = |name: &str| { + ArrowField::new( + name, + ArrowDataType::Struct(Fields::from(vec![ArrowField::new( + "col", + ArrowDataType::Int64, + true, + )])), + true, + ) + }; + + let stats_fields = Fields::from(vec![ + ArrowField::new("numRecords", ArrowDataType::Int64, true), + inner_struct_type("nullCount"), + inner_struct_type("minValues"), + inner_struct_type("maxValues"), + ]); + let stats_struct = StructArray::new( + stats_fields.clone(), + vec![ + Arc::new(num_records_array) as ArrayRef, + Arc::new(null_count_struct) as ArrayRef, + Arc::new(min_values_struct) as ArrayRef, + Arc::new(max_values_struct) as ArrayRef, + ], + None, + ); + + let schema = Arc::new(ArrowSchema::new(vec![ + ArrowField::new("path", ArrowDataType::Utf8, false), + ArrowField::new("stats", ArrowDataType::Struct(stats_fields), true), + ])); + + let batch = + RecordBatch::try_new(schema, vec![Arc::new(path_array), Arc::new(stats_struct)]) + .unwrap(); + + Box::new(ArrowEngineData::new(batch)) + } + + #[test] + fn test_verifier_with_empty_add_files() { + let columns = vec![(column_name!("col"), DataType::LONG)]; + let verifier = StatsColumnVerifier::new(columns); + let result = verifier.verify(&[]); + assert!(result.is_ok()); + } + + #[test] + fn test_verify_valid_stats() { + let batch = create_add_file_batch( + vec!["file1.parquet", "file2.parquet"], + vec![Some(100), Some(100)], + vec![Some(0), Some(5)], + vec![Some(1), Some(10)], + vec![Some(100), Some(50)], + ); + + let columns = vec![(column_name!("col"), DataType::LONG)]; + let verifier = StatsColumnVerifier::new(columns); + let result = verifier.verify(&[batch]); + assert!(result.is_ok()); + } + + #[test] + fn test_verify_missing_stat_category() { + let cases = [ + ("nullCount", vec![None], vec![Some(1)], vec![Some(100)]), + ("minValues", vec![Some(0)], vec![None], vec![Some(100)]), + ("maxValues", vec![Some(0)], vec![Some(1)], vec![None]), + ]; + for (category, null_counts, min_values, max_values) in cases { + let batch = create_add_file_batch( + vec!["file1.parquet"], + vec![Some(100)], + null_counts, + min_values, + max_values, + ); + let verifier = + StatsColumnVerifier::new(vec![(column_name!("col"), DataType::LONG)]); + let err_msg = verifier.verify(&[batch]).unwrap_err().to_string(); + assert!(err_msg.contains("file1.parquet"), "case: {category}"); + assert!(err_msg.contains(category), "case: {category}"); + } + } + + #[test] + fn test_verify_multiple_batches() { + let batch1 = create_add_file_batch( + vec!["good_file.parquet"], + vec![Some(100)], + vec![Some(0)], + vec![Some(1)], + vec![Some(100)], + ); + let batch2 = create_add_file_batch( + vec!["bad_file.parquet"], + vec![Some(100)], + vec![None], + vec![None], + vec![None], + ); + + let columns = vec![(column_name!("col"), DataType::LONG)]; + let verifier = StatsColumnVerifier::new(columns); + let result = verifier.verify(&[batch1, batch2]); + + assert!(result.is_err()); + let err_msg = result.unwrap_err().to_string(); + assert!(err_msg.contains("bad_file.parquet")); + assert!(!err_msg.contains("good_file.parquet")); + } + + #[test] + fn test_verify_no_required_columns() { + let batch = create_add_file_batch( + vec!["file1.parquet"], + vec![Some(100)], + vec![None], + vec![None], + vec![None], + ); + + let verifier = StatsColumnVerifier::new(vec![]); + let result = verifier.verify(&[batch]); + assert!(result.is_ok()); + } + + #[test] + fn test_verify_all_null_column_allows_null_min_max() { + // nullCount == numRecords means all rows are null, so null min/max is valid + let batch = create_add_file_batch( + vec!["file1.parquet"], + vec![Some(100)], + vec![Some(100)], + vec![None], + vec![None], + ); + + let columns = vec![(column_name!("col"), DataType::LONG)]; + let verifier = StatsColumnVerifier::new(columns); + assert!(verifier.verify(&[batch]).is_ok()); + } + + #[test] + fn test_verify_partial_null_column_requires_min_max() { + // nullCount < numRecords means not all rows are null, so min/max must be present + let batch = create_add_file_batch( + vec!["file1.parquet"], + vec![Some(100)], + vec![Some(50)], + vec![None], + vec![None], + ); + + let columns = vec![(column_name!("col"), DataType::LONG)]; + let verifier = StatsColumnVerifier::new(columns); + let result = verifier.verify(&[batch]); + assert!(matches!(result, Err(Error::StatsValidation(_)))); + let err = result.unwrap_err().to_string(); + assert!(err.contains("minValues")); + } + + /// Creates test data with two columns (col_a, col_b) in stats, both LONG. + #[allow(clippy::too_many_arguments)] + fn create_two_column_batch( + paths: Vec<&str>, + num_records: Vec>, + col_a_nullcount: Vec>, + col_a_min: Vec>, + col_a_max: Vec>, + col_b_nullcount: Vec>, + col_b_min: Vec>, + col_b_max: Vec>, + ) -> Box { + let path_array = StringArray::from(paths.to_vec()); + let col_a_field = Arc::new(ArrowField::new("col_a", ArrowDataType::Int64, true)); + let col_b_field = Arc::new(ArrowField::new("col_b", ArrowDataType::Int64, true)); + let both_fields = Fields::from(vec![col_a_field, col_b_field]); + + let make_struct = |a: Vec>, b: Vec>| { + StructArray::new( + both_fields.clone(), + vec![ + Arc::new(Int64Array::from(a)) as ArrayRef, + Arc::new(Int64Array::from(b)) as ArrayRef, + ], + None, + ) + }; + + let num_records_array = Int64Array::from(num_records); + let null_count_struct = make_struct(col_a_nullcount, col_b_nullcount); + let min_values_struct = make_struct(col_a_min, col_b_min); + let max_values_struct = make_struct(col_a_max, col_b_max); + + let inner_type = ArrowDataType::Struct(Fields::from(vec![ + ArrowField::new("col_a", ArrowDataType::Int64, true), + ArrowField::new("col_b", ArrowDataType::Int64, true), + ])); + let stats_fields = Fields::from(vec![ + ArrowField::new("numRecords", ArrowDataType::Int64, true), + ArrowField::new("nullCount", inner_type.clone(), true), + ArrowField::new("minValues", inner_type.clone(), true), + ArrowField::new("maxValues", inner_type, true), + ]); + let stats_struct = StructArray::new( + stats_fields.clone(), + vec![ + Arc::new(num_records_array) as ArrayRef, + Arc::new(null_count_struct) as ArrayRef, + Arc::new(min_values_struct) as ArrayRef, + Arc::new(max_values_struct) as ArrayRef, + ], + None, + ); + + let schema = Arc::new(ArrowSchema::new(vec![ + ArrowField::new("path", ArrowDataType::Utf8, false), + ArrowField::new("stats", ArrowDataType::Struct(stats_fields), true), + ])); + let batch = + RecordBatch::try_new(schema, vec![Arc::new(path_array), Arc::new(stats_struct)]) + .unwrap(); + Box::new(ArrowEngineData::new(batch)) + } + + #[test] + fn test_verify_multiple_columns() { + // Both columns have valid stats + let batch = create_two_column_batch( + vec!["file1.parquet"], + vec![Some(100)], + vec![Some(0)], + vec![Some(1)], + vec![Some(10)], + vec![Some(0)], + vec![Some(2)], + vec![Some(20)], + ); + let columns = vec![ + (column_name!("col_a"), DataType::LONG), + (column_name!("col_b"), DataType::LONG), + ]; + assert!(StatsColumnVerifier::new(columns).verify(&[batch]).is_ok()); + + // col_a valid, col_b missing minValues + let batch = create_two_column_batch( + vec!["file1.parquet"], + vec![Some(100)], + vec![Some(0)], + vec![Some(1)], + vec![Some(10)], + vec![Some(0)], + vec![None], + vec![Some(20)], + ); + let columns = vec![ + (column_name!("col_a"), DataType::LONG), + (column_name!("col_b"), DataType::LONG), + ]; + let err_msg = StatsColumnVerifier::new(columns) + .verify(&[batch]) + .unwrap_err() + .to_string(); + assert!(err_msg.contains("col_b")); + assert!(err_msg.contains("minValues")); + assert!(!err_msg.contains("col_a")); + } + + /// Verifies that stats collected from non-standard Arrow string representations + /// (LargeUtf8/LargeStringArray, Utf8View/StringViewArray) can be validated by + /// StatsColumnVerifier, which expects Delta's logical STRING type. Engines may use any of + /// these representations, and the stats pipeline must handle them without type errors. + #[rstest] + #[case::large_utf8(Arc::new(LargeStringArray::from(vec!["Austin", "Boston", "Chicago"])) as ArrayRef)] + #[case::utf8_view(Arc::new(StringViewArray::from(vec!["Austin", "Boston", "Chicago"])) as ArrayRef)] + fn test_verify_string_stats(#[case] values: ArrayRef) { + let schema = Arc::new(ArrowSchema::new(vec![ArrowField::new( + "city", + values.data_type().clone(), + false, + )])); + let batch = RecordBatch::try_new(schema, vec![values]).unwrap(); + + let stats = collect_stats(&batch, &[column_name!("city")]).unwrap(); + + let path_array = StringArray::from(vec!["file1.parquet"]); + let add_file_schema = Arc::new(ArrowSchema::new(vec![ + ArrowField::new("path", ArrowDataType::Utf8, false), + ArrowField::new("stats", stats.data_type().clone(), true), + ])); + let add_file_batch = RecordBatch::try_new( + add_file_schema, + vec![ + Arc::new(path_array) as ArrayRef, + Arc::new(stats) as ArrayRef, + ], + ) + .unwrap(); + + let engine_data: Box = Box::new(ArrowEngineData::new(add_file_batch)); + + let verifier = + StatsColumnVerifier::new(vec![(column_name!("city"), DataType::STRING)]); + verifier.verify(&[engine_data]).unwrap(); + } + + /// Verify collect_stats produces correct stats shape for all-null and empty batches. + /// These cases keep the column in minValues/maxValues with null values (so that + /// StatsColumnVerifier can find the field via visit_rows and check nullCount == numRecords). + #[rstest] + #[case::all_null_values(Arc::new(Int64Array::from(vec![None::, None, None])) as ArrayRef)] + #[case::empty_batch(Arc::new(Int64Array::from(Vec::>::new())) as ArrayRef)] + fn test_collected_stats_shape_for_all_null_and_empty(#[case] values: ArrayRef) { + let num_rows = values.len(); + let schema = Arc::new(ArrowSchema::new(vec![ArrowField::new( + "col", + values.data_type().clone(), + true, + )])); + let batch = RecordBatch::try_new(schema, vec![values]).unwrap(); + + let stats = collect_stats(&batch, &[column_name!("col")]).unwrap(); + + // numRecords should match row count + let num_records = stats + .column_by_name("numRecords") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(num_records.value(0), num_rows as i64); + + // All-null/empty columns are present in minValues/maxValues with null values + let min_values = stats + .column_by_name("minValues") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert!(min_values.column_by_name("col").unwrap().is_null(0)); + + let max_values = stats + .column_by_name("maxValues") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert!(max_values.column_by_name("col").unwrap().is_null(0)); + } + + /// Round-trip test: collect_stats produces stats that pass verification for every + /// stats-eligible type. Covers non-null, all-null, and empty patterns per type. + #[rstest] + // Note: BOOLEAN and BINARY are omitted for non-null cases because collect_stats does not + // produce min/max for those types (they fall through to the wildcard in compute_leaf_agg). + // All-null cases are still tested since null min/max is valid when nullCount == numRecords. + #[case::boolean_all_null( + Arc::new(BooleanArray::from(vec![None::, None, None])) as ArrayRef, + DataType::BOOLEAN, + )] + #[case::byte( + Arc::new(Int8Array::from(vec![Some(1i8), Some(2), Some(3)])) as ArrayRef, + DataType::BYTE, + )] + #[case::byte_all_null( + Arc::new(Int8Array::from(vec![None::, None, None])) as ArrayRef, + DataType::BYTE, + )] + #[case::short( + Arc::new(Int16Array::from(vec![Some(100i16), Some(200), Some(300)])) as ArrayRef, + DataType::SHORT, + )] + #[case::short_all_null( + Arc::new(Int16Array::from(vec![None::, None, None])) as ArrayRef, + DataType::SHORT, + )] + #[case::integer( + Arc::new(PrimitiveArray::::from(vec![Some(1), Some(2), Some(3)])) as ArrayRef, + DataType::INTEGER, + )] + #[case::integer_all_null( + Arc::new(PrimitiveArray::::from(vec![None::, None, None])) as ArrayRef, + DataType::INTEGER, + )] + #[case::long( + Arc::new(Int64Array::from(vec![Some(1i64), Some(2), Some(3)])) as ArrayRef, + DataType::LONG, + )] + #[case::long_all_null( + Arc::new(Int64Array::from(vec![None::, None, None])) as ArrayRef, + DataType::LONG, + )] + #[case::long_empty( + Arc::new(Int64Array::from(Vec::>::new())) as ArrayRef, + DataType::LONG, + )] + #[case::float( + Arc::new(PrimitiveArray::::from(vec![Some(1.0f32), Some(2.0), Some(3.0)])) as ArrayRef, + DataType::FLOAT, + )] + #[case::float_all_null( + Arc::new(PrimitiveArray::::from(vec![None::, None, None])) as ArrayRef, + DataType::FLOAT, + )] + #[case::double( + Arc::new(PrimitiveArray::::from(vec![Some(1.0f64), Some(2.0), Some(3.0)])) as ArrayRef, + DataType::DOUBLE, + )] + #[case::double_all_null( + Arc::new(PrimitiveArray::::from(vec![None::, None, None])) as ArrayRef, + DataType::DOUBLE, + )] + #[case::date( + Arc::new(PrimitiveArray::::from(vec![Some(18000), Some(19000), Some(20000)])) as ArrayRef, + DataType::DATE, + )] + #[case::date_all_null( + Arc::new(PrimitiveArray::::from(vec![None::, None, None])) as ArrayRef, + DataType::DATE, + )] + #[case::timestamp( + Arc::new(PrimitiveArray::::from(vec![Some(1_000_000i64), Some(2_000_000), Some(3_000_000)])) as ArrayRef, + DataType::TIMESTAMP, + )] + #[case::timestamp_all_null( + Arc::new(PrimitiveArray::::from(vec![None::, None, None])) as ArrayRef, + DataType::TIMESTAMP, + )] + #[case::timestamp_ntz( + Arc::new(PrimitiveArray::::from(vec![Some(1_000_000i64), Some(2_000_000), Some(3_000_000)])) as ArrayRef, + DataType::TIMESTAMP_NTZ, + )] + #[case::timestamp_ntz_all_null( + Arc::new(PrimitiveArray::::from(vec![None::, None, None])) as ArrayRef, + DataType::TIMESTAMP_NTZ, + )] + #[case::string( + Arc::new(StringArray::from(vec![Some("a"), Some("b"), Some("c")])) as ArrayRef, + DataType::STRING, + )] + #[case::string_all_null( + Arc::new(StringArray::from(vec![None::<&str>, None, None])) as ArrayRef, + DataType::STRING, + )] + #[case::binary_all_null( + Arc::new(BinaryArray::from(vec![None::<&[u8]>, None, None])) as ArrayRef, + DataType::BINARY, + )] + #[case::decimal( + Arc::new(PrimitiveArray::::from(vec![Some(100i128), Some(200), Some(300)]).with_precision_and_scale(10, 2).unwrap()) as ArrayRef, + DataType::decimal(10, 2).unwrap(), + )] + #[case::decimal_all_null( + Arc::new(PrimitiveArray::::from(vec![None::, None, None]).with_precision_and_scale(10, 2).unwrap()) as ArrayRef, + DataType::decimal(10, 2).unwrap(), + )] + fn test_collected_stats_pass_verification_all_types( + #[case] values: ArrayRef, + #[case] dt: DataType, + ) { + let schema = Arc::new(ArrowSchema::new(vec![ArrowField::new( + "col", + values.data_type().clone(), + true, + )])); + let batch = RecordBatch::try_new(schema, vec![values]).unwrap(); + + let stats = collect_stats(&batch, &[column_name!("col")]).unwrap(); + + let path_array = StringArray::from(vec!["file1.parquet"]); + let add_file_schema = Arc::new(ArrowSchema::new(vec![ + ArrowField::new("path", ArrowDataType::Utf8, false), + ArrowField::new("stats", stats.data_type().clone(), true), + ])); + let add_file_batch = RecordBatch::try_new( + add_file_schema, + vec![ + Arc::new(path_array) as ArrayRef, + Arc::new(stats) as ArrayRef, + ], + ) + .unwrap(); + + let engine_data: Box = Box::new(ArrowEngineData::new(add_file_batch)); + + let verifier = StatsColumnVerifier::new(vec![(column_name!("col"), dt)]); + verifier.verify(&[engine_data]).unwrap(); + } + + // ============================================================================ + // verify_num_records_present tests + // ============================================================================ + + #[rstest] + #[case::empty_input(vec![], None, vec![])] + #[case::all_present( + vec![vec![("a.parquet", Some(10)), ("b.parquet", Some(20))]], + None/* expected_first_offender */, + vec![]/* later_offenders */, + )] + #[case::first_offender_named_later_offenders_hidden( + vec![vec![("a.parquet", Some(10)), ("b.parquet", None), ("c.parquet", None)]], + Some("b.parquet"), + vec!["c.parquet"], + )] + #[case::short_circuits_across_batches( + vec![ + vec![("a.parquet", None), ("b.parquet", Some(20))], + vec![("c.parquet", Some(30)), ("d.parquet", None)], + ], + Some("a.parquet"), + vec!["d.parquet"], + )] + fn test_verify_num_records_present( + #[case] batches: Vec)>>, + #[case] expected_first_offender: Option<&str>, + #[case] later_offenders: Vec<&str>, + ) { + let batches: Vec> = batches + .into_iter() + .map(|rows| { + let (paths, num_records): (Vec<_>, Vec<_>) = rows.into_iter().unzip(); + create_add_file_batch_with_num_records(paths, num_records) + }) + .collect(); + let result = verify_num_records_present(&batches); + match expected_first_offender { + None => result.unwrap(), + Some(path) => { + let err = result.unwrap_err().to_string(); + assert!( + err.contains("'stats.numRecords' is required") && err.contains(path), + "expected error containing '{path}', but got: {err}", + ); + for later_offender in &later_offenders { + assert!( + !err.contains(later_offender), + "error should not mention '{later_offender}': {err}", + ); + } + } + } + } + + #[test] + fn test_verify_num_records_present_flags_add_without_stats() { + let batch = create_add_file_batch_with_stats_mask( + vec!["a.parquet", "b.parquet"], + vec![true, false], + ); + let err = verify_num_records_present(&[batch]) + .unwrap_err() + .to_string(); + assert!( + err.contains("'stats.numRecords' is required") && err.contains("b.parquet"), + "expected error naming b.parquet, got: {err}", + ); + assert!( + !err.contains("a.parquet"), + "row with stats present should not be flagged: {err}", + ); + } + + fn create_add_file_batch_with_num_records( + paths: Vec<&str>, + num_records: Vec>, + ) -> Box { + let n = paths.len(); + create_add_file_batch( + paths, + num_records, + vec![None; n], + vec![None; n], + vec![None; n], + ) + } + + /// Build an add-file batch where rows with `stats_present[i] == false` carry no `stats` + /// at all; other rows get `numRecords = 10`. + fn create_add_file_batch_with_stats_mask( + paths: Vec<&str>, + stats_present: Vec, + ) -> Box { + assert_eq!(paths.len(), stats_present.len()); + let n = paths.len(); + let num_records: Vec> = stats_present.iter().map(|p| p.then_some(10)).collect(); + let path_array = StringArray::from(paths); + + let col_field = Arc::new(ArrowField::new("col", ArrowDataType::Int64, true)); + let inner_struct_type = |name: &str| { + ArrowField::new( + name, + ArrowDataType::Struct(Fields::from(vec![ArrowField::new( + "col", + ArrowDataType::Int64, + true, + )])), + true, + ) + }; + let stats_fields = Fields::from(vec![ + ArrowField::new("numRecords", ArrowDataType::Int64, true), + inner_struct_type("nullCount"), + inner_struct_type("minValues"), + inner_struct_type("maxValues"), + ]); + let zero_inner_struct_array = StructArray::new( + Fields::from(vec![col_field]), + vec![Arc::new(Int64Array::from(vec![None as Option; n])) as ArrayRef], + None, /* null buffer */ + ); + let stats_struct_array = StructArray::new( + stats_fields.clone(), + vec![ + Arc::new(Int64Array::from(num_records)) as ArrayRef, + Arc::new(zero_inner_struct_array.clone()) as ArrayRef, + Arc::new(zero_inner_struct_array.clone()) as ArrayRef, + Arc::new(zero_inner_struct_array) as ArrayRef, + ], + // null buffer: rows with stats_present[i] == false are null + Some(stats_present.into_iter().collect()), + ); + let schema = Arc::new(ArrowSchema::new(vec![ + ArrowField::new("path", ArrowDataType::Utf8, false), + ArrowField::new("stats", ArrowDataType::Struct(stats_fields), true), + ])); + let batch = RecordBatch::try_new( + schema, + vec![Arc::new(path_array), Arc::new(stats_struct_array)], + ) + .unwrap(); + Box::new(ArrowEngineData::new(batch)) + } +} diff --git a/kernel/src/engine/default/storage.rs b/default-engine/src/storage.rs similarity index 82% rename from kernel/src/engine/default/storage.rs rename to default-engine/src/storage.rs index 3c17c463d2..c7bf7bf8b9 100644 --- a/kernel/src/engine/default/storage.rs +++ b/default-engine/src/storage.rs @@ -1,12 +1,11 @@ use std::collections::HashMap; use std::sync::{Arc, LazyLock, RwLock}; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::{self, Error, ObjectStore}; +use delta_kernel::Error as DeltaError; use url::Url; -use crate::object_store::path::Path; -use crate::object_store::{self, Error, ObjectStore}; -use crate::Error as DeltaError; - /// Alias for convenience type ClosureReturn = Result<(Box, Path), Error>; /// This type alias makes it easier to reference the handler closure(s) @@ -21,8 +20,9 @@ type Handlers = HashMap; static URL_REGISTRY: LazyLock> = LazyLock::new(|| RwLock::new(HashMap::default())); /// Insert a new URL handler for [store_from_url_opts] with the given `scheme`. This allows -/// users to provide their own custom URL handler to plug new [crate::object_store::ObjectStore] -/// instances into delta-kernel, which is used by [store_from_url_opts] to parse the URL. +/// users to provide their own custom URL handler to plug new +/// [delta_kernel::object_store::ObjectStore] instances into delta-kernel, which is used by +/// [store_from_url_opts] to parse the URL. pub fn insert_url_handler( scheme: impl AsRef, handler_closure: HandlerClosure, @@ -38,7 +38,7 @@ pub fn insert_url_handler( /// Create an [`ObjectStore`] from a URL. /// -/// Returns an `Arc` ready to use with [`crate::engine::default::DefaultEngine`]. +/// Returns an `Arc` ready to use with [`crate::DefaultEngine`]. /// /// This function checks for custom URL handlers registered via [`insert_url_handler`] /// before falling back to [`object_store`]'s default behavior. @@ -47,7 +47,7 @@ pub fn insert_url_handler( /// /// ```rust /// # use url::Url; -/// # use delta_kernel::engine::default::storage::store_from_url; +/// # use delta_kernel_default_engine::storage::store_from_url; /// # use delta_kernel::DeltaResult; /// # fn example() -> DeltaResult<()> { /// let url = Url::parse("file:///path/to/table")?; @@ -55,13 +55,13 @@ pub fn insert_url_handler( /// # Ok(()) /// # } /// ``` -pub fn store_from_url(url: &Url) -> crate::DeltaResult> { +pub fn store_from_url(url: &Url) -> delta_kernel::DeltaResult> { store_from_url_opts(url, std::iter::empty::<(&str, &str)>()) } /// Create an [`ObjectStore`] from a URL with custom options. /// -/// Returns an `Arc` ready to use with [`crate::engine::default::DefaultEngine`]. +/// Returns an `Arc` ready to use with [`crate::DefaultEngine`]. /// /// This function checks for custom URL handlers registered via [`insert_url_handler`] /// before falling back to [`object_store`]'s default behavior. @@ -71,7 +71,7 @@ pub fn store_from_url(url: &Url) -> crate::DeltaResult> { /// ```rust /// # use url::Url; /// # use std::collections::HashMap; -/// # use delta_kernel::engine::default::storage::store_from_url_opts; +/// # use delta_kernel_default_engine::storage::store_from_url_opts; /// # use delta_kernel::DeltaResult; /// # fn example() -> DeltaResult<()> { /// let url = Url::parse("s3://my-bucket/path/to/table")?; @@ -83,14 +83,14 @@ pub fn store_from_url(url: &Url) -> crate::DeltaResult> { pub fn store_from_url_opts( url: &Url, options: I, -) -> crate::DeltaResult> +) -> delta_kernel::DeltaResult> where I: IntoIterator, K: AsRef, V: Into, { // First attempt to use any schemes registered via insert_url_handler, - // falling back to the default behavior of crate::object_store::parse_url_opts + // falling back to the default behavior of delta_kernel::object_store::parse_url_opts let (store, _path) = if let Ok(handlers) = URL_REGISTRY.read() { if let Some(handler) = handlers.get(url.scheme()) { let options = options @@ -110,16 +110,20 @@ where #[cfg(test)] mod tests { - use super::*; + use std::collections::HashMap; - use crate::object_store::{self, path::Path}; + use delta_kernel::object_store::path::Path; + use delta_kernel::object_store::{self, ObjectStore}; use hdfs_native_object_store::HdfsObjectStoreBuilder; + use super::{insert_url_handler, store_from_url_opts, URL_REGISTRY}; + use crate::*; + /// Example funciton of doing testing of a custom [HdfsObjectStore] construction fn parse_url_opts_hdfs_native( url: &Url, options: I, - ) -> Result<(Box, Path), Error> + ) -> Result<(Box, Path), object_store::Error> where I: IntoIterator, K: AsRef, @@ -159,7 +163,10 @@ mod tests { // to connect to, so the only way to really verify that we got the object store we // expected is to inspect the `store` on the error v_v match store_from_url_opts(&url, options) { - Err(crate::Error::ObjectStore(object_store::Error::Generic { store, source: _ })) => { + Err(delta_kernel::Error::ObjectStore(object_store::Error::Generic { + store, + source: _, + })) => { assert_eq!(store, "HdfsObjectStore"); } Err(unexpected) => panic!("Unexpected error happened: {unexpected:?}"), diff --git a/default-engine/test-utils/Cargo.toml b/default-engine/test-utils/Cargo.toml new file mode 100644 index 0000000000..d990cb9ab2 --- /dev/null +++ b/default-engine/test-utils/Cargo.toml @@ -0,0 +1,15 @@ +[package] +name = "delta_kernel_default_engine_test_utils" +description = "Shared test helpers for delta_kernel_default_engine's own test suite." +publish = false +edition.workspace = true +license.workspace = true +repository.workspace = true +version.workspace = true +rust-version.workspace = true + +[package.metadata.release] +release = false + +[dependencies] +delta_kernel = { path = "../../kernel" } diff --git a/default-engine/test-utils/src/lib.rs b/default-engine/test-utils/src/lib.rs new file mode 100644 index 0000000000..2d348eed52 --- /dev/null +++ b/default-engine/test-utils/src/lib.rs @@ -0,0 +1,72 @@ +//! Shared test helpers for `delta_kernel_default_engine`'s own test suite. +//! +//! This crate sits inside `default-engine/` as a workspace member so default-engine's tests can +//! pull in common helpers (record-batch conversion, error assertions, etc.) without duplicating +//! them. It cannot live in the workspace `test_utils` crate because `test_utils` itself depends +//! on `delta_kernel_default_engine`, which would form an `default_engine -> test_utils -> +//! default_engine` cycle if added to default-engine as a normal dep. Keeping these helpers in +//! a sibling crate dev-depended by default-engine sidesteps that. + +use std::sync::Arc; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; + +use delta_kernel::arrow::array::{RecordBatch, StringArray}; +use delta_kernel::arrow::datatypes::{DataType, Field, Schema as ArrowSchema}; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::{DeltaResult, EngineData, Error}; + +/// Convert an `EngineData` into a `RecordBatch`. Panics if the underlying engine data is not +/// `ArrowEngineData`. +pub fn into_record_batch(engine_data: Box) -> RecordBatch { + ArrowEngineData::try_from_engine_data(engine_data) + .unwrap() + .into() +} + +/// `?`-friendly variant of [`into_record_batch`] for use inside iterators that yield +/// `DeltaResult>`. +pub fn try_into_record_batch( + engine_data: DeltaResult>, +) -> DeltaResult { + engine_data + .and_then(ArrowEngineData::try_from_engine_data) + .map(Into::into) +} + +/// Wrap a `StringArray` into the single-column `EngineData` shape expected by `parse_json`. +pub fn string_array_to_engine_data(string_array: StringArray) -> Box { + let string_field = Arc::new(Field::new("a", DataType::Utf8, true)); + let schema = Arc::new(ArrowSchema::new(vec![string_field])); + let batch = RecordBatch::try_new(schema, vec![Arc::new(string_array)]) + .expect("Can't convert to record batch"); + Box::new(ArrowEngineData::new(batch)) +} + +/// Returns the current time as a `Duration` since Unix epoch. +pub fn current_time_duration() -> DeltaResult { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|e| Error::generic(format!("System time before Unix epoch: {e}"))) +} + +/// Returns the current time in milliseconds since Unix epoch. +pub fn current_time_ms() -> DeltaResult { + let duration = current_time_duration()?; + i64::try_from(duration.as_millis()) + .map_err(|_| Error::generic("Current timestamp exceeds i64 millisecond range")) +} + +/// Assert that `res` is an `Err` whose `Display` contains `message`. +#[track_caller] +pub fn assert_result_error_with_message(res: Result, message: &str) { + match res { + Ok(_) => panic!("Expected error containing '{message}', but got Ok result"), + Err(e) => { + let actual = e.to_string(); + assert!( + actual.contains(message), + "Expected error containing '{message}', got: '{actual}'" + ); + } + } +} diff --git a/delta-kernel-unity-catalog/Cargo.toml b/delta-kernel-unity-catalog/Cargo.toml new file mode 100644 index 0000000000..8fcfc6f24c --- /dev/null +++ b/delta-kernel-unity-catalog/Cargo.toml @@ -0,0 +1,45 @@ +[package] +name = "delta-kernel-unity-catalog" +description = "Unity Catalog integration for delta_kernel." +edition.workspace = true +homepage.workspace = true +keywords.workspace = true +license.workspace = true +repository.workspace = true +readme = "README.md" +rust-version.workspace = true +version = "0.1.0" + +# for cargo-release +[package.metadata.release] +release = false + +[features] +default = ["arrow"] +arrow = ["arrow-59"] +arrow-59 = ["delta_kernel/arrow-59"] +arrow-58 = ["delta_kernel/arrow-58"] +# Live-server E2E tests. Gated so normal `cargo nextest` never compiles or runs them. +# Tests additionally skip at runtime when UC_SERVER_URL is unset. +integration-test = [] + +[dependencies] +delta_kernel = { path = "../kernel", version = "0.27.1", features = ["internal-api"] } +unity-catalog-delta-client-api = { path = "../unity-catalog-delta-client-api", version = "0.1.0" } +serde_json = "1.0" +tokio = { version = "1", features = ["full"] } +tracing = "0.1" +url = "2" + +[dev-dependencies] +delta_kernel = { path = "../kernel", features = ["default-engine-base", "test-utils", "internal-api"] } +delta_kernel_default_engine = { path = "../default-engine", features = ["rustls"] } +unity-catalog-delta-rest-client = { path = "../unity-catalog-delta-rest-client", features = ["test-utils"] } +unity-catalog-delta-client-api = { path = "../unity-catalog-delta-client-api" } +test_utils = { path = "../test-utils" } +rstest = "0.23" +tempfile = "3" +uuid = { version = "1", features = ["v4"] } +# Only used by the gated live-server CREATE test, which hand-rolls the staging-tables/tables POSTs. +# Mirrors the rest-client's reqwest config. +reqwest = { version = "0.13", default-features = false, features = ["charset", "http2", "json", "rustls", "system-proxy"] } diff --git a/delta-kernel-unity-catalog/README.md b/delta-kernel-unity-catalog/README.md new file mode 100644 index 0000000000..ae6360c777 --- /dev/null +++ b/delta-kernel-unity-catalog/README.md @@ -0,0 +1,31 @@ +# delta-kernel-unity-catalog + +> [!WARNING] +> This crate is experimental and under construction. It is not intended for production use. + +A crate connecting [`delta_kernel`] to Unity Catalog for catalog-managed Delta tables. + +A table with the `catalogManaged` table feature cannot be read or written by accessing the +transaction log on disk alone. This crate bridges UC's responses into kernel's APIs and implements +kernel's `Committer` so commits go through the catalog. + +It depends on [`unity-catalog-delta-client-api`] for the client contract, not on a concrete HTTP +client, so you can pair it with [`unity-catalog-delta-rest-client`] or your own implementation. + +It provides: + +- `UCCommitter`: kernel's `Committer` trait implemented against UC, so commits are staged and + ratified through the catalog rather than written straight to the log, plus `publish()` to promote + ratified commits. Requires a multi-threaded tokio runtime. +- `snapshot_builder_from_load_table()` and `log_tail_from_commits()`: build a kernel snapshot from a + `load_table` response, including commits UC knows about but has not yet published. +- `get_required_properties_for_disk()` and `build_uc_create_table_request()`: the table-creation + flow. +- `aws_object_store_options()`: maps UC-vended storage credentials onto object-store options. + +See the [Unity Catalog integration guide] for the end-to-end read, write, and create flows. + +[`delta_kernel`]: https://crates.io/crates/delta_kernel +[`unity-catalog-delta-client-api`]: https://github.com/delta-io/delta-kernel-rs/tree/main/unity-catalog-delta-client-api +[`unity-catalog-delta-rest-client`]: https://github.com/delta-io/delta-kernel-rs/tree/main/unity-catalog-delta-rest-client +[Unity Catalog integration guide]: https://github.com/delta-io/delta-kernel-rs/blob/main/docs/user-guide/src/unity_catalog/overview.md diff --git a/delta-kernel-unity-catalog/src/committer.rs b/delta-kernel-unity-catalog/src/committer.rs new file mode 100644 index 0000000000..1694eb3812 --- /dev/null +++ b/delta-kernel-unity-catalog/src/committer.rs @@ -0,0 +1,668 @@ +use std::panic::{catch_unwind, AssertUnwindSafe}; +use std::sync::Arc; + +use delta_kernel::committer::{ + CommitMetadata, CommitResponse, CommitType, Committer, PublishMetadata, +}; +use delta_kernel::{ + DeltaResult, DeltaResultIterator, Engine, Error as DeltaError, FileMeta, FilteredEngineData, +}; +use tracing::{debug, info}; +use unity_catalog_delta_client_api::{ + Commit, DeltaTableRequirement, DeltaTableUpdate, TableIdentifier, UpdateTableClient, + UpdateTableRequest, +}; + +use crate::constants::{ + CATALOG_MANAGED_FEATURE, CLUSTERING_DOMAIN_NAME, ENABLE_IN_COMMIT_TIMESTAMPS, + IN_COMMIT_TIMESTAMP_FEATURE, UC_TABLE_ID_KEY, VACUUM_PROTOCOL_CHECK_FEATURE, +}; +use crate::errors; + +/// Convenience macro: returns an error if a condition is not met. +macro_rules! require { + ($cond:expr, $err:expr) => { + if !($cond) { + return Err($err); + } + }; +} + +/// A [UCCommitter] is a Unity Catalog [`Committer`] implementation for committing to a specific +/// delta table in UC. +/// +/// Version 0 (table creation) writes the published commit directly to the delta log. +/// +/// For version >= 1, the committer writes a staged commit and calls the UC `update_table` API +/// to ratify it. +/// +/// NOTE: this [`Committer`] requires a multi-threaded tokio runtime. That is, whatever +/// implementation consumes the Committer to commit to the table, must call `commit` from within a +/// multi-threaded tokio runtime context. Since the default engine uses tokio, this is compatible, +/// but must ensure that the multi-threaded runtime is used. +#[derive(Debug, Clone)] +pub struct UCCommitter { + update_table_client: Arc, + table_id: String, + table: TableIdentifier, +} + +impl UCCommitter { + /// Build a committer that issues commits for the UC-managed table `table` via + /// `update_table_client`. + pub fn new( + update_table_client: Arc, + table_id: impl Into, + table: TableIdentifier, + ) -> Self { + UCCommitter { + update_table_client, + table_id: table_id.into(), + table, + } + } + + /// Returns true if the commit metadata has the `catalogManaged` feature in both reader and + /// writer features. + fn has_catalog_managed_feature(commit_metadata: &CommitMetadata) -> bool { + commit_metadata.has_writer_feature(CATALOG_MANAGED_FEATURE) + && commit_metadata.has_reader_feature(CATALOG_MANAGED_FEATURE) + } + + /// Validates that protocol features and metadata properties are correct for a UC + /// catalog-managed table. + fn validate_catalog_managed_state(&self, commit_metadata: &CommitMetadata) -> DeltaResult<()> { + require!( + commit_metadata.commit_type() != CommitType::UpgradeToCatalogManaged, + errors::upgrade_downgrade_unsupported("upgrade") + ); + require!( + commit_metadata.commit_type() != CommitType::DowngradeToPathBased, + errors::upgrade_downgrade_unsupported("downgrade") + ); + require!( + Self::has_catalog_managed_feature(commit_metadata), + errors::missing_feature(CATALOG_MANAGED_FEATURE) + ); + // UC mandates vacuumProtocolCheck for its catalog-managed tables. + require!( + commit_metadata.has_writer_feature(VACUUM_PROTOCOL_CHECK_FEATURE) + && commit_metadata.has_reader_feature(VACUUM_PROTOCOL_CHECK_FEATURE), + errors::missing_feature(VACUUM_PROTOCOL_CHECK_FEATURE) + ); + require!( + commit_metadata.has_writer_feature(IN_COMMIT_TIMESTAMP_FEATURE), + errors::missing_feature(IN_COMMIT_TIMESTAMP_FEATURE) + ); + + let config = commit_metadata + .metadata_configuration() + .ok_or_else(errors::missing_metadata_configuration)?; + let table_id = config + .get(UC_TABLE_ID_KEY) + .ok_or_else(|| errors::missing_property(UC_TABLE_ID_KEY))?; + require!( + table_id == &self.table_id, + errors::table_id_mismatch(&self.table_id, table_id) + ); + require!( + config.get(ENABLE_IN_COMMIT_TIMESTAMPS).map(String::as_str) == Some("true"), + errors::ict_not_enabled() + ); + Ok(()) + } + + /// Validates that this commit does not include ALTER TABLE changes (protocol, metadata, + /// or clustering column changes). + fn validate_no_alter_table_changes(commit_metadata: &CommitMetadata) -> DeltaResult<()> { + require!( + !commit_metadata.has_protocol_change(), + errors::alter_table_unsupported("protocol") + ); + require!( + !commit_metadata.has_metadata_change(), + errors::alter_table_unsupported("metadata") + ); + require!( + !commit_metadata.has_domain_metadata_change(CLUSTERING_DOMAIN_NAME), + errors::alter_table_unsupported("clustering columns") + ); + Ok(()) + } + + /// Commit version 0 (table creation). Validates that all required UC properties are present, + /// then writes the version 0 commit file directly to the published commit path. + fn commit_version_0( + &self, + engine: &dyn Engine, + actions: DeltaResultIterator<'_, FilteredEngineData>, + commit_metadata: &CommitMetadata, + ) -> DeltaResult { + debug_assert!( + commit_metadata.version() == 0, + "commit_version_0 called with version {}", + commit_metadata.version() + ); + self.validate_catalog_managed_state(commit_metadata)?; + let published_commit_path = commit_metadata.published_commit_path()?; + match engine.json_handler().write_json_file( + &published_commit_path, + Box::new(actions), + false, + ) { + Ok(()) => { + info!("wrote version 0 commit file for UC table creation"); + let file_meta = FileMeta::new( + published_commit_path, + commit_metadata.in_commit_timestamp(), + 0, + ); + Ok(CommitResponse::Committed { file_meta }) + } + Err(DeltaError::FileAlreadyExists(_)) => { + info!("version 0 commit conflict: commit file already exists"); + Ok(CommitResponse::Conflict { version: 0 }) + } + Err(e) => Err(e), + } + } + + /// Commit version >= 1. Validates catalog-managed status hasn't changed, writes a staged + /// commit file, and calls the UC commit API to ratify it. + fn commit_version_non_zero( + &self, + engine: &dyn Engine, + actions: DeltaResultIterator<'_, FilteredEngineData>, + commit_metadata: CommitMetadata, + ) -> DeltaResult + where + C: 'static, + { + debug_assert!( + commit_metadata.version() != 0, + "commit_version_non_zero called with version 0" + ); + self.validate_catalog_managed_state(&commit_metadata)?; + Self::validate_no_alter_table_changes(&commit_metadata)?; + let staged_commit_path = commit_metadata.staged_commit_path()?; + engine + .json_handler() + .write_json_file(&staged_commit_path, Box::new(actions), false)?; + + let committed = engine.storage_handler().head(&staged_commit_path)?; + debug!("wrote staged commit file: {:?}", committed); + + let mut updates = vec![DeltaTableUpdate::AddCommit { + commit: Commit { + version: u64_to_wire_i64(commit_metadata.version(), "commit version")?, + timestamp: commit_metadata.in_commit_timestamp(), + file_name: staged_commit_file_name(&staged_commit_path)?, + file_size: u64_to_wire_i64(committed.size, "committed size")?, + file_modification_timestamp: committed.last_modified, + }, + }]; + if let Some(max_pub) = commit_metadata.max_published_version() { + updates.push(DeltaTableUpdate::SetLatestBackfilledVersion { + latest_published_version: u64_to_wire_i64(max_pub, "max published version")?, + }); + } + let update_req = UpdateTableRequest::new( + vec![DeltaTableRequirement::AssertTableUuid { + uuid: self.table_id.clone(), + }], + updates, + ) + .map_err(|e| DeltaError::generic(format!("invalid UC update_table request: {e}")))?; + let target = self.table.clone(); + + let handle = tokio::runtime::Handle::try_current().map_err(|_| { + DeltaError::generic("UCCommitter may only be used within a tokio runtime") + })?; + // `block_in_place` panics if the current runtime isn't multi-threaded. We can't check for + // that up front: `runtime_flavor()` can't tell a real single-threaded runtime (where this + // panics) apart from the FFI case (single-threaded on top of multi-threaded, where it's + // fine). So we let it run and catch the panic. + let result = catch_unwind(AssertUnwindSafe(|| { + tokio::task::block_in_place(|| { + handle.block_on(async move { + self.update_table_client + .update_table(&target, update_req) + .await + }) + }) + })) + .map_err(|panic| { + let msg = panic + .downcast_ref::<&str>() + .map(|s| s.to_string()) + .or_else(|| panic.downcast_ref::().cloned()) + .unwrap_or_else(|| "unknown panic".to_string()); + DeltaError::generic(format!( + "UCCommitter commit panicked (requires a multi-threaded tokio runtime): {msg}" + )) + })?; + match result { + Ok(_) => Ok(CommitResponse::Committed { + file_meta: committed, + }), + // TODO(#2970): classify version conflicts as CommitResponse::Conflict so the + // transaction layer can rebase/retry, instead of collapsing every error to Generic. + Err(e) => Err(DeltaError::Generic(format!("UC update_table error: {e}"))), + } + } +} + +impl Committer for UCCommitter { + /// Commit the given `actions` to the delta table in UC. + /// + /// For version 0 (table creation), the committer writes `000.json` directly to the published + /// commit path; the caller finalizes the table in UC via the create-table API. For version >= + /// 1, connectors should publish staged commits to the delta log immediately after writing; + /// UC expects to be informed of the last known published version during commit. + fn commit( + &self, + engine: &dyn Engine, + actions: DeltaResultIterator<'_, FilteredEngineData>, + commit_metadata: CommitMetadata, + ) -> DeltaResult { + if commit_metadata.version() == 0 { + return self.commit_version_0(engine, actions, &commit_metadata); + } + self.commit_version_non_zero(engine, actions, commit_metadata) + } + + fn is_catalog_committer(&self) -> bool { + true + } + + fn publish(&self, engine: &dyn Engine, publish_metadata: PublishMetadata) -> DeltaResult<()> { + if publish_metadata.commits_to_publish().is_empty() { + return Ok(()); + } + + for catalog_commit in publish_metadata.commits_to_publish() { + let src = catalog_commit.location(); + let dest = catalog_commit.published_location(); + match engine.storage_handler().copy_atomic(src, dest) { + Ok(_) => (), + Err(DeltaError::FileAlreadyExists(_)) => (), + Err(e) => return Err(e), + } + } + + Ok(()) + } +} + +/// Convert a `u64` to the `i64` the UC wire types use, erroring if it does not fit. +fn u64_to_wire_i64(value: u64, field: &str) -> DeltaResult { + value + .try_into() + .map_err(|_| DeltaError::generic(format!("{field} does not fit into i64 for UC commit"))) +} + +fn staged_commit_file_name(path: &url::Url) -> DeltaResult { + path.path_segments() + .and_then(|mut segments| segments.next_back()) + .filter(|segment| !segment.is_empty()) + .map(str::to_string) + .ok_or_else(|| DeltaError::generic("staged commit path has no file name")) +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + use std::fs; + + use delta_kernel::committer::{CatalogCommit, CommitMetadata}; + use delta_kernel::object_store::local::LocalFileSystem; + use delta_kernel::Version; + use delta_kernel_default_engine::DefaultEngine; + use unity_catalog_delta_client_api::error::Result; + + use super::*; + + struct MockUpdateTableClient; + + impl UpdateTableClient for MockUpdateTableClient { + async fn update_table(&self, _: &TableIdentifier, _: UpdateTableRequest) -> Result<()> { + unimplemented!() + } + } + + fn test_committer() -> UCCommitter { + UCCommitter::new( + Arc::new(MockUpdateTableClient), + "test-table-id", + TableIdentifier::new("test_catalog", "test_schema", "test_table"), + ) + } + + /// Creates a valid catalog-managed CommitMetadata with all required UC features and properties. + fn catalog_managed_commit_metadata(table_root: url::Url, version: Version) -> CommitMetadata { + CommitMetadata::new_unchecked_with( + table_root, + version, + vec!["catalogManaged", "vacuumProtocolCheck"], + vec!["catalogManaged", "inCommitTimestamp", "vacuumProtocolCheck"], + HashMap::from([ + ( + "io.unitycatalog.tableId".to_string(), + "test-table-id".to_string(), + ), + ( + "delta.enableInCommitTimestamps".to_string(), + "true".to_string(), + ), + ]), + ) + .unwrap() + } + + #[test] + fn commit_version_0_writes_published_commit() { + let tmp_dir = tempfile::tempdir().unwrap(); + let table_root = url::Url::from_directory_path(tmp_dir.path()).unwrap(); + let commit_metadata = catalog_managed_commit_metadata(table_root.clone(), 0); + let committer = test_committer(); + let engine = DefaultEngine::builder(Arc::new(LocalFileSystem::new())).build(); + + // Create the _delta_log directory + fs::create_dir_all(tmp_dir.path().join("_delta_log")).unwrap(); + + let result = committer + .commit(&engine, Box::new(std::iter::empty()), commit_metadata) + .unwrap(); + match result { + CommitResponse::Committed { file_meta } => { + assert!( + file_meta + .location + .as_str() + .ends_with("00000000000000000000.json"), + "expected published path for version 0, got: {}", + file_meta.location + ); + assert_eq!(file_meta.size, 0); + // Verify the file was written to disk + let commit_path = tmp_dir.path().join("_delta_log/00000000000000000000.json"); + assert!(commit_path.exists(), "000.json should exist on disk"); + } + CommitResponse::Conflict { .. } => { + panic!("expected Committed for version 0, got Conflict") + } + } + } + + #[test] + fn commit_version_0_conflict_when_file_exists() { + let tmp_dir = tempfile::tempdir().unwrap(); + let table_root = url::Url::from_directory_path(tmp_dir.path()).unwrap(); + let committer = test_committer(); + let engine = DefaultEngine::builder(Arc::new(LocalFileSystem::new())).build(); + + // Pre-create the commit file to trigger a conflict + let delta_log = tmp_dir.path().join("_delta_log"); + fs::create_dir_all(&delta_log).unwrap(); + fs::write(delta_log.join("00000000000000000000.json"), "existing").unwrap(); + + let commit_metadata = catalog_managed_commit_metadata(table_root, 0); + let result = committer + .commit(&engine, Box::new(std::iter::empty()), commit_metadata) + .unwrap(); + assert!( + matches!(result, CommitResponse::Conflict { version: 0 }), + "expected Conflict for version 0 when file exists, got: {result:?}" + ); + } + + #[test] + fn commit_version_0_rejects_missing_catalog_managed_feature() { + let tmp_dir = tempfile::tempdir().unwrap(); + let table_root = url::Url::from_directory_path(tmp_dir.path()).unwrap(); + let commit_metadata = CommitMetadata::new_unchecked(table_root, 0).unwrap(); + let committer = test_committer(); + let engine = DefaultEngine::builder(Arc::new(LocalFileSystem::new())).build(); + fs::create_dir_all(tmp_dir.path().join("_delta_log")).unwrap(); + + let err = committer + .commit(&engine, Box::new(std::iter::empty()), commit_metadata) + .unwrap_err(); + assert!( + err.to_string().contains("catalogManaged"), + "expected catalogManaged error, got: {err}" + ); + } + + #[test] + fn commit_version_0_rejects_missing_table_id() { + let tmp_dir = tempfile::tempdir().unwrap(); + let table_root = url::Url::from_directory_path(tmp_dir.path()).unwrap(); + // Has features but missing io.unitycatalog.tableId in config + let commit_metadata = CommitMetadata::new_unchecked_with( + table_root, + 0, + vec!["catalogManaged", "vacuumProtocolCheck"], + vec!["catalogManaged", "inCommitTimestamp", "vacuumProtocolCheck"], + HashMap::from([( + "delta.enableInCommitTimestamps".to_string(), + "true".to_string(), + )]), + ) + .unwrap(); + let committer = test_committer(); + let engine = DefaultEngine::builder(Arc::new(LocalFileSystem::new())).build(); + fs::create_dir_all(tmp_dir.path().join("_delta_log")).unwrap(); + + let err = committer + .commit(&engine, Box::new(std::iter::empty()), commit_metadata) + .unwrap_err(); + assert!( + err.to_string().contains("io.unitycatalog.tableId"), + "expected tableId error, got: {err}" + ); + } + + #[test] + fn commit_version_0_rejects_missing_ict_enablement() { + let tmp_dir = tempfile::tempdir().unwrap(); + let table_root = url::Url::from_directory_path(tmp_dir.path()).unwrap(); + // Has features and tableId but missing delta.enableInCommitTimestamps=true + let commit_metadata = CommitMetadata::new_unchecked_with( + table_root, + 0, + vec!["catalogManaged", "vacuumProtocolCheck"], + vec!["catalogManaged", "inCommitTimestamp", "vacuumProtocolCheck"], + HashMap::from([( + "io.unitycatalog.tableId".to_string(), + "test-table-id".to_string(), + )]), + ) + .unwrap(); + let committer = test_committer(); + let engine = DefaultEngine::builder(Arc::new(LocalFileSystem::new())).build(); + fs::create_dir_all(tmp_dir.path().join("_delta_log")).unwrap(); + + let err = committer + .commit(&engine, Box::new(std::iter::empty()), commit_metadata) + .unwrap_err(); + assert!( + err.to_string().contains("enableInCommitTimestamps"), + "expected ICT enablement error, got: {err}" + ); + } + + #[test] + fn commit_version_non_zero_rejects_non_catalog_managed_table() { + let tmp_dir = tempfile::tempdir().unwrap(); + let table_root = url::Url::from_directory_path(tmp_dir.path()).unwrap(); + // Version >= 1 but without catalogManaged feature (simulates downgrade attempt) + let commit_metadata = CommitMetadata::new_unchecked(table_root, 1).unwrap(); + let committer = test_committer(); + let engine = DefaultEngine::builder(Arc::new(LocalFileSystem::new())).build(); + + let err = committer + .commit(&engine, Box::new(std::iter::empty()), commit_metadata) + .unwrap_err(); + assert!( + err.to_string().contains("catalogManaged"), + "expected catalogManaged error, got: {err}" + ); + } + + // A default `#[tokio::test]` is a current-thread runtime. `block_in_place` would panic there, + // so the committer must reject the flavor with an error instead. + #[tokio::test] + async fn commit_version_non_zero_rejects_current_thread_runtime() { + let tmp_dir = tempfile::tempdir().unwrap(); + let table_root = url::Url::from_directory_path(tmp_dir.path()).unwrap(); + let commit_metadata = catalog_managed_commit_metadata(table_root, 1); + let engine = DefaultEngine::builder(Arc::new(LocalFileSystem::new())).build(); + fs::create_dir_all(tmp_dir.path().join("_delta_log")).unwrap(); + + let err = test_committer() + .commit(&engine, Box::new(std::iter::empty()), commit_metadata) + .unwrap_err(); + assert!( + err.to_string().contains("multi-threaded"), + "expected a multi-threaded-runtime error, got: {err}" + ); + } + + #[test] + fn commit_version_non_zero_rejects_protocol_change() { + let tmp_dir = tempfile::tempdir().unwrap(); + let table_root = url::Url::from_directory_path(tmp_dir.path()).unwrap(); + let commit_metadata = catalog_managed_commit_metadata(table_root, 1).with_protocol_change(); + let engine = DefaultEngine::builder(Arc::new(LocalFileSystem::new())).build(); + + let err = test_committer() + .commit(&engine, Box::new(std::iter::empty()), commit_metadata) + .unwrap_err(); + assert!( + err.to_string().contains("table protocol"), + "expected protocol change error, got: {err}" + ); + } + + #[test] + fn commit_version_non_zero_rejects_metadata_change() { + let tmp_dir = tempfile::tempdir().unwrap(); + let table_root = url::Url::from_directory_path(tmp_dir.path()).unwrap(); + let commit_metadata = catalog_managed_commit_metadata(table_root, 1).with_metadata_change(); + let engine = DefaultEngine::builder(Arc::new(LocalFileSystem::new())).build(); + + let err = test_committer() + .commit(&engine, Box::new(std::iter::empty()), commit_metadata) + .unwrap_err(); + assert!( + err.to_string().contains("table metadata"), + "expected metadata change error, got: {err}" + ); + } + + #[test] + fn commit_version_non_zero_rejects_clustering_change() { + let tmp_dir = tempfile::tempdir().unwrap(); + let table_root = url::Url::from_directory_path(tmp_dir.path()).unwrap(); + let commit_metadata = + catalog_managed_commit_metadata(table_root, 1).with_domain_change("delta.clustering"); + let engine = DefaultEngine::builder(Arc::new(LocalFileSystem::new())).build(); + + let err = test_committer() + .commit(&engine, Box::new(std::iter::empty()), commit_metadata) + .unwrap_err(); + assert!( + err.to_string().contains("clustering columns"), + "expected clustering change error, got: {err}" + ); + } + + #[test] + fn commit_version_non_zero_rejects_mismatched_table_id() { + let tmp_dir = tempfile::tempdir().unwrap(); + let table_root = url::Url::from_directory_path(tmp_dir.path()).unwrap(); + let commit_metadata = catalog_managed_commit_metadata(table_root, 1); + // Committer initialized with a different table ID than what's in the metadata. + let committer = UCCommitter::new( + Arc::new(MockUpdateTableClient), + "different-table-id", + TableIdentifier::new("test_catalog", "test_schema", "test_table"), + ); + let engine = DefaultEngine::builder(Arc::new(LocalFileSystem::new())).build(); + + let err = committer + .commit(&engine, Box::new(std::iter::empty()), commit_metadata) + .unwrap_err(); + assert!( + err.to_string().contains("table ID mismatch"), + "expected table ID mismatch error, got: {err}" + ); + } + + fn staged_commit_url(table_root: &url::Url, version: Version) -> url::Url { + table_root + .join(&format!( + "_delta_log/_staged_commits/{version:020}.uuid.json" + )) + .unwrap() + } + + fn published_commit_url(table_root: &url::Url, version: Version) -> url::Url { + table_root + .join(&format!("_delta_log/{version:020}.json")) + .unwrap() + } + + #[tokio::test] + async fn test_publish() { + let tmp_dir = tempfile::tempdir().unwrap(); + let table_root = url::Url::from_directory_path(tmp_dir.path()).unwrap(); + let staged_dir = tmp_dir.path().join("_delta_log/_staged_commits"); + let versions = [10u64, 11, 12]; + + // ===== GIVEN ===== + // Create catalog commits + let catalog_commits: Vec = versions + .into_iter() + .map(|v| { + CatalogCommit::new_unchecked( + v, + staged_commit_url(&table_root, v), + published_commit_url(&table_root, v), + ) + }) + .collect(); + + // Write staged commit files to disk + fs::create_dir_all(&staged_dir).unwrap(); + for commit in &catalog_commits { + let path = commit.location().to_file_path().unwrap(); + fs::write(&path, format!("version: {}", commit.version())).unwrap(); + } + + // Write 10.json file to disk (should be skipped, not error) + let existing_published = published_commit_url(&table_root, 10) + .to_file_path() + .unwrap(); + fs::create_dir_all(existing_published.parent().unwrap()).unwrap(); + fs::write(&existing_published, "version: 10").unwrap(); + + // ===== WHEN ===== + let publish_metadata = PublishMetadata::try_new(12, catalog_commits).unwrap(); + let committer = UCCommitter::new( + Arc::new(MockUpdateTableClient), + "testUcTableId", + TableIdentifier::new("test_catalog", "test_schema", "test_table"), + ); + let engine = DefaultEngine::builder(Arc::new(LocalFileSystem::new())).build(); + committer.publish(&engine, publish_metadata).unwrap(); + + // ===== THEN ===== + for v in versions { + let path = published_commit_url(&table_root, v).to_file_path().unwrap(); + assert!(path.exists()); + assert_eq!(fs::read_to_string(&path).unwrap(), format!("version: {v}")); + } + } +} diff --git a/delta-kernel-unity-catalog/src/constants.rs b/delta-kernel-unity-catalog/src/constants.rs new file mode 100644 index 0000000000..bb02f17f48 --- /dev/null +++ b/delta-kernel-unity-catalog/src/constants.rs @@ -0,0 +1,38 @@ +//! Shared constants for UC catalog-managed table operations. + +/// Property key for the UC table ID, stored in Delta metadata configuration. +pub(crate) const UC_TABLE_ID_KEY: &str = "io.unitycatalog.tableId"; +/// Property key to enable in-commit timestamps. +pub(crate) const ENABLE_IN_COMMIT_TIMESTAMPS: &str = "delta.enableInCommitTimestamps"; +/// Feature supported value. +pub(crate) const FEATURE_SUPPORTED: &str = "supported"; +/// Feature signal key for catalog-managed tables. +pub(crate) const CATALOG_MANAGED_FEATURE_KEY: &str = "delta.feature.catalogManaged"; +/// Feature signal key for vacuum protocol check. +pub(crate) const VACUUM_PROTOCOL_CHECK_FEATURE_KEY: &str = "delta.feature.vacuumProtocolCheck"; +/// Feature signal key for v2 checkpoints. +pub(crate) const V2_CHECKPOINT_FEATURE_KEY: &str = "delta.feature.v2Checkpoint"; +/// Feature signal key for deletion vectors. +pub(crate) const DELETION_VECTORS_FEATURE_KEY: &str = "delta.feature.deletionVectors"; +/// Config property enabling deletion vectors. +pub(crate) const ENABLE_DELETION_VECTORS_KEY: &str = "delta.enableDeletionVectors"; +/// Config property selecting the checkpoint policy. +pub(crate) const CHECKPOINT_POLICY_KEY: &str = "delta.checkpointPolicy"; +/// Config property value selecting the v2 checkpoint policy. +pub(crate) const CHECKPOINT_POLICY_V2: &str = "v2"; +/// Config property writing checkpoint stats as a struct. +pub(crate) const CHECKPOINT_WRITE_STATS_AS_STRUCT_KEY: &str = "delta.checkpoint.writeStatsAsStruct"; +/// Config property writing checkpoint stats as JSON. +pub(crate) const CHECKPOINT_WRITE_STATS_AS_JSON_KEY: &str = "delta.checkpoint.writeStatsAsJson"; +/// Boolean-true config property value. +pub(crate) const CONFIG_TRUE: &str = "true"; +/// Feature name for catalog-managed tables (wire format). +pub(crate) const CATALOG_MANAGED_FEATURE: &str = "catalogManaged"; +/// Feature name for vacuum protocol check (wire format). +pub(crate) const VACUUM_PROTOCOL_CHECK_FEATURE: &str = "vacuumProtocolCheck"; +/// Feature name for in-commit timestamps (wire format). +pub(crate) const IN_COMMIT_TIMESTAMP_FEATURE: &str = "inCommitTimestamp"; +/// Domain name for clustering metadata. +pub(crate) const CLUSTERING_DOMAIN_NAME: &str = "delta.clustering"; +/// Domain name for row-tracking metadata. +pub(crate) const ROW_TRACKING_DOMAIN_NAME: &str = "delta.rowTracking"; diff --git a/delta-kernel-unity-catalog/src/errors.rs b/delta-kernel-unity-catalog/src/errors.rs new file mode 100644 index 0000000000..bfe32765ac --- /dev/null +++ b/delta-kernel-unity-catalog/src/errors.rs @@ -0,0 +1,43 @@ +//! Error helpers for UC operations. Centralizes error message construction to keep validation +//! logic concise. + +use delta_kernel::Error as DeltaError; + +pub(crate) fn missing_feature(feature: &str) -> DeltaError { + DeltaError::generic(format!( + "UC catalog-managed table requires the '{feature}' table feature" + )) +} + +pub(crate) fn missing_metadata_configuration() -> DeltaError { + DeltaError::generic("UC catalog-managed table requires metadata configuration") +} + +pub(crate) fn missing_property(key: &str) -> DeltaError { + DeltaError::generic(format!( + "UC catalog-managed table requires '{key}' in metadata configuration" + )) +} + +pub(crate) fn table_id_mismatch(expected: &str, actual: &str) -> DeltaError { + DeltaError::generic(format!( + "UC table ID mismatch: expected '{expected}' but found '{actual}'" + )) +} + +pub(crate) fn ict_not_enabled() -> DeltaError { + DeltaError::generic("UC catalog-managed table requires 'delta.enableInCommitTimestamps=true'") +} + +pub(crate) fn upgrade_downgrade_unsupported(direction: &str) -> DeltaError { + DeltaError::generic(format!( + "Table {direction} is not yet supported by the UCCommitter" + )) +} + +pub(crate) fn alter_table_unsupported(what: &str) -> DeltaError { + DeltaError::generic(format!( + "UCCommitter does not support commits that change the table {what}. \ + ALTER TABLE is not supported for catalog-managed tables." + )) +} diff --git a/delta-kernel-unity-catalog/src/lib.rs b/delta-kernel-unity-catalog/src/lib.rs new file mode 100644 index 0000000000..ed030df2c6 --- /dev/null +++ b/delta-kernel-unity-catalog/src/lib.rs @@ -0,0 +1,211 @@ +//! Kernel-side helpers for catalog-managed Delta tables in Unity Catalog. + +mod committer; +mod constants; +mod errors; +mod utils; + +pub use committer::UCCommitter; +use delta_kernel::snapshot::SnapshotBuilder; +use delta_kernel::{DeltaResult, Error, LogPath, Snapshot}; +use unity_catalog_delta_client_api::{Commit, LoadTableResponse}; +use url::Url; +pub use utils::{ + aws_object_store_options, build_uc_create_table_request, get_required_properties_for_disk, +}; + +/// Convert the inline `Commit`s returned by a UC `load_table` response into +/// kernel `LogPath`s suitable for `Snapshot::builder_for(...).with_log_tail(...)`. +/// +/// Sorts commits by version because they may arrive out of order. Normalizes `table_root` +/// with a trailing `/` for staged-commit path resolution. +/// +/// # Errors +/// +/// Returns an error if a commit's `file_size` is negative (does not fit in `FileSize`) or if +/// [`LogPath::staged_commit`] rejects the resolved path. +pub fn log_tail_from_commits(commits: &[Commit], mut table_root: Url) -> DeltaResult> { + // `load_table` returns the location without a trailing slash, but staged-commit path + // resolution requires the table root to end in `/`. + if !table_root.path().ends_with('/') { + table_root.set_path(&format!("{}/", table_root.path())); + } + let mut sorted: Vec<&Commit> = commits.iter().collect(); + sorted.sort_by_key(|c| c.version); + sorted + .into_iter() + .map(|c| { + let file_size = c.file_size.try_into().map_err(|_| { + Error::generic(format!( + "commit file_size {} does not fit in FileSize", + c.file_size + )) + })?; + LogPath::staged_commit( + table_root.clone(), + &c.file_name, + c.file_modification_timestamp, + file_size, + ) + }) + .collect() +} + +/// Build a [`SnapshotBuilder`] from a UC `load_table` response. +/// +/// Parses the table location, converts the inline commits into a log tail via +/// [`log_tail_from_commits`], and pins the catalog's ratified version with +/// `with_max_catalog_version`. The connector still owns the `load_table` call and the final +/// `build(engine)`; add `at_version(v)` to the returned builder to time-travel. +/// +/// # Errors +/// +/// Returns an error if the response's location is not a valid URL, if [`log_tail_from_commits`] +/// fails, or if `latest_table_version` is negative. +pub fn snapshot_builder_from_load_table(resp: &LoadTableResponse) -> DeltaResult { + let table_root = Url::parse(&resp.metadata.location) + .map_err(|e| Error::generic(format!("invalid table location: {e}")))?; + let log_tail = log_tail_from_commits(&resp.commits, table_root.clone())?; + let mut builder = Snapshot::builder_for(table_root).with_log_tail(log_tail); + if let Some(version) = resp.latest_table_version { + let max_catalog_version: u64 = version + .try_into() + .map_err(|_| Error::generic("catalog reported a negative table version"))?; + builder = builder.with_max_catalog_version(max_catalog_version); + } + Ok(builder) +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use delta_kernel::object_store::memory::InMemory; + use delta_kernel::Snapshot; + use delta_kernel_default_engine::DefaultEngineBuilder; + use rstest::rstest; + use unity_catalog_delta_client_api::{Commit, InMemoryUpdateTableClient, TableData}; + + use super::*; + + #[tokio::test] + async fn load_snapshot_errors_on_non_contiguous_commits() { + let client = InMemoryUpdateTableClient::new(); + client.insert_table( + "test_table", + TableData { + max_ratified_version: 3, + catalog_commits: vec![ + Commit::new( + 1, + 0, + "00000000000000000001.3a0d65cd-4056-49b8-937b-95f9e3ee90e5.json", + 100, + 0, + ), + Commit::new( + 3, + 0, + "00000000000000000003.3a0d65cd-4056-49b8-937b-95f9e3ee90e5.json", + 100, + 0, + ), // gap: version 2 missing + ], + }, + ); + let resp = client + .load_table_response("test_table", "memory:///") + .unwrap(); + + let table_url = Url::parse(&resp.metadata.location).unwrap(); + let log_tail = log_tail_from_commits(&resp.commits, table_url.clone()).unwrap(); + + let store = Arc::new(InMemory::new()); + let engine = DefaultEngineBuilder::new(store).build(); + let max_catalog_version: u64 = resp.latest_table_version.unwrap_or(0).try_into().unwrap(); + let result = Snapshot::builder_for(table_url) + .with_log_tail(log_tail) + .with_max_catalog_version(max_catalog_version) + .build(&engine); + + assert!(result + .unwrap_err() + .to_string() + .contains("Log tail versions 1 and 3 are not contiguous")); + } + + /// Builds a `LoadTableResponse` with no commits at the given location and ratified version. + fn load_table_response(location: &str, version: i64) -> LoadTableResponse { + let client = InMemoryUpdateTableClient::new(); + client.insert_table( + "test_table", + TableData { + max_ratified_version: version, + catalog_commits: vec![], + }, + ); + client.load_table_response("test_table", location).unwrap() + } + + #[test] + fn snapshot_builder_from_load_table_rejects_negative_version() { + let resp = load_table_response("memory:///my_table", -1); + let err = snapshot_builder_from_load_table(&resp).unwrap_err(); + assert!(err.to_string().contains("negative table version")); + } + + #[test] + fn snapshot_builder_from_load_table_rejects_invalid_location() { + let resp = load_table_response("not a url", 0); + let err = snapshot_builder_from_load_table(&resp).unwrap_err(); + assert!(err.to_string().contains("invalid table location")); + } + + #[rstest] + #[case::missing_trailing_slash("memory:///my_table")] + #[case::has_trailing_slash("memory:///my_table/")] + fn log_tail_from_commits_sorts_and_normalizes(#[case] table_root: &str) { + let names = [ + "00000000000000000000.3a0d65cd-4056-49b8-937b-95f9e3ee90e5.json", + "00000000000000000001.3a0d65cd-4056-49b8-937b-95f9e3ee90e5.json", + "00000000000000000002.3a0d65cd-4056-49b8-937b-95f9e3ee90e5.json", + ]; + // Insert out of order: 2, 0, 1. + let commits = vec![ + Commit::new(2, 0, names[2], 100, 0), + Commit::new(0, 0, names[0], 100, 0), + Commit::new(1, 0, names[1], 100, 0), + ]; + let normalized = Url::parse("memory:///my_table/").unwrap(); + + let log_tail = log_tail_from_commits(&commits, Url::parse(table_root).unwrap()).unwrap(); + + let expected: Vec = names + .iter() + .map(|name| LogPath::staged_commit(normalized.clone(), name, 0, 100).unwrap()) + .collect(); + assert_eq!(log_tail, expected); + } + + #[test] + fn log_tail_from_commits_empty_commits_is_empty() { + let table_root = Url::parse("memory:///my_table/").unwrap(); + assert!(log_tail_from_commits(&[], table_root).unwrap().is_empty()); + } + + #[test] + fn log_tail_from_commits_rejects_negative_file_size() { + let commits = vec![Commit::new( + 0, + 0, + "00000000000000000000.3a0d65cd-4056-49b8-937b-95f9e3ee90e5.json", + -1, + 0, + )]; + let table_root = Url::parse("memory:///my_table/").unwrap(); + + let err = log_tail_from_commits(&commits, table_root).unwrap_err(); + + assert!(err.to_string().contains("does not fit in FileSize")); + } +} diff --git a/delta-kernel-unity-catalog/src/utils/create_table.rs b/delta-kernel-unity-catalog/src/utils/create_table.rs new file mode 100644 index 0000000000..e19d8958a8 --- /dev/null +++ b/delta-kernel-unity-catalog/src/utils/create_table.rs @@ -0,0 +1,454 @@ +//! Utilities for Unity Catalog catalog-managed table creation. +//! +//! These utilities help connectors create UC-managed tables by providing the required properties +//! for both the Delta log (disk) and the UC server registration. +//! +//! TODO(#3032): add a high-level helper that runs the whole flow (reserve staging table, vend +//! credentials, kernel v0 commit, register with UC) in one call. +//! +//! # Usage +//! +//! ```ignore +//! // Step 1: Get staging info from UC +//! let staging_info = my_uc_client.post_staging_table(..).await?; +//! +//! // Step 2: Build and commit the create-table transaction +//! let disk_props = get_required_properties_for_disk(&staging_info.table_id); +//! let create_table_txn = kernel::create_table(path, schema, "MyApp/1.0") +//! .with_table_properties(disk_props) +//! .build(engine, committer); +//! create_table_txn.commit(engine)?; +//! +//! // Step 3: Finalize table in UC +//! let snapshot = /* load post-commit snapshot at version 0 */; +//! let create_req = build_uc_create_table_request(&snapshot, engine, table_name)?; +//! my_uc_client.post_create_table(create_req).await?; +//! ``` + +use std::collections::{HashMap, HashSet}; + +use delta_kernel::actions::Protocol; +use delta_kernel::{DeltaResult, Engine, Error, Snapshot}; +use unity_catalog_delta_client_api::{ + CreateTableRequest, Protocol as WireProtocol, StorageCredential, +}; + +use crate::constants::{ + CATALOG_MANAGED_FEATURE_KEY, CHECKPOINT_POLICY_KEY, CHECKPOINT_POLICY_V2, + CHECKPOINT_WRITE_STATS_AS_JSON_KEY, CHECKPOINT_WRITE_STATS_AS_STRUCT_KEY, + CLUSTERING_DOMAIN_NAME, CONFIG_TRUE, DELETION_VECTORS_FEATURE_KEY, ENABLE_DELETION_VECTORS_KEY, + FEATURE_SUPPORTED, ROW_TRACKING_DOMAIN_NAME, UC_TABLE_ID_KEY, V2_CHECKPOINT_FEATURE_KEY, + VACUUM_PROTOCOL_CHECK_FEATURE_KEY, +}; + +/// Returns the table properties that must be written to disk (in `000.json`) for a UC +/// catalog-managed table creation. +/// +/// These properties must be persisted in the Delta log so that the table is recognized as +/// catalog-managed. Note: ICT enablement is handled automatically by kernel's CREATE TABLE +/// when the `catalogManaged` feature is present. +/// +/// TODO(delta-io/delta#7179): this hardcodes the required table features and properties. The create +/// table flow should instead derive them from the `createStagingTable` response's advertised +/// `required_protocol` and `required_properties`, so it stays correct if UC changes what it +/// requires. +pub fn get_required_properties_for_disk(uc_table_id: &str) -> HashMap { + [ + (CATALOG_MANAGED_FEATURE_KEY, FEATURE_SUPPORTED), + (VACUUM_PROTOCOL_CHECK_FEATURE_KEY, FEATURE_SUPPORTED), + (V2_CHECKPOINT_FEATURE_KEY, FEATURE_SUPPORTED), + (DELETION_VECTORS_FEATURE_KEY, FEATURE_SUPPORTED), + (ENABLE_DELETION_VECTORS_KEY, CONFIG_TRUE), + (CHECKPOINT_WRITE_STATS_AS_STRUCT_KEY, CONFIG_TRUE), + (CHECKPOINT_WRITE_STATS_AS_JSON_KEY, CONFIG_TRUE), + (UC_TABLE_ID_KEY, uc_table_id), + ] + .into_iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect() +} + +/// Map AWS S3 credentials vended by UC (`CreateStagingTableResponse.storage_credentials` or +/// `get_table_credentials`) into `object_store` option keys, for building an engine over the +/// table's storage. +/// +/// This is a convenience over the public `store_from_url_opts`: connectors can build a store +/// themselves by passing option keys directly. It only translates S3's key names today. +/// +/// TODO(#3016): also translate GCS (`gcs.*`) and Azure (`azure.*`) credential keys so connectors +/// don't each re-derive the mapping. +pub fn aws_object_store_options( + creds: &[StorageCredential], + region: &str, +) -> HashMap { + let mut opts = HashMap::new(); + for cred in creds { + for (key, value) in &cred.config { + let mapped = match key.as_str() { + "s3.access-key-id" => "access_key_id", + "s3.secret-access-key" => "secret_access_key", + "s3.session-token" => "session_token", + other => other, + }; + opts.insert(mapped.to_string(), value.clone()); + } + } + opts.insert("region".to_string(), region.to_string()); + opts +} + +/// Build the typed `CreateTableRequest` body the connector sends to the UC `tables` endpoint after +/// the v0 commit succeeds, built from the post-commit v0 snapshot. +/// +/// User properties set via `create_table`'s `with_table_properties` flow through automatically +/// (they land in `metaData.configuration`). To add or override properties afterward, mutate +/// `req.properties` on the returned request before sending. +/// +/// UC requires `delta.checkpointPolicy=v2` in the request body, but kernel's CREATE TABLE rejects +/// it as a table property, so it never lands in the committed metadata. This function adds it to +/// the request body so callers get a complete, contract-valid request. +/// +/// # Errors +/// +/// Returns an error if `snapshot` is not at version 0, if the schema can't be serialized, or if the +/// engine fails to read clustering metadata or the commit timestamp. +pub fn build_uc_create_table_request( + snapshot: &Snapshot, + engine: &dyn Engine, + table_name: impl Into, +) -> DeltaResult { + if snapshot.version() != 0 { + return Err(Error::generic(format!( + "build_uc_create_table_request is only valid for version 0 (table creation) \ + snapshots, but snapshot is at version {}", + snapshot.version() + ))); + } + + let columns = serde_json::to_value(snapshot.schema().as_ref()) + .map_err(|e| Error::generic(format!("Failed to serialize table schema: {e}")))?; + + let table_config = snapshot.table_configuration(); + let metadata = table_config.metadata(); + let protocol = table_config.protocol(); + + let partition_columns = metadata.partition_columns().to_vec(); + let mut properties = metadata.configuration().clone(); + properties.insert( + CHECKPOINT_POLICY_KEY.to_string(), + CHECKPOINT_POLICY_V2.to_string(), + ); + let wire_protocol = to_wire_protocol(protocol); + + // UC only recognizes the `delta.clustering` and `delta.rowTracking` domain metadatas. + let uc_recognized_domains = HashSet::from([CLUSTERING_DOMAIN_NAME, ROW_TRACKING_DOMAIN_NAME]); + let mut domain_metadata: HashMap = HashMap::new(); + for (domain, dm) in + snapshot.get_domain_metadatas_internal(engine, Some(&uc_recognized_domains))? + { + let value = serde_json::from_str(dm.configuration()) + .map_err(|e| Error::generic(format!("malformed {domain} domain metadata: {e}")))?; + domain_metadata.insert(domain, value); + } + + let in_commit_timestamp_ms = snapshot.get_timestamp(engine)?; + + Ok(CreateTableRequest { + name: table_name.into(), + location: snapshot.table_root().to_string(), + table_type: "MANAGED".to_string(), + comment: None, + columns, + partition_columns, + protocol: wire_protocol, + properties, + domain_metadata, + last_commit_timestamp_ms: in_commit_timestamp_ms, + }) +} + +/// Convert a kernel `Protocol` into the api crate's wire `Protocol`. The api crate is kernel-free, +/// so it defines its own serializable transport type rather than reusing `delta_kernel::Protocol`. +fn to_wire_protocol(protocol: &Protocol) -> WireProtocol { + WireProtocol { + min_reader_version: protocol.min_reader_version(), + min_writer_version: protocol.min_writer_version(), + reader_features: protocol + .reader_features() + .into_iter() + .flatten() + .map(|f| f.as_ref().to_string()) + .collect(), + writer_features: protocol + .writer_features() + .into_iter() + .flatten() + .map(|f| f.as_ref().to_string()) + .collect(), + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use delta_kernel::expressions::column_name; + use delta_kernel::object_store::memory::InMemory; + use delta_kernel::schema::{schema_ref, DataType, SchemaRef, StructField, StructType}; + use delta_kernel::snapshot::Snapshot; + use delta_kernel::transaction::create_table::create_table; + use delta_kernel::transaction::data_layout::DataLayout; + use delta_kernel_default_engine::DefaultEngineBuilder; + use rstest::rstest; + use test_utils::TestCatalogCommitter; + + use super::*; + + /// Create a table at version 0 (optionally clustered), then build the UC create request from + /// its post-commit snapshot. Shared setup for the version-0 request tests. + fn create_v0_and_build_request( + engine: &dyn Engine, + table_path: &str, + schema: SchemaRef, + disk_props: HashMap, + data_layout: DataLayout, + ) -> CreateTableRequest { + create_table(table_path, schema, "Test/1.0") + .with_table_properties(disk_props) + .with_data_layout(data_layout) + .build(engine, Box::new(TestCatalogCommitter)) + .unwrap() + .commit(engine) + .unwrap() + .unwrap_committed(); + let snapshot = Snapshot::builder_for(table_path) + .with_max_catalog_version(0) + .build(engine) + .unwrap(); + assert_eq!(snapshot.version(), 0); + build_uc_create_table_request(&snapshot, engine, "my_table").unwrap() + } + + #[test] + fn test_get_required_properties_for_disk() { + let props = get_required_properties_for_disk("my-uc-table-123"); + assert_eq!(props.len(), 8); + assert_eq!(props["delta.feature.catalogManaged"], "supported"); + assert_eq!(props["delta.feature.vacuumProtocolCheck"], "supported"); + assert_eq!(props["delta.feature.v2Checkpoint"], "supported"); + assert_eq!(props["delta.feature.deletionVectors"], "supported"); + assert_eq!(props["delta.enableDeletionVectors"], "true"); + assert_eq!(props["delta.checkpoint.writeStatsAsStruct"], "true"); + assert_eq!(props["delta.checkpoint.writeStatsAsJson"], "true"); + assert_eq!(props["io.unitycatalog.tableId"], "my-uc-table-123"); + } + + #[tokio::test] + async fn test_build_uc_create_table_request() { + let storage = Arc::new(InMemory::new()); + let engine = DefaultEngineBuilder::new(storage).build(); + let table_path = "memory:///test_create_req/"; + let schema = Arc::new( + StructType::try_new(vec![ + StructField::new("id", DataType::INTEGER, true), + StructField::new("region", DataType::STRING, true), + ]) + .unwrap(), + ); + + let disk_props = get_required_properties_for_disk("test-table-id-456"); + let req = create_v0_and_build_request( + &engine, + table_path, + schema, + disk_props, + DataLayout::clustered(["region"]), + ); + + assert_eq!(req.protocol.min_reader_version, 3); + assert_eq!(req.protocol.min_writer_version, 7); + assert!(req + .protocol + .reader_features + .iter() + .any(|f| f == "catalogManaged")); + assert!(req + .protocol + .writer_features + .iter() + .any(|f| f == "inCommitTimestamp")); + + assert_eq!( + req.properties["io.unitycatalog.tableId"], + "test-table-id-456" + ); + assert_eq!(req.properties["delta.checkpointPolicy"], "v2"); + assert!(!req.properties.contains_key("delta.feature.catalogManaged")); + assert!(!req.properties.contains_key("delta.minReaderVersion")); + assert!(!req.properties.contains_key("delta.lastUpdateVersion")); + + // Clustering surfaces under domain_metadata in typed JSON form. + let clustering = req + .domain_metadata + .get("delta.clustering") + .expect("clustering domain metadata should be present"); + let cols = clustering + .get("clusteringColumns") + .and_then(|v| v.as_array()) + .expect("clusteringColumns should be a JSON array"); + assert_eq!(cols.len(), 1); + + assert_eq!( + req.columns.get("type").and_then(|v| v.as_str()), + Some("struct") + ); + assert_eq!(req.name, "my_table"); + assert!(req.location.contains("test_create_req")); + + assert!( + req.last_commit_timestamp_ms > 0, + "expected positive in-commit timestamp, got {}", + req.last_commit_timestamp_ms + ); + } + + #[tokio::test] + async fn test_build_uc_create_table_request_forwards_row_tracking() { + let storage = Arc::new(InMemory::new()); + let engine = DefaultEngineBuilder::new(storage).build(); + let table_path = "memory:///test_row_tracking/"; + let schema = schema_ref! { nullable "id": INTEGER }; + + let mut disk_props = get_required_properties_for_disk("test-table-id"); + disk_props.insert("delta.enableRowTracking".to_string(), "true".to_string()); + let req = + create_v0_and_build_request(&engine, table_path, schema, disk_props, DataLayout::None); + + let hwm = req.domain_metadata[ROW_TRACKING_DOMAIN_NAME]["rowIdHighWaterMark"].as_i64(); + assert_eq!( + hwm, + Some(-1), + "empty create should forward the initial watermark" + ); + } + + #[tokio::test] + async fn test_build_uc_create_table_request_silently_drops_user_domain_metadata() { + let storage = Arc::new(InMemory::new()); + let engine = DefaultEngineBuilder::new(storage).build(); + let table_path = "memory:///test_user_domain/"; + let schema = schema_ref! { nullable "id": INTEGER }; + + let mut disk_props = get_required_properties_for_disk("test-table-id"); + disk_props.insert( + "delta.feature.domainMetadata".to_string(), + "supported".to_string(), + ); + create_table(table_path, schema, "Test/1.0") + .with_table_properties(disk_props) + .build(&engine, Box::new(TestCatalogCommitter)) + .unwrap() + .with_domain_metadata("myApp.retention".to_string(), r#"{"days":30}"#.to_string()) + .commit(&engine) + .unwrap() + .unwrap_committed(); + + let snapshot = Snapshot::builder_for(table_path) + .with_max_catalog_version(0) + .build(&engine) + .unwrap(); + let req = build_uc_create_table_request(&snapshot, &engine, "test_user_domain").unwrap(); + assert!( + !req.domain_metadata.contains_key("myApp.retention"), + "user domain should be dropped, not forwarded" + ); + } + + // Clustering columns are forwarded verbatim from the committed `delta.clustering` domain. Under + // column mapping the committed paths are physical names, so the request must carry those, not + // the logical names. + #[rstest] + #[case::logical_names_without_column_mapping(false)] + #[case::physical_names_under_column_mapping(true)] + #[tokio::test] + async fn test_clustering_columns_forwarded(#[case] column_mapping: bool) { + let storage = Arc::new(InMemory::new()); + let engine = DefaultEngineBuilder::new(storage).build(); + let table_path = "memory:///test_clustering/"; + let schema = schema_ref! { + nullable "id": INTEGER, + nullable "region": STRING, + nullable "address": { + nullable "city": STRING, + nullable "zip": STRING, + }, + }; + + let mut disk_props = get_required_properties_for_disk("test-table-id"); + if column_mapping { + disk_props.insert("delta.columnMapping.mode".to_string(), "name".to_string()); + } + let req = create_v0_and_build_request( + &engine, + table_path, + schema, + disk_props, + DataLayout::Clustered { + columns: vec![column_name!("region"), column_name!("address", "city")], + }, + ); + + let cols = req.domain_metadata[CLUSTERING_DOMAIN_NAME]["clusteringColumns"].clone(); + let parsed: Vec> = serde_json::from_value(cols).unwrap(); + if column_mapping { + assert_eq!(parsed.len(), 2); + let flat: Vec<&str> = parsed.iter().flatten().map(String::as_str).collect(); + assert!(!flat.contains(&"region") && !flat.contains(&"city")); + } else { + assert_eq!(parsed, vec![vec!["region"], vec!["address", "city"]]); + } + } + + #[tokio::test] + async fn test_build_uc_create_table_request_rejects_non_zero_version() { + let storage = Arc::new(InMemory::new()); + let engine = DefaultEngineBuilder::new(storage).build(); + let table_path = "memory:///test_create_req_version/"; + let schema = Arc::new( + StructType::try_new(vec![StructField::new("id", DataType::INTEGER, true)]).unwrap(), + ); + + // Create a table (version 0) and append (version 1). + let disk_props = get_required_properties_for_disk("test-table-id"); + let _ = create_table(table_path, schema, "Test/1.0") + .with_table_properties(disk_props) + .build(&engine, Box::new(TestCatalogCommitter)) + .unwrap() + .commit(&engine) + .unwrap(); + let v0_snapshot = Snapshot::builder_for(table_path) + .with_max_catalog_version(0) + .build(&engine) + .unwrap(); + let result = v0_snapshot + .transaction(Box::new(TestCatalogCommitter), &engine) + .unwrap() + .commit(&engine) + .unwrap(); + assert!(result.is_committed()); + + // Load snapshot at version 1. + let snapshot = Snapshot::builder_for(table_path) + .with_max_catalog_version(1) + .build(&engine) + .unwrap(); + assert_eq!(snapshot.version(), 1); + + // Should fail because version != 0. + let err = build_uc_create_table_request(&snapshot, &engine, "x").unwrap_err(); + assert!( + err.to_string().contains("version 0"), + "expected version 0 error, got: {err}" + ); + } +} diff --git a/delta-kernel-unity-catalog/src/utils/mod.rs b/delta-kernel-unity-catalog/src/utils/mod.rs new file mode 100644 index 0000000000..e0557daa33 --- /dev/null +++ b/delta-kernel-unity-catalog/src/utils/mod.rs @@ -0,0 +1,5 @@ +pub(crate) mod create_table; + +pub use create_table::{ + aws_object_store_options, build_uc_create_table_request, get_required_properties_for_disk, +}; diff --git a/uc-catalog/tests/data/catalog_managed_0/_delta_log/00000000000000000000.json b/delta-kernel-unity-catalog/tests/data/catalog_managed_0/_delta_log/00000000000000000000.json similarity index 78% rename from uc-catalog/tests/data/catalog_managed_0/_delta_log/00000000000000000000.json rename to delta-kernel-unity-catalog/tests/data/catalog_managed_0/_delta_log/00000000000000000000.json index 86dc5e492b..4aa0f02197 100644 --- a/uc-catalog/tests/data/catalog_managed_0/_delta_log/00000000000000000000.json +++ b/delta-kernel-unity-catalog/tests/data/catalog_managed_0/_delta_log/00000000000000000000.json @@ -1,3 +1,3 @@ {"commitInfo":{"inCommitTimestamp":1749830855993,"timestamp":1749830855992,"operation":"CREATE TABLE","operationParameters":{"partitionBy":"[\"part1\"]","clusterBy":"[]","description":null,"isManaged":"false","properties":"{}"},"isolationLevel":"Serializable","isBlindAppend":true,"operationMetrics":{},"engineInfo":"Apache-Spark/4.0.0 Delta-Lake/4.0.0","txnId":"d108f896-9662-4eda-b4de-444a99850aa8"}} -{"metaData":{"id":"64dcd182-b3b4-4ee0-88e0-63c159a4121c","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"part1\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"col1\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["part1"],"configuration":{"delta.enableInCommitTimestamps":"true"},"createdTime":1749830855646}} -{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["catalogManaged"],"writerFeatures":["catalogManaged","inCommitTimestamp"]}} +{"metaData":{"id":"64dcd182-b3b4-4ee0-88e0-63c159a4121c","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"part1\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"col1\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["part1"],"configuration":{"delta.enableInCommitTimestamps":"true","io.unitycatalog.tableId":"64dcd182-b3b4-4ee0-88e0-63c159a4121c"},"createdTime":1749830855646}} +{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["catalogManaged","vacuumProtocolCheck"],"writerFeatures":["catalogManaged","inCommitTimestamp","vacuumProtocolCheck"]}} diff --git a/uc-catalog/tests/data/catalog_managed_0/_delta_log/_staged_commits/00000000000000000001.4cb9708e-b478-44de-b203-53f9ba9b2876.json b/delta-kernel-unity-catalog/tests/data/catalog_managed_0/_delta_log/_staged_commits/00000000000000000001.4cb9708e-b478-44de-b203-53f9ba9b2876.json similarity index 100% rename from uc-catalog/tests/data/catalog_managed_0/_delta_log/_staged_commits/00000000000000000001.4cb9708e-b478-44de-b203-53f9ba9b2876.json rename to delta-kernel-unity-catalog/tests/data/catalog_managed_0/_delta_log/_staged_commits/00000000000000000001.4cb9708e-b478-44de-b203-53f9ba9b2876.json diff --git a/uc-catalog/tests/data/catalog_managed_0/_delta_log/_staged_commits/00000000000000000002.5b9bba4a-0085-430d-a65e-b0d38c1afbe9.json b/delta-kernel-unity-catalog/tests/data/catalog_managed_0/_delta_log/_staged_commits/00000000000000000002.5b9bba4a-0085-430d-a65e-b0d38c1afbe9.json similarity index 100% rename from uc-catalog/tests/data/catalog_managed_0/_delta_log/_staged_commits/00000000000000000002.5b9bba4a-0085-430d-a65e-b0d38c1afbe9.json rename to delta-kernel-unity-catalog/tests/data/catalog_managed_0/_delta_log/_staged_commits/00000000000000000002.5b9bba4a-0085-430d-a65e-b0d38c1afbe9.json diff --git a/uc-catalog/tests/data/catalog_managed_0/part1=0/part-00000-13fefaba-8ec2-4762-b17e-aeda657451c5.c000.snappy.parquet b/delta-kernel-unity-catalog/tests/data/catalog_managed_0/part1=0/part-00000-13fefaba-8ec2-4762-b17e-aeda657451c5.c000.snappy.parquet similarity index 100% rename from uc-catalog/tests/data/catalog_managed_0/part1=0/part-00000-13fefaba-8ec2-4762-b17e-aeda657451c5.c000.snappy.parquet rename to delta-kernel-unity-catalog/tests/data/catalog_managed_0/part1=0/part-00000-13fefaba-8ec2-4762-b17e-aeda657451c5.c000.snappy.parquet diff --git a/uc-catalog/tests/data/catalog_managed_0/part1=1/part-00000-8afb1c56-2018-4af2-aa4f-4336c1b39efd.c000.snappy.parquet b/delta-kernel-unity-catalog/tests/data/catalog_managed_0/part1=1/part-00000-8afb1c56-2018-4af2-aa4f-4336c1b39efd.c000.snappy.parquet similarity index 100% rename from uc-catalog/tests/data/catalog_managed_0/part1=1/part-00000-8afb1c56-2018-4af2-aa4f-4336c1b39efd.c000.snappy.parquet rename to delta-kernel-unity-catalog/tests/data/catalog_managed_0/part1=1/part-00000-8afb1c56-2018-4af2-aa4f-4336c1b39efd.c000.snappy.parquet diff --git a/delta-kernel-unity-catalog/tests/e2e_in_memory.rs b/delta-kernel-unity-catalog/tests/e2e_in_memory.rs new file mode 100644 index 0000000000..7321d31660 --- /dev/null +++ b/delta-kernel-unity-catalog/tests/e2e_in_memory.rs @@ -0,0 +1,350 @@ +use std::path::PathBuf; +use std::sync::Arc; + +use delta_kernel::arrow::array::{ArrayRef, Int32Array, StringArray}; +use delta_kernel::object_store::local::LocalFileSystem; +use delta_kernel::schema::{DataType, StructField, StructType}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::{Engine, Snapshot}; +use delta_kernel_default_engine::executor::tokio::TokioMultiThreadExecutor; +use delta_kernel_default_engine::DefaultEngine; +use delta_kernel_unity_catalog::{ + get_required_properties_for_disk, snapshot_builder_from_load_table, UCCommitter, +}; +use test_utils::{insert_data_with, read_scan}; +use unity_catalog_delta_client_api::{ + Commit, InMemoryUpdateTableClient, TableData, TableIdentifier, +}; + +// ============================================================================ +// Test Setup +// ============================================================================ + +type TestError = Box; + +const TEST_CATALOG: &str = "test_catalog"; +const TEST_SCHEMA: &str = "test_schema"; +const TEST_TABLE: &str = "test_table"; +const TABLE_ID: &str = "64dcd182-b3b4-4ee0-88e0-63c159a4121c"; + +struct TestSetup { + update_table_client: Arc, + engine: DefaultEngine, + snapshot: Arc, + table_uri: url::Url, + /// Tests must bind this field (not ignore with `..` or `_`) to prevent the temp directory + /// from being dropped and cleaned up before the test completes. + _tmp_dir: tempfile::TempDir, +} + +/// Copies test data to temp dir and loads snapshot at v2 with in-memory commits client. +async fn setup() -> Result { + let src = PathBuf::from("./tests/data/catalog_managed_0/"); + let tmp_dir = tempfile::tempdir()?; + copy_dir_recursive(&src, tmp_dir.path())?; + + // v0 published, v1/v2 ratified but unpublished + let update_table_client = Arc::new(InMemoryUpdateTableClient::new()); + update_table_client.insert_table( + TABLE_ID, + TableData { + max_ratified_version: 2, + catalog_commits: vec![ + Commit::new( + 1, + 1749830871085, + "00000000000000000001.4cb9708e-b478-44de-b203-53f9ba9b2876.json", + 889, + 1749830870833, + ), + Commit::new( + 2, + 1749830881799, + "00000000000000000002.5b9bba4a-0085-430d-a65e-b0d38c1afbe9.json", + 891, + 1749830881779, + ), + ], + }, + ); + + let store = Arc::new(LocalFileSystem::new()); + let executor = Arc::new(TokioMultiThreadExecutor::new( + tokio::runtime::Handle::current(), + )); + let engine = delta_kernel_default_engine::DefaultEngineBuilder::new(store) + .with_task_executor(executor) + .build(); + let table_uri = url::Url::from_directory_path(tmp_dir.path()).map_err(|_| "invalid path")?; + + let resp = update_table_client.load_table_response(TABLE_ID, table_uri.as_str())?; + let snapshot = snapshot_builder_from_load_table(&resp)?.build(&engine)?; + + Ok(TestSetup { + update_table_client, + engine, + snapshot, + table_uri, + _tmp_dir: tmp_dir, + }) +} + +/// Recursively copies a directory tree. +fn copy_dir_recursive(src: &std::path::Path, dst: &std::path::Path) -> std::io::Result<()> { + std::fs::create_dir_all(dst)?; + for entry in std::fs::read_dir(src)? { + let entry = entry?; + let dst_path = dst.join(entry.file_name()); + if entry.path().is_dir() { + copy_dir_recursive(&entry.path(), &dst_path)?; + } else { + std::fs::copy(entry.path(), &dst_path)?; + } + } + Ok(()) +} + +fn uc_committer( + update_table_client: &Arc, +) -> UCCommitter { + UCCommitter::new( + update_table_client.clone(), + TABLE_ID, + TableIdentifier::new(TEST_CATALOG, TEST_SCHEMA, TEST_TABLE), + ) +} + +/// Commits an empty transaction and returns the post-commit snapshot. +fn commit( + snapshot: &Arc, + update_table_client: &Arc, + engine: &DefaultEngine, +) -> Result, TestError> { + Ok(snapshot + .clone() + .transaction(Box::new(uc_committer(update_table_client)), engine)? + .with_operation("WRITE".to_string()) + .commit(engine)? + .unwrap_post_commit_snapshot()) +} + +/// Loads a snapshot via the connector read path: `load_table` response -> snapshot builder. +fn build_snapshot( + client: &Arc, + engine: &DefaultEngine, + table_uri: &url::Url, +) -> Result, TestError> { + let resp = client.load_table_response(TABLE_ID, table_uri.as_str())?; + Ok(snapshot_builder_from_load_table(&resp)?.build(engine)?) +} + +// ============================================================================ +// Tests +// ============================================================================ + +// multi_thread required: UCCommitter uses block_on which panics on single-threaded runtime +#[tokio::test(flavor = "multi_thread")] +async fn test_scan_returns_fixture_rows() -> Result<(), TestError> { + let TestSetup { + engine, + snapshot, + _tmp_dir, + .. + } = setup().await?; + assert_eq!(snapshot.version(), 2); + + let scan = snapshot.scan_builder().build()?; + let engine_dyn: Arc = Arc::new(engine); + let batches = read_scan(&scan, engine_dyn)?; + + let total: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total, 200); + + let schema = batches + .first() + .ok_or("expected at least one batch")? + .schema(); + assert!(schema.column_with_name("part1").is_some()); + assert!(schema.column_with_name("col1").is_some()); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread")] +async fn test_insert_without_publish_hits_limit() -> Result<(), TestError> { + let TestSetup { + update_table_client, + engine, + mut snapshot, + _tmp_dir, + .. + } = setup().await?; + + // Start with 2 unpublished (v1, v2). Insert up to MAX, then the next should fail. + let max = TableData::MAX_UNPUBLISHED_COMMITS as u64; + for _ in 3..=max { + snapshot = commit(&snapshot, &update_table_client, &engine)?; + } + assert_eq!(snapshot.version(), max); + + // Next insert should fail with MaxUnpublishedCommitsExceeded + let committer = Box::new(uc_committer(&update_table_client)); + let err = snapshot + .clone() + .transaction(committer, &engine)? + .commit(&engine) + .unwrap_err(); + assert!( + matches!(err, delta_kernel::Error::Generic(msg) if msg.contains("Max unpublished commits")) + ); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread")] +async fn test_checkpoint_after_publish() -> Result<(), TestError> { + let TestSetup { + update_table_client, + engine, + snapshot, + table_uri, + _tmp_dir, + } = setup().await?; + + commit(&snapshot, &update_table_client, &engine)? + .publish(&engine, &uc_committer(&update_table_client))? + .checkpoint(&engine, None)?; + + // Load a fresh snapshot and verify checkpoint was written + let snapshot = Snapshot::builder_for(table_uri) + .with_max_catalog_version(3) + .build(&engine)?; + assert_eq!(snapshot.log_segment().checkpoint_version, Some(3)); + + Ok(()) +} + +#[tokio::test(flavor = "multi_thread")] +async fn test_cannot_checkpoint_unpublished_snapshot() -> Result<(), TestError> { + let TestSetup { + update_table_client, + engine, + snapshot, + _tmp_dir, + .. + } = setup().await?; + + let snapshot = commit(&snapshot, &update_table_client, &engine)?; + let err = snapshot.checkpoint(&engine, None).unwrap_err(); + assert!(matches!(err, delta_kernel::Error::Generic(msg) if msg.contains("not published"))); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread")] +async fn test_append_scan_back_and_incremental_read() -> Result<(), TestError> { + let tmp = tempfile::tempdir()?; + let table_uri = url::Url::from_directory_path(tmp.path()).map_err(|_| "invalid path")?; + + let store = Arc::new(LocalFileSystem::new()); + let executor = Arc::new(TokioMultiThreadExecutor::new( + tokio::runtime::Handle::current(), + )); + let engine = Arc::new( + delta_kernel_default_engine::DefaultEngineBuilder::new(store) + .with_task_executor(executor) + .build(), + ); + + let client = Arc::new(InMemoryUpdateTableClient::new()); + + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("val", DataType::STRING), + ])?); + + // v0 create: writes 000.json directly to storage, does not call the catalog. + create_table(table_uri.as_str(), schema, "delta-kernel-uc-test") + .with_table_properties(get_required_properties_for_disk(TABLE_ID)) + .build(engine.as_ref(), Box::new(uc_committer(&client)))? + .commit(engine.as_ref())? + .unwrap_committed(); + client.create_table(TABLE_ID)?; + + let snap_v0 = build_snapshot(&client, &engine, &table_uri)?; + assert_eq!(snap_v0.version(), 0); + let scan = snap_v0.clone().scan_builder().build()?; + let v0_rows: usize = read_scan(&scan, engine.clone() as Arc)? + .iter() + .map(|b| b.num_rows()) + .sum(); + assert_eq!(v0_rows, 0); + + // v1 append: 3 rows. + let id_col: ArrayRef = Arc::new(Int32Array::from(vec![1, 2, 3])); + let val_col: ArrayRef = Arc::new(StringArray::from(vec!["a", "b", "c"])); + insert_data_with( + snap_v0, + &engine, + vec![id_col, val_col], + Box::new(uc_committer(&client)), + "WRITE", + true, + false, + ) + .await? + .unwrap_committed(); + + let snap_v1 = build_snapshot(&client, &engine, &table_uri)?; + assert_eq!(snap_v1.version(), 1); + let scan = snap_v1.clone().scan_builder().build()?; + let v1_batches = read_scan(&scan, engine.clone() as Arc)?; + let v1_rows: usize = v1_batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(v1_rows, 3); + let mut ids: Vec = v1_batches + .iter() + .flat_map(|b| { + let col = b + .column_by_name("id") + .expect("id column") + .as_any() + .downcast_ref::() + .expect("id is int32"); + col.values().to_vec() + }) + .collect(); + // Scan row order is not guaranteed, so sort before comparing. + ids.sort_unstable(); + assert_eq!(ids, vec![1, 2, 3]); + + // Append 2 rows. A scan at v2 should see 5 rows. + let id_col: ArrayRef = Arc::new(Int32Array::from(vec![4, 5])); + let val_col: ArrayRef = Arc::new(StringArray::from(vec!["d", "e"])); + insert_data_with( + snap_v1, + &engine, + vec![id_col, val_col], + Box::new(uc_committer(&client)), + "WRITE", + true, + false, + ) + .await? + .unwrap_committed(); + + let snap_v2 = build_snapshot(&client, &engine, &table_uri)?; + assert_eq!(snap_v2.version(), 2); + let scan = snap_v2.clone().scan_builder().build()?; + let v2_rows: usize = read_scan(&scan, engine.clone() as Arc)? + .iter() + .map(|b| b.num_rows()) + .sum(); + assert_eq!(v2_rows, 5); + + // Publish, then confirm reads still return all rows. + let published = snap_v2.publish(engine.as_ref(), &uc_committer(&client))?; + let scan = published.scan_builder().build()?; + let pub_rows: usize = read_scan(&scan, engine.clone() as Arc)? + .iter() + .map(|b| b.num_rows()) + .sum(); + assert_eq!(pub_rows, 5); + Ok(()) +} diff --git a/delta-kernel-unity-catalog/tests/integration_live_server.rs b/delta-kernel-unity-catalog/tests/integration_live_server.rs new file mode 100644 index 0000000000..595dad6c74 --- /dev/null +++ b/delta-kernel-unity-catalog/tests/integration_live_server.rs @@ -0,0 +1,329 @@ +//! Live end-to-end tests against a running Unity Catalog server. +//! +//! These are gated behind the `integration-test` feature so they are never compiled by a normal +//! `cargo nextest`, and marked `#[ignore]` so that even with the feature enabled they report as +//! skipped rather than passing. Run them against a UC server with +//! `--run-ignored only -E 'test(live_)'` (see the UC CI job). Each also returns early if +//! `UC_SERVER_URL` is unset. +//! +//! Run against a local UC server: +//! ```bash +//! UC_SERVER_URL=http://localhost:8080 UC_TOKEN=not-used \ +//! cargo nextest run -p delta-kernel-unity-catalog --features integration-test +//! ``` +//! +//! Optional overrides for the read-path test: `UC_TEST_CATALOG`, `UC_TEST_SCHEMA`, +//! `UC_TEST_TABLE` (the read test skips unless `UC_TEST_TABLE` is set). +//! +//! The CREATE test (`live_create_table`) mutates the catalog, so point `UC_SERVER_URL` only at a +//! throwaway server or schema. +//! +//! These tests prove the RPCs round-trip, not that the payloads are semantically correct. +#![cfg(feature = "integration-test")] + +use std::sync::Arc; + +use delta_kernel::arrow::array::{ArrayRef, Int32Array, StringArray, StructArray}; +use delta_kernel::arrow::datatypes::{DataType as ArrowDataType, Field}; +use delta_kernel::expressions::column_name; +use delta_kernel::schema::{DataType, SchemaRef, StructField, StructType}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::transaction::data_layout::DataLayout; +use delta_kernel::{Engine, Snapshot}; +use delta_kernel_default_engine::storage::store_from_url_opts; +use delta_kernel_default_engine::DefaultEngineBuilder; +use delta_kernel_unity_catalog::{ + aws_object_store_options, build_uc_create_table_request, get_required_properties_for_disk, + log_tail_from_commits, snapshot_builder_from_load_table, UCCommitter, +}; +use test_utils::{insert_data_with, read_scan}; +use unity_catalog_delta_client_api::{ + CreateStagingTableRequest, LoadTableResponse, TableIdentifier, +}; +use unity_catalog_delta_rest_client::{ClientConfig, UCClient, UCUpdateTableRestClient}; +use url::Url; + +/// Returns `(server_url, token)` from the environment, or `None` to signal the caller to skip. +/// `UC_TOKEN` defaults to a dummy value since a dev-mode server runs with auth disabled. +fn server_env() -> Option<(String, String)> { + let url = std::env::var("UC_SERVER_URL").ok()?; + let token = std::env::var("UC_TOKEN").unwrap_or_else(|_| "not-used".to_string()); + Some((url, token)) +} + +fn client_config(url: &str, token: &str) -> ClientConfig { + ClientConfig::build(url, token) + .build() + .expect("failed to build ClientConfig") +} + +fn client(url: &str, token: &str) -> UCClient { + UCClient::new(client_config(url, token)).expect("failed to build UCClient") +} + +/// Normalize a UC table location to a root URL ending in `/`. UC returns the location without a +/// trailing slash, but staged-commit path resolution and snapshot building require one. +fn normalize_table_root(table_url: &Url) -> Url { + let mut root = table_url.clone(); + if !root.path().ends_with('/') { + root.set_path(&format!("{}/", root.path())); + } + root +} + +fn snapshot_from_load_table(resp: &LoadTableResponse, engine: &dyn Engine) -> Arc { + snapshot_builder_from_load_table(resp) + .expect("build snapshot builder from load_table response") + .build(engine) + .expect("build snapshot from load_table response") +} + +/// `load_table` returns parseable metadata + commits, and the commits convert to a kernel log tail. +/// Skips unless `UC_TEST_TABLE` names an existing managed delta table. This validates the read-path +/// wire types without requiring storage credentials (it does not read data files). +#[tokio::test(flavor = "multi_thread")] +#[ignore = "requires a running UC server; run with --run-ignored (UC CI job or manual)"] +async fn live_load_table_builds_log_tail() { + let Some((url, token)) = server_env() else { + eprintln!("UC_SERVER_URL unset; skipping live_load_table_builds_log_tail"); + return; + }; + let Some(table) = std::env::var("UC_TEST_TABLE").ok() else { + eprintln!("UC_TEST_TABLE unset; skipping live_load_table_builds_log_tail"); + return; + }; + let catalog = std::env::var("UC_TEST_CATALOG").unwrap_or_else(|_| "unity".to_string()); + let schema = std::env::var("UC_TEST_SCHEMA").unwrap_or_else(|_| "default".to_string()); + + let resp = client(&url, &token) + .load_table(&catalog, &schema, &table) + .await + .expect("load_table failed"); + + assert!( + !resp.metadata.table_uuid.is_empty(), + "expected a table_uuid in the load_table response" + ); + let table_url = Url::parse(&resp.metadata.location).expect("location is not a valid URL"); + + // Pure transform (no I/O): proves the commit wire shape maps onto kernel log paths. + let log_tail = log_tail_from_commits(&resp.commits, table_url) + .expect("log_tail_from_commits failed on the server's commit list"); + assert_eq!( + log_tail.len(), + resp.commits.len(), + "every returned commit should map to a log path" + ); +} + +/// Live CREATE through the full connector flow: `UCClient::create_staging_table` to reserve, +/// kernel for the v0 commit, then `UCClient::create_table` to register. +/// +/// The table enables row tracking and clustering so the create body and the write path exercise the +/// `delta.rowTracking` and `delta.clustering` domains in one round trip: the initial watermark (-1) +/// is forwarded at create and advances to 2 after the append (the CTAS concern), and the clustering +/// columns are forwarded under `delta.clustering`. +/// +/// This mutates the catalog, so point it only at a throwaway server or schema. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "requires a running UC server; run with --run-ignored (UC CI job or manual)"] +async fn live_create_table() { + let Some((url, token)) = server_env() else { + eprintln!("UC_SERVER_URL unset; skipping live_create_table"); + return; + }; + let catalog = std::env::var("UC_TEST_CATALOG").unwrap_or_else(|_| "unity".to_string()); + let schema_name = std::env::var("UC_TEST_SCHEMA").unwrap_or_else(|_| "default".to_string()); + // Unique per run so reruns never collide on the staging-tables POST (which 409s on a name + // already present). + let name = format!("kernel_rs_create_test_{}", uuid::Uuid::new_v4().simple()); + let name = name.as_str(); + + let uc_client = client(&url, &token); + + // ===== Step 1: reserve a staging table -> allocate uuid + storage + staging credentials ===== + let resp = uc_client + .create_staging_table( + &catalog, + &schema_name, + CreateStagingTableRequest { + name: name.to_string(), + }, + ) + .await + .expect("create_staging_table failed"); + + // ===== Step 2: Build the engine over the staging storage location ===== + let table_root = + normalize_table_root(&Url::parse(&resp.location).expect("location is not a valid URL")); + let store = store_from_url_opts( + &table_root, + aws_object_store_options(&resp.storage_credentials, "us-east-1"), + ) + .expect("failed to build object store from staging credentials"); + let engine = Arc::new(DefaultEngineBuilder::new(store).build()); + + // A server backed by local-FS storage (e.g. the server in CI) allocates the table location + // logically without creating the directory; kernel lists the table root during create-table and + // `LocalFileSystem` errors on a missing path (a cloud object store returns empty). Create it + // for file:// locations so the create-table build can proceed. + if table_root.scheme() == "file" { + if let Ok(path) = table_root.to_file_path() { + std::fs::create_dir_all(&path).expect("failed to create local table directory"); + } + } + + // ===== Step 3: Define the schema ===== + // A nested struct so clustering can target both a top-level and a nested + // column. + let schema: SchemaRef = Arc::new( + StructType::try_new(vec![ + StructField::not_null("id", DataType::INTEGER), + StructField::nullable("name", DataType::STRING), + StructField::nullable( + "address", + StructType::try_new(vec![ + StructField::nullable("city", DataType::STRING), + StructField::nullable("zip", DataType::STRING), + ]) + .expect("failed to build nested schema"), + ), + ]) + .expect("failed to build schema"), + ); + + // ===== Step 4: Invoke kernel to write the v0 commit (00.json) via the UC committer ===== + // The v0 path writes directly to storage and does not call the catalog. Enable row tracking so + // the create body carries `delta.rowTracking`, and cluster on `name` and the nested + // `address.city` so it carries `delta.clustering`. + let mut disk_props = get_required_properties_for_disk(&resp.table_id); + disk_props.insert("delta.enableRowTracking".to_string(), "true".to_string()); + let committer = Box::new(UCCommitter::new( + Arc::new(UCUpdateTableRestClient::new(client_config(&url, &token)).expect("update client")), + resp.table_id.clone(), + TableIdentifier::new(catalog.clone(), schema_name.clone(), name.to_string()), + )); + create_table(table_root.as_str(), schema, "delta-kernel-rs-live-test") + .with_table_properties(disk_props) + .with_data_layout(DataLayout::Clustered { + columns: vec![column_name!("name"), column_name!("address", "city")], + }) + .build(engine.as_ref(), committer) + .expect("failed to build create-table transaction") + .commit(engine.as_ref()) + .expect("failed to commit create-table transaction") + .unwrap_committed(); + + // ===== Step 5: Load the post-commit v0 snapshot ===== + // The table is catalog-managed, so the snapshot build requires the catalog's max version; the + // just-created table is at v0. + let snapshot = Snapshot::builder_for(table_root.clone()) + .with_max_catalog_version(0) + .build(engine.as_ref()) + .expect("failed to build post-commit snapshot"); + assert_eq!( + snapshot.version(), + 0, + "post-create snapshot should be at version 0" + ); + + // ===== Step 6: Build the typed create body and register the table with UC (POST tables) ===== + let req = build_uc_create_table_request(&snapshot, engine.as_ref(), name) + .expect("failed to build CreateTableRequest"); + // Empty create forwards the initial row-tracking watermark and the clustering columns. + assert_eq!( + req.domain_metadata["delta.rowTracking"]["rowIdHighWaterMark"].as_i64(), + Some(-1), + "empty create should forward the initial watermark" + ); + let clustering: Vec> = serde_json::from_value( + req.domain_metadata["delta.clustering"]["clusteringColumns"].clone(), + ) + .expect("clusteringColumns should deserialize"); + assert_eq!(clustering, vec![vec!["name"], vec!["address", "city"]]); + for (key, value) in [ + ("delta.checkpoint.writeStatsAsStruct", "true"), + ("delta.checkpoint.writeStatsAsJson", "true"), + ("delta.enableDeletionVectors", "true"), + ("delta.checkpointPolicy", "v2"), + ] { + assert_eq!( + req.properties.get(key).map(String::as_str), + Some(value), + "{key} should be present in the create body" + ); + } + uc_client + .create_table(&catalog, &schema_name, req) + .await + .expect("create_table failed"); + + // ===== Step 7: Verify registration: the table loads and its uuid matches the staging id ===== + let loaded = uc_client + .load_table(&catalog, &schema_name, name) + .await + .expect("load_table after create failed"); + assert_eq!( + loaded.metadata.table_uuid, resp.table_id, + "loaded table uuid should match the staging table id" + ); + + // ===== Step 8: Append 3 rows through the connector write path, then scan them back ===== + let uc = client(&url, &token); + let pre = uc + .load_table(&catalog, &schema_name, name) + .await + .expect("load_table before append failed"); + let snapshot = snapshot_from_load_table(&pre, engine.as_ref()); + + let id_col: ArrayRef = Arc::new(Int32Array::from(vec![10, 20, 30])); + let name_col: ArrayRef = Arc::new(StringArray::from(vec!["x", "y", "z"])); + let address_col: ArrayRef = Arc::new(StructArray::from(vec![ + ( + Arc::new(Field::new("city", ArrowDataType::Utf8, true)), + Arc::new(StringArray::from(vec!["nyc", "sf", "la"])) as ArrayRef, + ), + ( + Arc::new(Field::new("zip", ArrowDataType::Utf8, true)), + Arc::new(StringArray::from(vec!["10001", "94103", "90001"])) as ArrayRef, + ), + ])); + let append_committer = Box::new(UCCommitter::new( + Arc::new(UCUpdateTableRestClient::new(client_config(&url, &token)).expect("update client")), + resp.table_id.clone(), + TableIdentifier::new(catalog.clone(), schema_name.clone(), name.to_string()), + )); + insert_data_with( + snapshot, + &engine, + vec![id_col, name_col, address_col], + append_committer, + "WRITE", + true, + false, + ) + .await + .expect("append failed") + .unwrap_committed(); + + let post = uc + .load_table(&catalog, &schema_name, name) + .await + .expect("load_table after append failed"); + let snapshot = snapshot_from_load_table(&post, engine.as_ref()); + + // Appending 3 rows to a row-tracking table assigns IDs 0..=2, advancing the mark from -1 to 2. + let row_tracking = snapshot + .get_domain_metadata_internal("delta.rowTracking", engine.as_ref()) + .expect("failed to read delta.rowTracking domain metadata"); + assert_eq!( + row_tracking.as_deref(), + Some(r#"{"rowIdHighWaterMark":2}"#), + "after appending 3 rows the high water mark should advance from -1 to 2" + ); + + let scan = snapshot.scan_builder().build().expect("scan build failed"); + let batches = read_scan(&scan, engine.clone() as Arc).expect("read_scan failed"); + let rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(rows, 3, "appended rows should be returned by the scan"); +} diff --git a/derive-macros/Cargo.toml b/derive-macros/Cargo.toml index 47d8b670af..c62c6bee63 100644 --- a/derive-macros/Cargo.toml +++ b/derive-macros/Cargo.toml @@ -17,3 +17,6 @@ proc-macro = true proc-macro2 = "1" syn = { version = "2.0", features = ["extra-traits", "full"] } quote = "1.0" + +[dev-dependencies] +rstest = "0.23" diff --git a/derive-macros/src/lib.rs b/derive-macros/src/lib.rs index eeb7a1d124..160486d0ce 100644 --- a/derive-macros/src/lib.rs +++ b/derive-macros/src/lib.rs @@ -1,29 +1,113 @@ use proc_macro2::{Ident, Span, TokenStream}; use quote::{quote, quote_spanned, ToTokens}; -use syn::parse_macro_input; +use syn::parse::Parser; +use syn::punctuated::Punctuated; use syn::spanned::Spanned; use syn::{ - Data, DataStruct, DeriveInput, Error, Fields, Item, Meta, PathArguments, Type, Visibility, + parse_macro_input, Attribute, Data, DataStruct, DeriveInput, Error, Expr, ExprLit, Field, + Fields, Item, Lit, Meta, PathArguments, Token, Type, Visibility, }; -/// Parses a dot-delimited column name into an array of field names. See -/// `delta_kernel::expressions::column_name::column_name` macro for details. +mod schema_macro; + +// Builds a `StructType`; see `delta_kernel::schema::schema` re-export for details. +#[doc(hidden)] +#[proc_macro] +pub fn schema(input: proc_macro::TokenStream) -> proc_macro::TokenStream { + schema_macro::parse_schema(input, false, |block| block) +} + +// Fallible version of `schema!`; see `delta_kernel::schema::try_schema` re-export for details. +#[doc(hidden)] #[proc_macro] -pub fn parse_column_name(input: proc_macro::TokenStream) -> proc_macro::TokenStream { - let is_valid = |c: char| c.is_ascii_alphanumeric() || c == '_' || c == '.'; - let err = match syn::parse(input) { - Ok(syn::Lit::Str(name)) => match name.value().chars().find(|c| !is_valid(*c)) { - Some(bad_char) => Error::new(name.span(), format!("Invalid character: {bad_char:?}")), - _ => { - let path = name.value(); - let path = path.split('.').map(proc_macro2::Literal::string); - return quote_spanned! { name.span() => [#(#path),*] }.into(); +pub fn try_schema(input: proc_macro::TokenStream) -> proc_macro::TokenStream { + // Wrap the block in a closure that anchors the block's `?` operators + schema_macro::parse_schema(input, true, |block| { + quote! { (|| -> delta_kernel::DeltaResult<_> #block)() } + }) +} + +// `Arc`-wrapped `schema!`; see `delta_kernel::schema::schema_ref` for details. +#[doc(hidden)] +#[proc_macro] +pub fn schema_ref(input: proc_macro::TokenStream) -> proc_macro::TokenStream { + schema_macro::parse_schema(input, false, |block| quote!(::std::sync::Arc::new(#block))) +} + +// `LazyLock`-wrapped `schema!`; see `delta_kernel::schema::lazy_schema_ref`. +#[doc(hidden)] +#[proc_macro] +pub fn lazy_schema_ref(input: proc_macro::TokenStream) -> proc_macro::TokenStream { + schema_macro::parse_schema( + input, + false, + |block| quote!(::std::sync::LazyLock::new(|| ::std::sync::Arc::new(#block))), + ) +} + +/// Builds `&[&str]` from the arguments to `column_name!`. Each **string-literal** argument is a +/// dot-separated path, split into individual segments; every other (constant) argument is taken as +/// a single segment, validated at const-eval time. All segments must match `[a-zA-Z0-9_]+`. +#[proc_macro] +pub fn column_name_segments(input: proc_macro::TokenStream) -> proc_macro::TokenStream { + column_name_segments_impl(input.into()) + .unwrap_or_else(Error::into_compile_error) + .into() +} + +fn column_name_segments_impl(input: TokenStream) -> Result { + // `column_name!` requires at least one argument, so no empty check is needed here. + let mut emitted = Vec::new(); + for expr in Punctuated::::parse_terminated.parse2(input)? { + // Fragment-forwarding macros (col!, column_name!, joined_column_name!, ...) capture with + // `:literal`/`:expr` and re-emit, which wraps the argument in an invisible (none-delimited) + // group. Peel away those groups to recover an underlying string literal. + let mut inner = &expr; + while let Expr::Group(group) = inner { + inner = &group.expr; + } + match inner { + // A string literal is a dot-separated path: split it and validate each segment now. + Expr::Lit(ExprLit { + lit: Lit::Str(lit_str), + .. + }) => { + for segment in lit_str.value().split('.') { + validate_single_segment(segment, lit_str.span())?; + let literal = proc_macro2::Literal::string(segment); + emitted.push(quote_spanned! { lit_str.span() => #literal }); + } } - }, - Ok(lit) => Error::new(lit.span(), "Expected a string literal"), - Err(err) => err, - }; - err.into_compile_error().into() + // A string constant's value isn't visible until const-eval (after this macro expands), + // so wrap the value in a const fn validator with failure as a compile-time panic. + _ => emitted.push(quote_spanned! { expr.span() => + match ::delta_kernel::expressions::__require_valid_simple_column_segment(#expr) { + Some(segment) => segment, + None => panic!("String constants passed to column_name! must be simple names \ + matching [a-zA-Z0-9_]+; use a string literal for dot-separated \ + paths, or ColumnName::new() to disambiguate"), + } + }), + } + } + + Ok(quote! { &[ #(#emitted),* ] }) +} + +fn validate_single_segment(segment: &str, span: Span) -> Result<(), Error> { + if segment.is_empty() { + return Err(Error::new(span, "empty column name segment")); + } + if let Some(bad) = segment + .chars() + .find(|c| !(c.is_ascii_alphanumeric() || *c == '_')) + { + return Err(Error::new( + span, + format!("invalid character {bad:?} in column name segment {segment:?}"), + )); + } + Ok(()) } /// Derive a `delta_kernel::schemas::ToSchema` implementation for the annotated struct. The actual @@ -34,16 +118,32 @@ pub fn parse_column_name(input: proc_macro::TokenStream) -> proc_macro::TokenStr /// action (this macro allows the use of standard rust snake_case, and will convert to the correct /// delta schema camelCase version). /// -/// If a field sets `allow_null_container_values`, it means the underlying data can contain null in -/// the values of the container (i.e. a `key` -> `null` in a `HashMap`). Therefore the schema should -/// mark the value field as nullable, but those mappings will be dropped when converting to an -/// actual rust `HashMap`. Currently this can _only_ be set on `HashMap` fields. -#[proc_macro_derive(ToSchema, attributes(allow_null_container_values))] +/// Supported field attributes: +/// - `#[field_id = N]`: Sets the Parquet field ID for this field. `N` must be in `0..=i32::MAX` +/// (`0` is a valid reserved Iceberg field ID, e.g. the manifest-entry `status` field). +/// - `#[nested_field_id = N]`: Sets the Parquet field ID for the element of a list field (`Vec` +/// or `Option>`; rejected on any other type). Stored as `ColumnMappingNestedIds` metadata +/// (`{"fieldName.element": N}`) on the parent `StructField` and propagated to the inner Arrow +/// list element during schema conversion. `N` must be in `0..=i32::MAX`. +/// - `#[allow_null_container_values]`: Marks the value field of a container as nullable, i.e. the +/// underlying data can contain null in the values of the container (a `key` -> `null` in a +/// `HashMap`). Those mappings will be dropped when converting to an actual rust `HashMap`. +/// Currently this can _only_ be set on `HashMap` fields. +/// - `#[skip_schema]`: Excludes this field from the generated schema (and, on a struct that also +/// derives `IntoEngineData` / `IntoStructData`, from the produced engine data / struct scalar). +/// NOTE: `TryFromStructData` rejects skipped fields because it cannot reconstruct them. +#[proc_macro_derive( + ToSchema, + attributes(allow_null_container_values, field_id, nested_field_id, skip_schema) +)] pub fn derive_to_schema(input: proc_macro::TokenStream) -> proc_macro::TokenStream { let input = parse_macro_input!(input as DeriveInput); let struct_ident = input.ident; - let schema_fields = gen_schema_fields(&input.data); + let schema_fields = match gen_schema_fields(&input.data, struct_ident.span()) { + Ok(tokens) => tokens, + Err(e) => return e.to_compile_error().into(), + }; let output = quote! { #[automatically_derived] impl delta_kernel::schema::ToSchema for #struct_ident { @@ -82,60 +182,229 @@ fn get_schema_name(name: &Ident) -> Ident { Ident::new(&ret, name.span()) } -fn gen_schema_fields(data: &Data) -> TokenStream { - let fields = match data { - Data::Struct(DataStruct { - fields: Fields::Named(fields), - .. - }) => &fields.named, - _ => { - return Error::new( - Span::call_site(), - "this derive macro only works on structs with named fields", - ) - .to_compile_error() - } +/// Reads a `#[ = N]` field attribute, returning `Ok(None)` if absent. The value must be +/// an integer literal in `1..=i32::MAX` (the valid Parquet field-id range); a bare `#[]` +/// or `#[(..)]` shape, a non-integer, or an out-of-range value is a hard error. If the +/// attribute is repeated, the first occurrence wins. +fn get_named_attr_id( + field_attributes: &[Attribute], + attr_name: &str, +) -> Result, Error> { + // Match on the attribute path first so a malformed shape (bare path / list form) on the right + // name is a diagnosable error rather than silently skipped. + let Some(attr) = field_attributes + .iter() + .find(|attr| attr.path().is_ident(attr_name)) + else { + return Ok(None); }; + let Meta::NameValue(nv) = &attr.meta else { + return Err(Error::new( + attr.span(), + format!("{attr_name} error: expected `#[{attr_name} = N]`"), + )); + }; + let Expr::Lit(ExprLit { + lit: Lit::Int(lit_int), + .. + }) = &nv.value + else { + return Err(Error::new( + nv.value.span(), + format!("{attr_name} error: expected an integer"), + )); + }; + let value: i64 = lit_int + .base10_parse() + .map_err(|e| Error::new(lit_int.span(), format!("{attr_name} error: {e}")))?; + if !(0..=i64::from(i32::MAX)).contains(&value) { + return Err(Error::new( + lit_int.span(), + format!("{attr_name} error: field id {value} must be in 0..=2147483647"), + )); + } + Ok(Some(value)) +} - let schema_fields = fields.iter().map(|field| { - let name = field.ident.as_ref().unwrap(); // we know these are named fields - let name = get_schema_name(name); - let have_schema_null = field.attrs.iter().any(|attr| { - // check if we have allow_null_container_values attr - match &attr.meta { - Meta::Path(path) => path.get_ident().is_some_and(|ident| ident == "allow_null_container_values"), - _ => false, - } - }); - - match field.ty { - Type::Path(ref type_path) => { - let type_path_quoted = type_path.path.segments.iter().map(|segment| { - let segment_ident = &segment.ident; - match &segment.arguments { - PathArguments::None => quote! { #segment_ident :: }, - PathArguments::AngleBracketed(angle_args) => quote! { #segment_ident::#angle_args :: }, - _ => Error::new(segment.arguments.span(), "Can only handle <> type path args").to_compile_error() +/// Check if a path segment is `Option<{inner}<..>>` (e.g. `Option>`), matching on the +/// inner type's last segment identifier. +fn is_option_of(seg: &syn::PathSegment, inner: &str) -> bool { + if seg.ident != "Option" { + return false; + } + let PathArguments::AngleBracketed(angle_args) = &seg.arguments else { + return false; + }; + // Option has exactly one type argument + let Some(syn::GenericArgument::Type(Type::Path(inner_type))) = angle_args.args.first() else { + return false; + }; + inner_type + .path + .segments + .last() + .is_some_and(|seg| seg.ident == inner) +} + +/// Check if a path segment is a list-shaped type: `Vec` or `Option>`. +fn is_list_segment(seg: &syn::PathSegment) -> bool { + seg.ident == "Vec" || is_option_of(seg, "Vec") +} + +/// Check if any of `attrs` is a bare-path attribute named `name` (e.g. `#[skip_schema]`). +fn has_named_attr(attrs: &[Attribute], name: &str) -> bool { + attrs.iter().any(|attr| match &attr.meta { + Meta::Path(path) => path.is_ident(name), + _ => false, + }) +} + +fn gen_schema_field(field: &Field) -> TokenStream { + let name = get_schema_name(field.ident.as_ref().unwrap()); + let have_schema_null = has_named_attr(&field.attrs, "allow_null_container_values"); + + match field.ty { + Type::Path(ref type_path) => { + let type_path_quoted = type_path.path.segments.iter().map(|segment| { + let segment_ident = &segment.ident; + match &segment.arguments { + PathArguments::None => quote! { #segment_ident :: }, + PathArguments::AngleBracketed(angle_args) => { + quote! { #segment_ident::#angle_args :: } } - }); - if have_schema_null { - if let Some(last_ident) = type_path.path.segments.last().map(|seg| &seg.ident) { - if last_ident != "HashMap" { - return Error::new( - last_ident.span(), - format!("Can only use allow_null_container_values on HashMap fields, not {last_ident}") - ).to_compile_error() - } + _ => Error::new( + segment.arguments.span(), + "Can only handle <> type path args", + ) + .to_compile_error(), + } + }); + + let base_call = if have_schema_null { + if let Some(last_seg) = type_path.path.segments.last() { + let is_valid = last_seg.ident == "HashMap" || is_option_of(last_seg, "HashMap"); + if !is_valid { + return Error::new( + last_seg.ident.span(), + format!( + "Can only use allow_null_container_values on HashMap or \ + Option fields, not {}", + last_seg.ident + ), + ) + .to_compile_error(); } - quote_spanned! { field.span() => #(#type_path_quoted)* get_nullable_container_struct_field(stringify!(#name))} - } else { - quote_spanned! { field.span() => #(#type_path_quoted)* get_struct_field(stringify!(#name))} } + quote_spanned! { field.span() => #(#type_path_quoted)* get_nullable_container_struct_field(stringify!(#name)) } + } else { + quote_spanned! { field.span() => #(#type_path_quoted)* get_struct_field(stringify!(#name)) } + }; + + let field_id = match get_named_attr_id(&field.attrs, "field_id") { + Ok(v) => v, + Err(e) => return e.to_compile_error(), + }; + let nested_field_id = match get_named_attr_id(&field.attrs, "nested_field_id") { + Ok(v) => v, + Err(e) => return e.to_compile_error(), + }; + + let with_field_id = if let Some(id) = field_id { + quote_spanned! { field.span() => #base_call.add_metadata([(delta_kernel::schema::ColumnMetadataKey::ParquetFieldId.as_ref(), #id)]) } + } else { + quote_spanned! { field.span() => #base_call } + }; + + if let Some(elem_id) = nested_field_id { + // The `.element` nested id is only meaningful for a list element; reject it + // on any other type, mirroring the allow_null_container_values guard above. + let is_list = type_path.path.segments.last().is_some_and(is_list_segment); + if !is_list { + return Error::new( + field.span(), + format!( + "Can only use nested_field_id on Vec or Option fields, not {}", + type_path.path.segments.last().map_or_else( + || "".to_string(), + |seg| seg.ident.to_string() + ) + ), + ) + .to_compile_error(); + } + let nested_key = format!("{}.element", name); + quote_spanned! { field.span() => + #with_field_id.add_metadata([( + delta_kernel::schema::ColumnMetadataKey::ColumnMappingNestedIds.as_ref().to_string(), + delta_kernel::schema::MetadataValue::Other( + ::serde_json::json!({ #nested_key: #elem_id }) + ) + )]) + } + } else { + with_field_id } - _ => Error::new(field.span(), format!("Can't handle type: {:?}", field.ty)).to_compile_error() } - }); - quote! { #(#schema_fields),* } + _ => Error::new(field.span(), format!("Can't handle type: {:?}", field.ty)) + .to_compile_error(), + } +} + +fn has_skip_schema(field: &Field) -> bool { + has_named_attr(&field.attrs, "skip_schema") +} + +/// Schema-facing fields of a named struct, in declaration order, with `#[skip_schema]` dropped. +/// +/// Shared by all derives that walk a struct's fields, so they cannot disagree on which fields +/// participate (or in what order). +fn schema_fields<'a>( + data: &'a Data, + derive_name: &str, + span: Span, +) -> Result, Error> { + Ok(partition_fields(data, derive_name, span)?.0) +} + +/// Splits a named struct's fields into schema-facing fields and `#[skip_schema]` fields, each in +/// declaration order. Derives that must also account for skipped fields use this directly. +fn partition_fields<'a>( + data: &'a Data, + derive_name: &str, + span: Span, +) -> Result<(Vec<&'a Field>, Vec<&'a Field>), Error> { + let Data::Struct(DataStruct { + fields: Fields::Named(fields), + .. + }) = data + else { + return Err(Error::new( + span, + format!("{derive_name} can only be derived for structs with named fields"), + )); + }; + Ok(fields.named.iter().partition(|f| !has_skip_schema(f))) +} + +fn try_from_struct_data_fields<'a>( + data: &'a Data, + derive_name: &str, + span: Span, +) -> Result, Error> { + let (fields, skipped) = partition_fields(data, derive_name, span)?; + if let Some(field) = skipped.first() { + return Err(Error::new( + field.span(), + "TryFromStructData does not support #[skip_schema] fields", + )); + } + Ok(fields) +} + +fn gen_schema_fields(data: &Data, span: Span) -> Result { + let fields = schema_fields(data, "ToSchema", span)?; + let fields = fields.iter().map(|f| gen_schema_field(f)); + Ok(quote! { #(#fields),* }) } /// Derive an IntoEngineData trait for a struct that has all fields implement `TryInto`. @@ -148,22 +417,12 @@ pub fn into_engine_data_derive(input: proc_macro::TokenStream) -> proc_macro::To let input = parse_macro_input!(input as DeriveInput); let struct_name = &input.ident; - let Data::Struct(DataStruct { - fields: Fields::Named(fields), - .. - }) = &input.data - else { - return Error::new( - struct_name.span(), - "IntoEngineData can only be derived for structs with named fields", - ) - .to_compile_error() - .into(); + let fields = match schema_fields(&input.data, "IntoEngineData", struct_name.span()) { + Ok(fields) => fields, + Err(e) => return e.to_compile_error().into(), }; - - let fields = &fields.named; - let field_idents = fields.iter().map(|f| &f.ident); - let field_types: Vec<_> = fields.iter().map(|f| &f.ty).collect(); + let (field_idents, field_types): (Vec<_>, Vec<_>) = + fields.into_iter().map(|f| (&f.ident, &f.ty)).unzip(); let expanded = quote! { #[automatically_derived] @@ -191,6 +450,138 @@ pub fn into_engine_data_derive(input: proc_macro::TokenStream) -> proc_macro::To proc_macro::TokenStream::from(expanded) } +/// Derive `From` conversions into `StructData` and `Scalar` for a rust struct. +/// +/// Emits both: +/// - `From for StructData` — field values via `.into()`, schema from `ToSchema` +/// - `From for Scalar` — `Scalar::Struct(self.into())` +/// +/// Honors `#[skip_schema]`, producing the same field set as `ToSchema`. Every schema field type +/// must implement `Into` (and the struct must implement `ToSchema`). +#[proc_macro_derive(IntoStructData)] +pub fn into_struct_data_derive(input: proc_macro::TokenStream) -> proc_macro::TokenStream { + let input = parse_macro_input!(input as DeriveInput); + let struct_name = &input.ident; + + let fields = match schema_fields(&input.data, "IntoStructData", struct_name.span()) { + Ok(fields) => fields, + Err(e) => return e.to_compile_error().into(), + }; + let (field_idents, field_types): (Vec<_>, Vec<_>) = + fields.into_iter().map(|f| (&f.ident, &f.ty)).unzip(); + + let expanded = quote! { + #[automatically_derived] + impl From<#struct_name> for delta_kernel::expressions::StructData + where + #struct_name: delta_kernel::schema::ToSchema, + #(#field_types: Into,)* + { + fn from(value: #struct_name) -> Self { + Self::from_values_unchecked( + <#struct_name as delta_kernel::schema::ToSchema>::to_schema(), + vec![ #(value.#field_idents.into()),* ], + ) + } + } + + #[automatically_derived] + impl From<#struct_name> for delta_kernel::expressions::Scalar + where + #struct_name: Into, + { + fn from(value: #struct_name) -> Self { + Self::Struct(value.into()) + } + } + }; + + proc_macro::TokenStream::from(expanded) +} + +/// Derive the inverse of [`IntoStructData`](macro@IntoStructData): `TryFrom` conversions from +/// `StructData` and `Scalar` for a rust struct. +/// +/// Emits both: +/// - `TryFrom for Self` — values matched by schema field name and converted via each +/// field's `TryFrom` +/// - `TryFrom for Self` — unwraps `Scalar::Struct`, else errors +/// +/// Missing, duplicate, and unknown fields are errors. Every field type must implement +/// `TryFrom`. `#[skip_schema]` is rejected because the reverse conversion +/// cannot infer a value or schema for a field omitted by `ToSchema`. +#[proc_macro_derive(TryFromStructData, attributes(skip_schema))] +pub fn try_from_struct_data_derive(input: proc_macro::TokenStream) -> proc_macro::TokenStream { + let input = parse_macro_input!(input as DeriveInput); + try_from_struct_data_impl(&input) + .unwrap_or_else(Error::into_compile_error) + .into() +} + +fn try_from_struct_data_impl(input: &DeriveInput) -> Result { + let struct_name = &input.ident; + let fields = try_from_struct_data_fields(&input.data, "TryFromStructData", struct_name.span())?; + let (field_idents, field_types): (Vec<_>, Vec<_>) = + fields.into_iter().map(|f| (&f.ident, &f.ty)).unzip(); + let schema_field_names: Vec<_> = field_idents + .iter() + .map(|ident| ident.as_ref().map(get_schema_name)) + .collect::>() + .ok_or_else(|| { + Error::new( + struct_name.span(), + "TryFromStructData can only be derived for structs with named fields", + ) + })?; + + Ok(quote! { + #[automatically_derived] + impl TryFrom for #struct_name + where + #struct_name: delta_kernel::schema::ToSchema, + #(#field_types: + TryFrom,)* + { + type Error = delta_kernel::Error; + + fn try_from( + value: delta_kernel::expressions::StructData, + ) -> delta_kernel::DeltaResult { + let mut fields = + delta_kernel::schema::derive_macro_utils::StructDataFields::try_new( + value, + <#struct_name as delta_kernel::schema::ToSchema>::to_schema(), + )?; + let result = Self { + #(#field_idents: fields.take_field(stringify!(#schema_field_names))?,)* + }; + fields.finish()?; + Ok(result) + } + } + + #[automatically_derived] + impl TryFrom for #struct_name + where + #struct_name: TryFrom< + delta_kernel::expressions::StructData, + Error = delta_kernel::Error, + >, + { + type Error = delta_kernel::Error; + + fn try_from( + value: delta_kernel::expressions::Scalar, + ) -> delta_kernel::DeltaResult { + match value { + delta_kernel::expressions::Scalar::Struct(data) => data.try_into(), + other => Err(other.conversion_error(stringify!(#struct_name))), + } + } + } + }) +} + /// Mark items as `internal_api` to make them public iff the `internal-api` feature is enabled. /// /// NOTE: This macro does not support `mod` declarations because of nuances in how the mod expander @@ -268,3 +659,175 @@ fn make_public(mut item: Item) -> Item { item } + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + + /// Expand `gen_schema_fields` for `input` and return the generated tokens as a string. Macro + /// errors are embedded as `compile_error!` tokens in that string; `Err` only signals that the + /// input itself failed to parse as a `DeriveInput`. + fn schema_fields_tokens(input: &str) -> Result { + let input = syn::parse_str::(input).map_err(|e| e.to_string())?; + let tokens = gen_schema_fields(&input.data, input.ident.span()) + .unwrap_or_else(|e| e.to_compile_error()); + Ok(tokens.to_string()) + } + + #[test] + fn test_valid_field_id_parsing() { + let input = r#" + struct TestStruct { + #[field_id = 123] + valid_field: String, + + #[field_id = 456] + another_valid_field: i32, + + normal_field: bool, + } + "#; + + let tokens = schema_fields_tokens(input).unwrap(); + assert!(tokens.contains("123")); + assert!(tokens.contains("456")); + assert!(!tokens.contains("compile_error")); + } + + #[rstest] + #[case::zero("0", "0i64")] + #[case::one("1", "1i64")] + #[case::max("2147483647", "2147483647i64")] + fn field_id_in_range_renders_literal(#[case] id: &str, #[case] expected: &str) { + let input = format!("struct S {{ #[field_id = {id}] f: String }}"); + let tokens = schema_fields_tokens(&input).unwrap(); + let needle = format!( + "(delta_kernel :: schema :: ColumnMetadataKey :: ParquetFieldId . as_ref () , {expected})" + ); + assert!(tokens.contains(&needle), "found: {tokens}"); + } + + #[rstest] + #[case::negative("#[field_id = -1]")] + #[case::overflow_i32("#[field_id = 2147483648]")] + #[case::overflow_i64("#[field_id = 9223372036854775808]")] + #[case::not_an_integer(r#"#[field_id = "x"]"#)] + #[case::bare_path("#[field_id]")] + #[case::list_form("#[field_id(1)]")] + fn field_id_rejected(#[case] attr: &str) { + let input = format!("struct S {{ {attr} f: String }}"); + let tokens = schema_fields_tokens(&input).unwrap(); + assert!( + tokens.contains("compile_error"), + "expected rejection, found: {tokens}" + ); + } + + #[rstest] + // With a companion field_id, both metadata keys are emitted. + #[case::with_field_id("#[field_id = 132]\n#[nested_field_id = 133]", true)] + // Element id alone emits only the nested id, not ParquetFieldId. + #[case::without_field_id("#[nested_field_id = 133]", false)] + fn nested_field_id_on_list(#[case] attrs: &str, #[case] expect_parquet_field_id: bool) { + let input = format!("struct S {{ {attrs}\nsplit_offsets: Option> }}"); + let tokens = schema_fields_tokens(&input).unwrap(); + assert!(!tokens.contains("compile_error"), "found: {tokens}"); + assert!(tokens.contains("ColumnMappingNestedIds"), "found: {tokens}"); + assert!(tokens.contains("splitOffsets.element"), "found: {tokens}"); + assert!(tokens.contains("133i64"), "found: {tokens}"); + assert_eq!( + tokens.contains("ParquetFieldId . as_ref"), + expect_parquet_field_id, + "found: {tokens}" + ); + } + + #[rstest] + #[case::scalar("f: String")] + #[case::map("f: std::collections::HashMap")] + fn nested_field_id_rejected_on_non_list(#[case] field: &str) { + let input = format!("struct S {{ #[nested_field_id = 5] {field} }}"); + let tokens = schema_fields_tokens(&input).unwrap(); + assert!( + tokens.contains("compile_error"), + "expected rejection, found: {tokens}" + ); + } + + #[test] + fn skip_schema_excludes_field() { + let input = r#" + struct TestStruct { + #[skip_schema] + skipped: String, + kept: i32, + } + "#; + let tokens = schema_fields_tokens(input).unwrap(); + assert!(tokens.contains("kept"), "kept field must remain: {tokens}"); + assert!( + !tokens.contains("skipped"), + "skipped field must be absent: {tokens}" + ); + } + + #[test] + fn try_from_struct_data_rejects_skipped_field() { + let input = syn::parse_str::( + r#" + struct TestStruct { + #[skip_schema] + skipped: String, + kept: i32, + } + "#, + ) + .unwrap(); + let error = + try_from_struct_data_fields(&input.data, "TryFromStructData", input.ident.span()) + .unwrap_err(); + assert_eq!( + error.to_string(), + "TryFromStructData does not support #[skip_schema] fields" + ); + } + + #[rstest] + #[case::hashmap("std::collections::HashMap")] + #[case::option_hashmap("Option>")] + fn allow_null_container_values_on_map(#[case] ty: &str) { + let input = format!("struct S {{ #[allow_null_container_values] f: {ty} }}"); + let tokens = schema_fields_tokens(&input).unwrap(); + assert!(!tokens.contains("compile_error"), "found: {tokens}"); + assert!( + tokens.contains("get_nullable_container_struct_field"), + "found: {tokens}" + ); + } + + #[rstest] + #[case::scalar("String")] + #[case::vec("Vec")] + fn allow_null_container_values_rejected_on_non_map(#[case] ty: &str) { + let input = format!("struct S {{ #[allow_null_container_values] f: {ty} }}"); + let tokens = schema_fields_tokens(&input).unwrap(); + assert!( + tokens.contains("compile_error"), + "expected rejection, found: {tokens}" + ); + } + + #[rstest] + #[case::tuple("f: (i32, i32)")] + #[case::array("f: [i32; 4]")] + fn non_path_field_type_rejected(#[case] field: &str) { + let input = format!("struct S {{ {field} }}"); + let tokens = schema_fields_tokens(&input).unwrap(); + assert!( + tokens.contains("compile_error"), + "expected rejection, found: {tokens}" + ); + } +} diff --git a/derive-macros/src/schema_macro.rs b/derive-macros/src/schema_macro.rs new file mode 100644 index 0000000000..a65281bb0f --- /dev/null +++ b/derive-macros/src/schema_macro.rs @@ -0,0 +1,403 @@ +//! Parser and code generator for the `schema!` / `try_schema!` / `schema_ref!` function-like +//! macros. The grammar mirrors a Delta schema (the protocol `schemaString`) while letting the +//! caller interpolate arbitrary runtime values, in the spirit of `serde_json::json!`. See the +//! public macro docs in `delta_kernel::schema` for the full user-facing description. +//! +//! # Grammar +//! +//! ```text +//! body := (entry ',')* entry? // 0+ comma-separated entries, optional trailing comma +//! entry := nullability name ':' type // possibly nullable struct field +//! | '(' EXPR ')' // interpolate one StructField +//! | '..' '(' EXPR ')' // splice an `impl IntoIterator` +//! nullability := 'nullable' | 'not_null' +//! name := STR_LITERAL +//! | IDENT +//! | '(' EXPR ')' // interpolate an `impl Into` +//! type := '[' nullability type ']' // array with possibly-nullable elements +//! | '{' body '}' // nested struct +//! | '{' type '=>' nullability type '}' // map with possibly-nullable values +//! | '(' EXPR ')' // interpolate an `impl Into` +//! | IDENT // interpolate `DataType::` +//! ``` +//! +//! # Design +//! +//! A recursive-descent parser that emits schema-building code directly as it walks the token +//! stream, with no intermediate AST. Each grammar production maps one-to-one onto a code fragment, +//! so each `emit_*` function parses its slice of the input and returns the `TokenStream` for the +//! corresponding expression; children are recursively emitted and spliced into their parent's +//! `quote!`, producing nested expressions. The one cross-cutting concern, duplicate-name +//! detection, is per-struct-body and folds into the body parser. +//! +//! `syn` serves purely as the tokenizer/cursor (`ParseStream`, `peek`, delimiter macros, and +//! `Expr`/`LitStr`/`Ident` sub-parsers). The entry point can be a simple closure because +//! `FnOnce(ParseStream) -> syn::Result` satisfies `syn::parse::Parser`. + +use std::collections::hash_map::Entry as MapEntry; +use std::collections::HashMap; + +use proc_macro2::{Span, TokenStream}; +use quote::{quote, quote_spanned, ToTokens}; +use syn::parse::{ParseStream, Parser}; +use syn::spanned::Spanned; +use syn::token::{Brace, Bracket, Paren}; +use syn::{braced, bracketed, parenthesized, Expr, Ident, LitStr, Token}; + +mod kw { + syn::custom_keyword!(nullable); + syn::custom_keyword!(not_null); +} + +/// Parses the macro body and emits the struct-building code, rejecting statically-detectable +/// duplicate field names. +/// - `fallible`: build nested structs with `try_new` and `?`-propagate, vs. `new_unchecked`. +/// - `wrap`: post-processes the resulting top-level `StructType` (e.g. `Arc::new`, error closure). +/// +/// Duplicate-name errors accumulate in `errors` while parsing continues, and +/// are emitted only if parsing otherwise succeeds. A fatal parse error short-circuits via `?` and +/// is reported on its own (suppressing any duplicates seen to that point). +pub(crate) fn parse_schema( + input: proc_macro::TokenStream, + fallible: bool, + wrap: impl FnOnce(TokenStream) -> TokenStream, +) -> proc_macro::TokenStream { + // Emits a block of 1+ errors, in case parsing failed or duplicate fields were detected. + let compile_errors_block = |errors: Vec| { + let invocations = errors.iter().map(syn::Error::to_compile_error); + quote!({ #( #invocations )* }) + }; + try_parse_schema(input.into(), fallible) + .map_or_else(compile_errors_block, wrap) + .into() +} + +/// Runs the parser over a [`proc_macro2::TokenStream`], returning the schema-building tokens or the +/// accumulated errors. Splitting this out from [`parse_schema`] separates the wrapping of the +/// `compile_error!` block from its rendering, and lets unit tests drive the parser directly. +fn try_parse_schema(input: TokenStream, fallible: bool) -> Result> { + let mut errors: Vec = Vec::new(); + let parser = |input: ParseStream| emit_struct(input, fallible, &mut errors); + match parser.parse2(input) { + Ok(block) if errors.is_empty() => Ok(block), + Ok(_) => Err(errors), + Err(parse_error) => Err(vec![parse_error]), + } +} + +/// `body := (entry ',')* entry?` +/// +/// Emits code to create a `StructType` instance, by building up a vec of `StructField`s and +/// recording any duplicate field names in `errors`. The `fallible` flag determines whether +/// `StructType::try_new` or `StructType::new_unchecked` is used. +fn emit_struct( + input: ParseStream, + fallible: bool, + errors: &mut Vec, +) -> syn::Result { + // Track and reject duplicate literal field names and duplicate identifiers + let mut literals = HashMap::new(); + let mut idents = HashMap::new(); + let mut stmts = Vec::new(); + while !input.is_empty() { + let entry = emit_field_entry(input, fallible, &mut literals, &mut idents, errors); + stmts.push(entry?); + if input.is_empty() { + break; + } + input.parse::()?; + } + let ctor = if fallible { + quote!(try_new) + } else { + quote!(new_unchecked) + }; + Ok(quote! {{ + let mut __fields = ::std::vec::Vec::new(); + #( #stmts )* + delta_kernel::schema::StructType::#ctor(__fields) + }}) +} + +/// `entry := nullability name ':' type | '(' EXPR ')' | '..' '(' EXPR ')'` +/// +/// Emits one `__fields.push(...)` or `__fields.extend(...)` statement. +fn emit_field_entry( + input: ParseStream, + fallible: bool, + literals: &mut HashMap, + idents: &mut HashMap, + errors: &mut Vec, +) -> syn::Result { + // `..(xs)` -- splice an iterator of fields. + if input.peek(Token![..]) { + input.parse::()?; + let expr: Expr = parse_paren_to_end(input, "splice expression")?; + return Ok(quote_spanned! { expr.span() => + __fields.extend( + ::core::iter::IntoIterator::into_iter(#expr) + .map(delta_kernel::schema::ToSchemaField::to_schema_field) + ); + }); + } + // `(field)` -- interpolate one `StructField`. A named field always starts with a nullability + // keyword, so a leading `(` is unambiguous. + if input.peek(Paren) { + let expr: Expr = parse_paren_to_end(input, "field expression")?; + return Ok(quote_spanned! { expr.span() => + __fields.push(delta_kernel::schema::ToSchemaField::to_schema_field(#expr)); + }); + } + // `nullability NAME : type` + let nullable = parse_nullability(input)?; + let name = emit_field_name(input, literals, idents, errors)?; + input.parse::()?; + let dtype = emit_type(input, fallible, errors)?; + Ok(quote! { + __fields.push(delta_kernel::schema::StructField::new(#name, #dtype, #nullable)); + }) +} + +/// `name := STR_LITERAL | IDENT | '(' EXPR ')'` +/// +/// Returns the name tokens (all three forms evaluate to `impl Into`) and records the name +/// for duplicate detection. String literals are knowable by value (recorded case-insensitively, per +/// Delta's case-insensitive column rule); a bare ident is opaque but textually comparable; a +/// `(EXPR)` name is fully opaque and left to runtime validation. +fn emit_field_name( + input: ParseStream, + literals: &mut HashMap, + idents: &mut HashMap, + errors: &mut Vec, +) -> syn::Result { + if input.peek(LitStr) { + let lit: LitStr = input.parse()?; + check_for_duplicates(literals, lit.value().to_lowercase(), lit.span(), errors); + Ok(lit.to_token_stream()) + } else if input.peek(Paren) { + let expr: Expr = parse_paren_to_end(input, "field name expression")?; + Ok(expr.to_token_stream()) + } else { + let ident: Ident = input.parse()?; + check_for_duplicates(idents, ident.to_string(), ident.span(), errors); + Ok(ident.to_token_stream()) + } +} + +/// Emits a `DataType`-typed expression for the next `type` in the stream. `fallible` recursively +/// propagates the requesting macro's validation mode: when set, nested structs use `try_new` and +/// their results are `?`-propagated to the enclosing `DeltaResult` closure. +fn emit_type( + input: ParseStream, + fallible: bool, + errors: &mut Vec, +) -> syn::Result { + // `(EXPR)` -- anything that evaluates to `impl Into` + if input.peek(Paren) { + let expr: Expr = parse_paren_to_end(input, "parenthesized type")?; + return Ok(expr.to_token_stream()); + } + // `[ nullability element ]` -- array. + if input.peek(Bracket) { + let content; + bracketed!(content in input); + let contains_null = parse_nullability(&content)?; + let element = emit_type(&content, fallible, errors)?; + ensure_empty(&content, "array element type")?; + return Ok(quote! { delta_kernel::schema::ArrayType::new(#element, #contains_null) }); + } + // `{ ... }` -- struct or map. A map is `KEY => VALUE`; the key is a single token tree, so a + // `=>` in the second position unambiguously marks a map. Anything else (including empty) is a + // struct body. + if input.peek(Brace) { + let content; + braced!(content in input); + if !content.peek2(Token![=>]) { + let block = emit_struct(&content, fallible, errors)?; + return Ok(if fallible { quote!(#block?) } else { block }); + } + let key = emit_type(&content, fallible, errors)?; + content.parse::]>()?; + let value_contains_null = parse_nullability(&content)?; + let value = emit_type(&content, fallible, errors)?; + ensure_empty(&content, "map value type")?; + return Ok( + quote! { delta_kernel::schema::MapType::new(#key, #value, #value_contains_null) }, + ); + } + // Bare, single-segment path/call -> `DataType::` (e.g. `LONG`, `unshredded_variant()`). + let expr: Expr = input.parse()?; + validate_bare_datatype(&expr)?; + Ok(quote_spanned! { expr.span() => delta_kernel::schema::DataType::#expr }) +} + +/// `nullability := 'nullable' | 'not_null'`, returning `contains_null`-style booleans. +fn parse_nullability(input: ParseStream) -> syn::Result { + if input.peek(kw::nullable) { + input.parse::()?; + Ok(true) + } else if input.peek(kw::not_null) { + input.parse::()?; + Ok(false) + } else { + Err(input.error("expected `nullable` or `not_null`")) + } +} + +/// Validates that a bare type is a single-segment path or call (e.g. `LONG` -> `DataType::LONG`), +/// rejecting qualified paths (e.g. `a::b`) and non-path expressions. +fn validate_bare_datatype(expr: &Expr) -> syn::Result<()> { + let path = match expr { + Expr::Path(path) if path.qself.is_none() => &path.path, + Expr::Call(call) => match &*call.func { + Expr::Path(path) if path.qself.is_none() => &path.path, + other => return Err(syn::Error::new_spanned(other, "expected a data type")), + }, + other => return Err(syn::Error::new_spanned(other, "expected a data type")), + }; + // A turbofish (`foo::`) stays within a single segment, so segment count alone distinguishes + // it from a qualified path (`a::b`). + if path.leading_colon.is_some() || path.segments.len() != 1 { + return Err(syn::Error::new_spanned(path, "unexpected qualified path")); + } + Ok(()) +} + +/// Parses a parenthesized `( T )`, requiring `T` to be its sole remaining content. +fn parse_paren_to_end(input: ParseStream, what: &str) -> syn::Result { + let content; + parenthesized!(content in input); + let value = content.parse()?; + ensure_empty(&content, what)?; + Ok(value) +} + +/// Errors out (pointing at the trailing tokens) if `content` is not fully consumed. +fn ensure_empty(content: ParseStream, what: &str) -> syn::Result<()> { + if content.is_empty() { + Ok(()) + } else { + Err(content.error(format!("unexpected tokens after {what}"))) + } +} + +/// Records a name occurrence in the given namespace. When the same key reappears in this body, +/// pushes two errors -- one at the duplicate and one at the original occurrence -- which surface as +/// a `compile_error!` pointing at each. +fn check_for_duplicates( + seen: &mut HashMap, + key: String, + span: Span, + errors: &mut Vec, +) { + match seen.entry(key) { + MapEntry::Occupied(first) => { + errors.push(syn::Error::new(span, "duplicate field name")); + errors.push(syn::Error::new(*first.get(), "first defined here")); + } + MapEntry::Vacant(slot) => { + slot.insert(span); + } + } +} + +#[cfg(test)] +mod tests { + // These tests drive `try_parse_schema` directly (the same code path the proc macros run), + // asserting only on the parse outcome and on the *caller-facing* diagnostics -- the compile + // errors a user sees when they mistype an invocation. They deliberately avoid inspecting the + // generated token stream, since its exact shape is an implementation detail. + use rstest::rstest; + + use super::*; + + /// Malformed invocations that must be rejected with a specific, caller-facing diagnostic. + #[rstest] + #[case::missing_top_level_nullability(quote! { "a": INTEGER }, "nullable")] + #[case::missing_array_element_nullability(quote! { nullable "a": [ STRING ] }, "nullable")] + #[case::missing_map_value_nullability(quote! { nullable "a": { STRING => STRING } }, "nullable")] + #[case::qualified_path_type(quote! { nullable "a": a::b }, "qualified path")] + #[case::non_datatype_bare_type(quote! { nullable "a": 42 }, "data type")] + #[case::trailing_tokens_in_parenthesized_type(quote! { nullable "a": (foo bar) }, "unexpected tokens")] + #[case::trailing_tokens_in_array_element(quote! { nullable "a": [ nullable STRING extra ] }, "unexpected tokens")] + #[case::trailing_tokens_in_map_value(quote! { nullable "a": { STRING => nullable STRING extra } }, "unexpected tokens")] + fn rejects_with_diagnostic(#[case] input: TokenStream, #[case] needle: &str) { + let errors = try_parse_schema(input, false).expect_err("expected rejection"); + let messages = Vec::from_iter(errors.iter().map(ToString::to_string)); + assert!( + messages.iter().any(|m| m.contains(needle)), + "no error contained {needle:?}; got: {messages:?}", + ); + } + + /// Structurally broken invocations that `syn` rejects for us. We assert only that they fail, + /// not on the wording, to avoid coupling the tests to `syn`'s error messages. + #[rstest] + #[case::missing_colon(quote! { nullable "a" INTEGER })] + #[case::missing_comma_between_fields(quote! { not_null "a": INTEGER not_null "b": LONG })] + fn rejects_structurally_invalid_input(#[case] input: TokenStream) { + assert!(try_parse_schema(input, false).is_err()); + } + + /// Statically-detectable duplicate field names are rejected, with errors pointing at both the + /// duplicate and the original occurrence. Literals collide case-insensitively (per Delta's + /// column-name rule); bare identifiers collide by their text. + #[rstest] + #[case::case_insensitive_string_literals(quote! { nullable "id": INTEGER, nullable "ID": STRING })] + #[case::repeated_identifiers(quote! { nullable NAME: INTEGER, not_null NAME: STRING })] + fn rejects_duplicate_field_names(#[case] input: TokenStream) { + let errors = try_parse_schema(input, false).expect_err("expected duplicate rejection"); + let messages = Vec::from_iter(errors.iter().map(ToString::to_string)); + assert!( + messages.iter().any(|m| m.contains("duplicate field name")), + "got: {messages:?}" + ); + assert!( + messages.iter().any(|m| m.contains("first defined here")), + "got: {messages:?}" + ); + } + + /// Well-formed invocations parse cleanly in both the infallible and fallible modes (the grammar + /// is identical; only the generated constructor differs). The last two cases pin the boundary + /// of static duplicate detection: it is per-struct-body, and interpolated `(EXPR)` names are + /// opaque, so neither is flagged here (the fallible macro defers such checks to runtime). + #[rstest] + #[case::flat_fields(quote! { + not_null "id": LONG, + nullable "name": STRING, + not_null "score": (DataType::DOUBLE), + })] + #[case::nested_struct_array_and_map(quote! { + not_null "user": { nullable "city": STRING, nullable "zip": STRING }, + nullable "tags": [ not_null STRING ], + not_null "props": { STRING => nullable STRING }, + })] + #[case::interpolated_name_type_field_and_splice(quote! { + not_null (format!("col_{}", 1)): (DataType::LONG), + (StructField::nullable("z", DataType::STRING)), + ..(::std::vec::Vec::::new()), + })] + #[case::empty_body(quote! {})] + #[case::same_name_in_sibling_nested_structs(quote! { + not_null "a": { nullable "x": INTEGER }, + not_null "b": { nullable "x": INTEGER }, + })] + #[case::interpolated_name_collision_deferred(quote! { + nullable (n): INTEGER, + nullable (n): STRING, + })] + fn accepts_well_formed_schema( + #[case] input: TokenStream, + #[values(false, true)] fallible: bool, + ) { + let result = try_parse_schema(input, fallible); + assert!( + result.is_ok(), + "expected acceptance, got: {:?}", + result + .err() + .map(|e| Vec::from_iter(e.iter().map(ToString::to_string))), + ); + } +} diff --git a/doc/roadmap.md b/doc/roadmap.md deleted file mode 100644 index 4200270158..0000000000 --- a/doc/roadmap.md +++ /dev/null @@ -1,33 +0,0 @@ -# Roadmap - -This document represents a draft of a potential roadmap for delta-kernel-rs - -The whole premise of delta-kernel is to address gaps identified in the -proliferation of APIs and custom implementations of the Delta protocol. This -should learn from those and provide a simpler or easier way for _all_ those -integrations to make following Delta Lake easier. - - -## Data and AI Summit - -Goals: - -* Get a topic branch in `delta-rs` functional with a minimal use-case, - replacing some key functionality. - * (?) Perhaps a CDC or Deletion vector use-case, to showcase the benefit. - Denny is more interested in psuedo-code level -* Defined feature strategy for library vs user API surfaces - * Singular user API implemented, perhaps arrow2 to demonstrate the - hot-swappability versus strict delta-rs -* Basic scaffolding of traits defined with issues created in repository for - contributors to participate -* Strong CI pipeline for getting new contributors on-board. -* **EVERYTHING** that exists gratuitously documented in rustdoc with doc tests -* Show how using delta-kernel-rs would be simpler for an API implementor - * Selling this idea/API to future delta integrations - -Demo ideas (order of priority): - -* wasm in the browser for delta integration. -* Bootstrap simple Ruby client -* Bootstrap simple Node client diff --git a/docs/user-guide/.gitignore b/docs/user-guide/.gitignore new file mode 100644 index 0000000000..c48bfcd032 --- /dev/null +++ b/docs/user-guide/.gitignore @@ -0,0 +1,2 @@ +# mdbook build output +book diff --git a/docs/user-guide/CLAUDE.md b/docs/user-guide/CLAUDE.md new file mode 100644 index 0000000000..2fced13e2b --- /dev/null +++ b/docs/user-guide/CLAUDE.md @@ -0,0 +1,435 @@ +# Delta Kernel User Guide: Writing Standards + +This file governs the mdBook user guide. Every page is a Markdown file in +`src/`. The audience is Rust developers building Delta Lake connectors. Assume +Rust competence (ownership, traits, async, error handling) but NOT Delta Lake +expertise. Never explain Rust concepts. Always define Delta concepts +(transaction log, checkpoint, action types, etc.) on first use. + +## Quick reference + +``` +Voice: second person, present tense, active, contractions OK +Headings: sentence case (exception: type names like "The Engine trait") +Dashes: never use "--" in prose; restructure into two sentences +Code: MUST compile via mdbook test +Output: MUST show expected output after code that produces it +Errors: prefer ? over .unwrap(); hidden boilerplate returns DeltaResult +Terms: Snapshot, Scan, Transaction, Engine, connector, predicate +Dataset: Alice/30/Seattle, Bob/25/Portland, Carol/35/Denver +Links: relative paths, link first occurrence per section only +Scope: teach Kernel API usage, not Delta protocol internals +``` + +## Build and test commands + +Run from `docs/user-guide/`: + +```sh +mdbook serve # local hot-reload preview at localhost:3000 +make test # compile every Rust code block in the guide +mdbook build # render to HTML; also runs linkcheck if mdbook-linkcheck2 is installed +``` + +First-time setup (matches CI): + +```sh +cargo install --locked mdbook mdbook-mermaid mdbook-linkcheck2 +``` + +CI runs `make test` and `mdbook build` (with linkcheck) on every push, PR, +and merge group event. See `.github/workflows/user-guide.yml`. + +## Purpose and scope + +This guide teaches developers **how to use Kernel to build a connector**. It is +NOT a Delta protocol reference. The whole point of Kernel is that connector +authors do not need to understand protocol internals. + +- Explain Kernel's APIs, types, and patterns. Show how to call them correctly. +- Mention protocol concepts (log replay, data skipping, deletion vectors) only + to the degree needed to understand why a Kernel API exists or behaves a + certain way. +- Never deep-dive into protocol mechanics (action reconciliation rules, commit + JSON schema, checkpoint format internals, timestamp ordering semantics). If + the reader needs that level of detail, link to the + [Delta protocol spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md). + +### Protocol depth framework + +Use this three-tier system to decide how much protocol context to include: + +| Depth | When to use | Example | +|-------|-------------|---------| +| **Zero mention** | The protocol detail has no API-surface consequence | Checkpoint Parquet format internals, action reconciliation ordering rules | +| **One sentence** | The reader needs to know WHY an API exists or behaves a certain way | "Delta tables use deletion vectors to mark rows as deleted without rewriting files, which is why `DvInfo` exists on `ScanFile`." | +| **Short paragraph** (3-5 sentences) | The reader must make a decision or write code differently because of protocol behavior | The best-effort nature of data skipping, where the reader needs to decide whether to apply row-level filtering after the scan | + +**Concrete example of too deep vs. right amount:** + +Too deep: +> "During log replay, Kernel reads all JSON commit files and checkpoint Parquet +> files, then reconciles Add and Remove actions using the tombstone-retention +> rule: a Remove action cancels a matching Add if the Remove's timestamp is +> greater than or equal to the Add's timestamp within the same checkpoint +> interval." + +Right amount: +> "Kernel replays the Delta log to determine which files are currently active. +> You don't need to understand the replay rules. The `Scan` gives you the +> final file list." + +**The decision test:** does the connector author need to write code differently +because of this detail? If yes, explain enough for them to write correct code. +If no, one sentence of context at most. + +**Exception zones:** Catalog integration pages and custom Engine implementation +pages necessarily carry more protocol context than the reading/writing how-to +pages. The commit type taxonomy (staged, ratified, published) is required +vocabulary for catalog committer authors. Apply the decision test per-paragraph +rather than per-page for these sections. + +## Page structure + +Every page, regardless of type, MUST follow this skeleton: + +1. **H1 title** that names the concept or task. +2. **Opening sentence** appropriate to the page type (see opener patterns in + the Page types table below). Never start with "This page covers X." For + how-to pages, lead with the reader's goal. For explanation pages, lead with + a definition and why it matters. +3. **Prerequisite/context link** if the page assumes knowledge from another + page: "Before reading this page, make sure you understand + [Building a Scan](../reading/building_a_scan.md)." +4. **Body** using progressive disclosure: simplest case first, then advanced + options. Each section should motivate WHY before explaining HOW. +5. **"What's next"** section at the bottom with 2-3 links to logical next + pages. For sequential navigation (the natural reading order). +6. **"See also"** section (optional) for lateral cross-references to related + but non-sequential pages. Useful on reference and explanation pages. + +## Page types + +This guide uses the Diataxis model. There are four types of documentation. Each +demands a different voice, structure, and opener. MUST NOT mix types in one page. + +| Type | Purpose | Voice | Opener pattern | Our pages | +|------|---------|-------|----------------|-----------| +| **Tutorial** | Learning by doing | "Let's build..." | "In this tutorial, you will [outcome]." | Quick starts | +| **How-to** | Accomplish a task | "To do X, do Y" | "To [goal], you [action]." | Building a scan, Appending data, Filter pushdown | +| **Reference** | Look up exact details | Neutral, structured | "[Type/Trait] [does what]." One-sentence definition. | Engine trait, Schema and types, Implementing engine | +| **Explanation** | Understand why | Discursive, second person OK | "[Concept] is [definition]. This matters because [why]." | Architecture, Catalog overview | + +### How to decide which type a page is + +- Is the reader doing something for the first time with hand-holding? -> Tutorial +- Is the reader trying to accomplish a specific task and already knows the basics? -> How-to +- Is the reader looking up a specific API, type, trait, or method signature? -> Reference +- Is the reader trying to understand WHY something works the way it does, + or how pieces fit together at a high level? -> Explanation +- If none of the above fit, default to How-to (task-oriented). + +If a page feels like it needs to be two types (e.g., `implementing_engine.md` +is both "how to implement the Engine trait" and "reference for all handler +signatures"), split it into two pages or pick the dominant type and link to +the other. For example, a how-to page can link to a reference page for the +full trait signatures rather than inlining them all. + +**Page classifications:** + +| Page | Type | Rationale | +|------|------|-----------| +| `quick_start_read.md`, `quick_start_write.md` | Tutorial | Step-by-step, one path, builds a working thing | +| `installation.md`, `feature_flags.md` | Reference | Look up dependencies, flags, and their effects | +| `building_a_scan.md`, `column_selection.md`, `filter_pushdown.md` | How-to | "To read data, configure a scan like this" | +| `append.md`, `create_table.md`, `idempotent_writes.md` | How-to | "To write data, follow these steps" | +| `scan_metadata.md`, `parallel_scan_metadata.md` | How-to | "To distribute reads, use scan_metadata like this" | +| `architecture.md` | Explanation | Why Kernel is layered, how the pieces fit together | +| `engine_trait.md` (concepts) | Explanation | Why the Engine trait exists, what role it plays | +| `implementing_engine.md` | Reference | Trait signatures, key contracts, default implementations | +| `engine_data.md` | Reference | EngineData trait, visitor pattern, method contracts | +| `schema_and_types.md` | Reference | Type system listing, schema construction | +| `catalog_managed/overview.md` | Explanation | Why catalog-managed tables exist, how Kernel fits in | +| `catalog_managed/committer.md` | How-to | "To implement a catalog committer, do this" | +| `catalog_managed/reading.md`, `writing.md` | How-to | "To read/write catalog-managed tables, do this" | +| `connector/overview.md` | Explanation | What a connector is, Kernel vs. connector responsibilities | +| `configuring_storage.md` | How-to | "To configure S3/GCS/Azure storage, pass these options" | +| `checkpointing.md` | How-to | "To checkpoint a table, call these APIs" | +| `ffi/overview.md` | Explanation + Reference | Consider splitting if it grows | + +### Type-specific guidance + +**Tutorials:** Take the reader by the hand. One path, no choices. Every step +MUST produce a visible, verifiable result. Minimize explanation. Link to it +instead. It is OK to USE concepts (traits, `Result`, `Arc`) before they are +formally explained. This is the spiral curriculum pattern: hook the reader with +a working result, then circle back to explain the mechanics in later chapters. +Use explicit forward references: "We're using the default engine here. +[The Engine Trait](../concepts/engine_trait.md) explains how to customize it." + +**How-to pages:** Assume competence. Lead with the goal, not the page +description. Follow the recipe model: specific steps, specific outcome. + +**Reference pages:** Mirror the structure of the API, not the user's workflow. +Neutral voice is acceptable here (third person, impersonal constructions). An +exhaustive listing is expected. A brief "when you would use this" sentence +before trait signatures is still required to avoid the "wall of API" problem. + +**Explanation pages:** The opener rule relaxes here. "This section explains how +Kernel supports catalog-managed tables" is acceptable for explanation pages +because the reader's goal is understanding, not doing. Still lead with why it +matters, not what the page contains. Consider a "When NOT to use this" section +where applicable (e.g., "If you just need a DataFrame, use delta-rs directly. +Kernel is for building connectors that need protocol-level control."). + +## Voice and tone + +- MUST: second person ("you"), present tense, active voice. +- MUST: use contractions (don't, you'll, it's). Do not use slang. +- MUST NOT: use "we" to mean the project or the authors. "We" is acceptable + only in tutorial context ("Let's walk through this"). +- MUST NOT: use emojis, exclamation marks, or marketing superlatives + ("blazingly fast", "world-class", "cutting-edge"). +- SHOULD: sound like a knowledgeable colleague, not a textbook. +- For **reference pages**, neutral/impersonal voice is acceptable where it + reads more naturally (e.g., "The handler resolves columns by field ID" rather + than forcing "You resolve columns by field ID" when the handler does it). + +## Sentence-level rules + +MUST: +- One idea per sentence. +- Place conditions before instructions: "To compile the project, run + `cargo build`" not "Run `cargo build` to compile the project." +- When warning the reader, always explain the consequence, not only the rule. + +SHOULD: +- Eliminate "There is/are" constructions. Rewrite with a concrete subject. +- Replace weak verbs: "occurs" -> specific verb; "is able to" -> "can"; + "has the ability to" -> "can"; "in order to" -> "to". + +MUST NOT: +- Never say "simply", "just", "easy", or "obvious". +- Never use `--` (double dash) as an em dash in prose. Restructure into + two sentences, or use a period and start a new sentence. + +### Rewrites cheat sheet + +| Instead of... | Write... | +|-------------------------------------------|----------------------------------------| +| This page covers X | To accomplish X, you do Y *(how-to)* / X is Y. This matters because Z *(explanation)* | +| There is a method called `foo` that... | The `foo` method... | +| The table will be created by the engine | The engine creates the table | +| In order to read the table | To read the table | +| It is important to note that | *(just state the fact directly)* | +| Simply call `build()` | Call `build()` | +| The system is able to handle | The system handles | +| Run `cargo build` to compile | To compile, run `cargo build` | +| causes the triggering of | triggers | +| provides a detailed description of | describes | +| at this point in time | now | +| due to the fact that | because | +| in the event that | if | + +## Code examples + +### Code fence annotations + +- `` ```rust `` : compiles AND runs during `mdbook test`. Use for self-contained + examples that need no external resources. +- `` ```rust,no_run `` : compiles but does not run. Use for examples needing a + real table, network, or async runtime. +- `` ```rust,ignore `` : not compiled. Use for fragments shown in context of a + larger block above, or for trait/struct definitions from the Kernel API. +- `` ```rust,compile_fail `` : MUST fail to compile. Use for "break-then-fix" + teaching where you show a common mistake, display the compiler error, then + show the corrected version. +- `` ```rust,should_panic `` : compiles and runs, but MUST panic. Use for + examples demonstrating error conditions. +- `` ```toml ``, `` ```sh ``, `` ```text ``, `` ```json `` : non-Rust code. + +### Code example rules + +- MUST: every code example that can be made compilable MUST be compilable. + Wrap boilerplate (imports, engine setup, async runtime) in hidden `# ` lines + so the reader sees only the relevant code, but `mdbook test` still compiles. +- MUST: show expected output after every code block that produces output. Use a + `` ```text `` block. If the output is a table, show the ASCII table. +- SHOULD: use inline comments to narrate what each step does. Comments explain + intent ("why"), not syntax ("what"). +- SHOULD: number the steps in comments (`// 1. Parse the table`, + `// 2. Build engine`) for longer examples, matching the prose breakdown. +- For **error handling**, MUST use `?` over `.unwrap()`. Hidden boilerplate + should include a `main()` returning `DeltaResult<()>` (or + `Result<(), Box>` when mixing error types). Readers + copy examples verbatim. `.unwrap()` in examples teaches destructive habits. +- For **async examples**, include `tokio` in the hidden boilerplate + (`# #[tokio::main]` / `# async fn main()`). The reader should not need to + know about the async runtime to follow the example. +- For **external crate examples** (e.g., Unity Catalog crates in catalog + pages), use `rust,ignore` since the reader cannot compile them without the + external crate. Note the dependency explicitly in prose. See the root + `CLAUDE.md` for the canonical list of crates in this workspace. +- SHOULD: include experimentation prompts where natural. After an example, a + brief "Try changing `with_predicate(age < 30)` to `age > 30` and observe + which files are skipped" invites active learning. + +## Consistent example dataset + +Use the same Delta table throughout the guide wherever possible: + +```text +Schema: name (STRING), age (INTEGER), city (STRING) +Rows: ("Alice", 30, "Seattle"), ("Bob", 25, "Portland"), ("Carol", 35, "Denver") +``` + +**When to deviate:** +- Partitioned tables: add a partition column (e.g., `city`) +- Nested types: use a purpose-built schema showing the nesting +- Schema evolution: use a modified version of the canonical schema +- Reference pages listing types/traits: no dataset needed + +When deviating, always show the schema and a few sample rows. + +## Terminology + +Use these terms consistently. Do not alternate with synonyms. + +| Term | Not this | Notes | +|------|----------|-------| +| Snapshot | table state, table view | "version" is fine when referring to the numeric version (e.g., "table version 5"). Use "Snapshot" when referring to the object/type. | +| Scan | query, read operation | | +| Transaction | write operation | "commit" is fine when referring to the act of finalizing (e.g., "commit the transaction"). Do not use "commit" as a noun synonym for Transaction. | +| Engine | backend, provider, runtime | Refers to the Kernel `Engine` trait. | +| connector | integration, plugin, adapter | The code that glues a compute engine to Kernel. | +| predicate | | The Kernel API concept. "filter" is fine in user-facing context (e.g., "filter your data by age") but do not use "filter expression" as a synonym for `predicate` in API-level discussion. | +| data skipping | file pruning, file skipping | | +| EngineData | batch, record batch | "RecordBatch" is fine when specifically referring to the Arrow type. | + +Note: "compute engine" and "query engine" both refer to the external caller +(Spark, DataFusion, Flink, etc.), not to Kernel's `Engine` trait. Use +"compute engine" by default. "Query engine" is acceptable when discussing +query-specific behavior. + +Define Delta-specific terms (transaction log, checkpoint, action, partition, +deletion vector, etc.) on first use in each page. Rust-specific terms (trait, +`Arc`, lifetime, `?` operator) need no definition. + +## Formatting conventions + +- MUST: sentence case for headings ("Building a scan" not "Building a Scan"). + Exception: proper nouns and type names ("The Engine trait"). +- MUST: Oxford comma ("S3, GCS, and Azure"). +- MUST: backtick code font for identifiers: `Snapshot`, `scan_builder()`, + `cargo build`, file paths. +- MUST: `**bold**` for introducing a term for the first time in a definition. +- SHOULD: `_italic_` for emphasis within a sentence. +- Tables SHOULD have a header row and use consistent alignment. + +## Admonitions + +MUST use `> [!NOTE]`, `> [!WARNING]`, `> [!TIP]` for all callouts. Do not use +`> **Note:**` or other ad-hoc formats. + +Use sparingly. There is evidence readers skip elements outside their focus of +interest (Google developer style guide). Reserve for genuinely important info: + +- `> [!NOTE]`: supplementary information. Feature flag requirements go here. +- `> [!WARNING]`: ONLY for situations with real consequences (data loss, + silent incorrectness, hard-to-debug errors, irreversible actions). +- `> [!TIP]`: a non-obvious shortcut or best practice. + +## Cross-references + +- Link on first occurrence of a concept per section. Do not link every mention. +- Use relative paths: `[Building a Scan](../reading/building_a_scan.md)`. +- When deferring a concept, name the destination explicitly: "Deletion vectors + are covered in [Scan Metadata](./scan_metadata.md)." +- Use explicit forward references in tutorials: "We're using the default engine + here. [The Engine Trait](../concepts/engine_trait.md) explains how to + customize it." + +## Diagrams + +- Use ASCII art (`` ```text `` blocks) for architecture and flow diagrams. + ASCII art renders in all environments (terminal, browser, PDF) without + requiring image tooling. +- Use images in `src/images/` only when ASCII art cannot convey the concept + (e.g., screenshots, complex visual flows). All images MUST have alt text. +- Keep diagrams focused. If a diagram needs more than ~30 lines, consider + splitting it or simplifying. + +## Page length and scope + +- SHOULD: aim for 100-250 lines per page. Under 50 lines suggests the page is + too thin and should be merged into a parent page. Over 400 lines suggests + the page covers too many topics and should be split. +- Each page should cover one concept or one task. If you find yourself writing + multiple H2 sections that could each stand alone, consider splitting. +- Do not bleed per-operation details into pages that do not own them. If a + page needs to reference a concept (for example, which maintenance operations + Kernel supports, or whether log compaction is enabled), link to the page + that owns the topic rather than restating it. Ownership rules of thumb: + maintenance operations belong on the Maintenance Operations pages; protocol + details belong on the [Delta protocol spec](https://raw.githubusercontent.com/delta-io/delta/master/PROTOCOL.md) + or the page explicitly about that feature; connector-specific concerns + belong on the connector pages. + +## SUMMARY.md structure + +- Follow the reader's learning path: getting started first, then core concepts, + then reading, writing, advanced topics. Within each section, order by + increasing complexity. +- MUST NOT nest deeper than 2 levels (3 max in rare cases). Flat navigation is + easier to scan. +- Use `# Part Title` headings to separate major sections (e.g., + `# Getting Started`, `# Building Connectors`, `# Reference`). +- Use draft chapters (`[Future Topic]()`) for planned but unwritten content. + This is preferred over creating stub files. +- Use separators (`---`) before suffix content like Glossary and Contributing. + +## File hygiene + +- No legacy or stub files. If a page has fewer than 10 lines of content, either + finish it or remove its entry from SUMMARY.md and use a draft chapter instead. +- Root-level `src/*.md` files (outside of subdirectories) should only be + `SUMMARY.md` and the introduction. All other content belongs in a subdirectory. +- Images go in `src/images/`. +- Keep `SUMMARY.md` in sync with the actual file tree. + +## CI and tooling + +- MUST: run `mdbook test` in CI on every PR. This compiles all Rust code + examples and catches stale code. Stale examples are the #1 documentation + quality killer. +- MUST: run `mdbook-linkcheck2` in CI to catch broken internal links. It's a + maintained fork; the original `mdbook-linkcheck` is incompatible with + mdbook 0.5+. +- MUST: set `create-missing = false` in `book.toml`. The default `true` setting + silently creates empty files when SUMMARY.md paths are wrong, masking errors. + +## Anti-patterns + +These failures are NOT already covered by the rules above and deserve explicit +callouts. + +1. **"Wall of API"**: trait signatures or method lists without a motivating + example first. Before showing `pub trait ParquetHandler { ... }`, write one + sentence explaining when a connector author would implement it and show a + minimal use case. + +2. **The "obvious to the author" gap**: skipping steps that seem self-evident. + Bad: "Configure the engine, then run the scan." Good: "Pass an + `Arc` to `scan.execute()`. The `Arc` is required because the + iterator must keep the engine alive." + +3. **Undifferentiated FAQ**: if a question is asked frequently, the answer + belongs on the relevant page, not in a separate FAQ. FAQs become + unsearchable graveyards. + +4. **Stale stubs**: a page with just `# Observability` and nothing else wastes + the reader's time and erodes trust. Delete it or use a draft chapter. + +5. **Examples that skip imports**: never force the reader to guess what to + import. Use hidden `# ` lines so imports are compiled but visually + hidden. The eyeball icon in mdBook lets curious readers reveal them. diff --git a/docs/user-guide/LAST_VALIDATED.md b/docs/user-guide/LAST_VALIDATED.md new file mode 100644 index 0000000000..bdc4b4d184 --- /dev/null +++ b/docs/user-guide/LAST_VALIDATED.md @@ -0,0 +1,6 @@ + + +- Last validated: 2026-04-27 +- Last kernel SHA: 6486bd26f37632bbd61cf957db376fbfed9f7805 +- Last kernel branch: main diff --git a/docs/user-guide/Makefile b/docs/user-guide/Makefile new file mode 100644 index 0000000000..80c02643c5 --- /dev/null +++ b/docs/user-guide/Makefile @@ -0,0 +1,26 @@ +KERNEL_PATH := ../.. + +.PHONY: test serve build clean + +# Build the book +build: + mdbook build + +# Serve locally with hot reload +serve: + mdbook serve + +# Test that code examples compile. Build both crates in one cargo invocation so the +# resulting target/debug/deps contains a single delta_kernel rmeta with the unified +# feature set. Two separate cargo builds would leave two rmetas behind and mdbook test +# cannot pick one (E0464). +test: + cargo clean --manifest-path $(KERNEL_PATH)/kernel/Cargo.toml + cargo build --locked --manifest-path $(KERNEL_PATH)/Cargo.toml \ + -p delta_kernel -p delta_kernel_default_engine \ + --features "delta_kernel/internal-api,delta_kernel/prettyprint,delta_kernel_default_engine/rustls,delta_kernel_default_engine/arrow" + mdbook test -L $(KERNEL_PATH)/target/debug/deps + +# Clean build artifacts +clean: + mdbook clean diff --git a/docs/user-guide/README.md b/docs/user-guide/README.md new file mode 100644 index 0000000000..18579c0db8 --- /dev/null +++ b/docs/user-guide/README.md @@ -0,0 +1,19 @@ +# delta kernel mdbook + +This book serves as a user guide + developer guide + overall docs for the kernel. In addition to our +docs.rs and top-level README, the three provide comprehensive documentation for the project. + +## prerequisites + +This book is built with [`mdbook`]. Install the latest version with `cargo install mdbook` or whatever +their latest docs say. If running into issues with the `mermaid` preprocessor (e.g. "Unable to run the preprocessor `mermaid`"), run `cargo install mdbook-mermaid`. + +## building +The book is built in CI and deployed to [docs.delta.io/kernel/rust](https://docs.delta.io/kernel/rust/). When working on +the book locally you can preview changes with mdbook local server: + +```bash +mdbook serve # from docs/user-guide to serve the book on localhost:3000 +``` + +[`mdbook`]: https://github.com/rust-lang/mdBook diff --git a/docs/user-guide/book.toml b/docs/user-guide/book.toml new file mode 100644 index 0000000000..b8fc72874f --- /dev/null +++ b/docs/user-guide/book.toml @@ -0,0 +1,31 @@ +[book] +language = "en" +src = "src" +title = "Delta Kernel Documentation" + +[rust] +edition = "2021" + +[build] +create-missing = false + +[output.html] +additional-css = ["./theme/catppuccin.css"] +default-theme = "latte" +preferred-dark-theme = "macchiato" +additional-js = ["mermaid.min.js", "mermaid-init.js"] + +# Validates internal `[link](../foo.md)` references and #section anchors. +# Uses mdbook-linkcheck2 (a maintained fork; the original mdbook-linkcheck +# is incompatible with mdbook 0.5+). +# Skip web URLs to keep CI hermetic and immune to network flakes. +# optional = true: don't fail `mdbook build` if mdbook-linkcheck2 isn't +# installed locally; CI installs it explicitly. +[output.linkcheck2] +follow-web-links = false +optional = true + +[preprocessor] + +[preprocessor.mermaid] +command = "mdbook-mermaid" diff --git a/docs/user-guide/mermaid-init.js b/docs/user-guide/mermaid-init.js new file mode 100644 index 0000000000..0469ff1675 --- /dev/null +++ b/docs/user-guide/mermaid-init.js @@ -0,0 +1,39 @@ +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this +// file, You can obtain one at https://mozilla.org/MPL/2.0/. + +(() => { + const darkThemes = ['ayu', 'navy', 'coal']; + const lightThemes = ['light', 'rust']; + + const classList = document.getElementsByTagName('html')[0].classList; + + let lastThemeWasLight = true; + for (const cssClass of classList) { + if (darkThemes.includes(cssClass)) { + lastThemeWasLight = false; + break; + } + } + + const theme = lastThemeWasLight ? 'default' : 'dark'; + mermaid.initialize({ startOnLoad: true, theme }); + + // Simplest way to make mermaid re-render the diagrams in the new theme is via refreshing the page + + for (const darkTheme of darkThemes) { + document.getElementById(darkTheme).addEventListener('click', () => { + if (lastThemeWasLight) { + window.location.reload(); + } + }); + } + + for (const lightTheme of lightThemes) { + document.getElementById(lightTheme).addEventListener('click', () => { + if (!lastThemeWasLight) { + window.location.reload(); + } + }); + } +})(); diff --git a/docs/user-guide/mermaid.min.js b/docs/user-guide/mermaid.min.js new file mode 100644 index 0000000000..d5865da86d --- /dev/null +++ b/docs/user-guide/mermaid.min.js @@ -0,0 +1,2609 @@ +/* MIT Licensed. Copyright (c) 2014 - 2022 Knut Sveidqvist */ +/* For license information please see https://github.com/mermaid-js/mermaid/blob/develop/LICENSE */ +"use strict";var __esbuild_esm_mermaid=(()=>{var B2e=Object.create;var by=Object.defineProperty;var F2e=Object.getOwnPropertyDescriptor;var $2e=Object.getOwnPropertyNames;var z2e=Object.getPrototypeOf,G2e=Object.prototype.hasOwnProperty;var o=(t,e)=>by(t,"name",{value:e,configurable:!0});var N=(t,e)=>()=>(t&&(e=t(t=0)),e);var Mi=(t,e)=>()=>(e||t((e={exports:{}}).exports,e),e.exports),hr=(t,e)=>{for(var r in e)by(t,r,{get:e[r],enumerable:!0})},L4=(t,e,r,n)=>{if(e&&typeof e=="object"||typeof e=="function")for(let i of $2e(e))!G2e.call(t,i)&&i!==r&&by(t,i,{get:()=>e[i],enumerable:!(n=F2e(e,i))||n.enumerable});return t},Sr=(t,e,r)=>(L4(t,e,"default"),r&&L4(r,e,"default")),Sa=(t,e,r)=>(r=t!=null?B2e(z2e(t)):{},L4(e||!t||!t.__esModule?by(r,"default",{value:t,enumerable:!0}):r,t)),V2e=t=>L4(by({},"__esModule",{value:!0}),t);var R4=Mi((EC,SC)=>{"use strict";(function(t,e){typeof EC=="object"&&typeof SC<"u"?SC.exports=e():typeof define=="function"&&define.amd?define(e):(t=typeof globalThis<"u"?globalThis:t||self).dayjs=e()})(EC,function(){"use strict";var t=1e3,e=6e4,r=36e5,n="millisecond",i="second",a="minute",s="hour",l="day",u="week",h="month",f="quarter",d="year",p="date",m="Invalid Date",g=/^(\d{4})[-/]?(\d{1,2})?[-/]?(\d{0,2})[Tt\s]*(\d{1,2})?:?(\d{1,2})?:?(\d{1,2})?[.:]?(\d+)?$/,y=/\[([^\]]+)]|Y{1,4}|M{1,4}|D{1,2}|d{1,4}|H{1,2}|h{1,2}|a|A|m{1,2}|s{1,2}|Z{1,2}|SSS/g,v={name:"en",weekdays:"Sunday_Monday_Tuesday_Wednesday_Thursday_Friday_Saturday".split("_"),months:"January_February_March_April_May_June_July_August_September_October_November_December".split("_"),ordinal:o(function(k){var L=["th","st","nd","rd"],R=k%100;return"["+k+(L[(R-20)%10]||L[R]||L[0])+"]"},"ordinal")},x=o(function(k,L,R){var O=String(k);return!O||O.length>=L?k:""+Array(L+1-O.length).join(R)+k},"m"),b={s:x,z:o(function(k){var L=-k.utcOffset(),R=Math.abs(L),O=Math.floor(R/60),M=R%60;return(L<=0?"+":"-")+x(O,2,"0")+":"+x(M,2,"0")},"z"),m:o(function k(L,R){if(L.date()1)return k(F[0])}else{var P=L.name;C[P]=L,M=P}return!O&&M&&(w=M),M||!O&&w},"t"),S=o(function(k,L){if(E(k))return k.clone();var R=typeof L=="object"?L:{};return R.date=k,R.args=arguments,new I(R)},"O"),_=b;_.l=A,_.i=E,_.w=function(k,L){return S(k,{locale:L.$L,utc:L.$u,x:L.$x,$offset:L.$offset})};var I=function(){function k(R){this.$L=A(R.locale,null,!0),this.parse(R),this.$x=this.$x||R.x||{},this[T]=!0}o(k,"M");var L=k.prototype;return L.parse=function(R){this.$d=function(O){var M=O.date,B=O.utc;if(M===null)return new Date(NaN);if(_.u(M))return new Date;if(M instanceof Date)return new Date(M);if(typeof M=="string"&&!/Z$/i.test(M)){var F=M.match(g);if(F){var P=F[2]-1||0,z=(F[7]||"0").substring(0,3);return B?new Date(Date.UTC(F[1],P,F[3]||1,F[4]||0,F[5]||0,F[6]||0,z)):new Date(F[1],P,F[3]||1,F[4]||0,F[5]||0,F[6]||0,z)}}return new Date(M)}(R),this.init()},L.init=function(){var R=this.$d;this.$y=R.getFullYear(),this.$M=R.getMonth(),this.$D=R.getDate(),this.$W=R.getDay(),this.$H=R.getHours(),this.$m=R.getMinutes(),this.$s=R.getSeconds(),this.$ms=R.getMilliseconds()},L.$utils=function(){return _},L.isValid=function(){return this.$d.toString()!==m},L.isSame=function(R,O){var M=S(R);return this.startOf(O)<=M&&M<=this.endOf(O)},L.isAfter=function(R,O){return S(R){"use strict";CF=Sa(R4(),1),eu={trace:0,debug:1,info:2,warn:3,error:4,fatal:5},Y={trace:o((...t)=>{},"trace"),debug:o((...t)=>{},"debug"),info:o((...t)=>{},"info"),warn:o((...t)=>{},"warn"),error:o((...t)=>{},"error"),fatal:o((...t)=>{},"fatal")},wy=o(function(t="fatal"){let e=eu.fatal;typeof t=="string"?t.toLowerCase()in eu&&(e=eu[t]):typeof t=="number"&&(e=t),Y.trace=()=>{},Y.debug=()=>{},Y.info=()=>{},Y.warn=()=>{},Y.error=()=>{},Y.fatal=()=>{},e<=eu.fatal&&(Y.fatal=console.error?console.error.bind(console,bo("FATAL"),"color: orange"):console.log.bind(console,"\x1B[35m",bo("FATAL"))),e<=eu.error&&(Y.error=console.error?console.error.bind(console,bo("ERROR"),"color: orange"):console.log.bind(console,"\x1B[31m",bo("ERROR"))),e<=eu.warn&&(Y.warn=console.warn?console.warn.bind(console,bo("WARN"),"color: orange"):console.log.bind(console,"\x1B[33m",bo("WARN"))),e<=eu.info&&(Y.info=console.info?console.info.bind(console,bo("INFO"),"color: lightblue"):console.log.bind(console,"\x1B[34m",bo("INFO"))),e<=eu.debug&&(Y.debug=console.debug?console.debug.bind(console,bo("DEBUG"),"color: lightgreen"):console.log.bind(console,"\x1B[32m",bo("DEBUG"))),e<=eu.trace&&(Y.trace=console.debug?console.debug.bind(console,bo("TRACE"),"color: lightgreen"):console.log.bind(console,"\x1B[32m",bo("TRACE")))},"setLogLevel"),bo=o(t=>`%c${(0,CF.default)().format("ss.SSS")} : ${t} : `,"format")});var U2e,e0,CC,AF,N4=N(()=>{"use strict";U2e=Object.freeze({left:0,top:0,width:16,height:16}),e0=Object.freeze({rotate:0,vFlip:!1,hFlip:!1}),CC=Object.freeze({...U2e,...e0}),AF=Object.freeze({...CC,body:"",hidden:!1})});var H2e,_F,DF=N(()=>{"use strict";N4();H2e=Object.freeze({width:null,height:null}),_F=Object.freeze({...H2e,...e0})});var AC,M4,LF=N(()=>{"use strict";AC=o((t,e,r,n="")=>{let i=t.split(":");if(t.slice(0,1)==="@"){if(i.length<2||i.length>3)return null;n=i.shift().slice(1)}if(i.length>3||!i.length)return null;if(i.length>1){let l=i.pop(),u=i.pop(),h={provider:i.length>0?i[0]:n,prefix:u,name:l};return e&&!M4(h)?null:h}let a=i[0],s=a.split("-");if(s.length>1){let l={provider:n,prefix:s.shift(),name:s.join("-")};return e&&!M4(l)?null:l}if(r&&n===""){let l={provider:n,prefix:"",name:a};return e&&!M4(l,r)?null:l}return null},"stringToIcon"),M4=o((t,e)=>t?!!((e&&t.prefix===""||t.prefix)&&t.name):!1,"validateIconName")});function RF(t,e){let r={};!t.hFlip!=!e.hFlip&&(r.hFlip=!0),!t.vFlip!=!e.vFlip&&(r.vFlip=!0);let n=((t.rotate||0)+(e.rotate||0))%4;return n&&(r.rotate=n),r}var NF=N(()=>{"use strict";o(RF,"mergeIconTransformations")});function _C(t,e){let r=RF(t,e);for(let n in AF)n in e0?n in t&&!(n in r)&&(r[n]=e0[n]):n in e?r[n]=e[n]:n in t&&(r[n]=t[n]);return r}var MF=N(()=>{"use strict";N4();NF();o(_C,"mergeIconData")});function IF(t,e){let r=t.icons,n=t.aliases||Object.create(null),i=Object.create(null);function a(s){if(r[s])return i[s]=[];if(!(s in i)){i[s]=null;let l=n[s]&&n[s].parent,u=l&&a(l);u&&(i[s]=[l].concat(u))}return i[s]}return o(a,"resolve"),(e||Object.keys(r).concat(Object.keys(n))).forEach(a),i}var OF=N(()=>{"use strict";o(IF,"getIconsTree")});function PF(t,e,r){let n=t.icons,i=t.aliases||Object.create(null),a={};function s(l){a=_C(n[l]||i[l],a)}return o(s,"parse"),s(e),r.forEach(s),_C(t,a)}function DC(t,e){if(t.icons[e])return PF(t,e,[]);let r=IF(t,[e])[e];return r?PF(t,e,r):null}var BF=N(()=>{"use strict";MF();OF();o(PF,"internalGetIconData");o(DC,"getIconData")});function LC(t,e,r){if(e===1)return t;if(r=r||100,typeof t=="number")return Math.ceil(t*e*r)/r;if(typeof t!="string")return t;let n=t.split(W2e);if(n===null||!n.length)return t;let i=[],a=n.shift(),s=q2e.test(a);for(;;){if(s){let l=parseFloat(a);isNaN(l)?i.push(a):i.push(Math.ceil(l*e*r)/r)}else i.push(a);if(a=n.shift(),a===void 0)return i.join("");s=!s}}var W2e,q2e,FF=N(()=>{"use strict";W2e=/(-?[0-9.]*[0-9]+[0-9.]*)/g,q2e=/^-?[0-9.]*[0-9]+[0-9.]*$/g;o(LC,"calculateSize")});function Y2e(t,e="defs"){let r="",n=t.indexOf("<"+e);for(;n>=0;){let i=t.indexOf(">",n),a=t.indexOf("",a);if(s===-1)break;r+=t.slice(i+1,a).trim(),t=t.slice(0,n).trim()+t.slice(s+1)}return{defs:r,content:t}}function X2e(t,e){return t?""+t+""+e:e}function $F(t,e,r){let n=Y2e(t);return X2e(n.defs,e+n.content+r)}var zF=N(()=>{"use strict";o(Y2e,"splitSVGDefs");o(X2e,"mergeDefsAndContent");o($F,"wrapSVGContent")});function RC(t,e){let r={...CC,...t},n={..._F,...e},i={left:r.left,top:r.top,width:r.width,height:r.height},a=r.body;[r,n].forEach(y=>{let v=[],x=y.hFlip,b=y.vFlip,w=y.rotate;x?b?w+=2:(v.push("translate("+(i.width+i.left).toString()+" "+(0-i.top).toString()+")"),v.push("scale(-1 1)"),i.top=i.left=0):b&&(v.push("translate("+(0-i.left).toString()+" "+(i.height+i.top).toString()+")"),v.push("scale(1 -1)"),i.top=i.left=0);let C;switch(w<0&&(w-=Math.floor(w/4)*4),w=w%4,w){case 1:C=i.height/2+i.top,v.unshift("rotate(90 "+C.toString()+" "+C.toString()+")");break;case 2:v.unshift("rotate(180 "+(i.width/2+i.left).toString()+" "+(i.height/2+i.top).toString()+")");break;case 3:C=i.width/2+i.left,v.unshift("rotate(-90 "+C.toString()+" "+C.toString()+")");break}w%2===1&&(i.left!==i.top&&(C=i.left,i.left=i.top,i.top=C),i.width!==i.height&&(C=i.width,i.width=i.height,i.height=C)),v.length&&(a=$F(a,'',""))});let s=n.width,l=n.height,u=i.width,h=i.height,f,d;s===null?(d=l===null?"1em":l==="auto"?h:l,f=LC(d,u/h)):(f=s==="auto"?u:s,d=l===null?LC(f,h/u):l==="auto"?h:l);let p={},m=o((y,v)=>{j2e(v)||(p[y]=v.toString())},"setAttr");m("width",f),m("height",d);let g=[i.left,i.top,u,h];return p.viewBox=g.join(" "),{attributes:p,viewBox:g,body:a}}var j2e,GF=N(()=>{"use strict";N4();DF();FF();zF();j2e=o(t=>t==="unset"||t==="undefined"||t==="none","isUnsetKeyword");o(RC,"iconToSVG")});function NC(t,e=Q2e){let r=[],n;for(;n=K2e.exec(t);)r.push(n[1]);if(!r.length)return t;let i="suffix"+(Math.random()*16777216|Date.now()).toString(16);return r.forEach(a=>{let s=typeof e=="function"?e(a):e+(Z2e++).toString(),l=a.replace(/[.*+?^${}()|[\]\\]/g,"\\$&");t=t.replace(new RegExp('([#;"])('+l+')([")]|\\.[a-z])',"g"),"$1"+s+i+"$3")}),t=t.replace(new RegExp(i,"g"),""),t}var K2e,Q2e,Z2e,VF=N(()=>{"use strict";K2e=/\sid="(\S+)"/g,Q2e="IconifyId"+Date.now().toString(16)+(Math.random()*16777216|0).toString(16),Z2e=0;o(NC,"replaceIDs")});function MC(t,e){let r=t.indexOf("xlink:")===-1?"":' xmlns:xlink="http://www.w3.org/1999/xlink"';for(let n in e)r+=" "+n+'="'+e[n]+'"';return'"+t+""}var UF=N(()=>{"use strict";o(MC,"iconToHTML")});var WF=Mi((iit,HF)=>{"use strict";var t0=1e3,r0=t0*60,n0=r0*60,Wf=n0*24,J2e=Wf*7,exe=Wf*365.25;HF.exports=function(t,e){e=e||{};var r=typeof t;if(r==="string"&&t.length>0)return txe(t);if(r==="number"&&isFinite(t))return e.long?nxe(t):rxe(t);throw new Error("val is not a non-empty string or a valid number. val="+JSON.stringify(t))};function txe(t){if(t=String(t),!(t.length>100)){var e=/^(-?(?:\d+)?\.?\d+) *(milliseconds?|msecs?|ms|seconds?|secs?|s|minutes?|mins?|m|hours?|hrs?|h|days?|d|weeks?|w|years?|yrs?|y)?$/i.exec(t);if(e){var r=parseFloat(e[1]),n=(e[2]||"ms").toLowerCase();switch(n){case"years":case"year":case"yrs":case"yr":case"y":return r*exe;case"weeks":case"week":case"w":return r*J2e;case"days":case"day":case"d":return r*Wf;case"hours":case"hour":case"hrs":case"hr":case"h":return r*n0;case"minutes":case"minute":case"mins":case"min":case"m":return r*r0;case"seconds":case"second":case"secs":case"sec":case"s":return r*t0;case"milliseconds":case"millisecond":case"msecs":case"msec":case"ms":return r;default:return}}}}o(txe,"parse");function rxe(t){var e=Math.abs(t);return e>=Wf?Math.round(t/Wf)+"d":e>=n0?Math.round(t/n0)+"h":e>=r0?Math.round(t/r0)+"m":e>=t0?Math.round(t/t0)+"s":t+"ms"}o(rxe,"fmtShort");function nxe(t){var e=Math.abs(t);return e>=Wf?I4(t,e,Wf,"day"):e>=n0?I4(t,e,n0,"hour"):e>=r0?I4(t,e,r0,"minute"):e>=t0?I4(t,e,t0,"second"):t+" ms"}o(nxe,"fmtLong");function I4(t,e,r,n){var i=e>=r*1.5;return Math.round(t/r)+" "+n+(i?"s":"")}o(I4,"plural")});var YF=Mi((sit,qF)=>{"use strict";function ixe(t){r.debug=r,r.default=r,r.coerce=u,r.disable=s,r.enable=i,r.enabled=l,r.humanize=WF(),r.destroy=h,Object.keys(t).forEach(f=>{r[f]=t[f]}),r.names=[],r.skips=[],r.formatters={};function e(f){let d=0;for(let p=0;p{if(E==="%%")return"%";C++;let S=r.formatters[A];if(typeof S=="function"){let _=v[C];E=S.call(x,_),v.splice(C,1),C--}return E}),r.formatArgs.call(x,v),(x.log||r.log).apply(x,v)}return o(y,"debug"),y.namespace=f,y.useColors=r.useColors(),y.color=r.selectColor(f),y.extend=n,y.destroy=r.destroy,Object.defineProperty(y,"enabled",{enumerable:!0,configurable:!1,get:o(()=>p!==null?p:(m!==r.namespaces&&(m=r.namespaces,g=r.enabled(f)),g),"get"),set:o(v=>{p=v},"set")}),typeof r.init=="function"&&r.init(y),y}o(r,"createDebug");function n(f,d){let p=r(this.namespace+(typeof d>"u"?":":d)+f);return p.log=this.log,p}o(n,"extend");function i(f){r.save(f),r.namespaces=f,r.names=[],r.skips=[];let d=(typeof f=="string"?f:"").trim().replace(" ",",").split(",").filter(Boolean);for(let p of d)p[0]==="-"?r.skips.push(p.slice(1)):r.names.push(p)}o(i,"enable");function a(f,d){let p=0,m=0,g=-1,y=0;for(;p"-"+d)].join(",");return r.enable(""),f}o(s,"disable");function l(f){for(let d of r.skips)if(a(f,d))return!1;for(let d of r.names)if(a(f,d))return!0;return!1}o(l,"enabled");function u(f){return f instanceof Error?f.stack||f.message:f}o(u,"coerce");function h(){console.warn("Instance method `debug.destroy()` is deprecated and no longer does anything. It will be removed in the next major version of `debug`.")}return o(h,"destroy"),r.enable(r.load()),r}o(ixe,"setup");qF.exports=ixe});var XF=Mi((qs,O4)=>{"use strict";qs.formatArgs=sxe;qs.save=oxe;qs.load=lxe;qs.useColors=axe;qs.storage=cxe();qs.destroy=(()=>{let t=!1;return()=>{t||(t=!0,console.warn("Instance method `debug.destroy()` is deprecated and no longer does anything. It will be removed in the next major version of `debug`."))}})();qs.colors=["#0000CC","#0000FF","#0033CC","#0033FF","#0066CC","#0066FF","#0099CC","#0099FF","#00CC00","#00CC33","#00CC66","#00CC99","#00CCCC","#00CCFF","#3300CC","#3300FF","#3333CC","#3333FF","#3366CC","#3366FF","#3399CC","#3399FF","#33CC00","#33CC33","#33CC66","#33CC99","#33CCCC","#33CCFF","#6600CC","#6600FF","#6633CC","#6633FF","#66CC00","#66CC33","#9900CC","#9900FF","#9933CC","#9933FF","#99CC00","#99CC33","#CC0000","#CC0033","#CC0066","#CC0099","#CC00CC","#CC00FF","#CC3300","#CC3333","#CC3366","#CC3399","#CC33CC","#CC33FF","#CC6600","#CC6633","#CC9900","#CC9933","#CCCC00","#CCCC33","#FF0000","#FF0033","#FF0066","#FF0099","#FF00CC","#FF00FF","#FF3300","#FF3333","#FF3366","#FF3399","#FF33CC","#FF33FF","#FF6600","#FF6633","#FF9900","#FF9933","#FFCC00","#FFCC33"];function axe(){if(typeof window<"u"&&window.process&&(window.process.type==="renderer"||window.process.__nwjs))return!0;if(typeof navigator<"u"&&navigator.userAgent&&navigator.userAgent.toLowerCase().match(/(edge|trident)\/(\d+)/))return!1;let t;return typeof document<"u"&&document.documentElement&&document.documentElement.style&&document.documentElement.style.WebkitAppearance||typeof window<"u"&&window.console&&(window.console.firebug||window.console.exception&&window.console.table)||typeof navigator<"u"&&navigator.userAgent&&(t=navigator.userAgent.toLowerCase().match(/firefox\/(\d+)/))&&parseInt(t[1],10)>=31||typeof navigator<"u"&&navigator.userAgent&&navigator.userAgent.toLowerCase().match(/applewebkit\/(\d+)/)}o(axe,"useColors");function sxe(t){if(t[0]=(this.useColors?"%c":"")+this.namespace+(this.useColors?" %c":" ")+t[0]+(this.useColors?"%c ":" ")+"+"+O4.exports.humanize(this.diff),!this.useColors)return;let e="color: "+this.color;t.splice(1,0,e,"color: inherit");let r=0,n=0;t[0].replace(/%[a-zA-Z%]/g,i=>{i!=="%%"&&(r++,i==="%c"&&(n=r))}),t.splice(n,0,e)}o(sxe,"formatArgs");qs.log=console.debug||console.log||(()=>{});function oxe(t){try{t?qs.storage.setItem("debug",t):qs.storage.removeItem("debug")}catch{}}o(oxe,"save");function lxe(){let t;try{t=qs.storage.getItem("debug")}catch{}return!t&&typeof process<"u"&&"env"in process&&(t=process.env.DEBUG),t}o(lxe,"load");function cxe(){try{return localStorage}catch{}}o(cxe,"localstorage");O4.exports=YF()(qs);var{formatters:uxe}=O4.exports;uxe.j=function(t){try{return JSON.stringify(t)}catch(e){return"[UnexpectedJSONParseError]: "+e.message}}});var uit,jF=N(()=>{"use strict";LF();BF();GF();VF();UF();uit=Sa(XF(),1)});var OC,IC,KF,P4,hxe,wo,tu=N(()=>{"use strict";vt();jF();OC={body:'?',height:80,width:80},IC=new Map,KF=new Map,P4=o(t=>{for(let e of t){if(!e.name)throw new Error('Invalid icon loader. Must have a "name" property with non-empty string value.');if(Y.debug("Registering icon pack:",e.name),"loader"in e)KF.set(e.name,e.loader);else if("icons"in e)IC.set(e.name,e.icons);else throw Y.error("Invalid icon loader:",e),new Error('Invalid icon loader. Must have either "icons" or "loader" property.')}},"registerIconPacks"),hxe=o(async(t,e)=>{let r=AC(t,!0,e!==void 0);if(!r)throw new Error(`Invalid icon name: ${t}`);let n=r.prefix||e;if(!n)throw new Error(`Icon name must contain a prefix: ${t}`);let i=IC.get(n);if(!i){let s=KF.get(n);if(!s)throw new Error(`Icon set not found: ${r.prefix}`);try{i={...await s(),prefix:n},IC.set(n,i)}catch(l){throw Y.error(l),new Error(`Failed to load icon set: ${r.prefix}`)}}let a=DC(i,r.name);if(!a)throw new Error(`Icon not found: ${t}`);return a},"getRegisteredIconData"),wo=o(async(t,e)=>{let r;try{r=await hxe(t,e?.fallbackPrefix)}catch(a){Y.error(a),r=OC}let n=RC(r,e);return MC(NC(n.body),n.attributes)},"getIconSVG")});function B4(t){for(var e=[],r=1;r{"use strict";o(B4,"dedent")});var F4,qf,QF,$4=N(()=>{"use strict";F4=/^-{3}\s*[\n\r](.*?)[\n\r]-{3}\s*[\n\r]+/s,qf=/%{2}{\s*(?:(\w+)\s*:|(\w+))\s*(?:(\w+)|((?:(?!}%{2}).|\r?\n)*))?\s*(?:}%{2})?/gi,QF=/\s*%%.*\n/gm});var i0,BC=N(()=>{"use strict";i0=class extends Error{static{o(this,"UnknownDiagramError")}constructor(e){super(e),this.name="UnknownDiagramError"}}});var Yf,a0,z4,FC,ZF,Xf=N(()=>{"use strict";vt();$4();BC();Yf={},a0=o(function(t,e){t=t.replace(F4,"").replace(qf,"").replace(QF,` +`);for(let[r,{detector:n}]of Object.entries(Yf))if(n(t,e))return r;throw new i0(`No diagram type detected matching given configuration for text: ${t}`)},"detectType"),z4=o((...t)=>{for(let{id:e,detector:r,loader:n}of t)FC(e,r,n)},"registerLazyLoadedDiagrams"),FC=o((t,e,r)=>{Yf[t]&&Y.warn(`Detector with key ${t} already exists. Overwriting.`),Yf[t]={detector:e,loader:r},Y.debug(`Detector with key ${t} added${r?" with loader":""}`)},"addDetector"),ZF=o(t=>Yf[t].loader,"getDiagramLoader")});var Ty,JF,$C=N(()=>{"use strict";Ty=function(){var t=o(function($e,Re,Ie,be){for(Ie=Ie||{},be=$e.length;be--;Ie[$e[be]]=Re);return Ie},"o"),e=[1,24],r=[1,25],n=[1,26],i=[1,27],a=[1,28],s=[1,63],l=[1,64],u=[1,65],h=[1,66],f=[1,67],d=[1,68],p=[1,69],m=[1,29],g=[1,30],y=[1,31],v=[1,32],x=[1,33],b=[1,34],w=[1,35],C=[1,36],T=[1,37],E=[1,38],A=[1,39],S=[1,40],_=[1,41],I=[1,42],D=[1,43],k=[1,44],L=[1,45],R=[1,46],O=[1,47],M=[1,48],B=[1,50],F=[1,51],P=[1,52],z=[1,53],$=[1,54],H=[1,55],Q=[1,56],j=[1,57],ie=[1,58],ne=[1,59],le=[1,60],he=[14,42],K=[14,34,36,37,38,39,40,41,42,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74],X=[12,14,34,36,37,38,39,40,41,42,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74],te=[1,82],J=[1,83],se=[1,84],ue=[1,85],Z=[12,14,42],Se=[12,14,33,42],ce=[12,14,33,42,76,77,79,80],ae=[12,33],Oe=[34,36,37,38,39,40,41,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74],ge={trace:o(function(){},"trace"),yy:{},symbols_:{error:2,start:3,mermaidDoc:4,direction:5,direction_tb:6,direction_bt:7,direction_rl:8,direction_lr:9,graphConfig:10,C4_CONTEXT:11,NEWLINE:12,statements:13,EOF:14,C4_CONTAINER:15,C4_COMPONENT:16,C4_DYNAMIC:17,C4_DEPLOYMENT:18,otherStatements:19,diagramStatements:20,otherStatement:21,title:22,accDescription:23,acc_title:24,acc_title_value:25,acc_descr:26,acc_descr_value:27,acc_descr_multiline_value:28,boundaryStatement:29,boundaryStartStatement:30,boundaryStopStatement:31,boundaryStart:32,LBRACE:33,ENTERPRISE_BOUNDARY:34,attributes:35,SYSTEM_BOUNDARY:36,BOUNDARY:37,CONTAINER_BOUNDARY:38,NODE:39,NODE_L:40,NODE_R:41,RBRACE:42,diagramStatement:43,PERSON:44,PERSON_EXT:45,SYSTEM:46,SYSTEM_DB:47,SYSTEM_QUEUE:48,SYSTEM_EXT:49,SYSTEM_EXT_DB:50,SYSTEM_EXT_QUEUE:51,CONTAINER:52,CONTAINER_DB:53,CONTAINER_QUEUE:54,CONTAINER_EXT:55,CONTAINER_EXT_DB:56,CONTAINER_EXT_QUEUE:57,COMPONENT:58,COMPONENT_DB:59,COMPONENT_QUEUE:60,COMPONENT_EXT:61,COMPONENT_EXT_DB:62,COMPONENT_EXT_QUEUE:63,REL:64,BIREL:65,REL_U:66,REL_D:67,REL_L:68,REL_R:69,REL_B:70,REL_INDEX:71,UPDATE_EL_STYLE:72,UPDATE_REL_STYLE:73,UPDATE_LAYOUT_CONFIG:74,attribute:75,STR:76,STR_KEY:77,STR_VALUE:78,ATTRIBUTE:79,ATTRIBUTE_EMPTY:80,$accept:0,$end:1},terminals_:{2:"error",6:"direction_tb",7:"direction_bt",8:"direction_rl",9:"direction_lr",11:"C4_CONTEXT",12:"NEWLINE",14:"EOF",15:"C4_CONTAINER",16:"C4_COMPONENT",17:"C4_DYNAMIC",18:"C4_DEPLOYMENT",22:"title",23:"accDescription",24:"acc_title",25:"acc_title_value",26:"acc_descr",27:"acc_descr_value",28:"acc_descr_multiline_value",33:"LBRACE",34:"ENTERPRISE_BOUNDARY",36:"SYSTEM_BOUNDARY",37:"BOUNDARY",38:"CONTAINER_BOUNDARY",39:"NODE",40:"NODE_L",41:"NODE_R",42:"RBRACE",44:"PERSON",45:"PERSON_EXT",46:"SYSTEM",47:"SYSTEM_DB",48:"SYSTEM_QUEUE",49:"SYSTEM_EXT",50:"SYSTEM_EXT_DB",51:"SYSTEM_EXT_QUEUE",52:"CONTAINER",53:"CONTAINER_DB",54:"CONTAINER_QUEUE",55:"CONTAINER_EXT",56:"CONTAINER_EXT_DB",57:"CONTAINER_EXT_QUEUE",58:"COMPONENT",59:"COMPONENT_DB",60:"COMPONENT_QUEUE",61:"COMPONENT_EXT",62:"COMPONENT_EXT_DB",63:"COMPONENT_EXT_QUEUE",64:"REL",65:"BIREL",66:"REL_U",67:"REL_D",68:"REL_L",69:"REL_R",70:"REL_B",71:"REL_INDEX",72:"UPDATE_EL_STYLE",73:"UPDATE_REL_STYLE",74:"UPDATE_LAYOUT_CONFIG",76:"STR",77:"STR_KEY",78:"STR_VALUE",79:"ATTRIBUTE",80:"ATTRIBUTE_EMPTY"},productions_:[0,[3,1],[3,1],[5,1],[5,1],[5,1],[5,1],[4,1],[10,4],[10,4],[10,4],[10,4],[10,4],[13,1],[13,1],[13,2],[19,1],[19,2],[19,3],[21,1],[21,1],[21,2],[21,2],[21,1],[29,3],[30,3],[30,3],[30,4],[32,2],[32,2],[32,2],[32,2],[32,2],[32,2],[32,2],[31,1],[20,1],[20,2],[20,3],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,1],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[43,2],[35,1],[35,2],[75,1],[75,2],[75,1],[75,1]],performAction:o(function(Re,Ie,be,W,de,re,oe){var V=re.length-1;switch(de){case 3:W.setDirection("TB");break;case 4:W.setDirection("BT");break;case 5:W.setDirection("RL");break;case 6:W.setDirection("LR");break;case 8:case 9:case 10:case 11:case 12:W.setC4Type(re[V-3]);break;case 19:W.setTitle(re[V].substring(6)),this.$=re[V].substring(6);break;case 20:W.setAccDescription(re[V].substring(15)),this.$=re[V].substring(15);break;case 21:this.$=re[V].trim(),W.setTitle(this.$);break;case 22:case 23:this.$=re[V].trim(),W.setAccDescription(this.$);break;case 28:re[V].splice(2,0,"ENTERPRISE"),W.addPersonOrSystemBoundary(...re[V]),this.$=re[V];break;case 29:re[V].splice(2,0,"SYSTEM"),W.addPersonOrSystemBoundary(...re[V]),this.$=re[V];break;case 30:W.addPersonOrSystemBoundary(...re[V]),this.$=re[V];break;case 31:re[V].splice(2,0,"CONTAINER"),W.addContainerBoundary(...re[V]),this.$=re[V];break;case 32:W.addDeploymentNode("node",...re[V]),this.$=re[V];break;case 33:W.addDeploymentNode("nodeL",...re[V]),this.$=re[V];break;case 34:W.addDeploymentNode("nodeR",...re[V]),this.$=re[V];break;case 35:W.popBoundaryParseStack();break;case 39:W.addPersonOrSystem("person",...re[V]),this.$=re[V];break;case 40:W.addPersonOrSystem("external_person",...re[V]),this.$=re[V];break;case 41:W.addPersonOrSystem("system",...re[V]),this.$=re[V];break;case 42:W.addPersonOrSystem("system_db",...re[V]),this.$=re[V];break;case 43:W.addPersonOrSystem("system_queue",...re[V]),this.$=re[V];break;case 44:W.addPersonOrSystem("external_system",...re[V]),this.$=re[V];break;case 45:W.addPersonOrSystem("external_system_db",...re[V]),this.$=re[V];break;case 46:W.addPersonOrSystem("external_system_queue",...re[V]),this.$=re[V];break;case 47:W.addContainer("container",...re[V]),this.$=re[V];break;case 48:W.addContainer("container_db",...re[V]),this.$=re[V];break;case 49:W.addContainer("container_queue",...re[V]),this.$=re[V];break;case 50:W.addContainer("external_container",...re[V]),this.$=re[V];break;case 51:W.addContainer("external_container_db",...re[V]),this.$=re[V];break;case 52:W.addContainer("external_container_queue",...re[V]),this.$=re[V];break;case 53:W.addComponent("component",...re[V]),this.$=re[V];break;case 54:W.addComponent("component_db",...re[V]),this.$=re[V];break;case 55:W.addComponent("component_queue",...re[V]),this.$=re[V];break;case 56:W.addComponent("external_component",...re[V]),this.$=re[V];break;case 57:W.addComponent("external_component_db",...re[V]),this.$=re[V];break;case 58:W.addComponent("external_component_queue",...re[V]),this.$=re[V];break;case 60:W.addRel("rel",...re[V]),this.$=re[V];break;case 61:W.addRel("birel",...re[V]),this.$=re[V];break;case 62:W.addRel("rel_u",...re[V]),this.$=re[V];break;case 63:W.addRel("rel_d",...re[V]),this.$=re[V];break;case 64:W.addRel("rel_l",...re[V]),this.$=re[V];break;case 65:W.addRel("rel_r",...re[V]),this.$=re[V];break;case 66:W.addRel("rel_b",...re[V]),this.$=re[V];break;case 67:re[V].splice(0,1),W.addRel("rel",...re[V]),this.$=re[V];break;case 68:W.updateElStyle("update_el_style",...re[V]),this.$=re[V];break;case 69:W.updateRelStyle("update_rel_style",...re[V]),this.$=re[V];break;case 70:W.updateLayoutConfig("update_layout_config",...re[V]),this.$=re[V];break;case 71:this.$=[re[V]];break;case 72:re[V].unshift(re[V-1]),this.$=re[V];break;case 73:case 75:this.$=re[V].trim();break;case 74:let xe={};xe[re[V-1].trim()]=re[V].trim(),this.$=xe;break;case 76:this.$="";break}},"anonymous"),table:[{3:1,4:2,5:3,6:[1,5],7:[1,6],8:[1,7],9:[1,8],10:4,11:[1,9],15:[1,10],16:[1,11],17:[1,12],18:[1,13]},{1:[3]},{1:[2,1]},{1:[2,2]},{1:[2,7]},{1:[2,3]},{1:[2,4]},{1:[2,5]},{1:[2,6]},{12:[1,14]},{12:[1,15]},{12:[1,16]},{12:[1,17]},{12:[1,18]},{13:19,19:20,20:21,21:22,22:e,23:r,24:n,26:i,28:a,29:49,30:61,32:62,34:s,36:l,37:u,38:h,39:f,40:d,41:p,43:23,44:m,45:g,46:y,47:v,48:x,49:b,50:w,51:C,52:T,53:E,54:A,55:S,56:_,57:I,58:D,59:k,60:L,61:R,62:O,63:M,64:B,65:F,66:P,67:z,68:$,69:H,70:Q,71:j,72:ie,73:ne,74:le},{13:70,19:20,20:21,21:22,22:e,23:r,24:n,26:i,28:a,29:49,30:61,32:62,34:s,36:l,37:u,38:h,39:f,40:d,41:p,43:23,44:m,45:g,46:y,47:v,48:x,49:b,50:w,51:C,52:T,53:E,54:A,55:S,56:_,57:I,58:D,59:k,60:L,61:R,62:O,63:M,64:B,65:F,66:P,67:z,68:$,69:H,70:Q,71:j,72:ie,73:ne,74:le},{13:71,19:20,20:21,21:22,22:e,23:r,24:n,26:i,28:a,29:49,30:61,32:62,34:s,36:l,37:u,38:h,39:f,40:d,41:p,43:23,44:m,45:g,46:y,47:v,48:x,49:b,50:w,51:C,52:T,53:E,54:A,55:S,56:_,57:I,58:D,59:k,60:L,61:R,62:O,63:M,64:B,65:F,66:P,67:z,68:$,69:H,70:Q,71:j,72:ie,73:ne,74:le},{13:72,19:20,20:21,21:22,22:e,23:r,24:n,26:i,28:a,29:49,30:61,32:62,34:s,36:l,37:u,38:h,39:f,40:d,41:p,43:23,44:m,45:g,46:y,47:v,48:x,49:b,50:w,51:C,52:T,53:E,54:A,55:S,56:_,57:I,58:D,59:k,60:L,61:R,62:O,63:M,64:B,65:F,66:P,67:z,68:$,69:H,70:Q,71:j,72:ie,73:ne,74:le},{13:73,19:20,20:21,21:22,22:e,23:r,24:n,26:i,28:a,29:49,30:61,32:62,34:s,36:l,37:u,38:h,39:f,40:d,41:p,43:23,44:m,45:g,46:y,47:v,48:x,49:b,50:w,51:C,52:T,53:E,54:A,55:S,56:_,57:I,58:D,59:k,60:L,61:R,62:O,63:M,64:B,65:F,66:P,67:z,68:$,69:H,70:Q,71:j,72:ie,73:ne,74:le},{14:[1,74]},t(he,[2,13],{43:23,29:49,30:61,32:62,20:75,34:s,36:l,37:u,38:h,39:f,40:d,41:p,44:m,45:g,46:y,47:v,48:x,49:b,50:w,51:C,52:T,53:E,54:A,55:S,56:_,57:I,58:D,59:k,60:L,61:R,62:O,63:M,64:B,65:F,66:P,67:z,68:$,69:H,70:Q,71:j,72:ie,73:ne,74:le}),t(he,[2,14]),t(K,[2,16],{12:[1,76]}),t(he,[2,36],{12:[1,77]}),t(X,[2,19]),t(X,[2,20]),{25:[1,78]},{27:[1,79]},t(X,[2,23]),{35:80,75:81,76:te,77:J,79:se,80:ue},{35:86,75:81,76:te,77:J,79:se,80:ue},{35:87,75:81,76:te,77:J,79:se,80:ue},{35:88,75:81,76:te,77:J,79:se,80:ue},{35:89,75:81,76:te,77:J,79:se,80:ue},{35:90,75:81,76:te,77:J,79:se,80:ue},{35:91,75:81,76:te,77:J,79:se,80:ue},{35:92,75:81,76:te,77:J,79:se,80:ue},{35:93,75:81,76:te,77:J,79:se,80:ue},{35:94,75:81,76:te,77:J,79:se,80:ue},{35:95,75:81,76:te,77:J,79:se,80:ue},{35:96,75:81,76:te,77:J,79:se,80:ue},{35:97,75:81,76:te,77:J,79:se,80:ue},{35:98,75:81,76:te,77:J,79:se,80:ue},{35:99,75:81,76:te,77:J,79:se,80:ue},{35:100,75:81,76:te,77:J,79:se,80:ue},{35:101,75:81,76:te,77:J,79:se,80:ue},{35:102,75:81,76:te,77:J,79:se,80:ue},{35:103,75:81,76:te,77:J,79:se,80:ue},{35:104,75:81,76:te,77:J,79:se,80:ue},t(Z,[2,59]),{35:105,75:81,76:te,77:J,79:se,80:ue},{35:106,75:81,76:te,77:J,79:se,80:ue},{35:107,75:81,76:te,77:J,79:se,80:ue},{35:108,75:81,76:te,77:J,79:se,80:ue},{35:109,75:81,76:te,77:J,79:se,80:ue},{35:110,75:81,76:te,77:J,79:se,80:ue},{35:111,75:81,76:te,77:J,79:se,80:ue},{35:112,75:81,76:te,77:J,79:se,80:ue},{35:113,75:81,76:te,77:J,79:se,80:ue},{35:114,75:81,76:te,77:J,79:se,80:ue},{35:115,75:81,76:te,77:J,79:se,80:ue},{20:116,29:49,30:61,32:62,34:s,36:l,37:u,38:h,39:f,40:d,41:p,43:23,44:m,45:g,46:y,47:v,48:x,49:b,50:w,51:C,52:T,53:E,54:A,55:S,56:_,57:I,58:D,59:k,60:L,61:R,62:O,63:M,64:B,65:F,66:P,67:z,68:$,69:H,70:Q,71:j,72:ie,73:ne,74:le},{12:[1,118],33:[1,117]},{35:119,75:81,76:te,77:J,79:se,80:ue},{35:120,75:81,76:te,77:J,79:se,80:ue},{35:121,75:81,76:te,77:J,79:se,80:ue},{35:122,75:81,76:te,77:J,79:se,80:ue},{35:123,75:81,76:te,77:J,79:se,80:ue},{35:124,75:81,76:te,77:J,79:se,80:ue},{35:125,75:81,76:te,77:J,79:se,80:ue},{14:[1,126]},{14:[1,127]},{14:[1,128]},{14:[1,129]},{1:[2,8]},t(he,[2,15]),t(K,[2,17],{21:22,19:130,22:e,23:r,24:n,26:i,28:a}),t(he,[2,37],{19:20,20:21,21:22,43:23,29:49,30:61,32:62,13:131,22:e,23:r,24:n,26:i,28:a,34:s,36:l,37:u,38:h,39:f,40:d,41:p,44:m,45:g,46:y,47:v,48:x,49:b,50:w,51:C,52:T,53:E,54:A,55:S,56:_,57:I,58:D,59:k,60:L,61:R,62:O,63:M,64:B,65:F,66:P,67:z,68:$,69:H,70:Q,71:j,72:ie,73:ne,74:le}),t(X,[2,21]),t(X,[2,22]),t(Z,[2,39]),t(Se,[2,71],{75:81,35:132,76:te,77:J,79:se,80:ue}),t(ce,[2,73]),{78:[1,133]},t(ce,[2,75]),t(ce,[2,76]),t(Z,[2,40]),t(Z,[2,41]),t(Z,[2,42]),t(Z,[2,43]),t(Z,[2,44]),t(Z,[2,45]),t(Z,[2,46]),t(Z,[2,47]),t(Z,[2,48]),t(Z,[2,49]),t(Z,[2,50]),t(Z,[2,51]),t(Z,[2,52]),t(Z,[2,53]),t(Z,[2,54]),t(Z,[2,55]),t(Z,[2,56]),t(Z,[2,57]),t(Z,[2,58]),t(Z,[2,60]),t(Z,[2,61]),t(Z,[2,62]),t(Z,[2,63]),t(Z,[2,64]),t(Z,[2,65]),t(Z,[2,66]),t(Z,[2,67]),t(Z,[2,68]),t(Z,[2,69]),t(Z,[2,70]),{31:134,42:[1,135]},{12:[1,136]},{33:[1,137]},t(ae,[2,28]),t(ae,[2,29]),t(ae,[2,30]),t(ae,[2,31]),t(ae,[2,32]),t(ae,[2,33]),t(ae,[2,34]),{1:[2,9]},{1:[2,10]},{1:[2,11]},{1:[2,12]},t(K,[2,18]),t(he,[2,38]),t(Se,[2,72]),t(ce,[2,74]),t(Z,[2,24]),t(Z,[2,35]),t(Oe,[2,25]),t(Oe,[2,26],{12:[1,138]}),t(Oe,[2,27])],defaultActions:{2:[2,1],3:[2,2],4:[2,7],5:[2,3],6:[2,4],7:[2,5],8:[2,6],74:[2,8],126:[2,9],127:[2,10],128:[2,11],129:[2,12]},parseError:o(function(Re,Ie){if(Ie.recoverable)this.trace(Re);else{var be=new Error(Re);throw be.hash=Ie,be}},"parseError"),parse:o(function(Re){var Ie=this,be=[0],W=[],de=[null],re=[],oe=this.table,V="",xe=0,q=0,pe=0,ve=2,Pe=1,_e=re.slice.call(arguments,1),we=Object.create(this.lexer),Ve={yy:{}};for(var De in this.yy)Object.prototype.hasOwnProperty.call(this.yy,De)&&(Ve.yy[De]=this.yy[De]);we.setInput(Re,Ve.yy),Ve.yy.lexer=we,Ve.yy.parser=this,typeof we.yylloc>"u"&&(we.yylloc={});var qe=we.yylloc;re.push(qe);var at=we.options&&we.options.ranges;typeof Ve.yy.parseError=="function"?this.parseError=Ve.yy.parseError:this.parseError=Object.getPrototypeOf(this).parseError;function Rt(nt){be.length=be.length-2*nt,de.length=de.length-nt,re.length=re.length-nt}o(Rt,"popStack");function st(){var nt;return nt=W.pop()||we.lex()||Pe,typeof nt!="number"&&(nt instanceof Array&&(W=nt,nt=W.pop()),nt=Ie.symbols_[nt]||nt),nt}o(st,"lex");for(var Ue,ct,We,ot,Yt,bt,Mt={},xt,ut,Et,ft;;){if(We=be[be.length-1],this.defaultActions[We]?ot=this.defaultActions[We]:((Ue===null||typeof Ue>"u")&&(Ue=st()),ot=oe[We]&&oe[We][Ue]),typeof ot>"u"||!ot.length||!ot[0]){var yt="";ft=[];for(xt in oe[We])this.terminals_[xt]&&xt>ve&&ft.push("'"+this.terminals_[xt]+"'");we.showPosition?yt="Parse error on line "+(xe+1)+`: +`+we.showPosition()+` +Expecting `+ft.join(", ")+", got '"+(this.terminals_[Ue]||Ue)+"'":yt="Parse error on line "+(xe+1)+": Unexpected "+(Ue==Pe?"end of input":"'"+(this.terminals_[Ue]||Ue)+"'"),this.parseError(yt,{text:we.match,token:this.terminals_[Ue]||Ue,line:we.yylineno,loc:qe,expected:ft})}if(ot[0]instanceof Array&&ot.length>1)throw new Error("Parse Error: multiple actions possible at state: "+We+", token: "+Ue);switch(ot[0]){case 1:be.push(Ue),de.push(we.yytext),re.push(we.yylloc),be.push(ot[1]),Ue=null,ct?(Ue=ct,ct=null):(q=we.yyleng,V=we.yytext,xe=we.yylineno,qe=we.yylloc,pe>0&&pe--);break;case 2:if(ut=this.productions_[ot[1]][1],Mt.$=de[de.length-ut],Mt._$={first_line:re[re.length-(ut||1)].first_line,last_line:re[re.length-1].last_line,first_column:re[re.length-(ut||1)].first_column,last_column:re[re.length-1].last_column},at&&(Mt._$.range=[re[re.length-(ut||1)].range[0],re[re.length-1].range[1]]),bt=this.performAction.apply(Mt,[V,q,xe,Ve.yy,ot[1],de,re].concat(_e)),typeof bt<"u")return bt;ut&&(be=be.slice(0,-1*ut*2),de=de.slice(0,-1*ut),re=re.slice(0,-1*ut)),be.push(this.productions_[ot[1]][0]),de.push(Mt.$),re.push(Mt._$),Et=oe[be[be.length-2]][be[be.length-1]],be.push(Et);break;case 3:return!0}}return!0},"parse")},ze=function(){var $e={EOF:1,parseError:o(function(Ie,be){if(this.yy.parser)this.yy.parser.parseError(Ie,be);else throw new Error(Ie)},"parseError"),setInput:o(function(Re,Ie){return this.yy=Ie||this.yy||{},this._input=Re,this._more=this._backtrack=this.done=!1,this.yylineno=this.yyleng=0,this.yytext=this.matched=this.match="",this.conditionStack=["INITIAL"],this.yylloc={first_line:1,first_column:0,last_line:1,last_column:0},this.options.ranges&&(this.yylloc.range=[0,0]),this.offset=0,this},"setInput"),input:o(function(){var Re=this._input[0];this.yytext+=Re,this.yyleng++,this.offset++,this.match+=Re,this.matched+=Re;var Ie=Re.match(/(?:\r\n?|\n).*/g);return Ie?(this.yylineno++,this.yylloc.last_line++):this.yylloc.last_column++,this.options.ranges&&this.yylloc.range[1]++,this._input=this._input.slice(1),Re},"input"),unput:o(function(Re){var Ie=Re.length,be=Re.split(/(?:\r\n?|\n)/g);this._input=Re+this._input,this.yytext=this.yytext.substr(0,this.yytext.length-Ie),this.offset-=Ie;var W=this.match.split(/(?:\r\n?|\n)/g);this.match=this.match.substr(0,this.match.length-1),this.matched=this.matched.substr(0,this.matched.length-1),be.length-1&&(this.yylineno-=be.length-1);var de=this.yylloc.range;return this.yylloc={first_line:this.yylloc.first_line,last_line:this.yylineno+1,first_column:this.yylloc.first_column,last_column:be?(be.length===W.length?this.yylloc.first_column:0)+W[W.length-be.length].length-be[0].length:this.yylloc.first_column-Ie},this.options.ranges&&(this.yylloc.range=[de[0],de[0]+this.yyleng-Ie]),this.yyleng=this.yytext.length,this},"unput"),more:o(function(){return this._more=!0,this},"more"),reject:o(function(){if(this.options.backtrack_lexer)this._backtrack=!0;else return this.parseError("Lexical error on line "+(this.yylineno+1)+`. You can only invoke reject() in the lexer when the lexer is of the backtracking persuasion (options.backtrack_lexer = true). +`+this.showPosition(),{text:"",token:null,line:this.yylineno});return this},"reject"),less:o(function(Re){this.unput(this.match.slice(Re))},"less"),pastInput:o(function(){var Re=this.matched.substr(0,this.matched.length-this.match.length);return(Re.length>20?"...":"")+Re.substr(-20).replace(/\n/g,"")},"pastInput"),upcomingInput:o(function(){var Re=this.match;return Re.length<20&&(Re+=this._input.substr(0,20-Re.length)),(Re.substr(0,20)+(Re.length>20?"...":"")).replace(/\n/g,"")},"upcomingInput"),showPosition:o(function(){var Re=this.pastInput(),Ie=new Array(Re.length+1).join("-");return Re+this.upcomingInput()+` +`+Ie+"^"},"showPosition"),test_match:o(function(Re,Ie){var be,W,de;if(this.options.backtrack_lexer&&(de={yylineno:this.yylineno,yylloc:{first_line:this.yylloc.first_line,last_line:this.last_line,first_column:this.yylloc.first_column,last_column:this.yylloc.last_column},yytext:this.yytext,match:this.match,matches:this.matches,matched:this.matched,yyleng:this.yyleng,offset:this.offset,_more:this._more,_input:this._input,yy:this.yy,conditionStack:this.conditionStack.slice(0),done:this.done},this.options.ranges&&(de.yylloc.range=this.yylloc.range.slice(0))),W=Re[0].match(/(?:\r\n?|\n).*/g),W&&(this.yylineno+=W.length),this.yylloc={first_line:this.yylloc.last_line,last_line:this.yylineno+1,first_column:this.yylloc.last_column,last_column:W?W[W.length-1].length-W[W.length-1].match(/\r?\n?/)[0].length:this.yylloc.last_column+Re[0].length},this.yytext+=Re[0],this.match+=Re[0],this.matches=Re,this.yyleng=this.yytext.length,this.options.ranges&&(this.yylloc.range=[this.offset,this.offset+=this.yyleng]),this._more=!1,this._backtrack=!1,this._input=this._input.slice(Re[0].length),this.matched+=Re[0],be=this.performAction.call(this,this.yy,this,Ie,this.conditionStack[this.conditionStack.length-1]),this.done&&this._input&&(this.done=!1),be)return be;if(this._backtrack){for(var re in de)this[re]=de[re];return!1}return!1},"test_match"),next:o(function(){if(this.done)return this.EOF;this._input||(this.done=!0);var Re,Ie,be,W;this._more||(this.yytext="",this.match="");for(var de=this._currentRules(),re=0;reIe[0].length)){if(Ie=be,W=re,this.options.backtrack_lexer){if(Re=this.test_match(be,de[re]),Re!==!1)return Re;if(this._backtrack){Ie=!1;continue}else return!1}else if(!this.options.flex)break}return Ie?(Re=this.test_match(Ie,de[W]),Re!==!1?Re:!1):this._input===""?this.EOF:this.parseError("Lexical error on line "+(this.yylineno+1)+`. Unrecognized text. +`+this.showPosition(),{text:"",token:null,line:this.yylineno})},"next"),lex:o(function(){var Ie=this.next();return Ie||this.lex()},"lex"),begin:o(function(Ie){this.conditionStack.push(Ie)},"begin"),popState:o(function(){var Ie=this.conditionStack.length-1;return Ie>0?this.conditionStack.pop():this.conditionStack[0]},"popState"),_currentRules:o(function(){return this.conditionStack.length&&this.conditionStack[this.conditionStack.length-1]?this.conditions[this.conditionStack[this.conditionStack.length-1]].rules:this.conditions.INITIAL.rules},"_currentRules"),topState:o(function(Ie){return Ie=this.conditionStack.length-1-Math.abs(Ie||0),Ie>=0?this.conditionStack[Ie]:"INITIAL"},"topState"),pushState:o(function(Ie){this.begin(Ie)},"pushState"),stateStackSize:o(function(){return this.conditionStack.length},"stateStackSize"),options:{},performAction:o(function(Ie,be,W,de){var re=de;switch(W){case 0:return 6;case 1:return 7;case 2:return 8;case 3:return 9;case 4:return 22;case 5:return 23;case 6:return this.begin("acc_title"),24;break;case 7:return this.popState(),"acc_title_value";break;case 8:return this.begin("acc_descr"),26;break;case 9:return this.popState(),"acc_descr_value";break;case 10:this.begin("acc_descr_multiline");break;case 11:this.popState();break;case 12:return"acc_descr_multiline_value";case 13:break;case 14:c;break;case 15:return 12;case 16:break;case 17:return 11;case 18:return 15;case 19:return 16;case 20:return 17;case 21:return 18;case 22:return this.begin("person_ext"),45;break;case 23:return this.begin("person"),44;break;case 24:return this.begin("system_ext_queue"),51;break;case 25:return this.begin("system_ext_db"),50;break;case 26:return this.begin("system_ext"),49;break;case 27:return this.begin("system_queue"),48;break;case 28:return this.begin("system_db"),47;break;case 29:return this.begin("system"),46;break;case 30:return this.begin("boundary"),37;break;case 31:return this.begin("enterprise_boundary"),34;break;case 32:return this.begin("system_boundary"),36;break;case 33:return this.begin("container_ext_queue"),57;break;case 34:return this.begin("container_ext_db"),56;break;case 35:return this.begin("container_ext"),55;break;case 36:return this.begin("container_queue"),54;break;case 37:return this.begin("container_db"),53;break;case 38:return this.begin("container"),52;break;case 39:return this.begin("container_boundary"),38;break;case 40:return this.begin("component_ext_queue"),63;break;case 41:return this.begin("component_ext_db"),62;break;case 42:return this.begin("component_ext"),61;break;case 43:return this.begin("component_queue"),60;break;case 44:return this.begin("component_db"),59;break;case 45:return this.begin("component"),58;break;case 46:return this.begin("node"),39;break;case 47:return this.begin("node"),39;break;case 48:return this.begin("node_l"),40;break;case 49:return this.begin("node_r"),41;break;case 50:return this.begin("rel"),64;break;case 51:return this.begin("birel"),65;break;case 52:return this.begin("rel_u"),66;break;case 53:return this.begin("rel_u"),66;break;case 54:return this.begin("rel_d"),67;break;case 55:return this.begin("rel_d"),67;break;case 56:return this.begin("rel_l"),68;break;case 57:return this.begin("rel_l"),68;break;case 58:return this.begin("rel_r"),69;break;case 59:return this.begin("rel_r"),69;break;case 60:return this.begin("rel_b"),70;break;case 61:return this.begin("rel_index"),71;break;case 62:return this.begin("update_el_style"),72;break;case 63:return this.begin("update_rel_style"),73;break;case 64:return this.begin("update_layout_config"),74;break;case 65:return"EOF_IN_STRUCT";case 66:return this.begin("attribute"),"ATTRIBUTE_EMPTY";break;case 67:this.begin("attribute");break;case 68:this.popState(),this.popState();break;case 69:return 80;case 70:break;case 71:return 80;case 72:this.begin("string");break;case 73:this.popState();break;case 74:return"STR";case 75:this.begin("string_kv");break;case 76:return this.begin("string_kv_key"),"STR_KEY";break;case 77:this.popState(),this.begin("string_kv_value");break;case 78:return"STR_VALUE";case 79:this.popState(),this.popState();break;case 80:return"STR";case 81:return"LBRACE";case 82:return"RBRACE";case 83:return"SPACE";case 84:return"EOL";case 85:return 14}},"anonymous"),rules:[/^(?:.*direction\s+TB[^\n]*)/,/^(?:.*direction\s+BT[^\n]*)/,/^(?:.*direction\s+RL[^\n]*)/,/^(?:.*direction\s+LR[^\n]*)/,/^(?:title\s[^#\n;]+)/,/^(?:accDescription\s[^#\n;]+)/,/^(?:accTitle\s*:\s*)/,/^(?:(?!\n||)*[^\n]*)/,/^(?:accDescr\s*:\s*)/,/^(?:(?!\n||)*[^\n]*)/,/^(?:accDescr\s*\{\s*)/,/^(?:[\}])/,/^(?:[^\}]*)/,/^(?:%%(?!\{)*[^\n]*(\r?\n?)+)/,/^(?:%%[^\n]*(\r?\n)*)/,/^(?:\s*(\r?\n)+)/,/^(?:\s+)/,/^(?:C4Context\b)/,/^(?:C4Container\b)/,/^(?:C4Component\b)/,/^(?:C4Dynamic\b)/,/^(?:C4Deployment\b)/,/^(?:Person_Ext\b)/,/^(?:Person\b)/,/^(?:SystemQueue_Ext\b)/,/^(?:SystemDb_Ext\b)/,/^(?:System_Ext\b)/,/^(?:SystemQueue\b)/,/^(?:SystemDb\b)/,/^(?:System\b)/,/^(?:Boundary\b)/,/^(?:Enterprise_Boundary\b)/,/^(?:System_Boundary\b)/,/^(?:ContainerQueue_Ext\b)/,/^(?:ContainerDb_Ext\b)/,/^(?:Container_Ext\b)/,/^(?:ContainerQueue\b)/,/^(?:ContainerDb\b)/,/^(?:Container\b)/,/^(?:Container_Boundary\b)/,/^(?:ComponentQueue_Ext\b)/,/^(?:ComponentDb_Ext\b)/,/^(?:Component_Ext\b)/,/^(?:ComponentQueue\b)/,/^(?:ComponentDb\b)/,/^(?:Component\b)/,/^(?:Deployment_Node\b)/,/^(?:Node\b)/,/^(?:Node_L\b)/,/^(?:Node_R\b)/,/^(?:Rel\b)/,/^(?:BiRel\b)/,/^(?:Rel_Up\b)/,/^(?:Rel_U\b)/,/^(?:Rel_Down\b)/,/^(?:Rel_D\b)/,/^(?:Rel_Left\b)/,/^(?:Rel_L\b)/,/^(?:Rel_Right\b)/,/^(?:Rel_R\b)/,/^(?:Rel_Back\b)/,/^(?:RelIndex\b)/,/^(?:UpdateElementStyle\b)/,/^(?:UpdateRelStyle\b)/,/^(?:UpdateLayoutConfig\b)/,/^(?:$)/,/^(?:[(][ ]*[,])/,/^(?:[(])/,/^(?:[)])/,/^(?:,,)/,/^(?:,)/,/^(?:[ ]*["]["])/,/^(?:[ ]*["])/,/^(?:["])/,/^(?:[^"]*)/,/^(?:[ ]*[\$])/,/^(?:[^=]*)/,/^(?:[=][ ]*["])/,/^(?:[^"]+)/,/^(?:["])/,/^(?:[^,]+)/,/^(?:\{)/,/^(?:\})/,/^(?:[\s]+)/,/^(?:[\n\r]+)/,/^(?:$)/],conditions:{acc_descr_multiline:{rules:[11,12],inclusive:!1},acc_descr:{rules:[9],inclusive:!1},acc_title:{rules:[7],inclusive:!1},string_kv_value:{rules:[78,79],inclusive:!1},string_kv_key:{rules:[77],inclusive:!1},string_kv:{rules:[76],inclusive:!1},string:{rules:[73,74],inclusive:!1},attribute:{rules:[68,69,70,71,72,75,80],inclusive:!1},update_layout_config:{rules:[65,66,67,68],inclusive:!1},update_rel_style:{rules:[65,66,67,68],inclusive:!1},update_el_style:{rules:[65,66,67,68],inclusive:!1},rel_b:{rules:[65,66,67,68],inclusive:!1},rel_r:{rules:[65,66,67,68],inclusive:!1},rel_l:{rules:[65,66,67,68],inclusive:!1},rel_d:{rules:[65,66,67,68],inclusive:!1},rel_u:{rules:[65,66,67,68],inclusive:!1},rel_bi:{rules:[],inclusive:!1},rel:{rules:[65,66,67,68],inclusive:!1},node_r:{rules:[65,66,67,68],inclusive:!1},node_l:{rules:[65,66,67,68],inclusive:!1},node:{rules:[65,66,67,68],inclusive:!1},index:{rules:[],inclusive:!1},rel_index:{rules:[65,66,67,68],inclusive:!1},component_ext_queue:{rules:[],inclusive:!1},component_ext_db:{rules:[65,66,67,68],inclusive:!1},component_ext:{rules:[65,66,67,68],inclusive:!1},component_queue:{rules:[65,66,67,68],inclusive:!1},component_db:{rules:[65,66,67,68],inclusive:!1},component:{rules:[65,66,67,68],inclusive:!1},container_boundary:{rules:[65,66,67,68],inclusive:!1},container_ext_queue:{rules:[65,66,67,68],inclusive:!1},container_ext_db:{rules:[65,66,67,68],inclusive:!1},container_ext:{rules:[65,66,67,68],inclusive:!1},container_queue:{rules:[65,66,67,68],inclusive:!1},container_db:{rules:[65,66,67,68],inclusive:!1},container:{rules:[65,66,67,68],inclusive:!1},birel:{rules:[65,66,67,68],inclusive:!1},system_boundary:{rules:[65,66,67,68],inclusive:!1},enterprise_boundary:{rules:[65,66,67,68],inclusive:!1},boundary:{rules:[65,66,67,68],inclusive:!1},system_ext_queue:{rules:[65,66,67,68],inclusive:!1},system_ext_db:{rules:[65,66,67,68],inclusive:!1},system_ext:{rules:[65,66,67,68],inclusive:!1},system_queue:{rules:[65,66,67,68],inclusive:!1},system_db:{rules:[65,66,67,68],inclusive:!1},system:{rules:[65,66,67,68],inclusive:!1},person_ext:{rules:[65,66,67,68],inclusive:!1},person:{rules:[65,66,67,68],inclusive:!1},INITIAL:{rules:[0,1,2,3,4,5,6,8,10,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,81,82,83,84,85],inclusive:!0}}};return $e}();ge.lexer=ze;function He(){this.yy={}}return o(He,"Parser"),He.prototype=ge,ge.Parser=He,new He}();Ty.parser=Ty;JF=Ty});var zC,Gn,s0=N(()=>{"use strict";zC=o((t,e,{depth:r=2,clobber:n=!1}={})=>{let i={depth:r,clobber:n};return Array.isArray(e)&&!Array.isArray(t)?(e.forEach(a=>zC(t,a,i)),t):Array.isArray(e)&&Array.isArray(t)?(e.forEach(a=>{t.includes(a)||t.push(a)}),t):t===void 0||r<=0?t!=null&&typeof t=="object"&&typeof e=="object"?Object.assign(t,e):e:(e!==void 0&&typeof t=="object"&&typeof e=="object"&&Object.keys(e).forEach(a=>{typeof e[a]=="object"&&(t[a]===void 0||typeof t[a]=="object")?(t[a]===void 0&&(t[a]=Array.isArray(e[a])?[]:{}),t[a]=zC(t[a],e[a],{depth:r-1,clobber:n})):(n||typeof t[a]!="object"&&typeof e[a]!="object")&&(t[a]=e[a])}),t)},"assignWithDepth"),Gn=zC});var G4,e$,t$=N(()=>{"use strict";G4={min:{r:0,g:0,b:0,s:0,l:0,a:0},max:{r:255,g:255,b:255,h:360,s:100,l:100,a:1},clamp:{r:o(t=>t>=255?255:t<0?0:t,"r"),g:o(t=>t>=255?255:t<0?0:t,"g"),b:o(t=>t>=255?255:t<0?0:t,"b"),h:o(t=>t%360,"h"),s:o(t=>t>=100?100:t<0?0:t,"s"),l:o(t=>t>=100?100:t<0?0:t,"l"),a:o(t=>t>=1?1:t<0?0:t,"a")},toLinear:o(t=>{let e=t/255;return t>.03928?Math.pow((e+.055)/1.055,2.4):e/12.92},"toLinear"),hue2rgb:o((t,e,r)=>(r<0&&(r+=1),r>1&&(r-=1),r<.16666666666666666?t+(e-t)*6*r:r<.5?e:r<.6666666666666666?t+(e-t)*(.6666666666666666-r)*6:t),"hue2rgb"),hsl2rgb:o(({h:t,s:e,l:r},n)=>{if(!e)return r*2.55;t/=360,e/=100,r/=100;let i=r<.5?r*(1+e):r+e-r*e,a=2*r-i;switch(n){case"r":return G4.hue2rgb(a,i,t+.3333333333333333)*255;case"g":return G4.hue2rgb(a,i,t)*255;case"b":return G4.hue2rgb(a,i,t-.3333333333333333)*255}},"hsl2rgb"),rgb2hsl:o(({r:t,g:e,b:r},n)=>{t/=255,e/=255,r/=255;let i=Math.max(t,e,r),a=Math.min(t,e,r),s=(i+a)/2;if(n==="l")return s*100;if(i===a)return 0;let l=i-a,u=s>.5?l/(2-i-a):l/(i+a);if(n==="s")return u*100;switch(i){case t:return((e-r)/l+(e{"use strict";fxe={clamp:o((t,e,r)=>e>r?Math.min(e,Math.max(r,t)):Math.min(r,Math.max(e,t)),"clamp"),round:o(t=>Math.round(t*1e10)/1e10,"round")},r$=fxe});var dxe,i$,a$=N(()=>{"use strict";dxe={dec2hex:o(t=>{let e=Math.round(t).toString(16);return e.length>1?e:`0${e}`},"dec2hex")},i$=dxe});var pxe,jt,Wl=N(()=>{"use strict";t$();n$();a$();pxe={channel:e$,lang:r$,unit:i$},jt=pxe});var ru,Ii,ky=N(()=>{"use strict";Wl();ru={};for(let t=0;t<=255;t++)ru[t]=jt.unit.dec2hex(t);Ii={ALL:0,RGB:1,HSL:2}});var GC,s$,o$=N(()=>{"use strict";ky();GC=class{static{o(this,"Type")}constructor(){this.type=Ii.ALL}get(){return this.type}set(e){if(this.type&&this.type!==e)throw new Error("Cannot change both RGB and HSL channels at the same time");this.type=e}reset(){this.type=Ii.ALL}is(e){return this.type===e}},s$=GC});var VC,l$,c$=N(()=>{"use strict";Wl();o$();ky();VC=class{static{o(this,"Channels")}constructor(e,r){this.color=r,this.changed=!1,this.data=e,this.type=new s$}set(e,r){return this.color=r,this.changed=!1,this.data=e,this.type.type=Ii.ALL,this}_ensureHSL(){let e=this.data,{h:r,s:n,l:i}=e;r===void 0&&(e.h=jt.channel.rgb2hsl(e,"h")),n===void 0&&(e.s=jt.channel.rgb2hsl(e,"s")),i===void 0&&(e.l=jt.channel.rgb2hsl(e,"l"))}_ensureRGB(){let e=this.data,{r,g:n,b:i}=e;r===void 0&&(e.r=jt.channel.hsl2rgb(e,"r")),n===void 0&&(e.g=jt.channel.hsl2rgb(e,"g")),i===void 0&&(e.b=jt.channel.hsl2rgb(e,"b"))}get r(){let e=this.data,r=e.r;return!this.type.is(Ii.HSL)&&r!==void 0?r:(this._ensureHSL(),jt.channel.hsl2rgb(e,"r"))}get g(){let e=this.data,r=e.g;return!this.type.is(Ii.HSL)&&r!==void 0?r:(this._ensureHSL(),jt.channel.hsl2rgb(e,"g"))}get b(){let e=this.data,r=e.b;return!this.type.is(Ii.HSL)&&r!==void 0?r:(this._ensureHSL(),jt.channel.hsl2rgb(e,"b"))}get h(){let e=this.data,r=e.h;return!this.type.is(Ii.RGB)&&r!==void 0?r:(this._ensureRGB(),jt.channel.rgb2hsl(e,"h"))}get s(){let e=this.data,r=e.s;return!this.type.is(Ii.RGB)&&r!==void 0?r:(this._ensureRGB(),jt.channel.rgb2hsl(e,"s"))}get l(){let e=this.data,r=e.l;return!this.type.is(Ii.RGB)&&r!==void 0?r:(this._ensureRGB(),jt.channel.rgb2hsl(e,"l"))}get a(){return this.data.a}set r(e){this.type.set(Ii.RGB),this.changed=!0,this.data.r=e}set g(e){this.type.set(Ii.RGB),this.changed=!0,this.data.g=e}set b(e){this.type.set(Ii.RGB),this.changed=!0,this.data.b=e}set h(e){this.type.set(Ii.HSL),this.changed=!0,this.data.h=e}set s(e){this.type.set(Ii.HSL),this.changed=!0,this.data.s=e}set l(e){this.type.set(Ii.HSL),this.changed=!0,this.data.l=e}set a(e){this.changed=!0,this.data.a=e}},l$=VC});var mxe,ih,Ey=N(()=>{"use strict";c$();mxe=new l$({r:0,g:0,b:0,a:0},"transparent"),ih=mxe});var u$,jf,UC=N(()=>{"use strict";Ey();ky();u$={re:/^#((?:[a-f0-9]{2}){2,4}|[a-f0-9]{3})$/i,parse:o(t=>{if(t.charCodeAt(0)!==35)return;let e=t.match(u$.re);if(!e)return;let r=e[1],n=parseInt(r,16),i=r.length,a=i%4===0,s=i>4,l=s?1:17,u=s?8:4,h=a?0:-1,f=s?255:15;return ih.set({r:(n>>u*(h+3)&f)*l,g:(n>>u*(h+2)&f)*l,b:(n>>u*(h+1)&f)*l,a:a?(n&f)*l/255:1},t)},"parse"),stringify:o(t=>{let{r:e,g:r,b:n,a:i}=t;return i<1?`#${ru[Math.round(e)]}${ru[Math.round(r)]}${ru[Math.round(n)]}${ru[Math.round(i*255)]}`:`#${ru[Math.round(e)]}${ru[Math.round(r)]}${ru[Math.round(n)]}`},"stringify")},jf=u$});var V4,Sy,h$=N(()=>{"use strict";Wl();Ey();V4={re:/^hsla?\(\s*?(-?(?:\d+(?:\.\d+)?|(?:\.\d+))(?:e-?\d+)?(?:deg|grad|rad|turn)?)\s*?(?:,|\s)\s*?(-?(?:\d+(?:\.\d+)?|(?:\.\d+))(?:e-?\d+)?%)\s*?(?:,|\s)\s*?(-?(?:\d+(?:\.\d+)?|(?:\.\d+))(?:e-?\d+)?%)(?:\s*?(?:,|\/)\s*?\+?(-?(?:\d+(?:\.\d+)?|(?:\.\d+))(?:e-?\d+)?(%)?))?\s*?\)$/i,hueRe:/^(.+?)(deg|grad|rad|turn)$/i,_hue2deg:o(t=>{let e=t.match(V4.hueRe);if(e){let[,r,n]=e;switch(n){case"grad":return jt.channel.clamp.h(parseFloat(r)*.9);case"rad":return jt.channel.clamp.h(parseFloat(r)*180/Math.PI);case"turn":return jt.channel.clamp.h(parseFloat(r)*360)}}return jt.channel.clamp.h(parseFloat(t))},"_hue2deg"),parse:o(t=>{let e=t.charCodeAt(0);if(e!==104&&e!==72)return;let r=t.match(V4.re);if(!r)return;let[,n,i,a,s,l]=r;return ih.set({h:V4._hue2deg(n),s:jt.channel.clamp.s(parseFloat(i)),l:jt.channel.clamp.l(parseFloat(a)),a:s?jt.channel.clamp.a(l?parseFloat(s)/100:parseFloat(s)):1},t)},"parse"),stringify:o(t=>{let{h:e,s:r,l:n,a:i}=t;return i<1?`hsla(${jt.lang.round(e)}, ${jt.lang.round(r)}%, ${jt.lang.round(n)}%, ${i})`:`hsl(${jt.lang.round(e)}, ${jt.lang.round(r)}%, ${jt.lang.round(n)}%)`},"stringify")},Sy=V4});var U4,HC,f$=N(()=>{"use strict";UC();U4={colors:{aliceblue:"#f0f8ff",antiquewhite:"#faebd7",aqua:"#00ffff",aquamarine:"#7fffd4",azure:"#f0ffff",beige:"#f5f5dc",bisque:"#ffe4c4",black:"#000000",blanchedalmond:"#ffebcd",blue:"#0000ff",blueviolet:"#8a2be2",brown:"#a52a2a",burlywood:"#deb887",cadetblue:"#5f9ea0",chartreuse:"#7fff00",chocolate:"#d2691e",coral:"#ff7f50",cornflowerblue:"#6495ed",cornsilk:"#fff8dc",crimson:"#dc143c",cyanaqua:"#00ffff",darkblue:"#00008b",darkcyan:"#008b8b",darkgoldenrod:"#b8860b",darkgray:"#a9a9a9",darkgreen:"#006400",darkgrey:"#a9a9a9",darkkhaki:"#bdb76b",darkmagenta:"#8b008b",darkolivegreen:"#556b2f",darkorange:"#ff8c00",darkorchid:"#9932cc",darkred:"#8b0000",darksalmon:"#e9967a",darkseagreen:"#8fbc8f",darkslateblue:"#483d8b",darkslategray:"#2f4f4f",darkslategrey:"#2f4f4f",darkturquoise:"#00ced1",darkviolet:"#9400d3",deeppink:"#ff1493",deepskyblue:"#00bfff",dimgray:"#696969",dimgrey:"#696969",dodgerblue:"#1e90ff",firebrick:"#b22222",floralwhite:"#fffaf0",forestgreen:"#228b22",fuchsia:"#ff00ff",gainsboro:"#dcdcdc",ghostwhite:"#f8f8ff",gold:"#ffd700",goldenrod:"#daa520",gray:"#808080",green:"#008000",greenyellow:"#adff2f",grey:"#808080",honeydew:"#f0fff0",hotpink:"#ff69b4",indianred:"#cd5c5c",indigo:"#4b0082",ivory:"#fffff0",khaki:"#f0e68c",lavender:"#e6e6fa",lavenderblush:"#fff0f5",lawngreen:"#7cfc00",lemonchiffon:"#fffacd",lightblue:"#add8e6",lightcoral:"#f08080",lightcyan:"#e0ffff",lightgoldenrodyellow:"#fafad2",lightgray:"#d3d3d3",lightgreen:"#90ee90",lightgrey:"#d3d3d3",lightpink:"#ffb6c1",lightsalmon:"#ffa07a",lightseagreen:"#20b2aa",lightskyblue:"#87cefa",lightslategray:"#778899",lightslategrey:"#778899",lightsteelblue:"#b0c4de",lightyellow:"#ffffe0",lime:"#00ff00",limegreen:"#32cd32",linen:"#faf0e6",magenta:"#ff00ff",maroon:"#800000",mediumaquamarine:"#66cdaa",mediumblue:"#0000cd",mediumorchid:"#ba55d3",mediumpurple:"#9370db",mediumseagreen:"#3cb371",mediumslateblue:"#7b68ee",mediumspringgreen:"#00fa9a",mediumturquoise:"#48d1cc",mediumvioletred:"#c71585",midnightblue:"#191970",mintcream:"#f5fffa",mistyrose:"#ffe4e1",moccasin:"#ffe4b5",navajowhite:"#ffdead",navy:"#000080",oldlace:"#fdf5e6",olive:"#808000",olivedrab:"#6b8e23",orange:"#ffa500",orangered:"#ff4500",orchid:"#da70d6",palegoldenrod:"#eee8aa",palegreen:"#98fb98",paleturquoise:"#afeeee",palevioletred:"#db7093",papayawhip:"#ffefd5",peachpuff:"#ffdab9",peru:"#cd853f",pink:"#ffc0cb",plum:"#dda0dd",powderblue:"#b0e0e6",purple:"#800080",rebeccapurple:"#663399",red:"#ff0000",rosybrown:"#bc8f8f",royalblue:"#4169e1",saddlebrown:"#8b4513",salmon:"#fa8072",sandybrown:"#f4a460",seagreen:"#2e8b57",seashell:"#fff5ee",sienna:"#a0522d",silver:"#c0c0c0",skyblue:"#87ceeb",slateblue:"#6a5acd",slategray:"#708090",slategrey:"#708090",snow:"#fffafa",springgreen:"#00ff7f",tan:"#d2b48c",teal:"#008080",thistle:"#d8bfd8",transparent:"#00000000",turquoise:"#40e0d0",violet:"#ee82ee",wheat:"#f5deb3",white:"#ffffff",whitesmoke:"#f5f5f5",yellow:"#ffff00",yellowgreen:"#9acd32"},parse:o(t=>{t=t.toLowerCase();let e=U4.colors[t];if(e)return jf.parse(e)},"parse"),stringify:o(t=>{let e=jf.stringify(t);for(let r in U4.colors)if(U4.colors[r]===e)return r},"stringify")},HC=U4});var d$,Cy,p$=N(()=>{"use strict";Wl();Ey();d$={re:/^rgba?\(\s*?(-?(?:\d+(?:\.\d+)?|(?:\.\d+))(?:e\d+)?(%?))\s*?(?:,|\s)\s*?(-?(?:\d+(?:\.\d+)?|(?:\.\d+))(?:e\d+)?(%?))\s*?(?:,|\s)\s*?(-?(?:\d+(?:\.\d+)?|(?:\.\d+))(?:e\d+)?(%?))(?:\s*?(?:,|\/)\s*?\+?(-?(?:\d+(?:\.\d+)?|(?:\.\d+))(?:e\d+)?(%?)))?\s*?\)$/i,parse:o(t=>{let e=t.charCodeAt(0);if(e!==114&&e!==82)return;let r=t.match(d$.re);if(!r)return;let[,n,i,a,s,l,u,h,f]=r;return ih.set({r:jt.channel.clamp.r(i?parseFloat(n)*2.55:parseFloat(n)),g:jt.channel.clamp.g(s?parseFloat(a)*2.55:parseFloat(a)),b:jt.channel.clamp.b(u?parseFloat(l)*2.55:parseFloat(l)),a:h?jt.channel.clamp.a(f?parseFloat(h)/100:parseFloat(h)):1},t)},"parse"),stringify:o(t=>{let{r:e,g:r,b:n,a:i}=t;return i<1?`rgba(${jt.lang.round(e)}, ${jt.lang.round(r)}, ${jt.lang.round(n)}, ${jt.lang.round(i)})`:`rgb(${jt.lang.round(e)}, ${jt.lang.round(r)}, ${jt.lang.round(n)})`},"stringify")},Cy=d$});var gxe,Oi,nu=N(()=>{"use strict";UC();h$();f$();p$();ky();gxe={format:{keyword:HC,hex:jf,rgb:Cy,rgba:Cy,hsl:Sy,hsla:Sy},parse:o(t=>{if(typeof t!="string")return t;let e=jf.parse(t)||Cy.parse(t)||Sy.parse(t)||HC.parse(t);if(e)return e;throw new Error(`Unsupported color format: "${t}"`)},"parse"),stringify:o(t=>!t.changed&&t.color?t.color:t.type.is(Ii.HSL)||t.data.r===void 0?Sy.stringify(t):t.a<1||!Number.isInteger(t.r)||!Number.isInteger(t.g)||!Number.isInteger(t.b)?Cy.stringify(t):jf.stringify(t),"stringify")},Oi=gxe});var yxe,H4,WC=N(()=>{"use strict";Wl();nu();yxe=o((t,e)=>{let r=Oi.parse(t);for(let n in e)r[n]=jt.channel.clamp[n](e[n]);return Oi.stringify(r)},"change"),H4=yxe});var vxe,qa,qC=N(()=>{"use strict";Wl();Ey();nu();WC();vxe=o((t,e,r=0,n=1)=>{if(typeof t!="number")return H4(t,{a:e});let i=ih.set({r:jt.channel.clamp.r(t),g:jt.channel.clamp.g(e),b:jt.channel.clamp.b(r),a:jt.channel.clamp.a(n)});return Oi.stringify(i)},"rgba"),qa=vxe});var xxe,Kf,m$=N(()=>{"use strict";Wl();nu();xxe=o((t,e)=>jt.lang.round(Oi.parse(t)[e]),"channel"),Kf=xxe});var bxe,g$,y$=N(()=>{"use strict";Wl();nu();bxe=o(t=>{let{r:e,g:r,b:n}=Oi.parse(t),i=.2126*jt.channel.toLinear(e)+.7152*jt.channel.toLinear(r)+.0722*jt.channel.toLinear(n);return jt.lang.round(i)},"luminance"),g$=bxe});var wxe,v$,x$=N(()=>{"use strict";y$();wxe=o(t=>g$(t)>=.5,"isLight"),v$=wxe});var Txe,ca,b$=N(()=>{"use strict";x$();Txe=o(t=>!v$(t),"isDark"),ca=Txe});var kxe,W4,YC=N(()=>{"use strict";Wl();nu();kxe=o((t,e,r)=>{let n=Oi.parse(t),i=n[e],a=jt.channel.clamp[e](i+r);return i!==a&&(n[e]=a),Oi.stringify(n)},"adjustChannel"),W4=kxe});var Exe,Dt,w$=N(()=>{"use strict";YC();Exe=o((t,e)=>W4(t,"l",e),"lighten"),Dt=Exe});var Sxe,Ot,T$=N(()=>{"use strict";YC();Sxe=o((t,e)=>W4(t,"l",-e),"darken"),Ot=Sxe});var Cxe,Me,k$=N(()=>{"use strict";nu();WC();Cxe=o((t,e)=>{let r=Oi.parse(t),n={};for(let i in e)e[i]&&(n[i]=r[i]+e[i]);return H4(t,n)},"adjust"),Me=Cxe});var Axe,E$,S$=N(()=>{"use strict";nu();qC();Axe=o((t,e,r=50)=>{let{r:n,g:i,b:a,a:s}=Oi.parse(t),{r:l,g:u,b:h,a:f}=Oi.parse(e),d=r/100,p=d*2-1,m=s-f,y=((p*m===-1?p:(p+m)/(1+p*m))+1)/2,v=1-y,x=n*y+l*v,b=i*y+u*v,w=a*y+h*v,C=s*d+f*(1-d);return qa(x,b,w,C)},"mix"),E$=Axe});var _xe,wt,C$=N(()=>{"use strict";nu();S$();_xe=o((t,e=100)=>{let r=Oi.parse(t);return r.r=255-r.r,r.g=255-r.g,r.b=255-r.b,E$(r,t,e)},"invert"),wt=_xe});var A$=N(()=>{"use strict";qC();m$();b$();w$();T$();k$();C$()});var Ys=N(()=>{"use strict";A$()});var ah,sh,Ay=N(()=>{"use strict";ah="#ffffff",sh="#f2f2f2"});var Ti,o0=N(()=>{"use strict";Ys();Ti=o((t,e)=>e?Me(t,{s:-40,l:10}):Me(t,{s:-40,l:-10}),"mkBorder")});var jC,_$,D$=N(()=>{"use strict";Ys();Ay();o0();jC=class{static{o(this,"Theme")}constructor(){this.background="#f4f4f4",this.primaryColor="#fff4dd",this.noteBkgColor="#fff5ad",this.noteTextColor="#333",this.THEME_COLOR_LIMIT=12,this.fontFamily='"trebuchet ms", verdana, arial, sans-serif',this.fontSize="16px"}updateColors(){if(this.primaryTextColor=this.primaryTextColor||(this.darkMode?"#eee":"#333"),this.secondaryColor=this.secondaryColor||Me(this.primaryColor,{h:-120}),this.tertiaryColor=this.tertiaryColor||Me(this.primaryColor,{h:180,l:5}),this.primaryBorderColor=this.primaryBorderColor||Ti(this.primaryColor,this.darkMode),this.secondaryBorderColor=this.secondaryBorderColor||Ti(this.secondaryColor,this.darkMode),this.tertiaryBorderColor=this.tertiaryBorderColor||Ti(this.tertiaryColor,this.darkMode),this.noteBorderColor=this.noteBorderColor||Ti(this.noteBkgColor,this.darkMode),this.noteBkgColor=this.noteBkgColor||"#fff5ad",this.noteTextColor=this.noteTextColor||"#333",this.secondaryTextColor=this.secondaryTextColor||wt(this.secondaryColor),this.tertiaryTextColor=this.tertiaryTextColor||wt(this.tertiaryColor),this.lineColor=this.lineColor||wt(this.background),this.arrowheadColor=this.arrowheadColor||wt(this.background),this.textColor=this.textColor||this.primaryTextColor,this.border2=this.border2||this.tertiaryBorderColor,this.nodeBkg=this.nodeBkg||this.primaryColor,this.mainBkg=this.mainBkg||this.primaryColor,this.nodeBorder=this.nodeBorder||this.primaryBorderColor,this.clusterBkg=this.clusterBkg||this.tertiaryColor,this.clusterBorder=this.clusterBorder||this.tertiaryBorderColor,this.defaultLinkColor=this.defaultLinkColor||this.lineColor,this.titleColor=this.titleColor||this.tertiaryTextColor,this.edgeLabelBackground=this.edgeLabelBackground||(this.darkMode?Ot(this.secondaryColor,30):this.secondaryColor),this.nodeTextColor=this.nodeTextColor||this.primaryTextColor,this.actorBorder=this.actorBorder||this.primaryBorderColor,this.actorBkg=this.actorBkg||this.mainBkg,this.actorTextColor=this.actorTextColor||this.primaryTextColor,this.actorLineColor=this.actorLineColor||this.actorBorder,this.labelBoxBkgColor=this.labelBoxBkgColor||this.actorBkg,this.signalColor=this.signalColor||this.textColor,this.signalTextColor=this.signalTextColor||this.textColor,this.labelBoxBorderColor=this.labelBoxBorderColor||this.actorBorder,this.labelTextColor=this.labelTextColor||this.actorTextColor,this.loopTextColor=this.loopTextColor||this.actorTextColor,this.activationBorderColor=this.activationBorderColor||Ot(this.secondaryColor,10),this.activationBkgColor=this.activationBkgColor||this.secondaryColor,this.sequenceNumberColor=this.sequenceNumberColor||wt(this.lineColor),this.sectionBkgColor=this.sectionBkgColor||this.tertiaryColor,this.altSectionBkgColor=this.altSectionBkgColor||"white",this.sectionBkgColor=this.sectionBkgColor||this.secondaryColor,this.sectionBkgColor2=this.sectionBkgColor2||this.primaryColor,this.excludeBkgColor=this.excludeBkgColor||"#eeeeee",this.taskBorderColor=this.taskBorderColor||this.primaryBorderColor,this.taskBkgColor=this.taskBkgColor||this.primaryColor,this.activeTaskBorderColor=this.activeTaskBorderColor||this.primaryColor,this.activeTaskBkgColor=this.activeTaskBkgColor||Dt(this.primaryColor,23),this.gridColor=this.gridColor||"lightgrey",this.doneTaskBkgColor=this.doneTaskBkgColor||"lightgrey",this.doneTaskBorderColor=this.doneTaskBorderColor||"grey",this.critBorderColor=this.critBorderColor||"#ff8888",this.critBkgColor=this.critBkgColor||"red",this.todayLineColor=this.todayLineColor||"red",this.taskTextColor=this.taskTextColor||this.textColor,this.taskTextOutsideColor=this.taskTextOutsideColor||this.textColor,this.taskTextLightColor=this.taskTextLightColor||this.textColor,this.taskTextColor=this.taskTextColor||this.primaryTextColor,this.taskTextDarkColor=this.taskTextDarkColor||this.textColor,this.taskTextClickableColor=this.taskTextClickableColor||"#003163",this.personBorder=this.personBorder||this.primaryBorderColor,this.personBkg=this.personBkg||this.mainBkg,this.darkMode?(this.rowOdd=this.rowOdd||Ot(this.mainBkg,5)||"#ffffff",this.rowEven=this.rowEven||Ot(this.mainBkg,10)):(this.rowOdd=this.rowOdd||Dt(this.mainBkg,75)||"#ffffff",this.rowEven=this.rowEven||Dt(this.mainBkg,5)),this.transitionColor=this.transitionColor||this.lineColor,this.transitionLabelColor=this.transitionLabelColor||this.textColor,this.stateLabelColor=this.stateLabelColor||this.stateBkg||this.primaryTextColor,this.stateBkg=this.stateBkg||this.mainBkg,this.labelBackgroundColor=this.labelBackgroundColor||this.stateBkg,this.compositeBackground=this.compositeBackground||this.background||this.tertiaryColor,this.altBackground=this.altBackground||this.tertiaryColor,this.compositeTitleBackground=this.compositeTitleBackground||this.mainBkg,this.compositeBorder=this.compositeBorder||this.nodeBorder,this.innerEndBackground=this.nodeBorder,this.errorBkgColor=this.errorBkgColor||this.tertiaryColor,this.errorTextColor=this.errorTextColor||this.tertiaryTextColor,this.transitionColor=this.transitionColor||this.lineColor,this.specialStateColor=this.lineColor,this.cScale0=this.cScale0||this.primaryColor,this.cScale1=this.cScale1||this.secondaryColor,this.cScale2=this.cScale2||this.tertiaryColor,this.cScale3=this.cScale3||Me(this.primaryColor,{h:30}),this.cScale4=this.cScale4||Me(this.primaryColor,{h:60}),this.cScale5=this.cScale5||Me(this.primaryColor,{h:90}),this.cScale6=this.cScale6||Me(this.primaryColor,{h:120}),this.cScale7=this.cScale7||Me(this.primaryColor,{h:150}),this.cScale8=this.cScale8||Me(this.primaryColor,{h:210,l:150}),this.cScale9=this.cScale9||Me(this.primaryColor,{h:270}),this.cScale10=this.cScale10||Me(this.primaryColor,{h:300}),this.cScale11=this.cScale11||Me(this.primaryColor,{h:330}),this.darkMode)for(let r=0;r{this[n]=e[n]}),this.updateColors(),r.forEach(n=>{this[n]=e[n]})}},_$=o(t=>{let e=new jC;return e.calculate(t),e},"getThemeVariables")});var KC,L$,R$=N(()=>{"use strict";Ys();o0();KC=class{static{o(this,"Theme")}constructor(){this.background="#333",this.primaryColor="#1f2020",this.secondaryColor=Dt(this.primaryColor,16),this.tertiaryColor=Me(this.primaryColor,{h:-160}),this.primaryBorderColor=wt(this.background),this.secondaryBorderColor=Ti(this.secondaryColor,this.darkMode),this.tertiaryBorderColor=Ti(this.tertiaryColor,this.darkMode),this.primaryTextColor=wt(this.primaryColor),this.secondaryTextColor=wt(this.secondaryColor),this.tertiaryTextColor=wt(this.tertiaryColor),this.lineColor=wt(this.background),this.textColor=wt(this.background),this.mainBkg="#1f2020",this.secondBkg="calculated",this.mainContrastColor="lightgrey",this.darkTextColor=Dt(wt("#323D47"),10),this.lineColor="calculated",this.border1="#ccc",this.border2=qa(255,255,255,.25),this.arrowheadColor="calculated",this.fontFamily='"trebuchet ms", verdana, arial, sans-serif',this.fontSize="16px",this.labelBackground="#181818",this.textColor="#ccc",this.THEME_COLOR_LIMIT=12,this.nodeBkg="calculated",this.nodeBorder="calculated",this.clusterBkg="calculated",this.clusterBorder="calculated",this.defaultLinkColor="calculated",this.titleColor="#F9FFFE",this.edgeLabelBackground="calculated",this.actorBorder="calculated",this.actorBkg="calculated",this.actorTextColor="calculated",this.actorLineColor="calculated",this.signalColor="calculated",this.signalTextColor="calculated",this.labelBoxBkgColor="calculated",this.labelBoxBorderColor="calculated",this.labelTextColor="calculated",this.loopTextColor="calculated",this.noteBorderColor="calculated",this.noteBkgColor="#fff5ad",this.noteTextColor="calculated",this.activationBorderColor="calculated",this.activationBkgColor="calculated",this.sequenceNumberColor="black",this.sectionBkgColor=Ot("#EAE8D9",30),this.altSectionBkgColor="calculated",this.sectionBkgColor2="#EAE8D9",this.excludeBkgColor=Ot(this.sectionBkgColor,10),this.taskBorderColor=qa(255,255,255,70),this.taskBkgColor="calculated",this.taskTextColor="calculated",this.taskTextLightColor="calculated",this.taskTextOutsideColor="calculated",this.taskTextClickableColor="#003163",this.activeTaskBorderColor=qa(255,255,255,50),this.activeTaskBkgColor="#81B1DB",this.gridColor="calculated",this.doneTaskBkgColor="calculated",this.doneTaskBorderColor="grey",this.critBorderColor="#E83737",this.critBkgColor="#E83737",this.taskTextDarkColor="calculated",this.todayLineColor="#DB5757",this.personBorder=this.primaryBorderColor,this.personBkg=this.mainBkg,this.archEdgeColor="calculated",this.archEdgeArrowColor="calculated",this.archEdgeWidth="3",this.archGroupBorderColor=this.primaryBorderColor,this.archGroupBorderWidth="2px",this.rowOdd=this.rowOdd||Dt(this.mainBkg,5)||"#ffffff",this.rowEven=this.rowEven||Ot(this.mainBkg,10),this.labelColor="calculated",this.errorBkgColor="#a44141",this.errorTextColor="#ddd"}updateColors(){this.secondBkg=Dt(this.mainBkg,16),this.lineColor=this.mainContrastColor,this.arrowheadColor=this.mainContrastColor,this.nodeBkg=this.mainBkg,this.nodeBorder=this.border1,this.clusterBkg=this.secondBkg,this.clusterBorder=this.border2,this.defaultLinkColor=this.lineColor,this.edgeLabelBackground=Dt(this.labelBackground,25),this.actorBorder=this.border1,this.actorBkg=this.mainBkg,this.actorTextColor=this.mainContrastColor,this.actorLineColor=this.actorBorder,this.signalColor=this.mainContrastColor,this.signalTextColor=this.mainContrastColor,this.labelBoxBkgColor=this.actorBkg,this.labelBoxBorderColor=this.actorBorder,this.labelTextColor=this.mainContrastColor,this.loopTextColor=this.mainContrastColor,this.noteBorderColor=this.secondaryBorderColor,this.noteBkgColor=this.secondBkg,this.noteTextColor=this.secondaryTextColor,this.activationBorderColor=this.border1,this.activationBkgColor=this.secondBkg,this.altSectionBkgColor=this.background,this.taskBkgColor=Dt(this.mainBkg,23),this.taskTextColor=this.darkTextColor,this.taskTextLightColor=this.mainContrastColor,this.taskTextOutsideColor=this.taskTextLightColor,this.gridColor=this.mainContrastColor,this.doneTaskBkgColor=this.mainContrastColor,this.taskTextDarkColor=this.darkTextColor,this.archEdgeColor=this.lineColor,this.archEdgeArrowColor=this.lineColor,this.transitionColor=this.transitionColor||this.lineColor,this.transitionLabelColor=this.transitionLabelColor||this.textColor,this.stateLabelColor=this.stateLabelColor||this.stateBkg||this.primaryTextColor,this.stateBkg=this.stateBkg||this.mainBkg,this.labelBackgroundColor=this.labelBackgroundColor||this.stateBkg,this.compositeBackground=this.compositeBackground||this.background||this.tertiaryColor,this.altBackground=this.altBackground||"#555",this.compositeTitleBackground=this.compositeTitleBackground||this.mainBkg,this.compositeBorder=this.compositeBorder||this.nodeBorder,this.innerEndBackground=this.primaryBorderColor,this.specialStateColor="#f4f4f4",this.errorBkgColor=this.errorBkgColor||this.tertiaryColor,this.errorTextColor=this.errorTextColor||this.tertiaryTextColor,this.fillType0=this.primaryColor,this.fillType1=this.secondaryColor,this.fillType2=Me(this.primaryColor,{h:64}),this.fillType3=Me(this.secondaryColor,{h:64}),this.fillType4=Me(this.primaryColor,{h:-64}),this.fillType5=Me(this.secondaryColor,{h:-64}),this.fillType6=Me(this.primaryColor,{h:128}),this.fillType7=Me(this.secondaryColor,{h:128}),this.cScale1=this.cScale1||"#0b0000",this.cScale2=this.cScale2||"#4d1037",this.cScale3=this.cScale3||"#3f5258",this.cScale4=this.cScale4||"#4f2f1b",this.cScale5=this.cScale5||"#6e0a0a",this.cScale6=this.cScale6||"#3b0048",this.cScale7=this.cScale7||"#995a01",this.cScale8=this.cScale8||"#154706",this.cScale9=this.cScale9||"#161722",this.cScale10=this.cScale10||"#00296f",this.cScale11=this.cScale11||"#01629c",this.cScale12=this.cScale12||"#010029",this.cScale0=this.cScale0||this.primaryColor,this.cScale1=this.cScale1||this.secondaryColor,this.cScale2=this.cScale2||this.tertiaryColor,this.cScale3=this.cScale3||Me(this.primaryColor,{h:30}),this.cScale4=this.cScale4||Me(this.primaryColor,{h:60}),this.cScale5=this.cScale5||Me(this.primaryColor,{h:90}),this.cScale6=this.cScale6||Me(this.primaryColor,{h:120}),this.cScale7=this.cScale7||Me(this.primaryColor,{h:150}),this.cScale8=this.cScale8||Me(this.primaryColor,{h:210}),this.cScale9=this.cScale9||Me(this.primaryColor,{h:270}),this.cScale10=this.cScale10||Me(this.primaryColor,{h:300}),this.cScale11=this.cScale11||Me(this.primaryColor,{h:330});for(let e=0;e{this[n]=e[n]}),this.updateColors(),r.forEach(n=>{this[n]=e[n]})}},L$=o(t=>{let e=new KC;return e.calculate(t),e},"getThemeVariables")});var QC,oh,_y=N(()=>{"use strict";Ys();o0();Ay();QC=class{static{o(this,"Theme")}constructor(){this.background="#f4f4f4",this.primaryColor="#ECECFF",this.secondaryColor=Me(this.primaryColor,{h:120}),this.secondaryColor="#ffffde",this.tertiaryColor=Me(this.primaryColor,{h:-160}),this.primaryBorderColor=Ti(this.primaryColor,this.darkMode),this.secondaryBorderColor=Ti(this.secondaryColor,this.darkMode),this.tertiaryBorderColor=Ti(this.tertiaryColor,this.darkMode),this.primaryTextColor=wt(this.primaryColor),this.secondaryTextColor=wt(this.secondaryColor),this.tertiaryTextColor=wt(this.tertiaryColor),this.lineColor=wt(this.background),this.textColor=wt(this.background),this.background="white",this.mainBkg="#ECECFF",this.secondBkg="#ffffde",this.lineColor="#333333",this.border1="#9370DB",this.border2="#aaaa33",this.arrowheadColor="#333333",this.fontFamily='"trebuchet ms", verdana, arial, sans-serif',this.fontSize="16px",this.labelBackground="rgba(232,232,232, 0.8)",this.textColor="#333",this.THEME_COLOR_LIMIT=12,this.nodeBkg="calculated",this.nodeBorder="calculated",this.clusterBkg="calculated",this.clusterBorder="calculated",this.defaultLinkColor="calculated",this.titleColor="calculated",this.edgeLabelBackground="calculated",this.actorBorder="calculated",this.actorBkg="calculated",this.actorTextColor="black",this.actorLineColor="calculated",this.signalColor="calculated",this.signalTextColor="calculated",this.labelBoxBkgColor="calculated",this.labelBoxBorderColor="calculated",this.labelTextColor="calculated",this.loopTextColor="calculated",this.noteBorderColor="calculated",this.noteBkgColor="#fff5ad",this.noteTextColor="calculated",this.activationBorderColor="#666",this.activationBkgColor="#f4f4f4",this.sequenceNumberColor="white",this.sectionBkgColor="calculated",this.altSectionBkgColor="calculated",this.sectionBkgColor2="calculated",this.excludeBkgColor="#eeeeee",this.taskBorderColor="calculated",this.taskBkgColor="calculated",this.taskTextLightColor="calculated",this.taskTextColor=this.taskTextLightColor,this.taskTextDarkColor="calculated",this.taskTextOutsideColor=this.taskTextDarkColor,this.taskTextClickableColor="calculated",this.activeTaskBorderColor="calculated",this.activeTaskBkgColor="calculated",this.gridColor="calculated",this.doneTaskBkgColor="calculated",this.doneTaskBorderColor="calculated",this.critBorderColor="calculated",this.critBkgColor="calculated",this.todayLineColor="calculated",this.sectionBkgColor=qa(102,102,255,.49),this.altSectionBkgColor="white",this.sectionBkgColor2="#fff400",this.taskBorderColor="#534fbc",this.taskBkgColor="#8a90dd",this.taskTextLightColor="white",this.taskTextColor="calculated",this.taskTextDarkColor="black",this.taskTextOutsideColor="calculated",this.taskTextClickableColor="#003163",this.activeTaskBorderColor="#534fbc",this.activeTaskBkgColor="#bfc7ff",this.gridColor="lightgrey",this.doneTaskBkgColor="lightgrey",this.doneTaskBorderColor="grey",this.critBorderColor="#ff8888",this.critBkgColor="red",this.todayLineColor="red",this.personBorder=this.primaryBorderColor,this.personBkg=this.mainBkg,this.archEdgeColor="calculated",this.archEdgeArrowColor="calculated",this.archEdgeWidth="3",this.archGroupBorderColor=this.primaryBorderColor,this.archGroupBorderWidth="2px",this.rowOdd="calculated",this.rowEven="calculated",this.labelColor="black",this.errorBkgColor="#552222",this.errorTextColor="#552222",this.updateColors()}updateColors(){this.cScale0=this.cScale0||this.primaryColor,this.cScale1=this.cScale1||this.secondaryColor,this.cScale2=this.cScale2||this.tertiaryColor,this.cScale3=this.cScale3||Me(this.primaryColor,{h:30}),this.cScale4=this.cScale4||Me(this.primaryColor,{h:60}),this.cScale5=this.cScale5||Me(this.primaryColor,{h:90}),this.cScale6=this.cScale6||Me(this.primaryColor,{h:120}),this.cScale7=this.cScale7||Me(this.primaryColor,{h:150}),this.cScale8=this.cScale8||Me(this.primaryColor,{h:210}),this.cScale9=this.cScale9||Me(this.primaryColor,{h:270}),this.cScale10=this.cScale10||Me(this.primaryColor,{h:300}),this.cScale11=this.cScale11||Me(this.primaryColor,{h:330}),this.cScalePeer1=this.cScalePeer1||Ot(this.secondaryColor,45),this.cScalePeer2=this.cScalePeer2||Ot(this.tertiaryColor,40);for(let e=0;e{this[n]==="calculated"&&(this[n]=void 0)}),typeof e!="object"){this.updateColors();return}let r=Object.keys(e);r.forEach(n=>{this[n]=e[n]}),this.updateColors(),r.forEach(n=>{this[n]=e[n]})}},oh=o(t=>{let e=new QC;return e.calculate(t),e},"getThemeVariables")});var ZC,N$,M$=N(()=>{"use strict";Ys();Ay();o0();ZC=class{static{o(this,"Theme")}constructor(){this.background="#f4f4f4",this.primaryColor="#cde498",this.secondaryColor="#cdffb2",this.background="white",this.mainBkg="#cde498",this.secondBkg="#cdffb2",this.lineColor="green",this.border1="#13540c",this.border2="#6eaa49",this.arrowheadColor="green",this.fontFamily='"trebuchet ms", verdana, arial, sans-serif',this.fontSize="16px",this.tertiaryColor=Dt("#cde498",10),this.primaryBorderColor=Ti(this.primaryColor,this.darkMode),this.secondaryBorderColor=Ti(this.secondaryColor,this.darkMode),this.tertiaryBorderColor=Ti(this.tertiaryColor,this.darkMode),this.primaryTextColor=wt(this.primaryColor),this.secondaryTextColor=wt(this.secondaryColor),this.tertiaryTextColor=wt(this.primaryColor),this.lineColor=wt(this.background),this.textColor=wt(this.background),this.THEME_COLOR_LIMIT=12,this.nodeBkg="calculated",this.nodeBorder="calculated",this.clusterBkg="calculated",this.clusterBorder="calculated",this.defaultLinkColor="calculated",this.titleColor="#333",this.edgeLabelBackground="#e8e8e8",this.actorBorder="calculated",this.actorBkg="calculated",this.actorTextColor="black",this.actorLineColor="calculated",this.signalColor="#333",this.signalTextColor="#333",this.labelBoxBkgColor="calculated",this.labelBoxBorderColor="#326932",this.labelTextColor="calculated",this.loopTextColor="calculated",this.noteBorderColor="calculated",this.noteBkgColor="#fff5ad",this.noteTextColor="calculated",this.activationBorderColor="#666",this.activationBkgColor="#f4f4f4",this.sequenceNumberColor="white",this.sectionBkgColor="#6eaa49",this.altSectionBkgColor="white",this.sectionBkgColor2="#6eaa49",this.excludeBkgColor="#eeeeee",this.taskBorderColor="calculated",this.taskBkgColor="#487e3a",this.taskTextLightColor="white",this.taskTextColor="calculated",this.taskTextDarkColor="black",this.taskTextOutsideColor="calculated",this.taskTextClickableColor="#003163",this.activeTaskBorderColor="calculated",this.activeTaskBkgColor="calculated",this.gridColor="lightgrey",this.doneTaskBkgColor="lightgrey",this.doneTaskBorderColor="grey",this.critBorderColor="#ff8888",this.critBkgColor="red",this.todayLineColor="red",this.personBorder=this.primaryBorderColor,this.personBkg=this.mainBkg,this.archEdgeColor="calculated",this.archEdgeArrowColor="calculated",this.archEdgeWidth="3",this.archGroupBorderColor=this.primaryBorderColor,this.archGroupBorderWidth="2px",this.labelColor="black",this.errorBkgColor="#552222",this.errorTextColor="#552222"}updateColors(){this.actorBorder=Ot(this.mainBkg,20),this.actorBkg=this.mainBkg,this.labelBoxBkgColor=this.actorBkg,this.labelTextColor=this.actorTextColor,this.loopTextColor=this.actorTextColor,this.noteBorderColor=this.border2,this.noteTextColor=this.actorTextColor,this.actorLineColor=this.actorBorder,this.cScale0=this.cScale0||this.primaryColor,this.cScale1=this.cScale1||this.secondaryColor,this.cScale2=this.cScale2||this.tertiaryColor,this.cScale3=this.cScale3||Me(this.primaryColor,{h:30}),this.cScale4=this.cScale4||Me(this.primaryColor,{h:60}),this.cScale5=this.cScale5||Me(this.primaryColor,{h:90}),this.cScale6=this.cScale6||Me(this.primaryColor,{h:120}),this.cScale7=this.cScale7||Me(this.primaryColor,{h:150}),this.cScale8=this.cScale8||Me(this.primaryColor,{h:210}),this.cScale9=this.cScale9||Me(this.primaryColor,{h:270}),this.cScale10=this.cScale10||Me(this.primaryColor,{h:300}),this.cScale11=this.cScale11||Me(this.primaryColor,{h:330}),this.cScalePeer1=this.cScalePeer1||Ot(this.secondaryColor,45),this.cScalePeer2=this.cScalePeer2||Ot(this.tertiaryColor,40);for(let e=0;e{this[n]=e[n]}),this.updateColors(),r.forEach(n=>{this[n]=e[n]})}},N$=o(t=>{let e=new ZC;return e.calculate(t),e},"getThemeVariables")});var JC,I$,O$=N(()=>{"use strict";Ys();o0();Ay();JC=class{static{o(this,"Theme")}constructor(){this.primaryColor="#eee",this.contrast="#707070",this.secondaryColor=Dt(this.contrast,55),this.background="#ffffff",this.tertiaryColor=Me(this.primaryColor,{h:-160}),this.primaryBorderColor=Ti(this.primaryColor,this.darkMode),this.secondaryBorderColor=Ti(this.secondaryColor,this.darkMode),this.tertiaryBorderColor=Ti(this.tertiaryColor,this.darkMode),this.primaryTextColor=wt(this.primaryColor),this.secondaryTextColor=wt(this.secondaryColor),this.tertiaryTextColor=wt(this.tertiaryColor),this.lineColor=wt(this.background),this.textColor=wt(this.background),this.mainBkg="#eee",this.secondBkg="calculated",this.lineColor="#666",this.border1="#999",this.border2="calculated",this.note="#ffa",this.text="#333",this.critical="#d42",this.done="#bbb",this.arrowheadColor="#333333",this.fontFamily='"trebuchet ms", verdana, arial, sans-serif',this.fontSize="16px",this.THEME_COLOR_LIMIT=12,this.nodeBkg="calculated",this.nodeBorder="calculated",this.clusterBkg="calculated",this.clusterBorder="calculated",this.defaultLinkColor="calculated",this.titleColor="calculated",this.edgeLabelBackground="white",this.actorBorder="calculated",this.actorBkg="calculated",this.actorTextColor="calculated",this.actorLineColor=this.actorBorder,this.signalColor="calculated",this.signalTextColor="calculated",this.labelBoxBkgColor="calculated",this.labelBoxBorderColor="calculated",this.labelTextColor="calculated",this.loopTextColor="calculated",this.noteBorderColor="calculated",this.noteBkgColor="calculated",this.noteTextColor="calculated",this.activationBorderColor="#666",this.activationBkgColor="#f4f4f4",this.sequenceNumberColor="white",this.sectionBkgColor="calculated",this.altSectionBkgColor="white",this.sectionBkgColor2="calculated",this.excludeBkgColor="#eeeeee",this.taskBorderColor="calculated",this.taskBkgColor="calculated",this.taskTextLightColor="white",this.taskTextColor="calculated",this.taskTextDarkColor="calculated",this.taskTextOutsideColor="calculated",this.taskTextClickableColor="#003163",this.activeTaskBorderColor="calculated",this.activeTaskBkgColor="calculated",this.gridColor="calculated",this.doneTaskBkgColor="calculated",this.doneTaskBorderColor="calculated",this.critBkgColor="calculated",this.critBorderColor="calculated",this.todayLineColor="calculated",this.personBorder=this.primaryBorderColor,this.personBkg=this.mainBkg,this.archEdgeColor="calculated",this.archEdgeArrowColor="calculated",this.archEdgeWidth="3",this.archGroupBorderColor=this.primaryBorderColor,this.archGroupBorderWidth="2px",this.rowOdd=this.rowOdd||Dt(this.mainBkg,75)||"#ffffff",this.rowEven=this.rowEven||"#f4f4f4",this.labelColor="black",this.errorBkgColor="#552222",this.errorTextColor="#552222"}updateColors(){this.secondBkg=Dt(this.contrast,55),this.border2=this.contrast,this.actorBorder=Dt(this.border1,23),this.actorBkg=this.mainBkg,this.actorTextColor=this.text,this.actorLineColor=this.actorBorder,this.signalColor=this.text,this.signalTextColor=this.text,this.labelBoxBkgColor=this.actorBkg,this.labelBoxBorderColor=this.actorBorder,this.labelTextColor=this.text,this.loopTextColor=this.text,this.noteBorderColor="#999",this.noteBkgColor="#666",this.noteTextColor="#fff",this.cScale0=this.cScale0||"#555",this.cScale1=this.cScale1||"#F4F4F4",this.cScale2=this.cScale2||"#555",this.cScale3=this.cScale3||"#BBB",this.cScale4=this.cScale4||"#777",this.cScale5=this.cScale5||"#999",this.cScale6=this.cScale6||"#DDD",this.cScale7=this.cScale7||"#FFF",this.cScale8=this.cScale8||"#DDD",this.cScale9=this.cScale9||"#BBB",this.cScale10=this.cScale10||"#999",this.cScale11=this.cScale11||"#777";for(let e=0;e{this[n]=e[n]}),this.updateColors(),r.forEach(n=>{this[n]=e[n]})}},I$=o(t=>{let e=new JC;return e.calculate(t),e},"getThemeVariables")});var To,q4=N(()=>{"use strict";D$();R$();_y();M$();O$();To={base:{getThemeVariables:_$},dark:{getThemeVariables:L$},default:{getThemeVariables:oh},forest:{getThemeVariables:N$},neutral:{getThemeVariables:I$}}});var ql,P$=N(()=>{"use strict";ql={flowchart:{useMaxWidth:!0,titleTopMargin:25,subGraphTitleMargin:{top:0,bottom:0},diagramPadding:8,htmlLabels:!0,nodeSpacing:50,rankSpacing:50,curve:"basis",padding:15,defaultRenderer:"dagre-wrapper",wrappingWidth:200},sequence:{useMaxWidth:!0,hideUnusedParticipants:!1,activationWidth:10,diagramMarginX:50,diagramMarginY:10,actorMargin:50,width:150,height:65,boxMargin:10,boxTextMargin:5,noteMargin:10,messageMargin:35,messageAlign:"center",mirrorActors:!0,forceMenus:!1,bottomMarginAdj:1,rightAngles:!1,showSequenceNumbers:!1,actorFontSize:14,actorFontFamily:'"Open Sans", sans-serif',actorFontWeight:400,noteFontSize:14,noteFontFamily:'"trebuchet ms", verdana, arial, sans-serif',noteFontWeight:400,noteAlign:"center",messageFontSize:16,messageFontFamily:'"trebuchet ms", verdana, arial, sans-serif',messageFontWeight:400,wrap:!1,wrapPadding:10,labelBoxWidth:50,labelBoxHeight:20},gantt:{useMaxWidth:!0,titleTopMargin:25,barHeight:20,barGap:4,topPadding:50,rightPadding:75,leftPadding:75,gridLineStartPadding:35,fontSize:11,sectionFontSize:11,numberSectionStyles:4,axisFormat:"%Y-%m-%d",topAxis:!1,displayMode:"",weekday:"sunday"},journey:{useMaxWidth:!0,diagramMarginX:50,diagramMarginY:10,leftMargin:150,width:150,height:50,boxMargin:10,boxTextMargin:5,noteMargin:10,messageMargin:35,messageAlign:"center",bottomMarginAdj:1,rightAngles:!1,taskFontSize:14,taskFontFamily:'"Open Sans", sans-serif',taskMargin:50,activationWidth:10,textPlacement:"fo",actorColours:["#8FBC8F","#7CFC00","#00FFFF","#20B2AA","#B0E0E6","#FFFFE0"],sectionFills:["#191970","#8B008B","#4B0082","#2F4F4F","#800000","#8B4513","#00008B"],sectionColours:["#fff"]},class:{useMaxWidth:!0,titleTopMargin:25,arrowMarkerAbsolute:!1,dividerMargin:10,padding:5,textHeight:10,defaultRenderer:"dagre-wrapper",htmlLabels:!1,hideEmptyMembersBox:!1},state:{useMaxWidth:!0,titleTopMargin:25,dividerMargin:10,sizeUnit:5,padding:8,textHeight:10,titleShift:-15,noteMargin:10,forkWidth:70,forkHeight:7,miniPadding:2,fontSizeFactor:5.02,fontSize:24,labelHeight:16,edgeLengthFactor:"20",compositTitleSize:35,radius:5,defaultRenderer:"dagre-wrapper"},er:{useMaxWidth:!0,titleTopMargin:25,diagramPadding:20,layoutDirection:"TB",minEntityWidth:100,minEntityHeight:75,entityPadding:15,nodeSpacing:140,rankSpacing:80,stroke:"gray",fill:"honeydew",fontSize:12},pie:{useMaxWidth:!0,textPosition:.75},quadrantChart:{useMaxWidth:!0,chartWidth:500,chartHeight:500,titleFontSize:20,titlePadding:10,quadrantPadding:5,xAxisLabelPadding:5,yAxisLabelPadding:5,xAxisLabelFontSize:16,yAxisLabelFontSize:16,quadrantLabelFontSize:16,quadrantTextTopPadding:5,pointTextPadding:5,pointLabelFontSize:12,pointRadius:5,xAxisPosition:"top",yAxisPosition:"left",quadrantInternalBorderStrokeWidth:1,quadrantExternalBorderStrokeWidth:2},xyChart:{useMaxWidth:!0,width:700,height:500,titleFontSize:20,titlePadding:10,showTitle:!0,xAxis:{$ref:"#/$defs/XYChartAxisConfig",showLabel:!0,labelFontSize:14,labelPadding:5,showTitle:!0,titleFontSize:16,titlePadding:5,showTick:!0,tickLength:5,tickWidth:2,showAxisLine:!0,axisLineWidth:2},yAxis:{$ref:"#/$defs/XYChartAxisConfig",showLabel:!0,labelFontSize:14,labelPadding:5,showTitle:!0,titleFontSize:16,titlePadding:5,showTick:!0,tickLength:5,tickWidth:2,showAxisLine:!0,axisLineWidth:2},chartOrientation:"vertical",plotReservedSpacePercent:50},requirement:{useMaxWidth:!0,rect_fill:"#f9f9f9",text_color:"#333",rect_border_size:"0.5px",rect_border_color:"#bbb",rect_min_width:200,rect_min_height:200,fontSize:14,rect_padding:10,line_height:20},mindmap:{useMaxWidth:!0,padding:10,maxNodeWidth:200},kanban:{useMaxWidth:!0,padding:8,sectionWidth:200,ticketBaseUrl:""},timeline:{useMaxWidth:!0,diagramMarginX:50,diagramMarginY:10,leftMargin:150,width:150,height:50,boxMargin:10,boxTextMargin:5,noteMargin:10,messageMargin:35,messageAlign:"center",bottomMarginAdj:1,rightAngles:!1,taskFontSize:14,taskFontFamily:'"Open Sans", sans-serif',taskMargin:50,activationWidth:10,textPlacement:"fo",actorColours:["#8FBC8F","#7CFC00","#00FFFF","#20B2AA","#B0E0E6","#FFFFE0"],sectionFills:["#191970","#8B008B","#4B0082","#2F4F4F","#800000","#8B4513","#00008B"],sectionColours:["#fff"],disableMulticolor:!1},gitGraph:{useMaxWidth:!0,titleTopMargin:25,diagramPadding:8,nodeLabel:{width:75,height:100,x:-25,y:0},mainBranchName:"main",mainBranchOrder:0,showCommitLabel:!0,showBranches:!0,rotateCommitLabel:!0,parallelCommits:!1,arrowMarkerAbsolute:!1},c4:{useMaxWidth:!0,diagramMarginX:50,diagramMarginY:10,c4ShapeMargin:50,c4ShapePadding:20,width:216,height:60,boxMargin:10,c4ShapeInRow:4,nextLinePaddingX:0,c4BoundaryInRow:2,personFontSize:14,personFontFamily:'"Open Sans", sans-serif',personFontWeight:"normal",external_personFontSize:14,external_personFontFamily:'"Open Sans", sans-serif',external_personFontWeight:"normal",systemFontSize:14,systemFontFamily:'"Open Sans", sans-serif',systemFontWeight:"normal",external_systemFontSize:14,external_systemFontFamily:'"Open Sans", sans-serif',external_systemFontWeight:"normal",system_dbFontSize:14,system_dbFontFamily:'"Open Sans", sans-serif',system_dbFontWeight:"normal",external_system_dbFontSize:14,external_system_dbFontFamily:'"Open Sans", sans-serif',external_system_dbFontWeight:"normal",system_queueFontSize:14,system_queueFontFamily:'"Open Sans", sans-serif',system_queueFontWeight:"normal",external_system_queueFontSize:14,external_system_queueFontFamily:'"Open Sans", sans-serif',external_system_queueFontWeight:"normal",boundaryFontSize:14,boundaryFontFamily:'"Open Sans", sans-serif',boundaryFontWeight:"normal",messageFontSize:12,messageFontFamily:'"Open Sans", sans-serif',messageFontWeight:"normal",containerFontSize:14,containerFontFamily:'"Open Sans", sans-serif',containerFontWeight:"normal",external_containerFontSize:14,external_containerFontFamily:'"Open Sans", sans-serif',external_containerFontWeight:"normal",container_dbFontSize:14,container_dbFontFamily:'"Open Sans", sans-serif',container_dbFontWeight:"normal",external_container_dbFontSize:14,external_container_dbFontFamily:'"Open Sans", sans-serif',external_container_dbFontWeight:"normal",container_queueFontSize:14,container_queueFontFamily:'"Open Sans", sans-serif',container_queueFontWeight:"normal",external_container_queueFontSize:14,external_container_queueFontFamily:'"Open Sans", sans-serif',external_container_queueFontWeight:"normal",componentFontSize:14,componentFontFamily:'"Open Sans", sans-serif',componentFontWeight:"normal",external_componentFontSize:14,external_componentFontFamily:'"Open Sans", sans-serif',external_componentFontWeight:"normal",component_dbFontSize:14,component_dbFontFamily:'"Open Sans", sans-serif',component_dbFontWeight:"normal",external_component_dbFontSize:14,external_component_dbFontFamily:'"Open Sans", sans-serif',external_component_dbFontWeight:"normal",component_queueFontSize:14,component_queueFontFamily:'"Open Sans", sans-serif',component_queueFontWeight:"normal",external_component_queueFontSize:14,external_component_queueFontFamily:'"Open Sans", sans-serif',external_component_queueFontWeight:"normal",wrap:!0,wrapPadding:10,person_bg_color:"#08427B",person_border_color:"#073B6F",external_person_bg_color:"#686868",external_person_border_color:"#8A8A8A",system_bg_color:"#1168BD",system_border_color:"#3C7FC0",system_db_bg_color:"#1168BD",system_db_border_color:"#3C7FC0",system_queue_bg_color:"#1168BD",system_queue_border_color:"#3C7FC0",external_system_bg_color:"#999999",external_system_border_color:"#8A8A8A",external_system_db_bg_color:"#999999",external_system_db_border_color:"#8A8A8A",external_system_queue_bg_color:"#999999",external_system_queue_border_color:"#8A8A8A",container_bg_color:"#438DD5",container_border_color:"#3C7FC0",container_db_bg_color:"#438DD5",container_db_border_color:"#3C7FC0",container_queue_bg_color:"#438DD5",container_queue_border_color:"#3C7FC0",external_container_bg_color:"#B3B3B3",external_container_border_color:"#A6A6A6",external_container_db_bg_color:"#B3B3B3",external_container_db_border_color:"#A6A6A6",external_container_queue_bg_color:"#B3B3B3",external_container_queue_border_color:"#A6A6A6",component_bg_color:"#85BBF0",component_border_color:"#78A8D8",component_db_bg_color:"#85BBF0",component_db_border_color:"#78A8D8",component_queue_bg_color:"#85BBF0",component_queue_border_color:"#78A8D8",external_component_bg_color:"#CCCCCC",external_component_border_color:"#BFBFBF",external_component_db_bg_color:"#CCCCCC",external_component_db_border_color:"#BFBFBF",external_component_queue_bg_color:"#CCCCCC",external_component_queue_border_color:"#BFBFBF"},sankey:{useMaxWidth:!0,width:600,height:400,linkColor:"gradient",nodeAlignment:"justify",showValues:!0,prefix:"",suffix:""},block:{useMaxWidth:!0,padding:8},packet:{useMaxWidth:!0,rowHeight:32,bitWidth:32,bitsPerRow:32,showBits:!0,paddingX:5,paddingY:5},architecture:{useMaxWidth:!0,padding:40,iconSize:80,fontSize:16},radar:{useMaxWidth:!0,width:600,height:600,marginTop:50,marginRight:50,marginBottom:50,marginLeft:50,axisScaleFactor:1,axisLabelFactor:1.05,curveTension:.17},theme:"default",look:"classic",handDrawnSeed:0,layout:"dagre",maxTextSize:5e4,maxEdges:500,darkMode:!1,fontFamily:'"trebuchet ms", verdana, arial, sans-serif;',logLevel:5,securityLevel:"strict",startOnLoad:!0,arrowMarkerAbsolute:!1,secure:["secure","securityLevel","startOnLoad","maxTextSize","suppressErrorRendering","maxEdges"],legacyMathML:!1,forceLegacyMathML:!1,deterministicIds:!1,fontSize:16,markdownAutoWrap:!0,suppressErrorRendering:!1}});var B$,F$,$$,or,Ya=N(()=>{"use strict";q4();P$();B$={...ql,deterministicIDSeed:void 0,elk:{mergeEdges:!1,nodePlacementStrategy:"BRANDES_KOEPF"},themeCSS:void 0,themeVariables:To.default.getThemeVariables(),sequence:{...ql.sequence,messageFont:o(function(){return{fontFamily:this.messageFontFamily,fontSize:this.messageFontSize,fontWeight:this.messageFontWeight}},"messageFont"),noteFont:o(function(){return{fontFamily:this.noteFontFamily,fontSize:this.noteFontSize,fontWeight:this.noteFontWeight}},"noteFont"),actorFont:o(function(){return{fontFamily:this.actorFontFamily,fontSize:this.actorFontSize,fontWeight:this.actorFontWeight}},"actorFont")},class:{hideEmptyMembersBox:!1},gantt:{...ql.gantt,tickInterval:void 0,useWidth:void 0},c4:{...ql.c4,useWidth:void 0,personFont:o(function(){return{fontFamily:this.personFontFamily,fontSize:this.personFontSize,fontWeight:this.personFontWeight}},"personFont"),external_personFont:o(function(){return{fontFamily:this.external_personFontFamily,fontSize:this.external_personFontSize,fontWeight:this.external_personFontWeight}},"external_personFont"),systemFont:o(function(){return{fontFamily:this.systemFontFamily,fontSize:this.systemFontSize,fontWeight:this.systemFontWeight}},"systemFont"),external_systemFont:o(function(){return{fontFamily:this.external_systemFontFamily,fontSize:this.external_systemFontSize,fontWeight:this.external_systemFontWeight}},"external_systemFont"),system_dbFont:o(function(){return{fontFamily:this.system_dbFontFamily,fontSize:this.system_dbFontSize,fontWeight:this.system_dbFontWeight}},"system_dbFont"),external_system_dbFont:o(function(){return{fontFamily:this.external_system_dbFontFamily,fontSize:this.external_system_dbFontSize,fontWeight:this.external_system_dbFontWeight}},"external_system_dbFont"),system_queueFont:o(function(){return{fontFamily:this.system_queueFontFamily,fontSize:this.system_queueFontSize,fontWeight:this.system_queueFontWeight}},"system_queueFont"),external_system_queueFont:o(function(){return{fontFamily:this.external_system_queueFontFamily,fontSize:this.external_system_queueFontSize,fontWeight:this.external_system_queueFontWeight}},"external_system_queueFont"),containerFont:o(function(){return{fontFamily:this.containerFontFamily,fontSize:this.containerFontSize,fontWeight:this.containerFontWeight}},"containerFont"),external_containerFont:o(function(){return{fontFamily:this.external_containerFontFamily,fontSize:this.external_containerFontSize,fontWeight:this.external_containerFontWeight}},"external_containerFont"),container_dbFont:o(function(){return{fontFamily:this.container_dbFontFamily,fontSize:this.container_dbFontSize,fontWeight:this.container_dbFontWeight}},"container_dbFont"),external_container_dbFont:o(function(){return{fontFamily:this.external_container_dbFontFamily,fontSize:this.external_container_dbFontSize,fontWeight:this.external_container_dbFontWeight}},"external_container_dbFont"),container_queueFont:o(function(){return{fontFamily:this.container_queueFontFamily,fontSize:this.container_queueFontSize,fontWeight:this.container_queueFontWeight}},"container_queueFont"),external_container_queueFont:o(function(){return{fontFamily:this.external_container_queueFontFamily,fontSize:this.external_container_queueFontSize,fontWeight:this.external_container_queueFontWeight}},"external_container_queueFont"),componentFont:o(function(){return{fontFamily:this.componentFontFamily,fontSize:this.componentFontSize,fontWeight:this.componentFontWeight}},"componentFont"),external_componentFont:o(function(){return{fontFamily:this.external_componentFontFamily,fontSize:this.external_componentFontSize,fontWeight:this.external_componentFontWeight}},"external_componentFont"),component_dbFont:o(function(){return{fontFamily:this.component_dbFontFamily,fontSize:this.component_dbFontSize,fontWeight:this.component_dbFontWeight}},"component_dbFont"),external_component_dbFont:o(function(){return{fontFamily:this.external_component_dbFontFamily,fontSize:this.external_component_dbFontSize,fontWeight:this.external_component_dbFontWeight}},"external_component_dbFont"),component_queueFont:o(function(){return{fontFamily:this.component_queueFontFamily,fontSize:this.component_queueFontSize,fontWeight:this.component_queueFontWeight}},"component_queueFont"),external_component_queueFont:o(function(){return{fontFamily:this.external_component_queueFontFamily,fontSize:this.external_component_queueFontSize,fontWeight:this.external_component_queueFontWeight}},"external_component_queueFont"),boundaryFont:o(function(){return{fontFamily:this.boundaryFontFamily,fontSize:this.boundaryFontSize,fontWeight:this.boundaryFontWeight}},"boundaryFont"),messageFont:o(function(){return{fontFamily:this.messageFontFamily,fontSize:this.messageFontSize,fontWeight:this.messageFontWeight}},"messageFont")},pie:{...ql.pie,useWidth:984},xyChart:{...ql.xyChart,useWidth:void 0},requirement:{...ql.requirement,useWidth:void 0},packet:{...ql.packet},radar:{...ql.radar}},F$=o((t,e="")=>Object.keys(t).reduce((r,n)=>Array.isArray(t[n])?r:typeof t[n]=="object"&&t[n]!==null?[...r,e+n,...F$(t[n],"")]:[...r,e+n],[]),"keyify"),$$=new Set(F$(B$,"")),or=B$});var l0,Dxe,e7=N(()=>{"use strict";Ya();vt();l0=o(t=>{if(Y.debug("sanitizeDirective called with",t),!(typeof t!="object"||t==null)){if(Array.isArray(t)){t.forEach(e=>l0(e));return}for(let e of Object.keys(t)){if(Y.debug("Checking key",e),e.startsWith("__")||e.includes("proto")||e.includes("constr")||!$$.has(e)||t[e]==null){Y.debug("sanitize deleting key: ",e),delete t[e];continue}if(typeof t[e]=="object"){Y.debug("sanitizing object",e),l0(t[e]);continue}let r=["themeCSS","fontFamily","altFontFamily"];for(let n of r)e.includes(n)&&(Y.debug("sanitizing css option",e),t[e]=Dxe(t[e]))}if(t.themeVariables)for(let e of Object.keys(t.themeVariables)){let r=t.themeVariables[e];r?.match&&!r.match(/^[\d "#%(),.;A-Za-z]+$/)&&(t.themeVariables[e]="")}Y.debug("After sanitization",t)}},"sanitizeDirective"),Dxe=o(t=>{let e=0,r=0;for(let n of t){if(e{"use strict";s0();vt();q4();Ya();e7();lh=Object.freeze(or),xs=Gn({},lh),c0=[],Dy=Gn({},lh),Y4=o((t,e)=>{let r=Gn({},t),n={};for(let i of e)H$(i),n=Gn(n,i);if(r=Gn(r,n),n.theme&&n.theme in To){let i=Gn({},G$),a=Gn(i.themeVariables||{},n.themeVariables);r.theme&&r.theme in To&&(r.themeVariables=To[r.theme].getThemeVariables(a))}return Dy=r,q$(Dy),Dy},"updateCurrentConfig"),t7=o(t=>(xs=Gn({},lh),xs=Gn(xs,t),t.theme&&To[t.theme]&&(xs.themeVariables=To[t.theme].getThemeVariables(t.themeVariables)),Y4(xs,c0),xs),"setSiteConfig"),V$=o(t=>{G$=Gn({},t)},"saveConfigFromInitialize"),U$=o(t=>(xs=Gn(xs,t),Y4(xs,c0),xs),"updateSiteConfig"),r7=o(()=>Gn({},xs),"getSiteConfig"),X4=o(t=>(q$(t),Gn(Dy,t),cr()),"setConfig"),cr=o(()=>Gn({},Dy),"getConfig"),H$=o(t=>{t&&(["secure",...xs.secure??[]].forEach(e=>{Object.hasOwn(t,e)&&(Y.debug(`Denied attempt to modify a secure key ${e}`,t[e]),delete t[e])}),Object.keys(t).forEach(e=>{e.startsWith("__")&&delete t[e]}),Object.keys(t).forEach(e=>{typeof t[e]=="string"&&(t[e].includes("<")||t[e].includes(">")||t[e].includes("url(data:"))&&delete t[e],typeof t[e]=="object"&&H$(t[e])}))},"sanitize"),W$=o(t=>{l0(t),t.fontFamily&&!t.themeVariables?.fontFamily&&(t.themeVariables={...t.themeVariables,fontFamily:t.fontFamily}),c0.push(t),Y4(xs,c0)},"addDirective"),Ly=o((t=xs)=>{c0=[],Y4(t,c0)},"reset"),Lxe={LAZY_LOAD_DEPRECATED:"The configuration options lazyLoadedDiagrams and loadExternalDiagramsAtStartup are deprecated. Please use registerExternalDiagrams instead."},z$={},Rxe=o(t=>{z$[t]||(Y.warn(Lxe[t]),z$[t]=!0)},"issueWarning"),q$=o(t=>{t&&(t.lazyLoadedDiagrams||t.loadExternalDiagramsAtStartup)&&Rxe("LAZY_LOAD_DEPRECATED")},"checkConfig")});function Ka(t){return function(e){for(var r=arguments.length,n=new Array(r>1?r-1:0),i=1;i2&&arguments[2]!==void 0?arguments[2]:Q4;Y$&&Y$(t,null);let n=e.length;for(;n--;){let i=e[n];if(typeof i=="string"){let a=r(i);a!==i&&(Nxe(e)||(e[n]=a),i=a)}t[i]=!0}return t}function zxe(t){for(let e=0;e0&&arguments[0]!==void 0?arguments[0]:Qxe(),e=o(At=>sz(At),"DOMPurify");if(e.version="3.2.4",e.removed=[],!t||!t.document||t.document.nodeType!==Oy.document||!t.Element)return e.isSupported=!1,e;let{document:r}=t,n=r,i=n.currentScript,{DocumentFragment:a,HTMLTemplateElement:s,Node:l,Element:u,NodeFilter:h,NamedNodeMap:f=t.NamedNodeMap||t.MozNamedAttrMap,HTMLFormElement:d,DOMParser:p,trustedTypes:m}=t,g=u.prototype,y=Iy(g,"cloneNode"),v=Iy(g,"remove"),x=Iy(g,"nextSibling"),b=Iy(g,"childNodes"),w=Iy(g,"parentNode");if(typeof s=="function"){let At=r.createElement("template");At.content&&At.content.ownerDocument&&(r=At.content.ownerDocument)}let C,T="",{implementation:E,createNodeIterator:A,createDocumentFragment:S,getElementsByTagName:_}=r,{importNode:I}=n,D=tz();e.isSupported=typeof rz=="function"&&typeof w=="function"&&E&&E.createHTMLDocument!==void 0;let{MUSTACHE_EXPR:k,ERB_EXPR:L,TMPLIT_EXPR:R,DATA_ATTR:O,ARIA_ATTR:M,IS_SCRIPT_OR_DATA:B,ATTR_WHITESPACE:F,CUSTOM_ELEMENT:P}=ez,{IS_ALLOWED_URI:z}=ez,$=null,H=Cr({},[...K$,...i7,...a7,...s7,...Q$]),Q=null,j=Cr({},[...Z$,...o7,...J$,...K4]),ie=Object.seal(nz(null,{tagNameCheck:{writable:!0,configurable:!1,enumerable:!0,value:null},attributeNameCheck:{writable:!0,configurable:!1,enumerable:!0,value:null},allowCustomizedBuiltInElements:{writable:!0,configurable:!1,enumerable:!0,value:!1}})),ne=null,le=null,he=!0,K=!0,X=!1,te=!0,J=!1,se=!0,ue=!1,Z=!1,Se=!1,ce=!1,ae=!1,Oe=!1,ge=!0,ze=!1,He="user-content-",$e=!0,Re=!1,Ie={},be=null,W=Cr({},["annotation-xml","audio","colgroup","desc","foreignobject","head","iframe","math","mi","mn","mo","ms","mtext","noembed","noframes","noscript","plaintext","script","style","svg","template","thead","title","video","xmp"]),de=null,re=Cr({},["audio","video","img","source","image","track"]),oe=null,V=Cr({},["alt","class","for","id","label","name","pattern","placeholder","role","summary","title","value","style","xmlns"]),xe="http://www.w3.org/1998/Math/MathML",q="http://www.w3.org/2000/svg",pe="http://www.w3.org/1999/xhtml",ve=pe,Pe=!1,_e=null,we=Cr({},[xe,q,pe],n7),Ve=Cr({},["mi","mo","mn","ms","mtext"]),De=Cr({},["annotation-xml"]),qe=Cr({},["title","style","font","a","script"]),at=null,Rt=["application/xhtml+xml","text/html"],st="text/html",Ue=null,ct=null,We=r.createElement("form"),ot=o(function(Ce){return Ce instanceof RegExp||Ce instanceof Function},"isRegexOrFunction"),Yt=o(function(){let Ce=arguments.length>0&&arguments[0]!==void 0?arguments[0]:{};if(!(ct&&ct===Ce)){if((!Ce||typeof Ce!="object")&&(Ce={}),Ce=Qf(Ce),at=Rt.indexOf(Ce.PARSER_MEDIA_TYPE)===-1?st:Ce.PARSER_MEDIA_TYPE,Ue=at==="application/xhtml+xml"?n7:Q4,$=sl(Ce,"ALLOWED_TAGS")?Cr({},Ce.ALLOWED_TAGS,Ue):H,Q=sl(Ce,"ALLOWED_ATTR")?Cr({},Ce.ALLOWED_ATTR,Ue):j,_e=sl(Ce,"ALLOWED_NAMESPACES")?Cr({},Ce.ALLOWED_NAMESPACES,n7):we,oe=sl(Ce,"ADD_URI_SAFE_ATTR")?Cr(Qf(V),Ce.ADD_URI_SAFE_ATTR,Ue):V,de=sl(Ce,"ADD_DATA_URI_TAGS")?Cr(Qf(re),Ce.ADD_DATA_URI_TAGS,Ue):re,be=sl(Ce,"FORBID_CONTENTS")?Cr({},Ce.FORBID_CONTENTS,Ue):W,ne=sl(Ce,"FORBID_TAGS")?Cr({},Ce.FORBID_TAGS,Ue):{},le=sl(Ce,"FORBID_ATTR")?Cr({},Ce.FORBID_ATTR,Ue):{},Ie=sl(Ce,"USE_PROFILES")?Ce.USE_PROFILES:!1,he=Ce.ALLOW_ARIA_ATTR!==!1,K=Ce.ALLOW_DATA_ATTR!==!1,X=Ce.ALLOW_UNKNOWN_PROTOCOLS||!1,te=Ce.ALLOW_SELF_CLOSE_IN_ATTR!==!1,J=Ce.SAFE_FOR_TEMPLATES||!1,se=Ce.SAFE_FOR_XML!==!1,ue=Ce.WHOLE_DOCUMENT||!1,ce=Ce.RETURN_DOM||!1,ae=Ce.RETURN_DOM_FRAGMENT||!1,Oe=Ce.RETURN_TRUSTED_TYPE||!1,Se=Ce.FORCE_BODY||!1,ge=Ce.SANITIZE_DOM!==!1,ze=Ce.SANITIZE_NAMED_PROPS||!1,$e=Ce.KEEP_CONTENT!==!1,Re=Ce.IN_PLACE||!1,z=Ce.ALLOWED_URI_REGEXP||iz,ve=Ce.NAMESPACE||pe,Ve=Ce.MATHML_TEXT_INTEGRATION_POINTS||Ve,De=Ce.HTML_INTEGRATION_POINTS||De,ie=Ce.CUSTOM_ELEMENT_HANDLING||{},Ce.CUSTOM_ELEMENT_HANDLING&&ot(Ce.CUSTOM_ELEMENT_HANDLING.tagNameCheck)&&(ie.tagNameCheck=Ce.CUSTOM_ELEMENT_HANDLING.tagNameCheck),Ce.CUSTOM_ELEMENT_HANDLING&&ot(Ce.CUSTOM_ELEMENT_HANDLING.attributeNameCheck)&&(ie.attributeNameCheck=Ce.CUSTOM_ELEMENT_HANDLING.attributeNameCheck),Ce.CUSTOM_ELEMENT_HANDLING&&typeof Ce.CUSTOM_ELEMENT_HANDLING.allowCustomizedBuiltInElements=="boolean"&&(ie.allowCustomizedBuiltInElements=Ce.CUSTOM_ELEMENT_HANDLING.allowCustomizedBuiltInElements),J&&(K=!1),ae&&(ce=!0),Ie&&($=Cr({},Q$),Q=[],Ie.html===!0&&(Cr($,K$),Cr(Q,Z$)),Ie.svg===!0&&(Cr($,i7),Cr(Q,o7),Cr(Q,K4)),Ie.svgFilters===!0&&(Cr($,a7),Cr(Q,o7),Cr(Q,K4)),Ie.mathMl===!0&&(Cr($,s7),Cr(Q,J$),Cr(Q,K4))),Ce.ADD_TAGS&&($===H&&($=Qf($)),Cr($,Ce.ADD_TAGS,Ue)),Ce.ADD_ATTR&&(Q===j&&(Q=Qf(Q)),Cr(Q,Ce.ADD_ATTR,Ue)),Ce.ADD_URI_SAFE_ATTR&&Cr(oe,Ce.ADD_URI_SAFE_ATTR,Ue),Ce.FORBID_CONTENTS&&(be===W&&(be=Qf(be)),Cr(be,Ce.FORBID_CONTENTS,Ue)),$e&&($["#text"]=!0),ue&&Cr($,["html","head","body"]),$.table&&(Cr($,["tbody"]),delete ne.tbody),Ce.TRUSTED_TYPES_POLICY){if(typeof Ce.TRUSTED_TYPES_POLICY.createHTML!="function")throw My('TRUSTED_TYPES_POLICY configuration option must provide a "createHTML" hook.');if(typeof Ce.TRUSTED_TYPES_POLICY.createScriptURL!="function")throw My('TRUSTED_TYPES_POLICY configuration option must provide a "createScriptURL" hook.');C=Ce.TRUSTED_TYPES_POLICY,T=C.createHTML("")}else C===void 0&&(C=Zxe(m,i)),C!==null&&typeof T=="string"&&(T=C.createHTML(""));ja&&ja(Ce),ct=Ce}},"_parseConfig"),bt=Cr({},[...i7,...a7,...Gxe]),Mt=Cr({},[...s7,...Vxe]),xt=o(function(Ce){let tt=w(Ce);(!tt||!tt.tagName)&&(tt={namespaceURI:ve,tagName:"template"});let St=Q4(Ce.tagName),mr=Q4(tt.tagName);return _e[Ce.namespaceURI]?Ce.namespaceURI===q?tt.namespaceURI===pe?St==="svg":tt.namespaceURI===xe?St==="svg"&&(mr==="annotation-xml"||Ve[mr]):!!bt[St]:Ce.namespaceURI===xe?tt.namespaceURI===pe?St==="math":tt.namespaceURI===q?St==="math"&&De[mr]:!!Mt[St]:Ce.namespaceURI===pe?tt.namespaceURI===q&&!De[mr]||tt.namespaceURI===xe&&!Ve[mr]?!1:!Mt[St]&&(qe[St]||!bt[St]):!!(at==="application/xhtml+xml"&&_e[Ce.namespaceURI]):!1},"_checkValidNamespace"),ut=o(function(Ce){Ry(e.removed,{element:Ce});try{w(Ce).removeChild(Ce)}catch{v(Ce)}},"_forceRemove"),Et=o(function(Ce,tt){try{Ry(e.removed,{attribute:tt.getAttributeNode(Ce),from:tt})}catch{Ry(e.removed,{attribute:null,from:tt})}if(tt.removeAttribute(Ce),Ce==="is")if(ce||ae)try{ut(tt)}catch{}else try{tt.setAttribute(Ce,"")}catch{}},"_removeAttribute"),ft=o(function(Ce){let tt=null,St=null;if(Se)Ce=""+Ce;else{let gn=j$(Ce,/^[\r\n\t ]+/);St=gn&&gn[0]}at==="application/xhtml+xml"&&ve===pe&&(Ce=''+Ce+"");let mr=C?C.createHTML(Ce):Ce;if(ve===pe)try{tt=new p().parseFromString(mr,at)}catch{}if(!tt||!tt.documentElement){tt=E.createDocument(ve,"template",null);try{tt.documentElement.innerHTML=Pe?T:mr}catch{}}let rn=tt.body||tt.documentElement;return Ce&&St&&rn.insertBefore(r.createTextNode(St),rn.childNodes[0]||null),ve===pe?_.call(tt,ue?"html":"body")[0]:ue?tt.documentElement:rn},"_initDocument"),yt=o(function(Ce){return A.call(Ce.ownerDocument||Ce,Ce,h.SHOW_ELEMENT|h.SHOW_COMMENT|h.SHOW_TEXT|h.SHOW_PROCESSING_INSTRUCTION|h.SHOW_CDATA_SECTION,null)},"_createNodeIterator"),nt=o(function(Ce){return Ce instanceof d&&(typeof Ce.nodeName!="string"||typeof Ce.textContent!="string"||typeof Ce.removeChild!="function"||!(Ce.attributes instanceof f)||typeof Ce.removeAttribute!="function"||typeof Ce.setAttribute!="function"||typeof Ce.namespaceURI!="string"||typeof Ce.insertBefore!="function"||typeof Ce.hasChildNodes!="function")},"_isClobbered"),dn=o(function(Ce){return typeof l=="function"&&Ce instanceof l},"_isNode");function Tt(At,Ce,tt){j4(At,St=>{St.call(e,Ce,tt,ct)})}o(Tt,"_executeHooks");let On=o(function(Ce){let tt=null;if(Tt(D.beforeSanitizeElements,Ce,null),nt(Ce))return ut(Ce),!0;let St=Ue(Ce.nodeName);if(Tt(D.uponSanitizeElement,Ce,{tagName:St,allowedTags:$}),Ce.hasChildNodes()&&!dn(Ce.firstElementChild)&&Xa(/<[/\w]/g,Ce.innerHTML)&&Xa(/<[/\w]/g,Ce.textContent)||Ce.nodeType===Oy.progressingInstruction||se&&Ce.nodeType===Oy.comment&&Xa(/<[/\w]/g,Ce.data))return ut(Ce),!0;if(!$[St]||ne[St]){if(!ne[St]&&_r(St)&&(ie.tagNameCheck instanceof RegExp&&Xa(ie.tagNameCheck,St)||ie.tagNameCheck instanceof Function&&ie.tagNameCheck(St)))return!1;if($e&&!be[St]){let mr=w(Ce)||Ce.parentNode,rn=b(Ce)||Ce.childNodes;if(rn&&mr){let gn=rn.length;for(let Zr=gn-1;Zr>=0;--Zr){let Ni=y(rn[Zr],!0);Ni.__removalCount=(Ce.__removalCount||0)+1,mr.insertBefore(Ni,x(Ce))}}}return ut(Ce),!0}return Ce instanceof u&&!xt(Ce)||(St==="noscript"||St==="noembed"||St==="noframes")&&Xa(/<\/no(script|embed|frames)/i,Ce.innerHTML)?(ut(Ce),!0):(J&&Ce.nodeType===Oy.text&&(tt=Ce.textContent,j4([k,L,R],mr=>{tt=Ny(tt,mr," ")}),Ce.textContent!==tt&&(Ry(e.removed,{element:Ce.cloneNode()}),Ce.textContent=tt)),Tt(D.afterSanitizeElements,Ce,null),!1)},"_sanitizeElements"),tn=o(function(Ce,tt,St){if(ge&&(tt==="id"||tt==="name")&&(St in r||St in We))return!1;if(!(K&&!le[tt]&&Xa(O,tt))){if(!(he&&Xa(M,tt))){if(!Q[tt]||le[tt]){if(!(_r(Ce)&&(ie.tagNameCheck instanceof RegExp&&Xa(ie.tagNameCheck,Ce)||ie.tagNameCheck instanceof Function&&ie.tagNameCheck(Ce))&&(ie.attributeNameCheck instanceof RegExp&&Xa(ie.attributeNameCheck,tt)||ie.attributeNameCheck instanceof Function&&ie.attributeNameCheck(tt))||tt==="is"&&ie.allowCustomizedBuiltInElements&&(ie.tagNameCheck instanceof RegExp&&Xa(ie.tagNameCheck,St)||ie.tagNameCheck instanceof Function&&ie.tagNameCheck(St))))return!1}else if(!oe[tt]){if(!Xa(z,Ny(St,F,""))){if(!((tt==="src"||tt==="xlink:href"||tt==="href")&&Ce!=="script"&&Bxe(St,"data:")===0&&de[Ce])){if(!(X&&!Xa(B,Ny(St,F,"")))){if(St)return!1}}}}}}return!0},"_isValidAttribute"),_r=o(function(Ce){return Ce!=="annotation-xml"&&j$(Ce,P)},"_isBasicCustomElement"),Dr=o(function(Ce){Tt(D.beforeSanitizeAttributes,Ce,null);let{attributes:tt}=Ce;if(!tt||nt(Ce))return;let St={attrName:"",attrValue:"",keepAttr:!0,allowedAttributes:Q,forceKeepAttr:void 0},mr=tt.length;for(;mr--;){let rn=tt[mr],{name:gn,namespaceURI:Zr,value:Ni}=rn,Zn=Ue(gn),Sn=gn==="value"?Ni:Fxe(Ni);if(St.attrName=Zn,St.attrValue=Sn,St.keepAttr=!0,St.forceKeepAttr=void 0,Tt(D.uponSanitizeAttribute,Ce,St),Sn=St.attrValue,ze&&(Zn==="id"||Zn==="name")&&(Et(gn,Ce),Sn=He+Sn),se&&Xa(/((--!?|])>)|<\/(style|title)/i,Sn)){Et(gn,Ce);continue}if(St.forceKeepAttr||(Et(gn,Ce),!St.keepAttr))continue;if(!te&&Xa(/\/>/i,Sn)){Et(gn,Ce);continue}J&&j4([k,L,R],et=>{Sn=Ny(Sn,et," ")});let Hr=Ue(Ce.nodeName);if(tn(Hr,Zn,Sn)){if(C&&typeof m=="object"&&typeof m.getAttributeType=="function"&&!Zr)switch(m.getAttributeType(Hr,Zn)){case"TrustedHTML":{Sn=C.createHTML(Sn);break}case"TrustedScriptURL":{Sn=C.createScriptURL(Sn);break}}try{Zr?Ce.setAttributeNS(Zr,gn,Sn):Ce.setAttribute(gn,Sn),nt(Ce)?ut(Ce):X$(e.removed)}catch{}}}Tt(D.afterSanitizeAttributes,Ce,null)},"_sanitizeAttributes"),Pn=o(function At(Ce){let tt=null,St=yt(Ce);for(Tt(D.beforeSanitizeShadowDOM,Ce,null);tt=St.nextNode();)Tt(D.uponSanitizeShadowNode,tt,null),On(tt),Dr(tt),tt.content instanceof a&&At(tt.content);Tt(D.afterSanitizeShadowDOM,Ce,null)},"_sanitizeShadowDOM");return e.sanitize=function(At){let Ce=arguments.length>1&&arguments[1]!==void 0?arguments[1]:{},tt=null,St=null,mr=null,rn=null;if(Pe=!At,Pe&&(At=""),typeof At!="string"&&!dn(At))if(typeof At.toString=="function"){if(At=At.toString(),typeof At!="string")throw My("dirty is not a string, aborting")}else throw My("toString is not a function");if(!e.isSupported)return At;if(Z||Yt(Ce),e.removed=[],typeof At=="string"&&(Re=!1),Re){if(At.nodeName){let Ni=Ue(At.nodeName);if(!$[Ni]||ne[Ni])throw My("root node is forbidden and cannot be sanitized in-place")}}else if(At instanceof l)tt=ft(""),St=tt.ownerDocument.importNode(At,!0),St.nodeType===Oy.element&&St.nodeName==="BODY"||St.nodeName==="HTML"?tt=St:tt.appendChild(St);else{if(!ce&&!J&&!ue&&At.indexOf("<")===-1)return C&&Oe?C.createHTML(At):At;if(tt=ft(At),!tt)return ce?null:Oe?T:""}tt&&Se&&ut(tt.firstChild);let gn=yt(Re?At:tt);for(;mr=gn.nextNode();)On(mr),Dr(mr),mr.content instanceof a&&Pn(mr.content);if(Re)return At;if(ce){if(ae)for(rn=S.call(tt.ownerDocument);tt.firstChild;)rn.appendChild(tt.firstChild);else rn=tt;return(Q.shadowroot||Q.shadowrootmode)&&(rn=I.call(n,rn,!0)),rn}let Zr=ue?tt.outerHTML:tt.innerHTML;return ue&&$["!doctype"]&&tt.ownerDocument&&tt.ownerDocument.doctype&&tt.ownerDocument.doctype.name&&Xa(az,tt.ownerDocument.doctype.name)&&(Zr=" +`+Zr),J&&j4([k,L,R],Ni=>{Zr=Ny(Zr,Ni," ")}),C&&Oe?C.createHTML(Zr):Zr},e.setConfig=function(){let At=arguments.length>0&&arguments[0]!==void 0?arguments[0]:{};Yt(At),Z=!0},e.clearConfig=function(){ct=null,Z=!1},e.isValidAttribute=function(At,Ce,tt){ct||Yt({});let St=Ue(At),mr=Ue(Ce);return tn(St,mr,tt)},e.addHook=function(At,Ce){typeof Ce=="function"&&Ry(D[At],Ce)},e.removeHook=function(At,Ce){if(Ce!==void 0){let tt=Oxe(D[At],Ce);return tt===-1?void 0:Pxe(D[At],tt,1)[0]}return X$(D[At])},e.removeHooks=function(At){D[At]=[]},e.removeAllHooks=function(){D=tz()},e}var rz,Y$,Nxe,Mxe,Ixe,ja,ko,nz,l7,c7,j4,Oxe,X$,Ry,Pxe,Q4,n7,j$,Ny,Bxe,Fxe,sl,Xa,My,K$,i7,a7,Gxe,s7,Vxe,Q$,Z$,o7,J$,K4,Uxe,Hxe,Wxe,qxe,Yxe,iz,Xxe,jxe,az,Kxe,ez,Oy,Qxe,Zxe,tz,ch,u7=N(()=>{"use strict";({entries:rz,setPrototypeOf:Y$,isFrozen:Nxe,getPrototypeOf:Mxe,getOwnPropertyDescriptor:Ixe}=Object),{freeze:ja,seal:ko,create:nz}=Object,{apply:l7,construct:c7}=typeof Reflect<"u"&&Reflect;ja||(ja=o(function(e){return e},"freeze"));ko||(ko=o(function(e){return e},"seal"));l7||(l7=o(function(e,r,n){return e.apply(r,n)},"apply"));c7||(c7=o(function(e,r){return new e(...r)},"construct"));j4=Ka(Array.prototype.forEach),Oxe=Ka(Array.prototype.lastIndexOf),X$=Ka(Array.prototype.pop),Ry=Ka(Array.prototype.push),Pxe=Ka(Array.prototype.splice),Q4=Ka(String.prototype.toLowerCase),n7=Ka(String.prototype.toString),j$=Ka(String.prototype.match),Ny=Ka(String.prototype.replace),Bxe=Ka(String.prototype.indexOf),Fxe=Ka(String.prototype.trim),sl=Ka(Object.prototype.hasOwnProperty),Xa=Ka(RegExp.prototype.test),My=$xe(TypeError);o(Ka,"unapply");o($xe,"unconstruct");o(Cr,"addToSet");o(zxe,"cleanArray");o(Qf,"clone");o(Iy,"lookupGetter");K$=ja(["a","abbr","acronym","address","area","article","aside","audio","b","bdi","bdo","big","blink","blockquote","body","br","button","canvas","caption","center","cite","code","col","colgroup","content","data","datalist","dd","decorator","del","details","dfn","dialog","dir","div","dl","dt","element","em","fieldset","figcaption","figure","font","footer","form","h1","h2","h3","h4","h5","h6","head","header","hgroup","hr","html","i","img","input","ins","kbd","label","legend","li","main","map","mark","marquee","menu","menuitem","meter","nav","nobr","ol","optgroup","option","output","p","picture","pre","progress","q","rp","rt","ruby","s","samp","section","select","shadow","small","source","spacer","span","strike","strong","style","sub","summary","sup","table","tbody","td","template","textarea","tfoot","th","thead","time","tr","track","tt","u","ul","var","video","wbr"]),i7=ja(["svg","a","altglyph","altglyphdef","altglyphitem","animatecolor","animatemotion","animatetransform","circle","clippath","defs","desc","ellipse","filter","font","g","glyph","glyphref","hkern","image","line","lineargradient","marker","mask","metadata","mpath","path","pattern","polygon","polyline","radialgradient","rect","stop","style","switch","symbol","text","textpath","title","tref","tspan","view","vkern"]),a7=ja(["feBlend","feColorMatrix","feComponentTransfer","feComposite","feConvolveMatrix","feDiffuseLighting","feDisplacementMap","feDistantLight","feDropShadow","feFlood","feFuncA","feFuncB","feFuncG","feFuncR","feGaussianBlur","feImage","feMerge","feMergeNode","feMorphology","feOffset","fePointLight","feSpecularLighting","feSpotLight","feTile","feTurbulence"]),Gxe=ja(["animate","color-profile","cursor","discard","font-face","font-face-format","font-face-name","font-face-src","font-face-uri","foreignobject","hatch","hatchpath","mesh","meshgradient","meshpatch","meshrow","missing-glyph","script","set","solidcolor","unknown","use"]),s7=ja(["math","menclose","merror","mfenced","mfrac","mglyph","mi","mlabeledtr","mmultiscripts","mn","mo","mover","mpadded","mphantom","mroot","mrow","ms","mspace","msqrt","mstyle","msub","msup","msubsup","mtable","mtd","mtext","mtr","munder","munderover","mprescripts"]),Vxe=ja(["maction","maligngroup","malignmark","mlongdiv","mscarries","mscarry","msgroup","mstack","msline","msrow","semantics","annotation","annotation-xml","mprescripts","none"]),Q$=ja(["#text"]),Z$=ja(["accept","action","align","alt","autocapitalize","autocomplete","autopictureinpicture","autoplay","background","bgcolor","border","capture","cellpadding","cellspacing","checked","cite","class","clear","color","cols","colspan","controls","controlslist","coords","crossorigin","datetime","decoding","default","dir","disabled","disablepictureinpicture","disableremoteplayback","download","draggable","enctype","enterkeyhint","face","for","headers","height","hidden","high","href","hreflang","id","inputmode","integrity","ismap","kind","label","lang","list","loading","loop","low","max","maxlength","media","method","min","minlength","multiple","muted","name","nonce","noshade","novalidate","nowrap","open","optimum","pattern","placeholder","playsinline","popover","popovertarget","popovertargetaction","poster","preload","pubdate","radiogroup","readonly","rel","required","rev","reversed","role","rows","rowspan","spellcheck","scope","selected","shape","size","sizes","span","srclang","start","src","srcset","step","style","summary","tabindex","title","translate","type","usemap","valign","value","width","wrap","xmlns","slot"]),o7=ja(["accent-height","accumulate","additive","alignment-baseline","amplitude","ascent","attributename","attributetype","azimuth","basefrequency","baseline-shift","begin","bias","by","class","clip","clippathunits","clip-path","clip-rule","color","color-interpolation","color-interpolation-filters","color-profile","color-rendering","cx","cy","d","dx","dy","diffuseconstant","direction","display","divisor","dur","edgemode","elevation","end","exponent","fill","fill-opacity","fill-rule","filter","filterunits","flood-color","flood-opacity","font-family","font-size","font-size-adjust","font-stretch","font-style","font-variant","font-weight","fx","fy","g1","g2","glyph-name","glyphref","gradientunits","gradienttransform","height","href","id","image-rendering","in","in2","intercept","k","k1","k2","k3","k4","kerning","keypoints","keysplines","keytimes","lang","lengthadjust","letter-spacing","kernelmatrix","kernelunitlength","lighting-color","local","marker-end","marker-mid","marker-start","markerheight","markerunits","markerwidth","maskcontentunits","maskunits","max","mask","media","method","mode","min","name","numoctaves","offset","operator","opacity","order","orient","orientation","origin","overflow","paint-order","path","pathlength","patterncontentunits","patterntransform","patternunits","points","preservealpha","preserveaspectratio","primitiveunits","r","rx","ry","radius","refx","refy","repeatcount","repeatdur","restart","result","rotate","scale","seed","shape-rendering","slope","specularconstant","specularexponent","spreadmethod","startoffset","stddeviation","stitchtiles","stop-color","stop-opacity","stroke-dasharray","stroke-dashoffset","stroke-linecap","stroke-linejoin","stroke-miterlimit","stroke-opacity","stroke","stroke-width","style","surfacescale","systemlanguage","tabindex","tablevalues","targetx","targety","transform","transform-origin","text-anchor","text-decoration","text-rendering","textlength","type","u1","u2","unicode","values","viewbox","visibility","version","vert-adv-y","vert-origin-x","vert-origin-y","width","word-spacing","wrap","writing-mode","xchannelselector","ychannelselector","x","x1","x2","xmlns","y","y1","y2","z","zoomandpan"]),J$=ja(["accent","accentunder","align","bevelled","close","columnsalign","columnlines","columnspan","denomalign","depth","dir","display","displaystyle","encoding","fence","frame","height","href","id","largeop","length","linethickness","lspace","lquote","mathbackground","mathcolor","mathsize","mathvariant","maxsize","minsize","movablelimits","notation","numalign","open","rowalign","rowlines","rowspacing","rowspan","rspace","rquote","scriptlevel","scriptminsize","scriptsizemultiplier","selection","separator","separators","stretchy","subscriptshift","supscriptshift","symmetric","voffset","width","xmlns"]),K4=ja(["xlink:href","xml:id","xlink:title","xml:space","xmlns:xlink"]),Uxe=ko(/\{\{[\w\W]*|[\w\W]*\}\}/gm),Hxe=ko(/<%[\w\W]*|[\w\W]*%>/gm),Wxe=ko(/\$\{[\w\W]*/gm),qxe=ko(/^data-[\-\w.\u00B7-\uFFFF]+$/),Yxe=ko(/^aria-[\-\w]+$/),iz=ko(/^(?:(?:(?:f|ht)tps?|mailto|tel|callto|sms|cid|xmpp):|[^a-z]|[a-z+.\-]+(?:[^a-z+.\-:]|$))/i),Xxe=ko(/^(?:\w+script|data):/i),jxe=ko(/[\u0000-\u0020\u00A0\u1680\u180E\u2000-\u2029\u205F\u3000]/g),az=ko(/^html$/i),Kxe=ko(/^[a-z][.\w]*(-[.\w]+)+$/i),ez=Object.freeze({__proto__:null,ARIA_ATTR:Yxe,ATTR_WHITESPACE:jxe,CUSTOM_ELEMENT:Kxe,DATA_ATTR:qxe,DOCTYPE_NAME:az,ERB_EXPR:Hxe,IS_ALLOWED_URI:iz,IS_SCRIPT_OR_DATA:Xxe,MUSTACHE_EXPR:Uxe,TMPLIT_EXPR:Wxe}),Oy={element:1,attribute:2,text:3,cdataSection:4,entityReference:5,entityNode:6,progressingInstruction:7,comment:8,document:9,documentType:10,documentFragment:11,notation:12},Qxe=o(function(){return typeof window>"u"?null:window},"getGlobal"),Zxe=o(function(e,r){if(typeof e!="object"||typeof e.createPolicy!="function")return null;let n=null,i="data-tt-policy-suffix";r&&r.hasAttribute(i)&&(n=r.getAttribute(i));let a="dompurify"+(n?"#"+n:"");try{return e.createPolicy(a,{createHTML(s){return s},createScriptURL(s){return s}})}catch{return console.warn("TrustedTypes policy "+a+" could not be created."),null}},"_createTrustedTypesPolicy"),tz=o(function(){return{afterSanitizeAttributes:[],afterSanitizeElements:[],afterSanitizeShadowDOM:[],beforeSanitizeAttributes:[],beforeSanitizeElements:[],beforeSanitizeShadowDOM:[],uponSanitizeAttribute:[],uponSanitizeElement:[],uponSanitizeShadowNode:[]}},"_createHooksMap");o(sz,"createDOMPurify");ch=sz()});var MG={};hr(MG,{default:()=>q4e});function abe(t){return String(t).replace(ibe,e=>nbe[e])}function cbe(t){if(t.default)return t.default;var e=t.type,r=Array.isArray(e)?e[0]:e;if(typeof r!="string")return r.enum[0];switch(r){case"boolean":return!1;case"string":return"";case"number":return 0;case"object":return{}}}function gbe(t){for(var e=0;e=i[0]&&t<=i[1])return r.name}return null}function $z(t){for(var e=0;e=u3[e]&&t<=u3[e+1])return!0;return!1}function Abe(t,e){jl[t]=e}function P7(t,e,r){if(!jl[e])throw new Error("Font metrics not found for font: "+e+".");var n=t.charCodeAt(0),i=jl[e][n];if(!i&&t[0]in lz&&(n=lz[t[0]].charCodeAt(0),i=jl[e][n]),!i&&r==="text"&&$z(n)&&(i=jl[e][77]),i)return{depth:i[0],height:i[1],italic:i[2],skew:i[3],width:i[4]}}function _be(t){var e;if(t>=5?e=0:t>=3?e=1:e=2,!h7[e]){var r=h7[e]={cssEmPerMu:Z4.quad[e]/18};for(var n in Z4)Z4.hasOwnProperty(n)&&(r[n]=Z4[n][e])}return h7[e]}function hz(t){if(t instanceof Ts)return t;throw new Error("Expected symbolNode but got "+String(t)+".")}function Nbe(t){if(t instanceof td)return t;throw new Error("Expected span but got "+String(t)+".")}function G(t,e,r,n,i,a){An[t][i]={font:e,group:r,replace:n},a&&n&&(An[t][n]=An[t][i])}function Nt(t){for(var{type:e,names:r,props:n,handler:i,htmlBuilder:a,mathmlBuilder:s}=t,l={type:e,numArgs:n.numArgs,argTypes:n.argTypes,allowedInArgument:!!n.allowedInArgument,allowedInText:!!n.allowedInText,allowedInMath:n.allowedInMath===void 0?!0:n.allowedInMath,numOptionalArgs:n.numOptionalArgs||0,infix:!!n.infix,primitive:!!n.primitive,handler:i},u=0;u0&&(a.push(a3(s,e)),s=[]),a.push(n[l]));s.length>0&&a.push(a3(s,e));var h;r?(h=a3(Pi(r,e,!0)),h.classes=["tag"],a.push(h)):i&&a.push(i);var f=lu(["katex-html"],a);if(f.setAttribute("aria-hidden","true"),h){var d=h.children[0];d.style.height=kt(f.height+f.depth),f.depth&&(d.style.verticalAlign=kt(-f.depth))}return f}function Qz(t){return new ed(t)}function gz(t,e,r,n,i){var a=ks(t,r),s;a.length===1&&a[0]instanceof ws&&Jt.contains(["mrow","mtable"],a[0].type)?s=a[0]:s=new dt.MathNode("mrow",a);var l=new dt.MathNode("annotation",[new dt.TextNode(e)]);l.setAttribute("encoding","application/x-tex");var u=new dt.MathNode("semantics",[s,l]),h=new dt.MathNode("math",[u]);h.setAttribute("xmlns","http://www.w3.org/1998/Math/MathML"),n&&h.setAttribute("display","block");var f=i?"katex":"katex-mathml";return Be.makeSpan([f],[h])}function xr(t,e){if(!t||t.type!==e)throw new Error("Expected node of type "+e+", but got "+(t?"node of type "+t.type:String(t)));return t}function z7(t){var e=w3(t);if(!e)throw new Error("Expected node of symbol group type, but got "+(t?"node of type "+t.type:String(t)));return e}function w3(t){return t&&(t.type==="atom"||Ibe.hasOwnProperty(t.type))?t:null}function tG(t,e){var r=Pi(t.body,e,!0);return u4e([t.mclass],r,e)}function rG(t,e){var r,n=ks(t.body,e);return t.mclass==="minner"?r=new dt.MathNode("mpadded",n):t.mclass==="mord"?t.isCharacterBox?(r=n[0],r.type="mi"):r=new dt.MathNode("mi",n):(t.isCharacterBox?(r=n[0],r.type="mo"):r=new dt.MathNode("mo",n),t.mclass==="mbin"?(r.attributes.lspace="0.22em",r.attributes.rspace="0.22em"):t.mclass==="mpunct"?(r.attributes.lspace="0em",r.attributes.rspace="0.17em"):t.mclass==="mopen"||t.mclass==="mclose"?(r.attributes.lspace="0em",r.attributes.rspace="0em"):t.mclass==="minner"&&(r.attributes.lspace="0.0556em",r.attributes.width="+0.1111em")),r}function d4e(t,e,r){var n=h4e[t];switch(n){case"\\\\cdrightarrow":case"\\\\cdleftarrow":return r.callFunction(n,[e[0]],[e[1]]);case"\\uparrow":case"\\downarrow":{var i=r.callFunction("\\\\cdleft",[e[0]],[]),a={type:"atom",text:n,mode:"math",family:"rel"},s=r.callFunction("\\Big",[a],[]),l=r.callFunction("\\\\cdright",[e[1]],[]),u={type:"ordgroup",mode:"math",body:[i,s,l]};return r.callFunction("\\\\cdparent",[u],[])}case"\\\\cdlongequal":return r.callFunction("\\\\cdlongequal",[],[]);case"\\Vert":{var h={type:"textord",text:"\\Vert",mode:"math"};return r.callFunction("\\Big",[h],[])}default:return{type:"textord",text:" ",mode:"math"}}}function p4e(t){var e=[];for(t.gullet.beginGroup(),t.gullet.macros.set("\\cr","\\\\\\relax"),t.gullet.beginGroup();;){e.push(t.parseExpression(!1,"\\\\")),t.gullet.endGroup(),t.gullet.beginGroup();var r=t.fetch().text;if(r==="&"||r==="\\\\")t.consume();else if(r==="\\end"){e[e.length-1].length===0&&e.pop();break}else throw new gt("Expected \\\\ or \\cr or \\end",t.nextToken)}for(var n=[],i=[n],a=0;a-1))if("<>AV".indexOf(h)>-1)for(var d=0;d<2;d++){for(var p=!0,m=u+1;mAV=|." after @',s[u]);var g=d4e(h,f,t),y={type:"styling",body:[g],mode:"math",style:"display"};n.push(y),l=yz()}a%2===0?n.push(l):n.shift(),n=[],i.push(n)}t.gullet.endGroup(),t.gullet.endGroup();var v=new Array(i[0].length).fill({type:"align",align:"c",pregap:.25,postgap:.25});return{type:"array",mode:"math",body:i,arraystretch:1,addJot:!0,rowGaps:[null],cols:v,colSeparationType:"CD",hLinesBeforeRow:new Array(i.length+1).fill([])}}function k3(t,e){var r=w3(t);if(r&&Jt.contains(A4e,r.text))return r;throw r?new gt("Invalid delimiter '"+r.text+"' after '"+e.funcName+"'",t):new gt("Invalid delimiter type '"+t.type+"'",t)}function bz(t){if(!t.body)throw new Error("Bug: The leftright ParseNode wasn't fully parsed.")}function Ql(t){for(var{type:e,names:r,props:n,handler:i,htmlBuilder:a,mathmlBuilder:s}=t,l={type:e,numArgs:n.numArgs||0,allowedInText:!1,numOptionalArgs:0,handler:i},u=0;u1||!f)&&y.pop(),x.length{"use strict";Xs=class t{static{o(this,"SourceLocation")}constructor(e,r,n){this.lexer=void 0,this.start=void 0,this.end=void 0,this.lexer=e,this.start=r,this.end=n}static range(e,r){return r?!e||!e.loc||!r.loc||e.loc.lexer!==r.loc.lexer?null:new t(e.loc.lexer,e.loc.start,r.loc.end):e&&e.loc}},So=class t{static{o(this,"Token")}constructor(e,r){this.text=void 0,this.loc=void 0,this.noexpand=void 0,this.treatAsRelax=void 0,this.text=e,this.loc=r}range(e,r){return new t(r,Xs.range(this,e))}},gt=class t{static{o(this,"ParseError")}constructor(e,r){this.name=void 0,this.position=void 0,this.length=void 0,this.rawMessage=void 0;var n="KaTeX parse error: "+e,i,a,s=r&&r.loc;if(s&&s.start<=s.end){var l=s.lexer.input;i=s.start,a=s.end,i===l.length?n+=" at end of input: ":n+=" at position "+(i+1)+": ";var u=l.slice(i,a).replace(/[^]/g,"$&\u0332"),h;i>15?h="\u2026"+l.slice(i-15,i):h=l.slice(0,i);var f;a+15":">","<":"<",'"':""","'":"'"},ibe=/[&><"']/g;o(abe,"escape");Fz=o(function t(e){return e.type==="ordgroup"||e.type==="color"?e.body.length===1?t(e.body[0]):e:e.type==="font"?t(e.body):e},"getBaseElem"),sbe=o(function(e){var r=Fz(e);return r.type==="mathord"||r.type==="textord"||r.type==="atom"},"isCharacterBox"),obe=o(function(e){if(!e)throw new Error("Expected non-null, but got "+String(e));return e},"assert"),lbe=o(function(e){var r=/^[\x00-\x20]*([^\\/#?]*?)(:|�*58|�*3a|&colon)/i.exec(e);return r?r[2]!==":"||!/^[a-zA-Z][a-zA-Z0-9+\-.]*$/.test(r[1])?null:r[1].toLowerCase():"_relative"},"protocolFromUrl"),Jt={contains:Jxe,deflt:ebe,escape:abe,hyphenate:rbe,getBaseElem:Fz,isCharacterBox:sbe,protocolFromUrl:lbe},c3={displayMode:{type:"boolean",description:"Render math in display mode, which puts the math in display style (so \\int and \\sum are large, for example), and centers the math on the page on its own line.",cli:"-d, --display-mode"},output:{type:{enum:["htmlAndMathml","html","mathml"]},description:"Determines the markup language of the output.",cli:"-F, --format "},leqno:{type:"boolean",description:"Render display math in leqno style (left-justified tags)."},fleqn:{type:"boolean",description:"Render display math flush left."},throwOnError:{type:"boolean",default:!0,cli:"-t, --no-throw-on-error",cliDescription:"Render errors (in the color given by --error-color) instead of throwing a ParseError exception when encountering an error."},errorColor:{type:"string",default:"#cc0000",cli:"-c, --error-color ",cliDescription:"A color string given in the format 'rgb' or 'rrggbb' (no #). This option determines the color of errors rendered by the -t option.",cliProcessor:o(t=>"#"+t,"cliProcessor")},macros:{type:"object",cli:"-m, --macro ",cliDescription:"Define custom macro of the form '\\foo:expansion' (use multiple -m arguments for multiple macros).",cliDefault:[],cliProcessor:o((t,e)=>(e.push(t),e),"cliProcessor")},minRuleThickness:{type:"number",description:"Specifies a minimum thickness, in ems, for fraction lines, `\\sqrt` top lines, `{array}` vertical lines, `\\hline`, `\\hdashline`, `\\underline`, `\\overline`, and the borders of `\\fbox`, `\\boxed`, and `\\fcolorbox`.",processor:o(t=>Math.max(0,t),"processor"),cli:"--min-rule-thickness ",cliProcessor:parseFloat},colorIsTextColor:{type:"boolean",description:"Makes \\color behave like LaTeX's 2-argument \\textcolor, instead of LaTeX's one-argument \\color mode change.",cli:"-b, --color-is-text-color"},strict:{type:[{enum:["warn","ignore","error"]},"boolean","function"],description:"Turn on strict / LaTeX faithfulness mode, which throws an error if the input uses features that are not supported by LaTeX.",cli:"-S, --strict",cliDefault:!1},trust:{type:["boolean","function"],description:"Trust the input, enabling all HTML features such as \\url.",cli:"-T, --trust"},maxSize:{type:"number",default:1/0,description:"If non-zero, all user-specified sizes, e.g. in \\rule{500em}{500em}, will be capped to maxSize ems. Otherwise, elements and spaces can be arbitrarily large",processor:o(t=>Math.max(0,t),"processor"),cli:"-s, --max-size ",cliProcessor:parseInt},maxExpand:{type:"number",default:1e3,description:"Limit the number of macro expansions to the specified number, to prevent e.g. infinite macro loops. If set to Infinity, the macro expander will try to fully expand as in LaTeX.",processor:o(t=>Math.max(0,t),"processor"),cli:"-e, --max-expand ",cliProcessor:o(t=>t==="Infinity"?1/0:parseInt(t),"cliProcessor")},globalGroup:{type:"boolean",cli:!1}};o(cbe,"getDefaultValue");zy=class{static{o(this,"Settings")}constructor(e){this.displayMode=void 0,this.output=void 0,this.leqno=void 0,this.fleqn=void 0,this.throwOnError=void 0,this.errorColor=void 0,this.macros=void 0,this.minRuleThickness=void 0,this.colorIsTextColor=void 0,this.strict=void 0,this.trust=void 0,this.maxSize=void 0,this.maxExpand=void 0,this.globalGroup=void 0,e=e||{};for(var r in c3)if(c3.hasOwnProperty(r)){var n=c3[r];this[r]=e[r]!==void 0?n.processor?n.processor(e[r]):e[r]:cbe(n)}}reportNonstrict(e,r,n){var i=this.strict;if(typeof i=="function"&&(i=i(e,r,n)),!(!i||i==="ignore")){if(i===!0||i==="error")throw new gt("LaTeX-incompatible input and strict mode is set to 'error': "+(r+" ["+e+"]"),n);i==="warn"?typeof console<"u"&&console.warn("LaTeX-incompatible input and strict mode is set to 'warn': "+(r+" ["+e+"]")):typeof console<"u"&&console.warn("LaTeX-incompatible input and strict mode is set to "+("unrecognized '"+i+"': "+r+" ["+e+"]"))}}useStrictBehavior(e,r,n){var i=this.strict;if(typeof i=="function")try{i=i(e,r,n)}catch{i="error"}return!i||i==="ignore"?!1:i===!0||i==="error"?!0:i==="warn"?(typeof console<"u"&&console.warn("LaTeX-incompatible input and strict mode is set to 'warn': "+(r+" ["+e+"]")),!1):(typeof console<"u"&&console.warn("LaTeX-incompatible input and strict mode is set to "+("unrecognized '"+i+"': "+r+" ["+e+"]")),!1)}isTrusted(e){if(e.url&&!e.protocol){var r=Jt.protocolFromUrl(e.url);if(r==null)return!1;e.protocol=r}var n=typeof this.trust=="function"?this.trust(e):this.trust;return!!n}},Yl=class{static{o(this,"Style")}constructor(e,r,n){this.id=void 0,this.size=void 0,this.cramped=void 0,this.id=e,this.size=r,this.cramped=n}sup(){return Xl[ube[this.id]]}sub(){return Xl[hbe[this.id]]}fracNum(){return Xl[fbe[this.id]]}fracDen(){return Xl[dbe[this.id]]}cramp(){return Xl[pbe[this.id]]}text(){return Xl[mbe[this.id]]}isTight(){return this.size>=2}},O7=0,h3=1,f0=2,su=3,Gy=4,Eo=5,d0=6,Qa=7,Xl=[new Yl(O7,0,!1),new Yl(h3,0,!0),new Yl(f0,1,!1),new Yl(su,1,!0),new Yl(Gy,2,!1),new Yl(Eo,2,!0),new Yl(d0,3,!1),new Yl(Qa,3,!0)],ube=[Gy,Eo,Gy,Eo,d0,Qa,d0,Qa],hbe=[Eo,Eo,Eo,Eo,Qa,Qa,Qa,Qa],fbe=[f0,su,Gy,Eo,d0,Qa,d0,Qa],dbe=[su,su,Eo,Eo,Qa,Qa,Qa,Qa],pbe=[h3,h3,su,su,Eo,Eo,Qa,Qa],mbe=[O7,h3,f0,su,f0,su,f0,su],tr={DISPLAY:Xl[O7],TEXT:Xl[f0],SCRIPT:Xl[Gy],SCRIPTSCRIPT:Xl[d0]},k7=[{name:"latin",blocks:[[256,591],[768,879]]},{name:"cyrillic",blocks:[[1024,1279]]},{name:"armenian",blocks:[[1328,1423]]},{name:"brahmic",blocks:[[2304,4255]]},{name:"georgian",blocks:[[4256,4351]]},{name:"cjk",blocks:[[12288,12543],[19968,40879],[65280,65376]]},{name:"hangul",blocks:[[44032,55215]]}];o(gbe,"scriptFromCodepoint");u3=[];k7.forEach(t=>t.blocks.forEach(e=>u3.push(...e)));o($z,"supportedCodepoint");h0=80,ybe=o(function(e,r){return"M95,"+(622+e+r)+` +c-2.7,0,-7.17,-2.7,-13.5,-8c-5.8,-5.3,-9.5,-10,-9.5,-14 +c0,-2,0.3,-3.3,1,-4c1.3,-2.7,23.83,-20.7,67.5,-54 +c44.2,-33.3,65.8,-50.3,66.5,-51c1.3,-1.3,3,-2,5,-2c4.7,0,8.7,3.3,12,10 +s173,378,173,378c0.7,0,35.3,-71,104,-213c68.7,-142,137.5,-285,206.5,-429 +c69,-144,104.5,-217.7,106.5,-221 +l`+e/2.075+" -"+e+` +c5.3,-9.3,12,-14,20,-14 +H400000v`+(40+e)+`H845.2724 +s-225.272,467,-225.272,467s-235,486,-235,486c-2.7,4.7,-9,7,-19,7 +c-6,0,-10,-1,-12,-3s-194,-422,-194,-422s-65,47,-65,47z +M`+(834+e)+" "+r+"h400000v"+(40+e)+"h-400000z"},"sqrtMain"),vbe=o(function(e,r){return"M263,"+(601+e+r)+`c0.7,0,18,39.7,52,119 +c34,79.3,68.167,158.7,102.5,238c34.3,79.3,51.8,119.3,52.5,120 +c340,-704.7,510.7,-1060.3,512,-1067 +l`+e/2.084+" -"+e+` +c4.7,-7.3,11,-11,19,-11 +H40000v`+(40+e)+`H1012.3 +s-271.3,567,-271.3,567c-38.7,80.7,-84,175,-136,283c-52,108,-89.167,185.3,-111.5,232 +c-22.3,46.7,-33.8,70.3,-34.5,71c-4.7,4.7,-12.3,7,-23,7s-12,-1,-12,-1 +s-109,-253,-109,-253c-72.7,-168,-109.3,-252,-110,-252c-10.7,8,-22,16.7,-34,26 +c-22,17.3,-33.3,26,-34,26s-26,-26,-26,-26s76,-59,76,-59s76,-60,76,-60z +M`+(1001+e)+" "+r+"h400000v"+(40+e)+"h-400000z"},"sqrtSize1"),xbe=o(function(e,r){return"M983 "+(10+e+r)+` +l`+e/3.13+" -"+e+` +c4,-6.7,10,-10,18,-10 H400000v`+(40+e)+` +H1013.1s-83.4,268,-264.1,840c-180.7,572,-277,876.3,-289,913c-4.7,4.7,-12.7,7,-24,7 +s-12,0,-12,0c-1.3,-3.3,-3.7,-11.7,-7,-25c-35.3,-125.3,-106.7,-373.3,-214,-744 +c-10,12,-21,25,-33,39s-32,39,-32,39c-6,-5.3,-15,-14,-27,-26s25,-30,25,-30 +c26.7,-32.7,52,-63,76,-91s52,-60,52,-60s208,722,208,722 +c56,-175.3,126.3,-397.3,211,-666c84.7,-268.7,153.8,-488.2,207.5,-658.5 +c53.7,-170.3,84.5,-266.8,92.5,-289.5z +M`+(1001+e)+" "+r+"h400000v"+(40+e)+"h-400000z"},"sqrtSize2"),bbe=o(function(e,r){return"M424,"+(2398+e+r)+` +c-1.3,-0.7,-38.5,-172,-111.5,-514c-73,-342,-109.8,-513.3,-110.5,-514 +c0,-2,-10.7,14.3,-32,49c-4.7,7.3,-9.8,15.7,-15.5,25c-5.7,9.3,-9.8,16,-12.5,20 +s-5,7,-5,7c-4,-3.3,-8.3,-7.7,-13,-13s-13,-13,-13,-13s76,-122,76,-122s77,-121,77,-121 +s209,968,209,968c0,-2,84.7,-361.7,254,-1079c169.3,-717.3,254.7,-1077.7,256,-1081 +l`+e/4.223+" -"+e+`c4,-6.7,10,-10,18,-10 H400000 +v`+(40+e)+`H1014.6 +s-87.3,378.7,-272.6,1166c-185.3,787.3,-279.3,1182.3,-282,1185 +c-2,6,-10,9,-24,9 +c-8,0,-12,-0.7,-12,-2z M`+(1001+e)+" "+r+` +h400000v`+(40+e)+"h-400000z"},"sqrtSize3"),wbe=o(function(e,r){return"M473,"+(2713+e+r)+` +c339.3,-1799.3,509.3,-2700,510,-2702 l`+e/5.298+" -"+e+` +c3.3,-7.3,9.3,-11,18,-11 H400000v`+(40+e)+`H1017.7 +s-90.5,478,-276.2,1466c-185.7,988,-279.5,1483,-281.5,1485c-2,6,-10,9,-24,9 +c-8,0,-12,-0.7,-12,-2c0,-1.3,-5.3,-32,-16,-92c-50.7,-293.3,-119.7,-693.3,-207,-1200 +c0,-1.3,-5.3,8.7,-16,30c-10.7,21.3,-21.3,42.7,-32,64s-16,33,-16,33s-26,-26,-26,-26 +s76,-153,76,-153s77,-151,77,-151c0.7,0.7,35.7,202,105,604c67.3,400.7,102,602.7,104, +606zM`+(1001+e)+" "+r+"h400000v"+(40+e)+"H1017.7z"},"sqrtSize4"),Tbe=o(function(e){var r=e/2;return"M400000 "+e+" H0 L"+r+" 0 l65 45 L145 "+(e-80)+" H400000z"},"phasePath"),kbe=o(function(e,r,n){var i=n-54-r-e;return"M702 "+(e+r)+"H400000"+(40+e)+` +H742v`+i+`l-4 4-4 4c-.667.7 -2 1.5-4 2.5s-4.167 1.833-6.5 2.5-5.5 1-9.5 1 +h-12l-28-84c-16.667-52-96.667 -294.333-240-727l-212 -643 -85 170 +c-4-3.333-8.333-7.667-13 -13l-13-13l77-155 77-156c66 199.333 139 419.667 +219 661 l218 661zM702 `+r+"H400000v"+(40+e)+"H742z"},"sqrtTall"),Ebe=o(function(e,r,n){r=1e3*r;var i="";switch(e){case"sqrtMain":i=ybe(r,h0);break;case"sqrtSize1":i=vbe(r,h0);break;case"sqrtSize2":i=xbe(r,h0);break;case"sqrtSize3":i=bbe(r,h0);break;case"sqrtSize4":i=wbe(r,h0);break;case"sqrtTall":i=kbe(r,h0,n)}return i},"sqrtPath"),Sbe=o(function(e,r){switch(e){case"\u239C":return"M291 0 H417 V"+r+" H291z M291 0 H417 V"+r+" H291z";case"\u2223":return"M145 0 H188 V"+r+" H145z M145 0 H188 V"+r+" H145z";case"\u2225":return"M145 0 H188 V"+r+" H145z M145 0 H188 V"+r+" H145z"+("M367 0 H410 V"+r+" H367z M367 0 H410 V"+r+" H367z");case"\u239F":return"M457 0 H583 V"+r+" H457z M457 0 H583 V"+r+" H457z";case"\u23A2":return"M319 0 H403 V"+r+" H319z M319 0 H403 V"+r+" H319z";case"\u23A5":return"M263 0 H347 V"+r+" H263z M263 0 H347 V"+r+" H263z";case"\u23AA":return"M384 0 H504 V"+r+" H384z M384 0 H504 V"+r+" H384z";case"\u23D0":return"M312 0 H355 V"+r+" H312z M312 0 H355 V"+r+" H312z";case"\u2016":return"M257 0 H300 V"+r+" H257z M257 0 H300 V"+r+" H257z"+("M478 0 H521 V"+r+" H478z M478 0 H521 V"+r+" H478z");default:return""}},"innerPath"),oz={doubleleftarrow:`M262 157 +l10-10c34-36 62.7-77 86-123 3.3-8 5-13.3 5-16 0-5.3-6.7-8-20-8-7.3 + 0-12.2.5-14.5 1.5-2.3 1-4.8 4.5-7.5 10.5-49.3 97.3-121.7 169.3-217 216-28 + 14-57.3 25-88 33-6.7 2-11 3.8-13 5.5-2 1.7-3 4.2-3 7.5s1 5.8 3 7.5 +c2 1.7 6.3 3.5 13 5.5 68 17.3 128.2 47.8 180.5 91.5 52.3 43.7 93.8 96.2 124.5 + 157.5 9.3 8 15.3 12.3 18 13h6c12-.7 18-4 18-10 0-2-1.7-7-5-15-23.3-46-52-87 +-86-123l-10-10h399738v-40H218c328 0 0 0 0 0l-10-8c-26.7-20-65.7-43-117-69 2.7 +-2 6-3.7 10-5 36.7-16 72.3-37.3 107-64l10-8h399782v-40z +m8 0v40h399730v-40zm0 194v40h399730v-40z`,doublerightarrow:`M399738 392l +-10 10c-34 36-62.7 77-86 123-3.3 8-5 13.3-5 16 0 5.3 6.7 8 20 8 7.3 0 12.2-.5 + 14.5-1.5 2.3-1 4.8-4.5 7.5-10.5 49.3-97.3 121.7-169.3 217-216 28-14 57.3-25 88 +-33 6.7-2 11-3.8 13-5.5 2-1.7 3-4.2 3-7.5s-1-5.8-3-7.5c-2-1.7-6.3-3.5-13-5.5-68 +-17.3-128.2-47.8-180.5-91.5-52.3-43.7-93.8-96.2-124.5-157.5-9.3-8-15.3-12.3-18 +-13h-6c-12 .7-18 4-18 10 0 2 1.7 7 5 15 23.3 46 52 87 86 123l10 10H0v40h399782 +c-328 0 0 0 0 0l10 8c26.7 20 65.7 43 117 69-2.7 2-6 3.7-10 5-36.7 16-72.3 37.3 +-107 64l-10 8H0v40zM0 157v40h399730v-40zm0 194v40h399730v-40z`,leftarrow:`M400000 241H110l3-3c68.7-52.7 113.7-120 + 135-202 4-14.7 6-23 6-25 0-7.3-7-11-21-11-8 0-13.2.8-15.5 2.5-2.3 1.7-4.2 5.8 +-5.5 12.5-1.3 4.7-2.7 10.3-4 17-12 48.7-34.8 92-68.5 130S65.3 228.3 18 247 +c-10 4-16 7.7-18 11 0 8.7 6 14.3 18 17 47.3 18.7 87.8 47 121.5 85S196 441.3 208 + 490c.7 2 1.3 5 2 9s1.2 6.7 1.5 8c.3 1.3 1 3.3 2 6s2.2 4.5 3.5 5.5c1.3 1 3.3 + 1.8 6 2.5s6 1 10 1c14 0 21-3.7 21-11 0-2-2-10.3-6-25-20-79.3-65-146.7-135-202 + l-3-3h399890zM100 241v40h399900v-40z`,leftbrace:`M6 548l-6-6v-35l6-11c56-104 135.3-181.3 238-232 57.3-28.7 117 +-45 179-50h399577v120H403c-43.3 7-81 15-113 26-100.7 33-179.7 91-237 174-2.7 + 5-6 9-10 13-.7 1-7.3 1-20 1H6z`,leftbraceunder:`M0 6l6-6h17c12.688 0 19.313.3 20 1 4 4 7.313 8.3 10 13 + 35.313 51.3 80.813 93.8 136.5 127.5 55.688 33.7 117.188 55.8 184.5 66.5.688 + 0 2 .3 4 1 18.688 2.7 76 4.3 172 5h399450v120H429l-6-1c-124.688-8-235-61.7 +-331-161C60.687 138.7 32.312 99.3 7 54L0 41V6z`,leftgroup:`M400000 80 +H435C64 80 168.3 229.4 21 260c-5.9 1.2-18 0-18 0-2 0-3-1-3-3v-38C76 61 257 0 + 435 0h399565z`,leftgroupunder:`M400000 262 +H435C64 262 168.3 112.6 21 82c-5.9-1.2-18 0-18 0-2 0-3 1-3 3v38c76 158 257 219 + 435 219h399565z`,leftharpoon:`M0 267c.7 5.3 3 10 7 14h399993v-40H93c3.3 +-3.3 10.2-9.5 20.5-18.5s17.8-15.8 22.5-20.5c50.7-52 88-110.3 112-175 4-11.3 5 +-18.3 3-21-1.3-4-7.3-6-18-6-8 0-13 .7-15 2s-4.7 6.7-8 16c-42 98.7-107.3 174.7 +-196 228-6.7 4.7-10.7 8-12 10-1.3 2-2 5.7-2 11zm100-26v40h399900v-40z`,leftharpoonplus:`M0 267c.7 5.3 3 10 7 14h399993v-40H93c3.3-3.3 10.2-9.5 + 20.5-18.5s17.8-15.8 22.5-20.5c50.7-52 88-110.3 112-175 4-11.3 5-18.3 3-21-1.3 +-4-7.3-6-18-6-8 0-13 .7-15 2s-4.7 6.7-8 16c-42 98.7-107.3 174.7-196 228-6.7 4.7 +-10.7 8-12 10-1.3 2-2 5.7-2 11zm100-26v40h399900v-40zM0 435v40h400000v-40z +m0 0v40h400000v-40z`,leftharpoondown:`M7 241c-4 4-6.333 8.667-7 14 0 5.333.667 9 2 11s5.333 + 5.333 12 10c90.667 54 156 130 196 228 3.333 10.667 6.333 16.333 9 17 2 .667 5 + 1 9 1h5c10.667 0 16.667-2 18-6 2-2.667 1-9.667-3-21-32-87.333-82.667-157.667 +-152-211l-3-3h399907v-40zM93 281 H400000 v-40L7 241z`,leftharpoondownplus:`M7 435c-4 4-6.3 8.7-7 14 0 5.3.7 9 2 11s5.3 5.3 12 + 10c90.7 54 156 130 196 228 3.3 10.7 6.3 16.3 9 17 2 .7 5 1 9 1h5c10.7 0 16.7 +-2 18-6 2-2.7 1-9.7-3-21-32-87.3-82.7-157.7-152-211l-3-3h399907v-40H7zm93 0 +v40h399900v-40zM0 241v40h399900v-40zm0 0v40h399900v-40z`,lefthook:`M400000 281 H103s-33-11.2-61-33.5S0 197.3 0 164s14.2-61.2 42.5 +-83.5C70.8 58.2 104 47 142 47 c16.7 0 25 6.7 25 20 0 12-8.7 18.7-26 20-40 3.3 +-68.7 15.7-86 37-10 12-15 25.3-15 40 0 22.7 9.8 40.7 29.5 54 19.7 13.3 43.5 21 + 71.5 23h399859zM103 281v-40h399897v40z`,leftlinesegment:`M40 281 V428 H0 V94 H40 V241 H400000 v40z +M40 281 V428 H0 V94 H40 V241 H400000 v40z`,leftmapsto:`M40 281 V448H0V74H40V241H400000v40z +M40 281 V448H0V74H40V241H400000v40z`,leftToFrom:`M0 147h400000v40H0zm0 214c68 40 115.7 95.7 143 167h22c15.3 0 23 +-.3 23-1 0-1.3-5.3-13.7-16-37-18-35.3-41.3-69-70-101l-7-8h399905v-40H95l7-8 +c28.7-32 52-65.7 70-101 10.7-23.3 16-35.7 16-37 0-.7-7.7-1-23-1h-22C115.7 265.3 + 68 321 0 361zm0-174v-40h399900v40zm100 154v40h399900v-40z`,longequal:`M0 50 h400000 v40H0z m0 194h40000v40H0z +M0 50 h400000 v40H0z m0 194h40000v40H0z`,midbrace:`M200428 334 +c-100.7-8.3-195.3-44-280-108-55.3-42-101.7-93-139-153l-9-14c-2.7 4-5.7 8.7-9 14 +-53.3 86.7-123.7 153-211 199-66.7 36-137.3 56.3-212 62H0V214h199568c178.3-11.7 + 311.7-78.3 403-201 6-8 9.7-12 11-12 .7-.7 6.7-1 18-1s17.3.3 18 1c1.3 0 5 4 11 + 12 44.7 59.3 101.3 106.3 170 141s145.3 54.3 229 60h199572v120z`,midbraceunder:`M199572 214 +c100.7 8.3 195.3 44 280 108 55.3 42 101.7 93 139 153l9 14c2.7-4 5.7-8.7 9-14 + 53.3-86.7 123.7-153 211-199 66.7-36 137.3-56.3 212-62h199568v120H200432c-178.3 + 11.7-311.7 78.3-403 201-6 8-9.7 12-11 12-.7.7-6.7 1-18 1s-17.3-.3-18-1c-1.3 0 +-5-4-11-12-44.7-59.3-101.3-106.3-170-141s-145.3-54.3-229-60H0V214z`,oiintSize1:`M512.6 71.6c272.6 0 320.3 106.8 320.3 178.2 0 70.8-47.7 177.6 +-320.3 177.6S193.1 320.6 193.1 249.8c0-71.4 46.9-178.2 319.5-178.2z +m368.1 178.2c0-86.4-60.9-215.4-368.1-215.4-306.4 0-367.3 129-367.3 215.4 0 85.8 +60.9 214.8 367.3 214.8 307.2 0 368.1-129 368.1-214.8z`,oiintSize2:`M757.8 100.1c384.7 0 451.1 137.6 451.1 230 0 91.3-66.4 228.8 +-451.1 228.8-386.3 0-452.7-137.5-452.7-228.8 0-92.4 66.4-230 452.7-230z +m502.4 230c0-111.2-82.4-277.2-502.4-277.2s-504 166-504 277.2 +c0 110 84 276 504 276s502.4-166 502.4-276z`,oiiintSize1:`M681.4 71.6c408.9 0 480.5 106.8 480.5 178.2 0 70.8-71.6 177.6 +-480.5 177.6S202.1 320.6 202.1 249.8c0-71.4 70.5-178.2 479.3-178.2z +m525.8 178.2c0-86.4-86.8-215.4-525.7-215.4-437.9 0-524.7 129-524.7 215.4 0 +85.8 86.8 214.8 524.7 214.8 438.9 0 525.7-129 525.7-214.8z`,oiiintSize2:`M1021.2 53c603.6 0 707.8 165.8 707.8 277.2 0 110-104.2 275.8 +-707.8 275.8-606 0-710.2-165.8-710.2-275.8C311 218.8 415.2 53 1021.2 53z +m770.4 277.1c0-131.2-126.4-327.6-770.5-327.6S248.4 198.9 248.4 330.1 +c0 130 128.8 326.4 772.7 326.4s770.5-196.4 770.5-326.4z`,rightarrow:`M0 241v40h399891c-47.3 35.3-84 78-110 128 +-16.7 32-27.7 63.7-33 95 0 1.3-.2 2.7-.5 4-.3 1.3-.5 2.3-.5 3 0 7.3 6.7 11 20 + 11 8 0 13.2-.8 15.5-2.5 2.3-1.7 4.2-5.5 5.5-11.5 2-13.3 5.7-27 11-41 14.7-44.7 + 39-84.5 73-119.5s73.7-60.2 119-75.5c6-2 9-5.7 9-11s-3-9-9-11c-45.3-15.3-85 +-40.5-119-75.5s-58.3-74.8-73-119.5c-4.7-14-8.3-27.3-11-40-1.3-6.7-3.2-10.8-5.5 +-12.5-2.3-1.7-7.5-2.5-15.5-2.5-14 0-21 3.7-21 11 0 2 2 10.3 6 25 20.7 83.3 67 + 151.7 139 205zm0 0v40h399900v-40z`,rightbrace:`M400000 542l +-6 6h-17c-12.7 0-19.3-.3-20-1-4-4-7.3-8.3-10-13-35.3-51.3-80.8-93.8-136.5-127.5 +s-117.2-55.8-184.5-66.5c-.7 0-2-.3-4-1-18.7-2.7-76-4.3-172-5H0V214h399571l6 1 +c124.7 8 235 61.7 331 161 31.3 33.3 59.7 72.7 85 118l7 13v35z`,rightbraceunder:`M399994 0l6 6v35l-6 11c-56 104-135.3 181.3-238 232-57.3 + 28.7-117 45-179 50H-300V214h399897c43.3-7 81-15 113-26 100.7-33 179.7-91 237 +-174 2.7-5 6-9 10-13 .7-1 7.3-1 20-1h17z`,rightgroup:`M0 80h399565c371 0 266.7 149.4 414 180 5.9 1.2 18 0 18 0 2 0 + 3-1 3-3v-38c-76-158-257-219-435-219H0z`,rightgroupunder:`M0 262h399565c371 0 266.7-149.4 414-180 5.9-1.2 18 0 18 + 0 2 0 3 1 3 3v38c-76 158-257 219-435 219H0z`,rightharpoon:`M0 241v40h399993c4.7-4.7 7-9.3 7-14 0-9.3 +-3.7-15.3-11-18-92.7-56.7-159-133.7-199-231-3.3-9.3-6-14.7-8-16-2-1.3-7-2-15-2 +-10.7 0-16.7 2-18 6-2 2.7-1 9.7 3 21 15.3 42 36.7 81.8 64 119.5 27.3 37.7 58 + 69.2 92 94.5zm0 0v40h399900v-40z`,rightharpoonplus:`M0 241v40h399993c4.7-4.7 7-9.3 7-14 0-9.3-3.7-15.3-11 +-18-92.7-56.7-159-133.7-199-231-3.3-9.3-6-14.7-8-16-2-1.3-7-2-15-2-10.7 0-16.7 + 2-18 6-2 2.7-1 9.7 3 21 15.3 42 36.7 81.8 64 119.5 27.3 37.7 58 69.2 92 94.5z +m0 0v40h399900v-40z m100 194v40h399900v-40zm0 0v40h399900v-40z`,rightharpoondown:`M399747 511c0 7.3 6.7 11 20 11 8 0 13-.8 15-2.5s4.7-6.8 + 8-15.5c40-94 99.3-166.3 178-217 13.3-8 20.3-12.3 21-13 5.3-3.3 8.5-5.8 9.5 +-7.5 1-1.7 1.5-5.2 1.5-10.5s-2.3-10.3-7-15H0v40h399908c-34 25.3-64.7 57-92 95 +-27.3 38-48.7 77.7-64 119-3.3 8.7-5 14-5 16zM0 241v40h399900v-40z`,rightharpoondownplus:`M399747 705c0 7.3 6.7 11 20 11 8 0 13-.8 + 15-2.5s4.7-6.8 8-15.5c40-94 99.3-166.3 178-217 13.3-8 20.3-12.3 21-13 5.3-3.3 + 8.5-5.8 9.5-7.5 1-1.7 1.5-5.2 1.5-10.5s-2.3-10.3-7-15H0v40h399908c-34 25.3 +-64.7 57-92 95-27.3 38-48.7 77.7-64 119-3.3 8.7-5 14-5 16zM0 435v40h399900v-40z +m0-194v40h400000v-40zm0 0v40h400000v-40z`,righthook:`M399859 241c-764 0 0 0 0 0 40-3.3 68.7-15.7 86-37 10-12 15-25.3 + 15-40 0-22.7-9.8-40.7-29.5-54-19.7-13.3-43.5-21-71.5-23-17.3-1.3-26-8-26-20 0 +-13.3 8.7-20 26-20 38 0 71 11.2 99 33.5 0 0 7 5.6 21 16.7 14 11.2 21 33.5 21 + 66.8s-14 61.2-42 83.5c-28 22.3-61 33.5-99 33.5L0 241z M0 281v-40h399859v40z`,rightlinesegment:`M399960 241 V94 h40 V428 h-40 V281 H0 v-40z +M399960 241 V94 h40 V428 h-40 V281 H0 v-40z`,rightToFrom:`M400000 167c-70.7-42-118-97.7-142-167h-23c-15.3 0-23 .3-23 + 1 0 1.3 5.3 13.7 16 37 18 35.3 41.3 69 70 101l7 8H0v40h399905l-7 8c-28.7 32 +-52 65.7-70 101-10.7 23.3-16 35.7-16 37 0 .7 7.7 1 23 1h23c24-69.3 71.3-125 142 +-167z M100 147v40h399900v-40zM0 341v40h399900v-40z`,twoheadleftarrow:`M0 167c68 40 + 115.7 95.7 143 167h22c15.3 0 23-.3 23-1 0-1.3-5.3-13.7-16-37-18-35.3-41.3-69 +-70-101l-7-8h125l9 7c50.7 39.3 85 86 103 140h46c0-4.7-6.3-18.7-19-42-18-35.3 +-40-67.3-66-96l-9-9h399716v-40H284l9-9c26-28.7 48-60.7 66-96 12.7-23.333 19 +-37.333 19-42h-46c-18 54-52.3 100.7-103 140l-9 7H95l7-8c28.7-32 52-65.7 70-101 + 10.7-23.333 16-35.7 16-37 0-.7-7.7-1-23-1h-22C115.7 71.3 68 127 0 167z`,twoheadrightarrow:`M400000 167 +c-68-40-115.7-95.7-143-167h-22c-15.3 0-23 .3-23 1 0 1.3 5.3 13.7 16 37 18 35.3 + 41.3 69 70 101l7 8h-125l-9-7c-50.7-39.3-85-86-103-140h-46c0 4.7 6.3 18.7 19 42 + 18 35.3 40 67.3 66 96l9 9H0v40h399716l-9 9c-26 28.7-48 60.7-66 96-12.7 23.333 +-19 37.333-19 42h46c18-54 52.3-100.7 103-140l9-7h125l-7 8c-28.7 32-52 65.7-70 + 101-10.7 23.333-16 35.7-16 37 0 .7 7.7 1 23 1h22c27.3-71.3 75-127 143-167z`,tilde1:`M200 55.538c-77 0-168 73.953-177 73.953-3 0-7 +-2.175-9-5.437L2 97c-1-2-2-4-2-6 0-4 2-7 5-9l20-12C116 12 171 0 207 0c86 0 + 114 68 191 68 78 0 168-68 177-68 4 0 7 2 9 5l12 19c1 2.175 2 4.35 2 6.525 0 + 4.35-2 7.613-5 9.788l-19 13.05c-92 63.077-116.937 75.308-183 76.128 +-68.267.847-113-73.952-191-73.952z`,tilde2:`M344 55.266c-142 0-300.638 81.316-311.5 86.418 +-8.01 3.762-22.5 10.91-23.5 5.562L1 120c-1-2-1-3-1-4 0-5 3-9 8-10l18.4-9C160.9 + 31.9 283 0 358 0c148 0 188 122 331 122s314-97 326-97c4 0 8 2 10 7l7 21.114 +c1 2.14 1 3.21 1 4.28 0 5.347-3 9.626-7 10.696l-22.3 12.622C852.6 158.372 751 + 181.476 676 181.476c-149 0-189-126.21-332-126.21z`,tilde3:`M786 59C457 59 32 175.242 13 175.242c-6 0-10-3.457 +-11-10.37L.15 138c-1-7 3-12 10-13l19.2-6.4C378.4 40.7 634.3 0 804.3 0c337 0 + 411.8 157 746.8 157 328 0 754-112 773-112 5 0 10 3 11 9l1 14.075c1 8.066-.697 + 16.595-6.697 17.492l-21.052 7.31c-367.9 98.146-609.15 122.696-778.15 122.696 + -338 0-409-156.573-744-156.573z`,tilde4:`M786 58C457 58 32 177.487 13 177.487c-6 0-10-3.345 +-11-10.035L.15 143c-1-7 3-12 10-13l22-6.7C381.2 35 637.15 0 807.15 0c337 0 409 + 177 744 177 328 0 754-127 773-127 5 0 10 3 11 9l1 14.794c1 7.805-3 13.38-9 + 14.495l-20.7 5.574c-366.85 99.79-607.3 139.372-776.3 139.372-338 0-409 + -175.236-744-175.236z`,vec:`M377 20c0-5.333 1.833-10 5.5-14S391 0 397 0c4.667 0 8.667 1.667 12 5 +3.333 2.667 6.667 9 10 19 6.667 24.667 20.333 43.667 41 57 7.333 4.667 11 +10.667 11 18 0 6-1 10-3 12s-6.667 5-14 9c-28.667 14.667-53.667 35.667-75 63 +-1.333 1.333-3.167 3.5-5.5 6.5s-4 4.833-5 5.5c-1 .667-2.5 1.333-4.5 2s-4.333 1 +-7 1c-4.667 0-9.167-1.833-13.5-5.5S337 184 337 178c0-12.667 15.667-32.333 47-59 +H213l-171-1c-8.667-6-13-12.333-13-19 0-4.667 4.333-11.333 13-20h359 +c-16-25.333-24-45-24-59z`,widehat1:`M529 0h5l519 115c5 1 9 5 9 10 0 1-1 2-1 3l-4 22 +c-1 5-5 9-11 9h-2L532 67 19 159h-2c-5 0-9-4-11-9l-5-22c-1-6 2-12 8-13z`,widehat2:`M1181 0h2l1171 176c6 0 10 5 10 11l-2 23c-1 6-5 10 +-11 10h-1L1182 67 15 220h-1c-6 0-10-4-11-10l-2-23c-1-6 4-11 10-11z`,widehat3:`M1181 0h2l1171 236c6 0 10 5 10 11l-2 23c-1 6-5 10 +-11 10h-1L1182 67 15 280h-1c-6 0-10-4-11-10l-2-23c-1-6 4-11 10-11z`,widehat4:`M1181 0h2l1171 296c6 0 10 5 10 11l-2 23c-1 6-5 10 +-11 10h-1L1182 67 15 340h-1c-6 0-10-4-11-10l-2-23c-1-6 4-11 10-11z`,widecheck1:`M529,159h5l519,-115c5,-1,9,-5,9,-10c0,-1,-1,-2,-1,-3l-4,-22c-1, +-5,-5,-9,-11,-9h-2l-512,92l-513,-92h-2c-5,0,-9,4,-11,9l-5,22c-1,6,2,12,8,13z`,widecheck2:`M1181,220h2l1171,-176c6,0,10,-5,10,-11l-2,-23c-1,-6,-5,-10, +-11,-10h-1l-1168,153l-1167,-153h-1c-6,0,-10,4,-11,10l-2,23c-1,6,4,11,10,11z`,widecheck3:`M1181,280h2l1171,-236c6,0,10,-5,10,-11l-2,-23c-1,-6,-5,-10, +-11,-10h-1l-1168,213l-1167,-213h-1c-6,0,-10,4,-11,10l-2,23c-1,6,4,11,10,11z`,widecheck4:`M1181,340h2l1171,-296c6,0,10,-5,10,-11l-2,-23c-1,-6,-5,-10, +-11,-10h-1l-1168,273l-1167,-273h-1c-6,0,-10,4,-11,10l-2,23c-1,6,4,11,10,11z`,baraboveleftarrow:`M400000 620h-399890l3 -3c68.7 -52.7 113.7 -120 135 -202 +c4 -14.7 6 -23 6 -25c0 -7.3 -7 -11 -21 -11c-8 0 -13.2 0.8 -15.5 2.5 +c-2.3 1.7 -4.2 5.8 -5.5 12.5c-1.3 4.7 -2.7 10.3 -4 17c-12 48.7 -34.8 92 -68.5 130 +s-74.2 66.3 -121.5 85c-10 4 -16 7.7 -18 11c0 8.7 6 14.3 18 17c47.3 18.7 87.8 47 +121.5 85s56.5 81.3 68.5 130c0.7 2 1.3 5 2 9s1.2 6.7 1.5 8c0.3 1.3 1 3.3 2 6 +s2.2 4.5 3.5 5.5c1.3 1 3.3 1.8 6 2.5s6 1 10 1c14 0 21 -3.7 21 -11 +c0 -2 -2 -10.3 -6 -25c-20 -79.3 -65 -146.7 -135 -202l-3 -3h399890z +M100 620v40h399900v-40z M0 241v40h399900v-40zM0 241v40h399900v-40z`,rightarrowabovebar:`M0 241v40h399891c-47.3 35.3-84 78-110 128-16.7 32 +-27.7 63.7-33 95 0 1.3-.2 2.7-.5 4-.3 1.3-.5 2.3-.5 3 0 7.3 6.7 11 20 11 8 0 +13.2-.8 15.5-2.5 2.3-1.7 4.2-5.5 5.5-11.5 2-13.3 5.7-27 11-41 14.7-44.7 39 +-84.5 73-119.5s73.7-60.2 119-75.5c6-2 9-5.7 9-11s-3-9-9-11c-45.3-15.3-85-40.5 +-119-75.5s-58.3-74.8-73-119.5c-4.7-14-8.3-27.3-11-40-1.3-6.7-3.2-10.8-5.5 +-12.5-2.3-1.7-7.5-2.5-15.5-2.5-14 0-21 3.7-21 11 0 2 2 10.3 6 25 20.7 83.3 67 +151.7 139 205zm96 379h399894v40H0zm0 0h399904v40H0z`,baraboveshortleftharpoon:`M507,435c-4,4,-6.3,8.7,-7,14c0,5.3,0.7,9,2,11 +c1.3,2,5.3,5.3,12,10c90.7,54,156,130,196,228c3.3,10.7,6.3,16.3,9,17 +c2,0.7,5,1,9,1c0,0,5,0,5,0c10.7,0,16.7,-2,18,-6c2,-2.7,1,-9.7,-3,-21 +c-32,-87.3,-82.7,-157.7,-152,-211c0,0,-3,-3,-3,-3l399351,0l0,-40 +c-398570,0,-399437,0,-399437,0z M593 435 v40 H399500 v-40z +M0 281 v-40 H399908 v40z M0 281 v-40 H399908 v40z`,rightharpoonaboveshortbar:`M0,241 l0,40c399126,0,399993,0,399993,0 +c4.7,-4.7,7,-9.3,7,-14c0,-9.3,-3.7,-15.3,-11,-18c-92.7,-56.7,-159,-133.7,-199, +-231c-3.3,-9.3,-6,-14.7,-8,-16c-2,-1.3,-7,-2,-15,-2c-10.7,0,-16.7,2,-18,6 +c-2,2.7,-1,9.7,3,21c15.3,42,36.7,81.8,64,119.5c27.3,37.7,58,69.2,92,94.5z +M0 241 v40 H399908 v-40z M0 475 v-40 H399500 v40z M0 475 v-40 H399500 v40z`,shortbaraboveleftharpoon:`M7,435c-4,4,-6.3,8.7,-7,14c0,5.3,0.7,9,2,11 +c1.3,2,5.3,5.3,12,10c90.7,54,156,130,196,228c3.3,10.7,6.3,16.3,9,17c2,0.7,5,1,9, +1c0,0,5,0,5,0c10.7,0,16.7,-2,18,-6c2,-2.7,1,-9.7,-3,-21c-32,-87.3,-82.7,-157.7, +-152,-211c0,0,-3,-3,-3,-3l399907,0l0,-40c-399126,0,-399993,0,-399993,0z +M93 435 v40 H400000 v-40z M500 241 v40 H400000 v-40z M500 241 v40 H400000 v-40z`,shortrightharpoonabovebar:`M53,241l0,40c398570,0,399437,0,399437,0 +c4.7,-4.7,7,-9.3,7,-14c0,-9.3,-3.7,-15.3,-11,-18c-92.7,-56.7,-159,-133.7,-199, +-231c-3.3,-9.3,-6,-14.7,-8,-16c-2,-1.3,-7,-2,-15,-2c-10.7,0,-16.7,2,-18,6 +c-2,2.7,-1,9.7,3,21c15.3,42,36.7,81.8,64,119.5c27.3,37.7,58,69.2,92,94.5z +M500 241 v40 H399408 v-40z M500 435 v40 H400000 v-40z`},Cbe=o(function(e,r){switch(e){case"lbrack":return"M403 1759 V84 H666 V0 H319 V1759 v"+r+` v1759 h347 v-84 +H403z M403 1759 V0 H319 V1759 v`+r+" v1759 h84z";case"rbrack":return"M347 1759 V0 H0 V84 H263 V1759 v"+r+` v1759 H0 v84 H347z +M347 1759 V0 H263 V1759 v`+r+" v1759 h84z";case"vert":return"M145 15 v585 v"+r+` v585 c2.667,10,9.667,15,21,15 +c10,0,16.667,-5,20,-15 v-585 v`+-r+` v-585 c-2.667,-10,-9.667,-15,-21,-15 +c-10,0,-16.667,5,-20,15z M188 15 H145 v585 v`+r+" v585 h43z";case"doublevert":return"M145 15 v585 v"+r+` v585 c2.667,10,9.667,15,21,15 +c10,0,16.667,-5,20,-15 v-585 v`+-r+` v-585 c-2.667,-10,-9.667,-15,-21,-15 +c-10,0,-16.667,5,-20,15z M188 15 H145 v585 v`+r+` v585 h43z +M367 15 v585 v`+r+` v585 c2.667,10,9.667,15,21,15 +c10,0,16.667,-5,20,-15 v-585 v`+-r+` v-585 c-2.667,-10,-9.667,-15,-21,-15 +c-10,0,-16.667,5,-20,15z M410 15 H367 v585 v`+r+" v585 h43z";case"lfloor":return"M319 602 V0 H403 V602 v"+r+` v1715 h263 v84 H319z +MM319 602 V0 H403 V602 v`+r+" v1715 H319z";case"rfloor":return"M319 602 V0 H403 V602 v"+r+` v1799 H0 v-84 H319z +MM319 602 V0 H403 V602 v`+r+" v1715 H319z";case"lceil":return"M403 1759 V84 H666 V0 H319 V1759 v"+r+` v602 h84z +M403 1759 V0 H319 V1759 v`+r+" v602 h84z";case"rceil":return"M347 1759 V0 H0 V84 H263 V1759 v"+r+` v602 h84z +M347 1759 V0 h-84 V1759 v`+r+" v602 h84z";case"lparen":return`M863,9c0,-2,-2,-5,-6,-9c0,0,-17,0,-17,0c-12.7,0,-19.3,0.3,-20,1 +c-5.3,5.3,-10.3,11,-15,17c-242.7,294.7,-395.3,682,-458,1162c-21.3,163.3,-33.3,349, +-36,557 l0,`+(r+84)+`c0.2,6,0,26,0,60c2,159.3,10,310.7,24,454c53.3,528,210, +949.7,470,1265c4.7,6,9.7,11.7,15,17c0.7,0.7,7,1,19,1c0,0,18,0,18,0c4,-4,6,-7,6,-9 +c0,-2.7,-3.3,-8.7,-10,-18c-135.3,-192.7,-235.5,-414.3,-300.5,-665c-65,-250.7,-102.5, +-544.7,-112.5,-882c-2,-104,-3,-167,-3,-189 +l0,-`+(r+92)+`c0,-162.7,5.7,-314,17,-454c20.7,-272,63.7,-513,129,-723c65.3, +-210,155.3,-396.3,270,-559c6.7,-9.3,10,-15.3,10,-18z`;case"rparen":return`M76,0c-16.7,0,-25,3,-25,9c0,2,2,6.3,6,13c21.3,28.7,42.3,60.3, +63,95c96.7,156.7,172.8,332.5,228.5,527.5c55.7,195,92.8,416.5,111.5,664.5 +c11.3,139.3,17,290.7,17,454c0,28,1.7,43,3.3,45l0,`+(r+9)+` +c-3,4,-3.3,16.7,-3.3,38c0,162,-5.7,313.7,-17,455c-18.7,248,-55.8,469.3,-111.5,664 +c-55.7,194.7,-131.8,370.3,-228.5,527c-20.7,34.7,-41.7,66.3,-63,95c-2,3.3,-4,7,-6,11 +c0,7.3,5.7,11,17,11c0,0,11,0,11,0c9.3,0,14.3,-0.3,15,-1c5.3,-5.3,10.3,-11,15,-17 +c242.7,-294.7,395.3,-681.7,458,-1161c21.3,-164.7,33.3,-350.7,36,-558 +l0,-`+(r+144)+`c-2,-159.3,-10,-310.7,-24,-454c-53.3,-528,-210,-949.7, +-470,-1265c-4.7,-6,-9.7,-11.7,-15,-17c-0.7,-0.7,-6.7,-1,-18,-1z`;default:throw new Error("Unknown stretchy delimiter.")}},"tallDelim"),ed=class{static{o(this,"DocumentFragment")}constructor(e){this.children=void 0,this.classes=void 0,this.height=void 0,this.depth=void 0,this.maxFontSize=void 0,this.style=void 0,this.children=e,this.classes=[],this.height=0,this.depth=0,this.maxFontSize=0,this.style={}}hasClass(e){return Jt.contains(this.classes,e)}toNode(){for(var e=document.createDocumentFragment(),r=0;rr.toText(),"toText");return this.children.map(e).join("")}},jl={"AMS-Regular":{32:[0,0,0,0,.25],65:[0,.68889,0,0,.72222],66:[0,.68889,0,0,.66667],67:[0,.68889,0,0,.72222],68:[0,.68889,0,0,.72222],69:[0,.68889,0,0,.66667],70:[0,.68889,0,0,.61111],71:[0,.68889,0,0,.77778],72:[0,.68889,0,0,.77778],73:[0,.68889,0,0,.38889],74:[.16667,.68889,0,0,.5],75:[0,.68889,0,0,.77778],76:[0,.68889,0,0,.66667],77:[0,.68889,0,0,.94445],78:[0,.68889,0,0,.72222],79:[.16667,.68889,0,0,.77778],80:[0,.68889,0,0,.61111],81:[.16667,.68889,0,0,.77778],82:[0,.68889,0,0,.72222],83:[0,.68889,0,0,.55556],84:[0,.68889,0,0,.66667],85:[0,.68889,0,0,.72222],86:[0,.68889,0,0,.72222],87:[0,.68889,0,0,1],88:[0,.68889,0,0,.72222],89:[0,.68889,0,0,.72222],90:[0,.68889,0,0,.66667],107:[0,.68889,0,0,.55556],160:[0,0,0,0,.25],165:[0,.675,.025,0,.75],174:[.15559,.69224,0,0,.94666],240:[0,.68889,0,0,.55556],295:[0,.68889,0,0,.54028],710:[0,.825,0,0,2.33334],732:[0,.9,0,0,2.33334],770:[0,.825,0,0,2.33334],771:[0,.9,0,0,2.33334],989:[.08167,.58167,0,0,.77778],1008:[0,.43056,.04028,0,.66667],8245:[0,.54986,0,0,.275],8463:[0,.68889,0,0,.54028],8487:[0,.68889,0,0,.72222],8498:[0,.68889,0,0,.55556],8502:[0,.68889,0,0,.66667],8503:[0,.68889,0,0,.44445],8504:[0,.68889,0,0,.66667],8513:[0,.68889,0,0,.63889],8592:[-.03598,.46402,0,0,.5],8594:[-.03598,.46402,0,0,.5],8602:[-.13313,.36687,0,0,1],8603:[-.13313,.36687,0,0,1],8606:[.01354,.52239,0,0,1],8608:[.01354,.52239,0,0,1],8610:[.01354,.52239,0,0,1.11111],8611:[.01354,.52239,0,0,1.11111],8619:[0,.54986,0,0,1],8620:[0,.54986,0,0,1],8621:[-.13313,.37788,0,0,1.38889],8622:[-.13313,.36687,0,0,1],8624:[0,.69224,0,0,.5],8625:[0,.69224,0,0,.5],8630:[0,.43056,0,0,1],8631:[0,.43056,0,0,1],8634:[.08198,.58198,0,0,.77778],8635:[.08198,.58198,0,0,.77778],8638:[.19444,.69224,0,0,.41667],8639:[.19444,.69224,0,0,.41667],8642:[.19444,.69224,0,0,.41667],8643:[.19444,.69224,0,0,.41667],8644:[.1808,.675,0,0,1],8646:[.1808,.675,0,0,1],8647:[.1808,.675,0,0,1],8648:[.19444,.69224,0,0,.83334],8649:[.1808,.675,0,0,1],8650:[.19444,.69224,0,0,.83334],8651:[.01354,.52239,0,0,1],8652:[.01354,.52239,0,0,1],8653:[-.13313,.36687,0,0,1],8654:[-.13313,.36687,0,0,1],8655:[-.13313,.36687,0,0,1],8666:[.13667,.63667,0,0,1],8667:[.13667,.63667,0,0,1],8669:[-.13313,.37788,0,0,1],8672:[-.064,.437,0,0,1.334],8674:[-.064,.437,0,0,1.334],8705:[0,.825,0,0,.5],8708:[0,.68889,0,0,.55556],8709:[.08167,.58167,0,0,.77778],8717:[0,.43056,0,0,.42917],8722:[-.03598,.46402,0,0,.5],8724:[.08198,.69224,0,0,.77778],8726:[.08167,.58167,0,0,.77778],8733:[0,.69224,0,0,.77778],8736:[0,.69224,0,0,.72222],8737:[0,.69224,0,0,.72222],8738:[.03517,.52239,0,0,.72222],8739:[.08167,.58167,0,0,.22222],8740:[.25142,.74111,0,0,.27778],8741:[.08167,.58167,0,0,.38889],8742:[.25142,.74111,0,0,.5],8756:[0,.69224,0,0,.66667],8757:[0,.69224,0,0,.66667],8764:[-.13313,.36687,0,0,.77778],8765:[-.13313,.37788,0,0,.77778],8769:[-.13313,.36687,0,0,.77778],8770:[-.03625,.46375,0,0,.77778],8774:[.30274,.79383,0,0,.77778],8776:[-.01688,.48312,0,0,.77778],8778:[.08167,.58167,0,0,.77778],8782:[.06062,.54986,0,0,.77778],8783:[.06062,.54986,0,0,.77778],8785:[.08198,.58198,0,0,.77778],8786:[.08198,.58198,0,0,.77778],8787:[.08198,.58198,0,0,.77778],8790:[0,.69224,0,0,.77778],8791:[.22958,.72958,0,0,.77778],8796:[.08198,.91667,0,0,.77778],8806:[.25583,.75583,0,0,.77778],8807:[.25583,.75583,0,0,.77778],8808:[.25142,.75726,0,0,.77778],8809:[.25142,.75726,0,0,.77778],8812:[.25583,.75583,0,0,.5],8814:[.20576,.70576,0,0,.77778],8815:[.20576,.70576,0,0,.77778],8816:[.30274,.79383,0,0,.77778],8817:[.30274,.79383,0,0,.77778],8818:[.22958,.72958,0,0,.77778],8819:[.22958,.72958,0,0,.77778],8822:[.1808,.675,0,0,.77778],8823:[.1808,.675,0,0,.77778],8828:[.13667,.63667,0,0,.77778],8829:[.13667,.63667,0,0,.77778],8830:[.22958,.72958,0,0,.77778],8831:[.22958,.72958,0,0,.77778],8832:[.20576,.70576,0,0,.77778],8833:[.20576,.70576,0,0,.77778],8840:[.30274,.79383,0,0,.77778],8841:[.30274,.79383,0,0,.77778],8842:[.13597,.63597,0,0,.77778],8843:[.13597,.63597,0,0,.77778],8847:[.03517,.54986,0,0,.77778],8848:[.03517,.54986,0,0,.77778],8858:[.08198,.58198,0,0,.77778],8859:[.08198,.58198,0,0,.77778],8861:[.08198,.58198,0,0,.77778],8862:[0,.675,0,0,.77778],8863:[0,.675,0,0,.77778],8864:[0,.675,0,0,.77778],8865:[0,.675,0,0,.77778],8872:[0,.69224,0,0,.61111],8873:[0,.69224,0,0,.72222],8874:[0,.69224,0,0,.88889],8876:[0,.68889,0,0,.61111],8877:[0,.68889,0,0,.61111],8878:[0,.68889,0,0,.72222],8879:[0,.68889,0,0,.72222],8882:[.03517,.54986,0,0,.77778],8883:[.03517,.54986,0,0,.77778],8884:[.13667,.63667,0,0,.77778],8885:[.13667,.63667,0,0,.77778],8888:[0,.54986,0,0,1.11111],8890:[.19444,.43056,0,0,.55556],8891:[.19444,.69224,0,0,.61111],8892:[.19444,.69224,0,0,.61111],8901:[0,.54986,0,0,.27778],8903:[.08167,.58167,0,0,.77778],8905:[.08167,.58167,0,0,.77778],8906:[.08167,.58167,0,0,.77778],8907:[0,.69224,0,0,.77778],8908:[0,.69224,0,0,.77778],8909:[-.03598,.46402,0,0,.77778],8910:[0,.54986,0,0,.76042],8911:[0,.54986,0,0,.76042],8912:[.03517,.54986,0,0,.77778],8913:[.03517,.54986,0,0,.77778],8914:[0,.54986,0,0,.66667],8915:[0,.54986,0,0,.66667],8916:[0,.69224,0,0,.66667],8918:[.0391,.5391,0,0,.77778],8919:[.0391,.5391,0,0,.77778],8920:[.03517,.54986,0,0,1.33334],8921:[.03517,.54986,0,0,1.33334],8922:[.38569,.88569,0,0,.77778],8923:[.38569,.88569,0,0,.77778],8926:[.13667,.63667,0,0,.77778],8927:[.13667,.63667,0,0,.77778],8928:[.30274,.79383,0,0,.77778],8929:[.30274,.79383,0,0,.77778],8934:[.23222,.74111,0,0,.77778],8935:[.23222,.74111,0,0,.77778],8936:[.23222,.74111,0,0,.77778],8937:[.23222,.74111,0,0,.77778],8938:[.20576,.70576,0,0,.77778],8939:[.20576,.70576,0,0,.77778],8940:[.30274,.79383,0,0,.77778],8941:[.30274,.79383,0,0,.77778],8994:[.19444,.69224,0,0,.77778],8995:[.19444,.69224,0,0,.77778],9416:[.15559,.69224,0,0,.90222],9484:[0,.69224,0,0,.5],9488:[0,.69224,0,0,.5],9492:[0,.37788,0,0,.5],9496:[0,.37788,0,0,.5],9585:[.19444,.68889,0,0,.88889],9586:[.19444,.74111,0,0,.88889],9632:[0,.675,0,0,.77778],9633:[0,.675,0,0,.77778],9650:[0,.54986,0,0,.72222],9651:[0,.54986,0,0,.72222],9654:[.03517,.54986,0,0,.77778],9660:[0,.54986,0,0,.72222],9661:[0,.54986,0,0,.72222],9664:[.03517,.54986,0,0,.77778],9674:[.11111,.69224,0,0,.66667],9733:[.19444,.69224,0,0,.94445],10003:[0,.69224,0,0,.83334],10016:[0,.69224,0,0,.83334],10731:[.11111,.69224,0,0,.66667],10846:[.19444,.75583,0,0,.61111],10877:[.13667,.63667,0,0,.77778],10878:[.13667,.63667,0,0,.77778],10885:[.25583,.75583,0,0,.77778],10886:[.25583,.75583,0,0,.77778],10887:[.13597,.63597,0,0,.77778],10888:[.13597,.63597,0,0,.77778],10889:[.26167,.75726,0,0,.77778],10890:[.26167,.75726,0,0,.77778],10891:[.48256,.98256,0,0,.77778],10892:[.48256,.98256,0,0,.77778],10901:[.13667,.63667,0,0,.77778],10902:[.13667,.63667,0,0,.77778],10933:[.25142,.75726,0,0,.77778],10934:[.25142,.75726,0,0,.77778],10935:[.26167,.75726,0,0,.77778],10936:[.26167,.75726,0,0,.77778],10937:[.26167,.75726,0,0,.77778],10938:[.26167,.75726,0,0,.77778],10949:[.25583,.75583,0,0,.77778],10950:[.25583,.75583,0,0,.77778],10955:[.28481,.79383,0,0,.77778],10956:[.28481,.79383,0,0,.77778],57350:[.08167,.58167,0,0,.22222],57351:[.08167,.58167,0,0,.38889],57352:[.08167,.58167,0,0,.77778],57353:[0,.43056,.04028,0,.66667],57356:[.25142,.75726,0,0,.77778],57357:[.25142,.75726,0,0,.77778],57358:[.41951,.91951,0,0,.77778],57359:[.30274,.79383,0,0,.77778],57360:[.30274,.79383,0,0,.77778],57361:[.41951,.91951,0,0,.77778],57366:[.25142,.75726,0,0,.77778],57367:[.25142,.75726,0,0,.77778],57368:[.25142,.75726,0,0,.77778],57369:[.25142,.75726,0,0,.77778],57370:[.13597,.63597,0,0,.77778],57371:[.13597,.63597,0,0,.77778]},"Caligraphic-Regular":{32:[0,0,0,0,.25],65:[0,.68333,0,.19445,.79847],66:[0,.68333,.03041,.13889,.65681],67:[0,.68333,.05834,.13889,.52653],68:[0,.68333,.02778,.08334,.77139],69:[0,.68333,.08944,.11111,.52778],70:[0,.68333,.09931,.11111,.71875],71:[.09722,.68333,.0593,.11111,.59487],72:[0,.68333,.00965,.11111,.84452],73:[0,.68333,.07382,0,.54452],74:[.09722,.68333,.18472,.16667,.67778],75:[0,.68333,.01445,.05556,.76195],76:[0,.68333,0,.13889,.68972],77:[0,.68333,0,.13889,1.2009],78:[0,.68333,.14736,.08334,.82049],79:[0,.68333,.02778,.11111,.79611],80:[0,.68333,.08222,.08334,.69556],81:[.09722,.68333,0,.11111,.81667],82:[0,.68333,0,.08334,.8475],83:[0,.68333,.075,.13889,.60556],84:[0,.68333,.25417,0,.54464],85:[0,.68333,.09931,.08334,.62583],86:[0,.68333,.08222,0,.61278],87:[0,.68333,.08222,.08334,.98778],88:[0,.68333,.14643,.13889,.7133],89:[.09722,.68333,.08222,.08334,.66834],90:[0,.68333,.07944,.13889,.72473],160:[0,0,0,0,.25]},"Fraktur-Regular":{32:[0,0,0,0,.25],33:[0,.69141,0,0,.29574],34:[0,.69141,0,0,.21471],38:[0,.69141,0,0,.73786],39:[0,.69141,0,0,.21201],40:[.24982,.74947,0,0,.38865],41:[.24982,.74947,0,0,.38865],42:[0,.62119,0,0,.27764],43:[.08319,.58283,0,0,.75623],44:[0,.10803,0,0,.27764],45:[.08319,.58283,0,0,.75623],46:[0,.10803,0,0,.27764],47:[.24982,.74947,0,0,.50181],48:[0,.47534,0,0,.50181],49:[0,.47534,0,0,.50181],50:[0,.47534,0,0,.50181],51:[.18906,.47534,0,0,.50181],52:[.18906,.47534,0,0,.50181],53:[.18906,.47534,0,0,.50181],54:[0,.69141,0,0,.50181],55:[.18906,.47534,0,0,.50181],56:[0,.69141,0,0,.50181],57:[.18906,.47534,0,0,.50181],58:[0,.47534,0,0,.21606],59:[.12604,.47534,0,0,.21606],61:[-.13099,.36866,0,0,.75623],63:[0,.69141,0,0,.36245],65:[0,.69141,0,0,.7176],66:[0,.69141,0,0,.88397],67:[0,.69141,0,0,.61254],68:[0,.69141,0,0,.83158],69:[0,.69141,0,0,.66278],70:[.12604,.69141,0,0,.61119],71:[0,.69141,0,0,.78539],72:[.06302,.69141,0,0,.7203],73:[0,.69141,0,0,.55448],74:[.12604,.69141,0,0,.55231],75:[0,.69141,0,0,.66845],76:[0,.69141,0,0,.66602],77:[0,.69141,0,0,1.04953],78:[0,.69141,0,0,.83212],79:[0,.69141,0,0,.82699],80:[.18906,.69141,0,0,.82753],81:[.03781,.69141,0,0,.82699],82:[0,.69141,0,0,.82807],83:[0,.69141,0,0,.82861],84:[0,.69141,0,0,.66899],85:[0,.69141,0,0,.64576],86:[0,.69141,0,0,.83131],87:[0,.69141,0,0,1.04602],88:[0,.69141,0,0,.71922],89:[.18906,.69141,0,0,.83293],90:[.12604,.69141,0,0,.60201],91:[.24982,.74947,0,0,.27764],93:[.24982,.74947,0,0,.27764],94:[0,.69141,0,0,.49965],97:[0,.47534,0,0,.50046],98:[0,.69141,0,0,.51315],99:[0,.47534,0,0,.38946],100:[0,.62119,0,0,.49857],101:[0,.47534,0,0,.40053],102:[.18906,.69141,0,0,.32626],103:[.18906,.47534,0,0,.5037],104:[.18906,.69141,0,0,.52126],105:[0,.69141,0,0,.27899],106:[0,.69141,0,0,.28088],107:[0,.69141,0,0,.38946],108:[0,.69141,0,0,.27953],109:[0,.47534,0,0,.76676],110:[0,.47534,0,0,.52666],111:[0,.47534,0,0,.48885],112:[.18906,.52396,0,0,.50046],113:[.18906,.47534,0,0,.48912],114:[0,.47534,0,0,.38919],115:[0,.47534,0,0,.44266],116:[0,.62119,0,0,.33301],117:[0,.47534,0,0,.5172],118:[0,.52396,0,0,.5118],119:[0,.52396,0,0,.77351],120:[.18906,.47534,0,0,.38865],121:[.18906,.47534,0,0,.49884],122:[.18906,.47534,0,0,.39054],160:[0,0,0,0,.25],8216:[0,.69141,0,0,.21471],8217:[0,.69141,0,0,.21471],58112:[0,.62119,0,0,.49749],58113:[0,.62119,0,0,.4983],58114:[.18906,.69141,0,0,.33328],58115:[.18906,.69141,0,0,.32923],58116:[.18906,.47534,0,0,.50343],58117:[0,.69141,0,0,.33301],58118:[0,.62119,0,0,.33409],58119:[0,.47534,0,0,.50073]},"Main-Bold":{32:[0,0,0,0,.25],33:[0,.69444,0,0,.35],34:[0,.69444,0,0,.60278],35:[.19444,.69444,0,0,.95833],36:[.05556,.75,0,0,.575],37:[.05556,.75,0,0,.95833],38:[0,.69444,0,0,.89444],39:[0,.69444,0,0,.31944],40:[.25,.75,0,0,.44722],41:[.25,.75,0,0,.44722],42:[0,.75,0,0,.575],43:[.13333,.63333,0,0,.89444],44:[.19444,.15556,0,0,.31944],45:[0,.44444,0,0,.38333],46:[0,.15556,0,0,.31944],47:[.25,.75,0,0,.575],48:[0,.64444,0,0,.575],49:[0,.64444,0,0,.575],50:[0,.64444,0,0,.575],51:[0,.64444,0,0,.575],52:[0,.64444,0,0,.575],53:[0,.64444,0,0,.575],54:[0,.64444,0,0,.575],55:[0,.64444,0,0,.575],56:[0,.64444,0,0,.575],57:[0,.64444,0,0,.575],58:[0,.44444,0,0,.31944],59:[.19444,.44444,0,0,.31944],60:[.08556,.58556,0,0,.89444],61:[-.10889,.39111,0,0,.89444],62:[.08556,.58556,0,0,.89444],63:[0,.69444,0,0,.54305],64:[0,.69444,0,0,.89444],65:[0,.68611,0,0,.86944],66:[0,.68611,0,0,.81805],67:[0,.68611,0,0,.83055],68:[0,.68611,0,0,.88194],69:[0,.68611,0,0,.75555],70:[0,.68611,0,0,.72361],71:[0,.68611,0,0,.90416],72:[0,.68611,0,0,.9],73:[0,.68611,0,0,.43611],74:[0,.68611,0,0,.59444],75:[0,.68611,0,0,.90138],76:[0,.68611,0,0,.69166],77:[0,.68611,0,0,1.09166],78:[0,.68611,0,0,.9],79:[0,.68611,0,0,.86388],80:[0,.68611,0,0,.78611],81:[.19444,.68611,0,0,.86388],82:[0,.68611,0,0,.8625],83:[0,.68611,0,0,.63889],84:[0,.68611,0,0,.8],85:[0,.68611,0,0,.88472],86:[0,.68611,.01597,0,.86944],87:[0,.68611,.01597,0,1.18888],88:[0,.68611,0,0,.86944],89:[0,.68611,.02875,0,.86944],90:[0,.68611,0,0,.70277],91:[.25,.75,0,0,.31944],92:[.25,.75,0,0,.575],93:[.25,.75,0,0,.31944],94:[0,.69444,0,0,.575],95:[.31,.13444,.03194,0,.575],97:[0,.44444,0,0,.55902],98:[0,.69444,0,0,.63889],99:[0,.44444,0,0,.51111],100:[0,.69444,0,0,.63889],101:[0,.44444,0,0,.52708],102:[0,.69444,.10903,0,.35139],103:[.19444,.44444,.01597,0,.575],104:[0,.69444,0,0,.63889],105:[0,.69444,0,0,.31944],106:[.19444,.69444,0,0,.35139],107:[0,.69444,0,0,.60694],108:[0,.69444,0,0,.31944],109:[0,.44444,0,0,.95833],110:[0,.44444,0,0,.63889],111:[0,.44444,0,0,.575],112:[.19444,.44444,0,0,.63889],113:[.19444,.44444,0,0,.60694],114:[0,.44444,0,0,.47361],115:[0,.44444,0,0,.45361],116:[0,.63492,0,0,.44722],117:[0,.44444,0,0,.63889],118:[0,.44444,.01597,0,.60694],119:[0,.44444,.01597,0,.83055],120:[0,.44444,0,0,.60694],121:[.19444,.44444,.01597,0,.60694],122:[0,.44444,0,0,.51111],123:[.25,.75,0,0,.575],124:[.25,.75,0,0,.31944],125:[.25,.75,0,0,.575],126:[.35,.34444,0,0,.575],160:[0,0,0,0,.25],163:[0,.69444,0,0,.86853],168:[0,.69444,0,0,.575],172:[0,.44444,0,0,.76666],176:[0,.69444,0,0,.86944],177:[.13333,.63333,0,0,.89444],184:[.17014,0,0,0,.51111],198:[0,.68611,0,0,1.04166],215:[.13333,.63333,0,0,.89444],216:[.04861,.73472,0,0,.89444],223:[0,.69444,0,0,.59722],230:[0,.44444,0,0,.83055],247:[.13333,.63333,0,0,.89444],248:[.09722,.54167,0,0,.575],305:[0,.44444,0,0,.31944],338:[0,.68611,0,0,1.16944],339:[0,.44444,0,0,.89444],567:[.19444,.44444,0,0,.35139],710:[0,.69444,0,0,.575],711:[0,.63194,0,0,.575],713:[0,.59611,0,0,.575],714:[0,.69444,0,0,.575],715:[0,.69444,0,0,.575],728:[0,.69444,0,0,.575],729:[0,.69444,0,0,.31944],730:[0,.69444,0,0,.86944],732:[0,.69444,0,0,.575],733:[0,.69444,0,0,.575],915:[0,.68611,0,0,.69166],916:[0,.68611,0,0,.95833],920:[0,.68611,0,0,.89444],923:[0,.68611,0,0,.80555],926:[0,.68611,0,0,.76666],928:[0,.68611,0,0,.9],931:[0,.68611,0,0,.83055],933:[0,.68611,0,0,.89444],934:[0,.68611,0,0,.83055],936:[0,.68611,0,0,.89444],937:[0,.68611,0,0,.83055],8211:[0,.44444,.03194,0,.575],8212:[0,.44444,.03194,0,1.14999],8216:[0,.69444,0,0,.31944],8217:[0,.69444,0,0,.31944],8220:[0,.69444,0,0,.60278],8221:[0,.69444,0,0,.60278],8224:[.19444,.69444,0,0,.51111],8225:[.19444,.69444,0,0,.51111],8242:[0,.55556,0,0,.34444],8407:[0,.72444,.15486,0,.575],8463:[0,.69444,0,0,.66759],8465:[0,.69444,0,0,.83055],8467:[0,.69444,0,0,.47361],8472:[.19444,.44444,0,0,.74027],8476:[0,.69444,0,0,.83055],8501:[0,.69444,0,0,.70277],8592:[-.10889,.39111,0,0,1.14999],8593:[.19444,.69444,0,0,.575],8594:[-.10889,.39111,0,0,1.14999],8595:[.19444,.69444,0,0,.575],8596:[-.10889,.39111,0,0,1.14999],8597:[.25,.75,0,0,.575],8598:[.19444,.69444,0,0,1.14999],8599:[.19444,.69444,0,0,1.14999],8600:[.19444,.69444,0,0,1.14999],8601:[.19444,.69444,0,0,1.14999],8636:[-.10889,.39111,0,0,1.14999],8637:[-.10889,.39111,0,0,1.14999],8640:[-.10889,.39111,0,0,1.14999],8641:[-.10889,.39111,0,0,1.14999],8656:[-.10889,.39111,0,0,1.14999],8657:[.19444,.69444,0,0,.70277],8658:[-.10889,.39111,0,0,1.14999],8659:[.19444,.69444,0,0,.70277],8660:[-.10889,.39111,0,0,1.14999],8661:[.25,.75,0,0,.70277],8704:[0,.69444,0,0,.63889],8706:[0,.69444,.06389,0,.62847],8707:[0,.69444,0,0,.63889],8709:[.05556,.75,0,0,.575],8711:[0,.68611,0,0,.95833],8712:[.08556,.58556,0,0,.76666],8715:[.08556,.58556,0,0,.76666],8722:[.13333,.63333,0,0,.89444],8723:[.13333,.63333,0,0,.89444],8725:[.25,.75,0,0,.575],8726:[.25,.75,0,0,.575],8727:[-.02778,.47222,0,0,.575],8728:[-.02639,.47361,0,0,.575],8729:[-.02639,.47361,0,0,.575],8730:[.18,.82,0,0,.95833],8733:[0,.44444,0,0,.89444],8734:[0,.44444,0,0,1.14999],8736:[0,.69224,0,0,.72222],8739:[.25,.75,0,0,.31944],8741:[.25,.75,0,0,.575],8743:[0,.55556,0,0,.76666],8744:[0,.55556,0,0,.76666],8745:[0,.55556,0,0,.76666],8746:[0,.55556,0,0,.76666],8747:[.19444,.69444,.12778,0,.56875],8764:[-.10889,.39111,0,0,.89444],8768:[.19444,.69444,0,0,.31944],8771:[.00222,.50222,0,0,.89444],8773:[.027,.638,0,0,.894],8776:[.02444,.52444,0,0,.89444],8781:[.00222,.50222,0,0,.89444],8801:[.00222,.50222,0,0,.89444],8804:[.19667,.69667,0,0,.89444],8805:[.19667,.69667,0,0,.89444],8810:[.08556,.58556,0,0,1.14999],8811:[.08556,.58556,0,0,1.14999],8826:[.08556,.58556,0,0,.89444],8827:[.08556,.58556,0,0,.89444],8834:[.08556,.58556,0,0,.89444],8835:[.08556,.58556,0,0,.89444],8838:[.19667,.69667,0,0,.89444],8839:[.19667,.69667,0,0,.89444],8846:[0,.55556,0,0,.76666],8849:[.19667,.69667,0,0,.89444],8850:[.19667,.69667,0,0,.89444],8851:[0,.55556,0,0,.76666],8852:[0,.55556,0,0,.76666],8853:[.13333,.63333,0,0,.89444],8854:[.13333,.63333,0,0,.89444],8855:[.13333,.63333,0,0,.89444],8856:[.13333,.63333,0,0,.89444],8857:[.13333,.63333,0,0,.89444],8866:[0,.69444,0,0,.70277],8867:[0,.69444,0,0,.70277],8868:[0,.69444,0,0,.89444],8869:[0,.69444,0,0,.89444],8900:[-.02639,.47361,0,0,.575],8901:[-.02639,.47361,0,0,.31944],8902:[-.02778,.47222,0,0,.575],8968:[.25,.75,0,0,.51111],8969:[.25,.75,0,0,.51111],8970:[.25,.75,0,0,.51111],8971:[.25,.75,0,0,.51111],8994:[-.13889,.36111,0,0,1.14999],8995:[-.13889,.36111,0,0,1.14999],9651:[.19444,.69444,0,0,1.02222],9657:[-.02778,.47222,0,0,.575],9661:[.19444,.69444,0,0,1.02222],9667:[-.02778,.47222,0,0,.575],9711:[.19444,.69444,0,0,1.14999],9824:[.12963,.69444,0,0,.89444],9825:[.12963,.69444,0,0,.89444],9826:[.12963,.69444,0,0,.89444],9827:[.12963,.69444,0,0,.89444],9837:[0,.75,0,0,.44722],9838:[.19444,.69444,0,0,.44722],9839:[.19444,.69444,0,0,.44722],10216:[.25,.75,0,0,.44722],10217:[.25,.75,0,0,.44722],10815:[0,.68611,0,0,.9],10927:[.19667,.69667,0,0,.89444],10928:[.19667,.69667,0,0,.89444],57376:[.19444,.69444,0,0,0]},"Main-BoldItalic":{32:[0,0,0,0,.25],33:[0,.69444,.11417,0,.38611],34:[0,.69444,.07939,0,.62055],35:[.19444,.69444,.06833,0,.94444],37:[.05556,.75,.12861,0,.94444],38:[0,.69444,.08528,0,.88555],39:[0,.69444,.12945,0,.35555],40:[.25,.75,.15806,0,.47333],41:[.25,.75,.03306,0,.47333],42:[0,.75,.14333,0,.59111],43:[.10333,.60333,.03306,0,.88555],44:[.19444,.14722,0,0,.35555],45:[0,.44444,.02611,0,.41444],46:[0,.14722,0,0,.35555],47:[.25,.75,.15806,0,.59111],48:[0,.64444,.13167,0,.59111],49:[0,.64444,.13167,0,.59111],50:[0,.64444,.13167,0,.59111],51:[0,.64444,.13167,0,.59111],52:[.19444,.64444,.13167,0,.59111],53:[0,.64444,.13167,0,.59111],54:[0,.64444,.13167,0,.59111],55:[.19444,.64444,.13167,0,.59111],56:[0,.64444,.13167,0,.59111],57:[0,.64444,.13167,0,.59111],58:[0,.44444,.06695,0,.35555],59:[.19444,.44444,.06695,0,.35555],61:[-.10889,.39111,.06833,0,.88555],63:[0,.69444,.11472,0,.59111],64:[0,.69444,.09208,0,.88555],65:[0,.68611,0,0,.86555],66:[0,.68611,.0992,0,.81666],67:[0,.68611,.14208,0,.82666],68:[0,.68611,.09062,0,.87555],69:[0,.68611,.11431,0,.75666],70:[0,.68611,.12903,0,.72722],71:[0,.68611,.07347,0,.89527],72:[0,.68611,.17208,0,.8961],73:[0,.68611,.15681,0,.47166],74:[0,.68611,.145,0,.61055],75:[0,.68611,.14208,0,.89499],76:[0,.68611,0,0,.69777],77:[0,.68611,.17208,0,1.07277],78:[0,.68611,.17208,0,.8961],79:[0,.68611,.09062,0,.85499],80:[0,.68611,.0992,0,.78721],81:[.19444,.68611,.09062,0,.85499],82:[0,.68611,.02559,0,.85944],83:[0,.68611,.11264,0,.64999],84:[0,.68611,.12903,0,.7961],85:[0,.68611,.17208,0,.88083],86:[0,.68611,.18625,0,.86555],87:[0,.68611,.18625,0,1.15999],88:[0,.68611,.15681,0,.86555],89:[0,.68611,.19803,0,.86555],90:[0,.68611,.14208,0,.70888],91:[.25,.75,.1875,0,.35611],93:[.25,.75,.09972,0,.35611],94:[0,.69444,.06709,0,.59111],95:[.31,.13444,.09811,0,.59111],97:[0,.44444,.09426,0,.59111],98:[0,.69444,.07861,0,.53222],99:[0,.44444,.05222,0,.53222],100:[0,.69444,.10861,0,.59111],101:[0,.44444,.085,0,.53222],102:[.19444,.69444,.21778,0,.4],103:[.19444,.44444,.105,0,.53222],104:[0,.69444,.09426,0,.59111],105:[0,.69326,.11387,0,.35555],106:[.19444,.69326,.1672,0,.35555],107:[0,.69444,.11111,0,.53222],108:[0,.69444,.10861,0,.29666],109:[0,.44444,.09426,0,.94444],110:[0,.44444,.09426,0,.64999],111:[0,.44444,.07861,0,.59111],112:[.19444,.44444,.07861,0,.59111],113:[.19444,.44444,.105,0,.53222],114:[0,.44444,.11111,0,.50167],115:[0,.44444,.08167,0,.48694],116:[0,.63492,.09639,0,.385],117:[0,.44444,.09426,0,.62055],118:[0,.44444,.11111,0,.53222],119:[0,.44444,.11111,0,.76777],120:[0,.44444,.12583,0,.56055],121:[.19444,.44444,.105,0,.56166],122:[0,.44444,.13889,0,.49055],126:[.35,.34444,.11472,0,.59111],160:[0,0,0,0,.25],168:[0,.69444,.11473,0,.59111],176:[0,.69444,0,0,.94888],184:[.17014,0,0,0,.53222],198:[0,.68611,.11431,0,1.02277],216:[.04861,.73472,.09062,0,.88555],223:[.19444,.69444,.09736,0,.665],230:[0,.44444,.085,0,.82666],248:[.09722,.54167,.09458,0,.59111],305:[0,.44444,.09426,0,.35555],338:[0,.68611,.11431,0,1.14054],339:[0,.44444,.085,0,.82666],567:[.19444,.44444,.04611,0,.385],710:[0,.69444,.06709,0,.59111],711:[0,.63194,.08271,0,.59111],713:[0,.59444,.10444,0,.59111],714:[0,.69444,.08528,0,.59111],715:[0,.69444,0,0,.59111],728:[0,.69444,.10333,0,.59111],729:[0,.69444,.12945,0,.35555],730:[0,.69444,0,0,.94888],732:[0,.69444,.11472,0,.59111],733:[0,.69444,.11472,0,.59111],915:[0,.68611,.12903,0,.69777],916:[0,.68611,0,0,.94444],920:[0,.68611,.09062,0,.88555],923:[0,.68611,0,0,.80666],926:[0,.68611,.15092,0,.76777],928:[0,.68611,.17208,0,.8961],931:[0,.68611,.11431,0,.82666],933:[0,.68611,.10778,0,.88555],934:[0,.68611,.05632,0,.82666],936:[0,.68611,.10778,0,.88555],937:[0,.68611,.0992,0,.82666],8211:[0,.44444,.09811,0,.59111],8212:[0,.44444,.09811,0,1.18221],8216:[0,.69444,.12945,0,.35555],8217:[0,.69444,.12945,0,.35555],8220:[0,.69444,.16772,0,.62055],8221:[0,.69444,.07939,0,.62055]},"Main-Italic":{32:[0,0,0,0,.25],33:[0,.69444,.12417,0,.30667],34:[0,.69444,.06961,0,.51444],35:[.19444,.69444,.06616,0,.81777],37:[.05556,.75,.13639,0,.81777],38:[0,.69444,.09694,0,.76666],39:[0,.69444,.12417,0,.30667],40:[.25,.75,.16194,0,.40889],41:[.25,.75,.03694,0,.40889],42:[0,.75,.14917,0,.51111],43:[.05667,.56167,.03694,0,.76666],44:[.19444,.10556,0,0,.30667],45:[0,.43056,.02826,0,.35778],46:[0,.10556,0,0,.30667],47:[.25,.75,.16194,0,.51111],48:[0,.64444,.13556,0,.51111],49:[0,.64444,.13556,0,.51111],50:[0,.64444,.13556,0,.51111],51:[0,.64444,.13556,0,.51111],52:[.19444,.64444,.13556,0,.51111],53:[0,.64444,.13556,0,.51111],54:[0,.64444,.13556,0,.51111],55:[.19444,.64444,.13556,0,.51111],56:[0,.64444,.13556,0,.51111],57:[0,.64444,.13556,0,.51111],58:[0,.43056,.0582,0,.30667],59:[.19444,.43056,.0582,0,.30667],61:[-.13313,.36687,.06616,0,.76666],63:[0,.69444,.1225,0,.51111],64:[0,.69444,.09597,0,.76666],65:[0,.68333,0,0,.74333],66:[0,.68333,.10257,0,.70389],67:[0,.68333,.14528,0,.71555],68:[0,.68333,.09403,0,.755],69:[0,.68333,.12028,0,.67833],70:[0,.68333,.13305,0,.65277],71:[0,.68333,.08722,0,.77361],72:[0,.68333,.16389,0,.74333],73:[0,.68333,.15806,0,.38555],74:[0,.68333,.14028,0,.525],75:[0,.68333,.14528,0,.76888],76:[0,.68333,0,0,.62722],77:[0,.68333,.16389,0,.89666],78:[0,.68333,.16389,0,.74333],79:[0,.68333,.09403,0,.76666],80:[0,.68333,.10257,0,.67833],81:[.19444,.68333,.09403,0,.76666],82:[0,.68333,.03868,0,.72944],83:[0,.68333,.11972,0,.56222],84:[0,.68333,.13305,0,.71555],85:[0,.68333,.16389,0,.74333],86:[0,.68333,.18361,0,.74333],87:[0,.68333,.18361,0,.99888],88:[0,.68333,.15806,0,.74333],89:[0,.68333,.19383,0,.74333],90:[0,.68333,.14528,0,.61333],91:[.25,.75,.1875,0,.30667],93:[.25,.75,.10528,0,.30667],94:[0,.69444,.06646,0,.51111],95:[.31,.12056,.09208,0,.51111],97:[0,.43056,.07671,0,.51111],98:[0,.69444,.06312,0,.46],99:[0,.43056,.05653,0,.46],100:[0,.69444,.10333,0,.51111],101:[0,.43056,.07514,0,.46],102:[.19444,.69444,.21194,0,.30667],103:[.19444,.43056,.08847,0,.46],104:[0,.69444,.07671,0,.51111],105:[0,.65536,.1019,0,.30667],106:[.19444,.65536,.14467,0,.30667],107:[0,.69444,.10764,0,.46],108:[0,.69444,.10333,0,.25555],109:[0,.43056,.07671,0,.81777],110:[0,.43056,.07671,0,.56222],111:[0,.43056,.06312,0,.51111],112:[.19444,.43056,.06312,0,.51111],113:[.19444,.43056,.08847,0,.46],114:[0,.43056,.10764,0,.42166],115:[0,.43056,.08208,0,.40889],116:[0,.61508,.09486,0,.33222],117:[0,.43056,.07671,0,.53666],118:[0,.43056,.10764,0,.46],119:[0,.43056,.10764,0,.66444],120:[0,.43056,.12042,0,.46389],121:[.19444,.43056,.08847,0,.48555],122:[0,.43056,.12292,0,.40889],126:[.35,.31786,.11585,0,.51111],160:[0,0,0,0,.25],168:[0,.66786,.10474,0,.51111],176:[0,.69444,0,0,.83129],184:[.17014,0,0,0,.46],198:[0,.68333,.12028,0,.88277],216:[.04861,.73194,.09403,0,.76666],223:[.19444,.69444,.10514,0,.53666],230:[0,.43056,.07514,0,.71555],248:[.09722,.52778,.09194,0,.51111],338:[0,.68333,.12028,0,.98499],339:[0,.43056,.07514,0,.71555],710:[0,.69444,.06646,0,.51111],711:[0,.62847,.08295,0,.51111],713:[0,.56167,.10333,0,.51111],714:[0,.69444,.09694,0,.51111],715:[0,.69444,0,0,.51111],728:[0,.69444,.10806,0,.51111],729:[0,.66786,.11752,0,.30667],730:[0,.69444,0,0,.83129],732:[0,.66786,.11585,0,.51111],733:[0,.69444,.1225,0,.51111],915:[0,.68333,.13305,0,.62722],916:[0,.68333,0,0,.81777],920:[0,.68333,.09403,0,.76666],923:[0,.68333,0,0,.69222],926:[0,.68333,.15294,0,.66444],928:[0,.68333,.16389,0,.74333],931:[0,.68333,.12028,0,.71555],933:[0,.68333,.11111,0,.76666],934:[0,.68333,.05986,0,.71555],936:[0,.68333,.11111,0,.76666],937:[0,.68333,.10257,0,.71555],8211:[0,.43056,.09208,0,.51111],8212:[0,.43056,.09208,0,1.02222],8216:[0,.69444,.12417,0,.30667],8217:[0,.69444,.12417,0,.30667],8220:[0,.69444,.1685,0,.51444],8221:[0,.69444,.06961,0,.51444],8463:[0,.68889,0,0,.54028]},"Main-Regular":{32:[0,0,0,0,.25],33:[0,.69444,0,0,.27778],34:[0,.69444,0,0,.5],35:[.19444,.69444,0,0,.83334],36:[.05556,.75,0,0,.5],37:[.05556,.75,0,0,.83334],38:[0,.69444,0,0,.77778],39:[0,.69444,0,0,.27778],40:[.25,.75,0,0,.38889],41:[.25,.75,0,0,.38889],42:[0,.75,0,0,.5],43:[.08333,.58333,0,0,.77778],44:[.19444,.10556,0,0,.27778],45:[0,.43056,0,0,.33333],46:[0,.10556,0,0,.27778],47:[.25,.75,0,0,.5],48:[0,.64444,0,0,.5],49:[0,.64444,0,0,.5],50:[0,.64444,0,0,.5],51:[0,.64444,0,0,.5],52:[0,.64444,0,0,.5],53:[0,.64444,0,0,.5],54:[0,.64444,0,0,.5],55:[0,.64444,0,0,.5],56:[0,.64444,0,0,.5],57:[0,.64444,0,0,.5],58:[0,.43056,0,0,.27778],59:[.19444,.43056,0,0,.27778],60:[.0391,.5391,0,0,.77778],61:[-.13313,.36687,0,0,.77778],62:[.0391,.5391,0,0,.77778],63:[0,.69444,0,0,.47222],64:[0,.69444,0,0,.77778],65:[0,.68333,0,0,.75],66:[0,.68333,0,0,.70834],67:[0,.68333,0,0,.72222],68:[0,.68333,0,0,.76389],69:[0,.68333,0,0,.68056],70:[0,.68333,0,0,.65278],71:[0,.68333,0,0,.78472],72:[0,.68333,0,0,.75],73:[0,.68333,0,0,.36111],74:[0,.68333,0,0,.51389],75:[0,.68333,0,0,.77778],76:[0,.68333,0,0,.625],77:[0,.68333,0,0,.91667],78:[0,.68333,0,0,.75],79:[0,.68333,0,0,.77778],80:[0,.68333,0,0,.68056],81:[.19444,.68333,0,0,.77778],82:[0,.68333,0,0,.73611],83:[0,.68333,0,0,.55556],84:[0,.68333,0,0,.72222],85:[0,.68333,0,0,.75],86:[0,.68333,.01389,0,.75],87:[0,.68333,.01389,0,1.02778],88:[0,.68333,0,0,.75],89:[0,.68333,.025,0,.75],90:[0,.68333,0,0,.61111],91:[.25,.75,0,0,.27778],92:[.25,.75,0,0,.5],93:[.25,.75,0,0,.27778],94:[0,.69444,0,0,.5],95:[.31,.12056,.02778,0,.5],97:[0,.43056,0,0,.5],98:[0,.69444,0,0,.55556],99:[0,.43056,0,0,.44445],100:[0,.69444,0,0,.55556],101:[0,.43056,0,0,.44445],102:[0,.69444,.07778,0,.30556],103:[.19444,.43056,.01389,0,.5],104:[0,.69444,0,0,.55556],105:[0,.66786,0,0,.27778],106:[.19444,.66786,0,0,.30556],107:[0,.69444,0,0,.52778],108:[0,.69444,0,0,.27778],109:[0,.43056,0,0,.83334],110:[0,.43056,0,0,.55556],111:[0,.43056,0,0,.5],112:[.19444,.43056,0,0,.55556],113:[.19444,.43056,0,0,.52778],114:[0,.43056,0,0,.39167],115:[0,.43056,0,0,.39445],116:[0,.61508,0,0,.38889],117:[0,.43056,0,0,.55556],118:[0,.43056,.01389,0,.52778],119:[0,.43056,.01389,0,.72222],120:[0,.43056,0,0,.52778],121:[.19444,.43056,.01389,0,.52778],122:[0,.43056,0,0,.44445],123:[.25,.75,0,0,.5],124:[.25,.75,0,0,.27778],125:[.25,.75,0,0,.5],126:[.35,.31786,0,0,.5],160:[0,0,0,0,.25],163:[0,.69444,0,0,.76909],167:[.19444,.69444,0,0,.44445],168:[0,.66786,0,0,.5],172:[0,.43056,0,0,.66667],176:[0,.69444,0,0,.75],177:[.08333,.58333,0,0,.77778],182:[.19444,.69444,0,0,.61111],184:[.17014,0,0,0,.44445],198:[0,.68333,0,0,.90278],215:[.08333,.58333,0,0,.77778],216:[.04861,.73194,0,0,.77778],223:[0,.69444,0,0,.5],230:[0,.43056,0,0,.72222],247:[.08333,.58333,0,0,.77778],248:[.09722,.52778,0,0,.5],305:[0,.43056,0,0,.27778],338:[0,.68333,0,0,1.01389],339:[0,.43056,0,0,.77778],567:[.19444,.43056,0,0,.30556],710:[0,.69444,0,0,.5],711:[0,.62847,0,0,.5],713:[0,.56778,0,0,.5],714:[0,.69444,0,0,.5],715:[0,.69444,0,0,.5],728:[0,.69444,0,0,.5],729:[0,.66786,0,0,.27778],730:[0,.69444,0,0,.75],732:[0,.66786,0,0,.5],733:[0,.69444,0,0,.5],915:[0,.68333,0,0,.625],916:[0,.68333,0,0,.83334],920:[0,.68333,0,0,.77778],923:[0,.68333,0,0,.69445],926:[0,.68333,0,0,.66667],928:[0,.68333,0,0,.75],931:[0,.68333,0,0,.72222],933:[0,.68333,0,0,.77778],934:[0,.68333,0,0,.72222],936:[0,.68333,0,0,.77778],937:[0,.68333,0,0,.72222],8211:[0,.43056,.02778,0,.5],8212:[0,.43056,.02778,0,1],8216:[0,.69444,0,0,.27778],8217:[0,.69444,0,0,.27778],8220:[0,.69444,0,0,.5],8221:[0,.69444,0,0,.5],8224:[.19444,.69444,0,0,.44445],8225:[.19444,.69444,0,0,.44445],8230:[0,.123,0,0,1.172],8242:[0,.55556,0,0,.275],8407:[0,.71444,.15382,0,.5],8463:[0,.68889,0,0,.54028],8465:[0,.69444,0,0,.72222],8467:[0,.69444,0,.11111,.41667],8472:[.19444,.43056,0,.11111,.63646],8476:[0,.69444,0,0,.72222],8501:[0,.69444,0,0,.61111],8592:[-.13313,.36687,0,0,1],8593:[.19444,.69444,0,0,.5],8594:[-.13313,.36687,0,0,1],8595:[.19444,.69444,0,0,.5],8596:[-.13313,.36687,0,0,1],8597:[.25,.75,0,0,.5],8598:[.19444,.69444,0,0,1],8599:[.19444,.69444,0,0,1],8600:[.19444,.69444,0,0,1],8601:[.19444,.69444,0,0,1],8614:[.011,.511,0,0,1],8617:[.011,.511,0,0,1.126],8618:[.011,.511,0,0,1.126],8636:[-.13313,.36687,0,0,1],8637:[-.13313,.36687,0,0,1],8640:[-.13313,.36687,0,0,1],8641:[-.13313,.36687,0,0,1],8652:[.011,.671,0,0,1],8656:[-.13313,.36687,0,0,1],8657:[.19444,.69444,0,0,.61111],8658:[-.13313,.36687,0,0,1],8659:[.19444,.69444,0,0,.61111],8660:[-.13313,.36687,0,0,1],8661:[.25,.75,0,0,.61111],8704:[0,.69444,0,0,.55556],8706:[0,.69444,.05556,.08334,.5309],8707:[0,.69444,0,0,.55556],8709:[.05556,.75,0,0,.5],8711:[0,.68333,0,0,.83334],8712:[.0391,.5391,0,0,.66667],8715:[.0391,.5391,0,0,.66667],8722:[.08333,.58333,0,0,.77778],8723:[.08333,.58333,0,0,.77778],8725:[.25,.75,0,0,.5],8726:[.25,.75,0,0,.5],8727:[-.03472,.46528,0,0,.5],8728:[-.05555,.44445,0,0,.5],8729:[-.05555,.44445,0,0,.5],8730:[.2,.8,0,0,.83334],8733:[0,.43056,0,0,.77778],8734:[0,.43056,0,0,1],8736:[0,.69224,0,0,.72222],8739:[.25,.75,0,0,.27778],8741:[.25,.75,0,0,.5],8743:[0,.55556,0,0,.66667],8744:[0,.55556,0,0,.66667],8745:[0,.55556,0,0,.66667],8746:[0,.55556,0,0,.66667],8747:[.19444,.69444,.11111,0,.41667],8764:[-.13313,.36687,0,0,.77778],8768:[.19444,.69444,0,0,.27778],8771:[-.03625,.46375,0,0,.77778],8773:[-.022,.589,0,0,.778],8776:[-.01688,.48312,0,0,.77778],8781:[-.03625,.46375,0,0,.77778],8784:[-.133,.673,0,0,.778],8801:[-.03625,.46375,0,0,.77778],8804:[.13597,.63597,0,0,.77778],8805:[.13597,.63597,0,0,.77778],8810:[.0391,.5391,0,0,1],8811:[.0391,.5391,0,0,1],8826:[.0391,.5391,0,0,.77778],8827:[.0391,.5391,0,0,.77778],8834:[.0391,.5391,0,0,.77778],8835:[.0391,.5391,0,0,.77778],8838:[.13597,.63597,0,0,.77778],8839:[.13597,.63597,0,0,.77778],8846:[0,.55556,0,0,.66667],8849:[.13597,.63597,0,0,.77778],8850:[.13597,.63597,0,0,.77778],8851:[0,.55556,0,0,.66667],8852:[0,.55556,0,0,.66667],8853:[.08333,.58333,0,0,.77778],8854:[.08333,.58333,0,0,.77778],8855:[.08333,.58333,0,0,.77778],8856:[.08333,.58333,0,0,.77778],8857:[.08333,.58333,0,0,.77778],8866:[0,.69444,0,0,.61111],8867:[0,.69444,0,0,.61111],8868:[0,.69444,0,0,.77778],8869:[0,.69444,0,0,.77778],8872:[.249,.75,0,0,.867],8900:[-.05555,.44445,0,0,.5],8901:[-.05555,.44445,0,0,.27778],8902:[-.03472,.46528,0,0,.5],8904:[.005,.505,0,0,.9],8942:[.03,.903,0,0,.278],8943:[-.19,.313,0,0,1.172],8945:[-.1,.823,0,0,1.282],8968:[.25,.75,0,0,.44445],8969:[.25,.75,0,0,.44445],8970:[.25,.75,0,0,.44445],8971:[.25,.75,0,0,.44445],8994:[-.14236,.35764,0,0,1],8995:[-.14236,.35764,0,0,1],9136:[.244,.744,0,0,.412],9137:[.244,.745,0,0,.412],9651:[.19444,.69444,0,0,.88889],9657:[-.03472,.46528,0,0,.5],9661:[.19444,.69444,0,0,.88889],9667:[-.03472,.46528,0,0,.5],9711:[.19444,.69444,0,0,1],9824:[.12963,.69444,0,0,.77778],9825:[.12963,.69444,0,0,.77778],9826:[.12963,.69444,0,0,.77778],9827:[.12963,.69444,0,0,.77778],9837:[0,.75,0,0,.38889],9838:[.19444,.69444,0,0,.38889],9839:[.19444,.69444,0,0,.38889],10216:[.25,.75,0,0,.38889],10217:[.25,.75,0,0,.38889],10222:[.244,.744,0,0,.412],10223:[.244,.745,0,0,.412],10229:[.011,.511,0,0,1.609],10230:[.011,.511,0,0,1.638],10231:[.011,.511,0,0,1.859],10232:[.024,.525,0,0,1.609],10233:[.024,.525,0,0,1.638],10234:[.024,.525,0,0,1.858],10236:[.011,.511,0,0,1.638],10815:[0,.68333,0,0,.75],10927:[.13597,.63597,0,0,.77778],10928:[.13597,.63597,0,0,.77778],57376:[.19444,.69444,0,0,0]},"Math-BoldItalic":{32:[0,0,0,0,.25],48:[0,.44444,0,0,.575],49:[0,.44444,0,0,.575],50:[0,.44444,0,0,.575],51:[.19444,.44444,0,0,.575],52:[.19444,.44444,0,0,.575],53:[.19444,.44444,0,0,.575],54:[0,.64444,0,0,.575],55:[.19444,.44444,0,0,.575],56:[0,.64444,0,0,.575],57:[.19444,.44444,0,0,.575],65:[0,.68611,0,0,.86944],66:[0,.68611,.04835,0,.8664],67:[0,.68611,.06979,0,.81694],68:[0,.68611,.03194,0,.93812],69:[0,.68611,.05451,0,.81007],70:[0,.68611,.15972,0,.68889],71:[0,.68611,0,0,.88673],72:[0,.68611,.08229,0,.98229],73:[0,.68611,.07778,0,.51111],74:[0,.68611,.10069,0,.63125],75:[0,.68611,.06979,0,.97118],76:[0,.68611,0,0,.75555],77:[0,.68611,.11424,0,1.14201],78:[0,.68611,.11424,0,.95034],79:[0,.68611,.03194,0,.83666],80:[0,.68611,.15972,0,.72309],81:[.19444,.68611,0,0,.86861],82:[0,.68611,.00421,0,.87235],83:[0,.68611,.05382,0,.69271],84:[0,.68611,.15972,0,.63663],85:[0,.68611,.11424,0,.80027],86:[0,.68611,.25555,0,.67778],87:[0,.68611,.15972,0,1.09305],88:[0,.68611,.07778,0,.94722],89:[0,.68611,.25555,0,.67458],90:[0,.68611,.06979,0,.77257],97:[0,.44444,0,0,.63287],98:[0,.69444,0,0,.52083],99:[0,.44444,0,0,.51342],100:[0,.69444,0,0,.60972],101:[0,.44444,0,0,.55361],102:[.19444,.69444,.11042,0,.56806],103:[.19444,.44444,.03704,0,.5449],104:[0,.69444,0,0,.66759],105:[0,.69326,0,0,.4048],106:[.19444,.69326,.0622,0,.47083],107:[0,.69444,.01852,0,.6037],108:[0,.69444,.0088,0,.34815],109:[0,.44444,0,0,1.0324],110:[0,.44444,0,0,.71296],111:[0,.44444,0,0,.58472],112:[.19444,.44444,0,0,.60092],113:[.19444,.44444,.03704,0,.54213],114:[0,.44444,.03194,0,.5287],115:[0,.44444,0,0,.53125],116:[0,.63492,0,0,.41528],117:[0,.44444,0,0,.68102],118:[0,.44444,.03704,0,.56666],119:[0,.44444,.02778,0,.83148],120:[0,.44444,0,0,.65903],121:[.19444,.44444,.03704,0,.59028],122:[0,.44444,.04213,0,.55509],160:[0,0,0,0,.25],915:[0,.68611,.15972,0,.65694],916:[0,.68611,0,0,.95833],920:[0,.68611,.03194,0,.86722],923:[0,.68611,0,0,.80555],926:[0,.68611,.07458,0,.84125],928:[0,.68611,.08229,0,.98229],931:[0,.68611,.05451,0,.88507],933:[0,.68611,.15972,0,.67083],934:[0,.68611,0,0,.76666],936:[0,.68611,.11653,0,.71402],937:[0,.68611,.04835,0,.8789],945:[0,.44444,0,0,.76064],946:[.19444,.69444,.03403,0,.65972],947:[.19444,.44444,.06389,0,.59003],948:[0,.69444,.03819,0,.52222],949:[0,.44444,0,0,.52882],950:[.19444,.69444,.06215,0,.50833],951:[.19444,.44444,.03704,0,.6],952:[0,.69444,.03194,0,.5618],953:[0,.44444,0,0,.41204],954:[0,.44444,0,0,.66759],955:[0,.69444,0,0,.67083],956:[.19444,.44444,0,0,.70787],957:[0,.44444,.06898,0,.57685],958:[.19444,.69444,.03021,0,.50833],959:[0,.44444,0,0,.58472],960:[0,.44444,.03704,0,.68241],961:[.19444,.44444,0,0,.6118],962:[.09722,.44444,.07917,0,.42361],963:[0,.44444,.03704,0,.68588],964:[0,.44444,.13472,0,.52083],965:[0,.44444,.03704,0,.63055],966:[.19444,.44444,0,0,.74722],967:[.19444,.44444,0,0,.71805],968:[.19444,.69444,.03704,0,.75833],969:[0,.44444,.03704,0,.71782],977:[0,.69444,0,0,.69155],981:[.19444,.69444,0,0,.7125],982:[0,.44444,.03194,0,.975],1009:[.19444,.44444,0,0,.6118],1013:[0,.44444,0,0,.48333],57649:[0,.44444,0,0,.39352],57911:[.19444,.44444,0,0,.43889]},"Math-Italic":{32:[0,0,0,0,.25],48:[0,.43056,0,0,.5],49:[0,.43056,0,0,.5],50:[0,.43056,0,0,.5],51:[.19444,.43056,0,0,.5],52:[.19444,.43056,0,0,.5],53:[.19444,.43056,0,0,.5],54:[0,.64444,0,0,.5],55:[.19444,.43056,0,0,.5],56:[0,.64444,0,0,.5],57:[.19444,.43056,0,0,.5],65:[0,.68333,0,.13889,.75],66:[0,.68333,.05017,.08334,.75851],67:[0,.68333,.07153,.08334,.71472],68:[0,.68333,.02778,.05556,.82792],69:[0,.68333,.05764,.08334,.7382],70:[0,.68333,.13889,.08334,.64306],71:[0,.68333,0,.08334,.78625],72:[0,.68333,.08125,.05556,.83125],73:[0,.68333,.07847,.11111,.43958],74:[0,.68333,.09618,.16667,.55451],75:[0,.68333,.07153,.05556,.84931],76:[0,.68333,0,.02778,.68056],77:[0,.68333,.10903,.08334,.97014],78:[0,.68333,.10903,.08334,.80347],79:[0,.68333,.02778,.08334,.76278],80:[0,.68333,.13889,.08334,.64201],81:[.19444,.68333,0,.08334,.79056],82:[0,.68333,.00773,.08334,.75929],83:[0,.68333,.05764,.08334,.6132],84:[0,.68333,.13889,.08334,.58438],85:[0,.68333,.10903,.02778,.68278],86:[0,.68333,.22222,0,.58333],87:[0,.68333,.13889,0,.94445],88:[0,.68333,.07847,.08334,.82847],89:[0,.68333,.22222,0,.58056],90:[0,.68333,.07153,.08334,.68264],97:[0,.43056,0,0,.52859],98:[0,.69444,0,0,.42917],99:[0,.43056,0,.05556,.43276],100:[0,.69444,0,.16667,.52049],101:[0,.43056,0,.05556,.46563],102:[.19444,.69444,.10764,.16667,.48959],103:[.19444,.43056,.03588,.02778,.47697],104:[0,.69444,0,0,.57616],105:[0,.65952,0,0,.34451],106:[.19444,.65952,.05724,0,.41181],107:[0,.69444,.03148,0,.5206],108:[0,.69444,.01968,.08334,.29838],109:[0,.43056,0,0,.87801],110:[0,.43056,0,0,.60023],111:[0,.43056,0,.05556,.48472],112:[.19444,.43056,0,.08334,.50313],113:[.19444,.43056,.03588,.08334,.44641],114:[0,.43056,.02778,.05556,.45116],115:[0,.43056,0,.05556,.46875],116:[0,.61508,0,.08334,.36111],117:[0,.43056,0,.02778,.57246],118:[0,.43056,.03588,.02778,.48472],119:[0,.43056,.02691,.08334,.71592],120:[0,.43056,0,.02778,.57153],121:[.19444,.43056,.03588,.05556,.49028],122:[0,.43056,.04398,.05556,.46505],160:[0,0,0,0,.25],915:[0,.68333,.13889,.08334,.61528],916:[0,.68333,0,.16667,.83334],920:[0,.68333,.02778,.08334,.76278],923:[0,.68333,0,.16667,.69445],926:[0,.68333,.07569,.08334,.74236],928:[0,.68333,.08125,.05556,.83125],931:[0,.68333,.05764,.08334,.77986],933:[0,.68333,.13889,.05556,.58333],934:[0,.68333,0,.08334,.66667],936:[0,.68333,.11,.05556,.61222],937:[0,.68333,.05017,.08334,.7724],945:[0,.43056,.0037,.02778,.6397],946:[.19444,.69444,.05278,.08334,.56563],947:[.19444,.43056,.05556,0,.51773],948:[0,.69444,.03785,.05556,.44444],949:[0,.43056,0,.08334,.46632],950:[.19444,.69444,.07378,.08334,.4375],951:[.19444,.43056,.03588,.05556,.49653],952:[0,.69444,.02778,.08334,.46944],953:[0,.43056,0,.05556,.35394],954:[0,.43056,0,0,.57616],955:[0,.69444,0,0,.58334],956:[.19444,.43056,0,.02778,.60255],957:[0,.43056,.06366,.02778,.49398],958:[.19444,.69444,.04601,.11111,.4375],959:[0,.43056,0,.05556,.48472],960:[0,.43056,.03588,0,.57003],961:[.19444,.43056,0,.08334,.51702],962:[.09722,.43056,.07986,.08334,.36285],963:[0,.43056,.03588,0,.57141],964:[0,.43056,.1132,.02778,.43715],965:[0,.43056,.03588,.02778,.54028],966:[.19444,.43056,0,.08334,.65417],967:[.19444,.43056,0,.05556,.62569],968:[.19444,.69444,.03588,.11111,.65139],969:[0,.43056,.03588,0,.62245],977:[0,.69444,0,.08334,.59144],981:[.19444,.69444,0,.08334,.59583],982:[0,.43056,.02778,0,.82813],1009:[.19444,.43056,0,.08334,.51702],1013:[0,.43056,0,.05556,.4059],57649:[0,.43056,0,.02778,.32246],57911:[.19444,.43056,0,.08334,.38403]},"SansSerif-Bold":{32:[0,0,0,0,.25],33:[0,.69444,0,0,.36667],34:[0,.69444,0,0,.55834],35:[.19444,.69444,0,0,.91667],36:[.05556,.75,0,0,.55],37:[.05556,.75,0,0,1.02912],38:[0,.69444,0,0,.83056],39:[0,.69444,0,0,.30556],40:[.25,.75,0,0,.42778],41:[.25,.75,0,0,.42778],42:[0,.75,0,0,.55],43:[.11667,.61667,0,0,.85556],44:[.10556,.13056,0,0,.30556],45:[0,.45833,0,0,.36667],46:[0,.13056,0,0,.30556],47:[.25,.75,0,0,.55],48:[0,.69444,0,0,.55],49:[0,.69444,0,0,.55],50:[0,.69444,0,0,.55],51:[0,.69444,0,0,.55],52:[0,.69444,0,0,.55],53:[0,.69444,0,0,.55],54:[0,.69444,0,0,.55],55:[0,.69444,0,0,.55],56:[0,.69444,0,0,.55],57:[0,.69444,0,0,.55],58:[0,.45833,0,0,.30556],59:[.10556,.45833,0,0,.30556],61:[-.09375,.40625,0,0,.85556],63:[0,.69444,0,0,.51945],64:[0,.69444,0,0,.73334],65:[0,.69444,0,0,.73334],66:[0,.69444,0,0,.73334],67:[0,.69444,0,0,.70278],68:[0,.69444,0,0,.79445],69:[0,.69444,0,0,.64167],70:[0,.69444,0,0,.61111],71:[0,.69444,0,0,.73334],72:[0,.69444,0,0,.79445],73:[0,.69444,0,0,.33056],74:[0,.69444,0,0,.51945],75:[0,.69444,0,0,.76389],76:[0,.69444,0,0,.58056],77:[0,.69444,0,0,.97778],78:[0,.69444,0,0,.79445],79:[0,.69444,0,0,.79445],80:[0,.69444,0,0,.70278],81:[.10556,.69444,0,0,.79445],82:[0,.69444,0,0,.70278],83:[0,.69444,0,0,.61111],84:[0,.69444,0,0,.73334],85:[0,.69444,0,0,.76389],86:[0,.69444,.01528,0,.73334],87:[0,.69444,.01528,0,1.03889],88:[0,.69444,0,0,.73334],89:[0,.69444,.0275,0,.73334],90:[0,.69444,0,0,.67223],91:[.25,.75,0,0,.34306],93:[.25,.75,0,0,.34306],94:[0,.69444,0,0,.55],95:[.35,.10833,.03056,0,.55],97:[0,.45833,0,0,.525],98:[0,.69444,0,0,.56111],99:[0,.45833,0,0,.48889],100:[0,.69444,0,0,.56111],101:[0,.45833,0,0,.51111],102:[0,.69444,.07639,0,.33611],103:[.19444,.45833,.01528,0,.55],104:[0,.69444,0,0,.56111],105:[0,.69444,0,0,.25556],106:[.19444,.69444,0,0,.28611],107:[0,.69444,0,0,.53056],108:[0,.69444,0,0,.25556],109:[0,.45833,0,0,.86667],110:[0,.45833,0,0,.56111],111:[0,.45833,0,0,.55],112:[.19444,.45833,0,0,.56111],113:[.19444,.45833,0,0,.56111],114:[0,.45833,.01528,0,.37222],115:[0,.45833,0,0,.42167],116:[0,.58929,0,0,.40417],117:[0,.45833,0,0,.56111],118:[0,.45833,.01528,0,.5],119:[0,.45833,.01528,0,.74445],120:[0,.45833,0,0,.5],121:[.19444,.45833,.01528,0,.5],122:[0,.45833,0,0,.47639],126:[.35,.34444,0,0,.55],160:[0,0,0,0,.25],168:[0,.69444,0,0,.55],176:[0,.69444,0,0,.73334],180:[0,.69444,0,0,.55],184:[.17014,0,0,0,.48889],305:[0,.45833,0,0,.25556],567:[.19444,.45833,0,0,.28611],710:[0,.69444,0,0,.55],711:[0,.63542,0,0,.55],713:[0,.63778,0,0,.55],728:[0,.69444,0,0,.55],729:[0,.69444,0,0,.30556],730:[0,.69444,0,0,.73334],732:[0,.69444,0,0,.55],733:[0,.69444,0,0,.55],915:[0,.69444,0,0,.58056],916:[0,.69444,0,0,.91667],920:[0,.69444,0,0,.85556],923:[0,.69444,0,0,.67223],926:[0,.69444,0,0,.73334],928:[0,.69444,0,0,.79445],931:[0,.69444,0,0,.79445],933:[0,.69444,0,0,.85556],934:[0,.69444,0,0,.79445],936:[0,.69444,0,0,.85556],937:[0,.69444,0,0,.79445],8211:[0,.45833,.03056,0,.55],8212:[0,.45833,.03056,0,1.10001],8216:[0,.69444,0,0,.30556],8217:[0,.69444,0,0,.30556],8220:[0,.69444,0,0,.55834],8221:[0,.69444,0,0,.55834]},"SansSerif-Italic":{32:[0,0,0,0,.25],33:[0,.69444,.05733,0,.31945],34:[0,.69444,.00316,0,.5],35:[.19444,.69444,.05087,0,.83334],36:[.05556,.75,.11156,0,.5],37:[.05556,.75,.03126,0,.83334],38:[0,.69444,.03058,0,.75834],39:[0,.69444,.07816,0,.27778],40:[.25,.75,.13164,0,.38889],41:[.25,.75,.02536,0,.38889],42:[0,.75,.11775,0,.5],43:[.08333,.58333,.02536,0,.77778],44:[.125,.08333,0,0,.27778],45:[0,.44444,.01946,0,.33333],46:[0,.08333,0,0,.27778],47:[.25,.75,.13164,0,.5],48:[0,.65556,.11156,0,.5],49:[0,.65556,.11156,0,.5],50:[0,.65556,.11156,0,.5],51:[0,.65556,.11156,0,.5],52:[0,.65556,.11156,0,.5],53:[0,.65556,.11156,0,.5],54:[0,.65556,.11156,0,.5],55:[0,.65556,.11156,0,.5],56:[0,.65556,.11156,0,.5],57:[0,.65556,.11156,0,.5],58:[0,.44444,.02502,0,.27778],59:[.125,.44444,.02502,0,.27778],61:[-.13,.37,.05087,0,.77778],63:[0,.69444,.11809,0,.47222],64:[0,.69444,.07555,0,.66667],65:[0,.69444,0,0,.66667],66:[0,.69444,.08293,0,.66667],67:[0,.69444,.11983,0,.63889],68:[0,.69444,.07555,0,.72223],69:[0,.69444,.11983,0,.59722],70:[0,.69444,.13372,0,.56945],71:[0,.69444,.11983,0,.66667],72:[0,.69444,.08094,0,.70834],73:[0,.69444,.13372,0,.27778],74:[0,.69444,.08094,0,.47222],75:[0,.69444,.11983,0,.69445],76:[0,.69444,0,0,.54167],77:[0,.69444,.08094,0,.875],78:[0,.69444,.08094,0,.70834],79:[0,.69444,.07555,0,.73611],80:[0,.69444,.08293,0,.63889],81:[.125,.69444,.07555,0,.73611],82:[0,.69444,.08293,0,.64584],83:[0,.69444,.09205,0,.55556],84:[0,.69444,.13372,0,.68056],85:[0,.69444,.08094,0,.6875],86:[0,.69444,.1615,0,.66667],87:[0,.69444,.1615,0,.94445],88:[0,.69444,.13372,0,.66667],89:[0,.69444,.17261,0,.66667],90:[0,.69444,.11983,0,.61111],91:[.25,.75,.15942,0,.28889],93:[.25,.75,.08719,0,.28889],94:[0,.69444,.0799,0,.5],95:[.35,.09444,.08616,0,.5],97:[0,.44444,.00981,0,.48056],98:[0,.69444,.03057,0,.51667],99:[0,.44444,.08336,0,.44445],100:[0,.69444,.09483,0,.51667],101:[0,.44444,.06778,0,.44445],102:[0,.69444,.21705,0,.30556],103:[.19444,.44444,.10836,0,.5],104:[0,.69444,.01778,0,.51667],105:[0,.67937,.09718,0,.23889],106:[.19444,.67937,.09162,0,.26667],107:[0,.69444,.08336,0,.48889],108:[0,.69444,.09483,0,.23889],109:[0,.44444,.01778,0,.79445],110:[0,.44444,.01778,0,.51667],111:[0,.44444,.06613,0,.5],112:[.19444,.44444,.0389,0,.51667],113:[.19444,.44444,.04169,0,.51667],114:[0,.44444,.10836,0,.34167],115:[0,.44444,.0778,0,.38333],116:[0,.57143,.07225,0,.36111],117:[0,.44444,.04169,0,.51667],118:[0,.44444,.10836,0,.46111],119:[0,.44444,.10836,0,.68334],120:[0,.44444,.09169,0,.46111],121:[.19444,.44444,.10836,0,.46111],122:[0,.44444,.08752,0,.43472],126:[.35,.32659,.08826,0,.5],160:[0,0,0,0,.25],168:[0,.67937,.06385,0,.5],176:[0,.69444,0,0,.73752],184:[.17014,0,0,0,.44445],305:[0,.44444,.04169,0,.23889],567:[.19444,.44444,.04169,0,.26667],710:[0,.69444,.0799,0,.5],711:[0,.63194,.08432,0,.5],713:[0,.60889,.08776,0,.5],714:[0,.69444,.09205,0,.5],715:[0,.69444,0,0,.5],728:[0,.69444,.09483,0,.5],729:[0,.67937,.07774,0,.27778],730:[0,.69444,0,0,.73752],732:[0,.67659,.08826,0,.5],733:[0,.69444,.09205,0,.5],915:[0,.69444,.13372,0,.54167],916:[0,.69444,0,0,.83334],920:[0,.69444,.07555,0,.77778],923:[0,.69444,0,0,.61111],926:[0,.69444,.12816,0,.66667],928:[0,.69444,.08094,0,.70834],931:[0,.69444,.11983,0,.72222],933:[0,.69444,.09031,0,.77778],934:[0,.69444,.04603,0,.72222],936:[0,.69444,.09031,0,.77778],937:[0,.69444,.08293,0,.72222],8211:[0,.44444,.08616,0,.5],8212:[0,.44444,.08616,0,1],8216:[0,.69444,.07816,0,.27778],8217:[0,.69444,.07816,0,.27778],8220:[0,.69444,.14205,0,.5],8221:[0,.69444,.00316,0,.5]},"SansSerif-Regular":{32:[0,0,0,0,.25],33:[0,.69444,0,0,.31945],34:[0,.69444,0,0,.5],35:[.19444,.69444,0,0,.83334],36:[.05556,.75,0,0,.5],37:[.05556,.75,0,0,.83334],38:[0,.69444,0,0,.75834],39:[0,.69444,0,0,.27778],40:[.25,.75,0,0,.38889],41:[.25,.75,0,0,.38889],42:[0,.75,0,0,.5],43:[.08333,.58333,0,0,.77778],44:[.125,.08333,0,0,.27778],45:[0,.44444,0,0,.33333],46:[0,.08333,0,0,.27778],47:[.25,.75,0,0,.5],48:[0,.65556,0,0,.5],49:[0,.65556,0,0,.5],50:[0,.65556,0,0,.5],51:[0,.65556,0,0,.5],52:[0,.65556,0,0,.5],53:[0,.65556,0,0,.5],54:[0,.65556,0,0,.5],55:[0,.65556,0,0,.5],56:[0,.65556,0,0,.5],57:[0,.65556,0,0,.5],58:[0,.44444,0,0,.27778],59:[.125,.44444,0,0,.27778],61:[-.13,.37,0,0,.77778],63:[0,.69444,0,0,.47222],64:[0,.69444,0,0,.66667],65:[0,.69444,0,0,.66667],66:[0,.69444,0,0,.66667],67:[0,.69444,0,0,.63889],68:[0,.69444,0,0,.72223],69:[0,.69444,0,0,.59722],70:[0,.69444,0,0,.56945],71:[0,.69444,0,0,.66667],72:[0,.69444,0,0,.70834],73:[0,.69444,0,0,.27778],74:[0,.69444,0,0,.47222],75:[0,.69444,0,0,.69445],76:[0,.69444,0,0,.54167],77:[0,.69444,0,0,.875],78:[0,.69444,0,0,.70834],79:[0,.69444,0,0,.73611],80:[0,.69444,0,0,.63889],81:[.125,.69444,0,0,.73611],82:[0,.69444,0,0,.64584],83:[0,.69444,0,0,.55556],84:[0,.69444,0,0,.68056],85:[0,.69444,0,0,.6875],86:[0,.69444,.01389,0,.66667],87:[0,.69444,.01389,0,.94445],88:[0,.69444,0,0,.66667],89:[0,.69444,.025,0,.66667],90:[0,.69444,0,0,.61111],91:[.25,.75,0,0,.28889],93:[.25,.75,0,0,.28889],94:[0,.69444,0,0,.5],95:[.35,.09444,.02778,0,.5],97:[0,.44444,0,0,.48056],98:[0,.69444,0,0,.51667],99:[0,.44444,0,0,.44445],100:[0,.69444,0,0,.51667],101:[0,.44444,0,0,.44445],102:[0,.69444,.06944,0,.30556],103:[.19444,.44444,.01389,0,.5],104:[0,.69444,0,0,.51667],105:[0,.67937,0,0,.23889],106:[.19444,.67937,0,0,.26667],107:[0,.69444,0,0,.48889],108:[0,.69444,0,0,.23889],109:[0,.44444,0,0,.79445],110:[0,.44444,0,0,.51667],111:[0,.44444,0,0,.5],112:[.19444,.44444,0,0,.51667],113:[.19444,.44444,0,0,.51667],114:[0,.44444,.01389,0,.34167],115:[0,.44444,0,0,.38333],116:[0,.57143,0,0,.36111],117:[0,.44444,0,0,.51667],118:[0,.44444,.01389,0,.46111],119:[0,.44444,.01389,0,.68334],120:[0,.44444,0,0,.46111],121:[.19444,.44444,.01389,0,.46111],122:[0,.44444,0,0,.43472],126:[.35,.32659,0,0,.5],160:[0,0,0,0,.25],168:[0,.67937,0,0,.5],176:[0,.69444,0,0,.66667],184:[.17014,0,0,0,.44445],305:[0,.44444,0,0,.23889],567:[.19444,.44444,0,0,.26667],710:[0,.69444,0,0,.5],711:[0,.63194,0,0,.5],713:[0,.60889,0,0,.5],714:[0,.69444,0,0,.5],715:[0,.69444,0,0,.5],728:[0,.69444,0,0,.5],729:[0,.67937,0,0,.27778],730:[0,.69444,0,0,.66667],732:[0,.67659,0,0,.5],733:[0,.69444,0,0,.5],915:[0,.69444,0,0,.54167],916:[0,.69444,0,0,.83334],920:[0,.69444,0,0,.77778],923:[0,.69444,0,0,.61111],926:[0,.69444,0,0,.66667],928:[0,.69444,0,0,.70834],931:[0,.69444,0,0,.72222],933:[0,.69444,0,0,.77778],934:[0,.69444,0,0,.72222],936:[0,.69444,0,0,.77778],937:[0,.69444,0,0,.72222],8211:[0,.44444,.02778,0,.5],8212:[0,.44444,.02778,0,1],8216:[0,.69444,0,0,.27778],8217:[0,.69444,0,0,.27778],8220:[0,.69444,0,0,.5],8221:[0,.69444,0,0,.5]},"Script-Regular":{32:[0,0,0,0,.25],65:[0,.7,.22925,0,.80253],66:[0,.7,.04087,0,.90757],67:[0,.7,.1689,0,.66619],68:[0,.7,.09371,0,.77443],69:[0,.7,.18583,0,.56162],70:[0,.7,.13634,0,.89544],71:[0,.7,.17322,0,.60961],72:[0,.7,.29694,0,.96919],73:[0,.7,.19189,0,.80907],74:[.27778,.7,.19189,0,1.05159],75:[0,.7,.31259,0,.91364],76:[0,.7,.19189,0,.87373],77:[0,.7,.15981,0,1.08031],78:[0,.7,.3525,0,.9015],79:[0,.7,.08078,0,.73787],80:[0,.7,.08078,0,1.01262],81:[0,.7,.03305,0,.88282],82:[0,.7,.06259,0,.85],83:[0,.7,.19189,0,.86767],84:[0,.7,.29087,0,.74697],85:[0,.7,.25815,0,.79996],86:[0,.7,.27523,0,.62204],87:[0,.7,.27523,0,.80532],88:[0,.7,.26006,0,.94445],89:[0,.7,.2939,0,.70961],90:[0,.7,.24037,0,.8212],160:[0,0,0,0,.25]},"Size1-Regular":{32:[0,0,0,0,.25],40:[.35001,.85,0,0,.45834],41:[.35001,.85,0,0,.45834],47:[.35001,.85,0,0,.57778],91:[.35001,.85,0,0,.41667],92:[.35001,.85,0,0,.57778],93:[.35001,.85,0,0,.41667],123:[.35001,.85,0,0,.58334],125:[.35001,.85,0,0,.58334],160:[0,0,0,0,.25],710:[0,.72222,0,0,.55556],732:[0,.72222,0,0,.55556],770:[0,.72222,0,0,.55556],771:[0,.72222,0,0,.55556],8214:[-99e-5,.601,0,0,.77778],8593:[1e-5,.6,0,0,.66667],8595:[1e-5,.6,0,0,.66667],8657:[1e-5,.6,0,0,.77778],8659:[1e-5,.6,0,0,.77778],8719:[.25001,.75,0,0,.94445],8720:[.25001,.75,0,0,.94445],8721:[.25001,.75,0,0,1.05556],8730:[.35001,.85,0,0,1],8739:[-.00599,.606,0,0,.33333],8741:[-.00599,.606,0,0,.55556],8747:[.30612,.805,.19445,0,.47222],8748:[.306,.805,.19445,0,.47222],8749:[.306,.805,.19445,0,.47222],8750:[.30612,.805,.19445,0,.47222],8896:[.25001,.75,0,0,.83334],8897:[.25001,.75,0,0,.83334],8898:[.25001,.75,0,0,.83334],8899:[.25001,.75,0,0,.83334],8968:[.35001,.85,0,0,.47222],8969:[.35001,.85,0,0,.47222],8970:[.35001,.85,0,0,.47222],8971:[.35001,.85,0,0,.47222],9168:[-99e-5,.601,0,0,.66667],10216:[.35001,.85,0,0,.47222],10217:[.35001,.85,0,0,.47222],10752:[.25001,.75,0,0,1.11111],10753:[.25001,.75,0,0,1.11111],10754:[.25001,.75,0,0,1.11111],10756:[.25001,.75,0,0,.83334],10758:[.25001,.75,0,0,.83334]},"Size2-Regular":{32:[0,0,0,0,.25],40:[.65002,1.15,0,0,.59722],41:[.65002,1.15,0,0,.59722],47:[.65002,1.15,0,0,.81111],91:[.65002,1.15,0,0,.47222],92:[.65002,1.15,0,0,.81111],93:[.65002,1.15,0,0,.47222],123:[.65002,1.15,0,0,.66667],125:[.65002,1.15,0,0,.66667],160:[0,0,0,0,.25],710:[0,.75,0,0,1],732:[0,.75,0,0,1],770:[0,.75,0,0,1],771:[0,.75,0,0,1],8719:[.55001,1.05,0,0,1.27778],8720:[.55001,1.05,0,0,1.27778],8721:[.55001,1.05,0,0,1.44445],8730:[.65002,1.15,0,0,1],8747:[.86225,1.36,.44445,0,.55556],8748:[.862,1.36,.44445,0,.55556],8749:[.862,1.36,.44445,0,.55556],8750:[.86225,1.36,.44445,0,.55556],8896:[.55001,1.05,0,0,1.11111],8897:[.55001,1.05,0,0,1.11111],8898:[.55001,1.05,0,0,1.11111],8899:[.55001,1.05,0,0,1.11111],8968:[.65002,1.15,0,0,.52778],8969:[.65002,1.15,0,0,.52778],8970:[.65002,1.15,0,0,.52778],8971:[.65002,1.15,0,0,.52778],10216:[.65002,1.15,0,0,.61111],10217:[.65002,1.15,0,0,.61111],10752:[.55001,1.05,0,0,1.51112],10753:[.55001,1.05,0,0,1.51112],10754:[.55001,1.05,0,0,1.51112],10756:[.55001,1.05,0,0,1.11111],10758:[.55001,1.05,0,0,1.11111]},"Size3-Regular":{32:[0,0,0,0,.25],40:[.95003,1.45,0,0,.73611],41:[.95003,1.45,0,0,.73611],47:[.95003,1.45,0,0,1.04445],91:[.95003,1.45,0,0,.52778],92:[.95003,1.45,0,0,1.04445],93:[.95003,1.45,0,0,.52778],123:[.95003,1.45,0,0,.75],125:[.95003,1.45,0,0,.75],160:[0,0,0,0,.25],710:[0,.75,0,0,1.44445],732:[0,.75,0,0,1.44445],770:[0,.75,0,0,1.44445],771:[0,.75,0,0,1.44445],8730:[.95003,1.45,0,0,1],8968:[.95003,1.45,0,0,.58334],8969:[.95003,1.45,0,0,.58334],8970:[.95003,1.45,0,0,.58334],8971:[.95003,1.45,0,0,.58334],10216:[.95003,1.45,0,0,.75],10217:[.95003,1.45,0,0,.75]},"Size4-Regular":{32:[0,0,0,0,.25],40:[1.25003,1.75,0,0,.79167],41:[1.25003,1.75,0,0,.79167],47:[1.25003,1.75,0,0,1.27778],91:[1.25003,1.75,0,0,.58334],92:[1.25003,1.75,0,0,1.27778],93:[1.25003,1.75,0,0,.58334],123:[1.25003,1.75,0,0,.80556],125:[1.25003,1.75,0,0,.80556],160:[0,0,0,0,.25],710:[0,.825,0,0,1.8889],732:[0,.825,0,0,1.8889],770:[0,.825,0,0,1.8889],771:[0,.825,0,0,1.8889],8730:[1.25003,1.75,0,0,1],8968:[1.25003,1.75,0,0,.63889],8969:[1.25003,1.75,0,0,.63889],8970:[1.25003,1.75,0,0,.63889],8971:[1.25003,1.75,0,0,.63889],9115:[.64502,1.155,0,0,.875],9116:[1e-5,.6,0,0,.875],9117:[.64502,1.155,0,0,.875],9118:[.64502,1.155,0,0,.875],9119:[1e-5,.6,0,0,.875],9120:[.64502,1.155,0,0,.875],9121:[.64502,1.155,0,0,.66667],9122:[-99e-5,.601,0,0,.66667],9123:[.64502,1.155,0,0,.66667],9124:[.64502,1.155,0,0,.66667],9125:[-99e-5,.601,0,0,.66667],9126:[.64502,1.155,0,0,.66667],9127:[1e-5,.9,0,0,.88889],9128:[.65002,1.15,0,0,.88889],9129:[.90001,0,0,0,.88889],9130:[0,.3,0,0,.88889],9131:[1e-5,.9,0,0,.88889],9132:[.65002,1.15,0,0,.88889],9133:[.90001,0,0,0,.88889],9143:[.88502,.915,0,0,1.05556],10216:[1.25003,1.75,0,0,.80556],10217:[1.25003,1.75,0,0,.80556],57344:[-.00499,.605,0,0,1.05556],57345:[-.00499,.605,0,0,1.05556],57680:[0,.12,0,0,.45],57681:[0,.12,0,0,.45],57682:[0,.12,0,0,.45],57683:[0,.12,0,0,.45]},"Typewriter-Regular":{32:[0,0,0,0,.525],33:[0,.61111,0,0,.525],34:[0,.61111,0,0,.525],35:[0,.61111,0,0,.525],36:[.08333,.69444,0,0,.525],37:[.08333,.69444,0,0,.525],38:[0,.61111,0,0,.525],39:[0,.61111,0,0,.525],40:[.08333,.69444,0,0,.525],41:[.08333,.69444,0,0,.525],42:[0,.52083,0,0,.525],43:[-.08056,.53055,0,0,.525],44:[.13889,.125,0,0,.525],45:[-.08056,.53055,0,0,.525],46:[0,.125,0,0,.525],47:[.08333,.69444,0,0,.525],48:[0,.61111,0,0,.525],49:[0,.61111,0,0,.525],50:[0,.61111,0,0,.525],51:[0,.61111,0,0,.525],52:[0,.61111,0,0,.525],53:[0,.61111,0,0,.525],54:[0,.61111,0,0,.525],55:[0,.61111,0,0,.525],56:[0,.61111,0,0,.525],57:[0,.61111,0,0,.525],58:[0,.43056,0,0,.525],59:[.13889,.43056,0,0,.525],60:[-.05556,.55556,0,0,.525],61:[-.19549,.41562,0,0,.525],62:[-.05556,.55556,0,0,.525],63:[0,.61111,0,0,.525],64:[0,.61111,0,0,.525],65:[0,.61111,0,0,.525],66:[0,.61111,0,0,.525],67:[0,.61111,0,0,.525],68:[0,.61111,0,0,.525],69:[0,.61111,0,0,.525],70:[0,.61111,0,0,.525],71:[0,.61111,0,0,.525],72:[0,.61111,0,0,.525],73:[0,.61111,0,0,.525],74:[0,.61111,0,0,.525],75:[0,.61111,0,0,.525],76:[0,.61111,0,0,.525],77:[0,.61111,0,0,.525],78:[0,.61111,0,0,.525],79:[0,.61111,0,0,.525],80:[0,.61111,0,0,.525],81:[.13889,.61111,0,0,.525],82:[0,.61111,0,0,.525],83:[0,.61111,0,0,.525],84:[0,.61111,0,0,.525],85:[0,.61111,0,0,.525],86:[0,.61111,0,0,.525],87:[0,.61111,0,0,.525],88:[0,.61111,0,0,.525],89:[0,.61111,0,0,.525],90:[0,.61111,0,0,.525],91:[.08333,.69444,0,0,.525],92:[.08333,.69444,0,0,.525],93:[.08333,.69444,0,0,.525],94:[0,.61111,0,0,.525],95:[.09514,0,0,0,.525],96:[0,.61111,0,0,.525],97:[0,.43056,0,0,.525],98:[0,.61111,0,0,.525],99:[0,.43056,0,0,.525],100:[0,.61111,0,0,.525],101:[0,.43056,0,0,.525],102:[0,.61111,0,0,.525],103:[.22222,.43056,0,0,.525],104:[0,.61111,0,0,.525],105:[0,.61111,0,0,.525],106:[.22222,.61111,0,0,.525],107:[0,.61111,0,0,.525],108:[0,.61111,0,0,.525],109:[0,.43056,0,0,.525],110:[0,.43056,0,0,.525],111:[0,.43056,0,0,.525],112:[.22222,.43056,0,0,.525],113:[.22222,.43056,0,0,.525],114:[0,.43056,0,0,.525],115:[0,.43056,0,0,.525],116:[0,.55358,0,0,.525],117:[0,.43056,0,0,.525],118:[0,.43056,0,0,.525],119:[0,.43056,0,0,.525],120:[0,.43056,0,0,.525],121:[.22222,.43056,0,0,.525],122:[0,.43056,0,0,.525],123:[.08333,.69444,0,0,.525],124:[.08333,.69444,0,0,.525],125:[.08333,.69444,0,0,.525],126:[0,.61111,0,0,.525],127:[0,.61111,0,0,.525],160:[0,0,0,0,.525],176:[0,.61111,0,0,.525],184:[.19445,0,0,0,.525],305:[0,.43056,0,0,.525],567:[.22222,.43056,0,0,.525],711:[0,.56597,0,0,.525],713:[0,.56555,0,0,.525],714:[0,.61111,0,0,.525],715:[0,.61111,0,0,.525],728:[0,.61111,0,0,.525],730:[0,.61111,0,0,.525],770:[0,.61111,0,0,.525],771:[0,.61111,0,0,.525],776:[0,.61111,0,0,.525],915:[0,.61111,0,0,.525],916:[0,.61111,0,0,.525],920:[0,.61111,0,0,.525],923:[0,.61111,0,0,.525],926:[0,.61111,0,0,.525],928:[0,.61111,0,0,.525],931:[0,.61111,0,0,.525],933:[0,.61111,0,0,.525],934:[0,.61111,0,0,.525],936:[0,.61111,0,0,.525],937:[0,.61111,0,0,.525],8216:[0,.61111,0,0,.525],8217:[0,.61111,0,0,.525],8242:[0,.61111,0,0,.525],9251:[.11111,.21944,0,0,.525]}},Z4={slant:[.25,.25,.25],space:[0,0,0],stretch:[0,0,0],shrink:[0,0,0],xHeight:[.431,.431,.431],quad:[1,1.171,1.472],extraSpace:[0,0,0],num1:[.677,.732,.925],num2:[.394,.384,.387],num3:[.444,.471,.504],denom1:[.686,.752,1.025],denom2:[.345,.344,.532],sup1:[.413,.503,.504],sup2:[.363,.431,.404],sup3:[.289,.286,.294],sub1:[.15,.143,.2],sub2:[.247,.286,.4],supDrop:[.386,.353,.494],subDrop:[.05,.071,.1],delim1:[2.39,1.7,1.98],delim2:[1.01,1.157,1.42],axisHeight:[.25,.25,.25],defaultRuleThickness:[.04,.049,.049],bigOpSpacing1:[.111,.111,.111],bigOpSpacing2:[.166,.166,.166],bigOpSpacing3:[.2,.2,.2],bigOpSpacing4:[.6,.611,.611],bigOpSpacing5:[.1,.143,.143],sqrtRuleThickness:[.04,.04,.04],ptPerEm:[10,10,10],doubleRuleSep:[.2,.2,.2],arrayRuleWidth:[.04,.04,.04],fboxsep:[.3,.3,.3],fboxrule:[.04,.04,.04]},lz={\u00C5:"A",\u00D0:"D",\u00DE:"o",\u00E5:"a",\u00F0:"d",\u00FE:"o",\u0410:"A",\u0411:"B",\u0412:"B",\u0413:"F",\u0414:"A",\u0415:"E",\u0416:"K",\u0417:"3",\u0418:"N",\u0419:"N",\u041A:"K",\u041B:"N",\u041C:"M",\u041D:"H",\u041E:"O",\u041F:"N",\u0420:"P",\u0421:"C",\u0422:"T",\u0423:"y",\u0424:"O",\u0425:"X",\u0426:"U",\u0427:"h",\u0428:"W",\u0429:"W",\u042A:"B",\u042B:"X",\u042C:"B",\u042D:"3",\u042E:"X",\u042F:"R",\u0430:"a",\u0431:"b",\u0432:"a",\u0433:"r",\u0434:"y",\u0435:"e",\u0436:"m",\u0437:"e",\u0438:"n",\u0439:"n",\u043A:"n",\u043B:"n",\u043C:"m",\u043D:"n",\u043E:"o",\u043F:"n",\u0440:"p",\u0441:"c",\u0442:"o",\u0443:"y",\u0444:"b",\u0445:"x",\u0446:"n",\u0447:"n",\u0448:"w",\u0449:"w",\u044A:"a",\u044B:"m",\u044C:"a",\u044D:"e",\u044E:"m",\u044F:"r"};o(Abe,"setFontMetrics");o(P7,"getCharacterMetrics");h7={};o(_be,"getGlobalMetrics");Dbe=[[1,1,1],[2,1,1],[3,1,1],[4,2,1],[5,2,1],[6,3,1],[7,4,2],[8,6,3],[9,7,6],[10,8,7],[11,10,9]],cz=[.5,.6,.7,.8,.9,1,1.2,1.44,1.728,2.074,2.488],uz=o(function(e,r){return r.size<2?e:Dbe[e-1][r.size-1]},"sizeAtStyle"),f3=class t{static{o(this,"Options")}constructor(e){this.style=void 0,this.color=void 0,this.size=void 0,this.textSize=void 0,this.phantom=void 0,this.font=void 0,this.fontFamily=void 0,this.fontWeight=void 0,this.fontShape=void 0,this.sizeMultiplier=void 0,this.maxSize=void 0,this.minRuleThickness=void 0,this._fontMetrics=void 0,this.style=e.style,this.color=e.color,this.size=e.size||t.BASESIZE,this.textSize=e.textSize||this.size,this.phantom=!!e.phantom,this.font=e.font||"",this.fontFamily=e.fontFamily||"",this.fontWeight=e.fontWeight||"",this.fontShape=e.fontShape||"",this.sizeMultiplier=cz[this.size-1],this.maxSize=e.maxSize,this.minRuleThickness=e.minRuleThickness,this._fontMetrics=void 0}extend(e){var r={style:this.style,size:this.size,textSize:this.textSize,color:this.color,phantom:this.phantom,font:this.font,fontFamily:this.fontFamily,fontWeight:this.fontWeight,fontShape:this.fontShape,maxSize:this.maxSize,minRuleThickness:this.minRuleThickness};for(var n in e)e.hasOwnProperty(n)&&(r[n]=e[n]);return new t(r)}havingStyle(e){return this.style===e?this:this.extend({style:e,size:uz(this.textSize,e)})}havingCrampedStyle(){return this.havingStyle(this.style.cramp())}havingSize(e){return this.size===e&&this.textSize===e?this:this.extend({style:this.style.text(),size:e,textSize:e,sizeMultiplier:cz[e-1]})}havingBaseStyle(e){e=e||this.style.text();var r=uz(t.BASESIZE,e);return this.size===r&&this.textSize===t.BASESIZE&&this.style===e?this:this.extend({style:e,size:r})}havingBaseSizing(){var e;switch(this.style.id){case 4:case 5:e=3;break;case 6:case 7:e=1;break;default:e=6}return this.extend({style:this.style.text(),size:e})}withColor(e){return this.extend({color:e})}withPhantom(){return this.extend({phantom:!0})}withFont(e){return this.extend({font:e})}withTextFontFamily(e){return this.extend({fontFamily:e,font:""})}withTextFontWeight(e){return this.extend({fontWeight:e,font:""})}withTextFontShape(e){return this.extend({fontShape:e,font:""})}sizingClasses(e){return e.size!==this.size?["sizing","reset-size"+e.size,"size"+this.size]:[]}baseSizingClasses(){return this.size!==t.BASESIZE?["sizing","reset-size"+this.size,"size"+t.BASESIZE]:[]}fontMetrics(){return this._fontMetrics||(this._fontMetrics=_be(this.size)),this._fontMetrics}getColor(){return this.phantom?"transparent":this.color}};f3.BASESIZE=6;E7={pt:1,mm:7227/2540,cm:7227/254,in:72.27,bp:803/800,pc:12,dd:1238/1157,cc:14856/1157,nd:685/642,nc:1370/107,sp:1/65536,px:803/800},Lbe={ex:!0,em:!0,mu:!0},zz=o(function(e){return typeof e!="string"&&(e=e.unit),e in E7||e in Lbe||e==="ex"},"validUnit"),ti=o(function(e,r){var n;if(e.unit in E7)n=E7[e.unit]/r.fontMetrics().ptPerEm/r.sizeMultiplier;else if(e.unit==="mu")n=r.fontMetrics().cssEmPerMu;else{var i;if(r.style.isTight()?i=r.havingStyle(r.style.text()):i=r,e.unit==="ex")n=i.fontMetrics().xHeight;else if(e.unit==="em")n=i.fontMetrics().quad;else throw new gt("Invalid unit: '"+e.unit+"'");i!==r&&(n*=i.sizeMultiplier/r.sizeMultiplier)}return Math.min(e.number*n,r.maxSize)},"calculateSize"),kt=o(function(e){return+e.toFixed(4)+"em"},"makeEm"),fh=o(function(e){return e.filter(r=>r).join(" ")},"createClass"),Gz=o(function(e,r,n){if(this.classes=e||[],this.attributes={},this.height=0,this.depth=0,this.maxFontSize=0,this.style=n||{},r){r.style.isTight()&&this.classes.push("mtight");var i=r.getColor();i&&(this.style.color=i)}},"initNode"),Vz=o(function(e){var r=document.createElement(e);r.className=fh(this.classes);for(var n in this.style)this.style.hasOwnProperty(n)&&(r.style[n]=this.style[n]);for(var i in this.attributes)this.attributes.hasOwnProperty(i)&&r.setAttribute(i,this.attributes[i]);for(var a=0;a",r},"toMarkup"),td=class{static{o(this,"Span")}constructor(e,r,n,i){this.children=void 0,this.attributes=void 0,this.classes=void 0,this.height=void 0,this.depth=void 0,this.width=void 0,this.maxFontSize=void 0,this.style=void 0,Gz.call(this,e,n,i),this.children=r||[]}setAttribute(e,r){this.attributes[e]=r}hasClass(e){return Jt.contains(this.classes,e)}toNode(){return Vz.call(this,"span")}toMarkup(){return Uz.call(this,"span")}},Vy=class{static{o(this,"Anchor")}constructor(e,r,n,i){this.children=void 0,this.attributes=void 0,this.classes=void 0,this.height=void 0,this.depth=void 0,this.maxFontSize=void 0,this.style=void 0,Gz.call(this,r,i),this.children=n||[],this.setAttribute("href",e)}setAttribute(e,r){this.attributes[e]=r}hasClass(e){return Jt.contains(this.classes,e)}toNode(){return Vz.call(this,"a")}toMarkup(){return Uz.call(this,"a")}},S7=class{static{o(this,"Img")}constructor(e,r,n){this.src=void 0,this.alt=void 0,this.classes=void 0,this.height=void 0,this.depth=void 0,this.maxFontSize=void 0,this.style=void 0,this.alt=r,this.src=e,this.classes=["mord"],this.style=n}hasClass(e){return Jt.contains(this.classes,e)}toNode(){var e=document.createElement("img");e.src=this.src,e.alt=this.alt,e.className="mord";for(var r in this.style)this.style.hasOwnProperty(r)&&(e.style[r]=this.style[r]);return e}toMarkup(){var e=''+Jt.escape(this.alt)+'0&&(r=document.createElement("span"),r.style.marginRight=kt(this.italic)),this.classes.length>0&&(r=r||document.createElement("span"),r.className=fh(this.classes));for(var n in this.style)this.style.hasOwnProperty(n)&&(r=r||document.createElement("span"),r.style[n]=this.style[n]);return r?(r.appendChild(e),r):e}toMarkup(){var e=!1,r="0&&(n+="margin-right:"+this.italic+"em;");for(var i in this.style)this.style.hasOwnProperty(i)&&(n+=Jt.hyphenate(i)+":"+this.style[i]+";");n&&(e=!0,r+=' style="'+Jt.escape(n)+'"');var a=Jt.escape(this.text);return e?(r+=">",r+=a,r+="",r):a}},ll=class{static{o(this,"SvgNode")}constructor(e,r){this.children=void 0,this.attributes=void 0,this.children=e||[],this.attributes=r||{}}toNode(){var e="http://www.w3.org/2000/svg",r=document.createElementNS(e,"svg");for(var n in this.attributes)Object.prototype.hasOwnProperty.call(this.attributes,n)&&r.setAttribute(n,this.attributes[n]);for(var i=0;i':''}},Uy=class{static{o(this,"LineNode")}constructor(e){this.attributes=void 0,this.attributes=e||{}}toNode(){var e="http://www.w3.org/2000/svg",r=document.createElementNS(e,"line");for(var n in this.attributes)Object.prototype.hasOwnProperty.call(this.attributes,n)&&r.setAttribute(n,this.attributes[n]);return r}toMarkup(){var e="","\\gt",!0);G(U,ee,Ee,"\u2208","\\in",!0);G(U,ee,Ee,"\uE020","\\@not");G(U,ee,Ee,"\u2282","\\subset",!0);G(U,ee,Ee,"\u2283","\\supset",!0);G(U,ee,Ee,"\u2286","\\subseteq",!0);G(U,ee,Ee,"\u2287","\\supseteq",!0);G(U,ke,Ee,"\u2288","\\nsubseteq",!0);G(U,ke,Ee,"\u2289","\\nsupseteq",!0);G(U,ee,Ee,"\u22A8","\\models");G(U,ee,Ee,"\u2190","\\leftarrow",!0);G(U,ee,Ee,"\u2264","\\le");G(U,ee,Ee,"\u2264","\\leq",!0);G(U,ee,Ee,"<","\\lt",!0);G(U,ee,Ee,"\u2192","\\rightarrow",!0);G(U,ee,Ee,"\u2192","\\to");G(U,ke,Ee,"\u2271","\\ngeq",!0);G(U,ke,Ee,"\u2270","\\nleq",!0);G(U,ee,uu,"\xA0","\\ ");G(U,ee,uu,"\xA0","\\space");G(U,ee,uu,"\xA0","\\nobreakspace");G(it,ee,uu,"\xA0","\\ ");G(it,ee,uu,"\xA0"," ");G(it,ee,uu,"\xA0","\\space");G(it,ee,uu,"\xA0","\\nobreakspace");G(U,ee,uu,null,"\\nobreak");G(U,ee,uu,null,"\\allowbreak");G(U,ee,x3,",",",");G(U,ee,x3,";",";");G(U,ke,It,"\u22BC","\\barwedge",!0);G(U,ke,It,"\u22BB","\\veebar",!0);G(U,ee,It,"\u2299","\\odot",!0);G(U,ee,It,"\u2295","\\oplus",!0);G(U,ee,It,"\u2297","\\otimes",!0);G(U,ee,Le,"\u2202","\\partial",!0);G(U,ee,It,"\u2298","\\oslash",!0);G(U,ke,It,"\u229A","\\circledcirc",!0);G(U,ke,It,"\u22A1","\\boxdot",!0);G(U,ee,It,"\u25B3","\\bigtriangleup");G(U,ee,It,"\u25BD","\\bigtriangledown");G(U,ee,It,"\u2020","\\dagger");G(U,ee,It,"\u22C4","\\diamond");G(U,ee,It,"\u22C6","\\star");G(U,ee,It,"\u25C3","\\triangleleft");G(U,ee,It,"\u25B9","\\triangleright");G(U,ee,js,"{","\\{");G(it,ee,Le,"{","\\{");G(it,ee,Le,"{","\\textbraceleft");G(U,ee,Za,"}","\\}");G(it,ee,Le,"}","\\}");G(it,ee,Le,"}","\\textbraceright");G(U,ee,js,"{","\\lbrace");G(U,ee,Za,"}","\\rbrace");G(U,ee,js,"[","\\lbrack",!0);G(it,ee,Le,"[","\\lbrack",!0);G(U,ee,Za,"]","\\rbrack",!0);G(it,ee,Le,"]","\\rbrack",!0);G(U,ee,js,"(","\\lparen",!0);G(U,ee,Za,")","\\rparen",!0);G(it,ee,Le,"<","\\textless",!0);G(it,ee,Le,">","\\textgreater",!0);G(U,ee,js,"\u230A","\\lfloor",!0);G(U,ee,Za,"\u230B","\\rfloor",!0);G(U,ee,js,"\u2308","\\lceil",!0);G(U,ee,Za,"\u2309","\\rceil",!0);G(U,ee,Le,"\\","\\backslash");G(U,ee,Le,"\u2223","|");G(U,ee,Le,"\u2223","\\vert");G(it,ee,Le,"|","\\textbar",!0);G(U,ee,Le,"\u2225","\\|");G(U,ee,Le,"\u2225","\\Vert");G(it,ee,Le,"\u2225","\\textbardbl");G(it,ee,Le,"~","\\textasciitilde");G(it,ee,Le,"\\","\\textbackslash");G(it,ee,Le,"^","\\textasciicircum");G(U,ee,Ee,"\u2191","\\uparrow",!0);G(U,ee,Ee,"\u21D1","\\Uparrow",!0);G(U,ee,Ee,"\u2193","\\downarrow",!0);G(U,ee,Ee,"\u21D3","\\Downarrow",!0);G(U,ee,Ee,"\u2195","\\updownarrow",!0);G(U,ee,Ee,"\u21D5","\\Updownarrow",!0);G(U,ee,ki,"\u2210","\\coprod");G(U,ee,ki,"\u22C1","\\bigvee");G(U,ee,ki,"\u22C0","\\bigwedge");G(U,ee,ki,"\u2A04","\\biguplus");G(U,ee,ki,"\u22C2","\\bigcap");G(U,ee,ki,"\u22C3","\\bigcup");G(U,ee,ki,"\u222B","\\int");G(U,ee,ki,"\u222B","\\intop");G(U,ee,ki,"\u222C","\\iint");G(U,ee,ki,"\u222D","\\iiint");G(U,ee,ki,"\u220F","\\prod");G(U,ee,ki,"\u2211","\\sum");G(U,ee,ki,"\u2A02","\\bigotimes");G(U,ee,ki,"\u2A01","\\bigoplus");G(U,ee,ki,"\u2A00","\\bigodot");G(U,ee,ki,"\u222E","\\oint");G(U,ee,ki,"\u222F","\\oiint");G(U,ee,ki,"\u2230","\\oiiint");G(U,ee,ki,"\u2A06","\\bigsqcup");G(U,ee,ki,"\u222B","\\smallint");G(it,ee,p0,"\u2026","\\textellipsis");G(U,ee,p0,"\u2026","\\mathellipsis");G(it,ee,p0,"\u2026","\\ldots",!0);G(U,ee,p0,"\u2026","\\ldots",!0);G(U,ee,p0,"\u22EF","\\@cdots",!0);G(U,ee,p0,"\u22F1","\\ddots",!0);G(U,ee,Le,"\u22EE","\\varvdots");G(U,ee,Vn,"\u02CA","\\acute");G(U,ee,Vn,"\u02CB","\\grave");G(U,ee,Vn,"\xA8","\\ddot");G(U,ee,Vn,"~","\\tilde");G(U,ee,Vn,"\u02C9","\\bar");G(U,ee,Vn,"\u02D8","\\breve");G(U,ee,Vn,"\u02C7","\\check");G(U,ee,Vn,"^","\\hat");G(U,ee,Vn,"\u20D7","\\vec");G(U,ee,Vn,"\u02D9","\\dot");G(U,ee,Vn,"\u02DA","\\mathring");G(U,ee,er,"\uE131","\\@imath");G(U,ee,er,"\uE237","\\@jmath");G(U,ee,Le,"\u0131","\u0131");G(U,ee,Le,"\u0237","\u0237");G(it,ee,Le,"\u0131","\\i",!0);G(it,ee,Le,"\u0237","\\j",!0);G(it,ee,Le,"\xDF","\\ss",!0);G(it,ee,Le,"\xE6","\\ae",!0);G(it,ee,Le,"\u0153","\\oe",!0);G(it,ee,Le,"\xF8","\\o",!0);G(it,ee,Le,"\xC6","\\AE",!0);G(it,ee,Le,"\u0152","\\OE",!0);G(it,ee,Le,"\xD8","\\O",!0);G(it,ee,Vn,"\u02CA","\\'");G(it,ee,Vn,"\u02CB","\\`");G(it,ee,Vn,"\u02C6","\\^");G(it,ee,Vn,"\u02DC","\\~");G(it,ee,Vn,"\u02C9","\\=");G(it,ee,Vn,"\u02D8","\\u");G(it,ee,Vn,"\u02D9","\\.");G(it,ee,Vn,"\xB8","\\c");G(it,ee,Vn,"\u02DA","\\r");G(it,ee,Vn,"\u02C7","\\v");G(it,ee,Vn,"\xA8",'\\"');G(it,ee,Vn,"\u02DD","\\H");G(it,ee,Vn,"\u25EF","\\textcircled");Hz={"--":!0,"---":!0,"``":!0,"''":!0};G(it,ee,Le,"\u2013","--",!0);G(it,ee,Le,"\u2013","\\textendash");G(it,ee,Le,"\u2014","---",!0);G(it,ee,Le,"\u2014","\\textemdash");G(it,ee,Le,"\u2018","`",!0);G(it,ee,Le,"\u2018","\\textquoteleft");G(it,ee,Le,"\u2019","'",!0);G(it,ee,Le,"\u2019","\\textquoteright");G(it,ee,Le,"\u201C","``",!0);G(it,ee,Le,"\u201C","\\textquotedblleft");G(it,ee,Le,"\u201D","''",!0);G(it,ee,Le,"\u201D","\\textquotedblright");G(U,ee,Le,"\xB0","\\degree",!0);G(it,ee,Le,"\xB0","\\degree");G(it,ee,Le,"\xB0","\\textdegree",!0);G(U,ee,Le,"\xA3","\\pounds");G(U,ee,Le,"\xA3","\\mathsterling",!0);G(it,ee,Le,"\xA3","\\pounds");G(it,ee,Le,"\xA3","\\textsterling",!0);G(U,ke,Le,"\u2720","\\maltese");G(it,ke,Le,"\u2720","\\maltese");fz='0123456789/@."';for(J4=0;J40)return ol(a,h,i,r,s.concat(f));if(u){var d,p;if(u==="boldsymbol"){var m=Bbe(a,i,r,s,n);d=m.fontName,p=[m.fontClass]}else l?(d=Yz[u].fontName,p=[u]):(d=i3(u,r.fontWeight,r.fontShape),p=[u,r.fontWeight,r.fontShape]);if(b3(a,d,i).metrics)return ol(a,d,i,r,s.concat(p));if(Hz.hasOwnProperty(a)&&d.slice(0,10)==="Typewriter"){for(var g=[],y=0;y{if(fh(t.classes)!==fh(e.classes)||t.skew!==e.skew||t.maxFontSize!==e.maxFontSize)return!1;if(t.classes.length===1){var r=t.classes[0];if(r==="mbin"||r==="mord")return!1}for(var n in t.style)if(t.style.hasOwnProperty(n)&&t.style[n]!==e.style[n])return!1;for(var i in e.style)if(e.style.hasOwnProperty(i)&&t.style[i]!==e.style[i])return!1;return!0},"canCombine"),zbe=o(t=>{for(var e=0;er&&(r=s.height),s.depth>n&&(n=s.depth),s.maxFontSize>i&&(i=s.maxFontSize)}e.height=r,e.depth=n,e.maxFontSize=i},"sizeElementFromChildren"),bs=o(function(e,r,n,i){var a=new td(e,r,n,i);return B7(a),a},"makeSpan"),Wz=o((t,e,r,n)=>new td(t,e,r,n),"makeSvgSpan"),Gbe=o(function(e,r,n){var i=bs([e],[],r);return i.height=Math.max(n||r.fontMetrics().defaultRuleThickness,r.minRuleThickness),i.style.borderBottomWidth=kt(i.height),i.maxFontSize=1,i},"makeLineSpan"),Vbe=o(function(e,r,n,i){var a=new Vy(e,r,n,i);return B7(a),a},"makeAnchor"),qz=o(function(e){var r=new ed(e);return B7(r),r},"makeFragment"),Ube=o(function(e,r){return e instanceof ed?bs([],[e],r):e},"wrapFragment"),Hbe=o(function(e){if(e.positionType==="individualShift"){for(var r=e.children,n=[r[0]],i=-r[0].shift-r[0].elem.depth,a=i,s=1;s{var r=bs(["mspace"],[],e),n=ti(t,e);return r.style.marginRight=kt(n),r},"makeGlue"),i3=o(function(e,r,n){var i="";switch(e){case"amsrm":i="AMS";break;case"textrm":i="Main";break;case"textsf":i="SansSerif";break;case"texttt":i="Typewriter";break;default:i=e}var a;return r==="textbf"&&n==="textit"?a="BoldItalic":r==="textbf"?a="Bold":r==="textit"?a="Italic":a="Regular",i+"-"+a},"retrieveTextFontName"),Yz={mathbf:{variant:"bold",fontName:"Main-Bold"},mathrm:{variant:"normal",fontName:"Main-Regular"},textit:{variant:"italic",fontName:"Main-Italic"},mathit:{variant:"italic",fontName:"Main-Italic"},mathnormal:{variant:"italic",fontName:"Math-Italic"},mathbb:{variant:"double-struck",fontName:"AMS-Regular"},mathcal:{variant:"script",fontName:"Caligraphic-Regular"},mathfrak:{variant:"fraktur",fontName:"Fraktur-Regular"},mathscr:{variant:"script",fontName:"Script-Regular"},mathsf:{variant:"sans-serif",fontName:"SansSerif-Regular"},mathtt:{variant:"monospace",fontName:"Typewriter-Regular"}},Xz={vec:["vec",.471,.714],oiintSize1:["oiintSize1",.957,.499],oiintSize2:["oiintSize2",1.472,.659],oiiintSize1:["oiiintSize1",1.304,.499],oiiintSize2:["oiiintSize2",1.98,.659]},Ybe=o(function(e,r){var[n,i,a]=Xz[e],s=new Kl(n),l=new ll([s],{width:kt(i),height:kt(a),style:"width:"+kt(i),viewBox:"0 0 "+1e3*i+" "+1e3*a,preserveAspectRatio:"xMinYMin"}),u=Wz(["overlay"],[l],r);return u.height=a,u.style.height=kt(a),u.style.width=kt(i),u},"staticSvg"),Be={fontMap:Yz,makeSymbol:ol,mathsym:Pbe,makeSpan:bs,makeSvgSpan:Wz,makeLineSpan:Gbe,makeAnchor:Vbe,makeFragment:qz,wrapFragment:Ube,makeVList:Wbe,makeOrd:Fbe,makeGlue:qbe,staticSvg:Ybe,svgData:Xz,tryCombineChars:zbe},ei={number:3,unit:"mu"},Zf={number:4,unit:"mu"},au={number:5,unit:"mu"},Xbe={mord:{mop:ei,mbin:Zf,mrel:au,minner:ei},mop:{mord:ei,mop:ei,mrel:au,minner:ei},mbin:{mord:Zf,mop:Zf,mopen:Zf,minner:Zf},mrel:{mord:au,mop:au,mopen:au,minner:au},mopen:{},mclose:{mop:ei,mbin:Zf,mrel:au,minner:ei},mpunct:{mord:ei,mop:ei,mrel:au,mopen:ei,mclose:ei,mpunct:ei,minner:ei},minner:{mord:ei,mop:ei,mbin:Zf,mrel:au,mopen:ei,mpunct:ei,minner:ei}},jbe={mord:{mop:ei},mop:{mord:ei,mop:ei},mbin:{},mrel:{},mopen:{},mclose:{mop:ei},mpunct:{},minner:{mop:ei}},jz={},p3={},m3={};o(Nt,"defineFunction");o(rd,"defineFunctionBuilders");g3=o(function(e){return e.type==="ordgroup"&&e.body.length===1?e.body[0]:e},"normalizeArgument"),di=o(function(e){return e.type==="ordgroup"?e.body:[e]},"ordargument"),lu=Be.makeSpan,Kbe=["leftmost","mbin","mopen","mrel","mop","mpunct"],Qbe=["rightmost","mrel","mclose","mpunct"],Zbe={display:tr.DISPLAY,text:tr.TEXT,script:tr.SCRIPT,scriptscript:tr.SCRIPTSCRIPT},Jbe={mord:"mord",mop:"mop",mbin:"mbin",mrel:"mrel",mopen:"mopen",mclose:"mclose",mpunct:"mpunct",minner:"minner"},Pi=o(function(e,r,n,i){i===void 0&&(i=[null,null]);for(var a=[],s=0;s{var v=y.classes[0],x=g.classes[0];v==="mbin"&&Jt.contains(Qbe,x)?y.classes[0]="mord":x==="mbin"&&Jt.contains(Kbe,v)&&(g.classes[0]="mord")},{node:d},p,m),mz(a,(g,y)=>{var v=A7(y),x=A7(g),b=v&&x?g.hasClass("mtight")?jbe[v][x]:Xbe[v][x]:null;if(b)return Be.makeGlue(b,h)},{node:d},p,m),a},"buildExpression"),mz=o(function t(e,r,n,i,a){i&&e.push(i);for(var s=0;sp=>{e.splice(d+1,0,p),s++})(s)}i&&e.pop()},"traverseNonSpaceNodes"),Kz=o(function(e){return e instanceof ed||e instanceof Vy||e instanceof td&&e.hasClass("enclosing")?e:null},"checkPartialGroup"),e4e=o(function t(e,r){var n=Kz(e);if(n){var i=n.children;if(i.length){if(r==="right")return t(i[i.length-1],"right");if(r==="left")return t(i[0],"left")}}return e},"getOutermostNode"),A7=o(function(e,r){return e?(r&&(e=e4e(e,r)),Jbe[e.classes[0]]||null):null},"getTypeOfDomTree"),Hy=o(function(e,r){var n=["nulldelimiter"].concat(e.baseSizingClasses());return lu(r.concat(n))},"makeNullDelimiter"),Fr=o(function(e,r,n){if(!e)return lu();if(p3[e.type]){var i=p3[e.type](e,r);if(n&&r.size!==n.size){i=lu(r.sizingClasses(n),[i],r);var a=r.sizeMultiplier/n.sizeMultiplier;i.height*=a,i.depth*=a}return i}else throw new gt("Got group of unknown type: '"+e.type+"'")},"buildGroup");o(a3,"buildHTMLUnbreakable");o(_7,"buildHTML");o(Qz,"newDocumentFragment");ws=class{static{o(this,"MathNode")}constructor(e,r,n){this.type=void 0,this.attributes=void 0,this.children=void 0,this.classes=void 0,this.type=e,this.attributes={},this.children=r||[],this.classes=n||[]}setAttribute(e,r){this.attributes[e]=r}getAttribute(e){return this.attributes[e]}toNode(){var e=document.createElementNS("http://www.w3.org/1998/Math/MathML",this.type);for(var r in this.attributes)Object.prototype.hasOwnProperty.call(this.attributes,r)&&e.setAttribute(r,this.attributes[r]);this.classes.length>0&&(e.className=fh(this.classes));for(var n=0;n0&&(e+=' class ="'+Jt.escape(fh(this.classes))+'"'),e+=">";for(var n=0;n",e}toText(){return this.children.map(e=>e.toText()).join("")}},Jf=class{static{o(this,"TextNode")}constructor(e){this.text=void 0,this.text=e}toNode(){return document.createTextNode(this.text)}toMarkup(){return Jt.escape(this.toText())}toText(){return this.text}},D7=class{static{o(this,"SpaceNode")}constructor(e){this.width=void 0,this.character=void 0,this.width=e,e>=.05555&&e<=.05556?this.character="\u200A":e>=.1666&&e<=.1667?this.character="\u2009":e>=.2222&&e<=.2223?this.character="\u2005":e>=.2777&&e<=.2778?this.character="\u2005\u200A":e>=-.05556&&e<=-.05555?this.character="\u200A\u2063":e>=-.1667&&e<=-.1666?this.character="\u2009\u2063":e>=-.2223&&e<=-.2222?this.character="\u205F\u2063":e>=-.2778&&e<=-.2777?this.character="\u2005\u2063":this.character=null}toNode(){if(this.character)return document.createTextNode(this.character);var e=document.createElementNS("http://www.w3.org/1998/Math/MathML","mspace");return e.setAttribute("width",kt(this.width)),e}toMarkup(){return this.character?""+this.character+"":''}toText(){return this.character?this.character:" "}},dt={MathNode:ws,TextNode:Jf,SpaceNode:D7,newDocumentFragment:Qz},Co=o(function(e,r,n){return An[r][e]&&An[r][e].replace&&e.charCodeAt(0)!==55349&&!(Hz.hasOwnProperty(e)&&n&&(n.fontFamily&&n.fontFamily.slice(4,6)==="tt"||n.font&&n.font.slice(4,6)==="tt"))&&(e=An[r][e].replace),new dt.TextNode(e)},"makeText"),F7=o(function(e){return e.length===1?e[0]:new dt.MathNode("mrow",e)},"makeRow"),$7=o(function(e,r){if(r.fontFamily==="texttt")return"monospace";if(r.fontFamily==="textsf")return r.fontShape==="textit"&&r.fontWeight==="textbf"?"sans-serif-bold-italic":r.fontShape==="textit"?"sans-serif-italic":r.fontWeight==="textbf"?"bold-sans-serif":"sans-serif";if(r.fontShape==="textit"&&r.fontWeight==="textbf")return"bold-italic";if(r.fontShape==="textit")return"italic";if(r.fontWeight==="textbf")return"bold";var n=r.font;if(!n||n==="mathnormal")return null;var i=e.mode;if(n==="mathit")return"italic";if(n==="boldsymbol")return e.type==="textord"?"bold":"bold-italic";if(n==="mathbf")return"bold";if(n==="mathbb")return"double-struck";if(n==="mathfrak")return"fraktur";if(n==="mathscr"||n==="mathcal")return"script";if(n==="mathsf")return"sans-serif";if(n==="mathtt")return"monospace";var a=e.text;if(Jt.contains(["\\imath","\\jmath"],a))return null;An[i][a]&&An[i][a].replace&&(a=An[i][a].replace);var s=Be.fontMap[n].fontName;return P7(a,s,i)?Be.fontMap[n].variant:null},"getVariant"),ks=o(function(e,r,n){if(e.length===1){var i=yn(e[0],r);return n&&i instanceof ws&&i.type==="mo"&&(i.setAttribute("lspace","0em"),i.setAttribute("rspace","0em")),[i]}for(var a=[],s,l=0;l0&&(d.text=d.text.slice(0,1)+"\u0338"+d.text.slice(1),a.pop())}}}a.push(u),s=u}return a},"buildExpression"),dh=o(function(e,r,n){return F7(ks(e,r,n))},"buildExpressionRow"),yn=o(function(e,r){if(!e)return new dt.MathNode("mrow");if(m3[e.type]){var n=m3[e.type](e,r);return n}else throw new gt("Got group of unknown type: '"+e.type+"'")},"buildGroup");o(gz,"buildMathML");Zz=o(function(e){return new f3({style:e.displayMode?tr.DISPLAY:tr.TEXT,maxSize:e.maxSize,minRuleThickness:e.minRuleThickness})},"optionsFromSettings"),Jz=o(function(e,r){if(r.displayMode){var n=["katex-display"];r.leqno&&n.push("leqno"),r.fleqn&&n.push("fleqn"),e=Be.makeSpan(n,[e])}return e},"displayWrap"),t4e=o(function(e,r,n){var i=Zz(n),a;if(n.output==="mathml")return gz(e,r,i,n.displayMode,!0);if(n.output==="html"){var s=_7(e,i);a=Be.makeSpan(["katex"],[s])}else{var l=gz(e,r,i,n.displayMode,!1),u=_7(e,i);a=Be.makeSpan(["katex"],[l,u])}return Jz(a,n)},"buildTree"),r4e=o(function(e,r,n){var i=Zz(n),a=_7(e,i),s=Be.makeSpan(["katex"],[a]);return Jz(s,n)},"buildHTMLTree"),n4e={widehat:"^",widecheck:"\u02C7",widetilde:"~",utilde:"~",overleftarrow:"\u2190",underleftarrow:"\u2190",xleftarrow:"\u2190",overrightarrow:"\u2192",underrightarrow:"\u2192",xrightarrow:"\u2192",underbrace:"\u23DF",overbrace:"\u23DE",overgroup:"\u23E0",undergroup:"\u23E1",overleftrightarrow:"\u2194",underleftrightarrow:"\u2194",xleftrightarrow:"\u2194",Overrightarrow:"\u21D2",xRightarrow:"\u21D2",overleftharpoon:"\u21BC",xleftharpoonup:"\u21BC",overrightharpoon:"\u21C0",xrightharpoonup:"\u21C0",xLeftarrow:"\u21D0",xLeftrightarrow:"\u21D4",xhookleftarrow:"\u21A9",xhookrightarrow:"\u21AA",xmapsto:"\u21A6",xrightharpoondown:"\u21C1",xleftharpoondown:"\u21BD",xrightleftharpoons:"\u21CC",xleftrightharpoons:"\u21CB",xtwoheadleftarrow:"\u219E",xtwoheadrightarrow:"\u21A0",xlongequal:"=",xtofrom:"\u21C4",xrightleftarrows:"\u21C4",xrightequilibrium:"\u21CC",xleftequilibrium:"\u21CB","\\cdrightarrow":"\u2192","\\cdleftarrow":"\u2190","\\cdlongequal":"="},i4e=o(function(e){var r=new dt.MathNode("mo",[new dt.TextNode(n4e[e.replace(/^\\/,"")])]);return r.setAttribute("stretchy","true"),r},"mathMLnode"),a4e={overrightarrow:[["rightarrow"],.888,522,"xMaxYMin"],overleftarrow:[["leftarrow"],.888,522,"xMinYMin"],underrightarrow:[["rightarrow"],.888,522,"xMaxYMin"],underleftarrow:[["leftarrow"],.888,522,"xMinYMin"],xrightarrow:[["rightarrow"],1.469,522,"xMaxYMin"],"\\cdrightarrow":[["rightarrow"],3,522,"xMaxYMin"],xleftarrow:[["leftarrow"],1.469,522,"xMinYMin"],"\\cdleftarrow":[["leftarrow"],3,522,"xMinYMin"],Overrightarrow:[["doublerightarrow"],.888,560,"xMaxYMin"],xRightarrow:[["doublerightarrow"],1.526,560,"xMaxYMin"],xLeftarrow:[["doubleleftarrow"],1.526,560,"xMinYMin"],overleftharpoon:[["leftharpoon"],.888,522,"xMinYMin"],xleftharpoonup:[["leftharpoon"],.888,522,"xMinYMin"],xleftharpoondown:[["leftharpoondown"],.888,522,"xMinYMin"],overrightharpoon:[["rightharpoon"],.888,522,"xMaxYMin"],xrightharpoonup:[["rightharpoon"],.888,522,"xMaxYMin"],xrightharpoondown:[["rightharpoondown"],.888,522,"xMaxYMin"],xlongequal:[["longequal"],.888,334,"xMinYMin"],"\\cdlongequal":[["longequal"],3,334,"xMinYMin"],xtwoheadleftarrow:[["twoheadleftarrow"],.888,334,"xMinYMin"],xtwoheadrightarrow:[["twoheadrightarrow"],.888,334,"xMaxYMin"],overleftrightarrow:[["leftarrow","rightarrow"],.888,522],overbrace:[["leftbrace","midbrace","rightbrace"],1.6,548],underbrace:[["leftbraceunder","midbraceunder","rightbraceunder"],1.6,548],underleftrightarrow:[["leftarrow","rightarrow"],.888,522],xleftrightarrow:[["leftarrow","rightarrow"],1.75,522],xLeftrightarrow:[["doubleleftarrow","doublerightarrow"],1.75,560],xrightleftharpoons:[["leftharpoondownplus","rightharpoonplus"],1.75,716],xleftrightharpoons:[["leftharpoonplus","rightharpoondownplus"],1.75,716],xhookleftarrow:[["leftarrow","righthook"],1.08,522],xhookrightarrow:[["lefthook","rightarrow"],1.08,522],overlinesegment:[["leftlinesegment","rightlinesegment"],.888,522],underlinesegment:[["leftlinesegment","rightlinesegment"],.888,522],overgroup:[["leftgroup","rightgroup"],.888,342],undergroup:[["leftgroupunder","rightgroupunder"],.888,342],xmapsto:[["leftmapsto","rightarrow"],1.5,522],xtofrom:[["leftToFrom","rightToFrom"],1.75,528],xrightleftarrows:[["baraboveleftarrow","rightarrowabovebar"],1.75,901],xrightequilibrium:[["baraboveshortleftharpoon","rightharpoonaboveshortbar"],1.75,716],xleftequilibrium:[["shortbaraboveleftharpoon","shortrightharpoonabovebar"],1.75,716]},s4e=o(function(e){return e.type==="ordgroup"?e.body.length:1},"groupLength"),o4e=o(function(e,r){function n(){var l=4e5,u=e.label.slice(1);if(Jt.contains(["widehat","widecheck","widetilde","utilde"],u)){var h=e,f=s4e(h.base),d,p,m;if(f>5)u==="widehat"||u==="widecheck"?(d=420,l=2364,m=.42,p=u+"4"):(d=312,l=2340,m=.34,p="tilde4");else{var g=[1,1,2,2,3,3][f];u==="widehat"||u==="widecheck"?(l=[0,1062,2364,2364,2364][g],d=[0,239,300,360,420][g],m=[0,.24,.3,.3,.36,.42][g],p=u+g):(l=[0,600,1033,2339,2340][g],d=[0,260,286,306,312][g],m=[0,.26,.286,.3,.306,.34][g],p="tilde"+g)}var y=new Kl(p),v=new ll([y],{width:"100%",height:kt(m),viewBox:"0 0 "+l+" "+d,preserveAspectRatio:"none"});return{span:Be.makeSvgSpan([],[v],r),minWidth:0,height:m}}else{var x=[],b=a4e[u],[w,C,T]=b,E=T/1e3,A=w.length,S,_;if(A===1){var I=b[3];S=["hide-tail"],_=[I]}else if(A===2)S=["halfarrow-left","halfarrow-right"],_=["xMinYMin","xMaxYMin"];else if(A===3)S=["brace-left","brace-center","brace-right"],_=["xMinYMin","xMidYMin","xMaxYMin"];else throw new Error(`Correct katexImagesData or update code here to support + `+A+" children.");for(var D=0;D0&&(i.style.minWidth=kt(a)),i},"svgSpan"),l4e=o(function(e,r,n,i,a){var s,l=e.height+e.depth+n+i;if(/fbox|color|angl/.test(r)){if(s=Be.makeSpan(["stretchy",r],[],a),r==="fbox"){var u=a.color&&a.getColor();u&&(s.style.borderColor=u)}}else{var h=[];/^[bx]cancel$/.test(r)&&h.push(new Uy({x1:"0",y1:"0",x2:"100%",y2:"100%","stroke-width":"0.046em"})),/^x?cancel$/.test(r)&&h.push(new Uy({x1:"0",y1:"100%",x2:"100%",y2:"0","stroke-width":"0.046em"}));var f=new ll(h,{width:"100%",height:kt(l)});s=Be.makeSvgSpan([],[f],a)}return s.height=l,s.style.height=kt(l),s},"encloseSpan"),cu={encloseSpan:l4e,mathMLnode:i4e,svgSpan:o4e};o(xr,"assertNodeType");o(z7,"assertSymbolNodeType");o(w3,"checkSymbolNodeType");G7=o((t,e)=>{var r,n,i;t&&t.type==="supsub"?(n=xr(t.base,"accent"),r=n.base,t.base=r,i=Nbe(Fr(t,e)),t.base=n):(n=xr(t,"accent"),r=n.base);var a=Fr(r,e.havingCrampedStyle()),s=n.isShifty&&Jt.isCharacterBox(r),l=0;if(s){var u=Jt.getBaseElem(r),h=Fr(u,e.havingCrampedStyle());l=hz(h).skew}var f=n.label==="\\c",d=f?a.height+a.depth:Math.min(a.height,e.fontMetrics().xHeight),p;if(n.isStretchy)p=cu.svgSpan(n,e),p=Be.makeVList({positionType:"firstBaseline",children:[{type:"elem",elem:a},{type:"elem",elem:p,wrapperClasses:["svg-align"],wrapperStyle:l>0?{width:"calc(100% - "+kt(2*l)+")",marginLeft:kt(2*l)}:void 0}]},e);else{var m,g;n.label==="\\vec"?(m=Be.staticSvg("vec",e),g=Be.svgData.vec[1]):(m=Be.makeOrd({mode:n.mode,text:n.label},e,"textord"),m=hz(m),m.italic=0,g=m.width,f&&(d+=m.depth)),p=Be.makeSpan(["accent-body"],[m]);var y=n.label==="\\textcircled";y&&(p.classes.push("accent-full"),d=a.height);var v=l;y||(v-=g/2),p.style.left=kt(v),n.label==="\\textcircled"&&(p.style.top=".2em"),p=Be.makeVList({positionType:"firstBaseline",children:[{type:"elem",elem:a},{type:"kern",size:-d},{type:"elem",elem:p}]},e)}var x=Be.makeSpan(["mord","accent"],[p],e);return i?(i.children[0]=x,i.height=Math.max(x.height,i.height),i.classes[0]="mord",i):x},"htmlBuilder$a"),eG=o((t,e)=>{var r=t.isStretchy?cu.mathMLnode(t.label):new dt.MathNode("mo",[Co(t.label,t.mode)]),n=new dt.MathNode("mover",[yn(t.base,e),r]);return n.setAttribute("accent","true"),n},"mathmlBuilder$9"),c4e=new RegExp(["\\acute","\\grave","\\ddot","\\tilde","\\bar","\\breve","\\check","\\hat","\\vec","\\dot","\\mathring"].map(t=>"\\"+t).join("|"));Nt({type:"accent",names:["\\acute","\\grave","\\ddot","\\tilde","\\bar","\\breve","\\check","\\hat","\\vec","\\dot","\\mathring","\\widecheck","\\widehat","\\widetilde","\\overrightarrow","\\overleftarrow","\\Overrightarrow","\\overleftrightarrow","\\overgroup","\\overlinesegment","\\overleftharpoon","\\overrightharpoon"],props:{numArgs:1},handler:o((t,e)=>{var r=g3(e[0]),n=!c4e.test(t.funcName),i=!n||t.funcName==="\\widehat"||t.funcName==="\\widetilde"||t.funcName==="\\widecheck";return{type:"accent",mode:t.parser.mode,label:t.funcName,isStretchy:n,isShifty:i,base:r}},"handler"),htmlBuilder:G7,mathmlBuilder:eG});Nt({type:"accent",names:["\\'","\\`","\\^","\\~","\\=","\\u","\\.",'\\"',"\\c","\\r","\\H","\\v","\\textcircled"],props:{numArgs:1,allowedInText:!0,allowedInMath:!0,argTypes:["primitive"]},handler:o((t,e)=>{var r=e[0],n=t.parser.mode;return n==="math"&&(t.parser.settings.reportNonstrict("mathVsTextAccents","LaTeX's accent "+t.funcName+" works only in text mode"),n="text"),{type:"accent",mode:n,label:t.funcName,isStretchy:!1,isShifty:!0,base:r}},"handler"),htmlBuilder:G7,mathmlBuilder:eG});Nt({type:"accentUnder",names:["\\underleftarrow","\\underrightarrow","\\underleftrightarrow","\\undergroup","\\underlinesegment","\\utilde"],props:{numArgs:1},handler:o((t,e)=>{var{parser:r,funcName:n}=t,i=e[0];return{type:"accentUnder",mode:r.mode,label:n,base:i}},"handler"),htmlBuilder:o((t,e)=>{var r=Fr(t.base,e),n=cu.svgSpan(t,e),i=t.label==="\\utilde"?.12:0,a=Be.makeVList({positionType:"top",positionData:r.height,children:[{type:"elem",elem:n,wrapperClasses:["svg-align"]},{type:"kern",size:i},{type:"elem",elem:r}]},e);return Be.makeSpan(["mord","accentunder"],[a],e)},"htmlBuilder"),mathmlBuilder:o((t,e)=>{var r=cu.mathMLnode(t.label),n=new dt.MathNode("munder",[yn(t.base,e),r]);return n.setAttribute("accentunder","true"),n},"mathmlBuilder")});s3=o(t=>{var e=new dt.MathNode("mpadded",t?[t]:[]);return e.setAttribute("width","+0.6em"),e.setAttribute("lspace","0.3em"),e},"paddedNode");Nt({type:"xArrow",names:["\\xleftarrow","\\xrightarrow","\\xLeftarrow","\\xRightarrow","\\xleftrightarrow","\\xLeftrightarrow","\\xhookleftarrow","\\xhookrightarrow","\\xmapsto","\\xrightharpoondown","\\xrightharpoonup","\\xleftharpoondown","\\xleftharpoonup","\\xrightleftharpoons","\\xleftrightharpoons","\\xlongequal","\\xtwoheadrightarrow","\\xtwoheadleftarrow","\\xtofrom","\\xrightleftarrows","\\xrightequilibrium","\\xleftequilibrium","\\\\cdrightarrow","\\\\cdleftarrow","\\\\cdlongequal"],props:{numArgs:1,numOptionalArgs:1},handler(t,e,r){var{parser:n,funcName:i}=t;return{type:"xArrow",mode:n.mode,label:i,body:e[0],below:r[0]}},htmlBuilder(t,e){var r=e.style,n=e.havingStyle(r.sup()),i=Be.wrapFragment(Fr(t.body,n,e),e),a=t.label.slice(0,2)==="\\x"?"x":"cd";i.classes.push(a+"-arrow-pad");var s;t.below&&(n=e.havingStyle(r.sub()),s=Be.wrapFragment(Fr(t.below,n,e),e),s.classes.push(a+"-arrow-pad"));var l=cu.svgSpan(t,e),u=-e.fontMetrics().axisHeight+.5*l.height,h=-e.fontMetrics().axisHeight-.5*l.height-.111;(i.depth>.25||t.label==="\\xleftequilibrium")&&(h-=i.depth);var f;if(s){var d=-e.fontMetrics().axisHeight+s.height+.5*l.height+.111;f=Be.makeVList({positionType:"individualShift",children:[{type:"elem",elem:i,shift:h},{type:"elem",elem:l,shift:u},{type:"elem",elem:s,shift:d}]},e)}else f=Be.makeVList({positionType:"individualShift",children:[{type:"elem",elem:i,shift:h},{type:"elem",elem:l,shift:u}]},e);return f.children[0].children[0].children[1].classes.push("svg-align"),Be.makeSpan(["mrel","x-arrow"],[f],e)},mathmlBuilder(t,e){var r=cu.mathMLnode(t.label);r.setAttribute("minsize",t.label.charAt(0)==="x"?"1.75em":"3.0em");var n;if(t.body){var i=s3(yn(t.body,e));if(t.below){var a=s3(yn(t.below,e));n=new dt.MathNode("munderover",[r,a,i])}else n=new dt.MathNode("mover",[r,i])}else if(t.below){var s=s3(yn(t.below,e));n=new dt.MathNode("munder",[r,s])}else n=s3(),n=new dt.MathNode("mover",[r,n]);return n}});u4e=Be.makeSpan;o(tG,"htmlBuilder$9");o(rG,"mathmlBuilder$8");Nt({type:"mclass",names:["\\mathord","\\mathbin","\\mathrel","\\mathopen","\\mathclose","\\mathpunct","\\mathinner"],props:{numArgs:1,primitive:!0},handler(t,e){var{parser:r,funcName:n}=t,i=e[0];return{type:"mclass",mode:r.mode,mclass:"m"+n.slice(5),body:di(i),isCharacterBox:Jt.isCharacterBox(i)}},htmlBuilder:tG,mathmlBuilder:rG});T3=o(t=>{var e=t.type==="ordgroup"&&t.body.length?t.body[0]:t;return e.type==="atom"&&(e.family==="bin"||e.family==="rel")?"m"+e.family:"mord"},"binrelClass");Nt({type:"mclass",names:["\\@binrel"],props:{numArgs:2},handler(t,e){var{parser:r}=t;return{type:"mclass",mode:r.mode,mclass:T3(e[0]),body:di(e[1]),isCharacterBox:Jt.isCharacterBox(e[1])}}});Nt({type:"mclass",names:["\\stackrel","\\overset","\\underset"],props:{numArgs:2},handler(t,e){var{parser:r,funcName:n}=t,i=e[1],a=e[0],s;n!=="\\stackrel"?s=T3(i):s="mrel";var l={type:"op",mode:i.mode,limits:!0,alwaysHandleSupSub:!0,parentIsSupSub:!1,symbol:!1,suppressBaseShift:n!=="\\stackrel",body:di(i)},u={type:"supsub",mode:a.mode,base:l,sup:n==="\\underset"?null:a,sub:n==="\\underset"?a:null};return{type:"mclass",mode:r.mode,mclass:s,body:[u],isCharacterBox:Jt.isCharacterBox(u)}},htmlBuilder:tG,mathmlBuilder:rG});Nt({type:"pmb",names:["\\pmb"],props:{numArgs:1,allowedInText:!0},handler(t,e){var{parser:r}=t;return{type:"pmb",mode:r.mode,mclass:T3(e[0]),body:di(e[0])}},htmlBuilder(t,e){var r=Pi(t.body,e,!0),n=Be.makeSpan([t.mclass],r,e);return n.style.textShadow="0.02em 0.01em 0.04px",n},mathmlBuilder(t,e){var r=ks(t.body,e),n=new dt.MathNode("mstyle",r);return n.setAttribute("style","text-shadow: 0.02em 0.01em 0.04px"),n}});h4e={">":"\\\\cdrightarrow","<":"\\\\cdleftarrow","=":"\\\\cdlongequal",A:"\\uparrow",V:"\\downarrow","|":"\\Vert",".":"no arrow"},yz=o(()=>({type:"styling",body:[],mode:"math",style:"display"}),"newCell"),vz=o(t=>t.type==="textord"&&t.text==="@","isStartOfArrow"),f4e=o((t,e)=>(t.type==="mathord"||t.type==="atom")&&t.text===e,"isLabelEnd");o(d4e,"cdArrow");o(p4e,"parseCD");Nt({type:"cdlabel",names:["\\\\cdleft","\\\\cdright"],props:{numArgs:1},handler(t,e){var{parser:r,funcName:n}=t;return{type:"cdlabel",mode:r.mode,side:n.slice(4),label:e[0]}},htmlBuilder(t,e){var r=e.havingStyle(e.style.sup()),n=Be.wrapFragment(Fr(t.label,r,e),e);return n.classes.push("cd-label-"+t.side),n.style.bottom=kt(.8-n.depth),n.height=0,n.depth=0,n},mathmlBuilder(t,e){var r=new dt.MathNode("mrow",[yn(t.label,e)]);return r=new dt.MathNode("mpadded",[r]),r.setAttribute("width","0"),t.side==="left"&&r.setAttribute("lspace","-1width"),r.setAttribute("voffset","0.7em"),r=new dt.MathNode("mstyle",[r]),r.setAttribute("displaystyle","false"),r.setAttribute("scriptlevel","1"),r}});Nt({type:"cdlabelparent",names:["\\\\cdparent"],props:{numArgs:1},handler(t,e){var{parser:r}=t;return{type:"cdlabelparent",mode:r.mode,fragment:e[0]}},htmlBuilder(t,e){var r=Be.wrapFragment(Fr(t.fragment,e),e);return r.classes.push("cd-vert-arrow"),r},mathmlBuilder(t,e){return new dt.MathNode("mrow",[yn(t.fragment,e)])}});Nt({type:"textord",names:["\\@char"],props:{numArgs:1,allowedInText:!0},handler(t,e){for(var{parser:r}=t,n=xr(e[0],"ordgroup"),i=n.body,a="",s=0;s=1114111)throw new gt("\\@char with invalid code point "+a);return u<=65535?h=String.fromCharCode(u):(u-=65536,h=String.fromCharCode((u>>10)+55296,(u&1023)+56320)),{type:"textord",mode:r.mode,text:h}}});nG=o((t,e)=>{var r=Pi(t.body,e.withColor(t.color),!1);return Be.makeFragment(r)},"htmlBuilder$8"),iG=o((t,e)=>{var r=ks(t.body,e.withColor(t.color)),n=new dt.MathNode("mstyle",r);return n.setAttribute("mathcolor",t.color),n},"mathmlBuilder$7");Nt({type:"color",names:["\\textcolor"],props:{numArgs:2,allowedInText:!0,argTypes:["color","original"]},handler(t,e){var{parser:r}=t,n=xr(e[0],"color-token").color,i=e[1];return{type:"color",mode:r.mode,color:n,body:di(i)}},htmlBuilder:nG,mathmlBuilder:iG});Nt({type:"color",names:["\\color"],props:{numArgs:1,allowedInText:!0,argTypes:["color"]},handler(t,e){var{parser:r,breakOnTokenText:n}=t,i=xr(e[0],"color-token").color;r.gullet.macros.set("\\current@color",i);var a=r.parseExpression(!0,n);return{type:"color",mode:r.mode,color:i,body:a}},htmlBuilder:nG,mathmlBuilder:iG});Nt({type:"cr",names:["\\\\"],props:{numArgs:0,numOptionalArgs:0,allowedInText:!0},handler(t,e,r){var{parser:n}=t,i=n.gullet.future().text==="["?n.parseSizeGroup(!0):null,a=!n.settings.displayMode||!n.settings.useStrictBehavior("newLineInDisplayMode","In LaTeX, \\\\ or \\newline does nothing in display mode");return{type:"cr",mode:n.mode,newLine:a,size:i&&xr(i,"size").value}},htmlBuilder(t,e){var r=Be.makeSpan(["mspace"],[],e);return t.newLine&&(r.classes.push("newline"),t.size&&(r.style.marginTop=kt(ti(t.size,e)))),r},mathmlBuilder(t,e){var r=new dt.MathNode("mspace");return t.newLine&&(r.setAttribute("linebreak","newline"),t.size&&r.setAttribute("height",kt(ti(t.size,e)))),r}});L7={"\\global":"\\global","\\long":"\\\\globallong","\\\\globallong":"\\\\globallong","\\def":"\\gdef","\\gdef":"\\gdef","\\edef":"\\xdef","\\xdef":"\\xdef","\\let":"\\\\globallet","\\futurelet":"\\\\globalfuture"},aG=o(t=>{var e=t.text;if(/^(?:[\\{}$&#^_]|EOF)$/.test(e))throw new gt("Expected a control sequence",t);return e},"checkControlSequence"),m4e=o(t=>{var e=t.gullet.popToken();return e.text==="="&&(e=t.gullet.popToken(),e.text===" "&&(e=t.gullet.popToken())),e},"getRHS"),sG=o((t,e,r,n)=>{var i=t.gullet.macros.get(r.text);i==null&&(r.noexpand=!0,i={tokens:[r],numArgs:0,unexpandable:!t.gullet.isExpandable(r.text)}),t.gullet.macros.set(e,i,n)},"letCommand");Nt({type:"internal",names:["\\global","\\long","\\\\globallong"],props:{numArgs:0,allowedInText:!0},handler(t){var{parser:e,funcName:r}=t;e.consumeSpaces();var n=e.fetch();if(L7[n.text])return(r==="\\global"||r==="\\\\globallong")&&(n.text=L7[n.text]),xr(e.parseFunction(),"internal");throw new gt("Invalid token after macro prefix",n)}});Nt({type:"internal",names:["\\def","\\gdef","\\edef","\\xdef"],props:{numArgs:0,allowedInText:!0,primitive:!0},handler(t){var{parser:e,funcName:r}=t,n=e.gullet.popToken(),i=n.text;if(/^(?:[\\{}$&#^_]|EOF)$/.test(i))throw new gt("Expected a control sequence",n);for(var a=0,s,l=[[]];e.gullet.future().text!=="{";)if(n=e.gullet.popToken(),n.text==="#"){if(e.gullet.future().text==="{"){s=e.gullet.future(),l[a].push("{");break}if(n=e.gullet.popToken(),!/^[1-9]$/.test(n.text))throw new gt('Invalid argument number "'+n.text+'"');if(parseInt(n.text)!==a+1)throw new gt('Argument number "'+n.text+'" out of order');a++,l.push([])}else{if(n.text==="EOF")throw new gt("Expected a macro definition");l[a].push(n.text)}var{tokens:u}=e.gullet.consumeArg();return s&&u.unshift(s),(r==="\\edef"||r==="\\xdef")&&(u=e.gullet.expandTokens(u),u.reverse()),e.gullet.macros.set(i,{tokens:u,numArgs:a,delimiters:l},r===L7[r]),{type:"internal",mode:e.mode}}});Nt({type:"internal",names:["\\let","\\\\globallet"],props:{numArgs:0,allowedInText:!0,primitive:!0},handler(t){var{parser:e,funcName:r}=t,n=aG(e.gullet.popToken());e.gullet.consumeSpaces();var i=m4e(e);return sG(e,n,i,r==="\\\\globallet"),{type:"internal",mode:e.mode}}});Nt({type:"internal",names:["\\futurelet","\\\\globalfuture"],props:{numArgs:0,allowedInText:!0,primitive:!0},handler(t){var{parser:e,funcName:r}=t,n=aG(e.gullet.popToken()),i=e.gullet.popToken(),a=e.gullet.popToken();return sG(e,n,a,r==="\\\\globalfuture"),e.gullet.pushToken(a),e.gullet.pushToken(i),{type:"internal",mode:e.mode}}});Fy=o(function(e,r,n){var i=An.math[e]&&An.math[e].replace,a=P7(i||e,r,n);if(!a)throw new Error("Unsupported symbol "+e+" and font size "+r+".");return a},"getMetrics"),V7=o(function(e,r,n,i){var a=n.havingBaseStyle(r),s=Be.makeSpan(i.concat(a.sizingClasses(n)),[e],n),l=a.sizeMultiplier/n.sizeMultiplier;return s.height*=l,s.depth*=l,s.maxFontSize=a.sizeMultiplier,s},"styleWrap"),oG=o(function(e,r,n){var i=r.havingBaseStyle(n),a=(1-r.sizeMultiplier/i.sizeMultiplier)*r.fontMetrics().axisHeight;e.classes.push("delimcenter"),e.style.top=kt(a),e.height-=a,e.depth+=a},"centerSpan"),g4e=o(function(e,r,n,i,a,s){var l=Be.makeSymbol(e,"Main-Regular",a,i),u=V7(l,r,i,s);return n&&oG(u,i,r),u},"makeSmallDelim"),y4e=o(function(e,r,n,i){return Be.makeSymbol(e,"Size"+r+"-Regular",n,i)},"mathrmSize"),lG=o(function(e,r,n,i,a,s){var l=y4e(e,r,a,i),u=V7(Be.makeSpan(["delimsizing","size"+r],[l],i),tr.TEXT,i,s);return n&&oG(u,i,tr.TEXT),u},"makeLargeDelim"),p7=o(function(e,r,n){var i;r==="Size1-Regular"?i="delim-size1":i="delim-size4";var a=Be.makeSpan(["delimsizinginner",i],[Be.makeSpan([],[Be.makeSymbol(e,r,n)])]);return{type:"elem",elem:a}},"makeGlyphSpan"),m7=o(function(e,r,n){var i=jl["Size4-Regular"][e.charCodeAt(0)]?jl["Size4-Regular"][e.charCodeAt(0)][4]:jl["Size1-Regular"][e.charCodeAt(0)][4],a=new Kl("inner",Sbe(e,Math.round(1e3*r))),s=new ll([a],{width:kt(i),height:kt(r),style:"width:"+kt(i),viewBox:"0 0 "+1e3*i+" "+Math.round(1e3*r),preserveAspectRatio:"xMinYMin"}),l=Be.makeSvgSpan([],[s],n);return l.height=r,l.style.height=kt(r),l.style.width=kt(i),{type:"elem",elem:l}},"makeInner"),R7=.008,o3={type:"kern",size:-1*R7},v4e=["|","\\lvert","\\rvert","\\vert"],x4e=["\\|","\\lVert","\\rVert","\\Vert"],cG=o(function(e,r,n,i,a,s){var l,u,h,f,d="",p=0;l=h=f=e,u=null;var m="Size1-Regular";e==="\\uparrow"?h=f="\u23D0":e==="\\Uparrow"?h=f="\u2016":e==="\\downarrow"?l=h="\u23D0":e==="\\Downarrow"?l=h="\u2016":e==="\\updownarrow"?(l="\\uparrow",h="\u23D0",f="\\downarrow"):e==="\\Updownarrow"?(l="\\Uparrow",h="\u2016",f="\\Downarrow"):Jt.contains(v4e,e)?(h="\u2223",d="vert",p=333):Jt.contains(x4e,e)?(h="\u2225",d="doublevert",p=556):e==="["||e==="\\lbrack"?(l="\u23A1",h="\u23A2",f="\u23A3",m="Size4-Regular",d="lbrack",p=667):e==="]"||e==="\\rbrack"?(l="\u23A4",h="\u23A5",f="\u23A6",m="Size4-Regular",d="rbrack",p=667):e==="\\lfloor"||e==="\u230A"?(h=l="\u23A2",f="\u23A3",m="Size4-Regular",d="lfloor",p=667):e==="\\lceil"||e==="\u2308"?(l="\u23A1",h=f="\u23A2",m="Size4-Regular",d="lceil",p=667):e==="\\rfloor"||e==="\u230B"?(h=l="\u23A5",f="\u23A6",m="Size4-Regular",d="rfloor",p=667):e==="\\rceil"||e==="\u2309"?(l="\u23A4",h=f="\u23A5",m="Size4-Regular",d="rceil",p=667):e==="("||e==="\\lparen"?(l="\u239B",h="\u239C",f="\u239D",m="Size4-Regular",d="lparen",p=875):e===")"||e==="\\rparen"?(l="\u239E",h="\u239F",f="\u23A0",m="Size4-Regular",d="rparen",p=875):e==="\\{"||e==="\\lbrace"?(l="\u23A7",u="\u23A8",f="\u23A9",h="\u23AA",m="Size4-Regular"):e==="\\}"||e==="\\rbrace"?(l="\u23AB",u="\u23AC",f="\u23AD",h="\u23AA",m="Size4-Regular"):e==="\\lgroup"||e==="\u27EE"?(l="\u23A7",f="\u23A9",h="\u23AA",m="Size4-Regular"):e==="\\rgroup"||e==="\u27EF"?(l="\u23AB",f="\u23AD",h="\u23AA",m="Size4-Regular"):e==="\\lmoustache"||e==="\u23B0"?(l="\u23A7",f="\u23AD",h="\u23AA",m="Size4-Regular"):(e==="\\rmoustache"||e==="\u23B1")&&(l="\u23AB",f="\u23A9",h="\u23AA",m="Size4-Regular");var g=Fy(l,m,a),y=g.height+g.depth,v=Fy(h,m,a),x=v.height+v.depth,b=Fy(f,m,a),w=b.height+b.depth,C=0,T=1;if(u!==null){var E=Fy(u,m,a);C=E.height+E.depth,T=2}var A=y+w+C,S=Math.max(0,Math.ceil((r-A)/(T*x))),_=A+S*T*x,I=i.fontMetrics().axisHeight;n&&(I*=i.sizeMultiplier);var D=_/2-I,k=[];if(d.length>0){var L=_-y-w,R=Math.round(_*1e3),O=Cbe(d,Math.round(L*1e3)),M=new Kl(d,O),B=(p/1e3).toFixed(3)+"em",F=(R/1e3).toFixed(3)+"em",P=new ll([M],{width:B,height:F,viewBox:"0 0 "+p+" "+R}),z=Be.makeSvgSpan([],[P],i);z.height=R/1e3,z.style.width=B,z.style.height=F,k.push({type:"elem",elem:z})}else{if(k.push(p7(f,m,a)),k.push(o3),u===null){var $=_-y-w+2*R7;k.push(m7(h,$,i))}else{var H=(_-y-w-C)/2+2*R7;k.push(m7(h,H,i)),k.push(o3),k.push(p7(u,m,a)),k.push(o3),k.push(m7(h,H,i))}k.push(o3),k.push(p7(l,m,a))}var Q=i.havingBaseStyle(tr.TEXT),j=Be.makeVList({positionType:"bottom",positionData:D,children:k},Q);return V7(Be.makeSpan(["delimsizing","mult"],[j],Q),tr.TEXT,i,s)},"makeStackedDelim"),g7=80,y7=.08,v7=o(function(e,r,n,i,a){var s=Ebe(e,i,n),l=new Kl(e,s),u=new ll([l],{width:"400em",height:kt(r),viewBox:"0 0 400000 "+n,preserveAspectRatio:"xMinYMin slice"});return Be.makeSvgSpan(["hide-tail"],[u],a)},"sqrtSvg"),b4e=o(function(e,r){var n=r.havingBaseSizing(),i=dG("\\surd",e*n.sizeMultiplier,fG,n),a=n.sizeMultiplier,s=Math.max(0,r.minRuleThickness-r.fontMetrics().sqrtRuleThickness),l,u=0,h=0,f=0,d;return i.type==="small"?(f=1e3+1e3*s+g7,e<1?a=1:e<1.4&&(a=.7),u=(1+s+y7)/a,h=(1+s)/a,l=v7("sqrtMain",u,f,s,r),l.style.minWidth="0.853em",d=.833/a):i.type==="large"?(f=(1e3+g7)*$y[i.size],h=($y[i.size]+s)/a,u=($y[i.size]+s+y7)/a,l=v7("sqrtSize"+i.size,u,f,s,r),l.style.minWidth="1.02em",d=1/a):(u=e+s+y7,h=e+s,f=Math.floor(1e3*e+s)+g7,l=v7("sqrtTall",u,f,s,r),l.style.minWidth="0.742em",d=1.056),l.height=h,l.style.height=kt(u),{span:l,advanceWidth:d,ruleWidth:(r.fontMetrics().sqrtRuleThickness+s)*a}},"makeSqrtImage"),uG=["(","\\lparen",")","\\rparen","[","\\lbrack","]","\\rbrack","\\{","\\lbrace","\\}","\\rbrace","\\lfloor","\\rfloor","\u230A","\u230B","\\lceil","\\rceil","\u2308","\u2309","\\surd"],w4e=["\\uparrow","\\downarrow","\\updownarrow","\\Uparrow","\\Downarrow","\\Updownarrow","|","\\|","\\vert","\\Vert","\\lvert","\\rvert","\\lVert","\\rVert","\\lgroup","\\rgroup","\u27EE","\u27EF","\\lmoustache","\\rmoustache","\u23B0","\u23B1"],hG=["<",">","\\langle","\\rangle","/","\\backslash","\\lt","\\gt"],$y=[0,1.2,1.8,2.4,3],T4e=o(function(e,r,n,i,a){if(e==="<"||e==="\\lt"||e==="\u27E8"?e="\\langle":(e===">"||e==="\\gt"||e==="\u27E9")&&(e="\\rangle"),Jt.contains(uG,e)||Jt.contains(hG,e))return lG(e,r,!1,n,i,a);if(Jt.contains(w4e,e))return cG(e,$y[r],!1,n,i,a);throw new gt("Illegal delimiter: '"+e+"'")},"makeSizedDelim"),k4e=[{type:"small",style:tr.SCRIPTSCRIPT},{type:"small",style:tr.SCRIPT},{type:"small",style:tr.TEXT},{type:"large",size:1},{type:"large",size:2},{type:"large",size:3},{type:"large",size:4}],E4e=[{type:"small",style:tr.SCRIPTSCRIPT},{type:"small",style:tr.SCRIPT},{type:"small",style:tr.TEXT},{type:"stack"}],fG=[{type:"small",style:tr.SCRIPTSCRIPT},{type:"small",style:tr.SCRIPT},{type:"small",style:tr.TEXT},{type:"large",size:1},{type:"large",size:2},{type:"large",size:3},{type:"large",size:4},{type:"stack"}],S4e=o(function(e){if(e.type==="small")return"Main-Regular";if(e.type==="large")return"Size"+e.size+"-Regular";if(e.type==="stack")return"Size4-Regular";throw new Error("Add support for delim type '"+e.type+"' here.")},"delimTypeToFont"),dG=o(function(e,r,n,i){for(var a=Math.min(2,3-i.style.size),s=a;sr)return n[s]}return n[n.length-1]},"traverseSequence"),pG=o(function(e,r,n,i,a,s){e==="<"||e==="\\lt"||e==="\u27E8"?e="\\langle":(e===">"||e==="\\gt"||e==="\u27E9")&&(e="\\rangle");var l;Jt.contains(hG,e)?l=k4e:Jt.contains(uG,e)?l=fG:l=E4e;var u=dG(e,r,l,i);return u.type==="small"?g4e(e,u.style,n,i,a,s):u.type==="large"?lG(e,u.size,n,i,a,s):cG(e,r,n,i,a,s)},"makeCustomSizedDelim"),C4e=o(function(e,r,n,i,a,s){var l=i.fontMetrics().axisHeight*i.sizeMultiplier,u=901,h=5/i.fontMetrics().ptPerEm,f=Math.max(r-l,n+l),d=Math.max(f/500*u,2*f-h);return pG(e,d,!0,i,a,s)},"makeLeftRightDelim"),ou={sqrtImage:b4e,sizedDelim:T4e,sizeToMaxHeight:$y,customSizedDelim:pG,leftRightDelim:C4e},xz={"\\bigl":{mclass:"mopen",size:1},"\\Bigl":{mclass:"mopen",size:2},"\\biggl":{mclass:"mopen",size:3},"\\Biggl":{mclass:"mopen",size:4},"\\bigr":{mclass:"mclose",size:1},"\\Bigr":{mclass:"mclose",size:2},"\\biggr":{mclass:"mclose",size:3},"\\Biggr":{mclass:"mclose",size:4},"\\bigm":{mclass:"mrel",size:1},"\\Bigm":{mclass:"mrel",size:2},"\\biggm":{mclass:"mrel",size:3},"\\Biggm":{mclass:"mrel",size:4},"\\big":{mclass:"mord",size:1},"\\Big":{mclass:"mord",size:2},"\\bigg":{mclass:"mord",size:3},"\\Bigg":{mclass:"mord",size:4}},A4e=["(","\\lparen",")","\\rparen","[","\\lbrack","]","\\rbrack","\\{","\\lbrace","\\}","\\rbrace","\\lfloor","\\rfloor","\u230A","\u230B","\\lceil","\\rceil","\u2308","\u2309","<",">","\\langle","\u27E8","\\rangle","\u27E9","\\lt","\\gt","\\lvert","\\rvert","\\lVert","\\rVert","\\lgroup","\\rgroup","\u27EE","\u27EF","\\lmoustache","\\rmoustache","\u23B0","\u23B1","/","\\backslash","|","\\vert","\\|","\\Vert","\\uparrow","\\Uparrow","\\downarrow","\\Downarrow","\\updownarrow","\\Updownarrow","."];o(k3,"checkDelimiter");Nt({type:"delimsizing",names:["\\bigl","\\Bigl","\\biggl","\\Biggl","\\bigr","\\Bigr","\\biggr","\\Biggr","\\bigm","\\Bigm","\\biggm","\\Biggm","\\big","\\Big","\\bigg","\\Bigg"],props:{numArgs:1,argTypes:["primitive"]},handler:o((t,e)=>{var r=k3(e[0],t);return{type:"delimsizing",mode:t.parser.mode,size:xz[t.funcName].size,mclass:xz[t.funcName].mclass,delim:r.text}},"handler"),htmlBuilder:o((t,e)=>t.delim==="."?Be.makeSpan([t.mclass]):ou.sizedDelim(t.delim,t.size,e,t.mode,[t.mclass]),"htmlBuilder"),mathmlBuilder:o(t=>{var e=[];t.delim!=="."&&e.push(Co(t.delim,t.mode));var r=new dt.MathNode("mo",e);t.mclass==="mopen"||t.mclass==="mclose"?r.setAttribute("fence","true"):r.setAttribute("fence","false"),r.setAttribute("stretchy","true");var n=kt(ou.sizeToMaxHeight[t.size]);return r.setAttribute("minsize",n),r.setAttribute("maxsize",n),r},"mathmlBuilder")});o(bz,"assertParsed");Nt({type:"leftright-right",names:["\\right"],props:{numArgs:1,primitive:!0},handler:o((t,e)=>{var r=t.parser.gullet.macros.get("\\current@color");if(r&&typeof r!="string")throw new gt("\\current@color set to non-string in \\right");return{type:"leftright-right",mode:t.parser.mode,delim:k3(e[0],t).text,color:r}},"handler")});Nt({type:"leftright",names:["\\left"],props:{numArgs:1,primitive:!0},handler:o((t,e)=>{var r=k3(e[0],t),n=t.parser;++n.leftrightDepth;var i=n.parseExpression(!1);--n.leftrightDepth,n.expect("\\right",!1);var a=xr(n.parseFunction(),"leftright-right");return{type:"leftright",mode:n.mode,body:i,left:r.text,right:a.delim,rightColor:a.color}},"handler"),htmlBuilder:o((t,e)=>{bz(t);for(var r=Pi(t.body,e,!0,["mopen","mclose"]),n=0,i=0,a=!1,s=0;s{bz(t);var r=ks(t.body,e);if(t.left!=="."){var n=new dt.MathNode("mo",[Co(t.left,t.mode)]);n.setAttribute("fence","true"),r.unshift(n)}if(t.right!=="."){var i=new dt.MathNode("mo",[Co(t.right,t.mode)]);i.setAttribute("fence","true"),t.rightColor&&i.setAttribute("mathcolor",t.rightColor),r.push(i)}return F7(r)},"mathmlBuilder")});Nt({type:"middle",names:["\\middle"],props:{numArgs:1,primitive:!0},handler:o((t,e)=>{var r=k3(e[0],t);if(!t.parser.leftrightDepth)throw new gt("\\middle without preceding \\left",r);return{type:"middle",mode:t.parser.mode,delim:r.text}},"handler"),htmlBuilder:o((t,e)=>{var r;if(t.delim===".")r=Hy(e,[]);else{r=ou.sizedDelim(t.delim,1,e,t.mode,[]);var n={delim:t.delim,options:e};r.isMiddle=n}return r},"htmlBuilder"),mathmlBuilder:o((t,e)=>{var r=t.delim==="\\vert"||t.delim==="|"?Co("|","text"):Co(t.delim,t.mode),n=new dt.MathNode("mo",[r]);return n.setAttribute("fence","true"),n.setAttribute("lspace","0.05em"),n.setAttribute("rspace","0.05em"),n},"mathmlBuilder")});U7=o((t,e)=>{var r=Be.wrapFragment(Fr(t.body,e),e),n=t.label.slice(1),i=e.sizeMultiplier,a,s=0,l=Jt.isCharacterBox(t.body);if(n==="sout")a=Be.makeSpan(["stretchy","sout"]),a.height=e.fontMetrics().defaultRuleThickness/i,s=-.5*e.fontMetrics().xHeight;else if(n==="phase"){var u=ti({number:.6,unit:"pt"},e),h=ti({number:.35,unit:"ex"},e),f=e.havingBaseSizing();i=i/f.sizeMultiplier;var d=r.height+r.depth+u+h;r.style.paddingLeft=kt(d/2+u);var p=Math.floor(1e3*d*i),m=Tbe(p),g=new ll([new Kl("phase",m)],{width:"400em",height:kt(p/1e3),viewBox:"0 0 400000 "+p,preserveAspectRatio:"xMinYMin slice"});a=Be.makeSvgSpan(["hide-tail"],[g],e),a.style.height=kt(d),s=r.depth+u+h}else{/cancel/.test(n)?l||r.classes.push("cancel-pad"):n==="angl"?r.classes.push("anglpad"):r.classes.push("boxpad");var y=0,v=0,x=0;/box/.test(n)?(x=Math.max(e.fontMetrics().fboxrule,e.minRuleThickness),y=e.fontMetrics().fboxsep+(n==="colorbox"?0:x),v=y):n==="angl"?(x=Math.max(e.fontMetrics().defaultRuleThickness,e.minRuleThickness),y=4*x,v=Math.max(0,.25-r.depth)):(y=l?.2:0,v=y),a=cu.encloseSpan(r,n,y,v,e),/fbox|boxed|fcolorbox/.test(n)?(a.style.borderStyle="solid",a.style.borderWidth=kt(x)):n==="angl"&&x!==.049&&(a.style.borderTopWidth=kt(x),a.style.borderRightWidth=kt(x)),s=r.depth+v,t.backgroundColor&&(a.style.backgroundColor=t.backgroundColor,t.borderColor&&(a.style.borderColor=t.borderColor))}var b;if(t.backgroundColor)b=Be.makeVList({positionType:"individualShift",children:[{type:"elem",elem:a,shift:s},{type:"elem",elem:r,shift:0}]},e);else{var w=/cancel|phase/.test(n)?["svg-align"]:[];b=Be.makeVList({positionType:"individualShift",children:[{type:"elem",elem:r,shift:0},{type:"elem",elem:a,shift:s,wrapperClasses:w}]},e)}return/cancel/.test(n)&&(b.height=r.height,b.depth=r.depth),/cancel/.test(n)&&!l?Be.makeSpan(["mord","cancel-lap"],[b],e):Be.makeSpan(["mord"],[b],e)},"htmlBuilder$7"),H7=o((t,e)=>{var r=0,n=new dt.MathNode(t.label.indexOf("colorbox")>-1?"mpadded":"menclose",[yn(t.body,e)]);switch(t.label){case"\\cancel":n.setAttribute("notation","updiagonalstrike");break;case"\\bcancel":n.setAttribute("notation","downdiagonalstrike");break;case"\\phase":n.setAttribute("notation","phasorangle");break;case"\\sout":n.setAttribute("notation","horizontalstrike");break;case"\\fbox":n.setAttribute("notation","box");break;case"\\angl":n.setAttribute("notation","actuarial");break;case"\\fcolorbox":case"\\colorbox":if(r=e.fontMetrics().fboxsep*e.fontMetrics().ptPerEm,n.setAttribute("width","+"+2*r+"pt"),n.setAttribute("height","+"+2*r+"pt"),n.setAttribute("lspace",r+"pt"),n.setAttribute("voffset",r+"pt"),t.label==="\\fcolorbox"){var i=Math.max(e.fontMetrics().fboxrule,e.minRuleThickness);n.setAttribute("style","border: "+i+"em solid "+String(t.borderColor))}break;case"\\xcancel":n.setAttribute("notation","updiagonalstrike downdiagonalstrike");break}return t.backgroundColor&&n.setAttribute("mathbackground",t.backgroundColor),n},"mathmlBuilder$6");Nt({type:"enclose",names:["\\colorbox"],props:{numArgs:2,allowedInText:!0,argTypes:["color","text"]},handler(t,e,r){var{parser:n,funcName:i}=t,a=xr(e[0],"color-token").color,s=e[1];return{type:"enclose",mode:n.mode,label:i,backgroundColor:a,body:s}},htmlBuilder:U7,mathmlBuilder:H7});Nt({type:"enclose",names:["\\fcolorbox"],props:{numArgs:3,allowedInText:!0,argTypes:["color","color","text"]},handler(t,e,r){var{parser:n,funcName:i}=t,a=xr(e[0],"color-token").color,s=xr(e[1],"color-token").color,l=e[2];return{type:"enclose",mode:n.mode,label:i,backgroundColor:s,borderColor:a,body:l}},htmlBuilder:U7,mathmlBuilder:H7});Nt({type:"enclose",names:["\\fbox"],props:{numArgs:1,argTypes:["hbox"],allowedInText:!0},handler(t,e){var{parser:r}=t;return{type:"enclose",mode:r.mode,label:"\\fbox",body:e[0]}}});Nt({type:"enclose",names:["\\cancel","\\bcancel","\\xcancel","\\sout","\\phase"],props:{numArgs:1},handler(t,e){var{parser:r,funcName:n}=t,i=e[0];return{type:"enclose",mode:r.mode,label:n,body:i}},htmlBuilder:U7,mathmlBuilder:H7});Nt({type:"enclose",names:["\\angl"],props:{numArgs:1,argTypes:["hbox"],allowedInText:!1},handler(t,e){var{parser:r}=t;return{type:"enclose",mode:r.mode,label:"\\angl",body:e[0]}}});mG={};o(Ql,"defineEnvironment");gG={};o(fe,"defineMacro");o(wz,"getHLines");E3=o(t=>{var e=t.parser.settings;if(!e.displayMode)throw new gt("{"+t.envName+"} can be used only in display mode.")},"validateAmsEnvironmentContext");o(W7,"getAutoTag");o(ph,"parseArray");o(q7,"dCellStyle");Zl=o(function(e,r){var n,i,a=e.body.length,s=e.hLinesBeforeRow,l=0,u=new Array(a),h=[],f=Math.max(r.fontMetrics().arrayRuleWidth,r.minRuleThickness),d=1/r.fontMetrics().ptPerEm,p=5*d;if(e.colSeparationType&&e.colSeparationType==="small"){var m=r.havingStyle(tr.SCRIPT).sizeMultiplier;p=.2778*(m/r.sizeMultiplier)}var g=e.colSeparationType==="CD"?ti({number:3,unit:"ex"},r):12*d,y=3*d,v=e.arraystretch*g,x=.7*v,b=.3*v,w=0;function C(ae){for(var Oe=0;Oe0&&(w+=.25),h.push({pos:w,isDashed:ae[Oe]})}for(o(C,"setHLinePos"),C(s[0]),n=0;n0&&(D+=b,Aae))for(n=0;n=l)){var le=void 0;(i>0||e.hskipBeforeAndAfter)&&(le=Jt.deflt(H.pregap,p),le!==0&&(O=Be.makeSpan(["arraycolsep"],[]),O.style.width=kt(le),R.push(O)));var he=[];for(n=0;n0){for(var J=Be.makeLineSpan("hline",r,f),se=Be.makeLineSpan("hdashline",r,f),ue=[{type:"elem",elem:u,shift:0}];h.length>0;){var Z=h.pop(),Se=Z.pos-k;Z.isDashed?ue.push({type:"elem",elem:se,shift:Se}):ue.push({type:"elem",elem:J,shift:Se})}u=Be.makeVList({positionType:"individualShift",children:ue},r)}if(B.length===0)return Be.makeSpan(["mord"],[u],r);var ce=Be.makeVList({positionType:"individualShift",children:B},r);return ce=Be.makeSpan(["tag"],[ce],r),Be.makeFragment([u,ce])},"htmlBuilder"),_4e={c:"center ",l:"left ",r:"right "},Jl=o(function(e,r){for(var n=[],i=new dt.MathNode("mtd",[],["mtr-glue"]),a=new dt.MathNode("mtd",[],["mml-eqn-num"]),s=0;s0){var g=e.cols,y="",v=!1,x=0,b=g.length;g[0].type==="separator"&&(p+="top ",x=1),g[g.length-1].type==="separator"&&(p+="bottom ",b-=1);for(var w=x;w0?"left ":"",p+=S[S.length-1].length>0?"right ":"";for(var _=1;_-1?"alignat":"align",a=e.envName==="split",s=ph(e.parser,{cols:n,addJot:!0,autoTag:a?void 0:W7(e.envName),emptySingleRow:!0,colSeparationType:i,maxNumCols:a?2:void 0,leqno:e.parser.settings.leqno},"display"),l,u=0,h={type:"ordgroup",mode:e.mode,body:[]};if(r[0]&&r[0].type==="ordgroup"){for(var f="",d=0;d0&&m&&(v=1),n[g]={type:"align",align:y,pregap:v,postgap:0}}return s.colSeparationType=m?"align":"alignat",s},"alignedHandler");Ql({type:"array",names:["array","darray"],props:{numArgs:1},handler(t,e){var r=w3(e[0]),n=r?[e[0]]:xr(e[0],"ordgroup").body,i=n.map(function(s){var l=z7(s),u=l.text;if("lcr".indexOf(u)!==-1)return{type:"align",align:u};if(u==="|")return{type:"separator",separator:"|"};if(u===":")return{type:"separator",separator:":"};throw new gt("Unknown column alignment: "+u,s)}),a={cols:i,hskipBeforeAndAfter:!0,maxNumCols:i.length};return ph(t.parser,a,q7(t.envName))},htmlBuilder:Zl,mathmlBuilder:Jl});Ql({type:"array",names:["matrix","pmatrix","bmatrix","Bmatrix","vmatrix","Vmatrix","matrix*","pmatrix*","bmatrix*","Bmatrix*","vmatrix*","Vmatrix*"],props:{numArgs:0},handler(t){var e={matrix:null,pmatrix:["(",")"],bmatrix:["[","]"],Bmatrix:["\\{","\\}"],vmatrix:["|","|"],Vmatrix:["\\Vert","\\Vert"]}[t.envName.replace("*","")],r="c",n={hskipBeforeAndAfter:!1,cols:[{type:"align",align:r}]};if(t.envName.charAt(t.envName.length-1)==="*"){var i=t.parser;if(i.consumeSpaces(),i.fetch().text==="["){if(i.consume(),i.consumeSpaces(),r=i.fetch().text,"lcr".indexOf(r)===-1)throw new gt("Expected l or c or r",i.nextToken);i.consume(),i.consumeSpaces(),i.expect("]"),i.consume(),n.cols=[{type:"align",align:r}]}}var a=ph(t.parser,n,q7(t.envName)),s=Math.max(0,...a.body.map(l=>l.length));return a.cols=new Array(s).fill({type:"align",align:r}),e?{type:"leftright",mode:t.mode,body:[a],left:e[0],right:e[1],rightColor:void 0}:a},htmlBuilder:Zl,mathmlBuilder:Jl});Ql({type:"array",names:["smallmatrix"],props:{numArgs:0},handler(t){var e={arraystretch:.5},r=ph(t.parser,e,"script");return r.colSeparationType="small",r},htmlBuilder:Zl,mathmlBuilder:Jl});Ql({type:"array",names:["subarray"],props:{numArgs:1},handler(t,e){var r=w3(e[0]),n=r?[e[0]]:xr(e[0],"ordgroup").body,i=n.map(function(s){var l=z7(s),u=l.text;if("lc".indexOf(u)!==-1)return{type:"align",align:u};throw new gt("Unknown column alignment: "+u,s)});if(i.length>1)throw new gt("{subarray} can contain only one column");var a={cols:i,hskipBeforeAndAfter:!1,arraystretch:.5};if(a=ph(t.parser,a,"script"),a.body.length>0&&a.body[0].length>1)throw new gt("{subarray} can contain only one column");return a},htmlBuilder:Zl,mathmlBuilder:Jl});Ql({type:"array",names:["cases","dcases","rcases","drcases"],props:{numArgs:0},handler(t){var e={arraystretch:1.2,cols:[{type:"align",align:"l",pregap:0,postgap:1},{type:"align",align:"l",pregap:0,postgap:0}]},r=ph(t.parser,e,q7(t.envName));return{type:"leftright",mode:t.mode,body:[r],left:t.envName.indexOf("r")>-1?".":"\\{",right:t.envName.indexOf("r")>-1?"\\}":".",rightColor:void 0}},htmlBuilder:Zl,mathmlBuilder:Jl});Ql({type:"array",names:["align","align*","aligned","split"],props:{numArgs:0},handler:yG,htmlBuilder:Zl,mathmlBuilder:Jl});Ql({type:"array",names:["gathered","gather","gather*"],props:{numArgs:0},handler(t){Jt.contains(["gather","gather*"],t.envName)&&E3(t);var e={cols:[{type:"align",align:"c"}],addJot:!0,colSeparationType:"gather",autoTag:W7(t.envName),emptySingleRow:!0,leqno:t.parser.settings.leqno};return ph(t.parser,e,"display")},htmlBuilder:Zl,mathmlBuilder:Jl});Ql({type:"array",names:["alignat","alignat*","alignedat"],props:{numArgs:1},handler:yG,htmlBuilder:Zl,mathmlBuilder:Jl});Ql({type:"array",names:["equation","equation*"],props:{numArgs:0},handler(t){E3(t);var e={autoTag:W7(t.envName),emptySingleRow:!0,singleRow:!0,maxNumCols:1,leqno:t.parser.settings.leqno};return ph(t.parser,e,"display")},htmlBuilder:Zl,mathmlBuilder:Jl});Ql({type:"array",names:["CD"],props:{numArgs:0},handler(t){return E3(t),p4e(t.parser)},htmlBuilder:Zl,mathmlBuilder:Jl});fe("\\nonumber","\\gdef\\@eqnsw{0}");fe("\\notag","\\nonumber");Nt({type:"text",names:["\\hline","\\hdashline"],props:{numArgs:0,allowedInText:!0,allowedInMath:!0},handler(t,e){throw new gt(t.funcName+" valid only within array environment")}});Tz=mG;Nt({type:"environment",names:["\\begin","\\end"],props:{numArgs:1,argTypes:["text"]},handler(t,e){var{parser:r,funcName:n}=t,i=e[0];if(i.type!=="ordgroup")throw new gt("Invalid environment name",i);for(var a="",s=0;s{var r=t.font,n=e.withFont(r);return Fr(t.body,n)},"htmlBuilder$5"),xG=o((t,e)=>{var r=t.font,n=e.withFont(r);return yn(t.body,n)},"mathmlBuilder$4"),kz={"\\Bbb":"\\mathbb","\\bold":"\\mathbf","\\frak":"\\mathfrak","\\bm":"\\boldsymbol"};Nt({type:"font",names:["\\mathrm","\\mathit","\\mathbf","\\mathnormal","\\mathbb","\\mathcal","\\mathfrak","\\mathscr","\\mathsf","\\mathtt","\\Bbb","\\bold","\\frak"],props:{numArgs:1,allowedInArgument:!0},handler:o((t,e)=>{var{parser:r,funcName:n}=t,i=g3(e[0]),a=n;return a in kz&&(a=kz[a]),{type:"font",mode:r.mode,font:a.slice(1),body:i}},"handler"),htmlBuilder:vG,mathmlBuilder:xG});Nt({type:"mclass",names:["\\boldsymbol","\\bm"],props:{numArgs:1},handler:o((t,e)=>{var{parser:r}=t,n=e[0],i=Jt.isCharacterBox(n);return{type:"mclass",mode:r.mode,mclass:T3(n),body:[{type:"font",mode:r.mode,font:"boldsymbol",body:n}],isCharacterBox:i}},"handler")});Nt({type:"font",names:["\\rm","\\sf","\\tt","\\bf","\\it","\\cal"],props:{numArgs:0,allowedInText:!0},handler:o((t,e)=>{var{parser:r,funcName:n,breakOnTokenText:i}=t,{mode:a}=r,s=r.parseExpression(!0,i),l="math"+n.slice(1);return{type:"font",mode:a,font:l,body:{type:"ordgroup",mode:r.mode,body:s}}},"handler"),htmlBuilder:vG,mathmlBuilder:xG});bG=o((t,e)=>{var r=e;return t==="display"?r=r.id>=tr.SCRIPT.id?r.text():tr.DISPLAY:t==="text"&&r.size===tr.DISPLAY.size?r=tr.TEXT:t==="script"?r=tr.SCRIPT:t==="scriptscript"&&(r=tr.SCRIPTSCRIPT),r},"adjustStyle"),Y7=o((t,e)=>{var r=bG(t.size,e.style),n=r.fracNum(),i=r.fracDen(),a;a=e.havingStyle(n);var s=Fr(t.numer,a,e);if(t.continued){var l=8.5/e.fontMetrics().ptPerEm,u=3.5/e.fontMetrics().ptPerEm;s.height=s.height0?g=3*p:g=7*p,y=e.fontMetrics().denom1):(d>0?(m=e.fontMetrics().num2,g=p):(m=e.fontMetrics().num3,g=3*p),y=e.fontMetrics().denom2);var v;if(f){var b=e.fontMetrics().axisHeight;m-s.depth-(b+.5*d){var r=new dt.MathNode("mfrac",[yn(t.numer,e),yn(t.denom,e)]);if(!t.hasBarLine)r.setAttribute("linethickness","0px");else if(t.barSize){var n=ti(t.barSize,e);r.setAttribute("linethickness",kt(n))}var i=bG(t.size,e.style);if(i.size!==e.style.size){r=new dt.MathNode("mstyle",[r]);var a=i.size===tr.DISPLAY.size?"true":"false";r.setAttribute("displaystyle",a),r.setAttribute("scriptlevel","0")}if(t.leftDelim!=null||t.rightDelim!=null){var s=[];if(t.leftDelim!=null){var l=new dt.MathNode("mo",[new dt.TextNode(t.leftDelim.replace("\\",""))]);l.setAttribute("fence","true"),s.push(l)}if(s.push(r),t.rightDelim!=null){var u=new dt.MathNode("mo",[new dt.TextNode(t.rightDelim.replace("\\",""))]);u.setAttribute("fence","true"),s.push(u)}return F7(s)}return r},"mathmlBuilder$3");Nt({type:"genfrac",names:["\\dfrac","\\frac","\\tfrac","\\dbinom","\\binom","\\tbinom","\\\\atopfrac","\\\\bracefrac","\\\\brackfrac"],props:{numArgs:2,allowedInArgument:!0},handler:o((t,e)=>{var{parser:r,funcName:n}=t,i=e[0],a=e[1],s,l=null,u=null,h="auto";switch(n){case"\\dfrac":case"\\frac":case"\\tfrac":s=!0;break;case"\\\\atopfrac":s=!1;break;case"\\dbinom":case"\\binom":case"\\tbinom":s=!1,l="(",u=")";break;case"\\\\bracefrac":s=!1,l="\\{",u="\\}";break;case"\\\\brackfrac":s=!1,l="[",u="]";break;default:throw new Error("Unrecognized genfrac command")}switch(n){case"\\dfrac":case"\\dbinom":h="display";break;case"\\tfrac":case"\\tbinom":h="text";break}return{type:"genfrac",mode:r.mode,continued:!1,numer:i,denom:a,hasBarLine:s,leftDelim:l,rightDelim:u,size:h,barSize:null}},"handler"),htmlBuilder:Y7,mathmlBuilder:X7});Nt({type:"genfrac",names:["\\cfrac"],props:{numArgs:2},handler:o((t,e)=>{var{parser:r,funcName:n}=t,i=e[0],a=e[1];return{type:"genfrac",mode:r.mode,continued:!0,numer:i,denom:a,hasBarLine:!0,leftDelim:null,rightDelim:null,size:"display",barSize:null}},"handler")});Nt({type:"infix",names:["\\over","\\choose","\\atop","\\brace","\\brack"],props:{numArgs:0,infix:!0},handler(t){var{parser:e,funcName:r,token:n}=t,i;switch(r){case"\\over":i="\\frac";break;case"\\choose":i="\\binom";break;case"\\atop":i="\\\\atopfrac";break;case"\\brace":i="\\\\bracefrac";break;case"\\brack":i="\\\\brackfrac";break;default:throw new Error("Unrecognized infix genfrac command")}return{type:"infix",mode:e.mode,replaceWith:i,token:n}}});Ez=["display","text","script","scriptscript"],Sz=o(function(e){var r=null;return e.length>0&&(r=e,r=r==="."?null:r),r},"delimFromValue");Nt({type:"genfrac",names:["\\genfrac"],props:{numArgs:6,allowedInArgument:!0,argTypes:["math","math","size","text","math","math"]},handler(t,e){var{parser:r}=t,n=e[4],i=e[5],a=g3(e[0]),s=a.type==="atom"&&a.family==="open"?Sz(a.text):null,l=g3(e[1]),u=l.type==="atom"&&l.family==="close"?Sz(l.text):null,h=xr(e[2],"size"),f,d=null;h.isBlank?f=!0:(d=h.value,f=d.number>0);var p="auto",m=e[3];if(m.type==="ordgroup"){if(m.body.length>0){var g=xr(m.body[0],"textord");p=Ez[Number(g.text)]}}else m=xr(m,"textord"),p=Ez[Number(m.text)];return{type:"genfrac",mode:r.mode,numer:n,denom:i,continued:!1,hasBarLine:f,barSize:d,leftDelim:s,rightDelim:u,size:p}},htmlBuilder:Y7,mathmlBuilder:X7});Nt({type:"infix",names:["\\above"],props:{numArgs:1,argTypes:["size"],infix:!0},handler(t,e){var{parser:r,funcName:n,token:i}=t;return{type:"infix",mode:r.mode,replaceWith:"\\\\abovefrac",size:xr(e[0],"size").value,token:i}}});Nt({type:"genfrac",names:["\\\\abovefrac"],props:{numArgs:3,argTypes:["math","size","math"]},handler:o((t,e)=>{var{parser:r,funcName:n}=t,i=e[0],a=obe(xr(e[1],"infix").size),s=e[2],l=a.number>0;return{type:"genfrac",mode:r.mode,numer:i,denom:s,continued:!1,hasBarLine:l,barSize:a,leftDelim:null,rightDelim:null,size:"auto"}},"handler"),htmlBuilder:Y7,mathmlBuilder:X7});wG=o((t,e)=>{var r=e.style,n,i;t.type==="supsub"?(n=t.sup?Fr(t.sup,e.havingStyle(r.sup()),e):Fr(t.sub,e.havingStyle(r.sub()),e),i=xr(t.base,"horizBrace")):i=xr(t,"horizBrace");var a=Fr(i.base,e.havingBaseStyle(tr.DISPLAY)),s=cu.svgSpan(i,e),l;if(i.isOver?(l=Be.makeVList({positionType:"firstBaseline",children:[{type:"elem",elem:a},{type:"kern",size:.1},{type:"elem",elem:s}]},e),l.children[0].children[0].children[1].classes.push("svg-align")):(l=Be.makeVList({positionType:"bottom",positionData:a.depth+.1+s.height,children:[{type:"elem",elem:s},{type:"kern",size:.1},{type:"elem",elem:a}]},e),l.children[0].children[0].children[0].classes.push("svg-align")),n){var u=Be.makeSpan(["mord",i.isOver?"mover":"munder"],[l],e);i.isOver?l=Be.makeVList({positionType:"firstBaseline",children:[{type:"elem",elem:u},{type:"kern",size:.2},{type:"elem",elem:n}]},e):l=Be.makeVList({positionType:"bottom",positionData:u.depth+.2+n.height+n.depth,children:[{type:"elem",elem:n},{type:"kern",size:.2},{type:"elem",elem:u}]},e)}return Be.makeSpan(["mord",i.isOver?"mover":"munder"],[l],e)},"htmlBuilder$3"),D4e=o((t,e)=>{var r=cu.mathMLnode(t.label);return new dt.MathNode(t.isOver?"mover":"munder",[yn(t.base,e),r])},"mathmlBuilder$2");Nt({type:"horizBrace",names:["\\overbrace","\\underbrace"],props:{numArgs:1},handler(t,e){var{parser:r,funcName:n}=t;return{type:"horizBrace",mode:r.mode,label:n,isOver:/^\\over/.test(n),base:e[0]}},htmlBuilder:wG,mathmlBuilder:D4e});Nt({type:"href",names:["\\href"],props:{numArgs:2,argTypes:["url","original"],allowedInText:!0},handler:o((t,e)=>{var{parser:r}=t,n=e[1],i=xr(e[0],"url").url;return r.settings.isTrusted({command:"\\href",url:i})?{type:"href",mode:r.mode,href:i,body:di(n)}:r.formatUnsupportedCmd("\\href")},"handler"),htmlBuilder:o((t,e)=>{var r=Pi(t.body,e,!1);return Be.makeAnchor(t.href,[],r,e)},"htmlBuilder"),mathmlBuilder:o((t,e)=>{var r=dh(t.body,e);return r instanceof ws||(r=new ws("mrow",[r])),r.setAttribute("href",t.href),r},"mathmlBuilder")});Nt({type:"href",names:["\\url"],props:{numArgs:1,argTypes:["url"],allowedInText:!0},handler:o((t,e)=>{var{parser:r}=t,n=xr(e[0],"url").url;if(!r.settings.isTrusted({command:"\\url",url:n}))return r.formatUnsupportedCmd("\\url");for(var i=[],a=0;a{var{parser:r,funcName:n,token:i}=t,a=xr(e[0],"raw").string,s=e[1];r.settings.strict&&r.settings.reportNonstrict("htmlExtension","HTML extension is disabled on strict mode");var l,u={};switch(n){case"\\htmlClass":u.class=a,l={command:"\\htmlClass",class:a};break;case"\\htmlId":u.id=a,l={command:"\\htmlId",id:a};break;case"\\htmlStyle":u.style=a,l={command:"\\htmlStyle",style:a};break;case"\\htmlData":{for(var h=a.split(","),f=0;f{var r=Pi(t.body,e,!1),n=["enclosing"];t.attributes.class&&n.push(...t.attributes.class.trim().split(/\s+/));var i=Be.makeSpan(n,r,e);for(var a in t.attributes)a!=="class"&&t.attributes.hasOwnProperty(a)&&i.setAttribute(a,t.attributes[a]);return i},"htmlBuilder"),mathmlBuilder:o((t,e)=>dh(t.body,e),"mathmlBuilder")});Nt({type:"htmlmathml",names:["\\html@mathml"],props:{numArgs:2,allowedInText:!0},handler:o((t,e)=>{var{parser:r}=t;return{type:"htmlmathml",mode:r.mode,html:di(e[0]),mathml:di(e[1])}},"handler"),htmlBuilder:o((t,e)=>{var r=Pi(t.html,e,!1);return Be.makeFragment(r)},"htmlBuilder"),mathmlBuilder:o((t,e)=>dh(t.mathml,e),"mathmlBuilder")});x7=o(function(e){if(/^[-+]? *(\d+(\.\d*)?|\.\d+)$/.test(e))return{number:+e,unit:"bp"};var r=/([-+]?) *(\d+(?:\.\d*)?|\.\d+) *([a-z]{2})/.exec(e);if(!r)throw new gt("Invalid size: '"+e+"' in \\includegraphics");var n={number:+(r[1]+r[2]),unit:r[3]};if(!zz(n))throw new gt("Invalid unit: '"+n.unit+"' in \\includegraphics.");return n},"sizeData");Nt({type:"includegraphics",names:["\\includegraphics"],props:{numArgs:1,numOptionalArgs:1,argTypes:["raw","url"],allowedInText:!1},handler:o((t,e,r)=>{var{parser:n}=t,i={number:0,unit:"em"},a={number:.9,unit:"em"},s={number:0,unit:"em"},l="";if(r[0])for(var u=xr(r[0],"raw").string,h=u.split(","),f=0;f{var r=ti(t.height,e),n=0;t.totalheight.number>0&&(n=ti(t.totalheight,e)-r);var i=0;t.width.number>0&&(i=ti(t.width,e));var a={height:kt(r+n)};i>0&&(a.width=kt(i)),n>0&&(a.verticalAlign=kt(-n));var s=new S7(t.src,t.alt,a);return s.height=r,s.depth=n,s},"htmlBuilder"),mathmlBuilder:o((t,e)=>{var r=new dt.MathNode("mglyph",[]);r.setAttribute("alt",t.alt);var n=ti(t.height,e),i=0;if(t.totalheight.number>0&&(i=ti(t.totalheight,e)-n,r.setAttribute("valign",kt(-i))),r.setAttribute("height",kt(n+i)),t.width.number>0){var a=ti(t.width,e);r.setAttribute("width",kt(a))}return r.setAttribute("src",t.src),r},"mathmlBuilder")});Nt({type:"kern",names:["\\kern","\\mkern","\\hskip","\\mskip"],props:{numArgs:1,argTypes:["size"],primitive:!0,allowedInText:!0},handler(t,e){var{parser:r,funcName:n}=t,i=xr(e[0],"size");if(r.settings.strict){var a=n[1]==="m",s=i.value.unit==="mu";a?(s||r.settings.reportNonstrict("mathVsTextUnits","LaTeX's "+n+" supports only mu units, "+("not "+i.value.unit+" units")),r.mode!=="math"&&r.settings.reportNonstrict("mathVsTextUnits","LaTeX's "+n+" works only in math mode")):s&&r.settings.reportNonstrict("mathVsTextUnits","LaTeX's "+n+" doesn't support mu units")}return{type:"kern",mode:r.mode,dimension:i.value}},htmlBuilder(t,e){return Be.makeGlue(t.dimension,e)},mathmlBuilder(t,e){var r=ti(t.dimension,e);return new dt.SpaceNode(r)}});Nt({type:"lap",names:["\\mathllap","\\mathrlap","\\mathclap"],props:{numArgs:1,allowedInText:!0},handler:o((t,e)=>{var{parser:r,funcName:n}=t,i=e[0];return{type:"lap",mode:r.mode,alignment:n.slice(5),body:i}},"handler"),htmlBuilder:o((t,e)=>{var r;t.alignment==="clap"?(r=Be.makeSpan([],[Fr(t.body,e)]),r=Be.makeSpan(["inner"],[r],e)):r=Be.makeSpan(["inner"],[Fr(t.body,e)]);var n=Be.makeSpan(["fix"],[]),i=Be.makeSpan([t.alignment],[r,n],e),a=Be.makeSpan(["strut"]);return a.style.height=kt(i.height+i.depth),i.depth&&(a.style.verticalAlign=kt(-i.depth)),i.children.unshift(a),i=Be.makeSpan(["thinbox"],[i],e),Be.makeSpan(["mord","vbox"],[i],e)},"htmlBuilder"),mathmlBuilder:o((t,e)=>{var r=new dt.MathNode("mpadded",[yn(t.body,e)]);if(t.alignment!=="rlap"){var n=t.alignment==="llap"?"-1":"-0.5";r.setAttribute("lspace",n+"width")}return r.setAttribute("width","0px"),r},"mathmlBuilder")});Nt({type:"styling",names:["\\(","$"],props:{numArgs:0,allowedInText:!0,allowedInMath:!1},handler(t,e){var{funcName:r,parser:n}=t,i=n.mode;n.switchMode("math");var a=r==="\\("?"\\)":"$",s=n.parseExpression(!1,a);return n.expect(a),n.switchMode(i),{type:"styling",mode:n.mode,style:"text",body:s}}});Nt({type:"text",names:["\\)","\\]"],props:{numArgs:0,allowedInText:!0,allowedInMath:!1},handler(t,e){throw new gt("Mismatched "+t.funcName)}});Cz=o((t,e)=>{switch(e.style.size){case tr.DISPLAY.size:return t.display;case tr.TEXT.size:return t.text;case tr.SCRIPT.size:return t.script;case tr.SCRIPTSCRIPT.size:return t.scriptscript;default:return t.text}},"chooseMathStyle");Nt({type:"mathchoice",names:["\\mathchoice"],props:{numArgs:4,primitive:!0},handler:o((t,e)=>{var{parser:r}=t;return{type:"mathchoice",mode:r.mode,display:di(e[0]),text:di(e[1]),script:di(e[2]),scriptscript:di(e[3])}},"handler"),htmlBuilder:o((t,e)=>{var r=Cz(t,e),n=Pi(r,e,!1);return Be.makeFragment(n)},"htmlBuilder"),mathmlBuilder:o((t,e)=>{var r=Cz(t,e);return dh(r,e)},"mathmlBuilder")});TG=o((t,e,r,n,i,a,s)=>{t=Be.makeSpan([],[t]);var l=r&&Jt.isCharacterBox(r),u,h;if(e){var f=Fr(e,n.havingStyle(i.sup()),n);h={elem:f,kern:Math.max(n.fontMetrics().bigOpSpacing1,n.fontMetrics().bigOpSpacing3-f.depth)}}if(r){var d=Fr(r,n.havingStyle(i.sub()),n);u={elem:d,kern:Math.max(n.fontMetrics().bigOpSpacing2,n.fontMetrics().bigOpSpacing4-d.height)}}var p;if(h&&u){var m=n.fontMetrics().bigOpSpacing5+u.elem.height+u.elem.depth+u.kern+t.depth+s;p=Be.makeVList({positionType:"bottom",positionData:m,children:[{type:"kern",size:n.fontMetrics().bigOpSpacing5},{type:"elem",elem:u.elem,marginLeft:kt(-a)},{type:"kern",size:u.kern},{type:"elem",elem:t},{type:"kern",size:h.kern},{type:"elem",elem:h.elem,marginLeft:kt(a)},{type:"kern",size:n.fontMetrics().bigOpSpacing5}]},n)}else if(u){var g=t.height-s;p=Be.makeVList({positionType:"top",positionData:g,children:[{type:"kern",size:n.fontMetrics().bigOpSpacing5},{type:"elem",elem:u.elem,marginLeft:kt(-a)},{type:"kern",size:u.kern},{type:"elem",elem:t}]},n)}else if(h){var y=t.depth+s;p=Be.makeVList({positionType:"bottom",positionData:y,children:[{type:"elem",elem:t},{type:"kern",size:h.kern},{type:"elem",elem:h.elem,marginLeft:kt(a)},{type:"kern",size:n.fontMetrics().bigOpSpacing5}]},n)}else return t;var v=[p];if(u&&a!==0&&!l){var x=Be.makeSpan(["mspace"],[],n);x.style.marginRight=kt(a),v.unshift(x)}return Be.makeSpan(["mop","op-limits"],v,n)},"assembleSupSub"),kG=["\\smallint"],m0=o((t,e)=>{var r,n,i=!1,a;t.type==="supsub"?(r=t.sup,n=t.sub,a=xr(t.base,"op"),i=!0):a=xr(t,"op");var s=e.style,l=!1;s.size===tr.DISPLAY.size&&a.symbol&&!Jt.contains(kG,a.name)&&(l=!0);var u;if(a.symbol){var h=l?"Size2-Regular":"Size1-Regular",f="";if((a.name==="\\oiint"||a.name==="\\oiiint")&&(f=a.name.slice(1),a.name=f==="oiint"?"\\iint":"\\iiint"),u=Be.makeSymbol(a.name,h,"math",e,["mop","op-symbol",l?"large-op":"small-op"]),f.length>0){var d=u.italic,p=Be.staticSvg(f+"Size"+(l?"2":"1"),e);u=Be.makeVList({positionType:"individualShift",children:[{type:"elem",elem:u,shift:0},{type:"elem",elem:p,shift:l?.08:0}]},e),a.name="\\"+f,u.classes.unshift("mop"),u.italic=d}}else if(a.body){var m=Pi(a.body,e,!0);m.length===1&&m[0]instanceof Ts?(u=m[0],u.classes[0]="mop"):u=Be.makeSpan(["mop"],m,e)}else{for(var g=[],y=1;y{var r;if(t.symbol)r=new ws("mo",[Co(t.name,t.mode)]),Jt.contains(kG,t.name)&&r.setAttribute("largeop","false");else if(t.body)r=new ws("mo",ks(t.body,e));else{r=new ws("mi",[new Jf(t.name.slice(1))]);var n=new ws("mo",[Co("\u2061","text")]);t.parentIsSupSub?r=new ws("mrow",[r,n]):r=Qz([r,n])}return r},"mathmlBuilder$1"),L4e={"\u220F":"\\prod","\u2210":"\\coprod","\u2211":"\\sum","\u22C0":"\\bigwedge","\u22C1":"\\bigvee","\u22C2":"\\bigcap","\u22C3":"\\bigcup","\u2A00":"\\bigodot","\u2A01":"\\bigoplus","\u2A02":"\\bigotimes","\u2A04":"\\biguplus","\u2A06":"\\bigsqcup"};Nt({type:"op",names:["\\coprod","\\bigvee","\\bigwedge","\\biguplus","\\bigcap","\\bigcup","\\intop","\\prod","\\sum","\\bigotimes","\\bigoplus","\\bigodot","\\bigsqcup","\\smallint","\u220F","\u2210","\u2211","\u22C0","\u22C1","\u22C2","\u22C3","\u2A00","\u2A01","\u2A02","\u2A04","\u2A06"],props:{numArgs:0},handler:o((t,e)=>{var{parser:r,funcName:n}=t,i=n;return i.length===1&&(i=L4e[i]),{type:"op",mode:r.mode,limits:!0,parentIsSupSub:!1,symbol:!0,name:i}},"handler"),htmlBuilder:m0,mathmlBuilder:Wy});Nt({type:"op",names:["\\mathop"],props:{numArgs:1,primitive:!0},handler:o((t,e)=>{var{parser:r}=t,n=e[0];return{type:"op",mode:r.mode,limits:!1,parentIsSupSub:!1,symbol:!1,body:di(n)}},"handler"),htmlBuilder:m0,mathmlBuilder:Wy});R4e={"\u222B":"\\int","\u222C":"\\iint","\u222D":"\\iiint","\u222E":"\\oint","\u222F":"\\oiint","\u2230":"\\oiiint"};Nt({type:"op",names:["\\arcsin","\\arccos","\\arctan","\\arctg","\\arcctg","\\arg","\\ch","\\cos","\\cosec","\\cosh","\\cot","\\cotg","\\coth","\\csc","\\ctg","\\cth","\\deg","\\dim","\\exp","\\hom","\\ker","\\lg","\\ln","\\log","\\sec","\\sin","\\sinh","\\sh","\\tan","\\tanh","\\tg","\\th"],props:{numArgs:0},handler(t){var{parser:e,funcName:r}=t;return{type:"op",mode:e.mode,limits:!1,parentIsSupSub:!1,symbol:!1,name:r}},htmlBuilder:m0,mathmlBuilder:Wy});Nt({type:"op",names:["\\det","\\gcd","\\inf","\\lim","\\max","\\min","\\Pr","\\sup"],props:{numArgs:0},handler(t){var{parser:e,funcName:r}=t;return{type:"op",mode:e.mode,limits:!0,parentIsSupSub:!1,symbol:!1,name:r}},htmlBuilder:m0,mathmlBuilder:Wy});Nt({type:"op",names:["\\int","\\iint","\\iiint","\\oint","\\oiint","\\oiiint","\u222B","\u222C","\u222D","\u222E","\u222F","\u2230"],props:{numArgs:0},handler(t){var{parser:e,funcName:r}=t,n=r;return n.length===1&&(n=R4e[n]),{type:"op",mode:e.mode,limits:!1,parentIsSupSub:!1,symbol:!0,name:n}},htmlBuilder:m0,mathmlBuilder:Wy});EG=o((t,e)=>{var r,n,i=!1,a;t.type==="supsub"?(r=t.sup,n=t.sub,a=xr(t.base,"operatorname"),i=!0):a=xr(t,"operatorname");var s;if(a.body.length>0){for(var l=a.body.map(d=>{var p=d.text;return typeof p=="string"?{type:"textord",mode:d.mode,text:p}:d}),u=Pi(l,e.withFont("mathrm"),!0),h=0;h{for(var r=ks(t.body,e.withFont("mathrm")),n=!0,i=0;if.toText()).join("");r=[new dt.TextNode(l)]}var u=new dt.MathNode("mi",r);u.setAttribute("mathvariant","normal");var h=new dt.MathNode("mo",[Co("\u2061","text")]);return t.parentIsSupSub?new dt.MathNode("mrow",[u,h]):dt.newDocumentFragment([u,h])},"mathmlBuilder");Nt({type:"operatorname",names:["\\operatorname@","\\operatornamewithlimits"],props:{numArgs:1},handler:o((t,e)=>{var{parser:r,funcName:n}=t,i=e[0];return{type:"operatorname",mode:r.mode,body:di(i),alwaysHandleSupSub:n==="\\operatornamewithlimits",limits:!1,parentIsSupSub:!1}},"handler"),htmlBuilder:EG,mathmlBuilder:N4e});fe("\\operatorname","\\@ifstar\\operatornamewithlimits\\operatorname@");rd({type:"ordgroup",htmlBuilder(t,e){return t.semisimple?Be.makeFragment(Pi(t.body,e,!1)):Be.makeSpan(["mord"],Pi(t.body,e,!0),e)},mathmlBuilder(t,e){return dh(t.body,e,!0)}});Nt({type:"overline",names:["\\overline"],props:{numArgs:1},handler(t,e){var{parser:r}=t,n=e[0];return{type:"overline",mode:r.mode,body:n}},htmlBuilder(t,e){var r=Fr(t.body,e.havingCrampedStyle()),n=Be.makeLineSpan("overline-line",e),i=e.fontMetrics().defaultRuleThickness,a=Be.makeVList({positionType:"firstBaseline",children:[{type:"elem",elem:r},{type:"kern",size:3*i},{type:"elem",elem:n},{type:"kern",size:i}]},e);return Be.makeSpan(["mord","overline"],[a],e)},mathmlBuilder(t,e){var r=new dt.MathNode("mo",[new dt.TextNode("\u203E")]);r.setAttribute("stretchy","true");var n=new dt.MathNode("mover",[yn(t.body,e),r]);return n.setAttribute("accent","true"),n}});Nt({type:"phantom",names:["\\phantom"],props:{numArgs:1,allowedInText:!0},handler:o((t,e)=>{var{parser:r}=t,n=e[0];return{type:"phantom",mode:r.mode,body:di(n)}},"handler"),htmlBuilder:o((t,e)=>{var r=Pi(t.body,e.withPhantom(),!1);return Be.makeFragment(r)},"htmlBuilder"),mathmlBuilder:o((t,e)=>{var r=ks(t.body,e);return new dt.MathNode("mphantom",r)},"mathmlBuilder")});Nt({type:"hphantom",names:["\\hphantom"],props:{numArgs:1,allowedInText:!0},handler:o((t,e)=>{var{parser:r}=t,n=e[0];return{type:"hphantom",mode:r.mode,body:n}},"handler"),htmlBuilder:o((t,e)=>{var r=Be.makeSpan([],[Fr(t.body,e.withPhantom())]);if(r.height=0,r.depth=0,r.children)for(var n=0;n{var r=ks(di(t.body),e),n=new dt.MathNode("mphantom",r),i=new dt.MathNode("mpadded",[n]);return i.setAttribute("height","0px"),i.setAttribute("depth","0px"),i},"mathmlBuilder")});Nt({type:"vphantom",names:["\\vphantom"],props:{numArgs:1,allowedInText:!0},handler:o((t,e)=>{var{parser:r}=t,n=e[0];return{type:"vphantom",mode:r.mode,body:n}},"handler"),htmlBuilder:o((t,e)=>{var r=Be.makeSpan(["inner"],[Fr(t.body,e.withPhantom())]),n=Be.makeSpan(["fix"],[]);return Be.makeSpan(["mord","rlap"],[r,n],e)},"htmlBuilder"),mathmlBuilder:o((t,e)=>{var r=ks(di(t.body),e),n=new dt.MathNode("mphantom",r),i=new dt.MathNode("mpadded",[n]);return i.setAttribute("width","0px"),i},"mathmlBuilder")});Nt({type:"raisebox",names:["\\raisebox"],props:{numArgs:2,argTypes:["size","hbox"],allowedInText:!0},handler(t,e){var{parser:r}=t,n=xr(e[0],"size").value,i=e[1];return{type:"raisebox",mode:r.mode,dy:n,body:i}},htmlBuilder(t,e){var r=Fr(t.body,e),n=ti(t.dy,e);return Be.makeVList({positionType:"shift",positionData:-n,children:[{type:"elem",elem:r}]},e)},mathmlBuilder(t,e){var r=new dt.MathNode("mpadded",[yn(t.body,e)]),n=t.dy.number+t.dy.unit;return r.setAttribute("voffset",n),r}});Nt({type:"internal",names:["\\relax"],props:{numArgs:0,allowedInText:!0},handler(t){var{parser:e}=t;return{type:"internal",mode:e.mode}}});Nt({type:"rule",names:["\\rule"],props:{numArgs:2,numOptionalArgs:1,argTypes:["size","size","size"]},handler(t,e,r){var{parser:n}=t,i=r[0],a=xr(e[0],"size"),s=xr(e[1],"size");return{type:"rule",mode:n.mode,shift:i&&xr(i,"size").value,width:a.value,height:s.value}},htmlBuilder(t,e){var r=Be.makeSpan(["mord","rule"],[],e),n=ti(t.width,e),i=ti(t.height,e),a=t.shift?ti(t.shift,e):0;return r.style.borderRightWidth=kt(n),r.style.borderTopWidth=kt(i),r.style.bottom=kt(a),r.width=n,r.height=i+a,r.depth=-a,r.maxFontSize=i*1.125*e.sizeMultiplier,r},mathmlBuilder(t,e){var r=ti(t.width,e),n=ti(t.height,e),i=t.shift?ti(t.shift,e):0,a=e.color&&e.getColor()||"black",s=new dt.MathNode("mspace");s.setAttribute("mathbackground",a),s.setAttribute("width",kt(r)),s.setAttribute("height",kt(n));var l=new dt.MathNode("mpadded",[s]);return i>=0?l.setAttribute("height",kt(i)):(l.setAttribute("height",kt(i)),l.setAttribute("depth",kt(-i))),l.setAttribute("voffset",kt(i)),l}});o(SG,"sizingGroup");Az=["\\tiny","\\sixptsize","\\scriptsize","\\footnotesize","\\small","\\normalsize","\\large","\\Large","\\LARGE","\\huge","\\Huge"],M4e=o((t,e)=>{var r=e.havingSize(t.size);return SG(t.body,r,e)},"htmlBuilder");Nt({type:"sizing",names:Az,props:{numArgs:0,allowedInText:!0},handler:o((t,e)=>{var{breakOnTokenText:r,funcName:n,parser:i}=t,a=i.parseExpression(!1,r);return{type:"sizing",mode:i.mode,size:Az.indexOf(n)+1,body:a}},"handler"),htmlBuilder:M4e,mathmlBuilder:o((t,e)=>{var r=e.havingSize(t.size),n=ks(t.body,r),i=new dt.MathNode("mstyle",n);return i.setAttribute("mathsize",kt(r.sizeMultiplier)),i},"mathmlBuilder")});Nt({type:"smash",names:["\\smash"],props:{numArgs:1,numOptionalArgs:1,allowedInText:!0},handler:o((t,e,r)=>{var{parser:n}=t,i=!1,a=!1,s=r[0]&&xr(r[0],"ordgroup");if(s)for(var l="",u=0;u{var r=Be.makeSpan([],[Fr(t.body,e)]);if(!t.smashHeight&&!t.smashDepth)return r;if(t.smashHeight&&(r.height=0,r.children))for(var n=0;n{var r=new dt.MathNode("mpadded",[yn(t.body,e)]);return t.smashHeight&&r.setAttribute("height","0px"),t.smashDepth&&r.setAttribute("depth","0px"),r},"mathmlBuilder")});Nt({type:"sqrt",names:["\\sqrt"],props:{numArgs:1,numOptionalArgs:1},handler(t,e,r){var{parser:n}=t,i=r[0],a=e[0];return{type:"sqrt",mode:n.mode,body:a,index:i}},htmlBuilder(t,e){var r=Fr(t.body,e.havingCrampedStyle());r.height===0&&(r.height=e.fontMetrics().xHeight),r=Be.wrapFragment(r,e);var n=e.fontMetrics(),i=n.defaultRuleThickness,a=i;e.style.idr.height+r.depth+s&&(s=(s+d-r.height-r.depth)/2);var p=u.height-r.height-s-h;r.style.paddingLeft=kt(f);var m=Be.makeVList({positionType:"firstBaseline",children:[{type:"elem",elem:r,wrapperClasses:["svg-align"]},{type:"kern",size:-(r.height+p)},{type:"elem",elem:u},{type:"kern",size:h}]},e);if(t.index){var g=e.havingStyle(tr.SCRIPTSCRIPT),y=Fr(t.index,g,e),v=.6*(m.height-m.depth),x=Be.makeVList({positionType:"shift",positionData:-v,children:[{type:"elem",elem:y}]},e),b=Be.makeSpan(["root"],[x]);return Be.makeSpan(["mord","sqrt"],[b,m],e)}else return Be.makeSpan(["mord","sqrt"],[m],e)},mathmlBuilder(t,e){var{body:r,index:n}=t;return n?new dt.MathNode("mroot",[yn(r,e),yn(n,e)]):new dt.MathNode("msqrt",[yn(r,e)])}});_z={display:tr.DISPLAY,text:tr.TEXT,script:tr.SCRIPT,scriptscript:tr.SCRIPTSCRIPT};Nt({type:"styling",names:["\\displaystyle","\\textstyle","\\scriptstyle","\\scriptscriptstyle"],props:{numArgs:0,allowedInText:!0,primitive:!0},handler(t,e){var{breakOnTokenText:r,funcName:n,parser:i}=t,a=i.parseExpression(!0,r),s=n.slice(1,n.length-5);return{type:"styling",mode:i.mode,style:s,body:a}},htmlBuilder(t,e){var r=_z[t.style],n=e.havingStyle(r).withFont("");return SG(t.body,n,e)},mathmlBuilder(t,e){var r=_z[t.style],n=e.havingStyle(r),i=ks(t.body,n),a=new dt.MathNode("mstyle",i),s={display:["0","true"],text:["0","false"],script:["1","false"],scriptscript:["2","false"]},l=s[t.style];return a.setAttribute("scriptlevel",l[0]),a.setAttribute("displaystyle",l[1]),a}});I4e=o(function(e,r){var n=e.base;if(n)if(n.type==="op"){var i=n.limits&&(r.style.size===tr.DISPLAY.size||n.alwaysHandleSupSub);return i?m0:null}else if(n.type==="operatorname"){var a=n.alwaysHandleSupSub&&(r.style.size===tr.DISPLAY.size||n.limits);return a?EG:null}else{if(n.type==="accent")return Jt.isCharacterBox(n.base)?G7:null;if(n.type==="horizBrace"){var s=!e.sub;return s===n.isOver?wG:null}else return null}else return null},"htmlBuilderDelegate");rd({type:"supsub",htmlBuilder(t,e){var r=I4e(t,e);if(r)return r(t,e);var{base:n,sup:i,sub:a}=t,s=Fr(n,e),l,u,h=e.fontMetrics(),f=0,d=0,p=n&&Jt.isCharacterBox(n);if(i){var m=e.havingStyle(e.style.sup());l=Fr(i,m,e),p||(f=s.height-m.fontMetrics().supDrop*m.sizeMultiplier/e.sizeMultiplier)}if(a){var g=e.havingStyle(e.style.sub());u=Fr(a,g,e),p||(d=s.depth+g.fontMetrics().subDrop*g.sizeMultiplier/e.sizeMultiplier)}var y;e.style===tr.DISPLAY?y=h.sup1:e.style.cramped?y=h.sup3:y=h.sup2;var v=e.sizeMultiplier,x=kt(.5/h.ptPerEm/v),b=null;if(u){var w=t.base&&t.base.type==="op"&&t.base.name&&(t.base.name==="\\oiint"||t.base.name==="\\oiiint");(s instanceof Ts||w)&&(b=kt(-s.italic))}var C;if(l&&u){f=Math.max(f,y,l.depth+.25*h.xHeight),d=Math.max(d,h.sub2);var T=h.defaultRuleThickness,E=4*T;if(f-l.depth-(u.height-d)0&&(f+=A,d-=A)}var S=[{type:"elem",elem:u,shift:d,marginRight:x,marginLeft:b},{type:"elem",elem:l,shift:-f,marginRight:x}];C=Be.makeVList({positionType:"individualShift",children:S},e)}else if(u){d=Math.max(d,h.sub1,u.height-.8*h.xHeight);var _=[{type:"elem",elem:u,marginLeft:b,marginRight:x}];C=Be.makeVList({positionType:"shift",positionData:d,children:_},e)}else if(l)f=Math.max(f,y,l.depth+.25*h.xHeight),C=Be.makeVList({positionType:"shift",positionData:-f,children:[{type:"elem",elem:l,marginRight:x}]},e);else throw new Error("supsub must have either sup or sub.");var I=A7(s,"right")||"mord";return Be.makeSpan([I],[s,Be.makeSpan(["msupsub"],[C])],e)},mathmlBuilder(t,e){var r=!1,n,i;t.base&&t.base.type==="horizBrace"&&(i=!!t.sup,i===t.base.isOver&&(r=!0,n=t.base.isOver)),t.base&&(t.base.type==="op"||t.base.type==="operatorname")&&(t.base.parentIsSupSub=!0);var a=[yn(t.base,e)];t.sub&&a.push(yn(t.sub,e)),t.sup&&a.push(yn(t.sup,e));var s;if(r)s=n?"mover":"munder";else if(t.sub)if(t.sup){var h=t.base;h&&h.type==="op"&&h.limits&&e.style===tr.DISPLAY||h&&h.type==="operatorname"&&h.alwaysHandleSupSub&&(e.style===tr.DISPLAY||h.limits)?s="munderover":s="msubsup"}else{var u=t.base;u&&u.type==="op"&&u.limits&&(e.style===tr.DISPLAY||u.alwaysHandleSupSub)||u&&u.type==="operatorname"&&u.alwaysHandleSupSub&&(u.limits||e.style===tr.DISPLAY)?s="munder":s="msub"}else{var l=t.base;l&&l.type==="op"&&l.limits&&(e.style===tr.DISPLAY||l.alwaysHandleSupSub)||l&&l.type==="operatorname"&&l.alwaysHandleSupSub&&(l.limits||e.style===tr.DISPLAY)?s="mover":s="msup"}return new dt.MathNode(s,a)}});rd({type:"atom",htmlBuilder(t,e){return Be.mathsym(t.text,t.mode,e,["m"+t.family])},mathmlBuilder(t,e){var r=new dt.MathNode("mo",[Co(t.text,t.mode)]);if(t.family==="bin"){var n=$7(t,e);n==="bold-italic"&&r.setAttribute("mathvariant",n)}else t.family==="punct"?r.setAttribute("separator","true"):(t.family==="open"||t.family==="close")&&r.setAttribute("stretchy","false");return r}});CG={mi:"italic",mn:"normal",mtext:"normal"};rd({type:"mathord",htmlBuilder(t,e){return Be.makeOrd(t,e,"mathord")},mathmlBuilder(t,e){var r=new dt.MathNode("mi",[Co(t.text,t.mode,e)]),n=$7(t,e)||"italic";return n!==CG[r.type]&&r.setAttribute("mathvariant",n),r}});rd({type:"textord",htmlBuilder(t,e){return Be.makeOrd(t,e,"textord")},mathmlBuilder(t,e){var r=Co(t.text,t.mode,e),n=$7(t,e)||"normal",i;return t.mode==="text"?i=new dt.MathNode("mtext",[r]):/[0-9]/.test(t.text)?i=new dt.MathNode("mn",[r]):t.text==="\\prime"?i=new dt.MathNode("mo",[r]):i=new dt.MathNode("mi",[r]),n!==CG[i.type]&&i.setAttribute("mathvariant",n),i}});b7={"\\nobreak":"nobreak","\\allowbreak":"allowbreak"},w7={" ":{},"\\ ":{},"~":{className:"nobreak"},"\\space":{},"\\nobreakspace":{className:"nobreak"}};rd({type:"spacing",htmlBuilder(t,e){if(w7.hasOwnProperty(t.text)){var r=w7[t.text].className||"";if(t.mode==="text"){var n=Be.makeOrd(t,e,"textord");return n.classes.push(r),n}else return Be.makeSpan(["mspace",r],[Be.mathsym(t.text,t.mode,e)],e)}else{if(b7.hasOwnProperty(t.text))return Be.makeSpan(["mspace",b7[t.text]],[],e);throw new gt('Unknown type of space "'+t.text+'"')}},mathmlBuilder(t,e){var r;if(w7.hasOwnProperty(t.text))r=new dt.MathNode("mtext",[new dt.TextNode("\xA0")]);else{if(b7.hasOwnProperty(t.text))return new dt.MathNode("mspace");throw new gt('Unknown type of space "'+t.text+'"')}return r}});Dz=o(()=>{var t=new dt.MathNode("mtd",[]);return t.setAttribute("width","50%"),t},"pad");rd({type:"tag",mathmlBuilder(t,e){var r=new dt.MathNode("mtable",[new dt.MathNode("mtr",[Dz(),new dt.MathNode("mtd",[dh(t.body,e)]),Dz(),new dt.MathNode("mtd",[dh(t.tag,e)])])]);return r.setAttribute("width","100%"),r}});Lz={"\\text":void 0,"\\textrm":"textrm","\\textsf":"textsf","\\texttt":"texttt","\\textnormal":"textrm"},Rz={"\\textbf":"textbf","\\textmd":"textmd"},O4e={"\\textit":"textit","\\textup":"textup"},Nz=o((t,e)=>{var r=t.font;if(r){if(Lz[r])return e.withTextFontFamily(Lz[r]);if(Rz[r])return e.withTextFontWeight(Rz[r]);if(r==="\\emph")return e.fontShape==="textit"?e.withTextFontShape("textup"):e.withTextFontShape("textit")}else return e;return e.withTextFontShape(O4e[r])},"optionsWithFont");Nt({type:"text",names:["\\text","\\textrm","\\textsf","\\texttt","\\textnormal","\\textbf","\\textmd","\\textit","\\textup","\\emph"],props:{numArgs:1,argTypes:["text"],allowedInArgument:!0,allowedInText:!0},handler(t,e){var{parser:r,funcName:n}=t,i=e[0];return{type:"text",mode:r.mode,body:di(i),font:n}},htmlBuilder(t,e){var r=Nz(t,e),n=Pi(t.body,r,!0);return Be.makeSpan(["mord","text"],n,r)},mathmlBuilder(t,e){var r=Nz(t,e);return dh(t.body,r)}});Nt({type:"underline",names:["\\underline"],props:{numArgs:1,allowedInText:!0},handler(t,e){var{parser:r}=t;return{type:"underline",mode:r.mode,body:e[0]}},htmlBuilder(t,e){var r=Fr(t.body,e),n=Be.makeLineSpan("underline-line",e),i=e.fontMetrics().defaultRuleThickness,a=Be.makeVList({positionType:"top",positionData:r.height,children:[{type:"kern",size:i},{type:"elem",elem:n},{type:"kern",size:3*i},{type:"elem",elem:r}]},e);return Be.makeSpan(["mord","underline"],[a],e)},mathmlBuilder(t,e){var r=new dt.MathNode("mo",[new dt.TextNode("\u203E")]);r.setAttribute("stretchy","true");var n=new dt.MathNode("munder",[yn(t.body,e),r]);return n.setAttribute("accentunder","true"),n}});Nt({type:"vcenter",names:["\\vcenter"],props:{numArgs:1,argTypes:["original"],allowedInText:!1},handler(t,e){var{parser:r}=t;return{type:"vcenter",mode:r.mode,body:e[0]}},htmlBuilder(t,e){var r=Fr(t.body,e),n=e.fontMetrics().axisHeight,i=.5*(r.height-n-(r.depth+n));return Be.makeVList({positionType:"shift",positionData:i,children:[{type:"elem",elem:r}]},e)},mathmlBuilder(t,e){return new dt.MathNode("mpadded",[yn(t.body,e)],["vcenter"])}});Nt({type:"verb",names:["\\verb"],props:{numArgs:0,allowedInText:!0},handler(t,e,r){throw new gt("\\verb ended by end of line instead of matching delimiter")},htmlBuilder(t,e){for(var r=Mz(t),n=[],i=e.havingStyle(e.style.text()),a=0;at.body.replace(/ /g,t.star?"\u2423":"\xA0"),"makeVerb"),hh=jz,AG=`[ \r + ]`,P4e="\\\\[a-zA-Z@]+",B4e="\\\\[^\uD800-\uDFFF]",F4e="("+P4e+")"+AG+"*",$4e=`\\\\( +|[ \r ]+ +?)[ \r ]*`,N7="[\u0300-\u036F]",z4e=new RegExp(N7+"+$"),G4e="("+AG+"+)|"+($4e+"|")+"([!-\\[\\]-\u2027\u202A-\uD7FF\uF900-\uFFFF]"+(N7+"*")+"|[\uD800-\uDBFF][\uDC00-\uDFFF]"+(N7+"*")+"|\\\\verb\\*([^]).*?\\4|\\\\verb([^*a-zA-Z]).*?\\5"+("|"+F4e)+("|"+B4e+")"),y3=class{static{o(this,"Lexer")}constructor(e,r){this.input=void 0,this.settings=void 0,this.tokenRegex=void 0,this.catcodes=void 0,this.input=e,this.settings=r,this.tokenRegex=new RegExp(G4e,"g"),this.catcodes={"%":14,"~":13}}setCatcode(e,r){this.catcodes[e]=r}lex(){var e=this.input,r=this.tokenRegex.lastIndex;if(r===e.length)return new So("EOF",new Xs(this,r,r));var n=this.tokenRegex.exec(e);if(n===null||n.index!==r)throw new gt("Unexpected character: '"+e[r]+"'",new So(e[r],new Xs(this,r,r+1)));var i=n[6]||n[3]||(n[2]?"\\ ":" ");if(this.catcodes[i]===14){var a=e.indexOf(` +`,this.tokenRegex.lastIndex);return a===-1?(this.tokenRegex.lastIndex=e.length,this.settings.reportNonstrict("commentAtEnd","% comment has no terminating newline; LaTeX would fail because of commenting the end of math mode (e.g. $)")):this.tokenRegex.lastIndex=a+1,this.lex()}return new So(i,new Xs(this,r,this.tokenRegex.lastIndex))}},M7=class{static{o(this,"Namespace")}constructor(e,r){e===void 0&&(e={}),r===void 0&&(r={}),this.current=void 0,this.builtins=void 0,this.undefStack=void 0,this.current=r,this.builtins=e,this.undefStack=[]}beginGroup(){this.undefStack.push({})}endGroup(){if(this.undefStack.length===0)throw new gt("Unbalanced namespace destruction: attempt to pop global namespace; please report this as a bug");var e=this.undefStack.pop();for(var r in e)e.hasOwnProperty(r)&&(e[r]==null?delete this.current[r]:this.current[r]=e[r])}endGroups(){for(;this.undefStack.length>0;)this.endGroup()}has(e){return this.current.hasOwnProperty(e)||this.builtins.hasOwnProperty(e)}get(e){return this.current.hasOwnProperty(e)?this.current[e]:this.builtins[e]}set(e,r,n){if(n===void 0&&(n=!1),n){for(var i=0;i0&&(this.undefStack[this.undefStack.length-1][e]=r)}else{var a=this.undefStack[this.undefStack.length-1];a&&!a.hasOwnProperty(e)&&(a[e]=this.current[e])}r==null?delete this.current[e]:this.current[e]=r}},V4e=gG;fe("\\noexpand",function(t){var e=t.popToken();return t.isExpandable(e.text)&&(e.noexpand=!0,e.treatAsRelax=!0),{tokens:[e],numArgs:0}});fe("\\expandafter",function(t){var e=t.popToken();return t.expandOnce(!0),{tokens:[e],numArgs:0}});fe("\\@firstoftwo",function(t){var e=t.consumeArgs(2);return{tokens:e[0],numArgs:0}});fe("\\@secondoftwo",function(t){var e=t.consumeArgs(2);return{tokens:e[1],numArgs:0}});fe("\\@ifnextchar",function(t){var e=t.consumeArgs(3);t.consumeSpaces();var r=t.future();return e[0].length===1&&e[0][0].text===r.text?{tokens:e[1],numArgs:0}:{tokens:e[2],numArgs:0}});fe("\\@ifstar","\\@ifnextchar *{\\@firstoftwo{#1}}");fe("\\TextOrMath",function(t){var e=t.consumeArgs(2);return t.mode==="text"?{tokens:e[0],numArgs:0}:{tokens:e[1],numArgs:0}});Iz={0:0,1:1,2:2,3:3,4:4,5:5,6:6,7:7,8:8,9:9,a:10,A:10,b:11,B:11,c:12,C:12,d:13,D:13,e:14,E:14,f:15,F:15};fe("\\char",function(t){var e=t.popToken(),r,n="";if(e.text==="'")r=8,e=t.popToken();else if(e.text==='"')r=16,e=t.popToken();else if(e.text==="`")if(e=t.popToken(),e.text[0]==="\\")n=e.text.charCodeAt(1);else{if(e.text==="EOF")throw new gt("\\char` missing argument");n=e.text.charCodeAt(0)}else r=10;if(r){if(n=Iz[e.text],n==null||n>=r)throw new gt("Invalid base-"+r+" digit "+e.text);for(var i;(i=Iz[t.future().text])!=null&&i{var n=t.consumeArg().tokens;if(n.length!==1)throw new gt("\\newcommand's first argument must be a macro name");var i=n[0].text,a=t.isDefined(i);if(a&&!e)throw new gt("\\newcommand{"+i+"} attempting to redefine "+(i+"; use \\renewcommand"));if(!a&&!r)throw new gt("\\renewcommand{"+i+"} when command "+i+" does not yet exist; use \\newcommand");var s=0;if(n=t.consumeArg().tokens,n.length===1&&n[0].text==="["){for(var l="",u=t.expandNextToken();u.text!=="]"&&u.text!=="EOF";)l+=u.text,u=t.expandNextToken();if(!l.match(/^\s*[0-9]+\s*$/))throw new gt("Invalid number of arguments: "+l);s=parseInt(l),n=t.consumeArg().tokens}return t.macros.set(i,{tokens:n,numArgs:s}),""},"newcommand");fe("\\newcommand",t=>j7(t,!1,!0));fe("\\renewcommand",t=>j7(t,!0,!1));fe("\\providecommand",t=>j7(t,!0,!0));fe("\\message",t=>{var e=t.consumeArgs(1)[0];return console.log(e.reverse().map(r=>r.text).join("")),""});fe("\\errmessage",t=>{var e=t.consumeArgs(1)[0];return console.error(e.reverse().map(r=>r.text).join("")),""});fe("\\show",t=>{var e=t.popToken(),r=e.text;return console.log(e,t.macros.get(r),hh[r],An.math[r],An.text[r]),""});fe("\\bgroup","{");fe("\\egroup","}");fe("~","\\nobreakspace");fe("\\lq","`");fe("\\rq","'");fe("\\aa","\\r a");fe("\\AA","\\r A");fe("\\textcopyright","\\html@mathml{\\textcircled{c}}{\\char`\xA9}");fe("\\copyright","\\TextOrMath{\\textcopyright}{\\text{\\textcopyright}}");fe("\\textregistered","\\html@mathml{\\textcircled{\\scriptsize R}}{\\char`\xAE}");fe("\u212C","\\mathscr{B}");fe("\u2130","\\mathscr{E}");fe("\u2131","\\mathscr{F}");fe("\u210B","\\mathscr{H}");fe("\u2110","\\mathscr{I}");fe("\u2112","\\mathscr{L}");fe("\u2133","\\mathscr{M}");fe("\u211B","\\mathscr{R}");fe("\u212D","\\mathfrak{C}");fe("\u210C","\\mathfrak{H}");fe("\u2128","\\mathfrak{Z}");fe("\\Bbbk","\\Bbb{k}");fe("\xB7","\\cdotp");fe("\\llap","\\mathllap{\\textrm{#1}}");fe("\\rlap","\\mathrlap{\\textrm{#1}}");fe("\\clap","\\mathclap{\\textrm{#1}}");fe("\\mathstrut","\\vphantom{(}");fe("\\underbar","\\underline{\\text{#1}}");fe("\\not",'\\html@mathml{\\mathrel{\\mathrlap\\@not}}{\\char"338}');fe("\\neq","\\html@mathml{\\mathrel{\\not=}}{\\mathrel{\\char`\u2260}}");fe("\\ne","\\neq");fe("\u2260","\\neq");fe("\\notin","\\html@mathml{\\mathrel{{\\in}\\mathllap{/\\mskip1mu}}}{\\mathrel{\\char`\u2209}}");fe("\u2209","\\notin");fe("\u2258","\\html@mathml{\\mathrel{=\\kern{-1em}\\raisebox{0.4em}{$\\scriptsize\\frown$}}}{\\mathrel{\\char`\u2258}}");fe("\u2259","\\html@mathml{\\stackrel{\\tiny\\wedge}{=}}{\\mathrel{\\char`\u2258}}");fe("\u225A","\\html@mathml{\\stackrel{\\tiny\\vee}{=}}{\\mathrel{\\char`\u225A}}");fe("\u225B","\\html@mathml{\\stackrel{\\scriptsize\\star}{=}}{\\mathrel{\\char`\u225B}}");fe("\u225D","\\html@mathml{\\stackrel{\\tiny\\mathrm{def}}{=}}{\\mathrel{\\char`\u225D}}");fe("\u225E","\\html@mathml{\\stackrel{\\tiny\\mathrm{m}}{=}}{\\mathrel{\\char`\u225E}}");fe("\u225F","\\html@mathml{\\stackrel{\\tiny?}{=}}{\\mathrel{\\char`\u225F}}");fe("\u27C2","\\perp");fe("\u203C","\\mathclose{!\\mkern-0.8mu!}");fe("\u220C","\\notni");fe("\u231C","\\ulcorner");fe("\u231D","\\urcorner");fe("\u231E","\\llcorner");fe("\u231F","\\lrcorner");fe("\xA9","\\copyright");fe("\xAE","\\textregistered");fe("\uFE0F","\\textregistered");fe("\\ulcorner",'\\html@mathml{\\@ulcorner}{\\mathop{\\char"231c}}');fe("\\urcorner",'\\html@mathml{\\@urcorner}{\\mathop{\\char"231d}}');fe("\\llcorner",'\\html@mathml{\\@llcorner}{\\mathop{\\char"231e}}');fe("\\lrcorner",'\\html@mathml{\\@lrcorner}{\\mathop{\\char"231f}}');fe("\\vdots","\\mathord{\\varvdots\\rule{0pt}{15pt}}");fe("\u22EE","\\vdots");fe("\\varGamma","\\mathit{\\Gamma}");fe("\\varDelta","\\mathit{\\Delta}");fe("\\varTheta","\\mathit{\\Theta}");fe("\\varLambda","\\mathit{\\Lambda}");fe("\\varXi","\\mathit{\\Xi}");fe("\\varPi","\\mathit{\\Pi}");fe("\\varSigma","\\mathit{\\Sigma}");fe("\\varUpsilon","\\mathit{\\Upsilon}");fe("\\varPhi","\\mathit{\\Phi}");fe("\\varPsi","\\mathit{\\Psi}");fe("\\varOmega","\\mathit{\\Omega}");fe("\\substack","\\begin{subarray}{c}#1\\end{subarray}");fe("\\colon","\\nobreak\\mskip2mu\\mathpunct{}\\mathchoice{\\mkern-3mu}{\\mkern-3mu}{}{}{:}\\mskip6mu\\relax");fe("\\boxed","\\fbox{$\\displaystyle{#1}$}");fe("\\iff","\\DOTSB\\;\\Longleftrightarrow\\;");fe("\\implies","\\DOTSB\\;\\Longrightarrow\\;");fe("\\impliedby","\\DOTSB\\;\\Longleftarrow\\;");Oz={",":"\\dotsc","\\not":"\\dotsb","+":"\\dotsb","=":"\\dotsb","<":"\\dotsb",">":"\\dotsb","-":"\\dotsb","*":"\\dotsb",":":"\\dotsb","\\DOTSB":"\\dotsb","\\coprod":"\\dotsb","\\bigvee":"\\dotsb","\\bigwedge":"\\dotsb","\\biguplus":"\\dotsb","\\bigcap":"\\dotsb","\\bigcup":"\\dotsb","\\prod":"\\dotsb","\\sum":"\\dotsb","\\bigotimes":"\\dotsb","\\bigoplus":"\\dotsb","\\bigodot":"\\dotsb","\\bigsqcup":"\\dotsb","\\And":"\\dotsb","\\longrightarrow":"\\dotsb","\\Longrightarrow":"\\dotsb","\\longleftarrow":"\\dotsb","\\Longleftarrow":"\\dotsb","\\longleftrightarrow":"\\dotsb","\\Longleftrightarrow":"\\dotsb","\\mapsto":"\\dotsb","\\longmapsto":"\\dotsb","\\hookrightarrow":"\\dotsb","\\doteq":"\\dotsb","\\mathbin":"\\dotsb","\\mathrel":"\\dotsb","\\relbar":"\\dotsb","\\Relbar":"\\dotsb","\\xrightarrow":"\\dotsb","\\xleftarrow":"\\dotsb","\\DOTSI":"\\dotsi","\\int":"\\dotsi","\\oint":"\\dotsi","\\iint":"\\dotsi","\\iiint":"\\dotsi","\\iiiint":"\\dotsi","\\idotsint":"\\dotsi","\\DOTSX":"\\dotsx"};fe("\\dots",function(t){var e="\\dotso",r=t.expandAfterFuture().text;return r in Oz?e=Oz[r]:(r.slice(0,4)==="\\not"||r in An.math&&Jt.contains(["bin","rel"],An.math[r].group))&&(e="\\dotsb"),e});K7={")":!0,"]":!0,"\\rbrack":!0,"\\}":!0,"\\rbrace":!0,"\\rangle":!0,"\\rceil":!0,"\\rfloor":!0,"\\rgroup":!0,"\\rmoustache":!0,"\\right":!0,"\\bigr":!0,"\\biggr":!0,"\\Bigr":!0,"\\Biggr":!0,$:!0,";":!0,".":!0,",":!0};fe("\\dotso",function(t){var e=t.future().text;return e in K7?"\\ldots\\,":"\\ldots"});fe("\\dotsc",function(t){var e=t.future().text;return e in K7&&e!==","?"\\ldots\\,":"\\ldots"});fe("\\cdots",function(t){var e=t.future().text;return e in K7?"\\@cdots\\,":"\\@cdots"});fe("\\dotsb","\\cdots");fe("\\dotsm","\\cdots");fe("\\dotsi","\\!\\cdots");fe("\\dotsx","\\ldots\\,");fe("\\DOTSI","\\relax");fe("\\DOTSB","\\relax");fe("\\DOTSX","\\relax");fe("\\tmspace","\\TextOrMath{\\kern#1#3}{\\mskip#1#2}\\relax");fe("\\,","\\tmspace+{3mu}{.1667em}");fe("\\thinspace","\\,");fe("\\>","\\mskip{4mu}");fe("\\:","\\tmspace+{4mu}{.2222em}");fe("\\medspace","\\:");fe("\\;","\\tmspace+{5mu}{.2777em}");fe("\\thickspace","\\;");fe("\\!","\\tmspace-{3mu}{.1667em}");fe("\\negthinspace","\\!");fe("\\negmedspace","\\tmspace-{4mu}{.2222em}");fe("\\negthickspace","\\tmspace-{5mu}{.277em}");fe("\\enspace","\\kern.5em ");fe("\\enskip","\\hskip.5em\\relax");fe("\\quad","\\hskip1em\\relax");fe("\\qquad","\\hskip2em\\relax");fe("\\tag","\\@ifstar\\tag@literal\\tag@paren");fe("\\tag@paren","\\tag@literal{({#1})}");fe("\\tag@literal",t=>{if(t.macros.get("\\df@tag"))throw new gt("Multiple \\tag");return"\\gdef\\df@tag{\\text{#1}}"});fe("\\bmod","\\mathchoice{\\mskip1mu}{\\mskip1mu}{\\mskip5mu}{\\mskip5mu}\\mathbin{\\rm mod}\\mathchoice{\\mskip1mu}{\\mskip1mu}{\\mskip5mu}{\\mskip5mu}");fe("\\pod","\\allowbreak\\mathchoice{\\mkern18mu}{\\mkern8mu}{\\mkern8mu}{\\mkern8mu}(#1)");fe("\\pmod","\\pod{{\\rm mod}\\mkern6mu#1}");fe("\\mod","\\allowbreak\\mathchoice{\\mkern18mu}{\\mkern12mu}{\\mkern12mu}{\\mkern12mu}{\\rm mod}\\,\\,#1");fe("\\newline","\\\\\\relax");fe("\\TeX","\\textrm{\\html@mathml{T\\kern-.1667em\\raisebox{-.5ex}{E}\\kern-.125emX}{TeX}}");_G=kt(jl["Main-Regular"][84][1]-.7*jl["Main-Regular"][65][1]);fe("\\LaTeX","\\textrm{\\html@mathml{"+("L\\kern-.36em\\raisebox{"+_G+"}{\\scriptstyle A}")+"\\kern-.15em\\TeX}{LaTeX}}");fe("\\KaTeX","\\textrm{\\html@mathml{"+("K\\kern-.17em\\raisebox{"+_G+"}{\\scriptstyle A}")+"\\kern-.15em\\TeX}{KaTeX}}");fe("\\hspace","\\@ifstar\\@hspacer\\@hspace");fe("\\@hspace","\\hskip #1\\relax");fe("\\@hspacer","\\rule{0pt}{0pt}\\hskip #1\\relax");fe("\\ordinarycolon",":");fe("\\vcentcolon","\\mathrel{\\mathop\\ordinarycolon}");fe("\\dblcolon",'\\html@mathml{\\mathrel{\\vcentcolon\\mathrel{\\mkern-.9mu}\\vcentcolon}}{\\mathop{\\char"2237}}');fe("\\coloneqq",'\\html@mathml{\\mathrel{\\vcentcolon\\mathrel{\\mkern-1.2mu}=}}{\\mathop{\\char"2254}}');fe("\\Coloneqq",'\\html@mathml{\\mathrel{\\dblcolon\\mathrel{\\mkern-1.2mu}=}}{\\mathop{\\char"2237\\char"3d}}');fe("\\coloneq",'\\html@mathml{\\mathrel{\\vcentcolon\\mathrel{\\mkern-1.2mu}\\mathrel{-}}}{\\mathop{\\char"3a\\char"2212}}');fe("\\Coloneq",'\\html@mathml{\\mathrel{\\dblcolon\\mathrel{\\mkern-1.2mu}\\mathrel{-}}}{\\mathop{\\char"2237\\char"2212}}');fe("\\eqqcolon",'\\html@mathml{\\mathrel{=\\mathrel{\\mkern-1.2mu}\\vcentcolon}}{\\mathop{\\char"2255}}');fe("\\Eqqcolon",'\\html@mathml{\\mathrel{=\\mathrel{\\mkern-1.2mu}\\dblcolon}}{\\mathop{\\char"3d\\char"2237}}');fe("\\eqcolon",'\\html@mathml{\\mathrel{\\mathrel{-}\\mathrel{\\mkern-1.2mu}\\vcentcolon}}{\\mathop{\\char"2239}}');fe("\\Eqcolon",'\\html@mathml{\\mathrel{\\mathrel{-}\\mathrel{\\mkern-1.2mu}\\dblcolon}}{\\mathop{\\char"2212\\char"2237}}');fe("\\colonapprox",'\\html@mathml{\\mathrel{\\vcentcolon\\mathrel{\\mkern-1.2mu}\\approx}}{\\mathop{\\char"3a\\char"2248}}');fe("\\Colonapprox",'\\html@mathml{\\mathrel{\\dblcolon\\mathrel{\\mkern-1.2mu}\\approx}}{\\mathop{\\char"2237\\char"2248}}');fe("\\colonsim",'\\html@mathml{\\mathrel{\\vcentcolon\\mathrel{\\mkern-1.2mu}\\sim}}{\\mathop{\\char"3a\\char"223c}}');fe("\\Colonsim",'\\html@mathml{\\mathrel{\\dblcolon\\mathrel{\\mkern-1.2mu}\\sim}}{\\mathop{\\char"2237\\char"223c}}');fe("\u2237","\\dblcolon");fe("\u2239","\\eqcolon");fe("\u2254","\\coloneqq");fe("\u2255","\\eqqcolon");fe("\u2A74","\\Coloneqq");fe("\\ratio","\\vcentcolon");fe("\\coloncolon","\\dblcolon");fe("\\colonequals","\\coloneqq");fe("\\coloncolonequals","\\Coloneqq");fe("\\equalscolon","\\eqqcolon");fe("\\equalscoloncolon","\\Eqqcolon");fe("\\colonminus","\\coloneq");fe("\\coloncolonminus","\\Coloneq");fe("\\minuscolon","\\eqcolon");fe("\\minuscoloncolon","\\Eqcolon");fe("\\coloncolonapprox","\\Colonapprox");fe("\\coloncolonsim","\\Colonsim");fe("\\simcolon","\\mathrel{\\sim\\mathrel{\\mkern-1.2mu}\\vcentcolon}");fe("\\simcoloncolon","\\mathrel{\\sim\\mathrel{\\mkern-1.2mu}\\dblcolon}");fe("\\approxcolon","\\mathrel{\\approx\\mathrel{\\mkern-1.2mu}\\vcentcolon}");fe("\\approxcoloncolon","\\mathrel{\\approx\\mathrel{\\mkern-1.2mu}\\dblcolon}");fe("\\notni","\\html@mathml{\\not\\ni}{\\mathrel{\\char`\u220C}}");fe("\\limsup","\\DOTSB\\operatorname*{lim\\,sup}");fe("\\liminf","\\DOTSB\\operatorname*{lim\\,inf}");fe("\\injlim","\\DOTSB\\operatorname*{inj\\,lim}");fe("\\projlim","\\DOTSB\\operatorname*{proj\\,lim}");fe("\\varlimsup","\\DOTSB\\operatorname*{\\overline{lim}}");fe("\\varliminf","\\DOTSB\\operatorname*{\\underline{lim}}");fe("\\varinjlim","\\DOTSB\\operatorname*{\\underrightarrow{lim}}");fe("\\varprojlim","\\DOTSB\\operatorname*{\\underleftarrow{lim}}");fe("\\gvertneqq","\\html@mathml{\\@gvertneqq}{\u2269}");fe("\\lvertneqq","\\html@mathml{\\@lvertneqq}{\u2268}");fe("\\ngeqq","\\html@mathml{\\@ngeqq}{\u2271}");fe("\\ngeqslant","\\html@mathml{\\@ngeqslant}{\u2271}");fe("\\nleqq","\\html@mathml{\\@nleqq}{\u2270}");fe("\\nleqslant","\\html@mathml{\\@nleqslant}{\u2270}");fe("\\nshortmid","\\html@mathml{\\@nshortmid}{\u2224}");fe("\\nshortparallel","\\html@mathml{\\@nshortparallel}{\u2226}");fe("\\nsubseteqq","\\html@mathml{\\@nsubseteqq}{\u2288}");fe("\\nsupseteqq","\\html@mathml{\\@nsupseteqq}{\u2289}");fe("\\varsubsetneq","\\html@mathml{\\@varsubsetneq}{\u228A}");fe("\\varsubsetneqq","\\html@mathml{\\@varsubsetneqq}{\u2ACB}");fe("\\varsupsetneq","\\html@mathml{\\@varsupsetneq}{\u228B}");fe("\\varsupsetneqq","\\html@mathml{\\@varsupsetneqq}{\u2ACC}");fe("\\imath","\\html@mathml{\\@imath}{\u0131}");fe("\\jmath","\\html@mathml{\\@jmath}{\u0237}");fe("\\llbracket","\\html@mathml{\\mathopen{[\\mkern-3.2mu[}}{\\mathopen{\\char`\u27E6}}");fe("\\rrbracket","\\html@mathml{\\mathclose{]\\mkern-3.2mu]}}{\\mathclose{\\char`\u27E7}}");fe("\u27E6","\\llbracket");fe("\u27E7","\\rrbracket");fe("\\lBrace","\\html@mathml{\\mathopen{\\{\\mkern-3.2mu[}}{\\mathopen{\\char`\u2983}}");fe("\\rBrace","\\html@mathml{\\mathclose{]\\mkern-3.2mu\\}}}{\\mathclose{\\char`\u2984}}");fe("\u2983","\\lBrace");fe("\u2984","\\rBrace");fe("\\minuso","\\mathbin{\\html@mathml{{\\mathrlap{\\mathchoice{\\kern{0.145em}}{\\kern{0.145em}}{\\kern{0.1015em}}{\\kern{0.0725em}}\\circ}{-}}}{\\char`\u29B5}}");fe("\u29B5","\\minuso");fe("\\darr","\\downarrow");fe("\\dArr","\\Downarrow");fe("\\Darr","\\Downarrow");fe("\\lang","\\langle");fe("\\rang","\\rangle");fe("\\uarr","\\uparrow");fe("\\uArr","\\Uparrow");fe("\\Uarr","\\Uparrow");fe("\\N","\\mathbb{N}");fe("\\R","\\mathbb{R}");fe("\\Z","\\mathbb{Z}");fe("\\alef","\\aleph");fe("\\alefsym","\\aleph");fe("\\Alpha","\\mathrm{A}");fe("\\Beta","\\mathrm{B}");fe("\\bull","\\bullet");fe("\\Chi","\\mathrm{X}");fe("\\clubs","\\clubsuit");fe("\\cnums","\\mathbb{C}");fe("\\Complex","\\mathbb{C}");fe("\\Dagger","\\ddagger");fe("\\diamonds","\\diamondsuit");fe("\\empty","\\emptyset");fe("\\Epsilon","\\mathrm{E}");fe("\\Eta","\\mathrm{H}");fe("\\exist","\\exists");fe("\\harr","\\leftrightarrow");fe("\\hArr","\\Leftrightarrow");fe("\\Harr","\\Leftrightarrow");fe("\\hearts","\\heartsuit");fe("\\image","\\Im");fe("\\infin","\\infty");fe("\\Iota","\\mathrm{I}");fe("\\isin","\\in");fe("\\Kappa","\\mathrm{K}");fe("\\larr","\\leftarrow");fe("\\lArr","\\Leftarrow");fe("\\Larr","\\Leftarrow");fe("\\lrarr","\\leftrightarrow");fe("\\lrArr","\\Leftrightarrow");fe("\\Lrarr","\\Leftrightarrow");fe("\\Mu","\\mathrm{M}");fe("\\natnums","\\mathbb{N}");fe("\\Nu","\\mathrm{N}");fe("\\Omicron","\\mathrm{O}");fe("\\plusmn","\\pm");fe("\\rarr","\\rightarrow");fe("\\rArr","\\Rightarrow");fe("\\Rarr","\\Rightarrow");fe("\\real","\\Re");fe("\\reals","\\mathbb{R}");fe("\\Reals","\\mathbb{R}");fe("\\Rho","\\mathrm{P}");fe("\\sdot","\\cdot");fe("\\sect","\\S");fe("\\spades","\\spadesuit");fe("\\sub","\\subset");fe("\\sube","\\subseteq");fe("\\supe","\\supseteq");fe("\\Tau","\\mathrm{T}");fe("\\thetasym","\\vartheta");fe("\\weierp","\\wp");fe("\\Zeta","\\mathrm{Z}");fe("\\argmin","\\DOTSB\\operatorname*{arg\\,min}");fe("\\argmax","\\DOTSB\\operatorname*{arg\\,max}");fe("\\plim","\\DOTSB\\mathop{\\operatorname{plim}}\\limits");fe("\\bra","\\mathinner{\\langle{#1}|}");fe("\\ket","\\mathinner{|{#1}\\rangle}");fe("\\braket","\\mathinner{\\langle{#1}\\rangle}");fe("\\Bra","\\left\\langle#1\\right|");fe("\\Ket","\\left|#1\\right\\rangle");DG=o(t=>e=>{var r=e.consumeArg().tokens,n=e.consumeArg().tokens,i=e.consumeArg().tokens,a=e.consumeArg().tokens,s=e.macros.get("|"),l=e.macros.get("\\|");e.macros.beginGroup();var u=o(d=>p=>{t&&(p.macros.set("|",s),i.length&&p.macros.set("\\|",l));var m=d;if(!d&&i.length){var g=p.future();g.text==="|"&&(p.popToken(),m=!0)}return{tokens:m?i:n,numArgs:0}},"midMacro");e.macros.set("|",u(!1)),i.length&&e.macros.set("\\|",u(!0));var h=e.consumeArg().tokens,f=e.expandTokens([...a,...h,...r]);return e.macros.endGroup(),{tokens:f.reverse(),numArgs:0}},"braketHelper");fe("\\bra@ket",DG(!1));fe("\\bra@set",DG(!0));fe("\\Braket","\\bra@ket{\\left\\langle}{\\,\\middle\\vert\\,}{\\,\\middle\\vert\\,}{\\right\\rangle}");fe("\\Set","\\bra@set{\\left\\{\\:}{\\;\\middle\\vert\\;}{\\;\\middle\\Vert\\;}{\\:\\right\\}}");fe("\\set","\\bra@set{\\{\\,}{\\mid}{}{\\,\\}}");fe("\\angln","{\\angl n}");fe("\\blue","\\textcolor{##6495ed}{#1}");fe("\\orange","\\textcolor{##ffa500}{#1}");fe("\\pink","\\textcolor{##ff00af}{#1}");fe("\\red","\\textcolor{##df0030}{#1}");fe("\\green","\\textcolor{##28ae7b}{#1}");fe("\\gray","\\textcolor{gray}{#1}");fe("\\purple","\\textcolor{##9d38bd}{#1}");fe("\\blueA","\\textcolor{##ccfaff}{#1}");fe("\\blueB","\\textcolor{##80f6ff}{#1}");fe("\\blueC","\\textcolor{##63d9ea}{#1}");fe("\\blueD","\\textcolor{##11accd}{#1}");fe("\\blueE","\\textcolor{##0c7f99}{#1}");fe("\\tealA","\\textcolor{##94fff5}{#1}");fe("\\tealB","\\textcolor{##26edd5}{#1}");fe("\\tealC","\\textcolor{##01d1c1}{#1}");fe("\\tealD","\\textcolor{##01a995}{#1}");fe("\\tealE","\\textcolor{##208170}{#1}");fe("\\greenA","\\textcolor{##b6ffb0}{#1}");fe("\\greenB","\\textcolor{##8af281}{#1}");fe("\\greenC","\\textcolor{##74cf70}{#1}");fe("\\greenD","\\textcolor{##1fab54}{#1}");fe("\\greenE","\\textcolor{##0d923f}{#1}");fe("\\goldA","\\textcolor{##ffd0a9}{#1}");fe("\\goldB","\\textcolor{##ffbb71}{#1}");fe("\\goldC","\\textcolor{##ff9c39}{#1}");fe("\\goldD","\\textcolor{##e07d10}{#1}");fe("\\goldE","\\textcolor{##a75a05}{#1}");fe("\\redA","\\textcolor{##fca9a9}{#1}");fe("\\redB","\\textcolor{##ff8482}{#1}");fe("\\redC","\\textcolor{##f9685d}{#1}");fe("\\redD","\\textcolor{##e84d39}{#1}");fe("\\redE","\\textcolor{##bc2612}{#1}");fe("\\maroonA","\\textcolor{##ffbde0}{#1}");fe("\\maroonB","\\textcolor{##ff92c6}{#1}");fe("\\maroonC","\\textcolor{##ed5fa6}{#1}");fe("\\maroonD","\\textcolor{##ca337c}{#1}");fe("\\maroonE","\\textcolor{##9e034e}{#1}");fe("\\purpleA","\\textcolor{##ddd7ff}{#1}");fe("\\purpleB","\\textcolor{##c6b9fc}{#1}");fe("\\purpleC","\\textcolor{##aa87ff}{#1}");fe("\\purpleD","\\textcolor{##7854ab}{#1}");fe("\\purpleE","\\textcolor{##543b78}{#1}");fe("\\mintA","\\textcolor{##f5f9e8}{#1}");fe("\\mintB","\\textcolor{##edf2df}{#1}");fe("\\mintC","\\textcolor{##e0e5cc}{#1}");fe("\\grayA","\\textcolor{##f6f7f7}{#1}");fe("\\grayB","\\textcolor{##f0f1f2}{#1}");fe("\\grayC","\\textcolor{##e3e5e6}{#1}");fe("\\grayD","\\textcolor{##d6d8da}{#1}");fe("\\grayE","\\textcolor{##babec2}{#1}");fe("\\grayF","\\textcolor{##888d93}{#1}");fe("\\grayG","\\textcolor{##626569}{#1}");fe("\\grayH","\\textcolor{##3b3e40}{#1}");fe("\\grayI","\\textcolor{##21242c}{#1}");fe("\\kaBlue","\\textcolor{##314453}{#1}");fe("\\kaGreen","\\textcolor{##71B307}{#1}");LG={"^":!0,_:!0,"\\limits":!0,"\\nolimits":!0},I7=class{static{o(this,"MacroExpander")}constructor(e,r,n){this.settings=void 0,this.expansionCount=void 0,this.lexer=void 0,this.macros=void 0,this.stack=void 0,this.mode=void 0,this.settings=r,this.expansionCount=0,this.feed(e),this.macros=new M7(V4e,r.macros),this.mode=n,this.stack=[]}feed(e){this.lexer=new y3(e,this.settings)}switchMode(e){this.mode=e}beginGroup(){this.macros.beginGroup()}endGroup(){this.macros.endGroup()}endGroups(){this.macros.endGroups()}future(){return this.stack.length===0&&this.pushToken(this.lexer.lex()),this.stack[this.stack.length-1]}popToken(){return this.future(),this.stack.pop()}pushToken(e){this.stack.push(e)}pushTokens(e){this.stack.push(...e)}scanArgument(e){var r,n,i;if(e){if(this.consumeSpaces(),this.future().text!=="[")return null;r=this.popToken(),{tokens:i,end:n}=this.consumeArg(["]"])}else({tokens:i,start:r,end:n}=this.consumeArg());return this.pushToken(new So("EOF",n.loc)),this.pushTokens(i),r.range(n,"")}consumeSpaces(){for(;;){var e=this.future();if(e.text===" ")this.stack.pop();else break}}consumeArg(e){var r=[],n=e&&e.length>0;n||this.consumeSpaces();var i=this.future(),a,s=0,l=0;do{if(a=this.popToken(),r.push(a),a.text==="{")++s;else if(a.text==="}"){if(--s,s===-1)throw new gt("Extra }",a)}else if(a.text==="EOF")throw new gt("Unexpected end of input in a macro argument, expected '"+(e&&n?e[l]:"}")+"'",a);if(e&&n)if((s===0||s===1&&e[l]==="{")&&a.text===e[l]){if(++l,l===e.length){r.splice(-l,l);break}}else l=0}while(s!==0||n);return i.text==="{"&&r[r.length-1].text==="}"&&(r.pop(),r.shift()),r.reverse(),{tokens:r,start:i,end:a}}consumeArgs(e,r){if(r){if(r.length!==e+1)throw new gt("The length of delimiters doesn't match the number of args!");for(var n=r[0],i=0;ithis.settings.maxExpand)throw new gt("Too many expansions: infinite loop or need to increase maxExpand setting")}expandOnce(e){var r=this.popToken(),n=r.text,i=r.noexpand?null:this._getExpansion(n);if(i==null||e&&i.unexpandable){if(e&&i==null&&n[0]==="\\"&&!this.isDefined(n))throw new gt("Undefined control sequence: "+n);return this.pushToken(r),!1}this.countExpansion(1);var a=i.tokens,s=this.consumeArgs(i.numArgs,i.delimiters);if(i.numArgs){a=a.slice();for(var l=a.length-1;l>=0;--l){var u=a[l];if(u.text==="#"){if(l===0)throw new gt("Incomplete placeholder at end of macro body",u);if(u=a[--l],u.text==="#")a.splice(l+1,1);else if(/^[1-9]$/.test(u.text))a.splice(l,2,...s[+u.text-1]);else throw new gt("Not a valid argument number",u)}}}return this.pushTokens(a),a.length}expandAfterFuture(){return this.expandOnce(),this.future()}expandNextToken(){for(;;)if(this.expandOnce()===!1){var e=this.stack.pop();return e.treatAsRelax&&(e.text="\\relax"),e}throw new Error}expandMacro(e){return this.macros.has(e)?this.expandTokens([new So(e)]):void 0}expandTokens(e){var r=[],n=this.stack.length;for(this.pushTokens(e);this.stack.length>n;)if(this.expandOnce(!0)===!1){var i=this.stack.pop();i.treatAsRelax&&(i.noexpand=!1,i.treatAsRelax=!1),r.push(i)}return this.countExpansion(r.length),r}expandMacroAsText(e){var r=this.expandMacro(e);return r&&r.map(n=>n.text).join("")}_getExpansion(e){var r=this.macros.get(e);if(r==null)return r;if(e.length===1){var n=this.lexer.catcodes[e];if(n!=null&&n!==13)return}var i=typeof r=="function"?r(this):r;if(typeof i=="string"){var a=0;if(i.indexOf("#")!==-1)for(var s=i.replace(/##/g,"");s.indexOf("#"+(a+1))!==-1;)++a;for(var l=new y3(i,this.settings),u=[],h=l.lex();h.text!=="EOF";)u.push(h),h=l.lex();u.reverse();var f={tokens:u,numArgs:a};return f}return i}isDefined(e){return this.macros.has(e)||hh.hasOwnProperty(e)||An.math.hasOwnProperty(e)||An.text.hasOwnProperty(e)||LG.hasOwnProperty(e)}isExpandable(e){var r=this.macros.get(e);return r!=null?typeof r=="string"||typeof r=="function"||!r.unexpandable:hh.hasOwnProperty(e)&&!hh[e].primitive}},Pz=/^[₊₋₌₍₎₀₁₂₃₄₅₆₇₈₉ₐₑₕᵢⱼₖₗₘₙₒₚᵣₛₜᵤᵥₓᵦᵧᵨᵩᵪ]/,l3=Object.freeze({"\u208A":"+","\u208B":"-","\u208C":"=","\u208D":"(","\u208E":")","\u2080":"0","\u2081":"1","\u2082":"2","\u2083":"3","\u2084":"4","\u2085":"5","\u2086":"6","\u2087":"7","\u2088":"8","\u2089":"9","\u2090":"a","\u2091":"e","\u2095":"h","\u1D62":"i","\u2C7C":"j","\u2096":"k","\u2097":"l","\u2098":"m","\u2099":"n","\u2092":"o","\u209A":"p","\u1D63":"r","\u209B":"s","\u209C":"t","\u1D64":"u","\u1D65":"v","\u2093":"x","\u1D66":"\u03B2","\u1D67":"\u03B3","\u1D68":"\u03C1","\u1D69":"\u03D5","\u1D6A":"\u03C7","\u207A":"+","\u207B":"-","\u207C":"=","\u207D":"(","\u207E":")","\u2070":"0","\xB9":"1","\xB2":"2","\xB3":"3","\u2074":"4","\u2075":"5","\u2076":"6","\u2077":"7","\u2078":"8","\u2079":"9","\u1D2C":"A","\u1D2E":"B","\u1D30":"D","\u1D31":"E","\u1D33":"G","\u1D34":"H","\u1D35":"I","\u1D36":"J","\u1D37":"K","\u1D38":"L","\u1D39":"M","\u1D3A":"N","\u1D3C":"O","\u1D3E":"P","\u1D3F":"R","\u1D40":"T","\u1D41":"U","\u2C7D":"V","\u1D42":"W","\u1D43":"a","\u1D47":"b","\u1D9C":"c","\u1D48":"d","\u1D49":"e","\u1DA0":"f","\u1D4D":"g",\u02B0:"h","\u2071":"i",\u02B2:"j","\u1D4F":"k",\u02E1:"l","\u1D50":"m",\u207F:"n","\u1D52":"o","\u1D56":"p",\u02B3:"r",\u02E2:"s","\u1D57":"t","\u1D58":"u","\u1D5B":"v",\u02B7:"w",\u02E3:"x",\u02B8:"y","\u1DBB":"z","\u1D5D":"\u03B2","\u1D5E":"\u03B3","\u1D5F":"\u03B4","\u1D60":"\u03D5","\u1D61":"\u03C7","\u1DBF":"\u03B8"}),T7={"\u0301":{text:"\\'",math:"\\acute"},"\u0300":{text:"\\`",math:"\\grave"},"\u0308":{text:'\\"',math:"\\ddot"},"\u0303":{text:"\\~",math:"\\tilde"},"\u0304":{text:"\\=",math:"\\bar"},"\u0306":{text:"\\u",math:"\\breve"},"\u030C":{text:"\\v",math:"\\check"},"\u0302":{text:"\\^",math:"\\hat"},"\u0307":{text:"\\.",math:"\\dot"},"\u030A":{text:"\\r",math:"\\mathring"},"\u030B":{text:"\\H"},"\u0327":{text:"\\c"}},Bz={\u00E1:"a\u0301",\u00E0:"a\u0300",\u00E4:"a\u0308",\u01DF:"a\u0308\u0304",\u00E3:"a\u0303",\u0101:"a\u0304",\u0103:"a\u0306",\u1EAF:"a\u0306\u0301",\u1EB1:"a\u0306\u0300",\u1EB5:"a\u0306\u0303",\u01CE:"a\u030C",\u00E2:"a\u0302",\u1EA5:"a\u0302\u0301",\u1EA7:"a\u0302\u0300",\u1EAB:"a\u0302\u0303",\u0227:"a\u0307",\u01E1:"a\u0307\u0304",\u00E5:"a\u030A",\u01FB:"a\u030A\u0301",\u1E03:"b\u0307",\u0107:"c\u0301",\u1E09:"c\u0327\u0301",\u010D:"c\u030C",\u0109:"c\u0302",\u010B:"c\u0307",\u00E7:"c\u0327",\u010F:"d\u030C",\u1E0B:"d\u0307",\u1E11:"d\u0327",\u00E9:"e\u0301",\u00E8:"e\u0300",\u00EB:"e\u0308",\u1EBD:"e\u0303",\u0113:"e\u0304",\u1E17:"e\u0304\u0301",\u1E15:"e\u0304\u0300",\u0115:"e\u0306",\u1E1D:"e\u0327\u0306",\u011B:"e\u030C",\u00EA:"e\u0302",\u1EBF:"e\u0302\u0301",\u1EC1:"e\u0302\u0300",\u1EC5:"e\u0302\u0303",\u0117:"e\u0307",\u0229:"e\u0327",\u1E1F:"f\u0307",\u01F5:"g\u0301",\u1E21:"g\u0304",\u011F:"g\u0306",\u01E7:"g\u030C",\u011D:"g\u0302",\u0121:"g\u0307",\u0123:"g\u0327",\u1E27:"h\u0308",\u021F:"h\u030C",\u0125:"h\u0302",\u1E23:"h\u0307",\u1E29:"h\u0327",\u00ED:"i\u0301",\u00EC:"i\u0300",\u00EF:"i\u0308",\u1E2F:"i\u0308\u0301",\u0129:"i\u0303",\u012B:"i\u0304",\u012D:"i\u0306",\u01D0:"i\u030C",\u00EE:"i\u0302",\u01F0:"j\u030C",\u0135:"j\u0302",\u1E31:"k\u0301",\u01E9:"k\u030C",\u0137:"k\u0327",\u013A:"l\u0301",\u013E:"l\u030C",\u013C:"l\u0327",\u1E3F:"m\u0301",\u1E41:"m\u0307",\u0144:"n\u0301",\u01F9:"n\u0300",\u00F1:"n\u0303",\u0148:"n\u030C",\u1E45:"n\u0307",\u0146:"n\u0327",\u00F3:"o\u0301",\u00F2:"o\u0300",\u00F6:"o\u0308",\u022B:"o\u0308\u0304",\u00F5:"o\u0303",\u1E4D:"o\u0303\u0301",\u1E4F:"o\u0303\u0308",\u022D:"o\u0303\u0304",\u014D:"o\u0304",\u1E53:"o\u0304\u0301",\u1E51:"o\u0304\u0300",\u014F:"o\u0306",\u01D2:"o\u030C",\u00F4:"o\u0302",\u1ED1:"o\u0302\u0301",\u1ED3:"o\u0302\u0300",\u1ED7:"o\u0302\u0303",\u022F:"o\u0307",\u0231:"o\u0307\u0304",\u0151:"o\u030B",\u1E55:"p\u0301",\u1E57:"p\u0307",\u0155:"r\u0301",\u0159:"r\u030C",\u1E59:"r\u0307",\u0157:"r\u0327",\u015B:"s\u0301",\u1E65:"s\u0301\u0307",\u0161:"s\u030C",\u1E67:"s\u030C\u0307",\u015D:"s\u0302",\u1E61:"s\u0307",\u015F:"s\u0327",\u1E97:"t\u0308",\u0165:"t\u030C",\u1E6B:"t\u0307",\u0163:"t\u0327",\u00FA:"u\u0301",\u00F9:"u\u0300",\u00FC:"u\u0308",\u01D8:"u\u0308\u0301",\u01DC:"u\u0308\u0300",\u01D6:"u\u0308\u0304",\u01DA:"u\u0308\u030C",\u0169:"u\u0303",\u1E79:"u\u0303\u0301",\u016B:"u\u0304",\u1E7B:"u\u0304\u0308",\u016D:"u\u0306",\u01D4:"u\u030C",\u00FB:"u\u0302",\u016F:"u\u030A",\u0171:"u\u030B",\u1E7D:"v\u0303",\u1E83:"w\u0301",\u1E81:"w\u0300",\u1E85:"w\u0308",\u0175:"w\u0302",\u1E87:"w\u0307",\u1E98:"w\u030A",\u1E8D:"x\u0308",\u1E8B:"x\u0307",\u00FD:"y\u0301",\u1EF3:"y\u0300",\u00FF:"y\u0308",\u1EF9:"y\u0303",\u0233:"y\u0304",\u0177:"y\u0302",\u1E8F:"y\u0307",\u1E99:"y\u030A",\u017A:"z\u0301",\u017E:"z\u030C",\u1E91:"z\u0302",\u017C:"z\u0307",\u00C1:"A\u0301",\u00C0:"A\u0300",\u00C4:"A\u0308",\u01DE:"A\u0308\u0304",\u00C3:"A\u0303",\u0100:"A\u0304",\u0102:"A\u0306",\u1EAE:"A\u0306\u0301",\u1EB0:"A\u0306\u0300",\u1EB4:"A\u0306\u0303",\u01CD:"A\u030C",\u00C2:"A\u0302",\u1EA4:"A\u0302\u0301",\u1EA6:"A\u0302\u0300",\u1EAA:"A\u0302\u0303",\u0226:"A\u0307",\u01E0:"A\u0307\u0304",\u00C5:"A\u030A",\u01FA:"A\u030A\u0301",\u1E02:"B\u0307",\u0106:"C\u0301",\u1E08:"C\u0327\u0301",\u010C:"C\u030C",\u0108:"C\u0302",\u010A:"C\u0307",\u00C7:"C\u0327",\u010E:"D\u030C",\u1E0A:"D\u0307",\u1E10:"D\u0327",\u00C9:"E\u0301",\u00C8:"E\u0300",\u00CB:"E\u0308",\u1EBC:"E\u0303",\u0112:"E\u0304",\u1E16:"E\u0304\u0301",\u1E14:"E\u0304\u0300",\u0114:"E\u0306",\u1E1C:"E\u0327\u0306",\u011A:"E\u030C",\u00CA:"E\u0302",\u1EBE:"E\u0302\u0301",\u1EC0:"E\u0302\u0300",\u1EC4:"E\u0302\u0303",\u0116:"E\u0307",\u0228:"E\u0327",\u1E1E:"F\u0307",\u01F4:"G\u0301",\u1E20:"G\u0304",\u011E:"G\u0306",\u01E6:"G\u030C",\u011C:"G\u0302",\u0120:"G\u0307",\u0122:"G\u0327",\u1E26:"H\u0308",\u021E:"H\u030C",\u0124:"H\u0302",\u1E22:"H\u0307",\u1E28:"H\u0327",\u00CD:"I\u0301",\u00CC:"I\u0300",\u00CF:"I\u0308",\u1E2E:"I\u0308\u0301",\u0128:"I\u0303",\u012A:"I\u0304",\u012C:"I\u0306",\u01CF:"I\u030C",\u00CE:"I\u0302",\u0130:"I\u0307",\u0134:"J\u0302",\u1E30:"K\u0301",\u01E8:"K\u030C",\u0136:"K\u0327",\u0139:"L\u0301",\u013D:"L\u030C",\u013B:"L\u0327",\u1E3E:"M\u0301",\u1E40:"M\u0307",\u0143:"N\u0301",\u01F8:"N\u0300",\u00D1:"N\u0303",\u0147:"N\u030C",\u1E44:"N\u0307",\u0145:"N\u0327",\u00D3:"O\u0301",\u00D2:"O\u0300",\u00D6:"O\u0308",\u022A:"O\u0308\u0304",\u00D5:"O\u0303",\u1E4C:"O\u0303\u0301",\u1E4E:"O\u0303\u0308",\u022C:"O\u0303\u0304",\u014C:"O\u0304",\u1E52:"O\u0304\u0301",\u1E50:"O\u0304\u0300",\u014E:"O\u0306",\u01D1:"O\u030C",\u00D4:"O\u0302",\u1ED0:"O\u0302\u0301",\u1ED2:"O\u0302\u0300",\u1ED6:"O\u0302\u0303",\u022E:"O\u0307",\u0230:"O\u0307\u0304",\u0150:"O\u030B",\u1E54:"P\u0301",\u1E56:"P\u0307",\u0154:"R\u0301",\u0158:"R\u030C",\u1E58:"R\u0307",\u0156:"R\u0327",\u015A:"S\u0301",\u1E64:"S\u0301\u0307",\u0160:"S\u030C",\u1E66:"S\u030C\u0307",\u015C:"S\u0302",\u1E60:"S\u0307",\u015E:"S\u0327",\u0164:"T\u030C",\u1E6A:"T\u0307",\u0162:"T\u0327",\u00DA:"U\u0301",\u00D9:"U\u0300",\u00DC:"U\u0308",\u01D7:"U\u0308\u0301",\u01DB:"U\u0308\u0300",\u01D5:"U\u0308\u0304",\u01D9:"U\u0308\u030C",\u0168:"U\u0303",\u1E78:"U\u0303\u0301",\u016A:"U\u0304",\u1E7A:"U\u0304\u0308",\u016C:"U\u0306",\u01D3:"U\u030C",\u00DB:"U\u0302",\u016E:"U\u030A",\u0170:"U\u030B",\u1E7C:"V\u0303",\u1E82:"W\u0301",\u1E80:"W\u0300",\u1E84:"W\u0308",\u0174:"W\u0302",\u1E86:"W\u0307",\u1E8C:"X\u0308",\u1E8A:"X\u0307",\u00DD:"Y\u0301",\u1EF2:"Y\u0300",\u0178:"Y\u0308",\u1EF8:"Y\u0303",\u0232:"Y\u0304",\u0176:"Y\u0302",\u1E8E:"Y\u0307",\u0179:"Z\u0301",\u017D:"Z\u030C",\u1E90:"Z\u0302",\u017B:"Z\u0307",\u03AC:"\u03B1\u0301",\u1F70:"\u03B1\u0300",\u1FB1:"\u03B1\u0304",\u1FB0:"\u03B1\u0306",\u03AD:"\u03B5\u0301",\u1F72:"\u03B5\u0300",\u03AE:"\u03B7\u0301",\u1F74:"\u03B7\u0300",\u03AF:"\u03B9\u0301",\u1F76:"\u03B9\u0300",\u03CA:"\u03B9\u0308",\u0390:"\u03B9\u0308\u0301",\u1FD2:"\u03B9\u0308\u0300",\u1FD1:"\u03B9\u0304",\u1FD0:"\u03B9\u0306",\u03CC:"\u03BF\u0301",\u1F78:"\u03BF\u0300",\u03CD:"\u03C5\u0301",\u1F7A:"\u03C5\u0300",\u03CB:"\u03C5\u0308",\u03B0:"\u03C5\u0308\u0301",\u1FE2:"\u03C5\u0308\u0300",\u1FE1:"\u03C5\u0304",\u1FE0:"\u03C5\u0306",\u03CE:"\u03C9\u0301",\u1F7C:"\u03C9\u0300",\u038E:"\u03A5\u0301",\u1FEA:"\u03A5\u0300",\u03AB:"\u03A5\u0308",\u1FE9:"\u03A5\u0304",\u1FE8:"\u03A5\u0306",\u038F:"\u03A9\u0301",\u1FFA:"\u03A9\u0300"},v3=class t{static{o(this,"Parser")}constructor(e,r){this.mode=void 0,this.gullet=void 0,this.settings=void 0,this.leftrightDepth=void 0,this.nextToken=void 0,this.mode="math",this.gullet=new I7(e,r,this.mode),this.settings=r,this.leftrightDepth=0}expect(e,r){if(r===void 0&&(r=!0),this.fetch().text!==e)throw new gt("Expected '"+e+"', got '"+this.fetch().text+"'",this.fetch());r&&this.consume()}consume(){this.nextToken=null}fetch(){return this.nextToken==null&&(this.nextToken=this.gullet.expandNextToken()),this.nextToken}switchMode(e){this.mode=e,this.gullet.switchMode(e)}parse(){this.settings.globalGroup||this.gullet.beginGroup(),this.settings.colorIsTextColor&&this.gullet.macros.set("\\color","\\textcolor");try{var e=this.parseExpression(!1);return this.expect("EOF"),this.settings.globalGroup||this.gullet.endGroup(),e}finally{this.gullet.endGroups()}}subparse(e){var r=this.nextToken;this.consume(),this.gullet.pushToken(new So("}")),this.gullet.pushTokens(e);var n=this.parseExpression(!1);return this.expect("}"),this.nextToken=r,n}parseExpression(e,r){for(var n=[];;){this.mode==="math"&&this.consumeSpaces();var i=this.fetch();if(t.endOfExpression.indexOf(i.text)!==-1||r&&i.text===r||e&&hh[i.text]&&hh[i.text].infix)break;var a=this.parseAtom(r);if(a){if(a.type==="internal")continue}else break;n.push(a)}return this.mode==="text"&&this.formLigatures(n),this.handleInfixNodes(n)}handleInfixNodes(e){for(var r=-1,n,i=0;i=0&&this.settings.reportNonstrict("unicodeTextInMathMode",'Latin-1/Unicode text character "'+r[0]+'" used in math mode',e);var l=An[this.mode][r].group,u=Xs.range(e),h;if(Mbe.hasOwnProperty(l)){var f=l;h={type:"atom",mode:this.mode,family:f,loc:u,text:r}}else h={type:l,mode:this.mode,loc:u,text:r};s=h}else if(r.charCodeAt(0)>=128)this.settings.strict&&($z(r.charCodeAt(0))?this.mode==="math"&&this.settings.reportNonstrict("unicodeTextInMathMode",'Unicode text character "'+r[0]+'" used in math mode',e):this.settings.reportNonstrict("unknownSymbol",'Unrecognized Unicode character "'+r[0]+'"'+(" ("+r.charCodeAt(0)+")"),e)),s={type:"textord",mode:"text",loc:Xs.range(e),text:r};else return null;if(this.consume(),a)for(var d=0;d{e instanceof Element&&e.tagName==="A"&&e.hasAttribute("target")&&e.setAttribute(t,e.getAttribute("target")??"")}),ch.addHook("afterSanitizeAttributes",e=>{e instanceof Element&&e.tagName==="A"&&e.hasAttribute(t)&&(e.setAttribute("target",e.getAttribute(t)??""),e.removeAttribute(t),e.getAttribute("target")==="_blank"&&e.setAttribute("rel","noopener"))})}var nd,Y4e,X4e,BG,OG,Tr,K4e,Q4e,Z4e,J4e,FG,e3e,fr,t3e,r3e,ec,J7,n3e,i3e,PG,eA,pi,id,mh,Ze,gr=N(()=>{"use strict";u7();nd=//gi,Y4e=o(t=>t?FG(t).replace(/\\n/g,"#br#").split("#br#"):[""],"getRows"),X4e=(()=>{let t=!1;return()=>{t||(j4e(),t=!0)}})();o(j4e,"setupDompurifyHooks");BG=o(t=>(X4e(),ch.sanitize(t)),"removeScript"),OG=o((t,e)=>{if(e.flowchart?.htmlLabels!==!1){let r=e.securityLevel;r==="antiscript"||r==="strict"?t=BG(t):r!=="loose"&&(t=FG(t),t=t.replace(//g,">"),t=t.replace(/=/g,"="),t=J4e(t))}return t},"sanitizeMore"),Tr=o((t,e)=>t&&(e.dompurifyConfig?t=ch.sanitize(OG(t,e),e.dompurifyConfig).toString():t=ch.sanitize(OG(t,e),{FORBID_TAGS:["style"]}).toString(),t),"sanitizeText"),K4e=o((t,e)=>typeof t=="string"?Tr(t,e):t.flat().map(r=>Tr(r,e)),"sanitizeTextOrArray"),Q4e=o(t=>nd.test(t),"hasBreaks"),Z4e=o(t=>t.split(nd),"splitBreaks"),J4e=o(t=>t.replace(/#br#/g,"
"),"placeholderToBreak"),FG=o(t=>t.replace(nd,"#br#"),"breakToPlaceholder"),e3e=o(t=>{let e="";return t&&(e=window.location.protocol+"//"+window.location.host+window.location.pathname+window.location.search,e=e.replaceAll(/\(/g,"\\("),e=e.replaceAll(/\)/g,"\\)")),e},"getUrl"),fr=o(t=>!(t===!1||["false","null","0"].includes(String(t).trim().toLowerCase())),"evaluate"),t3e=o(function(...t){let e=t.filter(r=>!isNaN(r));return Math.max(...e)},"getMax"),r3e=o(function(...t){let e=t.filter(r=>!isNaN(r));return Math.min(...e)},"getMin"),ec=o(function(t){let e=t.split(/(,)/),r=[];for(let n=0;n0&&n+1Math.max(0,t.split(e).length-1),"countOccurrence"),n3e=o((t,e)=>{let r=J7(t,"~"),n=J7(e,"~");return r===1&&n===1},"shouldCombineSets"),i3e=o(t=>{let e=J7(t,"~"),r=!1;if(e<=1)return t;e%2!==0&&t.startsWith("~")&&(t=t.substring(1),r=!0);let n=[...t],i=n.indexOf("~"),a=n.lastIndexOf("~");for(;i!==-1&&a!==-1&&i!==a;)n[i]="<",n[a]=">",i=n.indexOf("~"),a=n.lastIndexOf("~");return r&&n.unshift("~"),n.join("")},"processSet"),PG=o(()=>window.MathMLElement!==void 0,"isMathMLSupported"),eA=/\$\$(.*)\$\$/g,pi=o(t=>(t.match(eA)?.length??0)>0,"hasKatex"),id=o(async(t,e)=>{t=await mh(t,e);let r=document.createElement("div");r.innerHTML=t,r.id="katex-temp",r.style.visibility="hidden",r.style.position="absolute",r.style.top="0",document.querySelector("body")?.insertAdjacentElement("beforeend",r);let i={width:r.clientWidth,height:r.clientHeight};return r.remove(),i},"calculateMathMLDimensions"),mh=o(async(t,e)=>{if(!pi(t))return t;if(!(PG()||e.legacyMathML||e.forceLegacyMathML))return t.replace(eA,"MathML is unsupported in this environment.");let{default:r}=await Promise.resolve().then(()=>(IG(),MG)),n=e.forceLegacyMathML||!PG()&&e.legacyMathML?"htmlAndMathml":"mathml";return t.split(nd).map(i=>pi(i)?`
${i}
`:`
${i}
`).join("").replace(eA,(i,a)=>r.renderToString(a,{throwOnError:!0,displayMode:!0,output:n}).replace(/\n/g," ").replace(//g,""))},"renderKatex"),Ze={getRows:Y4e,sanitizeText:Tr,sanitizeTextOrArray:K4e,hasBreaks:Q4e,splitBreaks:Z4e,lineBreakRegex:nd,removeScript:BG,getUrl:e3e,evaluate:fr,getMax:t3e,getMin:r3e}});var a3e,s3e,vn,Ao,Ei=N(()=>{"use strict";vt();a3e=o(function(t,e){for(let r of e)t.attr(r[0],r[1])},"d3Attrs"),s3e=o(function(t,e,r){let n=new Map;return r?(n.set("width","100%"),n.set("style",`max-width: ${e}px;`)):(n.set("height",t),n.set("width",e)),n},"calculateSvgSizeAttrs"),vn=o(function(t,e,r,n){let i=s3e(e,r,n);a3e(t,i)},"configureSvgSize"),Ao=o(function(t,e,r,n){let i=e.node().getBBox(),a=i.width,s=i.height;Y.info(`SVG bounds: ${a}x${s}`,i);let l=0,u=0;Y.info(`Graph bounds: ${l}x${u}`,t),l=a+r*2,u=s+r*2,Y.info(`Calculated bounds: ${l}x${u}`),vn(e,u,l,n);let h=`${i.x-r} ${i.y-r} ${i.width+2*r} ${i.height+2*r}`;e.attr("viewBox",h)},"setupGraphViewbox")});var S3,o3e,$G,zG,tA=N(()=>{"use strict";vt();S3={},o3e=o((t,e,r)=>{let n="";return t in S3&&S3[t]?n=S3[t](r):Y.warn(`No theme found for ${t}`),` & { + font-family: ${r.fontFamily}; + font-size: ${r.fontSize}; + fill: ${r.textColor} + } + @keyframes edge-animation-frame { + from { + stroke-dashoffset: 0; + } + } + @keyframes dash { + to { + stroke-dashoffset: 0; + } + } + & .edge-animation-slow { + stroke-dasharray: 9,5 !important; + stroke-dashoffset: 900; + animation: dash 50s linear infinite; + stroke-linecap: round; + } + & .edge-animation-fast { + stroke-dasharray: 9,5 !important; + stroke-dashoffset: 900; + animation: dash 20s linear infinite; + stroke-linecap: round; + } + /* Classes common for multiple diagrams */ + + & .error-icon { + fill: ${r.errorBkgColor}; + } + & .error-text { + fill: ${r.errorTextColor}; + stroke: ${r.errorTextColor}; + } + + & .edge-thickness-normal { + stroke-width: 1px; + } + & .edge-thickness-thick { + stroke-width: 3.5px + } + & .edge-pattern-solid { + stroke-dasharray: 0; + } + & .edge-thickness-invisible { + stroke-width: 0; + fill: none; + } + & .edge-pattern-dashed{ + stroke-dasharray: 3; + } + .edge-pattern-dotted { + stroke-dasharray: 2; + } + + & .marker { + fill: ${r.lineColor}; + stroke: ${r.lineColor}; + } + & .marker.cross { + stroke: ${r.lineColor}; + } + + & svg { + font-family: ${r.fontFamily}; + font-size: ${r.fontSize}; + } + & p { + margin: 0 + } + + ${n} + + ${e} +`},"getStyles"),$G=o((t,e)=>{e!==void 0&&(S3[t]=e)},"addStylesForDiagram"),zG=o3e});var qy={};hr(qy,{clear:()=>Ar,getAccDescription:()=>Mr,getAccTitle:()=>Rr,getDiagramTitle:()=>Ir,setAccDescription:()=>Nr,setAccTitle:()=>Lr,setDiagramTitle:()=>$r});var rA,nA,iA,aA,Ar,Lr,Rr,Nr,Mr,$r,Ir,mi=N(()=>{"use strict";gr();ji();rA="",nA="",iA="",aA=o(t=>Tr(t,cr()),"sanitizeText"),Ar=o(()=>{rA="",iA="",nA=""},"clear"),Lr=o(t=>{rA=aA(t).replace(/^\s+/g,"")},"setAccTitle"),Rr=o(()=>rA,"getAccTitle"),Nr=o(t=>{iA=aA(t).replace(/\n\s+/g,` +`)},"setAccDescription"),Mr=o(()=>iA,"getAccDescription"),$r=o(t=>{nA=aA(t)},"setDiagramTitle"),Ir=o(()=>nA,"getDiagramTitle")});var GG,l3e,me,Yy,A3,Xy,oA,c3e,C3,ad,jy,sA,zt=N(()=>{"use strict";Xf();vt();ji();gr();Ei();tA();mi();GG=Y,l3e=wy,me=cr,Yy=X4,A3=lh,Xy=o(t=>Tr(t,me()),"sanitizeText"),oA=Ao,c3e=o(()=>qy,"getCommonDb"),C3={},ad=o((t,e,r)=>{C3[t]&&GG.warn(`Diagram with id ${t} already registered. Overwriting.`),C3[t]=e,r&&FC(t,r),$G(t,e.styles),e.injectUtils?.(GG,l3e,me,Xy,oA,c3e(),()=>{})},"registerDiagram"),jy=o(t=>{if(t in C3)return C3[t];throw new sA(t)},"getDiagram"),sA=class extends Error{static{o(this,"DiagramNotFoundError")}constructor(e){super(`Diagram ${e} not found.`)}}});var ul,gh,Ja,cl,tc,Ky,lA,cA,_3,D3,VG,u3e,h3e,f3e,d3e,p3e,m3e,g3e,y3e,v3e,x3e,b3e,w3e,T3e,k3e,E3e,S3e,C3e,UG,A3e,_3e,HG,D3e,L3e,R3e,N3e,yh,M3e,I3e,O3e,P3e,B3e,Qy,uA=N(()=>{"use strict";zt();gr();mi();ul=[],gh=[""],Ja="global",cl="",tc=[{alias:"global",label:{text:"global"},type:{text:"global"},tags:null,link:null,parentBoundary:""}],Ky=[],lA="",cA=!1,_3=4,D3=2,u3e=o(function(){return VG},"getC4Type"),h3e=o(function(t){VG=Tr(t,me())},"setC4Type"),f3e=o(function(t,e,r,n,i,a,s,l,u){if(t==null||e===void 0||e===null||r===void 0||r===null||n===void 0||n===null)return;let h={},f=Ky.find(d=>d.from===e&&d.to===r);if(f?h=f:Ky.push(h),h.type=t,h.from=e,h.to=r,h.label={text:n},i==null)h.techn={text:""};else if(typeof i=="object"){let[d,p]=Object.entries(i)[0];h[d]={text:p}}else h.techn={text:i};if(a==null)h.descr={text:""};else if(typeof a=="object"){let[d,p]=Object.entries(a)[0];h[d]={text:p}}else h.descr={text:a};if(typeof s=="object"){let[d,p]=Object.entries(s)[0];h[d]=p}else h.sprite=s;if(typeof l=="object"){let[d,p]=Object.entries(l)[0];h[d]=p}else h.tags=l;if(typeof u=="object"){let[d,p]=Object.entries(u)[0];h[d]=p}else h.link=u;h.wrap=yh()},"addRel"),d3e=o(function(t,e,r,n,i,a,s){if(e===null||r===null)return;let l={},u=ul.find(h=>h.alias===e);if(u&&e===u.alias?l=u:(l.alias=e,ul.push(l)),r==null?l.label={text:""}:l.label={text:r},n==null)l.descr={text:""};else if(typeof n=="object"){let[h,f]=Object.entries(n)[0];l[h]={text:f}}else l.descr={text:n};if(typeof i=="object"){let[h,f]=Object.entries(i)[0];l[h]=f}else l.sprite=i;if(typeof a=="object"){let[h,f]=Object.entries(a)[0];l[h]=f}else l.tags=a;if(typeof s=="object"){let[h,f]=Object.entries(s)[0];l[h]=f}else l.link=s;l.typeC4Shape={text:t},l.parentBoundary=Ja,l.wrap=yh()},"addPersonOrSystem"),p3e=o(function(t,e,r,n,i,a,s,l){if(e===null||r===null)return;let u={},h=ul.find(f=>f.alias===e);if(h&&e===h.alias?u=h:(u.alias=e,ul.push(u)),r==null?u.label={text:""}:u.label={text:r},n==null)u.techn={text:""};else if(typeof n=="object"){let[f,d]=Object.entries(n)[0];u[f]={text:d}}else u.techn={text:n};if(i==null)u.descr={text:""};else if(typeof i=="object"){let[f,d]=Object.entries(i)[0];u[f]={text:d}}else u.descr={text:i};if(typeof a=="object"){let[f,d]=Object.entries(a)[0];u[f]=d}else u.sprite=a;if(typeof s=="object"){let[f,d]=Object.entries(s)[0];u[f]=d}else u.tags=s;if(typeof l=="object"){let[f,d]=Object.entries(l)[0];u[f]=d}else u.link=l;u.wrap=yh(),u.typeC4Shape={text:t},u.parentBoundary=Ja},"addContainer"),m3e=o(function(t,e,r,n,i,a,s,l){if(e===null||r===null)return;let u={},h=ul.find(f=>f.alias===e);if(h&&e===h.alias?u=h:(u.alias=e,ul.push(u)),r==null?u.label={text:""}:u.label={text:r},n==null)u.techn={text:""};else if(typeof n=="object"){let[f,d]=Object.entries(n)[0];u[f]={text:d}}else u.techn={text:n};if(i==null)u.descr={text:""};else if(typeof i=="object"){let[f,d]=Object.entries(i)[0];u[f]={text:d}}else u.descr={text:i};if(typeof a=="object"){let[f,d]=Object.entries(a)[0];u[f]=d}else u.sprite=a;if(typeof s=="object"){let[f,d]=Object.entries(s)[0];u[f]=d}else u.tags=s;if(typeof l=="object"){let[f,d]=Object.entries(l)[0];u[f]=d}else u.link=l;u.wrap=yh(),u.typeC4Shape={text:t},u.parentBoundary=Ja},"addComponent"),g3e=o(function(t,e,r,n,i){if(t===null||e===null)return;let a={},s=tc.find(l=>l.alias===t);if(s&&t===s.alias?a=s:(a.alias=t,tc.push(a)),e==null?a.label={text:""}:a.label={text:e},r==null)a.type={text:"system"};else if(typeof r=="object"){let[l,u]=Object.entries(r)[0];a[l]={text:u}}else a.type={text:r};if(typeof n=="object"){let[l,u]=Object.entries(n)[0];a[l]=u}else a.tags=n;if(typeof i=="object"){let[l,u]=Object.entries(i)[0];a[l]=u}else a.link=i;a.parentBoundary=Ja,a.wrap=yh(),cl=Ja,Ja=t,gh.push(cl)},"addPersonOrSystemBoundary"),y3e=o(function(t,e,r,n,i){if(t===null||e===null)return;let a={},s=tc.find(l=>l.alias===t);if(s&&t===s.alias?a=s:(a.alias=t,tc.push(a)),e==null?a.label={text:""}:a.label={text:e},r==null)a.type={text:"container"};else if(typeof r=="object"){let[l,u]=Object.entries(r)[0];a[l]={text:u}}else a.type={text:r};if(typeof n=="object"){let[l,u]=Object.entries(n)[0];a[l]=u}else a.tags=n;if(typeof i=="object"){let[l,u]=Object.entries(i)[0];a[l]=u}else a.link=i;a.parentBoundary=Ja,a.wrap=yh(),cl=Ja,Ja=t,gh.push(cl)},"addContainerBoundary"),v3e=o(function(t,e,r,n,i,a,s,l){if(e===null||r===null)return;let u={},h=tc.find(f=>f.alias===e);if(h&&e===h.alias?u=h:(u.alias=e,tc.push(u)),r==null?u.label={text:""}:u.label={text:r},n==null)u.type={text:"node"};else if(typeof n=="object"){let[f,d]=Object.entries(n)[0];u[f]={text:d}}else u.type={text:n};if(i==null)u.descr={text:""};else if(typeof i=="object"){let[f,d]=Object.entries(i)[0];u[f]={text:d}}else u.descr={text:i};if(typeof s=="object"){let[f,d]=Object.entries(s)[0];u[f]=d}else u.tags=s;if(typeof l=="object"){let[f,d]=Object.entries(l)[0];u[f]=d}else u.link=l;u.nodeType=t,u.parentBoundary=Ja,u.wrap=yh(),cl=Ja,Ja=e,gh.push(cl)},"addDeploymentNode"),x3e=o(function(){Ja=cl,gh.pop(),cl=gh.pop(),gh.push(cl)},"popBoundaryParseStack"),b3e=o(function(t,e,r,n,i,a,s,l,u,h,f){let d=ul.find(p=>p.alias===e);if(!(d===void 0&&(d=tc.find(p=>p.alias===e),d===void 0))){if(r!=null)if(typeof r=="object"){let[p,m]=Object.entries(r)[0];d[p]=m}else d.bgColor=r;if(n!=null)if(typeof n=="object"){let[p,m]=Object.entries(n)[0];d[p]=m}else d.fontColor=n;if(i!=null)if(typeof i=="object"){let[p,m]=Object.entries(i)[0];d[p]=m}else d.borderColor=i;if(a!=null)if(typeof a=="object"){let[p,m]=Object.entries(a)[0];d[p]=m}else d.shadowing=a;if(s!=null)if(typeof s=="object"){let[p,m]=Object.entries(s)[0];d[p]=m}else d.shape=s;if(l!=null)if(typeof l=="object"){let[p,m]=Object.entries(l)[0];d[p]=m}else d.sprite=l;if(u!=null)if(typeof u=="object"){let[p,m]=Object.entries(u)[0];d[p]=m}else d.techn=u;if(h!=null)if(typeof h=="object"){let[p,m]=Object.entries(h)[0];d[p]=m}else d.legendText=h;if(f!=null)if(typeof f=="object"){let[p,m]=Object.entries(f)[0];d[p]=m}else d.legendSprite=f}},"updateElStyle"),w3e=o(function(t,e,r,n,i,a,s){let l=Ky.find(u=>u.from===e&&u.to===r);if(l!==void 0){if(n!=null)if(typeof n=="object"){let[u,h]=Object.entries(n)[0];l[u]=h}else l.textColor=n;if(i!=null)if(typeof i=="object"){let[u,h]=Object.entries(i)[0];l[u]=h}else l.lineColor=i;if(a!=null)if(typeof a=="object"){let[u,h]=Object.entries(a)[0];l[u]=parseInt(h)}else l.offsetX=parseInt(a);if(s!=null)if(typeof s=="object"){let[u,h]=Object.entries(s)[0];l[u]=parseInt(h)}else l.offsetY=parseInt(s)}},"updateRelStyle"),T3e=o(function(t,e,r){let n=_3,i=D3;if(typeof e=="object"){let a=Object.values(e)[0];n=parseInt(a)}else n=parseInt(e);if(typeof r=="object"){let a=Object.values(r)[0];i=parseInt(a)}else i=parseInt(r);n>=1&&(_3=n),i>=1&&(D3=i)},"updateLayoutConfig"),k3e=o(function(){return _3},"getC4ShapeInRow"),E3e=o(function(){return D3},"getC4BoundaryInRow"),S3e=o(function(){return Ja},"getCurrentBoundaryParse"),C3e=o(function(){return cl},"getParentBoundaryParse"),UG=o(function(t){return t==null?ul:ul.filter(e=>e.parentBoundary===t)},"getC4ShapeArray"),A3e=o(function(t){return ul.find(e=>e.alias===t)},"getC4Shape"),_3e=o(function(t){return Object.keys(UG(t))},"getC4ShapeKeys"),HG=o(function(t){return t==null?tc:tc.filter(e=>e.parentBoundary===t)},"getBoundaries"),D3e=HG,L3e=o(function(){return Ky},"getRels"),R3e=o(function(){return lA},"getTitle"),N3e=o(function(t){cA=t},"setWrap"),yh=o(function(){return cA},"autoWrap"),M3e=o(function(){ul=[],tc=[{alias:"global",label:{text:"global"},type:{text:"global"},tags:null,link:null,parentBoundary:""}],cl="",Ja="global",gh=[""],Ky=[],gh=[""],lA="",cA=!1,_3=4,D3=2},"clear"),I3e={SOLID:0,DOTTED:1,NOTE:2,SOLID_CROSS:3,DOTTED_CROSS:4,SOLID_OPEN:5,DOTTED_OPEN:6,LOOP_START:10,LOOP_END:11,ALT_START:12,ALT_ELSE:13,ALT_END:14,OPT_START:15,OPT_END:16,ACTIVE_START:17,ACTIVE_END:18,PAR_START:19,PAR_AND:20,PAR_END:21,RECT_START:22,RECT_END:23,SOLID_POINT:24,DOTTED_POINT:25},O3e={FILLED:0,OPEN:1},P3e={LEFTOF:0,RIGHTOF:1,OVER:2},B3e=o(function(t){lA=Tr(t,me())},"setTitle"),Qy={addPersonOrSystem:d3e,addPersonOrSystemBoundary:g3e,addContainer:p3e,addContainerBoundary:y3e,addComponent:m3e,addDeploymentNode:v3e,popBoundaryParseStack:x3e,addRel:f3e,updateElStyle:b3e,updateRelStyle:w3e,updateLayoutConfig:T3e,autoWrap:yh,setWrap:N3e,getC4ShapeArray:UG,getC4Shape:A3e,getC4ShapeKeys:_3e,getBoundaries:HG,getBoundarys:D3e,getCurrentBoundaryParse:S3e,getParentBoundaryParse:C3e,getRels:L3e,getTitle:R3e,getC4Type:u3e,getC4ShapeInRow:k3e,getC4BoundaryInRow:E3e,setAccTitle:Lr,getAccTitle:Rr,getAccDescription:Mr,setAccDescription:Nr,getConfig:o(()=>me().c4,"getConfig"),clear:M3e,LINETYPE:I3e,ARROWTYPE:O3e,PLACEMENT:P3e,setTitle:B3e,setC4Type:h3e}});function sd(t,e){return t==null||e==null?NaN:te?1:t>=e?0:NaN}var hA=N(()=>{"use strict";o(sd,"ascending")});function fA(t,e){return t==null||e==null?NaN:et?1:e>=t?0:NaN}var WG=N(()=>{"use strict";o(fA,"descending")});function od(t){let e,r,n;t.length!==2?(e=sd,r=o((l,u)=>sd(t(l),u),"compare2"),n=o((l,u)=>t(l)-u,"delta")):(e=t===sd||t===fA?t:F3e,r=t,n=t);function i(l,u,h=0,f=l.length){if(h>>1;r(l[d],u)<0?h=d+1:f=d}while(h>>1;r(l[d],u)<=0?h=d+1:f=d}while(hh&&n(l[d-1],u)>-n(l[d],u)?d-1:d}return o(s,"center"),{left:i,center:s,right:a}}function F3e(){return 0}var dA=N(()=>{"use strict";hA();WG();o(od,"bisector");o(F3e,"zero")});function pA(t){return t===null?NaN:+t}var qG=N(()=>{"use strict";o(pA,"number")});var YG,XG,$3e,z3e,mA,jG=N(()=>{"use strict";hA();dA();qG();YG=od(sd),XG=YG.right,$3e=YG.left,z3e=od(pA).center,mA=XG});function KG({_intern:t,_key:e},r){let n=e(r);return t.has(n)?t.get(n):r}function G3e({_intern:t,_key:e},r){let n=e(r);return t.has(n)?t.get(n):(t.set(n,r),r)}function V3e({_intern:t,_key:e},r){let n=e(r);return t.has(n)&&(r=t.get(n),t.delete(n)),r}function U3e(t){return t!==null&&typeof t=="object"?t.valueOf():t}var g0,QG=N(()=>{"use strict";g0=class extends Map{static{o(this,"InternMap")}constructor(e,r=U3e){if(super(),Object.defineProperties(this,{_intern:{value:new Map},_key:{value:r}}),e!=null)for(let[n,i]of e)this.set(n,i)}get(e){return super.get(KG(this,e))}has(e){return super.has(KG(this,e))}set(e,r){return super.set(G3e(this,e),r)}delete(e){return super.delete(V3e(this,e))}};o(KG,"intern_get");o(G3e,"intern_set");o(V3e,"intern_delete");o(U3e,"keyof")});function L3(t,e,r){let n=(e-t)/Math.max(0,r),i=Math.floor(Math.log10(n)),a=n/Math.pow(10,i),s=a>=H3e?10:a>=W3e?5:a>=q3e?2:1,l,u,h;return i<0?(h=Math.pow(10,-i)/s,l=Math.round(t*h),u=Math.round(e*h),l/he&&--u,h=-h):(h=Math.pow(10,i)*s,l=Math.round(t/h),u=Math.round(e/h),l*he&&--u),u0))return[];if(t===e)return[t];let n=e=i))return[];let l=a-i+1,u=new Array(l);if(n)if(s<0)for(let h=0;h{"use strict";H3e=Math.sqrt(50),W3e=Math.sqrt(10),q3e=Math.sqrt(2);o(L3,"tickSpec");o(R3,"ticks");o(Zy,"tickIncrement");o(y0,"tickStep")});function N3(t,e){let r;if(e===void 0)for(let n of t)n!=null&&(r=n)&&(r=n);else{let n=-1;for(let i of t)(i=e(i,++n,t))!=null&&(r=i)&&(r=i)}return r}var JG=N(()=>{"use strict";o(N3,"max")});function M3(t,e){let r;if(e===void 0)for(let n of t)n!=null&&(r>n||r===void 0&&n>=n)&&(r=n);else{let n=-1;for(let i of t)(i=e(i,++n,t))!=null&&(r>i||r===void 0&&i>=i)&&(r=i)}return r}var eV=N(()=>{"use strict";o(M3,"min")});function I3(t,e,r){t=+t,e=+e,r=(i=arguments.length)<2?(e=t,t=0,1):i<3?1:+r;for(var n=-1,i=Math.max(0,Math.ceil((e-t)/r))|0,a=new Array(i);++n{"use strict";o(I3,"range")});var vh=N(()=>{"use strict";jG();dA();JG();eV();tV();ZG();QG()});function gA(t){return t}var rV=N(()=>{"use strict";o(gA,"default")});function Y3e(t){return"translate("+t+",0)"}function X3e(t){return"translate(0,"+t+")"}function j3e(t){return e=>+t(e)}function K3e(t,e){return e=Math.max(0,t.bandwidth()-e*2)/2,t.round()&&(e=Math.round(e)),r=>+t(r)+e}function Q3e(){return!this.__axis}function iV(t,e){var r=[],n=null,i=null,a=6,s=6,l=3,u=typeof window<"u"&&window.devicePixelRatio>1?0:.5,h=t===P3||t===O3?-1:1,f=t===O3||t===yA?"x":"y",d=t===P3||t===vA?Y3e:X3e;function p(m){var g=n??(e.ticks?e.ticks.apply(e,r):e.domain()),y=i??(e.tickFormat?e.tickFormat.apply(e,r):gA),v=Math.max(a,0)+l,x=e.range(),b=+x[0]+u,w=+x[x.length-1]+u,C=(e.bandwidth?K3e:j3e)(e.copy(),u),T=m.selection?m.selection():m,E=T.selectAll(".domain").data([null]),A=T.selectAll(".tick").data(g,e).order(),S=A.exit(),_=A.enter().append("g").attr("class","tick"),I=A.select("line"),D=A.select("text");E=E.merge(E.enter().insert("path",".tick").attr("class","domain").attr("stroke","currentColor")),A=A.merge(_),I=I.merge(_.append("line").attr("stroke","currentColor").attr(f+"2",h*a)),D=D.merge(_.append("text").attr("fill","currentColor").attr(f,h*v).attr("dy",t===P3?"0em":t===vA?"0.71em":"0.32em")),m!==T&&(E=E.transition(m),A=A.transition(m),I=I.transition(m),D=D.transition(m),S=S.transition(m).attr("opacity",nV).attr("transform",function(k){return isFinite(k=C(k))?d(k+u):this.getAttribute("transform")}),_.attr("opacity",nV).attr("transform",function(k){var L=this.parentNode.__axis;return d((L&&isFinite(L=L(k))?L:C(k))+u)})),S.remove(),E.attr("d",t===O3||t===yA?s?"M"+h*s+","+b+"H"+u+"V"+w+"H"+h*s:"M"+u+","+b+"V"+w:s?"M"+b+","+h*s+"V"+u+"H"+w+"V"+h*s:"M"+b+","+u+"H"+w),A.attr("opacity",1).attr("transform",function(k){return d(C(k)+u)}),I.attr(f+"2",h*a),D.attr(f,h*v).text(y),T.filter(Q3e).attr("fill","none").attr("font-size",10).attr("font-family","sans-serif").attr("text-anchor",t===yA?"start":t===O3?"end":"middle"),T.each(function(){this.__axis=C})}return o(p,"axis"),p.scale=function(m){return arguments.length?(e=m,p):e},p.ticks=function(){return r=Array.from(arguments),p},p.tickArguments=function(m){return arguments.length?(r=m==null?[]:Array.from(m),p):r.slice()},p.tickValues=function(m){return arguments.length?(n=m==null?null:Array.from(m),p):n&&n.slice()},p.tickFormat=function(m){return arguments.length?(i=m,p):i},p.tickSize=function(m){return arguments.length?(a=s=+m,p):a},p.tickSizeInner=function(m){return arguments.length?(a=+m,p):a},p.tickSizeOuter=function(m){return arguments.length?(s=+m,p):s},p.tickPadding=function(m){return arguments.length?(l=+m,p):l},p.offset=function(m){return arguments.length?(u=+m,p):u},p}function xA(t){return iV(P3,t)}function bA(t){return iV(vA,t)}var P3,yA,vA,O3,nV,aV=N(()=>{"use strict";rV();P3=1,yA=2,vA=3,O3=4,nV=1e-6;o(Y3e,"translateX");o(X3e,"translateY");o(j3e,"number");o(K3e,"center");o(Q3e,"entering");o(iV,"axis");o(xA,"axisTop");o(bA,"axisBottom")});var sV=N(()=>{"use strict";aV()});function lV(){for(var t=0,e=arguments.length,r={},n;t=0&&(n=r.slice(i+1),r=r.slice(0,i)),r&&!e.hasOwnProperty(r))throw new Error("unknown type: "+r);return{type:r,name:n}})}function e5e(t,e){for(var r=0,n=t.length,i;r{"use strict";Z3e={value:o(()=>{},"value")};o(lV,"dispatch");o(B3,"Dispatch");o(J3e,"parseTypenames");B3.prototype=lV.prototype={constructor:B3,on:o(function(t,e){var r=this._,n=J3e(t+"",r),i,a=-1,s=n.length;if(arguments.length<2){for(;++a0)for(var r=new Array(i),n=0,i,a;n{"use strict";cV()});var F3,kA,EA=N(()=>{"use strict";F3="http://www.w3.org/1999/xhtml",kA={svg:"http://www.w3.org/2000/svg",xhtml:F3,xlink:"http://www.w3.org/1999/xlink",xml:"http://www.w3.org/XML/1998/namespace",xmlns:"http://www.w3.org/2000/xmlns/"}});function rc(t){var e=t+="",r=e.indexOf(":");return r>=0&&(e=t.slice(0,r))!=="xmlns"&&(t=t.slice(r+1)),kA.hasOwnProperty(e)?{space:kA[e],local:t}:t}var $3=N(()=>{"use strict";EA();o(rc,"default")});function t5e(t){return function(){var e=this.ownerDocument,r=this.namespaceURI;return r===F3&&e.documentElement.namespaceURI===F3?e.createElement(t):e.createElementNS(r,t)}}function r5e(t){return function(){return this.ownerDocument.createElementNS(t.space,t.local)}}function Jy(t){var e=rc(t);return(e.local?r5e:t5e)(e)}var SA=N(()=>{"use strict";$3();EA();o(t5e,"creatorInherit");o(r5e,"creatorFixed");o(Jy,"default")});function n5e(){}function xh(t){return t==null?n5e:function(){return this.querySelector(t)}}var z3=N(()=>{"use strict";o(n5e,"none");o(xh,"default")});function CA(t){typeof t!="function"&&(t=xh(t));for(var e=this._groups,r=e.length,n=new Array(r),i=0;i{"use strict";hl();z3();o(CA,"default")});function AA(t){return t==null?[]:Array.isArray(t)?t:Array.from(t)}var hV=N(()=>{"use strict";o(AA,"array")});function i5e(){return[]}function v0(t){return t==null?i5e:function(){return this.querySelectorAll(t)}}var _A=N(()=>{"use strict";o(i5e,"empty");o(v0,"default")});function a5e(t){return function(){return AA(t.apply(this,arguments))}}function DA(t){typeof t=="function"?t=a5e(t):t=v0(t);for(var e=this._groups,r=e.length,n=[],i=[],a=0;a{"use strict";hl();hV();_A();o(a5e,"arrayAll");o(DA,"default")});function x0(t){return function(){return this.matches(t)}}function G3(t){return function(e){return e.matches(t)}}var ev=N(()=>{"use strict";o(x0,"default");o(G3,"childMatcher")});function o5e(t){return function(){return s5e.call(this.children,t)}}function l5e(){return this.firstElementChild}function LA(t){return this.select(t==null?l5e:o5e(typeof t=="function"?t:G3(t)))}var s5e,dV=N(()=>{"use strict";ev();s5e=Array.prototype.find;o(o5e,"childFind");o(l5e,"childFirst");o(LA,"default")});function u5e(){return Array.from(this.children)}function h5e(t){return function(){return c5e.call(this.children,t)}}function RA(t){return this.selectAll(t==null?u5e:h5e(typeof t=="function"?t:G3(t)))}var c5e,pV=N(()=>{"use strict";ev();c5e=Array.prototype.filter;o(u5e,"children");o(h5e,"childrenFilter");o(RA,"default")});function NA(t){typeof t!="function"&&(t=x0(t));for(var e=this._groups,r=e.length,n=new Array(r),i=0;i{"use strict";hl();ev();o(NA,"default")});function tv(t){return new Array(t.length)}var MA=N(()=>{"use strict";o(tv,"default")});function IA(){return new oi(this._enter||this._groups.map(tv),this._parents)}function rv(t,e){this.ownerDocument=t.ownerDocument,this.namespaceURI=t.namespaceURI,this._next=null,this._parent=t,this.__data__=e}var OA=N(()=>{"use strict";MA();hl();o(IA,"default");o(rv,"EnterNode");rv.prototype={constructor:rv,appendChild:o(function(t){return this._parent.insertBefore(t,this._next)},"appendChild"),insertBefore:o(function(t,e){return this._parent.insertBefore(t,e)},"insertBefore"),querySelector:o(function(t){return this._parent.querySelector(t)},"querySelector"),querySelectorAll:o(function(t){return this._parent.querySelectorAll(t)},"querySelectorAll")}});function PA(t){return function(){return t}}var gV=N(()=>{"use strict";o(PA,"default")});function f5e(t,e,r,n,i,a){for(var s=0,l,u=e.length,h=a.length;s=w&&(w=b+1);!(T=v[w])&&++w{"use strict";hl();OA();gV();o(f5e,"bindIndex");o(d5e,"bindKey");o(p5e,"datum");o(BA,"default");o(m5e,"arraylike")});function FA(){return new oi(this._exit||this._groups.map(tv),this._parents)}var vV=N(()=>{"use strict";MA();hl();o(FA,"default")});function $A(t,e,r){var n=this.enter(),i=this,a=this.exit();return typeof t=="function"?(n=t(n),n&&(n=n.selection())):n=n.append(t+""),e!=null&&(i=e(i),i&&(i=i.selection())),r==null?a.remove():r(a),n&&i?n.merge(i).order():i}var xV=N(()=>{"use strict";o($A,"default")});function zA(t){for(var e=t.selection?t.selection():t,r=this._groups,n=e._groups,i=r.length,a=n.length,s=Math.min(i,a),l=new Array(i),u=0;u{"use strict";hl();o(zA,"default")});function GA(){for(var t=this._groups,e=-1,r=t.length;++e=0;)(s=n[i])&&(a&&s.compareDocumentPosition(a)^4&&a.parentNode.insertBefore(s,a),a=s);return this}var wV=N(()=>{"use strict";o(GA,"default")});function VA(t){t||(t=g5e);function e(d,p){return d&&p?t(d.__data__,p.__data__):!d-!p}o(e,"compareNode");for(var r=this._groups,n=r.length,i=new Array(n),a=0;ae?1:t>=e?0:NaN}var TV=N(()=>{"use strict";hl();o(VA,"default");o(g5e,"ascending")});function UA(){var t=arguments[0];return arguments[0]=this,t.apply(null,arguments),this}var kV=N(()=>{"use strict";o(UA,"default")});function HA(){return Array.from(this)}var EV=N(()=>{"use strict";o(HA,"default")});function WA(){for(var t=this._groups,e=0,r=t.length;e{"use strict";o(WA,"default")});function qA(){let t=0;for(let e of this)++t;return t}var CV=N(()=>{"use strict";o(qA,"default")});function YA(){return!this.node()}var AV=N(()=>{"use strict";o(YA,"default")});function XA(t){for(var e=this._groups,r=0,n=e.length;r{"use strict";o(XA,"default")});function y5e(t){return function(){this.removeAttribute(t)}}function v5e(t){return function(){this.removeAttributeNS(t.space,t.local)}}function x5e(t,e){return function(){this.setAttribute(t,e)}}function b5e(t,e){return function(){this.setAttributeNS(t.space,t.local,e)}}function w5e(t,e){return function(){var r=e.apply(this,arguments);r==null?this.removeAttribute(t):this.setAttribute(t,r)}}function T5e(t,e){return function(){var r=e.apply(this,arguments);r==null?this.removeAttributeNS(t.space,t.local):this.setAttributeNS(t.space,t.local,r)}}function jA(t,e){var r=rc(t);if(arguments.length<2){var n=this.node();return r.local?n.getAttributeNS(r.space,r.local):n.getAttribute(r)}return this.each((e==null?r.local?v5e:y5e:typeof e=="function"?r.local?T5e:w5e:r.local?b5e:x5e)(r,e))}var DV=N(()=>{"use strict";$3();o(y5e,"attrRemove");o(v5e,"attrRemoveNS");o(x5e,"attrConstant");o(b5e,"attrConstantNS");o(w5e,"attrFunction");o(T5e,"attrFunctionNS");o(jA,"default")});function nv(t){return t.ownerDocument&&t.ownerDocument.defaultView||t.document&&t||t.defaultView}var KA=N(()=>{"use strict";o(nv,"default")});function k5e(t){return function(){this.style.removeProperty(t)}}function E5e(t,e,r){return function(){this.style.setProperty(t,e,r)}}function S5e(t,e,r){return function(){var n=e.apply(this,arguments);n==null?this.style.removeProperty(t):this.style.setProperty(t,n,r)}}function QA(t,e,r){return arguments.length>1?this.each((e==null?k5e:typeof e=="function"?S5e:E5e)(t,e,r??"")):bh(this.node(),t)}function bh(t,e){return t.style.getPropertyValue(e)||nv(t).getComputedStyle(t,null).getPropertyValue(e)}var ZA=N(()=>{"use strict";KA();o(k5e,"styleRemove");o(E5e,"styleConstant");o(S5e,"styleFunction");o(QA,"default");o(bh,"styleValue")});function C5e(t){return function(){delete this[t]}}function A5e(t,e){return function(){this[t]=e}}function _5e(t,e){return function(){var r=e.apply(this,arguments);r==null?delete this[t]:this[t]=r}}function JA(t,e){return arguments.length>1?this.each((e==null?C5e:typeof e=="function"?_5e:A5e)(t,e)):this.node()[t]}var LV=N(()=>{"use strict";o(C5e,"propertyRemove");o(A5e,"propertyConstant");o(_5e,"propertyFunction");o(JA,"default")});function RV(t){return t.trim().split(/^|\s+/)}function e8(t){return t.classList||new NV(t)}function NV(t){this._node=t,this._names=RV(t.getAttribute("class")||"")}function MV(t,e){for(var r=e8(t),n=-1,i=e.length;++n{"use strict";o(RV,"classArray");o(e8,"classList");o(NV,"ClassList");NV.prototype={add:o(function(t){var e=this._names.indexOf(t);e<0&&(this._names.push(t),this._node.setAttribute("class",this._names.join(" ")))},"add"),remove:o(function(t){var e=this._names.indexOf(t);e>=0&&(this._names.splice(e,1),this._node.setAttribute("class",this._names.join(" ")))},"remove"),contains:o(function(t){return this._names.indexOf(t)>=0},"contains")};o(MV,"classedAdd");o(IV,"classedRemove");o(D5e,"classedTrue");o(L5e,"classedFalse");o(R5e,"classedFunction");o(t8,"default")});function N5e(){this.textContent=""}function M5e(t){return function(){this.textContent=t}}function I5e(t){return function(){var e=t.apply(this,arguments);this.textContent=e??""}}function r8(t){return arguments.length?this.each(t==null?N5e:(typeof t=="function"?I5e:M5e)(t)):this.node().textContent}var PV=N(()=>{"use strict";o(N5e,"textRemove");o(M5e,"textConstant");o(I5e,"textFunction");o(r8,"default")});function O5e(){this.innerHTML=""}function P5e(t){return function(){this.innerHTML=t}}function B5e(t){return function(){var e=t.apply(this,arguments);this.innerHTML=e??""}}function n8(t){return arguments.length?this.each(t==null?O5e:(typeof t=="function"?B5e:P5e)(t)):this.node().innerHTML}var BV=N(()=>{"use strict";o(O5e,"htmlRemove");o(P5e,"htmlConstant");o(B5e,"htmlFunction");o(n8,"default")});function F5e(){this.nextSibling&&this.parentNode.appendChild(this)}function i8(){return this.each(F5e)}var FV=N(()=>{"use strict";o(F5e,"raise");o(i8,"default")});function $5e(){this.previousSibling&&this.parentNode.insertBefore(this,this.parentNode.firstChild)}function a8(){return this.each($5e)}var $V=N(()=>{"use strict";o($5e,"lower");o(a8,"default")});function s8(t){var e=typeof t=="function"?t:Jy(t);return this.select(function(){return this.appendChild(e.apply(this,arguments))})}var zV=N(()=>{"use strict";SA();o(s8,"default")});function z5e(){return null}function o8(t,e){var r=typeof t=="function"?t:Jy(t),n=e==null?z5e:typeof e=="function"?e:xh(e);return this.select(function(){return this.insertBefore(r.apply(this,arguments),n.apply(this,arguments)||null)})}var GV=N(()=>{"use strict";SA();z3();o(z5e,"constantNull");o(o8,"default")});function G5e(){var t=this.parentNode;t&&t.removeChild(this)}function l8(){return this.each(G5e)}var VV=N(()=>{"use strict";o(G5e,"remove");o(l8,"default")});function V5e(){var t=this.cloneNode(!1),e=this.parentNode;return e?e.insertBefore(t,this.nextSibling):t}function U5e(){var t=this.cloneNode(!0),e=this.parentNode;return e?e.insertBefore(t,this.nextSibling):t}function c8(t){return this.select(t?U5e:V5e)}var UV=N(()=>{"use strict";o(V5e,"selection_cloneShallow");o(U5e,"selection_cloneDeep");o(c8,"default")});function u8(t){return arguments.length?this.property("__data__",t):this.node().__data__}var HV=N(()=>{"use strict";o(u8,"default")});function H5e(t){return function(e){t.call(this,e,this.__data__)}}function W5e(t){return t.trim().split(/^|\s+/).map(function(e){var r="",n=e.indexOf(".");return n>=0&&(r=e.slice(n+1),e=e.slice(0,n)),{type:e,name:r}})}function q5e(t){return function(){var e=this.__on;if(e){for(var r=0,n=-1,i=e.length,a;r{"use strict";o(H5e,"contextListener");o(W5e,"parseTypenames");o(q5e,"onRemove");o(Y5e,"onAdd");o(h8,"default")});function qV(t,e,r){var n=nv(t),i=n.CustomEvent;typeof i=="function"?i=new i(e,r):(i=n.document.createEvent("Event"),r?(i.initEvent(e,r.bubbles,r.cancelable),i.detail=r.detail):i.initEvent(e,!1,!1)),t.dispatchEvent(i)}function X5e(t,e){return function(){return qV(this,t,e)}}function j5e(t,e){return function(){return qV(this,t,e.apply(this,arguments))}}function f8(t,e){return this.each((typeof e=="function"?j5e:X5e)(t,e))}var YV=N(()=>{"use strict";KA();o(qV,"dispatchEvent");o(X5e,"dispatchConstant");o(j5e,"dispatchFunction");o(f8,"default")});function*d8(){for(var t=this._groups,e=0,r=t.length;e{"use strict";o(d8,"default")});function oi(t,e){this._groups=t,this._parents=e}function jV(){return new oi([[document.documentElement]],p8)}function K5e(){return this}var p8,hu,hl=N(()=>{"use strict";uV();fV();dV();pV();mV();yV();OA();vV();xV();bV();wV();TV();kV();EV();SV();CV();AV();_V();DV();ZA();LV();OV();PV();BV();FV();$V();zV();GV();VV();UV();HV();WV();YV();XV();p8=[null];o(oi,"Selection");o(jV,"selection");o(K5e,"selection_selection");oi.prototype=jV.prototype={constructor:oi,select:CA,selectAll:DA,selectChild:LA,selectChildren:RA,filter:NA,data:BA,enter:IA,exit:FA,join:$A,merge:zA,selection:K5e,order:GA,sort:VA,call:UA,nodes:HA,node:WA,size:qA,empty:YA,each:XA,attr:jA,style:QA,property:JA,classed:t8,text:r8,html:n8,raise:i8,lower:a8,append:s8,insert:o8,remove:l8,clone:c8,datum:u8,on:h8,dispatch:f8,[Symbol.iterator]:d8};hu=jV});function Ge(t){return typeof t=="string"?new oi([[document.querySelector(t)]],[document.documentElement]):new oi([[t]],p8)}var KV=N(()=>{"use strict";hl();o(Ge,"default")});var fl=N(()=>{"use strict";ev();$3();KV();hl();z3();_A();ZA()});var QV=N(()=>{"use strict"});function wh(t,e,r){t.prototype=e.prototype=r,r.constructor=t}function b0(t,e){var r=Object.create(t.prototype);for(var n in e)r[n]=e[n];return r}var m8=N(()=>{"use strict";o(wh,"default");o(b0,"extend")});function Th(){}function JV(){return this.rgb().formatHex()}function iwe(){return this.rgb().formatHex8()}function awe(){return sU(this).formatHsl()}function eU(){return this.rgb().formatRgb()}function pl(t){var e,r;return t=(t+"").trim().toLowerCase(),(e=Q5e.exec(t))?(r=e[1].length,e=parseInt(e[1],16),r===6?tU(e):r===3?new ua(e>>8&15|e>>4&240,e>>4&15|e&240,(e&15)<<4|e&15,1):r===8?V3(e>>24&255,e>>16&255,e>>8&255,(e&255)/255):r===4?V3(e>>12&15|e>>8&240,e>>8&15|e>>4&240,e>>4&15|e&240,((e&15)<<4|e&15)/255):null):(e=Z5e.exec(t))?new ua(e[1],e[2],e[3],1):(e=J5e.exec(t))?new ua(e[1]*255/100,e[2]*255/100,e[3]*255/100,1):(e=ewe.exec(t))?V3(e[1],e[2],e[3],e[4]):(e=twe.exec(t))?V3(e[1]*255/100,e[2]*255/100,e[3]*255/100,e[4]):(e=rwe.exec(t))?iU(e[1],e[2]/100,e[3]/100,1):(e=nwe.exec(t))?iU(e[1],e[2]/100,e[3]/100,e[4]):ZV.hasOwnProperty(t)?tU(ZV[t]):t==="transparent"?new ua(NaN,NaN,NaN,0):null}function tU(t){return new ua(t>>16&255,t>>8&255,t&255,1)}function V3(t,e,r,n){return n<=0&&(t=e=r=NaN),new ua(t,e,r,n)}function y8(t){return t instanceof Th||(t=pl(t)),t?(t=t.rgb(),new ua(t.r,t.g,t.b,t.opacity)):new ua}function T0(t,e,r,n){return arguments.length===1?y8(t):new ua(t,e,r,n??1)}function ua(t,e,r,n){this.r=+t,this.g=+e,this.b=+r,this.opacity=+n}function rU(){return`#${ld(this.r)}${ld(this.g)}${ld(this.b)}`}function swe(){return`#${ld(this.r)}${ld(this.g)}${ld(this.b)}${ld((isNaN(this.opacity)?1:this.opacity)*255)}`}function nU(){let t=W3(this.opacity);return`${t===1?"rgb(":"rgba("}${cd(this.r)}, ${cd(this.g)}, ${cd(this.b)}${t===1?")":`, ${t})`}`}function W3(t){return isNaN(t)?1:Math.max(0,Math.min(1,t))}function cd(t){return Math.max(0,Math.min(255,Math.round(t)||0))}function ld(t){return t=cd(t),(t<16?"0":"")+t.toString(16)}function iU(t,e,r,n){return n<=0?t=e=r=NaN:r<=0||r>=1?t=e=NaN:e<=0&&(t=NaN),new dl(t,e,r,n)}function sU(t){if(t instanceof dl)return new dl(t.h,t.s,t.l,t.opacity);if(t instanceof Th||(t=pl(t)),!t)return new dl;if(t instanceof dl)return t;t=t.rgb();var e=t.r/255,r=t.g/255,n=t.b/255,i=Math.min(e,r,n),a=Math.max(e,r,n),s=NaN,l=a-i,u=(a+i)/2;return l?(e===a?s=(r-n)/l+(r0&&u<1?0:s,new dl(s,l,u,t.opacity)}function oU(t,e,r,n){return arguments.length===1?sU(t):new dl(t,e,r,n??1)}function dl(t,e,r,n){this.h=+t,this.s=+e,this.l=+r,this.opacity=+n}function aU(t){return t=(t||0)%360,t<0?t+360:t}function U3(t){return Math.max(0,Math.min(1,t||0))}function g8(t,e,r){return(t<60?e+(r-e)*t/60:t<180?r:t<240?e+(r-e)*(240-t)/60:e)*255}var iv,H3,w0,av,nc,Q5e,Z5e,J5e,ewe,twe,rwe,nwe,ZV,v8=N(()=>{"use strict";m8();o(Th,"Color");iv=.7,H3=1/iv,w0="\\s*([+-]?\\d+)\\s*",av="\\s*([+-]?(?:\\d*\\.)?\\d+(?:[eE][+-]?\\d+)?)\\s*",nc="\\s*([+-]?(?:\\d*\\.)?\\d+(?:[eE][+-]?\\d+)?)%\\s*",Q5e=/^#([0-9a-f]{3,8})$/,Z5e=new RegExp(`^rgb\\(${w0},${w0},${w0}\\)$`),J5e=new RegExp(`^rgb\\(${nc},${nc},${nc}\\)$`),ewe=new RegExp(`^rgba\\(${w0},${w0},${w0},${av}\\)$`),twe=new RegExp(`^rgba\\(${nc},${nc},${nc},${av}\\)$`),rwe=new RegExp(`^hsl\\(${av},${nc},${nc}\\)$`),nwe=new RegExp(`^hsla\\(${av},${nc},${nc},${av}\\)$`),ZV={aliceblue:15792383,antiquewhite:16444375,aqua:65535,aquamarine:8388564,azure:15794175,beige:16119260,bisque:16770244,black:0,blanchedalmond:16772045,blue:255,blueviolet:9055202,brown:10824234,burlywood:14596231,cadetblue:6266528,chartreuse:8388352,chocolate:13789470,coral:16744272,cornflowerblue:6591981,cornsilk:16775388,crimson:14423100,cyan:65535,darkblue:139,darkcyan:35723,darkgoldenrod:12092939,darkgray:11119017,darkgreen:25600,darkgrey:11119017,darkkhaki:12433259,darkmagenta:9109643,darkolivegreen:5597999,darkorange:16747520,darkorchid:10040012,darkred:9109504,darksalmon:15308410,darkseagreen:9419919,darkslateblue:4734347,darkslategray:3100495,darkslategrey:3100495,darkturquoise:52945,darkviolet:9699539,deeppink:16716947,deepskyblue:49151,dimgray:6908265,dimgrey:6908265,dodgerblue:2003199,firebrick:11674146,floralwhite:16775920,forestgreen:2263842,fuchsia:16711935,gainsboro:14474460,ghostwhite:16316671,gold:16766720,goldenrod:14329120,gray:8421504,green:32768,greenyellow:11403055,grey:8421504,honeydew:15794160,hotpink:16738740,indianred:13458524,indigo:4915330,ivory:16777200,khaki:15787660,lavender:15132410,lavenderblush:16773365,lawngreen:8190976,lemonchiffon:16775885,lightblue:11393254,lightcoral:15761536,lightcyan:14745599,lightgoldenrodyellow:16448210,lightgray:13882323,lightgreen:9498256,lightgrey:13882323,lightpink:16758465,lightsalmon:16752762,lightseagreen:2142890,lightskyblue:8900346,lightslategray:7833753,lightslategrey:7833753,lightsteelblue:11584734,lightyellow:16777184,lime:65280,limegreen:3329330,linen:16445670,magenta:16711935,maroon:8388608,mediumaquamarine:6737322,mediumblue:205,mediumorchid:12211667,mediumpurple:9662683,mediumseagreen:3978097,mediumslateblue:8087790,mediumspringgreen:64154,mediumturquoise:4772300,mediumvioletred:13047173,midnightblue:1644912,mintcream:16121850,mistyrose:16770273,moccasin:16770229,navajowhite:16768685,navy:128,oldlace:16643558,olive:8421376,olivedrab:7048739,orange:16753920,orangered:16729344,orchid:14315734,palegoldenrod:15657130,palegreen:10025880,paleturquoise:11529966,palevioletred:14381203,papayawhip:16773077,peachpuff:16767673,peru:13468991,pink:16761035,plum:14524637,powderblue:11591910,purple:8388736,rebeccapurple:6697881,red:16711680,rosybrown:12357519,royalblue:4286945,saddlebrown:9127187,salmon:16416882,sandybrown:16032864,seagreen:3050327,seashell:16774638,sienna:10506797,silver:12632256,skyblue:8900331,slateblue:6970061,slategray:7372944,slategrey:7372944,snow:16775930,springgreen:65407,steelblue:4620980,tan:13808780,teal:32896,thistle:14204888,tomato:16737095,turquoise:4251856,violet:15631086,wheat:16113331,white:16777215,whitesmoke:16119285,yellow:16776960,yellowgreen:10145074};wh(Th,pl,{copy(t){return Object.assign(new this.constructor,this,t)},displayable(){return this.rgb().displayable()},hex:JV,formatHex:JV,formatHex8:iwe,formatHsl:awe,formatRgb:eU,toString:eU});o(JV,"color_formatHex");o(iwe,"color_formatHex8");o(awe,"color_formatHsl");o(eU,"color_formatRgb");o(pl,"color");o(tU,"rgbn");o(V3,"rgba");o(y8,"rgbConvert");o(T0,"rgb");o(ua,"Rgb");wh(ua,T0,b0(Th,{brighter(t){return t=t==null?H3:Math.pow(H3,t),new ua(this.r*t,this.g*t,this.b*t,this.opacity)},darker(t){return t=t==null?iv:Math.pow(iv,t),new ua(this.r*t,this.g*t,this.b*t,this.opacity)},rgb(){return this},clamp(){return new ua(cd(this.r),cd(this.g),cd(this.b),W3(this.opacity))},displayable(){return-.5<=this.r&&this.r<255.5&&-.5<=this.g&&this.g<255.5&&-.5<=this.b&&this.b<255.5&&0<=this.opacity&&this.opacity<=1},hex:rU,formatHex:rU,formatHex8:swe,formatRgb:nU,toString:nU}));o(rU,"rgb_formatHex");o(swe,"rgb_formatHex8");o(nU,"rgb_formatRgb");o(W3,"clampa");o(cd,"clampi");o(ld,"hex");o(iU,"hsla");o(sU,"hslConvert");o(oU,"hsl");o(dl,"Hsl");wh(dl,oU,b0(Th,{brighter(t){return t=t==null?H3:Math.pow(H3,t),new dl(this.h,this.s,this.l*t,this.opacity)},darker(t){return t=t==null?iv:Math.pow(iv,t),new dl(this.h,this.s,this.l*t,this.opacity)},rgb(){var t=this.h%360+(this.h<0)*360,e=isNaN(t)||isNaN(this.s)?0:this.s,r=this.l,n=r+(r<.5?r:1-r)*e,i=2*r-n;return new ua(g8(t>=240?t-240:t+120,i,n),g8(t,i,n),g8(t<120?t+240:t-120,i,n),this.opacity)},clamp(){return new dl(aU(this.h),U3(this.s),U3(this.l),W3(this.opacity))},displayable(){return(0<=this.s&&this.s<=1||isNaN(this.s))&&0<=this.l&&this.l<=1&&0<=this.opacity&&this.opacity<=1},formatHsl(){let t=W3(this.opacity);return`${t===1?"hsl(":"hsla("}${aU(this.h)}, ${U3(this.s)*100}%, ${U3(this.l)*100}%${t===1?")":`, ${t})`}`}}));o(aU,"clamph");o(U3,"clampt");o(g8,"hsl2rgb")});var lU,cU,uU=N(()=>{"use strict";lU=Math.PI/180,cU=180/Math.PI});function gU(t){if(t instanceof ic)return new ic(t.l,t.a,t.b,t.opacity);if(t instanceof fu)return yU(t);t instanceof ua||(t=y8(t));var e=T8(t.r),r=T8(t.g),n=T8(t.b),i=x8((.2225045*e+.7168786*r+.0606169*n)/fU),a,s;return e===r&&r===n?a=s=i:(a=x8((.4360747*e+.3850649*r+.1430804*n)/hU),s=x8((.0139322*e+.0971045*r+.7141733*n)/dU)),new ic(116*i-16,500*(a-i),200*(i-s),t.opacity)}function k8(t,e,r,n){return arguments.length===1?gU(t):new ic(t,e,r,n??1)}function ic(t,e,r,n){this.l=+t,this.a=+e,this.b=+r,this.opacity=+n}function x8(t){return t>owe?Math.pow(t,1/3):t/mU+pU}function b8(t){return t>k0?t*t*t:mU*(t-pU)}function w8(t){return 255*(t<=.0031308?12.92*t:1.055*Math.pow(t,1/2.4)-.055)}function T8(t){return(t/=255)<=.04045?t/12.92:Math.pow((t+.055)/1.055,2.4)}function lwe(t){if(t instanceof fu)return new fu(t.h,t.c,t.l,t.opacity);if(t instanceof ic||(t=gU(t)),t.a===0&&t.b===0)return new fu(NaN,0{"use strict";m8();v8();uU();q3=18,hU=.96422,fU=1,dU=.82521,pU=4/29,k0=6/29,mU=3*k0*k0,owe=k0*k0*k0;o(gU,"labConvert");o(k8,"lab");o(ic,"Lab");wh(ic,k8,b0(Th,{brighter(t){return new ic(this.l+q3*(t??1),this.a,this.b,this.opacity)},darker(t){return new ic(this.l-q3*(t??1),this.a,this.b,this.opacity)},rgb(){var t=(this.l+16)/116,e=isNaN(this.a)?t:t+this.a/500,r=isNaN(this.b)?t:t-this.b/200;return e=hU*b8(e),t=fU*b8(t),r=dU*b8(r),new ua(w8(3.1338561*e-1.6168667*t-.4906146*r),w8(-.9787684*e+1.9161415*t+.033454*r),w8(.0719453*e-.2289914*t+1.4052427*r),this.opacity)}}));o(x8,"xyz2lab");o(b8,"lab2xyz");o(w8,"lrgb2rgb");o(T8,"rgb2lrgb");o(lwe,"hclConvert");o(sv,"hcl");o(fu,"Hcl");o(yU,"hcl2lab");wh(fu,sv,b0(Th,{brighter(t){return new fu(this.h,this.c,this.l+q3*(t??1),this.opacity)},darker(t){return new fu(this.h,this.c,this.l-q3*(t??1),this.opacity)},rgb(){return yU(this).rgb()}}))});var E0=N(()=>{"use strict";v8();vU()});function E8(t,e,r,n,i){var a=t*t,s=a*t;return((1-3*t+3*a-s)*e+(4-6*a+3*s)*r+(1+3*t+3*a-3*s)*n+s*i)/6}function S8(t){var e=t.length-1;return function(r){var n=r<=0?r=0:r>=1?(r=1,e-1):Math.floor(r*e),i=t[n],a=t[n+1],s=n>0?t[n-1]:2*i-a,l=n{"use strict";o(E8,"basis");o(S8,"default")});function A8(t){var e=t.length;return function(r){var n=Math.floor(((r%=1)<0?++r:r)*e),i=t[(n+e-1)%e],a=t[n%e],s=t[(n+1)%e],l=t[(n+2)%e];return E8((r-n/e)*e,i,a,s,l)}}var xU=N(()=>{"use strict";C8();o(A8,"default")});var S0,_8=N(()=>{"use strict";S0=o(t=>()=>t,"default")});function bU(t,e){return function(r){return t+r*e}}function cwe(t,e,r){return t=Math.pow(t,r),e=Math.pow(e,r)-t,r=1/r,function(n){return Math.pow(t+n*e,r)}}function wU(t,e){var r=e-t;return r?bU(t,r>180||r<-180?r-360*Math.round(r/360):r):S0(isNaN(t)?e:t)}function TU(t){return(t=+t)==1?du:function(e,r){return r-e?cwe(e,r,t):S0(isNaN(e)?r:e)}}function du(t,e){var r=e-t;return r?bU(t,r):S0(isNaN(t)?e:t)}var D8=N(()=>{"use strict";_8();o(bU,"linear");o(cwe,"exponential");o(wU,"hue");o(TU,"gamma");o(du,"nogamma")});function kU(t){return function(e){var r=e.length,n=new Array(r),i=new Array(r),a=new Array(r),s,l;for(s=0;s{"use strict";E0();C8();xU();D8();ud=o(function t(e){var r=TU(e);function n(i,a){var s=r((i=T0(i)).r,(a=T0(a)).r),l=r(i.g,a.g),u=r(i.b,a.b),h=du(i.opacity,a.opacity);return function(f){return i.r=s(f),i.g=l(f),i.b=u(f),i.opacity=h(f),i+""}}return o(n,"rgb"),n.gamma=t,n},"rgbGamma")(1);o(kU,"rgbSpline");uwe=kU(S8),hwe=kU(A8)});function R8(t,e){e||(e=[]);var r=t?Math.min(e.length,t.length):0,n=e.slice(),i;return function(a){for(i=0;i{"use strict";o(R8,"default");o(EU,"isNumberArray")});function CU(t,e){var r=e?e.length:0,n=t?Math.min(r,t.length):0,i=new Array(n),a=new Array(r),s;for(s=0;s{"use strict";Y3();o(CU,"genericArray")});function N8(t,e){var r=new Date;return t=+t,e=+e,function(n){return r.setTime(t*(1-n)+e*n),r}}var _U=N(()=>{"use strict";o(N8,"default")});function Ki(t,e){return t=+t,e=+e,function(r){return t*(1-r)+e*r}}var ov=N(()=>{"use strict";o(Ki,"default")});function M8(t,e){var r={},n={},i;(t===null||typeof t!="object")&&(t={}),(e===null||typeof e!="object")&&(e={});for(i in e)i in t?r[i]=kh(t[i],e[i]):n[i]=e[i];return function(a){for(i in r)n[i]=r[i](a);return n}}var DU=N(()=>{"use strict";Y3();o(M8,"default")});function fwe(t){return function(){return t}}function dwe(t){return function(e){return t(e)+""}}function C0(t,e){var r=O8.lastIndex=I8.lastIndex=0,n,i,a,s=-1,l=[],u=[];for(t=t+"",e=e+"";(n=O8.exec(t))&&(i=I8.exec(e));)(a=i.index)>r&&(a=e.slice(r,a),l[s]?l[s]+=a:l[++s]=a),(n=n[0])===(i=i[0])?l[s]?l[s]+=i:l[++s]=i:(l[++s]=null,u.push({i:s,x:Ki(n,i)})),r=I8.lastIndex;return r{"use strict";ov();O8=/[-+]?(?:\d+\.?\d*|\.?\d+)(?:[eE][-+]?\d+)?/g,I8=new RegExp(O8.source,"g");o(fwe,"zero");o(dwe,"one");o(C0,"default")});function kh(t,e){var r=typeof e,n;return e==null||r==="boolean"?S0(e):(r==="number"?Ki:r==="string"?(n=pl(e))?(e=n,ud):C0:e instanceof pl?ud:e instanceof Date?N8:EU(e)?R8:Array.isArray(e)?CU:typeof e.valueOf!="function"&&typeof e.toString!="function"||isNaN(e)?M8:Ki)(t,e)}var Y3=N(()=>{"use strict";E0();L8();AU();_U();ov();DU();P8();_8();SU();o(kh,"default")});function X3(t,e){return t=+t,e=+e,function(r){return Math.round(t*(1-r)+e*r)}}var LU=N(()=>{"use strict";o(X3,"default")});function K3(t,e,r,n,i,a){var s,l,u;return(s=Math.sqrt(t*t+e*e))&&(t/=s,e/=s),(u=t*r+e*n)&&(r-=t*u,n-=e*u),(l=Math.sqrt(r*r+n*n))&&(r/=l,n/=l,u/=l),t*n{"use strict";RU=180/Math.PI,j3={translateX:0,translateY:0,rotate:0,skewX:0,scaleX:1,scaleY:1};o(K3,"default")});function MU(t){let e=new(typeof DOMMatrix=="function"?DOMMatrix:WebKitCSSMatrix)(t+"");return e.isIdentity?j3:K3(e.a,e.b,e.c,e.d,e.e,e.f)}function IU(t){return t==null?j3:(Q3||(Q3=document.createElementNS("http://www.w3.org/2000/svg","g")),Q3.setAttribute("transform",t),(t=Q3.transform.baseVal.consolidate())?(t=t.matrix,K3(t.a,t.b,t.c,t.d,t.e,t.f)):j3)}var Q3,OU=N(()=>{"use strict";NU();o(MU,"parseCss");o(IU,"parseSvg")});function PU(t,e,r,n){function i(h){return h.length?h.pop()+" ":""}o(i,"pop");function a(h,f,d,p,m,g){if(h!==d||f!==p){var y=m.push("translate(",null,e,null,r);g.push({i:y-4,x:Ki(h,d)},{i:y-2,x:Ki(f,p)})}else(d||p)&&m.push("translate("+d+e+p+r)}o(a,"translate");function s(h,f,d,p){h!==f?(h-f>180?f+=360:f-h>180&&(h+=360),p.push({i:d.push(i(d)+"rotate(",null,n)-2,x:Ki(h,f)})):f&&d.push(i(d)+"rotate("+f+n)}o(s,"rotate");function l(h,f,d,p){h!==f?p.push({i:d.push(i(d)+"skewX(",null,n)-2,x:Ki(h,f)}):f&&d.push(i(d)+"skewX("+f+n)}o(l,"skewX");function u(h,f,d,p,m,g){if(h!==d||f!==p){var y=m.push(i(m)+"scale(",null,",",null,")");g.push({i:y-4,x:Ki(h,d)},{i:y-2,x:Ki(f,p)})}else(d!==1||p!==1)&&m.push(i(m)+"scale("+d+","+p+")")}return o(u,"scale"),function(h,f){var d=[],p=[];return h=t(h),f=t(f),a(h.translateX,h.translateY,f.translateX,f.translateY,d,p),s(h.rotate,f.rotate,d,p),l(h.skewX,f.skewX,d,p),u(h.scaleX,h.scaleY,f.scaleX,f.scaleY,d,p),h=f=null,function(m){for(var g=-1,y=p.length,v;++g{"use strict";ov();OU();o(PU,"interpolateTransform");B8=PU(MU,"px, ","px)","deg)"),F8=PU(IU,", ",")",")")});function FU(t){return function(e,r){var n=t((e=sv(e)).h,(r=sv(r)).h),i=du(e.c,r.c),a=du(e.l,r.l),s=du(e.opacity,r.opacity);return function(l){return e.h=n(l),e.c=i(l),e.l=a(l),e.opacity=s(l),e+""}}}var $8,pwe,$U=N(()=>{"use strict";E0();D8();o(FU,"hcl");$8=FU(wU),pwe=FU(du)});var A0=N(()=>{"use strict";Y3();ov();LU();P8();BU();L8();$U()});function dv(){return hd||(VU(mwe),hd=hv.now()+e5)}function mwe(){hd=0}function fv(){this._call=this._time=this._next=null}function t5(t,e,r){var n=new fv;return n.restart(t,e,r),n}function UU(){dv(),++_0;for(var t=Z3,e;t;)(e=hd-t._time)>=0&&t._call.call(void 0,e),t=t._next;--_0}function zU(){hd=(J3=hv.now())+e5,_0=cv=0;try{UU()}finally{_0=0,ywe(),hd=0}}function gwe(){var t=hv.now(),e=t-J3;e>GU&&(e5-=e,J3=t)}function ywe(){for(var t,e=Z3,r,n=1/0;e;)e._call?(n>e._time&&(n=e._time),t=e,e=e._next):(r=e._next,e._next=null,e=t?t._next=r:Z3=r);uv=t,z8(n)}function z8(t){if(!_0){cv&&(cv=clearTimeout(cv));var e=t-hd;e>24?(t<1/0&&(cv=setTimeout(zU,t-hv.now()-e5)),lv&&(lv=clearInterval(lv))):(lv||(J3=hv.now(),lv=setInterval(gwe,GU)),_0=1,VU(zU))}}var _0,cv,lv,GU,Z3,uv,J3,hd,e5,hv,VU,G8=N(()=>{"use strict";_0=0,cv=0,lv=0,GU=1e3,J3=0,hd=0,e5=0,hv=typeof performance=="object"&&performance.now?performance:Date,VU=typeof window=="object"&&window.requestAnimationFrame?window.requestAnimationFrame.bind(window):function(t){setTimeout(t,17)};o(dv,"now");o(mwe,"clearNow");o(fv,"Timer");fv.prototype=t5.prototype={constructor:fv,restart:o(function(t,e,r){if(typeof t!="function")throw new TypeError("callback is not a function");r=(r==null?dv():+r)+(e==null?0:+e),!this._next&&uv!==this&&(uv?uv._next=this:Z3=this,uv=this),this._call=t,this._time=r,z8()},"restart"),stop:o(function(){this._call&&(this._call=null,this._time=1/0,z8())},"stop")};o(t5,"timer");o(UU,"timerFlush");o(zU,"wake");o(gwe,"poke");o(ywe,"nap");o(z8,"sleep")});function pv(t,e,r){var n=new fv;return e=e==null?0:+e,n.restart(i=>{n.stop(),t(i+e)},e,r),n}var HU=N(()=>{"use strict";G8();o(pv,"default")});var r5=N(()=>{"use strict";G8();HU()});function pu(t,e,r,n,i,a){var s=t.__transition;if(!s)t.__transition={};else if(r in s)return;bwe(t,r,{name:e,index:n,group:i,on:vwe,tween:xwe,time:a.time,delay:a.delay,duration:a.duration,ease:a.ease,timer:null,state:YU})}function gv(t,e){var r=Bi(t,e);if(r.state>YU)throw new Error("too late; already scheduled");return r}function ha(t,e){var r=Bi(t,e);if(r.state>n5)throw new Error("too late; already running");return r}function Bi(t,e){var r=t.__transition;if(!r||!(r=r[e]))throw new Error("transition not found");return r}function bwe(t,e,r){var n=t.__transition,i;n[e]=r,r.timer=t5(a,0,r.time);function a(h){r.state=WU,r.timer.restart(s,r.delay,r.time),r.delay<=h&&s(h-r.delay)}o(a,"schedule");function s(h){var f,d,p,m;if(r.state!==WU)return u();for(f in n)if(m=n[f],m.name===r.name){if(m.state===n5)return pv(s);m.state===qU?(m.state=mv,m.timer.stop(),m.on.call("interrupt",t,t.__data__,m.index,m.group),delete n[f]):+f{"use strict";TA();r5();vwe=wA("start","end","cancel","interrupt"),xwe=[],YU=0,WU=1,i5=2,n5=3,qU=4,a5=5,mv=6;o(pu,"default");o(gv,"init");o(ha,"set");o(Bi,"get");o(bwe,"create")});function yv(t,e){var r=t.__transition,n,i,a=!0,s;if(r){e=e==null?null:e+"";for(s in r){if((n=r[s]).name!==e){a=!1;continue}i=n.state>i5&&n.state{"use strict";Es();o(yv,"default")});function V8(t){return this.each(function(){yv(this,t)})}var jU=N(()=>{"use strict";XU();o(V8,"default")});function wwe(t,e){var r,n;return function(){var i=ha(this,t),a=i.tween;if(a!==r){n=r=a;for(var s=0,l=n.length;s{"use strict";Es();o(wwe,"tweenRemove");o(Twe,"tweenFunction");o(U8,"default");o(D0,"tweenValue")});function xv(t,e){var r;return(typeof e=="number"?Ki:e instanceof pl?ud:(r=pl(e))?(e=r,ud):C0)(t,e)}var H8=N(()=>{"use strict";E0();A0();o(xv,"default")});function kwe(t){return function(){this.removeAttribute(t)}}function Ewe(t){return function(){this.removeAttributeNS(t.space,t.local)}}function Swe(t,e,r){var n,i=r+"",a;return function(){var s=this.getAttribute(t);return s===i?null:s===n?a:a=e(n=s,r)}}function Cwe(t,e,r){var n,i=r+"",a;return function(){var s=this.getAttributeNS(t.space,t.local);return s===i?null:s===n?a:a=e(n=s,r)}}function Awe(t,e,r){var n,i,a;return function(){var s,l=r(this),u;return l==null?void this.removeAttribute(t):(s=this.getAttribute(t),u=l+"",s===u?null:s===n&&u===i?a:(i=u,a=e(n=s,l)))}}function _we(t,e,r){var n,i,a;return function(){var s,l=r(this),u;return l==null?void this.removeAttributeNS(t.space,t.local):(s=this.getAttributeNS(t.space,t.local),u=l+"",s===u?null:s===n&&u===i?a:(i=u,a=e(n=s,l)))}}function W8(t,e){var r=rc(t),n=r==="transform"?F8:xv;return this.attrTween(t,typeof e=="function"?(r.local?_we:Awe)(r,n,D0(this,"attr."+t,e)):e==null?(r.local?Ewe:kwe)(r):(r.local?Cwe:Swe)(r,n,e))}var KU=N(()=>{"use strict";A0();fl();vv();H8();o(kwe,"attrRemove");o(Ewe,"attrRemoveNS");o(Swe,"attrConstant");o(Cwe,"attrConstantNS");o(Awe,"attrFunction");o(_we,"attrFunctionNS");o(W8,"default")});function Dwe(t,e){return function(r){this.setAttribute(t,e.call(this,r))}}function Lwe(t,e){return function(r){this.setAttributeNS(t.space,t.local,e.call(this,r))}}function Rwe(t,e){var r,n;function i(){var a=e.apply(this,arguments);return a!==n&&(r=(n=a)&&Lwe(t,a)),r}return o(i,"tween"),i._value=e,i}function Nwe(t,e){var r,n;function i(){var a=e.apply(this,arguments);return a!==n&&(r=(n=a)&&Dwe(t,a)),r}return o(i,"tween"),i._value=e,i}function q8(t,e){var r="attr."+t;if(arguments.length<2)return(r=this.tween(r))&&r._value;if(e==null)return this.tween(r,null);if(typeof e!="function")throw new Error;var n=rc(t);return this.tween(r,(n.local?Rwe:Nwe)(n,e))}var QU=N(()=>{"use strict";fl();o(Dwe,"attrInterpolate");o(Lwe,"attrInterpolateNS");o(Rwe,"attrTweenNS");o(Nwe,"attrTween");o(q8,"default")});function Mwe(t,e){return function(){gv(this,t).delay=+e.apply(this,arguments)}}function Iwe(t,e){return e=+e,function(){gv(this,t).delay=e}}function Y8(t){var e=this._id;return arguments.length?this.each((typeof t=="function"?Mwe:Iwe)(e,t)):Bi(this.node(),e).delay}var ZU=N(()=>{"use strict";Es();o(Mwe,"delayFunction");o(Iwe,"delayConstant");o(Y8,"default")});function Owe(t,e){return function(){ha(this,t).duration=+e.apply(this,arguments)}}function Pwe(t,e){return e=+e,function(){ha(this,t).duration=e}}function X8(t){var e=this._id;return arguments.length?this.each((typeof t=="function"?Owe:Pwe)(e,t)):Bi(this.node(),e).duration}var JU=N(()=>{"use strict";Es();o(Owe,"durationFunction");o(Pwe,"durationConstant");o(X8,"default")});function Bwe(t,e){if(typeof e!="function")throw new Error;return function(){ha(this,t).ease=e}}function j8(t){var e=this._id;return arguments.length?this.each(Bwe(e,t)):Bi(this.node(),e).ease}var eH=N(()=>{"use strict";Es();o(Bwe,"easeConstant");o(j8,"default")});function Fwe(t,e){return function(){var r=e.apply(this,arguments);if(typeof r!="function")throw new Error;ha(this,t).ease=r}}function K8(t){if(typeof t!="function")throw new Error;return this.each(Fwe(this._id,t))}var tH=N(()=>{"use strict";Es();o(Fwe,"easeVarying");o(K8,"default")});function Q8(t){typeof t!="function"&&(t=x0(t));for(var e=this._groups,r=e.length,n=new Array(r),i=0;i{"use strict";fl();fd();o(Q8,"default")});function Z8(t){if(t._id!==this._id)throw new Error;for(var e=this._groups,r=t._groups,n=e.length,i=r.length,a=Math.min(n,i),s=new Array(n),l=0;l{"use strict";fd();o(Z8,"default")});function $we(t){return(t+"").trim().split(/^|\s+/).every(function(e){var r=e.indexOf(".");return r>=0&&(e=e.slice(0,r)),!e||e==="start"})}function zwe(t,e,r){var n,i,a=$we(e)?gv:ha;return function(){var s=a(this,t),l=s.on;l!==n&&(i=(n=l).copy()).on(e,r),s.on=i}}function J8(t,e){var r=this._id;return arguments.length<2?Bi(this.node(),r).on.on(t):this.each(zwe(r,t,e))}var iH=N(()=>{"use strict";Es();o($we,"start");o(zwe,"onFunction");o(J8,"default")});function Gwe(t){return function(){var e=this.parentNode;for(var r in this.__transition)if(+r!==t)return;e&&e.removeChild(this)}}function e_(){return this.on("end.remove",Gwe(this._id))}var aH=N(()=>{"use strict";o(Gwe,"removeFunction");o(e_,"default")});function t_(t){var e=this._name,r=this._id;typeof t!="function"&&(t=xh(t));for(var n=this._groups,i=n.length,a=new Array(i),s=0;s{"use strict";fl();fd();Es();o(t_,"default")});function r_(t){var e=this._name,r=this._id;typeof t!="function"&&(t=v0(t));for(var n=this._groups,i=n.length,a=[],s=[],l=0;l{"use strict";fl();fd();Es();o(r_,"default")});function n_(){return new Vwe(this._groups,this._parents)}var Vwe,lH=N(()=>{"use strict";fl();Vwe=hu.prototype.constructor;o(n_,"default")});function Uwe(t,e){var r,n,i;return function(){var a=bh(this,t),s=(this.style.removeProperty(t),bh(this,t));return a===s?null:a===r&&s===n?i:i=e(r=a,n=s)}}function cH(t){return function(){this.style.removeProperty(t)}}function Hwe(t,e,r){var n,i=r+"",a;return function(){var s=bh(this,t);return s===i?null:s===n?a:a=e(n=s,r)}}function Wwe(t,e,r){var n,i,a;return function(){var s=bh(this,t),l=r(this),u=l+"";return l==null&&(u=l=(this.style.removeProperty(t),bh(this,t))),s===u?null:s===n&&u===i?a:(i=u,a=e(n=s,l))}}function qwe(t,e){var r,n,i,a="style."+e,s="end."+a,l;return function(){var u=ha(this,t),h=u.on,f=u.value[a]==null?l||(l=cH(e)):void 0;(h!==r||i!==f)&&(n=(r=h).copy()).on(s,i=f),u.on=n}}function i_(t,e,r){var n=(t+="")=="transform"?B8:xv;return e==null?this.styleTween(t,Uwe(t,n)).on("end.style."+t,cH(t)):typeof e=="function"?this.styleTween(t,Wwe(t,n,D0(this,"style."+t,e))).each(qwe(this._id,t)):this.styleTween(t,Hwe(t,n,e),r).on("end.style."+t,null)}var uH=N(()=>{"use strict";A0();fl();Es();vv();H8();o(Uwe,"styleNull");o(cH,"styleRemove");o(Hwe,"styleConstant");o(Wwe,"styleFunction");o(qwe,"styleMaybeRemove");o(i_,"default")});function Ywe(t,e,r){return function(n){this.style.setProperty(t,e.call(this,n),r)}}function Xwe(t,e,r){var n,i;function a(){var s=e.apply(this,arguments);return s!==i&&(n=(i=s)&&Ywe(t,s,r)),n}return o(a,"tween"),a._value=e,a}function a_(t,e,r){var n="style."+(t+="");if(arguments.length<2)return(n=this.tween(n))&&n._value;if(e==null)return this.tween(n,null);if(typeof e!="function")throw new Error;return this.tween(n,Xwe(t,e,r??""))}var hH=N(()=>{"use strict";o(Ywe,"styleInterpolate");o(Xwe,"styleTween");o(a_,"default")});function jwe(t){return function(){this.textContent=t}}function Kwe(t){return function(){var e=t(this);this.textContent=e??""}}function s_(t){return this.tween("text",typeof t=="function"?Kwe(D0(this,"text",t)):jwe(t==null?"":t+""))}var fH=N(()=>{"use strict";vv();o(jwe,"textConstant");o(Kwe,"textFunction");o(s_,"default")});function Qwe(t){return function(e){this.textContent=t.call(this,e)}}function Zwe(t){var e,r;function n(){var i=t.apply(this,arguments);return i!==r&&(e=(r=i)&&Qwe(i)),e}return o(n,"tween"),n._value=t,n}function o_(t){var e="text";if(arguments.length<1)return(e=this.tween(e))&&e._value;if(t==null)return this.tween(e,null);if(typeof t!="function")throw new Error;return this.tween(e,Zwe(t))}var dH=N(()=>{"use strict";o(Qwe,"textInterpolate");o(Zwe,"textTween");o(o_,"default")});function l_(){for(var t=this._name,e=this._id,r=s5(),n=this._groups,i=n.length,a=0;a{"use strict";fd();Es();o(l_,"default")});function c_(){var t,e,r=this,n=r._id,i=r.size();return new Promise(function(a,s){var l={value:s},u={value:o(function(){--i===0&&a()},"value")};r.each(function(){var h=ha(this,n),f=h.on;f!==t&&(e=(t=f).copy(),e._.cancel.push(l),e._.interrupt.push(l),e._.end.push(u)),h.on=e}),i===0&&a()})}var mH=N(()=>{"use strict";Es();o(c_,"default")});function es(t,e,r,n){this._groups=t,this._parents=e,this._name=r,this._id=n}function gH(t){return hu().transition(t)}function s5(){return++Jwe}var Jwe,mu,fd=N(()=>{"use strict";fl();KU();QU();ZU();JU();eH();tH();rH();nH();iH();aH();sH();oH();lH();uH();hH();fH();dH();pH();vv();mH();Jwe=0;o(es,"Transition");o(gH,"transition");o(s5,"newId");mu=hu.prototype;es.prototype=gH.prototype={constructor:es,select:t_,selectAll:r_,selectChild:mu.selectChild,selectChildren:mu.selectChildren,filter:Q8,merge:Z8,selection:n_,transition:l_,call:mu.call,nodes:mu.nodes,node:mu.node,size:mu.size,empty:mu.empty,each:mu.each,on:J8,attr:W8,attrTween:q8,style:i_,styleTween:a_,text:s_,textTween:o_,remove:e_,tween:U8,delay:Y8,duration:X8,ease:j8,easeVarying:K8,end:c_,[Symbol.iterator]:mu[Symbol.iterator]}});function o5(t){return((t*=2)<=1?t*t*t:(t-=2)*t*t+2)/2}var yH=N(()=>{"use strict";o(o5,"cubicInOut")});var u_=N(()=>{"use strict";yH()});function tTe(t,e){for(var r;!(r=t.__transition)||!(r=r[e]);)if(!(t=t.parentNode))throw new Error(`transition ${e} not found`);return r}function h_(t){var e,r;t instanceof es?(e=t._id,t=t._name):(e=s5(),(r=eTe).time=dv(),t=t==null?null:t+"");for(var n=this._groups,i=n.length,a=0;a{"use strict";fd();Es();u_();r5();eTe={time:null,delay:0,duration:250,ease:o5};o(tTe,"inherit");o(h_,"default")});var xH=N(()=>{"use strict";fl();jU();vH();hu.prototype.interrupt=V8;hu.prototype.transition=h_});var l5=N(()=>{"use strict";xH()});var bH=N(()=>{"use strict"});var wH=N(()=>{"use strict"});var TH=N(()=>{"use strict"});function kH(t){return[+t[0],+t[1]]}function rTe(t){return[kH(t[0]),kH(t[1])]}function f_(t){return{type:t}}var Z0t,J0t,emt,tmt,rmt,nmt,EH=N(()=>{"use strict";l5();bH();wH();TH();({abs:Z0t,max:J0t,min:emt}=Math);o(kH,"number1");o(rTe,"number2");tmt={name:"x",handles:["w","e"].map(f_),input:o(function(t,e){return t==null?null:[[+t[0],e[0][1]],[+t[1],e[1][1]]]},"input"),output:o(function(t){return t&&[t[0][0],t[1][0]]},"output")},rmt={name:"y",handles:["n","s"].map(f_),input:o(function(t,e){return t==null?null:[[e[0][0],+t[0]],[e[1][0],+t[1]]]},"input"),output:o(function(t){return t&&[t[0][1],t[1][1]]},"output")},nmt={name:"xy",handles:["n","w","e","s","nw","ne","sw","se"].map(f_),input:o(function(t){return t==null?null:rTe(t)},"input"),output:o(function(t){return t},"output")};o(f_,"type")});var SH=N(()=>{"use strict";EH()});function CH(t){this._+=t[0];for(let e=1,r=t.length;e=0))throw new Error(`invalid digits: ${t}`);if(e>15)return CH;let r=10**e;return function(n){this._+=n[0];for(let i=1,a=n.length;i{"use strict";d_=Math.PI,p_=2*d_,dd=1e-6,nTe=p_-dd;o(CH,"append");o(iTe,"appendRound");pd=class{static{o(this,"Path")}constructor(e){this._x0=this._y0=this._x1=this._y1=null,this._="",this._append=e==null?CH:iTe(e)}moveTo(e,r){this._append`M${this._x0=this._x1=+e},${this._y0=this._y1=+r}`}closePath(){this._x1!==null&&(this._x1=this._x0,this._y1=this._y0,this._append`Z`)}lineTo(e,r){this._append`L${this._x1=+e},${this._y1=+r}`}quadraticCurveTo(e,r,n,i){this._append`Q${+e},${+r},${this._x1=+n},${this._y1=+i}`}bezierCurveTo(e,r,n,i,a,s){this._append`C${+e},${+r},${+n},${+i},${this._x1=+a},${this._y1=+s}`}arcTo(e,r,n,i,a){if(e=+e,r=+r,n=+n,i=+i,a=+a,a<0)throw new Error(`negative radius: ${a}`);let s=this._x1,l=this._y1,u=n-e,h=i-r,f=s-e,d=l-r,p=f*f+d*d;if(this._x1===null)this._append`M${this._x1=e},${this._y1=r}`;else if(p>dd)if(!(Math.abs(d*u-h*f)>dd)||!a)this._append`L${this._x1=e},${this._y1=r}`;else{let m=n-s,g=i-l,y=u*u+h*h,v=m*m+g*g,x=Math.sqrt(y),b=Math.sqrt(p),w=a*Math.tan((d_-Math.acos((y+p-v)/(2*x*b)))/2),C=w/b,T=w/x;Math.abs(C-1)>dd&&this._append`L${e+C*f},${r+C*d}`,this._append`A${a},${a},0,0,${+(d*m>f*g)},${this._x1=e+T*u},${this._y1=r+T*h}`}}arc(e,r,n,i,a,s){if(e=+e,r=+r,n=+n,s=!!s,n<0)throw new Error(`negative radius: ${n}`);let l=n*Math.cos(i),u=n*Math.sin(i),h=e+l,f=r+u,d=1^s,p=s?i-a:a-i;this._x1===null?this._append`M${h},${f}`:(Math.abs(this._x1-h)>dd||Math.abs(this._y1-f)>dd)&&this._append`L${h},${f}`,n&&(p<0&&(p=p%p_+p_),p>nTe?this._append`A${n},${n},0,1,${d},${e-l},${r-u}A${n},${n},0,1,${d},${this._x1=h},${this._y1=f}`:p>dd&&this._append`A${n},${n},0,${+(p>=d_)},${d},${this._x1=e+n*Math.cos(a)},${this._y1=r+n*Math.sin(a)}`)}rect(e,r,n,i){this._append`M${this._x0=this._x1=+e},${this._y0=this._y1=+r}h${n=+n}v${+i}h${-n}Z`}toString(){return this._}};o(AH,"path");AH.prototype=pd.prototype});var m_=N(()=>{"use strict";_H()});var DH=N(()=>{"use strict"});var LH=N(()=>{"use strict"});var RH=N(()=>{"use strict"});var NH=N(()=>{"use strict"});var MH=N(()=>{"use strict"});var IH=N(()=>{"use strict"});var OH=N(()=>{"use strict"});function g_(t){return Math.abs(t=Math.round(t))>=1e21?t.toLocaleString("en").replace(/,/g,""):t.toString(10)}function md(t,e){if((r=(t=e?t.toExponential(e-1):t.toExponential()).indexOf("e"))<0)return null;var r,n=t.slice(0,r);return[n.length>1?n[0]+n.slice(2):n,+t.slice(r+1)]}var bv=N(()=>{"use strict";o(g_,"default");o(md,"formatDecimalParts")});function ml(t){return t=md(Math.abs(t)),t?t[1]:NaN}var wv=N(()=>{"use strict";bv();o(ml,"default")});function y_(t,e){return function(r,n){for(var i=r.length,a=[],s=0,l=t[0],u=0;i>0&&l>0&&(u+l+1>n&&(l=Math.max(1,n-u)),a.push(r.substring(i-=l,i+l)),!((u+=l+1)>n));)l=t[s=(s+1)%t.length];return a.reverse().join(e)}}var PH=N(()=>{"use strict";o(y_,"default")});function v_(t){return function(e){return e.replace(/[0-9]/g,function(r){return t[+r]})}}var BH=N(()=>{"use strict";o(v_,"default")});function Eh(t){if(!(e=aTe.exec(t)))throw new Error("invalid format: "+t);var e;return new c5({fill:e[1],align:e[2],sign:e[3],symbol:e[4],zero:e[5],width:e[6],comma:e[7],precision:e[8]&&e[8].slice(1),trim:e[9],type:e[10]})}function c5(t){this.fill=t.fill===void 0?" ":t.fill+"",this.align=t.align===void 0?">":t.align+"",this.sign=t.sign===void 0?"-":t.sign+"",this.symbol=t.symbol===void 0?"":t.symbol+"",this.zero=!!t.zero,this.width=t.width===void 0?void 0:+t.width,this.comma=!!t.comma,this.precision=t.precision===void 0?void 0:+t.precision,this.trim=!!t.trim,this.type=t.type===void 0?"":t.type+""}var aTe,x_=N(()=>{"use strict";aTe=/^(?:(.)?([<>=^]))?([+\-( ])?([$#])?(0)?(\d+)?(,)?(\.\d+)?(~)?([a-z%])?$/i;o(Eh,"formatSpecifier");Eh.prototype=c5.prototype;o(c5,"FormatSpecifier");c5.prototype.toString=function(){return this.fill+this.align+this.sign+this.symbol+(this.zero?"0":"")+(this.width===void 0?"":Math.max(1,this.width|0))+(this.comma?",":"")+(this.precision===void 0?"":"."+Math.max(0,this.precision|0))+(this.trim?"~":"")+this.type}});function b_(t){e:for(var e=t.length,r=1,n=-1,i;r0&&(n=0);break}return n>0?t.slice(0,n)+t.slice(i+1):t}var FH=N(()=>{"use strict";o(b_,"default")});function T_(t,e){var r=md(t,e);if(!r)return t+"";var n=r[0],i=r[1],a=i-(w_=Math.max(-8,Math.min(8,Math.floor(i/3)))*3)+1,s=n.length;return a===s?n:a>s?n+new Array(a-s+1).join("0"):a>0?n.slice(0,a)+"."+n.slice(a):"0."+new Array(1-a).join("0")+md(t,Math.max(0,e+a-1))[0]}var w_,k_=N(()=>{"use strict";bv();o(T_,"default")});function u5(t,e){var r=md(t,e);if(!r)return t+"";var n=r[0],i=r[1];return i<0?"0."+new Array(-i).join("0")+n:n.length>i+1?n.slice(0,i+1)+"."+n.slice(i+1):n+new Array(i-n.length+2).join("0")}var $H=N(()=>{"use strict";bv();o(u5,"default")});var E_,zH=N(()=>{"use strict";bv();k_();$H();E_={"%":o((t,e)=>(t*100).toFixed(e),"%"),b:o(t=>Math.round(t).toString(2),"b"),c:o(t=>t+"","c"),d:g_,e:o((t,e)=>t.toExponential(e),"e"),f:o((t,e)=>t.toFixed(e),"f"),g:o((t,e)=>t.toPrecision(e),"g"),o:o(t=>Math.round(t).toString(8),"o"),p:o((t,e)=>u5(t*100,e),"p"),r:u5,s:T_,X:o(t=>Math.round(t).toString(16).toUpperCase(),"X"),x:o(t=>Math.round(t).toString(16),"x")}});function h5(t){return t}var GH=N(()=>{"use strict";o(h5,"default")});function S_(t){var e=t.grouping===void 0||t.thousands===void 0?h5:y_(VH.call(t.grouping,Number),t.thousands+""),r=t.currency===void 0?"":t.currency[0]+"",n=t.currency===void 0?"":t.currency[1]+"",i=t.decimal===void 0?".":t.decimal+"",a=t.numerals===void 0?h5:v_(VH.call(t.numerals,String)),s=t.percent===void 0?"%":t.percent+"",l=t.minus===void 0?"\u2212":t.minus+"",u=t.nan===void 0?"NaN":t.nan+"";function h(d){d=Eh(d);var p=d.fill,m=d.align,g=d.sign,y=d.symbol,v=d.zero,x=d.width,b=d.comma,w=d.precision,C=d.trim,T=d.type;T==="n"?(b=!0,T="g"):E_[T]||(w===void 0&&(w=12),C=!0,T="g"),(v||p==="0"&&m==="=")&&(v=!0,p="0",m="=");var E=y==="$"?r:y==="#"&&/[boxX]/.test(T)?"0"+T.toLowerCase():"",A=y==="$"?n:/[%p]/.test(T)?s:"",S=E_[T],_=/[defgprs%]/.test(T);w=w===void 0?6:/[gprs]/.test(T)?Math.max(1,Math.min(21,w)):Math.max(0,Math.min(20,w));function I(D){var k=E,L=A,R,O,M;if(T==="c")L=S(D)+L,D="";else{D=+D;var B=D<0||1/D<0;if(D=isNaN(D)?u:S(Math.abs(D),w),C&&(D=b_(D)),B&&+D==0&&g!=="+"&&(B=!1),k=(B?g==="("?g:l:g==="-"||g==="("?"":g)+k,L=(T==="s"?UH[8+w_/3]:"")+L+(B&&g==="("?")":""),_){for(R=-1,O=D.length;++RM||M>57){L=(M===46?i+D.slice(R+1):D.slice(R))+L,D=D.slice(0,R);break}}}b&&!v&&(D=e(D,1/0));var F=k.length+D.length+L.length,P=F>1)+k+D+L+P.slice(F);break;default:D=P+k+D+L;break}return a(D)}return o(I,"format"),I.toString=function(){return d+""},I}o(h,"newFormat");function f(d,p){var m=h((d=Eh(d),d.type="f",d)),g=Math.max(-8,Math.min(8,Math.floor(ml(p)/3)))*3,y=Math.pow(10,-g),v=UH[8+g/3];return function(x){return m(y*x)+v}}return o(f,"formatPrefix"),{format:h,formatPrefix:f}}var VH,UH,HH=N(()=>{"use strict";wv();PH();BH();x_();FH();zH();k_();GH();VH=Array.prototype.map,UH=["y","z","a","f","p","n","\xB5","m","","k","M","G","T","P","E","Z","Y"];o(S_,"default")});function C_(t){return f5=S_(t),d5=f5.format,p5=f5.formatPrefix,f5}var f5,d5,p5,WH=N(()=>{"use strict";HH();C_({thousands:",",grouping:[3],currency:["$",""]});o(C_,"defaultLocale")});function m5(t){return Math.max(0,-ml(Math.abs(t)))}var qH=N(()=>{"use strict";wv();o(m5,"default")});function g5(t,e){return Math.max(0,Math.max(-8,Math.min(8,Math.floor(ml(e)/3)))*3-ml(Math.abs(t)))}var YH=N(()=>{"use strict";wv();o(g5,"default")});function y5(t,e){return t=Math.abs(t),e=Math.abs(e)-t,Math.max(0,ml(e)-ml(t))+1}var XH=N(()=>{"use strict";wv();o(y5,"default")});var A_=N(()=>{"use strict";WH();x_();qH();YH();XH()});var jH=N(()=>{"use strict"});var KH=N(()=>{"use strict"});var QH=N(()=>{"use strict"});var ZH=N(()=>{"use strict"});function Sh(t,e){switch(arguments.length){case 0:break;case 1:this.range(t);break;default:this.range(e).domain(t);break}return this}var Tv=N(()=>{"use strict";o(Sh,"initRange")});function gu(){var t=new g0,e=[],r=[],n=__;function i(a){let s=t.get(a);if(s===void 0){if(n!==__)return n;t.set(a,s=e.push(a)-1)}return r[s%r.length]}return o(i,"scale"),i.domain=function(a){if(!arguments.length)return e.slice();e=[],t=new g0;for(let s of a)t.has(s)||t.set(s,e.push(s)-1);return i},i.range=function(a){return arguments.length?(r=Array.from(a),i):r.slice()},i.unknown=function(a){return arguments.length?(n=a,i):n},i.copy=function(){return gu(e,r).unknown(n)},Sh.apply(i,arguments),i}var __,D_=N(()=>{"use strict";vh();Tv();__=Symbol("implicit");o(gu,"ordinal")});function L0(){var t=gu().unknown(void 0),e=t.domain,r=t.range,n=0,i=1,a,s,l=!1,u=0,h=0,f=.5;delete t.unknown;function d(){var p=e().length,m=i{"use strict";vh();Tv();D_();o(L0,"band")});function L_(t){return function(){return t}}var eW=N(()=>{"use strict";o(L_,"constants")});function R_(t){return+t}var tW=N(()=>{"use strict";o(R_,"number")});function R0(t){return t}function N_(t,e){return(e-=t=+t)?function(r){return(r-t)/e}:L_(isNaN(e)?NaN:.5)}function sTe(t,e){var r;return t>e&&(r=t,t=e,e=r),function(n){return Math.max(t,Math.min(e,n))}}function oTe(t,e,r){var n=t[0],i=t[1],a=e[0],s=e[1];return i2?lTe:oTe,u=h=null,d}o(f,"rescale");function d(p){return p==null||isNaN(p=+p)?a:(u||(u=l(t.map(n),e,r)))(n(s(p)))}return o(d,"scale"),d.invert=function(p){return s(i((h||(h=l(e,t.map(n),Ki)))(p)))},d.domain=function(p){return arguments.length?(t=Array.from(p,R_),f()):t.slice()},d.range=function(p){return arguments.length?(e=Array.from(p),f()):e.slice()},d.rangeRound=function(p){return e=Array.from(p),r=X3,f()},d.clamp=function(p){return arguments.length?(s=p?!0:R0,f()):s!==R0},d.interpolate=function(p){return arguments.length?(r=p,f()):r},d.unknown=function(p){return arguments.length?(a=p,d):a},function(p,m){return n=p,i=m,f()}}function kv(){return cTe()(R0,R0)}var rW,M_=N(()=>{"use strict";vh();A0();eW();tW();rW=[0,1];o(R0,"identity");o(N_,"normalize");o(sTe,"clamper");o(oTe,"bimap");o(lTe,"polymap");o(v5,"copy");o(cTe,"transformer");o(kv,"continuous")});function I_(t,e,r,n){var i=y0(t,e,r),a;switch(n=Eh(n??",f"),n.type){case"s":{var s=Math.max(Math.abs(t),Math.abs(e));return n.precision==null&&!isNaN(a=g5(i,s))&&(n.precision=a),p5(n,s)}case"":case"e":case"g":case"p":case"r":{n.precision==null&&!isNaN(a=y5(i,Math.max(Math.abs(t),Math.abs(e))))&&(n.precision=a-(n.type==="e"));break}case"f":case"%":{n.precision==null&&!isNaN(a=m5(i))&&(n.precision=a-(n.type==="%")*2);break}}return d5(n)}var nW=N(()=>{"use strict";vh();A_();o(I_,"tickFormat")});function uTe(t){var e=t.domain;return t.ticks=function(r){var n=e();return R3(n[0],n[n.length-1],r??10)},t.tickFormat=function(r,n){var i=e();return I_(i[0],i[i.length-1],r??10,n)},t.nice=function(r){r==null&&(r=10);var n=e(),i=0,a=n.length-1,s=n[i],l=n[a],u,h,f=10;for(l0;){if(h=Zy(s,l,r),h===u)return n[i]=s,n[a]=l,e(n);if(h>0)s=Math.floor(s/h)*h,l=Math.ceil(l/h)*h;else if(h<0)s=Math.ceil(s*h)/h,l=Math.floor(l*h)/h;else break;u=h}return t},t}function gl(){var t=kv();return t.copy=function(){return v5(t,gl())},Sh.apply(t,arguments),uTe(t)}var iW=N(()=>{"use strict";vh();M_();Tv();nW();o(uTe,"linearish");o(gl,"linear")});function O_(t,e){t=t.slice();var r=0,n=t.length-1,i=t[r],a=t[n],s;return a{"use strict";o(O_,"nice")});function xn(t,e,r,n){function i(a){return t(a=arguments.length===0?new Date:new Date(+a)),a}return o(i,"interval"),i.floor=a=>(t(a=new Date(+a)),a),i.ceil=a=>(t(a=new Date(a-1)),e(a,1),t(a),a),i.round=a=>{let s=i(a),l=i.ceil(a);return a-s(e(a=new Date(+a),s==null?1:Math.floor(s)),a),i.range=(a,s,l)=>{let u=[];if(a=i.ceil(a),l=l==null?1:Math.floor(l),!(a0))return u;let h;do u.push(h=new Date(+a)),e(a,l),t(a);while(hxn(s=>{if(s>=s)for(;t(s),!a(s);)s.setTime(s-1)},(s,l)=>{if(s>=s)if(l<0)for(;++l<=0;)for(;e(s,-1),!a(s););else for(;--l>=0;)for(;e(s,1),!a(s););}),r&&(i.count=(a,s)=>(P_.setTime(+a),B_.setTime(+s),t(P_),t(B_),Math.floor(r(P_,B_))),i.every=a=>(a=Math.floor(a),!isFinite(a)||!(a>0)?null:a>1?i.filter(n?s=>n(s)%a===0:s=>i.count(0,s)%a===0):i)),i}var P_,B_,yu=N(()=>{"use strict";P_=new Date,B_=new Date;o(xn,"timeInterval")});var ac,sW,F_=N(()=>{"use strict";yu();ac=xn(()=>{},(t,e)=>{t.setTime(+t+e)},(t,e)=>e-t);ac.every=t=>(t=Math.floor(t),!isFinite(t)||!(t>0)?null:t>1?xn(e=>{e.setTime(Math.floor(e/t)*t)},(e,r)=>{e.setTime(+e+r*t)},(e,r)=>(r-e)/t):ac);sW=ac.range});var Ks,oW,$_=N(()=>{"use strict";yu();Ks=xn(t=>{t.setTime(t-t.getMilliseconds())},(t,e)=>{t.setTime(+t+e*1e3)},(t,e)=>(e-t)/1e3,t=>t.getUTCSeconds()),oW=Ks.range});var vu,hTe,x5,fTe,z_=N(()=>{"use strict";yu();vu=xn(t=>{t.setTime(t-t.getMilliseconds()-t.getSeconds()*1e3)},(t,e)=>{t.setTime(+t+e*6e4)},(t,e)=>(e-t)/6e4,t=>t.getMinutes()),hTe=vu.range,x5=xn(t=>{t.setUTCSeconds(0,0)},(t,e)=>{t.setTime(+t+e*6e4)},(t,e)=>(e-t)/6e4,t=>t.getUTCMinutes()),fTe=x5.range});var xu,dTe,b5,pTe,G_=N(()=>{"use strict";yu();xu=xn(t=>{t.setTime(t-t.getMilliseconds()-t.getSeconds()*1e3-t.getMinutes()*6e4)},(t,e)=>{t.setTime(+t+e*36e5)},(t,e)=>(e-t)/36e5,t=>t.getHours()),dTe=xu.range,b5=xn(t=>{t.setUTCMinutes(0,0,0)},(t,e)=>{t.setTime(+t+e*36e5)},(t,e)=>(e-t)/36e5,t=>t.getUTCHours()),pTe=b5.range});var _o,mTe,Sv,gTe,w5,yTe,V_=N(()=>{"use strict";yu();_o=xn(t=>t.setHours(0,0,0,0),(t,e)=>t.setDate(t.getDate()+e),(t,e)=>(e-t-(e.getTimezoneOffset()-t.getTimezoneOffset())*6e4)/864e5,t=>t.getDate()-1),mTe=_o.range,Sv=xn(t=>{t.setUTCHours(0,0,0,0)},(t,e)=>{t.setUTCDate(t.getUTCDate()+e)},(t,e)=>(e-t)/864e5,t=>t.getUTCDate()-1),gTe=Sv.range,w5=xn(t=>{t.setUTCHours(0,0,0,0)},(t,e)=>{t.setUTCDate(t.getUTCDate()+e)},(t,e)=>(e-t)/864e5,t=>Math.floor(t/864e5)),yTe=w5.range});function vd(t){return xn(e=>{e.setDate(e.getDate()-(e.getDay()+7-t)%7),e.setHours(0,0,0,0)},(e,r)=>{e.setDate(e.getDate()+r*7)},(e,r)=>(r-e-(r.getTimezoneOffset()-e.getTimezoneOffset())*6e4)/6048e5)}function xd(t){return xn(e=>{e.setUTCDate(e.getUTCDate()-(e.getUTCDay()+7-t)%7),e.setUTCHours(0,0,0,0)},(e,r)=>{e.setUTCDate(e.getUTCDate()+r*7)},(e,r)=>(r-e)/6048e5)}var yl,Ch,T5,k5,oc,E5,S5,cW,vTe,xTe,bTe,wTe,TTe,kTe,bd,N0,uW,hW,Ah,fW,dW,pW,ETe,STe,CTe,ATe,_Te,DTe,U_=N(()=>{"use strict";yu();o(vd,"timeWeekday");yl=vd(0),Ch=vd(1),T5=vd(2),k5=vd(3),oc=vd(4),E5=vd(5),S5=vd(6),cW=yl.range,vTe=Ch.range,xTe=T5.range,bTe=k5.range,wTe=oc.range,TTe=E5.range,kTe=S5.range;o(xd,"utcWeekday");bd=xd(0),N0=xd(1),uW=xd(2),hW=xd(3),Ah=xd(4),fW=xd(5),dW=xd(6),pW=bd.range,ETe=N0.range,STe=uW.range,CTe=hW.range,ATe=Ah.range,_Te=fW.range,DTe=dW.range});var bu,LTe,C5,RTe,H_=N(()=>{"use strict";yu();bu=xn(t=>{t.setDate(1),t.setHours(0,0,0,0)},(t,e)=>{t.setMonth(t.getMonth()+e)},(t,e)=>e.getMonth()-t.getMonth()+(e.getFullYear()-t.getFullYear())*12,t=>t.getMonth()),LTe=bu.range,C5=xn(t=>{t.setUTCDate(1),t.setUTCHours(0,0,0,0)},(t,e)=>{t.setUTCMonth(t.getUTCMonth()+e)},(t,e)=>e.getUTCMonth()-t.getUTCMonth()+(e.getUTCFullYear()-t.getUTCFullYear())*12,t=>t.getUTCMonth()),RTe=C5.range});var Qs,NTe,vl,MTe,W_=N(()=>{"use strict";yu();Qs=xn(t=>{t.setMonth(0,1),t.setHours(0,0,0,0)},(t,e)=>{t.setFullYear(t.getFullYear()+e)},(t,e)=>e.getFullYear()-t.getFullYear(),t=>t.getFullYear());Qs.every=t=>!isFinite(t=Math.floor(t))||!(t>0)?null:xn(e=>{e.setFullYear(Math.floor(e.getFullYear()/t)*t),e.setMonth(0,1),e.setHours(0,0,0,0)},(e,r)=>{e.setFullYear(e.getFullYear()+r*t)});NTe=Qs.range,vl=xn(t=>{t.setUTCMonth(0,1),t.setUTCHours(0,0,0,0)},(t,e)=>{t.setUTCFullYear(t.getUTCFullYear()+e)},(t,e)=>e.getUTCFullYear()-t.getUTCFullYear(),t=>t.getUTCFullYear());vl.every=t=>!isFinite(t=Math.floor(t))||!(t>0)?null:xn(e=>{e.setUTCFullYear(Math.floor(e.getUTCFullYear()/t)*t),e.setUTCMonth(0,1),e.setUTCHours(0,0,0,0)},(e,r)=>{e.setUTCFullYear(e.getUTCFullYear()+r*t)});MTe=vl.range});function gW(t,e,r,n,i,a){let s=[[Ks,1,1e3],[Ks,5,5*1e3],[Ks,15,15*1e3],[Ks,30,30*1e3],[a,1,6e4],[a,5,5*6e4],[a,15,15*6e4],[a,30,30*6e4],[i,1,36e5],[i,3,3*36e5],[i,6,6*36e5],[i,12,12*36e5],[n,1,864e5],[n,2,2*864e5],[r,1,6048e5],[e,1,2592e6],[e,3,3*2592e6],[t,1,31536e6]];function l(h,f,d){let p=fv).right(s,p);if(m===s.length)return t.every(y0(h/31536e6,f/31536e6,d));if(m===0)return ac.every(Math.max(y0(h,f,d),1));let[g,y]=s[p/s[m-1][2]{"use strict";vh();F_();$_();z_();G_();V_();U_();H_();W_();o(gW,"ticker");[OTe,PTe]=gW(vl,C5,bd,w5,b5,x5),[q_,Y_]=gW(Qs,bu,yl,_o,xu,vu)});var A5=N(()=>{"use strict";F_();$_();z_();G_();V_();U_();H_();W_();yW()});function X_(t){if(0<=t.y&&t.y<100){var e=new Date(-1,t.m,t.d,t.H,t.M,t.S,t.L);return e.setFullYear(t.y),e}return new Date(t.y,t.m,t.d,t.H,t.M,t.S,t.L)}function j_(t){if(0<=t.y&&t.y<100){var e=new Date(Date.UTC(-1,t.m,t.d,t.H,t.M,t.S,t.L));return e.setUTCFullYear(t.y),e}return new Date(Date.UTC(t.y,t.m,t.d,t.H,t.M,t.S,t.L))}function Cv(t,e,r){return{y:t,m:e,d:r,H:0,M:0,S:0,L:0}}function K_(t){var e=t.dateTime,r=t.date,n=t.time,i=t.periods,a=t.days,s=t.shortDays,l=t.months,u=t.shortMonths,h=Av(i),f=_v(i),d=Av(a),p=_v(a),m=Av(s),g=_v(s),y=Av(l),v=_v(l),x=Av(u),b=_v(u),w={a:B,A:F,b:P,B:z,c:null,d:kW,e:kW,f:ake,g:mke,G:yke,H:rke,I:nke,j:ike,L:_W,m:ske,M:oke,p:$,q:H,Q:CW,s:AW,S:lke,u:cke,U:uke,V:hke,w:fke,W:dke,x:null,X:null,y:pke,Y:gke,Z:vke,"%":SW},C={a:Q,A:j,b:ie,B:ne,c:null,d:EW,e:EW,f:Tke,g:Nke,G:Ike,H:xke,I:bke,j:wke,L:LW,m:kke,M:Eke,p:le,q:he,Q:CW,s:AW,S:Ske,u:Cke,U:Ake,V:_ke,w:Dke,W:Lke,x:null,X:null,y:Rke,Y:Mke,Z:Oke,"%":SW},T={a:I,A:D,b:k,B:L,c:R,d:wW,e:wW,f:ZTe,g:bW,G:xW,H:TW,I:TW,j:XTe,L:QTe,m:YTe,M:jTe,p:_,q:qTe,Q:eke,s:tke,S:KTe,u:GTe,U:VTe,V:UTe,w:zTe,W:HTe,x:O,X:M,y:bW,Y:xW,Z:WTe,"%":JTe};w.x=E(r,w),w.X=E(n,w),w.c=E(e,w),C.x=E(r,C),C.X=E(n,C),C.c=E(e,C);function E(K,X){return function(te){var J=[],se=-1,ue=0,Z=K.length,Se,ce,ae;for(te instanceof Date||(te=new Date(+te));++se53)return null;"w"in J||(J.w=1),"Z"in J?(ue=j_(Cv(J.y,0,1)),Z=ue.getUTCDay(),ue=Z>4||Z===0?N0.ceil(ue):N0(ue),ue=Sv.offset(ue,(J.V-1)*7),J.y=ue.getUTCFullYear(),J.m=ue.getUTCMonth(),J.d=ue.getUTCDate()+(J.w+6)%7):(ue=X_(Cv(J.y,0,1)),Z=ue.getDay(),ue=Z>4||Z===0?Ch.ceil(ue):Ch(ue),ue=_o.offset(ue,(J.V-1)*7),J.y=ue.getFullYear(),J.m=ue.getMonth(),J.d=ue.getDate()+(J.w+6)%7)}else("W"in J||"U"in J)&&("w"in J||(J.w="u"in J?J.u%7:"W"in J?1:0),Z="Z"in J?j_(Cv(J.y,0,1)).getUTCDay():X_(Cv(J.y,0,1)).getDay(),J.m=0,J.d="W"in J?(J.w+6)%7+J.W*7-(Z+5)%7:J.w+J.U*7-(Z+6)%7);return"Z"in J?(J.H+=J.Z/100|0,J.M+=J.Z%100,j_(J)):X_(J)}}o(A,"newParse");function S(K,X,te,J){for(var se=0,ue=X.length,Z=te.length,Se,ce;se=Z)return-1;if(Se=X.charCodeAt(se++),Se===37){if(Se=X.charAt(se++),ce=T[Se in vW?X.charAt(se++):Se],!ce||(J=ce(K,te,J))<0)return-1}else if(Se!=te.charCodeAt(J++))return-1}return J}o(S,"parseSpecifier");function _(K,X,te){var J=h.exec(X.slice(te));return J?(K.p=f.get(J[0].toLowerCase()),te+J[0].length):-1}o(_,"parsePeriod");function I(K,X,te){var J=m.exec(X.slice(te));return J?(K.w=g.get(J[0].toLowerCase()),te+J[0].length):-1}o(I,"parseShortWeekday");function D(K,X,te){var J=d.exec(X.slice(te));return J?(K.w=p.get(J[0].toLowerCase()),te+J[0].length):-1}o(D,"parseWeekday");function k(K,X,te){var J=x.exec(X.slice(te));return J?(K.m=b.get(J[0].toLowerCase()),te+J[0].length):-1}o(k,"parseShortMonth");function L(K,X,te){var J=y.exec(X.slice(te));return J?(K.m=v.get(J[0].toLowerCase()),te+J[0].length):-1}o(L,"parseMonth");function R(K,X,te){return S(K,e,X,te)}o(R,"parseLocaleDateTime");function O(K,X,te){return S(K,r,X,te)}o(O,"parseLocaleDate");function M(K,X,te){return S(K,n,X,te)}o(M,"parseLocaleTime");function B(K){return s[K.getDay()]}o(B,"formatShortWeekday");function F(K){return a[K.getDay()]}o(F,"formatWeekday");function P(K){return u[K.getMonth()]}o(P,"formatShortMonth");function z(K){return l[K.getMonth()]}o(z,"formatMonth");function $(K){return i[+(K.getHours()>=12)]}o($,"formatPeriod");function H(K){return 1+~~(K.getMonth()/3)}o(H,"formatQuarter");function Q(K){return s[K.getUTCDay()]}o(Q,"formatUTCShortWeekday");function j(K){return a[K.getUTCDay()]}o(j,"formatUTCWeekday");function ie(K){return u[K.getUTCMonth()]}o(ie,"formatUTCShortMonth");function ne(K){return l[K.getUTCMonth()]}o(ne,"formatUTCMonth");function le(K){return i[+(K.getUTCHours()>=12)]}o(le,"formatUTCPeriod");function he(K){return 1+~~(K.getUTCMonth()/3)}return o(he,"formatUTCQuarter"),{format:o(function(K){var X=E(K+="",w);return X.toString=function(){return K},X},"format"),parse:o(function(K){var X=A(K+="",!1);return X.toString=function(){return K},X},"parse"),utcFormat:o(function(K){var X=E(K+="",C);return X.toString=function(){return K},X},"utcFormat"),utcParse:o(function(K){var X=A(K+="",!0);return X.toString=function(){return K},X},"utcParse")}}function Wr(t,e,r){var n=t<0?"-":"",i=(n?-t:t)+"",a=i.length;return n+(a[e.toLowerCase(),r]))}function zTe(t,e,r){var n=Qi.exec(e.slice(r,r+1));return n?(t.w=+n[0],r+n[0].length):-1}function GTe(t,e,r){var n=Qi.exec(e.slice(r,r+1));return n?(t.u=+n[0],r+n[0].length):-1}function VTe(t,e,r){var n=Qi.exec(e.slice(r,r+2));return n?(t.U=+n[0],r+n[0].length):-1}function UTe(t,e,r){var n=Qi.exec(e.slice(r,r+2));return n?(t.V=+n[0],r+n[0].length):-1}function HTe(t,e,r){var n=Qi.exec(e.slice(r,r+2));return n?(t.W=+n[0],r+n[0].length):-1}function xW(t,e,r){var n=Qi.exec(e.slice(r,r+4));return n?(t.y=+n[0],r+n[0].length):-1}function bW(t,e,r){var n=Qi.exec(e.slice(r,r+2));return n?(t.y=+n[0]+(+n[0]>68?1900:2e3),r+n[0].length):-1}function WTe(t,e,r){var n=/^(Z)|([+-]\d\d)(?::?(\d\d))?/.exec(e.slice(r,r+6));return n?(t.Z=n[1]?0:-(n[2]+(n[3]||"00")),r+n[0].length):-1}function qTe(t,e,r){var n=Qi.exec(e.slice(r,r+1));return n?(t.q=n[0]*3-3,r+n[0].length):-1}function YTe(t,e,r){var n=Qi.exec(e.slice(r,r+2));return n?(t.m=n[0]-1,r+n[0].length):-1}function wW(t,e,r){var n=Qi.exec(e.slice(r,r+2));return n?(t.d=+n[0],r+n[0].length):-1}function XTe(t,e,r){var n=Qi.exec(e.slice(r,r+3));return n?(t.m=0,t.d=+n[0],r+n[0].length):-1}function TW(t,e,r){var n=Qi.exec(e.slice(r,r+2));return n?(t.H=+n[0],r+n[0].length):-1}function jTe(t,e,r){var n=Qi.exec(e.slice(r,r+2));return n?(t.M=+n[0],r+n[0].length):-1}function KTe(t,e,r){var n=Qi.exec(e.slice(r,r+2));return n?(t.S=+n[0],r+n[0].length):-1}function QTe(t,e,r){var n=Qi.exec(e.slice(r,r+3));return n?(t.L=+n[0],r+n[0].length):-1}function ZTe(t,e,r){var n=Qi.exec(e.slice(r,r+6));return n?(t.L=Math.floor(n[0]/1e3),r+n[0].length):-1}function JTe(t,e,r){var n=BTe.exec(e.slice(r,r+1));return n?r+n[0].length:-1}function eke(t,e,r){var n=Qi.exec(e.slice(r));return n?(t.Q=+n[0],r+n[0].length):-1}function tke(t,e,r){var n=Qi.exec(e.slice(r));return n?(t.s=+n[0],r+n[0].length):-1}function kW(t,e){return Wr(t.getDate(),e,2)}function rke(t,e){return Wr(t.getHours(),e,2)}function nke(t,e){return Wr(t.getHours()%12||12,e,2)}function ike(t,e){return Wr(1+_o.count(Qs(t),t),e,3)}function _W(t,e){return Wr(t.getMilliseconds(),e,3)}function ake(t,e){return _W(t,e)+"000"}function ske(t,e){return Wr(t.getMonth()+1,e,2)}function oke(t,e){return Wr(t.getMinutes(),e,2)}function lke(t,e){return Wr(t.getSeconds(),e,2)}function cke(t){var e=t.getDay();return e===0?7:e}function uke(t,e){return Wr(yl.count(Qs(t)-1,t),e,2)}function DW(t){var e=t.getDay();return e>=4||e===0?oc(t):oc.ceil(t)}function hke(t,e){return t=DW(t),Wr(oc.count(Qs(t),t)+(Qs(t).getDay()===4),e,2)}function fke(t){return t.getDay()}function dke(t,e){return Wr(Ch.count(Qs(t)-1,t),e,2)}function pke(t,e){return Wr(t.getFullYear()%100,e,2)}function mke(t,e){return t=DW(t),Wr(t.getFullYear()%100,e,2)}function gke(t,e){return Wr(t.getFullYear()%1e4,e,4)}function yke(t,e){var r=t.getDay();return t=r>=4||r===0?oc(t):oc.ceil(t),Wr(t.getFullYear()%1e4,e,4)}function vke(t){var e=t.getTimezoneOffset();return(e>0?"-":(e*=-1,"+"))+Wr(e/60|0,"0",2)+Wr(e%60,"0",2)}function EW(t,e){return Wr(t.getUTCDate(),e,2)}function xke(t,e){return Wr(t.getUTCHours(),e,2)}function bke(t,e){return Wr(t.getUTCHours()%12||12,e,2)}function wke(t,e){return Wr(1+Sv.count(vl(t),t),e,3)}function LW(t,e){return Wr(t.getUTCMilliseconds(),e,3)}function Tke(t,e){return LW(t,e)+"000"}function kke(t,e){return Wr(t.getUTCMonth()+1,e,2)}function Eke(t,e){return Wr(t.getUTCMinutes(),e,2)}function Ske(t,e){return Wr(t.getUTCSeconds(),e,2)}function Cke(t){var e=t.getUTCDay();return e===0?7:e}function Ake(t,e){return Wr(bd.count(vl(t)-1,t),e,2)}function RW(t){var e=t.getUTCDay();return e>=4||e===0?Ah(t):Ah.ceil(t)}function _ke(t,e){return t=RW(t),Wr(Ah.count(vl(t),t)+(vl(t).getUTCDay()===4),e,2)}function Dke(t){return t.getUTCDay()}function Lke(t,e){return Wr(N0.count(vl(t)-1,t),e,2)}function Rke(t,e){return Wr(t.getUTCFullYear()%100,e,2)}function Nke(t,e){return t=RW(t),Wr(t.getUTCFullYear()%100,e,2)}function Mke(t,e){return Wr(t.getUTCFullYear()%1e4,e,4)}function Ike(t,e){var r=t.getUTCDay();return t=r>=4||r===0?Ah(t):Ah.ceil(t),Wr(t.getUTCFullYear()%1e4,e,4)}function Oke(){return"+0000"}function SW(){return"%"}function CW(t){return+t}function AW(t){return Math.floor(+t/1e3)}var vW,Qi,BTe,FTe,NW=N(()=>{"use strict";A5();o(X_,"localDate");o(j_,"utcDate");o(Cv,"newDate");o(K_,"formatLocale");vW={"-":"",_:" ",0:"0"},Qi=/^\s*\d+/,BTe=/^%/,FTe=/[\\^$*+?|[\]().{}]/g;o(Wr,"pad");o($Te,"requote");o(Av,"formatRe");o(_v,"formatLookup");o(zTe,"parseWeekdayNumberSunday");o(GTe,"parseWeekdayNumberMonday");o(VTe,"parseWeekNumberSunday");o(UTe,"parseWeekNumberISO");o(HTe,"parseWeekNumberMonday");o(xW,"parseFullYear");o(bW,"parseYear");o(WTe,"parseZone");o(qTe,"parseQuarter");o(YTe,"parseMonthNumber");o(wW,"parseDayOfMonth");o(XTe,"parseDayOfYear");o(TW,"parseHour24");o(jTe,"parseMinutes");o(KTe,"parseSeconds");o(QTe,"parseMilliseconds");o(ZTe,"parseMicroseconds");o(JTe,"parseLiteralPercent");o(eke,"parseUnixTimestamp");o(tke,"parseUnixTimestampSeconds");o(kW,"formatDayOfMonth");o(rke,"formatHour24");o(nke,"formatHour12");o(ike,"formatDayOfYear");o(_W,"formatMilliseconds");o(ake,"formatMicroseconds");o(ske,"formatMonthNumber");o(oke,"formatMinutes");o(lke,"formatSeconds");o(cke,"formatWeekdayNumberMonday");o(uke,"formatWeekNumberSunday");o(DW,"dISO");o(hke,"formatWeekNumberISO");o(fke,"formatWeekdayNumberSunday");o(dke,"formatWeekNumberMonday");o(pke,"formatYear");o(mke,"formatYearISO");o(gke,"formatFullYear");o(yke,"formatFullYearISO");o(vke,"formatZone");o(EW,"formatUTCDayOfMonth");o(xke,"formatUTCHour24");o(bke,"formatUTCHour12");o(wke,"formatUTCDayOfYear");o(LW,"formatUTCMilliseconds");o(Tke,"formatUTCMicroseconds");o(kke,"formatUTCMonthNumber");o(Eke,"formatUTCMinutes");o(Ske,"formatUTCSeconds");o(Cke,"formatUTCWeekdayNumberMonday");o(Ake,"formatUTCWeekNumberSunday");o(RW,"UTCdISO");o(_ke,"formatUTCWeekNumberISO");o(Dke,"formatUTCWeekdayNumberSunday");o(Lke,"formatUTCWeekNumberMonday");o(Rke,"formatUTCYear");o(Nke,"formatUTCYearISO");o(Mke,"formatUTCFullYear");o(Ike,"formatUTCFullYearISO");o(Oke,"formatUTCZone");o(SW,"formatLiteralPercent");o(CW,"formatUnixTimestamp");o(AW,"formatUnixTimestampSeconds")});function Q_(t){return M0=K_(t),wd=M0.format,MW=M0.parse,IW=M0.utcFormat,OW=M0.utcParse,M0}var M0,wd,MW,IW,OW,PW=N(()=>{"use strict";NW();Q_({dateTime:"%x, %X",date:"%-m/%-d/%Y",time:"%-I:%M:%S %p",periods:["AM","PM"],days:["Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"],shortDays:["Sun","Mon","Tue","Wed","Thu","Fri","Sat"],months:["January","February","March","April","May","June","July","August","September","October","November","December"],shortMonths:["Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"]});o(Q_,"defaultLocale")});var Z_=N(()=>{"use strict";PW()});function Pke(t){return new Date(t)}function Bke(t){return t instanceof Date?+t:+new Date(+t)}function BW(t,e,r,n,i,a,s,l,u,h){var f=kv(),d=f.invert,p=f.domain,m=h(".%L"),g=h(":%S"),y=h("%I:%M"),v=h("%I %p"),x=h("%a %d"),b=h("%b %d"),w=h("%B"),C=h("%Y");function T(E){return(u(E){"use strict";A5();Z_();M_();Tv();aW();o(Pke,"date");o(Bke,"number");o(BW,"calendar");o(_5,"time")});var $W=N(()=>{"use strict";JH();iW();D_();FW()});function J_(t){for(var e=t.length/6|0,r=new Array(e),n=0;n{"use strict";o(J_,"default")});var e9,GW=N(()=>{"use strict";zW();e9=J_("4e79a7f28e2ce1575976b7b259a14fedc949af7aa1ff9da79c755fbab0ab")});var VW=N(()=>{"use strict";GW()});function Bn(t){return o(function(){return t},"constant")}var D5=N(()=>{"use strict";o(Bn,"default")});function HW(t){return t>1?0:t<-1?I0:Math.acos(t)}function r9(t){return t>=1?Dv:t<=-1?-Dv:Math.asin(t)}var t9,fa,_h,UW,L5,xl,Td,Zi,I0,Dv,O0,R5=N(()=>{"use strict";t9=Math.abs,fa=Math.atan2,_h=Math.cos,UW=Math.max,L5=Math.min,xl=Math.sin,Td=Math.sqrt,Zi=1e-12,I0=Math.PI,Dv=I0/2,O0=2*I0;o(HW,"acos");o(r9,"asin")});function N5(t){let e=3;return t.digits=function(r){if(!arguments.length)return e;if(r==null)e=null;else{let n=Math.floor(r);if(!(n>=0))throw new RangeError(`invalid digits: ${r}`);e=n}return t},()=>new pd(e)}var n9=N(()=>{"use strict";m_();o(N5,"withPath")});function Fke(t){return t.innerRadius}function $ke(t){return t.outerRadius}function zke(t){return t.startAngle}function Gke(t){return t.endAngle}function Vke(t){return t&&t.padAngle}function Uke(t,e,r,n,i,a,s,l){var u=r-t,h=n-e,f=s-i,d=l-a,p=d*u-f*h;if(!(p*pR*R+O*O&&(S=I,_=D),{cx:S,cy:_,x01:-f,y01:-d,x11:S*(i/T-1),y11:_*(i/T-1)}}function bl(){var t=Fke,e=$ke,r=Bn(0),n=null,i=zke,a=Gke,s=Vke,l=null,u=N5(h);function h(){var f,d,p=+t.apply(this,arguments),m=+e.apply(this,arguments),g=i.apply(this,arguments)-Dv,y=a.apply(this,arguments)-Dv,v=t9(y-g),x=y>g;if(l||(l=f=u()),mZi))l.moveTo(0,0);else if(v>O0-Zi)l.moveTo(m*_h(g),m*xl(g)),l.arc(0,0,m,g,y,!x),p>Zi&&(l.moveTo(p*_h(y),p*xl(y)),l.arc(0,0,p,y,g,x));else{var b=g,w=y,C=g,T=y,E=v,A=v,S=s.apply(this,arguments)/2,_=S>Zi&&(n?+n.apply(this,arguments):Td(p*p+m*m)),I=L5(t9(m-p)/2,+r.apply(this,arguments)),D=I,k=I,L,R;if(_>Zi){var O=r9(_/p*xl(S)),M=r9(_/m*xl(S));(E-=O*2)>Zi?(O*=x?1:-1,C+=O,T-=O):(E=0,C=T=(g+y)/2),(A-=M*2)>Zi?(M*=x?1:-1,b+=M,w-=M):(A=0,b=w=(g+y)/2)}var B=m*_h(b),F=m*xl(b),P=p*_h(T),z=p*xl(T);if(I>Zi){var $=m*_h(w),H=m*xl(w),Q=p*_h(C),j=p*xl(C),ie;if(vZi?k>Zi?(L=M5(Q,j,B,F,m,k,x),R=M5($,H,P,z,m,k,x),l.moveTo(L.cx+L.x01,L.cy+L.y01),kZi)||!(E>Zi)?l.lineTo(P,z):D>Zi?(L=M5(P,z,$,H,p,-D,x),R=M5(B,F,Q,j,p,-D,x),l.lineTo(L.cx+L.x01,L.cy+L.y01),D{"use strict";D5();R5();n9();o(Fke,"arcInnerRadius");o($ke,"arcOuterRadius");o(zke,"arcStartAngle");o(Gke,"arcEndAngle");o(Vke,"arcPadAngle");o(Uke,"intersect");o(M5,"cornerTangents");o(bl,"default")});function Lv(t){return typeof t=="object"&&"length"in t?t:Array.from(t)}var Nyt,i9=N(()=>{"use strict";Nyt=Array.prototype.slice;o(Lv,"default")});function qW(t){this._context=t}function wu(t){return new qW(t)}var a9=N(()=>{"use strict";o(qW,"Linear");qW.prototype={areaStart:o(function(){this._line=0},"areaStart"),areaEnd:o(function(){this._line=NaN},"areaEnd"),lineStart:o(function(){this._point=0},"lineStart"),lineEnd:o(function(){(this._line||this._line!==0&&this._point===1)&&this._context.closePath(),this._line=1-this._line},"lineEnd"),point:o(function(t,e){switch(t=+t,e=+e,this._point){case 0:this._point=1,this._line?this._context.lineTo(t,e):this._context.moveTo(t,e);break;case 1:this._point=2;default:this._context.lineTo(t,e);break}},"point")};o(wu,"default")});function YW(t){return t[0]}function XW(t){return t[1]}var jW=N(()=>{"use strict";o(YW,"x");o(XW,"y")});function wl(t,e){var r=Bn(!0),n=null,i=wu,a=null,s=N5(l);t=typeof t=="function"?t:t===void 0?YW:Bn(t),e=typeof e=="function"?e:e===void 0?XW:Bn(e);function l(u){var h,f=(u=Lv(u)).length,d,p=!1,m;for(n==null&&(a=i(m=s())),h=0;h<=f;++h)!(h{"use strict";i9();D5();a9();n9();jW();o(wl,"default")});function s9(t,e){return et?1:e>=t?0:NaN}var QW=N(()=>{"use strict";o(s9,"default")});function o9(t){return t}var ZW=N(()=>{"use strict";o(o9,"default")});function I5(){var t=o9,e=s9,r=null,n=Bn(0),i=Bn(O0),a=Bn(0);function s(l){var u,h=(l=Lv(l)).length,f,d,p=0,m=new Array(h),g=new Array(h),y=+n.apply(this,arguments),v=Math.min(O0,Math.max(-O0,i.apply(this,arguments)-y)),x,b=Math.min(Math.abs(v)/h,a.apply(this,arguments)),w=b*(v<0?-1:1),C;for(u=0;u0&&(p+=C);for(e!=null?m.sort(function(T,E){return e(g[T],g[E])}):r!=null&&m.sort(function(T,E){return r(l[T],l[E])}),u=0,d=p?(v-h*w)/p:0;u0?C*d:0)+w,g[f]={data:l[f],index:u,value:C,startAngle:y,endAngle:x,padAngle:b};return g}return o(s,"pie"),s.value=function(l){return arguments.length?(t=typeof l=="function"?l:Bn(+l),s):t},s.sortValues=function(l){return arguments.length?(e=l,r=null,s):e},s.sort=function(l){return arguments.length?(r=l,e=null,s):r},s.startAngle=function(l){return arguments.length?(n=typeof l=="function"?l:Bn(+l),s):n},s.endAngle=function(l){return arguments.length?(i=typeof l=="function"?l:Bn(+l),s):i},s.padAngle=function(l){return arguments.length?(a=typeof l=="function"?l:Bn(+l),s):a},s}var JW=N(()=>{"use strict";i9();D5();QW();ZW();R5();o(I5,"default")});function Rv(t){return new O5(t,!0)}function Nv(t){return new O5(t,!1)}var O5,eq=N(()=>{"use strict";O5=class{static{o(this,"Bump")}constructor(e,r){this._context=e,this._x=r}areaStart(){this._line=0}areaEnd(){this._line=NaN}lineStart(){this._point=0}lineEnd(){(this._line||this._line!==0&&this._point===1)&&this._context.closePath(),this._line=1-this._line}point(e,r){switch(e=+e,r=+r,this._point){case 0:{this._point=1,this._line?this._context.lineTo(e,r):this._context.moveTo(e,r);break}case 1:this._point=2;default:{this._x?this._context.bezierCurveTo(this._x0=(this._x0+e)/2,this._y0,this._x0,r,e,r):this._context.bezierCurveTo(this._x0,this._y0=(this._y0+r)/2,e,this._y0,e,r);break}}this._x0=e,this._y0=r}};o(Rv,"bumpX");o(Nv,"bumpY")});function Zs(){}var Mv=N(()=>{"use strict";o(Zs,"default")});function P0(t,e,r){t._context.bezierCurveTo((2*t._x0+t._x1)/3,(2*t._y0+t._y1)/3,(t._x0+2*t._x1)/3,(t._y0+2*t._y1)/3,(t._x0+4*t._x1+e)/6,(t._y0+4*t._y1+r)/6)}function Iv(t){this._context=t}function Do(t){return new Iv(t)}var Ov=N(()=>{"use strict";o(P0,"point");o(Iv,"Basis");Iv.prototype={areaStart:o(function(){this._line=0},"areaStart"),areaEnd:o(function(){this._line=NaN},"areaEnd"),lineStart:o(function(){this._x0=this._x1=this._y0=this._y1=NaN,this._point=0},"lineStart"),lineEnd:o(function(){switch(this._point){case 3:P0(this,this._x1,this._y1);case 2:this._context.lineTo(this._x1,this._y1);break}(this._line||this._line!==0&&this._point===1)&&this._context.closePath(),this._line=1-this._line},"lineEnd"),point:o(function(t,e){switch(t=+t,e=+e,this._point){case 0:this._point=1,this._line?this._context.lineTo(t,e):this._context.moveTo(t,e);break;case 1:this._point=2;break;case 2:this._point=3,this._context.lineTo((5*this._x0+this._x1)/6,(5*this._y0+this._y1)/6);default:P0(this,t,e);break}this._x0=this._x1,this._x1=t,this._y0=this._y1,this._y1=e},"point")};o(Do,"default")});function tq(t){this._context=t}function P5(t){return new tq(t)}var rq=N(()=>{"use strict";Mv();Ov();o(tq,"BasisClosed");tq.prototype={areaStart:Zs,areaEnd:Zs,lineStart:o(function(){this._x0=this._x1=this._x2=this._x3=this._x4=this._y0=this._y1=this._y2=this._y3=this._y4=NaN,this._point=0},"lineStart"),lineEnd:o(function(){switch(this._point){case 1:{this._context.moveTo(this._x2,this._y2),this._context.closePath();break}case 2:{this._context.moveTo((this._x2+2*this._x3)/3,(this._y2+2*this._y3)/3),this._context.lineTo((this._x3+2*this._x2)/3,(this._y3+2*this._y2)/3),this._context.closePath();break}case 3:{this.point(this._x2,this._y2),this.point(this._x3,this._y3),this.point(this._x4,this._y4);break}}},"lineEnd"),point:o(function(t,e){switch(t=+t,e=+e,this._point){case 0:this._point=1,this._x2=t,this._y2=e;break;case 1:this._point=2,this._x3=t,this._y3=e;break;case 2:this._point=3,this._x4=t,this._y4=e,this._context.moveTo((this._x0+4*this._x1+t)/6,(this._y0+4*this._y1+e)/6);break;default:P0(this,t,e);break}this._x0=this._x1,this._x1=t,this._y0=this._y1,this._y1=e},"point")};o(P5,"default")});function nq(t){this._context=t}function B5(t){return new nq(t)}var iq=N(()=>{"use strict";Ov();o(nq,"BasisOpen");nq.prototype={areaStart:o(function(){this._line=0},"areaStart"),areaEnd:o(function(){this._line=NaN},"areaEnd"),lineStart:o(function(){this._x0=this._x1=this._y0=this._y1=NaN,this._point=0},"lineStart"),lineEnd:o(function(){(this._line||this._line!==0&&this._point===3)&&this._context.closePath(),this._line=1-this._line},"lineEnd"),point:o(function(t,e){switch(t=+t,e=+e,this._point){case 0:this._point=1;break;case 1:this._point=2;break;case 2:this._point=3;var r=(this._x0+4*this._x1+t)/6,n=(this._y0+4*this._y1+e)/6;this._line?this._context.lineTo(r,n):this._context.moveTo(r,n);break;case 3:this._point=4;default:P0(this,t,e);break}this._x0=this._x1,this._x1=t,this._y0=this._y1,this._y1=e},"point")};o(B5,"default")});function aq(t,e){this._basis=new Iv(t),this._beta=e}var l9,sq=N(()=>{"use strict";Ov();o(aq,"Bundle");aq.prototype={lineStart:o(function(){this._x=[],this._y=[],this._basis.lineStart()},"lineStart"),lineEnd:o(function(){var t=this._x,e=this._y,r=t.length-1;if(r>0)for(var n=t[0],i=e[0],a=t[r]-n,s=e[r]-i,l=-1,u;++l<=r;)u=l/r,this._basis.point(this._beta*t[l]+(1-this._beta)*(n+u*a),this._beta*e[l]+(1-this._beta)*(i+u*s));this._x=this._y=null,this._basis.lineEnd()},"lineEnd"),point:o(function(t,e){this._x.push(+t),this._y.push(+e)},"point")};l9=o(function t(e){function r(n){return e===1?new Iv(n):new aq(n,e)}return o(r,"bundle"),r.beta=function(n){return t(+n)},r},"custom")(.85)});function B0(t,e,r){t._context.bezierCurveTo(t._x1+t._k*(t._x2-t._x0),t._y1+t._k*(t._y2-t._y0),t._x2+t._k*(t._x1-e),t._y2+t._k*(t._y1-r),t._x2,t._y2)}function F5(t,e){this._context=t,this._k=(1-e)/6}var Pv,Bv=N(()=>{"use strict";o(B0,"point");o(F5,"Cardinal");F5.prototype={areaStart:o(function(){this._line=0},"areaStart"),areaEnd:o(function(){this._line=NaN},"areaEnd"),lineStart:o(function(){this._x0=this._x1=this._x2=this._y0=this._y1=this._y2=NaN,this._point=0},"lineStart"),lineEnd:o(function(){switch(this._point){case 2:this._context.lineTo(this._x2,this._y2);break;case 3:B0(this,this._x1,this._y1);break}(this._line||this._line!==0&&this._point===1)&&this._context.closePath(),this._line=1-this._line},"lineEnd"),point:o(function(t,e){switch(t=+t,e=+e,this._point){case 0:this._point=1,this._line?this._context.lineTo(t,e):this._context.moveTo(t,e);break;case 1:this._point=2,this._x1=t,this._y1=e;break;case 2:this._point=3;default:B0(this,t,e);break}this._x0=this._x1,this._x1=this._x2,this._x2=t,this._y0=this._y1,this._y1=this._y2,this._y2=e},"point")};Pv=o(function t(e){function r(n){return new F5(n,e)}return o(r,"cardinal"),r.tension=function(n){return t(+n)},r},"custom")(0)});function $5(t,e){this._context=t,this._k=(1-e)/6}var c9,u9=N(()=>{"use strict";Mv();Bv();o($5,"CardinalClosed");$5.prototype={areaStart:Zs,areaEnd:Zs,lineStart:o(function(){this._x0=this._x1=this._x2=this._x3=this._x4=this._x5=this._y0=this._y1=this._y2=this._y3=this._y4=this._y5=NaN,this._point=0},"lineStart"),lineEnd:o(function(){switch(this._point){case 1:{this._context.moveTo(this._x3,this._y3),this._context.closePath();break}case 2:{this._context.lineTo(this._x3,this._y3),this._context.closePath();break}case 3:{this.point(this._x3,this._y3),this.point(this._x4,this._y4),this.point(this._x5,this._y5);break}}},"lineEnd"),point:o(function(t,e){switch(t=+t,e=+e,this._point){case 0:this._point=1,this._x3=t,this._y3=e;break;case 1:this._point=2,this._context.moveTo(this._x4=t,this._y4=e);break;case 2:this._point=3,this._x5=t,this._y5=e;break;default:B0(this,t,e);break}this._x0=this._x1,this._x1=this._x2,this._x2=t,this._y0=this._y1,this._y1=this._y2,this._y2=e},"point")};c9=o(function t(e){function r(n){return new $5(n,e)}return o(r,"cardinal"),r.tension=function(n){return t(+n)},r},"custom")(0)});function z5(t,e){this._context=t,this._k=(1-e)/6}var h9,f9=N(()=>{"use strict";Bv();o(z5,"CardinalOpen");z5.prototype={areaStart:o(function(){this._line=0},"areaStart"),areaEnd:o(function(){this._line=NaN},"areaEnd"),lineStart:o(function(){this._x0=this._x1=this._x2=this._y0=this._y1=this._y2=NaN,this._point=0},"lineStart"),lineEnd:o(function(){(this._line||this._line!==0&&this._point===3)&&this._context.closePath(),this._line=1-this._line},"lineEnd"),point:o(function(t,e){switch(t=+t,e=+e,this._point){case 0:this._point=1;break;case 1:this._point=2;break;case 2:this._point=3,this._line?this._context.lineTo(this._x2,this._y2):this._context.moveTo(this._x2,this._y2);break;case 3:this._point=4;default:B0(this,t,e);break}this._x0=this._x1,this._x1=this._x2,this._x2=t,this._y0=this._y1,this._y1=this._y2,this._y2=e},"point")};h9=o(function t(e){function r(n){return new z5(n,e)}return o(r,"cardinal"),r.tension=function(n){return t(+n)},r},"custom")(0)});function Fv(t,e,r){var n=t._x1,i=t._y1,a=t._x2,s=t._y2;if(t._l01_a>Zi){var l=2*t._l01_2a+3*t._l01_a*t._l12_a+t._l12_2a,u=3*t._l01_a*(t._l01_a+t._l12_a);n=(n*l-t._x0*t._l12_2a+t._x2*t._l01_2a)/u,i=(i*l-t._y0*t._l12_2a+t._y2*t._l01_2a)/u}if(t._l23_a>Zi){var h=2*t._l23_2a+3*t._l23_a*t._l12_a+t._l12_2a,f=3*t._l23_a*(t._l23_a+t._l12_a);a=(a*h+t._x1*t._l23_2a-e*t._l12_2a)/f,s=(s*h+t._y1*t._l23_2a-r*t._l12_2a)/f}t._context.bezierCurveTo(n,i,a,s,t._x2,t._y2)}function oq(t,e){this._context=t,this._alpha=e}var $v,G5=N(()=>{"use strict";R5();Bv();o(Fv,"point");o(oq,"CatmullRom");oq.prototype={areaStart:o(function(){this._line=0},"areaStart"),areaEnd:o(function(){this._line=NaN},"areaEnd"),lineStart:o(function(){this._x0=this._x1=this._x2=this._y0=this._y1=this._y2=NaN,this._l01_a=this._l12_a=this._l23_a=this._l01_2a=this._l12_2a=this._l23_2a=this._point=0},"lineStart"),lineEnd:o(function(){switch(this._point){case 2:this._context.lineTo(this._x2,this._y2);break;case 3:this.point(this._x2,this._y2);break}(this._line||this._line!==0&&this._point===1)&&this._context.closePath(),this._line=1-this._line},"lineEnd"),point:o(function(t,e){if(t=+t,e=+e,this._point){var r=this._x2-t,n=this._y2-e;this._l23_a=Math.sqrt(this._l23_2a=Math.pow(r*r+n*n,this._alpha))}switch(this._point){case 0:this._point=1,this._line?this._context.lineTo(t,e):this._context.moveTo(t,e);break;case 1:this._point=2;break;case 2:this._point=3;default:Fv(this,t,e);break}this._l01_a=this._l12_a,this._l12_a=this._l23_a,this._l01_2a=this._l12_2a,this._l12_2a=this._l23_2a,this._x0=this._x1,this._x1=this._x2,this._x2=t,this._y0=this._y1,this._y1=this._y2,this._y2=e},"point")};$v=o(function t(e){function r(n){return e?new oq(n,e):new F5(n,0)}return o(r,"catmullRom"),r.alpha=function(n){return t(+n)},r},"custom")(.5)});function lq(t,e){this._context=t,this._alpha=e}var d9,cq=N(()=>{"use strict";u9();Mv();G5();o(lq,"CatmullRomClosed");lq.prototype={areaStart:Zs,areaEnd:Zs,lineStart:o(function(){this._x0=this._x1=this._x2=this._x3=this._x4=this._x5=this._y0=this._y1=this._y2=this._y3=this._y4=this._y5=NaN,this._l01_a=this._l12_a=this._l23_a=this._l01_2a=this._l12_2a=this._l23_2a=this._point=0},"lineStart"),lineEnd:o(function(){switch(this._point){case 1:{this._context.moveTo(this._x3,this._y3),this._context.closePath();break}case 2:{this._context.lineTo(this._x3,this._y3),this._context.closePath();break}case 3:{this.point(this._x3,this._y3),this.point(this._x4,this._y4),this.point(this._x5,this._y5);break}}},"lineEnd"),point:o(function(t,e){if(t=+t,e=+e,this._point){var r=this._x2-t,n=this._y2-e;this._l23_a=Math.sqrt(this._l23_2a=Math.pow(r*r+n*n,this._alpha))}switch(this._point){case 0:this._point=1,this._x3=t,this._y3=e;break;case 1:this._point=2,this._context.moveTo(this._x4=t,this._y4=e);break;case 2:this._point=3,this._x5=t,this._y5=e;break;default:Fv(this,t,e);break}this._l01_a=this._l12_a,this._l12_a=this._l23_a,this._l01_2a=this._l12_2a,this._l12_2a=this._l23_2a,this._x0=this._x1,this._x1=this._x2,this._x2=t,this._y0=this._y1,this._y1=this._y2,this._y2=e},"point")};d9=o(function t(e){function r(n){return e?new lq(n,e):new $5(n,0)}return o(r,"catmullRom"),r.alpha=function(n){return t(+n)},r},"custom")(.5)});function uq(t,e){this._context=t,this._alpha=e}var p9,hq=N(()=>{"use strict";f9();G5();o(uq,"CatmullRomOpen");uq.prototype={areaStart:o(function(){this._line=0},"areaStart"),areaEnd:o(function(){this._line=NaN},"areaEnd"),lineStart:o(function(){this._x0=this._x1=this._x2=this._y0=this._y1=this._y2=NaN,this._l01_a=this._l12_a=this._l23_a=this._l01_2a=this._l12_2a=this._l23_2a=this._point=0},"lineStart"),lineEnd:o(function(){(this._line||this._line!==0&&this._point===3)&&this._context.closePath(),this._line=1-this._line},"lineEnd"),point:o(function(t,e){if(t=+t,e=+e,this._point){var r=this._x2-t,n=this._y2-e;this._l23_a=Math.sqrt(this._l23_2a=Math.pow(r*r+n*n,this._alpha))}switch(this._point){case 0:this._point=1;break;case 1:this._point=2;break;case 2:this._point=3,this._line?this._context.lineTo(this._x2,this._y2):this._context.moveTo(this._x2,this._y2);break;case 3:this._point=4;default:Fv(this,t,e);break}this._l01_a=this._l12_a,this._l12_a=this._l23_a,this._l01_2a=this._l12_2a,this._l12_2a=this._l23_2a,this._x0=this._x1,this._x1=this._x2,this._x2=t,this._y0=this._y1,this._y1=this._y2,this._y2=e},"point")};p9=o(function t(e){function r(n){return e?new uq(n,e):new z5(n,0)}return o(r,"catmullRom"),r.alpha=function(n){return t(+n)},r},"custom")(.5)});function fq(t){this._context=t}function V5(t){return new fq(t)}var dq=N(()=>{"use strict";Mv();o(fq,"LinearClosed");fq.prototype={areaStart:Zs,areaEnd:Zs,lineStart:o(function(){this._point=0},"lineStart"),lineEnd:o(function(){this._point&&this._context.closePath()},"lineEnd"),point:o(function(t,e){t=+t,e=+e,this._point?this._context.lineTo(t,e):(this._point=1,this._context.moveTo(t,e))},"point")};o(V5,"default")});function pq(t){return t<0?-1:1}function mq(t,e,r){var n=t._x1-t._x0,i=e-t._x1,a=(t._y1-t._y0)/(n||i<0&&-0),s=(r-t._y1)/(i||n<0&&-0),l=(a*i+s*n)/(n+i);return(pq(a)+pq(s))*Math.min(Math.abs(a),Math.abs(s),.5*Math.abs(l))||0}function gq(t,e){var r=t._x1-t._x0;return r?(3*(t._y1-t._y0)/r-e)/2:e}function m9(t,e,r){var n=t._x0,i=t._y0,a=t._x1,s=t._y1,l=(a-n)/3;t._context.bezierCurveTo(n+l,i+l*e,a-l,s-l*r,a,s)}function U5(t){this._context=t}function yq(t){this._context=new vq(t)}function vq(t){this._context=t}function zv(t){return new U5(t)}function Gv(t){return new yq(t)}var xq=N(()=>{"use strict";o(pq,"sign");o(mq,"slope3");o(gq,"slope2");o(m9,"point");o(U5,"MonotoneX");U5.prototype={areaStart:o(function(){this._line=0},"areaStart"),areaEnd:o(function(){this._line=NaN},"areaEnd"),lineStart:o(function(){this._x0=this._x1=this._y0=this._y1=this._t0=NaN,this._point=0},"lineStart"),lineEnd:o(function(){switch(this._point){case 2:this._context.lineTo(this._x1,this._y1);break;case 3:m9(this,this._t0,gq(this,this._t0));break}(this._line||this._line!==0&&this._point===1)&&this._context.closePath(),this._line=1-this._line},"lineEnd"),point:o(function(t,e){var r=NaN;if(t=+t,e=+e,!(t===this._x1&&e===this._y1)){switch(this._point){case 0:this._point=1,this._line?this._context.lineTo(t,e):this._context.moveTo(t,e);break;case 1:this._point=2;break;case 2:this._point=3,m9(this,gq(this,r=mq(this,t,e)),r);break;default:m9(this,this._t0,r=mq(this,t,e));break}this._x0=this._x1,this._x1=t,this._y0=this._y1,this._y1=e,this._t0=r}},"point")};o(yq,"MonotoneY");(yq.prototype=Object.create(U5.prototype)).point=function(t,e){U5.prototype.point.call(this,e,t)};o(vq,"ReflectContext");vq.prototype={moveTo:o(function(t,e){this._context.moveTo(e,t)},"moveTo"),closePath:o(function(){this._context.closePath()},"closePath"),lineTo:o(function(t,e){this._context.lineTo(e,t)},"lineTo"),bezierCurveTo:o(function(t,e,r,n,i,a){this._context.bezierCurveTo(e,t,n,r,a,i)},"bezierCurveTo")};o(zv,"monotoneX");o(Gv,"monotoneY")});function wq(t){this._context=t}function bq(t){var e,r=t.length-1,n,i=new Array(r),a=new Array(r),s=new Array(r);for(i[0]=0,a[0]=2,s[0]=t[0]+2*t[1],e=1;e=0;--e)i[e]=(s[e]-i[e+1])/a[e];for(a[r-1]=(t[r]+i[r-1])/2,e=0;e{"use strict";o(wq,"Natural");wq.prototype={areaStart:o(function(){this._line=0},"areaStart"),areaEnd:o(function(){this._line=NaN},"areaEnd"),lineStart:o(function(){this._x=[],this._y=[]},"lineStart"),lineEnd:o(function(){var t=this._x,e=this._y,r=t.length;if(r)if(this._line?this._context.lineTo(t[0],e[0]):this._context.moveTo(t[0],e[0]),r===2)this._context.lineTo(t[1],e[1]);else for(var n=bq(t),i=bq(e),a=0,s=1;s{"use strict";o(H5,"Step");H5.prototype={areaStart:o(function(){this._line=0},"areaStart"),areaEnd:o(function(){this._line=NaN},"areaEnd"),lineStart:o(function(){this._x=this._y=NaN,this._point=0},"lineStart"),lineEnd:o(function(){0=0&&(this._t=1-this._t,this._line=1-this._line)},"lineEnd"),point:o(function(t,e){switch(t=+t,e=+e,this._point){case 0:this._point=1,this._line?this._context.lineTo(t,e):this._context.moveTo(t,e);break;case 1:this._point=2;default:{if(this._t<=0)this._context.lineTo(this._x,e),this._context.lineTo(t,e);else{var r=this._x*(1-this._t)+t*this._t;this._context.lineTo(r,this._y),this._context.lineTo(r,e)}break}}this._x=t,this._y=e},"point")};o($0,"default");o(Vv,"stepBefore");o(Uv,"stepAfter")});var Eq=N(()=>{"use strict";WW();KW();JW();rq();iq();Ov();eq();sq();u9();f9();Bv();cq();hq();G5();dq();a9();xq();Tq();kq()});var Sq=N(()=>{"use strict"});var Cq=N(()=>{"use strict"});function Dh(t,e,r){this.k=t,this.x=e,this.y=r}function y9(t){for(;!t.__zoom;)if(!(t=t.parentNode))return g9;return t.__zoom}var g9,v9=N(()=>{"use strict";o(Dh,"Transform");Dh.prototype={constructor:Dh,scale:o(function(t){return t===1?this:new Dh(this.k*t,this.x,this.y)},"scale"),translate:o(function(t,e){return t===0&e===0?this:new Dh(this.k,this.x+this.k*t,this.y+this.k*e)},"translate"),apply:o(function(t){return[t[0]*this.k+this.x,t[1]*this.k+this.y]},"apply"),applyX:o(function(t){return t*this.k+this.x},"applyX"),applyY:o(function(t){return t*this.k+this.y},"applyY"),invert:o(function(t){return[(t[0]-this.x)/this.k,(t[1]-this.y)/this.k]},"invert"),invertX:o(function(t){return(t-this.x)/this.k},"invertX"),invertY:o(function(t){return(t-this.y)/this.k},"invertY"),rescaleX:o(function(t){return t.copy().domain(t.range().map(this.invertX,this).map(t.invert,t))},"rescaleX"),rescaleY:o(function(t){return t.copy().domain(t.range().map(this.invertY,this).map(t.invert,t))},"rescaleY"),toString:o(function(){return"translate("+this.x+","+this.y+") scale("+this.k+")"},"toString")};g9=new Dh(1,0,0);y9.prototype=Dh.prototype;o(y9,"transform")});var Aq=N(()=>{"use strict"});var _q=N(()=>{"use strict";l5();Sq();Cq();v9();Aq()});var Dq=N(()=>{"use strict";_q();v9()});var dr=N(()=>{"use strict";vh();sV();SH();DH();E0();LH();RH();TA();QV();NH();u_();MH();OH();A_();jH();KH();A0();m_();QH();IH();ZH();$W();VW();fl();Eq();A5();Z_();r5();l5();Dq()});var Lq=Mi(Ji=>{"use strict";Object.defineProperty(Ji,"__esModule",{value:!0});Ji.BLANK_URL=Ji.relativeFirstCharacters=Ji.whitespaceEscapeCharsRegex=Ji.urlSchemeRegex=Ji.ctrlCharactersRegex=Ji.htmlCtrlEntityRegex=Ji.htmlEntitiesRegex=Ji.invalidProtocolRegex=void 0;Ji.invalidProtocolRegex=/^([^\w]*)(javascript|data|vbscript)/im;Ji.htmlEntitiesRegex=/&#(\w+)(^\w|;)?/g;Ji.htmlCtrlEntityRegex=/&(newline|tab);/gi;Ji.ctrlCharactersRegex=/[\u0000-\u001F\u007F-\u009F\u2000-\u200D\uFEFF]/gim;Ji.urlSchemeRegex=/^.+(:|:)/gim;Ji.whitespaceEscapeCharsRegex=/(\\|%5[cC])((%(6[eE]|72|74))|[nrt])/g;Ji.relativeFirstCharacters=[".","/"];Ji.BLANK_URL="about:blank"});var z0=Mi(W5=>{"use strict";Object.defineProperty(W5,"__esModule",{value:!0});W5.sanitizeUrl=void 0;var Aa=Lq();function Hke(t){return Aa.relativeFirstCharacters.indexOf(t[0])>-1}o(Hke,"isRelativeUrlWithoutProtocol");function Wke(t){var e=t.replace(Aa.ctrlCharactersRegex,"");return e.replace(Aa.htmlEntitiesRegex,function(r,n){return String.fromCharCode(n)})}o(Wke,"decodeHtmlCharacters");function qke(t){return URL.canParse(t)}o(qke,"isValidUrl");function Rq(t){try{return decodeURIComponent(t)}catch{return t}}o(Rq,"decodeURI");function Yke(t){if(!t)return Aa.BLANK_URL;var e,r=Rq(t.trim());do r=Wke(r).replace(Aa.htmlCtrlEntityRegex,"").replace(Aa.ctrlCharactersRegex,"").replace(Aa.whitespaceEscapeCharsRegex,"").trim(),r=Rq(r),e=r.match(Aa.ctrlCharactersRegex)||r.match(Aa.htmlEntitiesRegex)||r.match(Aa.htmlCtrlEntityRegex)||r.match(Aa.whitespaceEscapeCharsRegex);while(e&&e.length>0);var n=r;if(!n)return Aa.BLANK_URL;if(Hke(n))return n;var i=n.trimStart(),a=i.match(Aa.urlSchemeRegex);if(!a)return n;var s=a[0].toLowerCase().trim();if(Aa.invalidProtocolRegex.test(s))return Aa.BLANK_URL;var l=i.replace(/\\/g,"/");if(s==="mailto:"||s.includes("://"))return l;if(s==="http:"||s==="https:"){if(!qke(l))return Aa.BLANK_URL;var u=new URL(l);return u.protocol=u.protocol.toLowerCase(),u.hostname=u.hostname.toLowerCase(),u.toString()}return l}o(Yke,"sanitizeUrl");W5.sanitizeUrl=Yke});var x9,kd,q5,Nq,Mq,Iq,Tl,Hv,Wv=N(()=>{"use strict";x9=Sa(z0(),1);gr();kd=o((t,e)=>{let r=t.append("rect");if(r.attr("x",e.x),r.attr("y",e.y),r.attr("fill",e.fill),r.attr("stroke",e.stroke),r.attr("width",e.width),r.attr("height",e.height),e.name&&r.attr("name",e.name),e.rx&&r.attr("rx",e.rx),e.ry&&r.attr("ry",e.ry),e.attrs!==void 0)for(let n in e.attrs)r.attr(n,e.attrs[n]);return e.class&&r.attr("class",e.class),r},"drawRect"),q5=o((t,e)=>{let r={x:e.startx,y:e.starty,width:e.stopx-e.startx,height:e.stopy-e.starty,fill:e.fill,stroke:e.stroke,class:"rect"};kd(t,r).lower()},"drawBackgroundRect"),Nq=o((t,e)=>{let r=e.text.replace(nd," "),n=t.append("text");n.attr("x",e.x),n.attr("y",e.y),n.attr("class","legend"),n.style("text-anchor",e.anchor),e.class&&n.attr("class",e.class);let i=n.append("tspan");return i.attr("x",e.x+e.textMargin*2),i.text(r),n},"drawText"),Mq=o((t,e,r,n)=>{let i=t.append("image");i.attr("x",e),i.attr("y",r);let a=(0,x9.sanitizeUrl)(n);i.attr("xlink:href",a)},"drawImage"),Iq=o((t,e,r,n)=>{let i=t.append("use");i.attr("x",e),i.attr("y",r);let a=(0,x9.sanitizeUrl)(n);i.attr("xlink:href",`#${a}`)},"drawEmbeddedImage"),Tl=o(()=>({x:0,y:0,width:100,height:100,fill:"#EDF2AE",stroke:"#666",anchor:"start",rx:0,ry:0}),"getNoteRect"),Hv=o(()=>({x:0,y:0,width:100,height:100,"text-anchor":"start",style:"#666",textMargin:0,rx:0,ry:0,tspan:!0}),"getTextObj")});var Oq,b9,Pq,Xke,jke,Kke,Qke,Zke,Jke,eEe,tEe,rEe,nEe,iEe,aEe,Tu,kl,Bq=N(()=>{"use strict";gr();Wv();Oq=Sa(z0(),1),b9=o(function(t,e){return kd(t,e)},"drawRect"),Pq=o(function(t,e,r,n,i,a){let s=t.append("image");s.attr("width",e),s.attr("height",r),s.attr("x",n),s.attr("y",i);let l=a.startsWith("data:image/png;base64")?a:(0,Oq.sanitizeUrl)(a);s.attr("xlink:href",l)},"drawImage"),Xke=o((t,e,r)=>{let n=t.append("g"),i=0;for(let a of e){let s=a.textColor?a.textColor:"#444444",l=a.lineColor?a.lineColor:"#444444",u=a.offsetX?parseInt(a.offsetX):0,h=a.offsetY?parseInt(a.offsetY):0,f="";if(i===0){let p=n.append("line");p.attr("x1",a.startPoint.x),p.attr("y1",a.startPoint.y),p.attr("x2",a.endPoint.x),p.attr("y2",a.endPoint.y),p.attr("stroke-width","1"),p.attr("stroke",l),p.style("fill","none"),a.type!=="rel_b"&&p.attr("marker-end","url("+f+"#arrowhead)"),(a.type==="birel"||a.type==="rel_b")&&p.attr("marker-start","url("+f+"#arrowend)"),i=-1}else{let p=n.append("path");p.attr("fill","none").attr("stroke-width","1").attr("stroke",l).attr("d","Mstartx,starty Qcontrolx,controly stopx,stopy ".replaceAll("startx",a.startPoint.x).replaceAll("starty",a.startPoint.y).replaceAll("controlx",a.startPoint.x+(a.endPoint.x-a.startPoint.x)/2-(a.endPoint.x-a.startPoint.x)/4).replaceAll("controly",a.startPoint.y+(a.endPoint.y-a.startPoint.y)/2).replaceAll("stopx",a.endPoint.x).replaceAll("stopy",a.endPoint.y)),a.type!=="rel_b"&&p.attr("marker-end","url("+f+"#arrowhead)"),(a.type==="birel"||a.type==="rel_b")&&p.attr("marker-start","url("+f+"#arrowend)")}let d=r.messageFont();Tu(r)(a.label.text,n,Math.min(a.startPoint.x,a.endPoint.x)+Math.abs(a.endPoint.x-a.startPoint.x)/2+u,Math.min(a.startPoint.y,a.endPoint.y)+Math.abs(a.endPoint.y-a.startPoint.y)/2+h,a.label.width,a.label.height,{fill:s},d),a.techn&&a.techn.text!==""&&(d=r.messageFont(),Tu(r)("["+a.techn.text+"]",n,Math.min(a.startPoint.x,a.endPoint.x)+Math.abs(a.endPoint.x-a.startPoint.x)/2+u,Math.min(a.startPoint.y,a.endPoint.y)+Math.abs(a.endPoint.y-a.startPoint.y)/2+r.messageFontSize+5+h,Math.max(a.label.width,a.techn.width),a.techn.height,{fill:s,"font-style":"italic"},d))}},"drawRels"),jke=o(function(t,e,r){let n=t.append("g"),i=e.bgColor?e.bgColor:"none",a=e.borderColor?e.borderColor:"#444444",s=e.fontColor?e.fontColor:"black",l={"stroke-width":1,"stroke-dasharray":"7.0,7.0"};e.nodeType&&(l={"stroke-width":1});let u={x:e.x,y:e.y,fill:i,stroke:a,width:e.width,height:e.height,rx:2.5,ry:2.5,attrs:l};b9(n,u);let h=r.boundaryFont();h.fontWeight="bold",h.fontSize=h.fontSize+2,h.fontColor=s,Tu(r)(e.label.text,n,e.x,e.y+e.label.Y,e.width,e.height,{fill:"#444444"},h),e.type&&e.type.text!==""&&(h=r.boundaryFont(),h.fontColor=s,Tu(r)(e.type.text,n,e.x,e.y+e.type.Y,e.width,e.height,{fill:"#444444"},h)),e.descr&&e.descr.text!==""&&(h=r.boundaryFont(),h.fontSize=h.fontSize-2,h.fontColor=s,Tu(r)(e.descr.text,n,e.x,e.y+e.descr.Y,e.width,e.height,{fill:"#444444"},h))},"drawBoundary"),Kke=o(function(t,e,r){let n=e.bgColor?e.bgColor:r[e.typeC4Shape.text+"_bg_color"],i=e.borderColor?e.borderColor:r[e.typeC4Shape.text+"_border_color"],a=e.fontColor?e.fontColor:"#FFFFFF",s="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAIAAADYYG7QAAACD0lEQVR4Xu2YoU4EMRCGT+4j8Ai8AhaH4QHgAUjQuFMECUgMIUgwJAgMhgQsAYUiJCiQIBBY+EITsjfTdme6V24v4c8vyGbb+ZjOtN0bNcvjQXmkH83WvYBWto6PLm6v7p7uH1/w2fXD+PBycX1Pv2l3IdDm/vn7x+dXQiAubRzoURa7gRZWd0iGRIiJbOnhnfYBQZNJjNbuyY2eJG8fkDE3bbG4ep6MHUAsgYxmE3nVs6VsBWJSGccsOlFPmLIViMzLOB7pCVO2AtHJMohH7Fh6zqitQK7m0rJvAVYgGcEpe//PLdDz65sM4pF9N7ICcXDKIB5Nv6j7tD0NoSdM2QrU9Gg0ewE1LqBhHR3BBdvj2vapnidjHxD/q6vd7Pvhr31AwcY8eXMTXAKECZZJFXuEq27aLgQK5uLMohCenGGuGewOxSjBvYBqeG6B+Nqiblggdjnc+ZXDy+FNFpFzw76O3UBAROuXh6FoiAcf5g9eTvUgzy0nWg6I8cXHRUpg5bOVBCo+KDpFajOf23GgPme7RSQ+lacIENUgJ6gg1k6HjgOlqnLqip4tEuhv0hNEMXUD0clyXE3p6pZA0S2nnvTlXwLJEZWlb7cTQH1+USgTN4VhAenm/wea1OCAOmqo6fE1WCb9WSKBah+rbUWPWAmE2Rvk0ApiB45eOyNAzU8xcTvj8KvkKEoOaIYeHNA3ZuygAvFMUO0AAAAASUVORK5CYII=";switch(e.typeC4Shape.text){case"person":s="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAIAAADYYG7QAAACD0lEQVR4Xu2YoU4EMRCGT+4j8Ai8AhaH4QHgAUjQuFMECUgMIUgwJAgMhgQsAYUiJCiQIBBY+EITsjfTdme6V24v4c8vyGbb+ZjOtN0bNcvjQXmkH83WvYBWto6PLm6v7p7uH1/w2fXD+PBycX1Pv2l3IdDm/vn7x+dXQiAubRzoURa7gRZWd0iGRIiJbOnhnfYBQZNJjNbuyY2eJG8fkDE3bbG4ep6MHUAsgYxmE3nVs6VsBWJSGccsOlFPmLIViMzLOB7pCVO2AtHJMohH7Fh6zqitQK7m0rJvAVYgGcEpe//PLdDz65sM4pF9N7ICcXDKIB5Nv6j7tD0NoSdM2QrU9Gg0ewE1LqBhHR3BBdvj2vapnidjHxD/q6vd7Pvhr31AwcY8eXMTXAKECZZJFXuEq27aLgQK5uLMohCenGGuGewOxSjBvYBqeG6B+Nqiblggdjnc+ZXDy+FNFpFzw76O3UBAROuXh6FoiAcf5g9eTvUgzy0nWg6I8cXHRUpg5bOVBCo+KDpFajOf23GgPme7RSQ+lacIENUgJ6gg1k6HjgOlqnLqip4tEuhv0hNEMXUD0clyXE3p6pZA0S2nnvTlXwLJEZWlb7cTQH1+USgTN4VhAenm/wea1OCAOmqo6fE1WCb9WSKBah+rbUWPWAmE2Rvk0ApiB45eOyNAzU8xcTvj8KvkKEoOaIYeHNA3ZuygAvFMUO0AAAAASUVORK5CYII=";break;case"external_person":s="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAIAAADYYG7QAAAB6ElEQVR4Xu2YLY+EMBCG9+dWr0aj0Wg0Go1Go0+j8Xdv2uTCvv1gpt0ebHKPuhDaeW4605Z9mJvx4AdXUyTUdd08z+u6flmWZRnHsWkafk9DptAwDPu+f0eAYtu2PEaGWuj5fCIZrBAC2eLBAnRCsEkkxmeaJp7iDJ2QMDdHsLg8SxKFEJaAo8lAXnmuOFIhTMpxxKATebo4UiFknuNo4OniSIXQyRxEA3YsnjGCVEjVXD7yLUAqxBGUyPv/Y4W2beMgGuS7kVQIBycH0fD+oi5pezQETxdHKmQKGk1eQEYldK+jw5GxPfZ9z7Mk0Qnhf1W1m3w//EUn5BDmSZsbR44QQLBEqrBHqOrmSKaQAxdnLArCrxZcM7A7ZKs4ioRq8LFC+NpC3WCBJsvpVw5edm9iEXFuyNfxXAgSwfrFQ1c0iNda8AdejvUgnktOtJQQxmcfFzGglc5WVCj7oDgFqU18boeFSs52CUh8LE8BIVQDT1ABrB0HtgSEYlX5doJnCwv9TXocKCaKbnwhdDKPq4lf3SwU3HLq4V/+WYhHVMa/3b4IlfyikAduCkcBc7mQ3/z/Qq/cTuikhkzB12Ae/mcJC9U+Vo8Ej1gWAtgbeGgFsAMHr50BIWOLCbezvhpBFUdY6EJuJ/QDW0XoMX60zZ0AAAAASUVORK5CYII=";break}let l=t.append("g");l.attr("class","person-man");let u=Tl();switch(e.typeC4Shape.text){case"person":case"external_person":case"system":case"external_system":case"container":case"external_container":case"component":case"external_component":u.x=e.x,u.y=e.y,u.fill=n,u.width=e.width,u.height=e.height,u.stroke=i,u.rx=2.5,u.ry=2.5,u.attrs={"stroke-width":.5},b9(l,u);break;case"system_db":case"external_system_db":case"container_db":case"external_container_db":case"component_db":case"external_component_db":l.append("path").attr("fill",n).attr("stroke-width","0.5").attr("stroke",i).attr("d","Mstartx,startyc0,-10 half,-10 half,-10c0,0 half,0 half,10l0,heightc0,10 -half,10 -half,10c0,0 -half,0 -half,-10l0,-height".replaceAll("startx",e.x).replaceAll("starty",e.y).replaceAll("half",e.width/2).replaceAll("height",e.height)),l.append("path").attr("fill","none").attr("stroke-width","0.5").attr("stroke",i).attr("d","Mstartx,startyc0,10 half,10 half,10c0,0 half,0 half,-10".replaceAll("startx",e.x).replaceAll("starty",e.y).replaceAll("half",e.width/2));break;case"system_queue":case"external_system_queue":case"container_queue":case"external_container_queue":case"component_queue":case"external_component_queue":l.append("path").attr("fill",n).attr("stroke-width","0.5").attr("stroke",i).attr("d","Mstartx,startylwidth,0c5,0 5,half 5,halfc0,0 0,half -5,halfl-width,0c-5,0 -5,-half -5,-halfc0,0 0,-half 5,-half".replaceAll("startx",e.x).replaceAll("starty",e.y).replaceAll("width",e.width).replaceAll("half",e.height/2)),l.append("path").attr("fill","none").attr("stroke-width","0.5").attr("stroke",i).attr("d","Mstartx,startyc-5,0 -5,half -5,halfc0,half 5,half 5,half".replaceAll("startx",e.x+e.width).replaceAll("starty",e.y).replaceAll("half",e.height/2));break}let h=aEe(r,e.typeC4Shape.text);switch(l.append("text").attr("fill",a).attr("font-family",h.fontFamily).attr("font-size",h.fontSize-2).attr("font-style","italic").attr("lengthAdjust","spacing").attr("textLength",e.typeC4Shape.width).attr("x",e.x+e.width/2-e.typeC4Shape.width/2).attr("y",e.y+e.typeC4Shape.Y).text("<<"+e.typeC4Shape.text+">>"),e.typeC4Shape.text){case"person":case"external_person":Pq(l,48,48,e.x+e.width/2-24,e.y+e.image.Y,s);break}let f=r[e.typeC4Shape.text+"Font"]();return f.fontWeight="bold",f.fontSize=f.fontSize+2,f.fontColor=a,Tu(r)(e.label.text,l,e.x,e.y+e.label.Y,e.width,e.height,{fill:a},f),f=r[e.typeC4Shape.text+"Font"](),f.fontColor=a,e.techn&&e.techn?.text!==""?Tu(r)(e.techn.text,l,e.x,e.y+e.techn.Y,e.width,e.height,{fill:a,"font-style":"italic"},f):e.type&&e.type.text!==""&&Tu(r)(e.type.text,l,e.x,e.y+e.type.Y,e.width,e.height,{fill:a,"font-style":"italic"},f),e.descr&&e.descr.text!==""&&(f=r.personFont(),f.fontColor=a,Tu(r)(e.descr.text,l,e.x,e.y+e.descr.Y,e.width,e.height,{fill:a},f)),e.height},"drawC4Shape"),Qke=o(function(t){t.append("defs").append("symbol").attr("id","database").attr("fill-rule","evenodd").attr("clip-rule","evenodd").append("path").attr("transform","scale(.5)").attr("d","M12.258.001l.256.004.255.005.253.008.251.01.249.012.247.015.246.016.242.019.241.02.239.023.236.024.233.027.231.028.229.031.225.032.223.034.22.036.217.038.214.04.211.041.208.043.205.045.201.046.198.048.194.05.191.051.187.053.183.054.18.056.175.057.172.059.168.06.163.061.16.063.155.064.15.066.074.033.073.033.071.034.07.034.069.035.068.035.067.035.066.035.064.036.064.036.062.036.06.036.06.037.058.037.058.037.055.038.055.038.053.038.052.038.051.039.05.039.048.039.047.039.045.04.044.04.043.04.041.04.04.041.039.041.037.041.036.041.034.041.033.042.032.042.03.042.029.042.027.042.026.043.024.043.023.043.021.043.02.043.018.044.017.043.015.044.013.044.012.044.011.045.009.044.007.045.006.045.004.045.002.045.001.045v17l-.001.045-.002.045-.004.045-.006.045-.007.045-.009.044-.011.045-.012.044-.013.044-.015.044-.017.043-.018.044-.02.043-.021.043-.023.043-.024.043-.026.043-.027.042-.029.042-.03.042-.032.042-.033.042-.034.041-.036.041-.037.041-.039.041-.04.041-.041.04-.043.04-.044.04-.045.04-.047.039-.048.039-.05.039-.051.039-.052.038-.053.038-.055.038-.055.038-.058.037-.058.037-.06.037-.06.036-.062.036-.064.036-.064.036-.066.035-.067.035-.068.035-.069.035-.07.034-.071.034-.073.033-.074.033-.15.066-.155.064-.16.063-.163.061-.168.06-.172.059-.175.057-.18.056-.183.054-.187.053-.191.051-.194.05-.198.048-.201.046-.205.045-.208.043-.211.041-.214.04-.217.038-.22.036-.223.034-.225.032-.229.031-.231.028-.233.027-.236.024-.239.023-.241.02-.242.019-.246.016-.247.015-.249.012-.251.01-.253.008-.255.005-.256.004-.258.001-.258-.001-.256-.004-.255-.005-.253-.008-.251-.01-.249-.012-.247-.015-.245-.016-.243-.019-.241-.02-.238-.023-.236-.024-.234-.027-.231-.028-.228-.031-.226-.032-.223-.034-.22-.036-.217-.038-.214-.04-.211-.041-.208-.043-.204-.045-.201-.046-.198-.048-.195-.05-.19-.051-.187-.053-.184-.054-.179-.056-.176-.057-.172-.059-.167-.06-.164-.061-.159-.063-.155-.064-.151-.066-.074-.033-.072-.033-.072-.034-.07-.034-.069-.035-.068-.035-.067-.035-.066-.035-.064-.036-.063-.036-.062-.036-.061-.036-.06-.037-.058-.037-.057-.037-.056-.038-.055-.038-.053-.038-.052-.038-.051-.039-.049-.039-.049-.039-.046-.039-.046-.04-.044-.04-.043-.04-.041-.04-.04-.041-.039-.041-.037-.041-.036-.041-.034-.041-.033-.042-.032-.042-.03-.042-.029-.042-.027-.042-.026-.043-.024-.043-.023-.043-.021-.043-.02-.043-.018-.044-.017-.043-.015-.044-.013-.044-.012-.044-.011-.045-.009-.044-.007-.045-.006-.045-.004-.045-.002-.045-.001-.045v-17l.001-.045.002-.045.004-.045.006-.045.007-.045.009-.044.011-.045.012-.044.013-.044.015-.044.017-.043.018-.044.02-.043.021-.043.023-.043.024-.043.026-.043.027-.042.029-.042.03-.042.032-.042.033-.042.034-.041.036-.041.037-.041.039-.041.04-.041.041-.04.043-.04.044-.04.046-.04.046-.039.049-.039.049-.039.051-.039.052-.038.053-.038.055-.038.056-.038.057-.037.058-.037.06-.037.061-.036.062-.036.063-.036.064-.036.066-.035.067-.035.068-.035.069-.035.07-.034.072-.034.072-.033.074-.033.151-.066.155-.064.159-.063.164-.061.167-.06.172-.059.176-.057.179-.056.184-.054.187-.053.19-.051.195-.05.198-.048.201-.046.204-.045.208-.043.211-.041.214-.04.217-.038.22-.036.223-.034.226-.032.228-.031.231-.028.234-.027.236-.024.238-.023.241-.02.243-.019.245-.016.247-.015.249-.012.251-.01.253-.008.255-.005.256-.004.258-.001.258.001zm-9.258 20.499v.01l.001.021.003.021.004.022.005.021.006.022.007.022.009.023.01.022.011.023.012.023.013.023.015.023.016.024.017.023.018.024.019.024.021.024.022.025.023.024.024.025.052.049.056.05.061.051.066.051.07.051.075.051.079.052.084.052.088.052.092.052.097.052.102.051.105.052.11.052.114.051.119.051.123.051.127.05.131.05.135.05.139.048.144.049.147.047.152.047.155.047.16.045.163.045.167.043.171.043.176.041.178.041.183.039.187.039.19.037.194.035.197.035.202.033.204.031.209.03.212.029.216.027.219.025.222.024.226.021.23.02.233.018.236.016.24.015.243.012.246.01.249.008.253.005.256.004.259.001.26-.001.257-.004.254-.005.25-.008.247-.011.244-.012.241-.014.237-.016.233-.018.231-.021.226-.021.224-.024.22-.026.216-.027.212-.028.21-.031.205-.031.202-.034.198-.034.194-.036.191-.037.187-.039.183-.04.179-.04.175-.042.172-.043.168-.044.163-.045.16-.046.155-.046.152-.047.148-.048.143-.049.139-.049.136-.05.131-.05.126-.05.123-.051.118-.052.114-.051.11-.052.106-.052.101-.052.096-.052.092-.052.088-.053.083-.051.079-.052.074-.052.07-.051.065-.051.06-.051.056-.05.051-.05.023-.024.023-.025.021-.024.02-.024.019-.024.018-.024.017-.024.015-.023.014-.024.013-.023.012-.023.01-.023.01-.022.008-.022.006-.022.006-.022.004-.022.004-.021.001-.021.001-.021v-4.127l-.077.055-.08.053-.083.054-.085.053-.087.052-.09.052-.093.051-.095.05-.097.05-.1.049-.102.049-.105.048-.106.047-.109.047-.111.046-.114.045-.115.045-.118.044-.12.043-.122.042-.124.042-.126.041-.128.04-.13.04-.132.038-.134.038-.135.037-.138.037-.139.035-.142.035-.143.034-.144.033-.147.032-.148.031-.15.03-.151.03-.153.029-.154.027-.156.027-.158.026-.159.025-.161.024-.162.023-.163.022-.165.021-.166.02-.167.019-.169.018-.169.017-.171.016-.173.015-.173.014-.175.013-.175.012-.177.011-.178.01-.179.008-.179.008-.181.006-.182.005-.182.004-.184.003-.184.002h-.37l-.184-.002-.184-.003-.182-.004-.182-.005-.181-.006-.179-.008-.179-.008-.178-.01-.176-.011-.176-.012-.175-.013-.173-.014-.172-.015-.171-.016-.17-.017-.169-.018-.167-.019-.166-.02-.165-.021-.163-.022-.162-.023-.161-.024-.159-.025-.157-.026-.156-.027-.155-.027-.153-.029-.151-.03-.15-.03-.148-.031-.146-.032-.145-.033-.143-.034-.141-.035-.14-.035-.137-.037-.136-.037-.134-.038-.132-.038-.13-.04-.128-.04-.126-.041-.124-.042-.122-.042-.12-.044-.117-.043-.116-.045-.113-.045-.112-.046-.109-.047-.106-.047-.105-.048-.102-.049-.1-.049-.097-.05-.095-.05-.093-.052-.09-.051-.087-.052-.085-.053-.083-.054-.08-.054-.077-.054v4.127zm0-5.654v.011l.001.021.003.021.004.021.005.022.006.022.007.022.009.022.01.022.011.023.012.023.013.023.015.024.016.023.017.024.018.024.019.024.021.024.022.024.023.025.024.024.052.05.056.05.061.05.066.051.07.051.075.052.079.051.084.052.088.052.092.052.097.052.102.052.105.052.11.051.114.051.119.052.123.05.127.051.131.05.135.049.139.049.144.048.147.048.152.047.155.046.16.045.163.045.167.044.171.042.176.042.178.04.183.04.187.038.19.037.194.036.197.034.202.033.204.032.209.03.212.028.216.027.219.025.222.024.226.022.23.02.233.018.236.016.24.014.243.012.246.01.249.008.253.006.256.003.259.001.26-.001.257-.003.254-.006.25-.008.247-.01.244-.012.241-.015.237-.016.233-.018.231-.02.226-.022.224-.024.22-.025.216-.027.212-.029.21-.03.205-.032.202-.033.198-.035.194-.036.191-.037.187-.039.183-.039.179-.041.175-.042.172-.043.168-.044.163-.045.16-.045.155-.047.152-.047.148-.048.143-.048.139-.05.136-.049.131-.05.126-.051.123-.051.118-.051.114-.052.11-.052.106-.052.101-.052.096-.052.092-.052.088-.052.083-.052.079-.052.074-.051.07-.052.065-.051.06-.05.056-.051.051-.049.023-.025.023-.024.021-.025.02-.024.019-.024.018-.024.017-.024.015-.023.014-.023.013-.024.012-.022.01-.023.01-.023.008-.022.006-.022.006-.022.004-.021.004-.022.001-.021.001-.021v-4.139l-.077.054-.08.054-.083.054-.085.052-.087.053-.09.051-.093.051-.095.051-.097.05-.1.049-.102.049-.105.048-.106.047-.109.047-.111.046-.114.045-.115.044-.118.044-.12.044-.122.042-.124.042-.126.041-.128.04-.13.039-.132.039-.134.038-.135.037-.138.036-.139.036-.142.035-.143.033-.144.033-.147.033-.148.031-.15.03-.151.03-.153.028-.154.028-.156.027-.158.026-.159.025-.161.024-.162.023-.163.022-.165.021-.166.02-.167.019-.169.018-.169.017-.171.016-.173.015-.173.014-.175.013-.175.012-.177.011-.178.009-.179.009-.179.007-.181.007-.182.005-.182.004-.184.003-.184.002h-.37l-.184-.002-.184-.003-.182-.004-.182-.005-.181-.007-.179-.007-.179-.009-.178-.009-.176-.011-.176-.012-.175-.013-.173-.014-.172-.015-.171-.016-.17-.017-.169-.018-.167-.019-.166-.02-.165-.021-.163-.022-.162-.023-.161-.024-.159-.025-.157-.026-.156-.027-.155-.028-.153-.028-.151-.03-.15-.03-.148-.031-.146-.033-.145-.033-.143-.033-.141-.035-.14-.036-.137-.036-.136-.037-.134-.038-.132-.039-.13-.039-.128-.04-.126-.041-.124-.042-.122-.043-.12-.043-.117-.044-.116-.044-.113-.046-.112-.046-.109-.046-.106-.047-.105-.048-.102-.049-.1-.049-.097-.05-.095-.051-.093-.051-.09-.051-.087-.053-.085-.052-.083-.054-.08-.054-.077-.054v4.139zm0-5.666v.011l.001.02.003.022.004.021.005.022.006.021.007.022.009.023.01.022.011.023.012.023.013.023.015.023.016.024.017.024.018.023.019.024.021.025.022.024.023.024.024.025.052.05.056.05.061.05.066.051.07.051.075.052.079.051.084.052.088.052.092.052.097.052.102.052.105.051.11.052.114.051.119.051.123.051.127.05.131.05.135.05.139.049.144.048.147.048.152.047.155.046.16.045.163.045.167.043.171.043.176.042.178.04.183.04.187.038.19.037.194.036.197.034.202.033.204.032.209.03.212.028.216.027.219.025.222.024.226.021.23.02.233.018.236.017.24.014.243.012.246.01.249.008.253.006.256.003.259.001.26-.001.257-.003.254-.006.25-.008.247-.01.244-.013.241-.014.237-.016.233-.018.231-.02.226-.022.224-.024.22-.025.216-.027.212-.029.21-.03.205-.032.202-.033.198-.035.194-.036.191-.037.187-.039.183-.039.179-.041.175-.042.172-.043.168-.044.163-.045.16-.045.155-.047.152-.047.148-.048.143-.049.139-.049.136-.049.131-.051.126-.05.123-.051.118-.052.114-.051.11-.052.106-.052.101-.052.096-.052.092-.052.088-.052.083-.052.079-.052.074-.052.07-.051.065-.051.06-.051.056-.05.051-.049.023-.025.023-.025.021-.024.02-.024.019-.024.018-.024.017-.024.015-.023.014-.024.013-.023.012-.023.01-.022.01-.023.008-.022.006-.022.006-.022.004-.022.004-.021.001-.021.001-.021v-4.153l-.077.054-.08.054-.083.053-.085.053-.087.053-.09.051-.093.051-.095.051-.097.05-.1.049-.102.048-.105.048-.106.048-.109.046-.111.046-.114.046-.115.044-.118.044-.12.043-.122.043-.124.042-.126.041-.128.04-.13.039-.132.039-.134.038-.135.037-.138.036-.139.036-.142.034-.143.034-.144.033-.147.032-.148.032-.15.03-.151.03-.153.028-.154.028-.156.027-.158.026-.159.024-.161.024-.162.023-.163.023-.165.021-.166.02-.167.019-.169.018-.169.017-.171.016-.173.015-.173.014-.175.013-.175.012-.177.01-.178.01-.179.009-.179.007-.181.006-.182.006-.182.004-.184.003-.184.001-.185.001-.185-.001-.184-.001-.184-.003-.182-.004-.182-.006-.181-.006-.179-.007-.179-.009-.178-.01-.176-.01-.176-.012-.175-.013-.173-.014-.172-.015-.171-.016-.17-.017-.169-.018-.167-.019-.166-.02-.165-.021-.163-.023-.162-.023-.161-.024-.159-.024-.157-.026-.156-.027-.155-.028-.153-.028-.151-.03-.15-.03-.148-.032-.146-.032-.145-.033-.143-.034-.141-.034-.14-.036-.137-.036-.136-.037-.134-.038-.132-.039-.13-.039-.128-.041-.126-.041-.124-.041-.122-.043-.12-.043-.117-.044-.116-.044-.113-.046-.112-.046-.109-.046-.106-.048-.105-.048-.102-.048-.1-.05-.097-.049-.095-.051-.093-.051-.09-.052-.087-.052-.085-.053-.083-.053-.08-.054-.077-.054v4.153zm8.74-8.179l-.257.004-.254.005-.25.008-.247.011-.244.012-.241.014-.237.016-.233.018-.231.021-.226.022-.224.023-.22.026-.216.027-.212.028-.21.031-.205.032-.202.033-.198.034-.194.036-.191.038-.187.038-.183.04-.179.041-.175.042-.172.043-.168.043-.163.045-.16.046-.155.046-.152.048-.148.048-.143.048-.139.049-.136.05-.131.05-.126.051-.123.051-.118.051-.114.052-.11.052-.106.052-.101.052-.096.052-.092.052-.088.052-.083.052-.079.052-.074.051-.07.052-.065.051-.06.05-.056.05-.051.05-.023.025-.023.024-.021.024-.02.025-.019.024-.018.024-.017.023-.015.024-.014.023-.013.023-.012.023-.01.023-.01.022-.008.022-.006.023-.006.021-.004.022-.004.021-.001.021-.001.021.001.021.001.021.004.021.004.022.006.021.006.023.008.022.01.022.01.023.012.023.013.023.014.023.015.024.017.023.018.024.019.024.02.025.021.024.023.024.023.025.051.05.056.05.06.05.065.051.07.052.074.051.079.052.083.052.088.052.092.052.096.052.101.052.106.052.11.052.114.052.118.051.123.051.126.051.131.05.136.05.139.049.143.048.148.048.152.048.155.046.16.046.163.045.168.043.172.043.175.042.179.041.183.04.187.038.191.038.194.036.198.034.202.033.205.032.21.031.212.028.216.027.22.026.224.023.226.022.231.021.233.018.237.016.241.014.244.012.247.011.25.008.254.005.257.004.26.001.26-.001.257-.004.254-.005.25-.008.247-.011.244-.012.241-.014.237-.016.233-.018.231-.021.226-.022.224-.023.22-.026.216-.027.212-.028.21-.031.205-.032.202-.033.198-.034.194-.036.191-.038.187-.038.183-.04.179-.041.175-.042.172-.043.168-.043.163-.045.16-.046.155-.046.152-.048.148-.048.143-.048.139-.049.136-.05.131-.05.126-.051.123-.051.118-.051.114-.052.11-.052.106-.052.101-.052.096-.052.092-.052.088-.052.083-.052.079-.052.074-.051.07-.052.065-.051.06-.05.056-.05.051-.05.023-.025.023-.024.021-.024.02-.025.019-.024.018-.024.017-.023.015-.024.014-.023.013-.023.012-.023.01-.023.01-.022.008-.022.006-.023.006-.021.004-.022.004-.021.001-.021.001-.021-.001-.021-.001-.021-.004-.021-.004-.022-.006-.021-.006-.023-.008-.022-.01-.022-.01-.023-.012-.023-.013-.023-.014-.023-.015-.024-.017-.023-.018-.024-.019-.024-.02-.025-.021-.024-.023-.024-.023-.025-.051-.05-.056-.05-.06-.05-.065-.051-.07-.052-.074-.051-.079-.052-.083-.052-.088-.052-.092-.052-.096-.052-.101-.052-.106-.052-.11-.052-.114-.052-.118-.051-.123-.051-.126-.051-.131-.05-.136-.05-.139-.049-.143-.048-.148-.048-.152-.048-.155-.046-.16-.046-.163-.045-.168-.043-.172-.043-.175-.042-.179-.041-.183-.04-.187-.038-.191-.038-.194-.036-.198-.034-.202-.033-.205-.032-.21-.031-.212-.028-.216-.027-.22-.026-.224-.023-.226-.022-.231-.021-.233-.018-.237-.016-.241-.014-.244-.012-.247-.011-.25-.008-.254-.005-.257-.004-.26-.001-.26.001z")},"insertDatabaseIcon"),Zke=o(function(t){t.append("defs").append("symbol").attr("id","computer").attr("width","24").attr("height","24").append("path").attr("transform","scale(.5)").attr("d","M2 2v13h20v-13h-20zm18 11h-16v-9h16v9zm-10.228 6l.466-1h3.524l.467 1h-4.457zm14.228 3h-24l2-6h2.104l-1.33 4h18.45l-1.297-4h2.073l2 6zm-5-10h-14v-7h14v7z")},"insertComputerIcon"),Jke=o(function(t){t.append("defs").append("symbol").attr("id","clock").attr("width","24").attr("height","24").append("path").attr("transform","scale(.5)").attr("d","M12 2c5.514 0 10 4.486 10 10s-4.486 10-10 10-10-4.486-10-10 4.486-10 10-10zm0-2c-6.627 0-12 5.373-12 12s5.373 12 12 12 12-5.373 12-12-5.373-12-12-12zm5.848 12.459c.202.038.202.333.001.372-1.907.361-6.045 1.111-6.547 1.111-.719 0-1.301-.582-1.301-1.301 0-.512.77-5.447 1.125-7.445.034-.192.312-.181.343.014l.985 6.238 5.394 1.011z")},"insertClockIcon"),eEe=o(function(t){t.append("defs").append("marker").attr("id","arrowhead").attr("refX",9).attr("refY",5).attr("markerUnits","userSpaceOnUse").attr("markerWidth",12).attr("markerHeight",12).attr("orient","auto").append("path").attr("d","M 0 0 L 10 5 L 0 10 z")},"insertArrowHead"),tEe=o(function(t){t.append("defs").append("marker").attr("id","arrowend").attr("refX",1).attr("refY",5).attr("markerUnits","userSpaceOnUse").attr("markerWidth",12).attr("markerHeight",12).attr("orient","auto").append("path").attr("d","M 10 0 L 0 5 L 10 10 z")},"insertArrowEnd"),rEe=o(function(t){t.append("defs").append("marker").attr("id","filled-head").attr("refX",18).attr("refY",7).attr("markerWidth",20).attr("markerHeight",28).attr("orient","auto").append("path").attr("d","M 18,7 L9,13 L14,7 L9,1 Z")},"insertArrowFilledHead"),nEe=o(function(t){t.append("defs").append("marker").attr("id","sequencenumber").attr("refX",15).attr("refY",15).attr("markerWidth",60).attr("markerHeight",40).attr("orient","auto").append("circle").attr("cx",15).attr("cy",15).attr("r",6)},"insertDynamicNumber"),iEe=o(function(t){let r=t.append("defs").append("marker").attr("id","crosshead").attr("markerWidth",15).attr("markerHeight",8).attr("orient","auto").attr("refX",16).attr("refY",4);r.append("path").attr("fill","black").attr("stroke","#000000").style("stroke-dasharray","0, 0").attr("stroke-width","1px").attr("d","M 9,2 V 6 L16,4 Z"),r.append("path").attr("fill","none").attr("stroke","#000000").style("stroke-dasharray","0, 0").attr("stroke-width","1px").attr("d","M 0,1 L 6,7 M 6,1 L 0,7")},"insertArrowCrossHead"),aEe=o((t,e)=>({fontFamily:t[e+"FontFamily"],fontSize:t[e+"FontSize"],fontWeight:t[e+"FontWeight"]}),"getC4ShapeFont"),Tu=function(){function t(i,a,s,l,u,h,f){let d=a.append("text").attr("x",s+u/2).attr("y",l+h/2+5).style("text-anchor","middle").text(i);n(d,f)}o(t,"byText");function e(i,a,s,l,u,h,f,d){let{fontSize:p,fontFamily:m,fontWeight:g}=d,y=i.split(Ze.lineBreakRegex);for(let v=0;v{"use strict";sEe=typeof global=="object"&&global&&global.Object===Object&&global,X5=sEe});var oEe,lEe,li,Lo=N(()=>{"use strict";w9();oEe=typeof self=="object"&&self&&self.Object===Object&&self,lEe=X5||oEe||Function("return this")(),li=lEe});var cEe,ea,Ed=N(()=>{"use strict";Lo();cEe=li.Symbol,ea=cEe});function fEe(t){var e=uEe.call(t,qv),r=t[qv];try{t[qv]=void 0;var n=!0}catch{}var i=hEe.call(t);return n&&(e?t[qv]=r:delete t[qv]),i}var Fq,uEe,hEe,qv,$q,zq=N(()=>{"use strict";Ed();Fq=Object.prototype,uEe=Fq.hasOwnProperty,hEe=Fq.toString,qv=ea?ea.toStringTag:void 0;o(fEe,"getRawTag");$q=fEe});function mEe(t){return pEe.call(t)}var dEe,pEe,Gq,Vq=N(()=>{"use strict";dEe=Object.prototype,pEe=dEe.toString;o(mEe,"objectToString");Gq=mEe});function vEe(t){return t==null?t===void 0?yEe:gEe:Uq&&Uq in Object(t)?$q(t):Gq(t)}var gEe,yEe,Uq,da,ku=N(()=>{"use strict";Ed();zq();Vq();gEe="[object Null]",yEe="[object Undefined]",Uq=ea?ea.toStringTag:void 0;o(vEe,"baseGetTag");da=vEe});function xEe(t){var e=typeof t;return t!=null&&(e=="object"||e=="function")}var bn,Js=N(()=>{"use strict";o(xEe,"isObject");bn=xEe});function EEe(t){if(!bn(t))return!1;var e=da(t);return e==wEe||e==TEe||e==bEe||e==kEe}var bEe,wEe,TEe,kEe,Si,Yv=N(()=>{"use strict";ku();Js();bEe="[object AsyncFunction]",wEe="[object Function]",TEe="[object GeneratorFunction]",kEe="[object Proxy]";o(EEe,"isFunction");Si=EEe});var SEe,j5,Hq=N(()=>{"use strict";Lo();SEe=li["__core-js_shared__"],j5=SEe});function CEe(t){return!!Wq&&Wq in t}var Wq,qq,Yq=N(()=>{"use strict";Hq();Wq=function(){var t=/[^.]+$/.exec(j5&&j5.keys&&j5.keys.IE_PROTO||"");return t?"Symbol(src)_1."+t:""}();o(CEe,"isMasked");qq=CEe});function DEe(t){if(t!=null){try{return _Ee.call(t)}catch{}try{return t+""}catch{}}return""}var AEe,_Ee,Eu,T9=N(()=>{"use strict";AEe=Function.prototype,_Ee=AEe.toString;o(DEe,"toSource");Eu=DEe});function BEe(t){if(!bn(t)||qq(t))return!1;var e=Si(t)?PEe:REe;return e.test(Eu(t))}var LEe,REe,NEe,MEe,IEe,OEe,PEe,Xq,jq=N(()=>{"use strict";Yv();Yq();Js();T9();LEe=/[\\^$.*+?()[\]{}|]/g,REe=/^\[object .+?Constructor\]$/,NEe=Function.prototype,MEe=Object.prototype,IEe=NEe.toString,OEe=MEe.hasOwnProperty,PEe=RegExp("^"+IEe.call(OEe).replace(LEe,"\\$&").replace(/hasOwnProperty|(function).*?(?=\\\()| for .+?(?=\\\])/g,"$1.*?")+"$");o(BEe,"baseIsNative");Xq=BEe});function FEe(t,e){return t?.[e]}var Kq,Qq=N(()=>{"use strict";o(FEe,"getValue");Kq=FEe});function $Ee(t,e){var r=Kq(t,e);return Xq(r)?r:void 0}var Ss,Lh=N(()=>{"use strict";jq();Qq();o($Ee,"getNative");Ss=$Ee});var zEe,Su,Xv=N(()=>{"use strict";Lh();zEe=Ss(Object,"create"),Su=zEe});function GEe(){this.__data__=Su?Su(null):{},this.size=0}var Zq,Jq=N(()=>{"use strict";Xv();o(GEe,"hashClear");Zq=GEe});function VEe(t){var e=this.has(t)&&delete this.__data__[t];return this.size-=e?1:0,e}var eY,tY=N(()=>{"use strict";o(VEe,"hashDelete");eY=VEe});function qEe(t){var e=this.__data__;if(Su){var r=e[t];return r===UEe?void 0:r}return WEe.call(e,t)?e[t]:void 0}var UEe,HEe,WEe,rY,nY=N(()=>{"use strict";Xv();UEe="__lodash_hash_undefined__",HEe=Object.prototype,WEe=HEe.hasOwnProperty;o(qEe,"hashGet");rY=qEe});function jEe(t){var e=this.__data__;return Su?e[t]!==void 0:XEe.call(e,t)}var YEe,XEe,iY,aY=N(()=>{"use strict";Xv();YEe=Object.prototype,XEe=YEe.hasOwnProperty;o(jEe,"hashHas");iY=jEe});function QEe(t,e){var r=this.__data__;return this.size+=this.has(t)?0:1,r[t]=Su&&e===void 0?KEe:e,this}var KEe,sY,oY=N(()=>{"use strict";Xv();KEe="__lodash_hash_undefined__";o(QEe,"hashSet");sY=QEe});function G0(t){var e=-1,r=t==null?0:t.length;for(this.clear();++e{"use strict";Jq();tY();nY();aY();oY();o(G0,"Hash");G0.prototype.clear=Zq;G0.prototype.delete=eY;G0.prototype.get=rY;G0.prototype.has=iY;G0.prototype.set=sY;k9=G0});function ZEe(){this.__data__=[],this.size=0}var cY,uY=N(()=>{"use strict";o(ZEe,"listCacheClear");cY=ZEe});function JEe(t,e){return t===e||t!==t&&e!==e}var Ro,Sd=N(()=>{"use strict";o(JEe,"eq");Ro=JEe});function e6e(t,e){for(var r=t.length;r--;)if(Ro(t[r][0],e))return r;return-1}var Rh,jv=N(()=>{"use strict";Sd();o(e6e,"assocIndexOf");Rh=e6e});function n6e(t){var e=this.__data__,r=Rh(e,t);if(r<0)return!1;var n=e.length-1;return r==n?e.pop():r6e.call(e,r,1),--this.size,!0}var t6e,r6e,hY,fY=N(()=>{"use strict";jv();t6e=Array.prototype,r6e=t6e.splice;o(n6e,"listCacheDelete");hY=n6e});function i6e(t){var e=this.__data__,r=Rh(e,t);return r<0?void 0:e[r][1]}var dY,pY=N(()=>{"use strict";jv();o(i6e,"listCacheGet");dY=i6e});function a6e(t){return Rh(this.__data__,t)>-1}var mY,gY=N(()=>{"use strict";jv();o(a6e,"listCacheHas");mY=a6e});function s6e(t,e){var r=this.__data__,n=Rh(r,t);return n<0?(++this.size,r.push([t,e])):r[n][1]=e,this}var yY,vY=N(()=>{"use strict";jv();o(s6e,"listCacheSet");yY=s6e});function V0(t){var e=-1,r=t==null?0:t.length;for(this.clear();++e{"use strict";uY();fY();pY();gY();vY();o(V0,"ListCache");V0.prototype.clear=cY;V0.prototype.delete=hY;V0.prototype.get=dY;V0.prototype.has=mY;V0.prototype.set=yY;Nh=V0});var o6e,Mh,K5=N(()=>{"use strict";Lh();Lo();o6e=Ss(li,"Map"),Mh=o6e});function l6e(){this.size=0,this.__data__={hash:new k9,map:new(Mh||Nh),string:new k9}}var xY,bY=N(()=>{"use strict";lY();Kv();K5();o(l6e,"mapCacheClear");xY=l6e});function c6e(t){var e=typeof t;return e=="string"||e=="number"||e=="symbol"||e=="boolean"?t!=="__proto__":t===null}var wY,TY=N(()=>{"use strict";o(c6e,"isKeyable");wY=c6e});function u6e(t,e){var r=t.__data__;return wY(e)?r[typeof e=="string"?"string":"hash"]:r.map}var Ih,Qv=N(()=>{"use strict";TY();o(u6e,"getMapData");Ih=u6e});function h6e(t){var e=Ih(this,t).delete(t);return this.size-=e?1:0,e}var kY,EY=N(()=>{"use strict";Qv();o(h6e,"mapCacheDelete");kY=h6e});function f6e(t){return Ih(this,t).get(t)}var SY,CY=N(()=>{"use strict";Qv();o(f6e,"mapCacheGet");SY=f6e});function d6e(t){return Ih(this,t).has(t)}var AY,_Y=N(()=>{"use strict";Qv();o(d6e,"mapCacheHas");AY=d6e});function p6e(t,e){var r=Ih(this,t),n=r.size;return r.set(t,e),this.size+=r.size==n?0:1,this}var DY,LY=N(()=>{"use strict";Qv();o(p6e,"mapCacheSet");DY=p6e});function U0(t){var e=-1,r=t==null?0:t.length;for(this.clear();++e{"use strict";bY();EY();CY();_Y();LY();o(U0,"MapCache");U0.prototype.clear=xY;U0.prototype.delete=kY;U0.prototype.get=SY;U0.prototype.has=AY;U0.prototype.set=DY;Cd=U0});function E9(t,e){if(typeof t!="function"||e!=null&&typeof e!="function")throw new TypeError(m6e);var r=o(function(){var n=arguments,i=e?e.apply(this,n):n[0],a=r.cache;if(a.has(i))return a.get(i);var s=t.apply(this,n);return r.cache=a.set(i,s)||a,s},"memoized");return r.cache=new(E9.Cache||Cd),r}var m6e,H0,S9=N(()=>{"use strict";Q5();m6e="Expected a function";o(E9,"memoize");E9.Cache=Cd;H0=E9});function g6e(){this.__data__=new Nh,this.size=0}var RY,NY=N(()=>{"use strict";Kv();o(g6e,"stackClear");RY=g6e});function y6e(t){var e=this.__data__,r=e.delete(t);return this.size=e.size,r}var MY,IY=N(()=>{"use strict";o(y6e,"stackDelete");MY=y6e});function v6e(t){return this.__data__.get(t)}var OY,PY=N(()=>{"use strict";o(v6e,"stackGet");OY=v6e});function x6e(t){return this.__data__.has(t)}var BY,FY=N(()=>{"use strict";o(x6e,"stackHas");BY=x6e});function w6e(t,e){var r=this.__data__;if(r instanceof Nh){var n=r.__data__;if(!Mh||n.length{"use strict";Kv();K5();Q5();b6e=200;o(w6e,"stackSet");$Y=w6e});function W0(t){var e=this.__data__=new Nh(t);this.size=e.size}var lc,Zv=N(()=>{"use strict";Kv();NY();IY();PY();FY();zY();o(W0,"Stack");W0.prototype.clear=RY;W0.prototype.delete=MY;W0.prototype.get=OY;W0.prototype.has=BY;W0.prototype.set=$Y;lc=W0});var T6e,q0,C9=N(()=>{"use strict";Lh();T6e=function(){try{var t=Ss(Object,"defineProperty");return t({},"",{}),t}catch{}}(),q0=T6e});function k6e(t,e,r){e=="__proto__"&&q0?q0(t,e,{configurable:!0,enumerable:!0,value:r,writable:!0}):t[e]=r}var cc,Y0=N(()=>{"use strict";C9();o(k6e,"baseAssignValue");cc=k6e});function E6e(t,e,r){(r!==void 0&&!Ro(t[e],r)||r===void 0&&!(e in t))&&cc(t,e,r)}var Jv,A9=N(()=>{"use strict";Y0();Sd();o(E6e,"assignMergeValue");Jv=E6e});function S6e(t){return function(e,r,n){for(var i=-1,a=Object(e),s=n(e),l=s.length;l--;){var u=s[t?l:++i];if(r(a[u],u,a)===!1)break}return e}}var GY,VY=N(()=>{"use strict";o(S6e,"createBaseFor");GY=S6e});var C6e,X0,Z5=N(()=>{"use strict";VY();C6e=GY(),X0=C6e});function _6e(t,e){if(e)return t.slice();var r=t.length,n=WY?WY(r):new t.constructor(r);return t.copy(n),n}var qY,UY,A6e,HY,WY,J5,_9=N(()=>{"use strict";Lo();qY=typeof exports=="object"&&exports&&!exports.nodeType&&exports,UY=qY&&typeof module=="object"&&module&&!module.nodeType&&module,A6e=UY&&UY.exports===qY,HY=A6e?li.Buffer:void 0,WY=HY?HY.allocUnsafe:void 0;o(_6e,"cloneBuffer");J5=_6e});var D6e,j0,D9=N(()=>{"use strict";Lo();D6e=li.Uint8Array,j0=D6e});function L6e(t){var e=new t.constructor(t.byteLength);return new j0(e).set(new j0(t)),e}var K0,ew=N(()=>{"use strict";D9();o(L6e,"cloneArrayBuffer");K0=L6e});function R6e(t,e){var r=e?K0(t.buffer):t.buffer;return new t.constructor(r,t.byteOffset,t.length)}var tw,L9=N(()=>{"use strict";ew();o(R6e,"cloneTypedArray");tw=R6e});function N6e(t,e){var r=-1,n=t.length;for(e||(e=Array(n));++r{"use strict";o(N6e,"copyArray");rw=N6e});var YY,M6e,XY,jY=N(()=>{"use strict";Js();YY=Object.create,M6e=function(){function t(){}return o(t,"object"),function(e){if(!bn(e))return{};if(YY)return YY(e);t.prototype=e;var r=new t;return t.prototype=void 0,r}}(),XY=M6e});function I6e(t,e){return function(r){return t(e(r))}}var nw,N9=N(()=>{"use strict";o(I6e,"overArg");nw=I6e});var O6e,Q0,iw=N(()=>{"use strict";N9();O6e=nw(Object.getPrototypeOf,Object),Q0=O6e});function B6e(t){var e=t&&t.constructor,r=typeof e=="function"&&e.prototype||P6e;return t===r}var P6e,uc,Z0=N(()=>{"use strict";P6e=Object.prototype;o(B6e,"isPrototype");uc=B6e});function F6e(t){return typeof t.constructor=="function"&&!uc(t)?XY(Q0(t)):{}}var aw,M9=N(()=>{"use strict";jY();iw();Z0();o(F6e,"initCloneObject");aw=F6e});function $6e(t){return t!=null&&typeof t=="object"}var ri,No=N(()=>{"use strict";o($6e,"isObjectLike");ri=$6e});function G6e(t){return ri(t)&&da(t)==z6e}var z6e,I9,KY=N(()=>{"use strict";ku();No();z6e="[object Arguments]";o(G6e,"baseIsArguments");I9=G6e});var QY,V6e,U6e,H6e,El,J0=N(()=>{"use strict";KY();No();QY=Object.prototype,V6e=QY.hasOwnProperty,U6e=QY.propertyIsEnumerable,H6e=I9(function(){return arguments}())?I9:function(t){return ri(t)&&V6e.call(t,"callee")&&!U6e.call(t,"callee")},El=H6e});var W6e,Pt,Un=N(()=>{"use strict";W6e=Array.isArray,Pt=W6e});function Y6e(t){return typeof t=="number"&&t>-1&&t%1==0&&t<=q6e}var q6e,em,sw=N(()=>{"use strict";q6e=9007199254740991;o(Y6e,"isLength");em=Y6e});function X6e(t){return t!=null&&em(t.length)&&!Si(t)}var ci,Mo=N(()=>{"use strict";Yv();sw();o(X6e,"isArrayLike");ci=X6e});function j6e(t){return ri(t)&&ci(t)}var Ad,ow=N(()=>{"use strict";Mo();No();o(j6e,"isArrayLikeObject");Ad=j6e});function K6e(){return!1}var ZY,JY=N(()=>{"use strict";o(K6e,"stubFalse");ZY=K6e});var rX,eX,Q6e,tX,Z6e,J6e,Sl,tm=N(()=>{"use strict";Lo();JY();rX=typeof exports=="object"&&exports&&!exports.nodeType&&exports,eX=rX&&typeof module=="object"&&module&&!module.nodeType&&module,Q6e=eX&&eX.exports===rX,tX=Q6e?li.Buffer:void 0,Z6e=tX?tX.isBuffer:void 0,J6e=Z6e||ZY,Sl=J6e});function aSe(t){if(!ri(t)||da(t)!=eSe)return!1;var e=Q0(t);if(e===null)return!0;var r=nSe.call(e,"constructor")&&e.constructor;return typeof r=="function"&&r instanceof r&&nX.call(r)==iSe}var eSe,tSe,rSe,nX,nSe,iSe,iX,aX=N(()=>{"use strict";ku();iw();No();eSe="[object Object]",tSe=Function.prototype,rSe=Object.prototype,nX=tSe.toString,nSe=rSe.hasOwnProperty,iSe=nX.call(Object);o(aSe,"isPlainObject");iX=aSe});function LSe(t){return ri(t)&&em(t.length)&&!!Fn[da(t)]}var sSe,oSe,lSe,cSe,uSe,hSe,fSe,dSe,pSe,mSe,gSe,ySe,vSe,xSe,bSe,wSe,TSe,kSe,ESe,SSe,CSe,ASe,_Se,DSe,Fn,sX,oX=N(()=>{"use strict";ku();sw();No();sSe="[object Arguments]",oSe="[object Array]",lSe="[object Boolean]",cSe="[object Date]",uSe="[object Error]",hSe="[object Function]",fSe="[object Map]",dSe="[object Number]",pSe="[object Object]",mSe="[object RegExp]",gSe="[object Set]",ySe="[object String]",vSe="[object WeakMap]",xSe="[object ArrayBuffer]",bSe="[object DataView]",wSe="[object Float32Array]",TSe="[object Float64Array]",kSe="[object Int8Array]",ESe="[object Int16Array]",SSe="[object Int32Array]",CSe="[object Uint8Array]",ASe="[object Uint8ClampedArray]",_Se="[object Uint16Array]",DSe="[object Uint32Array]",Fn={};Fn[wSe]=Fn[TSe]=Fn[kSe]=Fn[ESe]=Fn[SSe]=Fn[CSe]=Fn[ASe]=Fn[_Se]=Fn[DSe]=!0;Fn[sSe]=Fn[oSe]=Fn[xSe]=Fn[lSe]=Fn[bSe]=Fn[cSe]=Fn[uSe]=Fn[hSe]=Fn[fSe]=Fn[dSe]=Fn[pSe]=Fn[mSe]=Fn[gSe]=Fn[ySe]=Fn[vSe]=!1;o(LSe,"baseIsTypedArray");sX=LSe});function RSe(t){return function(e){return t(e)}}var Io,_d=N(()=>{"use strict";o(RSe,"baseUnary");Io=RSe});var lX,e2,NSe,O9,MSe,Oo,t2=N(()=>{"use strict";w9();lX=typeof exports=="object"&&exports&&!exports.nodeType&&exports,e2=lX&&typeof module=="object"&&module&&!module.nodeType&&module,NSe=e2&&e2.exports===lX,O9=NSe&&X5.process,MSe=function(){try{var t=e2&&e2.require&&e2.require("util").types;return t||O9&&O9.binding&&O9.binding("util")}catch{}}(),Oo=MSe});var cX,ISe,Oh,r2=N(()=>{"use strict";oX();_d();t2();cX=Oo&&Oo.isTypedArray,ISe=cX?Io(cX):sX,Oh=ISe});function OSe(t,e){if(!(e==="constructor"&&typeof t[e]=="function")&&e!="__proto__")return t[e]}var n2,P9=N(()=>{"use strict";o(OSe,"safeGet");n2=OSe});function FSe(t,e,r){var n=t[e];(!(BSe.call(t,e)&&Ro(n,r))||r===void 0&&!(e in t))&&cc(t,e,r)}var PSe,BSe,hc,rm=N(()=>{"use strict";Y0();Sd();PSe=Object.prototype,BSe=PSe.hasOwnProperty;o(FSe,"assignValue");hc=FSe});function $Se(t,e,r,n){var i=!r;r||(r={});for(var a=-1,s=e.length;++a{"use strict";rm();Y0();o($Se,"copyObject");Po=$Se});function zSe(t,e){for(var r=-1,n=Array(t);++r{"use strict";o(zSe,"baseTimes");uX=zSe});function USe(t,e){var r=typeof t;return e=e??GSe,!!e&&(r=="number"||r!="symbol"&&VSe.test(t))&&t>-1&&t%1==0&&t{"use strict";GSe=9007199254740991,VSe=/^(?:0|[1-9]\d*)$/;o(USe,"isIndex");Ph=USe});function qSe(t,e){var r=Pt(t),n=!r&&El(t),i=!r&&!n&&Sl(t),a=!r&&!n&&!i&&Oh(t),s=r||n||i||a,l=s?uX(t.length,String):[],u=l.length;for(var h in t)(e||WSe.call(t,h))&&!(s&&(h=="length"||i&&(h=="offset"||h=="parent")||a&&(h=="buffer"||h=="byteLength"||h=="byteOffset")||Ph(h,u)))&&l.push(h);return l}var HSe,WSe,lw,B9=N(()=>{"use strict";hX();J0();Un();tm();i2();r2();HSe=Object.prototype,WSe=HSe.hasOwnProperty;o(qSe,"arrayLikeKeys");lw=qSe});function YSe(t){var e=[];if(t!=null)for(var r in Object(t))e.push(r);return e}var fX,dX=N(()=>{"use strict";o(YSe,"nativeKeysIn");fX=YSe});function KSe(t){if(!bn(t))return fX(t);var e=uc(t),r=[];for(var n in t)n=="constructor"&&(e||!jSe.call(t,n))||r.push(n);return r}var XSe,jSe,pX,mX=N(()=>{"use strict";Js();Z0();dX();XSe=Object.prototype,jSe=XSe.hasOwnProperty;o(KSe,"baseKeysIn");pX=KSe});function QSe(t){return ci(t)?lw(t,!0):pX(t)}var Cs,Bh=N(()=>{"use strict";B9();mX();Mo();o(QSe,"keysIn");Cs=QSe});function ZSe(t){return Po(t,Cs(t))}var gX,yX=N(()=>{"use strict";Dd();Bh();o(ZSe,"toPlainObject");gX=ZSe});function JSe(t,e,r,n,i,a,s){var l=n2(t,r),u=n2(e,r),h=s.get(u);if(h){Jv(t,r,h);return}var f=a?a(l,u,r+"",t,e,s):void 0,d=f===void 0;if(d){var p=Pt(u),m=!p&&Sl(u),g=!p&&!m&&Oh(u);f=u,p||m||g?Pt(l)?f=l:Ad(l)?f=rw(l):m?(d=!1,f=J5(u,!0)):g?(d=!1,f=tw(u,!0)):f=[]:iX(u)||El(u)?(f=l,El(l)?f=gX(l):(!bn(l)||Si(l))&&(f=aw(u))):d=!1}d&&(s.set(u,f),i(f,u,n,a,s),s.delete(u)),Jv(t,r,f)}var vX,xX=N(()=>{"use strict";A9();_9();L9();R9();M9();J0();Un();ow();tm();Yv();Js();aX();r2();P9();yX();o(JSe,"baseMergeDeep");vX=JSe});function bX(t,e,r,n,i){t!==e&&X0(e,function(a,s){if(i||(i=new lc),bn(a))vX(t,e,s,r,bX,n,i);else{var l=n?n(n2(t,s),a,s+"",t,e,i):void 0;l===void 0&&(l=a),Jv(t,s,l)}},Cs)}var wX,TX=N(()=>{"use strict";Zv();A9();Z5();xX();Js();Bh();P9();o(bX,"baseMerge");wX=bX});function eCe(t){return t}var ta,Cu=N(()=>{"use strict";o(eCe,"identity");ta=eCe});function tCe(t,e,r){switch(r.length){case 0:return t.call(e);case 1:return t.call(e,r[0]);case 2:return t.call(e,r[0],r[1]);case 3:return t.call(e,r[0],r[1],r[2])}return t.apply(e,r)}var kX,EX=N(()=>{"use strict";o(tCe,"apply");kX=tCe});function rCe(t,e,r){return e=SX(e===void 0?t.length-1:e,0),function(){for(var n=arguments,i=-1,a=SX(n.length-e,0),s=Array(a);++i{"use strict";EX();SX=Math.max;o(rCe,"overRest");cw=rCe});function nCe(t){return function(){return t}}var As,$9=N(()=>{"use strict";o(nCe,"constant");As=nCe});var iCe,CX,AX=N(()=>{"use strict";$9();C9();Cu();iCe=q0?function(t,e){return q0(t,"toString",{configurable:!0,enumerable:!1,value:As(e),writable:!0})}:ta,CX=iCe});function lCe(t){var e=0,r=0;return function(){var n=oCe(),i=sCe-(n-r);if(r=n,i>0){if(++e>=aCe)return arguments[0]}else e=0;return t.apply(void 0,arguments)}}var aCe,sCe,oCe,_X,DX=N(()=>{"use strict";aCe=800,sCe=16,oCe=Date.now;o(lCe,"shortOut");_X=lCe});var cCe,uw,z9=N(()=>{"use strict";AX();DX();cCe=_X(CX),uw=cCe});function uCe(t,e){return uw(cw(t,e,ta),t+"")}var fc,nm=N(()=>{"use strict";Cu();F9();z9();o(uCe,"baseRest");fc=uCe});function hCe(t,e,r){if(!bn(r))return!1;var n=typeof e;return(n=="number"?ci(r)&&Ph(e,r.length):n=="string"&&e in r)?Ro(r[e],t):!1}var eo,Ld=N(()=>{"use strict";Sd();Mo();i2();Js();o(hCe,"isIterateeCall");eo=hCe});function fCe(t){return fc(function(e,r){var n=-1,i=r.length,a=i>1?r[i-1]:void 0,s=i>2?r[2]:void 0;for(a=t.length>3&&typeof a=="function"?(i--,a):void 0,s&&eo(r[0],r[1],s)&&(a=i<3?void 0:a,i=1),e=Object(e);++n{"use strict";nm();Ld();o(fCe,"createAssigner");hw=fCe});var dCe,Fh,V9=N(()=>{"use strict";TX();G9();dCe=hw(function(t,e,r){wX(t,e,r)}),Fh=dCe});function W9(t,e){if(!t)return e;let r=`curve${t.charAt(0).toUpperCase()+t.slice(1)}`;return pCe[r]??e}function vCe(t,e){let r=t.trim();if(r)return e.securityLevel!=="loose"?(0,NX.sanitizeUrl)(r):r}function OX(t,e){return!t||!e?0:Math.sqrt(Math.pow(e.x-t.x,2)+Math.pow(e.y-t.y,2))}function bCe(t){let e,r=0;t.forEach(i=>{r+=OX(i,e),e=i});let n=r/2;return q9(t,n)}function wCe(t){return t.length===1?t[0]:bCe(t)}function kCe(t,e,r){let n=structuredClone(r);Y.info("our points",n),e!=="start_left"&&e!=="start_right"&&n.reverse();let i=25+t,a=q9(n,i),s=10+t*.5,l=Math.atan2(n[0].y-a.y,n[0].x-a.x),u={x:0,y:0};return e==="start_left"?(u.x=Math.sin(l+Math.PI)*s+(n[0].x+a.x)/2,u.y=-Math.cos(l+Math.PI)*s+(n[0].y+a.y)/2):e==="end_right"?(u.x=Math.sin(l-Math.PI)*s+(n[0].x+a.x)/2-5,u.y=-Math.cos(l-Math.PI)*s+(n[0].y+a.y)/2-5):e==="end_left"?(u.x=Math.sin(l)*s+(n[0].x+a.x)/2-5,u.y=-Math.cos(l)*s+(n[0].y+a.y)/2-5):(u.x=Math.sin(l)*s+(n[0].x+a.x)/2,u.y=-Math.cos(l)*s+(n[0].y+a.y)/2),u}function Y9(t){let e="",r="";for(let n of t)n!==void 0&&(n.startsWith("color:")||n.startsWith("text-align:")?r=r+n+";":e=e+n+";");return{style:e,labelStyle:r}}function ECe(t){let e="",r="0123456789abcdef",n=r.length;for(let i=0;i{"use strict";NX=Sa(z0(),1);dr();gr();e7();vt();Xf();s0();S9();V9();$4();H9="\u200B",pCe={curveBasis:Do,curveBasisClosed:P5,curveBasisOpen:B5,curveBumpX:Rv,curveBumpY:Nv,curveBundle:l9,curveCardinalClosed:c9,curveCardinalOpen:h9,curveCardinal:Pv,curveCatmullRomClosed:d9,curveCatmullRomOpen:p9,curveCatmullRom:$v,curveLinear:wu,curveLinearClosed:V5,curveMonotoneX:zv,curveMonotoneY:Gv,curveNatural:F0,curveStep:$0,curveStepAfter:Uv,curveStepBefore:Vv},mCe=/\s*(?:(\w+)(?=:):|(\w+))\s*(?:(\w+)|((?:(?!}%{2}).|\r?\n)*))?\s*(?:}%{2})?/gi,gCe=o(function(t,e){let r=MX(t,/(?:init\b)|(?:initialize\b)/),n={};if(Array.isArray(r)){let s=r.map(l=>l.args);l0(s),n=Gn(n,[...s])}else n=r.args;if(!n)return;let i=a0(t,e),a="config";return n[a]!==void 0&&(i==="flowchart-v2"&&(i="flowchart"),n[i]=n[a],delete n[a]),n},"detectInit"),MX=o(function(t,e=null){try{let r=new RegExp(`[%]{2}(?![{]${mCe.source})(?=[}][%]{2}).* +`,"ig");t=t.trim().replace(r,"").replace(/'/gm,'"'),Y.debug(`Detecting diagram directive${e!==null?" type:"+e:""} based on the text:${t}`);let n,i=[];for(;(n=qf.exec(t))!==null;)if(n.index===qf.lastIndex&&qf.lastIndex++,n&&!e||e&&n[1]?.match(e)||e&&n[2]?.match(e)){let a=n[1]?n[1]:n[2],s=n[3]?n[3].trim():n[4]?JSON.parse(n[4].trim()):null;i.push({type:a,args:s})}return i.length===0?{type:t,args:null}:i.length===1?i[0]:i}catch(r){return Y.error(`ERROR: ${r.message} - Unable to parse directive type: '${e}' based on the text: '${t}'`),{type:void 0,args:null}}},"detectDirective"),IX=o(function(t){return t.replace(qf,"")},"removeDirectives"),yCe=o(function(t,e){for(let[r,n]of e.entries())if(n.match(t))return r;return-1},"isSubstringInArray");o(W9,"interpolateToCurve");o(vCe,"formatUrl");xCe=o((t,...e)=>{let r=t.split("."),n=r.length-1,i=r[n],a=window;for(let s=0;s{let r=Math.pow(10,e);return Math.round(t*r)/r},"roundNumber"),q9=o((t,e)=>{let r,n=e;for(let i of t){if(r){let a=OX(i,r);if(a===0)return r;if(a=1)return{x:i.x,y:i.y};if(s>0&&s<1)return{x:LX((1-s)*r.x+s*i.x,5),y:LX((1-s)*r.y+s*i.y,5)}}}r=i}throw new Error("Could not find a suitable point for the given distance")},"calculatePoint"),TCe=o((t,e,r)=>{Y.info(`our points ${JSON.stringify(e)}`),e[0]!==r&&(e=e.reverse());let i=q9(e,25),a=t?10:5,s=Math.atan2(e[0].y-i.y,e[0].x-i.x),l={x:0,y:0};return l.x=Math.sin(s)*a+(e[0].x+i.x)/2,l.y=-Math.cos(s)*a+(e[0].y+i.y)/2,l},"calcCardinalityPosition");o(kCe,"calcTerminalLabelPosition");o(Y9,"getStylesFromArray");RX=0,X9=o(()=>(RX++,"id-"+Math.random().toString(36).substr(2,12)+"-"+RX),"generateId");o(ECe,"makeRandomHex");j9=o(t=>ECe(t.length),"random"),SCe=o(function(){return{x:0,y:0,fill:void 0,anchor:"start",style:"#666",width:100,height:100,textMargin:0,rx:0,ry:0,valign:void 0,text:""}},"getTextObj"),CCe=o(function(t,e){let r=e.text.replace(Ze.lineBreakRegex," "),[,n]=Bo(e.fontSize),i=t.append("text");i.attr("x",e.x),i.attr("y",e.y),i.style("text-anchor",e.anchor),i.style("font-family",e.fontFamily),i.style("font-size",n),i.style("font-weight",e.fontWeight),i.attr("fill",e.fill),e.class!==void 0&&i.attr("class",e.class);let a=i.append("tspan");return a.attr("x",e.x+e.textMargin*2),a.attr("fill",e.fill),a.text(r),i},"drawSimpleText"),K9=H0((t,e,r)=>{if(!t||(r=Object.assign({fontSize:12,fontWeight:400,fontFamily:"Arial",joinWith:"
"},r),Ze.lineBreakRegex.test(t)))return t;let n=t.split(" ").filter(Boolean),i=[],a="";return n.forEach((s,l)=>{let u=ra(`${s} `,r),h=ra(a,r);if(u>e){let{hyphenatedStrings:p,remainingWord:m}=ACe(s,e,"-",r);i.push(a,...p),a=m}else h+u>=e?(i.push(a),a=s):a=[a,s].filter(Boolean).join(" ");l+1===n.length&&i.push(a)}),i.filter(s=>s!=="").join(r.joinWith)},(t,e,r)=>`${t}${e}${r.fontSize}${r.fontWeight}${r.fontFamily}${r.joinWith}`),ACe=H0((t,e,r="-",n)=>{n=Object.assign({fontSize:12,fontWeight:400,fontFamily:"Arial",margin:0},n);let i=[...t],a=[],s="";return i.forEach((l,u)=>{let h=`${s}${l}`;if(ra(h,n)>=e){let d=u+1,p=i.length===d,m=`${h}${r}`;a.push(p?h:m),s=""}else s=h}),{hyphenatedStrings:a,remainingWord:s}},(t,e,r="-",n)=>`${t}${e}${r}${n.fontSize}${n.fontWeight}${n.fontFamily}`);o(dw,"calculateTextHeight");o(ra,"calculateTextWidth");Q9=H0((t,e)=>{let{fontSize:r=12,fontFamily:n="Arial",fontWeight:i=400}=e;if(!t)return{width:0,height:0};let[,a]=Bo(r),s=["sans-serif",n],l=t.split(Ze.lineBreakRegex),u=[],h=Ge("body");if(!h.remove)return{width:0,height:0,lineHeight:0};let f=h.append("svg");for(let p of s){let m=0,g={width:0,height:0,lineHeight:0};for(let y of l){let v=SCe();v.text=y||H9;let x=CCe(f,v).style("font-size",a).style("font-weight",i).style("font-family",p),b=(x._groups||x)[0][0].getBBox();if(b.width===0&&b.height===0)throw new Error("svg element not in render tree");g.width=Math.round(Math.max(g.width,b.width)),m=Math.round(b.height),g.height+=m,g.lineHeight=Math.round(Math.max(g.lineHeight,m))}u.push(g)}f.remove();let d=isNaN(u[1].height)||isNaN(u[1].width)||isNaN(u[1].lineHeight)||u[0].height>u[1].height&&u[0].width>u[1].width&&u[0].lineHeight>u[1].lineHeight?0:1;return u[d]},(t,e)=>`${t}${e.fontSize}${e.fontWeight}${e.fontFamily}`),U9=class{constructor(e=!1,r){this.count=0;this.count=r?r.length:0,this.next=e?()=>this.count++:()=>Date.now()}static{o(this,"InitIDGenerator")}},_Ce=o(function(t){return fw=fw||document.createElement("div"),t=escape(t).replace(/%26/g,"&").replace(/%23/g,"#").replace(/%3B/g,";"),fw.innerHTML=t,unescape(fw.textContent)},"entityDecode");o(Z9,"isDetailedError");DCe=o((t,e,r,n)=>{if(!n)return;let i=t.node()?.getBBox();i&&t.append("text").text(n).attr("text-anchor","middle").attr("x",i.x+i.width/2).attr("y",-r).attr("class",e)},"insertTitle"),Bo=o(t=>{if(typeof t=="number")return[t,t+"px"];let e=parseInt(t??"",10);return Number.isNaN(e)?[void 0,void 0]:t===String(e)?[e,t+"px"]:[e,t]},"parseFontSize");o(Fi,"cleanAndMerge");Gt={assignWithDepth:Gn,wrapLabel:K9,calculateTextHeight:dw,calculateTextWidth:ra,calculateTextDimensions:Q9,cleanAndMerge:Fi,detectInit:gCe,detectDirective:MX,isSubstringInArray:yCe,interpolateToCurve:W9,calcLabelPosition:wCe,calcCardinalityPosition:TCe,calcTerminalLabelPosition:kCe,formatUrl:vCe,getStylesFromArray:Y9,generateId:X9,random:j9,runFunc:xCe,entityDecode:_Ce,insertTitle:DCe,parseFontSize:Bo,InitIDGenerator:U9},PX=o(function(t){let e=t;return e=e.replace(/style.*:\S*#.*;/g,function(r){return r.substring(0,r.length-1)}),e=e.replace(/classDef.*:\S*#.*;/g,function(r){return r.substring(0,r.length-1)}),e=e.replace(/#\w+;/g,function(r){let n=r.substring(1,r.length-1);return/^\+?\d+$/.test(n)?"\uFB02\xB0\xB0"+n+"\xB6\xDF":"\uFB02\xB0"+n+"\xB6\xDF"}),e},"encodeEntities"),na=o(function(t){return t.replace(/fl°°/g,"&#").replace(/fl°/g,"&").replace(/¶ß/g,";")},"decodeEntities"),$h=o((t,e,{counter:r=0,prefix:n,suffix:i},a)=>a||`${n?`${n}_`:""}${t}_${e}_${r}${i?`_${i}`:""}`,"getEdgeId");o($n,"handleUndefinedAttr")});function Cl(t,e,r,n,i){if(!e[t].width)if(r)e[t].text=K9(e[t].text,i,n),e[t].textLines=e[t].text.split(Ze.lineBreakRegex).length,e[t].width=i,e[t].height=dw(e[t].text,n);else{let a=e[t].text.split(Ze.lineBreakRegex);e[t].textLines=a.length;let s=0;e[t].height=0,e[t].width=0;for(let l of a)e[t].width=Math.max(ra(l,n),e[t].width),s=dw(l,n),e[t].height=e[t].height+s}}function GX(t,e,r,n,i){let a=new yw(i);a.data.widthLimit=r.data.widthLimit/Math.min(J9,n.length);for(let[s,l]of n.entries()){let u=0;l.image={width:0,height:0,Y:0},l.sprite&&(l.image.width=48,l.image.height=48,l.image.Y=u,u=l.image.Y+l.image.height);let h=l.wrap&&Vt.wrap,f=pw(Vt);if(f.fontSize=f.fontSize+2,f.fontWeight="bold",Cl("label",l,h,f,a.data.widthLimit),l.label.Y=u+8,u=l.label.Y+l.label.height,l.type&&l.type.text!==""){l.type.text="["+l.type.text+"]";let g=pw(Vt);Cl("type",l,h,g,a.data.widthLimit),l.type.Y=u+5,u=l.type.Y+l.type.height}if(l.descr&&l.descr.text!==""){let g=pw(Vt);g.fontSize=g.fontSize-2,Cl("descr",l,h,g,a.data.widthLimit),l.descr.Y=u+20,u=l.descr.Y+l.descr.height}if(s==0||s%J9===0){let g=r.data.startx+Vt.diagramMarginX,y=r.data.stopy+Vt.diagramMarginY+u;a.setData(g,g,y,y)}else{let g=a.data.stopx!==a.data.startx?a.data.stopx+Vt.diagramMarginX:a.data.startx,y=a.data.starty;a.setData(g,g,y,y)}a.name=l.alias;let d=i.db.getC4ShapeArray(l.alias),p=i.db.getC4ShapeKeys(l.alias);p.length>0&&zX(a,t,d,p),e=l.alias;let m=i.db.getBoundarys(e);m.length>0&&GX(t,e,a,m,i),l.alias!=="global"&&$X(t,l,a),r.data.stopy=Math.max(a.data.stopy+Vt.c4ShapeMargin,r.data.stopy),r.data.stopx=Math.max(a.data.stopx+Vt.c4ShapeMargin,r.data.stopx),mw=Math.max(mw,r.data.stopx),gw=Math.max(gw,r.data.stopy)}}var mw,gw,FX,J9,Vt,yw,eD,a2,pw,LCe,$X,zX,_s,BX,RCe,NCe,MCe,tD,VX=N(()=>{"use strict";dr();Bq();vt();$C();gr();uA();zt();s0();ir();Ei();mw=0,gw=0,FX=4,J9=2;Ty.yy=Qy;Vt={},yw=class{static{o(this,"Bounds")}constructor(e){this.name="",this.data={},this.data.startx=void 0,this.data.stopx=void 0,this.data.starty=void 0,this.data.stopy=void 0,this.data.widthLimit=void 0,this.nextData={},this.nextData.startx=void 0,this.nextData.stopx=void 0,this.nextData.starty=void 0,this.nextData.stopy=void 0,this.nextData.cnt=0,eD(e.db.getConfig())}setData(e,r,n,i){this.nextData.startx=this.data.startx=e,this.nextData.stopx=this.data.stopx=r,this.nextData.starty=this.data.starty=n,this.nextData.stopy=this.data.stopy=i}updateVal(e,r,n,i){e[r]===void 0?e[r]=n:e[r]=i(n,e[r])}insert(e){this.nextData.cnt=this.nextData.cnt+1;let r=this.nextData.startx===this.nextData.stopx?this.nextData.stopx+e.margin:this.nextData.stopx+e.margin*2,n=r+e.width,i=this.nextData.starty+e.margin*2,a=i+e.height;(r>=this.data.widthLimit||n>=this.data.widthLimit||this.nextData.cnt>FX)&&(r=this.nextData.startx+e.margin+Vt.nextLinePaddingX,i=this.nextData.stopy+e.margin*2,this.nextData.stopx=n=r+e.width,this.nextData.starty=this.nextData.stopy,this.nextData.stopy=a=i+e.height,this.nextData.cnt=1),e.x=r,e.y=i,this.updateVal(this.data,"startx",r,Math.min),this.updateVal(this.data,"starty",i,Math.min),this.updateVal(this.data,"stopx",n,Math.max),this.updateVal(this.data,"stopy",a,Math.max),this.updateVal(this.nextData,"startx",r,Math.min),this.updateVal(this.nextData,"starty",i,Math.min),this.updateVal(this.nextData,"stopx",n,Math.max),this.updateVal(this.nextData,"stopy",a,Math.max)}init(e){this.name="",this.data={startx:void 0,stopx:void 0,starty:void 0,stopy:void 0,widthLimit:void 0},this.nextData={startx:void 0,stopx:void 0,starty:void 0,stopy:void 0,cnt:0},eD(e.db.getConfig())}bumpLastMargin(e){this.data.stopx+=e,this.data.stopy+=e}},eD=o(function(t){Gn(Vt,t),t.fontFamily&&(Vt.personFontFamily=Vt.systemFontFamily=Vt.messageFontFamily=t.fontFamily),t.fontSize&&(Vt.personFontSize=Vt.systemFontSize=Vt.messageFontSize=t.fontSize),t.fontWeight&&(Vt.personFontWeight=Vt.systemFontWeight=Vt.messageFontWeight=t.fontWeight)},"setConf"),a2=o((t,e)=>({fontFamily:t[e+"FontFamily"],fontSize:t[e+"FontSize"],fontWeight:t[e+"FontWeight"]}),"c4ShapeFont"),pw=o(t=>({fontFamily:t.boundaryFontFamily,fontSize:t.boundaryFontSize,fontWeight:t.boundaryFontWeight}),"boundaryFont"),LCe=o(t=>({fontFamily:t.messageFontFamily,fontSize:t.messageFontSize,fontWeight:t.messageFontWeight}),"messageFont");o(Cl,"calcC4ShapeTextWH");$X=o(function(t,e,r){e.x=r.data.startx,e.y=r.data.starty,e.width=r.data.stopx-r.data.startx,e.height=r.data.stopy-r.data.starty,e.label.y=Vt.c4ShapeMargin-35;let n=e.wrap&&Vt.wrap,i=pw(Vt);i.fontSize=i.fontSize+2,i.fontWeight="bold";let a=ra(e.label.text,i);Cl("label",e,n,i,a),kl.drawBoundary(t,e,Vt)},"drawBoundary"),zX=o(function(t,e,r,n){let i=0;for(let a of n){i=0;let s=r[a],l=a2(Vt,s.typeC4Shape.text);switch(l.fontSize=l.fontSize-2,s.typeC4Shape.width=ra("\xAB"+s.typeC4Shape.text+"\xBB",l),s.typeC4Shape.height=l.fontSize+2,s.typeC4Shape.Y=Vt.c4ShapePadding,i=s.typeC4Shape.Y+s.typeC4Shape.height-4,s.image={width:0,height:0,Y:0},s.typeC4Shape.text){case"person":case"external_person":s.image.width=48,s.image.height=48,s.image.Y=i,i=s.image.Y+s.image.height;break}s.sprite&&(s.image.width=48,s.image.height=48,s.image.Y=i,i=s.image.Y+s.image.height);let u=s.wrap&&Vt.wrap,h=Vt.width-Vt.c4ShapePadding*2,f=a2(Vt,s.typeC4Shape.text);if(f.fontSize=f.fontSize+2,f.fontWeight="bold",Cl("label",s,u,f,h),s.label.Y=i+8,i=s.label.Y+s.label.height,s.type&&s.type.text!==""){s.type.text="["+s.type.text+"]";let m=a2(Vt,s.typeC4Shape.text);Cl("type",s,u,m,h),s.type.Y=i+5,i=s.type.Y+s.type.height}else if(s.techn&&s.techn.text!==""){s.techn.text="["+s.techn.text+"]";let m=a2(Vt,s.techn.text);Cl("techn",s,u,m,h),s.techn.Y=i+5,i=s.techn.Y+s.techn.height}let d=i,p=s.label.width;if(s.descr&&s.descr.text!==""){let m=a2(Vt,s.typeC4Shape.text);Cl("descr",s,u,m,h),s.descr.Y=i+20,i=s.descr.Y+s.descr.height,p=Math.max(s.label.width,s.descr.width),d=i-s.descr.textLines*5}p=p+Vt.c4ShapePadding,s.width=Math.max(s.width||Vt.width,p,Vt.width),s.height=Math.max(s.height||Vt.height,d,Vt.height),s.margin=s.margin||Vt.c4ShapeMargin,t.insert(s),kl.drawC4Shape(e,s,Vt)}t.bumpLastMargin(Vt.c4ShapeMargin)},"drawC4ShapeArray"),_s=class{static{o(this,"Point")}constructor(e,r){this.x=e,this.y=r}},BX=o(function(t,e){let r=t.x,n=t.y,i=e.x,a=e.y,s=r+t.width/2,l=n+t.height/2,u=Math.abs(r-i),h=Math.abs(n-a),f=h/u,d=t.height/t.width,p=null;return n==a&&ri?p=new _s(r,l):r==i&&na&&(p=new _s(s,n)),r>i&&n=f?p=new _s(r,l+f*t.width/2):p=new _s(s-u/h*t.height/2,n+t.height):r=f?p=new _s(r+t.width,l+f*t.width/2):p=new _s(s+u/h*t.height/2,n+t.height):ra?d>=f?p=new _s(r+t.width,l-f*t.width/2):p=new _s(s+t.height/2*u/h,n):r>i&&n>a&&(d>=f?p=new _s(r,l-t.width/2*f):p=new _s(s-t.height/2*u/h,n)),p},"getIntersectPoint"),RCe=o(function(t,e){let r={x:0,y:0};r.x=e.x+e.width/2,r.y=e.y+e.height/2;let n=BX(t,r);r.x=t.x+t.width/2,r.y=t.y+t.height/2;let i=BX(e,r);return{startPoint:n,endPoint:i}},"getIntersectPoints"),NCe=o(function(t,e,r,n){let i=0;for(let a of e){i=i+1;let s=a.wrap&&Vt.wrap,l=LCe(Vt);n.db.getC4Type()==="C4Dynamic"&&(a.label.text=i+": "+a.label.text);let h=ra(a.label.text,l);Cl("label",a,s,l,h),a.techn&&a.techn.text!==""&&(h=ra(a.techn.text,l),Cl("techn",a,s,l,h)),a.descr&&a.descr.text!==""&&(h=ra(a.descr.text,l),Cl("descr",a,s,l,h));let f=r(a.from),d=r(a.to),p=RCe(f,d);a.startPoint=p.startPoint,a.endPoint=p.endPoint}kl.drawRels(t,e,Vt)},"drawRels");o(GX,"drawInsideBoundary");MCe=o(function(t,e,r,n){Vt=me().c4;let i=me().securityLevel,a;i==="sandbox"&&(a=Ge("#i"+e));let s=i==="sandbox"?Ge(a.nodes()[0].contentDocument.body):Ge("body"),l=n.db;n.db.setWrap(Vt.wrap),FX=l.getC4ShapeInRow(),J9=l.getC4BoundaryInRow(),Y.debug(`C:${JSON.stringify(Vt,null,2)}`);let u=i==="sandbox"?s.select(`[id="${e}"]`):Ge(`[id="${e}"]`);kl.insertComputerIcon(u),kl.insertDatabaseIcon(u),kl.insertClockIcon(u);let h=new yw(n);h.setData(Vt.diagramMarginX,Vt.diagramMarginX,Vt.diagramMarginY,Vt.diagramMarginY),h.data.widthLimit=screen.availWidth,mw=Vt.diagramMarginX,gw=Vt.diagramMarginY;let f=n.db.getTitle(),d=n.db.getBoundarys("");GX(u,"",h,d,n),kl.insertArrowHead(u),kl.insertArrowEnd(u),kl.insertArrowCrossHead(u),kl.insertArrowFilledHead(u),NCe(u,n.db.getRels(),n.db.getC4Shape,n),h.data.stopx=mw,h.data.stopy=gw;let p=h.data,g=p.stopy-p.starty+2*Vt.diagramMarginY,v=p.stopx-p.startx+2*Vt.diagramMarginX;f&&u.append("text").text(f).attr("x",(p.stopx-p.startx)/2-4*Vt.diagramMarginX).attr("y",p.starty+Vt.diagramMarginY),vn(u,g,v,Vt.useMaxWidth);let x=f?60:0;u.attr("viewBox",p.startx-Vt.diagramMarginX+" -"+(Vt.diagramMarginY+x)+" "+v+" "+(g+x)),Y.debug("models:",p)},"draw"),tD={drawPersonOrSystemArray:zX,drawBoundary:$X,setConf:eD,draw:MCe}});var ICe,UX,HX=N(()=>{"use strict";ICe=o(t=>`.person { + stroke: ${t.personBorder}; + fill: ${t.personBkg}; + } +`,"getStyles"),UX=ICe});var WX={};hr(WX,{diagram:()=>OCe});var OCe,qX=N(()=>{"use strict";$C();uA();VX();HX();OCe={parser:JF,db:Qy,renderer:tD,styles:UX,init:o(({c4:t,wrap:e})=>{tD.setConf(t),Qy.setWrap(e)},"init")}});function uj(t){return typeof t>"u"||t===null}function $Ce(t){return typeof t=="object"&&t!==null}function zCe(t){return Array.isArray(t)?t:uj(t)?[]:[t]}function GCe(t,e){var r,n,i,a;if(e)for(a=Object.keys(e),r=0,n=a.length;rl&&(a=" ... ",e=n-l+a.length),r-n>l&&(s=" ...",r=n+l-s.length),{str:a+t.slice(e,r).replace(/\t/g,"\u2192")+s,pos:n-e+a.length}}function nD(t,e){return $i.repeat(" ",e-t.length)+t}function KCe(t,e){if(e=Object.create(e||null),!t.buffer)return null;e.maxLength||(e.maxLength=79),typeof e.indent!="number"&&(e.indent=1),typeof e.linesBefore!="number"&&(e.linesBefore=3),typeof e.linesAfter!="number"&&(e.linesAfter=2);for(var r=/\r?\n|\r|\0/g,n=[0],i=[],a,s=-1;a=r.exec(t.buffer);)i.push(a.index),n.push(a.index+a[0].length),t.position<=a.index&&s<0&&(s=n.length-2);s<0&&(s=n.length-1);var l="",u,h,f=Math.min(t.line+e.linesAfter,i.length).toString().length,d=e.maxLength-(e.indent+f+3);for(u=1;u<=e.linesBefore&&!(s-u<0);u++)h=rD(t.buffer,n[s-u],i[s-u],t.position-(n[s]-n[s-u]),d),l=$i.repeat(" ",e.indent)+nD((t.line-u+1).toString(),f)+" | "+h.str+` +`+l;for(h=rD(t.buffer,n[s],i[s],t.position,d),l+=$i.repeat(" ",e.indent)+nD((t.line+1).toString(),f)+" | "+h.str+` +`,l+=$i.repeat("-",e.indent+f+3+h.pos)+`^ +`,u=1;u<=e.linesAfter&&!(s+u>=i.length);u++)h=rD(t.buffer,n[s+u],i[s+u],t.position-(n[s]-n[s+u]),d),l+=$i.repeat(" ",e.indent)+nD((t.line+u+1).toString(),f)+" | "+h.str+` +`;return l.replace(/\n$/,"")}function e7e(t){var e={};return t!==null&&Object.keys(t).forEach(function(r){t[r].forEach(function(n){e[String(n)]=r})}),e}function t7e(t,e){if(e=e||{},Object.keys(e).forEach(function(r){if(ZCe.indexOf(r)===-1)throw new Ds('Unknown option "'+r+'" is met in definition of "'+t+'" YAML type.')}),this.options=e,this.tag=t,this.kind=e.kind||null,this.resolve=e.resolve||function(){return!0},this.construct=e.construct||function(r){return r},this.instanceOf=e.instanceOf||null,this.predicate=e.predicate||null,this.represent=e.represent||null,this.representName=e.representName||null,this.defaultStyle=e.defaultStyle||null,this.multi=e.multi||!1,this.styleAliases=e7e(e.styleAliases||null),JCe.indexOf(this.kind)===-1)throw new Ds('Unknown kind "'+this.kind+'" is specified for "'+t+'" YAML type.')}function jX(t,e){var r=[];return t[e].forEach(function(n){var i=r.length;r.forEach(function(a,s){a.tag===n.tag&&a.kind===n.kind&&a.multi===n.multi&&(i=s)}),r[i]=n}),r}function r7e(){var t={scalar:{},sequence:{},mapping:{},fallback:{},multi:{scalar:[],sequence:[],mapping:[],fallback:[]}},e,r;function n(i){i.multi?(t.multi[i.kind].push(i),t.multi.fallback.push(i)):t[i.kind][i.tag]=t.fallback[i.tag]=i}for(o(n,"collectType"),e=0,r=arguments.length;e=0&&(e=e.slice(1)),e===".inf"?r===1?Number.POSITIVE_INFINITY:Number.NEGATIVE_INFINITY:e===".nan"?NaN:r*parseFloat(e,10)}function A7e(t,e){var r;if(isNaN(t))switch(e){case"lowercase":return".nan";case"uppercase":return".NAN";case"camelcase":return".NaN"}else if(Number.POSITIVE_INFINITY===t)switch(e){case"lowercase":return".inf";case"uppercase":return".INF";case"camelcase":return".Inf"}else if(Number.NEGATIVE_INFINITY===t)switch(e){case"lowercase":return"-.inf";case"uppercase":return"-.INF";case"camelcase":return"-.Inf"}else if($i.isNegativeZero(t))return"-0.0";return r=t.toString(10),C7e.test(r)?r.replace("e",".e"):r}function _7e(t){return Object.prototype.toString.call(t)==="[object Number]"&&(t%1!==0||$i.isNegativeZero(t))}function R7e(t){return t===null?!1:dj.exec(t)!==null||pj.exec(t)!==null}function N7e(t){var e,r,n,i,a,s,l,u=0,h=null,f,d,p;if(e=dj.exec(t),e===null&&(e=pj.exec(t)),e===null)throw new Error("Date resolve error");if(r=+e[1],n=+e[2]-1,i=+e[3],!e[4])return new Date(Date.UTC(r,n,i));if(a=+e[4],s=+e[5],l=+e[6],e[7]){for(u=e[7].slice(0,3);u.length<3;)u+="0";u=+u}return e[9]&&(f=+e[10],d=+(e[11]||0),h=(f*60+d)*6e4,e[9]==="-"&&(h=-h)),p=new Date(Date.UTC(r,n,i,a,s,l,u)),h&&p.setTime(p.getTime()-h),p}function M7e(t){return t.toISOString()}function O7e(t){return t==="<<"||t===null}function B7e(t){if(t===null)return!1;var e,r,n=0,i=t.length,a=uD;for(r=0;r64)){if(e<0)return!1;n+=6}return n%8===0}function F7e(t){var e,r,n=t.replace(/[\r\n=]/g,""),i=n.length,a=uD,s=0,l=[];for(e=0;e>16&255),l.push(s>>8&255),l.push(s&255)),s=s<<6|a.indexOf(n.charAt(e));return r=i%4*6,r===0?(l.push(s>>16&255),l.push(s>>8&255),l.push(s&255)):r===18?(l.push(s>>10&255),l.push(s>>2&255)):r===12&&l.push(s>>4&255),new Uint8Array(l)}function $7e(t){var e="",r=0,n,i,a=t.length,s=uD;for(n=0;n>18&63],e+=s[r>>12&63],e+=s[r>>6&63],e+=s[r&63]),r=(r<<8)+t[n];return i=a%3,i===0?(e+=s[r>>18&63],e+=s[r>>12&63],e+=s[r>>6&63],e+=s[r&63]):i===2?(e+=s[r>>10&63],e+=s[r>>4&63],e+=s[r<<2&63],e+=s[64]):i===1&&(e+=s[r>>2&63],e+=s[r<<4&63],e+=s[64],e+=s[64]),e}function z7e(t){return Object.prototype.toString.call(t)==="[object Uint8Array]"}function H7e(t){if(t===null)return!0;var e=[],r,n,i,a,s,l=t;for(r=0,n=l.length;r>10)+55296,(t-65536&1023)+56320)}function cAe(t,e){this.input=t,this.filename=e.filename||null,this.schema=e.schema||mj,this.onWarning=e.onWarning||null,this.legacy=e.legacy||!1,this.json=e.json||!1,this.listener=e.listener||null,this.implicitTypes=this.schema.compiledImplicit,this.typeMap=this.schema.compiledTypeMap,this.length=t.length,this.position=0,this.line=0,this.lineStart=0,this.lineIndent=0,this.firstTabInLine=-1,this.documents=[]}function Tj(t,e){var r={name:t.filename,buffer:t.input.slice(0,-1),position:t.position,line:t.line,column:t.position-t.lineStart};return r.snippet=QCe(r),new Ds(e,r)}function Qt(t,e){throw Tj(t,e)}function bw(t,e){t.onWarning&&t.onWarning.call(null,Tj(t,e))}function zh(t,e,r,n){var i,a,s,l;if(e1&&(t.result+=$i.repeat(` +`,e-1))}function uAe(t,e,r){var n,i,a,s,l,u,h,f,d=t.kind,p=t.result,m;if(m=t.input.charCodeAt(t.position),Ls(m)||am(m)||m===35||m===38||m===42||m===33||m===124||m===62||m===39||m===34||m===37||m===64||m===96||(m===63||m===45)&&(i=t.input.charCodeAt(t.position+1),Ls(i)||r&&am(i)))return!1;for(t.kind="scalar",t.result="",a=s=t.position,l=!1;m!==0;){if(m===58){if(i=t.input.charCodeAt(t.position+1),Ls(i)||r&&am(i))break}else if(m===35){if(n=t.input.charCodeAt(t.position-1),Ls(n))break}else{if(t.position===t.lineStart&&kw(t)||r&&am(m))break;if(dc(m))if(u=t.line,h=t.lineStart,f=t.lineIndent,Ci(t,!1,-1),t.lineIndent>=e){l=!0,m=t.input.charCodeAt(t.position);continue}else{t.position=s,t.line=u,t.lineStart=h,t.lineIndent=f;break}}l&&(zh(t,a,s,!1),fD(t,t.line-u),a=s=t.position,l=!1),Nd(m)||(s=t.position+1),m=t.input.charCodeAt(++t.position)}return zh(t,a,s,!1),t.result?!0:(t.kind=d,t.result=p,!1)}function hAe(t,e){var r,n,i;if(r=t.input.charCodeAt(t.position),r!==39)return!1;for(t.kind="scalar",t.result="",t.position++,n=i=t.position;(r=t.input.charCodeAt(t.position))!==0;)if(r===39)if(zh(t,n,t.position,!0),r=t.input.charCodeAt(++t.position),r===39)n=t.position,t.position++,i=t.position;else return!0;else dc(r)?(zh(t,n,i,!0),fD(t,Ci(t,!1,e)),n=i=t.position):t.position===t.lineStart&&kw(t)?Qt(t,"unexpected end of the document within a single quoted scalar"):(t.position++,i=t.position);Qt(t,"unexpected end of the stream within a single quoted scalar")}function fAe(t,e){var r,n,i,a,s,l;if(l=t.input.charCodeAt(t.position),l!==34)return!1;for(t.kind="scalar",t.result="",t.position++,r=n=t.position;(l=t.input.charCodeAt(t.position))!==0;){if(l===34)return zh(t,r,t.position,!0),t.position++,!0;if(l===92){if(zh(t,r,t.position,!0),l=t.input.charCodeAt(++t.position),dc(l))Ci(t,!1,e);else if(l<256&&bj[l])t.result+=wj[l],t.position++;else if((s=sAe(l))>0){for(i=s,a=0;i>0;i--)l=t.input.charCodeAt(++t.position),(s=aAe(l))>=0?a=(a<<4)+s:Qt(t,"expected hexadecimal character");t.result+=lAe(a),t.position++}else Qt(t,"unknown escape sequence");r=n=t.position}else dc(l)?(zh(t,r,n,!0),fD(t,Ci(t,!1,e)),r=n=t.position):t.position===t.lineStart&&kw(t)?Qt(t,"unexpected end of the document within a double quoted scalar"):(t.position++,n=t.position)}Qt(t,"unexpected end of the stream within a double quoted scalar")}function dAe(t,e){var r=!0,n,i,a,s=t.tag,l,u=t.anchor,h,f,d,p,m,g=Object.create(null),y,v,x,b;if(b=t.input.charCodeAt(t.position),b===91)f=93,m=!1,l=[];else if(b===123)f=125,m=!0,l={};else return!1;for(t.anchor!==null&&(t.anchorMap[t.anchor]=l),b=t.input.charCodeAt(++t.position);b!==0;){if(Ci(t,!0,e),b=t.input.charCodeAt(t.position),b===f)return t.position++,t.tag=s,t.anchor=u,t.kind=m?"mapping":"sequence",t.result=l,!0;r?b===44&&Qt(t,"expected the node content, but found ','"):Qt(t,"missed comma between flow collection entries"),v=y=x=null,d=p=!1,b===63&&(h=t.input.charCodeAt(t.position+1),Ls(h)&&(d=p=!0,t.position++,Ci(t,!0,e))),n=t.line,i=t.lineStart,a=t.position,om(t,e,vw,!1,!0),v=t.tag,y=t.result,Ci(t,!0,e),b=t.input.charCodeAt(t.position),(p||t.line===n)&&b===58&&(d=!0,b=t.input.charCodeAt(++t.position),Ci(t,!0,e),om(t,e,vw,!1,!0),x=t.result),m?sm(t,l,g,v,y,x,n,i,a):d?l.push(sm(t,null,g,v,y,x,n,i,a)):l.push(y),Ci(t,!0,e),b=t.input.charCodeAt(t.position),b===44?(r=!0,b=t.input.charCodeAt(++t.position)):r=!1}Qt(t,"unexpected end of the stream within a flow collection")}function pAe(t,e){var r,n,i=iD,a=!1,s=!1,l=e,u=0,h=!1,f,d;if(d=t.input.charCodeAt(t.position),d===124)n=!1;else if(d===62)n=!0;else return!1;for(t.kind="scalar",t.result="";d!==0;)if(d=t.input.charCodeAt(++t.position),d===43||d===45)iD===i?i=d===43?KX:tAe:Qt(t,"repeat of a chomping mode identifier");else if((f=oAe(d))>=0)f===0?Qt(t,"bad explicit indentation width of a block scalar; it cannot be less than one"):s?Qt(t,"repeat of an indentation width identifier"):(l=e+f-1,s=!0);else break;if(Nd(d)){do d=t.input.charCodeAt(++t.position);while(Nd(d));if(d===35)do d=t.input.charCodeAt(++t.position);while(!dc(d)&&d!==0)}for(;d!==0;){for(hD(t),t.lineIndent=0,d=t.input.charCodeAt(t.position);(!s||t.lineIndentl&&(l=t.lineIndent),dc(d)){u++;continue}if(t.lineIndente)&&u!==0)Qt(t,"bad indentation of a sequence entry");else if(t.lineIndente)&&(v&&(s=t.line,l=t.lineStart,u=t.position),om(t,e,xw,!0,i)&&(v?g=t.result:y=t.result),v||(sm(t,d,p,m,g,y,s,l,u),m=g=y=null),Ci(t,!0,-1),b=t.input.charCodeAt(t.position)),(t.line===a||t.lineIndent>e)&&b!==0)Qt(t,"bad indentation of a mapping entry");else if(t.lineIndente?u=1:t.lineIndent===e?u=0:t.lineIndente?u=1:t.lineIndent===e?u=0:t.lineIndent tag; it should be "scalar", not "'+t.kind+'"'),d=0,p=t.implicitTypes.length;d"),t.result!==null&&g.kind!==t.kind&&Qt(t,"unacceptable node kind for !<"+t.tag+'> tag; it should be "'+g.kind+'", not "'+t.kind+'"'),g.resolve(t.result,t.tag)?(t.result=g.construct(t.result,t.tag),t.anchor!==null&&(t.anchorMap[t.anchor]=t.result)):Qt(t,"cannot resolve a node with !<"+t.tag+"> explicit tag")}return t.listener!==null&&t.listener("close",t),t.tag!==null||t.anchor!==null||f}function xAe(t){var e=t.position,r,n,i,a=!1,s;for(t.version=null,t.checkLineBreaks=t.legacy,t.tagMap=Object.create(null),t.anchorMap=Object.create(null);(s=t.input.charCodeAt(t.position))!==0&&(Ci(t,!0,-1),s=t.input.charCodeAt(t.position),!(t.lineIndent>0||s!==37));){for(a=!0,s=t.input.charCodeAt(++t.position),r=t.position;s!==0&&!Ls(s);)s=t.input.charCodeAt(++t.position);for(n=t.input.slice(r,t.position),i=[],n.length<1&&Qt(t,"directive name must not be less than one character in length");s!==0;){for(;Nd(s);)s=t.input.charCodeAt(++t.position);if(s===35){do s=t.input.charCodeAt(++t.position);while(s!==0&&!dc(s));break}if(dc(s))break;for(r=t.position;s!==0&&!Ls(s);)s=t.input.charCodeAt(++t.position);i.push(t.input.slice(r,t.position))}s!==0&&hD(t),Gh.call(JX,n)?JX[n](t,n,i):bw(t,'unknown document directive "'+n+'"')}if(Ci(t,!0,-1),t.lineIndent===0&&t.input.charCodeAt(t.position)===45&&t.input.charCodeAt(t.position+1)===45&&t.input.charCodeAt(t.position+2)===45?(t.position+=3,Ci(t,!0,-1)):a&&Qt(t,"directives end mark is expected"),om(t,t.lineIndent-1,xw,!1,!0),Ci(t,!0,-1),t.checkLineBreaks&&nAe.test(t.input.slice(e,t.position))&&bw(t,"non-ASCII line breaks are interpreted as content"),t.documents.push(t.result),t.position===t.lineStart&&kw(t)){t.input.charCodeAt(t.position)===46&&(t.position+=3,Ci(t,!0,-1));return}if(t.position"u"&&(r=e,e=null);var n=kj(t,r);if(typeof e!="function")return n;for(var i=0,a=n.length;i=55296&&r<=56319&&e+1=56320&&n<=57343)?(r-55296)*1024+n-56320+65536:r}function Nj(t){var e=/^\n* /;return e.test(t)}function jAe(t,e,r,n,i,a,s,l){var u,h=0,f=null,d=!1,p=!1,m=n!==-1,g=-1,y=YAe(s2(t,0))&&XAe(s2(t,t.length-1));if(e||s)for(u=0;u=65536?u+=2:u++){if(h=s2(t,u),!u2(h))return im;y=y&&ij(h,f,l),f=h}else{for(u=0;u=65536?u+=2:u++){if(h=s2(t,u),h===l2)d=!0,m&&(p=p||u-g-1>n&&t[g+1]!==" ",g=u);else if(!u2(h))return im;y=y&&ij(h,f,l),f=h}p=p||m&&u-g-1>n&&t[g+1]!==" "}return!d&&!p?y&&!s&&!i(t)?Mj:a===c2?im:lD:r>9&&Nj(t)?im:s?a===c2?im:lD:p?Oj:Ij}function KAe(t,e,r,n,i){t.dump=function(){if(e.length===0)return t.quotingType===c2?'""':"''";if(!t.noCompatMode&&(zAe.indexOf(e)!==-1||GAe.test(e)))return t.quotingType===c2?'"'+e+'"':"'"+e+"'";var a=t.indent*Math.max(1,r),s=t.lineWidth===-1?-1:Math.max(Math.min(t.lineWidth,40),t.lineWidth-a),l=n||t.flowLevel>-1&&r>=t.flowLevel;function u(h){return qAe(t,h)}switch(o(u,"testAmbiguity"),jAe(e,l,t.indent,s,u,t.quotingType,t.forceQuotes&&!n,i)){case Mj:return e;case lD:return"'"+e.replace(/'/g,"''")+"'";case Ij:return"|"+aj(e,t.indent)+sj(rj(e,a));case Oj:return">"+aj(e,t.indent)+sj(rj(QAe(e,s),a));case im:return'"'+ZAe(e)+'"';default:throw new Ds("impossible error: invalid scalar style")}}()}function aj(t,e){var r=Nj(t)?String(e):"",n=t[t.length-1]===` +`,i=n&&(t[t.length-2]===` +`||t===` +`),a=i?"+":n?"":"-";return r+a+` +`}function sj(t){return t[t.length-1]===` +`?t.slice(0,-1):t}function QAe(t,e){for(var r=/(\n+)([^\n]*)/g,n=function(){var h=t.indexOf(` +`);return h=h!==-1?h:t.length,r.lastIndex=h,oj(t.slice(0,h),e)}(),i=t[0]===` +`||t[0]===" ",a,s;s=r.exec(t);){var l=s[1],u=s[2];a=u[0]===" ",n+=l+(!i&&!a&&u!==""?` +`:"")+oj(u,e),i=a}return n}function oj(t,e){if(t===""||t[0]===" ")return t;for(var r=/ [^ ]/g,n,i=0,a,s=0,l=0,u="";n=r.exec(t);)l=n.index,l-i>e&&(a=s>i?s:l,u+=` +`+t.slice(i,a),i=a+1),s=l;return u+=` +`,t.length-i>e&&s>i?u+=t.slice(i,s)+` +`+t.slice(s+1):u+=t.slice(i),u.slice(1)}function ZAe(t){for(var e="",r=0,n,i=0;i=65536?i+=2:i++)r=s2(t,i),n=Da[r],!n&&u2(r)?(e+=t[i],r>=65536&&(e+=t[i+1])):e+=n||UAe(r);return e}function JAe(t,e,r){var n="",i=t.tag,a,s,l;for(a=0,s=r.length;a"u"&&Au(t,e,null,!1,!1))&&(n!==""&&(n+=","+(t.condenseFlow?"":" ")),n+=t.dump);t.tag=i,t.dump="["+n+"]"}function lj(t,e,r,n){var i="",a=t.tag,s,l,u;for(s=0,l=r.length;s"u"&&Au(t,e+1,null,!0,!0,!1,!0))&&((!n||i!=="")&&(i+=oD(t,e)),t.dump&&l2===t.dump.charCodeAt(0)?i+="-":i+="- ",i+=t.dump);t.tag=a,t.dump=i||"[]"}function e8e(t,e,r){var n="",i=t.tag,a=Object.keys(r),s,l,u,h,f;for(s=0,l=a.length;s1024&&(f+="? "),f+=t.dump+(t.condenseFlow?'"':"")+":"+(t.condenseFlow?"":" "),Au(t,e,h,!1,!1)&&(f+=t.dump,n+=f));t.tag=i,t.dump="{"+n+"}"}function t8e(t,e,r,n){var i="",a=t.tag,s=Object.keys(r),l,u,h,f,d,p;if(t.sortKeys===!0)s.sort();else if(typeof t.sortKeys=="function")s.sort(t.sortKeys);else if(t.sortKeys)throw new Ds("sortKeys must be a boolean or a function");for(l=0,u=s.length;l1024,d&&(t.dump&&l2===t.dump.charCodeAt(0)?p+="?":p+="? "),p+=t.dump,d&&(p+=oD(t,e)),Au(t,e+1,f,!0,d)&&(t.dump&&l2===t.dump.charCodeAt(0)?p+=":":p+=": ",p+=t.dump,i+=p));t.tag=a,t.dump=i||"{}"}function cj(t,e,r){var n,i,a,s,l,u;for(i=r?t.explicitTypes:t.implicitTypes,a=0,s=i.length;a tag resolver accepts not "'+u+'" style');t.dump=n}return!0}return!1}function Au(t,e,r,n,i,a,s){t.tag=null,t.dump=r,cj(t,r,!1)||cj(t,r,!0);var l=Sj.call(t.dump),u=n,h;n&&(n=t.flowLevel<0||t.flowLevel>e);var f=l==="[object Object]"||l==="[object Array]",d,p;if(f&&(d=t.duplicates.indexOf(r),p=d!==-1),(t.tag!==null&&t.tag!=="?"||p||t.indent!==2&&e>0)&&(i=!1),p&&t.usedDuplicates[d])t.dump="*ref_"+d;else{if(f&&p&&!t.usedDuplicates[d]&&(t.usedDuplicates[d]=!0),l==="[object Object]")n&&Object.keys(t.dump).length!==0?(t8e(t,e,t.dump,i),p&&(t.dump="&ref_"+d+t.dump)):(e8e(t,e,t.dump),p&&(t.dump="&ref_"+d+" "+t.dump));else if(l==="[object Array]")n&&t.dump.length!==0?(t.noArrayIndent&&!s&&e>0?lj(t,e-1,t.dump,i):lj(t,e,t.dump,i),p&&(t.dump="&ref_"+d+t.dump)):(JAe(t,e,t.dump),p&&(t.dump="&ref_"+d+" "+t.dump));else if(l==="[object String]")t.tag!=="?"&&KAe(t,t.dump,e,a,u);else{if(l==="[object Undefined]")return!1;if(t.skipInvalid)return!1;throw new Ds("unacceptable kind of an object to dump "+l)}t.tag!==null&&t.tag!=="?"&&(h=encodeURI(t.tag[0]==="!"?t.tag.slice(1):t.tag).replace(/!/g,"%21"),t.tag[0]==="!"?h="!"+h:h.slice(0,18)==="tag:yaml.org,2002:"?h="!!"+h.slice(18):h="!<"+h+">",t.dump=h+" "+t.dump)}return!0}function r8e(t,e){var r=[],n=[],i,a;for(cD(t,r,n),i=0,a=n.length;i{"use strict";o(uj,"isNothing");o($Ce,"isObject");o(zCe,"toArray");o(GCe,"extend");o(VCe,"repeat");o(UCe,"isNegativeZero");HCe=uj,WCe=$Ce,qCe=zCe,YCe=VCe,XCe=UCe,jCe=GCe,$i={isNothing:HCe,isObject:WCe,toArray:qCe,repeat:YCe,isNegativeZero:XCe,extend:jCe};o(hj,"formatError");o(o2,"YAMLException$1");o2.prototype=Object.create(Error.prototype);o2.prototype.constructor=o2;o2.prototype.toString=o(function(e){return this.name+": "+hj(this,e)},"toString");Ds=o2;o(rD,"getLine");o(nD,"padStart");o(KCe,"makeSnippet");QCe=KCe,ZCe=["kind","multi","resolve","construct","instanceOf","predicate","represent","representName","defaultStyle","styleAliases"],JCe=["scalar","sequence","mapping"];o(e7e,"compileStyleAliases");o(t7e,"Type$1");_a=t7e;o(jX,"compileList");o(r7e,"compileMap");o(aD,"Schema$1");aD.prototype.extend=o(function(e){var r=[],n=[];if(e instanceof _a)n.push(e);else if(Array.isArray(e))n=n.concat(e);else if(e&&(Array.isArray(e.implicit)||Array.isArray(e.explicit)))e.implicit&&(r=r.concat(e.implicit)),e.explicit&&(n=n.concat(e.explicit));else throw new Ds("Schema.extend argument should be a Type, [ Type ], or a schema definition ({ implicit: [...], explicit: [...] })");r.forEach(function(a){if(!(a instanceof _a))throw new Ds("Specified list of YAML types (or a single Type object) contains a non-Type object.");if(a.loadKind&&a.loadKind!=="scalar")throw new Ds("There is a non-scalar type in the implicit list of a schema. Implicit resolving of such types is not supported.");if(a.multi)throw new Ds("There is a multi type in the implicit list of a schema. Multi tags can only be listed as explicit.")}),n.forEach(function(a){if(!(a instanceof _a))throw new Ds("Specified list of YAML types (or a single Type object) contains a non-Type object.")});var i=Object.create(aD.prototype);return i.implicit=(this.implicit||[]).concat(r),i.explicit=(this.explicit||[]).concat(n),i.compiledImplicit=jX(i,"implicit"),i.compiledExplicit=jX(i,"explicit"),i.compiledTypeMap=r7e(i.compiledImplicit,i.compiledExplicit),i},"extend");n7e=aD,i7e=new _a("tag:yaml.org,2002:str",{kind:"scalar",construct:o(function(t){return t!==null?t:""},"construct")}),a7e=new _a("tag:yaml.org,2002:seq",{kind:"sequence",construct:o(function(t){return t!==null?t:[]},"construct")}),s7e=new _a("tag:yaml.org,2002:map",{kind:"mapping",construct:o(function(t){return t!==null?t:{}},"construct")}),o7e=new n7e({explicit:[i7e,a7e,s7e]});o(l7e,"resolveYamlNull");o(c7e,"constructYamlNull");o(u7e,"isNull");h7e=new _a("tag:yaml.org,2002:null",{kind:"scalar",resolve:l7e,construct:c7e,predicate:u7e,represent:{canonical:o(function(){return"~"},"canonical"),lowercase:o(function(){return"null"},"lowercase"),uppercase:o(function(){return"NULL"},"uppercase"),camelcase:o(function(){return"Null"},"camelcase"),empty:o(function(){return""},"empty")},defaultStyle:"lowercase"});o(f7e,"resolveYamlBoolean");o(d7e,"constructYamlBoolean");o(p7e,"isBoolean");m7e=new _a("tag:yaml.org,2002:bool",{kind:"scalar",resolve:f7e,construct:d7e,predicate:p7e,represent:{lowercase:o(function(t){return t?"true":"false"},"lowercase"),uppercase:o(function(t){return t?"TRUE":"FALSE"},"uppercase"),camelcase:o(function(t){return t?"True":"False"},"camelcase")},defaultStyle:"lowercase"});o(g7e,"isHexCode");o(y7e,"isOctCode");o(v7e,"isDecCode");o(x7e,"resolveYamlInteger");o(b7e,"constructYamlInteger");o(w7e,"isInteger");T7e=new _a("tag:yaml.org,2002:int",{kind:"scalar",resolve:x7e,construct:b7e,predicate:w7e,represent:{binary:o(function(t){return t>=0?"0b"+t.toString(2):"-0b"+t.toString(2).slice(1)},"binary"),octal:o(function(t){return t>=0?"0o"+t.toString(8):"-0o"+t.toString(8).slice(1)},"octal"),decimal:o(function(t){return t.toString(10)},"decimal"),hexadecimal:o(function(t){return t>=0?"0x"+t.toString(16).toUpperCase():"-0x"+t.toString(16).toUpperCase().slice(1)},"hexadecimal")},defaultStyle:"decimal",styleAliases:{binary:[2,"bin"],octal:[8,"oct"],decimal:[10,"dec"],hexadecimal:[16,"hex"]}}),k7e=new RegExp("^(?:[-+]?(?:[0-9][0-9_]*)(?:\\.[0-9_]*)?(?:[eE][-+]?[0-9]+)?|\\.[0-9_]+(?:[eE][-+]?[0-9]+)?|[-+]?\\.(?:inf|Inf|INF)|\\.(?:nan|NaN|NAN))$");o(E7e,"resolveYamlFloat");o(S7e,"constructYamlFloat");C7e=/^[-+]?[0-9]+e/;o(A7e,"representYamlFloat");o(_7e,"isFloat");D7e=new _a("tag:yaml.org,2002:float",{kind:"scalar",resolve:E7e,construct:S7e,predicate:_7e,represent:A7e,defaultStyle:"lowercase"}),fj=o7e.extend({implicit:[h7e,m7e,T7e,D7e]}),L7e=fj,dj=new RegExp("^([0-9][0-9][0-9][0-9])-([0-9][0-9])-([0-9][0-9])$"),pj=new RegExp("^([0-9][0-9][0-9][0-9])-([0-9][0-9]?)-([0-9][0-9]?)(?:[Tt]|[ \\t]+)([0-9][0-9]?):([0-9][0-9]):([0-9][0-9])(?:\\.([0-9]*))?(?:[ \\t]*(Z|([-+])([0-9][0-9]?)(?::([0-9][0-9]))?))?$");o(R7e,"resolveYamlTimestamp");o(N7e,"constructYamlTimestamp");o(M7e,"representYamlTimestamp");I7e=new _a("tag:yaml.org,2002:timestamp",{kind:"scalar",resolve:R7e,construct:N7e,instanceOf:Date,represent:M7e});o(O7e,"resolveYamlMerge");P7e=new _a("tag:yaml.org,2002:merge",{kind:"scalar",resolve:O7e}),uD=`ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/= +\r`;o(B7e,"resolveYamlBinary");o(F7e,"constructYamlBinary");o($7e,"representYamlBinary");o(z7e,"isBinary");G7e=new _a("tag:yaml.org,2002:binary",{kind:"scalar",resolve:B7e,construct:F7e,predicate:z7e,represent:$7e}),V7e=Object.prototype.hasOwnProperty,U7e=Object.prototype.toString;o(H7e,"resolveYamlOmap");o(W7e,"constructYamlOmap");q7e=new _a("tag:yaml.org,2002:omap",{kind:"sequence",resolve:H7e,construct:W7e}),Y7e=Object.prototype.toString;o(X7e,"resolveYamlPairs");o(j7e,"constructYamlPairs");K7e=new _a("tag:yaml.org,2002:pairs",{kind:"sequence",resolve:X7e,construct:j7e}),Q7e=Object.prototype.hasOwnProperty;o(Z7e,"resolveYamlSet");o(J7e,"constructYamlSet");eAe=new _a("tag:yaml.org,2002:set",{kind:"mapping",resolve:Z7e,construct:J7e}),mj=L7e.extend({implicit:[I7e,P7e],explicit:[G7e,q7e,K7e,eAe]}),Gh=Object.prototype.hasOwnProperty,vw=1,gj=2,yj=3,xw=4,iD=1,tAe=2,KX=3,rAe=/[\x00-\x08\x0B\x0C\x0E-\x1F\x7F-\x84\x86-\x9F\uFFFE\uFFFF]|[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?:[^\uD800-\uDBFF]|^)[\uDC00-\uDFFF]/,nAe=/[\x85\u2028\u2029]/,iAe=/[,\[\]\{\}]/,vj=/^(?:!|!!|![a-z\-]+!)$/i,xj=/^(?:!|[^,\[\]\{\}])(?:%[0-9a-f]{2}|[0-9a-z\-#;\/\?:@&=\+\$,_\.!~\*'\(\)\[\]])*$/i;o(QX,"_class");o(dc,"is_EOL");o(Nd,"is_WHITE_SPACE");o(Ls,"is_WS_OR_EOL");o(am,"is_FLOW_INDICATOR");o(aAe,"fromHexCode");o(sAe,"escapedHexLen");o(oAe,"fromDecimalCode");o(ZX,"simpleEscapeSequence");o(lAe,"charFromCodepoint");bj=new Array(256),wj=new Array(256);for(Rd=0;Rd<256;Rd++)bj[Rd]=ZX(Rd)?1:0,wj[Rd]=ZX(Rd);o(cAe,"State$1");o(Tj,"generateError");o(Qt,"throwError");o(bw,"throwWarning");JX={YAML:o(function(e,r,n){var i,a,s;e.version!==null&&Qt(e,"duplication of %YAML directive"),n.length!==1&&Qt(e,"YAML directive accepts exactly one argument"),i=/^([0-9]+)\.([0-9]+)$/.exec(n[0]),i===null&&Qt(e,"ill-formed argument of the YAML directive"),a=parseInt(i[1],10),s=parseInt(i[2],10),a!==1&&Qt(e,"unacceptable YAML version of the document"),e.version=n[0],e.checkLineBreaks=s<2,s!==1&&s!==2&&bw(e,"unsupported YAML version of the document")},"handleYamlDirective"),TAG:o(function(e,r,n){var i,a;n.length!==2&&Qt(e,"TAG directive accepts exactly two arguments"),i=n[0],a=n[1],vj.test(i)||Qt(e,"ill-formed tag handle (first argument) of the TAG directive"),Gh.call(e.tagMap,i)&&Qt(e,'there is a previously declared suffix for "'+i+'" tag handle'),xj.test(a)||Qt(e,"ill-formed tag prefix (second argument) of the TAG directive");try{a=decodeURIComponent(a)}catch{Qt(e,"tag prefix is malformed: "+a)}e.tagMap[i]=a},"handleTagDirective")};o(zh,"captureSegment");o(ej,"mergeMappings");o(sm,"storeMappingPair");o(hD,"readLineBreak");o(Ci,"skipSeparationSpace");o(kw,"testDocumentSeparator");o(fD,"writeFoldedLines");o(uAe,"readPlainScalar");o(hAe,"readSingleQuotedScalar");o(fAe,"readDoubleQuotedScalar");o(dAe,"readFlowCollection");o(pAe,"readBlockScalar");o(tj,"readBlockSequence");o(mAe,"readBlockMapping");o(gAe,"readTagProperty");o(yAe,"readAnchorProperty");o(vAe,"readAlias");o(om,"composeNode");o(xAe,"readDocument");o(kj,"loadDocuments");o(bAe,"loadAll$1");o(wAe,"load$1");TAe=bAe,kAe=wAe,Ej={loadAll:TAe,load:kAe},Sj=Object.prototype.toString,Cj=Object.prototype.hasOwnProperty,dD=65279,EAe=9,l2=10,SAe=13,CAe=32,AAe=33,_Ae=34,sD=35,DAe=37,LAe=38,RAe=39,NAe=42,Aj=44,MAe=45,ww=58,IAe=61,OAe=62,PAe=63,BAe=64,_j=91,Dj=93,FAe=96,Lj=123,$Ae=124,Rj=125,Da={};Da[0]="\\0";Da[7]="\\a";Da[8]="\\b";Da[9]="\\t";Da[10]="\\n";Da[11]="\\v";Da[12]="\\f";Da[13]="\\r";Da[27]="\\e";Da[34]='\\"';Da[92]="\\\\";Da[133]="\\N";Da[160]="\\_";Da[8232]="\\L";Da[8233]="\\P";zAe=["y","Y","yes","Yes","YES","on","On","ON","n","N","no","No","NO","off","Off","OFF"],GAe=/^[-+]?[0-9_]+(?::[0-9_]+)+(?:\.[0-9_]*)?$/;o(VAe,"compileStyleMap");o(UAe,"encodeHex");HAe=1,c2=2;o(WAe,"State");o(rj,"indentString");o(oD,"generateNextLine");o(qAe,"testImplicitResolving");o(Tw,"isWhitespace");o(u2,"isPrintable");o(nj,"isNsCharOrWhitespace");o(ij,"isPlainSafe");o(YAe,"isPlainSafeFirst");o(XAe,"isPlainSafeLast");o(s2,"codePointAt");o(Nj,"needIndentIndicator");Mj=1,lD=2,Ij=3,Oj=4,im=5;o(jAe,"chooseScalarStyle");o(KAe,"writeScalar");o(aj,"blockHeader");o(sj,"dropEndingNewline");o(QAe,"foldString");o(oj,"foldLine");o(ZAe,"escapeString");o(JAe,"writeFlowSequence");o(lj,"writeBlockSequence");o(e8e,"writeFlowMapping");o(t8e,"writeBlockMapping");o(cj,"detectType");o(Au,"writeNode");o(r8e,"getDuplicateReferences");o(cD,"inspectNode");o(n8e,"dump$1");i8e=n8e,a8e={dump:i8e};o(pD,"renamed");lm=fj,cm=Ej.load,okt=Ej.loadAll,lkt=a8e.dump,ckt=pD("safeLoad","load"),ukt=pD("safeLoadAll","loadAll"),hkt=pD("safeDump","dump")});function vD(){return{async:!1,breaks:!1,extensions:null,gfm:!0,hooks:null,pedantic:!1,renderer:null,silent:!1,tokenizer:null,walkTokens:null}}function Gj(t){Id=t}function nn(t,e=""){let r=typeof t=="string"?t:t.source,n={replace:o((i,a)=>{let s=typeof a=="string"?a:a.source;return s=s.replace(ts.caret,"$1"),r=r.replace(i,s),n},"replace"),getRegex:o(()=>new RegExp(r,e),"getRegex")};return n}function pc(t,e){if(e){if(ts.escapeTest.test(t))return t.replace(ts.escapeReplace,Bj)}else if(ts.escapeTestNoEncode.test(t))return t.replace(ts.escapeReplaceNoEncode,Bj);return t}function Fj(t){try{t=encodeURI(t).replace(ts.percentDecode,"%")}catch{return null}return t}function $j(t,e){let r=t.replace(ts.findPipe,(a,s,l)=>{let u=!1,h=s;for(;--h>=0&&l[h]==="\\";)u=!u;return u?"|":" |"}),n=r.split(ts.splitPipe),i=0;if(n[0].trim()||n.shift(),n.length>0&&!n.at(-1)?.trim()&&n.pop(),e)if(n.length>e)n.splice(e);else for(;n.length{let s=a.match(r.other.beginningSpace);if(s===null)return a;let[l]=s;return l.length>=i.length?a.slice(i.length):a}).join(` +`)}function Jr(t,e){return Md.parse(t,e)}var Id,d2,ts,s8e,o8e,l8e,m2,c8e,xD,Vj,Uj,u8e,bD,h8e,wD,f8e,d8e,Aw,TD,p8e,Hj,m8e,kD,Pj,g8e,y8e,v8e,x8e,Wj,b8e,_w,ED,qj,w8e,Yj,T8e,k8e,E8e,Xj,S8e,C8e,jj,A8e,_8e,D8e,L8e,R8e,N8e,M8e,Cw,I8e,Kj,Qj,O8e,SD,P8e,gD,B8e,Sw,h2,F8e,Bj,hm,Al,fm,p2,_l,um,yD,Md,dkt,pkt,mkt,gkt,ykt,vkt,xkt,Zj=N(()=>{"use strict";o(vD,"_getDefaults");Id=vD();o(Gj,"changeDefaults");d2={exec:o(()=>null,"exec")};o(nn,"edit");ts={codeRemoveIndent:/^(?: {1,4}| {0,3}\t)/gm,outputLinkReplace:/\\([\[\]])/g,indentCodeCompensation:/^(\s+)(?:```)/,beginningSpace:/^\s+/,endingHash:/#$/,startingSpaceChar:/^ /,endingSpaceChar:/ $/,nonSpaceChar:/[^ ]/,newLineCharGlobal:/\n/g,tabCharGlobal:/\t/g,multipleSpaceGlobal:/\s+/g,blankLine:/^[ \t]*$/,doubleBlankLine:/\n[ \t]*\n[ \t]*$/,blockquoteStart:/^ {0,3}>/,blockquoteSetextReplace:/\n {0,3}((?:=+|-+) *)(?=\n|$)/g,blockquoteSetextReplace2:/^ {0,3}>[ \t]?/gm,listReplaceTabs:/^\t+/,listReplaceNesting:/^ {1,4}(?=( {4})*[^ ])/g,listIsTask:/^\[[ xX]\] /,listReplaceTask:/^\[[ xX]\] +/,anyLine:/\n.*\n/,hrefBrackets:/^<(.*)>$/,tableDelimiter:/[:|]/,tableAlignChars:/^\||\| *$/g,tableRowBlankLine:/\n[ \t]*$/,tableAlignRight:/^ *-+: *$/,tableAlignCenter:/^ *:-+: *$/,tableAlignLeft:/^ *:-+ *$/,startATag:/^/i,startPreScriptTag:/^<(pre|code|kbd|script)(\s|>)/i,endPreScriptTag:/^<\/(pre|code|kbd|script)(\s|>)/i,startAngleBracket:/^$/,pedanticHrefTitle:/^([^'"]*[^\s])\s+(['"])(.*)\2/,unicodeAlphaNumeric:/[\p{L}\p{N}]/u,escapeTest:/[&<>"']/,escapeReplace:/[&<>"']/g,escapeTestNoEncode:/[<>"']|&(?!(#\d{1,7}|#[Xx][a-fA-F0-9]{1,6}|\w+);)/,escapeReplaceNoEncode:/[<>"']|&(?!(#\d{1,7}|#[Xx][a-fA-F0-9]{1,6}|\w+);)/g,unescapeTest:/&(#(?:\d+)|(?:#x[0-9A-Fa-f]+)|(?:\w+));?/ig,caret:/(^|[^\[])\^/g,percentDecode:/%25/g,findPipe:/\|/g,splitPipe:/ \|/,slashPipe:/\\\|/g,carriageReturn:/\r\n|\r/g,spaceLine:/^ +$/gm,notSpaceStart:/^\S*/,endingNewline:/\n$/,listItemRegex:o(t=>new RegExp(`^( {0,3}${t})((?:[ ][^\\n]*)?(?:\\n|$))`),"listItemRegex"),nextBulletRegex:o(t=>new RegExp(`^ {0,${Math.min(3,t-1)}}(?:[*+-]|\\d{1,9}[.)])((?:[ ][^\\n]*)?(?:\\n|$))`),"nextBulletRegex"),hrRegex:o(t=>new RegExp(`^ {0,${Math.min(3,t-1)}}((?:- *){3,}|(?:_ *){3,}|(?:\\* *){3,})(?:\\n+|$)`),"hrRegex"),fencesBeginRegex:o(t=>new RegExp(`^ {0,${Math.min(3,t-1)}}(?:\`\`\`|~~~)`),"fencesBeginRegex"),headingBeginRegex:o(t=>new RegExp(`^ {0,${Math.min(3,t-1)}}#`),"headingBeginRegex"),htmlBeginRegex:o(t=>new RegExp(`^ {0,${Math.min(3,t-1)}}<(?:[a-z].*>|!--)`,"i"),"htmlBeginRegex")},s8e=/^(?:[ \t]*(?:\n|$))+/,o8e=/^((?: {4}| {0,3}\t)[^\n]+(?:\n(?:[ \t]*(?:\n|$))*)?)+/,l8e=/^ {0,3}(`{3,}(?=[^`\n]*(?:\n|$))|~{3,})([^\n]*)(?:\n|$)(?:|([\s\S]*?)(?:\n|$))(?: {0,3}\1[~`]* *(?=\n|$)|$)/,m2=/^ {0,3}((?:-[\t ]*){3,}|(?:_[ \t]*){3,}|(?:\*[ \t]*){3,})(?:\n+|$)/,c8e=/^ {0,3}(#{1,6})(?=\s|$)(.*)(?:\n+|$)/,xD=/(?:[*+-]|\d{1,9}[.)])/,Vj=/^(?!bull |blockCode|fences|blockquote|heading|html|table)((?:.|\n(?!\s*?\n|bull |blockCode|fences|blockquote|heading|html|table))+?)\n {0,3}(=+|-+) *(?:\n+|$)/,Uj=nn(Vj).replace(/bull/g,xD).replace(/blockCode/g,/(?: {4}| {0,3}\t)/).replace(/fences/g,/ {0,3}(?:`{3,}|~{3,})/).replace(/blockquote/g,/ {0,3}>/).replace(/heading/g,/ {0,3}#{1,6}/).replace(/html/g,/ {0,3}<[^\n>]+>\n/).replace(/\|table/g,"").getRegex(),u8e=nn(Vj).replace(/bull/g,xD).replace(/blockCode/g,/(?: {4}| {0,3}\t)/).replace(/fences/g,/ {0,3}(?:`{3,}|~{3,})/).replace(/blockquote/g,/ {0,3}>/).replace(/heading/g,/ {0,3}#{1,6}/).replace(/html/g,/ {0,3}<[^\n>]+>\n/).replace(/table/g,/ {0,3}\|?(?:[:\- ]*\|)+[\:\- ]*\n/).getRegex(),bD=/^([^\n]+(?:\n(?!hr|heading|lheading|blockquote|fences|list|html|table| +\n)[^\n]+)*)/,h8e=/^[^\n]+/,wD=/(?!\s*\])(?:\\.|[^\[\]\\])+/,f8e=nn(/^ {0,3}\[(label)\]: *(?:\n[ \t]*)?([^<\s][^\s]*|<.*?>)(?:(?: +(?:\n[ \t]*)?| *\n[ \t]*)(title))? *(?:\n+|$)/).replace("label",wD).replace("title",/(?:"(?:\\"?|[^"\\])*"|'[^'\n]*(?:\n[^'\n]+)*\n?'|\([^()]*\))/).getRegex(),d8e=nn(/^( {0,3}bull)([ \t][^\n]+?)?(?:\n|$)/).replace(/bull/g,xD).getRegex(),Aw="address|article|aside|base|basefont|blockquote|body|caption|center|col|colgroup|dd|details|dialog|dir|div|dl|dt|fieldset|figcaption|figure|footer|form|frame|frameset|h[1-6]|head|header|hr|html|iframe|legend|li|link|main|menu|menuitem|meta|nav|noframes|ol|optgroup|option|p|param|search|section|summary|table|tbody|td|tfoot|th|thead|title|tr|track|ul",TD=/|$))/,p8e=nn("^ {0,3}(?:<(script|pre|style|textarea)[\\s>][\\s\\S]*?(?:[^\\n]*\\n+|$)|comment[^\\n]*(\\n+|$)|<\\?[\\s\\S]*?(?:\\?>\\n*|$)|\\n*|$)|\\n*|$)|)[\\s\\S]*?(?:(?:\\n[ ]*)+\\n|$)|<(?!script|pre|style|textarea)([a-z][\\w-]*)(?:attribute)*? */?>(?=[ \\t]*(?:\\n|$))[\\s\\S]*?(?:(?:\\n[ ]*)+\\n|$)|(?=[ \\t]*(?:\\n|$))[\\s\\S]*?(?:(?:\\n[ ]*)+\\n|$))","i").replace("comment",TD).replace("tag",Aw).replace("attribute",/ +[a-zA-Z:_][\w.:-]*(?: *= *"[^"\n]*"| *= *'[^'\n]*'| *= *[^\s"'=<>`]+)?/).getRegex(),Hj=nn(bD).replace("hr",m2).replace("heading"," {0,3}#{1,6}(?:\\s|$)").replace("|lheading","").replace("|table","").replace("blockquote"," {0,3}>").replace("fences"," {0,3}(?:`{3,}(?=[^`\\n]*\\n)|~{3,})[^\\n]*\\n").replace("list"," {0,3}(?:[*+-]|1[.)]) ").replace("html",")|<(?:script|pre|style|textarea|!--)").replace("tag",Aw).getRegex(),m8e=nn(/^( {0,3}> ?(paragraph|[^\n]*)(?:\n|$))+/).replace("paragraph",Hj).getRegex(),kD={blockquote:m8e,code:o8e,def:f8e,fences:l8e,heading:c8e,hr:m2,html:p8e,lheading:Uj,list:d8e,newline:s8e,paragraph:Hj,table:d2,text:h8e},Pj=nn("^ *([^\\n ].*)\\n {0,3}((?:\\| *)?:?-+:? *(?:\\| *:?-+:? *)*(?:\\| *)?)(?:\\n((?:(?! *\\n|hr|heading|blockquote|code|fences|list|html).*(?:\\n|$))*)\\n*|$)").replace("hr",m2).replace("heading"," {0,3}#{1,6}(?:\\s|$)").replace("blockquote"," {0,3}>").replace("code","(?: {4}| {0,3} )[^\\n]").replace("fences"," {0,3}(?:`{3,}(?=[^`\\n]*\\n)|~{3,})[^\\n]*\\n").replace("list"," {0,3}(?:[*+-]|1[.)]) ").replace("html",")|<(?:script|pre|style|textarea|!--)").replace("tag",Aw).getRegex(),g8e={...kD,lheading:u8e,table:Pj,paragraph:nn(bD).replace("hr",m2).replace("heading"," {0,3}#{1,6}(?:\\s|$)").replace("|lheading","").replace("table",Pj).replace("blockquote"," {0,3}>").replace("fences"," {0,3}(?:`{3,}(?=[^`\\n]*\\n)|~{3,})[^\\n]*\\n").replace("list"," {0,3}(?:[*+-]|1[.)]) ").replace("html",")|<(?:script|pre|style|textarea|!--)").replace("tag",Aw).getRegex()},y8e={...kD,html:nn(`^ *(?:comment *(?:\\n|\\s*$)|<(tag)[\\s\\S]+? *(?:\\n{2,}|\\s*$)|\\s]*)*?/?> *(?:\\n{2,}|\\s*$))`).replace("comment",TD).replace(/tag/g,"(?!(?:a|em|strong|small|s|cite|q|dfn|abbr|data|time|code|var|samp|kbd|sub|sup|i|b|u|mark|ruby|rt|rp|bdi|bdo|span|br|wbr|ins|del|img)\\b)\\w+(?!:|[^\\w\\s@]*@)\\b").getRegex(),def:/^ *\[([^\]]+)\]: *]+)>?(?: +(["(][^\n]+[")]))? *(?:\n+|$)/,heading:/^(#{1,6})(.*)(?:\n+|$)/,fences:d2,lheading:/^(.+?)\n {0,3}(=+|-+) *(?:\n+|$)/,paragraph:nn(bD).replace("hr",m2).replace("heading",` *#{1,6} *[^ +]`).replace("lheading",Uj).replace("|table","").replace("blockquote"," {0,3}>").replace("|fences","").replace("|list","").replace("|html","").replace("|tag","").getRegex()},v8e=/^\\([!"#$%&'()*+,\-./:;<=>?@\[\]\\^_`{|}~])/,x8e=/^(`+)([^`]|[^`][\s\S]*?[^`])\1(?!`)/,Wj=/^( {2,}|\\)\n(?!\s*$)/,b8e=/^(`+|[^`])(?:(?= {2,}\n)|[\s\S]*?(?:(?=[\\]*?>/g,Xj=/^(?:\*+(?:((?!\*)punct)|[^\s*]))|^_+(?:((?!_)punct)|([^\s_]))/,S8e=nn(Xj,"u").replace(/punct/g,_w).getRegex(),C8e=nn(Xj,"u").replace(/punct/g,Yj).getRegex(),jj="^[^_*]*?__[^_*]*?\\*[^_*]*?(?=__)|[^*]+(?=[^*])|(?!\\*)punct(\\*+)(?=[\\s]|$)|notPunctSpace(\\*+)(?!\\*)(?=punctSpace|$)|(?!\\*)punctSpace(\\*+)(?=notPunctSpace)|[\\s](\\*+)(?!\\*)(?=punct)|(?!\\*)punct(\\*+)(?!\\*)(?=punct)|notPunctSpace(\\*+)(?=notPunctSpace)",A8e=nn(jj,"gu").replace(/notPunctSpace/g,qj).replace(/punctSpace/g,ED).replace(/punct/g,_w).getRegex(),_8e=nn(jj,"gu").replace(/notPunctSpace/g,k8e).replace(/punctSpace/g,T8e).replace(/punct/g,Yj).getRegex(),D8e=nn("^[^_*]*?\\*\\*[^_*]*?_[^_*]*?(?=\\*\\*)|[^_]+(?=[^_])|(?!_)punct(_+)(?=[\\s]|$)|notPunctSpace(_+)(?!_)(?=punctSpace|$)|(?!_)punctSpace(_+)(?=notPunctSpace)|[\\s](_+)(?!_)(?=punct)|(?!_)punct(_+)(?!_)(?=punct)","gu").replace(/notPunctSpace/g,qj).replace(/punctSpace/g,ED).replace(/punct/g,_w).getRegex(),L8e=nn(/\\(punct)/,"gu").replace(/punct/g,_w).getRegex(),R8e=nn(/^<(scheme:[^\s\x00-\x1f<>]*|email)>/).replace("scheme",/[a-zA-Z][a-zA-Z0-9+.-]{1,31}/).replace("email",/[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+(@)[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)+(?![-_])/).getRegex(),N8e=nn(TD).replace("(?:-->|$)","-->").getRegex(),M8e=nn("^comment|^|^<[a-zA-Z][\\w-]*(?:attribute)*?\\s*/?>|^<\\?[\\s\\S]*?\\?>|^|^").replace("comment",N8e).replace("attribute",/\s+[a-zA-Z:_][\w.:-]*(?:\s*=\s*"[^"]*"|\s*=\s*'[^']*'|\s*=\s*[^\s"'=<>`]+)?/).getRegex(),Cw=/(?:\[(?:\\.|[^\[\]\\])*\]|\\.|`[^`]*`|[^\[\]\\`])*?/,I8e=nn(/^!?\[(label)\]\(\s*(href)(?:\s+(title))?\s*\)/).replace("label",Cw).replace("href",/<(?:\\.|[^\n<>\\])+>|[^\s\x00-\x1f]*/).replace("title",/"(?:\\"?|[^"\\])*"|'(?:\\'?|[^'\\])*'|\((?:\\\)?|[^)\\])*\)/).getRegex(),Kj=nn(/^!?\[(label)\]\[(ref)\]/).replace("label",Cw).replace("ref",wD).getRegex(),Qj=nn(/^!?\[(ref)\](?:\[\])?/).replace("ref",wD).getRegex(),O8e=nn("reflink|nolink(?!\\()","g").replace("reflink",Kj).replace("nolink",Qj).getRegex(),SD={_backpedal:d2,anyPunctuation:L8e,autolink:R8e,blockSkip:E8e,br:Wj,code:x8e,del:d2,emStrongLDelim:S8e,emStrongRDelimAst:A8e,emStrongRDelimUnd:D8e,escape:v8e,link:I8e,nolink:Qj,punctuation:w8e,reflink:Kj,reflinkSearch:O8e,tag:M8e,text:b8e,url:d2},P8e={...SD,link:nn(/^!?\[(label)\]\((.*?)\)/).replace("label",Cw).getRegex(),reflink:nn(/^!?\[(label)\]\s*\[([^\]]*)\]/).replace("label",Cw).getRegex()},gD={...SD,emStrongRDelimAst:_8e,emStrongLDelim:C8e,url:nn(/^((?:ftp|https?):\/\/|www\.)(?:[a-zA-Z0-9\-]+\.?)+[^\s<]*|^email/,"i").replace("email",/[A-Za-z0-9._+-]+(@)[a-zA-Z0-9-_]+(?:\.[a-zA-Z0-9-_]*[a-zA-Z0-9])+(?![-_])/).getRegex(),_backpedal:/(?:[^?!.,:;*_'"~()&]+|\([^)]*\)|&(?![a-zA-Z0-9]+;$)|[?!.,:;*_'"~)]+(?!$))+/,del:/^(~~?)(?=[^\s~])((?:\\.|[^\\])*?(?:\\.|[^\s~\\]))\1(?=[^~]|$)/,text:/^([`~]+|[^`~])(?:(?= {2,}\n)|(?=[a-zA-Z0-9.!#$%&'*+\/=?_`{\|}~-]+@)|[\s\S]*?(?:(?=[\\":">",'"':""","'":"'"},Bj=o(t=>F8e[t],"getEscapeReplacement");o(pc,"escape");o(Fj,"cleanUrl");o($j,"splitCells");o(f2,"rtrim");o($8e,"findClosingBracket");o(zj,"outputLink");o(z8e,"indentCodeCompensation");hm=class{static{o(this,"_Tokenizer")}options;rules;lexer;constructor(e){this.options=e||Id}space(e){let r=this.rules.block.newline.exec(e);if(r&&r[0].length>0)return{type:"space",raw:r[0]}}code(e){let r=this.rules.block.code.exec(e);if(r){let n=r[0].replace(this.rules.other.codeRemoveIndent,"");return{type:"code",raw:r[0],codeBlockStyle:"indented",text:this.options.pedantic?n:f2(n,` +`)}}}fences(e){let r=this.rules.block.fences.exec(e);if(r){let n=r[0],i=z8e(n,r[3]||"",this.rules);return{type:"code",raw:n,lang:r[2]?r[2].trim().replace(this.rules.inline.anyPunctuation,"$1"):r[2],text:i}}}heading(e){let r=this.rules.block.heading.exec(e);if(r){let n=r[2].trim();if(this.rules.other.endingHash.test(n)){let i=f2(n,"#");(this.options.pedantic||!i||this.rules.other.endingSpaceChar.test(i))&&(n=i.trim())}return{type:"heading",raw:r[0],depth:r[1].length,text:n,tokens:this.lexer.inline(n)}}}hr(e){let r=this.rules.block.hr.exec(e);if(r)return{type:"hr",raw:f2(r[0],` +`)}}blockquote(e){let r=this.rules.block.blockquote.exec(e);if(r){let n=f2(r[0],` +`).split(` +`),i="",a="",s=[];for(;n.length>0;){let l=!1,u=[],h;for(h=0;h1,a={type:"list",raw:"",ordered:i,start:i?+n.slice(0,-1):"",loose:!1,items:[]};n=i?`\\d{1,9}\\${n.slice(-1)}`:`\\${n}`,this.options.pedantic&&(n=i?n:"[*+-]");let s=this.rules.other.listItemRegex(n),l=!1;for(;e;){let h=!1,f="",d="";if(!(r=s.exec(e))||this.rules.block.hr.test(e))break;f=r[0],e=e.substring(f.length);let p=r[2].split(` +`,1)[0].replace(this.rules.other.listReplaceTabs,b=>" ".repeat(3*b.length)),m=e.split(` +`,1)[0],g=!p.trim(),y=0;if(this.options.pedantic?(y=2,d=p.trimStart()):g?y=r[1].length+1:(y=r[2].search(this.rules.other.nonSpaceChar),y=y>4?1:y,d=p.slice(y),y+=r[1].length),g&&this.rules.other.blankLine.test(m)&&(f+=m+` +`,e=e.substring(m.length+1),h=!0),!h){let b=this.rules.other.nextBulletRegex(y),w=this.rules.other.hrRegex(y),C=this.rules.other.fencesBeginRegex(y),T=this.rules.other.headingBeginRegex(y),E=this.rules.other.htmlBeginRegex(y);for(;e;){let A=e.split(` +`,1)[0],S;if(m=A,this.options.pedantic?(m=m.replace(this.rules.other.listReplaceNesting," "),S=m):S=m.replace(this.rules.other.tabCharGlobal," "),C.test(m)||T.test(m)||E.test(m)||b.test(m)||w.test(m))break;if(S.search(this.rules.other.nonSpaceChar)>=y||!m.trim())d+=` +`+S.slice(y);else{if(g||p.replace(this.rules.other.tabCharGlobal," ").search(this.rules.other.nonSpaceChar)>=4||C.test(p)||T.test(p)||w.test(p))break;d+=` +`+m}!g&&!m.trim()&&(g=!0),f+=A+` +`,e=e.substring(A.length+1),p=S.slice(y)}}a.loose||(l?a.loose=!0:this.rules.other.doubleBlankLine.test(f)&&(l=!0));let v=null,x;this.options.gfm&&(v=this.rules.other.listIsTask.exec(d),v&&(x=v[0]!=="[ ] ",d=d.replace(this.rules.other.listReplaceTask,""))),a.items.push({type:"list_item",raw:f,task:!!v,checked:x,loose:!1,text:d,tokens:[]}),a.raw+=f}let u=a.items.at(-1);if(u)u.raw=u.raw.trimEnd(),u.text=u.text.trimEnd();else return;a.raw=a.raw.trimEnd();for(let h=0;hp.type==="space"),d=f.length>0&&f.some(p=>this.rules.other.anyLine.test(p.raw));a.loose=d}if(a.loose)for(let h=0;h({text:u,tokens:this.lexer.inline(u),header:!1,align:s.align[h]})));return s}}lheading(e){let r=this.rules.block.lheading.exec(e);if(r)return{type:"heading",raw:r[0],depth:r[2].charAt(0)==="="?1:2,text:r[1],tokens:this.lexer.inline(r[1])}}paragraph(e){let r=this.rules.block.paragraph.exec(e);if(r){let n=r[1].charAt(r[1].length-1)===` +`?r[1].slice(0,-1):r[1];return{type:"paragraph",raw:r[0],text:n,tokens:this.lexer.inline(n)}}}text(e){let r=this.rules.block.text.exec(e);if(r)return{type:"text",raw:r[0],text:r[0],tokens:this.lexer.inline(r[0])}}escape(e){let r=this.rules.inline.escape.exec(e);if(r)return{type:"escape",raw:r[0],text:r[1]}}tag(e){let r=this.rules.inline.tag.exec(e);if(r)return!this.lexer.state.inLink&&this.rules.other.startATag.test(r[0])?this.lexer.state.inLink=!0:this.lexer.state.inLink&&this.rules.other.endATag.test(r[0])&&(this.lexer.state.inLink=!1),!this.lexer.state.inRawBlock&&this.rules.other.startPreScriptTag.test(r[0])?this.lexer.state.inRawBlock=!0:this.lexer.state.inRawBlock&&this.rules.other.endPreScriptTag.test(r[0])&&(this.lexer.state.inRawBlock=!1),{type:"html",raw:r[0],inLink:this.lexer.state.inLink,inRawBlock:this.lexer.state.inRawBlock,block:!1,text:r[0]}}link(e){let r=this.rules.inline.link.exec(e);if(r){let n=r[2].trim();if(!this.options.pedantic&&this.rules.other.startAngleBracket.test(n)){if(!this.rules.other.endAngleBracket.test(n))return;let s=f2(n.slice(0,-1),"\\");if((n.length-s.length)%2===0)return}else{let s=$8e(r[2],"()");if(s>-1){let u=(r[0].indexOf("!")===0?5:4)+r[1].length+s;r[2]=r[2].substring(0,s),r[0]=r[0].substring(0,u).trim(),r[3]=""}}let i=r[2],a="";if(this.options.pedantic){let s=this.rules.other.pedanticHrefTitle.exec(i);s&&(i=s[1],a=s[3])}else a=r[3]?r[3].slice(1,-1):"";return i=i.trim(),this.rules.other.startAngleBracket.test(i)&&(this.options.pedantic&&!this.rules.other.endAngleBracket.test(n)?i=i.slice(1):i=i.slice(1,-1)),zj(r,{href:i&&i.replace(this.rules.inline.anyPunctuation,"$1"),title:a&&a.replace(this.rules.inline.anyPunctuation,"$1")},r[0],this.lexer,this.rules)}}reflink(e,r){let n;if((n=this.rules.inline.reflink.exec(e))||(n=this.rules.inline.nolink.exec(e))){let i=(n[2]||n[1]).replace(this.rules.other.multipleSpaceGlobal," "),a=r[i.toLowerCase()];if(!a){let s=n[0].charAt(0);return{type:"text",raw:s,text:s}}return zj(n,a,n[0],this.lexer,this.rules)}}emStrong(e,r,n=""){let i=this.rules.inline.emStrongLDelim.exec(e);if(!i||i[3]&&n.match(this.rules.other.unicodeAlphaNumeric))return;if(!(i[1]||i[2]||"")||!n||this.rules.inline.punctuation.exec(n)){let s=[...i[0]].length-1,l,u,h=s,f=0,d=i[0][0]==="*"?this.rules.inline.emStrongRDelimAst:this.rules.inline.emStrongRDelimUnd;for(d.lastIndex=0,r=r.slice(-1*e.length+s);(i=d.exec(r))!=null;){if(l=i[1]||i[2]||i[3]||i[4]||i[5]||i[6],!l)continue;if(u=[...l].length,i[3]||i[4]){h+=u;continue}else if((i[5]||i[6])&&s%3&&!((s+u)%3)){f+=u;continue}if(h-=u,h>0)continue;u=Math.min(u,u+h+f);let p=[...i[0]][0].length,m=e.slice(0,s+i.index+p+u);if(Math.min(s,u)%2){let y=m.slice(1,-1);return{type:"em",raw:m,text:y,tokens:this.lexer.inlineTokens(y)}}let g=m.slice(2,-2);return{type:"strong",raw:m,text:g,tokens:this.lexer.inlineTokens(g)}}}}codespan(e){let r=this.rules.inline.code.exec(e);if(r){let n=r[2].replace(this.rules.other.newLineCharGlobal," "),i=this.rules.other.nonSpaceChar.test(n),a=this.rules.other.startingSpaceChar.test(n)&&this.rules.other.endingSpaceChar.test(n);return i&&a&&(n=n.substring(1,n.length-1)),{type:"codespan",raw:r[0],text:n}}}br(e){let r=this.rules.inline.br.exec(e);if(r)return{type:"br",raw:r[0]}}del(e){let r=this.rules.inline.del.exec(e);if(r)return{type:"del",raw:r[0],text:r[2],tokens:this.lexer.inlineTokens(r[2])}}autolink(e){let r=this.rules.inline.autolink.exec(e);if(r){let n,i;return r[2]==="@"?(n=r[1],i="mailto:"+n):(n=r[1],i=n),{type:"link",raw:r[0],text:n,href:i,tokens:[{type:"text",raw:n,text:n}]}}}url(e){let r;if(r=this.rules.inline.url.exec(e)){let n,i;if(r[2]==="@")n=r[0],i="mailto:"+n;else{let a;do a=r[0],r[0]=this.rules.inline._backpedal.exec(r[0])?.[0]??"";while(a!==r[0]);n=r[0],r[1]==="www."?i="http://"+r[0]:i=r[0]}return{type:"link",raw:r[0],text:n,href:i,tokens:[{type:"text",raw:n,text:n}]}}}inlineText(e){let r=this.rules.inline.text.exec(e);if(r){let n=this.lexer.state.inRawBlock;return{type:"text",raw:r[0],text:r[0],escaped:n}}}},Al=class t{static{o(this,"_Lexer")}tokens;options;state;tokenizer;inlineQueue;constructor(e){this.tokens=[],this.tokens.links=Object.create(null),this.options=e||Id,this.options.tokenizer=this.options.tokenizer||new hm,this.tokenizer=this.options.tokenizer,this.tokenizer.options=this.options,this.tokenizer.lexer=this,this.inlineQueue=[],this.state={inLink:!1,inRawBlock:!1,top:!0};let r={other:ts,block:Sw.normal,inline:h2.normal};this.options.pedantic?(r.block=Sw.pedantic,r.inline=h2.pedantic):this.options.gfm&&(r.block=Sw.gfm,this.options.breaks?r.inline=h2.breaks:r.inline=h2.gfm),this.tokenizer.rules=r}static get rules(){return{block:Sw,inline:h2}}static lex(e,r){return new t(r).lex(e)}static lexInline(e,r){return new t(r).inlineTokens(e)}lex(e){e=e.replace(ts.carriageReturn,` +`),this.blockTokens(e,this.tokens);for(let r=0;r(i=s.call({lexer:this},e,r))?(e=e.substring(i.raw.length),r.push(i),!0):!1))continue;if(i=this.tokenizer.space(e)){e=e.substring(i.raw.length);let s=r.at(-1);i.raw.length===1&&s!==void 0?s.raw+=` +`:r.push(i);continue}if(i=this.tokenizer.code(e)){e=e.substring(i.raw.length);let s=r.at(-1);s?.type==="paragraph"||s?.type==="text"?(s.raw+=` +`+i.raw,s.text+=` +`+i.text,this.inlineQueue.at(-1).src=s.text):r.push(i);continue}if(i=this.tokenizer.fences(e)){e=e.substring(i.raw.length),r.push(i);continue}if(i=this.tokenizer.heading(e)){e=e.substring(i.raw.length),r.push(i);continue}if(i=this.tokenizer.hr(e)){e=e.substring(i.raw.length),r.push(i);continue}if(i=this.tokenizer.blockquote(e)){e=e.substring(i.raw.length),r.push(i);continue}if(i=this.tokenizer.list(e)){e=e.substring(i.raw.length),r.push(i);continue}if(i=this.tokenizer.html(e)){e=e.substring(i.raw.length),r.push(i);continue}if(i=this.tokenizer.def(e)){e=e.substring(i.raw.length);let s=r.at(-1);s?.type==="paragraph"||s?.type==="text"?(s.raw+=` +`+i.raw,s.text+=` +`+i.raw,this.inlineQueue.at(-1).src=s.text):this.tokens.links[i.tag]||(this.tokens.links[i.tag]={href:i.href,title:i.title});continue}if(i=this.tokenizer.table(e)){e=e.substring(i.raw.length),r.push(i);continue}if(i=this.tokenizer.lheading(e)){e=e.substring(i.raw.length),r.push(i);continue}let a=e;if(this.options.extensions?.startBlock){let s=1/0,l=e.slice(1),u;this.options.extensions.startBlock.forEach(h=>{u=h.call({lexer:this},l),typeof u=="number"&&u>=0&&(s=Math.min(s,u))}),s<1/0&&s>=0&&(a=e.substring(0,s+1))}if(this.state.top&&(i=this.tokenizer.paragraph(a))){let s=r.at(-1);n&&s?.type==="paragraph"?(s.raw+=` +`+i.raw,s.text+=` +`+i.text,this.inlineQueue.pop(),this.inlineQueue.at(-1).src=s.text):r.push(i),n=a.length!==e.length,e=e.substring(i.raw.length);continue}if(i=this.tokenizer.text(e)){e=e.substring(i.raw.length);let s=r.at(-1);s?.type==="text"?(s.raw+=` +`+i.raw,s.text+=` +`+i.text,this.inlineQueue.pop(),this.inlineQueue.at(-1).src=s.text):r.push(i);continue}if(e){let s="Infinite loop on byte: "+e.charCodeAt(0);if(this.options.silent){console.error(s);break}else throw new Error(s)}}return this.state.top=!0,r}inline(e,r=[]){return this.inlineQueue.push({src:e,tokens:r}),r}inlineTokens(e,r=[]){let n=e,i=null;if(this.tokens.links){let l=Object.keys(this.tokens.links);if(l.length>0)for(;(i=this.tokenizer.rules.inline.reflinkSearch.exec(n))!=null;)l.includes(i[0].slice(i[0].lastIndexOf("[")+1,-1))&&(n=n.slice(0,i.index)+"["+"a".repeat(i[0].length-2)+"]"+n.slice(this.tokenizer.rules.inline.reflinkSearch.lastIndex))}for(;(i=this.tokenizer.rules.inline.blockSkip.exec(n))!=null;)n=n.slice(0,i.index)+"["+"a".repeat(i[0].length-2)+"]"+n.slice(this.tokenizer.rules.inline.blockSkip.lastIndex);for(;(i=this.tokenizer.rules.inline.anyPunctuation.exec(n))!=null;)n=n.slice(0,i.index)+"++"+n.slice(this.tokenizer.rules.inline.anyPunctuation.lastIndex);let a=!1,s="";for(;e;){a||(s=""),a=!1;let l;if(this.options.extensions?.inline?.some(h=>(l=h.call({lexer:this},e,r))?(e=e.substring(l.raw.length),r.push(l),!0):!1))continue;if(l=this.tokenizer.escape(e)){e=e.substring(l.raw.length),r.push(l);continue}if(l=this.tokenizer.tag(e)){e=e.substring(l.raw.length),r.push(l);continue}if(l=this.tokenizer.link(e)){e=e.substring(l.raw.length),r.push(l);continue}if(l=this.tokenizer.reflink(e,this.tokens.links)){e=e.substring(l.raw.length);let h=r.at(-1);l.type==="text"&&h?.type==="text"?(h.raw+=l.raw,h.text+=l.text):r.push(l);continue}if(l=this.tokenizer.emStrong(e,n,s)){e=e.substring(l.raw.length),r.push(l);continue}if(l=this.tokenizer.codespan(e)){e=e.substring(l.raw.length),r.push(l);continue}if(l=this.tokenizer.br(e)){e=e.substring(l.raw.length),r.push(l);continue}if(l=this.tokenizer.del(e)){e=e.substring(l.raw.length),r.push(l);continue}if(l=this.tokenizer.autolink(e)){e=e.substring(l.raw.length),r.push(l);continue}if(!this.state.inLink&&(l=this.tokenizer.url(e))){e=e.substring(l.raw.length),r.push(l);continue}let u=e;if(this.options.extensions?.startInline){let h=1/0,f=e.slice(1),d;this.options.extensions.startInline.forEach(p=>{d=p.call({lexer:this},f),typeof d=="number"&&d>=0&&(h=Math.min(h,d))}),h<1/0&&h>=0&&(u=e.substring(0,h+1))}if(l=this.tokenizer.inlineText(u)){e=e.substring(l.raw.length),l.raw.slice(-1)!=="_"&&(s=l.raw.slice(-1)),a=!0;let h=r.at(-1);h?.type==="text"?(h.raw+=l.raw,h.text+=l.text):r.push(l);continue}if(e){let h="Infinite loop on byte: "+e.charCodeAt(0);if(this.options.silent){console.error(h);break}else throw new Error(h)}}return r}},fm=class{static{o(this,"_Renderer")}options;parser;constructor(e){this.options=e||Id}space(e){return""}code({text:e,lang:r,escaped:n}){let i=(r||"").match(ts.notSpaceStart)?.[0],a=e.replace(ts.endingNewline,"")+` +`;return i?'
'+(n?a:pc(a,!0))+`
+`:"
"+(n?a:pc(a,!0))+`
+`}blockquote({tokens:e}){return`
+${this.parser.parse(e)}
+`}html({text:e}){return e}heading({tokens:e,depth:r}){return`${this.parser.parseInline(e)} +`}hr(e){return`
+`}list(e){let r=e.ordered,n=e.start,i="";for(let l=0;l +`+i+" +`}listitem(e){let r="";if(e.task){let n=this.checkbox({checked:!!e.checked});e.loose?e.tokens[0]?.type==="paragraph"?(e.tokens[0].text=n+" "+e.tokens[0].text,e.tokens[0].tokens&&e.tokens[0].tokens.length>0&&e.tokens[0].tokens[0].type==="text"&&(e.tokens[0].tokens[0].text=n+" "+pc(e.tokens[0].tokens[0].text),e.tokens[0].tokens[0].escaped=!0)):e.tokens.unshift({type:"text",raw:n+" ",text:n+" ",escaped:!0}):r+=n+" "}return r+=this.parser.parse(e.tokens,!!e.loose),`
  • ${r}
    • +`}checkbox({checked:e}){return"'}paragraph({tokens:e}){return`

    ${this.parser.parseInline(e)}

    +`}table(e){let r="",n="";for(let a=0;a${i}`),`
    + +`+r+` +`+i+`
    +`}tablerow({text:e}){return` +${e} +`}tablecell(e){let r=this.parser.parseInline(e.tokens),n=e.header?"th":"td";return(e.align?`<${n} align="${e.align}">`:`<${n}>`)+r+` +`}strong({tokens:e}){return`${this.parser.parseInline(e)}`}em({tokens:e}){return`${this.parser.parseInline(e)}`}codespan({text:e}){return`${pc(e,!0)}`}br(e){return"
    "}del({tokens:e}){return`${this.parser.parseInline(e)}`}link({href:e,title:r,tokens:n}){let i=this.parser.parseInline(n),a=Fj(e);if(a===null)return i;e=a;let s='    ",s}image({href:e,title:r,text:n}){let i=Fj(e);if(i===null)return pc(n);e=i;let a=`${n}{let l=a[s].flat(1/0);n=n.concat(this.walkTokens(l,r))}):a.tokens&&(n=n.concat(this.walkTokens(a.tokens,r)))}}return n}use(...e){let r=this.defaults.extensions||{renderers:{},childTokens:{}};return e.forEach(n=>{let i={...n};if(i.async=this.defaults.async||i.async||!1,n.extensions&&(n.extensions.forEach(a=>{if(!a.name)throw new Error("extension name required");if("renderer"in a){let s=r.renderers[a.name];s?r.renderers[a.name]=function(...l){let u=a.renderer.apply(this,l);return u===!1&&(u=s.apply(this,l)),u}:r.renderers[a.name]=a.renderer}if("tokenizer"in a){if(!a.level||a.level!=="block"&&a.level!=="inline")throw new Error("extension level must be 'block' or 'inline'");let s=r[a.level];s?s.unshift(a.tokenizer):r[a.level]=[a.tokenizer],a.start&&(a.level==="block"?r.startBlock?r.startBlock.push(a.start):r.startBlock=[a.start]:a.level==="inline"&&(r.startInline?r.startInline.push(a.start):r.startInline=[a.start]))}"childTokens"in a&&a.childTokens&&(r.childTokens[a.name]=a.childTokens)}),i.extensions=r),n.renderer){let a=this.defaults.renderer||new fm(this.defaults);for(let s in n.renderer){if(!(s in a))throw new Error(`renderer '${s}' does not exist`);if(["options","parser"].includes(s))continue;let l=s,u=n.renderer[l],h=a[l];a[l]=(...f)=>{let d=u.apply(a,f);return d===!1&&(d=h.apply(a,f)),d||""}}i.renderer=a}if(n.tokenizer){let a=this.defaults.tokenizer||new hm(this.defaults);for(let s in n.tokenizer){if(!(s in a))throw new Error(`tokenizer '${s}' does not exist`);if(["options","rules","lexer"].includes(s))continue;let l=s,u=n.tokenizer[l],h=a[l];a[l]=(...f)=>{let d=u.apply(a,f);return d===!1&&(d=h.apply(a,f)),d}}i.tokenizer=a}if(n.hooks){let a=this.defaults.hooks||new um;for(let s in n.hooks){if(!(s in a))throw new Error(`hook '${s}' does not exist`);if(["options","block"].includes(s))continue;let l=s,u=n.hooks[l],h=a[l];um.passThroughHooks.has(s)?a[l]=f=>{if(this.defaults.async)return Promise.resolve(u.call(a,f)).then(p=>h.call(a,p));let d=u.call(a,f);return h.call(a,d)}:a[l]=(...f)=>{let d=u.apply(a,f);return d===!1&&(d=h.apply(a,f)),d}}i.hooks=a}if(n.walkTokens){let a=this.defaults.walkTokens,s=n.walkTokens;i.walkTokens=function(l){let u=[];return u.push(s.call(this,l)),a&&(u=u.concat(a.call(this,l))),u}}this.defaults={...this.defaults,...i}}),this}setOptions(e){return this.defaults={...this.defaults,...e},this}lexer(e,r){return Al.lex(e,r??this.defaults)}parser(e,r){return _l.parse(e,r??this.defaults)}parseMarkdown(e){return o((n,i)=>{let a={...i},s={...this.defaults,...a},l=this.onError(!!s.silent,!!s.async);if(this.defaults.async===!0&&a.async===!1)return l(new Error("marked(): The async option was set to true by an extension. Remove async: false from the parse options object to return a Promise."));if(typeof n>"u"||n===null)return l(new Error("marked(): input parameter is undefined or null"));if(typeof n!="string")return l(new Error("marked(): input parameter is of type "+Object.prototype.toString.call(n)+", string expected"));s.hooks&&(s.hooks.options=s,s.hooks.block=e);let u=s.hooks?s.hooks.provideLexer():e?Al.lex:Al.lexInline,h=s.hooks?s.hooks.provideParser():e?_l.parse:_l.parseInline;if(s.async)return Promise.resolve(s.hooks?s.hooks.preprocess(n):n).then(f=>u(f,s)).then(f=>s.hooks?s.hooks.processAllTokens(f):f).then(f=>s.walkTokens?Promise.all(this.walkTokens(f,s.walkTokens)).then(()=>f):f).then(f=>h(f,s)).then(f=>s.hooks?s.hooks.postprocess(f):f).catch(l);try{s.hooks&&(n=s.hooks.preprocess(n));let f=u(n,s);s.hooks&&(f=s.hooks.processAllTokens(f)),s.walkTokens&&this.walkTokens(f,s.walkTokens);let d=h(f,s);return s.hooks&&(d=s.hooks.postprocess(d)),d}catch(f){return l(f)}},"parse")}onError(e,r){return n=>{if(n.message+=` +Please report this to https://github.com/markedjs/marked.`,e){let i="

    An error occurred:

    "+pc(n.message+"",!0)+"
    ";return r?Promise.resolve(i):i}if(r)return Promise.reject(n);throw n}}},Md=new yD;o(Jr,"marked");Jr.options=Jr.setOptions=function(t){return Md.setOptions(t),Jr.defaults=Md.defaults,Gj(Jr.defaults),Jr};Jr.getDefaults=vD;Jr.defaults=Id;Jr.use=function(...t){return Md.use(...t),Jr.defaults=Md.defaults,Gj(Jr.defaults),Jr};Jr.walkTokens=function(t,e){return Md.walkTokens(t,e)};Jr.parseInline=Md.parseInline;Jr.Parser=_l;Jr.parser=_l.parse;Jr.Renderer=fm;Jr.TextRenderer=p2;Jr.Lexer=Al;Jr.lexer=Al.lex;Jr.Tokenizer=hm;Jr.Hooks=um;Jr.parse=Jr;dkt=Jr.options,pkt=Jr.setOptions,mkt=Jr.use,gkt=Jr.walkTokens,ykt=Jr.parseInline,vkt=_l.parse,xkt=Al.lex});function G8e(t,{markdownAutoWrap:e}){let n=t.replace(//g,` +`).replace(/\n{2,}/g,` +`),i=B4(n);return e===!1?i.replace(/ /g," "):i}function Jj(t,e={}){let r=G8e(t,e),n=Jr.lexer(r),i=[[]],a=0;function s(l,u="normal"){l.type==="text"?l.text.split(` +`).forEach((f,d)=>{d!==0&&(a++,i.push([])),f.split(" ").forEach(p=>{p=p.replace(/'/g,"'"),p&&i[a].push({content:p,type:u})})}):l.type==="strong"||l.type==="em"?l.tokens.forEach(h=>{s(h,l.type)}):l.type==="html"&&i[a].push({content:l.text,type:"normal"})}return o(s,"processNode"),n.forEach(l=>{l.type==="paragraph"?l.tokens?.forEach(u=>{s(u)}):l.type==="html"&&i[a].push({content:l.text,type:"normal"})}),i}function eK(t,{markdownAutoWrap:e}={}){let r=Jr.lexer(t);function n(i){return i.type==="text"?e===!1?i.text.replace(/\n */g,"
    ").replace(/ /g," "):i.text.replace(/\n */g,"
    "):i.type==="strong"?`${i.tokens?.map(n).join("")}`:i.type==="em"?`${i.tokens?.map(n).join("")}`:i.type==="paragraph"?`

    ${i.tokens?.map(n).join("")}

    `:i.type==="space"?"":i.type==="html"?`${i.text}`:i.type==="escape"?i.text:`Unsupported markdown: ${i.type}`}return o(n,"output"),r.map(n).join("")}var tK=N(()=>{"use strict";Zj();PC();o(G8e,"preprocessMarkdown");o(Jj,"markdownToLines");o(eK,"markdownToHTML")});function V8e(t){return Intl.Segmenter?[...new Intl.Segmenter().segment(t)].map(e=>e.segment):[...t]}function U8e(t,e){let r=V8e(e.content);return rK(t,[],r,e.type)}function rK(t,e,r,n){if(r.length===0)return[{content:e.join(""),type:n},{content:"",type:n}];let[i,...a]=r,s=[...e,i];return t([{content:s.join(""),type:n}])?rK(t,s,a,n):(e.length===0&&i&&(e.push(i),r.shift()),[{content:e.join(""),type:n},{content:r.join(""),type:n}])}function nK(t,e){if(t.some(({content:r})=>r.includes(` +`)))throw new Error("splitLineToFitWidth does not support newlines in the line");return CD(t,e)}function CD(t,e,r=[],n=[]){if(t.length===0)return n.length>0&&r.push(n),r.length>0?r:[];let i="";t[0].content===" "&&(i=" ",t.shift());let a=t.shift()??{content:" ",type:"normal"},s=[...n];if(i!==""&&s.push({content:i,type:"normal"}),s.push(a),e(s))return CD(t,e,r,s);if(n.length>0)r.push(n),t.unshift(a);else if(a.content){let[l,u]=U8e(e,a);r.push([l]),u.content&&t.unshift(u)}return CD(t,e,r)}var iK=N(()=>{"use strict";o(V8e,"splitTextToChars");o(U8e,"splitWordToFitWidth");o(rK,"splitWordToFitWidthRecursion");o(nK,"splitLineToFitWidth");o(CD,"splitLineToFitWidthRecursion")});function aK(t,e){e&&t.attr("style",e)}async function H8e(t,e,r,n,i=!1){let a=t.append("foreignObject");a.attr("width",`${10*r}px`),a.attr("height",`${10*r}px`);let s=a.append("xhtml:div"),l=e.label;e.label&&pi(e.label)&&(l=await mh(e.label.replace(Ze.lineBreakRegex,` +`),me()));let u=e.isNode?"nodeLabel":"edgeLabel",h=s.append("span");h.html(l),aK(h,e.labelStyle),h.attr("class",`${u} ${n}`),aK(s,e.labelStyle),s.style("display","table-cell"),s.style("white-space","nowrap"),s.style("line-height","1.5"),s.style("max-width",r+"px"),s.style("text-align","center"),s.attr("xmlns","http://www.w3.org/1999/xhtml"),i&&s.attr("class","labelBkg");let f=s.node().getBoundingClientRect();return f.width===r&&(s.style("display","table"),s.style("white-space","break-spaces"),s.style("width",r+"px"),f=s.node().getBoundingClientRect()),a.node()}function AD(t,e,r){return t.append("tspan").attr("class","text-outer-tspan").attr("x",0).attr("y",e*r-.1+"em").attr("dy",r+"em")}function W8e(t,e,r){let n=t.append("text"),i=AD(n,1,e);_D(i,r);let a=i.node().getComputedTextLength();return n.remove(),a}function sK(t,e,r){let n=t.append("text"),i=AD(n,1,e);_D(i,[{content:r,type:"normal"}]);let a=i.node()?.getBoundingClientRect();return a&&n.remove(),a}function q8e(t,e,r,n=!1){let a=e.append("g"),s=a.insert("rect").attr("class","background").attr("style","stroke: none"),l=a.append("text").attr("y","-10.1"),u=0;for(let h of r){let f=o(p=>W8e(a,1.1,p)<=t,"checkWidth"),d=f(h)?[h]:nK(h,f);for(let p of d){let m=AD(l,u,1.1);_D(m,p),u++}}if(n){let h=l.node().getBBox(),f=2;return s.attr("x",h.x-f).attr("y",h.y-f).attr("width",h.width+2*f).attr("height",h.height+2*f),a.node()}else return l.node()}function _D(t,e){t.text(""),e.forEach((r,n)=>{let i=t.append("tspan").attr("font-style",r.type==="em"?"italic":"normal").attr("class","text-inner-tspan").attr("font-weight",r.type==="strong"?"bold":"normal");n===0?i.text(r.content):i.text(" "+r.content)})}function DD(t){return t.replace(/fa[bklrs]?:fa-[\w-]+/g,e=>``)}var Hn,to=N(()=>{"use strict";zt();gr();dr();vt();tK();ir();iK();o(aK,"applyStyle");o(H8e,"addHtmlSpan");o(AD,"createTspan");o(W8e,"computeWidthOfText");o(sK,"computeDimensionOfText");o(q8e,"createFormattedText");o(_D,"updateTextContentAndStyles");o(DD,"replaceIconSubstring");Hn=o(async(t,e="",{style:r="",isTitle:n=!1,classes:i="",useHtmlLabels:a=!0,isNode:s=!0,width:l=200,addSvgBackground:u=!1}={},h)=>{if(Y.debug("XYZ createText",e,r,n,i,a,s,"addSvgBackground: ",u),a){let f=eK(e,h),d=DD(na(f)),p=e.replace(/\\\\/g,"\\"),m={isNode:s,label:pi(e)?p:d,labelStyle:r.replace("fill:","color:")};return await H8e(t,m,l,i,u)}else{let f=e.replace(//g,"
    "),d=Jj(f.replace("
    ","
    "),h),p=q8e(l,t,d,e?u:!1);if(s){/stroke:/.exec(r)&&(r=r.replace("stroke:","lineColor:"));let m=r.replace(/stroke:[^;]+;?/g,"").replace(/stroke-width:[^;]+;?/g,"").replace(/fill:[^;]+;?/g,"").replace(/color:/g,"fill:");Ge(p).attr("style",m)}else{let m=r.replace(/stroke:[^;]+;?/g,"").replace(/stroke-width:[^;]+;?/g,"").replace(/fill:[^;]+;?/g,"").replace(/background:/g,"fill:");Ge(p).select("rect").attr("style",m.replace(/background:/g,"fill:"));let g=r.replace(/stroke:[^;]+;?/g,"").replace(/stroke-width:[^;]+;?/g,"").replace(/fill:[^;]+;?/g,"").replace(/color:/g,"fill:");Ge(p).select("text").attr("style",g)}return p}},"createText")});function Xt(t){let e=t.map((r,n)=>`${n===0?"M":"L"}${r.x},${r.y}`);return e.push("Z"),e.join(" ")}function Fo(t,e,r,n,i,a){let s=[],u=r-t,h=n-e,f=u/a,d=2*Math.PI/f,p=e+h/2;for(let m=0;m<=50;m++){let g=m/50,y=t+g*u,v=p+i*Math.sin(d*(y-t));s.push({x:y,y:v})}return s}function Lw(t,e,r,n,i,a){let s=[],l=i*Math.PI/180,f=(a*Math.PI/180-l)/(n-1);for(let d=0;d{"use strict";to();zt();dr();Ya();gr();ir();pt=o(async(t,e,r)=>{let n,i=e.useHtmlLabels||fr(me()?.htmlLabels);r?n=r:n="node default";let a=t.insert("g").attr("class",n).attr("id",e.domId||e.id),s=a.insert("g").attr("class","label").attr("style",$n(e.labelStyle)),l;e.label===void 0?l="":l=typeof e.label=="string"?e.label:e.label[0];let u=await Hn(s,Tr(na(l),me()),{useHtmlLabels:i,width:e.width||me().flowchart?.wrappingWidth,cssClasses:"markdown-node-label",style:e.labelStyle,addSvgBackground:!!e.icon||!!e.img}),h=u.getBBox(),f=(e?.padding??0)/2;if(i){let d=u.children[0],p=Ge(u),m=d.getElementsByTagName("img");if(m){let g=l.replace(/]*>/g,"").trim()==="";await Promise.all([...m].map(y=>new Promise(v=>{function x(){if(y.style.display="flex",y.style.flexDirection="column",g){let b=me().fontSize?me().fontSize:window.getComputedStyle(document.body).fontSize,w=5,[C=or.fontSize]=Bo(b),T=C*w+"px";y.style.minWidth=T,y.style.maxWidth=T}else y.style.width="100%";v(y)}o(x,"setupImage"),setTimeout(()=>{y.complete&&x()}),y.addEventListener("error",x),y.addEventListener("load",x)})))}h=d.getBoundingClientRect(),p.attr("width",h.width),p.attr("height",h.height)}return i?s.attr("transform","translate("+-h.width/2+", "+-h.height/2+")"):s.attr("transform","translate(0, "+-h.height/2+")"),e.centerLabel&&s.attr("transform","translate("+-h.width/2+", "+-h.height/2+")"),s.insert("rect",":first-child"),{shapeSvg:a,bbox:h,halfPadding:f,label:s}},"labelHelper"),Dw=o(async(t,e,r)=>{let n=r.useHtmlLabels||fr(me()?.flowchart?.htmlLabels),i=t.insert("g").attr("class","label").attr("style",r.labelStyle||""),a=await Hn(i,Tr(na(e),me()),{useHtmlLabels:n,width:r.width||me()?.flowchart?.wrappingWidth,style:r.labelStyle,addSvgBackground:!!r.icon||!!r.img}),s=a.getBBox(),l=r.padding/2;if(fr(me()?.flowchart?.htmlLabels)){let u=a.children[0],h=Ge(a);s=u.getBoundingClientRect(),h.attr("width",s.width),h.attr("height",s.height)}return n?i.attr("transform","translate("+-s.width/2+", "+-s.height/2+")"):i.attr("transform","translate(0, "+-s.height/2+")"),r.centerLabel&&i.attr("transform","translate("+-s.width/2+", "+-s.height/2+")"),i.insert("rect",":first-child"),{shapeSvg:t,bbox:s,halfPadding:l,label:i}},"insertLabel"),je=o((t,e)=>{let r=e.node().getBBox();t.width=r.width,t.height=r.height},"updateNodeBounds"),ht=o((t,e)=>(t.look==="handDrawn"?"rough-node":"node")+" "+t.cssClasses+" "+(e||""),"getNodeClasses");o(Xt,"createPathFromPoints");o(Fo,"generateFullSineWavePoints");o(Lw,"generateCirclePoints")});function Y8e(t,e){return t.intersect(e)}var oK,lK=N(()=>{"use strict";o(Y8e,"intersectNode");oK=Y8e});function X8e(t,e,r,n){var i=t.x,a=t.y,s=i-n.x,l=a-n.y,u=Math.sqrt(e*e*l*l+r*r*s*s),h=Math.abs(e*r*s/u);n.x{"use strict";o(X8e,"intersectEllipse");Rw=X8e});function j8e(t,e,r){return Rw(t,e,e,r)}var cK,uK=N(()=>{"use strict";LD();o(j8e,"intersectCircle");cK=j8e});function K8e(t,e,r,n){var i,a,s,l,u,h,f,d,p,m,g,y,v,x,b;if(i=e.y-t.y,s=t.x-e.x,u=e.x*t.y-t.x*e.y,p=i*r.x+s*r.y+u,m=i*n.x+s*n.y+u,!(p!==0&&m!==0&&hK(p,m))&&(a=n.y-r.y,l=r.x-n.x,h=n.x*r.y-r.x*n.y,f=a*t.x+l*t.y+h,d=a*e.x+l*e.y+h,!(f!==0&&d!==0&&hK(f,d))&&(g=i*l-a*s,g!==0)))return y=Math.abs(g/2),v=s*h-l*u,x=v<0?(v-y)/g:(v+y)/g,v=a*u-i*h,b=v<0?(v-y)/g:(v+y)/g,{x,y:b}}function hK(t,e){return t*e>0}var fK,dK=N(()=>{"use strict";o(K8e,"intersectLine");o(hK,"sameSign");fK=K8e});function Q8e(t,e,r){let n=t.x,i=t.y,a=[],s=Number.POSITIVE_INFINITY,l=Number.POSITIVE_INFINITY;typeof e.forEach=="function"?e.forEach(function(f){s=Math.min(s,f.x),l=Math.min(l,f.y)}):(s=Math.min(s,e.x),l=Math.min(l,e.y));let u=n-t.width/2-s,h=i-t.height/2-l;for(let f=0;f1&&a.sort(function(f,d){let p=f.x-r.x,m=f.y-r.y,g=Math.sqrt(p*p+m*m),y=d.x-r.x,v=d.y-r.y,x=Math.sqrt(y*y+v*v);return g{"use strict";dK();o(Q8e,"intersectPolygon");pK=Q8e});var Z8e,Vh,RD=N(()=>{"use strict";Z8e=o((t,e)=>{var r=t.x,n=t.y,i=e.x-r,a=e.y-n,s=t.width/2,l=t.height/2,u,h;return Math.abs(a)*s>Math.abs(i)*l?(a<0&&(l=-l),u=a===0?0:l*i/a,h=l):(i<0&&(s=-s),u=s,h=i===0?0:s*a/i),{x:r+u,y:n+h}},"intersectRect"),Vh=Z8e});var Ye,Ht=N(()=>{"use strict";lK();uK();LD();mK();RD();Ye={node:oK,circle:cK,ellipse:Rw,polygon:pK,rect:Vh}});var gK,mc,J8e,ND,Qe,Ke,Ut=N(()=>{"use strict";zt();gK=o(t=>{let{handDrawnSeed:e}=me();return{fill:t,hachureAngle:120,hachureGap:4,fillWeight:2,roughness:.7,stroke:t,seed:e}},"solidStateFill"),mc=o(t=>{let e=J8e([...t.cssCompiledStyles||[],...t.cssStyles||[]]);return{stylesMap:e,stylesArray:[...e]}},"compileStyles"),J8e=o(t=>{let e=new Map;return t.forEach(r=>{let[n,i]=r.split(":");e.set(n.trim(),i?.trim())}),e},"styles2Map"),ND=o(t=>t==="color"||t==="font-size"||t==="font-family"||t==="font-weight"||t==="font-style"||t==="text-decoration"||t==="text-align"||t==="text-transform"||t==="line-height"||t==="letter-spacing"||t==="word-spacing"||t==="text-shadow"||t==="text-overflow"||t==="white-space"||t==="word-wrap"||t==="word-break"||t==="overflow-wrap"||t==="hyphens","isLabelStyle"),Qe=o(t=>{let{stylesArray:e}=mc(t),r=[],n=[],i=[],a=[];return e.forEach(s=>{let l=s[0];ND(l)?r.push(s.join(":")+" !important"):(n.push(s.join(":")+" !important"),l.includes("stroke")&&i.push(s.join(":")+" !important"),l==="fill"&&a.push(s.join(":")+" !important"))}),{labelStyles:r.join(";"),nodeStyles:n.join(";"),stylesArray:e,borderStyles:i,backgroundStyles:a}},"styles2String"),Ke=o((t,e)=>{let{themeVariables:r,handDrawnSeed:n}=me(),{nodeBorder:i,mainBkg:a}=r,{stylesMap:s}=mc(t);return Object.assign({roughness:.7,fill:s.get("fill")||a,fillStyle:"hachure",fillWeight:4,hachureGap:5.2,stroke:s.get("stroke")||i,seed:n,strokeWidth:s.get("stroke-width")?.replace("px","")||1.3,fillLineDash:[0,0]},e)},"userNodeOverrides")});function MD(t,e,r){if(t&&t.length){let[n,i]=e,a=Math.PI/180*r,s=Math.cos(a),l=Math.sin(a);for(let u of t){let[h,f]=u;u[0]=(h-n)*s-(f-i)*l+n,u[1]=(h-n)*l+(f-i)*s+i}}}function e_e(t,e){return t[0]===e[0]&&t[1]===e[1]}function t_e(t,e,r,n=1){let i=r,a=Math.max(e,.1),s=t[0]&&t[0][0]&&typeof t[0][0]=="number"?[t]:t,l=[0,0];if(i)for(let h of s)MD(h,l,i);let u=function(h,f,d){let p=[];for(let b of h){let w=[...b];e_e(w[0],w[w.length-1])||w.push([w[0][0],w[0][1]]),w.length>2&&p.push(w)}let m=[];f=Math.max(f,.1);let g=[];for(let b of p)for(let w=0;wb.yminw.ymin?1:b.xw.x?1:b.ymax===w.ymax?0:(b.ymax-w.ymax)/Math.abs(b.ymax-w.ymax)),!g.length)return m;let y=[],v=g[0].ymin,x=0;for(;y.length||g.length;){if(g.length){let b=-1;for(let w=0;wv);w++)b=w;g.splice(0,b+1).forEach(w=>{y.push({s:v,edge:w})})}if(y=y.filter(b=>!(b.edge.ymax<=v)),y.sort((b,w)=>b.edge.x===w.edge.x?0:(b.edge.x-w.edge.x)/Math.abs(b.edge.x-w.edge.x)),(d!==1||x%f==0)&&y.length>1)for(let b=0;b=y.length)break;let C=y[b].edge,T=y[w].edge;m.push([[Math.round(C.x),v],[Math.round(T.x),v]])}v+=d,y.forEach(b=>{b.edge.x=b.edge.x+d*b.edge.islope}),x++}return m}(s,a,n);if(i){for(let h of s)MD(h,l,-i);(function(h,f,d){let p=[];h.forEach(m=>p.push(...m)),MD(p,f,d)})(u,l,-i)}return u}function x2(t,e){var r;let n=e.hachureAngle+90,i=e.hachureGap;i<0&&(i=4*e.strokeWidth),i=Math.round(Math.max(i,.1));let a=1;return e.roughness>=1&&(((r=e.randomizer)===null||r===void 0?void 0:r.next())||Math.random())>.7&&(a=i),t_e(t,i,n,a||1)}function zw(t){let e=t[0],r=t[1];return Math.sqrt(Math.pow(e[0]-r[0],2)+Math.pow(e[1]-r[1],2))}function OD(t,e){return t.type===e}function jD(t){let e=[],r=function(s){let l=new Array;for(;s!=="";)if(s.match(/^([ \t\r\n,]+)/))s=s.substr(RegExp.$1.length);else if(s.match(/^([aAcChHlLmMqQsStTvVzZ])/))l[l.length]={type:r_e,text:RegExp.$1},s=s.substr(RegExp.$1.length);else{if(!s.match(/^(([-+]?[0-9]+(\.[0-9]*)?|[-+]?\.[0-9]+)([eE][-+]?[0-9]+)?)/))return[];l[l.length]={type:ID,text:`${parseFloat(RegExp.$1)}`},s=s.substr(RegExp.$1.length)}return l[l.length]={type:yK,text:""},l}(t),n="BOD",i=0,a=r[i];for(;!OD(a,yK);){let s=0,l=[];if(n==="BOD"){if(a.text!=="M"&&a.text!=="m")return jD("M0,0"+t);i++,s=Nw[a.text],n=a.text}else OD(a,ID)?s=Nw[n]:(i++,s=Nw[a.text],n=a.text);if(!(i+sf%2?h+r:h+e);a.push({key:"C",data:u}),e=u[4],r=u[5];break}case"Q":a.push({key:"Q",data:[...l]}),e=l[2],r=l[3];break;case"q":{let u=l.map((h,f)=>f%2?h+r:h+e);a.push({key:"Q",data:u}),e=u[2],r=u[3];break}case"A":a.push({key:"A",data:[...l]}),e=l[5],r=l[6];break;case"a":e+=l[5],r+=l[6],a.push({key:"A",data:[l[0],l[1],l[2],l[3],l[4],e,r]});break;case"H":a.push({key:"H",data:[...l]}),e=l[0];break;case"h":e+=l[0],a.push({key:"H",data:[e]});break;case"V":a.push({key:"V",data:[...l]}),r=l[0];break;case"v":r+=l[0],a.push({key:"V",data:[r]});break;case"S":a.push({key:"S",data:[...l]}),e=l[2],r=l[3];break;case"s":{let u=l.map((h,f)=>f%2?h+r:h+e);a.push({key:"S",data:u}),e=u[2],r=u[3];break}case"T":a.push({key:"T",data:[...l]}),e=l[0],r=l[1];break;case"t":e+=l[0],r+=l[1],a.push({key:"T",data:[e,r]});break;case"Z":case"z":a.push({key:"Z",data:[]}),e=n,r=i}return a}function CK(t){let e=[],r="",n=0,i=0,a=0,s=0,l=0,u=0;for(let{key:h,data:f}of t){switch(h){case"M":e.push({key:"M",data:[...f]}),[n,i]=f,[a,s]=f;break;case"C":e.push({key:"C",data:[...f]}),n=f[4],i=f[5],l=f[2],u=f[3];break;case"L":e.push({key:"L",data:[...f]}),[n,i]=f;break;case"H":n=f[0],e.push({key:"L",data:[n,i]});break;case"V":i=f[0],e.push({key:"L",data:[n,i]});break;case"S":{let d=0,p=0;r==="C"||r==="S"?(d=n+(n-l),p=i+(i-u)):(d=n,p=i),e.push({key:"C",data:[d,p,...f]}),l=f[0],u=f[1],n=f[2],i=f[3];break}case"T":{let[d,p]=f,m=0,g=0;r==="Q"||r==="T"?(m=n+(n-l),g=i+(i-u)):(m=n,g=i);let y=n+2*(m-n)/3,v=i+2*(g-i)/3,x=d+2*(m-d)/3,b=p+2*(g-p)/3;e.push({key:"C",data:[y,v,x,b,d,p]}),l=m,u=g,n=d,i=p;break}case"Q":{let[d,p,m,g]=f,y=n+2*(d-n)/3,v=i+2*(p-i)/3,x=m+2*(d-m)/3,b=g+2*(p-g)/3;e.push({key:"C",data:[y,v,x,b,m,g]}),l=d,u=p,n=m,i=g;break}case"A":{let d=Math.abs(f[0]),p=Math.abs(f[1]),m=f[2],g=f[3],y=f[4],v=f[5],x=f[6];d===0||p===0?(e.push({key:"C",data:[n,i,v,x,v,x]}),n=v,i=x):(n!==v||i!==x)&&(AK(n,i,v,x,d,p,m,g,y).forEach(function(b){e.push({key:"C",data:b})}),n=v,i=x);break}case"Z":e.push({key:"Z",data:[]}),n=a,i=s}r=h}return e}function g2(t,e,r){return[t*Math.cos(r)-e*Math.sin(r),t*Math.sin(r)+e*Math.cos(r)]}function AK(t,e,r,n,i,a,s,l,u,h){let f=(d=s,Math.PI*d/180);var d;let p=[],m=0,g=0,y=0,v=0;if(h)[m,g,y,v]=h;else{[t,e]=g2(t,e,-f),[r,n]=g2(r,n,-f);let L=(t-r)/2,R=(e-n)/2,O=L*L/(i*i)+R*R/(a*a);O>1&&(O=Math.sqrt(O),i*=O,a*=O);let M=i*i,B=a*a,F=M*B-M*R*R-B*L*L,P=M*R*R+B*L*L,z=(l===u?-1:1)*Math.sqrt(Math.abs(F/P));y=z*i*R/a+(t+r)/2,v=z*-a*L/i+(e+n)/2,m=Math.asin(parseFloat(((e-v)/a).toFixed(9))),g=Math.asin(parseFloat(((n-v)/a).toFixed(9))),tg&&(m-=2*Math.PI),!u&&g>m&&(g-=2*Math.PI)}let x=g-m;if(Math.abs(x)>120*Math.PI/180){let L=g,R=r,O=n;g=u&&g>m?m+120*Math.PI/180*1:m+120*Math.PI/180*-1,p=AK(r=y+i*Math.cos(g),n=v+a*Math.sin(g),R,O,i,a,s,0,u,[g,L,y,v])}x=g-m;let b=Math.cos(m),w=Math.sin(m),C=Math.cos(g),T=Math.sin(g),E=Math.tan(x/4),A=4/3*i*E,S=4/3*a*E,_=[t,e],I=[t+A*w,e-S*b],D=[r+A*T,n-S*C],k=[r,n];if(I[0]=2*_[0]-I[0],I[1]=2*_[1]-I[1],h)return[I,D,k].concat(p);{p=[I,D,k].concat(p);let L=[];for(let R=0;R2){let i=[];for(let a=0;a2*Math.PI&&(m=0,g=2*Math.PI);let y=2*Math.PI/u.curveStepCount,v=Math.min(y/2,(g-m)/2),x=kK(v,h,f,d,p,m,g,1,u);if(!u.disableMultiStroke){let b=kK(v,h,f,d,p,m,g,1.5,u);x.push(...b)}return s&&(l?x.push(...Uh(h,f,h+d*Math.cos(m),f+p*Math.sin(m),u),...Uh(h,f,h+d*Math.cos(g),f+p*Math.sin(g),u)):x.push({op:"lineTo",data:[h,f]},{op:"lineTo",data:[h+d*Math.cos(m),f+p*Math.sin(m)]})),{type:"path",ops:x}}function bK(t,e){let r=CK(SK(jD(t))),n=[],i=[0,0],a=[0,0];for(let{key:s,data:l}of r)switch(s){case"M":a=[l[0],l[1]],i=[l[0],l[1]];break;case"L":n.push(...Uh(a[0],a[1],l[0],l[1],e)),a=[l[0],l[1]];break;case"C":{let[u,h,f,d,p,m]=l;n.push(...a_e(u,h,f,d,p,m,a,e)),a=[p,m];break}case"Z":n.push(...Uh(a[0],a[1],i[0],i[1],e)),a=[i[0],i[1]]}return{type:"path",ops:n}}function PD(t,e){let r=[];for(let n of t)if(n.length){let i=e.maxRandomnessOffset||0,a=n.length;if(a>2){r.push({op:"move",data:[n[0][0]+nr(i,e),n[0][1]+nr(i,e)]});for(let s=1;s500?.4:-.0016668*u+1.233334;let f=i.maxRandomnessOffset||0;f*f*100>l&&(f=u/10);let d=f/2,p=.2+.2*LK(i),m=i.bowing*i.maxRandomnessOffset*(n-e)/200,g=i.bowing*i.maxRandomnessOffset*(t-r)/200;m=nr(m,i,h),g=nr(g,i,h);let y=[],v=o(()=>nr(d,i,h),"M"),x=o(()=>nr(f,i,h),"k"),b=i.preserveVertices;return a&&(s?y.push({op:"move",data:[t+(b?0:v()),e+(b?0:v())]}):y.push({op:"move",data:[t+(b?0:nr(f,i,h)),e+(b?0:nr(f,i,h))]})),s?y.push({op:"bcurveTo",data:[m+t+(r-t)*p+v(),g+e+(n-e)*p+v(),m+t+2*(r-t)*p+v(),g+e+2*(n-e)*p+v(),r+(b?0:v()),n+(b?0:v())]}):y.push({op:"bcurveTo",data:[m+t+(r-t)*p+x(),g+e+(n-e)*p+x(),m+t+2*(r-t)*p+x(),g+e+2*(n-e)*p+x(),r+(b?0:x()),n+(b?0:x())]}),y}function Mw(t,e,r){if(!t.length)return[];let n=[];n.push([t[0][0]+nr(e,r),t[0][1]+nr(e,r)]),n.push([t[0][0]+nr(e,r),t[0][1]+nr(e,r)]);for(let i=1;i3){let a=[],s=1-r.curveTightness;i.push({op:"move",data:[t[1][0],t[1][1]]});for(let l=1;l+21&&i.push(l)):i.push(l),i.push(t[e+3])}else{let u=t[e+0],h=t[e+1],f=t[e+2],d=t[e+3],p=Od(u,h,.5),m=Od(h,f,.5),g=Od(f,d,.5),y=Od(p,m,.5),v=Od(m,g,.5),x=Od(y,v,.5);qD([u,p,y,x],0,r,i),qD([x,v,g,d],0,r,i)}var a,s;return i}function o_e(t,e){return $w(t,0,t.length,e)}function $w(t,e,r,n,i){let a=i||[],s=t[e],l=t[r-1],u=0,h=1;for(let f=e+1;fu&&(u=d,h=f)}return Math.sqrt(u)>n?($w(t,e,h+1,n,a),$w(t,h,r,n,a)):(a.length||a.push(s),a.push(l)),a}function BD(t,e=.15,r){let n=[],i=(t.length-1)/3;for(let a=0;a0?$w(n,0,n.length,r):n}var v2,FD,$D,zD,GD,VD,Rs,UD,r_e,ID,yK,Nw,n_e,ro,pm,YD,Iw,XD,Xe,Wt=N(()=>{"use strict";o(MD,"t");o(e_e,"e");o(t_e,"s");o(x2,"n");v2=class{static{o(this,"o")}constructor(e){this.helper=e}fillPolygons(e,r){return this._fillPolygons(e,r)}_fillPolygons(e,r){let n=x2(e,r);return{type:"fillSketch",ops:this.renderLines(n,r)}}renderLines(e,r){let n=[];for(let i of e)n.push(...this.helper.doubleLineOps(i[0][0],i[0][1],i[1][0],i[1][1],r));return n}};o(zw,"a");FD=class extends v2{static{o(this,"h")}fillPolygons(e,r){let n=r.hachureGap;n<0&&(n=4*r.strokeWidth),n=Math.max(n,.1);let i=x2(e,Object.assign({},r,{hachureGap:n})),a=Math.PI/180*r.hachureAngle,s=[],l=.5*n*Math.cos(a),u=.5*n*Math.sin(a);for(let[h,f]of i)zw([h,f])&&s.push([[h[0]-l,h[1]+u],[...f]],[[h[0]+l,h[1]-u],[...f]]);return{type:"fillSketch",ops:this.renderLines(s,r)}}},$D=class extends v2{static{o(this,"r")}fillPolygons(e,r){let n=this._fillPolygons(e,r),i=Object.assign({},r,{hachureAngle:r.hachureAngle+90}),a=this._fillPolygons(e,i);return n.ops=n.ops.concat(a.ops),n}},zD=class{static{o(this,"i")}constructor(e){this.helper=e}fillPolygons(e,r){let n=x2(e,r=Object.assign({},r,{hachureAngle:0}));return this.dotsOnLines(n,r)}dotsOnLines(e,r){let n=[],i=r.hachureGap;i<0&&(i=4*r.strokeWidth),i=Math.max(i,.1);let a=r.fillWeight;a<0&&(a=r.strokeWidth/2);let s=i/4;for(let l of e){let u=zw(l),h=u/i,f=Math.ceil(h)-1,d=u-f*i,p=(l[0][0]+l[1][0])/2-i/4,m=Math.min(l[0][1],l[1][1]);for(let g=0;g{let l=zw(s),u=Math.floor(l/(n+i)),h=(l+i-u*(n+i))/2,f=s[0],d=s[1];f[0]>d[0]&&(f=s[1],d=s[0]);let p=Math.atan((d[1]-f[1])/(d[0]-f[0]));for(let m=0;m{let s=zw(a),l=Math.round(s/(2*r)),u=a[0],h=a[1];u[0]>h[0]&&(u=a[1],h=a[0]);let f=Math.atan((h[1]-u[1])/(h[0]-u[0]));for(let d=0;d2*Math.PI&&(A=0,S=2*Math.PI);let _=(S-A)/b.curveStepCount,I=[];for(let D=A;D<=S;D+=_)I.push([w+T*Math.cos(D),C+E*Math.sin(D)]);return I.push([w+T*Math.cos(S),C+E*Math.sin(S)]),I.push([w,C]),dm([I],b)}(e,r,n,i,a,s,h));return h.stroke!==ro&&f.push(d),this._d("arc",f,h)}curve(e,r){let n=this._o(r),i=[],a=vK(e,n);if(n.fill&&n.fill!==ro)if(n.fillStyle==="solid"){let s=vK(e,Object.assign(Object.assign({},n),{disableMultiStroke:!0,roughness:n.roughness?n.roughness+n.fillShapeRoughnessGain:0}));i.push({type:"fillPath",ops:this._mergedShape(s.ops)})}else{let s=[],l=e;if(l.length){let u=typeof l[0][0]=="number"?[l]:l;for(let h of u)h.length<3?s.push(...h):h.length===3?s.push(...BD(EK([h[0],h[0],h[1],h[2]]),10,(1+n.roughness)/2)):s.push(...BD(EK(h),10,(1+n.roughness)/2))}s.length&&i.push(dm([s],n))}return n.stroke!==ro&&i.push(a),this._d("curve",i,n)}polygon(e,r){let n=this._o(r),i=[],a=Ow(e,!0,n);return n.fill&&(n.fillStyle==="solid"?i.push(PD([e],n)):i.push(dm([e],n))),n.stroke!==ro&&i.push(a),this._d("polygon",i,n)}path(e,r){let n=this._o(r),i=[];if(!e)return this._d("path",i,n);e=(e||"").replace(/\n/g," ").replace(/(-\s)/g,"-").replace("/(ss)/g"," ");let a=n.fill&&n.fill!=="transparent"&&n.fill!==ro,s=n.stroke!==ro,l=!!(n.simplification&&n.simplification<1),u=function(f,d,p){let m=CK(SK(jD(f))),g=[],y=[],v=[0,0],x=[],b=o(()=>{x.length>=4&&y.push(...BD(x,d)),x=[]},"i"),w=o(()=>{b(),y.length&&(g.push(y),y=[])},"c");for(let{key:T,data:E}of m)switch(T){case"M":w(),v=[E[0],E[1]],y.push(v);break;case"L":b(),y.push([E[0],E[1]]);break;case"C":if(!x.length){let A=y.length?y[y.length-1]:v;x.push([A[0],A[1]])}x.push([E[0],E[1]]),x.push([E[2],E[3]]),x.push([E[4],E[5]]);break;case"Z":b(),y.push([v[0],v[1]])}if(w(),!p)return g;let C=[];for(let T of g){let E=o_e(T,p);E.length&&C.push(E)}return C}(e,1,l?4-4*(n.simplification||1):(1+n.roughness)/2),h=bK(e,n);if(a)if(n.fillStyle==="solid")if(u.length===1){let f=bK(e,Object.assign(Object.assign({},n),{disableMultiStroke:!0,roughness:n.roughness?n.roughness+n.fillShapeRoughnessGain:0}));i.push({type:"fillPath",ops:this._mergedShape(f.ops)})}else i.push(PD(u,n));else i.push(dm(u,n));return s&&(l?u.forEach(f=>{i.push(Ow(f,!1,n))}):i.push(h)),this._d("path",i,n)}opsToPath(e,r){let n="";for(let i of e.ops){let a=typeof r=="number"&&r>=0?i.data.map(s=>+s.toFixed(r)):i.data;switch(i.op){case"move":n+=`M${a[0]} ${a[1]} `;break;case"bcurveTo":n+=`C${a[0]} ${a[1]}, ${a[2]} ${a[3]}, ${a[4]} ${a[5]} `;break;case"lineTo":n+=`L${a[0]} ${a[1]} `}}return n.trim()}toPaths(e){let r=e.sets||[],n=e.options||this.defaultOptions,i=[];for(let a of r){let s=null;switch(a.type){case"path":s={d:this.opsToPath(a),stroke:n.stroke,strokeWidth:n.strokeWidth,fill:ro};break;case"fillPath":s={d:this.opsToPath(a),stroke:ro,strokeWidth:0,fill:n.fill||ro};break;case"fillSketch":s=this.fillSketch(a,n)}s&&i.push(s)}return i}fillSketch(e,r){let n=r.fillWeight;return n<0&&(n=r.strokeWidth/2),{d:this.opsToPath(e),stroke:r.fill||ro,strokeWidth:n,fill:ro}}_mergedShape(e){return e.filter((r,n)=>n===0||r.op!=="move")}},YD=class{static{o(this,"st")}constructor(e,r){this.canvas=e,this.ctx=this.canvas.getContext("2d"),this.gen=new pm(r)}draw(e){let r=e.sets||[],n=e.options||this.getDefaultOptions(),i=this.ctx,a=e.options.fixedDecimalPlaceDigits;for(let s of r)switch(s.type){case"path":i.save(),i.strokeStyle=n.stroke==="none"?"transparent":n.stroke,i.lineWidth=n.strokeWidth,n.strokeLineDash&&i.setLineDash(n.strokeLineDash),n.strokeLineDashOffset&&(i.lineDashOffset=n.strokeLineDashOffset),this._drawToContext(i,s,a),i.restore();break;case"fillPath":{i.save(),i.fillStyle=n.fill||"";let l=e.shape==="curve"||e.shape==="polygon"||e.shape==="path"?"evenodd":"nonzero";this._drawToContext(i,s,a,l),i.restore();break}case"fillSketch":this.fillSketch(i,s,n)}}fillSketch(e,r,n){let i=n.fillWeight;i<0&&(i=n.strokeWidth/2),e.save(),n.fillLineDash&&e.setLineDash(n.fillLineDash),n.fillLineDashOffset&&(e.lineDashOffset=n.fillLineDashOffset),e.strokeStyle=n.fill||"",e.lineWidth=i,this._drawToContext(e,r,n.fixedDecimalPlaceDigits),e.restore()}_drawToContext(e,r,n,i="nonzero"){e.beginPath();for(let a of r.ops){let s=typeof n=="number"&&n>=0?a.data.map(l=>+l.toFixed(n)):a.data;switch(a.op){case"move":e.moveTo(s[0],s[1]);break;case"bcurveTo":e.bezierCurveTo(s[0],s[1],s[2],s[3],s[4],s[5]);break;case"lineTo":e.lineTo(s[0],s[1])}}r.type==="fillPath"?e.fill(i):e.stroke()}get generator(){return this.gen}getDefaultOptions(){return this.gen.defaultOptions}line(e,r,n,i,a){let s=this.gen.line(e,r,n,i,a);return this.draw(s),s}rectangle(e,r,n,i,a){let s=this.gen.rectangle(e,r,n,i,a);return this.draw(s),s}ellipse(e,r,n,i,a){let s=this.gen.ellipse(e,r,n,i,a);return this.draw(s),s}circle(e,r,n,i){let a=this.gen.circle(e,r,n,i);return this.draw(a),a}linearPath(e,r){let n=this.gen.linearPath(e,r);return this.draw(n),n}polygon(e,r){let n=this.gen.polygon(e,r);return this.draw(n),n}arc(e,r,n,i,a,s,l=!1,u){let h=this.gen.arc(e,r,n,i,a,s,l,u);return this.draw(h),h}curve(e,r){let n=this.gen.curve(e,r);return this.draw(n),n}path(e,r){let n=this.gen.path(e,r);return this.draw(n),n}},Iw="http://www.w3.org/2000/svg",XD=class{static{o(this,"ot")}constructor(e,r){this.svg=e,this.gen=new pm(r)}draw(e){let r=e.sets||[],n=e.options||this.getDefaultOptions(),i=this.svg.ownerDocument||window.document,a=i.createElementNS(Iw,"g"),s=e.options.fixedDecimalPlaceDigits;for(let l of r){let u=null;switch(l.type){case"path":u=i.createElementNS(Iw,"path"),u.setAttribute("d",this.opsToPath(l,s)),u.setAttribute("stroke",n.stroke),u.setAttribute("stroke-width",n.strokeWidth+""),u.setAttribute("fill","none"),n.strokeLineDash&&u.setAttribute("stroke-dasharray",n.strokeLineDash.join(" ").trim()),n.strokeLineDashOffset&&u.setAttribute("stroke-dashoffset",`${n.strokeLineDashOffset}`);break;case"fillPath":u=i.createElementNS(Iw,"path"),u.setAttribute("d",this.opsToPath(l,s)),u.setAttribute("stroke","none"),u.setAttribute("stroke-width","0"),u.setAttribute("fill",n.fill||""),e.shape!=="curve"&&e.shape!=="polygon"||u.setAttribute("fill-rule","evenodd");break;case"fillSketch":u=this.fillSketch(i,l,n)}u&&a.appendChild(u)}return a}fillSketch(e,r,n){let i=n.fillWeight;i<0&&(i=n.strokeWidth/2);let a=e.createElementNS(Iw,"path");return a.setAttribute("d",this.opsToPath(r,n.fixedDecimalPlaceDigits)),a.setAttribute("stroke",n.fill||""),a.setAttribute("stroke-width",i+""),a.setAttribute("fill","none"),n.fillLineDash&&a.setAttribute("stroke-dasharray",n.fillLineDash.join(" ").trim()),n.fillLineDashOffset&&a.setAttribute("stroke-dashoffset",`${n.fillLineDashOffset}`),a}get generator(){return this.gen}getDefaultOptions(){return this.gen.defaultOptions}opsToPath(e,r){return this.gen.opsToPath(e,r)}line(e,r,n,i,a){let s=this.gen.line(e,r,n,i,a);return this.draw(s)}rectangle(e,r,n,i,a){let s=this.gen.rectangle(e,r,n,i,a);return this.draw(s)}ellipse(e,r,n,i,a){let s=this.gen.ellipse(e,r,n,i,a);return this.draw(s)}circle(e,r,n,i){let a=this.gen.circle(e,r,n,i);return this.draw(a)}linearPath(e,r){let n=this.gen.linearPath(e,r);return this.draw(n)}polygon(e,r){let n=this.gen.polygon(e,r);return this.draw(n)}arc(e,r,n,i,a,s,l=!1,u){let h=this.gen.arc(e,r,n,i,a,s,l,u);return this.draw(h)}curve(e,r){let n=this.gen.curve(e,r);return this.draw(n)}path(e,r){let n=this.gen.path(e,r);return this.draw(n)}},Xe={canvas:o((t,e)=>new YD(t,e),"canvas"),svg:o((t,e)=>new XD(t,e),"svg"),generator:o(t=>new pm(t),"generator"),newSeed:o(()=>pm.newSeed(),"newSeed")}});function RK(t,e){let{labelStyles:r}=Qe(e);e.labelStyle=r;let n=ht(e),i=n;n||(i="anchor");let a=t.insert("g").attr("class",i).attr("id",e.domId||e.id),s=1,{cssStyles:l}=e,u=Xe.svg(a),h=Ke(e,{fill:"black",stroke:"none",fillStyle:"solid"});e.look!=="handDrawn"&&(h.roughness=0);let f=u.circle(0,0,s*2,h),d=a.insert(()=>f,":first-child");return d.attr("class","anchor").attr("style",$n(l)),je(e,d),e.intersect=function(p){return Y.info("Circle intersect",e,s,p),Ye.circle(e,s,p)},a}var NK=N(()=>{"use strict";vt();Ft();Ht();Ut();Wt();ir();o(RK,"anchor")});function MK(t,e,r,n,i,a,s){let u=(t+r)/2,h=(e+n)/2,f=Math.atan2(n-e,r-t),d=(r-t)/2,p=(n-e)/2,m=d/i,g=p/a,y=Math.sqrt(m**2+g**2);if(y>1)throw new Error("The given radii are too small to create an arc between the points.");let v=Math.sqrt(1-y**2),x=u+v*a*Math.sin(f)*(s?-1:1),b=h-v*i*Math.cos(f)*(s?-1:1),w=Math.atan2((e-b)/a,(t-x)/i),T=Math.atan2((n-b)/a,(r-x)/i)-w;s&&T<0&&(T+=2*Math.PI),!s&&T>0&&(T-=2*Math.PI);let E=[];for(let A=0;A<20;A++){let S=A/19,_=w+S*T,I=x+i*Math.cos(_),D=b+a*Math.sin(_);E.push({x:I,y:D})}return E}async function IK(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a}=await pt(t,e,ht(e)),s=a.width+e.padding+20,l=a.height+e.padding,u=l/2,h=u/(2.5+l/50),{cssStyles:f}=e,d=[{x:s/2,y:-l/2},{x:-s/2,y:-l/2},...MK(-s/2,-l/2,-s/2,l/2,h,u,!1),{x:s/2,y:l/2},...MK(s/2,l/2,s/2,-l/2,h,u,!0)],p=Xe.svg(i),m=Ke(e,{});e.look!=="handDrawn"&&(m.roughness=0,m.fillStyle="solid");let g=Xt(d),y=p.path(g,m),v=i.insert(()=>y,":first-child");return v.attr("class","basic label-container"),f&&e.look!=="handDrawn"&&v.selectAll("path").attr("style",f),n&&e.look!=="handDrawn"&&v.selectAll("path").attr("style",n),v.attr("transform",`translate(${h/2}, 0)`),je(e,v),e.intersect=function(x){return Ye.polygon(e,d,x)},i}var OK=N(()=>{"use strict";Ft();Ht();Ut();Wt();o(MK,"generateArcPoints");o(IK,"bowTieRect")});function La(t,e,r,n){return t.insert("polygon",":first-child").attr("points",n.map(function(i){return i.x+","+i.y}).join(" ")).attr("class","label-container").attr("transform","translate("+-e/2+","+r/2+")")}var _u=N(()=>{"use strict";o(La,"insertPolygonShape")});async function PK(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a}=await pt(t,e,ht(e)),s=a.height+e.padding,l=12,u=a.width+e.padding+l,h=0,f=u,d=-s,p=0,m=[{x:h+l,y:d},{x:f,y:d},{x:f,y:p},{x:h,y:p},{x:h,y:d+l},{x:h+l,y:d}],g,{cssStyles:y}=e;if(e.look==="handDrawn"){let v=Xe.svg(i),x=Ke(e,{}),b=Xt(m),w=v.path(b,x);g=i.insert(()=>w,":first-child").attr("transform",`translate(${-u/2}, ${s/2})`),y&&g.attr("style",y)}else g=La(i,u,s,m);return n&&g.attr("style",n),je(e,g),e.intersect=function(v){return Ye.polygon(e,m,v)},i}var BK=N(()=>{"use strict";Ft();Ht();Ut();Wt();_u();Ft();o(PK,"card")});function FK(t,e){let{nodeStyles:r}=Qe(e);e.label="";let n=t.insert("g").attr("class",ht(e)).attr("id",e.domId??e.id),{cssStyles:i}=e,a=Math.max(28,e.width??0),s=[{x:0,y:a/2},{x:a/2,y:0},{x:0,y:-a/2},{x:-a/2,y:0}],l=Xe.svg(n),u=Ke(e,{});e.look!=="handDrawn"&&(u.roughness=0,u.fillStyle="solid");let h=Xt(s),f=l.path(h,u),d=n.insert(()=>f,":first-child");return i&&e.look!=="handDrawn"&&d.selectAll("path").attr("style",i),r&&e.look!=="handDrawn"&&d.selectAll("path").attr("style",r),e.width=28,e.height=28,e.intersect=function(p){return Ye.polygon(e,s,p)},n}var $K=N(()=>{"use strict";Ht();Wt();Ut();Ft();o(FK,"choice")});async function zK(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a,halfPadding:s}=await pt(t,e,ht(e)),l=a.width/2+s,u,{cssStyles:h}=e;if(e.look==="handDrawn"){let f=Xe.svg(i),d=Ke(e,{}),p=f.circle(0,0,l*2,d);u=i.insert(()=>p,":first-child"),u.attr("class","basic label-container").attr("style",$n(h))}else u=i.insert("circle",":first-child").attr("class","basic label-container").attr("style",n).attr("r",l).attr("cx",0).attr("cy",0);return je(e,u),e.intersect=function(f){return Y.info("Circle intersect",e,l,f),Ye.circle(e,l,f)},i}var GK=N(()=>{"use strict";vt();Ft();Ht();Ut();Wt();ir();o(zK,"circle")});function l_e(t){let e=Math.cos(Math.PI/4),r=Math.sin(Math.PI/4),n=t*2,i={x:n/2*e,y:n/2*r},a={x:-(n/2)*e,y:n/2*r},s={x:-(n/2)*e,y:-(n/2)*r},l={x:n/2*e,y:-(n/2)*r};return`M ${a.x},${a.y} L ${l.x},${l.y} + M ${i.x},${i.y} L ${s.x},${s.y}`}function VK(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r,e.label="";let i=t.insert("g").attr("class",ht(e)).attr("id",e.domId??e.id),a=Math.max(30,e?.width??0),{cssStyles:s}=e,l=Xe.svg(i),u=Ke(e,{});e.look!=="handDrawn"&&(u.roughness=0,u.fillStyle="solid");let h=l.circle(0,0,a*2,u),f=l_e(a),d=l.path(f,u),p=i.insert(()=>h,":first-child");return p.insert(()=>d),s&&e.look!=="handDrawn"&&p.selectAll("path").attr("style",s),n&&e.look!=="handDrawn"&&p.selectAll("path").attr("style",n),je(e,p),e.intersect=function(m){return Y.info("crossedCircle intersect",e,{radius:a,point:m}),Ye.circle(e,a,m)},i}var UK=N(()=>{"use strict";vt();Ft();Ut();Wt();Ht();o(l_e,"createLine");o(VK,"crossedCircle")});function Hh(t,e,r,n=100,i=0,a=180){let s=[],l=i*Math.PI/180,f=(a*Math.PI/180-l)/(n-1);for(let d=0;dw,":first-child").attr("stroke-opacity",0),C.insert(()=>x,":first-child"),C.attr("class","text"),f&&e.look!=="handDrawn"&&C.selectAll("path").attr("style",f),n&&e.look!=="handDrawn"&&C.selectAll("path").attr("style",n),C.attr("transform",`translate(${h}, 0)`),s.attr("transform",`translate(${-l/2+h-(a.x-(a.left??0))},${-u/2+(e.padding??0)/2-(a.y-(a.top??0))})`),je(e,C),e.intersect=function(T){return Ye.polygon(e,p,T)},i}var WK=N(()=>{"use strict";Ft();Ht();Ut();Wt();o(Hh,"generateCirclePoints");o(HK,"curlyBraceLeft")});function Wh(t,e,r,n=100,i=0,a=180){let s=[],l=i*Math.PI/180,f=(a*Math.PI/180-l)/(n-1);for(let d=0;dw,":first-child").attr("stroke-opacity",0),C.insert(()=>x,":first-child"),C.attr("class","text"),f&&e.look!=="handDrawn"&&C.selectAll("path").attr("style",f),n&&e.look!=="handDrawn"&&C.selectAll("path").attr("style",n),C.attr("transform",`translate(${-h}, 0)`),s.attr("transform",`translate(${-l/2+(e.padding??0)/2-(a.x-(a.left??0))},${-u/2+(e.padding??0)/2-(a.y-(a.top??0))})`),je(e,C),e.intersect=function(T){return Ye.polygon(e,p,T)},i}var YK=N(()=>{"use strict";Ft();Ht();Ut();Wt();o(Wh,"generateCirclePoints");o(qK,"curlyBraceRight")});function Ra(t,e,r,n=100,i=0,a=180){let s=[],l=i*Math.PI/180,f=(a*Math.PI/180-l)/(n-1);for(let d=0;dA,":first-child").attr("stroke-opacity",0),S.insert(()=>b,":first-child"),S.insert(()=>T,":first-child"),S.attr("class","text"),f&&e.look!=="handDrawn"&&S.selectAll("path").attr("style",f),n&&e.look!=="handDrawn"&&S.selectAll("path").attr("style",n),S.attr("transform",`translate(${h-h/4}, 0)`),s.attr("transform",`translate(${-l/2+(e.padding??0)/2-(a.x-(a.left??0))},${-u/2+(e.padding??0)/2-(a.y-(a.top??0))})`),je(e,S),e.intersect=function(_){return Ye.polygon(e,m,_)},i}var jK=N(()=>{"use strict";Ft();Ht();Ut();Wt();o(Ra,"generateCirclePoints");o(XK,"curlyBraces")});async function KK(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a}=await pt(t,e,ht(e)),s=80,l=20,u=Math.max(s,(a.width+(e.padding??0)*2)*1.25,e?.width??0),h=Math.max(l,a.height+(e.padding??0)*2,e?.height??0),f=h/2,{cssStyles:d}=e,p=Xe.svg(i),m=Ke(e,{});e.look!=="handDrawn"&&(m.roughness=0,m.fillStyle="solid");let g=u,y=h,v=g-f,x=y/4,b=[{x:v,y:0},{x,y:0},{x:0,y:y/2},{x,y},{x:v,y},...Lw(-v,-y/2,f,50,270,90)],w=Xt(b),C=p.path(w,m),T=i.insert(()=>C,":first-child");return T.attr("class","basic label-container"),d&&e.look!=="handDrawn"&&T.selectChildren("path").attr("style",d),n&&e.look!=="handDrawn"&&T.selectChildren("path").attr("style",n),T.attr("transform",`translate(${-u/2}, ${-h/2})`),je(e,T),e.intersect=function(E){return Ye.polygon(e,b,E)},i}var QK=N(()=>{"use strict";Ft();Ht();Ut();Wt();o(KK,"curvedTrapezoid")});async function ZK(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a,label:s}=await pt(t,e,ht(e)),l=Math.max(a.width+e.padding,e.width??0),u=l/2,h=u/(2.5+l/50),f=Math.max(a.height+h+e.padding,e.height??0),d,{cssStyles:p}=e;if(e.look==="handDrawn"){let m=Xe.svg(i),g=u_e(0,0,l,f,u,h),y=h_e(0,h,l,f,u,h),v=m.path(g,Ke(e,{})),x=m.path(y,Ke(e,{fill:"none"}));d=i.insert(()=>x,":first-child"),d=i.insert(()=>v,":first-child"),d.attr("class","basic label-container"),p&&d.attr("style",p)}else{let m=c_e(0,0,l,f,u,h);d=i.insert("path",":first-child").attr("d",m).attr("class","basic label-container").attr("style",$n(p)).attr("style",n)}return d.attr("label-offset-y",h),d.attr("transform",`translate(${-l/2}, ${-(f/2+h)})`),je(e,d),s.attr("transform",`translate(${-(a.width/2)-(a.x-(a.left??0))}, ${-(a.height/2)+(e.padding??0)/1.5-(a.y-(a.top??0))})`),e.intersect=function(m){let g=Ye.rect(e,m),y=g.x-(e.x??0);if(u!=0&&(Math.abs(y)<(e.width??0)/2||Math.abs(y)==(e.width??0)/2&&Math.abs(g.y-(e.y??0))>(e.height??0)/2-h)){let v=h*h*(1-y*y/(u*u));v>0&&(v=Math.sqrt(v)),v=h-v,m.y-(e.y??0)>0&&(v=-v),g.y+=v}return g},i}var c_e,u_e,h_e,JK=N(()=>{"use strict";Ft();Ht();Ut();Wt();ir();c_e=o((t,e,r,n,i,a)=>[`M${t},${e+a}`,`a${i},${a} 0,0,0 ${r},0`,`a${i},${a} 0,0,0 ${-r},0`,`l0,${n}`,`a${i},${a} 0,0,0 ${r},0`,`l0,${-n}`].join(" "),"createCylinderPathD"),u_e=o((t,e,r,n,i,a)=>[`M${t},${e+a}`,`M${t+r},${e+a}`,`a${i},${a} 0,0,0 ${-r},0`,`l0,${n}`,`a${i},${a} 0,0,0 ${r},0`,`l0,${-n}`].join(" "),"createOuterCylinderPathD"),h_e=o((t,e,r,n,i,a)=>[`M${t-r/2},${-n/2}`,`a${i},${a} 0,0,0 ${r},0`].join(" "),"createInnerCylinderPathD");o(ZK,"cylinder")});async function eQ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a,label:s}=await pt(t,e,ht(e)),l=a.width+e.padding,u=a.height+e.padding,h=u*.2,f=-l/2,d=-u/2-h/2,{cssStyles:p}=e,m=Xe.svg(i),g=Ke(e,{});e.look!=="handDrawn"&&(g.roughness=0,g.fillStyle="solid");let y=[{x:f,y:d+h},{x:-f,y:d+h},{x:-f,y:-d},{x:f,y:-d},{x:f,y:d},{x:-f,y:d},{x:-f,y:d+h}],v=m.polygon(y.map(b=>[b.x,b.y]),g),x=i.insert(()=>v,":first-child");return x.attr("class","basic label-container"),p&&e.look!=="handDrawn"&&x.selectAll("path").attr("style",p),n&&e.look!=="handDrawn"&&x.selectAll("path").attr("style",n),s.attr("transform",`translate(${f+(e.padding??0)/2-(a.x-(a.left??0))}, ${d+h+(e.padding??0)/2-(a.y-(a.top??0))})`),je(e,x),e.intersect=function(b){return Ye.rect(e,b)},i}var tQ=N(()=>{"use strict";Ft();Ht();Ut();Wt();o(eQ,"dividedRectangle")});async function rQ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a,halfPadding:s}=await pt(t,e,ht(e)),u=a.width/2+s+5,h=a.width/2+s,f,{cssStyles:d}=e;if(e.look==="handDrawn"){let p=Xe.svg(i),m=Ke(e,{roughness:.2,strokeWidth:2.5}),g=Ke(e,{roughness:.2,strokeWidth:1.5}),y=p.circle(0,0,u*2,m),v=p.circle(0,0,h*2,g);f=i.insert("g",":first-child"),f.attr("class",$n(e.cssClasses)).attr("style",$n(d)),f.node()?.appendChild(y),f.node()?.appendChild(v)}else{f=i.insert("g",":first-child");let p=f.insert("circle",":first-child"),m=f.insert("circle");f.attr("class","basic label-container").attr("style",n),p.attr("class","outer-circle").attr("style",n).attr("r",u).attr("cx",0).attr("cy",0),m.attr("class","inner-circle").attr("style",n).attr("r",h).attr("cx",0).attr("cy",0)}return je(e,f),e.intersect=function(p){return Y.info("DoubleCircle intersect",e,u,p),Ye.circle(e,u,p)},i}var nQ=N(()=>{"use strict";vt();Ft();Ht();Ut();Wt();ir();o(rQ,"doublecircle")});function iQ(t,e,{config:{themeVariables:r}}){let{labelStyles:n,nodeStyles:i}=Qe(e);e.label="",e.labelStyle=n;let a=t.insert("g").attr("class",ht(e)).attr("id",e.domId??e.id),s=7,{cssStyles:l}=e,u=Xe.svg(a),{nodeBorder:h}=r,f=Ke(e,{fillStyle:"solid"});e.look!=="handDrawn"&&(f.roughness=0);let d=u.circle(0,0,s*2,f),p=a.insert(()=>d,":first-child");return p.selectAll("path").attr("style",`fill: ${h} !important;`),l&&l.length>0&&e.look!=="handDrawn"&&p.selectAll("path").attr("style",l),i&&e.look!=="handDrawn"&&p.selectAll("path").attr("style",i),je(e,p),e.intersect=function(m){return Y.info("filledCircle intersect",e,{radius:s,point:m}),Ye.circle(e,s,m)},a}var aQ=N(()=>{"use strict";Wt();vt();Ht();Ut();Ft();o(iQ,"filledCircle")});async function sQ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a,label:s}=await pt(t,e,ht(e)),l=a.width+(e.padding??0),u=l+a.height,h=l+a.height,f=[{x:0,y:-u},{x:h,y:-u},{x:h/2,y:0}],{cssStyles:d}=e,p=Xe.svg(i),m=Ke(e,{});e.look!=="handDrawn"&&(m.roughness=0,m.fillStyle="solid");let g=Xt(f),y=p.path(g,m),v=i.insert(()=>y,":first-child").attr("transform",`translate(${-u/2}, ${u/2})`);return d&&e.look!=="handDrawn"&&v.selectChildren("path").attr("style",d),n&&e.look!=="handDrawn"&&v.selectChildren("path").attr("style",n),e.width=l,e.height=u,je(e,v),s.attr("transform",`translate(${-a.width/2-(a.x-(a.left??0))}, ${-u/2+(e.padding??0)/2+(a.y-(a.top??0))})`),e.intersect=function(x){return Y.info("Triangle intersect",e,f,x),Ye.polygon(e,f,x)},i}var oQ=N(()=>{"use strict";vt();Ft();Ht();Ut();Wt();Ft();o(sQ,"flippedTriangle")});function lQ(t,e,{dir:r,config:{state:n,themeVariables:i}}){let{nodeStyles:a}=Qe(e);e.label="";let s=t.insert("g").attr("class",ht(e)).attr("id",e.domId??e.id),{cssStyles:l}=e,u=Math.max(70,e?.width??0),h=Math.max(10,e?.height??0);r==="LR"&&(u=Math.max(10,e?.width??0),h=Math.max(70,e?.height??0));let f=-1*u/2,d=-1*h/2,p=Xe.svg(s),m=Ke(e,{stroke:i.lineColor,fill:i.lineColor});e.look!=="handDrawn"&&(m.roughness=0,m.fillStyle="solid");let g=p.rectangle(f,d,u,h,m),y=s.insert(()=>g,":first-child");l&&e.look!=="handDrawn"&&y.selectAll("path").attr("style",l),a&&e.look!=="handDrawn"&&y.selectAll("path").attr("style",a),je(e,y);let v=n?.padding??0;return e.width&&e.height&&(e.width+=v/2||0,e.height+=v/2||0),e.intersect=function(x){return Ye.rect(e,x)},s}var cQ=N(()=>{"use strict";Wt();Ht();Ut();Ft();o(lQ,"forkJoin")});async function uQ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let i=80,a=50,{shapeSvg:s,bbox:l}=await pt(t,e,ht(e)),u=Math.max(i,l.width+(e.padding??0)*2,e?.width??0),h=Math.max(a,l.height+(e.padding??0)*2,e?.height??0),f=h/2,{cssStyles:d}=e,p=Xe.svg(s),m=Ke(e,{});e.look!=="handDrawn"&&(m.roughness=0,m.fillStyle="solid");let g=[{x:-u/2,y:-h/2},{x:u/2-f,y:-h/2},...Lw(-u/2+f,0,f,50,90,270),{x:u/2-f,y:h/2},{x:-u/2,y:h/2}],y=Xt(g),v=p.path(y,m),x=s.insert(()=>v,":first-child");return x.attr("class","basic label-container"),d&&e.look!=="handDrawn"&&x.selectChildren("path").attr("style",d),n&&e.look!=="handDrawn"&&x.selectChildren("path").attr("style",n),je(e,x),e.intersect=function(b){return Y.info("Pill intersect",e,{radius:f,point:b}),Ye.polygon(e,g,b)},s}var hQ=N(()=>{"use strict";vt();Ft();Ht();Ut();Wt();o(uQ,"halfRoundedRectangle")});async function fQ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a}=await pt(t,e,ht(e)),s=4,l=a.height+e.padding,u=l/s,h=a.width+2*u+e.padding,f=[{x:u,y:0},{x:h-u,y:0},{x:h,y:-l/2},{x:h-u,y:-l},{x:u,y:-l},{x:0,y:-l/2}],d,{cssStyles:p}=e;if(e.look==="handDrawn"){let m=Xe.svg(i),g=Ke(e,{}),y=f_e(0,0,h,l,u),v=m.path(y,g);d=i.insert(()=>v,":first-child").attr("transform",`translate(${-h/2}, ${l/2})`),p&&d.attr("style",p)}else d=La(i,h,l,f);return n&&d.attr("style",n),e.width=h,e.height=l,je(e,d),e.intersect=function(m){return Ye.polygon(e,f,m)},i}var f_e,dQ=N(()=>{"use strict";Ft();Ht();Ut();Wt();_u();f_e=o((t,e,r,n,i)=>[`M${t+i},${e}`,`L${t+r-i},${e}`,`L${t+r},${e-n/2}`,`L${t+r-i},${e-n}`,`L${t+i},${e-n}`,`L${t},${e-n/2}`,"Z"].join(" "),"createHexagonPathD");o(fQ,"hexagon")});async function pQ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.label="",e.labelStyle=r;let{shapeSvg:i}=await pt(t,e,ht(e)),a=Math.max(30,e?.width??0),s=Math.max(30,e?.height??0),{cssStyles:l}=e,u=Xe.svg(i),h=Ke(e,{});e.look!=="handDrawn"&&(h.roughness=0,h.fillStyle="solid");let f=[{x:0,y:0},{x:a,y:0},{x:0,y:s},{x:a,y:s}],d=Xt(f),p=u.path(d,h),m=i.insert(()=>p,":first-child");return m.attr("class","basic label-container"),l&&e.look!=="handDrawn"&&m.selectChildren("path").attr("style",l),n&&e.look!=="handDrawn"&&m.selectChildren("path").attr("style",n),m.attr("transform",`translate(${-a/2}, ${-s/2})`),je(e,m),e.intersect=function(g){return Y.info("Pill intersect",e,{points:f}),Ye.polygon(e,f,g)},i}var mQ=N(()=>{"use strict";vt();Ft();Ht();Ut();Wt();o(pQ,"hourglass")});async function gQ(t,e,{config:{themeVariables:r,flowchart:n}}){let{labelStyles:i}=Qe(e);e.labelStyle=i;let a=e.assetHeight??48,s=e.assetWidth??48,l=Math.max(a,s),u=n?.wrappingWidth;e.width=Math.max(l,u??0);let{shapeSvg:h,bbox:f,label:d}=await pt(t,e,"icon-shape default"),p=e.pos==="t",m=l,g=l,{nodeBorder:y}=r,{stylesMap:v}=mc(e),x=-g/2,b=-m/2,w=e.label?8:0,C=Xe.svg(h),T=Ke(e,{stroke:"none",fill:"none"});e.look!=="handDrawn"&&(T.roughness=0,T.fillStyle="solid");let E=C.rectangle(x,b,g,m,T),A=Math.max(g,f.width),S=m+f.height+w,_=C.rectangle(-A/2,-S/2,A,S,{...T,fill:"transparent",stroke:"none"}),I=h.insert(()=>E,":first-child"),D=h.insert(()=>_);if(e.icon){let k=h.append("g");k.html(`${await wo(e.icon,{height:l,width:l,fallbackPrefix:""})}`);let L=k.node().getBBox(),R=L.width,O=L.height,M=L.x,B=L.y;k.attr("transform",`translate(${-R/2-M},${p?f.height/2+w/2-O/2-B:-f.height/2-w/2-O/2-B})`),k.attr("style",`color: ${v.get("stroke")??y};`)}return d.attr("transform",`translate(${-f.width/2-(f.x-(f.left??0))},${p?-S/2:S/2-f.height})`),I.attr("transform",`translate(0,${p?f.height/2+w/2:-f.height/2-w/2})`),je(e,D),e.intersect=function(k){if(Y.info("iconSquare intersect",e,k),!e.label)return Ye.rect(e,k);let L=e.x??0,R=e.y??0,O=e.height??0,M=[];return p?M=[{x:L-f.width/2,y:R-O/2},{x:L+f.width/2,y:R-O/2},{x:L+f.width/2,y:R-O/2+f.height+w},{x:L+g/2,y:R-O/2+f.height+w},{x:L+g/2,y:R+O/2},{x:L-g/2,y:R+O/2},{x:L-g/2,y:R-O/2+f.height+w},{x:L-f.width/2,y:R-O/2+f.height+w}]:M=[{x:L-g/2,y:R-O/2},{x:L+g/2,y:R-O/2},{x:L+g/2,y:R-O/2+m},{x:L+f.width/2,y:R-O/2+m},{x:L+f.width/2/2,y:R+O/2},{x:L-f.width/2,y:R+O/2},{x:L-f.width/2,y:R-O/2+m},{x:L-g/2,y:R-O/2+m}],Ye.polygon(e,M,k)},h}var yQ=N(()=>{"use strict";Wt();vt();tu();Ht();Ut();Ft();o(gQ,"icon")});async function vQ(t,e,{config:{themeVariables:r,flowchart:n}}){let{labelStyles:i}=Qe(e);e.labelStyle=i;let a=e.assetHeight??48,s=e.assetWidth??48,l=Math.max(a,s),u=n?.wrappingWidth;e.width=Math.max(l,u??0);let{shapeSvg:h,bbox:f,label:d}=await pt(t,e,"icon-shape default"),p=20,m=e.label?8:0,g=e.pos==="t",{nodeBorder:y,mainBkg:v}=r,{stylesMap:x}=mc(e),b=Xe.svg(h),w=Ke(e,{});e.look!=="handDrawn"&&(w.roughness=0,w.fillStyle="solid");let C=x.get("fill");w.stroke=C??v;let T=h.append("g");e.icon&&T.html(`${await wo(e.icon,{height:l,width:l,fallbackPrefix:""})}`);let E=T.node().getBBox(),A=E.width,S=E.height,_=E.x,I=E.y,D=Math.max(A,S)*Math.SQRT2+p*2,k=b.circle(0,0,D,w),L=Math.max(D,f.width),R=D+f.height+m,O=b.rectangle(-L/2,-R/2,L,R,{...w,fill:"transparent",stroke:"none"}),M=h.insert(()=>k,":first-child"),B=h.insert(()=>O);return T.attr("transform",`translate(${-A/2-_},${g?f.height/2+m/2-S/2-I:-f.height/2-m/2-S/2-I})`),T.attr("style",`color: ${x.get("stroke")??y};`),d.attr("transform",`translate(${-f.width/2-(f.x-(f.left??0))},${g?-R/2:R/2-f.height})`),M.attr("transform",`translate(0,${g?f.height/2+m/2:-f.height/2-m/2})`),je(e,B),e.intersect=function(F){return Y.info("iconSquare intersect",e,F),Ye.rect(e,F)},h}var xQ=N(()=>{"use strict";Wt();vt();tu();Ht();Ut();Ft();o(vQ,"iconCircle")});var Na,qh=N(()=>{"use strict";Na=o((t,e,r,n,i)=>["M",t+i,e,"H",t+r-i,"A",i,i,0,0,1,t+r,e+i,"V",e+n-i,"A",i,i,0,0,1,t+r-i,e+n,"H",t+i,"A",i,i,0,0,1,t,e+n-i,"V",e+i,"A",i,i,0,0,1,t+i,e,"Z"].join(" "),"createRoundedRectPathD")});async function bQ(t,e,{config:{themeVariables:r,flowchart:n}}){let{labelStyles:i}=Qe(e);e.labelStyle=i;let a=e.assetHeight??48,s=e.assetWidth??48,l=Math.max(a,s),u=n?.wrappingWidth;e.width=Math.max(l,u??0);let{shapeSvg:h,bbox:f,halfPadding:d,label:p}=await pt(t,e,"icon-shape default"),m=e.pos==="t",g=l+d*2,y=l+d*2,{nodeBorder:v,mainBkg:x}=r,{stylesMap:b}=mc(e),w=-y/2,C=-g/2,T=e.label?8:0,E=Xe.svg(h),A=Ke(e,{});e.look!=="handDrawn"&&(A.roughness=0,A.fillStyle="solid");let S=b.get("fill");A.stroke=S??x;let _=E.path(Na(w,C,y,g,5),A),I=Math.max(y,f.width),D=g+f.height+T,k=E.rectangle(-I/2,-D/2,I,D,{...A,fill:"transparent",stroke:"none"}),L=h.insert(()=>_,":first-child").attr("class","icon-shape2"),R=h.insert(()=>k);if(e.icon){let O=h.append("g");O.html(`${await wo(e.icon,{height:l,width:l,fallbackPrefix:""})}`);let M=O.node().getBBox(),B=M.width,F=M.height,P=M.x,z=M.y;O.attr("transform",`translate(${-B/2-P},${m?f.height/2+T/2-F/2-z:-f.height/2-T/2-F/2-z})`),O.attr("style",`color: ${b.get("stroke")??v};`)}return p.attr("transform",`translate(${-f.width/2-(f.x-(f.left??0))},${m?-D/2:D/2-f.height})`),L.attr("transform",`translate(0,${m?f.height/2+T/2:-f.height/2-T/2})`),je(e,R),e.intersect=function(O){if(Y.info("iconSquare intersect",e,O),!e.label)return Ye.rect(e,O);let M=e.x??0,B=e.y??0,F=e.height??0,P=[];return m?P=[{x:M-f.width/2,y:B-F/2},{x:M+f.width/2,y:B-F/2},{x:M+f.width/2,y:B-F/2+f.height+T},{x:M+y/2,y:B-F/2+f.height+T},{x:M+y/2,y:B+F/2},{x:M-y/2,y:B+F/2},{x:M-y/2,y:B-F/2+f.height+T},{x:M-f.width/2,y:B-F/2+f.height+T}]:P=[{x:M-y/2,y:B-F/2},{x:M+y/2,y:B-F/2},{x:M+y/2,y:B-F/2+g},{x:M+f.width/2,y:B-F/2+g},{x:M+f.width/2/2,y:B+F/2},{x:M-f.width/2,y:B+F/2},{x:M-f.width/2,y:B-F/2+g},{x:M-y/2,y:B-F/2+g}],Ye.polygon(e,P,O)},h}var wQ=N(()=>{"use strict";Wt();vt();tu();Ht();Ut();qh();Ft();o(bQ,"iconRounded")});async function TQ(t,e,{config:{themeVariables:r,flowchart:n}}){let{labelStyles:i}=Qe(e);e.labelStyle=i;let a=e.assetHeight??48,s=e.assetWidth??48,l=Math.max(a,s),u=n?.wrappingWidth;e.width=Math.max(l,u??0);let{shapeSvg:h,bbox:f,halfPadding:d,label:p}=await pt(t,e,"icon-shape default"),m=e.pos==="t",g=l+d*2,y=l+d*2,{nodeBorder:v,mainBkg:x}=r,{stylesMap:b}=mc(e),w=-y/2,C=-g/2,T=e.label?8:0,E=Xe.svg(h),A=Ke(e,{});e.look!=="handDrawn"&&(A.roughness=0,A.fillStyle="solid");let S=b.get("fill");A.stroke=S??x;let _=E.path(Na(w,C,y,g,.1),A),I=Math.max(y,f.width),D=g+f.height+T,k=E.rectangle(-I/2,-D/2,I,D,{...A,fill:"transparent",stroke:"none"}),L=h.insert(()=>_,":first-child"),R=h.insert(()=>k);if(e.icon){let O=h.append("g");O.html(`${await wo(e.icon,{height:l,width:l,fallbackPrefix:""})}`);let M=O.node().getBBox(),B=M.width,F=M.height,P=M.x,z=M.y;O.attr("transform",`translate(${-B/2-P},${m?f.height/2+T/2-F/2-z:-f.height/2-T/2-F/2-z})`),O.attr("style",`color: ${b.get("stroke")??v};`)}return p.attr("transform",`translate(${-f.width/2-(f.x-(f.left??0))},${m?-D/2:D/2-f.height})`),L.attr("transform",`translate(0,${m?f.height/2+T/2:-f.height/2-T/2})`),je(e,R),e.intersect=function(O){if(Y.info("iconSquare intersect",e,O),!e.label)return Ye.rect(e,O);let M=e.x??0,B=e.y??0,F=e.height??0,P=[];return m?P=[{x:M-f.width/2,y:B-F/2},{x:M+f.width/2,y:B-F/2},{x:M+f.width/2,y:B-F/2+f.height+T},{x:M+y/2,y:B-F/2+f.height+T},{x:M+y/2,y:B+F/2},{x:M-y/2,y:B+F/2},{x:M-y/2,y:B-F/2+f.height+T},{x:M-f.width/2,y:B-F/2+f.height+T}]:P=[{x:M-y/2,y:B-F/2},{x:M+y/2,y:B-F/2},{x:M+y/2,y:B-F/2+g},{x:M+f.width/2,y:B-F/2+g},{x:M+f.width/2/2,y:B+F/2},{x:M-f.width/2,y:B+F/2},{x:M-f.width/2,y:B-F/2+g},{x:M-y/2,y:B-F/2+g}],Ye.polygon(e,P,O)},h}var kQ=N(()=>{"use strict";Wt();vt();tu();Ht();qh();Ut();Ft();o(TQ,"iconSquare")});async function EQ(t,e,{config:{flowchart:r}}){let n=new Image;n.src=e?.img??"",await n.decode();let i=Number(n.naturalWidth.toString().replace("px","")),a=Number(n.naturalHeight.toString().replace("px",""));e.imageAspectRatio=i/a;let{labelStyles:s}=Qe(e);e.labelStyle=s;let l=r?.wrappingWidth;e.defaultWidth=r?.wrappingWidth;let u=Math.max(e.label?l??0:0,e?.assetWidth??i),h=e.constraint==="on"&&e?.assetHeight?e.assetHeight*e.imageAspectRatio:u,f=e.constraint==="on"?h/e.imageAspectRatio:e?.assetHeight??a;e.width=Math.max(h,l??0);let{shapeSvg:d,bbox:p,label:m}=await pt(t,e,"image-shape default"),g=e.pos==="t",y=-h/2,v=-f/2,x=e.label?8:0,b=Xe.svg(d),w=Ke(e,{});e.look!=="handDrawn"&&(w.roughness=0,w.fillStyle="solid");let C=b.rectangle(y,v,h,f,w),T=Math.max(h,p.width),E=f+p.height+x,A=b.rectangle(-T/2,-E/2,T,E,{...w,fill:"none",stroke:"none"}),S=d.insert(()=>C,":first-child"),_=d.insert(()=>A);if(e.img){let I=d.append("image");I.attr("href",e.img),I.attr("width",h),I.attr("height",f),I.attr("preserveAspectRatio","none"),I.attr("transform",`translate(${-h/2},${g?E/2-f:-E/2})`)}return m.attr("transform",`translate(${-p.width/2-(p.x-(p.left??0))},${g?-f/2-p.height/2-x/2:f/2-p.height/2+x/2})`),S.attr("transform",`translate(0,${g?p.height/2+x/2:-p.height/2-x/2})`),je(e,_),e.intersect=function(I){if(Y.info("iconSquare intersect",e,I),!e.label)return Ye.rect(e,I);let D=e.x??0,k=e.y??0,L=e.height??0,R=[];return g?R=[{x:D-p.width/2,y:k-L/2},{x:D+p.width/2,y:k-L/2},{x:D+p.width/2,y:k-L/2+p.height+x},{x:D+h/2,y:k-L/2+p.height+x},{x:D+h/2,y:k+L/2},{x:D-h/2,y:k+L/2},{x:D-h/2,y:k-L/2+p.height+x},{x:D-p.width/2,y:k-L/2+p.height+x}]:R=[{x:D-h/2,y:k-L/2},{x:D+h/2,y:k-L/2},{x:D+h/2,y:k-L/2+f},{x:D+p.width/2,y:k-L/2+f},{x:D+p.width/2/2,y:k+L/2},{x:D-p.width/2,y:k+L/2},{x:D-p.width/2,y:k-L/2+f},{x:D-h/2,y:k-L/2+f}],Ye.polygon(e,R,I)},d}var SQ=N(()=>{"use strict";Wt();vt();Ht();Ut();Ft();o(EQ,"imageSquare")});async function CQ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a}=await pt(t,e,ht(e)),s=Math.max(a.width+(e.padding??0)*2,e?.width??0),l=Math.max(a.height+(e.padding??0)*2,e?.height??0),u=[{x:0,y:0},{x:s,y:0},{x:s+3*l/6,y:-l},{x:-3*l/6,y:-l}],h,{cssStyles:f}=e;if(e.look==="handDrawn"){let d=Xe.svg(i),p=Ke(e,{}),m=Xt(u),g=d.path(m,p);h=i.insert(()=>g,":first-child").attr("transform",`translate(${-s/2}, ${l/2})`),f&&h.attr("style",f)}else h=La(i,s,l,u);return n&&h.attr("style",n),e.width=s,e.height=l,je(e,h),e.intersect=function(d){return Ye.polygon(e,u,d)},i}var AQ=N(()=>{"use strict";Ft();Ht();Ut();Wt();_u();o(CQ,"inv_trapezoid")});async function Du(t,e,r){let{labelStyles:n,nodeStyles:i}=Qe(e);e.labelStyle=n;let{shapeSvg:a,bbox:s}=await pt(t,e,ht(e)),l=Math.max(s.width+r.labelPaddingX*2,e?.width||0),u=Math.max(s.height+r.labelPaddingY*2,e?.height||0),h=-l/2,f=-u/2,d,{rx:p,ry:m}=e,{cssStyles:g}=e;if(r?.rx&&r.ry&&(p=r.rx,m=r.ry),e.look==="handDrawn"){let y=Xe.svg(a),v=Ke(e,{}),x=p||m?y.path(Na(h,f,l,u,p||0),v):y.rectangle(h,f,l,u,v);d=a.insert(()=>x,":first-child"),d.attr("class","basic label-container").attr("style",$n(g))}else d=a.insert("rect",":first-child"),d.attr("class","basic label-container").attr("style",i).attr("rx",$n(p)).attr("ry",$n(m)).attr("x",h).attr("y",f).attr("width",l).attr("height",u);return je(e,d),e.intersect=function(y){return Ye.rect(e,y)},a}var mm=N(()=>{"use strict";Ft();Ht();qh();Ut();Wt();ir();o(Du,"drawRect")});async function _Q(t,e){let{shapeSvg:r,bbox:n,label:i}=await pt(t,e,"label"),a=r.insert("rect",":first-child");return a.attr("width",.1).attr("height",.1),r.attr("class","label edgeLabel"),i.attr("transform",`translate(${-(n.width/2)-(n.x-(n.left??0))}, ${-(n.height/2)-(n.y-(n.top??0))})`),je(e,a),e.intersect=function(u){return Ye.rect(e,u)},r}var DQ=N(()=>{"use strict";mm();Ft();Ht();o(_Q,"labelRect")});async function LQ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a}=await pt(t,e,ht(e)),s=Math.max(a.width+(e.padding??0),e?.width??0),l=Math.max(a.height+(e.padding??0),e?.height??0),u=[{x:0,y:0},{x:s+3*l/6,y:0},{x:s,y:-l},{x:-(3*l)/6,y:-l}],h,{cssStyles:f}=e;if(e.look==="handDrawn"){let d=Xe.svg(i),p=Ke(e,{}),m=Xt(u),g=d.path(m,p);h=i.insert(()=>g,":first-child").attr("transform",`translate(${-s/2}, ${l/2})`),f&&h.attr("style",f)}else h=La(i,s,l,u);return n&&h.attr("style",n),e.width=s,e.height=l,je(e,h),e.intersect=function(d){return Ye.polygon(e,u,d)},i}var RQ=N(()=>{"use strict";Ft();Ht();Ut();Wt();_u();o(LQ,"lean_left")});async function NQ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a}=await pt(t,e,ht(e)),s=Math.max(a.width+(e.padding??0),e?.width??0),l=Math.max(a.height+(e.padding??0),e?.height??0),u=[{x:-3*l/6,y:0},{x:s,y:0},{x:s+3*l/6,y:-l},{x:0,y:-l}],h,{cssStyles:f}=e;if(e.look==="handDrawn"){let d=Xe.svg(i),p=Ke(e,{}),m=Xt(u),g=d.path(m,p);h=i.insert(()=>g,":first-child").attr("transform",`translate(${-s/2}, ${l/2})`),f&&h.attr("style",f)}else h=La(i,s,l,u);return n&&h.attr("style",n),e.width=s,e.height=l,je(e,h),e.intersect=function(d){return Ye.polygon(e,u,d)},i}var MQ=N(()=>{"use strict";Ft();Ht();Ut();Wt();_u();o(NQ,"lean_right")});function IQ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.label="",e.labelStyle=r;let i=t.insert("g").attr("class",ht(e)).attr("id",e.domId??e.id),{cssStyles:a}=e,s=Math.max(35,e?.width??0),l=Math.max(35,e?.height??0),u=7,h=[{x:s,y:0},{x:0,y:l+u/2},{x:s-2*u,y:l+u/2},{x:0,y:2*l},{x:s,y:l-u/2},{x:2*u,y:l-u/2}],f=Xe.svg(i),d=Ke(e,{});e.look!=="handDrawn"&&(d.roughness=0,d.fillStyle="solid");let p=Xt(h),m=f.path(p,d),g=i.insert(()=>m,":first-child");return a&&e.look!=="handDrawn"&&g.selectAll("path").attr("style",a),n&&e.look!=="handDrawn"&&g.selectAll("path").attr("style",n),g.attr("transform",`translate(-${s/2},${-l})`),je(e,g),e.intersect=function(y){return Y.info("lightningBolt intersect",e,y),Ye.polygon(e,h,y)},i}var OQ=N(()=>{"use strict";vt();Ft();Ut();Wt();Ht();Ft();o(IQ,"lightningBolt")});async function PQ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a,label:s}=await pt(t,e,ht(e)),l=Math.max(a.width+(e.padding??0),e.width??0),u=l/2,h=u/(2.5+l/50),f=Math.max(a.height+h+(e.padding??0),e.height??0),d=f*.1,p,{cssStyles:m}=e;if(e.look==="handDrawn"){let g=Xe.svg(i),y=p_e(0,0,l,f,u,h,d),v=m_e(0,h,l,f,u,h),x=Ke(e,{}),b=g.path(y,x),w=g.path(v,x);i.insert(()=>w,":first-child").attr("class","line"),p=i.insert(()=>b,":first-child"),p.attr("class","basic label-container"),m&&p.attr("style",m)}else{let g=d_e(0,0,l,f,u,h,d);p=i.insert("path",":first-child").attr("d",g).attr("class","basic label-container").attr("style",$n(m)).attr("style",n)}return p.attr("label-offset-y",h),p.attr("transform",`translate(${-l/2}, ${-(f/2+h)})`),je(e,p),s.attr("transform",`translate(${-(a.width/2)-(a.x-(a.left??0))}, ${-(a.height/2)+h-(a.y-(a.top??0))})`),e.intersect=function(g){let y=Ye.rect(e,g),v=y.x-(e.x??0);if(u!=0&&(Math.abs(v)<(e.width??0)/2||Math.abs(v)==(e.width??0)/2&&Math.abs(y.y-(e.y??0))>(e.height??0)/2-h)){let x=h*h*(1-v*v/(u*u));x>0&&(x=Math.sqrt(x)),x=h-x,g.y-(e.y??0)>0&&(x=-x),y.y+=x}return y},i}var d_e,p_e,m_e,BQ=N(()=>{"use strict";Ft();Ht();Ut();Wt();ir();d_e=o((t,e,r,n,i,a,s)=>[`M${t},${e+a}`,`a${i},${a} 0,0,0 ${r},0`,`a${i},${a} 0,0,0 ${-r},0`,`l0,${n}`,`a${i},${a} 0,0,0 ${r},0`,`l0,${-n}`,`M${t},${e+a+s}`,`a${i},${a} 0,0,0 ${r},0`].join(" "),"createCylinderPathD"),p_e=o((t,e,r,n,i,a,s)=>[`M${t},${e+a}`,`M${t+r},${e+a}`,`a${i},${a} 0,0,0 ${-r},0`,`l0,${n}`,`a${i},${a} 0,0,0 ${r},0`,`l0,${-n}`,`M${t},${e+a+s}`,`a${i},${a} 0,0,0 ${r},0`].join(" "),"createOuterCylinderPathD"),m_e=o((t,e,r,n,i,a)=>[`M${t-r/2},${-n/2}`,`a${i},${a} 0,0,0 ${r},0`].join(" "),"createInnerCylinderPathD");o(PQ,"linedCylinder")});async function FQ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a,label:s}=await pt(t,e,ht(e)),l=Math.max(a.width+(e.padding??0)*2,e?.width??0),u=Math.max(a.height+(e.padding??0)*2,e?.height??0),h=u/4,f=u+h,{cssStyles:d}=e,p=Xe.svg(i),m=Ke(e,{});e.look!=="handDrawn"&&(m.roughness=0,m.fillStyle="solid");let g=[{x:-l/2-l/2*.1,y:-f/2},{x:-l/2-l/2*.1,y:f/2},...Fo(-l/2-l/2*.1,f/2,l/2+l/2*.1,f/2,h,.8),{x:l/2+l/2*.1,y:-f/2},{x:-l/2-l/2*.1,y:-f/2},{x:-l/2,y:-f/2},{x:-l/2,y:f/2*1.1},{x:-l/2,y:-f/2}],y=p.polygon(g.map(x=>[x.x,x.y]),m),v=i.insert(()=>y,":first-child");return v.attr("class","basic label-container"),d&&e.look!=="handDrawn"&&v.selectAll("path").attr("style",d),n&&e.look!=="handDrawn"&&v.selectAll("path").attr("style",n),v.attr("transform",`translate(0,${-h/2})`),s.attr("transform",`translate(${-l/2+(e.padding??0)+l/2*.1/2-(a.x-(a.left??0))},${-u/2+(e.padding??0)-h/2-(a.y-(a.top??0))})`),je(e,v),e.intersect=function(x){return Ye.polygon(e,g,x)},i}var $Q=N(()=>{"use strict";Ft();Ht();Wt();Ut();o(FQ,"linedWaveEdgedRect")});async function zQ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a,label:s}=await pt(t,e,ht(e)),l=Math.max(a.width+(e.padding??0)*2,e?.width??0),u=Math.max(a.height+(e.padding??0)*2,e?.height??0),h=5,f=-l/2,d=-u/2,{cssStyles:p}=e,m=Xe.svg(i),g=Ke(e,{}),y=[{x:f-h,y:d+h},{x:f-h,y:d+u+h},{x:f+l-h,y:d+u+h},{x:f+l-h,y:d+u},{x:f+l,y:d+u},{x:f+l,y:d+u-h},{x:f+l+h,y:d+u-h},{x:f+l+h,y:d-h},{x:f+h,y:d-h},{x:f+h,y:d},{x:f,y:d},{x:f,y:d+h}],v=[{x:f,y:d+h},{x:f+l-h,y:d+h},{x:f+l-h,y:d+u},{x:f+l,y:d+u},{x:f+l,y:d},{x:f,y:d}];e.look!=="handDrawn"&&(g.roughness=0,g.fillStyle="solid");let x=Xt(y),b=m.path(x,g),w=Xt(v),C=m.path(w,{...g,fill:"none"}),T=i.insert(()=>C,":first-child");return T.insert(()=>b,":first-child"),T.attr("class","basic label-container"),p&&e.look!=="handDrawn"&&T.selectAll("path").attr("style",p),n&&e.look!=="handDrawn"&&T.selectAll("path").attr("style",n),s.attr("transform",`translate(${-(a.width/2)-h-(a.x-(a.left??0))}, ${-(a.height/2)+h-(a.y-(a.top??0))})`),je(e,T),e.intersect=function(E){return Ye.polygon(e,y,E)},i}var GQ=N(()=>{"use strict";Ft();Ut();Wt();Ht();o(zQ,"multiRect")});async function VQ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a,label:s}=await pt(t,e,ht(e)),l=Math.max(a.width+(e.padding??0)*2,e?.width??0),u=Math.max(a.height+(e.padding??0)*2,e?.height??0),h=u/4,f=u+h,d=-l/2,p=-f/2,m=5,{cssStyles:g}=e,y=Fo(d-m,p+f+m,d+l-m,p+f+m,h,.8),v=y?.[y.length-1],x=[{x:d-m,y:p+m},{x:d-m,y:p+f+m},...y,{x:d+l-m,y:v.y-m},{x:d+l,y:v.y-m},{x:d+l,y:v.y-2*m},{x:d+l+m,y:v.y-2*m},{x:d+l+m,y:p-m},{x:d+m,y:p-m},{x:d+m,y:p},{x:d,y:p},{x:d,y:p+m}],b=[{x:d,y:p+m},{x:d+l-m,y:p+m},{x:d+l-m,y:v.y-m},{x:d+l,y:v.y-m},{x:d+l,y:p},{x:d,y:p}],w=Xe.svg(i),C=Ke(e,{});e.look!=="handDrawn"&&(C.roughness=0,C.fillStyle="solid");let T=Xt(x),E=w.path(T,C),A=Xt(b),S=w.path(A,C),_=i.insert(()=>E,":first-child");return _.insert(()=>S),_.attr("class","basic label-container"),g&&e.look!=="handDrawn"&&_.selectAll("path").attr("style",g),n&&e.look!=="handDrawn"&&_.selectAll("path").attr("style",n),_.attr("transform",`translate(0,${-h/2})`),s.attr("transform",`translate(${-(a.width/2)-m-(a.x-(a.left??0))}, ${-(a.height/2)+m-h/2-(a.y-(a.top??0))})`),je(e,_),e.intersect=function(I){return Ye.polygon(e,x,I)},i}var UQ=N(()=>{"use strict";Ft();Ht();Wt();Ut();o(VQ,"multiWaveEdgedRectangle")});async function HQ(t,e,{config:{themeVariables:r}}){let{labelStyles:n,nodeStyles:i}=Qe(e);e.labelStyle=n,e.useHtmlLabels||cr().flowchart?.htmlLabels!==!1||(e.centerLabel=!0);let{shapeSvg:s,bbox:l}=await pt(t,e,ht(e)),u=Math.max(l.width+(e.padding??0)*2,e?.width??0),h=Math.max(l.height+(e.padding??0)*2,e?.height??0),f=-u/2,d=-h/2,{cssStyles:p}=e,m=Xe.svg(s),g=Ke(e,{fill:r.noteBkgColor,stroke:r.noteBorderColor});e.look!=="handDrawn"&&(g.roughness=0,g.fillStyle="solid");let y=m.rectangle(f,d,u,h,g),v=s.insert(()=>y,":first-child");return v.attr("class","basic label-container"),p&&e.look!=="handDrawn"&&v.selectAll("path").attr("style",p),i&&e.look!=="handDrawn"&&v.selectAll("path").attr("style",i),je(e,v),e.intersect=function(x){return Ye.rect(e,x)},s}var WQ=N(()=>{"use strict";Wt();Ht();Ut();Ft();ji();o(HQ,"note")});async function qQ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a}=await pt(t,e,ht(e)),s=a.width+e.padding,l=a.height+e.padding,u=s+l,h=[{x:u/2,y:0},{x:u,y:-u/2},{x:u/2,y:-u},{x:0,y:-u/2}],f,{cssStyles:d}=e;if(e.look==="handDrawn"){let p=Xe.svg(i),m=Ke(e,{}),g=g_e(0,0,u),y=p.path(g,m);f=i.insert(()=>y,":first-child").attr("transform",`translate(${-u/2}, ${u/2})`),d&&f.attr("style",d)}else f=La(i,u,u,h);return n&&f.attr("style",n),je(e,f),e.intersect=function(p){return Y.debug(`APA12 Intersect called SPLIT +point:`,p,` +node: +`,e,` +res:`,Ye.polygon(e,h,p)),Ye.polygon(e,h,p)},i}var g_e,YQ=N(()=>{"use strict";vt();Ft();Ht();Ut();Wt();_u();g_e=o((t,e,r)=>[`M${t+r/2},${e}`,`L${t+r},${e-r/2}`,`L${t+r/2},${e-r}`,`L${t},${e-r/2}`,"Z"].join(" "),"createDecisionBoxPathD");o(qQ,"question")});async function XQ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a,label:s}=await pt(t,e,ht(e)),l=Math.max(a.width+(e.padding??0),e?.width??0),u=Math.max(a.height+(e.padding??0),e?.height??0),h=-l/2,f=-u/2,d=f/2,p=[{x:h+d,y:f},{x:h,y:0},{x:h+d,y:-f},{x:-h,y:-f},{x:-h,y:f}],{cssStyles:m}=e,g=Xe.svg(i),y=Ke(e,{});e.look!=="handDrawn"&&(y.roughness=0,y.fillStyle="solid");let v=Xt(p),x=g.path(v,y),b=i.insert(()=>x,":first-child");return b.attr("class","basic label-container"),m&&e.look!=="handDrawn"&&b.selectAll("path").attr("style",m),n&&e.look!=="handDrawn"&&b.selectAll("path").attr("style",n),b.attr("transform",`translate(${-d/2},0)`),s.attr("transform",`translate(${-d/2-a.width/2-(a.x-(a.left??0))}, ${-(a.height/2)-(a.y-(a.top??0))})`),je(e,b),e.intersect=function(w){return Ye.polygon(e,p,w)},i}var jQ=N(()=>{"use strict";Ft();Ht();Ut();Wt();o(XQ,"rect_left_inv_arrow")});function y_e(t,e){e&&t.attr("style",e)}async function v_e(t){let e=Ge(document.createElementNS("http://www.w3.org/2000/svg","foreignObject")),r=e.append("xhtml:div"),n=t.label;t.label&&pi(t.label)&&(n=await mh(t.label.replace(Ze.lineBreakRegex,` +`),me()));let i=t.isNode?"nodeLabel":"edgeLabel";return r.html('"+n+""),y_e(r,t.labelStyle),r.style("display","inline-block"),r.style("padding-right","1px"),r.style("white-space","nowrap"),r.attr("xmlns","http://www.w3.org/1999/xhtml"),e.node()}var x_e,gc,Gw=N(()=>{"use strict";dr();vt();zt();gr();ir();o(y_e,"applyStyle");o(v_e,"addHtmlLabel");x_e=o(async(t,e,r,n)=>{let i=t||"";if(typeof i=="object"&&(i=i[0]),fr(me().flowchart.htmlLabels)){i=i.replace(/\\n|\n/g,"
    "),Y.info("vertexText"+i);let a={isNode:n,label:na(i).replace(/fa[blrs]?:fa-[\w-]+/g,l=>``),labelStyle:e&&e.replace("fill:","color:")};return await v_e(a)}else{let a=document.createElementNS("http://www.w3.org/2000/svg","text");a.setAttribute("style",e.replace("color:","fill:"));let s=[];typeof i=="string"?s=i.split(/\\n|\n|/gi):Array.isArray(i)?s=i:s=[];for(let l of s){let u=document.createElementNS("http://www.w3.org/2000/svg","tspan");u.setAttributeNS("http://www.w3.org/XML/1998/namespace","xml:space","preserve"),u.setAttribute("dy","1em"),u.setAttribute("x","0"),r?u.setAttribute("class","title-row"):u.setAttribute("class","row"),u.textContent=l.trim(),a.appendChild(u)}return a}},"createLabel"),gc=x_e});async function KQ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let i;e.cssClasses?i="node "+e.cssClasses:i="node default";let a=t.insert("g").attr("class",i).attr("id",e.domId||e.id),s=a.insert("g"),l=a.insert("g").attr("class","label").attr("style",n),u=e.description,h=e.label,f=l.node().appendChild(await gc(h,e.labelStyle,!0,!0)),d={width:0,height:0};if(fr(me()?.flowchart?.htmlLabels)){let S=f.children[0],_=Ge(f);d=S.getBoundingClientRect(),_.attr("width",d.width),_.attr("height",d.height)}Y.info("Text 2",u);let p=u||[],m=f.getBBox(),g=l.node().appendChild(await gc(p.join?p.join("
    "):p,e.labelStyle,!0,!0)),y=g.children[0],v=Ge(g);d=y.getBoundingClientRect(),v.attr("width",d.width),v.attr("height",d.height);let x=(e.padding||0)/2;Ge(g).attr("transform","translate( "+(d.width>m.width?0:(m.width-d.width)/2)+", "+(m.height+x+5)+")"),Ge(f).attr("transform","translate( "+(d.width(Y.debug("Rough node insert CXC",I),D),":first-child"),E=a.insert(()=>(Y.debug("Rough node insert CXC",I),I),":first-child")}else E=s.insert("rect",":first-child"),A=s.insert("line"),E.attr("class","outer title-state").attr("style",n).attr("x",-d.width/2-x).attr("y",-d.height/2-x).attr("width",d.width+(e.padding||0)).attr("height",d.height+(e.padding||0)),A.attr("class","divider").attr("x1",-d.width/2-x).attr("x2",d.width/2+x).attr("y1",-d.height/2-x+m.height+x).attr("y2",-d.height/2-x+m.height+x);return je(e,E),e.intersect=function(S){return Ye.rect(e,S)},a}var QQ=N(()=>{"use strict";dr();gr();Ft();Gw();Ht();Ut();Wt();zt();qh();vt();o(KQ,"rectWithTitle")});async function ZQ(t,e){let r={rx:5,ry:5,classes:"",labelPaddingX:(e?.padding||0)*1,labelPaddingY:(e?.padding||0)*1};return Du(t,e,r)}var JQ=N(()=>{"use strict";mm();o(ZQ,"roundedRect")});async function eZ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a,label:s}=await pt(t,e,ht(e)),l=e?.padding??0,u=Math.max(a.width+(e.padding??0)*2,e?.width??0),h=Math.max(a.height+(e.padding??0)*2,e?.height??0),f=-a.width/2-l,d=-a.height/2-l,{cssStyles:p}=e,m=Xe.svg(i),g=Ke(e,{});e.look!=="handDrawn"&&(g.roughness=0,g.fillStyle="solid");let y=[{x:f,y:d},{x:f+u+8,y:d},{x:f+u+8,y:d+h},{x:f-8,y:d+h},{x:f-8,y:d},{x:f,y:d},{x:f,y:d+h}],v=m.polygon(y.map(b=>[b.x,b.y]),g),x=i.insert(()=>v,":first-child");return x.attr("class","basic label-container").attr("style",$n(p)),n&&e.look!=="handDrawn"&&x.selectAll("path").attr("style",n),p&&e.look!=="handDrawn"&&x.selectAll("path").attr("style",n),s.attr("transform",`translate(${-u/2+4+(e.padding??0)-(a.x-(a.left??0))},${-h/2+(e.padding??0)-(a.y-(a.top??0))})`),je(e,x),e.intersect=function(b){return Ye.rect(e,b)},i}var tZ=N(()=>{"use strict";Ft();Ht();Ut();Wt();ir();o(eZ,"shadedProcess")});async function rZ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a,label:s}=await pt(t,e,ht(e)),l=Math.max(a.width+(e.padding??0)*2,e?.width??0),u=Math.max(a.height+(e.padding??0)*2,e?.height??0),h=-l/2,f=-u/2,{cssStyles:d}=e,p=Xe.svg(i),m=Ke(e,{});e.look!=="handDrawn"&&(m.roughness=0,m.fillStyle="solid");let g=[{x:h,y:f},{x:h,y:f+u},{x:h+l,y:f+u},{x:h+l,y:f-u/2}],y=Xt(g),v=p.path(y,m),x=i.insert(()=>v,":first-child");return x.attr("class","basic label-container"),d&&e.look!=="handDrawn"&&x.selectChildren("path").attr("style",d),n&&e.look!=="handDrawn"&&x.selectChildren("path").attr("style",n),x.attr("transform",`translate(0, ${u/4})`),s.attr("transform",`translate(${-l/2+(e.padding??0)-(a.x-(a.left??0))}, ${-u/4+(e.padding??0)-(a.y-(a.top??0))})`),je(e,x),e.intersect=function(b){return Ye.polygon(e,g,b)},i}var nZ=N(()=>{"use strict";Ft();Ht();Ut();Wt();o(rZ,"slopedRect")});async function iZ(t,e){let r={rx:0,ry:0,classes:"",labelPaddingX:(e?.padding||0)*2,labelPaddingY:(e?.padding||0)*1};return Du(t,e,r)}var aZ=N(()=>{"use strict";mm();o(iZ,"squareRect")});async function sZ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a}=await pt(t,e,ht(e)),s=a.height+e.padding,l=a.width+s/4+e.padding,u,{cssStyles:h}=e;if(e.look==="handDrawn"){let f=Xe.svg(i),d=Ke(e,{}),p=Na(-l/2,-s/2,l,s,s/2),m=f.path(p,d);u=i.insert(()=>m,":first-child"),u.attr("class","basic label-container").attr("style",$n(h))}else u=i.insert("rect",":first-child"),u.attr("class","basic label-container").attr("style",n).attr("rx",s/2).attr("ry",s/2).attr("x",-l/2).attr("y",-s/2).attr("width",l).attr("height",s);return je(e,u),e.intersect=function(f){return Ye.rect(e,f)},i}var oZ=N(()=>{"use strict";Ft();Ht();Ut();Wt();qh();ir();o(sZ,"stadium")});async function lZ(t,e){return Du(t,e,{rx:5,ry:5,classes:"flowchart-node"})}var cZ=N(()=>{"use strict";mm();o(lZ,"state")});function uZ(t,e,{config:{themeVariables:r}}){let{labelStyles:n,nodeStyles:i}=Qe(e);e.labelStyle=n;let{cssStyles:a}=e,{lineColor:s,stateBorder:l,nodeBorder:u}=r,h=t.insert("g").attr("class","node default").attr("id",e.domId||e.id),f=Xe.svg(h),d=Ke(e,{});e.look!=="handDrawn"&&(d.roughness=0,d.fillStyle="solid");let p=f.circle(0,0,14,{...d,stroke:s,strokeWidth:2}),m=l??u,g=f.circle(0,0,5,{...d,fill:m,stroke:m,strokeWidth:2,fillStyle:"solid"}),y=h.insert(()=>p,":first-child");return y.insert(()=>g),a&&y.selectAll("path").attr("style",a),i&&y.selectAll("path").attr("style",i),je(e,y),e.intersect=function(v){return Ye.circle(e,7,v)},h}var hZ=N(()=>{"use strict";Wt();Ht();Ut();Ft();o(uZ,"stateEnd")});function fZ(t,e,{config:{themeVariables:r}}){let{lineColor:n}=r,i=t.insert("g").attr("class","node default").attr("id",e.domId||e.id),a;if(e.look==="handDrawn"){let l=Xe.svg(i).circle(0,0,14,gK(n));a=i.insert(()=>l),a.attr("class","state-start").attr("r",7).attr("width",14).attr("height",14)}else a=i.insert("circle",":first-child"),a.attr("class","state-start").attr("r",7).attr("width",14).attr("height",14);return je(e,a),e.intersect=function(s){return Ye.circle(e,7,s)},i}var dZ=N(()=>{"use strict";Wt();Ht();Ut();Ft();o(fZ,"stateStart")});async function pZ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a}=await pt(t,e,ht(e)),s=(e?.padding||0)/2,l=a.width+e.padding,u=a.height+e.padding,h=-a.width/2-s,f=-a.height/2-s,d=[{x:0,y:0},{x:l,y:0},{x:l,y:-u},{x:0,y:-u},{x:0,y:0},{x:-8,y:0},{x:l+8,y:0},{x:l+8,y:-u},{x:-8,y:-u},{x:-8,y:0}];if(e.look==="handDrawn"){let p=Xe.svg(i),m=Ke(e,{}),g=p.rectangle(h-8,f,l+16,u,m),y=p.line(h,f,h,f+u,m),v=p.line(h+l,f,h+l,f+u,m);i.insert(()=>y,":first-child"),i.insert(()=>v,":first-child");let x=i.insert(()=>g,":first-child"),{cssStyles:b}=e;x.attr("class","basic label-container").attr("style",$n(b)),je(e,x)}else{let p=La(i,l,u,d);n&&p.attr("style",n),je(e,p)}return e.intersect=function(p){return Ye.polygon(e,d,p)},i}var mZ=N(()=>{"use strict";Ft();Ht();Ut();Wt();_u();ir();o(pZ,"subroutine")});async function gZ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a}=await pt(t,e,ht(e)),s=Math.max(a.width+(e.padding??0)*2,e?.width??0),l=Math.max(a.height+(e.padding??0)*2,e?.height??0),u=-s/2,h=-l/2,f=.2*l,d=.2*l,{cssStyles:p}=e,m=Xe.svg(i),g=Ke(e,{}),y=[{x:u-f/2,y:h},{x:u+s+f/2,y:h},{x:u+s+f/2,y:h+l},{x:u-f/2,y:h+l}],v=[{x:u+s-f/2,y:h+l},{x:u+s+f/2,y:h+l},{x:u+s+f/2,y:h+l-d}];e.look!=="handDrawn"&&(g.roughness=0,g.fillStyle="solid");let x=Xt(y),b=m.path(x,g),w=Xt(v),C=m.path(w,{...g,fillStyle:"solid"}),T=i.insert(()=>C,":first-child");return T.insert(()=>b,":first-child"),T.attr("class","basic label-container"),p&&e.look!=="handDrawn"&&T.selectAll("path").attr("style",p),n&&e.look!=="handDrawn"&&T.selectAll("path").attr("style",n),je(e,T),e.intersect=function(E){return Ye.polygon(e,y,E)},i}var yZ=N(()=>{"use strict";Ft();Ut();Wt();Ht();o(gZ,"taggedRect")});async function vZ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a,label:s}=await pt(t,e,ht(e)),l=Math.max(a.width+(e.padding??0)*2,e?.width??0),u=Math.max(a.height+(e.padding??0)*2,e?.height??0),h=u/4,f=.2*l,d=.2*u,p=u+h,{cssStyles:m}=e,g=Xe.svg(i),y=Ke(e,{});e.look!=="handDrawn"&&(y.roughness=0,y.fillStyle="solid");let v=[{x:-l/2-l/2*.1,y:p/2},...Fo(-l/2-l/2*.1,p/2,l/2+l/2*.1,p/2,h,.8),{x:l/2+l/2*.1,y:-p/2},{x:-l/2-l/2*.1,y:-p/2}],x=-l/2+l/2*.1,b=-p/2-d*.4,w=[{x:x+l-f,y:(b+u)*1.4},{x:x+l,y:b+u-d},{x:x+l,y:(b+u)*.9},...Fo(x+l,(b+u)*1.3,x+l-f,(b+u)*1.5,-u*.03,.5)],C=Xt(v),T=g.path(C,y),E=Xt(w),A=g.path(E,{...y,fillStyle:"solid"}),S=i.insert(()=>A,":first-child");return S.insert(()=>T,":first-child"),S.attr("class","basic label-container"),m&&e.look!=="handDrawn"&&S.selectAll("path").attr("style",m),n&&e.look!=="handDrawn"&&S.selectAll("path").attr("style",n),S.attr("transform",`translate(0,${-h/2})`),s.attr("transform",`translate(${-l/2+(e.padding??0)-(a.x-(a.left??0))},${-u/2+(e.padding??0)-h/2-(a.y-(a.top??0))})`),je(e,S),e.intersect=function(_){return Ye.polygon(e,v,_)},i}var xZ=N(()=>{"use strict";Ft();Ht();Wt();Ut();o(vZ,"taggedWaveEdgedRectangle")});async function bZ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a}=await pt(t,e,ht(e)),s=Math.max(a.width+e.padding,e?.width||0),l=Math.max(a.height+e.padding,e?.height||0),u=-s/2,h=-l/2,f=i.insert("rect",":first-child");return f.attr("class","text").attr("style",n).attr("rx",0).attr("ry",0).attr("x",u).attr("y",h).attr("width",s).attr("height",l),je(e,f),e.intersect=function(d){return Ye.rect(e,d)},i}var wZ=N(()=>{"use strict";Ft();Ht();Ut();o(bZ,"text")});async function TZ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a,label:s,halfPadding:l}=await pt(t,e,ht(e)),u=e.look==="neo"?l*2:l,h=a.height+u,f=h/2,d=f/(2.5+h/50),p=a.width+d+u,{cssStyles:m}=e,g;if(e.look==="handDrawn"){let y=Xe.svg(i),v=w_e(0,0,p,h,d,f),x=T_e(0,0,p,h,d,f),b=y.path(v,Ke(e,{})),w=y.path(x,Ke(e,{fill:"none"}));g=i.insert(()=>w,":first-child"),g=i.insert(()=>b,":first-child"),g.attr("class","basic label-container"),m&&g.attr("style",m)}else{let y=b_e(0,0,p,h,d,f);g=i.insert("path",":first-child").attr("d",y).attr("class","basic label-container").attr("style",$n(m)).attr("style",n),g.attr("class","basic label-container"),m&&g.selectAll("path").attr("style",m),n&&g.selectAll("path").attr("style",n)}return g.attr("label-offset-x",d),g.attr("transform",`translate(${-p/2}, ${h/2} )`),s.attr("transform",`translate(${-(a.width/2)-d-(a.x-(a.left??0))}, ${-(a.height/2)-(a.y-(a.top??0))})`),je(e,g),e.intersect=function(y){let v=Ye.rect(e,y),x=v.y-(e.y??0);if(f!=0&&(Math.abs(x)<(e.height??0)/2||Math.abs(x)==(e.height??0)/2&&Math.abs(v.x-(e.x??0))>(e.width??0)/2-d)){let b=d*d*(1-x*x/(f*f));b!=0&&(b=Math.sqrt(Math.abs(b))),b=d-b,y.x-(e.x??0)>0&&(b=-b),v.x+=b}return v},i}var b_e,w_e,T_e,kZ=N(()=>{"use strict";Ft();Ut();Wt();Ht();ir();b_e=o((t,e,r,n,i,a)=>`M${t},${e} + a${i},${a} 0,0,1 0,${-n} + l${r},0 + a${i},${a} 0,0,1 0,${n} + M${r},${-n} + a${i},${a} 0,0,0 0,${n} + l${-r},0`,"createCylinderPathD"),w_e=o((t,e,r,n,i,a)=>[`M${t},${e}`,`M${t+r},${e}`,`a${i},${a} 0,0,0 0,${-n}`,`l${-r},0`,`a${i},${a} 0,0,0 0,${n}`,`l${r},0`].join(" "),"createOuterCylinderPathD"),T_e=o((t,e,r,n,i,a)=>[`M${t+r/2},${-n/2}`,`a${i},${a} 0,0,0 0,${n}`].join(" "),"createInnerCylinderPathD");o(TZ,"tiltedCylinder")});async function EZ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a}=await pt(t,e,ht(e)),s=a.width+e.padding,l=a.height+e.padding,u=[{x:-3*l/6,y:0},{x:s+3*l/6,y:0},{x:s,y:-l},{x:0,y:-l}],h,{cssStyles:f}=e;if(e.look==="handDrawn"){let d=Xe.svg(i),p=Ke(e,{}),m=Xt(u),g=d.path(m,p);h=i.insert(()=>g,":first-child").attr("transform",`translate(${-s/2}, ${l/2})`),f&&h.attr("style",f)}else h=La(i,s,l,u);return n&&h.attr("style",n),e.width=s,e.height=l,je(e,h),e.intersect=function(d){return Ye.polygon(e,u,d)},i}var SZ=N(()=>{"use strict";Ft();Ht();Ut();Wt();_u();o(EZ,"trapezoid")});async function CZ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a}=await pt(t,e,ht(e)),s=60,l=20,u=Math.max(s,a.width+(e.padding??0)*2,e?.width??0),h=Math.max(l,a.height+(e.padding??0)*2,e?.height??0),{cssStyles:f}=e,d=Xe.svg(i),p=Ke(e,{});e.look!=="handDrawn"&&(p.roughness=0,p.fillStyle="solid");let m=[{x:-u/2*.8,y:-h/2},{x:u/2*.8,y:-h/2},{x:u/2,y:-h/2*.6},{x:u/2,y:h/2},{x:-u/2,y:h/2},{x:-u/2,y:-h/2*.6}],g=Xt(m),y=d.path(g,p),v=i.insert(()=>y,":first-child");return v.attr("class","basic label-container"),f&&e.look!=="handDrawn"&&v.selectChildren("path").attr("style",f),n&&e.look!=="handDrawn"&&v.selectChildren("path").attr("style",n),je(e,v),e.intersect=function(x){return Ye.polygon(e,m,x)},i}var AZ=N(()=>{"use strict";Ft();Ht();Ut();Wt();o(CZ,"trapezoidalPentagon")});async function _Z(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a,label:s}=await pt(t,e,ht(e)),l=fr(me().flowchart?.htmlLabels),u=a.width+(e.padding??0),h=u+a.height,f=u+a.height,d=[{x:0,y:0},{x:f,y:0},{x:f/2,y:-h}],{cssStyles:p}=e,m=Xe.svg(i),g=Ke(e,{});e.look!=="handDrawn"&&(g.roughness=0,g.fillStyle="solid");let y=Xt(d),v=m.path(y,g),x=i.insert(()=>v,":first-child").attr("transform",`translate(${-h/2}, ${h/2})`);return p&&e.look!=="handDrawn"&&x.selectChildren("path").attr("style",p),n&&e.look!=="handDrawn"&&x.selectChildren("path").attr("style",n),e.width=u,e.height=h,je(e,x),s.attr("transform",`translate(${-a.width/2-(a.x-(a.left??0))}, ${h/2-(a.height+(e.padding??0)/(l?2:1)-(a.y-(a.top??0)))})`),e.intersect=function(b){return Y.info("Triangle intersect",e,d,b),Ye.polygon(e,d,b)},i}var DZ=N(()=>{"use strict";vt();Ft();Ht();Ut();Wt();Ft();gr();zt();o(_Z,"triangle")});async function LZ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a,label:s}=await pt(t,e,ht(e)),l=Math.max(a.width+(e.padding??0)*2,e?.width??0),u=Math.max(a.height+(e.padding??0)*2,e?.height??0),h=u/8,f=u+h,{cssStyles:d}=e,m=70-l,g=m>0?m/2:0,y=Xe.svg(i),v=Ke(e,{});e.look!=="handDrawn"&&(v.roughness=0,v.fillStyle="solid");let x=[{x:-l/2-g,y:f/2},...Fo(-l/2-g,f/2,l/2+g,f/2,h,.8),{x:l/2+g,y:-f/2},{x:-l/2-g,y:-f/2}],b=Xt(x),w=y.path(b,v),C=i.insert(()=>w,":first-child");return C.attr("class","basic label-container"),d&&e.look!=="handDrawn"&&C.selectAll("path").attr("style",d),n&&e.look!=="handDrawn"&&C.selectAll("path").attr("style",n),C.attr("transform",`translate(0,${-h/2})`),s.attr("transform",`translate(${-l/2+(e.padding??0)-(a.x-(a.left??0))},${-u/2+(e.padding??0)-h-(a.y-(a.top??0))})`),je(e,C),e.intersect=function(T){return Ye.polygon(e,x,T)},i}var RZ=N(()=>{"use strict";Ft();Ht();Wt();Ut();o(LZ,"waveEdgedRectangle")});async function NZ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a}=await pt(t,e,ht(e)),s=100,l=50,u=Math.max(a.width+(e.padding??0)*2,e?.width??0),h=Math.max(a.height+(e.padding??0)*2,e?.height??0),f=u/h,d=u,p=h;d>p*f?p=d/f:d=p*f,d=Math.max(d,s),p=Math.max(p,l);let m=Math.min(p*.2,p/4),g=p+m*2,{cssStyles:y}=e,v=Xe.svg(i),x=Ke(e,{});e.look!=="handDrawn"&&(x.roughness=0,x.fillStyle="solid");let b=[{x:-d/2,y:g/2},...Fo(-d/2,g/2,d/2,g/2,m,1),{x:d/2,y:-g/2},...Fo(d/2,-g/2,-d/2,-g/2,m,-1)],w=Xt(b),C=v.path(w,x),T=i.insert(()=>C,":first-child");return T.attr("class","basic label-container"),y&&e.look!=="handDrawn"&&T.selectAll("path").attr("style",y),n&&e.look!=="handDrawn"&&T.selectAll("path").attr("style",n),je(e,T),e.intersect=function(E){return Ye.polygon(e,b,E)},i}var MZ=N(()=>{"use strict";Ft();Ht();Ut();Wt();o(NZ,"waveRectangle")});async function IZ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let{shapeSvg:i,bbox:a,label:s}=await pt(t,e,ht(e)),l=Math.max(a.width+(e.padding??0)*2,e?.width??0),u=Math.max(a.height+(e.padding??0)*2,e?.height??0),h=5,f=-l/2,d=-u/2,{cssStyles:p}=e,m=Xe.svg(i),g=Ke(e,{}),y=[{x:f-h,y:d-h},{x:f-h,y:d+u},{x:f+l,y:d+u},{x:f+l,y:d-h}],v=`M${f-h},${d-h} L${f+l},${d-h} L${f+l},${d+u} L${f-h},${d+u} L${f-h},${d-h} + M${f-h},${d} L${f+l},${d} + M${f},${d-h} L${f},${d+u}`;e.look!=="handDrawn"&&(g.roughness=0,g.fillStyle="solid");let x=m.path(v,g),b=i.insert(()=>x,":first-child");return b.attr("transform",`translate(${h/2}, ${h/2})`),b.attr("class","basic label-container"),p&&e.look!=="handDrawn"&&b.selectAll("path").attr("style",p),n&&e.look!=="handDrawn"&&b.selectAll("path").attr("style",n),s.attr("transform",`translate(${-(a.width/2)+h/2-(a.x-(a.left??0))}, ${-(a.height/2)+h/2-(a.y-(a.top??0))})`),je(e,b),e.intersect=function(w){return Ye.polygon(e,y,w)},i}var OZ=N(()=>{"use strict";Ft();Ut();Wt();Ht();o(IZ,"windowPane")});async function KD(t,e){let r=e;if(r.alias&&(e.label=r.alias),e.look==="handDrawn"){let{themeVariables:P}=cr(),{background:z}=P,$={...e,id:e.id+"-background",look:"default",cssStyles:["stroke: none",`fill: ${z}`]};await KD(t,$)}let n=cr();e.useHtmlLabels=n.htmlLabels;let i=n.er?.diagramPadding??10,a=n.er?.entityPadding??6,{cssStyles:s}=e,{labelStyles:l}=Qe(e);if(r.attributes.length===0&&e.label){let P={rx:0,ry:0,labelPaddingX:i,labelPaddingY:i*1.5,classes:""};ra(e.label,n)+P.labelPaddingX*20){let P=f.width+i*2-(m+g+y+v);m+=P/w,g+=P/w,y>0&&(y+=P/w),v>0&&(v+=P/w)}let T=m+g+y+v,E=Xe.svg(h),A=Ke(e,{});e.look!=="handDrawn"&&(A.roughness=0,A.fillStyle="solid");let S=Math.max(C.width+i*2,e?.width||0,T),_=Math.max(C.height+(p[0]||d)+a,e?.height||0),I=-S/2,D=-_/2;h.selectAll("g:not(:first-child)").each((P,z,$)=>{let H=Ge($[z]),Q=H.attr("transform"),j=0,ie=0;if(Q){let le=RegExp(/translate\(([^,]+),([^)]+)\)/).exec(Q);le&&(j=parseFloat(le[1]),ie=parseFloat(le[2]),H.attr("class").includes("attribute-name")?j+=m:H.attr("class").includes("attribute-keys")?j+=m+g:H.attr("class").includes("attribute-comment")&&(j+=m+g+y))}H.attr("transform",`translate(${I+i/2+j}, ${ie+D+f.height+a/2})`)}),h.select(".name").attr("transform","translate("+-f.width/2+", "+(D+a/2)+")");let k=E.rectangle(I,D,S,_,A),L=h.insert(()=>k,":first-child").attr("style",s.join("")),{themeVariables:R}=cr(),{rowEven:O,rowOdd:M,nodeBorder:B}=R;p.push(0);for(let[P,z]of p.entries()){if(P===0&&p.length>1)continue;let $=P%2===0&&z!==0,H=E.rectangle(I,f.height+D+z,S,f.height,{...A,fill:$?O:M,stroke:B});h.insert(()=>H,"g.label").attr("style",s.join("")).attr("class",`row-rect-${P%2===0?"even":"odd"}`)}let F=E.line(I,f.height+D,S+I,f.height+D,A);h.insert(()=>F).attr("class","divider"),F=E.line(m+I,f.height+D,m+I,_+D,A),h.insert(()=>F).attr("class","divider"),x&&(F=E.line(m+g+I,f.height+D,m+g+I,_+D,A),h.insert(()=>F).attr("class","divider")),b&&(F=E.line(m+g+y+I,f.height+D,m+g+y+I,_+D,A),h.insert(()=>F).attr("class","divider"));for(let P of p)F=E.line(I,f.height+D+P,S+I,f.height+D+P,A),h.insert(()=>F).attr("class","divider");return je(e,L),e.intersect=function(P){return Ye.rect(e,P)},h}async function b2(t,e,r,n=0,i=0,a=[],s=""){let l=t.insert("g").attr("class",`label ${a.join(" ")}`).attr("transform",`translate(${n}, ${i})`).attr("style",s);e!==ec(e)&&(e=ec(e),e=e.replaceAll("<","<").replaceAll(">",">"));let u=l.node().appendChild(await Hn(l,e,{width:ra(e,r)+100,style:s,useHtmlLabels:r.htmlLabels},r));if(e.includes("<")||e.includes(">")){let f=u.children[0];for(f.textContent=f.textContent.replaceAll("<","<").replaceAll(">",">");f.childNodes[0];)f=f.childNodes[0],f.textContent=f.textContent.replaceAll("<","<").replaceAll(">",">")}let h=u.getBBox();if(fr(r.htmlLabels)){let f=u.children[0];f.style.textAlign="start";let d=Ge(u);h=f.getBoundingClientRect(),d.attr("width",h.width),d.attr("height",h.height)}return h}var PZ=N(()=>{"use strict";Ft();Ht();Ut();Wt();mm();ji();to();gr();dr();ir();o(KD,"erBox");o(b2,"addText")});async function BZ(t,e,r,n,i=r.class.padding??12){let a=n?0:3,s=t.insert("g").attr("class",ht(e)).attr("id",e.domId||e.id),l=null,u=null,h=null,f=null,d=0,p=0,m=0;if(l=s.insert("g").attr("class","annotation-group text"),e.annotations.length>0){let b=e.annotations[0];await Vw(l,{text:`\xAB${b}\xBB`},0),d=l.node().getBBox().height}u=s.insert("g").attr("class","label-group text"),await Vw(u,e,0,["font-weight: bolder"]);let g=u.node().getBBox();p=g.height,h=s.insert("g").attr("class","members-group text");let y=0;for(let b of e.members){let w=await Vw(h,b,y,[b.parseClassifier()]);y+=w+a}m=h.node().getBBox().height,m<=0&&(m=i/2),f=s.insert("g").attr("class","methods-group text");let v=0;for(let b of e.methods){let w=await Vw(f,b,v,[b.parseClassifier()]);v+=w+a}let x=s.node().getBBox();if(l!==null){let b=l.node().getBBox();l.attr("transform",`translate(${-b.width/2})`)}return u.attr("transform",`translate(${-g.width/2}, ${d})`),x=s.node().getBBox(),h.attr("transform",`translate(0, ${d+p+i*2})`),x=s.node().getBBox(),f.attr("transform",`translate(0, ${d+p+(m?m+i*4:i*2)})`),x=s.node().getBBox(),{shapeSvg:s,bbox:x}}async function Vw(t,e,r,n=[]){let i=t.insert("g").attr("class","label").attr("style",n.join("; ")),a=cr(),s="useHtmlLabels"in e?e.useHtmlLabels:fr(a.htmlLabels)??!0,l="";"text"in e?l=e.text:l=e.label,!s&&l.startsWith("\\")&&(l=l.substring(1)),pi(l)&&(s=!0);let u=await Hn(i,Xy(na(l)),{width:ra(l,a)+50,classes:"markdown-node-label",useHtmlLabels:s},a),h,f=1;if(s){let d=u.children[0],p=Ge(u);f=d.innerHTML.split("
    ").length,d.innerHTML.includes("")&&(f+=d.innerHTML.split("").length-1);let m=d.getElementsByTagName("img");if(m){let g=l.replace(/]*>/g,"").trim()==="";await Promise.all([...m].map(y=>new Promise(v=>{function x(){if(y.style.display="flex",y.style.flexDirection="column",g){let b=a.fontSize?.toString()??window.getComputedStyle(document.body).fontSize,C=parseInt(b,10)*5+"px";y.style.minWidth=C,y.style.maxWidth=C}else y.style.width="100%";v(y)}o(x,"setupImage"),setTimeout(()=>{y.complete&&x()}),y.addEventListener("error",x),y.addEventListener("load",x)})))}h=d.getBoundingClientRect(),p.attr("width",h.width),p.attr("height",h.height)}else{n.includes("font-weight: bolder")&&Ge(u).selectAll("tspan").attr("font-weight",""),f=u.children.length;let d=u.children[0];(u.textContent===""||u.textContent.includes(">"))&&(d.textContent=l[0]+l.substring(1).replaceAll(">",">").replaceAll("<","<").trim(),l[1]===" "&&(d.textContent=d.textContent[0]+" "+d.textContent.substring(1))),d.textContent==="undefined"&&(d.textContent=""),h=u.getBBox()}return i.attr("transform","translate(0,"+(-h.height/(2*f)+r)+")"),h.height}var FZ=N(()=>{"use strict";dr();ji();Ft();ir();zt();to();gr();o(BZ,"textHelper");o(Vw,"addText")});async function $Z(t,e){let r=me(),n=r.class.padding??12,i=n,a=e.useHtmlLabels??fr(r.htmlLabels)??!0,s=e;s.annotations=s.annotations??[],s.members=s.members??[],s.methods=s.methods??[];let{shapeSvg:l,bbox:u}=await BZ(t,e,r,a,i),{labelStyles:h,nodeStyles:f}=Qe(e);e.labelStyle=h,e.cssStyles=s.styles||"";let d=s.styles?.join(";")||f||"";e.cssStyles||(e.cssStyles=d.replaceAll("!important","").split(";"));let p=s.members.length===0&&s.methods.length===0&&!r.class?.hideEmptyMembersBox,m=Xe.svg(l),g=Ke(e,{});e.look!=="handDrawn"&&(g.roughness=0,g.fillStyle="solid");let y=u.width,v=u.height;s.members.length===0&&s.methods.length===0?v+=i:s.members.length>0&&s.methods.length===0&&(v+=i*2);let x=-y/2,b=-v/2,w=m.rectangle(x-n,b-n-(p?n:s.members.length===0&&s.methods.length===0?-n/2:0),y+2*n,v+2*n+(p?n*2:s.members.length===0&&s.methods.length===0?-n:0),g),C=l.insert(()=>w,":first-child");C.attr("class","basic label-container");let T=C.node().getBBox();l.selectAll(".text").each((_,I,D)=>{let k=Ge(D[I]),L=k.attr("transform"),R=0;if(L){let F=RegExp(/translate\(([^,]+),([^)]+)\)/).exec(L);F&&(R=parseFloat(F[2]))}let O=R+b+n-(p?n:s.members.length===0&&s.methods.length===0?-n/2:0);a||(O-=4);let M=x;(k.attr("class").includes("label-group")||k.attr("class").includes("annotation-group"))&&(M=-k.node()?.getBBox().width/2||0,l.selectAll("text").each(function(B,F,P){window.getComputedStyle(P[F]).textAnchor==="middle"&&(M=0)})),k.attr("transform",`translate(${M}, ${O})`)});let E=l.select(".annotation-group").node().getBBox().height-(p?n/2:0)||0,A=l.select(".label-group").node().getBBox().height-(p?n/2:0)||0,S=l.select(".members-group").node().getBBox().height-(p?n/2:0)||0;if(s.members.length>0||s.methods.length>0||p){let _=m.line(T.x,E+A+b+n,T.x+T.width,E+A+b+n,g);l.insert(()=>_).attr("class","divider").attr("style",d)}if(p||s.members.length>0||s.methods.length>0){let _=m.line(T.x,E+A+S+b+i*2+n,T.x+T.width,E+A+S+b+n+i*2,g);l.insert(()=>_).attr("class","divider").attr("style",d)}if(s.look!=="handDrawn"&&l.selectAll("path").attr("style",d),C.select(":nth-child(2)").attr("style",d),l.selectAll(".divider").select("path").attr("style",d),e.labelStyle?l.selectAll("span").attr("style",e.labelStyle):l.selectAll("span").attr("style",d),!a){let _=RegExp(/color\s*:\s*([^;]*)/),I=_.exec(d);if(I){let D=I[0].replace("color","fill");l.selectAll("tspan").attr("style",D)}else if(h){let D=_.exec(h);if(D){let k=D[0].replace("color","fill");l.selectAll("tspan").attr("style",k)}}}return je(e,C),e.intersect=function(_){return Ye.rect(e,_)},l}var zZ=N(()=>{"use strict";Ft();zt();dr();Wt();Ut();Ht();FZ();gr();o($Z,"classBox")});async function GZ(t,e){let{labelStyles:r,nodeStyles:n}=Qe(e);e.labelStyle=r;let i=e,a=e,s=20,l=20,u="verifyMethod"in e,h=ht(e),f=t.insert("g").attr("class",h).attr("id",e.domId??e.id),d;u?d=await Lu(f,`<<${i.type}>>`,0,e.labelStyle):d=await Lu(f,"<<Element>>",0,e.labelStyle);let p=d,m=await Lu(f,i.name,p,e.labelStyle+"; font-weight: bold;");if(p+=m+l,u){let E=await Lu(f,`${i.requirementId?`Id: ${i.requirementId}`:""}`,p,e.labelStyle);p+=E;let A=await Lu(f,`${i.text?`Text: ${i.text}`:""}`,p,e.labelStyle);p+=A;let S=await Lu(f,`${i.risk?`Risk: ${i.risk}`:""}`,p,e.labelStyle);p+=S,await Lu(f,`${i.verifyMethod?`Verification: ${i.verifyMethod}`:""}`,p,e.labelStyle)}else{let E=await Lu(f,`${a.type?`Type: ${a.type}`:""}`,p,e.labelStyle);p+=E,await Lu(f,`${a.docRef?`Doc Ref: ${a.docRef}`:""}`,p,e.labelStyle)}let g=(f.node()?.getBBox().width??200)+s,y=(f.node()?.getBBox().height??200)+s,v=-g/2,x=-y/2,b=Xe.svg(f),w=Ke(e,{});e.look!=="handDrawn"&&(w.roughness=0,w.fillStyle="solid");let C=b.rectangle(v,x,g,y,w),T=f.insert(()=>C,":first-child");if(T.attr("class","basic label-container").attr("style",n),f.selectAll(".label").each((E,A,S)=>{let _=Ge(S[A]),I=_.attr("transform"),D=0,k=0;if(I){let M=RegExp(/translate\(([^,]+),([^)]+)\)/).exec(I);M&&(D=parseFloat(M[1]),k=parseFloat(M[2]))}let L=k-y/2,R=v+s/2;(A===0||A===1)&&(R=D),_.attr("transform",`translate(${R}, ${L+s})`)}),p>d+m+l){let E=b.line(v,x+d+m+l,v+g,x+d+m+l,w);f.insert(()=>E).attr("style",n)}return je(e,T),e.intersect=function(E){return Ye.rect(e,E)},f}async function Lu(t,e,r,n=""){if(e==="")return 0;let i=t.insert("g").attr("class","label").attr("style",n),a=me(),s=a.htmlLabels??!0,l=await Hn(i,Xy(na(e)),{width:ra(e,a)+50,classes:"markdown-node-label",useHtmlLabels:s,style:n},a),u;if(s){let h=l.children[0],f=Ge(l);u=h.getBoundingClientRect(),f.attr("width",u.width),f.attr("height",u.height)}else{let h=l.children[0];for(let f of h.children)f.textContent=f.textContent.replaceAll(">",">").replaceAll("<","<"),n&&f.setAttribute("style",n);u=l.getBBox(),u.height+=6}return i.attr("transform",`translate(${-u.width/2},${-u.height/2+r})`),u.height}var VZ=N(()=>{"use strict";Ft();Ht();Ut();Wt();ir();zt();to();dr();o(GZ,"requirementBox");o(Lu,"addText")});async function UZ(t,e,{config:r}){let{labelStyles:n,nodeStyles:i}=Qe(e);e.labelStyle=n||"";let a=10,s=e.width;e.width=(e.width??200)-10;let{shapeSvg:l,bbox:u,label:h}=await pt(t,e,ht(e)),f=e.padding||10,d="",p;"ticket"in e&&e.ticket&&r?.kanban?.ticketBaseUrl&&(d=r?.kanban?.ticketBaseUrl.replace("#TICKET#",e.ticket),p=l.insert("svg:a",":first-child").attr("class","kanban-ticket-link").attr("xlink:href",d).attr("target","_blank"));let m={useHtmlLabels:e.useHtmlLabels,labelStyle:e.labelStyle||"",width:e.width,img:e.img,padding:e.padding||8,centerLabel:!1},g,y;p?{label:g,bbox:y}=await Dw(p,"ticket"in e&&e.ticket||"",m):{label:g,bbox:y}=await Dw(l,"ticket"in e&&e.ticket||"",m);let{label:v,bbox:x}=await Dw(l,"assigned"in e&&e.assigned||"",m);e.width=s;let b=10,w=e?.width||0,C=Math.max(y.height,x.height)/2,T=Math.max(u.height+b*2,e?.height||0)+C,E=-w/2,A=-T/2;h.attr("transform","translate("+(f-w/2)+", "+(-C-u.height/2)+")"),g.attr("transform","translate("+(f-w/2)+", "+(-C+u.height/2)+")"),v.attr("transform","translate("+(f+w/2-x.width-2*a)+", "+(-C+u.height/2)+")");let S,{rx:_,ry:I}=e,{cssStyles:D}=e;if(e.look==="handDrawn"){let k=Xe.svg(l),L=Ke(e,{}),R=_||I?k.path(Na(E,A,w,T,_||0),L):k.rectangle(E,A,w,T,L);S=l.insert(()=>R,":first-child"),S.attr("class","basic label-container").attr("style",D||null)}else{S=l.insert("rect",":first-child"),S.attr("class","basic label-container __APA__").attr("style",i).attr("rx",_??5).attr("ry",I??5).attr("x",E).attr("y",A).attr("width",w).attr("height",T);let k="priority"in e&&e.priority;if(k){let L=l.append("line"),R=E+2,O=A+Math.floor((_??0)/2),M=A+T-Math.floor((_??0)/2);L.attr("x1",R).attr("y1",O).attr("x2",R).attr("y2",M).attr("stroke-width","4").attr("stroke",k_e(k))}}return je(e,S),e.height=T,e.intersect=function(k){return Ye.rect(e,k)},l}var k_e,HZ=N(()=>{"use strict";Ft();Ht();qh();Ut();Wt();k_e=o(t=>{switch(t){case"Very High":return"red";case"High":return"orange";case"Medium":return null;case"Low":return"blue";case"Very Low":return"lightblue"}},"colorFromPriority");o(UZ,"kanbanItem")});function WZ(t){return t in QD}var E_e,S_e,QD,ZD=N(()=>{"use strict";NK();OK();BK();$K();GK();UK();WK();YK();jK();QK();JK();tQ();nQ();aQ();oQ();cQ();hQ();dQ();mQ();yQ();xQ();wQ();kQ();SQ();AQ();DQ();RQ();MQ();OQ();BQ();$Q();GQ();UQ();WQ();YQ();jQ();QQ();JQ();tZ();nZ();aZ();oZ();cZ();hZ();dZ();mZ();yZ();xZ();wZ();kZ();SZ();AZ();DZ();RZ();MZ();OZ();PZ();zZ();VZ();HZ();E_e=[{semanticName:"Process",name:"Rectangle",shortName:"rect",description:"Standard process shape",aliases:["proc","process","rectangle"],internalAliases:["squareRect"],handler:iZ},{semanticName:"Event",name:"Rounded Rectangle",shortName:"rounded",description:"Represents an event",aliases:["event"],internalAliases:["roundedRect"],handler:ZQ},{semanticName:"Terminal Point",name:"Stadium",shortName:"stadium",description:"Terminal point",aliases:["terminal","pill"],handler:sZ},{semanticName:"Subprocess",name:"Framed Rectangle",shortName:"fr-rect",description:"Subprocess",aliases:["subprocess","subproc","framed-rectangle","subroutine"],handler:pZ},{semanticName:"Database",name:"Cylinder",shortName:"cyl",description:"Database storage",aliases:["db","database","cylinder"],handler:ZK},{semanticName:"Start",name:"Circle",shortName:"circle",description:"Starting point",aliases:["circ"],handler:zK},{semanticName:"Decision",name:"Diamond",shortName:"diam",description:"Decision-making step",aliases:["decision","diamond","question"],handler:qQ},{semanticName:"Prepare Conditional",name:"Hexagon",shortName:"hex",description:"Preparation or condition step",aliases:["hexagon","prepare"],handler:fQ},{semanticName:"Data Input/Output",name:"Lean Right",shortName:"lean-r",description:"Represents input or output",aliases:["lean-right","in-out"],internalAliases:["lean_right"],handler:NQ},{semanticName:"Data Input/Output",name:"Lean Left",shortName:"lean-l",description:"Represents output or input",aliases:["lean-left","out-in"],internalAliases:["lean_left"],handler:LQ},{semanticName:"Priority Action",name:"Trapezoid Base Bottom",shortName:"trap-b",description:"Priority action",aliases:["priority","trapezoid-bottom","trapezoid"],handler:EZ},{semanticName:"Manual Operation",name:"Trapezoid Base Top",shortName:"trap-t",description:"Represents a manual task",aliases:["manual","trapezoid-top","inv-trapezoid"],internalAliases:["inv_trapezoid"],handler:CQ},{semanticName:"Stop",name:"Double Circle",shortName:"dbl-circ",description:"Represents a stop point",aliases:["double-circle"],internalAliases:["doublecircle"],handler:rQ},{semanticName:"Text Block",name:"Text Block",shortName:"text",description:"Text block",handler:bZ},{semanticName:"Card",name:"Notched Rectangle",shortName:"notch-rect",description:"Represents a card",aliases:["card","notched-rectangle"],handler:PK},{semanticName:"Lined/Shaded Process",name:"Lined Rectangle",shortName:"lin-rect",description:"Lined process shape",aliases:["lined-rectangle","lined-process","lin-proc","shaded-process"],handler:eZ},{semanticName:"Start",name:"Small Circle",shortName:"sm-circ",description:"Small starting point",aliases:["start","small-circle"],internalAliases:["stateStart"],handler:fZ},{semanticName:"Stop",name:"Framed Circle",shortName:"fr-circ",description:"Stop point",aliases:["stop","framed-circle"],internalAliases:["stateEnd"],handler:uZ},{semanticName:"Fork/Join",name:"Filled Rectangle",shortName:"fork",description:"Fork or join in process flow",aliases:["join"],internalAliases:["forkJoin"],handler:lQ},{semanticName:"Collate",name:"Hourglass",shortName:"hourglass",description:"Represents a collate operation",aliases:["hourglass","collate"],handler:pQ},{semanticName:"Comment",name:"Curly Brace",shortName:"brace",description:"Adds a comment",aliases:["comment","brace-l"],handler:HK},{semanticName:"Comment Right",name:"Curly Brace",shortName:"brace-r",description:"Adds a comment",handler:qK},{semanticName:"Comment with braces on both sides",name:"Curly Braces",shortName:"braces",description:"Adds a comment",handler:XK},{semanticName:"Com Link",name:"Lightning Bolt",shortName:"bolt",description:"Communication link",aliases:["com-link","lightning-bolt"],handler:IQ},{semanticName:"Document",name:"Document",shortName:"doc",description:"Represents a document",aliases:["doc","document"],handler:LZ},{semanticName:"Delay",name:"Half-Rounded Rectangle",shortName:"delay",description:"Represents a delay",aliases:["half-rounded-rectangle"],handler:uQ},{semanticName:"Direct Access Storage",name:"Horizontal Cylinder",shortName:"h-cyl",description:"Direct access storage",aliases:["das","horizontal-cylinder"],handler:TZ},{semanticName:"Disk Storage",name:"Lined Cylinder",shortName:"lin-cyl",description:"Disk storage",aliases:["disk","lined-cylinder"],handler:PQ},{semanticName:"Display",name:"Curved Trapezoid",shortName:"curv-trap",description:"Represents a display",aliases:["curved-trapezoid","display"],handler:KK},{semanticName:"Divided Process",name:"Divided Rectangle",shortName:"div-rect",description:"Divided process shape",aliases:["div-proc","divided-rectangle","divided-process"],handler:eQ},{semanticName:"Extract",name:"Triangle",shortName:"tri",description:"Extraction process",aliases:["extract","triangle"],handler:_Z},{semanticName:"Internal Storage",name:"Window Pane",shortName:"win-pane",description:"Internal storage",aliases:["internal-storage","window-pane"],handler:IZ},{semanticName:"Junction",name:"Filled Circle",shortName:"f-circ",description:"Junction point",aliases:["junction","filled-circle"],handler:iQ},{semanticName:"Loop Limit",name:"Trapezoidal Pentagon",shortName:"notch-pent",description:"Loop limit step",aliases:["loop-limit","notched-pentagon"],handler:CZ},{semanticName:"Manual File",name:"Flipped Triangle",shortName:"flip-tri",description:"Manual file operation",aliases:["manual-file","flipped-triangle"],handler:sQ},{semanticName:"Manual Input",name:"Sloped Rectangle",shortName:"sl-rect",description:"Manual input step",aliases:["manual-input","sloped-rectangle"],handler:rZ},{semanticName:"Multi-Document",name:"Stacked Document",shortName:"docs",description:"Multiple documents",aliases:["documents","st-doc","stacked-document"],handler:VQ},{semanticName:"Multi-Process",name:"Stacked Rectangle",shortName:"st-rect",description:"Multiple processes",aliases:["procs","processes","stacked-rectangle"],handler:zQ},{semanticName:"Stored Data",name:"Bow Tie Rectangle",shortName:"bow-rect",description:"Stored data",aliases:["stored-data","bow-tie-rectangle"],handler:IK},{semanticName:"Summary",name:"Crossed Circle",shortName:"cross-circ",description:"Summary",aliases:["summary","crossed-circle"],handler:VK},{semanticName:"Tagged Document",name:"Tagged Document",shortName:"tag-doc",description:"Tagged document",aliases:["tag-doc","tagged-document"],handler:vZ},{semanticName:"Tagged Process",name:"Tagged Rectangle",shortName:"tag-rect",description:"Tagged process",aliases:["tagged-rectangle","tag-proc","tagged-process"],handler:gZ},{semanticName:"Paper Tape",name:"Flag",shortName:"flag",description:"Paper tape",aliases:["paper-tape"],handler:NZ},{semanticName:"Odd",name:"Odd",shortName:"odd",description:"Odd shape",internalAliases:["rect_left_inv_arrow"],handler:XQ},{semanticName:"Lined Document",name:"Lined Document",shortName:"lin-doc",description:"Lined document",aliases:["lined-document"],handler:FQ}],S_e=o(()=>{let e=[...Object.entries({state:lZ,choice:FK,note:HQ,rectWithTitle:KQ,labelRect:_Q,iconSquare:TQ,iconCircle:vQ,icon:gQ,iconRounded:bQ,imageSquare:EQ,anchor:RK,kanbanItem:UZ,classBox:$Z,erBox:KD,requirementBox:GZ}),...E_e.flatMap(r=>[r.shortName,..."aliases"in r?r.aliases:[],..."internalAliases"in r?r.internalAliases:[]].map(i=>[i,r.handler]))];return Object.fromEntries(e)},"generateShapeMap"),QD=S_e();o(WZ,"isValidShape")});var C_e,Uw,qZ=N(()=>{"use strict";dr();Ew();zt();vt();ZD();ir();gr();mi();C_e="flowchart-",Uw=class{constructor(){this.vertexCounter=0;this.config=me();this.vertices=new Map;this.edges=[];this.classes=new Map;this.subGraphs=[];this.subGraphLookup=new Map;this.tooltips=new Map;this.subCount=0;this.firstGraphFlag=!0;this.secCount=-1;this.posCrossRef=[];this.funs=[];this.setAccTitle=Lr;this.setAccDescription=Nr;this.setDiagramTitle=$r;this.getAccTitle=Rr;this.getAccDescription=Mr;this.getDiagramTitle=Ir;this.funs.push(this.setupToolTips.bind(this)),this.addVertex=this.addVertex.bind(this),this.firstGraph=this.firstGraph.bind(this),this.setDirection=this.setDirection.bind(this),this.addSubGraph=this.addSubGraph.bind(this),this.addLink=this.addLink.bind(this),this.setLink=this.setLink.bind(this),this.updateLink=this.updateLink.bind(this),this.addClass=this.addClass.bind(this),this.setClass=this.setClass.bind(this),this.destructLink=this.destructLink.bind(this),this.setClickEvent=this.setClickEvent.bind(this),this.setTooltip=this.setTooltip.bind(this),this.updateLinkInterpolate=this.updateLinkInterpolate.bind(this),this.setClickFun=this.setClickFun.bind(this),this.bindFunctions=this.bindFunctions.bind(this),this.lex={firstGraph:this.firstGraph.bind(this)},this.clear(),this.setGen("gen-2")}static{o(this,"FlowDB")}sanitizeText(e){return Ze.sanitizeText(e,this.config)}lookUpDomId(e){for(let r of this.vertices.values())if(r.id===e)return r.domId;return e}addVertex(e,r,n,i,a,s,l={},u){if(!e||e.trim().length===0)return;let h;if(u!==void 0){let m;u.includes(` +`)?m=u+` +`:m=`{ +`+u+` +}`,h=cm(m,{schema:lm})}let f=this.edges.find(m=>m.id===e);if(f){let m=h;m?.animate!==void 0&&(f.animate=m.animate),m?.animation!==void 0&&(f.animation=m.animation);return}let d,p=this.vertices.get(e);if(p===void 0&&(p={id:e,labelType:"text",domId:C_e+e+"-"+this.vertexCounter,styles:[],classes:[]},this.vertices.set(e,p)),this.vertexCounter++,r!==void 0?(this.config=me(),d=this.sanitizeText(r.text.trim()),p.labelType=r.type,d.startsWith('"')&&d.endsWith('"')&&(d=d.substring(1,d.length-1)),p.text=d):p.text===void 0&&(p.text=e),n!==void 0&&(p.type=n),i?.forEach(m=>{p.styles.push(m)}),a?.forEach(m=>{p.classes.push(m)}),s!==void 0&&(p.dir=s),p.props===void 0?p.props=l:l!==void 0&&Object.assign(p.props,l),h!==void 0){if(h.shape){if(h.shape!==h.shape.toLowerCase()||h.shape.includes("_"))throw new Error(`No such shape: ${h.shape}. Shape names should be lowercase.`);if(!WZ(h.shape))throw new Error(`No such shape: ${h.shape}.`);p.type=h?.shape}h?.label&&(p.text=h?.label),h?.icon&&(p.icon=h?.icon,!h.label?.trim()&&p.text===e&&(p.text="")),h?.form&&(p.form=h?.form),h?.pos&&(p.pos=h?.pos),h?.img&&(p.img=h?.img,!h.label?.trim()&&p.text===e&&(p.text="")),h?.constraint&&(p.constraint=h.constraint),h.w&&(p.assetWidth=Number(h.w)),h.h&&(p.assetHeight=Number(h.h))}}addSingleLink(e,r,n,i){let l={start:e,end:r,type:void 0,text:"",labelType:"text",classes:[],isUserDefinedId:!1,interpolate:this.edges.defaultInterpolate};Y.info("abc78 Got edge...",l);let u=n.text;if(u!==void 0&&(l.text=this.sanitizeText(u.text.trim()),l.text.startsWith('"')&&l.text.endsWith('"')&&(l.text=l.text.substring(1,l.text.length-1)),l.labelType=u.type),n!==void 0&&(l.type=n.type,l.stroke=n.stroke,l.length=n.length>10?10:n.length),i&&!this.edges.some(h=>h.id===i))l.id=i,l.isUserDefinedId=!0;else{let h=this.edges.filter(f=>f.start===l.start&&f.end===l.end);h.length===0?l.id=$h(l.start,l.end,{counter:0,prefix:"L"}):l.id=$h(l.start,l.end,{counter:h.length+1,prefix:"L"})}if(this.edges.length<(this.config.maxEdges??500))Y.info("Pushing edge..."),this.edges.push(l);else throw new Error(`Edge limit exceeded. ${this.edges.length} edges found, but the limit is ${this.config.maxEdges}. + +Initialize mermaid with maxEdges set to a higher number to allow more edges. +You cannot set this config via configuration inside the diagram as it is a secure config. +You have to call mermaid.initialize.`)}isLinkData(e){return e!==null&&typeof e=="object"&&"id"in e&&typeof e.id=="string"}addLink(e,r,n){let i=this.isLinkData(n)?n.id.replace("@",""):void 0;Y.info("addLink",e,r,i);for(let a of e)for(let s of r){let l=a===e[e.length-1],u=s===r[0];l&&u?this.addSingleLink(a,s,n,i):this.addSingleLink(a,s,n,void 0)}}updateLinkInterpolate(e,r){e.forEach(n=>{n==="default"?this.edges.defaultInterpolate=r:this.edges[n].interpolate=r})}updateLink(e,r){e.forEach(n=>{if(typeof n=="number"&&n>=this.edges.length)throw new Error(`The index ${n} for linkStyle is out of bounds. Valid indices for linkStyle are between 0 and ${this.edges.length-1}. (Help: Ensure that the index is within the range of existing edges.)`);n==="default"?this.edges.defaultStyle=r:(this.edges[n].style=r,(this.edges[n]?.style?.length??0)>0&&!this.edges[n]?.style?.some(i=>i?.startsWith("fill"))&&this.edges[n]?.style?.push("fill:none"))})}addClass(e,r){let n=r.join().replace(/\\,/g,"\xA7\xA7\xA7").replace(/,/g,";").replace(/§§§/g,",").split(";");e.split(",").forEach(i=>{let a=this.classes.get(i);a===void 0&&(a={id:i,styles:[],textStyles:[]},this.classes.set(i,a)),n?.forEach(s=>{if(/color/.exec(s)){let l=s.replace("fill","bgFill");a.textStyles.push(l)}a.styles.push(s)})})}setDirection(e){this.direction=e,/.*/.exec(this.direction)&&(this.direction="LR"),/.*v/.exec(this.direction)&&(this.direction="TB"),this.direction==="TD"&&(this.direction="TB")}setClass(e,r){for(let n of e.split(",")){let i=this.vertices.get(n);i&&i.classes.push(r);let a=this.edges.find(l=>l.id===n);a&&a.classes.push(r);let s=this.subGraphLookup.get(n);s&&s.classes.push(r)}}setTooltip(e,r){if(r!==void 0){r=this.sanitizeText(r);for(let n of e.split(","))this.tooltips.set(this.version==="gen-1"?this.lookUpDomId(n):n,r)}}setClickFun(e,r,n){let i=this.lookUpDomId(e);if(me().securityLevel!=="loose"||r===void 0)return;let a=[];if(typeof n=="string"){a=n.split(/,(?=(?:(?:[^"]*"){2})*[^"]*$)/);for(let l=0;l{let l=document.querySelector(`[id="${i}"]`);l!==null&&l.addEventListener("click",()=>{Gt.runFunc(r,...a)},!1)}))}setLink(e,r,n){e.split(",").forEach(i=>{let a=this.vertices.get(i);a!==void 0&&(a.link=Gt.formatUrl(r,this.config),a.linkTarget=n)}),this.setClass(e,"clickable")}getTooltip(e){return this.tooltips.get(e)}setClickEvent(e,r,n){e.split(",").forEach(i=>{this.setClickFun(i,r,n)}),this.setClass(e,"clickable")}bindFunctions(e){this.funs.forEach(r=>{r(e)})}getDirection(){return this.direction?.trim()}getVertices(){return this.vertices}getEdges(){return this.edges}getClasses(){return this.classes}setupToolTips(e){let r=Ge(".mermaidTooltip");(r._groups||r)[0][0]===null&&(r=Ge("body").append("div").attr("class","mermaidTooltip").style("opacity",0)),Ge(e).select("svg").selectAll("g.node").on("mouseover",a=>{let s=Ge(a.currentTarget);if(s.attr("title")===null)return;let u=a.currentTarget?.getBoundingClientRect();r.transition().duration(200).style("opacity",".9"),r.text(s.attr("title")).style("left",window.scrollX+u.left+(u.right-u.left)/2+"px").style("top",window.scrollY+u.bottom+"px"),r.html(r.html().replace(/<br\/>/g,"
    ")),s.classed("hover",!0)}).on("mouseout",a=>{r.transition().duration(500).style("opacity",0),Ge(a.currentTarget).classed("hover",!1)})}clear(e="gen-2"){this.vertices=new Map,this.classes=new Map,this.edges=[],this.funs=[this.setupToolTips.bind(this)],this.subGraphs=[],this.subGraphLookup=new Map,this.subCount=0,this.tooltips=new Map,this.firstGraphFlag=!0,this.version=e,this.config=me(),Ar()}setGen(e){this.version=e||"gen-2"}defaultStyle(){return"fill:#ffa;stroke: #f66; stroke-width: 3px; stroke-dasharray: 5, 5;fill:#ffa;stroke: #666;"}addSubGraph(e,r,n){let i=e.text.trim(),a=n.text;e===n&&/\s/.exec(n.text)&&(i=void 0);let s=o(f=>{let d={boolean:{},number:{},string:{}},p=[],m;return{nodeList:f.filter(function(y){let v=typeof y;return y.stmt&&y.stmt==="dir"?(m=y.value,!1):y.trim()===""?!1:v in d?d[v].hasOwnProperty(y)?!1:d[v][y]=!0:p.includes(y)?!1:p.push(y)}),dir:m}},"uniq"),{nodeList:l,dir:u}=s(r.flat());if(this.version==="gen-1")for(let f=0;f2e3)return{result:!1,count:0};if(this.posCrossRef[this.secCount]=r,this.subGraphs[r].id===e)return{result:!0,count:0};let i=0,a=1;for(;i=0){let l=this.indexNodes2(e,s);if(l.result)return{result:!0,count:a+l.count};a=a+l.count}i=i+1}return{result:!1,count:a}}getDepthFirstPos(e){return this.posCrossRef[e]}indexNodes(){this.secCount=-1,this.subGraphs.length>0&&this.indexNodes2("none",this.subGraphs.length-1)}getSubGraphs(){return this.subGraphs}firstGraph(){return this.firstGraphFlag?(this.firstGraphFlag=!1,!0):!1}destructStartLink(e){let r=e.trim(),n="arrow_open";switch(r[0]){case"<":n="arrow_point",r=r.slice(1);break;case"x":n="arrow_cross",r=r.slice(1);break;case"o":n="arrow_circle",r=r.slice(1);break}let i="normal";return r.includes("=")&&(i="thick"),r.includes(".")&&(i="dotted"),{type:n,stroke:i}}countChar(e,r){let n=r.length,i=0;for(let a=0;a":i="arrow_point",r.startsWith("<")&&(i="double_"+i,n=n.slice(1));break;case"o":i="arrow_circle",r.startsWith("o")&&(i="double_"+i,n=n.slice(1));break}let a="normal",s=n.length-1;n.startsWith("=")&&(a="thick"),n.startsWith("~")&&(a="invisible");let l=this.countChar(".",n);return l&&(a="dotted",s=l),{type:i,stroke:a,length:s}}destructLink(e,r){let n=this.destructEndLink(e),i;if(r){if(i=this.destructStartLink(r),i.stroke!==n.stroke)return{type:"INVALID",stroke:"INVALID"};if(i.type==="arrow_open")i.type=n.type;else{if(i.type!==n.type)return{type:"INVALID",stroke:"INVALID"};i.type="double_"+i.type}return i.type==="double_arrow"&&(i.type="double_arrow_point"),i.length=n.length,i}return n}exists(e,r){for(let n of e)if(n.nodes.includes(r))return!0;return!1}makeUniq(e,r){let n=[];return e.nodes.forEach((i,a)=>{this.exists(r,i)||n.push(e.nodes[a])}),{nodes:n}}getTypeFromVertex(e){if(e.img)return"imageSquare";if(e.icon)return e.form==="circle"?"iconCircle":e.form==="square"?"iconSquare":e.form==="rounded"?"iconRounded":"icon";switch(e.type){case"square":case void 0:return"squareRect";case"round":return"roundedRect";case"ellipse":return"ellipse";default:return e.type}}findNode(e,r){return e.find(n=>n.id===r)}destructEdgeType(e){let r="none",n="arrow_point";switch(e){case"arrow_point":case"arrow_circle":case"arrow_cross":n=e;break;case"double_arrow_point":case"double_arrow_circle":case"double_arrow_cross":r=e.replace("double_",""),n=r;break}return{arrowTypeStart:r,arrowTypeEnd:n}}addNodeFromVertex(e,r,n,i,a,s){let l=n.get(e.id),u=i.get(e.id)??!1,h=this.findNode(r,e.id);if(h)h.cssStyles=e.styles,h.cssCompiledStyles=this.getCompiledStyles(e.classes),h.cssClasses=e.classes.join(" ");else{let f={id:e.id,label:e.text,labelStyle:"",parentId:l,padding:a.flowchart?.padding||8,cssStyles:e.styles,cssCompiledStyles:this.getCompiledStyles(["default","node",...e.classes]),cssClasses:"default "+e.classes.join(" "),dir:e.dir,domId:e.domId,look:s,link:e.link,linkTarget:e.linkTarget,tooltip:this.getTooltip(e.id),icon:e.icon,pos:e.pos,img:e.img,assetWidth:e.assetWidth,assetHeight:e.assetHeight,constraint:e.constraint};u?r.push({...f,isGroup:!0,shape:"rect"}):r.push({...f,isGroup:!1,shape:this.getTypeFromVertex(e)})}}getCompiledStyles(e){let r=[];for(let n of e){let i=this.classes.get(n);i?.styles&&(r=[...r,...i.styles??[]].map(a=>a.trim())),i?.textStyles&&(r=[...r,...i.textStyles??[]].map(a=>a.trim()))}return r}getData(){let e=me(),r=[],n=[],i=this.getSubGraphs(),a=new Map,s=new Map;for(let h=i.length-1;h>=0;h--){let f=i[h];f.nodes.length>0&&s.set(f.id,!0);for(let d of f.nodes)a.set(d,f.id)}for(let h=i.length-1;h>=0;h--){let f=i[h];r.push({id:f.id,label:f.title,labelStyle:"",parentId:a.get(f.id),padding:8,cssCompiledStyles:this.getCompiledStyles(f.classes),cssClasses:f.classes.join(" "),shape:"rect",dir:f.dir,isGroup:!0,look:e.look})}this.getVertices().forEach(h=>{this.addNodeFromVertex(h,r,a,s,e,e.look||"classic")});let u=this.getEdges();return u.forEach((h,f)=>{let{arrowTypeStart:d,arrowTypeEnd:p}=this.destructEdgeType(h.type),m=[...u.defaultStyle??[]];h.style&&m.push(...h.style);let g={id:$h(h.start,h.end,{counter:f,prefix:"L"},h.id),isUserDefinedId:h.isUserDefinedId,start:h.start,end:h.end,type:h.type??"normal",label:h.text,labelpos:"c",thickness:h.stroke,minlen:h.length,classes:h?.stroke==="invisible"?"":"edge-thickness-normal edge-pattern-solid flowchart-link",arrowTypeStart:h?.stroke==="invisible"||h?.type==="arrow_open"?"none":d,arrowTypeEnd:h?.stroke==="invisible"||h?.type==="arrow_open"?"none":p,arrowheadStyle:"fill: #333",cssCompiledStyles:this.getCompiledStyles(h.classes),labelStyle:m,style:m,pattern:h.stroke,look:e.look,animate:h.animate,animation:h.animation,curve:h.interpolate||this.edges.defaultInterpolate||e.flowchart?.curve};n.push(g)}),{nodes:r,edges:n,other:{},config:e}}defaultConfig(){return A3.flowchart}}});var yc,gm=N(()=>{"use strict";dr();yc=o((t,e)=>{let r;return e==="sandbox"&&(r=Ge("#i"+t)),(e==="sandbox"?Ge(r.nodes()[0].contentDocument.body):Ge("body")).select(`[id="${t}"]`)},"getDiagramElement")});var Ru,w2=N(()=>{"use strict";Ru=o(({flowchart:t})=>{let e=t?.subGraphTitleMargin?.top??0,r=t?.subGraphTitleMargin?.bottom??0,n=e+r;return{subGraphTitleTopMargin:e,subGraphTitleBottomMargin:r,subGraphTitleTotalMargin:n}},"getSubGraphTitleMargins")});var YZ,A_e,__e,D_e,L_e,R_e,N_e,XZ,ym,jZ,Hw=N(()=>{"use strict";zt();gr();vt();w2();dr();Wt();to();RD();Gw();qh();Ut();YZ=o(async(t,e)=>{Y.info("Creating subgraph rect for ",e.id,e);let r=me(),{themeVariables:n,handDrawnSeed:i}=r,{clusterBkg:a,clusterBorder:s}=n,{labelStyles:l,nodeStyles:u,borderStyles:h,backgroundStyles:f}=Qe(e),d=t.insert("g").attr("class","cluster "+e.cssClasses).attr("id",e.id).attr("data-look",e.look),p=fr(r.flowchart.htmlLabels),m=d.insert("g").attr("class","cluster-label "),g=await Hn(m,e.label,{style:e.labelStyle,useHtmlLabels:p,isNode:!0}),y=g.getBBox();if(fr(r.flowchart.htmlLabels)){let A=g.children[0],S=Ge(g);y=A.getBoundingClientRect(),S.attr("width",y.width),S.attr("height",y.height)}let v=e.width<=y.width+e.padding?y.width+e.padding:e.width;e.width<=y.width+e.padding?e.diff=(v-e.width)/2-e.padding:e.diff=-e.padding;let x=e.height,b=e.x-v/2,w=e.y-x/2;Y.trace("Data ",e,JSON.stringify(e));let C;if(e.look==="handDrawn"){let A=Xe.svg(d),S=Ke(e,{roughness:.7,fill:a,stroke:s,fillWeight:3,seed:i}),_=A.path(Na(b,w,v,x,0),S);C=d.insert(()=>(Y.debug("Rough node insert CXC",_),_),":first-child"),C.select("path:nth-child(2)").attr("style",h.join(";")),C.select("path").attr("style",f.join(";").replace("fill","stroke"))}else C=d.insert("rect",":first-child"),C.attr("style",u).attr("rx",e.rx).attr("ry",e.ry).attr("x",b).attr("y",w).attr("width",v).attr("height",x);let{subGraphTitleTopMargin:T}=Ru(r);if(m.attr("transform",`translate(${e.x-y.width/2}, ${e.y-e.height/2+T})`),l){let A=m.select("span");A&&A.attr("style",l)}let E=C.node().getBBox();return e.offsetX=0,e.width=E.width,e.height=E.height,e.offsetY=y.height-e.padding/2,e.intersect=function(A){return Vh(e,A)},{cluster:d,labelBBox:y}},"rect"),A_e=o((t,e)=>{let r=t.insert("g").attr("class","note-cluster").attr("id",e.id),n=r.insert("rect",":first-child"),i=0*e.padding,a=i/2;n.attr("rx",e.rx).attr("ry",e.ry).attr("x",e.x-e.width/2-a).attr("y",e.y-e.height/2-a).attr("width",e.width+i).attr("height",e.height+i).attr("fill","none");let s=n.node().getBBox();return e.width=s.width,e.height=s.height,e.intersect=function(l){return Vh(e,l)},{cluster:r,labelBBox:{width:0,height:0}}},"noteGroup"),__e=o(async(t,e)=>{let r=me(),{themeVariables:n,handDrawnSeed:i}=r,{altBackground:a,compositeBackground:s,compositeTitleBackground:l,nodeBorder:u}=n,h=t.insert("g").attr("class",e.cssClasses).attr("id",e.id).attr("data-id",e.id).attr("data-look",e.look),f=h.insert("g",":first-child"),d=h.insert("g").attr("class","cluster-label"),p=h.append("rect"),m=d.node().appendChild(await gc(e.label,e.labelStyle,void 0,!0)),g=m.getBBox();if(fr(r.flowchart.htmlLabels)){let _=m.children[0],I=Ge(m);g=_.getBoundingClientRect(),I.attr("width",g.width),I.attr("height",g.height)}let y=0*e.padding,v=y/2,x=(e.width<=g.width+e.padding?g.width+e.padding:e.width)+y;e.width<=g.width+e.padding?e.diff=(x-e.width)/2-e.padding:e.diff=-e.padding;let b=e.height+y,w=e.height+y-g.height-6,C=e.x-x/2,T=e.y-b/2;e.width=x;let E=e.y-e.height/2-v+g.height+2,A;if(e.look==="handDrawn"){let _=e.cssClasses.includes("statediagram-cluster-alt"),I=Xe.svg(h),D=e.rx||e.ry?I.path(Na(C,T,x,b,10),{roughness:.7,fill:l,fillStyle:"solid",stroke:u,seed:i}):I.rectangle(C,T,x,b,{seed:i});A=h.insert(()=>D,":first-child");let k=I.rectangle(C,E,x,w,{fill:_?a:s,fillStyle:_?"hachure":"solid",stroke:u,seed:i});A=h.insert(()=>D,":first-child"),p=h.insert(()=>k)}else A=f.insert("rect",":first-child"),A.attr("class","outer").attr("x",C).attr("y",T).attr("width",x).attr("height",b).attr("data-look",e.look),p.attr("class","inner").attr("x",C).attr("y",E).attr("width",x).attr("height",w);d.attr("transform",`translate(${e.x-g.width/2}, ${T+1-(fr(r.flowchart.htmlLabels)?0:3)})`);let S=A.node().getBBox();return e.height=S.height,e.offsetX=0,e.offsetY=g.height-e.padding/2,e.labelBBox=g,e.intersect=function(_){return Vh(e,_)},{cluster:h,labelBBox:g}},"roundedWithTitle"),D_e=o(async(t,e)=>{Y.info("Creating subgraph rect for ",e.id,e);let r=me(),{themeVariables:n,handDrawnSeed:i}=r,{clusterBkg:a,clusterBorder:s}=n,{labelStyles:l,nodeStyles:u,borderStyles:h,backgroundStyles:f}=Qe(e),d=t.insert("g").attr("class","cluster "+e.cssClasses).attr("id",e.id).attr("data-look",e.look),p=fr(r.flowchart.htmlLabels),m=d.insert("g").attr("class","cluster-label "),g=await Hn(m,e.label,{style:e.labelStyle,useHtmlLabels:p,isNode:!0,width:e.width}),y=g.getBBox();if(fr(r.flowchart.htmlLabels)){let A=g.children[0],S=Ge(g);y=A.getBoundingClientRect(),S.attr("width",y.width),S.attr("height",y.height)}let v=e.width<=y.width+e.padding?y.width+e.padding:e.width;e.width<=y.width+e.padding?e.diff=(v-e.width)/2-e.padding:e.diff=-e.padding;let x=e.height,b=e.x-v/2,w=e.y-x/2;Y.trace("Data ",e,JSON.stringify(e));let C;if(e.look==="handDrawn"){let A=Xe.svg(d),S=Ke(e,{roughness:.7,fill:a,stroke:s,fillWeight:4,seed:i}),_=A.path(Na(b,w,v,x,e.rx),S);C=d.insert(()=>(Y.debug("Rough node insert CXC",_),_),":first-child"),C.select("path:nth-child(2)").attr("style",h.join(";")),C.select("path").attr("style",f.join(";").replace("fill","stroke"))}else C=d.insert("rect",":first-child"),C.attr("style",u).attr("rx",e.rx).attr("ry",e.ry).attr("x",b).attr("y",w).attr("width",v).attr("height",x);let{subGraphTitleTopMargin:T}=Ru(r);if(m.attr("transform",`translate(${e.x-y.width/2}, ${e.y-e.height/2+T})`),l){let A=m.select("span");A&&A.attr("style",l)}let E=C.node().getBBox();return e.offsetX=0,e.width=E.width,e.height=E.height,e.offsetY=y.height-e.padding/2,e.intersect=function(A){return Vh(e,A)},{cluster:d,labelBBox:y}},"kanbanSection"),L_e=o((t,e)=>{let r=me(),{themeVariables:n,handDrawnSeed:i}=r,{nodeBorder:a}=n,s=t.insert("g").attr("class",e.cssClasses).attr("id",e.id).attr("data-look",e.look),l=s.insert("g",":first-child"),u=0*e.padding,h=e.width+u;e.diff=-e.padding;let f=e.height+u,d=e.x-h/2,p=e.y-f/2;e.width=h;let m;if(e.look==="handDrawn"){let v=Xe.svg(s).rectangle(d,p,h,f,{fill:"lightgrey",roughness:.5,strokeLineDash:[5],stroke:a,seed:i});m=s.insert(()=>v,":first-child")}else m=l.insert("rect",":first-child"),m.attr("class","divider").attr("x",d).attr("y",p).attr("width",h).attr("height",f).attr("data-look",e.look);let g=m.node().getBBox();return e.height=g.height,e.offsetX=0,e.offsetY=0,e.intersect=function(y){return Vh(e,y)},{cluster:s,labelBBox:{}}},"divider"),R_e=YZ,N_e={rect:YZ,squareRect:R_e,roundedWithTitle:__e,noteGroup:A_e,divider:L_e,kanbanSection:D_e},XZ=new Map,ym=o(async(t,e)=>{let r=e.shape||"rect",n=await N_e[r](t,e);return XZ.set(e.id,n),n},"insertCluster"),jZ=o(()=>{XZ=new Map},"clear")});function Ww(t,e){if(t===void 0||e===void 0)return{angle:0,deltaX:0,deltaY:0};t=Wn(t),e=Wn(e);let[r,n]=[t.x,t.y],[i,a]=[e.x,e.y],s=i-r,l=a-n;return{angle:Math.atan(l/s),deltaX:s,deltaY:l}}var $o,Wn,qw,JD=N(()=>{"use strict";$o={aggregation:18,extension:18,composition:18,dependency:6,lollipop:13.5,arrow_point:4};o(Ww,"calculateDeltaAndAngle");Wn=o(t=>Array.isArray(t)?{x:t[0],y:t[1]}:t,"pointTransformer"),qw=o(t=>({x:o(function(e,r,n){let i=0,a=Wn(n[0]).x=0?1:-1)}else if(r===n.length-1&&Object.hasOwn($o,t.arrowTypeEnd)){let{angle:m,deltaX:g}=Ww(n[n.length-1],n[n.length-2]);i=$o[t.arrowTypeEnd]*Math.cos(m)*(g>=0?1:-1)}let s=Math.abs(Wn(e).x-Wn(n[n.length-1]).x),l=Math.abs(Wn(e).y-Wn(n[n.length-1]).y),u=Math.abs(Wn(e).x-Wn(n[0]).x),h=Math.abs(Wn(e).y-Wn(n[0]).y),f=$o[t.arrowTypeStart],d=$o[t.arrowTypeEnd],p=1;if(s0&&l0&&h=0?1:-1)}else if(r===n.length-1&&Object.hasOwn($o,t.arrowTypeEnd)){let{angle:m,deltaY:g}=Ww(n[n.length-1],n[n.length-2]);i=$o[t.arrowTypeEnd]*Math.abs(Math.sin(m))*(g>=0?1:-1)}let s=Math.abs(Wn(e).y-Wn(n[n.length-1]).y),l=Math.abs(Wn(e).x-Wn(n[n.length-1]).x),u=Math.abs(Wn(e).y-Wn(n[0]).y),h=Math.abs(Wn(e).x-Wn(n[0]).x),f=$o[t.arrowTypeStart],d=$o[t.arrowTypeEnd],p=1;if(s0&&l0&&h{"use strict";vt();QZ=o((t,e,r,n,i,a)=>{e.arrowTypeStart&&KZ(t,"start",e.arrowTypeStart,r,n,i,a),e.arrowTypeEnd&&KZ(t,"end",e.arrowTypeEnd,r,n,i,a)},"addEdgeMarkers"),M_e={arrow_cross:{type:"cross",fill:!1},arrow_point:{type:"point",fill:!0},arrow_barb:{type:"barb",fill:!0},arrow_circle:{type:"circle",fill:!1},aggregation:{type:"aggregation",fill:!1},extension:{type:"extension",fill:!1},composition:{type:"composition",fill:!0},dependency:{type:"dependency",fill:!0},lollipop:{type:"lollipop",fill:!1},only_one:{type:"onlyOne",fill:!1},zero_or_one:{type:"zeroOrOne",fill:!1},one_or_more:{type:"oneOrMore",fill:!1},zero_or_more:{type:"zeroOrMore",fill:!1},requirement_arrow:{type:"requirement_arrow",fill:!1},requirement_contains:{type:"requirement_contains",fill:!1}},KZ=o((t,e,r,n,i,a,s)=>{let l=M_e[r];if(!l){Y.warn(`Unknown arrow type: ${r}`);return}let u=l.type,f=`${i}_${a}-${u}${e==="start"?"Start":"End"}`;if(s&&s.trim()!==""){let d=s.replace(/[^\dA-Za-z]/g,"_"),p=`${f}_${d}`;if(!document.getElementById(p)){let m=document.getElementById(f);if(m){let g=m.cloneNode(!0);g.id=p,g.querySelectorAll("path, circle, line").forEach(v=>{v.setAttribute("stroke",s),l.fill&&v.setAttribute("fill",s)}),m.parentNode?.appendChild(g)}}t.attr(`marker-${e}`,`url(${n}#${p})`)}else t.attr(`marker-${e}`,`url(${n}#${f})`)},"addEdgeMarker")});function Yw(t,e){me().flowchart.htmlLabels&&t&&(t.style.width=e.length*9+"px",t.style.height="12px")}function P_e(t){let e=[],r=[];for(let n=1;n5&&Math.abs(a.y-i.y)>5||i.y===a.y&&a.x===s.x&&Math.abs(a.x-i.x)>5&&Math.abs(a.y-s.y)>5)&&(e.push(a),r.push(n))}return{cornerPoints:e,cornerPointPositions:r}}var Xw,pa,tJ,T2,jw,Kw,I_e,O_e,JZ,eJ,B_e,Qw,eL=N(()=>{"use strict";zt();gr();vt();to();ir();JD();w2();dr();Wt();Gw();ZZ();Ut();Xw=new Map,pa=new Map,tJ=o(()=>{Xw.clear(),pa.clear()},"clear"),T2=o(t=>t?t.reduce((r,n)=>r+";"+n,""):"","getLabelStyles"),jw=o(async(t,e)=>{let r=fr(me().flowchart.htmlLabels),n=await Hn(t,e.label,{style:T2(e.labelStyle),useHtmlLabels:r,addSvgBackground:!0,isNode:!1});Y.info("abc82",e,e.labelType);let i=t.insert("g").attr("class","edgeLabel"),a=i.insert("g").attr("class","label");a.node().appendChild(n);let s=n.getBBox();if(r){let u=n.children[0],h=Ge(n);s=u.getBoundingClientRect(),h.attr("width",s.width),h.attr("height",s.height)}a.attr("transform","translate("+-s.width/2+", "+-s.height/2+")"),Xw.set(e.id,i),e.width=s.width,e.height=s.height;let l;if(e.startLabelLeft){let u=await gc(e.startLabelLeft,T2(e.labelStyle)),h=t.insert("g").attr("class","edgeTerminals"),f=h.insert("g").attr("class","inner");l=f.node().appendChild(u);let d=u.getBBox();f.attr("transform","translate("+-d.width/2+", "+-d.height/2+")"),pa.get(e.id)||pa.set(e.id,{}),pa.get(e.id).startLeft=h,Yw(l,e.startLabelLeft)}if(e.startLabelRight){let u=await gc(e.startLabelRight,T2(e.labelStyle)),h=t.insert("g").attr("class","edgeTerminals"),f=h.insert("g").attr("class","inner");l=h.node().appendChild(u),f.node().appendChild(u);let d=u.getBBox();f.attr("transform","translate("+-d.width/2+", "+-d.height/2+")"),pa.get(e.id)||pa.set(e.id,{}),pa.get(e.id).startRight=h,Yw(l,e.startLabelRight)}if(e.endLabelLeft){let u=await gc(e.endLabelLeft,T2(e.labelStyle)),h=t.insert("g").attr("class","edgeTerminals"),f=h.insert("g").attr("class","inner");l=f.node().appendChild(u);let d=u.getBBox();f.attr("transform","translate("+-d.width/2+", "+-d.height/2+")"),h.node().appendChild(u),pa.get(e.id)||pa.set(e.id,{}),pa.get(e.id).endLeft=h,Yw(l,e.endLabelLeft)}if(e.endLabelRight){let u=await gc(e.endLabelRight,T2(e.labelStyle)),h=t.insert("g").attr("class","edgeTerminals"),f=h.insert("g").attr("class","inner");l=f.node().appendChild(u);let d=u.getBBox();f.attr("transform","translate("+-d.width/2+", "+-d.height/2+")"),h.node().appendChild(u),pa.get(e.id)||pa.set(e.id,{}),pa.get(e.id).endRight=h,Yw(l,e.endLabelRight)}return n},"insertEdgeLabel");o(Yw,"setTerminalWidth");Kw=o((t,e)=>{Y.debug("Moving label abc88 ",t.id,t.label,Xw.get(t.id),e);let r=e.updatedPath?e.updatedPath:e.originalPath,n=me(),{subGraphTitleTotalMargin:i}=Ru(n);if(t.label){let a=Xw.get(t.id),s=t.x,l=t.y;if(r){let u=Gt.calcLabelPosition(r);Y.debug("Moving label "+t.label+" from (",s,",",l,") to (",u.x,",",u.y,") abc88"),e.updatedPath&&(s=u.x,l=u.y)}a.attr("transform",`translate(${s}, ${l+i/2})`)}if(t.startLabelLeft){let a=pa.get(t.id).startLeft,s=t.x,l=t.y;if(r){let u=Gt.calcTerminalLabelPosition(t.arrowTypeStart?10:0,"start_left",r);s=u.x,l=u.y}a.attr("transform",`translate(${s}, ${l})`)}if(t.startLabelRight){let a=pa.get(t.id).startRight,s=t.x,l=t.y;if(r){let u=Gt.calcTerminalLabelPosition(t.arrowTypeStart?10:0,"start_right",r);s=u.x,l=u.y}a.attr("transform",`translate(${s}, ${l})`)}if(t.endLabelLeft){let a=pa.get(t.id).endLeft,s=t.x,l=t.y;if(r){let u=Gt.calcTerminalLabelPosition(t.arrowTypeEnd?10:0,"end_left",r);s=u.x,l=u.y}a.attr("transform",`translate(${s}, ${l})`)}if(t.endLabelRight){let a=pa.get(t.id).endRight,s=t.x,l=t.y;if(r){let u=Gt.calcTerminalLabelPosition(t.arrowTypeEnd?10:0,"end_right",r);s=u.x,l=u.y}a.attr("transform",`translate(${s}, ${l})`)}},"positionEdgeLabel"),I_e=o((t,e)=>{let r=t.x,n=t.y,i=Math.abs(e.x-r),a=Math.abs(e.y-n),s=t.width/2,l=t.height/2;return i>=s||a>=l},"outsideNode"),O_e=o((t,e,r)=>{Y.debug(`intersection calc abc89: + outsidePoint: ${JSON.stringify(e)} + insidePoint : ${JSON.stringify(r)} + node : x:${t.x} y:${t.y} w:${t.width} h:${t.height}`);let n=t.x,i=t.y,a=Math.abs(n-r.x),s=t.width/2,l=r.xMath.abs(n-e.x)*u){let d=r.y{Y.warn("abc88 cutPathAtIntersect",t,e);let r=[],n=t[0],i=!1;return t.forEach(a=>{if(Y.info("abc88 checking point",a,e),!I_e(e,a)&&!i){let s=O_e(e,n,a);Y.debug("abc88 inside",a,n,s),Y.debug("abc88 intersection",s,e);let l=!1;r.forEach(u=>{l=l||u.x===s.x&&u.y===s.y}),r.some(u=>u.x===s.x&&u.y===s.y)?Y.warn("abc88 no intersect",s,r):r.push(s),i=!0}else Y.warn("abc88 outside",a,n),n=a,i||r.push(a)}),Y.debug("returning points",r),r},"cutPathAtIntersect");o(P_e,"extractCornerPoints");eJ=o(function(t,e,r){let n=e.x-t.x,i=e.y-t.y,a=Math.sqrt(n*n+i*i),s=r/a;return{x:e.x-s*n,y:e.y-s*i}},"findAdjacentPoint"),B_e=o(function(t){let{cornerPointPositions:e}=P_e(t),r=[];for(let n=0;n10&&Math.abs(a.y-i.y)>=10){Y.debug("Corner point fixing",Math.abs(a.x-i.x),Math.abs(a.y-i.y));let m=5;s.x===l.x?p={x:h<0?l.x-m+d:l.x+m-d,y:f<0?l.y-d:l.y+d}:p={x:h<0?l.x-d:l.x+d,y:f<0?l.y-m+d:l.y+m-d}}else Y.debug("Corner point skipping fixing",Math.abs(a.x-i.x),Math.abs(a.y-i.y));r.push(p,u)}else r.push(t[n]);return r},"fixCorners"),Qw=o(function(t,e,r,n,i,a,s){let{handDrawnSeed:l}=me(),u=e.points,h=!1,f=i;var d=a;let p=[];for(let _ in e.cssCompiledStyles)ND(_)||p.push(e.cssCompiledStyles[_]);d.intersect&&f.intersect&&(u=u.slice(1,e.points.length-1),u.unshift(f.intersect(u[0])),Y.debug("Last point APA12",e.start,"-->",e.end,u[u.length-1],d,d.intersect(u[u.length-1])),u.push(d.intersect(u[u.length-1]))),e.toCluster&&(Y.info("to cluster abc88",r.get(e.toCluster)),u=JZ(e.points,r.get(e.toCluster).node),h=!0),e.fromCluster&&(Y.debug("from cluster abc88",r.get(e.fromCluster),JSON.stringify(u,null,2)),u=JZ(u.reverse(),r.get(e.fromCluster).node).reverse(),h=!0);let m=u.filter(_=>!Number.isNaN(_.y));m=B_e(m);let g=Do;switch(g=wu,e.curve){case"linear":g=wu;break;case"basis":g=Do;break;case"cardinal":g=Pv;break;case"bumpX":g=Rv;break;case"bumpY":g=Nv;break;case"catmullRom":g=$v;break;case"monotoneX":g=zv;break;case"monotoneY":g=Gv;break;case"natural":g=F0;break;case"step":g=$0;break;case"stepAfter":g=Uv;break;case"stepBefore":g=Vv;break;default:g=Do}let{x:y,y:v}=qw(e),x=wl().x(y).y(v).curve(g),b;switch(e.thickness){case"normal":b="edge-thickness-normal";break;case"thick":b="edge-thickness-thick";break;case"invisible":b="edge-thickness-invisible";break;default:b="edge-thickness-normal"}switch(e.pattern){case"solid":b+=" edge-pattern-solid";break;case"dotted":b+=" edge-pattern-dotted";break;case"dashed":b+=" edge-pattern-dashed";break;default:b+=" edge-pattern-solid"}let w,C=x(m),T=Array.isArray(e.style)?e.style:[e.style],E=T.find(_=>_?.startsWith("stroke:"));if(e.look==="handDrawn"){let _=Xe.svg(t);Object.assign([],m);let I=_.path(C,{roughness:.3,seed:l});b+=" transition",w=Ge(I).select("path").attr("id",e.id).attr("class"," "+b+(e.classes?" "+e.classes:"")).attr("style",T?T.reduce((k,L)=>k+";"+L,""):"");let D=w.attr("d");w.attr("d",D),t.node().appendChild(w.node())}else{let _=p.join(";"),I=T?T.reduce((L,R)=>L+R+";",""):"",D="";e.animate&&(D=" edge-animation-fast"),e.animation&&(D=" edge-animation-"+e.animation);let k=_?_+";"+I+";":I;w=t.append("path").attr("d",C).attr("id",e.id).attr("class"," "+b+(e.classes?" "+e.classes:"")+(D??"")).attr("style",k),E=k.match(/stroke:([^;]+)/)?.[1]}let A="";(me().flowchart.arrowMarkerAbsolute||me().state.arrowMarkerAbsolute)&&(A=window.location.protocol+"//"+window.location.host+window.location.pathname+window.location.search,A=A.replace(/\(/g,"\\(").replace(/\)/g,"\\)")),Y.info("arrowTypeStart",e.arrowTypeStart),Y.info("arrowTypeEnd",e.arrowTypeEnd),QZ(w,e,A,s,n,E);let S={};return h&&(S.updatedPath=u),S.originalPath=e.points,S},"insertEdge")});var F_e,$_e,z_e,G_e,V_e,U_e,H_e,W_e,q_e,Y_e,X_e,j_e,K_e,Q_e,Z_e,J_e,e9e,Zw,tL=N(()=>{"use strict";vt();F_e=o((t,e,r,n)=>{e.forEach(i=>{e9e[i](t,r,n)})},"insertMarkers"),$_e=o((t,e,r)=>{Y.trace("Making markers for ",r),t.append("defs").append("marker").attr("id",r+"_"+e+"-extensionStart").attr("class","marker extension "+e).attr("refX",18).attr("refY",7).attr("markerWidth",190).attr("markerHeight",240).attr("orient","auto").append("path").attr("d","M 1,7 L18,13 V 1 Z"),t.append("defs").append("marker").attr("id",r+"_"+e+"-extensionEnd").attr("class","marker extension "+e).attr("refX",1).attr("refY",7).attr("markerWidth",20).attr("markerHeight",28).attr("orient","auto").append("path").attr("d","M 1,1 V 13 L18,7 Z")},"extension"),z_e=o((t,e,r)=>{t.append("defs").append("marker").attr("id",r+"_"+e+"-compositionStart").attr("class","marker composition "+e).attr("refX",18).attr("refY",7).attr("markerWidth",190).attr("markerHeight",240).attr("orient","auto").append("path").attr("d","M 18,7 L9,13 L1,7 L9,1 Z"),t.append("defs").append("marker").attr("id",r+"_"+e+"-compositionEnd").attr("class","marker composition "+e).attr("refX",1).attr("refY",7).attr("markerWidth",20).attr("markerHeight",28).attr("orient","auto").append("path").attr("d","M 18,7 L9,13 L1,7 L9,1 Z")},"composition"),G_e=o((t,e,r)=>{t.append("defs").append("marker").attr("id",r+"_"+e+"-aggregationStart").attr("class","marker aggregation "+e).attr("refX",18).attr("refY",7).attr("markerWidth",190).attr("markerHeight",240).attr("orient","auto").append("path").attr("d","M 18,7 L9,13 L1,7 L9,1 Z"),t.append("defs").append("marker").attr("id",r+"_"+e+"-aggregationEnd").attr("class","marker aggregation "+e).attr("refX",1).attr("refY",7).attr("markerWidth",20).attr("markerHeight",28).attr("orient","auto").append("path").attr("d","M 18,7 L9,13 L1,7 L9,1 Z")},"aggregation"),V_e=o((t,e,r)=>{t.append("defs").append("marker").attr("id",r+"_"+e+"-dependencyStart").attr("class","marker dependency "+e).attr("refX",6).attr("refY",7).attr("markerWidth",190).attr("markerHeight",240).attr("orient","auto").append("path").attr("d","M 5,7 L9,13 L1,7 L9,1 Z"),t.append("defs").append("marker").attr("id",r+"_"+e+"-dependencyEnd").attr("class","marker dependency "+e).attr("refX",13).attr("refY",7).attr("markerWidth",20).attr("markerHeight",28).attr("orient","auto").append("path").attr("d","M 18,7 L9,13 L14,7 L9,1 Z")},"dependency"),U_e=o((t,e,r)=>{t.append("defs").append("marker").attr("id",r+"_"+e+"-lollipopStart").attr("class","marker lollipop "+e).attr("refX",13).attr("refY",7).attr("markerWidth",190).attr("markerHeight",240).attr("orient","auto").append("circle").attr("stroke","black").attr("fill","transparent").attr("cx",7).attr("cy",7).attr("r",6),t.append("defs").append("marker").attr("id",r+"_"+e+"-lollipopEnd").attr("class","marker lollipop "+e).attr("refX",1).attr("refY",7).attr("markerWidth",190).attr("markerHeight",240).attr("orient","auto").append("circle").attr("stroke","black").attr("fill","transparent").attr("cx",7).attr("cy",7).attr("r",6)},"lollipop"),H_e=o((t,e,r)=>{t.append("marker").attr("id",r+"_"+e+"-pointEnd").attr("class","marker "+e).attr("viewBox","0 0 10 10").attr("refX",5).attr("refY",5).attr("markerUnits","userSpaceOnUse").attr("markerWidth",8).attr("markerHeight",8).attr("orient","auto").append("path").attr("d","M 0 0 L 10 5 L 0 10 z").attr("class","arrowMarkerPath").style("stroke-width",1).style("stroke-dasharray","1,0"),t.append("marker").attr("id",r+"_"+e+"-pointStart").attr("class","marker "+e).attr("viewBox","0 0 10 10").attr("refX",4.5).attr("refY",5).attr("markerUnits","userSpaceOnUse").attr("markerWidth",8).attr("markerHeight",8).attr("orient","auto").append("path").attr("d","M 0 5 L 10 10 L 10 0 z").attr("class","arrowMarkerPath").style("stroke-width",1).style("stroke-dasharray","1,0")},"point"),W_e=o((t,e,r)=>{t.append("marker").attr("id",r+"_"+e+"-circleEnd").attr("class","marker "+e).attr("viewBox","0 0 10 10").attr("refX",11).attr("refY",5).attr("markerUnits","userSpaceOnUse").attr("markerWidth",11).attr("markerHeight",11).attr("orient","auto").append("circle").attr("cx","5").attr("cy","5").attr("r","5").attr("class","arrowMarkerPath").style("stroke-width",1).style("stroke-dasharray","1,0"),t.append("marker").attr("id",r+"_"+e+"-circleStart").attr("class","marker "+e).attr("viewBox","0 0 10 10").attr("refX",-1).attr("refY",5).attr("markerUnits","userSpaceOnUse").attr("markerWidth",11).attr("markerHeight",11).attr("orient","auto").append("circle").attr("cx","5").attr("cy","5").attr("r","5").attr("class","arrowMarkerPath").style("stroke-width",1).style("stroke-dasharray","1,0")},"circle"),q_e=o((t,e,r)=>{t.append("marker").attr("id",r+"_"+e+"-crossEnd").attr("class","marker cross "+e).attr("viewBox","0 0 11 11").attr("refX",12).attr("refY",5.2).attr("markerUnits","userSpaceOnUse").attr("markerWidth",11).attr("markerHeight",11).attr("orient","auto").append("path").attr("d","M 1,1 l 9,9 M 10,1 l -9,9").attr("class","arrowMarkerPath").style("stroke-width",2).style("stroke-dasharray","1,0"),t.append("marker").attr("id",r+"_"+e+"-crossStart").attr("class","marker cross "+e).attr("viewBox","0 0 11 11").attr("refX",-1).attr("refY",5.2).attr("markerUnits","userSpaceOnUse").attr("markerWidth",11).attr("markerHeight",11).attr("orient","auto").append("path").attr("d","M 1,1 l 9,9 M 10,1 l -9,9").attr("class","arrowMarkerPath").style("stroke-width",2).style("stroke-dasharray","1,0")},"cross"),Y_e=o((t,e,r)=>{t.append("defs").append("marker").attr("id",r+"_"+e+"-barbEnd").attr("refX",19).attr("refY",7).attr("markerWidth",20).attr("markerHeight",14).attr("markerUnits","userSpaceOnUse").attr("orient","auto").append("path").attr("d","M 19,7 L9,13 L14,7 L9,1 Z")},"barb"),X_e=o((t,e,r)=>{t.append("defs").append("marker").attr("id",r+"_"+e+"-onlyOneStart").attr("class","marker onlyOne "+e).attr("refX",0).attr("refY",9).attr("markerWidth",18).attr("markerHeight",18).attr("orient","auto").append("path").attr("d","M9,0 L9,18 M15,0 L15,18"),t.append("defs").append("marker").attr("id",r+"_"+e+"-onlyOneEnd").attr("class","marker onlyOne "+e).attr("refX",18).attr("refY",9).attr("markerWidth",18).attr("markerHeight",18).attr("orient","auto").append("path").attr("d","M3,0 L3,18 M9,0 L9,18")},"only_one"),j_e=o((t,e,r)=>{let n=t.append("defs").append("marker").attr("id",r+"_"+e+"-zeroOrOneStart").attr("class","marker zeroOrOne "+e).attr("refX",0).attr("refY",9).attr("markerWidth",30).attr("markerHeight",18).attr("orient","auto");n.append("circle").attr("fill","white").attr("cx",21).attr("cy",9).attr("r",6),n.append("path").attr("d","M9,0 L9,18");let i=t.append("defs").append("marker").attr("id",r+"_"+e+"-zeroOrOneEnd").attr("class","marker zeroOrOne "+e).attr("refX",30).attr("refY",9).attr("markerWidth",30).attr("markerHeight",18).attr("orient","auto");i.append("circle").attr("fill","white").attr("cx",9).attr("cy",9).attr("r",6),i.append("path").attr("d","M21,0 L21,18")},"zero_or_one"),K_e=o((t,e,r)=>{t.append("defs").append("marker").attr("id",r+"_"+e+"-oneOrMoreStart").attr("class","marker oneOrMore "+e).attr("refX",18).attr("refY",18).attr("markerWidth",45).attr("markerHeight",36).attr("orient","auto").append("path").attr("d","M0,18 Q 18,0 36,18 Q 18,36 0,18 M42,9 L42,27"),t.append("defs").append("marker").attr("id",r+"_"+e+"-oneOrMoreEnd").attr("class","marker oneOrMore "+e).attr("refX",27).attr("refY",18).attr("markerWidth",45).attr("markerHeight",36).attr("orient","auto").append("path").attr("d","M3,9 L3,27 M9,18 Q27,0 45,18 Q27,36 9,18")},"one_or_more"),Q_e=o((t,e,r)=>{let n=t.append("defs").append("marker").attr("id",r+"_"+e+"-zeroOrMoreStart").attr("class","marker zeroOrMore "+e).attr("refX",18).attr("refY",18).attr("markerWidth",57).attr("markerHeight",36).attr("orient","auto");n.append("circle").attr("fill","white").attr("cx",48).attr("cy",18).attr("r",6),n.append("path").attr("d","M0,18 Q18,0 36,18 Q18,36 0,18");let i=t.append("defs").append("marker").attr("id",r+"_"+e+"-zeroOrMoreEnd").attr("class","marker zeroOrMore "+e).attr("refX",39).attr("refY",18).attr("markerWidth",57).attr("markerHeight",36).attr("orient","auto");i.append("circle").attr("fill","white").attr("cx",9).attr("cy",18).attr("r",6),i.append("path").attr("d","M21,18 Q39,0 57,18 Q39,36 21,18")},"zero_or_more"),Z_e=o((t,e,r)=>{t.append("defs").append("marker").attr("id",r+"_"+e+"-requirement_arrowEnd").attr("refX",20).attr("refY",10).attr("markerWidth",20).attr("markerHeight",20).attr("orient","auto").append("path").attr("d",`M0,0 + L20,10 + M20,10 + L0,20`)},"requirement_arrow"),J_e=o((t,e,r)=>{let n=t.append("defs").append("marker").attr("id",r+"_"+e+"-requirement_containsStart").attr("refX",0).attr("refY",10).attr("markerWidth",20).attr("markerHeight",20).attr("orient","auto").append("g");n.append("circle").attr("cx",10).attr("cy",10).attr("r",9).attr("fill","none"),n.append("line").attr("x1",1).attr("x2",19).attr("y1",10).attr("y2",10),n.append("line").attr("y1",1).attr("y2",19).attr("x1",10).attr("x2",10)},"requirement_contains"),e9e={extension:$_e,composition:z_e,aggregation:G_e,dependency:V_e,lollipop:U_e,point:H_e,circle:W_e,cross:q_e,barb:Y_e,only_one:X_e,zero_or_one:j_e,one_or_more:K_e,zero_or_more:Q_e,requirement_arrow:Z_e,requirement_contains:J_e},Zw=F_e});async function vm(t,e,r){let n,i;e.shape==="rect"&&(e.rx&&e.ry?e.shape="roundedRect":e.shape="squareRect");let a=e.shape?QD[e.shape]:void 0;if(!a)throw new Error(`No such shape: ${e.shape}. Please check your syntax.`);if(e.link){let s;r.config.securityLevel==="sandbox"?s="_top":e.linkTarget&&(s=e.linkTarget||"_blank"),n=t.insert("svg:a").attr("xlink:href",e.link).attr("target",s??null),i=await a(n,e,r)}else i=await a(t,e,r),n=i;return e.tooltip&&i.attr("title",e.tooltip),Jw.set(e.id,n),e.haveCallback&&n.attr("class",n.attr("class")+" clickable"),n}var Jw,rJ,nJ,k2,eT=N(()=>{"use strict";vt();ZD();Jw=new Map;o(vm,"insertNode");rJ=o((t,e)=>{Jw.set(e.id,t)},"setNodeElem"),nJ=o(()=>{Jw.clear()},"clear"),k2=o(t=>{let e=Jw.get(t.id);Y.trace("Transforming node",t.diff,t,"translate("+(t.x-t.width/2-5)+", "+t.width/2+")");let r=8,n=t.diff||0;return t.clusterNode?e.attr("transform","translate("+(t.x+n-t.width/2)+", "+(t.y-t.height/2-r)+")"):e.attr("transform","translate("+t.x+", "+t.y+")"),n},"positionNode")});var iJ,aJ=N(()=>{"use strict";ji();gr();vt();Hw();eL();tL();eT();Ft();ir();iJ={common:Ze,getConfig:cr,insertCluster:ym,insertEdge:Qw,insertEdgeLabel:jw,insertMarkers:Zw,insertNode:vm,interpolateToCurve:W9,labelHelper:pt,log:Y,positionEdgeLabel:Kw}});function r9e(t){return typeof t=="symbol"||ri(t)&&da(t)==t9e}var t9e,no,Pd=N(()=>{"use strict";ku();No();t9e="[object Symbol]";o(r9e,"isSymbol");no=r9e});function n9e(t,e){for(var r=-1,n=t==null?0:t.length,i=Array(n);++r{"use strict";o(n9e,"arrayMap");Ns=n9e});function lJ(t){if(typeof t=="string")return t;if(Pt(t))return Ns(t,lJ)+"";if(no(t))return oJ?oJ.call(t):"";var e=t+"";return e=="0"&&1/t==-i9e?"-0":e}var i9e,sJ,oJ,cJ,uJ=N(()=>{"use strict";Ed();Bd();Un();Pd();i9e=1/0,sJ=ea?ea.prototype:void 0,oJ=sJ?sJ.toString:void 0;o(lJ,"baseToString");cJ=lJ});function s9e(t){for(var e=t.length;e--&&a9e.test(t.charAt(e)););return e}var a9e,hJ,fJ=N(()=>{"use strict";a9e=/\s/;o(s9e,"trimmedEndIndex");hJ=s9e});function l9e(t){return t&&t.slice(0,hJ(t)+1).replace(o9e,"")}var o9e,dJ,pJ=N(()=>{"use strict";fJ();o9e=/^\s+/;o(l9e,"baseTrim");dJ=l9e});function d9e(t){if(typeof t=="number")return t;if(no(t))return mJ;if(bn(t)){var e=typeof t.valueOf=="function"?t.valueOf():t;t=bn(e)?e+"":e}if(typeof t!="string")return t===0?t:+t;t=dJ(t);var r=u9e.test(t);return r||h9e.test(t)?f9e(t.slice(2),r?2:8):c9e.test(t)?mJ:+t}var mJ,c9e,u9e,h9e,f9e,gJ,yJ=N(()=>{"use strict";pJ();Js();Pd();mJ=NaN,c9e=/^[-+]0x[0-9a-f]+$/i,u9e=/^0b[01]+$/i,h9e=/^0o[0-7]+$/i,f9e=parseInt;o(d9e,"toNumber");gJ=d9e});function m9e(t){if(!t)return t===0?t:0;if(t=gJ(t),t===vJ||t===-vJ){var e=t<0?-1:1;return e*p9e}return t===t?t:0}var vJ,p9e,xm,rL=N(()=>{"use strict";yJ();vJ=1/0,p9e=17976931348623157e292;o(m9e,"toFinite");xm=m9e});function g9e(t){var e=xm(t),r=e%1;return e===e?r?e-r:e:0}var vc,bm=N(()=>{"use strict";rL();o(g9e,"toInteger");vc=g9e});var y9e,tT,xJ=N(()=>{"use strict";Lh();Lo();y9e=Ss(li,"WeakMap"),tT=y9e});function v9e(){}var ni,nL=N(()=>{"use strict";o(v9e,"noop");ni=v9e});function x9e(t,e){for(var r=-1,n=t==null?0:t.length;++r{"use strict";o(x9e,"arrayEach");rT=x9e});function b9e(t,e,r,n){for(var i=t.length,a=r+(n?1:-1);n?a--:++a{"use strict";o(b9e,"baseFindIndex");nT=b9e});function w9e(t){return t!==t}var bJ,wJ=N(()=>{"use strict";o(w9e,"baseIsNaN");bJ=w9e});function T9e(t,e,r){for(var n=r-1,i=t.length;++n{"use strict";o(T9e,"strictIndexOf");TJ=T9e});function k9e(t,e,r){return e===e?TJ(t,e,r):nT(t,bJ,r)}var wm,iT=N(()=>{"use strict";aL();wJ();kJ();o(k9e,"baseIndexOf");wm=k9e});function E9e(t,e){var r=t==null?0:t.length;return!!r&&wm(t,e,0)>-1}var aT,sL=N(()=>{"use strict";iT();o(E9e,"arrayIncludes");aT=E9e});var S9e,EJ,SJ=N(()=>{"use strict";N9();S9e=nw(Object.keys,Object),EJ=S9e});function _9e(t){if(!uc(t))return EJ(t);var e=[];for(var r in Object(t))A9e.call(t,r)&&r!="constructor"&&e.push(r);return e}var C9e,A9e,Tm,sT=N(()=>{"use strict";Z0();SJ();C9e=Object.prototype,A9e=C9e.hasOwnProperty;o(_9e,"baseKeys");Tm=_9e});function D9e(t){return ci(t)?lw(t):Tm(t)}var zr,xc=N(()=>{"use strict";B9();sT();Mo();o(D9e,"keys");zr=D9e});var L9e,R9e,N9e,ma,CJ=N(()=>{"use strict";rm();Dd();G9();Mo();Z0();xc();L9e=Object.prototype,R9e=L9e.hasOwnProperty,N9e=hw(function(t,e){if(uc(e)||ci(e)){Po(e,zr(e),t);return}for(var r in e)R9e.call(e,r)&&hc(t,r,e[r])}),ma=N9e});function O9e(t,e){if(Pt(t))return!1;var r=typeof t;return r=="number"||r=="symbol"||r=="boolean"||t==null||no(t)?!0:I9e.test(t)||!M9e.test(t)||e!=null&&t in Object(e)}var M9e,I9e,km,oT=N(()=>{"use strict";Un();Pd();M9e=/\.|\[(?:[^[\]]*|(["'])(?:(?!\1)[^\\]|\\.)*?\1)\]/,I9e=/^\w*$/;o(O9e,"isKey");km=O9e});function B9e(t){var e=H0(t,function(n){return r.size===P9e&&r.clear(),n}),r=e.cache;return e}var P9e,AJ,_J=N(()=>{"use strict";S9();P9e=500;o(B9e,"memoizeCapped");AJ=B9e});var F9e,$9e,z9e,DJ,LJ=N(()=>{"use strict";_J();F9e=/[^.[\]]+|\[(?:(-?\d+(?:\.\d+)?)|(["'])((?:(?!\2)[^\\]|\\.)*?)\2)\]|(?=(?:\.|\[\])(?:\.|\[\]|$))/g,$9e=/\\(\\)?/g,z9e=AJ(function(t){var e=[];return t.charCodeAt(0)===46&&e.push(""),t.replace(F9e,function(r,n,i,a){e.push(i?a.replace($9e,"$1"):n||r)}),e}),DJ=z9e});function G9e(t){return t==null?"":cJ(t)}var lT,oL=N(()=>{"use strict";uJ();o(G9e,"toString");lT=G9e});function V9e(t,e){return Pt(t)?t:km(t,e)?[t]:DJ(lT(t))}var Yh,E2=N(()=>{"use strict";Un();oT();LJ();oL();o(V9e,"castPath");Yh=V9e});function H9e(t){if(typeof t=="string"||no(t))return t;var e=t+"";return e=="0"&&1/t==-U9e?"-0":e}var U9e,bc,Em=N(()=>{"use strict";Pd();U9e=1/0;o(H9e,"toKey");bc=H9e});function W9e(t,e){e=Yh(e,t);for(var r=0,n=e.length;t!=null&&r{"use strict";E2();Em();o(W9e,"baseGet");Xh=W9e});function q9e(t,e,r){var n=t==null?void 0:Xh(t,e);return n===void 0?r:n}var RJ,NJ=N(()=>{"use strict";S2();o(q9e,"get");RJ=q9e});function Y9e(t,e){for(var r=-1,n=e.length,i=t.length;++r{"use strict";o(Y9e,"arrayPush");Sm=Y9e});function X9e(t){return Pt(t)||El(t)||!!(MJ&&t&&t[MJ])}var MJ,IJ,OJ=N(()=>{"use strict";Ed();J0();Un();MJ=ea?ea.isConcatSpreadable:void 0;o(X9e,"isFlattenable");IJ=X9e});function PJ(t,e,r,n,i){var a=-1,s=t.length;for(r||(r=IJ),i||(i=[]);++a0&&r(l)?e>1?PJ(l,e-1,r,n,i):Sm(i,l):n||(i[i.length]=l)}return i}var wc,Cm=N(()=>{"use strict";cT();OJ();o(PJ,"baseFlatten");wc=PJ});function j9e(t){var e=t==null?0:t.length;return e?wc(t,1):[]}var qr,uT=N(()=>{"use strict";Cm();o(j9e,"flatten");qr=j9e});function K9e(t){return uw(cw(t,void 0,qr),t+"")}var BJ,FJ=N(()=>{"use strict";uT();F9();z9();o(K9e,"flatRest");BJ=K9e});function Q9e(t,e,r){var n=-1,i=t.length;e<0&&(e=-e>i?0:i+e),r=r>i?i:r,r<0&&(r+=i),i=e>r?0:r-e>>>0,e>>>=0;for(var a=Array(i);++n{"use strict";o(Q9e,"baseSlice");hT=Q9e});function sDe(t){return aDe.test(t)}var Z9e,J9e,eDe,tDe,rDe,nDe,iDe,aDe,$J,zJ=N(()=>{"use strict";Z9e="\\ud800-\\udfff",J9e="\\u0300-\\u036f",eDe="\\ufe20-\\ufe2f",tDe="\\u20d0-\\u20ff",rDe=J9e+eDe+tDe,nDe="\\ufe0e\\ufe0f",iDe="\\u200d",aDe=RegExp("["+iDe+Z9e+rDe+nDe+"]");o(sDe,"hasUnicode");$J=sDe});function oDe(t,e,r,n){var i=-1,a=t==null?0:t.length;for(n&&a&&(r=t[++i]);++i{"use strict";o(oDe,"arrayReduce");GJ=oDe});function lDe(t,e){return t&&Po(e,zr(e),t)}var UJ,HJ=N(()=>{"use strict";Dd();xc();o(lDe,"baseAssign");UJ=lDe});function cDe(t,e){return t&&Po(e,Cs(e),t)}var WJ,qJ=N(()=>{"use strict";Dd();Bh();o(cDe,"baseAssignIn");WJ=cDe});function uDe(t,e){for(var r=-1,n=t==null?0:t.length,i=0,a=[];++r{"use strict";o(uDe,"arrayFilter");Am=uDe});function hDe(){return[]}var dT,cL=N(()=>{"use strict";o(hDe,"stubArray");dT=hDe});var fDe,dDe,YJ,pDe,_m,pT=N(()=>{"use strict";fT();cL();fDe=Object.prototype,dDe=fDe.propertyIsEnumerable,YJ=Object.getOwnPropertySymbols,pDe=YJ?function(t){return t==null?[]:(t=Object(t),Am(YJ(t),function(e){return dDe.call(t,e)}))}:dT,_m=pDe});function mDe(t,e){return Po(t,_m(t),e)}var XJ,jJ=N(()=>{"use strict";Dd();pT();o(mDe,"copySymbols");XJ=mDe});var gDe,yDe,mT,uL=N(()=>{"use strict";cT();iw();pT();cL();gDe=Object.getOwnPropertySymbols,yDe=gDe?function(t){for(var e=[];t;)Sm(e,_m(t)),t=Q0(t);return e}:dT,mT=yDe});function vDe(t,e){return Po(t,mT(t),e)}var KJ,QJ=N(()=>{"use strict";Dd();uL();o(vDe,"copySymbolsIn");KJ=vDe});function xDe(t,e,r){var n=e(t);return Pt(t)?n:Sm(n,r(t))}var gT,hL=N(()=>{"use strict";cT();Un();o(xDe,"baseGetAllKeys");gT=xDe});function bDe(t){return gT(t,zr,_m)}var C2,fL=N(()=>{"use strict";hL();pT();xc();o(bDe,"getAllKeys");C2=bDe});function wDe(t){return gT(t,Cs,mT)}var yT,dL=N(()=>{"use strict";hL();uL();Bh();o(wDe,"getAllKeysIn");yT=wDe});var TDe,vT,ZJ=N(()=>{"use strict";Lh();Lo();TDe=Ss(li,"DataView"),vT=TDe});var kDe,xT,JJ=N(()=>{"use strict";Lh();Lo();kDe=Ss(li,"Promise"),xT=kDe});var EDe,jh,pL=N(()=>{"use strict";Lh();Lo();EDe=Ss(li,"Set"),jh=EDe});var eee,SDe,tee,ree,nee,iee,CDe,ADe,_De,DDe,LDe,Fd,io,$d=N(()=>{"use strict";ZJ();K5();JJ();pL();xJ();ku();T9();eee="[object Map]",SDe="[object Object]",tee="[object Promise]",ree="[object Set]",nee="[object WeakMap]",iee="[object DataView]",CDe=Eu(vT),ADe=Eu(Mh),_De=Eu(xT),DDe=Eu(jh),LDe=Eu(tT),Fd=da;(vT&&Fd(new vT(new ArrayBuffer(1)))!=iee||Mh&&Fd(new Mh)!=eee||xT&&Fd(xT.resolve())!=tee||jh&&Fd(new jh)!=ree||tT&&Fd(new tT)!=nee)&&(Fd=o(function(t){var e=da(t),r=e==SDe?t.constructor:void 0,n=r?Eu(r):"";if(n)switch(n){case CDe:return iee;case ADe:return eee;case _De:return tee;case DDe:return ree;case LDe:return nee}return e},"getTag"));io=Fd});function MDe(t){var e=t.length,r=new t.constructor(e);return e&&typeof t[0]=="string"&&NDe.call(t,"index")&&(r.index=t.index,r.input=t.input),r}var RDe,NDe,aee,see=N(()=>{"use strict";RDe=Object.prototype,NDe=RDe.hasOwnProperty;o(MDe,"initCloneArray");aee=MDe});function IDe(t,e){var r=e?K0(t.buffer):t.buffer;return new t.constructor(r,t.byteOffset,t.byteLength)}var oee,lee=N(()=>{"use strict";ew();o(IDe,"cloneDataView");oee=IDe});function PDe(t){var e=new t.constructor(t.source,ODe.exec(t));return e.lastIndex=t.lastIndex,e}var ODe,cee,uee=N(()=>{"use strict";ODe=/\w*$/;o(PDe,"cloneRegExp");cee=PDe});function BDe(t){return fee?Object(fee.call(t)):{}}var hee,fee,dee,pee=N(()=>{"use strict";Ed();hee=ea?ea.prototype:void 0,fee=hee?hee.valueOf:void 0;o(BDe,"cloneSymbol");dee=BDe});function nLe(t,e,r){var n=t.constructor;switch(e){case qDe:return K0(t);case FDe:case $De:return new n(+t);case YDe:return oee(t,r);case XDe:case jDe:case KDe:case QDe:case ZDe:case JDe:case eLe:case tLe:case rLe:return tw(t,r);case zDe:return new n;case GDe:case HDe:return new n(t);case VDe:return cee(t);case UDe:return new n;case WDe:return dee(t)}}var FDe,$De,zDe,GDe,VDe,UDe,HDe,WDe,qDe,YDe,XDe,jDe,KDe,QDe,ZDe,JDe,eLe,tLe,rLe,mee,gee=N(()=>{"use strict";ew();lee();uee();pee();L9();FDe="[object Boolean]",$De="[object Date]",zDe="[object Map]",GDe="[object Number]",VDe="[object RegExp]",UDe="[object Set]",HDe="[object String]",WDe="[object Symbol]",qDe="[object ArrayBuffer]",YDe="[object DataView]",XDe="[object Float32Array]",jDe="[object Float64Array]",KDe="[object Int8Array]",QDe="[object Int16Array]",ZDe="[object Int32Array]",JDe="[object Uint8Array]",eLe="[object Uint8ClampedArray]",tLe="[object Uint16Array]",rLe="[object Uint32Array]";o(nLe,"initCloneByTag");mee=nLe});function aLe(t){return ri(t)&&io(t)==iLe}var iLe,yee,vee=N(()=>{"use strict";$d();No();iLe="[object Map]";o(aLe,"baseIsMap");yee=aLe});var xee,sLe,bee,wee=N(()=>{"use strict";vee();_d();t2();xee=Oo&&Oo.isMap,sLe=xee?Io(xee):yee,bee=sLe});function lLe(t){return ri(t)&&io(t)==oLe}var oLe,Tee,kee=N(()=>{"use strict";$d();No();oLe="[object Set]";o(lLe,"baseIsSet");Tee=lLe});var Eee,cLe,See,Cee=N(()=>{"use strict";kee();_d();t2();Eee=Oo&&Oo.isSet,cLe=Eee?Io(Eee):Tee,See=cLe});function bT(t,e,r,n,i,a){var s,l=e&uLe,u=e&hLe,h=e&fLe;if(r&&(s=i?r(t,n,i,a):r(t)),s!==void 0)return s;if(!bn(t))return t;var f=Pt(t);if(f){if(s=aee(t),!l)return rw(t,s)}else{var d=io(t),p=d==_ee||d==yLe;if(Sl(t))return J5(t,l);if(d==Dee||d==Aee||p&&!i){if(s=u||p?{}:aw(t),!l)return u?KJ(t,WJ(s,t)):XJ(t,UJ(s,t))}else{if(!_n[d])return i?t:{};s=mee(t,d,l)}}a||(a=new lc);var m=a.get(t);if(m)return m;a.set(t,s),See(t)?t.forEach(function(v){s.add(bT(v,e,r,v,t,a))}):bee(t)&&t.forEach(function(v,x){s.set(x,bT(v,e,r,x,t,a))});var g=h?u?yT:C2:u?Cs:zr,y=f?void 0:g(t);return rT(y||t,function(v,x){y&&(x=v,v=t[x]),hc(s,x,bT(v,e,r,x,t,a))}),s}var uLe,hLe,fLe,Aee,dLe,pLe,mLe,gLe,_ee,yLe,vLe,xLe,Dee,bLe,wLe,TLe,kLe,ELe,SLe,CLe,ALe,_Le,DLe,LLe,RLe,NLe,MLe,ILe,OLe,_n,wT,mL=N(()=>{"use strict";Zv();iL();rm();HJ();qJ();_9();R9();jJ();QJ();fL();dL();$d();see();gee();M9();Un();tm();wee();Js();Cee();xc();Bh();uLe=1,hLe=2,fLe=4,Aee="[object Arguments]",dLe="[object Array]",pLe="[object Boolean]",mLe="[object Date]",gLe="[object Error]",_ee="[object Function]",yLe="[object GeneratorFunction]",vLe="[object Map]",xLe="[object Number]",Dee="[object Object]",bLe="[object RegExp]",wLe="[object Set]",TLe="[object String]",kLe="[object Symbol]",ELe="[object WeakMap]",SLe="[object ArrayBuffer]",CLe="[object DataView]",ALe="[object Float32Array]",_Le="[object Float64Array]",DLe="[object Int8Array]",LLe="[object Int16Array]",RLe="[object Int32Array]",NLe="[object Uint8Array]",MLe="[object Uint8ClampedArray]",ILe="[object Uint16Array]",OLe="[object Uint32Array]",_n={};_n[Aee]=_n[dLe]=_n[SLe]=_n[CLe]=_n[pLe]=_n[mLe]=_n[ALe]=_n[_Le]=_n[DLe]=_n[LLe]=_n[RLe]=_n[vLe]=_n[xLe]=_n[Dee]=_n[bLe]=_n[wLe]=_n[TLe]=_n[kLe]=_n[NLe]=_n[MLe]=_n[ILe]=_n[OLe]=!0;_n[gLe]=_n[_ee]=_n[ELe]=!1;o(bT,"baseClone");wT=bT});function BLe(t){return wT(t,PLe)}var PLe,an,gL=N(()=>{"use strict";mL();PLe=4;o(BLe,"clone");an=BLe});function zLe(t){return wT(t,FLe|$Le)}var FLe,$Le,yL,Lee=N(()=>{"use strict";mL();FLe=1,$Le=4;o(zLe,"cloneDeep");yL=zLe});function GLe(t){for(var e=-1,r=t==null?0:t.length,n=0,i=[];++e{"use strict";o(GLe,"compact");Tc=GLe});function ULe(t){return this.__data__.set(t,VLe),this}var VLe,Nee,Mee=N(()=>{"use strict";VLe="__lodash_hash_undefined__";o(ULe,"setCacheAdd");Nee=ULe});function HLe(t){return this.__data__.has(t)}var Iee,Oee=N(()=>{"use strict";o(HLe,"setCacheHas");Iee=HLe});function TT(t){var e=-1,r=t==null?0:t.length;for(this.__data__=new Cd;++e{"use strict";Q5();Mee();Oee();o(TT,"SetCache");TT.prototype.add=TT.prototype.push=Nee;TT.prototype.has=Iee;Dm=TT});function WLe(t,e){for(var r=-1,n=t==null?0:t.length;++r{"use strict";o(WLe,"arraySome");ET=WLe});function qLe(t,e){return t.has(e)}var Lm,ST=N(()=>{"use strict";o(qLe,"cacheHas");Lm=qLe});function jLe(t,e,r,n,i,a){var s=r&YLe,l=t.length,u=e.length;if(l!=u&&!(s&&u>l))return!1;var h=a.get(t),f=a.get(e);if(h&&f)return h==e&&f==t;var d=-1,p=!0,m=r&XLe?new Dm:void 0;for(a.set(t,e),a.set(e,t);++d{"use strict";kT();vL();ST();YLe=1,XLe=2;o(jLe,"equalArrays");CT=jLe});function KLe(t){var e=-1,r=Array(t.size);return t.forEach(function(n,i){r[++e]=[i,n]}),r}var Pee,Bee=N(()=>{"use strict";o(KLe,"mapToArray");Pee=KLe});function QLe(t){var e=-1,r=Array(t.size);return t.forEach(function(n){r[++e]=n}),r}var Rm,AT=N(()=>{"use strict";o(QLe,"setToArray");Rm=QLe});function hRe(t,e,r,n,i,a,s){switch(r){case uRe:if(t.byteLength!=e.byteLength||t.byteOffset!=e.byteOffset)return!1;t=t.buffer,e=e.buffer;case cRe:return!(t.byteLength!=e.byteLength||!a(new j0(t),new j0(e)));case eRe:case tRe:case iRe:return Ro(+t,+e);case rRe:return t.name==e.name&&t.message==e.message;case aRe:case oRe:return t==e+"";case nRe:var l=Pee;case sRe:var u=n&ZLe;if(l||(l=Rm),t.size!=e.size&&!u)return!1;var h=s.get(t);if(h)return h==e;n|=JLe,s.set(t,e);var f=CT(l(t),l(e),n,i,a,s);return s.delete(t),f;case lRe:if(bL)return bL.call(t)==bL.call(e)}return!1}var ZLe,JLe,eRe,tRe,rRe,nRe,iRe,aRe,sRe,oRe,lRe,cRe,uRe,Fee,bL,$ee,zee=N(()=>{"use strict";Ed();D9();Sd();xL();Bee();AT();ZLe=1,JLe=2,eRe="[object Boolean]",tRe="[object Date]",rRe="[object Error]",nRe="[object Map]",iRe="[object Number]",aRe="[object RegExp]",sRe="[object Set]",oRe="[object String]",lRe="[object Symbol]",cRe="[object ArrayBuffer]",uRe="[object DataView]",Fee=ea?ea.prototype:void 0,bL=Fee?Fee.valueOf:void 0;o(hRe,"equalByTag");$ee=hRe});function mRe(t,e,r,n,i,a){var s=r&fRe,l=C2(t),u=l.length,h=C2(e),f=h.length;if(u!=f&&!s)return!1;for(var d=u;d--;){var p=l[d];if(!(s?p in e:pRe.call(e,p)))return!1}var m=a.get(t),g=a.get(e);if(m&&g)return m==e&&g==t;var y=!0;a.set(t,e),a.set(e,t);for(var v=s;++d{"use strict";fL();fRe=1,dRe=Object.prototype,pRe=dRe.hasOwnProperty;o(mRe,"equalObjects");Gee=mRe});function vRe(t,e,r,n,i,a){var s=Pt(t),l=Pt(e),u=s?Hee:io(t),h=l?Hee:io(e);u=u==Uee?_T:u,h=h==Uee?_T:h;var f=u==_T,d=h==_T,p=u==h;if(p&&Sl(t)){if(!Sl(e))return!1;s=!0,f=!1}if(p&&!f)return a||(a=new lc),s||Oh(t)?CT(t,e,r,n,i,a):$ee(t,e,u,r,n,i,a);if(!(r&gRe)){var m=f&&Wee.call(t,"__wrapped__"),g=d&&Wee.call(e,"__wrapped__");if(m||g){var y=m?t.value():t,v=g?e.value():e;return a||(a=new lc),i(y,v,r,n,a)}}return p?(a||(a=new lc),Gee(t,e,r,n,i,a)):!1}var gRe,Uee,Hee,_T,yRe,Wee,qee,Yee=N(()=>{"use strict";Zv();xL();zee();Vee();$d();Un();tm();r2();gRe=1,Uee="[object Arguments]",Hee="[object Array]",_T="[object Object]",yRe=Object.prototype,Wee=yRe.hasOwnProperty;o(vRe,"baseIsEqualDeep");qee=vRe});function Xee(t,e,r,n,i){return t===e?!0:t==null||e==null||!ri(t)&&!ri(e)?t!==t&&e!==e:qee(t,e,r,n,Xee,i)}var DT,wL=N(()=>{"use strict";Yee();No();o(Xee,"baseIsEqual");DT=Xee});function wRe(t,e,r,n){var i=r.length,a=i,s=!n;if(t==null)return!a;for(t=Object(t);i--;){var l=r[i];if(s&&l[2]?l[1]!==t[l[0]]:!(l[0]in t))return!1}for(;++i{"use strict";Zv();wL();xRe=1,bRe=2;o(wRe,"baseIsMatch");jee=wRe});function TRe(t){return t===t&&!bn(t)}var LT,TL=N(()=>{"use strict";Js();o(TRe,"isStrictComparable");LT=TRe});function kRe(t){for(var e=zr(t),r=e.length;r--;){var n=e[r],i=t[n];e[r]=[n,i,LT(i)]}return e}var Qee,Zee=N(()=>{"use strict";TL();xc();o(kRe,"getMatchData");Qee=kRe});function ERe(t,e){return function(r){return r==null?!1:r[t]===e&&(e!==void 0||t in Object(r))}}var RT,kL=N(()=>{"use strict";o(ERe,"matchesStrictComparable");RT=ERe});function SRe(t){var e=Qee(t);return e.length==1&&e[0][2]?RT(e[0][0],e[0][1]):function(r){return r===t||jee(r,t,e)}}var Jee,ete=N(()=>{"use strict";Kee();Zee();kL();o(SRe,"baseMatches");Jee=SRe});function CRe(t,e){return t!=null&&e in Object(t)}var tte,rte=N(()=>{"use strict";o(CRe,"baseHasIn");tte=CRe});function ARe(t,e,r){e=Yh(e,t);for(var n=-1,i=e.length,a=!1;++n{"use strict";E2();J0();Un();i2();sw();Em();o(ARe,"hasPath");NT=ARe});function _Re(t,e){return t!=null&&NT(t,e,tte)}var MT,SL=N(()=>{"use strict";rte();EL();o(_Re,"hasIn");MT=_Re});function RRe(t,e){return km(t)&<(e)?RT(bc(t),e):function(r){var n=RJ(r,t);return n===void 0&&n===e?MT(r,t):DT(e,n,DRe|LRe)}}var DRe,LRe,nte,ite=N(()=>{"use strict";wL();NJ();SL();oT();TL();kL();Em();DRe=1,LRe=2;o(RRe,"baseMatchesProperty");nte=RRe});function NRe(t){return function(e){return e?.[t]}}var IT,CL=N(()=>{"use strict";o(NRe,"baseProperty");IT=NRe});function MRe(t){return function(e){return Xh(e,t)}}var ate,ste=N(()=>{"use strict";S2();o(MRe,"basePropertyDeep");ate=MRe});function IRe(t){return km(t)?IT(bc(t)):ate(t)}var ote,lte=N(()=>{"use strict";CL();ste();oT();Em();o(IRe,"property");ote=IRe});function ORe(t){return typeof t=="function"?t:t==null?ta:typeof t=="object"?Pt(t)?nte(t[0],t[1]):Jee(t):ote(t)}var pn,rs=N(()=>{"use strict";ete();ite();Cu();Un();lte();o(ORe,"baseIteratee");pn=ORe});function PRe(t,e,r,n){for(var i=-1,a=t==null?0:t.length;++i{"use strict";o(PRe,"arrayAggregator");cte=PRe});function BRe(t,e){return t&&X0(t,e,zr)}var Nm,OT=N(()=>{"use strict";Z5();xc();o(BRe,"baseForOwn");Nm=BRe});function FRe(t,e){return function(r,n){if(r==null)return r;if(!ci(r))return t(r,n);for(var i=r.length,a=e?i:-1,s=Object(r);(e?a--:++a{"use strict";Mo();o(FRe,"createBaseEach");hte=FRe});var $Re,Ms,Kh=N(()=>{"use strict";OT();fte();$Re=hte(Nm),Ms=$Re});function zRe(t,e,r,n){return Ms(t,function(i,a,s){e(n,i,r(i),s)}),n}var dte,pte=N(()=>{"use strict";Kh();o(zRe,"baseAggregator");dte=zRe});function GRe(t,e){return function(r,n){var i=Pt(r)?cte:dte,a=e?e():{};return i(r,t,pn(n,2),a)}}var mte,gte=N(()=>{"use strict";ute();pte();rs();Un();o(GRe,"createAggregator");mte=GRe});var VRe,PT,yte=N(()=>{"use strict";Lo();VRe=o(function(){return li.Date.now()},"now"),PT=VRe});var vte,URe,HRe,Qh,xte=N(()=>{"use strict";nm();Sd();Ld();Bh();vte=Object.prototype,URe=vte.hasOwnProperty,HRe=fc(function(t,e){t=Object(t);var r=-1,n=e.length,i=n>2?e[2]:void 0;for(i&&eo(e[0],e[1],i)&&(n=1);++r{"use strict";o(WRe,"arrayIncludesWith");BT=WRe});function YRe(t,e,r,n){var i=-1,a=aT,s=!0,l=t.length,u=[],h=e.length;if(!l)return u;r&&(e=Ns(e,Io(r))),n?(a=BT,s=!1):e.length>=qRe&&(a=Lm,s=!1,e=new Dm(e));e:for(;++i{"use strict";kT();sL();AL();Bd();_d();ST();qRe=200;o(YRe,"baseDifference");bte=YRe});var XRe,Zh,Tte=N(()=>{"use strict";wte();Cm();nm();ow();XRe=fc(function(t,e){return Ad(t)?bte(t,wc(e,1,Ad,!0)):[]}),Zh=XRe});function jRe(t){var e=t==null?0:t.length;return e?t[e-1]:void 0}var ga,kte=N(()=>{"use strict";o(jRe,"last");ga=jRe});function KRe(t,e,r){var n=t==null?0:t.length;return n?(e=r||e===void 0?1:vc(e),hT(t,e<0?0:e,n)):[]}var gi,Ete=N(()=>{"use strict";lL();bm();o(KRe,"drop");gi=KRe});function QRe(t,e,r){var n=t==null?0:t.length;return n?(e=r||e===void 0?1:vc(e),e=n-e,hT(t,0,e<0?0:e)):[]}var Nu,Ste=N(()=>{"use strict";lL();bm();o(QRe,"dropRight");Nu=QRe});function ZRe(t){return typeof t=="function"?t:ta}var Mm,FT=N(()=>{"use strict";Cu();o(ZRe,"castFunction");Mm=ZRe});function JRe(t,e){var r=Pt(t)?rT:Ms;return r(t,Mm(e))}var Ae,$T=N(()=>{"use strict";iL();Kh();FT();Un();o(JRe,"forEach");Ae=JRe});var Cte=N(()=>{"use strict";$T()});function eNe(t,e){for(var r=-1,n=t==null?0:t.length;++r{"use strict";o(eNe,"arrayEvery");Ate=eNe});function tNe(t,e){var r=!0;return Ms(t,function(n,i,a){return r=!!e(n,i,a),r}),r}var Dte,Lte=N(()=>{"use strict";Kh();o(tNe,"baseEvery");Dte=tNe});function rNe(t,e,r){var n=Pt(t)?Ate:Dte;return r&&eo(t,e,r)&&(e=void 0),n(t,pn(e,3))}var Ma,Rte=N(()=>{"use strict";_te();Lte();rs();Un();Ld();o(rNe,"every");Ma=rNe});function nNe(t,e){var r=[];return Ms(t,function(n,i,a){e(n,i,a)&&r.push(n)}),r}var zT,_L=N(()=>{"use strict";Kh();o(nNe,"baseFilter");zT=nNe});function iNe(t,e){var r=Pt(t)?Am:zT;return r(t,pn(e,3))}var Yr,DL=N(()=>{"use strict";fT();_L();rs();Un();o(iNe,"filter");Yr=iNe});function aNe(t){return function(e,r,n){var i=Object(e);if(!ci(e)){var a=pn(r,3);e=zr(e),r=o(function(l){return a(i[l],l,i)},"predicate")}var s=t(e,r,n);return s>-1?i[a?e[s]:s]:void 0}}var Nte,Mte=N(()=>{"use strict";rs();Mo();xc();o(aNe,"createFind");Nte=aNe});function oNe(t,e,r){var n=t==null?0:t.length;if(!n)return-1;var i=r==null?0:vc(r);return i<0&&(i=sNe(n+i,0)),nT(t,pn(e,3),i)}var sNe,Ite,Ote=N(()=>{"use strict";aL();rs();bm();sNe=Math.max;o(oNe,"findIndex");Ite=oNe});var lNe,ns,Pte=N(()=>{"use strict";Mte();Ote();lNe=Nte(Ite),ns=lNe});function cNe(t){return t&&t.length?t[0]:void 0}var ia,Bte=N(()=>{"use strict";o(cNe,"head");ia=cNe});var Fte=N(()=>{"use strict";Bte()});function uNe(t,e){var r=-1,n=ci(t)?Array(t.length):[];return Ms(t,function(i,a,s){n[++r]=e(i,a,s)}),n}var GT,LL=N(()=>{"use strict";Kh();Mo();o(uNe,"baseMap");GT=uNe});function hNe(t,e){var r=Pt(t)?Ns:GT;return r(t,pn(e,3))}var Je,Im=N(()=>{"use strict";Bd();rs();LL();Un();o(hNe,"map");Je=hNe});function fNe(t,e){return wc(Je(t,e),1)}var ya,RL=N(()=>{"use strict";Cm();Im();o(fNe,"flatMap");ya=fNe});function dNe(t,e){return t==null?t:X0(t,Mm(e),Cs)}var NL,$te=N(()=>{"use strict";Z5();FT();Bh();o(dNe,"forIn");NL=dNe});function pNe(t,e){return t&&Nm(t,Mm(e))}var ML,zte=N(()=>{"use strict";OT();FT();o(pNe,"forOwn");ML=pNe});var mNe,gNe,yNe,IL,Gte=N(()=>{"use strict";Y0();gte();mNe=Object.prototype,gNe=mNe.hasOwnProperty,yNe=mte(function(t,e,r){gNe.call(t,r)?t[r].push(e):cc(t,r,[e])}),IL=yNe});function vNe(t,e){return t>e}var Vte,Ute=N(()=>{"use strict";o(vNe,"baseGt");Vte=vNe});function wNe(t,e){return t!=null&&bNe.call(t,e)}var xNe,bNe,Hte,Wte=N(()=>{"use strict";xNe=Object.prototype,bNe=xNe.hasOwnProperty;o(wNe,"baseHas");Hte=wNe});function TNe(t,e){return t!=null&&NT(t,e,Hte)}var Bt,qte=N(()=>{"use strict";Wte();EL();o(TNe,"has");Bt=TNe});function ENe(t){return typeof t=="string"||!Pt(t)&&ri(t)&&da(t)==kNe}var kNe,yi,VT=N(()=>{"use strict";ku();Un();No();kNe="[object String]";o(ENe,"isString");yi=ENe});function SNe(t,e){return Ns(e,function(r){return t[r]})}var Yte,Xte=N(()=>{"use strict";Bd();o(SNe,"baseValues");Yte=SNe});function CNe(t){return t==null?[]:Yte(t,zr(t))}var br,OL=N(()=>{"use strict";Xte();xc();o(CNe,"values");br=CNe});function _Ne(t,e,r,n){t=ci(t)?t:br(t),r=r&&!n?vc(r):0;var i=t.length;return r<0&&(r=ANe(i+r,0)),yi(t)?r<=i&&t.indexOf(e,r)>-1:!!i&&wm(t,e,r)>-1}var ANe,qn,jte=N(()=>{"use strict";iT();Mo();VT();bm();OL();ANe=Math.max;o(_Ne,"includes");qn=_Ne});function LNe(t,e,r){var n=t==null?0:t.length;if(!n)return-1;var i=r==null?0:vc(r);return i<0&&(i=DNe(n+i,0)),wm(t,e,i)}var DNe,UT,Kte=N(()=>{"use strict";iT();bm();DNe=Math.max;o(LNe,"indexOf");UT=LNe});function ONe(t){if(t==null)return!0;if(ci(t)&&(Pt(t)||typeof t=="string"||typeof t.splice=="function"||Sl(t)||Oh(t)||El(t)))return!t.length;var e=io(t);if(e==RNe||e==NNe)return!t.size;if(uc(t))return!Tm(t).length;for(var r in t)if(INe.call(t,r))return!1;return!0}var RNe,NNe,MNe,INe,ur,HT=N(()=>{"use strict";sT();$d();J0();Un();Mo();tm();Z0();r2();RNe="[object Map]",NNe="[object Set]",MNe=Object.prototype,INe=MNe.hasOwnProperty;o(ONe,"isEmpty");ur=ONe});function BNe(t){return ri(t)&&da(t)==PNe}var PNe,Qte,Zte=N(()=>{"use strict";ku();No();PNe="[object RegExp]";o(BNe,"baseIsRegExp");Qte=BNe});var Jte,FNe,zo,ere=N(()=>{"use strict";Zte();_d();t2();Jte=Oo&&Oo.isRegExp,FNe=Jte?Io(Jte):Qte,zo=FNe});function $Ne(t){return t===void 0}var pr,tre=N(()=>{"use strict";o($Ne,"isUndefined");pr=$Ne});function zNe(t,e){return t{"use strict";o(zNe,"baseLt");WT=zNe});function GNe(t,e){var r={};return e=pn(e,3),Nm(t,function(n,i,a){cc(r,i,e(n,i,a))}),r}var zd,rre=N(()=>{"use strict";Y0();OT();rs();o(GNe,"mapValues");zd=GNe});function VNe(t,e,r){for(var n=-1,i=t.length;++n{"use strict";Pd();o(VNe,"baseExtremum");Om=VNe});function UNe(t){return t&&t.length?Om(t,ta,Vte):void 0}var Is,nre=N(()=>{"use strict";qT();Ute();Cu();o(UNe,"max");Is=UNe});function HNe(t){return t&&t.length?Om(t,ta,WT):void 0}var Dl,BL=N(()=>{"use strict";qT();PL();Cu();o(HNe,"min");Dl=HNe});function WNe(t,e){return t&&t.length?Om(t,pn(e,2),WT):void 0}var Gd,ire=N(()=>{"use strict";qT();rs();PL();o(WNe,"minBy");Gd=WNe});function YNe(t){if(typeof t!="function")throw new TypeError(qNe);return function(){var e=arguments;switch(e.length){case 0:return!t.call(this);case 1:return!t.call(this,e[0]);case 2:return!t.call(this,e[0],e[1]);case 3:return!t.call(this,e[0],e[1],e[2])}return!t.apply(this,e)}}var qNe,are,sre=N(()=>{"use strict";qNe="Expected a function";o(YNe,"negate");are=YNe});function XNe(t,e,r,n){if(!bn(t))return t;e=Yh(e,t);for(var i=-1,a=e.length,s=a-1,l=t;l!=null&&++i{"use strict";rm();E2();i2();Js();Em();o(XNe,"baseSet");ore=XNe});function jNe(t,e,r){for(var n=-1,i=e.length,a={};++n{"use strict";S2();lre();E2();o(jNe,"basePickBy");YT=jNe});function KNe(t,e){if(t==null)return{};var r=Ns(yT(t),function(n){return[n]});return e=pn(e),YT(t,r,function(n,i){return e(n,i[0])})}var Os,cre=N(()=>{"use strict";Bd();rs();FL();dL();o(KNe,"pickBy");Os=KNe});function QNe(t,e){var r=t.length;for(t.sort(e);r--;)t[r]=t[r].value;return t}var ure,hre=N(()=>{"use strict";o(QNe,"baseSortBy");ure=QNe});function ZNe(t,e){if(t!==e){var r=t!==void 0,n=t===null,i=t===t,a=no(t),s=e!==void 0,l=e===null,u=e===e,h=no(e);if(!l&&!h&&!a&&t>e||a&&s&&u&&!l&&!h||n&&s&&u||!r&&u||!i)return 1;if(!n&&!a&&!h&&t{"use strict";Pd();o(ZNe,"compareAscending");fre=ZNe});function JNe(t,e,r){for(var n=-1,i=t.criteria,a=e.criteria,s=i.length,l=r.length;++n=l)return u;var h=r[n];return u*(h=="desc"?-1:1)}}return t.index-e.index}var pre,mre=N(()=>{"use strict";dre();o(JNe,"compareMultiple");pre=JNe});function eMe(t,e,r){e.length?e=Ns(e,function(a){return Pt(a)?function(s){return Xh(s,a.length===1?a[0]:a)}:a}):e=[ta];var n=-1;e=Ns(e,Io(pn));var i=GT(t,function(a,s,l){var u=Ns(e,function(h){return h(a)});return{criteria:u,index:++n,value:a}});return ure(i,function(a,s){return pre(a,s,r)})}var gre,yre=N(()=>{"use strict";Bd();S2();rs();LL();hre();_d();mre();Cu();Un();o(eMe,"baseOrderBy");gre=eMe});var tMe,vre,xre=N(()=>{"use strict";CL();tMe=IT("length"),vre=tMe});function dMe(t){for(var e=bre.lastIndex=0;bre.test(t);)++e;return e}var wre,rMe,nMe,iMe,aMe,sMe,oMe,$L,zL,lMe,Tre,kre,Ere,cMe,Sre,Cre,uMe,hMe,fMe,bre,Are,_re=N(()=>{"use strict";wre="\\ud800-\\udfff",rMe="\\u0300-\\u036f",nMe="\\ufe20-\\ufe2f",iMe="\\u20d0-\\u20ff",aMe=rMe+nMe+iMe,sMe="\\ufe0e\\ufe0f",oMe="["+wre+"]",$L="["+aMe+"]",zL="\\ud83c[\\udffb-\\udfff]",lMe="(?:"+$L+"|"+zL+")",Tre="[^"+wre+"]",kre="(?:\\ud83c[\\udde6-\\uddff]){2}",Ere="[\\ud800-\\udbff][\\udc00-\\udfff]",cMe="\\u200d",Sre=lMe+"?",Cre="["+sMe+"]?",uMe="(?:"+cMe+"(?:"+[Tre,kre,Ere].join("|")+")"+Cre+Sre+")*",hMe=Cre+Sre+uMe,fMe="(?:"+[Tre+$L+"?",$L,kre,Ere,oMe].join("|")+")",bre=RegExp(zL+"(?="+zL+")|"+fMe+hMe,"g");o(dMe,"unicodeSize");Are=dMe});function pMe(t){return $J(t)?Are(t):vre(t)}var Dre,Lre=N(()=>{"use strict";xre();zJ();_re();o(pMe,"stringSize");Dre=pMe});function mMe(t,e){return YT(t,e,function(r,n){return MT(t,n)})}var Rre,Nre=N(()=>{"use strict";FL();SL();o(mMe,"basePick");Rre=mMe});var gMe,Vd,Mre=N(()=>{"use strict";Nre();FJ();gMe=BJ(function(t,e){return t==null?{}:Rre(t,e)}),Vd=gMe});function xMe(t,e,r,n){for(var i=-1,a=vMe(yMe((e-t)/(r||1)),0),s=Array(a);a--;)s[n?a:++i]=t,t+=r;return s}var yMe,vMe,Ire,Ore=N(()=>{"use strict";yMe=Math.ceil,vMe=Math.max;o(xMe,"baseRange");Ire=xMe});function bMe(t){return function(e,r,n){return n&&typeof n!="number"&&eo(e,r,n)&&(r=n=void 0),e=xm(e),r===void 0?(r=e,e=0):r=xm(r),n=n===void 0?e{"use strict";Ore();Ld();rL();o(bMe,"createRange");Pre=bMe});var wMe,Go,Fre=N(()=>{"use strict";Bre();wMe=Pre(),Go=wMe});function TMe(t,e,r,n,i){return i(t,function(a,s,l){r=n?(n=!1,a):e(r,a,s,l)}),r}var $re,zre=N(()=>{"use strict";o(TMe,"baseReduce");$re=TMe});function kMe(t,e,r){var n=Pt(t)?GJ:$re,i=arguments.length<3;return n(t,pn(e,4),r,i,Ms)}var Xr,GL=N(()=>{"use strict";VJ();Kh();rs();zre();Un();o(kMe,"reduce");Xr=kMe});function EMe(t,e){var r=Pt(t)?Am:zT;return r(t,are(pn(e,3)))}var Jh,Gre=N(()=>{"use strict";fT();_L();rs();Un();sre();o(EMe,"reject");Jh=EMe});function AMe(t){if(t==null)return 0;if(ci(t))return yi(t)?Dre(t):t.length;var e=io(t);return e==SMe||e==CMe?t.size:Tm(t).length}var SMe,CMe,VL,Vre=N(()=>{"use strict";sT();$d();Mo();VT();Lre();SMe="[object Map]",CMe="[object Set]";o(AMe,"size");VL=AMe});function _Me(t,e){var r;return Ms(t,function(n,i,a){return r=e(n,i,a),!r}),!!r}var Ure,Hre=N(()=>{"use strict";Kh();o(_Me,"baseSome");Ure=_Me});function DMe(t,e,r){var n=Pt(t)?ET:Ure;return r&&eo(t,e,r)&&(e=void 0),n(t,pn(e,3))}var A2,Wre=N(()=>{"use strict";vL();rs();Hre();Un();Ld();o(DMe,"some");A2=DMe});var LMe,kc,qre=N(()=>{"use strict";Cm();yre();nm();Ld();LMe=fc(function(t,e){if(t==null)return[];var r=e.length;return r>1&&eo(t,e[0],e[1])?e=[]:r>2&&eo(e[0],e[1],e[2])&&(e=[e[0]]),gre(t,wc(e,1),[])}),kc=LMe});var RMe,NMe,Yre,Xre=N(()=>{"use strict";pL();nL();AT();RMe=1/0,NMe=jh&&1/Rm(new jh([,-0]))[1]==RMe?function(t){return new jh(t)}:ni,Yre=NMe});function IMe(t,e,r){var n=-1,i=aT,a=t.length,s=!0,l=[],u=l;if(r)s=!1,i=BT;else if(a>=MMe){var h=e?null:Yre(t);if(h)return Rm(h);s=!1,i=Lm,u=new Dm}else u=e?[]:l;e:for(;++n{"use strict";kT();sL();AL();ST();Xre();AT();MMe=200;o(IMe,"baseUniq");Pm=IMe});var OMe,UL,jre=N(()=>{"use strict";Cm();nm();XT();ow();OMe=fc(function(t){return Pm(wc(t,1,Ad,!0))}),UL=OMe});function PMe(t){return t&&t.length?Pm(t):[]}var Bm,Kre=N(()=>{"use strict";XT();o(PMe,"uniq");Bm=PMe});function BMe(t,e){return t&&t.length?Pm(t,pn(e,2)):[]}var Qre,Zre=N(()=>{"use strict";rs();XT();o(BMe,"uniqBy");Qre=BMe});function $Me(t){var e=++FMe;return lT(t)+e}var FMe,Ud,Jre=N(()=>{"use strict";oL();FMe=0;o($Me,"uniqueId");Ud=$Me});function zMe(t,e,r){for(var n=-1,i=t.length,a=e.length,s={};++n{"use strict";o(zMe,"baseZipObject");ene=zMe});function GMe(t,e){return ene(t||[],e||[],hc)}var jT,rne=N(()=>{"use strict";rm();tne();o(GMe,"zipObject");jT=GMe});var qt=N(()=>{"use strict";CJ();gL();Lee();Ree();$9();xte();Tte();Ete();Ste();Cte();Rte();DL();Pte();Fte();RL();uT();$T();$te();zte();Gte();qte();Cu();jte();Kte();Un();HT();Yv();Js();ere();VT();tre();xc();kte();Im();rre();nre();V9();BL();ire();nL();yte();Mre();cre();Fre();GL();Gre();Vre();Wre();qre();jre();Kre();Jre();OL();rne();});function ine(t,e){t[e]?t[e]++:t[e]=1}function ane(t,e){--t[e]||delete t[e]}function _2(t,e,r,n){var i=""+e,a=""+r;if(!t&&i>a){var s=i;i=a,a=s}return i+nne+a+nne+(pr(n)?VMe:n)}function UMe(t,e,r,n){var i=""+e,a=""+r;if(!t&&i>a){var s=i;i=a,a=s}var l={v:i,w:a};return n&&(l.name=n),l}function HL(t,e){return _2(t,e.v,e.w,e.name)}var VMe,Hd,nne,sn,KT=N(()=>{"use strict";qt();VMe="\0",Hd="\0",nne="",sn=class{static{o(this,"Graph")}constructor(e={}){this._isDirected=Object.prototype.hasOwnProperty.call(e,"directed")?e.directed:!0,this._isMultigraph=Object.prototype.hasOwnProperty.call(e,"multigraph")?e.multigraph:!1,this._isCompound=Object.prototype.hasOwnProperty.call(e,"compound")?e.compound:!1,this._label=void 0,this._defaultNodeLabelFn=As(void 0),this._defaultEdgeLabelFn=As(void 0),this._nodes={},this._isCompound&&(this._parent={},this._children={},this._children[Hd]={}),this._in={},this._preds={},this._out={},this._sucs={},this._edgeObjs={},this._edgeLabels={}}isDirected(){return this._isDirected}isMultigraph(){return this._isMultigraph}isCompound(){return this._isCompound}setGraph(e){return this._label=e,this}graph(){return this._label}setDefaultNodeLabel(e){return Si(e)||(e=As(e)),this._defaultNodeLabelFn=e,this}nodeCount(){return this._nodeCount}nodes(){return zr(this._nodes)}sources(){var e=this;return Yr(this.nodes(),function(r){return ur(e._in[r])})}sinks(){var e=this;return Yr(this.nodes(),function(r){return ur(e._out[r])})}setNodes(e,r){var n=arguments,i=this;return Ae(e,function(a){n.length>1?i.setNode(a,r):i.setNode(a)}),this}setNode(e,r){return Object.prototype.hasOwnProperty.call(this._nodes,e)?(arguments.length>1&&(this._nodes[e]=r),this):(this._nodes[e]=arguments.length>1?r:this._defaultNodeLabelFn(e),this._isCompound&&(this._parent[e]=Hd,this._children[e]={},this._children[Hd][e]=!0),this._in[e]={},this._preds[e]={},this._out[e]={},this._sucs[e]={},++this._nodeCount,this)}node(e){return this._nodes[e]}hasNode(e){return Object.prototype.hasOwnProperty.call(this._nodes,e)}removeNode(e){if(Object.prototype.hasOwnProperty.call(this._nodes,e)){var r=o(n=>this.removeEdge(this._edgeObjs[n]),"removeEdge");delete this._nodes[e],this._isCompound&&(this._removeFromParentsChildList(e),delete this._parent[e],Ae(this.children(e),n=>{this.setParent(n)}),delete this._children[e]),Ae(zr(this._in[e]),r),delete this._in[e],delete this._preds[e],Ae(zr(this._out[e]),r),delete this._out[e],delete this._sucs[e],--this._nodeCount}return this}setParent(e,r){if(!this._isCompound)throw new Error("Cannot set parent in a non-compound graph");if(pr(r))r=Hd;else{r+="";for(var n=r;!pr(n);n=this.parent(n))if(n===e)throw new Error("Setting "+r+" as parent of "+e+" would create a cycle");this.setNode(r)}return this.setNode(e),this._removeFromParentsChildList(e),this._parent[e]=r,this._children[r][e]=!0,this}_removeFromParentsChildList(e){delete this._children[this._parent[e]][e]}parent(e){if(this._isCompound){var r=this._parent[e];if(r!==Hd)return r}}children(e){if(pr(e)&&(e=Hd),this._isCompound){var r=this._children[e];if(r)return zr(r)}else{if(e===Hd)return this.nodes();if(this.hasNode(e))return[]}}predecessors(e){var r=this._preds[e];if(r)return zr(r)}successors(e){var r=this._sucs[e];if(r)return zr(r)}neighbors(e){var r=this.predecessors(e);if(r)return UL(r,this.successors(e))}isLeaf(e){var r;return this.isDirected()?r=this.successors(e):r=this.neighbors(e),r.length===0}filterNodes(e){var r=new this.constructor({directed:this._isDirected,multigraph:this._isMultigraph,compound:this._isCompound});r.setGraph(this.graph());var n=this;Ae(this._nodes,function(s,l){e(l)&&r.setNode(l,s)}),Ae(this._edgeObjs,function(s){r.hasNode(s.v)&&r.hasNode(s.w)&&r.setEdge(s,n.edge(s))});var i={};function a(s){var l=n.parent(s);return l===void 0||r.hasNode(l)?(i[s]=l,l):l in i?i[l]:a(l)}return o(a,"findParent"),this._isCompound&&Ae(r.nodes(),function(s){r.setParent(s,a(s))}),r}setDefaultEdgeLabel(e){return Si(e)||(e=As(e)),this._defaultEdgeLabelFn=e,this}edgeCount(){return this._edgeCount}edges(){return br(this._edgeObjs)}setPath(e,r){var n=this,i=arguments;return Xr(e,function(a,s){return i.length>1?n.setEdge(a,s,r):n.setEdge(a,s),s}),this}setEdge(){var e,r,n,i,a=!1,s=arguments[0];typeof s=="object"&&s!==null&&"v"in s?(e=s.v,r=s.w,n=s.name,arguments.length===2&&(i=arguments[1],a=!0)):(e=s,r=arguments[1],n=arguments[3],arguments.length>2&&(i=arguments[2],a=!0)),e=""+e,r=""+r,pr(n)||(n=""+n);var l=_2(this._isDirected,e,r,n);if(Object.prototype.hasOwnProperty.call(this._edgeLabels,l))return a&&(this._edgeLabels[l]=i),this;if(!pr(n)&&!this._isMultigraph)throw new Error("Cannot set a named edge when isMultigraph = false");this.setNode(e),this.setNode(r),this._edgeLabels[l]=a?i:this._defaultEdgeLabelFn(e,r,n);var u=UMe(this._isDirected,e,r,n);return e=u.v,r=u.w,Object.freeze(u),this._edgeObjs[l]=u,ine(this._preds[r],e),ine(this._sucs[e],r),this._in[r][l]=u,this._out[e][l]=u,this._edgeCount++,this}edge(e,r,n){var i=arguments.length===1?HL(this._isDirected,arguments[0]):_2(this._isDirected,e,r,n);return this._edgeLabels[i]}hasEdge(e,r,n){var i=arguments.length===1?HL(this._isDirected,arguments[0]):_2(this._isDirected,e,r,n);return Object.prototype.hasOwnProperty.call(this._edgeLabels,i)}removeEdge(e,r,n){var i=arguments.length===1?HL(this._isDirected,arguments[0]):_2(this._isDirected,e,r,n),a=this._edgeObjs[i];return a&&(e=a.v,r=a.w,delete this._edgeLabels[i],delete this._edgeObjs[i],ane(this._preds[r],e),ane(this._sucs[e],r),delete this._in[r][i],delete this._out[e][i],this._edgeCount--),this}inEdges(e,r){var n=this._in[e];if(n){var i=br(n);return r?Yr(i,function(a){return a.v===r}):i}}outEdges(e,r){var n=this._out[e];if(n){var i=br(n);return r?Yr(i,function(a){return a.w===r}):i}}nodeEdges(e,r){var n=this.inEdges(e,r);if(n)return n.concat(this.outEdges(e,r))}};sn.prototype._nodeCount=0;sn.prototype._edgeCount=0;o(ine,"incrementOrInitEntry");o(ane,"decrementOrRemoveEntry");o(_2,"edgeArgsToId");o(UMe,"edgeArgsToObj");o(HL,"edgeObjToId")});var Vo=N(()=>{"use strict";KT()});function sne(t){t._prev._next=t._next,t._next._prev=t._prev,delete t._next,delete t._prev}function HMe(t,e){if(t!=="_next"&&t!=="_prev")return e}var ZT,one=N(()=>{"use strict";ZT=class{static{o(this,"List")}constructor(){var e={};e._next=e._prev=e,this._sentinel=e}dequeue(){var e=this._sentinel,r=e._prev;if(r!==e)return sne(r),r}enqueue(e){var r=this._sentinel;e._prev&&e._next&&sne(e),e._next=r._next,r._next._prev=e,r._next=e,e._prev=r}toString(){for(var e=[],r=this._sentinel,n=r._prev;n!==r;)e.push(JSON.stringify(n,HMe)),n=n._prev;return"["+e.join(", ")+"]"}};o(sne,"unlink");o(HMe,"filterOutLinks")});function lne(t,e){if(t.nodeCount()<=1)return[];var r=YMe(t,e||WMe),n=qMe(r.graph,r.buckets,r.zeroIdx);return qr(Je(n,function(i){return t.outEdges(i.v,i.w)}))}function qMe(t,e,r){for(var n=[],i=e[e.length-1],a=e[0],s;t.nodeCount();){for(;s=a.dequeue();)WL(t,e,r,s);for(;s=i.dequeue();)WL(t,e,r,s);if(t.nodeCount()){for(var l=e.length-2;l>0;--l)if(s=e[l].dequeue(),s){n=n.concat(WL(t,e,r,s,!0));break}}}return n}function WL(t,e,r,n,i){var a=i?[]:void 0;return Ae(t.inEdges(n.v),function(s){var l=t.edge(s),u=t.node(s.v);i&&a.push({v:s.v,w:s.w}),u.out-=l,qL(e,r,u)}),Ae(t.outEdges(n.v),function(s){var l=t.edge(s),u=s.w,h=t.node(u);h.in-=l,qL(e,r,h)}),t.removeNode(n.v),a}function YMe(t,e){var r=new sn,n=0,i=0;Ae(t.nodes(),function(l){r.setNode(l,{v:l,in:0,out:0})}),Ae(t.edges(),function(l){var u=r.edge(l.v,l.w)||0,h=e(l),f=u+h;r.setEdge(l.v,l.w,f),i=Math.max(i,r.node(l.v).out+=h),n=Math.max(n,r.node(l.w).in+=h)});var a=Go(i+n+3).map(function(){return new ZT}),s=n+1;return Ae(r.nodes(),function(l){qL(a,s,r.node(l))}),{graph:r,buckets:a,zeroIdx:s}}function qL(t,e,r){r.out?r.in?t[r.out-r.in+e].enqueue(r):t[t.length-1].enqueue(r):t[0].enqueue(r)}var WMe,cne=N(()=>{"use strict";qt();Vo();one();WMe=As(1);o(lne,"greedyFAS");o(qMe,"doGreedyFAS");o(WL,"removeNode");o(YMe,"buildState");o(qL,"assignBucket")});function une(t){var e=t.graph().acyclicer==="greedy"?lne(t,r(t)):XMe(t);Ae(e,function(n){var i=t.edge(n);t.removeEdge(n),i.forwardName=n.name,i.reversed=!0,t.setEdge(n.w,n.v,i,Ud("rev"))});function r(n){return function(i){return n.edge(i).weight}}o(r,"weightFn")}function XMe(t){var e=[],r={},n={};function i(a){Object.prototype.hasOwnProperty.call(n,a)||(n[a]=!0,r[a]=!0,Ae(t.outEdges(a),function(s){Object.prototype.hasOwnProperty.call(r,s.w)?e.push(s):i(s.w)}),delete r[a])}return o(i,"dfs"),Ae(t.nodes(),i),e}function hne(t){Ae(t.edges(),function(e){var r=t.edge(e);if(r.reversed){t.removeEdge(e);var n=r.forwardName;delete r.reversed,delete r.forwardName,t.setEdge(e.w,e.v,r,n)}})}var YL=N(()=>{"use strict";qt();cne();o(une,"run");o(XMe,"dfsFAS");o(hne,"undo")});function Ec(t,e,r,n){var i;do i=Ud(n);while(t.hasNode(i));return r.dummy=e,t.setNode(i,r),i}function dne(t){var e=new sn().setGraph(t.graph());return Ae(t.nodes(),function(r){e.setNode(r,t.node(r))}),Ae(t.edges(),function(r){var n=e.edge(r.v,r.w)||{weight:0,minlen:1},i=t.edge(r);e.setEdge(r.v,r.w,{weight:n.weight+i.weight,minlen:Math.max(n.minlen,i.minlen)})}),e}function JT(t){var e=new sn({multigraph:t.isMultigraph()}).setGraph(t.graph());return Ae(t.nodes(),function(r){t.children(r).length||e.setNode(r,t.node(r))}),Ae(t.edges(),function(r){e.setEdge(r,t.edge(r))}),e}function XL(t,e){var r=t.x,n=t.y,i=e.x-r,a=e.y-n,s=t.width/2,l=t.height/2;if(!i&&!a)throw new Error("Not possible to find intersection inside of the rectangle");var u,h;return Math.abs(a)*s>Math.abs(i)*l?(a<0&&(l=-l),u=l*i/a,h=l):(i<0&&(s=-s),u=s,h=s*a/i),{x:r+u,y:n+h}}function ef(t){var e=Je(Go(KL(t)+1),function(){return[]});return Ae(t.nodes(),function(r){var n=t.node(r),i=n.rank;pr(i)||(e[i][n.order]=r)}),e}function pne(t){var e=Dl(Je(t.nodes(),function(r){return t.node(r).rank}));Ae(t.nodes(),function(r){var n=t.node(r);Bt(n,"rank")&&(n.rank-=e)})}function mne(t){var e=Dl(Je(t.nodes(),function(a){return t.node(a).rank})),r=[];Ae(t.nodes(),function(a){var s=t.node(a).rank-e;r[s]||(r[s]=[]),r[s].push(a)});var n=0,i=t.graph().nodeRankFactor;Ae(r,function(a,s){pr(a)&&s%i!==0?--n:n&&Ae(a,function(l){t.node(l).rank+=n})})}function jL(t,e,r,n){var i={width:0,height:0};return arguments.length>=4&&(i.rank=r,i.order=n),Ec(t,"border",i,e)}function KL(t){return Is(Je(t.nodes(),function(e){var r=t.node(e).rank;if(!pr(r))return r}))}function gne(t,e){var r={lhs:[],rhs:[]};return Ae(t,function(n){e(n)?r.lhs.push(n):r.rhs.push(n)}),r}function yne(t,e){var r=PT();try{return e()}finally{console.log(t+" time: "+(PT()-r)+"ms")}}function vne(t,e){return e()}var Sc=N(()=>{"use strict";qt();Vo();o(Ec,"addDummyNode");o(dne,"simplify");o(JT,"asNonCompoundGraph");o(XL,"intersectRect");o(ef,"buildLayerMatrix");o(pne,"normalizeRanks");o(mne,"removeEmptyRanks");o(jL,"addBorderNode");o(KL,"maxRank");o(gne,"partition");o(yne,"time");o(vne,"notime")});function bne(t){function e(r){var n=t.children(r),i=t.node(r);if(n.length&&Ae(n,e),Object.prototype.hasOwnProperty.call(i,"minRank")){i.borderLeft=[],i.borderRight=[];for(var a=i.minRank,s=i.maxRank+1;a{"use strict";qt();Sc();o(bne,"addBorderSegments");o(xne,"addBorderNode")});function kne(t){var e=t.graph().rankdir.toLowerCase();(e==="lr"||e==="rl")&&Sne(t)}function Ene(t){var e=t.graph().rankdir.toLowerCase();(e==="bt"||e==="rl")&&jMe(t),(e==="lr"||e==="rl")&&(KMe(t),Sne(t))}function Sne(t){Ae(t.nodes(),function(e){Tne(t.node(e))}),Ae(t.edges(),function(e){Tne(t.edge(e))})}function Tne(t){var e=t.width;t.width=t.height,t.height=e}function jMe(t){Ae(t.nodes(),function(e){QL(t.node(e))}),Ae(t.edges(),function(e){var r=t.edge(e);Ae(r.points,QL),Object.prototype.hasOwnProperty.call(r,"y")&&QL(r)})}function QL(t){t.y=-t.y}function KMe(t){Ae(t.nodes(),function(e){ZL(t.node(e))}),Ae(t.edges(),function(e){var r=t.edge(e);Ae(r.points,ZL),Object.prototype.hasOwnProperty.call(r,"x")&&ZL(r)})}function ZL(t){var e=t.x;t.x=t.y,t.y=e}var Cne=N(()=>{"use strict";qt();o(kne,"adjust");o(Ene,"undo");o(Sne,"swapWidthHeight");o(Tne,"swapWidthHeightOne");o(jMe,"reverseY");o(QL,"reverseYOne");o(KMe,"swapXY");o(ZL,"swapXYOne")});function Ane(t){t.graph().dummyChains=[],Ae(t.edges(),function(e){ZMe(t,e)})}function ZMe(t,e){var r=e.v,n=t.node(r).rank,i=e.w,a=t.node(i).rank,s=e.name,l=t.edge(e),u=l.labelRank;if(a!==n+1){t.removeEdge(e);var h=void 0,f,d;for(d=0,++n;n{"use strict";qt();Sc();o(Ane,"run");o(ZMe,"normalizeEdge");o(_ne,"undo")});function D2(t){var e={};function r(n){var i=t.node(n);if(Object.prototype.hasOwnProperty.call(e,n))return i.rank;e[n]=!0;var a=Dl(Je(t.outEdges(n),function(s){return r(s.w)-t.edge(s).minlen}));return(a===Number.POSITIVE_INFINITY||a===void 0||a===null)&&(a=0),i.rank=a}o(r,"dfs"),Ae(t.sources(),r)}function Wd(t,e){return t.node(e.w).rank-t.node(e.v).rank-t.edge(e).minlen}var ek=N(()=>{"use strict";qt();o(D2,"longestPath");o(Wd,"slack")});function tk(t){var e=new sn({directed:!1}),r=t.nodes()[0],n=t.nodeCount();e.setNode(r,{});for(var i,a;JMe(e,t){"use strict";qt();Vo();ek();o(tk,"feasibleTree");o(JMe,"tightTree");o(eIe,"findMinSlackEdge");o(tIe,"shiftRanks")});var Lne=N(()=>{"use strict"});var tR=N(()=>{"use strict"});var cWt,rR=N(()=>{"use strict";qt();tR();cWt=As(1)});var Rne=N(()=>{"use strict";rR()});var nR=N(()=>{"use strict"});var Nne=N(()=>{"use strict";nR()});var bWt,Mne=N(()=>{"use strict";qt();bWt=As(1)});function iR(t){var e={},r={},n=[];function i(a){if(Object.prototype.hasOwnProperty.call(r,a))throw new L2;Object.prototype.hasOwnProperty.call(e,a)||(r[a]=!0,e[a]=!0,Ae(t.predecessors(a),i),delete r[a],n.push(a))}if(o(i,"visit"),Ae(t.sinks(),i),VL(e)!==t.nodeCount())throw new L2;return n}function L2(){}var aR=N(()=>{"use strict";qt();iR.CycleException=L2;o(iR,"topsort");o(L2,"CycleException");L2.prototype=new Error});var Ine=N(()=>{"use strict";aR()});function rk(t,e,r){Pt(e)||(e=[e]);var n=(t.isDirected()?t.successors:t.neighbors).bind(t),i=[],a={};return Ae(e,function(s){if(!t.hasNode(s))throw new Error("Graph does not have node: "+s);One(t,s,r==="post",a,n,i)}),i}function One(t,e,r,n,i,a){Object.prototype.hasOwnProperty.call(n,e)||(n[e]=!0,r||a.push(e),Ae(i(e),function(s){One(t,s,r,n,i,a)}),r&&a.push(e))}var sR=N(()=>{"use strict";qt();o(rk,"dfs");o(One,"doDfs")});function oR(t,e){return rk(t,e,"post")}var Pne=N(()=>{"use strict";sR();o(oR,"postorder")});function lR(t,e){return rk(t,e,"pre")}var Bne=N(()=>{"use strict";sR();o(lR,"preorder")});var Fne=N(()=>{"use strict";tR();KT()});var $ne=N(()=>{"use strict";Lne();rR();Rne();Nne();Mne();Ine();Pne();Bne();Fne();nR();aR()});function rf(t){t=dne(t),D2(t);var e=tk(t);uR(e),cR(e,t);for(var r,n;r=Une(e);)n=Hne(e,t,r),Wne(e,t,r,n)}function cR(t,e){var r=oR(t,t.nodes());r=r.slice(0,r.length-1),Ae(r,function(n){sIe(t,e,n)})}function sIe(t,e,r){var n=t.node(r),i=n.parent;t.edge(r,i).cutvalue=Gne(t,e,r)}function Gne(t,e,r){var n=t.node(r),i=n.parent,a=!0,s=e.edge(r,i),l=0;return s||(a=!1,s=e.edge(i,r)),l=s.weight,Ae(e.nodeEdges(r),function(u){var h=u.v===r,f=h?u.w:u.v;if(f!==i){var d=h===a,p=e.edge(u).weight;if(l+=d?p:-p,lIe(t,r,f)){var m=t.edge(r,f).cutvalue;l+=d?-m:m}}}),l}function uR(t,e){arguments.length<2&&(e=t.nodes()[0]),Vne(t,{},1,e)}function Vne(t,e,r,n,i){var a=r,s=t.node(n);return e[n]=!0,Ae(t.neighbors(n),function(l){Object.prototype.hasOwnProperty.call(e,l)||(r=Vne(t,e,r,l,n))}),s.low=a,s.lim=r++,i?s.parent=i:delete s.parent,r}function Une(t){return ns(t.edges(),function(e){return t.edge(e).cutvalue<0})}function Hne(t,e,r){var n=r.v,i=r.w;e.hasEdge(n,i)||(n=r.w,i=r.v);var a=t.node(n),s=t.node(i),l=a,u=!1;a.lim>s.lim&&(l=s,u=!0);var h=Yr(e.edges(),function(f){return u===zne(t,t.node(f.v),l)&&u!==zne(t,t.node(f.w),l)});return Gd(h,function(f){return Wd(e,f)})}function Wne(t,e,r,n){var i=r.v,a=r.w;t.removeEdge(i,a),t.setEdge(n.v,n.w,{}),uR(t),cR(t,e),oIe(t,e)}function oIe(t,e){var r=ns(t.nodes(),function(i){return!e.node(i).parent}),n=lR(t,r);n=n.slice(1),Ae(n,function(i){var a=t.node(i).parent,s=e.edge(i,a),l=!1;s||(s=e.edge(a,i),l=!0),e.node(i).rank=e.node(a).rank+(l?s.minlen:-s.minlen)})}function lIe(t,e,r){return t.hasEdge(e,r)}function zne(t,e,r){return r.low<=e.lim&&e.lim<=r.lim}var qne=N(()=>{"use strict";qt();$ne();Sc();eR();ek();rf.initLowLimValues=uR;rf.initCutValues=cR;rf.calcCutValue=Gne;rf.leaveEdge=Une;rf.enterEdge=Hne;rf.exchangeEdges=Wne;o(rf,"networkSimplex");o(cR,"initCutValues");o(sIe,"assignCutValue");o(Gne,"calcCutValue");o(uR,"initLowLimValues");o(Vne,"dfsAssignLowLim");o(Une,"leaveEdge");o(Hne,"enterEdge");o(Wne,"exchangeEdges");o(oIe,"updateRanks");o(lIe,"isTreeEdge");o(zne,"isDescendant")});function hR(t){switch(t.graph().ranker){case"network-simplex":Yne(t);break;case"tight-tree":uIe(t);break;case"longest-path":cIe(t);break;default:Yne(t)}}function uIe(t){D2(t),tk(t)}function Yne(t){rf(t)}var cIe,fR=N(()=>{"use strict";eR();qne();ek();o(hR,"rank");cIe=D2;o(uIe,"tightTreeRanker");o(Yne,"networkSimplexRanker")});function Xne(t){var e=Ec(t,"root",{},"_root"),r=hIe(t),n=Is(br(r))-1,i=2*n+1;t.graph().nestingRoot=e,Ae(t.edges(),function(s){t.edge(s).minlen*=i});var a=fIe(t)+1;Ae(t.children(),function(s){jne(t,e,i,a,n,r,s)}),t.graph().nodeRankFactor=i}function jne(t,e,r,n,i,a,s){var l=t.children(s);if(!l.length){s!==e&&t.setEdge(e,s,{weight:0,minlen:r});return}var u=jL(t,"_bt"),h=jL(t,"_bb"),f=t.node(s);t.setParent(u,s),f.borderTop=u,t.setParent(h,s),f.borderBottom=h,Ae(l,function(d){jne(t,e,r,n,i,a,d);var p=t.node(d),m=p.borderTop?p.borderTop:d,g=p.borderBottom?p.borderBottom:d,y=p.borderTop?n:2*n,v=m!==g?1:i-a[s]+1;t.setEdge(u,m,{weight:y,minlen:v,nestingEdge:!0}),t.setEdge(g,h,{weight:y,minlen:v,nestingEdge:!0})}),t.parent(s)||t.setEdge(e,u,{weight:0,minlen:i+a[s]})}function hIe(t){var e={};function r(n,i){var a=t.children(n);a&&a.length&&Ae(a,function(s){r(s,i+1)}),e[n]=i}return o(r,"dfs"),Ae(t.children(),function(n){r(n,1)}),e}function fIe(t){return Xr(t.edges(),function(e,r){return e+t.edge(r).weight},0)}function Kne(t){var e=t.graph();t.removeNode(e.nestingRoot),delete e.nestingRoot,Ae(t.edges(),function(r){var n=t.edge(r);n.nestingEdge&&t.removeEdge(r)})}var Qne=N(()=>{"use strict";qt();Sc();o(Xne,"run");o(jne,"dfs");o(hIe,"treeDepths");o(fIe,"sumWeights");o(Kne,"cleanup")});function Zne(t,e,r){var n={},i;Ae(r,function(a){for(var s=t.parent(a),l,u;s;){if(l=t.parent(s),l?(u=n[l],n[l]=s):(u=i,i=s),u&&u!==s){e.setEdge(u,s);return}s=l}})}var Jne=N(()=>{"use strict";qt();o(Zne,"addSubgraphConstraints")});function eie(t,e,r){var n=pIe(t),i=new sn({compound:!0}).setGraph({root:n}).setDefaultNodeLabel(function(a){return t.node(a)});return Ae(t.nodes(),function(a){var s=t.node(a),l=t.parent(a);(s.rank===e||s.minRank<=e&&e<=s.maxRank)&&(i.setNode(a),i.setParent(a,l||n),Ae(t[r](a),function(u){var h=u.v===a?u.w:u.v,f=i.edge(h,a),d=pr(f)?0:f.weight;i.setEdge(h,a,{weight:t.edge(u).weight+d})}),Object.prototype.hasOwnProperty.call(s,"minRank")&&i.setNode(a,{borderLeft:s.borderLeft[e],borderRight:s.borderRight[e]}))}),i}function pIe(t){for(var e;t.hasNode(e=Ud("_root")););return e}var tie=N(()=>{"use strict";qt();Vo();o(eie,"buildLayerGraph");o(pIe,"createRootNode")});function rie(t,e){for(var r=0,n=1;n0;)f%2&&(d+=l[f+1]),f=f-1>>1,l[f]+=h.weight;u+=h.weight*d})),u}var nie=N(()=>{"use strict";qt();o(rie,"crossCount");o(mIe,"twoLayerCrossCount")});function iie(t){var e={},r=Yr(t.nodes(),function(l){return!t.children(l).length}),n=Is(Je(r,function(l){return t.node(l).rank})),i=Je(Go(n+1),function(){return[]});function a(l){if(!Bt(e,l)){e[l]=!0;var u=t.node(l);i[u.rank].push(l),Ae(t.successors(l),a)}}o(a,"dfs");var s=kc(r,function(l){return t.node(l).rank});return Ae(s,a),i}var aie=N(()=>{"use strict";qt();o(iie,"initOrder")});function sie(t,e){return Je(e,function(r){var n=t.inEdges(r);if(n.length){var i=Xr(n,function(a,s){var l=t.edge(s),u=t.node(s.v);return{sum:a.sum+l.weight*u.order,weight:a.weight+l.weight}},{sum:0,weight:0});return{v:r,barycenter:i.sum/i.weight,weight:i.weight}}else return{v:r}})}var oie=N(()=>{"use strict";qt();o(sie,"barycenter")});function lie(t,e){var r={};Ae(t,function(i,a){var s=r[i.v]={indegree:0,in:[],out:[],vs:[i.v],i:a};pr(i.barycenter)||(s.barycenter=i.barycenter,s.weight=i.weight)}),Ae(e.edges(),function(i){var a=r[i.v],s=r[i.w];!pr(a)&&!pr(s)&&(s.indegree++,a.out.push(r[i.w]))});var n=Yr(r,function(i){return!i.indegree});return gIe(n)}function gIe(t){var e=[];function r(a){return function(s){s.merged||(pr(s.barycenter)||pr(a.barycenter)||s.barycenter>=a.barycenter)&&yIe(a,s)}}o(r,"handleIn");function n(a){return function(s){s.in.push(a),--s.indegree===0&&t.push(s)}}for(o(n,"handleOut");t.length;){var i=t.pop();e.push(i),Ae(i.in.reverse(),r(i)),Ae(i.out,n(i))}return Je(Yr(e,function(a){return!a.merged}),function(a){return Vd(a,["vs","i","barycenter","weight"])})}function yIe(t,e){var r=0,n=0;t.weight&&(r+=t.barycenter*t.weight,n+=t.weight),e.weight&&(r+=e.barycenter*e.weight,n+=e.weight),t.vs=e.vs.concat(t.vs),t.barycenter=r/n,t.weight=n,t.i=Math.min(e.i,t.i),e.merged=!0}var cie=N(()=>{"use strict";qt();o(lie,"resolveConflicts");o(gIe,"doResolveConflicts");o(yIe,"mergeEntries")});function hie(t,e){var r=gne(t,function(f){return Object.prototype.hasOwnProperty.call(f,"barycenter")}),n=r.lhs,i=kc(r.rhs,function(f){return-f.i}),a=[],s=0,l=0,u=0;n.sort(vIe(!!e)),u=uie(a,i,u),Ae(n,function(f){u+=f.vs.length,a.push(f.vs),s+=f.barycenter*f.weight,l+=f.weight,u=uie(a,i,u)});var h={vs:qr(a)};return l&&(h.barycenter=s/l,h.weight=l),h}function uie(t,e,r){for(var n;e.length&&(n=ga(e)).i<=r;)e.pop(),t.push(n.vs),r++;return r}function vIe(t){return function(e,r){return e.barycenterr.barycenter?1:t?r.i-e.i:e.i-r.i}}var fie=N(()=>{"use strict";qt();Sc();o(hie,"sort");o(uie,"consumeUnsortable");o(vIe,"compareWithBias")});function dR(t,e,r,n){var i=t.children(e),a=t.node(e),s=a?a.borderLeft:void 0,l=a?a.borderRight:void 0,u={};s&&(i=Yr(i,function(g){return g!==s&&g!==l}));var h=sie(t,i);Ae(h,function(g){if(t.children(g.v).length){var y=dR(t,g.v,r,n);u[g.v]=y,Object.prototype.hasOwnProperty.call(y,"barycenter")&&bIe(g,y)}});var f=lie(h,r);xIe(f,u);var d=hie(f,n);if(s&&(d.vs=qr([s,d.vs,l]),t.predecessors(s).length)){var p=t.node(t.predecessors(s)[0]),m=t.node(t.predecessors(l)[0]);Object.prototype.hasOwnProperty.call(d,"barycenter")||(d.barycenter=0,d.weight=0),d.barycenter=(d.barycenter*d.weight+p.order+m.order)/(d.weight+2),d.weight+=2}return d}function xIe(t,e){Ae(t,function(r){r.vs=qr(r.vs.map(function(n){return e[n]?e[n].vs:n}))})}function bIe(t,e){pr(t.barycenter)?(t.barycenter=e.barycenter,t.weight=e.weight):(t.barycenter=(t.barycenter*t.weight+e.barycenter*e.weight)/(t.weight+e.weight),t.weight+=e.weight)}var die=N(()=>{"use strict";qt();oie();cie();fie();o(dR,"sortSubgraph");o(xIe,"expandSubgraphs");o(bIe,"mergeBarycenters")});function gie(t){var e=KL(t),r=pie(t,Go(1,e+1),"inEdges"),n=pie(t,Go(e-1,-1,-1),"outEdges"),i=iie(t);mie(t,i);for(var a=Number.POSITIVE_INFINITY,s,l=0,u=0;u<4;++l,++u){wIe(l%2?r:n,l%4>=2),i=ef(t);var h=rie(t,i);h{"use strict";qt();Vo();Sc();Jne();tie();nie();aie();die();o(gie,"order");o(pie,"buildLayerGraphs");o(wIe,"sweepLayerGraphs");o(mie,"assignOrder")});function vie(t){var e=kIe(t);Ae(t.graph().dummyChains,function(r){for(var n=t.node(r),i=n.edgeObj,a=TIe(t,e,i.v,i.w),s=a.path,l=a.lca,u=0,h=s[u],f=!0;r!==i.w;){if(n=t.node(r),f){for(;(h=s[u])!==l&&t.node(h).maxRanks||l>e[u].lim));for(h=u,u=n;(u=t.parent(u))!==h;)a.push(u);return{path:i.concat(a.reverse()),lca:h}}function kIe(t){var e={},r=0;function n(i){var a=r;Ae(t.children(i),n),e[i]={low:a,lim:r++}}return o(n,"dfs"),Ae(t.children(),n),e}var xie=N(()=>{"use strict";qt();o(vie,"parentDummyChains");o(TIe,"findPath");o(kIe,"postorder")});function EIe(t,e){var r={};function n(i,a){var s=0,l=0,u=i.length,h=ga(a);return Ae(a,function(f,d){var p=CIe(t,f),m=p?t.node(p).order:u;(p||f===h)&&(Ae(a.slice(l,d+1),function(g){Ae(t.predecessors(g),function(y){var v=t.node(y),x=v.order;(xh)&&bie(r,p,f)})})}o(n,"scan");function i(a,s){var l=-1,u,h=0;return Ae(s,function(f,d){if(t.node(f).dummy==="border"){var p=t.predecessors(f);p.length&&(u=t.node(p[0]).order,n(s,h,d,l,u),h=d,l=u)}n(s,h,s.length,u,a.length)}),s}return o(i,"visitLayer"),Xr(e,i),r}function CIe(t,e){if(t.node(e).dummy)return ns(t.predecessors(e),function(r){return t.node(r).dummy})}function bie(t,e,r){if(e>r){var n=e;e=r,r=n}var i=t[e];i||(t[e]=i={}),i[r]=!0}function AIe(t,e,r){if(e>r){var n=e;e=r,r=n}return!!t[e]&&Object.prototype.hasOwnProperty.call(t[e],r)}function _Ie(t,e,r,n){var i={},a={},s={};return Ae(e,function(l){Ae(l,function(u,h){i[u]=u,a[u]=u,s[u]=h})}),Ae(e,function(l){var u=-1;Ae(l,function(h){var f=n(h);if(f.length){f=kc(f,function(y){return s[y]});for(var d=(f.length-1)/2,p=Math.floor(d),m=Math.ceil(d);p<=m;++p){var g=f[p];a[h]===h&&u{"use strict";qt();Vo();Sc();o(EIe,"findType1Conflicts");o(SIe,"findType2Conflicts");o(CIe,"findOtherInnerSegmentNode");o(bie,"addConflict");o(AIe,"hasConflict");o(_Ie,"verticalAlignment");o(DIe,"horizontalCompaction");o(LIe,"buildBlockGraph");o(RIe,"findSmallestWidthAlignment");o(NIe,"alignCoordinates");o(MIe,"balance");o(wie,"positionX");o(IIe,"sep");o(OIe,"width")});function kie(t){t=JT(t),PIe(t),ML(wie(t),function(e,r){t.node(r).x=e})}function PIe(t){var e=ef(t),r=t.graph().ranksep,n=0;Ae(e,function(i){var a=Is(Je(i,function(s){return t.node(s).height}));Ae(i,function(s){t.node(s).y=n+a/2}),n+=a+r})}var Eie=N(()=>{"use strict";qt();Sc();Tie();o(kie,"position");o(PIe,"positionY")});function R2(t,e){var r=e&&e.debugTiming?yne:vne;r("layout",()=>{var n=r(" buildLayoutGraph",()=>YIe(t));r(" runLayout",()=>BIe(n,r)),r(" updateInputGraph",()=>FIe(t,n))})}function BIe(t,e){e(" makeSpaceForEdgeLabels",()=>XIe(t)),e(" removeSelfEdges",()=>nOe(t)),e(" acyclic",()=>une(t)),e(" nestingGraph.run",()=>Xne(t)),e(" rank",()=>hR(JT(t))),e(" injectEdgeLabelProxies",()=>jIe(t)),e(" removeEmptyRanks",()=>mne(t)),e(" nestingGraph.cleanup",()=>Kne(t)),e(" normalizeRanks",()=>pne(t)),e(" assignRankMinMax",()=>KIe(t)),e(" removeEdgeLabelProxies",()=>QIe(t)),e(" normalize.run",()=>Ane(t)),e(" parentDummyChains",()=>vie(t)),e(" addBorderSegments",()=>bne(t)),e(" order",()=>gie(t)),e(" insertSelfEdges",()=>iOe(t)),e(" adjustCoordinateSystem",()=>kne(t)),e(" position",()=>kie(t)),e(" positionSelfEdges",()=>aOe(t)),e(" removeBorderNodes",()=>rOe(t)),e(" normalize.undo",()=>_ne(t)),e(" fixupEdgeLabelCoords",()=>eOe(t)),e(" undoCoordinateSystem",()=>Ene(t)),e(" translateGraph",()=>ZIe(t)),e(" assignNodeIntersects",()=>JIe(t)),e(" reversePoints",()=>tOe(t)),e(" acyclic.undo",()=>hne(t))}function FIe(t,e){Ae(t.nodes(),function(r){var n=t.node(r),i=e.node(r);n&&(n.x=i.x,n.y=i.y,e.children(r).length&&(n.width=i.width,n.height=i.height))}),Ae(t.edges(),function(r){var n=t.edge(r),i=e.edge(r);n.points=i.points,Object.prototype.hasOwnProperty.call(i,"x")&&(n.x=i.x,n.y=i.y)}),t.graph().width=e.graph().width,t.graph().height=e.graph().height}function YIe(t){var e=new sn({multigraph:!0,compound:!0}),r=mR(t.graph());return e.setGraph(Fh({},zIe,pR(r,$Ie),Vd(r,GIe))),Ae(t.nodes(),function(n){var i=mR(t.node(n));e.setNode(n,Qh(pR(i,VIe),UIe)),e.setParent(n,t.parent(n))}),Ae(t.edges(),function(n){var i=mR(t.edge(n));e.setEdge(n,Fh({},WIe,pR(i,HIe),Vd(i,qIe)))}),e}function XIe(t){var e=t.graph();e.ranksep/=2,Ae(t.edges(),function(r){var n=t.edge(r);n.minlen*=2,n.labelpos.toLowerCase()!=="c"&&(e.rankdir==="TB"||e.rankdir==="BT"?n.width+=n.labeloffset:n.height+=n.labeloffset)})}function jIe(t){Ae(t.edges(),function(e){var r=t.edge(e);if(r.width&&r.height){var n=t.node(e.v),i=t.node(e.w),a={rank:(i.rank-n.rank)/2+n.rank,e};Ec(t,"edge-proxy",a,"_ep")}})}function KIe(t){var e=0;Ae(t.nodes(),function(r){var n=t.node(r);n.borderTop&&(n.minRank=t.node(n.borderTop).rank,n.maxRank=t.node(n.borderBottom).rank,e=Is(e,n.maxRank))}),t.graph().maxRank=e}function QIe(t){Ae(t.nodes(),function(e){var r=t.node(e);r.dummy==="edge-proxy"&&(t.edge(r.e).labelRank=r.rank,t.removeNode(e))})}function ZIe(t){var e=Number.POSITIVE_INFINITY,r=0,n=Number.POSITIVE_INFINITY,i=0,a=t.graph(),s=a.marginx||0,l=a.marginy||0;function u(h){var f=h.x,d=h.y,p=h.width,m=h.height;e=Math.min(e,f-p/2),r=Math.max(r,f+p/2),n=Math.min(n,d-m/2),i=Math.max(i,d+m/2)}o(u,"getExtremes"),Ae(t.nodes(),function(h){u(t.node(h))}),Ae(t.edges(),function(h){var f=t.edge(h);Object.prototype.hasOwnProperty.call(f,"x")&&u(f)}),e-=s,n-=l,Ae(t.nodes(),function(h){var f=t.node(h);f.x-=e,f.y-=n}),Ae(t.edges(),function(h){var f=t.edge(h);Ae(f.points,function(d){d.x-=e,d.y-=n}),Object.prototype.hasOwnProperty.call(f,"x")&&(f.x-=e),Object.prototype.hasOwnProperty.call(f,"y")&&(f.y-=n)}),a.width=r-e+s,a.height=i-n+l}function JIe(t){Ae(t.edges(),function(e){var r=t.edge(e),n=t.node(e.v),i=t.node(e.w),a,s;r.points?(a=r.points[0],s=r.points[r.points.length-1]):(r.points=[],a=i,s=n),r.points.unshift(XL(n,a)),r.points.push(XL(i,s))})}function eOe(t){Ae(t.edges(),function(e){var r=t.edge(e);if(Object.prototype.hasOwnProperty.call(r,"x"))switch((r.labelpos==="l"||r.labelpos==="r")&&(r.width-=r.labeloffset),r.labelpos){case"l":r.x-=r.width/2+r.labeloffset;break;case"r":r.x+=r.width/2+r.labeloffset;break}})}function tOe(t){Ae(t.edges(),function(e){var r=t.edge(e);r.reversed&&r.points.reverse()})}function rOe(t){Ae(t.nodes(),function(e){if(t.children(e).length){var r=t.node(e),n=t.node(r.borderTop),i=t.node(r.borderBottom),a=t.node(ga(r.borderLeft)),s=t.node(ga(r.borderRight));r.width=Math.abs(s.x-a.x),r.height=Math.abs(i.y-n.y),r.x=a.x+r.width/2,r.y=n.y+r.height/2}}),Ae(t.nodes(),function(e){t.node(e).dummy==="border"&&t.removeNode(e)})}function nOe(t){Ae(t.edges(),function(e){if(e.v===e.w){var r=t.node(e.v);r.selfEdges||(r.selfEdges=[]),r.selfEdges.push({e,label:t.edge(e)}),t.removeEdge(e)}})}function iOe(t){var e=ef(t);Ae(e,function(r){var n=0;Ae(r,function(i,a){var s=t.node(i);s.order=a+n,Ae(s.selfEdges,function(l){Ec(t,"selfedge",{width:l.label.width,height:l.label.height,rank:s.rank,order:a+ ++n,e:l.e,label:l.label},"_se")}),delete s.selfEdges})})}function aOe(t){Ae(t.nodes(),function(e){var r=t.node(e);if(r.dummy==="selfedge"){var n=t.node(r.e.v),i=n.x+n.width/2,a=n.y,s=r.x-i,l=n.height/2;t.setEdge(r.e,r.label),t.removeNode(e),r.label.points=[{x:i+2*s/3,y:a-l},{x:i+5*s/6,y:a-l},{x:i+s,y:a},{x:i+5*s/6,y:a+l},{x:i+2*s/3,y:a+l}],r.label.x=r.x,r.label.y=r.y}})}function pR(t,e){return zd(Vd(t,e),Number)}function mR(t){var e={};return Ae(t,function(r,n){e[n.toLowerCase()]=r}),e}var $Ie,zIe,GIe,VIe,UIe,HIe,WIe,qIe,Sie=N(()=>{"use strict";qt();Vo();wne();Cne();YL();JL();fR();Qne();yie();xie();Eie();Sc();o(R2,"layout");o(BIe,"runLayout");o(FIe,"updateInputGraph");$Ie=["nodesep","edgesep","ranksep","marginx","marginy"],zIe={ranksep:50,edgesep:20,nodesep:50,rankdir:"tb"},GIe=["acyclicer","ranker","rankdir","align"],VIe=["width","height"],UIe={width:0,height:0},HIe=["minlen","weight","width","height","labeloffset"],WIe={minlen:1,weight:1,width:0,height:0,labeloffset:10,labelpos:"r"},qIe=["labelpos"];o(YIe,"buildLayoutGraph");o(XIe,"makeSpaceForEdgeLabels");o(jIe,"injectEdgeLabelProxies");o(KIe,"assignRankMinMax");o(QIe,"removeEdgeLabelProxies");o(ZIe,"translateGraph");o(JIe,"assignNodeIntersects");o(eOe,"fixupEdgeLabelCoords");o(tOe,"reversePointsForReversedEdges");o(rOe,"removeBorderNodes");o(nOe,"removeSelfEdges");o(iOe,"insertSelfEdges");o(aOe,"positionSelfEdges");o(pR,"selectNumberAttrs");o(mR,"canonicalize")});var gR=N(()=>{"use strict";YL();Sie();JL();fR()});function Uo(t){var e={options:{directed:t.isDirected(),multigraph:t.isMultigraph(),compound:t.isCompound()},nodes:sOe(t),edges:oOe(t)};return pr(t.graph())||(e.value=an(t.graph())),e}function sOe(t){return Je(t.nodes(),function(e){var r=t.node(e),n=t.parent(e),i={v:e};return pr(r)||(i.value=r),pr(n)||(i.parent=n),i})}function oOe(t){return Je(t.edges(),function(e){var r=t.edge(e),n={v:e.v,w:e.w};return pr(e.name)||(n.name=e.name),pr(r)||(n.value=r),n})}var yR=N(()=>{"use strict";qt();KT();o(Uo,"write");o(sOe,"writeNodes");o(oOe,"writeEdges")});var wr,qd,_ie,Die,nk,lOe,Lie,Rie,cOe,Fm,Aie,Nie,Mie,Iie,Oie,Pie=N(()=>{"use strict";vt();Vo();yR();wr=new Map,qd=new Map,_ie=new Map,Die=o(()=>{qd.clear(),_ie.clear(),wr.clear()},"clear"),nk=o((t,e)=>{let r=qd.get(e)||[];return Y.trace("In isDescendant",e," ",t," = ",r.includes(t)),r.includes(t)},"isDescendant"),lOe=o((t,e)=>{let r=qd.get(e)||[];return Y.info("Descendants of ",e," is ",r),Y.info("Edge is ",t),t.v===e||t.w===e?!1:r?r.includes(t.v)||nk(t.v,e)||nk(t.w,e)||r.includes(t.w):(Y.debug("Tilt, ",e,",not in descendants"),!1)},"edgeInCluster"),Lie=o((t,e,r,n)=>{Y.warn("Copying children of ",t,"root",n,"data",e.node(t),n);let i=e.children(t)||[];t!==n&&i.push(t),Y.warn("Copying (nodes) clusterId",t,"nodes",i),i.forEach(a=>{if(e.children(a).length>0)Lie(a,e,r,n);else{let s=e.node(a);Y.info("cp ",a," to ",n," with parent ",t),r.setNode(a,s),n!==e.parent(a)&&(Y.warn("Setting parent",a,e.parent(a)),r.setParent(a,e.parent(a))),t!==n&&a!==t?(Y.debug("Setting parent",a,t),r.setParent(a,t)):(Y.info("In copy ",t,"root",n,"data",e.node(t),n),Y.debug("Not Setting parent for node=",a,"cluster!==rootId",t!==n,"node!==clusterId",a!==t));let l=e.edges(a);Y.debug("Copying Edges",l),l.forEach(u=>{Y.info("Edge",u);let h=e.edge(u.v,u.w,u.name);Y.info("Edge data",h,n);try{lOe(u,n)?(Y.info("Copying as ",u.v,u.w,h,u.name),r.setEdge(u.v,u.w,h,u.name),Y.info("newGraph edges ",r.edges(),r.edge(r.edges()[0]))):Y.info("Skipping copy of edge ",u.v,"-->",u.w," rootId: ",n," clusterId:",t)}catch(f){Y.error(f)}})}Y.debug("Removing node",a),e.removeNode(a)})},"copy"),Rie=o((t,e)=>{let r=e.children(t),n=[...r];for(let i of r)_ie.set(i,t),n=[...n,...Rie(i,e)];return n},"extractDescendants"),cOe=o((t,e,r)=>{let n=t.edges().filter(u=>u.v===e||u.w===e),i=t.edges().filter(u=>u.v===r||u.w===r),a=n.map(u=>({v:u.v===e?r:u.v,w:u.w===e?e:u.w})),s=i.map(u=>({v:u.v,w:u.w}));return a.filter(u=>s.some(h=>u.v===h.v&&u.w===h.w))},"findCommonEdges"),Fm=o((t,e,r)=>{let n=e.children(t);if(Y.trace("Searching children of id ",t,n),n.length<1)return t;let i;for(let a of n){let s=Fm(a,e,r),l=cOe(e,r,s);if(s)if(l.length>0)i=s;else return s}return i},"findNonClusterChild"),Aie=o(t=>!wr.has(t)||!wr.get(t).externalConnections?t:wr.has(t)?wr.get(t).id:t,"getAnchorId"),Nie=o((t,e)=>{if(!t||e>10){Y.debug("Opting out, no graph ");return}else Y.debug("Opting in, graph ");t.nodes().forEach(function(r){t.children(r).length>0&&(Y.warn("Cluster identified",r," Replacement id in edges: ",Fm(r,t,r)),qd.set(r,Rie(r,t)),wr.set(r,{id:Fm(r,t,r),clusterData:t.node(r)}))}),t.nodes().forEach(function(r){let n=t.children(r),i=t.edges();n.length>0?(Y.debug("Cluster identified",r,qd),i.forEach(a=>{let s=nk(a.v,r),l=nk(a.w,r);s^l&&(Y.warn("Edge: ",a," leaves cluster ",r),Y.warn("Descendants of XXX ",r,": ",qd.get(r)),wr.get(r).externalConnections=!0)})):Y.debug("Not a cluster ",r,qd)});for(let r of wr.keys()){let n=wr.get(r).id,i=t.parent(n);i!==r&&wr.has(i)&&!wr.get(i).externalConnections&&(wr.get(r).id=i)}t.edges().forEach(function(r){let n=t.edge(r);Y.warn("Edge "+r.v+" -> "+r.w+": "+JSON.stringify(r)),Y.warn("Edge "+r.v+" -> "+r.w+": "+JSON.stringify(t.edge(r)));let i=r.v,a=r.w;if(Y.warn("Fix XXX",wr,"ids:",r.v,r.w,"Translating: ",wr.get(r.v)," --- ",wr.get(r.w)),wr.get(r.v)||wr.get(r.w)){if(Y.warn("Fixing and trying - removing XXX",r.v,r.w,r.name),i=Aie(r.v),a=Aie(r.w),t.removeEdge(r.v,r.w,r.name),i!==r.v){let s=t.parent(i);wr.get(s).externalConnections=!0,n.fromCluster=r.v}if(a!==r.w){let s=t.parent(a);wr.get(s).externalConnections=!0,n.toCluster=r.w}Y.warn("Fix Replacing with XXX",i,a,r.name),t.setEdge(i,a,n,r.name)}}),Y.warn("Adjusted Graph",Uo(t)),Mie(t,0),Y.trace(wr)},"adjustClustersAndEdges"),Mie=o((t,e)=>{if(Y.warn("extractor - ",e,Uo(t),t.children("D")),e>10){Y.error("Bailing out");return}let r=t.nodes(),n=!1;for(let i of r){let a=t.children(i);n=n||a.length>0}if(!n){Y.debug("Done, no node has children",t.nodes());return}Y.debug("Nodes = ",r,e);for(let i of r)if(Y.debug("Extracting node",i,wr,wr.has(i)&&!wr.get(i).externalConnections,!t.parent(i),t.node(i),t.children("D")," Depth ",e),!wr.has(i))Y.debug("Not a cluster",i,e);else if(!wr.get(i).externalConnections&&t.children(i)&&t.children(i).length>0){Y.warn("Cluster without external connections, without a parent and with children",i,e);let s=t.graph().rankdir==="TB"?"LR":"TB";wr.get(i)?.clusterData?.dir&&(s=wr.get(i).clusterData.dir,Y.warn("Fixing dir",wr.get(i).clusterData.dir,s));let l=new sn({multigraph:!0,compound:!0}).setGraph({rankdir:s,nodesep:50,ranksep:50,marginx:8,marginy:8}).setDefaultEdgeLabel(function(){return{}});Y.warn("Old graph before copy",Uo(t)),Lie(i,t,l,i),t.setNode(i,{clusterNode:!0,id:i,clusterData:wr.get(i).clusterData,label:wr.get(i).label,graph:l}),Y.warn("New graph after copy node: (",i,")",Uo(l)),Y.debug("Old graph after copy",Uo(t))}else Y.warn("Cluster ** ",i," **not meeting the criteria !externalConnections:",!wr.get(i).externalConnections," no parent: ",!t.parent(i)," children ",t.children(i)&&t.children(i).length>0,t.children("D"),e),Y.debug(wr);r=t.nodes(),Y.warn("New list of nodes",r);for(let i of r){let a=t.node(i);Y.warn(" Now next level",i,a),a?.clusterNode&&Mie(a.graph,e+1)}},"extractor"),Iie=o((t,e)=>{if(e.length===0)return[];let r=Object.assign([],e);return e.forEach(n=>{let i=t.children(n),a=Iie(t,i);r=[...r,...a]}),r},"sorter"),Oie=o(t=>Iie(t,t.children()),"sortNodesByHierarchy")});var Fie={};hr(Fie,{render:()=>uOe});var Bie,uOe,$ie=N(()=>{"use strict";gR();yR();Vo();tL();Ft();Pie();eT();Hw();eL();vt();w2();zt();Bie=o(async(t,e,r,n,i,a)=>{Y.warn("Graph in recursive render:XAX",Uo(e),i);let s=e.graph().rankdir;Y.trace("Dir in recursive render - dir:",s);let l=t.insert("g").attr("class","root");e.nodes()?Y.info("Recursive render XXX",e.nodes()):Y.info("No nodes found for",e),e.edges().length>0&&Y.info("Recursive edges",e.edge(e.edges()[0]));let u=l.insert("g").attr("class","clusters"),h=l.insert("g").attr("class","edgePaths"),f=l.insert("g").attr("class","edgeLabels"),d=l.insert("g").attr("class","nodes");await Promise.all(e.nodes().map(async function(y){let v=e.node(y);if(i!==void 0){let x=JSON.parse(JSON.stringify(i.clusterData));Y.trace(`Setting data for parent cluster XXX + Node.id = `,y,` + data=`,x.height,` +Parent cluster`,i.height),e.setNode(i.id,x),e.parent(y)||(Y.trace("Setting parent",y,i.id),e.setParent(y,i.id,x))}if(Y.info("(Insert) Node XXX"+y+": "+JSON.stringify(e.node(y))),v?.clusterNode){Y.info("Cluster identified XBX",y,v.width,e.node(y));let{ranksep:x,nodesep:b}=e.graph();v.graph.setGraph({...v.graph.graph(),ranksep:x+25,nodesep:b});let w=await Bie(d,v.graph,r,n,e.node(y),a),C=w.elem;je(v,C),v.diff=w.diff||0,Y.info("New compound node after recursive render XAX",y,"width",v.width,"height",v.height),rJ(C,v)}else e.children(y).length>0?(Y.trace("Cluster - the non recursive path XBX",y,v.id,v,v.width,"Graph:",e),Y.trace(Fm(v.id,e)),wr.set(v.id,{id:Fm(v.id,e),node:v})):(Y.trace("Node - the non recursive path XAX",y,d,e.node(y),s),await vm(d,e.node(y),{config:a,dir:s}))})),await o(async()=>{let y=e.edges().map(async function(v){let x=e.edge(v.v,v.w,v.name);Y.info("Edge "+v.v+" -> "+v.w+": "+JSON.stringify(v)),Y.info("Edge "+v.v+" -> "+v.w+": ",v," ",JSON.stringify(e.edge(v))),Y.info("Fix",wr,"ids:",v.v,v.w,"Translating: ",wr.get(v.v),wr.get(v.w)),await jw(f,x)});await Promise.all(y)},"processEdges")(),Y.info("Graph before layout:",JSON.stringify(Uo(e))),Y.info("############################################# XXX"),Y.info("### Layout ### XXX"),Y.info("############################################# XXX"),R2(e),Y.info("Graph after layout:",JSON.stringify(Uo(e)));let m=0,{subGraphTitleTotalMargin:g}=Ru(a);return await Promise.all(Oie(e).map(async function(y){let v=e.node(y);if(Y.info("Position XBX => "+y+": ("+v.x,","+v.y,") width: ",v.width," height: ",v.height),v?.clusterNode)v.y+=g,Y.info("A tainted cluster node XBX1",y,v.id,v.width,v.height,v.x,v.y,e.parent(y)),wr.get(v.id).node=v,k2(v);else if(e.children(y).length>0){Y.info("A pure cluster node XBX1",y,v.id,v.x,v.y,v.width,v.height,e.parent(y)),v.height+=g,e.node(v.parentId);let x=v?.padding/2||0,b=v?.labelBBox?.height||0,w=b-x||0;Y.debug("OffsetY",w,"labelHeight",b,"halfPadding",x),await ym(u,v),wr.get(v.id).node=v}else{let x=e.node(v.parentId);v.y+=g/2,Y.info("A regular node XBX1 - using the padding",v.id,"parent",v.parentId,v.width,v.height,v.x,v.y,"offsetY",v.offsetY,"parent",x,x?.offsetY,v),k2(v)}})),e.edges().forEach(function(y){let v=e.edge(y);Y.info("Edge "+y.v+" -> "+y.w+": "+JSON.stringify(v),v),v.points.forEach(C=>C.y+=g/2);let x=e.node(y.v);var b=e.node(y.w);let w=Qw(h,v,wr,r,x,b,n);Kw(v,w)}),e.nodes().forEach(function(y){let v=e.node(y);Y.info(y,v.type,v.diff),v.isGroup&&(m=v.diff)}),Y.warn("Returning from recursive render XAX",l,m),{elem:l,diff:m}},"recursiveRender"),uOe=o(async(t,e)=>{let r=new sn({multigraph:!0,compound:!0}).setGraph({rankdir:t.direction,nodesep:t.config?.nodeSpacing||t.config?.flowchart?.nodeSpacing||t.nodeSpacing,ranksep:t.config?.rankSpacing||t.config?.flowchart?.rankSpacing||t.rankSpacing,marginx:8,marginy:8}).setDefaultEdgeLabel(function(){return{}}),n=e.select("g");Zw(n,t.markers,t.type,t.diagramId),nJ(),tJ(),jZ(),Die(),t.nodes.forEach(a=>{r.setNode(a.id,{...a}),a.parentId&&r.setParent(a.id,a.parentId)}),Y.debug("Edges:",t.edges),t.edges.forEach(a=>{if(a.start===a.end){let s=a.start,l=s+"---"+s+"---1",u=s+"---"+s+"---2",h=r.node(s);r.setNode(l,{domId:l,id:l,parentId:h.parentId,labelStyle:"",label:"",padding:0,shape:"labelRect",style:"",width:10,height:10}),r.setParent(l,h.parentId),r.setNode(u,{domId:u,id:u,parentId:h.parentId,labelStyle:"",padding:0,shape:"labelRect",label:"",style:"",width:10,height:10}),r.setParent(u,h.parentId);let f=structuredClone(a),d=structuredClone(a),p=structuredClone(a);f.label="",f.arrowTypeEnd="none",f.id=s+"-cyclic-special-1",d.arrowTypeStart="none",d.arrowTypeEnd="none",d.id=s+"-cyclic-special-mid",p.label="",h.isGroup&&(f.fromCluster=s,p.toCluster=s),p.id=s+"-cyclic-special-2",p.arrowTypeStart="none",r.setEdge(s,l,f,s+"-cyclic-special-0"),r.setEdge(l,u,d,s+"-cyclic-special-1"),r.setEdge(u,s,p,s+"-cyc{"use strict";aJ();vt();N2={},vR=o(t=>{for(let e of t)N2[e.name]=e},"registerLayoutLoaders"),hOe=o(()=>{vR([{name:"dagre",loader:o(async()=>await Promise.resolve().then(()=>($ie(),Fie)),"loader")}])},"registerDefaultLayoutLoaders");hOe();Cc=o(async(t,e)=>{if(!(t.layoutAlgorithm in N2))throw new Error(`Unknown layout algorithm: ${t.layoutAlgorithm}`);let r=N2[t.layoutAlgorithm];return(await r.loader()).render(t,e,iJ,{algorithm:r.algorithm})},"render"),nf=o((t="",{fallback:e="dagre"}={})=>{if(t in N2)return t;if(e in N2)return Y.warn(`Layout algorithm ${t} is not registered. Using ${e} as fallback.`),e;throw new Error(`Both layout algorithms ${t} and ${e} are not registered.`)},"getRegisteredLayoutAlgorithm")});var Ac,fOe,dOe,$m=N(()=>{"use strict";Ei();vt();Ac=o((t,e,r,n)=>{t.attr("class",r);let{width:i,height:a,x:s,y:l}=fOe(t,e);vn(t,a,i,n);let u=dOe(s,l,i,a,e);t.attr("viewBox",u),Y.debug(`viewBox configured: ${u} with padding: ${e}`)},"setupViewPortForSVG"),fOe=o((t,e)=>{let r=t.node()?.getBBox()||{width:0,height:0,x:0,y:0};return{width:r.width+e*2,height:r.height+e*2,x:r.x,y:r.y}},"calculateDimensionsWithPadding"),dOe=o((t,e,r,n,i)=>`${t-i} ${e-i} ${r} ${n}`,"createViewBox")});var pOe,mOe,zie,Gie=N(()=>{"use strict";dr();zt();vt();gm();Yd();$m();ir();pOe=o(function(t,e){return e.db.getClasses()},"getClasses"),mOe=o(async function(t,e,r,n){Y.info("REF0:"),Y.info("Drawing state diagram (v2)",e);let{securityLevel:i,flowchart:a,layout:s}=me(),l;i==="sandbox"&&(l=Ge("#i"+e));let u=i==="sandbox"?l.nodes()[0].contentDocument:document;Y.debug("Before getData: ");let h=n.db.getData();Y.debug("Data: ",h);let f=yc(e,i),d=n.db.getDirection();h.type=n.type,h.layoutAlgorithm=nf(s),h.layoutAlgorithm==="dagre"&&s==="elk"&&Y.warn("flowchart-elk was moved to an external package in Mermaid v11. Please refer [release notes](https://github.com/mermaid-js/mermaid/releases/tag/v11.0.0) for more details. This diagram will be rendered using `dagre` layout as a fallback."),h.direction=d,h.nodeSpacing=a?.nodeSpacing||50,h.rankSpacing=a?.rankSpacing||50,h.markers=["point","circle","cross"],h.diagramId=e,Y.debug("REF1:",h),await Cc(h,f);let p=h.config.flowchart?.diagramPadding??8;Gt.insertTitle(f,"flowchartTitleText",a?.titleTopMargin||0,n.db.getDiagramTitle()),Ac(f,p,"flowchart",a?.useMaxWidth||!1);for(let m of h.nodes){let g=Ge(`#${e} [id="${m.id}"]`);if(!g||!m.link)continue;let y=u.createElementNS("http://www.w3.org/2000/svg","a");y.setAttributeNS("http://www.w3.org/2000/svg","class",m.cssClasses),y.setAttributeNS("http://www.w3.org/2000/svg","rel","noopener"),i==="sandbox"?y.setAttributeNS("http://www.w3.org/2000/svg","target","_top"):m.linkTarget&&y.setAttributeNS("http://www.w3.org/2000/svg","target",m.linkTarget);let v=g.insert(function(){return y},":first-child"),x=g.select(".label-container");x&&v.append(function(){return x.node()});let b=g.select(".label");b&&v.append(function(){return b.node()})}},"draw"),zie={getClasses:pOe,draw:mOe}});var xR,bR,Vie=N(()=>{"use strict";xR=function(){var t=o(function(Hr,et,mt,Kt){for(mt=mt||{},Kt=Hr.length;Kt--;mt[Hr[Kt]]=et);return mt},"o"),e=[1,4],r=[1,3],n=[1,5],i=[1,8,9,10,11,27,34,36,38,44,60,84,85,86,87,88,89,102,105,106,109,111,114,115,116,121,122,123,124],a=[2,2],s=[1,13],l=[1,14],u=[1,15],h=[1,16],f=[1,23],d=[1,25],p=[1,26],m=[1,27],g=[1,49],y=[1,48],v=[1,29],x=[1,30],b=[1,31],w=[1,32],C=[1,33],T=[1,44],E=[1,46],A=[1,42],S=[1,47],_=[1,43],I=[1,50],D=[1,45],k=[1,51],L=[1,52],R=[1,34],O=[1,35],M=[1,36],B=[1,37],F=[1,57],P=[1,8,9,10,11,27,32,34,36,38,44,60,84,85,86,87,88,89,102,105,106,109,111,114,115,116,121,122,123,124],z=[1,61],$=[1,60],H=[1,62],Q=[8,9,11,75,77,78],j=[1,78],ie=[1,91],ne=[1,96],le=[1,95],he=[1,92],K=[1,88],X=[1,94],te=[1,90],J=[1,97],se=[1,93],ue=[1,98],Z=[1,89],Se=[8,9,10,11,40,75,77,78],ce=[8,9,10,11,40,46,75,77,78],ae=[8,9,10,11,29,40,44,46,48,50,52,54,56,58,60,63,65,67,68,70,75,77,78,89,102,105,106,109,111,114,115,116],Oe=[8,9,11,44,60,75,77,78,89,102,105,106,109,111,114,115,116],ge=[44,60,89,102,105,106,109,111,114,115,116],ze=[1,121],He=[1,122],$e=[1,124],Re=[1,123],Ie=[44,60,62,74,89,102,105,106,109,111,114,115,116],be=[1,133],W=[1,147],de=[1,148],re=[1,149],oe=[1,150],V=[1,135],xe=[1,137],q=[1,141],pe=[1,142],ve=[1,143],Pe=[1,144],_e=[1,145],we=[1,146],Ve=[1,151],De=[1,152],qe=[1,131],at=[1,132],Rt=[1,139],st=[1,134],Ue=[1,138],ct=[1,136],We=[8,9,10,11,27,32,34,36,38,44,60,84,85,86,87,88,89,102,105,106,109,111,114,115,116,121,122,123,124],ot=[1,154],Yt=[1,156],bt=[8,9,11],Mt=[8,9,10,11,14,44,60,89,105,106,109,111,114,115,116],xt=[1,176],ut=[1,172],Et=[1,173],ft=[1,177],yt=[1,174],nt=[1,175],dn=[77,116,119],Tt=[8,9,10,11,12,14,27,29,32,44,60,75,84,85,86,87,88,89,90,105,109,111,114,115,116],On=[10,106],tn=[31,49,51,53,55,57,62,64,66,67,69,71,116,117,118],_r=[1,247],Dr=[1,245],Pn=[1,249],At=[1,243],Ce=[1,244],tt=[1,246],St=[1,248],mr=[1,250],rn=[1,268],gn=[8,9,11,106],Zr=[8,9,10,11,60,84,105,106,109,110,111,112],Ni={trace:o(function(){},"trace"),yy:{},symbols_:{error:2,start:3,graphConfig:4,document:5,line:6,statement:7,SEMI:8,NEWLINE:9,SPACE:10,EOF:11,GRAPH:12,NODIR:13,DIR:14,FirstStmtSeparator:15,ending:16,endToken:17,spaceList:18,spaceListNewline:19,vertexStatement:20,separator:21,styleStatement:22,linkStyleStatement:23,classDefStatement:24,classStatement:25,clickStatement:26,subgraph:27,textNoTags:28,SQS:29,text:30,SQE:31,end:32,direction:33,acc_title:34,acc_title_value:35,acc_descr:36,acc_descr_value:37,acc_descr_multiline_value:38,shapeData:39,SHAPE_DATA:40,link:41,node:42,styledVertex:43,AMP:44,vertex:45,STYLE_SEPARATOR:46,idString:47,DOUBLECIRCLESTART:48,DOUBLECIRCLEEND:49,PS:50,PE:51,"(-":52,"-)":53,STADIUMSTART:54,STADIUMEND:55,SUBROUTINESTART:56,SUBROUTINEEND:57,VERTEX_WITH_PROPS_START:58,"NODE_STRING[field]":59,COLON:60,"NODE_STRING[value]":61,PIPE:62,CYLINDERSTART:63,CYLINDEREND:64,DIAMOND_START:65,DIAMOND_STOP:66,TAGEND:67,TRAPSTART:68,TRAPEND:69,INVTRAPSTART:70,INVTRAPEND:71,linkStatement:72,arrowText:73,TESTSTR:74,START_LINK:75,edgeText:76,LINK:77,LINK_ID:78,edgeTextToken:79,STR:80,MD_STR:81,textToken:82,keywords:83,STYLE:84,LINKSTYLE:85,CLASSDEF:86,CLASS:87,CLICK:88,DOWN:89,UP:90,textNoTagsToken:91,stylesOpt:92,"idString[vertex]":93,"idString[class]":94,CALLBACKNAME:95,CALLBACKARGS:96,HREF:97,LINK_TARGET:98,"STR[link]":99,"STR[tooltip]":100,alphaNum:101,DEFAULT:102,numList:103,INTERPOLATE:104,NUM:105,COMMA:106,style:107,styleComponent:108,NODE_STRING:109,UNIT:110,BRKT:111,PCT:112,idStringToken:113,MINUS:114,MULT:115,UNICODE_TEXT:116,TEXT:117,TAGSTART:118,EDGE_TEXT:119,alphaNumToken:120,direction_tb:121,direction_bt:122,direction_rl:123,direction_lr:124,$accept:0,$end:1},terminals_:{2:"error",8:"SEMI",9:"NEWLINE",10:"SPACE",11:"EOF",12:"GRAPH",13:"NODIR",14:"DIR",27:"subgraph",29:"SQS",31:"SQE",32:"end",34:"acc_title",35:"acc_title_value",36:"acc_descr",37:"acc_descr_value",38:"acc_descr_multiline_value",40:"SHAPE_DATA",44:"AMP",46:"STYLE_SEPARATOR",48:"DOUBLECIRCLESTART",49:"DOUBLECIRCLEEND",50:"PS",51:"PE",52:"(-",53:"-)",54:"STADIUMSTART",55:"STADIUMEND",56:"SUBROUTINESTART",57:"SUBROUTINEEND",58:"VERTEX_WITH_PROPS_START",59:"NODE_STRING[field]",60:"COLON",61:"NODE_STRING[value]",62:"PIPE",63:"CYLINDERSTART",64:"CYLINDEREND",65:"DIAMOND_START",66:"DIAMOND_STOP",67:"TAGEND",68:"TRAPSTART",69:"TRAPEND",70:"INVTRAPSTART",71:"INVTRAPEND",74:"TESTSTR",75:"START_LINK",77:"LINK",78:"LINK_ID",80:"STR",81:"MD_STR",84:"STYLE",85:"LINKSTYLE",86:"CLASSDEF",87:"CLASS",88:"CLICK",89:"DOWN",90:"UP",93:"idString[vertex]",94:"idString[class]",95:"CALLBACKNAME",96:"CALLBACKARGS",97:"HREF",98:"LINK_TARGET",99:"STR[link]",100:"STR[tooltip]",102:"DEFAULT",104:"INTERPOLATE",105:"NUM",106:"COMMA",109:"NODE_STRING",110:"UNIT",111:"BRKT",112:"PCT",114:"MINUS",115:"MULT",116:"UNICODE_TEXT",117:"TEXT",118:"TAGSTART",119:"EDGE_TEXT",121:"direction_tb",122:"direction_bt",123:"direction_rl",124:"direction_lr"},productions_:[0,[3,2],[5,0],[5,2],[6,1],[6,1],[6,1],[6,1],[6,1],[4,2],[4,2],[4,2],[4,3],[16,2],[16,1],[17,1],[17,1],[17,1],[15,1],[15,1],[15,2],[19,2],[19,2],[19,1],[19,1],[18,2],[18,1],[7,2],[7,2],[7,2],[7,2],[7,2],[7,2],[7,9],[7,6],[7,4],[7,1],[7,2],[7,2],[7,1],[21,1],[21,1],[21,1],[39,2],[39,1],[20,4],[20,3],[20,4],[20,2],[20,2],[20,1],[42,1],[42,6],[42,5],[43,1],[43,3],[45,4],[45,4],[45,6],[45,4],[45,4],[45,4],[45,8],[45,4],[45,4],[45,4],[45,6],[45,4],[45,4],[45,4],[45,4],[45,4],[45,1],[41,2],[41,3],[41,3],[41,1],[41,3],[41,4],[76,1],[76,2],[76,1],[76,1],[72,1],[72,2],[73,3],[30,1],[30,2],[30,1],[30,1],[83,1],[83,1],[83,1],[83,1],[83,1],[83,1],[83,1],[83,1],[83,1],[83,1],[83,1],[28,1],[28,2],[28,1],[28,1],[24,5],[25,5],[26,2],[26,4],[26,3],[26,5],[26,3],[26,5],[26,5],[26,7],[26,2],[26,4],[26,2],[26,4],[26,4],[26,6],[22,5],[23,5],[23,5],[23,9],[23,9],[23,7],[23,7],[103,1],[103,3],[92,1],[92,3],[107,1],[107,2],[108,1],[108,1],[108,1],[108,1],[108,1],[108,1],[108,1],[108,1],[113,1],[113,1],[113,1],[113,1],[113,1],[113,1],[113,1],[113,1],[113,1],[113,1],[113,1],[82,1],[82,1],[82,1],[82,1],[91,1],[91,1],[91,1],[91,1],[91,1],[91,1],[91,1],[91,1],[91,1],[91,1],[91,1],[79,1],[79,1],[120,1],[120,1],[120,1],[120,1],[120,1],[120,1],[120,1],[120,1],[120,1],[120,1],[120,1],[47,1],[47,2],[101,1],[101,2],[33,1],[33,1],[33,1],[33,1]],performAction:o(function(et,mt,Kt,lt,Cn,ye,Vf){var Te=ye.length-1;switch(Cn){case 2:this.$=[];break;case 3:(!Array.isArray(ye[Te])||ye[Te].length>0)&&ye[Te-1].push(ye[Te]),this.$=ye[Te-1];break;case 4:case 183:this.$=ye[Te];break;case 11:lt.setDirection("TB"),this.$="TB";break;case 12:lt.setDirection(ye[Te-1]),this.$=ye[Te-1];break;case 27:this.$=ye[Te-1].nodes;break;case 28:case 29:case 30:case 31:case 32:this.$=[];break;case 33:this.$=lt.addSubGraph(ye[Te-6],ye[Te-1],ye[Te-4]);break;case 34:this.$=lt.addSubGraph(ye[Te-3],ye[Te-1],ye[Te-3]);break;case 35:this.$=lt.addSubGraph(void 0,ye[Te-1],void 0);break;case 37:this.$=ye[Te].trim(),lt.setAccTitle(this.$);break;case 38:case 39:this.$=ye[Te].trim(),lt.setAccDescription(this.$);break;case 43:this.$=ye[Te-1]+ye[Te];break;case 44:this.$=ye[Te];break;case 45:lt.addVertex(ye[Te-1][ye[Te-1].length-1],void 0,void 0,void 0,void 0,void 0,void 0,ye[Te]),lt.addLink(ye[Te-3].stmt,ye[Te-1],ye[Te-2]),this.$={stmt:ye[Te-1],nodes:ye[Te-1].concat(ye[Te-3].nodes)};break;case 46:lt.addLink(ye[Te-2].stmt,ye[Te],ye[Te-1]),this.$={stmt:ye[Te],nodes:ye[Te].concat(ye[Te-2].nodes)};break;case 47:lt.addLink(ye[Te-3].stmt,ye[Te-1],ye[Te-2]),this.$={stmt:ye[Te-1],nodes:ye[Te-1].concat(ye[Te-3].nodes)};break;case 48:this.$={stmt:ye[Te-1],nodes:ye[Te-1]};break;case 49:lt.addVertex(ye[Te-1][ye[Te-1].length-1],void 0,void 0,void 0,void 0,void 0,void 0,ye[Te]),this.$={stmt:ye[Te-1],nodes:ye[Te-1],shapeData:ye[Te]};break;case 50:this.$={stmt:ye[Te],nodes:ye[Te]};break;case 51:this.$=[ye[Te]];break;case 52:lt.addVertex(ye[Te-5][ye[Te-5].length-1],void 0,void 0,void 0,void 0,void 0,void 0,ye[Te-4]),this.$=ye[Te-5].concat(ye[Te]);break;case 53:this.$=ye[Te-4].concat(ye[Te]);break;case 54:this.$=ye[Te];break;case 55:this.$=ye[Te-2],lt.setClass(ye[Te-2],ye[Te]);break;case 56:this.$=ye[Te-3],lt.addVertex(ye[Te-3],ye[Te-1],"square");break;case 57:this.$=ye[Te-3],lt.addVertex(ye[Te-3],ye[Te-1],"doublecircle");break;case 58:this.$=ye[Te-5],lt.addVertex(ye[Te-5],ye[Te-2],"circle");break;case 59:this.$=ye[Te-3],lt.addVertex(ye[Te-3],ye[Te-1],"ellipse");break;case 60:this.$=ye[Te-3],lt.addVertex(ye[Te-3],ye[Te-1],"stadium");break;case 61:this.$=ye[Te-3],lt.addVertex(ye[Te-3],ye[Te-1],"subroutine");break;case 62:this.$=ye[Te-7],lt.addVertex(ye[Te-7],ye[Te-1],"rect",void 0,void 0,void 0,Object.fromEntries([[ye[Te-5],ye[Te-3]]]));break;case 63:this.$=ye[Te-3],lt.addVertex(ye[Te-3],ye[Te-1],"cylinder");break;case 64:this.$=ye[Te-3],lt.addVertex(ye[Te-3],ye[Te-1],"round");break;case 65:this.$=ye[Te-3],lt.addVertex(ye[Te-3],ye[Te-1],"diamond");break;case 66:this.$=ye[Te-5],lt.addVertex(ye[Te-5],ye[Te-2],"hexagon");break;case 67:this.$=ye[Te-3],lt.addVertex(ye[Te-3],ye[Te-1],"odd");break;case 68:this.$=ye[Te-3],lt.addVertex(ye[Te-3],ye[Te-1],"trapezoid");break;case 69:this.$=ye[Te-3],lt.addVertex(ye[Te-3],ye[Te-1],"inv_trapezoid");break;case 70:this.$=ye[Te-3],lt.addVertex(ye[Te-3],ye[Te-1],"lean_right");break;case 71:this.$=ye[Te-3],lt.addVertex(ye[Te-3],ye[Te-1],"lean_left");break;case 72:this.$=ye[Te],lt.addVertex(ye[Te]);break;case 73:ye[Te-1].text=ye[Te],this.$=ye[Te-1];break;case 74:case 75:ye[Te-2].text=ye[Te-1],this.$=ye[Te-2];break;case 76:this.$=ye[Te];break;case 77:var wi=lt.destructLink(ye[Te],ye[Te-2]);this.$={type:wi.type,stroke:wi.stroke,length:wi.length,text:ye[Te-1]};break;case 78:var wi=lt.destructLink(ye[Te],ye[Te-2]);this.$={type:wi.type,stroke:wi.stroke,length:wi.length,text:ye[Te-1],id:ye[Te-3]};break;case 79:this.$={text:ye[Te],type:"text"};break;case 80:this.$={text:ye[Te-1].text+""+ye[Te],type:ye[Te-1].type};break;case 81:this.$={text:ye[Te],type:"string"};break;case 82:this.$={text:ye[Te],type:"markdown"};break;case 83:var wi=lt.destructLink(ye[Te]);this.$={type:wi.type,stroke:wi.stroke,length:wi.length};break;case 84:var wi=lt.destructLink(ye[Te]);this.$={type:wi.type,stroke:wi.stroke,length:wi.length,id:ye[Te-1]};break;case 85:this.$=ye[Te-1];break;case 86:this.$={text:ye[Te],type:"text"};break;case 87:this.$={text:ye[Te-1].text+""+ye[Te],type:ye[Te-1].type};break;case 88:this.$={text:ye[Te],type:"string"};break;case 89:case 104:this.$={text:ye[Te],type:"markdown"};break;case 101:this.$={text:ye[Te],type:"text"};break;case 102:this.$={text:ye[Te-1].text+""+ye[Te],type:ye[Te-1].type};break;case 103:this.$={text:ye[Te],type:"text"};break;case 105:this.$=ye[Te-4],lt.addClass(ye[Te-2],ye[Te]);break;case 106:this.$=ye[Te-4],lt.setClass(ye[Te-2],ye[Te]);break;case 107:case 115:this.$=ye[Te-1],lt.setClickEvent(ye[Te-1],ye[Te]);break;case 108:case 116:this.$=ye[Te-3],lt.setClickEvent(ye[Te-3],ye[Te-2]),lt.setTooltip(ye[Te-3],ye[Te]);break;case 109:this.$=ye[Te-2],lt.setClickEvent(ye[Te-2],ye[Te-1],ye[Te]);break;case 110:this.$=ye[Te-4],lt.setClickEvent(ye[Te-4],ye[Te-3],ye[Te-2]),lt.setTooltip(ye[Te-4],ye[Te]);break;case 111:this.$=ye[Te-2],lt.setLink(ye[Te-2],ye[Te]);break;case 112:this.$=ye[Te-4],lt.setLink(ye[Te-4],ye[Te-2]),lt.setTooltip(ye[Te-4],ye[Te]);break;case 113:this.$=ye[Te-4],lt.setLink(ye[Te-4],ye[Te-2],ye[Te]);break;case 114:this.$=ye[Te-6],lt.setLink(ye[Te-6],ye[Te-4],ye[Te]),lt.setTooltip(ye[Te-6],ye[Te-2]);break;case 117:this.$=ye[Te-1],lt.setLink(ye[Te-1],ye[Te]);break;case 118:this.$=ye[Te-3],lt.setLink(ye[Te-3],ye[Te-2]),lt.setTooltip(ye[Te-3],ye[Te]);break;case 119:this.$=ye[Te-3],lt.setLink(ye[Te-3],ye[Te-2],ye[Te]);break;case 120:this.$=ye[Te-5],lt.setLink(ye[Te-5],ye[Te-4],ye[Te]),lt.setTooltip(ye[Te-5],ye[Te-2]);break;case 121:this.$=ye[Te-4],lt.addVertex(ye[Te-2],void 0,void 0,ye[Te]);break;case 122:this.$=ye[Te-4],lt.updateLink([ye[Te-2]],ye[Te]);break;case 123:this.$=ye[Te-4],lt.updateLink(ye[Te-2],ye[Te]);break;case 124:this.$=ye[Te-8],lt.updateLinkInterpolate([ye[Te-6]],ye[Te-2]),lt.updateLink([ye[Te-6]],ye[Te]);break;case 125:this.$=ye[Te-8],lt.updateLinkInterpolate(ye[Te-6],ye[Te-2]),lt.updateLink(ye[Te-6],ye[Te]);break;case 126:this.$=ye[Te-6],lt.updateLinkInterpolate([ye[Te-4]],ye[Te]);break;case 127:this.$=ye[Te-6],lt.updateLinkInterpolate(ye[Te-4],ye[Te]);break;case 128:case 130:this.$=[ye[Te]];break;case 129:case 131:ye[Te-2].push(ye[Te]),this.$=ye[Te-2];break;case 133:this.$=ye[Te-1]+ye[Te];break;case 181:this.$=ye[Te];break;case 182:this.$=ye[Te-1]+""+ye[Te];break;case 184:this.$=ye[Te-1]+""+ye[Te];break;case 185:this.$={stmt:"dir",value:"TB"};break;case 186:this.$={stmt:"dir",value:"BT"};break;case 187:this.$={stmt:"dir",value:"RL"};break;case 188:this.$={stmt:"dir",value:"LR"};break}},"anonymous"),table:[{3:1,4:2,9:e,10:r,12:n},{1:[3]},t(i,a,{5:6}),{4:7,9:e,10:r,12:n},{4:8,9:e,10:r,12:n},{13:[1,9],14:[1,10]},{1:[2,1],6:11,7:12,8:s,9:l,10:u,11:h,20:17,22:18,23:19,24:20,25:21,26:22,27:f,33:24,34:d,36:p,38:m,42:28,43:38,44:g,45:39,47:40,60:y,84:v,85:x,86:b,87:w,88:C,89:T,102:E,105:A,106:S,109:_,111:I,113:41,114:D,115:k,116:L,121:R,122:O,123:M,124:B},t(i,[2,9]),t(i,[2,10]),t(i,[2,11]),{8:[1,54],9:[1,55],10:F,15:53,18:56},t(P,[2,3]),t(P,[2,4]),t(P,[2,5]),t(P,[2,6]),t(P,[2,7]),t(P,[2,8]),{8:z,9:$,11:H,21:58,41:59,72:63,75:[1,64],77:[1,66],78:[1,65]},{8:z,9:$,11:H,21:67},{8:z,9:$,11:H,21:68},{8:z,9:$,11:H,21:69},{8:z,9:$,11:H,21:70},{8:z,9:$,11:H,21:71},{8:z,9:$,10:[1,72],11:H,21:73},t(P,[2,36]),{35:[1,74]},{37:[1,75]},t(P,[2,39]),t(Q,[2,50],{18:76,39:77,10:F,40:j}),{10:[1,79]},{10:[1,80]},{10:[1,81]},{10:[1,82]},{14:ie,44:ne,60:le,80:[1,86],89:he,95:[1,83],97:[1,84],101:85,105:K,106:X,109:te,111:J,114:se,115:ue,116:Z,120:87},t(P,[2,185]),t(P,[2,186]),t(P,[2,187]),t(P,[2,188]),t(Se,[2,51]),t(Se,[2,54],{46:[1,99]}),t(ce,[2,72],{113:112,29:[1,100],44:g,48:[1,101],50:[1,102],52:[1,103],54:[1,104],56:[1,105],58:[1,106],60:y,63:[1,107],65:[1,108],67:[1,109],68:[1,110],70:[1,111],89:T,102:E,105:A,106:S,109:_,111:I,114:D,115:k,116:L}),t(ae,[2,181]),t(ae,[2,142]),t(ae,[2,143]),t(ae,[2,144]),t(ae,[2,145]),t(ae,[2,146]),t(ae,[2,147]),t(ae,[2,148]),t(ae,[2,149]),t(ae,[2,150]),t(ae,[2,151]),t(ae,[2,152]),t(i,[2,12]),t(i,[2,18]),t(i,[2,19]),{9:[1,113]},t(Oe,[2,26],{18:114,10:F}),t(P,[2,27]),{42:115,43:38,44:g,45:39,47:40,60:y,89:T,102:E,105:A,106:S,109:_,111:I,113:41,114:D,115:k,116:L},t(P,[2,40]),t(P,[2,41]),t(P,[2,42]),t(ge,[2,76],{73:116,62:[1,118],74:[1,117]}),{76:119,79:120,80:ze,81:He,116:$e,119:Re},{75:[1,125],77:[1,126]},t(Ie,[2,83]),t(P,[2,28]),t(P,[2,29]),t(P,[2,30]),t(P,[2,31]),t(P,[2,32]),{10:be,12:W,14:de,27:re,28:127,32:oe,44:V,60:xe,75:q,80:[1,129],81:[1,130],83:140,84:pe,85:ve,86:Pe,87:_e,88:we,89:Ve,90:De,91:128,105:qe,109:at,111:Rt,114:st,115:Ue,116:ct},t(We,a,{5:153}),t(P,[2,37]),t(P,[2,38]),t(Q,[2,48],{44:ot}),t(Q,[2,49],{18:155,10:F,40:Yt}),t(Se,[2,44]),{44:g,47:157,60:y,89:T,102:E,105:A,106:S,109:_,111:I,113:41,114:D,115:k,116:L},{102:[1,158],103:159,105:[1,160]},{44:g,47:161,60:y,89:T,102:E,105:A,106:S,109:_,111:I,113:41,114:D,115:k,116:L},{44:g,47:162,60:y,89:T,102:E,105:A,106:S,109:_,111:I,113:41,114:D,115:k,116:L},t(bt,[2,107],{10:[1,163],96:[1,164]}),{80:[1,165]},t(bt,[2,115],{120:167,10:[1,166],14:ie,44:ne,60:le,89:he,105:K,106:X,109:te,111:J,114:se,115:ue,116:Z}),t(bt,[2,117],{10:[1,168]}),t(Mt,[2,183]),t(Mt,[2,170]),t(Mt,[2,171]),t(Mt,[2,172]),t(Mt,[2,173]),t(Mt,[2,174]),t(Mt,[2,175]),t(Mt,[2,176]),t(Mt,[2,177]),t(Mt,[2,178]),t(Mt,[2,179]),t(Mt,[2,180]),{44:g,47:169,60:y,89:T,102:E,105:A,106:S,109:_,111:I,113:41,114:D,115:k,116:L},{30:170,67:xt,80:ut,81:Et,82:171,116:ft,117:yt,118:nt},{30:178,67:xt,80:ut,81:Et,82:171,116:ft,117:yt,118:nt},{30:180,50:[1,179],67:xt,80:ut,81:Et,82:171,116:ft,117:yt,118:nt},{30:181,67:xt,80:ut,81:Et,82:171,116:ft,117:yt,118:nt},{30:182,67:xt,80:ut,81:Et,82:171,116:ft,117:yt,118:nt},{30:183,67:xt,80:ut,81:Et,82:171,116:ft,117:yt,118:nt},{109:[1,184]},{30:185,67:xt,80:ut,81:Et,82:171,116:ft,117:yt,118:nt},{30:186,65:[1,187],67:xt,80:ut,81:Et,82:171,116:ft,117:yt,118:nt},{30:188,67:xt,80:ut,81:Et,82:171,116:ft,117:yt,118:nt},{30:189,67:xt,80:ut,81:Et,82:171,116:ft,117:yt,118:nt},{30:190,67:xt,80:ut,81:Et,82:171,116:ft,117:yt,118:nt},t(ae,[2,182]),t(i,[2,20]),t(Oe,[2,25]),t(Q,[2,46],{39:191,18:192,10:F,40:j}),t(ge,[2,73],{10:[1,193]}),{10:[1,194]},{30:195,67:xt,80:ut,81:Et,82:171,116:ft,117:yt,118:nt},{77:[1,196],79:197,116:$e,119:Re},t(dn,[2,79]),t(dn,[2,81]),t(dn,[2,82]),t(dn,[2,168]),t(dn,[2,169]),{76:198,79:120,80:ze,81:He,116:$e,119:Re},t(Ie,[2,84]),{8:z,9:$,10:be,11:H,12:W,14:de,21:200,27:re,29:[1,199],32:oe,44:V,60:xe,75:q,83:140,84:pe,85:ve,86:Pe,87:_e,88:we,89:Ve,90:De,91:201,105:qe,109:at,111:Rt,114:st,115:Ue,116:ct},t(Tt,[2,101]),t(Tt,[2,103]),t(Tt,[2,104]),t(Tt,[2,157]),t(Tt,[2,158]),t(Tt,[2,159]),t(Tt,[2,160]),t(Tt,[2,161]),t(Tt,[2,162]),t(Tt,[2,163]),t(Tt,[2,164]),t(Tt,[2,165]),t(Tt,[2,166]),t(Tt,[2,167]),t(Tt,[2,90]),t(Tt,[2,91]),t(Tt,[2,92]),t(Tt,[2,93]),t(Tt,[2,94]),t(Tt,[2,95]),t(Tt,[2,96]),t(Tt,[2,97]),t(Tt,[2,98]),t(Tt,[2,99]),t(Tt,[2,100]),{6:11,7:12,8:s,9:l,10:u,11:h,20:17,22:18,23:19,24:20,25:21,26:22,27:f,32:[1,202],33:24,34:d,36:p,38:m,42:28,43:38,44:g,45:39,47:40,60:y,84:v,85:x,86:b,87:w,88:C,89:T,102:E,105:A,106:S,109:_,111:I,113:41,114:D,115:k,116:L,121:R,122:O,123:M,124:B},{10:F,18:203},{44:[1,204]},t(Se,[2,43]),{10:[1,205],44:g,60:y,89:T,102:E,105:A,106:S,109:_,111:I,113:112,114:D,115:k,116:L},{10:[1,206]},{10:[1,207],106:[1,208]},t(On,[2,128]),{10:[1,209],44:g,60:y,89:T,102:E,105:A,106:S,109:_,111:I,113:112,114:D,115:k,116:L},{10:[1,210],44:g,60:y,89:T,102:E,105:A,106:S,109:_,111:I,113:112,114:D,115:k,116:L},{80:[1,211]},t(bt,[2,109],{10:[1,212]}),t(bt,[2,111],{10:[1,213]}),{80:[1,214]},t(Mt,[2,184]),{80:[1,215],98:[1,216]},t(Se,[2,55],{113:112,44:g,60:y,89:T,102:E,105:A,106:S,109:_,111:I,114:D,115:k,116:L}),{31:[1,217],67:xt,82:218,116:ft,117:yt,118:nt},t(tn,[2,86]),t(tn,[2,88]),t(tn,[2,89]),t(tn,[2,153]),t(tn,[2,154]),t(tn,[2,155]),t(tn,[2,156]),{49:[1,219],67:xt,82:218,116:ft,117:yt,118:nt},{30:220,67:xt,80:ut,81:Et,82:171,116:ft,117:yt,118:nt},{51:[1,221],67:xt,82:218,116:ft,117:yt,118:nt},{53:[1,222],67:xt,82:218,116:ft,117:yt,118:nt},{55:[1,223],67:xt,82:218,116:ft,117:yt,118:nt},{57:[1,224],67:xt,82:218,116:ft,117:yt,118:nt},{60:[1,225]},{64:[1,226],67:xt,82:218,116:ft,117:yt,118:nt},{66:[1,227],67:xt,82:218,116:ft,117:yt,118:nt},{30:228,67:xt,80:ut,81:Et,82:171,116:ft,117:yt,118:nt},{31:[1,229],67:xt,82:218,116:ft,117:yt,118:nt},{67:xt,69:[1,230],71:[1,231],82:218,116:ft,117:yt,118:nt},{67:xt,69:[1,233],71:[1,232],82:218,116:ft,117:yt,118:nt},t(Q,[2,45],{18:155,10:F,40:Yt}),t(Q,[2,47],{44:ot}),t(ge,[2,75]),t(ge,[2,74]),{62:[1,234],67:xt,82:218,116:ft,117:yt,118:nt},t(ge,[2,77]),t(dn,[2,80]),{77:[1,235],79:197,116:$e,119:Re},{30:236,67:xt,80:ut,81:Et,82:171,116:ft,117:yt,118:nt},t(We,a,{5:237}),t(Tt,[2,102]),t(P,[2,35]),{43:238,44:g,45:39,47:40,60:y,89:T,102:E,105:A,106:S,109:_,111:I,113:41,114:D,115:k,116:L},{10:F,18:239},{10:_r,60:Dr,84:Pn,92:240,105:At,107:241,108:242,109:Ce,110:tt,111:St,112:mr},{10:_r,60:Dr,84:Pn,92:251,104:[1,252],105:At,107:241,108:242,109:Ce,110:tt,111:St,112:mr},{10:_r,60:Dr,84:Pn,92:253,104:[1,254],105:At,107:241,108:242,109:Ce,110:tt,111:St,112:mr},{105:[1,255]},{10:_r,60:Dr,84:Pn,92:256,105:At,107:241,108:242,109:Ce,110:tt,111:St,112:mr},{44:g,47:257,60:y,89:T,102:E,105:A,106:S,109:_,111:I,113:41,114:D,115:k,116:L},t(bt,[2,108]),{80:[1,258]},{80:[1,259],98:[1,260]},t(bt,[2,116]),t(bt,[2,118],{10:[1,261]}),t(bt,[2,119]),t(ce,[2,56]),t(tn,[2,87]),t(ce,[2,57]),{51:[1,262],67:xt,82:218,116:ft,117:yt,118:nt},t(ce,[2,64]),t(ce,[2,59]),t(ce,[2,60]),t(ce,[2,61]),{109:[1,263]},t(ce,[2,63]),t(ce,[2,65]),{66:[1,264],67:xt,82:218,116:ft,117:yt,118:nt},t(ce,[2,67]),t(ce,[2,68]),t(ce,[2,70]),t(ce,[2,69]),t(ce,[2,71]),t([10,44,60,89,102,105,106,109,111,114,115,116],[2,85]),t(ge,[2,78]),{31:[1,265],67:xt,82:218,116:ft,117:yt,118:nt},{6:11,7:12,8:s,9:l,10:u,11:h,20:17,22:18,23:19,24:20,25:21,26:22,27:f,32:[1,266],33:24,34:d,36:p,38:m,42:28,43:38,44:g,45:39,47:40,60:y,84:v,85:x,86:b,87:w,88:C,89:T,102:E,105:A,106:S,109:_,111:I,113:41,114:D,115:k,116:L,121:R,122:O,123:M,124:B},t(Se,[2,53]),{43:267,44:g,45:39,47:40,60:y,89:T,102:E,105:A,106:S,109:_,111:I,113:41,114:D,115:k,116:L},t(bt,[2,121],{106:rn}),t(gn,[2,130],{108:269,10:_r,60:Dr,84:Pn,105:At,109:Ce,110:tt,111:St,112:mr}),t(Zr,[2,132]),t(Zr,[2,134]),t(Zr,[2,135]),t(Zr,[2,136]),t(Zr,[2,137]),t(Zr,[2,138]),t(Zr,[2,139]),t(Zr,[2,140]),t(Zr,[2,141]),t(bt,[2,122],{106:rn}),{10:[1,270]},t(bt,[2,123],{106:rn}),{10:[1,271]},t(On,[2,129]),t(bt,[2,105],{106:rn}),t(bt,[2,106],{113:112,44:g,60:y,89:T,102:E,105:A,106:S,109:_,111:I,114:D,115:k,116:L}),t(bt,[2,110]),t(bt,[2,112],{10:[1,272]}),t(bt,[2,113]),{98:[1,273]},{51:[1,274]},{62:[1,275]},{66:[1,276]},{8:z,9:$,11:H,21:277},t(P,[2,34]),t(Se,[2,52]),{10:_r,60:Dr,84:Pn,105:At,107:278,108:242,109:Ce,110:tt,111:St,112:mr},t(Zr,[2,133]),{14:ie,44:ne,60:le,89:he,101:279,105:K,106:X,109:te,111:J,114:se,115:ue,116:Z,120:87},{14:ie,44:ne,60:le,89:he,101:280,105:K,106:X,109:te,111:J,114:se,115:ue,116:Z,120:87},{98:[1,281]},t(bt,[2,120]),t(ce,[2,58]),{30:282,67:xt,80:ut,81:Et,82:171,116:ft,117:yt,118:nt},t(ce,[2,66]),t(We,a,{5:283}),t(gn,[2,131],{108:269,10:_r,60:Dr,84:Pn,105:At,109:Ce,110:tt,111:St,112:mr}),t(bt,[2,126],{120:167,10:[1,284],14:ie,44:ne,60:le,89:he,105:K,106:X,109:te,111:J,114:se,115:ue,116:Z}),t(bt,[2,127],{120:167,10:[1,285],14:ie,44:ne,60:le,89:he,105:K,106:X,109:te,111:J,114:se,115:ue,116:Z}),t(bt,[2,114]),{31:[1,286],67:xt,82:218,116:ft,117:yt,118:nt},{6:11,7:12,8:s,9:l,10:u,11:h,20:17,22:18,23:19,24:20,25:21,26:22,27:f,32:[1,287],33:24,34:d,36:p,38:m,42:28,43:38,44:g,45:39,47:40,60:y,84:v,85:x,86:b,87:w,88:C,89:T,102:E,105:A,106:S,109:_,111:I,113:41,114:D,115:k,116:L,121:R,122:O,123:M,124:B},{10:_r,60:Dr,84:Pn,92:288,105:At,107:241,108:242,109:Ce,110:tt,111:St,112:mr},{10:_r,60:Dr,84:Pn,92:289,105:At,107:241,108:242,109:Ce,110:tt,111:St,112:mr},t(ce,[2,62]),t(P,[2,33]),t(bt,[2,124],{106:rn}),t(bt,[2,125],{106:rn})],defaultActions:{},parseError:o(function(et,mt){if(mt.recoverable)this.trace(et);else{var Kt=new Error(et);throw Kt.hash=mt,Kt}},"parseError"),parse:o(function(et){var mt=this,Kt=[0],lt=[],Cn=[null],ye=[],Vf=this.table,Te="",wi=0,TF=0,kF=0,M2e=2,EF=1,I2e=ye.slice.call(arguments,1),Xi=Object.create(this.lexer),Uf={yy:{}};for(var xC in this.yy)Object.prototype.hasOwnProperty.call(this.yy,xC)&&(Uf.yy[xC]=this.yy[xC]);Xi.setInput(et,Uf.yy),Uf.yy.lexer=Xi,Uf.yy.parser=this,typeof Xi.yylloc>"u"&&(Xi.yylloc={});var bC=Xi.yylloc;ye.push(bC);var O2e=Xi.options&&Xi.options.ranges;typeof Uf.yy.parseError=="function"?this.parseError=Uf.yy.parseError:this.parseError=Object.getPrototypeOf(this).parseError;function wnt(Ws){Kt.length=Kt.length-2*Ws,Cn.length=Cn.length-Ws,ye.length=ye.length-Ws}o(wnt,"popStack");function P2e(){var Ws;return Ws=lt.pop()||Xi.lex()||EF,typeof Ws!="number"&&(Ws instanceof Array&&(lt=Ws,Ws=lt.pop()),Ws=mt.symbols_[Ws]||Ws),Ws}o(P2e,"lex");for(var Wa,wC,Hf,xo,Tnt,TC,Jp={},_4,Jc,SF,D4;;){if(Hf=Kt[Kt.length-1],this.defaultActions[Hf]?xo=this.defaultActions[Hf]:((Wa===null||typeof Wa>"u")&&(Wa=P2e()),xo=Vf[Hf]&&Vf[Hf][Wa]),typeof xo>"u"||!xo.length||!xo[0]){var kC="";D4=[];for(_4 in Vf[Hf])this.terminals_[_4]&&_4>M2e&&D4.push("'"+this.terminals_[_4]+"'");Xi.showPosition?kC="Parse error on line "+(wi+1)+`: +`+Xi.showPosition()+` +Expecting `+D4.join(", ")+", got '"+(this.terminals_[Wa]||Wa)+"'":kC="Parse error on line "+(wi+1)+": Unexpected "+(Wa==EF?"end of input":"'"+(this.terminals_[Wa]||Wa)+"'"),this.parseError(kC,{text:Xi.match,token:this.terminals_[Wa]||Wa,line:Xi.yylineno,loc:bC,expected:D4})}if(xo[0]instanceof Array&&xo.length>1)throw new Error("Parse Error: multiple actions possible at state: "+Hf+", token: "+Wa);switch(xo[0]){case 1:Kt.push(Wa),Cn.push(Xi.yytext),ye.push(Xi.yylloc),Kt.push(xo[1]),Wa=null,wC?(Wa=wC,wC=null):(TF=Xi.yyleng,Te=Xi.yytext,wi=Xi.yylineno,bC=Xi.yylloc,kF>0&&kF--);break;case 2:if(Jc=this.productions_[xo[1]][1],Jp.$=Cn[Cn.length-Jc],Jp._$={first_line:ye[ye.length-(Jc||1)].first_line,last_line:ye[ye.length-1].last_line,first_column:ye[ye.length-(Jc||1)].first_column,last_column:ye[ye.length-1].last_column},O2e&&(Jp._$.range=[ye[ye.length-(Jc||1)].range[0],ye[ye.length-1].range[1]]),TC=this.performAction.apply(Jp,[Te,TF,wi,Uf.yy,xo[1],Cn,ye].concat(I2e)),typeof TC<"u")return TC;Jc&&(Kt=Kt.slice(0,-1*Jc*2),Cn=Cn.slice(0,-1*Jc),ye=ye.slice(0,-1*Jc)),Kt.push(this.productions_[xo[1]][0]),Cn.push(Jp.$),ye.push(Jp._$),SF=Vf[Kt[Kt.length-2]][Kt[Kt.length-1]],Kt.push(SF);break;case 3:return!0}}return!0},"parse")},Zn=function(){var Hr={EOF:1,parseError:o(function(mt,Kt){if(this.yy.parser)this.yy.parser.parseError(mt,Kt);else throw new Error(mt)},"parseError"),setInput:o(function(et,mt){return this.yy=mt||this.yy||{},this._input=et,this._more=this._backtrack=this.done=!1,this.yylineno=this.yyleng=0,this.yytext=this.matched=this.match="",this.conditionStack=["INITIAL"],this.yylloc={first_line:1,first_column:0,last_line:1,last_column:0},this.options.ranges&&(this.yylloc.range=[0,0]),this.offset=0,this},"setInput"),input:o(function(){var et=this._input[0];this.yytext+=et,this.yyleng++,this.offset++,this.match+=et,this.matched+=et;var mt=et.match(/(?:\r\n?|\n).*/g);return mt?(this.yylineno++,this.yylloc.last_line++):this.yylloc.last_column++,this.options.ranges&&this.yylloc.range[1]++,this._input=this._input.slice(1),et},"input"),unput:o(function(et){var mt=et.length,Kt=et.split(/(?:\r\n?|\n)/g);this._input=et+this._input,this.yytext=this.yytext.substr(0,this.yytext.length-mt),this.offset-=mt;var lt=this.match.split(/(?:\r\n?|\n)/g);this.match=this.match.substr(0,this.match.length-1),this.matched=this.matched.substr(0,this.matched.length-1),Kt.length-1&&(this.yylineno-=Kt.length-1);var Cn=this.yylloc.range;return this.yylloc={first_line:this.yylloc.first_line,last_line:this.yylineno+1,first_column:this.yylloc.first_column,last_column:Kt?(Kt.length===lt.length?this.yylloc.first_column:0)+lt[lt.length-Kt.length].length-Kt[0].length:this.yylloc.first_column-mt},this.options.ranges&&(this.yylloc.range=[Cn[0],Cn[0]+this.yyleng-mt]),this.yyleng=this.yytext.length,this},"unput"),more:o(function(){return this._more=!0,this},"more"),reject:o(function(){if(this.options.backtrack_lexer)this._backtrack=!0;else return this.parseError("Lexical error on line "+(this.yylineno+1)+`. You can only invoke reject() in the lexer when the lexer is of the backtracking persuasion (options.backtrack_lexer = true). +`+this.showPosition(),{text:"",token:null,line:this.yylineno});return this},"reject"),less:o(function(et){this.unput(this.match.slice(et))},"less"),pastInput:o(function(){var et=this.matched.substr(0,this.matched.length-this.match.length);return(et.length>20?"...":"")+et.substr(-20).replace(/\n/g,"")},"pastInput"),upcomingInput:o(function(){var et=this.match;return et.length<20&&(et+=this._input.substr(0,20-et.length)),(et.substr(0,20)+(et.length>20?"...":"")).replace(/\n/g,"")},"upcomingInput"),showPosition:o(function(){var et=this.pastInput(),mt=new Array(et.length+1).join("-");return et+this.upcomingInput()+` +`+mt+"^"},"showPosition"),test_match:o(function(et,mt){var Kt,lt,Cn;if(this.options.backtrack_lexer&&(Cn={yylineno:this.yylineno,yylloc:{first_line:this.yylloc.first_line,last_line:this.last_line,first_column:this.yylloc.first_column,last_column:this.yylloc.last_column},yytext:this.yytext,match:this.match,matches:this.matches,matched:this.matched,yyleng:this.yyleng,offset:this.offset,_more:this._more,_input:this._input,yy:this.yy,conditionStack:this.conditionStack.slice(0),done:this.done},this.options.ranges&&(Cn.yylloc.range=this.yylloc.range.slice(0))),lt=et[0].match(/(?:\r\n?|\n).*/g),lt&&(this.yylineno+=lt.length),this.yylloc={first_line:this.yylloc.last_line,last_line:this.yylineno+1,first_column:this.yylloc.last_column,last_column:lt?lt[lt.length-1].length-lt[lt.length-1].match(/\r?\n?/)[0].length:this.yylloc.last_column+et[0].length},this.yytext+=et[0],this.match+=et[0],this.matches=et,this.yyleng=this.yytext.length,this.options.ranges&&(this.yylloc.range=[this.offset,this.offset+=this.yyleng]),this._more=!1,this._backtrack=!1,this._input=this._input.slice(et[0].length),this.matched+=et[0],Kt=this.performAction.call(this,this.yy,this,mt,this.conditionStack[this.conditionStack.length-1]),this.done&&this._input&&(this.done=!1),Kt)return Kt;if(this._backtrack){for(var ye in Cn)this[ye]=Cn[ye];return!1}return!1},"test_match"),next:o(function(){if(this.done)return this.EOF;this._input||(this.done=!0);var et,mt,Kt,lt;this._more||(this.yytext="",this.match="");for(var Cn=this._currentRules(),ye=0;yemt[0].length)){if(mt=Kt,lt=ye,this.options.backtrack_lexer){if(et=this.test_match(Kt,Cn[ye]),et!==!1)return et;if(this._backtrack){mt=!1;continue}else return!1}else if(!this.options.flex)break}return mt?(et=this.test_match(mt,Cn[lt]),et!==!1?et:!1):this._input===""?this.EOF:this.parseError("Lexical error on line "+(this.yylineno+1)+`. Unrecognized text. +`+this.showPosition(),{text:"",token:null,line:this.yylineno})},"next"),lex:o(function(){var mt=this.next();return mt||this.lex()},"lex"),begin:o(function(mt){this.conditionStack.push(mt)},"begin"),popState:o(function(){var mt=this.conditionStack.length-1;return mt>0?this.conditionStack.pop():this.conditionStack[0]},"popState"),_currentRules:o(function(){return this.conditionStack.length&&this.conditionStack[this.conditionStack.length-1]?this.conditions[this.conditionStack[this.conditionStack.length-1]].rules:this.conditions.INITIAL.rules},"_currentRules"),topState:o(function(mt){return mt=this.conditionStack.length-1-Math.abs(mt||0),mt>=0?this.conditionStack[mt]:"INITIAL"},"topState"),pushState:o(function(mt){this.begin(mt)},"pushState"),stateStackSize:o(function(){return this.conditionStack.length},"stateStackSize"),options:{},performAction:o(function(mt,Kt,lt,Cn){var ye=Cn;switch(lt){case 0:return this.begin("acc_title"),34;break;case 1:return this.popState(),"acc_title_value";break;case 2:return this.begin("acc_descr"),36;break;case 3:return this.popState(),"acc_descr_value";break;case 4:this.begin("acc_descr_multiline");break;case 5:this.popState();break;case 6:return"acc_descr_multiline_value";case 7:return this.pushState("shapeData"),Kt.yytext="",40;break;case 8:return this.pushState("shapeDataStr"),40;break;case 9:return this.popState(),40;break;case 10:let Vf=/\n\s*/g;return Kt.yytext=Kt.yytext.replace(Vf,"
    "),40;break;case 11:return 40;case 12:this.popState();break;case 13:this.begin("callbackname");break;case 14:this.popState();break;case 15:this.popState(),this.begin("callbackargs");break;case 16:return 95;case 17:this.popState();break;case 18:return 96;case 19:return"MD_STR";case 20:this.popState();break;case 21:this.begin("md_string");break;case 22:return"STR";case 23:this.popState();break;case 24:this.pushState("string");break;case 25:return 84;case 26:return 102;case 27:return 85;case 28:return 104;case 29:return 86;case 30:return 87;case 31:return 97;case 32:this.begin("click");break;case 33:this.popState();break;case 34:return 88;case 35:return mt.lex.firstGraph()&&this.begin("dir"),12;break;case 36:return mt.lex.firstGraph()&&this.begin("dir"),12;break;case 37:return mt.lex.firstGraph()&&this.begin("dir"),12;break;case 38:return 27;case 39:return 32;case 40:return 98;case 41:return 98;case 42:return 98;case 43:return 98;case 44:return this.popState(),13;break;case 45:return this.popState(),14;break;case 46:return this.popState(),14;break;case 47:return this.popState(),14;break;case 48:return this.popState(),14;break;case 49:return this.popState(),14;break;case 50:return this.popState(),14;break;case 51:return this.popState(),14;break;case 52:return this.popState(),14;break;case 53:return this.popState(),14;break;case 54:return this.popState(),14;break;case 55:return 121;case 56:return 122;case 57:return 123;case 58:return 124;case 59:return 78;case 60:return 105;case 61:return 111;case 62:return 46;case 63:return 60;case 64:return 44;case 65:return 8;case 66:return 106;case 67:return 115;case 68:return this.popState(),77;break;case 69:return this.pushState("edgeText"),75;break;case 70:return 119;case 71:return this.popState(),77;break;case 72:return this.pushState("thickEdgeText"),75;break;case 73:return 119;case 74:return this.popState(),77;break;case 75:return this.pushState("dottedEdgeText"),75;break;case 76:return 119;case 77:return 77;case 78:return this.popState(),53;break;case 79:return"TEXT";case 80:return this.pushState("ellipseText"),52;break;case 81:return this.popState(),55;break;case 82:return this.pushState("text"),54;break;case 83:return this.popState(),57;break;case 84:return this.pushState("text"),56;break;case 85:return 58;case 86:return this.pushState("text"),67;break;case 87:return this.popState(),64;break;case 88:return this.pushState("text"),63;break;case 89:return this.popState(),49;break;case 90:return this.pushState("text"),48;break;case 91:return this.popState(),69;break;case 92:return this.popState(),71;break;case 93:return 117;case 94:return this.pushState("trapText"),68;break;case 95:return this.pushState("trapText"),70;break;case 96:return 118;case 97:return 67;case 98:return 90;case 99:return"SEP";case 100:return 89;case 101:return 115;case 102:return 111;case 103:return 44;case 104:return 109;case 105:return 114;case 106:return 116;case 107:return this.popState(),62;break;case 108:return this.pushState("text"),62;break;case 109:return this.popState(),51;break;case 110:return this.pushState("text"),50;break;case 111:return this.popState(),31;break;case 112:return this.pushState("text"),29;break;case 113:return this.popState(),66;break;case 114:return this.pushState("text"),65;break;case 115:return"TEXT";case 116:return"QUOTE";case 117:return 9;case 118:return 10;case 119:return 11}},"anonymous"),rules:[/^(?:accTitle\s*:\s*)/,/^(?:(?!\n||)*[^\n]*)/,/^(?:accDescr\s*:\s*)/,/^(?:(?!\n||)*[^\n]*)/,/^(?:accDescr\s*\{\s*)/,/^(?:[\}])/,/^(?:[^\}]*)/,/^(?:@\{)/,/^(?:["])/,/^(?:["])/,/^(?:[^\"]+)/,/^(?:[^}^"]+)/,/^(?:\})/,/^(?:call[\s]+)/,/^(?:\([\s]*\))/,/^(?:\()/,/^(?:[^(]*)/,/^(?:\))/,/^(?:[^)]*)/,/^(?:[^`"]+)/,/^(?:[`]["])/,/^(?:["][`])/,/^(?:[^"]+)/,/^(?:["])/,/^(?:["])/,/^(?:style\b)/,/^(?:default\b)/,/^(?:linkStyle\b)/,/^(?:interpolate\b)/,/^(?:classDef\b)/,/^(?:class\b)/,/^(?:href[\s])/,/^(?:click[\s]+)/,/^(?:[\s\n])/,/^(?:[^\s\n]*)/,/^(?:flowchart-elk\b)/,/^(?:graph\b)/,/^(?:flowchart\b)/,/^(?:subgraph\b)/,/^(?:end\b\s*)/,/^(?:_self\b)/,/^(?:_blank\b)/,/^(?:_parent\b)/,/^(?:_top\b)/,/^(?:(\r?\n)*\s*\n)/,/^(?:\s*LR\b)/,/^(?:\s*RL\b)/,/^(?:\s*TB\b)/,/^(?:\s*BT\b)/,/^(?:\s*TD\b)/,/^(?:\s*BR\b)/,/^(?:\s*<)/,/^(?:\s*>)/,/^(?:\s*\^)/,/^(?:\s*v\b)/,/^(?:.*direction\s+TB[^\n]*)/,/^(?:.*direction\s+BT[^\n]*)/,/^(?:.*direction\s+RL[^\n]*)/,/^(?:.*direction\s+LR[^\n]*)/,/^(?:[^\s\"]+@(?=[^\{\"]))/,/^(?:[0-9]+)/,/^(?:#)/,/^(?::::)/,/^(?::)/,/^(?:&)/,/^(?:;)/,/^(?:,)/,/^(?:\*)/,/^(?:\s*[xo<]?--+[-xo>]\s*)/,/^(?:\s*[xo<]?--\s*)/,/^(?:[^-]|-(?!-)+)/,/^(?:\s*[xo<]?==+[=xo>]\s*)/,/^(?:\s*[xo<]?==\s*)/,/^(?:[^=]|=(?!))/,/^(?:\s*[xo<]?-?\.+-[xo>]?\s*)/,/^(?:\s*[xo<]?-\.\s*)/,/^(?:[^\.]|\.(?!))/,/^(?:\s*~~[\~]+\s*)/,/^(?:[-/\)][\)])/,/^(?:[^\(\)\[\]\{\}]|!\)+)/,/^(?:\(-)/,/^(?:\]\))/,/^(?:\(\[)/,/^(?:\]\])/,/^(?:\[\[)/,/^(?:\[\|)/,/^(?:>)/,/^(?:\)\])/,/^(?:\[\()/,/^(?:\)\)\))/,/^(?:\(\(\()/,/^(?:[\\(?=\])][\]])/,/^(?:\/(?=\])\])/,/^(?:\/(?!\])|\\(?!\])|[^\\\[\]\(\)\{\}\/]+)/,/^(?:\[\/)/,/^(?:\[\\)/,/^(?:<)/,/^(?:>)/,/^(?:\^)/,/^(?:\\\|)/,/^(?:v\b)/,/^(?:\*)/,/^(?:#)/,/^(?:&)/,/^(?:([A-Za-z0-9!"\#$%&'*+\.`?\\_\/]|-(?=[^\>\-\.])|(?!))+)/,/^(?:-)/,/^(?:[\u00AA\u00B5\u00BA\u00C0-\u00D6\u00D8-\u00F6]|[\u00F8-\u02C1\u02C6-\u02D1\u02E0-\u02E4\u02EC\u02EE\u0370-\u0374\u0376\u0377]|[\u037A-\u037D\u0386\u0388-\u038A\u038C\u038E-\u03A1\u03A3-\u03F5]|[\u03F7-\u0481\u048A-\u0527\u0531-\u0556\u0559\u0561-\u0587\u05D0-\u05EA]|[\u05F0-\u05F2\u0620-\u064A\u066E\u066F\u0671-\u06D3\u06D5\u06E5\u06E6\u06EE]|[\u06EF\u06FA-\u06FC\u06FF\u0710\u0712-\u072F\u074D-\u07A5\u07B1\u07CA-\u07EA]|[\u07F4\u07F5\u07FA\u0800-\u0815\u081A\u0824\u0828\u0840-\u0858\u08A0]|[\u08A2-\u08AC\u0904-\u0939\u093D\u0950\u0958-\u0961\u0971-\u0977]|[\u0979-\u097F\u0985-\u098C\u098F\u0990\u0993-\u09A8\u09AA-\u09B0\u09B2]|[\u09B6-\u09B9\u09BD\u09CE\u09DC\u09DD\u09DF-\u09E1\u09F0\u09F1\u0A05-\u0A0A]|[\u0A0F\u0A10\u0A13-\u0A28\u0A2A-\u0A30\u0A32\u0A33\u0A35\u0A36\u0A38\u0A39]|[\u0A59-\u0A5C\u0A5E\u0A72-\u0A74\u0A85-\u0A8D\u0A8F-\u0A91\u0A93-\u0AA8]|[\u0AAA-\u0AB0\u0AB2\u0AB3\u0AB5-\u0AB9\u0ABD\u0AD0\u0AE0\u0AE1\u0B05-\u0B0C]|[\u0B0F\u0B10\u0B13-\u0B28\u0B2A-\u0B30\u0B32\u0B33\u0B35-\u0B39\u0B3D\u0B5C]|[\u0B5D\u0B5F-\u0B61\u0B71\u0B83\u0B85-\u0B8A\u0B8E-\u0B90\u0B92-\u0B95\u0B99]|[\u0B9A\u0B9C\u0B9E\u0B9F\u0BA3\u0BA4\u0BA8-\u0BAA\u0BAE-\u0BB9\u0BD0]|[\u0C05-\u0C0C\u0C0E-\u0C10\u0C12-\u0C28\u0C2A-\u0C33\u0C35-\u0C39\u0C3D]|[\u0C58\u0C59\u0C60\u0C61\u0C85-\u0C8C\u0C8E-\u0C90\u0C92-\u0CA8\u0CAA-\u0CB3]|[\u0CB5-\u0CB9\u0CBD\u0CDE\u0CE0\u0CE1\u0CF1\u0CF2\u0D05-\u0D0C\u0D0E-\u0D10]|[\u0D12-\u0D3A\u0D3D\u0D4E\u0D60\u0D61\u0D7A-\u0D7F\u0D85-\u0D96\u0D9A-\u0DB1]|[\u0DB3-\u0DBB\u0DBD\u0DC0-\u0DC6\u0E01-\u0E30\u0E32\u0E33\u0E40-\u0E46\u0E81]|[\u0E82\u0E84\u0E87\u0E88\u0E8A\u0E8D\u0E94-\u0E97\u0E99-\u0E9F\u0EA1-\u0EA3]|[\u0EA5\u0EA7\u0EAA\u0EAB\u0EAD-\u0EB0\u0EB2\u0EB3\u0EBD\u0EC0-\u0EC4\u0EC6]|[\u0EDC-\u0EDF\u0F00\u0F40-\u0F47\u0F49-\u0F6C\u0F88-\u0F8C\u1000-\u102A]|[\u103F\u1050-\u1055\u105A-\u105D\u1061\u1065\u1066\u106E-\u1070\u1075-\u1081]|[\u108E\u10A0-\u10C5\u10C7\u10CD\u10D0-\u10FA\u10FC-\u1248\u124A-\u124D]|[\u1250-\u1256\u1258\u125A-\u125D\u1260-\u1288\u128A-\u128D\u1290-\u12B0]|[\u12B2-\u12B5\u12B8-\u12BE\u12C0\u12C2-\u12C5\u12C8-\u12D6\u12D8-\u1310]|[\u1312-\u1315\u1318-\u135A\u1380-\u138F\u13A0-\u13F4\u1401-\u166C]|[\u166F-\u167F\u1681-\u169A\u16A0-\u16EA\u1700-\u170C\u170E-\u1711]|[\u1720-\u1731\u1740-\u1751\u1760-\u176C\u176E-\u1770\u1780-\u17B3\u17D7]|[\u17DC\u1820-\u1877\u1880-\u18A8\u18AA\u18B0-\u18F5\u1900-\u191C]|[\u1950-\u196D\u1970-\u1974\u1980-\u19AB\u19C1-\u19C7\u1A00-\u1A16]|[\u1A20-\u1A54\u1AA7\u1B05-\u1B33\u1B45-\u1B4B\u1B83-\u1BA0\u1BAE\u1BAF]|[\u1BBA-\u1BE5\u1C00-\u1C23\u1C4D-\u1C4F\u1C5A-\u1C7D\u1CE9-\u1CEC]|[\u1CEE-\u1CF1\u1CF5\u1CF6\u1D00-\u1DBF\u1E00-\u1F15\u1F18-\u1F1D]|[\u1F20-\u1F45\u1F48-\u1F4D\u1F50-\u1F57\u1F59\u1F5B\u1F5D\u1F5F-\u1F7D]|[\u1F80-\u1FB4\u1FB6-\u1FBC\u1FBE\u1FC2-\u1FC4\u1FC6-\u1FCC\u1FD0-\u1FD3]|[\u1FD6-\u1FDB\u1FE0-\u1FEC\u1FF2-\u1FF4\u1FF6-\u1FFC\u2071\u207F]|[\u2090-\u209C\u2102\u2107\u210A-\u2113\u2115\u2119-\u211D\u2124\u2126\u2128]|[\u212A-\u212D\u212F-\u2139\u213C-\u213F\u2145-\u2149\u214E\u2183\u2184]|[\u2C00-\u2C2E\u2C30-\u2C5E\u2C60-\u2CE4\u2CEB-\u2CEE\u2CF2\u2CF3]|[\u2D00-\u2D25\u2D27\u2D2D\u2D30-\u2D67\u2D6F\u2D80-\u2D96\u2DA0-\u2DA6]|[\u2DA8-\u2DAE\u2DB0-\u2DB6\u2DB8-\u2DBE\u2DC0-\u2DC6\u2DC8-\u2DCE]|[\u2DD0-\u2DD6\u2DD8-\u2DDE\u2E2F\u3005\u3006\u3031-\u3035\u303B\u303C]|[\u3041-\u3096\u309D-\u309F\u30A1-\u30FA\u30FC-\u30FF\u3105-\u312D]|[\u3131-\u318E\u31A0-\u31BA\u31F0-\u31FF\u3400-\u4DB5\u4E00-\u9FCC]|[\uA000-\uA48C\uA4D0-\uA4FD\uA500-\uA60C\uA610-\uA61F\uA62A\uA62B]|[\uA640-\uA66E\uA67F-\uA697\uA6A0-\uA6E5\uA717-\uA71F\uA722-\uA788]|[\uA78B-\uA78E\uA790-\uA793\uA7A0-\uA7AA\uA7F8-\uA801\uA803-\uA805]|[\uA807-\uA80A\uA80C-\uA822\uA840-\uA873\uA882-\uA8B3\uA8F2-\uA8F7\uA8FB]|[\uA90A-\uA925\uA930-\uA946\uA960-\uA97C\uA984-\uA9B2\uA9CF\uAA00-\uAA28]|[\uAA40-\uAA42\uAA44-\uAA4B\uAA60-\uAA76\uAA7A\uAA80-\uAAAF\uAAB1\uAAB5]|[\uAAB6\uAAB9-\uAABD\uAAC0\uAAC2\uAADB-\uAADD\uAAE0-\uAAEA\uAAF2-\uAAF4]|[\uAB01-\uAB06\uAB09-\uAB0E\uAB11-\uAB16\uAB20-\uAB26\uAB28-\uAB2E]|[\uABC0-\uABE2\uAC00-\uD7A3\uD7B0-\uD7C6\uD7CB-\uD7FB\uF900-\uFA6D]|[\uFA70-\uFAD9\uFB00-\uFB06\uFB13-\uFB17\uFB1D\uFB1F-\uFB28\uFB2A-\uFB36]|[\uFB38-\uFB3C\uFB3E\uFB40\uFB41\uFB43\uFB44\uFB46-\uFBB1\uFBD3-\uFD3D]|[\uFD50-\uFD8F\uFD92-\uFDC7\uFDF0-\uFDFB\uFE70-\uFE74\uFE76-\uFEFC]|[\uFF21-\uFF3A\uFF41-\uFF5A\uFF66-\uFFBE\uFFC2-\uFFC7\uFFCA-\uFFCF]|[\uFFD2-\uFFD7\uFFDA-\uFFDC])/,/^(?:\|)/,/^(?:\|)/,/^(?:\))/,/^(?:\()/,/^(?:\])/,/^(?:\[)/,/^(?:(\}))/,/^(?:\{)/,/^(?:[^\[\]\(\)\{\}\|\"]+)/,/^(?:")/,/^(?:(\r?\n)+)/,/^(?:\s)/,/^(?:$)/],conditions:{shapeDataEndBracket:{rules:[21,24,77,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},shapeDataStr:{rules:[9,10,21,24,77,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},shapeData:{rules:[8,11,12,21,24,77,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},callbackargs:{rules:[17,18,21,24,77,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},callbackname:{rules:[14,15,16,21,24,77,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},href:{rules:[21,24,77,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},click:{rules:[21,24,33,34,77,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},dottedEdgeText:{rules:[21,24,74,76,77,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},thickEdgeText:{rules:[21,24,71,73,77,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},edgeText:{rules:[21,24,68,70,77,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},trapText:{rules:[21,24,77,80,82,84,88,90,91,92,93,94,95,108,110,112,114],inclusive:!1},ellipseText:{rules:[21,24,77,78,79,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},text:{rules:[21,24,77,80,81,82,83,84,87,88,89,90,94,95,107,108,109,110,111,112,113,114,115],inclusive:!1},vertex:{rules:[21,24,77,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},dir:{rules:[21,24,44,45,46,47,48,49,50,51,52,53,54,77,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},acc_descr_multiline:{rules:[5,6,21,24,77,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},acc_descr:{rules:[3,21,24,77,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},acc_title:{rules:[1,21,24,77,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},md_string:{rules:[19,20,21,24,77,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},string:{rules:[21,22,23,24,77,80,82,84,88,90,94,95,108,110,112,114],inclusive:!1},INITIAL:{rules:[0,2,4,7,13,21,24,25,26,27,28,29,30,31,32,35,36,37,38,39,40,41,42,43,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,71,72,74,75,77,80,82,84,85,86,88,90,94,95,96,97,98,99,100,101,102,103,104,105,106,108,110,112,114,116,117,118,119],inclusive:!0}}};return Hr}();Ni.lexer=Zn;function Sn(){this.yy={}}return o(Sn,"Parser"),Sn.prototype=Ni,Ni.Parser=Sn,new Sn}();xR.parser=xR;bR=xR});var Uie,Hie,Wie=N(()=>{"use strict";Vie();Uie=Object.assign({},bR);Uie.parse=t=>{let e=t.replace(/}\s*\n/g,`} +`);return bR.parse(e)};Hie=Uie});var gOe,yOe,qie,Yie=N(()=>{"use strict";Ys();gOe=o((t,e)=>{let r=Kf,n=r(t,"r"),i=r(t,"g"),a=r(t,"b");return qa(n,i,a,e)},"fade"),yOe=o(t=>`.label { + font-family: ${t.fontFamily}; + color: ${t.nodeTextColor||t.textColor}; + } + .cluster-label text { + fill: ${t.titleColor}; + } + .cluster-label span { + color: ${t.titleColor}; + } + .cluster-label span p { + background-color: transparent; + } + + .label text,span { + fill: ${t.nodeTextColor||t.textColor}; + color: ${t.nodeTextColor||t.textColor}; + } + + .node rect, + .node circle, + .node ellipse, + .node polygon, + .node path { + fill: ${t.mainBkg}; + stroke: ${t.nodeBorder}; + stroke-width: 1px; + } + .rough-node .label text , .node .label text, .image-shape .label, .icon-shape .label { + text-anchor: middle; + } + // .flowchart-label .text-outer-tspan { + // text-anchor: middle; + // } + // .flowchart-label .text-inner-tspan { + // text-anchor: start; + // } + + .node .katex path { + fill: #000; + stroke: #000; + stroke-width: 1px; + } + + .rough-node .label,.node .label, .image-shape .label, .icon-shape .label { + text-align: center; + } + .node.clickable { + cursor: pointer; + } + + + .root .anchor path { + fill: ${t.lineColor} !important; + stroke-width: 0; + stroke: ${t.lineColor}; + } + + .arrowheadPath { + fill: ${t.arrowheadColor}; + } + + .edgePath .path { + stroke: ${t.lineColor}; + stroke-width: 2.0px; + } + + .flowchart-link { + stroke: ${t.lineColor}; + fill: none; + } + + .edgeLabel { + background-color: ${t.edgeLabelBackground}; + p { + background-color: ${t.edgeLabelBackground}; + } + rect { + opacity: 0.5; + background-color: ${t.edgeLabelBackground}; + fill: ${t.edgeLabelBackground}; + } + text-align: center; + } + + /* For html labels only */ + .labelBkg { + background-color: ${gOe(t.edgeLabelBackground,.5)}; + // background-color: + } + + .cluster rect { + fill: ${t.clusterBkg}; + stroke: ${t.clusterBorder}; + stroke-width: 1px; + } + + .cluster text { + fill: ${t.titleColor}; + } + + .cluster span { + color: ${t.titleColor}; + } + /* .cluster div { + color: ${t.titleColor}; + } */ + + div.mermaidTooltip { + position: absolute; + text-align: center; + max-width: 200px; + padding: 2px; + font-family: ${t.fontFamily}; + font-size: 12px; + background: ${t.tertiaryColor}; + border: 1px solid ${t.border2}; + border-radius: 2px; + pointer-events: none; + z-index: 100; + } + + .flowchartTitleText { + text-anchor: middle; + font-size: 18px; + fill: ${t.textColor}; + } + + rect.text { + fill: none; + stroke-width: 0; + } + + .icon-shape, .image-shape { + background-color: ${t.edgeLabelBackground}; + p { + background-color: ${t.edgeLabelBackground}; + padding: 2px; + } + rect { + opacity: 0.5; + background-color: ${t.edgeLabelBackground}; + fill: ${t.edgeLabelBackground}; + } + text-align: center; + } +`,"getStyles"),qie=yOe});var ik={};hr(ik,{diagram:()=>vOe});var vOe,ak=N(()=>{"use strict";zt();qZ();Gie();Wie();Yie();vOe={parser:Hie,get db(){return new Uw},renderer:zie,styles:qie,init:o(t=>{t.flowchart||(t.flowchart={}),t.layout&&Yy({layout:t.layout}),t.flowchart.arrowMarkerAbsolute=t.arrowMarkerAbsolute,Yy({flowchart:{arrowMarkerAbsolute:t.arrowMarkerAbsolute}})},"init")}});var wR,Zie,Jie=N(()=>{"use strict";wR=function(){var t=o(function(J,se,ue,Z){for(ue=ue||{},Z=J.length;Z--;ue[J[Z]]=se);return ue},"o"),e=[6,8,10,22,24,26,28,33,34,35,36,37,40,43,44,50],r=[1,10],n=[1,11],i=[1,12],a=[1,13],s=[1,20],l=[1,21],u=[1,22],h=[1,23],f=[1,24],d=[1,19],p=[1,25],m=[1,26],g=[1,18],y=[1,33],v=[1,34],x=[1,35],b=[1,36],w=[1,37],C=[6,8,10,13,15,17,20,21,22,24,26,28,33,34,35,36,37,40,43,44,50,63,64,65,66,67],T=[1,42],E=[1,43],A=[1,52],S=[40,50,68,69],_=[1,63],I=[1,61],D=[1,58],k=[1,62],L=[1,64],R=[6,8,10,13,17,22,24,26,28,33,34,35,36,37,40,41,42,43,44,48,49,50,63,64,65,66,67],O=[63,64,65,66,67],M=[1,81],B=[1,80],F=[1,78],P=[1,79],z=[6,10,42,47],$=[6,10,13,41,42,47,48,49],H=[1,89],Q=[1,88],j=[1,87],ie=[19,56],ne=[1,98],le=[1,97],he=[19,56,58,60],K={trace:o(function(){},"trace"),yy:{},symbols_:{error:2,start:3,ER_DIAGRAM:4,document:5,EOF:6,line:7,SPACE:8,statement:9,NEWLINE:10,entityName:11,relSpec:12,COLON:13,role:14,STYLE_SEPARATOR:15,idList:16,BLOCK_START:17,attributes:18,BLOCK_STOP:19,SQS:20,SQE:21,title:22,title_value:23,acc_title:24,acc_title_value:25,acc_descr:26,acc_descr_value:27,acc_descr_multiline_value:28,direction:29,classDefStatement:30,classStatement:31,styleStatement:32,direction_tb:33,direction_bt:34,direction_rl:35,direction_lr:36,CLASSDEF:37,stylesOpt:38,separator:39,UNICODE_TEXT:40,STYLE_TEXT:41,COMMA:42,CLASS:43,STYLE:44,style:45,styleComponent:46,SEMI:47,NUM:48,BRKT:49,ENTITY_NAME:50,attribute:51,attributeType:52,attributeName:53,attributeKeyTypeList:54,attributeComment:55,ATTRIBUTE_WORD:56,attributeKeyType:57,",":58,ATTRIBUTE_KEY:59,COMMENT:60,cardinality:61,relType:62,ZERO_OR_ONE:63,ZERO_OR_MORE:64,ONE_OR_MORE:65,ONLY_ONE:66,MD_PARENT:67,NON_IDENTIFYING:68,IDENTIFYING:69,WORD:70,$accept:0,$end:1},terminals_:{2:"error",4:"ER_DIAGRAM",6:"EOF",8:"SPACE",10:"NEWLINE",13:"COLON",15:"STYLE_SEPARATOR",17:"BLOCK_START",19:"BLOCK_STOP",20:"SQS",21:"SQE",22:"title",23:"title_value",24:"acc_title",25:"acc_title_value",26:"acc_descr",27:"acc_descr_value",28:"acc_descr_multiline_value",33:"direction_tb",34:"direction_bt",35:"direction_rl",36:"direction_lr",37:"CLASSDEF",40:"UNICODE_TEXT",41:"STYLE_TEXT",42:"COMMA",43:"CLASS",44:"STYLE",47:"SEMI",48:"NUM",49:"BRKT",50:"ENTITY_NAME",56:"ATTRIBUTE_WORD",58:",",59:"ATTRIBUTE_KEY",60:"COMMENT",63:"ZERO_OR_ONE",64:"ZERO_OR_MORE",65:"ONE_OR_MORE",66:"ONLY_ONE",67:"MD_PARENT",68:"NON_IDENTIFYING",69:"IDENTIFYING",70:"WORD"},productions_:[0,[3,3],[5,0],[5,2],[7,2],[7,1],[7,1],[7,1],[9,5],[9,9],[9,7],[9,7],[9,4],[9,6],[9,3],[9,5],[9,1],[9,3],[9,7],[9,9],[9,6],[9,8],[9,4],[9,6],[9,2],[9,2],[9,2],[9,1],[9,1],[9,1],[9,1],[9,1],[29,1],[29,1],[29,1],[29,1],[30,4],[16,1],[16,1],[16,3],[16,3],[31,3],[32,4],[38,1],[38,3],[45,1],[45,2],[39,1],[39,1],[39,1],[46,1],[46,1],[46,1],[46,1],[11,1],[11,1],[18,1],[18,2],[51,2],[51,3],[51,3],[51,4],[52,1],[53,1],[54,1],[54,3],[57,1],[55,1],[12,3],[61,1],[61,1],[61,1],[61,1],[61,1],[62,1],[62,1],[14,1],[14,1],[14,1]],performAction:o(function(se,ue,Z,Se,ce,ae,Oe){var ge=ae.length-1;switch(ce){case 1:break;case 2:this.$=[];break;case 3:ae[ge-1].push(ae[ge]),this.$=ae[ge-1];break;case 4:case 5:this.$=ae[ge];break;case 6:case 7:this.$=[];break;case 8:Se.addEntity(ae[ge-4]),Se.addEntity(ae[ge-2]),Se.addRelationship(ae[ge-4],ae[ge],ae[ge-2],ae[ge-3]);break;case 9:Se.addEntity(ae[ge-8]),Se.addEntity(ae[ge-4]),Se.addRelationship(ae[ge-8],ae[ge],ae[ge-4],ae[ge-5]),Se.setClass([ae[ge-8]],ae[ge-6]),Se.setClass([ae[ge-4]],ae[ge-2]);break;case 10:Se.addEntity(ae[ge-6]),Se.addEntity(ae[ge-2]),Se.addRelationship(ae[ge-6],ae[ge],ae[ge-2],ae[ge-3]),Se.setClass([ae[ge-6]],ae[ge-4]);break;case 11:Se.addEntity(ae[ge-6]),Se.addEntity(ae[ge-4]),Se.addRelationship(ae[ge-6],ae[ge],ae[ge-4],ae[ge-5]),Se.setClass([ae[ge-4]],ae[ge-2]);break;case 12:Se.addEntity(ae[ge-3]),Se.addAttributes(ae[ge-3],ae[ge-1]);break;case 13:Se.addEntity(ae[ge-5]),Se.addAttributes(ae[ge-5],ae[ge-1]),Se.setClass([ae[ge-5]],ae[ge-3]);break;case 14:Se.addEntity(ae[ge-2]);break;case 15:Se.addEntity(ae[ge-4]),Se.setClass([ae[ge-4]],ae[ge-2]);break;case 16:Se.addEntity(ae[ge]);break;case 17:Se.addEntity(ae[ge-2]),Se.setClass([ae[ge-2]],ae[ge]);break;case 18:Se.addEntity(ae[ge-6],ae[ge-4]),Se.addAttributes(ae[ge-6],ae[ge-1]);break;case 19:Se.addEntity(ae[ge-8],ae[ge-6]),Se.addAttributes(ae[ge-8],ae[ge-1]),Se.setClass([ae[ge-8]],ae[ge-3]);break;case 20:Se.addEntity(ae[ge-5],ae[ge-3]);break;case 21:Se.addEntity(ae[ge-7],ae[ge-5]),Se.setClass([ae[ge-7]],ae[ge-2]);break;case 22:Se.addEntity(ae[ge-3],ae[ge-1]);break;case 23:Se.addEntity(ae[ge-5],ae[ge-3]),Se.setClass([ae[ge-5]],ae[ge]);break;case 24:case 25:this.$=ae[ge].trim(),Se.setAccTitle(this.$);break;case 26:case 27:this.$=ae[ge].trim(),Se.setAccDescription(this.$);break;case 32:Se.setDirection("TB");break;case 33:Se.setDirection("BT");break;case 34:Se.setDirection("RL");break;case 35:Se.setDirection("LR");break;case 36:this.$=ae[ge-3],Se.addClass(ae[ge-2],ae[ge-1]);break;case 37:case 38:case 56:case 64:this.$=[ae[ge]];break;case 39:case 40:this.$=ae[ge-2].concat([ae[ge]]);break;case 41:this.$=ae[ge-2],Se.setClass(ae[ge-1],ae[ge]);break;case 42:this.$=ae[ge-3],Se.addCssStyles(ae[ge-2],ae[ge-1]);break;case 43:this.$=[ae[ge]];break;case 44:ae[ge-2].push(ae[ge]),this.$=ae[ge-2];break;case 46:this.$=ae[ge-1]+ae[ge];break;case 54:case 76:case 77:this.$=ae[ge].replace(/"/g,"");break;case 55:case 78:this.$=ae[ge];break;case 57:ae[ge].push(ae[ge-1]),this.$=ae[ge];break;case 58:this.$={type:ae[ge-1],name:ae[ge]};break;case 59:this.$={type:ae[ge-2],name:ae[ge-1],keys:ae[ge]};break;case 60:this.$={type:ae[ge-2],name:ae[ge-1],comment:ae[ge]};break;case 61:this.$={type:ae[ge-3],name:ae[ge-2],keys:ae[ge-1],comment:ae[ge]};break;case 62:case 63:case 66:this.$=ae[ge];break;case 65:ae[ge-2].push(ae[ge]),this.$=ae[ge-2];break;case 67:this.$=ae[ge].replace(/"/g,"");break;case 68:this.$={cardA:ae[ge],relType:ae[ge-1],cardB:ae[ge-2]};break;case 69:this.$=Se.Cardinality.ZERO_OR_ONE;break;case 70:this.$=Se.Cardinality.ZERO_OR_MORE;break;case 71:this.$=Se.Cardinality.ONE_OR_MORE;break;case 72:this.$=Se.Cardinality.ONLY_ONE;break;case 73:this.$=Se.Cardinality.MD_PARENT;break;case 74:this.$=Se.Identification.NON_IDENTIFYING;break;case 75:this.$=Se.Identification.IDENTIFYING;break}},"anonymous"),table:[{3:1,4:[1,2]},{1:[3]},t(e,[2,2],{5:3}),{6:[1,4],7:5,8:[1,6],9:7,10:[1,8],11:9,22:r,24:n,26:i,28:a,29:14,30:15,31:16,32:17,33:s,34:l,35:u,36:h,37:f,40:d,43:p,44:m,50:g},t(e,[2,7],{1:[2,1]}),t(e,[2,3]),{9:27,11:9,22:r,24:n,26:i,28:a,29:14,30:15,31:16,32:17,33:s,34:l,35:u,36:h,37:f,40:d,43:p,44:m,50:g},t(e,[2,5]),t(e,[2,6]),t(e,[2,16],{12:28,61:32,15:[1,29],17:[1,30],20:[1,31],63:y,64:v,65:x,66:b,67:w}),{23:[1,38]},{25:[1,39]},{27:[1,40]},t(e,[2,27]),t(e,[2,28]),t(e,[2,29]),t(e,[2,30]),t(e,[2,31]),t(C,[2,54]),t(C,[2,55]),t(e,[2,32]),t(e,[2,33]),t(e,[2,34]),t(e,[2,35]),{16:41,40:T,41:E},{16:44,40:T,41:E},{16:45,40:T,41:E},t(e,[2,4]),{11:46,40:d,50:g},{16:47,40:T,41:E},{18:48,19:[1,49],51:50,52:51,56:A},{11:53,40:d,50:g},{62:54,68:[1,55],69:[1,56]},t(S,[2,69]),t(S,[2,70]),t(S,[2,71]),t(S,[2,72]),t(S,[2,73]),t(e,[2,24]),t(e,[2,25]),t(e,[2,26]),{13:_,38:57,41:I,42:D,45:59,46:60,48:k,49:L},t(R,[2,37]),t(R,[2,38]),{16:65,40:T,41:E,42:D},{13:_,38:66,41:I,42:D,45:59,46:60,48:k,49:L},{13:[1,67],15:[1,68]},t(e,[2,17],{61:32,12:69,17:[1,70],42:D,63:y,64:v,65:x,66:b,67:w}),{19:[1,71]},t(e,[2,14]),{18:72,19:[2,56],51:50,52:51,56:A},{53:73,56:[1,74]},{56:[2,62]},{21:[1,75]},{61:76,63:y,64:v,65:x,66:b,67:w},t(O,[2,74]),t(O,[2,75]),{6:M,10:B,39:77,42:F,47:P},{40:[1,82],41:[1,83]},t(z,[2,43],{46:84,13:_,41:I,48:k,49:L}),t($,[2,45]),t($,[2,50]),t($,[2,51]),t($,[2,52]),t($,[2,53]),t(e,[2,41],{42:D}),{6:M,10:B,39:85,42:F,47:P},{14:86,40:H,50:Q,70:j},{16:90,40:T,41:E},{11:91,40:d,50:g},{18:92,19:[1,93],51:50,52:51,56:A},t(e,[2,12]),{19:[2,57]},t(ie,[2,58],{54:94,55:95,57:96,59:ne,60:le}),t([19,56,59,60],[2,63]),t(e,[2,22],{15:[1,100],17:[1,99]}),t([40,50],[2,68]),t(e,[2,36]),{13:_,41:I,45:101,46:60,48:k,49:L},t(e,[2,47]),t(e,[2,48]),t(e,[2,49]),t(R,[2,39]),t(R,[2,40]),t($,[2,46]),t(e,[2,42]),t(e,[2,8]),t(e,[2,76]),t(e,[2,77]),t(e,[2,78]),{13:[1,102],42:D},{13:[1,104],15:[1,103]},{19:[1,105]},t(e,[2,15]),t(ie,[2,59],{55:106,58:[1,107],60:le}),t(ie,[2,60]),t(he,[2,64]),t(ie,[2,67]),t(he,[2,66]),{18:108,19:[1,109],51:50,52:51,56:A},{16:110,40:T,41:E},t(z,[2,44],{46:84,13:_,41:I,48:k,49:L}),{14:111,40:H,50:Q,70:j},{16:112,40:T,41:E},{14:113,40:H,50:Q,70:j},t(e,[2,13]),t(ie,[2,61]),{57:114,59:ne},{19:[1,115]},t(e,[2,20]),t(e,[2,23],{17:[1,116],42:D}),t(e,[2,11]),{13:[1,117],42:D},t(e,[2,10]),t(he,[2,65]),t(e,[2,18]),{18:118,19:[1,119],51:50,52:51,56:A},{14:120,40:H,50:Q,70:j},{19:[1,121]},t(e,[2,21]),t(e,[2,9]),t(e,[2,19])],defaultActions:{52:[2,62],72:[2,57]},parseError:o(function(se,ue){if(ue.recoverable)this.trace(se);else{var Z=new Error(se);throw Z.hash=ue,Z}},"parseError"),parse:o(function(se){var ue=this,Z=[0],Se=[],ce=[null],ae=[],Oe=this.table,ge="",ze=0,He=0,$e=0,Re=2,Ie=1,be=ae.slice.call(arguments,1),W=Object.create(this.lexer),de={yy:{}};for(var re in this.yy)Object.prototype.hasOwnProperty.call(this.yy,re)&&(de.yy[re]=this.yy[re]);W.setInput(se,de.yy),de.yy.lexer=W,de.yy.parser=this,typeof W.yylloc>"u"&&(W.yylloc={});var oe=W.yylloc;ae.push(oe);var V=W.options&&W.options.ranges;typeof de.yy.parseError=="function"?this.parseError=de.yy.parseError:this.parseError=Object.getPrototypeOf(this).parseError;function xe(ct){Z.length=Z.length-2*ct,ce.length=ce.length-ct,ae.length=ae.length-ct}o(xe,"popStack");function q(){var ct;return ct=Se.pop()||W.lex()||Ie,typeof ct!="number"&&(ct instanceof Array&&(Se=ct,ct=Se.pop()),ct=ue.symbols_[ct]||ct),ct}o(q,"lex");for(var pe,ve,Pe,_e,we,Ve,De={},qe,at,Rt,st;;){if(Pe=Z[Z.length-1],this.defaultActions[Pe]?_e=this.defaultActions[Pe]:((pe===null||typeof pe>"u")&&(pe=q()),_e=Oe[Pe]&&Oe[Pe][pe]),typeof _e>"u"||!_e.length||!_e[0]){var Ue="";st=[];for(qe in Oe[Pe])this.terminals_[qe]&&qe>Re&&st.push("'"+this.terminals_[qe]+"'");W.showPosition?Ue="Parse error on line "+(ze+1)+`: +`+W.showPosition()+` +Expecting `+st.join(", ")+", got '"+(this.terminals_[pe]||pe)+"'":Ue="Parse error on line "+(ze+1)+": Unexpected "+(pe==Ie?"end of input":"'"+(this.terminals_[pe]||pe)+"'"),this.parseError(Ue,{text:W.match,token:this.terminals_[pe]||pe,line:W.yylineno,loc:oe,expected:st})}if(_e[0]instanceof Array&&_e.length>1)throw new Error("Parse Error: multiple actions possible at state: "+Pe+", token: "+pe);switch(_e[0]){case 1:Z.push(pe),ce.push(W.yytext),ae.push(W.yylloc),Z.push(_e[1]),pe=null,ve?(pe=ve,ve=null):(He=W.yyleng,ge=W.yytext,ze=W.yylineno,oe=W.yylloc,$e>0&&$e--);break;case 2:if(at=this.productions_[_e[1]][1],De.$=ce[ce.length-at],De._$={first_line:ae[ae.length-(at||1)].first_line,last_line:ae[ae.length-1].last_line,first_column:ae[ae.length-(at||1)].first_column,last_column:ae[ae.length-1].last_column},V&&(De._$.range=[ae[ae.length-(at||1)].range[0],ae[ae.length-1].range[1]]),Ve=this.performAction.apply(De,[ge,He,ze,de.yy,_e[1],ce,ae].concat(be)),typeof Ve<"u")return Ve;at&&(Z=Z.slice(0,-1*at*2),ce=ce.slice(0,-1*at),ae=ae.slice(0,-1*at)),Z.push(this.productions_[_e[1]][0]),ce.push(De.$),ae.push(De._$),Rt=Oe[Z[Z.length-2]][Z[Z.length-1]],Z.push(Rt);break;case 3:return!0}}return!0},"parse")},X=function(){var J={EOF:1,parseError:o(function(ue,Z){if(this.yy.parser)this.yy.parser.parseError(ue,Z);else throw new Error(ue)},"parseError"),setInput:o(function(se,ue){return this.yy=ue||this.yy||{},this._input=se,this._more=this._backtrack=this.done=!1,this.yylineno=this.yyleng=0,this.yytext=this.matched=this.match="",this.conditionStack=["INITIAL"],this.yylloc={first_line:1,first_column:0,last_line:1,last_column:0},this.options.ranges&&(this.yylloc.range=[0,0]),this.offset=0,this},"setInput"),input:o(function(){var se=this._input[0];this.yytext+=se,this.yyleng++,this.offset++,this.match+=se,this.matched+=se;var ue=se.match(/(?:\r\n?|\n).*/g);return ue?(this.yylineno++,this.yylloc.last_line++):this.yylloc.last_column++,this.options.ranges&&this.yylloc.range[1]++,this._input=this._input.slice(1),se},"input"),unput:o(function(se){var ue=se.length,Z=se.split(/(?:\r\n?|\n)/g);this._input=se+this._input,this.yytext=this.yytext.substr(0,this.yytext.length-ue),this.offset-=ue;var Se=this.match.split(/(?:\r\n?|\n)/g);this.match=this.match.substr(0,this.match.length-1),this.matched=this.matched.substr(0,this.matched.length-1),Z.length-1&&(this.yylineno-=Z.length-1);var ce=this.yylloc.range;return this.yylloc={first_line:this.yylloc.first_line,last_line:this.yylineno+1,first_column:this.yylloc.first_column,last_column:Z?(Z.length===Se.length?this.yylloc.first_column:0)+Se[Se.length-Z.length].length-Z[0].length:this.yylloc.first_column-ue},this.options.ranges&&(this.yylloc.range=[ce[0],ce[0]+this.yyleng-ue]),this.yyleng=this.yytext.length,this},"unput"),more:o(function(){return this._more=!0,this},"more"),reject:o(function(){if(this.options.backtrack_lexer)this._backtrack=!0;else return this.parseError("Lexical error on line "+(this.yylineno+1)+`. You can only invoke reject() in the lexer when the lexer is of the backtracking persuasion (options.backtrack_lexer = true). +`+this.showPosition(),{text:"",token:null,line:this.yylineno});return this},"reject"),less:o(function(se){this.unput(this.match.slice(se))},"less"),pastInput:o(function(){var se=this.matched.substr(0,this.matched.length-this.match.length);return(se.length>20?"...":"")+se.substr(-20).replace(/\n/g,"")},"pastInput"),upcomingInput:o(function(){var se=this.match;return se.length<20&&(se+=this._input.substr(0,20-se.length)),(se.substr(0,20)+(se.length>20?"...":"")).replace(/\n/g,"")},"upcomingInput"),showPosition:o(function(){var se=this.pastInput(),ue=new Array(se.length+1).join("-");return se+this.upcomingInput()+` +`+ue+"^"},"showPosition"),test_match:o(function(se,ue){var Z,Se,ce;if(this.options.backtrack_lexer&&(ce={yylineno:this.yylineno,yylloc:{first_line:this.yylloc.first_line,last_line:this.last_line,first_column:this.yylloc.first_column,last_column:this.yylloc.last_column},yytext:this.yytext,match:this.match,matches:this.matches,matched:this.matched,yyleng:this.yyleng,offset:this.offset,_more:this._more,_input:this._input,yy:this.yy,conditionStack:this.conditionStack.slice(0),done:this.done},this.options.ranges&&(ce.yylloc.range=this.yylloc.range.slice(0))),Se=se[0].match(/(?:\r\n?|\n).*/g),Se&&(this.yylineno+=Se.length),this.yylloc={first_line:this.yylloc.last_line,last_line:this.yylineno+1,first_column:this.yylloc.last_column,last_column:Se?Se[Se.length-1].length-Se[Se.length-1].match(/\r?\n?/)[0].length:this.yylloc.last_column+se[0].length},this.yytext+=se[0],this.match+=se[0],this.matches=se,this.yyleng=this.yytext.length,this.options.ranges&&(this.yylloc.range=[this.offset,this.offset+=this.yyleng]),this._more=!1,this._backtrack=!1,this._input=this._input.slice(se[0].length),this.matched+=se[0],Z=this.performAction.call(this,this.yy,this,ue,this.conditionStack[this.conditionStack.length-1]),this.done&&this._input&&(this.done=!1),Z)return Z;if(this._backtrack){for(var ae in ce)this[ae]=ce[ae];return!1}return!1},"test_match"),next:o(function(){if(this.done)return this.EOF;this._input||(this.done=!0);var se,ue,Z,Se;this._more||(this.yytext="",this.match="");for(var ce=this._currentRules(),ae=0;aeue[0].length)){if(ue=Z,Se=ae,this.options.backtrack_lexer){if(se=this.test_match(Z,ce[ae]),se!==!1)return se;if(this._backtrack){ue=!1;continue}else return!1}else if(!this.options.flex)break}return ue?(se=this.test_match(ue,ce[Se]),se!==!1?se:!1):this._input===""?this.EOF:this.parseError("Lexical error on line "+(this.yylineno+1)+`. Unrecognized text. +`+this.showPosition(),{text:"",token:null,line:this.yylineno})},"next"),lex:o(function(){var ue=this.next();return ue||this.lex()},"lex"),begin:o(function(ue){this.conditionStack.push(ue)},"begin"),popState:o(function(){var ue=this.conditionStack.length-1;return ue>0?this.conditionStack.pop():this.conditionStack[0]},"popState"),_currentRules:o(function(){return this.conditionStack.length&&this.conditionStack[this.conditionStack.length-1]?this.conditions[this.conditionStack[this.conditionStack.length-1]].rules:this.conditions.INITIAL.rules},"_currentRules"),topState:o(function(ue){return ue=this.conditionStack.length-1-Math.abs(ue||0),ue>=0?this.conditionStack[ue]:"INITIAL"},"topState"),pushState:o(function(ue){this.begin(ue)},"pushState"),stateStackSize:o(function(){return this.conditionStack.length},"stateStackSize"),options:{"case-insensitive":!0},performAction:o(function(ue,Z,Se,ce){var ae=ce;switch(Se){case 0:return this.begin("acc_title"),24;break;case 1:return this.popState(),"acc_title_value";break;case 2:return this.begin("acc_descr"),26;break;case 3:return this.popState(),"acc_descr_value";break;case 4:this.begin("acc_descr_multiline");break;case 5:this.popState();break;case 6:return"acc_descr_multiline_value";case 7:return 33;case 8:return 34;case 9:return 35;case 10:return 36;case 11:return 10;case 12:break;case 13:return 8;case 14:return 50;case 15:return 70;case 16:return 4;case 17:return this.begin("block"),17;break;case 18:return 49;case 19:return 49;case 20:return 42;case 21:return 15;case 22:return 13;case 23:break;case 24:return 59;case 25:return 56;case 26:return 56;case 27:return 60;case 28:break;case 29:return this.popState(),19;break;case 30:return Z.yytext[0];case 31:return 20;case 32:return 21;case 33:return this.begin("style"),44;break;case 34:return this.popState(),10;break;case 35:break;case 36:return 13;case 37:return 42;case 38:return 49;case 39:return this.begin("style"),37;break;case 40:return 43;case 41:return 63;case 42:return 65;case 43:return 65;case 44:return 65;case 45:return 63;case 46:return 63;case 47:return 64;case 48:return 64;case 49:return 64;case 50:return 64;case 51:return 64;case 52:return 65;case 53:return 64;case 54:return 65;case 55:return 66;case 56:return 66;case 57:return 66;case 58:return 66;case 59:return 63;case 60:return 64;case 61:return 65;case 62:return 67;case 63:return 68;case 64:return 69;case 65:return 69;case 66:return 68;case 67:return 68;case 68:return 68;case 69:return 41;case 70:return 47;case 71:return 40;case 72:return 48;case 73:return Z.yytext[0];case 74:return 6}},"anonymous"),rules:[/^(?:accTitle\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*\{\s*)/i,/^(?:[\}])/i,/^(?:[^\}]*)/i,/^(?:.*direction\s+TB[^\n]*)/i,/^(?:.*direction\s+BT[^\n]*)/i,/^(?:.*direction\s+RL[^\n]*)/i,/^(?:.*direction\s+LR[^\n]*)/i,/^(?:[\n]+)/i,/^(?:\s+)/i,/^(?:[\s]+)/i,/^(?:"[^"%\r\n\v\b\\]+")/i,/^(?:"[^"]*")/i,/^(?:erDiagram\b)/i,/^(?:\{)/i,/^(?:#)/i,/^(?:#)/i,/^(?:,)/i,/^(?::::)/i,/^(?::)/i,/^(?:\s+)/i,/^(?:\b((?:PK)|(?:FK)|(?:UK))\b)/i,/^(?:([^\s]*)[~].*[~]([^\s]*))/i,/^(?:([\*A-Za-z_\u00C0-\uFFFF][A-Za-z0-9\-\_\[\]\(\)\u00C0-\uFFFF\*]*))/i,/^(?:"[^"]*")/i,/^(?:[\n]+)/i,/^(?:\})/i,/^(?:.)/i,/^(?:\[)/i,/^(?:\])/i,/^(?:style\b)/i,/^(?:[\n]+)/i,/^(?:\s+)/i,/^(?::)/i,/^(?:,)/i,/^(?:#)/i,/^(?:classDef\b)/i,/^(?:class\b)/i,/^(?:one or zero\b)/i,/^(?:one or more\b)/i,/^(?:one or many\b)/i,/^(?:1\+)/i,/^(?:\|o\b)/i,/^(?:zero or one\b)/i,/^(?:zero or more\b)/i,/^(?:zero or many\b)/i,/^(?:0\+)/i,/^(?:\}o\b)/i,/^(?:many\(0\))/i,/^(?:many\(1\))/i,/^(?:many\b)/i,/^(?:\}\|)/i,/^(?:one\b)/i,/^(?:only one\b)/i,/^(?:1\b)/i,/^(?:\|\|)/i,/^(?:o\|)/i,/^(?:o\{)/i,/^(?:\|\{)/i,/^(?:\s*u\b)/i,/^(?:\.\.)/i,/^(?:--)/i,/^(?:to\b)/i,/^(?:optionally to\b)/i,/^(?:\.-)/i,/^(?:-\.)/i,/^(?:([^\x00-\x7F]|\w|-|\*)+)/i,/^(?:;)/i,/^(?:([^\x00-\x7F]|\w|-|\*)+)/i,/^(?:[0-9])/i,/^(?:.)/i,/^(?:$)/i],conditions:{style:{rules:[34,35,36,37,38,69,70],inclusive:!1},acc_descr_multiline:{rules:[5,6],inclusive:!1},acc_descr:{rules:[3],inclusive:!1},acc_title:{rules:[1],inclusive:!1},block:{rules:[23,24,25,26,27,28,29,30],inclusive:!1},INITIAL:{rules:[0,2,4,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,31,32,33,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,71,72,73,74],inclusive:!0}}};return J}();K.lexer=X;function te(){this.yy={}}return o(te,"Parser"),te.prototype=K,K.Parser=te,new te}();wR.parser=wR;Zie=wR});var sk,eae=N(()=>{"use strict";vt();zt();mi();ir();sk=class{constructor(){this.entities=new Map;this.relationships=[];this.classes=new Map;this.direction="TB";this.Cardinality={ZERO_OR_ONE:"ZERO_OR_ONE",ZERO_OR_MORE:"ZERO_OR_MORE",ONE_OR_MORE:"ONE_OR_MORE",ONLY_ONE:"ONLY_ONE",MD_PARENT:"MD_PARENT"};this.Identification={NON_IDENTIFYING:"NON_IDENTIFYING",IDENTIFYING:"IDENTIFYING"};this.setAccTitle=Lr;this.getAccTitle=Rr;this.setAccDescription=Nr;this.getAccDescription=Mr;this.setDiagramTitle=$r;this.getDiagramTitle=Ir;this.getConfig=o(()=>me().er,"getConfig");this.clear(),this.addEntity=this.addEntity.bind(this),this.addAttributes=this.addAttributes.bind(this),this.addRelationship=this.addRelationship.bind(this),this.setDirection=this.setDirection.bind(this),this.addCssStyles=this.addCssStyles.bind(this),this.addClass=this.addClass.bind(this),this.setClass=this.setClass.bind(this),this.setAccTitle=this.setAccTitle.bind(this),this.setAccDescription=this.setAccDescription.bind(this)}static{o(this,"ErDB")}addEntity(e,r=""){return this.entities.has(e)?!this.entities.get(e)?.alias&&r&&(this.entities.get(e).alias=r,Y.info(`Add alias '${r}' to entity '${e}'`)):(this.entities.set(e,{id:`entity-${e}-${this.entities.size}`,label:e,attributes:[],alias:r,shape:"erBox",look:me().look??"default",cssClasses:"default",cssStyles:[]}),Y.info("Added new entity :",e)),this.entities.get(e)}getEntity(e){return this.entities.get(e)}getEntities(){return this.entities}getClasses(){return this.classes}addAttributes(e,r){let n=this.addEntity(e),i;for(i=r.length-1;i>=0;i--)r[i].keys||(r[i].keys=[]),r[i].comment||(r[i].comment=""),n.attributes.push(r[i]),Y.debug("Added attribute ",r[i].name)}addRelationship(e,r,n,i){let a=this.entities.get(e),s=this.entities.get(n);if(!a||!s)return;let l={entityA:a.id,roleA:r,entityB:s.id,relSpec:i};this.relationships.push(l),Y.debug("Added new relationship :",l)}getRelationships(){return this.relationships}getDirection(){return this.direction}setDirection(e){this.direction=e}getCompiledStyles(e){let r=[];for(let n of e){let i=this.classes.get(n);i?.styles&&(r=[...r,...i.styles??[]].map(a=>a.trim())),i?.textStyles&&(r=[...r,...i.textStyles??[]].map(a=>a.trim()))}return r}addCssStyles(e,r){for(let n of e){let i=this.entities.get(n);if(!r||!i)return;for(let a of r)i.cssStyles.push(a)}}addClass(e,r){e.forEach(n=>{let i=this.classes.get(n);i===void 0&&(i={id:n,styles:[],textStyles:[]},this.classes.set(n,i)),r&&r.forEach(function(a){if(/color/.exec(a)){let s=a.replace("fill","bgFill");i.textStyles.push(s)}i.styles.push(a)})})}setClass(e,r){for(let n of e){let i=this.entities.get(n);if(i)for(let a of r)i.cssClasses+=" "+a}}clear(){this.entities=new Map,this.classes=new Map,this.relationships=[],Ar()}getData(){let e=[],r=[],n=me();for(let a of this.entities.keys()){let s=this.entities.get(a);s&&(s.cssCompiledStyles=this.getCompiledStyles(s.cssClasses.split(" ")),e.push(s))}let i=0;for(let a of this.relationships){let s={id:$h(a.entityA,a.entityB,{prefix:"id",counter:i++}),type:"normal",curve:"basis",start:a.entityA,end:a.entityB,label:a.roleA,labelpos:"c",thickness:"normal",classes:"relationshipLine",arrowTypeStart:a.relSpec.cardB.toLowerCase(),arrowTypeEnd:a.relSpec.cardA.toLowerCase(),pattern:a.relSpec.relType=="IDENTIFYING"?"solid":"dashed",look:n.look};r.push(s)}return{nodes:e,edges:r,other:{},config:n,direction:"TB"}}}});var TR={};hr(TR,{draw:()=>SOe});var SOe,tae=N(()=>{"use strict";zt();vt();gm();Yd();$m();ir();dr();SOe=o(async function(t,e,r,n){Y.info("REF0:"),Y.info("Drawing er diagram (unified)",e);let{securityLevel:i,er:a,layout:s}=me(),l=n.db.getData(),u=yc(e,i);l.type=n.type,l.layoutAlgorithm=nf(s),l.config.flowchart.nodeSpacing=a?.nodeSpacing||140,l.config.flowchart.rankSpacing=a?.rankSpacing||80,l.direction=n.db.getDirection(),l.markers=["only_one","zero_or_one","one_or_more","zero_or_more"],l.diagramId=e,await Cc(l,u),l.layoutAlgorithm==="elk"&&u.select(".edges").lower();let h=u.selectAll('[id*="-background"]');Array.from(h).length>0&&h.each(function(){let d=Ge(this),m=d.attr("id").replace("-background",""),g=u.select(`#${CSS.escape(m)}`);if(!g.empty()){let y=g.attr("transform");d.attr("transform",y)}});let f=8;Gt.insertTitle(u,"erDiagramTitleText",a?.titleTopMargin??25,n.db.getDiagramTitle()),Ac(u,f,"erDiagram",a?.useMaxWidth??!0)},"draw")});var COe,AOe,rae,nae=N(()=>{"use strict";Ys();COe=o((t,e)=>{let r=Kf,n=r(t,"r"),i=r(t,"g"),a=r(t,"b");return qa(n,i,a,e)},"fade"),AOe=o(t=>` + .entityBox { + fill: ${t.mainBkg}; + stroke: ${t.nodeBorder}; + } + + .relationshipLabelBox { + fill: ${t.tertiaryColor}; + opacity: 0.7; + background-color: ${t.tertiaryColor}; + rect { + opacity: 0.5; + } + } + + .labelBkg { + background-color: ${COe(t.tertiaryColor,.5)}; + } + + .edgeLabel .label { + fill: ${t.nodeBorder}; + font-size: 14px; + } + + .label { + font-family: ${t.fontFamily}; + color: ${t.nodeTextColor||t.textColor}; + } + + .edge-pattern-dashed { + stroke-dasharray: 8,8; + } + + .node rect, + .node circle, + .node ellipse, + .node polygon + { + fill: ${t.mainBkg}; + stroke: ${t.nodeBorder}; + stroke-width: 1px; + } + + .relationshipLine { + stroke: ${t.lineColor}; + stroke-width: 1; + fill: none; + } + + .marker { + fill: none !important; + stroke: ${t.lineColor} !important; + stroke-width: 1; + } +`,"getStyles"),rae=AOe});var iae={};hr(iae,{diagram:()=>_Oe});var _Oe,aae=N(()=>{"use strict";Jie();eae();tae();nae();_Oe={parser:Zie,get db(){return new sk},renderer:TR,styles:rae}});function ii(t){return typeof t=="object"&&t!==null&&typeof t.$type=="string"}function va(t){return typeof t=="object"&&t!==null&&typeof t.$refText=="string"}function kR(t){return typeof t=="object"&&t!==null&&typeof t.name=="string"&&typeof t.type=="string"&&typeof t.path=="string"}function jd(t){return typeof t=="object"&&t!==null&&ii(t.container)&&va(t.reference)&&typeof t.message=="string"}function Ll(t){return typeof t=="object"&&t!==null&&Array.isArray(t.content)}function af(t){return typeof t=="object"&&t!==null&&typeof t.tokenType=="object"}function M2(t){return Ll(t)&&typeof t.fullText=="string"}var Xd,Rl=N(()=>{"use strict";o(ii,"isAstNode");o(va,"isReference");o(kR,"isAstNodeDescription");o(jd,"isLinkingError");Xd=class{static{o(this,"AbstractAstReflection")}constructor(){this.subtypes={},this.allSubtypes={}}isInstance(e,r){return ii(e)&&this.isSubtype(e.$type,r)}isSubtype(e,r){if(e===r)return!0;let n=this.subtypes[e];n||(n=this.subtypes[e]={});let i=n[r];if(i!==void 0)return i;{let a=this.computeIsSubtype(e,r);return n[r]=a,a}}getAllSubTypes(e){let r=this.allSubtypes[e];if(r)return r;{let n=this.getAllTypes(),i=[];for(let a of n)this.isSubtype(a,e)&&i.push(a);return this.allSubtypes[e]=i,i}}};o(Ll,"isCompositeCstNode");o(af,"isLeafCstNode");o(M2,"isRootCstNode")});function NOe(t){return typeof t=="string"?t:typeof t>"u"?"undefined":typeof t.toString=="function"?t.toString():Object.prototype.toString.call(t)}function ok(t){return!!t&&typeof t[Symbol.iterator]=="function"}function en(...t){if(t.length===1){let e=t[0];if(e instanceof ao)return e;if(ok(e))return new ao(()=>e[Symbol.iterator](),r=>r.next());if(typeof e.length=="number")return new ao(()=>({index:0}),r=>r.index1?new ao(()=>({collIndex:0,arrIndex:0}),e=>{do{if(e.iterator){let r=e.iterator.next();if(!r.done)return r;e.iterator=void 0}if(e.array){if(e.arrIndex{"use strict";ao=class t{static{o(this,"StreamImpl")}constructor(e,r){this.startFn=e,this.nextFn=r}iterator(){let e={state:this.startFn(),next:o(()=>this.nextFn(e.state),"next"),[Symbol.iterator]:()=>e};return e}[Symbol.iterator](){return this.iterator()}isEmpty(){return!!this.iterator().next().done}count(){let e=this.iterator(),r=0,n=e.next();for(;!n.done;)r++,n=e.next();return r}toArray(){let e=[],r=this.iterator(),n;do n=r.next(),n.value!==void 0&&e.push(n.value);while(!n.done);return e}toSet(){return new Set(this)}toMap(e,r){let n=this.map(i=>[e?e(i):i,r?r(i):i]);return new Map(n)}toString(){return this.join()}concat(e){return new t(()=>({first:this.startFn(),firstDone:!1,iterator:e[Symbol.iterator]()}),r=>{let n;if(!r.firstDone){do if(n=this.nextFn(r.first),!n.done)return n;while(!n.done);r.firstDone=!0}do if(n=r.iterator.next(),!n.done)return n;while(!n.done);return Ia})}join(e=","){let r=this.iterator(),n="",i,a=!1;do i=r.next(),i.done||(a&&(n+=e),n+=NOe(i.value)),a=!0;while(!i.done);return n}indexOf(e,r=0){let n=this.iterator(),i=0,a=n.next();for(;!a.done;){if(i>=r&&a.value===e)return i;a=n.next(),i++}return-1}every(e){let r=this.iterator(),n=r.next();for(;!n.done;){if(!e(n.value))return!1;n=r.next()}return!0}some(e){let r=this.iterator(),n=r.next();for(;!n.done;){if(e(n.value))return!0;n=r.next()}return!1}forEach(e){let r=this.iterator(),n=0,i=r.next();for(;!i.done;)e(i.value,n),i=r.next(),n++}map(e){return new t(this.startFn,r=>{let{done:n,value:i}=this.nextFn(r);return n?Ia:{done:!1,value:e(i)}})}filter(e){return new t(this.startFn,r=>{let n;do if(n=this.nextFn(r),!n.done&&e(n.value))return n;while(!n.done);return Ia})}nonNullable(){return this.filter(e=>e!=null)}reduce(e,r){let n=this.iterator(),i=r,a=n.next();for(;!a.done;)i===void 0?i=a.value:i=e(i,a.value),a=n.next();return i}reduceRight(e,r){return this.recursiveReduce(this.iterator(),e,r)}recursiveReduce(e,r,n){let i=e.next();if(i.done)return n;let a=this.recursiveReduce(e,r,n);return a===void 0?i.value:r(a,i.value)}find(e){let r=this.iterator(),n=r.next();for(;!n.done;){if(e(n.value))return n.value;n=r.next()}}findIndex(e){let r=this.iterator(),n=0,i=r.next();for(;!i.done;){if(e(i.value))return n;i=r.next(),n++}return-1}includes(e){let r=this.iterator(),n=r.next();for(;!n.done;){if(n.value===e)return!0;n=r.next()}return!1}flatMap(e){return new t(()=>({this:this.startFn()}),r=>{do{if(r.iterator){let a=r.iterator.next();if(a.done)r.iterator=void 0;else return a}let{done:n,value:i}=this.nextFn(r.this);if(!n){let a=e(i);if(ok(a))r.iterator=a[Symbol.iterator]();else return{done:!1,value:a}}}while(r.iterator);return Ia})}flat(e){if(e===void 0&&(e=1),e<=0)return this;let r=e>1?this.flat(e-1):this;return new t(()=>({this:r.startFn()}),n=>{do{if(n.iterator){let s=n.iterator.next();if(s.done)n.iterator=void 0;else return s}let{done:i,value:a}=r.nextFn(n.this);if(!i)if(ok(a))n.iterator=a[Symbol.iterator]();else return{done:!1,value:a}}while(n.iterator);return Ia})}head(){let r=this.iterator().next();if(!r.done)return r.value}tail(e=1){return new t(()=>{let r=this.startFn();for(let n=0;n({size:0,state:this.startFn()}),r=>(r.size++,r.size>e?Ia:this.nextFn(r.state)))}distinct(e){return new t(()=>({set:new Set,internalState:this.startFn()}),r=>{let n;do if(n=this.nextFn(r.internalState),!n.done){let i=e?e(n.value):n.value;if(!r.set.has(i))return r.set.add(i),n}while(!n.done);return Ia})}exclude(e,r){let n=new Set;for(let i of e){let a=r?r(i):i;n.add(a)}return this.filter(i=>{let a=r?r(i):i;return!n.has(a)})}};o(NOe,"toString");o(ok,"isIterable");I2=new ao(()=>{},()=>Ia),Ia=Object.freeze({done:!0,value:void 0});o(en,"stream");_c=class extends ao{static{o(this,"TreeStreamImpl")}constructor(e,r,n){super(()=>({iterators:n?.includeRoot?[[e][Symbol.iterator]()]:[r(e)[Symbol.iterator]()],pruned:!1}),i=>{for(i.pruned&&(i.iterators.pop(),i.pruned=!1);i.iterators.length>0;){let s=i.iterators[i.iterators.length-1].next();if(s.done)i.iterators.pop();else return i.iterators.push(r(s.value)[Symbol.iterator]()),s}return Ia})}iterator(){let e={state:this.startFn(),next:o(()=>this.nextFn(e.state),"next"),prune:o(()=>{e.state.pruned=!0},"prune"),[Symbol.iterator]:()=>e};return e}};(function(t){function e(a){return a.reduce((s,l)=>s+l,0)}o(e,"sum"),t.sum=e;function r(a){return a.reduce((s,l)=>s*l,0)}o(r,"product"),t.product=r;function n(a){return a.reduce((s,l)=>Math.min(s,l))}o(n,"min"),t.min=n;function i(a){return a.reduce((s,l)=>Math.max(s,l))}o(i,"max"),t.max=i})(zm||(zm={}))});var ck={};hr(ck,{DefaultNameRegexp:()=>lk,RangeComparison:()=>Dc,compareRange:()=>cae,findCommentNode:()=>AR,findDeclarationNodeAtOffset:()=>IOe,findLeafNodeAtOffset:()=>_R,findLeafNodeBeforeOffset:()=>uae,flattenCst:()=>MOe,getInteriorNodes:()=>BOe,getNextNode:()=>OOe,getPreviousNode:()=>fae,getStartlineNode:()=>POe,inRange:()=>CR,isChildNode:()=>SR,isCommentNode:()=>ER,streamCst:()=>Kd,toDocumentSegment:()=>Qd,tokenToRange:()=>Gm});function Kd(t){return new _c(t,e=>Ll(e)?e.content:[],{includeRoot:!0})}function MOe(t){return Kd(t).filter(af)}function SR(t,e){for(;t.container;)if(t=t.container,t===e)return!0;return!1}function Gm(t){return{start:{character:t.startColumn-1,line:t.startLine-1},end:{character:t.endColumn,line:t.endLine-1}}}function Qd(t){if(!t)return;let{offset:e,end:r,range:n}=t;return{range:n,offset:e,end:r,length:r-e}}function cae(t,e){if(t.end.linee.end.line||t.start.line===e.end.line&&t.start.character>=e.end.character)return Dc.After;let r=t.start.line>e.start.line||t.start.line===e.start.line&&t.start.character>=e.start.character,n=t.end.lineDc.After}function IOe(t,e,r=lk){if(t){if(e>0){let n=e-t.offset,i=t.text.charAt(n);r.test(i)||e--}return _R(t,e)}}function AR(t,e){if(t){let r=fae(t,!0);if(r&&ER(r,e))return r;if(M2(t)){let n=t.content.findIndex(i=>!i.hidden);for(let i=n-1;i>=0;i--){let a=t.content[i];if(ER(a,e))return a}}}}function ER(t,e){return af(t)&&e.includes(t.tokenType.name)}function _R(t,e){if(af(t))return t;if(Ll(t)){let r=hae(t,e,!1);if(r)return _R(r,e)}}function uae(t,e){if(af(t))return t;if(Ll(t)){let r=hae(t,e,!0);if(r)return uae(r,e)}}function hae(t,e,r){let n=0,i=t.content.length-1,a;for(;n<=i;){let s=Math.floor((n+i)/2),l=t.content[s];if(l.offset<=e&&l.end>e)return l;l.end<=e?(a=r?l:void 0,n=s+1):i=s-1}return a}function fae(t,e=!0){for(;t.container;){let r=t.container,n=r.content.indexOf(t);for(;n>0;){n--;let i=r.content[n];if(e||!i.hidden)return i}t=r}}function OOe(t,e=!0){for(;t.container;){let r=t.container,n=r.content.indexOf(t),i=r.content.length-1;for(;n{"use strict";Rl();Ps();o(Kd,"streamCst");o(MOe,"flattenCst");o(SR,"isChildNode");o(Gm,"tokenToRange");o(Qd,"toDocumentSegment");(function(t){t[t.Before=0]="Before",t[t.After=1]="After",t[t.OverlapFront=2]="OverlapFront",t[t.OverlapBack=3]="OverlapBack",t[t.Inside=4]="Inside",t[t.Outside=5]="Outside"})(Dc||(Dc={}));o(cae,"compareRange");o(CR,"inRange");lk=/^[\w\p{L}]$/u;o(IOe,"findDeclarationNodeAtOffset");o(AR,"findCommentNode");o(ER,"isCommentNode");o(_R,"findLeafNodeAtOffset");o(uae,"findLeafNodeBeforeOffset");o(hae,"binarySearch");o(fae,"getPreviousNode");o(OOe,"getNextNode");o(POe,"getStartlineNode");o(BOe,"getInteriorNodes");o(FOe,"getCommonParent");o(lae,"getParentChain")});function Lc(t){throw new Error("Error! The input value was not handled.")}var Zd,uk=N(()=>{"use strict";Zd=class extends Error{static{o(this,"ErrorWithLocation")}constructor(e,r){super(e?`${r} at ${e.range.start.line}:${e.range.start.character}`:r)}};o(Lc,"assertUnreachable")});var U2={};hr(U2,{AbstractElement:()=>Hm,AbstractRule:()=>Vm,AbstractType:()=>Um,Action:()=>cg,Alternatives:()=>ug,ArrayLiteral:()=>Wm,ArrayType:()=>qm,Assignment:()=>hg,BooleanLiteral:()=>Ym,CharacterRange:()=>fg,Condition:()=>O2,Conjunction:()=>Xm,CrossReference:()=>dg,Disjunction:()=>jm,EndOfFile:()=>pg,Grammar:()=>Km,GrammarImport:()=>B2,Group:()=>mg,InferredType:()=>Qm,Interface:()=>Zm,Keyword:()=>gg,LangiumGrammarAstReflection:()=>Cg,LangiumGrammarTerminals:()=>$Oe,NamedArgument:()=>F2,NegatedToken:()=>yg,Negation:()=>Jm,NumberLiteral:()=>eg,Parameter:()=>tg,ParameterReference:()=>rg,ParserRule:()=>ng,ReferenceType:()=>ig,RegexToken:()=>vg,ReturnType:()=>$2,RuleCall:()=>xg,SimpleType:()=>ag,StringLiteral:()=>sg,TerminalAlternatives:()=>bg,TerminalGroup:()=>wg,TerminalRule:()=>Jd,TerminalRuleCall:()=>Tg,Type:()=>og,TypeAttribute:()=>z2,TypeDefinition:()=>hk,UnionType:()=>lg,UnorderedGroup:()=>kg,UntilToken:()=>Eg,ValueLiteral:()=>P2,Wildcard:()=>Sg,isAbstractElement:()=>G2,isAbstractRule:()=>zOe,isAbstractType:()=>GOe,isAction:()=>Mu,isAlternatives:()=>mk,isArrayLiteral:()=>qOe,isArrayType:()=>DR,isAssignment:()=>Ml,isBooleanLiteral:()=>LR,isCharacterRange:()=>FR,isCondition:()=>VOe,isConjunction:()=>RR,isCrossReference:()=>ep,isDisjunction:()=>NR,isEndOfFile:()=>$R,isFeatureName:()=>UOe,isGrammar:()=>YOe,isGrammarImport:()=>XOe,isGroup:()=>sf,isInferredType:()=>fk,isInterface:()=>dk,isKeyword:()=>Ho,isNamedArgument:()=>jOe,isNegatedToken:()=>zR,isNegation:()=>MR,isNumberLiteral:()=>KOe,isParameter:()=>QOe,isParameterReference:()=>IR,isParserRule:()=>Oa,isPrimitiveType:()=>dae,isReferenceType:()=>OR,isRegexToken:()=>GR,isReturnType:()=>PR,isRuleCall:()=>Il,isSimpleType:()=>pk,isStringLiteral:()=>ZOe,isTerminalAlternatives:()=>VR,isTerminalGroup:()=>UR,isTerminalRule:()=>so,isTerminalRuleCall:()=>gk,isType:()=>V2,isTypeAttribute:()=>JOe,isTypeDefinition:()=>HOe,isUnionType:()=>BR,isUnorderedGroup:()=>yk,isUntilToken:()=>HR,isValueLiteral:()=>WOe,isWildcard:()=>WR,reflection:()=>lr});function zOe(t){return lr.isInstance(t,Vm)}function GOe(t){return lr.isInstance(t,Um)}function VOe(t){return lr.isInstance(t,O2)}function UOe(t){return dae(t)||t==="current"||t==="entry"||t==="extends"||t==="false"||t==="fragment"||t==="grammar"||t==="hidden"||t==="import"||t==="interface"||t==="returns"||t==="terminal"||t==="true"||t==="type"||t==="infer"||t==="infers"||t==="with"||typeof t=="string"&&/\^?[_a-zA-Z][\w_]*/.test(t)}function dae(t){return t==="string"||t==="number"||t==="boolean"||t==="Date"||t==="bigint"}function HOe(t){return lr.isInstance(t,hk)}function WOe(t){return lr.isInstance(t,P2)}function G2(t){return lr.isInstance(t,Hm)}function qOe(t){return lr.isInstance(t,Wm)}function DR(t){return lr.isInstance(t,qm)}function LR(t){return lr.isInstance(t,Ym)}function RR(t){return lr.isInstance(t,Xm)}function NR(t){return lr.isInstance(t,jm)}function YOe(t){return lr.isInstance(t,Km)}function XOe(t){return lr.isInstance(t,B2)}function fk(t){return lr.isInstance(t,Qm)}function dk(t){return lr.isInstance(t,Zm)}function jOe(t){return lr.isInstance(t,F2)}function MR(t){return lr.isInstance(t,Jm)}function KOe(t){return lr.isInstance(t,eg)}function QOe(t){return lr.isInstance(t,tg)}function IR(t){return lr.isInstance(t,rg)}function Oa(t){return lr.isInstance(t,ng)}function OR(t){return lr.isInstance(t,ig)}function PR(t){return lr.isInstance(t,$2)}function pk(t){return lr.isInstance(t,ag)}function ZOe(t){return lr.isInstance(t,sg)}function so(t){return lr.isInstance(t,Jd)}function V2(t){return lr.isInstance(t,og)}function JOe(t){return lr.isInstance(t,z2)}function BR(t){return lr.isInstance(t,lg)}function Mu(t){return lr.isInstance(t,cg)}function mk(t){return lr.isInstance(t,ug)}function Ml(t){return lr.isInstance(t,hg)}function FR(t){return lr.isInstance(t,fg)}function ep(t){return lr.isInstance(t,dg)}function $R(t){return lr.isInstance(t,pg)}function sf(t){return lr.isInstance(t,mg)}function Ho(t){return lr.isInstance(t,gg)}function zR(t){return lr.isInstance(t,yg)}function GR(t){return lr.isInstance(t,vg)}function Il(t){return lr.isInstance(t,xg)}function VR(t){return lr.isInstance(t,bg)}function UR(t){return lr.isInstance(t,wg)}function gk(t){return lr.isInstance(t,Tg)}function yk(t){return lr.isInstance(t,kg)}function HR(t){return lr.isInstance(t,Eg)}function WR(t){return lr.isInstance(t,Sg)}var $Oe,Vm,Um,O2,hk,P2,Hm,Wm,qm,Ym,Xm,jm,Km,B2,Qm,Zm,F2,Jm,eg,tg,rg,ng,ig,$2,ag,sg,Jd,og,z2,lg,cg,ug,hg,fg,dg,pg,mg,gg,yg,vg,xg,bg,wg,Tg,kg,Eg,Sg,Cg,lr,Rc=N(()=>{"use strict";Rl();$Oe={ID:/\^?[_a-zA-Z][\w_]*/,STRING:/"(\\.|[^"\\])*"|'(\\.|[^'\\])*'/,NUMBER:/NaN|-?((\d*\.\d+|\d+)([Ee][+-]?\d+)?|Infinity)/,RegexLiteral:/\/(?![*+?])(?:[^\r\n\[/\\]|\\.|\[(?:[^\r\n\]\\]|\\.)*\])+\/[a-z]*/,WS:/\s+/,ML_COMMENT:/\/\*[\s\S]*?\*\//,SL_COMMENT:/\/\/[^\n\r]*/},Vm="AbstractRule";o(zOe,"isAbstractRule");Um="AbstractType";o(GOe,"isAbstractType");O2="Condition";o(VOe,"isCondition");o(UOe,"isFeatureName");o(dae,"isPrimitiveType");hk="TypeDefinition";o(HOe,"isTypeDefinition");P2="ValueLiteral";o(WOe,"isValueLiteral");Hm="AbstractElement";o(G2,"isAbstractElement");Wm="ArrayLiteral";o(qOe,"isArrayLiteral");qm="ArrayType";o(DR,"isArrayType");Ym="BooleanLiteral";o(LR,"isBooleanLiteral");Xm="Conjunction";o(RR,"isConjunction");jm="Disjunction";o(NR,"isDisjunction");Km="Grammar";o(YOe,"isGrammar");B2="GrammarImport";o(XOe,"isGrammarImport");Qm="InferredType";o(fk,"isInferredType");Zm="Interface";o(dk,"isInterface");F2="NamedArgument";o(jOe,"isNamedArgument");Jm="Negation";o(MR,"isNegation");eg="NumberLiteral";o(KOe,"isNumberLiteral");tg="Parameter";o(QOe,"isParameter");rg="ParameterReference";o(IR,"isParameterReference");ng="ParserRule";o(Oa,"isParserRule");ig="ReferenceType";o(OR,"isReferenceType");$2="ReturnType";o(PR,"isReturnType");ag="SimpleType";o(pk,"isSimpleType");sg="StringLiteral";o(ZOe,"isStringLiteral");Jd="TerminalRule";o(so,"isTerminalRule");og="Type";o(V2,"isType");z2="TypeAttribute";o(JOe,"isTypeAttribute");lg="UnionType";o(BR,"isUnionType");cg="Action";o(Mu,"isAction");ug="Alternatives";o(mk,"isAlternatives");hg="Assignment";o(Ml,"isAssignment");fg="CharacterRange";o(FR,"isCharacterRange");dg="CrossReference";o(ep,"isCrossReference");pg="EndOfFile";o($R,"isEndOfFile");mg="Group";o(sf,"isGroup");gg="Keyword";o(Ho,"isKeyword");yg="NegatedToken";o(zR,"isNegatedToken");vg="RegexToken";o(GR,"isRegexToken");xg="RuleCall";o(Il,"isRuleCall");bg="TerminalAlternatives";o(VR,"isTerminalAlternatives");wg="TerminalGroup";o(UR,"isTerminalGroup");Tg="TerminalRuleCall";o(gk,"isTerminalRuleCall");kg="UnorderedGroup";o(yk,"isUnorderedGroup");Eg="UntilToken";o(HR,"isUntilToken");Sg="Wildcard";o(WR,"isWildcard");Cg=class extends Xd{static{o(this,"LangiumGrammarAstReflection")}getAllTypes(){return[Hm,Vm,Um,cg,ug,Wm,qm,hg,Ym,fg,O2,Xm,dg,jm,pg,Km,B2,mg,Qm,Zm,gg,F2,yg,Jm,eg,tg,rg,ng,ig,vg,$2,xg,ag,sg,bg,wg,Jd,Tg,og,z2,hk,lg,kg,Eg,P2,Sg]}computeIsSubtype(e,r){switch(e){case cg:case ug:case hg:case fg:case dg:case pg:case mg:case gg:case yg:case vg:case xg:case bg:case wg:case Tg:case kg:case Eg:case Sg:return this.isSubtype(Hm,r);case Wm:case eg:case sg:return this.isSubtype(P2,r);case qm:case ig:case ag:case lg:return this.isSubtype(hk,r);case Ym:return this.isSubtype(O2,r)||this.isSubtype(P2,r);case Xm:case jm:case Jm:case rg:return this.isSubtype(O2,r);case Qm:case Zm:case og:return this.isSubtype(Um,r);case ng:return this.isSubtype(Vm,r)||this.isSubtype(Um,r);case Jd:return this.isSubtype(Vm,r);default:return!1}}getReferenceType(e){let r=`${e.container.$type}:${e.property}`;switch(r){case"Action:type":case"CrossReference:type":case"Interface:superTypes":case"ParserRule:returnType":case"SimpleType:typeRef":return Um;case"Grammar:hiddenTokens":case"ParserRule:hiddenTokens":case"RuleCall:rule":return Vm;case"Grammar:usedGrammars":return Km;case"NamedArgument:parameter":case"ParameterReference:parameter":return tg;case"TerminalRuleCall:rule":return Jd;default:throw new Error(`${r} is not a valid reference id.`)}}getTypeMetaData(e){switch(e){case Hm:return{name:Hm,properties:[{name:"cardinality"},{name:"lookahead"}]};case Wm:return{name:Wm,properties:[{name:"elements",defaultValue:[]}]};case qm:return{name:qm,properties:[{name:"elementType"}]};case Ym:return{name:Ym,properties:[{name:"true",defaultValue:!1}]};case Xm:return{name:Xm,properties:[{name:"left"},{name:"right"}]};case jm:return{name:jm,properties:[{name:"left"},{name:"right"}]};case Km:return{name:Km,properties:[{name:"definesHiddenTokens",defaultValue:!1},{name:"hiddenTokens",defaultValue:[]},{name:"imports",defaultValue:[]},{name:"interfaces",defaultValue:[]},{name:"isDeclared",defaultValue:!1},{name:"name"},{name:"rules",defaultValue:[]},{name:"types",defaultValue:[]},{name:"usedGrammars",defaultValue:[]}]};case B2:return{name:B2,properties:[{name:"path"}]};case Qm:return{name:Qm,properties:[{name:"name"}]};case Zm:return{name:Zm,properties:[{name:"attributes",defaultValue:[]},{name:"name"},{name:"superTypes",defaultValue:[]}]};case F2:return{name:F2,properties:[{name:"calledByName",defaultValue:!1},{name:"parameter"},{name:"value"}]};case Jm:return{name:Jm,properties:[{name:"value"}]};case eg:return{name:eg,properties:[{name:"value"}]};case tg:return{name:tg,properties:[{name:"name"}]};case rg:return{name:rg,properties:[{name:"parameter"}]};case ng:return{name:ng,properties:[{name:"dataType"},{name:"definesHiddenTokens",defaultValue:!1},{name:"definition"},{name:"entry",defaultValue:!1},{name:"fragment",defaultValue:!1},{name:"hiddenTokens",defaultValue:[]},{name:"inferredType"},{name:"name"},{name:"parameters",defaultValue:[]},{name:"returnType"},{name:"wildcard",defaultValue:!1}]};case ig:return{name:ig,properties:[{name:"referenceType"}]};case $2:return{name:$2,properties:[{name:"name"}]};case ag:return{name:ag,properties:[{name:"primitiveType"},{name:"stringType"},{name:"typeRef"}]};case sg:return{name:sg,properties:[{name:"value"}]};case Jd:return{name:Jd,properties:[{name:"definition"},{name:"fragment",defaultValue:!1},{name:"hidden",defaultValue:!1},{name:"name"},{name:"type"}]};case og:return{name:og,properties:[{name:"name"},{name:"type"}]};case z2:return{name:z2,properties:[{name:"defaultValue"},{name:"isOptional",defaultValue:!1},{name:"name"},{name:"type"}]};case lg:return{name:lg,properties:[{name:"types",defaultValue:[]}]};case cg:return{name:cg,properties:[{name:"cardinality"},{name:"feature"},{name:"inferredType"},{name:"lookahead"},{name:"operator"},{name:"type"}]};case ug:return{name:ug,properties:[{name:"cardinality"},{name:"elements",defaultValue:[]},{name:"lookahead"}]};case hg:return{name:hg,properties:[{name:"cardinality"},{name:"feature"},{name:"lookahead"},{name:"operator"},{name:"terminal"}]};case fg:return{name:fg,properties:[{name:"cardinality"},{name:"left"},{name:"lookahead"},{name:"right"}]};case dg:return{name:dg,properties:[{name:"cardinality"},{name:"deprecatedSyntax",defaultValue:!1},{name:"lookahead"},{name:"terminal"},{name:"type"}]};case pg:return{name:pg,properties:[{name:"cardinality"},{name:"lookahead"}]};case mg:return{name:mg,properties:[{name:"cardinality"},{name:"elements",defaultValue:[]},{name:"guardCondition"},{name:"lookahead"}]};case gg:return{name:gg,properties:[{name:"cardinality"},{name:"lookahead"},{name:"value"}]};case yg:return{name:yg,properties:[{name:"cardinality"},{name:"lookahead"},{name:"terminal"}]};case vg:return{name:vg,properties:[{name:"cardinality"},{name:"lookahead"},{name:"regex"}]};case xg:return{name:xg,properties:[{name:"arguments",defaultValue:[]},{name:"cardinality"},{name:"lookahead"},{name:"rule"}]};case bg:return{name:bg,properties:[{name:"cardinality"},{name:"elements",defaultValue:[]},{name:"lookahead"}]};case wg:return{name:wg,properties:[{name:"cardinality"},{name:"elements",defaultValue:[]},{name:"lookahead"}]};case Tg:return{name:Tg,properties:[{name:"cardinality"},{name:"lookahead"},{name:"rule"}]};case kg:return{name:kg,properties:[{name:"cardinality"},{name:"elements",defaultValue:[]},{name:"lookahead"}]};case Eg:return{name:Eg,properties:[{name:"cardinality"},{name:"lookahead"},{name:"terminal"}]};case Sg:return{name:Sg,properties:[{name:"cardinality"},{name:"lookahead"}]};default:return{name:e,properties:[]}}}},lr=new Cg});var xk={};hr(xk,{assignMandatoryProperties:()=>XR,copyAstNode:()=>YR,findLocalReferences:()=>tPe,findRootNode:()=>H2,getContainerOfType:()=>tp,getDocument:()=>Pa,hasContainerOfType:()=>ePe,linkContentToContainer:()=>vk,streamAllContents:()=>Nc,streamAst:()=>Wo,streamContents:()=>W2,streamReferences:()=>Ag});function vk(t){for(let[e,r]of Object.entries(t))e.startsWith("$")||(Array.isArray(r)?r.forEach((n,i)=>{ii(n)&&(n.$container=t,n.$containerProperty=e,n.$containerIndex=i)}):ii(r)&&(r.$container=t,r.$containerProperty=e))}function tp(t,e){let r=t;for(;r;){if(e(r))return r;r=r.$container}}function ePe(t,e){let r=t;for(;r;){if(e(r))return!0;r=r.$container}return!1}function Pa(t){let r=H2(t).$document;if(!r)throw new Error("AST node has no document.");return r}function H2(t){for(;t.$container;)t=t.$container;return t}function W2(t,e){if(!t)throw new Error("Node must be an AstNode.");let r=e?.range;return new ao(()=>({keys:Object.keys(t),keyIndex:0,arrayIndex:0}),n=>{for(;n.keyIndexW2(r,e))}function Wo(t,e){if(t){if(e?.range&&!qR(t,e.range))return new _c(t,()=>[])}else throw new Error("Root node must be an AstNode.");return new _c(t,r=>W2(r,e),{includeRoot:!0})}function qR(t,e){var r;if(!e)return!0;let n=(r=t.$cstNode)===null||r===void 0?void 0:r.range;return n?CR(n,e):!1}function Ag(t){return new ao(()=>({keys:Object.keys(t),keyIndex:0,arrayIndex:0}),e=>{for(;e.keyIndex{Ag(n).forEach(i=>{i.reference.ref===t&&r.push(i.reference)})}),en(r)}function XR(t,e){let r=t.getTypeMetaData(e.$type),n=e;for(let i of r.properties)i.defaultValue!==void 0&&n[i.name]===void 0&&(n[i.name]=pae(i.defaultValue))}function pae(t){return Array.isArray(t)?[...t.map(pae)]:t}function YR(t,e){let r={$type:t.$type};for(let[n,i]of Object.entries(t))if(!n.startsWith("$"))if(ii(i))r[n]=YR(i,e);else if(va(i))r[n]=e(r,n,i.$refNode,i.$refText);else if(Array.isArray(i)){let a=[];for(let s of i)ii(s)?a.push(YR(s,e)):va(s)?a.push(e(r,n,s.$refNode,s.$refText)):a.push(s);r[n]=a}else r[n]=i;return vk(r),r}var is=N(()=>{"use strict";Rl();Ps();Nl();o(vk,"linkContentToContainer");o(tp,"getContainerOfType");o(ePe,"hasContainerOfType");o(Pa,"getDocument");o(H2,"findRootNode");o(W2,"streamContents");o(Nc,"streamAllContents");o(Wo,"streamAst");o(qR,"isAstNodeInRange");o(Ag,"streamReferences");o(tPe,"findLocalReferences");o(XR,"assignMandatoryProperties");o(pae,"copyDefaultValue");o(YR,"copyAstNode")});function ar(t){return t.charCodeAt(0)}function bk(t,e){Array.isArray(t)?t.forEach(function(r){e.push(r)}):e.push(t)}function _g(t,e){if(t[e]===!0)throw"duplicate flag "+e;let r=t[e];t[e]=!0}function rp(t){if(t===void 0)throw Error("Internal Error - Should never get here!");return!0}function q2(){throw Error("Internal Error - Should never get here!")}function jR(t){return t.type==="Character"}var KR=N(()=>{"use strict";o(ar,"cc");o(bk,"insertToSet");o(_g,"addFlag");o(rp,"ASSERT_EXISTS");o(q2,"ASSERT_NEVER_REACH_HERE");o(jR,"isCharacter")});var Y2,X2,QR,mae=N(()=>{"use strict";KR();Y2=[];for(let t=ar("0");t<=ar("9");t++)Y2.push(t);X2=[ar("_")].concat(Y2);for(let t=ar("a");t<=ar("z");t++)X2.push(t);for(let t=ar("A");t<=ar("Z");t++)X2.push(t);QR=[ar(" "),ar("\f"),ar(` +`),ar("\r"),ar(" "),ar("\v"),ar(" "),ar("\xA0"),ar("\u1680"),ar("\u2000"),ar("\u2001"),ar("\u2002"),ar("\u2003"),ar("\u2004"),ar("\u2005"),ar("\u2006"),ar("\u2007"),ar("\u2008"),ar("\u2009"),ar("\u200A"),ar("\u2028"),ar("\u2029"),ar("\u202F"),ar("\u205F"),ar("\u3000"),ar("\uFEFF")]});var rPe,wk,nPe,np,gae=N(()=>{"use strict";KR();mae();rPe=/[0-9a-fA-F]/,wk=/[0-9]/,nPe=/[1-9]/,np=class{static{o(this,"RegExpParser")}constructor(){this.idx=0,this.input="",this.groupIdx=0}saveState(){return{idx:this.idx,input:this.input,groupIdx:this.groupIdx}}restoreState(e){this.idx=e.idx,this.input=e.input,this.groupIdx=e.groupIdx}pattern(e){this.idx=0,this.input=e,this.groupIdx=0,this.consumeChar("/");let r=this.disjunction();this.consumeChar("/");let n={type:"Flags",loc:{begin:this.idx,end:e.length},global:!1,ignoreCase:!1,multiLine:!1,unicode:!1,sticky:!1};for(;this.isRegExpFlag();)switch(this.popChar()){case"g":_g(n,"global");break;case"i":_g(n,"ignoreCase");break;case"m":_g(n,"multiLine");break;case"u":_g(n,"unicode");break;case"y":_g(n,"sticky");break}if(this.idx!==this.input.length)throw Error("Redundant input: "+this.input.substring(this.idx));return{type:"Pattern",flags:n,value:r,loc:this.loc(0)}}disjunction(){let e=[],r=this.idx;for(e.push(this.alternative());this.peekChar()==="|";)this.consumeChar("|"),e.push(this.alternative());return{type:"Disjunction",value:e,loc:this.loc(r)}}alternative(){let e=[],r=this.idx;for(;this.isTerm();)e.push(this.term());return{type:"Alternative",value:e,loc:this.loc(r)}}term(){return this.isAssertion()?this.assertion():this.atom()}assertion(){let e=this.idx;switch(this.popChar()){case"^":return{type:"StartAnchor",loc:this.loc(e)};case"$":return{type:"EndAnchor",loc:this.loc(e)};case"\\":switch(this.popChar()){case"b":return{type:"WordBoundary",loc:this.loc(e)};case"B":return{type:"NonWordBoundary",loc:this.loc(e)}}throw Error("Invalid Assertion Escape");case"(":this.consumeChar("?");let r;switch(this.popChar()){case"=":r="Lookahead";break;case"!":r="NegativeLookahead";break}rp(r);let n=this.disjunction();return this.consumeChar(")"),{type:r,value:n,loc:this.loc(e)}}return q2()}quantifier(e=!1){let r,n=this.idx;switch(this.popChar()){case"*":r={atLeast:0,atMost:1/0};break;case"+":r={atLeast:1,atMost:1/0};break;case"?":r={atLeast:0,atMost:1};break;case"{":let i=this.integerIncludingZero();switch(this.popChar()){case"}":r={atLeast:i,atMost:i};break;case",":let a;this.isDigit()?(a=this.integerIncludingZero(),r={atLeast:i,atMost:a}):r={atLeast:i,atMost:1/0},this.consumeChar("}");break}if(e===!0&&r===void 0)return;rp(r);break}if(!(e===!0&&r===void 0)&&rp(r))return this.peekChar(0)==="?"?(this.consumeChar("?"),r.greedy=!1):r.greedy=!0,r.type="Quantifier",r.loc=this.loc(n),r}atom(){let e,r=this.idx;switch(this.peekChar()){case".":e=this.dotAll();break;case"\\":e=this.atomEscape();break;case"[":e=this.characterClass();break;case"(":e=this.group();break}return e===void 0&&this.isPatternCharacter()&&(e=this.patternCharacter()),rp(e)?(e.loc=this.loc(r),this.isQuantifier()&&(e.quantifier=this.quantifier()),e):q2()}dotAll(){return this.consumeChar("."),{type:"Set",complement:!0,value:[ar(` +`),ar("\r"),ar("\u2028"),ar("\u2029")]}}atomEscape(){switch(this.consumeChar("\\"),this.peekChar()){case"1":case"2":case"3":case"4":case"5":case"6":case"7":case"8":case"9":return this.decimalEscapeAtom();case"d":case"D":case"s":case"S":case"w":case"W":return this.characterClassEscape();case"f":case"n":case"r":case"t":case"v":return this.controlEscapeAtom();case"c":return this.controlLetterEscapeAtom();case"0":return this.nulCharacterAtom();case"x":return this.hexEscapeSequenceAtom();case"u":return this.regExpUnicodeEscapeSequenceAtom();default:return this.identityEscapeAtom()}}decimalEscapeAtom(){return{type:"GroupBackReference",value:this.positiveInteger()}}characterClassEscape(){let e,r=!1;switch(this.popChar()){case"d":e=Y2;break;case"D":e=Y2,r=!0;break;case"s":e=QR;break;case"S":e=QR,r=!0;break;case"w":e=X2;break;case"W":e=X2,r=!0;break}return rp(e)?{type:"Set",value:e,complement:r}:q2()}controlEscapeAtom(){let e;switch(this.popChar()){case"f":e=ar("\f");break;case"n":e=ar(` +`);break;case"r":e=ar("\r");break;case"t":e=ar(" ");break;case"v":e=ar("\v");break}return rp(e)?{type:"Character",value:e}:q2()}controlLetterEscapeAtom(){this.consumeChar("c");let e=this.popChar();if(/[a-zA-Z]/.test(e)===!1)throw Error("Invalid ");return{type:"Character",value:e.toUpperCase().charCodeAt(0)-64}}nulCharacterAtom(){return this.consumeChar("0"),{type:"Character",value:ar("\0")}}hexEscapeSequenceAtom(){return this.consumeChar("x"),this.parseHexDigits(2)}regExpUnicodeEscapeSequenceAtom(){return this.consumeChar("u"),this.parseHexDigits(4)}identityEscapeAtom(){let e=this.popChar();return{type:"Character",value:ar(e)}}classPatternCharacterAtom(){switch(this.peekChar()){case` +`:case"\r":case"\u2028":case"\u2029":case"\\":case"]":throw Error("TBD");default:let e=this.popChar();return{type:"Character",value:ar(e)}}}characterClass(){let e=[],r=!1;for(this.consumeChar("["),this.peekChar(0)==="^"&&(this.consumeChar("^"),r=!0);this.isClassAtom();){let n=this.classAtom(),i=n.type==="Character";if(jR(n)&&this.isRangeDash()){this.consumeChar("-");let a=this.classAtom(),s=a.type==="Character";if(jR(a)){if(a.value=this.input.length)throw Error("Unexpected end of input");this.idx++}loc(e){return{begin:e,end:this.idx}}}});var Mc,yae=N(()=>{"use strict";Mc=class{static{o(this,"BaseRegExpVisitor")}visitChildren(e){for(let r in e){let n=e[r];e.hasOwnProperty(r)&&(n.type!==void 0?this.visit(n):Array.isArray(n)&&n.forEach(i=>{this.visit(i)},this))}}visit(e){switch(e.type){case"Pattern":this.visitPattern(e);break;case"Flags":this.visitFlags(e);break;case"Disjunction":this.visitDisjunction(e);break;case"Alternative":this.visitAlternative(e);break;case"StartAnchor":this.visitStartAnchor(e);break;case"EndAnchor":this.visitEndAnchor(e);break;case"WordBoundary":this.visitWordBoundary(e);break;case"NonWordBoundary":this.visitNonWordBoundary(e);break;case"Lookahead":this.visitLookahead(e);break;case"NegativeLookahead":this.visitNegativeLookahead(e);break;case"Character":this.visitCharacter(e);break;case"Set":this.visitSet(e);break;case"Group":this.visitGroup(e);break;case"GroupBackReference":this.visitGroupBackReference(e);break;case"Quantifier":this.visitQuantifier(e);break}this.visitChildren(e)}visitPattern(e){}visitFlags(e){}visitDisjunction(e){}visitAlternative(e){}visitStartAnchor(e){}visitEndAnchor(e){}visitWordBoundary(e){}visitNonWordBoundary(e){}visitLookahead(e){}visitNegativeLookahead(e){}visitCharacter(e){}visitSet(e){}visitGroup(e){}visitGroupBackReference(e){}visitQuantifier(e){}}});var j2=N(()=>{"use strict";gae();yae()});var Tk={};hr(Tk,{NEWLINE_REGEXP:()=>JR,escapeRegExp:()=>ap,getCaseInsensitivePattern:()=>tN,getTerminalParts:()=>iPe,isMultilineComment:()=>eN,isWhitespace:()=>Dg,partialMatches:()=>rN,partialRegExp:()=>bae,whitespaceCharacters:()=>xae});function iPe(t){try{typeof t!="string"&&(t=t.source),t=`/${t}/`;let e=vae.pattern(t),r=[];for(let n of e.value.value)ip.reset(t),ip.visit(n),r.push({start:ip.startRegexp,end:ip.endRegex});return r}catch{return[]}}function eN(t){try{return typeof t=="string"&&(t=new RegExp(t)),t=t.toString(),ip.reset(t),ip.visit(vae.pattern(t)),ip.multiline}catch{return!1}}function Dg(t){let e=typeof t=="string"?new RegExp(t):t;return xae.some(r=>e.test(r))}function ap(t){return t.replace(/[.*+?^${}()|[\]\\]/g,"\\$&")}function tN(t){return Array.prototype.map.call(t,e=>/\w/.test(e)?`[${e.toLowerCase()}${e.toUpperCase()}]`:ap(e)).join("")}function rN(t,e){let r=bae(t),n=e.match(r);return!!n&&n[0].length>0}function bae(t){typeof t=="string"&&(t=new RegExp(t));let e=t,r=t.source,n=0;function i(){let a="",s;function l(h){a+=r.substr(n,h),n+=h}o(l,"appendRaw");function u(h){a+="(?:"+r.substr(n,h)+"|$)",n+=h}for(o(u,"appendOptional");n",n)-n+1);break;default:u(2);break}break;case"[":s=/\[(?:\\.|.)*?\]/g,s.lastIndex=n,s=s.exec(r)||[],u(s[0].length);break;case"|":case"^":case"$":case"*":case"+":case"?":l(1);break;case"{":s=/\{\d+,?\d*\}/g,s.lastIndex=n,s=s.exec(r),s?l(s[0].length):u(1);break;case"(":if(r[n+1]==="?")switch(r[n+2]){case":":a+="(?:",n+=3,a+=i()+"|$)";break;case"=":a+="(?=",n+=3,a+=i()+")";break;case"!":s=n,n+=3,i(),a+=r.substr(s,n-s);break;case"<":switch(r[n+3]){case"=":case"!":s=n,n+=4,i(),a+=r.substr(s,n-s);break;default:l(r.indexOf(">",n)-n+1),a+=i()+"|$)";break}break}else l(1),a+=i()+"|$)";break;case")":return++n,a;default:u(1);break}return a}return o(i,"process"),new RegExp(i(),t.flags)}var JR,vae,ZR,ip,xae,Lg=N(()=>{"use strict";j2();JR=/\r?\n/gm,vae=new np,ZR=class extends Mc{static{o(this,"TerminalRegExpVisitor")}constructor(){super(...arguments),this.isStarting=!0,this.endRegexpStack=[],this.multiline=!1}get endRegex(){return this.endRegexpStack.join("")}reset(e){this.multiline=!1,this.regex=e,this.startRegexp="",this.isStarting=!0,this.endRegexpStack=[]}visitGroup(e){e.quantifier&&(this.isStarting=!1,this.endRegexpStack=[])}visitCharacter(e){let r=String.fromCharCode(e.value);if(!this.multiline&&r===` +`&&(this.multiline=!0),e.quantifier)this.isStarting=!1,this.endRegexpStack=[];else{let n=ap(r);this.endRegexpStack.push(n),this.isStarting&&(this.startRegexp+=n)}}visitSet(e){if(!this.multiline){let r=this.regex.substring(e.loc.begin,e.loc.end),n=new RegExp(r);this.multiline=!!` +`.match(n)}if(e.quantifier)this.isStarting=!1,this.endRegexpStack=[];else{let r=this.regex.substring(e.loc.begin,e.loc.end);this.endRegexpStack.push(r),this.isStarting&&(this.startRegexp+=r)}}visitChildren(e){e.type==="Group"&&e.quantifier||super.visitChildren(e)}},ip=new ZR;o(iPe,"getTerminalParts");o(eN,"isMultilineComment");xae=`\f +\r \v \xA0\u1680\u2000\u2001\u2002\u2003\u2004\u2005\u2006\u2007\u2008\u2009\u200A\u2028\u2029\u202F\u205F\u3000\uFEFF`.split("");o(Dg,"isWhitespace");o(ap,"escapeRegExp");o(tN,"getCaseInsensitivePattern");o(rN,"partialMatches");o(bae,"partialRegExp")});var Ek={};hr(Ek,{findAssignment:()=>hN,findNameAssignment:()=>kk,findNodeForKeyword:()=>cN,findNodeForProperty:()=>Q2,findNodesForKeyword:()=>aPe,findNodesForKeywordInternal:()=>uN,findNodesForProperty:()=>oN,getActionAtElement:()=>Sae,getActionType:()=>Aae,getAllReachableRules:()=>K2,getCrossReferenceTerminal:()=>aN,getEntryRule:()=>wae,getExplicitRuleType:()=>Rg,getHiddenRules:()=>Tae,getRuleType:()=>fN,getRuleTypeName:()=>uPe,getTypeName:()=>J2,isArrayCardinality:()=>oPe,isArrayOperator:()=>lPe,isCommentTerminal:()=>sN,isDataType:()=>cPe,isDataTypeRule:()=>Z2,isOptionalCardinality:()=>sPe,terminalRegex:()=>Ng});function wae(t){return t.rules.find(e=>Oa(e)&&e.entry)}function Tae(t){return t.rules.filter(e=>so(e)&&e.hidden)}function K2(t,e){let r=new Set,n=wae(t);if(!n)return new Set(t.rules);let i=[n].concat(Tae(t));for(let s of i)kae(s,r,e);let a=new Set;for(let s of t.rules)(r.has(s.name)||so(s)&&s.hidden)&&a.add(s);return a}function kae(t,e,r){e.add(t.name),Nc(t).forEach(n=>{if(Il(n)||r&&gk(n)){let i=n.rule.ref;i&&!e.has(i.name)&&kae(i,e,r)}})}function aN(t){if(t.terminal)return t.terminal;if(t.type.ref){let e=kk(t.type.ref);return e?.terminal}}function sN(t){return t.hidden&&!Dg(Ng(t))}function oN(t,e){return!t||!e?[]:lN(t,e,t.astNode,!0)}function Q2(t,e,r){if(!t||!e)return;let n=lN(t,e,t.astNode,!0);if(n.length!==0)return r!==void 0?r=Math.max(0,Math.min(r,n.length-1)):r=0,n[r]}function lN(t,e,r,n){if(!n){let i=tp(t.grammarSource,Ml);if(i&&i.feature===e)return[t]}return Ll(t)&&t.astNode===r?t.content.flatMap(i=>lN(i,e,r,!1)):[]}function aPe(t,e){return t?uN(t,e,t?.astNode):[]}function cN(t,e,r){if(!t)return;let n=uN(t,e,t?.astNode);if(n.length!==0)return r!==void 0?r=Math.max(0,Math.min(r,n.length-1)):r=0,n[r]}function uN(t,e,r){if(t.astNode!==r)return[];if(Ho(t.grammarSource)&&t.grammarSource.value===e)return[t];let n=Kd(t).iterator(),i,a=[];do if(i=n.next(),!i.done){let s=i.value;s.astNode===r?Ho(s.grammarSource)&&s.grammarSource.value===e&&a.push(s):n.prune()}while(!i.done);return a}function hN(t){var e;let r=t.astNode;for(;r===((e=t.container)===null||e===void 0?void 0:e.astNode);){let n=tp(t.grammarSource,Ml);if(n)return n;t=t.container}}function kk(t){let e=t;return fk(e)&&(Mu(e.$container)?e=e.$container.$container:Oa(e.$container)?e=e.$container:Lc(e.$container)),Eae(t,e,new Map)}function Eae(t,e,r){var n;function i(a,s){let l;return tp(a,Ml)||(l=Eae(s,s,r)),r.set(t,l),l}if(o(i,"go"),r.has(t))return r.get(t);r.set(t,void 0);for(let a of Nc(e)){if(Ml(a)&&a.feature.toLowerCase()==="name")return r.set(t,a),a;if(Il(a)&&Oa(a.rule.ref))return i(a,a.rule.ref);if(pk(a)&&(!((n=a.typeRef)===null||n===void 0)&&n.ref))return i(a,a.typeRef.ref)}}function Sae(t){let e=t.$container;if(sf(e)){let r=e.elements,n=r.indexOf(t);for(let i=n-1;i>=0;i--){let a=r[i];if(Mu(a))return a;{let s=Nc(r[i]).find(Mu);if(s)return s}}}if(G2(e))return Sae(e)}function sPe(t,e){return t==="?"||t==="*"||sf(e)&&!!e.guardCondition}function oPe(t){return t==="*"||t==="+"}function lPe(t){return t==="+="}function Z2(t){return Cae(t,new Set)}function Cae(t,e){if(e.has(t))return!0;e.add(t);for(let r of Nc(t))if(Il(r)){if(!r.rule.ref||Oa(r.rule.ref)&&!Cae(r.rule.ref,e))return!1}else{if(Ml(r))return!1;if(Mu(r))return!1}return!!t.definition}function cPe(t){return iN(t.type,new Set)}function iN(t,e){if(e.has(t))return!0;if(e.add(t),DR(t))return!1;if(OR(t))return!1;if(BR(t))return t.types.every(r=>iN(r,e));if(pk(t)){if(t.primitiveType!==void 0)return!0;if(t.stringType!==void 0)return!0;if(t.typeRef!==void 0){let r=t.typeRef.ref;return V2(r)?iN(r.type,e):!1}else return!1}else return!1}function Rg(t){if(t.inferredType)return t.inferredType.name;if(t.dataType)return t.dataType;if(t.returnType){let e=t.returnType.ref;if(e){if(Oa(e))return e.name;if(dk(e)||V2(e))return e.name}}}function J2(t){var e;if(Oa(t))return Z2(t)?t.name:(e=Rg(t))!==null&&e!==void 0?e:t.name;if(dk(t)||V2(t)||PR(t))return t.name;if(Mu(t)){let r=Aae(t);if(r)return r}else if(fk(t))return t.name;throw new Error("Cannot get name of Unknown Type")}function Aae(t){var e;if(t.inferredType)return t.inferredType.name;if(!((e=t.type)===null||e===void 0)&&e.ref)return J2(t.type.ref)}function uPe(t){var e,r,n;return so(t)?(r=(e=t.type)===null||e===void 0?void 0:e.name)!==null&&r!==void 0?r:"string":Z2(t)?t.name:(n=Rg(t))!==null&&n!==void 0?n:t.name}function fN(t){var e,r,n;return so(t)?(r=(e=t.type)===null||e===void 0?void 0:e.name)!==null&&r!==void 0?r:"string":(n=Rg(t))!==null&&n!==void 0?n:t.name}function Ng(t){let e={s:!1,i:!1,u:!1},r=Mg(t.definition,e),n=Object.entries(e).filter(([,i])=>i).map(([i])=>i).join("");return new RegExp(r,n)}function Mg(t,e){if(VR(t))return hPe(t);if(UR(t))return fPe(t);if(FR(t))return mPe(t);if(gk(t)){let r=t.rule.ref;if(!r)throw new Error("Missing rule reference.");return Iu(Mg(r.definition),{cardinality:t.cardinality,lookahead:t.lookahead})}else{if(zR(t))return pPe(t);if(HR(t))return dPe(t);if(GR(t)){let r=t.regex.lastIndexOf("/"),n=t.regex.substring(1,r),i=t.regex.substring(r+1);return e&&(e.i=i.includes("i"),e.s=i.includes("s"),e.u=i.includes("u")),Iu(n,{cardinality:t.cardinality,lookahead:t.lookahead,wrap:!1})}else{if(WR(t))return Iu(dN,{cardinality:t.cardinality,lookahead:t.lookahead});throw new Error(`Invalid terminal element: ${t?.$type}`)}}}function hPe(t){return Iu(t.elements.map(e=>Mg(e)).join("|"),{cardinality:t.cardinality,lookahead:t.lookahead})}function fPe(t){return Iu(t.elements.map(e=>Mg(e)).join(""),{cardinality:t.cardinality,lookahead:t.lookahead})}function dPe(t){return Iu(`${dN}*?${Mg(t.terminal)}`,{cardinality:t.cardinality,lookahead:t.lookahead})}function pPe(t){return Iu(`(?!${Mg(t.terminal)})${dN}*?`,{cardinality:t.cardinality,lookahead:t.lookahead})}function mPe(t){return t.right?Iu(`[${nN(t.left)}-${nN(t.right)}]`,{cardinality:t.cardinality,lookahead:t.lookahead,wrap:!1}):Iu(nN(t.left),{cardinality:t.cardinality,lookahead:t.lookahead,wrap:!1})}function nN(t){return ap(t.value)}function Iu(t,e){var r;return(e.wrap!==!1||e.lookahead)&&(t=`(${(r=e.lookahead)!==null&&r!==void 0?r:""}${t})`),e.cardinality?`${t}${e.cardinality}`:t}var dN,Ol=N(()=>{"use strict";uk();Rc();Rl();is();Nl();Lg();o(wae,"getEntryRule");o(Tae,"getHiddenRules");o(K2,"getAllReachableRules");o(kae,"ruleDfs");o(aN,"getCrossReferenceTerminal");o(sN,"isCommentTerminal");o(oN,"findNodesForProperty");o(Q2,"findNodeForProperty");o(lN,"findNodesForPropertyInternal");o(aPe,"findNodesForKeyword");o(cN,"findNodeForKeyword");o(uN,"findNodesForKeywordInternal");o(hN,"findAssignment");o(kk,"findNameAssignment");o(Eae,"findNameAssignmentInternal");o(Sae,"getActionAtElement");o(sPe,"isOptionalCardinality");o(oPe,"isArrayCardinality");o(lPe,"isArrayOperator");o(Z2,"isDataTypeRule");o(Cae,"isDataTypeRuleInternal");o(cPe,"isDataType");o(iN,"isDataTypeInternal");o(Rg,"getExplicitRuleType");o(J2,"getTypeName");o(Aae,"getActionType");o(uPe,"getRuleTypeName");o(fN,"getRuleType");o(Ng,"terminalRegex");dN=/[\s\S]/.source;o(Mg,"abstractElementToRegex");o(hPe,"terminalAlternativesToRegex");o(fPe,"terminalGroupToRegex");o(dPe,"untilTokenToRegex");o(pPe,"negateTokenToRegex");o(mPe,"characterRangeToRegex");o(nN,"keywordToRegex");o(Iu,"withCardinality")});function pN(t){let e=[],r=t.Grammar;for(let n of r.rules)so(n)&&sN(n)&&eN(Ng(n))&&e.push(n.name);return{multilineCommentRules:e,nameRegexp:lk}}var mN=N(()=>{"use strict";Nl();Ol();Lg();Rc();o(pN,"createGrammarConfig")});var gN=N(()=>{"use strict"});function Ig(t){console&&console.error&&console.error(`Error: ${t}`)}function ex(t){console&&console.warn&&console.warn(`Warning: ${t}`)}var _ae=N(()=>{"use strict";o(Ig,"PRINT_ERROR");o(ex,"PRINT_WARNING")});function tx(t){let e=new Date().getTime(),r=t();return{time:new Date().getTime()-e,value:r}}var Dae=N(()=>{"use strict";o(tx,"timer")});function rx(t){function e(){}o(e,"FakeConstructor"),e.prototype=t;let r=new e;function n(){return typeof r.bar}return o(n,"fakeAccess"),n(),n(),t;(0,eval)(t)}var Lae=N(()=>{"use strict";o(rx,"toFastProperties")});var Og=N(()=>{"use strict";_ae();Dae();Lae()});function gPe(t){return yPe(t)?t.LABEL:t.name}function yPe(t){return yi(t.LABEL)&&t.LABEL!==""}function Sk(t){return Je(t,Pg)}function Pg(t){function e(r){return Je(r,Pg)}if(o(e,"convertDefinition"),t instanceof on){let r={type:"NonTerminal",name:t.nonTerminalName,idx:t.idx};return yi(t.label)&&(r.label=t.label),r}else{if(t instanceof Dn)return{type:"Alternative",definition:e(t.definition)};if(t instanceof ln)return{type:"Option",idx:t.idx,definition:e(t.definition)};if(t instanceof Ln)return{type:"RepetitionMandatory",idx:t.idx,definition:e(t.definition)};if(t instanceof Rn)return{type:"RepetitionMandatoryWithSeparator",idx:t.idx,separator:Pg(new kr({terminalType:t.separator})),definition:e(t.definition)};if(t instanceof wn)return{type:"RepetitionWithSeparator",idx:t.idx,separator:Pg(new kr({terminalType:t.separator})),definition:e(t.definition)};if(t instanceof Or)return{type:"Repetition",idx:t.idx,definition:e(t.definition)};if(t instanceof Tn)return{type:"Alternation",idx:t.idx,definition:e(t.definition)};if(t instanceof kr){let r={type:"Terminal",name:t.terminalType.name,label:gPe(t.terminalType),idx:t.idx};yi(t.label)&&(r.terminalLabel=t.label);let n=t.terminalType.PATTERN;return t.terminalType.PATTERN&&(r.pattern=zo(n)?n.source:n),r}else{if(t instanceof as)return{type:"Rule",name:t.name,orgText:t.orgText,definition:e(t.definition)};throw Error("non exhaustive match")}}}var oo,on,as,Dn,ln,Ln,Rn,Or,wn,Tn,kr,Ck=N(()=>{"use strict";qt();o(gPe,"tokenLabel");o(yPe,"hasTokenLabel");oo=class{static{o(this,"AbstractProduction")}get definition(){return this._definition}set definition(e){this._definition=e}constructor(e){this._definition=e}accept(e){e.visit(this),Ae(this.definition,r=>{r.accept(e)})}},on=class extends oo{static{o(this,"NonTerminal")}constructor(e){super([]),this.idx=1,ma(this,Os(e,r=>r!==void 0))}set definition(e){}get definition(){return this.referencedRule!==void 0?this.referencedRule.definition:[]}accept(e){e.visit(this)}},as=class extends oo{static{o(this,"Rule")}constructor(e){super(e.definition),this.orgText="",ma(this,Os(e,r=>r!==void 0))}},Dn=class extends oo{static{o(this,"Alternative")}constructor(e){super(e.definition),this.ignoreAmbiguities=!1,ma(this,Os(e,r=>r!==void 0))}},ln=class extends oo{static{o(this,"Option")}constructor(e){super(e.definition),this.idx=1,ma(this,Os(e,r=>r!==void 0))}},Ln=class extends oo{static{o(this,"RepetitionMandatory")}constructor(e){super(e.definition),this.idx=1,ma(this,Os(e,r=>r!==void 0))}},Rn=class extends oo{static{o(this,"RepetitionMandatoryWithSeparator")}constructor(e){super(e.definition),this.idx=1,ma(this,Os(e,r=>r!==void 0))}},Or=class extends oo{static{o(this,"Repetition")}constructor(e){super(e.definition),this.idx=1,ma(this,Os(e,r=>r!==void 0))}},wn=class extends oo{static{o(this,"RepetitionWithSeparator")}constructor(e){super(e.definition),this.idx=1,ma(this,Os(e,r=>r!==void 0))}},Tn=class extends oo{static{o(this,"Alternation")}get definition(){return this._definition}set definition(e){this._definition=e}constructor(e){super(e.definition),this.idx=1,this.ignoreAmbiguities=!1,this.hasPredicates=!1,ma(this,Os(e,r=>r!==void 0))}},kr=class{static{o(this,"Terminal")}constructor(e){this.idx=1,ma(this,Os(e,r=>r!==void 0))}accept(e){e.visit(this)}};o(Sk,"serializeGrammar");o(Pg,"serializeProduction")});var ss,Rae=N(()=>{"use strict";Ck();ss=class{static{o(this,"GAstVisitor")}visit(e){let r=e;switch(r.constructor){case on:return this.visitNonTerminal(r);case Dn:return this.visitAlternative(r);case ln:return this.visitOption(r);case Ln:return this.visitRepetitionMandatory(r);case Rn:return this.visitRepetitionMandatoryWithSeparator(r);case wn:return this.visitRepetitionWithSeparator(r);case Or:return this.visitRepetition(r);case Tn:return this.visitAlternation(r);case kr:return this.visitTerminal(r);case as:return this.visitRule(r);default:throw Error("non exhaustive match")}}visitNonTerminal(e){}visitAlternative(e){}visitOption(e){}visitRepetition(e){}visitRepetitionMandatory(e){}visitRepetitionMandatoryWithSeparator(e){}visitRepetitionWithSeparator(e){}visitAlternation(e){}visitTerminal(e){}visitRule(e){}}});function yN(t){return t instanceof Dn||t instanceof ln||t instanceof Or||t instanceof Ln||t instanceof Rn||t instanceof wn||t instanceof kr||t instanceof as}function sp(t,e=[]){return t instanceof ln||t instanceof Or||t instanceof wn?!0:t instanceof Tn?A2(t.definition,n=>sp(n,e)):t instanceof on&&qn(e,t)?!1:t instanceof oo?(t instanceof on&&e.push(t),Ma(t.definition,n=>sp(n,e))):!1}function vN(t){return t instanceof Tn}function Bs(t){if(t instanceof on)return"SUBRULE";if(t instanceof ln)return"OPTION";if(t instanceof Tn)return"OR";if(t instanceof Ln)return"AT_LEAST_ONE";if(t instanceof Rn)return"AT_LEAST_ONE_SEP";if(t instanceof wn)return"MANY_SEP";if(t instanceof Or)return"MANY";if(t instanceof kr)return"CONSUME";throw Error("non exhaustive match")}var Nae=N(()=>{"use strict";qt();Ck();o(yN,"isSequenceProd");o(sp,"isOptionalProd");o(vN,"isBranchingProd");o(Bs,"getProductionDslName")});var os=N(()=>{"use strict";Ck();Rae();Nae()});function Mae(t,e,r){return[new ln({definition:[new kr({terminalType:t.separator})].concat(t.definition)})].concat(e,r)}var Ou,Ak=N(()=>{"use strict";qt();os();Ou=class{static{o(this,"RestWalker")}walk(e,r=[]){Ae(e.definition,(n,i)=>{let a=gi(e.definition,i+1);if(n instanceof on)this.walkProdRef(n,a,r);else if(n instanceof kr)this.walkTerminal(n,a,r);else if(n instanceof Dn)this.walkFlat(n,a,r);else if(n instanceof ln)this.walkOption(n,a,r);else if(n instanceof Ln)this.walkAtLeastOne(n,a,r);else if(n instanceof Rn)this.walkAtLeastOneSep(n,a,r);else if(n instanceof wn)this.walkManySep(n,a,r);else if(n instanceof Or)this.walkMany(n,a,r);else if(n instanceof Tn)this.walkOr(n,a,r);else throw Error("non exhaustive match")})}walkTerminal(e,r,n){}walkProdRef(e,r,n){}walkFlat(e,r,n){let i=r.concat(n);this.walk(e,i)}walkOption(e,r,n){let i=r.concat(n);this.walk(e,i)}walkAtLeastOne(e,r,n){let i=[new ln({definition:e.definition})].concat(r,n);this.walk(e,i)}walkAtLeastOneSep(e,r,n){let i=Mae(e,r,n);this.walk(e,i)}walkMany(e,r,n){let i=[new ln({definition:e.definition})].concat(r,n);this.walk(e,i)}walkManySep(e,r,n){let i=Mae(e,r,n);this.walk(e,i)}walkOr(e,r,n){let i=r.concat(n);Ae(e.definition,a=>{let s=new Dn({definition:[a]});this.walk(s,i)})}};o(Mae,"restForRepetitionWithSeparator")});function op(t){if(t instanceof on)return op(t.referencedRule);if(t instanceof kr)return bPe(t);if(yN(t))return vPe(t);if(vN(t))return xPe(t);throw Error("non exhaustive match")}function vPe(t){let e=[],r=t.definition,n=0,i=r.length>n,a,s=!0;for(;i&&s;)a=r[n],s=sp(a),e=e.concat(op(a)),n=n+1,i=r.length>n;return Bm(e)}function xPe(t){let e=Je(t.definition,r=>op(r));return Bm(qr(e))}function bPe(t){return[t.terminalType]}var xN=N(()=>{"use strict";qt();os();o(op,"first");o(vPe,"firstForSequence");o(xPe,"firstForBranching");o(bPe,"firstForTerminal")});var _k,bN=N(()=>{"use strict";_k="_~IN~_"});function Iae(t){let e={};return Ae(t,r=>{let n=new wN(r).startWalking();ma(e,n)}),e}function wPe(t,e){return t.name+e+_k}var wN,Oae=N(()=>{"use strict";Ak();xN();qt();bN();os();wN=class extends Ou{static{o(this,"ResyncFollowsWalker")}constructor(e){super(),this.topProd=e,this.follows={}}startWalking(){return this.walk(this.topProd),this.follows}walkTerminal(e,r,n){}walkProdRef(e,r,n){let i=wPe(e.referencedRule,e.idx)+this.topProd.name,a=r.concat(n),s=new Dn({definition:a}),l=op(s);this.follows[i]=l}};o(Iae,"computeAllProdsFollows");o(wPe,"buildBetweenProdsFollowPrefix")});function Bg(t){let e=t.toString();if(Dk.hasOwnProperty(e))return Dk[e];{let r=TPe.pattern(e);return Dk[e]=r,r}}function Pae(){Dk={}}var Dk,TPe,Lk=N(()=>{"use strict";j2();Dk={},TPe=new np;o(Bg,"getRegExpAst");o(Pae,"clearRegExpParserCache")});function $ae(t,e=!1){try{let r=Bg(t);return TN(r.value,{},r.flags.ignoreCase)}catch(r){if(r.message===Fae)e&&ex(`${nx} Unable to optimize: < ${t.toString()} > + Complement Sets cannot be automatically optimized. + This will disable the lexer's first char optimizations. + See: https://chevrotain.io/docs/guide/resolving_lexer_errors.html#COMPLEMENT for details.`);else{let n="";e&&(n=` + This will disable the lexer's first char optimizations. + See: https://chevrotain.io/docs/guide/resolving_lexer_errors.html#REGEXP_PARSING for details.`),Ig(`${nx} + Failed parsing: < ${t.toString()} > + Using the @chevrotain/regexp-to-ast library + Please open an issue at: https://github.com/chevrotain/chevrotain/issues`+n)}}return[]}function TN(t,e,r){switch(t.type){case"Disjunction":for(let i=0;i{if(typeof u=="number")Rk(u,e,r);else{let h=u;if(r===!0)for(let f=h.from;f<=h.to;f++)Rk(f,e,r);else{for(let f=h.from;f<=h.to&&f=Fg){let f=h.from>=Fg?h.from:Fg,d=h.to,p=Ic(f),m=Ic(d);for(let g=p;g<=m;g++)e[g]=g}}}});break;case"Group":TN(s.value,e,r);break;default:throw Error("Non Exhaustive Match")}let l=s.quantifier!==void 0&&s.quantifier.atLeast===0;if(s.type==="Group"&&kN(s)===!1||s.type!=="Group"&&l===!1)break}break;default:throw Error("non exhaustive match!")}return br(e)}function Rk(t,e,r){let n=Ic(t);e[n]=n,r===!0&&kPe(t,e)}function kPe(t,e){let r=String.fromCharCode(t),n=r.toUpperCase();if(n!==r){let i=Ic(n.charCodeAt(0));e[i]=i}else{let i=r.toLowerCase();if(i!==r){let a=Ic(i.charCodeAt(0));e[a]=a}}}function Bae(t,e){return ns(t.value,r=>{if(typeof r=="number")return qn(e,r);{let n=r;return ns(e,i=>n.from<=i&&i<=n.to)!==void 0}})}function kN(t){let e=t.quantifier;return e&&e.atLeast===0?!0:t.value?Pt(t.value)?Ma(t.value,kN):kN(t.value):!1}function Nk(t,e){if(e instanceof RegExp){let r=Bg(e),n=new EN(t);return n.visit(r),n.found}else return ns(e,r=>qn(t,r.charCodeAt(0)))!==void 0}var Fae,nx,EN,zae=N(()=>{"use strict";j2();qt();Og();Lk();SN();Fae="Complement Sets are not supported for first char optimization",nx=`Unable to use "first char" lexer optimizations: +`;o($ae,"getOptimizedStartCodesIndices");o(TN,"firstCharOptimizedIndices");o(Rk,"addOptimizedIdxToResult");o(kPe,"handleIgnoreCase");o(Bae,"findCode");o(kN,"isWholeOptional");EN=class extends Mc{static{o(this,"CharCodeFinder")}constructor(e){super(),this.targetCharCodes=e,this.found=!1}visitChildren(e){if(this.found!==!0){switch(e.type){case"Lookahead":this.visitLookahead(e);return;case"NegativeLookahead":this.visitNegativeLookahead(e);return}super.visitChildren(e)}}visitCharacter(e){qn(this.targetCharCodes,e.value)&&(this.found=!0)}visitSet(e){e.complement?Bae(e,this.targetCharCodes)===void 0&&(this.found=!0):Bae(e,this.targetCharCodes)!==void 0&&(this.found=!0)}};o(Nk,"canMatchCharCode")});function Uae(t,e){e=Qh(e,{useSticky:AN,debug:!1,safeMode:!1,positionTracking:"full",lineTerminatorCharacters:["\r",` +`],tracer:o((b,w)=>w(),"tracer")});let r=e.tracer;r("initCharCodeToOptimizedIndexMap",()=>{GPe()});let n;r("Reject Lexer.NA",()=>{n=Jh(t,b=>b[lp]===Xn.NA)});let i=!1,a;r("Transform Patterns",()=>{i=!1,a=Je(n,b=>{let w=b[lp];if(zo(w)){let C=w.source;return C.length===1&&C!=="^"&&C!=="$"&&C!=="."&&!w.ignoreCase?C:C.length===2&&C[0]==="\\"&&!qn(["d","D","s","S","t","r","n","t","0","c","b","B","f","v","w","W"],C[1])?C[1]:e.useSticky?Vae(w):Gae(w)}else{if(Si(w))return i=!0,{exec:w};if(typeof w=="object")return i=!0,w;if(typeof w=="string"){if(w.length===1)return w;{let C=w.replace(/[\\^$.*+?()[\]{}|]/g,"\\$&"),T=new RegExp(C);return e.useSticky?Vae(T):Gae(T)}}else throw Error("non exhaustive match")}})});let s,l,u,h,f;r("misc mapping",()=>{s=Je(n,b=>b.tokenTypeIdx),l=Je(n,b=>{let w=b.GROUP;if(w!==Xn.SKIPPED){if(yi(w))return w;if(pr(w))return!1;throw Error("non exhaustive match")}}),u=Je(n,b=>{let w=b.LONGER_ALT;if(w)return Pt(w)?Je(w,T=>UT(n,T)):[UT(n,w)]}),h=Je(n,b=>b.PUSH_MODE),f=Je(n,b=>Bt(b,"POP_MODE"))});let d;r("Line Terminator Handling",()=>{let b=Qae(e.lineTerminatorCharacters);d=Je(n,w=>!1),e.positionTracking!=="onlyOffset"&&(d=Je(n,w=>Bt(w,"LINE_BREAKS")?!!w.LINE_BREAKS:Kae(w,b)===!1&&Nk(b,w.PATTERN)))});let p,m,g,y;r("Misc Mapping #2",()=>{p=Je(n,Xae),m=Je(a,$Pe),g=Xr(n,(b,w)=>{let C=w.GROUP;return yi(C)&&C!==Xn.SKIPPED&&(b[C]=[]),b},{}),y=Je(a,(b,w)=>({pattern:a[w],longerAlt:u[w],canLineTerminator:d[w],isCustom:p[w],short:m[w],group:l[w],push:h[w],pop:f[w],tokenTypeIdx:s[w],tokenType:n[w]}))});let v=!0,x=[];return e.safeMode||r("First Char Optimization",()=>{x=Xr(n,(b,w,C)=>{if(typeof w.PATTERN=="string"){let T=w.PATTERN.charCodeAt(0),E=Ic(T);CN(b,E,y[C])}else if(Pt(w.START_CHARS_HINT)){let T;Ae(w.START_CHARS_HINT,E=>{let A=typeof E=="string"?E.charCodeAt(0):E,S=Ic(A);T!==S&&(T=S,CN(b,S,y[C]))})}else if(zo(w.PATTERN))if(w.PATTERN.unicode)v=!1,e.ensureOptimizations&&Ig(`${nx} Unable to analyze < ${w.PATTERN.toString()} > pattern. + The regexp unicode flag is not currently supported by the regexp-to-ast library. + This will disable the lexer's first char optimizations. + For details See: https://chevrotain.io/docs/guide/resolving_lexer_errors.html#UNICODE_OPTIMIZE`);else{let T=$ae(w.PATTERN,e.ensureOptimizations);ur(T)&&(v=!1),Ae(T,E=>{CN(b,E,y[C])})}else e.ensureOptimizations&&Ig(`${nx} TokenType: <${w.name}> is using a custom token pattern without providing parameter. + This will disable the lexer's first char optimizations. + For details See: https://chevrotain.io/docs/guide/resolving_lexer_errors.html#CUSTOM_OPTIMIZE`),v=!1;return b},[])}),{emptyGroups:g,patternIdxToConfig:y,charCodeToPatternIdxToConfig:x,hasCustom:i,canBeOptimized:v}}function Hae(t,e){let r=[],n=SPe(t);r=r.concat(n.errors);let i=CPe(n.valid),a=i.valid;return r=r.concat(i.errors),r=r.concat(EPe(a)),r=r.concat(IPe(a)),r=r.concat(OPe(a,e)),r=r.concat(PPe(a)),r}function EPe(t){let e=[],r=Yr(t,n=>zo(n[lp]));return e=e.concat(_Pe(r)),e=e.concat(RPe(r)),e=e.concat(NPe(r)),e=e.concat(MPe(r)),e=e.concat(DPe(r)),e}function SPe(t){let e=Yr(t,i=>!Bt(i,lp)),r=Je(e,i=>({message:"Token Type: ->"+i.name+"<- missing static 'PATTERN' property",type:Yn.MISSING_PATTERN,tokenTypes:[i]})),n=Zh(t,e);return{errors:r,valid:n}}function CPe(t){let e=Yr(t,i=>{let a=i[lp];return!zo(a)&&!Si(a)&&!Bt(a,"exec")&&!yi(a)}),r=Je(e,i=>({message:"Token Type: ->"+i.name+"<- static 'PATTERN' can only be a RegExp, a Function matching the {CustomPatternMatcherFunc} type or an Object matching the {ICustomPattern} interface.",type:Yn.INVALID_PATTERN,tokenTypes:[i]})),n=Zh(t,e);return{errors:r,valid:n}}function _Pe(t){class e extends Mc{static{o(this,"EndAnchorFinder")}constructor(){super(...arguments),this.found=!1}visitEndAnchor(a){this.found=!0}}let r=Yr(t,i=>{let a=i.PATTERN;try{let s=Bg(a),l=new e;return l.visit(s),l.found}catch{return APe.test(a.source)}});return Je(r,i=>({message:`Unexpected RegExp Anchor Error: + Token Type: ->`+i.name+`<- static 'PATTERN' cannot contain end of input anchor '$' + See chevrotain.io/docs/guide/resolving_lexer_errors.html#ANCHORS for details.`,type:Yn.EOI_ANCHOR_FOUND,tokenTypes:[i]}))}function DPe(t){let e=Yr(t,n=>n.PATTERN.test(""));return Je(e,n=>({message:"Token Type: ->"+n.name+"<- static 'PATTERN' must not match an empty string",type:Yn.EMPTY_MATCH_PATTERN,tokenTypes:[n]}))}function RPe(t){class e extends Mc{static{o(this,"StartAnchorFinder")}constructor(){super(...arguments),this.found=!1}visitStartAnchor(a){this.found=!0}}let r=Yr(t,i=>{let a=i.PATTERN;try{let s=Bg(a),l=new e;return l.visit(s),l.found}catch{return LPe.test(a.source)}});return Je(r,i=>({message:`Unexpected RegExp Anchor Error: + Token Type: ->`+i.name+`<- static 'PATTERN' cannot contain start of input anchor '^' + See https://chevrotain.io/docs/guide/resolving_lexer_errors.html#ANCHORS for details.`,type:Yn.SOI_ANCHOR_FOUND,tokenTypes:[i]}))}function NPe(t){let e=Yr(t,n=>{let i=n[lp];return i instanceof RegExp&&(i.multiline||i.global)});return Je(e,n=>({message:"Token Type: ->"+n.name+"<- static 'PATTERN' may NOT contain global('g') or multiline('m')",type:Yn.UNSUPPORTED_FLAGS_FOUND,tokenTypes:[n]}))}function MPe(t){let e=[],r=Je(t,a=>Xr(t,(s,l)=>(a.PATTERN.source===l.PATTERN.source&&!qn(e,l)&&l.PATTERN!==Xn.NA&&(e.push(l),s.push(l)),s),[]));r=Tc(r);let n=Yr(r,a=>a.length>1);return Je(n,a=>{let s=Je(a,u=>u.name);return{message:`The same RegExp pattern ->${ia(a).PATTERN}<-has been used in all of the following Token Types: ${s.join(", ")} <-`,type:Yn.DUPLICATE_PATTERNS_FOUND,tokenTypes:a}})}function IPe(t){let e=Yr(t,n=>{if(!Bt(n,"GROUP"))return!1;let i=n.GROUP;return i!==Xn.SKIPPED&&i!==Xn.NA&&!yi(i)});return Je(e,n=>({message:"Token Type: ->"+n.name+"<- static 'GROUP' can only be Lexer.SKIPPED/Lexer.NA/A String",type:Yn.INVALID_GROUP_TYPE_FOUND,tokenTypes:[n]}))}function OPe(t,e){let r=Yr(t,i=>i.PUSH_MODE!==void 0&&!qn(e,i.PUSH_MODE));return Je(r,i=>({message:`Token Type: ->${i.name}<- static 'PUSH_MODE' value cannot refer to a Lexer Mode ->${i.PUSH_MODE}<-which does not exist`,type:Yn.PUSH_MODE_DOES_NOT_EXIST,tokenTypes:[i]}))}function PPe(t){let e=[],r=Xr(t,(n,i,a)=>{let s=i.PATTERN;return s===Xn.NA||(yi(s)?n.push({str:s,idx:a,tokenType:i}):zo(s)&&FPe(s)&&n.push({str:s.source,idx:a,tokenType:i})),n},[]);return Ae(t,(n,i)=>{Ae(r,({str:a,idx:s,tokenType:l})=>{if(i${l.name}<- can never be matched. +Because it appears AFTER the Token Type ->${n.name}<-in the lexer's definition. +See https://chevrotain.io/docs/guide/resolving_lexer_errors.html#UNREACHABLE`;e.push({message:u,type:Yn.UNREACHABLE_PATTERN,tokenTypes:[n,l]})}})}),e}function BPe(t,e){if(zo(e)){let r=e.exec(t);return r!==null&&r.index===0}else{if(Si(e))return e(t,0,[],{});if(Bt(e,"exec"))return e.exec(t,0,[],{});if(typeof e=="string")return e===t;throw Error("non exhaustive match")}}function FPe(t){return ns([".","\\","[","]","|","^","$","(",")","?","*","+","{"],r=>t.source.indexOf(r)!==-1)===void 0}function Gae(t){let e=t.ignoreCase?"i":"";return new RegExp(`^(?:${t.source})`,e)}function Vae(t){let e=t.ignoreCase?"iy":"y";return new RegExp(`${t.source}`,e)}function Wae(t,e,r){let n=[];return Bt(t,$g)||n.push({message:"A MultiMode Lexer cannot be initialized without a <"+$g+`> property in its definition +`,type:Yn.MULTI_MODE_LEXER_WITHOUT_DEFAULT_MODE}),Bt(t,Mk)||n.push({message:"A MultiMode Lexer cannot be initialized without a <"+Mk+`> property in its definition +`,type:Yn.MULTI_MODE_LEXER_WITHOUT_MODES_PROPERTY}),Bt(t,Mk)&&Bt(t,$g)&&!Bt(t.modes,t.defaultMode)&&n.push({message:`A MultiMode Lexer cannot be initialized with a ${$g}: <${t.defaultMode}>which does not exist +`,type:Yn.MULTI_MODE_LEXER_DEFAULT_MODE_VALUE_DOES_NOT_EXIST}),Bt(t,Mk)&&Ae(t.modes,(i,a)=>{Ae(i,(s,l)=>{if(pr(s))n.push({message:`A Lexer cannot be initialized using an undefined Token Type. Mode:<${a}> at index: <${l}> +`,type:Yn.LEXER_DEFINITION_CANNOT_CONTAIN_UNDEFINED});else if(Bt(s,"LONGER_ALT")){let u=Pt(s.LONGER_ALT)?s.LONGER_ALT:[s.LONGER_ALT];Ae(u,h=>{!pr(h)&&!qn(i,h)&&n.push({message:`A MultiMode Lexer cannot be initialized with a longer_alt <${h.name}> on token <${s.name}> outside of mode <${a}> +`,type:Yn.MULTI_MODE_LEXER_LONGER_ALT_NOT_IN_CURRENT_MODE})})}})}),n}function qae(t,e,r){let n=[],i=!1,a=Tc(qr(br(t.modes))),s=Jh(a,u=>u[lp]===Xn.NA),l=Qae(r);return e&&Ae(s,u=>{let h=Kae(u,l);if(h!==!1){let d={message:zPe(u,h),type:h.issue,tokenType:u};n.push(d)}else Bt(u,"LINE_BREAKS")?u.LINE_BREAKS===!0&&(i=!0):Nk(l,u.PATTERN)&&(i=!0)}),e&&!i&&n.push({message:`Warning: No LINE_BREAKS Found. + This Lexer has been defined to track line and column information, + But none of the Token Types can be identified as matching a line terminator. + See https://chevrotain.io/docs/guide/resolving_lexer_errors.html#LINE_BREAKS + for details.`,type:Yn.NO_LINE_BREAKS_FLAGS}),n}function Yae(t){let e={},r=zr(t);return Ae(r,n=>{let i=t[n];if(Pt(i))e[n]=[];else throw Error("non exhaustive match")}),e}function Xae(t){let e=t.PATTERN;if(zo(e))return!1;if(Si(e))return!0;if(Bt(e,"exec"))return!0;if(yi(e))return!1;throw Error("non exhaustive match")}function $Pe(t){return yi(t)&&t.length===1?t.charCodeAt(0):!1}function Kae(t,e){if(Bt(t,"LINE_BREAKS"))return!1;if(zo(t.PATTERN)){try{Nk(e,t.PATTERN)}catch(r){return{issue:Yn.IDENTIFY_TERMINATOR,errMsg:r.message}}return!1}else{if(yi(t.PATTERN))return!1;if(Xae(t))return{issue:Yn.CUSTOM_LINE_BREAK};throw Error("non exhaustive match")}}function zPe(t,e){if(e.issue===Yn.IDENTIFY_TERMINATOR)return`Warning: unable to identify line terminator usage in pattern. + The problem is in the <${t.name}> Token Type + Root cause: ${e.errMsg}. + For details See: https://chevrotain.io/docs/guide/resolving_lexer_errors.html#IDENTIFY_TERMINATOR`;if(e.issue===Yn.CUSTOM_LINE_BREAK)return`Warning: A Custom Token Pattern should specify the option. + The problem is in the <${t.name}> Token Type + For details See: https://chevrotain.io/docs/guide/resolving_lexer_errors.html#CUSTOM_LINE_BREAK`;throw Error("non exhaustive match")}function Qae(t){return Je(t,r=>yi(r)?r.charCodeAt(0):r)}function CN(t,e,r){t[e]===void 0?t[e]=[r]:t[e].push(r)}function Ic(t){return t255?255+~~(t/255):t}}var lp,$g,Mk,AN,APe,LPe,jae,Fg,Ik,SN=N(()=>{"use strict";j2();ix();qt();Og();zae();Lk();lp="PATTERN",$g="defaultMode",Mk="modes",AN=typeof new RegExp("(?:)").sticky=="boolean";o(Uae,"analyzeTokenTypes");o(Hae,"validatePatterns");o(EPe,"validateRegExpPattern");o(SPe,"findMissingPatterns");o(CPe,"findInvalidPatterns");APe=/[^\\][$]/;o(_Pe,"findEndOfInputAnchor");o(DPe,"findEmptyMatchRegExps");LPe=/[^\\[][\^]|^\^/;o(RPe,"findStartOfInputAnchor");o(NPe,"findUnsupportedFlags");o(MPe,"findDuplicatePatterns");o(IPe,"findInvalidGroupType");o(OPe,"findModesThatDoNotExist");o(PPe,"findUnreachablePatterns");o(BPe,"testTokenType");o(FPe,"noMetaChar");o(Gae,"addStartOfInput");o(Vae,"addStickyFlag");o(Wae,"performRuntimeChecks");o(qae,"performWarningRuntimeChecks");o(Yae,"cloneEmptyGroups");o(Xae,"isCustomPattern");o($Pe,"isShortPattern");jae={test:o(function(t){let e=t.length;for(let r=this.lastIndex;r{r.isParent=r.categoryMatches.length>0})}function VPe(t){let e=an(t),r=t,n=!0;for(;n;){r=Tc(qr(Je(r,a=>a.CATEGORIES)));let i=Zh(r,e);e=e.concat(i),ur(i)?n=!1:r=i}return e}function UPe(t){Ae(t,e=>{_N(e)||(ese[Zae]=e,e.tokenTypeIdx=Zae++),Jae(e)&&!Pt(e.CATEGORIES)&&(e.CATEGORIES=[e.CATEGORIES]),Jae(e)||(e.CATEGORIES=[]),qPe(e)||(e.categoryMatches=[]),YPe(e)||(e.categoryMatchesMap={})})}function HPe(t){Ae(t,e=>{e.categoryMatches=[],Ae(e.categoryMatchesMap,(r,n)=>{e.categoryMatches.push(ese[n].tokenTypeIdx)})})}function WPe(t){Ae(t,e=>{tse([],e)})}function tse(t,e){Ae(t,r=>{e.categoryMatchesMap[r.tokenTypeIdx]=!0}),Ae(e.CATEGORIES,r=>{let n=t.concat(e);qn(n,r)||tse(n,r)})}function _N(t){return Bt(t,"tokenTypeIdx")}function Jae(t){return Bt(t,"CATEGORIES")}function qPe(t){return Bt(t,"categoryMatches")}function YPe(t){return Bt(t,"categoryMatchesMap")}function rse(t){return Bt(t,"tokenTypeIdx")}var Zae,ese,cp=N(()=>{"use strict";qt();o(Pu,"tokenStructuredMatcher");o(zg,"tokenStructuredMatcherNoCategories");Zae=1,ese={};o(Bu,"augmentTokenTypes");o(VPe,"expandCategories");o(UPe,"assignTokenDefaultProps");o(HPe,"assignCategoriesTokensProp");o(WPe,"assignCategoriesMapProp");o(tse,"singleAssignCategoriesToksMap");o(_N,"hasShortKeyProperty");o(Jae,"hasCategoriesProperty");o(qPe,"hasExtendingTokensTypesProperty");o(YPe,"hasExtendingTokensTypesMapProperty");o(rse,"isTokenType")});var Gg,DN=N(()=>{"use strict";Gg={buildUnableToPopLexerModeMessage(t){return`Unable to pop Lexer Mode after encountering Token ->${t.image}<- The Mode Stack is empty`},buildUnexpectedCharactersMessage(t,e,r,n,i){return`unexpected character: ->${t.charAt(e)}<- at offset: ${e}, skipped ${r} characters.`}}});var Yn,ax,Xn,ix=N(()=>{"use strict";SN();qt();Og();cp();DN();Lk();(function(t){t[t.MISSING_PATTERN=0]="MISSING_PATTERN",t[t.INVALID_PATTERN=1]="INVALID_PATTERN",t[t.EOI_ANCHOR_FOUND=2]="EOI_ANCHOR_FOUND",t[t.UNSUPPORTED_FLAGS_FOUND=3]="UNSUPPORTED_FLAGS_FOUND",t[t.DUPLICATE_PATTERNS_FOUND=4]="DUPLICATE_PATTERNS_FOUND",t[t.INVALID_GROUP_TYPE_FOUND=5]="INVALID_GROUP_TYPE_FOUND",t[t.PUSH_MODE_DOES_NOT_EXIST=6]="PUSH_MODE_DOES_NOT_EXIST",t[t.MULTI_MODE_LEXER_WITHOUT_DEFAULT_MODE=7]="MULTI_MODE_LEXER_WITHOUT_DEFAULT_MODE",t[t.MULTI_MODE_LEXER_WITHOUT_MODES_PROPERTY=8]="MULTI_MODE_LEXER_WITHOUT_MODES_PROPERTY",t[t.MULTI_MODE_LEXER_DEFAULT_MODE_VALUE_DOES_NOT_EXIST=9]="MULTI_MODE_LEXER_DEFAULT_MODE_VALUE_DOES_NOT_EXIST",t[t.LEXER_DEFINITION_CANNOT_CONTAIN_UNDEFINED=10]="LEXER_DEFINITION_CANNOT_CONTAIN_UNDEFINED",t[t.SOI_ANCHOR_FOUND=11]="SOI_ANCHOR_FOUND",t[t.EMPTY_MATCH_PATTERN=12]="EMPTY_MATCH_PATTERN",t[t.NO_LINE_BREAKS_FLAGS=13]="NO_LINE_BREAKS_FLAGS",t[t.UNREACHABLE_PATTERN=14]="UNREACHABLE_PATTERN",t[t.IDENTIFY_TERMINATOR=15]="IDENTIFY_TERMINATOR",t[t.CUSTOM_LINE_BREAK=16]="CUSTOM_LINE_BREAK",t[t.MULTI_MODE_LEXER_LONGER_ALT_NOT_IN_CURRENT_MODE=17]="MULTI_MODE_LEXER_LONGER_ALT_NOT_IN_CURRENT_MODE"})(Yn||(Yn={}));ax={deferDefinitionErrorsHandling:!1,positionTracking:"full",lineTerminatorsPattern:/\n|\r\n?/g,lineTerminatorCharacters:[` +`,"\r"],ensureOptimizations:!1,safeMode:!1,errorMessageProvider:Gg,traceInitPerf:!1,skipValidations:!1,recoveryEnabled:!0};Object.freeze(ax);Xn=class{static{o(this,"Lexer")}constructor(e,r=ax){if(this.lexerDefinition=e,this.lexerDefinitionErrors=[],this.lexerDefinitionWarning=[],this.patternIdxToConfig={},this.charCodeToPatternIdxToConfig={},this.modes=[],this.emptyGroups={},this.trackStartLines=!0,this.trackEndLines=!0,this.hasCustom=!1,this.canModeBeOptimized={},this.TRACE_INIT=(i,a)=>{if(this.traceInitPerf===!0){this.traceInitIndent++;let s=new Array(this.traceInitIndent+1).join(" ");this.traceInitIndent <${i}>`);let{time:l,value:u}=tx(a),h=l>10?console.warn:console.log;return this.traceInitIndent time: ${l}ms`),this.traceInitIndent--,u}else return a()},typeof r=="boolean")throw Error(`The second argument to the Lexer constructor is now an ILexerConfig Object. +a boolean 2nd argument is no longer supported`);this.config=ma({},ax,r);let n=this.config.traceInitPerf;n===!0?(this.traceInitMaxIdent=1/0,this.traceInitPerf=!0):typeof n=="number"&&(this.traceInitMaxIdent=n,this.traceInitPerf=!0),this.traceInitIndent=-1,this.TRACE_INIT("Lexer Constructor",()=>{let i,a=!0;this.TRACE_INIT("Lexer Config handling",()=>{if(this.config.lineTerminatorsPattern===ax.lineTerminatorsPattern)this.config.lineTerminatorsPattern=jae;else if(this.config.lineTerminatorCharacters===ax.lineTerminatorCharacters)throw Error(`Error: Missing property on the Lexer config. + For details See: https://chevrotain.io/docs/guide/resolving_lexer_errors.html#MISSING_LINE_TERM_CHARS`);if(r.safeMode&&r.ensureOptimizations)throw Error('"safeMode" and "ensureOptimizations" flags are mutually exclusive.');this.trackStartLines=/full|onlyStart/i.test(this.config.positionTracking),this.trackEndLines=/full/i.test(this.config.positionTracking),Pt(e)?i={modes:{defaultMode:an(e)},defaultMode:$g}:(a=!1,i=an(e))}),this.config.skipValidations===!1&&(this.TRACE_INIT("performRuntimeChecks",()=>{this.lexerDefinitionErrors=this.lexerDefinitionErrors.concat(Wae(i,this.trackStartLines,this.config.lineTerminatorCharacters))}),this.TRACE_INIT("performWarningRuntimeChecks",()=>{this.lexerDefinitionWarning=this.lexerDefinitionWarning.concat(qae(i,this.trackStartLines,this.config.lineTerminatorCharacters))})),i.modes=i.modes?i.modes:{},Ae(i.modes,(l,u)=>{i.modes[u]=Jh(l,h=>pr(h))});let s=zr(i.modes);if(Ae(i.modes,(l,u)=>{this.TRACE_INIT(`Mode: <${u}> processing`,()=>{if(this.modes.push(u),this.config.skipValidations===!1&&this.TRACE_INIT("validatePatterns",()=>{this.lexerDefinitionErrors=this.lexerDefinitionErrors.concat(Hae(l,s))}),ur(this.lexerDefinitionErrors)){Bu(l);let h;this.TRACE_INIT("analyzeTokenTypes",()=>{h=Uae(l,{lineTerminatorCharacters:this.config.lineTerminatorCharacters,positionTracking:r.positionTracking,ensureOptimizations:r.ensureOptimizations,safeMode:r.safeMode,tracer:this.TRACE_INIT})}),this.patternIdxToConfig[u]=h.patternIdxToConfig,this.charCodeToPatternIdxToConfig[u]=h.charCodeToPatternIdxToConfig,this.emptyGroups=ma({},this.emptyGroups,h.emptyGroups),this.hasCustom=h.hasCustom||this.hasCustom,this.canModeBeOptimized[u]=h.canBeOptimized}})}),this.defaultMode=i.defaultMode,!ur(this.lexerDefinitionErrors)&&!this.config.deferDefinitionErrorsHandling){let u=Je(this.lexerDefinitionErrors,h=>h.message).join(`----------------------- +`);throw new Error(`Errors detected in definition of Lexer: +`+u)}Ae(this.lexerDefinitionWarning,l=>{ex(l.message)}),this.TRACE_INIT("Choosing sub-methods implementations",()=>{if(AN?(this.chopInput=ta,this.match=this.matchWithTest):(this.updateLastIndex=ni,this.match=this.matchWithExec),a&&(this.handleModes=ni),this.trackStartLines===!1&&(this.computeNewColumn=ta),this.trackEndLines===!1&&(this.updateTokenEndLineColumnLocation=ni),/full/i.test(this.config.positionTracking))this.createTokenInstance=this.createFullToken;else if(/onlyStart/i.test(this.config.positionTracking))this.createTokenInstance=this.createStartOnlyToken;else if(/onlyOffset/i.test(this.config.positionTracking))this.createTokenInstance=this.createOffsetOnlyToken;else throw Error(`Invalid config option: "${this.config.positionTracking}"`);this.hasCustom?(this.addToken=this.addTokenUsingPush,this.handlePayload=this.handlePayloadWithCustom):(this.addToken=this.addTokenUsingMemberAccess,this.handlePayload=this.handlePayloadNoCustom)}),this.TRACE_INIT("Failed Optimization Warnings",()=>{let l=Xr(this.canModeBeOptimized,(u,h,f)=>(h===!1&&u.push(f),u),[]);if(r.ensureOptimizations&&!ur(l))throw Error(`Lexer Modes: < ${l.join(", ")} > cannot be optimized. + Disable the "ensureOptimizations" lexer config flag to silently ignore this and run the lexer in an un-optimized mode. + Or inspect the console log for details on how to resolve these issues.`)}),this.TRACE_INIT("clearRegExpParserCache",()=>{Pae()}),this.TRACE_INIT("toFastProperties",()=>{rx(this)})})}tokenize(e,r=this.defaultMode){if(!ur(this.lexerDefinitionErrors)){let i=Je(this.lexerDefinitionErrors,a=>a.message).join(`----------------------- +`);throw new Error(`Unable to Tokenize because Errors detected in definition of Lexer: +`+i)}return this.tokenizeInternal(e,r)}tokenizeInternal(e,r){let n,i,a,s,l,u,h,f,d,p,m,g,y,v,x,b,w=e,C=w.length,T=0,E=0,A=this.hasCustom?0:Math.floor(e.length/10),S=new Array(A),_=[],I=this.trackStartLines?1:void 0,D=this.trackStartLines?1:void 0,k=Yae(this.emptyGroups),L=this.trackStartLines,R=this.config.lineTerminatorsPattern,O=0,M=[],B=[],F=[],P=[];Object.freeze(P);let z;function $(){return M}o($,"getPossiblePatternsSlow");function H(le){let he=Ic(le),K=B[he];return K===void 0?P:K}o(H,"getPossiblePatternsOptimized");let Q=o(le=>{if(F.length===1&&le.tokenType.PUSH_MODE===void 0){let he=this.config.errorMessageProvider.buildUnableToPopLexerModeMessage(le);_.push({offset:le.startOffset,line:le.startLine,column:le.startColumn,length:le.image.length,message:he})}else{F.pop();let he=ga(F);M=this.patternIdxToConfig[he],B=this.charCodeToPatternIdxToConfig[he],O=M.length;let K=this.canModeBeOptimized[he]&&this.config.safeMode===!1;B&&K?z=H:z=$}},"pop_mode");function j(le){F.push(le),B=this.charCodeToPatternIdxToConfig[le],M=this.patternIdxToConfig[le],O=M.length,O=M.length;let he=this.canModeBeOptimized[le]&&this.config.safeMode===!1;B&&he?z=H:z=$}o(j,"push_mode"),j.call(this,r);let ie,ne=this.config.recoveryEnabled;for(;Tu.length){u=s,h=f,ie=se;break}}}break}}if(u!==null){if(d=u.length,p=ie.group,p!==void 0&&(m=ie.tokenTypeIdx,g=this.createTokenInstance(u,T,m,ie.tokenType,I,D,d),this.handlePayload(g,h),p===!1?E=this.addToken(S,E,g):k[p].push(g)),e=this.chopInput(e,d),T=T+d,D=this.computeNewColumn(D,d),L===!0&&ie.canLineTerminator===!0){let X=0,te,J;R.lastIndex=0;do te=R.test(u),te===!0&&(J=R.lastIndex-1,X++);while(te===!0);X!==0&&(I=I+X,D=d-J,this.updateTokenEndLineColumnLocation(g,p,J,X,I,D,d))}this.handleModes(ie,Q,j,g)}else{let X=T,te=I,J=D,se=ne===!1;for(;se===!1&&T{"use strict";qt();ix();cp();o(Fu,"tokenLabel");o(LN,"hasTokenLabel");XPe="parent",nse="categories",ise="label",ase="group",sse="push_mode",ose="pop_mode",lse="longer_alt",cse="line_breaks",use="start_chars_hint";o(of,"createToken");o(jPe,"createTokenInternal");lo=of({name:"EOF",pattern:Xn.NA});Bu([lo]);o($u,"createTokenInstance");o(sx,"tokenMatcher")});var zu,hse,Pl,Vg=N(()=>{"use strict";up();qt();os();zu={buildMismatchTokenMessage({expected:t,actual:e,previous:r,ruleName:n}){return`Expecting ${LN(t)?`--> ${Fu(t)} <--`:`token of type --> ${t.name} <--`} but found --> '${e.image}' <--`},buildNotAllInputParsedMessage({firstRedundant:t,ruleName:e}){return"Redundant input, expecting EOF but found: "+t.image},buildNoViableAltMessage({expectedPathsPerAlt:t,actual:e,previous:r,customUserDescription:n,ruleName:i}){let a="Expecting: ",l=` +but found: '`+ia(e).image+"'";if(n)return a+n+l;{let u=Xr(t,(p,m)=>p.concat(m),[]),h=Je(u,p=>`[${Je(p,m=>Fu(m)).join(", ")}]`),d=`one of these possible Token sequences: +${Je(h,(p,m)=>` ${m+1}. ${p}`).join(` +`)}`;return a+d+l}},buildEarlyExitMessage({expectedIterationPaths:t,actual:e,customUserDescription:r,ruleName:n}){let i="Expecting: ",s=` +but found: '`+ia(e).image+"'";if(r)return i+r+s;{let u=`expecting at least one iteration which starts with one of these possible Token sequences:: + <${Je(t,h=>`[${Je(h,f=>Fu(f)).join(",")}]`).join(" ,")}>`;return i+u+s}}};Object.freeze(zu);hse={buildRuleNotFoundError(t,e){return"Invalid grammar, reference to a rule which is not defined: ->"+e.nonTerminalName+`<- +inside top level rule: ->`+t.name+"<-"}},Pl={buildDuplicateFoundError(t,e){function r(f){return f instanceof kr?f.terminalType.name:f instanceof on?f.nonTerminalName:""}o(r,"getExtraProductionArgument");let n=t.name,i=ia(e),a=i.idx,s=Bs(i),l=r(i),u=a>0,h=`->${s}${u?a:""}<- ${l?`with argument: ->${l}<-`:""} + appears more than once (${e.length} times) in the top level rule: ->${n}<-. + For further details see: https://chevrotain.io/docs/FAQ.html#NUMERICAL_SUFFIXES + `;return h=h.replace(/[ \t]+/g," "),h=h.replace(/\s\s+/g,` +`),h},buildNamespaceConflictError(t){return`Namespace conflict found in grammar. +The grammar has both a Terminal(Token) and a Non-Terminal(Rule) named: <${t.name}>. +To resolve this make sure each Terminal and Non-Terminal names are unique +This is easy to accomplish by using the convention that Terminal names start with an uppercase letter +and Non-Terminal names start with a lower case letter.`},buildAlternationPrefixAmbiguityError(t){let e=Je(t.prefixPath,i=>Fu(i)).join(", "),r=t.alternation.idx===0?"":t.alternation.idx;return`Ambiguous alternatives: <${t.ambiguityIndices.join(" ,")}> due to common lookahead prefix +in inside <${t.topLevelRule.name}> Rule, +<${e}> may appears as a prefix path in all these alternatives. +See: https://chevrotain.io/docs/guide/resolving_grammar_errors.html#COMMON_PREFIX +For Further details.`},buildAlternationAmbiguityError(t){let e=Je(t.prefixPath,i=>Fu(i)).join(", "),r=t.alternation.idx===0?"":t.alternation.idx,n=`Ambiguous Alternatives Detected: <${t.ambiguityIndices.join(" ,")}> in inside <${t.topLevelRule.name}> Rule, +<${e}> may appears as a prefix path in all these alternatives. +`;return n=n+`See: https://chevrotain.io/docs/guide/resolving_grammar_errors.html#AMBIGUOUS_ALTERNATIVES +For Further details.`,n},buildEmptyRepetitionError(t){let e=Bs(t.repetition);return t.repetition.idx!==0&&(e+=t.repetition.idx),`The repetition <${e}> within Rule <${t.topLevelRule.name}> can never consume any tokens. +This could lead to an infinite loop.`},buildTokenNameError(t){return"deprecated"},buildEmptyAlternationError(t){return`Ambiguous empty alternative: <${t.emptyChoiceIdx+1}> in inside <${t.topLevelRule.name}> Rule. +Only the last alternative may be an empty alternative.`},buildTooManyAlternativesError(t){return`An Alternation cannot have more than 256 alternatives: + inside <${t.topLevelRule.name}> Rule. + has ${t.alternation.definition.length+1} alternatives.`},buildLeftRecursionError(t){let e=t.topLevelRule.name,r=Je(t.leftRecursionPath,a=>a.name),n=`${e} --> ${r.concat([e]).join(" --> ")}`;return`Left Recursion found in grammar. +rule: <${e}> can be invoked from itself (directly or indirectly) +without consuming any Tokens. The grammar path that causes this is: + ${n} + To fix this refactor your grammar to remove the left recursion. +see: https://en.wikipedia.org/wiki/LL_parser#Left_factoring.`},buildInvalidRuleNameError(t){return"deprecated"},buildDuplicateRuleNameError(t){let e;return t.topLevelRule instanceof as?e=t.topLevelRule.name:e=t.topLevelRule,`Duplicate definition, rule: ->${e}<- is already defined in the grammar: ->${t.grammarName}<-`}}});function fse(t,e){let r=new RN(t,e);return r.resolveRefs(),r.errors}var RN,dse=N(()=>{"use strict";Fs();qt();os();o(fse,"resolveGrammar");RN=class extends ss{static{o(this,"GastRefResolverVisitor")}constructor(e,r){super(),this.nameToTopRule=e,this.errMsgProvider=r,this.errors=[]}resolveRefs(){Ae(br(this.nameToTopRule),e=>{this.currTopLevel=e,e.accept(this)})}visitNonTerminal(e){let r=this.nameToTopRule[e.nonTerminalName];if(r)e.referencedRule=r;else{let n=this.errMsgProvider.buildRuleNotFoundError(this.currTopLevel,e);this.errors.push({message:n,type:zi.UNRESOLVED_SUBRULE_REF,ruleName:this.currTopLevel.name,unresolvedRefName:e.nonTerminalName})}}}});function Fk(t,e,r=[]){r=an(r);let n=[],i=0;function a(l){return l.concat(gi(t,i+1))}o(a,"remainingPathWith");function s(l){let u=Fk(a(l),e,r);return n.concat(u)}for(o(s,"getAlternativesForProd");r.length{ur(u.definition)===!1&&(n=s(u.definition))}),n;if(l instanceof kr)r.push(l.terminalType);else throw Error("non exhaustive match")}i++}return n.push({partialPath:r,suffixDef:gi(t,i)}),n}function $k(t,e,r,n){let i="EXIT_NONE_TERMINAL",a=[i],s="EXIT_ALTERNATIVE",l=!1,u=e.length,h=u-n-1,f=[],d=[];for(d.push({idx:-1,def:t,ruleStack:[],occurrenceStack:[]});!ur(d);){let p=d.pop();if(p===s){l&&ga(d).idx<=h&&d.pop();continue}let m=p.def,g=p.idx,y=p.ruleStack,v=p.occurrenceStack;if(ur(m))continue;let x=m[0];if(x===i){let b={idx:g,def:gi(m),ruleStack:Nu(y),occurrenceStack:Nu(v)};d.push(b)}else if(x instanceof kr)if(g=0;b--){let w=x.definition[b],C={idx:g,def:w.definition.concat(gi(m)),ruleStack:y,occurrenceStack:v};d.push(C),d.push(s)}else if(x instanceof Dn)d.push({idx:g,def:x.definition.concat(gi(m)),ruleStack:y,occurrenceStack:v});else if(x instanceof as)d.push(KPe(x,g,y,v));else throw Error("non exhaustive match")}return f}function KPe(t,e,r,n){let i=an(r);i.push(t.name);let a=an(n);return a.push(1),{idx:e,def:t.definition,ruleStack:i,occurrenceStack:a}}var NN,Ok,Ug,Pk,ox,Bk,lx,cx=N(()=>{"use strict";qt();xN();Ak();os();NN=class extends Ou{static{o(this,"AbstractNextPossibleTokensWalker")}constructor(e,r){super(),this.topProd=e,this.path=r,this.possibleTokTypes=[],this.nextProductionName="",this.nextProductionOccurrence=0,this.found=!1,this.isAtEndOfPath=!1}startWalking(){if(this.found=!1,this.path.ruleStack[0]!==this.topProd.name)throw Error("The path does not start with the walker's top Rule!");return this.ruleStack=an(this.path.ruleStack).reverse(),this.occurrenceStack=an(this.path.occurrenceStack).reverse(),this.ruleStack.pop(),this.occurrenceStack.pop(),this.updateExpectedNext(),this.walk(this.topProd),this.possibleTokTypes}walk(e,r=[]){this.found||super.walk(e,r)}walkProdRef(e,r,n){if(e.referencedRule.name===this.nextProductionName&&e.idx===this.nextProductionOccurrence){let i=r.concat(n);this.updateExpectedNext(),this.walk(e.referencedRule,i)}}updateExpectedNext(){ur(this.ruleStack)?(this.nextProductionName="",this.nextProductionOccurrence=0,this.isAtEndOfPath=!0):(this.nextProductionName=this.ruleStack.pop(),this.nextProductionOccurrence=this.occurrenceStack.pop())}},Ok=class extends NN{static{o(this,"NextAfterTokenWalker")}constructor(e,r){super(e,r),this.path=r,this.nextTerminalName="",this.nextTerminalOccurrence=0,this.nextTerminalName=this.path.lastTok.name,this.nextTerminalOccurrence=this.path.lastTokOccurrence}walkTerminal(e,r,n){if(this.isAtEndOfPath&&e.terminalType.name===this.nextTerminalName&&e.idx===this.nextTerminalOccurrence&&!this.found){let i=r.concat(n),a=new Dn({definition:i});this.possibleTokTypes=op(a),this.found=!0}}},Ug=class extends Ou{static{o(this,"AbstractNextTerminalAfterProductionWalker")}constructor(e,r){super(),this.topRule=e,this.occurrence=r,this.result={token:void 0,occurrence:void 0,isEndOfRule:void 0}}startWalking(){return this.walk(this.topRule),this.result}},Pk=class extends Ug{static{o(this,"NextTerminalAfterManyWalker")}walkMany(e,r,n){if(e.idx===this.occurrence){let i=ia(r.concat(n));this.result.isEndOfRule=i===void 0,i instanceof kr&&(this.result.token=i.terminalType,this.result.occurrence=i.idx)}else super.walkMany(e,r,n)}},ox=class extends Ug{static{o(this,"NextTerminalAfterManySepWalker")}walkManySep(e,r,n){if(e.idx===this.occurrence){let i=ia(r.concat(n));this.result.isEndOfRule=i===void 0,i instanceof kr&&(this.result.token=i.terminalType,this.result.occurrence=i.idx)}else super.walkManySep(e,r,n)}},Bk=class extends Ug{static{o(this,"NextTerminalAfterAtLeastOneWalker")}walkAtLeastOne(e,r,n){if(e.idx===this.occurrence){let i=ia(r.concat(n));this.result.isEndOfRule=i===void 0,i instanceof kr&&(this.result.token=i.terminalType,this.result.occurrence=i.idx)}else super.walkAtLeastOne(e,r,n)}},lx=class extends Ug{static{o(this,"NextTerminalAfterAtLeastOneSepWalker")}walkAtLeastOneSep(e,r,n){if(e.idx===this.occurrence){let i=ia(r.concat(n));this.result.isEndOfRule=i===void 0,i instanceof kr&&(this.result.token=i.terminalType,this.result.occurrence=i.idx)}else super.walkAtLeastOneSep(e,r,n)}};o(Fk,"possiblePathsFrom");o($k,"nextPossibleTokensAfter");o(KPe,"expandTopLevelRule")});function ux(t){if(t instanceof ln||t==="Option")return jn.OPTION;if(t instanceof Or||t==="Repetition")return jn.REPETITION;if(t instanceof Ln||t==="RepetitionMandatory")return jn.REPETITION_MANDATORY;if(t instanceof Rn||t==="RepetitionMandatoryWithSeparator")return jn.REPETITION_MANDATORY_WITH_SEPARATOR;if(t instanceof wn||t==="RepetitionWithSeparator")return jn.REPETITION_WITH_SEPARATOR;if(t instanceof Tn||t==="Alternation")return jn.ALTERNATION;throw Error("non exhaustive match")}function Gk(t){let{occurrence:e,rule:r,prodType:n,maxLookahead:i}=t,a=ux(n);return a===jn.ALTERNATION?Hg(e,r,i):Wg(e,r,a,i)}function mse(t,e,r,n,i,a){let s=Hg(t,e,r),l=wse(s)?zg:Pu;return a(s,n,l,i)}function gse(t,e,r,n,i,a){let s=Wg(t,e,i,r),l=wse(s)?zg:Pu;return a(s[0],l,n)}function yse(t,e,r,n){let i=t.length,a=Ma(t,s=>Ma(s,l=>l.length===1));if(e)return function(s){let l=Je(s,u=>u.GATE);for(let u=0;uqr(u)),l=Xr(s,(u,h,f)=>(Ae(h,d=>{Bt(u,d.tokenTypeIdx)||(u[d.tokenTypeIdx]=f),Ae(d.categoryMatches,p=>{Bt(u,p)||(u[p]=f)})}),u),{});return function(){let u=this.LA(1);return l[u.tokenTypeIdx]}}else return function(){for(let s=0;sa.length===1),i=t.length;if(n&&!r){let a=qr(t);if(a.length===1&&ur(a[0].categoryMatches)){let l=a[0].tokenTypeIdx;return function(){return this.LA(1).tokenTypeIdx===l}}else{let s=Xr(a,(l,u,h)=>(l[u.tokenTypeIdx]=!0,Ae(u.categoryMatches,f=>{l[f]=!0}),l),[]);return function(){let l=this.LA(1);return s[l.tokenTypeIdx]===!0}}}else return function(){e:for(let a=0;aFk([s],1)),n=pse(r.length),i=Je(r,s=>{let l={};return Ae(s,u=>{let h=MN(u.partialPath);Ae(h,f=>{l[f]=!0})}),l}),a=r;for(let s=1;s<=e;s++){let l=a;a=pse(l.length);for(let u=0;u{let x=MN(v.partialPath);Ae(x,b=>{i[u][b]=!0})})}}}}return n}function Hg(t,e,r,n){let i=new zk(t,jn.ALTERNATION,n);return e.accept(i),xse(i.result,r)}function Wg(t,e,r,n){let i=new zk(t,r);e.accept(i);let a=i.result,l=new IN(e,t,r).startWalking(),u=new Dn({definition:a}),h=new Dn({definition:l});return xse([u,h],n)}function Vk(t,e){e:for(let r=0;r{let i=e[n];return r===i||i.categoryMatchesMap[r.tokenTypeIdx]})}function wse(t){return Ma(t,e=>Ma(e,r=>Ma(r,n=>ur(n.categoryMatches))))}var jn,IN,zk,qg=N(()=>{"use strict";qt();cx();Ak();cp();os();(function(t){t[t.OPTION=0]="OPTION",t[t.REPETITION=1]="REPETITION",t[t.REPETITION_MANDATORY=2]="REPETITION_MANDATORY",t[t.REPETITION_MANDATORY_WITH_SEPARATOR=3]="REPETITION_MANDATORY_WITH_SEPARATOR",t[t.REPETITION_WITH_SEPARATOR=4]="REPETITION_WITH_SEPARATOR",t[t.ALTERNATION=5]="ALTERNATION"})(jn||(jn={}));o(ux,"getProdType");o(Gk,"getLookaheadPaths");o(mse,"buildLookaheadFuncForOr");o(gse,"buildLookaheadFuncForOptionalProd");o(yse,"buildAlternativesLookAheadFunc");o(vse,"buildSingleAlternativeLookaheadFunction");IN=class extends Ou{static{o(this,"RestDefinitionFinderWalker")}constructor(e,r,n){super(),this.topProd=e,this.targetOccurrence=r,this.targetProdType=n}startWalking(){return this.walk(this.topProd),this.restDef}checkIsTarget(e,r,n,i){return e.idx===this.targetOccurrence&&this.targetProdType===r?(this.restDef=n.concat(i),!0):!1}walkOption(e,r,n){this.checkIsTarget(e,jn.OPTION,r,n)||super.walkOption(e,r,n)}walkAtLeastOne(e,r,n){this.checkIsTarget(e,jn.REPETITION_MANDATORY,r,n)||super.walkOption(e,r,n)}walkAtLeastOneSep(e,r,n){this.checkIsTarget(e,jn.REPETITION_MANDATORY_WITH_SEPARATOR,r,n)||super.walkOption(e,r,n)}walkMany(e,r,n){this.checkIsTarget(e,jn.REPETITION,r,n)||super.walkOption(e,r,n)}walkManySep(e,r,n){this.checkIsTarget(e,jn.REPETITION_WITH_SEPARATOR,r,n)||super.walkOption(e,r,n)}},zk=class extends ss{static{o(this,"InsideDefinitionFinderVisitor")}constructor(e,r,n){super(),this.targetOccurrence=e,this.targetProdType=r,this.targetRef=n,this.result=[]}checkIsTarget(e,r){e.idx===this.targetOccurrence&&this.targetProdType===r&&(this.targetRef===void 0||e===this.targetRef)&&(this.result=e.definition)}visitOption(e){this.checkIsTarget(e,jn.OPTION)}visitRepetition(e){this.checkIsTarget(e,jn.REPETITION)}visitRepetitionMandatory(e){this.checkIsTarget(e,jn.REPETITION_MANDATORY)}visitRepetitionMandatoryWithSeparator(e){this.checkIsTarget(e,jn.REPETITION_MANDATORY_WITH_SEPARATOR)}visitRepetitionWithSeparator(e){this.checkIsTarget(e,jn.REPETITION_WITH_SEPARATOR)}visitAlternation(e){this.checkIsTarget(e,jn.ALTERNATION)}};o(pse,"initializeArrayOfArrays");o(MN,"pathToHashKeys");o(QPe,"isUniquePrefixHash");o(xse,"lookAheadSequenceFromAlternatives");o(Hg,"getLookaheadPathsForOr");o(Wg,"getLookaheadPathsForOptionalProd");o(Vk,"containsPath");o(bse,"isStrictPrefixOfPath");o(wse,"areTokenCategoriesNotUsed")});function Tse(t){let e=t.lookaheadStrategy.validate({rules:t.rules,tokenTypes:t.tokenTypes,grammarName:t.grammarName});return Je(e,r=>Object.assign({type:zi.CUSTOM_LOOKAHEAD_VALIDATION},r))}function kse(t,e,r,n){let i=ya(t,u=>ZPe(u,r)),a=iBe(t,e,r),s=ya(t,u=>tBe(u,r)),l=ya(t,u=>eBe(u,t,n,r));return i.concat(a,s,l)}function ZPe(t,e){let r=new ON;t.accept(r);let n=r.allProductions,i=IL(n,JPe),a=Os(i,l=>l.length>1);return Je(br(a),l=>{let u=ia(l),h=e.buildDuplicateFoundError(t,l),f=Bs(u),d={message:h,type:zi.DUPLICATE_PRODUCTIONS,ruleName:t.name,dslName:f,occurrence:u.idx},p=Ese(u);return p&&(d.parameter=p),d})}function JPe(t){return`${Bs(t)}_#_${t.idx}_#_${Ese(t)}`}function Ese(t){return t instanceof kr?t.terminalType.name:t instanceof on?t.nonTerminalName:""}function eBe(t,e,r,n){let i=[];if(Xr(e,(s,l)=>l.name===t.name?s+1:s,0)>1){let s=n.buildDuplicateRuleNameError({topLevelRule:t,grammarName:r});i.push({message:s,type:zi.DUPLICATE_RULE_NAME,ruleName:t.name})}return i}function Sse(t,e,r){let n=[],i;return qn(e,t)||(i=`Invalid rule override, rule: ->${t}<- cannot be overridden in the grammar: ->${r}<-as it is not defined in any of the super grammars `,n.push({message:i,type:zi.INVALID_RULE_OVERRIDE,ruleName:t})),n}function BN(t,e,r,n=[]){let i=[],a=Uk(e.definition);if(ur(a))return[];{let s=t.name;qn(a,t)&&i.push({message:r.buildLeftRecursionError({topLevelRule:t,leftRecursionPath:n}),type:zi.LEFT_RECURSION,ruleName:s});let u=Zh(a,n.concat([t])),h=ya(u,f=>{let d=an(n);return d.push(f),BN(t,f,r,d)});return i.concat(h)}}function Uk(t){let e=[];if(ur(t))return e;let r=ia(t);if(r instanceof on)e.push(r.referencedRule);else if(r instanceof Dn||r instanceof ln||r instanceof Ln||r instanceof Rn||r instanceof wn||r instanceof Or)e=e.concat(Uk(r.definition));else if(r instanceof Tn)e=qr(Je(r.definition,a=>Uk(a.definition)));else if(!(r instanceof kr))throw Error("non exhaustive match");let n=sp(r),i=t.length>1;if(n&&i){let a=gi(t);return e.concat(Uk(a))}else return e}function Cse(t,e){let r=new hx;t.accept(r);let n=r.alternations;return ya(n,a=>{let s=Nu(a.definition);return ya(s,(l,u)=>{let h=$k([l],[],Pu,1);return ur(h)?[{message:e.buildEmptyAlternationError({topLevelRule:t,alternation:a,emptyChoiceIdx:u}),type:zi.NONE_LAST_EMPTY_ALT,ruleName:t.name,occurrence:a.idx,alternative:u+1}]:[]})})}function Ase(t,e,r){let n=new hx;t.accept(n);let i=n.alternations;return i=Jh(i,s=>s.ignoreAmbiguities===!0),ya(i,s=>{let l=s.idx,u=s.maxLookahead||e,h=Hg(l,t,u,s),f=rBe(h,s,t,r),d=nBe(h,s,t,r);return f.concat(d)})}function tBe(t,e){let r=new hx;t.accept(r);let n=r.alternations;return ya(n,a=>a.definition.length>255?[{message:e.buildTooManyAlternativesError({topLevelRule:t,alternation:a}),type:zi.TOO_MANY_ALTS,ruleName:t.name,occurrence:a.idx}]:[])}function _se(t,e,r){let n=[];return Ae(t,i=>{let a=new PN;i.accept(a);let s=a.allProductions;Ae(s,l=>{let u=ux(l),h=l.maxLookahead||e,f=l.idx,p=Wg(f,i,u,h)[0];if(ur(qr(p))){let m=r.buildEmptyRepetitionError({topLevelRule:i,repetition:l});n.push({message:m,type:zi.NO_NON_EMPTY_LOOKAHEAD,ruleName:i.name})}})}),n}function rBe(t,e,r,n){let i=[],a=Xr(t,(l,u,h)=>(e.definition[h].ignoreAmbiguities===!0||Ae(u,f=>{let d=[h];Ae(t,(p,m)=>{h!==m&&Vk(p,f)&&e.definition[m].ignoreAmbiguities!==!0&&d.push(m)}),d.length>1&&!Vk(i,f)&&(i.push(f),l.push({alts:d,path:f}))}),l),[]);return Je(a,l=>{let u=Je(l.alts,f=>f+1);return{message:n.buildAlternationAmbiguityError({topLevelRule:r,alternation:e,ambiguityIndices:u,prefixPath:l.path}),type:zi.AMBIGUOUS_ALTS,ruleName:r.name,occurrence:e.idx,alternatives:l.alts}})}function nBe(t,e,r,n){let i=Xr(t,(s,l,u)=>{let h=Je(l,f=>({idx:u,path:f}));return s.concat(h)},[]);return Tc(ya(i,s=>{if(e.definition[s.idx].ignoreAmbiguities===!0)return[];let u=s.idx,h=s.path,f=Yr(i,p=>e.definition[p.idx].ignoreAmbiguities!==!0&&p.idx{let m=[p.idx+1,u+1],g=e.idx===0?"":e.idx;return{message:n.buildAlternationPrefixAmbiguityError({topLevelRule:r,alternation:e,ambiguityIndices:m,prefixPath:p.path}),type:zi.AMBIGUOUS_PREFIX_ALTS,ruleName:r.name,occurrence:g,alternatives:m}})}))}function iBe(t,e,r){let n=[],i=Je(e,a=>a.name);return Ae(t,a=>{let s=a.name;if(qn(i,s)){let l=r.buildNamespaceConflictError(a);n.push({message:l,type:zi.CONFLICT_TOKENS_RULES_NAMESPACE,ruleName:s})}}),n}var ON,hx,PN,fx=N(()=>{"use strict";qt();Fs();os();qg();cx();cp();o(Tse,"validateLookahead");o(kse,"validateGrammar");o(ZPe,"validateDuplicateProductions");o(JPe,"identifyProductionForDuplicates");o(Ese,"getExtraProductionArgument");ON=class extends ss{static{o(this,"OccurrenceValidationCollector")}constructor(){super(...arguments),this.allProductions=[]}visitNonTerminal(e){this.allProductions.push(e)}visitOption(e){this.allProductions.push(e)}visitRepetitionWithSeparator(e){this.allProductions.push(e)}visitRepetitionMandatory(e){this.allProductions.push(e)}visitRepetitionMandatoryWithSeparator(e){this.allProductions.push(e)}visitRepetition(e){this.allProductions.push(e)}visitAlternation(e){this.allProductions.push(e)}visitTerminal(e){this.allProductions.push(e)}};o(eBe,"validateRuleDoesNotAlreadyExist");o(Sse,"validateRuleIsOverridden");o(BN,"validateNoLeftRecursion");o(Uk,"getFirstNoneTerminal");hx=class extends ss{static{o(this,"OrCollector")}constructor(){super(...arguments),this.alternations=[]}visitAlternation(e){this.alternations.push(e)}};o(Cse,"validateEmptyOrAlternative");o(Ase,"validateAmbiguousAlternationAlternatives");PN=class extends ss{static{o(this,"RepetitionCollector")}constructor(){super(...arguments),this.allProductions=[]}visitRepetitionWithSeparator(e){this.allProductions.push(e)}visitRepetitionMandatory(e){this.allProductions.push(e)}visitRepetitionMandatoryWithSeparator(e){this.allProductions.push(e)}visitRepetition(e){this.allProductions.push(e)}};o(tBe,"validateTooManyAlts");o(_se,"validateSomeNonEmptyLookaheadPath");o(rBe,"checkAlternativesAmbiguities");o(nBe,"checkPrefixAlternativesAmbiguities");o(iBe,"checkTerminalAndNoneTerminalsNameSpace")});function Dse(t){let e=Qh(t,{errMsgProvider:hse}),r={};return Ae(t.rules,n=>{r[n.name]=n}),fse(r,e.errMsgProvider)}function Lse(t){return t=Qh(t,{errMsgProvider:Pl}),kse(t.rules,t.tokenTypes,t.errMsgProvider,t.grammarName)}var Rse=N(()=>{"use strict";qt();dse();fx();Vg();o(Dse,"resolveGrammar");o(Lse,"validateGrammar")});function lf(t){return qn(Pse,t.name)}var Nse,Mse,Ise,Ose,Pse,Yg,hp,dx,px,mx,Xg=N(()=>{"use strict";qt();Nse="MismatchedTokenException",Mse="NoViableAltException",Ise="EarlyExitException",Ose="NotAllInputParsedException",Pse=[Nse,Mse,Ise,Ose];Object.freeze(Pse);o(lf,"isRecognitionException");Yg=class extends Error{static{o(this,"RecognitionException")}constructor(e,r){super(e),this.token=r,this.resyncedTokens=[],Object.setPrototypeOf(this,new.target.prototype),Error.captureStackTrace&&Error.captureStackTrace(this,this.constructor)}},hp=class extends Yg{static{o(this,"MismatchedTokenException")}constructor(e,r,n){super(e,r),this.previousToken=n,this.name=Nse}},dx=class extends Yg{static{o(this,"NoViableAltException")}constructor(e,r,n){super(e,r),this.previousToken=n,this.name=Mse}},px=class extends Yg{static{o(this,"NotAllInputParsedException")}constructor(e,r){super(e,r),this.name=Ose}},mx=class extends Yg{static{o(this,"EarlyExitException")}constructor(e,r,n){super(e,r),this.previousToken=n,this.name=Ise}}});function aBe(t,e,r,n,i,a,s){let l=this.getKeyForAutomaticLookahead(n,i),u=this.firstAfterRepMap[l];if(u===void 0){let p=this.getCurrRuleFullName(),m=this.getGAstProductions()[p];u=new a(m,i).startWalking(),this.firstAfterRepMap[l]=u}let h=u.token,f=u.occurrence,d=u.isEndOfRule;this.RULE_STACK.length===1&&d&&h===void 0&&(h=lo,f=1),!(h===void 0||f===void 0)&&this.shouldInRepetitionRecoveryBeTried(h,f,s)&&this.tryInRepetitionRecovery(t,e,r,h)}var FN,zN,$N,Hk,GN=N(()=>{"use strict";up();qt();Xg();bN();Fs();FN={},zN="InRuleRecoveryException",$N=class extends Error{static{o(this,"InRuleRecoveryException")}constructor(e){super(e),this.name=zN}},Hk=class{static{o(this,"Recoverable")}initRecoverable(e){this.firstAfterRepMap={},this.resyncFollows={},this.recoveryEnabled=Bt(e,"recoveryEnabled")?e.recoveryEnabled:ls.recoveryEnabled,this.recoveryEnabled&&(this.attemptInRepetitionRecovery=aBe)}getTokenToInsert(e){let r=$u(e,"",NaN,NaN,NaN,NaN,NaN,NaN);return r.isInsertedInRecovery=!0,r}canTokenTypeBeInsertedInRecovery(e){return!0}canTokenTypeBeDeletedInRecovery(e){return!0}tryInRepetitionRecovery(e,r,n,i){let a=this.findReSyncTokenType(),s=this.exportLexerState(),l=[],u=!1,h=this.LA(1),f=this.LA(1),d=o(()=>{let p=this.LA(0),m=this.errorMessageProvider.buildMismatchTokenMessage({expected:i,actual:h,previous:p,ruleName:this.getCurrRuleFullName()}),g=new hp(m,h,this.LA(0));g.resyncedTokens=Nu(l),this.SAVE_ERROR(g)},"generateErrorMessage");for(;!u;)if(this.tokenMatcher(f,i)){d();return}else if(n.call(this)){d(),e.apply(this,r);return}else this.tokenMatcher(f,a)?u=!0:(f=this.SKIP_TOKEN(),this.addToResyncTokens(f,l));this.importLexerState(s)}shouldInRepetitionRecoveryBeTried(e,r,n){return!(n===!1||this.tokenMatcher(this.LA(1),e)||this.isBackTracking()||this.canPerformInRuleRecovery(e,this.getFollowsForInRuleRecovery(e,r)))}getFollowsForInRuleRecovery(e,r){let n=this.getCurrentGrammarPath(e,r);return this.getNextPossibleTokenTypes(n)}tryInRuleRecovery(e,r){if(this.canRecoverWithSingleTokenInsertion(e,r))return this.getTokenToInsert(e);if(this.canRecoverWithSingleTokenDeletion(e)){let n=this.SKIP_TOKEN();return this.consumeToken(),n}throw new $N("sad sad panda")}canPerformInRuleRecovery(e,r){return this.canRecoverWithSingleTokenInsertion(e,r)||this.canRecoverWithSingleTokenDeletion(e)}canRecoverWithSingleTokenInsertion(e,r){if(!this.canTokenTypeBeInsertedInRecovery(e)||ur(r))return!1;let n=this.LA(1);return ns(r,a=>this.tokenMatcher(n,a))!==void 0}canRecoverWithSingleTokenDeletion(e){return this.canTokenTypeBeDeletedInRecovery(e)?this.tokenMatcher(this.LA(2),e):!1}isInCurrentRuleReSyncSet(e){let r=this.getCurrFollowKey(),n=this.getFollowSetFromFollowKey(r);return qn(n,e)}findReSyncTokenType(){let e=this.flattenFollowSet(),r=this.LA(1),n=2;for(;;){let i=ns(e,a=>sx(r,a));if(i!==void 0)return i;r=this.LA(n),n++}}getCurrFollowKey(){if(this.RULE_STACK.length===1)return FN;let e=this.getLastExplicitRuleShortName(),r=this.getLastExplicitRuleOccurrenceIndex(),n=this.getPreviousExplicitRuleShortName();return{ruleName:this.shortRuleNameToFullName(e),idxInCallingRule:r,inRule:this.shortRuleNameToFullName(n)}}buildFullFollowKeyStack(){let e=this.RULE_STACK,r=this.RULE_OCCURRENCE_STACK;return Je(e,(n,i)=>i===0?FN:{ruleName:this.shortRuleNameToFullName(n),idxInCallingRule:r[i],inRule:this.shortRuleNameToFullName(e[i-1])})}flattenFollowSet(){let e=Je(this.buildFullFollowKeyStack(),r=>this.getFollowSetFromFollowKey(r));return qr(e)}getFollowSetFromFollowKey(e){if(e===FN)return[lo];let r=e.ruleName+e.idxInCallingRule+_k+e.inRule;return this.resyncFollows[r]}addToResyncTokens(e,r){return this.tokenMatcher(e,lo)||r.push(e),r}reSyncTo(e){let r=[],n=this.LA(1);for(;this.tokenMatcher(n,e)===!1;)n=this.SKIP_TOKEN(),this.addToResyncTokens(n,r);return Nu(r)}attemptInRepetitionRecovery(e,r,n,i,a,s,l){}getCurrentGrammarPath(e,r){let n=this.getHumanReadableRuleStack(),i=an(this.RULE_OCCURRENCE_STACK);return{ruleStack:n,occurrenceStack:i,lastTok:e,lastTokOccurrence:r}}getHumanReadableRuleStack(){return Je(this.RULE_STACK,e=>this.shortRuleNameToFullName(e))}};o(aBe,"attemptInRepetitionRecovery")});function Wk(t,e,r){return r|e|t}var qk=N(()=>{"use strict";o(Wk,"getKeyForAutomaticLookahead")});var Gu,VN=N(()=>{"use strict";qt();Vg();Fs();fx();qg();Gu=class{static{o(this,"LLkLookaheadStrategy")}constructor(e){var r;this.maxLookahead=(r=e?.maxLookahead)!==null&&r!==void 0?r:ls.maxLookahead}validate(e){let r=this.validateNoLeftRecursion(e.rules);if(ur(r)){let n=this.validateEmptyOrAlternatives(e.rules),i=this.validateAmbiguousAlternationAlternatives(e.rules,this.maxLookahead),a=this.validateSomeNonEmptyLookaheadPath(e.rules,this.maxLookahead);return[...r,...n,...i,...a]}return r}validateNoLeftRecursion(e){return ya(e,r=>BN(r,r,Pl))}validateEmptyOrAlternatives(e){return ya(e,r=>Cse(r,Pl))}validateAmbiguousAlternationAlternatives(e,r){return ya(e,n=>Ase(n,r,Pl))}validateSomeNonEmptyLookaheadPath(e,r){return _se(e,r,Pl)}buildLookaheadForAlternation(e){return mse(e.prodOccurrence,e.rule,e.maxLookahead,e.hasPredicates,e.dynamicTokensEnabled,yse)}buildLookaheadForOptional(e){return gse(e.prodOccurrence,e.rule,e.maxLookahead,e.dynamicTokensEnabled,ux(e.prodType),vse)}}});function sBe(t){Yk.reset(),t.accept(Yk);let e=Yk.dslMethods;return Yk.reset(),e}var Xk,UN,Yk,Bse=N(()=>{"use strict";qt();Fs();qk();os();VN();Xk=class{static{o(this,"LooksAhead")}initLooksAhead(e){this.dynamicTokensEnabled=Bt(e,"dynamicTokensEnabled")?e.dynamicTokensEnabled:ls.dynamicTokensEnabled,this.maxLookahead=Bt(e,"maxLookahead")?e.maxLookahead:ls.maxLookahead,this.lookaheadStrategy=Bt(e,"lookaheadStrategy")?e.lookaheadStrategy:new Gu({maxLookahead:this.maxLookahead}),this.lookAheadFuncsCache=new Map}preComputeLookaheadFunctions(e){Ae(e,r=>{this.TRACE_INIT(`${r.name} Rule Lookahead`,()=>{let{alternation:n,repetition:i,option:a,repetitionMandatory:s,repetitionMandatoryWithSeparator:l,repetitionWithSeparator:u}=sBe(r);Ae(n,h=>{let f=h.idx===0?"":h.idx;this.TRACE_INIT(`${Bs(h)}${f}`,()=>{let d=this.lookaheadStrategy.buildLookaheadForAlternation({prodOccurrence:h.idx,rule:r,maxLookahead:h.maxLookahead||this.maxLookahead,hasPredicates:h.hasPredicates,dynamicTokensEnabled:this.dynamicTokensEnabled}),p=Wk(this.fullRuleNameToShort[r.name],256,h.idx);this.setLaFuncCache(p,d)})}),Ae(i,h=>{this.computeLookaheadFunc(r,h.idx,768,"Repetition",h.maxLookahead,Bs(h))}),Ae(a,h=>{this.computeLookaheadFunc(r,h.idx,512,"Option",h.maxLookahead,Bs(h))}),Ae(s,h=>{this.computeLookaheadFunc(r,h.idx,1024,"RepetitionMandatory",h.maxLookahead,Bs(h))}),Ae(l,h=>{this.computeLookaheadFunc(r,h.idx,1536,"RepetitionMandatoryWithSeparator",h.maxLookahead,Bs(h))}),Ae(u,h=>{this.computeLookaheadFunc(r,h.idx,1280,"RepetitionWithSeparator",h.maxLookahead,Bs(h))})})})}computeLookaheadFunc(e,r,n,i,a,s){this.TRACE_INIT(`${s}${r===0?"":r}`,()=>{let l=this.lookaheadStrategy.buildLookaheadForOptional({prodOccurrence:r,rule:e,maxLookahead:a||this.maxLookahead,dynamicTokensEnabled:this.dynamicTokensEnabled,prodType:i}),u=Wk(this.fullRuleNameToShort[e.name],n,r);this.setLaFuncCache(u,l)})}getKeyForAutomaticLookahead(e,r){let n=this.getLastExplicitRuleShortName();return Wk(n,e,r)}getLaFuncFromCache(e){return this.lookAheadFuncsCache.get(e)}setLaFuncCache(e,r){this.lookAheadFuncsCache.set(e,r)}},UN=class extends ss{static{o(this,"DslMethodsCollectorVisitor")}constructor(){super(...arguments),this.dslMethods={option:[],alternation:[],repetition:[],repetitionWithSeparator:[],repetitionMandatory:[],repetitionMandatoryWithSeparator:[]}}reset(){this.dslMethods={option:[],alternation:[],repetition:[],repetitionWithSeparator:[],repetitionMandatory:[],repetitionMandatoryWithSeparator:[]}}visitOption(e){this.dslMethods.option.push(e)}visitRepetitionWithSeparator(e){this.dslMethods.repetitionWithSeparator.push(e)}visitRepetitionMandatory(e){this.dslMethods.repetitionMandatory.push(e)}visitRepetitionMandatoryWithSeparator(e){this.dslMethods.repetitionMandatoryWithSeparator.push(e)}visitRepetition(e){this.dslMethods.repetition.push(e)}visitAlternation(e){this.dslMethods.alternation.push(e)}},Yk=new UN;o(sBe,"collectMethods")});function qN(t,e){isNaN(t.startOffset)===!0?(t.startOffset=e.startOffset,t.endOffset=e.endOffset):t.endOffset{"use strict";o(qN,"setNodeLocationOnlyOffset");o(YN,"setNodeLocationFull");o(Fse,"addTerminalToCst");o($se,"addNoneTerminalToCst")});function XN(t,e){Object.defineProperty(t,oBe,{enumerable:!1,configurable:!0,writable:!1,value:e})}var oBe,Gse=N(()=>{"use strict";oBe="name";o(XN,"defineNameProp")});function lBe(t,e){let r=zr(t),n=r.length;for(let i=0;is.msg);throw Error(`Errors Detected in CST Visitor <${this.constructor.name}>: + ${a.join(` + +`).replace(/\n/g,` + `)}`)}},"validateVisitor")};return r.prototype=n,r.prototype.constructor=r,r._RULE_NAMES=e,r}function Use(t,e,r){let n=o(function(){},"derivedConstructor");XN(n,t+"BaseSemanticsWithDefaults");let i=Object.create(r.prototype);return Ae(e,a=>{i[a]=lBe}),n.prototype=i,n.prototype.constructor=n,n}function cBe(t,e){return uBe(t,e)}function uBe(t,e){let r=Yr(e,i=>Si(t[i])===!1),n=Je(r,i=>({msg:`Missing visitor method: <${i}> on ${t.constructor.name} CST Visitor.`,type:jN.MISSING_METHOD,methodName:i}));return Tc(n)}var jN,Hse=N(()=>{"use strict";qt();Gse();o(lBe,"defaultVisit");o(Vse,"createBaseSemanticVisitorConstructor");o(Use,"createBaseVisitorConstructorWithDefaults");(function(t){t[t.REDUNDANT_METHOD=0]="REDUNDANT_METHOD",t[t.MISSING_METHOD=1]="MISSING_METHOD"})(jN||(jN={}));o(cBe,"validateVisitor");o(uBe,"validateMissingCstMethods")});var Zk,Wse=N(()=>{"use strict";zse();qt();Hse();Fs();Zk=class{static{o(this,"TreeBuilder")}initTreeBuilder(e){if(this.CST_STACK=[],this.outputCst=e.outputCst,this.nodeLocationTracking=Bt(e,"nodeLocationTracking")?e.nodeLocationTracking:ls.nodeLocationTracking,!this.outputCst)this.cstInvocationStateUpdate=ni,this.cstFinallyStateUpdate=ni,this.cstPostTerminal=ni,this.cstPostNonTerminal=ni,this.cstPostRule=ni;else if(/full/i.test(this.nodeLocationTracking))this.recoveryEnabled?(this.setNodeLocationFromToken=YN,this.setNodeLocationFromNode=YN,this.cstPostRule=ni,this.setInitialNodeLocation=this.setInitialNodeLocationFullRecovery):(this.setNodeLocationFromToken=ni,this.setNodeLocationFromNode=ni,this.cstPostRule=this.cstPostRuleFull,this.setInitialNodeLocation=this.setInitialNodeLocationFullRegular);else if(/onlyOffset/i.test(this.nodeLocationTracking))this.recoveryEnabled?(this.setNodeLocationFromToken=qN,this.setNodeLocationFromNode=qN,this.cstPostRule=ni,this.setInitialNodeLocation=this.setInitialNodeLocationOnlyOffsetRecovery):(this.setNodeLocationFromToken=ni,this.setNodeLocationFromNode=ni,this.cstPostRule=this.cstPostRuleOnlyOffset,this.setInitialNodeLocation=this.setInitialNodeLocationOnlyOffsetRegular);else if(/none/i.test(this.nodeLocationTracking))this.setNodeLocationFromToken=ni,this.setNodeLocationFromNode=ni,this.cstPostRule=ni,this.setInitialNodeLocation=ni;else throw Error(`Invalid config option: "${e.nodeLocationTracking}"`)}setInitialNodeLocationOnlyOffsetRecovery(e){e.location={startOffset:NaN,endOffset:NaN}}setInitialNodeLocationOnlyOffsetRegular(e){e.location={startOffset:this.LA(1).startOffset,endOffset:NaN}}setInitialNodeLocationFullRecovery(e){e.location={startOffset:NaN,startLine:NaN,startColumn:NaN,endOffset:NaN,endLine:NaN,endColumn:NaN}}setInitialNodeLocationFullRegular(e){let r=this.LA(1);e.location={startOffset:r.startOffset,startLine:r.startLine,startColumn:r.startColumn,endOffset:NaN,endLine:NaN,endColumn:NaN}}cstInvocationStateUpdate(e){let r={name:e,children:Object.create(null)};this.setInitialNodeLocation(r),this.CST_STACK.push(r)}cstFinallyStateUpdate(){this.CST_STACK.pop()}cstPostRuleFull(e){let r=this.LA(0),n=e.location;n.startOffset<=r.startOffset?(n.endOffset=r.endOffset,n.endLine=r.endLine,n.endColumn=r.endColumn):(n.startOffset=NaN,n.startLine=NaN,n.startColumn=NaN)}cstPostRuleOnlyOffset(e){let r=this.LA(0),n=e.location;n.startOffset<=r.startOffset?n.endOffset=r.endOffset:n.startOffset=NaN}cstPostTerminal(e,r){let n=this.CST_STACK[this.CST_STACK.length-1];Fse(n,r,e),this.setNodeLocationFromToken(n.location,r)}cstPostNonTerminal(e,r){let n=this.CST_STACK[this.CST_STACK.length-1];$se(n,r,e),this.setNodeLocationFromNode(n.location,e.location)}getBaseCstVisitorConstructor(){if(pr(this.baseCstVisitorConstructor)){let e=Vse(this.className,zr(this.gastProductionsCache));return this.baseCstVisitorConstructor=e,e}return this.baseCstVisitorConstructor}getBaseCstVisitorConstructorWithDefaults(){if(pr(this.baseCstVisitorWithDefaultsConstructor)){let e=Use(this.className,zr(this.gastProductionsCache),this.getBaseCstVisitorConstructor());return this.baseCstVisitorWithDefaultsConstructor=e,e}return this.baseCstVisitorWithDefaultsConstructor}getLastExplicitRuleShortName(){let e=this.RULE_STACK;return e[e.length-1]}getPreviousExplicitRuleShortName(){let e=this.RULE_STACK;return e[e.length-2]}getLastExplicitRuleOccurrenceIndex(){let e=this.RULE_OCCURRENCE_STACK;return e[e.length-1]}}});var Jk,qse=N(()=>{"use strict";Fs();Jk=class{static{o(this,"LexerAdapter")}initLexerAdapter(){this.tokVector=[],this.tokVectorLength=0,this.currIdx=-1}set input(e){if(this.selfAnalysisDone!==!0)throw Error("Missing invocation at the end of the Parser's constructor.");this.reset(),this.tokVector=e,this.tokVectorLength=e.length}get input(){return this.tokVector}SKIP_TOKEN(){return this.currIdx<=this.tokVector.length-2?(this.consumeToken(),this.LA(1)):jg}LA(e){let r=this.currIdx+e;return r<0||this.tokVectorLength<=r?jg:this.tokVector[r]}consumeToken(){this.currIdx++}exportLexerState(){return this.currIdx}importLexerState(e){this.currIdx=e}resetLexerState(){this.currIdx=-1}moveToTerminatedState(){this.currIdx=this.tokVector.length-1}getLexerPosition(){return this.exportLexerState()}}});var eE,Yse=N(()=>{"use strict";qt();Xg();Fs();Vg();fx();os();eE=class{static{o(this,"RecognizerApi")}ACTION(e){return e.call(this)}consume(e,r,n){return this.consumeInternal(r,e,n)}subrule(e,r,n){return this.subruleInternal(r,e,n)}option(e,r){return this.optionInternal(r,e)}or(e,r){return this.orInternal(r,e)}many(e,r){return this.manyInternal(e,r)}atLeastOne(e,r){return this.atLeastOneInternal(e,r)}CONSUME(e,r){return this.consumeInternal(e,0,r)}CONSUME1(e,r){return this.consumeInternal(e,1,r)}CONSUME2(e,r){return this.consumeInternal(e,2,r)}CONSUME3(e,r){return this.consumeInternal(e,3,r)}CONSUME4(e,r){return this.consumeInternal(e,4,r)}CONSUME5(e,r){return this.consumeInternal(e,5,r)}CONSUME6(e,r){return this.consumeInternal(e,6,r)}CONSUME7(e,r){return this.consumeInternal(e,7,r)}CONSUME8(e,r){return this.consumeInternal(e,8,r)}CONSUME9(e,r){return this.consumeInternal(e,9,r)}SUBRULE(e,r){return this.subruleInternal(e,0,r)}SUBRULE1(e,r){return this.subruleInternal(e,1,r)}SUBRULE2(e,r){return this.subruleInternal(e,2,r)}SUBRULE3(e,r){return this.subruleInternal(e,3,r)}SUBRULE4(e,r){return this.subruleInternal(e,4,r)}SUBRULE5(e,r){return this.subruleInternal(e,5,r)}SUBRULE6(e,r){return this.subruleInternal(e,6,r)}SUBRULE7(e,r){return this.subruleInternal(e,7,r)}SUBRULE8(e,r){return this.subruleInternal(e,8,r)}SUBRULE9(e,r){return this.subruleInternal(e,9,r)}OPTION(e){return this.optionInternal(e,0)}OPTION1(e){return this.optionInternal(e,1)}OPTION2(e){return this.optionInternal(e,2)}OPTION3(e){return this.optionInternal(e,3)}OPTION4(e){return this.optionInternal(e,4)}OPTION5(e){return this.optionInternal(e,5)}OPTION6(e){return this.optionInternal(e,6)}OPTION7(e){return this.optionInternal(e,7)}OPTION8(e){return this.optionInternal(e,8)}OPTION9(e){return this.optionInternal(e,9)}OR(e){return this.orInternal(e,0)}OR1(e){return this.orInternal(e,1)}OR2(e){return this.orInternal(e,2)}OR3(e){return this.orInternal(e,3)}OR4(e){return this.orInternal(e,4)}OR5(e){return this.orInternal(e,5)}OR6(e){return this.orInternal(e,6)}OR7(e){return this.orInternal(e,7)}OR8(e){return this.orInternal(e,8)}OR9(e){return this.orInternal(e,9)}MANY(e){this.manyInternal(0,e)}MANY1(e){this.manyInternal(1,e)}MANY2(e){this.manyInternal(2,e)}MANY3(e){this.manyInternal(3,e)}MANY4(e){this.manyInternal(4,e)}MANY5(e){this.manyInternal(5,e)}MANY6(e){this.manyInternal(6,e)}MANY7(e){this.manyInternal(7,e)}MANY8(e){this.manyInternal(8,e)}MANY9(e){this.manyInternal(9,e)}MANY_SEP(e){this.manySepFirstInternal(0,e)}MANY_SEP1(e){this.manySepFirstInternal(1,e)}MANY_SEP2(e){this.manySepFirstInternal(2,e)}MANY_SEP3(e){this.manySepFirstInternal(3,e)}MANY_SEP4(e){this.manySepFirstInternal(4,e)}MANY_SEP5(e){this.manySepFirstInternal(5,e)}MANY_SEP6(e){this.manySepFirstInternal(6,e)}MANY_SEP7(e){this.manySepFirstInternal(7,e)}MANY_SEP8(e){this.manySepFirstInternal(8,e)}MANY_SEP9(e){this.manySepFirstInternal(9,e)}AT_LEAST_ONE(e){this.atLeastOneInternal(0,e)}AT_LEAST_ONE1(e){return this.atLeastOneInternal(1,e)}AT_LEAST_ONE2(e){this.atLeastOneInternal(2,e)}AT_LEAST_ONE3(e){this.atLeastOneInternal(3,e)}AT_LEAST_ONE4(e){this.atLeastOneInternal(4,e)}AT_LEAST_ONE5(e){this.atLeastOneInternal(5,e)}AT_LEAST_ONE6(e){this.atLeastOneInternal(6,e)}AT_LEAST_ONE7(e){this.atLeastOneInternal(7,e)}AT_LEAST_ONE8(e){this.atLeastOneInternal(8,e)}AT_LEAST_ONE9(e){this.atLeastOneInternal(9,e)}AT_LEAST_ONE_SEP(e){this.atLeastOneSepFirstInternal(0,e)}AT_LEAST_ONE_SEP1(e){this.atLeastOneSepFirstInternal(1,e)}AT_LEAST_ONE_SEP2(e){this.atLeastOneSepFirstInternal(2,e)}AT_LEAST_ONE_SEP3(e){this.atLeastOneSepFirstInternal(3,e)}AT_LEAST_ONE_SEP4(e){this.atLeastOneSepFirstInternal(4,e)}AT_LEAST_ONE_SEP5(e){this.atLeastOneSepFirstInternal(5,e)}AT_LEAST_ONE_SEP6(e){this.atLeastOneSepFirstInternal(6,e)}AT_LEAST_ONE_SEP7(e){this.atLeastOneSepFirstInternal(7,e)}AT_LEAST_ONE_SEP8(e){this.atLeastOneSepFirstInternal(8,e)}AT_LEAST_ONE_SEP9(e){this.atLeastOneSepFirstInternal(9,e)}RULE(e,r,n=Kg){if(qn(this.definedRulesNames,e)){let s={message:Pl.buildDuplicateRuleNameError({topLevelRule:e,grammarName:this.className}),type:zi.DUPLICATE_RULE_NAME,ruleName:e};this.definitionErrors.push(s)}this.definedRulesNames.push(e);let i=this.defineRule(e,r,n);return this[e]=i,i}OVERRIDE_RULE(e,r,n=Kg){let i=Sse(e,this.definedRulesNames,this.className);this.definitionErrors=this.definitionErrors.concat(i);let a=this.defineRule(e,r,n);return this[e]=a,a}BACKTRACK(e,r){return function(){this.isBackTrackingStack.push(1);let n=this.saveRecogState();try{return e.apply(this,r),!0}catch(i){if(lf(i))return!1;throw i}finally{this.reloadRecogState(n),this.isBackTrackingStack.pop()}}}getGAstProductions(){return this.gastProductionsCache}getSerializedGastProductions(){return Sk(br(this.gastProductionsCache))}}});var tE,Xse=N(()=>{"use strict";qt();qk();Xg();qg();cx();Fs();GN();up();cp();tE=class{static{o(this,"RecognizerEngine")}initRecognizerEngine(e,r){if(this.className=this.constructor.name,this.shortRuleNameToFull={},this.fullRuleNameToShort={},this.ruleShortNameIdx=256,this.tokenMatcher=zg,this.subruleIdx=0,this.definedRulesNames=[],this.tokensMap={},this.isBackTrackingStack=[],this.RULE_STACK=[],this.RULE_OCCURRENCE_STACK=[],this.gastProductionsCache={},Bt(r,"serializedGrammar"))throw Error(`The Parser's configuration can no longer contain a property. + See: https://chevrotain.io/docs/changes/BREAKING_CHANGES.html#_6-0-0 + For Further details.`);if(Pt(e)){if(ur(e))throw Error(`A Token Vocabulary cannot be empty. + Note that the first argument for the parser constructor + is no longer a Token vector (since v4.0).`);if(typeof e[0].startOffset=="number")throw Error(`The Parser constructor no longer accepts a token vector as the first argument. + See: https://chevrotain.io/docs/changes/BREAKING_CHANGES.html#_4-0-0 + For Further details.`)}if(Pt(e))this.tokensMap=Xr(e,(a,s)=>(a[s.name]=s,a),{});else if(Bt(e,"modes")&&Ma(qr(br(e.modes)),rse)){let a=qr(br(e.modes)),s=Bm(a);this.tokensMap=Xr(s,(l,u)=>(l[u.name]=u,l),{})}else if(bn(e))this.tokensMap=an(e);else throw new Error(" argument must be An Array of Token constructors, A dictionary of Token constructors or an IMultiModeLexerDefinition");this.tokensMap.EOF=lo;let n=Bt(e,"modes")?qr(br(e.modes)):br(e),i=Ma(n,a=>ur(a.categoryMatches));this.tokenMatcher=i?zg:Pu,Bu(br(this.tokensMap))}defineRule(e,r,n){if(this.selfAnalysisDone)throw Error(`Grammar rule <${e}> may not be defined after the 'performSelfAnalysis' method has been called' +Make sure that all grammar rule definitions are done before 'performSelfAnalysis' is called.`);let i=Bt(n,"resyncEnabled")?n.resyncEnabled:Kg.resyncEnabled,a=Bt(n,"recoveryValueFunc")?n.recoveryValueFunc:Kg.recoveryValueFunc,s=this.ruleShortNameIdx<<12;this.ruleShortNameIdx++,this.shortRuleNameToFull[s]=e,this.fullRuleNameToShort[e]=s;let l;return this.outputCst===!0?l=o(function(...f){try{this.ruleInvocationStateUpdate(s,e,this.subruleIdx),r.apply(this,f);let d=this.CST_STACK[this.CST_STACK.length-1];return this.cstPostRule(d),d}catch(d){return this.invokeRuleCatch(d,i,a)}finally{this.ruleFinallyStateUpdate()}},"invokeRuleWithTry"):l=o(function(...f){try{return this.ruleInvocationStateUpdate(s,e,this.subruleIdx),r.apply(this,f)}catch(d){return this.invokeRuleCatch(d,i,a)}finally{this.ruleFinallyStateUpdate()}},"invokeRuleWithTryCst"),Object.assign(l,{ruleName:e,originalGrammarAction:r})}invokeRuleCatch(e,r,n){let i=this.RULE_STACK.length===1,a=r&&!this.isBackTracking()&&this.recoveryEnabled;if(lf(e)){let s=e;if(a){let l=this.findReSyncTokenType();if(this.isInCurrentRuleReSyncSet(l))if(s.resyncedTokens=this.reSyncTo(l),this.outputCst){let u=this.CST_STACK[this.CST_STACK.length-1];return u.recoveredNode=!0,u}else return n(e);else{if(this.outputCst){let u=this.CST_STACK[this.CST_STACK.length-1];u.recoveredNode=!0,s.partialCstResult=u}throw s}}else{if(i)return this.moveToTerminatedState(),n(e);throw s}}else throw e}optionInternal(e,r){let n=this.getKeyForAutomaticLookahead(512,r);return this.optionInternalLogic(e,r,n)}optionInternalLogic(e,r,n){let i=this.getLaFuncFromCache(n),a;if(typeof e!="function"){a=e.DEF;let s=e.GATE;if(s!==void 0){let l=i;i=o(()=>s.call(this)&&l.call(this),"lookAheadFunc")}}else a=e;if(i.call(this)===!0)return a.call(this)}atLeastOneInternal(e,r){let n=this.getKeyForAutomaticLookahead(1024,e);return this.atLeastOneInternalLogic(e,r,n)}atLeastOneInternalLogic(e,r,n){let i=this.getLaFuncFromCache(n),a;if(typeof r!="function"){a=r.DEF;let s=r.GATE;if(s!==void 0){let l=i;i=o(()=>s.call(this)&&l.call(this),"lookAheadFunc")}}else a=r;if(i.call(this)===!0){let s=this.doSingleRepetition(a);for(;i.call(this)===!0&&s===!0;)s=this.doSingleRepetition(a)}else throw this.raiseEarlyExitException(e,jn.REPETITION_MANDATORY,r.ERR_MSG);this.attemptInRepetitionRecovery(this.atLeastOneInternal,[e,r],i,1024,e,Bk)}atLeastOneSepFirstInternal(e,r){let n=this.getKeyForAutomaticLookahead(1536,e);this.atLeastOneSepFirstInternalLogic(e,r,n)}atLeastOneSepFirstInternalLogic(e,r,n){let i=r.DEF,a=r.SEP;if(this.getLaFuncFromCache(n).call(this)===!0){i.call(this);let l=o(()=>this.tokenMatcher(this.LA(1),a),"separatorLookAheadFunc");for(;this.tokenMatcher(this.LA(1),a)===!0;)this.CONSUME(a),i.call(this);this.attemptInRepetitionRecovery(this.repetitionSepSecondInternal,[e,a,l,i,lx],l,1536,e,lx)}else throw this.raiseEarlyExitException(e,jn.REPETITION_MANDATORY_WITH_SEPARATOR,r.ERR_MSG)}manyInternal(e,r){let n=this.getKeyForAutomaticLookahead(768,e);return this.manyInternalLogic(e,r,n)}manyInternalLogic(e,r,n){let i=this.getLaFuncFromCache(n),a;if(typeof r!="function"){a=r.DEF;let l=r.GATE;if(l!==void 0){let u=i;i=o(()=>l.call(this)&&u.call(this),"lookaheadFunction")}}else a=r;let s=!0;for(;i.call(this)===!0&&s===!0;)s=this.doSingleRepetition(a);this.attemptInRepetitionRecovery(this.manyInternal,[e,r],i,768,e,Pk,s)}manySepFirstInternal(e,r){let n=this.getKeyForAutomaticLookahead(1280,e);this.manySepFirstInternalLogic(e,r,n)}manySepFirstInternalLogic(e,r,n){let i=r.DEF,a=r.SEP;if(this.getLaFuncFromCache(n).call(this)===!0){i.call(this);let l=o(()=>this.tokenMatcher(this.LA(1),a),"separatorLookAheadFunc");for(;this.tokenMatcher(this.LA(1),a)===!0;)this.CONSUME(a),i.call(this);this.attemptInRepetitionRecovery(this.repetitionSepSecondInternal,[e,a,l,i,ox],l,1280,e,ox)}}repetitionSepSecondInternal(e,r,n,i,a){for(;n();)this.CONSUME(r),i.call(this);this.attemptInRepetitionRecovery(this.repetitionSepSecondInternal,[e,r,n,i,a],n,1536,e,a)}doSingleRepetition(e){let r=this.getLexerPosition();return e.call(this),this.getLexerPosition()>r}orInternal(e,r){let n=this.getKeyForAutomaticLookahead(256,r),i=Pt(e)?e:e.DEF,s=this.getLaFuncFromCache(n).call(this,i);if(s!==void 0)return i[s].ALT.call(this);this.raiseNoAltException(r,e.ERR_MSG)}ruleFinallyStateUpdate(){if(this.RULE_STACK.pop(),this.RULE_OCCURRENCE_STACK.pop(),this.cstFinallyStateUpdate(),this.RULE_STACK.length===0&&this.isAtEndOfInput()===!1){let e=this.LA(1),r=this.errorMessageProvider.buildNotAllInputParsedMessage({firstRedundant:e,ruleName:this.getCurrRuleFullName()});this.SAVE_ERROR(new px(r,e))}}subruleInternal(e,r,n){let i;try{let a=n!==void 0?n.ARGS:void 0;return this.subruleIdx=r,i=e.apply(this,a),this.cstPostNonTerminal(i,n!==void 0&&n.LABEL!==void 0?n.LABEL:e.ruleName),i}catch(a){throw this.subruleInternalError(a,n,e.ruleName)}}subruleInternalError(e,r,n){throw lf(e)&&e.partialCstResult!==void 0&&(this.cstPostNonTerminal(e.partialCstResult,r!==void 0&&r.LABEL!==void 0?r.LABEL:n),delete e.partialCstResult),e}consumeInternal(e,r,n){let i;try{let a=this.LA(1);this.tokenMatcher(a,e)===!0?(this.consumeToken(),i=a):this.consumeInternalError(e,a,n)}catch(a){i=this.consumeInternalRecovery(e,r,a)}return this.cstPostTerminal(n!==void 0&&n.LABEL!==void 0?n.LABEL:e.name,i),i}consumeInternalError(e,r,n){let i,a=this.LA(0);throw n!==void 0&&n.ERR_MSG?i=n.ERR_MSG:i=this.errorMessageProvider.buildMismatchTokenMessage({expected:e,actual:r,previous:a,ruleName:this.getCurrRuleFullName()}),this.SAVE_ERROR(new hp(i,r,a))}consumeInternalRecovery(e,r,n){if(this.recoveryEnabled&&n.name==="MismatchedTokenException"&&!this.isBackTracking()){let i=this.getFollowsForInRuleRecovery(e,r);try{return this.tryInRuleRecovery(e,i)}catch(a){throw a.name===zN?n:a}}else throw n}saveRecogState(){let e=this.errors,r=an(this.RULE_STACK);return{errors:e,lexerState:this.exportLexerState(),RULE_STACK:r,CST_STACK:this.CST_STACK}}reloadRecogState(e){this.errors=e.errors,this.importLexerState(e.lexerState),this.RULE_STACK=e.RULE_STACK}ruleInvocationStateUpdate(e,r,n){this.RULE_OCCURRENCE_STACK.push(n),this.RULE_STACK.push(e),this.cstInvocationStateUpdate(r)}isBackTracking(){return this.isBackTrackingStack.length!==0}getCurrRuleFullName(){let e=this.getLastExplicitRuleShortName();return this.shortRuleNameToFull[e]}shortRuleNameToFullName(e){return this.shortRuleNameToFull[e]}isAtEndOfInput(){return this.tokenMatcher(this.LA(1),lo)}reset(){this.resetLexerState(),this.subruleIdx=0,this.isBackTrackingStack=[],this.errors=[],this.RULE_STACK=[],this.CST_STACK=[],this.RULE_OCCURRENCE_STACK=[]}}});var rE,jse=N(()=>{"use strict";Xg();qt();qg();Fs();rE=class{static{o(this,"ErrorHandler")}initErrorHandler(e){this._errors=[],this.errorMessageProvider=Bt(e,"errorMessageProvider")?e.errorMessageProvider:ls.errorMessageProvider}SAVE_ERROR(e){if(lf(e))return e.context={ruleStack:this.getHumanReadableRuleStack(),ruleOccurrenceStack:an(this.RULE_OCCURRENCE_STACK)},this._errors.push(e),e;throw Error("Trying to save an Error which is not a RecognitionException")}get errors(){return an(this._errors)}set errors(e){this._errors=e}raiseEarlyExitException(e,r,n){let i=this.getCurrRuleFullName(),a=this.getGAstProductions()[i],l=Wg(e,a,r,this.maxLookahead)[0],u=[];for(let f=1;f<=this.maxLookahead;f++)u.push(this.LA(f));let h=this.errorMessageProvider.buildEarlyExitMessage({expectedIterationPaths:l,actual:u,previous:this.LA(0),customUserDescription:n,ruleName:i});throw this.SAVE_ERROR(new mx(h,this.LA(1),this.LA(0)))}raiseNoAltException(e,r){let n=this.getCurrRuleFullName(),i=this.getGAstProductions()[n],a=Hg(e,i,this.maxLookahead),s=[];for(let h=1;h<=this.maxLookahead;h++)s.push(this.LA(h));let l=this.LA(0),u=this.errorMessageProvider.buildNoViableAltMessage({expectedPathsPerAlt:a,actual:s,previous:l,customUserDescription:r,ruleName:this.getCurrRuleFullName()});throw this.SAVE_ERROR(new dx(u,this.LA(1),l))}}});var nE,Kse=N(()=>{"use strict";cx();qt();nE=class{static{o(this,"ContentAssist")}initContentAssist(){}computeContentAssist(e,r){let n=this.gastProductionsCache[e];if(pr(n))throw Error(`Rule ->${e}<- does not exist in this grammar.`);return $k([n],r,this.tokenMatcher,this.maxLookahead)}getNextPossibleTokenTypes(e){let r=ia(e.ruleStack),i=this.getGAstProductions()[r];return new Ok(i,e).startWalking()}}});function yx(t,e,r,n=!1){aE(r);let i=ga(this.recordingProdStack),a=Si(e)?e:e.DEF,s=new t({definition:[],idx:r});return n&&(s.separator=e.SEP),Bt(e,"MAX_LOOKAHEAD")&&(s.maxLookahead=e.MAX_LOOKAHEAD),this.recordingProdStack.push(s),a.call(this),i.definition.push(s),this.recordingProdStack.pop(),sE}function dBe(t,e){aE(e);let r=ga(this.recordingProdStack),n=Pt(t)===!1,i=n===!1?t:t.DEF,a=new Tn({definition:[],idx:e,ignoreAmbiguities:n&&t.IGNORE_AMBIGUITIES===!0});Bt(t,"MAX_LOOKAHEAD")&&(a.maxLookahead=t.MAX_LOOKAHEAD);let s=A2(i,l=>Si(l.GATE));return a.hasPredicates=s,r.definition.push(a),Ae(i,l=>{let u=new Dn({definition:[]});a.definition.push(u),Bt(l,"IGNORE_AMBIGUITIES")?u.ignoreAmbiguities=l.IGNORE_AMBIGUITIES:Bt(l,"GATE")&&(u.ignoreAmbiguities=!0),this.recordingProdStack.push(u),l.ALT.call(this),this.recordingProdStack.pop()}),sE}function Jse(t){return t===0?"":`${t}`}function aE(t){if(t<0||t>Zse){let e=new Error(`Invalid DSL Method idx value: <${t}> + Idx value must be a none negative value smaller than ${Zse+1}`);throw e.KNOWN_RECORDER_ERROR=!0,e}}var sE,Qse,Zse,eoe,toe,fBe,iE,roe=N(()=>{"use strict";qt();os();ix();cp();up();Fs();qk();sE={description:"This Object indicates the Parser is during Recording Phase"};Object.freeze(sE);Qse=!0,Zse=Math.pow(2,8)-1,eoe=of({name:"RECORDING_PHASE_TOKEN",pattern:Xn.NA});Bu([eoe]);toe=$u(eoe,`This IToken indicates the Parser is in Recording Phase + See: https://chevrotain.io/docs/guide/internals.html#grammar-recording for details`,-1,-1,-1,-1,-1,-1);Object.freeze(toe);fBe={name:`This CSTNode indicates the Parser is in Recording Phase + See: https://chevrotain.io/docs/guide/internals.html#grammar-recording for details`,children:{}},iE=class{static{o(this,"GastRecorder")}initGastRecorder(e){this.recordingProdStack=[],this.RECORDING_PHASE=!1}enableRecording(){this.RECORDING_PHASE=!0,this.TRACE_INIT("Enable Recording",()=>{for(let e=0;e<10;e++){let r=e>0?e:"";this[`CONSUME${r}`]=function(n,i){return this.consumeInternalRecord(n,e,i)},this[`SUBRULE${r}`]=function(n,i){return this.subruleInternalRecord(n,e,i)},this[`OPTION${r}`]=function(n){return this.optionInternalRecord(n,e)},this[`OR${r}`]=function(n){return this.orInternalRecord(n,e)},this[`MANY${r}`]=function(n){this.manyInternalRecord(e,n)},this[`MANY_SEP${r}`]=function(n){this.manySepFirstInternalRecord(e,n)},this[`AT_LEAST_ONE${r}`]=function(n){this.atLeastOneInternalRecord(e,n)},this[`AT_LEAST_ONE_SEP${r}`]=function(n){this.atLeastOneSepFirstInternalRecord(e,n)}}this.consume=function(e,r,n){return this.consumeInternalRecord(r,e,n)},this.subrule=function(e,r,n){return this.subruleInternalRecord(r,e,n)},this.option=function(e,r){return this.optionInternalRecord(r,e)},this.or=function(e,r){return this.orInternalRecord(r,e)},this.many=function(e,r){this.manyInternalRecord(e,r)},this.atLeastOne=function(e,r){this.atLeastOneInternalRecord(e,r)},this.ACTION=this.ACTION_RECORD,this.BACKTRACK=this.BACKTRACK_RECORD,this.LA=this.LA_RECORD})}disableRecording(){this.RECORDING_PHASE=!1,this.TRACE_INIT("Deleting Recording methods",()=>{let e=this;for(let r=0;r<10;r++){let n=r>0?r:"";delete e[`CONSUME${n}`],delete e[`SUBRULE${n}`],delete e[`OPTION${n}`],delete e[`OR${n}`],delete e[`MANY${n}`],delete e[`MANY_SEP${n}`],delete e[`AT_LEAST_ONE${n}`],delete e[`AT_LEAST_ONE_SEP${n}`]}delete e.consume,delete e.subrule,delete e.option,delete e.or,delete e.many,delete e.atLeastOne,delete e.ACTION,delete e.BACKTRACK,delete e.LA})}ACTION_RECORD(e){}BACKTRACK_RECORD(e,r){return()=>!0}LA_RECORD(e){return jg}topLevelRuleRecord(e,r){try{let n=new as({definition:[],name:e});return n.name=e,this.recordingProdStack.push(n),r.call(this),this.recordingProdStack.pop(),n}catch(n){if(n.KNOWN_RECORDER_ERROR!==!0)try{n.message=n.message+` + This error was thrown during the "grammar recording phase" For more info see: + https://chevrotain.io/docs/guide/internals.html#grammar-recording`}catch{throw n}throw n}}optionInternalRecord(e,r){return yx.call(this,ln,e,r)}atLeastOneInternalRecord(e,r){yx.call(this,Ln,r,e)}atLeastOneSepFirstInternalRecord(e,r){yx.call(this,Rn,r,e,Qse)}manyInternalRecord(e,r){yx.call(this,Or,r,e)}manySepFirstInternalRecord(e,r){yx.call(this,wn,r,e,Qse)}orInternalRecord(e,r){return dBe.call(this,e,r)}subruleInternalRecord(e,r,n){if(aE(r),!e||Bt(e,"ruleName")===!1){let l=new Error(` argument is invalid expecting a Parser method reference but got: <${JSON.stringify(e)}> + inside top level rule: <${this.recordingProdStack[0].name}>`);throw l.KNOWN_RECORDER_ERROR=!0,l}let i=ga(this.recordingProdStack),a=e.ruleName,s=new on({idx:r,nonTerminalName:a,label:n?.LABEL,referencedRule:void 0});return i.definition.push(s),this.outputCst?fBe:sE}consumeInternalRecord(e,r,n){if(aE(r),!_N(e)){let s=new Error(` argument is invalid expecting a TokenType reference but got: <${JSON.stringify(e)}> + inside top level rule: <${this.recordingProdStack[0].name}>`);throw s.KNOWN_RECORDER_ERROR=!0,s}let i=ga(this.recordingProdStack),a=new kr({idx:r,terminalType:e,label:n?.LABEL});return i.definition.push(a),toe}};o(yx,"recordProd");o(dBe,"recordOrProd");o(Jse,"getIdxSuffix");o(aE,"assertMethodIdxIsValid")});var oE,noe=N(()=>{"use strict";qt();Og();Fs();oE=class{static{o(this,"PerformanceTracer")}initPerformanceTracer(e){if(Bt(e,"traceInitPerf")){let r=e.traceInitPerf,n=typeof r=="number";this.traceInitMaxIdent=n?r:1/0,this.traceInitPerf=n?r>0:r}else this.traceInitMaxIdent=0,this.traceInitPerf=ls.traceInitPerf;this.traceInitIndent=-1}TRACE_INIT(e,r){if(this.traceInitPerf===!0){this.traceInitIndent++;let n=new Array(this.traceInitIndent+1).join(" ");this.traceInitIndent <${e}>`);let{time:i,value:a}=tx(r),s=i>10?console.warn:console.log;return this.traceInitIndent time: ${i}ms`),this.traceInitIndent--,a}else return r()}}});function ioe(t,e){e.forEach(r=>{let n=r.prototype;Object.getOwnPropertyNames(n).forEach(i=>{if(i==="constructor")return;let a=Object.getOwnPropertyDescriptor(n,i);a&&(a.get||a.set)?Object.defineProperty(t.prototype,i,a):t.prototype[i]=r.prototype[i]})})}var aoe=N(()=>{"use strict";o(ioe,"applyMixins")});function lE(t=void 0){return function(){return t}}var jg,ls,Kg,zi,vx,xx,Fs=N(()=>{"use strict";qt();Og();Oae();up();Vg();Rse();GN();Bse();Wse();qse();Yse();Xse();jse();Kse();roe();noe();aoe();fx();jg=$u(lo,"",NaN,NaN,NaN,NaN,NaN,NaN);Object.freeze(jg);ls=Object.freeze({recoveryEnabled:!1,maxLookahead:3,dynamicTokensEnabled:!1,outputCst:!0,errorMessageProvider:zu,nodeLocationTracking:"none",traceInitPerf:!1,skipValidations:!1}),Kg=Object.freeze({recoveryValueFunc:o(()=>{},"recoveryValueFunc"),resyncEnabled:!0});(function(t){t[t.INVALID_RULE_NAME=0]="INVALID_RULE_NAME",t[t.DUPLICATE_RULE_NAME=1]="DUPLICATE_RULE_NAME",t[t.INVALID_RULE_OVERRIDE=2]="INVALID_RULE_OVERRIDE",t[t.DUPLICATE_PRODUCTIONS=3]="DUPLICATE_PRODUCTIONS",t[t.UNRESOLVED_SUBRULE_REF=4]="UNRESOLVED_SUBRULE_REF",t[t.LEFT_RECURSION=5]="LEFT_RECURSION",t[t.NONE_LAST_EMPTY_ALT=6]="NONE_LAST_EMPTY_ALT",t[t.AMBIGUOUS_ALTS=7]="AMBIGUOUS_ALTS",t[t.CONFLICT_TOKENS_RULES_NAMESPACE=8]="CONFLICT_TOKENS_RULES_NAMESPACE",t[t.INVALID_TOKEN_NAME=9]="INVALID_TOKEN_NAME",t[t.NO_NON_EMPTY_LOOKAHEAD=10]="NO_NON_EMPTY_LOOKAHEAD",t[t.AMBIGUOUS_PREFIX_ALTS=11]="AMBIGUOUS_PREFIX_ALTS",t[t.TOO_MANY_ALTS=12]="TOO_MANY_ALTS",t[t.CUSTOM_LOOKAHEAD_VALIDATION=13]="CUSTOM_LOOKAHEAD_VALIDATION"})(zi||(zi={}));o(lE,"EMPTY_ALT");vx=class t{static{o(this,"Parser")}static performSelfAnalysis(e){throw Error("The **static** `performSelfAnalysis` method has been deprecated. \nUse the **instance** method with the same name instead.")}performSelfAnalysis(){this.TRACE_INIT("performSelfAnalysis",()=>{let e;this.selfAnalysisDone=!0;let r=this.className;this.TRACE_INIT("toFastProps",()=>{rx(this)}),this.TRACE_INIT("Grammar Recording",()=>{try{this.enableRecording(),Ae(this.definedRulesNames,i=>{let s=this[i].originalGrammarAction,l;this.TRACE_INIT(`${i} Rule`,()=>{l=this.topLevelRuleRecord(i,s)}),this.gastProductionsCache[i]=l})}finally{this.disableRecording()}});let n=[];if(this.TRACE_INIT("Grammar Resolving",()=>{n=Dse({rules:br(this.gastProductionsCache)}),this.definitionErrors=this.definitionErrors.concat(n)}),this.TRACE_INIT("Grammar Validations",()=>{if(ur(n)&&this.skipValidations===!1){let i=Lse({rules:br(this.gastProductionsCache),tokenTypes:br(this.tokensMap),errMsgProvider:Pl,grammarName:r}),a=Tse({lookaheadStrategy:this.lookaheadStrategy,rules:br(this.gastProductionsCache),tokenTypes:br(this.tokensMap),grammarName:r});this.definitionErrors=this.definitionErrors.concat(i,a)}}),ur(this.definitionErrors)&&(this.recoveryEnabled&&this.TRACE_INIT("computeAllProdsFollows",()=>{let i=Iae(br(this.gastProductionsCache));this.resyncFollows=i}),this.TRACE_INIT("ComputeLookaheadFunctions",()=>{var i,a;(a=(i=this.lookaheadStrategy).initialize)===null||a===void 0||a.call(i,{rules:br(this.gastProductionsCache)}),this.preComputeLookaheadFunctions(br(this.gastProductionsCache))})),!t.DEFER_DEFINITION_ERRORS_HANDLING&&!ur(this.definitionErrors))throw e=Je(this.definitionErrors,i=>i.message),new Error(`Parser Definition Errors detected: + ${e.join(` +------------------------------- +`)}`)})}constructor(e,r){this.definitionErrors=[],this.selfAnalysisDone=!1;let n=this;if(n.initErrorHandler(r),n.initLexerAdapter(),n.initLooksAhead(r),n.initRecognizerEngine(e,r),n.initRecoverable(r),n.initTreeBuilder(r),n.initContentAssist(),n.initGastRecorder(r),n.initPerformanceTracer(r),Bt(r,"ignoredIssues"))throw new Error(`The IParserConfig property has been deprecated. + Please use the flag on the relevant DSL method instead. + See: https://chevrotain.io/docs/guide/resolving_grammar_errors.html#IGNORING_AMBIGUITIES + For further details.`);this.skipValidations=Bt(r,"skipValidations")?r.skipValidations:ls.skipValidations}};vx.DEFER_DEFINITION_ERRORS_HANDLING=!1;ioe(vx,[Hk,Xk,Zk,Jk,tE,eE,rE,nE,iE,oE]);xx=class extends vx{static{o(this,"EmbeddedActionsParser")}constructor(e,r=ls){let n=an(r);n.outputCst=!1,super(e,n)}}});var soe=N(()=>{"use strict";os()});var ooe=N(()=>{"use strict"});var loe=N(()=>{"use strict";soe();ooe()});var coe=N(()=>{"use strict";gN()});var cf=N(()=>{"use strict";gN();Fs();ix();up();qg();VN();Vg();Xg();DN();os();os();loe();coe()});function fp(t,e,r){return`${t.name}_${e}_${r}`}function doe(t){let e={decisionMap:{},decisionStates:[],ruleToStartState:new Map,ruleToStopState:new Map,states:[]};bBe(e,t);let r=t.length;for(let n=0;npoe(t,e,s));return e1(t,e,n,r,...i)}function CBe(t,e,r){let n=aa(t,e,r,{type:uf});hf(t,n);let i=e1(t,e,n,r,dp(t,e,r));return ABe(t,e,r,i)}function dp(t,e,r){let n=Yr(Je(r.definition,i=>poe(t,e,i)),i=>i!==void 0);return n.length===1?n[0]:n.length===0?void 0:DBe(t,n)}function moe(t,e,r,n,i){let a=n.left,s=n.right,l=aa(t,e,r,{type:xBe});hf(t,l);let u=aa(t,e,r,{type:foe});return a.loopback=l,u.loopback=l,t.decisionMap[fp(e,i?"RepetitionMandatoryWithSeparator":"RepetitionMandatory",r.idx)]=l,Ai(s,l),i===void 0?(Ai(l,a),Ai(l,u)):(Ai(l,u),Ai(l,i.left),Ai(i.right,a)),{left:a,right:u}}function goe(t,e,r,n,i){let a=n.left,s=n.right,l=aa(t,e,r,{type:vBe});hf(t,l);let u=aa(t,e,r,{type:foe}),h=aa(t,e,r,{type:yBe});return l.loopback=h,u.loopback=h,Ai(l,a),Ai(l,u),Ai(s,h),i!==void 0?(Ai(h,u),Ai(h,i.left),Ai(i.right,a)):Ai(h,l),t.decisionMap[fp(e,i?"RepetitionWithSeparator":"Repetition",r.idx)]=l,{left:l,right:u}}function ABe(t,e,r,n){let i=n.left,a=n.right;return Ai(i,a),t.decisionMap[fp(e,"Option",r.idx)]=i,n}function hf(t,e){return t.decisionStates.push(e),e.decision=t.decisionStates.length-1,e.decision}function e1(t,e,r,n,...i){let a=aa(t,e,n,{type:gBe,start:r});r.end=a;for(let l of i)l!==void 0?(Ai(r,l.left),Ai(l.right,a)):Ai(r,a);let s={left:r,right:a};return t.decisionMap[fp(e,_Be(n),n.idx)]=r,s}function _Be(t){if(t instanceof Tn)return"Alternation";if(t instanceof ln)return"Option";if(t instanceof Or)return"Repetition";if(t instanceof wn)return"RepetitionWithSeparator";if(t instanceof Ln)return"RepetitionMandatory";if(t instanceof Rn)return"RepetitionMandatoryWithSeparator";throw new Error("Invalid production type encountered")}function DBe(t,e){let r=e.length;for(let a=0;a{"use strict";Im();DL();cf();o(fp,"buildATNKey");uf=1,mBe=2,uoe=4,hoe=5,Jg=7,gBe=8,yBe=9,vBe=10,xBe=11,foe=12,bx=class{static{o(this,"AbstractTransition")}constructor(e){this.target=e}isEpsilon(){return!1}},Qg=class extends bx{static{o(this,"AtomTransition")}constructor(e,r){super(e),this.tokenType=r}},wx=class extends bx{static{o(this,"EpsilonTransition")}constructor(e){super(e)}isEpsilon(){return!0}},Zg=class extends bx{static{o(this,"RuleTransition")}constructor(e,r,n){super(e),this.rule=r,this.followState=n}isEpsilon(){return!0}};o(doe,"createATN");o(bBe,"createRuleStartAndStopATNStates");o(poe,"atom");o(wBe,"repetition");o(TBe,"repetitionSep");o(kBe,"repetitionMandatory");o(EBe,"repetitionMandatorySep");o(SBe,"alternation");o(CBe,"option");o(dp,"block");o(moe,"plus");o(goe,"star");o(ABe,"optional");o(hf,"defineDecisionState");o(e1,"makeAlts");o(_Be,"getProdType");o(DBe,"makeBlock");o(QN,"tokenRef");o(LBe,"ruleRef");o(RBe,"buildRuleHandle");o(Ai,"epsilon");o(aa,"newState");o(ZN,"addTransition");o(NBe,"removeState")});function JN(t,e=!0){return`${e?`a${t.alt}`:""}s${t.state.stateNumber}:${t.stack.map(r=>r.stateNumber.toString()).join("_")}`}var Tx,t1,voe=N(()=>{"use strict";Im();Tx={},t1=class{static{o(this,"ATNConfigSet")}constructor(){this.map={},this.configs=[]}get size(){return this.configs.length}finalize(){this.map={}}add(e){let r=JN(e);r in this.map||(this.map[r]=this.configs.length,this.configs.push(e))}get elements(){return this.configs}get alts(){return Je(this.configs,e=>e.alt)}get key(){let e="";for(let r in this.map)e+=r+":";return e}};o(JN,"getATNConfigKey")});function MBe(t,e){let r={};return n=>{let i=n.toString(),a=r[i];return a!==void 0||(a={atnStartState:t,decision:e,states:{}},r[i]=a),a}}function boe(t,e=!0){let r=new Set;for(let n of t){let i=new Set;for(let a of n){if(a===void 0){if(e)break;return!1}let s=[a.tokenTypeIdx].concat(a.categoryMatches);for(let l of s)if(r.has(l)){if(!i.has(l))return!1}else r.add(l),i.add(l)}}return!0}function IBe(t){let e=t.decisionStates.length,r=Array(e);for(let n=0;nFu(i)).join(", "),r=t.production.idx===0?"":t.production.idx,n=`Ambiguous Alternatives Detected: <${t.ambiguityIndices.join(", ")}> in <${$Be(t.production)}${r}> inside <${t.topLevelRule.name}> Rule, +<${e}> may appears as a prefix path in all these alternatives. +`;return n=n+`See: https://chevrotain.io/docs/guide/resolving_grammar_errors.html#AMBIGUOUS_ALTERNATIVES +For Further details.`,n}function $Be(t){if(t instanceof on)return"SUBRULE";if(t instanceof ln)return"OPTION";if(t instanceof Tn)return"OR";if(t instanceof Ln)return"AT_LEAST_ONE";if(t instanceof Rn)return"AT_LEAST_ONE_SEP";if(t instanceof wn)return"MANY_SEP";if(t instanceof Or)return"MANY";if(t instanceof kr)return"CONSUME";throw Error("non exhaustive match")}function zBe(t,e,r){let n=ya(e.configs.elements,a=>a.state.transitions),i=Qre(n.filter(a=>a instanceof Qg).map(a=>a.tokenType),a=>a.tokenTypeIdx);return{actualToken:r,possibleTokenTypes:i,tokenPath:t}}function GBe(t,e){return t.edges[e.tokenTypeIdx]}function VBe(t,e,r){let n=new t1,i=[];for(let s of t.elements){if(r.is(s.alt)===!1)continue;if(s.state.type===Jg){i.push(s);continue}let l=s.state.transitions.length;for(let u=0;u0&&!YBe(a))for(let s of i)a.add(s);return a}function UBe(t,e){if(t instanceof Qg&&sx(e,t.tokenType))return t.target}function HBe(t,e){let r;for(let n of t.elements)if(e.is(n.alt)===!0){if(r===void 0)r=n.alt;else if(r!==n.alt)return}return r}function Toe(t){return{configs:t,edges:{},isAcceptState:!1,prediction:-1}}function woe(t,e,r,n){return n=koe(t,n),e.edges[r.tokenTypeIdx]=n,n}function koe(t,e){if(e===Tx)return e;let r=e.configs.key,n=t.states[r];return n!==void 0?n:(e.configs.finalize(),t.states[r]=e,e)}function WBe(t){let e=new t1,r=t.transitions.length;for(let n=0;n0){let i=[...t.stack],s={state:i.pop(),alt:t.alt,stack:i};uE(s,e)}else e.add(t);return}r.epsilonOnlyTransitions||e.add(t);let n=r.transitions.length;for(let i=0;i1)return!0;return!1}function ZBe(t){for(let e of Array.from(t.values()))if(Object.keys(e).length===1)return!0;return!1}var cE,xoe,kx,Eoe=N(()=>{"use strict";cf();yoe();voe();BL();RL();Zre();Im();uT();$T();HT();GL();o(MBe,"createDFACache");cE=class{static{o(this,"PredicateSet")}constructor(){this.predicates=[]}is(e){return e>=this.predicates.length||this.predicates[e]}set(e,r){this.predicates[e]=r}toString(){let e="",r=this.predicates.length;for(let n=0;nconsole.log(n)}initialize(e){this.atn=doe(e.rules),this.dfas=IBe(this.atn)}validateAmbiguousAlternationAlternatives(){return[]}validateEmptyOrAlternatives(){return[]}buildLookaheadForAlternation(e){let{prodOccurrence:r,rule:n,hasPredicates:i,dynamicTokensEnabled:a}=e,s=this.dfas,l=this.logging,u=fp(n,"Alternation",r),f=this.atn.decisionMap[u].decision,d=Je(Gk({maxLookahead:1,occurrence:r,prodType:"Alternation",rule:n}),p=>Je(p,m=>m[0]));if(boe(d,!1)&&!a){let p=Xr(d,(m,g,y)=>(Ae(g,v=>{v&&(m[v.tokenTypeIdx]=y,Ae(v.categoryMatches,x=>{m[x]=y}))}),m),{});return i?function(m){var g;let y=this.LA(1),v=p[y.tokenTypeIdx];if(m!==void 0&&v!==void 0){let x=(g=m[v])===null||g===void 0?void 0:g.GATE;if(x!==void 0&&x.call(this)===!1)return}return v}:function(){let m=this.LA(1);return p[m.tokenTypeIdx]}}else return i?function(p){let m=new cE,g=p===void 0?0:p.length;for(let v=0;vJe(p,m=>m[0]));if(boe(d)&&d[0][0]&&!a){let p=d[0],m=qr(p);if(m.length===1&&ur(m[0].categoryMatches)){let y=m[0].tokenTypeIdx;return function(){return this.LA(1).tokenTypeIdx===y}}else{let g=Xr(m,(y,v)=>(v!==void 0&&(y[v.tokenTypeIdx]=!0,Ae(v.categoryMatches,x=>{y[x]=!0})),y),{});return function(){let y=this.LA(1);return g[y.tokenTypeIdx]===!0}}}return function(){let p=eM.call(this,s,f,xoe,l);return typeof p=="object"?!1:p===0}}};o(boe,"isLL1Sequence");o(IBe,"initATNSimulator");o(eM,"adaptivePredict");o(OBe,"performLookahead");o(PBe,"computeLookaheadTarget");o(BBe,"reportLookaheadAmbiguity");o(FBe,"buildAmbiguityError");o($Be,"getProductionDslName");o(zBe,"buildAdaptivePredictError");o(GBe,"getExistingTargetState");o(VBe,"computeReachSet");o(UBe,"getReachableTarget");o(HBe,"getUniqueAlt");o(Toe,"newDFAState");o(woe,"addDFAEdge");o(koe,"addDFAState");o(WBe,"computeStartState");o(uE,"closure");o(qBe,"getEpsilonTarget");o(YBe,"hasConfigInRuleStopState");o(XBe,"allConfigsInRuleStopStates");o(jBe,"hasConflictTerminatingPrediction");o(KBe,"getConflictingAltSets");o(QBe,"hasConflictingAltSet");o(ZBe,"hasStateAssociatedWithOneAlt")});var Soe=N(()=>{"use strict";Eoe()});var Coe,tM,Aoe,hE,jr,Pr,fE,_oe,rM,Doe,Loe,Roe,Noe,nM,Moe,Ioe,Ooe,dE,r1,n1,iM,i1,Poe,aM,sM,oM,lM,cM,Boe,Foe,uM,$oe,hM,Ex,zoe,Goe,Voe,Uoe,Hoe,Woe,qoe,Yoe,pE,Xoe,joe,Koe,Qoe,Zoe,Joe,ele,tle,rle,nle,ile,mE,ale,sle,ole,lle,cle,ule,hle,fle,dle,ple,mle,gle,yle,fM,dM,vle,xle,ble,wle,Tle,kle,Ele,Sle,Cle,pM,Fe,mM=N(()=>{"use strict";(function(t){function e(r){return typeof r=="string"}o(e,"is"),t.is=e})(Coe||(Coe={}));(function(t){function e(r){return typeof r=="string"}o(e,"is"),t.is=e})(tM||(tM={}));(function(t){t.MIN_VALUE=-2147483648,t.MAX_VALUE=2147483647;function e(r){return typeof r=="number"&&t.MIN_VALUE<=r&&r<=t.MAX_VALUE}o(e,"is"),t.is=e})(Aoe||(Aoe={}));(function(t){t.MIN_VALUE=0,t.MAX_VALUE=2147483647;function e(r){return typeof r=="number"&&t.MIN_VALUE<=r&&r<=t.MAX_VALUE}o(e,"is"),t.is=e})(hE||(hE={}));(function(t){function e(n,i){return n===Number.MAX_VALUE&&(n=hE.MAX_VALUE),i===Number.MAX_VALUE&&(i=hE.MAX_VALUE),{line:n,character:i}}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.objectLiteral(i)&&Fe.uinteger(i.line)&&Fe.uinteger(i.character)}o(r,"is"),t.is=r})(jr||(jr={}));(function(t){function e(n,i,a,s){if(Fe.uinteger(n)&&Fe.uinteger(i)&&Fe.uinteger(a)&&Fe.uinteger(s))return{start:jr.create(n,i),end:jr.create(a,s)};if(jr.is(n)&&jr.is(i))return{start:n,end:i};throw new Error(`Range#create called with invalid arguments[${n}, ${i}, ${a}, ${s}]`)}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.objectLiteral(i)&&jr.is(i.start)&&jr.is(i.end)}o(r,"is"),t.is=r})(Pr||(Pr={}));(function(t){function e(n,i){return{uri:n,range:i}}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.objectLiteral(i)&&Pr.is(i.range)&&(Fe.string(i.uri)||Fe.undefined(i.uri))}o(r,"is"),t.is=r})(fE||(fE={}));(function(t){function e(n,i,a,s){return{targetUri:n,targetRange:i,targetSelectionRange:a,originSelectionRange:s}}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.objectLiteral(i)&&Pr.is(i.targetRange)&&Fe.string(i.targetUri)&&Pr.is(i.targetSelectionRange)&&(Pr.is(i.originSelectionRange)||Fe.undefined(i.originSelectionRange))}o(r,"is"),t.is=r})(_oe||(_oe={}));(function(t){function e(n,i,a,s){return{red:n,green:i,blue:a,alpha:s}}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.objectLiteral(i)&&Fe.numberRange(i.red,0,1)&&Fe.numberRange(i.green,0,1)&&Fe.numberRange(i.blue,0,1)&&Fe.numberRange(i.alpha,0,1)}o(r,"is"),t.is=r})(rM||(rM={}));(function(t){function e(n,i){return{range:n,color:i}}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.objectLiteral(i)&&Pr.is(i.range)&&rM.is(i.color)}o(r,"is"),t.is=r})(Doe||(Doe={}));(function(t){function e(n,i,a){return{label:n,textEdit:i,additionalTextEdits:a}}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.objectLiteral(i)&&Fe.string(i.label)&&(Fe.undefined(i.textEdit)||n1.is(i))&&(Fe.undefined(i.additionalTextEdits)||Fe.typedArray(i.additionalTextEdits,n1.is))}o(r,"is"),t.is=r})(Loe||(Loe={}));(function(t){t.Comment="comment",t.Imports="imports",t.Region="region"})(Roe||(Roe={}));(function(t){function e(n,i,a,s,l,u){let h={startLine:n,endLine:i};return Fe.defined(a)&&(h.startCharacter=a),Fe.defined(s)&&(h.endCharacter=s),Fe.defined(l)&&(h.kind=l),Fe.defined(u)&&(h.collapsedText=u),h}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.objectLiteral(i)&&Fe.uinteger(i.startLine)&&Fe.uinteger(i.startLine)&&(Fe.undefined(i.startCharacter)||Fe.uinteger(i.startCharacter))&&(Fe.undefined(i.endCharacter)||Fe.uinteger(i.endCharacter))&&(Fe.undefined(i.kind)||Fe.string(i.kind))}o(r,"is"),t.is=r})(Noe||(Noe={}));(function(t){function e(n,i){return{location:n,message:i}}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.defined(i)&&fE.is(i.location)&&Fe.string(i.message)}o(r,"is"),t.is=r})(nM||(nM={}));(function(t){t.Error=1,t.Warning=2,t.Information=3,t.Hint=4})(Moe||(Moe={}));(function(t){t.Unnecessary=1,t.Deprecated=2})(Ioe||(Ioe={}));(function(t){function e(r){let n=r;return Fe.objectLiteral(n)&&Fe.string(n.href)}o(e,"is"),t.is=e})(Ooe||(Ooe={}));(function(t){function e(n,i,a,s,l,u){let h={range:n,message:i};return Fe.defined(a)&&(h.severity=a),Fe.defined(s)&&(h.code=s),Fe.defined(l)&&(h.source=l),Fe.defined(u)&&(h.relatedInformation=u),h}o(e,"create"),t.create=e;function r(n){var i;let a=n;return Fe.defined(a)&&Pr.is(a.range)&&Fe.string(a.message)&&(Fe.number(a.severity)||Fe.undefined(a.severity))&&(Fe.integer(a.code)||Fe.string(a.code)||Fe.undefined(a.code))&&(Fe.undefined(a.codeDescription)||Fe.string((i=a.codeDescription)===null||i===void 0?void 0:i.href))&&(Fe.string(a.source)||Fe.undefined(a.source))&&(Fe.undefined(a.relatedInformation)||Fe.typedArray(a.relatedInformation,nM.is))}o(r,"is"),t.is=r})(dE||(dE={}));(function(t){function e(n,i,...a){let s={title:n,command:i};return Fe.defined(a)&&a.length>0&&(s.arguments=a),s}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.defined(i)&&Fe.string(i.title)&&Fe.string(i.command)}o(r,"is"),t.is=r})(r1||(r1={}));(function(t){function e(a,s){return{range:a,newText:s}}o(e,"replace"),t.replace=e;function r(a,s){return{range:{start:a,end:a},newText:s}}o(r,"insert"),t.insert=r;function n(a){return{range:a,newText:""}}o(n,"del"),t.del=n;function i(a){let s=a;return Fe.objectLiteral(s)&&Fe.string(s.newText)&&Pr.is(s.range)}o(i,"is"),t.is=i})(n1||(n1={}));(function(t){function e(n,i,a){let s={label:n};return i!==void 0&&(s.needsConfirmation=i),a!==void 0&&(s.description=a),s}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.objectLiteral(i)&&Fe.string(i.label)&&(Fe.boolean(i.needsConfirmation)||i.needsConfirmation===void 0)&&(Fe.string(i.description)||i.description===void 0)}o(r,"is"),t.is=r})(iM||(iM={}));(function(t){function e(r){let n=r;return Fe.string(n)}o(e,"is"),t.is=e})(i1||(i1={}));(function(t){function e(a,s,l){return{range:a,newText:s,annotationId:l}}o(e,"replace"),t.replace=e;function r(a,s,l){return{range:{start:a,end:a},newText:s,annotationId:l}}o(r,"insert"),t.insert=r;function n(a,s){return{range:a,newText:"",annotationId:s}}o(n,"del"),t.del=n;function i(a){let s=a;return n1.is(s)&&(iM.is(s.annotationId)||i1.is(s.annotationId))}o(i,"is"),t.is=i})(Poe||(Poe={}));(function(t){function e(n,i){return{textDocument:n,edits:i}}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.defined(i)&&uM.is(i.textDocument)&&Array.isArray(i.edits)}o(r,"is"),t.is=r})(aM||(aM={}));(function(t){function e(n,i,a){let s={kind:"create",uri:n};return i!==void 0&&(i.overwrite!==void 0||i.ignoreIfExists!==void 0)&&(s.options=i),a!==void 0&&(s.annotationId=a),s}o(e,"create"),t.create=e;function r(n){let i=n;return i&&i.kind==="create"&&Fe.string(i.uri)&&(i.options===void 0||(i.options.overwrite===void 0||Fe.boolean(i.options.overwrite))&&(i.options.ignoreIfExists===void 0||Fe.boolean(i.options.ignoreIfExists)))&&(i.annotationId===void 0||i1.is(i.annotationId))}o(r,"is"),t.is=r})(sM||(sM={}));(function(t){function e(n,i,a,s){let l={kind:"rename",oldUri:n,newUri:i};return a!==void 0&&(a.overwrite!==void 0||a.ignoreIfExists!==void 0)&&(l.options=a),s!==void 0&&(l.annotationId=s),l}o(e,"create"),t.create=e;function r(n){let i=n;return i&&i.kind==="rename"&&Fe.string(i.oldUri)&&Fe.string(i.newUri)&&(i.options===void 0||(i.options.overwrite===void 0||Fe.boolean(i.options.overwrite))&&(i.options.ignoreIfExists===void 0||Fe.boolean(i.options.ignoreIfExists)))&&(i.annotationId===void 0||i1.is(i.annotationId))}o(r,"is"),t.is=r})(oM||(oM={}));(function(t){function e(n,i,a){let s={kind:"delete",uri:n};return i!==void 0&&(i.recursive!==void 0||i.ignoreIfNotExists!==void 0)&&(s.options=i),a!==void 0&&(s.annotationId=a),s}o(e,"create"),t.create=e;function r(n){let i=n;return i&&i.kind==="delete"&&Fe.string(i.uri)&&(i.options===void 0||(i.options.recursive===void 0||Fe.boolean(i.options.recursive))&&(i.options.ignoreIfNotExists===void 0||Fe.boolean(i.options.ignoreIfNotExists)))&&(i.annotationId===void 0||i1.is(i.annotationId))}o(r,"is"),t.is=r})(lM||(lM={}));(function(t){function e(r){let n=r;return n&&(n.changes!==void 0||n.documentChanges!==void 0)&&(n.documentChanges===void 0||n.documentChanges.every(i=>Fe.string(i.kind)?sM.is(i)||oM.is(i)||lM.is(i):aM.is(i)))}o(e,"is"),t.is=e})(cM||(cM={}));(function(t){function e(n){return{uri:n}}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.defined(i)&&Fe.string(i.uri)}o(r,"is"),t.is=r})(Boe||(Boe={}));(function(t){function e(n,i){return{uri:n,version:i}}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.defined(i)&&Fe.string(i.uri)&&Fe.integer(i.version)}o(r,"is"),t.is=r})(Foe||(Foe={}));(function(t){function e(n,i){return{uri:n,version:i}}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.defined(i)&&Fe.string(i.uri)&&(i.version===null||Fe.integer(i.version))}o(r,"is"),t.is=r})(uM||(uM={}));(function(t){function e(n,i,a,s){return{uri:n,languageId:i,version:a,text:s}}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.defined(i)&&Fe.string(i.uri)&&Fe.string(i.languageId)&&Fe.integer(i.version)&&Fe.string(i.text)}o(r,"is"),t.is=r})($oe||($oe={}));(function(t){t.PlainText="plaintext",t.Markdown="markdown";function e(r){let n=r;return n===t.PlainText||n===t.Markdown}o(e,"is"),t.is=e})(hM||(hM={}));(function(t){function e(r){let n=r;return Fe.objectLiteral(r)&&hM.is(n.kind)&&Fe.string(n.value)}o(e,"is"),t.is=e})(Ex||(Ex={}));(function(t){t.Text=1,t.Method=2,t.Function=3,t.Constructor=4,t.Field=5,t.Variable=6,t.Class=7,t.Interface=8,t.Module=9,t.Property=10,t.Unit=11,t.Value=12,t.Enum=13,t.Keyword=14,t.Snippet=15,t.Color=16,t.File=17,t.Reference=18,t.Folder=19,t.EnumMember=20,t.Constant=21,t.Struct=22,t.Event=23,t.Operator=24,t.TypeParameter=25})(zoe||(zoe={}));(function(t){t.PlainText=1,t.Snippet=2})(Goe||(Goe={}));(function(t){t.Deprecated=1})(Voe||(Voe={}));(function(t){function e(n,i,a){return{newText:n,insert:i,replace:a}}o(e,"create"),t.create=e;function r(n){let i=n;return i&&Fe.string(i.newText)&&Pr.is(i.insert)&&Pr.is(i.replace)}o(r,"is"),t.is=r})(Uoe||(Uoe={}));(function(t){t.asIs=1,t.adjustIndentation=2})(Hoe||(Hoe={}));(function(t){function e(r){let n=r;return n&&(Fe.string(n.detail)||n.detail===void 0)&&(Fe.string(n.description)||n.description===void 0)}o(e,"is"),t.is=e})(Woe||(Woe={}));(function(t){function e(r){return{label:r}}o(e,"create"),t.create=e})(qoe||(qoe={}));(function(t){function e(r,n){return{items:r||[],isIncomplete:!!n}}o(e,"create"),t.create=e})(Yoe||(Yoe={}));(function(t){function e(n){return n.replace(/[\\`*_{}[\]()#+\-.!]/g,"\\$&")}o(e,"fromPlainText"),t.fromPlainText=e;function r(n){let i=n;return Fe.string(i)||Fe.objectLiteral(i)&&Fe.string(i.language)&&Fe.string(i.value)}o(r,"is"),t.is=r})(pE||(pE={}));(function(t){function e(r){let n=r;return!!n&&Fe.objectLiteral(n)&&(Ex.is(n.contents)||pE.is(n.contents)||Fe.typedArray(n.contents,pE.is))&&(r.range===void 0||Pr.is(r.range))}o(e,"is"),t.is=e})(Xoe||(Xoe={}));(function(t){function e(r,n){return n?{label:r,documentation:n}:{label:r}}o(e,"create"),t.create=e})(joe||(joe={}));(function(t){function e(r,n,...i){let a={label:r};return Fe.defined(n)&&(a.documentation=n),Fe.defined(i)?a.parameters=i:a.parameters=[],a}o(e,"create"),t.create=e})(Koe||(Koe={}));(function(t){t.Text=1,t.Read=2,t.Write=3})(Qoe||(Qoe={}));(function(t){function e(r,n){let i={range:r};return Fe.number(n)&&(i.kind=n),i}o(e,"create"),t.create=e})(Zoe||(Zoe={}));(function(t){t.File=1,t.Module=2,t.Namespace=3,t.Package=4,t.Class=5,t.Method=6,t.Property=7,t.Field=8,t.Constructor=9,t.Enum=10,t.Interface=11,t.Function=12,t.Variable=13,t.Constant=14,t.String=15,t.Number=16,t.Boolean=17,t.Array=18,t.Object=19,t.Key=20,t.Null=21,t.EnumMember=22,t.Struct=23,t.Event=24,t.Operator=25,t.TypeParameter=26})(Joe||(Joe={}));(function(t){t.Deprecated=1})(ele||(ele={}));(function(t){function e(r,n,i,a,s){let l={name:r,kind:n,location:{uri:a,range:i}};return s&&(l.containerName=s),l}o(e,"create"),t.create=e})(tle||(tle={}));(function(t){function e(r,n,i,a){return a!==void 0?{name:r,kind:n,location:{uri:i,range:a}}:{name:r,kind:n,location:{uri:i}}}o(e,"create"),t.create=e})(rle||(rle={}));(function(t){function e(n,i,a,s,l,u){let h={name:n,detail:i,kind:a,range:s,selectionRange:l};return u!==void 0&&(h.children=u),h}o(e,"create"),t.create=e;function r(n){let i=n;return i&&Fe.string(i.name)&&Fe.number(i.kind)&&Pr.is(i.range)&&Pr.is(i.selectionRange)&&(i.detail===void 0||Fe.string(i.detail))&&(i.deprecated===void 0||Fe.boolean(i.deprecated))&&(i.children===void 0||Array.isArray(i.children))&&(i.tags===void 0||Array.isArray(i.tags))}o(r,"is"),t.is=r})(nle||(nle={}));(function(t){t.Empty="",t.QuickFix="quickfix",t.Refactor="refactor",t.RefactorExtract="refactor.extract",t.RefactorInline="refactor.inline",t.RefactorRewrite="refactor.rewrite",t.Source="source",t.SourceOrganizeImports="source.organizeImports",t.SourceFixAll="source.fixAll"})(ile||(ile={}));(function(t){t.Invoked=1,t.Automatic=2})(mE||(mE={}));(function(t){function e(n,i,a){let s={diagnostics:n};return i!=null&&(s.only=i),a!=null&&(s.triggerKind=a),s}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.defined(i)&&Fe.typedArray(i.diagnostics,dE.is)&&(i.only===void 0||Fe.typedArray(i.only,Fe.string))&&(i.triggerKind===void 0||i.triggerKind===mE.Invoked||i.triggerKind===mE.Automatic)}o(r,"is"),t.is=r})(ale||(ale={}));(function(t){function e(n,i,a){let s={title:n},l=!0;return typeof i=="string"?(l=!1,s.kind=i):r1.is(i)?s.command=i:s.edit=i,l&&a!==void 0&&(s.kind=a),s}o(e,"create"),t.create=e;function r(n){let i=n;return i&&Fe.string(i.title)&&(i.diagnostics===void 0||Fe.typedArray(i.diagnostics,dE.is))&&(i.kind===void 0||Fe.string(i.kind))&&(i.edit!==void 0||i.command!==void 0)&&(i.command===void 0||r1.is(i.command))&&(i.isPreferred===void 0||Fe.boolean(i.isPreferred))&&(i.edit===void 0||cM.is(i.edit))}o(r,"is"),t.is=r})(sle||(sle={}));(function(t){function e(n,i){let a={range:n};return Fe.defined(i)&&(a.data=i),a}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.defined(i)&&Pr.is(i.range)&&(Fe.undefined(i.command)||r1.is(i.command))}o(r,"is"),t.is=r})(ole||(ole={}));(function(t){function e(n,i){return{tabSize:n,insertSpaces:i}}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.defined(i)&&Fe.uinteger(i.tabSize)&&Fe.boolean(i.insertSpaces)}o(r,"is"),t.is=r})(lle||(lle={}));(function(t){function e(n,i,a){return{range:n,target:i,data:a}}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.defined(i)&&Pr.is(i.range)&&(Fe.undefined(i.target)||Fe.string(i.target))}o(r,"is"),t.is=r})(cle||(cle={}));(function(t){function e(n,i){return{range:n,parent:i}}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.objectLiteral(i)&&Pr.is(i.range)&&(i.parent===void 0||t.is(i.parent))}o(r,"is"),t.is=r})(ule||(ule={}));(function(t){t.namespace="namespace",t.type="type",t.class="class",t.enum="enum",t.interface="interface",t.struct="struct",t.typeParameter="typeParameter",t.parameter="parameter",t.variable="variable",t.property="property",t.enumMember="enumMember",t.event="event",t.function="function",t.method="method",t.macro="macro",t.keyword="keyword",t.modifier="modifier",t.comment="comment",t.string="string",t.number="number",t.regexp="regexp",t.operator="operator",t.decorator="decorator"})(hle||(hle={}));(function(t){t.declaration="declaration",t.definition="definition",t.readonly="readonly",t.static="static",t.deprecated="deprecated",t.abstract="abstract",t.async="async",t.modification="modification",t.documentation="documentation",t.defaultLibrary="defaultLibrary"})(fle||(fle={}));(function(t){function e(r){let n=r;return Fe.objectLiteral(n)&&(n.resultId===void 0||typeof n.resultId=="string")&&Array.isArray(n.data)&&(n.data.length===0||typeof n.data[0]=="number")}o(e,"is"),t.is=e})(dle||(dle={}));(function(t){function e(n,i){return{range:n,text:i}}o(e,"create"),t.create=e;function r(n){let i=n;return i!=null&&Pr.is(i.range)&&Fe.string(i.text)}o(r,"is"),t.is=r})(ple||(ple={}));(function(t){function e(n,i,a){return{range:n,variableName:i,caseSensitiveLookup:a}}o(e,"create"),t.create=e;function r(n){let i=n;return i!=null&&Pr.is(i.range)&&Fe.boolean(i.caseSensitiveLookup)&&(Fe.string(i.variableName)||i.variableName===void 0)}o(r,"is"),t.is=r})(mle||(mle={}));(function(t){function e(n,i){return{range:n,expression:i}}o(e,"create"),t.create=e;function r(n){let i=n;return i!=null&&Pr.is(i.range)&&(Fe.string(i.expression)||i.expression===void 0)}o(r,"is"),t.is=r})(gle||(gle={}));(function(t){function e(n,i){return{frameId:n,stoppedLocation:i}}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.defined(i)&&Pr.is(n.stoppedLocation)}o(r,"is"),t.is=r})(yle||(yle={}));(function(t){t.Type=1,t.Parameter=2;function e(r){return r===1||r===2}o(e,"is"),t.is=e})(fM||(fM={}));(function(t){function e(n){return{value:n}}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.objectLiteral(i)&&(i.tooltip===void 0||Fe.string(i.tooltip)||Ex.is(i.tooltip))&&(i.location===void 0||fE.is(i.location))&&(i.command===void 0||r1.is(i.command))}o(r,"is"),t.is=r})(dM||(dM={}));(function(t){function e(n,i,a){let s={position:n,label:i};return a!==void 0&&(s.kind=a),s}o(e,"create"),t.create=e;function r(n){let i=n;return Fe.objectLiteral(i)&&jr.is(i.position)&&(Fe.string(i.label)||Fe.typedArray(i.label,dM.is))&&(i.kind===void 0||fM.is(i.kind))&&i.textEdits===void 0||Fe.typedArray(i.textEdits,n1.is)&&(i.tooltip===void 0||Fe.string(i.tooltip)||Ex.is(i.tooltip))&&(i.paddingLeft===void 0||Fe.boolean(i.paddingLeft))&&(i.paddingRight===void 0||Fe.boolean(i.paddingRight))}o(r,"is"),t.is=r})(vle||(vle={}));(function(t){function e(r){return{kind:"snippet",value:r}}o(e,"createSnippet"),t.createSnippet=e})(xle||(xle={}));(function(t){function e(r,n,i,a){return{insertText:r,filterText:n,range:i,command:a}}o(e,"create"),t.create=e})(ble||(ble={}));(function(t){function e(r){return{items:r}}o(e,"create"),t.create=e})(wle||(wle={}));(function(t){t.Invoked=0,t.Automatic=1})(Tle||(Tle={}));(function(t){function e(r,n){return{range:r,text:n}}o(e,"create"),t.create=e})(kle||(kle={}));(function(t){function e(r,n){return{triggerKind:r,selectedCompletionInfo:n}}o(e,"create"),t.create=e})(Ele||(Ele={}));(function(t){function e(r){let n=r;return Fe.objectLiteral(n)&&tM.is(n.uri)&&Fe.string(n.name)}o(e,"is"),t.is=e})(Sle||(Sle={}));(function(t){function e(a,s,l,u){return new pM(a,s,l,u)}o(e,"create"),t.create=e;function r(a){let s=a;return!!(Fe.defined(s)&&Fe.string(s.uri)&&(Fe.undefined(s.languageId)||Fe.string(s.languageId))&&Fe.uinteger(s.lineCount)&&Fe.func(s.getText)&&Fe.func(s.positionAt)&&Fe.func(s.offsetAt))}o(r,"is"),t.is=r;function n(a,s){let l=a.getText(),u=i(s,(f,d)=>{let p=f.range.start.line-d.range.start.line;return p===0?f.range.start.character-d.range.start.character:p}),h=l.length;for(let f=u.length-1;f>=0;f--){let d=u[f],p=a.offsetAt(d.range.start),m=a.offsetAt(d.range.end);if(m<=h)l=l.substring(0,p)+d.newText+l.substring(m,l.length);else throw new Error("Overlapping edit");h=p}return l}o(n,"applyEdits"),t.applyEdits=n;function i(a,s){if(a.length<=1)return a;let l=a.length/2|0,u=a.slice(0,l),h=a.slice(l);i(u,s),i(h,s);let f=0,d=0,p=0;for(;f0&&e.push(r.length),this._lineOffsets=e}return this._lineOffsets}positionAt(e){e=Math.max(Math.min(e,this._content.length),0);let r=this.getLineOffsets(),n=0,i=r.length;if(i===0)return jr.create(0,e);for(;ne?i=s:n=s+1}let a=n-1;return jr.create(a,e-r[a])}offsetAt(e){let r=this.getLineOffsets();if(e.line>=r.length)return this._content.length;if(e.line<0)return 0;let n=r[e.line],i=e.line+1"u"}o(n,"undefined"),t.undefined=n;function i(m){return m===!0||m===!1}o(i,"boolean"),t.boolean=i;function a(m){return e.call(m)==="[object String]"}o(a,"string"),t.string=a;function s(m){return e.call(m)==="[object Number]"}o(s,"number"),t.number=s;function l(m,g,y){return e.call(m)==="[object Number]"&&g<=m&&m<=y}o(l,"numberRange"),t.numberRange=l;function u(m){return e.call(m)==="[object Number]"&&-2147483648<=m&&m<=2147483647}o(u,"integer"),t.integer=u;function h(m){return e.call(m)==="[object Number]"&&0<=m&&m<=2147483647}o(h,"uinteger"),t.uinteger=h;function f(m){return e.call(m)==="[object Function]"}o(f,"func"),t.func=f;function d(m){return m!==null&&typeof m=="object"}o(d,"objectLiteral"),t.objectLiteral=d;function p(m,g){return Array.isArray(m)&&m.every(g)}o(p,"typedArray"),t.typedArray=p})(Fe||(Fe={}))});var Sx,Cx,pp,mp,gM,a1,gE=N(()=>{"use strict";mM();Nl();Sx=class{static{o(this,"CstNodeBuilder")}constructor(){this.nodeStack=[]}get current(){var e;return(e=this.nodeStack[this.nodeStack.length-1])!==null&&e!==void 0?e:this.rootNode}buildRootNode(e){return this.rootNode=new a1(e),this.rootNode.root=this.rootNode,this.nodeStack=[this.rootNode],this.rootNode}buildCompositeNode(e){let r=new mp;return r.grammarSource=e,r.root=this.rootNode,this.current.content.push(r),this.nodeStack.push(r),r}buildLeafNode(e,r){let n=new pp(e.startOffset,e.image.length,Gm(e),e.tokenType,!r);return n.grammarSource=r,n.root=this.rootNode,this.current.content.push(n),n}removeNode(e){let r=e.container;if(r){let n=r.content.indexOf(e);n>=0&&r.content.splice(n,1)}}addHiddenNodes(e){let r=[];for(let a of e){let s=new pp(a.startOffset,a.image.length,Gm(a),a.tokenType,!0);s.root=this.rootNode,r.push(s)}let n=this.current,i=!1;if(n.content.length>0){n.content.push(...r);return}for(;n.container;){let a=n.container.content.indexOf(n);if(a>0){n.container.content.splice(a,0,...r),i=!0;break}n=n.container}i||this.rootNode.content.unshift(...r)}construct(e){let r=this.current;typeof e.$type=="string"&&(this.current.astNode=e),e.$cstNode=r;let n=this.nodeStack.pop();n?.content.length===0&&this.removeNode(n)}},Cx=class{static{o(this,"AbstractCstNode")}get parent(){return this.container}get feature(){return this.grammarSource}get hidden(){return!1}get astNode(){var e,r;let n=typeof((e=this._astNode)===null||e===void 0?void 0:e.$type)=="string"?this._astNode:(r=this.container)===null||r===void 0?void 0:r.astNode;if(!n)throw new Error("This node has no associated AST element");return n}set astNode(e){this._astNode=e}get element(){return this.astNode}get text(){return this.root.fullText.substring(this.offset,this.end)}},pp=class extends Cx{static{o(this,"LeafCstNodeImpl")}get offset(){return this._offset}get length(){return this._length}get end(){return this._offset+this._length}get hidden(){return this._hidden}get tokenType(){return this._tokenType}get range(){return this._range}constructor(e,r,n,i,a=!1){super(),this._hidden=a,this._offset=e,this._tokenType=i,this._length=r,this._range=n}},mp=class extends Cx{static{o(this,"CompositeCstNodeImpl")}constructor(){super(...arguments),this.content=new gM(this)}get children(){return this.content}get offset(){var e,r;return(r=(e=this.firstNonHiddenNode)===null||e===void 0?void 0:e.offset)!==null&&r!==void 0?r:0}get length(){return this.end-this.offset}get end(){var e,r;return(r=(e=this.lastNonHiddenNode)===null||e===void 0?void 0:e.end)!==null&&r!==void 0?r:0}get range(){let e=this.firstNonHiddenNode,r=this.lastNonHiddenNode;if(e&&r){if(this._rangeCache===void 0){let{range:n}=e,{range:i}=r;this._rangeCache={start:n.start,end:i.end.line=0;e--){let r=this.content[e];if(!r.hidden)return r}return this.content[this.content.length-1]}},gM=class t extends Array{static{o(this,"CstNodeContainer")}constructor(e){super(),this.parent=e,Object.setPrototypeOf(this,t.prototype)}push(...e){return this.addParents(e),super.push(...e)}unshift(...e){return this.addParents(e),super.unshift(...e)}splice(e,r,...n){return this.addParents(n),super.splice(e,r,...n)}addParents(e){for(let r of e)r.container=this.parent}},a1=class extends mp{static{o(this,"RootCstNodeImpl")}get text(){return this._text.substring(this.offset,this.end)}get fullText(){return this._text}constructor(e){super(),this._text="",this._text=e??""}}});function yM(t){return t.$type===yE}var yE,Ale,_le,Ax,_x,vE,s1,Dx,JBe,vM,Lx=N(()=>{"use strict";cf();Soe();Rc();Ol();is();gE();yE=Symbol("Datatype");o(yM,"isDataTypeNode");Ale="\u200B",_le=o(t=>t.endsWith(Ale)?t:t+Ale,"withRuleSuffix"),Ax=class{static{o(this,"AbstractLangiumParser")}constructor(e){this._unorderedGroups=new Map,this.allRules=new Map,this.lexer=e.parser.Lexer;let r=this.lexer.definition,n=e.LanguageMetaData.mode==="production";this.wrapper=new vM(r,Object.assign(Object.assign({},e.parser.ParserConfig),{skipValidations:n,errorMessageProvider:e.parser.ParserErrorMessageProvider}))}alternatives(e,r){this.wrapper.wrapOr(e,r)}optional(e,r){this.wrapper.wrapOption(e,r)}many(e,r){this.wrapper.wrapMany(e,r)}atLeastOne(e,r){this.wrapper.wrapAtLeastOne(e,r)}getRule(e){return this.allRules.get(e)}isRecording(){return this.wrapper.IS_RECORDING}get unorderedGroups(){return this._unorderedGroups}getRuleStack(){return this.wrapper.RULE_STACK}finalize(){this.wrapper.wrapSelfAnalysis()}},_x=class extends Ax{static{o(this,"LangiumParser")}get current(){return this.stack[this.stack.length-1]}constructor(e){super(e),this.nodeBuilder=new Sx,this.stack=[],this.assignmentMap=new Map,this.linker=e.references.Linker,this.converter=e.parser.ValueConverter,this.astReflection=e.shared.AstReflection}rule(e,r){let n=this.computeRuleType(e),i=this.wrapper.DEFINE_RULE(_le(e.name),this.startImplementation(n,r).bind(this));return this.allRules.set(e.name,i),e.entry&&(this.mainRule=i),i}computeRuleType(e){if(!e.fragment){if(Z2(e))return yE;{let r=Rg(e);return r??e.name}}}parse(e,r={}){this.nodeBuilder.buildRootNode(e);let n=this.lexerResult=this.lexer.tokenize(e);this.wrapper.input=n.tokens;let i=r.rule?this.allRules.get(r.rule):this.mainRule;if(!i)throw new Error(r.rule?`No rule found with name '${r.rule}'`:"No main rule available.");let a=i.call(this.wrapper,{});return this.nodeBuilder.addHiddenNodes(n.hidden),this.unorderedGroups.clear(),this.lexerResult=void 0,{value:a,lexerErrors:n.errors,lexerReport:n.report,parserErrors:this.wrapper.errors}}startImplementation(e,r){return n=>{let i=!this.isRecording()&&e!==void 0;if(i){let s={$type:e};this.stack.push(s),e===yE&&(s.value="")}let a;try{a=r(n)}catch{a=void 0}return a===void 0&&i&&(a=this.construct()),a}}extractHiddenTokens(e){let r=this.lexerResult.hidden;if(!r.length)return[];let n=e.startOffset;for(let i=0;in)return r.splice(0,i);return r.splice(0,r.length)}consume(e,r,n){let i=this.wrapper.wrapConsume(e,r);if(!this.isRecording()&&this.isValidToken(i)){let a=this.extractHiddenTokens(i);this.nodeBuilder.addHiddenNodes(a);let s=this.nodeBuilder.buildLeafNode(i,n),{assignment:l,isCrossRef:u}=this.getAssignment(n),h=this.current;if(l){let f=Ho(n)?i.image:this.converter.convert(i.image,s);this.assign(l.operator,l.feature,f,s,u)}else if(yM(h)){let f=i.image;Ho(n)||(f=this.converter.convert(f,s).toString()),h.value+=f}}}isValidToken(e){return!e.isInsertedInRecovery&&!isNaN(e.startOffset)&&typeof e.endOffset=="number"&&!isNaN(e.endOffset)}subrule(e,r,n,i,a){let s;!this.isRecording()&&!n&&(s=this.nodeBuilder.buildCompositeNode(i));let l=this.wrapper.wrapSubrule(e,r,a);!this.isRecording()&&s&&s.length>0&&this.performSubruleAssignment(l,i,s)}performSubruleAssignment(e,r,n){let{assignment:i,isCrossRef:a}=this.getAssignment(r);if(i)this.assign(i.operator,i.feature,e,n,a);else if(!i){let s=this.current;if(yM(s))s.value+=e.toString();else if(typeof e=="object"&&e){let u=this.assignWithoutOverride(e,s);this.stack.pop(),this.stack.push(u)}}}action(e,r){if(!this.isRecording()){let n=this.current;if(r.feature&&r.operator){n=this.construct(),this.nodeBuilder.removeNode(n.$cstNode),this.nodeBuilder.buildCompositeNode(r).content.push(n.$cstNode);let a={$type:e};this.stack.push(a),this.assign(r.operator,r.feature,n,n.$cstNode,!1)}else n.$type=e}}construct(){if(this.isRecording())return;let e=this.current;return vk(e),this.nodeBuilder.construct(e),this.stack.pop(),yM(e)?this.converter.convert(e.value,e.$cstNode):(XR(this.astReflection,e),e)}getAssignment(e){if(!this.assignmentMap.has(e)){let r=tp(e,Ml);this.assignmentMap.set(e,{assignment:r,isCrossRef:r?ep(r.terminal):!1})}return this.assignmentMap.get(e)}assign(e,r,n,i,a){let s=this.current,l;switch(a&&typeof n=="string"?l=this.linker.buildReference(s,r,i,n):l=n,e){case"=":{s[r]=l;break}case"?=":{s[r]=!0;break}case"+=":Array.isArray(s[r])||(s[r]=[]),s[r].push(l)}}assignWithoutOverride(e,r){for(let[i,a]of Object.entries(r)){let s=e[i];s===void 0?e[i]=a:Array.isArray(s)&&Array.isArray(a)&&(a.push(...s),e[i]=a)}let n=e.$cstNode;return n&&(n.astNode=void 0,e.$cstNode=void 0),e}get definitionErrors(){return this.wrapper.definitionErrors}},vE=class{static{o(this,"AbstractParserErrorMessageProvider")}buildMismatchTokenMessage(e){return zu.buildMismatchTokenMessage(e)}buildNotAllInputParsedMessage(e){return zu.buildNotAllInputParsedMessage(e)}buildNoViableAltMessage(e){return zu.buildNoViableAltMessage(e)}buildEarlyExitMessage(e){return zu.buildEarlyExitMessage(e)}},s1=class extends vE{static{o(this,"LangiumParserErrorMessageProvider")}buildMismatchTokenMessage({expected:e,actual:r}){return`Expecting ${e.LABEL?"`"+e.LABEL+"`":e.name.endsWith(":KW")?`keyword '${e.name.substring(0,e.name.length-3)}'`:`token of type '${e.name}'`} but found \`${r.image}\`.`}buildNotAllInputParsedMessage({firstRedundant:e}){return`Expecting end of file but found \`${e.image}\`.`}},Dx=class extends Ax{static{o(this,"LangiumCompletionParser")}constructor(){super(...arguments),this.tokens=[],this.elementStack=[],this.lastElementStack=[],this.nextTokenIndex=0,this.stackSize=0}action(){}construct(){}parse(e){this.resetState();let r=this.lexer.tokenize(e,{mode:"partial"});return this.tokens=r.tokens,this.wrapper.input=[...this.tokens],this.mainRule.call(this.wrapper,{}),this.unorderedGroups.clear(),{tokens:this.tokens,elementStack:[...this.lastElementStack],tokenIndex:this.nextTokenIndex}}rule(e,r){let n=this.wrapper.DEFINE_RULE(_le(e.name),this.startImplementation(r).bind(this));return this.allRules.set(e.name,n),e.entry&&(this.mainRule=n),n}resetState(){this.elementStack=[],this.lastElementStack=[],this.nextTokenIndex=0,this.stackSize=0}startImplementation(e){return r=>{let n=this.keepStackSize();try{e(r)}finally{this.resetStackSize(n)}}}removeUnexpectedElements(){this.elementStack.splice(this.stackSize)}keepStackSize(){let e=this.elementStack.length;return this.stackSize=e,e}resetStackSize(e){this.removeUnexpectedElements(),this.stackSize=e}consume(e,r,n){this.wrapper.wrapConsume(e,r),this.isRecording()||(this.lastElementStack=[...this.elementStack,n],this.nextTokenIndex=this.currIdx+1)}subrule(e,r,n,i,a){this.before(i),this.wrapper.wrapSubrule(e,r,a),this.after(i)}before(e){this.isRecording()||this.elementStack.push(e)}after(e){if(!this.isRecording()){let r=this.elementStack.lastIndexOf(e);r>=0&&this.elementStack.splice(r)}}get currIdx(){return this.wrapper.currIdx}},JBe={recoveryEnabled:!0,nodeLocationTracking:"full",skipValidations:!0,errorMessageProvider:new s1},vM=class extends xx{static{o(this,"ChevrotainWrapper")}constructor(e,r){let n=r&&"maxLookahead"in r;super(e,Object.assign(Object.assign(Object.assign({},JBe),{lookaheadStrategy:n?new Gu({maxLookahead:r.maxLookahead}):new kx({logging:r.skipValidations?()=>{}:void 0})}),r))}get IS_RECORDING(){return this.RECORDING_PHASE}DEFINE_RULE(e,r){return this.RULE(e,r)}wrapSelfAnalysis(){this.performSelfAnalysis()}wrapConsume(e,r){return this.consume(e,r)}wrapSubrule(e,r,n){return this.subrule(e,r,{ARGS:[n]})}wrapOr(e,r){this.or(e,r)}wrapOption(e,r){this.option(e,r)}wrapMany(e,r){this.many(e,r)}wrapAtLeastOne(e,r){this.atLeastOne(e,r)}}});function Rx(t,e,r){return eFe({parser:e,tokens:r,ruleNames:new Map},t),e}function eFe(t,e){let r=K2(e,!1),n=en(e.rules).filter(Oa).filter(i=>r.has(i));for(let i of n){let a=Object.assign(Object.assign({},t),{consume:1,optional:1,subrule:1,many:1,or:1});t.parser.rule(i,gp(a,i.definition))}}function gp(t,e,r=!1){let n;if(Ho(e))n=oFe(t,e);else if(Mu(e))n=tFe(t,e);else if(Ml(e))n=gp(t,e.terminal);else if(ep(e))n=Dle(t,e);else if(Il(e))n=rFe(t,e);else if(mk(e))n=iFe(t,e);else if(yk(e))n=aFe(t,e);else if(sf(e))n=sFe(t,e);else if($R(e)){let i=t.consume++;n=o(()=>t.parser.consume(i,lo,e),"method")}else throw new Zd(e.$cstNode,`Unexpected element type: ${e.$type}`);return Lle(t,r?void 0:xE(e),n,e.cardinality)}function tFe(t,e){let r=J2(e);return()=>t.parser.action(r,e)}function rFe(t,e){let r=e.rule.ref;if(Oa(r)){let n=t.subrule++,i=r.fragment,a=e.arguments.length>0?nFe(r,e.arguments):()=>({});return s=>t.parser.subrule(n,Rle(t,r),i,e,a(s))}else if(so(r)){let n=t.consume++,i=xM(t,r.name);return()=>t.parser.consume(n,i,e)}else if(r)Lc(r);else throw new Zd(e.$cstNode,`Undefined rule: ${e.rule.$refText}`)}function nFe(t,e){let r=e.map(n=>Vu(n.value));return n=>{let i={};for(let a=0;ae(n)||r(n)}else if(RR(t)){let e=Vu(t.left),r=Vu(t.right);return n=>e(n)&&r(n)}else if(MR(t)){let e=Vu(t.value);return r=>!e(r)}else if(IR(t)){let e=t.parameter.ref.name;return r=>r!==void 0&&r[e]===!0}else if(LR(t)){let e=!!t.true;return()=>e}Lc(t)}function iFe(t,e){if(e.elements.length===1)return gp(t,e.elements[0]);{let r=[];for(let i of e.elements){let a={ALT:gp(t,i,!0)},s=xE(i);s&&(a.GATE=Vu(s)),r.push(a)}let n=t.or++;return i=>t.parser.alternatives(n,r.map(a=>{let s={ALT:o(()=>a.ALT(i),"ALT")},l=a.GATE;return l&&(s.GATE=()=>l(i)),s}))}}function aFe(t,e){if(e.elements.length===1)return gp(t,e.elements[0]);let r=[];for(let l of e.elements){let u={ALT:gp(t,l,!0)},h=xE(l);h&&(u.GATE=Vu(h)),r.push(u)}let n=t.or++,i=o((l,u)=>{let h=u.getRuleStack().join("-");return`uGroup_${l}_${h}`},"idFunc"),a=o(l=>t.parser.alternatives(n,r.map((u,h)=>{let f={ALT:o(()=>!0,"ALT")},d=t.parser;f.ALT=()=>{if(u.ALT(l),!d.isRecording()){let m=i(n,d);d.unorderedGroups.get(m)||d.unorderedGroups.set(m,[]);let g=d.unorderedGroups.get(m);typeof g?.[h]>"u"&&(g[h]=!0)}};let p=u.GATE;return p?f.GATE=()=>p(l):f.GATE=()=>{let m=d.unorderedGroups.get(i(n,d));return!m?.[h]},f})),"alternatives"),s=Lle(t,xE(e),a,"*");return l=>{s(l),t.parser.isRecording()||t.parser.unorderedGroups.delete(i(n,t.parser))}}function sFe(t,e){let r=e.elements.map(n=>gp(t,n));return n=>r.forEach(i=>i(n))}function xE(t){if(sf(t))return t.guardCondition}function Dle(t,e,r=e.terminal){if(r)if(Il(r)&&Oa(r.rule.ref)){let n=r.rule.ref,i=t.subrule++;return a=>t.parser.subrule(i,Rle(t,n),!1,e,a)}else if(Il(r)&&so(r.rule.ref)){let n=t.consume++,i=xM(t,r.rule.ref.name);return()=>t.parser.consume(n,i,e)}else if(Ho(r)){let n=t.consume++,i=xM(t,r.value);return()=>t.parser.consume(n,i,e)}else throw new Error("Could not build cross reference parser");else{if(!e.type.ref)throw new Error("Could not resolve reference to type: "+e.type.$refText);let n=kk(e.type.ref),i=n?.terminal;if(!i)throw new Error("Could not find name assignment for type: "+J2(e.type.ref));return Dle(t,e,i)}}function oFe(t,e){let r=t.consume++,n=t.tokens[e.value];if(!n)throw new Error("Could not find token for keyword: "+e.value);return()=>t.parser.consume(r,n,e)}function Lle(t,e,r,n){let i=e&&Vu(e);if(!n)if(i){let a=t.or++;return s=>t.parser.alternatives(a,[{ALT:o(()=>r(s),"ALT"),GATE:o(()=>i(s),"GATE")},{ALT:lE(),GATE:o(()=>!i(s),"GATE")}])}else return r;if(n==="*"){let a=t.many++;return s=>t.parser.many(a,{DEF:o(()=>r(s),"DEF"),GATE:i?()=>i(s):void 0})}else if(n==="+"){let a=t.many++;if(i){let s=t.or++;return l=>t.parser.alternatives(s,[{ALT:o(()=>t.parser.atLeastOne(a,{DEF:o(()=>r(l),"DEF")}),"ALT"),GATE:o(()=>i(l),"GATE")},{ALT:lE(),GATE:o(()=>!i(l),"GATE")}])}else return s=>t.parser.atLeastOne(a,{DEF:o(()=>r(s),"DEF")})}else if(n==="?"){let a=t.optional++;return s=>t.parser.optional(a,{DEF:o(()=>r(s),"DEF"),GATE:i?()=>i(s):void 0})}else Lc(n)}function Rle(t,e){let r=lFe(t,e),n=t.parser.getRule(r);if(!n)throw new Error(`Rule "${r}" not found."`);return n}function lFe(t,e){if(Oa(e))return e.name;if(t.ruleNames.has(e))return t.ruleNames.get(e);{let r=e,n=r.$container,i=e.$type;for(;!Oa(n);)(sf(n)||mk(n)||yk(n))&&(i=n.elements.indexOf(r).toString()+":"+i),r=n,n=n.$container;return i=n.name+":"+i,t.ruleNames.set(e,i),i}}function xM(t,e){let r=t.tokens[e];if(!r)throw new Error(`Token "${e}" not found."`);return r}var bE=N(()=>{"use strict";cf();Rc();uk();Ps();Ol();o(Rx,"createParser");o(eFe,"buildRules");o(gp,"buildElement");o(tFe,"buildAction");o(rFe,"buildRuleCall");o(nFe,"buildRuleCallPredicate");o(Vu,"buildPredicate");o(iFe,"buildAlternatives");o(aFe,"buildUnorderedGroup");o(sFe,"buildGroup");o(xE,"getGuardCondition");o(Dle,"buildCrossReference");o(oFe,"buildKeyword");o(Lle,"wrap");o(Rle,"getRule");o(lFe,"getRuleName");o(xM,"getToken")});function bM(t){let e=t.Grammar,r=t.parser.Lexer,n=new Dx(t);return Rx(e,n,r.definition),n.finalize(),n}var wM=N(()=>{"use strict";Lx();bE();o(bM,"createCompletionParser")});function TM(t){let e=Nle(t);return e.finalize(),e}function Nle(t){let e=t.Grammar,r=t.parser.Lexer,n=new _x(t);return Rx(e,n,r.definition)}var kM=N(()=>{"use strict";Lx();bE();o(TM,"createLangiumParser");o(Nle,"prepareLangiumParser")});var Uu,wE=N(()=>{"use strict";cf();Rc();is();Ol();Lg();Ps();Uu=class{static{o(this,"DefaultTokenBuilder")}constructor(){this.diagnostics=[]}buildTokens(e,r){let n=en(K2(e,!1)),i=this.buildTerminalTokens(n),a=this.buildKeywordTokens(n,i,r);return i.forEach(s=>{let l=s.PATTERN;typeof l=="object"&&l&&"test"in l&&Dg(l)?a.unshift(s):a.push(s)}),a}flushLexingReport(e){return{diagnostics:this.popDiagnostics()}}popDiagnostics(){let e=[...this.diagnostics];return this.diagnostics=[],e}buildTerminalTokens(e){return e.filter(so).filter(r=>!r.fragment).map(r=>this.buildTerminalToken(r)).toArray()}buildTerminalToken(e){let r=Ng(e),n=this.requiresCustomPattern(r)?this.regexPatternFunction(r):r,i={name:e.name,PATTERN:n};return typeof n=="function"&&(i.LINE_BREAKS=!0),e.hidden&&(i.GROUP=Dg(r)?Xn.SKIPPED:"hidden"),i}requiresCustomPattern(e){return e.flags.includes("u")||e.flags.includes("s")?!0:!!(e.source.includes("?<=")||e.source.includes("?(r.lastIndex=i,r.exec(n))}buildKeywordTokens(e,r,n){return e.filter(Oa).flatMap(i=>Nc(i).filter(Ho)).distinct(i=>i.value).toArray().sort((i,a)=>a.value.length-i.value.length).map(i=>this.buildKeywordToken(i,r,!!n?.caseInsensitive))}buildKeywordToken(e,r,n){let i=this.buildKeywordPattern(e,n),a={name:e.value,PATTERN:i,LONGER_ALT:this.findLongerAlt(e,r)};return typeof i=="function"&&(a.LINE_BREAKS=!0),a}buildKeywordPattern(e,r){return r?new RegExp(tN(e.value)):e.value}findLongerAlt(e,r){return r.reduce((n,i)=>{let a=i?.PATTERN;return a?.source&&rN("^"+a.source+"$",e.value)&&n.push(i),n},[])}}});var yp,Oc,EM=N(()=>{"use strict";Rc();Ol();yp=class{static{o(this,"DefaultValueConverter")}convert(e,r){let n=r.grammarSource;if(ep(n)&&(n=aN(n)),Il(n)){let i=n.rule.ref;if(!i)throw new Error("This cst node was not parsed by a rule.");return this.runConverter(i,e,r)}return e}runConverter(e,r,n){var i;switch(e.name.toUpperCase()){case"INT":return Oc.convertInt(r);case"STRING":return Oc.convertString(r);case"ID":return Oc.convertID(r)}switch((i=fN(e))===null||i===void 0?void 0:i.toLowerCase()){case"number":return Oc.convertNumber(r);case"boolean":return Oc.convertBoolean(r);case"bigint":return Oc.convertBigint(r);case"date":return Oc.convertDate(r);default:return r}}};(function(t){function e(h){let f="";for(let d=1;d{"use strict";Object.defineProperty(AM,"__esModule",{value:!0});var SM;function CM(){if(SM===void 0)throw new Error("No runtime abstraction layer installed");return SM}o(CM,"RAL");(function(t){function e(r){if(r===void 0)throw new Error("No runtime abstraction layer provided");SM=r}o(e,"install"),t.install=e})(CM||(CM={}));AM.default=CM});var Ole=Mi(Ba=>{"use strict";Object.defineProperty(Ba,"__esModule",{value:!0});Ba.stringArray=Ba.array=Ba.func=Ba.error=Ba.number=Ba.string=Ba.boolean=void 0;function cFe(t){return t===!0||t===!1}o(cFe,"boolean");Ba.boolean=cFe;function Mle(t){return typeof t=="string"||t instanceof String}o(Mle,"string");Ba.string=Mle;function uFe(t){return typeof t=="number"||t instanceof Number}o(uFe,"number");Ba.number=uFe;function hFe(t){return t instanceof Error}o(hFe,"error");Ba.error=hFe;function fFe(t){return typeof t=="function"}o(fFe,"func");Ba.func=fFe;function Ile(t){return Array.isArray(t)}o(Ile,"array");Ba.array=Ile;function dFe(t){return Ile(t)&&t.every(e=>Mle(e))}o(dFe,"stringArray");Ba.stringArray=dFe});var LM=Mi(o1=>{"use strict";Object.defineProperty(o1,"__esModule",{value:!0});o1.Emitter=o1.Event=void 0;var pFe=_M(),Ple;(function(t){let e={dispose(){}};t.None=function(){return e}})(Ple||(o1.Event=Ple={}));var DM=class{static{o(this,"CallbackList")}add(e,r=null,n){this._callbacks||(this._callbacks=[],this._contexts=[]),this._callbacks.push(e),this._contexts.push(r),Array.isArray(n)&&n.push({dispose:o(()=>this.remove(e,r),"dispose")})}remove(e,r=null){if(!this._callbacks)return;let n=!1;for(let i=0,a=this._callbacks.length;i{this._callbacks||(this._callbacks=new DM),this._options&&this._options.onFirstListenerAdd&&this._callbacks.isEmpty()&&this._options.onFirstListenerAdd(this),this._callbacks.add(e,r);let i={dispose:o(()=>{this._callbacks&&(this._callbacks.remove(e,r),i.dispose=t._noop,this._options&&this._options.onLastListenerRemove&&this._callbacks.isEmpty()&&this._options.onLastListenerRemove(this))},"dispose")};return Array.isArray(n)&&n.push(i),i}),this._event}fire(e){this._callbacks&&this._callbacks.invoke.call(this._callbacks,e)}dispose(){this._callbacks&&(this._callbacks.dispose(),this._callbacks=void 0)}};o1.Emitter=TE;TE._noop=function(){}});var Ble=Mi(l1=>{"use strict";Object.defineProperty(l1,"__esModule",{value:!0});l1.CancellationTokenSource=l1.CancellationToken=void 0;var mFe=_M(),gFe=Ole(),RM=LM(),kE;(function(t){t.None=Object.freeze({isCancellationRequested:!1,onCancellationRequested:RM.Event.None}),t.Cancelled=Object.freeze({isCancellationRequested:!0,onCancellationRequested:RM.Event.None});function e(r){let n=r;return n&&(n===t.None||n===t.Cancelled||gFe.boolean(n.isCancellationRequested)&&!!n.onCancellationRequested)}o(e,"is"),t.is=e})(kE||(l1.CancellationToken=kE={}));var yFe=Object.freeze(function(t,e){let r=(0,mFe.default)().timer.setTimeout(t.bind(e),0);return{dispose(){r.dispose()}}}),EE=class{static{o(this,"MutableToken")}constructor(){this._isCancelled=!1}cancel(){this._isCancelled||(this._isCancelled=!0,this._emitter&&(this._emitter.fire(void 0),this.dispose()))}get isCancellationRequested(){return this._isCancelled}get onCancellationRequested(){return this._isCancelled?yFe:(this._emitter||(this._emitter=new RM.Emitter),this._emitter.event)}dispose(){this._emitter&&(this._emitter.dispose(),this._emitter=void 0)}},NM=class{static{o(this,"CancellationTokenSource")}get token(){return this._token||(this._token=new EE),this._token}cancel(){this._token?this._token.cancel():this._token=kE.Cancelled}dispose(){this._token?this._token instanceof EE&&this._token.dispose():this._token=kE.None}};l1.CancellationTokenSource=NM});var yr={};var qo=N(()=>{"use strict";Sr(yr,Sa(Ble(),1))});function MM(){return new Promise(t=>{typeof setImmediate>"u"?setTimeout(t,0):setImmediate(t)})}function CE(){return SE=performance.now(),new yr.CancellationTokenSource}function $le(t){Fle=t}function Bc(t){return t===Pc}async function xi(t){if(t===yr.CancellationToken.None)return;let e=performance.now();if(e-SE>=Fle&&(SE=e,await MM(),SE=performance.now()),t.isCancellationRequested)throw Pc}var SE,Fle,Pc,cs,Yo=N(()=>{"use strict";qo();o(MM,"delayNextTick");SE=0,Fle=10;o(CE,"startCancelableOperation");o($le,"setInterruptionPeriod");Pc=Symbol("OperationCancelled");o(Bc,"isOperationCancelled");o(xi,"interruptAndCheck");cs=class{static{o(this,"Deferred")}constructor(){this.promise=new Promise((e,r)=>{this.resolve=n=>(e(n),this),this.reject=n=>(r(n),this)})}}});function IM(t,e){if(t.length<=1)return t;let r=t.length/2|0,n=t.slice(0,r),i=t.slice(r);IM(n,e),IM(i,e);let a=0,s=0,l=0;for(;ar.line||e.line===r.line&&e.character>r.character?{start:r,end:e}:t}function vFe(t){let e=Vle(t.range);return e!==t.range?{newText:t.newText,range:e}:t}var AE,c1,Ule=N(()=>{"use strict";AE=class t{static{o(this,"FullTextDocument")}constructor(e,r,n,i){this._uri=e,this._languageId=r,this._version=n,this._content=i,this._lineOffsets=void 0}get uri(){return this._uri}get languageId(){return this._languageId}get version(){return this._version}getText(e){if(e){let r=this.offsetAt(e.start),n=this.offsetAt(e.end);return this._content.substring(r,n)}return this._content}update(e,r){for(let n of e)if(t.isIncremental(n)){let i=Vle(n.range),a=this.offsetAt(i.start),s=this.offsetAt(i.end);this._content=this._content.substring(0,a)+n.text+this._content.substring(s,this._content.length);let l=Math.max(i.start.line,0),u=Math.max(i.end.line,0),h=this._lineOffsets,f=zle(n.text,!1,a);if(u-l===f.length)for(let p=0,m=f.length;pe?i=s:n=s+1}let a=n-1;return e=this.ensureBeforeEOL(e,r[a]),{line:a,character:e-r[a]}}offsetAt(e){let r=this.getLineOffsets();if(e.line>=r.length)return this._content.length;if(e.line<0)return 0;let n=r[e.line];if(e.character<=0)return n;let i=e.line+1r&&Gle(this._content.charCodeAt(e-1));)e--;return e}get lineCount(){return this.getLineOffsets().length}static isIncremental(e){let r=e;return r!=null&&typeof r.text=="string"&&r.range!==void 0&&(r.rangeLength===void 0||typeof r.rangeLength=="number")}static isFull(e){let r=e;return r!=null&&typeof r.text=="string"&&r.range===void 0&&r.rangeLength===void 0}};(function(t){function e(i,a,s,l){return new AE(i,a,s,l)}o(e,"create"),t.create=e;function r(i,a,s){if(i instanceof AE)return i.update(a,s),i;throw new Error("TextDocument.update: document must be created by TextDocument.create")}o(r,"update"),t.update=r;function n(i,a){let s=i.getText(),l=IM(a.map(vFe),(f,d)=>{let p=f.range.start.line-d.range.start.line;return p===0?f.range.start.character-d.range.start.character:p}),u=0,h=[];for(let f of l){let d=i.offsetAt(f.range.start);if(du&&h.push(s.substring(u,d)),f.newText.length&&h.push(f.newText),u=i.offsetAt(f.range.end)}return h.push(s.substr(u)),h.join("")}o(n,"applyEdits"),t.applyEdits=n})(c1||(c1={}));o(IM,"mergeSort");o(zle,"computeLineOffsets");o(Gle,"isEOL");o(Vle,"getWellformedRange");o(vFe,"getWellformedEdit")});var Hle,us,u1,OM=N(()=>{"use strict";(()=>{"use strict";var t={470:i=>{function a(u){if(typeof u!="string")throw new TypeError("Path must be a string. Received "+JSON.stringify(u))}o(a,"e");function s(u,h){for(var f,d="",p=0,m=-1,g=0,y=0;y<=u.length;++y){if(y2){var v=d.lastIndexOf("/");if(v!==d.length-1){v===-1?(d="",p=0):p=(d=d.slice(0,v)).length-1-d.lastIndexOf("/"),m=y,g=0;continue}}else if(d.length===2||d.length===1){d="",p=0,m=y,g=0;continue}}h&&(d.length>0?d+="/..":d="..",p=2)}else d.length>0?d+="/"+u.slice(m+1,y):d=u.slice(m+1,y),p=y-m-1;m=y,g=0}else f===46&&g!==-1?++g:g=-1}return d}o(s,"r");var l={resolve:o(function(){for(var u,h="",f=!1,d=arguments.length-1;d>=-1&&!f;d--){var p;d>=0?p=arguments[d]:(u===void 0&&(u=process.cwd()),p=u),a(p),p.length!==0&&(h=p+"/"+h,f=p.charCodeAt(0)===47)}return h=s(h,!f),f?h.length>0?"/"+h:"/":h.length>0?h:"."},"resolve"),normalize:o(function(u){if(a(u),u.length===0)return".";var h=u.charCodeAt(0)===47,f=u.charCodeAt(u.length-1)===47;return(u=s(u,!h)).length!==0||h||(u="."),u.length>0&&f&&(u+="/"),h?"/"+u:u},"normalize"),isAbsolute:o(function(u){return a(u),u.length>0&&u.charCodeAt(0)===47},"isAbsolute"),join:o(function(){if(arguments.length===0)return".";for(var u,h=0;h0&&(u===void 0?u=f:u+="/"+f)}return u===void 0?".":l.normalize(u)},"join"),relative:o(function(u,h){if(a(u),a(h),u===h||(u=l.resolve(u))===(h=l.resolve(h)))return"";for(var f=1;fy){if(h.charCodeAt(m+x)===47)return h.slice(m+x+1);if(x===0)return h.slice(m+x)}else p>y&&(u.charCodeAt(f+x)===47?v=x:x===0&&(v=0));break}var b=u.charCodeAt(f+x);if(b!==h.charCodeAt(m+x))break;b===47&&(v=x)}var w="";for(x=f+v+1;x<=d;++x)x!==d&&u.charCodeAt(x)!==47||(w.length===0?w+="..":w+="/..");return w.length>0?w+h.slice(m+v):(m+=v,h.charCodeAt(m)===47&&++m,h.slice(m))},"relative"),_makeLong:o(function(u){return u},"_makeLong"),dirname:o(function(u){if(a(u),u.length===0)return".";for(var h=u.charCodeAt(0),f=h===47,d=-1,p=!0,m=u.length-1;m>=1;--m)if((h=u.charCodeAt(m))===47){if(!p){d=m;break}}else p=!1;return d===-1?f?"/":".":f&&d===1?"//":u.slice(0,d)},"dirname"),basename:o(function(u,h){if(h!==void 0&&typeof h!="string")throw new TypeError('"ext" argument must be a string');a(u);var f,d=0,p=-1,m=!0;if(h!==void 0&&h.length>0&&h.length<=u.length){if(h.length===u.length&&h===u)return"";var g=h.length-1,y=-1;for(f=u.length-1;f>=0;--f){var v=u.charCodeAt(f);if(v===47){if(!m){d=f+1;break}}else y===-1&&(m=!1,y=f+1),g>=0&&(v===h.charCodeAt(g)?--g==-1&&(p=f):(g=-1,p=y))}return d===p?p=y:p===-1&&(p=u.length),u.slice(d,p)}for(f=u.length-1;f>=0;--f)if(u.charCodeAt(f)===47){if(!m){d=f+1;break}}else p===-1&&(m=!1,p=f+1);return p===-1?"":u.slice(d,p)},"basename"),extname:o(function(u){a(u);for(var h=-1,f=0,d=-1,p=!0,m=0,g=u.length-1;g>=0;--g){var y=u.charCodeAt(g);if(y!==47)d===-1&&(p=!1,d=g+1),y===46?h===-1?h=g:m!==1&&(m=1):h!==-1&&(m=-1);else if(!p){f=g+1;break}}return h===-1||d===-1||m===0||m===1&&h===d-1&&h===f+1?"":u.slice(h,d)},"extname"),format:o(function(u){if(u===null||typeof u!="object")throw new TypeError('The "pathObject" argument must be of type Object. Received type '+typeof u);return function(h,f){var d=f.dir||f.root,p=f.base||(f.name||"")+(f.ext||"");return d?d===f.root?d+p:d+"/"+p:p}(0,u)},"format"),parse:o(function(u){a(u);var h={root:"",dir:"",base:"",ext:"",name:""};if(u.length===0)return h;var f,d=u.charCodeAt(0),p=d===47;p?(h.root="/",f=1):f=0;for(var m=-1,g=0,y=-1,v=!0,x=u.length-1,b=0;x>=f;--x)if((d=u.charCodeAt(x))!==47)y===-1&&(v=!1,y=x+1),d===46?m===-1?m=x:b!==1&&(b=1):m!==-1&&(b=-1);else if(!v){g=x+1;break}return m===-1||y===-1||b===0||b===1&&m===y-1&&m===g+1?y!==-1&&(h.base=h.name=g===0&&p?u.slice(1,y):u.slice(g,y)):(g===0&&p?(h.name=u.slice(1,m),h.base=u.slice(1,y)):(h.name=u.slice(g,m),h.base=u.slice(g,y)),h.ext=u.slice(m,y)),g>0?h.dir=u.slice(0,g-1):p&&(h.dir="/"),h},"parse"),sep:"/",delimiter:":",win32:null,posix:null};l.posix=l,i.exports=l}},e={};function r(i){var a=e[i];if(a!==void 0)return a.exports;var s=e[i]={exports:{}};return t[i](s,s.exports,r),s.exports}o(r,"r"),r.d=(i,a)=>{for(var s in a)r.o(a,s)&&!r.o(i,s)&&Object.defineProperty(i,s,{enumerable:!0,get:a[s]})},r.o=(i,a)=>Object.prototype.hasOwnProperty.call(i,a),r.r=i=>{typeof Symbol<"u"&&Symbol.toStringTag&&Object.defineProperty(i,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(i,"__esModule",{value:!0})};var n={};(()=>{let i;r.r(n),r.d(n,{URI:o(()=>p,"URI"),Utils:o(()=>I,"Utils")}),typeof process=="object"?i=process.platform==="win32":typeof navigator=="object"&&(i=navigator.userAgent.indexOf("Windows")>=0);let a=/^\w[\w\d+.-]*$/,s=/^\//,l=/^\/\//;function u(D,k){if(!D.scheme&&k)throw new Error(`[UriError]: Scheme is missing: {scheme: "", authority: "${D.authority}", path: "${D.path}", query: "${D.query}", fragment: "${D.fragment}"}`);if(D.scheme&&!a.test(D.scheme))throw new Error("[UriError]: Scheme contains illegal characters.");if(D.path){if(D.authority){if(!s.test(D.path))throw new Error('[UriError]: If a URI contains an authority component, then the path component must either be empty or begin with a slash ("/") character')}else if(l.test(D.path))throw new Error('[UriError]: If a URI does not contain an authority component, then the path cannot begin with two slash characters ("//")')}}o(u,"s");let h="",f="/",d=/^(([^:/?#]+?):)?(\/\/([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?/;class p{static{o(this,"f")}static isUri(k){return k instanceof p||!!k&&typeof k.authority=="string"&&typeof k.fragment=="string"&&typeof k.path=="string"&&typeof k.query=="string"&&typeof k.scheme=="string"&&typeof k.fsPath=="string"&&typeof k.with=="function"&&typeof k.toString=="function"}scheme;authority;path;query;fragment;constructor(k,L,R,O,M,B=!1){typeof k=="object"?(this.scheme=k.scheme||h,this.authority=k.authority||h,this.path=k.path||h,this.query=k.query||h,this.fragment=k.fragment||h):(this.scheme=function(F,P){return F||P?F:"file"}(k,B),this.authority=L||h,this.path=function(F,P){switch(F){case"https":case"http":case"file":P?P[0]!==f&&(P=f+P):P=f}return P}(this.scheme,R||h),this.query=O||h,this.fragment=M||h,u(this,B))}get fsPath(){return b(this,!1)}with(k){if(!k)return this;let{scheme:L,authority:R,path:O,query:M,fragment:B}=k;return L===void 0?L=this.scheme:L===null&&(L=h),R===void 0?R=this.authority:R===null&&(R=h),O===void 0?O=this.path:O===null&&(O=h),M===void 0?M=this.query:M===null&&(M=h),B===void 0?B=this.fragment:B===null&&(B=h),L===this.scheme&&R===this.authority&&O===this.path&&M===this.query&&B===this.fragment?this:new g(L,R,O,M,B)}static parse(k,L=!1){let R=d.exec(k);return R?new g(R[2]||h,E(R[4]||h),E(R[5]||h),E(R[7]||h),E(R[9]||h),L):new g(h,h,h,h,h)}static file(k){let L=h;if(i&&(k=k.replace(/\\/g,f)),k[0]===f&&k[1]===f){let R=k.indexOf(f,2);R===-1?(L=k.substring(2),k=f):(L=k.substring(2,R),k=k.substring(R)||f)}return new g("file",L,k,h,h)}static from(k){let L=new g(k.scheme,k.authority,k.path,k.query,k.fragment);return u(L,!0),L}toString(k=!1){return w(this,k)}toJSON(){return this}static revive(k){if(k){if(k instanceof p)return k;{let L=new g(k);return L._formatted=k.external,L._fsPath=k._sep===m?k.fsPath:null,L}}return k}}let m=i?1:void 0;class g extends p{static{o(this,"l")}_formatted=null;_fsPath=null;get fsPath(){return this._fsPath||(this._fsPath=b(this,!1)),this._fsPath}toString(k=!1){return k?w(this,!0):(this._formatted||(this._formatted=w(this,!1)),this._formatted)}toJSON(){let k={$mid:1};return this._fsPath&&(k.fsPath=this._fsPath,k._sep=m),this._formatted&&(k.external=this._formatted),this.path&&(k.path=this.path),this.scheme&&(k.scheme=this.scheme),this.authority&&(k.authority=this.authority),this.query&&(k.query=this.query),this.fragment&&(k.fragment=this.fragment),k}}let y={58:"%3A",47:"%2F",63:"%3F",35:"%23",91:"%5B",93:"%5D",64:"%40",33:"%21",36:"%24",38:"%26",39:"%27",40:"%28",41:"%29",42:"%2A",43:"%2B",44:"%2C",59:"%3B",61:"%3D",32:"%20"};function v(D,k,L){let R,O=-1;for(let M=0;M=97&&B<=122||B>=65&&B<=90||B>=48&&B<=57||B===45||B===46||B===95||B===126||k&&B===47||L&&B===91||L&&B===93||L&&B===58)O!==-1&&(R+=encodeURIComponent(D.substring(O,M)),O=-1),R!==void 0&&(R+=D.charAt(M));else{R===void 0&&(R=D.substr(0,M));let F=y[B];F!==void 0?(O!==-1&&(R+=encodeURIComponent(D.substring(O,M)),O=-1),R+=F):O===-1&&(O=M)}}return O!==-1&&(R+=encodeURIComponent(D.substring(O))),R!==void 0?R:D}o(v,"d");function x(D){let k;for(let L=0;L1&&D.scheme==="file"?`//${D.authority}${D.path}`:D.path.charCodeAt(0)===47&&(D.path.charCodeAt(1)>=65&&D.path.charCodeAt(1)<=90||D.path.charCodeAt(1)>=97&&D.path.charCodeAt(1)<=122)&&D.path.charCodeAt(2)===58?k?D.path.substr(1):D.path[1].toLowerCase()+D.path.substr(2):D.path,i&&(L=L.replace(/\//g,"\\")),L}o(b,"m");function w(D,k){let L=k?x:v,R="",{scheme:O,authority:M,path:B,query:F,fragment:P}=D;if(O&&(R+=O,R+=":"),(M||O==="file")&&(R+=f,R+=f),M){let z=M.indexOf("@");if(z!==-1){let $=M.substr(0,z);M=M.substr(z+1),z=$.lastIndexOf(":"),z===-1?R+=L($,!1,!1):(R+=L($.substr(0,z),!1,!1),R+=":",R+=L($.substr(z+1),!1,!0)),R+="@"}M=M.toLowerCase(),z=M.lastIndexOf(":"),z===-1?R+=L(M,!1,!0):(R+=L(M.substr(0,z),!1,!0),R+=M.substr(z))}if(B){if(B.length>=3&&B.charCodeAt(0)===47&&B.charCodeAt(2)===58){let z=B.charCodeAt(1);z>=65&&z<=90&&(B=`/${String.fromCharCode(z+32)}:${B.substr(3)}`)}else if(B.length>=2&&B.charCodeAt(1)===58){let z=B.charCodeAt(0);z>=65&&z<=90&&(B=`${String.fromCharCode(z+32)}:${B.substr(2)}`)}R+=L(B,!0,!1)}return F&&(R+="?",R+=L(F,!1,!1)),P&&(R+="#",R+=k?P:v(P,!1,!1)),R}o(w,"y");function C(D){try{return decodeURIComponent(D)}catch{return D.length>3?D.substr(0,3)+C(D.substr(3)):D}}o(C,"v");let T=/(%[0-9A-Za-z][0-9A-Za-z])+/g;function E(D){return D.match(T)?D.replace(T,k=>C(k)):D}o(E,"C");var A=r(470);let S=A.posix||A,_="/";var I;(function(D){D.joinPath=function(k,...L){return k.with({path:S.join(k.path,...L)})},D.resolvePath=function(k,...L){let R=k.path,O=!1;R[0]!==_&&(R=_+R,O=!0);let M=S.resolve(R,...L);return O&&M[0]===_&&!k.authority&&(M=M.substring(1)),k.with({path:M})},D.dirname=function(k){if(k.path.length===0||k.path===_)return k;let L=S.dirname(k.path);return L.length===1&&L.charCodeAt(0)===46&&(L=""),k.with({path:L})},D.basename=function(k){return S.basename(k.path)},D.extname=function(k){return S.extname(k.path)}})(I||(I={}))})(),Hle=n})();({URI:us,Utils:u1}=Hle)});var hs,Fc=N(()=>{"use strict";OM();(function(t){t.basename=u1.basename,t.dirname=u1.dirname,t.extname=u1.extname,t.joinPath=u1.joinPath,t.resolvePath=u1.resolvePath;function e(i,a){return i?.toString()===a?.toString()}o(e,"equals"),t.equals=e;function r(i,a){let s=typeof i=="string"?i:i.path,l=typeof a=="string"?a:a.path,u=s.split("/").filter(m=>m.length>0),h=l.split("/").filter(m=>m.length>0),f=0;for(;f{"use strict";Ule();h1();qo();Ps();Fc();(function(t){t[t.Changed=0]="Changed",t[t.Parsed=1]="Parsed",t[t.IndexedContent=2]="IndexedContent",t[t.ComputedScopes=3]="ComputedScopes",t[t.Linked=4]="Linked",t[t.IndexedReferences=5]="IndexedReferences",t[t.Validated=6]="Validated"})(kn||(kn={}));Nx=class{static{o(this,"DefaultLangiumDocumentFactory")}constructor(e){this.serviceRegistry=e.ServiceRegistry,this.textDocuments=e.workspace.TextDocuments,this.fileSystemProvider=e.workspace.FileSystemProvider}async fromUri(e,r=yr.CancellationToken.None){let n=await this.fileSystemProvider.readFile(e);return this.createAsync(e,n,r)}fromTextDocument(e,r,n){return r=r??us.parse(e.uri),yr.CancellationToken.is(n)?this.createAsync(r,e,n):this.create(r,e,n)}fromString(e,r,n){return yr.CancellationToken.is(n)?this.createAsync(r,e,n):this.create(r,e,n)}fromModel(e,r){return this.create(r,{$model:e})}create(e,r,n){if(typeof r=="string"){let i=this.parse(e,r,n);return this.createLangiumDocument(i,e,void 0,r)}else if("$model"in r){let i={value:r.$model,parserErrors:[],lexerErrors:[]};return this.createLangiumDocument(i,e)}else{let i=this.parse(e,r.getText(),n);return this.createLangiumDocument(i,e,r)}}async createAsync(e,r,n){if(typeof r=="string"){let i=await this.parseAsync(e,r,n);return this.createLangiumDocument(i,e,void 0,r)}else{let i=await this.parseAsync(e,r.getText(),n);return this.createLangiumDocument(i,e,r)}}createLangiumDocument(e,r,n,i){let a;if(n)a={parseResult:e,uri:r,state:kn.Parsed,references:[],textDocument:n};else{let s=this.createTextDocumentGetter(r,i);a={parseResult:e,uri:r,state:kn.Parsed,references:[],get textDocument(){return s()}}}return e.value.$document=a,a}async update(e,r){var n,i;let a=(n=e.parseResult.value.$cstNode)===null||n===void 0?void 0:n.root.fullText,s=(i=this.textDocuments)===null||i===void 0?void 0:i.get(e.uri.toString()),l=s?s.getText():await this.fileSystemProvider.readFile(e.uri);if(s)Object.defineProperty(e,"textDocument",{value:s});else{let u=this.createTextDocumentGetter(e.uri,l);Object.defineProperty(e,"textDocument",{get:u})}return a!==l&&(e.parseResult=await this.parseAsync(e.uri,l,r),e.parseResult.value.$document=e),e.state=kn.Parsed,e}parse(e,r,n){return this.serviceRegistry.getServices(e).parser.LangiumParser.parse(r,n)}parseAsync(e,r,n){return this.serviceRegistry.getServices(e).parser.AsyncParser.parse(r,n)}createTextDocumentGetter(e,r){let n=this.serviceRegistry,i;return()=>i??(i=c1.create(e.toString(),n.getServices(e).LanguageMetaData.languageId,0,r??""))}},Mx=class{static{o(this,"DefaultLangiumDocuments")}constructor(e){this.documentMap=new Map,this.langiumDocumentFactory=e.workspace.LangiumDocumentFactory,this.serviceRegistry=e.ServiceRegistry}get all(){return en(this.documentMap.values())}addDocument(e){let r=e.uri.toString();if(this.documentMap.has(r))throw new Error(`A document with the URI '${r}' is already present.`);this.documentMap.set(r,e)}getDocument(e){let r=e.toString();return this.documentMap.get(r)}async getOrCreateDocument(e,r){let n=this.getDocument(e);return n||(n=await this.langiumDocumentFactory.fromUri(e,r),this.addDocument(n),n)}createDocument(e,r,n){if(n)return this.langiumDocumentFactory.fromString(r,e,n).then(i=>(this.addDocument(i),i));{let i=this.langiumDocumentFactory.fromString(r,e);return this.addDocument(i),i}}hasDocument(e){return this.documentMap.has(e.toString())}invalidateDocument(e){let r=e.toString(),n=this.documentMap.get(r);return n&&(this.serviceRegistry.getServices(e).references.Linker.unlink(n),n.state=kn.Changed,n.precomputedScopes=void 0,n.diagnostics=void 0),n}deleteDocument(e){let r=e.toString(),n=this.documentMap.get(r);return n&&(n.state=kn.Changed,this.documentMap.delete(r)),n}}});var PM,Ix,BM=N(()=>{"use strict";qo();Rl();is();Yo();h1();PM=Symbol("ref_resolving"),Ix=class{static{o(this,"DefaultLinker")}constructor(e){this.reflection=e.shared.AstReflection,this.langiumDocuments=()=>e.shared.workspace.LangiumDocuments,this.scopeProvider=e.references.ScopeProvider,this.astNodeLocator=e.workspace.AstNodeLocator}async link(e,r=yr.CancellationToken.None){for(let n of Wo(e.parseResult.value))await xi(r),Ag(n).forEach(i=>this.doLink(i,e))}doLink(e,r){var n;let i=e.reference;if(i._ref===void 0){i._ref=PM;try{let a=this.getCandidate(e);if(jd(a))i._ref=a;else if(i._nodeDescription=a,this.langiumDocuments().hasDocument(a.documentUri)){let s=this.loadAstNode(a);i._ref=s??this.createLinkingError(e,a)}else i._ref=void 0}catch(a){console.error(`An error occurred while resolving reference to '${i.$refText}':`,a);let s=(n=a.message)!==null&&n!==void 0?n:String(a);i._ref=Object.assign(Object.assign({},e),{message:`An error occurred while resolving reference to '${i.$refText}': ${s}`})}r.references.push(i)}}unlink(e){for(let r of e.references)delete r._ref,delete r._nodeDescription;e.references=[]}getCandidate(e){let n=this.scopeProvider.getScope(e).getElement(e.reference.$refText);return n??this.createLinkingError(e)}buildReference(e,r,n,i){let a=this,s={$refNode:n,$refText:i,get ref(){var l;if(ii(this._ref))return this._ref;if(kR(this._nodeDescription)){let u=a.loadAstNode(this._nodeDescription);this._ref=u??a.createLinkingError({reference:s,container:e,property:r},this._nodeDescription)}else if(this._ref===void 0){this._ref=PM;let u=H2(e).$document,h=a.getLinkedNode({reference:s,container:e,property:r});if(h.error&&u&&u.state{"use strict";Ol();o(Wle,"isNamed");Ox=class{static{o(this,"DefaultNameProvider")}getName(e){if(Wle(e))return e.name}getNameNode(e){return Q2(e.$cstNode,"name")}}});var Px,$M=N(()=>{"use strict";Ol();Rl();is();Nl();Ps();Fc();Px=class{static{o(this,"DefaultReferences")}constructor(e){this.nameProvider=e.references.NameProvider,this.index=e.shared.workspace.IndexManager,this.nodeLocator=e.workspace.AstNodeLocator}findDeclaration(e){if(e){let r=hN(e),n=e.astNode;if(r&&n){let i=n[r.feature];if(va(i))return i.ref;if(Array.isArray(i)){for(let a of i)if(va(a)&&a.$refNode&&a.$refNode.offset<=e.offset&&a.$refNode.end>=e.end)return a.ref}}if(n){let i=this.nameProvider.getNameNode(n);if(i&&(i===e||SR(e,i)))return n}}}findDeclarationNode(e){let r=this.findDeclaration(e);if(r?.$cstNode){let n=this.nameProvider.getNameNode(r);return n??r.$cstNode}}findReferences(e,r){let n=[];if(r.includeDeclaration){let a=this.getReferenceToSelf(e);a&&n.push(a)}let i=this.index.findAllReferences(e,this.nodeLocator.getAstNodePath(e));return r.documentUri&&(i=i.filter(a=>hs.equals(a.sourceUri,r.documentUri))),n.push(...i),en(n)}getReferenceToSelf(e){let r=this.nameProvider.getNameNode(e);if(r){let n=Pa(e),i=this.nodeLocator.getAstNodePath(e);return{sourceUri:n.uri,sourcePath:i,targetUri:n.uri,targetPath:i,segment:Qd(r),local:!0}}}}});var Bl,vp,f1=N(()=>{"use strict";Ps();Bl=class{static{o(this,"MultiMap")}constructor(e){if(this.map=new Map,e)for(let[r,n]of e)this.add(r,n)}get size(){return zm.sum(en(this.map.values()).map(e=>e.length))}clear(){this.map.clear()}delete(e,r){if(r===void 0)return this.map.delete(e);{let n=this.map.get(e);if(n){let i=n.indexOf(r);if(i>=0)return n.length===1?this.map.delete(e):n.splice(i,1),!0}return!1}}get(e){var r;return(r=this.map.get(e))!==null&&r!==void 0?r:[]}has(e,r){if(r===void 0)return this.map.has(e);{let n=this.map.get(e);return n?n.indexOf(r)>=0:!1}}add(e,r){return this.map.has(e)?this.map.get(e).push(r):this.map.set(e,[r]),this}addAll(e,r){return this.map.has(e)?this.map.get(e).push(...r):this.map.set(e,Array.from(r)),this}forEach(e){this.map.forEach((r,n)=>r.forEach(i=>e(i,n,this)))}[Symbol.iterator](){return this.entries().iterator()}entries(){return en(this.map.entries()).flatMap(([e,r])=>r.map(n=>[e,n]))}keys(){return en(this.map.keys())}values(){return en(this.map.values()).flat()}entriesGroupedByKey(){return en(this.map.entries())}},vp=class{static{o(this,"BiMap")}get size(){return this.map.size}constructor(e){if(this.map=new Map,this.inverse=new Map,e)for(let[r,n]of e)this.set(r,n)}clear(){this.map.clear(),this.inverse.clear()}set(e,r){return this.map.set(e,r),this.inverse.set(r,e),this}get(e){return this.map.get(e)}getKey(e){return this.inverse.get(e)}delete(e){let r=this.map.get(e);return r!==void 0?(this.map.delete(e),this.inverse.delete(r),!0):!1}}});var Bx,zM=N(()=>{"use strict";qo();is();f1();Yo();Bx=class{static{o(this,"DefaultScopeComputation")}constructor(e){this.nameProvider=e.references.NameProvider,this.descriptions=e.workspace.AstNodeDescriptionProvider}async computeExports(e,r=yr.CancellationToken.None){return this.computeExportsForNode(e.parseResult.value,e,void 0,r)}async computeExportsForNode(e,r,n=W2,i=yr.CancellationToken.None){let a=[];this.exportNode(e,a,r);for(let s of n(e))await xi(i),this.exportNode(s,a,r);return a}exportNode(e,r,n){let i=this.nameProvider.getName(e);i&&r.push(this.descriptions.createDescription(e,i,n))}async computeLocalScopes(e,r=yr.CancellationToken.None){let n=e.parseResult.value,i=new Bl;for(let a of Nc(n))await xi(r),this.processNode(a,e,i);return i}processNode(e,r,n){let i=e.$container;if(i){let a=this.nameProvider.getName(e);a&&n.add(i,this.descriptions.createDescription(e,a,r))}}}});var d1,Fx,xFe,GM=N(()=>{"use strict";Ps();d1=class{static{o(this,"StreamScope")}constructor(e,r,n){var i;this.elements=e,this.outerScope=r,this.caseInsensitive=(i=n?.caseInsensitive)!==null&&i!==void 0?i:!1}getAllElements(){return this.outerScope?this.elements.concat(this.outerScope.getAllElements()):this.elements}getElement(e){let r=this.caseInsensitive?this.elements.find(n=>n.name.toLowerCase()===e.toLowerCase()):this.elements.find(n=>n.name===e);if(r)return r;if(this.outerScope)return this.outerScope.getElement(e)}},Fx=class{static{o(this,"MapScope")}constructor(e,r,n){var i;this.elements=new Map,this.caseInsensitive=(i=n?.caseInsensitive)!==null&&i!==void 0?i:!1;for(let a of e){let s=this.caseInsensitive?a.name.toLowerCase():a.name;this.elements.set(s,a)}this.outerScope=r}getElement(e){let r=this.caseInsensitive?e.toLowerCase():e,n=this.elements.get(r);if(n)return n;if(this.outerScope)return this.outerScope.getElement(e)}getAllElements(){let e=en(this.elements.values());return this.outerScope&&(e=e.concat(this.outerScope.getAllElements())),e}},xFe={getElement(){},getAllElements(){return I2}}});var p1,$x,xp,_E,m1,DE=N(()=>{"use strict";p1=class{static{o(this,"DisposableCache")}constructor(){this.toDispose=[],this.isDisposed=!1}onDispose(e){this.toDispose.push(e)}dispose(){this.throwIfDisposed(),this.clear(),this.isDisposed=!0,this.toDispose.forEach(e=>e.dispose())}throwIfDisposed(){if(this.isDisposed)throw new Error("This cache has already been disposed")}},$x=class extends p1{static{o(this,"SimpleCache")}constructor(){super(...arguments),this.cache=new Map}has(e){return this.throwIfDisposed(),this.cache.has(e)}set(e,r){this.throwIfDisposed(),this.cache.set(e,r)}get(e,r){if(this.throwIfDisposed(),this.cache.has(e))return this.cache.get(e);if(r){let n=r();return this.cache.set(e,n),n}else return}delete(e){return this.throwIfDisposed(),this.cache.delete(e)}clear(){this.throwIfDisposed(),this.cache.clear()}},xp=class extends p1{static{o(this,"ContextCache")}constructor(e){super(),this.cache=new Map,this.converter=e??(r=>r)}has(e,r){return this.throwIfDisposed(),this.cacheForContext(e).has(r)}set(e,r,n){this.throwIfDisposed(),this.cacheForContext(e).set(r,n)}get(e,r,n){this.throwIfDisposed();let i=this.cacheForContext(e);if(i.has(r))return i.get(r);if(n){let a=n();return i.set(r,a),a}else return}delete(e,r){return this.throwIfDisposed(),this.cacheForContext(e).delete(r)}clear(e){if(this.throwIfDisposed(),e){let r=this.converter(e);this.cache.delete(r)}else this.cache.clear()}cacheForContext(e){let r=this.converter(e),n=this.cache.get(r);return n||(n=new Map,this.cache.set(r,n)),n}},_E=class extends xp{static{o(this,"DocumentCache")}constructor(e,r){super(n=>n.toString()),r?(this.toDispose.push(e.workspace.DocumentBuilder.onDocumentPhase(r,n=>{this.clear(n.uri.toString())})),this.toDispose.push(e.workspace.DocumentBuilder.onUpdate((n,i)=>{for(let a of i)this.clear(a)}))):this.toDispose.push(e.workspace.DocumentBuilder.onUpdate((n,i)=>{let a=n.concat(i);for(let s of a)this.clear(s)}))}},m1=class extends $x{static{o(this,"WorkspaceCache")}constructor(e,r){super(),r?(this.toDispose.push(e.workspace.DocumentBuilder.onBuildPhase(r,()=>{this.clear()})),this.toDispose.push(e.workspace.DocumentBuilder.onUpdate((n,i)=>{i.length>0&&this.clear()}))):this.toDispose.push(e.workspace.DocumentBuilder.onUpdate(()=>{this.clear()}))}}});var zx,VM=N(()=>{"use strict";GM();is();Ps();DE();zx=class{static{o(this,"DefaultScopeProvider")}constructor(e){this.reflection=e.shared.AstReflection,this.nameProvider=e.references.NameProvider,this.descriptions=e.workspace.AstNodeDescriptionProvider,this.indexManager=e.shared.workspace.IndexManager,this.globalScopeCache=new m1(e.shared)}getScope(e){let r=[],n=this.reflection.getReferenceType(e),i=Pa(e.container).precomputedScopes;if(i){let s=e.container;do{let l=i.get(s);l.length>0&&r.push(en(l).filter(u=>this.reflection.isSubtype(u.type,n))),s=s.$container}while(s)}let a=this.getGlobalScope(n,e);for(let s=r.length-1;s>=0;s--)a=this.createScope(r[s],a);return a}createScope(e,r,n){return new d1(en(e),r,n)}createScopeForNodes(e,r,n){let i=en(e).map(a=>{let s=this.nameProvider.getName(a);if(s)return this.descriptions.createDescription(a,s)}).nonNullable();return new d1(i,r,n)}getGlobalScope(e,r){return this.globalScopeCache.get(e,()=>new Fx(this.indexManager.allElements(e)))}}});function UM(t){return typeof t.$comment=="string"}function qle(t){return typeof t=="object"&&!!t&&("$ref"in t||"$error"in t)}var Gx,LE=N(()=>{"use strict";OM();Rl();is();Ol();o(UM,"isAstNodeWithComment");o(qle,"isIntermediateReference");Gx=class{static{o(this,"DefaultJsonSerializer")}constructor(e){this.ignoreProperties=new Set(["$container","$containerProperty","$containerIndex","$document","$cstNode"]),this.langiumDocuments=e.shared.workspace.LangiumDocuments,this.astNodeLocator=e.workspace.AstNodeLocator,this.nameProvider=e.references.NameProvider,this.commentProvider=e.documentation.CommentProvider}serialize(e,r){let n=r??{},i=r?.replacer,a=o((l,u)=>this.replacer(l,u,n),"defaultReplacer"),s=i?(l,u)=>i(l,u,a):a;try{return this.currentDocument=Pa(e),JSON.stringify(e,s,r?.space)}finally{this.currentDocument=void 0}}deserialize(e,r){let n=r??{},i=JSON.parse(e);return this.linkNode(i,i,n),i}replacer(e,r,{refText:n,sourceText:i,textRegions:a,comments:s,uriConverter:l}){var u,h,f,d;if(!this.ignoreProperties.has(e))if(va(r)){let p=r.ref,m=n?r.$refText:void 0;if(p){let g=Pa(p),y="";this.currentDocument&&this.currentDocument!==g&&(l?y=l(g.uri,r):y=g.uri.toString());let v=this.astNodeLocator.getAstNodePath(p);return{$ref:`${y}#${v}`,$refText:m}}else return{$error:(h=(u=r.error)===null||u===void 0?void 0:u.message)!==null&&h!==void 0?h:"Could not resolve reference",$refText:m}}else if(ii(r)){let p;if(a&&(p=this.addAstNodeRegionWithAssignmentsTo(Object.assign({},r)),(!e||r.$document)&&p?.$textRegion&&(p.$textRegion.documentURI=(f=this.currentDocument)===null||f===void 0?void 0:f.uri.toString())),i&&!e&&(p??(p=Object.assign({},r)),p.$sourceText=(d=r.$cstNode)===null||d===void 0?void 0:d.text),s){p??(p=Object.assign({},r));let m=this.commentProvider.getComment(r);m&&(p.$comment=m.replace(/\r/g,""))}return p??r}else return r}addAstNodeRegionWithAssignmentsTo(e){let r=o(n=>({offset:n.offset,end:n.end,length:n.length,range:n.range}),"createDocumentSegment");if(e.$cstNode){let n=e.$textRegion=r(e.$cstNode),i=n.assignments={};return Object.keys(e).filter(a=>!a.startsWith("$")).forEach(a=>{let s=oN(e.$cstNode,a).map(r);s.length!==0&&(i[a]=s)}),e}}linkNode(e,r,n,i,a,s){for(let[u,h]of Object.entries(e))if(Array.isArray(h))for(let f=0;f{"use strict";Fc();Vx=class{static{o(this,"DefaultServiceRegistry")}get map(){return this.fileExtensionMap}constructor(e){this.languageIdMap=new Map,this.fileExtensionMap=new Map,this.textDocuments=e?.workspace.TextDocuments}register(e){let r=e.LanguageMetaData;for(let n of r.fileExtensions)this.fileExtensionMap.has(n)&&console.warn(`The file extension ${n} is used by multiple languages. It is now assigned to '${r.languageId}'.`),this.fileExtensionMap.set(n,e);this.languageIdMap.set(r.languageId,e),this.languageIdMap.size===1?this.singleton=e:this.singleton=void 0}getServices(e){var r,n;if(this.singleton!==void 0)return this.singleton;if(this.languageIdMap.size===0)throw new Error("The service registry is empty. Use `register` to register the services of a language.");let i=(n=(r=this.textDocuments)===null||r===void 0?void 0:r.get(e))===null||n===void 0?void 0:n.languageId;if(i!==void 0){let l=this.languageIdMap.get(i);if(l)return l}let a=hs.extname(e),s=this.fileExtensionMap.get(a);if(!s)throw i?new Error(`The service registry contains no services for the extension '${a}' for language '${i}'.`):new Error(`The service registry contains no services for the extension '${a}'.`);return s}hasServices(e){try{return this.getServices(e),!0}catch{return!1}}get all(){return Array.from(this.languageIdMap.values())}}});function bp(t){return{code:t}}var g1,Ux,Hx=N(()=>{"use strict";Xo();f1();Yo();Ps();o(bp,"diagnosticData");(function(t){t.all=["fast","slow","built-in"]})(g1||(g1={}));Ux=class{static{o(this,"ValidationRegistry")}constructor(e){this.entries=new Bl,this.entriesBefore=[],this.entriesAfter=[],this.reflection=e.shared.AstReflection}register(e,r=this,n="fast"){if(n==="built-in")throw new Error("The 'built-in' category is reserved for lexer, parser, and linker errors.");for(let[i,a]of Object.entries(e)){let s=a;if(Array.isArray(s))for(let l of s){let u={check:this.wrapValidationException(l,r),category:n};this.addEntry(i,u)}else if(typeof s=="function"){let l={check:this.wrapValidationException(s,r),category:n};this.addEntry(i,l)}else Lc(s)}}wrapValidationException(e,r){return async(n,i,a)=>{await this.handleException(()=>e.call(r,n,i,a),"An error occurred during validation",i,n)}}async handleException(e,r,n,i){try{await e()}catch(a){if(Bc(a))throw a;console.error(`${r}:`,a),a instanceof Error&&a.stack&&console.error(a.stack);let s=a instanceof Error?a.message:String(a);n("error",`${r}: ${s}`,{node:i})}}addEntry(e,r){if(e==="AstNode"){this.entries.add("AstNode",r);return}for(let n of this.reflection.getAllSubTypes(e))this.entries.add(n,r)}getChecks(e,r){let n=en(this.entries.get(e)).concat(this.entries.get("AstNode"));return r&&(n=n.filter(i=>r.includes(i.category))),n.map(i=>i.check)}registerBeforeDocument(e,r=this){this.entriesBefore.push(this.wrapPreparationException(e,"An error occurred during set-up of the validation",r))}registerAfterDocument(e,r=this){this.entriesAfter.push(this.wrapPreparationException(e,"An error occurred during tear-down of the validation",r))}wrapPreparationException(e,r,n){return async(i,a,s,l)=>{await this.handleException(()=>e.call(n,i,a,s,l),r,a,i)}}get checksBefore(){return this.entriesBefore}get checksAfter(){return this.entriesAfter}}});function Yle(t){if(t.range)return t.range;let e;return typeof t.property=="string"?e=Q2(t.node.$cstNode,t.property,t.index):typeof t.keyword=="string"&&(e=cN(t.node.$cstNode,t.keyword,t.index)),e??(e=t.node.$cstNode),e?e.range:{start:{line:0,character:0},end:{line:0,character:0}}}function RE(t){switch(t){case"error":return 1;case"warning":return 2;case"info":return 3;case"hint":return 4;default:throw new Error("Invalid diagnostic severity: "+t)}}function Xle(t){switch(t){case"error":return bp(jo.LexingError);case"warning":return bp(jo.LexingWarning);case"info":return bp(jo.LexingInfo);case"hint":return bp(jo.LexingHint);default:throw new Error("Invalid diagnostic severity: "+t)}}var Wx,jo,WM=N(()=>{"use strict";qo();Ol();is();Nl();Yo();Hx();Wx=class{static{o(this,"DefaultDocumentValidator")}constructor(e){this.validationRegistry=e.validation.ValidationRegistry,this.metadata=e.LanguageMetaData}async validateDocument(e,r={},n=yr.CancellationToken.None){let i=e.parseResult,a=[];if(await xi(n),(!r.categories||r.categories.includes("built-in"))&&(this.processLexingErrors(i,a,r),r.stopAfterLexingErrors&&a.some(s=>{var l;return((l=s.data)===null||l===void 0?void 0:l.code)===jo.LexingError})||(this.processParsingErrors(i,a,r),r.stopAfterParsingErrors&&a.some(s=>{var l;return((l=s.data)===null||l===void 0?void 0:l.code)===jo.ParsingError}))||(this.processLinkingErrors(e,a,r),r.stopAfterLinkingErrors&&a.some(s=>{var l;return((l=s.data)===null||l===void 0?void 0:l.code)===jo.LinkingError}))))return a;try{a.push(...await this.validateAst(i.value,r,n))}catch(s){if(Bc(s))throw s;console.error("An error occurred during validation:",s)}return await xi(n),a}processLexingErrors(e,r,n){var i,a,s;let l=[...e.lexerErrors,...(a=(i=e.lexerReport)===null||i===void 0?void 0:i.diagnostics)!==null&&a!==void 0?a:[]];for(let u of l){let h=(s=u.severity)!==null&&s!==void 0?s:"error",f={severity:RE(h),range:{start:{line:u.line-1,character:u.column-1},end:{line:u.line-1,character:u.column+u.length-1}},message:u.message,data:Xle(h),source:this.getSource()};r.push(f)}}processParsingErrors(e,r,n){for(let i of e.parserErrors){let a;if(isNaN(i.token.startOffset)){if("previousToken"in i){let s=i.previousToken;if(isNaN(s.startOffset)){let l={line:0,character:0};a={start:l,end:l}}else{let l={line:s.endLine-1,character:s.endColumn};a={start:l,end:l}}}}else a=Gm(i.token);if(a){let s={severity:RE("error"),range:a,message:i.message,data:bp(jo.ParsingError),source:this.getSource()};r.push(s)}}}processLinkingErrors(e,r,n){for(let i of e.references){let a=i.error;if(a){let s={node:a.container,property:a.property,index:a.index,data:{code:jo.LinkingError,containerType:a.container.$type,property:a.property,refText:a.reference.$refText}};r.push(this.toDiagnostic("error",a.message,s))}}}async validateAst(e,r,n=yr.CancellationToken.None){let i=[],a=o((s,l,u)=>{i.push(this.toDiagnostic(s,l,u))},"acceptor");return await this.validateAstBefore(e,r,a,n),await this.validateAstNodes(e,r,a,n),await this.validateAstAfter(e,r,a,n),i}async validateAstBefore(e,r,n,i=yr.CancellationToken.None){var a;let s=this.validationRegistry.checksBefore;for(let l of s)await xi(i),await l(e,n,(a=r.categories)!==null&&a!==void 0?a:[],i)}async validateAstNodes(e,r,n,i=yr.CancellationToken.None){await Promise.all(Wo(e).map(async a=>{await xi(i);let s=this.validationRegistry.getChecks(a.$type,r.categories);for(let l of s)await l(a,n,i)}))}async validateAstAfter(e,r,n,i=yr.CancellationToken.None){var a;let s=this.validationRegistry.checksAfter;for(let l of s)await xi(i),await l(e,n,(a=r.categories)!==null&&a!==void 0?a:[],i)}toDiagnostic(e,r,n){return{message:r,range:Yle(n),severity:RE(e),code:n.code,codeDescription:n.codeDescription,tags:n.tags,relatedInformation:n.relatedInformation,data:n.data,source:this.getSource()}}getSource(){return this.metadata.languageId}};o(Yle,"getDiagnosticRange");o(RE,"toDiagnosticSeverity");o(Xle,"toDiagnosticData");(function(t){t.LexingError="lexing-error",t.LexingWarning="lexing-warning",t.LexingInfo="lexing-info",t.LexingHint="lexing-hint",t.ParsingError="parsing-error",t.LinkingError="linking-error"})(jo||(jo={}))});var qx,Yx,qM=N(()=>{"use strict";qo();Rl();is();Nl();Yo();Fc();qx=class{static{o(this,"DefaultAstNodeDescriptionProvider")}constructor(e){this.astNodeLocator=e.workspace.AstNodeLocator,this.nameProvider=e.references.NameProvider}createDescription(e,r,n){let i=n??Pa(e);r??(r=this.nameProvider.getName(e));let a=this.astNodeLocator.getAstNodePath(e);if(!r)throw new Error(`Node at path ${a} has no name.`);let s,l=o(()=>{var u;return s??(s=Qd((u=this.nameProvider.getNameNode(e))!==null&&u!==void 0?u:e.$cstNode))},"nameSegmentGetter");return{node:e,name:r,get nameSegment(){return l()},selectionSegment:Qd(e.$cstNode),type:e.$type,documentUri:i.uri,path:a}}},Yx=class{static{o(this,"DefaultReferenceDescriptionProvider")}constructor(e){this.nodeLocator=e.workspace.AstNodeLocator}async createDescriptions(e,r=yr.CancellationToken.None){let n=[],i=e.parseResult.value;for(let a of Wo(i))await xi(r),Ag(a).filter(s=>!jd(s)).forEach(s=>{let l=this.createDescription(s);l&&n.push(l)});return n}createDescription(e){let r=e.reference.$nodeDescription,n=e.reference.$refNode;if(!r||!n)return;let i=Pa(e.container).uri;return{sourceUri:i,sourcePath:this.nodeLocator.getAstNodePath(e.container),targetUri:r.documentUri,targetPath:r.path,segment:Qd(n),local:hs.equals(r.documentUri,i)}}}});var Xx,YM=N(()=>{"use strict";Xx=class{static{o(this,"DefaultAstNodeLocator")}constructor(){this.segmentSeparator="/",this.indexSeparator="@"}getAstNodePath(e){if(e.$container){let r=this.getAstNodePath(e.$container),n=this.getPathSegment(e);return r+this.segmentSeparator+n}return""}getPathSegment({$containerProperty:e,$containerIndex:r}){if(!e)throw new Error("Missing '$containerProperty' in AST node.");return r!==void 0?e+this.indexSeparator+r:e}getAstNode(e,r){return r.split(this.segmentSeparator).reduce((i,a)=>{if(!i||a.length===0)return i;let s=a.indexOf(this.indexSeparator);if(s>0){let l=a.substring(0,s),u=parseInt(a.substring(s+1)),h=i[l];return h?.[u]}return i[a]},e)}}});var Kn={};var NE=N(()=>{"use strict";Sr(Kn,Sa(LM(),1))});var jx,XM=N(()=>{"use strict";NE();Yo();jx=class{static{o(this,"DefaultConfigurationProvider")}constructor(e){this._ready=new cs,this.settings={},this.workspaceConfig=!1,this.onConfigurationSectionUpdateEmitter=new Kn.Emitter,this.serviceRegistry=e.ServiceRegistry}get ready(){return this._ready.promise}initialize(e){var r,n;this.workspaceConfig=(n=(r=e.capabilities.workspace)===null||r===void 0?void 0:r.configuration)!==null&&n!==void 0?n:!1}async initialized(e){if(this.workspaceConfig){if(e.register){let r=this.serviceRegistry.all;e.register({section:r.map(n=>this.toSectionName(n.LanguageMetaData.languageId))})}if(e.fetchConfiguration){let r=this.serviceRegistry.all.map(i=>({section:this.toSectionName(i.LanguageMetaData.languageId)})),n=await e.fetchConfiguration(r);r.forEach((i,a)=>{this.updateSectionConfiguration(i.section,n[a])})}}this._ready.resolve()}updateConfiguration(e){e.settings&&Object.keys(e.settings).forEach(r=>{let n=e.settings[r];this.updateSectionConfiguration(r,n),this.onConfigurationSectionUpdateEmitter.fire({section:r,configuration:n})})}updateSectionConfiguration(e,r){this.settings[e]=r}async getConfiguration(e,r){await this.ready;let n=this.toSectionName(e);if(this.settings[n])return this.settings[n][r]}toSectionName(e){return`${e}`}get onConfigurationSectionUpdate(){return this.onConfigurationSectionUpdateEmitter.event}}});var ff,jM=N(()=>{"use strict";(function(t){function e(r){return{dispose:o(async()=>await r(),"dispose")}}o(e,"create"),t.create=e})(ff||(ff={}))});var Kx,KM=N(()=>{"use strict";qo();jM();f1();Yo();Ps();Hx();h1();Kx=class{static{o(this,"DefaultDocumentBuilder")}constructor(e){this.updateBuildOptions={validation:{categories:["built-in","fast"]}},this.updateListeners=[],this.buildPhaseListeners=new Bl,this.documentPhaseListeners=new Bl,this.buildState=new Map,this.documentBuildWaiters=new Map,this.currentState=kn.Changed,this.langiumDocuments=e.workspace.LangiumDocuments,this.langiumDocumentFactory=e.workspace.LangiumDocumentFactory,this.textDocuments=e.workspace.TextDocuments,this.indexManager=e.workspace.IndexManager,this.serviceRegistry=e.ServiceRegistry}async build(e,r={},n=yr.CancellationToken.None){var i,a;for(let s of e){let l=s.uri.toString();if(s.state===kn.Validated){if(typeof r.validation=="boolean"&&r.validation)s.state=kn.IndexedReferences,s.diagnostics=void 0,this.buildState.delete(l);else if(typeof r.validation=="object"){let u=this.buildState.get(l),h=(i=u?.result)===null||i===void 0?void 0:i.validationChecks;if(h){let d=((a=r.validation.categories)!==null&&a!==void 0?a:g1.all).filter(p=>!h.includes(p));d.length>0&&(this.buildState.set(l,{completed:!1,options:{validation:Object.assign(Object.assign({},r.validation),{categories:d})},result:u.result}),s.state=kn.IndexedReferences)}}}else this.buildState.delete(l)}this.currentState=kn.Changed,await this.emitUpdate(e.map(s=>s.uri),[]),await this.buildDocuments(e,r,n)}async update(e,r,n=yr.CancellationToken.None){this.currentState=kn.Changed;for(let s of r)this.langiumDocuments.deleteDocument(s),this.buildState.delete(s.toString()),this.indexManager.remove(s);for(let s of e){if(!this.langiumDocuments.invalidateDocument(s)){let u=this.langiumDocumentFactory.fromModel({$type:"INVALID"},s);u.state=kn.Changed,this.langiumDocuments.addDocument(u)}this.buildState.delete(s.toString())}let i=en(e).concat(r).map(s=>s.toString()).toSet();this.langiumDocuments.all.filter(s=>!i.has(s.uri.toString())&&this.shouldRelink(s,i)).forEach(s=>{this.serviceRegistry.getServices(s.uri).references.Linker.unlink(s),s.state=Math.min(s.state,kn.ComputedScopes),s.diagnostics=void 0}),await this.emitUpdate(e,r),await xi(n);let a=this.sortDocuments(this.langiumDocuments.all.filter(s=>{var l;return s.staten(e,r)))}sortDocuments(e){let r=0,n=e.length-1;for(;r=0&&!this.hasTextDocument(e[n]);)n--;rn.error!==void 0)?!0:this.indexManager.isAffected(e,r)}onUpdate(e){return this.updateListeners.push(e),ff.create(()=>{let r=this.updateListeners.indexOf(e);r>=0&&this.updateListeners.splice(r,1)})}async buildDocuments(e,r,n){this.prepareBuild(e,r),await this.runCancelable(e,kn.Parsed,n,a=>this.langiumDocumentFactory.update(a,n)),await this.runCancelable(e,kn.IndexedContent,n,a=>this.indexManager.updateContent(a,n)),await this.runCancelable(e,kn.ComputedScopes,n,async a=>{let s=this.serviceRegistry.getServices(a.uri).references.ScopeComputation;a.precomputedScopes=await s.computeLocalScopes(a,n)}),await this.runCancelable(e,kn.Linked,n,a=>this.serviceRegistry.getServices(a.uri).references.Linker.link(a,n)),await this.runCancelable(e,kn.IndexedReferences,n,a=>this.indexManager.updateReferences(a,n));let i=e.filter(a=>this.shouldValidate(a));await this.runCancelable(i,kn.Validated,n,a=>this.validate(a,n));for(let a of e){let s=this.buildState.get(a.uri.toString());s&&(s.completed=!0)}}prepareBuild(e,r){for(let n of e){let i=n.uri.toString(),a=this.buildState.get(i);(!a||a.completed)&&this.buildState.set(i,{completed:!1,options:r,result:a?.result})}}async runCancelable(e,r,n,i){let a=e.filter(l=>l.statel.state===r);await this.notifyBuildPhase(s,r,n),this.currentState=r}onBuildPhase(e,r){return this.buildPhaseListeners.add(e,r),ff.create(()=>{this.buildPhaseListeners.delete(e,r)})}onDocumentPhase(e,r){return this.documentPhaseListeners.add(e,r),ff.create(()=>{this.documentPhaseListeners.delete(e,r)})}waitUntil(e,r,n){let i;if(r&&"path"in r?i=r:n=r,n??(n=yr.CancellationToken.None),i){let a=this.langiumDocuments.getDocument(i);if(a&&a.state>e)return Promise.resolve(i)}return this.currentState>=e?Promise.resolve(void 0):n.isCancellationRequested?Promise.reject(Pc):new Promise((a,s)=>{let l=this.onBuildPhase(e,()=>{if(l.dispose(),u.dispose(),i){let h=this.langiumDocuments.getDocument(i);a(h?.uri)}else a(void 0)}),u=n.onCancellationRequested(()=>{l.dispose(),u.dispose(),s(Pc)})})}async notifyDocumentPhase(e,r,n){let a=this.documentPhaseListeners.get(r).slice();for(let s of a)try{await s(e,n)}catch(l){if(!Bc(l))throw l}}async notifyBuildPhase(e,r,n){if(e.length===0)return;let a=this.buildPhaseListeners.get(r).slice();for(let s of a)await xi(n),await s(e,n)}shouldValidate(e){return!!this.getBuildOptions(e).validation}async validate(e,r){var n,i;let a=this.serviceRegistry.getServices(e.uri).validation.DocumentValidator,s=this.getBuildOptions(e).validation,l=typeof s=="object"?s:void 0,u=await a.validateDocument(e,l,r);e.diagnostics?e.diagnostics.push(...u):e.diagnostics=u;let h=this.buildState.get(e.uri.toString());if(h){(n=h.result)!==null&&n!==void 0||(h.result={});let f=(i=l?.categories)!==null&&i!==void 0?i:g1.all;h.result.validationChecks?h.result.validationChecks.push(...f):h.result.validationChecks=[...f]}}getBuildOptions(e){var r,n;return(n=(r=this.buildState.get(e.uri.toString()))===null||r===void 0?void 0:r.options)!==null&&n!==void 0?n:{}}}});var Qx,QM=N(()=>{"use strict";is();DE();qo();Ps();Fc();Qx=class{static{o(this,"DefaultIndexManager")}constructor(e){this.symbolIndex=new Map,this.symbolByTypeIndex=new xp,this.referenceIndex=new Map,this.documents=e.workspace.LangiumDocuments,this.serviceRegistry=e.ServiceRegistry,this.astReflection=e.AstReflection}findAllReferences(e,r){let n=Pa(e).uri,i=[];return this.referenceIndex.forEach(a=>{a.forEach(s=>{hs.equals(s.targetUri,n)&&s.targetPath===r&&i.push(s)})}),en(i)}allElements(e,r){let n=en(this.symbolIndex.keys());return r&&(n=n.filter(i=>!r||r.has(i))),n.map(i=>this.getFileDescriptions(i,e)).flat()}getFileDescriptions(e,r){var n;return r?this.symbolByTypeIndex.get(e,r,()=>{var a;return((a=this.symbolIndex.get(e))!==null&&a!==void 0?a:[]).filter(l=>this.astReflection.isSubtype(l.type,r))}):(n=this.symbolIndex.get(e))!==null&&n!==void 0?n:[]}remove(e){let r=e.toString();this.symbolIndex.delete(r),this.symbolByTypeIndex.clear(r),this.referenceIndex.delete(r)}async updateContent(e,r=yr.CancellationToken.None){let i=await this.serviceRegistry.getServices(e.uri).references.ScopeComputation.computeExports(e,r),a=e.uri.toString();this.symbolIndex.set(a,i),this.symbolByTypeIndex.clear(a)}async updateReferences(e,r=yr.CancellationToken.None){let i=await this.serviceRegistry.getServices(e.uri).workspace.ReferenceDescriptionProvider.createDescriptions(e,r);this.referenceIndex.set(e.uri.toString(),i)}isAffected(e,r){let n=this.referenceIndex.get(e.uri.toString());return n?n.some(i=>!i.local&&r.has(i.targetUri.toString())):!1}}});var Zx,ZM=N(()=>{"use strict";qo();Yo();Fc();Zx=class{static{o(this,"DefaultWorkspaceManager")}constructor(e){this.initialBuildOptions={},this._ready=new cs,this.serviceRegistry=e.ServiceRegistry,this.langiumDocuments=e.workspace.LangiumDocuments,this.documentBuilder=e.workspace.DocumentBuilder,this.fileSystemProvider=e.workspace.FileSystemProvider,this.mutex=e.workspace.WorkspaceLock}get ready(){return this._ready.promise}get workspaceFolders(){return this.folders}initialize(e){var r;this.folders=(r=e.workspaceFolders)!==null&&r!==void 0?r:void 0}initialized(e){return this.mutex.write(r=>{var n;return this.initializeWorkspace((n=this.folders)!==null&&n!==void 0?n:[],r)})}async initializeWorkspace(e,r=yr.CancellationToken.None){let n=await this.performStartup(e);await xi(r),await this.documentBuilder.build(n,this.initialBuildOptions,r)}async performStartup(e){let r=this.serviceRegistry.all.flatMap(a=>a.LanguageMetaData.fileExtensions),n=[],i=o(a=>{n.push(a),this.langiumDocuments.hasDocument(a.uri)||this.langiumDocuments.addDocument(a)},"collector");return await this.loadAdditionalDocuments(e,i),await Promise.all(e.map(a=>[a,this.getRootFolder(a)]).map(async a=>this.traverseFolder(...a,r,i))),this._ready.resolve(),n}loadAdditionalDocuments(e,r){return Promise.resolve()}getRootFolder(e){return us.parse(e.uri)}async traverseFolder(e,r,n,i){let a=await this.fileSystemProvider.readDirectory(r);await Promise.all(a.map(async s=>{if(this.includeEntry(e,s,n)){if(s.isDirectory)await this.traverseFolder(e,s.uri,n,i);else if(s.isFile){let l=await this.langiumDocuments.getOrCreateDocument(s.uri);i(l)}}}))}includeEntry(e,r,n){let i=hs.basename(r.uri);if(i.startsWith("."))return!1;if(r.isDirectory)return i!=="node_modules"&&i!=="out";if(r.isFile){let a=hs.extname(r.uri);return n.includes(a)}return!1}}});function IE(t){return Array.isArray(t)&&(t.length===0||"name"in t[0])}function eI(t){return t&&"modes"in t&&"defaultMode"in t}function JM(t){return!IE(t)&&!eI(t)}var Jx,ME,wp,OE=N(()=>{"use strict";cf();Jx=class{static{o(this,"DefaultLexerErrorMessageProvider")}buildUnexpectedCharactersMessage(e,r,n,i,a){return Gg.buildUnexpectedCharactersMessage(e,r,n,i,a)}buildUnableToPopLexerModeMessage(e){return Gg.buildUnableToPopLexerModeMessage(e)}},ME={mode:"full"},wp=class{static{o(this,"DefaultLexer")}constructor(e){this.errorMessageProvider=e.parser.LexerErrorMessageProvider,this.tokenBuilder=e.parser.TokenBuilder;let r=this.tokenBuilder.buildTokens(e.Grammar,{caseInsensitive:e.LanguageMetaData.caseInsensitive});this.tokenTypes=this.toTokenTypeDictionary(r);let n=JM(r)?Object.values(r):r,i=e.LanguageMetaData.mode==="production";this.chevrotainLexer=new Xn(n,{positionTracking:"full",skipValidations:i,errorMessageProvider:this.errorMessageProvider})}get definition(){return this.tokenTypes}tokenize(e,r=ME){var n,i,a;let s=this.chevrotainLexer.tokenize(e);return{tokens:s.tokens,errors:s.errors,hidden:(n=s.groups.hidden)!==null&&n!==void 0?n:[],report:(a=(i=this.tokenBuilder).flushLexingReport)===null||a===void 0?void 0:a.call(i,e)}}toTokenTypeDictionary(e){if(JM(e))return e;let r=eI(e)?Object.values(e.modes).flat():e,n={};return r.forEach(i=>n[i.name]=i),n}};o(IE,"isTokenTypeArray");o(eI,"isIMultiModeLexerDefinition");o(JM,"isTokenTypeDictionary")});function nI(t,e,r){let n,i;typeof t=="string"?(i=e,n=r):(i=t.range.start,n=e),i||(i=jr.create(0,0));let a=Qle(t),s=aI(n),l=wFe({lines:a,position:i,options:s});return CFe({index:0,tokens:l,position:i})}function iI(t,e){let r=aI(e),n=Qle(t);if(n.length===0)return!1;let i=n[0],a=n[n.length-1],s=r.start,l=r.end;return!!s?.exec(i)&&!!l?.exec(a)}function Qle(t){let e="";return typeof t=="string"?e=t:e=t.text,e.split(JR)}function wFe(t){var e,r,n;let i=[],a=t.position.line,s=t.position.character;for(let l=0;l=f.length){if(i.length>0){let m=jr.create(a,s);i.push({type:"break",content:"",range:Pr.create(m,m)})}}else{jle.lastIndex=d;let m=jle.exec(f);if(m){let g=m[0],y=m[1],v=jr.create(a,s+d),x=jr.create(a,s+d+g.length);i.push({type:"tag",content:y,range:Pr.create(v,x)}),d+=g.length,d=rI(f,d)}if(d0&&i[i.length-1].type==="break"?i.slice(0,-1):i}function TFe(t,e,r,n){let i=[];if(t.length===0){let a=jr.create(r,n),s=jr.create(r,n+e.length);i.push({type:"text",content:e,range:Pr.create(a,s)})}else{let a=0;for(let l of t){let u=l.index,h=e.substring(a,u);h.length>0&&i.push({type:"text",content:e.substring(a,u),range:Pr.create(jr.create(r,a+n),jr.create(r,u+n))});let f=h.length+1,d=l[1];if(i.push({type:"inline-tag",content:d,range:Pr.create(jr.create(r,a+f+n),jr.create(r,a+f+d.length+n))}),f+=d.length,l.length===4){f+=l[2].length;let p=l[3];i.push({type:"text",content:p,range:Pr.create(jr.create(r,a+f+n),jr.create(r,a+f+p.length+n))})}else i.push({type:"text",content:"",range:Pr.create(jr.create(r,a+f+n),jr.create(r,a+f+n))});a=u+l[0].length}let s=e.substring(a);s.length>0&&i.push({type:"text",content:s,range:Pr.create(jr.create(r,a+n),jr.create(r,a+n+s.length))})}return i}function rI(t,e){let r=t.substring(e).match(kFe);return r?e+r.index:t.length}function SFe(t){let e=t.match(EFe);if(e&&typeof e.index=="number")return e.index}function CFe(t){var e,r,n,i;let a=jr.create(t.position.line,t.position.character);if(t.tokens.length===0)return new PE([],Pr.create(a,a));let s=[];for(;t.index0){let u=rI(e,a);s=e.substring(u),e=e.substring(0,a)}return(t==="linkcode"||t==="link"&&r.link==="code")&&(s=`\`${s}\``),(i=(n=r.renderLink)===null||n===void 0?void 0:n.call(r,e,s))!==null&&i!==void 0?i:RFe(e,s)}}function RFe(t,e){try{return us.parse(t,!0),`[${e}](${t})`}catch{return t}}function Kle(t){return t.endsWith(` +`)?` +`:` + +`}var jle,bFe,kFe,EFe,PE,eb,tb,BE,sI=N(()=>{"use strict";mM();Lg();Fc();o(nI,"parseJSDoc");o(iI,"isJSDoc");o(Qle,"getLines");jle=/\s*(@([\p{L}][\p{L}\p{N}]*)?)/uy,bFe=/\{(@[\p{L}][\p{L}\p{N}]*)(\s*)([^\r\n}]+)?\}/gu;o(wFe,"tokenize");o(TFe,"buildInlineTokens");kFe=/\S/,EFe=/\s*$/;o(rI,"skipWhitespace");o(SFe,"lastCharacter");o(CFe,"parseJSDocComment");o(AFe,"parseJSDocElement");o(_Fe,"appendEmptyLine");o(Zle,"parseJSDocText");o(DFe,"parseJSDocInline");o(Jle,"parseJSDocTag");o(ece,"parseJSDocLine");o(aI,"normalizeOptions");o(tI,"normalizeOption");PE=class{static{o(this,"JSDocCommentImpl")}constructor(e,r){this.elements=e,this.range=r}getTag(e){return this.getAllTags().find(r=>r.name===e)}getTags(e){return this.getAllTags().filter(r=>r.name===e)}getAllTags(){return this.elements.filter(e=>"name"in e)}toString(){let e="";for(let r of this.elements)if(e.length===0)e=r.toString();else{let n=r.toString();e+=Kle(e)+n}return e.trim()}toMarkdown(e){let r="";for(let n of this.elements)if(r.length===0)r=n.toMarkdown(e);else{let i=n.toMarkdown(e);r+=Kle(r)+i}return r.trim()}},eb=class{static{o(this,"JSDocTagImpl")}constructor(e,r,n,i){this.name=e,this.content=r,this.inline=n,this.range=i}toString(){let e=`@${this.name}`,r=this.content.toString();return this.content.inlines.length===1?e=`${e} ${r}`:this.content.inlines.length>1&&(e=`${e} +${r}`),this.inline?`{${e}}`:e}toMarkdown(e){var r,n;return(n=(r=e?.renderTag)===null||r===void 0?void 0:r.call(e,this))!==null&&n!==void 0?n:this.toMarkdownDefault(e)}toMarkdownDefault(e){let r=this.content.toMarkdown(e);if(this.inline){let a=LFe(this.name,r,e??{});if(typeof a=="string")return a}let n="";e?.tag==="italic"||e?.tag===void 0?n="*":e?.tag==="bold"?n="**":e?.tag==="bold-italic"&&(n="***");let i=`${n}@${this.name}${n}`;return this.content.inlines.length===1?i=`${i} \u2014 ${r}`:this.content.inlines.length>1&&(i=`${i} +${r}`),this.inline?`{${i}}`:i}};o(LFe,"renderInlineTag");o(RFe,"renderLinkDefault");tb=class{static{o(this,"JSDocTextImpl")}constructor(e,r){this.inlines=e,this.range=r}toString(){let e="";for(let r=0;rn.range.start.line&&(e+=` +`)}return e}toMarkdown(e){let r="";for(let n=0;ni.range.start.line&&(r+=` +`)}return r}},BE=class{static{o(this,"JSDocLineImpl")}constructor(e,r){this.text=e,this.range=r}toString(){return this.text}toMarkdown(){return this.text}};o(Kle,"fillNewlines")});var rb,oI=N(()=>{"use strict";is();sI();rb=class{static{o(this,"JSDocDocumentationProvider")}constructor(e){this.indexManager=e.shared.workspace.IndexManager,this.commentProvider=e.documentation.CommentProvider}getDocumentation(e){let r=this.commentProvider.getComment(e);if(r&&iI(r))return nI(r).toMarkdown({renderLink:o((i,a)=>this.documentationLinkRenderer(e,i,a),"renderLink"),renderTag:o(i=>this.documentationTagRenderer(e,i),"renderTag")})}documentationLinkRenderer(e,r,n){var i;let a=(i=this.findNameInPrecomputedScopes(e,r))!==null&&i!==void 0?i:this.findNameInGlobalScope(e,r);if(a&&a.nameSegment){let s=a.nameSegment.range.start.line+1,l=a.nameSegment.range.start.character+1,u=a.documentUri.with({fragment:`L${s},${l}`});return`[${n}](${u.toString()})`}else return}documentationTagRenderer(e,r){}findNameInPrecomputedScopes(e,r){let i=Pa(e).precomputedScopes;if(!i)return;let a=e;do{let l=i.get(a).find(u=>u.name===r);if(l)return l;a=a.$container}while(a)}findNameInGlobalScope(e,r){return this.indexManager.allElements().find(i=>i.name===r)}}});var nb,lI=N(()=>{"use strict";LE();Nl();nb=class{static{o(this,"DefaultCommentProvider")}constructor(e){this.grammarConfig=()=>e.parser.GrammarConfig}getComment(e){var r;return UM(e)?e.$comment:(r=AR(e.$cstNode,this.grammarConfig().multilineCommentRules))===null||r===void 0?void 0:r.text}}});var ib,cI,uI,hI=N(()=>{"use strict";Yo();NE();ib=class{static{o(this,"DefaultAsyncParser")}constructor(e){this.syncParser=e.parser.LangiumParser}parse(e,r){return Promise.resolve(this.syncParser.parse(e))}},cI=class{static{o(this,"AbstractThreadedAsyncParser")}constructor(e){this.threadCount=8,this.terminationDelay=200,this.workerPool=[],this.queue=[],this.hydrator=e.serializer.Hydrator}initializeWorkers(){for(;this.workerPool.length{if(this.queue.length>0){let r=this.queue.shift();r&&(e.lock(),r.resolve(e))}}),this.workerPool.push(e)}}async parse(e,r){let n=await this.acquireParserWorker(r),i=new cs,a,s=r.onCancellationRequested(()=>{a=setTimeout(()=>{this.terminateWorker(n)},this.terminationDelay)});return n.parse(e).then(l=>{let u=this.hydrator.hydrate(l);i.resolve(u)}).catch(l=>{i.reject(l)}).finally(()=>{s.dispose(),clearTimeout(a)}),i.promise}terminateWorker(e){e.terminate();let r=this.workerPool.indexOf(e);r>=0&&this.workerPool.splice(r,1)}async acquireParserWorker(e){this.initializeWorkers();for(let n of this.workerPool)if(n.ready)return n.lock(),n;let r=new cs;return e.onCancellationRequested(()=>{let n=this.queue.indexOf(r);n>=0&&this.queue.splice(n,1),r.reject(Pc)}),this.queue.push(r),r.promise}},uI=class{static{o(this,"ParserWorker")}get ready(){return this._ready}get onReady(){return this.onReadyEmitter.event}constructor(e,r,n,i){this.onReadyEmitter=new Kn.Emitter,this.deferred=new cs,this._ready=!0,this._parsing=!1,this.sendMessage=e,this._terminate=i,r(a=>{let s=a;this.deferred.resolve(s),this.unlock()}),n(a=>{this.deferred.reject(a),this.unlock()})}terminate(){this.deferred.reject(Pc),this._terminate()}lock(){this._ready=!1}unlock(){this._parsing=!1,this._ready=!0,this.onReadyEmitter.fire()}parse(e){if(this._parsing)throw new Error("Parser worker is busy");return this._parsing=!0,this.deferred=new cs,this.sendMessage(e),this.deferred.promise}}});var ab,fI=N(()=>{"use strict";qo();Yo();ab=class{static{o(this,"DefaultWorkspaceLock")}constructor(){this.previousTokenSource=new yr.CancellationTokenSource,this.writeQueue=[],this.readQueue=[],this.done=!0}write(e){this.cancelWrite();let r=CE();return this.previousTokenSource=r,this.enqueue(this.writeQueue,e,r.token)}read(e){return this.enqueue(this.readQueue,e)}enqueue(e,r,n=yr.CancellationToken.None){let i=new cs,a={action:r,deferred:i,cancellationToken:n};return e.push(a),this.performNextOperation(),i.promise}async performNextOperation(){if(!this.done)return;let e=[];if(this.writeQueue.length>0)e.push(this.writeQueue.shift());else if(this.readQueue.length>0)e.push(...this.readQueue.splice(0,this.readQueue.length));else return;this.done=!1,await Promise.all(e.map(async({action:r,deferred:n,cancellationToken:i})=>{try{let a=await Promise.resolve().then(()=>r(i));n.resolve(a)}catch(a){Bc(a)?n.resolve(void 0):n.reject(a)}})),this.done=!0,this.performNextOperation()}cancelWrite(){this.previousTokenSource.cancel()}}});var sb,dI=N(()=>{"use strict";gE();Rc();Rl();is();f1();Nl();sb=class{static{o(this,"DefaultHydrator")}constructor(e){this.grammarElementIdMap=new vp,this.tokenTypeIdMap=new vp,this.grammar=e.Grammar,this.lexer=e.parser.Lexer,this.linker=e.references.Linker}dehydrate(e){return{lexerErrors:e.lexerErrors,lexerReport:e.lexerReport?this.dehydrateLexerReport(e.lexerReport):void 0,parserErrors:e.parserErrors.map(r=>Object.assign(Object.assign({},r),{message:r.message})),value:this.dehydrateAstNode(e.value,this.createDehyrationContext(e.value))}}dehydrateLexerReport(e){return e}createDehyrationContext(e){let r=new Map,n=new Map;for(let i of Wo(e))r.set(i,{});if(e.$cstNode)for(let i of Kd(e.$cstNode))n.set(i,{});return{astNodes:r,cstNodes:n}}dehydrateAstNode(e,r){let n=r.astNodes.get(e);n.$type=e.$type,n.$containerIndex=e.$containerIndex,n.$containerProperty=e.$containerProperty,e.$cstNode!==void 0&&(n.$cstNode=this.dehydrateCstNode(e.$cstNode,r));for(let[i,a]of Object.entries(e))if(!i.startsWith("$"))if(Array.isArray(a)){let s=[];n[i]=s;for(let l of a)ii(l)?s.push(this.dehydrateAstNode(l,r)):va(l)?s.push(this.dehydrateReference(l,r)):s.push(l)}else ii(a)?n[i]=this.dehydrateAstNode(a,r):va(a)?n[i]=this.dehydrateReference(a,r):a!==void 0&&(n[i]=a);return n}dehydrateReference(e,r){let n={};return n.$refText=e.$refText,e.$refNode&&(n.$refNode=r.cstNodes.get(e.$refNode)),n}dehydrateCstNode(e,r){let n=r.cstNodes.get(e);return M2(e)?n.fullText=e.fullText:n.grammarSource=this.getGrammarElementId(e.grammarSource),n.hidden=e.hidden,n.astNode=r.astNodes.get(e.astNode),Ll(e)?n.content=e.content.map(i=>this.dehydrateCstNode(i,r)):af(e)&&(n.tokenType=e.tokenType.name,n.offset=e.offset,n.length=e.length,n.startLine=e.range.start.line,n.startColumn=e.range.start.character,n.endLine=e.range.end.line,n.endColumn=e.range.end.character),n}hydrate(e){let r=e.value,n=this.createHydrationContext(r);return"$cstNode"in r&&this.hydrateCstNode(r.$cstNode,n),{lexerErrors:e.lexerErrors,lexerReport:e.lexerReport,parserErrors:e.parserErrors,value:this.hydrateAstNode(r,n)}}createHydrationContext(e){let r=new Map,n=new Map;for(let a of Wo(e))r.set(a,{});let i;if(e.$cstNode)for(let a of Kd(e.$cstNode)){let s;"fullText"in a?(s=new a1(a.fullText),i=s):"content"in a?s=new mp:"tokenType"in a&&(s=this.hydrateCstLeafNode(a)),s&&(n.set(a,s),s.root=i)}return{astNodes:r,cstNodes:n}}hydrateAstNode(e,r){let n=r.astNodes.get(e);n.$type=e.$type,n.$containerIndex=e.$containerIndex,n.$containerProperty=e.$containerProperty,e.$cstNode&&(n.$cstNode=r.cstNodes.get(e.$cstNode));for(let[i,a]of Object.entries(e))if(!i.startsWith("$"))if(Array.isArray(a)){let s=[];n[i]=s;for(let l of a)ii(l)?s.push(this.setParent(this.hydrateAstNode(l,r),n)):va(l)?s.push(this.hydrateReference(l,n,i,r)):s.push(l)}else ii(a)?n[i]=this.setParent(this.hydrateAstNode(a,r),n):va(a)?n[i]=this.hydrateReference(a,n,i,r):a!==void 0&&(n[i]=a);return n}setParent(e,r){return e.$container=r,e}hydrateReference(e,r,n,i){return this.linker.buildReference(r,n,i.cstNodes.get(e.$refNode),e.$refText)}hydrateCstNode(e,r,n=0){let i=r.cstNodes.get(e);if(typeof e.grammarSource=="number"&&(i.grammarSource=this.getGrammarElement(e.grammarSource)),i.astNode=r.astNodes.get(e.astNode),Ll(i))for(let a of e.content){let s=this.hydrateCstNode(a,r,n++);i.content.push(s)}return i}hydrateCstLeafNode(e){let r=this.getTokenType(e.tokenType),n=e.offset,i=e.length,a=e.startLine,s=e.startColumn,l=e.endLine,u=e.endColumn,h=e.hidden;return new pp(n,i,{start:{line:a,character:s},end:{line:l,character:u}},r,h)}getTokenType(e){return this.lexer.definition[e]}getGrammarElementId(e){if(e)return this.grammarElementIdMap.size===0&&this.createGrammarElementIdMap(),this.grammarElementIdMap.get(e)}getGrammarElement(e){return this.grammarElementIdMap.size===0&&this.createGrammarElementIdMap(),this.grammarElementIdMap.getKey(e)}createGrammarElementIdMap(){let e=0;for(let r of Wo(this.grammar))G2(r)&&this.grammarElementIdMap.set(r,e++)}}});function fs(t){return{documentation:{CommentProvider:o(e=>new nb(e),"CommentProvider"),DocumentationProvider:o(e=>new rb(e),"DocumentationProvider")},parser:{AsyncParser:o(e=>new ib(e),"AsyncParser"),GrammarConfig:o(e=>pN(e),"GrammarConfig"),LangiumParser:o(e=>TM(e),"LangiumParser"),CompletionParser:o(e=>bM(e),"CompletionParser"),ValueConverter:o(()=>new yp,"ValueConverter"),TokenBuilder:o(()=>new Uu,"TokenBuilder"),Lexer:o(e=>new wp(e),"Lexer"),ParserErrorMessageProvider:o(()=>new s1,"ParserErrorMessageProvider"),LexerErrorMessageProvider:o(()=>new Jx,"LexerErrorMessageProvider")},workspace:{AstNodeLocator:o(()=>new Xx,"AstNodeLocator"),AstNodeDescriptionProvider:o(e=>new qx(e),"AstNodeDescriptionProvider"),ReferenceDescriptionProvider:o(e=>new Yx(e),"ReferenceDescriptionProvider")},references:{Linker:o(e=>new Ix(e),"Linker"),NameProvider:o(()=>new Ox,"NameProvider"),ScopeProvider:o(e=>new zx(e),"ScopeProvider"),ScopeComputation:o(e=>new Bx(e),"ScopeComputation"),References:o(e=>new Px(e),"References")},serializer:{Hydrator:o(e=>new sb(e),"Hydrator"),JsonSerializer:o(e=>new Gx(e),"JsonSerializer")},validation:{DocumentValidator:o(e=>new Wx(e),"DocumentValidator"),ValidationRegistry:o(e=>new Ux(e),"ValidationRegistry")},shared:o(()=>t.shared,"shared")}}function ds(t){return{ServiceRegistry:o(e=>new Vx(e),"ServiceRegistry"),workspace:{LangiumDocuments:o(e=>new Mx(e),"LangiumDocuments"),LangiumDocumentFactory:o(e=>new Nx(e),"LangiumDocumentFactory"),DocumentBuilder:o(e=>new Kx(e),"DocumentBuilder"),IndexManager:o(e=>new Qx(e),"IndexManager"),WorkspaceManager:o(e=>new Zx(e),"WorkspaceManager"),FileSystemProvider:o(e=>t.fileSystemProvider(e),"FileSystemProvider"),WorkspaceLock:o(()=>new ab,"WorkspaceLock"),ConfigurationProvider:o(e=>new jx(e),"ConfigurationProvider")}}}var pI=N(()=>{"use strict";mN();wM();kM();wE();EM();BM();FM();$M();zM();VM();LE();HM();WM();Hx();qM();YM();XM();KM();h1();QM();ZM();OE();oI();lI();Lx();hI();fI();dI();o(fs,"createDefaultCoreModule");o(ds,"createDefaultSharedCoreModule")});function ui(t,e,r,n,i,a,s,l,u){let h=[t,e,r,n,i,a,s,l,u].reduce(FE,{});return ace(h)}function ice(t){if(t&&t[nce])for(let e of Object.values(t))ice(e);return t}function ace(t,e){let r=new Proxy({},{deleteProperty:o(()=>!1,"deleteProperty"),set:o(()=>{throw new Error("Cannot set property on injected service container")},"set"),get:o((n,i)=>i===nce?!0:rce(n,i,t,e||r),"get"),getOwnPropertyDescriptor:o((n,i)=>(rce(n,i,t,e||r),Object.getOwnPropertyDescriptor(n,i)),"getOwnPropertyDescriptor"),has:o((n,i)=>i in t,"has"),ownKeys:o(()=>[...Object.getOwnPropertyNames(t)],"ownKeys")});return r}function rce(t,e,r,n){if(e in t){if(t[e]instanceof Error)throw new Error("Construction failure. Please make sure that your dependencies are constructable.",{cause:t[e]});if(t[e]===tce)throw new Error('Cycle detected. Please make "'+String(e)+'" lazy. Visit https://langium.org/docs/reference/configuration-services/#resolving-cyclic-dependencies');return t[e]}else if(e in r){let i=r[e];t[e]=tce;try{t[e]=typeof i=="function"?i(n):ace(i,n)}catch(a){throw t[e]=a instanceof Error?a:void 0,a}return t[e]}else return}function FE(t,e){if(e){for(let[r,n]of Object.entries(e))if(n!==void 0){let i=t[r];i!==null&&n!==null&&typeof i=="object"&&typeof n=="object"?t[r]=FE(i,n):t[r]=n}}return t}var mI,nce,tce,gI=N(()=>{"use strict";(function(t){t.merge=(e,r)=>FE(FE({},e),r)})(mI||(mI={}));o(ui,"inject");nce=Symbol("isProxy");o(ice,"eagerLoad");o(ace,"_inject");tce=Symbol();o(rce,"_resolve");o(FE,"_merge")});var sce=N(()=>{"use strict"});var oce=N(()=>{"use strict";lI();oI();sI()});var lce=N(()=>{"use strict"});var cce=N(()=>{"use strict";mN();lce()});var yI,Tp,$E,vI,uce=N(()=>{"use strict";cf();wE();OE();yI={indentTokenName:"INDENT",dedentTokenName:"DEDENT",whitespaceTokenName:"WS",ignoreIndentationDelimiters:[]};(function(t){t.REGULAR="indentation-sensitive",t.IGNORE_INDENTATION="ignore-indentation"})(Tp||(Tp={}));$E=class extends Uu{static{o(this,"IndentationAwareTokenBuilder")}constructor(e=yI){super(),this.indentationStack=[0],this.whitespaceRegExp=/[ \t]+/y,this.options=Object.assign(Object.assign({},yI),e),this.indentTokenType=of({name:this.options.indentTokenName,pattern:this.indentMatcher.bind(this),line_breaks:!1}),this.dedentTokenType=of({name:this.options.dedentTokenName,pattern:this.dedentMatcher.bind(this),line_breaks:!1})}buildTokens(e,r){let n=super.buildTokens(e,r);if(!IE(n))throw new Error("Invalid tokens built by default builder");let{indentTokenName:i,dedentTokenName:a,whitespaceTokenName:s,ignoreIndentationDelimiters:l}=this.options,u,h,f,d=[];for(let p of n){for(let[m,g]of l)p.name===m?p.PUSH_MODE=Tp.IGNORE_INDENTATION:p.name===g&&(p.POP_MODE=!0);p.name===a?u=p:p.name===i?h=p:p.name===s?f=p:d.push(p)}if(!u||!h||!f)throw new Error("Some indentation/whitespace tokens not found!");return l.length>0?{modes:{[Tp.REGULAR]:[u,h,...d,f],[Tp.IGNORE_INDENTATION]:[...d,f]},defaultMode:Tp.REGULAR}:[u,h,f,...d]}flushLexingReport(e){let r=super.flushLexingReport(e);return Object.assign(Object.assign({},r),{remainingDedents:this.flushRemainingDedents(e)})}isStartOfLine(e,r){return r===0||`\r +`.includes(e[r-1])}matchWhitespace(e,r,n,i){var a;this.whitespaceRegExp.lastIndex=r;let s=this.whitespaceRegExp.exec(e);return{currIndentLevel:(a=s?.[0].length)!==null&&a!==void 0?a:0,prevIndentLevel:this.indentationStack.at(-1),match:s}}createIndentationTokenInstance(e,r,n,i){let a=this.getLineNumber(r,i);return $u(e,n,i,i+n.length,a,a,1,n.length)}getLineNumber(e,r){return e.substring(0,r).split(/\r\n|\r|\n/).length}indentMatcher(e,r,n,i){if(!this.isStartOfLine(e,r))return null;let{currIndentLevel:a,prevIndentLevel:s,match:l}=this.matchWhitespace(e,r,n,i);return a<=s?null:(this.indentationStack.push(a),l)}dedentMatcher(e,r,n,i){var a,s,l,u;if(!this.isStartOfLine(e,r))return null;let{currIndentLevel:h,prevIndentLevel:f,match:d}=this.matchWhitespace(e,r,n,i);if(h>=f)return null;let p=this.indentationStack.lastIndexOf(h);if(p===-1)return this.diagnostics.push({severity:"error",message:`Invalid dedent level ${h} at offset: ${r}. Current indentation stack: ${this.indentationStack}`,offset:r,length:(s=(a=d?.[0])===null||a===void 0?void 0:a.length)!==null&&s!==void 0?s:0,line:this.getLineNumber(e,r),column:1}),null;let m=this.indentationStack.length-p-1,g=(u=(l=e.substring(0,r).match(/[\r\n]+$/))===null||l===void 0?void 0:l[0].length)!==null&&u!==void 0?u:1;for(let y=0;y1;)r.push(this.createIndentationTokenInstance(this.dedentTokenType,e,"",e.length)),this.indentationStack.pop();return this.indentationStack=[0],r}},vI=class extends wp{static{o(this,"IndentationAwareLexer")}constructor(e){if(super(e),e.parser.TokenBuilder instanceof $E)this.indentationTokenBuilder=e.parser.TokenBuilder;else throw new Error("IndentationAwareLexer requires an accompanying IndentationAwareTokenBuilder")}tokenize(e,r=ME){let n=super.tokenize(e),i=n.report;r?.mode==="full"&&n.tokens.push(...i.remainingDedents),i.remainingDedents=[];let{indentTokenType:a,dedentTokenType:s}=this.indentationTokenBuilder,l=a.tokenTypeIdx,u=s.tokenTypeIdx,h=[],f=n.tokens.length-1;for(let d=0;d=0&&h.push(n.tokens[f]),n.tokens=h,n}}});var hce=N(()=>{"use strict"});var fce=N(()=>{"use strict";hI();wM();gE();uce();kM();Lx();OE();bE();hce();wE();EM()});var dce=N(()=>{"use strict";BM();FM();$M();GM();zM();VM()});var pce=N(()=>{"use strict";dI();LE()});var zE,ps,xI=N(()=>{"use strict";zE=class{static{o(this,"EmptyFileSystemProvider")}readFile(){throw new Error("No file system is available.")}async readDirectory(){return[]}},ps={fileSystemProvider:o(()=>new zE,"fileSystemProvider")}});function IFe(){let t=ui(ds(ps),MFe),e=ui(fs({shared:t}),NFe);return t.ServiceRegistry.register(e),e}function Hu(t){var e;let r=IFe(),n=r.serializer.JsonSerializer.deserialize(t);return r.shared.workspace.LangiumDocumentFactory.fromModel(n,us.parse(`memory://${(e=n.name)!==null&&e!==void 0?e:"grammar"}.langium`)),n}var NFe,MFe,mce=N(()=>{"use strict";pI();gI();Rc();xI();Fc();NFe={Grammar:o(()=>{},"Grammar"),LanguageMetaData:o(()=>({caseInsensitive:!1,fileExtensions:[".langium"],languageId:"langium"}),"LanguageMetaData")},MFe={AstReflection:o(()=>new Cg,"AstReflection")};o(IFe,"createMinimalGrammarServices");o(Hu,"loadGrammarFromJson")});var Gr={};hr(Gr,{AstUtils:()=>xk,BiMap:()=>vp,Cancellation:()=>yr,ContextCache:()=>xp,CstUtils:()=>ck,DONE_RESULT:()=>Ia,Deferred:()=>cs,Disposable:()=>ff,DisposableCache:()=>p1,DocumentCache:()=>_E,EMPTY_STREAM:()=>I2,ErrorWithLocation:()=>Zd,GrammarUtils:()=>Ek,MultiMap:()=>Bl,OperationCancelled:()=>Pc,Reduction:()=>zm,RegExpUtils:()=>Tk,SimpleCache:()=>$x,StreamImpl:()=>ao,TreeStreamImpl:()=>_c,URI:()=>us,UriUtils:()=>hs,WorkspaceCache:()=>m1,assertUnreachable:()=>Lc,delayNextTick:()=>MM,interruptAndCheck:()=>xi,isOperationCancelled:()=>Bc,loadGrammarFromJson:()=>Hu,setInterruptionPeriod:()=>$le,startCancelableOperation:()=>CE,stream:()=>en});var gce=N(()=>{"use strict";DE();NE();Sr(Gr,Kn);f1();jM();uk();mce();Yo();Ps();Fc();is();qo();Nl();Ol();Lg()});var yce=N(()=>{"use strict";WM();Hx()});var vce=N(()=>{"use strict";qM();YM();XM();KM();h1();xI();QM();fI();ZM()});var xa={};hr(xa,{AbstractAstReflection:()=>Xd,AbstractCstNode:()=>Cx,AbstractLangiumParser:()=>Ax,AbstractParserErrorMessageProvider:()=>vE,AbstractThreadedAsyncParser:()=>cI,AstUtils:()=>xk,BiMap:()=>vp,Cancellation:()=>yr,CompositeCstNodeImpl:()=>mp,ContextCache:()=>xp,CstNodeBuilder:()=>Sx,CstUtils:()=>ck,DEFAULT_TOKENIZE_OPTIONS:()=>ME,DONE_RESULT:()=>Ia,DatatypeSymbol:()=>yE,DefaultAstNodeDescriptionProvider:()=>qx,DefaultAstNodeLocator:()=>Xx,DefaultAsyncParser:()=>ib,DefaultCommentProvider:()=>nb,DefaultConfigurationProvider:()=>jx,DefaultDocumentBuilder:()=>Kx,DefaultDocumentValidator:()=>Wx,DefaultHydrator:()=>sb,DefaultIndexManager:()=>Qx,DefaultJsonSerializer:()=>Gx,DefaultLangiumDocumentFactory:()=>Nx,DefaultLangiumDocuments:()=>Mx,DefaultLexer:()=>wp,DefaultLexerErrorMessageProvider:()=>Jx,DefaultLinker:()=>Ix,DefaultNameProvider:()=>Ox,DefaultReferenceDescriptionProvider:()=>Yx,DefaultReferences:()=>Px,DefaultScopeComputation:()=>Bx,DefaultScopeProvider:()=>zx,DefaultServiceRegistry:()=>Vx,DefaultTokenBuilder:()=>Uu,DefaultValueConverter:()=>yp,DefaultWorkspaceLock:()=>ab,DefaultWorkspaceManager:()=>Zx,Deferred:()=>cs,Disposable:()=>ff,DisposableCache:()=>p1,DocumentCache:()=>_E,DocumentState:()=>kn,DocumentValidator:()=>jo,EMPTY_SCOPE:()=>xFe,EMPTY_STREAM:()=>I2,EmptyFileSystem:()=>ps,EmptyFileSystemProvider:()=>zE,ErrorWithLocation:()=>Zd,GrammarAST:()=>U2,GrammarUtils:()=>Ek,IndentationAwareLexer:()=>vI,IndentationAwareTokenBuilder:()=>$E,JSDocDocumentationProvider:()=>rb,LangiumCompletionParser:()=>Dx,LangiumParser:()=>_x,LangiumParserErrorMessageProvider:()=>s1,LeafCstNodeImpl:()=>pp,LexingMode:()=>Tp,MapScope:()=>Fx,Module:()=>mI,MultiMap:()=>Bl,OperationCancelled:()=>Pc,ParserWorker:()=>uI,Reduction:()=>zm,RegExpUtils:()=>Tk,RootCstNodeImpl:()=>a1,SimpleCache:()=>$x,StreamImpl:()=>ao,StreamScope:()=>d1,TextDocument:()=>c1,TreeStreamImpl:()=>_c,URI:()=>us,UriUtils:()=>hs,ValidationCategory:()=>g1,ValidationRegistry:()=>Ux,ValueConverter:()=>Oc,WorkspaceCache:()=>m1,assertUnreachable:()=>Lc,createCompletionParser:()=>bM,createDefaultCoreModule:()=>fs,createDefaultSharedCoreModule:()=>ds,createGrammarConfig:()=>pN,createLangiumParser:()=>TM,createParser:()=>Rx,delayNextTick:()=>MM,diagnosticData:()=>bp,eagerLoad:()=>ice,getDiagnosticRange:()=>Yle,indentationBuilderDefaultOptions:()=>yI,inject:()=>ui,interruptAndCheck:()=>xi,isAstNode:()=>ii,isAstNodeDescription:()=>kR,isAstNodeWithComment:()=>UM,isCompositeCstNode:()=>Ll,isIMultiModeLexerDefinition:()=>eI,isJSDoc:()=>iI,isLeafCstNode:()=>af,isLinkingError:()=>jd,isNamed:()=>Wle,isOperationCancelled:()=>Bc,isReference:()=>va,isRootCstNode:()=>M2,isTokenTypeArray:()=>IE,isTokenTypeDictionary:()=>JM,loadGrammarFromJson:()=>Hu,parseJSDoc:()=>nI,prepareLangiumParser:()=>Nle,setInterruptionPeriod:()=>$le,startCancelableOperation:()=>CE,stream:()=>en,toDiagnosticData:()=>Xle,toDiagnosticSeverity:()=>RE});var Xo=N(()=>{"use strict";pI();gI();HM();sce();Rl();oce();cce();fce();dce();pce();gce();Sr(xa,Gr);yce();vce();Rc()});function Sce(t){return Fl.isInstance(t,ob)}function Cce(t){return Fl.isInstance(t,y1)}function Ace(t){return Fl.isInstance(t,v1)}function _ce(t){return Fl.isInstance(t,WE)}function Dce(t){return Fl.isInstance(t,x1)}function Lce(t){return Fl.isInstance(t,lb)}function Rce(t){return Fl.isInstance(t,b1)}function Nce(t){return Fl.isInstance(t,cb)}function Mce(t){return Fl.isInstance(t,ub)}function Ice(t){return Fl.isInstance(t,hb)}function Oce(t){return Fl.isInstance(t,fb)}var OFe,Lt,AI,ob,GE,y1,VE,UE,v1,WE,bI,wI,TI,x1,kI,lb,EI,b1,SI,cb,ub,hb,fb,qE,CI,HE,Pce,Fl,xce,PFe,bce,BFe,wce,FFe,Tce,$Fe,kce,zFe,Ece,GFe,VFe,UFe,HFe,WFe,qFe,YFe,co,_I,DI,LI,RI,NI,MI,XFe,jFe,KFe,QFe,w1,Wu,$s,ZFe,zs=N(()=>{"use strict";Xo();Xo();Xo();Xo();OFe=Object.defineProperty,Lt=o((t,e)=>OFe(t,"name",{value:e,configurable:!0}),"__name"),AI="Statement",ob="Architecture";o(Sce,"isArchitecture");Lt(Sce,"isArchitecture");GE="Axis",y1="Branch";o(Cce,"isBranch");Lt(Cce,"isBranch");VE="Checkout",UE="CherryPicking",v1="Commit";o(Ace,"isCommit");Lt(Ace,"isCommit");WE="Common";o(_ce,"isCommon");Lt(_ce,"isCommon");bI="Curve",wI="Edge",TI="Entry",x1="GitGraph";o(Dce,"isGitGraph");Lt(Dce,"isGitGraph");kI="Group",lb="Info";o(Lce,"isInfo");Lt(Lce,"isInfo");EI="Junction",b1="Merge";o(Rce,"isMerge");Lt(Rce,"isMerge");SI="Option",cb="Packet";o(Nce,"isPacket");Lt(Nce,"isPacket");ub="PacketBlock";o(Mce,"isPacketBlock");Lt(Mce,"isPacketBlock");hb="Pie";o(Ice,"isPie");Lt(Ice,"isPie");fb="PieSection";o(Oce,"isPieSection");Lt(Oce,"isPieSection");qE="Radar",CI="Service",HE="Direction",Pce=class extends Xd{static{o(this,"MermaidAstReflection")}static{Lt(this,"MermaidAstReflection")}getAllTypes(){return[ob,GE,y1,VE,UE,v1,WE,bI,HE,wI,TI,x1,kI,lb,EI,b1,SI,cb,ub,hb,fb,qE,CI,AI]}computeIsSubtype(t,e){switch(t){case y1:case VE:case UE:case v1:case b1:return this.isSubtype(AI,e);case HE:return this.isSubtype(x1,e);default:return!1}}getReferenceType(t){let e=`${t.container.$type}:${t.property}`;switch(e){case"Entry:axis":return GE;default:throw new Error(`${e} is not a valid reference id.`)}}getTypeMetaData(t){switch(t){case ob:return{name:ob,properties:[{name:"accDescr"},{name:"accTitle"},{name:"edges",defaultValue:[]},{name:"groups",defaultValue:[]},{name:"junctions",defaultValue:[]},{name:"services",defaultValue:[]},{name:"title"}]};case GE:return{name:GE,properties:[{name:"label"},{name:"name"}]};case y1:return{name:y1,properties:[{name:"name"},{name:"order"}]};case VE:return{name:VE,properties:[{name:"branch"}]};case UE:return{name:UE,properties:[{name:"id"},{name:"parent"},{name:"tags",defaultValue:[]}]};case v1:return{name:v1,properties:[{name:"id"},{name:"message"},{name:"tags",defaultValue:[]},{name:"type"}]};case WE:return{name:WE,properties:[{name:"accDescr"},{name:"accTitle"},{name:"title"}]};case bI:return{name:bI,properties:[{name:"entries",defaultValue:[]},{name:"label"},{name:"name"}]};case wI:return{name:wI,properties:[{name:"lhsDir"},{name:"lhsGroup",defaultValue:!1},{name:"lhsId"},{name:"lhsInto",defaultValue:!1},{name:"rhsDir"},{name:"rhsGroup",defaultValue:!1},{name:"rhsId"},{name:"rhsInto",defaultValue:!1},{name:"title"}]};case TI:return{name:TI,properties:[{name:"axis"},{name:"value"}]};case x1:return{name:x1,properties:[{name:"accDescr"},{name:"accTitle"},{name:"statements",defaultValue:[]},{name:"title"}]};case kI:return{name:kI,properties:[{name:"icon"},{name:"id"},{name:"in"},{name:"title"}]};case lb:return{name:lb,properties:[{name:"accDescr"},{name:"accTitle"},{name:"title"}]};case EI:return{name:EI,properties:[{name:"id"},{name:"in"}]};case b1:return{name:b1,properties:[{name:"branch"},{name:"id"},{name:"tags",defaultValue:[]},{name:"type"}]};case SI:return{name:SI,properties:[{name:"name"},{name:"value",defaultValue:!1}]};case cb:return{name:cb,properties:[{name:"accDescr"},{name:"accTitle"},{name:"blocks",defaultValue:[]},{name:"title"}]};case ub:return{name:ub,properties:[{name:"end"},{name:"label"},{name:"start"}]};case hb:return{name:hb,properties:[{name:"accDescr"},{name:"accTitle"},{name:"sections",defaultValue:[]},{name:"showData",defaultValue:!1},{name:"title"}]};case fb:return{name:fb,properties:[{name:"label"},{name:"value"}]};case qE:return{name:qE,properties:[{name:"accDescr"},{name:"accTitle"},{name:"axes",defaultValue:[]},{name:"curves",defaultValue:[]},{name:"options",defaultValue:[]},{name:"title"}]};case CI:return{name:CI,properties:[{name:"icon"},{name:"iconText"},{name:"id"},{name:"in"},{name:"title"}]};case HE:return{name:HE,properties:[{name:"accDescr"},{name:"accTitle"},{name:"dir"},{name:"statements",defaultValue:[]},{name:"title"}]};default:return{name:t,properties:[]}}}},Fl=new Pce,PFe=Lt(()=>xce??(xce=Hu('{"$type":"Grammar","isDeclared":true,"name":"Info","imports":[],"rules":[{"$type":"ParserRule","entry":true,"name":"Info","definition":{"$type":"Group","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@3"},"arguments":[],"cardinality":"*"},{"$type":"Keyword","value":"info"},{"$type":"RuleCall","rule":{"$ref":"#/rules@3"},"arguments":[],"cardinality":"*"},{"$type":"Group","elements":[{"$type":"Keyword","value":"showInfo"},{"$type":"RuleCall","rule":{"$ref":"#/rules@3"},"arguments":[],"cardinality":"*"}],"cardinality":"?"},{"$type":"RuleCall","rule":{"$ref":"#/rules@1"},"arguments":[],"cardinality":"?"}]},"definesHiddenTokens":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","fragment":true,"name":"TitleAndAccessibilities","definition":{"$type":"Group","elements":[{"$type":"Alternatives","elements":[{"$type":"Assignment","feature":"accDescr","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@4"},"arguments":[]}},{"$type":"Assignment","feature":"accTitle","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@5"},"arguments":[]}},{"$type":"Assignment","feature":"title","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@6"},"arguments":[]}}]},{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[]}],"cardinality":"+"},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","fragment":true,"name":"EOL","dataType":"string","definition":{"$type":"Alternatives","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@3"},"arguments":[],"cardinality":"+"},{"$type":"EndOfFile"}]},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"TerminalRule","name":"NEWLINE","definition":{"$type":"RegexToken","regex":"/\\\\r?\\\\n/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ACC_DESCR","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*accDescr(?:[\\\\t ]*:([^\\\\n\\\\r]*?(?=%%)|[^\\\\n\\\\r]*)|\\\\s*{([^}]*)})/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ACC_TITLE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*accTitle[\\\\t ]*:(?:[^\\\\n\\\\r]*?(?=%%)|[^\\\\n\\\\r]*)/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"TITLE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*title(?:[\\\\t ][^\\\\n\\\\r]*?(?=%%)|[\\\\t ][^\\\\n\\\\r]*|)/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","hidden":true,"name":"WHITESPACE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]+/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"YAML","definition":{"$type":"RegexToken","regex":"/---[\\\\t ]*\\\\r?\\\\n(?:[\\\\S\\\\s]*?\\\\r?\\\\n)?---(?:\\\\r?\\\\n|(?!\\\\S))/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"DIRECTIVE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*%%{[\\\\S\\\\s]*?}%%(?:\\\\r?\\\\n|(?!\\\\S))/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"SINGLE_LINE_COMMENT","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*%%[^\\\\n\\\\r]*/"},"fragment":false}],"definesHiddenTokens":false,"hiddenTokens":[],"interfaces":[{"$type":"Interface","name":"Common","attributes":[{"$type":"TypeAttribute","name":"accDescr","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}},{"$type":"TypeAttribute","name":"accTitle","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}},{"$type":"TypeAttribute","name":"title","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}}],"superTypes":[]}],"types":[],"usedGrammars":[]}')),"InfoGrammar"),BFe=Lt(()=>bce??(bce=Hu(`{"$type":"Grammar","isDeclared":true,"name":"Packet","imports":[],"rules":[{"$type":"ParserRule","entry":true,"name":"Packet","definition":{"$type":"Group","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@6"},"arguments":[],"cardinality":"*"},{"$type":"Keyword","value":"packet-beta"},{"$type":"Alternatives","elements":[{"$type":"Group","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@6"},"arguments":[],"cardinality":"*"},{"$type":"RuleCall","rule":{"$ref":"#/rules@4"},"arguments":[]},{"$type":"Assignment","feature":"blocks","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@1"},"arguments":[]},"cardinality":"*"}]},{"$type":"Group","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@6"},"arguments":[],"cardinality":"+"},{"$type":"Assignment","feature":"blocks","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@1"},"arguments":[]},"cardinality":"+"}]},{"$type":"RuleCall","rule":{"$ref":"#/rules@6"},"arguments":[],"cardinality":"*"}]}]},"definesHiddenTokens":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"PacketBlock","definition":{"$type":"Group","elements":[{"$type":"Assignment","feature":"start","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[]}},{"$type":"Group","elements":[{"$type":"Keyword","value":"-"},{"$type":"Assignment","feature":"end","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[]}}],"cardinality":"?"},{"$type":"Keyword","value":":"},{"$type":"Assignment","feature":"label","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@3"},"arguments":[]}},{"$type":"RuleCall","rule":{"$ref":"#/rules@5"},"arguments":[]}]},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"TerminalRule","name":"INT","type":{"$type":"ReturnType","name":"number"},"definition":{"$type":"RegexToken","regex":"/0|[1-9][0-9]*/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"STRING","definition":{"$type":"RegexToken","regex":"/\\"[^\\"]*\\"|'[^']*'/"},"fragment":false,"hidden":false},{"$type":"ParserRule","fragment":true,"name":"TitleAndAccessibilities","definition":{"$type":"Group","elements":[{"$type":"Alternatives","elements":[{"$type":"Assignment","feature":"accDescr","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@7"},"arguments":[]}},{"$type":"Assignment","feature":"accTitle","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@8"},"arguments":[]}},{"$type":"Assignment","feature":"title","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@9"},"arguments":[]}}]},{"$type":"RuleCall","rule":{"$ref":"#/rules@5"},"arguments":[]}],"cardinality":"+"},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","fragment":true,"name":"EOL","dataType":"string","definition":{"$type":"Alternatives","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@6"},"arguments":[],"cardinality":"+"},{"$type":"EndOfFile"}]},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"TerminalRule","name":"NEWLINE","definition":{"$type":"RegexToken","regex":"/\\\\r?\\\\n/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ACC_DESCR","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*accDescr(?:[\\\\t ]*:([^\\\\n\\\\r]*?(?=%%)|[^\\\\n\\\\r]*)|\\\\s*{([^}]*)})/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ACC_TITLE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*accTitle[\\\\t ]*:(?:[^\\\\n\\\\r]*?(?=%%)|[^\\\\n\\\\r]*)/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"TITLE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*title(?:[\\\\t ][^\\\\n\\\\r]*?(?=%%)|[\\\\t ][^\\\\n\\\\r]*|)/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","hidden":true,"name":"WHITESPACE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]+/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"YAML","definition":{"$type":"RegexToken","regex":"/---[\\\\t ]*\\\\r?\\\\n(?:[\\\\S\\\\s]*?\\\\r?\\\\n)?---(?:\\\\r?\\\\n|(?!\\\\S))/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"DIRECTIVE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*%%{[\\\\S\\\\s]*?}%%(?:\\\\r?\\\\n|(?!\\\\S))/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"SINGLE_LINE_COMMENT","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*%%[^\\\\n\\\\r]*/"},"fragment":false}],"definesHiddenTokens":false,"hiddenTokens":[],"interfaces":[{"$type":"Interface","name":"Common","attributes":[{"$type":"TypeAttribute","name":"accDescr","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}},{"$type":"TypeAttribute","name":"accTitle","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}},{"$type":"TypeAttribute","name":"title","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}}],"superTypes":[]}],"types":[],"usedGrammars":[]}`)),"PacketGrammar"),FFe=Lt(()=>wce??(wce=Hu('{"$type":"Grammar","isDeclared":true,"name":"Pie","imports":[],"rules":[{"$type":"ParserRule","entry":true,"name":"Pie","definition":{"$type":"Group","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@6"},"arguments":[],"cardinality":"*"},{"$type":"Keyword","value":"pie"},{"$type":"Assignment","feature":"showData","operator":"?=","terminal":{"$type":"Keyword","value":"showData"},"cardinality":"?"},{"$type":"Alternatives","elements":[{"$type":"Group","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@6"},"arguments":[],"cardinality":"*"},{"$type":"RuleCall","rule":{"$ref":"#/rules@4"},"arguments":[]},{"$type":"Assignment","feature":"sections","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@1"},"arguments":[]},"cardinality":"*"}]},{"$type":"Group","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@6"},"arguments":[],"cardinality":"+"},{"$type":"Assignment","feature":"sections","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@1"},"arguments":[]},"cardinality":"+"}]},{"$type":"RuleCall","rule":{"$ref":"#/rules@6"},"arguments":[],"cardinality":"*"}]}]},"definesHiddenTokens":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"PieSection","definition":{"$type":"Group","elements":[{"$type":"Assignment","feature":"label","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[]}},{"$type":"Keyword","value":":"},{"$type":"Assignment","feature":"value","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@3"},"arguments":[]}},{"$type":"RuleCall","rule":{"$ref":"#/rules@5"},"arguments":[]}]},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"TerminalRule","name":"PIE_SECTION_LABEL","definition":{"$type":"RegexToken","regex":"/\\"[^\\"]+\\"/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"PIE_SECTION_VALUE","type":{"$type":"ReturnType","name":"number"},"definition":{"$type":"RegexToken","regex":"/(0|[1-9][0-9]*)(\\\\.[0-9]+)?/"},"fragment":false,"hidden":false},{"$type":"ParserRule","fragment":true,"name":"TitleAndAccessibilities","definition":{"$type":"Group","elements":[{"$type":"Alternatives","elements":[{"$type":"Assignment","feature":"accDescr","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@7"},"arguments":[]}},{"$type":"Assignment","feature":"accTitle","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@8"},"arguments":[]}},{"$type":"Assignment","feature":"title","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@9"},"arguments":[]}}]},{"$type":"RuleCall","rule":{"$ref":"#/rules@5"},"arguments":[]}],"cardinality":"+"},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","fragment":true,"name":"EOL","dataType":"string","definition":{"$type":"Alternatives","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@6"},"arguments":[],"cardinality":"+"},{"$type":"EndOfFile"}]},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"TerminalRule","name":"NEWLINE","definition":{"$type":"RegexToken","regex":"/\\\\r?\\\\n/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ACC_DESCR","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*accDescr(?:[\\\\t ]*:([^\\\\n\\\\r]*?(?=%%)|[^\\\\n\\\\r]*)|\\\\s*{([^}]*)})/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ACC_TITLE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*accTitle[\\\\t ]*:(?:[^\\\\n\\\\r]*?(?=%%)|[^\\\\n\\\\r]*)/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"TITLE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*title(?:[\\\\t ][^\\\\n\\\\r]*?(?=%%)|[\\\\t ][^\\\\n\\\\r]*|)/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","hidden":true,"name":"WHITESPACE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]+/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"YAML","definition":{"$type":"RegexToken","regex":"/---[\\\\t ]*\\\\r?\\\\n(?:[\\\\S\\\\s]*?\\\\r?\\\\n)?---(?:\\\\r?\\\\n|(?!\\\\S))/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"DIRECTIVE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*%%{[\\\\S\\\\s]*?}%%(?:\\\\r?\\\\n|(?!\\\\S))/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"SINGLE_LINE_COMMENT","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*%%[^\\\\n\\\\r]*/"},"fragment":false}],"definesHiddenTokens":false,"hiddenTokens":[],"interfaces":[{"$type":"Interface","name":"Common","attributes":[{"$type":"TypeAttribute","name":"accDescr","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}},{"$type":"TypeAttribute","name":"accTitle","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}},{"$type":"TypeAttribute","name":"title","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}}],"superTypes":[]}],"types":[],"usedGrammars":[]}')),"PieGrammar"),$Fe=Lt(()=>Tce??(Tce=Hu('{"$type":"Grammar","isDeclared":true,"name":"Architecture","imports":[],"rules":[{"$type":"ParserRule","entry":true,"name":"Architecture","definition":{"$type":"Group","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@18"},"arguments":[],"cardinality":"*"},{"$type":"Keyword","value":"architecture-beta"},{"$type":"Alternatives","elements":[{"$type":"Group","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@18"},"arguments":[],"cardinality":"*"},{"$type":"RuleCall","rule":{"$ref":"#/rules@16"},"arguments":[]}]},{"$type":"Group","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@18"},"arguments":[],"cardinality":"*"},{"$type":"RuleCall","rule":{"$ref":"#/rules@1"},"arguments":[],"cardinality":"*"}]},{"$type":"RuleCall","rule":{"$ref":"#/rules@18"},"arguments":[],"cardinality":"*"}]}]},"definesHiddenTokens":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","fragment":true,"name":"Statement","definition":{"$type":"Alternatives","elements":[{"$type":"Assignment","feature":"groups","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@5"},"arguments":[]}},{"$type":"Assignment","feature":"services","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@6"},"arguments":[]}},{"$type":"Assignment","feature":"junctions","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@7"},"arguments":[]}},{"$type":"Assignment","feature":"edges","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@8"},"arguments":[]}}]},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","fragment":true,"name":"LeftPort","definition":{"$type":"Group","elements":[{"$type":"Keyword","value":":"},{"$type":"Assignment","feature":"lhsDir","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@9"},"arguments":[]}}]},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","fragment":true,"name":"RightPort","definition":{"$type":"Group","elements":[{"$type":"Assignment","feature":"rhsDir","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@9"},"arguments":[]}},{"$type":"Keyword","value":":"}]},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","fragment":true,"name":"Arrow","definition":{"$type":"Group","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[]},{"$type":"Assignment","feature":"lhsInto","operator":"?=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@15"},"arguments":[]},"cardinality":"?"},{"$type":"Alternatives","elements":[{"$type":"Keyword","value":"--"},{"$type":"Group","elements":[{"$type":"Keyword","value":"-"},{"$type":"Assignment","feature":"title","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@13"},"arguments":[]}},{"$type":"Keyword","value":"-"}]}]},{"$type":"Assignment","feature":"rhsInto","operator":"?=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@15"},"arguments":[]},"cardinality":"?"},{"$type":"RuleCall","rule":{"$ref":"#/rules@3"},"arguments":[]}]},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"Group","definition":{"$type":"Group","elements":[{"$type":"Keyword","value":"group"},{"$type":"Assignment","feature":"id","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@10"},"arguments":[]}},{"$type":"Assignment","feature":"icon","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@12"},"arguments":[]},"cardinality":"?"},{"$type":"Assignment","feature":"title","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@13"},"arguments":[]},"cardinality":"?"},{"$type":"Group","elements":[{"$type":"Keyword","value":"in"},{"$type":"Assignment","feature":"in","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@10"},"arguments":[]}}],"cardinality":"?"},{"$type":"RuleCall","rule":{"$ref":"#/rules@17"},"arguments":[]}]},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"Service","definition":{"$type":"Group","elements":[{"$type":"Keyword","value":"service"},{"$type":"Assignment","feature":"id","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@10"},"arguments":[]}},{"$type":"Alternatives","elements":[{"$type":"Assignment","feature":"iconText","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@11"},"arguments":[]}},{"$type":"Assignment","feature":"icon","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@12"},"arguments":[]}}],"cardinality":"?"},{"$type":"Assignment","feature":"title","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@13"},"arguments":[]},"cardinality":"?"},{"$type":"Group","elements":[{"$type":"Keyword","value":"in"},{"$type":"Assignment","feature":"in","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@10"},"arguments":[]}}],"cardinality":"?"},{"$type":"RuleCall","rule":{"$ref":"#/rules@17"},"arguments":[]}]},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"Junction","definition":{"$type":"Group","elements":[{"$type":"Keyword","value":"junction"},{"$type":"Assignment","feature":"id","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@10"},"arguments":[]}},{"$type":"Group","elements":[{"$type":"Keyword","value":"in"},{"$type":"Assignment","feature":"in","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@10"},"arguments":[]}}],"cardinality":"?"},{"$type":"RuleCall","rule":{"$ref":"#/rules@17"},"arguments":[]}]},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"Edge","definition":{"$type":"Group","elements":[{"$type":"Assignment","feature":"lhsId","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@10"},"arguments":[]}},{"$type":"Assignment","feature":"lhsGroup","operator":"?=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@14"},"arguments":[]},"cardinality":"?"},{"$type":"RuleCall","rule":{"$ref":"#/rules@4"},"arguments":[]},{"$type":"Assignment","feature":"rhsId","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@10"},"arguments":[]}},{"$type":"Assignment","feature":"rhsGroup","operator":"?=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@14"},"arguments":[]},"cardinality":"?"},{"$type":"RuleCall","rule":{"$ref":"#/rules@17"},"arguments":[]}]},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"TerminalRule","name":"ARROW_DIRECTION","definition":{"$type":"TerminalAlternatives","elements":[{"$type":"TerminalAlternatives","elements":[{"$type":"TerminalAlternatives","elements":[{"$type":"CharacterRange","left":{"$type":"Keyword","value":"L"}},{"$type":"CharacterRange","left":{"$type":"Keyword","value":"R"}}]},{"$type":"CharacterRange","left":{"$type":"Keyword","value":"T"}}]},{"$type":"CharacterRange","left":{"$type":"Keyword","value":"B"}}]},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ARCH_ID","definition":{"$type":"RegexToken","regex":"/[\\\\w]+/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ARCH_TEXT_ICON","definition":{"$type":"RegexToken","regex":"/\\\\(\\"[^\\"]+\\"\\\\)/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ARCH_ICON","definition":{"$type":"RegexToken","regex":"/\\\\([\\\\w-:]+\\\\)/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ARCH_TITLE","definition":{"$type":"RegexToken","regex":"/\\\\[[\\\\w ]+\\\\]/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ARROW_GROUP","definition":{"$type":"RegexToken","regex":"/\\\\{group\\\\}/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ARROW_INTO","definition":{"$type":"RegexToken","regex":"/<|>/"},"fragment":false,"hidden":false},{"$type":"ParserRule","fragment":true,"name":"TitleAndAccessibilities","definition":{"$type":"Group","elements":[{"$type":"Alternatives","elements":[{"$type":"Assignment","feature":"accDescr","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@19"},"arguments":[]}},{"$type":"Assignment","feature":"accTitle","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@20"},"arguments":[]}},{"$type":"Assignment","feature":"title","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@21"},"arguments":[]}}]},{"$type":"RuleCall","rule":{"$ref":"#/rules@17"},"arguments":[]}],"cardinality":"+"},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","fragment":true,"name":"EOL","dataType":"string","definition":{"$type":"Alternatives","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@18"},"arguments":[],"cardinality":"+"},{"$type":"EndOfFile"}]},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"TerminalRule","name":"NEWLINE","definition":{"$type":"RegexToken","regex":"/\\\\r?\\\\n/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ACC_DESCR","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*accDescr(?:[\\\\t ]*:([^\\\\n\\\\r]*?(?=%%)|[^\\\\n\\\\r]*)|\\\\s*{([^}]*)})/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ACC_TITLE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*accTitle[\\\\t ]*:(?:[^\\\\n\\\\r]*?(?=%%)|[^\\\\n\\\\r]*)/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"TITLE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*title(?:[\\\\t ][^\\\\n\\\\r]*?(?=%%)|[\\\\t ][^\\\\n\\\\r]*|)/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","hidden":true,"name":"WHITESPACE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]+/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"YAML","definition":{"$type":"RegexToken","regex":"/---[\\\\t ]*\\\\r?\\\\n(?:[\\\\S\\\\s]*?\\\\r?\\\\n)?---(?:\\\\r?\\\\n|(?!\\\\S))/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"DIRECTIVE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*%%{[\\\\S\\\\s]*?}%%(?:\\\\r?\\\\n|(?!\\\\S))/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"SINGLE_LINE_COMMENT","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*%%[^\\\\n\\\\r]*/"},"fragment":false}],"definesHiddenTokens":false,"hiddenTokens":[],"interfaces":[{"$type":"Interface","name":"Common","attributes":[{"$type":"TypeAttribute","name":"accDescr","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}},{"$type":"TypeAttribute","name":"accTitle","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}},{"$type":"TypeAttribute","name":"title","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}}],"superTypes":[]}],"types":[],"usedGrammars":[]}')),"ArchitectureGrammar"),zFe=Lt(()=>kce??(kce=Hu(`{"$type":"Grammar","isDeclared":true,"name":"GitGraph","interfaces":[{"$type":"Interface","name":"Common","attributes":[{"$type":"TypeAttribute","name":"accDescr","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}},{"$type":"TypeAttribute","name":"accTitle","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}},{"$type":"TypeAttribute","name":"title","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}}],"superTypes":[]}],"rules":[{"$type":"ParserRule","fragment":true,"name":"TitleAndAccessibilities","definition":{"$type":"Group","elements":[{"$type":"Alternatives","elements":[{"$type":"Assignment","feature":"accDescr","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@3"},"arguments":[]}},{"$type":"Assignment","feature":"accTitle","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@4"},"arguments":[]}},{"$type":"Assignment","feature":"title","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@5"},"arguments":[]}}]},{"$type":"RuleCall","rule":{"$ref":"#/rules@1"},"arguments":[]}],"cardinality":"+"},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","fragment":true,"name":"EOL","dataType":"string","definition":{"$type":"Alternatives","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[],"cardinality":"+"},{"$type":"EndOfFile"}]},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"TerminalRule","name":"NEWLINE","definition":{"$type":"RegexToken","regex":"/\\\\r?\\\\n/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ACC_DESCR","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*accDescr(?:[\\\\t ]*:([^\\\\n\\\\r]*?(?=%%)|[^\\\\n\\\\r]*)|\\\\s*{([^}]*)})/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ACC_TITLE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*accTitle[\\\\t ]*:(?:[^\\\\n\\\\r]*?(?=%%)|[^\\\\n\\\\r]*)/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"TITLE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*title(?:[\\\\t ][^\\\\n\\\\r]*?(?=%%)|[\\\\t ][^\\\\n\\\\r]*|)/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","hidden":true,"name":"WHITESPACE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]+/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"YAML","definition":{"$type":"RegexToken","regex":"/---[\\\\t ]*\\\\r?\\\\n(?:[\\\\S\\\\s]*?\\\\r?\\\\n)?---(?:\\\\r?\\\\n|(?!\\\\S))/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"DIRECTIVE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*%%{[\\\\S\\\\s]*?}%%(?:\\\\r?\\\\n|(?!\\\\S))/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"SINGLE_LINE_COMMENT","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*%%[^\\\\n\\\\r]*/"},"fragment":false},{"$type":"ParserRule","entry":true,"name":"GitGraph","definition":{"$type":"Group","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[],"cardinality":"*"},{"$type":"Alternatives","elements":[{"$type":"Keyword","value":"gitGraph"},{"$type":"Group","elements":[{"$type":"Keyword","value":"gitGraph"},{"$type":"Keyword","value":":"}]},{"$type":"Keyword","value":"gitGraph:"},{"$type":"Group","elements":[{"$type":"Keyword","value":"gitGraph"},{"$type":"RuleCall","rule":{"$ref":"#/rules@12"},"arguments":[]},{"$type":"Keyword","value":":"}]}]},{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[],"cardinality":"*"},{"$type":"Group","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[],"cardinality":"*"},{"$type":"Alternatives","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@0"},"arguments":[]},{"$type":"Assignment","feature":"statements","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@11"},"arguments":[]}},{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[]}],"cardinality":"*"}]}]},"definesHiddenTokens":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"Statement","definition":{"$type":"Alternatives","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@13"},"arguments":[]},{"$type":"RuleCall","rule":{"$ref":"#/rules@14"},"arguments":[]},{"$type":"RuleCall","rule":{"$ref":"#/rules@15"},"arguments":[]},{"$type":"RuleCall","rule":{"$ref":"#/rules@16"},"arguments":[]},{"$type":"RuleCall","rule":{"$ref":"#/rules@17"},"arguments":[]}]},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"Direction","definition":{"$type":"Assignment","feature":"dir","operator":"=","terminal":{"$type":"Alternatives","elements":[{"$type":"Keyword","value":"LR"},{"$type":"Keyword","value":"TB"},{"$type":"Keyword","value":"BT"}]}},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"Commit","definition":{"$type":"Group","elements":[{"$type":"Keyword","value":"commit"},{"$type":"Alternatives","elements":[{"$type":"Group","elements":[{"$type":"Keyword","value":"id:"},{"$type":"Assignment","feature":"id","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@20"},"arguments":[]}}]},{"$type":"Group","elements":[{"$type":"Keyword","value":"msg:","cardinality":"?"},{"$type":"Assignment","feature":"message","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@20"},"arguments":[]}}]},{"$type":"Group","elements":[{"$type":"Keyword","value":"tag:"},{"$type":"Assignment","feature":"tags","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@20"},"arguments":[]}}]},{"$type":"Group","elements":[{"$type":"Keyword","value":"type:"},{"$type":"Assignment","feature":"type","operator":"=","terminal":{"$type":"Alternatives","elements":[{"$type":"Keyword","value":"NORMAL"},{"$type":"Keyword","value":"REVERSE"},{"$type":"Keyword","value":"HIGHLIGHT"}]}}]}],"cardinality":"*"},{"$type":"RuleCall","rule":{"$ref":"#/rules@1"},"arguments":[]}]},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"Branch","definition":{"$type":"Group","elements":[{"$type":"Keyword","value":"branch"},{"$type":"Assignment","feature":"name","operator":"=","terminal":{"$type":"Alternatives","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@19"},"arguments":[]},{"$type":"RuleCall","rule":{"$ref":"#/rules@20"},"arguments":[]}]}},{"$type":"Group","elements":[{"$type":"Keyword","value":"order:"},{"$type":"Assignment","feature":"order","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@18"},"arguments":[]}}],"cardinality":"?"},{"$type":"RuleCall","rule":{"$ref":"#/rules@1"},"arguments":[]}]},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"Merge","definition":{"$type":"Group","elements":[{"$type":"Keyword","value":"merge"},{"$type":"Assignment","feature":"branch","operator":"=","terminal":{"$type":"Alternatives","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@19"},"arguments":[]},{"$type":"RuleCall","rule":{"$ref":"#/rules@20"},"arguments":[]}]}},{"$type":"Alternatives","elements":[{"$type":"Group","elements":[{"$type":"Keyword","value":"id:"},{"$type":"Assignment","feature":"id","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@20"},"arguments":[]}}]},{"$type":"Group","elements":[{"$type":"Keyword","value":"tag:"},{"$type":"Assignment","feature":"tags","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@20"},"arguments":[]}}]},{"$type":"Group","elements":[{"$type":"Keyword","value":"type:"},{"$type":"Assignment","feature":"type","operator":"=","terminal":{"$type":"Alternatives","elements":[{"$type":"Keyword","value":"NORMAL"},{"$type":"Keyword","value":"REVERSE"},{"$type":"Keyword","value":"HIGHLIGHT"}]}}]}],"cardinality":"*"},{"$type":"RuleCall","rule":{"$ref":"#/rules@1"},"arguments":[]}]},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"Checkout","definition":{"$type":"Group","elements":[{"$type":"Alternatives","elements":[{"$type":"Keyword","value":"checkout"},{"$type":"Keyword","value":"switch"}]},{"$type":"Assignment","feature":"branch","operator":"=","terminal":{"$type":"Alternatives","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@19"},"arguments":[]},{"$type":"RuleCall","rule":{"$ref":"#/rules@20"},"arguments":[]}]}},{"$type":"RuleCall","rule":{"$ref":"#/rules@1"},"arguments":[]}]},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"CherryPicking","definition":{"$type":"Group","elements":[{"$type":"Keyword","value":"cherry-pick"},{"$type":"Alternatives","elements":[{"$type":"Group","elements":[{"$type":"Keyword","value":"id:"},{"$type":"Assignment","feature":"id","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@20"},"arguments":[]}}]},{"$type":"Group","elements":[{"$type":"Keyword","value":"tag:"},{"$type":"Assignment","feature":"tags","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@20"},"arguments":[]}}]},{"$type":"Group","elements":[{"$type":"Keyword","value":"parent:"},{"$type":"Assignment","feature":"parent","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@20"},"arguments":[]}}]}],"cardinality":"*"},{"$type":"RuleCall","rule":{"$ref":"#/rules@1"},"arguments":[]}]},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"TerminalRule","name":"INT","type":{"$type":"ReturnType","name":"number"},"definition":{"$type":"RegexToken","regex":"/[0-9]+(?=\\\\s)/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ID","type":{"$type":"ReturnType","name":"string"},"definition":{"$type":"RegexToken","regex":"/\\\\w([-\\\\./\\\\w]*[-\\\\w])?/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"STRING","definition":{"$type":"RegexToken","regex":"/\\"[^\\"]*\\"|'[^']*'/"},"fragment":false,"hidden":false}],"definesHiddenTokens":false,"hiddenTokens":[],"imports":[],"types":[],"usedGrammars":[]}`)),"GitGraphGrammar"),GFe=Lt(()=>Ece??(Ece=Hu(`{"$type":"Grammar","isDeclared":true,"name":"Radar","interfaces":[{"$type":"Interface","name":"Common","attributes":[{"$type":"TypeAttribute","name":"accDescr","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}},{"$type":"TypeAttribute","name":"accTitle","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}},{"$type":"TypeAttribute","name":"title","isOptional":true,"type":{"$type":"SimpleType","primitiveType":"string"}}],"superTypes":[]},{"$type":"Interface","name":"Entry","attributes":[{"$type":"TypeAttribute","name":"axis","isOptional":true,"type":{"$type":"ReferenceType","referenceType":{"$type":"SimpleType","typeRef":{"$ref":"#/rules@12"}}}},{"$type":"TypeAttribute","name":"value","type":{"$type":"SimpleType","primitiveType":"number"},"isOptional":false}],"superTypes":[]}],"rules":[{"$type":"ParserRule","fragment":true,"name":"TitleAndAccessibilities","definition":{"$type":"Group","elements":[{"$type":"Alternatives","elements":[{"$type":"Assignment","feature":"accDescr","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@3"},"arguments":[]}},{"$type":"Assignment","feature":"accTitle","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@4"},"arguments":[]}},{"$type":"Assignment","feature":"title","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@5"},"arguments":[]}}]},{"$type":"RuleCall","rule":{"$ref":"#/rules@1"},"arguments":[]}],"cardinality":"+"},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","fragment":true,"name":"EOL","dataType":"string","definition":{"$type":"Alternatives","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[],"cardinality":"+"},{"$type":"EndOfFile"}]},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"TerminalRule","name":"NEWLINE","definition":{"$type":"RegexToken","regex":"/\\\\r?\\\\n/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ACC_DESCR","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*accDescr(?:[\\\\t ]*:([^\\\\n\\\\r]*?(?=%%)|[^\\\\n\\\\r]*)|\\\\s*{([^}]*)})/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ACC_TITLE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*accTitle[\\\\t ]*:(?:[^\\\\n\\\\r]*?(?=%%)|[^\\\\n\\\\r]*)/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"TITLE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*title(?:[\\\\t ][^\\\\n\\\\r]*?(?=%%)|[\\\\t ][^\\\\n\\\\r]*|)/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","hidden":true,"name":"WHITESPACE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]+/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"YAML","definition":{"$type":"RegexToken","regex":"/---[\\\\t ]*\\\\r?\\\\n(?:[\\\\S\\\\s]*?\\\\r?\\\\n)?---(?:\\\\r?\\\\n|(?!\\\\S))/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"DIRECTIVE","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*%%{[\\\\S\\\\s]*?}%%(?:\\\\r?\\\\n|(?!\\\\S))/"},"fragment":false},{"$type":"TerminalRule","hidden":true,"name":"SINGLE_LINE_COMMENT","definition":{"$type":"RegexToken","regex":"/[\\\\t ]*%%[^\\\\n\\\\r]*/"},"fragment":false},{"$type":"ParserRule","entry":true,"name":"Radar","definition":{"$type":"Group","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[],"cardinality":"*"},{"$type":"Alternatives","elements":[{"$type":"Keyword","value":"radar-beta"},{"$type":"Keyword","value":"radar-beta:"},{"$type":"Group","elements":[{"$type":"Keyword","value":"radar-beta"},{"$type":"Keyword","value":":"}]}]},{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[],"cardinality":"*"},{"$type":"Alternatives","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@0"},"arguments":[]},{"$type":"Group","elements":[{"$type":"Keyword","value":"axis"},{"$type":"Assignment","feature":"axes","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@12"},"arguments":[]}},{"$type":"Group","elements":[{"$type":"Keyword","value":","},{"$type":"Assignment","feature":"axes","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@12"},"arguments":[]}}],"cardinality":"*"}]},{"$type":"Group","elements":[{"$type":"Keyword","value":"curve"},{"$type":"Assignment","feature":"curves","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@13"},"arguments":[]}},{"$type":"Group","elements":[{"$type":"Keyword","value":","},{"$type":"Assignment","feature":"curves","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@13"},"arguments":[]}}],"cardinality":"*"}]},{"$type":"Group","elements":[{"$type":"Assignment","feature":"options","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@17"},"arguments":[]}},{"$type":"Group","elements":[{"$type":"Keyword","value":","},{"$type":"Assignment","feature":"options","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@17"},"arguments":[]}}],"cardinality":"*"}]},{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[]}],"cardinality":"*"}]},"definesHiddenTokens":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","fragment":true,"name":"Label","definition":{"$type":"Group","elements":[{"$type":"Keyword","value":"["},{"$type":"Assignment","feature":"label","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@22"},"arguments":[]}},{"$type":"Keyword","value":"]"}]},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"Axis","definition":{"$type":"Group","elements":[{"$type":"Assignment","feature":"name","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@21"},"arguments":[]}},{"$type":"RuleCall","rule":{"$ref":"#/rules@11"},"arguments":[],"cardinality":"?"}]},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"Curve","definition":{"$type":"Group","elements":[{"$type":"Assignment","feature":"name","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@21"},"arguments":[]}},{"$type":"RuleCall","rule":{"$ref":"#/rules@11"},"arguments":[],"cardinality":"?"},{"$type":"Keyword","value":"{"},{"$type":"RuleCall","rule":{"$ref":"#/rules@14"},"arguments":[]},{"$type":"Keyword","value":"}"}]},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","fragment":true,"name":"Entries","definition":{"$type":"Alternatives","elements":[{"$type":"Group","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[],"cardinality":"*"},{"$type":"Assignment","feature":"entries","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@16"},"arguments":[]}},{"$type":"Group","elements":[{"$type":"Keyword","value":","},{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[],"cardinality":"*"},{"$type":"Assignment","feature":"entries","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@16"},"arguments":[]}}],"cardinality":"*"},{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[],"cardinality":"*"}]},{"$type":"Group","elements":[{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[],"cardinality":"*"},{"$type":"Assignment","feature":"entries","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@15"},"arguments":[]}},{"$type":"Group","elements":[{"$type":"Keyword","value":","},{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[],"cardinality":"*"},{"$type":"Assignment","feature":"entries","operator":"+=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@15"},"arguments":[]}}],"cardinality":"*"},{"$type":"RuleCall","rule":{"$ref":"#/rules@2"},"arguments":[],"cardinality":"*"}]}]},"definesHiddenTokens":false,"entry":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"DetailedEntry","returnType":{"$ref":"#/interfaces@1"},"definition":{"$type":"Group","elements":[{"$type":"Assignment","feature":"axis","operator":"=","terminal":{"$type":"CrossReference","type":{"$ref":"#/rules@12"},"terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@21"},"arguments":[]},"deprecatedSyntax":false}},{"$type":"Keyword","value":":","cardinality":"?"},{"$type":"Assignment","feature":"value","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@18"},"arguments":[]}}]},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"NumberEntry","returnType":{"$ref":"#/interfaces@1"},"definition":{"$type":"Assignment","feature":"value","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@18"},"arguments":[]}},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"ParserRule","name":"Option","definition":{"$type":"Alternatives","elements":[{"$type":"Group","elements":[{"$type":"Assignment","feature":"name","operator":"=","terminal":{"$type":"Keyword","value":"showLegend"}},{"$type":"Assignment","feature":"value","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@19"},"arguments":[]}}]},{"$type":"Group","elements":[{"$type":"Assignment","feature":"name","operator":"=","terminal":{"$type":"Keyword","value":"ticks"}},{"$type":"Assignment","feature":"value","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@18"},"arguments":[]}}]},{"$type":"Group","elements":[{"$type":"Assignment","feature":"name","operator":"=","terminal":{"$type":"Keyword","value":"max"}},{"$type":"Assignment","feature":"value","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@18"},"arguments":[]}}]},{"$type":"Group","elements":[{"$type":"Assignment","feature":"name","operator":"=","terminal":{"$type":"Keyword","value":"min"}},{"$type":"Assignment","feature":"value","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@18"},"arguments":[]}}]},{"$type":"Group","elements":[{"$type":"Assignment","feature":"name","operator":"=","terminal":{"$type":"Keyword","value":"graticule"}},{"$type":"Assignment","feature":"value","operator":"=","terminal":{"$type":"RuleCall","rule":{"$ref":"#/rules@20"},"arguments":[]}}]}]},"definesHiddenTokens":false,"entry":false,"fragment":false,"hiddenTokens":[],"parameters":[],"wildcard":false},{"$type":"TerminalRule","name":"NUMBER","type":{"$type":"ReturnType","name":"number"},"definition":{"$type":"RegexToken","regex":"/(0|[1-9][0-9]*)(\\\\.[0-9]+)?/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"BOOLEAN","type":{"$type":"ReturnType","name":"boolean"},"definition":{"$type":"TerminalAlternatives","elements":[{"$type":"CharacterRange","left":{"$type":"Keyword","value":"true"}},{"$type":"CharacterRange","left":{"$type":"Keyword","value":"false"}}]},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"GRATICULE","type":{"$type":"ReturnType","name":"string"},"definition":{"$type":"TerminalAlternatives","elements":[{"$type":"CharacterRange","left":{"$type":"Keyword","value":"circle"}},{"$type":"CharacterRange","left":{"$type":"Keyword","value":"polygon"}}]},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"ID","type":{"$type":"ReturnType","name":"string"},"definition":{"$type":"RegexToken","regex":"/[a-zA-Z_][a-zA-Z0-9\\\\-_]*/"},"fragment":false,"hidden":false},{"$type":"TerminalRule","name":"STRING","definition":{"$type":"RegexToken","regex":"/\\"[^\\"]*\\"|'[^']*'/"},"fragment":false,"hidden":false}],"definesHiddenTokens":false,"hiddenTokens":[],"imports":[],"types":[],"usedGrammars":[]}`)),"RadarGrammar"),VFe={languageId:"info",fileExtensions:[".mmd",".mermaid"],caseInsensitive:!1,mode:"production"},UFe={languageId:"packet",fileExtensions:[".mmd",".mermaid"],caseInsensitive:!1,mode:"production"},HFe={languageId:"pie",fileExtensions:[".mmd",".mermaid"],caseInsensitive:!1,mode:"production"},WFe={languageId:"architecture",fileExtensions:[".mmd",".mermaid"],caseInsensitive:!1,mode:"production"},qFe={languageId:"gitGraph",fileExtensions:[".mmd",".mermaid"],caseInsensitive:!1,mode:"production"},YFe={languageId:"radar",fileExtensions:[".mmd",".mermaid"],caseInsensitive:!1,mode:"production"},co={AstReflection:Lt(()=>new Pce,"AstReflection")},_I={Grammar:Lt(()=>PFe(),"Grammar"),LanguageMetaData:Lt(()=>VFe,"LanguageMetaData"),parser:{}},DI={Grammar:Lt(()=>BFe(),"Grammar"),LanguageMetaData:Lt(()=>UFe,"LanguageMetaData"),parser:{}},LI={Grammar:Lt(()=>FFe(),"Grammar"),LanguageMetaData:Lt(()=>HFe,"LanguageMetaData"),parser:{}},RI={Grammar:Lt(()=>$Fe(),"Grammar"),LanguageMetaData:Lt(()=>WFe,"LanguageMetaData"),parser:{}},NI={Grammar:Lt(()=>zFe(),"Grammar"),LanguageMetaData:Lt(()=>qFe,"LanguageMetaData"),parser:{}},MI={Grammar:Lt(()=>GFe(),"Grammar"),LanguageMetaData:Lt(()=>YFe,"LanguageMetaData"),parser:{}},XFe=/accDescr(?:[\t ]*:([^\n\r]*)|\s*{([^}]*)})/,jFe=/accTitle[\t ]*:([^\n\r]*)/,KFe=/title([\t ][^\n\r]*|)/,QFe={ACC_DESCR:XFe,ACC_TITLE:jFe,TITLE:KFe},w1=class extends yp{static{o(this,"AbstractMermaidValueConverter")}static{Lt(this,"AbstractMermaidValueConverter")}runConverter(t,e,r){let n=this.runCommonConverter(t,e,r);return n===void 0&&(n=this.runCustomConverter(t,e,r)),n===void 0?super.runConverter(t,e,r):n}runCommonConverter(t,e,r){let n=QFe[t.name];if(n===void 0)return;let i=n.exec(e);if(i!==null){if(i[1]!==void 0)return i[1].trim().replace(/[\t ]{2,}/gm," ");if(i[2]!==void 0)return i[2].replace(/^\s*/gm,"").replace(/\s+$/gm,"").replace(/[\t ]{2,}/gm," ").replace(/[\n\r]{2,}/gm,` +`)}}},Wu=class extends w1{static{o(this,"CommonValueConverter")}static{Lt(this,"CommonValueConverter")}runCustomConverter(t,e,r){}},$s=class extends Uu{static{o(this,"AbstractMermaidTokenBuilder")}static{Lt(this,"AbstractMermaidTokenBuilder")}constructor(t){super(),this.keywords=new Set(t)}buildKeywordTokens(t,e,r){let n=super.buildKeywordTokens(t,e,r);return n.forEach(i=>{this.keywords.has(i.name)&&i.PATTERN!==void 0&&(i.PATTERN=new RegExp(i.PATTERN.toString()+"(?:(?=%%)|(?!\\S))"))}),n}},ZFe=class extends $s{static{o(this,"CommonTokenBuilder")}static{Lt(this,"CommonTokenBuilder")}}});function XE(t=ps){let e=ui(ds(t),co),r=ui(fs({shared:e}),NI,YE);return e.ServiceRegistry.register(r),{shared:e,GitGraph:r}}var JFe,YE,II=N(()=>{"use strict";zs();Xo();JFe=class extends $s{static{o(this,"GitGraphTokenBuilder")}static{Lt(this,"GitGraphTokenBuilder")}constructor(){super(["gitGraph"])}},YE={parser:{TokenBuilder:Lt(()=>new JFe,"TokenBuilder"),ValueConverter:Lt(()=>new Wu,"ValueConverter")}};o(XE,"createGitGraphServices");Lt(XE,"createGitGraphServices")});function KE(t=ps){let e=ui(ds(t),co),r=ui(fs({shared:e}),_I,jE);return e.ServiceRegistry.register(r),{shared:e,Info:r}}var e$e,jE,OI=N(()=>{"use strict";zs();Xo();e$e=class extends $s{static{o(this,"InfoTokenBuilder")}static{Lt(this,"InfoTokenBuilder")}constructor(){super(["info","showInfo"])}},jE={parser:{TokenBuilder:Lt(()=>new e$e,"TokenBuilder"),ValueConverter:Lt(()=>new Wu,"ValueConverter")}};o(KE,"createInfoServices");Lt(KE,"createInfoServices")});function ZE(t=ps){let e=ui(ds(t),co),r=ui(fs({shared:e}),DI,QE);return e.ServiceRegistry.register(r),{shared:e,Packet:r}}var t$e,QE,PI=N(()=>{"use strict";zs();Xo();t$e=class extends $s{static{o(this,"PacketTokenBuilder")}static{Lt(this,"PacketTokenBuilder")}constructor(){super(["packet-beta"])}},QE={parser:{TokenBuilder:Lt(()=>new t$e,"TokenBuilder"),ValueConverter:Lt(()=>new Wu,"ValueConverter")}};o(ZE,"createPacketServices");Lt(ZE,"createPacketServices")});function e6(t=ps){let e=ui(ds(t),co),r=ui(fs({shared:e}),LI,JE);return e.ServiceRegistry.register(r),{shared:e,Pie:r}}var r$e,n$e,JE,BI=N(()=>{"use strict";zs();Xo();r$e=class extends $s{static{o(this,"PieTokenBuilder")}static{Lt(this,"PieTokenBuilder")}constructor(){super(["pie","showData"])}},n$e=class extends w1{static{o(this,"PieValueConverter")}static{Lt(this,"PieValueConverter")}runCustomConverter(t,e,r){if(t.name==="PIE_SECTION_LABEL")return e.replace(/"/g,"").trim()}},JE={parser:{TokenBuilder:Lt(()=>new r$e,"TokenBuilder"),ValueConverter:Lt(()=>new n$e,"ValueConverter")}};o(e6,"createPieServices");Lt(e6,"createPieServices")});function r6(t=ps){let e=ui(ds(t),co),r=ui(fs({shared:e}),RI,t6);return e.ServiceRegistry.register(r),{shared:e,Architecture:r}}var i$e,a$e,t6,FI=N(()=>{"use strict";zs();Xo();i$e=class extends $s{static{o(this,"ArchitectureTokenBuilder")}static{Lt(this,"ArchitectureTokenBuilder")}constructor(){super(["architecture"])}},a$e=class extends w1{static{o(this,"ArchitectureValueConverter")}static{Lt(this,"ArchitectureValueConverter")}runCustomConverter(t,e,r){if(t.name==="ARCH_ICON")return e.replace(/[()]/g,"").trim();if(t.name==="ARCH_TEXT_ICON")return e.replace(/["()]/g,"");if(t.name==="ARCH_TITLE")return e.replace(/[[\]]/g,"").trim()}},t6={parser:{TokenBuilder:Lt(()=>new i$e,"TokenBuilder"),ValueConverter:Lt(()=>new a$e,"ValueConverter")}};o(r6,"createArchitectureServices");Lt(r6,"createArchitectureServices")});function i6(t=ps){let e=ui(ds(t),co),r=ui(fs({shared:e}),MI,n6);return e.ServiceRegistry.register(r),{shared:e,Radar:r}}var s$e,n6,$I=N(()=>{"use strict";zs();Xo();s$e=class extends $s{static{o(this,"RadarTokenBuilder")}static{Lt(this,"RadarTokenBuilder")}constructor(){super(["radar-beta"])}},n6={parser:{TokenBuilder:Lt(()=>new s$e,"TokenBuilder"),ValueConverter:Lt(()=>new Wu,"ValueConverter")}};o(i6,"createRadarServices");Lt(i6,"createRadarServices")});var Bce={};hr(Bce,{InfoModule:()=>jE,createInfoServices:()=>KE});var Fce=N(()=>{"use strict";OI();zs()});var $ce={};hr($ce,{PacketModule:()=>QE,createPacketServices:()=>ZE});var zce=N(()=>{"use strict";PI();zs()});var Gce={};hr(Gce,{PieModule:()=>JE,createPieServices:()=>e6});var Vce=N(()=>{"use strict";BI();zs()});var Uce={};hr(Uce,{ArchitectureModule:()=>t6,createArchitectureServices:()=>r6});var Hce=N(()=>{"use strict";FI();zs()});var Wce={};hr(Wce,{GitGraphModule:()=>YE,createGitGraphServices:()=>XE});var qce=N(()=>{"use strict";II();zs()});var Yce={};hr(Yce,{RadarModule:()=>n6,createRadarServices:()=>i6});var Xce=N(()=>{"use strict";$I();zs()});async function uo(t,e){let r=o$e[t];if(!r)throw new Error(`Unknown diagram type: ${t}`);df[t]||await r();let i=df[t].parse(e);if(i.lexerErrors.length>0||i.parserErrors.length>0)throw new l$e(i);return i.value}var df,o$e,l$e,kp=N(()=>{"use strict";II();OI();PI();BI();FI();$I();zs();df={},o$e={info:Lt(async()=>{let{createInfoServices:t}=await Promise.resolve().then(()=>(Fce(),Bce)),e=t().Info.parser.LangiumParser;df.info=e},"info"),packet:Lt(async()=>{let{createPacketServices:t}=await Promise.resolve().then(()=>(zce(),$ce)),e=t().Packet.parser.LangiumParser;df.packet=e},"packet"),pie:Lt(async()=>{let{createPieServices:t}=await Promise.resolve().then(()=>(Vce(),Gce)),e=t().Pie.parser.LangiumParser;df.pie=e},"pie"),architecture:Lt(async()=>{let{createArchitectureServices:t}=await Promise.resolve().then(()=>(Hce(),Uce)),e=t().Architecture.parser.LangiumParser;df.architecture=e},"architecture"),gitGraph:Lt(async()=>{let{createGitGraphServices:t}=await Promise.resolve().then(()=>(qce(),Wce)),e=t().GitGraph.parser.LangiumParser;df.gitGraph=e},"gitGraph"),radar:Lt(async()=>{let{createRadarServices:t}=await Promise.resolve().then(()=>(Xce(),Yce)),e=t().Radar.parser.LangiumParser;df.radar=e},"radar")};o(uo,"parse");Lt(uo,"parse");l$e=class extends Error{static{o(this,"MermaidParseError")}constructor(t){let e=t.lexerErrors.map(n=>n.message).join(` +`),r=t.parserErrors.map(n=>n.message).join(` +`);super(`Parsing failed: ${e} ${r}`),this.result=t}static{Lt(this,"MermaidParseError")}}});function $c(t,e){t.accDescr&&e.setAccDescription?.(t.accDescr),t.accTitle&&e.setAccTitle?.(t.accTitle),t.title&&e.setDiagramTitle?.(t.title)}var T1=N(()=>{"use strict";o($c,"populateCommonDb")});var Kr,a6=N(()=>{"use strict";Kr={NORMAL:0,REVERSE:1,HIGHLIGHT:2,MERGE:3,CHERRY_PICK:4}});var pf,s6=N(()=>{"use strict";pf=class{constructor(e){this.init=e;this.records=this.init()}static{o(this,"ImperativeState")}reset(){this.records=this.init()}}});function zI(){return j9({length:7})}function u$e(t,e){let r=Object.create(null);return t.reduce((n,i)=>{let a=e(i);return r[a]||(r[a]=!0,n.push(i)),n},[])}function jce(t,e,r){let n=t.indexOf(e);n===-1?t.push(r):t.splice(n,1,r)}function Qce(t){let e=t.reduce((i,a)=>i.seq>a.seq?i:a,t[0]),r="";t.forEach(function(i){i===e?r+=" *":r+=" |"});let n=[r,e.id,e.seq];for(let i in _t.records.branches)_t.records.branches.get(i)===e.id&&n.push(i);if(Y.debug(n.join(" ")),e.parents&&e.parents.length==2&&e.parents[0]&&e.parents[1]){let i=_t.records.commits.get(e.parents[0]);jce(t,e,i),e.parents[1]&&t.push(_t.records.commits.get(e.parents[1]))}else{if(e.parents.length==0)return;if(e.parents[0]){let i=_t.records.commits.get(e.parents[0]);jce(t,e,i)}}t=u$e(t,i=>i.id),Qce(t)}var c$e,Ep,_t,h$e,f$e,d$e,p$e,m$e,g$e,y$e,Kce,v$e,x$e,b$e,w$e,T$e,Zce,k$e,E$e,S$e,o6,GI=N(()=>{"use strict";vt();ir();ji();gr();mi();a6();s6();Ya();c$e=or.gitGraph,Ep=o(()=>Fi({...c$e,...cr().gitGraph}),"getConfig"),_t=new pf(()=>{let t=Ep(),e=t.mainBranchName,r=t.mainBranchOrder;return{mainBranchName:e,commits:new Map,head:null,branchConfig:new Map([[e,{name:e,order:r}]]),branches:new Map([[e,null]]),currBranch:e,direction:"LR",seq:0,options:{}}});o(zI,"getID");o(u$e,"uniqBy");h$e=o(function(t){_t.records.direction=t},"setDirection"),f$e=o(function(t){Y.debug("options str",t),t=t?.trim(),t=t||"{}";try{_t.records.options=JSON.parse(t)}catch(e){Y.error("error while parsing gitGraph options",e.message)}},"setOptions"),d$e=o(function(){return _t.records.options},"getOptions"),p$e=o(function(t){let e=t.msg,r=t.id,n=t.type,i=t.tags;Y.info("commit",e,r,n,i),Y.debug("Entering commit:",e,r,n,i);let a=Ep();r=Ze.sanitizeText(r,a),e=Ze.sanitizeText(e,a),i=i?.map(l=>Ze.sanitizeText(l,a));let s={id:r||_t.records.seq+"-"+zI(),message:e,seq:_t.records.seq++,type:n??Kr.NORMAL,tags:i??[],parents:_t.records.head==null?[]:[_t.records.head.id],branch:_t.records.currBranch};_t.records.head=s,Y.info("main branch",a.mainBranchName),_t.records.commits.set(s.id,s),_t.records.branches.set(_t.records.currBranch,s.id),Y.debug("in pushCommit "+s.id)},"commit"),m$e=o(function(t){let e=t.name,r=t.order;if(e=Ze.sanitizeText(e,Ep()),_t.records.branches.has(e))throw new Error(`Trying to create an existing branch. (Help: Either use a new name if you want create a new branch or try using "checkout ${e}")`);_t.records.branches.set(e,_t.records.head!=null?_t.records.head.id:null),_t.records.branchConfig.set(e,{name:e,order:r}),Kce(e),Y.debug("in createBranch")},"branch"),g$e=o(t=>{let e=t.branch,r=t.id,n=t.type,i=t.tags,a=Ep();e=Ze.sanitizeText(e,a),r&&(r=Ze.sanitizeText(r,a));let s=_t.records.branches.get(_t.records.currBranch),l=_t.records.branches.get(e),u=s?_t.records.commits.get(s):void 0,h=l?_t.records.commits.get(l):void 0;if(u&&h&&u.branch===e)throw new Error(`Cannot merge branch '${e}' into itself.`);if(_t.records.currBranch===e){let p=new Error('Incorrect usage of "merge". Cannot merge a branch to itself');throw p.hash={text:`merge ${e}`,token:`merge ${e}`,expected:["branch abc"]},p}if(u===void 0||!u){let p=new Error(`Incorrect usage of "merge". Current branch (${_t.records.currBranch})has no commits`);throw p.hash={text:`merge ${e}`,token:`merge ${e}`,expected:["commit"]},p}if(!_t.records.branches.has(e)){let p=new Error('Incorrect usage of "merge". Branch to be merged ('+e+") does not exist");throw p.hash={text:`merge ${e}`,token:`merge ${e}`,expected:[`branch ${e}`]},p}if(h===void 0||!h){let p=new Error('Incorrect usage of "merge". Branch to be merged ('+e+") has no commits");throw p.hash={text:`merge ${e}`,token:`merge ${e}`,expected:['"commit"']},p}if(u===h){let p=new Error('Incorrect usage of "merge". Both branches have same head');throw p.hash={text:`merge ${e}`,token:`merge ${e}`,expected:["branch abc"]},p}if(r&&_t.records.commits.has(r)){let p=new Error('Incorrect usage of "merge". Commit with id:'+r+" already exists, use different custom Id");throw p.hash={text:`merge ${e} ${r} ${n} ${i?.join(" ")}`,token:`merge ${e} ${r} ${n} ${i?.join(" ")}`,expected:[`merge ${e} ${r}_UNIQUE ${n} ${i?.join(" ")}`]},p}let f=l||"",d={id:r||`${_t.records.seq}-${zI()}`,message:`merged branch ${e} into ${_t.records.currBranch}`,seq:_t.records.seq++,parents:_t.records.head==null?[]:[_t.records.head.id,f],branch:_t.records.currBranch,type:Kr.MERGE,customType:n,customId:!!r,tags:i??[]};_t.records.head=d,_t.records.commits.set(d.id,d),_t.records.branches.set(_t.records.currBranch,d.id),Y.debug(_t.records.branches),Y.debug("in mergeBranch")},"merge"),y$e=o(function(t){let e=t.id,r=t.targetId,n=t.tags,i=t.parent;Y.debug("Entering cherryPick:",e,r,n);let a=Ep();if(e=Ze.sanitizeText(e,a),r=Ze.sanitizeText(r,a),n=n?.map(u=>Ze.sanitizeText(u,a)),i=Ze.sanitizeText(i,a),!e||!_t.records.commits.has(e)){let u=new Error('Incorrect usage of "cherryPick". Source commit id should exist and provided');throw u.hash={text:`cherryPick ${e} ${r}`,token:`cherryPick ${e} ${r}`,expected:["cherry-pick abc"]},u}let s=_t.records.commits.get(e);if(s===void 0||!s)throw new Error('Incorrect usage of "cherryPick". Source commit id should exist and provided');if(i&&!(Array.isArray(s.parents)&&s.parents.includes(i)))throw new Error("Invalid operation: The specified parent commit is not an immediate parent of the cherry-picked commit.");let l=s.branch;if(s.type===Kr.MERGE&&!i)throw new Error("Incorrect usage of cherry-pick: If the source commit is a merge commit, an immediate parent commit must be specified.");if(!r||!_t.records.commits.has(r)){if(l===_t.records.currBranch){let d=new Error('Incorrect usage of "cherryPick". Source commit is already on current branch');throw d.hash={text:`cherryPick ${e} ${r}`,token:`cherryPick ${e} ${r}`,expected:["cherry-pick abc"]},d}let u=_t.records.branches.get(_t.records.currBranch);if(u===void 0||!u){let d=new Error(`Incorrect usage of "cherry-pick". Current branch (${_t.records.currBranch})has no commits`);throw d.hash={text:`cherryPick ${e} ${r}`,token:`cherryPick ${e} ${r}`,expected:["cherry-pick abc"]},d}let h=_t.records.commits.get(u);if(h===void 0||!h){let d=new Error(`Incorrect usage of "cherry-pick". Current branch (${_t.records.currBranch})has no commits`);throw d.hash={text:`cherryPick ${e} ${r}`,token:`cherryPick ${e} ${r}`,expected:["cherry-pick abc"]},d}let f={id:_t.records.seq+"-"+zI(),message:`cherry-picked ${s?.message} into ${_t.records.currBranch}`,seq:_t.records.seq++,parents:_t.records.head==null?[]:[_t.records.head.id,s.id],branch:_t.records.currBranch,type:Kr.CHERRY_PICK,tags:n?n.filter(Boolean):[`cherry-pick:${s.id}${s.type===Kr.MERGE?`|parent:${i}`:""}`]};_t.records.head=f,_t.records.commits.set(f.id,f),_t.records.branches.set(_t.records.currBranch,f.id),Y.debug(_t.records.branches),Y.debug("in cherryPick")}},"cherryPick"),Kce=o(function(t){if(t=Ze.sanitizeText(t,Ep()),_t.records.branches.has(t)){_t.records.currBranch=t;let e=_t.records.branches.get(_t.records.currBranch);e===void 0||!e?_t.records.head=null:_t.records.head=_t.records.commits.get(e)??null}else{let e=new Error(`Trying to checkout branch which is not yet created. (Help try using "branch ${t}")`);throw e.hash={text:`checkout ${t}`,token:`checkout ${t}`,expected:[`branch ${t}`]},e}},"checkout");o(jce,"upsert");o(Qce,"prettyPrintCommitHistory");v$e=o(function(){Y.debug(_t.records.commits);let t=Zce()[0];Qce([t])},"prettyPrint"),x$e=o(function(){_t.reset(),Ar()},"clear"),b$e=o(function(){return[..._t.records.branchConfig.values()].map((e,r)=>e.order!==null&&e.order!==void 0?e:{...e,order:parseFloat(`0.${r}`)}).sort((e,r)=>(e.order??0)-(r.order??0)).map(({name:e})=>({name:e}))},"getBranchesAsObjArray"),w$e=o(function(){return _t.records.branches},"getBranches"),T$e=o(function(){return _t.records.commits},"getCommits"),Zce=o(function(){let t=[..._t.records.commits.values()];return t.forEach(function(e){Y.debug(e.id)}),t.sort((e,r)=>e.seq-r.seq),t},"getCommitsArray"),k$e=o(function(){return _t.records.currBranch},"getCurrentBranch"),E$e=o(function(){return _t.records.direction},"getDirection"),S$e=o(function(){return _t.records.head},"getHead"),o6={commitType:Kr,getConfig:Ep,setDirection:h$e,setOptions:f$e,getOptions:d$e,commit:p$e,branch:m$e,merge:g$e,cherryPick:y$e,checkout:Kce,prettyPrint:v$e,clear:x$e,getBranchesAsObjArray:b$e,getBranches:w$e,getCommits:T$e,getCommitsArray:Zce,getCurrentBranch:k$e,getDirection:E$e,getHead:S$e,setAccTitle:Lr,getAccTitle:Rr,getAccDescription:Mr,setAccDescription:Nr,setDiagramTitle:$r,getDiagramTitle:Ir}});var C$e,A$e,_$e,D$e,L$e,R$e,N$e,Jce,eue=N(()=>{"use strict";kp();vt();T1();GI();a6();C$e=o((t,e)=>{$c(t,e),t.dir&&e.setDirection(t.dir);for(let r of t.statements)A$e(r,e)},"populate"),A$e=o((t,e)=>{let n={Commit:o(i=>e.commit(_$e(i)),"Commit"),Branch:o(i=>e.branch(D$e(i)),"Branch"),Merge:o(i=>e.merge(L$e(i)),"Merge"),Checkout:o(i=>e.checkout(R$e(i)),"Checkout"),CherryPicking:o(i=>e.cherryPick(N$e(i)),"CherryPicking")}[t.$type];n?n(t):Y.error(`Unknown statement type: ${t.$type}`)},"parseStatement"),_$e=o(t=>({id:t.id,msg:t.message??"",type:t.type!==void 0?Kr[t.type]:Kr.NORMAL,tags:t.tags??void 0}),"parseCommit"),D$e=o(t=>({name:t.name,order:t.order??0}),"parseBranch"),L$e=o(t=>({branch:t.branch,id:t.id??"",type:t.type!==void 0?Kr[t.type]:void 0,tags:t.tags??void 0}),"parseMerge"),R$e=o(t=>t.branch,"parseCheckout"),N$e=o(t=>({id:t.id,targetId:"",tags:t.tags?.length===0?void 0:t.tags,parent:t.parent}),"parseCherryPicking"),Jce={parse:o(async t=>{let e=await uo("gitGraph",t);Y.debug(e),C$e(e,o6)},"parse")}});var M$e,Ko,gf,yf,zc,qu,Sp,Gs,Vs,l6,db,c6,mf,Br,I$e,rue,nue,O$e,P$e,B$e,F$e,$$e,z$e,G$e,V$e,U$e,H$e,W$e,q$e,tue,Y$e,pb,X$e,j$e,K$e,Q$e,Z$e,iue,aue=N(()=>{"use strict";dr();zt();vt();ir();a6();M$e=me(),Ko=M$e?.gitGraph,gf=10,yf=40,zc=4,qu=2,Sp=8,Gs=new Map,Vs=new Map,l6=30,db=new Map,c6=[],mf=0,Br="LR",I$e=o(()=>{Gs.clear(),Vs.clear(),db.clear(),mf=0,c6=[],Br="LR"},"clear"),rue=o(t=>{let e=document.createElementNS("http://www.w3.org/2000/svg","text");return(typeof t=="string"?t.split(/\\n|\n|/gi):t).forEach(n=>{let i=document.createElementNS("http://www.w3.org/2000/svg","tspan");i.setAttributeNS("http://www.w3.org/XML/1998/namespace","xml:space","preserve"),i.setAttribute("dy","1em"),i.setAttribute("x","0"),i.setAttribute("class","row"),i.textContent=n.trim(),e.appendChild(i)}),e},"drawText"),nue=o(t=>{let e,r,n;return Br==="BT"?(r=o((i,a)=>i<=a,"comparisonFunc"),n=1/0):(r=o((i,a)=>i>=a,"comparisonFunc"),n=0),t.forEach(i=>{let a=Br==="TB"||Br=="BT"?Vs.get(i)?.y:Vs.get(i)?.x;a!==void 0&&r(a,n)&&(e=i,n=a)}),e},"findClosestParent"),O$e=o(t=>{let e="",r=1/0;return t.forEach(n=>{let i=Vs.get(n).y;i<=r&&(e=n,r=i)}),e||void 0},"findClosestParentBT"),P$e=o((t,e,r)=>{let n=r,i=r,a=[];t.forEach(s=>{let l=e.get(s);if(!l)throw new Error(`Commit not found for key ${s}`);l.parents.length?(n=F$e(l),i=Math.max(n,i)):a.push(l),$$e(l,n)}),n=i,a.forEach(s=>{z$e(s,n,r)}),t.forEach(s=>{let l=e.get(s);if(l?.parents.length){let u=O$e(l.parents);n=Vs.get(u).y-yf,n<=i&&(i=n);let h=Gs.get(l.branch).pos,f=n-gf;Vs.set(l.id,{x:h,y:f})}})},"setParallelBTPos"),B$e=o(t=>{let e=nue(t.parents.filter(n=>n!==null));if(!e)throw new Error(`Closest parent not found for commit ${t.id}`);let r=Vs.get(e)?.y;if(r===void 0)throw new Error(`Closest parent position not found for commit ${t.id}`);return r},"findClosestParentPos"),F$e=o(t=>B$e(t)+yf,"calculateCommitPosition"),$$e=o((t,e)=>{let r=Gs.get(t.branch);if(!r)throw new Error(`Branch not found for commit ${t.id}`);let n=r.pos,i=e+gf;return Vs.set(t.id,{x:n,y:i}),{x:n,y:i}},"setCommitPosition"),z$e=o((t,e,r)=>{let n=Gs.get(t.branch);if(!n)throw new Error(`Branch not found for commit ${t.id}`);let i=e+r,a=n.pos;Vs.set(t.id,{x:a,y:i})},"setRootPosition"),G$e=o((t,e,r,n,i,a)=>{if(a===Kr.HIGHLIGHT)t.append("rect").attr("x",r.x-10).attr("y",r.y-10).attr("width",20).attr("height",20).attr("class",`commit ${e.id} commit-highlight${i%Sp} ${n}-outer`),t.append("rect").attr("x",r.x-6).attr("y",r.y-6).attr("width",12).attr("height",12).attr("class",`commit ${e.id} commit${i%Sp} ${n}-inner`);else if(a===Kr.CHERRY_PICK)t.append("circle").attr("cx",r.x).attr("cy",r.y).attr("r",10).attr("class",`commit ${e.id} ${n}`),t.append("circle").attr("cx",r.x-3).attr("cy",r.y+2).attr("r",2.75).attr("fill","#fff").attr("class",`commit ${e.id} ${n}`),t.append("circle").attr("cx",r.x+3).attr("cy",r.y+2).attr("r",2.75).attr("fill","#fff").attr("class",`commit ${e.id} ${n}`),t.append("line").attr("x1",r.x+3).attr("y1",r.y+1).attr("x2",r.x).attr("y2",r.y-5).attr("stroke","#fff").attr("class",`commit ${e.id} ${n}`),t.append("line").attr("x1",r.x-3).attr("y1",r.y+1).attr("x2",r.x).attr("y2",r.y-5).attr("stroke","#fff").attr("class",`commit ${e.id} ${n}`);else{let s=t.append("circle");if(s.attr("cx",r.x),s.attr("cy",r.y),s.attr("r",e.type===Kr.MERGE?9:10),s.attr("class",`commit ${e.id} commit${i%Sp}`),a===Kr.MERGE){let l=t.append("circle");l.attr("cx",r.x),l.attr("cy",r.y),l.attr("r",6),l.attr("class",`commit ${n} ${e.id} commit${i%Sp}`)}a===Kr.REVERSE&&t.append("path").attr("d",`M ${r.x-5},${r.y-5}L${r.x+5},${r.y+5}M${r.x-5},${r.y+5}L${r.x+5},${r.y-5}`).attr("class",`commit ${n} ${e.id} commit${i%Sp}`)}},"drawCommitBullet"),V$e=o((t,e,r,n)=>{if(e.type!==Kr.CHERRY_PICK&&(e.customId&&e.type===Kr.MERGE||e.type!==Kr.MERGE)&&Ko?.showCommitLabel){let i=t.append("g"),a=i.insert("rect").attr("class","commit-label-bkg"),s=i.append("text").attr("x",n).attr("y",r.y+25).attr("class","commit-label").text(e.id),l=s.node()?.getBBox();if(l&&(a.attr("x",r.posWithOffset-l.width/2-qu).attr("y",r.y+13.5).attr("width",l.width+2*qu).attr("height",l.height+2*qu),Br==="TB"||Br==="BT"?(a.attr("x",r.x-(l.width+4*zc+5)).attr("y",r.y-12),s.attr("x",r.x-(l.width+4*zc)).attr("y",r.y+l.height-12)):s.attr("x",r.posWithOffset-l.width/2),Ko.rotateCommitLabel))if(Br==="TB"||Br==="BT")s.attr("transform","rotate(-45, "+r.x+", "+r.y+")"),a.attr("transform","rotate(-45, "+r.x+", "+r.y+")");else{let u=-7.5-(l.width+10)/25*9.5,h=10+l.width/25*8.5;i.attr("transform","translate("+u+", "+h+") rotate(-45, "+n+", "+r.y+")")}}},"drawCommitLabel"),U$e=o((t,e,r,n)=>{if(e.tags.length>0){let i=0,a=0,s=0,l=[];for(let u of e.tags.reverse()){let h=t.insert("polygon"),f=t.append("circle"),d=t.append("text").attr("y",r.y-16-i).attr("class","tag-label").text(u),p=d.node()?.getBBox();if(!p)throw new Error("Tag bbox not found");a=Math.max(a,p.width),s=Math.max(s,p.height),d.attr("x",r.posWithOffset-p.width/2),l.push({tag:d,hole:f,rect:h,yOffset:i}),i+=20}for(let{tag:u,hole:h,rect:f,yOffset:d}of l){let p=s/2,m=r.y-19.2-d;if(f.attr("class","tag-label-bkg").attr("points",` + ${n-a/2-zc/2},${m+qu} + ${n-a/2-zc/2},${m-qu} + ${r.posWithOffset-a/2-zc},${m-p-qu} + ${r.posWithOffset+a/2+zc},${m-p-qu} + ${r.posWithOffset+a/2+zc},${m+p+qu} + ${r.posWithOffset-a/2-zc},${m+p+qu}`),h.attr("cy",m).attr("cx",n-a/2+zc/2).attr("r",1.5).attr("class","tag-hole"),Br==="TB"||Br==="BT"){let g=n+d;f.attr("class","tag-label-bkg").attr("points",` + ${r.x},${g+2} + ${r.x},${g-2} + ${r.x+gf},${g-p-2} + ${r.x+gf+a+4},${g-p-2} + ${r.x+gf+a+4},${g+p+2} + ${r.x+gf},${g+p+2}`).attr("transform","translate(12,12) rotate(45, "+r.x+","+n+")"),h.attr("cx",r.x+zc/2).attr("cy",g).attr("transform","translate(12,12) rotate(45, "+r.x+","+n+")"),u.attr("x",r.x+5).attr("y",g+3).attr("transform","translate(14,14) rotate(45, "+r.x+","+n+")")}}}},"drawCommitTags"),H$e=o(t=>{switch(t.customType??t.type){case Kr.NORMAL:return"commit-normal";case Kr.REVERSE:return"commit-reverse";case Kr.HIGHLIGHT:return"commit-highlight";case Kr.MERGE:return"commit-merge";case Kr.CHERRY_PICK:return"commit-cherry-pick";default:return"commit-normal"}},"getCommitClassType"),W$e=o((t,e,r,n)=>{let i={x:0,y:0};if(t.parents.length>0){let a=nue(t.parents);if(a){let s=n.get(a)??i;return e==="TB"?s.y+yf:e==="BT"?(n.get(t.id)??i).y-yf:s.x+yf}}else return e==="TB"?l6:e==="BT"?(n.get(t.id)??i).y-yf:0;return 0},"calculatePosition"),q$e=o((t,e,r)=>{let n=Br==="BT"&&r?e:e+gf,i=Br==="TB"||Br==="BT"?n:Gs.get(t.branch)?.pos,a=Br==="TB"||Br==="BT"?Gs.get(t.branch)?.pos:n;if(a===void 0||i===void 0)throw new Error(`Position were undefined for commit ${t.id}`);return{x:a,y:i,posWithOffset:n}},"getCommitPosition"),tue=o((t,e,r)=>{if(!Ko)throw new Error("GitGraph config not found");let n=t.append("g").attr("class","commit-bullets"),i=t.append("g").attr("class","commit-labels"),a=Br==="TB"||Br==="BT"?l6:0,s=[...e.keys()],l=Ko?.parallelCommits??!1,u=o((f,d)=>{let p=e.get(f)?.seq,m=e.get(d)?.seq;return p!==void 0&&m!==void 0?p-m:0},"sortKeys"),h=s.sort(u);Br==="BT"&&(l&&P$e(h,e,a),h=h.reverse()),h.forEach(f=>{let d=e.get(f);if(!d)throw new Error(`Commit not found for key ${f}`);l&&(a=W$e(d,Br,a,Vs));let p=q$e(d,a,l);if(r){let m=H$e(d),g=d.customType??d.type,y=Gs.get(d.branch)?.index??0;G$e(n,d,p,m,y,g),V$e(i,d,p,a),U$e(i,d,p,a)}Br==="TB"||Br==="BT"?Vs.set(d.id,{x:p.x,y:p.posWithOffset}):Vs.set(d.id,{x:p.posWithOffset,y:p.y}),a=Br==="BT"&&l?a+yf:a+yf+gf,a>mf&&(mf=a)})},"drawCommits"),Y$e=o((t,e,r,n,i)=>{let s=(Br==="TB"||Br==="BT"?r.xh.branch===s,"isOnBranchToGetCurve"),u=o(h=>h.seq>t.seq&&h.sequ(h)&&l(h))},"shouldRerouteArrow"),pb=o((t,e,r=0)=>{let n=t+Math.abs(t-e)/2;if(r>5)return n;if(c6.every(s=>Math.abs(s-n)>=10))return c6.push(n),n;let a=Math.abs(t-e);return pb(t,e-a/5,r+1)},"findLane"),X$e=o((t,e,r,n)=>{let i=Vs.get(e.id),a=Vs.get(r.id);if(i===void 0||a===void 0)throw new Error(`Commit positions not found for commits ${e.id} and ${r.id}`);let s=Y$e(e,r,i,a,n),l="",u="",h=0,f=0,d=Gs.get(r.branch)?.index;r.type===Kr.MERGE&&e.id!==r.parents[0]&&(d=Gs.get(e.branch)?.index);let p;if(s){l="A 10 10, 0, 0, 0,",u="A 10 10, 0, 0, 1,",h=10,f=10;let m=i.ya.x&&(l="A 20 20, 0, 0, 0,",u="A 20 20, 0, 0, 1,",h=20,f=20,r.type===Kr.MERGE&&e.id!==r.parents[0]?p=`M ${i.x} ${i.y} L ${i.x} ${a.y-h} ${u} ${i.x-f} ${a.y} L ${a.x} ${a.y}`:p=`M ${i.x} ${i.y} L ${a.x+h} ${i.y} ${l} ${a.x} ${i.y+f} L ${a.x} ${a.y}`),i.x===a.x&&(p=`M ${i.x} ${i.y} L ${a.x} ${a.y}`)):Br==="BT"?(i.xa.x&&(l="A 20 20, 0, 0, 0,",u="A 20 20, 0, 0, 1,",h=20,f=20,r.type===Kr.MERGE&&e.id!==r.parents[0]?p=`M ${i.x} ${i.y} L ${i.x} ${a.y+h} ${l} ${i.x-f} ${a.y} L ${a.x} ${a.y}`:p=`M ${i.x} ${i.y} L ${a.x-h} ${i.y} ${l} ${a.x} ${i.y-f} L ${a.x} ${a.y}`),i.x===a.x&&(p=`M ${i.x} ${i.y} L ${a.x} ${a.y}`)):(i.ya.y&&(r.type===Kr.MERGE&&e.id!==r.parents[0]?p=`M ${i.x} ${i.y} L ${a.x-h} ${i.y} ${l} ${a.x} ${i.y-f} L ${a.x} ${a.y}`:p=`M ${i.x} ${i.y} L ${i.x} ${a.y+h} ${u} ${i.x+f} ${a.y} L ${a.x} ${a.y}`),i.y===a.y&&(p=`M ${i.x} ${i.y} L ${a.x} ${a.y}`));if(p===void 0)throw new Error("Line definition not found");t.append("path").attr("d",p).attr("class","arrow arrow"+d%Sp)},"drawArrow"),j$e=o((t,e)=>{let r=t.append("g").attr("class","commit-arrows");[...e.keys()].forEach(n=>{let i=e.get(n);i.parents&&i.parents.length>0&&i.parents.forEach(a=>{X$e(r,e.get(a),i,e)})})},"drawArrows"),K$e=o((t,e)=>{let r=t.append("g");e.forEach((n,i)=>{let a=i%Sp,s=Gs.get(n.name)?.pos;if(s===void 0)throw new Error(`Position not found for branch ${n.name}`);let l=r.append("line");l.attr("x1",0),l.attr("y1",s),l.attr("x2",mf),l.attr("y2",s),l.attr("class","branch branch"+a),Br==="TB"?(l.attr("y1",l6),l.attr("x1",s),l.attr("y2",mf),l.attr("x2",s)):Br==="BT"&&(l.attr("y1",mf),l.attr("x1",s),l.attr("y2",l6),l.attr("x2",s)),c6.push(s);let u=n.name,h=rue(u),f=r.insert("rect"),p=r.insert("g").attr("class","branchLabel").insert("g").attr("class","label branch-label"+a);p.node().appendChild(h);let m=h.getBBox();f.attr("class","branchLabelBkg label"+a).attr("rx",4).attr("ry",4).attr("x",-m.width-4-(Ko?.rotateCommitLabel===!0?30:0)).attr("y",-m.height/2+8).attr("width",m.width+18).attr("height",m.height+4),p.attr("transform","translate("+(-m.width-14-(Ko?.rotateCommitLabel===!0?30:0))+", "+(s-m.height/2-1)+")"),Br==="TB"?(f.attr("x",s-m.width/2-10).attr("y",0),p.attr("transform","translate("+(s-m.width/2-5)+", 0)")):Br==="BT"?(f.attr("x",s-m.width/2-10).attr("y",mf),p.attr("transform","translate("+(s-m.width/2-5)+", "+mf+")")):f.attr("transform","translate(-19, "+(s-m.height/2)+")")})},"drawBranches"),Q$e=o(function(t,e,r,n,i){return Gs.set(t,{pos:e,index:r}),e+=50+(i?40:0)+(Br==="TB"||Br==="BT"?n.width/2:0),e},"setBranchPosition"),Z$e=o(function(t,e,r,n){if(I$e(),Y.debug("in gitgraph renderer",t+` +`,"id:",e,r),!Ko)throw new Error("GitGraph config not found");let i=Ko.rotateCommitLabel??!1,a=n.db;db=a.getCommits();let s=a.getBranchesAsObjArray();Br=a.getDirection();let l=Ge(`[id="${e}"]`),u=0;s.forEach((h,f)=>{let d=rue(h.name),p=l.append("g"),m=p.insert("g").attr("class","branchLabel"),g=m.insert("g").attr("class","label branch-label");g.node()?.appendChild(d);let y=d.getBBox();u=Q$e(h.name,u,f,y,i),g.remove(),m.remove(),p.remove()}),tue(l,db,!1),Ko.showBranches&&K$e(l,s),j$e(l,db),tue(l,db,!0),Gt.insertTitle(l,"gitTitleText",Ko.titleTopMargin??0,a.getDiagramTitle()),oA(void 0,l,Ko.diagramPadding,Ko.useMaxWidth)},"draw"),iue={draw:Z$e}});var J$e,sue,oue=N(()=>{"use strict";J$e=o(t=>` + .commit-id, + .commit-msg, + .branch-label { + fill: lightgrey; + color: lightgrey; + font-family: 'trebuchet ms', verdana, arial, sans-serif; + font-family: var(--mermaid-font-family); + } + ${[0,1,2,3,4,5,6,7].map(e=>` + .branch-label${e} { fill: ${t["gitBranchLabel"+e]}; } + .commit${e} { stroke: ${t["git"+e]}; fill: ${t["git"+e]}; } + .commit-highlight${e} { stroke: ${t["gitInv"+e]}; fill: ${t["gitInv"+e]}; } + .label${e} { fill: ${t["git"+e]}; } + .arrow${e} { stroke: ${t["git"+e]}; } + `).join(` +`)} + + .branch { + stroke-width: 1; + stroke: ${t.lineColor}; + stroke-dasharray: 2; + } + .commit-label { font-size: ${t.commitLabelFontSize}; fill: ${t.commitLabelColor};} + .commit-label-bkg { font-size: ${t.commitLabelFontSize}; fill: ${t.commitLabelBackground}; opacity: 0.5; } + .tag-label { font-size: ${t.tagLabelFontSize}; fill: ${t.tagLabelColor};} + .tag-label-bkg { fill: ${t.tagLabelBackground}; stroke: ${t.tagLabelBorder}; } + .tag-hole { fill: ${t.textColor}; } + + .commit-merge { + stroke: ${t.primaryColor}; + fill: ${t.primaryColor}; + } + .commit-reverse { + stroke: ${t.primaryColor}; + fill: ${t.primaryColor}; + stroke-width: 3; + } + .commit-highlight-outer { + } + .commit-highlight-inner { + stroke: ${t.primaryColor}; + fill: ${t.primaryColor}; + } + + .arrow { stroke-width: 8; stroke-linecap: round; fill: none} + .gitTitleText { + text-anchor: middle; + font-size: 18px; + fill: ${t.textColor}; + } +`,"getStyles"),sue=J$e});var lue={};hr(lue,{diagram:()=>eze});var eze,cue=N(()=>{"use strict";eue();GI();aue();oue();eze={parser:Jce,db:o6,renderer:iue,styles:sue}});var VI,fue,due=N(()=>{"use strict";VI=function(){var t=o(function(L,R,O,M){for(O=O||{},M=L.length;M--;O[L[M]]=R);return O},"o"),e=[6,8,10,12,13,14,15,16,17,18,20,21,22,23,24,25,26,27,28,29,30,31,33,35,36,38,40],r=[1,26],n=[1,27],i=[1,28],a=[1,29],s=[1,30],l=[1,31],u=[1,32],h=[1,33],f=[1,34],d=[1,9],p=[1,10],m=[1,11],g=[1,12],y=[1,13],v=[1,14],x=[1,15],b=[1,16],w=[1,19],C=[1,20],T=[1,21],E=[1,22],A=[1,23],S=[1,25],_=[1,35],I={trace:o(function(){},"trace"),yy:{},symbols_:{error:2,start:3,gantt:4,document:5,EOF:6,line:7,SPACE:8,statement:9,NL:10,weekday:11,weekday_monday:12,weekday_tuesday:13,weekday_wednesday:14,weekday_thursday:15,weekday_friday:16,weekday_saturday:17,weekday_sunday:18,weekend:19,weekend_friday:20,weekend_saturday:21,dateFormat:22,inclusiveEndDates:23,topAxis:24,axisFormat:25,tickInterval:26,excludes:27,includes:28,todayMarker:29,title:30,acc_title:31,acc_title_value:32,acc_descr:33,acc_descr_value:34,acc_descr_multiline_value:35,section:36,clickStatement:37,taskTxt:38,taskData:39,click:40,callbackname:41,callbackargs:42,href:43,clickStatementDebug:44,$accept:0,$end:1},terminals_:{2:"error",4:"gantt",6:"EOF",8:"SPACE",10:"NL",12:"weekday_monday",13:"weekday_tuesday",14:"weekday_wednesday",15:"weekday_thursday",16:"weekday_friday",17:"weekday_saturday",18:"weekday_sunday",20:"weekend_friday",21:"weekend_saturday",22:"dateFormat",23:"inclusiveEndDates",24:"topAxis",25:"axisFormat",26:"tickInterval",27:"excludes",28:"includes",29:"todayMarker",30:"title",31:"acc_title",32:"acc_title_value",33:"acc_descr",34:"acc_descr_value",35:"acc_descr_multiline_value",36:"section",38:"taskTxt",39:"taskData",40:"click",41:"callbackname",42:"callbackargs",43:"href"},productions_:[0,[3,3],[5,0],[5,2],[7,2],[7,1],[7,1],[7,1],[11,1],[11,1],[11,1],[11,1],[11,1],[11,1],[11,1],[19,1],[19,1],[9,1],[9,1],[9,1],[9,1],[9,1],[9,1],[9,1],[9,1],[9,1],[9,1],[9,1],[9,2],[9,2],[9,1],[9,1],[9,1],[9,2],[37,2],[37,3],[37,3],[37,4],[37,3],[37,4],[37,2],[44,2],[44,3],[44,3],[44,4],[44,3],[44,4],[44,2]],performAction:o(function(R,O,M,B,F,P,z){var $=P.length-1;switch(F){case 1:return P[$-1];case 2:this.$=[];break;case 3:P[$-1].push(P[$]),this.$=P[$-1];break;case 4:case 5:this.$=P[$];break;case 6:case 7:this.$=[];break;case 8:B.setWeekday("monday");break;case 9:B.setWeekday("tuesday");break;case 10:B.setWeekday("wednesday");break;case 11:B.setWeekday("thursday");break;case 12:B.setWeekday("friday");break;case 13:B.setWeekday("saturday");break;case 14:B.setWeekday("sunday");break;case 15:B.setWeekend("friday");break;case 16:B.setWeekend("saturday");break;case 17:B.setDateFormat(P[$].substr(11)),this.$=P[$].substr(11);break;case 18:B.enableInclusiveEndDates(),this.$=P[$].substr(18);break;case 19:B.TopAxis(),this.$=P[$].substr(8);break;case 20:B.setAxisFormat(P[$].substr(11)),this.$=P[$].substr(11);break;case 21:B.setTickInterval(P[$].substr(13)),this.$=P[$].substr(13);break;case 22:B.setExcludes(P[$].substr(9)),this.$=P[$].substr(9);break;case 23:B.setIncludes(P[$].substr(9)),this.$=P[$].substr(9);break;case 24:B.setTodayMarker(P[$].substr(12)),this.$=P[$].substr(12);break;case 27:B.setDiagramTitle(P[$].substr(6)),this.$=P[$].substr(6);break;case 28:this.$=P[$].trim(),B.setAccTitle(this.$);break;case 29:case 30:this.$=P[$].trim(),B.setAccDescription(this.$);break;case 31:B.addSection(P[$].substr(8)),this.$=P[$].substr(8);break;case 33:B.addTask(P[$-1],P[$]),this.$="task";break;case 34:this.$=P[$-1],B.setClickEvent(P[$-1],P[$],null);break;case 35:this.$=P[$-2],B.setClickEvent(P[$-2],P[$-1],P[$]);break;case 36:this.$=P[$-2],B.setClickEvent(P[$-2],P[$-1],null),B.setLink(P[$-2],P[$]);break;case 37:this.$=P[$-3],B.setClickEvent(P[$-3],P[$-2],P[$-1]),B.setLink(P[$-3],P[$]);break;case 38:this.$=P[$-2],B.setClickEvent(P[$-2],P[$],null),B.setLink(P[$-2],P[$-1]);break;case 39:this.$=P[$-3],B.setClickEvent(P[$-3],P[$-1],P[$]),B.setLink(P[$-3],P[$-2]);break;case 40:this.$=P[$-1],B.setLink(P[$-1],P[$]);break;case 41:case 47:this.$=P[$-1]+" "+P[$];break;case 42:case 43:case 45:this.$=P[$-2]+" "+P[$-1]+" "+P[$];break;case 44:case 46:this.$=P[$-3]+" "+P[$-2]+" "+P[$-1]+" "+P[$];break}},"anonymous"),table:[{3:1,4:[1,2]},{1:[3]},t(e,[2,2],{5:3}),{6:[1,4],7:5,8:[1,6],9:7,10:[1,8],11:17,12:r,13:n,14:i,15:a,16:s,17:l,18:u,19:18,20:h,21:f,22:d,23:p,24:m,25:g,26:y,27:v,28:x,29:b,30:w,31:C,33:T,35:E,36:A,37:24,38:S,40:_},t(e,[2,7],{1:[2,1]}),t(e,[2,3]),{9:36,11:17,12:r,13:n,14:i,15:a,16:s,17:l,18:u,19:18,20:h,21:f,22:d,23:p,24:m,25:g,26:y,27:v,28:x,29:b,30:w,31:C,33:T,35:E,36:A,37:24,38:S,40:_},t(e,[2,5]),t(e,[2,6]),t(e,[2,17]),t(e,[2,18]),t(e,[2,19]),t(e,[2,20]),t(e,[2,21]),t(e,[2,22]),t(e,[2,23]),t(e,[2,24]),t(e,[2,25]),t(e,[2,26]),t(e,[2,27]),{32:[1,37]},{34:[1,38]},t(e,[2,30]),t(e,[2,31]),t(e,[2,32]),{39:[1,39]},t(e,[2,8]),t(e,[2,9]),t(e,[2,10]),t(e,[2,11]),t(e,[2,12]),t(e,[2,13]),t(e,[2,14]),t(e,[2,15]),t(e,[2,16]),{41:[1,40],43:[1,41]},t(e,[2,4]),t(e,[2,28]),t(e,[2,29]),t(e,[2,33]),t(e,[2,34],{42:[1,42],43:[1,43]}),t(e,[2,40],{41:[1,44]}),t(e,[2,35],{43:[1,45]}),t(e,[2,36]),t(e,[2,38],{42:[1,46]}),t(e,[2,37]),t(e,[2,39])],defaultActions:{},parseError:o(function(R,O){if(O.recoverable)this.trace(R);else{var M=new Error(R);throw M.hash=O,M}},"parseError"),parse:o(function(R){var O=this,M=[0],B=[],F=[null],P=[],z=this.table,$="",H=0,Q=0,j=0,ie=2,ne=1,le=P.slice.call(arguments,1),he=Object.create(this.lexer),K={yy:{}};for(var X in this.yy)Object.prototype.hasOwnProperty.call(this.yy,X)&&(K.yy[X]=this.yy[X]);he.setInput(R,K.yy),K.yy.lexer=he,K.yy.parser=this,typeof he.yylloc>"u"&&(he.yylloc={});var te=he.yylloc;P.push(te);var J=he.options&&he.options.ranges;typeof K.yy.parseError=="function"?this.parseError=K.yy.parseError:this.parseError=Object.getPrototypeOf(this).parseError;function se(W){M.length=M.length-2*W,F.length=F.length-W,P.length=P.length-W}o(se,"popStack");function ue(){var W;return W=B.pop()||he.lex()||ne,typeof W!="number"&&(W instanceof Array&&(B=W,W=B.pop()),W=O.symbols_[W]||W),W}o(ue,"lex");for(var Z,Se,ce,ae,Oe,ge,ze={},He,$e,Re,Ie;;){if(ce=M[M.length-1],this.defaultActions[ce]?ae=this.defaultActions[ce]:((Z===null||typeof Z>"u")&&(Z=ue()),ae=z[ce]&&z[ce][Z]),typeof ae>"u"||!ae.length||!ae[0]){var be="";Ie=[];for(He in z[ce])this.terminals_[He]&&He>ie&&Ie.push("'"+this.terminals_[He]+"'");he.showPosition?be="Parse error on line "+(H+1)+`: +`+he.showPosition()+` +Expecting `+Ie.join(", ")+", got '"+(this.terminals_[Z]||Z)+"'":be="Parse error on line "+(H+1)+": Unexpected "+(Z==ne?"end of input":"'"+(this.terminals_[Z]||Z)+"'"),this.parseError(be,{text:he.match,token:this.terminals_[Z]||Z,line:he.yylineno,loc:te,expected:Ie})}if(ae[0]instanceof Array&&ae.length>1)throw new Error("Parse Error: multiple actions possible at state: "+ce+", token: "+Z);switch(ae[0]){case 1:M.push(Z),F.push(he.yytext),P.push(he.yylloc),M.push(ae[1]),Z=null,Se?(Z=Se,Se=null):(Q=he.yyleng,$=he.yytext,H=he.yylineno,te=he.yylloc,j>0&&j--);break;case 2:if($e=this.productions_[ae[1]][1],ze.$=F[F.length-$e],ze._$={first_line:P[P.length-($e||1)].first_line,last_line:P[P.length-1].last_line,first_column:P[P.length-($e||1)].first_column,last_column:P[P.length-1].last_column},J&&(ze._$.range=[P[P.length-($e||1)].range[0],P[P.length-1].range[1]]),ge=this.performAction.apply(ze,[$,Q,H,K.yy,ae[1],F,P].concat(le)),typeof ge<"u")return ge;$e&&(M=M.slice(0,-1*$e*2),F=F.slice(0,-1*$e),P=P.slice(0,-1*$e)),M.push(this.productions_[ae[1]][0]),F.push(ze.$),P.push(ze._$),Re=z[M[M.length-2]][M[M.length-1]],M.push(Re);break;case 3:return!0}}return!0},"parse")},D=function(){var L={EOF:1,parseError:o(function(O,M){if(this.yy.parser)this.yy.parser.parseError(O,M);else throw new Error(O)},"parseError"),setInput:o(function(R,O){return this.yy=O||this.yy||{},this._input=R,this._more=this._backtrack=this.done=!1,this.yylineno=this.yyleng=0,this.yytext=this.matched=this.match="",this.conditionStack=["INITIAL"],this.yylloc={first_line:1,first_column:0,last_line:1,last_column:0},this.options.ranges&&(this.yylloc.range=[0,0]),this.offset=0,this},"setInput"),input:o(function(){var R=this._input[0];this.yytext+=R,this.yyleng++,this.offset++,this.match+=R,this.matched+=R;var O=R.match(/(?:\r\n?|\n).*/g);return O?(this.yylineno++,this.yylloc.last_line++):this.yylloc.last_column++,this.options.ranges&&this.yylloc.range[1]++,this._input=this._input.slice(1),R},"input"),unput:o(function(R){var O=R.length,M=R.split(/(?:\r\n?|\n)/g);this._input=R+this._input,this.yytext=this.yytext.substr(0,this.yytext.length-O),this.offset-=O;var B=this.match.split(/(?:\r\n?|\n)/g);this.match=this.match.substr(0,this.match.length-1),this.matched=this.matched.substr(0,this.matched.length-1),M.length-1&&(this.yylineno-=M.length-1);var F=this.yylloc.range;return this.yylloc={first_line:this.yylloc.first_line,last_line:this.yylineno+1,first_column:this.yylloc.first_column,last_column:M?(M.length===B.length?this.yylloc.first_column:0)+B[B.length-M.length].length-M[0].length:this.yylloc.first_column-O},this.options.ranges&&(this.yylloc.range=[F[0],F[0]+this.yyleng-O]),this.yyleng=this.yytext.length,this},"unput"),more:o(function(){return this._more=!0,this},"more"),reject:o(function(){if(this.options.backtrack_lexer)this._backtrack=!0;else return this.parseError("Lexical error on line "+(this.yylineno+1)+`. You can only invoke reject() in the lexer when the lexer is of the backtracking persuasion (options.backtrack_lexer = true). +`+this.showPosition(),{text:"",token:null,line:this.yylineno});return this},"reject"),less:o(function(R){this.unput(this.match.slice(R))},"less"),pastInput:o(function(){var R=this.matched.substr(0,this.matched.length-this.match.length);return(R.length>20?"...":"")+R.substr(-20).replace(/\n/g,"")},"pastInput"),upcomingInput:o(function(){var R=this.match;return R.length<20&&(R+=this._input.substr(0,20-R.length)),(R.substr(0,20)+(R.length>20?"...":"")).replace(/\n/g,"")},"upcomingInput"),showPosition:o(function(){var R=this.pastInput(),O=new Array(R.length+1).join("-");return R+this.upcomingInput()+` +`+O+"^"},"showPosition"),test_match:o(function(R,O){var M,B,F;if(this.options.backtrack_lexer&&(F={yylineno:this.yylineno,yylloc:{first_line:this.yylloc.first_line,last_line:this.last_line,first_column:this.yylloc.first_column,last_column:this.yylloc.last_column},yytext:this.yytext,match:this.match,matches:this.matches,matched:this.matched,yyleng:this.yyleng,offset:this.offset,_more:this._more,_input:this._input,yy:this.yy,conditionStack:this.conditionStack.slice(0),done:this.done},this.options.ranges&&(F.yylloc.range=this.yylloc.range.slice(0))),B=R[0].match(/(?:\r\n?|\n).*/g),B&&(this.yylineno+=B.length),this.yylloc={first_line:this.yylloc.last_line,last_line:this.yylineno+1,first_column:this.yylloc.last_column,last_column:B?B[B.length-1].length-B[B.length-1].match(/\r?\n?/)[0].length:this.yylloc.last_column+R[0].length},this.yytext+=R[0],this.match+=R[0],this.matches=R,this.yyleng=this.yytext.length,this.options.ranges&&(this.yylloc.range=[this.offset,this.offset+=this.yyleng]),this._more=!1,this._backtrack=!1,this._input=this._input.slice(R[0].length),this.matched+=R[0],M=this.performAction.call(this,this.yy,this,O,this.conditionStack[this.conditionStack.length-1]),this.done&&this._input&&(this.done=!1),M)return M;if(this._backtrack){for(var P in F)this[P]=F[P];return!1}return!1},"test_match"),next:o(function(){if(this.done)return this.EOF;this._input||(this.done=!0);var R,O,M,B;this._more||(this.yytext="",this.match="");for(var F=this._currentRules(),P=0;PO[0].length)){if(O=M,B=P,this.options.backtrack_lexer){if(R=this.test_match(M,F[P]),R!==!1)return R;if(this._backtrack){O=!1;continue}else return!1}else if(!this.options.flex)break}return O?(R=this.test_match(O,F[B]),R!==!1?R:!1):this._input===""?this.EOF:this.parseError("Lexical error on line "+(this.yylineno+1)+`. Unrecognized text. +`+this.showPosition(),{text:"",token:null,line:this.yylineno})},"next"),lex:o(function(){var O=this.next();return O||this.lex()},"lex"),begin:o(function(O){this.conditionStack.push(O)},"begin"),popState:o(function(){var O=this.conditionStack.length-1;return O>0?this.conditionStack.pop():this.conditionStack[0]},"popState"),_currentRules:o(function(){return this.conditionStack.length&&this.conditionStack[this.conditionStack.length-1]?this.conditions[this.conditionStack[this.conditionStack.length-1]].rules:this.conditions.INITIAL.rules},"_currentRules"),topState:o(function(O){return O=this.conditionStack.length-1-Math.abs(O||0),O>=0?this.conditionStack[O]:"INITIAL"},"topState"),pushState:o(function(O){this.begin(O)},"pushState"),stateStackSize:o(function(){return this.conditionStack.length},"stateStackSize"),options:{"case-insensitive":!0},performAction:o(function(O,M,B,F){var P=F;switch(B){case 0:return this.begin("open_directive"),"open_directive";break;case 1:return this.begin("acc_title"),31;break;case 2:return this.popState(),"acc_title_value";break;case 3:return this.begin("acc_descr"),33;break;case 4:return this.popState(),"acc_descr_value";break;case 5:this.begin("acc_descr_multiline");break;case 6:this.popState();break;case 7:return"acc_descr_multiline_value";case 8:break;case 9:break;case 10:break;case 11:return 10;case 12:break;case 13:break;case 14:this.begin("href");break;case 15:this.popState();break;case 16:return 43;case 17:this.begin("callbackname");break;case 18:this.popState();break;case 19:this.popState(),this.begin("callbackargs");break;case 20:return 41;case 21:this.popState();break;case 22:return 42;case 23:this.begin("click");break;case 24:this.popState();break;case 25:return 40;case 26:return 4;case 27:return 22;case 28:return 23;case 29:return 24;case 30:return 25;case 31:return 26;case 32:return 28;case 33:return 27;case 34:return 29;case 35:return 12;case 36:return 13;case 37:return 14;case 38:return 15;case 39:return 16;case 40:return 17;case 41:return 18;case 42:return 20;case 43:return 21;case 44:return"date";case 45:return 30;case 46:return"accDescription";case 47:return 36;case 48:return 38;case 49:return 39;case 50:return":";case 51:return 6;case 52:return"INVALID"}},"anonymous"),rules:[/^(?:%%\{)/i,/^(?:accTitle\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*\{\s*)/i,/^(?:[\}])/i,/^(?:[^\}]*)/i,/^(?:%%(?!\{)*[^\n]*)/i,/^(?:[^\}]%%*[^\n]*)/i,/^(?:%%*[^\n]*[\n]*)/i,/^(?:[\n]+)/i,/^(?:\s+)/i,/^(?:%[^\n]*)/i,/^(?:href[\s]+["])/i,/^(?:["])/i,/^(?:[^"]*)/i,/^(?:call[\s]+)/i,/^(?:\([\s]*\))/i,/^(?:\()/i,/^(?:[^(]*)/i,/^(?:\))/i,/^(?:[^)]*)/i,/^(?:click[\s]+)/i,/^(?:[\s\n])/i,/^(?:[^\s\n]*)/i,/^(?:gantt\b)/i,/^(?:dateFormat\s[^#\n;]+)/i,/^(?:inclusiveEndDates\b)/i,/^(?:topAxis\b)/i,/^(?:axisFormat\s[^#\n;]+)/i,/^(?:tickInterval\s[^#\n;]+)/i,/^(?:includes\s[^#\n;]+)/i,/^(?:excludes\s[^#\n;]+)/i,/^(?:todayMarker\s[^\n;]+)/i,/^(?:weekday\s+monday\b)/i,/^(?:weekday\s+tuesday\b)/i,/^(?:weekday\s+wednesday\b)/i,/^(?:weekday\s+thursday\b)/i,/^(?:weekday\s+friday\b)/i,/^(?:weekday\s+saturday\b)/i,/^(?:weekday\s+sunday\b)/i,/^(?:weekend\s+friday\b)/i,/^(?:weekend\s+saturday\b)/i,/^(?:\d\d\d\d-\d\d-\d\d\b)/i,/^(?:title\s[^\n]+)/i,/^(?:accDescription\s[^#\n;]+)/i,/^(?:section\s[^\n]+)/i,/^(?:[^:\n]+)/i,/^(?::[^#\n;]+)/i,/^(?::)/i,/^(?:$)/i,/^(?:.)/i],conditions:{acc_descr_multiline:{rules:[6,7],inclusive:!1},acc_descr:{rules:[4],inclusive:!1},acc_title:{rules:[2],inclusive:!1},callbackargs:{rules:[21,22],inclusive:!1},callbackname:{rules:[18,19,20],inclusive:!1},href:{rules:[15,16],inclusive:!1},click:{rules:[24,25],inclusive:!1},INITIAL:{rules:[0,1,3,5,8,9,10,11,12,13,14,17,23,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52],inclusive:!0}}};return L}();I.lexer=D;function k(){this.yy={}}return o(k,"Parser"),k.prototype=I,I.Parser=k,new k}();VI.parser=VI;fue=VI});var pue=Mi((UI,HI)=>{"use strict";(function(t,e){typeof UI=="object"&&typeof HI<"u"?HI.exports=e():typeof define=="function"&&define.amd?define(e):(t=typeof globalThis<"u"?globalThis:t||self).dayjs_plugin_isoWeek=e()})(UI,function(){"use strict";var t="day";return function(e,r,n){var i=o(function(l){return l.add(4-l.isoWeekday(),t)},"a"),a=r.prototype;a.isoWeekYear=function(){return i(this).year()},a.isoWeek=function(l){if(!this.$utils().u(l))return this.add(7*(l-this.isoWeek()),t);var u,h,f,d,p=i(this),m=(u=this.isoWeekYear(),h=this.$u,f=(h?n.utc:n)().year(u).startOf("year"),d=4-f.isoWeekday(),f.isoWeekday()>4&&(d+=7),f.add(d,t));return p.diff(m,"week")+1},a.isoWeekday=function(l){return this.$utils().u(l)?this.day()||7:this.day(this.day()%7?l:l-7)};var s=a.startOf;a.startOf=function(l,u){var h=this.$utils(),f=!!h.u(u)||u;return h.p(l)==="isoweek"?f?this.date(this.date()-(this.isoWeekday()-1)).startOf("day"):this.date(this.date()-1-(this.isoWeekday()-1)+7).endOf("day"):s.bind(this)(l,u)}}})});var mue=Mi((WI,qI)=>{"use strict";(function(t,e){typeof WI=="object"&&typeof qI<"u"?qI.exports=e():typeof define=="function"&&define.amd?define(e):(t=typeof globalThis<"u"?globalThis:t||self).dayjs_plugin_customParseFormat=e()})(WI,function(){"use strict";var t={LTS:"h:mm:ss A",LT:"h:mm A",L:"MM/DD/YYYY",LL:"MMMM D, YYYY",LLL:"MMMM D, YYYY h:mm A",LLLL:"dddd, MMMM D, YYYY h:mm A"},e=/(\[[^[]*\])|([-_:/.,()\s]+)|(A|a|Q|YYYY|YY?|ww?|MM?M?M?|Do|DD?|hh?|HH?|mm?|ss?|S{1,3}|z|ZZ?)/g,r=/\d/,n=/\d\d/,i=/\d\d?/,a=/\d*[^-_:/,()\s\d]+/,s={},l=o(function(g){return(g=+g)+(g>68?1900:2e3)},"a"),u=o(function(g){return function(y){this[g]=+y}},"f"),h=[/[+-]\d\d:?(\d\d)?|Z/,function(g){(this.zone||(this.zone={})).offset=function(y){if(!y||y==="Z")return 0;var v=y.match(/([+-]|\d\d)/g),x=60*v[1]+(+v[2]||0);return x===0?0:v[0]==="+"?-x:x}(g)}],f=o(function(g){var y=s[g];return y&&(y.indexOf?y:y.s.concat(y.f))},"u"),d=o(function(g,y){var v,x=s.meridiem;if(x){for(var b=1;b<=24;b+=1)if(g.indexOf(x(b,0,y))>-1){v=b>12;break}}else v=g===(y?"pm":"PM");return v},"d"),p={A:[a,function(g){this.afternoon=d(g,!1)}],a:[a,function(g){this.afternoon=d(g,!0)}],Q:[r,function(g){this.month=3*(g-1)+1}],S:[r,function(g){this.milliseconds=100*+g}],SS:[n,function(g){this.milliseconds=10*+g}],SSS:[/\d{3}/,function(g){this.milliseconds=+g}],s:[i,u("seconds")],ss:[i,u("seconds")],m:[i,u("minutes")],mm:[i,u("minutes")],H:[i,u("hours")],h:[i,u("hours")],HH:[i,u("hours")],hh:[i,u("hours")],D:[i,u("day")],DD:[n,u("day")],Do:[a,function(g){var y=s.ordinal,v=g.match(/\d+/);if(this.day=v[0],y)for(var x=1;x<=31;x+=1)y(x).replace(/\[|\]/g,"")===g&&(this.day=x)}],w:[i,u("week")],ww:[n,u("week")],M:[i,u("month")],MM:[n,u("month")],MMM:[a,function(g){var y=f("months"),v=(f("monthsShort")||y.map(function(x){return x.slice(0,3)})).indexOf(g)+1;if(v<1)throw new Error;this.month=v%12||v}],MMMM:[a,function(g){var y=f("months").indexOf(g)+1;if(y<1)throw new Error;this.month=y%12||y}],Y:[/[+-]?\d+/,u("year")],YY:[n,function(g){this.year=l(g)}],YYYY:[/\d{4}/,u("year")],Z:h,ZZ:h};function m(g){var y,v;y=g,v=s&&s.formats;for(var x=(g=y.replace(/(\[[^\]]+])|(LTS?|l{1,4}|L{1,4})/g,function(S,_,I){var D=I&&I.toUpperCase();return _||v[I]||t[I]||v[D].replace(/(\[[^\]]+])|(MMMM|MM|DD|dddd)/g,function(k,L,R){return L||R.slice(1)})})).match(e),b=x.length,w=0;w-1)return new Date((M==="X"?1e3:1)*O);var P=m(M)(O),z=P.year,$=P.month,H=P.day,Q=P.hours,j=P.minutes,ie=P.seconds,ne=P.milliseconds,le=P.zone,he=P.week,K=new Date,X=H||(z||$?1:K.getDate()),te=z||K.getFullYear(),J=0;z&&!$||(J=$>0?$-1:K.getMonth());var se,ue=Q||0,Z=j||0,Se=ie||0,ce=ne||0;return le?new Date(Date.UTC(te,J,X,ue,Z,Se,ce+60*le.offset*1e3)):B?new Date(Date.UTC(te,J,X,ue,Z,Se,ce)):(se=new Date(te,J,X,ue,Z,Se,ce),he&&(se=F(se).week(he).toDate()),se)}catch{return new Date("")}}(C,A,T,v),this.init(),D&&D!==!0&&(this.$L=this.locale(D).$L),I&&C!=this.format(A)&&(this.$d=new Date("")),s={}}else if(A instanceof Array)for(var k=A.length,L=1;L<=k;L+=1){E[1]=A[L-1];var R=v.apply(this,E);if(R.isValid()){this.$d=R.$d,this.$L=R.$L,this.init();break}L===k&&(this.$d=new Date(""))}else b.call(this,w)}}})});var gue=Mi((YI,XI)=>{"use strict";(function(t,e){typeof YI=="object"&&typeof XI<"u"?XI.exports=e():typeof define=="function"&&define.amd?define(e):(t=typeof globalThis<"u"?globalThis:t||self).dayjs_plugin_advancedFormat=e()})(YI,function(){"use strict";return function(t,e){var r=e.prototype,n=r.format;r.format=function(i){var a=this,s=this.$locale();if(!this.isValid())return n.bind(this)(i);var l=this.$utils(),u=(i||"YYYY-MM-DDTHH:mm:ssZ").replace(/\[([^\]]+)]|Q|wo|ww|w|WW|W|zzz|z|gggg|GGGG|Do|X|x|k{1,2}|S/g,function(h){switch(h){case"Q":return Math.ceil((a.$M+1)/3);case"Do":return s.ordinal(a.$D);case"gggg":return a.weekYear();case"GGGG":return a.isoWeekYear();case"wo":return s.ordinal(a.week(),"W");case"w":case"ww":return l.s(a.week(),h==="w"?1:2,"0");case"W":case"WW":return l.s(a.isoWeek(),h==="W"?1:2,"0");case"k":case"kk":return l.s(String(a.$H===0?24:a.$H),h==="k"?1:2,"0");case"X":return Math.floor(a.$d.getTime()/1e3);case"x":return a.$d.getTime();case"z":return"["+a.offsetName()+"]";case"zzz":return"["+a.offsetName("long")+"]";default:return h}});return n.bind(this)(u)}}})});function Nue(t,e,r){let n=!0;for(;n;)n=!1,r.forEach(function(i){let a="^\\s*"+i+"\\s*$",s=new RegExp(a);t[0].match(s)&&(e[i]=!0,t.shift(1),n=!0)})}var xue,ho,bue,wue,Tue,yue,Gc,ZI,JI,eO,mb,gb,tO,rO,f6,E1,nO,kue,iO,yb,aO,sO,d6,jI,ize,aze,sze,oze,lze,cze,uze,hze,fze,dze,pze,mze,gze,yze,vze,xze,bze,wze,Tze,kze,Eze,Sze,Cze,Eue,Aze,_ze,Dze,Sue,Lze,KI,Cue,Aue,u6,k1,Rze,Nze,QI,h6,Gi,_ue,Mze,Cp,Ize,vue,Oze,Due,Pze,Lue,Bze,Fze,Rue,Mue=N(()=>{"use strict";xue=Sa(z0(),1),ho=Sa(R4(),1),bue=Sa(pue(),1),wue=Sa(mue(),1),Tue=Sa(gue(),1);vt();zt();ir();mi();ho.default.extend(bue.default);ho.default.extend(wue.default);ho.default.extend(Tue.default);yue={friday:5,saturday:6},Gc="",ZI="",eO="",mb=[],gb=[],tO=new Map,rO=[],f6=[],E1="",nO="",kue=["active","done","crit","milestone"],iO=[],yb=!1,aO=!1,sO="sunday",d6="saturday",jI=0,ize=o(function(){rO=[],f6=[],E1="",iO=[],u6=0,QI=void 0,h6=void 0,Gi=[],Gc="",ZI="",nO="",JI=void 0,eO="",mb=[],gb=[],yb=!1,aO=!1,jI=0,tO=new Map,Ar(),sO="sunday",d6="saturday"},"clear"),aze=o(function(t){ZI=t},"setAxisFormat"),sze=o(function(){return ZI},"getAxisFormat"),oze=o(function(t){JI=t},"setTickInterval"),lze=o(function(){return JI},"getTickInterval"),cze=o(function(t){eO=t},"setTodayMarker"),uze=o(function(){return eO},"getTodayMarker"),hze=o(function(t){Gc=t},"setDateFormat"),fze=o(function(){yb=!0},"enableInclusiveEndDates"),dze=o(function(){return yb},"endDatesAreInclusive"),pze=o(function(){aO=!0},"enableTopAxis"),mze=o(function(){return aO},"topAxisEnabled"),gze=o(function(t){nO=t},"setDisplayMode"),yze=o(function(){return nO},"getDisplayMode"),vze=o(function(){return Gc},"getDateFormat"),xze=o(function(t){mb=t.toLowerCase().split(/[\s,]+/)},"setIncludes"),bze=o(function(){return mb},"getIncludes"),wze=o(function(t){gb=t.toLowerCase().split(/[\s,]+/)},"setExcludes"),Tze=o(function(){return gb},"getExcludes"),kze=o(function(){return tO},"getLinks"),Eze=o(function(t){E1=t,rO.push(t)},"addSection"),Sze=o(function(){return rO},"getSections"),Cze=o(function(){let t=vue(),e=10,r=0;for(;!t&&r[\d\w- ]+)/.exec(r);if(i!==null){let s=null;for(let u of i.groups.ids.split(" ")){let h=Cp(u);h!==void 0&&(!s||h.endTime>s.endTime)&&(s=h)}if(s)return s.endTime;let l=new Date;return l.setHours(0,0,0,0),l}let a=(0,ho.default)(r,e.trim(),!0);if(a.isValid())return a.toDate();{Y.debug("Invalid date:"+r),Y.debug("With date format:"+e.trim());let s=new Date(r);if(s===void 0||isNaN(s.getTime())||s.getFullYear()<-1e4||s.getFullYear()>1e4)throw new Error("Invalid date:"+r);return s}},"getStartDate"),Cue=o(function(t){let e=/^(\d+(?:\.\d+)?)([Mdhmswy]|ms)$/.exec(t.trim());return e!==null?[Number.parseFloat(e[1]),e[2]]:[NaN,"ms"]},"parseDuration"),Aue=o(function(t,e,r,n=!1){r=r.trim();let a=/^until\s+(?[\d\w- ]+)/.exec(r);if(a!==null){let f=null;for(let p of a.groups.ids.split(" ")){let m=Cp(p);m!==void 0&&(!f||m.startTime{window.open(r,"_self")}),tO.set(n,r))}),Due(t,"clickable")},"setLink"),Due=o(function(t,e){t.split(",").forEach(function(r){let n=Cp(r);n!==void 0&&n.classes.push(e)})},"setClass"),Pze=o(function(t,e,r){if(me().securityLevel!=="loose"||e===void 0)return;let n=[];if(typeof r=="string"){n=r.split(/,(?=(?:(?:[^"]*"){2})*[^"]*$)/);for(let a=0;a{Gt.runFunc(e,...n)})},"setClickFun"),Lue=o(function(t,e){iO.push(function(){let r=document.querySelector(`[id="${t}"]`);r!==null&&r.addEventListener("click",function(){e()})},function(){let r=document.querySelector(`[id="${t}-text"]`);r!==null&&r.addEventListener("click",function(){e()})})},"pushFun"),Bze=o(function(t,e,r){t.split(",").forEach(function(n){Pze(n,e,r)}),Due(t,"clickable")},"setClickEvent"),Fze=o(function(t){iO.forEach(function(e){e(t)})},"bindFunctions"),Rue={getConfig:o(()=>me().gantt,"getConfig"),clear:ize,setDateFormat:hze,getDateFormat:vze,enableInclusiveEndDates:fze,endDatesAreInclusive:dze,enableTopAxis:pze,topAxisEnabled:mze,setAxisFormat:aze,getAxisFormat:sze,setTickInterval:oze,getTickInterval:lze,setTodayMarker:cze,getTodayMarker:uze,setAccTitle:Lr,getAccTitle:Rr,setDiagramTitle:$r,getDiagramTitle:Ir,setDisplayMode:gze,getDisplayMode:yze,setAccDescription:Nr,getAccDescription:Mr,addSection:Eze,getSections:Sze,getTasks:Cze,addTask:Mze,findTaskById:Cp,addTaskOrg:Ize,setIncludes:xze,getIncludes:bze,setExcludes:wze,getExcludes:Tze,setClickEvent:Bze,setLink:Oze,getLinks:kze,bindFunctions:Fze,parseDuration:Cue,isInvalidDate:Eue,setWeekday:Aze,getWeekday:_ze,setWeekend:Dze};o(Nue,"getTaskTags")});var p6,$ze,Iue,zze,Yu,Gze,Oue,Pue=N(()=>{"use strict";p6=Sa(R4(),1);vt();dr();gr();zt();Ei();$ze=o(function(){Y.debug("Something is calling, setConf, remove the call")},"setConf"),Iue={monday:Ch,tuesday:T5,wednesday:k5,thursday:oc,friday:E5,saturday:S5,sunday:yl},zze=o((t,e)=>{let r=[...t].map(()=>-1/0),n=[...t].sort((a,s)=>a.startTime-s.startTime||a.order-s.order),i=0;for(let a of n)for(let s=0;s=r[s]){r[s]=a.endTime,a.order=s+e,s>i&&(i=s);break}return i},"getMaxIntersections"),Gze=o(function(t,e,r,n){let i=me().gantt,a=me().securityLevel,s;a==="sandbox"&&(s=Ge("#i"+e));let l=a==="sandbox"?Ge(s.nodes()[0].contentDocument.body):Ge("body"),u=a==="sandbox"?s.nodes()[0].contentDocument:document,h=u.getElementById(e);Yu=h.parentElement.offsetWidth,Yu===void 0&&(Yu=1200),i.useWidth!==void 0&&(Yu=i.useWidth);let f=n.db.getTasks(),d=[];for(let S of f)d.push(S.type);d=A(d);let p={},m=2*i.topPadding;if(n.db.getDisplayMode()==="compact"||i.displayMode==="compact"){let S={};for(let I of f)S[I.section]===void 0?S[I.section]=[I]:S[I.section].push(I);let _=0;for(let I of Object.keys(S)){let D=zze(S[I],_)+1;_+=D,m+=D*(i.barHeight+i.barGap),p[I]=D}}else{m+=f.length*(i.barHeight+i.barGap);for(let S of d)p[S]=f.filter(_=>_.type===S).length}h.setAttribute("viewBox","0 0 "+Yu+" "+m);let g=l.select(`[id="${e}"]`),y=_5().domain([M3(f,function(S){return S.startTime}),N3(f,function(S){return S.endTime})]).rangeRound([0,Yu-i.leftPadding-i.rightPadding]);function v(S,_){let I=S.startTime,D=_.startTime,k=0;return I>D?k=1:Iz.order))].map(z=>S.find($=>$.order===z));g.append("g").selectAll("rect").data(M).enter().append("rect").attr("x",0).attr("y",function(z,$){return $=z.order,$*_+I-2}).attr("width",function(){return R-i.rightPadding/2}).attr("height",_).attr("class",function(z){for(let[$,H]of d.entries())if(z.type===H)return"section section"+$%i.numberSectionStyles;return"section section0"});let B=g.append("g").selectAll("rect").data(S).enter(),F=n.db.getLinks();if(B.append("rect").attr("id",function(z){return z.id}).attr("rx",3).attr("ry",3).attr("x",function(z){return z.milestone?y(z.startTime)+D+.5*(y(z.endTime)-y(z.startTime))-.5*k:y(z.startTime)+D}).attr("y",function(z,$){return $=z.order,$*_+I}).attr("width",function(z){return z.milestone?k:y(z.renderEndTime||z.endTime)-y(z.startTime)}).attr("height",k).attr("transform-origin",function(z,$){return $=z.order,(y(z.startTime)+D+.5*(y(z.endTime)-y(z.startTime))).toString()+"px "+($*_+I+.5*k).toString()+"px"}).attr("class",function(z){let $="task",H="";z.classes.length>0&&(H=z.classes.join(" "));let Q=0;for(let[ie,ne]of d.entries())z.type===ne&&(Q=ie%i.numberSectionStyles);let j="";return z.active?z.crit?j+=" activeCrit":j=" active":z.done?z.crit?j=" doneCrit":j=" done":z.crit&&(j+=" crit"),j.length===0&&(j=" task"),z.milestone&&(j=" milestone "+j),j+=Q,j+=" "+H,$+j}),B.append("text").attr("id",function(z){return z.id+"-text"}).text(function(z){return z.task}).attr("font-size",i.fontSize).attr("x",function(z){let $=y(z.startTime),H=y(z.renderEndTime||z.endTime);z.milestone&&($+=.5*(y(z.endTime)-y(z.startTime))-.5*k),z.milestone&&(H=$+k);let Q=this.getBBox().width;return Q>H-$?H+Q+1.5*i.leftPadding>R?$+D-5:H+D+5:(H-$)/2+$+D}).attr("y",function(z,$){return $=z.order,$*_+i.barHeight/2+(i.fontSize/2-2)+I}).attr("text-height",k).attr("class",function(z){let $=y(z.startTime),H=y(z.endTime);z.milestone&&(H=$+k);let Q=this.getBBox().width,j="";z.classes.length>0&&(j=z.classes.join(" "));let ie=0;for(let[le,he]of d.entries())z.type===he&&(ie=le%i.numberSectionStyles);let ne="";return z.active&&(z.crit?ne="activeCritText"+ie:ne="activeText"+ie),z.done?z.crit?ne=ne+" doneCritText"+ie:ne=ne+" doneText"+ie:z.crit&&(ne=ne+" critText"+ie),z.milestone&&(ne+=" milestoneText"),Q>H-$?H+Q+1.5*i.leftPadding>R?j+" taskTextOutsideLeft taskTextOutside"+ie+" "+ne:j+" taskTextOutsideRight taskTextOutside"+ie+" "+ne+" width-"+Q:j+" taskText taskText"+ie+" "+ne+" width-"+Q}),me().securityLevel==="sandbox"){let z;z=Ge("#i"+e);let $=z.nodes()[0].contentDocument;B.filter(function(H){return F.has(H.id)}).each(function(H){var Q=$.querySelector("#"+H.id),j=$.querySelector("#"+H.id+"-text");let ie=Q.parentNode;var ne=$.createElement("a");ne.setAttribute("xlink:href",F.get(H.id)),ne.setAttribute("target","_top"),ie.appendChild(ne),ne.appendChild(Q),ne.appendChild(j)})}}o(b,"drawRects");function w(S,_,I,D,k,L,R,O){if(R.length===0&&O.length===0)return;let M,B;for(let{startTime:Q,endTime:j}of L)(M===void 0||QB)&&(B=j);if(!M||!B)return;if((0,p6.default)(B).diff((0,p6.default)(M),"year")>5){Y.warn("The difference between the min and max time is more than 5 years. This will cause performance issues. Skipping drawing exclude days.");return}let F=n.db.getDateFormat(),P=[],z=null,$=(0,p6.default)(M);for(;$.valueOf()<=B;)n.db.isInvalidDate($,F,R,O)?z?z.end=$:z={start:$,end:$}:z&&(P.push(z),z=null),$=$.add(1,"d");g.append("g").selectAll("rect").data(P).enter().append("rect").attr("id",function(Q){return"exclude-"+Q.start.format("YYYY-MM-DD")}).attr("x",function(Q){return y(Q.start)+I}).attr("y",i.gridLineStartPadding).attr("width",function(Q){let j=Q.end.add(1,"day");return y(j)-y(Q.start)}).attr("height",k-_-i.gridLineStartPadding).attr("transform-origin",function(Q,j){return(y(Q.start)+I+.5*(y(Q.end)-y(Q.start))).toString()+"px "+(j*S+.5*k).toString()+"px"}).attr("class","exclude-range")}o(w,"drawExcludeDays");function C(S,_,I,D){let k=bA(y).tickSize(-D+_+i.gridLineStartPadding).tickFormat(wd(n.db.getAxisFormat()||i.axisFormat||"%Y-%m-%d")),R=/^([1-9]\d*)(millisecond|second|minute|hour|day|week|month)$/.exec(n.db.getTickInterval()||i.tickInterval);if(R!==null){let O=R[1],M=R[2],B=n.db.getWeekday()||i.weekday;switch(M){case"millisecond":k.ticks(ac.every(O));break;case"second":k.ticks(Ks.every(O));break;case"minute":k.ticks(vu.every(O));break;case"hour":k.ticks(xu.every(O));break;case"day":k.ticks(_o.every(O));break;case"week":k.ticks(Iue[B].every(O));break;case"month":k.ticks(bu.every(O));break}}if(g.append("g").attr("class","grid").attr("transform","translate("+S+", "+(D-50)+")").call(k).selectAll("text").style("text-anchor","middle").attr("fill","#000").attr("stroke","none").attr("font-size",10).attr("dy","1em"),n.db.topAxisEnabled()||i.topAxis){let O=xA(y).tickSize(-D+_+i.gridLineStartPadding).tickFormat(wd(n.db.getAxisFormat()||i.axisFormat||"%Y-%m-%d"));if(R!==null){let M=R[1],B=R[2],F=n.db.getWeekday()||i.weekday;switch(B){case"millisecond":O.ticks(ac.every(M));break;case"second":O.ticks(Ks.every(M));break;case"minute":O.ticks(vu.every(M));break;case"hour":O.ticks(xu.every(M));break;case"day":O.ticks(_o.every(M));break;case"week":O.ticks(Iue[F].every(M));break;case"month":O.ticks(bu.every(M));break}}g.append("g").attr("class","grid").attr("transform","translate("+S+", "+_+")").call(O).selectAll("text").style("text-anchor","middle").attr("fill","#000").attr("stroke","none").attr("font-size",10)}}o(C,"makeGrid");function T(S,_){let I=0,D=Object.keys(p).map(k=>[k,p[k]]);g.append("g").selectAll("text").data(D).enter().append(function(k){let L=k[0].split(Ze.lineBreakRegex),R=-(L.length-1)/2,O=u.createElementNS("http://www.w3.org/2000/svg","text");O.setAttribute("dy",R+"em");for(let[M,B]of L.entries()){let F=u.createElementNS("http://www.w3.org/2000/svg","tspan");F.setAttribute("alignment-baseline","central"),F.setAttribute("x","10"),M>0&&F.setAttribute("dy","1em"),F.textContent=B,O.appendChild(F)}return O}).attr("x",10).attr("y",function(k,L){if(L>0)for(let R=0;R{"use strict";Vze=o(t=>` + .mermaid-main-font { + font-family: ${t.fontFamily}; + } + + .exclude-range { + fill: ${t.excludeBkgColor}; + } + + .section { + stroke: none; + opacity: 0.2; + } + + .section0 { + fill: ${t.sectionBkgColor}; + } + + .section2 { + fill: ${t.sectionBkgColor2}; + } + + .section1, + .section3 { + fill: ${t.altSectionBkgColor}; + opacity: 0.2; + } + + .sectionTitle0 { + fill: ${t.titleColor}; + } + + .sectionTitle1 { + fill: ${t.titleColor}; + } + + .sectionTitle2 { + fill: ${t.titleColor}; + } + + .sectionTitle3 { + fill: ${t.titleColor}; + } + + .sectionTitle { + text-anchor: start; + font-family: ${t.fontFamily}; + } + + + /* Grid and axis */ + + .grid .tick { + stroke: ${t.gridColor}; + opacity: 0.8; + shape-rendering: crispEdges; + } + + .grid .tick text { + font-family: ${t.fontFamily}; + fill: ${t.textColor}; + } + + .grid path { + stroke-width: 0; + } + + + /* Today line */ + + .today { + fill: none; + stroke: ${t.todayLineColor}; + stroke-width: 2px; + } + + + /* Task styling */ + + /* Default task */ + + .task { + stroke-width: 2; + } + + .taskText { + text-anchor: middle; + font-family: ${t.fontFamily}; + } + + .taskTextOutsideRight { + fill: ${t.taskTextDarkColor}; + text-anchor: start; + font-family: ${t.fontFamily}; + } + + .taskTextOutsideLeft { + fill: ${t.taskTextDarkColor}; + text-anchor: end; + } + + + /* Special case clickable */ + + .task.clickable { + cursor: pointer; + } + + .taskText.clickable { + cursor: pointer; + fill: ${t.taskTextClickableColor} !important; + font-weight: bold; + } + + .taskTextOutsideLeft.clickable { + cursor: pointer; + fill: ${t.taskTextClickableColor} !important; + font-weight: bold; + } + + .taskTextOutsideRight.clickable { + cursor: pointer; + fill: ${t.taskTextClickableColor} !important; + font-weight: bold; + } + + + /* Specific task settings for the sections*/ + + .taskText0, + .taskText1, + .taskText2, + .taskText3 { + fill: ${t.taskTextColor}; + } + + .task0, + .task1, + .task2, + .task3 { + fill: ${t.taskBkgColor}; + stroke: ${t.taskBorderColor}; + } + + .taskTextOutside0, + .taskTextOutside2 + { + fill: ${t.taskTextOutsideColor}; + } + + .taskTextOutside1, + .taskTextOutside3 { + fill: ${t.taskTextOutsideColor}; + } + + + /* Active task */ + + .active0, + .active1, + .active2, + .active3 { + fill: ${t.activeTaskBkgColor}; + stroke: ${t.activeTaskBorderColor}; + } + + .activeText0, + .activeText1, + .activeText2, + .activeText3 { + fill: ${t.taskTextDarkColor} !important; + } + + + /* Completed task */ + + .done0, + .done1, + .done2, + .done3 { + stroke: ${t.doneTaskBorderColor}; + fill: ${t.doneTaskBkgColor}; + stroke-width: 2; + } + + .doneText0, + .doneText1, + .doneText2, + .doneText3 { + fill: ${t.taskTextDarkColor} !important; + } + + + /* Tasks on the critical line */ + + .crit0, + .crit1, + .crit2, + .crit3 { + stroke: ${t.critBorderColor}; + fill: ${t.critBkgColor}; + stroke-width: 2; + } + + .activeCrit0, + .activeCrit1, + .activeCrit2, + .activeCrit3 { + stroke: ${t.critBorderColor}; + fill: ${t.activeTaskBkgColor}; + stroke-width: 2; + } + + .doneCrit0, + .doneCrit1, + .doneCrit2, + .doneCrit3 { + stroke: ${t.critBorderColor}; + fill: ${t.doneTaskBkgColor}; + stroke-width: 2; + cursor: pointer; + shape-rendering: crispEdges; + } + + .milestone { + transform: rotate(45deg) scale(0.8,0.8); + } + + .milestoneText { + font-style: italic; + } + .doneCritText0, + .doneCritText1, + .doneCritText2, + .doneCritText3 { + fill: ${t.taskTextDarkColor} !important; + } + + .activeCritText0, + .activeCritText1, + .activeCritText2, + .activeCritText3 { + fill: ${t.taskTextDarkColor} !important; + } + + .titleText { + text-anchor: middle; + font-size: 18px; + fill: ${t.titleColor||t.textColor}; + font-family: ${t.fontFamily}; + } +`,"getStyles"),Bue=Vze});var $ue={};hr($ue,{diagram:()=>Uze});var Uze,zue=N(()=>{"use strict";due();Mue();Pue();Fue();Uze={parser:fue,db:Rue,renderer:Oue,styles:Bue}});var Uue,Hue=N(()=>{"use strict";kp();vt();Uue={parse:o(async t=>{let e=await uo("info",t);Y.debug(e)},"parse")}});var vb,oO=N(()=>{vb={name:"mermaid",version:"11.6.0",description:"Markdown-ish syntax for generating flowcharts, mindmaps, sequence diagrams, class diagrams, gantt charts, git graphs and more.",type:"module",module:"./dist/mermaid.core.mjs",types:"./dist/mermaid.d.ts",exports:{".":{types:"./dist/mermaid.d.ts",import:"./dist/mermaid.core.mjs",default:"./dist/mermaid.core.mjs"},"./*":"./*"},keywords:["diagram","markdown","flowchart","sequence diagram","gantt","class diagram","git graph","mindmap","packet diagram","c4 diagram","er diagram","pie chart","pie diagram","quadrant chart","requirement diagram","graph"],scripts:{clean:"rimraf dist",dev:"pnpm -w dev","docs:code":"typedoc src/defaultConfig.ts src/config.ts src/mermaid.ts && prettier --write ./src/docs/config/setup","docs:build":"rimraf ../../docs && pnpm docs:code && pnpm docs:spellcheck && tsx scripts/docs.cli.mts","docs:verify":"pnpm docs:code && pnpm docs:spellcheck && tsx scripts/docs.cli.mts --verify","docs:pre:vitepress":"pnpm --filter ./src/docs prefetch && rimraf src/vitepress && pnpm docs:code && tsx scripts/docs.cli.mts --vitepress && pnpm --filter ./src/vitepress install --no-frozen-lockfile --ignore-scripts","docs:build:vitepress":"pnpm docs:pre:vitepress && (cd src/vitepress && pnpm run build) && cpy --flat src/docs/landing/ ./src/vitepress/.vitepress/dist/landing","docs:dev":'pnpm docs:pre:vitepress && concurrently "pnpm --filter ./src/vitepress dev" "tsx scripts/docs.cli.mts --watch --vitepress"',"docs:dev:docker":'pnpm docs:pre:vitepress && concurrently "pnpm --filter ./src/vitepress dev:docker" "tsx scripts/docs.cli.mts --watch --vitepress"',"docs:serve":"pnpm docs:build:vitepress && vitepress serve src/vitepress","docs:spellcheck":'cspell "src/docs/**/*.md"',"docs:release-version":"tsx scripts/update-release-version.mts","docs:verify-version":"tsx scripts/update-release-version.mts --verify","types:build-config":"tsx scripts/create-types-from-json-schema.mts","types:verify-config":"tsx scripts/create-types-from-json-schema.mts --verify",checkCircle:"npx madge --circular ./src",prepublishOnly:"pnpm docs:verify-version"},repository:{type:"git",url:"https://github.com/mermaid-js/mermaid"},author:"Knut Sveidqvist",license:"MIT",standard:{ignore:["**/parser/*.js","dist/**/*.js","cypress/**/*.js"],globals:["page"]},dependencies:{"@braintree/sanitize-url":"^7.0.4","@iconify/utils":"^2.1.33","@mermaid-js/parser":"workspace:^","@types/d3":"^7.4.3",cytoscape:"^3.29.3","cytoscape-cose-bilkent":"^4.1.0","cytoscape-fcose":"^2.2.0",d3:"^7.9.0","d3-sankey":"^0.12.3","dagre-d3-es":"7.0.11",dayjs:"^1.11.13",dompurify:"^3.2.4",katex:"^0.16.9",khroma:"^2.1.0","lodash-es":"^4.17.21",marked:"^15.0.7",roughjs:"^4.6.6",stylis:"^4.3.6","ts-dedent":"^2.2.0",uuid:"^11.1.0"},devDependencies:{"@adobe/jsonschema2md":"^8.0.2","@iconify/types":"^2.0.0","@types/cytoscape":"^3.21.9","@types/cytoscape-fcose":"^2.2.4","@types/d3-sankey":"^0.12.4","@types/d3-scale":"^4.0.9","@types/d3-scale-chromatic":"^3.1.0","@types/d3-selection":"^3.0.11","@types/d3-shape":"^3.1.7","@types/jsdom":"^21.1.7","@types/katex":"^0.16.7","@types/lodash-es":"^4.17.12","@types/micromatch":"^4.0.9","@types/stylis":"^4.2.7","@types/uuid":"^10.0.0",ajv:"^8.17.1",chokidar:"^4.0.3",concurrently:"^9.1.2","csstree-validator":"^4.0.1",globby:"^14.0.2",jison:"^0.4.18","js-base64":"^3.7.7",jsdom:"^26.0.0","json-schema-to-typescript":"^15.0.4",micromatch:"^4.0.8","path-browserify":"^1.0.1",prettier:"^3.5.2",remark:"^15.0.1","remark-frontmatter":"^5.0.0","remark-gfm":"^4.0.1",rimraf:"^6.0.1","start-server-and-test":"^2.0.10","type-fest":"^4.35.0",typedoc:"^0.27.8","typedoc-plugin-markdown":"^4.4.2",typescript:"~5.7.3","unist-util-flatmap":"^1.0.0","unist-util-visit":"^5.0.0",vitepress:"^1.0.2","vitepress-plugin-search":"1.0.4-alpha.22"},files:["dist/","README.md"],publishConfig:{access:"public"}}});var Xze,jze,Wue,que=N(()=>{"use strict";oO();Xze={version:vb.version},jze=o(()=>Xze.version,"getVersion"),Wue={getVersion:jze}});var sa,Vc=N(()=>{"use strict";dr();zt();sa=o(t=>{let{securityLevel:e}=me(),r=Ge("body");if(e==="sandbox"){let a=Ge(`#i${t}`).node()?.contentDocument??document;r=Ge(a.body)}return r.select(`#${t}`)},"selectSvgElement")});var Kze,Yue,Xue=N(()=>{"use strict";vt();Vc();Ei();Kze=o((t,e,r)=>{Y.debug(`rendering info diagram +`+t);let n=sa(e);vn(n,100,400,!0),n.append("g").append("text").attr("x",100).attr("y",40).attr("class","version").attr("font-size",32).style("text-anchor","middle").text(`v${r}`)},"draw"),Yue={draw:Kze}});var jue={};hr(jue,{diagram:()=>Qze});var Qze,Kue=N(()=>{"use strict";Hue();que();Xue();Qze={parser:Uue,db:Wue,renderer:Yue}});var Jue,lO,m6,cO,eGe,tGe,rGe,nGe,iGe,aGe,sGe,g6,uO=N(()=>{"use strict";vt();mi();Ya();Jue=or.pie,lO={sections:new Map,showData:!1,config:Jue},m6=lO.sections,cO=lO.showData,eGe=structuredClone(Jue),tGe=o(()=>structuredClone(eGe),"getConfig"),rGe=o(()=>{m6=new Map,cO=lO.showData,Ar()},"clear"),nGe=o(({label:t,value:e})=>{m6.has(t)||(m6.set(t,e),Y.debug(`added new section: ${t}, with value: ${e}`))},"addSection"),iGe=o(()=>m6,"getSections"),aGe=o(t=>{cO=t},"setShowData"),sGe=o(()=>cO,"getShowData"),g6={getConfig:tGe,clear:rGe,setDiagramTitle:$r,getDiagramTitle:Ir,setAccTitle:Lr,getAccTitle:Rr,setAccDescription:Nr,getAccDescription:Mr,addSection:nGe,getSections:iGe,setShowData:aGe,getShowData:sGe}});var oGe,ehe,the=N(()=>{"use strict";kp();vt();T1();uO();oGe=o((t,e)=>{$c(t,e),e.setShowData(t.showData),t.sections.map(e.addSection)},"populateDb"),ehe={parse:o(async t=>{let e=await uo("pie",t);Y.debug(e),oGe(e,g6)},"parse")}});var lGe,rhe,nhe=N(()=>{"use strict";lGe=o(t=>` + .pieCircle{ + stroke: ${t.pieStrokeColor}; + stroke-width : ${t.pieStrokeWidth}; + opacity : ${t.pieOpacity}; + } + .pieOuterCircle{ + stroke: ${t.pieOuterStrokeColor}; + stroke-width: ${t.pieOuterStrokeWidth}; + fill: none; + } + .pieTitleText { + text-anchor: middle; + font-size: ${t.pieTitleTextSize}; + fill: ${t.pieTitleTextColor}; + font-family: ${t.fontFamily}; + } + .slice { + font-family: ${t.fontFamily}; + fill: ${t.pieSectionTextColor}; + font-size:${t.pieSectionTextSize}; + // fill: white; + } + .legend text { + fill: ${t.pieLegendTextColor}; + font-family: ${t.fontFamily}; + font-size: ${t.pieLegendTextSize}; + } +`,"getStyles"),rhe=lGe});var cGe,uGe,ihe,ahe=N(()=>{"use strict";dr();zt();vt();Vc();Ei();ir();cGe=o(t=>{let e=[...t.entries()].map(n=>({label:n[0],value:n[1]})).sort((n,i)=>i.value-n.value);return I5().value(n=>n.value)(e)},"createPieArcs"),uGe=o((t,e,r,n)=>{Y.debug(`rendering pie chart +`+t);let i=n.db,a=me(),s=Fi(i.getConfig(),a.pie),l=40,u=18,h=4,f=450,d=f,p=sa(e),m=p.append("g");m.attr("transform","translate("+d/2+","+f/2+")");let{themeVariables:g}=a,[y]=Bo(g.pieOuterStrokeWidth);y??=2;let v=s.textPosition,x=Math.min(d,f)/2-l,b=bl().innerRadius(0).outerRadius(x),w=bl().innerRadius(x*v).outerRadius(x*v);m.append("circle").attr("cx",0).attr("cy",0).attr("r",x+y/2).attr("class","pieOuterCircle");let C=i.getSections(),T=cGe(C),E=[g.pie1,g.pie2,g.pie3,g.pie4,g.pie5,g.pie6,g.pie7,g.pie8,g.pie9,g.pie10,g.pie11,g.pie12],A=gu(E);m.selectAll("mySlices").data(T).enter().append("path").attr("d",b).attr("fill",k=>A(k.data.label)).attr("class","pieCircle");let S=0;C.forEach(k=>{S+=k}),m.selectAll("mySlices").data(T).enter().append("text").text(k=>(k.data.value/S*100).toFixed(0)+"%").attr("transform",k=>"translate("+w.centroid(k)+")").style("text-anchor","middle").attr("class","slice"),m.append("text").text(i.getDiagramTitle()).attr("x",0).attr("y",-(f-50)/2).attr("class","pieTitleText");let _=m.selectAll(".legend").data(A.domain()).enter().append("g").attr("class","legend").attr("transform",(k,L)=>{let R=u+h,O=R*A.domain().length/2,M=12*u,B=L*R-O;return"translate("+M+","+B+")"});_.append("rect").attr("width",u).attr("height",u).style("fill",A).style("stroke",A),_.data(T).append("text").attr("x",u+h).attr("y",u-h).text(k=>{let{label:L,value:R}=k.data;return i.getShowData()?`${L} [${R}]`:L});let I=Math.max(..._.selectAll("text").nodes().map(k=>k?.getBoundingClientRect().width??0)),D=d+l+u+h+I;p.attr("viewBox",`0 0 ${D} ${f}`),vn(p,f,D,s.useMaxWidth)},"draw"),ihe={draw:uGe}});var she={};hr(she,{diagram:()=>hGe});var hGe,ohe=N(()=>{"use strict";the();uO();nhe();ahe();hGe={parser:ehe,db:g6,renderer:ihe,styles:rhe}});var hO,uhe,hhe=N(()=>{"use strict";hO=function(){var t=o(function(xe,q,pe,ve){for(pe=pe||{},ve=xe.length;ve--;pe[xe[ve]]=q);return pe},"o"),e=[1,3],r=[1,4],n=[1,5],i=[1,6],a=[1,7],s=[1,4,5,10,12,13,14,18,25,35,37,39,41,42,48,50,51,52,53,54,55,56,57,60,61,63,64,65,66,67],l=[1,4,5,10,12,13,14,18,25,28,35,37,39,41,42,48,50,51,52,53,54,55,56,57,60,61,63,64,65,66,67],u=[55,56,57],h=[2,36],f=[1,37],d=[1,36],p=[1,38],m=[1,35],g=[1,43],y=[1,41],v=[1,14],x=[1,23],b=[1,18],w=[1,19],C=[1,20],T=[1,21],E=[1,22],A=[1,24],S=[1,25],_=[1,26],I=[1,27],D=[1,28],k=[1,29],L=[1,32],R=[1,33],O=[1,34],M=[1,39],B=[1,40],F=[1,42],P=[1,44],z=[1,62],$=[1,61],H=[4,5,8,10,12,13,14,18,44,47,49,55,56,57,63,64,65,66,67],Q=[1,65],j=[1,66],ie=[1,67],ne=[1,68],le=[1,69],he=[1,70],K=[1,71],X=[1,72],te=[1,73],J=[1,74],se=[1,75],ue=[1,76],Z=[4,5,6,7,8,9,10,11,12,13,14,15,18],Se=[1,90],ce=[1,91],ae=[1,92],Oe=[1,99],ge=[1,93],ze=[1,96],He=[1,94],$e=[1,95],Re=[1,97],Ie=[1,98],be=[1,102],W=[10,55,56,57],de=[4,5,6,8,10,11,13,17,18,19,20,55,56,57],re={trace:o(function(){},"trace"),yy:{},symbols_:{error:2,idStringToken:3,ALPHA:4,NUM:5,NODE_STRING:6,DOWN:7,MINUS:8,DEFAULT:9,COMMA:10,COLON:11,AMP:12,BRKT:13,MULT:14,UNICODE_TEXT:15,styleComponent:16,UNIT:17,SPACE:18,STYLE:19,PCT:20,idString:21,style:22,stylesOpt:23,classDefStatement:24,CLASSDEF:25,start:26,eol:27,QUADRANT:28,document:29,line:30,statement:31,axisDetails:32,quadrantDetails:33,points:34,title:35,title_value:36,acc_title:37,acc_title_value:38,acc_descr:39,acc_descr_value:40,acc_descr_multiline_value:41,section:42,text:43,point_start:44,point_x:45,point_y:46,class_name:47,"X-AXIS":48,"AXIS-TEXT-DELIMITER":49,"Y-AXIS":50,QUADRANT_1:51,QUADRANT_2:52,QUADRANT_3:53,QUADRANT_4:54,NEWLINE:55,SEMI:56,EOF:57,alphaNumToken:58,textNoTagsToken:59,STR:60,MD_STR:61,alphaNum:62,PUNCTUATION:63,PLUS:64,EQUALS:65,DOT:66,UNDERSCORE:67,$accept:0,$end:1},terminals_:{2:"error",4:"ALPHA",5:"NUM",6:"NODE_STRING",7:"DOWN",8:"MINUS",9:"DEFAULT",10:"COMMA",11:"COLON",12:"AMP",13:"BRKT",14:"MULT",15:"UNICODE_TEXT",17:"UNIT",18:"SPACE",19:"STYLE",20:"PCT",25:"CLASSDEF",28:"QUADRANT",35:"title",36:"title_value",37:"acc_title",38:"acc_title_value",39:"acc_descr",40:"acc_descr_value",41:"acc_descr_multiline_value",42:"section",44:"point_start",45:"point_x",46:"point_y",47:"class_name",48:"X-AXIS",49:"AXIS-TEXT-DELIMITER",50:"Y-AXIS",51:"QUADRANT_1",52:"QUADRANT_2",53:"QUADRANT_3",54:"QUADRANT_4",55:"NEWLINE",56:"SEMI",57:"EOF",60:"STR",61:"MD_STR",63:"PUNCTUATION",64:"PLUS",65:"EQUALS",66:"DOT",67:"UNDERSCORE"},productions_:[0,[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[16,1],[16,1],[16,1],[16,1],[16,1],[16,1],[16,1],[16,1],[16,1],[16,1],[21,1],[21,2],[22,1],[22,2],[23,1],[23,3],[24,5],[26,2],[26,2],[26,2],[29,0],[29,2],[30,2],[31,0],[31,1],[31,2],[31,1],[31,1],[31,1],[31,2],[31,2],[31,2],[31,1],[31,1],[34,4],[34,5],[34,5],[34,6],[32,4],[32,3],[32,2],[32,4],[32,3],[32,2],[33,2],[33,2],[33,2],[33,2],[27,1],[27,1],[27,1],[43,1],[43,2],[43,1],[43,1],[62,1],[62,2],[58,1],[58,1],[58,1],[58,1],[58,1],[58,1],[58,1],[58,1],[58,1],[58,1],[58,1],[59,1],[59,1],[59,1]],performAction:o(function(q,pe,ve,Pe,_e,we,Ve){var De=we.length-1;switch(_e){case 23:this.$=we[De];break;case 24:this.$=we[De-1]+""+we[De];break;case 26:this.$=we[De-1]+we[De];break;case 27:this.$=[we[De].trim()];break;case 28:we[De-2].push(we[De].trim()),this.$=we[De-2];break;case 29:this.$=we[De-4],Pe.addClass(we[De-2],we[De]);break;case 37:this.$=[];break;case 42:this.$=we[De].trim(),Pe.setDiagramTitle(this.$);break;case 43:this.$=we[De].trim(),Pe.setAccTitle(this.$);break;case 44:case 45:this.$=we[De].trim(),Pe.setAccDescription(this.$);break;case 46:Pe.addSection(we[De].substr(8)),this.$=we[De].substr(8);break;case 47:Pe.addPoint(we[De-3],"",we[De-1],we[De],[]);break;case 48:Pe.addPoint(we[De-4],we[De-3],we[De-1],we[De],[]);break;case 49:Pe.addPoint(we[De-4],"",we[De-2],we[De-1],we[De]);break;case 50:Pe.addPoint(we[De-5],we[De-4],we[De-2],we[De-1],we[De]);break;case 51:Pe.setXAxisLeftText(we[De-2]),Pe.setXAxisRightText(we[De]);break;case 52:we[De-1].text+=" \u27F6 ",Pe.setXAxisLeftText(we[De-1]);break;case 53:Pe.setXAxisLeftText(we[De]);break;case 54:Pe.setYAxisBottomText(we[De-2]),Pe.setYAxisTopText(we[De]);break;case 55:we[De-1].text+=" \u27F6 ",Pe.setYAxisBottomText(we[De-1]);break;case 56:Pe.setYAxisBottomText(we[De]);break;case 57:Pe.setQuadrant1Text(we[De]);break;case 58:Pe.setQuadrant2Text(we[De]);break;case 59:Pe.setQuadrant3Text(we[De]);break;case 60:Pe.setQuadrant4Text(we[De]);break;case 64:this.$={text:we[De],type:"text"};break;case 65:this.$={text:we[De-1].text+""+we[De],type:we[De-1].type};break;case 66:this.$={text:we[De],type:"text"};break;case 67:this.$={text:we[De],type:"markdown"};break;case 68:this.$=we[De];break;case 69:this.$=we[De-1]+""+we[De];break}},"anonymous"),table:[{18:e,26:1,27:2,28:r,55:n,56:i,57:a},{1:[3]},{18:e,26:8,27:2,28:r,55:n,56:i,57:a},{18:e,26:9,27:2,28:r,55:n,56:i,57:a},t(s,[2,33],{29:10}),t(l,[2,61]),t(l,[2,62]),t(l,[2,63]),{1:[2,30]},{1:[2,31]},t(u,h,{30:11,31:12,24:13,32:15,33:16,34:17,43:30,58:31,1:[2,32],4:f,5:d,10:p,12:m,13:g,14:y,18:v,25:x,35:b,37:w,39:C,41:T,42:E,48:A,50:S,51:_,52:I,53:D,54:k,60:L,61:R,63:O,64:M,65:B,66:F,67:P}),t(s,[2,34]),{27:45,55:n,56:i,57:a},t(u,[2,37]),t(u,h,{24:13,32:15,33:16,34:17,43:30,58:31,31:46,4:f,5:d,10:p,12:m,13:g,14:y,18:v,25:x,35:b,37:w,39:C,41:T,42:E,48:A,50:S,51:_,52:I,53:D,54:k,60:L,61:R,63:O,64:M,65:B,66:F,67:P}),t(u,[2,39]),t(u,[2,40]),t(u,[2,41]),{36:[1,47]},{38:[1,48]},{40:[1,49]},t(u,[2,45]),t(u,[2,46]),{18:[1,50]},{4:f,5:d,10:p,12:m,13:g,14:y,43:51,58:31,60:L,61:R,63:O,64:M,65:B,66:F,67:P},{4:f,5:d,10:p,12:m,13:g,14:y,43:52,58:31,60:L,61:R,63:O,64:M,65:B,66:F,67:P},{4:f,5:d,10:p,12:m,13:g,14:y,43:53,58:31,60:L,61:R,63:O,64:M,65:B,66:F,67:P},{4:f,5:d,10:p,12:m,13:g,14:y,43:54,58:31,60:L,61:R,63:O,64:M,65:B,66:F,67:P},{4:f,5:d,10:p,12:m,13:g,14:y,43:55,58:31,60:L,61:R,63:O,64:M,65:B,66:F,67:P},{4:f,5:d,10:p,12:m,13:g,14:y,43:56,58:31,60:L,61:R,63:O,64:M,65:B,66:F,67:P},{4:f,5:d,8:z,10:p,12:m,13:g,14:y,18:$,44:[1,57],47:[1,58],58:60,59:59,63:O,64:M,65:B,66:F,67:P},t(H,[2,64]),t(H,[2,66]),t(H,[2,67]),t(H,[2,70]),t(H,[2,71]),t(H,[2,72]),t(H,[2,73]),t(H,[2,74]),t(H,[2,75]),t(H,[2,76]),t(H,[2,77]),t(H,[2,78]),t(H,[2,79]),t(H,[2,80]),t(s,[2,35]),t(u,[2,38]),t(u,[2,42]),t(u,[2,43]),t(u,[2,44]),{3:64,4:Q,5:j,6:ie,7:ne,8:le,9:he,10:K,11:X,12:te,13:J,14:se,15:ue,21:63},t(u,[2,53],{59:59,58:60,4:f,5:d,8:z,10:p,12:m,13:g,14:y,18:$,49:[1,77],63:O,64:M,65:B,66:F,67:P}),t(u,[2,56],{59:59,58:60,4:f,5:d,8:z,10:p,12:m,13:g,14:y,18:$,49:[1,78],63:O,64:M,65:B,66:F,67:P}),t(u,[2,57],{59:59,58:60,4:f,5:d,8:z,10:p,12:m,13:g,14:y,18:$,63:O,64:M,65:B,66:F,67:P}),t(u,[2,58],{59:59,58:60,4:f,5:d,8:z,10:p,12:m,13:g,14:y,18:$,63:O,64:M,65:B,66:F,67:P}),t(u,[2,59],{59:59,58:60,4:f,5:d,8:z,10:p,12:m,13:g,14:y,18:$,63:O,64:M,65:B,66:F,67:P}),t(u,[2,60],{59:59,58:60,4:f,5:d,8:z,10:p,12:m,13:g,14:y,18:$,63:O,64:M,65:B,66:F,67:P}),{45:[1,79]},{44:[1,80]},t(H,[2,65]),t(H,[2,81]),t(H,[2,82]),t(H,[2,83]),{3:82,4:Q,5:j,6:ie,7:ne,8:le,9:he,10:K,11:X,12:te,13:J,14:se,15:ue,18:[1,81]},t(Z,[2,23]),t(Z,[2,1]),t(Z,[2,2]),t(Z,[2,3]),t(Z,[2,4]),t(Z,[2,5]),t(Z,[2,6]),t(Z,[2,7]),t(Z,[2,8]),t(Z,[2,9]),t(Z,[2,10]),t(Z,[2,11]),t(Z,[2,12]),t(u,[2,52],{58:31,43:83,4:f,5:d,10:p,12:m,13:g,14:y,60:L,61:R,63:O,64:M,65:B,66:F,67:P}),t(u,[2,55],{58:31,43:84,4:f,5:d,10:p,12:m,13:g,14:y,60:L,61:R,63:O,64:M,65:B,66:F,67:P}),{46:[1,85]},{45:[1,86]},{4:Se,5:ce,6:ae,8:Oe,11:ge,13:ze,16:89,17:He,18:$e,19:Re,20:Ie,22:88,23:87},t(Z,[2,24]),t(u,[2,51],{59:59,58:60,4:f,5:d,8:z,10:p,12:m,13:g,14:y,18:$,63:O,64:M,65:B,66:F,67:P}),t(u,[2,54],{59:59,58:60,4:f,5:d,8:z,10:p,12:m,13:g,14:y,18:$,63:O,64:M,65:B,66:F,67:P}),t(u,[2,47],{22:88,16:89,23:100,4:Se,5:ce,6:ae,8:Oe,11:ge,13:ze,17:He,18:$e,19:Re,20:Ie}),{46:[1,101]},t(u,[2,29],{10:be}),t(W,[2,27],{16:103,4:Se,5:ce,6:ae,8:Oe,11:ge,13:ze,17:He,18:$e,19:Re,20:Ie}),t(de,[2,25]),t(de,[2,13]),t(de,[2,14]),t(de,[2,15]),t(de,[2,16]),t(de,[2,17]),t(de,[2,18]),t(de,[2,19]),t(de,[2,20]),t(de,[2,21]),t(de,[2,22]),t(u,[2,49],{10:be}),t(u,[2,48],{22:88,16:89,23:104,4:Se,5:ce,6:ae,8:Oe,11:ge,13:ze,17:He,18:$e,19:Re,20:Ie}),{4:Se,5:ce,6:ae,8:Oe,11:ge,13:ze,16:89,17:He,18:$e,19:Re,20:Ie,22:105},t(de,[2,26]),t(u,[2,50],{10:be}),t(W,[2,28],{16:103,4:Se,5:ce,6:ae,8:Oe,11:ge,13:ze,17:He,18:$e,19:Re,20:Ie})],defaultActions:{8:[2,30],9:[2,31]},parseError:o(function(q,pe){if(pe.recoverable)this.trace(q);else{var ve=new Error(q);throw ve.hash=pe,ve}},"parseError"),parse:o(function(q){var pe=this,ve=[0],Pe=[],_e=[null],we=[],Ve=this.table,De="",qe=0,at=0,Rt=0,st=2,Ue=1,ct=we.slice.call(arguments,1),We=Object.create(this.lexer),ot={yy:{}};for(var Yt in this.yy)Object.prototype.hasOwnProperty.call(this.yy,Yt)&&(ot.yy[Yt]=this.yy[Yt]);We.setInput(q,ot.yy),ot.yy.lexer=We,ot.yy.parser=this,typeof We.yylloc>"u"&&(We.yylloc={});var bt=We.yylloc;we.push(bt);var Mt=We.options&&We.options.ranges;typeof ot.yy.parseError=="function"?this.parseError=ot.yy.parseError:this.parseError=Object.getPrototypeOf(this).parseError;function xt(Ce){ve.length=ve.length-2*Ce,_e.length=_e.length-Ce,we.length=we.length-Ce}o(xt,"popStack");function ut(){var Ce;return Ce=Pe.pop()||We.lex()||Ue,typeof Ce!="number"&&(Ce instanceof Array&&(Pe=Ce,Ce=Pe.pop()),Ce=pe.symbols_[Ce]||Ce),Ce}o(ut,"lex");for(var Et,ft,yt,nt,dn,Tt,On={},tn,_r,Dr,Pn;;){if(yt=ve[ve.length-1],this.defaultActions[yt]?nt=this.defaultActions[yt]:((Et===null||typeof Et>"u")&&(Et=ut()),nt=Ve[yt]&&Ve[yt][Et]),typeof nt>"u"||!nt.length||!nt[0]){var At="";Pn=[];for(tn in Ve[yt])this.terminals_[tn]&&tn>st&&Pn.push("'"+this.terminals_[tn]+"'");We.showPosition?At="Parse error on line "+(qe+1)+`: +`+We.showPosition()+` +Expecting `+Pn.join(", ")+", got '"+(this.terminals_[Et]||Et)+"'":At="Parse error on line "+(qe+1)+": Unexpected "+(Et==Ue?"end of input":"'"+(this.terminals_[Et]||Et)+"'"),this.parseError(At,{text:We.match,token:this.terminals_[Et]||Et,line:We.yylineno,loc:bt,expected:Pn})}if(nt[0]instanceof Array&&nt.length>1)throw new Error("Parse Error: multiple actions possible at state: "+yt+", token: "+Et);switch(nt[0]){case 1:ve.push(Et),_e.push(We.yytext),we.push(We.yylloc),ve.push(nt[1]),Et=null,ft?(Et=ft,ft=null):(at=We.yyleng,De=We.yytext,qe=We.yylineno,bt=We.yylloc,Rt>0&&Rt--);break;case 2:if(_r=this.productions_[nt[1]][1],On.$=_e[_e.length-_r],On._$={first_line:we[we.length-(_r||1)].first_line,last_line:we[we.length-1].last_line,first_column:we[we.length-(_r||1)].first_column,last_column:we[we.length-1].last_column},Mt&&(On._$.range=[we[we.length-(_r||1)].range[0],we[we.length-1].range[1]]),Tt=this.performAction.apply(On,[De,at,qe,ot.yy,nt[1],_e,we].concat(ct)),typeof Tt<"u")return Tt;_r&&(ve=ve.slice(0,-1*_r*2),_e=_e.slice(0,-1*_r),we=we.slice(0,-1*_r)),ve.push(this.productions_[nt[1]][0]),_e.push(On.$),we.push(On._$),Dr=Ve[ve[ve.length-2]][ve[ve.length-1]],ve.push(Dr);break;case 3:return!0}}return!0},"parse")},oe=function(){var xe={EOF:1,parseError:o(function(pe,ve){if(this.yy.parser)this.yy.parser.parseError(pe,ve);else throw new Error(pe)},"parseError"),setInput:o(function(q,pe){return this.yy=pe||this.yy||{},this._input=q,this._more=this._backtrack=this.done=!1,this.yylineno=this.yyleng=0,this.yytext=this.matched=this.match="",this.conditionStack=["INITIAL"],this.yylloc={first_line:1,first_column:0,last_line:1,last_column:0},this.options.ranges&&(this.yylloc.range=[0,0]),this.offset=0,this},"setInput"),input:o(function(){var q=this._input[0];this.yytext+=q,this.yyleng++,this.offset++,this.match+=q,this.matched+=q;var pe=q.match(/(?:\r\n?|\n).*/g);return pe?(this.yylineno++,this.yylloc.last_line++):this.yylloc.last_column++,this.options.ranges&&this.yylloc.range[1]++,this._input=this._input.slice(1),q},"input"),unput:o(function(q){var pe=q.length,ve=q.split(/(?:\r\n?|\n)/g);this._input=q+this._input,this.yytext=this.yytext.substr(0,this.yytext.length-pe),this.offset-=pe;var Pe=this.match.split(/(?:\r\n?|\n)/g);this.match=this.match.substr(0,this.match.length-1),this.matched=this.matched.substr(0,this.matched.length-1),ve.length-1&&(this.yylineno-=ve.length-1);var _e=this.yylloc.range;return this.yylloc={first_line:this.yylloc.first_line,last_line:this.yylineno+1,first_column:this.yylloc.first_column,last_column:ve?(ve.length===Pe.length?this.yylloc.first_column:0)+Pe[Pe.length-ve.length].length-ve[0].length:this.yylloc.first_column-pe},this.options.ranges&&(this.yylloc.range=[_e[0],_e[0]+this.yyleng-pe]),this.yyleng=this.yytext.length,this},"unput"),more:o(function(){return this._more=!0,this},"more"),reject:o(function(){if(this.options.backtrack_lexer)this._backtrack=!0;else return this.parseError("Lexical error on line "+(this.yylineno+1)+`. You can only invoke reject() in the lexer when the lexer is of the backtracking persuasion (options.backtrack_lexer = true). +`+this.showPosition(),{text:"",token:null,line:this.yylineno});return this},"reject"),less:o(function(q){this.unput(this.match.slice(q))},"less"),pastInput:o(function(){var q=this.matched.substr(0,this.matched.length-this.match.length);return(q.length>20?"...":"")+q.substr(-20).replace(/\n/g,"")},"pastInput"),upcomingInput:o(function(){var q=this.match;return q.length<20&&(q+=this._input.substr(0,20-q.length)),(q.substr(0,20)+(q.length>20?"...":"")).replace(/\n/g,"")},"upcomingInput"),showPosition:o(function(){var q=this.pastInput(),pe=new Array(q.length+1).join("-");return q+this.upcomingInput()+` +`+pe+"^"},"showPosition"),test_match:o(function(q,pe){var ve,Pe,_e;if(this.options.backtrack_lexer&&(_e={yylineno:this.yylineno,yylloc:{first_line:this.yylloc.first_line,last_line:this.last_line,first_column:this.yylloc.first_column,last_column:this.yylloc.last_column},yytext:this.yytext,match:this.match,matches:this.matches,matched:this.matched,yyleng:this.yyleng,offset:this.offset,_more:this._more,_input:this._input,yy:this.yy,conditionStack:this.conditionStack.slice(0),done:this.done},this.options.ranges&&(_e.yylloc.range=this.yylloc.range.slice(0))),Pe=q[0].match(/(?:\r\n?|\n).*/g),Pe&&(this.yylineno+=Pe.length),this.yylloc={first_line:this.yylloc.last_line,last_line:this.yylineno+1,first_column:this.yylloc.last_column,last_column:Pe?Pe[Pe.length-1].length-Pe[Pe.length-1].match(/\r?\n?/)[0].length:this.yylloc.last_column+q[0].length},this.yytext+=q[0],this.match+=q[0],this.matches=q,this.yyleng=this.yytext.length,this.options.ranges&&(this.yylloc.range=[this.offset,this.offset+=this.yyleng]),this._more=!1,this._backtrack=!1,this._input=this._input.slice(q[0].length),this.matched+=q[0],ve=this.performAction.call(this,this.yy,this,pe,this.conditionStack[this.conditionStack.length-1]),this.done&&this._input&&(this.done=!1),ve)return ve;if(this._backtrack){for(var we in _e)this[we]=_e[we];return!1}return!1},"test_match"),next:o(function(){if(this.done)return this.EOF;this._input||(this.done=!0);var q,pe,ve,Pe;this._more||(this.yytext="",this.match="");for(var _e=this._currentRules(),we=0;we<_e.length;we++)if(ve=this._input.match(this.rules[_e[we]]),ve&&(!pe||ve[0].length>pe[0].length)){if(pe=ve,Pe=we,this.options.backtrack_lexer){if(q=this.test_match(ve,_e[we]),q!==!1)return q;if(this._backtrack){pe=!1;continue}else return!1}else if(!this.options.flex)break}return pe?(q=this.test_match(pe,_e[Pe]),q!==!1?q:!1):this._input===""?this.EOF:this.parseError("Lexical error on line "+(this.yylineno+1)+`. Unrecognized text. +`+this.showPosition(),{text:"",token:null,line:this.yylineno})},"next"),lex:o(function(){var pe=this.next();return pe||this.lex()},"lex"),begin:o(function(pe){this.conditionStack.push(pe)},"begin"),popState:o(function(){var pe=this.conditionStack.length-1;return pe>0?this.conditionStack.pop():this.conditionStack[0]},"popState"),_currentRules:o(function(){return this.conditionStack.length&&this.conditionStack[this.conditionStack.length-1]?this.conditions[this.conditionStack[this.conditionStack.length-1]].rules:this.conditions.INITIAL.rules},"_currentRules"),topState:o(function(pe){return pe=this.conditionStack.length-1-Math.abs(pe||0),pe>=0?this.conditionStack[pe]:"INITIAL"},"topState"),pushState:o(function(pe){this.begin(pe)},"pushState"),stateStackSize:o(function(){return this.conditionStack.length},"stateStackSize"),options:{"case-insensitive":!0},performAction:o(function(pe,ve,Pe,_e){var we=_e;switch(Pe){case 0:break;case 1:break;case 2:return 55;case 3:break;case 4:return this.begin("title"),35;break;case 5:return this.popState(),"title_value";break;case 6:return this.begin("acc_title"),37;break;case 7:return this.popState(),"acc_title_value";break;case 8:return this.begin("acc_descr"),39;break;case 9:return this.popState(),"acc_descr_value";break;case 10:this.begin("acc_descr_multiline");break;case 11:this.popState();break;case 12:return"acc_descr_multiline_value";case 13:return 48;case 14:return 50;case 15:return 49;case 16:return 51;case 17:return 52;case 18:return 53;case 19:return 54;case 20:return 25;case 21:this.begin("md_string");break;case 22:return"MD_STR";case 23:this.popState();break;case 24:this.begin("string");break;case 25:this.popState();break;case 26:return"STR";case 27:this.begin("class_name");break;case 28:return this.popState(),47;break;case 29:return this.begin("point_start"),44;break;case 30:return this.begin("point_x"),45;break;case 31:this.popState();break;case 32:this.popState(),this.begin("point_y");break;case 33:return this.popState(),46;break;case 34:return 28;case 35:return 4;case 36:return 11;case 37:return 64;case 38:return 10;case 39:return 65;case 40:return 65;case 41:return 14;case 42:return 13;case 43:return 67;case 44:return 66;case 45:return 12;case 46:return 8;case 47:return 5;case 48:return 18;case 49:return 56;case 50:return 63;case 51:return 57}},"anonymous"),rules:[/^(?:%%(?!\{)[^\n]*)/i,/^(?:[^\}]%%[^\n]*)/i,/^(?:[\n\r]+)/i,/^(?:%%[^\n]*)/i,/^(?:title\b)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accTitle\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*\{\s*)/i,/^(?:[\}])/i,/^(?:[^\}]*)/i,/^(?: *x-axis *)/i,/^(?: *y-axis *)/i,/^(?: *--+> *)/i,/^(?: *quadrant-1 *)/i,/^(?: *quadrant-2 *)/i,/^(?: *quadrant-3 *)/i,/^(?: *quadrant-4 *)/i,/^(?:classDef\b)/i,/^(?:["][`])/i,/^(?:[^`"]+)/i,/^(?:[`]["])/i,/^(?:["])/i,/^(?:["])/i,/^(?:[^"]*)/i,/^(?::::)/i,/^(?:^\w+)/i,/^(?:\s*:\s*\[\s*)/i,/^(?:(1)|(0(.\d+)?))/i,/^(?:\s*\] *)/i,/^(?:\s*,\s*)/i,/^(?:(1)|(0(.\d+)?))/i,/^(?: *quadrantChart *)/i,/^(?:[A-Za-z]+)/i,/^(?::)/i,/^(?:\+)/i,/^(?:,)/i,/^(?:=)/i,/^(?:=)/i,/^(?:\*)/i,/^(?:#)/i,/^(?:[\_])/i,/^(?:\.)/i,/^(?:&)/i,/^(?:-)/i,/^(?:[0-9]+)/i,/^(?:\s)/i,/^(?:;)/i,/^(?:[!"#$%&'*+,-.`?\\_/])/i,/^(?:$)/i],conditions:{class_name:{rules:[28],inclusive:!1},point_y:{rules:[33],inclusive:!1},point_x:{rules:[32],inclusive:!1},point_start:{rules:[30,31],inclusive:!1},acc_descr_multiline:{rules:[11,12],inclusive:!1},acc_descr:{rules:[9],inclusive:!1},acc_title:{rules:[7],inclusive:!1},title:{rules:[5],inclusive:!1},md_string:{rules:[22,23],inclusive:!1},string:{rules:[25,26],inclusive:!1},INITIAL:{rules:[0,1,2,3,4,6,8,10,13,14,15,16,17,18,19,20,21,24,27,29,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51],inclusive:!0}}};return xe}();re.lexer=oe;function V(){this.yy={}}return o(V,"Parser"),V.prototype=re,re.Parser=V,new V}();hO.parser=hO;uhe=hO});var ms,y6,fhe=N(()=>{"use strict";dr();Ya();vt();_y();ms=oh(),y6=class{constructor(){this.classes=new Map;this.config=this.getDefaultConfig(),this.themeConfig=this.getDefaultThemeConfig(),this.data=this.getDefaultData()}static{o(this,"QuadrantBuilder")}getDefaultData(){return{titleText:"",quadrant1Text:"",quadrant2Text:"",quadrant3Text:"",quadrant4Text:"",xAxisLeftText:"",xAxisRightText:"",yAxisBottomText:"",yAxisTopText:"",points:[]}}getDefaultConfig(){return{showXAxis:!0,showYAxis:!0,showTitle:!0,chartHeight:or.quadrantChart?.chartWidth||500,chartWidth:or.quadrantChart?.chartHeight||500,titlePadding:or.quadrantChart?.titlePadding||10,titleFontSize:or.quadrantChart?.titleFontSize||20,quadrantPadding:or.quadrantChart?.quadrantPadding||5,xAxisLabelPadding:or.quadrantChart?.xAxisLabelPadding||5,yAxisLabelPadding:or.quadrantChart?.yAxisLabelPadding||5,xAxisLabelFontSize:or.quadrantChart?.xAxisLabelFontSize||16,yAxisLabelFontSize:or.quadrantChart?.yAxisLabelFontSize||16,quadrantLabelFontSize:or.quadrantChart?.quadrantLabelFontSize||16,quadrantTextTopPadding:or.quadrantChart?.quadrantTextTopPadding||5,pointTextPadding:or.quadrantChart?.pointTextPadding||5,pointLabelFontSize:or.quadrantChart?.pointLabelFontSize||12,pointRadius:or.quadrantChart?.pointRadius||5,xAxisPosition:or.quadrantChart?.xAxisPosition||"top",yAxisPosition:or.quadrantChart?.yAxisPosition||"left",quadrantInternalBorderStrokeWidth:or.quadrantChart?.quadrantInternalBorderStrokeWidth||1,quadrantExternalBorderStrokeWidth:or.quadrantChart?.quadrantExternalBorderStrokeWidth||2}}getDefaultThemeConfig(){return{quadrant1Fill:ms.quadrant1Fill,quadrant2Fill:ms.quadrant2Fill,quadrant3Fill:ms.quadrant3Fill,quadrant4Fill:ms.quadrant4Fill,quadrant1TextFill:ms.quadrant1TextFill,quadrant2TextFill:ms.quadrant2TextFill,quadrant3TextFill:ms.quadrant3TextFill,quadrant4TextFill:ms.quadrant4TextFill,quadrantPointFill:ms.quadrantPointFill,quadrantPointTextFill:ms.quadrantPointTextFill,quadrantXAxisTextFill:ms.quadrantXAxisTextFill,quadrantYAxisTextFill:ms.quadrantYAxisTextFill,quadrantTitleFill:ms.quadrantTitleFill,quadrantInternalBorderStrokeFill:ms.quadrantInternalBorderStrokeFill,quadrantExternalBorderStrokeFill:ms.quadrantExternalBorderStrokeFill}}clear(){this.config=this.getDefaultConfig(),this.themeConfig=this.getDefaultThemeConfig(),this.data=this.getDefaultData(),this.classes=new Map,Y.info("clear called")}setData(e){this.data={...this.data,...e}}addPoints(e){this.data.points=[...e,...this.data.points]}addClass(e,r){this.classes.set(e,r)}setConfig(e){Y.trace("setConfig called with: ",e),this.config={...this.config,...e}}setThemeConfig(e){Y.trace("setThemeConfig called with: ",e),this.themeConfig={...this.themeConfig,...e}}calculateSpace(e,r,n,i){let a=this.config.xAxisLabelPadding*2+this.config.xAxisLabelFontSize,s={top:e==="top"&&r?a:0,bottom:e==="bottom"&&r?a:0},l=this.config.yAxisLabelPadding*2+this.config.yAxisLabelFontSize,u={left:this.config.yAxisPosition==="left"&&n?l:0,right:this.config.yAxisPosition==="right"&&n?l:0},h=this.config.titleFontSize+this.config.titlePadding*2,f={top:i?h:0},d=this.config.quadrantPadding+u.left,p=this.config.quadrantPadding+s.top+f.top,m=this.config.chartWidth-this.config.quadrantPadding*2-u.left-u.right,g=this.config.chartHeight-this.config.quadrantPadding*2-s.top-s.bottom-f.top,y=m/2,v=g/2;return{xAxisSpace:s,yAxisSpace:u,titleSpace:f,quadrantSpace:{quadrantLeft:d,quadrantTop:p,quadrantWidth:m,quadrantHalfWidth:y,quadrantHeight:g,quadrantHalfHeight:v}}}getAxisLabels(e,r,n,i){let{quadrantSpace:a,titleSpace:s}=i,{quadrantHalfHeight:l,quadrantHeight:u,quadrantLeft:h,quadrantHalfWidth:f,quadrantTop:d,quadrantWidth:p}=a,m=!!this.data.xAxisRightText,g=!!this.data.yAxisTopText,y=[];return this.data.xAxisLeftText&&r&&y.push({text:this.data.xAxisLeftText,fill:this.themeConfig.quadrantXAxisTextFill,x:h+(m?f/2:0),y:e==="top"?this.config.xAxisLabelPadding+s.top:this.config.xAxisLabelPadding+d+u+this.config.quadrantPadding,fontSize:this.config.xAxisLabelFontSize,verticalPos:m?"center":"left",horizontalPos:"top",rotation:0}),this.data.xAxisRightText&&r&&y.push({text:this.data.xAxisRightText,fill:this.themeConfig.quadrantXAxisTextFill,x:h+f+(m?f/2:0),y:e==="top"?this.config.xAxisLabelPadding+s.top:this.config.xAxisLabelPadding+d+u+this.config.quadrantPadding,fontSize:this.config.xAxisLabelFontSize,verticalPos:m?"center":"left",horizontalPos:"top",rotation:0}),this.data.yAxisBottomText&&n&&y.push({text:this.data.yAxisBottomText,fill:this.themeConfig.quadrantYAxisTextFill,x:this.config.yAxisPosition==="left"?this.config.yAxisLabelPadding:this.config.yAxisLabelPadding+h+p+this.config.quadrantPadding,y:d+u-(g?l/2:0),fontSize:this.config.yAxisLabelFontSize,verticalPos:g?"center":"left",horizontalPos:"top",rotation:-90}),this.data.yAxisTopText&&n&&y.push({text:this.data.yAxisTopText,fill:this.themeConfig.quadrantYAxisTextFill,x:this.config.yAxisPosition==="left"?this.config.yAxisLabelPadding:this.config.yAxisLabelPadding+h+p+this.config.quadrantPadding,y:d+l-(g?l/2:0),fontSize:this.config.yAxisLabelFontSize,verticalPos:g?"center":"left",horizontalPos:"top",rotation:-90}),y}getQuadrants(e){let{quadrantSpace:r}=e,{quadrantHalfHeight:n,quadrantLeft:i,quadrantHalfWidth:a,quadrantTop:s}=r,l=[{text:{text:this.data.quadrant1Text,fill:this.themeConfig.quadrant1TextFill,x:0,y:0,fontSize:this.config.quadrantLabelFontSize,verticalPos:"center",horizontalPos:"middle",rotation:0},x:i+a,y:s,width:a,height:n,fill:this.themeConfig.quadrant1Fill},{text:{text:this.data.quadrant2Text,fill:this.themeConfig.quadrant2TextFill,x:0,y:0,fontSize:this.config.quadrantLabelFontSize,verticalPos:"center",horizontalPos:"middle",rotation:0},x:i,y:s,width:a,height:n,fill:this.themeConfig.quadrant2Fill},{text:{text:this.data.quadrant3Text,fill:this.themeConfig.quadrant3TextFill,x:0,y:0,fontSize:this.config.quadrantLabelFontSize,verticalPos:"center",horizontalPos:"middle",rotation:0},x:i,y:s+n,width:a,height:n,fill:this.themeConfig.quadrant3Fill},{text:{text:this.data.quadrant4Text,fill:this.themeConfig.quadrant4TextFill,x:0,y:0,fontSize:this.config.quadrantLabelFontSize,verticalPos:"center",horizontalPos:"middle",rotation:0},x:i+a,y:s+n,width:a,height:n,fill:this.themeConfig.quadrant4Fill}];for(let u of l)u.text.x=u.x+u.width/2,this.data.points.length===0?(u.text.y=u.y+u.height/2,u.text.horizontalPos="middle"):(u.text.y=u.y+this.config.quadrantTextTopPadding,u.text.horizontalPos="top");return l}getQuadrantPoints(e){let{quadrantSpace:r}=e,{quadrantHeight:n,quadrantLeft:i,quadrantTop:a,quadrantWidth:s}=r,l=gl().domain([0,1]).range([i,s+i]),u=gl().domain([0,1]).range([n+a,a]);return this.data.points.map(f=>{let d=this.classes.get(f.className);return d&&(f={...d,...f}),{x:l(f.x),y:u(f.y),fill:f.color??this.themeConfig.quadrantPointFill,radius:f.radius??this.config.pointRadius,text:{text:f.text,fill:this.themeConfig.quadrantPointTextFill,x:l(f.x),y:u(f.y)+this.config.pointTextPadding,verticalPos:"center",horizontalPos:"top",fontSize:this.config.pointLabelFontSize,rotation:0},strokeColor:f.strokeColor??this.themeConfig.quadrantPointFill,strokeWidth:f.strokeWidth??"0px"}})}getBorders(e){let r=this.config.quadrantExternalBorderStrokeWidth/2,{quadrantSpace:n}=e,{quadrantHalfHeight:i,quadrantHeight:a,quadrantLeft:s,quadrantHalfWidth:l,quadrantTop:u,quadrantWidth:h}=n;return[{strokeFill:this.themeConfig.quadrantExternalBorderStrokeFill,strokeWidth:this.config.quadrantExternalBorderStrokeWidth,x1:s-r,y1:u,x2:s+h+r,y2:u},{strokeFill:this.themeConfig.quadrantExternalBorderStrokeFill,strokeWidth:this.config.quadrantExternalBorderStrokeWidth,x1:s+h,y1:u+r,x2:s+h,y2:u+a-r},{strokeFill:this.themeConfig.quadrantExternalBorderStrokeFill,strokeWidth:this.config.quadrantExternalBorderStrokeWidth,x1:s-r,y1:u+a,x2:s+h+r,y2:u+a},{strokeFill:this.themeConfig.quadrantExternalBorderStrokeFill,strokeWidth:this.config.quadrantExternalBorderStrokeWidth,x1:s,y1:u+r,x2:s,y2:u+a-r},{strokeFill:this.themeConfig.quadrantInternalBorderStrokeFill,strokeWidth:this.config.quadrantInternalBorderStrokeWidth,x1:s+l,y1:u+r,x2:s+l,y2:u+a-r},{strokeFill:this.themeConfig.quadrantInternalBorderStrokeFill,strokeWidth:this.config.quadrantInternalBorderStrokeWidth,x1:s+r,y1:u+i,x2:s+h-r,y2:u+i}]}getTitle(e){if(e)return{text:this.data.titleText,fill:this.themeConfig.quadrantTitleFill,fontSize:this.config.titleFontSize,horizontalPos:"top",verticalPos:"center",rotation:0,y:this.config.titlePadding,x:this.config.chartWidth/2}}build(){let e=this.config.showXAxis&&!!(this.data.xAxisLeftText||this.data.xAxisRightText),r=this.config.showYAxis&&!!(this.data.yAxisTopText||this.data.yAxisBottomText),n=this.config.showTitle&&!!this.data.titleText,i=this.data.points.length>0?"bottom":this.config.xAxisPosition,a=this.calculateSpace(i,e,r,n);return{points:this.getQuadrantPoints(a),quadrants:this.getQuadrants(a),axisLabels:this.getAxisLabels(i,e,r,a),borderLines:this.getBorders(a),title:this.getTitle(n)}}}});function fO(t){return!/^#?([\dA-Fa-f]{6}|[\dA-Fa-f]{3})$/.test(t)}function dhe(t){return!/^\d+$/.test(t)}function phe(t){return!/^\d+px$/.test(t)}var Ap,mhe=N(()=>{"use strict";Ap=class extends Error{static{o(this,"InvalidStyleError")}constructor(e,r,n){super(`value for ${e} ${r} is invalid, please use a valid ${n}`),this.name="InvalidStyleError"}};o(fO,"validateHexCode");o(dhe,"validateNumber");o(phe,"validateSizeInPixels")});function Xu(t){return Tr(t.trim(),pGe)}function mGe(t){ba.setData({quadrant1Text:Xu(t.text)})}function gGe(t){ba.setData({quadrant2Text:Xu(t.text)})}function yGe(t){ba.setData({quadrant3Text:Xu(t.text)})}function vGe(t){ba.setData({quadrant4Text:Xu(t.text)})}function xGe(t){ba.setData({xAxisLeftText:Xu(t.text)})}function bGe(t){ba.setData({xAxisRightText:Xu(t.text)})}function wGe(t){ba.setData({yAxisTopText:Xu(t.text)})}function TGe(t){ba.setData({yAxisBottomText:Xu(t.text)})}function dO(t){let e={};for(let r of t){let[n,i]=r.trim().split(/\s*:\s*/);if(n==="radius"){if(dhe(i))throw new Ap(n,i,"number");e.radius=parseInt(i)}else if(n==="color"){if(fO(i))throw new Ap(n,i,"hex code");e.color=i}else if(n==="stroke-color"){if(fO(i))throw new Ap(n,i,"hex code");e.strokeColor=i}else if(n==="stroke-width"){if(phe(i))throw new Ap(n,i,"number of pixels (eg. 10px)");e.strokeWidth=i}else throw new Error(`style named ${n} is not supported.`)}return e}function kGe(t,e,r,n,i){let a=dO(i);ba.addPoints([{x:r,y:n,text:Xu(t.text),className:e,...a}])}function EGe(t,e){ba.addClass(t,dO(e))}function SGe(t){ba.setConfig({chartWidth:t})}function CGe(t){ba.setConfig({chartHeight:t})}function AGe(){let t=me(),{themeVariables:e,quadrantChart:r}=t;return r&&ba.setConfig(r),ba.setThemeConfig({quadrant1Fill:e.quadrant1Fill,quadrant2Fill:e.quadrant2Fill,quadrant3Fill:e.quadrant3Fill,quadrant4Fill:e.quadrant4Fill,quadrant1TextFill:e.quadrant1TextFill,quadrant2TextFill:e.quadrant2TextFill,quadrant3TextFill:e.quadrant3TextFill,quadrant4TextFill:e.quadrant4TextFill,quadrantPointFill:e.quadrantPointFill,quadrantPointTextFill:e.quadrantPointTextFill,quadrantXAxisTextFill:e.quadrantXAxisTextFill,quadrantYAxisTextFill:e.quadrantYAxisTextFill,quadrantExternalBorderStrokeFill:e.quadrantExternalBorderStrokeFill,quadrantInternalBorderStrokeFill:e.quadrantInternalBorderStrokeFill,quadrantTitleFill:e.quadrantTitleFill}),ba.setData({titleText:Ir()}),ba.build()}var pGe,ba,_Ge,ghe,yhe=N(()=>{"use strict";zt();gr();mi();fhe();mhe();pGe=me();o(Xu,"textSanitizer");ba=new y6;o(mGe,"setQuadrant1Text");o(gGe,"setQuadrant2Text");o(yGe,"setQuadrant3Text");o(vGe,"setQuadrant4Text");o(xGe,"setXAxisLeftText");o(bGe,"setXAxisRightText");o(wGe,"setYAxisTopText");o(TGe,"setYAxisBottomText");o(dO,"parseStyles");o(kGe,"addPoint");o(EGe,"addClass");o(SGe,"setWidth");o(CGe,"setHeight");o(AGe,"getQuadrantData");_Ge=o(function(){ba.clear(),Ar()},"clear"),ghe={setWidth:SGe,setHeight:CGe,setQuadrant1Text:mGe,setQuadrant2Text:gGe,setQuadrant3Text:yGe,setQuadrant4Text:vGe,setXAxisLeftText:xGe,setXAxisRightText:bGe,setYAxisTopText:wGe,setYAxisBottomText:TGe,parseStyles:dO,addPoint:kGe,addClass:EGe,getQuadrantData:AGe,clear:_Ge,setAccTitle:Lr,getAccTitle:Rr,setDiagramTitle:$r,getDiagramTitle:Ir,getAccDescription:Mr,setAccDescription:Nr}});var DGe,vhe,xhe=N(()=>{"use strict";dr();zt();vt();Ei();DGe=o((t,e,r,n)=>{function i(S){return S==="top"?"hanging":"middle"}o(i,"getDominantBaseLine");function a(S){return S==="left"?"start":"middle"}o(a,"getTextAnchor");function s(S){return`translate(${S.x}, ${S.y}) rotate(${S.rotation||0})`}o(s,"getTransformation");let l=me();Y.debug(`Rendering quadrant chart +`+t);let u=l.securityLevel,h;u==="sandbox"&&(h=Ge("#i"+e));let d=(u==="sandbox"?Ge(h.nodes()[0].contentDocument.body):Ge("body")).select(`[id="${e}"]`),p=d.append("g").attr("class","main"),m=l.quadrantChart?.chartWidth??500,g=l.quadrantChart?.chartHeight??500;vn(d,g,m,l.quadrantChart?.useMaxWidth??!0),d.attr("viewBox","0 0 "+m+" "+g),n.db.setHeight(g),n.db.setWidth(m);let y=n.db.getQuadrantData(),v=p.append("g").attr("class","quadrants"),x=p.append("g").attr("class","border"),b=p.append("g").attr("class","data-points"),w=p.append("g").attr("class","labels"),C=p.append("g").attr("class","title");y.title&&C.append("text").attr("x",0).attr("y",0).attr("fill",y.title.fill).attr("font-size",y.title.fontSize).attr("dominant-baseline",i(y.title.horizontalPos)).attr("text-anchor",a(y.title.verticalPos)).attr("transform",s(y.title)).text(y.title.text),y.borderLines&&x.selectAll("line").data(y.borderLines).enter().append("line").attr("x1",S=>S.x1).attr("y1",S=>S.y1).attr("x2",S=>S.x2).attr("y2",S=>S.y2).style("stroke",S=>S.strokeFill).style("stroke-width",S=>S.strokeWidth);let T=v.selectAll("g.quadrant").data(y.quadrants).enter().append("g").attr("class","quadrant");T.append("rect").attr("x",S=>S.x).attr("y",S=>S.y).attr("width",S=>S.width).attr("height",S=>S.height).attr("fill",S=>S.fill),T.append("text").attr("x",0).attr("y",0).attr("fill",S=>S.text.fill).attr("font-size",S=>S.text.fontSize).attr("dominant-baseline",S=>i(S.text.horizontalPos)).attr("text-anchor",S=>a(S.text.verticalPos)).attr("transform",S=>s(S.text)).text(S=>S.text.text),w.selectAll("g.label").data(y.axisLabels).enter().append("g").attr("class","label").append("text").attr("x",0).attr("y",0).text(S=>S.text).attr("fill",S=>S.fill).attr("font-size",S=>S.fontSize).attr("dominant-baseline",S=>i(S.horizontalPos)).attr("text-anchor",S=>a(S.verticalPos)).attr("transform",S=>s(S));let A=b.selectAll("g.data-point").data(y.points).enter().append("g").attr("class","data-point");A.append("circle").attr("cx",S=>S.x).attr("cy",S=>S.y).attr("r",S=>S.radius).attr("fill",S=>S.fill).attr("stroke",S=>S.strokeColor).attr("stroke-width",S=>S.strokeWidth),A.append("text").attr("x",0).attr("y",0).text(S=>S.text.text).attr("fill",S=>S.text.fill).attr("font-size",S=>S.text.fontSize).attr("dominant-baseline",S=>i(S.text.horizontalPos)).attr("text-anchor",S=>a(S.text.verticalPos)).attr("transform",S=>s(S.text))},"draw"),vhe={draw:DGe}});var bhe={};hr(bhe,{diagram:()=>LGe});var LGe,whe=N(()=>{"use strict";hhe();yhe();xhe();LGe={parser:uhe,db:ghe,renderer:vhe,styles:o(()=>"","styles")}});var pO,Ehe,She=N(()=>{"use strict";pO=function(){var t=o(function(O,M,B,F){for(B=B||{},F=O.length;F--;B[O[F]]=M);return B},"o"),e=[1,10,12,14,16,18,19,21,23],r=[2,6],n=[1,3],i=[1,5],a=[1,6],s=[1,7],l=[1,5,10,12,14,16,18,19,21,23,34,35,36],u=[1,25],h=[1,26],f=[1,28],d=[1,29],p=[1,30],m=[1,31],g=[1,32],y=[1,33],v=[1,34],x=[1,35],b=[1,36],w=[1,37],C=[1,43],T=[1,42],E=[1,47],A=[1,50],S=[1,10,12,14,16,18,19,21,23,34,35,36],_=[1,10,12,14,16,18,19,21,23,24,26,27,28,34,35,36],I=[1,10,12,14,16,18,19,21,23,24,26,27,28,34,35,36,41,42,43,44,45,46,47,48,49,50],D=[1,64],k={trace:o(function(){},"trace"),yy:{},symbols_:{error:2,start:3,eol:4,XYCHART:5,chartConfig:6,document:7,CHART_ORIENTATION:8,statement:9,title:10,text:11,X_AXIS:12,parseXAxis:13,Y_AXIS:14,parseYAxis:15,LINE:16,plotData:17,BAR:18,acc_title:19,acc_title_value:20,acc_descr:21,acc_descr_value:22,acc_descr_multiline_value:23,SQUARE_BRACES_START:24,commaSeparatedNumbers:25,SQUARE_BRACES_END:26,NUMBER_WITH_DECIMAL:27,COMMA:28,xAxisData:29,bandData:30,ARROW_DELIMITER:31,commaSeparatedTexts:32,yAxisData:33,NEWLINE:34,SEMI:35,EOF:36,alphaNum:37,STR:38,MD_STR:39,alphaNumToken:40,AMP:41,NUM:42,ALPHA:43,PLUS:44,EQUALS:45,MULT:46,DOT:47,BRKT:48,MINUS:49,UNDERSCORE:50,$accept:0,$end:1},terminals_:{2:"error",5:"XYCHART",8:"CHART_ORIENTATION",10:"title",12:"X_AXIS",14:"Y_AXIS",16:"LINE",18:"BAR",19:"acc_title",20:"acc_title_value",21:"acc_descr",22:"acc_descr_value",23:"acc_descr_multiline_value",24:"SQUARE_BRACES_START",26:"SQUARE_BRACES_END",27:"NUMBER_WITH_DECIMAL",28:"COMMA",31:"ARROW_DELIMITER",34:"NEWLINE",35:"SEMI",36:"EOF",38:"STR",39:"MD_STR",41:"AMP",42:"NUM",43:"ALPHA",44:"PLUS",45:"EQUALS",46:"MULT",47:"DOT",48:"BRKT",49:"MINUS",50:"UNDERSCORE"},productions_:[0,[3,2],[3,3],[3,2],[3,1],[6,1],[7,0],[7,2],[9,2],[9,2],[9,2],[9,2],[9,2],[9,3],[9,2],[9,3],[9,2],[9,2],[9,1],[17,3],[25,3],[25,1],[13,1],[13,2],[13,1],[29,1],[29,3],[30,3],[32,3],[32,1],[15,1],[15,2],[15,1],[33,3],[4,1],[4,1],[4,1],[11,1],[11,1],[11,1],[37,1],[37,2],[40,1],[40,1],[40,1],[40,1],[40,1],[40,1],[40,1],[40,1],[40,1],[40,1]],performAction:o(function(M,B,F,P,z,$,H){var Q=$.length-1;switch(z){case 5:P.setOrientation($[Q]);break;case 9:P.setDiagramTitle($[Q].text.trim());break;case 12:P.setLineData({text:"",type:"text"},$[Q]);break;case 13:P.setLineData($[Q-1],$[Q]);break;case 14:P.setBarData({text:"",type:"text"},$[Q]);break;case 15:P.setBarData($[Q-1],$[Q]);break;case 16:this.$=$[Q].trim(),P.setAccTitle(this.$);break;case 17:case 18:this.$=$[Q].trim(),P.setAccDescription(this.$);break;case 19:this.$=$[Q-1];break;case 20:this.$=[Number($[Q-2]),...$[Q]];break;case 21:this.$=[Number($[Q])];break;case 22:P.setXAxisTitle($[Q]);break;case 23:P.setXAxisTitle($[Q-1]);break;case 24:P.setXAxisTitle({type:"text",text:""});break;case 25:P.setXAxisBand($[Q]);break;case 26:P.setXAxisRangeData(Number($[Q-2]),Number($[Q]));break;case 27:this.$=$[Q-1];break;case 28:this.$=[$[Q-2],...$[Q]];break;case 29:this.$=[$[Q]];break;case 30:P.setYAxisTitle($[Q]);break;case 31:P.setYAxisTitle($[Q-1]);break;case 32:P.setYAxisTitle({type:"text",text:""});break;case 33:P.setYAxisRangeData(Number($[Q-2]),Number($[Q]));break;case 37:this.$={text:$[Q],type:"text"};break;case 38:this.$={text:$[Q],type:"text"};break;case 39:this.$={text:$[Q],type:"markdown"};break;case 40:this.$=$[Q];break;case 41:this.$=$[Q-1]+""+$[Q];break}},"anonymous"),table:[t(e,r,{3:1,4:2,7:4,5:n,34:i,35:a,36:s}),{1:[3]},t(e,r,{4:2,7:4,3:8,5:n,34:i,35:a,36:s}),t(e,r,{4:2,7:4,6:9,3:10,5:n,8:[1,11],34:i,35:a,36:s}),{1:[2,4],9:12,10:[1,13],12:[1,14],14:[1,15],16:[1,16],18:[1,17],19:[1,18],21:[1,19],23:[1,20]},t(l,[2,34]),t(l,[2,35]),t(l,[2,36]),{1:[2,1]},t(e,r,{4:2,7:4,3:21,5:n,34:i,35:a,36:s}),{1:[2,3]},t(l,[2,5]),t(e,[2,7],{4:22,34:i,35:a,36:s}),{11:23,37:24,38:u,39:h,40:27,41:f,42:d,43:p,44:m,45:g,46:y,47:v,48:x,49:b,50:w},{11:39,13:38,24:C,27:T,29:40,30:41,37:24,38:u,39:h,40:27,41:f,42:d,43:p,44:m,45:g,46:y,47:v,48:x,49:b,50:w},{11:45,15:44,27:E,33:46,37:24,38:u,39:h,40:27,41:f,42:d,43:p,44:m,45:g,46:y,47:v,48:x,49:b,50:w},{11:49,17:48,24:A,37:24,38:u,39:h,40:27,41:f,42:d,43:p,44:m,45:g,46:y,47:v,48:x,49:b,50:w},{11:52,17:51,24:A,37:24,38:u,39:h,40:27,41:f,42:d,43:p,44:m,45:g,46:y,47:v,48:x,49:b,50:w},{20:[1,53]},{22:[1,54]},t(S,[2,18]),{1:[2,2]},t(S,[2,8]),t(S,[2,9]),t(_,[2,37],{40:55,41:f,42:d,43:p,44:m,45:g,46:y,47:v,48:x,49:b,50:w}),t(_,[2,38]),t(_,[2,39]),t(I,[2,40]),t(I,[2,42]),t(I,[2,43]),t(I,[2,44]),t(I,[2,45]),t(I,[2,46]),t(I,[2,47]),t(I,[2,48]),t(I,[2,49]),t(I,[2,50]),t(I,[2,51]),t(S,[2,10]),t(S,[2,22],{30:41,29:56,24:C,27:T}),t(S,[2,24]),t(S,[2,25]),{31:[1,57]},{11:59,32:58,37:24,38:u,39:h,40:27,41:f,42:d,43:p,44:m,45:g,46:y,47:v,48:x,49:b,50:w},t(S,[2,11]),t(S,[2,30],{33:60,27:E}),t(S,[2,32]),{31:[1,61]},t(S,[2,12]),{17:62,24:A},{25:63,27:D},t(S,[2,14]),{17:65,24:A},t(S,[2,16]),t(S,[2,17]),t(I,[2,41]),t(S,[2,23]),{27:[1,66]},{26:[1,67]},{26:[2,29],28:[1,68]},t(S,[2,31]),{27:[1,69]},t(S,[2,13]),{26:[1,70]},{26:[2,21],28:[1,71]},t(S,[2,15]),t(S,[2,26]),t(S,[2,27]),{11:59,32:72,37:24,38:u,39:h,40:27,41:f,42:d,43:p,44:m,45:g,46:y,47:v,48:x,49:b,50:w},t(S,[2,33]),t(S,[2,19]),{25:73,27:D},{26:[2,28]},{26:[2,20]}],defaultActions:{8:[2,1],10:[2,3],21:[2,2],72:[2,28],73:[2,20]},parseError:o(function(M,B){if(B.recoverable)this.trace(M);else{var F=new Error(M);throw F.hash=B,F}},"parseError"),parse:o(function(M){var B=this,F=[0],P=[],z=[null],$=[],H=this.table,Q="",j=0,ie=0,ne=0,le=2,he=1,K=$.slice.call(arguments,1),X=Object.create(this.lexer),te={yy:{}};for(var J in this.yy)Object.prototype.hasOwnProperty.call(this.yy,J)&&(te.yy[J]=this.yy[J]);X.setInput(M,te.yy),te.yy.lexer=X,te.yy.parser=this,typeof X.yylloc>"u"&&(X.yylloc={});var se=X.yylloc;$.push(se);var ue=X.options&&X.options.ranges;typeof te.yy.parseError=="function"?this.parseError=te.yy.parseError:this.parseError=Object.getPrototypeOf(this).parseError;function Z(re){F.length=F.length-2*re,z.length=z.length-re,$.length=$.length-re}o(Z,"popStack");function Se(){var re;return re=P.pop()||X.lex()||he,typeof re!="number"&&(re instanceof Array&&(P=re,re=P.pop()),re=B.symbols_[re]||re),re}o(Se,"lex");for(var ce,ae,Oe,ge,ze,He,$e={},Re,Ie,be,W;;){if(Oe=F[F.length-1],this.defaultActions[Oe]?ge=this.defaultActions[Oe]:((ce===null||typeof ce>"u")&&(ce=Se()),ge=H[Oe]&&H[Oe][ce]),typeof ge>"u"||!ge.length||!ge[0]){var de="";W=[];for(Re in H[Oe])this.terminals_[Re]&&Re>le&&W.push("'"+this.terminals_[Re]+"'");X.showPosition?de="Parse error on line "+(j+1)+`: +`+X.showPosition()+` +Expecting `+W.join(", ")+", got '"+(this.terminals_[ce]||ce)+"'":de="Parse error on line "+(j+1)+": Unexpected "+(ce==he?"end of input":"'"+(this.terminals_[ce]||ce)+"'"),this.parseError(de,{text:X.match,token:this.terminals_[ce]||ce,line:X.yylineno,loc:se,expected:W})}if(ge[0]instanceof Array&&ge.length>1)throw new Error("Parse Error: multiple actions possible at state: "+Oe+", token: "+ce);switch(ge[0]){case 1:F.push(ce),z.push(X.yytext),$.push(X.yylloc),F.push(ge[1]),ce=null,ae?(ce=ae,ae=null):(ie=X.yyleng,Q=X.yytext,j=X.yylineno,se=X.yylloc,ne>0&&ne--);break;case 2:if(Ie=this.productions_[ge[1]][1],$e.$=z[z.length-Ie],$e._$={first_line:$[$.length-(Ie||1)].first_line,last_line:$[$.length-1].last_line,first_column:$[$.length-(Ie||1)].first_column,last_column:$[$.length-1].last_column},ue&&($e._$.range=[$[$.length-(Ie||1)].range[0],$[$.length-1].range[1]]),He=this.performAction.apply($e,[Q,ie,j,te.yy,ge[1],z,$].concat(K)),typeof He<"u")return He;Ie&&(F=F.slice(0,-1*Ie*2),z=z.slice(0,-1*Ie),$=$.slice(0,-1*Ie)),F.push(this.productions_[ge[1]][0]),z.push($e.$),$.push($e._$),be=H[F[F.length-2]][F[F.length-1]],F.push(be);break;case 3:return!0}}return!0},"parse")},L=function(){var O={EOF:1,parseError:o(function(B,F){if(this.yy.parser)this.yy.parser.parseError(B,F);else throw new Error(B)},"parseError"),setInput:o(function(M,B){return this.yy=B||this.yy||{},this._input=M,this._more=this._backtrack=this.done=!1,this.yylineno=this.yyleng=0,this.yytext=this.matched=this.match="",this.conditionStack=["INITIAL"],this.yylloc={first_line:1,first_column:0,last_line:1,last_column:0},this.options.ranges&&(this.yylloc.range=[0,0]),this.offset=0,this},"setInput"),input:o(function(){var M=this._input[0];this.yytext+=M,this.yyleng++,this.offset++,this.match+=M,this.matched+=M;var B=M.match(/(?:\r\n?|\n).*/g);return B?(this.yylineno++,this.yylloc.last_line++):this.yylloc.last_column++,this.options.ranges&&this.yylloc.range[1]++,this._input=this._input.slice(1),M},"input"),unput:o(function(M){var B=M.length,F=M.split(/(?:\r\n?|\n)/g);this._input=M+this._input,this.yytext=this.yytext.substr(0,this.yytext.length-B),this.offset-=B;var P=this.match.split(/(?:\r\n?|\n)/g);this.match=this.match.substr(0,this.match.length-1),this.matched=this.matched.substr(0,this.matched.length-1),F.length-1&&(this.yylineno-=F.length-1);var z=this.yylloc.range;return this.yylloc={first_line:this.yylloc.first_line,last_line:this.yylineno+1,first_column:this.yylloc.first_column,last_column:F?(F.length===P.length?this.yylloc.first_column:0)+P[P.length-F.length].length-F[0].length:this.yylloc.first_column-B},this.options.ranges&&(this.yylloc.range=[z[0],z[0]+this.yyleng-B]),this.yyleng=this.yytext.length,this},"unput"),more:o(function(){return this._more=!0,this},"more"),reject:o(function(){if(this.options.backtrack_lexer)this._backtrack=!0;else return this.parseError("Lexical error on line "+(this.yylineno+1)+`. You can only invoke reject() in the lexer when the lexer is of the backtracking persuasion (options.backtrack_lexer = true). +`+this.showPosition(),{text:"",token:null,line:this.yylineno});return this},"reject"),less:o(function(M){this.unput(this.match.slice(M))},"less"),pastInput:o(function(){var M=this.matched.substr(0,this.matched.length-this.match.length);return(M.length>20?"...":"")+M.substr(-20).replace(/\n/g,"")},"pastInput"),upcomingInput:o(function(){var M=this.match;return M.length<20&&(M+=this._input.substr(0,20-M.length)),(M.substr(0,20)+(M.length>20?"...":"")).replace(/\n/g,"")},"upcomingInput"),showPosition:o(function(){var M=this.pastInput(),B=new Array(M.length+1).join("-");return M+this.upcomingInput()+` +`+B+"^"},"showPosition"),test_match:o(function(M,B){var F,P,z;if(this.options.backtrack_lexer&&(z={yylineno:this.yylineno,yylloc:{first_line:this.yylloc.first_line,last_line:this.last_line,first_column:this.yylloc.first_column,last_column:this.yylloc.last_column},yytext:this.yytext,match:this.match,matches:this.matches,matched:this.matched,yyleng:this.yyleng,offset:this.offset,_more:this._more,_input:this._input,yy:this.yy,conditionStack:this.conditionStack.slice(0),done:this.done},this.options.ranges&&(z.yylloc.range=this.yylloc.range.slice(0))),P=M[0].match(/(?:\r\n?|\n).*/g),P&&(this.yylineno+=P.length),this.yylloc={first_line:this.yylloc.last_line,last_line:this.yylineno+1,first_column:this.yylloc.last_column,last_column:P?P[P.length-1].length-P[P.length-1].match(/\r?\n?/)[0].length:this.yylloc.last_column+M[0].length},this.yytext+=M[0],this.match+=M[0],this.matches=M,this.yyleng=this.yytext.length,this.options.ranges&&(this.yylloc.range=[this.offset,this.offset+=this.yyleng]),this._more=!1,this._backtrack=!1,this._input=this._input.slice(M[0].length),this.matched+=M[0],F=this.performAction.call(this,this.yy,this,B,this.conditionStack[this.conditionStack.length-1]),this.done&&this._input&&(this.done=!1),F)return F;if(this._backtrack){for(var $ in z)this[$]=z[$];return!1}return!1},"test_match"),next:o(function(){if(this.done)return this.EOF;this._input||(this.done=!0);var M,B,F,P;this._more||(this.yytext="",this.match="");for(var z=this._currentRules(),$=0;$B[0].length)){if(B=F,P=$,this.options.backtrack_lexer){if(M=this.test_match(F,z[$]),M!==!1)return M;if(this._backtrack){B=!1;continue}else return!1}else if(!this.options.flex)break}return B?(M=this.test_match(B,z[P]),M!==!1?M:!1):this._input===""?this.EOF:this.parseError("Lexical error on line "+(this.yylineno+1)+`. Unrecognized text. +`+this.showPosition(),{text:"",token:null,line:this.yylineno})},"next"),lex:o(function(){var B=this.next();return B||this.lex()},"lex"),begin:o(function(B){this.conditionStack.push(B)},"begin"),popState:o(function(){var B=this.conditionStack.length-1;return B>0?this.conditionStack.pop():this.conditionStack[0]},"popState"),_currentRules:o(function(){return this.conditionStack.length&&this.conditionStack[this.conditionStack.length-1]?this.conditions[this.conditionStack[this.conditionStack.length-1]].rules:this.conditions.INITIAL.rules},"_currentRules"),topState:o(function(B){return B=this.conditionStack.length-1-Math.abs(B||0),B>=0?this.conditionStack[B]:"INITIAL"},"topState"),pushState:o(function(B){this.begin(B)},"pushState"),stateStackSize:o(function(){return this.conditionStack.length},"stateStackSize"),options:{"case-insensitive":!0},performAction:o(function(B,F,P,z){var $=z;switch(P){case 0:break;case 1:break;case 2:return this.popState(),34;break;case 3:return this.popState(),34;break;case 4:return 34;case 5:break;case 6:return 10;case 7:return this.pushState("acc_title"),19;break;case 8:return this.popState(),"acc_title_value";break;case 9:return this.pushState("acc_descr"),21;break;case 10:return this.popState(),"acc_descr_value";break;case 11:this.pushState("acc_descr_multiline");break;case 12:this.popState();break;case 13:return"acc_descr_multiline_value";case 14:return 5;case 15:return 8;case 16:return this.pushState("axis_data"),"X_AXIS";break;case 17:return this.pushState("axis_data"),"Y_AXIS";break;case 18:return this.pushState("axis_band_data"),24;break;case 19:return 31;case 20:return this.pushState("data"),16;break;case 21:return this.pushState("data"),18;break;case 22:return this.pushState("data_inner"),24;break;case 23:return 27;case 24:return this.popState(),26;break;case 25:this.popState();break;case 26:this.pushState("string");break;case 27:this.popState();break;case 28:return"STR";case 29:return 24;case 30:return 26;case 31:return 43;case 32:return"COLON";case 33:return 44;case 34:return 28;case 35:return 45;case 36:return 46;case 37:return 48;case 38:return 50;case 39:return 47;case 40:return 41;case 41:return 49;case 42:return 42;case 43:break;case 44:return 35;case 45:return 36}},"anonymous"),rules:[/^(?:%%(?!\{)[^\n]*)/i,/^(?:[^\}]%%[^\n]*)/i,/^(?:(\r?\n))/i,/^(?:(\r?\n))/i,/^(?:[\n\r]+)/i,/^(?:%%[^\n]*)/i,/^(?:title\b)/i,/^(?:accTitle\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*\{\s*)/i,/^(?:\{)/i,/^(?:[^\}]*)/i,/^(?:xychart-beta\b)/i,/^(?:(?:vertical|horizontal))/i,/^(?:x-axis\b)/i,/^(?:y-axis\b)/i,/^(?:\[)/i,/^(?:-->)/i,/^(?:line\b)/i,/^(?:bar\b)/i,/^(?:\[)/i,/^(?:[+-]?(?:\d+(?:\.\d+)?|\.\d+))/i,/^(?:\])/i,/^(?:(?:`\) \{ this\.pushState\(md_string\); \}\n\(\?:\(\?!`"\)\.\)\+ \{ return MD_STR; \}\n\(\?:`))/i,/^(?:["])/i,/^(?:["])/i,/^(?:[^"]*)/i,/^(?:\[)/i,/^(?:\])/i,/^(?:[A-Za-z]+)/i,/^(?::)/i,/^(?:\+)/i,/^(?:,)/i,/^(?:=)/i,/^(?:\*)/i,/^(?:#)/i,/^(?:[\_])/i,/^(?:\.)/i,/^(?:&)/i,/^(?:-)/i,/^(?:[0-9]+)/i,/^(?:\s+)/i,/^(?:;)/i,/^(?:$)/i],conditions:{data_inner:{rules:[0,1,4,5,6,7,9,11,14,15,16,17,20,21,23,24,25,26,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45],inclusive:!0},data:{rules:[0,1,3,4,5,6,7,9,11,14,15,16,17,20,21,22,25,26,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45],inclusive:!0},axis_band_data:{rules:[0,1,4,5,6,7,9,11,14,15,16,17,20,21,24,25,26,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45],inclusive:!0},axis_data:{rules:[0,1,2,4,5,6,7,9,11,14,15,16,17,18,19,20,21,23,25,26,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45],inclusive:!0},acc_descr_multiline:{rules:[12,13],inclusive:!1},acc_descr:{rules:[10],inclusive:!1},acc_title:{rules:[8],inclusive:!1},title:{rules:[],inclusive:!1},md_string:{rules:[],inclusive:!1},string:{rules:[27,28],inclusive:!1},INITIAL:{rules:[0,1,4,5,6,7,9,11,14,15,16,17,20,21,25,26,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45],inclusive:!0}}};return O}();k.lexer=L;function R(){this.yy={}}return o(R,"Parser"),R.prototype=k,k.Parser=R,new R}();pO.parser=pO;Ehe=pO});function mO(t){return t.type==="bar"}function v6(t){return t.type==="band"}function S1(t){return t.type==="linear"}var x6=N(()=>{"use strict";o(mO,"isBarPlot");o(v6,"isBandAxisData");o(S1,"isLinearAxisData")});var C1,gO=N(()=>{"use strict";to();C1=class{constructor(e){this.parentGroup=e}static{o(this,"TextDimensionCalculatorWithFont")}getMaxDimension(e,r){if(!this.parentGroup)return{width:e.reduce((a,s)=>Math.max(s.length,a),0)*r,height:r};let n={width:0,height:0},i=this.parentGroup.append("g").attr("visibility","hidden").attr("font-size",r);for(let a of e){let s=sK(i,1,a),l=s?s.width:a.length*r,u=s?s.height:r;n.width=Math.max(n.width,l),n.height=Math.max(n.height,u)}return i.remove(),n}}});var A1,yO=N(()=>{"use strict";A1=class{constructor(e,r,n,i){this.axisConfig=e;this.title=r;this.textDimensionCalculator=n;this.axisThemeConfig=i;this.boundingRect={x:0,y:0,width:0,height:0};this.axisPosition="left";this.showTitle=!1;this.showLabel=!1;this.showTick=!1;this.showAxisLine=!1;this.outerPadding=0;this.titleTextHeight=0;this.labelTextHeight=0;this.range=[0,10],this.boundingRect={x:0,y:0,width:0,height:0},this.axisPosition="left"}static{o(this,"BaseAxis")}setRange(e){this.range=e,this.axisPosition==="left"||this.axisPosition==="right"?this.boundingRect.height=e[1]-e[0]:this.boundingRect.width=e[1]-e[0],this.recalculateScale()}getRange(){return[this.range[0]+this.outerPadding,this.range[1]-this.outerPadding]}setAxisPosition(e){this.axisPosition=e,this.setRange(this.range)}getTickDistance(){let e=this.getRange();return Math.abs(e[0]-e[1])/this.getTickValues().length}getAxisOuterPadding(){return this.outerPadding}getLabelDimension(){return this.textDimensionCalculator.getMaxDimension(this.getTickValues().map(e=>e.toString()),this.axisConfig.labelFontSize)}recalculateOuterPaddingToDrawBar(){.7*this.getTickDistance()>this.outerPadding*2&&(this.outerPadding=Math.floor(.7*this.getTickDistance()/2)),this.recalculateScale()}calculateSpaceIfDrawnHorizontally(e){let r=e.height;if(this.axisConfig.showAxisLine&&r>this.axisConfig.axisLineWidth&&(r-=this.axisConfig.axisLineWidth,this.showAxisLine=!0),this.axisConfig.showLabel){let n=this.getLabelDimension(),i=.2*e.width;this.outerPadding=Math.min(n.width/2,i);let a=n.height+this.axisConfig.labelPadding*2;this.labelTextHeight=n.height,a<=r&&(r-=a,this.showLabel=!0)}if(this.axisConfig.showTick&&r>=this.axisConfig.tickLength&&(this.showTick=!0,r-=this.axisConfig.tickLength),this.axisConfig.showTitle&&this.title){let n=this.textDimensionCalculator.getMaxDimension([this.title],this.axisConfig.titleFontSize),i=n.height+this.axisConfig.titlePadding*2;this.titleTextHeight=n.height,i<=r&&(r-=i,this.showTitle=!0)}this.boundingRect.width=e.width,this.boundingRect.height=e.height-r}calculateSpaceIfDrawnVertical(e){let r=e.width;if(this.axisConfig.showAxisLine&&r>this.axisConfig.axisLineWidth&&(r-=this.axisConfig.axisLineWidth,this.showAxisLine=!0),this.axisConfig.showLabel){let n=this.getLabelDimension(),i=.2*e.height;this.outerPadding=Math.min(n.height/2,i);let a=n.width+this.axisConfig.labelPadding*2;a<=r&&(r-=a,this.showLabel=!0)}if(this.axisConfig.showTick&&r>=this.axisConfig.tickLength&&(this.showTick=!0,r-=this.axisConfig.tickLength),this.axisConfig.showTitle&&this.title){let n=this.textDimensionCalculator.getMaxDimension([this.title],this.axisConfig.titleFontSize),i=n.height+this.axisConfig.titlePadding*2;this.titleTextHeight=n.height,i<=r&&(r-=i,this.showTitle=!0)}this.boundingRect.width=e.width-r,this.boundingRect.height=e.height}calculateSpace(e){return this.axisPosition==="left"||this.axisPosition==="right"?this.calculateSpaceIfDrawnVertical(e):this.calculateSpaceIfDrawnHorizontally(e),this.recalculateScale(),{width:this.boundingRect.width,height:this.boundingRect.height}}setBoundingBoxXY(e){this.boundingRect.x=e.x,this.boundingRect.y=e.y}getDrawableElementsForLeftAxis(){let e=[];if(this.showAxisLine){let r=this.boundingRect.x+this.boundingRect.width-this.axisConfig.axisLineWidth/2;e.push({type:"path",groupTexts:["left-axis","axisl-line"],data:[{path:`M ${r},${this.boundingRect.y} L ${r},${this.boundingRect.y+this.boundingRect.height} `,strokeFill:this.axisThemeConfig.axisLineColor,strokeWidth:this.axisConfig.axisLineWidth}]})}if(this.showLabel&&e.push({type:"text",groupTexts:["left-axis","label"],data:this.getTickValues().map(r=>({text:r.toString(),x:this.boundingRect.x+this.boundingRect.width-(this.showLabel?this.axisConfig.labelPadding:0)-(this.showTick?this.axisConfig.tickLength:0)-(this.showAxisLine?this.axisConfig.axisLineWidth:0),y:this.getScaleValue(r),fill:this.axisThemeConfig.labelColor,fontSize:this.axisConfig.labelFontSize,rotation:0,verticalPos:"middle",horizontalPos:"right"}))}),this.showTick){let r=this.boundingRect.x+this.boundingRect.width-(this.showAxisLine?this.axisConfig.axisLineWidth:0);e.push({type:"path",groupTexts:["left-axis","ticks"],data:this.getTickValues().map(n=>({path:`M ${r},${this.getScaleValue(n)} L ${r-this.axisConfig.tickLength},${this.getScaleValue(n)}`,strokeFill:this.axisThemeConfig.tickColor,strokeWidth:this.axisConfig.tickWidth}))})}return this.showTitle&&e.push({type:"text",groupTexts:["left-axis","title"],data:[{text:this.title,x:this.boundingRect.x+this.axisConfig.titlePadding,y:this.boundingRect.y+this.boundingRect.height/2,fill:this.axisThemeConfig.titleColor,fontSize:this.axisConfig.titleFontSize,rotation:270,verticalPos:"top",horizontalPos:"center"}]}),e}getDrawableElementsForBottomAxis(){let e=[];if(this.showAxisLine){let r=this.boundingRect.y+this.axisConfig.axisLineWidth/2;e.push({type:"path",groupTexts:["bottom-axis","axis-line"],data:[{path:`M ${this.boundingRect.x},${r} L ${this.boundingRect.x+this.boundingRect.width},${r}`,strokeFill:this.axisThemeConfig.axisLineColor,strokeWidth:this.axisConfig.axisLineWidth}]})}if(this.showLabel&&e.push({type:"text",groupTexts:["bottom-axis","label"],data:this.getTickValues().map(r=>({text:r.toString(),x:this.getScaleValue(r),y:this.boundingRect.y+this.axisConfig.labelPadding+(this.showTick?this.axisConfig.tickLength:0)+(this.showAxisLine?this.axisConfig.axisLineWidth:0),fill:this.axisThemeConfig.labelColor,fontSize:this.axisConfig.labelFontSize,rotation:0,verticalPos:"top",horizontalPos:"center"}))}),this.showTick){let r=this.boundingRect.y+(this.showAxisLine?this.axisConfig.axisLineWidth:0);e.push({type:"path",groupTexts:["bottom-axis","ticks"],data:this.getTickValues().map(n=>({path:`M ${this.getScaleValue(n)},${r} L ${this.getScaleValue(n)},${r+this.axisConfig.tickLength}`,strokeFill:this.axisThemeConfig.tickColor,strokeWidth:this.axisConfig.tickWidth}))})}return this.showTitle&&e.push({type:"text",groupTexts:["bottom-axis","title"],data:[{text:this.title,x:this.range[0]+(this.range[1]-this.range[0])/2,y:this.boundingRect.y+this.boundingRect.height-this.axisConfig.titlePadding-this.titleTextHeight,fill:this.axisThemeConfig.titleColor,fontSize:this.axisConfig.titleFontSize,rotation:0,verticalPos:"top",horizontalPos:"center"}]}),e}getDrawableElementsForTopAxis(){let e=[];if(this.showAxisLine){let r=this.boundingRect.y+this.boundingRect.height-this.axisConfig.axisLineWidth/2;e.push({type:"path",groupTexts:["top-axis","axis-line"],data:[{path:`M ${this.boundingRect.x},${r} L ${this.boundingRect.x+this.boundingRect.width},${r}`,strokeFill:this.axisThemeConfig.axisLineColor,strokeWidth:this.axisConfig.axisLineWidth}]})}if(this.showLabel&&e.push({type:"text",groupTexts:["top-axis","label"],data:this.getTickValues().map(r=>({text:r.toString(),x:this.getScaleValue(r),y:this.boundingRect.y+(this.showTitle?this.titleTextHeight+this.axisConfig.titlePadding*2:0)+this.axisConfig.labelPadding,fill:this.axisThemeConfig.labelColor,fontSize:this.axisConfig.labelFontSize,rotation:0,verticalPos:"top",horizontalPos:"center"}))}),this.showTick){let r=this.boundingRect.y;e.push({type:"path",groupTexts:["top-axis","ticks"],data:this.getTickValues().map(n=>({path:`M ${this.getScaleValue(n)},${r+this.boundingRect.height-(this.showAxisLine?this.axisConfig.axisLineWidth:0)} L ${this.getScaleValue(n)},${r+this.boundingRect.height-this.axisConfig.tickLength-(this.showAxisLine?this.axisConfig.axisLineWidth:0)}`,strokeFill:this.axisThemeConfig.tickColor,strokeWidth:this.axisConfig.tickWidth}))})}return this.showTitle&&e.push({type:"text",groupTexts:["top-axis","title"],data:[{text:this.title,x:this.boundingRect.x+this.boundingRect.width/2,y:this.boundingRect.y+this.axisConfig.titlePadding,fill:this.axisThemeConfig.titleColor,fontSize:this.axisConfig.titleFontSize,rotation:0,verticalPos:"top",horizontalPos:"center"}]}),e}getDrawableElements(){if(this.axisPosition==="left")return this.getDrawableElementsForLeftAxis();if(this.axisPosition==="right")throw Error("Drawing of right axis is not implemented");return this.axisPosition==="bottom"?this.getDrawableElementsForBottomAxis():this.axisPosition==="top"?this.getDrawableElementsForTopAxis():[]}}});var b6,Che=N(()=>{"use strict";dr();vt();yO();b6=class extends A1{static{o(this,"BandAxis")}constructor(e,r,n,i,a){super(e,i,a,r),this.categories=n,this.scale=L0().domain(this.categories).range(this.getRange())}setRange(e){super.setRange(e)}recalculateScale(){this.scale=L0().domain(this.categories).range(this.getRange()).paddingInner(1).paddingOuter(0).align(.5),Y.trace("BandAxis axis final categories, range: ",this.categories,this.getRange())}getTickValues(){return this.categories}getScaleValue(e){return this.scale(e)??this.getRange()[0]}}});var w6,Ahe=N(()=>{"use strict";dr();yO();w6=class extends A1{static{o(this,"LinearAxis")}constructor(e,r,n,i,a){super(e,i,a,r),this.domain=n,this.scale=gl().domain(this.domain).range(this.getRange())}getTickValues(){return this.scale.ticks()}recalculateScale(){let e=[...this.domain];this.axisPosition==="left"&&e.reverse(),this.scale=gl().domain(e).range(this.getRange())}getScaleValue(e){return this.scale(e)}}});function vO(t,e,r,n){let i=new C1(n);return v6(t)?new b6(e,r,t.categories,t.title,i):new w6(e,r,[t.min,t.max],t.title,i)}var _he=N(()=>{"use strict";x6();gO();Che();Ahe();o(vO,"getAxis")});function Dhe(t,e,r,n){let i=new C1(n);return new xO(i,t,e,r)}var xO,Lhe=N(()=>{"use strict";gO();xO=class{constructor(e,r,n,i){this.textDimensionCalculator=e;this.chartConfig=r;this.chartData=n;this.chartThemeConfig=i;this.boundingRect={x:0,y:0,width:0,height:0},this.showChartTitle=!1}static{o(this,"ChartTitle")}setBoundingBoxXY(e){this.boundingRect.x=e.x,this.boundingRect.y=e.y}calculateSpace(e){let r=this.textDimensionCalculator.getMaxDimension([this.chartData.title],this.chartConfig.titleFontSize),n=Math.max(r.width,e.width),i=r.height+2*this.chartConfig.titlePadding;return r.width<=n&&r.height<=i&&this.chartConfig.showTitle&&this.chartData.title&&(this.boundingRect.width=n,this.boundingRect.height=i,this.showChartTitle=!0),{width:this.boundingRect.width,height:this.boundingRect.height}}getDrawableElements(){let e=[];return this.showChartTitle&&e.push({groupTexts:["chart-title"],type:"text",data:[{fontSize:this.chartConfig.titleFontSize,text:this.chartData.title,verticalPos:"middle",horizontalPos:"center",x:this.boundingRect.x+this.boundingRect.width/2,y:this.boundingRect.y+this.boundingRect.height/2,fill:this.chartThemeConfig.titleColor,rotation:0}]}),e}};o(Dhe,"getChartTitleComponent")});var T6,Rhe=N(()=>{"use strict";dr();T6=class{constructor(e,r,n,i,a){this.plotData=e;this.xAxis=r;this.yAxis=n;this.orientation=i;this.plotIndex=a}static{o(this,"LinePlot")}getDrawableElement(){let e=this.plotData.data.map(n=>[this.xAxis.getScaleValue(n[0]),this.yAxis.getScaleValue(n[1])]),r;return this.orientation==="horizontal"?r=wl().y(n=>n[0]).x(n=>n[1])(e):r=wl().x(n=>n[0]).y(n=>n[1])(e),r?[{groupTexts:["plot",`line-plot-${this.plotIndex}`],type:"path",data:[{path:r,strokeFill:this.plotData.strokeFill,strokeWidth:this.plotData.strokeWidth}]}]:[]}}});var k6,Nhe=N(()=>{"use strict";k6=class{constructor(e,r,n,i,a,s){this.barData=e;this.boundingRect=r;this.xAxis=n;this.yAxis=i;this.orientation=a;this.plotIndex=s}static{o(this,"BarPlot")}getDrawableElement(){let e=this.barData.data.map(a=>[this.xAxis.getScaleValue(a[0]),this.yAxis.getScaleValue(a[1])]),n=Math.min(this.xAxis.getAxisOuterPadding()*2,this.xAxis.getTickDistance())*(1-.05),i=n/2;return this.orientation==="horizontal"?[{groupTexts:["plot",`bar-plot-${this.plotIndex}`],type:"rect",data:e.map(a=>({x:this.boundingRect.x,y:a[0]-i,height:n,width:a[1]-this.boundingRect.x,fill:this.barData.fill,strokeWidth:0,strokeFill:this.barData.fill}))}]:[{groupTexts:["plot",`bar-plot-${this.plotIndex}`],type:"rect",data:e.map(a=>({x:a[0]-i,y:a[1],width:n,height:this.boundingRect.y+this.boundingRect.height-a[1],fill:this.barData.fill,strokeWidth:0,strokeFill:this.barData.fill}))}]}}});function Mhe(t,e,r){return new bO(t,e,r)}var bO,Ihe=N(()=>{"use strict";Rhe();Nhe();bO=class{constructor(e,r,n){this.chartConfig=e;this.chartData=r;this.chartThemeConfig=n;this.boundingRect={x:0,y:0,width:0,height:0}}static{o(this,"BasePlot")}setAxes(e,r){this.xAxis=e,this.yAxis=r}setBoundingBoxXY(e){this.boundingRect.x=e.x,this.boundingRect.y=e.y}calculateSpace(e){return this.boundingRect.width=e.width,this.boundingRect.height=e.height,{width:this.boundingRect.width,height:this.boundingRect.height}}getDrawableElements(){if(!(this.xAxis&&this.yAxis))throw Error("Axes must be passed to render Plots");let e=[];for(let[r,n]of this.chartData.plots.entries())switch(n.type){case"line":{let i=new T6(n,this.xAxis,this.yAxis,this.chartConfig.chartOrientation,r);e.push(...i.getDrawableElement())}break;case"bar":{let i=new k6(n,this.boundingRect,this.xAxis,this.yAxis,this.chartConfig.chartOrientation,r);e.push(...i.getDrawableElement())}break}return e}};o(Mhe,"getPlotComponent")});var E6,Ohe=N(()=>{"use strict";_he();Lhe();Ihe();x6();E6=class{constructor(e,r,n,i){this.chartConfig=e;this.chartData=r;this.componentStore={title:Dhe(e,r,n,i),plot:Mhe(e,r,n),xAxis:vO(r.xAxis,e.xAxis,{titleColor:n.xAxisTitleColor,labelColor:n.xAxisLabelColor,tickColor:n.xAxisTickColor,axisLineColor:n.xAxisLineColor},i),yAxis:vO(r.yAxis,e.yAxis,{titleColor:n.yAxisTitleColor,labelColor:n.yAxisLabelColor,tickColor:n.yAxisTickColor,axisLineColor:n.yAxisLineColor},i)}}static{o(this,"Orchestrator")}calculateVerticalSpace(){let e=this.chartConfig.width,r=this.chartConfig.height,n=0,i=0,a=Math.floor(e*this.chartConfig.plotReservedSpacePercent/100),s=Math.floor(r*this.chartConfig.plotReservedSpacePercent/100),l=this.componentStore.plot.calculateSpace({width:a,height:s});e-=l.width,r-=l.height,l=this.componentStore.title.calculateSpace({width:this.chartConfig.width,height:r}),i=l.height,r-=l.height,this.componentStore.xAxis.setAxisPosition("bottom"),l=this.componentStore.xAxis.calculateSpace({width:e,height:r}),r-=l.height,this.componentStore.yAxis.setAxisPosition("left"),l=this.componentStore.yAxis.calculateSpace({width:e,height:r}),n=l.width,e-=l.width,e>0&&(a+=e,e=0),r>0&&(s+=r,r=0),this.componentStore.plot.calculateSpace({width:a,height:s}),this.componentStore.plot.setBoundingBoxXY({x:n,y:i}),this.componentStore.xAxis.setRange([n,n+a]),this.componentStore.xAxis.setBoundingBoxXY({x:n,y:i+s}),this.componentStore.yAxis.setRange([i,i+s]),this.componentStore.yAxis.setBoundingBoxXY({x:0,y:i}),this.chartData.plots.some(u=>mO(u))&&this.componentStore.xAxis.recalculateOuterPaddingToDrawBar()}calculateHorizontalSpace(){let e=this.chartConfig.width,r=this.chartConfig.height,n=0,i=0,a=0,s=Math.floor(e*this.chartConfig.plotReservedSpacePercent/100),l=Math.floor(r*this.chartConfig.plotReservedSpacePercent/100),u=this.componentStore.plot.calculateSpace({width:s,height:l});e-=u.width,r-=u.height,u=this.componentStore.title.calculateSpace({width:this.chartConfig.width,height:r}),n=u.height,r-=u.height,this.componentStore.xAxis.setAxisPosition("left"),u=this.componentStore.xAxis.calculateSpace({width:e,height:r}),e-=u.width,i=u.width,this.componentStore.yAxis.setAxisPosition("top"),u=this.componentStore.yAxis.calculateSpace({width:e,height:r}),r-=u.height,a=n+u.height,e>0&&(s+=e,e=0),r>0&&(l+=r,r=0),this.componentStore.plot.calculateSpace({width:s,height:l}),this.componentStore.plot.setBoundingBoxXY({x:i,y:a}),this.componentStore.yAxis.setRange([i,i+s]),this.componentStore.yAxis.setBoundingBoxXY({x:i,y:n}),this.componentStore.xAxis.setRange([a,a+l]),this.componentStore.xAxis.setBoundingBoxXY({x:0,y:a}),this.chartData.plots.some(h=>mO(h))&&this.componentStore.xAxis.recalculateOuterPaddingToDrawBar()}calculateSpace(){this.chartConfig.chartOrientation==="horizontal"?this.calculateHorizontalSpace():this.calculateVerticalSpace()}getDrawableElement(){this.calculateSpace();let e=[];this.componentStore.plot.setAxes(this.componentStore.xAxis,this.componentStore.yAxis);for(let r of Object.values(this.componentStore))e.push(...r.getDrawableElements());return e}}});var S6,Phe=N(()=>{"use strict";Ohe();S6=class{static{o(this,"XYChartBuilder")}static build(e,r,n,i){return new E6(e,r,n,i).getDrawableElement()}}});function Fhe(){let t=oh(),e=cr();return Fi(t.xyChart,e.themeVariables.xyChart)}function $he(){let t=cr();return Fi(or.xyChart,t.xyChart)}function zhe(){return{yAxis:{type:"linear",title:"",min:1/0,max:-1/0},xAxis:{type:"band",title:"",categories:[]},title:"",plots:[]}}function kO(t){let e=cr();return Tr(t.trim(),e)}function IGe(t){Bhe=t}function OGe(t){t==="horizontal"?bb.chartOrientation="horizontal":bb.chartOrientation="vertical"}function PGe(t){fn.xAxis.title=kO(t.text)}function Ghe(t,e){fn.xAxis={type:"linear",title:fn.xAxis.title,min:t,max:e},C6=!0}function BGe(t){fn.xAxis={type:"band",title:fn.xAxis.title,categories:t.map(e=>kO(e.text))},C6=!0}function FGe(t){fn.yAxis.title=kO(t.text)}function $Ge(t,e){fn.yAxis={type:"linear",title:fn.yAxis.title,min:t,max:e},TO=!0}function zGe(t){let e=Math.min(...t),r=Math.max(...t),n=S1(fn.yAxis)?fn.yAxis.min:1/0,i=S1(fn.yAxis)?fn.yAxis.max:-1/0;fn.yAxis={type:"linear",title:fn.yAxis.title,min:Math.min(n,e),max:Math.max(i,r)}}function Vhe(t){let e=[];if(t.length===0)return e;if(!C6){let r=S1(fn.xAxis)?fn.xAxis.min:1/0,n=S1(fn.xAxis)?fn.xAxis.max:-1/0;Ghe(Math.min(r,1),Math.max(n,t.length))}if(TO||zGe(t),v6(fn.xAxis)&&(e=fn.xAxis.categories.map((r,n)=>[r,t[n]])),S1(fn.xAxis)){let r=fn.xAxis.min,n=fn.xAxis.max,i=(n-r)/(t.length-1),a=[];for(let s=r;s<=n;s+=i)a.push(`${s}`);e=a.map((s,l)=>[s,t[l]])}return e}function Uhe(t){return wO[t===0?0:t%wO.length]}function GGe(t,e){let r=Vhe(e);fn.plots.push({type:"line",strokeFill:Uhe(xb),strokeWidth:2,data:r}),xb++}function VGe(t,e){let r=Vhe(e);fn.plots.push({type:"bar",fill:Uhe(xb),data:r}),xb++}function UGe(){if(fn.plots.length===0)throw Error("No Plot to render, please provide a plot with some data");return fn.title=Ir(),S6.build(bb,fn,wb,Bhe)}function HGe(){return wb}function WGe(){return bb}var xb,Bhe,bb,wb,fn,wO,C6,TO,qGe,Hhe,Whe=N(()=>{"use strict";ji();Ya();_y();ir();gr();mi();Phe();x6();xb=0,bb=$he(),wb=Fhe(),fn=zhe(),wO=wb.plotColorPalette.split(",").map(t=>t.trim()),C6=!1,TO=!1;o(Fhe,"getChartDefaultThemeConfig");o($he,"getChartDefaultConfig");o(zhe,"getChartDefaultData");o(kO,"textSanitizer");o(IGe,"setTmpSVGG");o(OGe,"setOrientation");o(PGe,"setXAxisTitle");o(Ghe,"setXAxisRangeData");o(BGe,"setXAxisBand");o(FGe,"setYAxisTitle");o($Ge,"setYAxisRangeData");o(zGe,"setYAxisRangeFromPlotData");o(Vhe,"transformDataWithoutCategory");o(Uhe,"getPlotColorFromPalette");o(GGe,"setLineData");o(VGe,"setBarData");o(UGe,"getDrawableElem");o(HGe,"getChartThemeConfig");o(WGe,"getChartConfig");qGe=o(function(){Ar(),xb=0,bb=$he(),fn=zhe(),wb=Fhe(),wO=wb.plotColorPalette.split(",").map(t=>t.trim()),C6=!1,TO=!1},"clear"),Hhe={getDrawableElem:UGe,clear:qGe,setAccTitle:Lr,getAccTitle:Rr,setDiagramTitle:$r,getDiagramTitle:Ir,getAccDescription:Mr,setAccDescription:Nr,setOrientation:OGe,setXAxisTitle:PGe,setXAxisRangeData:Ghe,setXAxisBand:BGe,setYAxisTitle:FGe,setYAxisRangeData:$Ge,setLineData:GGe,setBarData:VGe,setTmpSVGG:IGe,getChartThemeConfig:HGe,getChartConfig:WGe}});var YGe,qhe,Yhe=N(()=>{"use strict";vt();Vc();Ei();YGe=o((t,e,r,n)=>{let i=n.db,a=i.getChartThemeConfig(),s=i.getChartConfig();function l(v){return v==="top"?"text-before-edge":"middle"}o(l,"getDominantBaseLine");function u(v){return v==="left"?"start":v==="right"?"end":"middle"}o(u,"getTextAnchor");function h(v){return`translate(${v.x}, ${v.y}) rotate(${v.rotation||0})`}o(h,"getTextTransformation"),Y.debug(`Rendering xychart chart +`+t);let f=sa(e),d=f.append("g").attr("class","main"),p=d.append("rect").attr("width",s.width).attr("height",s.height).attr("class","background");vn(f,s.height,s.width,!0),f.attr("viewBox",`0 0 ${s.width} ${s.height}`),p.attr("fill",a.backgroundColor),i.setTmpSVGG(f.append("g").attr("class","mermaid-tmp-group"));let m=i.getDrawableElem(),g={};function y(v){let x=d,b="";for(let[w]of v.entries()){let C=d;w>0&&g[b]&&(C=g[b]),b+=v[w],x=g[b],x||(x=g[b]=C.append("g").attr("class",v[w]))}return x}o(y,"getGroup");for(let v of m){if(v.data.length===0)continue;let x=y(v.groupTexts);switch(v.type){case"rect":x.selectAll("rect").data(v.data).enter().append("rect").attr("x",b=>b.x).attr("y",b=>b.y).attr("width",b=>b.width).attr("height",b=>b.height).attr("fill",b=>b.fill).attr("stroke",b=>b.strokeFill).attr("stroke-width",b=>b.strokeWidth);break;case"text":x.selectAll("text").data(v.data).enter().append("text").attr("x",0).attr("y",0).attr("fill",b=>b.fill).attr("font-size",b=>b.fontSize).attr("dominant-baseline",b=>l(b.verticalPos)).attr("text-anchor",b=>u(b.horizontalPos)).attr("transform",b=>h(b)).text(b=>b.text);break;case"path":x.selectAll("path").data(v.data).enter().append("path").attr("d",b=>b.path).attr("fill",b=>b.fill?b.fill:"none").attr("stroke",b=>b.strokeFill).attr("stroke-width",b=>b.strokeWidth);break}}},"draw"),qhe={draw:YGe}});var Xhe={};hr(Xhe,{diagram:()=>XGe});var XGe,jhe=N(()=>{"use strict";She();Whe();Yhe();XGe={parser:Ehe,db:Hhe,renderer:qhe}});var EO,Zhe,Jhe=N(()=>{"use strict";EO=function(){var t=o(function(re,oe,V,xe){for(V=V||{},xe=re.length;xe--;V[re[xe]]=oe);return V},"o"),e=[1,3],r=[1,4],n=[1,5],i=[1,6],a=[5,6,8,9,11,13,21,22,23,24,41,42,43,44,45,46,54,72,74,77,89,90],s=[1,22],l=[2,7],u=[1,26],h=[1,27],f=[1,28],d=[1,29],p=[1,33],m=[1,34],g=[1,35],y=[1,36],v=[1,37],x=[1,38],b=[1,24],w=[1,31],C=[1,32],T=[1,30],E=[1,39],A=[1,40],S=[5,8,9,11,13,21,22,23,24,41,42,43,44,45,46,54,72,74,77,89,90],_=[1,61],I=[89,90],D=[5,8,9,11,13,21,22,23,24,27,29,41,42,43,44,45,46,54,61,63,72,74,75,76,77,80,81,82,83,84,85,86,87,88,89,90],k=[27,29],L=[1,70],R=[1,71],O=[1,72],M=[1,73],B=[1,74],F=[1,75],P=[1,76],z=[1,83],$=[1,80],H=[1,84],Q=[1,85],j=[1,86],ie=[1,87],ne=[1,88],le=[1,89],he=[1,90],K=[1,91],X=[1,92],te=[5,8,9,11,13,21,22,23,24,27,41,42,43,44,45,46,54,72,74,75,76,77,80,81,82,83,84,85,86,87,88,89,90],J=[63,64],se=[1,101],ue=[5,8,9,11,13,21,22,23,24,41,42,43,44,45,46,54,72,74,76,77,89,90],Z=[5,8,9,11,13,21,22,23,24,41,42,43,44,45,46,54,72,74,75,76,77,80,81,82,83,84,85,86,87,88,89,90],Se=[1,110],ce=[1,106],ae=[1,107],Oe=[1,108],ge=[1,109],ze=[1,111],He=[1,116],$e=[1,117],Re=[1,114],Ie=[1,115],be={trace:o(function(){},"trace"),yy:{},symbols_:{error:2,start:3,directive:4,NEWLINE:5,RD:6,diagram:7,EOF:8,acc_title:9,acc_title_value:10,acc_descr:11,acc_descr_value:12,acc_descr_multiline_value:13,requirementDef:14,elementDef:15,relationshipDef:16,direction:17,styleStatement:18,classDefStatement:19,classStatement:20,direction_tb:21,direction_bt:22,direction_rl:23,direction_lr:24,requirementType:25,requirementName:26,STRUCT_START:27,requirementBody:28,STYLE_SEPARATOR:29,idList:30,ID:31,COLONSEP:32,id:33,TEXT:34,text:35,RISK:36,riskLevel:37,VERIFYMTHD:38,verifyType:39,STRUCT_STOP:40,REQUIREMENT:41,FUNCTIONAL_REQUIREMENT:42,INTERFACE_REQUIREMENT:43,PERFORMANCE_REQUIREMENT:44,PHYSICAL_REQUIREMENT:45,DESIGN_CONSTRAINT:46,LOW_RISK:47,MED_RISK:48,HIGH_RISK:49,VERIFY_ANALYSIS:50,VERIFY_DEMONSTRATION:51,VERIFY_INSPECTION:52,VERIFY_TEST:53,ELEMENT:54,elementName:55,elementBody:56,TYPE:57,type:58,DOCREF:59,ref:60,END_ARROW_L:61,relationship:62,LINE:63,END_ARROW_R:64,CONTAINS:65,COPIES:66,DERIVES:67,SATISFIES:68,VERIFIES:69,REFINES:70,TRACES:71,CLASSDEF:72,stylesOpt:73,CLASS:74,ALPHA:75,COMMA:76,STYLE:77,style:78,styleComponent:79,NUM:80,COLON:81,UNIT:82,SPACE:83,BRKT:84,PCT:85,MINUS:86,LABEL:87,SEMICOLON:88,unqString:89,qString:90,$accept:0,$end:1},terminals_:{2:"error",5:"NEWLINE",6:"RD",8:"EOF",9:"acc_title",10:"acc_title_value",11:"acc_descr",12:"acc_descr_value",13:"acc_descr_multiline_value",21:"direction_tb",22:"direction_bt",23:"direction_rl",24:"direction_lr",27:"STRUCT_START",29:"STYLE_SEPARATOR",31:"ID",32:"COLONSEP",34:"TEXT",36:"RISK",38:"VERIFYMTHD",40:"STRUCT_STOP",41:"REQUIREMENT",42:"FUNCTIONAL_REQUIREMENT",43:"INTERFACE_REQUIREMENT",44:"PERFORMANCE_REQUIREMENT",45:"PHYSICAL_REQUIREMENT",46:"DESIGN_CONSTRAINT",47:"LOW_RISK",48:"MED_RISK",49:"HIGH_RISK",50:"VERIFY_ANALYSIS",51:"VERIFY_DEMONSTRATION",52:"VERIFY_INSPECTION",53:"VERIFY_TEST",54:"ELEMENT",57:"TYPE",59:"DOCREF",61:"END_ARROW_L",63:"LINE",64:"END_ARROW_R",65:"CONTAINS",66:"COPIES",67:"DERIVES",68:"SATISFIES",69:"VERIFIES",70:"REFINES",71:"TRACES",72:"CLASSDEF",74:"CLASS",75:"ALPHA",76:"COMMA",77:"STYLE",80:"NUM",81:"COLON",82:"UNIT",83:"SPACE",84:"BRKT",85:"PCT",86:"MINUS",87:"LABEL",88:"SEMICOLON",89:"unqString",90:"qString"},productions_:[0,[3,3],[3,2],[3,4],[4,2],[4,2],[4,1],[7,0],[7,2],[7,2],[7,2],[7,2],[7,2],[7,2],[7,2],[7,2],[7,2],[17,1],[17,1],[17,1],[17,1],[14,5],[14,7],[28,5],[28,5],[28,5],[28,5],[28,2],[28,1],[25,1],[25,1],[25,1],[25,1],[25,1],[25,1],[37,1],[37,1],[37,1],[39,1],[39,1],[39,1],[39,1],[15,5],[15,7],[56,5],[56,5],[56,2],[56,1],[16,5],[16,5],[62,1],[62,1],[62,1],[62,1],[62,1],[62,1],[62,1],[19,3],[20,3],[20,3],[30,1],[30,3],[30,1],[30,3],[18,3],[73,1],[73,3],[78,1],[78,2],[79,1],[79,1],[79,1],[79,1],[79,1],[79,1],[79,1],[79,1],[79,1],[79,1],[26,1],[26,1],[33,1],[33,1],[35,1],[35,1],[55,1],[55,1],[58,1],[58,1],[60,1],[60,1]],performAction:o(function(oe,V,xe,q,pe,ve,Pe){var _e=ve.length-1;switch(pe){case 4:this.$=ve[_e].trim(),q.setAccTitle(this.$);break;case 5:case 6:this.$=ve[_e].trim(),q.setAccDescription(this.$);break;case 7:this.$=[];break;case 17:q.setDirection("TB");break;case 18:q.setDirection("BT");break;case 19:q.setDirection("RL");break;case 20:q.setDirection("LR");break;case 21:q.addRequirement(ve[_e-3],ve[_e-4]);break;case 22:q.addRequirement(ve[_e-5],ve[_e-6]),q.setClass([ve[_e-5]],ve[_e-3]);break;case 23:q.setNewReqId(ve[_e-2]);break;case 24:q.setNewReqText(ve[_e-2]);break;case 25:q.setNewReqRisk(ve[_e-2]);break;case 26:q.setNewReqVerifyMethod(ve[_e-2]);break;case 29:this.$=q.RequirementType.REQUIREMENT;break;case 30:this.$=q.RequirementType.FUNCTIONAL_REQUIREMENT;break;case 31:this.$=q.RequirementType.INTERFACE_REQUIREMENT;break;case 32:this.$=q.RequirementType.PERFORMANCE_REQUIREMENT;break;case 33:this.$=q.RequirementType.PHYSICAL_REQUIREMENT;break;case 34:this.$=q.RequirementType.DESIGN_CONSTRAINT;break;case 35:this.$=q.RiskLevel.LOW_RISK;break;case 36:this.$=q.RiskLevel.MED_RISK;break;case 37:this.$=q.RiskLevel.HIGH_RISK;break;case 38:this.$=q.VerifyType.VERIFY_ANALYSIS;break;case 39:this.$=q.VerifyType.VERIFY_DEMONSTRATION;break;case 40:this.$=q.VerifyType.VERIFY_INSPECTION;break;case 41:this.$=q.VerifyType.VERIFY_TEST;break;case 42:q.addElement(ve[_e-3]);break;case 43:q.addElement(ve[_e-5]),q.setClass([ve[_e-5]],ve[_e-3]);break;case 44:q.setNewElementType(ve[_e-2]);break;case 45:q.setNewElementDocRef(ve[_e-2]);break;case 48:q.addRelationship(ve[_e-2],ve[_e],ve[_e-4]);break;case 49:q.addRelationship(ve[_e-2],ve[_e-4],ve[_e]);break;case 50:this.$=q.Relationships.CONTAINS;break;case 51:this.$=q.Relationships.COPIES;break;case 52:this.$=q.Relationships.DERIVES;break;case 53:this.$=q.Relationships.SATISFIES;break;case 54:this.$=q.Relationships.VERIFIES;break;case 55:this.$=q.Relationships.REFINES;break;case 56:this.$=q.Relationships.TRACES;break;case 57:this.$=ve[_e-2],q.defineClass(ve[_e-1],ve[_e]);break;case 58:q.setClass(ve[_e-1],ve[_e]);break;case 59:q.setClass([ve[_e-2]],ve[_e]);break;case 60:case 62:this.$=[ve[_e]];break;case 61:case 63:this.$=ve[_e-2].concat([ve[_e]]);break;case 64:this.$=ve[_e-2],q.setCssStyle(ve[_e-1],ve[_e]);break;case 65:this.$=[ve[_e]];break;case 66:ve[_e-2].push(ve[_e]),this.$=ve[_e-2];break;case 68:this.$=ve[_e-1]+ve[_e];break}},"anonymous"),table:[{3:1,4:2,6:e,9:r,11:n,13:i},{1:[3]},{3:8,4:2,5:[1,7],6:e,9:r,11:n,13:i},{5:[1,9]},{10:[1,10]},{12:[1,11]},t(a,[2,6]),{3:12,4:2,6:e,9:r,11:n,13:i},{1:[2,2]},{4:17,5:s,7:13,8:l,9:r,11:n,13:i,14:14,15:15,16:16,17:18,18:19,19:20,20:21,21:u,22:h,23:f,24:d,25:23,33:25,41:p,42:m,43:g,44:y,45:v,46:x,54:b,72:w,74:C,77:T,89:E,90:A},t(a,[2,4]),t(a,[2,5]),{1:[2,1]},{8:[1,41]},{4:17,5:s,7:42,8:l,9:r,11:n,13:i,14:14,15:15,16:16,17:18,18:19,19:20,20:21,21:u,22:h,23:f,24:d,25:23,33:25,41:p,42:m,43:g,44:y,45:v,46:x,54:b,72:w,74:C,77:T,89:E,90:A},{4:17,5:s,7:43,8:l,9:r,11:n,13:i,14:14,15:15,16:16,17:18,18:19,19:20,20:21,21:u,22:h,23:f,24:d,25:23,33:25,41:p,42:m,43:g,44:y,45:v,46:x,54:b,72:w,74:C,77:T,89:E,90:A},{4:17,5:s,7:44,8:l,9:r,11:n,13:i,14:14,15:15,16:16,17:18,18:19,19:20,20:21,21:u,22:h,23:f,24:d,25:23,33:25,41:p,42:m,43:g,44:y,45:v,46:x,54:b,72:w,74:C,77:T,89:E,90:A},{4:17,5:s,7:45,8:l,9:r,11:n,13:i,14:14,15:15,16:16,17:18,18:19,19:20,20:21,21:u,22:h,23:f,24:d,25:23,33:25,41:p,42:m,43:g,44:y,45:v,46:x,54:b,72:w,74:C,77:T,89:E,90:A},{4:17,5:s,7:46,8:l,9:r,11:n,13:i,14:14,15:15,16:16,17:18,18:19,19:20,20:21,21:u,22:h,23:f,24:d,25:23,33:25,41:p,42:m,43:g,44:y,45:v,46:x,54:b,72:w,74:C,77:T,89:E,90:A},{4:17,5:s,7:47,8:l,9:r,11:n,13:i,14:14,15:15,16:16,17:18,18:19,19:20,20:21,21:u,22:h,23:f,24:d,25:23,33:25,41:p,42:m,43:g,44:y,45:v,46:x,54:b,72:w,74:C,77:T,89:E,90:A},{4:17,5:s,7:48,8:l,9:r,11:n,13:i,14:14,15:15,16:16,17:18,18:19,19:20,20:21,21:u,22:h,23:f,24:d,25:23,33:25,41:p,42:m,43:g,44:y,45:v,46:x,54:b,72:w,74:C,77:T,89:E,90:A},{4:17,5:s,7:49,8:l,9:r,11:n,13:i,14:14,15:15,16:16,17:18,18:19,19:20,20:21,21:u,22:h,23:f,24:d,25:23,33:25,41:p,42:m,43:g,44:y,45:v,46:x,54:b,72:w,74:C,77:T,89:E,90:A},{4:17,5:s,7:50,8:l,9:r,11:n,13:i,14:14,15:15,16:16,17:18,18:19,19:20,20:21,21:u,22:h,23:f,24:d,25:23,33:25,41:p,42:m,43:g,44:y,45:v,46:x,54:b,72:w,74:C,77:T,89:E,90:A},{26:51,89:[1,52],90:[1,53]},{55:54,89:[1,55],90:[1,56]},{29:[1,59],61:[1,57],63:[1,58]},t(S,[2,17]),t(S,[2,18]),t(S,[2,19]),t(S,[2,20]),{30:60,33:62,75:_,89:E,90:A},{30:63,33:62,75:_,89:E,90:A},{30:64,33:62,75:_,89:E,90:A},t(I,[2,29]),t(I,[2,30]),t(I,[2,31]),t(I,[2,32]),t(I,[2,33]),t(I,[2,34]),t(D,[2,81]),t(D,[2,82]),{1:[2,3]},{8:[2,8]},{8:[2,9]},{8:[2,10]},{8:[2,11]},{8:[2,12]},{8:[2,13]},{8:[2,14]},{8:[2,15]},{8:[2,16]},{27:[1,65],29:[1,66]},t(k,[2,79]),t(k,[2,80]),{27:[1,67],29:[1,68]},t(k,[2,85]),t(k,[2,86]),{62:69,65:L,66:R,67:O,68:M,69:B,70:F,71:P},{62:77,65:L,66:R,67:O,68:M,69:B,70:F,71:P},{30:78,33:62,75:_,89:E,90:A},{73:79,75:z,76:$,78:81,79:82,80:H,81:Q,82:j,83:ie,84:ne,85:le,86:he,87:K,88:X},t(te,[2,60]),t(te,[2,62]),{73:93,75:z,76:$,78:81,79:82,80:H,81:Q,82:j,83:ie,84:ne,85:le,86:he,87:K,88:X},{30:94,33:62,75:_,76:$,89:E,90:A},{5:[1,95]},{30:96,33:62,75:_,89:E,90:A},{5:[1,97]},{30:98,33:62,75:_,89:E,90:A},{63:[1,99]},t(J,[2,50]),t(J,[2,51]),t(J,[2,52]),t(J,[2,53]),t(J,[2,54]),t(J,[2,55]),t(J,[2,56]),{64:[1,100]},t(S,[2,59],{76:$}),t(S,[2,64],{76:se}),{33:103,75:[1,102],89:E,90:A},t(ue,[2,65],{79:104,75:z,80:H,81:Q,82:j,83:ie,84:ne,85:le,86:he,87:K,88:X}),t(Z,[2,67]),t(Z,[2,69]),t(Z,[2,70]),t(Z,[2,71]),t(Z,[2,72]),t(Z,[2,73]),t(Z,[2,74]),t(Z,[2,75]),t(Z,[2,76]),t(Z,[2,77]),t(Z,[2,78]),t(S,[2,57],{76:se}),t(S,[2,58],{76:$}),{5:Se,28:105,31:ce,34:ae,36:Oe,38:ge,40:ze},{27:[1,112],76:$},{5:He,40:$e,56:113,57:Re,59:Ie},{27:[1,118],76:$},{33:119,89:E,90:A},{33:120,89:E,90:A},{75:z,78:121,79:82,80:H,81:Q,82:j,83:ie,84:ne,85:le,86:he,87:K,88:X},t(te,[2,61]),t(te,[2,63]),t(Z,[2,68]),t(S,[2,21]),{32:[1,122]},{32:[1,123]},{32:[1,124]},{32:[1,125]},{5:Se,28:126,31:ce,34:ae,36:Oe,38:ge,40:ze},t(S,[2,28]),{5:[1,127]},t(S,[2,42]),{32:[1,128]},{32:[1,129]},{5:He,40:$e,56:130,57:Re,59:Ie},t(S,[2,47]),{5:[1,131]},t(S,[2,48]),t(S,[2,49]),t(ue,[2,66],{79:104,75:z,80:H,81:Q,82:j,83:ie,84:ne,85:le,86:he,87:K,88:X}),{33:132,89:E,90:A},{35:133,89:[1,134],90:[1,135]},{37:136,47:[1,137],48:[1,138],49:[1,139]},{39:140,50:[1,141],51:[1,142],52:[1,143],53:[1,144]},t(S,[2,27]),{5:Se,28:145,31:ce,34:ae,36:Oe,38:ge,40:ze},{58:146,89:[1,147],90:[1,148]},{60:149,89:[1,150],90:[1,151]},t(S,[2,46]),{5:He,40:$e,56:152,57:Re,59:Ie},{5:[1,153]},{5:[1,154]},{5:[2,83]},{5:[2,84]},{5:[1,155]},{5:[2,35]},{5:[2,36]},{5:[2,37]},{5:[1,156]},{5:[2,38]},{5:[2,39]},{5:[2,40]},{5:[2,41]},t(S,[2,22]),{5:[1,157]},{5:[2,87]},{5:[2,88]},{5:[1,158]},{5:[2,89]},{5:[2,90]},t(S,[2,43]),{5:Se,28:159,31:ce,34:ae,36:Oe,38:ge,40:ze},{5:Se,28:160,31:ce,34:ae,36:Oe,38:ge,40:ze},{5:Se,28:161,31:ce,34:ae,36:Oe,38:ge,40:ze},{5:Se,28:162,31:ce,34:ae,36:Oe,38:ge,40:ze},{5:He,40:$e,56:163,57:Re,59:Ie},{5:He,40:$e,56:164,57:Re,59:Ie},t(S,[2,23]),t(S,[2,24]),t(S,[2,25]),t(S,[2,26]),t(S,[2,44]),t(S,[2,45])],defaultActions:{8:[2,2],12:[2,1],41:[2,3],42:[2,8],43:[2,9],44:[2,10],45:[2,11],46:[2,12],47:[2,13],48:[2,14],49:[2,15],50:[2,16],134:[2,83],135:[2,84],137:[2,35],138:[2,36],139:[2,37],141:[2,38],142:[2,39],143:[2,40],144:[2,41],147:[2,87],148:[2,88],150:[2,89],151:[2,90]},parseError:o(function(oe,V){if(V.recoverable)this.trace(oe);else{var xe=new Error(oe);throw xe.hash=V,xe}},"parseError"),parse:o(function(oe){var V=this,xe=[0],q=[],pe=[null],ve=[],Pe=this.table,_e="",we=0,Ve=0,De=0,qe=2,at=1,Rt=ve.slice.call(arguments,1),st=Object.create(this.lexer),Ue={yy:{}};for(var ct in this.yy)Object.prototype.hasOwnProperty.call(this.yy,ct)&&(Ue.yy[ct]=this.yy[ct]);st.setInput(oe,Ue.yy),Ue.yy.lexer=st,Ue.yy.parser=this,typeof st.yylloc>"u"&&(st.yylloc={});var We=st.yylloc;ve.push(We);var ot=st.options&&st.options.ranges;typeof Ue.yy.parseError=="function"?this.parseError=Ue.yy.parseError:this.parseError=Object.getPrototypeOf(this).parseError;function Yt(Dr){xe.length=xe.length-2*Dr,pe.length=pe.length-Dr,ve.length=ve.length-Dr}o(Yt,"popStack");function bt(){var Dr;return Dr=q.pop()||st.lex()||at,typeof Dr!="number"&&(Dr instanceof Array&&(q=Dr,Dr=q.pop()),Dr=V.symbols_[Dr]||Dr),Dr}o(bt,"lex");for(var Mt,xt,ut,Et,ft,yt,nt={},dn,Tt,On,tn;;){if(ut=xe[xe.length-1],this.defaultActions[ut]?Et=this.defaultActions[ut]:((Mt===null||typeof Mt>"u")&&(Mt=bt()),Et=Pe[ut]&&Pe[ut][Mt]),typeof Et>"u"||!Et.length||!Et[0]){var _r="";tn=[];for(dn in Pe[ut])this.terminals_[dn]&&dn>qe&&tn.push("'"+this.terminals_[dn]+"'");st.showPosition?_r="Parse error on line "+(we+1)+`: +`+st.showPosition()+` +Expecting `+tn.join(", ")+", got '"+(this.terminals_[Mt]||Mt)+"'":_r="Parse error on line "+(we+1)+": Unexpected "+(Mt==at?"end of input":"'"+(this.terminals_[Mt]||Mt)+"'"),this.parseError(_r,{text:st.match,token:this.terminals_[Mt]||Mt,line:st.yylineno,loc:We,expected:tn})}if(Et[0]instanceof Array&&Et.length>1)throw new Error("Parse Error: multiple actions possible at state: "+ut+", token: "+Mt);switch(Et[0]){case 1:xe.push(Mt),pe.push(st.yytext),ve.push(st.yylloc),xe.push(Et[1]),Mt=null,xt?(Mt=xt,xt=null):(Ve=st.yyleng,_e=st.yytext,we=st.yylineno,We=st.yylloc,De>0&&De--);break;case 2:if(Tt=this.productions_[Et[1]][1],nt.$=pe[pe.length-Tt],nt._$={first_line:ve[ve.length-(Tt||1)].first_line,last_line:ve[ve.length-1].last_line,first_column:ve[ve.length-(Tt||1)].first_column,last_column:ve[ve.length-1].last_column},ot&&(nt._$.range=[ve[ve.length-(Tt||1)].range[0],ve[ve.length-1].range[1]]),yt=this.performAction.apply(nt,[_e,Ve,we,Ue.yy,Et[1],pe,ve].concat(Rt)),typeof yt<"u")return yt;Tt&&(xe=xe.slice(0,-1*Tt*2),pe=pe.slice(0,-1*Tt),ve=ve.slice(0,-1*Tt)),xe.push(this.productions_[Et[1]][0]),pe.push(nt.$),ve.push(nt._$),On=Pe[xe[xe.length-2]][xe[xe.length-1]],xe.push(On);break;case 3:return!0}}return!0},"parse")},W=function(){var re={EOF:1,parseError:o(function(V,xe){if(this.yy.parser)this.yy.parser.parseError(V,xe);else throw new Error(V)},"parseError"),setInput:o(function(oe,V){return this.yy=V||this.yy||{},this._input=oe,this._more=this._backtrack=this.done=!1,this.yylineno=this.yyleng=0,this.yytext=this.matched=this.match="",this.conditionStack=["INITIAL"],this.yylloc={first_line:1,first_column:0,last_line:1,last_column:0},this.options.ranges&&(this.yylloc.range=[0,0]),this.offset=0,this},"setInput"),input:o(function(){var oe=this._input[0];this.yytext+=oe,this.yyleng++,this.offset++,this.match+=oe,this.matched+=oe;var V=oe.match(/(?:\r\n?|\n).*/g);return V?(this.yylineno++,this.yylloc.last_line++):this.yylloc.last_column++,this.options.ranges&&this.yylloc.range[1]++,this._input=this._input.slice(1),oe},"input"),unput:o(function(oe){var V=oe.length,xe=oe.split(/(?:\r\n?|\n)/g);this._input=oe+this._input,this.yytext=this.yytext.substr(0,this.yytext.length-V),this.offset-=V;var q=this.match.split(/(?:\r\n?|\n)/g);this.match=this.match.substr(0,this.match.length-1),this.matched=this.matched.substr(0,this.matched.length-1),xe.length-1&&(this.yylineno-=xe.length-1);var pe=this.yylloc.range;return this.yylloc={first_line:this.yylloc.first_line,last_line:this.yylineno+1,first_column:this.yylloc.first_column,last_column:xe?(xe.length===q.length?this.yylloc.first_column:0)+q[q.length-xe.length].length-xe[0].length:this.yylloc.first_column-V},this.options.ranges&&(this.yylloc.range=[pe[0],pe[0]+this.yyleng-V]),this.yyleng=this.yytext.length,this},"unput"),more:o(function(){return this._more=!0,this},"more"),reject:o(function(){if(this.options.backtrack_lexer)this._backtrack=!0;else return this.parseError("Lexical error on line "+(this.yylineno+1)+`. You can only invoke reject() in the lexer when the lexer is of the backtracking persuasion (options.backtrack_lexer = true). +`+this.showPosition(),{text:"",token:null,line:this.yylineno});return this},"reject"),less:o(function(oe){this.unput(this.match.slice(oe))},"less"),pastInput:o(function(){var oe=this.matched.substr(0,this.matched.length-this.match.length);return(oe.length>20?"...":"")+oe.substr(-20).replace(/\n/g,"")},"pastInput"),upcomingInput:o(function(){var oe=this.match;return oe.length<20&&(oe+=this._input.substr(0,20-oe.length)),(oe.substr(0,20)+(oe.length>20?"...":"")).replace(/\n/g,"")},"upcomingInput"),showPosition:o(function(){var oe=this.pastInput(),V=new Array(oe.length+1).join("-");return oe+this.upcomingInput()+` +`+V+"^"},"showPosition"),test_match:o(function(oe,V){var xe,q,pe;if(this.options.backtrack_lexer&&(pe={yylineno:this.yylineno,yylloc:{first_line:this.yylloc.first_line,last_line:this.last_line,first_column:this.yylloc.first_column,last_column:this.yylloc.last_column},yytext:this.yytext,match:this.match,matches:this.matches,matched:this.matched,yyleng:this.yyleng,offset:this.offset,_more:this._more,_input:this._input,yy:this.yy,conditionStack:this.conditionStack.slice(0),done:this.done},this.options.ranges&&(pe.yylloc.range=this.yylloc.range.slice(0))),q=oe[0].match(/(?:\r\n?|\n).*/g),q&&(this.yylineno+=q.length),this.yylloc={first_line:this.yylloc.last_line,last_line:this.yylineno+1,first_column:this.yylloc.last_column,last_column:q?q[q.length-1].length-q[q.length-1].match(/\r?\n?/)[0].length:this.yylloc.last_column+oe[0].length},this.yytext+=oe[0],this.match+=oe[0],this.matches=oe,this.yyleng=this.yytext.length,this.options.ranges&&(this.yylloc.range=[this.offset,this.offset+=this.yyleng]),this._more=!1,this._backtrack=!1,this._input=this._input.slice(oe[0].length),this.matched+=oe[0],xe=this.performAction.call(this,this.yy,this,V,this.conditionStack[this.conditionStack.length-1]),this.done&&this._input&&(this.done=!1),xe)return xe;if(this._backtrack){for(var ve in pe)this[ve]=pe[ve];return!1}return!1},"test_match"),next:o(function(){if(this.done)return this.EOF;this._input||(this.done=!0);var oe,V,xe,q;this._more||(this.yytext="",this.match="");for(var pe=this._currentRules(),ve=0;veV[0].length)){if(V=xe,q=ve,this.options.backtrack_lexer){if(oe=this.test_match(xe,pe[ve]),oe!==!1)return oe;if(this._backtrack){V=!1;continue}else return!1}else if(!this.options.flex)break}return V?(oe=this.test_match(V,pe[q]),oe!==!1?oe:!1):this._input===""?this.EOF:this.parseError("Lexical error on line "+(this.yylineno+1)+`. Unrecognized text. +`+this.showPosition(),{text:"",token:null,line:this.yylineno})},"next"),lex:o(function(){var V=this.next();return V||this.lex()},"lex"),begin:o(function(V){this.conditionStack.push(V)},"begin"),popState:o(function(){var V=this.conditionStack.length-1;return V>0?this.conditionStack.pop():this.conditionStack[0]},"popState"),_currentRules:o(function(){return this.conditionStack.length&&this.conditionStack[this.conditionStack.length-1]?this.conditions[this.conditionStack[this.conditionStack.length-1]].rules:this.conditions.INITIAL.rules},"_currentRules"),topState:o(function(V){return V=this.conditionStack.length-1-Math.abs(V||0),V>=0?this.conditionStack[V]:"INITIAL"},"topState"),pushState:o(function(V){this.begin(V)},"pushState"),stateStackSize:o(function(){return this.conditionStack.length},"stateStackSize"),options:{"case-insensitive":!0},performAction:o(function(V,xe,q,pe){var ve=pe;switch(q){case 0:return"title";case 1:return this.begin("acc_title"),9;break;case 2:return this.popState(),"acc_title_value";break;case 3:return this.begin("acc_descr"),11;break;case 4:return this.popState(),"acc_descr_value";break;case 5:this.begin("acc_descr_multiline");break;case 6:this.popState();break;case 7:return"acc_descr_multiline_value";case 8:return 21;case 9:return 22;case 10:return 23;case 11:return 24;case 12:return 5;case 13:break;case 14:break;case 15:break;case 16:return 8;case 17:return 6;case 18:return 27;case 19:return 40;case 20:return 29;case 21:return 32;case 22:return 31;case 23:return 34;case 24:return 36;case 25:return 38;case 26:return 41;case 27:return 42;case 28:return 43;case 29:return 44;case 30:return 45;case 31:return 46;case 32:return 47;case 33:return 48;case 34:return 49;case 35:return 50;case 36:return 51;case 37:return 52;case 38:return 53;case 39:return 54;case 40:return 65;case 41:return 66;case 42:return 67;case 43:return 68;case 44:return 69;case 45:return 70;case 46:return 71;case 47:return 57;case 48:return 59;case 49:return this.begin("style"),77;break;case 50:return 75;case 51:return 81;case 52:return 88;case 53:return"PERCENT";case 54:return 86;case 55:return 84;case 56:break;case 57:this.begin("string");break;case 58:this.popState();break;case 59:return this.begin("style"),72;break;case 60:return this.begin("style"),74;break;case 61:return 61;case 62:return 64;case 63:return 63;case 64:this.begin("string");break;case 65:this.popState();break;case 66:return"qString";case 67:return xe.yytext=xe.yytext.trim(),89;break;case 68:return 75;case 69:return 80;case 70:return 76}},"anonymous"),rules:[/^(?:title\s[^#\n;]+)/i,/^(?:accTitle\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*\{\s*)/i,/^(?:[\}])/i,/^(?:[^\}]*)/i,/^(?:.*direction\s+TB[^\n]*)/i,/^(?:.*direction\s+BT[^\n]*)/i,/^(?:.*direction\s+RL[^\n]*)/i,/^(?:.*direction\s+LR[^\n]*)/i,/^(?:(\r?\n)+)/i,/^(?:\s+)/i,/^(?:#[^\n]*)/i,/^(?:%[^\n]*)/i,/^(?:$)/i,/^(?:requirementDiagram\b)/i,/^(?:\{)/i,/^(?:\})/i,/^(?::{3})/i,/^(?::)/i,/^(?:id\b)/i,/^(?:text\b)/i,/^(?:risk\b)/i,/^(?:verifyMethod\b)/i,/^(?:requirement\b)/i,/^(?:functionalRequirement\b)/i,/^(?:interfaceRequirement\b)/i,/^(?:performanceRequirement\b)/i,/^(?:physicalRequirement\b)/i,/^(?:designConstraint\b)/i,/^(?:low\b)/i,/^(?:medium\b)/i,/^(?:high\b)/i,/^(?:analysis\b)/i,/^(?:demonstration\b)/i,/^(?:inspection\b)/i,/^(?:test\b)/i,/^(?:element\b)/i,/^(?:contains\b)/i,/^(?:copies\b)/i,/^(?:derives\b)/i,/^(?:satisfies\b)/i,/^(?:verifies\b)/i,/^(?:refines\b)/i,/^(?:traces\b)/i,/^(?:type\b)/i,/^(?:docref\b)/i,/^(?:style\b)/i,/^(?:\w+)/i,/^(?::)/i,/^(?:;)/i,/^(?:%)/i,/^(?:-)/i,/^(?:#)/i,/^(?: )/i,/^(?:["])/i,/^(?:\n)/i,/^(?:classDef\b)/i,/^(?:class\b)/i,/^(?:<-)/i,/^(?:->)/i,/^(?:-)/i,/^(?:["])/i,/^(?:["])/i,/^(?:[^"]*)/i,/^(?:[\w][^:,\r\n\{\<\>\-\=]*)/i,/^(?:\w+)/i,/^(?:[0-9]+)/i,/^(?:,)/i],conditions:{acc_descr_multiline:{rules:[6,7,68,69,70],inclusive:!1},acc_descr:{rules:[4,68,69,70],inclusive:!1},acc_title:{rules:[2,68,69,70],inclusive:!1},style:{rules:[50,51,52,53,54,55,56,57,58,68,69,70],inclusive:!1},unqString:{rules:[68,69,70],inclusive:!1},token:{rules:[68,69,70],inclusive:!1},string:{rules:[65,66,68,69,70],inclusive:!1},INITIAL:{rules:[0,1,3,5,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,59,60,61,62,63,64,67,68,69,70],inclusive:!0}}};return re}();be.lexer=W;function de(){this.yy={}}return o(de,"Parser"),de.prototype=be,be.Parser=de,new de}();EO.parser=EO;Zhe=EO});var A6,efe=N(()=>{"use strict";zt();vt();mi();A6=class{constructor(){this.relations=[];this.latestRequirement=this.getInitialRequirement();this.requirements=new Map;this.latestElement=this.getInitialElement();this.elements=new Map;this.classes=new Map;this.direction="TB";this.RequirementType={REQUIREMENT:"Requirement",FUNCTIONAL_REQUIREMENT:"Functional Requirement",INTERFACE_REQUIREMENT:"Interface Requirement",PERFORMANCE_REQUIREMENT:"Performance Requirement",PHYSICAL_REQUIREMENT:"Physical Requirement",DESIGN_CONSTRAINT:"Design Constraint"};this.RiskLevel={LOW_RISK:"Low",MED_RISK:"Medium",HIGH_RISK:"High"};this.VerifyType={VERIFY_ANALYSIS:"Analysis",VERIFY_DEMONSTRATION:"Demonstration",VERIFY_INSPECTION:"Inspection",VERIFY_TEST:"Test"};this.Relationships={CONTAINS:"contains",COPIES:"copies",DERIVES:"derives",SATISFIES:"satisfies",VERIFIES:"verifies",REFINES:"refines",TRACES:"traces"};this.setAccTitle=Lr;this.getAccTitle=Rr;this.setAccDescription=Nr;this.getAccDescription=Mr;this.setDiagramTitle=$r;this.getDiagramTitle=Ir;this.getConfig=o(()=>me().requirement,"getConfig");this.clear(),this.setDirection=this.setDirection.bind(this),this.addRequirement=this.addRequirement.bind(this),this.setNewReqId=this.setNewReqId.bind(this),this.setNewReqRisk=this.setNewReqRisk.bind(this),this.setNewReqText=this.setNewReqText.bind(this),this.setNewReqVerifyMethod=this.setNewReqVerifyMethod.bind(this),this.addElement=this.addElement.bind(this),this.setNewElementType=this.setNewElementType.bind(this),this.setNewElementDocRef=this.setNewElementDocRef.bind(this),this.addRelationship=this.addRelationship.bind(this),this.setCssStyle=this.setCssStyle.bind(this),this.setClass=this.setClass.bind(this),this.defineClass=this.defineClass.bind(this),this.setAccTitle=this.setAccTitle.bind(this),this.setAccDescription=this.setAccDescription.bind(this)}static{o(this,"RequirementDB")}getDirection(){return this.direction}setDirection(e){this.direction=e}resetLatestRequirement(){this.latestRequirement=this.getInitialRequirement()}resetLatestElement(){this.latestElement=this.getInitialElement()}getInitialRequirement(){return{requirementId:"",text:"",risk:"",verifyMethod:"",name:"",type:"",cssStyles:[],classes:["default"]}}getInitialElement(){return{name:"",type:"",docRef:"",cssStyles:[],classes:["default"]}}addRequirement(e,r){return this.requirements.has(e)||this.requirements.set(e,{name:e,type:r,requirementId:this.latestRequirement.requirementId,text:this.latestRequirement.text,risk:this.latestRequirement.risk,verifyMethod:this.latestRequirement.verifyMethod,cssStyles:[],classes:["default"]}),this.resetLatestRequirement(),this.requirements.get(e)}getRequirements(){return this.requirements}setNewReqId(e){this.latestRequirement!==void 0&&(this.latestRequirement.requirementId=e)}setNewReqText(e){this.latestRequirement!==void 0&&(this.latestRequirement.text=e)}setNewReqRisk(e){this.latestRequirement!==void 0&&(this.latestRequirement.risk=e)}setNewReqVerifyMethod(e){this.latestRequirement!==void 0&&(this.latestRequirement.verifyMethod=e)}addElement(e){return this.elements.has(e)||(this.elements.set(e,{name:e,type:this.latestElement.type,docRef:this.latestElement.docRef,cssStyles:[],classes:["default"]}),Y.info("Added new element: ",e)),this.resetLatestElement(),this.elements.get(e)}getElements(){return this.elements}setNewElementType(e){this.latestElement!==void 0&&(this.latestElement.type=e)}setNewElementDocRef(e){this.latestElement!==void 0&&(this.latestElement.docRef=e)}addRelationship(e,r,n){this.relations.push({type:e,src:r,dst:n})}getRelationships(){return this.relations}clear(){this.relations=[],this.resetLatestRequirement(),this.requirements=new Map,this.resetLatestElement(),this.elements=new Map,this.classes=new Map,Ar()}setCssStyle(e,r){for(let n of e){let i=this.requirements.get(n)??this.elements.get(n);if(!r||!i)return;for(let a of r)a.includes(",")?i.cssStyles.push(...a.split(",")):i.cssStyles.push(a)}}setClass(e,r){for(let n of e){let i=this.requirements.get(n)??this.elements.get(n);if(i)for(let a of r){i.classes.push(a);let s=this.classes.get(a)?.styles;s&&i.cssStyles.push(...s)}}}defineClass(e,r){for(let n of e){let i=this.classes.get(n);i===void 0&&(i={id:n,styles:[],textStyles:[]},this.classes.set(n,i)),r&&r.forEach(function(a){if(/color/.exec(a)){let s=a.replace("fill","bgFill");i.textStyles.push(s)}i.styles.push(a)}),this.requirements.forEach(a=>{a.classes.includes(n)&&a.cssStyles.push(...r.flatMap(s=>s.split(",")))}),this.elements.forEach(a=>{a.classes.includes(n)&&a.cssStyles.push(...r.flatMap(s=>s.split(",")))})}}getClasses(){return this.classes}getData(){let e=me(),r=[],n=[];for(let i of this.requirements.values()){let a=i;a.id=i.name,a.cssStyles=i.cssStyles,a.cssClasses=i.classes.join(" "),a.shape="requirementBox",a.look=e.look,r.push(a)}for(let i of this.elements.values()){let a=i;a.shape="requirementBox",a.look=e.look,a.id=i.name,a.cssStyles=i.cssStyles,a.cssClasses=i.classes.join(" "),r.push(a)}for(let i of this.relations){let a=0,s=i.type===this.Relationships.CONTAINS,l={id:`${i.src}-${i.dst}-${a}`,start:this.requirements.get(i.src)?.name??this.elements.get(i.src)?.name,end:this.requirements.get(i.dst)?.name??this.elements.get(i.dst)?.name,label:`<<${i.type}>>`,classes:"relationshipLine",style:["fill:none",s?"":"stroke-dasharray: 10,7"],labelpos:"c",thickness:"normal",type:"normal",pattern:s?"normal":"dashed",arrowTypeStart:s?"requirement_contains":"",arrowTypeEnd:s?"":"requirement_arrow",look:e.look};n.push(l),a++}return{nodes:r,edges:n,other:{},config:e,direction:this.getDirection()}}}});var ZGe,tfe,rfe=N(()=>{"use strict";ZGe=o(t=>` + + marker { + fill: ${t.relationColor}; + stroke: ${t.relationColor}; + } + + marker.cross { + stroke: ${t.lineColor}; + } + + svg { + font-family: ${t.fontFamily}; + font-size: ${t.fontSize}; + } + + .reqBox { + fill: ${t.requirementBackground}; + fill-opacity: 1.0; + stroke: ${t.requirementBorderColor}; + stroke-width: ${t.requirementBorderSize}; + } + + .reqTitle, .reqLabel{ + fill: ${t.requirementTextColor}; + } + .reqLabelBox { + fill: ${t.relationLabelBackground}; + fill-opacity: 1.0; + } + + .req-title-line { + stroke: ${t.requirementBorderColor}; + stroke-width: ${t.requirementBorderSize}; + } + .relationshipLine { + stroke: ${t.relationColor}; + stroke-width: 1; + } + .relationshipLabel { + fill: ${t.relationLabelColor}; + } + .divider { + stroke: ${t.nodeBorder}; + stroke-width: 1; + } + .label { + font-family: ${t.fontFamily}; + color: ${t.nodeTextColor||t.textColor}; + } + .label text,span { + fill: ${t.nodeTextColor||t.textColor}; + color: ${t.nodeTextColor||t.textColor}; + } + .labelBkg { + background-color: ${t.edgeLabelBackground}; + } + +`,"getStyles"),tfe=ZGe});var SO={};hr(SO,{draw:()=>JGe});var JGe,nfe=N(()=>{"use strict";zt();vt();gm();Yd();$m();ir();JGe=o(async function(t,e,r,n){Y.info("REF0:"),Y.info("Drawing requirement diagram (unified)",e);let{securityLevel:i,state:a,layout:s}=me(),l=n.db.getData(),u=yc(e,i);l.type=n.type,l.layoutAlgorithm=nf(s),l.nodeSpacing=a?.nodeSpacing??50,l.rankSpacing=a?.rankSpacing??50,l.markers=["requirement_contains","requirement_arrow"],l.diagramId=e,await Cc(l,u);let h=8;Gt.insertTitle(u,"requirementDiagramTitleText",a?.titleTopMargin??25,n.db.getDiagramTitle()),Ac(u,h,"requirementDiagram",a?.useMaxWidth??!0)},"draw")});var ife={};hr(ife,{diagram:()=>eVe});var eVe,afe=N(()=>{"use strict";Jhe();efe();rfe();nfe();eVe={parser:Zhe,get db(){return new A6},renderer:SO,styles:tfe}});var CO,lfe,cfe=N(()=>{"use strict";CO=function(){var t=o(function(K,X,te,J){for(te=te||{},J=K.length;J--;te[K[J]]=X);return te},"o"),e=[1,2],r=[1,3],n=[1,4],i=[2,4],a=[1,9],s=[1,11],l=[1,13],u=[1,14],h=[1,16],f=[1,17],d=[1,18],p=[1,24],m=[1,25],g=[1,26],y=[1,27],v=[1,28],x=[1,29],b=[1,30],w=[1,31],C=[1,32],T=[1,33],E=[1,34],A=[1,35],S=[1,36],_=[1,37],I=[1,38],D=[1,39],k=[1,41],L=[1,42],R=[1,43],O=[1,44],M=[1,45],B=[1,46],F=[1,4,5,13,14,16,18,21,23,29,30,31,33,35,36,37,38,39,41,43,44,46,47,48,49,50,52,53,54,59,60,61,62,70],P=[4,5,16,50,52,53],z=[4,5,13,14,16,18,21,23,29,30,31,33,35,36,37,38,39,41,43,44,46,50,52,53,54,59,60,61,62,70],$=[4,5,13,14,16,18,21,23,29,30,31,33,35,36,37,38,39,41,43,44,46,49,50,52,53,54,59,60,61,62,70],H=[4,5,13,14,16,18,21,23,29,30,31,33,35,36,37,38,39,41,43,44,46,48,50,52,53,54,59,60,61,62,70],Q=[4,5,13,14,16,18,21,23,29,30,31,33,35,36,37,38,39,41,43,44,46,47,50,52,53,54,59,60,61,62,70],j=[68,69,70],ie=[1,122],ne={trace:o(function(){},"trace"),yy:{},symbols_:{error:2,start:3,SPACE:4,NEWLINE:5,SD:6,document:7,line:8,statement:9,box_section:10,box_line:11,participant_statement:12,create:13,box:14,restOfLine:15,end:16,signal:17,autonumber:18,NUM:19,off:20,activate:21,actor:22,deactivate:23,note_statement:24,links_statement:25,link_statement:26,properties_statement:27,details_statement:28,title:29,legacy_title:30,acc_title:31,acc_title_value:32,acc_descr:33,acc_descr_value:34,acc_descr_multiline_value:35,loop:36,rect:37,opt:38,alt:39,else_sections:40,par:41,par_sections:42,par_over:43,critical:44,option_sections:45,break:46,option:47,and:48,else:49,participant:50,AS:51,participant_actor:52,destroy:53,note:54,placement:55,text2:56,over:57,actor_pair:58,links:59,link:60,properties:61,details:62,spaceList:63,",":64,left_of:65,right_of:66,signaltype:67,"+":68,"-":69,ACTOR:70,SOLID_OPEN_ARROW:71,DOTTED_OPEN_ARROW:72,SOLID_ARROW:73,BIDIRECTIONAL_SOLID_ARROW:74,DOTTED_ARROW:75,BIDIRECTIONAL_DOTTED_ARROW:76,SOLID_CROSS:77,DOTTED_CROSS:78,SOLID_POINT:79,DOTTED_POINT:80,TXT:81,$accept:0,$end:1},terminals_:{2:"error",4:"SPACE",5:"NEWLINE",6:"SD",13:"create",14:"box",15:"restOfLine",16:"end",18:"autonumber",19:"NUM",20:"off",21:"activate",23:"deactivate",29:"title",30:"legacy_title",31:"acc_title",32:"acc_title_value",33:"acc_descr",34:"acc_descr_value",35:"acc_descr_multiline_value",36:"loop",37:"rect",38:"opt",39:"alt",41:"par",43:"par_over",44:"critical",46:"break",47:"option",48:"and",49:"else",50:"participant",51:"AS",52:"participant_actor",53:"destroy",54:"note",57:"over",59:"links",60:"link",61:"properties",62:"details",64:",",65:"left_of",66:"right_of",68:"+",69:"-",70:"ACTOR",71:"SOLID_OPEN_ARROW",72:"DOTTED_OPEN_ARROW",73:"SOLID_ARROW",74:"BIDIRECTIONAL_SOLID_ARROW",75:"DOTTED_ARROW",76:"BIDIRECTIONAL_DOTTED_ARROW",77:"SOLID_CROSS",78:"DOTTED_CROSS",79:"SOLID_POINT",80:"DOTTED_POINT",81:"TXT"},productions_:[0,[3,2],[3,2],[3,2],[7,0],[7,2],[8,2],[8,1],[8,1],[10,0],[10,2],[11,2],[11,1],[11,1],[9,1],[9,2],[9,4],[9,2],[9,4],[9,3],[9,3],[9,2],[9,3],[9,3],[9,2],[9,2],[9,2],[9,2],[9,2],[9,1],[9,1],[9,2],[9,2],[9,1],[9,4],[9,4],[9,4],[9,4],[9,4],[9,4],[9,4],[9,4],[45,1],[45,4],[42,1],[42,4],[40,1],[40,4],[12,5],[12,3],[12,5],[12,3],[12,3],[24,4],[24,4],[25,3],[26,3],[27,3],[28,3],[63,2],[63,1],[58,3],[58,1],[55,1],[55,1],[17,5],[17,5],[17,4],[22,1],[67,1],[67,1],[67,1],[67,1],[67,1],[67,1],[67,1],[67,1],[67,1],[67,1],[56,1]],performAction:o(function(X,te,J,se,ue,Z,Se){var ce=Z.length-1;switch(ue){case 3:return se.apply(Z[ce]),Z[ce];break;case 4:case 9:this.$=[];break;case 5:case 10:Z[ce-1].push(Z[ce]),this.$=Z[ce-1];break;case 6:case 7:case 11:case 12:this.$=Z[ce];break;case 8:case 13:this.$=[];break;case 15:Z[ce].type="createParticipant",this.$=Z[ce];break;case 16:Z[ce-1].unshift({type:"boxStart",boxData:se.parseBoxData(Z[ce-2])}),Z[ce-1].push({type:"boxEnd",boxText:Z[ce-2]}),this.$=Z[ce-1];break;case 18:this.$={type:"sequenceIndex",sequenceIndex:Number(Z[ce-2]),sequenceIndexStep:Number(Z[ce-1]),sequenceVisible:!0,signalType:se.LINETYPE.AUTONUMBER};break;case 19:this.$={type:"sequenceIndex",sequenceIndex:Number(Z[ce-1]),sequenceIndexStep:1,sequenceVisible:!0,signalType:se.LINETYPE.AUTONUMBER};break;case 20:this.$={type:"sequenceIndex",sequenceVisible:!1,signalType:se.LINETYPE.AUTONUMBER};break;case 21:this.$={type:"sequenceIndex",sequenceVisible:!0,signalType:se.LINETYPE.AUTONUMBER};break;case 22:this.$={type:"activeStart",signalType:se.LINETYPE.ACTIVE_START,actor:Z[ce-1].actor};break;case 23:this.$={type:"activeEnd",signalType:se.LINETYPE.ACTIVE_END,actor:Z[ce-1].actor};break;case 29:se.setDiagramTitle(Z[ce].substring(6)),this.$=Z[ce].substring(6);break;case 30:se.setDiagramTitle(Z[ce].substring(7)),this.$=Z[ce].substring(7);break;case 31:this.$=Z[ce].trim(),se.setAccTitle(this.$);break;case 32:case 33:this.$=Z[ce].trim(),se.setAccDescription(this.$);break;case 34:Z[ce-1].unshift({type:"loopStart",loopText:se.parseMessage(Z[ce-2]),signalType:se.LINETYPE.LOOP_START}),Z[ce-1].push({type:"loopEnd",loopText:Z[ce-2],signalType:se.LINETYPE.LOOP_END}),this.$=Z[ce-1];break;case 35:Z[ce-1].unshift({type:"rectStart",color:se.parseMessage(Z[ce-2]),signalType:se.LINETYPE.RECT_START}),Z[ce-1].push({type:"rectEnd",color:se.parseMessage(Z[ce-2]),signalType:se.LINETYPE.RECT_END}),this.$=Z[ce-1];break;case 36:Z[ce-1].unshift({type:"optStart",optText:se.parseMessage(Z[ce-2]),signalType:se.LINETYPE.OPT_START}),Z[ce-1].push({type:"optEnd",optText:se.parseMessage(Z[ce-2]),signalType:se.LINETYPE.OPT_END}),this.$=Z[ce-1];break;case 37:Z[ce-1].unshift({type:"altStart",altText:se.parseMessage(Z[ce-2]),signalType:se.LINETYPE.ALT_START}),Z[ce-1].push({type:"altEnd",signalType:se.LINETYPE.ALT_END}),this.$=Z[ce-1];break;case 38:Z[ce-1].unshift({type:"parStart",parText:se.parseMessage(Z[ce-2]),signalType:se.LINETYPE.PAR_START}),Z[ce-1].push({type:"parEnd",signalType:se.LINETYPE.PAR_END}),this.$=Z[ce-1];break;case 39:Z[ce-1].unshift({type:"parStart",parText:se.parseMessage(Z[ce-2]),signalType:se.LINETYPE.PAR_OVER_START}),Z[ce-1].push({type:"parEnd",signalType:se.LINETYPE.PAR_END}),this.$=Z[ce-1];break;case 40:Z[ce-1].unshift({type:"criticalStart",criticalText:se.parseMessage(Z[ce-2]),signalType:se.LINETYPE.CRITICAL_START}),Z[ce-1].push({type:"criticalEnd",signalType:se.LINETYPE.CRITICAL_END}),this.$=Z[ce-1];break;case 41:Z[ce-1].unshift({type:"breakStart",breakText:se.parseMessage(Z[ce-2]),signalType:se.LINETYPE.BREAK_START}),Z[ce-1].push({type:"breakEnd",optText:se.parseMessage(Z[ce-2]),signalType:se.LINETYPE.BREAK_END}),this.$=Z[ce-1];break;case 43:this.$=Z[ce-3].concat([{type:"option",optionText:se.parseMessage(Z[ce-1]),signalType:se.LINETYPE.CRITICAL_OPTION},Z[ce]]);break;case 45:this.$=Z[ce-3].concat([{type:"and",parText:se.parseMessage(Z[ce-1]),signalType:se.LINETYPE.PAR_AND},Z[ce]]);break;case 47:this.$=Z[ce-3].concat([{type:"else",altText:se.parseMessage(Z[ce-1]),signalType:se.LINETYPE.ALT_ELSE},Z[ce]]);break;case 48:Z[ce-3].draw="participant",Z[ce-3].type="addParticipant",Z[ce-3].description=se.parseMessage(Z[ce-1]),this.$=Z[ce-3];break;case 49:Z[ce-1].draw="participant",Z[ce-1].type="addParticipant",this.$=Z[ce-1];break;case 50:Z[ce-3].draw="actor",Z[ce-3].type="addParticipant",Z[ce-3].description=se.parseMessage(Z[ce-1]),this.$=Z[ce-3];break;case 51:Z[ce-1].draw="actor",Z[ce-1].type="addParticipant",this.$=Z[ce-1];break;case 52:Z[ce-1].type="destroyParticipant",this.$=Z[ce-1];break;case 53:this.$=[Z[ce-1],{type:"addNote",placement:Z[ce-2],actor:Z[ce-1].actor,text:Z[ce]}];break;case 54:Z[ce-2]=[].concat(Z[ce-1],Z[ce-1]).slice(0,2),Z[ce-2][0]=Z[ce-2][0].actor,Z[ce-2][1]=Z[ce-2][1].actor,this.$=[Z[ce-1],{type:"addNote",placement:se.PLACEMENT.OVER,actor:Z[ce-2].slice(0,2),text:Z[ce]}];break;case 55:this.$=[Z[ce-1],{type:"addLinks",actor:Z[ce-1].actor,text:Z[ce]}];break;case 56:this.$=[Z[ce-1],{type:"addALink",actor:Z[ce-1].actor,text:Z[ce]}];break;case 57:this.$=[Z[ce-1],{type:"addProperties",actor:Z[ce-1].actor,text:Z[ce]}];break;case 58:this.$=[Z[ce-1],{type:"addDetails",actor:Z[ce-1].actor,text:Z[ce]}];break;case 61:this.$=[Z[ce-2],Z[ce]];break;case 62:this.$=Z[ce];break;case 63:this.$=se.PLACEMENT.LEFTOF;break;case 64:this.$=se.PLACEMENT.RIGHTOF;break;case 65:this.$=[Z[ce-4],Z[ce-1],{type:"addMessage",from:Z[ce-4].actor,to:Z[ce-1].actor,signalType:Z[ce-3],msg:Z[ce],activate:!0},{type:"activeStart",signalType:se.LINETYPE.ACTIVE_START,actor:Z[ce-1].actor}];break;case 66:this.$=[Z[ce-4],Z[ce-1],{type:"addMessage",from:Z[ce-4].actor,to:Z[ce-1].actor,signalType:Z[ce-3],msg:Z[ce]},{type:"activeEnd",signalType:se.LINETYPE.ACTIVE_END,actor:Z[ce-4].actor}];break;case 67:this.$=[Z[ce-3],Z[ce-1],{type:"addMessage",from:Z[ce-3].actor,to:Z[ce-1].actor,signalType:Z[ce-2],msg:Z[ce]}];break;case 68:this.$={type:"addParticipant",actor:Z[ce]};break;case 69:this.$=se.LINETYPE.SOLID_OPEN;break;case 70:this.$=se.LINETYPE.DOTTED_OPEN;break;case 71:this.$=se.LINETYPE.SOLID;break;case 72:this.$=se.LINETYPE.BIDIRECTIONAL_SOLID;break;case 73:this.$=se.LINETYPE.DOTTED;break;case 74:this.$=se.LINETYPE.BIDIRECTIONAL_DOTTED;break;case 75:this.$=se.LINETYPE.SOLID_CROSS;break;case 76:this.$=se.LINETYPE.DOTTED_CROSS;break;case 77:this.$=se.LINETYPE.SOLID_POINT;break;case 78:this.$=se.LINETYPE.DOTTED_POINT;break;case 79:this.$=se.parseMessage(Z[ce].trim().substring(1));break}},"anonymous"),table:[{3:1,4:e,5:r,6:n},{1:[3]},{3:5,4:e,5:r,6:n},{3:6,4:e,5:r,6:n},t([1,4,5,13,14,18,21,23,29,30,31,33,35,36,37,38,39,41,43,44,46,50,52,53,54,59,60,61,62,70],i,{7:7}),{1:[2,1]},{1:[2,2]},{1:[2,3],4:a,5:s,8:8,9:10,12:12,13:l,14:u,17:15,18:h,21:f,22:40,23:d,24:19,25:20,26:21,27:22,28:23,29:p,30:m,31:g,33:y,35:v,36:x,37:b,38:w,39:C,41:T,43:E,44:A,46:S,50:_,52:I,53:D,54:k,59:L,60:R,61:O,62:M,70:B},t(F,[2,5]),{9:47,12:12,13:l,14:u,17:15,18:h,21:f,22:40,23:d,24:19,25:20,26:21,27:22,28:23,29:p,30:m,31:g,33:y,35:v,36:x,37:b,38:w,39:C,41:T,43:E,44:A,46:S,50:_,52:I,53:D,54:k,59:L,60:R,61:O,62:M,70:B},t(F,[2,7]),t(F,[2,8]),t(F,[2,14]),{12:48,50:_,52:I,53:D},{15:[1,49]},{5:[1,50]},{5:[1,53],19:[1,51],20:[1,52]},{22:54,70:B},{22:55,70:B},{5:[1,56]},{5:[1,57]},{5:[1,58]},{5:[1,59]},{5:[1,60]},t(F,[2,29]),t(F,[2,30]),{32:[1,61]},{34:[1,62]},t(F,[2,33]),{15:[1,63]},{15:[1,64]},{15:[1,65]},{15:[1,66]},{15:[1,67]},{15:[1,68]},{15:[1,69]},{15:[1,70]},{22:71,70:B},{22:72,70:B},{22:73,70:B},{67:74,71:[1,75],72:[1,76],73:[1,77],74:[1,78],75:[1,79],76:[1,80],77:[1,81],78:[1,82],79:[1,83],80:[1,84]},{55:85,57:[1,86],65:[1,87],66:[1,88]},{22:89,70:B},{22:90,70:B},{22:91,70:B},{22:92,70:B},t([5,51,64,71,72,73,74,75,76,77,78,79,80,81],[2,68]),t(F,[2,6]),t(F,[2,15]),t(P,[2,9],{10:93}),t(F,[2,17]),{5:[1,95],19:[1,94]},{5:[1,96]},t(F,[2,21]),{5:[1,97]},{5:[1,98]},t(F,[2,24]),t(F,[2,25]),t(F,[2,26]),t(F,[2,27]),t(F,[2,28]),t(F,[2,31]),t(F,[2,32]),t(z,i,{7:99}),t(z,i,{7:100}),t(z,i,{7:101}),t($,i,{40:102,7:103}),t(H,i,{42:104,7:105}),t(H,i,{7:105,42:106}),t(Q,i,{45:107,7:108}),t(z,i,{7:109}),{5:[1,111],51:[1,110]},{5:[1,113],51:[1,112]},{5:[1,114]},{22:117,68:[1,115],69:[1,116],70:B},t(j,[2,69]),t(j,[2,70]),t(j,[2,71]),t(j,[2,72]),t(j,[2,73]),t(j,[2,74]),t(j,[2,75]),t(j,[2,76]),t(j,[2,77]),t(j,[2,78]),{22:118,70:B},{22:120,58:119,70:B},{70:[2,63]},{70:[2,64]},{56:121,81:ie},{56:123,81:ie},{56:124,81:ie},{56:125,81:ie},{4:[1,128],5:[1,130],11:127,12:129,16:[1,126],50:_,52:I,53:D},{5:[1,131]},t(F,[2,19]),t(F,[2,20]),t(F,[2,22]),t(F,[2,23]),{4:a,5:s,8:8,9:10,12:12,13:l,14:u,16:[1,132],17:15,18:h,21:f,22:40,23:d,24:19,25:20,26:21,27:22,28:23,29:p,30:m,31:g,33:y,35:v,36:x,37:b,38:w,39:C,41:T,43:E,44:A,46:S,50:_,52:I,53:D,54:k,59:L,60:R,61:O,62:M,70:B},{4:a,5:s,8:8,9:10,12:12,13:l,14:u,16:[1,133],17:15,18:h,21:f,22:40,23:d,24:19,25:20,26:21,27:22,28:23,29:p,30:m,31:g,33:y,35:v,36:x,37:b,38:w,39:C,41:T,43:E,44:A,46:S,50:_,52:I,53:D,54:k,59:L,60:R,61:O,62:M,70:B},{4:a,5:s,8:8,9:10,12:12,13:l,14:u,16:[1,134],17:15,18:h,21:f,22:40,23:d,24:19,25:20,26:21,27:22,28:23,29:p,30:m,31:g,33:y,35:v,36:x,37:b,38:w,39:C,41:T,43:E,44:A,46:S,50:_,52:I,53:D,54:k,59:L,60:R,61:O,62:M,70:B},{16:[1,135]},{4:a,5:s,8:8,9:10,12:12,13:l,14:u,16:[2,46],17:15,18:h,21:f,22:40,23:d,24:19,25:20,26:21,27:22,28:23,29:p,30:m,31:g,33:y,35:v,36:x,37:b,38:w,39:C,41:T,43:E,44:A,46:S,49:[1,136],50:_,52:I,53:D,54:k,59:L,60:R,61:O,62:M,70:B},{16:[1,137]},{4:a,5:s,8:8,9:10,12:12,13:l,14:u,16:[2,44],17:15,18:h,21:f,22:40,23:d,24:19,25:20,26:21,27:22,28:23,29:p,30:m,31:g,33:y,35:v,36:x,37:b,38:w,39:C,41:T,43:E,44:A,46:S,48:[1,138],50:_,52:I,53:D,54:k,59:L,60:R,61:O,62:M,70:B},{16:[1,139]},{16:[1,140]},{4:a,5:s,8:8,9:10,12:12,13:l,14:u,16:[2,42],17:15,18:h,21:f,22:40,23:d,24:19,25:20,26:21,27:22,28:23,29:p,30:m,31:g,33:y,35:v,36:x,37:b,38:w,39:C,41:T,43:E,44:A,46:S,47:[1,141],50:_,52:I,53:D,54:k,59:L,60:R,61:O,62:M,70:B},{4:a,5:s,8:8,9:10,12:12,13:l,14:u,16:[1,142],17:15,18:h,21:f,22:40,23:d,24:19,25:20,26:21,27:22,28:23,29:p,30:m,31:g,33:y,35:v,36:x,37:b,38:w,39:C,41:T,43:E,44:A,46:S,50:_,52:I,53:D,54:k,59:L,60:R,61:O,62:M,70:B},{15:[1,143]},t(F,[2,49]),{15:[1,144]},t(F,[2,51]),t(F,[2,52]),{22:145,70:B},{22:146,70:B},{56:147,81:ie},{56:148,81:ie},{56:149,81:ie},{64:[1,150],81:[2,62]},{5:[2,55]},{5:[2,79]},{5:[2,56]},{5:[2,57]},{5:[2,58]},t(F,[2,16]),t(P,[2,10]),{12:151,50:_,52:I,53:D},t(P,[2,12]),t(P,[2,13]),t(F,[2,18]),t(F,[2,34]),t(F,[2,35]),t(F,[2,36]),t(F,[2,37]),{15:[1,152]},t(F,[2,38]),{15:[1,153]},t(F,[2,39]),t(F,[2,40]),{15:[1,154]},t(F,[2,41]),{5:[1,155]},{5:[1,156]},{56:157,81:ie},{56:158,81:ie},{5:[2,67]},{5:[2,53]},{5:[2,54]},{22:159,70:B},t(P,[2,11]),t($,i,{7:103,40:160}),t(H,i,{7:105,42:161}),t(Q,i,{7:108,45:162}),t(F,[2,48]),t(F,[2,50]),{5:[2,65]},{5:[2,66]},{81:[2,61]},{16:[2,47]},{16:[2,45]},{16:[2,43]}],defaultActions:{5:[2,1],6:[2,2],87:[2,63],88:[2,64],121:[2,55],122:[2,79],123:[2,56],124:[2,57],125:[2,58],147:[2,67],148:[2,53],149:[2,54],157:[2,65],158:[2,66],159:[2,61],160:[2,47],161:[2,45],162:[2,43]},parseError:o(function(X,te){if(te.recoverable)this.trace(X);else{var J=new Error(X);throw J.hash=te,J}},"parseError"),parse:o(function(X){var te=this,J=[0],se=[],ue=[null],Z=[],Se=this.table,ce="",ae=0,Oe=0,ge=0,ze=2,He=1,$e=Z.slice.call(arguments,1),Re=Object.create(this.lexer),Ie={yy:{}};for(var be in this.yy)Object.prototype.hasOwnProperty.call(this.yy,be)&&(Ie.yy[be]=this.yy[be]);Re.setInput(X,Ie.yy),Ie.yy.lexer=Re,Ie.yy.parser=this,typeof Re.yylloc>"u"&&(Re.yylloc={});var W=Re.yylloc;Z.push(W);var de=Re.options&&Re.options.ranges;typeof Ie.yy.parseError=="function"?this.parseError=Ie.yy.parseError:this.parseError=Object.getPrototypeOf(this).parseError;function re(Rt){J.length=J.length-2*Rt,ue.length=ue.length-Rt,Z.length=Z.length-Rt}o(re,"popStack");function oe(){var Rt;return Rt=se.pop()||Re.lex()||He,typeof Rt!="number"&&(Rt instanceof Array&&(se=Rt,Rt=se.pop()),Rt=te.symbols_[Rt]||Rt),Rt}o(oe,"lex");for(var V,xe,q,pe,ve,Pe,_e={},we,Ve,De,qe;;){if(q=J[J.length-1],this.defaultActions[q]?pe=this.defaultActions[q]:((V===null||typeof V>"u")&&(V=oe()),pe=Se[q]&&Se[q][V]),typeof pe>"u"||!pe.length||!pe[0]){var at="";qe=[];for(we in Se[q])this.terminals_[we]&&we>ze&&qe.push("'"+this.terminals_[we]+"'");Re.showPosition?at="Parse error on line "+(ae+1)+`: +`+Re.showPosition()+` +Expecting `+qe.join(", ")+", got '"+(this.terminals_[V]||V)+"'":at="Parse error on line "+(ae+1)+": Unexpected "+(V==He?"end of input":"'"+(this.terminals_[V]||V)+"'"),this.parseError(at,{text:Re.match,token:this.terminals_[V]||V,line:Re.yylineno,loc:W,expected:qe})}if(pe[0]instanceof Array&&pe.length>1)throw new Error("Parse Error: multiple actions possible at state: "+q+", token: "+V);switch(pe[0]){case 1:J.push(V),ue.push(Re.yytext),Z.push(Re.yylloc),J.push(pe[1]),V=null,xe?(V=xe,xe=null):(Oe=Re.yyleng,ce=Re.yytext,ae=Re.yylineno,W=Re.yylloc,ge>0&&ge--);break;case 2:if(Ve=this.productions_[pe[1]][1],_e.$=ue[ue.length-Ve],_e._$={first_line:Z[Z.length-(Ve||1)].first_line,last_line:Z[Z.length-1].last_line,first_column:Z[Z.length-(Ve||1)].first_column,last_column:Z[Z.length-1].last_column},de&&(_e._$.range=[Z[Z.length-(Ve||1)].range[0],Z[Z.length-1].range[1]]),Pe=this.performAction.apply(_e,[ce,Oe,ae,Ie.yy,pe[1],ue,Z].concat($e)),typeof Pe<"u")return Pe;Ve&&(J=J.slice(0,-1*Ve*2),ue=ue.slice(0,-1*Ve),Z=Z.slice(0,-1*Ve)),J.push(this.productions_[pe[1]][0]),ue.push(_e.$),Z.push(_e._$),De=Se[J[J.length-2]][J[J.length-1]],J.push(De);break;case 3:return!0}}return!0},"parse")},le=function(){var K={EOF:1,parseError:o(function(te,J){if(this.yy.parser)this.yy.parser.parseError(te,J);else throw new Error(te)},"parseError"),setInput:o(function(X,te){return this.yy=te||this.yy||{},this._input=X,this._more=this._backtrack=this.done=!1,this.yylineno=this.yyleng=0,this.yytext=this.matched=this.match="",this.conditionStack=["INITIAL"],this.yylloc={first_line:1,first_column:0,last_line:1,last_column:0},this.options.ranges&&(this.yylloc.range=[0,0]),this.offset=0,this},"setInput"),input:o(function(){var X=this._input[0];this.yytext+=X,this.yyleng++,this.offset++,this.match+=X,this.matched+=X;var te=X.match(/(?:\r\n?|\n).*/g);return te?(this.yylineno++,this.yylloc.last_line++):this.yylloc.last_column++,this.options.ranges&&this.yylloc.range[1]++,this._input=this._input.slice(1),X},"input"),unput:o(function(X){var te=X.length,J=X.split(/(?:\r\n?|\n)/g);this._input=X+this._input,this.yytext=this.yytext.substr(0,this.yytext.length-te),this.offset-=te;var se=this.match.split(/(?:\r\n?|\n)/g);this.match=this.match.substr(0,this.match.length-1),this.matched=this.matched.substr(0,this.matched.length-1),J.length-1&&(this.yylineno-=J.length-1);var ue=this.yylloc.range;return this.yylloc={first_line:this.yylloc.first_line,last_line:this.yylineno+1,first_column:this.yylloc.first_column,last_column:J?(J.length===se.length?this.yylloc.first_column:0)+se[se.length-J.length].length-J[0].length:this.yylloc.first_column-te},this.options.ranges&&(this.yylloc.range=[ue[0],ue[0]+this.yyleng-te]),this.yyleng=this.yytext.length,this},"unput"),more:o(function(){return this._more=!0,this},"more"),reject:o(function(){if(this.options.backtrack_lexer)this._backtrack=!0;else return this.parseError("Lexical error on line "+(this.yylineno+1)+`. You can only invoke reject() in the lexer when the lexer is of the backtracking persuasion (options.backtrack_lexer = true). +`+this.showPosition(),{text:"",token:null,line:this.yylineno});return this},"reject"),less:o(function(X){this.unput(this.match.slice(X))},"less"),pastInput:o(function(){var X=this.matched.substr(0,this.matched.length-this.match.length);return(X.length>20?"...":"")+X.substr(-20).replace(/\n/g,"")},"pastInput"),upcomingInput:o(function(){var X=this.match;return X.length<20&&(X+=this._input.substr(0,20-X.length)),(X.substr(0,20)+(X.length>20?"...":"")).replace(/\n/g,"")},"upcomingInput"),showPosition:o(function(){var X=this.pastInput(),te=new Array(X.length+1).join("-");return X+this.upcomingInput()+` +`+te+"^"},"showPosition"),test_match:o(function(X,te){var J,se,ue;if(this.options.backtrack_lexer&&(ue={yylineno:this.yylineno,yylloc:{first_line:this.yylloc.first_line,last_line:this.last_line,first_column:this.yylloc.first_column,last_column:this.yylloc.last_column},yytext:this.yytext,match:this.match,matches:this.matches,matched:this.matched,yyleng:this.yyleng,offset:this.offset,_more:this._more,_input:this._input,yy:this.yy,conditionStack:this.conditionStack.slice(0),done:this.done},this.options.ranges&&(ue.yylloc.range=this.yylloc.range.slice(0))),se=X[0].match(/(?:\r\n?|\n).*/g),se&&(this.yylineno+=se.length),this.yylloc={first_line:this.yylloc.last_line,last_line:this.yylineno+1,first_column:this.yylloc.last_column,last_column:se?se[se.length-1].length-se[se.length-1].match(/\r?\n?/)[0].length:this.yylloc.last_column+X[0].length},this.yytext+=X[0],this.match+=X[0],this.matches=X,this.yyleng=this.yytext.length,this.options.ranges&&(this.yylloc.range=[this.offset,this.offset+=this.yyleng]),this._more=!1,this._backtrack=!1,this._input=this._input.slice(X[0].length),this.matched+=X[0],J=this.performAction.call(this,this.yy,this,te,this.conditionStack[this.conditionStack.length-1]),this.done&&this._input&&(this.done=!1),J)return J;if(this._backtrack){for(var Z in ue)this[Z]=ue[Z];return!1}return!1},"test_match"),next:o(function(){if(this.done)return this.EOF;this._input||(this.done=!0);var X,te,J,se;this._more||(this.yytext="",this.match="");for(var ue=this._currentRules(),Z=0;Zte[0].length)){if(te=J,se=Z,this.options.backtrack_lexer){if(X=this.test_match(J,ue[Z]),X!==!1)return X;if(this._backtrack){te=!1;continue}else return!1}else if(!this.options.flex)break}return te?(X=this.test_match(te,ue[se]),X!==!1?X:!1):this._input===""?this.EOF:this.parseError("Lexical error on line "+(this.yylineno+1)+`. Unrecognized text. +`+this.showPosition(),{text:"",token:null,line:this.yylineno})},"next"),lex:o(function(){var te=this.next();return te||this.lex()},"lex"),begin:o(function(te){this.conditionStack.push(te)},"begin"),popState:o(function(){var te=this.conditionStack.length-1;return te>0?this.conditionStack.pop():this.conditionStack[0]},"popState"),_currentRules:o(function(){return this.conditionStack.length&&this.conditionStack[this.conditionStack.length-1]?this.conditions[this.conditionStack[this.conditionStack.length-1]].rules:this.conditions.INITIAL.rules},"_currentRules"),topState:o(function(te){return te=this.conditionStack.length-1-Math.abs(te||0),te>=0?this.conditionStack[te]:"INITIAL"},"topState"),pushState:o(function(te){this.begin(te)},"pushState"),stateStackSize:o(function(){return this.conditionStack.length},"stateStackSize"),options:{"case-insensitive":!0},performAction:o(function(te,J,se,ue){var Z=ue;switch(se){case 0:return 5;case 1:break;case 2:break;case 3:break;case 4:break;case 5:break;case 6:return 19;case 7:return this.begin("LINE"),14;break;case 8:return this.begin("ID"),50;break;case 9:return this.begin("ID"),52;break;case 10:return 13;case 11:return this.begin("ID"),53;break;case 12:return J.yytext=J.yytext.trim(),this.begin("ALIAS"),70;break;case 13:return this.popState(),this.popState(),this.begin("LINE"),51;break;case 14:return this.popState(),this.popState(),5;break;case 15:return this.begin("LINE"),36;break;case 16:return this.begin("LINE"),37;break;case 17:return this.begin("LINE"),38;break;case 18:return this.begin("LINE"),39;break;case 19:return this.begin("LINE"),49;break;case 20:return this.begin("LINE"),41;break;case 21:return this.begin("LINE"),43;break;case 22:return this.begin("LINE"),48;break;case 23:return this.begin("LINE"),44;break;case 24:return this.begin("LINE"),47;break;case 25:return this.begin("LINE"),46;break;case 26:return this.popState(),15;break;case 27:return 16;case 28:return 65;case 29:return 66;case 30:return 59;case 31:return 60;case 32:return 61;case 33:return 62;case 34:return 57;case 35:return 54;case 36:return this.begin("ID"),21;break;case 37:return this.begin("ID"),23;break;case 38:return 29;case 39:return 30;case 40:return this.begin("acc_title"),31;break;case 41:return this.popState(),"acc_title_value";break;case 42:return this.begin("acc_descr"),33;break;case 43:return this.popState(),"acc_descr_value";break;case 44:this.begin("acc_descr_multiline");break;case 45:this.popState();break;case 46:return"acc_descr_multiline_value";case 47:return 6;case 48:return 18;case 49:return 20;case 50:return 64;case 51:return 5;case 52:return J.yytext=J.yytext.trim(),70;break;case 53:return 73;case 54:return 74;case 55:return 75;case 56:return 76;case 57:return 71;case 58:return 72;case 59:return 77;case 60:return 78;case 61:return 79;case 62:return 80;case 63:return 81;case 64:return 68;case 65:return 69;case 66:return 5;case 67:return"INVALID"}},"anonymous"),rules:[/^(?:[\n]+)/i,/^(?:\s+)/i,/^(?:((?!\n)\s)+)/i,/^(?:#[^\n]*)/i,/^(?:%(?!\{)[^\n]*)/i,/^(?:[^\}]%%[^\n]*)/i,/^(?:[0-9]+(?=[ \n]+))/i,/^(?:box\b)/i,/^(?:participant\b)/i,/^(?:actor\b)/i,/^(?:create\b)/i,/^(?:destroy\b)/i,/^(?:[^\<->\->:\n,;]+?([\-]*[^\<->\->:\n,;]+?)*?(?=((?!\n)\s)+as(?!\n)\s|[#\n;]|$))/i,/^(?:as\b)/i,/^(?:(?:))/i,/^(?:loop\b)/i,/^(?:rect\b)/i,/^(?:opt\b)/i,/^(?:alt\b)/i,/^(?:else\b)/i,/^(?:par\b)/i,/^(?:par_over\b)/i,/^(?:and\b)/i,/^(?:critical\b)/i,/^(?:option\b)/i,/^(?:break\b)/i,/^(?:(?:[:]?(?:no)?wrap)?[^#\n;]*)/i,/^(?:end\b)/i,/^(?:left of\b)/i,/^(?:right of\b)/i,/^(?:links\b)/i,/^(?:link\b)/i,/^(?:properties\b)/i,/^(?:details\b)/i,/^(?:over\b)/i,/^(?:note\b)/i,/^(?:activate\b)/i,/^(?:deactivate\b)/i,/^(?:title\s[^#\n;]+)/i,/^(?:title:\s[^#\n;]+)/i,/^(?:accTitle\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*\{\s*)/i,/^(?:[\}])/i,/^(?:[^\}]*)/i,/^(?:sequenceDiagram\b)/i,/^(?:autonumber\b)/i,/^(?:off\b)/i,/^(?:,)/i,/^(?:;)/i,/^(?:[^\+\<->\->:\n,;]+((?!(-x|--x|-\)|--\)))[\-]*[^\+\<->\->:\n,;]+)*)/i,/^(?:->>)/i,/^(?:<<->>)/i,/^(?:-->>)/i,/^(?:<<-->>)/i,/^(?:->)/i,/^(?:-->)/i,/^(?:-[x])/i,/^(?:--[x])/i,/^(?:-[\)])/i,/^(?:--[\)])/i,/^(?::(?:(?:no)?wrap)?[^#\n;]+)/i,/^(?:\+)/i,/^(?:-)/i,/^(?:$)/i,/^(?:.)/i],conditions:{acc_descr_multiline:{rules:[45,46],inclusive:!1},acc_descr:{rules:[43],inclusive:!1},acc_title:{rules:[41],inclusive:!1},ID:{rules:[2,3,12],inclusive:!1},ALIAS:{rules:[2,3,13,14],inclusive:!1},LINE:{rules:[2,3,26],inclusive:!1},INITIAL:{rules:[0,1,3,4,5,6,7,8,9,10,11,15,16,17,18,19,20,21,22,23,24,25,27,28,29,30,31,32,33,34,35,36,37,38,39,40,42,44,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67],inclusive:!0}}};return K}();ne.lexer=le;function he(){this.yy={}}return o(he,"Parser"),he.prototype=ne,ne.Parser=he,new he}();CO.parser=CO;lfe=CO});var iVe,aVe,sVe,_6,ufe=N(()=>{"use strict";zt();vt();s6();gr();mi();iVe={SOLID:0,DOTTED:1,NOTE:2,SOLID_CROSS:3,DOTTED_CROSS:4,SOLID_OPEN:5,DOTTED_OPEN:6,LOOP_START:10,LOOP_END:11,ALT_START:12,ALT_ELSE:13,ALT_END:14,OPT_START:15,OPT_END:16,ACTIVE_START:17,ACTIVE_END:18,PAR_START:19,PAR_AND:20,PAR_END:21,RECT_START:22,RECT_END:23,SOLID_POINT:24,DOTTED_POINT:25,AUTONUMBER:26,CRITICAL_START:27,CRITICAL_OPTION:28,CRITICAL_END:29,BREAK_START:30,BREAK_END:31,PAR_OVER_START:32,BIDIRECTIONAL_SOLID:33,BIDIRECTIONAL_DOTTED:34},aVe={FILLED:0,OPEN:1},sVe={LEFTOF:0,RIGHTOF:1,OVER:2},_6=class{constructor(){this.state=new pf(()=>({prevActor:void 0,actors:new Map,createdActors:new Map,destroyedActors:new Map,boxes:[],messages:[],notes:[],sequenceNumbersEnabled:!1,wrapEnabled:void 0,currentBox:void 0,lastCreated:void 0,lastDestroyed:void 0}));this.setAccTitle=Lr;this.setAccDescription=Nr;this.setDiagramTitle=$r;this.getAccTitle=Rr;this.getAccDescription=Mr;this.getDiagramTitle=Ir;this.apply=this.apply.bind(this),this.parseBoxData=this.parseBoxData.bind(this),this.parseMessage=this.parseMessage.bind(this),this.clear(),this.setWrap(me().wrap),this.LINETYPE=iVe,this.ARROWTYPE=aVe,this.PLACEMENT=sVe}static{o(this,"SequenceDB")}addBox(e){this.state.records.boxes.push({name:e.text,wrap:e.wrap??this.autoWrap(),fill:e.color,actorKeys:[]}),this.state.records.currentBox=this.state.records.boxes.slice(-1)[0]}addActor(e,r,n,i){let a=this.state.records.currentBox,s=this.state.records.actors.get(e);if(s){if(this.state.records.currentBox&&s.box&&this.state.records.currentBox!==s.box)throw new Error(`A same participant should only be defined in one Box: ${s.name} can't be in '${s.box.name}' and in '${this.state.records.currentBox.name}' at the same time.`);if(a=s.box?s.box:this.state.records.currentBox,s.box=a,s&&r===s.name&&n==null)return}if(n?.text==null&&(n={text:r,type:i}),(i==null||n.text==null)&&(n={text:r,type:i}),this.state.records.actors.set(e,{box:a,name:r,description:n.text,wrap:n.wrap??this.autoWrap(),prevActor:this.state.records.prevActor,links:{},properties:{},actorCnt:null,rectData:null,type:i??"participant"}),this.state.records.prevActor){let l=this.state.records.actors.get(this.state.records.prevActor);l&&(l.nextActor=e)}this.state.records.currentBox&&this.state.records.currentBox.actorKeys.push(e),this.state.records.prevActor=e}activationCount(e){let r,n=0;if(!e)return 0;for(r=0;r>-",token:"->>-",line:"1",loc:{first_line:1,last_line:1,first_column:1,last_column:1},expected:["'ACTIVE_PARTICIPANT'"]},l}return this.state.records.messages.push({id:this.state.records.messages.length.toString(),from:e,to:r,message:n?.text??"",wrap:n?.wrap??this.autoWrap(),type:i,activate:a}),!0}hasAtLeastOneBox(){return this.state.records.boxes.length>0}hasAtLeastOneBoxWithTitle(){return this.state.records.boxes.some(e=>e.name)}getMessages(){return this.state.records.messages}getBoxes(){return this.state.records.boxes}getActors(){return this.state.records.actors}getCreatedActors(){return this.state.records.createdActors}getDestroyedActors(){return this.state.records.destroyedActors}getActor(e){return this.state.records.actors.get(e)}getActorKeys(){return[...this.state.records.actors.keys()]}enableSequenceNumbers(){this.state.records.sequenceNumbersEnabled=!0}disableSequenceNumbers(){this.state.records.sequenceNumbersEnabled=!1}showSequenceNumbers(){return this.state.records.sequenceNumbersEnabled}setWrap(e){this.state.records.wrapEnabled=e}extractWrap(e){if(e===void 0)return{};e=e.trim();let r=/^:?wrap:/.exec(e)!==null?!0:/^:?nowrap:/.exec(e)!==null?!1:void 0;return{cleanedText:(r===void 0?e:e.replace(/^:?(?:no)?wrap:/,"")).trim(),wrap:r}}autoWrap(){return this.state.records.wrapEnabled!==void 0?this.state.records.wrapEnabled:me().sequence?.wrap??!1}clear(){this.state.reset(),Ar()}parseMessage(e){let r=e.trim(),{wrap:n,cleanedText:i}=this.extractWrap(r),a={text:i,wrap:n};return Y.debug(`parseMessage: ${JSON.stringify(a)}`),a}parseBoxData(e){let r=/^((?:rgba?|hsla?)\s*\(.*\)|\w*)(.*)$/.exec(e),n=r?.[1]?r[1].trim():"transparent",i=r?.[2]?r[2].trim():void 0;if(window?.CSS)window.CSS.supports("color",n)||(n="transparent",i=e.trim());else{let l=new Option().style;l.color=n,l.color!==n&&(n="transparent",i=e.trim())}let{wrap:a,cleanedText:s}=this.extractWrap(i);return{text:s?Tr(s,me()):void 0,color:n,wrap:a}}addNote(e,r,n){let i={actor:e,placement:r,message:n.text,wrap:n.wrap??this.autoWrap()},a=[].concat(e,e);this.state.records.notes.push(i),this.state.records.messages.push({id:this.state.records.messages.length.toString(),from:a[0],to:a[1],message:n.text,wrap:n.wrap??this.autoWrap(),type:this.LINETYPE.NOTE,placement:r})}addLinks(e,r){let n=this.getActor(e);try{let i=Tr(r.text,me());i=i.replace(/=/g,"="),i=i.replace(/&/g,"&");let a=JSON.parse(i);this.insertLinks(n,a)}catch(i){Y.error("error while parsing actor link text",i)}}addALink(e,r){let n=this.getActor(e);try{let i={},a=Tr(r.text,me()),s=a.indexOf("@");a=a.replace(/=/g,"="),a=a.replace(/&/g,"&");let l=a.slice(0,s-1).trim(),u=a.slice(s+1).trim();i[l]=u,this.insertLinks(n,i)}catch(i){Y.error("error while parsing actor link text",i)}}insertLinks(e,r){if(e.links==null)e.links=r;else for(let n in r)e.links[n]=r[n]}addProperties(e,r){let n=this.getActor(e);try{let i=Tr(r.text,me()),a=JSON.parse(i);this.insertProperties(n,a)}catch(i){Y.error("error while parsing actor properties text",i)}}insertProperties(e,r){if(e.properties==null)e.properties=r;else for(let n in r)e.properties[n]=r[n]}boxEnd(){this.state.records.currentBox=void 0}addDetails(e,r){let n=this.getActor(e),i=document.getElementById(r.text);try{let a=i.innerHTML,s=JSON.parse(a);s.properties&&this.insertProperties(n,s.properties),s.links&&this.insertLinks(n,s.links)}catch(a){Y.error("error while parsing actor details text",a)}}getActorProperty(e,r){if(e?.properties!==void 0)return e.properties[r]}apply(e){if(Array.isArray(e))e.forEach(r=>{this.apply(r)});else switch(e.type){case"sequenceIndex":this.state.records.messages.push({id:this.state.records.messages.length.toString(),from:void 0,to:void 0,message:{start:e.sequenceIndex,step:e.sequenceIndexStep,visible:e.sequenceVisible},wrap:!1,type:e.signalType});break;case"addParticipant":this.addActor(e.actor,e.actor,e.description,e.draw);break;case"createParticipant":if(this.state.records.actors.has(e.actor))throw new Error("It is not possible to have actors with the same id, even if one is destroyed before the next is created. Use 'AS' aliases to simulate the behavior");this.state.records.lastCreated=e.actor,this.addActor(e.actor,e.actor,e.description,e.draw),this.state.records.createdActors.set(e.actor,this.state.records.messages.length);break;case"destroyParticipant":this.state.records.lastDestroyed=e.actor,this.state.records.destroyedActors.set(e.actor,this.state.records.messages.length);break;case"activeStart":this.addSignal(e.actor,void 0,void 0,e.signalType);break;case"activeEnd":this.addSignal(e.actor,void 0,void 0,e.signalType);break;case"addNote":this.addNote(e.actor,e.placement,e.text);break;case"addLinks":this.addLinks(e.actor,e.text);break;case"addALink":this.addALink(e.actor,e.text);break;case"addProperties":this.addProperties(e.actor,e.text);break;case"addDetails":this.addDetails(e.actor,e.text);break;case"addMessage":if(this.state.records.lastCreated){if(e.to!==this.state.records.lastCreated)throw new Error("The created participant "+this.state.records.lastCreated.name+" does not have an associated creating message after its declaration. Please check the sequence diagram.");this.state.records.lastCreated=void 0}else if(this.state.records.lastDestroyed){if(e.to!==this.state.records.lastDestroyed&&e.from!==this.state.records.lastDestroyed)throw new Error("The destroyed participant "+this.state.records.lastDestroyed.name+" does not have an associated destroying message after its declaration. Please check the sequence diagram.");this.state.records.lastDestroyed=void 0}this.addSignal(e.from,e.to,e.msg,e.signalType,e.activate);break;case"boxStart":this.addBox(e.boxData);break;case"boxEnd":this.boxEnd();break;case"loopStart":this.addSignal(void 0,void 0,e.loopText,e.signalType);break;case"loopEnd":this.addSignal(void 0,void 0,void 0,e.signalType);break;case"rectStart":this.addSignal(void 0,void 0,e.color,e.signalType);break;case"rectEnd":this.addSignal(void 0,void 0,void 0,e.signalType);break;case"optStart":this.addSignal(void 0,void 0,e.optText,e.signalType);break;case"optEnd":this.addSignal(void 0,void 0,void 0,e.signalType);break;case"altStart":this.addSignal(void 0,void 0,e.altText,e.signalType);break;case"else":this.addSignal(void 0,void 0,e.altText,e.signalType);break;case"altEnd":this.addSignal(void 0,void 0,void 0,e.signalType);break;case"setAccTitle":Lr(e.text);break;case"parStart":this.addSignal(void 0,void 0,e.parText,e.signalType);break;case"and":this.addSignal(void 0,void 0,e.parText,e.signalType);break;case"parEnd":this.addSignal(void 0,void 0,void 0,e.signalType);break;case"criticalStart":this.addSignal(void 0,void 0,e.criticalText,e.signalType);break;case"option":this.addSignal(void 0,void 0,e.optionText,e.signalType);break;case"criticalEnd":this.addSignal(void 0,void 0,void 0,e.signalType);break;case"breakStart":this.addSignal(void 0,void 0,e.breakText,e.signalType);break;case"breakEnd":this.addSignal(void 0,void 0,void 0,e.signalType);break}}getConfig(){return me().sequence}}});var oVe,hfe,ffe=N(()=>{"use strict";oVe=o(t=>`.actor { + stroke: ${t.actorBorder}; + fill: ${t.actorBkg}; + } + + text.actor > tspan { + fill: ${t.actorTextColor}; + stroke: none; + } + + .actor-line { + stroke: ${t.actorLineColor}; + } + + .messageLine0 { + stroke-width: 1.5; + stroke-dasharray: none; + stroke: ${t.signalColor}; + } + + .messageLine1 { + stroke-width: 1.5; + stroke-dasharray: 2, 2; + stroke: ${t.signalColor}; + } + + #arrowhead path { + fill: ${t.signalColor}; + stroke: ${t.signalColor}; + } + + .sequenceNumber { + fill: ${t.sequenceNumberColor}; + } + + #sequencenumber { + fill: ${t.signalColor}; + } + + #crosshead path { + fill: ${t.signalColor}; + stroke: ${t.signalColor}; + } + + .messageText { + fill: ${t.signalTextColor}; + stroke: none; + } + + .labelBox { + stroke: ${t.labelBoxBorderColor}; + fill: ${t.labelBoxBkgColor}; + } + + .labelText, .labelText > tspan { + fill: ${t.labelTextColor}; + stroke: none; + } + + .loopText, .loopText > tspan { + fill: ${t.loopTextColor}; + stroke: none; + } + + .loopLine { + stroke-width: 2px; + stroke-dasharray: 2, 2; + stroke: ${t.labelBoxBorderColor}; + fill: ${t.labelBoxBorderColor}; + } + + .note { + //stroke: #decc93; + stroke: ${t.noteBorderColor}; + fill: ${t.noteBkgColor}; + } + + .noteText, .noteText > tspan { + fill: ${t.noteTextColor}; + stroke: none; + } + + .activation0 { + fill: ${t.activationBkgColor}; + stroke: ${t.activationBorderColor}; + } + + .activation1 { + fill: ${t.activationBkgColor}; + stroke: ${t.activationBorderColor}; + } + + .activation2 { + fill: ${t.activationBkgColor}; + stroke: ${t.activationBorderColor}; + } + + .actorPopupMenu { + position: absolute; + } + + .actorPopupMenuPanel { + position: absolute; + fill: ${t.actorBkg}; + box-shadow: 0px 8px 16px 0px rgba(0,0,0,0.2); + filter: drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4)); +} + .actor-man line { + stroke: ${t.actorBorder}; + fill: ${t.actorBkg}; + } + .actor-man circle, line { + stroke: ${t.actorBorder}; + fill: ${t.actorBkg}; + stroke-width: 2px; + } +`,"getStyles"),hfe=oVe});var AO,vf,pfe,mfe,lVe,dfe,_O,cVe,uVe,Tb,_p,gfe,Uc,DO,hVe,fVe,dVe,pVe,mVe,gVe,yVe,yfe,vVe,xVe,bVe,wVe,TVe,kVe,EVe,vfe,SVe,LO,CVe,hi,xfe=N(()=>{"use strict";gr();Wv();ir();AO=Sa(z0(),1);ji();vf=18*2,pfe="actor-top",mfe="actor-bottom",lVe="actor-box",dfe="actor-man",_O=o(function(t,e){return kd(t,e)},"drawRect"),cVe=o(function(t,e,r,n,i){if(e.links===void 0||e.links===null||Object.keys(e.links).length===0)return{height:0,width:0};let a=e.links,s=e.actorCnt,l=e.rectData;var u="none";i&&(u="block !important");let h=t.append("g");h.attr("id","actor"+s+"_popup"),h.attr("class","actorPopupMenu"),h.attr("display",u);var f="";l.class!==void 0&&(f=" "+l.class);let d=l.width>r?l.width:r,p=h.append("rect");if(p.attr("class","actorPopupMenuPanel"+f),p.attr("x",l.x),p.attr("y",l.height),p.attr("fill",l.fill),p.attr("stroke",l.stroke),p.attr("width",d),p.attr("height",l.height),p.attr("rx",l.rx),p.attr("ry",l.ry),a!=null){var m=20;for(let v in a){var g=h.append("a"),y=(0,AO.sanitizeUrl)(a[v]);g.attr("xlink:href",y),g.attr("target","_blank"),CVe(n)(v,g,l.x+10,l.height+m,d,20,{class:"actor"},n),m+=30}}return p.attr("height",m),{height:l.height+m,width:d}},"drawPopup"),uVe=o(function(t){return"var pu = document.getElementById('"+t+"'); if (pu != null) { pu.style.display = pu.style.display == 'block' ? 'none' : 'block'; }"},"popupMenuToggle"),Tb=o(async function(t,e,r=null){let n=t.append("foreignObject"),i=await mh(e.text,cr()),s=n.append("xhtml:div").attr("style","width: fit-content;").attr("xmlns","http://www.w3.org/1999/xhtml").html(i).node().getBoundingClientRect();if(n.attr("height",Math.round(s.height)).attr("width",Math.round(s.width)),e.class==="noteText"){let l=t.node().firstChild;l.setAttribute("height",s.height+2*e.textMargin);let u=l.getBBox();n.attr("x",Math.round(u.x+u.width/2-s.width/2)).attr("y",Math.round(u.y+u.height/2-s.height/2))}else if(r){let{startx:l,stopx:u,starty:h}=r;if(l>u){let f=l;l=u,u=f}n.attr("x",Math.round(l+Math.abs(l-u)/2-s.width/2)),e.class==="loopText"?n.attr("y",Math.round(h)):n.attr("y",Math.round(h-s.height))}return[n]},"drawKatex"),_p=o(function(t,e){let r=0,n=0,i=e.text.split(Ze.lineBreakRegex),[a,s]=Bo(e.fontSize),l=[],u=0,h=o(()=>e.y,"yfunc");if(e.valign!==void 0&&e.textMargin!==void 0&&e.textMargin>0)switch(e.valign){case"top":case"start":h=o(()=>Math.round(e.y+e.textMargin),"yfunc");break;case"middle":case"center":h=o(()=>Math.round(e.y+(r+n+e.textMargin)/2),"yfunc");break;case"bottom":case"end":h=o(()=>Math.round(e.y+(r+n+2*e.textMargin)-e.textMargin),"yfunc");break}if(e.anchor!==void 0&&e.textMargin!==void 0&&e.width!==void 0)switch(e.anchor){case"left":case"start":e.x=Math.round(e.x+e.textMargin),e.anchor="start",e.dominantBaseline="middle",e.alignmentBaseline="middle";break;case"middle":case"center":e.x=Math.round(e.x+e.width/2),e.anchor="middle",e.dominantBaseline="middle",e.alignmentBaseline="middle";break;case"right":case"end":e.x=Math.round(e.x+e.width-e.textMargin),e.anchor="end",e.dominantBaseline="middle",e.alignmentBaseline="middle";break}for(let[f,d]of i.entries()){e.textMargin!==void 0&&e.textMargin===0&&a!==void 0&&(u=f*a);let p=t.append("text");p.attr("x",e.x),p.attr("y",h()),e.anchor!==void 0&&p.attr("text-anchor",e.anchor).attr("dominant-baseline",e.dominantBaseline).attr("alignment-baseline",e.alignmentBaseline),e.fontFamily!==void 0&&p.style("font-family",e.fontFamily),s!==void 0&&p.style("font-size",s),e.fontWeight!==void 0&&p.style("font-weight",e.fontWeight),e.fill!==void 0&&p.attr("fill",e.fill),e.class!==void 0&&p.attr("class",e.class),e.dy!==void 0?p.attr("dy",e.dy):u!==0&&p.attr("dy",u);let m=d||H9;if(e.tspan){let g=p.append("tspan");g.attr("x",e.x),e.fill!==void 0&&g.attr("fill",e.fill),g.text(m)}else p.text(m);e.valign!==void 0&&e.textMargin!==void 0&&e.textMargin>0&&(n+=(p._groups||p)[0][0].getBBox().height,r=n),l.push(p)}return l},"drawText"),gfe=o(function(t,e){function r(i,a,s,l,u){return i+","+a+" "+(i+s)+","+a+" "+(i+s)+","+(a+l-u)+" "+(i+s-u*1.2)+","+(a+l)+" "+i+","+(a+l)}o(r,"genPoints");let n=t.append("polygon");return n.attr("points",r(e.x,e.y,e.width,e.height,7)),n.attr("class","labelBox"),e.y=e.y+e.height/2,_p(t,e),n},"drawLabel"),Uc=-1,DO=o((t,e,r,n)=>{t.select&&r.forEach(i=>{let a=e.get(i),s=t.select("#actor"+a.actorCnt);!n.mirrorActors&&a.stopy?s.attr("y2",a.stopy+a.height/2):n.mirrorActors&&s.attr("y2",a.stopy)})},"fixLifeLineHeights"),hVe=o(function(t,e,r,n){let i=n?e.stopy:e.starty,a=e.x+e.width/2,s=i+e.height,l=t.append("g").lower();var u=l;n||(Uc++,Object.keys(e.links||{}).length&&!r.forceMenus&&u.attr("onclick",uVe(`actor${Uc}_popup`)).attr("cursor","pointer"),u.append("line").attr("id","actor"+Uc).attr("x1",a).attr("y1",s).attr("x2",a).attr("y2",2e3).attr("class","actor-line 200").attr("stroke-width","0.5px").attr("stroke","#999").attr("name",e.name),u=l.append("g"),e.actorCnt=Uc,e.links!=null&&u.attr("id","root-"+Uc));let h=Tl();var f="actor";e.properties?.class?f=e.properties.class:h.fill="#eaeaea",n?f+=` ${mfe}`:f+=` ${pfe}`,h.x=e.x,h.y=i,h.width=e.width,h.height=e.height,h.class=f,h.rx=3,h.ry=3,h.name=e.name;let d=_O(u,h);if(e.rectData=h,e.properties?.icon){let m=e.properties.icon.trim();m.charAt(0)==="@"?Iq(u,h.x+h.width-20,h.y+10,m.substr(1)):Mq(u,h.x+h.width-20,h.y+10,m)}LO(r,pi(e.description))(e.description,u,h.x,h.y,h.width,h.height,{class:`actor ${lVe}`},r);let p=e.height;if(d.node){let m=d.node().getBBox();e.height=m.height,p=m.height}return p},"drawActorTypeParticipant"),fVe=o(function(t,e,r,n){let i=n?e.stopy:e.starty,a=e.x+e.width/2,s=i+80,l=t.append("g").lower();n||(Uc++,l.append("line").attr("id","actor"+Uc).attr("x1",a).attr("y1",s).attr("x2",a).attr("y2",2e3).attr("class","actor-line 200").attr("stroke-width","0.5px").attr("stroke","#999").attr("name",e.name),e.actorCnt=Uc);let u=t.append("g"),h=dfe;n?h+=` ${mfe}`:h+=` ${pfe}`,u.attr("class",h),u.attr("name",e.name);let f=Tl();f.x=e.x,f.y=i,f.fill="#eaeaea",f.width=e.width,f.height=e.height,f.class="actor",f.rx=3,f.ry=3,u.append("line").attr("id","actor-man-torso"+Uc).attr("x1",a).attr("y1",i+25).attr("x2",a).attr("y2",i+45),u.append("line").attr("id","actor-man-arms"+Uc).attr("x1",a-vf/2).attr("y1",i+33).attr("x2",a+vf/2).attr("y2",i+33),u.append("line").attr("x1",a-vf/2).attr("y1",i+60).attr("x2",a).attr("y2",i+45),u.append("line").attr("x1",a).attr("y1",i+45).attr("x2",a+vf/2-2).attr("y2",i+60);let d=u.append("circle");d.attr("cx",e.x+e.width/2),d.attr("cy",i+10),d.attr("r",15),d.attr("width",e.width),d.attr("height",e.height);let p=u.node().getBBox();return e.height=p.height,LO(r,pi(e.description))(e.description,u,f.x,f.y+35,f.width,f.height,{class:`actor ${dfe}`},r),e.height},"drawActorTypeActor"),dVe=o(async function(t,e,r,n){switch(e.type){case"actor":return await fVe(t,e,r,n);case"participant":return await hVe(t,e,r,n)}},"drawActor"),pVe=o(function(t,e,r){let i=t.append("g");yfe(i,e),e.name&&LO(r)(e.name,i,e.x,e.y+(e.textMaxHeight||0)/2,e.width,0,{class:"text"},r),i.lower()},"drawBox"),mVe=o(function(t){return t.append("g")},"anchorElement"),gVe=o(function(t,e,r,n,i){let a=Tl(),s=e.anchored;a.x=e.startx,a.y=e.starty,a.class="activation"+i%3,a.width=e.stopx-e.startx,a.height=r-e.starty,_O(s,a)},"drawActivation"),yVe=o(async function(t,e,r,n){let{boxMargin:i,boxTextMargin:a,labelBoxHeight:s,labelBoxWidth:l,messageFontFamily:u,messageFontSize:h,messageFontWeight:f}=n,d=t.append("g"),p=o(function(y,v,x,b){return d.append("line").attr("x1",y).attr("y1",v).attr("x2",x).attr("y2",b).attr("class","loopLine")},"drawLoopLine");p(e.startx,e.starty,e.stopx,e.starty),p(e.stopx,e.starty,e.stopx,e.stopy),p(e.startx,e.stopy,e.stopx,e.stopy),p(e.startx,e.starty,e.startx,e.stopy),e.sections!==void 0&&e.sections.forEach(function(y){p(e.startx,y.y,e.stopx,y.y).style("stroke-dasharray","3, 3")});let m=Hv();m.text=r,m.x=e.startx,m.y=e.starty,m.fontFamily=u,m.fontSize=h,m.fontWeight=f,m.anchor="middle",m.valign="middle",m.tspan=!1,m.width=l||50,m.height=s||20,m.textMargin=a,m.class="labelText",gfe(d,m),m=vfe(),m.text=e.title,m.x=e.startx+l/2+(e.stopx-e.startx)/2,m.y=e.starty+i+a,m.anchor="middle",m.valign="middle",m.textMargin=a,m.class="loopText",m.fontFamily=u,m.fontSize=h,m.fontWeight=f,m.wrap=!0;let g=pi(m.text)?await Tb(d,m,e):_p(d,m);if(e.sectionTitles!==void 0){for(let[y,v]of Object.entries(e.sectionTitles))if(v.message){m.text=v.message,m.x=e.startx+(e.stopx-e.startx)/2,m.y=e.sections[y].y+i+a,m.class="loopText",m.anchor="middle",m.valign="middle",m.tspan=!1,m.fontFamily=u,m.fontSize=h,m.fontWeight=f,m.wrap=e.wrap,pi(m.text)?(e.starty=e.sections[y].y,await Tb(d,m,e)):_p(d,m);let x=Math.round(g.map(b=>(b._groups||b)[0][0].getBBox().height).reduce((b,w)=>b+w));e.sections[y].height+=x-(i+a)}}return e.height=Math.round(e.stopy-e.starty),d},"drawLoop"),yfe=o(function(t,e){q5(t,e)},"drawBackgroundRect"),vVe=o(function(t){t.append("defs").append("symbol").attr("id","database").attr("fill-rule","evenodd").attr("clip-rule","evenodd").append("path").attr("transform","scale(.5)").attr("d","M12.258.001l.256.004.255.005.253.008.251.01.249.012.247.015.246.016.242.019.241.02.239.023.236.024.233.027.231.028.229.031.225.032.223.034.22.036.217.038.214.04.211.041.208.043.205.045.201.046.198.048.194.05.191.051.187.053.183.054.18.056.175.057.172.059.168.06.163.061.16.063.155.064.15.066.074.033.073.033.071.034.07.034.069.035.068.035.067.035.066.035.064.036.064.036.062.036.06.036.06.037.058.037.058.037.055.038.055.038.053.038.052.038.051.039.05.039.048.039.047.039.045.04.044.04.043.04.041.04.04.041.039.041.037.041.036.041.034.041.033.042.032.042.03.042.029.042.027.042.026.043.024.043.023.043.021.043.02.043.018.044.017.043.015.044.013.044.012.044.011.045.009.044.007.045.006.045.004.045.002.045.001.045v17l-.001.045-.002.045-.004.045-.006.045-.007.045-.009.044-.011.045-.012.044-.013.044-.015.044-.017.043-.018.044-.02.043-.021.043-.023.043-.024.043-.026.043-.027.042-.029.042-.03.042-.032.042-.033.042-.034.041-.036.041-.037.041-.039.041-.04.041-.041.04-.043.04-.044.04-.045.04-.047.039-.048.039-.05.039-.051.039-.052.038-.053.038-.055.038-.055.038-.058.037-.058.037-.06.037-.06.036-.062.036-.064.036-.064.036-.066.035-.067.035-.068.035-.069.035-.07.034-.071.034-.073.033-.074.033-.15.066-.155.064-.16.063-.163.061-.168.06-.172.059-.175.057-.18.056-.183.054-.187.053-.191.051-.194.05-.198.048-.201.046-.205.045-.208.043-.211.041-.214.04-.217.038-.22.036-.223.034-.225.032-.229.031-.231.028-.233.027-.236.024-.239.023-.241.02-.242.019-.246.016-.247.015-.249.012-.251.01-.253.008-.255.005-.256.004-.258.001-.258-.001-.256-.004-.255-.005-.253-.008-.251-.01-.249-.012-.247-.015-.245-.016-.243-.019-.241-.02-.238-.023-.236-.024-.234-.027-.231-.028-.228-.031-.226-.032-.223-.034-.22-.036-.217-.038-.214-.04-.211-.041-.208-.043-.204-.045-.201-.046-.198-.048-.195-.05-.19-.051-.187-.053-.184-.054-.179-.056-.176-.057-.172-.059-.167-.06-.164-.061-.159-.063-.155-.064-.151-.066-.074-.033-.072-.033-.072-.034-.07-.034-.069-.035-.068-.035-.067-.035-.066-.035-.064-.036-.063-.036-.062-.036-.061-.036-.06-.037-.058-.037-.057-.037-.056-.038-.055-.038-.053-.038-.052-.038-.051-.039-.049-.039-.049-.039-.046-.039-.046-.04-.044-.04-.043-.04-.041-.04-.04-.041-.039-.041-.037-.041-.036-.041-.034-.041-.033-.042-.032-.042-.03-.042-.029-.042-.027-.042-.026-.043-.024-.043-.023-.043-.021-.043-.02-.043-.018-.044-.017-.043-.015-.044-.013-.044-.012-.044-.011-.045-.009-.044-.007-.045-.006-.045-.004-.045-.002-.045-.001-.045v-17l.001-.045.002-.045.004-.045.006-.045.007-.045.009-.044.011-.045.012-.044.013-.044.015-.044.017-.043.018-.044.02-.043.021-.043.023-.043.024-.043.026-.043.027-.042.029-.042.03-.042.032-.042.033-.042.034-.041.036-.041.037-.041.039-.041.04-.041.041-.04.043-.04.044-.04.046-.04.046-.039.049-.039.049-.039.051-.039.052-.038.053-.038.055-.038.056-.038.057-.037.058-.037.06-.037.061-.036.062-.036.063-.036.064-.036.066-.035.067-.035.068-.035.069-.035.07-.034.072-.034.072-.033.074-.033.151-.066.155-.064.159-.063.164-.061.167-.06.172-.059.176-.057.179-.056.184-.054.187-.053.19-.051.195-.05.198-.048.201-.046.204-.045.208-.043.211-.041.214-.04.217-.038.22-.036.223-.034.226-.032.228-.031.231-.028.234-.027.236-.024.238-.023.241-.02.243-.019.245-.016.247-.015.249-.012.251-.01.253-.008.255-.005.256-.004.258-.001.258.001zm-9.258 20.499v.01l.001.021.003.021.004.022.005.021.006.022.007.022.009.023.01.022.011.023.012.023.013.023.015.023.016.024.017.023.018.024.019.024.021.024.022.025.023.024.024.025.052.049.056.05.061.051.066.051.07.051.075.051.079.052.084.052.088.052.092.052.097.052.102.051.105.052.11.052.114.051.119.051.123.051.127.05.131.05.135.05.139.048.144.049.147.047.152.047.155.047.16.045.163.045.167.043.171.043.176.041.178.041.183.039.187.039.19.037.194.035.197.035.202.033.204.031.209.03.212.029.216.027.219.025.222.024.226.021.23.02.233.018.236.016.24.015.243.012.246.01.249.008.253.005.256.004.259.001.26-.001.257-.004.254-.005.25-.008.247-.011.244-.012.241-.014.237-.016.233-.018.231-.021.226-.021.224-.024.22-.026.216-.027.212-.028.21-.031.205-.031.202-.034.198-.034.194-.036.191-.037.187-.039.183-.04.179-.04.175-.042.172-.043.168-.044.163-.045.16-.046.155-.046.152-.047.148-.048.143-.049.139-.049.136-.05.131-.05.126-.05.123-.051.118-.052.114-.051.11-.052.106-.052.101-.052.096-.052.092-.052.088-.053.083-.051.079-.052.074-.052.07-.051.065-.051.06-.051.056-.05.051-.05.023-.024.023-.025.021-.024.02-.024.019-.024.018-.024.017-.024.015-.023.014-.024.013-.023.012-.023.01-.023.01-.022.008-.022.006-.022.006-.022.004-.022.004-.021.001-.021.001-.021v-4.127l-.077.055-.08.053-.083.054-.085.053-.087.052-.09.052-.093.051-.095.05-.097.05-.1.049-.102.049-.105.048-.106.047-.109.047-.111.046-.114.045-.115.045-.118.044-.12.043-.122.042-.124.042-.126.041-.128.04-.13.04-.132.038-.134.038-.135.037-.138.037-.139.035-.142.035-.143.034-.144.033-.147.032-.148.031-.15.03-.151.03-.153.029-.154.027-.156.027-.158.026-.159.025-.161.024-.162.023-.163.022-.165.021-.166.02-.167.019-.169.018-.169.017-.171.016-.173.015-.173.014-.175.013-.175.012-.177.011-.178.01-.179.008-.179.008-.181.006-.182.005-.182.004-.184.003-.184.002h-.37l-.184-.002-.184-.003-.182-.004-.182-.005-.181-.006-.179-.008-.179-.008-.178-.01-.176-.011-.176-.012-.175-.013-.173-.014-.172-.015-.171-.016-.17-.017-.169-.018-.167-.019-.166-.02-.165-.021-.163-.022-.162-.023-.161-.024-.159-.025-.157-.026-.156-.027-.155-.027-.153-.029-.151-.03-.15-.03-.148-.031-.146-.032-.145-.033-.143-.034-.141-.035-.14-.035-.137-.037-.136-.037-.134-.038-.132-.038-.13-.04-.128-.04-.126-.041-.124-.042-.122-.042-.12-.044-.117-.043-.116-.045-.113-.045-.112-.046-.109-.047-.106-.047-.105-.048-.102-.049-.1-.049-.097-.05-.095-.05-.093-.052-.09-.051-.087-.052-.085-.053-.083-.054-.08-.054-.077-.054v4.127zm0-5.654v.011l.001.021.003.021.004.021.005.022.006.022.007.022.009.022.01.022.011.023.012.023.013.023.015.024.016.023.017.024.018.024.019.024.021.024.022.024.023.025.024.024.052.05.056.05.061.05.066.051.07.051.075.052.079.051.084.052.088.052.092.052.097.052.102.052.105.052.11.051.114.051.119.052.123.05.127.051.131.05.135.049.139.049.144.048.147.048.152.047.155.046.16.045.163.045.167.044.171.042.176.042.178.04.183.04.187.038.19.037.194.036.197.034.202.033.204.032.209.03.212.028.216.027.219.025.222.024.226.022.23.02.233.018.236.016.24.014.243.012.246.01.249.008.253.006.256.003.259.001.26-.001.257-.003.254-.006.25-.008.247-.01.244-.012.241-.015.237-.016.233-.018.231-.02.226-.022.224-.024.22-.025.216-.027.212-.029.21-.03.205-.032.202-.033.198-.035.194-.036.191-.037.187-.039.183-.039.179-.041.175-.042.172-.043.168-.044.163-.045.16-.045.155-.047.152-.047.148-.048.143-.048.139-.05.136-.049.131-.05.126-.051.123-.051.118-.051.114-.052.11-.052.106-.052.101-.052.096-.052.092-.052.088-.052.083-.052.079-.052.074-.051.07-.052.065-.051.06-.05.056-.051.051-.049.023-.025.023-.024.021-.025.02-.024.019-.024.018-.024.017-.024.015-.023.014-.023.013-.024.012-.022.01-.023.01-.023.008-.022.006-.022.006-.022.004-.021.004-.022.001-.021.001-.021v-4.139l-.077.054-.08.054-.083.054-.085.052-.087.053-.09.051-.093.051-.095.051-.097.05-.1.049-.102.049-.105.048-.106.047-.109.047-.111.046-.114.045-.115.044-.118.044-.12.044-.122.042-.124.042-.126.041-.128.04-.13.039-.132.039-.134.038-.135.037-.138.036-.139.036-.142.035-.143.033-.144.033-.147.033-.148.031-.15.03-.151.03-.153.028-.154.028-.156.027-.158.026-.159.025-.161.024-.162.023-.163.022-.165.021-.166.02-.167.019-.169.018-.169.017-.171.016-.173.015-.173.014-.175.013-.175.012-.177.011-.178.009-.179.009-.179.007-.181.007-.182.005-.182.004-.184.003-.184.002h-.37l-.184-.002-.184-.003-.182-.004-.182-.005-.181-.007-.179-.007-.179-.009-.178-.009-.176-.011-.176-.012-.175-.013-.173-.014-.172-.015-.171-.016-.17-.017-.169-.018-.167-.019-.166-.02-.165-.021-.163-.022-.162-.023-.161-.024-.159-.025-.157-.026-.156-.027-.155-.028-.153-.028-.151-.03-.15-.03-.148-.031-.146-.033-.145-.033-.143-.033-.141-.035-.14-.036-.137-.036-.136-.037-.134-.038-.132-.039-.13-.039-.128-.04-.126-.041-.124-.042-.122-.043-.12-.043-.117-.044-.116-.044-.113-.046-.112-.046-.109-.046-.106-.047-.105-.048-.102-.049-.1-.049-.097-.05-.095-.051-.093-.051-.09-.051-.087-.053-.085-.052-.083-.054-.08-.054-.077-.054v4.139zm0-5.666v.011l.001.02.003.022.004.021.005.022.006.021.007.022.009.023.01.022.011.023.012.023.013.023.015.023.016.024.017.024.018.023.019.024.021.025.022.024.023.024.024.025.052.05.056.05.061.05.066.051.07.051.075.052.079.051.084.052.088.052.092.052.097.052.102.052.105.051.11.052.114.051.119.051.123.051.127.05.131.05.135.05.139.049.144.048.147.048.152.047.155.046.16.045.163.045.167.043.171.043.176.042.178.04.183.04.187.038.19.037.194.036.197.034.202.033.204.032.209.03.212.028.216.027.219.025.222.024.226.021.23.02.233.018.236.017.24.014.243.012.246.01.249.008.253.006.256.003.259.001.26-.001.257-.003.254-.006.25-.008.247-.01.244-.013.241-.014.237-.016.233-.018.231-.02.226-.022.224-.024.22-.025.216-.027.212-.029.21-.03.205-.032.202-.033.198-.035.194-.036.191-.037.187-.039.183-.039.179-.041.175-.042.172-.043.168-.044.163-.045.16-.045.155-.047.152-.047.148-.048.143-.049.139-.049.136-.049.131-.051.126-.05.123-.051.118-.052.114-.051.11-.052.106-.052.101-.052.096-.052.092-.052.088-.052.083-.052.079-.052.074-.052.07-.051.065-.051.06-.051.056-.05.051-.049.023-.025.023-.025.021-.024.02-.024.019-.024.018-.024.017-.024.015-.023.014-.024.013-.023.012-.023.01-.022.01-.023.008-.022.006-.022.006-.022.004-.022.004-.021.001-.021.001-.021v-4.153l-.077.054-.08.054-.083.053-.085.053-.087.053-.09.051-.093.051-.095.051-.097.05-.1.049-.102.048-.105.048-.106.048-.109.046-.111.046-.114.046-.115.044-.118.044-.12.043-.122.043-.124.042-.126.041-.128.04-.13.039-.132.039-.134.038-.135.037-.138.036-.139.036-.142.034-.143.034-.144.033-.147.032-.148.032-.15.03-.151.03-.153.028-.154.028-.156.027-.158.026-.159.024-.161.024-.162.023-.163.023-.165.021-.166.02-.167.019-.169.018-.169.017-.171.016-.173.015-.173.014-.175.013-.175.012-.177.01-.178.01-.179.009-.179.007-.181.006-.182.006-.182.004-.184.003-.184.001-.185.001-.185-.001-.184-.001-.184-.003-.182-.004-.182-.006-.181-.006-.179-.007-.179-.009-.178-.01-.176-.01-.176-.012-.175-.013-.173-.014-.172-.015-.171-.016-.17-.017-.169-.018-.167-.019-.166-.02-.165-.021-.163-.023-.162-.023-.161-.024-.159-.024-.157-.026-.156-.027-.155-.028-.153-.028-.151-.03-.15-.03-.148-.032-.146-.032-.145-.033-.143-.034-.141-.034-.14-.036-.137-.036-.136-.037-.134-.038-.132-.039-.13-.039-.128-.041-.126-.041-.124-.041-.122-.043-.12-.043-.117-.044-.116-.044-.113-.046-.112-.046-.109-.046-.106-.048-.105-.048-.102-.048-.1-.05-.097-.049-.095-.051-.093-.051-.09-.052-.087-.052-.085-.053-.083-.053-.08-.054-.077-.054v4.153zm8.74-8.179l-.257.004-.254.005-.25.008-.247.011-.244.012-.241.014-.237.016-.233.018-.231.021-.226.022-.224.023-.22.026-.216.027-.212.028-.21.031-.205.032-.202.033-.198.034-.194.036-.191.038-.187.038-.183.04-.179.041-.175.042-.172.043-.168.043-.163.045-.16.046-.155.046-.152.048-.148.048-.143.048-.139.049-.136.05-.131.05-.126.051-.123.051-.118.051-.114.052-.11.052-.106.052-.101.052-.096.052-.092.052-.088.052-.083.052-.079.052-.074.051-.07.052-.065.051-.06.05-.056.05-.051.05-.023.025-.023.024-.021.024-.02.025-.019.024-.018.024-.017.023-.015.024-.014.023-.013.023-.012.023-.01.023-.01.022-.008.022-.006.023-.006.021-.004.022-.004.021-.001.021-.001.021.001.021.001.021.004.021.004.022.006.021.006.023.008.022.01.022.01.023.012.023.013.023.014.023.015.024.017.023.018.024.019.024.02.025.021.024.023.024.023.025.051.05.056.05.06.05.065.051.07.052.074.051.079.052.083.052.088.052.092.052.096.052.101.052.106.052.11.052.114.052.118.051.123.051.126.051.131.05.136.05.139.049.143.048.148.048.152.048.155.046.16.046.163.045.168.043.172.043.175.042.179.041.183.04.187.038.191.038.194.036.198.034.202.033.205.032.21.031.212.028.216.027.22.026.224.023.226.022.231.021.233.018.237.016.241.014.244.012.247.011.25.008.254.005.257.004.26.001.26-.001.257-.004.254-.005.25-.008.247-.011.244-.012.241-.014.237-.016.233-.018.231-.021.226-.022.224-.023.22-.026.216-.027.212-.028.21-.031.205-.032.202-.033.198-.034.194-.036.191-.038.187-.038.183-.04.179-.041.175-.042.172-.043.168-.043.163-.045.16-.046.155-.046.152-.048.148-.048.143-.048.139-.049.136-.05.131-.05.126-.051.123-.051.118-.051.114-.052.11-.052.106-.052.101-.052.096-.052.092-.052.088-.052.083-.052.079-.052.074-.051.07-.052.065-.051.06-.05.056-.05.051-.05.023-.025.023-.024.021-.024.02-.025.019-.024.018-.024.017-.023.015-.024.014-.023.013-.023.012-.023.01-.023.01-.022.008-.022.006-.023.006-.021.004-.022.004-.021.001-.021.001-.021-.001-.021-.001-.021-.004-.021-.004-.022-.006-.021-.006-.023-.008-.022-.01-.022-.01-.023-.012-.023-.013-.023-.014-.023-.015-.024-.017-.023-.018-.024-.019-.024-.02-.025-.021-.024-.023-.024-.023-.025-.051-.05-.056-.05-.06-.05-.065-.051-.07-.052-.074-.051-.079-.052-.083-.052-.088-.052-.092-.052-.096-.052-.101-.052-.106-.052-.11-.052-.114-.052-.118-.051-.123-.051-.126-.051-.131-.05-.136-.05-.139-.049-.143-.048-.148-.048-.152-.048-.155-.046-.16-.046-.163-.045-.168-.043-.172-.043-.175-.042-.179-.041-.183-.04-.187-.038-.191-.038-.194-.036-.198-.034-.202-.033-.205-.032-.21-.031-.212-.028-.216-.027-.22-.026-.224-.023-.226-.022-.231-.021-.233-.018-.237-.016-.241-.014-.244-.012-.247-.011-.25-.008-.254-.005-.257-.004-.26-.001-.26.001z")},"insertDatabaseIcon"),xVe=o(function(t){t.append("defs").append("symbol").attr("id","computer").attr("width","24").attr("height","24").append("path").attr("transform","scale(.5)").attr("d","M2 2v13h20v-13h-20zm18 11h-16v-9h16v9zm-10.228 6l.466-1h3.524l.467 1h-4.457zm14.228 3h-24l2-6h2.104l-1.33 4h18.45l-1.297-4h2.073l2 6zm-5-10h-14v-7h14v7z")},"insertComputerIcon"),bVe=o(function(t){t.append("defs").append("symbol").attr("id","clock").attr("width","24").attr("height","24").append("path").attr("transform","scale(.5)").attr("d","M12 2c5.514 0 10 4.486 10 10s-4.486 10-10 10-10-4.486-10-10 4.486-10 10-10zm0-2c-6.627 0-12 5.373-12 12s5.373 12 12 12 12-5.373 12-12-5.373-12-12-12zm5.848 12.459c.202.038.202.333.001.372-1.907.361-6.045 1.111-6.547 1.111-.719 0-1.301-.582-1.301-1.301 0-.512.77-5.447 1.125-7.445.034-.192.312-.181.343.014l.985 6.238 5.394 1.011z")},"insertClockIcon"),wVe=o(function(t){t.append("defs").append("marker").attr("id","arrowhead").attr("refX",7.9).attr("refY",5).attr("markerUnits","userSpaceOnUse").attr("markerWidth",12).attr("markerHeight",12).attr("orient","auto-start-reverse").append("path").attr("d","M -1 0 L 10 5 L 0 10 z")},"insertArrowHead"),TVe=o(function(t){t.append("defs").append("marker").attr("id","filled-head").attr("refX",15.5).attr("refY",7).attr("markerWidth",20).attr("markerHeight",28).attr("orient","auto").append("path").attr("d","M 18,7 L9,13 L14,7 L9,1 Z")},"insertArrowFilledHead"),kVe=o(function(t){t.append("defs").append("marker").attr("id","sequencenumber").attr("refX",15).attr("refY",15).attr("markerWidth",60).attr("markerHeight",40).attr("orient","auto").append("circle").attr("cx",15).attr("cy",15).attr("r",6)},"insertSequenceNumber"),EVe=o(function(t){t.append("defs").append("marker").attr("id","crosshead").attr("markerWidth",15).attr("markerHeight",8).attr("orient","auto").attr("refX",4).attr("refY",4.5).append("path").attr("fill","none").attr("stroke","#000000").style("stroke-dasharray","0, 0").attr("stroke-width","1pt").attr("d","M 1,2 L 6,7 M 6,2 L 1,7")},"insertArrowCrossHead"),vfe=o(function(){return{x:0,y:0,fill:void 0,anchor:void 0,style:"#666",width:void 0,height:void 0,textMargin:0,rx:0,ry:0,tspan:!0,valign:void 0}},"getTextObj"),SVe=o(function(){return{x:0,y:0,fill:"#EDF2AE",stroke:"#666",width:100,anchor:"start",height:100,rx:0,ry:0}},"getNoteRect"),LO=function(){function t(a,s,l,u,h,f,d){let p=s.append("text").attr("x",l+h/2).attr("y",u+f/2+5).style("text-anchor","middle").text(a);i(p,d)}o(t,"byText");function e(a,s,l,u,h,f,d,p){let{actorFontSize:m,actorFontFamily:g,actorFontWeight:y}=p,[v,x]=Bo(m),b=a.split(Ze.lineBreakRegex);for(let w=0;w{let s=Dp(Ne),l=a.actorKeys.reduce((f,d)=>f+=t.get(d).width+(t.get(d).margin||0),0);l-=2*Ne.boxTextMargin,a.wrap&&(a.name=Gt.wrapLabel(a.name,l-2*Ne.wrapPadding,s));let u=Gt.calculateTextDimensions(a.name,s);i=Ze.getMax(u.height,i);let h=Ze.getMax(l,u.width+2*Ne.wrapPadding);if(a.margin=Ne.boxTextMargin,la.textMaxHeight=i),Ze.getMax(n,Ne.height)}var Ne,rt,AVe,Dp,_1,RO,DVe,LVe,NO,wfe,Tfe,D6,bfe,NVe,IVe,PVe,BVe,FVe,kfe,Efe=N(()=>{"use strict";dr();xfe();vt();gr();Wv();zt();s0();ir();Ei();Ne={},rt={data:{startx:void 0,stopx:void 0,starty:void 0,stopy:void 0},verticalPos:0,sequenceItems:[],activations:[],models:{getHeight:o(function(){return Math.max.apply(null,this.actors.length===0?[0]:this.actors.map(t=>t.height||0))+(this.loops.length===0?0:this.loops.map(t=>t.height||0).reduce((t,e)=>t+e))+(this.messages.length===0?0:this.messages.map(t=>t.height||0).reduce((t,e)=>t+e))+(this.notes.length===0?0:this.notes.map(t=>t.height||0).reduce((t,e)=>t+e))},"getHeight"),clear:o(function(){this.actors=[],this.boxes=[],this.loops=[],this.messages=[],this.notes=[]},"clear"),addBox:o(function(t){this.boxes.push(t)},"addBox"),addActor:o(function(t){this.actors.push(t)},"addActor"),addLoop:o(function(t){this.loops.push(t)},"addLoop"),addMessage:o(function(t){this.messages.push(t)},"addMessage"),addNote:o(function(t){this.notes.push(t)},"addNote"),lastActor:o(function(){return this.actors[this.actors.length-1]},"lastActor"),lastLoop:o(function(){return this.loops[this.loops.length-1]},"lastLoop"),lastMessage:o(function(){return this.messages[this.messages.length-1]},"lastMessage"),lastNote:o(function(){return this.notes[this.notes.length-1]},"lastNote"),actors:[],boxes:[],loops:[],messages:[],notes:[]},init:o(function(){this.sequenceItems=[],this.activations=[],this.models.clear(),this.data={startx:void 0,stopx:void 0,starty:void 0,stopy:void 0},this.verticalPos=0,Tfe(me())},"init"),updateVal:o(function(t,e,r,n){t[e]===void 0?t[e]=r:t[e]=n(r,t[e])},"updateVal"),updateBounds:o(function(t,e,r,n){let i=this,a=0;function s(l){return o(function(h){a++;let f=i.sequenceItems.length-a+1;i.updateVal(h,"starty",e-f*Ne.boxMargin,Math.min),i.updateVal(h,"stopy",n+f*Ne.boxMargin,Math.max),i.updateVal(rt.data,"startx",t-f*Ne.boxMargin,Math.min),i.updateVal(rt.data,"stopx",r+f*Ne.boxMargin,Math.max),l!=="activation"&&(i.updateVal(h,"startx",t-f*Ne.boxMargin,Math.min),i.updateVal(h,"stopx",r+f*Ne.boxMargin,Math.max),i.updateVal(rt.data,"starty",e-f*Ne.boxMargin,Math.min),i.updateVal(rt.data,"stopy",n+f*Ne.boxMargin,Math.max))},"updateItemBounds")}o(s,"updateFn"),this.sequenceItems.forEach(s()),this.activations.forEach(s("activation"))},"updateBounds"),insert:o(function(t,e,r,n){let i=Ze.getMin(t,r),a=Ze.getMax(t,r),s=Ze.getMin(e,n),l=Ze.getMax(e,n);this.updateVal(rt.data,"startx",i,Math.min),this.updateVal(rt.data,"starty",s,Math.min),this.updateVal(rt.data,"stopx",a,Math.max),this.updateVal(rt.data,"stopy",l,Math.max),this.updateBounds(i,s,a,l)},"insert"),newActivation:o(function(t,e,r){let n=r.get(t.from),i=D6(t.from).length||0,a=n.x+n.width/2+(i-1)*Ne.activationWidth/2;this.activations.push({startx:a,starty:this.verticalPos+2,stopx:a+Ne.activationWidth,stopy:void 0,actor:t.from,anchored:hi.anchorElement(e)})},"newActivation"),endActivation:o(function(t){let e=this.activations.map(function(r){return r.actor}).lastIndexOf(t.from);return this.activations.splice(e,1)[0]},"endActivation"),createLoop:o(function(t={message:void 0,wrap:!1,width:void 0},e){return{startx:void 0,starty:this.verticalPos,stopx:void 0,stopy:void 0,title:t.message,wrap:t.wrap,width:t.width,height:0,fill:e}},"createLoop"),newLoop:o(function(t={message:void 0,wrap:!1,width:void 0},e){this.sequenceItems.push(this.createLoop(t,e))},"newLoop"),endLoop:o(function(){return this.sequenceItems.pop()},"endLoop"),isLoopOverlap:o(function(){return this.sequenceItems.length?this.sequenceItems[this.sequenceItems.length-1].overlap:!1},"isLoopOverlap"),addSectionToLoop:o(function(t){let e=this.sequenceItems.pop();e.sections=e.sections||[],e.sectionTitles=e.sectionTitles||[],e.sections.push({y:rt.getVerticalPos(),height:0}),e.sectionTitles.push(t),this.sequenceItems.push(e)},"addSectionToLoop"),saveVerticalPos:o(function(){this.isLoopOverlap()&&(this.savedVerticalPos=this.verticalPos)},"saveVerticalPos"),resetVerticalPos:o(function(){this.isLoopOverlap()&&(this.verticalPos=this.savedVerticalPos)},"resetVerticalPos"),bumpVerticalPos:o(function(t){this.verticalPos=this.verticalPos+t,this.data.stopy=Ze.getMax(this.data.stopy,this.verticalPos)},"bumpVerticalPos"),getVerticalPos:o(function(){return this.verticalPos},"getVerticalPos"),getBounds:o(function(){return{bounds:this.data,models:this.models}},"getBounds")},AVe=o(async function(t,e){rt.bumpVerticalPos(Ne.boxMargin),e.height=Ne.boxMargin,e.starty=rt.getVerticalPos();let r=Tl();r.x=e.startx,r.y=e.starty,r.width=e.width||Ne.width,r.class="note";let n=t.append("g"),i=hi.drawRect(n,r),a=Hv();a.x=e.startx,a.y=e.starty,a.width=r.width,a.dy="1em",a.text=e.message,a.class="noteText",a.fontFamily=Ne.noteFontFamily,a.fontSize=Ne.noteFontSize,a.fontWeight=Ne.noteFontWeight,a.anchor=Ne.noteAlign,a.textMargin=Ne.noteMargin,a.valign="center";let s=pi(a.text)?await Tb(n,a):_p(n,a),l=Math.round(s.map(u=>(u._groups||u)[0][0].getBBox().height).reduce((u,h)=>u+h));i.attr("height",l+2*Ne.noteMargin),e.height+=l+2*Ne.noteMargin,rt.bumpVerticalPos(l+2*Ne.noteMargin),e.stopy=e.starty+l+2*Ne.noteMargin,e.stopx=e.startx+r.width,rt.insert(e.startx,e.starty,e.stopx,e.stopy),rt.models.addNote(e)},"drawNote"),Dp=o(t=>({fontFamily:t.messageFontFamily,fontSize:t.messageFontSize,fontWeight:t.messageFontWeight}),"messageFont"),_1=o(t=>({fontFamily:t.noteFontFamily,fontSize:t.noteFontSize,fontWeight:t.noteFontWeight}),"noteFont"),RO=o(t=>({fontFamily:t.actorFontFamily,fontSize:t.actorFontSize,fontWeight:t.actorFontWeight}),"actorFont");o(_Ve,"boundMessage");DVe=o(async function(t,e,r,n){let{startx:i,stopx:a,starty:s,message:l,type:u,sequenceIndex:h,sequenceVisible:f}=e,d=Gt.calculateTextDimensions(l,Dp(Ne)),p=Hv();p.x=i,p.y=s+10,p.width=a-i,p.class="messageText",p.dy="1em",p.text=l,p.fontFamily=Ne.messageFontFamily,p.fontSize=Ne.messageFontSize,p.fontWeight=Ne.messageFontWeight,p.anchor=Ne.messageAlign,p.valign="center",p.textMargin=Ne.wrapPadding,p.tspan=!1,pi(p.text)?await Tb(t,p,{startx:i,stopx:a,starty:r}):_p(t,p);let m=d.width,g;i===a?Ne.rightAngles?g=t.append("path").attr("d",`M ${i},${r} H ${i+Ze.getMax(Ne.width/2,m/2)} V ${r+25} H ${i}`):g=t.append("path").attr("d","M "+i+","+r+" C "+(i+60)+","+(r-10)+" "+(i+60)+","+(r+30)+" "+i+","+(r+20)):(g=t.append("line"),g.attr("x1",i),g.attr("y1",r),g.attr("x2",a),g.attr("y2",r)),u===n.db.LINETYPE.DOTTED||u===n.db.LINETYPE.DOTTED_CROSS||u===n.db.LINETYPE.DOTTED_POINT||u===n.db.LINETYPE.DOTTED_OPEN||u===n.db.LINETYPE.BIDIRECTIONAL_DOTTED?(g.style("stroke-dasharray","3, 3"),g.attr("class","messageLine1")):g.attr("class","messageLine0");let y="";Ne.arrowMarkerAbsolute&&(y=window.location.protocol+"//"+window.location.host+window.location.pathname+window.location.search,y=y.replace(/\(/g,"\\("),y=y.replace(/\)/g,"\\)")),g.attr("stroke-width",2),g.attr("stroke","none"),g.style("fill","none"),(u===n.db.LINETYPE.SOLID||u===n.db.LINETYPE.DOTTED)&&g.attr("marker-end","url("+y+"#arrowhead)"),(u===n.db.LINETYPE.BIDIRECTIONAL_SOLID||u===n.db.LINETYPE.BIDIRECTIONAL_DOTTED)&&(g.attr("marker-start","url("+y+"#arrowhead)"),g.attr("marker-end","url("+y+"#arrowhead)")),(u===n.db.LINETYPE.SOLID_POINT||u===n.db.LINETYPE.DOTTED_POINT)&&g.attr("marker-end","url("+y+"#filled-head)"),(u===n.db.LINETYPE.SOLID_CROSS||u===n.db.LINETYPE.DOTTED_CROSS)&&g.attr("marker-end","url("+y+"#crosshead)"),(f||Ne.showSequenceNumbers)&&(g.attr("marker-start","url("+y+"#sequencenumber)"),t.append("text").attr("x",i).attr("y",r+4).attr("font-family","sans-serif").attr("font-size","12px").attr("text-anchor","middle").attr("class","sequenceNumber").text(h))},"drawMessage"),LVe=o(function(t,e,r,n,i,a,s){let l=0,u=0,h,f=0;for(let d of n){let p=e.get(d),m=p.box;h&&h!=m&&(s||rt.models.addBox(h),u+=Ne.boxMargin+h.margin),m&&m!=h&&(s||(m.x=l+u,m.y=i),u+=m.margin),p.width=p.width||Ne.width,p.height=Ze.getMax(p.height||Ne.height,Ne.height),p.margin=p.margin||Ne.actorMargin,f=Ze.getMax(f,p.height),r.get(p.name)&&(u+=p.width/2),p.x=l+u,p.starty=rt.getVerticalPos(),rt.insert(p.x,i,p.x+p.width,p.height),l+=p.width+u,p.box&&(p.box.width=l+m.margin-p.box.x),u=p.margin,h=p.box,rt.models.addActor(p)}h&&!s&&rt.models.addBox(h),rt.bumpVerticalPos(f)},"addActorRenderingData"),NO=o(async function(t,e,r,n){if(n){let i=0;rt.bumpVerticalPos(Ne.boxMargin*2);for(let a of r){let s=e.get(a);s.stopy||(s.stopy=rt.getVerticalPos());let l=await hi.drawActor(t,s,Ne,!0);i=Ze.getMax(i,l)}rt.bumpVerticalPos(i+Ne.boxMargin)}else for(let i of r){let a=e.get(i);await hi.drawActor(t,a,Ne,!1)}},"drawActors"),wfe=o(function(t,e,r,n){let i=0,a=0;for(let s of r){let l=e.get(s),u=IVe(l),h=hi.drawPopup(t,l,u,Ne,Ne.forceMenus,n);h.height>i&&(i=h.height),h.width+l.x>a&&(a=h.width+l.x)}return{maxHeight:i,maxWidth:a}},"drawActorsPopup"),Tfe=o(function(t){Gn(Ne,t),t.fontFamily&&(Ne.actorFontFamily=Ne.noteFontFamily=Ne.messageFontFamily=t.fontFamily),t.fontSize&&(Ne.actorFontSize=Ne.noteFontSize=Ne.messageFontSize=t.fontSize),t.fontWeight&&(Ne.actorFontWeight=Ne.noteFontWeight=Ne.messageFontWeight=t.fontWeight)},"setConf"),D6=o(function(t){return rt.activations.filter(function(e){return e.actor===t})},"actorActivations"),bfe=o(function(t,e){let r=e.get(t),n=D6(t),i=n.reduce(function(s,l){return Ze.getMin(s,l.startx)},r.x+r.width/2-1),a=n.reduce(function(s,l){return Ze.getMax(s,l.stopx)},r.x+r.width/2+1);return[i,a]},"activationBounds");o(Hc,"adjustLoopHeightForWrap");o(RVe,"adjustCreatedDestroyedData");NVe=o(async function(t,e,r,n){let{securityLevel:i,sequence:a}=me();Ne=a;let s;i==="sandbox"&&(s=Ge("#i"+e));let l=i==="sandbox"?Ge(s.nodes()[0].contentDocument.body):Ge("body"),u=i==="sandbox"?s.nodes()[0].contentDocument:document;rt.init(),Y.debug(n.db);let h=i==="sandbox"?l.select(`[id="${e}"]`):Ge(`[id="${e}"]`),f=n.db.getActors(),d=n.db.getCreatedActors(),p=n.db.getDestroyedActors(),m=n.db.getBoxes(),g=n.db.getActorKeys(),y=n.db.getMessages(),v=n.db.getDiagramTitle(),x=n.db.hasAtLeastOneBox(),b=n.db.hasAtLeastOneBoxWithTitle(),w=await MVe(f,y,n);if(Ne.height=await OVe(f,w,m),hi.insertComputerIcon(h),hi.insertDatabaseIcon(h),hi.insertClockIcon(h),x&&(rt.bumpVerticalPos(Ne.boxMargin),b&&rt.bumpVerticalPos(m[0].textMaxHeight)),Ne.hideUnusedParticipants===!0){let F=new Set;y.forEach(P=>{F.add(P.from),F.add(P.to)}),g=g.filter(P=>F.has(P))}LVe(h,f,d,g,0,y,!1);let C=await FVe(y,f,w,n);hi.insertArrowHead(h),hi.insertArrowCrossHead(h),hi.insertArrowFilledHead(h),hi.insertSequenceNumber(h);function T(F,P){let z=rt.endActivation(F);z.starty+18>P&&(z.starty=P-6,P+=12),hi.drawActivation(h,z,P,Ne,D6(F.from).length),rt.insert(z.startx,P-10,z.stopx,P)}o(T,"activeEnd");let E=1,A=1,S=[],_=[],I=0;for(let F of y){let P,z,$;switch(F.type){case n.db.LINETYPE.NOTE:rt.resetVerticalPos(),z=F.noteModel,await AVe(h,z);break;case n.db.LINETYPE.ACTIVE_START:rt.newActivation(F,h,f);break;case n.db.LINETYPE.ACTIVE_END:T(F,rt.getVerticalPos());break;case n.db.LINETYPE.LOOP_START:Hc(C,F,Ne.boxMargin,Ne.boxMargin+Ne.boxTextMargin,H=>rt.newLoop(H));break;case n.db.LINETYPE.LOOP_END:P=rt.endLoop(),await hi.drawLoop(h,P,"loop",Ne),rt.bumpVerticalPos(P.stopy-rt.getVerticalPos()),rt.models.addLoop(P);break;case n.db.LINETYPE.RECT_START:Hc(C,F,Ne.boxMargin,Ne.boxMargin,H=>rt.newLoop(void 0,H.message));break;case n.db.LINETYPE.RECT_END:P=rt.endLoop(),_.push(P),rt.models.addLoop(P),rt.bumpVerticalPos(P.stopy-rt.getVerticalPos());break;case n.db.LINETYPE.OPT_START:Hc(C,F,Ne.boxMargin,Ne.boxMargin+Ne.boxTextMargin,H=>rt.newLoop(H));break;case n.db.LINETYPE.OPT_END:P=rt.endLoop(),await hi.drawLoop(h,P,"opt",Ne),rt.bumpVerticalPos(P.stopy-rt.getVerticalPos()),rt.models.addLoop(P);break;case n.db.LINETYPE.ALT_START:Hc(C,F,Ne.boxMargin,Ne.boxMargin+Ne.boxTextMargin,H=>rt.newLoop(H));break;case n.db.LINETYPE.ALT_ELSE:Hc(C,F,Ne.boxMargin+Ne.boxTextMargin,Ne.boxMargin,H=>rt.addSectionToLoop(H));break;case n.db.LINETYPE.ALT_END:P=rt.endLoop(),await hi.drawLoop(h,P,"alt",Ne),rt.bumpVerticalPos(P.stopy-rt.getVerticalPos()),rt.models.addLoop(P);break;case n.db.LINETYPE.PAR_START:case n.db.LINETYPE.PAR_OVER_START:Hc(C,F,Ne.boxMargin,Ne.boxMargin+Ne.boxTextMargin,H=>rt.newLoop(H)),rt.saveVerticalPos();break;case n.db.LINETYPE.PAR_AND:Hc(C,F,Ne.boxMargin+Ne.boxTextMargin,Ne.boxMargin,H=>rt.addSectionToLoop(H));break;case n.db.LINETYPE.PAR_END:P=rt.endLoop(),await hi.drawLoop(h,P,"par",Ne),rt.bumpVerticalPos(P.stopy-rt.getVerticalPos()),rt.models.addLoop(P);break;case n.db.LINETYPE.AUTONUMBER:E=F.message.start||E,A=F.message.step||A,F.message.visible?n.db.enableSequenceNumbers():n.db.disableSequenceNumbers();break;case n.db.LINETYPE.CRITICAL_START:Hc(C,F,Ne.boxMargin,Ne.boxMargin+Ne.boxTextMargin,H=>rt.newLoop(H));break;case n.db.LINETYPE.CRITICAL_OPTION:Hc(C,F,Ne.boxMargin+Ne.boxTextMargin,Ne.boxMargin,H=>rt.addSectionToLoop(H));break;case n.db.LINETYPE.CRITICAL_END:P=rt.endLoop(),await hi.drawLoop(h,P,"critical",Ne),rt.bumpVerticalPos(P.stopy-rt.getVerticalPos()),rt.models.addLoop(P);break;case n.db.LINETYPE.BREAK_START:Hc(C,F,Ne.boxMargin,Ne.boxMargin+Ne.boxTextMargin,H=>rt.newLoop(H));break;case n.db.LINETYPE.BREAK_END:P=rt.endLoop(),await hi.drawLoop(h,P,"break",Ne),rt.bumpVerticalPos(P.stopy-rt.getVerticalPos()),rt.models.addLoop(P);break;default:try{$=F.msgModel,$.starty=rt.getVerticalPos(),$.sequenceIndex=E,$.sequenceVisible=n.db.showSequenceNumbers();let H=await _Ve(h,$);RVe(F,$,H,I,f,d,p),S.push({messageModel:$,lineStartY:H}),rt.models.addMessage($)}catch(H){Y.error("error while drawing message",H)}}[n.db.LINETYPE.SOLID_OPEN,n.db.LINETYPE.DOTTED_OPEN,n.db.LINETYPE.SOLID,n.db.LINETYPE.DOTTED,n.db.LINETYPE.SOLID_CROSS,n.db.LINETYPE.DOTTED_CROSS,n.db.LINETYPE.SOLID_POINT,n.db.LINETYPE.DOTTED_POINT,n.db.LINETYPE.BIDIRECTIONAL_SOLID,n.db.LINETYPE.BIDIRECTIONAL_DOTTED].includes(F.type)&&(E=E+A),I++}Y.debug("createdActors",d),Y.debug("destroyedActors",p),await NO(h,f,g,!1);for(let F of S)await DVe(h,F.messageModel,F.lineStartY,n);Ne.mirrorActors&&await NO(h,f,g,!0),_.forEach(F=>hi.drawBackgroundRect(h,F)),DO(h,f,g,Ne);for(let F of rt.models.boxes)F.height=rt.getVerticalPos()-F.y,rt.insert(F.x,F.y,F.x+F.width,F.height),F.startx=F.x,F.starty=F.y,F.stopx=F.startx+F.width,F.stopy=F.starty+F.height,F.stroke="rgb(0,0,0, 0.5)",hi.drawBox(h,F,Ne);x&&rt.bumpVerticalPos(Ne.boxMargin);let D=wfe(h,f,g,u),{bounds:k}=rt.getBounds();k.startx===void 0&&(k.startx=0),k.starty===void 0&&(k.starty=0),k.stopx===void 0&&(k.stopx=0),k.stopy===void 0&&(k.stopy=0);let L=k.stopy-k.starty;L2,d=o(y=>l?-y:y,"adjustValue");t.from===t.to?h=u:(t.activate&&!f&&(h+=d(Ne.activationWidth/2-1)),[r.db.LINETYPE.SOLID_OPEN,r.db.LINETYPE.DOTTED_OPEN].includes(t.type)||(h+=d(3)),[r.db.LINETYPE.BIDIRECTIONAL_SOLID,r.db.LINETYPE.BIDIRECTIONAL_DOTTED].includes(t.type)&&(u-=d(3)));let p=[n,i,a,s],m=Math.abs(u-h);t.wrap&&t.message&&(t.message=Gt.wrapLabel(t.message,Ze.getMax(m+2*Ne.wrapPadding,Ne.width),Dp(Ne)));let g=Gt.calculateTextDimensions(t.message,Dp(Ne));return{width:Ze.getMax(t.wrap?0:g.width+2*Ne.wrapPadding,m+2*Ne.wrapPadding,Ne.width),height:0,startx:u,stopx:h,starty:0,stopy:0,message:t.message,type:t.type,wrap:t.wrap,fromBounds:Math.min.apply(null,p),toBounds:Math.max.apply(null,p)}},"buildMessageModel"),FVe=o(async function(t,e,r,n){let i={},a=[],s,l,u;for(let h of t){switch(h.type){case n.db.LINETYPE.LOOP_START:case n.db.LINETYPE.ALT_START:case n.db.LINETYPE.OPT_START:case n.db.LINETYPE.PAR_START:case n.db.LINETYPE.PAR_OVER_START:case n.db.LINETYPE.CRITICAL_START:case n.db.LINETYPE.BREAK_START:a.push({id:h.id,msg:h.message,from:Number.MAX_SAFE_INTEGER,to:Number.MIN_SAFE_INTEGER,width:0});break;case n.db.LINETYPE.ALT_ELSE:case n.db.LINETYPE.PAR_AND:case n.db.LINETYPE.CRITICAL_OPTION:h.message&&(s=a.pop(),i[s.id]=s,i[h.id]=s,a.push(s));break;case n.db.LINETYPE.LOOP_END:case n.db.LINETYPE.ALT_END:case n.db.LINETYPE.OPT_END:case n.db.LINETYPE.PAR_END:case n.db.LINETYPE.CRITICAL_END:case n.db.LINETYPE.BREAK_END:s=a.pop(),i[s.id]=s;break;case n.db.LINETYPE.ACTIVE_START:{let d=e.get(h.from?h.from:h.to.actor),p=D6(h.from?h.from:h.to.actor).length,m=d.x+d.width/2+(p-1)*Ne.activationWidth/2,g={startx:m,stopx:m+Ne.activationWidth,actor:h.from,enabled:!0};rt.activations.push(g)}break;case n.db.LINETYPE.ACTIVE_END:{let d=rt.activations.map(p=>p.actor).lastIndexOf(h.from);rt.activations.splice(d,1).splice(0,1)}break}h.placement!==void 0?(l=await PVe(h,e,n),h.noteModel=l,a.forEach(d=>{s=d,s.from=Ze.getMin(s.from,l.startx),s.to=Ze.getMax(s.to,l.startx+l.width),s.width=Ze.getMax(s.width,Math.abs(s.from-s.to))-Ne.labelBoxWidth})):(u=BVe(h,e,n),h.msgModel=u,u.startx&&u.stopx&&a.length>0&&a.forEach(d=>{if(s=d,u.startx===u.stopx){let p=e.get(h.from),m=e.get(h.to);s.from=Ze.getMin(p.x-u.width/2,p.x-p.width/2,s.from),s.to=Ze.getMax(m.x+u.width/2,m.x+p.width/2,s.to),s.width=Ze.getMax(s.width,Math.abs(s.to-s.from))-Ne.labelBoxWidth}else s.from=Ze.getMin(u.startx,s.from),s.to=Ze.getMax(u.stopx,s.to),s.width=Ze.getMax(s.width,u.width)-Ne.labelBoxWidth}))}return rt.activations=[],Y.debug("Loop type widths:",i),i},"calculateLoopBounds"),kfe={bounds:rt,drawActors:NO,drawActorsPopup:wfe,setConf:Tfe,draw:NVe}});var Sfe={};hr(Sfe,{diagram:()=>$Ve});var $Ve,Cfe=N(()=>{"use strict";cfe();ufe();ffe();zt();Efe();$Ve={parser:lfe,get db(){return new _6},renderer:kfe,styles:hfe,init:o(t=>{t.sequence||(t.sequence={}),t.wrap&&(t.sequence.wrap=t.wrap,Yy({sequence:{wrap:t.wrap}}))},"init")}});var MO,L6,IO=N(()=>{"use strict";MO=function(){var t=o(function(Ie,be,W,de){for(W=W||{},de=Ie.length;de--;W[Ie[de]]=be);return W},"o"),e=[1,18],r=[1,19],n=[1,20],i=[1,41],a=[1,42],s=[1,26],l=[1,24],u=[1,25],h=[1,32],f=[1,33],d=[1,34],p=[1,45],m=[1,35],g=[1,36],y=[1,37],v=[1,38],x=[1,27],b=[1,28],w=[1,29],C=[1,30],T=[1,31],E=[1,44],A=[1,46],S=[1,43],_=[1,47],I=[1,9],D=[1,8,9],k=[1,58],L=[1,59],R=[1,60],O=[1,61],M=[1,62],B=[1,63],F=[1,64],P=[1,8,9,41],z=[1,76],$=[1,8,9,12,13,22,39,41,44,66,67,68,69,70,71,72,77,79],H=[1,8,9,12,13,17,20,22,39,41,44,48,58,66,67,68,69,70,71,72,77,79,84,99,101,102],Q=[13,58,84,99,101,102],j=[13,58,71,72,84,99,101,102],ie=[13,58,66,67,68,69,70,84,99,101,102],ne=[1,98],le=[1,115],he=[1,107],K=[1,113],X=[1,108],te=[1,109],J=[1,110],se=[1,111],ue=[1,112],Z=[1,114],Se=[22,58,59,80,84,85,86,87,88,89],ce=[1,8,9,39,41,44],ae=[1,8,9,22],Oe=[1,143],ge=[1,8,9,59],ze=[1,8,9,22,58,59,80,84,85,86,87,88,89],He={trace:o(function(){},"trace"),yy:{},symbols_:{error:2,start:3,mermaidDoc:4,statements:5,graphConfig:6,CLASS_DIAGRAM:7,NEWLINE:8,EOF:9,statement:10,classLabel:11,SQS:12,STR:13,SQE:14,namespaceName:15,alphaNumToken:16,DOT:17,className:18,classLiteralName:19,GENERICTYPE:20,relationStatement:21,LABEL:22,namespaceStatement:23,classStatement:24,memberStatement:25,annotationStatement:26,clickStatement:27,styleStatement:28,cssClassStatement:29,noteStatement:30,classDefStatement:31,direction:32,acc_title:33,acc_title_value:34,acc_descr:35,acc_descr_value:36,acc_descr_multiline_value:37,namespaceIdentifier:38,STRUCT_START:39,classStatements:40,STRUCT_STOP:41,NAMESPACE:42,classIdentifier:43,STYLE_SEPARATOR:44,members:45,CLASS:46,ANNOTATION_START:47,ANNOTATION_END:48,MEMBER:49,SEPARATOR:50,relation:51,NOTE_FOR:52,noteText:53,NOTE:54,CLASSDEF:55,classList:56,stylesOpt:57,ALPHA:58,COMMA:59,direction_tb:60,direction_bt:61,direction_rl:62,direction_lr:63,relationType:64,lineType:65,AGGREGATION:66,EXTENSION:67,COMPOSITION:68,DEPENDENCY:69,LOLLIPOP:70,LINE:71,DOTTED_LINE:72,CALLBACK:73,LINK:74,LINK_TARGET:75,CLICK:76,CALLBACK_NAME:77,CALLBACK_ARGS:78,HREF:79,STYLE:80,CSSCLASS:81,style:82,styleComponent:83,NUM:84,COLON:85,UNIT:86,SPACE:87,BRKT:88,PCT:89,commentToken:90,textToken:91,graphCodeTokens:92,textNoTagsToken:93,TAGSTART:94,TAGEND:95,"==":96,"--":97,DEFAULT:98,MINUS:99,keywords:100,UNICODE_TEXT:101,BQUOTE_STR:102,$accept:0,$end:1},terminals_:{2:"error",7:"CLASS_DIAGRAM",8:"NEWLINE",9:"EOF",12:"SQS",13:"STR",14:"SQE",17:"DOT",20:"GENERICTYPE",22:"LABEL",33:"acc_title",34:"acc_title_value",35:"acc_descr",36:"acc_descr_value",37:"acc_descr_multiline_value",39:"STRUCT_START",41:"STRUCT_STOP",42:"NAMESPACE",44:"STYLE_SEPARATOR",46:"CLASS",47:"ANNOTATION_START",48:"ANNOTATION_END",49:"MEMBER",50:"SEPARATOR",52:"NOTE_FOR",54:"NOTE",55:"CLASSDEF",58:"ALPHA",59:"COMMA",60:"direction_tb",61:"direction_bt",62:"direction_rl",63:"direction_lr",66:"AGGREGATION",67:"EXTENSION",68:"COMPOSITION",69:"DEPENDENCY",70:"LOLLIPOP",71:"LINE",72:"DOTTED_LINE",73:"CALLBACK",74:"LINK",75:"LINK_TARGET",76:"CLICK",77:"CALLBACK_NAME",78:"CALLBACK_ARGS",79:"HREF",80:"STYLE",81:"CSSCLASS",84:"NUM",85:"COLON",86:"UNIT",87:"SPACE",88:"BRKT",89:"PCT",92:"graphCodeTokens",94:"TAGSTART",95:"TAGEND",96:"==",97:"--",98:"DEFAULT",99:"MINUS",100:"keywords",101:"UNICODE_TEXT",102:"BQUOTE_STR"},productions_:[0,[3,1],[3,1],[4,1],[6,4],[5,1],[5,2],[5,3],[11,3],[15,1],[15,3],[15,2],[18,1],[18,3],[18,1],[18,2],[18,2],[18,2],[10,1],[10,2],[10,1],[10,1],[10,1],[10,1],[10,1],[10,1],[10,1],[10,1],[10,1],[10,1],[10,2],[10,2],[10,1],[23,4],[23,5],[38,2],[40,1],[40,2],[40,3],[24,1],[24,3],[24,4],[24,6],[43,2],[43,3],[26,4],[45,1],[45,2],[25,1],[25,2],[25,1],[25,1],[21,3],[21,4],[21,4],[21,5],[30,3],[30,2],[31,3],[56,1],[56,3],[32,1],[32,1],[32,1],[32,1],[51,3],[51,2],[51,2],[51,1],[64,1],[64,1],[64,1],[64,1],[64,1],[65,1],[65,1],[27,3],[27,4],[27,3],[27,4],[27,4],[27,5],[27,3],[27,4],[27,4],[27,5],[27,4],[27,5],[27,5],[27,6],[28,3],[29,3],[57,1],[57,3],[82,1],[82,2],[83,1],[83,1],[83,1],[83,1],[83,1],[83,1],[83,1],[83,1],[83,1],[90,1],[90,1],[91,1],[91,1],[91,1],[91,1],[91,1],[91,1],[91,1],[93,1],[93,1],[93,1],[93,1],[16,1],[16,1],[16,1],[16,1],[19,1],[53,1]],performAction:o(function(be,W,de,re,oe,V,xe){var q=V.length-1;switch(oe){case 8:this.$=V[q-1];break;case 9:case 12:case 14:this.$=V[q];break;case 10:case 13:this.$=V[q-2]+"."+V[q];break;case 11:case 15:this.$=V[q-1]+V[q];break;case 16:case 17:this.$=V[q-1]+"~"+V[q]+"~";break;case 18:re.addRelation(V[q]);break;case 19:V[q-1].title=re.cleanupLabel(V[q]),re.addRelation(V[q-1]);break;case 30:this.$=V[q].trim(),re.setAccTitle(this.$);break;case 31:case 32:this.$=V[q].trim(),re.setAccDescription(this.$);break;case 33:re.addClassesToNamespace(V[q-3],V[q-1]);break;case 34:re.addClassesToNamespace(V[q-4],V[q-1]);break;case 35:this.$=V[q],re.addNamespace(V[q]);break;case 36:this.$=[V[q]];break;case 37:this.$=[V[q-1]];break;case 38:V[q].unshift(V[q-2]),this.$=V[q];break;case 40:re.setCssClass(V[q-2],V[q]);break;case 41:re.addMembers(V[q-3],V[q-1]);break;case 42:re.setCssClass(V[q-5],V[q-3]),re.addMembers(V[q-5],V[q-1]);break;case 43:this.$=V[q],re.addClass(V[q]);break;case 44:this.$=V[q-1],re.addClass(V[q-1]),re.setClassLabel(V[q-1],V[q]);break;case 45:re.addAnnotation(V[q],V[q-2]);break;case 46:case 59:this.$=[V[q]];break;case 47:V[q].push(V[q-1]),this.$=V[q];break;case 48:break;case 49:re.addMember(V[q-1],re.cleanupLabel(V[q]));break;case 50:break;case 51:break;case 52:this.$={id1:V[q-2],id2:V[q],relation:V[q-1],relationTitle1:"none",relationTitle2:"none"};break;case 53:this.$={id1:V[q-3],id2:V[q],relation:V[q-1],relationTitle1:V[q-2],relationTitle2:"none"};break;case 54:this.$={id1:V[q-3],id2:V[q],relation:V[q-2],relationTitle1:"none",relationTitle2:V[q-1]};break;case 55:this.$={id1:V[q-4],id2:V[q],relation:V[q-2],relationTitle1:V[q-3],relationTitle2:V[q-1]};break;case 56:re.addNote(V[q],V[q-1]);break;case 57:re.addNote(V[q]);break;case 58:this.$=V[q-2],re.defineClass(V[q-1],V[q]);break;case 60:this.$=V[q-2].concat([V[q]]);break;case 61:re.setDirection("TB");break;case 62:re.setDirection("BT");break;case 63:re.setDirection("RL");break;case 64:re.setDirection("LR");break;case 65:this.$={type1:V[q-2],type2:V[q],lineType:V[q-1]};break;case 66:this.$={type1:"none",type2:V[q],lineType:V[q-1]};break;case 67:this.$={type1:V[q-1],type2:"none",lineType:V[q]};break;case 68:this.$={type1:"none",type2:"none",lineType:V[q]};break;case 69:this.$=re.relationType.AGGREGATION;break;case 70:this.$=re.relationType.EXTENSION;break;case 71:this.$=re.relationType.COMPOSITION;break;case 72:this.$=re.relationType.DEPENDENCY;break;case 73:this.$=re.relationType.LOLLIPOP;break;case 74:this.$=re.lineType.LINE;break;case 75:this.$=re.lineType.DOTTED_LINE;break;case 76:case 82:this.$=V[q-2],re.setClickEvent(V[q-1],V[q]);break;case 77:case 83:this.$=V[q-3],re.setClickEvent(V[q-2],V[q-1]),re.setTooltip(V[q-2],V[q]);break;case 78:this.$=V[q-2],re.setLink(V[q-1],V[q]);break;case 79:this.$=V[q-3],re.setLink(V[q-2],V[q-1],V[q]);break;case 80:this.$=V[q-3],re.setLink(V[q-2],V[q-1]),re.setTooltip(V[q-2],V[q]);break;case 81:this.$=V[q-4],re.setLink(V[q-3],V[q-2],V[q]),re.setTooltip(V[q-3],V[q-1]);break;case 84:this.$=V[q-3],re.setClickEvent(V[q-2],V[q-1],V[q]);break;case 85:this.$=V[q-4],re.setClickEvent(V[q-3],V[q-2],V[q-1]),re.setTooltip(V[q-3],V[q]);break;case 86:this.$=V[q-3],re.setLink(V[q-2],V[q]);break;case 87:this.$=V[q-4],re.setLink(V[q-3],V[q-1],V[q]);break;case 88:this.$=V[q-4],re.setLink(V[q-3],V[q-1]),re.setTooltip(V[q-3],V[q]);break;case 89:this.$=V[q-5],re.setLink(V[q-4],V[q-2],V[q]),re.setTooltip(V[q-4],V[q-1]);break;case 90:this.$=V[q-2],re.setCssStyle(V[q-1],V[q]);break;case 91:re.setCssClass(V[q-1],V[q]);break;case 92:this.$=[V[q]];break;case 93:V[q-2].push(V[q]),this.$=V[q-2];break;case 95:this.$=V[q-1]+V[q];break}},"anonymous"),table:[{3:1,4:2,5:3,6:4,7:[1,6],10:5,16:39,18:21,19:40,21:7,23:8,24:9,25:10,26:11,27:12,28:13,29:14,30:15,31:16,32:17,33:e,35:r,37:n,38:22,42:i,43:23,46:a,47:s,49:l,50:u,52:h,54:f,55:d,58:p,60:m,61:g,62:y,63:v,73:x,74:b,76:w,80:C,81:T,84:E,99:A,101:S,102:_},{1:[3]},{1:[2,1]},{1:[2,2]},{1:[2,3]},t(I,[2,5],{8:[1,48]}),{8:[1,49]},t(D,[2,18],{22:[1,50]}),t(D,[2,20]),t(D,[2,21]),t(D,[2,22]),t(D,[2,23]),t(D,[2,24]),t(D,[2,25]),t(D,[2,26]),t(D,[2,27]),t(D,[2,28]),t(D,[2,29]),{34:[1,51]},{36:[1,52]},t(D,[2,32]),t(D,[2,48],{51:53,64:56,65:57,13:[1,54],22:[1,55],66:k,67:L,68:R,69:O,70:M,71:B,72:F}),{39:[1,65]},t(P,[2,39],{39:[1,67],44:[1,66]}),t(D,[2,50]),t(D,[2,51]),{16:68,58:p,84:E,99:A,101:S},{16:39,18:69,19:40,58:p,84:E,99:A,101:S,102:_},{16:39,18:70,19:40,58:p,84:E,99:A,101:S,102:_},{16:39,18:71,19:40,58:p,84:E,99:A,101:S,102:_},{58:[1,72]},{13:[1,73]},{16:39,18:74,19:40,58:p,84:E,99:A,101:S,102:_},{13:z,53:75},{56:77,58:[1,78]},t(D,[2,61]),t(D,[2,62]),t(D,[2,63]),t(D,[2,64]),t($,[2,12],{16:39,19:40,18:80,17:[1,79],20:[1,81],58:p,84:E,99:A,101:S,102:_}),t($,[2,14],{20:[1,82]}),{15:83,16:84,58:p,84:E,99:A,101:S},{16:39,18:85,19:40,58:p,84:E,99:A,101:S,102:_},t(H,[2,118]),t(H,[2,119]),t(H,[2,120]),t(H,[2,121]),t([1,8,9,12,13,20,22,39,41,44,66,67,68,69,70,71,72,77,79],[2,122]),t(I,[2,6],{10:5,21:7,23:8,24:9,25:10,26:11,27:12,28:13,29:14,30:15,31:16,32:17,18:21,38:22,43:23,16:39,19:40,5:86,33:e,35:r,37:n,42:i,46:a,47:s,49:l,50:u,52:h,54:f,55:d,58:p,60:m,61:g,62:y,63:v,73:x,74:b,76:w,80:C,81:T,84:E,99:A,101:S,102:_}),{5:87,10:5,16:39,18:21,19:40,21:7,23:8,24:9,25:10,26:11,27:12,28:13,29:14,30:15,31:16,32:17,33:e,35:r,37:n,38:22,42:i,43:23,46:a,47:s,49:l,50:u,52:h,54:f,55:d,58:p,60:m,61:g,62:y,63:v,73:x,74:b,76:w,80:C,81:T,84:E,99:A,101:S,102:_},t(D,[2,19]),t(D,[2,30]),t(D,[2,31]),{13:[1,89],16:39,18:88,19:40,58:p,84:E,99:A,101:S,102:_},{51:90,64:56,65:57,66:k,67:L,68:R,69:O,70:M,71:B,72:F},t(D,[2,49]),{65:91,71:B,72:F},t(Q,[2,68],{64:92,66:k,67:L,68:R,69:O,70:M}),t(j,[2,69]),t(j,[2,70]),t(j,[2,71]),t(j,[2,72]),t(j,[2,73]),t(ie,[2,74]),t(ie,[2,75]),{8:[1,94],24:95,40:93,43:23,46:a},{16:96,58:p,84:E,99:A,101:S},{45:97,49:ne},{48:[1,99]},{13:[1,100]},{13:[1,101]},{77:[1,102],79:[1,103]},{22:le,57:104,58:he,80:K,82:105,83:106,84:X,85:te,86:J,87:se,88:ue,89:Z},{58:[1,116]},{13:z,53:117},t(D,[2,57]),t(D,[2,123]),{22:le,57:118,58:he,59:[1,119],80:K,82:105,83:106,84:X,85:te,86:J,87:se,88:ue,89:Z},t(Se,[2,59]),{16:39,18:120,19:40,58:p,84:E,99:A,101:S,102:_},t($,[2,15]),t($,[2,16]),t($,[2,17]),{39:[2,35]},{15:122,16:84,17:[1,121],39:[2,9],58:p,84:E,99:A,101:S},t(ce,[2,43],{11:123,12:[1,124]}),t(I,[2,7]),{9:[1,125]},t(ae,[2,52]),{16:39,18:126,19:40,58:p,84:E,99:A,101:S,102:_},{13:[1,128],16:39,18:127,19:40,58:p,84:E,99:A,101:S,102:_},t(Q,[2,67],{64:129,66:k,67:L,68:R,69:O,70:M}),t(Q,[2,66]),{41:[1,130]},{24:95,40:131,43:23,46:a},{8:[1,132],41:[2,36]},t(P,[2,40],{39:[1,133]}),{41:[1,134]},{41:[2,46],45:135,49:ne},{16:39,18:136,19:40,58:p,84:E,99:A,101:S,102:_},t(D,[2,76],{13:[1,137]}),t(D,[2,78],{13:[1,139],75:[1,138]}),t(D,[2,82],{13:[1,140],78:[1,141]}),{13:[1,142]},t(D,[2,90],{59:Oe}),t(ge,[2,92],{83:144,22:le,58:he,80:K,84:X,85:te,86:J,87:se,88:ue,89:Z}),t(ze,[2,94]),t(ze,[2,96]),t(ze,[2,97]),t(ze,[2,98]),t(ze,[2,99]),t(ze,[2,100]),t(ze,[2,101]),t(ze,[2,102]),t(ze,[2,103]),t(ze,[2,104]),t(D,[2,91]),t(D,[2,56]),t(D,[2,58],{59:Oe}),{58:[1,145]},t($,[2,13]),{15:146,16:84,58:p,84:E,99:A,101:S},{39:[2,11]},t(ce,[2,44]),{13:[1,147]},{1:[2,4]},t(ae,[2,54]),t(ae,[2,53]),{16:39,18:148,19:40,58:p,84:E,99:A,101:S,102:_},t(Q,[2,65]),t(D,[2,33]),{41:[1,149]},{24:95,40:150,41:[2,37],43:23,46:a},{45:151,49:ne},t(P,[2,41]),{41:[2,47]},t(D,[2,45]),t(D,[2,77]),t(D,[2,79]),t(D,[2,80],{75:[1,152]}),t(D,[2,83]),t(D,[2,84],{13:[1,153]}),t(D,[2,86],{13:[1,155],75:[1,154]}),{22:le,58:he,80:K,82:156,83:106,84:X,85:te,86:J,87:se,88:ue,89:Z},t(ze,[2,95]),t(Se,[2,60]),{39:[2,10]},{14:[1,157]},t(ae,[2,55]),t(D,[2,34]),{41:[2,38]},{41:[1,158]},t(D,[2,81]),t(D,[2,85]),t(D,[2,87]),t(D,[2,88],{75:[1,159]}),t(ge,[2,93],{83:144,22:le,58:he,80:K,84:X,85:te,86:J,87:se,88:ue,89:Z}),t(ce,[2,8]),t(P,[2,42]),t(D,[2,89])],defaultActions:{2:[2,1],3:[2,2],4:[2,3],83:[2,35],122:[2,11],125:[2,4],135:[2,47],146:[2,10],150:[2,38]},parseError:o(function(be,W){if(W.recoverable)this.trace(be);else{var de=new Error(be);throw de.hash=W,de}},"parseError"),parse:o(function(be){var W=this,de=[0],re=[],oe=[null],V=[],xe=this.table,q="",pe=0,ve=0,Pe=0,_e=2,we=1,Ve=V.slice.call(arguments,1),De=Object.create(this.lexer),qe={yy:{}};for(var at in this.yy)Object.prototype.hasOwnProperty.call(this.yy,at)&&(qe.yy[at]=this.yy[at]);De.setInput(be,qe.yy),qe.yy.lexer=De,qe.yy.parser=this,typeof De.yylloc>"u"&&(De.yylloc={});var Rt=De.yylloc;V.push(Rt);var st=De.options&&De.options.ranges;typeof qe.yy.parseError=="function"?this.parseError=qe.yy.parseError:this.parseError=Object.getPrototypeOf(this).parseError;function Ue(Tt){de.length=de.length-2*Tt,oe.length=oe.length-Tt,V.length=V.length-Tt}o(Ue,"popStack");function ct(){var Tt;return Tt=re.pop()||De.lex()||we,typeof Tt!="number"&&(Tt instanceof Array&&(re=Tt,Tt=re.pop()),Tt=W.symbols_[Tt]||Tt),Tt}o(ct,"lex");for(var We,ot,Yt,bt,Mt,xt,ut={},Et,ft,yt,nt;;){if(Yt=de[de.length-1],this.defaultActions[Yt]?bt=this.defaultActions[Yt]:((We===null||typeof We>"u")&&(We=ct()),bt=xe[Yt]&&xe[Yt][We]),typeof bt>"u"||!bt.length||!bt[0]){var dn="";nt=[];for(Et in xe[Yt])this.terminals_[Et]&&Et>_e&&nt.push("'"+this.terminals_[Et]+"'");De.showPosition?dn="Parse error on line "+(pe+1)+`: +`+De.showPosition()+` +Expecting `+nt.join(", ")+", got '"+(this.terminals_[We]||We)+"'":dn="Parse error on line "+(pe+1)+": Unexpected "+(We==we?"end of input":"'"+(this.terminals_[We]||We)+"'"),this.parseError(dn,{text:De.match,token:this.terminals_[We]||We,line:De.yylineno,loc:Rt,expected:nt})}if(bt[0]instanceof Array&&bt.length>1)throw new Error("Parse Error: multiple actions possible at state: "+Yt+", token: "+We);switch(bt[0]){case 1:de.push(We),oe.push(De.yytext),V.push(De.yylloc),de.push(bt[1]),We=null,ot?(We=ot,ot=null):(ve=De.yyleng,q=De.yytext,pe=De.yylineno,Rt=De.yylloc,Pe>0&&Pe--);break;case 2:if(ft=this.productions_[bt[1]][1],ut.$=oe[oe.length-ft],ut._$={first_line:V[V.length-(ft||1)].first_line,last_line:V[V.length-1].last_line,first_column:V[V.length-(ft||1)].first_column,last_column:V[V.length-1].last_column},st&&(ut._$.range=[V[V.length-(ft||1)].range[0],V[V.length-1].range[1]]),xt=this.performAction.apply(ut,[q,ve,pe,qe.yy,bt[1],oe,V].concat(Ve)),typeof xt<"u")return xt;ft&&(de=de.slice(0,-1*ft*2),oe=oe.slice(0,-1*ft),V=V.slice(0,-1*ft)),de.push(this.productions_[bt[1]][0]),oe.push(ut.$),V.push(ut._$),yt=xe[de[de.length-2]][de[de.length-1]],de.push(yt);break;case 3:return!0}}return!0},"parse")},$e=function(){var Ie={EOF:1,parseError:o(function(W,de){if(this.yy.parser)this.yy.parser.parseError(W,de);else throw new Error(W)},"parseError"),setInput:o(function(be,W){return this.yy=W||this.yy||{},this._input=be,this._more=this._backtrack=this.done=!1,this.yylineno=this.yyleng=0,this.yytext=this.matched=this.match="",this.conditionStack=["INITIAL"],this.yylloc={first_line:1,first_column:0,last_line:1,last_column:0},this.options.ranges&&(this.yylloc.range=[0,0]),this.offset=0,this},"setInput"),input:o(function(){var be=this._input[0];this.yytext+=be,this.yyleng++,this.offset++,this.match+=be,this.matched+=be;var W=be.match(/(?:\r\n?|\n).*/g);return W?(this.yylineno++,this.yylloc.last_line++):this.yylloc.last_column++,this.options.ranges&&this.yylloc.range[1]++,this._input=this._input.slice(1),be},"input"),unput:o(function(be){var W=be.length,de=be.split(/(?:\r\n?|\n)/g);this._input=be+this._input,this.yytext=this.yytext.substr(0,this.yytext.length-W),this.offset-=W;var re=this.match.split(/(?:\r\n?|\n)/g);this.match=this.match.substr(0,this.match.length-1),this.matched=this.matched.substr(0,this.matched.length-1),de.length-1&&(this.yylineno-=de.length-1);var oe=this.yylloc.range;return this.yylloc={first_line:this.yylloc.first_line,last_line:this.yylineno+1,first_column:this.yylloc.first_column,last_column:de?(de.length===re.length?this.yylloc.first_column:0)+re[re.length-de.length].length-de[0].length:this.yylloc.first_column-W},this.options.ranges&&(this.yylloc.range=[oe[0],oe[0]+this.yyleng-W]),this.yyleng=this.yytext.length,this},"unput"),more:o(function(){return this._more=!0,this},"more"),reject:o(function(){if(this.options.backtrack_lexer)this._backtrack=!0;else return this.parseError("Lexical error on line "+(this.yylineno+1)+`. You can only invoke reject() in the lexer when the lexer is of the backtracking persuasion (options.backtrack_lexer = true). +`+this.showPosition(),{text:"",token:null,line:this.yylineno});return this},"reject"),less:o(function(be){this.unput(this.match.slice(be))},"less"),pastInput:o(function(){var be=this.matched.substr(0,this.matched.length-this.match.length);return(be.length>20?"...":"")+be.substr(-20).replace(/\n/g,"")},"pastInput"),upcomingInput:o(function(){var be=this.match;return be.length<20&&(be+=this._input.substr(0,20-be.length)),(be.substr(0,20)+(be.length>20?"...":"")).replace(/\n/g,"")},"upcomingInput"),showPosition:o(function(){var be=this.pastInput(),W=new Array(be.length+1).join("-");return be+this.upcomingInput()+` +`+W+"^"},"showPosition"),test_match:o(function(be,W){var de,re,oe;if(this.options.backtrack_lexer&&(oe={yylineno:this.yylineno,yylloc:{first_line:this.yylloc.first_line,last_line:this.last_line,first_column:this.yylloc.first_column,last_column:this.yylloc.last_column},yytext:this.yytext,match:this.match,matches:this.matches,matched:this.matched,yyleng:this.yyleng,offset:this.offset,_more:this._more,_input:this._input,yy:this.yy,conditionStack:this.conditionStack.slice(0),done:this.done},this.options.ranges&&(oe.yylloc.range=this.yylloc.range.slice(0))),re=be[0].match(/(?:\r\n?|\n).*/g),re&&(this.yylineno+=re.length),this.yylloc={first_line:this.yylloc.last_line,last_line:this.yylineno+1,first_column:this.yylloc.last_column,last_column:re?re[re.length-1].length-re[re.length-1].match(/\r?\n?/)[0].length:this.yylloc.last_column+be[0].length},this.yytext+=be[0],this.match+=be[0],this.matches=be,this.yyleng=this.yytext.length,this.options.ranges&&(this.yylloc.range=[this.offset,this.offset+=this.yyleng]),this._more=!1,this._backtrack=!1,this._input=this._input.slice(be[0].length),this.matched+=be[0],de=this.performAction.call(this,this.yy,this,W,this.conditionStack[this.conditionStack.length-1]),this.done&&this._input&&(this.done=!1),de)return de;if(this._backtrack){for(var V in oe)this[V]=oe[V];return!1}return!1},"test_match"),next:o(function(){if(this.done)return this.EOF;this._input||(this.done=!0);var be,W,de,re;this._more||(this.yytext="",this.match="");for(var oe=this._currentRules(),V=0;VW[0].length)){if(W=de,re=V,this.options.backtrack_lexer){if(be=this.test_match(de,oe[V]),be!==!1)return be;if(this._backtrack){W=!1;continue}else return!1}else if(!this.options.flex)break}return W?(be=this.test_match(W,oe[re]),be!==!1?be:!1):this._input===""?this.EOF:this.parseError("Lexical error on line "+(this.yylineno+1)+`. Unrecognized text. +`+this.showPosition(),{text:"",token:null,line:this.yylineno})},"next"),lex:o(function(){var W=this.next();return W||this.lex()},"lex"),begin:o(function(W){this.conditionStack.push(W)},"begin"),popState:o(function(){var W=this.conditionStack.length-1;return W>0?this.conditionStack.pop():this.conditionStack[0]},"popState"),_currentRules:o(function(){return this.conditionStack.length&&this.conditionStack[this.conditionStack.length-1]?this.conditions[this.conditionStack[this.conditionStack.length-1]].rules:this.conditions.INITIAL.rules},"_currentRules"),topState:o(function(W){return W=this.conditionStack.length-1-Math.abs(W||0),W>=0?this.conditionStack[W]:"INITIAL"},"topState"),pushState:o(function(W){this.begin(W)},"pushState"),stateStackSize:o(function(){return this.conditionStack.length},"stateStackSize"),options:{},performAction:o(function(W,de,re,oe){var V=oe;switch(re){case 0:return 60;case 1:return 61;case 2:return 62;case 3:return 63;case 4:break;case 5:break;case 6:return this.begin("acc_title"),33;break;case 7:return this.popState(),"acc_title_value";break;case 8:return this.begin("acc_descr"),35;break;case 9:return this.popState(),"acc_descr_value";break;case 10:this.begin("acc_descr_multiline");break;case 11:this.popState();break;case 12:return"acc_descr_multiline_value";case 13:return 8;case 14:break;case 15:return 7;case 16:return 7;case 17:return"EDGE_STATE";case 18:this.begin("callback_name");break;case 19:this.popState();break;case 20:this.popState(),this.begin("callback_args");break;case 21:return 77;case 22:this.popState();break;case 23:return 78;case 24:this.popState();break;case 25:return"STR";case 26:this.begin("string");break;case 27:return 80;case 28:return 55;case 29:return this.begin("namespace"),42;break;case 30:return this.popState(),8;break;case 31:break;case 32:return this.begin("namespace-body"),39;break;case 33:return this.popState(),41;break;case 34:return"EOF_IN_STRUCT";case 35:return 8;case 36:break;case 37:return"EDGE_STATE";case 38:return this.begin("class"),46;break;case 39:return this.popState(),8;break;case 40:break;case 41:return this.popState(),this.popState(),41;break;case 42:return this.begin("class-body"),39;break;case 43:return this.popState(),41;break;case 44:return"EOF_IN_STRUCT";case 45:return"EDGE_STATE";case 46:return"OPEN_IN_STRUCT";case 47:break;case 48:return"MEMBER";case 49:return 81;case 50:return 73;case 51:return 74;case 52:return 76;case 53:return 52;case 54:return 54;case 55:return 47;case 56:return 48;case 57:return 79;case 58:this.popState();break;case 59:return"GENERICTYPE";case 60:this.begin("generic");break;case 61:this.popState();break;case 62:return"BQUOTE_STR";case 63:this.begin("bqstring");break;case 64:return 75;case 65:return 75;case 66:return 75;case 67:return 75;case 68:return 67;case 69:return 67;case 70:return 69;case 71:return 69;case 72:return 68;case 73:return 66;case 74:return 70;case 75:return 71;case 76:return 72;case 77:return 22;case 78:return 44;case 79:return 99;case 80:return 17;case 81:return"PLUS";case 82:return 85;case 83:return 59;case 84:return 88;case 85:return 88;case 86:return 89;case 87:return"EQUALS";case 88:return"EQUALS";case 89:return 58;case 90:return 12;case 91:return 14;case 92:return"PUNCTUATION";case 93:return 84;case 94:return 101;case 95:return 87;case 96:return 87;case 97:return 9}},"anonymous"),rules:[/^(?:.*direction\s+TB[^\n]*)/,/^(?:.*direction\s+BT[^\n]*)/,/^(?:.*direction\s+RL[^\n]*)/,/^(?:.*direction\s+LR[^\n]*)/,/^(?:%%(?!\{)*[^\n]*(\r?\n?)+)/,/^(?:%%[^\n]*(\r?\n)*)/,/^(?:accTitle\s*:\s*)/,/^(?:(?!\n||)*[^\n]*)/,/^(?:accDescr\s*:\s*)/,/^(?:(?!\n||)*[^\n]*)/,/^(?:accDescr\s*\{\s*)/,/^(?:[\}])/,/^(?:[^\}]*)/,/^(?:\s*(\r?\n)+)/,/^(?:\s+)/,/^(?:classDiagram-v2\b)/,/^(?:classDiagram\b)/,/^(?:\[\*\])/,/^(?:call[\s]+)/,/^(?:\([\s]*\))/,/^(?:\()/,/^(?:[^(]*)/,/^(?:\))/,/^(?:[^)]*)/,/^(?:["])/,/^(?:[^"]*)/,/^(?:["])/,/^(?:style\b)/,/^(?:classDef\b)/,/^(?:namespace\b)/,/^(?:\s*(\r?\n)+)/,/^(?:\s+)/,/^(?:[{])/,/^(?:[}])/,/^(?:$)/,/^(?:\s*(\r?\n)+)/,/^(?:\s+)/,/^(?:\[\*\])/,/^(?:class\b)/,/^(?:\s*(\r?\n)+)/,/^(?:\s+)/,/^(?:[}])/,/^(?:[{])/,/^(?:[}])/,/^(?:$)/,/^(?:\[\*\])/,/^(?:[{])/,/^(?:[\n])/,/^(?:[^{}\n]*)/,/^(?:cssClass\b)/,/^(?:callback\b)/,/^(?:link\b)/,/^(?:click\b)/,/^(?:note for\b)/,/^(?:note\b)/,/^(?:<<)/,/^(?:>>)/,/^(?:href\b)/,/^(?:[~])/,/^(?:[^~]*)/,/^(?:~)/,/^(?:[`])/,/^(?:[^`]+)/,/^(?:[`])/,/^(?:_self\b)/,/^(?:_blank\b)/,/^(?:_parent\b)/,/^(?:_top\b)/,/^(?:\s*<\|)/,/^(?:\s*\|>)/,/^(?:\s*>)/,/^(?:\s*<)/,/^(?:\s*\*)/,/^(?:\s*o\b)/,/^(?:\s*\(\))/,/^(?:--)/,/^(?:\.\.)/,/^(?::{1}[^:\n;]+)/,/^(?::{3})/,/^(?:-)/,/^(?:\.)/,/^(?:\+)/,/^(?::)/,/^(?:,)/,/^(?:#)/,/^(?:#)/,/^(?:%)/,/^(?:=)/,/^(?:=)/,/^(?:\w+)/,/^(?:\[)/,/^(?:\])/,/^(?:[!"#$%&'*+,-.`?\\/])/,/^(?:[0-9]+)/,/^(?:[\u00AA\u00B5\u00BA\u00C0-\u00D6\u00D8-\u00F6]|[\u00F8-\u02C1\u02C6-\u02D1\u02E0-\u02E4\u02EC\u02EE\u0370-\u0374\u0376\u0377]|[\u037A-\u037D\u0386\u0388-\u038A\u038C\u038E-\u03A1\u03A3-\u03F5]|[\u03F7-\u0481\u048A-\u0527\u0531-\u0556\u0559\u0561-\u0587\u05D0-\u05EA]|[\u05F0-\u05F2\u0620-\u064A\u066E\u066F\u0671-\u06D3\u06D5\u06E5\u06E6\u06EE]|[\u06EF\u06FA-\u06FC\u06FF\u0710\u0712-\u072F\u074D-\u07A5\u07B1\u07CA-\u07EA]|[\u07F4\u07F5\u07FA\u0800-\u0815\u081A\u0824\u0828\u0840-\u0858\u08A0]|[\u08A2-\u08AC\u0904-\u0939\u093D\u0950\u0958-\u0961\u0971-\u0977]|[\u0979-\u097F\u0985-\u098C\u098F\u0990\u0993-\u09A8\u09AA-\u09B0\u09B2]|[\u09B6-\u09B9\u09BD\u09CE\u09DC\u09DD\u09DF-\u09E1\u09F0\u09F1\u0A05-\u0A0A]|[\u0A0F\u0A10\u0A13-\u0A28\u0A2A-\u0A30\u0A32\u0A33\u0A35\u0A36\u0A38\u0A39]|[\u0A59-\u0A5C\u0A5E\u0A72-\u0A74\u0A85-\u0A8D\u0A8F-\u0A91\u0A93-\u0AA8]|[\u0AAA-\u0AB0\u0AB2\u0AB3\u0AB5-\u0AB9\u0ABD\u0AD0\u0AE0\u0AE1\u0B05-\u0B0C]|[\u0B0F\u0B10\u0B13-\u0B28\u0B2A-\u0B30\u0B32\u0B33\u0B35-\u0B39\u0B3D\u0B5C]|[\u0B5D\u0B5F-\u0B61\u0B71\u0B83\u0B85-\u0B8A\u0B8E-\u0B90\u0B92-\u0B95\u0B99]|[\u0B9A\u0B9C\u0B9E\u0B9F\u0BA3\u0BA4\u0BA8-\u0BAA\u0BAE-\u0BB9\u0BD0]|[\u0C05-\u0C0C\u0C0E-\u0C10\u0C12-\u0C28\u0C2A-\u0C33\u0C35-\u0C39\u0C3D]|[\u0C58\u0C59\u0C60\u0C61\u0C85-\u0C8C\u0C8E-\u0C90\u0C92-\u0CA8\u0CAA-\u0CB3]|[\u0CB5-\u0CB9\u0CBD\u0CDE\u0CE0\u0CE1\u0CF1\u0CF2\u0D05-\u0D0C\u0D0E-\u0D10]|[\u0D12-\u0D3A\u0D3D\u0D4E\u0D60\u0D61\u0D7A-\u0D7F\u0D85-\u0D96\u0D9A-\u0DB1]|[\u0DB3-\u0DBB\u0DBD\u0DC0-\u0DC6\u0E01-\u0E30\u0E32\u0E33\u0E40-\u0E46\u0E81]|[\u0E82\u0E84\u0E87\u0E88\u0E8A\u0E8D\u0E94-\u0E97\u0E99-\u0E9F\u0EA1-\u0EA3]|[\u0EA5\u0EA7\u0EAA\u0EAB\u0EAD-\u0EB0\u0EB2\u0EB3\u0EBD\u0EC0-\u0EC4\u0EC6]|[\u0EDC-\u0EDF\u0F00\u0F40-\u0F47\u0F49-\u0F6C\u0F88-\u0F8C\u1000-\u102A]|[\u103F\u1050-\u1055\u105A-\u105D\u1061\u1065\u1066\u106E-\u1070\u1075-\u1081]|[\u108E\u10A0-\u10C5\u10C7\u10CD\u10D0-\u10FA\u10FC-\u1248\u124A-\u124D]|[\u1250-\u1256\u1258\u125A-\u125D\u1260-\u1288\u128A-\u128D\u1290-\u12B0]|[\u12B2-\u12B5\u12B8-\u12BE\u12C0\u12C2-\u12C5\u12C8-\u12D6\u12D8-\u1310]|[\u1312-\u1315\u1318-\u135A\u1380-\u138F\u13A0-\u13F4\u1401-\u166C]|[\u166F-\u167F\u1681-\u169A\u16A0-\u16EA\u1700-\u170C\u170E-\u1711]|[\u1720-\u1731\u1740-\u1751\u1760-\u176C\u176E-\u1770\u1780-\u17B3\u17D7]|[\u17DC\u1820-\u1877\u1880-\u18A8\u18AA\u18B0-\u18F5\u1900-\u191C]|[\u1950-\u196D\u1970-\u1974\u1980-\u19AB\u19C1-\u19C7\u1A00-\u1A16]|[\u1A20-\u1A54\u1AA7\u1B05-\u1B33\u1B45-\u1B4B\u1B83-\u1BA0\u1BAE\u1BAF]|[\u1BBA-\u1BE5\u1C00-\u1C23\u1C4D-\u1C4F\u1C5A-\u1C7D\u1CE9-\u1CEC]|[\u1CEE-\u1CF1\u1CF5\u1CF6\u1D00-\u1DBF\u1E00-\u1F15\u1F18-\u1F1D]|[\u1F20-\u1F45\u1F48-\u1F4D\u1F50-\u1F57\u1F59\u1F5B\u1F5D\u1F5F-\u1F7D]|[\u1F80-\u1FB4\u1FB6-\u1FBC\u1FBE\u1FC2-\u1FC4\u1FC6-\u1FCC\u1FD0-\u1FD3]|[\u1FD6-\u1FDB\u1FE0-\u1FEC\u1FF2-\u1FF4\u1FF6-\u1FFC\u2071\u207F]|[\u2090-\u209C\u2102\u2107\u210A-\u2113\u2115\u2119-\u211D\u2124\u2126\u2128]|[\u212A-\u212D\u212F-\u2139\u213C-\u213F\u2145-\u2149\u214E\u2183\u2184]|[\u2C00-\u2C2E\u2C30-\u2C5E\u2C60-\u2CE4\u2CEB-\u2CEE\u2CF2\u2CF3]|[\u2D00-\u2D25\u2D27\u2D2D\u2D30-\u2D67\u2D6F\u2D80-\u2D96\u2DA0-\u2DA6]|[\u2DA8-\u2DAE\u2DB0-\u2DB6\u2DB8-\u2DBE\u2DC0-\u2DC6\u2DC8-\u2DCE]|[\u2DD0-\u2DD6\u2DD8-\u2DDE\u2E2F\u3005\u3006\u3031-\u3035\u303B\u303C]|[\u3041-\u3096\u309D-\u309F\u30A1-\u30FA\u30FC-\u30FF\u3105-\u312D]|[\u3131-\u318E\u31A0-\u31BA\u31F0-\u31FF\u3400-\u4DB5\u4E00-\u9FCC]|[\uA000-\uA48C\uA4D0-\uA4FD\uA500-\uA60C\uA610-\uA61F\uA62A\uA62B]|[\uA640-\uA66E\uA67F-\uA697\uA6A0-\uA6E5\uA717-\uA71F\uA722-\uA788]|[\uA78B-\uA78E\uA790-\uA793\uA7A0-\uA7AA\uA7F8-\uA801\uA803-\uA805]|[\uA807-\uA80A\uA80C-\uA822\uA840-\uA873\uA882-\uA8B3\uA8F2-\uA8F7\uA8FB]|[\uA90A-\uA925\uA930-\uA946\uA960-\uA97C\uA984-\uA9B2\uA9CF\uAA00-\uAA28]|[\uAA40-\uAA42\uAA44-\uAA4B\uAA60-\uAA76\uAA7A\uAA80-\uAAAF\uAAB1\uAAB5]|[\uAAB6\uAAB9-\uAABD\uAAC0\uAAC2\uAADB-\uAADD\uAAE0-\uAAEA\uAAF2-\uAAF4]|[\uAB01-\uAB06\uAB09-\uAB0E\uAB11-\uAB16\uAB20-\uAB26\uAB28-\uAB2E]|[\uABC0-\uABE2\uAC00-\uD7A3\uD7B0-\uD7C6\uD7CB-\uD7FB\uF900-\uFA6D]|[\uFA70-\uFAD9\uFB00-\uFB06\uFB13-\uFB17\uFB1D\uFB1F-\uFB28\uFB2A-\uFB36]|[\uFB38-\uFB3C\uFB3E\uFB40\uFB41\uFB43\uFB44\uFB46-\uFBB1\uFBD3-\uFD3D]|[\uFD50-\uFD8F\uFD92-\uFDC7\uFDF0-\uFDFB\uFE70-\uFE74\uFE76-\uFEFC]|[\uFF21-\uFF3A\uFF41-\uFF5A\uFF66-\uFFBE\uFFC2-\uFFC7\uFFCA-\uFFCF]|[\uFFD2-\uFFD7\uFFDA-\uFFDC])/,/^(?:\s)/,/^(?:\s)/,/^(?:$)/],conditions:{"namespace-body":{rules:[26,33,34,35,36,37,38,49,50,51,52,53,54,55,56,57,60,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,86,87,88,89,90,91,92,93,94,95,97],inclusive:!1},namespace:{rules:[26,29,30,31,32,49,50,51,52,53,54,55,56,57,60,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,86,87,88,89,90,91,92,93,94,95,97],inclusive:!1},"class-body":{rules:[26,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,60,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,86,87,88,89,90,91,92,93,94,95,97],inclusive:!1},class:{rules:[26,39,40,41,42,49,50,51,52,53,54,55,56,57,60,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,86,87,88,89,90,91,92,93,94,95,97],inclusive:!1},acc_descr_multiline:{rules:[11,12,26,49,50,51,52,53,54,55,56,57,60,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,86,87,88,89,90,91,92,93,94,95,97],inclusive:!1},acc_descr:{rules:[9,26,49,50,51,52,53,54,55,56,57,60,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,86,87,88,89,90,91,92,93,94,95,97],inclusive:!1},acc_title:{rules:[7,26,49,50,51,52,53,54,55,56,57,60,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,86,87,88,89,90,91,92,93,94,95,97],inclusive:!1},callback_args:{rules:[22,23,26,49,50,51,52,53,54,55,56,57,60,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,86,87,88,89,90,91,92,93,94,95,97],inclusive:!1},callback_name:{rules:[19,20,21,26,49,50,51,52,53,54,55,56,57,60,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,86,87,88,89,90,91,92,93,94,95,97],inclusive:!1},href:{rules:[26,49,50,51,52,53,54,55,56,57,60,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,86,87,88,89,90,91,92,93,94,95,97],inclusive:!1},struct:{rules:[26,49,50,51,52,53,54,55,56,57,60,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,86,87,88,89,90,91,92,93,94,95,97],inclusive:!1},generic:{rules:[26,49,50,51,52,53,54,55,56,57,58,59,60,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,86,87,88,89,90,91,92,93,94,95,97],inclusive:!1},bqstring:{rules:[26,49,50,51,52,53,54,55,56,57,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,86,87,88,89,90,91,92,93,94,95,97],inclusive:!1},string:{rules:[24,25,26,49,50,51,52,53,54,55,56,57,60,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,86,87,88,89,90,91,92,93,94,95,97],inclusive:!1},INITIAL:{rules:[0,1,2,3,4,5,6,8,10,13,14,15,16,17,18,26,27,28,29,38,49,50,51,52,53,54,55,56,57,60,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97],inclusive:!0}}};return Ie}();He.lexer=$e;function Re(){this.yy={}}return o(Re,"Parser"),Re.prototype=He,He.Parser=Re,new Re}();MO.parser=MO;L6=MO});var Dfe,kb,Lfe=N(()=>{"use strict";zt();gr();Dfe=["#","+","~","-",""],kb=class{static{o(this,"ClassMember")}constructor(e,r){this.memberType=r,this.visibility="",this.classifier="",this.text="";let n=Tr(e,me());this.parseMember(n)}getDisplayDetails(){let e=this.visibility+ec(this.id);this.memberType==="method"&&(e+=`(${ec(this.parameters.trim())})`,this.returnType&&(e+=" : "+ec(this.returnType))),e=e.trim();let r=this.parseClassifier();return{displayText:e,cssStyle:r}}parseMember(e){let r="";if(this.memberType==="method"){let a=/([#+~-])?(.+)\((.*)\)([\s$*])?(.*)([$*])?/.exec(e);if(a){let s=a[1]?a[1].trim():"";if(Dfe.includes(s)&&(this.visibility=s),this.id=a[2],this.parameters=a[3]?a[3].trim():"",r=a[4]?a[4].trim():"",this.returnType=a[5]?a[5].trim():"",r===""){let l=this.returnType.substring(this.returnType.length-1);/[$*]/.exec(l)&&(r=l,this.returnType=this.returnType.substring(0,this.returnType.length-1))}}}else{let i=e.length,a=e.substring(0,1),s=e.substring(i-1);Dfe.includes(a)&&(this.visibility=a),/[$*]/.exec(s)&&(r=s),this.id=e.substring(this.visibility===""?0:1,r===""?i:i-1)}this.classifier=r,this.id=this.id.startsWith(" ")?" "+this.id.trim():this.id.trim();let n=`${this.visibility?"\\"+this.visibility:""}${ec(this.id)}${this.memberType==="method"?`(${ec(this.parameters)})${this.returnType?" : "+ec(this.returnType):""}`:""}`;this.text=n.replaceAll("<","<").replaceAll(">",">"),this.text.startsWith("\\<")&&(this.text=this.text.replace("\\<","~"))}parseClassifier(){switch(this.classifier){case"*":return"font-style:italic;";case"$":return"text-decoration:underline;";default:return""}}}});var R6,Rfe,Lp,D1,OO=N(()=>{"use strict";dr();vt();zt();gr();ir();mi();Lfe();R6="classId-",Rfe=0,Lp=o(t=>Ze.sanitizeText(t,me()),"sanitizeText"),D1=class{constructor(){this.relations=[];this.classes=new Map;this.styleClasses=new Map;this.notes=[];this.interfaces=[];this.namespaces=new Map;this.namespaceCounter=0;this.functions=[];this.lineType={LINE:0,DOTTED_LINE:1};this.relationType={AGGREGATION:0,EXTENSION:1,COMPOSITION:2,DEPENDENCY:3,LOLLIPOP:4};this.setupToolTips=o(e=>{let r=Ge(".mermaidTooltip");(r._groups||r)[0][0]===null&&(r=Ge("body").append("div").attr("class","mermaidTooltip").style("opacity",0)),Ge(e).select("svg").selectAll("g.node").on("mouseover",a=>{let s=Ge(a.currentTarget);if(s.attr("title")===null)return;let u=this.getBoundingClientRect();r.transition().duration(200).style("opacity",".9"),r.text(s.attr("title")).style("left",window.scrollX+u.left+(u.right-u.left)/2+"px").style("top",window.scrollY+u.top-14+document.body.scrollTop+"px"),r.html(r.html().replace(/<br\/>/g,"
    ")),s.classed("hover",!0)}).on("mouseout",a=>{r.transition().duration(500).style("opacity",0),Ge(a.currentTarget).classed("hover",!1)})},"setupToolTips");this.direction="TB";this.setAccTitle=Lr;this.getAccTitle=Rr;this.setAccDescription=Nr;this.getAccDescription=Mr;this.setDiagramTitle=$r;this.getDiagramTitle=Ir;this.getConfig=o(()=>me().class,"getConfig");this.functions.push(this.setupToolTips.bind(this)),this.clear(),this.addRelation=this.addRelation.bind(this),this.addClassesToNamespace=this.addClassesToNamespace.bind(this),this.addNamespace=this.addNamespace.bind(this),this.setCssClass=this.setCssClass.bind(this),this.addMembers=this.addMembers.bind(this),this.addClass=this.addClass.bind(this),this.setClassLabel=this.setClassLabel.bind(this),this.addAnnotation=this.addAnnotation.bind(this),this.addMember=this.addMember.bind(this),this.cleanupLabel=this.cleanupLabel.bind(this),this.addNote=this.addNote.bind(this),this.defineClass=this.defineClass.bind(this),this.setDirection=this.setDirection.bind(this),this.setLink=this.setLink.bind(this),this.bindFunctions=this.bindFunctions.bind(this),this.clear=this.clear.bind(this),this.setTooltip=this.setTooltip.bind(this),this.setClickEvent=this.setClickEvent.bind(this),this.setCssStyle=this.setCssStyle.bind(this)}static{o(this,"ClassDB")}splitClassNameAndType(e){let r=Ze.sanitizeText(e,me()),n="",i=r;if(r.indexOf("~")>0){let a=r.split("~");i=Lp(a[0]),n=Lp(a[1])}return{className:i,type:n}}setClassLabel(e,r){let n=Ze.sanitizeText(e,me());r&&(r=Lp(r));let{className:i}=this.splitClassNameAndType(n);this.classes.get(i).label=r,this.classes.get(i).text=`${r}${this.classes.get(i).type?`<${this.classes.get(i).type}>`:""}`}addClass(e){let r=Ze.sanitizeText(e,me()),{className:n,type:i}=this.splitClassNameAndType(r);if(this.classes.has(n))return;let a=Ze.sanitizeText(n,me());this.classes.set(a,{id:a,type:i,label:a,text:`${a}${i?`<${i}>`:""}`,shape:"classBox",cssClasses:"default",methods:[],members:[],annotations:[],styles:[],domId:R6+a+"-"+Rfe}),Rfe++}addInterface(e,r){let n={id:`interface${this.interfaces.length}`,label:e,classId:r};this.interfaces.push(n)}lookUpDomId(e){let r=Ze.sanitizeText(e,me());if(this.classes.has(r))return this.classes.get(r).domId;throw new Error("Class not found: "+r)}clear(){this.relations=[],this.classes=new Map,this.notes=[],this.interfaces=[],this.functions=[],this.functions.push(this.setupToolTips.bind(this)),this.namespaces=new Map,this.namespaceCounter=0,this.direction="TB",Ar()}getClass(e){return this.classes.get(e)}getClasses(){return this.classes}getRelations(){return this.relations}getNotes(){return this.notes}addRelation(e){Y.debug("Adding relation: "+JSON.stringify(e));let r=[this.relationType.LOLLIPOP,this.relationType.AGGREGATION,this.relationType.COMPOSITION,this.relationType.DEPENDENCY,this.relationType.EXTENSION];e.relation.type1===this.relationType.LOLLIPOP&&!r.includes(e.relation.type2)?(this.addClass(e.id2),this.addInterface(e.id1,e.id2),e.id1=`interface${this.interfaces.length-1}`):e.relation.type2===this.relationType.LOLLIPOP&&!r.includes(e.relation.type1)?(this.addClass(e.id1),this.addInterface(e.id2,e.id1),e.id2=`interface${this.interfaces.length-1}`):(this.addClass(e.id1),this.addClass(e.id2)),e.id1=this.splitClassNameAndType(e.id1).className,e.id2=this.splitClassNameAndType(e.id2).className,e.relationTitle1=Ze.sanitizeText(e.relationTitle1.trim(),me()),e.relationTitle2=Ze.sanitizeText(e.relationTitle2.trim(),me()),this.relations.push(e)}addAnnotation(e,r){let n=this.splitClassNameAndType(e).className;this.classes.get(n).annotations.push(r)}addMember(e,r){this.addClass(e);let n=this.splitClassNameAndType(e).className,i=this.classes.get(n);if(typeof r=="string"){let a=r.trim();a.startsWith("<<")&&a.endsWith(">>")?i.annotations.push(Lp(a.substring(2,a.length-2))):a.indexOf(")")>0?i.methods.push(new kb(a,"method")):a&&i.members.push(new kb(a,"attribute"))}}addMembers(e,r){Array.isArray(r)&&(r.reverse(),r.forEach(n=>this.addMember(e,n)))}addNote(e,r){let n={id:`note${this.notes.length}`,class:r,text:e};this.notes.push(n)}cleanupLabel(e){return e.startsWith(":")&&(e=e.substring(1)),Lp(e.trim())}setCssClass(e,r){e.split(",").forEach(n=>{let i=n;/\d/.exec(n[0])&&(i=R6+i);let a=this.classes.get(i);a&&(a.cssClasses+=" "+r)})}defineClass(e,r){for(let n of e){let i=this.styleClasses.get(n);i===void 0&&(i={id:n,styles:[],textStyles:[]},this.styleClasses.set(n,i)),r&&r.forEach(a=>{if(/color/.exec(a)){let s=a.replace("fill","bgFill");i.textStyles.push(s)}i.styles.push(a)}),this.classes.forEach(a=>{a.cssClasses.includes(n)&&a.styles.push(...r.flatMap(s=>s.split(",")))})}}setTooltip(e,r){e.split(",").forEach(n=>{r!==void 0&&(this.classes.get(n).tooltip=Lp(r))})}getTooltip(e,r){return r&&this.namespaces.has(r)?this.namespaces.get(r).classes.get(e).tooltip:this.classes.get(e).tooltip}setLink(e,r,n){let i=me();e.split(",").forEach(a=>{let s=a;/\d/.exec(a[0])&&(s=R6+s);let l=this.classes.get(s);l&&(l.link=Gt.formatUrl(r,i),i.securityLevel==="sandbox"?l.linkTarget="_top":typeof n=="string"?l.linkTarget=Lp(n):l.linkTarget="_blank")}),this.setCssClass(e,"clickable")}setClickEvent(e,r,n){e.split(",").forEach(i=>{this.setClickFunc(i,r,n),this.classes.get(i).haveCallback=!0}),this.setCssClass(e,"clickable")}setClickFunc(e,r,n){let i=Ze.sanitizeText(e,me());if(me().securityLevel!=="loose"||r===void 0)return;let s=i;if(this.classes.has(s)){let l=this.lookUpDomId(s),u=[];if(typeof n=="string"){u=n.split(/,(?=(?:(?:[^"]*"){2})*[^"]*$)/);for(let h=0;h{let h=document.querySelector(`[id="${l}"]`);h!==null&&h.addEventListener("click",()=>{Gt.runFunc(r,...u)},!1)})}}bindFunctions(e){this.functions.forEach(r=>{r(e)})}getDirection(){return this.direction}setDirection(e){this.direction=e}addNamespace(e){this.namespaces.has(e)||(this.namespaces.set(e,{id:e,classes:new Map,children:{},domId:R6+e+"-"+this.namespaceCounter}),this.namespaceCounter++)}getNamespace(e){return this.namespaces.get(e)}getNamespaces(){return this.namespaces}addClassesToNamespace(e,r){if(this.namespaces.has(e))for(let n of r){let{className:i}=this.splitClassNameAndType(n);this.classes.get(i).parent=e,this.namespaces.get(e).classes.set(i,this.classes.get(i))}}setCssStyle(e,r){let n=this.classes.get(e);if(!(!r||!n))for(let i of r)i.includes(",")?n.styles.push(...i.split(",")):n.styles.push(i)}getArrowMarker(e){let r;switch(e){case 0:r="aggregation";break;case 1:r="extension";break;case 2:r="composition";break;case 3:r="dependency";break;case 4:r="lollipop";break;default:r="none"}return r}getData(){let e=[],r=[],n=me();for(let a of this.namespaces.keys()){let s=this.namespaces.get(a);if(s){let l={id:s.id,label:s.id,isGroup:!0,padding:n.class.padding??16,shape:"rect",cssStyles:["fill: none","stroke: black"],look:n.look};e.push(l)}}for(let a of this.classes.keys()){let s=this.classes.get(a);if(s){let l=s;l.parentId=s.parent,l.look=n.look,e.push(l)}}let i=0;for(let a of this.notes){i++;let s={id:a.id,label:a.text,isGroup:!1,shape:"note",padding:n.class.padding??6,cssStyles:["text-align: left","white-space: nowrap",`fill: ${n.themeVariables.noteBkgColor}`,`stroke: ${n.themeVariables.noteBorderColor}`],look:n.look};e.push(s);let l=this.classes.get(a.class)?.id??"";if(l){let u={id:`edgeNote${i}`,start:a.id,end:l,type:"normal",thickness:"normal",classes:"relation",arrowTypeStart:"none",arrowTypeEnd:"none",arrowheadStyle:"",labelStyle:[""],style:["fill: none"],pattern:"dotted",look:n.look};r.push(u)}}for(let a of this.interfaces){let s={id:a.id,label:a.label,isGroup:!1,shape:"rect",cssStyles:["opacity: 0;"],look:n.look};e.push(s)}i=0;for(let a of this.relations){i++;let s={id:$h(a.id1,a.id2,{prefix:"id",counter:i}),start:a.id1,end:a.id2,type:"normal",label:a.title,labelpos:"c",thickness:"normal",classes:"relation",arrowTypeStart:this.getArrowMarker(a.relation.type1),arrowTypeEnd:this.getArrowMarker(a.relation.type2),startLabelRight:a.relationTitle1==="none"?"":a.relationTitle1,endLabelLeft:a.relationTitle2==="none"?"":a.relationTitle2,arrowheadStyle:"",labelStyle:["display: inline-block"],style:a.style||"",pattern:a.relation.lineType==1?"dashed":"solid",look:n.look};r.push(s)}return{nodes:e,edges:r,other:{},config:n,direction:this.getDirection()}}}});var UVe,N6,PO=N(()=>{"use strict";UVe=o(t=>`g.classGroup text { + fill: ${t.nodeBorder||t.classText}; + stroke: none; + font-family: ${t.fontFamily}; + font-size: 10px; + + .title { + font-weight: bolder; + } + +} + +.nodeLabel, .edgeLabel { + color: ${t.classText}; +} +.edgeLabel .label rect { + fill: ${t.mainBkg}; +} +.label text { + fill: ${t.classText}; +} + +.labelBkg { + background: ${t.mainBkg}; +} +.edgeLabel .label span { + background: ${t.mainBkg}; +} + +.classTitle { + font-weight: bolder; +} +.node rect, + .node circle, + .node ellipse, + .node polygon, + .node path { + fill: ${t.mainBkg}; + stroke: ${t.nodeBorder}; + stroke-width: 1px; + } + + +.divider { + stroke: ${t.nodeBorder}; + stroke-width: 1; +} + +g.clickable { + cursor: pointer; +} + +g.classGroup rect { + fill: ${t.mainBkg}; + stroke: ${t.nodeBorder}; +} + +g.classGroup line { + stroke: ${t.nodeBorder}; + stroke-width: 1; +} + +.classLabel .box { + stroke: none; + stroke-width: 0; + fill: ${t.mainBkg}; + opacity: 0.5; +} + +.classLabel .label { + fill: ${t.nodeBorder}; + font-size: 10px; +} + +.relation { + stroke: ${t.lineColor}; + stroke-width: 1; + fill: none; +} + +.dashed-line{ + stroke-dasharray: 3; +} + +.dotted-line{ + stroke-dasharray: 1 2; +} + +#compositionStart, .composition { + fill: ${t.lineColor} !important; + stroke: ${t.lineColor} !important; + stroke-width: 1; +} + +#compositionEnd, .composition { + fill: ${t.lineColor} !important; + stroke: ${t.lineColor} !important; + stroke-width: 1; +} + +#dependencyStart, .dependency { + fill: ${t.lineColor} !important; + stroke: ${t.lineColor} !important; + stroke-width: 1; +} + +#dependencyStart, .dependency { + fill: ${t.lineColor} !important; + stroke: ${t.lineColor} !important; + stroke-width: 1; +} + +#extensionStart, .extension { + fill: transparent !important; + stroke: ${t.lineColor} !important; + stroke-width: 1; +} + +#extensionEnd, .extension { + fill: transparent !important; + stroke: ${t.lineColor} !important; + stroke-width: 1; +} + +#aggregationStart, .aggregation { + fill: transparent !important; + stroke: ${t.lineColor} !important; + stroke-width: 1; +} + +#aggregationEnd, .aggregation { + fill: transparent !important; + stroke: ${t.lineColor} !important; + stroke-width: 1; +} + +#lollipopStart, .lollipop { + fill: ${t.mainBkg} !important; + stroke: ${t.lineColor} !important; + stroke-width: 1; +} + +#lollipopEnd, .lollipop { + fill: ${t.mainBkg} !important; + stroke: ${t.lineColor} !important; + stroke-width: 1; +} + +.edgeTerminals { + font-size: 11px; + line-height: initial; +} + +.classTitleText { + text-anchor: middle; + font-size: 18px; + fill: ${t.textColor}; +} +`,"getStyles"),N6=UVe});var HVe,WVe,qVe,M6,BO=N(()=>{"use strict";zt();vt();gm();Yd();$m();ir();HVe=o((t,e="TB")=>{if(!t.doc)return e;let r=e;for(let n of t.doc)n.stmt==="dir"&&(r=n.value);return r},"getDir"),WVe=o(function(t,e){return e.db.getClasses()},"getClasses"),qVe=o(async function(t,e,r,n){Y.info("REF0:"),Y.info("Drawing class diagram (v3)",e);let{securityLevel:i,state:a,layout:s}=me(),l=n.db.getData(),u=yc(e,i);l.type=n.type,l.layoutAlgorithm=nf(s),l.nodeSpacing=a?.nodeSpacing||50,l.rankSpacing=a?.rankSpacing||50,l.markers=["aggregation","extension","composition","dependency","lollipop"],l.diagramId=e,await Cc(l,u);let h=8;Gt.insertTitle(u,"classDiagramTitleText",a?.titleTopMargin??25,n.db.getDiagramTitle()),Ac(u,h,"classDiagram",a?.useMaxWidth??!0)},"draw"),M6={getClasses:WVe,draw:qVe,getDir:HVe}});var Nfe={};hr(Nfe,{diagram:()=>YVe});var YVe,Mfe=N(()=>{"use strict";IO();OO();PO();BO();YVe={parser:L6,get db(){return new D1},renderer:M6,styles:N6,init:o(t=>{t.class||(t.class={}),t.class.arrowMarkerAbsolute=t.arrowMarkerAbsolute},"init")}});var Pfe={};hr(Pfe,{diagram:()=>QVe});var QVe,Bfe=N(()=>{"use strict";IO();OO();PO();BO();QVe={parser:L6,get db(){return new D1},renderer:M6,styles:N6,init:o(t=>{t.class||(t.class={}),t.class.arrowMarkerAbsolute=t.arrowMarkerAbsolute},"init")}});var FO,I6,$O=N(()=>{"use strict";FO=function(){var t=o(function(F,P,z,$){for(z=z||{},$=F.length;$--;z[F[$]]=P);return z},"o"),e=[1,2],r=[1,3],n=[1,4],i=[2,4],a=[1,9],s=[1,11],l=[1,16],u=[1,17],h=[1,18],f=[1,19],d=[1,32],p=[1,20],m=[1,21],g=[1,22],y=[1,23],v=[1,24],x=[1,26],b=[1,27],w=[1,28],C=[1,29],T=[1,30],E=[1,31],A=[1,34],S=[1,35],_=[1,36],I=[1,37],D=[1,33],k=[1,4,5,16,17,19,21,22,24,25,26,27,28,29,33,35,37,38,42,45,48,49,50,51,54],L=[1,4,5,14,15,16,17,19,21,22,24,25,26,27,28,29,33,35,37,38,42,45,48,49,50,51,54],R=[4,5,16,17,19,21,22,24,25,26,27,28,29,33,35,37,38,42,45,48,49,50,51,54],O={trace:o(function(){},"trace"),yy:{},symbols_:{error:2,start:3,SPACE:4,NL:5,SD:6,document:7,line:8,statement:9,classDefStatement:10,styleStatement:11,cssClassStatement:12,idStatement:13,DESCR:14,"-->":15,HIDE_EMPTY:16,scale:17,WIDTH:18,COMPOSIT_STATE:19,STRUCT_START:20,STRUCT_STOP:21,STATE_DESCR:22,AS:23,ID:24,FORK:25,JOIN:26,CHOICE:27,CONCURRENT:28,note:29,notePosition:30,NOTE_TEXT:31,direction:32,acc_title:33,acc_title_value:34,acc_descr:35,acc_descr_value:36,acc_descr_multiline_value:37,classDef:38,CLASSDEF_ID:39,CLASSDEF_STYLEOPTS:40,DEFAULT:41,style:42,STYLE_IDS:43,STYLEDEF_STYLEOPTS:44,class:45,CLASSENTITY_IDS:46,STYLECLASS:47,direction_tb:48,direction_bt:49,direction_rl:50,direction_lr:51,eol:52,";":53,EDGE_STATE:54,STYLE_SEPARATOR:55,left_of:56,right_of:57,$accept:0,$end:1},terminals_:{2:"error",4:"SPACE",5:"NL",6:"SD",14:"DESCR",15:"-->",16:"HIDE_EMPTY",17:"scale",18:"WIDTH",19:"COMPOSIT_STATE",20:"STRUCT_START",21:"STRUCT_STOP",22:"STATE_DESCR",23:"AS",24:"ID",25:"FORK",26:"JOIN",27:"CHOICE",28:"CONCURRENT",29:"note",31:"NOTE_TEXT",33:"acc_title",34:"acc_title_value",35:"acc_descr",36:"acc_descr_value",37:"acc_descr_multiline_value",38:"classDef",39:"CLASSDEF_ID",40:"CLASSDEF_STYLEOPTS",41:"DEFAULT",42:"style",43:"STYLE_IDS",44:"STYLEDEF_STYLEOPTS",45:"class",46:"CLASSENTITY_IDS",47:"STYLECLASS",48:"direction_tb",49:"direction_bt",50:"direction_rl",51:"direction_lr",53:";",54:"EDGE_STATE",55:"STYLE_SEPARATOR",56:"left_of",57:"right_of"},productions_:[0,[3,2],[3,2],[3,2],[7,0],[7,2],[8,2],[8,1],[8,1],[9,1],[9,1],[9,1],[9,1],[9,2],[9,3],[9,4],[9,1],[9,2],[9,1],[9,4],[9,3],[9,6],[9,1],[9,1],[9,1],[9,1],[9,4],[9,4],[9,1],[9,2],[9,2],[9,1],[10,3],[10,3],[11,3],[12,3],[32,1],[32,1],[32,1],[32,1],[52,1],[52,1],[13,1],[13,1],[13,3],[13,3],[30,1],[30,1]],performAction:o(function(P,z,$,H,Q,j,ie){var ne=j.length-1;switch(Q){case 3:return H.setRootDoc(j[ne]),j[ne];break;case 4:this.$=[];break;case 5:j[ne]!="nl"&&(j[ne-1].push(j[ne]),this.$=j[ne-1]);break;case 6:case 7:this.$=j[ne];break;case 8:this.$="nl";break;case 12:this.$=j[ne];break;case 13:let X=j[ne-1];X.description=H.trimColon(j[ne]),this.$=X;break;case 14:this.$={stmt:"relation",state1:j[ne-2],state2:j[ne]};break;case 15:let te=H.trimColon(j[ne]);this.$={stmt:"relation",state1:j[ne-3],state2:j[ne-1],description:te};break;case 19:this.$={stmt:"state",id:j[ne-3],type:"default",description:"",doc:j[ne-1]};break;case 20:var le=j[ne],he=j[ne-2].trim();if(j[ne].match(":")){var K=j[ne].split(":");le=K[0],he=[he,K[1]]}this.$={stmt:"state",id:le,type:"default",description:he};break;case 21:this.$={stmt:"state",id:j[ne-3],type:"default",description:j[ne-5],doc:j[ne-1]};break;case 22:this.$={stmt:"state",id:j[ne],type:"fork"};break;case 23:this.$={stmt:"state",id:j[ne],type:"join"};break;case 24:this.$={stmt:"state",id:j[ne],type:"choice"};break;case 25:this.$={stmt:"state",id:H.getDividerId(),type:"divider"};break;case 26:this.$={stmt:"state",id:j[ne-1].trim(),note:{position:j[ne-2].trim(),text:j[ne].trim()}};break;case 29:this.$=j[ne].trim(),H.setAccTitle(this.$);break;case 30:case 31:this.$=j[ne].trim(),H.setAccDescription(this.$);break;case 32:case 33:this.$={stmt:"classDef",id:j[ne-1].trim(),classes:j[ne].trim()};break;case 34:this.$={stmt:"style",id:j[ne-1].trim(),styleClass:j[ne].trim()};break;case 35:this.$={stmt:"applyClass",id:j[ne-1].trim(),styleClass:j[ne].trim()};break;case 36:H.setDirection("TB"),this.$={stmt:"dir",value:"TB"};break;case 37:H.setDirection("BT"),this.$={stmt:"dir",value:"BT"};break;case 38:H.setDirection("RL"),this.$={stmt:"dir",value:"RL"};break;case 39:H.setDirection("LR"),this.$={stmt:"dir",value:"LR"};break;case 42:case 43:this.$={stmt:"state",id:j[ne].trim(),type:"default",description:""};break;case 44:this.$={stmt:"state",id:j[ne-2].trim(),classes:[j[ne].trim()],type:"default",description:""};break;case 45:this.$={stmt:"state",id:j[ne-2].trim(),classes:[j[ne].trim()],type:"default",description:""};break}},"anonymous"),table:[{3:1,4:e,5:r,6:n},{1:[3]},{3:5,4:e,5:r,6:n},{3:6,4:e,5:r,6:n},t([1,4,5,16,17,19,22,24,25,26,27,28,29,33,35,37,38,42,45,48,49,50,51,54],i,{7:7}),{1:[2,1]},{1:[2,2]},{1:[2,3],4:a,5:s,8:8,9:10,10:12,11:13,12:14,13:15,16:l,17:u,19:h,22:f,24:d,25:p,26:m,27:g,28:y,29:v,32:25,33:x,35:b,37:w,38:C,42:T,45:E,48:A,49:S,50:_,51:I,54:D},t(k,[2,5]),{9:38,10:12,11:13,12:14,13:15,16:l,17:u,19:h,22:f,24:d,25:p,26:m,27:g,28:y,29:v,32:25,33:x,35:b,37:w,38:C,42:T,45:E,48:A,49:S,50:_,51:I,54:D},t(k,[2,7]),t(k,[2,8]),t(k,[2,9]),t(k,[2,10]),t(k,[2,11]),t(k,[2,12],{14:[1,39],15:[1,40]}),t(k,[2,16]),{18:[1,41]},t(k,[2,18],{20:[1,42]}),{23:[1,43]},t(k,[2,22]),t(k,[2,23]),t(k,[2,24]),t(k,[2,25]),{30:44,31:[1,45],56:[1,46],57:[1,47]},t(k,[2,28]),{34:[1,48]},{36:[1,49]},t(k,[2,31]),{39:[1,50],41:[1,51]},{43:[1,52]},{46:[1,53]},t(L,[2,42],{55:[1,54]}),t(L,[2,43],{55:[1,55]}),t(k,[2,36]),t(k,[2,37]),t(k,[2,38]),t(k,[2,39]),t(k,[2,6]),t(k,[2,13]),{13:56,24:d,54:D},t(k,[2,17]),t(R,i,{7:57}),{24:[1,58]},{24:[1,59]},{23:[1,60]},{24:[2,46]},{24:[2,47]},t(k,[2,29]),t(k,[2,30]),{40:[1,61]},{40:[1,62]},{44:[1,63]},{47:[1,64]},{24:[1,65]},{24:[1,66]},t(k,[2,14],{14:[1,67]}),{4:a,5:s,8:8,9:10,10:12,11:13,12:14,13:15,16:l,17:u,19:h,21:[1,68],22:f,24:d,25:p,26:m,27:g,28:y,29:v,32:25,33:x,35:b,37:w,38:C,42:T,45:E,48:A,49:S,50:_,51:I,54:D},t(k,[2,20],{20:[1,69]}),{31:[1,70]},{24:[1,71]},t(k,[2,32]),t(k,[2,33]),t(k,[2,34]),t(k,[2,35]),t(L,[2,44]),t(L,[2,45]),t(k,[2,15]),t(k,[2,19]),t(R,i,{7:72}),t(k,[2,26]),t(k,[2,27]),{4:a,5:s,8:8,9:10,10:12,11:13,12:14,13:15,16:l,17:u,19:h,21:[1,73],22:f,24:d,25:p,26:m,27:g,28:y,29:v,32:25,33:x,35:b,37:w,38:C,42:T,45:E,48:A,49:S,50:_,51:I,54:D},t(k,[2,21])],defaultActions:{5:[2,1],6:[2,2],46:[2,46],47:[2,47]},parseError:o(function(P,z){if(z.recoverable)this.trace(P);else{var $=new Error(P);throw $.hash=z,$}},"parseError"),parse:o(function(P){var z=this,$=[0],H=[],Q=[null],j=[],ie=this.table,ne="",le=0,he=0,K=0,X=2,te=1,J=j.slice.call(arguments,1),se=Object.create(this.lexer),ue={yy:{}};for(var Z in this.yy)Object.prototype.hasOwnProperty.call(this.yy,Z)&&(ue.yy[Z]=this.yy[Z]);se.setInput(P,ue.yy),ue.yy.lexer=se,ue.yy.parser=this,typeof se.yylloc>"u"&&(se.yylloc={});var Se=se.yylloc;j.push(Se);var ce=se.options&&se.options.ranges;typeof ue.yy.parseError=="function"?this.parseError=ue.yy.parseError:this.parseError=Object.getPrototypeOf(this).parseError;function ae(xe){$.length=$.length-2*xe,Q.length=Q.length-xe,j.length=j.length-xe}o(ae,"popStack");function Oe(){var xe;return xe=H.pop()||se.lex()||te,typeof xe!="number"&&(xe instanceof Array&&(H=xe,xe=H.pop()),xe=z.symbols_[xe]||xe),xe}o(Oe,"lex");for(var ge,ze,He,$e,Re,Ie,be={},W,de,re,oe;;){if(He=$[$.length-1],this.defaultActions[He]?$e=this.defaultActions[He]:((ge===null||typeof ge>"u")&&(ge=Oe()),$e=ie[He]&&ie[He][ge]),typeof $e>"u"||!$e.length||!$e[0]){var V="";oe=[];for(W in ie[He])this.terminals_[W]&&W>X&&oe.push("'"+this.terminals_[W]+"'");se.showPosition?V="Parse error on line "+(le+1)+`: +`+se.showPosition()+` +Expecting `+oe.join(", ")+", got '"+(this.terminals_[ge]||ge)+"'":V="Parse error on line "+(le+1)+": Unexpected "+(ge==te?"end of input":"'"+(this.terminals_[ge]||ge)+"'"),this.parseError(V,{text:se.match,token:this.terminals_[ge]||ge,line:se.yylineno,loc:Se,expected:oe})}if($e[0]instanceof Array&&$e.length>1)throw new Error("Parse Error: multiple actions possible at state: "+He+", token: "+ge);switch($e[0]){case 1:$.push(ge),Q.push(se.yytext),j.push(se.yylloc),$.push($e[1]),ge=null,ze?(ge=ze,ze=null):(he=se.yyleng,ne=se.yytext,le=se.yylineno,Se=se.yylloc,K>0&&K--);break;case 2:if(de=this.productions_[$e[1]][1],be.$=Q[Q.length-de],be._$={first_line:j[j.length-(de||1)].first_line,last_line:j[j.length-1].last_line,first_column:j[j.length-(de||1)].first_column,last_column:j[j.length-1].last_column},ce&&(be._$.range=[j[j.length-(de||1)].range[0],j[j.length-1].range[1]]),Ie=this.performAction.apply(be,[ne,he,le,ue.yy,$e[1],Q,j].concat(J)),typeof Ie<"u")return Ie;de&&($=$.slice(0,-1*de*2),Q=Q.slice(0,-1*de),j=j.slice(0,-1*de)),$.push(this.productions_[$e[1]][0]),Q.push(be.$),j.push(be._$),re=ie[$[$.length-2]][$[$.length-1]],$.push(re);break;case 3:return!0}}return!0},"parse")},M=function(){var F={EOF:1,parseError:o(function(z,$){if(this.yy.parser)this.yy.parser.parseError(z,$);else throw new Error(z)},"parseError"),setInput:o(function(P,z){return this.yy=z||this.yy||{},this._input=P,this._more=this._backtrack=this.done=!1,this.yylineno=this.yyleng=0,this.yytext=this.matched=this.match="",this.conditionStack=["INITIAL"],this.yylloc={first_line:1,first_column:0,last_line:1,last_column:0},this.options.ranges&&(this.yylloc.range=[0,0]),this.offset=0,this},"setInput"),input:o(function(){var P=this._input[0];this.yytext+=P,this.yyleng++,this.offset++,this.match+=P,this.matched+=P;var z=P.match(/(?:\r\n?|\n).*/g);return z?(this.yylineno++,this.yylloc.last_line++):this.yylloc.last_column++,this.options.ranges&&this.yylloc.range[1]++,this._input=this._input.slice(1),P},"input"),unput:o(function(P){var z=P.length,$=P.split(/(?:\r\n?|\n)/g);this._input=P+this._input,this.yytext=this.yytext.substr(0,this.yytext.length-z),this.offset-=z;var H=this.match.split(/(?:\r\n?|\n)/g);this.match=this.match.substr(0,this.match.length-1),this.matched=this.matched.substr(0,this.matched.length-1),$.length-1&&(this.yylineno-=$.length-1);var Q=this.yylloc.range;return this.yylloc={first_line:this.yylloc.first_line,last_line:this.yylineno+1,first_column:this.yylloc.first_column,last_column:$?($.length===H.length?this.yylloc.first_column:0)+H[H.length-$.length].length-$[0].length:this.yylloc.first_column-z},this.options.ranges&&(this.yylloc.range=[Q[0],Q[0]+this.yyleng-z]),this.yyleng=this.yytext.length,this},"unput"),more:o(function(){return this._more=!0,this},"more"),reject:o(function(){if(this.options.backtrack_lexer)this._backtrack=!0;else return this.parseError("Lexical error on line "+(this.yylineno+1)+`. You can only invoke reject() in the lexer when the lexer is of the backtracking persuasion (options.backtrack_lexer = true). +`+this.showPosition(),{text:"",token:null,line:this.yylineno});return this},"reject"),less:o(function(P){this.unput(this.match.slice(P))},"less"),pastInput:o(function(){var P=this.matched.substr(0,this.matched.length-this.match.length);return(P.length>20?"...":"")+P.substr(-20).replace(/\n/g,"")},"pastInput"),upcomingInput:o(function(){var P=this.match;return P.length<20&&(P+=this._input.substr(0,20-P.length)),(P.substr(0,20)+(P.length>20?"...":"")).replace(/\n/g,"")},"upcomingInput"),showPosition:o(function(){var P=this.pastInput(),z=new Array(P.length+1).join("-");return P+this.upcomingInput()+` +`+z+"^"},"showPosition"),test_match:o(function(P,z){var $,H,Q;if(this.options.backtrack_lexer&&(Q={yylineno:this.yylineno,yylloc:{first_line:this.yylloc.first_line,last_line:this.last_line,first_column:this.yylloc.first_column,last_column:this.yylloc.last_column},yytext:this.yytext,match:this.match,matches:this.matches,matched:this.matched,yyleng:this.yyleng,offset:this.offset,_more:this._more,_input:this._input,yy:this.yy,conditionStack:this.conditionStack.slice(0),done:this.done},this.options.ranges&&(Q.yylloc.range=this.yylloc.range.slice(0))),H=P[0].match(/(?:\r\n?|\n).*/g),H&&(this.yylineno+=H.length),this.yylloc={first_line:this.yylloc.last_line,last_line:this.yylineno+1,first_column:this.yylloc.last_column,last_column:H?H[H.length-1].length-H[H.length-1].match(/\r?\n?/)[0].length:this.yylloc.last_column+P[0].length},this.yytext+=P[0],this.match+=P[0],this.matches=P,this.yyleng=this.yytext.length,this.options.ranges&&(this.yylloc.range=[this.offset,this.offset+=this.yyleng]),this._more=!1,this._backtrack=!1,this._input=this._input.slice(P[0].length),this.matched+=P[0],$=this.performAction.call(this,this.yy,this,z,this.conditionStack[this.conditionStack.length-1]),this.done&&this._input&&(this.done=!1),$)return $;if(this._backtrack){for(var j in Q)this[j]=Q[j];return!1}return!1},"test_match"),next:o(function(){if(this.done)return this.EOF;this._input||(this.done=!0);var P,z,$,H;this._more||(this.yytext="",this.match="");for(var Q=this._currentRules(),j=0;jz[0].length)){if(z=$,H=j,this.options.backtrack_lexer){if(P=this.test_match($,Q[j]),P!==!1)return P;if(this._backtrack){z=!1;continue}else return!1}else if(!this.options.flex)break}return z?(P=this.test_match(z,Q[H]),P!==!1?P:!1):this._input===""?this.EOF:this.parseError("Lexical error on line "+(this.yylineno+1)+`. Unrecognized text. +`+this.showPosition(),{text:"",token:null,line:this.yylineno})},"next"),lex:o(function(){var z=this.next();return z||this.lex()},"lex"),begin:o(function(z){this.conditionStack.push(z)},"begin"),popState:o(function(){var z=this.conditionStack.length-1;return z>0?this.conditionStack.pop():this.conditionStack[0]},"popState"),_currentRules:o(function(){return this.conditionStack.length&&this.conditionStack[this.conditionStack.length-1]?this.conditions[this.conditionStack[this.conditionStack.length-1]].rules:this.conditions.INITIAL.rules},"_currentRules"),topState:o(function(z){return z=this.conditionStack.length-1-Math.abs(z||0),z>=0?this.conditionStack[z]:"INITIAL"},"topState"),pushState:o(function(z){this.begin(z)},"pushState"),stateStackSize:o(function(){return this.conditionStack.length},"stateStackSize"),options:{"case-insensitive":!0},performAction:o(function(z,$,H,Q){var j=Q;switch(H){case 0:return 41;case 1:return 48;case 2:return 49;case 3:return 50;case 4:return 51;case 5:break;case 6:break;case 7:return 5;case 8:break;case 9:break;case 10:break;case 11:break;case 12:return this.pushState("SCALE"),17;break;case 13:return 18;case 14:this.popState();break;case 15:return this.begin("acc_title"),33;break;case 16:return this.popState(),"acc_title_value";break;case 17:return this.begin("acc_descr"),35;break;case 18:return this.popState(),"acc_descr_value";break;case 19:this.begin("acc_descr_multiline");break;case 20:this.popState();break;case 21:return"acc_descr_multiline_value";case 22:return this.pushState("CLASSDEF"),38;break;case 23:return this.popState(),this.pushState("CLASSDEFID"),"DEFAULT_CLASSDEF_ID";break;case 24:return this.popState(),this.pushState("CLASSDEFID"),39;break;case 25:return this.popState(),40;break;case 26:return this.pushState("CLASS"),45;break;case 27:return this.popState(),this.pushState("CLASS_STYLE"),46;break;case 28:return this.popState(),47;break;case 29:return this.pushState("STYLE"),42;break;case 30:return this.popState(),this.pushState("STYLEDEF_STYLES"),43;break;case 31:return this.popState(),44;break;case 32:return this.pushState("SCALE"),17;break;case 33:return 18;case 34:this.popState();break;case 35:this.pushState("STATE");break;case 36:return this.popState(),$.yytext=$.yytext.slice(0,-8).trim(),25;break;case 37:return this.popState(),$.yytext=$.yytext.slice(0,-8).trim(),26;break;case 38:return this.popState(),$.yytext=$.yytext.slice(0,-10).trim(),27;break;case 39:return this.popState(),$.yytext=$.yytext.slice(0,-8).trim(),25;break;case 40:return this.popState(),$.yytext=$.yytext.slice(0,-8).trim(),26;break;case 41:return this.popState(),$.yytext=$.yytext.slice(0,-10).trim(),27;break;case 42:return 48;case 43:return 49;case 44:return 50;case 45:return 51;case 46:this.pushState("STATE_STRING");break;case 47:return this.pushState("STATE_ID"),"AS";break;case 48:return this.popState(),"ID";break;case 49:this.popState();break;case 50:return"STATE_DESCR";case 51:return 19;case 52:this.popState();break;case 53:return this.popState(),this.pushState("struct"),20;break;case 54:break;case 55:return this.popState(),21;break;case 56:break;case 57:return this.begin("NOTE"),29;break;case 58:return this.popState(),this.pushState("NOTE_ID"),56;break;case 59:return this.popState(),this.pushState("NOTE_ID"),57;break;case 60:this.popState(),this.pushState("FLOATING_NOTE");break;case 61:return this.popState(),this.pushState("FLOATING_NOTE_ID"),"AS";break;case 62:break;case 63:return"NOTE_TEXT";case 64:return this.popState(),"ID";break;case 65:return this.popState(),this.pushState("NOTE_TEXT"),24;break;case 66:return this.popState(),$.yytext=$.yytext.substr(2).trim(),31;break;case 67:return this.popState(),$.yytext=$.yytext.slice(0,-8).trim(),31;break;case 68:return 6;case 69:return 6;case 70:return 16;case 71:return 54;case 72:return 24;case 73:return $.yytext=$.yytext.trim(),14;break;case 74:return 15;case 75:return 28;case 76:return 55;case 77:return 5;case 78:return"INVALID"}},"anonymous"),rules:[/^(?:default\b)/i,/^(?:.*direction\s+TB[^\n]*)/i,/^(?:.*direction\s+BT[^\n]*)/i,/^(?:.*direction\s+RL[^\n]*)/i,/^(?:.*direction\s+LR[^\n]*)/i,/^(?:%%(?!\{)[^\n]*)/i,/^(?:[^\}]%%[^\n]*)/i,/^(?:[\n]+)/i,/^(?:[\s]+)/i,/^(?:((?!\n)\s)+)/i,/^(?:#[^\n]*)/i,/^(?:%[^\n]*)/i,/^(?:scale\s+)/i,/^(?:\d+)/i,/^(?:\s+width\b)/i,/^(?:accTitle\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*\{\s*)/i,/^(?:[\}])/i,/^(?:[^\}]*)/i,/^(?:classDef\s+)/i,/^(?:DEFAULT\s+)/i,/^(?:\w+\s+)/i,/^(?:[^\n]*)/i,/^(?:class\s+)/i,/^(?:(\w+)+((,\s*\w+)*))/i,/^(?:[^\n]*)/i,/^(?:style\s+)/i,/^(?:[\w,]+\s+)/i,/^(?:[^\n]*)/i,/^(?:scale\s+)/i,/^(?:\d+)/i,/^(?:\s+width\b)/i,/^(?:state\s+)/i,/^(?:.*<>)/i,/^(?:.*<>)/i,/^(?:.*<>)/i,/^(?:.*\[\[fork\]\])/i,/^(?:.*\[\[join\]\])/i,/^(?:.*\[\[choice\]\])/i,/^(?:.*direction\s+TB[^\n]*)/i,/^(?:.*direction\s+BT[^\n]*)/i,/^(?:.*direction\s+RL[^\n]*)/i,/^(?:.*direction\s+LR[^\n]*)/i,/^(?:["])/i,/^(?:\s*as\s+)/i,/^(?:[^\n\{]*)/i,/^(?:["])/i,/^(?:[^"]*)/i,/^(?:[^\n\s\{]+)/i,/^(?:\n)/i,/^(?:\{)/i,/^(?:%%(?!\{)[^\n]*)/i,/^(?:\})/i,/^(?:[\n])/i,/^(?:note\s+)/i,/^(?:left of\b)/i,/^(?:right of\b)/i,/^(?:")/i,/^(?:\s*as\s*)/i,/^(?:["])/i,/^(?:[^"]*)/i,/^(?:[^\n]*)/i,/^(?:\s*[^:\n\s\-]+)/i,/^(?:\s*:[^:\n;]+)/i,/^(?:[\s\S]*?end note\b)/i,/^(?:stateDiagram\s+)/i,/^(?:stateDiagram-v2\s+)/i,/^(?:hide empty description\b)/i,/^(?:\[\*\])/i,/^(?:[^:\n\s\-\{]+)/i,/^(?:\s*:[^:\n;]+)/i,/^(?:-->)/i,/^(?:--)/i,/^(?::::)/i,/^(?:$)/i,/^(?:.)/i],conditions:{LINE:{rules:[9,10],inclusive:!1},struct:{rules:[9,10,22,26,29,35,42,43,44,45,54,55,56,57,71,72,73,74,75],inclusive:!1},FLOATING_NOTE_ID:{rules:[64],inclusive:!1},FLOATING_NOTE:{rules:[61,62,63],inclusive:!1},NOTE_TEXT:{rules:[66,67],inclusive:!1},NOTE_ID:{rules:[65],inclusive:!1},NOTE:{rules:[58,59,60],inclusive:!1},STYLEDEF_STYLEOPTS:{rules:[],inclusive:!1},STYLEDEF_STYLES:{rules:[31],inclusive:!1},STYLE_IDS:{rules:[],inclusive:!1},STYLE:{rules:[30],inclusive:!1},CLASS_STYLE:{rules:[28],inclusive:!1},CLASS:{rules:[27],inclusive:!1},CLASSDEFID:{rules:[25],inclusive:!1},CLASSDEF:{rules:[23,24],inclusive:!1},acc_descr_multiline:{rules:[20,21],inclusive:!1},acc_descr:{rules:[18],inclusive:!1},acc_title:{rules:[16],inclusive:!1},SCALE:{rules:[13,14,33,34],inclusive:!1},ALIAS:{rules:[],inclusive:!1},STATE_ID:{rules:[48],inclusive:!1},STATE_STRING:{rules:[49,50],inclusive:!1},FORK_STATE:{rules:[],inclusive:!1},STATE:{rules:[9,10,36,37,38,39,40,41,46,47,51,52,53],inclusive:!1},ID:{rules:[9,10],inclusive:!1},INITIAL:{rules:[0,1,2,3,4,5,6,7,8,10,11,12,15,17,19,22,26,29,32,35,53,57,68,69,70,71,72,73,74,76,77,78],inclusive:!0}}};return F}();O.lexer=M;function B(){this.yy={}}return o(B,"Parser"),B.prototype=O,O.Parser=B,new B}();FO.parser=FO;I6=FO});var zfe,O6,zO,L1,Eb,Gfe,Vfe,Ufe,Rp,P6,GO,VO,UO,HO,WO,B6,F6,Hfe,Wfe,qO,YO,qfe,Yfe,R1,tUe,Xfe,XO,rUe,nUe,jfe,Kfe,iUe,Qfe,aUe,Zfe,jO,KO,Jfe,$6,ede,QO,z6=N(()=>{"use strict";zfe="TB",O6="TB",zO="dir",L1="state",Eb="relation",Gfe="classDef",Vfe="style",Ufe="applyClass",Rp="default",P6="divider",GO="fill:none",VO="fill: #333",UO="c",HO="text",WO="normal",B6="rect",F6="rectWithTitle",Hfe="stateStart",Wfe="stateEnd",qO="divider",YO="roundedWithTitle",qfe="note",Yfe="noteGroup",R1="statediagram",tUe="state",Xfe=`${R1}-${tUe}`,XO="transition",rUe="note",nUe="note-edge",jfe=`${XO} ${nUe}`,Kfe=`${R1}-${rUe}`,iUe="cluster",Qfe=`${R1}-${iUe}`,aUe="cluster-alt",Zfe=`${R1}-${aUe}`,jO="parent",KO="note",Jfe="state",$6="----",ede=`${$6}${KO}`,QO=`${$6}${jO}`});function ZO(t="",e=0,r="",n=$6){let i=r!==null&&r.length>0?`${n}${r}`:"";return`${Jfe}-${t}${i}-${e}`}function G6(t,e,r){if(!e.id||e.id===""||e.id==="")return;e.cssClasses&&(Array.isArray(e.cssCompiledStyles)||(e.cssCompiledStyles=[]),e.cssClasses.split(" ").forEach(i=>{if(r.get(i)){let a=r.get(i);e.cssCompiledStyles=[...e.cssCompiledStyles,...a.styles]}}));let n=t.find(i=>i.id===e.id);n?Object.assign(n,e):t.push(e)}function oUe(t){return t?.classes?.join(" ")??""}function lUe(t){return t?.styles??[]}var V6,xf,sUe,tde,N1,rde,nde=N(()=>{"use strict";zt();vt();gr();z6();V6=new Map,xf=0;o(ZO,"stateDomId");sUe=o((t,e,r,n,i,a,s,l)=>{Y.trace("items",e),e.forEach(u=>{switch(u.stmt){case L1:N1(t,u,r,n,i,a,s,l);break;case Rp:N1(t,u,r,n,i,a,s,l);break;case Eb:{N1(t,u.state1,r,n,i,a,s,l),N1(t,u.state2,r,n,i,a,s,l);let h={id:"edge"+xf,start:u.state1.id,end:u.state2.id,arrowhead:"normal",arrowTypeEnd:"arrow_barb",style:GO,labelStyle:"",label:Ze.sanitizeText(u.description,me()),arrowheadStyle:VO,labelpos:UO,labelType:HO,thickness:WO,classes:XO,look:s};i.push(h),xf++}break}})},"setupDoc"),tde=o((t,e=O6)=>{let r=e;if(t.doc)for(let n of t.doc)n.stmt==="dir"&&(r=n.value);return r},"getDir");o(G6,"insertOrUpdateNode");o(oUe,"getClassesFromDbInfo");o(lUe,"getStylesFromDbInfo");N1=o((t,e,r,n,i,a,s,l)=>{let u=e.id,h=r.get(u),f=oUe(h),d=lUe(h);if(Y.info("dataFetcher parsedItem",e,h,d),u!=="root"){let p=B6;e.start===!0?p=Hfe:e.start===!1&&(p=Wfe),e.type!==Rp&&(p=e.type),V6.get(u)||V6.set(u,{id:u,shape:p,description:Ze.sanitizeText(u,me()),cssClasses:`${f} ${Xfe}`,cssStyles:d});let m=V6.get(u);e.description&&(Array.isArray(m.description)?(m.shape=F6,m.description.push(e.description)):m.description?.length>0?(m.shape=F6,m.description===u?m.description=[e.description]:m.description=[m.description,e.description]):(m.shape=B6,m.description=e.description),m.description=Ze.sanitizeTextOrArray(m.description,me())),m.description?.length===1&&m.shape===F6&&(m.type==="group"?m.shape=YO:m.shape=B6),!m.type&&e.doc&&(Y.info("Setting cluster for XCX",u,tde(e)),m.type="group",m.isGroup=!0,m.dir=tde(e),m.shape=e.type===P6?qO:YO,m.cssClasses=`${m.cssClasses} ${Qfe} ${a?Zfe:""}`);let g={labelStyle:"",shape:m.shape,label:m.description,cssClasses:m.cssClasses,cssCompiledStyles:[],cssStyles:m.cssStyles,id:u,dir:m.dir,domId:ZO(u,xf),type:m.type,isGroup:m.type==="group",padding:8,rx:10,ry:10,look:s};if(g.shape===qO&&(g.label=""),t&&t.id!=="root"&&(Y.trace("Setting node ",u," to be child of its parent ",t.id),g.parentId=t.id),g.centerLabel=!0,e.note){let y={labelStyle:"",shape:qfe,label:e.note.text,cssClasses:Kfe,cssStyles:[],cssCompilesStyles:[],id:u+ede+"-"+xf,domId:ZO(u,xf,KO),type:m.type,isGroup:m.type==="group",padding:me().flowchart.padding,look:s,position:e.note.position},v=u+QO,x={labelStyle:"",shape:Yfe,label:e.note.text,cssClasses:m.cssClasses,cssStyles:[],id:u+QO,domId:ZO(u,xf,jO),type:"group",isGroup:!0,padding:16,look:s,position:e.note.position};xf++,x.id=v,y.parentId=v,G6(n,x,l),G6(n,y,l),G6(n,g,l);let b=u,w=y.id;e.note.position==="left of"&&(b=y.id,w=u),i.push({id:b+"-"+w,start:b,end:w,arrowhead:"none",arrowTypeEnd:"",style:GO,labelStyle:"",classes:jfe,arrowheadStyle:VO,labelpos:UO,labelType:HO,thickness:WO,look:s})}else G6(n,g,l)}e.doc&&(Y.trace("Adding nodes children "),sUe(e,e.doc,r,n,i,!a,s,l))},"dataFetcher"),rde=o(()=>{V6.clear(),xf=0},"reset")});var JO,cUe,uUe,ide,eP=N(()=>{"use strict";zt();vt();gm();Yd();$m();ir();z6();JO=o((t,e=O6)=>{if(!t.doc)return e;let r=e;for(let n of t.doc)n.stmt==="dir"&&(r=n.value);return r},"getDir"),cUe=o(function(t,e){return e.db.getClasses()},"getClasses"),uUe=o(async function(t,e,r,n){Y.info("REF0:"),Y.info("Drawing state diagram (v2)",e);let{securityLevel:i,state:a,layout:s}=me();n.db.extract(n.db.getRootDocV2());let l=n.db.getData(),u=yc(e,i);l.type=n.type,l.layoutAlgorithm=s,l.nodeSpacing=a?.nodeSpacing||50,l.rankSpacing=a?.rankSpacing||50,l.markers=["barb"],l.diagramId=e,await Cc(l,u);let h=8;Gt.insertTitle(u,"statediagramTitleText",a?.titleTopMargin??25,n.db.getDiagramTitle()),Ac(u,h,R1,a?.useMaxWidth??!0)},"draw"),ide={getClasses:cUe,draw:uUe,getDir:JO}});function ude(){return new Map}var tP,ade,sde,ode,lde,cde,hUe,fUe,hde,U6,Qo,H6=N(()=>{"use strict";zt();vt();ir();gr();mi();nde();eP();z6();tP="[*]",ade="start",sde=tP,ode="end",lde="color",cde="fill",hUe="bgFill",fUe=",";o(ude,"newClassesList");hde=o(()=>({relations:[],states:new Map,documents:{}}),"newDoc"),U6=o(t=>JSON.parse(JSON.stringify(t)),"clone"),Qo=class{static{o(this,"StateDB")}constructor(e){this.clear(),this.version=e,this.setRootDoc=this.setRootDoc.bind(this),this.getDividerId=this.getDividerId.bind(this),this.setDirection=this.setDirection.bind(this),this.trimColon=this.trimColon.bind(this)}version;nodes=[];edges=[];rootDoc=[];classes=ude();documents={root:hde()};currentDocument=this.documents.root;startEndCount=0;dividerCnt=0;static relationType={AGGREGATION:0,EXTENSION:1,COMPOSITION:2,DEPENDENCY:3};setRootDoc(e){Y.info("Setting root doc",e),this.rootDoc=e,this.version===1?this.extract(e):this.extract(this.getRootDocV2())}getRootDoc(){return this.rootDoc}docTranslator(e,r,n){if(r.stmt===Eb)this.docTranslator(e,r.state1,!0),this.docTranslator(e,r.state2,!1);else if(r.stmt===L1&&(r.id==="[*]"?(r.id=n?e.id+"_start":e.id+"_end",r.start=n):r.id=r.id.trim()),r.doc){let i=[],a=[],s;for(s=0;s0&&a.length>0){let l={stmt:L1,id:X9(),type:"divider",doc:U6(a)};i.push(U6(l)),r.doc=i}r.doc.forEach(l=>this.docTranslator(r,l,!0))}}getRootDocV2(){return this.docTranslator({id:"root"},{id:"root",doc:this.rootDoc},!0),{id:"root",doc:this.rootDoc}}extract(e){let r;e.doc?r=e.doc:r=e,Y.info(r),this.clear(!0),Y.info("Extract initial document:",r),r.forEach(s=>{switch(Y.warn("Statement",s.stmt),s.stmt){case L1:this.addState(s.id.trim(),s.type,s.doc,s.description,s.note,s.classes,s.styles,s.textStyles);break;case Eb:this.addRelation(s.state1,s.state2,s.description);break;case Gfe:this.addStyleClass(s.id.trim(),s.classes);break;case Vfe:{let l=s.id.trim().split(","),u=s.styleClass.split(",");l.forEach(h=>{let f=this.getState(h);if(f===void 0){let d=h.trim();this.addState(d),f=this.getState(d)}f.styles=u.map(d=>d.replace(/;/g,"")?.trim())})}break;case Ufe:this.setCssClass(s.id.trim(),s.styleClass);break}});let n=this.getStates(),a=me().look;rde(),N1(void 0,this.getRootDocV2(),n,this.nodes,this.edges,!0,a,this.classes),this.nodes.forEach(s=>{if(Array.isArray(s.label)){if(s.description=s.label.slice(1),s.isGroup&&s.description.length>0)throw new Error("Group nodes can only have label. Remove the additional description for node ["+s.id+"]");s.label=s.label[0]}})}addState(e,r=Rp,n=null,i=null,a=null,s=null,l=null,u=null){let h=e?.trim();if(this.currentDocument.states.has(h)?(this.currentDocument.states.get(h).doc||(this.currentDocument.states.get(h).doc=n),this.currentDocument.states.get(h).type||(this.currentDocument.states.get(h).type=r)):(Y.info("Adding state ",h,i),this.currentDocument.states.set(h,{id:h,descriptions:[],type:r,doc:n,note:a,classes:[],styles:[],textStyles:[]})),i&&(Y.info("Setting state description",h,i),typeof i=="string"&&this.addDescription(h,i.trim()),typeof i=="object"&&i.forEach(f=>this.addDescription(h,f.trim()))),a){let f=this.currentDocument.states.get(h);f.note=a,f.note.text=Ze.sanitizeText(f.note.text,me())}s&&(Y.info("Setting state classes",h,s),(typeof s=="string"?[s]:s).forEach(d=>this.setCssClass(h,d.trim()))),l&&(Y.info("Setting state styles",h,l),(typeof l=="string"?[l]:l).forEach(d=>this.setStyle(h,d.trim()))),u&&(Y.info("Setting state styles",h,l),(typeof u=="string"?[u]:u).forEach(d=>this.setTextStyle(h,d.trim())))}clear(e){this.nodes=[],this.edges=[],this.documents={root:hde()},this.currentDocument=this.documents.root,this.startEndCount=0,this.classes=ude(),e||Ar()}getState(e){return this.currentDocument.states.get(e)}getStates(){return this.currentDocument.states}logDocuments(){Y.info("Documents = ",this.documents)}getRelations(){return this.currentDocument.relations}startIdIfNeeded(e=""){let r=e;return e===tP&&(this.startEndCount++,r=`${ade}${this.startEndCount}`),r}startTypeIfNeeded(e="",r=Rp){return e===tP?ade:r}endIdIfNeeded(e=""){let r=e;return e===sde&&(this.startEndCount++,r=`${ode}${this.startEndCount}`),r}endTypeIfNeeded(e="",r=Rp){return e===sde?ode:r}addRelationObjs(e,r,n){let i=this.startIdIfNeeded(e.id.trim()),a=this.startTypeIfNeeded(e.id.trim(),e.type),s=this.startIdIfNeeded(r.id.trim()),l=this.startTypeIfNeeded(r.id.trim(),r.type);this.addState(i,a,e.doc,e.description,e.note,e.classes,e.styles,e.textStyles),this.addState(s,l,r.doc,r.description,r.note,r.classes,r.styles,r.textStyles),this.currentDocument.relations.push({id1:i,id2:s,relationTitle:Ze.sanitizeText(n,me())})}addRelation(e,r,n){if(typeof e=="object")this.addRelationObjs(e,r,n);else{let i=this.startIdIfNeeded(e.trim()),a=this.startTypeIfNeeded(e),s=this.endIdIfNeeded(r.trim()),l=this.endTypeIfNeeded(r);this.addState(i,a),this.addState(s,l),this.currentDocument.relations.push({id1:i,id2:s,title:Ze.sanitizeText(n,me())})}}addDescription(e,r){let n=this.currentDocument.states.get(e),i=r.startsWith(":")?r.replace(":","").trim():r;n.descriptions.push(Ze.sanitizeText(i,me()))}cleanupLabel(e){return e.substring(0,1)===":"?e.substr(2).trim():e.trim()}getDividerId(){return this.dividerCnt++,"divider-id-"+this.dividerCnt}addStyleClass(e,r=""){this.classes.has(e)||this.classes.set(e,{id:e,styles:[],textStyles:[]});let n=this.classes.get(e);r?.split(fUe).forEach(i=>{let a=i.replace(/([^;]*);/,"$1").trim();if(RegExp(lde).exec(i)){let l=a.replace(cde,hUe).replace(lde,cde);n.textStyles.push(l)}n.styles.push(a)})}getClasses(){return this.classes}setCssClass(e,r){e.split(",").forEach(n=>{let i=this.getState(n);if(i===void 0){let a=n.trim();this.addState(a),i=this.getState(a)}i.classes.push(r)})}setStyle(e,r){let n=this.getState(e);n!==void 0&&n.styles.push(r)}setTextStyle(e,r){let n=this.getState(e);n!==void 0&&n.textStyles.push(r)}getDirectionStatement(){return this.rootDoc.find(e=>e.stmt===zO)}getDirection(){return this.getDirectionStatement()?.value??zfe}setDirection(e){let r=this.getDirectionStatement();r?r.value=e:this.rootDoc.unshift({stmt:zO,value:e})}trimColon(e){return e&&e[0]===":"?e.substr(1).trim():e.trim()}getData(){let e=me();return{nodes:this.nodes,edges:this.edges,other:{},config:e,direction:JO(this.getRootDocV2())}}getConfig(){return me().state}getAccTitle=Rr;setAccTitle=Lr;getAccDescription=Mr;setAccDescription=Nr;setDiagramTitle=$r;getDiagramTitle=Ir}});var dUe,W6,rP=N(()=>{"use strict";dUe=o(t=>` +defs #statediagram-barbEnd { + fill: ${t.transitionColor}; + stroke: ${t.transitionColor}; + } +g.stateGroup text { + fill: ${t.nodeBorder}; + stroke: none; + font-size: 10px; +} +g.stateGroup text { + fill: ${t.textColor}; + stroke: none; + font-size: 10px; + +} +g.stateGroup .state-title { + font-weight: bolder; + fill: ${t.stateLabelColor}; +} + +g.stateGroup rect { + fill: ${t.mainBkg}; + stroke: ${t.nodeBorder}; +} + +g.stateGroup line { + stroke: ${t.lineColor}; + stroke-width: 1; +} + +.transition { + stroke: ${t.transitionColor}; + stroke-width: 1; + fill: none; +} + +.stateGroup .composit { + fill: ${t.background}; + border-bottom: 1px +} + +.stateGroup .alt-composit { + fill: #e0e0e0; + border-bottom: 1px +} + +.state-note { + stroke: ${t.noteBorderColor}; + fill: ${t.noteBkgColor}; + + text { + fill: ${t.noteTextColor}; + stroke: none; + font-size: 10px; + } +} + +.stateLabel .box { + stroke: none; + stroke-width: 0; + fill: ${t.mainBkg}; + opacity: 0.5; +} + +.edgeLabel .label rect { + fill: ${t.labelBackgroundColor}; + opacity: 0.5; +} +.edgeLabel { + background-color: ${t.edgeLabelBackground}; + p { + background-color: ${t.edgeLabelBackground}; + } + rect { + opacity: 0.5; + background-color: ${t.edgeLabelBackground}; + fill: ${t.edgeLabelBackground}; + } + text-align: center; +} +.edgeLabel .label text { + fill: ${t.transitionLabelColor||t.tertiaryTextColor}; +} +.label div .edgeLabel { + color: ${t.transitionLabelColor||t.tertiaryTextColor}; +} + +.stateLabel text { + fill: ${t.stateLabelColor}; + font-size: 10px; + font-weight: bold; +} + +.node circle.state-start { + fill: ${t.specialStateColor}; + stroke: ${t.specialStateColor}; +} + +.node .fork-join { + fill: ${t.specialStateColor}; + stroke: ${t.specialStateColor}; +} + +.node circle.state-end { + fill: ${t.innerEndBackground}; + stroke: ${t.background}; + stroke-width: 1.5 +} +.end-state-inner { + fill: ${t.compositeBackground||t.background}; + // stroke: ${t.background}; + stroke-width: 1.5 +} + +.node rect { + fill: ${t.stateBkg||t.mainBkg}; + stroke: ${t.stateBorder||t.nodeBorder}; + stroke-width: 1px; +} +.node polygon { + fill: ${t.mainBkg}; + stroke: ${t.stateBorder||t.nodeBorder};; + stroke-width: 1px; +} +#statediagram-barbEnd { + fill: ${t.lineColor}; +} + +.statediagram-cluster rect { + fill: ${t.compositeTitleBackground}; + stroke: ${t.stateBorder||t.nodeBorder}; + stroke-width: 1px; +} + +.cluster-label, .nodeLabel { + color: ${t.stateLabelColor}; + // line-height: 1; +} + +.statediagram-cluster rect.outer { + rx: 5px; + ry: 5px; +} +.statediagram-state .divider { + stroke: ${t.stateBorder||t.nodeBorder}; +} + +.statediagram-state .title-state { + rx: 5px; + ry: 5px; +} +.statediagram-cluster.statediagram-cluster .inner { + fill: ${t.compositeBackground||t.background}; +} +.statediagram-cluster.statediagram-cluster-alt .inner { + fill: ${t.altBackground?t.altBackground:"#efefef"}; +} + +.statediagram-cluster .inner { + rx:0; + ry:0; +} + +.statediagram-state rect.basic { + rx: 5px; + ry: 5px; +} +.statediagram-state rect.divider { + stroke-dasharray: 10,10; + fill: ${t.altBackground?t.altBackground:"#efefef"}; +} + +.note-edge { + stroke-dasharray: 5; +} + +.statediagram-note rect { + fill: ${t.noteBkgColor}; + stroke: ${t.noteBorderColor}; + stroke-width: 1px; + rx: 0; + ry: 0; +} +.statediagram-note rect { + fill: ${t.noteBkgColor}; + stroke: ${t.noteBorderColor}; + stroke-width: 1px; + rx: 0; + ry: 0; +} + +.statediagram-note text { + fill: ${t.noteTextColor}; +} + +.statediagram-note .nodeLabel { + color: ${t.noteTextColor}; +} +.statediagram .edgeLabel { + color: red; // ${t.noteTextColor}; +} + +#dependencyStart, #dependencyEnd { + fill: ${t.lineColor}; + stroke: ${t.lineColor}; + stroke-width: 1; +} + +.statediagramTitleText { + text-anchor: middle; + font-size: 18px; + fill: ${t.textColor}; +} +`,"getStyles"),W6=dUe});var nP,pUe,mUe,fde,gUe,dde,pde=N(()=>{"use strict";nP={},pUe=o((t,e)=>{nP[t]=e},"set"),mUe=o(t=>nP[t],"get"),fde=o(()=>Object.keys(nP),"keys"),gUe=o(()=>fde().length,"size"),dde={get:mUe,set:pUe,keys:fde,size:gUe}});var yUe,vUe,xUe,bUe,gde,wUe,TUe,kUe,EUe,iP,mde,yde,vde=N(()=>{"use strict";dr();pde();H6();ir();gr();zt();vt();yUe=o(t=>t.append("circle").attr("class","start-state").attr("r",me().state.sizeUnit).attr("cx",me().state.padding+me().state.sizeUnit).attr("cy",me().state.padding+me().state.sizeUnit),"drawStartState"),vUe=o(t=>t.append("line").style("stroke","grey").style("stroke-dasharray","3").attr("x1",me().state.textHeight).attr("class","divider").attr("x2",me().state.textHeight*2).attr("y1",0).attr("y2",0),"drawDivider"),xUe=o((t,e)=>{let r=t.append("text").attr("x",2*me().state.padding).attr("y",me().state.textHeight+2*me().state.padding).attr("font-size",me().state.fontSize).attr("class","state-title").text(e.id),n=r.node().getBBox();return t.insert("rect",":first-child").attr("x",me().state.padding).attr("y",me().state.padding).attr("width",n.width+2*me().state.padding).attr("height",n.height+2*me().state.padding).attr("rx",me().state.radius),r},"drawSimpleState"),bUe=o((t,e)=>{let r=o(function(p,m,g){let y=p.append("tspan").attr("x",2*me().state.padding).text(m);g||y.attr("dy",me().state.textHeight)},"addTspan"),i=t.append("text").attr("x",2*me().state.padding).attr("y",me().state.textHeight+1.3*me().state.padding).attr("font-size",me().state.fontSize).attr("class","state-title").text(e.descriptions[0]).node().getBBox(),a=i.height,s=t.append("text").attr("x",me().state.padding).attr("y",a+me().state.padding*.4+me().state.dividerMargin+me().state.textHeight).attr("class","state-description"),l=!0,u=!0;e.descriptions.forEach(function(p){l||(r(s,p,u),u=!1),l=!1});let h=t.append("line").attr("x1",me().state.padding).attr("y1",me().state.padding+a+me().state.dividerMargin/2).attr("y2",me().state.padding+a+me().state.dividerMargin/2).attr("class","descr-divider"),f=s.node().getBBox(),d=Math.max(f.width,i.width);return h.attr("x2",d+3*me().state.padding),t.insert("rect",":first-child").attr("x",me().state.padding).attr("y",me().state.padding).attr("width",d+2*me().state.padding).attr("height",f.height+a+2*me().state.padding).attr("rx",me().state.radius),t},"drawDescrState"),gde=o((t,e,r)=>{let n=me().state.padding,i=2*me().state.padding,a=t.node().getBBox(),s=a.width,l=a.x,u=t.append("text").attr("x",0).attr("y",me().state.titleShift).attr("font-size",me().state.fontSize).attr("class","state-title").text(e.id),f=u.node().getBBox().width+i,d=Math.max(f,s);d===s&&(d=d+i);let p,m=t.node().getBBox();e.doc,p=l-n,f>s&&(p=(s-d)/2+n),Math.abs(l-m.x)s&&(p=l-(f-s)/2);let g=1-me().state.textHeight;return t.insert("rect",":first-child").attr("x",p).attr("y",g).attr("class",r?"alt-composit":"composit").attr("width",d).attr("height",m.height+me().state.textHeight+me().state.titleShift+1).attr("rx","0"),u.attr("x",p+n),f<=s&&u.attr("x",l+(d-i)/2-f/2+n),t.insert("rect",":first-child").attr("x",p).attr("y",me().state.titleShift-me().state.textHeight-me().state.padding).attr("width",d).attr("height",me().state.textHeight*3).attr("rx",me().state.radius),t.insert("rect",":first-child").attr("x",p).attr("y",me().state.titleShift-me().state.textHeight-me().state.padding).attr("width",d).attr("height",m.height+3+2*me().state.textHeight).attr("rx",me().state.radius),t},"addTitleAndBox"),wUe=o(t=>(t.append("circle").attr("class","end-state-outer").attr("r",me().state.sizeUnit+me().state.miniPadding).attr("cx",me().state.padding+me().state.sizeUnit+me().state.miniPadding).attr("cy",me().state.padding+me().state.sizeUnit+me().state.miniPadding),t.append("circle").attr("class","end-state-inner").attr("r",me().state.sizeUnit).attr("cx",me().state.padding+me().state.sizeUnit+2).attr("cy",me().state.padding+me().state.sizeUnit+2)),"drawEndState"),TUe=o((t,e)=>{let r=me().state.forkWidth,n=me().state.forkHeight;if(e.parentId){let i=r;r=n,n=i}return t.append("rect").style("stroke","black").style("fill","black").attr("width",r).attr("height",n).attr("x",me().state.padding).attr("y",me().state.padding)},"drawForkJoinState"),kUe=o((t,e,r,n)=>{let i=0,a=n.append("text");a.style("text-anchor","start"),a.attr("class","noteText");let s=t.replace(/\r\n/g,"
    ");s=s.replace(/\n/g,"
    ");let l=s.split(Ze.lineBreakRegex),u=1.25*me().state.noteMargin;for(let h of l){let f=h.trim();if(f.length>0){let d=a.append("tspan");if(d.text(f),u===0){let p=d.node().getBBox();u+=p.height}i+=u,d.attr("x",e+me().state.noteMargin),d.attr("y",r+i+1.25*me().state.noteMargin)}}return{textWidth:a.node().getBBox().width,textHeight:i}},"_drawLongText"),EUe=o((t,e)=>{e.attr("class","state-note");let r=e.append("rect").attr("x",0).attr("y",me().state.padding),n=e.append("g"),{textWidth:i,textHeight:a}=kUe(t,0,0,n);return r.attr("height",a+2*me().state.noteMargin),r.attr("width",i+me().state.noteMargin*2),r},"drawNote"),iP=o(function(t,e){let r=e.id,n={id:r,label:e.id,width:0,height:0},i=t.append("g").attr("id",r).attr("class","stateGroup");e.type==="start"&&yUe(i),e.type==="end"&&wUe(i),(e.type==="fork"||e.type==="join")&&TUe(i,e),e.type==="note"&&EUe(e.note.text,i),e.type==="divider"&&vUe(i),e.type==="default"&&e.descriptions.length===0&&xUe(i,e),e.type==="default"&&e.descriptions.length>0&&bUe(i,e);let a=i.node().getBBox();return n.width=a.width+2*me().state.padding,n.height=a.height+2*me().state.padding,dde.set(r,n),n},"drawState"),mde=0,yde=o(function(t,e,r){let n=o(function(u){switch(u){case Qo.relationType.AGGREGATION:return"aggregation";case Qo.relationType.EXTENSION:return"extension";case Qo.relationType.COMPOSITION:return"composition";case Qo.relationType.DEPENDENCY:return"dependency"}},"getRelationType");e.points=e.points.filter(u=>!Number.isNaN(u.y));let i=e.points,a=wl().x(function(u){return u.x}).y(function(u){return u.y}).curve(Do),s=t.append("path").attr("d",a(i)).attr("id","edge"+mde).attr("class","transition"),l="";if(me().state.arrowMarkerAbsolute&&(l=window.location.protocol+"//"+window.location.host+window.location.pathname+window.location.search,l=l.replace(/\(/g,"\\("),l=l.replace(/\)/g,"\\)")),s.attr("marker-end","url("+l+"#"+n(Qo.relationType.DEPENDENCY)+"End)"),r.title!==void 0){let u=t.append("g").attr("class","stateLabel"),{x:h,y:f}=Gt.calcLabelPosition(e.points),d=Ze.getRows(r.title),p=0,m=[],g=0,y=0;for(let b=0;b<=d.length;b++){let w=u.append("text").attr("text-anchor","middle").text(d[b]).attr("x",h).attr("y",f+p),C=w.node().getBBox();g=Math.max(g,C.width),y=Math.min(y,C.x),Y.info(C.x,h,f+p),p===0&&(p=w.node().getBBox().height,Y.info("Title height",p,f)),m.push(w)}let v=p*d.length;if(d.length>1){let b=(d.length-1)*p*.5;m.forEach((w,C)=>w.attr("y",f+C*p-b)),v=p*d.length}let x=u.node().getBBox();u.insert("rect",":first-child").attr("class","box").attr("x",h-g/2-me().state.padding/2).attr("y",f-v/2-me().state.padding/2-3.5).attr("width",g+me().state.padding).attr("height",v+me().state.padding),Y.info(x)}mde++},"drawEdge")});var fo,aP,SUe,CUe,AUe,_Ue,xde,bde,wde=N(()=>{"use strict";dr();gR();Vo();vt();gr();vde();zt();Ei();aP={},SUe=o(function(){},"setConf"),CUe=o(function(t){t.append("defs").append("marker").attr("id","dependencyEnd").attr("refX",19).attr("refY",7).attr("markerWidth",20).attr("markerHeight",28).attr("orient","auto").append("path").attr("d","M 19,7 L9,13 L14,7 L9,1 Z")},"insertMarkers"),AUe=o(function(t,e,r,n){fo=me().state;let i=me().securityLevel,a;i==="sandbox"&&(a=Ge("#i"+e));let s=i==="sandbox"?Ge(a.nodes()[0].contentDocument.body):Ge("body"),l=i==="sandbox"?a.nodes()[0].contentDocument:document;Y.debug("Rendering diagram "+t);let u=s.select(`[id='${e}']`);CUe(u);let h=n.db.getRootDoc();xde(h,u,void 0,!1,s,l,n);let f=fo.padding,d=u.node().getBBox(),p=d.width+f*2,m=d.height+f*2,g=p*1.75;vn(u,m,g,fo.useMaxWidth),u.attr("viewBox",`${d.x-fo.padding} ${d.y-fo.padding} `+p+" "+m)},"draw"),_Ue=o(t=>t?t.length*fo.fontSizeFactor:1,"getLabelWidth"),xde=o((t,e,r,n,i,a,s)=>{let l=new sn({compound:!0,multigraph:!0}),u,h=!0;for(u=0;u{let T=C.parentElement,E=0,A=0;T&&(T.parentElement&&(E=T.parentElement.getBBox().width),A=parseInt(T.getAttribute("data-x-shift"),10),Number.isNaN(A)&&(A=0)),C.setAttribute("x1",0-A+8),C.setAttribute("x2",E-A-8)})):Y.debug("No Node "+b+": "+JSON.stringify(l.node(b)))});let v=y.getBBox();l.edges().forEach(function(b){b!==void 0&&l.edge(b)!==void 0&&(Y.debug("Edge "+b.v+" -> "+b.w+": "+JSON.stringify(l.edge(b))),yde(e,l.edge(b),l.edge(b).relation))}),v=y.getBBox();let x={id:r||"root",label:r||"root",width:0,height:0};return x.width=v.width+2*fo.padding,x.height=v.height+2*fo.padding,Y.debug("Doc rendered",x,l),x},"renderDoc"),bde={setConf:SUe,draw:AUe}});var Tde={};hr(Tde,{diagram:()=>DUe});var DUe,kde=N(()=>{"use strict";$O();H6();rP();wde();DUe={parser:I6,get db(){return new Qo(1)},renderer:bde,styles:W6,init:o(t=>{t.state||(t.state={}),t.state.arrowMarkerAbsolute=t.arrowMarkerAbsolute},"init")}});var Cde={};hr(Cde,{diagram:()=>MUe});var MUe,Ade=N(()=>{"use strict";$O();H6();rP();eP();MUe={parser:I6,get db(){return new Qo(2)},renderer:ide,styles:W6,init:o(t=>{t.state||(t.state={}),t.state.arrowMarkerAbsolute=t.arrowMarkerAbsolute},"init")}});var sP,Lde,Rde=N(()=>{"use strict";sP=function(){var t=o(function(d,p,m,g){for(m=m||{},g=d.length;g--;m[d[g]]=p);return m},"o"),e=[6,8,10,11,12,14,16,17,18],r=[1,9],n=[1,10],i=[1,11],a=[1,12],s=[1,13],l=[1,14],u={trace:o(function(){},"trace"),yy:{},symbols_:{error:2,start:3,journey:4,document:5,EOF:6,line:7,SPACE:8,statement:9,NEWLINE:10,title:11,acc_title:12,acc_title_value:13,acc_descr:14,acc_descr_value:15,acc_descr_multiline_value:16,section:17,taskName:18,taskData:19,$accept:0,$end:1},terminals_:{2:"error",4:"journey",6:"EOF",8:"SPACE",10:"NEWLINE",11:"title",12:"acc_title",13:"acc_title_value",14:"acc_descr",15:"acc_descr_value",16:"acc_descr_multiline_value",17:"section",18:"taskName",19:"taskData"},productions_:[0,[3,3],[5,0],[5,2],[7,2],[7,1],[7,1],[7,1],[9,1],[9,2],[9,2],[9,1],[9,1],[9,2]],performAction:o(function(p,m,g,y,v,x,b){var w=x.length-1;switch(v){case 1:return x[w-1];case 2:this.$=[];break;case 3:x[w-1].push(x[w]),this.$=x[w-1];break;case 4:case 5:this.$=x[w];break;case 6:case 7:this.$=[];break;case 8:y.setDiagramTitle(x[w].substr(6)),this.$=x[w].substr(6);break;case 9:this.$=x[w].trim(),y.setAccTitle(this.$);break;case 10:case 11:this.$=x[w].trim(),y.setAccDescription(this.$);break;case 12:y.addSection(x[w].substr(8)),this.$=x[w].substr(8);break;case 13:y.addTask(x[w-1],x[w]),this.$="task";break}},"anonymous"),table:[{3:1,4:[1,2]},{1:[3]},t(e,[2,2],{5:3}),{6:[1,4],7:5,8:[1,6],9:7,10:[1,8],11:r,12:n,14:i,16:a,17:s,18:l},t(e,[2,7],{1:[2,1]}),t(e,[2,3]),{9:15,11:r,12:n,14:i,16:a,17:s,18:l},t(e,[2,5]),t(e,[2,6]),t(e,[2,8]),{13:[1,16]},{15:[1,17]},t(e,[2,11]),t(e,[2,12]),{19:[1,18]},t(e,[2,4]),t(e,[2,9]),t(e,[2,10]),t(e,[2,13])],defaultActions:{},parseError:o(function(p,m){if(m.recoverable)this.trace(p);else{var g=new Error(p);throw g.hash=m,g}},"parseError"),parse:o(function(p){var m=this,g=[0],y=[],v=[null],x=[],b=this.table,w="",C=0,T=0,E=0,A=2,S=1,_=x.slice.call(arguments,1),I=Object.create(this.lexer),D={yy:{}};for(var k in this.yy)Object.prototype.hasOwnProperty.call(this.yy,k)&&(D.yy[k]=this.yy[k]);I.setInput(p,D.yy),D.yy.lexer=I,D.yy.parser=this,typeof I.yylloc>"u"&&(I.yylloc={});var L=I.yylloc;x.push(L);var R=I.options&&I.options.ranges;typeof D.yy.parseError=="function"?this.parseError=D.yy.parseError:this.parseError=Object.getPrototypeOf(this).parseError;function O(K){g.length=g.length-2*K,v.length=v.length-K,x.length=x.length-K}o(O,"popStack");function M(){var K;return K=y.pop()||I.lex()||S,typeof K!="number"&&(K instanceof Array&&(y=K,K=y.pop()),K=m.symbols_[K]||K),K}o(M,"lex");for(var B,F,P,z,$,H,Q={},j,ie,ne,le;;){if(P=g[g.length-1],this.defaultActions[P]?z=this.defaultActions[P]:((B===null||typeof B>"u")&&(B=M()),z=b[P]&&b[P][B]),typeof z>"u"||!z.length||!z[0]){var he="";le=[];for(j in b[P])this.terminals_[j]&&j>A&&le.push("'"+this.terminals_[j]+"'");I.showPosition?he="Parse error on line "+(C+1)+`: +`+I.showPosition()+` +Expecting `+le.join(", ")+", got '"+(this.terminals_[B]||B)+"'":he="Parse error on line "+(C+1)+": Unexpected "+(B==S?"end of input":"'"+(this.terminals_[B]||B)+"'"),this.parseError(he,{text:I.match,token:this.terminals_[B]||B,line:I.yylineno,loc:L,expected:le})}if(z[0]instanceof Array&&z.length>1)throw new Error("Parse Error: multiple actions possible at state: "+P+", token: "+B);switch(z[0]){case 1:g.push(B),v.push(I.yytext),x.push(I.yylloc),g.push(z[1]),B=null,F?(B=F,F=null):(T=I.yyleng,w=I.yytext,C=I.yylineno,L=I.yylloc,E>0&&E--);break;case 2:if(ie=this.productions_[z[1]][1],Q.$=v[v.length-ie],Q._$={first_line:x[x.length-(ie||1)].first_line,last_line:x[x.length-1].last_line,first_column:x[x.length-(ie||1)].first_column,last_column:x[x.length-1].last_column},R&&(Q._$.range=[x[x.length-(ie||1)].range[0],x[x.length-1].range[1]]),H=this.performAction.apply(Q,[w,T,C,D.yy,z[1],v,x].concat(_)),typeof H<"u")return H;ie&&(g=g.slice(0,-1*ie*2),v=v.slice(0,-1*ie),x=x.slice(0,-1*ie)),g.push(this.productions_[z[1]][0]),v.push(Q.$),x.push(Q._$),ne=b[g[g.length-2]][g[g.length-1]],g.push(ne);break;case 3:return!0}}return!0},"parse")},h=function(){var d={EOF:1,parseError:o(function(m,g){if(this.yy.parser)this.yy.parser.parseError(m,g);else throw new Error(m)},"parseError"),setInput:o(function(p,m){return this.yy=m||this.yy||{},this._input=p,this._more=this._backtrack=this.done=!1,this.yylineno=this.yyleng=0,this.yytext=this.matched=this.match="",this.conditionStack=["INITIAL"],this.yylloc={first_line:1,first_column:0,last_line:1,last_column:0},this.options.ranges&&(this.yylloc.range=[0,0]),this.offset=0,this},"setInput"),input:o(function(){var p=this._input[0];this.yytext+=p,this.yyleng++,this.offset++,this.match+=p,this.matched+=p;var m=p.match(/(?:\r\n?|\n).*/g);return m?(this.yylineno++,this.yylloc.last_line++):this.yylloc.last_column++,this.options.ranges&&this.yylloc.range[1]++,this._input=this._input.slice(1),p},"input"),unput:o(function(p){var m=p.length,g=p.split(/(?:\r\n?|\n)/g);this._input=p+this._input,this.yytext=this.yytext.substr(0,this.yytext.length-m),this.offset-=m;var y=this.match.split(/(?:\r\n?|\n)/g);this.match=this.match.substr(0,this.match.length-1),this.matched=this.matched.substr(0,this.matched.length-1),g.length-1&&(this.yylineno-=g.length-1);var v=this.yylloc.range;return this.yylloc={first_line:this.yylloc.first_line,last_line:this.yylineno+1,first_column:this.yylloc.first_column,last_column:g?(g.length===y.length?this.yylloc.first_column:0)+y[y.length-g.length].length-g[0].length:this.yylloc.first_column-m},this.options.ranges&&(this.yylloc.range=[v[0],v[0]+this.yyleng-m]),this.yyleng=this.yytext.length,this},"unput"),more:o(function(){return this._more=!0,this},"more"),reject:o(function(){if(this.options.backtrack_lexer)this._backtrack=!0;else return this.parseError("Lexical error on line "+(this.yylineno+1)+`. You can only invoke reject() in the lexer when the lexer is of the backtracking persuasion (options.backtrack_lexer = true). +`+this.showPosition(),{text:"",token:null,line:this.yylineno});return this},"reject"),less:o(function(p){this.unput(this.match.slice(p))},"less"),pastInput:o(function(){var p=this.matched.substr(0,this.matched.length-this.match.length);return(p.length>20?"...":"")+p.substr(-20).replace(/\n/g,"")},"pastInput"),upcomingInput:o(function(){var p=this.match;return p.length<20&&(p+=this._input.substr(0,20-p.length)),(p.substr(0,20)+(p.length>20?"...":"")).replace(/\n/g,"")},"upcomingInput"),showPosition:o(function(){var p=this.pastInput(),m=new Array(p.length+1).join("-");return p+this.upcomingInput()+` +`+m+"^"},"showPosition"),test_match:o(function(p,m){var g,y,v;if(this.options.backtrack_lexer&&(v={yylineno:this.yylineno,yylloc:{first_line:this.yylloc.first_line,last_line:this.last_line,first_column:this.yylloc.first_column,last_column:this.yylloc.last_column},yytext:this.yytext,match:this.match,matches:this.matches,matched:this.matched,yyleng:this.yyleng,offset:this.offset,_more:this._more,_input:this._input,yy:this.yy,conditionStack:this.conditionStack.slice(0),done:this.done},this.options.ranges&&(v.yylloc.range=this.yylloc.range.slice(0))),y=p[0].match(/(?:\r\n?|\n).*/g),y&&(this.yylineno+=y.length),this.yylloc={first_line:this.yylloc.last_line,last_line:this.yylineno+1,first_column:this.yylloc.last_column,last_column:y?y[y.length-1].length-y[y.length-1].match(/\r?\n?/)[0].length:this.yylloc.last_column+p[0].length},this.yytext+=p[0],this.match+=p[0],this.matches=p,this.yyleng=this.yytext.length,this.options.ranges&&(this.yylloc.range=[this.offset,this.offset+=this.yyleng]),this._more=!1,this._backtrack=!1,this._input=this._input.slice(p[0].length),this.matched+=p[0],g=this.performAction.call(this,this.yy,this,m,this.conditionStack[this.conditionStack.length-1]),this.done&&this._input&&(this.done=!1),g)return g;if(this._backtrack){for(var x in v)this[x]=v[x];return!1}return!1},"test_match"),next:o(function(){if(this.done)return this.EOF;this._input||(this.done=!0);var p,m,g,y;this._more||(this.yytext="",this.match="");for(var v=this._currentRules(),x=0;xm[0].length)){if(m=g,y=x,this.options.backtrack_lexer){if(p=this.test_match(g,v[x]),p!==!1)return p;if(this._backtrack){m=!1;continue}else return!1}else if(!this.options.flex)break}return m?(p=this.test_match(m,v[y]),p!==!1?p:!1):this._input===""?this.EOF:this.parseError("Lexical error on line "+(this.yylineno+1)+`. Unrecognized text. +`+this.showPosition(),{text:"",token:null,line:this.yylineno})},"next"),lex:o(function(){var m=this.next();return m||this.lex()},"lex"),begin:o(function(m){this.conditionStack.push(m)},"begin"),popState:o(function(){var m=this.conditionStack.length-1;return m>0?this.conditionStack.pop():this.conditionStack[0]},"popState"),_currentRules:o(function(){return this.conditionStack.length&&this.conditionStack[this.conditionStack.length-1]?this.conditions[this.conditionStack[this.conditionStack.length-1]].rules:this.conditions.INITIAL.rules},"_currentRules"),topState:o(function(m){return m=this.conditionStack.length-1-Math.abs(m||0),m>=0?this.conditionStack[m]:"INITIAL"},"topState"),pushState:o(function(m){this.begin(m)},"pushState"),stateStackSize:o(function(){return this.conditionStack.length},"stateStackSize"),options:{"case-insensitive":!0},performAction:o(function(m,g,y,v){var x=v;switch(y){case 0:break;case 1:break;case 2:return 10;case 3:break;case 4:break;case 5:return 4;case 6:return 11;case 7:return this.begin("acc_title"),12;break;case 8:return this.popState(),"acc_title_value";break;case 9:return this.begin("acc_descr"),14;break;case 10:return this.popState(),"acc_descr_value";break;case 11:this.begin("acc_descr_multiline");break;case 12:this.popState();break;case 13:return"acc_descr_multiline_value";case 14:return 17;case 15:return 18;case 16:return 19;case 17:return":";case 18:return 6;case 19:return"INVALID"}},"anonymous"),rules:[/^(?:%(?!\{)[^\n]*)/i,/^(?:[^\}]%%[^\n]*)/i,/^(?:[\n]+)/i,/^(?:\s+)/i,/^(?:#[^\n]*)/i,/^(?:journey\b)/i,/^(?:title\s[^#\n;]+)/i,/^(?:accTitle\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*\{\s*)/i,/^(?:[\}])/i,/^(?:[^\}]*)/i,/^(?:section\s[^#:\n;]+)/i,/^(?:[^#:\n;]+)/i,/^(?::[^#\n;]+)/i,/^(?::)/i,/^(?:$)/i,/^(?:.)/i],conditions:{acc_descr_multiline:{rules:[12,13],inclusive:!1},acc_descr:{rules:[10],inclusive:!1},acc_title:{rules:[8],inclusive:!1},INITIAL:{rules:[0,1,2,3,4,5,6,7,9,11,14,15,16,17,18,19],inclusive:!0}}};return d}();u.lexer=h;function f(){this.yy={}}return o(f,"Parser"),f.prototype=u,u.Parser=f,new f}();sP.parser=sP;Lde=sP});var M1,oP,Sb,Cb,BUe,FUe,$Ue,zUe,GUe,VUe,UUe,Nde,HUe,lP,Mde=N(()=>{"use strict";zt();mi();M1="",oP=[],Sb=[],Cb=[],BUe=o(function(){oP.length=0,Sb.length=0,M1="",Cb.length=0,Ar()},"clear"),FUe=o(function(t){M1=t,oP.push(t)},"addSection"),$Ue=o(function(){return oP},"getSections"),zUe=o(function(){let t=Nde(),e=100,r=0;for(;!t&&r{r.people&&t.push(...r.people)}),[...new Set(t)].sort()},"updateActors"),VUe=o(function(t,e){let r=e.substr(1).split(":"),n=0,i=[];r.length===1?(n=Number(r[0]),i=[]):(n=Number(r[0]),i=r[1].split(","));let a=i.map(l=>l.trim()),s={section:M1,type:M1,people:a,task:t,score:n};Cb.push(s)},"addTask"),UUe=o(function(t){let e={section:M1,type:M1,description:t,task:t,classes:[]};Sb.push(e)},"addTaskOrg"),Nde=o(function(){let t=o(function(r){return Cb[r].processed},"compileTask"),e=!0;for(let[r,n]of Cb.entries())t(r),e=e&&n.processed;return e},"compileTasks"),HUe=o(function(){return GUe()},"getActors"),lP={getConfig:o(()=>me().journey,"getConfig"),clear:BUe,setDiagramTitle:$r,getDiagramTitle:Ir,setAccTitle:Lr,getAccTitle:Rr,setAccDescription:Nr,getAccDescription:Mr,addSection:FUe,getSections:$Ue,getTasks:zUe,addTask:VUe,addTaskOrg:UUe,getActors:HUe}});var WUe,Ide,Ode=N(()=>{"use strict";WUe=o(t=>`.label { + font-family: ${t.fontFamily}; + color: ${t.textColor}; + } + .mouth { + stroke: #666; + } + + line { + stroke: ${t.textColor} + } + + .legend { + fill: ${t.textColor}; + font-family: ${t.fontFamily}; + } + + .label text { + fill: #333; + } + .label { + color: ${t.textColor} + } + + .face { + ${t.faceColor?`fill: ${t.faceColor}`:"fill: #FFF8DC"}; + stroke: #999; + } + + .node rect, + .node circle, + .node ellipse, + .node polygon, + .node path { + fill: ${t.mainBkg}; + stroke: ${t.nodeBorder}; + stroke-width: 1px; + } + + .node .label { + text-align: center; + } + .node.clickable { + cursor: pointer; + } + + .arrowheadPath { + fill: ${t.arrowheadColor}; + } + + .edgePath .path { + stroke: ${t.lineColor}; + stroke-width: 1.5px; + } + + .flowchart-link { + stroke: ${t.lineColor}; + fill: none; + } + + .edgeLabel { + background-color: ${t.edgeLabelBackground}; + rect { + opacity: 0.5; + } + text-align: center; + } + + .cluster rect { + } + + .cluster text { + fill: ${t.titleColor}; + } + + div.mermaidTooltip { + position: absolute; + text-align: center; + max-width: 200px; + padding: 2px; + font-family: ${t.fontFamily}; + font-size: 12px; + background: ${t.tertiaryColor}; + border: 1px solid ${t.border2}; + border-radius: 2px; + pointer-events: none; + z-index: 100; + } + + .task-type-0, .section-type-0 { + ${t.fillType0?`fill: ${t.fillType0}`:""}; + } + .task-type-1, .section-type-1 { + ${t.fillType0?`fill: ${t.fillType1}`:""}; + } + .task-type-2, .section-type-2 { + ${t.fillType0?`fill: ${t.fillType2}`:""}; + } + .task-type-3, .section-type-3 { + ${t.fillType0?`fill: ${t.fillType3}`:""}; + } + .task-type-4, .section-type-4 { + ${t.fillType0?`fill: ${t.fillType4}`:""}; + } + .task-type-5, .section-type-5 { + ${t.fillType0?`fill: ${t.fillType5}`:""}; + } + .task-type-6, .section-type-6 { + ${t.fillType0?`fill: ${t.fillType6}`:""}; + } + .task-type-7, .section-type-7 { + ${t.fillType0?`fill: ${t.fillType7}`:""}; + } + + .actor-0 { + ${t.actor0?`fill: ${t.actor0}`:""}; + } + .actor-1 { + ${t.actor1?`fill: ${t.actor1}`:""}; + } + .actor-2 { + ${t.actor2?`fill: ${t.actor2}`:""}; + } + .actor-3 { + ${t.actor3?`fill: ${t.actor3}`:""}; + } + .actor-4 { + ${t.actor4?`fill: ${t.actor4}`:""}; + } + .actor-5 { + ${t.actor5?`fill: ${t.actor5}`:""}; + } +`,"getStyles"),Ide=WUe});var cP,qUe,Bde,Fde,YUe,XUe,Pde,jUe,KUe,$de,QUe,I1,zde=N(()=>{"use strict";dr();Wv();cP=o(function(t,e){return kd(t,e)},"drawRect"),qUe=o(function(t,e){let n=t.append("circle").attr("cx",e.cx).attr("cy",e.cy).attr("class","face").attr("r",15).attr("stroke-width",2).attr("overflow","visible"),i=t.append("g");i.append("circle").attr("cx",e.cx-15/3).attr("cy",e.cy-15/3).attr("r",1.5).attr("stroke-width",2).attr("fill","#666").attr("stroke","#666"),i.append("circle").attr("cx",e.cx+15/3).attr("cy",e.cy-15/3).attr("r",1.5).attr("stroke-width",2).attr("fill","#666").attr("stroke","#666");function a(u){let h=bl().startAngle(Math.PI/2).endAngle(3*(Math.PI/2)).innerRadius(7.5).outerRadius(6.8181818181818175);u.append("path").attr("class","mouth").attr("d",h).attr("transform","translate("+e.cx+","+(e.cy+2)+")")}o(a,"smile");function s(u){let h=bl().startAngle(3*Math.PI/2).endAngle(5*(Math.PI/2)).innerRadius(7.5).outerRadius(6.8181818181818175);u.append("path").attr("class","mouth").attr("d",h).attr("transform","translate("+e.cx+","+(e.cy+7)+")")}o(s,"sad");function l(u){u.append("line").attr("class","mouth").attr("stroke",2).attr("x1",e.cx-5).attr("y1",e.cy+7).attr("x2",e.cx+5).attr("y2",e.cy+7).attr("class","mouth").attr("stroke-width","1px").attr("stroke","#666")}return o(l,"ambivalent"),e.score>3?a(i):e.score<3?s(i):l(i),n},"drawFace"),Bde=o(function(t,e){let r=t.append("circle");return r.attr("cx",e.cx),r.attr("cy",e.cy),r.attr("class","actor-"+e.pos),r.attr("fill",e.fill),r.attr("stroke",e.stroke),r.attr("r",e.r),r.class!==void 0&&r.attr("class",r.class),e.title!==void 0&&r.append("title").text(e.title),r},"drawCircle"),Fde=o(function(t,e){return Nq(t,e)},"drawText"),YUe=o(function(t,e){function r(i,a,s,l,u){return i+","+a+" "+(i+s)+","+a+" "+(i+s)+","+(a+l-u)+" "+(i+s-u*1.2)+","+(a+l)+" "+i+","+(a+l)}o(r,"genPoints");let n=t.append("polygon");n.attr("points",r(e.x,e.y,50,20,7)),n.attr("class","labelBox"),e.y=e.y+e.labelMargin,e.x=e.x+.5*e.labelMargin,Fde(t,e)},"drawLabel"),XUe=o(function(t,e,r){let n=t.append("g"),i=Tl();i.x=e.x,i.y=e.y,i.fill=e.fill,i.width=r.width*e.taskCount+r.diagramMarginX*(e.taskCount-1),i.height=r.height,i.class="journey-section section-type-"+e.num,i.rx=3,i.ry=3,cP(n,i),$de(r)(e.text,n,i.x,i.y,i.width,i.height,{class:"journey-section section-type-"+e.num},r,e.colour)},"drawSection"),Pde=-1,jUe=o(function(t,e,r){let n=e.x+r.width/2,i=t.append("g");Pde++;let a=300+5*30;i.append("line").attr("id","task"+Pde).attr("x1",n).attr("y1",e.y).attr("x2",n).attr("y2",a).attr("class","task-line").attr("stroke-width","1px").attr("stroke-dasharray","4 2").attr("stroke","#666"),qUe(i,{cx:n,cy:300+(5-e.score)*30,score:e.score});let s=Tl();s.x=e.x,s.y=e.y,s.fill=e.fill,s.width=r.width,s.height=r.height,s.class="task task-type-"+e.num,s.rx=3,s.ry=3,cP(i,s);let l=e.x+14;e.people.forEach(u=>{let h=e.actors[u].color,f={cx:l,cy:e.y,r:7,fill:h,stroke:"#000",title:u,pos:e.actors[u].position};Bde(i,f),l+=10}),$de(r)(e.task,i,s.x,s.y,s.width,s.height,{class:"task"},r,e.colour)},"drawTask"),KUe=o(function(t,e){q5(t,e)},"drawBackgroundRect"),$de=function(){function t(i,a,s,l,u,h,f,d){let p=a.append("text").attr("x",s+u/2).attr("y",l+h/2+5).style("font-color",d).style("text-anchor","middle").text(i);n(p,f)}o(t,"byText");function e(i,a,s,l,u,h,f,d,p){let{taskFontSize:m,taskFontFamily:g}=d,y=i.split(//gi);for(let v=0;v{let i=ju[n].color,a={cx:20,cy:r,r:7,fill:i,stroke:"#000",pos:ju[n].position};I1.drawCircle(t,a);let s={x:40,y:r+7,fill:"#666",text:n,textMargin:e.boxTextMargin|5};I1.drawText(t,s),r+=20})}var ZUe,ju,q6,Np,eHe,Zo,uP,Gde,tHe,hP,Vde=N(()=>{"use strict";dr();zde();zt();Ei();ZUe=o(function(t){Object.keys(t).forEach(function(r){q6[r]=t[r]})},"setConf"),ju={};o(JUe,"drawActorLegend");q6=me().journey,Np=q6.leftMargin,eHe=o(function(t,e,r,n){let i=me().journey,a=me().securityLevel,s;a==="sandbox"&&(s=Ge("#i"+e));let l=a==="sandbox"?Ge(s.nodes()[0].contentDocument.body):Ge("body");Zo.init();let u=l.select("#"+e);I1.initGraphics(u);let h=n.db.getTasks(),f=n.db.getDiagramTitle(),d=n.db.getActors();for(let x in ju)delete ju[x];let p=0;d.forEach(x=>{ju[x]={color:i.actorColours[p%i.actorColours.length],position:p},p++}),JUe(u),Zo.insert(0,0,Np,Object.keys(ju).length*50),tHe(u,h,0);let m=Zo.getBounds();f&&u.append("text").text(f).attr("x",Np).attr("font-size","4ex").attr("font-weight","bold").attr("y",25);let g=m.stopy-m.starty+2*i.diagramMarginY,y=Np+m.stopx+2*i.diagramMarginX;vn(u,g,y,i.useMaxWidth),u.append("line").attr("x1",Np).attr("y1",i.height*4).attr("x2",y-Np-4).attr("y2",i.height*4).attr("stroke-width",4).attr("stroke","black").attr("marker-end","url(#arrowhead)");let v=f?70:0;u.attr("viewBox",`${m.startx} -25 ${y} ${g+v}`),u.attr("preserveAspectRatio","xMinYMin meet"),u.attr("height",g+v+25)},"draw"),Zo={data:{startx:void 0,stopx:void 0,starty:void 0,stopy:void 0},verticalPos:0,sequenceItems:[],init:o(function(){this.sequenceItems=[],this.data={startx:void 0,stopx:void 0,starty:void 0,stopy:void 0},this.verticalPos=0},"init"),updateVal:o(function(t,e,r,n){t[e]===void 0?t[e]=r:t[e]=n(r,t[e])},"updateVal"),updateBounds:o(function(t,e,r,n){let i=me().journey,a=this,s=0;function l(u){return o(function(f){s++;let d=a.sequenceItems.length-s+1;a.updateVal(f,"starty",e-d*i.boxMargin,Math.min),a.updateVal(f,"stopy",n+d*i.boxMargin,Math.max),a.updateVal(Zo.data,"startx",t-d*i.boxMargin,Math.min),a.updateVal(Zo.data,"stopx",r+d*i.boxMargin,Math.max),u!=="activation"&&(a.updateVal(f,"startx",t-d*i.boxMargin,Math.min),a.updateVal(f,"stopx",r+d*i.boxMargin,Math.max),a.updateVal(Zo.data,"starty",e-d*i.boxMargin,Math.min),a.updateVal(Zo.data,"stopy",n+d*i.boxMargin,Math.max))},"updateItemBounds")}o(l,"updateFn"),this.sequenceItems.forEach(l())},"updateBounds"),insert:o(function(t,e,r,n){let i=Math.min(t,r),a=Math.max(t,r),s=Math.min(e,n),l=Math.max(e,n);this.updateVal(Zo.data,"startx",i,Math.min),this.updateVal(Zo.data,"starty",s,Math.min),this.updateVal(Zo.data,"stopx",a,Math.max),this.updateVal(Zo.data,"stopy",l,Math.max),this.updateBounds(i,s,a,l)},"insert"),bumpVerticalPos:o(function(t){this.verticalPos=this.verticalPos+t,this.data.stopy=this.verticalPos},"bumpVerticalPos"),getVerticalPos:o(function(){return this.verticalPos},"getVerticalPos"),getBounds:o(function(){return this.data},"getBounds")},uP=q6.sectionFills,Gde=q6.sectionColours,tHe=o(function(t,e,r){let n=me().journey,i="",a=n.height*2+n.diagramMarginY,s=r+a,l=0,u="#CCC",h="black",f=0;for(let[d,p]of e.entries()){if(i!==p.section){u=uP[l%uP.length],f=l%uP.length,h=Gde[l%Gde.length];let g=0,y=p.section;for(let x=d;x(ju[y]&&(g[y]=ju[y]),g),{});p.x=d*n.taskMargin+d*n.width+Np,p.y=s,p.width=n.diagramMarginX,p.height=n.diagramMarginY,p.colour=h,p.fill=u,p.num=f,p.actors=m,I1.drawTask(t,p,n),Zo.insert(p.x,p.y,p.x+p.width+n.taskMargin,300+5*30)}},"drawTasks"),hP={setConf:ZUe,draw:eHe}});var Ude={};hr(Ude,{diagram:()=>rHe});var rHe,Hde=N(()=>{"use strict";Rde();Mde();Ode();Vde();rHe={parser:Lde,db:lP,renderer:hP,styles:Ide,init:o(t=>{hP.setConf(t.journey),lP.clear()},"init")}});var dP,Qde,Zde=N(()=>{"use strict";dP=function(){var t=o(function(p,m,g,y){for(g=g||{},y=p.length;y--;g[p[y]]=m);return g},"o"),e=[6,8,10,11,12,14,16,17,20,21],r=[1,9],n=[1,10],i=[1,11],a=[1,12],s=[1,13],l=[1,16],u=[1,17],h={trace:o(function(){},"trace"),yy:{},symbols_:{error:2,start:3,timeline:4,document:5,EOF:6,line:7,SPACE:8,statement:9,NEWLINE:10,title:11,acc_title:12,acc_title_value:13,acc_descr:14,acc_descr_value:15,acc_descr_multiline_value:16,section:17,period_statement:18,event_statement:19,period:20,event:21,$accept:0,$end:1},terminals_:{2:"error",4:"timeline",6:"EOF",8:"SPACE",10:"NEWLINE",11:"title",12:"acc_title",13:"acc_title_value",14:"acc_descr",15:"acc_descr_value",16:"acc_descr_multiline_value",17:"section",20:"period",21:"event"},productions_:[0,[3,3],[5,0],[5,2],[7,2],[7,1],[7,1],[7,1],[9,1],[9,2],[9,2],[9,1],[9,1],[9,1],[9,1],[18,1],[19,1]],performAction:o(function(m,g,y,v,x,b,w){var C=b.length-1;switch(x){case 1:return b[C-1];case 2:this.$=[];break;case 3:b[C-1].push(b[C]),this.$=b[C-1];break;case 4:case 5:this.$=b[C];break;case 6:case 7:this.$=[];break;case 8:v.getCommonDb().setDiagramTitle(b[C].substr(6)),this.$=b[C].substr(6);break;case 9:this.$=b[C].trim(),v.getCommonDb().setAccTitle(this.$);break;case 10:case 11:this.$=b[C].trim(),v.getCommonDb().setAccDescription(this.$);break;case 12:v.addSection(b[C].substr(8)),this.$=b[C].substr(8);break;case 15:v.addTask(b[C],0,""),this.$=b[C];break;case 16:v.addEvent(b[C].substr(2)),this.$=b[C];break}},"anonymous"),table:[{3:1,4:[1,2]},{1:[3]},t(e,[2,2],{5:3}),{6:[1,4],7:5,8:[1,6],9:7,10:[1,8],11:r,12:n,14:i,16:a,17:s,18:14,19:15,20:l,21:u},t(e,[2,7],{1:[2,1]}),t(e,[2,3]),{9:18,11:r,12:n,14:i,16:a,17:s,18:14,19:15,20:l,21:u},t(e,[2,5]),t(e,[2,6]),t(e,[2,8]),{13:[1,19]},{15:[1,20]},t(e,[2,11]),t(e,[2,12]),t(e,[2,13]),t(e,[2,14]),t(e,[2,15]),t(e,[2,16]),t(e,[2,4]),t(e,[2,9]),t(e,[2,10])],defaultActions:{},parseError:o(function(m,g){if(g.recoverable)this.trace(m);else{var y=new Error(m);throw y.hash=g,y}},"parseError"),parse:o(function(m){var g=this,y=[0],v=[],x=[null],b=[],w=this.table,C="",T=0,E=0,A=0,S=2,_=1,I=b.slice.call(arguments,1),D=Object.create(this.lexer),k={yy:{}};for(var L in this.yy)Object.prototype.hasOwnProperty.call(this.yy,L)&&(k.yy[L]=this.yy[L]);D.setInput(m,k.yy),k.yy.lexer=D,k.yy.parser=this,typeof D.yylloc>"u"&&(D.yylloc={});var R=D.yylloc;b.push(R);var O=D.options&&D.options.ranges;typeof k.yy.parseError=="function"?this.parseError=k.yy.parseError:this.parseError=Object.getPrototypeOf(this).parseError;function M(X){y.length=y.length-2*X,x.length=x.length-X,b.length=b.length-X}o(M,"popStack");function B(){var X;return X=v.pop()||D.lex()||_,typeof X!="number"&&(X instanceof Array&&(v=X,X=v.pop()),X=g.symbols_[X]||X),X}o(B,"lex");for(var F,P,z,$,H,Q,j={},ie,ne,le,he;;){if(z=y[y.length-1],this.defaultActions[z]?$=this.defaultActions[z]:((F===null||typeof F>"u")&&(F=B()),$=w[z]&&w[z][F]),typeof $>"u"||!$.length||!$[0]){var K="";he=[];for(ie in w[z])this.terminals_[ie]&&ie>S&&he.push("'"+this.terminals_[ie]+"'");D.showPosition?K="Parse error on line "+(T+1)+`: +`+D.showPosition()+` +Expecting `+he.join(", ")+", got '"+(this.terminals_[F]||F)+"'":K="Parse error on line "+(T+1)+": Unexpected "+(F==_?"end of input":"'"+(this.terminals_[F]||F)+"'"),this.parseError(K,{text:D.match,token:this.terminals_[F]||F,line:D.yylineno,loc:R,expected:he})}if($[0]instanceof Array&&$.length>1)throw new Error("Parse Error: multiple actions possible at state: "+z+", token: "+F);switch($[0]){case 1:y.push(F),x.push(D.yytext),b.push(D.yylloc),y.push($[1]),F=null,P?(F=P,P=null):(E=D.yyleng,C=D.yytext,T=D.yylineno,R=D.yylloc,A>0&&A--);break;case 2:if(ne=this.productions_[$[1]][1],j.$=x[x.length-ne],j._$={first_line:b[b.length-(ne||1)].first_line,last_line:b[b.length-1].last_line,first_column:b[b.length-(ne||1)].first_column,last_column:b[b.length-1].last_column},O&&(j._$.range=[b[b.length-(ne||1)].range[0],b[b.length-1].range[1]]),Q=this.performAction.apply(j,[C,E,T,k.yy,$[1],x,b].concat(I)),typeof Q<"u")return Q;ne&&(y=y.slice(0,-1*ne*2),x=x.slice(0,-1*ne),b=b.slice(0,-1*ne)),y.push(this.productions_[$[1]][0]),x.push(j.$),b.push(j._$),le=w[y[y.length-2]][y[y.length-1]],y.push(le);break;case 3:return!0}}return!0},"parse")},f=function(){var p={EOF:1,parseError:o(function(g,y){if(this.yy.parser)this.yy.parser.parseError(g,y);else throw new Error(g)},"parseError"),setInput:o(function(m,g){return this.yy=g||this.yy||{},this._input=m,this._more=this._backtrack=this.done=!1,this.yylineno=this.yyleng=0,this.yytext=this.matched=this.match="",this.conditionStack=["INITIAL"],this.yylloc={first_line:1,first_column:0,last_line:1,last_column:0},this.options.ranges&&(this.yylloc.range=[0,0]),this.offset=0,this},"setInput"),input:o(function(){var m=this._input[0];this.yytext+=m,this.yyleng++,this.offset++,this.match+=m,this.matched+=m;var g=m.match(/(?:\r\n?|\n).*/g);return g?(this.yylineno++,this.yylloc.last_line++):this.yylloc.last_column++,this.options.ranges&&this.yylloc.range[1]++,this._input=this._input.slice(1),m},"input"),unput:o(function(m){var g=m.length,y=m.split(/(?:\r\n?|\n)/g);this._input=m+this._input,this.yytext=this.yytext.substr(0,this.yytext.length-g),this.offset-=g;var v=this.match.split(/(?:\r\n?|\n)/g);this.match=this.match.substr(0,this.match.length-1),this.matched=this.matched.substr(0,this.matched.length-1),y.length-1&&(this.yylineno-=y.length-1);var x=this.yylloc.range;return this.yylloc={first_line:this.yylloc.first_line,last_line:this.yylineno+1,first_column:this.yylloc.first_column,last_column:y?(y.length===v.length?this.yylloc.first_column:0)+v[v.length-y.length].length-y[0].length:this.yylloc.first_column-g},this.options.ranges&&(this.yylloc.range=[x[0],x[0]+this.yyleng-g]),this.yyleng=this.yytext.length,this},"unput"),more:o(function(){return this._more=!0,this},"more"),reject:o(function(){if(this.options.backtrack_lexer)this._backtrack=!0;else return this.parseError("Lexical error on line "+(this.yylineno+1)+`. You can only invoke reject() in the lexer when the lexer is of the backtracking persuasion (options.backtrack_lexer = true). +`+this.showPosition(),{text:"",token:null,line:this.yylineno});return this},"reject"),less:o(function(m){this.unput(this.match.slice(m))},"less"),pastInput:o(function(){var m=this.matched.substr(0,this.matched.length-this.match.length);return(m.length>20?"...":"")+m.substr(-20).replace(/\n/g,"")},"pastInput"),upcomingInput:o(function(){var m=this.match;return m.length<20&&(m+=this._input.substr(0,20-m.length)),(m.substr(0,20)+(m.length>20?"...":"")).replace(/\n/g,"")},"upcomingInput"),showPosition:o(function(){var m=this.pastInput(),g=new Array(m.length+1).join("-");return m+this.upcomingInput()+` +`+g+"^"},"showPosition"),test_match:o(function(m,g){var y,v,x;if(this.options.backtrack_lexer&&(x={yylineno:this.yylineno,yylloc:{first_line:this.yylloc.first_line,last_line:this.last_line,first_column:this.yylloc.first_column,last_column:this.yylloc.last_column},yytext:this.yytext,match:this.match,matches:this.matches,matched:this.matched,yyleng:this.yyleng,offset:this.offset,_more:this._more,_input:this._input,yy:this.yy,conditionStack:this.conditionStack.slice(0),done:this.done},this.options.ranges&&(x.yylloc.range=this.yylloc.range.slice(0))),v=m[0].match(/(?:\r\n?|\n).*/g),v&&(this.yylineno+=v.length),this.yylloc={first_line:this.yylloc.last_line,last_line:this.yylineno+1,first_column:this.yylloc.last_column,last_column:v?v[v.length-1].length-v[v.length-1].match(/\r?\n?/)[0].length:this.yylloc.last_column+m[0].length},this.yytext+=m[0],this.match+=m[0],this.matches=m,this.yyleng=this.yytext.length,this.options.ranges&&(this.yylloc.range=[this.offset,this.offset+=this.yyleng]),this._more=!1,this._backtrack=!1,this._input=this._input.slice(m[0].length),this.matched+=m[0],y=this.performAction.call(this,this.yy,this,g,this.conditionStack[this.conditionStack.length-1]),this.done&&this._input&&(this.done=!1),y)return y;if(this._backtrack){for(var b in x)this[b]=x[b];return!1}return!1},"test_match"),next:o(function(){if(this.done)return this.EOF;this._input||(this.done=!0);var m,g,y,v;this._more||(this.yytext="",this.match="");for(var x=this._currentRules(),b=0;bg[0].length)){if(g=y,v=b,this.options.backtrack_lexer){if(m=this.test_match(y,x[b]),m!==!1)return m;if(this._backtrack){g=!1;continue}else return!1}else if(!this.options.flex)break}return g?(m=this.test_match(g,x[v]),m!==!1?m:!1):this._input===""?this.EOF:this.parseError("Lexical error on line "+(this.yylineno+1)+`. Unrecognized text. +`+this.showPosition(),{text:"",token:null,line:this.yylineno})},"next"),lex:o(function(){var g=this.next();return g||this.lex()},"lex"),begin:o(function(g){this.conditionStack.push(g)},"begin"),popState:o(function(){var g=this.conditionStack.length-1;return g>0?this.conditionStack.pop():this.conditionStack[0]},"popState"),_currentRules:o(function(){return this.conditionStack.length&&this.conditionStack[this.conditionStack.length-1]?this.conditions[this.conditionStack[this.conditionStack.length-1]].rules:this.conditions.INITIAL.rules},"_currentRules"),topState:o(function(g){return g=this.conditionStack.length-1-Math.abs(g||0),g>=0?this.conditionStack[g]:"INITIAL"},"topState"),pushState:o(function(g){this.begin(g)},"pushState"),stateStackSize:o(function(){return this.conditionStack.length},"stateStackSize"),options:{"case-insensitive":!0},performAction:o(function(g,y,v,x){var b=x;switch(v){case 0:break;case 1:break;case 2:return 10;case 3:break;case 4:break;case 5:return 4;case 6:return 11;case 7:return this.begin("acc_title"),12;break;case 8:return this.popState(),"acc_title_value";break;case 9:return this.begin("acc_descr"),14;break;case 10:return this.popState(),"acc_descr_value";break;case 11:this.begin("acc_descr_multiline");break;case 12:this.popState();break;case 13:return"acc_descr_multiline_value";case 14:return 17;case 15:return 21;case 16:return 20;case 17:return 6;case 18:return"INVALID"}},"anonymous"),rules:[/^(?:%(?!\{)[^\n]*)/i,/^(?:[^\}]%%[^\n]*)/i,/^(?:[\n]+)/i,/^(?:\s+)/i,/^(?:#[^\n]*)/i,/^(?:timeline\b)/i,/^(?:title\s[^\n]+)/i,/^(?:accTitle\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*:\s*)/i,/^(?:(?!\n||)*[^\n]*)/i,/^(?:accDescr\s*\{\s*)/i,/^(?:[\}])/i,/^(?:[^\}]*)/i,/^(?:section\s[^:\n]+)/i,/^(?::\s[^:\n]+)/i,/^(?:[^#:\n]+)/i,/^(?:$)/i,/^(?:.)/i],conditions:{acc_descr_multiline:{rules:[12,13],inclusive:!1},acc_descr:{rules:[10],inclusive:!1},acc_title:{rules:[8],inclusive:!1},INITIAL:{rules:[0,1,2,3,4,5,6,7,9,11,14,15,16,17,18],inclusive:!0}}};return p}();h.lexer=f;function d(){this.yy={}}return o(d,"Parser"),d.prototype=h,h.Parser=d,new d}();dP.parser=dP;Qde=dP});var mP={};hr(mP,{addEvent:()=>ope,addSection:()=>npe,addTask:()=>spe,addTaskOrg:()=>lpe,clear:()=>rpe,default:()=>hHe,getCommonDb:()=>tpe,getSections:()=>ipe,getTasks:()=>ape});var O1,epe,pP,Y6,P1,tpe,rpe,npe,ipe,ape,spe,ope,lpe,Jde,hHe,cpe=N(()=>{"use strict";mi();O1="",epe=0,pP=[],Y6=[],P1=[],tpe=o(()=>qy,"getCommonDb"),rpe=o(function(){pP.length=0,Y6.length=0,O1="",P1.length=0,Ar()},"clear"),npe=o(function(t){O1=t,pP.push(t)},"addSection"),ipe=o(function(){return pP},"getSections"),ape=o(function(){let t=Jde(),e=100,r=0;for(;!t&&rr.id===epe-1).events.push(t)},"addEvent"),lpe=o(function(t){let e={section:O1,type:O1,description:t,task:t,classes:[]};Y6.push(e)},"addTaskOrg"),Jde=o(function(){let t=o(function(r){return P1[r].processed},"compileTask"),e=!0;for(let[r,n]of P1.entries())t(r),e=e&&n.processed;return e},"compileTasks"),hHe={clear:rpe,getCommonDb:tpe,addSection:npe,getSections:ipe,getTasks:ape,addTask:spe,addTaskOrg:lpe,addEvent:ope}});function dpe(t,e){t.each(function(){var r=Ge(this),n=r.text().split(/(\s+|
    )/).reverse(),i,a=[],s=1.1,l=r.attr("y"),u=parseFloat(r.attr("dy")),h=r.text(null).append("tspan").attr("x",0).attr("y",l).attr("dy",u+"em");for(let f=0;fe||i==="
    ")&&(a.pop(),h.text(a.join(" ").trim()),i==="
    "?a=[""]:a=[i],h=r.append("tspan").attr("x",0).attr("y",l).attr("dy",s+"em").text(i))})}var fHe,X6,dHe,pHe,hpe,mHe,gHe,upe,yHe,vHe,xHe,gP,fpe,bHe,wHe,THe,kHe,bf,ppe=N(()=>{"use strict";dr();fHe=12,X6=o(function(t,e){let r=t.append("rect");return r.attr("x",e.x),r.attr("y",e.y),r.attr("fill",e.fill),r.attr("stroke",e.stroke),r.attr("width",e.width),r.attr("height",e.height),r.attr("rx",e.rx),r.attr("ry",e.ry),e.class!==void 0&&r.attr("class",e.class),r},"drawRect"),dHe=o(function(t,e){let n=t.append("circle").attr("cx",e.cx).attr("cy",e.cy).attr("class","face").attr("r",15).attr("stroke-width",2).attr("overflow","visible"),i=t.append("g");i.append("circle").attr("cx",e.cx-15/3).attr("cy",e.cy-15/3).attr("r",1.5).attr("stroke-width",2).attr("fill","#666").attr("stroke","#666"),i.append("circle").attr("cx",e.cx+15/3).attr("cy",e.cy-15/3).attr("r",1.5).attr("stroke-width",2).attr("fill","#666").attr("stroke","#666");function a(u){let h=bl().startAngle(Math.PI/2).endAngle(3*(Math.PI/2)).innerRadius(7.5).outerRadius(6.8181818181818175);u.append("path").attr("class","mouth").attr("d",h).attr("transform","translate("+e.cx+","+(e.cy+2)+")")}o(a,"smile");function s(u){let h=bl().startAngle(3*Math.PI/2).endAngle(5*(Math.PI/2)).innerRadius(7.5).outerRadius(6.8181818181818175);u.append("path").attr("class","mouth").attr("d",h).attr("transform","translate("+e.cx+","+(e.cy+7)+")")}o(s,"sad");function l(u){u.append("line").attr("class","mouth").attr("stroke",2).attr("x1",e.cx-5).attr("y1",e.cy+7).attr("x2",e.cx+5).attr("y2",e.cy+7).attr("class","mouth").attr("stroke-width","1px").attr("stroke","#666")}return o(l,"ambivalent"),e.score>3?a(i):e.score<3?s(i):l(i),n},"drawFace"),pHe=o(function(t,e){let r=t.append("circle");return r.attr("cx",e.cx),r.attr("cy",e.cy),r.attr("class","actor-"+e.pos),r.attr("fill",e.fill),r.attr("stroke",e.stroke),r.attr("r",e.r),r.class!==void 0&&r.attr("class",r.class),e.title!==void 0&&r.append("title").text(e.title),r},"drawCircle"),hpe=o(function(t,e){let r=e.text.replace(//gi," "),n=t.append("text");n.attr("x",e.x),n.attr("y",e.y),n.attr("class","legend"),n.style("text-anchor",e.anchor),e.class!==void 0&&n.attr("class",e.class);let i=n.append("tspan");return i.attr("x",e.x+e.textMargin*2),i.text(r),n},"drawText"),mHe=o(function(t,e){function r(i,a,s,l,u){return i+","+a+" "+(i+s)+","+a+" "+(i+s)+","+(a+l-u)+" "+(i+s-u*1.2)+","+(a+l)+" "+i+","+(a+l)}o(r,"genPoints");let n=t.append("polygon");n.attr("points",r(e.x,e.y,50,20,7)),n.attr("class","labelBox"),e.y=e.y+e.labelMargin,e.x=e.x+.5*e.labelMargin,hpe(t,e)},"drawLabel"),gHe=o(function(t,e,r){let n=t.append("g"),i=gP();i.x=e.x,i.y=e.y,i.fill=e.fill,i.width=r.width,i.height=r.height,i.class="journey-section section-type-"+e.num,i.rx=3,i.ry=3,X6(n,i),fpe(r)(e.text,n,i.x,i.y,i.width,i.height,{class:"journey-section section-type-"+e.num},r,e.colour)},"drawSection"),upe=-1,yHe=o(function(t,e,r){let n=e.x+r.width/2,i=t.append("g");upe++;let a=300+5*30;i.append("line").attr("id","task"+upe).attr("x1",n).attr("y1",e.y).attr("x2",n).attr("y2",a).attr("class","task-line").attr("stroke-width","1px").attr("stroke-dasharray","4 2").attr("stroke","#666"),dHe(i,{cx:n,cy:300+(5-e.score)*30,score:e.score});let s=gP();s.x=e.x,s.y=e.y,s.fill=e.fill,s.width=r.width,s.height=r.height,s.class="task task-type-"+e.num,s.rx=3,s.ry=3,X6(i,s),fpe(r)(e.task,i,s.x,s.y,s.width,s.height,{class:"task"},r,e.colour)},"drawTask"),vHe=o(function(t,e){X6(t,{x:e.startx,y:e.starty,width:e.stopx-e.startx,height:e.stopy-e.starty,fill:e.fill,class:"rect"}).lower()},"drawBackgroundRect"),xHe=o(function(){return{x:0,y:0,fill:void 0,"text-anchor":"start",width:100,height:100,textMargin:0,rx:0,ry:0}},"getTextObj"),gP=o(function(){return{x:0,y:0,width:100,anchor:"start",height:100,rx:0,ry:0}},"getNoteRect"),fpe=function(){function t(i,a,s,l,u,h,f,d){let p=a.append("text").attr("x",s+u/2).attr("y",l+h/2+5).style("font-color",d).style("text-anchor","middle").text(i);n(p,f)}o(t,"byText");function e(i,a,s,l,u,h,f,d,p){let{taskFontSize:m,taskFontFamily:g}=d,y=i.split(//gi);for(let v=0;v{"use strict";dr();ppe();vt();zt();Ei();EHe=o(function(t,e,r,n){let i=me(),a=i.leftMargin??50;Y.debug("timeline",n.db);let s=i.securityLevel,l;s==="sandbox"&&(l=Ge("#i"+e));let h=(s==="sandbox"?Ge(l.nodes()[0].contentDocument.body):Ge("body")).select("#"+e);h.append("g");let f=n.db.getTasks(),d=n.db.getCommonDb().getDiagramTitle();Y.debug("task",f),bf.initGraphics(h);let p=n.db.getSections();Y.debug("sections",p);let m=0,g=0,y=0,v=0,x=50+a,b=50;v=50;let w=0,C=!0;p.forEach(function(_){let I={number:w,descr:_,section:w,width:150,padding:20,maxHeight:m},D=bf.getVirtualNodeHeight(h,I,i);Y.debug("sectionHeight before draw",D),m=Math.max(m,D+20)});let T=0,E=0;Y.debug("tasks.length",f.length);for(let[_,I]of f.entries()){let D={number:_,descr:I,section:I.section,width:150,padding:20,maxHeight:g},k=bf.getVirtualNodeHeight(h,D,i);Y.debug("taskHeight before draw",k),g=Math.max(g,k+20),T=Math.max(T,I.events.length);let L=0;for(let R of I.events){let O={descr:R,section:I.section,number:I.section,width:150,padding:20,maxHeight:50};L+=bf.getVirtualNodeHeight(h,O,i)}E=Math.max(E,L)}Y.debug("maxSectionHeight before draw",m),Y.debug("maxTaskHeight before draw",g),p&&p.length>0?p.forEach(_=>{let I=f.filter(R=>R.section===_),D={number:w,descr:_,section:w,width:200*Math.max(I.length,1)-50,padding:20,maxHeight:m};Y.debug("sectionNode",D);let k=h.append("g"),L=bf.drawNode(k,D,w,i);Y.debug("sectionNode output",L),k.attr("transform",`translate(${x}, ${v})`),b+=m+50,I.length>0&&mpe(h,I,w,x,b,g,i,T,E,m,!1),x+=200*Math.max(I.length,1),b=v,w++}):(C=!1,mpe(h,f,w,x,b,g,i,T,E,m,!0));let A=h.node().getBBox();Y.debug("bounds",A),d&&h.append("text").text(d).attr("x",A.width/2-a).attr("font-size","4ex").attr("font-weight","bold").attr("y",20),y=C?m+g+150:g+100,h.append("g").attr("class","lineWrapper").append("line").attr("x1",a).attr("y1",y).attr("x2",A.width+3*a).attr("y2",y).attr("stroke-width",4).attr("stroke","black").attr("marker-end","url(#arrowhead)"),Ao(void 0,h,i.timeline?.padding??50,i.timeline?.useMaxWidth??!1)},"draw"),mpe=o(function(t,e,r,n,i,a,s,l,u,h,f){for(let d of e){let p={descr:d.task,section:r,number:r,width:150,padding:20,maxHeight:a};Y.debug("taskNode",p);let m=t.append("g").attr("class","taskWrapper"),y=bf.drawNode(m,p,r,s).height;if(Y.debug("taskHeight after draw",y),m.attr("transform",`translate(${n}, ${i})`),a=Math.max(a,y),d.events){let v=t.append("g").attr("class","lineWrapper"),x=a;i+=100,x=x+SHe(t,d.events,r,n,i,s),i-=100,v.append("line").attr("x1",n+190/2).attr("y1",i+a).attr("x2",n+190/2).attr("y2",i+a+(f?a:h)+u+120).attr("stroke-width",2).attr("stroke","black").attr("marker-end","url(#arrowhead)").attr("stroke-dasharray","5,5")}n=n+200,f&&!s.timeline?.disableMulticolor&&r++}i=i-10},"drawTasks"),SHe=o(function(t,e,r,n,i,a){let s=0,l=i;i=i+100;for(let u of e){let h={descr:u,section:r,number:r,width:150,padding:20,maxHeight:50};Y.debug("eventNode",h);let f=t.append("g").attr("class","eventWrapper"),p=bf.drawNode(f,h,r,a).height;s=s+p,f.attr("transform",`translate(${n}, ${i})`),i=i+10+p}return i=l,s},"drawEvents"),gpe={setConf:o(()=>{},"setConf"),draw:EHe}});var CHe,AHe,vpe,xpe=N(()=>{"use strict";Ys();CHe=o(t=>{let e="";for(let r=0;r` + .edge { + stroke-width: 3; + } + ${CHe(t)} + .section-root rect, .section-root path, .section-root circle { + fill: ${t.git0}; + } + .section-root text { + fill: ${t.gitBranchLabel0}; + } + .icon-container { + height:100%; + display: flex; + justify-content: center; + align-items: center; + } + .edge { + fill: none; + } + .eventWrapper { + filter: brightness(120%); + } +`,"getStyles"),vpe=AHe});var bpe={};hr(bpe,{diagram:()=>_He});var _He,wpe=N(()=>{"use strict";Zde();cpe();ype();xpe();_He={db:mP,renderer:gpe,parser:Qde,styles:vpe}});var yP,Epe,Spe=N(()=>{"use strict";yP=function(){var t=o(function(C,T,E,A){for(E=E||{},A=C.length;A--;E[C[A]]=T);return E},"o"),e=[1,4],r=[1,13],n=[1,12],i=[1,15],a=[1,16],s=[1,20],l=[1,19],u=[6,7,8],h=[1,26],f=[1,24],d=[1,25],p=[6,7,11],m=[1,6,13,15,16,19,22],g=[1,33],y=[1,34],v=[1,6,7,11,13,15,16,19,22],x={trace:o(function(){},"trace"),yy:{},symbols_:{error:2,start:3,mindMap:4,spaceLines:5,SPACELINE:6,NL:7,MINDMAP:8,document:9,stop:10,EOF:11,statement:12,SPACELIST:13,node:14,ICON:15,CLASS:16,nodeWithId:17,nodeWithoutId:18,NODE_DSTART:19,NODE_DESCR:20,NODE_DEND:21,NODE_ID:22,$accept:0,$end:1},terminals_:{2:"error",6:"SPACELINE",7:"NL",8:"MINDMAP",11:"EOF",13:"SPACELIST",15:"ICON",16:"CLASS",19:"NODE_DSTART",20:"NODE_DESCR",21:"NODE_DEND",22:"NODE_ID"},productions_:[0,[3,1],[3,2],[5,1],[5,2],[5,2],[4,2],[4,3],[10,1],[10,1],[10,1],[10,2],[10,2],[9,3],[9,2],[12,2],[12,2],[12,2],[12,1],[12,1],[12,1],[12,1],[12,1],[14,1],[14,1],[18,3],[17,1],[17,4]],performAction:o(function(T,E,A,S,_,I,D){var k=I.length-1;switch(_){case 6:case 7:return S;case 8:S.getLogger().trace("Stop NL ");break;case 9:S.getLogger().trace("Stop EOF ");break;case 11:S.getLogger().trace("Stop NL2 ");break;case 12:S.getLogger().trace("Stop EOF2 ");break;case 15:S.getLogger().info("Node: ",I[k].id),S.addNode(I[k-1].length,I[k].id,I[k].descr,I[k].type);break;case 16:S.getLogger().trace("Icon: ",I[k]),S.decorateNode({icon:I[k]});break;case 17:case 21:S.decorateNode({class:I[k]});break;case 18:S.getLogger().trace("SPACELIST");break;case 19:S.getLogger().trace("Node: ",I[k].id),S.addNode(0,I[k].id,I[k].descr,I[k].type);break;case 20:S.decorateNode({icon:I[k]});break;case 25:S.getLogger().trace("node found ..",I[k-2]),this.$={id:I[k-1],descr:I[k-1],type:S.getType(I[k-2],I[k])};break;case 26:this.$={id:I[k],descr:I[k],type:S.nodeType.DEFAULT};break;case 27:S.getLogger().trace("node found ..",I[k-3]),this.$={id:I[k-3],descr:I[k-1],type:S.getType(I[k-2],I[k])};break}},"anonymous"),table:[{3:1,4:2,5:3,6:[1,5],8:e},{1:[3]},{1:[2,1]},{4:6,6:[1,7],7:[1,8],8:e},{6:r,7:[1,10],9:9,12:11,13:n,14:14,15:i,16:a,17:17,18:18,19:s,22:l},t(u,[2,3]),{1:[2,2]},t(u,[2,4]),t(u,[2,5]),{1:[2,6],6:r,12:21,13:n,14:14,15:i,16:a,17:17,18:18,19:s,22:l},{6:r,9:22,12:11,13:n,14:14,15:i,16:a,17:17,18:18,19:s,22:l},{6:h,7:f,10:23,11:d},t(p,[2,22],{17:17,18:18,14:27,15:[1,28],16:[1,29],19:s,22:l}),t(p,[2,18]),t(p,[2,19]),t(p,[2,20]),t(p,[2,21]),t(p,[2,23]),t(p,[2,24]),t(p,[2,26],{19:[1,30]}),{20:[1,31]},{6:h,7:f,10:32,11:d},{1:[2,7],6:r,12:21,13:n,14:14,15:i,16:a,17:17,18:18,19:s,22:l},t(m,[2,14],{7:g,11:y}),t(v,[2,8]),t(v,[2,9]),t(v,[2,10]),t(p,[2,15]),t(p,[2,16]),t(p,[2,17]),{20:[1,35]},{21:[1,36]},t(m,[2,13],{7:g,11:y}),t(v,[2,11]),t(v,[2,12]),{21:[1,37]},t(p,[2,25]),t(p,[2,27])],defaultActions:{2:[2,1],6:[2,2]},parseError:o(function(T,E){if(E.recoverable)this.trace(T);else{var A=new Error(T);throw A.hash=E,A}},"parseError"),parse:o(function(T){var E=this,A=[0],S=[],_=[null],I=[],D=this.table,k="",L=0,R=0,O=0,M=2,B=1,F=I.slice.call(arguments,1),P=Object.create(this.lexer),z={yy:{}};for(var $ in this.yy)Object.prototype.hasOwnProperty.call(this.yy,$)&&(z.yy[$]=this.yy[$]);P.setInput(T,z.yy),z.yy.lexer=P,z.yy.parser=this,typeof P.yylloc>"u"&&(P.yylloc={});var H=P.yylloc;I.push(H);var Q=P.options&&P.options.ranges;typeof z.yy.parseError=="function"?this.parseError=z.yy.parseError:this.parseError=Object.getPrototypeOf(this).parseError;function j(ae){A.length=A.length-2*ae,_.length=_.length-ae,I.length=I.length-ae}o(j,"popStack");function ie(){var ae;return ae=S.pop()||P.lex()||B,typeof ae!="number"&&(ae instanceof Array&&(S=ae,ae=S.pop()),ae=E.symbols_[ae]||ae),ae}o(ie,"lex");for(var ne,le,he,K,X,te,J={},se,ue,Z,Se;;){if(he=A[A.length-1],this.defaultActions[he]?K=this.defaultActions[he]:((ne===null||typeof ne>"u")&&(ne=ie()),K=D[he]&&D[he][ne]),typeof K>"u"||!K.length||!K[0]){var ce="";Se=[];for(se in D[he])this.terminals_[se]&&se>M&&Se.push("'"+this.terminals_[se]+"'");P.showPosition?ce="Parse error on line "+(L+1)+`: +`+P.showPosition()+` +Expecting `+Se.join(", ")+", got '"+(this.terminals_[ne]||ne)+"'":ce="Parse error on line "+(L+1)+": Unexpected "+(ne==B?"end of input":"'"+(this.terminals_[ne]||ne)+"'"),this.parseError(ce,{text:P.match,token:this.terminals_[ne]||ne,line:P.yylineno,loc:H,expected:Se})}if(K[0]instanceof Array&&K.length>1)throw new Error("Parse Error: multiple actions possible at state: "+he+", token: "+ne);switch(K[0]){case 1:A.push(ne),_.push(P.yytext),I.push(P.yylloc),A.push(K[1]),ne=null,le?(ne=le,le=null):(R=P.yyleng,k=P.yytext,L=P.yylineno,H=P.yylloc,O>0&&O--);break;case 2:if(ue=this.productions_[K[1]][1],J.$=_[_.length-ue],J._$={first_line:I[I.length-(ue||1)].first_line,last_line:I[I.length-1].last_line,first_column:I[I.length-(ue||1)].first_column,last_column:I[I.length-1].last_column},Q&&(J._$.range=[I[I.length-(ue||1)].range[0],I[I.length-1].range[1]]),te=this.performAction.apply(J,[k,R,L,z.yy,K[1],_,I].concat(F)),typeof te<"u")return te;ue&&(A=A.slice(0,-1*ue*2),_=_.slice(0,-1*ue),I=I.slice(0,-1*ue)),A.push(this.productions_[K[1]][0]),_.push(J.$),I.push(J._$),Z=D[A[A.length-2]][A[A.length-1]],A.push(Z);break;case 3:return!0}}return!0},"parse")},b=function(){var C={EOF:1,parseError:o(function(E,A){if(this.yy.parser)this.yy.parser.parseError(E,A);else throw new Error(E)},"parseError"),setInput:o(function(T,E){return this.yy=E||this.yy||{},this._input=T,this._more=this._backtrack=this.done=!1,this.yylineno=this.yyleng=0,this.yytext=this.matched=this.match="",this.conditionStack=["INITIAL"],this.yylloc={first_line:1,first_column:0,last_line:1,last_column:0},this.options.ranges&&(this.yylloc.range=[0,0]),this.offset=0,this},"setInput"),input:o(function(){var T=this._input[0];this.yytext+=T,this.yyleng++,this.offset++,this.match+=T,this.matched+=T;var E=T.match(/(?:\r\n?|\n).*/g);return E?(this.yylineno++,this.yylloc.last_line++):this.yylloc.last_column++,this.options.ranges&&this.yylloc.range[1]++,this._input=this._input.slice(1),T},"input"),unput:o(function(T){var E=T.length,A=T.split(/(?:\r\n?|\n)/g);this._input=T+this._input,this.yytext=this.yytext.substr(0,this.yytext.length-E),this.offset-=E;var S=this.match.split(/(?:\r\n?|\n)/g);this.match=this.match.substr(0,this.match.length-1),this.matched=this.matched.substr(0,this.matched.length-1),A.length-1&&(this.yylineno-=A.length-1);var _=this.yylloc.range;return this.yylloc={first_line:this.yylloc.first_line,last_line:this.yylineno+1,first_column:this.yylloc.first_column,last_column:A?(A.length===S.length?this.yylloc.first_column:0)+S[S.length-A.length].length-A[0].length:this.yylloc.first_column-E},this.options.ranges&&(this.yylloc.range=[_[0],_[0]+this.yyleng-E]),this.yyleng=this.yytext.length,this},"unput"),more:o(function(){return this._more=!0,this},"more"),reject:o(function(){if(this.options.backtrack_lexer)this._backtrack=!0;else return this.parseError("Lexical error on line "+(this.yylineno+1)+`. You can only invoke reject() in the lexer when the lexer is of the backtracking persuasion (options.backtrack_lexer = true). +`+this.showPosition(),{text:"",token:null,line:this.yylineno});return this},"reject"),less:o(function(T){this.unput(this.match.slice(T))},"less"),pastInput:o(function(){var T=this.matched.substr(0,this.matched.length-this.match.length);return(T.length>20?"...":"")+T.substr(-20).replace(/\n/g,"")},"pastInput"),upcomingInput:o(function(){var T=this.match;return T.length<20&&(T+=this._input.substr(0,20-T.length)),(T.substr(0,20)+(T.length>20?"...":"")).replace(/\n/g,"")},"upcomingInput"),showPosition:o(function(){var T=this.pastInput(),E=new Array(T.length+1).join("-");return T+this.upcomingInput()+` +`+E+"^"},"showPosition"),test_match:o(function(T,E){var A,S,_;if(this.options.backtrack_lexer&&(_={yylineno:this.yylineno,yylloc:{first_line:this.yylloc.first_line,last_line:this.last_line,first_column:this.yylloc.first_column,last_column:this.yylloc.last_column},yytext:this.yytext,match:this.match,matches:this.matches,matched:this.matched,yyleng:this.yyleng,offset:this.offset,_more:this._more,_input:this._input,yy:this.yy,conditionStack:this.conditionStack.slice(0),done:this.done},this.options.ranges&&(_.yylloc.range=this.yylloc.range.slice(0))),S=T[0].match(/(?:\r\n?|\n).*/g),S&&(this.yylineno+=S.length),this.yylloc={first_line:this.yylloc.last_line,last_line:this.yylineno+1,first_column:this.yylloc.last_column,last_column:S?S[S.length-1].length-S[S.length-1].match(/\r?\n?/)[0].length:this.yylloc.last_column+T[0].length},this.yytext+=T[0],this.match+=T[0],this.matches=T,this.yyleng=this.yytext.length,this.options.ranges&&(this.yylloc.range=[this.offset,this.offset+=this.yyleng]),this._more=!1,this._backtrack=!1,this._input=this._input.slice(T[0].length),this.matched+=T[0],A=this.performAction.call(this,this.yy,this,E,this.conditionStack[this.conditionStack.length-1]),this.done&&this._input&&(this.done=!1),A)return A;if(this._backtrack){for(var I in _)this[I]=_[I];return!1}return!1},"test_match"),next:o(function(){if(this.done)return this.EOF;this._input||(this.done=!0);var T,E,A,S;this._more||(this.yytext="",this.match="");for(var _=this._currentRules(),I=0;I<_.length;I++)if(A=this._input.match(this.rules[_[I]]),A&&(!E||A[0].length>E[0].length)){if(E=A,S=I,this.options.backtrack_lexer){if(T=this.test_match(A,_[I]),T!==!1)return T;if(this._backtrack){E=!1;continue}else return!1}else if(!this.options.flex)break}return E?(T=this.test_match(E,_[S]),T!==!1?T:!1):this._input===""?this.EOF:this.parseError("Lexical error on line "+(this.yylineno+1)+`. Unrecognized text. +`+this.showPosition(),{text:"",token:null,line:this.yylineno})},"next"),lex:o(function(){var E=this.next();return E||this.lex()},"lex"),begin:o(function(E){this.conditionStack.push(E)},"begin"),popState:o(function(){var E=this.conditionStack.length-1;return E>0?this.conditionStack.pop():this.conditionStack[0]},"popState"),_currentRules:o(function(){return this.conditionStack.length&&this.conditionStack[this.conditionStack.length-1]?this.conditions[this.conditionStack[this.conditionStack.length-1]].rules:this.conditions.INITIAL.rules},"_currentRules"),topState:o(function(E){return E=this.conditionStack.length-1-Math.abs(E||0),E>=0?this.conditionStack[E]:"INITIAL"},"topState"),pushState:o(function(E){this.begin(E)},"pushState"),stateStackSize:o(function(){return this.conditionStack.length},"stateStackSize"),options:{"case-insensitive":!0},performAction:o(function(E,A,S,_){var I=_;switch(S){case 0:return E.getLogger().trace("Found comment",A.yytext),6;break;case 1:return 8;case 2:this.begin("CLASS");break;case 3:return this.popState(),16;break;case 4:this.popState();break;case 5:E.getLogger().trace("Begin icon"),this.begin("ICON");break;case 6:return E.getLogger().trace("SPACELINE"),6;break;case 7:return 7;case 8:return 15;case 9:E.getLogger().trace("end icon"),this.popState();break;case 10:return E.getLogger().trace("Exploding node"),this.begin("NODE"),19;break;case 11:return E.getLogger().trace("Cloud"),this.begin("NODE"),19;break;case 12:return E.getLogger().trace("Explosion Bang"),this.begin("NODE"),19;break;case 13:return E.getLogger().trace("Cloud Bang"),this.begin("NODE"),19;break;case 14:return this.begin("NODE"),19;break;case 15:return this.begin("NODE"),19;break;case 16:return this.begin("NODE"),19;break;case 17:return this.begin("NODE"),19;break;case 18:return 13;case 19:return 22;case 20:return 11;case 21:this.begin("NSTR2");break;case 22:return"NODE_DESCR";case 23:this.popState();break;case 24:E.getLogger().trace("Starting NSTR"),this.begin("NSTR");break;case 25:return E.getLogger().trace("description:",A.yytext),"NODE_DESCR";break;case 26:this.popState();break;case 27:return this.popState(),E.getLogger().trace("node end ))"),"NODE_DEND";break;case 28:return this.popState(),E.getLogger().trace("node end )"),"NODE_DEND";break;case 29:return this.popState(),E.getLogger().trace("node end ...",A.yytext),"NODE_DEND";break;case 30:return this.popState(),E.getLogger().trace("node end (("),"NODE_DEND";break;case 31:return this.popState(),E.getLogger().trace("node end (-"),"NODE_DEND";break;case 32:return this.popState(),E.getLogger().trace("node end (-"),"NODE_DEND";break;case 33:return this.popState(),E.getLogger().trace("node end (("),"NODE_DEND";break;case 34:return this.popState(),E.getLogger().trace("node end (("),"NODE_DEND";break;case 35:return E.getLogger().trace("Long description:",A.yytext),20;break;case 36:return E.getLogger().trace("Long description:",A.yytext),20;break}},"anonymous"),rules:[/^(?:\s*%%.*)/i,/^(?:mindmap\b)/i,/^(?::::)/i,/^(?:.+)/i,/^(?:\n)/i,/^(?:::icon\()/i,/^(?:[\s]+[\n])/i,/^(?:[\n]+)/i,/^(?:[^\)]+)/i,/^(?:\))/i,/^(?:-\))/i,/^(?:\(-)/i,/^(?:\)\))/i,/^(?:\))/i,/^(?:\(\()/i,/^(?:\{\{)/i,/^(?:\()/i,/^(?:\[)/i,/^(?:[\s]+)/i,/^(?:[^\(\[\n\)\{\}]+)/i,/^(?:$)/i,/^(?:["][`])/i,/^(?:[^`"]+)/i,/^(?:[`]["])/i,/^(?:["])/i,/^(?:[^"]+)/i,/^(?:["])/i,/^(?:[\)]\))/i,/^(?:[\)])/i,/^(?:[\]])/i,/^(?:\}\})/i,/^(?:\(-)/i,/^(?:-\))/i,/^(?:\(\()/i,/^(?:\()/i,/^(?:[^\)\]\(\}]+)/i,/^(?:.+(?!\(\())/i],conditions:{CLASS:{rules:[3,4],inclusive:!1},ICON:{rules:[8,9],inclusive:!1},NSTR2:{rules:[22,23],inclusive:!1},NSTR:{rules:[25,26],inclusive:!1},NODE:{rules:[21,24,27,28,29,30,31,32,33,34,35,36],inclusive:!1},INITIAL:{rules:[0,1,2,5,6,7,10,11,12,13,14,15,16,17,18,19,20],inclusive:!0}}};return C}();x.lexer=b;function w(){this.yy={}}return o(w,"Parser"),w.prototype=x,x.Parser=w,new w}();yP.parser=yP;Epe=yP});var $l,Cpe,vP,NHe,MHe,IHe,OHe,Vi,PHe,BHe,FHe,$He,zHe,GHe,VHe,Ape,_pe=N(()=>{"use strict";zt();gr();vt();Ya();$l=[],Cpe=0,vP={},NHe=o(()=>{$l=[],Cpe=0,vP={}},"clear"),MHe=o(function(t){for(let e=$l.length-1;e>=0;e--)if($l[e].level$l.length>0?$l[0]:null,"getMindmap"),OHe=o((t,e,r,n)=>{Y.info("addNode",t,e,r,n);let i=me(),a=i.mindmap?.padding??or.mindmap.padding;switch(n){case Vi.ROUNDED_RECT:case Vi.RECT:case Vi.HEXAGON:a*=2}let s={id:Cpe++,nodeId:Tr(e,i),level:t,descr:Tr(r,i),type:n,children:[],width:i.mindmap?.maxNodeWidth??or.mindmap.maxNodeWidth,padding:a},l=MHe(t);if(l)l.children.push(s),$l.push(s);else if($l.length===0)$l.push(s);else throw new Error('There can be only one root. No parent could be found for ("'+s.descr+'")')},"addNode"),Vi={DEFAULT:0,NO_BORDER:0,ROUNDED_RECT:1,RECT:2,CIRCLE:3,CLOUD:4,BANG:5,HEXAGON:6},PHe=o((t,e)=>{switch(Y.debug("In get type",t,e),t){case"[":return Vi.RECT;case"(":return e===")"?Vi.ROUNDED_RECT:Vi.CLOUD;case"((":return Vi.CIRCLE;case")":return Vi.CLOUD;case"))":return Vi.BANG;case"{{":return Vi.HEXAGON;default:return Vi.DEFAULT}},"getType"),BHe=o((t,e)=>{vP[t]=e},"setElementForId"),FHe=o(t=>{if(!t)return;let e=me(),r=$l[$l.length-1];t.icon&&(r.icon=Tr(t.icon,e)),t.class&&(r.class=Tr(t.class,e))},"decorateNode"),$He=o(t=>{switch(t){case Vi.DEFAULT:return"no-border";case Vi.RECT:return"rect";case Vi.ROUNDED_RECT:return"rounded-rect";case Vi.CIRCLE:return"circle";case Vi.CLOUD:return"cloud";case Vi.BANG:return"bang";case Vi.HEXAGON:return"hexgon";default:return"no-border"}},"type2Str"),zHe=o(()=>Y,"getLogger"),GHe=o(t=>vP[t],"getElementById"),VHe={clear:NHe,addNode:OHe,getMindmap:IHe,nodeType:Vi,getType:PHe,setElementForId:BHe,decorateNode:FHe,type2Str:$He,getLogger:zHe,getElementById:GHe},Ape=VHe});function Wi(t){"@babel/helpers - typeof";return Wi=typeof Symbol=="function"&&typeof Symbol.iterator=="symbol"?function(e){return typeof e}:function(e){return e&&typeof Symbol=="function"&&e.constructor===Symbol&&e!==Symbol.prototype?"symbol":typeof e},Wi(t)}function Mf(t,e){if(!(t instanceof e))throw new TypeError("Cannot call a class as a function")}function Dpe(t,e){for(var r=0;rt.length)&&(e=t.length);for(var r=0,n=new Array(e);r=t.length?{done:!0}:{done:!1,value:t[n++]}},"n"),e:o(function(u){throw u},"e"),f:i}}throw new TypeError(`Invalid attempt to iterate non-iterable instance. +In order to be iterable, non-array objects must have a [Symbol.iterator]() method.`)}var a=!0,s=!1,l;return{s:o(function(){r=r.call(t)},"s"),n:o(function(){var u=r.next();return a=u.done,u},"n"),e:o(function(u){s=!0,l=u},"e"),f:o(function(){try{!a&&r.return!=null&&r.return()}finally{if(s)throw l}},"f")}}function yWe(t){var e=typeof t;return t!=null&&(e=="object"||e=="function")}function vWe(t,e){return e={exports:{}},t(e,e.exports),e.exports}function SWe(t){for(var e=t.length;e--&&EWe.test(t.charAt(e)););return e}function _We(t){return t&&t.slice(0,CWe(t)+1).replace(AWe,"")}function MWe(t){var e=RWe.call(t,Ab),r=t[Ab];try{t[Ab]=void 0;var n=!0}catch{}var i=NWe.call(t);return n&&(e?t[Ab]=r:delete t[Ab]),i}function BWe(t){return PWe.call(t)}function GWe(t){return t==null?t===void 0?zWe:$We:Npe&&Npe in Object(t)?IWe(t):FWe(t)}function VWe(t){return t!=null&&typeof t=="object"}function WWe(t){return typeof t=="symbol"||UWe(t)&&ame(t)==HWe}function KWe(t){if(typeof t=="number")return t;if(r4(t))return Mpe;if(zp(t)){var e=typeof t.valueOf=="function"?t.valueOf():t;t=zp(e)?e+"":e}if(typeof t!="string")return t===0?t:+t;t=DWe(t);var r=YWe.test(t);return r||XWe.test(t)?jWe(t.slice(2),r?2:8):qWe.test(t)?Mpe:+t}function eqe(t,e,r){var n,i,a,s,l,u,h=0,f=!1,d=!1,p=!0;if(typeof t!="function")throw new TypeError(QWe);e=Ipe(e)||0,zp(r)&&(f=!!r.leading,d="maxWait"in r,a=d?ZWe(Ipe(r.maxWait)||0,e):a,p="trailing"in r?!!r.trailing:p);function m(E){var A=n,S=i;return n=i=void 0,h=E,s=t.apply(S,A),s}o(m,"invokeFunc");function g(E){return h=E,l=setTimeout(x,e),f?m(E):s}o(g,"leadingEdge");function y(E){var A=E-u,S=E-h,_=e-A;return d?JWe(_,a-S):_}o(y,"remainingWait");function v(E){var A=E-u,S=E-h;return u===void 0||A>=e||A<0||d&&S>=a}o(v,"shouldInvoke");function x(){var E=xP();if(v(E))return b(E);l=setTimeout(x,y(E))}o(x,"timerExpired");function b(E){return l=void 0,p&&n?m(E):(n=i=void 0,s)}o(b,"trailingEdge");function w(){l!==void 0&&clearTimeout(l),h=0,n=u=i=l=void 0}o(w,"cancel");function C(){return l===void 0?s:b(xP())}o(C,"flush");function T(){var E=xP(),A=v(E);if(n=arguments,i=this,u=E,A){if(l===void 0)return g(u);if(d)return clearTimeout(l),l=setTimeout(x,e),m(u)}return l===void 0&&(l=setTimeout(x,e)),s}return o(T,"debounced"),T.cancel=w,T.flush=C,T}function IS(t,e,r,n,i,a){var s;return si(t)?s=t:s=Q1[t]||Q1.euclidean,e===0&&si(t)?s(i,a):s(e,r,n,i,a)}function qYe(t,e){if(OS(t))return!1;var r=typeof t;return r=="number"||r=="symbol"||r=="boolean"||t==null||r4(t)?!0:WYe.test(t)||!HYe.test(t)||e!=null&&t in Object(e)}function ZYe(t){if(!zp(t))return!1;var e=ame(t);return e==jYe||e==KYe||e==XYe||e==QYe}function tXe(t){return!!e0e&&e0e in t}function aXe(t){if(t!=null){try{return iXe.call(t)}catch{}try{return t+""}catch{}}return""}function pXe(t){if(!zp(t)||rXe(t))return!1;var e=JYe(t)?dXe:lXe;return e.test(sXe(t))}function gXe(t,e){return t?.[e]}function vXe(t,e){var r=yXe(t,e);return mXe(r)?r:void 0}function bXe(){this.__data__=jb?jb(null):{},this.size=0}function TXe(t){var e=this.has(t)&&delete this.__data__[t];return this.size-=e?1:0,e}function AXe(t){var e=this.__data__;if(jb){var r=e[t];return r===EXe?void 0:r}return CXe.call(e,t)?e[t]:void 0}function RXe(t){var e=this.__data__;return jb?e[t]!==void 0:LXe.call(e,t)}function IXe(t,e){var r=this.__data__;return this.size+=this.has(t)?0:1,r[t]=jb&&e===void 0?MXe:e,this}function ty(t){var e=-1,r=t==null?0:t.length;for(this.clear();++e-1}function XXe(t,e){var r=this.__data__,n=PS(r,t);return n<0?(++this.size,r.push([t,e])):r[n][1]=e,this}function ry(t){var e=-1,r=t==null?0:t.length;for(this.clear();++e-1&&t%1==0&&t0;){var f=i.shift();e(f),a.add(f.id()),l&&n(i,a,f)}return t}function Fme(t,e,r){if(r.isParent())for(var n=r._private.children,i=0;i0&&arguments[0]!==void 0?arguments[0]:NKe,e=arguments.length>1?arguments[1]:void 0,r=0;r0?k=R:D=R;while(Math.abs(L)>s&&++O=a?b(I,O):M===0?O:C(I,D,D+h)}o(T,"getTForX");var E=!1;function A(){E=!0,(t!==e||r!==n)&&w()}o(A,"precompute");var S=o(function(D){return E||A(),t===e&&r===n?D:D===0?0:D===1?1:v(T(D),e,n)},"f");S.getControlPoints=function(){return[{x:t,y:e},{x:r,y:n}]};var _="generateBezier("+[t,e,r,n]+")";return S.toString=function(){return _},S}function x0e(t,e,r,n,i){if(n===1||e===r)return r;var a=i(e,r,n);return t==null||((t.roundValue||t.color)&&(a=Math.round(a)),t.min!==void 0&&(a=Math.max(a,t.min)),t.max!==void 0&&(a=Math.min(a,t.max))),a}function b0e(t,e){return t.pfValue!=null||t.value!=null?t.pfValue!=null&&(e==null||e.type.units!=="%")?t.pfValue:t.value:t}function $1(t,e,r,n,i){var a=i!=null?i.type:null;r<0?r=0:r>1&&(r=1);var s=b0e(t,i),l=b0e(e,i);if(Ct(s)&&Ct(l))return x0e(a,s,l,r,n);if(En(s)&&En(l)){for(var u=[],h=0;h0?(m==="spring"&&g.push(s.duration),s.easingImpl=dS[m].apply(null,g)):s.easingImpl=dS[m]}var y=s.easingImpl,v;if(s.duration===0?v=1:v=(r-u)/s.duration,s.applying&&(v=s.progress),v<0?v=0:v>1&&(v=1),s.delay==null){var x=s.startPosition,b=s.position;if(b&&i&&!t.locked()){var w={};Rb(x.x,b.x)&&(w.x=$1(x.x,b.x,v,y)),Rb(x.y,b.y)&&(w.y=$1(x.y,b.y,v,y)),t.position(w)}var C=s.startPan,T=s.pan,E=a.pan,A=T!=null&&n;A&&(Rb(C.x,T.x)&&(E.x=$1(C.x,T.x,v,y)),Rb(C.y,T.y)&&(E.y=$1(C.y,T.y,v,y)),t.emit("pan"));var S=s.startZoom,_=s.zoom,I=_!=null&&n;I&&(Rb(S,_)&&(a.zoom=Yb(a.minZoom,$1(S,_,v,y),a.maxZoom)),t.emit("zoom")),(A||I)&&t.emit("viewport");var D=s.style;if(D&&D.length>0&&i){for(var k=0;k=0;A--){var S=E[A];S()}E.splice(0,E.length)},"callbacks"),b=m.length-1;b>=0;b--){var w=m[b],C=w._private;if(C.stopped){m.splice(b,1),C.hooked=!1,C.playing=!1,C.started=!1,x(C.frames);continue}!C.playing&&!C.applying||(C.playing&&C.applying&&(C.applying=!1),C.started||qKe(f,w,t),WKe(f,w,t,d),C.applying&&(C.applying=!1),x(C.frames),C.step!=null&&C.step(t),w.completed()&&(m.splice(b,1),C.hooked=!1,C.playing=!1,C.started=!1,x(C.completes)),y=!0)}return!d&&m.length===0&&g.length===0&&n.push(f),y}o(i,"stepOne");for(var a=!1,s=0;s0?e.notify("draw",r):e.notify("draw")),r.unmerge(n),e.emit("step")}function tge(t){this.options=rr({},eQe,tQe,t)}function rge(t){this.options=rr({},rQe,t)}function nge(t){this.options=rr({},nQe,t)}function HS(t){this.options=rr({},iQe,t),this.options.layout=this;var e=this.options.eles.nodes(),r=this.options.eles.edges(),n=r.filter(function(i){var a=i.source().data("id"),s=i.target().data("id"),l=e.some(function(h){return h.data("id")===a}),u=e.some(function(h){return h.data("id")===s});return!l||!u});this.options.eles=this.options.eles.not(n)}function age(t){this.options=rr({},wQe,t)}function gB(t){this.options=rr({},TQe,t)}function sge(t){this.options=rr({},kQe,t)}function oge(t){this.options=rr({},EQe,t)}function lge(t){this.options=t,this.notifications=0}function hge(t,e){e.radius===0?t.lineTo(e.cx,e.cy):t.arc(e.cx,e.cy,e.radius,e.startAngle,e.endAngle,e.counterClockwise)}function vB(t,e,r,n){var i=arguments.length>4&&arguments[4]!==void 0?arguments[4]:!0;return n===0||e.radius===0?{cx:e.x,cy:e.y,radius:0,startX:e.x,startY:e.y,stopX:e.x,stopY:e.y,startAngle:void 0,endAngle:void 0,counterClockwise:void 0}:(AQe(t,e,r,n,i),{cx:HP,cy:WP,radius:Bp,startX:cge,startY:uge,stopX:qP,stopY:YP,startAngle:qc.ang+Math.PI/2*Fp,endAngle:Jo.ang-Math.PI/2*Fp,counterClockwise:gS})}function fge(t){var e=[];if(t!=null){for(var r=0;r5&&arguments[5]!==void 0?arguments[5]:5,s=arguments.length>6?arguments[6]:void 0;t.beginPath(),t.moveTo(e+a,r),t.lineTo(e+n-a,r),t.quadraticCurveTo(e+n,r,e+n,r+a),t.lineTo(e+n,r+i-a),t.quadraticCurveTo(e+n,r+i,e+n-a,r+i),t.lineTo(e+a,r+i),t.quadraticCurveTo(e,r+i,e,r+i-a),t.lineTo(e,r+a),t.quadraticCurveTo(e,r,e+a,r),t.closePath(),s?t.stroke():t.fill()}function z0e(t,e,r){var n=t.createShader(e);if(t.shaderSource(n,r),t.compileShader(n),!t.getShaderParameter(n,t.COMPILE_STATUS))throw new Error(t.getShaderInfoLog(n));return n}function pZe(t,e,r){var n=z0e(t,t.VERTEX_SHADER,e),i=z0e(t,t.FRAGMENT_SHADER,r),a=t.createProgram();if(t.attachShader(a,n),t.attachShader(a,i),t.linkProgram(a),!t.getProgramParameter(a,t.LINK_STATUS))throw new Error("Could not initialize shaders");return a}function mZe(t,e,r){r===void 0&&(r=e);var n=t.makeOffscreenCanvas(e,r),i=n.context=n.getContext("2d");return n.clear=function(){return i.clearRect(0,0,n.width,n.height)},n.clear(),n}function wB(t){var e=t.pixelRatio,r=t.cy.zoom(),n=t.cy.pan();return{zoom:r*e,pan:{x:n.x*e,y:n.y*e}}}function NP(t,e,r,n,i){var a=n*r+e.x,s=i*r+e.y;return s=Math.round(t.canvasHeight-s),[a,s]}function oS(t,e,r){var n=t[0]/255,i=t[1]/255,a=t[2]/255,s=e,l=r||new Array(4);return l[0]=n*s,l[1]=i*s,l[2]=a*s,l[3]=s,l}function lS(t,e){var r=e||new Array(4);return r[0]=(t>>0&255)/255,r[1]=(t>>8&255)/255,r[2]=(t>>16&255)/255,r[3]=(t>>24&255)/255,r}function gZe(t){return t[0]+(t[1]<<8)+(t[2]<<16)+(t[3]<<24)}function yZe(t,e){var r=t.createTexture();return r.buffer=function(n){t.bindTexture(t.TEXTURE_2D,r),t.texParameteri(t.TEXTURE_2D,t.TEXTURE_WRAP_S,t.CLAMP_TO_EDGE),t.texParameteri(t.TEXTURE_2D,t.TEXTURE_WRAP_T,t.CLAMP_TO_EDGE),t.texParameteri(t.TEXTURE_2D,t.TEXTURE_MAG_FILTER,t.LINEAR),t.texParameteri(t.TEXTURE_2D,t.TEXTURE_MIN_FILTER,t.LINEAR_MIPMAP_NEAREST),t.pixelStorei(t.UNPACK_PREMULTIPLY_ALPHA_WEBGL,!0),t.texImage2D(t.TEXTURE_2D,0,t.RGBA,t.RGBA,t.UNSIGNED_BYTE,n),t.generateMipmap(t.TEXTURE_2D),t.bindTexture(t.TEXTURE_2D,null)},r.deleteTexture=function(){t.deleteTexture(r)},r}function Sge(t,e){switch(e){case"float":return[1,t.FLOAT,4];case"vec2":return[2,t.FLOAT,4];case"vec3":return[3,t.FLOAT,4];case"vec4":return[4,t.FLOAT,4];case"int":return[1,t.INT,4];case"ivec2":return[2,t.INT,4]}}function Cge(t,e,r){switch(e){case t.FLOAT:return new Float32Array(r);case t.INT:return new Int32Array(r)}}function vZe(t,e,r,n,i,a){switch(e){case t.FLOAT:return new Float32Array(r.buffer,a*n,i);case t.INT:return new Int32Array(r.buffer,a*n,i)}}function xZe(t,e,r,n){var i=Sge(t,e),a=_i(i,2),s=a[0],l=a[1],u=Cge(t,l,n),h=t.createBuffer();return t.bindBuffer(t.ARRAY_BUFFER,h),t.bufferData(t.ARRAY_BUFFER,u,t.STATIC_DRAW),l===t.FLOAT?t.vertexAttribPointer(r,s,l,!1,0,0):l===t.INT&&t.vertexAttribIPointer(r,s,l,0,0),t.enableVertexAttribArray(r),t.bindBuffer(t.ARRAY_BUFFER,null),h}function po(t,e,r,n){var i=Sge(t,r),a=_i(i,3),s=a[0],l=a[1],u=a[2],h=Cge(t,l,e*s),f=s*u,d=t.createBuffer();t.bindBuffer(t.ARRAY_BUFFER,d),t.bufferData(t.ARRAY_BUFFER,e*f,t.DYNAMIC_DRAW),t.enableVertexAttribArray(n),l===t.FLOAT?t.vertexAttribPointer(n,s,l,!1,f,0):l===t.INT&&t.vertexAttribIPointer(n,s,l,f,0),t.vertexAttribDivisor(n,1),t.bindBuffer(t.ARRAY_BUFFER,null);for(var p=new Array(e),m=0;mbge?(RZe(t),e.call(t,a)):(NZe(t),Rge(t,a,Vb.SCREEN)))}}{var r=t.matchCanvasSize;t.matchCanvasSize=function(a){r.call(t,a),t.pickingFrameBuffer.setFramebufferAttachmentSizes(t.canvasWidth,t.canvasHeight),t.pickingFrameBuffer.needsDraw=!0}}t.findNearestElements=function(a,s,l,u){return FZe(t,a,s)};{var n=t.invalidateCachedZSortedEles;t.invalidateCachedZSortedEles=function(){n.call(t),t.pickingFrameBuffer.needsDraw=!0}}{var i=t.notify;t.notify=function(a,s){i.call(t,a,s),a==="viewport"||a==="bounds"?t.pickingFrameBuffer.needsDraw=!0:a==="background"&&t.eleDrawing.invalidate(s,{type:"node-body"})}}}function RZe(t){var e=t.data.contexts[t.WEBGL];e.clear(e.COLOR_BUFFER_BIT|e.DEPTH_BUFFER_BIT)}function NZe(t){var e=o(function(n){n.save(),n.setTransform(1,0,0,1,0,0),n.clearRect(0,0,t.canvasWidth,t.canvasHeight),n.restore()},"clear");e(t.data.contexts[t.NODE]),e(t.data.contexts[t.DRAG])}function MZe(t){var e=t.canvasWidth,r=t.canvasHeight,n=wB(t),i=n.pan,a=n.zoom,s=Gb();DS(s,s,[i.x,i.y]),TB(s,s,[a,a]);var l=Gb();TZe(l,e,r);var u=Gb();return wZe(u,l,s),u}function Lge(t,e){var r=t.canvasWidth,n=t.canvasHeight,i=wB(t),a=i.pan,s=i.zoom;e.setTransform(1,0,0,1,0,0),e.clearRect(0,0,r,n),e.translate(a.x,a.y),e.scale(s,s)}function IZe(t,e){t.drawSelectionRectangle(e,function(r){return Lge(t,r)})}function OZe(t){var e=t.data.contexts[t.NODE];e.save(),Lge(t,e),e.strokeStyle="rgba(0, 0, 0, 0.3)",e.beginPath(),e.moveTo(-1e3,0),e.lineTo(1e3,0),e.stroke(),e.beginPath(),e.moveTo(0,-1e3),e.lineTo(0,1e3),e.stroke(),e.restore()}function PZe(t){var e=o(function(i,a,s){for(var l=i.atlasManager.getRenderTypeOpts(a),u=t.data.contexts[t.NODE],h=.125,f=l.atlasCollection.atlases,d=0;d=0&&k.add(O)}return k}function FZe(t,e,r){var n=BZe(t,e,r),i=t.getCachedZSortedEles(),a,s,l=mo(n),u;try{for(l.s();!(u=l.n()).done;){var h=u.value,f=i[h];if(!a&&f.isNode()&&(a=f),!s&&f.isEdge()&&(s=f),a&&s)break}}catch(d){l.e(d)}finally{l.f()}return[a,s].filter(Boolean)}function Rge(t,e,r){var n,i;t.webglDebug&&(i=[],n=performance.now());var a=t.eleDrawing,s=0;if(r.screen&&t.data.canvasNeedsRedraw[t.SELECT_BOX]&&IZe(t,e),t.data.canvasNeedsRedraw[t.NODE]||r.picking){var l=o(function(k,L){L+=1,k.isNode()?(a.drawTexture(k,L,"node-underlay"),a.drawTexture(k,L,"node-body"),a.drawTexture(k,L,"node-label"),a.drawTexture(k,L,"node-overlay")):(a.drawEdgeLine(k,L),a.drawEdgeArrow(k,L,"source"),a.drawEdgeArrow(k,L,"target"),a.drawTexture(k,L,"edge-label"))},"draw"),u=t.data.contexts[t.WEBGL];r.screen?(u.clearColor(0,0,0,0),u.enable(u.BLEND),u.blendFunc(u.ONE,u.ONE_MINUS_SRC_ALPHA)):u.disable(u.BLEND),u.clear(u.COLOR_BUFFER_BIT|u.DEPTH_BUFFER_BIT),u.viewport(0,0,u.canvas.width,u.canvas.height);var h=MZe(t),f=t.getCachedZSortedEles();if(s=f.length,a.startFrame(h,i,r),r.screen){for(var d=0;d{"use strict";o(Wi,"_typeof");o(Mf,"_classCallCheck");o(Dpe,"_defineProperties");o(If,"_createClass");o(X0e,"_defineProperty$1");o(_i,"_slicedToArray");o(j0e,"_toConsumableArray");o(UHe,"_arrayWithoutHoles");o(HHe,"_arrayWithHoles");o(WHe,"_iterableToArray");o(qHe,"_iterableToArrayLimit");o(ZP,"_unsupportedIterableToArray");o(OP,"_arrayLikeToArray");o(YHe,"_nonIterableSpread");o(XHe,"_nonIterableRest");o(mo,"_createForOfIteratorHelper");Ui=typeof window>"u"?null:window,Lpe=Ui?Ui.navigator:null;Ui&&Ui.document;jHe=Wi(""),K0e=Wi({}),KHe=Wi(function(){}),QHe=typeof HTMLElement>"u"?"undefined":Wi(HTMLElement),e4=o(function(e){return e&&e.instanceString&&si(e.instanceString)?e.instanceString():null},"instanceStr"),Zt=o(function(e){return e!=null&&Wi(e)==jHe},"string"),si=o(function(e){return e!=null&&Wi(e)===KHe},"fn"),En=o(function(e){return!go(e)&&(Array.isArray?Array.isArray(e):e!=null&&e instanceof Array)},"array"),Ur=o(function(e){return e!=null&&Wi(e)===K0e&&!En(e)&&e.constructor===Object},"plainObject"),ZHe=o(function(e){return e!=null&&Wi(e)===K0e},"object"),Ct=o(function(e){return e!=null&&Wi(e)===Wi(1)&&!isNaN(e)},"number"),JHe=o(function(e){return Ct(e)&&Math.floor(e)===e},"integer"),vS=o(function(e){if(QHe!=="undefined")return e!=null&&e instanceof HTMLElement},"htmlElement"),go=o(function(e){return t4(e)||Q0e(e)},"elementOrCollection"),t4=o(function(e){return e4(e)==="collection"&&e._private.single},"element"),Q0e=o(function(e){return e4(e)==="collection"&&!e._private.single},"collection"),JP=o(function(e){return e4(e)==="core"},"core"),Z0e=o(function(e){return e4(e)==="stylesheet"},"stylesheet"),eWe=o(function(e){return e4(e)==="event"},"event"),Af=o(function(e){return e==null?!0:!!(e===""||e.match(/^\s+$/))},"emptyString"),tWe=o(function(e){return typeof HTMLElement>"u"?!1:e instanceof HTMLElement},"domElement"),rWe=o(function(e){return Ur(e)&&Ct(e.x1)&&Ct(e.x2)&&Ct(e.y1)&&Ct(e.y2)},"boundingBox"),nWe=o(function(e){return ZHe(e)&&si(e.then)},"promise"),iWe=o(function(){return Lpe&&Lpe.userAgent.match(/msie|trident|edge/i)},"ms"),Ub=o(function(e,r){r||(r=o(function(){if(arguments.length===1)return arguments[0];if(arguments.length===0)return"undefined";for(var a=[],s=0;sr?1:0},"ascending"),hWe=o(function(e,r){return-1*eme(e,r)},"descending"),rr=Object.assign!=null?Object.assign.bind(Object):function(t){for(var e=arguments,r=1;r1&&(v-=1),v<1/6?g+(y-g)*6*v:v<1/2?y:v<2/3?g+(y-g)*(2/3-v)*6:g}o(f,"hue2rgb");var d=new RegExp("^"+oWe+"$").exec(e);if(d){if(n=parseInt(d[1]),n<0?n=(360- -1*n%360)%360:n>360&&(n=n%360),n/=360,i=parseFloat(d[2]),i<0||i>100||(i=i/100,a=parseFloat(d[3]),a<0||a>100)||(a=a/100,s=d[4],s!==void 0&&(s=parseFloat(s),s<0||s>1)))return;if(i===0)l=u=h=Math.round(a*255);else{var p=a<.5?a*(1+i):a+i-a*i,m=2*a-p;l=Math.round(255*f(m,p,n+1/3)),u=Math.round(255*f(m,p,n)),h=Math.round(255*f(m,p,n-1/3))}r=[l,u,h,s]}return r},"hsl2tuple"),pWe=o(function(e){var r,n=new RegExp("^"+aWe+"$").exec(e);if(n){r=[];for(var i=[],a=1;a<=3;a++){var s=n[a];if(s[s.length-1]==="%"&&(i[a]=!0),s=parseFloat(s),i[a]&&(s=s/100*255),s<0||s>255)return;r.push(Math.floor(s))}var l=i[1]||i[2]||i[3],u=i[1]&&i[2]&&i[3];if(l&&!u)return;var h=n[4];if(h!==void 0){if(h=parseFloat(h),h<0||h>1)return;r.push(h)}}return r},"rgb2tuple"),mWe=o(function(e){return gWe[e.toLowerCase()]},"colorname2tuple"),tme=o(function(e){return(En(e)?e:null)||mWe(e)||fWe(e)||pWe(e)||dWe(e)},"color2tuple"),gWe={transparent:[0,0,0,0],aliceblue:[240,248,255],antiquewhite:[250,235,215],aqua:[0,255,255],aquamarine:[127,255,212],azure:[240,255,255],beige:[245,245,220],bisque:[255,228,196],black:[0,0,0],blanchedalmond:[255,235,205],blue:[0,0,255],blueviolet:[138,43,226],brown:[165,42,42],burlywood:[222,184,135],cadetblue:[95,158,160],chartreuse:[127,255,0],chocolate:[210,105,30],coral:[255,127,80],cornflowerblue:[100,149,237],cornsilk:[255,248,220],crimson:[220,20,60],cyan:[0,255,255],darkblue:[0,0,139],darkcyan:[0,139,139],darkgoldenrod:[184,134,11],darkgray:[169,169,169],darkgreen:[0,100,0],darkgrey:[169,169,169],darkkhaki:[189,183,107],darkmagenta:[139,0,139],darkolivegreen:[85,107,47],darkorange:[255,140,0],darkorchid:[153,50,204],darkred:[139,0,0],darksalmon:[233,150,122],darkseagreen:[143,188,143],darkslateblue:[72,61,139],darkslategray:[47,79,79],darkslategrey:[47,79,79],darkturquoise:[0,206,209],darkviolet:[148,0,211],deeppink:[255,20,147],deepskyblue:[0,191,255],dimgray:[105,105,105],dimgrey:[105,105,105],dodgerblue:[30,144,255],firebrick:[178,34,34],floralwhite:[255,250,240],forestgreen:[34,139,34],fuchsia:[255,0,255],gainsboro:[220,220,220],ghostwhite:[248,248,255],gold:[255,215,0],goldenrod:[218,165,32],gray:[128,128,128],grey:[128,128,128],green:[0,128,0],greenyellow:[173,255,47],honeydew:[240,255,240],hotpink:[255,105,180],indianred:[205,92,92],indigo:[75,0,130],ivory:[255,255,240],khaki:[240,230,140],lavender:[230,230,250],lavenderblush:[255,240,245],lawngreen:[124,252,0],lemonchiffon:[255,250,205],lightblue:[173,216,230],lightcoral:[240,128,128],lightcyan:[224,255,255],lightgoldenrodyellow:[250,250,210],lightgray:[211,211,211],lightgreen:[144,238,144],lightgrey:[211,211,211],lightpink:[255,182,193],lightsalmon:[255,160,122],lightseagreen:[32,178,170],lightskyblue:[135,206,250],lightslategray:[119,136,153],lightslategrey:[119,136,153],lightsteelblue:[176,196,222],lightyellow:[255,255,224],lime:[0,255,0],limegreen:[50,205,50],linen:[250,240,230],magenta:[255,0,255],maroon:[128,0,0],mediumaquamarine:[102,205,170],mediumblue:[0,0,205],mediumorchid:[186,85,211],mediumpurple:[147,112,219],mediumseagreen:[60,179,113],mediumslateblue:[123,104,238],mediumspringgreen:[0,250,154],mediumturquoise:[72,209,204],mediumvioletred:[199,21,133],midnightblue:[25,25,112],mintcream:[245,255,250],mistyrose:[255,228,225],moccasin:[255,228,181],navajowhite:[255,222,173],navy:[0,0,128],oldlace:[253,245,230],olive:[128,128,0],olivedrab:[107,142,35],orange:[255,165,0],orangered:[255,69,0],orchid:[218,112,214],palegoldenrod:[238,232,170],palegreen:[152,251,152],paleturquoise:[175,238,238],palevioletred:[219,112,147],papayawhip:[255,239,213],peachpuff:[255,218,185],peru:[205,133,63],pink:[255,192,203],plum:[221,160,221],powderblue:[176,224,230],purple:[128,0,128],red:[255,0,0],rosybrown:[188,143,143],royalblue:[65,105,225],saddlebrown:[139,69,19],salmon:[250,128,114],sandybrown:[244,164,96],seagreen:[46,139,87],seashell:[255,245,238],sienna:[160,82,45],silver:[192,192,192],skyblue:[135,206,235],slateblue:[106,90,205],slategray:[112,128,144],slategrey:[112,128,144],snow:[255,250,250],springgreen:[0,255,127],steelblue:[70,130,180],tan:[210,180,140],teal:[0,128,128],thistle:[216,191,216],tomato:[255,99,71],turquoise:[64,224,208],violet:[238,130,238],wheat:[245,222,179],white:[255,255,255],whitesmoke:[245,245,245],yellow:[255,255,0],yellowgreen:[154,205,50]},rme=o(function(e){for(var r=e.map,n=e.keys,i=n.length,a=0;a1&&arguments[1]!==void 0?arguments[1]:V1,n=r,i;i=e.next(),!i.done;)n=n*ome+i.value|0;return n},"hashIterableInts"),Hb=o(function(e){var r=arguments.length>1&&arguments[1]!==void 0?arguments[1]:V1;return r*ome+e|0},"hashInt"),Wb=o(function(e){var r=arguments.length>1&&arguments[1]!==void 0?arguments[1]:Ob;return(r<<5)+r+e|0},"hashIntAlt"),rqe=o(function(e,r){return e*2097152+r},"combineHashes"),wf=o(function(e){return e[0]*2097152+e[1]},"combineHashesArray"),j6=o(function(e,r){return[Hb(e[0],r[0]),Wb(e[1],r[1])]},"hashArrays"),nqe=o(function(e,r){var n={value:0,done:!1},i=0,a=e.length,s={next:o(function(){return i=0&&!(e[i]===r&&(e.splice(i,1),n));i--);},"removeFromArray"),nB=o(function(e){e.splice(0,e.length)},"clearArray"),uqe=o(function(e,r){for(var n=0;n"u"?"undefined":Wi(Set))!==fqe?Set:dqe,NS=o(function(e,r){var n=arguments.length>2&&arguments[2]!==void 0?arguments[2]:!0;if(e===void 0||r===void 0||!JP(e)){ai("An element must have a core reference and parameters set");return}var i=r.group;if(i==null&&(r.data&&r.data.source!=null&&r.data.target!=null?i="edges":i="nodes"),i!=="nodes"&&i!=="edges"){ai("An element must be of type `nodes` or `edges`; you specified `"+i+"`");return}this.length=1,this[0]=this;var a=this._private={cy:e,single:!0,data:r.data||{},position:r.position||{x:0,y:0},autoWidth:void 0,autoHeight:void 0,autoPadding:void 0,compoundBoundsClean:!1,listeners:[],group:i,style:{},rstyle:{},styleCxts:[],styleKeys:{},removed:!0,selected:!!r.selected,selectable:r.selectable===void 0?!0:!!r.selectable,locked:!!r.locked,grabbed:!1,grabbable:r.grabbable===void 0?!0:!!r.grabbable,pannable:r.pannable===void 0?i==="edges":!!r.pannable,active:!1,classes:new J1,animation:{current:[],queue:[]},rscratch:{},scratch:r.scratch||{},edges:[],children:[],parent:r.parent&&r.parent.isNode()?r.parent:null,traversalCache:{},backgrounding:!1,bbCache:null,bbCacheShift:{x:0,y:0},bodyBounds:null,overlayBounds:null,labelBounds:{all:null,source:null,target:null,main:null},arrowBounds:{source:null,target:null,"mid-source":null,"mid-target":null}};if(a.position.x==null&&(a.position.x=0),a.position.y==null&&(a.position.y=0),r.renderedPosition){var s=r.renderedPosition,l=e.pan(),u=e.zoom();a.position={x:(s.x-l.x)/u,y:(s.y-l.y)/u}}var h=[];En(r.classes)?h=r.classes:Zt(r.classes)&&(h=r.classes.split(/\s+/));for(var f=0,d=h.length;fb?1:0},"defaultCmp"),f=o(function(x,b,w,C,T){var E;if(w==null&&(w=0),T==null&&(T=n),w<0)throw new Error("lo must be non-negative");for(C==null&&(C=x.length);wI;0<=I?_++:_--)S.push(_);return S}.apply(this).reverse(),A=[],C=0,T=E.length;CD;0<=D?++S:--S)k.push(s(x,w));return k},"nsmallest"),y=o(function(x,b,w,C){var T,E,A;for(C==null&&(C=n),T=x[w];w>b;){if(A=w-1>>1,E=x[A],C(T,E)<0){x[w]=E,w=A;continue}break}return x[w]=T},"_siftdown"),v=o(function(x,b,w){var C,T,E,A,S;for(w==null&&(w=n),T=x.length,S=b,E=x[b],C=2*b+1;C0;){var E=b.pop(),A=v(E),S=E.id();if(p[S]=A,A!==1/0)for(var _=E.neighborhood().intersect(g),I=0;I<_.length;I++){var D=_[I],k=D.id(),L=T(E,D),R=A+L.dist;R0)for(F.unshift(B);d[z];){var $=d[z];F.unshift($.edge),F.unshift($.node),P=$.node,z=P.id()}return l.spawn(F)},"pathTo")}},"dijkstra")},yqe={kruskal:o(function(e){e=e||function(w){return 1};for(var r=this.byGroup(),n=r.nodes,i=r.edges,a=n.length,s=new Array(a),l=n,u=o(function(C){for(var T=0;T0;){if(T(),A++,C===f){for(var S=[],_=a,I=f,D=x[I];S.unshift(_),D!=null&&S.unshift(D),_=v[I],_!=null;)I=_.id(),D=x[I];return{found:!0,distance:d[C],path:this.spawn(S),steps:A}}m[C]=!0;for(var k=w._private.edges,L=0;LD&&(g[I]=D,b[I]=_,w[I]=T),!a){var k=_*f+S;!a&&g[k]>D&&(g[k]=D,b[k]=S,w[k]=T)}}}for(var L=0;L1&&arguments[1]!==void 0?arguments[1]:s,ge=w(ae),ze=[],He=ge;;){if(He==null)return r.spawn();var $e=b(He),Re=$e.edge,Ie=$e.pred;if(ze.unshift(He[0]),He.same(Oe)&&ze.length>0)break;Re!=null&&ze.unshift(Re),He=Ie}return u.spawn(ze)},"pathTo"),E=0;E=0;f--){var d=h[f],p=d[1],m=d[2];(r[p]===l&&r[m]===u||r[p]===u&&r[m]===l)&&h.splice(f,1)}for(var g=0;gi;){var a=Math.floor(Math.random()*r.length);r=Sqe(a,e,r),n--}return r},"contractUntil"),Cqe={kargerStein:o(function(){var e=this,r=this.byGroup(),n=r.nodes,i=r.edges;i.unmergeBy(function(F){return F.isLoop()});var a=n.length,s=i.length,l=Math.ceil(Math.pow(Math.log(a)/Math.LN2,2)),u=Math.floor(a/Eqe);if(a<2){ai("At least 2 nodes are required for Karger-Stein algorithm");return}for(var h=[],f=0;f1&&arguments[1]!==void 0?arguments[1]:0,n=arguments.length>2&&arguments[2]!==void 0?arguments[2]:e.length,i=1/0,a=r;a1&&arguments[1]!==void 0?arguments[1]:0,n=arguments.length>2&&arguments[2]!==void 0?arguments[2]:e.length,i=-1/0,a=r;a1&&arguments[1]!==void 0?arguments[1]:0,n=arguments.length>2&&arguments[2]!==void 0?arguments[2]:e.length,i=0,a=0,s=r;s1&&arguments[1]!==void 0?arguments[1]:0,n=arguments.length>2&&arguments[2]!==void 0?arguments[2]:e.length,i=arguments.length>3&&arguments[3]!==void 0?arguments[3]:!0,a=arguments.length>4&&arguments[4]!==void 0?arguments[4]:!0,s=arguments.length>5&&arguments[5]!==void 0?arguments[5]:!0;i?e=e.slice(r,n):(n0&&e.splice(0,r));for(var l=0,u=e.length-1;u>=0;u--){var h=e[u];s?isFinite(h)||(e[u]=-1/0,l++):e.splice(u,1)}a&&e.sort(function(p,m){return p-m});var f=e.length,d=Math.floor(f/2);return f%2!==0?e[d+1+l]:(e[d-1+l]+e[d+l])/2},"median"),Nqe=o(function(e){return Math.PI*e/180},"deg2rad"),K6=o(function(e,r){return Math.atan2(r,e)-Math.PI/2},"getAngleFromDisp"),iB=Math.log2||function(t){return Math.log(t)/Math.log(2)},mme=o(function(e){return e>0?1:e<0?-1:0},"signum"),Gp=o(function(e,r){return Math.sqrt(Op(e,r))},"dist"),Op=o(function(e,r){var n=r.x-e.x,i=r.y-e.y;return n*n+i*i},"sqdist"),Mqe=o(function(e){for(var r=e.length,n=0,i=0;i=e.x1&&e.y2>=e.y1)return{x1:e.x1,y1:e.y1,x2:e.x2,y2:e.y2,w:e.x2-e.x1,h:e.y2-e.y1};if(e.w!=null&&e.h!=null&&e.w>=0&&e.h>=0)return{x1:e.x1,y1:e.y1,x2:e.x1+e.w,y2:e.y1+e.h,w:e.w,h:e.h}}},"makeBoundingBox"),Oqe=o(function(e){return{x1:e.x1,x2:e.x2,w:e.w,y1:e.y1,y2:e.y2,h:e.h}},"copyBoundingBox"),Pqe=o(function(e){e.x1=1/0,e.y1=1/0,e.x2=-1/0,e.y2=-1/0,e.w=0,e.h=0},"clearBoundingBox"),Bqe=o(function(e,r,n){return{x1:e.x1+r,x2:e.x2+r,y1:e.y1+n,y2:e.y2+n,w:e.w,h:e.h}},"shiftBoundingBox"),gme=o(function(e,r){e.x1=Math.min(e.x1,r.x1),e.x2=Math.max(e.x2,r.x2),e.w=e.x2-e.x1,e.y1=Math.min(e.y1,r.y1),e.y2=Math.max(e.y2,r.y2),e.h=e.y2-e.y1},"updateBoundingBox"),Fqe=o(function(e,r,n){e.x1=Math.min(e.x1,r),e.x2=Math.max(e.x2,r),e.w=e.x2-e.x1,e.y1=Math.min(e.y1,n),e.y2=Math.max(e.y2,n),e.h=e.y2-e.y1},"expandBoundingBoxByPoint"),cS=o(function(e){var r=arguments.length>1&&arguments[1]!==void 0?arguments[1]:0;return e.x1-=r,e.x2+=r,e.y1-=r,e.y2+=r,e.w=e.x2-e.x1,e.h=e.y2-e.y1,e},"expandBoundingBox"),uS=o(function(e){var r=arguments.length>1&&arguments[1]!==void 0?arguments[1]:[0],n,i,a,s;if(r.length===1)n=i=a=s=r[0];else if(r.length===2)n=a=r[0],s=i=r[1];else if(r.length===4){var l=_i(r,4);n=l[0],i=l[1],a=l[2],s=l[3]}return e.x1-=s,e.x2+=i,e.y1-=n,e.y2+=a,e.w=e.x2-e.x1,e.h=e.y2-e.y1,e},"expandBoundingBoxSides"),Fpe=o(function(e,r){e.x1=r.x1,e.y1=r.y1,e.x2=r.x2,e.y2=r.y2,e.w=e.x2-e.x1,e.h=e.y2-e.y1},"assignBoundingBox"),aB=o(function(e,r){return!(e.x1>r.x2||r.x1>e.x2||e.x2r.y2||r.y1>e.y2)},"boundingBoxesIntersect"),K1=o(function(e,r,n){return e.x1<=r&&r<=e.x2&&e.y1<=n&&n<=e.y2},"inBoundingBox"),$qe=o(function(e,r){return K1(e,r.x,r.y)},"pointInBoundingBox"),yme=o(function(e,r){return K1(e,r.x1,r.y1)&&K1(e,r.x2,r.y2)},"boundingBoxInBoundingBox"),vme=o(function(e,r,n,i,a,s,l){var u=arguments.length>7&&arguments[7]!==void 0?arguments[7]:"auto",h=u==="auto"?Vp(a,s):u,f=a/2,d=s/2;h=Math.min(h,f,d);var p=h!==f,m=h!==d,g;if(p){var y=n-f+h-l,v=i-d-l,x=n+f-h+l,b=v;if(g=Ef(e,r,n,i,y,v,x,b,!1),g.length>0)return g}if(m){var w=n+f+l,C=i-d+h-l,T=w,E=i+d-h+l;if(g=Ef(e,r,n,i,w,C,T,E,!1),g.length>0)return g}if(p){var A=n-f+h-l,S=i+d+l,_=n+f-h+l,I=S;if(g=Ef(e,r,n,i,A,S,_,I,!1),g.length>0)return g}if(m){var D=n-f-l,k=i-d+h-l,L=D,R=i+d-h+l;if(g=Ef(e,r,n,i,D,k,L,R,!1),g.length>0)return g}var O;{var M=n-f+h,B=i-d+h;if(O=Pb(e,r,n,i,M,B,h+l),O.length>0&&O[0]<=M&&O[1]<=B)return[O[0],O[1]]}{var F=n+f-h,P=i-d+h;if(O=Pb(e,r,n,i,F,P,h+l),O.length>0&&O[0]>=F&&O[1]<=P)return[O[0],O[1]]}{var z=n+f-h,$=i+d-h;if(O=Pb(e,r,n,i,z,$,h+l),O.length>0&&O[0]>=z&&O[1]>=$)return[O[0],O[1]]}{var H=n-f+h,Q=i+d-h;if(O=Pb(e,r,n,i,H,Q,h+l),O.length>0&&O[0]<=H&&O[1]>=Q)return[O[0],O[1]]}return[]},"roundRectangleIntersectLine"),zqe=o(function(e,r,n,i,a,s,l){var u=l,h=Math.min(n,a),f=Math.max(n,a),d=Math.min(i,s),p=Math.max(i,s);return h-u<=e&&e<=f+u&&d-u<=r&&r<=p+u},"inLineVicinity"),Gqe=o(function(e,r,n,i,a,s,l,u,h){var f={x1:Math.min(n,l,a)-h,x2:Math.max(n,l,a)+h,y1:Math.min(i,u,s)-h,y2:Math.max(i,u,s)+h};return!(ef.x2||rf.y2)},"inBezierVicinity"),Vqe=o(function(e,r,n,i){n-=i;var a=r*r-4*e*n;if(a<0)return[];var s=Math.sqrt(a),l=2*e,u=(-r+s)/l,h=(-r-s)/l;return[u,h]},"solveQuadratic"),Uqe=o(function(e,r,n,i,a){var s=1e-5;e===0&&(e=s),r/=e,n/=e,i/=e;var l,u,h,f,d,p,m,g;if(u=(3*n-r*r)/9,h=-(27*i)+r*(9*n-2*(r*r)),h/=54,l=u*u*u+h*h,a[1]=0,m=r/3,l>0){d=h+Math.sqrt(l),d=d<0?-Math.pow(-d,1/3):Math.pow(d,1/3),p=h-Math.sqrt(l),p=p<0?-Math.pow(-p,1/3):Math.pow(p,1/3),a[0]=-m+d+p,m+=(d+p)/2,a[4]=a[2]=-m,m=Math.sqrt(3)*(-p+d)/2,a[3]=m,a[5]=-m;return}if(a[5]=a[3]=0,l===0){g=h<0?-Math.pow(-h,1/3):Math.pow(h,1/3),a[0]=-m+2*g,a[4]=a[2]=-(g+m);return}u=-u,f=u*u*u,f=Math.acos(h/Math.sqrt(f)),g=2*Math.sqrt(u),a[0]=-m+g*Math.cos(f/3),a[2]=-m+g*Math.cos((f+2*Math.PI)/3),a[4]=-m+g*Math.cos((f+4*Math.PI)/3)},"solveCubic"),Hqe=o(function(e,r,n,i,a,s,l,u){var h=1*n*n-4*n*a+2*n*l+4*a*a-4*a*l+l*l+i*i-4*i*s+2*i*u+4*s*s-4*s*u+u*u,f=1*9*n*a-3*n*n-3*n*l-6*a*a+3*a*l+9*i*s-3*i*i-3*i*u-6*s*s+3*s*u,d=1*3*n*n-6*n*a+n*l-n*e+2*a*a+2*a*e-l*e+3*i*i-6*i*s+i*u-i*r+2*s*s+2*s*r-u*r,p=1*n*a-n*n+n*e-a*e+i*s-i*i+i*r-s*r,m=[];Uqe(h,f,d,p,m);for(var g=1e-7,y=[],v=0;v<6;v+=2)Math.abs(m[v+1])=0&&m[v]<=1&&y.push(m[v]);y.push(1),y.push(0);for(var x=-1,b,w,C,T=0;T=0?Ch?(e-a)*(e-a)+(r-s)*(r-s):f-p},"sqdistToFiniteLine"),Us=o(function(e,r,n){for(var i,a,s,l,u,h=0,f=0;f=e&&e>=s||i<=e&&e<=s)u=(e-i)/(s-i)*(l-a)+a,u>r&&h++;else continue;return h%2!==0},"pointInsidePolygonPoints"),Zu=o(function(e,r,n,i,a,s,l,u,h){var f=new Array(n.length),d;u[0]!=null?(d=Math.atan(u[1]/u[0]),u[0]<0?d=d+Math.PI/2:d=-d-Math.PI/2):d=u;for(var p=Math.cos(-d),m=Math.sin(-d),g=0;g0){var v=TS(f,-h);y=wS(v)}else y=f;return Us(e,r,y)},"pointInsidePolygon"),qqe=o(function(e,r,n,i,a,s,l,u){for(var h=new Array(n.length*2),f=0;f=0&&v<=1&&b.push(v),x>=0&&x<=1&&b.push(x),b.length===0)return[];var w=b[0]*u[0]+e,C=b[0]*u[1]+r;if(b.length>1){if(b[0]==b[1])return[w,C];var T=b[1]*u[0]+e,E=b[1]*u[1]+r;return[w,C,T,E]}else return[w,C]},"intersectLineCircle"),TP=o(function(e,r,n){return r<=e&&e<=n||n<=e&&e<=r?e:e<=r&&r<=n||n<=r&&r<=e?r:n},"midOfThree"),Ef=o(function(e,r,n,i,a,s,l,u,h){var f=e-a,d=n-e,p=l-a,m=r-s,g=i-r,y=u-s,v=p*m-y*f,x=d*m-g*f,b=y*d-p*g;if(b!==0){var w=v/b,C=x/b,T=.001,E=0-T,A=1+T;return E<=w&&w<=A&&E<=C&&C<=A?[e+w*d,r+w*g]:h?[e+w*d,r+w*g]:[]}else return v===0||x===0?TP(e,n,l)===l?[l,u]:TP(e,n,a)===a?[a,s]:TP(a,l,n)===n?[n,i]:[]:[]},"finiteLinesIntersect"),Xb=o(function(e,r,n,i,a,s,l,u){var h=[],f,d=new Array(n.length),p=!0;s==null&&(p=!1);var m;if(p){for(var g=0;g0){var y=TS(d,-u);m=wS(y)}else m=d}else m=n;for(var v,x,b,w,C=0;C2){for(var g=[f[0],f[1]],y=Math.pow(g[0]-e,2)+Math.pow(g[1]-r,2),v=1;vf&&(f=C)},"set"),get:o(function(w){return h[w]},"get")},p=0;p0?M=O.edgesTo(R)[0]:M=R.edgesTo(O)[0];var B=i(M);R=R.id(),S[R]>S[k]+B&&(S[R]=S[k]+B,_.nodes.indexOf(R)<0?_.push(R):_.updateItem(R),A[R]=0,E[R]=[]),S[R]==S[k]+B&&(A[R]=A[R]+A[k],E[R].push(k))}else for(var F=0;F0;){for(var H=T.pop(),Q=0;Q0&&l.push(n[u]);l.length!==0&&a.push(i.collection(l))}return a},"assign"),lYe=o(function(e,r){for(var n=0;n5&&arguments[5]!==void 0?arguments[5]:hYe,l=i,u,h,f=0;f=2?_b(e,r,n,0,Upe,fYe):_b(e,r,n,0,Vpe)},"euclidean"),squaredEuclidean:o(function(e,r,n){return _b(e,r,n,0,Upe)},"squaredEuclidean"),manhattan:o(function(e,r,n){return _b(e,r,n,0,Vpe)},"manhattan"),max:o(function(e,r,n){return _b(e,r,n,-1/0,dYe)},"max")};Q1["squared-euclidean"]=Q1.squaredEuclidean;Q1.squaredeuclidean=Q1.squaredEuclidean;o(IS,"clusteringDistance");pYe=la({k:2,m:2,sensitivityThreshold:1e-4,distance:"euclidean",maxIterations:10,attributes:[],testMode:!1,testCentroids:null}),oB=o(function(e){return pYe(e)},"setOptions"),kS=o(function(e,r,n,i,a){var s=a!=="kMedoids",l=s?function(d){return n[d]}:function(d){return i[d](n)},u=o(function(p){return i[p](r)},"getQ"),h=n,f=r;return IS(e,i.length,l,u,h,f)},"getDist"),kP=o(function(e,r,n){for(var i=n.length,a=new Array(i),s=new Array(i),l=new Array(r),u=null,h=0;hn)return!1}return!0},"haveMatricesConverged"),yYe=o(function(e,r,n){for(var i=0;il&&(l=r[h][f],u=f);a[u].push(e[h])}for(var d=0;d=a.threshold||a.mode==="dendrogram"&&e.length===1)return!1;var g=r[s],y=r[i[s]],v;a.mode==="dendrogram"?v={left:g,right:y,key:g.key}:v={value:g.value.concat(y.value),key:g.key},e[g.index]=v,e.splice(y.index,1),r[g.key]=v;for(var x=0;xn[y.key][b.key]&&(u=n[y.key][b.key])):a.linkage==="max"?(u=n[g.key][b.key],n[g.key][b.key]0&&i.push(a);return i},"findExemplars"),jpe=o(function(e,r,n){for(var i=[],a=0;al&&(s=h,l=r[a*e+h])}s>0&&i.push(s)}for(var f=0;fh&&(u=f,h=d)}n[a]=s[u]}return i=jpe(e,r,n),i},"assign"),Kpe=o(function(e){for(var r=this.cy(),n=this.nodes(),i=RYe(e),a={},s=0;s=D?(k=D,D=R,L=O):R>k&&(k=R);for(var M=0;M0?1:0;A[_%i.minIterations*l+H]=Q,$+=Q}if($>0&&(_>=i.minIterations-1||_==i.maxIterations-1)){for(var j=0,ie=0;ie1||E>1)&&(l=!0),d[w]=[],b.outgoers().forEach(function(S){S.isEdge()&&d[w].push(S.id())})}else p[w]=[void 0,b.target().id()]}):s.forEach(function(b){var w=b.id();if(b.isNode()){var C=b.degree(!0);C%2&&(u?h?l=!0:h=w:u=w),d[w]=[],b.connectedEdges().forEach(function(T){return d[w].push(T.id())})}else p[w]=[b.source().id(),b.target().id()]});var m={found:!1,trail:void 0};if(l)return m;if(h&&u)if(a){if(f&&h!=f)return m;f=h}else{if(f&&h!=f&&u!=f)return m;f||(f=h)}else f||(f=s[0].id());var g=o(function(w){for(var C=w,T=[w],E,A,S;d[C].length;)E=d[C].shift(),A=p[E][0],S=p[E][1],C!=S?(d[S]=d[S].filter(function(_){return _!=E}),C=S):!a&&C!=A&&(d[A]=d[A].filter(function(_){return _!=E}),C=A),T.unshift(E),T.unshift(C);return T},"walk"),y=[],v=[];for(v=g(f);v.length!=1;)d[v[0]].length==0?(y.unshift(s.getElementById(v.shift())),y.unshift(s.getElementById(v.shift()))):v=g(v.shift()).concat(v);y.unshift(s.getElementById(v.shift()));for(var x in d)if(d[x].length)return m;return m.found=!0,m.trail=this.spawn(y,!0),m},"hierholzer")},J6=o(function(){var e=this,r={},n=0,i=0,a=[],s=[],l={},u=o(function(p,m){for(var g=s.length-1,y=[],v=e.spawn();s[g].x!=p||s[g].y!=m;)y.push(s.pop().edge),g--;y.push(s.pop().edge),y.forEach(function(x){var b=x.connectedNodes().intersection(e);v.merge(x),b.forEach(function(w){var C=w.id(),T=w.connectedEdges().intersection(e);v.merge(w),r[C].cutVertex?v.merge(T.filter(function(E){return E.isLoop()})):v.merge(T)})}),a.push(v)},"buildComponent"),h=o(function d(p,m,g){p===g&&(i+=1),r[m]={id:n,low:n++,cutVertex:!1};var y=e.getElementById(m).connectedEdges().intersection(e);if(y.size()===0)a.push(e.spawn(e.getElementById(m)));else{var v,x,b,w;y.forEach(function(C){v=C.source().id(),x=C.target().id(),b=v===m?x:v,b!==g&&(w=C.id(),l[w]||(l[w]=!0,s.push({x:m,y:b,edge:C})),b in r?r[m].low=Math.min(r[m].low,r[b].id):(d(p,b,m),r[m].low=Math.min(r[m].low,r[b].low),r[m].id<=r[b].low&&(r[m].cutVertex=!0,u(m,b))))})}},"biconnectedSearch");e.forEach(function(d){if(d.isNode()){var p=d.id();p in r||(i=0,h(p,p),r[p].cutVertex=i>1)}});var f=Object.keys(r).filter(function(d){return r[d].cutVertex}).map(function(d){return e.getElementById(d)});return{cut:e.spawn(f),components:a}},"hopcroftTarjanBiconnected"),$Ye={hopcroftTarjanBiconnected:J6,htbc:J6,htb:J6,hopcroftTarjanBiconnectedComponents:J6},eS=o(function(){var e=this,r={},n=0,i=[],a=[],s=e.spawn(e),l=o(function u(h){a.push(h),r[h]={index:n,low:n++,explored:!1};var f=e.getElementById(h).connectedEdges().intersection(e);if(f.forEach(function(y){var v=y.target().id();v!==h&&(v in r||u(v),r[v].explored||(r[h].low=Math.min(r[h].low,r[v].low)))}),r[h].index===r[h].low){for(var d=e.spawn();;){var p=a.pop();if(d.merge(e.getElementById(p)),r[p].low=r[h].index,r[p].explored=!0,p===h)break}var m=d.edgesWith(d),g=d.merge(m);i.push(g),s=s.difference(g)}},"stronglyConnectedSearch");return e.forEach(function(u){if(u.isNode()){var h=u.id();h in r||l(h)}}),{cut:s,components:i}},"tarjanStronglyConnected"),zYe={tarjanStronglyConnected:eS,tsc:eS,tscc:eS,tarjanStronglyConnectedComponents:eS},Sme={};[qb,gqe,yqe,xqe,wqe,kqe,Cqe,Qqe,q1,Y1,FP,uYe,kYe,DYe,PYe,FYe,$Ye,zYe].forEach(function(t){rr(Sme,t)});Cme=0,Ame=1,_me=2,Ju=o(function t(e){if(!(this instanceof t))return new t(e);this.id="Thenable/1.0.7",this.state=Cme,this.fulfillValue=void 0,this.rejectReason=void 0,this.onFulfilled=[],this.onRejected=[],this.proxy={then:this.then.bind(this)},typeof e=="function"&&e.call(this,this.fulfill.bind(this),this.reject.bind(this))},"api");Ju.prototype={fulfill:o(function(e){return Qpe(this,Ame,"fulfillValue",e)},"fulfill"),reject:o(function(e){return Qpe(this,_me,"rejectReason",e)},"reject"),then:o(function(e,r){var n=this,i=new Ju;return n.onFulfilled.push(Jpe(e,i,"fulfill")),n.onRejected.push(Jpe(r,i,"reject")),Dme(n),i.proxy},"then")};Qpe=o(function(e,r,n,i){return e.state===Cme&&(e.state=r,e[n]=i,Dme(e)),e},"deliver"),Dme=o(function(e){e.state===Ame?Zpe(e,"onFulfilled",e.fulfillValue):e.state===_me&&Zpe(e,"onRejected",e.rejectReason)},"execute"),Zpe=o(function(e,r,n){if(e[r].length!==0){var i=e[r];e[r]=[];var a=o(function(){for(var l=0;l0},"animatedImpl")},"animated"),clearQueue:o(function(){return o(function(){var r=this,n=r.length!==void 0,i=n?r:[r],a=this._private.cy||this;if(!a.styleEnabled())return this;for(var s=0;s0&&this.spawn(i).updateStyle().emit("class"),r},"classes"),addClass:o(function(e){return this.toggleClass(e,!0)},"addClass"),hasClass:o(function(e){var r=this[0];return r!=null&&r._private.classes.has(e)},"hasClass"),toggleClass:o(function(e,r){En(e)||(e=e.match(/\S+/g)||[]);for(var n=this,i=r===void 0,a=[],s=0,l=n.length;s0&&this.spawn(a).updateStyle().emit("class"),n},"toggleClass"),removeClass:o(function(e){return this.toggleClass(e,!1)},"removeClass"),flashClass:o(function(e,r){var n=this;if(r==null)r=250;else if(r===0)return n;return n.addClass(e),setTimeout(function(){n.removeClass(e)},r),n},"flashClass")};hS.className=hS.classNames=hS.classes;Vr={metaChar:"[\\!\\\"\\#\\$\\%\\&\\'\\(\\)\\*\\+\\,\\.\\/\\:\\;\\<\\=\\>\\?\\@\\[\\]\\^\\`\\{\\|\\}\\~]",comparatorOp:"=|\\!=|>|>=|<|<=|\\$=|\\^=|\\*=",boolOp:"\\?|\\!|\\^",string:`"(?:\\\\"|[^"])*"|'(?:\\\\'|[^'])*'`,number:Hi,meta:"degree|indegree|outdegree",separator:"\\s*,\\s*",descendant:"\\s+",child:"\\s+>\\s+",subject:"\\$",group:"node|edge|\\*",directedEdge:"\\s+->\\s+",undirectedEdge:"\\s+<->\\s+"};Vr.variable="(?:[\\w-.]|(?:\\\\"+Vr.metaChar+"))+";Vr.className="(?:[\\w-]|(?:\\\\"+Vr.metaChar+"))+";Vr.value=Vr.string+"|"+Vr.number;Vr.id=Vr.variable;(function(){var t,e,r;for(t=Vr.comparatorOp.split("|"),r=0;r=0)&&e!=="="&&(Vr.comparatorOp+="|\\!"+e)})();mn=o(function(){return{checks:[]}},"newQuery"),$t={GROUP:0,COLLECTION:1,FILTER:2,DATA_COMPARE:3,DATA_EXIST:4,DATA_BOOL:5,META_COMPARE:6,STATE:7,ID:8,CLASS:9,UNDIRECTED_EDGE:10,DIRECTED_EDGE:11,NODE_SOURCE:12,NODE_TARGET:13,NODE_NEIGHBOR:14,CHILD:15,DESCENDANT:16,PARENT:17,ANCESTOR:18,COMPOUND_SPLIT:19,TRUE:20},zP=[{selector:":selected",matches:o(function(e){return e.selected()},"matches")},{selector:":unselected",matches:o(function(e){return!e.selected()},"matches")},{selector:":selectable",matches:o(function(e){return e.selectable()},"matches")},{selector:":unselectable",matches:o(function(e){return!e.selectable()},"matches")},{selector:":locked",matches:o(function(e){return e.locked()},"matches")},{selector:":unlocked",matches:o(function(e){return!e.locked()},"matches")},{selector:":visible",matches:o(function(e){return e.visible()},"matches")},{selector:":hidden",matches:o(function(e){return!e.visible()},"matches")},{selector:":transparent",matches:o(function(e){return e.transparent()},"matches")},{selector:":grabbed",matches:o(function(e){return e.grabbed()},"matches")},{selector:":free",matches:o(function(e){return!e.grabbed()},"matches")},{selector:":removed",matches:o(function(e){return e.removed()},"matches")},{selector:":inside",matches:o(function(e){return!e.removed()},"matches")},{selector:":grabbable",matches:o(function(e){return e.grabbable()},"matches")},{selector:":ungrabbable",matches:o(function(e){return!e.grabbable()},"matches")},{selector:":animated",matches:o(function(e){return e.animated()},"matches")},{selector:":unanimated",matches:o(function(e){return!e.animated()},"matches")},{selector:":parent",matches:o(function(e){return e.isParent()},"matches")},{selector:":childless",matches:o(function(e){return e.isChildless()},"matches")},{selector:":child",matches:o(function(e){return e.isChild()},"matches")},{selector:":orphan",matches:o(function(e){return e.isOrphan()},"matches")},{selector:":nonorphan",matches:o(function(e){return e.isChild()},"matches")},{selector:":compound",matches:o(function(e){return e.isNode()?e.isParent():e.source().isParent()||e.target().isParent()},"matches")},{selector:":loop",matches:o(function(e){return e.isLoop()},"matches")},{selector:":simple",matches:o(function(e){return e.isSimple()},"matches")},{selector:":active",matches:o(function(e){return e.active()},"matches")},{selector:":inactive",matches:o(function(e){return!e.active()},"matches")},{selector:":backgrounding",matches:o(function(e){return e.backgrounding()},"matches")},{selector:":nonbackgrounding",matches:o(function(e){return!e.backgrounding()},"matches")}].sort(function(t,e){return hWe(t.selector,e.selector)}),Jje=function(){for(var t={},e,r=0;r0&&f.edgeCount>0)return un("The selector `"+e+"` is invalid because it uses both a compound selector and an edge selector"),!1;if(f.edgeCount>1)return un("The selector `"+e+"` is invalid because it uses multiple edge selectors"),!1;f.edgeCount===1&&un("The selector `"+e+"` is deprecated. Edge selectors do not take effect on changes to source and target nodes after an edge is added, for performance reasons. Use a class or data selector on edges instead, updating the class or data of an edge when your app detects a change in source or target nodes.")}return!0},"parse"),aKe=o(function(){if(this.toStringCache!=null)return this.toStringCache;for(var e=o(function(f){return f??""},"clean"),r=o(function(f){return Zt(f)?'"'+f+'"':e(f)},"cleanVal"),n=o(function(f){return" "+f+" "},"space"),i=o(function(f,d){var p=f.type,m=f.value;switch(p){case $t.GROUP:{var g=e(m);return g.substring(0,g.length-1)}case $t.DATA_COMPARE:{var y=f.field,v=f.operator;return"["+y+n(e(v))+r(m)+"]"}case $t.DATA_BOOL:{var x=f.operator,b=f.field;return"["+e(x)+b+"]"}case $t.DATA_EXIST:{var w=f.field;return"["+w+"]"}case $t.META_COMPARE:{var C=f.operator,T=f.field;return"[["+T+n(e(C))+r(m)+"]]"}case $t.STATE:return m;case $t.ID:return"#"+m;case $t.CLASS:return"."+m;case $t.PARENT:case $t.CHILD:return a(f.parent,d)+n(">")+a(f.child,d);case $t.ANCESTOR:case $t.DESCENDANT:return a(f.ancestor,d)+" "+a(f.descendant,d);case $t.COMPOUND_SPLIT:{var E=a(f.left,d),A=a(f.subject,d),S=a(f.right,d);return E+(E.length>0?" ":"")+A+S}case $t.TRUE:return""}},"checkToString"),a=o(function(f,d){return f.checks.reduce(function(p,m,g){return p+(d===f&&g===0?"$":"")+i(m,d)},"")},"queryToString"),s="",l=0;l1&&l=0&&(r=r.replace("!",""),d=!0),r.indexOf("@")>=0&&(r=r.replace("@",""),f=!0),(a||l||f)&&(u=!a&&!s?"":""+e,h=""+n),f&&(e=u=u.toLowerCase(),n=h=h.toLowerCase()),r){case"*=":i=u.indexOf(h)>=0;break;case"$=":i=u.indexOf(h,u.length-h.length)>=0;break;case"^=":i=u.indexOf(h)===0;break;case"=":i=e===n;break;case">":p=!0,i=e>n;break;case">=":p=!0,i=e>=n;break;case"<":p=!0,i=e1&&arguments[1]!==void 0?arguments[1]:!0;return fB(this,t,e,Fme)};o($me,"addParent");Z1.forEachUp=function(t){var e=arguments.length>1&&arguments[1]!==void 0?arguments[1]:!0;return fB(this,t,e,$me)};o(dKe,"addParentAndChildren");Z1.forEachUpAndDown=function(t){var e=arguments.length>1&&arguments[1]!==void 0?arguments[1]:!0;return fB(this,t,e,dKe)};Z1.ancestors=Z1.parents;Kb=zme={data:cn.data({field:"data",bindingEvent:"data",allowBinding:!0,allowSetting:!0,settingEvent:"data",settingTriggersEvent:!0,triggerFnName:"trigger",allowGetting:!0,immutableKeys:{id:!0,source:!0,target:!0,parent:!0},updateStyle:!0}),removeData:cn.removeData({field:"data",event:"data",triggerFnName:"trigger",triggerEvent:!0,immutableKeys:{id:!0,source:!0,target:!0,parent:!0},updateStyle:!0}),scratch:cn.data({field:"scratch",bindingEvent:"scratch",allowBinding:!0,allowSetting:!0,settingEvent:"scratch",settingTriggersEvent:!0,triggerFnName:"trigger",allowGetting:!0,updateStyle:!0}),removeScratch:cn.removeData({field:"scratch",event:"scratch",triggerFnName:"trigger",triggerEvent:!0,updateStyle:!0}),rscratch:cn.data({field:"rscratch",allowBinding:!1,allowSetting:!0,settingTriggersEvent:!1,allowGetting:!0}),removeRscratch:cn.removeData({field:"rscratch",triggerEvent:!1}),id:o(function(){var e=this[0];if(e)return e._private.data.id},"id")};Kb.attr=Kb.data;Kb.removeAttr=Kb.removeData;pKe=zme,FS={};o(SP,"defineDegreeFunction");rr(FS,{degree:SP(function(t,e){return e.source().same(e.target())?2:1}),indegree:SP(function(t,e){return e.target().same(t)?1:0}),outdegree:SP(function(t,e){return e.source().same(t)?1:0})});o(F1,"defineDegreeBoundsFunction");rr(FS,{minDegree:F1("degree",function(t,e){return te}),minIndegree:F1("indegree",function(t,e){return te}),minOutdegree:F1("outdegree",function(t,e){return te})});rr(FS,{totalDegree:o(function(e){for(var r=0,n=this.nodes(),i=0;i0,p=d;d&&(f=f[0]);var m=p?f.position():{x:0,y:0};r!==void 0?h.position(e,r+m[e]):a!==void 0&&h.position({x:a.x+m.x,y:a.y+m.y})}else{var g=n.position(),y=l?n.parent():null,v=y&&y.length>0,x=v;v&&(y=y[0]);var b=x?y.position():{x:0,y:0};return a={x:g.x-b.x,y:g.y-b.y},e===void 0?a:a[e]}else if(!s)return;return this},"relativePosition")};Vl.modelPosition=Vl.point=Vl.position;Vl.modelPositions=Vl.points=Vl.positions;Vl.renderedPoint=Vl.renderedPosition;Vl.relativePoint=Vl.relativePosition;mKe=Gme;X1=Of={};Of.renderedBoundingBox=function(t){var e=this.boundingBox(t),r=this.cy(),n=r.zoom(),i=r.pan(),a=e.x1*n+i.x,s=e.x2*n+i.x,l=e.y1*n+i.y,u=e.y2*n+i.y;return{x1:a,x2:s,y1:l,y2:u,w:s-a,h:u-l}};Of.dirtyCompoundBoundsCache=function(){var t=arguments.length>0&&arguments[0]!==void 0?arguments[0]:!1,e=this.cy();return!e.styleEnabled()||!e.hasCompoundNodes()?this:(this.forEachUp(function(r){if(r.isParent()){var n=r._private;n.compoundBoundsClean=!1,n.bbCache=null,t||r.emitAndNotify("bounds")}}),this)};Of.updateCompoundBounds=function(){var t=arguments.length>0&&arguments[0]!==void 0?arguments[0]:!1,e=this.cy();if(!e.styleEnabled()||!e.hasCompoundNodes())return this;if(!t&&e.batching())return this;function r(s){if(!s.isParent())return;var l=s._private,u=s.children(),h=s.pstyle("compound-sizing-wrt-labels").value==="include",f={width:{val:s.pstyle("min-width").pfValue,left:s.pstyle("min-width-bias-left"),right:s.pstyle("min-width-bias-right")},height:{val:s.pstyle("min-height").pfValue,top:s.pstyle("min-height-bias-top"),bottom:s.pstyle("min-height-bias-bottom")}},d=u.boundingBox({includeLabels:h,includeOverlays:!1,useCache:!1}),p=l.position;(d.w===0||d.h===0)&&(d={w:s.pstyle("width").pfValue,h:s.pstyle("height").pfValue},d.x1=p.x-d.w/2,d.x2=p.x+d.w/2,d.y1=p.y-d.h/2,d.y2=p.y+d.h/2);function m(_,I,D){var k=0,L=0,R=I+D;return _>0&&R>0&&(k=I/R*_,L=D/R*_),{biasDiff:k,biasComplementDiff:L}}o(m,"computeBiasValues");function g(_,I,D,k){if(D.units==="%")switch(k){case"width":return _>0?D.pfValue*_:0;case"height":return I>0?D.pfValue*I:0;case"average":return _>0&&I>0?D.pfValue*(_+I)/2:0;case"min":return _>0&&I>0?_>I?D.pfValue*I:D.pfValue*_:0;case"max":return _>0&&I>0?_>I?D.pfValue*_:D.pfValue*I:0;default:return 0}else return D.units==="px"?D.pfValue:0}o(g,"computePaddingValues");var y=f.width.left.value;f.width.left.units==="px"&&f.width.val>0&&(y=y*100/f.width.val);var v=f.width.right.value;f.width.right.units==="px"&&f.width.val>0&&(v=v*100/f.width.val);var x=f.height.top.value;f.height.top.units==="px"&&f.height.val>0&&(x=x*100/f.height.val);var b=f.height.bottom.value;f.height.bottom.units==="px"&&f.height.val>0&&(b=b*100/f.height.val);var w=m(f.width.val-d.w,y,v),C=w.biasDiff,T=w.biasComplementDiff,E=m(f.height.val-d.h,x,b),A=E.biasDiff,S=E.biasComplementDiff;l.autoPadding=g(d.w,d.h,s.pstyle("padding"),s.pstyle("padding-relative-to").value),l.autoWidth=Math.max(d.w,f.width.val),p.x=(-C+d.x1+d.x2+T)/2,l.autoHeight=Math.max(d.h,f.height.val),p.y=(-A+d.y1+d.y2+S)/2}o(r,"update");for(var n=0;ne.x2?i:e.x2,e.y1=ne.y2?a:e.y2,e.w=e.x2-e.x1,e.h=e.y2-e.y1)},"updateBounds"),Pp=o(function(e,r){return r==null?e:zl(e,r.x1,r.y1,r.x2,r.y2)},"updateBoundsFromBox"),Db=o(function(e,r,n){return Gl(e,r,n)},"prefixedProperty"),tS=o(function(e,r,n){if(!r.cy().headless()){var i=r._private,a=i.rstyle,s=a.arrowWidth/2,l=r.pstyle(n+"-arrow-shape").value,u,h;if(l!=="none"){n==="source"?(u=a.srcX,h=a.srcY):n==="target"?(u=a.tgtX,h=a.tgtY):(u=a.midX,h=a.midY);var f=i.arrowBounds=i.arrowBounds||{},d=f[n]=f[n]||{};d.x1=u-s,d.y1=h-s,d.x2=u+s,d.y2=h+s,d.w=d.x2-d.x1,d.h=d.y2-d.y1,cS(d,1),zl(e,d.x1,d.y1,d.x2,d.y2)}}},"updateBoundsFromArrow"),CP=o(function(e,r,n){if(!r.cy().headless()){var i;n?i=n+"-":i="";var a=r._private,s=a.rstyle,l=r.pstyle(i+"label").strValue;if(l){var u=r.pstyle("text-halign"),h=r.pstyle("text-valign"),f=Db(s,"labelWidth",n),d=Db(s,"labelHeight",n),p=Db(s,"labelX",n),m=Db(s,"labelY",n),g=r.pstyle(i+"text-margin-x").pfValue,y=r.pstyle(i+"text-margin-y").pfValue,v=r.isEdge(),x=r.pstyle(i+"text-rotation"),b=r.pstyle("text-outline-width").pfValue,w=r.pstyle("text-border-width").pfValue,C=w/2,T=r.pstyle("text-background-padding").pfValue,E=2,A=d,S=f,_=S/2,I=A/2,D,k,L,R;if(v)D=p-_,k=p+_,L=m-I,R=m+I;else{switch(u.value){case"left":D=p-S,k=p;break;case"center":D=p-_,k=p+_;break;case"right":D=p,k=p+S;break}switch(h.value){case"top":L=m-A,R=m;break;case"center":L=m-I,R=m+I;break;case"bottom":L=m,R=m+A;break}}var O=g-Math.max(b,C)-T-E,M=g+Math.max(b,C)+T+E,B=y-Math.max(b,C)-T-E,F=y+Math.max(b,C)+T+E;D+=O,k+=M,L+=B,R+=F;var P=n||"main",z=a.labelBounds,$=z[P]=z[P]||{};$.x1=D,$.y1=L,$.x2=k,$.y2=R,$.w=k-D,$.h=R-L,$.leftPad=O,$.rightPad=M,$.topPad=B,$.botPad=F;var H=v&&x.strValue==="autorotate",Q=x.pfValue!=null&&x.pfValue!==0;if(H||Q){var j=H?Db(a.rstyle,"labelAngle",n):x.pfValue,ie=Math.cos(j),ne=Math.sin(j),le=(D+k)/2,he=(L+R)/2;if(!v){switch(u.value){case"left":le=k;break;case"right":le=D;break}switch(h.value){case"top":he=R;break;case"bottom":he=L;break}}var K=o(function(ce,ae){return ce=ce-le,ae=ae-he,{x:ce*ie-ae*ne+le,y:ce*ne+ae*ie+he}},"rotate"),X=K(D,L),te=K(D,R),J=K(k,L),se=K(k,R);D=Math.min(X.x,te.x,J.x,se.x),k=Math.max(X.x,te.x,J.x,se.x),L=Math.min(X.y,te.y,J.y,se.y),R=Math.max(X.y,te.y,J.y,se.y)}var ue=P+"Rot",Z=z[ue]=z[ue]||{};Z.x1=D,Z.y1=L,Z.x2=k,Z.y2=R,Z.w=k-D,Z.h=R-L,zl(e,D,L,k,R),zl(a.labelBounds.all,D,L,k,R)}return e}},"updateBoundsFromLabel"),gKe=o(function(e,r){if(!r.cy().headless()){var n=r.pstyle("outline-opacity").value,i=r.pstyle("outline-width").value;if(n>0&&i>0){var a=r.pstyle("outline-offset").value,s=r.pstyle("shape").value,l=i+a,u=(e.w+l*2)/e.w,h=(e.h+l*2)/e.h,f=0,d=0;["diamond","pentagon","round-triangle"].includes(s)?(u=(e.w+l*2.4)/e.w,d=-l/3.6):["concave-hexagon","rhomboid","right-rhomboid"].includes(s)?u=(e.w+l*2.4)/e.w:s==="star"?(u=(e.w+l*2.8)/e.w,h=(e.h+l*2.6)/e.h,d=-l/3.8):s==="triangle"?(u=(e.w+l*2.8)/e.w,h=(e.h+l*2.4)/e.h,d=-l/1.4):s==="vee"&&(u=(e.w+l*4.4)/e.w,h=(e.h+l*3.8)/e.h,d=-l*.5);var p=e.h*h-e.h,m=e.w*u-e.w;if(uS(e,[Math.ceil(p/2),Math.ceil(m/2)]),f!=0||d!==0){var g=Bqe(e,f,d);gme(e,g)}}}},"updateBoundsFromOutline"),yKe=o(function(e,r){var n=e._private.cy,i=n.styleEnabled(),a=n.headless(),s=Hs(),l=e._private,u=e.isNode(),h=e.isEdge(),f,d,p,m,g,y,v=l.rstyle,x=u&&i?e.pstyle("bounds-expansion").pfValue:[0],b=o(function(Se){return Se.pstyle("display").value!=="none"},"isDisplayed"),w=!i||b(e)&&(!h||b(e.source())&&b(e.target()));if(w){var C=0,T=0;i&&r.includeOverlays&&(C=e.pstyle("overlay-opacity").value,C!==0&&(T=e.pstyle("overlay-padding").value));var E=0,A=0;i&&r.includeUnderlays&&(E=e.pstyle("underlay-opacity").value,E!==0&&(A=e.pstyle("underlay-padding").value));var S=Math.max(T,A),_=0,I=0;if(i&&(_=e.pstyle("width").pfValue,I=_/2),u&&r.includeNodes){var D=e.position();g=D.x,y=D.y;var k=e.outerWidth(),L=k/2,R=e.outerHeight(),O=R/2;f=g-L,d=g+L,p=y-O,m=y+O,zl(s,f,p,d,m),i&&r.includeOutlines&&gKe(s,e)}else if(h&&r.includeEdges)if(i&&!a){var M=e.pstyle("curve-style").strValue;if(f=Math.min(v.srcX,v.midX,v.tgtX),d=Math.max(v.srcX,v.midX,v.tgtX),p=Math.min(v.srcY,v.midY,v.tgtY),m=Math.max(v.srcY,v.midY,v.tgtY),f-=I,d+=I,p-=I,m+=I,zl(s,f,p,d,m),M==="haystack"){var B=v.haystackPts;if(B&&B.length===2){if(f=B[0].x,p=B[0].y,d=B[1].x,m=B[1].y,f>d){var F=f;f=d,d=F}if(p>m){var P=p;p=m,m=P}zl(s,f-I,p-I,d+I,m+I)}}else if(M==="bezier"||M==="unbundled-bezier"||M.endsWith("segments")||M.endsWith("taxi")){var z;switch(M){case"bezier":case"unbundled-bezier":z=v.bezierPts;break;case"segments":case"taxi":case"round-segments":case"round-taxi":z=v.linePts;break}if(z!=null)for(var $=0;$d){var le=f;f=d,d=le}if(p>m){var he=p;p=m,m=he}f-=I,d+=I,p-=I,m+=I,zl(s,f,p,d,m)}if(i&&r.includeEdges&&h&&(tS(s,e,"mid-source"),tS(s,e,"mid-target"),tS(s,e,"source"),tS(s,e,"target")),i){var K=e.pstyle("ghost").value==="yes";if(K){var X=e.pstyle("ghost-offset-x").pfValue,te=e.pstyle("ghost-offset-y").pfValue;zl(s,s.x1+X,s.y1+te,s.x2+X,s.y2+te)}}var J=l.bodyBounds=l.bodyBounds||{};Fpe(J,s),uS(J,x),cS(J,1),i&&(f=s.x1,d=s.x2,p=s.y1,m=s.y2,zl(s,f-S,p-S,d+S,m+S));var se=l.overlayBounds=l.overlayBounds||{};Fpe(se,s),uS(se,x),cS(se,1);var ue=l.labelBounds=l.labelBounds||{};ue.all!=null?Pqe(ue.all):ue.all=Hs(),i&&r.includeLabels&&(r.includeMainLabels&&CP(s,e,null),h&&(r.includeSourceLabels&&CP(s,e,"source"),r.includeTargetLabels&&CP(s,e,"target")))}return s.x1=el(s.x1),s.y1=el(s.y1),s.x2=el(s.x2),s.y2=el(s.y2),s.w=el(s.x2-s.x1),s.h=el(s.y2-s.y1),s.w>0&&s.h>0&&w&&(uS(s,x),cS(s,1)),s},"boundingBoxImpl"),Ume=o(function(e){var r=0,n=o(function(s){return(s?1:0)<=0;l--)s(l);return this};Nf.removeAllListeners=function(){return this.removeListener("*")};Nf.emit=Nf.trigger=function(t,e,r){var n=this.listeners,i=n.length;return this.emitting++,En(e)||(e=[e]),MKe(this,function(a,s){r!=null&&(n=[{event:s.event,type:s.type,namespace:s.namespace,callback:r}],i=n.length);for(var l=o(function(f){var d=n[f];if(d.type===s.type&&(!d.namespace||d.namespace===s.namespace||d.namespace===RKe)&&a.eventMatches(a.context,d,s)){var p=[s];e!=null&&uqe(p,e),a.beforeEmit(a.context,d,s),d.conf&&d.conf.one&&(a.listeners=a.listeners.filter(function(y){return y!==d}));var m=a.callbackContext(a.context,d,s),g=d.callback.apply(m,p);a.afterEmit(a.context,d,s),g===!1&&(s.stopPropagation(),s.preventDefault())}},"_loop2"),u=0;u1&&!s){var l=this.length-1,u=this[l],h=u._private.data.id;this[l]=void 0,this[e]=u,a.set(h,{ele:u,index:e})}return this.length--,this},"unmergeAt"),unmergeOne:o(function(e){e=e[0];var r=this._private,n=e._private.data.id,i=r.map,a=i.get(n);if(!a)return this;var s=a.index;return this.unmergeAt(s),this},"unmergeOne"),unmerge:o(function(e){var r=this._private.cy;if(!e)return this;if(e&&Zt(e)){var n=e;e=r.mutableElements().filter(n)}for(var i=0;i=0;r--){var n=this[r];e(n)&&this.unmergeAt(r)}return this},"unmergeBy"),map:o(function(e,r){for(var n=[],i=this,a=0;an&&(n=u,i=l)}return{value:n,ele:i}},"max"),min:o(function(e,r){for(var n=1/0,i,a=this,s=0;s=0&&a"u"?"undefined":Wi(Symbol))!=e&&Wi(Symbol.iterator)!=e;r&&(ES[Symbol.iterator]=function(){var n=this,i={value:void 0,done:!1},a=0,s=this.length;return X0e({next:o(function(){return a1&&arguments[1]!==void 0?arguments[1]:!0,n=this[0],i=n.cy();if(i.styleEnabled()&&n){n._private.styleDirty&&(n._private.styleDirty=!1,i.style().apply(n));var a=n._private.style[e];return a??(r?i.style().getDefaultProperty(e):null)}},"parsedStyle"),numericStyle:o(function(e){var r=this[0];if(r.cy().styleEnabled()&&r){var n=r.pstyle(e);return n.pfValue!==void 0?n.pfValue:n.value}},"numericStyle"),numericStyleUnits:o(function(e){var r=this[0];if(r.cy().styleEnabled()&&r)return r.pstyle(e).units},"numericStyleUnits"),renderedStyle:o(function(e){var r=this.cy();if(!r.styleEnabled())return this;var n=this[0];if(n)return r.style().getRenderedStyle(n,e)},"renderedStyle"),style:o(function(e,r){var n=this.cy();if(!n.styleEnabled())return this;var i=!1,a=n.style();if(Ur(e)){var s=e;a.applyBypass(this,s,i),this.emitAndNotify("style")}else if(Zt(e))if(r===void 0){var l=this[0];return l?a.getStylePropertyValue(l,e):void 0}else a.applyBypass(this,e,r,i),this.emitAndNotify("style");else if(e===void 0){var u=this[0];return u?a.getRawStyle(u):void 0}return this},"style"),removeStyle:o(function(e){var r=this.cy();if(!r.styleEnabled())return this;var n=!1,i=r.style(),a=this;if(e===void 0)for(var s=0;s0&&e.push(f[0]),e.push(l[0])}return this.spawn(e,!0).filter(t)},"neighborhood"),closedNeighborhood:o(function(e){return this.neighborhood().add(this).filter(e)},"closedNeighborhood"),openNeighborhood:o(function(e){return this.neighborhood(e)},"openNeighborhood")});$a.neighbourhood=$a.neighborhood;$a.closedNeighbourhood=$a.closedNeighborhood;$a.openNeighbourhood=$a.openNeighborhood;rr($a,{source:tl(o(function(e){var r=this[0],n;return r&&(n=r._private.source||r.cy().collection()),n&&e?n.filter(e):n},"sourceImpl"),"source"),target:tl(o(function(e){var r=this[0],n;return r&&(n=r._private.target||r.cy().collection()),n&&e?n.filter(e):n},"targetImpl"),"target"),sources:g0e({attr:"source"}),targets:g0e({attr:"target"})});o(g0e,"defineSourceFunction");rr($a,{edgesWith:tl(y0e(),"edgesWith"),edgesTo:tl(y0e({thisIsSrc:!0}),"edgesTo")});o(y0e,"defineEdgesWithFunction");rr($a,{connectedEdges:tl(function(t){for(var e=[],r=this,n=0;n0);return s},"components"),component:o(function(){var e=this[0];return e.cy().mutableElements().components(e)[0]},"component")});$a.componentsOf=$a.components;ka=o(function(e,r){var n=arguments.length>2&&arguments[2]!==void 0?arguments[2]:!1,i=arguments.length>3&&arguments[3]!==void 0?arguments[3]:!1;if(e===void 0){ai("A collection must have a reference to the core");return}var a=new Xc,s=!1;if(!r)r=[];else if(r.length>0&&Ur(r[0])&&!t4(r[0])){s=!0;for(var l=[],u=new J1,h=0,f=r.length;h0&&arguments[0]!==void 0?arguments[0]:!0,e=arguments.length>1&&arguments[1]!==void 0?arguments[1]:!0,r=this,n=r.cy(),i=n._private,a=[],s=[],l,u=0,h=r.length;u0){for(var P=l.length===r.length?r:new ka(n,l),z=0;z0&&arguments[0]!==void 0?arguments[0]:!0,e=arguments.length>1&&arguments[1]!==void 0?arguments[1]:!0,r=this,n=[],i={},a=r._private.cy;function s(R){for(var O=R._private.edges,M=0;M0&&(t?D.emitAndNotify("remove"):e&&D.emit("remove"));for(var k=0;kf&&Math.abs(g.v)>f;);return p?function(y){return u[y*(u.length-1)|0]}:h},"springRK4Factory")}(),Nn=o(function(e,r,n,i){var a=UKe(e,r,n,i);return function(s,l,u){return s+(l-s)*a(u)}},"cubicBezier"),dS={linear:o(function(e,r,n){return e+(r-e)*n},"linear"),ease:Nn(.25,.1,.25,1),"ease-in":Nn(.42,0,1,1),"ease-out":Nn(0,0,.58,1),"ease-in-out":Nn(.42,0,.58,1),"ease-in-sine":Nn(.47,0,.745,.715),"ease-out-sine":Nn(.39,.575,.565,1),"ease-in-out-sine":Nn(.445,.05,.55,.95),"ease-in-quad":Nn(.55,.085,.68,.53),"ease-out-quad":Nn(.25,.46,.45,.94),"ease-in-out-quad":Nn(.455,.03,.515,.955),"ease-in-cubic":Nn(.55,.055,.675,.19),"ease-out-cubic":Nn(.215,.61,.355,1),"ease-in-out-cubic":Nn(.645,.045,.355,1),"ease-in-quart":Nn(.895,.03,.685,.22),"ease-out-quart":Nn(.165,.84,.44,1),"ease-in-out-quart":Nn(.77,0,.175,1),"ease-in-quint":Nn(.755,.05,.855,.06),"ease-out-quint":Nn(.23,1,.32,1),"ease-in-out-quint":Nn(.86,0,.07,1),"ease-in-expo":Nn(.95,.05,.795,.035),"ease-out-expo":Nn(.19,1,.22,1),"ease-in-out-expo":Nn(1,0,0,1),"ease-in-circ":Nn(.6,.04,.98,.335),"ease-out-circ":Nn(.075,.82,.165,1),"ease-in-out-circ":Nn(.785,.135,.15,.86),spring:o(function(e,r,n){if(n===0)return dS.linear;var i=HKe(e,r,n);return function(a,s,l){return a+(s-a)*i(l)}},"spring"),"cubic-bezier":Nn};o(x0e,"getEasedValue");o(b0e,"getValue");o($1,"ease");o(WKe,"step$1");o(Rb,"valid");o(qKe,"startAnimation");o(w0e,"stepAll");YKe={animate:cn.animate(),animation:cn.animation(),animated:cn.animated(),clearQueue:cn.clearQueue(),delay:cn.delay(),delayAnimation:cn.delayAnimation(),stop:cn.stop(),addToAnimationPool:o(function(e){var r=this;r.styleEnabled()&&r._private.aniEles.merge(e)},"addToAnimationPool"),stopAnimationLoop:o(function(){this._private.animationsRunning=!1},"stopAnimationLoop"),startAnimationLoop:o(function(){var e=this;if(e._private.animationsRunning=!0,!e.styleEnabled())return;function r(){e._private.animationsRunning&&xS(o(function(a){w0e(a,e),r()},"animationStep"))}o(r,"headlessStep");var n=e.renderer();n&&n.beforeRender?n.beforeRender(o(function(a,s){w0e(s,e)},"rendererAnimationStep"),n.beforeRenderPriorities.animations):r()},"startAnimationLoop")},XKe={qualifierCompare:o(function(e,r){return e==null||r==null?e==null&&r==null:e.sameText(r)},"qualifierCompare"),eventMatches:o(function(e,r,n){var i=r.qualifier;return i!=null?e!==n.target&&t4(n.target)&&i.matches(n.target):!0},"eventMatches"),addEventFields:o(function(e,r){r.cy=e,r.target=e},"addEventFields"),callbackContext:o(function(e,r,n){return r.qualifier!=null?n.target:e},"callbackContext")},iS=o(function(e){return Zt(e)?new Lf(e):e},"argSelector"),ege={createEmitter:o(function(){var e=this._private;return e.emitter||(e.emitter=new $S(XKe,this)),this},"createEmitter"),emitter:o(function(){return this._private.emitter},"emitter"),on:o(function(e,r,n){return this.emitter().on(e,iS(r),n),this},"on"),removeListener:o(function(e,r,n){return this.emitter().removeListener(e,iS(r),n),this},"removeListener"),removeAllListeners:o(function(){return this.emitter().removeAllListeners(),this},"removeAllListeners"),one:o(function(e,r,n){return this.emitter().one(e,iS(r),n),this},"one"),once:o(function(e,r,n){return this.emitter().one(e,iS(r),n),this},"once"),emit:o(function(e,r){return this.emitter().emit(e,r),this},"emit"),emitAndNotify:o(function(e,r){return this.emit(e),this.notify(e,r),this},"emitAndNotify")};cn.eventAliasesOn(ege);VP={png:o(function(e){var r=this._private.renderer;return e=e||{},r.png(e)},"png"),jpg:o(function(e){var r=this._private.renderer;return e=e||{},e.bg=e.bg||"#fff",r.jpg(e)},"jpg")};VP.jpeg=VP.jpg;pS={layout:o(function(e){var r=this;if(e==null){ai("Layout options must be specified to make a layout");return}if(e.name==null){ai("A `name` must be specified to make a layout");return}var n=e.name,i=r.extension("layout",n);if(i==null){ai("No such layout `"+n+"` found. Did you forget to import it and `cytoscape.use()` it?");return}var a;Zt(e.eles)?a=r.$(e.eles):a=e.eles!=null?e.eles:r.$();var s=new i(rr({},e,{cy:r,eles:a}));return s},"layout")};pS.createLayout=pS.makeLayout=pS.layout;jKe={notify:o(function(e,r){var n=this._private;if(this.batching()){n.batchNotifications=n.batchNotifications||{};var i=n.batchNotifications[e]=n.batchNotifications[e]||this.collection();r!=null&&i.merge(r);return}if(n.notificationsEnabled){var a=this.renderer();this.destroyed()||!a||a.notify(e,r)}},"notify"),notifications:o(function(e){var r=this._private;return e===void 0?r.notificationsEnabled:(r.notificationsEnabled=!!e,this)},"notifications"),noNotifications:o(function(e){this.notifications(!1),e(),this.notifications(!0)},"noNotifications"),batching:o(function(){return this._private.batchCount>0},"batching"),startBatch:o(function(){var e=this._private;return e.batchCount==null&&(e.batchCount=0),e.batchCount===0&&(e.batchStyleEles=this.collection(),e.batchNotifications={}),e.batchCount++,this},"startBatch"),endBatch:o(function(){var e=this._private;if(e.batchCount===0)return this;if(e.batchCount--,e.batchCount===0){e.batchStyleEles.updateStyle();var r=this.renderer();Object.keys(e.batchNotifications).forEach(function(n){var i=e.batchNotifications[n];i.empty()?r.notify(n):r.notify(n,i)})}return this},"endBatch"),batch:o(function(e){return this.startBatch(),e(),this.endBatch(),this},"batch"),batchData:o(function(e){var r=this;return this.batch(function(){for(var n=Object.keys(e),i=0;i0;)r.removeChild(r.childNodes[0]);e._private.renderer=null,e.mutableElements().forEach(function(n){var i=n._private;i.rscratch={},i.rstyle={},i.animation.current=[],i.animation.queue=[]})},"destroyRenderer"),onRender:o(function(e){return this.on("render",e)},"onRender"),offRender:o(function(e){return this.off("render",e)},"offRender")};UP.invalidateDimensions=UP.resize;mS={collection:o(function(e,r){return Zt(e)?this.$(e):go(e)?e.collection():En(e)?(r||(r={}),new ka(this,e,r.unique,r.removed)):new ka(this)},"collection"),nodes:o(function(e){var r=this.$(function(n){return n.isNode()});return e?r.filter(e):r},"nodes"),edges:o(function(e){var r=this.$(function(n){return n.isEdge()});return e?r.filter(e):r},"edges"),$:o(function(e){var r=this._private.elements;return e?r.filter(e):r.spawnSelf()},"$"),mutableElements:o(function(){return this._private.elements},"mutableElements")};mS.elements=mS.filter=mS.$;Ga={},$b="t",QKe="f";Ga.apply=function(t){for(var e=this,r=e._private,n=r.cy,i=n.collection(),a=0;a0;if(p||d&&m){var g=void 0;p&&m||p?g=h.properties:m&&(g=h.mappedProperties);for(var y=0;y1&&(C=1),l.color){var E=n.valueMin[0],A=n.valueMax[0],S=n.valueMin[1],_=n.valueMax[1],I=n.valueMin[2],D=n.valueMax[2],k=n.valueMin[3]==null?1:n.valueMin[3],L=n.valueMax[3]==null?1:n.valueMax[3],R=[Math.round(E+(A-E)*C),Math.round(S+(_-S)*C),Math.round(I+(D-I)*C),Math.round(k+(L-k)*C)];a={bypass:n.bypass,name:n.name,value:R,strValue:"rgb("+R[0]+", "+R[1]+", "+R[2]+")"}}else if(l.number){var O=n.valueMin+(n.valueMax-n.valueMin)*C;a=this.parse(n.name,O,n.bypass,p)}else return!1;if(!a)return y(),!1;a.mapping=n,n=a;break}case s.data:{for(var M=n.field.split("."),B=d.data,F=0;F0&&a>0){for(var l={},u=!1,h=0;h0?t.delayAnimation(s).play().promise().then(w):w()}).then(function(){return t.animation({style:l,duration:a,easing:t.pstyle("transition-timing-function").value,queue:!1}).play().promise()}).then(function(){r.removeBypasses(t,i),t.emitAndNotify("style"),n.transitioning=!1})}else n.transitioning&&(this.removeBypasses(t,i),t.emitAndNotify("style"),n.transitioning=!1)};Ga.checkTrigger=function(t,e,r,n,i,a){var s=this.properties[e],l=i(s);l!=null&&l(r,n)&&a(s)};Ga.checkZOrderTrigger=function(t,e,r,n){var i=this;this.checkTrigger(t,e,r,n,function(a){return a.triggersZOrder},function(){i._private.cy.notify("zorder",t)})};Ga.checkBoundsTrigger=function(t,e,r,n){this.checkTrigger(t,e,r,n,function(i){return i.triggersBounds},function(i){t.dirtyCompoundBoundsCache(),t.dirtyBoundingBoxCache(),i.triggersBoundsOfParallelBeziers&&e==="curve-style"&&(r==="bezier"||n==="bezier")&&t.parallelEdges().forEach(function(a){a.dirtyBoundingBoxCache()}),i.triggersBoundsOfConnectedEdges&&e==="display"&&(r==="none"||n==="none")&&t.connectedEdges().forEach(function(a){a.dirtyBoundingBoxCache()})})};Ga.checkTriggers=function(t,e,r,n){t.dirtyStyleCache(),this.checkZOrderTrigger(t,e,r,n),this.checkBoundsTrigger(t,e,r,n)};s4={};s4.applyBypass=function(t,e,r,n){var i=this,a=[],s=!0;if(e==="*"||e==="**"){if(r!==void 0)for(var l=0;li.length?n=n.substr(i.length):n=""}o(l,"removeSelAndBlockFromRemaining");function u(){a.length>s.length?a=a.substr(s.length):a=""}for(o(u,"removePropAndValFromRem");;){var h=n.match(/^\s*$/);if(h)break;var f=n.match(/^\s*((?:.|\s)+?)\s*\{((?:.|\s)+?)\}/);if(!f){un("Halting stylesheet parsing: String stylesheet contains more to parse but no selector and block found in: "+n);break}i=f[0];var d=f[1];if(d!=="core"){var p=new Lf(d);if(p.invalid){un("Skipping parsing of block: Invalid selector found in string stylesheet: "+d),l();continue}}var m=f[2],g=!1;a=m;for(var y=[];;){var v=a.match(/^\s*$/);if(v)break;var x=a.match(/^\s*(.+?)\s*:\s*(.+?)(?:\s*;|\s*$)/);if(!x){un("Skipping parsing of block: Invalid formatting of style property and value definitions found in:"+m),g=!0;break}s=x[0];var b=x[1],w=x[2],C=e.properties[b];if(!C){un("Skipping property: Invalid property name in: "+s),u();continue}var T=r.parse(b,w);if(!T){un("Skipping property: Invalid property definition in: "+s),u();continue}y.push({name:b,val:w}),u()}if(g){l();break}r.selector(d);for(var E=0;E=7&&e[0]==="d"&&(f=new RegExp(l.data.regex).exec(e))){if(r)return!1;var p=l.data;return{name:t,value:f,strValue:""+e,mapped:p,field:f[1],bypass:r}}else if(e.length>=10&&e[0]==="m"&&(d=new RegExp(l.mapData.regex).exec(e))){if(r||h.multiple)return!1;var m=l.mapData;if(!(h.color||h.number))return!1;var g=this.parse(t,d[4]);if(!g||g.mapped)return!1;var y=this.parse(t,d[5]);if(!y||y.mapped)return!1;if(g.pfValue===y.pfValue||g.strValue===y.strValue)return un("`"+t+": "+e+"` is not a valid mapper because the output range is zero; converting to `"+t+": "+g.strValue+"`"),this.parse(t,g.strValue);if(h.color){var v=g.value,x=y.value,b=v[0]===x[0]&&v[1]===x[1]&&v[2]===x[2]&&(v[3]===x[3]||(v[3]==null||v[3]===1)&&(x[3]==null||x[3]===1));if(b)return!1}return{name:t,value:d,strValue:""+e,mapped:m,field:d[1],fieldMin:parseFloat(d[2]),fieldMax:parseFloat(d[3]),valueMin:g.value,valueMax:y.value,bypass:r}}}if(h.multiple&&n!=="multiple"){var w;if(u?w=e.split(/\s+/):En(e)?w=e:w=[e],h.evenMultiple&&w.length%2!==0)return null;for(var C=[],T=[],E=[],A="",S=!1,_=0;_0?" ":"")+I.strValue}return h.validate&&!h.validate(C,T)?null:h.singleEnum&&S?C.length===1&&Zt(C[0])?{name:t,value:C[0],strValue:C[0],bypass:r}:null:{name:t,value:C,pfValue:E,strValue:A,bypass:r,units:T}}var D=o(function(){for(var K=0;Kh.max||h.strictMax&&e===h.max))return null;var M={name:t,value:e,strValue:""+e+(k||""),units:k,bypass:r};return h.unitless||k!=="px"&&k!=="em"?M.pfValue=e:M.pfValue=k==="px"||!k?e:this.getEmSizeInPixels()*e,(k==="ms"||k==="s")&&(M.pfValue=k==="ms"?e:1e3*e),(k==="deg"||k==="rad")&&(M.pfValue=k==="rad"?e:Nqe(e)),k==="%"&&(M.pfValue=e/100),M}else if(h.propList){var B=[],F=""+e;if(F!=="none"){for(var P=F.split(/\s*,\s*|\s+/),z=0;z0&&l>0&&!isNaN(n.w)&&!isNaN(n.h)&&n.w>0&&n.h>0){u=Math.min((s-2*r)/n.w,(l-2*r)/n.h),u=u>this._private.maxZoom?this._private.maxZoom:u,u=u=n.minZoom&&(n.maxZoom=r),this},"zoomRange"),minZoom:o(function(e){return e===void 0?this._private.minZoom:this.zoomRange({min:e})},"minZoom"),maxZoom:o(function(e){return e===void 0?this._private.maxZoom:this.zoomRange({max:e})},"maxZoom"),getZoomedViewport:o(function(e){var r=this._private,n=r.pan,i=r.zoom,a,s,l=!1;if(r.zoomingEnabled||(l=!0),Ct(e)?s=e:Ur(e)&&(s=e.level,e.position!=null?a=MS(e.position,i,n):e.renderedPosition!=null&&(a=e.renderedPosition),a!=null&&!r.panningEnabled&&(l=!0)),s=s>r.maxZoom?r.maxZoom:s,s=sr.maxZoom||!r.zoomingEnabled?s=!0:(r.zoom=u,a.push("zoom"))}if(i&&(!s||!e.cancelOnFailedZoom)&&r.panningEnabled){var h=e.pan;Ct(h.x)&&(r.pan.x=h.x,l=!1),Ct(h.y)&&(r.pan.y=h.y,l=!1),l||a.push("pan")}return a.length>0&&(a.push("viewport"),this.emit(a.join(" ")),this.notify("viewport")),this},"viewport"),center:o(function(e){var r=this.getCenterPan(e);return r&&(this._private.pan=r,this.emit("pan viewport"),this.notify("viewport")),this},"center"),getCenterPan:o(function(e,r){if(this._private.panningEnabled){if(Zt(e)){var n=e;e=this.mutableElements().filter(n)}else go(e)||(e=this.mutableElements());if(e.length!==0){var i=e.boundingBox(),a=this.width(),s=this.height();r=r===void 0?this._private.zoom:r;var l={x:(a-r*(i.x1+i.x2))/2,y:(s-r*(i.y1+i.y2))/2};return l}}},"getCenterPan"),reset:o(function(){return!this._private.panningEnabled||!this._private.zoomingEnabled?this:(this.viewport({pan:{x:0,y:0},zoom:1}),this)},"reset"),invalidateSize:o(function(){this._private.sizeCache=null},"invalidateSize"),size:o(function(){var e=this._private,r=e.container,n=this;return e.sizeCache=e.sizeCache||(r?function(){var i=n.window().getComputedStyle(r),a=o(function(l){return parseFloat(i.getPropertyValue(l))},"val");return{width:r.clientWidth-a("padding-left")-a("padding-right"),height:r.clientHeight-a("padding-top")-a("padding-bottom")}}():{width:1,height:1})},"size"),width:o(function(){return this.size().width},"width"),height:o(function(){return this.size().height},"height"),extent:o(function(){var e=this._private.pan,r=this._private.zoom,n=this.renderedExtent(),i={x1:(n.x1-e.x)/r,x2:(n.x2-e.x)/r,y1:(n.y1-e.y)/r,y2:(n.y2-e.y)/r};return i.w=i.x2-i.x1,i.h=i.y2-i.y1,i},"extent"),renderedExtent:o(function(){var e=this.width(),r=this.height();return{x1:0,y1:0,x2:e,y2:r,w:e,h:r}},"renderedExtent"),multiClickDebounceTime:o(function(e){if(e)this._private.multiClickDebounceTime=e;else return this._private.multiClickDebounceTime;return this},"multiClickDebounceTime")};Hp.centre=Hp.center;Hp.autolockNodes=Hp.autolock;Hp.autoungrabifyNodes=Hp.autoungrabify;Zb={data:cn.data({field:"data",bindingEvent:"data",allowBinding:!0,allowSetting:!0,settingEvent:"data",settingTriggersEvent:!0,triggerFnName:"trigger",allowGetting:!0,updateStyle:!0}),removeData:cn.removeData({field:"data",event:"data",triggerFnName:"trigger",triggerEvent:!0,updateStyle:!0}),scratch:cn.data({field:"scratch",bindingEvent:"scratch",allowBinding:!0,allowSetting:!0,settingEvent:"scratch",settingTriggersEvent:!0,triggerFnName:"trigger",allowGetting:!0,updateStyle:!0}),removeScratch:cn.removeData({field:"scratch",event:"scratch",triggerFnName:"trigger",triggerEvent:!0,updateStyle:!0})};Zb.attr=Zb.data;Zb.removeAttr=Zb.removeData;Jb=o(function(e){var r=this;e=rr({},e);var n=e.container;n&&!vS(n)&&vS(n[0])&&(n=n[0]);var i=n?n._cyreg:null;i=i||{},i&&i.cy&&(i.cy.destroy(),i={});var a=i.readies=i.readies||[];n&&(n._cyreg=i),i.cy=r;var s=Ui!==void 0&&n!==void 0&&!e.headless,l=e;l.layout=rr({name:s?"grid":"null"},l.layout),l.renderer=rr({name:s?"canvas":"null"},l.renderer);var u=o(function(g,y,v){return y!==void 0?y:v!==void 0?v:g},"defVal"),h=this._private={container:n,ready:!1,options:l,elements:new ka(this),listeners:[],aniEles:new ka(this),data:l.data||{},scratch:{},layout:null,renderer:null,destroyed:!1,notificationsEnabled:!0,minZoom:1e-50,maxZoom:1e50,zoomingEnabled:u(!0,l.zoomingEnabled),userZoomingEnabled:u(!0,l.userZoomingEnabled),panningEnabled:u(!0,l.panningEnabled),userPanningEnabled:u(!0,l.userPanningEnabled),boxSelectionEnabled:u(!0,l.boxSelectionEnabled),autolock:u(!1,l.autolock,l.autolockNodes),autoungrabify:u(!1,l.autoungrabify,l.autoungrabifyNodes),autounselectify:u(!1,l.autounselectify),styleEnabled:l.styleEnabled===void 0?s:l.styleEnabled,zoom:Ct(l.zoom)?l.zoom:1,pan:{x:Ur(l.pan)&&Ct(l.pan.x)?l.pan.x:0,y:Ur(l.pan)&&Ct(l.pan.y)?l.pan.y:0},animation:{current:[],queue:[]},hasCompoundNodes:!1,multiClickDebounceTime:u(250,l.multiClickDebounceTime)};this.createEmitter(),this.selectionType(l.selectionType),this.zoomRange({min:l.minZoom,max:l.maxZoom});var f=o(function(g,y){var v=g.some(nWe);if(v)return ey.all(g).then(y);y(g)},"loadExtData");h.styleEnabled&&r.setStyle([]);var d=rr({},l,l.renderer);r.initRenderer(d);var p=o(function(g,y,v){r.notifications(!1);var x=r.mutableElements();x.length>0&&x.remove(),g!=null&&(Ur(g)||En(g))&&r.add(g),r.one("layoutready",function(w){r.notifications(!0),r.emit(w),r.one("load",y),r.emitAndNotify("load")}).one("layoutstop",function(){r.one("done",v),r.emit("done")});var b=rr({},r._private.options.layout);b.eles=r.elements(),r.layout(b).run()},"setElesAndLayout");f([l.style,l.elements],function(m){var g=m[0],y=m[1];h.styleEnabled&&r.style().append(g),p(y,function(){r.startAnimationLoop(),h.ready=!0,si(l.ready)&&r.on("ready",l.ready);for(var v=0;v0,l=!!t.boundingBox,u=e.extent(),h=Hs(l?t.boundingBox:{x1:u.x1,y1:u.y1,w:u.w,h:u.h}),f;if(go(t.roots))f=t.roots;else if(En(t.roots)){for(var d=[],p=0;p0;){var O=R(),M=I(O,k);if(M)O.outgoers().filter(function(ae){return ae.isNode()&&r.has(ae)}).forEach(L);else if(M===null){un("Detected double maximal shift for node `"+O.id()+"`. Bailing maximal adjustment due to cycle. Use `options.maximal: true` only on DAGs.");break}}}var B=0;if(t.avoidOverlap)for(var F=0;F0&&b[0].length<=3?$e/2:0),Ie=2*Math.PI/b[ze].length*He;return ze===0&&b[0].length===1&&(Re=1),{x:se.x+Re*Math.cos(Ie),y:se.y+Re*Math.sin(Ie)}}else{var be=b[ze].length,W=Math.max(be===1?0:l?(h.w-t.padding*2-ue.w)/((t.grid?Se:be)-1):(h.w-t.padding*2-ue.w)/((t.grid?Se:be)+1),B),de={x:se.x+(He+1-(be+1)/2)*W,y:se.y+(ze+1-(ne+1)/2)*Z};return de}},"getPosition");return r.nodes().layoutPositions(this,t,ce),this};rQe={fit:!0,padding:30,boundingBox:void 0,avoidOverlap:!0,nodeDimensionsIncludeLabels:!1,spacingFactor:void 0,radius:void 0,startAngle:3/2*Math.PI,sweep:void 0,clockwise:!0,sort:void 0,animate:!1,animationDuration:500,animationEasing:void 0,animateFilter:o(function(e,r){return!0},"animateFilter"),ready:void 0,stop:void 0,transform:o(function(e,r){return r},"transform")};o(rge,"CircleLayout");rge.prototype.run=function(){var t=this.options,e=t,r=t.cy,n=e.eles,i=e.counterclockwise!==void 0?!e.counterclockwise:e.clockwise,a=n.nodes().not(":parent");e.sort&&(a=a.sort(e.sort));for(var s=Hs(e.boundingBox?e.boundingBox:{x1:0,y1:0,w:r.width(),h:r.height()}),l={x:s.x1+s.w/2,y:s.y1+s.h/2},u=e.sweep===void 0?2*Math.PI-2*Math.PI/a.length:e.sweep,h=u/Math.max(1,a.length-1),f,d=0,p=0;p1&&e.avoidOverlap){d*=1.75;var x=Math.cos(h)-Math.cos(0),b=Math.sin(h)-Math.sin(0),w=Math.sqrt(d*d/(x*x+b*b));f=Math.max(w,f)}var C=o(function(E,A){var S=e.startAngle+A*h*(i?1:-1),_=f*Math.cos(S),I=f*Math.sin(S),D={x:l.x+_,y:l.y+I};return D},"getPos");return n.nodes().layoutPositions(this,e,C),this};nQe={fit:!0,padding:30,startAngle:3/2*Math.PI,sweep:void 0,clockwise:!0,equidistant:!1,minNodeSpacing:10,boundingBox:void 0,avoidOverlap:!0,nodeDimensionsIncludeLabels:!1,height:void 0,width:void 0,spacingFactor:void 0,concentric:o(function(e){return e.degree()},"concentric"),levelWidth:o(function(e){return e.maxDegree()/4},"levelWidth"),animate:!1,animationDuration:500,animationEasing:void 0,animateFilter:o(function(e,r){return!0},"animateFilter"),ready:void 0,stop:void 0,transform:o(function(e,r){return r},"transform")};o(nge,"ConcentricLayout");nge.prototype.run=function(){for(var t=this.options,e=t,r=e.counterclockwise!==void 0?!e.counterclockwise:e.clockwise,n=t.cy,i=e.eles,a=i.nodes().not(":parent"),s=Hs(e.boundingBox?e.boundingBox:{x1:0,y1:0,w:n.width(),h:n.height()}),l={x:s.x1+s.w/2,y:s.y1+s.h/2},u=[],h=0,f=0;f0){var T=Math.abs(b[0].value-C.value);T>=v&&(b=[],x.push(b))}b.push(C)}var E=h+e.minNodeSpacing;if(!e.avoidOverlap){var A=x.length>0&&x[0].length>1,S=Math.min(s.w,s.h)/2-E,_=S/(x.length+A?1:0);E=Math.min(E,_)}for(var I=0,D=0;D1&&e.avoidOverlap){var O=Math.cos(R)-Math.cos(0),M=Math.sin(R)-Math.sin(0),B=Math.sqrt(E*E/(O*O+M*M));I=Math.max(B,I)}k.r=I,I+=E}if(e.equidistant){for(var F=0,P=0,z=0;z=t.numIter||(hQe(n,t),n.temperature=n.temperature*t.coolingFactor,n.temperature=t.animationThreshold&&a(),xS(d)}},"frame");f()}else{for(;h;)h=s(u),u++;E0e(n,t),l()}return this};HS.prototype.stop=function(){return this.stopped=!0,this.thread&&this.thread.stop(),this.emit("layoutstop"),this};HS.prototype.destroy=function(){return this.thread&&this.thread.stop(),this};aQe=o(function(e,r,n){for(var i=n.eles.edges(),a=n.eles.nodes(),s=Hs(n.boundingBox?n.boundingBox:{x1:0,y1:0,w:e.width(),h:e.height()}),l={isCompound:e.hasCompoundNodes(),layoutNodes:[],idToIndex:{},nodeSize:a.size(),graphSet:[],indexToGraph:[],layoutEdges:[],edgeSize:i.size(),temperature:n.initialTemp,clientWidth:s.w,clientHeight:s.h,boundingBox:s},u=n.eles.components(),h={},f=0;f0){l.graphSet.push(S);for(var f=0;fi.count?0:i.graph},"findLCA"),oQe=o(function t(e,r,n,i){var a=i.graphSet[n];if(-10)var d=i.nodeOverlap*f,p=Math.sqrt(l*l+u*u),m=d*l/p,g=d*u/p;else var y=CS(e,l,u),v=CS(r,-1*l,-1*u),x=v.x-y.x,b=v.y-y.y,w=x*x+b*b,p=Math.sqrt(w),d=(e.nodeRepulsion+r.nodeRepulsion)/w,m=d*x/p,g=d*b/p;e.isLocked||(e.offsetX-=m,e.offsetY-=g),r.isLocked||(r.offsetX+=m,r.offsetY+=g)}},"nodeRepulsion"),pQe=o(function(e,r,n,i){if(n>0)var a=e.maxX-r.minX;else var a=r.maxX-e.minX;if(i>0)var s=e.maxY-r.minY;else var s=r.maxY-e.minY;return a>=0&&s>=0?Math.sqrt(a*a+s*s):0},"nodesOverlap"),CS=o(function(e,r,n){var i=e.positionX,a=e.positionY,s=e.height||1,l=e.width||1,u=n/r,h=s/l,f={};return r===0&&0n?(f.x=i,f.y=a+s/2,f):0r&&-1*h<=u&&u<=h?(f.x=i-l/2,f.y=a-l*n/2/r,f):0=h)?(f.x=i+s*r/2/n,f.y=a+s/2,f):(0>n&&(u<=-1*h||u>=h)&&(f.x=i-s*r/2/n,f.y=a-s/2),f)},"findClippingPoint"),mQe=o(function(e,r){for(var n=0;nn){var v=r.gravity*m/y,x=r.gravity*g/y;p.offsetX+=v,p.offsetY+=x}}}}},"calculateGravityForces"),yQe=o(function(e,r){var n=[],i=0,a=-1;for(n.push.apply(n,e.graphSet[0]),a+=e.graphSet[0].length;i<=a;){var s=n[i++],l=e.idToIndex[s],u=e.layoutNodes[l],h=u.children;if(0n)var a={x:n*e/i,y:n*r/i};else var a={x:e,y:r};return a},"limitForce"),bQe=o(function t(e,r){var n=e.parentId;if(n!=null){var i=r.layoutNodes[r.idToIndex[n]],a=!1;if((i.maxX==null||e.maxX+i.padRight>i.maxX)&&(i.maxX=e.maxX+i.padRight,a=!0),(i.minX==null||e.minX-i.padLefti.maxY)&&(i.maxY=e.maxY+i.padBottom,a=!0),(i.minY==null||e.minY-i.padTopx&&(g+=v+r.componentSpacing,m=0,y=0,v=0)}}},"separateComponents"),wQe={fit:!0,padding:30,boundingBox:void 0,avoidOverlap:!0,avoidOverlapPadding:10,nodeDimensionsIncludeLabels:!1,spacingFactor:void 0,condense:!1,rows:void 0,cols:void 0,position:o(function(e){},"position"),sort:void 0,animate:!1,animationDuration:500,animationEasing:void 0,animateFilter:o(function(e,r){return!0},"animateFilter"),ready:void 0,stop:void 0,transform:o(function(e,r){return r},"transform")};o(age,"GridLayout");age.prototype.run=function(){var t=this.options,e=t,r=t.cy,n=e.eles,i=n.nodes().not(":parent");e.sort&&(i=i.sort(e.sort));var a=Hs(e.boundingBox?e.boundingBox:{x1:0,y1:0,w:r.width(),h:r.height()});if(a.h===0||a.w===0)n.nodes().layoutPositions(this,e,function(Q){return{x:a.x1,y:a.y1}});else{var s=i.size(),l=Math.sqrt(s*a.h/a.w),u=Math.round(l),h=Math.round(a.w/a.h*l),f=o(function(j){if(j==null)return Math.min(u,h);var ie=Math.min(u,h);ie==u?u=j:h=j},"small"),d=o(function(j){if(j==null)return Math.max(u,h);var ie=Math.max(u,h);ie==u?u=j:h=j},"large"),p=e.rows,m=e.cols!=null?e.cols:e.columns;if(p!=null&&m!=null)u=p,h=m;else if(p!=null&&m==null)u=p,h=Math.ceil(s/u);else if(p==null&&m!=null)h=m,u=Math.ceil(s/h);else if(h*u>s){var g=f(),y=d();(g-1)*y>=s?f(g-1):(y-1)*g>=s&&d(y-1)}else for(;h*u=s?d(x+1):f(v+1)}var b=a.w/h,w=a.h/u;if(e.condense&&(b=0,w=0),e.avoidOverlap)for(var C=0;C=h&&(O=0,R++)},"moveToNextCell"),B={},F=0;F(O=Wqe(t,e,M[B],M[B+1],M[B+2],M[B+3])))return v(A,O),!0}else if(_.edgeType==="bezier"||_.edgeType==="multibezier"||_.edgeType==="self"||_.edgeType==="compound"){for(var M=_.allpts,B=0;B+5<_.allpts.length;B+=4)if(Gqe(t,e,M[B],M[B+1],M[B+2],M[B+3],M[B+4],M[B+5],R)&&L>(O=Hqe(t,e,M[B],M[B+1],M[B+2],M[B+3],M[B+4],M[B+5])))return v(A,O),!0}for(var F=F||S.source,P=P||S.target,z=i.getArrowWidth(I,D),$=[{name:"source",x:_.arrowStartX,y:_.arrowStartY,angle:_.srcArrowAngle},{name:"target",x:_.arrowEndX,y:_.arrowEndY,angle:_.tgtArrowAngle},{name:"mid-source",x:_.midX,y:_.midY,angle:_.midsrcArrowAngle},{name:"mid-target",x:_.midX,y:_.midY,angle:_.midtgtArrowAngle}],B=0;B<$.length;B++){var H=$[B],Q=a.arrowShapes[A.pstyle(H.name+"-arrow-shape").value],j=A.pstyle("width").pfValue;if(Q.roughCollide(t,e,z,H.angle,{x:H.x,y:H.y},j,f)&&Q.collide(t,e,z,H.angle,{x:H.x,y:H.y},j,f))return v(A),!0}h&&l.length>0&&(x(F),x(P))}o(b,"checkEdge");function w(A,S,_){return Gl(A,S,_)}o(w,"preprop");function C(A,S){var _=A._private,I=p,D;S?D=S+"-":D="",A.boundingBox();var k=_.labelBounds[S||"main"],L=A.pstyle(D+"label").value,R=A.pstyle("text-events").strValue==="yes";if(!(!R||!L)){var O=w(_.rscratch,"labelX",S),M=w(_.rscratch,"labelY",S),B=w(_.rscratch,"labelAngle",S),F=A.pstyle(D+"text-margin-x").pfValue,P=A.pstyle(D+"text-margin-y").pfValue,z=k.x1-I-F,$=k.x2+I-F,H=k.y1-I-P,Q=k.y2+I-P;if(B){var j=Math.cos(B),ie=Math.sin(B),ne=o(function(se,ue){return se=se-O,ue=ue-M,{x:se*j-ue*ie+O,y:se*ie+ue*j+M}},"rotate"),le=ne(z,H),he=ne(z,Q),K=ne($,H),X=ne($,Q),te=[le.x+F,le.y+P,K.x+F,K.y+P,X.x+F,X.y+P,he.x+F,he.y+P];if(Us(t,e,te))return v(A),!0}else if(K1(k,t,e))return v(A),!0}}o(C,"checkLabel");for(var T=s.length-1;T>=0;T--){var E=s[T];E.isNode()?x(E)||C(E):b(E)||C(E)||C(E,"source")||C(E,"target")}return l};qp.getAllInBox=function(t,e,r,n){var i=this.getCachedZSortedEles().interactive,a=[],s=Math.min(t,r),l=Math.max(t,r),u=Math.min(e,n),h=Math.max(e,n);t=s,r=l,e=u,n=h;for(var f=Hs({x1:t,y1:e,x2:r,y2:n}),d=0;d0?-(Math.PI-e.ang):Math.PI+e.ang},"invertVec"),AQe=o(function(e,r,n,i,a){if(e!==D0e?L0e(r,e,qc):CQe(Jo,qc),L0e(r,n,Jo),A0e=qc.nx*Jo.ny-qc.ny*Jo.nx,_0e=qc.nx*Jo.nx-qc.ny*-Jo.ny,Ku=Math.asin(Math.max(-1,Math.min(1,A0e))),Math.abs(Ku)<1e-6){HP=r.x,WP=r.y,Bp=G1=0;return}Fp=1,gS=!1,_0e<0?Ku<0?Ku=Math.PI+Ku:(Ku=Math.PI-Ku,Fp=-1,gS=!0):Ku>0&&(Fp=-1,gS=!0),r.radius!==void 0?G1=r.radius:G1=i,Mp=Ku/2,aS=Math.min(qc.len/2,Jo.len/2),a?(Wc=Math.abs(Math.cos(Mp)*G1/Math.sin(Mp)),Wc>aS?(Wc=aS,Bp=Math.abs(Wc*Math.sin(Mp)/Math.cos(Mp))):Bp=G1):(Wc=Math.min(aS,G1),Bp=Math.abs(Wc*Math.sin(Mp)/Math.cos(Mp))),qP=r.x+Jo.nx*Wc,YP=r.y+Jo.ny*Wc,HP=qP-Jo.ny*Bp*Fp,WP=YP+Jo.nx*Bp*Fp,cge=r.x+qc.nx*Wc,uge=r.y+qc.ny*Wc,D0e=r},"calcCornerArc");o(hge,"drawPreparedRoundCorner");o(vB,"getRoundCorner");Va={};Va.findMidptPtsEtc=function(t,e){var r=e.posPts,n=e.intersectionPts,i=e.vectorNormInverse,a,s=t.pstyle("source-endpoint"),l=t.pstyle("target-endpoint"),u=s.units!=null&&l.units!=null,h=o(function(T,E,A,S){var _=S-E,I=A-T,D=Math.sqrt(I*I+_*_);return{x:-_/D,y:I/D}},"recalcVectorNormInverse"),f=t.pstyle("edge-distances").value;switch(f){case"node-position":a=r;break;case"intersection":a=n;break;case"endpoints":{if(u){var d=this.manualEndptToPx(t.source()[0],s),p=_i(d,2),m=p[0],g=p[1],y=this.manualEndptToPx(t.target()[0],l),v=_i(y,2),x=v[0],b=v[1],w={x1:m,y1:g,x2:x,y2:b};i=h(m,g,x,b),a=w}else un("Edge ".concat(t.id()," has edge-distances:endpoints specified without manual endpoints specified via source-endpoint and target-endpoint. Falling back on edge-distances:intersection (default).")),a=n;break}}return{midptPts:a,vectorNormInverse:i}};Va.findHaystackPoints=function(t){for(var e=0;e0?Math.max(q-pe,0):Math.min(q+pe,0)},"subDWH"),L=k(I,S),R=k(D,_),O=!1;b===h?x=Math.abs(L)>Math.abs(R)?i:n:b===u||b===l?(x=n,O=!0):(b===a||b===s)&&(x=i,O=!0);var M=x===n,B=M?R:L,F=M?D:I,P=mme(F),z=!1;!(O&&(C||E))&&(b===l&&F<0||b===u&&F>0||b===a&&F>0||b===s&&F<0)&&(P*=-1,B=P*Math.abs(B),z=!0);var $;if(C){var H=T<0?1+T:T;$=H*B}else{var Q=T<0?B:0;$=Q+T*P}var j=o(function(q){return Math.abs(q)=Math.abs(B)},"getIsTooClose"),ie=j($),ne=j(Math.abs(B)-Math.abs($)),le=ie||ne;if(le&&!z)if(M){var he=Math.abs(F)<=p/2,K=Math.abs(I)<=m/2;if(he){var X=(f.x1+f.x2)/2,te=f.y1,J=f.y2;r.segpts=[X,te,X,J]}else if(K){var se=(f.y1+f.y2)/2,ue=f.x1,Z=f.x2;r.segpts=[ue,se,Z,se]}else r.segpts=[f.x1,f.y2]}else{var Se=Math.abs(F)<=d/2,ce=Math.abs(D)<=g/2;if(Se){var ae=(f.y1+f.y2)/2,Oe=f.x1,ge=f.x2;r.segpts=[Oe,ae,ge,ae]}else if(ce){var ze=(f.x1+f.x2)/2,He=f.y1,$e=f.y2;r.segpts=[ze,He,ze,$e]}else r.segpts=[f.x2,f.y1]}else if(M){var Re=f.y1+$+(v?p/2*P:0),Ie=f.x1,be=f.x2;r.segpts=[Ie,Re,be,Re]}else{var W=f.x1+$+(v?d/2*P:0),de=f.y1,re=f.y2;r.segpts=[W,de,W,re]}if(r.isRound){var oe=t.pstyle("taxi-radius").value,V=t.pstyle("radius-type").value[0]==="arc-radius";r.radii=new Array(r.segpts.length/2).fill(oe),r.isArcRadius=new Array(r.segpts.length/2).fill(V)}};Va.tryToCorrectInvalidPoints=function(t,e){var r=t._private.rscratch;if(r.edgeType==="bezier"){var n=e.srcPos,i=e.tgtPos,a=e.srcW,s=e.srcH,l=e.tgtW,u=e.tgtH,h=e.srcShape,f=e.tgtShape,d=e.srcCornerRadius,p=e.tgtCornerRadius,m=e.srcRs,g=e.tgtRs,y=!Ct(r.startX)||!Ct(r.startY),v=!Ct(r.arrowStartX)||!Ct(r.arrowStartY),x=!Ct(r.endX)||!Ct(r.endY),b=!Ct(r.arrowEndX)||!Ct(r.arrowEndY),w=3,C=this.getArrowWidth(t.pstyle("width").pfValue,t.pstyle("arrow-scale").value)*this.arrowShapeWidth,T=w*C,E=Gp({x:r.ctrlpts[0],y:r.ctrlpts[1]},{x:r.startX,y:r.startY}),A=ER.poolIndex()){var O=L;L=R,R=O}var M=_.srcPos=L.position(),B=_.tgtPos=R.position(),F=_.srcW=L.outerWidth(),P=_.srcH=L.outerHeight(),z=_.tgtW=R.outerWidth(),$=_.tgtH=R.outerHeight(),H=_.srcShape=r.nodeShapes[e.getNodeShape(L)],Q=_.tgtShape=r.nodeShapes[e.getNodeShape(R)],j=_.srcCornerRadius=L.pstyle("corner-radius").value==="auto"?"auto":L.pstyle("corner-radius").pfValue,ie=_.tgtCornerRadius=R.pstyle("corner-radius").value==="auto"?"auto":R.pstyle("corner-radius").pfValue,ne=_.tgtRs=R._private.rscratch,le=_.srcRs=L._private.rscratch;_.dirCounts={north:0,west:0,south:0,east:0,northwest:0,southwest:0,northeast:0,southeast:0};for(var he=0;he<_.eles.length;he++){var K=_.eles[he],X=K[0]._private.rscratch,te=K.pstyle("curve-style").value,J=te==="unbundled-bezier"||te.endsWith("segments")||te.endsWith("taxi"),se=!L.same(K.source());if(!_.calculatedIntersection&&L!==R&&(_.hasBezier||_.hasUnbundled)){_.calculatedIntersection=!0;var ue=H.intersectLine(M.x,M.y,F,P,B.x,B.y,0,j,le),Z=_.srcIntn=ue,Se=Q.intersectLine(B.x,B.y,z,$,M.x,M.y,0,ie,ne),ce=_.tgtIntn=Se,ae=_.intersectionPts={x1:ue[0],x2:Se[0],y1:ue[1],y2:Se[1]},Oe=_.posPts={x1:M.x,x2:B.x,y1:M.y,y2:B.y},ge=Se[1]-ue[1],ze=Se[0]-ue[0],He=Math.sqrt(ze*ze+ge*ge),$e=_.vector={x:ze,y:ge},Re=_.vectorNorm={x:$e.x/He,y:$e.y/He},Ie={x:-Re.y,y:Re.x};_.nodesOverlap=!Ct(He)||Q.checkPoint(ue[0],ue[1],0,z,$,B.x,B.y,ie,ne)||H.checkPoint(Se[0],Se[1],0,F,P,M.x,M.y,j,le),_.vectorNormInverse=Ie,I={nodesOverlap:_.nodesOverlap,dirCounts:_.dirCounts,calculatedIntersection:!0,hasBezier:_.hasBezier,hasUnbundled:_.hasUnbundled,eles:_.eles,srcPos:B,srcRs:ne,tgtPos:M,tgtRs:le,srcW:z,srcH:$,tgtW:F,tgtH:P,srcIntn:ce,tgtIntn:Z,srcShape:Q,tgtShape:H,posPts:{x1:Oe.x2,y1:Oe.y2,x2:Oe.x1,y2:Oe.y1},intersectionPts:{x1:ae.x2,y1:ae.y2,x2:ae.x1,y2:ae.y1},vector:{x:-$e.x,y:-$e.y},vectorNorm:{x:-Re.x,y:-Re.y},vectorNormInverse:{x:-Ie.x,y:-Ie.y}}}var be=se?I:_;X.nodesOverlap=be.nodesOverlap,X.srcIntn=be.srcIntn,X.tgtIntn=be.tgtIntn,X.isRound=te.startsWith("round"),i&&(L.isParent()||L.isChild()||R.isParent()||R.isChild())&&(L.parents().anySame(R)||R.parents().anySame(L)||L.same(R)&&L.isParent())?e.findCompoundLoopPoints(K,be,he,J):L===R?e.findLoopPoints(K,be,he,J):te.endsWith("segments")?e.findSegmentsPoints(K,be):te.endsWith("taxi")?e.findTaxiPoints(K,be):te==="straight"||!J&&_.eles.length%2===1&&he===Math.floor(_.eles.length/2)?e.findStraightEdgePoints(K):e.findBezierPoints(K,be,he,J,se),e.findEndpoints(K),e.tryToCorrectInvalidPoints(K,be),e.checkForInvalidEdgeWarning(K),e.storeAllpts(K),e.storeEdgeProjections(K),e.calculateArrowAngles(K),e.recalculateEdgeLabelProjections(K),e.calculateLabelAngles(K)}},"_loop"),T=0;T0){var J=a,se=Op(J,U1(r)),ue=Op(J,U1(te)),Z=se;if(ue2){var Se=Op(J,{x:te[2],y:te[3]});Se0){var re=s,oe=Op(re,U1(r)),V=Op(re,U1(de)),xe=oe;if(V2){var q=Op(re,{x:de[2],y:de[3]});q=g||A){v={cp:C,segment:E};break}}if(v)break}var S=v.cp,_=v.segment,I=(g-x)/_.length,D=_.t1-_.t0,k=m?_.t0+D*I:_.t1-D*I;k=Yb(0,k,1),e=W1(S.p0,S.p1,S.p2,k),p=DQe(S.p0,S.p1,S.p2,k);break}case"straight":case"segments":case"haystack":{for(var L=0,R,O,M,B,F=n.allpts.length,P=0;P+3=g));P+=2);var z=g-O,$=z/R;$=Yb(0,$,1),e=Iqe(M,B,$),p=pge(M,B);break}}s("labelX",d,e.x),s("labelY",d,e.y),s("labelAutoAngle",d,p)}},"calculateEndProjection");h("source"),h("target"),this.applyLabelDimensions(t)}};Kc.applyLabelDimensions=function(t){this.applyPrefixedLabelDimensions(t),t.isEdge()&&(this.applyPrefixedLabelDimensions(t,"source"),this.applyPrefixedLabelDimensions(t,"target"))};Kc.applyPrefixedLabelDimensions=function(t,e){var r=t._private,n=this.getLabelText(t,e),i=this.calculateLabelDimensions(t,n),a=t.pstyle("line-height").pfValue,s=t.pstyle("text-wrap").strValue,l=Gl(r.rscratch,"labelWrapCachedLines",e)||[],u=s!=="wrap"?1:Math.max(l.length,1),h=i.height/u,f=h*a,d=i.width,p=i.height+(u-1)*(a-1)*h;kf(r.rstyle,"labelWidth",e,d),kf(r.rscratch,"labelWidth",e,d),kf(r.rstyle,"labelHeight",e,p),kf(r.rscratch,"labelHeight",e,p),kf(r.rscratch,"labelLineHeight",e,f)};Kc.getLabelText=function(t,e){var r=t._private,n=e?e+"-":"",i=t.pstyle(n+"label").strValue,a=t.pstyle("text-transform").value,s=o(function(Q,j){return j?(kf(r.rscratch,Q,e,j),j):Gl(r.rscratch,Q,e)},"rscratch");if(!i)return"";a=="none"||(a=="uppercase"?i=i.toUpperCase():a=="lowercase"&&(i=i.toLowerCase()));var l=t.pstyle("text-wrap").value;if(l==="wrap"){var u=s("labelKey");if(u!=null&&s("labelWrapKey")===u)return s("labelWrapCachedText");for(var h="\u200B",f=i.split(` +`),d=t.pstyle("text-max-width").pfValue,p=t.pstyle("text-overflow-wrap").value,m=p==="anywhere",g=[],y=/[\s\u200b]+|$/g,v=0;vd){var T=x.matchAll(y),E="",A=0,S=mo(T),_;try{for(S.s();!(_=S.n()).done;){var I=_.value,D=I[0],k=x.substring(A,I.index);A=I.index+D.length;var L=E.length===0?k:E+k+D,R=this.calculateLabelDimensions(t,L),O=R.width;O<=d?E+=k+D:(E&&g.push(E),E=k+D)}}catch(H){S.e(H)}finally{S.f()}E.match(/^[\s\u200b]+$/)||g.push(E)}else g.push(x)}s("labelWrapCachedLines",g),i=s("labelWrapCachedText",g.join(` +`)),s("labelWrapKey",u)}else if(l==="ellipsis"){var M=t.pstyle("text-max-width").pfValue,B="",F="\u2026",P=!1;if(this.calculateLabelDimensions(t,i).widthM)break;B+=i[z],z===i.length-1&&(P=!0)}return P||(B+=F),B}return i};Kc.getLabelJustification=function(t){var e=t.pstyle("text-justification").strValue,r=t.pstyle("text-halign").strValue;if(e==="auto")if(t.isNode())switch(r){case"left":return"right";case"right":return"left";default:return"center"}else return"center";else return e};Kc.calculateLabelDimensions=function(t,e){var r=this,n=r.cy.window(),i=n.document,a=_f(e,t._private.labelDimsKey),s=r.labelDimCache||(r.labelDimCache=[]),l=s[a];if(l!=null)return l;var u=0,h=t.pstyle("font-style").strValue,f=t.pstyle("font-size").pfValue,d=t.pstyle("font-family").strValue,p=t.pstyle("font-weight").strValue,m=this.labelCalcCanvas,g=this.labelCalcCanvasContext;if(!m){m=this.labelCalcCanvas=i.createElement("canvas"),g=this.labelCalcCanvasContext=m.getContext("2d");var y=m.style;y.position="absolute",y.left="-9999px",y.top="-9999px",y.zIndex="-1",y.visibility="hidden",y.pointerEvents="none"}g.font="".concat(h," ").concat(p," ").concat(f,"px ").concat(d);for(var v=0,x=0,b=e.split(` +`),w=0;w1&&arguments[1]!==void 0?arguments[1]:!0;if(e.merge(s),l)for(var u=0;u=t.desktopTapThreshold2}var ot=a(W);at&&(t.hoverData.tapholdCancelled=!0);var Yt=o(function(){var Tt=t.hoverData.dragDelta=t.hoverData.dragDelta||[];Tt.length===0?(Tt.push(De[0]),Tt.push(De[1])):(Tt[0]+=De[0],Tt[1]+=De[1])},"updateDragDelta");re=!0,i(_e,["mousemove","vmousemove","tapdrag"],W,{x:q[0],y:q[1]});var bt=o(function(){t.data.bgActivePosistion=void 0,t.hoverData.selecting||oe.emit({originalEvent:W,type:"boxstart",position:{x:q[0],y:q[1]}}),Pe[4]=1,t.hoverData.selecting=!0,t.redrawHint("select",!0),t.redraw()},"goIntoBoxMode");if(t.hoverData.which===3){if(at){var Mt={originalEvent:W,type:"cxtdrag",position:{x:q[0],y:q[1]}};Ve?Ve.emit(Mt):oe.emit(Mt),t.hoverData.cxtDragged=!0,(!t.hoverData.cxtOver||_e!==t.hoverData.cxtOver)&&(t.hoverData.cxtOver&&t.hoverData.cxtOver.emit({originalEvent:W,type:"cxtdragout",position:{x:q[0],y:q[1]}}),t.hoverData.cxtOver=_e,_e&&_e.emit({originalEvent:W,type:"cxtdragover",position:{x:q[0],y:q[1]}}))}}else if(t.hoverData.dragging){if(re=!0,oe.panningEnabled()&&oe.userPanningEnabled()){var xt;if(t.hoverData.justStartedPan){var ut=t.hoverData.mdownPos;xt={x:(q[0]-ut[0])*V,y:(q[1]-ut[1])*V},t.hoverData.justStartedPan=!1}else xt={x:De[0]*V,y:De[1]*V};oe.panBy(xt),oe.emit("dragpan"),t.hoverData.dragged=!0}q=t.projectIntoViewport(W.clientX,W.clientY)}else if(Pe[4]==1&&(Ve==null||Ve.pannable())){if(at){if(!t.hoverData.dragging&&oe.boxSelectionEnabled()&&(ot||!oe.panningEnabled()||!oe.userPanningEnabled()))bt();else if(!t.hoverData.selecting&&oe.panningEnabled()&&oe.userPanningEnabled()){var Et=s(Ve,t.hoverData.downs);Et&&(t.hoverData.dragging=!0,t.hoverData.justStartedPan=!0,Pe[4]=0,t.data.bgActivePosistion=U1(pe),t.redrawHint("select",!0),t.redraw())}Ve&&Ve.pannable()&&Ve.active()&&Ve.unactivate()}}else{if(Ve&&Ve.pannable()&&Ve.active()&&Ve.unactivate(),(!Ve||!Ve.grabbed())&&_e!=we&&(we&&i(we,["mouseout","tapdragout"],W,{x:q[0],y:q[1]}),_e&&i(_e,["mouseover","tapdragover"],W,{x:q[0],y:q[1]}),t.hoverData.last=_e),Ve)if(at){if(oe.boxSelectionEnabled()&&ot)Ve&&Ve.grabbed()&&(x(qe),Ve.emit("freeon"),qe.emit("free"),t.dragData.didDrag&&(Ve.emit("dragfreeon"),qe.emit("dragfree"))),bt();else if(Ve&&Ve.grabbed()&&t.nodeIsDraggable(Ve)){var ft=!t.dragData.didDrag;ft&&t.redrawHint("eles",!0),t.dragData.didDrag=!0,t.hoverData.draggingEles||y(qe,{inDragLayer:!0});var yt={x:0,y:0};if(Ct(De[0])&&Ct(De[1])&&(yt.x+=De[0],yt.y+=De[1],ft)){var nt=t.hoverData.dragDelta;nt&&Ct(nt[0])&&Ct(nt[1])&&(yt.x+=nt[0],yt.y+=nt[1])}t.hoverData.draggingEles=!0,qe.silentShift(yt).emit("position drag"),t.redrawHint("drag",!0),t.redraw()}}else Yt();re=!0}if(Pe[2]=q[0],Pe[3]=q[1],re)return W.stopPropagation&&W.stopPropagation(),W.preventDefault&&W.preventDefault(),!1}},"mousemoveHandler"),!1);var k,L,R;t.registerBinding(e,"mouseup",o(function(W){if(!(t.hoverData.which===1&&W.which!==1&&t.hoverData.capture)){var de=t.hoverData.capture;if(de){t.hoverData.capture=!1;var re=t.cy,oe=t.projectIntoViewport(W.clientX,W.clientY),V=t.selection,xe=t.findNearestElement(oe[0],oe[1],!0,!1),q=t.dragData.possibleDragElements,pe=t.hoverData.down,ve=a(W);if(t.data.bgActivePosistion&&(t.redrawHint("select",!0),t.redraw()),t.hoverData.tapholdCancelled=!0,t.data.bgActivePosistion=void 0,pe&&pe.unactivate(),t.hoverData.which===3){var Pe={originalEvent:W,type:"cxttapend",position:{x:oe[0],y:oe[1]}};if(pe?pe.emit(Pe):re.emit(Pe),!t.hoverData.cxtDragged){var _e={originalEvent:W,type:"cxttap",position:{x:oe[0],y:oe[1]}};pe?pe.emit(_e):re.emit(_e)}t.hoverData.cxtDragged=!1,t.hoverData.which=null}else if(t.hoverData.which===1){if(i(xe,["mouseup","tapend","vmouseup"],W,{x:oe[0],y:oe[1]}),!t.dragData.didDrag&&!t.hoverData.dragged&&!t.hoverData.selecting&&!t.hoverData.isOverThresholdDrag&&(i(pe,["click","tap","vclick"],W,{x:oe[0],y:oe[1]}),L=!1,W.timeStamp-R<=re.multiClickDebounceTime()?(k&&clearTimeout(k),L=!0,R=null,i(pe,["dblclick","dbltap","vdblclick"],W,{x:oe[0],y:oe[1]})):(k=setTimeout(function(){L||i(pe,["oneclick","onetap","voneclick"],W,{x:oe[0],y:oe[1]})},re.multiClickDebounceTime()),R=W.timeStamp)),pe==null&&!t.dragData.didDrag&&!t.hoverData.selecting&&!t.hoverData.dragged&&!a(W)&&(re.$(r).unselect(["tapunselect"]),q.length>0&&t.redrawHint("eles",!0),t.dragData.possibleDragElements=q=re.collection()),xe==pe&&!t.dragData.didDrag&&!t.hoverData.selecting&&xe!=null&&xe._private.selectable&&(t.hoverData.dragging||(re.selectionType()==="additive"||ve?xe.selected()?xe.unselect(["tapunselect"]):xe.select(["tapselect"]):ve||(re.$(r).unmerge(xe).unselect(["tapunselect"]),xe.select(["tapselect"]))),t.redrawHint("eles",!0)),t.hoverData.selecting){var we=re.collection(t.getAllInBox(V[0],V[1],V[2],V[3]));t.redrawHint("select",!0),we.length>0&&t.redrawHint("eles",!0),re.emit({type:"boxend",originalEvent:W,position:{x:oe[0],y:oe[1]}});var Ve=o(function(at){return at.selectable()&&!at.selected()},"eleWouldBeSelected");re.selectionType()==="additive"||ve||re.$(r).unmerge(we).unselect(),we.emit("box").stdFilter(Ve).select().emit("boxselect"),t.redraw()}if(t.hoverData.dragging&&(t.hoverData.dragging=!1,t.redrawHint("select",!0),t.redrawHint("eles",!0),t.redraw()),!V[4]){t.redrawHint("drag",!0),t.redrawHint("eles",!0);var De=pe&&pe.grabbed();x(q),De&&(pe.emit("freeon"),q.emit("free"),t.dragData.didDrag&&(pe.emit("dragfreeon"),q.emit("dragfree")))}}V[4]=0,t.hoverData.down=null,t.hoverData.cxtStarted=!1,t.hoverData.draggingEles=!1,t.hoverData.selecting=!1,t.hoverData.isOverThresholdDrag=!1,t.dragData.didDrag=!1,t.hoverData.dragged=!1,t.hoverData.dragDelta=[],t.hoverData.mdownPos=null,t.hoverData.mdownGPos=null,t.hoverData.which=null}}},"mouseupHandler"),!1);var O=o(function(W){if(!t.scrollingPage){var de=t.cy,re=de.zoom(),oe=de.pan(),V=t.projectIntoViewport(W.clientX,W.clientY),xe=[V[0]*re+oe.x,V[1]*re+oe.y];if(t.hoverData.draggingEles||t.hoverData.dragging||t.hoverData.cxtStarted||_()){W.preventDefault();return}if(de.panningEnabled()&&de.userPanningEnabled()&&de.zoomingEnabled()&&de.userZoomingEnabled()){W.preventDefault(),t.data.wheelZooming=!0,clearTimeout(t.data.wheelTimeout),t.data.wheelTimeout=setTimeout(function(){t.data.wheelZooming=!1,t.redrawHint("eles",!0),t.redraw()},150);var q;W.deltaY!=null?q=W.deltaY/-250:W.wheelDeltaY!=null?q=W.wheelDeltaY/1e3:q=W.wheelDelta/1e3,q=q*t.wheelSensitivity;var pe=W.deltaMode===1;pe&&(q*=33);var ve=de.zoom()*Math.pow(10,q);W.type==="gesturechange"&&(ve=t.gestureStartZoom*W.scale),de.zoom({level:ve,renderedPosition:{x:xe[0],y:xe[1]}}),de.emit(W.type==="gesturechange"?"pinchzoom":"scrollzoom")}}},"wheelHandler");t.registerBinding(t.container,"wheel",O,!0),t.registerBinding(e,"scroll",o(function(W){t.scrollingPage=!0,clearTimeout(t.scrollingPageTimeout),t.scrollingPageTimeout=setTimeout(function(){t.scrollingPage=!1},250)},"scrollHandler"),!0),t.registerBinding(t.container,"gesturestart",o(function(W){t.gestureStartZoom=t.cy.zoom(),t.hasTouchStarted||W.preventDefault()},"gestureStartHandler"),!0),t.registerBinding(t.container,"gesturechange",function(be){t.hasTouchStarted||O(be)},!0),t.registerBinding(t.container,"mouseout",o(function(W){var de=t.projectIntoViewport(W.clientX,W.clientY);t.cy.emit({originalEvent:W,type:"mouseout",position:{x:de[0],y:de[1]}})},"mouseOutHandler"),!1),t.registerBinding(t.container,"mouseover",o(function(W){var de=t.projectIntoViewport(W.clientX,W.clientY);t.cy.emit({originalEvent:W,type:"mouseover",position:{x:de[0],y:de[1]}})},"mouseOverHandler"),!1);var M,B,F,P,z,$,H,Q,j,ie,ne,le,he,K=o(function(W,de,re,oe){return Math.sqrt((re-W)*(re-W)+(oe-de)*(oe-de))},"distance"),X=o(function(W,de,re,oe){return(re-W)*(re-W)+(oe-de)*(oe-de)},"distanceSq"),te;t.registerBinding(t.container,"touchstart",te=o(function(W){if(t.hasTouchStarted=!0,!!I(W)){w(),t.touchData.capture=!0,t.data.bgActivePosistion=void 0;var de=t.cy,re=t.touchData.now,oe=t.touchData.earlier;if(W.touches[0]){var V=t.projectIntoViewport(W.touches[0].clientX,W.touches[0].clientY);re[0]=V[0],re[1]=V[1]}if(W.touches[1]){var V=t.projectIntoViewport(W.touches[1].clientX,W.touches[1].clientY);re[2]=V[0],re[3]=V[1]}if(W.touches[2]){var V=t.projectIntoViewport(W.touches[2].clientX,W.touches[2].clientY);re[4]=V[0],re[5]=V[1]}if(W.touches[1]){t.touchData.singleTouchMoved=!0,x(t.dragData.touchDragEles);var xe=t.findContainerClientCoords();j=xe[0],ie=xe[1],ne=xe[2],le=xe[3],M=W.touches[0].clientX-j,B=W.touches[0].clientY-ie,F=W.touches[1].clientX-j,P=W.touches[1].clientY-ie,he=0<=M&&M<=ne&&0<=F&&F<=ne&&0<=B&&B<=le&&0<=P&&P<=le;var q=de.pan(),pe=de.zoom();z=K(M,B,F,P),$=X(M,B,F,P),H=[(M+F)/2,(B+P)/2],Q=[(H[0]-q.x)/pe,(H[1]-q.y)/pe];var ve=200,Pe=ve*ve;if($=1){for(var st=t.touchData.startPosition=[null,null,null,null,null,null],Ue=0;Ue=t.touchTapThreshold2}if(de&&t.touchData.cxt){W.preventDefault();var st=W.touches[0].clientX-j,Ue=W.touches[0].clientY-ie,ct=W.touches[1].clientX-j,We=W.touches[1].clientY-ie,ot=X(st,Ue,ct,We),Yt=ot/$,bt=150,Mt=bt*bt,xt=1.5,ut=xt*xt;if(Yt>=ut||ot>=Mt){t.touchData.cxt=!1,t.data.bgActivePosistion=void 0,t.redrawHint("select",!0);var Et={originalEvent:W,type:"cxttapend",position:{x:V[0],y:V[1]}};t.touchData.start?(t.touchData.start.unactivate().emit(Et),t.touchData.start=null):oe.emit(Et)}}if(de&&t.touchData.cxt){var Et={originalEvent:W,type:"cxtdrag",position:{x:V[0],y:V[1]}};t.data.bgActivePosistion=void 0,t.redrawHint("select",!0),t.touchData.start?t.touchData.start.emit(Et):oe.emit(Et),t.touchData.start&&(t.touchData.start._private.grabbed=!1),t.touchData.cxtDragged=!0;var ft=t.findNearestElement(V[0],V[1],!0,!0);(!t.touchData.cxtOver||ft!==t.touchData.cxtOver)&&(t.touchData.cxtOver&&t.touchData.cxtOver.emit({originalEvent:W,type:"cxtdragout",position:{x:V[0],y:V[1]}}),t.touchData.cxtOver=ft,ft&&ft.emit({originalEvent:W,type:"cxtdragover",position:{x:V[0],y:V[1]}}))}else if(de&&W.touches[2]&&oe.boxSelectionEnabled())W.preventDefault(),t.data.bgActivePosistion=void 0,this.lastThreeTouch=+new Date,t.touchData.selecting||oe.emit({originalEvent:W,type:"boxstart",position:{x:V[0],y:V[1]}}),t.touchData.selecting=!0,t.touchData.didSelect=!0,re[4]=1,!re||re.length===0||re[0]===void 0?(re[0]=(V[0]+V[2]+V[4])/3,re[1]=(V[1]+V[3]+V[5])/3,re[2]=(V[0]+V[2]+V[4])/3+1,re[3]=(V[1]+V[3]+V[5])/3+1):(re[2]=(V[0]+V[2]+V[4])/3,re[3]=(V[1]+V[3]+V[5])/3),t.redrawHint("select",!0),t.redraw();else if(de&&W.touches[1]&&!t.touchData.didSelect&&oe.zoomingEnabled()&&oe.panningEnabled()&&oe.userZoomingEnabled()&&oe.userPanningEnabled()){W.preventDefault(),t.data.bgActivePosistion=void 0,t.redrawHint("select",!0);var yt=t.dragData.touchDragEles;if(yt){t.redrawHint("drag",!0);for(var nt=0;nt0&&!t.hoverData.draggingEles&&!t.swipePanning&&t.data.bgActivePosistion!=null&&(t.data.bgActivePosistion=void 0,t.redrawHint("select",!0),t.redraw())}},"touchmoveHandler"),!1);var se;t.registerBinding(e,"touchcancel",se=o(function(W){var de=t.touchData.start;t.touchData.capture=!1,de&&de.unactivate()},"touchcancelHandler"));var ue,Z,Se,ce;if(t.registerBinding(e,"touchend",ue=o(function(W){var de=t.touchData.start,re=t.touchData.capture;if(re)W.touches.length===0&&(t.touchData.capture=!1),W.preventDefault();else return;var oe=t.selection;t.swipePanning=!1,t.hoverData.draggingEles=!1;var V=t.cy,xe=V.zoom(),q=t.touchData.now,pe=t.touchData.earlier;if(W.touches[0]){var ve=t.projectIntoViewport(W.touches[0].clientX,W.touches[0].clientY);q[0]=ve[0],q[1]=ve[1]}if(W.touches[1]){var ve=t.projectIntoViewport(W.touches[1].clientX,W.touches[1].clientY);q[2]=ve[0],q[3]=ve[1]}if(W.touches[2]){var ve=t.projectIntoViewport(W.touches[2].clientX,W.touches[2].clientY);q[4]=ve[0],q[5]=ve[1]}de&&de.unactivate();var Pe;if(t.touchData.cxt){if(Pe={originalEvent:W,type:"cxttapend",position:{x:q[0],y:q[1]}},de?de.emit(Pe):V.emit(Pe),!t.touchData.cxtDragged){var _e={originalEvent:W,type:"cxttap",position:{x:q[0],y:q[1]}};de?de.emit(_e):V.emit(_e)}t.touchData.start&&(t.touchData.start._private.grabbed=!1),t.touchData.cxt=!1,t.touchData.start=null,t.redraw();return}if(!W.touches[2]&&V.boxSelectionEnabled()&&t.touchData.selecting){t.touchData.selecting=!1;var we=V.collection(t.getAllInBox(oe[0],oe[1],oe[2],oe[3]));oe[0]=void 0,oe[1]=void 0,oe[2]=void 0,oe[3]=void 0,oe[4]=0,t.redrawHint("select",!0),V.emit({type:"boxend",originalEvent:W,position:{x:q[0],y:q[1]}});var Ve=o(function(Mt){return Mt.selectable()&&!Mt.selected()},"eleWouldBeSelected");we.emit("box").stdFilter(Ve).select().emit("boxselect"),we.nonempty()&&t.redrawHint("eles",!0),t.redraw()}if(de?.unactivate(),W.touches[2])t.data.bgActivePosistion=void 0,t.redrawHint("select",!0);else if(!W.touches[1]){if(!W.touches[0]){if(!W.touches[0]){t.data.bgActivePosistion=void 0,t.redrawHint("select",!0);var De=t.dragData.touchDragEles;if(de!=null){var qe=de._private.grabbed;x(De),t.redrawHint("drag",!0),t.redrawHint("eles",!0),qe&&(de.emit("freeon"),De.emit("free"),t.dragData.didDrag&&(de.emit("dragfreeon"),De.emit("dragfree"))),i(de,["touchend","tapend","vmouseup","tapdragout"],W,{x:q[0],y:q[1]}),de.unactivate(),t.touchData.start=null}else{var at=t.findNearestElement(q[0],q[1],!0,!0);i(at,["touchend","tapend","vmouseup","tapdragout"],W,{x:q[0],y:q[1]})}var Rt=t.touchData.startPosition[0]-q[0],st=Rt*Rt,Ue=t.touchData.startPosition[1]-q[1],ct=Ue*Ue,We=st+ct,ot=We*xe*xe;t.touchData.singleTouchMoved||(de||V.$(":selected").unselect(["tapunselect"]),i(de,["tap","vclick"],W,{x:q[0],y:q[1]}),Z=!1,W.timeStamp-ce<=V.multiClickDebounceTime()?(Se&&clearTimeout(Se),Z=!0,ce=null,i(de,["dbltap","vdblclick"],W,{x:q[0],y:q[1]})):(Se=setTimeout(function(){Z||i(de,["onetap","voneclick"],W,{x:q[0],y:q[1]})},V.multiClickDebounceTime()),ce=W.timeStamp)),de!=null&&!t.dragData.didDrag&&de._private.selectable&&ot"u"){var ae=[],Oe=o(function(W){return{clientX:W.clientX,clientY:W.clientY,force:1,identifier:W.pointerId,pageX:W.pageX,pageY:W.pageY,radiusX:W.width/2,radiusY:W.height/2,screenX:W.screenX,screenY:W.screenY,target:W.target}},"makeTouch"),ge=o(function(W){return{event:W,touch:Oe(W)}},"makePointer"),ze=o(function(W){ae.push(ge(W))},"addPointer"),He=o(function(W){for(var de=0;de0)return H[0]}return null},"getCurveT"),g=Object.keys(p),y=0;y0?m:vme(a,s,e,r,n,i,l,u)},"intersectLine"),checkPoint:o(function(e,r,n,i,a,s,l,u){u=u==="auto"?Vp(i,a):u;var h=2*u;if(Zu(e,r,this.points,s,l,i,a-h,[0,-1],n)||Zu(e,r,this.points,s,l,i-h,a,[0,-1],n))return!0;var f=i/2+2*n,d=a/2+2*n,p=[s-f,l-d,s-f,l,s+f,l,s+f,l-d];return!!(Us(e,r,p)||$p(e,r,h,h,s+i/2-u,l+a/2-u,n)||$p(e,r,h,h,s-i/2+u,l+a/2-u,n))},"checkPoint")}};eh.registerNodeShapes=function(){var t=this.nodeShapes={},e=this;this.generateEllipse(),this.generatePolygon("triangle",gs(3,0)),this.generateRoundPolygon("round-triangle",gs(3,0)),this.generatePolygon("rectangle",gs(4,0)),t.square=t.rectangle,this.generateRoundRectangle(),this.generateCutRectangle(),this.generateBarrel(),this.generateBottomRoundrectangle();{var r=[0,1,1,0,0,-1,-1,0];this.generatePolygon("diamond",r),this.generateRoundPolygon("round-diamond",r)}this.generatePolygon("pentagon",gs(5,0)),this.generateRoundPolygon("round-pentagon",gs(5,0)),this.generatePolygon("hexagon",gs(6,0)),this.generateRoundPolygon("round-hexagon",gs(6,0)),this.generatePolygon("heptagon",gs(7,0)),this.generateRoundPolygon("round-heptagon",gs(7,0)),this.generatePolygon("octagon",gs(8,0)),this.generateRoundPolygon("round-octagon",gs(8,0));var n=new Array(20);{var i=PP(5,0),a=PP(5,Math.PI/5),s=.5*(3-Math.sqrt(5));s*=1.57;for(var l=0;l=e.deqFastCost*C)break}else if(h){if(b>=e.deqCost*m||b>=e.deqAvgCost*p)break}else if(w>=e.deqNoDrawCost*DP)break;var T=e.deq(n,v,y);if(T.length>0)for(var E=0;E0&&(e.onDeqd(n,g),!h&&e.shouldRedraw(n,g,v,y)&&a())},"dequeue"),l=e.priority||rB;i.beforeRender(s,l(n))}},"setupDequeueingImpl")},"setupDequeueing")},RQe=function(){function t(e){var r=arguments.length>1&&arguments[1]!==void 0?arguments[1]:bS;Mf(this,t),this.idsByKey=new Xc,this.keyForId=new Xc,this.cachesByLvl=new Xc,this.lvls=[],this.getKey=e,this.doesEleInvalidateKey=r}return o(t,"ElementTextureCacheLookup"),If(t,[{key:"getIdsFor",value:o(function(r){r==null&&ai("Can not get id list for null key");var n=this.idsByKey,i=this.idsByKey.get(r);return i||(i=new J1,n.set(r,i)),i},"getIdsFor")},{key:"addIdForKey",value:o(function(r,n){r!=null&&this.getIdsFor(r).add(n)},"addIdForKey")},{key:"deleteIdForKey",value:o(function(r,n){r!=null&&this.getIdsFor(r).delete(n)},"deleteIdForKey")},{key:"getNumberOfIdsForKey",value:o(function(r){return r==null?0:this.getIdsFor(r).size},"getNumberOfIdsForKey")},{key:"updateKeyMappingFor",value:o(function(r){var n=r.id(),i=this.keyForId.get(n),a=this.getKey(r);this.deleteIdForKey(i,n),this.addIdForKey(a,n),this.keyForId.set(n,a)},"updateKeyMappingFor")},{key:"deleteKeyMappingFor",value:o(function(r){var n=r.id(),i=this.keyForId.get(n);this.deleteIdForKey(i,n),this.keyForId.delete(n)},"deleteKeyMappingFor")},{key:"keyHasChangedFor",value:o(function(r){var n=r.id(),i=this.keyForId.get(n),a=this.getKey(r);return i!==a},"keyHasChangedFor")},{key:"isInvalid",value:o(function(r){return this.keyHasChangedFor(r)||this.doesEleInvalidateKey(r)},"isInvalid")},{key:"getCachesAt",value:o(function(r){var n=this.cachesByLvl,i=this.lvls,a=n.get(r);return a||(a=new Xc,n.set(r,a),i.push(r)),a},"getCachesAt")},{key:"getCache",value:o(function(r,n){return this.getCachesAt(n).get(r)},"getCache")},{key:"get",value:o(function(r,n){var i=this.getKey(r),a=this.getCache(i,n);return a!=null&&this.updateKeyMappingFor(r),a},"get")},{key:"getForCachedKey",value:o(function(r,n){var i=this.keyForId.get(r.id()),a=this.getCache(i,n);return a},"getForCachedKey")},{key:"hasCache",value:o(function(r,n){return this.getCachesAt(n).has(r)},"hasCache")},{key:"has",value:o(function(r,n){var i=this.getKey(r);return this.hasCache(i,n)},"has")},{key:"setCache",value:o(function(r,n,i){i.key=r,this.getCachesAt(n).set(r,i)},"setCache")},{key:"set",value:o(function(r,n,i){var a=this.getKey(r);this.setCache(a,n,i),this.updateKeyMappingFor(r)},"set")},{key:"deleteCache",value:o(function(r,n){this.getCachesAt(n).delete(r)},"deleteCache")},{key:"delete",value:o(function(r,n){var i=this.getKey(r);this.deleteCache(i,n)},"_delete")},{key:"invalidateKey",value:o(function(r){var n=this;this.lvls.forEach(function(i){return n.deleteCache(r,i)})},"invalidateKey")},{key:"invalidate",value:o(function(r){var n=r.id(),i=this.keyForId.get(n);this.deleteKeyMappingFor(r);var a=this.doesEleInvalidateKey(r);return a&&this.invalidateKey(i),a||this.getNumberOfIdsForKey(i)===0},"invalidate")}]),t}(),I0e=25,sS=50,yS=-4,XP=3,bge=7.99,NQe=8,MQe=1024,IQe=1024,OQe=1024,PQe=.2,BQe=.8,FQe=10,$Qe=.15,zQe=.1,GQe=.9,VQe=.9,UQe=100,HQe=1,H1={dequeue:"dequeue",downscale:"downscale",highQuality:"highQuality"},WQe=la({getKey:null,doesEleInvalidateKey:bS,drawElement:null,getBoundingBox:null,getRotationPoint:null,getRotationOffset:null,isVisible:ume,allowEdgeTxrCaching:!0,allowParentTxrCaching:!0}),Fb=o(function(e,r){var n=this;n.renderer=e,n.onDequeues=[];var i=WQe(r);rr(n,i),n.lookup=new RQe(i.getKey,i.doesEleInvalidateKey),n.setupDequeueing()},"ElementTextureCache"),qi=Fb.prototype;qi.reasons=H1;qi.getTextureQueue=function(t){var e=this;return e.eleImgCaches=e.eleImgCaches||{},e.eleImgCaches[t]=e.eleImgCaches[t]||[]};qi.getRetiredTextureQueue=function(t){var e=this,r=e.eleImgCaches.retired=e.eleImgCaches.retired||{},n=r[t]=r[t]||[];return n};qi.getElementQueue=function(){var t=this,e=t.eleCacheQueue=t.eleCacheQueue||new i4(function(r,n){return n.reqs-r.reqs});return e};qi.getElementKeyToQueue=function(){var t=this,e=t.eleKeyToCacheQueue=t.eleKeyToCacheQueue||{};return e};qi.getElement=function(t,e,r,n,i){var a=this,s=this.renderer,l=s.cy.zoom(),u=this.lookup;if(!e||e.w===0||e.h===0||isNaN(e.w)||isNaN(e.h)||!t.visible()||t.removed()||!a.allowEdgeTxrCaching&&t.isEdge()||!a.allowParentTxrCaching&&t.isParent())return null;if(n==null&&(n=Math.ceil(iB(l*r))),n=bge||n>XP)return null;var h=Math.pow(2,n),f=e.h*h,d=e.w*h,p=s.eleTextBiggerThanMin(t,h);if(!this.isVisible(t,p))return null;var m=u.get(t,n);if(m&&m.invalidated&&(m.invalidated=!1,m.texture.invalidatedWidth-=m.width),m)return m;var g;if(f<=I0e?g=I0e:f<=sS?g=sS:g=Math.ceil(f/sS)*sS,f>OQe||d>IQe)return null;var y=a.getTextureQueue(g),v=y[y.length-2],x=o(function(){return a.recycleTexture(g,d)||a.addTexture(g,d)},"addNewTxr");v||(v=y[y.length-1]),v||(v=x()),v.width-v.usedWidthn;D--)_=a.getElement(t,e,r,D,H1.downscale);I()}else return a.queueElement(t,E.level-1),E;else{var k;if(!w&&!C&&!T)for(var L=n-1;L>=yS;L--){var R=u.get(t,L);if(R){k=R;break}}if(b(k))return a.queueElement(t,n),k;v.context.translate(v.usedWidth,0),v.context.scale(h,h),this.drawElement(v.context,t,e,p,!1),v.context.scale(1/h,1/h),v.context.translate(-v.usedWidth,0)}return m={x:v.usedWidth,texture:v,level:n,scale:h,width:d,height:f,scaledLabelShown:p},v.usedWidth+=Math.ceil(d+NQe),v.eleCaches.push(m),u.set(t,n,m),a.checkTextureFullness(v),m};qi.invalidateElements=function(t){for(var e=0;e=PQe*t.width&&this.retireTexture(t)};qi.checkTextureFullness=function(t){var e=this,r=e.getTextureQueue(t.height);t.usedWidth/t.width>BQe&&t.fullnessChecks>=FQe?Df(r,t):t.fullnessChecks++};qi.retireTexture=function(t){var e=this,r=t.height,n=e.getTextureQueue(r),i=this.lookup;Df(n,t),t.retired=!0;for(var a=t.eleCaches,s=0;s=e)return s.retired=!1,s.usedWidth=0,s.invalidatedWidth=0,s.fullnessChecks=0,nB(s.eleCaches),s.context.setTransform(1,0,0,1,0,0),s.context.clearRect(0,0,s.width,s.height),Df(i,s),n.push(s),s}};qi.queueElement=function(t,e){var r=this,n=r.getElementQueue(),i=r.getElementKeyToQueue(),a=this.getKey(t),s=i[a];if(s)s.level=Math.max(s.level,e),s.eles.merge(t),s.reqs++,n.updateItem(s);else{var l={eles:t.spawn().merge(t),level:e,reqs:1,key:a};n.push(l),i[a]=l}};qi.dequeue=function(t){for(var e=this,r=e.getElementQueue(),n=e.getElementKeyToQueue(),i=[],a=e.lookup,s=0;s0;s++){var l=r.pop(),u=l.key,h=l.eles[0],f=a.hasCache(h,l.level);if(n[u]=null,f)continue;i.push(l);var d=e.getBoundingBox(h);e.getElement(h,d,t,l.level,H1.dequeue)}return i};qi.removeFromQueue=function(t){var e=this,r=e.getElementQueue(),n=e.getElementKeyToQueue(),i=this.getKey(t),a=n[i];a!=null&&(a.eles.length===1?(a.reqs=tB,r.updateItem(a),r.pop(),n[i]=null):a.eles.unmerge(t))};qi.onDequeue=function(t){this.onDequeues.push(t)};qi.offDequeue=function(t){Df(this.onDequeues,t)};qi.setupDequeueing=xge.setupDequeueing({deqRedrawThreshold:UQe,deqCost:$Qe,deqAvgCost:zQe,deqNoDrawCost:GQe,deqFastCost:VQe,deq:o(function(e,r,n){return e.dequeue(r,n)},"deq"),onDeqd:o(function(e,r){for(var n=0;n=YQe||r>_S)return null}n.validateLayersElesOrdering(r,t);var u=n.layersByLevel,h=Math.pow(2,r),f=u[r]=u[r]||[],d,p=n.levelIsComplete(r,t),m,g=o(function(){var I=o(function(O){if(n.validateLayersElesOrdering(O,t),n.levelIsComplete(O,t))return m=u[O],!0},"canUseAsTmpLvl"),D=o(function(O){if(!m)for(var M=r+O;zb<=M&&M<=_S&&!I(M);M+=O);},"checkLvls");D(1),D(-1);for(var k=f.length-1;k>=0;k--){var L=f[k];L.invalid&&Df(f,L)}},"checkTempLevels");if(!p)g();else return f;var y=o(function(){if(!d){d=Hs();for(var I=0;IP0e||L>P0e)return null;var R=k*L;if(R>tZe)return null;var O=n.makeLayer(d,r);if(D!=null){var M=f.indexOf(D)+1;f.splice(M,0,O)}else(I.insert===void 0||I.insert)&&f.unshift(O);return O},"makeLayer");if(n.skipping&&!l)return null;for(var x=null,b=t.length/qQe,w=!l,C=0;C=b||!yme(x.bb,T.boundingBox()))&&(x=v({insert:!0,after:x}),!x))return null;m||w?n.queueLayer(x,T):n.drawEleInLayer(x,T,r,e),x.eles.push(T),A[r]=x}return m||(w?null:f)};Ea.getEleLevelForLayerLevel=function(t,e){return t};Ea.drawEleInLayer=function(t,e,r,n){var i=this,a=this.renderer,s=t.context,l=e.boundingBox();l.w===0||l.h===0||!e.visible()||(r=i.getEleLevelForLayerLevel(r,n),a.setImgSmoothing(s,!1),a.drawCachedElement(s,e,null,null,r,rZe),a.setImgSmoothing(s,!0))};Ea.levelIsComplete=function(t,e){var r=this,n=r.layersByLevel[t];if(!n||n.length===0)return!1;for(var i=0,a=0;a0||s.invalid)return!1;i+=s.eles.length}return i===e.length};Ea.validateLayersElesOrdering=function(t,e){var r=this.layersByLevel[t];if(r)for(var n=0;n0){e=!0;break}}return e};Ea.invalidateElements=function(t){var e=this;t.length!==0&&(e.lastInvalidationTime=Qu(),!(t.length===0||!e.haveLayers())&&e.updateElementsInLayers(t,o(function(n,i,a){e.invalidateLayer(n)},"invalAssocLayers")))};Ea.invalidateLayer=function(t){if(this.lastInvalidationTime=Qu(),!t.invalid){var e=t.level,r=t.eles,n=this.layersByLevel[e];Df(n,t),t.elesQueue=[],t.invalid=!0,t.replacement&&(t.replacement.invalid=!0);for(var i=0;i3&&arguments[3]!==void 0?arguments[3]:!0,i=arguments.length>4&&arguments[4]!==void 0?arguments[4]:!0,a=arguments.length>5&&arguments[5]!==void 0?arguments[5]:!0,s=this,l=e._private.rscratch;if(!(a&&!e.visible())&&!(l.badLine||l.allpts==null||isNaN(l.allpts[0]))){var u;r&&(u=r,t.translate(-u.x1,-u.y1));var h=a?e.pstyle("opacity").value:1,f=a?e.pstyle("line-opacity").value:1,d=e.pstyle("curve-style").value,p=e.pstyle("line-style").value,m=e.pstyle("width").pfValue,g=e.pstyle("line-cap").value,y=e.pstyle("line-outline-width").value,v=e.pstyle("line-outline-color").value,x=h*f,b=h*f,w=o(function(){var O=arguments.length>0&&arguments[0]!==void 0?arguments[0]:x;d==="straight-triangle"?(s.eleStrokeStyle(t,e,O),s.drawEdgeTrianglePath(e,t,l.allpts)):(t.lineWidth=m,t.lineCap=g,s.eleStrokeStyle(t,e,O),s.drawEdgePath(e,t,l.allpts,p),t.lineCap="butt")},"drawLine"),C=o(function(){var O=arguments.length>0&&arguments[0]!==void 0?arguments[0]:x;if(t.lineWidth=m+y,t.lineCap=g,y>0)s.colorStrokeStyle(t,v[0],v[1],v[2],O);else{t.lineCap="butt";return}d==="straight-triangle"?s.drawEdgeTrianglePath(e,t,l.allpts):(s.drawEdgePath(e,t,l.allpts,p),t.lineCap="butt")},"drawLineOutline"),T=o(function(){i&&s.drawEdgeOverlay(t,e)},"drawOverlay"),E=o(function(){i&&s.drawEdgeUnderlay(t,e)},"drawUnderlay"),A=o(function(){var O=arguments.length>0&&arguments[0]!==void 0?arguments[0]:b;s.drawArrowheads(t,e,O)},"drawArrows"),S=o(function(){s.drawElementText(t,e,null,n)},"drawText");t.lineJoin="round";var _=e.pstyle("ghost").value==="yes";if(_){var I=e.pstyle("ghost-offset-x").pfValue,D=e.pstyle("ghost-offset-y").pfValue,k=e.pstyle("ghost-opacity").value,L=x*k;t.translate(I,D),w(L),A(L),t.translate(-I,-D)}else C();E(),w(),A(),T(),S(),r&&t.translate(u.x1,u.y1)}};kge=o(function(e){if(!["overlay","underlay"].includes(e))throw new Error("Invalid state");return function(r,n){if(n.visible()){var i=n.pstyle("".concat(e,"-opacity")).value;if(i!==0){var a=this,s=a.usePaths(),l=n._private.rscratch,u=n.pstyle("".concat(e,"-padding")).pfValue,h=2*u,f=n.pstyle("".concat(e,"-color")).value;r.lineWidth=h,l.edgeType==="self"&&!s?r.lineCap="butt":r.lineCap="round",a.colorStrokeStyle(r,f[0],f[1],f[2],i),a.drawEdgePath(n,r,l.allpts,"solid")}}}},"drawEdgeOverlayUnderlay");th.drawEdgeOverlay=kge("overlay");th.drawEdgeUnderlay=kge("underlay");th.drawEdgePath=function(t,e,r,n){var i=t._private.rscratch,a=e,s,l=!1,u=this.usePaths(),h=t.pstyle("line-dash-pattern").pfValue,f=t.pstyle("line-dash-offset").pfValue;if(u){var d=r.join("$"),p=i.pathCacheKey&&i.pathCacheKey===d;p?(s=e=i.pathCache,l=!0):(s=e=new Path2D,i.pathCacheKey=d,i.pathCache=s)}if(a.setLineDash)switch(n){case"dotted":a.setLineDash([1,1]);break;case"dashed":a.setLineDash(h),a.lineDashOffset=f;break;case"solid":a.setLineDash([]);break}if(!l&&!i.badLine)switch(e.beginPath&&e.beginPath(),e.moveTo(r[0],r[1]),i.edgeType){case"bezier":case"self":case"compound":case"multibezier":for(var m=2;m+35&&arguments[5]!==void 0?arguments[5]:!0,s=this;if(n==null){if(a&&!s.eleTextBiggerThanMin(e))return}else if(n===!1)return;if(e.isNode()){var l=e.pstyle("label");if(!l||!l.value)return;var u=s.getLabelJustification(e);t.textAlign=u,t.textBaseline="bottom"}else{var h=e.element()._private.rscratch.badLine,f=e.pstyle("label"),d=e.pstyle("source-label"),p=e.pstyle("target-label");if(h||(!f||!f.value)&&(!d||!d.value)&&(!p||!p.value))return;t.textAlign="center",t.textBaseline="bottom"}var m=!r,g;r&&(g=r,t.translate(-g.x1,-g.y1)),i==null?(s.drawText(t,e,null,m,a),e.isEdge()&&(s.drawText(t,e,"source",m,a),s.drawText(t,e,"target",m,a))):s.drawText(t,e,i,m,a),r&&t.translate(g.x1,g.y1)};Yp.getFontCache=function(t){var e;this.fontCaches=this.fontCaches||[];for(var r=0;r2&&arguments[2]!==void 0?arguments[2]:!0,n=e.pstyle("font-style").strValue,i=e.pstyle("font-size").pfValue+"px",a=e.pstyle("font-family").strValue,s=e.pstyle("font-weight").strValue,l=r?e.effectiveOpacity()*e.pstyle("text-opacity").value:1,u=e.pstyle("text-outline-opacity").value*l,h=e.pstyle("color").value,f=e.pstyle("text-outline-color").value;t.font=n+" "+s+" "+i+" "+a,t.lineJoin="round",this.colorFillStyle(t,h[0],h[1],h[2],l),this.colorStrokeStyle(t,f[0],f[1],f[2],u)};o(RP,"roundRect");Yp.getTextAngle=function(t,e){var r,n=t._private,i=n.rscratch,a=e?e+"-":"",s=t.pstyle(a+"text-rotation");if(s.strValue==="autorotate"){var l=Gl(i,"labelAngle",e);r=t.isEdge()?l:0}else s.strValue==="none"?r=0:r=s.pfValue;return r};Yp.drawText=function(t,e,r){var n=arguments.length>3&&arguments[3]!==void 0?arguments[3]:!0,i=arguments.length>4&&arguments[4]!==void 0?arguments[4]:!0,a=e._private,s=a.rscratch,l=i?e.effectiveOpacity():1;if(!(i&&(l===0||e.pstyle("text-opacity").value===0))){r==="main"&&(r=null);var u=Gl(s,"labelX",r),h=Gl(s,"labelY",r),f,d,p=this.getLabelText(e,r);if(p!=null&&p!==""&&!isNaN(u)&&!isNaN(h)){this.setupTextStyle(t,e,i);var m=r?r+"-":"",g=Gl(s,"labelWidth",r),y=Gl(s,"labelHeight",r),v=e.pstyle(m+"text-margin-x").pfValue,x=e.pstyle(m+"text-margin-y").pfValue,b=e.isEdge(),w=e.pstyle("text-halign").value,C=e.pstyle("text-valign").value;b&&(w="center",C="center"),u+=v,h+=x;var T;switch(n?T=this.getTextAngle(e,r):T=0,T!==0&&(f=u,d=h,t.translate(f,d),t.rotate(T),u=0,h=0),C){case"top":break;case"center":h+=y/2;break;case"bottom":h+=y;break}var E=e.pstyle("text-background-opacity").value,A=e.pstyle("text-border-opacity").value,S=e.pstyle("text-border-width").pfValue,_=e.pstyle("text-background-padding").pfValue,I=e.pstyle("text-background-shape").strValue,D=I.indexOf("round")===0,k=2;if(E>0||S>0&&A>0){var L=u-_;switch(w){case"left":L-=g;break;case"center":L-=g/2;break}var R=h-y-_,O=g+2*_,M=y+2*_;if(E>0){var B=t.fillStyle,F=e.pstyle("text-background-color").value;t.fillStyle="rgba("+F[0]+","+F[1]+","+F[2]+","+E*l+")",D?RP(t,L,R,O,M,k):t.fillRect(L,R,O,M),t.fillStyle=B}if(S>0&&A>0){var P=t.strokeStyle,z=t.lineWidth,$=e.pstyle("text-border-color").value,H=e.pstyle("text-border-style").value;if(t.strokeStyle="rgba("+$[0]+","+$[1]+","+$[2]+","+A*l+")",t.lineWidth=S,t.setLineDash)switch(H){case"dotted":t.setLineDash([1,1]);break;case"dashed":t.setLineDash([4,2]);break;case"double":t.lineWidth=S/4,t.setLineDash([]);break;case"solid":t.setLineDash([]);break}if(D?RP(t,L,R,O,M,k,"stroke"):t.strokeRect(L,R,O,M),H==="double"){var Q=S/2;D?RP(t,L+Q,R+Q,O-Q*2,M-Q*2,k,"stroke"):t.strokeRect(L+Q,R+Q,O-Q*2,M-Q*2)}t.setLineDash&&t.setLineDash([]),t.lineWidth=z,t.strokeStyle=P}}var j=2*e.pstyle("text-outline-width").pfValue;if(j>0&&(t.lineWidth=j),e.pstyle("text-wrap").value==="wrap"){var ie=Gl(s,"labelWrapCachedLines",r),ne=Gl(s,"labelLineHeight",r),le=g/2,he=this.getLabelJustification(e);switch(he==="auto"||(w==="left"?he==="left"?u+=-g:he==="center"&&(u+=-le):w==="center"?he==="left"?u+=-le:he==="right"&&(u+=le):w==="right"&&(he==="center"?u+=le:he==="right"&&(u+=g))),C){case"top":h-=(ie.length-1)*ne;break;case"center":case"bottom":h-=(ie.length-1)*ne;break}for(var K=0;K0&&t.strokeText(ie[K],u,h),t.fillText(ie[K],u,h),h+=ne}else j>0&&t.strokeText(p,u,h),t.fillText(p,u,h);T!==0&&(t.rotate(-T),t.translate(-f,-d))}}};ly={};ly.drawNode=function(t,e,r){var n=arguments.length>3&&arguments[3]!==void 0?arguments[3]:!0,i=arguments.length>4&&arguments[4]!==void 0?arguments[4]:!0,a=arguments.length>5&&arguments[5]!==void 0?arguments[5]:!0,s=this,l,u,h=e._private,f=h.rscratch,d=e.position();if(!(!Ct(d.x)||!Ct(d.y))&&!(a&&!e.visible())){var p=a?e.effectiveOpacity():1,m=s.usePaths(),g,y=!1,v=e.padding();l=e.width()+2*v,u=e.height()+2*v;var x;r&&(x=r,t.translate(-x.x1,-x.y1));for(var b=e.pstyle("background-image"),w=b.value,C=new Array(w.length),T=new Array(w.length),E=0,A=0;A0&&arguments[0]!==void 0?arguments[0]:L;s.eleFillStyle(t,e,oe)},"setupShapeColor"),K=o(function(){var oe=arguments.length>0&&arguments[0]!==void 0?arguments[0]:$;s.colorStrokeStyle(t,R[0],R[1],R[2],oe)},"setupBorderColor"),X=o(function(){var oe=arguments.length>0&&arguments[0]!==void 0?arguments[0]:ie;s.colorStrokeStyle(t,Q[0],Q[1],Q[2],oe)},"setupOutlineColor"),te=o(function(oe,V,xe,q){var pe=s.nodePathCache=s.nodePathCache||[],ve=cme(xe==="polygon"?xe+","+q.join(","):xe,""+V,""+oe,""+le),Pe=pe[ve],_e,we=!1;return Pe!=null?(_e=Pe,we=!0,f.pathCache=_e):(_e=new Path2D,pe[ve]=f.pathCache=_e),{path:_e,cacheHit:we}},"getPath"),J=e.pstyle("shape").strValue,se=e.pstyle("shape-polygon-points").pfValue;if(m){t.translate(d.x,d.y);var ue=te(l,u,J,se);g=ue.path,y=ue.cacheHit}var Z=o(function(){if(!y){var oe=d;m&&(oe={x:0,y:0}),s.nodeShapes[s.getNodeShape(e)].draw(g||t,oe.x,oe.y,l,u,le,f)}m?t.fill(g):t.fill()},"drawShape"),Se=o(function(){for(var oe=arguments.length>0&&arguments[0]!==void 0?arguments[0]:p,V=arguments.length>1&&arguments[1]!==void 0?arguments[1]:!0,xe=h.backgrounding,q=0,pe=0;pe0&&arguments[0]!==void 0?arguments[0]:!1,V=arguments.length>1&&arguments[1]!==void 0?arguments[1]:p;s.hasPie(e)&&(s.drawPie(t,e,V),oe&&(m||s.nodeShapes[s.getNodeShape(e)].draw(t,d.x,d.y,l,u,le,f)))},"drawPie"),ae=o(function(){var oe=arguments.length>0&&arguments[0]!==void 0?arguments[0]:p,V=(D>0?D:-D)*oe,xe=D>0?0:255;D!==0&&(s.colorFillStyle(t,xe,xe,xe,V),m?t.fill(g):t.fill())},"darken"),Oe=o(function(){if(k>0){if(t.lineWidth=k,t.lineCap=B,t.lineJoin=M,t.setLineDash)switch(O){case"dotted":t.setLineDash([1,1]);break;case"dashed":t.setLineDash(P),t.lineDashOffset=z;break;case"solid":case"double":t.setLineDash([]);break}if(F!=="center"){if(t.save(),t.lineWidth*=2,F==="inside")m?t.clip(g):t.clip();else{var oe=new Path2D;oe.rect(-l/2-k,-u/2-k,l+2*k,u+2*k),oe.addPath(g),t.clip(oe,"evenodd")}m?t.stroke(g):t.stroke(),t.restore()}else m?t.stroke(g):t.stroke();if(O==="double"){t.lineWidth=k/3;var V=t.globalCompositeOperation;t.globalCompositeOperation="destination-out",m?t.stroke(g):t.stroke(),t.globalCompositeOperation=V}t.setLineDash&&t.setLineDash([])}},"drawBorder"),ge=o(function(){if(H>0){if(t.lineWidth=H,t.lineCap="butt",t.setLineDash)switch(j){case"dotted":t.setLineDash([1,1]);break;case"dashed":t.setLineDash([4,2]);break;case"solid":case"double":t.setLineDash([]);break}var oe=d;m&&(oe={x:0,y:0});var V=s.getNodeShape(e),xe=k;F==="inside"&&(xe=0),F==="outside"&&(xe*=2);var q=(l+xe+(H+ne))/l,pe=(u+xe+(H+ne))/u,ve=l*q,Pe=u*pe,_e=s.nodeShapes[V].points,we;if(m){var Ve=te(ve,Pe,V,_e);we=Ve.path}if(V==="ellipse")s.drawEllipsePath(we||t,oe.x,oe.y,ve,Pe);else if(["round-diamond","round-heptagon","round-hexagon","round-octagon","round-pentagon","round-polygon","round-triangle","round-tag"].includes(V)){var De=0,qe=0,at=0;V==="round-diamond"?De=(xe+ne+H)*1.4:V==="round-heptagon"?(De=(xe+ne+H)*1.075,at=-(xe/2+ne+H)/35):V==="round-hexagon"?De=(xe+ne+H)*1.12:V==="round-pentagon"?(De=(xe+ne+H)*1.13,at=-(xe/2+ne+H)/15):V==="round-tag"?(De=(xe+ne+H)*1.12,qe=(xe/2+H+ne)*.07):V==="round-triangle"&&(De=(xe+ne+H)*(Math.PI/2),at=-(xe+ne/2+H)/Math.PI),De!==0&&(q=(l+De)/l,ve=l*q,["round-hexagon","round-tag"].includes(V)||(pe=(u+De)/u,Pe=u*pe)),le=le==="auto"?bme(ve,Pe):le;for(var Rt=ve/2,st=Pe/2,Ue=le+(xe+H+ne)/2,ct=new Array(_e.length/2),We=new Array(_e.length/2),ot=0;ot<_e.length/2;ot++)ct[ot]={x:oe.x+qe+Rt*_e[ot*2],y:oe.y+at+st*_e[ot*2+1]};var Yt,bt,Mt,xt,ut=ct.length;for(bt=ct[ut-1],Yt=0;Yt0){if(i=i||n.position(),a==null||s==null){var m=n.padding();a=n.width()+2*m,s=n.height()+2*m}l.colorFillStyle(r,f[0],f[1],f[2],h),l.nodeShapes[d].draw(r,i.x,i.y,a+u*2,s+u*2,p),r.fill()}}}},"drawNodeOverlayUnderlay");ly.drawNodeOverlay=Ege("overlay");ly.drawNodeUnderlay=Ege("underlay");ly.hasPie=function(t){return t=t[0],t._private.hasPie};ly.drawPie=function(t,e,r,n){e=e[0],n=n||e.position();var i=e.cy().style(),a=e.pstyle("pie-size"),s=n.x,l=n.y,u=e.width(),h=e.height(),f=Math.min(u,h)/2,d=0,p=this.usePaths();p&&(s=0,l=0),a.units==="%"?f=f*a.pfValue:a.pfValue!==void 0&&(f=a.pfValue/2);for(var m=1;m<=i.pieBackgroundN;m++){var g=e.pstyle("pie-"+m+"-background-size").value,y=e.pstyle("pie-"+m+"-background-color").value,v=e.pstyle("pie-"+m+"-background-opacity").value*r,x=g/100;x+d>1&&(x=1-d);var b=1.5*Math.PI+2*Math.PI*d,w=2*Math.PI*x,C=b+w;g===0||d>=1||d+x>1||(t.beginPath(),t.moveTo(s,l),t.arc(s,l,f,b,C),t.closePath(),this.colorFillStyle(t,y[0],y[1],y[2],v),t.fill(),d+=x)}};ys={},dZe=100;ys.getPixelRatio=function(){var t=this.data.contexts[0];if(this.forcedPixelRatio!=null)return this.forcedPixelRatio;var e=this.cy.window(),r=t.backingStorePixelRatio||t.webkitBackingStorePixelRatio||t.mozBackingStorePixelRatio||t.msBackingStorePixelRatio||t.oBackingStorePixelRatio||t.backingStorePixelRatio||1;return(e.devicePixelRatio||1)/r};ys.paintCache=function(t){for(var e=this.paintCaches=this.paintCaches||[],r=!0,n,i=0;ie.minMbLowQualFrames&&(e.motionBlurPxRatio=e.mbPxRBlurry)),e.clearingMotionBlur&&(e.motionBlurPxRatio=1),e.textureDrawLastFrame&&!d&&(f[e.NODE]=!0,f[e.SELECT_BOX]=!0);var b=r.style(),w=r.zoom(),C=s!==void 0?s:w,T=r.pan(),E={x:T.x,y:T.y},A={zoom:w,pan:{x:T.x,y:T.y}},S=e.prevViewport,_=S===void 0||A.zoom!==S.zoom||A.pan.x!==S.pan.x||A.pan.y!==S.pan.y;!_&&!(y&&!g)&&(e.motionBlurPxRatio=1),l&&(E=l),C*=u,E.x*=u,E.y*=u;var I=e.getCachedZSortedEles();function D(K,X,te,J,se){var ue=K.globalCompositeOperation;K.globalCompositeOperation="destination-out",e.colorFillStyle(K,255,255,255,e.motionBlurTransparency),K.fillRect(X,te,J,se),K.globalCompositeOperation=ue}o(D,"mbclear");function k(K,X){var te,J,se,ue;!e.clearingMotionBlur&&(K===h.bufferContexts[e.MOTIONBLUR_BUFFER_NODE]||K===h.bufferContexts[e.MOTIONBLUR_BUFFER_DRAG])?(te={x:T.x*m,y:T.y*m},J=w*m,se=e.canvasWidth*m,ue=e.canvasHeight*m):(te=E,J=C,se=e.canvasWidth,ue=e.canvasHeight),K.setTransform(1,0,0,1,0,0),X==="motionBlur"?D(K,0,0,se,ue):!n&&(X===void 0||X)&&K.clearRect(0,0,se,ue),i||(K.translate(te.x,te.y),K.scale(J,J)),l&&K.translate(l.x,l.y),s&&K.scale(s,s)}if(o(k,"setContextTransform"),d||(e.textureDrawLastFrame=!1),d){if(e.textureDrawLastFrame=!0,!e.textureCache){e.textureCache={},e.textureCache.bb=r.mutableElements().boundingBox(),e.textureCache.texture=e.data.bufferCanvases[e.TEXTURE_BUFFER];var L=e.data.bufferContexts[e.TEXTURE_BUFFER];L.setTransform(1,0,0,1,0,0),L.clearRect(0,0,e.canvasWidth*e.textureMult,e.canvasHeight*e.textureMult),e.render({forcedContext:L,drawOnlyNodeLayer:!0,forcedPxRatio:u*e.textureMult});var A=e.textureCache.viewport={zoom:r.zoom(),pan:r.pan(),width:e.canvasWidth,height:e.canvasHeight};A.mpan={x:(0-A.pan.x)/A.zoom,y:(0-A.pan.y)/A.zoom}}f[e.DRAG]=!1,f[e.NODE]=!1;var R=h.contexts[e.NODE],O=e.textureCache.texture,A=e.textureCache.viewport;R.setTransform(1,0,0,1,0,0),p?D(R,0,0,A.width,A.height):R.clearRect(0,0,A.width,A.height);var M=b.core("outside-texture-bg-color").value,B=b.core("outside-texture-bg-opacity").value;e.colorFillStyle(R,M[0],M[1],M[2],B),R.fillRect(0,0,A.width,A.height);var w=r.zoom();k(R,!1),R.clearRect(A.mpan.x,A.mpan.y,A.width/A.zoom/u,A.height/A.zoom/u),R.drawImage(O,A.mpan.x,A.mpan.y,A.width/A.zoom/u,A.height/A.zoom/u)}else e.textureOnViewport&&!n&&(e.textureCache=null);var F=r.extent(),P=e.pinching||e.hoverData.dragging||e.swipePanning||e.data.wheelZooming||e.hoverData.draggingEles||e.cy.animated(),z=e.hideEdgesOnViewport&&P,$=[];if($[e.NODE]=!f[e.NODE]&&p&&!e.clearedForMotionBlur[e.NODE]||e.clearingMotionBlur,$[e.NODE]&&(e.clearedForMotionBlur[e.NODE]=!0),$[e.DRAG]=!f[e.DRAG]&&p&&!e.clearedForMotionBlur[e.DRAG]||e.clearingMotionBlur,$[e.DRAG]&&(e.clearedForMotionBlur[e.DRAG]=!0),f[e.NODE]||i||a||$[e.NODE]){var H=p&&!$[e.NODE]&&m!==1,R=n||(H?e.data.bufferContexts[e.MOTIONBLUR_BUFFER_NODE]:h.contexts[e.NODE]),Q=p&&!H?"motionBlur":void 0;k(R,Q),z?e.drawCachedNodes(R,I.nondrag,u,F):e.drawLayeredElements(R,I.nondrag,u,F),e.debug&&e.drawDebugPoints(R,I.nondrag),!i&&!p&&(f[e.NODE]=!1)}if(!a&&(f[e.DRAG]||i||$[e.DRAG])){var H=p&&!$[e.DRAG]&&m!==1,R=n||(H?e.data.bufferContexts[e.MOTIONBLUR_BUFFER_DRAG]:h.contexts[e.DRAG]);k(R,p&&!H?"motionBlur":void 0),z?e.drawCachedNodes(R,I.drag,u,F):e.drawCachedElements(R,I.drag,u,F),e.debug&&e.drawDebugPoints(R,I.drag),!i&&!p&&(f[e.DRAG]=!1)}if(this.drawSelectionRectangle(t,k),p&&m!==1){var j=h.contexts[e.NODE],ie=e.data.bufferCanvases[e.MOTIONBLUR_BUFFER_NODE],ne=h.contexts[e.DRAG],le=e.data.bufferCanvases[e.MOTIONBLUR_BUFFER_DRAG],he=o(function(X,te,J){X.setTransform(1,0,0,1,0,0),J||!x?X.clearRect(0,0,e.canvasWidth,e.canvasHeight):D(X,0,0,e.canvasWidth,e.canvasHeight);var se=m;X.drawImage(te,0,0,e.canvasWidth*se,e.canvasHeight*se,0,0,e.canvasWidth,e.canvasHeight)},"drawMotionBlur");(f[e.NODE]||$[e.NODE])&&(he(j,ie,$[e.NODE]),f[e.NODE]=!1),(f[e.DRAG]||$[e.DRAG])&&(he(ne,le,$[e.DRAG]),f[e.DRAG]=!1)}e.prevViewport=A,e.clearingMotionBlur&&(e.clearingMotionBlur=!1,e.motionBlurCleared=!0,e.motionBlur=!0),p&&(e.motionBlurTimeout=setTimeout(function(){e.motionBlurTimeout=null,e.clearedForMotionBlur[e.NODE]=!1,e.clearedForMotionBlur[e.DRAG]=!1,e.motionBlur=!1,e.clearingMotionBlur=!d,e.mbFrames=0,f[e.NODE]=!0,f[e.DRAG]=!0,e.redraw()},dZe)),n||r.emit("render")};ys.drawSelectionRectangle=function(t,e){var r=this,n=r.cy,i=r.data,a=n.style(),s=t.drawOnlyNodeLayer,l=t.drawAllLayers,u=i.canvasNeedsRedraw,h=t.forcedContext;if(r.showFps||!s&&u[r.SELECT_BOX]&&!l){var f=h||i.contexts[r.SELECT_BOX];if(e(f),r.selection[4]==1&&(r.hoverData.selecting||r.touchData.selecting)){var d=r.cy.zoom(),p=a.core("selection-box-border-width").value/d;f.lineWidth=p,f.fillStyle="rgba("+a.core("selection-box-color").value[0]+","+a.core("selection-box-color").value[1]+","+a.core("selection-box-color").value[2]+","+a.core("selection-box-opacity").value+")",f.fillRect(r.selection[0],r.selection[1],r.selection[2]-r.selection[0],r.selection[3]-r.selection[1]),p>0&&(f.strokeStyle="rgba("+a.core("selection-box-border-color").value[0]+","+a.core("selection-box-border-color").value[1]+","+a.core("selection-box-border-color").value[2]+","+a.core("selection-box-opacity").value+")",f.strokeRect(r.selection[0],r.selection[1],r.selection[2]-r.selection[0],r.selection[3]-r.selection[1]))}if(i.bgActivePosistion&&!r.hoverData.selecting){var d=r.cy.zoom(),m=i.bgActivePosistion;f.fillStyle="rgba("+a.core("active-bg-color").value[0]+","+a.core("active-bg-color").value[1]+","+a.core("active-bg-color").value[2]+","+a.core("active-bg-opacity").value+")",f.beginPath(),f.arc(m.x,m.y,a.core("active-bg-size").pfValue/d,0,2*Math.PI),f.fill()}var g=r.lastRedrawTime;if(r.showFps&&g){g=Math.round(g);var y=Math.round(1e3/g),v="1 frame = "+g+" ms = "+y+" fps";if(f.setTransform(1,0,0,1,0,0),f.fillStyle="rgba(255, 0, 0, 0.75)",f.strokeStyle="rgba(255, 0, 0, 0.75)",f.font="30px Arial",!Nb){var x=f.measureText(v);Nb=x.actualBoundingBoxAscent}f.fillText(v,0,Nb);var b=60;f.strokeRect(0,Nb+10,250,20),f.fillRect(0,Nb+10,250*Math.min(y/b,1),20)}l||(u[r.SELECT_BOX]=!1)}};o(z0e,"compileShader");o(pZe,"createProgram");o(mZe,"createTextureCanvas");o(wB,"getEffectivePanZoom");o(NP,"modelToRenderedPosition");o(oS,"toWebGLColor");o(lS,"indexToVec4");o(gZe,"vec4ToIndex");o(yZe,"createTexture");o(Sge,"getTypeInfo");o(Cge,"createTypedArray");o(vZe,"createTypedArrayView");o(xZe,"createBufferStaticDraw");o(po,"createBufferDynamicDraw");o(bZe,"createPickingFrameBuffer");G0e=typeof Float32Array<"u"?Float32Array:Array;Math.hypot||(Math.hypot=function(){for(var t=0,e=arguments.length;e--;)t+=arguments[e]*arguments[e];return Math.sqrt(t)});o(Gb,"create");o(Age,"identity");o(wZe,"multiply");o(DS,"translate");o(_ge,"rotate");o(TB,"scale");o(TZe,"projection");Vb={SCREEN:{name:"screen",screen:!0},PICKING:{name:"picking",picking:!0}},Mb=la({getKey:null,drawElement:null,getBoundingBox:null,getRotation:null,getRotationPoint:null,getRotationOffset:null,isVisible:null,getPadding:null}),kZe=function(){function t(e,r){Mf(this,t),this.debugID=Math.floor(Math.random()*1e4),this.r=e,this.atlasSize=r.webglTexSize,this.rows=r.webglTexRows,this.enableWrapping=r.enableWrapping,this.texHeight=Math.floor(this.atlasSize/this.rows),this.maxTexWidth=this.atlasSize,this.texture=null,this.canvas=null,this.needsBuffer=!0,this.freePointer={x:0,row:0},this.keyToLocation=new Map,this.canvas=r.createTextureCanvas(e,this.atlasSize,this.atlasSize),this.scratch=r.createTextureCanvas(e,this.atlasSize,this.texHeight,"scratch")}return o(t,"Atlas"),If(t,[{key:"getKeys",value:o(function(){return new Set(this.keyToLocation.keys())},"getKeys")},{key:"getScale",value:o(function(r){var n=r.w,i=r.h,a=this.texHeight,s=this.maxTexWidth,l=a/i,u=n*l,h=i*l;return u>s&&(l=s/n,u=n*l,h=i*l),{scale:l,texW:u,texH:h}},"getScale")},{key:"draw",value:o(function(r,n,i){var a=this,s=this.atlasSize,l=this.rows,u=this.texHeight,h=this.getScale(n),f=h.scale,d=h.texW,p=h.texH,m=[null,null],g=o(function(w,C){if(i&&C){var T=C.context,E=w.x,A=w.row,S=E,_=u*A;T.save(),T.translate(S,_),T.scale(f,f),i(T,n),T.restore()}},"drawAt"),y=o(function(){g(a.freePointer,a.canvas),m[0]={x:a.freePointer.x,y:a.freePointer.row*u,w:d,h:p},m[1]={x:a.freePointer.x+d,y:a.freePointer.row*u,w:0,h:p},a.freePointer.x+=d,a.freePointer.x==s&&(a.freePointer.x=0,a.freePointer.row++)},"drawNormal"),v=o(function(){var w=a.scratch,C=a.canvas;w.clear(),g({x:0,row:0},w);var T=s-a.freePointer.x,E=d-T,A=u;{var S=a.freePointer.x,_=a.freePointer.row*u,I=T;C.context.drawImage(w,0,0,I,A,S,_,I,A),m[0]={x:S,y:_,w:I,h:p}}{var D=T,k=(a.freePointer.row+1)*u,L=E;C&&C.context.drawImage(w,D,0,L,A,0,k,L,A),m[1]={x:0,y:k,w:L,h:p}}a.freePointer.x=E,a.freePointer.row++},"drawWrapped"),x=o(function(){a.freePointer.x=0,a.freePointer.row++},"moveToStartOfNextRow");if(this.freePointer.x+d<=s)y();else{if(this.freePointer.row>=l-1)return!1;this.freePointer.x===s?(x(),y()):this.enableWrapping?v():(x(),y())}return this.keyToLocation.set(r,m),this.needsBuffer=!0,m},"draw")},{key:"getOffsets",value:o(function(r){return this.keyToLocation.get(r)},"getOffsets")},{key:"isEmpty",value:o(function(){return this.freePointer.x===0&&this.freePointer.row===0},"isEmpty")},{key:"canFit",value:o(function(r){var n=this.atlasSize,i=this.rows,a=this.getScale(r),s=a.texW;return this.freePointer.x+s>n?this.freePointer.row1&&arguments[1]!==void 0?arguments[1]:{},i=n.forceRedraw,a=i===void 0?!1:i,s=n.filterEle,l=s===void 0?function(){return!0}:s,u=n.filterType,h=u===void 0?function(){return!0}:u,f=!1,d=mo(r),p;try{for(d.s();!(p=d.n()).done;){var m=p.value;if(l(m)){var g=m.id(),y=mo(this.getRenderTypes()),v;try{for(y.s();!(v=y.n()).done;){var x=v.value;if(h(x.type)){var b=x.getKey(m);a?(x.atlasCollection.deleteKey(g,b),x.atlasCollection.styleKeyNeedsRedraw.add(b),f=!0):f|=x.atlasCollection.checkKeyIsInvalid(g,b)}}}catch(w){y.e(w)}finally{y.f()}}}}catch(w){d.e(w)}finally{d.f()}return f},"invalidate")},{key:"gc",value:o(function(){var r=mo(this.getRenderTypes()),n;try{for(r.s();!(n=r.n()).done;){var i=n.value;i.atlasCollection.gc()}}catch(a){r.e(a)}finally{r.f()}},"gc")},{key:"isRenderable",value:o(function(r,n){var i=this.getRenderTypeOpts(n);return i&&i.isVisible(r)},"isRenderable")},{key:"startBatch",value:o(function(){this.batchAtlases=[]},"startBatch")},{key:"getAtlasCount",value:o(function(){return this.batchAtlases.length},"getAtlasCount")},{key:"getAtlases",value:o(function(){return this.batchAtlases},"getAtlases")},{key:"getOrCreateAtlas",value:o(function(r,n,i){var a=this.renderTypes.get(i),s=a.getKey(r),l=r.id();return a.atlasCollection.draw(l,s,n,function(u){a.drawElement(u,r,n,!0,!0)})},"getOrCreateAtlas")},{key:"getAtlasIndexForBatch",value:o(function(r){var n=this.batchAtlases.indexOf(r);if(n<0){if(this.batchAtlases.length===this.maxAtlasesPerBatch)return;this.batchAtlases.push(r),n=this.batchAtlases.length-1}return n},"getAtlasIndexForBatch")},{key:"getIndexArray",value:o(function(){return Array.from({length:this.maxAtlases},function(r,n){return n})},"getIndexArray")},{key:"getAtlasInfo",value:o(function(r,n){var i=this.renderTypes.get(n),a=i.getBoundingBox(r),s=this.getOrCreateAtlas(r,a,n),l=this.getAtlasIndexForBatch(s);if(l!==void 0){var u=i.getKey(r),h=s.getOffsets(u),f=_i(h,2),d=f[0],p=f[1];return{atlasID:l,tex:d,tex1:d,tex2:p,bb:a,type:n,styleKey:u}}},"getAtlasInfo")},{key:"canAddToCurrentBatch",value:o(function(r,n){if(this.batchAtlases.length===this.maxAtlasesPerBatch){var i=this.renderTypes.get(n),a=i.getKey(r),s=i.atlasCollection.getAtlas(a);return s&&this.batchAtlases.includes(s)}return!0},"canAddToCurrentBatch")},{key:"setTransformMatrix",value:o(function(r,n,i){var a=arguments.length>3&&arguments[3]!==void 0?arguments[3]:!0,s=n.bb,l=n.type,u=n.tex1,h=n.tex2,f=this.getRenderTypeOpts(l),d=f.getPadding?f.getPadding(i):0,p=u.w/(u.w+h.w);a||(p=1-p);var m=this.getAdjustedBB(s,d,a,p),g,y;Age(r);var v=f.getRotation?f.getRotation(i):0;if(v!==0){var x=f.getRotationPoint(i),b=x.x,w=x.y;DS(r,r,[b,w]),_ge(r,r,v);var C=f.getRotationOffset(i);g=C.x+m.xOffset,y=C.y}else g=m.x1,y=m.y1;DS(r,r,[g,y]),TB(r,r,[m.w,m.h])},"setTransformMatrix")},{key:"getTransformMatrix",value:o(function(r,n){var i=arguments.length>2&&arguments[2]!==void 0?arguments[2]:!0,a=Gb();return this.setTransformMatrix(a,r,n,i),a},"getTransformMatrix")},{key:"getAdjustedBB",value:o(function(r,n,i,a){var s=r.x1,l=r.y1,u=r.w,h=r.h;n&&(s-=n,l-=n,u+=2*n,h+=2*n);var f=0,d=u*a;return i&&a<1?u=d:!i&&a<1&&(f=u-d,s+=f,u=d),{x1:s,y1:l,w:u,h,xOffset:f}},"getAdjustedBB")},{key:"getDebugInfo",value:o(function(){var r=[],n=mo(this.renderTypes),i;try{for(n.s();!(i=n.n()).done;){var a=_i(i.value,2),s=a[0],l=a[1],u=l.atlasCollection.getCounts(),h=u.keyCount,f=u.atlasCount;r.push({type:s,keyCount:h,atlasCount:f})}}catch(d){n.e(d)}finally{n.f()}return r},"getDebugInfo")}]),t}(),MP=0,V0e=1,U0e=2,IP=3,AZe=function(){function t(e,r,n){Mf(this,t),this.r=e,this.gl=r,this.maxInstances=n.webglBatchSize,this.maxAtlases=n.webglTexPerBatch,this.atlasSize=n.webglTexSize,this.bgColor=n.bgColor,n.enableWrapping=!0,n.createTextureCanvas=mZe,this.atlasManager=new CZe(e,n),this.program=this.createShaderProgram(Vb.SCREEN),this.pickingProgram=this.createShaderProgram(Vb.PICKING),this.vao=this.createVAO(),this.debugInfo=[]}return o(t,"ElementDrawingWebGL"),If(t,[{key:"addTextureRenderType",value:o(function(r,n){this.atlasManager.addRenderType(r,n)},"addTextureRenderType")},{key:"invalidate",value:o(function(r){var n=arguments.length>1&&arguments[1]!==void 0?arguments[1]:{},i=n.type,a=this.atlasManager;return i?a.invalidate(r,{filterType:o(function(l){return l===i},"filterType"),forceRedraw:!0}):a.invalidate(r)},"invalidate")},{key:"gc",value:o(function(){this.atlasManager.gc()},"gc")},{key:"createShaderProgram",value:o(function(r){var n=this.gl,i=`#version 300 es + precision highp float; + + uniform mat3 uPanZoomMatrix; + uniform int uAtlasSize; + + // instanced + in vec2 aPosition; + + // what are we rendering? + in int aVertType; + + // for picking + in vec4 aIndex; + + // For textures + in int aAtlasId; // which shader unit/atlas to use + in vec4 aTex1; // x/y/w/h of texture in atlas + in vec4 aTex2; + + // for any transforms that are needed + in vec4 aScaleRotate1; // vectors use fewer attributes than matrices + in vec2 aTranslate1; + in vec4 aScaleRotate2; + in vec2 aTranslate2; + + // for edges + in vec4 aPointAPointB; + in vec4 aPointCPointD; + in float aLineWidth; + in vec4 aEdgeColor; + + out vec2 vTexCoord; + out vec4 vEdgeColor; + flat out int vAtlasId; + flat out vec4 vIndex; + flat out int vVertType; + + void main(void) { + int vid = gl_VertexID; + vec2 position = aPosition; + + if(aVertType == `.concat(MP,`) { + float texX; + float texY; + float texW; + float texH; + mat3 texMatrix; + + int vid = gl_VertexID; + if(vid <= 5) { + texX = aTex1.x; + texY = aTex1.y; + texW = aTex1.z; + texH = aTex1.w; + texMatrix = mat3( + vec3(aScaleRotate1.xy, 0.0), + vec3(aScaleRotate2.zw, 0.0), + vec3(aTranslate1, 1.0) + ); + } else { + texX = aTex2.x; + texY = aTex2.y; + texW = aTex2.z; + texH = aTex2.w; + texMatrix = mat3( + vec3(aScaleRotate2.xy, 0.0), + vec3(aScaleRotate2.zw, 0.0), + vec3(aTranslate2, 1.0) + ); + } + + if(vid == 1 || vid == 2 || vid == 4 || vid == 7 || vid == 8 || vid == 10) { + texX += texW; + } + if(vid == 2 || vid == 4 || vid == 5 || vid == 8 || vid == 10 || vid == 11) { + texY += texH; + } + + float d = float(uAtlasSize); + vTexCoord = vec2(texX / d, texY / d); // tex coords must be between 0 and 1 + + gl_Position = vec4(uPanZoomMatrix * texMatrix * vec3(position, 1.0), 1.0); + } + else if(aVertType == `).concat(V0e,` && vid < 6) { + vec2 source = aPointAPointB.xy; + vec2 target = aPointAPointB.zw; + + // adjust the geometry so that the line is centered on the edge + position.y = position.y - 0.5; + + vec2 xBasis = target - source; + vec2 yBasis = normalize(vec2(-xBasis.y, xBasis.x)); + vec2 point = source + xBasis * position.x + yBasis * aLineWidth * position.y; + + gl_Position = vec4(uPanZoomMatrix * vec3(point, 1.0), 1.0); + vEdgeColor = aEdgeColor; + } + else if(aVertType == `).concat(U0e,` && vid < 6) { + vec2 pointA = aPointAPointB.xy; + vec2 pointB = aPointAPointB.zw; + vec2 pointC = aPointCPointD.xy; + vec2 pointD = aPointCPointD.zw; + + // adjust the geometry so that the line is centered on the edge + position.y = position.y - 0.5; + + vec2 p0 = pointA; + vec2 p1 = pointB; + vec2 p2 = pointC; + vec2 pos = position; + if(position.x == 1.0) { + p0 = pointD; + p1 = pointC; + p2 = pointB; + pos = vec2(0.0, -position.y); + } + + vec2 p01 = p1 - p0; + vec2 p12 = p2 - p1; + vec2 p21 = p1 - p2; + + // Find the normal vector. + vec2 tangent = normalize(normalize(p12) + normalize(p01)); + vec2 normal = vec2(-tangent.y, tangent.x); + + // Find the vector perpendicular to p0 -> p1. + vec2 p01Norm = normalize(vec2(-p01.y, p01.x)); + + // Determine the bend direction. + float sigma = sign(dot(p01 + p21, normal)); + float width = aLineWidth; + + if(sign(pos.y) == -sigma) { + // This is an intersecting vertex. Adjust the position so that there's no overlap. + vec2 point = 0.5 * width * normal * -sigma / dot(normal, p01Norm); + gl_Position = vec4(uPanZoomMatrix * vec3(p1 + point, 1.0), 1.0); + } else { + // This is a non-intersecting vertex. Treat it like a mitre join. + vec2 point = 0.5 * width * normal * sigma * dot(normal, p01Norm); + gl_Position = vec4(uPanZoomMatrix * vec3(p1 + point, 1.0), 1.0); + } + + vEdgeColor = aEdgeColor; + } + else if(aVertType == `).concat(IP,` && vid < 3) { + // massage the first triangle into an edge arrow + if(vid == 0) + position = vec2(-0.15, -0.3); + if(vid == 1) + position = vec2( 0.0, 0.0); + if(vid == 2) + position = vec2( 0.15, -0.3); + + mat3 transform = mat3( + vec3(aScaleRotate1.xy, 0.0), + vec3(aScaleRotate1.zw, 0.0), + vec3(aTranslate1, 1.0) + ); + gl_Position = vec4(uPanZoomMatrix * transform * vec3(position, 1.0), 1.0); + vEdgeColor = aEdgeColor; + } else { + gl_Position = vec4(2.0, 0.0, 0.0, 1.0); // discard vertex by putting it outside webgl clip space + } + + vAtlasId = aAtlasId; + vIndex = aIndex; + vVertType = aVertType; + } + `),a=this.atlasManager.getIndexArray(),s=`#version 300 es + precision highp float; + + // define texture unit for each node in the batch + `.concat(a.map(function(h){return"uniform sampler2D uTexture".concat(h,";")}).join(` + `),` + + uniform vec4 uBGColor; + + in vec2 vTexCoord; + in vec4 vEdgeColor; + flat in int vAtlasId; + flat in vec4 vIndex; + flat in int vVertType; + + out vec4 outColor; + + void main(void) { + if(vVertType == `).concat(MP,`) { + `).concat(a.map(function(h){return"if(vAtlasId == ".concat(h,") outColor = texture(uTexture").concat(h,", vTexCoord);")}).join(` + else `),` + } else if(vVertType == `).concat(IP,`) { + // blend arrow color with background (using premultiplied alpha) + outColor.rgb = vEdgeColor.rgb + (uBGColor.rgb * (1.0 - vEdgeColor.a)); + outColor.a = 1.0; // make opaque, masks out line under arrow + } else { + outColor = vEdgeColor; + } + + `).concat(r.picking?`if(outColor.a == 0.0) discard; + else outColor = vIndex;`:"",` + } + `),l=pZe(n,i,s);l.aPosition=n.getAttribLocation(l,"aPosition"),l.aIndex=n.getAttribLocation(l,"aIndex"),l.aVertType=n.getAttribLocation(l,"aVertType"),l.aAtlasId=n.getAttribLocation(l,"aAtlasId"),l.aTex1=n.getAttribLocation(l,"aTex1"),l.aTex2=n.getAttribLocation(l,"aTex2"),l.aScaleRotate1=n.getAttribLocation(l,"aScaleRotate1"),l.aTranslate1=n.getAttribLocation(l,"aTranslate1"),l.aScaleRotate2=n.getAttribLocation(l,"aScaleRotate2"),l.aTranslate2=n.getAttribLocation(l,"aTranslate2"),l.aPointAPointB=n.getAttribLocation(l,"aPointAPointB"),l.aPointCPointD=n.getAttribLocation(l,"aPointCPointD"),l.aLineWidth=n.getAttribLocation(l,"aLineWidth"),l.aEdgeColor=n.getAttribLocation(l,"aEdgeColor"),l.uPanZoomMatrix=n.getUniformLocation(l,"uPanZoomMatrix"),l.uAtlasSize=n.getUniformLocation(l,"uAtlasSize"),l.uBGColor=n.getUniformLocation(l,"uBGColor"),l.uTextures=[];for(var u=0;u2&&arguments[2]!==void 0?arguments[2]:Vb.SCREEN;this.panZoomMatrix=r,this.debugInfo=n,this.renderTarget=i,this.startBatch()},"startFrame")},{key:"startBatch",value:o(function(){this.instanceCount=0,this.atlasManager.startBatch()},"startBatch")},{key:"endFrame",value:o(function(){this.endBatch()},"endFrame")},{key:"getTempMatrix",value:o(function(){return this.tempMatrix=this.tempMatrix||Gb()},"getTempMatrix")},{key:"drawTexture",value:o(function(r,n,i){var a=this.atlasManager;if(a.isRenderable(r,i)){a.canAddToCurrentBatch(r,i)||this.endBatch();var s=this.instanceCount;this.vertTypeBuffer.getView(s)[0]=MP;var l=this.indexBuffer.getView(s);lS(n,l);var u=a.getAtlasInfo(r,i,u),h=u.atlasID,f=u.tex1,d=u.tex2,p=this.atlasIdBuffer.getView(s);p[0]=h;var m=this.tex1Buffer.getView(s);m[0]=f.x,m[1]=f.y,m[2]=f.w,m[3]=f.h;var g=this.tex2Buffer.getView(s);g[0]=d.x,g[1]=d.y,g[2]=d.w,g[3]=d.h;for(var y=this.getTempMatrix(),v=0,x=[1,2];v=this.maxInstances&&this.endBatch()}},"drawTexture")},{key:"drawEdgeArrow",value:o(function(r,n,i){var a=r._private.rscratch,s,l,u;if(i==="source"?(s=a.arrowStartX,l=a.arrowStartY,u=a.srcArrowAngle):(s=a.arrowEndX,l=a.arrowEndY,u=a.tgtArrowAngle),!(isNaN(s)||s==null||isNaN(l)||l==null||isNaN(u)||u==null)){var h=r.pstyle(i+"-arrow-shape").value;if(h!=="none"){var f=r.pstyle(i+"-arrow-color").value,d=r.pstyle("opacity").value,p=r.pstyle("line-opacity").value,m=d*p,g=r.pstyle("width").pfValue,y=r.pstyle("arrow-scale").value,v=this.r.getArrowWidth(g,y),x=this.getTempMatrix();Age(x),DS(x,x,[s,l]),TB(x,x,[v,v]),_ge(x,x,u);var b=this.instanceCount;this.vertTypeBuffer.getView(b)[0]=IP;var w=this.indexBuffer.getView(b);lS(n,w);var C=this.edgeColorBuffer.getView(b);oS(f,m,C);var T=this.scaleRotate1Buffer.getView(b);T[0]=x[0],T[1]=x[1],T[2]=x[3],T[3]=x[4];var E=this.translate1Buffer.getView(b);E[0]=x[6],E[1]=x[7],this.instanceCount++,this.instanceCount>=this.maxInstances&&this.endBatch()}}},"drawEdgeArrow")},{key:"drawEdgeLine",value:o(function(r,n){var i=r.pstyle("opacity").value,a=r.pstyle("line-opacity").value,s=r.pstyle("width").pfValue,l=r.pstyle("line-color").value,u=i*a,h=this.getEdgePoints(r);if(h.length/2+this.instanceCount>this.maxInstances&&this.endBatch(),h.length==4){var f=this.instanceCount;this.vertTypeBuffer.getView(f)[0]=V0e;var d=this.indexBuffer.getView(f);lS(n,d);var p=this.edgeColorBuffer.getView(f);oS(l,u,p);var m=this.lineWidthBuffer.getView(f);m[0]=s;var g=this.pointAPointBBuffer.getView(f);g[0]=h[0],g[1]=h[1],g[2]=h[2],g[3]=h[3],this.instanceCount++,this.instanceCount>=this.maxInstances&&this.endBatch()}else for(var y=0;y=this.maxInstances&&this.endBatch()}},"drawEdgeLine")},{key:"getEdgePoints",value:o(function(r){var n=r._private.rscratch,i=n.allpts;if(i.length==4)return i;var a=this.getNumSegments(r);return this.getCurveSegmentPoints(i,a)},"getEdgePoints")},{key:"getNumSegments",value:o(function(r){var n=15;return Math.min(Math.max(n,5),this.maxInstances)},"getNumSegments")},{key:"getCurveSegmentPoints",value:o(function(r,n){if(r.length==4)return r;for(var i=Array((n+1)*2),a=0;a<=n;a++)if(a==0)i[0]=r[0],i[1]=r[1];else if(a==n)i[a*2]=r[r.length-2],i[a*2+1]=r[r.length-1];else{var s=a/n;this.setCurvePoint(r,s,i,a*2)}return i},"getCurveSegmentPoints")},{key:"setCurvePoint",value:o(function(r,n,i,a){if(r.length<=2)i[a]=r[0],i[a+1]=r[1];else{for(var s=Array(r.length-2),l=0;l0},"isVisible")},{key:"getStyle",value:o(function(r,n){var i=n.pstyle("".concat(r,"-opacity")).value,a=n.pstyle("".concat(r,"-color")).value,s=n.pstyle("".concat(r,"-shape")).value;return{opacity:i,color:a,shape:s}},"getStyle")},{key:"getPadding",value:o(function(r,n){return n.pstyle("".concat(r,"-padding")).pfValue},"getPadding")},{key:"draw",value:o(function(r,n,i,a){if(this.isVisible(r,i)){var s=this.r,l=a.w,u=a.h,h=l/2,f=u/2,d=this.getStyle(r,i),p=d.shape,m=d.color,g=d.opacity;n.save(),n.fillStyle=H0e(m,g),p==="round-rectangle"||p==="roundrectangle"?s.drawRoundRectanglePath(n,h,f,l,u,"auto"):p==="ellipse"&&s.drawEllipsePath(n,h,f,l,u),n.fill(),n.restore()}},"draw")}]),t}();o(DZe,"getBGColor");Dge={};Dge.initWebgl=function(t,e){var r=this,n=r.data.contexts[r.WEBGL],i=t.cy.container();t.bgColor=DZe(i),t.webglTexSize=Math.min(t.webglTexSize,n.getParameter(n.MAX_TEXTURE_SIZE)),t.webglTexRows=Math.min(t.webglTexRows,54),t.webglBatchSize=Math.min(t.webglBatchSize,16384),t.webglTexPerBatch=Math.min(t.webglTexPerBatch,n.getParameter(n.MAX_TEXTURE_IMAGE_UNITS)),r.webglDebug=t.webglDebug,r.webglDebugShowAtlases=t.webglDebugShowAtlases,console.log("max texture units",n.getParameter(n.MAX_TEXTURE_IMAGE_UNITS)),console.log("max texture size",n.getParameter(n.MAX_TEXTURE_SIZE)),console.log("webgl options",t),r.pickingFrameBuffer=bZe(n),r.pickingFrameBuffer.needsDraw=!0;var a=o(function(f){return r.getTextAngle(f,null)},"getLabelRotation"),s=o(function(f){var d=f.pstyle("label");return d&&d.value},"isLabelVisible");r.eleDrawing=new AZe(r,n,t);var l=new _Ze(r);r.eleDrawing.addTextureRenderType("node-body",Mb({getKey:e.getStyleKey,getBoundingBox:e.getElementBox,drawElement:e.drawElement,isVisible:o(function(f){return f.visible()},"isVisible")})),r.eleDrawing.addTextureRenderType("node-label",Mb({getKey:e.getLabelKey,getBoundingBox:e.getLabelBox,drawElement:e.drawLabel,getRotation:a,getRotationPoint:e.getLabelRotationPoint,getRotationOffset:e.getLabelRotationOffset,isVisible:s})),r.eleDrawing.addTextureRenderType("node-overlay",Mb({getBoundingBox:e.getElementBox,getKey:o(function(f){return l.getStyleKey("overlay",f)},"getKey"),drawElement:o(function(f,d,p){return l.draw("overlay",f,d,p)},"drawElement"),isVisible:o(function(f){return l.isVisible("overlay",f)},"isVisible"),getPadding:o(function(f){return l.getPadding("overlay",f)},"getPadding")})),r.eleDrawing.addTextureRenderType("node-underlay",Mb({getBoundingBox:e.getElementBox,getKey:o(function(f){return l.getStyleKey("underlay",f)},"getKey"),drawElement:o(function(f,d,p){return l.draw("underlay",f,d,p)},"drawElement"),isVisible:o(function(f){return l.isVisible("underlay",f)},"isVisible"),getPadding:o(function(f){return l.getPadding("underlay",f)},"getPadding")})),r.eleDrawing.addTextureRenderType("edge-label",Mb({getKey:e.getLabelKey,getBoundingBox:e.getLabelBox,drawElement:e.drawLabel,getRotation:a,getRotationPoint:e.getLabelRotationPoint,getRotationOffset:e.getLabelRotationOffset,isVisible:s}));var u=n4(function(){console.log("garbage collect flag set"),r.data.gc=!0},1e4);r.onUpdateEleCalcs(function(h,f){var d=!1;f&&f.length>0&&(d|=r.eleDrawing.invalidate(f)),d&&u()}),LZe(r)};o(LZe,"overrideCanvasRendererFunctions");o(RZe,"clearWebgl");o(NZe,"clearCanvas");o(MZe,"createPanZoomMatrix");o(Lge,"setContextTransform");o(IZe,"drawSelectionRectangle");o(OZe,"drawAxes");o(PZe,"drawAtlases");o(BZe,"getPickingIndexes");o(FZe,"findNearestElementsWebgl");o(Rge,"renderWebgl");Pf={};Pf.drawPolygonPath=function(t,e,r,n,i,a){var s=n/2,l=i/2;t.beginPath&&t.beginPath(),t.moveTo(e+s*a[0],r+l*a[1]);for(var u=1;u0&&s>0){m.clearRect(0,0,a,s),m.globalCompositeOperation="source-over";var g=this.getCachedZSortedEles();if(t.full)m.translate(-n.x1*h,-n.y1*h),m.scale(h,h),this.drawElements(m,g),m.scale(1/h,1/h),m.translate(n.x1*h,n.y1*h);else{var y=e.pan(),v={x:y.x*h,y:y.y*h};h*=e.zoom(),m.translate(v.x,v.y),m.scale(h,h),this.drawElements(m,g),m.scale(1/h,1/h),m.translate(-v.x,-v.y)}t.bg&&(m.globalCompositeOperation="destination-over",m.fillStyle=t.bg,m.rect(0,0,a,s),m.fill())}return p};o($Ze,"b64ToBlob");o(Y0e,"b64UriToB64");o(Mge,"output");c4.png=function(t){return Mge(t,this.bufferCanvasImage(t),"image/png")};c4.jpg=function(t){return Mge(t,this.bufferCanvasImage(t),"image/jpeg")};Ige={};Ige.nodeShapeImpl=function(t,e,r,n,i,a,s,l){switch(t){case"ellipse":return this.drawEllipsePath(e,r,n,i,a);case"polygon":return this.drawPolygonPath(e,r,n,i,a,s);case"round-polygon":return this.drawRoundPolygonPath(e,r,n,i,a,s,l);case"roundrectangle":case"round-rectangle":return this.drawRoundRectanglePath(e,r,n,i,a,l);case"cutrectangle":case"cut-rectangle":return this.drawCutRectanglePath(e,r,n,i,a,s,l);case"bottomroundrectangle":case"bottom-round-rectangle":return this.drawBottomRoundRectanglePath(e,r,n,i,a,l);case"barrel":return this.drawBarrelPath(e,r,n,i,a)}};zZe=Oge,Er=Oge.prototype;Er.CANVAS_LAYERS=3;Er.SELECT_BOX=0;Er.DRAG=1;Er.NODE=2;Er.WEBGL=3;Er.CANVAS_TYPES=["2d","2d","2d","webgl2"];Er.BUFFER_COUNT=3;Er.TEXTURE_BUFFER=0;Er.MOTIONBLUR_BUFFER_NODE=1;Er.MOTIONBLUR_BUFFER_DRAG=2;o(Oge,"CanvasRenderer");Er.redrawHint=function(t,e){var r=this;switch(t){case"eles":r.data.canvasNeedsRedraw[Er.NODE]=e;break;case"drag":r.data.canvasNeedsRedraw[Er.DRAG]=e;break;case"select":r.data.canvasNeedsRedraw[Er.SELECT_BOX]=e;break;case"gc":r.data.gc=!0;break}};GZe=typeof Path2D<"u";Er.path2dEnabled=function(t){if(t===void 0)return this.pathsEnabled;this.pathsEnabled=!!t};Er.usePaths=function(){return GZe&&this.pathsEnabled};Er.setImgSmoothing=function(t,e){t.imageSmoothingEnabled!=null?t.imageSmoothingEnabled=e:(t.webkitImageSmoothingEnabled=e,t.mozImageSmoothingEnabled=e,t.msImageSmoothingEnabled=e)};Er.getImgSmoothing=function(t){return t.imageSmoothingEnabled!=null?t.imageSmoothingEnabled:t.webkitImageSmoothingEnabled||t.mozImageSmoothingEnabled||t.msImageSmoothingEnabled};Er.makeOffscreenCanvas=function(t,e){var r;if((typeof OffscreenCanvas>"u"?"undefined":Wi(OffscreenCanvas))!=="undefined")r=new OffscreenCanvas(t,e);else{var n=this.cy.window(),i=n.document;r=i.createElement("canvas"),r.width=t,r.height=e}return r};[Tge,Qc,th,bB,Yp,ly,ys,Dge,Pf,c4,Ige].forEach(function(t){rr(Er,t)});VZe=[{name:"null",impl:lge},{name:"base",impl:vge},{name:"canvas",impl:zZe}],UZe=[{type:"layout",extensions:SQe},{type:"renderer",extensions:VZe}],Pge={},Bge={};o(Fge,"setExtension");o($ge,"getExtension");o(HZe,"setModule");o(WZe,"getModule");QP=o(function(){if(arguments.length===2)return $ge.apply(null,arguments);if(arguments.length===3)return Fge.apply(null,arguments);if(arguments.length===4)return WZe.apply(null,arguments);if(arguments.length===5)return HZe.apply(null,arguments);ai("Invalid extension access syntax")},"extension");Jb.prototype.extension=QP;UZe.forEach(function(t){t.extensions.forEach(function(e){Fge(t.type,e.name,e.impl)})});zge=o(function t(){if(!(this instanceof t))return new t;this.length=0},"Stylesheet"),Wp=zge.prototype;Wp.instanceString=function(){return"stylesheet"};Wp.selector=function(t){var e=this.length++;return this[e]={selector:t,properties:[]},this};Wp.css=function(t,e){var r=this.length-1;if(Zt(t))this[r].properties.push({name:t,value:e});else if(Ur(t))for(var n=t,i=Object.keys(n),a=0;a{"use strict";o(function(e,r){typeof u4=="object"&&typeof EB=="object"?EB.exports=r():typeof define=="function"&&define.amd?define([],r):typeof u4=="object"?u4.layoutBase=r():e.layoutBase=r()},"webpackUniversalModuleDefinition")(u4,function(){return function(t){var e={};function r(n){if(e[n])return e[n].exports;var i=e[n]={i:n,l:!1,exports:{}};return t[n].call(i.exports,i,i.exports,r),i.l=!0,i.exports}return o(r,"__webpack_require__"),r.m=t,r.c=e,r.i=function(n){return n},r.d=function(n,i,a){r.o(n,i)||Object.defineProperty(n,i,{configurable:!1,enumerable:!0,get:a})},r.n=function(n){var i=n&&n.__esModule?o(function(){return n.default},"getDefault"):o(function(){return n},"getModuleExports");return r.d(i,"a",i),i},r.o=function(n,i){return Object.prototype.hasOwnProperty.call(n,i)},r.p="",r(r.s=26)}([function(t,e,r){"use strict";function n(){}o(n,"LayoutConstants"),n.QUALITY=1,n.DEFAULT_CREATE_BENDS_AS_NEEDED=!1,n.DEFAULT_INCREMENTAL=!1,n.DEFAULT_ANIMATION_ON_LAYOUT=!0,n.DEFAULT_ANIMATION_DURING_LAYOUT=!1,n.DEFAULT_ANIMATION_PERIOD=50,n.DEFAULT_UNIFORM_LEAF_NODE_SIZES=!1,n.DEFAULT_GRAPH_MARGIN=15,n.NODE_DIMENSIONS_INCLUDE_LABELS=!1,n.SIMPLE_NODE_SIZE=40,n.SIMPLE_NODE_HALF_SIZE=n.SIMPLE_NODE_SIZE/2,n.EMPTY_COMPOUND_NODE_SIZE=40,n.MIN_EDGE_LENGTH=1,n.WORLD_BOUNDARY=1e6,n.INITIAL_WORLD_BOUNDARY=n.WORLD_BOUNDARY/1e3,n.WORLD_CENTER_X=1200,n.WORLD_CENTER_Y=900,t.exports=n},function(t,e,r){"use strict";var n=r(2),i=r(8),a=r(9);function s(u,h,f){n.call(this,f),this.isOverlapingSourceAndTarget=!1,this.vGraphObject=f,this.bendpoints=[],this.source=u,this.target=h}o(s,"LEdge"),s.prototype=Object.create(n.prototype);for(var l in n)s[l]=n[l];s.prototype.getSource=function(){return this.source},s.prototype.getTarget=function(){return this.target},s.prototype.isInterGraph=function(){return this.isInterGraph},s.prototype.getLength=function(){return this.length},s.prototype.isOverlapingSourceAndTarget=function(){return this.isOverlapingSourceAndTarget},s.prototype.getBendpoints=function(){return this.bendpoints},s.prototype.getLca=function(){return this.lca},s.prototype.getSourceInLca=function(){return this.sourceInLca},s.prototype.getTargetInLca=function(){return this.targetInLca},s.prototype.getOtherEnd=function(u){if(this.source===u)return this.target;if(this.target===u)return this.source;throw"Node is not incident with this edge"},s.prototype.getOtherEndInGraph=function(u,h){for(var f=this.getOtherEnd(u),d=h.getGraphManager().getRoot();;){if(f.getOwner()==h)return f;if(f.getOwner()==d)break;f=f.getOwner().getParent()}return null},s.prototype.updateLength=function(){var u=new Array(4);this.isOverlapingSourceAndTarget=i.getIntersection(this.target.getRect(),this.source.getRect(),u),this.isOverlapingSourceAndTarget||(this.lengthX=u[0]-u[2],this.lengthY=u[1]-u[3],Math.abs(this.lengthX)<1&&(this.lengthX=a.sign(this.lengthX)),Math.abs(this.lengthY)<1&&(this.lengthY=a.sign(this.lengthY)),this.length=Math.sqrt(this.lengthX*this.lengthX+this.lengthY*this.lengthY))},s.prototype.updateLengthSimple=function(){this.lengthX=this.target.getCenterX()-this.source.getCenterX(),this.lengthY=this.target.getCenterY()-this.source.getCenterY(),Math.abs(this.lengthX)<1&&(this.lengthX=a.sign(this.lengthX)),Math.abs(this.lengthY)<1&&(this.lengthY=a.sign(this.lengthY)),this.length=Math.sqrt(this.lengthX*this.lengthX+this.lengthY*this.lengthY)},t.exports=s},function(t,e,r){"use strict";function n(i){this.vGraphObject=i}o(n,"LGraphObject"),t.exports=n},function(t,e,r){"use strict";var n=r(2),i=r(10),a=r(13),s=r(0),l=r(16),u=r(4);function h(d,p,m,g){m==null&&g==null&&(g=p),n.call(this,g),d.graphManager!=null&&(d=d.graphManager),this.estimatedSize=i.MIN_VALUE,this.inclusionTreeDepth=i.MAX_VALUE,this.vGraphObject=g,this.edges=[],this.graphManager=d,m!=null&&p!=null?this.rect=new a(p.x,p.y,m.width,m.height):this.rect=new a}o(h,"LNode"),h.prototype=Object.create(n.prototype);for(var f in n)h[f]=n[f];h.prototype.getEdges=function(){return this.edges},h.prototype.getChild=function(){return this.child},h.prototype.getOwner=function(){return this.owner},h.prototype.getWidth=function(){return this.rect.width},h.prototype.setWidth=function(d){this.rect.width=d},h.prototype.getHeight=function(){return this.rect.height},h.prototype.setHeight=function(d){this.rect.height=d},h.prototype.getCenterX=function(){return this.rect.x+this.rect.width/2},h.prototype.getCenterY=function(){return this.rect.y+this.rect.height/2},h.prototype.getCenter=function(){return new u(this.rect.x+this.rect.width/2,this.rect.y+this.rect.height/2)},h.prototype.getLocation=function(){return new u(this.rect.x,this.rect.y)},h.prototype.getRect=function(){return this.rect},h.prototype.getDiagonal=function(){return Math.sqrt(this.rect.width*this.rect.width+this.rect.height*this.rect.height)},h.prototype.getHalfTheDiagonal=function(){return Math.sqrt(this.rect.height*this.rect.height+this.rect.width*this.rect.width)/2},h.prototype.setRect=function(d,p){this.rect.x=d.x,this.rect.y=d.y,this.rect.width=p.width,this.rect.height=p.height},h.prototype.setCenter=function(d,p){this.rect.x=d-this.rect.width/2,this.rect.y=p-this.rect.height/2},h.prototype.setLocation=function(d,p){this.rect.x=d,this.rect.y=p},h.prototype.moveBy=function(d,p){this.rect.x+=d,this.rect.y+=p},h.prototype.getEdgeListToNode=function(d){var p=[],m,g=this;return g.edges.forEach(function(y){if(y.target==d){if(y.source!=g)throw"Incorrect edge source!";p.push(y)}}),p},h.prototype.getEdgesBetween=function(d){var p=[],m,g=this;return g.edges.forEach(function(y){if(!(y.source==g||y.target==g))throw"Incorrect edge source and/or target";(y.target==d||y.source==d)&&p.push(y)}),p},h.prototype.getNeighborsList=function(){var d=new Set,p=this;return p.edges.forEach(function(m){if(m.source==p)d.add(m.target);else{if(m.target!=p)throw"Incorrect incidency!";d.add(m.source)}}),d},h.prototype.withChildren=function(){var d=new Set,p,m;if(d.add(this),this.child!=null)for(var g=this.child.getNodes(),y=0;yp&&(this.rect.x-=(this.labelWidth-p)/2,this.setWidth(this.labelWidth)),this.labelHeight>m&&(this.labelPos=="center"?this.rect.y-=(this.labelHeight-m)/2:this.labelPos=="top"&&(this.rect.y-=this.labelHeight-m),this.setHeight(this.labelHeight))}}},h.prototype.getInclusionTreeDepth=function(){if(this.inclusionTreeDepth==i.MAX_VALUE)throw"assert failed";return this.inclusionTreeDepth},h.prototype.transform=function(d){var p=this.rect.x;p>s.WORLD_BOUNDARY?p=s.WORLD_BOUNDARY:p<-s.WORLD_BOUNDARY&&(p=-s.WORLD_BOUNDARY);var m=this.rect.y;m>s.WORLD_BOUNDARY?m=s.WORLD_BOUNDARY:m<-s.WORLD_BOUNDARY&&(m=-s.WORLD_BOUNDARY);var g=new u(p,m),y=d.inverseTransformPoint(g);this.setLocation(y.x,y.y)},h.prototype.getLeft=function(){return this.rect.x},h.prototype.getRight=function(){return this.rect.x+this.rect.width},h.prototype.getTop=function(){return this.rect.y},h.prototype.getBottom=function(){return this.rect.y+this.rect.height},h.prototype.getParent=function(){return this.owner==null?null:this.owner.getParent()},t.exports=h},function(t,e,r){"use strict";function n(i,a){i==null&&a==null?(this.x=0,this.y=0):(this.x=i,this.y=a)}o(n,"PointD"),n.prototype.getX=function(){return this.x},n.prototype.getY=function(){return this.y},n.prototype.setX=function(i){this.x=i},n.prototype.setY=function(i){this.y=i},n.prototype.getDifference=function(i){return new DimensionD(this.x-i.x,this.y-i.y)},n.prototype.getCopy=function(){return new n(this.x,this.y)},n.prototype.translate=function(i){return this.x+=i.width,this.y+=i.height,this},t.exports=n},function(t,e,r){"use strict";var n=r(2),i=r(10),a=r(0),s=r(6),l=r(3),u=r(1),h=r(13),f=r(12),d=r(11);function p(g,y,v){n.call(this,v),this.estimatedSize=i.MIN_VALUE,this.margin=a.DEFAULT_GRAPH_MARGIN,this.edges=[],this.nodes=[],this.isConnected=!1,this.parent=g,y!=null&&y instanceof s?this.graphManager=y:y!=null&&y instanceof Layout&&(this.graphManager=y.graphManager)}o(p,"LGraph"),p.prototype=Object.create(n.prototype);for(var m in n)p[m]=n[m];p.prototype.getNodes=function(){return this.nodes},p.prototype.getEdges=function(){return this.edges},p.prototype.getGraphManager=function(){return this.graphManager},p.prototype.getParent=function(){return this.parent},p.prototype.getLeft=function(){return this.left},p.prototype.getRight=function(){return this.right},p.prototype.getTop=function(){return this.top},p.prototype.getBottom=function(){return this.bottom},p.prototype.isConnected=function(){return this.isConnected},p.prototype.add=function(g,y,v){if(y==null&&v==null){var x=g;if(this.graphManager==null)throw"Graph has no graph mgr!";if(this.getNodes().indexOf(x)>-1)throw"Node already in graph!";return x.owner=this,this.getNodes().push(x),x}else{var b=g;if(!(this.getNodes().indexOf(y)>-1&&this.getNodes().indexOf(v)>-1))throw"Source or target not in graph!";if(!(y.owner==v.owner&&y.owner==this))throw"Both owners must be this graph!";return y.owner!=v.owner?null:(b.source=y,b.target=v,b.isInterGraph=!1,this.getEdges().push(b),y.edges.push(b),v!=y&&v.edges.push(b),b)}},p.prototype.remove=function(g){var y=g;if(g instanceof l){if(y==null)throw"Node is null!";if(!(y.owner!=null&&y.owner==this))throw"Owner graph is invalid!";if(this.graphManager==null)throw"Owner graph manager is invalid!";for(var v=y.edges.slice(),x,b=v.length,w=0;w-1&&E>-1))throw"Source and/or target doesn't know this edge!";x.source.edges.splice(T,1),x.target!=x.source&&x.target.edges.splice(E,1);var C=x.source.owner.getEdges().indexOf(x);if(C==-1)throw"Not in owner's edge list!";x.source.owner.getEdges().splice(C,1)}},p.prototype.updateLeftTop=function(){for(var g=i.MAX_VALUE,y=i.MAX_VALUE,v,x,b,w=this.getNodes(),C=w.length,T=0;Tv&&(g=v),y>x&&(y=x)}return g==i.MAX_VALUE?null:(w[0].getParent().paddingLeft!=null?b=w[0].getParent().paddingLeft:b=this.margin,this.left=y-b,this.top=g-b,new f(this.left,this.top))},p.prototype.updateBounds=function(g){for(var y=i.MAX_VALUE,v=-i.MAX_VALUE,x=i.MAX_VALUE,b=-i.MAX_VALUE,w,C,T,E,A,S=this.nodes,_=S.length,I=0;I<_;I++){var D=S[I];g&&D.child!=null&&D.updateBounds(),w=D.getLeft(),C=D.getRight(),T=D.getTop(),E=D.getBottom(),y>w&&(y=w),vT&&(x=T),bw&&(y=w),vT&&(x=T),b=this.nodes.length){var _=0;v.forEach(function(I){I.owner==g&&_++}),_==this.nodes.length&&(this.isConnected=!0)}},t.exports=p},function(t,e,r){"use strict";var n,i=r(1);function a(s){n=r(5),this.layout=s,this.graphs=[],this.edges=[]}o(a,"LGraphManager"),a.prototype.addRoot=function(){var s=this.layout.newGraph(),l=this.layout.newNode(null),u=this.add(s,l);return this.setRootGraph(u),this.rootGraph},a.prototype.add=function(s,l,u,h,f){if(u==null&&h==null&&f==null){if(s==null)throw"Graph is null!";if(l==null)throw"Parent node is null!";if(this.graphs.indexOf(s)>-1)throw"Graph already in this graph mgr!";if(this.graphs.push(s),s.parent!=null)throw"Already has a parent!";if(l.child!=null)throw"Already has a child!";return s.parent=l,l.child=s,s}else{f=u,h=l,u=s;var d=h.getOwner(),p=f.getOwner();if(!(d!=null&&d.getGraphManager()==this))throw"Source not in this graph mgr!";if(!(p!=null&&p.getGraphManager()==this))throw"Target not in this graph mgr!";if(d==p)return u.isInterGraph=!1,d.add(u,h,f);if(u.isInterGraph=!0,u.source=h,u.target=f,this.edges.indexOf(u)>-1)throw"Edge already in inter-graph edge list!";if(this.edges.push(u),!(u.source!=null&&u.target!=null))throw"Edge source and/or target is null!";if(!(u.source.edges.indexOf(u)==-1&&u.target.edges.indexOf(u)==-1))throw"Edge already in source and/or target incidency list!";return u.source.edges.push(u),u.target.edges.push(u),u}},a.prototype.remove=function(s){if(s instanceof n){var l=s;if(l.getGraphManager()!=this)throw"Graph not in this graph mgr";if(!(l==this.rootGraph||l.parent!=null&&l.parent.graphManager==this))throw"Invalid parent node!";var u=[];u=u.concat(l.getEdges());for(var h,f=u.length,d=0;d=s.getRight()?l[0]+=Math.min(s.getX()-a.getX(),a.getRight()-s.getRight()):s.getX()<=a.getX()&&s.getRight()>=a.getRight()&&(l[0]+=Math.min(a.getX()-s.getX(),s.getRight()-a.getRight())),a.getY()<=s.getY()&&a.getBottom()>=s.getBottom()?l[1]+=Math.min(s.getY()-a.getY(),a.getBottom()-s.getBottom()):s.getY()<=a.getY()&&s.getBottom()>=a.getBottom()&&(l[1]+=Math.min(a.getY()-s.getY(),s.getBottom()-a.getBottom()));var f=Math.abs((s.getCenterY()-a.getCenterY())/(s.getCenterX()-a.getCenterX()));s.getCenterY()===a.getCenterY()&&s.getCenterX()===a.getCenterX()&&(f=1);var d=f*l[0],p=l[1]/f;l[0]d)return l[0]=u,l[1]=m,l[2]=f,l[3]=S,!1;if(hf)return l[0]=p,l[1]=h,l[2]=E,l[3]=d,!1;if(uf?(l[0]=y,l[1]=v,k=!0):(l[0]=g,l[1]=m,k=!0):R===M&&(u>f?(l[0]=p,l[1]=m,k=!0):(l[0]=x,l[1]=v,k=!0)),-O===M?f>u?(l[2]=A,l[3]=S,L=!0):(l[2]=E,l[3]=T,L=!0):O===M&&(f>u?(l[2]=C,l[3]=T,L=!0):(l[2]=_,l[3]=S,L=!0)),k&&L)return!1;if(u>f?h>d?(B=this.getCardinalDirection(R,M,4),F=this.getCardinalDirection(O,M,2)):(B=this.getCardinalDirection(-R,M,3),F=this.getCardinalDirection(-O,M,1)):h>d?(B=this.getCardinalDirection(-R,M,1),F=this.getCardinalDirection(-O,M,3)):(B=this.getCardinalDirection(R,M,2),F=this.getCardinalDirection(O,M,4)),!k)switch(B){case 1:z=m,P=u+-w/M,l[0]=P,l[1]=z;break;case 2:P=x,z=h+b*M,l[0]=P,l[1]=z;break;case 3:z=v,P=u+w/M,l[0]=P,l[1]=z;break;case 4:P=y,z=h+-b*M,l[0]=P,l[1]=z;break}if(!L)switch(F){case 1:H=T,$=f+-D/M,l[2]=$,l[3]=H;break;case 2:$=_,H=d+I*M,l[2]=$,l[3]=H;break;case 3:H=S,$=f+D/M,l[2]=$,l[3]=H;break;case 4:$=A,H=d+-I*M,l[2]=$,l[3]=H;break}}return!1},i.getCardinalDirection=function(a,s,l){return a>s?l:1+l%4},i.getIntersection=function(a,s,l,u){if(u==null)return this.getIntersection2(a,s,l);var h=a.x,f=a.y,d=s.x,p=s.y,m=l.x,g=l.y,y=u.x,v=u.y,x=void 0,b=void 0,w=void 0,C=void 0,T=void 0,E=void 0,A=void 0,S=void 0,_=void 0;return w=p-f,T=h-d,A=d*f-h*p,C=v-g,E=m-y,S=y*g-m*v,_=w*E-C*T,_===0?null:(x=(T*S-E*A)/_,b=(C*A-w*S)/_,new n(x,b))},i.angleOfVector=function(a,s,l,u){var h=void 0;return a!==l?(h=Math.atan((u-s)/(l-a)),l0?1:i<0?-1:0},n.floor=function(i){return i<0?Math.ceil(i):Math.floor(i)},n.ceil=function(i){return i<0?Math.floor(i):Math.ceil(i)},t.exports=n},function(t,e,r){"use strict";function n(){}o(n,"Integer"),n.MAX_VALUE=2147483647,n.MIN_VALUE=-2147483648,t.exports=n},function(t,e,r){"use strict";var n=function(){function h(f,d){for(var p=0;p"u"?"undefined":n(a);return a==null||s!="object"&&s!="function"},t.exports=i},function(t,e,r){"use strict";function n(m){if(Array.isArray(m)){for(var g=0,y=Array(m.length);g0&&g;){for(w.push(T[0]);w.length>0&&g;){var E=w[0];w.splice(0,1),b.add(E);for(var A=E.getEdges(),x=0;x-1&&T.splice(D,1)}b=new Set,C=new Map}}return m},p.prototype.createDummyNodesForBendpoints=function(m){for(var g=[],y=m.source,v=this.graphManager.calcLowestCommonAncestor(m.source,m.target),x=0;x0){for(var v=this.edgeToDummyNodes.get(y),x=0;x=0&&g.splice(S,1);var _=C.getNeighborsList();_.forEach(function(k){if(y.indexOf(k)<0){var L=v.get(k),R=L-1;R==1&&E.push(k),v.set(k,R)}})}y=y.concat(E),(g.length==1||g.length==2)&&(x=!0,b=g[0])}return b},p.prototype.setGraphManager=function(m){this.graphManager=m},t.exports=p},function(t,e,r){"use strict";function n(){}o(n,"RandomSeed"),n.seed=1,n.x=0,n.nextDouble=function(){return n.x=Math.sin(n.seed++)*1e4,n.x-Math.floor(n.x)},t.exports=n},function(t,e,r){"use strict";var n=r(4);function i(a,s){this.lworldOrgX=0,this.lworldOrgY=0,this.ldeviceOrgX=0,this.ldeviceOrgY=0,this.lworldExtX=1,this.lworldExtY=1,this.ldeviceExtX=1,this.ldeviceExtY=1}o(i,"Transform"),i.prototype.getWorldOrgX=function(){return this.lworldOrgX},i.prototype.setWorldOrgX=function(a){this.lworldOrgX=a},i.prototype.getWorldOrgY=function(){return this.lworldOrgY},i.prototype.setWorldOrgY=function(a){this.lworldOrgY=a},i.prototype.getWorldExtX=function(){return this.lworldExtX},i.prototype.setWorldExtX=function(a){this.lworldExtX=a},i.prototype.getWorldExtY=function(){return this.lworldExtY},i.prototype.setWorldExtY=function(a){this.lworldExtY=a},i.prototype.getDeviceOrgX=function(){return this.ldeviceOrgX},i.prototype.setDeviceOrgX=function(a){this.ldeviceOrgX=a},i.prototype.getDeviceOrgY=function(){return this.ldeviceOrgY},i.prototype.setDeviceOrgY=function(a){this.ldeviceOrgY=a},i.prototype.getDeviceExtX=function(){return this.ldeviceExtX},i.prototype.setDeviceExtX=function(a){this.ldeviceExtX=a},i.prototype.getDeviceExtY=function(){return this.ldeviceExtY},i.prototype.setDeviceExtY=function(a){this.ldeviceExtY=a},i.prototype.transformX=function(a){var s=0,l=this.lworldExtX;return l!=0&&(s=this.ldeviceOrgX+(a-this.lworldOrgX)*this.ldeviceExtX/l),s},i.prototype.transformY=function(a){var s=0,l=this.lworldExtY;return l!=0&&(s=this.ldeviceOrgY+(a-this.lworldOrgY)*this.ldeviceExtY/l),s},i.prototype.inverseTransformX=function(a){var s=0,l=this.ldeviceExtX;return l!=0&&(s=this.lworldOrgX+(a-this.ldeviceOrgX)*this.lworldExtX/l),s},i.prototype.inverseTransformY=function(a){var s=0,l=this.ldeviceExtY;return l!=0&&(s=this.lworldOrgY+(a-this.ldeviceOrgY)*this.lworldExtY/l),s},i.prototype.inverseTransformPoint=function(a){var s=new n(this.inverseTransformX(a.x),this.inverseTransformY(a.y));return s},t.exports=i},function(t,e,r){"use strict";function n(d){if(Array.isArray(d)){for(var p=0,m=Array(d.length);pa.ADAPTATION_LOWER_NODE_LIMIT&&(this.coolingFactor=Math.max(this.coolingFactor*a.COOLING_ADAPTATION_FACTOR,this.coolingFactor-(d-a.ADAPTATION_LOWER_NODE_LIMIT)/(a.ADAPTATION_UPPER_NODE_LIMIT-a.ADAPTATION_LOWER_NODE_LIMIT)*this.coolingFactor*(1-a.COOLING_ADAPTATION_FACTOR))),this.maxNodeDisplacement=a.MAX_NODE_DISPLACEMENT_INCREMENTAL):(d>a.ADAPTATION_LOWER_NODE_LIMIT?this.coolingFactor=Math.max(a.COOLING_ADAPTATION_FACTOR,1-(d-a.ADAPTATION_LOWER_NODE_LIMIT)/(a.ADAPTATION_UPPER_NODE_LIMIT-a.ADAPTATION_LOWER_NODE_LIMIT)*(1-a.COOLING_ADAPTATION_FACTOR)):this.coolingFactor=1,this.initialCoolingFactor=this.coolingFactor,this.maxNodeDisplacement=a.MAX_NODE_DISPLACEMENT),this.maxIterations=Math.max(this.getAllNodes().length*5,this.maxIterations),this.totalDisplacementThreshold=this.displacementThresholdPerNode*this.getAllNodes().length,this.repulsionRange=this.calcRepulsionRange()},h.prototype.calcSpringForces=function(){for(var d=this.getAllEdges(),p,m=0;m0&&arguments[0]!==void 0?arguments[0]:!0,p=arguments.length>1&&arguments[1]!==void 0?arguments[1]:!1,m,g,y,v,x=this.getAllNodes(),b;if(this.useFRGridVariant)for(this.totalIterations%a.GRID_CALCULATION_CHECK_PERIOD==1&&d&&this.updateGrid(),b=new Set,m=0;mw||b>w)&&(d.gravitationForceX=-this.gravityConstant*y,d.gravitationForceY=-this.gravityConstant*v)):(w=p.getEstimatedSize()*this.compoundGravityRangeFactor,(x>w||b>w)&&(d.gravitationForceX=-this.gravityConstant*y*this.compoundGravityConstant,d.gravitationForceY=-this.gravityConstant*v*this.compoundGravityConstant))},h.prototype.isConverged=function(){var d,p=!1;return this.totalIterations>this.maxIterations/3&&(p=Math.abs(this.totalDisplacement-this.oldTotalDisplacement)<2),d=this.totalDisplacement=x.length||w>=x[0].length)){for(var C=0;Ch},"_defaultCompareFunction")}]),l}();t.exports=s},function(t,e,r){"use strict";var n=function(){function s(l,u){for(var h=0;h2&&arguments[2]!==void 0?arguments[2]:1,f=arguments.length>3&&arguments[3]!==void 0?arguments[3]:-1,d=arguments.length>4&&arguments[4]!==void 0?arguments[4]:-1;i(this,s),this.sequence1=l,this.sequence2=u,this.match_score=h,this.mismatch_penalty=f,this.gap_penalty=d,this.iMax=l.length+1,this.jMax=u.length+1,this.grid=new Array(this.iMax);for(var p=0;p=0;l--){var u=this.listeners[l];u.event===a&&u.callback===s&&this.listeners.splice(l,1)}},i.emit=function(a,s){for(var l=0;l{"use strict";o(function(e,r){typeof h4=="object"&&typeof CB=="object"?CB.exports=r(SB()):typeof define=="function"&&define.amd?define(["layout-base"],r):typeof h4=="object"?h4.coseBase=r(SB()):e.coseBase=r(e.layoutBase)},"webpackUniversalModuleDefinition")(h4,function(t){return function(e){var r={};function n(i){if(r[i])return r[i].exports;var a=r[i]={i,l:!1,exports:{}};return e[i].call(a.exports,a,a.exports,n),a.l=!0,a.exports}return o(n,"__webpack_require__"),n.m=e,n.c=r,n.i=function(i){return i},n.d=function(i,a,s){n.o(i,a)||Object.defineProperty(i,a,{configurable:!1,enumerable:!0,get:s})},n.n=function(i){var a=i&&i.__esModule?o(function(){return i.default},"getDefault"):o(function(){return i},"getModuleExports");return n.d(a,"a",a),a},n.o=function(i,a){return Object.prototype.hasOwnProperty.call(i,a)},n.p="",n(n.s=7)}([function(e,r){e.exports=t},function(e,r,n){"use strict";var i=n(0).FDLayoutConstants;function a(){}o(a,"CoSEConstants");for(var s in i)a[s]=i[s];a.DEFAULT_USE_MULTI_LEVEL_SCALING=!1,a.DEFAULT_RADIAL_SEPARATION=i.DEFAULT_EDGE_LENGTH,a.DEFAULT_COMPONENT_SEPERATION=60,a.TILE=!0,a.TILING_PADDING_VERTICAL=10,a.TILING_PADDING_HORIZONTAL=10,a.TREE_REDUCTION_ON_INCREMENTAL=!1,e.exports=a},function(e,r,n){"use strict";var i=n(0).FDLayoutEdge;function a(l,u,h){i.call(this,l,u,h)}o(a,"CoSEEdge"),a.prototype=Object.create(i.prototype);for(var s in i)a[s]=i[s];e.exports=a},function(e,r,n){"use strict";var i=n(0).LGraph;function a(l,u,h){i.call(this,l,u,h)}o(a,"CoSEGraph"),a.prototype=Object.create(i.prototype);for(var s in i)a[s]=i[s];e.exports=a},function(e,r,n){"use strict";var i=n(0).LGraphManager;function a(l){i.call(this,l)}o(a,"CoSEGraphManager"),a.prototype=Object.create(i.prototype);for(var s in i)a[s]=i[s];e.exports=a},function(e,r,n){"use strict";var i=n(0).FDLayoutNode,a=n(0).IMath;function s(u,h,f,d){i.call(this,u,h,f,d)}o(s,"CoSENode"),s.prototype=Object.create(i.prototype);for(var l in i)s[l]=i[l];s.prototype.move=function(){var u=this.graphManager.getLayout();this.displacementX=u.coolingFactor*(this.springForceX+this.repulsionForceX+this.gravitationForceX)/this.noOfChildren,this.displacementY=u.coolingFactor*(this.springForceY+this.repulsionForceY+this.gravitationForceY)/this.noOfChildren,Math.abs(this.displacementX)>u.coolingFactor*u.maxNodeDisplacement&&(this.displacementX=u.coolingFactor*u.maxNodeDisplacement*a.sign(this.displacementX)),Math.abs(this.displacementY)>u.coolingFactor*u.maxNodeDisplacement&&(this.displacementY=u.coolingFactor*u.maxNodeDisplacement*a.sign(this.displacementY)),this.child==null?this.moveBy(this.displacementX,this.displacementY):this.child.getNodes().length==0?this.moveBy(this.displacementX,this.displacementY):this.propogateDisplacementToChildren(this.displacementX,this.displacementY),u.totalDisplacement+=Math.abs(this.displacementX)+Math.abs(this.displacementY),this.springForceX=0,this.springForceY=0,this.repulsionForceX=0,this.repulsionForceY=0,this.gravitationForceX=0,this.gravitationForceY=0,this.displacementX=0,this.displacementY=0},s.prototype.propogateDisplacementToChildren=function(u,h){for(var f=this.getChild().getNodes(),d,p=0;p0)this.positionNodesRadially(T);else{this.reduceTrees(),this.graphManager.resetAllNodesToApplyGravitation();var E=new Set(this.getAllNodes()),A=this.nodesWithGravity.filter(function(S){return E.has(S)});this.graphManager.setAllNodesToApplyGravitation(A),this.positionNodesRandomly()}}return this.initSpringEmbedder(),this.runSpringEmbedder(),!0},w.prototype.tick=function(){if(this.totalIterations++,this.totalIterations===this.maxIterations&&!this.isTreeGrowing&&!this.isGrowthFinished)if(this.prunedNodesAll.length>0)this.isTreeGrowing=!0;else return!0;if(this.totalIterations%f.CONVERGENCE_CHECK_PERIOD==0&&!this.isTreeGrowing&&!this.isGrowthFinished){if(this.isConverged())if(this.prunedNodesAll.length>0)this.isTreeGrowing=!0;else return!0;this.coolingCycle++,this.layoutQuality==0?this.coolingAdjuster=this.coolingCycle:this.layoutQuality==1&&(this.coolingAdjuster=this.coolingCycle/3),this.coolingFactor=Math.max(this.initialCoolingFactor-Math.pow(this.coolingCycle,Math.log(100*(this.initialCoolingFactor-this.finalTemperature))/Math.log(this.maxCoolingCycle))/100*this.coolingAdjuster,this.finalTemperature),this.animationPeriod=Math.ceil(this.initialAnimationPeriod*Math.sqrt(this.coolingFactor))}if(this.isTreeGrowing){if(this.growTreeIterations%10==0)if(this.prunedNodesAll.length>0){this.graphManager.updateBounds(),this.updateGrid(),this.growTree(this.prunedNodesAll),this.graphManager.resetAllNodesToApplyGravitation();var T=new Set(this.getAllNodes()),E=this.nodesWithGravity.filter(function(_){return T.has(_)});this.graphManager.setAllNodesToApplyGravitation(E),this.graphManager.updateBounds(),this.updateGrid(),this.coolingFactor=f.DEFAULT_COOLING_FACTOR_INCREMENTAL}else this.isTreeGrowing=!1,this.isGrowthFinished=!0;this.growTreeIterations++}if(this.isGrowthFinished){if(this.isConverged())return!0;this.afterGrowthIterations%10==0&&(this.graphManager.updateBounds(),this.updateGrid()),this.coolingFactor=f.DEFAULT_COOLING_FACTOR_INCREMENTAL*((100-this.afterGrowthIterations)/100),this.afterGrowthIterations++}var A=!this.isTreeGrowing&&!this.isGrowthFinished,S=this.growTreeIterations%10==1&&this.isTreeGrowing||this.afterGrowthIterations%10==1&&this.isGrowthFinished;return this.totalDisplacement=0,this.graphManager.updateBounds(),this.calcSpringForces(),this.calcRepulsionForces(A,S),this.calcGravitationalForces(),this.moveNodes(),this.animate(),!1},w.prototype.getPositionsData=function(){for(var T=this.graphManager.getAllNodes(),E={},A=0;A1){var k;for(k=0;kS&&(S=Math.floor(D.y)),I=Math.floor(D.x+h.DEFAULT_COMPONENT_SEPERATION)}this.transform(new m(d.WORLD_CENTER_X-D.x/2,d.WORLD_CENTER_Y-D.y/2))},w.radialLayout=function(T,E,A){var S=Math.max(this.maxDiagonalInTree(T),h.DEFAULT_RADIAL_SEPARATION);w.branchRadialLayout(E,null,0,359,0,S);var _=x.calculateBounds(T),I=new b;I.setDeviceOrgX(_.getMinX()),I.setDeviceOrgY(_.getMinY()),I.setWorldOrgX(A.x),I.setWorldOrgY(A.y);for(var D=0;D1;){var Q=H[0];H.splice(0,1);var j=B.indexOf(Q);j>=0&&B.splice(j,1),z--,F--}E!=null?$=(B.indexOf(H[0])+1)%z:$=0;for(var ie=Math.abs(S-A)/F,ne=$;P!=F;ne=++ne%z){var le=B[ne].getOtherEnd(T);if(le!=E){var he=(A+P*ie)%360,K=(he+ie)%360;w.branchRadialLayout(le,T,he,K,_+I,I),P++}}},w.maxDiagonalInTree=function(T){for(var E=y.MIN_VALUE,A=0;AE&&(E=_)}return E},w.prototype.calcRepulsionRange=function(){return 2*(this.level+1)*this.idealEdgeLength},w.prototype.groupZeroDegreeMembers=function(){var T=this,E={};this.memberGroups={},this.idToDummyNode={};for(var A=[],S=this.graphManager.getAllNodes(),_=0;_"u"&&(E[k]=[]),E[k]=E[k].concat(I)}Object.keys(E).forEach(function(L){if(E[L].length>1){var R="DummyCompound_"+L;T.memberGroups[R]=E[L];var O=E[L][0].getParent(),M=new l(T.graphManager);M.id=R,M.paddingLeft=O.paddingLeft||0,M.paddingRight=O.paddingRight||0,M.paddingBottom=O.paddingBottom||0,M.paddingTop=O.paddingTop||0,T.idToDummyNode[R]=M;var B=T.getGraphManager().add(T.newGraph(),M),F=O.getChild();F.add(M);for(var P=0;P=0;T--){var E=this.compoundOrder[T],A=E.id,S=E.paddingLeft,_=E.paddingTop;this.adjustLocations(this.tiledMemberPack[A],E.rect.x,E.rect.y,S,_)}},w.prototype.repopulateZeroDegreeMembers=function(){var T=this,E=this.tiledZeroDegreePack;Object.keys(E).forEach(function(A){var S=T.idToDummyNode[A],_=S.paddingLeft,I=S.paddingTop;T.adjustLocations(E[A],S.rect.x,S.rect.y,_,I)})},w.prototype.getToBeTiled=function(T){var E=T.id;if(this.toBeTiled[E]!=null)return this.toBeTiled[E];var A=T.getChild();if(A==null)return this.toBeTiled[E]=!1,!1;for(var S=A.getNodes(),_=0;_0)return this.toBeTiled[E]=!1,!1;if(I.getChild()==null){this.toBeTiled[I.id]=!1;continue}if(!this.getToBeTiled(I))return this.toBeTiled[E]=!1,!1}return this.toBeTiled[E]=!0,!0},w.prototype.getNodeDegree=function(T){for(var E=T.id,A=T.getEdges(),S=0,_=0;_L&&(L=O.rect.height)}A+=L+T.verticalPadding}},w.prototype.tileCompoundMembers=function(T,E){var A=this;this.tiledMemberPack=[],Object.keys(T).forEach(function(S){var _=E[S];A.tiledMemberPack[S]=A.tileNodes(T[S],_.paddingLeft+_.paddingRight),_.rect.width=A.tiledMemberPack[S].width,_.rect.height=A.tiledMemberPack[S].height})},w.prototype.tileNodes=function(T,E){var A=h.TILING_PADDING_VERTICAL,S=h.TILING_PADDING_HORIZONTAL,_={rows:[],rowWidth:[],rowHeight:[],width:0,height:E,verticalPadding:A,horizontalPadding:S};T.sort(function(k,L){return k.rect.width*k.rect.height>L.rect.width*L.rect.height?-1:k.rect.width*k.rect.height0&&(D+=T.horizontalPadding),T.rowWidth[A]=D,T.width0&&(k+=T.verticalPadding);var L=0;k>T.rowHeight[A]&&(L=T.rowHeight[A],T.rowHeight[A]=k,L=T.rowHeight[A]-L),T.height+=L,T.rows[A].push(E)},w.prototype.getShortestRowIndex=function(T){for(var E=-1,A=Number.MAX_VALUE,S=0;SA&&(E=S,A=T.rowWidth[S]);return E},w.prototype.canAddHorizontal=function(T,E,A){var S=this.getShortestRowIndex(T);if(S<0)return!0;var _=T.rowWidth[S];if(_+T.horizontalPadding+E<=T.width)return!0;var I=0;T.rowHeight[S]0&&(I=A+T.verticalPadding-T.rowHeight[S]);var D;T.width-_>=E+T.horizontalPadding?D=(T.height+I)/(_+E+T.horizontalPadding):D=(T.height+I)/T.width,I=A+T.verticalPadding;var k;return T.widthI&&E!=A){S.splice(-1,1),T.rows[A].push(_),T.rowWidth[E]=T.rowWidth[E]-I,T.rowWidth[A]=T.rowWidth[A]+I,T.width=T.rowWidth[instance.getLongestRowIndex(T)];for(var D=Number.MIN_VALUE,k=0;kD&&(D=S[k].height);E>0&&(D+=T.verticalPadding);var L=T.rowHeight[E]+T.rowHeight[A];T.rowHeight[E]=D,T.rowHeight[A]<_.height+T.verticalPadding&&(T.rowHeight[A]=_.height+T.verticalPadding);var R=T.rowHeight[E]+T.rowHeight[A];T.height+=R-L,this.shiftToLastRow(T)}},w.prototype.tilingPreLayout=function(){h.TILE&&(this.groupZeroDegreeMembers(),this.clearCompounds(),this.clearZeroDegreeMembers())},w.prototype.tilingPostLayout=function(){h.TILE&&(this.repopulateZeroDegreeMembers(),this.repopulateCompounds())},w.prototype.reduceTrees=function(){for(var T=[],E=!0,A;E;){var S=this.graphManager.getAllNodes(),_=[];E=!1;for(var I=0;I0)for(var F=_;F<=I;F++)B[0]+=this.grid[F][D-1].length+this.grid[F][D].length-1;if(I0)for(var F=D;F<=k;F++)B[3]+=this.grid[_-1][F].length+this.grid[_][F].length-1;for(var P=y.MAX_VALUE,z,$,H=0;H{"use strict";o(function(e,r){typeof f4=="object"&&typeof _B=="object"?_B.exports=r(AB()):typeof define=="function"&&define.amd?define(["cose-base"],r):typeof f4=="object"?f4.cytoscapeCoseBilkent=r(AB()):e.cytoscapeCoseBilkent=r(e.coseBase)},"webpackUniversalModuleDefinition")(f4,function(t){return function(e){var r={};function n(i){if(r[i])return r[i].exports;var a=r[i]={i,l:!1,exports:{}};return e[i].call(a.exports,a,a.exports,n),a.l=!0,a.exports}return o(n,"__webpack_require__"),n.m=e,n.c=r,n.i=function(i){return i},n.d=function(i,a,s){n.o(i,a)||Object.defineProperty(i,a,{configurable:!1,enumerable:!0,get:s})},n.n=function(i){var a=i&&i.__esModule?o(function(){return i.default},"getDefault"):o(function(){return i},"getModuleExports");return n.d(a,"a",a),a},n.o=function(i,a){return Object.prototype.hasOwnProperty.call(i,a)},n.p="",n(n.s=1)}([function(e,r){e.exports=t},function(e,r,n){"use strict";var i=n(0).layoutBase.LayoutConstants,a=n(0).layoutBase.FDLayoutConstants,s=n(0).CoSEConstants,l=n(0).CoSELayout,u=n(0).CoSENode,h=n(0).layoutBase.PointD,f=n(0).layoutBase.DimensionD,d={ready:o(function(){},"ready"),stop:o(function(){},"stop"),quality:"default",nodeDimensionsIncludeLabels:!1,refresh:30,fit:!0,padding:10,randomize:!0,nodeRepulsion:4500,idealEdgeLength:50,edgeElasticity:.45,nestingFactor:.1,gravity:.25,numIter:2500,tile:!0,animate:"end",animationDuration:500,tilingPaddingVertical:10,tilingPaddingHorizontal:10,gravityRangeCompound:1.5,gravityCompound:1,gravityRange:3.8,initialEnergyOnIncremental:.5};function p(v,x){var b={};for(var w in v)b[w]=v[w];for(var w in x)b[w]=x[w];return b}o(p,"extend");function m(v){this.options=p(d,v),g(this.options)}o(m,"_CoSELayout");var g=o(function(x){x.nodeRepulsion!=null&&(s.DEFAULT_REPULSION_STRENGTH=a.DEFAULT_REPULSION_STRENGTH=x.nodeRepulsion),x.idealEdgeLength!=null&&(s.DEFAULT_EDGE_LENGTH=a.DEFAULT_EDGE_LENGTH=x.idealEdgeLength),x.edgeElasticity!=null&&(s.DEFAULT_SPRING_STRENGTH=a.DEFAULT_SPRING_STRENGTH=x.edgeElasticity),x.nestingFactor!=null&&(s.PER_LEVEL_IDEAL_EDGE_LENGTH_FACTOR=a.PER_LEVEL_IDEAL_EDGE_LENGTH_FACTOR=x.nestingFactor),x.gravity!=null&&(s.DEFAULT_GRAVITY_STRENGTH=a.DEFAULT_GRAVITY_STRENGTH=x.gravity),x.numIter!=null&&(s.MAX_ITERATIONS=a.MAX_ITERATIONS=x.numIter),x.gravityRange!=null&&(s.DEFAULT_GRAVITY_RANGE_FACTOR=a.DEFAULT_GRAVITY_RANGE_FACTOR=x.gravityRange),x.gravityCompound!=null&&(s.DEFAULT_COMPOUND_GRAVITY_STRENGTH=a.DEFAULT_COMPOUND_GRAVITY_STRENGTH=x.gravityCompound),x.gravityRangeCompound!=null&&(s.DEFAULT_COMPOUND_GRAVITY_RANGE_FACTOR=a.DEFAULT_COMPOUND_GRAVITY_RANGE_FACTOR=x.gravityRangeCompound),x.initialEnergyOnIncremental!=null&&(s.DEFAULT_COOLING_FACTOR_INCREMENTAL=a.DEFAULT_COOLING_FACTOR_INCREMENTAL=x.initialEnergyOnIncremental),x.quality=="draft"?i.QUALITY=0:x.quality=="proof"?i.QUALITY=2:i.QUALITY=1,s.NODE_DIMENSIONS_INCLUDE_LABELS=a.NODE_DIMENSIONS_INCLUDE_LABELS=i.NODE_DIMENSIONS_INCLUDE_LABELS=x.nodeDimensionsIncludeLabels,s.DEFAULT_INCREMENTAL=a.DEFAULT_INCREMENTAL=i.DEFAULT_INCREMENTAL=!x.randomize,s.ANIMATE=a.ANIMATE=i.ANIMATE=x.animate,s.TILE=x.tile,s.TILING_PADDING_VERTICAL=typeof x.tilingPaddingVertical=="function"?x.tilingPaddingVertical.call():x.tilingPaddingVertical,s.TILING_PADDING_HORIZONTAL=typeof x.tilingPaddingHorizontal=="function"?x.tilingPaddingHorizontal.call():x.tilingPaddingHorizontal},"getUserOptions");m.prototype.run=function(){var v,x,b=this.options,w=this.idToLNode={},C=this.layout=new l,T=this;T.stopped=!1,this.cy=this.options.cy,this.cy.trigger({type:"layoutstart",layout:this});var E=C.newGraphManager();this.gm=E;var A=this.options.eles.nodes(),S=this.options.eles.edges();this.root=E.addRoot(),this.processChildrenList(this.root,this.getTopMostNodes(A),C);for(var _=0;_0){var k;k=b.getGraphManager().add(b.newGraph(),A),this.processChildrenList(k,E,b)}}},m.prototype.stop=function(){return this.stopped=!0,this};var y=o(function(x){x("layout","cose-bilkent",m)},"register");typeof cytoscape<"u"&&y(cytoscape),e.exports=y}])})});function JZe(t,e,r,n,i){return t.insert("polygon",":first-child").attr("points",n.map(function(a){return a.x+","+a.y}).join(" ")).attr("transform","translate("+(i.width-e)/2+", "+r+")")}var YZe,XZe,jZe,KZe,QZe,ZZe,eJe,tJe,Vge,Uge,Hge=N(()=>{"use strict";to();ir();YZe=12,XZe=o(function(t,e,r,n){e.append("path").attr("id","node-"+r.id).attr("class","node-bkg node-"+t.type2Str(r.type)).attr("d",`M0 ${r.height-5} v${-r.height+2*5} q0,-5 5,-5 h${r.width-2*5} q5,0 5,5 v${r.height-5} H0 Z`),e.append("line").attr("class","node-line-"+n).attr("x1",0).attr("y1",r.height).attr("x2",r.width).attr("y2",r.height)},"defaultBkg"),jZe=o(function(t,e,r){e.append("rect").attr("id","node-"+r.id).attr("class","node-bkg node-"+t.type2Str(r.type)).attr("height",r.height).attr("width",r.width)},"rectBkg"),KZe=o(function(t,e,r){let n=r.width,i=r.height,a=.15*n,s=.25*n,l=.35*n,u=.2*n;e.append("path").attr("id","node-"+r.id).attr("class","node-bkg node-"+t.type2Str(r.type)).attr("d",`M0 0 a${a},${a} 0 0,1 ${n*.25},${-1*n*.1} + a${l},${l} 1 0,1 ${n*.4},${-1*n*.1} + a${s},${s} 1 0,1 ${n*.35},${1*n*.2} + + a${a},${a} 1 0,1 ${n*.15},${1*i*.35} + a${u},${u} 1 0,1 ${-1*n*.15},${1*i*.65} + + a${s},${a} 1 0,1 ${-1*n*.25},${n*.15} + a${l},${l} 1 0,1 ${-1*n*.5},0 + a${a},${a} 1 0,1 ${-1*n*.25},${-1*n*.15} + + a${a},${a} 1 0,1 ${-1*n*.1},${-1*i*.35} + a${u},${u} 1 0,1 ${n*.1},${-1*i*.65} + + H0 V0 Z`)},"cloudBkg"),QZe=o(function(t,e,r){let n=r.width,i=r.height,a=.15*n;e.append("path").attr("id","node-"+r.id).attr("class","node-bkg node-"+t.type2Str(r.type)).attr("d",`M0 0 a${a},${a} 1 0,0 ${n*.25},${-1*i*.1} + a${a},${a} 1 0,0 ${n*.25},0 + a${a},${a} 1 0,0 ${n*.25},0 + a${a},${a} 1 0,0 ${n*.25},${1*i*.1} + + a${a},${a} 1 0,0 ${n*.15},${1*i*.33} + a${a*.8},${a*.8} 1 0,0 0,${1*i*.34} + a${a},${a} 1 0,0 ${-1*n*.15},${1*i*.33} + + a${a},${a} 1 0,0 ${-1*n*.25},${i*.15} + a${a},${a} 1 0,0 ${-1*n*.25},0 + a${a},${a} 1 0,0 ${-1*n*.25},0 + a${a},${a} 1 0,0 ${-1*n*.25},${-1*i*.15} + + a${a},${a} 1 0,0 ${-1*n*.1},${-1*i*.33} + a${a*.8},${a*.8} 1 0,0 0,${-1*i*.34} + a${a},${a} 1 0,0 ${n*.1},${-1*i*.33} + + H0 V0 Z`)},"bangBkg"),ZZe=o(function(t,e,r){e.append("circle").attr("id","node-"+r.id).attr("class","node-bkg node-"+t.type2Str(r.type)).attr("r",r.width/2)},"circleBkg");o(JZe,"insertPolygonShape");eJe=o(function(t,e,r){let n=r.height,a=n/4,s=r.width-r.padding+2*a,l=[{x:a,y:0},{x:s-a,y:0},{x:s,y:-n/2},{x:s-a,y:-n},{x:a,y:-n},{x:0,y:-n/2}];JZe(e,s,n,l,r)},"hexagonBkg"),tJe=o(function(t,e,r){e.append("rect").attr("id","node-"+r.id).attr("class","node-bkg node-"+t.type2Str(r.type)).attr("height",r.height).attr("rx",r.padding).attr("ry",r.padding).attr("width",r.width)},"roundedRectBkg"),Vge=o(async function(t,e,r,n,i){let a=i.htmlLabels,s=n%(YZe-1),l=e.append("g");r.section=s;let u="section-"+s;s<0&&(u+=" section-root"),l.attr("class",(r.class?r.class+" ":"")+"mindmap-node "+u);let h=l.append("g"),f=l.append("g"),d=r.descr.replace(/()/g,` +`);await Hn(f,d,{useHtmlLabels:a,width:r.width,classes:"mindmap-node-label"},i),a||f.attr("dy","1em").attr("alignment-baseline","middle").attr("dominant-baseline","middle").attr("text-anchor","middle");let p=f.node().getBBox(),[m]=Bo(i.fontSize);if(r.height=p.height+m*1.1*.5+r.padding,r.width=p.width+2*r.padding,r.icon)if(r.type===t.nodeType.CIRCLE)r.height+=50,r.width+=50,l.append("foreignObject").attr("height","50px").attr("width",r.width).attr("style","text-align: center;").append("div").attr("class","icon-container").append("i").attr("class","node-icon-"+s+" "+r.icon),f.attr("transform","translate("+r.width/2+", "+(r.height/2-1.5*r.padding)+")");else{r.width+=50;let g=r.height;r.height=Math.max(g,60);let y=Math.abs(r.height-g);l.append("foreignObject").attr("width","60px").attr("height",r.height).attr("style","text-align: center;margin-top:"+y/2+"px;").append("div").attr("class","icon-container").append("i").attr("class","node-icon-"+s+" "+r.icon),f.attr("transform","translate("+(25+r.width/2)+", "+(y/2+r.padding/2)+")")}else if(a){let g=(r.width-p.width)/2,y=(r.height-p.height)/2;f.attr("transform","translate("+g+", "+y+")")}else{let g=r.width/2,y=r.padding/2;f.attr("transform","translate("+g+", "+y+")")}switch(r.type){case t.nodeType.DEFAULT:XZe(t,h,r,s);break;case t.nodeType.ROUNDED_RECT:tJe(t,h,r,s);break;case t.nodeType.RECT:jZe(t,h,r,s);break;case t.nodeType.CIRCLE:h.attr("transform","translate("+r.width/2+", "+ +r.height/2+")"),ZZe(t,h,r,s);break;case t.nodeType.CLOUD:KZe(t,h,r,s);break;case t.nodeType.BANG:QZe(t,h,r,s);break;case t.nodeType.HEXAGON:eJe(t,h,r,s);break}return t.setElementForId(r.id,l),r.height},"drawNode"),Uge=o(function(t,e){let r=t.getElementById(e.id),n=e.x||0,i=e.y||0;r.attr("transform","translate("+n+","+i+")")},"positionNode")});async function qge(t,e,r,n,i){await Vge(t,e,r,n,i),r.children&&await Promise.all(r.children.map((a,s)=>qge(t,e,a,n<0?s:n,i)))}function rJe(t,e){e.edges().map((r,n)=>{let i=r.data();if(r[0]._private.bodyBounds){let a=r[0]._private.rscratch;Y.trace("Edge: ",n,i),t.insert("path").attr("d",`M ${a.startX},${a.startY} L ${a.midX},${a.midY} L${a.endX},${a.endY} `).attr("class","edge section-edge-"+i.section+" edge-depth-"+i.depth)}})}function Yge(t,e,r,n){e.add({group:"nodes",data:{id:t.id.toString(),labelText:t.descr,height:t.height,width:t.width,level:n,nodeId:t.id,padding:t.padding,type:t.type},position:{x:t.x,y:t.y}}),t.children&&t.children.forEach(i=>{Yge(i,e,r,n+1),e.add({group:"edges",data:{id:`${t.id}_${i.id}`,source:t.id,target:i.id,depth:n,section:i.section}})})}function nJe(t,e){return new Promise(r=>{let n=Ge("body").append("div").attr("id","cy").attr("style","display:none"),i=rl({container:document.getElementById("cy"),style:[{selector:"edge",style:{"curve-style":"bezier"}}]});n.remove(),Yge(t,i,e,0),i.nodes().forEach(function(a){a.layoutDimensions=()=>{let s=a.data();return{w:s.width,h:s.height}}}),i.layout({name:"cose-bilkent",quality:"proof",styleEnabled:!1,animate:!1}).run(),i.ready(a=>{Y.info("Ready",a),r(i)})})}function iJe(t,e){e.nodes().map((r,n)=>{let i=r.data();i.x=r.position().x,i.y=r.position().y,Uge(t,i);let a=t.getElementById(i.nodeId);Y.info("Id:",n,"Position: (",r.position().x,", ",r.position().y,")",i),a.attr("transform",`translate(${r.position().x-i.width/2}, ${r.position().y-i.height/2})`),a.attr("attr",`apa-${n})`)})}var Wge,aJe,Xge,jge=N(()=>{"use strict";kB();Wge=Sa(Gge(),1);dr();zt();vt();Vc();Ei();Hge();Ya();rl.use(Wge.default);o(qge,"drawNodes");o(rJe,"drawEdges");o(Yge,"addNodes");o(nJe,"layoutMindmap");o(iJe,"positionNodes");aJe=o(async(t,e,r,n)=>{Y.debug(`Rendering mindmap diagram +`+t);let i=n.db,a=i.getMindmap();if(!a)return;let s=me();s.htmlLabels=!1;let l=sa(e),u=l.append("g");u.attr("class","mindmap-edges");let h=l.append("g");h.attr("class","mindmap-nodes"),await qge(i,h,a,-1,s);let f=await nJe(a,s);rJe(u,f),iJe(i,f),Ao(void 0,l,s.mindmap?.padding??or.mindmap.padding,s.mindmap?.useMaxWidth??or.mindmap.useMaxWidth)},"draw"),Xge={draw:aJe}});var sJe,oJe,Kge,Qge=N(()=>{"use strict";Ys();sJe=o(t=>{let e="";for(let r=0;r` + .edge { + stroke-width: 3; + } + ${sJe(t)} + .section-root rect, .section-root path, .section-root circle, .section-root polygon { + fill: ${t.git0}; + } + .section-root text { + fill: ${t.gitBranchLabel0}; + } + .icon-container { + height:100%; + display: flex; + justify-content: center; + align-items: center; + } + .edge { + fill: none; + } + .mindmap-node-label { + dy: 1em; + alignment-baseline: middle; + text-anchor: middle; + dominant-baseline: middle; + text-align: center; + } +`,"getStyles"),Kge=oJe});var Zge={};hr(Zge,{diagram:()=>lJe});var lJe,Jge=N(()=>{"use strict";Spe();_pe();jge();Qge();lJe={db:Ape,renderer:Xge,parser:Epe,styles:Kge}});var DB,r1e,n1e=N(()=>{"use strict";DB=function(){var t=o(function(A,S,_,I){for(_=_||{},I=A.length;I--;_[A[I]]=S);return _},"o"),e=[1,4],r=[1,13],n=[1,12],i=[1,15],a=[1,16],s=[1,20],l=[1,19],u=[6,7,8],h=[1,26],f=[1,24],d=[1,25],p=[6,7,11],m=[1,31],g=[6,7,11,24],y=[1,6,13,16,17,20,23],v=[1,35],x=[1,36],b=[1,6,7,11,13,16,17,20,23],w=[1,38],C={trace:o(function(){},"trace"),yy:{},symbols_:{error:2,start:3,mindMap:4,spaceLines:5,SPACELINE:6,NL:7,KANBAN:8,document:9,stop:10,EOF:11,statement:12,SPACELIST:13,node:14,shapeData:15,ICON:16,CLASS:17,nodeWithId:18,nodeWithoutId:19,NODE_DSTART:20,NODE_DESCR:21,NODE_DEND:22,NODE_ID:23,SHAPE_DATA:24,$accept:0,$end:1},terminals_:{2:"error",6:"SPACELINE",7:"NL",8:"KANBAN",11:"EOF",13:"SPACELIST",16:"ICON",17:"CLASS",20:"NODE_DSTART",21:"NODE_DESCR",22:"NODE_DEND",23:"NODE_ID",24:"SHAPE_DATA"},productions_:[0,[3,1],[3,2],[5,1],[5,2],[5,2],[4,2],[4,3],[10,1],[10,1],[10,1],[10,2],[10,2],[9,3],[9,2],[12,3],[12,2],[12,2],[12,2],[12,1],[12,2],[12,1],[12,1],[12,1],[12,1],[14,1],[14,1],[19,3],[18,1],[18,4],[15,2],[15,1]],performAction:o(function(S,_,I,D,k,L,R){var O=L.length-1;switch(k){case 6:case 7:return D;case 8:D.getLogger().trace("Stop NL ");break;case 9:D.getLogger().trace("Stop EOF ");break;case 11:D.getLogger().trace("Stop NL2 ");break;case 12:D.getLogger().trace("Stop EOF2 ");break;case 15:D.getLogger().info("Node: ",L[O-1].id),D.addNode(L[O-2].length,L[O-1].id,L[O-1].descr,L[O-1].type,L[O]);break;case 16:D.getLogger().info("Node: ",L[O].id),D.addNode(L[O-1].length,L[O].id,L[O].descr,L[O].type);break;case 17:D.getLogger().trace("Icon: ",L[O]),D.decorateNode({icon:L[O]});break;case 18:case 23:D.decorateNode({class:L[O]});break;case 19:D.getLogger().trace("SPACELIST");break;case 20:D.getLogger().trace("Node: ",L[O-1].id),D.addNode(0,L[O-1].id,L[O-1].descr,L[O-1].type,L[O]);break;case 21:D.getLogger().trace("Node: ",L[O].id),D.addNode(0,L[O].id,L[O].descr,L[O].type);break;case 22:D.decorateNode({icon:L[O]});break;case 27:D.getLogger().trace("node found ..",L[O-2]),this.$={id:L[O-1],descr:L[O-1],type:D.getType(L[O-2],L[O])};break;case 28:this.$={id:L[O],descr:L[O],type:0};break;case 29:D.getLogger().trace("node found ..",L[O-3]),this.$={id:L[O-3],descr:L[O-1],type:D.getType(L[O-2],L[O])};break;case 30:this.$=L[O-1]+L[O];break;case 31:this.$=L[O];break}},"anonymous"),table:[{3:1,4:2,5:3,6:[1,5],8:e},{1:[3]},{1:[2,1]},{4:6,6:[1,7],7:[1,8],8:e},{6:r,7:[1,10],9:9,12:11,13:n,14:14,16:i,17:a,18:17,19:18,20:s,23:l},t(u,[2,3]),{1:[2,2]},t(u,[2,4]),t(u,[2,5]),{1:[2,6],6:r,12:21,13:n,14:14,16:i,17:a,18:17,19:18,20:s,23:l},{6:r,9:22,12:11,13:n,14:14,16:i,17:a,18:17,19:18,20:s,23:l},{6:h,7:f,10:23,11:d},t(p,[2,24],{18:17,19:18,14:27,16:[1,28],17:[1,29],20:s,23:l}),t(p,[2,19]),t(p,[2,21],{15:30,24:m}),t(p,[2,22]),t(p,[2,23]),t(g,[2,25]),t(g,[2,26]),t(g,[2,28],{20:[1,32]}),{21:[1,33]},{6:h,7:f,10:34,11:d},{1:[2,7],6:r,12:21,13:n,14:14,16:i,17:a,18:17,19:18,20:s,23:l},t(y,[2,14],{7:v,11:x}),t(b,[2,8]),t(b,[2,9]),t(b,[2,10]),t(p,[2,16],{15:37,24:m}),t(p,[2,17]),t(p,[2,18]),t(p,[2,20],{24:w}),t(g,[2,31]),{21:[1,39]},{22:[1,40]},t(y,[2,13],{7:v,11:x}),t(b,[2,11]),t(b,[2,12]),t(p,[2,15],{24:w}),t(g,[2,30]),{22:[1,41]},t(g,[2,27]),t(g,[2,29])],defaultActions:{2:[2,1],6:[2,2]},parseError:o(function(S,_){if(_.recoverable)this.trace(S);else{var I=new Error(S);throw I.hash=_,I}},"parseError"),parse:o(function(S){var _=this,I=[0],D=[],k=[null],L=[],R=this.table,O="",M=0,B=0,F=0,P=2,z=1,$=L.slice.call(arguments,1),H=Object.create(this.lexer),Q={yy:{}};for(var j in this.yy)Object.prototype.hasOwnProperty.call(this.yy,j)&&(Q.yy[j]=this.yy[j]);H.setInput(S,Q.yy),Q.yy.lexer=H,Q.yy.parser=this,typeof H.yylloc>"u"&&(H.yylloc={});var ie=H.yylloc;L.push(ie);var ne=H.options&&H.options.ranges;typeof Q.yy.parseError=="function"?this.parseError=Q.yy.parseError:this.parseError=Object.getPrototypeOf(this).parseError;function le(ze){I.length=I.length-2*ze,k.length=k.length-ze,L.length=L.length-ze}o(le,"popStack");function he(){var ze;return ze=D.pop()||H.lex()||z,typeof ze!="number"&&(ze instanceof Array&&(D=ze,ze=D.pop()),ze=_.symbols_[ze]||ze),ze}o(he,"lex");for(var K,X,te,J,se,ue,Z={},Se,ce,ae,Oe;;){if(te=I[I.length-1],this.defaultActions[te]?J=this.defaultActions[te]:((K===null||typeof K>"u")&&(K=he()),J=R[te]&&R[te][K]),typeof J>"u"||!J.length||!J[0]){var ge="";Oe=[];for(Se in R[te])this.terminals_[Se]&&Se>P&&Oe.push("'"+this.terminals_[Se]+"'");H.showPosition?ge="Parse error on line "+(M+1)+`: +`+H.showPosition()+` +Expecting `+Oe.join(", ")+", got '"+(this.terminals_[K]||K)+"'":ge="Parse error on line "+(M+1)+": Unexpected "+(K==z?"end of input":"'"+(this.terminals_[K]||K)+"'"),this.parseError(ge,{text:H.match,token:this.terminals_[K]||K,line:H.yylineno,loc:ie,expected:Oe})}if(J[0]instanceof Array&&J.length>1)throw new Error("Parse Error: multiple actions possible at state: "+te+", token: "+K);switch(J[0]){case 1:I.push(K),k.push(H.yytext),L.push(H.yylloc),I.push(J[1]),K=null,X?(K=X,X=null):(B=H.yyleng,O=H.yytext,M=H.yylineno,ie=H.yylloc,F>0&&F--);break;case 2:if(ce=this.productions_[J[1]][1],Z.$=k[k.length-ce],Z._$={first_line:L[L.length-(ce||1)].first_line,last_line:L[L.length-1].last_line,first_column:L[L.length-(ce||1)].first_column,last_column:L[L.length-1].last_column},ne&&(Z._$.range=[L[L.length-(ce||1)].range[0],L[L.length-1].range[1]]),ue=this.performAction.apply(Z,[O,B,M,Q.yy,J[1],k,L].concat($)),typeof ue<"u")return ue;ce&&(I=I.slice(0,-1*ce*2),k=k.slice(0,-1*ce),L=L.slice(0,-1*ce)),I.push(this.productions_[J[1]][0]),k.push(Z.$),L.push(Z._$),ae=R[I[I.length-2]][I[I.length-1]],I.push(ae);break;case 3:return!0}}return!0},"parse")},T=function(){var A={EOF:1,parseError:o(function(_,I){if(this.yy.parser)this.yy.parser.parseError(_,I);else throw new Error(_)},"parseError"),setInput:o(function(S,_){return this.yy=_||this.yy||{},this._input=S,this._more=this._backtrack=this.done=!1,this.yylineno=this.yyleng=0,this.yytext=this.matched=this.match="",this.conditionStack=["INITIAL"],this.yylloc={first_line:1,first_column:0,last_line:1,last_column:0},this.options.ranges&&(this.yylloc.range=[0,0]),this.offset=0,this},"setInput"),input:o(function(){var S=this._input[0];this.yytext+=S,this.yyleng++,this.offset++,this.match+=S,this.matched+=S;var _=S.match(/(?:\r\n?|\n).*/g);return _?(this.yylineno++,this.yylloc.last_line++):this.yylloc.last_column++,this.options.ranges&&this.yylloc.range[1]++,this._input=this._input.slice(1),S},"input"),unput:o(function(S){var _=S.length,I=S.split(/(?:\r\n?|\n)/g);this._input=S+this._input,this.yytext=this.yytext.substr(0,this.yytext.length-_),this.offset-=_;var D=this.match.split(/(?:\r\n?|\n)/g);this.match=this.match.substr(0,this.match.length-1),this.matched=this.matched.substr(0,this.matched.length-1),I.length-1&&(this.yylineno-=I.length-1);var k=this.yylloc.range;return this.yylloc={first_line:this.yylloc.first_line,last_line:this.yylineno+1,first_column:this.yylloc.first_column,last_column:I?(I.length===D.length?this.yylloc.first_column:0)+D[D.length-I.length].length-I[0].length:this.yylloc.first_column-_},this.options.ranges&&(this.yylloc.range=[k[0],k[0]+this.yyleng-_]),this.yyleng=this.yytext.length,this},"unput"),more:o(function(){return this._more=!0,this},"more"),reject:o(function(){if(this.options.backtrack_lexer)this._backtrack=!0;else return this.parseError("Lexical error on line "+(this.yylineno+1)+`. You can only invoke reject() in the lexer when the lexer is of the backtracking persuasion (options.backtrack_lexer = true). +`+this.showPosition(),{text:"",token:null,line:this.yylineno});return this},"reject"),less:o(function(S){this.unput(this.match.slice(S))},"less"),pastInput:o(function(){var S=this.matched.substr(0,this.matched.length-this.match.length);return(S.length>20?"...":"")+S.substr(-20).replace(/\n/g,"")},"pastInput"),upcomingInput:o(function(){var S=this.match;return S.length<20&&(S+=this._input.substr(0,20-S.length)),(S.substr(0,20)+(S.length>20?"...":"")).replace(/\n/g,"")},"upcomingInput"),showPosition:o(function(){var S=this.pastInput(),_=new Array(S.length+1).join("-");return S+this.upcomingInput()+` +`+_+"^"},"showPosition"),test_match:o(function(S,_){var I,D,k;if(this.options.backtrack_lexer&&(k={yylineno:this.yylineno,yylloc:{first_line:this.yylloc.first_line,last_line:this.last_line,first_column:this.yylloc.first_column,last_column:this.yylloc.last_column},yytext:this.yytext,match:this.match,matches:this.matches,matched:this.matched,yyleng:this.yyleng,offset:this.offset,_more:this._more,_input:this._input,yy:this.yy,conditionStack:this.conditionStack.slice(0),done:this.done},this.options.ranges&&(k.yylloc.range=this.yylloc.range.slice(0))),D=S[0].match(/(?:\r\n?|\n).*/g),D&&(this.yylineno+=D.length),this.yylloc={first_line:this.yylloc.last_line,last_line:this.yylineno+1,first_column:this.yylloc.last_column,last_column:D?D[D.length-1].length-D[D.length-1].match(/\r?\n?/)[0].length:this.yylloc.last_column+S[0].length},this.yytext+=S[0],this.match+=S[0],this.matches=S,this.yyleng=this.yytext.length,this.options.ranges&&(this.yylloc.range=[this.offset,this.offset+=this.yyleng]),this._more=!1,this._backtrack=!1,this._input=this._input.slice(S[0].length),this.matched+=S[0],I=this.performAction.call(this,this.yy,this,_,this.conditionStack[this.conditionStack.length-1]),this.done&&this._input&&(this.done=!1),I)return I;if(this._backtrack){for(var L in k)this[L]=k[L];return!1}return!1},"test_match"),next:o(function(){if(this.done)return this.EOF;this._input||(this.done=!0);var S,_,I,D;this._more||(this.yytext="",this.match="");for(var k=this._currentRules(),L=0;L_[0].length)){if(_=I,D=L,this.options.backtrack_lexer){if(S=this.test_match(I,k[L]),S!==!1)return S;if(this._backtrack){_=!1;continue}else return!1}else if(!this.options.flex)break}return _?(S=this.test_match(_,k[D]),S!==!1?S:!1):this._input===""?this.EOF:this.parseError("Lexical error on line "+(this.yylineno+1)+`. Unrecognized text. +`+this.showPosition(),{text:"",token:null,line:this.yylineno})},"next"),lex:o(function(){var _=this.next();return _||this.lex()},"lex"),begin:o(function(_){this.conditionStack.push(_)},"begin"),popState:o(function(){var _=this.conditionStack.length-1;return _>0?this.conditionStack.pop():this.conditionStack[0]},"popState"),_currentRules:o(function(){return this.conditionStack.length&&this.conditionStack[this.conditionStack.length-1]?this.conditions[this.conditionStack[this.conditionStack.length-1]].rules:this.conditions.INITIAL.rules},"_currentRules"),topState:o(function(_){return _=this.conditionStack.length-1-Math.abs(_||0),_>=0?this.conditionStack[_]:"INITIAL"},"topState"),pushState:o(function(_){this.begin(_)},"pushState"),stateStackSize:o(function(){return this.conditionStack.length},"stateStackSize"),options:{"case-insensitive":!0},performAction:o(function(_,I,D,k){var L=k;switch(D){case 0:return this.pushState("shapeData"),I.yytext="",24;break;case 1:return this.pushState("shapeDataStr"),24;break;case 2:return this.popState(),24;break;case 3:let R=/\n\s*/g;return I.yytext=I.yytext.replace(R,"
    "),24;break;case 4:return 24;case 5:this.popState();break;case 6:return _.getLogger().trace("Found comment",I.yytext),6;break;case 7:return 8;case 8:this.begin("CLASS");break;case 9:return this.popState(),17;break;case 10:this.popState();break;case 11:_.getLogger().trace("Begin icon"),this.begin("ICON");break;case 12:return _.getLogger().trace("SPACELINE"),6;break;case 13:return 7;case 14:return 16;case 15:_.getLogger().trace("end icon"),this.popState();break;case 16:return _.getLogger().trace("Exploding node"),this.begin("NODE"),20;break;case 17:return _.getLogger().trace("Cloud"),this.begin("NODE"),20;break;case 18:return _.getLogger().trace("Explosion Bang"),this.begin("NODE"),20;break;case 19:return _.getLogger().trace("Cloud Bang"),this.begin("NODE"),20;break;case 20:return this.begin("NODE"),20;break;case 21:return this.begin("NODE"),20;break;case 22:return this.begin("NODE"),20;break;case 23:return this.begin("NODE"),20;break;case 24:return 13;case 25:return 23;case 26:return 11;case 27:this.begin("NSTR2");break;case 28:return"NODE_DESCR";case 29:this.popState();break;case 30:_.getLogger().trace("Starting NSTR"),this.begin("NSTR");break;case 31:return _.getLogger().trace("description:",I.yytext),"NODE_DESCR";break;case 32:this.popState();break;case 33:return this.popState(),_.getLogger().trace("node end ))"),"NODE_DEND";break;case 34:return this.popState(),_.getLogger().trace("node end )"),"NODE_DEND";break;case 35:return this.popState(),_.getLogger().trace("node end ...",I.yytext),"NODE_DEND";break;case 36:return this.popState(),_.getLogger().trace("node end (("),"NODE_DEND";break;case 37:return this.popState(),_.getLogger().trace("node end (-"),"NODE_DEND";break;case 38:return this.popState(),_.getLogger().trace("node end (-"),"NODE_DEND";break;case 39:return this.popState(),_.getLogger().trace("node end (("),"NODE_DEND";break;case 40:return this.popState(),_.getLogger().trace("node end (("),"NODE_DEND";break;case 41:return _.getLogger().trace("Long description:",I.yytext),21;break;case 42:return _.getLogger().trace("Long description:",I.yytext),21;break}},"anonymous"),rules:[/^(?:@\{)/i,/^(?:["])/i,/^(?:["])/i,/^(?:[^\"]+)/i,/^(?:[^}^"]+)/i,/^(?:\})/i,/^(?:\s*%%.*)/i,/^(?:kanban\b)/i,/^(?::::)/i,/^(?:.+)/i,/^(?:\n)/i,/^(?:::icon\()/i,/^(?:[\s]+[\n])/i,/^(?:[\n]+)/i,/^(?:[^\)]+)/i,/^(?:\))/i,/^(?:-\))/i,/^(?:\(-)/i,/^(?:\)\))/i,/^(?:\))/i,/^(?:\(\()/i,/^(?:\{\{)/i,/^(?:\()/i,/^(?:\[)/i,/^(?:[\s]+)/i,/^(?:[^\(\[\n\)\{\}@]+)/i,/^(?:$)/i,/^(?:["][`])/i,/^(?:[^`"]+)/i,/^(?:[`]["])/i,/^(?:["])/i,/^(?:[^"]+)/i,/^(?:["])/i,/^(?:[\)]\))/i,/^(?:[\)])/i,/^(?:[\]])/i,/^(?:\}\})/i,/^(?:\(-)/i,/^(?:-\))/i,/^(?:\(\()/i,/^(?:\()/i,/^(?:[^\)\]\(\}]+)/i,/^(?:.+(?!\(\())/i],conditions:{shapeDataEndBracket:{rules:[],inclusive:!1},shapeDataStr:{rules:[2,3],inclusive:!1},shapeData:{rules:[1,4,5],inclusive:!1},CLASS:{rules:[9,10],inclusive:!1},ICON:{rules:[14,15],inclusive:!1},NSTR2:{rules:[28,29],inclusive:!1},NSTR:{rules:[31,32],inclusive:!1},NODE:{rules:[27,30,33,34,35,36,37,38,39,40,41,42],inclusive:!1},INITIAL:{rules:[0,6,7,8,11,12,13,16,17,18,19,20,21,22,23,24,25,26],inclusive:!0}}};return A}();C.lexer=T;function E(){this.yy={}}return o(E,"Parser"),E.prototype=C,C.Parser=E,new E}();DB.parser=DB;r1e=DB});var nl,RB,LB,NB,fJe,dJe,i1e,pJe,mJe,Yi,gJe,yJe,vJe,xJe,bJe,wJe,TJe,a1e,s1e=N(()=>{"use strict";zt();gr();vt();Ya();Ew();nl=[],RB=[],LB=0,NB={},fJe=o(()=>{nl=[],RB=[],LB=0,NB={}},"clear"),dJe=o(t=>{if(nl.length===0)return null;let e=nl[0].level,r=null;for(let n=nl.length-1;n>=0;n--)if(nl[n].level===e&&!r&&(r=nl[n]),nl[n].levell.parentId===i.id);for(let l of s){let u={id:l.id,parentId:i.id,label:Tr(l.label??"",n),isGroup:!1,ticket:l?.ticket,priority:l?.priority,assigned:l?.assigned,icon:l?.icon,shape:"kanbanItem",level:l.level,rx:5,ry:5,cssStyles:["text-align: left"]};e.push(u)}}return{nodes:e,edges:t,other:{},config:me()}},"getData"),mJe=o((t,e,r,n,i)=>{let a=me(),s=a.mindmap?.padding??or.mindmap.padding;switch(n){case Yi.ROUNDED_RECT:case Yi.RECT:case Yi.HEXAGON:s*=2}let l={id:Tr(e,a)||"kbn"+LB++,level:t,label:Tr(r,a),width:a.mindmap?.maxNodeWidth??or.mindmap.maxNodeWidth,padding:s,isGroup:!1};if(i!==void 0){let h;i.includes(` +`)?h=i+` +`:h=`{ +`+i+` +}`;let f=cm(h,{schema:lm});if(f.shape&&(f.shape!==f.shape.toLowerCase()||f.shape.includes("_")))throw new Error(`No such shape: ${f.shape}. Shape names should be lowercase.`);f?.shape&&f.shape==="kanbanItem"&&(l.shape=f?.shape),f?.label&&(l.label=f?.label),f?.icon&&(l.icon=f?.icon.toString()),f?.assigned&&(l.assigned=f?.assigned.toString()),f?.ticket&&(l.ticket=f?.ticket.toString()),f?.priority&&(l.priority=f?.priority)}let u=dJe(t);u?l.parentId=u.id||"kbn"+LB++:RB.push(l),nl.push(l)},"addNode"),Yi={DEFAULT:0,NO_BORDER:0,ROUNDED_RECT:1,RECT:2,CIRCLE:3,CLOUD:4,BANG:5,HEXAGON:6},gJe=o((t,e)=>{switch(Y.debug("In get type",t,e),t){case"[":return Yi.RECT;case"(":return e===")"?Yi.ROUNDED_RECT:Yi.CLOUD;case"((":return Yi.CIRCLE;case")":return Yi.CLOUD;case"))":return Yi.BANG;case"{{":return Yi.HEXAGON;default:return Yi.DEFAULT}},"getType"),yJe=o((t,e)=>{NB[t]=e},"setElementForId"),vJe=o(t=>{if(!t)return;let e=me(),r=nl[nl.length-1];t.icon&&(r.icon=Tr(t.icon,e)),t.class&&(r.cssClasses=Tr(t.class,e))},"decorateNode"),xJe=o(t=>{switch(t){case Yi.DEFAULT:return"no-border";case Yi.RECT:return"rect";case Yi.ROUNDED_RECT:return"rounded-rect";case Yi.CIRCLE:return"circle";case Yi.CLOUD:return"cloud";case Yi.BANG:return"bang";case Yi.HEXAGON:return"hexgon";default:return"no-border"}},"type2Str"),bJe=o(()=>Y,"getLogger"),wJe=o(t=>NB[t],"getElementById"),TJe={clear:fJe,addNode:mJe,getSections:i1e,getData:pJe,nodeType:Yi,getType:gJe,setElementForId:yJe,decorateNode:vJe,type2Str:xJe,getLogger:bJe,getElementById:wJe},a1e=TJe});var kJe,o1e,l1e=N(()=>{"use strict";zt();vt();Vc();Ei();Ya();Hw();eT();kJe=o(async(t,e,r,n)=>{Y.debug(`Rendering kanban diagram +`+t);let a=n.db.getData(),s=me();s.htmlLabels=!1;let l=sa(e),u=l.append("g");u.attr("class","sections");let h=l.append("g");h.attr("class","items");let f=a.nodes.filter(v=>v.isGroup),d=0,p=10,m=[],g=25;for(let v of f){let x=s?.kanban?.sectionWidth||200;d=d+1,v.x=x*d+(d-1)*p/2,v.width=x,v.y=0,v.height=x*3,v.rx=5,v.ry=5,v.cssClasses=v.cssClasses+" section-"+d;let b=await ym(u,v);g=Math.max(g,b?.labelBBox?.height),m.push(b)}let y=0;for(let v of f){let x=m[y];y=y+1;let b=s?.kanban?.sectionWidth||200,w=-b*3/2+g,C=w,T=a.nodes.filter(S=>S.parentId===v.id);for(let S of T){if(S.isGroup)throw new Error("Groups within groups are not allowed in Kanban diagrams");S.x=v.x,S.width=b-1.5*p;let I=(await vm(h,S,{config:s})).node().getBBox();S.y=C+I.height/2,await k2(S),C=S.y+I.height/2+p/2}let E=x.cluster.select("rect"),A=Math.max(C-w+3*p,50)+(g-25);E.attr("height",A)}Ao(void 0,l,s.mindmap?.padding??or.kanban.padding,s.mindmap?.useMaxWidth??or.kanban.useMaxWidth)},"draw"),o1e={draw:kJe}});var EJe,SJe,c1e,u1e=N(()=>{"use strict";Ys();EJe=o(t=>{let e="";for(let n=0;nt.darkMode?Ot(n,i):Dt(n,i),"adjuster");for(let n=0;n` + .edge { + stroke-width: 3; + } + ${EJe(t)} + .section-root rect, .section-root path, .section-root circle, .section-root polygon { + fill: ${t.git0}; + } + .section-root text { + fill: ${t.gitBranchLabel0}; + } + .icon-container { + height:100%; + display: flex; + justify-content: center; + align-items: center; + } + .edge { + fill: none; + } + .cluster-label, .label { + color: ${t.textColor}; + fill: ${t.textColor}; + } + .kanban-label { + dy: 1em; + alignment-baseline: middle; + text-anchor: middle; + dominant-baseline: middle; + text-align: center; + } +`,"getStyles"),c1e=SJe});var h1e={};hr(h1e,{diagram:()=>CJe});var CJe,f1e=N(()=>{"use strict";n1e();s1e();l1e();u1e();CJe={db:a1e,renderer:o1e,parser:r1e,styles:c1e}});var MB,d4,m1e=N(()=>{"use strict";MB=function(){var t=o(function(l,u,h,f){for(h=h||{},f=l.length;f--;h[l[f]]=u);return h},"o"),e=[1,9],r=[1,10],n=[1,5,10,12],i={trace:o(function(){},"trace"),yy:{},symbols_:{error:2,start:3,SANKEY:4,NEWLINE:5,csv:6,opt_eof:7,record:8,csv_tail:9,EOF:10,"field[source]":11,COMMA:12,"field[target]":13,"field[value]":14,field:15,escaped:16,non_escaped:17,DQUOTE:18,ESCAPED_TEXT:19,NON_ESCAPED_TEXT:20,$accept:0,$end:1},terminals_:{2:"error",4:"SANKEY",5:"NEWLINE",10:"EOF",11:"field[source]",12:"COMMA",13:"field[target]",14:"field[value]",18:"DQUOTE",19:"ESCAPED_TEXT",20:"NON_ESCAPED_TEXT"},productions_:[0,[3,4],[6,2],[9,2],[9,0],[7,1],[7,0],[8,5],[15,1],[15,1],[16,3],[17,1]],performAction:o(function(u,h,f,d,p,m,g){var y=m.length-1;switch(p){case 7:let v=d.findOrCreateNode(m[y-4].trim().replaceAll('""','"')),x=d.findOrCreateNode(m[y-2].trim().replaceAll('""','"')),b=parseFloat(m[y].trim());d.addLink(v,x,b);break;case 8:case 9:case 11:this.$=m[y];break;case 10:this.$=m[y-1];break}},"anonymous"),table:[{3:1,4:[1,2]},{1:[3]},{5:[1,3]},{6:4,8:5,15:6,16:7,17:8,18:e,20:r},{1:[2,6],7:11,10:[1,12]},t(r,[2,4],{9:13,5:[1,14]}),{12:[1,15]},t(n,[2,8]),t(n,[2,9]),{19:[1,16]},t(n,[2,11]),{1:[2,1]},{1:[2,5]},t(r,[2,2]),{6:17,8:5,15:6,16:7,17:8,18:e,20:r},{15:18,16:7,17:8,18:e,20:r},{18:[1,19]},t(r,[2,3]),{12:[1,20]},t(n,[2,10]),{15:21,16:7,17:8,18:e,20:r},t([1,5,10],[2,7])],defaultActions:{11:[2,1],12:[2,5]},parseError:o(function(u,h){if(h.recoverable)this.trace(u);else{var f=new Error(u);throw f.hash=h,f}},"parseError"),parse:o(function(u){var h=this,f=[0],d=[],p=[null],m=[],g=this.table,y="",v=0,x=0,b=0,w=2,C=1,T=m.slice.call(arguments,1),E=Object.create(this.lexer),A={yy:{}};for(var S in this.yy)Object.prototype.hasOwnProperty.call(this.yy,S)&&(A.yy[S]=this.yy[S]);E.setInput(u,A.yy),A.yy.lexer=E,A.yy.parser=this,typeof E.yylloc>"u"&&(E.yylloc={});var _=E.yylloc;m.push(_);var I=E.options&&E.options.ranges;typeof A.yy.parseError=="function"?this.parseError=A.yy.parseError:this.parseError=Object.getPrototypeOf(this).parseError;function D(ie){f.length=f.length-2*ie,p.length=p.length-ie,m.length=m.length-ie}o(D,"popStack");function k(){var ie;return ie=d.pop()||E.lex()||C,typeof ie!="number"&&(ie instanceof Array&&(d=ie,ie=d.pop()),ie=h.symbols_[ie]||ie),ie}o(k,"lex");for(var L,R,O,M,B,F,P={},z,$,H,Q;;){if(O=f[f.length-1],this.defaultActions[O]?M=this.defaultActions[O]:((L===null||typeof L>"u")&&(L=k()),M=g[O]&&g[O][L]),typeof M>"u"||!M.length||!M[0]){var j="";Q=[];for(z in g[O])this.terminals_[z]&&z>w&&Q.push("'"+this.terminals_[z]+"'");E.showPosition?j="Parse error on line "+(v+1)+`: +`+E.showPosition()+` +Expecting `+Q.join(", ")+", got '"+(this.terminals_[L]||L)+"'":j="Parse error on line "+(v+1)+": Unexpected "+(L==C?"end of input":"'"+(this.terminals_[L]||L)+"'"),this.parseError(j,{text:E.match,token:this.terminals_[L]||L,line:E.yylineno,loc:_,expected:Q})}if(M[0]instanceof Array&&M.length>1)throw new Error("Parse Error: multiple actions possible at state: "+O+", token: "+L);switch(M[0]){case 1:f.push(L),p.push(E.yytext),m.push(E.yylloc),f.push(M[1]),L=null,R?(L=R,R=null):(x=E.yyleng,y=E.yytext,v=E.yylineno,_=E.yylloc,b>0&&b--);break;case 2:if($=this.productions_[M[1]][1],P.$=p[p.length-$],P._$={first_line:m[m.length-($||1)].first_line,last_line:m[m.length-1].last_line,first_column:m[m.length-($||1)].first_column,last_column:m[m.length-1].last_column},I&&(P._$.range=[m[m.length-($||1)].range[0],m[m.length-1].range[1]]),F=this.performAction.apply(P,[y,x,v,A.yy,M[1],p,m].concat(T)),typeof F<"u")return F;$&&(f=f.slice(0,-1*$*2),p=p.slice(0,-1*$),m=m.slice(0,-1*$)),f.push(this.productions_[M[1]][0]),p.push(P.$),m.push(P._$),H=g[f[f.length-2]][f[f.length-1]],f.push(H);break;case 3:return!0}}return!0},"parse")},a=function(){var l={EOF:1,parseError:o(function(h,f){if(this.yy.parser)this.yy.parser.parseError(h,f);else throw new Error(h)},"parseError"),setInput:o(function(u,h){return this.yy=h||this.yy||{},this._input=u,this._more=this._backtrack=this.done=!1,this.yylineno=this.yyleng=0,this.yytext=this.matched=this.match="",this.conditionStack=["INITIAL"],this.yylloc={first_line:1,first_column:0,last_line:1,last_column:0},this.options.ranges&&(this.yylloc.range=[0,0]),this.offset=0,this},"setInput"),input:o(function(){var u=this._input[0];this.yytext+=u,this.yyleng++,this.offset++,this.match+=u,this.matched+=u;var h=u.match(/(?:\r\n?|\n).*/g);return h?(this.yylineno++,this.yylloc.last_line++):this.yylloc.last_column++,this.options.ranges&&this.yylloc.range[1]++,this._input=this._input.slice(1),u},"input"),unput:o(function(u){var h=u.length,f=u.split(/(?:\r\n?|\n)/g);this._input=u+this._input,this.yytext=this.yytext.substr(0,this.yytext.length-h),this.offset-=h;var d=this.match.split(/(?:\r\n?|\n)/g);this.match=this.match.substr(0,this.match.length-1),this.matched=this.matched.substr(0,this.matched.length-1),f.length-1&&(this.yylineno-=f.length-1);var p=this.yylloc.range;return this.yylloc={first_line:this.yylloc.first_line,last_line:this.yylineno+1,first_column:this.yylloc.first_column,last_column:f?(f.length===d.length?this.yylloc.first_column:0)+d[d.length-f.length].length-f[0].length:this.yylloc.first_column-h},this.options.ranges&&(this.yylloc.range=[p[0],p[0]+this.yyleng-h]),this.yyleng=this.yytext.length,this},"unput"),more:o(function(){return this._more=!0,this},"more"),reject:o(function(){if(this.options.backtrack_lexer)this._backtrack=!0;else return this.parseError("Lexical error on line "+(this.yylineno+1)+`. You can only invoke reject() in the lexer when the lexer is of the backtracking persuasion (options.backtrack_lexer = true). +`+this.showPosition(),{text:"",token:null,line:this.yylineno});return this},"reject"),less:o(function(u){this.unput(this.match.slice(u))},"less"),pastInput:o(function(){var u=this.matched.substr(0,this.matched.length-this.match.length);return(u.length>20?"...":"")+u.substr(-20).replace(/\n/g,"")},"pastInput"),upcomingInput:o(function(){var u=this.match;return u.length<20&&(u+=this._input.substr(0,20-u.length)),(u.substr(0,20)+(u.length>20?"...":"")).replace(/\n/g,"")},"upcomingInput"),showPosition:o(function(){var u=this.pastInput(),h=new Array(u.length+1).join("-");return u+this.upcomingInput()+` +`+h+"^"},"showPosition"),test_match:o(function(u,h){var f,d,p;if(this.options.backtrack_lexer&&(p={yylineno:this.yylineno,yylloc:{first_line:this.yylloc.first_line,last_line:this.last_line,first_column:this.yylloc.first_column,last_column:this.yylloc.last_column},yytext:this.yytext,match:this.match,matches:this.matches,matched:this.matched,yyleng:this.yyleng,offset:this.offset,_more:this._more,_input:this._input,yy:this.yy,conditionStack:this.conditionStack.slice(0),done:this.done},this.options.ranges&&(p.yylloc.range=this.yylloc.range.slice(0))),d=u[0].match(/(?:\r\n?|\n).*/g),d&&(this.yylineno+=d.length),this.yylloc={first_line:this.yylloc.last_line,last_line:this.yylineno+1,first_column:this.yylloc.last_column,last_column:d?d[d.length-1].length-d[d.length-1].match(/\r?\n?/)[0].length:this.yylloc.last_column+u[0].length},this.yytext+=u[0],this.match+=u[0],this.matches=u,this.yyleng=this.yytext.length,this.options.ranges&&(this.yylloc.range=[this.offset,this.offset+=this.yyleng]),this._more=!1,this._backtrack=!1,this._input=this._input.slice(u[0].length),this.matched+=u[0],f=this.performAction.call(this,this.yy,this,h,this.conditionStack[this.conditionStack.length-1]),this.done&&this._input&&(this.done=!1),f)return f;if(this._backtrack){for(var m in p)this[m]=p[m];return!1}return!1},"test_match"),next:o(function(){if(this.done)return this.EOF;this._input||(this.done=!0);var u,h,f,d;this._more||(this.yytext="",this.match="");for(var p=this._currentRules(),m=0;mh[0].length)){if(h=f,d=m,this.options.backtrack_lexer){if(u=this.test_match(f,p[m]),u!==!1)return u;if(this._backtrack){h=!1;continue}else return!1}else if(!this.options.flex)break}return h?(u=this.test_match(h,p[d]),u!==!1?u:!1):this._input===""?this.EOF:this.parseError("Lexical error on line "+(this.yylineno+1)+`. Unrecognized text. +`+this.showPosition(),{text:"",token:null,line:this.yylineno})},"next"),lex:o(function(){var h=this.next();return h||this.lex()},"lex"),begin:o(function(h){this.conditionStack.push(h)},"begin"),popState:o(function(){var h=this.conditionStack.length-1;return h>0?this.conditionStack.pop():this.conditionStack[0]},"popState"),_currentRules:o(function(){return this.conditionStack.length&&this.conditionStack[this.conditionStack.length-1]?this.conditions[this.conditionStack[this.conditionStack.length-1]].rules:this.conditions.INITIAL.rules},"_currentRules"),topState:o(function(h){return h=this.conditionStack.length-1-Math.abs(h||0),h>=0?this.conditionStack[h]:"INITIAL"},"topState"),pushState:o(function(h){this.begin(h)},"pushState"),stateStackSize:o(function(){return this.conditionStack.length},"stateStackSize"),options:{"case-insensitive":!0},performAction:o(function(h,f,d,p){var m=p;switch(d){case 0:return this.pushState("csv"),4;break;case 1:return 10;case 2:return 5;case 3:return 12;case 4:return this.pushState("escaped_text"),18;break;case 5:return 20;case 6:return this.popState("escaped_text"),18;break;case 7:return 19}},"anonymous"),rules:[/^(?:sankey-beta\b)/i,/^(?:$)/i,/^(?:((\u000D\u000A)|(\u000A)))/i,/^(?:(\u002C))/i,/^(?:(\u0022))/i,/^(?:([\u0020-\u0021\u0023-\u002B\u002D-\u007E])*)/i,/^(?:(\u0022)(?!(\u0022)))/i,/^(?:(([\u0020-\u0021\u0023-\u002B\u002D-\u007E])|(\u002C)|(\u000D)|(\u000A)|(\u0022)(\u0022))*)/i],conditions:{csv:{rules:[1,2,3,4,5,6,7],inclusive:!1},escaped_text:{rules:[6,7],inclusive:!1},INITIAL:{rules:[0,1,2,3,4,5,6,7],inclusive:!0}}};return l}();i.lexer=a;function s(){this.yy={}}return o(s,"Parser"),s.prototype=i,i.Parser=s,new s}();MB.parser=MB;d4=MB});var XS,jS,YS,LJe,IB,RJe,OB,NJe,MJe,IJe,OJe,g1e,y1e=N(()=>{"use strict";zt();gr();mi();XS=[],jS=[],YS=new Map,LJe=o(()=>{XS=[],jS=[],YS=new Map,Ar()},"clear"),IB=class{constructor(e,r,n=0){this.source=e;this.target=r;this.value=n}static{o(this,"SankeyLink")}},RJe=o((t,e,r)=>{XS.push(new IB(t,e,r))},"addLink"),OB=class{constructor(e){this.ID=e}static{o(this,"SankeyNode")}},NJe=o(t=>{t=Ze.sanitizeText(t,me());let e=YS.get(t);return e===void 0&&(e=new OB(t),YS.set(t,e),jS.push(e)),e},"findOrCreateNode"),MJe=o(()=>jS,"getNodes"),IJe=o(()=>XS,"getLinks"),OJe=o(()=>({nodes:jS.map(t=>({id:t.ID})),links:XS.map(t=>({source:t.source.ID,target:t.target.ID,value:t.value}))}),"getGraph"),g1e={nodesMap:YS,getConfig:o(()=>me().sankey,"getConfig"),getNodes:MJe,getLinks:IJe,getGraph:OJe,addLink:RJe,findOrCreateNode:NJe,getAccTitle:Rr,setAccTitle:Lr,getAccDescription:Mr,setAccDescription:Nr,getDiagramTitle:Ir,setDiagramTitle:$r,clear:LJe}});function p4(t,e){let r;if(e===void 0)for(let n of t)n!=null&&(r=n)&&(r=n);else{let n=-1;for(let i of t)(i=e(i,++n,t))!=null&&(r=i)&&(r=i)}return r}var v1e=N(()=>{"use strict";o(p4,"max")});function cy(t,e){let r;if(e===void 0)for(let n of t)n!=null&&(r>n||r===void 0&&n>=n)&&(r=n);else{let n=-1;for(let i of t)(i=e(i,++n,t))!=null&&(r>i||r===void 0&&i>=i)&&(r=i)}return r}var x1e=N(()=>{"use strict";o(cy,"min")});function uy(t,e){let r=0;if(e===void 0)for(let n of t)(n=+n)&&(r+=n);else{let n=-1;for(let i of t)(i=+e(i,++n,t))&&(r+=i)}return r}var b1e=N(()=>{"use strict";o(uy,"sum")});var PB=N(()=>{"use strict";v1e();x1e();b1e()});function PJe(t){return t.target.depth}function BB(t){return t.depth}function FB(t,e){return e-1-t.height}function m4(t,e){return t.sourceLinks.length?t.depth:e-1}function $B(t){return t.targetLinks.length?t.depth:t.sourceLinks.length?cy(t.sourceLinks,PJe)-1:0}var zB=N(()=>{"use strict";PB();o(PJe,"targetDepth");o(BB,"left");o(FB,"right");o(m4,"justify");o($B,"center")});function hy(t){return function(){return t}}var w1e=N(()=>{"use strict";o(hy,"constant")});function T1e(t,e){return KS(t.source,e.source)||t.index-e.index}function k1e(t,e){return KS(t.target,e.target)||t.index-e.index}function KS(t,e){return t.y0-e.y0}function GB(t){return t.value}function BJe(t){return t.index}function FJe(t){return t.nodes}function $Je(t){return t.links}function E1e(t,e){let r=t.get(e);if(!r)throw new Error("missing: "+e);return r}function S1e({nodes:t}){for(let e of t){let r=e.y0,n=r;for(let i of e.sourceLinks)i.y0=r+i.width/2,r+=i.width;for(let i of e.targetLinks)i.y1=n+i.width/2,n+=i.width}}function QS(){let t=0,e=0,r=1,n=1,i=24,a=8,s,l=BJe,u=m4,h,f,d=FJe,p=$Je,m=6;function g(){let O={nodes:d.apply(null,arguments),links:p.apply(null,arguments)};return y(O),v(O),x(O),b(O),T(O),S1e(O),O}o(g,"sankey"),g.update=function(O){return S1e(O),O},g.nodeId=function(O){return arguments.length?(l=typeof O=="function"?O:hy(O),g):l},g.nodeAlign=function(O){return arguments.length?(u=typeof O=="function"?O:hy(O),g):u},g.nodeSort=function(O){return arguments.length?(h=O,g):h},g.nodeWidth=function(O){return arguments.length?(i=+O,g):i},g.nodePadding=function(O){return arguments.length?(a=s=+O,g):a},g.nodes=function(O){return arguments.length?(d=typeof O=="function"?O:hy(O),g):d},g.links=function(O){return arguments.length?(p=typeof O=="function"?O:hy(O),g):p},g.linkSort=function(O){return arguments.length?(f=O,g):f},g.size=function(O){return arguments.length?(t=e=0,r=+O[0],n=+O[1],g):[r-t,n-e]},g.extent=function(O){return arguments.length?(t=+O[0][0],r=+O[1][0],e=+O[0][1],n=+O[1][1],g):[[t,e],[r,n]]},g.iterations=function(O){return arguments.length?(m=+O,g):m};function y({nodes:O,links:M}){for(let[F,P]of O.entries())P.index=F,P.sourceLinks=[],P.targetLinks=[];let B=new Map(O.map((F,P)=>[l(F,P,O),F]));for(let[F,P]of M.entries()){P.index=F;let{source:z,target:$}=P;typeof z!="object"&&(z=P.source=E1e(B,z)),typeof $!="object"&&($=P.target=E1e(B,$)),z.sourceLinks.push(P),$.targetLinks.push(P)}if(f!=null)for(let{sourceLinks:F,targetLinks:P}of O)F.sort(f),P.sort(f)}o(y,"computeNodeLinks");function v({nodes:O}){for(let M of O)M.value=M.fixedValue===void 0?Math.max(uy(M.sourceLinks,GB),uy(M.targetLinks,GB)):M.fixedValue}o(v,"computeNodeValues");function x({nodes:O}){let M=O.length,B=new Set(O),F=new Set,P=0;for(;B.size;){for(let z of B){z.depth=P;for(let{target:$}of z.sourceLinks)F.add($)}if(++P>M)throw new Error("circular link");B=F,F=new Set}}o(x,"computeNodeDepths");function b({nodes:O}){let M=O.length,B=new Set(O),F=new Set,P=0;for(;B.size;){for(let z of B){z.height=P;for(let{source:$}of z.targetLinks)F.add($)}if(++P>M)throw new Error("circular link");B=F,F=new Set}}o(b,"computeNodeHeights");function w({nodes:O}){let M=p4(O,P=>P.depth)+1,B=(r-t-i)/(M-1),F=new Array(M);for(let P of O){let z=Math.max(0,Math.min(M-1,Math.floor(u.call(null,P,M))));P.layer=z,P.x0=t+z*B,P.x1=P.x0+i,F[z]?F[z].push(P):F[z]=[P]}if(h)for(let P of F)P.sort(h);return F}o(w,"computeNodeLayers");function C(O){let M=cy(O,B=>(n-e-(B.length-1)*s)/uy(B,GB));for(let B of O){let F=e;for(let P of B){P.y0=F,P.y1=F+P.value*M,F=P.y1+s;for(let z of P.sourceLinks)z.width=z.value*M}F=(n-F+s)/(B.length+1);for(let P=0;PB.length)-1)),C(M);for(let B=0;B0))continue;let j=(H/Q-$.y0)*M;$.y0+=j,$.y1+=j,D($)}h===void 0&&z.sort(KS),S(z,B)}}o(E,"relaxLeftToRight");function A(O,M,B){for(let F=O.length,P=F-2;P>=0;--P){let z=O[P];for(let $ of z){let H=0,Q=0;for(let{target:ie,value:ne}of $.sourceLinks){let le=ne*(ie.layer-$.layer);H+=R($,ie)*le,Q+=le}if(!(Q>0))continue;let j=(H/Q-$.y0)*M;$.y0+=j,$.y1+=j,D($)}h===void 0&&z.sort(KS),S(z,B)}}o(A,"relaxRightToLeft");function S(O,M){let B=O.length>>1,F=O[B];I(O,F.y0-s,B-1,M),_(O,F.y1+s,B+1,M),I(O,n,O.length-1,M),_(O,e,0,M)}o(S,"resolveCollisions");function _(O,M,B,F){for(;B1e-6&&(P.y0+=z,P.y1+=z),M=P.y1+s}}o(_,"resolveCollisionsTopToBottom");function I(O,M,B,F){for(;B>=0;--B){let P=O[B],z=(P.y1-M)*F;z>1e-6&&(P.y0-=z,P.y1-=z),M=P.y0-s}}o(I,"resolveCollisionsBottomToTop");function D({sourceLinks:O,targetLinks:M}){if(f===void 0){for(let{source:{sourceLinks:B}}of M)B.sort(k1e);for(let{target:{targetLinks:B}}of O)B.sort(T1e)}}o(D,"reorderNodeLinks");function k(O){if(f===void 0)for(let{sourceLinks:M,targetLinks:B}of O)M.sort(k1e),B.sort(T1e)}o(k,"reorderLinks");function L(O,M){let B=O.y0-(O.sourceLinks.length-1)*s/2;for(let{target:F,width:P}of O.sourceLinks){if(F===M)break;B+=P+s}for(let{source:F,width:P}of M.targetLinks){if(F===O)break;B-=P}return B}o(L,"targetTop");function R(O,M){let B=M.y0-(M.targetLinks.length-1)*s/2;for(let{source:F,width:P}of M.targetLinks){if(F===O)break;B+=P+s}for(let{target:F,width:P}of O.sourceLinks){if(F===M)break;B-=P}return B}return o(R,"sourceTop"),g}var C1e=N(()=>{"use strict";PB();zB();w1e();o(T1e,"ascendingSourceBreadth");o(k1e,"ascendingTargetBreadth");o(KS,"ascendingBreadth");o(GB,"value");o(BJe,"defaultId");o(FJe,"defaultNodes");o($Je,"defaultLinks");o(E1e,"find");o(S1e,"computeLinkBreadths");o(QS,"Sankey")});function HB(){this._x0=this._y0=this._x1=this._y1=null,this._=""}function A1e(){return new HB}var VB,UB,Xp,zJe,WB,_1e=N(()=>{"use strict";VB=Math.PI,UB=2*VB,Xp=1e-6,zJe=UB-Xp;o(HB,"Path");o(A1e,"path");HB.prototype=A1e.prototype={constructor:HB,moveTo:o(function(t,e){this._+="M"+(this._x0=this._x1=+t)+","+(this._y0=this._y1=+e)},"moveTo"),closePath:o(function(){this._x1!==null&&(this._x1=this._x0,this._y1=this._y0,this._+="Z")},"closePath"),lineTo:o(function(t,e){this._+="L"+(this._x1=+t)+","+(this._y1=+e)},"lineTo"),quadraticCurveTo:o(function(t,e,r,n){this._+="Q"+ +t+","+ +e+","+(this._x1=+r)+","+(this._y1=+n)},"quadraticCurveTo"),bezierCurveTo:o(function(t,e,r,n,i,a){this._+="C"+ +t+","+ +e+","+ +r+","+ +n+","+(this._x1=+i)+","+(this._y1=+a)},"bezierCurveTo"),arcTo:o(function(t,e,r,n,i){t=+t,e=+e,r=+r,n=+n,i=+i;var a=this._x1,s=this._y1,l=r-t,u=n-e,h=a-t,f=s-e,d=h*h+f*f;if(i<0)throw new Error("negative radius: "+i);if(this._x1===null)this._+="M"+(this._x1=t)+","+(this._y1=e);else if(d>Xp)if(!(Math.abs(f*l-u*h)>Xp)||!i)this._+="L"+(this._x1=t)+","+(this._y1=e);else{var p=r-a,m=n-s,g=l*l+u*u,y=p*p+m*m,v=Math.sqrt(g),x=Math.sqrt(d),b=i*Math.tan((VB-Math.acos((g+d-y)/(2*v*x)))/2),w=b/x,C=b/v;Math.abs(w-1)>Xp&&(this._+="L"+(t+w*h)+","+(e+w*f)),this._+="A"+i+","+i+",0,0,"+ +(f*p>h*m)+","+(this._x1=t+C*l)+","+(this._y1=e+C*u)}},"arcTo"),arc:o(function(t,e,r,n,i,a){t=+t,e=+e,r=+r,a=!!a;var s=r*Math.cos(n),l=r*Math.sin(n),u=t+s,h=e+l,f=1^a,d=a?n-i:i-n;if(r<0)throw new Error("negative radius: "+r);this._x1===null?this._+="M"+u+","+h:(Math.abs(this._x1-u)>Xp||Math.abs(this._y1-h)>Xp)&&(this._+="L"+u+","+h),r&&(d<0&&(d=d%UB+UB),d>zJe?this._+="A"+r+","+r+",0,1,"+f+","+(t-s)+","+(e-l)+"A"+r+","+r+",0,1,"+f+","+(this._x1=u)+","+(this._y1=h):d>Xp&&(this._+="A"+r+","+r+",0,"+ +(d>=VB)+","+f+","+(this._x1=t+r*Math.cos(i))+","+(this._y1=e+r*Math.sin(i))))},"arc"),rect:o(function(t,e,r,n){this._+="M"+(this._x0=this._x1=+t)+","+(this._y0=this._y1=+e)+"h"+ +r+"v"+ +n+"h"+-r+"Z"},"rect"),toString:o(function(){return this._},"toString")};WB=A1e});var D1e=N(()=>{"use strict";_1e()});function ZS(t){return o(function(){return t},"constant")}var L1e=N(()=>{"use strict";o(ZS,"default")});function R1e(t){return t[0]}function N1e(t){return t[1]}var M1e=N(()=>{"use strict";o(R1e,"x");o(N1e,"y")});var I1e,O1e=N(()=>{"use strict";I1e=Array.prototype.slice});function GJe(t){return t.source}function VJe(t){return t.target}function UJe(t){var e=GJe,r=VJe,n=R1e,i=N1e,a=null;function s(){var l,u=I1e.call(arguments),h=e.apply(this,u),f=r.apply(this,u);if(a||(a=l=WB()),t(a,+n.apply(this,(u[0]=h,u)),+i.apply(this,u),+n.apply(this,(u[0]=f,u)),+i.apply(this,u)),l)return a=null,l+""||null}return o(s,"link"),s.source=function(l){return arguments.length?(e=l,s):e},s.target=function(l){return arguments.length?(r=l,s):r},s.x=function(l){return arguments.length?(n=typeof l=="function"?l:ZS(+l),s):n},s.y=function(l){return arguments.length?(i=typeof l=="function"?l:ZS(+l),s):i},s.context=function(l){return arguments.length?(a=l??null,s):a},s}function HJe(t,e,r,n,i){t.moveTo(e,r),t.bezierCurveTo(e=(e+n)/2,r,e,i,n,i)}function qB(){return UJe(HJe)}var P1e=N(()=>{"use strict";D1e();O1e();L1e();M1e();o(GJe,"linkSource");o(VJe,"linkTarget");o(UJe,"link");o(HJe,"curveHorizontal");o(qB,"linkHorizontal")});var B1e=N(()=>{"use strict";P1e()});function WJe(t){return[t.source.x1,t.y0]}function qJe(t){return[t.target.x0,t.y1]}function JS(){return qB().source(WJe).target(qJe)}var F1e=N(()=>{"use strict";B1e();o(WJe,"horizontalSource");o(qJe,"horizontalTarget");o(JS,"default")});var $1e=N(()=>{"use strict";C1e();zB();F1e()});var g4,z1e=N(()=>{"use strict";g4=class t{static{o(this,"Uid")}static{this.count=0}static next(e){return new t(e+ ++t.count)}constructor(e){this.id=e,this.href=`#${e}`}toString(){return"url("+this.href+")"}}});var YJe,XJe,G1e,V1e=N(()=>{"use strict";zt();dr();$1e();Ei();z1e();YJe={left:BB,right:FB,center:$B,justify:m4},XJe=o(function(t,e,r,n){let{securityLevel:i,sankey:a}=me(),s=A3.sankey,l;i==="sandbox"&&(l=Ge("#i"+e));let u=i==="sandbox"?Ge(l.nodes()[0].contentDocument.body):Ge("body"),h=i==="sandbox"?u.select(`[id="${e}"]`):Ge(`[id="${e}"]`),f=a?.width??s.width,d=a?.height??s.width,p=a?.useMaxWidth??s.useMaxWidth,m=a?.nodeAlignment??s.nodeAlignment,g=a?.prefix??s.prefix,y=a?.suffix??s.suffix,v=a?.showValues??s.showValues,x=n.db.getGraph(),b=YJe[m];QS().nodeId(I=>I.id).nodeWidth(10).nodePadding(10+(v?15:0)).nodeAlign(b).extent([[0,0],[f,d]])(x);let T=gu(e9);h.append("g").attr("class","nodes").selectAll(".node").data(x.nodes).join("g").attr("class","node").attr("id",I=>(I.uid=g4.next("node-")).id).attr("transform",function(I){return"translate("+I.x0+","+I.y0+")"}).attr("x",I=>I.x0).attr("y",I=>I.y0).append("rect").attr("height",I=>I.y1-I.y0).attr("width",I=>I.x1-I.x0).attr("fill",I=>T(I.id));let E=o(({id:I,value:D})=>v?`${I} +${g}${Math.round(D*100)/100}${y}`:I,"getText");h.append("g").attr("class","node-labels").attr("font-size",14).selectAll("text").data(x.nodes).join("text").attr("x",I=>I.x0(I.y1+I.y0)/2).attr("dy",`${v?"0":"0.35"}em`).attr("text-anchor",I=>I.x0(D.uid=g4.next("linearGradient-")).id).attr("gradientUnits","userSpaceOnUse").attr("x1",D=>D.source.x1).attr("x2",D=>D.target.x0);I.append("stop").attr("offset","0%").attr("stop-color",D=>T(D.source.id)),I.append("stop").attr("offset","100%").attr("stop-color",D=>T(D.target.id))}let _;switch(S){case"gradient":_=o(I=>I.uid,"coloring");break;case"source":_=o(I=>T(I.source.id),"coloring");break;case"target":_=o(I=>T(I.target.id),"coloring");break;default:_=S}A.append("path").attr("d",JS()).attr("stroke",_).attr("stroke-width",I=>Math.max(1,I.width)),Ao(void 0,h,0,p)},"draw"),G1e={draw:XJe}});var U1e,H1e=N(()=>{"use strict";U1e=o(t=>t.replaceAll(/^[^\S\n\r]+|[^\S\n\r]+$/g,"").replaceAll(/([\n\r])+/g,` +`).trim(),"prepareTextForParsing")});var jJe,W1e,q1e=N(()=>{"use strict";jJe=o(t=>`.label { + font-family: ${t.fontFamily}; + }`,"getStyles"),W1e=jJe});var Y1e={};hr(Y1e,{diagram:()=>QJe});var KJe,QJe,X1e=N(()=>{"use strict";m1e();y1e();V1e();H1e();q1e();KJe=d4.parse.bind(d4);d4.parse=t=>KJe(U1e(t));QJe={styles:W1e,parser:d4,db:g1e,renderer:G1e}});var Q1e,YB,tet,ret,net,iet,aet,Bf,XB=N(()=>{"use strict";ji();Ya();ir();mi();Q1e={packet:[]},YB=structuredClone(Q1e),tet=or.packet,ret=o(()=>{let t=Fi({...tet,...cr().packet});return t.showBits&&(t.paddingY+=10),t},"getConfig"),net=o(()=>YB.packet,"getPacket"),iet=o(t=>{t.length>0&&YB.packet.push(t)},"pushWord"),aet=o(()=>{Ar(),YB=structuredClone(Q1e)},"clear"),Bf={pushWord:iet,getPacket:net,getConfig:ret,clear:aet,setAccTitle:Lr,getAccTitle:Rr,setDiagramTitle:$r,getDiagramTitle:Ir,getAccDescription:Mr,setAccDescription:Nr}});var set,oet,cet,Z1e,J1e=N(()=>{"use strict";kp();vt();T1();XB();set=1e4,oet=o(t=>{$c(t,Bf);let e=-1,r=[],n=1,{bitsPerRow:i}=Bf.getConfig();for(let{start:a,end:s,label:l}of t.blocks){if(s&&s{if(t.end===void 0&&(t.end=t.start),t.start>t.end)throw new Error(`Block start ${t.start} is greater than block end ${t.end}.`);return t.end+1<=e*r?[t,void 0]:[{start:t.start,end:e*r-1,label:t.label},{start:e*r,end:t.end,label:t.label}]},"getNextFittingBlock"),Z1e={parse:o(async t=>{let e=await uo("packet",t);Y.debug(e),oet(e)},"parse")}});var uet,het,eye,tye=N(()=>{"use strict";Vc();Ei();uet=o((t,e,r,n)=>{let i=n.db,a=i.getConfig(),{rowHeight:s,paddingY:l,bitWidth:u,bitsPerRow:h}=a,f=i.getPacket(),d=i.getDiagramTitle(),p=s+l,m=p*(f.length+1)-(d?0:s),g=u*h+2,y=sa(e);y.attr("viewbox",`0 0 ${g} ${m}`),vn(y,m,g,a.useMaxWidth);for(let[v,x]of f.entries())het(y,x,v,a);y.append("text").text(d).attr("x",g/2).attr("y",m-p/2).attr("dominant-baseline","middle").attr("text-anchor","middle").attr("class","packetTitle")},"draw"),het=o((t,e,r,{rowHeight:n,paddingX:i,paddingY:a,bitWidth:s,bitsPerRow:l,showBits:u})=>{let h=t.append("g"),f=r*(n+a)+a;for(let d of e){let p=d.start%l*s+1,m=(d.end-d.start+1)*s-i;if(h.append("rect").attr("x",p).attr("y",f).attr("width",m).attr("height",n).attr("class","packetBlock"),h.append("text").attr("x",p+m/2).attr("y",f+n/2).attr("class","packetLabel").attr("dominant-baseline","middle").attr("text-anchor","middle").text(d.label),!u)continue;let g=d.end===d.start,y=f-2;h.append("text").attr("x",p+(g?m/2:0)).attr("y",y).attr("class","packetByte start").attr("dominant-baseline","auto").attr("text-anchor",g?"middle":"start").text(d.start),g||h.append("text").attr("x",p+m).attr("y",y).attr("class","packetByte end").attr("dominant-baseline","auto").attr("text-anchor","end").text(d.end)}},"drawWord"),eye={draw:uet}});var fet,rye,nye=N(()=>{"use strict";ir();fet={byteFontSize:"10px",startByteColor:"black",endByteColor:"black",labelColor:"black",labelFontSize:"12px",titleColor:"black",titleFontSize:"14px",blockStrokeColor:"black",blockStrokeWidth:"1",blockFillColor:"#efefef"},rye=o(({packet:t}={})=>{let e=Fi(fet,t);return` + .packetByte { + font-size: ${e.byteFontSize}; + } + .packetByte.start { + fill: ${e.startByteColor}; + } + .packetByte.end { + fill: ${e.endByteColor}; + } + .packetLabel { + fill: ${e.labelColor}; + font-size: ${e.labelFontSize}; + } + .packetTitle { + fill: ${e.titleColor}; + font-size: ${e.titleFontSize}; + } + .packetBlock { + stroke: ${e.blockStrokeColor}; + stroke-width: ${e.blockStrokeWidth}; + fill: ${e.blockFillColor}; + } + `},"styles")});var iye={};hr(iye,{diagram:()=>det});var det,aye=N(()=>{"use strict";XB();J1e();tye();nye();det={parser:Z1e,db:Bf,renderer:eye,styles:rye}});var fy,lye,jp,get,yet,cye,vet,xet,bet,wet,Tet,ket,Eet,Kp,jB=N(()=>{"use strict";ji();Ya();ir();mi();fy={showLegend:!0,ticks:5,max:null,min:0,graticule:"circle"},lye={axes:[],curves:[],options:fy},jp=structuredClone(lye),get=or.radar,yet=o(()=>Fi({...get,...cr().radar}),"getConfig"),cye=o(()=>jp.axes,"getAxes"),vet=o(()=>jp.curves,"getCurves"),xet=o(()=>jp.options,"getOptions"),bet=o(t=>{jp.axes=t.map(e=>({name:e.name,label:e.label??e.name}))},"setAxes"),wet=o(t=>{jp.curves=t.map(e=>({name:e.name,label:e.label??e.name,entries:Tet(e.entries)}))},"setCurves"),Tet=o(t=>{if(t[0].axis==null)return t.map(r=>r.value);let e=cye();if(e.length===0)throw new Error("Axes must be populated before curves for reference entries");return e.map(r=>{let n=t.find(i=>i.axis?.$refText===r.name);if(n===void 0)throw new Error("Missing entry for axis "+r.label);return n.value})},"computeCurveEntries"),ket=o(t=>{let e=t.reduce((r,n)=>(r[n.name]=n,r),{});jp.options={showLegend:e.showLegend?.value??fy.showLegend,ticks:e.ticks?.value??fy.ticks,max:e.max?.value??fy.max,min:e.min?.value??fy.min,graticule:e.graticule?.value??fy.graticule}},"setOptions"),Eet=o(()=>{Ar(),jp=structuredClone(lye)},"clear"),Kp={getAxes:cye,getCurves:vet,getOptions:xet,setAxes:bet,setCurves:wet,setOptions:ket,getConfig:yet,clear:Eet,setAccTitle:Lr,getAccTitle:Rr,setDiagramTitle:$r,getDiagramTitle:Ir,getAccDescription:Mr,setAccDescription:Nr}});var Cet,uye,hye=N(()=>{"use strict";kp();vt();T1();jB();Cet=o(t=>{$c(t,Kp);let{axes:e,curves:r,options:n}=t;Kp.setAxes(e),Kp.setCurves(r),Kp.setOptions(n)},"populate"),uye={parse:o(async t=>{let e=await uo("radar",t);Y.debug(e),Cet(e)},"parse")}});function Ret(t,e,r,n,i,a,s){let l=e.length,u=Math.min(s.width,s.height)/2;r.forEach((h,f)=>{if(h.entries.length!==l)return;let d=h.entries.map((p,m)=>{let g=2*Math.PI*m/l-Math.PI/2,y=Net(p,n,i,u),v=y*Math.cos(g),x=y*Math.sin(g);return{x:v,y:x}});a==="circle"?t.append("path").attr("d",Met(d,s.curveTension)).attr("class",`radarCurve-${f}`):a==="polygon"&&t.append("polygon").attr("points",d.map(p=>`${p.x},${p.y}`).join(" ")).attr("class",`radarCurve-${f}`)})}function Net(t,e,r,n){let i=Math.min(Math.max(t,e),r);return n*(i-e)/(r-e)}function Met(t,e){let r=t.length,n=`M${t[0].x},${t[0].y}`;for(let i=0;i{let h=t.append("g").attr("transform",`translate(${i}, ${a+u*s})`);h.append("rect").attr("width",12).attr("height",12).attr("class",`radarLegendBox-${u}`),h.append("text").attr("x",16).attr("y",0).attr("class","radarLegendText").text(l.label)})}var Aet,_et,Det,Let,fye,dye=N(()=>{"use strict";Vc();Aet=o((t,e,r,n)=>{let i=n.db,a=i.getAxes(),s=i.getCurves(),l=i.getOptions(),u=i.getConfig(),h=i.getDiagramTitle(),f=sa(e),d=_et(f,u),p=l.max??Math.max(...s.map(y=>Math.max(...y.entries))),m=l.min,g=Math.min(u.width,u.height)/2;Det(d,a,g,l.ticks,l.graticule),Let(d,a,g,u),Ret(d,a,s,m,p,l.graticule,u),Iet(d,s,l.showLegend,u),d.append("text").attr("class","radarTitle").text(h).attr("x",0).attr("y",-u.height/2-u.marginTop)},"draw"),_et=o((t,e)=>{let r=e.width+e.marginLeft+e.marginRight,n=e.height+e.marginTop+e.marginBottom,i={x:e.marginLeft+e.width/2,y:e.marginTop+e.height/2};return t.attr("viewbox",`0 0 ${r} ${n}`).attr("width",r).attr("height",n),t.append("g").attr("transform",`translate(${i.x}, ${i.y})`)},"drawFrame"),Det=o((t,e,r,n,i)=>{if(i==="circle")for(let a=0;a{let d=2*f*Math.PI/a-Math.PI/2,p=l*Math.cos(d),m=l*Math.sin(d);return`${p},${m}`}).join(" ");t.append("polygon").attr("points",u).attr("class","radarGraticule")}}},"drawGraticule"),Let=o((t,e,r,n)=>{let i=e.length;for(let a=0;a{"use strict";ir();_y();ji();Oet=o((t,e)=>{let r="";for(let n=0;n{let e=oh(),r=cr(),n=Fi(e,r.themeVariables),i=Fi(n.radar,t);return{themeVariables:n,radarOptions:i}},"buildRadarStyleOptions"),pye=o(({radar:t}={})=>{let{themeVariables:e,radarOptions:r}=Pet(t);return` + .radarTitle { + font-size: ${e.fontSize}; + color: ${e.titleColor}; + dominant-baseline: hanging; + text-anchor: middle; + } + .radarAxisLine { + stroke: ${r.axisColor}; + stroke-width: ${r.axisStrokeWidth}; + } + .radarAxisLabel { + dominant-baseline: middle; + text-anchor: middle; + font-size: ${r.axisLabelFontSize}px; + color: ${r.axisColor}; + } + .radarGraticule { + fill: ${r.graticuleColor}; + fill-opacity: ${r.graticuleOpacity}; + stroke: ${r.graticuleColor}; + stroke-width: ${r.graticuleStrokeWidth}; + } + .radarLegendText { + text-anchor: start; + font-size: ${r.legendFontSize}px; + dominant-baseline: hanging; + } + ${Oet(e,r)} + `},"styles")});var gye={};hr(gye,{diagram:()=>Bet});var Bet,yye=N(()=>{"use strict";jB();hye();dye();mye();Bet={parser:uye,db:Kp,renderer:fye,styles:pye}});var KB,bye,wye=N(()=>{"use strict";KB=function(){var t=o(function(w,C,T,E){for(T=T||{},E=w.length;E--;T[w[E]]=C);return T},"o"),e=[1,7],r=[1,13],n=[1,14],i=[1,15],a=[1,19],s=[1,16],l=[1,17],u=[1,18],h=[8,30],f=[8,21,28,29,30,31,32,40,44,47],d=[1,23],p=[1,24],m=[8,15,16,21,28,29,30,31,32,40,44,47],g=[8,15,16,21,27,28,29,30,31,32,40,44,47],y=[1,49],v={trace:o(function(){},"trace"),yy:{},symbols_:{error:2,spaceLines:3,SPACELINE:4,NL:5,separator:6,SPACE:7,EOF:8,start:9,BLOCK_DIAGRAM_KEY:10,document:11,stop:12,statement:13,link:14,LINK:15,START_LINK:16,LINK_LABEL:17,STR:18,nodeStatement:19,columnsStatement:20,SPACE_BLOCK:21,blockStatement:22,classDefStatement:23,cssClassStatement:24,styleStatement:25,node:26,SIZE:27,COLUMNS:28,"id-block":29,end:30,block:31,NODE_ID:32,nodeShapeNLabel:33,dirList:34,DIR:35,NODE_DSTART:36,NODE_DEND:37,BLOCK_ARROW_START:38,BLOCK_ARROW_END:39,classDef:40,CLASSDEF_ID:41,CLASSDEF_STYLEOPTS:42,DEFAULT:43,class:44,CLASSENTITY_IDS:45,STYLECLASS:46,style:47,STYLE_ENTITY_IDS:48,STYLE_DEFINITION_DATA:49,$accept:0,$end:1},terminals_:{2:"error",4:"SPACELINE",5:"NL",7:"SPACE",8:"EOF",10:"BLOCK_DIAGRAM_KEY",15:"LINK",16:"START_LINK",17:"LINK_LABEL",18:"STR",21:"SPACE_BLOCK",27:"SIZE",28:"COLUMNS",29:"id-block",30:"end",31:"block",32:"NODE_ID",35:"DIR",36:"NODE_DSTART",37:"NODE_DEND",38:"BLOCK_ARROW_START",39:"BLOCK_ARROW_END",40:"classDef",41:"CLASSDEF_ID",42:"CLASSDEF_STYLEOPTS",43:"DEFAULT",44:"class",45:"CLASSENTITY_IDS",46:"STYLECLASS",47:"style",48:"STYLE_ENTITY_IDS",49:"STYLE_DEFINITION_DATA"},productions_:[0,[3,1],[3,2],[3,2],[6,1],[6,1],[6,1],[9,3],[12,1],[12,1],[12,2],[12,2],[11,1],[11,2],[14,1],[14,4],[13,1],[13,1],[13,1],[13,1],[13,1],[13,1],[13,1],[19,3],[19,2],[19,1],[20,1],[22,4],[22,3],[26,1],[26,2],[34,1],[34,2],[33,3],[33,4],[23,3],[23,3],[24,3],[25,3]],performAction:o(function(C,T,E,A,S,_,I){var D=_.length-1;switch(S){case 4:A.getLogger().debug("Rule: separator (NL) ");break;case 5:A.getLogger().debug("Rule: separator (Space) ");break;case 6:A.getLogger().debug("Rule: separator (EOF) ");break;case 7:A.getLogger().debug("Rule: hierarchy: ",_[D-1]),A.setHierarchy(_[D-1]);break;case 8:A.getLogger().debug("Stop NL ");break;case 9:A.getLogger().debug("Stop EOF ");break;case 10:A.getLogger().debug("Stop NL2 ");break;case 11:A.getLogger().debug("Stop EOF2 ");break;case 12:A.getLogger().debug("Rule: statement: ",_[D]),typeof _[D].length=="number"?this.$=_[D]:this.$=[_[D]];break;case 13:A.getLogger().debug("Rule: statement #2: ",_[D-1]),this.$=[_[D-1]].concat(_[D]);break;case 14:A.getLogger().debug("Rule: link: ",_[D],C),this.$={edgeTypeStr:_[D],label:""};break;case 15:A.getLogger().debug("Rule: LABEL link: ",_[D-3],_[D-1],_[D]),this.$={edgeTypeStr:_[D],label:_[D-1]};break;case 18:let k=parseInt(_[D]),L=A.generateId();this.$={id:L,type:"space",label:"",width:k,children:[]};break;case 23:A.getLogger().debug("Rule: (nodeStatement link node) ",_[D-2],_[D-1],_[D]," typestr: ",_[D-1].edgeTypeStr);let R=A.edgeStrToEdgeData(_[D-1].edgeTypeStr);this.$=[{id:_[D-2].id,label:_[D-2].label,type:_[D-2].type,directions:_[D-2].directions},{id:_[D-2].id+"-"+_[D].id,start:_[D-2].id,end:_[D].id,label:_[D-1].label,type:"edge",directions:_[D].directions,arrowTypeEnd:R,arrowTypeStart:"arrow_open"},{id:_[D].id,label:_[D].label,type:A.typeStr2Type(_[D].typeStr),directions:_[D].directions}];break;case 24:A.getLogger().debug("Rule: nodeStatement (abc88 node size) ",_[D-1],_[D]),this.$={id:_[D-1].id,label:_[D-1].label,type:A.typeStr2Type(_[D-1].typeStr),directions:_[D-1].directions,widthInColumns:parseInt(_[D],10)};break;case 25:A.getLogger().debug("Rule: nodeStatement (node) ",_[D]),this.$={id:_[D].id,label:_[D].label,type:A.typeStr2Type(_[D].typeStr),directions:_[D].directions,widthInColumns:1};break;case 26:A.getLogger().debug("APA123",this?this:"na"),A.getLogger().debug("COLUMNS: ",_[D]),this.$={type:"column-setting",columns:_[D]==="auto"?-1:parseInt(_[D])};break;case 27:A.getLogger().debug("Rule: id-block statement : ",_[D-2],_[D-1]);let O=A.generateId();this.$={..._[D-2],type:"composite",children:_[D-1]};break;case 28:A.getLogger().debug("Rule: blockStatement : ",_[D-2],_[D-1],_[D]);let M=A.generateId();this.$={id:M,type:"composite",label:"",children:_[D-1]};break;case 29:A.getLogger().debug("Rule: node (NODE_ID separator): ",_[D]),this.$={id:_[D]};break;case 30:A.getLogger().debug("Rule: node (NODE_ID nodeShapeNLabel separator): ",_[D-1],_[D]),this.$={id:_[D-1],label:_[D].label,typeStr:_[D].typeStr,directions:_[D].directions};break;case 31:A.getLogger().debug("Rule: dirList: ",_[D]),this.$=[_[D]];break;case 32:A.getLogger().debug("Rule: dirList: ",_[D-1],_[D]),this.$=[_[D-1]].concat(_[D]);break;case 33:A.getLogger().debug("Rule: nodeShapeNLabel: ",_[D-2],_[D-1],_[D]),this.$={typeStr:_[D-2]+_[D],label:_[D-1]};break;case 34:A.getLogger().debug("Rule: BLOCK_ARROW nodeShapeNLabel: ",_[D-3],_[D-2]," #3:",_[D-1],_[D]),this.$={typeStr:_[D-3]+_[D],label:_[D-2],directions:_[D-1]};break;case 35:case 36:this.$={type:"classDef",id:_[D-1].trim(),css:_[D].trim()};break;case 37:this.$={type:"applyClass",id:_[D-1].trim(),styleClass:_[D].trim()};break;case 38:this.$={type:"applyStyles",id:_[D-1].trim(),stylesStr:_[D].trim()};break}},"anonymous"),table:[{9:1,10:[1,2]},{1:[3]},{11:3,13:4,19:5,20:6,21:e,22:8,23:9,24:10,25:11,26:12,28:r,29:n,31:i,32:a,40:s,44:l,47:u},{8:[1,20]},t(h,[2,12],{13:4,19:5,20:6,22:8,23:9,24:10,25:11,26:12,11:21,21:e,28:r,29:n,31:i,32:a,40:s,44:l,47:u}),t(f,[2,16],{14:22,15:d,16:p}),t(f,[2,17]),t(f,[2,18]),t(f,[2,19]),t(f,[2,20]),t(f,[2,21]),t(f,[2,22]),t(m,[2,25],{27:[1,25]}),t(f,[2,26]),{19:26,26:12,32:a},{11:27,13:4,19:5,20:6,21:e,22:8,23:9,24:10,25:11,26:12,28:r,29:n,31:i,32:a,40:s,44:l,47:u},{41:[1,28],43:[1,29]},{45:[1,30]},{48:[1,31]},t(g,[2,29],{33:32,36:[1,33],38:[1,34]}),{1:[2,7]},t(h,[2,13]),{26:35,32:a},{32:[2,14]},{17:[1,36]},t(m,[2,24]),{11:37,13:4,14:22,15:d,16:p,19:5,20:6,21:e,22:8,23:9,24:10,25:11,26:12,28:r,29:n,31:i,32:a,40:s,44:l,47:u},{30:[1,38]},{42:[1,39]},{42:[1,40]},{46:[1,41]},{49:[1,42]},t(g,[2,30]),{18:[1,43]},{18:[1,44]},t(m,[2,23]),{18:[1,45]},{30:[1,46]},t(f,[2,28]),t(f,[2,35]),t(f,[2,36]),t(f,[2,37]),t(f,[2,38]),{37:[1,47]},{34:48,35:y},{15:[1,50]},t(f,[2,27]),t(g,[2,33]),{39:[1,51]},{34:52,35:y,39:[2,31]},{32:[2,15]},t(g,[2,34]),{39:[2,32]}],defaultActions:{20:[2,7],23:[2,14],50:[2,15],52:[2,32]},parseError:o(function(C,T){if(T.recoverable)this.trace(C);else{var E=new Error(C);throw E.hash=T,E}},"parseError"),parse:o(function(C){var T=this,E=[0],A=[],S=[null],_=[],I=this.table,D="",k=0,L=0,R=0,O=2,M=1,B=_.slice.call(arguments,1),F=Object.create(this.lexer),P={yy:{}};for(var z in this.yy)Object.prototype.hasOwnProperty.call(this.yy,z)&&(P.yy[z]=this.yy[z]);F.setInput(C,P.yy),P.yy.lexer=F,P.yy.parser=this,typeof F.yylloc>"u"&&(F.yylloc={});var $=F.yylloc;_.push($);var H=F.options&&F.options.ranges;typeof P.yy.parseError=="function"?this.parseError=P.yy.parseError:this.parseError=Object.getPrototypeOf(this).parseError;function Q(ce){E.length=E.length-2*ce,S.length=S.length-ce,_.length=_.length-ce}o(Q,"popStack");function j(){var ce;return ce=A.pop()||F.lex()||M,typeof ce!="number"&&(ce instanceof Array&&(A=ce,ce=A.pop()),ce=T.symbols_[ce]||ce),ce}o(j,"lex");for(var ie,ne,le,he,K,X,te={},J,se,ue,Z;;){if(le=E[E.length-1],this.defaultActions[le]?he=this.defaultActions[le]:((ie===null||typeof ie>"u")&&(ie=j()),he=I[le]&&I[le][ie]),typeof he>"u"||!he.length||!he[0]){var Se="";Z=[];for(J in I[le])this.terminals_[J]&&J>O&&Z.push("'"+this.terminals_[J]+"'");F.showPosition?Se="Parse error on line "+(k+1)+`: +`+F.showPosition()+` +Expecting `+Z.join(", ")+", got '"+(this.terminals_[ie]||ie)+"'":Se="Parse error on line "+(k+1)+": Unexpected "+(ie==M?"end of input":"'"+(this.terminals_[ie]||ie)+"'"),this.parseError(Se,{text:F.match,token:this.terminals_[ie]||ie,line:F.yylineno,loc:$,expected:Z})}if(he[0]instanceof Array&&he.length>1)throw new Error("Parse Error: multiple actions possible at state: "+le+", token: "+ie);switch(he[0]){case 1:E.push(ie),S.push(F.yytext),_.push(F.yylloc),E.push(he[1]),ie=null,ne?(ie=ne,ne=null):(L=F.yyleng,D=F.yytext,k=F.yylineno,$=F.yylloc,R>0&&R--);break;case 2:if(se=this.productions_[he[1]][1],te.$=S[S.length-se],te._$={first_line:_[_.length-(se||1)].first_line,last_line:_[_.length-1].last_line,first_column:_[_.length-(se||1)].first_column,last_column:_[_.length-1].last_column},H&&(te._$.range=[_[_.length-(se||1)].range[0],_[_.length-1].range[1]]),X=this.performAction.apply(te,[D,L,k,P.yy,he[1],S,_].concat(B)),typeof X<"u")return X;se&&(E=E.slice(0,-1*se*2),S=S.slice(0,-1*se),_=_.slice(0,-1*se)),E.push(this.productions_[he[1]][0]),S.push(te.$),_.push(te._$),ue=I[E[E.length-2]][E[E.length-1]],E.push(ue);break;case 3:return!0}}return!0},"parse")},x=function(){var w={EOF:1,parseError:o(function(T,E){if(this.yy.parser)this.yy.parser.parseError(T,E);else throw new Error(T)},"parseError"),setInput:o(function(C,T){return this.yy=T||this.yy||{},this._input=C,this._more=this._backtrack=this.done=!1,this.yylineno=this.yyleng=0,this.yytext=this.matched=this.match="",this.conditionStack=["INITIAL"],this.yylloc={first_line:1,first_column:0,last_line:1,last_column:0},this.options.ranges&&(this.yylloc.range=[0,0]),this.offset=0,this},"setInput"),input:o(function(){var C=this._input[0];this.yytext+=C,this.yyleng++,this.offset++,this.match+=C,this.matched+=C;var T=C.match(/(?:\r\n?|\n).*/g);return T?(this.yylineno++,this.yylloc.last_line++):this.yylloc.last_column++,this.options.ranges&&this.yylloc.range[1]++,this._input=this._input.slice(1),C},"input"),unput:o(function(C){var T=C.length,E=C.split(/(?:\r\n?|\n)/g);this._input=C+this._input,this.yytext=this.yytext.substr(0,this.yytext.length-T),this.offset-=T;var A=this.match.split(/(?:\r\n?|\n)/g);this.match=this.match.substr(0,this.match.length-1),this.matched=this.matched.substr(0,this.matched.length-1),E.length-1&&(this.yylineno-=E.length-1);var S=this.yylloc.range;return this.yylloc={first_line:this.yylloc.first_line,last_line:this.yylineno+1,first_column:this.yylloc.first_column,last_column:E?(E.length===A.length?this.yylloc.first_column:0)+A[A.length-E.length].length-E[0].length:this.yylloc.first_column-T},this.options.ranges&&(this.yylloc.range=[S[0],S[0]+this.yyleng-T]),this.yyleng=this.yytext.length,this},"unput"),more:o(function(){return this._more=!0,this},"more"),reject:o(function(){if(this.options.backtrack_lexer)this._backtrack=!0;else return this.parseError("Lexical error on line "+(this.yylineno+1)+`. You can only invoke reject() in the lexer when the lexer is of the backtracking persuasion (options.backtrack_lexer = true). +`+this.showPosition(),{text:"",token:null,line:this.yylineno});return this},"reject"),less:o(function(C){this.unput(this.match.slice(C))},"less"),pastInput:o(function(){var C=this.matched.substr(0,this.matched.length-this.match.length);return(C.length>20?"...":"")+C.substr(-20).replace(/\n/g,"")},"pastInput"),upcomingInput:o(function(){var C=this.match;return C.length<20&&(C+=this._input.substr(0,20-C.length)),(C.substr(0,20)+(C.length>20?"...":"")).replace(/\n/g,"")},"upcomingInput"),showPosition:o(function(){var C=this.pastInput(),T=new Array(C.length+1).join("-");return C+this.upcomingInput()+` +`+T+"^"},"showPosition"),test_match:o(function(C,T){var E,A,S;if(this.options.backtrack_lexer&&(S={yylineno:this.yylineno,yylloc:{first_line:this.yylloc.first_line,last_line:this.last_line,first_column:this.yylloc.first_column,last_column:this.yylloc.last_column},yytext:this.yytext,match:this.match,matches:this.matches,matched:this.matched,yyleng:this.yyleng,offset:this.offset,_more:this._more,_input:this._input,yy:this.yy,conditionStack:this.conditionStack.slice(0),done:this.done},this.options.ranges&&(S.yylloc.range=this.yylloc.range.slice(0))),A=C[0].match(/(?:\r\n?|\n).*/g),A&&(this.yylineno+=A.length),this.yylloc={first_line:this.yylloc.last_line,last_line:this.yylineno+1,first_column:this.yylloc.last_column,last_column:A?A[A.length-1].length-A[A.length-1].match(/\r?\n?/)[0].length:this.yylloc.last_column+C[0].length},this.yytext+=C[0],this.match+=C[0],this.matches=C,this.yyleng=this.yytext.length,this.options.ranges&&(this.yylloc.range=[this.offset,this.offset+=this.yyleng]),this._more=!1,this._backtrack=!1,this._input=this._input.slice(C[0].length),this.matched+=C[0],E=this.performAction.call(this,this.yy,this,T,this.conditionStack[this.conditionStack.length-1]),this.done&&this._input&&(this.done=!1),E)return E;if(this._backtrack){for(var _ in S)this[_]=S[_];return!1}return!1},"test_match"),next:o(function(){if(this.done)return this.EOF;this._input||(this.done=!0);var C,T,E,A;this._more||(this.yytext="",this.match="");for(var S=this._currentRules(),_=0;_T[0].length)){if(T=E,A=_,this.options.backtrack_lexer){if(C=this.test_match(E,S[_]),C!==!1)return C;if(this._backtrack){T=!1;continue}else return!1}else if(!this.options.flex)break}return T?(C=this.test_match(T,S[A]),C!==!1?C:!1):this._input===""?this.EOF:this.parseError("Lexical error on line "+(this.yylineno+1)+`. Unrecognized text. +`+this.showPosition(),{text:"",token:null,line:this.yylineno})},"next"),lex:o(function(){var T=this.next();return T||this.lex()},"lex"),begin:o(function(T){this.conditionStack.push(T)},"begin"),popState:o(function(){var T=this.conditionStack.length-1;return T>0?this.conditionStack.pop():this.conditionStack[0]},"popState"),_currentRules:o(function(){return this.conditionStack.length&&this.conditionStack[this.conditionStack.length-1]?this.conditions[this.conditionStack[this.conditionStack.length-1]].rules:this.conditions.INITIAL.rules},"_currentRules"),topState:o(function(T){return T=this.conditionStack.length-1-Math.abs(T||0),T>=0?this.conditionStack[T]:"INITIAL"},"topState"),pushState:o(function(T){this.begin(T)},"pushState"),stateStackSize:o(function(){return this.conditionStack.length},"stateStackSize"),options:{},performAction:o(function(T,E,A,S){var _=S;switch(A){case 0:return 10;case 1:return T.getLogger().debug("Found space-block"),31;break;case 2:return T.getLogger().debug("Found nl-block"),31;break;case 3:return T.getLogger().debug("Found space-block"),29;break;case 4:T.getLogger().debug(".",E.yytext);break;case 5:T.getLogger().debug("_",E.yytext);break;case 6:return 5;case 7:return E.yytext=-1,28;break;case 8:return E.yytext=E.yytext.replace(/columns\s+/,""),T.getLogger().debug("COLUMNS (LEX)",E.yytext),28;break;case 9:this.pushState("md_string");break;case 10:return"MD_STR";case 11:this.popState();break;case 12:this.pushState("string");break;case 13:T.getLogger().debug("LEX: POPPING STR:",E.yytext),this.popState();break;case 14:return T.getLogger().debug("LEX: STR end:",E.yytext),"STR";break;case 15:return E.yytext=E.yytext.replace(/space\:/,""),T.getLogger().debug("SPACE NUM (LEX)",E.yytext),21;break;case 16:return E.yytext="1",T.getLogger().debug("COLUMNS (LEX)",E.yytext),21;break;case 17:return 43;case 18:return"LINKSTYLE";case 19:return"INTERPOLATE";case 20:return this.pushState("CLASSDEF"),40;break;case 21:return this.popState(),this.pushState("CLASSDEFID"),"DEFAULT_CLASSDEF_ID";break;case 22:return this.popState(),this.pushState("CLASSDEFID"),41;break;case 23:return this.popState(),42;break;case 24:return this.pushState("CLASS"),44;break;case 25:return this.popState(),this.pushState("CLASS_STYLE"),45;break;case 26:return this.popState(),46;break;case 27:return this.pushState("STYLE_STMNT"),47;break;case 28:return this.popState(),this.pushState("STYLE_DEFINITION"),48;break;case 29:return this.popState(),49;break;case 30:return this.pushState("acc_title"),"acc_title";break;case 31:return this.popState(),"acc_title_value";break;case 32:return this.pushState("acc_descr"),"acc_descr";break;case 33:return this.popState(),"acc_descr_value";break;case 34:this.pushState("acc_descr_multiline");break;case 35:this.popState();break;case 36:return"acc_descr_multiline_value";case 37:return 30;case 38:return this.popState(),T.getLogger().debug("Lex: (("),"NODE_DEND";break;case 39:return this.popState(),T.getLogger().debug("Lex: (("),"NODE_DEND";break;case 40:return this.popState(),T.getLogger().debug("Lex: ))"),"NODE_DEND";break;case 41:return this.popState(),T.getLogger().debug("Lex: (("),"NODE_DEND";break;case 42:return this.popState(),T.getLogger().debug("Lex: (("),"NODE_DEND";break;case 43:return this.popState(),T.getLogger().debug("Lex: (-"),"NODE_DEND";break;case 44:return this.popState(),T.getLogger().debug("Lex: -)"),"NODE_DEND";break;case 45:return this.popState(),T.getLogger().debug("Lex: (("),"NODE_DEND";break;case 46:return this.popState(),T.getLogger().debug("Lex: ]]"),"NODE_DEND";break;case 47:return this.popState(),T.getLogger().debug("Lex: ("),"NODE_DEND";break;case 48:return this.popState(),T.getLogger().debug("Lex: ])"),"NODE_DEND";break;case 49:return this.popState(),T.getLogger().debug("Lex: /]"),"NODE_DEND";break;case 50:return this.popState(),T.getLogger().debug("Lex: /]"),"NODE_DEND";break;case 51:return this.popState(),T.getLogger().debug("Lex: )]"),"NODE_DEND";break;case 52:return this.popState(),T.getLogger().debug("Lex: )"),"NODE_DEND";break;case 53:return this.popState(),T.getLogger().debug("Lex: ]>"),"NODE_DEND";break;case 54:return this.popState(),T.getLogger().debug("Lex: ]"),"NODE_DEND";break;case 55:return T.getLogger().debug("Lexa: -)"),this.pushState("NODE"),36;break;case 56:return T.getLogger().debug("Lexa: (-"),this.pushState("NODE"),36;break;case 57:return T.getLogger().debug("Lexa: ))"),this.pushState("NODE"),36;break;case 58:return T.getLogger().debug("Lexa: )"),this.pushState("NODE"),36;break;case 59:return T.getLogger().debug("Lex: ((("),this.pushState("NODE"),36;break;case 60:return T.getLogger().debug("Lexa: )"),this.pushState("NODE"),36;break;case 61:return T.getLogger().debug("Lexa: )"),this.pushState("NODE"),36;break;case 62:return T.getLogger().debug("Lexa: )"),this.pushState("NODE"),36;break;case 63:return T.getLogger().debug("Lexc: >"),this.pushState("NODE"),36;break;case 64:return T.getLogger().debug("Lexa: (["),this.pushState("NODE"),36;break;case 65:return T.getLogger().debug("Lexa: )"),this.pushState("NODE"),36;break;case 66:return this.pushState("NODE"),36;break;case 67:return this.pushState("NODE"),36;break;case 68:return this.pushState("NODE"),36;break;case 69:return this.pushState("NODE"),36;break;case 70:return this.pushState("NODE"),36;break;case 71:return this.pushState("NODE"),36;break;case 72:return this.pushState("NODE"),36;break;case 73:return T.getLogger().debug("Lexa: ["),this.pushState("NODE"),36;break;case 74:return this.pushState("BLOCK_ARROW"),T.getLogger().debug("LEX ARR START"),38;break;case 75:return T.getLogger().debug("Lex: NODE_ID",E.yytext),32;break;case 76:return T.getLogger().debug("Lex: EOF",E.yytext),8;break;case 77:this.pushState("md_string");break;case 78:this.pushState("md_string");break;case 79:return"NODE_DESCR";case 80:this.popState();break;case 81:T.getLogger().debug("Lex: Starting string"),this.pushState("string");break;case 82:T.getLogger().debug("LEX ARR: Starting string"),this.pushState("string");break;case 83:return T.getLogger().debug("LEX: NODE_DESCR:",E.yytext),"NODE_DESCR";break;case 84:T.getLogger().debug("LEX POPPING"),this.popState();break;case 85:T.getLogger().debug("Lex: =>BAE"),this.pushState("ARROW_DIR");break;case 86:return E.yytext=E.yytext.replace(/^,\s*/,""),T.getLogger().debug("Lex (right): dir:",E.yytext),"DIR";break;case 87:return E.yytext=E.yytext.replace(/^,\s*/,""),T.getLogger().debug("Lex (left):",E.yytext),"DIR";break;case 88:return E.yytext=E.yytext.replace(/^,\s*/,""),T.getLogger().debug("Lex (x):",E.yytext),"DIR";break;case 89:return E.yytext=E.yytext.replace(/^,\s*/,""),T.getLogger().debug("Lex (y):",E.yytext),"DIR";break;case 90:return E.yytext=E.yytext.replace(/^,\s*/,""),T.getLogger().debug("Lex (up):",E.yytext),"DIR";break;case 91:return E.yytext=E.yytext.replace(/^,\s*/,""),T.getLogger().debug("Lex (down):",E.yytext),"DIR";break;case 92:return E.yytext="]>",T.getLogger().debug("Lex (ARROW_DIR end):",E.yytext),this.popState(),this.popState(),"BLOCK_ARROW_END";break;case 93:return T.getLogger().debug("Lex: LINK","#"+E.yytext+"#"),15;break;case 94:return T.getLogger().debug("Lex: LINK",E.yytext),15;break;case 95:return T.getLogger().debug("Lex: LINK",E.yytext),15;break;case 96:return T.getLogger().debug("Lex: LINK",E.yytext),15;break;case 97:return T.getLogger().debug("Lex: START_LINK",E.yytext),this.pushState("LLABEL"),16;break;case 98:return T.getLogger().debug("Lex: START_LINK",E.yytext),this.pushState("LLABEL"),16;break;case 99:return T.getLogger().debug("Lex: START_LINK",E.yytext),this.pushState("LLABEL"),16;break;case 100:this.pushState("md_string");break;case 101:return T.getLogger().debug("Lex: Starting string"),this.pushState("string"),"LINK_LABEL";break;case 102:return this.popState(),T.getLogger().debug("Lex: LINK","#"+E.yytext+"#"),15;break;case 103:return this.popState(),T.getLogger().debug("Lex: LINK",E.yytext),15;break;case 104:return this.popState(),T.getLogger().debug("Lex: LINK",E.yytext),15;break;case 105:return T.getLogger().debug("Lex: COLON",E.yytext),E.yytext=E.yytext.slice(1),27;break}},"anonymous"),rules:[/^(?:block-beta\b)/,/^(?:block\s+)/,/^(?:block\n+)/,/^(?:block:)/,/^(?:[\s]+)/,/^(?:[\n]+)/,/^(?:((\u000D\u000A)|(\u000A)))/,/^(?:columns\s+auto\b)/,/^(?:columns\s+[\d]+)/,/^(?:["][`])/,/^(?:[^`"]+)/,/^(?:[`]["])/,/^(?:["])/,/^(?:["])/,/^(?:[^"]*)/,/^(?:space[:]\d+)/,/^(?:space\b)/,/^(?:default\b)/,/^(?:linkStyle\b)/,/^(?:interpolate\b)/,/^(?:classDef\s+)/,/^(?:DEFAULT\s+)/,/^(?:\w+\s+)/,/^(?:[^\n]*)/,/^(?:class\s+)/,/^(?:(\w+)+((,\s*\w+)*))/,/^(?:[^\n]*)/,/^(?:style\s+)/,/^(?:(\w+)+((,\s*\w+)*))/,/^(?:[^\n]*)/,/^(?:accTitle\s*:\s*)/,/^(?:(?!\n||)*[^\n]*)/,/^(?:accDescr\s*:\s*)/,/^(?:(?!\n||)*[^\n]*)/,/^(?:accDescr\s*\{\s*)/,/^(?:[\}])/,/^(?:[^\}]*)/,/^(?:end\b\s*)/,/^(?:\(\(\()/,/^(?:\)\)\))/,/^(?:[\)]\))/,/^(?:\}\})/,/^(?:\})/,/^(?:\(-)/,/^(?:-\))/,/^(?:\(\()/,/^(?:\]\])/,/^(?:\()/,/^(?:\]\))/,/^(?:\\\])/,/^(?:\/\])/,/^(?:\)\])/,/^(?:[\)])/,/^(?:\]>)/,/^(?:[\]])/,/^(?:-\))/,/^(?:\(-)/,/^(?:\)\))/,/^(?:\))/,/^(?:\(\(\()/,/^(?:\(\()/,/^(?:\{\{)/,/^(?:\{)/,/^(?:>)/,/^(?:\(\[)/,/^(?:\()/,/^(?:\[\[)/,/^(?:\[\|)/,/^(?:\[\()/,/^(?:\)\)\))/,/^(?:\[\\)/,/^(?:\[\/)/,/^(?:\[\\)/,/^(?:\[)/,/^(?:<\[)/,/^(?:[^\(\[\n\-\)\{\}\s\<\>:]+)/,/^(?:$)/,/^(?:["][`])/,/^(?:["][`])/,/^(?:[^`"]+)/,/^(?:[`]["])/,/^(?:["])/,/^(?:["])/,/^(?:[^"]+)/,/^(?:["])/,/^(?:\]>\s*\()/,/^(?:,?\s*right\s*)/,/^(?:,?\s*left\s*)/,/^(?:,?\s*x\s*)/,/^(?:,?\s*y\s*)/,/^(?:,?\s*up\s*)/,/^(?:,?\s*down\s*)/,/^(?:\)\s*)/,/^(?:\s*[xo<]?--+[-xo>]\s*)/,/^(?:\s*[xo<]?==+[=xo>]\s*)/,/^(?:\s*[xo<]?-?\.+-[xo>]?\s*)/,/^(?:\s*~~[\~]+\s*)/,/^(?:\s*[xo<]?--\s*)/,/^(?:\s*[xo<]?==\s*)/,/^(?:\s*[xo<]?-\.\s*)/,/^(?:["][`])/,/^(?:["])/,/^(?:\s*[xo<]?--+[-xo>]\s*)/,/^(?:\s*[xo<]?==+[=xo>]\s*)/,/^(?:\s*[xo<]?-?\.+-[xo>]?\s*)/,/^(?::\d+)/],conditions:{STYLE_DEFINITION:{rules:[29],inclusive:!1},STYLE_STMNT:{rules:[28],inclusive:!1},CLASSDEFID:{rules:[23],inclusive:!1},CLASSDEF:{rules:[21,22],inclusive:!1},CLASS_STYLE:{rules:[26],inclusive:!1},CLASS:{rules:[25],inclusive:!1},LLABEL:{rules:[100,101,102,103,104],inclusive:!1},ARROW_DIR:{rules:[86,87,88,89,90,91,92],inclusive:!1},BLOCK_ARROW:{rules:[77,82,85],inclusive:!1},NODE:{rules:[38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,78,81],inclusive:!1},md_string:{rules:[10,11,79,80],inclusive:!1},space:{rules:[],inclusive:!1},string:{rules:[13,14,83,84],inclusive:!1},acc_descr_multiline:{rules:[35,36],inclusive:!1},acc_descr:{rules:[33],inclusive:!1},acc_title:{rules:[31],inclusive:!1},INITIAL:{rules:[0,1,2,3,4,5,6,7,8,9,12,15,16,17,18,19,20,24,27,30,32,34,37,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,93,94,95,96,97,98,99,105],inclusive:!0}}};return w}();v.lexer=x;function b(){this.yy={}}return o(b,"Parser"),b.prototype=v,v.Parser=b,new b}();KB.parser=KB;bye=KB});function Yet(t){switch(Y.debug("typeStr2Type",t),t){case"[]":return"square";case"()":return Y.debug("we have a round"),"round";case"(())":return"circle";case">]":return"rect_left_inv_arrow";case"{}":return"diamond";case"{{}}":return"hexagon";case"([])":return"stadium";case"[[]]":return"subroutine";case"[()]":return"cylinder";case"((()))":return"doublecircle";case"[//]":return"lean_right";case"[\\\\]":return"lean_left";case"[/\\]":return"trapezoid";case"[\\/]":return"inv_trapezoid";case"<[]>":return"block_arrow";default:return"na"}}function Xet(t){switch(Y.debug("typeStr2Type",t),t){case"==":return"thick";default:return"normal"}}function jet(t){switch(t.trim()){case"--x":return"arrow_cross";case"--o":return"arrow_circle";default:return"arrow_point"}}var Ul,ZB,QB,Tye,kye,zet,Sye,Get,eC,Vet,Uet,Het,Wet,Cye,JB,y4,qet,Eye,Ket,Qet,Zet,Jet,ett,ttt,rtt,ntt,itt,att,stt,Aye,_ye=N(()=>{"use strict";gL();ji();zt();vt();gr();mi();Ul=new Map,ZB=[],QB=new Map,Tye="color",kye="fill",zet="bgFill",Sye=",",Get=me(),eC=new Map,Vet=o(t=>Ze.sanitizeText(t,Get),"sanitizeText"),Uet=o(function(t,e=""){let r=eC.get(t);r||(r={id:t,styles:[],textStyles:[]},eC.set(t,r)),e?.split(Sye).forEach(n=>{let i=n.replace(/([^;]*);/,"$1").trim();if(RegExp(Tye).exec(n)){let s=i.replace(kye,zet).replace(Tye,kye);r.textStyles.push(s)}r.styles.push(i)})},"addStyleClass"),Het=o(function(t,e=""){let r=Ul.get(t);e!=null&&(r.styles=e.split(Sye))},"addStyle2Node"),Wet=o(function(t,e){t.split(",").forEach(function(r){let n=Ul.get(r);if(n===void 0){let i=r.trim();n={id:i,type:"na",children:[]},Ul.set(i,n)}n.classes||(n.classes=[]),n.classes.push(e)})},"setCssClass"),Cye=o((t,e)=>{let r=t.flat(),n=[];for(let i of r){if(i.label&&(i.label=Vet(i.label)),i.type==="classDef"){Uet(i.id,i.css);continue}if(i.type==="applyClass"){Wet(i.id,i?.styleClass??"");continue}if(i.type==="applyStyles"){i?.stylesStr&&Het(i.id,i?.stylesStr);continue}if(i.type==="column-setting")e.columns=i.columns??-1;else if(i.type==="edge"){let a=(QB.get(i.id)??0)+1;QB.set(i.id,a),i.id=a+"-"+i.id,ZB.push(i)}else{i.label||(i.type==="composite"?i.label="":i.label=i.id);let a=Ul.get(i.id);if(a===void 0?Ul.set(i.id,i):(i.type!=="na"&&(a.type=i.type),i.label!==i.id&&(a.label=i.label)),i.children&&Cye(i.children,i),i.type==="space"){let s=i.width??1;for(let l=0;l{Y.debug("Clear called"),Ar(),y4={id:"root",type:"composite",children:[],columns:-1},Ul=new Map([["root",y4]]),JB=[],eC=new Map,ZB=[],QB=new Map},"clear");o(Yet,"typeStr2Type");o(Xet,"edgeTypeStr2Type");o(jet,"edgeStrToEdgeData");Eye=0,Ket=o(()=>(Eye++,"id-"+Math.random().toString(36).substr(2,12)+"-"+Eye),"generateId"),Qet=o(t=>{y4.children=t,Cye(t,y4),JB=y4.children},"setHierarchy"),Zet=o(t=>{let e=Ul.get(t);return e?e.columns?e.columns:e.children?e.children.length:-1:-1},"getColumns"),Jet=o(()=>[...Ul.values()],"getBlocksFlat"),ett=o(()=>JB||[],"getBlocks"),ttt=o(()=>ZB,"getEdges"),rtt=o(t=>Ul.get(t),"getBlock"),ntt=o(t=>{Ul.set(t.id,t)},"setBlock"),itt=o(()=>console,"getLogger"),att=o(function(){return eC},"getClasses"),stt={getConfig:o(()=>cr().block,"getConfig"),typeStr2Type:Yet,edgeTypeStr2Type:Xet,edgeStrToEdgeData:jet,getLogger:itt,getBlocksFlat:Jet,getBlocks:ett,getEdges:ttt,setHierarchy:Qet,getBlock:rtt,setBlock:ntt,getColumns:Zet,getClasses:att,clear:qet,generateId:Ket},Aye=stt});var tC,ott,Dye,Lye=N(()=>{"use strict";Ys();tC=o((t,e)=>{let r=Kf,n=r(t,"r"),i=r(t,"g"),a=r(t,"b");return qa(n,i,a,e)},"fade"),ott=o(t=>`.label { + font-family: ${t.fontFamily}; + color: ${t.nodeTextColor||t.textColor}; + } + .cluster-label text { + fill: ${t.titleColor}; + } + .cluster-label span,p { + color: ${t.titleColor}; + } + + + + .label text,span,p { + fill: ${t.nodeTextColor||t.textColor}; + color: ${t.nodeTextColor||t.textColor}; + } + + .node rect, + .node circle, + .node ellipse, + .node polygon, + .node path { + fill: ${t.mainBkg}; + stroke: ${t.nodeBorder}; + stroke-width: 1px; + } + .flowchart-label text { + text-anchor: middle; + } + // .flowchart-label .text-outer-tspan { + // text-anchor: middle; + // } + // .flowchart-label .text-inner-tspan { + // text-anchor: start; + // } + + .node .label { + text-align: center; + } + .node.clickable { + cursor: pointer; + } + + .arrowheadPath { + fill: ${t.arrowheadColor}; + } + + .edgePath .path { + stroke: ${t.lineColor}; + stroke-width: 2.0px; + } + + .flowchart-link { + stroke: ${t.lineColor}; + fill: none; + } + + .edgeLabel { + background-color: ${t.edgeLabelBackground}; + rect { + opacity: 0.5; + background-color: ${t.edgeLabelBackground}; + fill: ${t.edgeLabelBackground}; + } + text-align: center; + } + + /* For html labels only */ + .labelBkg { + background-color: ${tC(t.edgeLabelBackground,.5)}; + // background-color: + } + + .node .cluster { + // fill: ${tC(t.mainBkg,.5)}; + fill: ${tC(t.clusterBkg,.5)}; + stroke: ${tC(t.clusterBorder,.2)}; + box-shadow: rgba(50, 50, 93, 0.25) 0px 13px 27px -5px, rgba(0, 0, 0, 0.3) 0px 8px 16px -8px; + stroke-width: 1px; + } + + .cluster text { + fill: ${t.titleColor}; + } + + .cluster span,p { + color: ${t.titleColor}; + } + /* .cluster div { + color: ${t.titleColor}; + } */ + + div.mermaidTooltip { + position: absolute; + text-align: center; + max-width: 200px; + padding: 2px; + font-family: ${t.fontFamily}; + font-size: 12px; + background: ${t.tertiaryColor}; + border: 1px solid ${t.border2}; + border-radius: 2px; + pointer-events: none; + z-index: 100; + } + + .flowchartTitleText { + text-anchor: middle; + font-size: 18px; + fill: ${t.textColor}; + } +`,"getStyles"),Dye=ott});var ltt,ctt,utt,htt,ftt,dtt,ptt,mtt,gtt,ytt,vtt,Rye,Nye=N(()=>{"use strict";vt();ltt=o((t,e,r,n)=>{e.forEach(i=>{vtt[i](t,r,n)})},"insertMarkers"),ctt=o((t,e,r)=>{Y.trace("Making markers for ",r),t.append("defs").append("marker").attr("id",r+"_"+e+"-extensionStart").attr("class","marker extension "+e).attr("refX",18).attr("refY",7).attr("markerWidth",190).attr("markerHeight",240).attr("orient","auto").append("path").attr("d","M 1,7 L18,13 V 1 Z"),t.append("defs").append("marker").attr("id",r+"_"+e+"-extensionEnd").attr("class","marker extension "+e).attr("refX",1).attr("refY",7).attr("markerWidth",20).attr("markerHeight",28).attr("orient","auto").append("path").attr("d","M 1,1 V 13 L18,7 Z")},"extension"),utt=o((t,e,r)=>{t.append("defs").append("marker").attr("id",r+"_"+e+"-compositionStart").attr("class","marker composition "+e).attr("refX",18).attr("refY",7).attr("markerWidth",190).attr("markerHeight",240).attr("orient","auto").append("path").attr("d","M 18,7 L9,13 L1,7 L9,1 Z"),t.append("defs").append("marker").attr("id",r+"_"+e+"-compositionEnd").attr("class","marker composition "+e).attr("refX",1).attr("refY",7).attr("markerWidth",20).attr("markerHeight",28).attr("orient","auto").append("path").attr("d","M 18,7 L9,13 L1,7 L9,1 Z")},"composition"),htt=o((t,e,r)=>{t.append("defs").append("marker").attr("id",r+"_"+e+"-aggregationStart").attr("class","marker aggregation "+e).attr("refX",18).attr("refY",7).attr("markerWidth",190).attr("markerHeight",240).attr("orient","auto").append("path").attr("d","M 18,7 L9,13 L1,7 L9,1 Z"),t.append("defs").append("marker").attr("id",r+"_"+e+"-aggregationEnd").attr("class","marker aggregation "+e).attr("refX",1).attr("refY",7).attr("markerWidth",20).attr("markerHeight",28).attr("orient","auto").append("path").attr("d","M 18,7 L9,13 L1,7 L9,1 Z")},"aggregation"),ftt=o((t,e,r)=>{t.append("defs").append("marker").attr("id",r+"_"+e+"-dependencyStart").attr("class","marker dependency "+e).attr("refX",6).attr("refY",7).attr("markerWidth",190).attr("markerHeight",240).attr("orient","auto").append("path").attr("d","M 5,7 L9,13 L1,7 L9,1 Z"),t.append("defs").append("marker").attr("id",r+"_"+e+"-dependencyEnd").attr("class","marker dependency "+e).attr("refX",13).attr("refY",7).attr("markerWidth",20).attr("markerHeight",28).attr("orient","auto").append("path").attr("d","M 18,7 L9,13 L14,7 L9,1 Z")},"dependency"),dtt=o((t,e,r)=>{t.append("defs").append("marker").attr("id",r+"_"+e+"-lollipopStart").attr("class","marker lollipop "+e).attr("refX",13).attr("refY",7).attr("markerWidth",190).attr("markerHeight",240).attr("orient","auto").append("circle").attr("stroke","black").attr("fill","transparent").attr("cx",7).attr("cy",7).attr("r",6),t.append("defs").append("marker").attr("id",r+"_"+e+"-lollipopEnd").attr("class","marker lollipop "+e).attr("refX",1).attr("refY",7).attr("markerWidth",190).attr("markerHeight",240).attr("orient","auto").append("circle").attr("stroke","black").attr("fill","transparent").attr("cx",7).attr("cy",7).attr("r",6)},"lollipop"),ptt=o((t,e,r)=>{t.append("marker").attr("id",r+"_"+e+"-pointEnd").attr("class","marker "+e).attr("viewBox","0 0 10 10").attr("refX",6).attr("refY",5).attr("markerUnits","userSpaceOnUse").attr("markerWidth",12).attr("markerHeight",12).attr("orient","auto").append("path").attr("d","M 0 0 L 10 5 L 0 10 z").attr("class","arrowMarkerPath").style("stroke-width",1).style("stroke-dasharray","1,0"),t.append("marker").attr("id",r+"_"+e+"-pointStart").attr("class","marker "+e).attr("viewBox","0 0 10 10").attr("refX",4.5).attr("refY",5).attr("markerUnits","userSpaceOnUse").attr("markerWidth",12).attr("markerHeight",12).attr("orient","auto").append("path").attr("d","M 0 5 L 10 10 L 10 0 z").attr("class","arrowMarkerPath").style("stroke-width",1).style("stroke-dasharray","1,0")},"point"),mtt=o((t,e,r)=>{t.append("marker").attr("id",r+"_"+e+"-circleEnd").attr("class","marker "+e).attr("viewBox","0 0 10 10").attr("refX",11).attr("refY",5).attr("markerUnits","userSpaceOnUse").attr("markerWidth",11).attr("markerHeight",11).attr("orient","auto").append("circle").attr("cx","5").attr("cy","5").attr("r","5").attr("class","arrowMarkerPath").style("stroke-width",1).style("stroke-dasharray","1,0"),t.append("marker").attr("id",r+"_"+e+"-circleStart").attr("class","marker "+e).attr("viewBox","0 0 10 10").attr("refX",-1).attr("refY",5).attr("markerUnits","userSpaceOnUse").attr("markerWidth",11).attr("markerHeight",11).attr("orient","auto").append("circle").attr("cx","5").attr("cy","5").attr("r","5").attr("class","arrowMarkerPath").style("stroke-width",1).style("stroke-dasharray","1,0")},"circle"),gtt=o((t,e,r)=>{t.append("marker").attr("id",r+"_"+e+"-crossEnd").attr("class","marker cross "+e).attr("viewBox","0 0 11 11").attr("refX",12).attr("refY",5.2).attr("markerUnits","userSpaceOnUse").attr("markerWidth",11).attr("markerHeight",11).attr("orient","auto").append("path").attr("d","M 1,1 l 9,9 M 10,1 l -9,9").attr("class","arrowMarkerPath").style("stroke-width",2).style("stroke-dasharray","1,0"),t.append("marker").attr("id",r+"_"+e+"-crossStart").attr("class","marker cross "+e).attr("viewBox","0 0 11 11").attr("refX",-1).attr("refY",5.2).attr("markerUnits","userSpaceOnUse").attr("markerWidth",11).attr("markerHeight",11).attr("orient","auto").append("path").attr("d","M 1,1 l 9,9 M 10,1 l -9,9").attr("class","arrowMarkerPath").style("stroke-width",2).style("stroke-dasharray","1,0")},"cross"),ytt=o((t,e,r)=>{t.append("defs").append("marker").attr("id",r+"_"+e+"-barbEnd").attr("refX",19).attr("refY",7).attr("markerWidth",20).attr("markerHeight",14).attr("markerUnits","strokeWidth").attr("orient","auto").append("path").attr("d","M 19,7 L9,13 L14,7 L9,1 Z")},"barb"),vtt={extension:ctt,composition:utt,aggregation:htt,dependency:ftt,lollipop:dtt,point:ptt,circle:mtt,cross:gtt,barb:ytt},Rye=ltt});function xtt(t,e){if(t===0||!Number.isInteger(t))throw new Error("Columns must be an integer !== 0.");if(e<0||!Number.isInteger(e))throw new Error("Position must be a non-negative integer."+e);if(t<0)return{px:e,py:0};if(t===1)return{px:0,py:e};let r=e%t,n=Math.floor(e/t);return{px:r,py:n}}function eF(t,e,r=0,n=0){Y.debug("setBlockSizes abc95 (start)",t.id,t?.size?.x,"block width =",t?.size,"sieblingWidth",r),t?.size?.width||(t.size={width:r,height:n,x:0,y:0});let i=0,a=0;if(t.children?.length>0){for(let m of t.children)eF(m,e);let s=btt(t);i=s.width,a=s.height,Y.debug("setBlockSizes abc95 maxWidth of",t.id,":s children is ",i,a);for(let m of t.children)m.size&&(Y.debug(`abc95 Setting size of children of ${t.id} id=${m.id} ${i} ${a} ${JSON.stringify(m.size)}`),m.size.width=i*(m.widthInColumns??1)+bi*((m.widthInColumns??1)-1),m.size.height=a,m.size.x=0,m.size.y=0,Y.debug(`abc95 updating size of ${t.id} children child:${m.id} maxWidth:${i} maxHeight:${a}`));for(let m of t.children)eF(m,e,i,a);let l=t.columns??-1,u=0;for(let m of t.children)u+=m.widthInColumns??1;let h=t.children.length;l>0&&l0?Math.min(t.children.length,l):t.children.length;if(m>0){let g=(d-m*bi-bi)/m;Y.debug("abc95 (growing to fit) width",t.id,d,t.size?.width,g);for(let y of t.children)y.size&&(y.size.width=g)}}t.size={width:d,height:p,x:0,y:0}}Y.debug("setBlockSizes abc94 (done)",t.id,t?.size?.x,t?.size?.width,t?.size?.y,t?.size?.height)}function Mye(t,e){Y.debug(`abc85 layout blocks (=>layoutBlocks) ${t.id} x: ${t?.size?.x} y: ${t?.size?.y} width: ${t?.size?.width}`);let r=t.columns??-1;if(Y.debug("layoutBlocks columns abc95",t.id,"=>",r,t),t.children&&t.children.length>0){let n=t?.children[0]?.size?.width??0,i=t.children.length*n+(t.children.length-1)*bi;Y.debug("widthOfChildren 88",i,"posX");let a=0;Y.debug("abc91 block?.size?.x",t.id,t?.size?.x);let s=t?.size?.x?t?.size?.x+(-t?.size?.width/2||0):-bi,l=0;for(let u of t.children){let h=t;if(!u.size)continue;let{width:f,height:d}=u.size,{px:p,py:m}=xtt(r,a);if(m!=l&&(l=m,s=t?.size?.x?t?.size?.x+(-t?.size?.width/2||0):-bi,Y.debug("New row in layout for block",t.id," and child ",u.id,l)),Y.debug(`abc89 layout blocks (child) id: ${u.id} Pos: ${a} (px, py) ${p},${m} (${h?.size?.x},${h?.size?.y}) parent: ${h.id} width: ${f}${bi}`),h.size){let g=f/2;u.size.x=s+bi+g,Y.debug(`abc91 layout blocks (calc) px, pyid:${u.id} startingPos=X${s} new startingPosX${u.size.x} ${g} padding=${bi} width=${f} halfWidth=${g} => x:${u.size.x} y:${u.size.y} ${u.widthInColumns} (width * (child?.w || 1)) / 2 ${f*(u?.widthInColumns??1)/2}`),s=u.size.x+g,u.size.y=h.size.y-h.size.height/2+m*(d+bi)+d/2+bi,Y.debug(`abc88 layout blocks (calc) px, pyid:${u.id}startingPosX${s}${bi}${g}=>x:${u.size.x}y:${u.size.y}${u.widthInColumns}(width * (child?.w || 1)) / 2${f*(u?.widthInColumns??1)/2}`)}u.children&&Mye(u,e),a+=u?.widthInColumns??1,Y.debug("abc88 columnsPos",u,a)}}Y.debug(`layout blocks (<==layoutBlocks) ${t.id} x: ${t?.size?.x} y: ${t?.size?.y} width: ${t?.size?.width}`)}function Iye(t,{minX:e,minY:r,maxX:n,maxY:i}={minX:0,minY:0,maxX:0,maxY:0}){if(t.size&&t.id!=="root"){let{x:a,y:s,width:l,height:u}=t.size;a-l/2n&&(n=a+l/2),s+u/2>i&&(i=s+u/2)}if(t.children)for(let a of t.children)({minX:e,minY:r,maxX:n,maxY:i}=Iye(a,{minX:e,minY:r,maxX:n,maxY:i}));return{minX:e,minY:r,maxX:n,maxY:i}}function Oye(t){let e=t.getBlock("root");if(!e)return;eF(e,t,0,0),Mye(e,t),Y.debug("getBlocks",JSON.stringify(e,null,2));let{minX:r,minY:n,maxX:i,maxY:a}=Iye(e),s=a-n,l=i-r;return{x:r,y:n,width:l,height:s}}var bi,btt,Pye=N(()=>{"use strict";vt();zt();bi=me()?.block?.padding??8;o(xtt,"calculateBlockPosition");btt=o(t=>{let e=0,r=0;for(let n of t.children){let{width:i,height:a,x:s,y:l}=n.size??{width:0,height:0,x:0,y:0};Y.debug("getMaxChildSize abc95 child:",n.id,"width:",i,"height:",a,"x:",s,"y:",l,n.type),n.type!=="space"&&(i>e&&(e=i/(t.widthInColumns??1)),a>r&&(r=a))}return{width:e,height:r}},"getMaxChildSize");o(eF,"setBlockSizes");o(Mye,"layoutBlocks");o(Iye,"findBounds");o(Oye,"layout")});function Bye(t,e){e&&t.attr("style",e)}function wtt(t){let e=Ge(document.createElementNS("http://www.w3.org/2000/svg","foreignObject")),r=e.append("xhtml:div"),n=t.label,i=t.isNode?"nodeLabel":"edgeLabel",a=r.append("span");return a.html(n),Bye(a,t.labelStyle),a.attr("class",i),Bye(r,t.labelStyle),r.style("display","inline-block"),r.style("white-space","nowrap"),r.attr("xmlns","http://www.w3.org/1999/xhtml"),e.node()}var Ttt,vs,rC=N(()=>{"use strict";dr();vt();zt();gr();ir();to();o(Bye,"applyStyle");o(wtt,"addHtmlLabel");Ttt=o((t,e,r,n)=>{let i=t||"";if(typeof i=="object"&&(i=i[0]),fr(me().flowchart.htmlLabels)){i=i.replace(/\\n|\n/g,"
    "),Y.debug("vertexText"+i);let a={isNode:n,label:DD(na(i)),labelStyle:e.replace("fill:","color:")};return wtt(a)}else{let a=document.createElementNS("http://www.w3.org/2000/svg","text");a.setAttribute("style",e.replace("color:","fill:"));let s=[];typeof i=="string"?s=i.split(/\\n|\n|/gi):Array.isArray(i)?s=i:s=[];for(let l of s){let u=document.createElementNS("http://www.w3.org/2000/svg","tspan");u.setAttributeNS("http://www.w3.org/XML/1998/namespace","xml:space","preserve"),u.setAttribute("dy","1em"),u.setAttribute("x","0"),r?u.setAttribute("class","title-row"):u.setAttribute("class","row"),u.textContent=l.trim(),a.appendChild(u)}return a}},"createLabel"),vs=Ttt});var $ye,ktt,Fye,zye=N(()=>{"use strict";vt();$ye=o((t,e,r,n,i)=>{e.arrowTypeStart&&Fye(t,"start",e.arrowTypeStart,r,n,i),e.arrowTypeEnd&&Fye(t,"end",e.arrowTypeEnd,r,n,i)},"addEdgeMarkers"),ktt={arrow_cross:"cross",arrow_point:"point",arrow_barb:"barb",arrow_circle:"circle",aggregation:"aggregation",extension:"extension",composition:"composition",dependency:"dependency",lollipop:"lollipop"},Fye=o((t,e,r,n,i,a)=>{let s=ktt[r];if(!s){Y.warn(`Unknown arrow type: ${r}`);return}let l=e==="start"?"Start":"End";t.attr(`marker-${e}`,`url(${n}#${i}_${a}-${s}${l})`)},"addEdgeMarker")});function nC(t,e){me().flowchart.htmlLabels&&t&&(t.style.width=e.length*9+"px",t.style.height="12px")}var tF,Ua,Vye,Uye,Ett,Stt,Gye,Hye,Wye=N(()=>{"use strict";vt();rC();to();dr();zt();ir();gr();JD();w2();zye();tF={},Ua={},Vye=o((t,e)=>{let r=me(),n=fr(r.flowchart.htmlLabels),i=e.labelType==="markdown"?Hn(t,e.label,{style:e.labelStyle,useHtmlLabels:n,addSvgBackground:!0},r):vs(e.label,e.labelStyle),a=t.insert("g").attr("class","edgeLabel"),s=a.insert("g").attr("class","label");s.node().appendChild(i);let l=i.getBBox();if(n){let h=i.children[0],f=Ge(i);l=h.getBoundingClientRect(),f.attr("width",l.width),f.attr("height",l.height)}s.attr("transform","translate("+-l.width/2+", "+-l.height/2+")"),tF[e.id]=a,e.width=l.width,e.height=l.height;let u;if(e.startLabelLeft){let h=vs(e.startLabelLeft,e.labelStyle),f=t.insert("g").attr("class","edgeTerminals"),d=f.insert("g").attr("class","inner");u=d.node().appendChild(h);let p=h.getBBox();d.attr("transform","translate("+-p.width/2+", "+-p.height/2+")"),Ua[e.id]||(Ua[e.id]={}),Ua[e.id].startLeft=f,nC(u,e.startLabelLeft)}if(e.startLabelRight){let h=vs(e.startLabelRight,e.labelStyle),f=t.insert("g").attr("class","edgeTerminals"),d=f.insert("g").attr("class","inner");u=f.node().appendChild(h),d.node().appendChild(h);let p=h.getBBox();d.attr("transform","translate("+-p.width/2+", "+-p.height/2+")"),Ua[e.id]||(Ua[e.id]={}),Ua[e.id].startRight=f,nC(u,e.startLabelRight)}if(e.endLabelLeft){let h=vs(e.endLabelLeft,e.labelStyle),f=t.insert("g").attr("class","edgeTerminals"),d=f.insert("g").attr("class","inner");u=d.node().appendChild(h);let p=h.getBBox();d.attr("transform","translate("+-p.width/2+", "+-p.height/2+")"),f.node().appendChild(h),Ua[e.id]||(Ua[e.id]={}),Ua[e.id].endLeft=f,nC(u,e.endLabelLeft)}if(e.endLabelRight){let h=vs(e.endLabelRight,e.labelStyle),f=t.insert("g").attr("class","edgeTerminals"),d=f.insert("g").attr("class","inner");u=d.node().appendChild(h);let p=h.getBBox();d.attr("transform","translate("+-p.width/2+", "+-p.height/2+")"),f.node().appendChild(h),Ua[e.id]||(Ua[e.id]={}),Ua[e.id].endRight=f,nC(u,e.endLabelRight)}return i},"insertEdgeLabel");o(nC,"setTerminalWidth");Uye=o((t,e)=>{Y.debug("Moving label abc88 ",t.id,t.label,tF[t.id],e);let r=e.updatedPath?e.updatedPath:e.originalPath,n=me(),{subGraphTitleTotalMargin:i}=Ru(n);if(t.label){let a=tF[t.id],s=t.x,l=t.y;if(r){let u=Gt.calcLabelPosition(r);Y.debug("Moving label "+t.label+" from (",s,",",l,") to (",u.x,",",u.y,") abc88"),e.updatedPath&&(s=u.x,l=u.y)}a.attr("transform",`translate(${s}, ${l+i/2})`)}if(t.startLabelLeft){let a=Ua[t.id].startLeft,s=t.x,l=t.y;if(r){let u=Gt.calcTerminalLabelPosition(t.arrowTypeStart?10:0,"start_left",r);s=u.x,l=u.y}a.attr("transform",`translate(${s}, ${l})`)}if(t.startLabelRight){let a=Ua[t.id].startRight,s=t.x,l=t.y;if(r){let u=Gt.calcTerminalLabelPosition(t.arrowTypeStart?10:0,"start_right",r);s=u.x,l=u.y}a.attr("transform",`translate(${s}, ${l})`)}if(t.endLabelLeft){let a=Ua[t.id].endLeft,s=t.x,l=t.y;if(r){let u=Gt.calcTerminalLabelPosition(t.arrowTypeEnd?10:0,"end_left",r);s=u.x,l=u.y}a.attr("transform",`translate(${s}, ${l})`)}if(t.endLabelRight){let a=Ua[t.id].endRight,s=t.x,l=t.y;if(r){let u=Gt.calcTerminalLabelPosition(t.arrowTypeEnd?10:0,"end_right",r);s=u.x,l=u.y}a.attr("transform",`translate(${s}, ${l})`)}},"positionEdgeLabel"),Ett=o((t,e)=>{let r=t.x,n=t.y,i=Math.abs(e.x-r),a=Math.abs(e.y-n),s=t.width/2,l=t.height/2;return i>=s||a>=l},"outsideNode"),Stt=o((t,e,r)=>{Y.debug(`intersection calc abc89: + outsidePoint: ${JSON.stringify(e)} + insidePoint : ${JSON.stringify(r)} + node : x:${t.x} y:${t.y} w:${t.width} h:${t.height}`);let n=t.x,i=t.y,a=Math.abs(n-r.x),s=t.width/2,l=r.xMath.abs(n-e.x)*u){let d=r.y{Y.debug("abc88 cutPathAtIntersect",t,e);let r=[],n=t[0],i=!1;return t.forEach(a=>{if(!Ett(e,a)&&!i){let s=Stt(e,n,a),l=!1;r.forEach(u=>{l=l||u.x===s.x&&u.y===s.y}),r.some(u=>u.x===s.x&&u.y===s.y)||r.push(s),i=!0}else n=a,i||r.push(a)}),r},"cutPathAtIntersect"),Hye=o(function(t,e,r,n,i,a,s){let l=r.points;Y.debug("abc88 InsertEdge: edge=",r,"e=",e);let u=!1,h=a.node(e.v);var f=a.node(e.w);f?.intersect&&h?.intersect&&(l=l.slice(1,r.points.length-1),l.unshift(h.intersect(l[0])),l.push(f.intersect(l[l.length-1]))),r.toCluster&&(Y.debug("to cluster abc88",n[r.toCluster]),l=Gye(r.points,n[r.toCluster].node),u=!0),r.fromCluster&&(Y.debug("from cluster abc88",n[r.fromCluster]),l=Gye(l.reverse(),n[r.fromCluster].node).reverse(),u=!0);let d=l.filter(C=>!Number.isNaN(C.y)),p=Do;r.curve&&(i==="graph"||i==="flowchart")&&(p=r.curve);let{x:m,y:g}=qw(r),y=wl().x(m).y(g).curve(p),v;switch(r.thickness){case"normal":v="edge-thickness-normal";break;case"thick":v="edge-thickness-thick";break;case"invisible":v="edge-thickness-thick";break;default:v=""}switch(r.pattern){case"solid":v+=" edge-pattern-solid";break;case"dotted":v+=" edge-pattern-dotted";break;case"dashed":v+=" edge-pattern-dashed";break}let x=t.append("path").attr("d",y(d)).attr("id",r.id).attr("class"," "+v+(r.classes?" "+r.classes:"")).attr("style",r.style),b="";(me().flowchart.arrowMarkerAbsolute||me().state.arrowMarkerAbsolute)&&(b=window.location.protocol+"//"+window.location.host+window.location.pathname+window.location.search,b=b.replace(/\(/g,"\\("),b=b.replace(/\)/g,"\\)")),$ye(x,r,b,s,i);let w={};return u&&(w.updatedPath=l),w.originalPath=r.points,w},"insertEdge")});var Ctt,qye,Yye=N(()=>{"use strict";Ctt=o(t=>{let e=new Set;for(let r of t)switch(r){case"x":e.add("right"),e.add("left");break;case"y":e.add("up"),e.add("down");break;default:e.add(r);break}return e},"expandAndDeduplicateDirections"),qye=o((t,e,r)=>{let n=Ctt(t),i=2,a=e.height+2*r.padding,s=a/i,l=e.width+2*s+r.padding,u=r.padding/2;return n.has("right")&&n.has("left")&&n.has("up")&&n.has("down")?[{x:0,y:0},{x:s,y:0},{x:l/2,y:2*u},{x:l-s,y:0},{x:l,y:0},{x:l,y:-a/3},{x:l+2*u,y:-a/2},{x:l,y:-2*a/3},{x:l,y:-a},{x:l-s,y:-a},{x:l/2,y:-a-2*u},{x:s,y:-a},{x:0,y:-a},{x:0,y:-2*a/3},{x:-2*u,y:-a/2},{x:0,y:-a/3}]:n.has("right")&&n.has("left")&&n.has("up")?[{x:s,y:0},{x:l-s,y:0},{x:l,y:-a/2},{x:l-s,y:-a},{x:s,y:-a},{x:0,y:-a/2}]:n.has("right")&&n.has("left")&&n.has("down")?[{x:0,y:0},{x:s,y:-a},{x:l-s,y:-a},{x:l,y:0}]:n.has("right")&&n.has("up")&&n.has("down")?[{x:0,y:0},{x:l,y:-s},{x:l,y:-a+s},{x:0,y:-a}]:n.has("left")&&n.has("up")&&n.has("down")?[{x:l,y:0},{x:0,y:-s},{x:0,y:-a+s},{x:l,y:-a}]:n.has("right")&&n.has("left")?[{x:s,y:0},{x:s,y:-u},{x:l-s,y:-u},{x:l-s,y:0},{x:l,y:-a/2},{x:l-s,y:-a},{x:l-s,y:-a+u},{x:s,y:-a+u},{x:s,y:-a},{x:0,y:-a/2}]:n.has("up")&&n.has("down")?[{x:l/2,y:0},{x:0,y:-u},{x:s,y:-u},{x:s,y:-a+u},{x:0,y:-a+u},{x:l/2,y:-a},{x:l,y:-a+u},{x:l-s,y:-a+u},{x:l-s,y:-u},{x:l,y:-u}]:n.has("right")&&n.has("up")?[{x:0,y:0},{x:l,y:-s},{x:0,y:-a}]:n.has("right")&&n.has("down")?[{x:0,y:0},{x:l,y:0},{x:0,y:-a}]:n.has("left")&&n.has("up")?[{x:l,y:0},{x:0,y:-s},{x:l,y:-a}]:n.has("left")&&n.has("down")?[{x:l,y:0},{x:0,y:0},{x:l,y:-a}]:n.has("right")?[{x:s,y:-u},{x:s,y:-u},{x:l-s,y:-u},{x:l-s,y:0},{x:l,y:-a/2},{x:l-s,y:-a},{x:l-s,y:-a+u},{x:s,y:-a+u},{x:s,y:-a+u}]:n.has("left")?[{x:s,y:0},{x:s,y:-u},{x:l-s,y:-u},{x:l-s,y:-a+u},{x:s,y:-a+u},{x:s,y:-a},{x:0,y:-a/2}]:n.has("up")?[{x:s,y:-u},{x:s,y:-a+u},{x:0,y:-a+u},{x:l/2,y:-a},{x:l,y:-a+u},{x:l-s,y:-a+u},{x:l-s,y:-u}]:n.has("down")?[{x:l/2,y:0},{x:0,y:-u},{x:s,y:-u},{x:s,y:-a+u},{x:l-s,y:-a+u},{x:l-s,y:-u},{x:l,y:-u}]:[{x:0,y:0}]},"getArrowPoints")});function Att(t,e){return t.intersect(e)}var Xye,jye=N(()=>{"use strict";o(Att,"intersectNode");Xye=Att});function _tt(t,e,r,n){var i=t.x,a=t.y,s=i-n.x,l=a-n.y,u=Math.sqrt(e*e*l*l+r*r*s*s),h=Math.abs(e*r*s/u);n.x{"use strict";o(_tt,"intersectEllipse");iC=_tt});function Dtt(t,e,r){return iC(t,e,e,r)}var Kye,Qye=N(()=>{"use strict";rF();o(Dtt,"intersectCircle");Kye=Dtt});function Ltt(t,e,r,n){var i,a,s,l,u,h,f,d,p,m,g,y,v,x,b;if(i=e.y-t.y,s=t.x-e.x,u=e.x*t.y-t.x*e.y,p=i*r.x+s*r.y+u,m=i*n.x+s*n.y+u,!(p!==0&&m!==0&&Zye(p,m))&&(a=n.y-r.y,l=r.x-n.x,h=n.x*r.y-r.x*n.y,f=a*t.x+l*t.y+h,d=a*e.x+l*e.y+h,!(f!==0&&d!==0&&Zye(f,d))&&(g=i*l-a*s,g!==0)))return y=Math.abs(g/2),v=s*h-l*u,x=v<0?(v-y)/g:(v+y)/g,v=a*u-i*h,b=v<0?(v-y)/g:(v+y)/g,{x,y:b}}function Zye(t,e){return t*e>0}var Jye,eve=N(()=>{"use strict";o(Ltt,"intersectLine");o(Zye,"sameSign");Jye=Ltt});function Rtt(t,e,r){var n=t.x,i=t.y,a=[],s=Number.POSITIVE_INFINITY,l=Number.POSITIVE_INFINITY;typeof e.forEach=="function"?e.forEach(function(g){s=Math.min(s,g.x),l=Math.min(l,g.y)}):(s=Math.min(s,e.x),l=Math.min(l,e.y));for(var u=n-t.width/2-s,h=i-t.height/2-l,f=0;f1&&a.sort(function(g,y){var v=g.x-r.x,x=g.y-r.y,b=Math.sqrt(v*v+x*x),w=y.x-r.x,C=y.y-r.y,T=Math.sqrt(w*w+C*C);return b{"use strict";eve();tve=Rtt;o(Rtt,"intersectPolygon")});var Ntt,nve,ive=N(()=>{"use strict";Ntt=o((t,e)=>{var r=t.x,n=t.y,i=e.x-r,a=e.y-n,s=t.width/2,l=t.height/2,u,h;return Math.abs(a)*s>Math.abs(i)*l?(a<0&&(l=-l),u=a===0?0:l*i/a,h=l):(i<0&&(s=-s),u=s,h=i===0?0:s*a/i),{x:r+u,y:n+h}},"intersectRect"),nve=Ntt});var In,nF=N(()=>{"use strict";jye();Qye();rF();rve();ive();In={node:Xye,circle:Kye,ellipse:iC,polygon:tve,rect:nve}});function Hl(t,e,r,n){return t.insert("polygon",":first-child").attr("points",n.map(function(i){return i.x+","+i.y}).join(" ")).attr("class","label-container").attr("transform","translate("+-e/2+","+r/2+")")}var Di,Qn,iF=N(()=>{"use strict";rC();to();zt();dr();gr();ir();Di=o(async(t,e,r,n)=>{let i=me(),a,s=e.useHtmlLabels||fr(i.flowchart.htmlLabels);r?a=r:a="node default";let l=t.insert("g").attr("class",a).attr("id",e.domId||e.id),u=l.insert("g").attr("class","label").attr("style",e.labelStyle),h;e.labelText===void 0?h="":h=typeof e.labelText=="string"?e.labelText:e.labelText[0];let f=u.node(),d;e.labelType==="markdown"?d=Hn(u,Tr(na(h),i),{useHtmlLabels:s,width:e.width||i.flowchart.wrappingWidth,classes:"markdown-node-label"},i):d=f.appendChild(vs(Tr(na(h),i),e.labelStyle,!1,n));let p=d.getBBox(),m=e.padding/2;if(fr(i.flowchart.htmlLabels)){let g=d.children[0],y=Ge(d),v=g.getElementsByTagName("img");if(v){let x=h.replace(/]*>/g,"").trim()==="";await Promise.all([...v].map(b=>new Promise(w=>{function C(){if(b.style.display="flex",b.style.flexDirection="column",x){let T=i.fontSize?i.fontSize:window.getComputedStyle(document.body).fontSize,A=parseInt(T,10)*5+"px";b.style.minWidth=A,b.style.maxWidth=A}else b.style.width="100%";w(b)}o(C,"setupImage"),setTimeout(()=>{b.complete&&C()}),b.addEventListener("error",C),b.addEventListener("load",C)})))}p=g.getBoundingClientRect(),y.attr("width",p.width),y.attr("height",p.height)}return s?u.attr("transform","translate("+-p.width/2+", "+-p.height/2+")"):u.attr("transform","translate(0, "+-p.height/2+")"),e.centerLabel&&u.attr("transform","translate("+-p.width/2+", "+-p.height/2+")"),u.insert("rect",":first-child"),{shapeSvg:l,bbox:p,halfPadding:m,label:u}},"labelHelper"),Qn=o((t,e)=>{let r=e.node().getBBox();t.width=r.width,t.height=r.height},"updateNodeBounds");o(Hl,"insertPolygonShape")});var Mtt,ave,sve=N(()=>{"use strict";iF();vt();zt();nF();Mtt=o(async(t,e)=>{e.useHtmlLabels||me().flowchart.htmlLabels||(e.centerLabel=!0);let{shapeSvg:n,bbox:i,halfPadding:a}=await Di(t,e,"node "+e.classes,!0);Y.info("Classes = ",e.classes);let s=n.insert("rect",":first-child");return s.attr("rx",e.rx).attr("ry",e.ry).attr("x",-i.width/2-a).attr("y",-i.height/2-a).attr("width",i.width+e.padding).attr("height",i.height+e.padding),Qn(e,s),e.intersect=function(l){return In.rect(e,l)},n},"note"),ave=Mtt});function aF(t,e,r,n){let i=[],a=o(l=>{i.push(l,0)},"addBorder"),s=o(l=>{i.push(0,l)},"skipBorder");e.includes("t")?(Y.debug("add top border"),a(r)):s(r),e.includes("r")?(Y.debug("add right border"),a(n)):s(n),e.includes("b")?(Y.debug("add bottom border"),a(r)):s(r),e.includes("l")?(Y.debug("add left border"),a(n)):s(n),t.attr("stroke-dasharray",i.join(" "))}var ove,yo,lve,Itt,Ott,Ptt,Btt,Ftt,$tt,ztt,Gtt,Vtt,Utt,Htt,Wtt,qtt,Ytt,Xtt,jtt,Ktt,Qtt,Ztt,cve,Jtt,ert,uve,aC,sF,hve,fve=N(()=>{"use strict";dr();zt();gr();vt();Yye();rC();nF();sve();iF();ove=o(t=>t?" "+t:"","formatClass"),yo=o((t,e)=>`${e||"node default"}${ove(t.classes)} ${ove(t.class)}`,"getClassesFromNode"),lve=o(async(t,e)=>{let{shapeSvg:r,bbox:n}=await Di(t,e,yo(e,void 0),!0),i=n.width+e.padding,a=n.height+e.padding,s=i+a,l=[{x:s/2,y:0},{x:s,y:-s/2},{x:s/2,y:-s},{x:0,y:-s/2}];Y.info("Question main (Circle)");let u=Hl(r,s,s,l);return u.attr("style",e.style),Qn(e,u),e.intersect=function(h){return Y.warn("Intersect called"),In.polygon(e,l,h)},r},"question"),Itt=o((t,e)=>{let r=t.insert("g").attr("class","node default").attr("id",e.domId||e.id),n=28,i=[{x:0,y:n/2},{x:n/2,y:0},{x:0,y:-n/2},{x:-n/2,y:0}];return r.insert("polygon",":first-child").attr("points",i.map(function(s){return s.x+","+s.y}).join(" ")).attr("class","state-start").attr("r",7).attr("width",28).attr("height",28),e.width=28,e.height=28,e.intersect=function(s){return In.circle(e,14,s)},r},"choice"),Ott=o(async(t,e)=>{let{shapeSvg:r,bbox:n}=await Di(t,e,yo(e,void 0),!0),i=4,a=n.height+e.padding,s=a/i,l=n.width+2*s+e.padding,u=[{x:s,y:0},{x:l-s,y:0},{x:l,y:-a/2},{x:l-s,y:-a},{x:s,y:-a},{x:0,y:-a/2}],h=Hl(r,l,a,u);return h.attr("style",e.style),Qn(e,h),e.intersect=function(f){return In.polygon(e,u,f)},r},"hexagon"),Ptt=o(async(t,e)=>{let{shapeSvg:r,bbox:n}=await Di(t,e,void 0,!0),i=2,a=n.height+2*e.padding,s=a/i,l=n.width+2*s+e.padding,u=qye(e.directions,n,e),h=Hl(r,l,a,u);return h.attr("style",e.style),Qn(e,h),e.intersect=function(f){return In.polygon(e,u,f)},r},"block_arrow"),Btt=o(async(t,e)=>{let{shapeSvg:r,bbox:n}=await Di(t,e,yo(e,void 0),!0),i=n.width+e.padding,a=n.height+e.padding,s=[{x:-a/2,y:0},{x:i,y:0},{x:i,y:-a},{x:-a/2,y:-a},{x:0,y:-a/2}];return Hl(r,i,a,s).attr("style",e.style),e.width=i+a,e.height=a,e.intersect=function(u){return In.polygon(e,s,u)},r},"rect_left_inv_arrow"),Ftt=o(async(t,e)=>{let{shapeSvg:r,bbox:n}=await Di(t,e,yo(e),!0),i=n.width+e.padding,a=n.height+e.padding,s=[{x:-2*a/6,y:0},{x:i-a/6,y:0},{x:i+2*a/6,y:-a},{x:a/6,y:-a}],l=Hl(r,i,a,s);return l.attr("style",e.style),Qn(e,l),e.intersect=function(u){return In.polygon(e,s,u)},r},"lean_right"),$tt=o(async(t,e)=>{let{shapeSvg:r,bbox:n}=await Di(t,e,yo(e,void 0),!0),i=n.width+e.padding,a=n.height+e.padding,s=[{x:2*a/6,y:0},{x:i+a/6,y:0},{x:i-2*a/6,y:-a},{x:-a/6,y:-a}],l=Hl(r,i,a,s);return l.attr("style",e.style),Qn(e,l),e.intersect=function(u){return In.polygon(e,s,u)},r},"lean_left"),ztt=o(async(t,e)=>{let{shapeSvg:r,bbox:n}=await Di(t,e,yo(e,void 0),!0),i=n.width+e.padding,a=n.height+e.padding,s=[{x:-2*a/6,y:0},{x:i+2*a/6,y:0},{x:i-a/6,y:-a},{x:a/6,y:-a}],l=Hl(r,i,a,s);return l.attr("style",e.style),Qn(e,l),e.intersect=function(u){return In.polygon(e,s,u)},r},"trapezoid"),Gtt=o(async(t,e)=>{let{shapeSvg:r,bbox:n}=await Di(t,e,yo(e,void 0),!0),i=n.width+e.padding,a=n.height+e.padding,s=[{x:a/6,y:0},{x:i-a/6,y:0},{x:i+2*a/6,y:-a},{x:-2*a/6,y:-a}],l=Hl(r,i,a,s);return l.attr("style",e.style),Qn(e,l),e.intersect=function(u){return In.polygon(e,s,u)},r},"inv_trapezoid"),Vtt=o(async(t,e)=>{let{shapeSvg:r,bbox:n}=await Di(t,e,yo(e,void 0),!0),i=n.width+e.padding,a=n.height+e.padding,s=[{x:0,y:0},{x:i+a/2,y:0},{x:i,y:-a/2},{x:i+a/2,y:-a},{x:0,y:-a}],l=Hl(r,i,a,s);return l.attr("style",e.style),Qn(e,l),e.intersect=function(u){return In.polygon(e,s,u)},r},"rect_right_inv_arrow"),Utt=o(async(t,e)=>{let{shapeSvg:r,bbox:n}=await Di(t,e,yo(e,void 0),!0),i=n.width+e.padding,a=i/2,s=a/(2.5+i/50),l=n.height+s+e.padding,u="M 0,"+s+" a "+a+","+s+" 0,0,0 "+i+" 0 a "+a+","+s+" 0,0,0 "+-i+" 0 l 0,"+l+" a "+a+","+s+" 0,0,0 "+i+" 0 l 0,"+-l,h=r.attr("label-offset-y",s).insert("path",":first-child").attr("style",e.style).attr("d",u).attr("transform","translate("+-i/2+","+-(l/2+s)+")");return Qn(e,h),e.intersect=function(f){let d=In.rect(e,f),p=d.x-e.x;if(a!=0&&(Math.abs(p)e.height/2-s)){let m=s*s*(1-p*p/(a*a));m!=0&&(m=Math.sqrt(m)),m=s-m,f.y-e.y>0&&(m=-m),d.y+=m}return d},r},"cylinder"),Htt=o(async(t,e)=>{let{shapeSvg:r,bbox:n,halfPadding:i}=await Di(t,e,"node "+e.classes+" "+e.class,!0),a=r.insert("rect",":first-child"),s=e.positioned?e.width:n.width+e.padding,l=e.positioned?e.height:n.height+e.padding,u=e.positioned?-s/2:-n.width/2-i,h=e.positioned?-l/2:-n.height/2-i;if(a.attr("class","basic label-container").attr("style",e.style).attr("rx",e.rx).attr("ry",e.ry).attr("x",u).attr("y",h).attr("width",s).attr("height",l),e.props){let f=new Set(Object.keys(e.props));e.props.borders&&(aF(a,e.props.borders,s,l),f.delete("borders")),f.forEach(d=>{Y.warn(`Unknown node property ${d}`)})}return Qn(e,a),e.intersect=function(f){return In.rect(e,f)},r},"rect"),Wtt=o(async(t,e)=>{let{shapeSvg:r,bbox:n,halfPadding:i}=await Di(t,e,"node "+e.classes,!0),a=r.insert("rect",":first-child"),s=e.positioned?e.width:n.width+e.padding,l=e.positioned?e.height:n.height+e.padding,u=e.positioned?-s/2:-n.width/2-i,h=e.positioned?-l/2:-n.height/2-i;if(a.attr("class","basic cluster composite label-container").attr("style",e.style).attr("rx",e.rx).attr("ry",e.ry).attr("x",u).attr("y",h).attr("width",s).attr("height",l),e.props){let f=new Set(Object.keys(e.props));e.props.borders&&(aF(a,e.props.borders,s,l),f.delete("borders")),f.forEach(d=>{Y.warn(`Unknown node property ${d}`)})}return Qn(e,a),e.intersect=function(f){return In.rect(e,f)},r},"composite"),qtt=o(async(t,e)=>{let{shapeSvg:r}=await Di(t,e,"label",!0);Y.trace("Classes = ",e.class);let n=r.insert("rect",":first-child"),i=0,a=0;if(n.attr("width",i).attr("height",a),r.attr("class","label edgeLabel"),e.props){let s=new Set(Object.keys(e.props));e.props.borders&&(aF(n,e.props.borders,i,a),s.delete("borders")),s.forEach(l=>{Y.warn(`Unknown node property ${l}`)})}return Qn(e,n),e.intersect=function(s){return In.rect(e,s)},r},"labelRect");o(aF,"applyNodePropertyBorders");Ytt=o((t,e)=>{let r;e.classes?r="node "+e.classes:r="node default";let n=t.insert("g").attr("class",r).attr("id",e.domId||e.id),i=n.insert("rect",":first-child"),a=n.insert("line"),s=n.insert("g").attr("class","label"),l=e.labelText.flat?e.labelText.flat():e.labelText,u="";typeof l=="object"?u=l[0]:u=l,Y.info("Label text abc79",u,l,typeof l=="object");let h=s.node().appendChild(vs(u,e.labelStyle,!0,!0)),f={width:0,height:0};if(fr(me().flowchart.htmlLabels)){let y=h.children[0],v=Ge(h);f=y.getBoundingClientRect(),v.attr("width",f.width),v.attr("height",f.height)}Y.info("Text 2",l);let d=l.slice(1,l.length),p=h.getBBox(),m=s.node().appendChild(vs(d.join?d.join("
    "):d,e.labelStyle,!0,!0));if(fr(me().flowchart.htmlLabels)){let y=m.children[0],v=Ge(m);f=y.getBoundingClientRect(),v.attr("width",f.width),v.attr("height",f.height)}let g=e.padding/2;return Ge(m).attr("transform","translate( "+(f.width>p.width?0:(p.width-f.width)/2)+", "+(p.height+g+5)+")"),Ge(h).attr("transform","translate( "+(f.width{let{shapeSvg:r,bbox:n}=await Di(t,e,yo(e,void 0),!0),i=n.height+e.padding,a=n.width+i/4+e.padding,s=r.insert("rect",":first-child").attr("style",e.style).attr("rx",i/2).attr("ry",i/2).attr("x",-a/2).attr("y",-i/2).attr("width",a).attr("height",i);return Qn(e,s),e.intersect=function(l){return In.rect(e,l)},r},"stadium"),jtt=o(async(t,e)=>{let{shapeSvg:r,bbox:n,halfPadding:i}=await Di(t,e,yo(e,void 0),!0),a=r.insert("circle",":first-child");return a.attr("style",e.style).attr("rx",e.rx).attr("ry",e.ry).attr("r",n.width/2+i).attr("width",n.width+e.padding).attr("height",n.height+e.padding),Y.info("Circle main"),Qn(e,a),e.intersect=function(s){return Y.info("Circle intersect",e,n.width/2+i,s),In.circle(e,n.width/2+i,s)},r},"circle"),Ktt=o(async(t,e)=>{let{shapeSvg:r,bbox:n,halfPadding:i}=await Di(t,e,yo(e,void 0),!0),a=5,s=r.insert("g",":first-child"),l=s.insert("circle"),u=s.insert("circle");return s.attr("class",e.class),l.attr("style",e.style).attr("rx",e.rx).attr("ry",e.ry).attr("r",n.width/2+i+a).attr("width",n.width+e.padding+a*2).attr("height",n.height+e.padding+a*2),u.attr("style",e.style).attr("rx",e.rx).attr("ry",e.ry).attr("r",n.width/2+i).attr("width",n.width+e.padding).attr("height",n.height+e.padding),Y.info("DoubleCircle main"),Qn(e,l),e.intersect=function(h){return Y.info("DoubleCircle intersect",e,n.width/2+i+a,h),In.circle(e,n.width/2+i+a,h)},r},"doublecircle"),Qtt=o(async(t,e)=>{let{shapeSvg:r,bbox:n}=await Di(t,e,yo(e,void 0),!0),i=n.width+e.padding,a=n.height+e.padding,s=[{x:0,y:0},{x:i,y:0},{x:i,y:-a},{x:0,y:-a},{x:0,y:0},{x:-8,y:0},{x:i+8,y:0},{x:i+8,y:-a},{x:-8,y:-a},{x:-8,y:0}],l=Hl(r,i,a,s);return l.attr("style",e.style),Qn(e,l),e.intersect=function(u){return In.polygon(e,s,u)},r},"subroutine"),Ztt=o((t,e)=>{let r=t.insert("g").attr("class","node default").attr("id",e.domId||e.id),n=r.insert("circle",":first-child");return n.attr("class","state-start").attr("r",7).attr("width",14).attr("height",14),Qn(e,n),e.intersect=function(i){return In.circle(e,7,i)},r},"start"),cve=o((t,e,r)=>{let n=t.insert("g").attr("class","node default").attr("id",e.domId||e.id),i=70,a=10;r==="LR"&&(i=10,a=70);let s=n.append("rect").attr("x",-1*i/2).attr("y",-1*a/2).attr("width",i).attr("height",a).attr("class","fork-join");return Qn(e,s),e.height=e.height+e.padding/2,e.width=e.width+e.padding/2,e.intersect=function(l){return In.rect(e,l)},n},"forkJoin"),Jtt=o((t,e)=>{let r=t.insert("g").attr("class","node default").attr("id",e.domId||e.id),n=r.insert("circle",":first-child"),i=r.insert("circle",":first-child");return i.attr("class","state-start").attr("r",7).attr("width",14).attr("height",14),n.attr("class","state-end").attr("r",5).attr("width",10).attr("height",10),Qn(e,i),e.intersect=function(a){return In.circle(e,7,a)},r},"end"),ert=o((t,e)=>{let r=e.padding/2,n=4,i=8,a;e.classes?a="node "+e.classes:a="node default";let s=t.insert("g").attr("class",a).attr("id",e.domId||e.id),l=s.insert("rect",":first-child"),u=s.insert("line"),h=s.insert("line"),f=0,d=n,p=s.insert("g").attr("class","label"),m=0,g=e.classData.annotations?.[0],y=e.classData.annotations[0]?"\xAB"+e.classData.annotations[0]+"\xBB":"",v=p.node().appendChild(vs(y,e.labelStyle,!0,!0)),x=v.getBBox();if(fr(me().flowchart.htmlLabels)){let S=v.children[0],_=Ge(v);x=S.getBoundingClientRect(),_.attr("width",x.width),_.attr("height",x.height)}e.classData.annotations[0]&&(d+=x.height+n,f+=x.width);let b=e.classData.label;e.classData.type!==void 0&&e.classData.type!==""&&(me().flowchart.htmlLabels?b+="<"+e.classData.type+">":b+="<"+e.classData.type+">");let w=p.node().appendChild(vs(b,e.labelStyle,!0,!0));Ge(w).attr("class","classTitle");let C=w.getBBox();if(fr(me().flowchart.htmlLabels)){let S=w.children[0],_=Ge(w);C=S.getBoundingClientRect(),_.attr("width",C.width),_.attr("height",C.height)}d+=C.height+n,C.width>f&&(f=C.width);let T=[];e.classData.members.forEach(S=>{let _=S.getDisplayDetails(),I=_.displayText;me().flowchart.htmlLabels&&(I=I.replace(//g,">"));let D=p.node().appendChild(vs(I,_.cssStyle?_.cssStyle:e.labelStyle,!0,!0)),k=D.getBBox();if(fr(me().flowchart.htmlLabels)){let L=D.children[0],R=Ge(D);k=L.getBoundingClientRect(),R.attr("width",k.width),R.attr("height",k.height)}k.width>f&&(f=k.width),d+=k.height+n,T.push(D)}),d+=i;let E=[];if(e.classData.methods.forEach(S=>{let _=S.getDisplayDetails(),I=_.displayText;me().flowchart.htmlLabels&&(I=I.replace(//g,">"));let D=p.node().appendChild(vs(I,_.cssStyle?_.cssStyle:e.labelStyle,!0,!0)),k=D.getBBox();if(fr(me().flowchart.htmlLabels)){let L=D.children[0],R=Ge(D);k=L.getBoundingClientRect(),R.attr("width",k.width),R.attr("height",k.height)}k.width>f&&(f=k.width),d+=k.height+n,E.push(D)}),d+=i,g){let S=(f-x.width)/2;Ge(v).attr("transform","translate( "+(-1*f/2+S)+", "+-1*d/2+")"),m=x.height+n}let A=(f-C.width)/2;return Ge(w).attr("transform","translate( "+(-1*f/2+A)+", "+(-1*d/2+m)+")"),m+=C.height+n,u.attr("class","divider").attr("x1",-f/2-r).attr("x2",f/2+r).attr("y1",-d/2-r+i+m).attr("y2",-d/2-r+i+m),m+=i,T.forEach(S=>{Ge(S).attr("transform","translate( "+-f/2+", "+(-1*d/2+m+i/2)+")");let _=S?.getBBox();m+=(_?.height??0)+n}),m+=i,h.attr("class","divider").attr("x1",-f/2-r).attr("x2",f/2+r).attr("y1",-d/2-r+i+m).attr("y2",-d/2-r+i+m),m+=i,E.forEach(S=>{Ge(S).attr("transform","translate( "+-f/2+", "+(-1*d/2+m)+")");let _=S?.getBBox();m+=(_?.height??0)+n}),l.attr("style",e.style).attr("class","outer title-state").attr("x",-f/2-r).attr("y",-(d/2)-r).attr("width",f+e.padding).attr("height",d+e.padding),Qn(e,l),e.intersect=function(S){return In.rect(e,S)},s},"class_box"),uve={rhombus:lve,composite:Wtt,question:lve,rect:Htt,labelRect:qtt,rectWithTitle:Ytt,choice:Itt,circle:jtt,doublecircle:Ktt,stadium:Xtt,hexagon:Ott,block_arrow:Ptt,rect_left_inv_arrow:Btt,lean_right:Ftt,lean_left:$tt,trapezoid:ztt,inv_trapezoid:Gtt,rect_right_inv_arrow:Vtt,cylinder:Utt,start:Ztt,end:Jtt,note:ave,subroutine:Qtt,fork:cve,join:cve,class_box:ert},aC={},sF=o(async(t,e,r)=>{let n,i;if(e.link){let a;me().securityLevel==="sandbox"?a="_top":e.linkTarget&&(a=e.linkTarget||"_blank"),n=t.insert("svg:a").attr("xlink:href",e.link).attr("target",a),i=await uve[e.shape](n,e,r)}else i=await uve[e.shape](t,e,r),n=i;return e.tooltip&&i.attr("title",e.tooltip),e.class&&i.attr("class","node default "+e.class),aC[e.id]=n,e.haveCallback&&aC[e.id].attr("class",aC[e.id].attr("class")+" clickable"),n},"insertNode"),hve=o(t=>{let e=aC[t.id];Y.trace("Transforming node",t.diff,t,"translate("+(t.x-t.width/2-5)+", "+t.width/2+")");let r=8,n=t.diff||0;return t.clusterNode?e.attr("transform","translate("+(t.x+n-t.width/2)+", "+(t.y-t.height/2-r)+")"):e.attr("transform","translate("+t.x+", "+t.y+")"),n},"positionNode")});function dve(t,e,r=!1){let n=t,i="default";(n?.classes?.length||0)>0&&(i=(n?.classes??[]).join(" ")),i=i+" flowchart-label";let a=0,s="",l;switch(n.type){case"round":a=5,s="rect";break;case"composite":a=0,s="composite",l=0;break;case"square":s="rect";break;case"diamond":s="question";break;case"hexagon":s="hexagon";break;case"block_arrow":s="block_arrow";break;case"odd":s="rect_left_inv_arrow";break;case"lean_right":s="lean_right";break;case"lean_left":s="lean_left";break;case"trapezoid":s="trapezoid";break;case"inv_trapezoid":s="inv_trapezoid";break;case"rect_left_inv_arrow":s="rect_left_inv_arrow";break;case"circle":s="circle";break;case"ellipse":s="ellipse";break;case"stadium":s="stadium";break;case"subroutine":s="subroutine";break;case"cylinder":s="cylinder";break;case"group":s="rect";break;case"doublecircle":s="doublecircle";break;default:s="rect"}let u=Y9(n?.styles??[]),h=n.label,f=n.size??{width:0,height:0,x:0,y:0};return{labelStyle:u.labelStyle,shape:s,labelText:h,rx:a,ry:a,class:i,style:u.style,id:n.id,directions:n.directions,width:f.width,height:f.height,x:f.x,y:f.y,positioned:r,intersect:void 0,type:n.type,padding:l??cr()?.block?.padding??0}}async function trt(t,e,r){let n=dve(e,r,!1);if(n.type==="group")return;let i=cr(),a=await sF(t,n,{config:i}),s=a.node().getBBox(),l=r.getBlock(n.id);l.size={width:s.width,height:s.height,x:0,y:0,node:a},r.setBlock(l),a.remove()}async function rrt(t,e,r){let n=dve(e,r,!0);if(r.getBlock(n.id).type!=="space"){let a=cr();await sF(t,n,{config:a}),e.intersect=n?.intersect,hve(n)}}async function oF(t,e,r,n){for(let i of e)await n(t,i,r),i.children&&await oF(t,i.children,r,n)}async function pve(t,e,r){await oF(t,e,r,trt)}async function mve(t,e,r){await oF(t,e,r,rrt)}async function gve(t,e,r,n,i){let a=new sn({multigraph:!0,compound:!0});a.setGraph({rankdir:"TB",nodesep:10,ranksep:10,marginx:8,marginy:8});for(let s of r)s.size&&a.setNode(s.id,{width:s.size.width,height:s.size.height,intersect:s.intersect});for(let s of e)if(s.start&&s.end){let l=n.getBlock(s.start),u=n.getBlock(s.end);if(l?.size&&u?.size){let h=l.size,f=u.size,d=[{x:h.x,y:h.y},{x:h.x+(f.x-h.x)/2,y:h.y+(f.y-h.y)/2},{x:f.x,y:f.y}];Hye(t,{v:s.start,w:s.end,name:s.id},{...s,arrowTypeEnd:s.arrowTypeEnd,arrowTypeStart:s.arrowTypeStart,points:d,classes:"edge-thickness-normal edge-pattern-solid flowchart-link LS-a1 LE-b1"},void 0,"block",a,i),s.label&&(await Vye(t,{...s,label:s.label,labelStyle:"stroke: #333; stroke-width: 1.5px;fill:none;",arrowTypeEnd:s.arrowTypeEnd,arrowTypeStart:s.arrowTypeStart,points:d,classes:"edge-thickness-normal edge-pattern-solid flowchart-link LS-a1 LE-b1"}),Uye({...s,x:d[1].x,y:d[1].y},{originalPath:d}))}}}var yve=N(()=>{"use strict";Vo();ji();Wye();fve();ir();o(dve,"getNodeFromBlock");o(trt,"calculateBlockSize");o(rrt,"insertBlockPositioned");o(oF,"performOperations");o(pve,"calculateBlockSizes");o(mve,"insertBlocks");o(gve,"insertEdges")});var nrt,irt,vve,xve=N(()=>{"use strict";dr();ji();Nye();vt();Ei();Pye();yve();nrt=o(function(t,e){return e.db.getClasses()},"getClasses"),irt=o(async function(t,e,r,n){let{securityLevel:i,block:a}=cr(),s=n.db,l;i==="sandbox"&&(l=Ge("#i"+e));let u=i==="sandbox"?Ge(l.nodes()[0].contentDocument.body):Ge("body"),h=i==="sandbox"?u.select(`[id="${e}"]`):Ge(`[id="${e}"]`);Rye(h,["point","circle","cross"],n.type,e);let d=s.getBlocks(),p=s.getBlocksFlat(),m=s.getEdges(),g=h.insert("g").attr("class","block");await pve(g,d,s);let y=Oye(s);if(await mve(g,d,s),await gve(g,m,p,s,e),y){let v=y,x=Math.max(1,Math.round(.125*(v.width/v.height))),b=v.height+x+10,w=v.width+10,{useMaxWidth:C}=a;vn(h,b,w,!!C),Y.debug("Here Bounds",y,v),h.attr("viewBox",`${v.x-5} ${v.y-5} ${v.width+10} ${v.height+10}`)}},"draw"),vve={draw:irt,getClasses:nrt}});var bve={};hr(bve,{diagram:()=>art});var art,wve=N(()=>{"use strict";wye();_ye();Lye();xve();art={parser:bye,db:Aye,renderer:vve,styles:Dye}});var lF,cF,v4,Eve,uF,Ha,Zc,x4,Sve,crt,b4,Cve,Ave,_ve,Dve,Lve,sC,Ff,oC=N(()=>{"use strict";lF={L:"left",R:"right",T:"top",B:"bottom"},cF={L:o(t=>`${t},${t/2} 0,${t} 0,0`,"L"),R:o(t=>`0,${t/2} ${t},0 ${t},${t}`,"R"),T:o(t=>`0,0 ${t},0 ${t/2},${t}`,"T"),B:o(t=>`${t/2},0 ${t},${t} 0,${t}`,"B")},v4={L:o((t,e)=>t-e+2,"L"),R:o((t,e)=>t-2,"R"),T:o((t,e)=>t-e+2,"T"),B:o((t,e)=>t-2,"B")},Eve=o(function(t){return Ha(t)?t==="L"?"R":"L":t==="T"?"B":"T"},"getOppositeArchitectureDirection"),uF=o(function(t){let e=t;return e==="L"||e==="R"||e==="T"||e==="B"},"isArchitectureDirection"),Ha=o(function(t){let e=t;return e==="L"||e==="R"},"isArchitectureDirectionX"),Zc=o(function(t){let e=t;return e==="T"||e==="B"},"isArchitectureDirectionY"),x4=o(function(t,e){let r=Ha(t)&&Zc(e),n=Zc(t)&&Ha(e);return r||n},"isArchitectureDirectionXY"),Sve=o(function(t){let e=t[0],r=t[1],n=Ha(e)&&Zc(r),i=Zc(e)&&Ha(r);return n||i},"isArchitecturePairXY"),crt=o(function(t){return t!=="LL"&&t!=="RR"&&t!=="TT"&&t!=="BB"},"isValidArchitectureDirectionPair"),b4=o(function(t,e){let r=`${t}${e}`;return crt(r)?r:void 0},"getArchitectureDirectionPair"),Cve=o(function([t,e],r){let n=r[0],i=r[1];return Ha(n)?Zc(i)?[t+(n==="L"?-1:1),e+(i==="T"?1:-1)]:[t+(n==="L"?-1:1),e]:Ha(i)?[t+(i==="L"?1:-1),e+(n==="T"?1:-1)]:[t,e+(n==="T"?1:-1)]},"shiftPositionByArchitectureDirectionPair"),Ave=o(function(t){return t==="LT"||t==="TL"?[1,1]:t==="BL"||t==="LB"?[1,-1]:t==="BR"||t==="RB"?[-1,-1]:[-1,1]},"getArchitectureDirectionXYFactors"),_ve=o(function(t,e){return x4(t,e)?"bend":Ha(t)?"horizontal":"vertical"},"getArchitectureDirectionAlignment"),Dve=o(function(t){return t.type==="service"},"isArchitectureService"),Lve=o(function(t){return t.type==="junction"},"isArchitectureJunction"),sC=o(t=>t.data(),"edgeData"),Ff=o(t=>t.data(),"nodeData")});function Li(t){let e=me().architecture;return e?.[t]?e[t]:Rve[t]}var Rve,vr,urt,hrt,frt,drt,prt,mrt,hF,grt,yrt,vrt,xrt,brt,wrt,Trt,Qp,w4=N(()=>{"use strict";Ya();zt();s6();mi();oC();Rve=or.architecture,vr=new pf(()=>({nodes:{},groups:{},edges:[],registeredIds:{},config:Rve,dataStructures:void 0,elements:{}})),urt=o(()=>{vr.reset(),Ar()},"clear"),hrt=o(function({id:t,icon:e,in:r,title:n,iconText:i}){if(vr.records.registeredIds[t]!==void 0)throw new Error(`The service id [${t}] is already in use by another ${vr.records.registeredIds[t]}`);if(r!==void 0){if(t===r)throw new Error(`The service [${t}] cannot be placed within itself`);if(vr.records.registeredIds[r]===void 0)throw new Error(`The service [${t}]'s parent does not exist. Please make sure the parent is created before this service`);if(vr.records.registeredIds[r]==="node")throw new Error(`The service [${t}]'s parent is not a group`)}vr.records.registeredIds[t]="node",vr.records.nodes[t]={id:t,type:"service",icon:e,iconText:i,title:n,edges:[],in:r}},"addService"),frt=o(()=>Object.values(vr.records.nodes).filter(Dve),"getServices"),drt=o(function({id:t,in:e}){vr.records.registeredIds[t]="node",vr.records.nodes[t]={id:t,type:"junction",edges:[],in:e}},"addJunction"),prt=o(()=>Object.values(vr.records.nodes).filter(Lve),"getJunctions"),mrt=o(()=>Object.values(vr.records.nodes),"getNodes"),hF=o(t=>vr.records.nodes[t],"getNode"),grt=o(function({id:t,icon:e,in:r,title:n}){if(vr.records.registeredIds[t]!==void 0)throw new Error(`The group id [${t}] is already in use by another ${vr.records.registeredIds[t]}`);if(r!==void 0){if(t===r)throw new Error(`The group [${t}] cannot be placed within itself`);if(vr.records.registeredIds[r]===void 0)throw new Error(`The group [${t}]'s parent does not exist. Please make sure the parent is created before this group`);if(vr.records.registeredIds[r]==="node")throw new Error(`The group [${t}]'s parent is not a group`)}vr.records.registeredIds[t]="group",vr.records.groups[t]={id:t,icon:e,title:n,in:r}},"addGroup"),yrt=o(()=>Object.values(vr.records.groups),"getGroups"),vrt=o(function({lhsId:t,rhsId:e,lhsDir:r,rhsDir:n,lhsInto:i,rhsInto:a,lhsGroup:s,rhsGroup:l,title:u}){if(!uF(r))throw new Error(`Invalid direction given for left hand side of edge ${t}--${e}. Expected (L,R,T,B) got ${r}`);if(!uF(n))throw new Error(`Invalid direction given for right hand side of edge ${t}--${e}. Expected (L,R,T,B) got ${n}`);if(vr.records.nodes[t]===void 0&&vr.records.groups[t]===void 0)throw new Error(`The left-hand id [${t}] does not yet exist. Please create the service/group before declaring an edge to it.`);if(vr.records.nodes[e]===void 0&&vr.records.groups[t]===void 0)throw new Error(`The right-hand id [${e}] does not yet exist. Please create the service/group before declaring an edge to it.`);let h=vr.records.nodes[t].in,f=vr.records.nodes[e].in;if(s&&h&&f&&h==f)throw new Error(`The left-hand id [${t}] is modified to traverse the group boundary, but the edge does not pass through two groups.`);if(l&&h&&f&&h==f)throw new Error(`The right-hand id [${e}] is modified to traverse the group boundary, but the edge does not pass through two groups.`);let d={lhsId:t,lhsDir:r,lhsInto:i,lhsGroup:s,rhsId:e,rhsDir:n,rhsInto:a,rhsGroup:l,title:u};vr.records.edges.push(d),vr.records.nodes[t]&&vr.records.nodes[e]&&(vr.records.nodes[t].edges.push(vr.records.edges[vr.records.edges.length-1]),vr.records.nodes[e].edges.push(vr.records.edges[vr.records.edges.length-1]))},"addEdge"),xrt=o(()=>vr.records.edges,"getEdges"),brt=o(()=>{if(vr.records.dataStructures===void 0){let t={},e=Object.entries(vr.records.nodes).reduce((l,[u,h])=>(l[u]=h.edges.reduce((f,d)=>{let p=hF(d.lhsId)?.in,m=hF(d.rhsId)?.in;if(p&&m&&p!==m){let g=_ve(d.lhsDir,d.rhsDir);g!=="bend"&&(t[p]??={},t[p][m]=g,t[m]??={},t[m][p]=g)}if(d.lhsId===u){let g=b4(d.lhsDir,d.rhsDir);g&&(f[g]=d.rhsId)}else{let g=b4(d.rhsDir,d.lhsDir);g&&(f[g]=d.lhsId)}return f},{}),l),{}),r=Object.keys(e)[0],n={[r]:1},i=Object.keys(e).reduce((l,u)=>u===r?l:{...l,[u]:1},{}),a=o(l=>{let u={[l]:[0,0]},h=[l];for(;h.length>0;){let f=h.shift();if(f){n[f]=1,delete i[f];let d=e[f],[p,m]=u[f];Object.entries(d).forEach(([g,y])=>{n[y]||(u[y]=Cve([p,m],g),h.push(y))})}}return u},"BFS"),s=[a(r)];for(;Object.keys(i).length>0;)s.push(a(Object.keys(i)[0]));vr.records.dataStructures={adjList:e,spatialMaps:s,groupAlignments:t}}return vr.records.dataStructures},"getDataStructures"),wrt=o((t,e)=>{vr.records.elements[t]=e},"setElementForId"),Trt=o(t=>vr.records.elements[t],"getElementById"),Qp={clear:urt,setDiagramTitle:$r,getDiagramTitle:Ir,setAccTitle:Lr,getAccTitle:Rr,setAccDescription:Nr,getAccDescription:Mr,addService:hrt,getServices:frt,addJunction:drt,getJunctions:prt,getNodes:mrt,getNode:hF,addGroup:grt,getGroups:yrt,addEdge:vrt,getEdges:xrt,setElementForId:wrt,getElementById:Trt,getDataStructures:brt};o(Li,"getConfigField")});var krt,Nve,Mve=N(()=>{"use strict";kp();vt();T1();w4();krt=o((t,e)=>{$c(t,e),t.groups.map(e.addGroup),t.services.map(r=>e.addService({...r,type:"service"})),t.junctions.map(r=>e.addJunction({...r,type:"junction"})),t.edges.map(e.addEdge)},"populateDb"),Nve={parse:o(async t=>{let e=await uo("architecture",t);Y.debug(e),krt(e,Qp)},"parse")}});var Ert,Ive,Ove=N(()=>{"use strict";Ert=o(t=>` + .edge { + stroke-width: ${t.archEdgeWidth}; + stroke: ${t.archEdgeColor}; + fill: none; + } + + .arrow { + fill: ${t.archEdgeArrowColor}; + } + + .node-bkg { + fill: none; + stroke: ${t.archGroupBorderColor}; + stroke-width: ${t.archGroupBorderWidth}; + stroke-dasharray: 8; + } + .node-icon-text { + display: flex; + align-items: center; + } + + .node-icon-text > div { + color: #fff; + margin: 1px; + height: fit-content; + text-align: center; + overflow: hidden; + display: -webkit-box; + -webkit-box-orient: vertical; + } +`,"getStyles"),Ive=Ert});var dF=Mi((T4,fF)=>{"use strict";o(function(e,r){typeof T4=="object"&&typeof fF=="object"?fF.exports=r():typeof define=="function"&&define.amd?define([],r):typeof T4=="object"?T4.layoutBase=r():e.layoutBase=r()},"webpackUniversalModuleDefinition")(T4,function(){return function(t){var e={};function r(n){if(e[n])return e[n].exports;var i=e[n]={i:n,l:!1,exports:{}};return t[n].call(i.exports,i,i.exports,r),i.l=!0,i.exports}return o(r,"__webpack_require__"),r.m=t,r.c=e,r.i=function(n){return n},r.d=function(n,i,a){r.o(n,i)||Object.defineProperty(n,i,{configurable:!1,enumerable:!0,get:a})},r.n=function(n){var i=n&&n.__esModule?o(function(){return n.default},"getDefault"):o(function(){return n},"getModuleExports");return r.d(i,"a",i),i},r.o=function(n,i){return Object.prototype.hasOwnProperty.call(n,i)},r.p="",r(r.s=28)}([function(t,e,r){"use strict";function n(){}o(n,"LayoutConstants"),n.QUALITY=1,n.DEFAULT_CREATE_BENDS_AS_NEEDED=!1,n.DEFAULT_INCREMENTAL=!1,n.DEFAULT_ANIMATION_ON_LAYOUT=!0,n.DEFAULT_ANIMATION_DURING_LAYOUT=!1,n.DEFAULT_ANIMATION_PERIOD=50,n.DEFAULT_UNIFORM_LEAF_NODE_SIZES=!1,n.DEFAULT_GRAPH_MARGIN=15,n.NODE_DIMENSIONS_INCLUDE_LABELS=!1,n.SIMPLE_NODE_SIZE=40,n.SIMPLE_NODE_HALF_SIZE=n.SIMPLE_NODE_SIZE/2,n.EMPTY_COMPOUND_NODE_SIZE=40,n.MIN_EDGE_LENGTH=1,n.WORLD_BOUNDARY=1e6,n.INITIAL_WORLD_BOUNDARY=n.WORLD_BOUNDARY/1e3,n.WORLD_CENTER_X=1200,n.WORLD_CENTER_Y=900,t.exports=n},function(t,e,r){"use strict";var n=r(2),i=r(8),a=r(9);function s(u,h,f){n.call(this,f),this.isOverlapingSourceAndTarget=!1,this.vGraphObject=f,this.bendpoints=[],this.source=u,this.target=h}o(s,"LEdge"),s.prototype=Object.create(n.prototype);for(var l in n)s[l]=n[l];s.prototype.getSource=function(){return this.source},s.prototype.getTarget=function(){return this.target},s.prototype.isInterGraph=function(){return this.isInterGraph},s.prototype.getLength=function(){return this.length},s.prototype.isOverlapingSourceAndTarget=function(){return this.isOverlapingSourceAndTarget},s.prototype.getBendpoints=function(){return this.bendpoints},s.prototype.getLca=function(){return this.lca},s.prototype.getSourceInLca=function(){return this.sourceInLca},s.prototype.getTargetInLca=function(){return this.targetInLca},s.prototype.getOtherEnd=function(u){if(this.source===u)return this.target;if(this.target===u)return this.source;throw"Node is not incident with this edge"},s.prototype.getOtherEndInGraph=function(u,h){for(var f=this.getOtherEnd(u),d=h.getGraphManager().getRoot();;){if(f.getOwner()==h)return f;if(f.getOwner()==d)break;f=f.getOwner().getParent()}return null},s.prototype.updateLength=function(){var u=new Array(4);this.isOverlapingSourceAndTarget=i.getIntersection(this.target.getRect(),this.source.getRect(),u),this.isOverlapingSourceAndTarget||(this.lengthX=u[0]-u[2],this.lengthY=u[1]-u[3],Math.abs(this.lengthX)<1&&(this.lengthX=a.sign(this.lengthX)),Math.abs(this.lengthY)<1&&(this.lengthY=a.sign(this.lengthY)),this.length=Math.sqrt(this.lengthX*this.lengthX+this.lengthY*this.lengthY))},s.prototype.updateLengthSimple=function(){this.lengthX=this.target.getCenterX()-this.source.getCenterX(),this.lengthY=this.target.getCenterY()-this.source.getCenterY(),Math.abs(this.lengthX)<1&&(this.lengthX=a.sign(this.lengthX)),Math.abs(this.lengthY)<1&&(this.lengthY=a.sign(this.lengthY)),this.length=Math.sqrt(this.lengthX*this.lengthX+this.lengthY*this.lengthY)},t.exports=s},function(t,e,r){"use strict";function n(i){this.vGraphObject=i}o(n,"LGraphObject"),t.exports=n},function(t,e,r){"use strict";var n=r(2),i=r(10),a=r(13),s=r(0),l=r(16),u=r(5);function h(d,p,m,g){m==null&&g==null&&(g=p),n.call(this,g),d.graphManager!=null&&(d=d.graphManager),this.estimatedSize=i.MIN_VALUE,this.inclusionTreeDepth=i.MAX_VALUE,this.vGraphObject=g,this.edges=[],this.graphManager=d,m!=null&&p!=null?this.rect=new a(p.x,p.y,m.width,m.height):this.rect=new a}o(h,"LNode"),h.prototype=Object.create(n.prototype);for(var f in n)h[f]=n[f];h.prototype.getEdges=function(){return this.edges},h.prototype.getChild=function(){return this.child},h.prototype.getOwner=function(){return this.owner},h.prototype.getWidth=function(){return this.rect.width},h.prototype.setWidth=function(d){this.rect.width=d},h.prototype.getHeight=function(){return this.rect.height},h.prototype.setHeight=function(d){this.rect.height=d},h.prototype.getCenterX=function(){return this.rect.x+this.rect.width/2},h.prototype.getCenterY=function(){return this.rect.y+this.rect.height/2},h.prototype.getCenter=function(){return new u(this.rect.x+this.rect.width/2,this.rect.y+this.rect.height/2)},h.prototype.getLocation=function(){return new u(this.rect.x,this.rect.y)},h.prototype.getRect=function(){return this.rect},h.prototype.getDiagonal=function(){return Math.sqrt(this.rect.width*this.rect.width+this.rect.height*this.rect.height)},h.prototype.getHalfTheDiagonal=function(){return Math.sqrt(this.rect.height*this.rect.height+this.rect.width*this.rect.width)/2},h.prototype.setRect=function(d,p){this.rect.x=d.x,this.rect.y=d.y,this.rect.width=p.width,this.rect.height=p.height},h.prototype.setCenter=function(d,p){this.rect.x=d-this.rect.width/2,this.rect.y=p-this.rect.height/2},h.prototype.setLocation=function(d,p){this.rect.x=d,this.rect.y=p},h.prototype.moveBy=function(d,p){this.rect.x+=d,this.rect.y+=p},h.prototype.getEdgeListToNode=function(d){var p=[],m,g=this;return g.edges.forEach(function(y){if(y.target==d){if(y.source!=g)throw"Incorrect edge source!";p.push(y)}}),p},h.prototype.getEdgesBetween=function(d){var p=[],m,g=this;return g.edges.forEach(function(y){if(!(y.source==g||y.target==g))throw"Incorrect edge source and/or target";(y.target==d||y.source==d)&&p.push(y)}),p},h.prototype.getNeighborsList=function(){var d=new Set,p=this;return p.edges.forEach(function(m){if(m.source==p)d.add(m.target);else{if(m.target!=p)throw"Incorrect incidency!";d.add(m.source)}}),d},h.prototype.withChildren=function(){var d=new Set,p,m;if(d.add(this),this.child!=null)for(var g=this.child.getNodes(),y=0;yp?(this.rect.x-=(this.labelWidth-p)/2,this.setWidth(this.labelWidth)):this.labelPosHorizontal=="right"&&this.setWidth(p+this.labelWidth)),this.labelHeight&&(this.labelPosVertical=="top"?(this.rect.y-=this.labelHeight,this.setHeight(m+this.labelHeight)):this.labelPosVertical=="center"&&this.labelHeight>m?(this.rect.y-=(this.labelHeight-m)/2,this.setHeight(this.labelHeight)):this.labelPosVertical=="bottom"&&this.setHeight(m+this.labelHeight))}}},h.prototype.getInclusionTreeDepth=function(){if(this.inclusionTreeDepth==i.MAX_VALUE)throw"assert failed";return this.inclusionTreeDepth},h.prototype.transform=function(d){var p=this.rect.x;p>s.WORLD_BOUNDARY?p=s.WORLD_BOUNDARY:p<-s.WORLD_BOUNDARY&&(p=-s.WORLD_BOUNDARY);var m=this.rect.y;m>s.WORLD_BOUNDARY?m=s.WORLD_BOUNDARY:m<-s.WORLD_BOUNDARY&&(m=-s.WORLD_BOUNDARY);var g=new u(p,m),y=d.inverseTransformPoint(g);this.setLocation(y.x,y.y)},h.prototype.getLeft=function(){return this.rect.x},h.prototype.getRight=function(){return this.rect.x+this.rect.width},h.prototype.getTop=function(){return this.rect.y},h.prototype.getBottom=function(){return this.rect.y+this.rect.height},h.prototype.getParent=function(){return this.owner==null?null:this.owner.getParent()},t.exports=h},function(t,e,r){"use strict";var n=r(0);function i(){}o(i,"FDLayoutConstants");for(var a in n)i[a]=n[a];i.MAX_ITERATIONS=2500,i.DEFAULT_EDGE_LENGTH=50,i.DEFAULT_SPRING_STRENGTH=.45,i.DEFAULT_REPULSION_STRENGTH=4500,i.DEFAULT_GRAVITY_STRENGTH=.4,i.DEFAULT_COMPOUND_GRAVITY_STRENGTH=1,i.DEFAULT_GRAVITY_RANGE_FACTOR=3.8,i.DEFAULT_COMPOUND_GRAVITY_RANGE_FACTOR=1.5,i.DEFAULT_USE_SMART_IDEAL_EDGE_LENGTH_CALCULATION=!0,i.DEFAULT_USE_SMART_REPULSION_RANGE_CALCULATION=!0,i.DEFAULT_COOLING_FACTOR_INCREMENTAL=.3,i.COOLING_ADAPTATION_FACTOR=.33,i.ADAPTATION_LOWER_NODE_LIMIT=1e3,i.ADAPTATION_UPPER_NODE_LIMIT=5e3,i.MAX_NODE_DISPLACEMENT_INCREMENTAL=100,i.MAX_NODE_DISPLACEMENT=i.MAX_NODE_DISPLACEMENT_INCREMENTAL*3,i.MIN_REPULSION_DIST=i.DEFAULT_EDGE_LENGTH/10,i.CONVERGENCE_CHECK_PERIOD=100,i.PER_LEVEL_IDEAL_EDGE_LENGTH_FACTOR=.1,i.MIN_EDGE_LENGTH=1,i.GRID_CALCULATION_CHECK_PERIOD=10,t.exports=i},function(t,e,r){"use strict";function n(i,a){i==null&&a==null?(this.x=0,this.y=0):(this.x=i,this.y=a)}o(n,"PointD"),n.prototype.getX=function(){return this.x},n.prototype.getY=function(){return this.y},n.prototype.setX=function(i){this.x=i},n.prototype.setY=function(i){this.y=i},n.prototype.getDifference=function(i){return new DimensionD(this.x-i.x,this.y-i.y)},n.prototype.getCopy=function(){return new n(this.x,this.y)},n.prototype.translate=function(i){return this.x+=i.width,this.y+=i.height,this},t.exports=n},function(t,e,r){"use strict";var n=r(2),i=r(10),a=r(0),s=r(7),l=r(3),u=r(1),h=r(13),f=r(12),d=r(11);function p(g,y,v){n.call(this,v),this.estimatedSize=i.MIN_VALUE,this.margin=a.DEFAULT_GRAPH_MARGIN,this.edges=[],this.nodes=[],this.isConnected=!1,this.parent=g,y!=null&&y instanceof s?this.graphManager=y:y!=null&&y instanceof Layout&&(this.graphManager=y.graphManager)}o(p,"LGraph"),p.prototype=Object.create(n.prototype);for(var m in n)p[m]=n[m];p.prototype.getNodes=function(){return this.nodes},p.prototype.getEdges=function(){return this.edges},p.prototype.getGraphManager=function(){return this.graphManager},p.prototype.getParent=function(){return this.parent},p.prototype.getLeft=function(){return this.left},p.prototype.getRight=function(){return this.right},p.prototype.getTop=function(){return this.top},p.prototype.getBottom=function(){return this.bottom},p.prototype.isConnected=function(){return this.isConnected},p.prototype.add=function(g,y,v){if(y==null&&v==null){var x=g;if(this.graphManager==null)throw"Graph has no graph mgr!";if(this.getNodes().indexOf(x)>-1)throw"Node already in graph!";return x.owner=this,this.getNodes().push(x),x}else{var b=g;if(!(this.getNodes().indexOf(y)>-1&&this.getNodes().indexOf(v)>-1))throw"Source or target not in graph!";if(!(y.owner==v.owner&&y.owner==this))throw"Both owners must be this graph!";return y.owner!=v.owner?null:(b.source=y,b.target=v,b.isInterGraph=!1,this.getEdges().push(b),y.edges.push(b),v!=y&&v.edges.push(b),b)}},p.prototype.remove=function(g){var y=g;if(g instanceof l){if(y==null)throw"Node is null!";if(!(y.owner!=null&&y.owner==this))throw"Owner graph is invalid!";if(this.graphManager==null)throw"Owner graph manager is invalid!";for(var v=y.edges.slice(),x,b=v.length,w=0;w-1&&E>-1))throw"Source and/or target doesn't know this edge!";x.source.edges.splice(T,1),x.target!=x.source&&x.target.edges.splice(E,1);var C=x.source.owner.getEdges().indexOf(x);if(C==-1)throw"Not in owner's edge list!";x.source.owner.getEdges().splice(C,1)}},p.prototype.updateLeftTop=function(){for(var g=i.MAX_VALUE,y=i.MAX_VALUE,v,x,b,w=this.getNodes(),C=w.length,T=0;Tv&&(g=v),y>x&&(y=x)}return g==i.MAX_VALUE?null:(w[0].getParent().paddingLeft!=null?b=w[0].getParent().paddingLeft:b=this.margin,this.left=y-b,this.top=g-b,new f(this.left,this.top))},p.prototype.updateBounds=function(g){for(var y=i.MAX_VALUE,v=-i.MAX_VALUE,x=i.MAX_VALUE,b=-i.MAX_VALUE,w,C,T,E,A,S=this.nodes,_=S.length,I=0;I<_;I++){var D=S[I];g&&D.child!=null&&D.updateBounds(),w=D.getLeft(),C=D.getRight(),T=D.getTop(),E=D.getBottom(),y>w&&(y=w),vT&&(x=T),bw&&(y=w),vT&&(x=T),b=this.nodes.length){var _=0;v.forEach(function(I){I.owner==g&&_++}),_==this.nodes.length&&(this.isConnected=!0)}},t.exports=p},function(t,e,r){"use strict";var n,i=r(1);function a(s){n=r(6),this.layout=s,this.graphs=[],this.edges=[]}o(a,"LGraphManager"),a.prototype.addRoot=function(){var s=this.layout.newGraph(),l=this.layout.newNode(null),u=this.add(s,l);return this.setRootGraph(u),this.rootGraph},a.prototype.add=function(s,l,u,h,f){if(u==null&&h==null&&f==null){if(s==null)throw"Graph is null!";if(l==null)throw"Parent node is null!";if(this.graphs.indexOf(s)>-1)throw"Graph already in this graph mgr!";if(this.graphs.push(s),s.parent!=null)throw"Already has a parent!";if(l.child!=null)throw"Already has a child!";return s.parent=l,l.child=s,s}else{f=u,h=l,u=s;var d=h.getOwner(),p=f.getOwner();if(!(d!=null&&d.getGraphManager()==this))throw"Source not in this graph mgr!";if(!(p!=null&&p.getGraphManager()==this))throw"Target not in this graph mgr!";if(d==p)return u.isInterGraph=!1,d.add(u,h,f);if(u.isInterGraph=!0,u.source=h,u.target=f,this.edges.indexOf(u)>-1)throw"Edge already in inter-graph edge list!";if(this.edges.push(u),!(u.source!=null&&u.target!=null))throw"Edge source and/or target is null!";if(!(u.source.edges.indexOf(u)==-1&&u.target.edges.indexOf(u)==-1))throw"Edge already in source and/or target incidency list!";return u.source.edges.push(u),u.target.edges.push(u),u}},a.prototype.remove=function(s){if(s instanceof n){var l=s;if(l.getGraphManager()!=this)throw"Graph not in this graph mgr";if(!(l==this.rootGraph||l.parent!=null&&l.parent.graphManager==this))throw"Invalid parent node!";var u=[];u=u.concat(l.getEdges());for(var h,f=u.length,d=0;d=s.getRight()?l[0]+=Math.min(s.getX()-a.getX(),a.getRight()-s.getRight()):s.getX()<=a.getX()&&s.getRight()>=a.getRight()&&(l[0]+=Math.min(a.getX()-s.getX(),s.getRight()-a.getRight())),a.getY()<=s.getY()&&a.getBottom()>=s.getBottom()?l[1]+=Math.min(s.getY()-a.getY(),a.getBottom()-s.getBottom()):s.getY()<=a.getY()&&s.getBottom()>=a.getBottom()&&(l[1]+=Math.min(a.getY()-s.getY(),s.getBottom()-a.getBottom()));var f=Math.abs((s.getCenterY()-a.getCenterY())/(s.getCenterX()-a.getCenterX()));s.getCenterY()===a.getCenterY()&&s.getCenterX()===a.getCenterX()&&(f=1);var d=f*l[0],p=l[1]/f;l[0]d)return l[0]=u,l[1]=m,l[2]=f,l[3]=S,!1;if(hf)return l[0]=p,l[1]=h,l[2]=E,l[3]=d,!1;if(uf?(l[0]=y,l[1]=v,k=!0):(l[0]=g,l[1]=m,k=!0):R===M&&(u>f?(l[0]=p,l[1]=m,k=!0):(l[0]=x,l[1]=v,k=!0)),-O===M?f>u?(l[2]=A,l[3]=S,L=!0):(l[2]=E,l[3]=T,L=!0):O===M&&(f>u?(l[2]=C,l[3]=T,L=!0):(l[2]=_,l[3]=S,L=!0)),k&&L)return!1;if(u>f?h>d?(B=this.getCardinalDirection(R,M,4),F=this.getCardinalDirection(O,M,2)):(B=this.getCardinalDirection(-R,M,3),F=this.getCardinalDirection(-O,M,1)):h>d?(B=this.getCardinalDirection(-R,M,1),F=this.getCardinalDirection(-O,M,3)):(B=this.getCardinalDirection(R,M,2),F=this.getCardinalDirection(O,M,4)),!k)switch(B){case 1:z=m,P=u+-w/M,l[0]=P,l[1]=z;break;case 2:P=x,z=h+b*M,l[0]=P,l[1]=z;break;case 3:z=v,P=u+w/M,l[0]=P,l[1]=z;break;case 4:P=y,z=h+-b*M,l[0]=P,l[1]=z;break}if(!L)switch(F){case 1:H=T,$=f+-D/M,l[2]=$,l[3]=H;break;case 2:$=_,H=d+I*M,l[2]=$,l[3]=H;break;case 3:H=S,$=f+D/M,l[2]=$,l[3]=H;break;case 4:$=A,H=d+-I*M,l[2]=$,l[3]=H;break}}return!1},i.getCardinalDirection=function(a,s,l){return a>s?l:1+l%4},i.getIntersection=function(a,s,l,u){if(u==null)return this.getIntersection2(a,s,l);var h=a.x,f=a.y,d=s.x,p=s.y,m=l.x,g=l.y,y=u.x,v=u.y,x=void 0,b=void 0,w=void 0,C=void 0,T=void 0,E=void 0,A=void 0,S=void 0,_=void 0;return w=p-f,T=h-d,A=d*f-h*p,C=v-g,E=m-y,S=y*g-m*v,_=w*E-C*T,_===0?null:(x=(T*S-E*A)/_,b=(C*A-w*S)/_,new n(x,b))},i.angleOfVector=function(a,s,l,u){var h=void 0;return a!==l?(h=Math.atan((u-s)/(l-a)),l=0){var v=(-m+Math.sqrt(m*m-4*p*g))/(2*p),x=(-m-Math.sqrt(m*m-4*p*g))/(2*p),b=null;return v>=0&&v<=1?[v]:x>=0&&x<=1?[x]:b}else return null},i.HALF_PI=.5*Math.PI,i.ONE_AND_HALF_PI=1.5*Math.PI,i.TWO_PI=2*Math.PI,i.THREE_PI=3*Math.PI,t.exports=i},function(t,e,r){"use strict";function n(){}o(n,"IMath"),n.sign=function(i){return i>0?1:i<0?-1:0},n.floor=function(i){return i<0?Math.ceil(i):Math.floor(i)},n.ceil=function(i){return i<0?Math.floor(i):Math.ceil(i)},t.exports=n},function(t,e,r){"use strict";function n(){}o(n,"Integer"),n.MAX_VALUE=2147483647,n.MIN_VALUE=-2147483648,t.exports=n},function(t,e,r){"use strict";var n=function(){function h(f,d){for(var p=0;p"u"?"undefined":n(a);return a==null||s!="object"&&s!="function"},t.exports=i},function(t,e,r){"use strict";function n(m){if(Array.isArray(m)){for(var g=0,y=Array(m.length);g0&&g;){for(w.push(T[0]);w.length>0&&g;){var E=w[0];w.splice(0,1),b.add(E);for(var A=E.getEdges(),x=0;x-1&&T.splice(D,1)}b=new Set,C=new Map}}return m},p.prototype.createDummyNodesForBendpoints=function(m){for(var g=[],y=m.source,v=this.graphManager.calcLowestCommonAncestor(m.source,m.target),x=0;x0){for(var v=this.edgeToDummyNodes.get(y),x=0;x=0&&g.splice(S,1);var _=C.getNeighborsList();_.forEach(function(k){if(y.indexOf(k)<0){var L=v.get(k),R=L-1;R==1&&E.push(k),v.set(k,R)}})}y=y.concat(E),(g.length==1||g.length==2)&&(x=!0,b=g[0])}return b},p.prototype.setGraphManager=function(m){this.graphManager=m},t.exports=p},function(t,e,r){"use strict";function n(){}o(n,"RandomSeed"),n.seed=1,n.x=0,n.nextDouble=function(){return n.x=Math.sin(n.seed++)*1e4,n.x-Math.floor(n.x)},t.exports=n},function(t,e,r){"use strict";var n=r(5);function i(a,s){this.lworldOrgX=0,this.lworldOrgY=0,this.ldeviceOrgX=0,this.ldeviceOrgY=0,this.lworldExtX=1,this.lworldExtY=1,this.ldeviceExtX=1,this.ldeviceExtY=1}o(i,"Transform"),i.prototype.getWorldOrgX=function(){return this.lworldOrgX},i.prototype.setWorldOrgX=function(a){this.lworldOrgX=a},i.prototype.getWorldOrgY=function(){return this.lworldOrgY},i.prototype.setWorldOrgY=function(a){this.lworldOrgY=a},i.prototype.getWorldExtX=function(){return this.lworldExtX},i.prototype.setWorldExtX=function(a){this.lworldExtX=a},i.prototype.getWorldExtY=function(){return this.lworldExtY},i.prototype.setWorldExtY=function(a){this.lworldExtY=a},i.prototype.getDeviceOrgX=function(){return this.ldeviceOrgX},i.prototype.setDeviceOrgX=function(a){this.ldeviceOrgX=a},i.prototype.getDeviceOrgY=function(){return this.ldeviceOrgY},i.prototype.setDeviceOrgY=function(a){this.ldeviceOrgY=a},i.prototype.getDeviceExtX=function(){return this.ldeviceExtX},i.prototype.setDeviceExtX=function(a){this.ldeviceExtX=a},i.prototype.getDeviceExtY=function(){return this.ldeviceExtY},i.prototype.setDeviceExtY=function(a){this.ldeviceExtY=a},i.prototype.transformX=function(a){var s=0,l=this.lworldExtX;return l!=0&&(s=this.ldeviceOrgX+(a-this.lworldOrgX)*this.ldeviceExtX/l),s},i.prototype.transformY=function(a){var s=0,l=this.lworldExtY;return l!=0&&(s=this.ldeviceOrgY+(a-this.lworldOrgY)*this.ldeviceExtY/l),s},i.prototype.inverseTransformX=function(a){var s=0,l=this.ldeviceExtX;return l!=0&&(s=this.lworldOrgX+(a-this.ldeviceOrgX)*this.lworldExtX/l),s},i.prototype.inverseTransformY=function(a){var s=0,l=this.ldeviceExtY;return l!=0&&(s=this.lworldOrgY+(a-this.ldeviceOrgY)*this.lworldExtY/l),s},i.prototype.inverseTransformPoint=function(a){var s=new n(this.inverseTransformX(a.x),this.inverseTransformY(a.y));return s},t.exports=i},function(t,e,r){"use strict";function n(d){if(Array.isArray(d)){for(var p=0,m=Array(d.length);pa.ADAPTATION_LOWER_NODE_LIMIT&&(this.coolingFactor=Math.max(this.coolingFactor*a.COOLING_ADAPTATION_FACTOR,this.coolingFactor-(d-a.ADAPTATION_LOWER_NODE_LIMIT)/(a.ADAPTATION_UPPER_NODE_LIMIT-a.ADAPTATION_LOWER_NODE_LIMIT)*this.coolingFactor*(1-a.COOLING_ADAPTATION_FACTOR))),this.maxNodeDisplacement=a.MAX_NODE_DISPLACEMENT_INCREMENTAL):(d>a.ADAPTATION_LOWER_NODE_LIMIT?this.coolingFactor=Math.max(a.COOLING_ADAPTATION_FACTOR,1-(d-a.ADAPTATION_LOWER_NODE_LIMIT)/(a.ADAPTATION_UPPER_NODE_LIMIT-a.ADAPTATION_LOWER_NODE_LIMIT)*(1-a.COOLING_ADAPTATION_FACTOR)):this.coolingFactor=1,this.initialCoolingFactor=this.coolingFactor,this.maxNodeDisplacement=a.MAX_NODE_DISPLACEMENT),this.maxIterations=Math.max(this.getAllNodes().length*5,this.maxIterations),this.displacementThresholdPerNode=3*a.DEFAULT_EDGE_LENGTH/100,this.totalDisplacementThreshold=this.displacementThresholdPerNode*this.getAllNodes().length,this.repulsionRange=this.calcRepulsionRange()},h.prototype.calcSpringForces=function(){for(var d=this.getAllEdges(),p,m=0;m0&&arguments[0]!==void 0?arguments[0]:!0,p=arguments.length>1&&arguments[1]!==void 0?arguments[1]:!1,m,g,y,v,x=this.getAllNodes(),b;if(this.useFRGridVariant)for(this.totalIterations%a.GRID_CALCULATION_CHECK_PERIOD==1&&d&&this.updateGrid(),b=new Set,m=0;mw||b>w)&&(d.gravitationForceX=-this.gravityConstant*y,d.gravitationForceY=-this.gravityConstant*v)):(w=p.getEstimatedSize()*this.compoundGravityRangeFactor,(x>w||b>w)&&(d.gravitationForceX=-this.gravityConstant*y*this.compoundGravityConstant,d.gravitationForceY=-this.gravityConstant*v*this.compoundGravityConstant))},h.prototype.isConverged=function(){var d,p=!1;return this.totalIterations>this.maxIterations/3&&(p=Math.abs(this.totalDisplacement-this.oldTotalDisplacement)<2),d=this.totalDisplacement=x.length||w>=x[0].length)){for(var C=0;Ch},"_defaultCompareFunction")}]),l}();t.exports=s},function(t,e,r){"use strict";function n(){}o(n,"SVD"),n.svd=function(i){this.U=null,this.V=null,this.s=null,this.m=0,this.n=0,this.m=i.length,this.n=i[0].length;var a=Math.min(this.m,this.n);this.s=function(xt){for(var ut=[];xt-- >0;)ut.push(0);return ut}(Math.min(this.m+1,this.n)),this.U=function(xt){var ut=o(function Et(ft){if(ft.length==0)return 0;for(var yt=[],nt=0;nt0;)ut.push(0);return ut}(this.n),l=function(xt){for(var ut=[];xt-- >0;)ut.push(0);return ut}(this.m),u=!0,h=!0,f=Math.min(this.m-1,this.n),d=Math.max(0,Math.min(this.n-2,this.m)),p=0;p=0;M--)if(this.s[M]!==0){for(var B=M+1;B=0;j--){if(function(xt,ut){return xt&&ut}(j0;){var ue=void 0,Z=void 0;for(ue=L-2;ue>=-1&&ue!==-1;ue--)if(Math.abs(s[ue])<=se+J*(Math.abs(this.s[ue])+Math.abs(this.s[ue+1]))){s[ue]=0;break}if(ue===L-2)Z=4;else{var Se=void 0;for(Se=L-1;Se>=ue&&Se!==ue;Se--){var ce=(Se!==L?Math.abs(s[Se]):0)+(Se!==ue+1?Math.abs(s[Se-1]):0);if(Math.abs(this.s[Se])<=se+J*ce){this.s[Se]=0;break}}Se===ue?Z=3:Se===L-1?Z=1:(Z=2,ue=Se)}switch(ue++,Z){case 1:{var ae=s[L-2];s[L-2]=0;for(var Oe=L-2;Oe>=ue;Oe--){var ge=n.hypot(this.s[Oe],ae),ze=this.s[Oe]/ge,He=ae/ge;if(this.s[Oe]=ge,Oe!==ue&&(ae=-He*s[Oe-1],s[Oe-1]=ze*s[Oe-1]),h)for(var $e=0;$e=this.s[ue+1]);){var ot=this.s[ue];if(this.s[ue]=this.s[ue+1],this.s[ue+1]=ot,h&&ueMath.abs(a)?(s=a/i,s=Math.abs(i)*Math.sqrt(1+s*s)):a!=0?(s=i/a,s=Math.abs(a)*Math.sqrt(1+s*s)):s=0,s},t.exports=n},function(t,e,r){"use strict";var n=function(){function s(l,u){for(var h=0;h2&&arguments[2]!==void 0?arguments[2]:1,f=arguments.length>3&&arguments[3]!==void 0?arguments[3]:-1,d=arguments.length>4&&arguments[4]!==void 0?arguments[4]:-1;i(this,s),this.sequence1=l,this.sequence2=u,this.match_score=h,this.mismatch_penalty=f,this.gap_penalty=d,this.iMax=l.length+1,this.jMax=u.length+1,this.grid=new Array(this.iMax);for(var p=0;p=0;l--){var u=this.listeners[l];u.event===a&&u.callback===s&&this.listeners.splice(l,1)}},i.emit=function(a,s){for(var l=0;l{"use strict";o(function(e,r){typeof k4=="object"&&typeof pF=="object"?pF.exports=r(dF()):typeof define=="function"&&define.amd?define(["layout-base"],r):typeof k4=="object"?k4.coseBase=r(dF()):e.coseBase=r(e.layoutBase)},"webpackUniversalModuleDefinition")(k4,function(t){return(()=>{"use strict";var e={45:(a,s,l)=>{var u={};u.layoutBase=l(551),u.CoSEConstants=l(806),u.CoSEEdge=l(767),u.CoSEGraph=l(880),u.CoSEGraphManager=l(578),u.CoSELayout=l(765),u.CoSENode=l(991),u.ConstraintHandler=l(902),a.exports=u},806:(a,s,l)=>{var u=l(551).FDLayoutConstants;function h(){}o(h,"CoSEConstants");for(var f in u)h[f]=u[f];h.DEFAULT_USE_MULTI_LEVEL_SCALING=!1,h.DEFAULT_RADIAL_SEPARATION=u.DEFAULT_EDGE_LENGTH,h.DEFAULT_COMPONENT_SEPERATION=60,h.TILE=!0,h.TILING_PADDING_VERTICAL=10,h.TILING_PADDING_HORIZONTAL=10,h.TRANSFORM_ON_CONSTRAINT_HANDLING=!0,h.ENFORCE_CONSTRAINTS=!0,h.APPLY_LAYOUT=!0,h.RELAX_MOVEMENT_ON_CONSTRAINTS=!0,h.TREE_REDUCTION_ON_INCREMENTAL=!0,h.PURE_INCREMENTAL=h.DEFAULT_INCREMENTAL,a.exports=h},767:(a,s,l)=>{var u=l(551).FDLayoutEdge;function h(d,p,m){u.call(this,d,p,m)}o(h,"CoSEEdge"),h.prototype=Object.create(u.prototype);for(var f in u)h[f]=u[f];a.exports=h},880:(a,s,l)=>{var u=l(551).LGraph;function h(d,p,m){u.call(this,d,p,m)}o(h,"CoSEGraph"),h.prototype=Object.create(u.prototype);for(var f in u)h[f]=u[f];a.exports=h},578:(a,s,l)=>{var u=l(551).LGraphManager;function h(d){u.call(this,d)}o(h,"CoSEGraphManager"),h.prototype=Object.create(u.prototype);for(var f in u)h[f]=u[f];a.exports=h},765:(a,s,l)=>{var u=l(551).FDLayout,h=l(578),f=l(880),d=l(991),p=l(767),m=l(806),g=l(902),y=l(551).FDLayoutConstants,v=l(551).LayoutConstants,x=l(551).Point,b=l(551).PointD,w=l(551).DimensionD,C=l(551).Layout,T=l(551).Integer,E=l(551).IGeometry,A=l(551).LGraph,S=l(551).Transform,_=l(551).LinkedList;function I(){u.call(this),this.toBeTiled={},this.constraints={}}o(I,"CoSELayout"),I.prototype=Object.create(u.prototype);for(var D in u)I[D]=u[D];I.prototype.newGraphManager=function(){var k=new h(this);return this.graphManager=k,k},I.prototype.newGraph=function(k){return new f(null,this.graphManager,k)},I.prototype.newNode=function(k){return new d(this.graphManager,k)},I.prototype.newEdge=function(k){return new p(null,null,k)},I.prototype.initParameters=function(){u.prototype.initParameters.call(this,arguments),this.isSubLayout||(m.DEFAULT_EDGE_LENGTH<10?this.idealEdgeLength=10:this.idealEdgeLength=m.DEFAULT_EDGE_LENGTH,this.useSmartIdealEdgeLengthCalculation=m.DEFAULT_USE_SMART_IDEAL_EDGE_LENGTH_CALCULATION,this.gravityConstant=y.DEFAULT_GRAVITY_STRENGTH,this.compoundGravityConstant=y.DEFAULT_COMPOUND_GRAVITY_STRENGTH,this.gravityRangeFactor=y.DEFAULT_GRAVITY_RANGE_FACTOR,this.compoundGravityRangeFactor=y.DEFAULT_COMPOUND_GRAVITY_RANGE_FACTOR,this.prunedNodesAll=[],this.growTreeIterations=0,this.afterGrowthIterations=0,this.isTreeGrowing=!1,this.isGrowthFinished=!1)},I.prototype.initSpringEmbedder=function(){u.prototype.initSpringEmbedder.call(this),this.coolingCycle=0,this.maxCoolingCycle=this.maxIterations/y.CONVERGENCE_CHECK_PERIOD,this.finalTemperature=.04,this.coolingAdjuster=1},I.prototype.layout=function(){var k=v.DEFAULT_CREATE_BENDS_AS_NEEDED;return k&&(this.createBendpoints(),this.graphManager.resetAllEdges()),this.level=0,this.classicLayout()},I.prototype.classicLayout=function(){if(this.nodesWithGravity=this.calculateNodesToApplyGravitationTo(),this.graphManager.setAllNodesToApplyGravitation(this.nodesWithGravity),this.calcNoOfChildrenForAllNodes(),this.graphManager.calcLowestCommonAncestors(),this.graphManager.calcInclusionTreeDepths(),this.graphManager.getRoot().calcEstimatedSize(),this.calcIdealEdgeLengths(),this.incremental){if(m.TREE_REDUCTION_ON_INCREMENTAL){this.reduceTrees(),this.graphManager.resetAllNodesToApplyGravitation();var L=new Set(this.getAllNodes()),R=this.nodesWithGravity.filter(function(B){return L.has(B)});this.graphManager.setAllNodesToApplyGravitation(R)}}else{var k=this.getFlatForest();if(k.length>0)this.positionNodesRadially(k);else{this.reduceTrees(),this.graphManager.resetAllNodesToApplyGravitation();var L=new Set(this.getAllNodes()),R=this.nodesWithGravity.filter(function(O){return L.has(O)});this.graphManager.setAllNodesToApplyGravitation(R),this.positionNodesRandomly()}}return Object.keys(this.constraints).length>0&&(g.handleConstraints(this),this.initConstraintVariables()),this.initSpringEmbedder(),m.APPLY_LAYOUT&&this.runSpringEmbedder(),!0},I.prototype.tick=function(){if(this.totalIterations++,this.totalIterations===this.maxIterations&&!this.isTreeGrowing&&!this.isGrowthFinished)if(this.prunedNodesAll.length>0)this.isTreeGrowing=!0;else return!0;if(this.totalIterations%y.CONVERGENCE_CHECK_PERIOD==0&&!this.isTreeGrowing&&!this.isGrowthFinished){if(this.isConverged())if(this.prunedNodesAll.length>0)this.isTreeGrowing=!0;else return!0;this.coolingCycle++,this.layoutQuality==0?this.coolingAdjuster=this.coolingCycle:this.layoutQuality==1&&(this.coolingAdjuster=this.coolingCycle/3),this.coolingFactor=Math.max(this.initialCoolingFactor-Math.pow(this.coolingCycle,Math.log(100*(this.initialCoolingFactor-this.finalTemperature))/Math.log(this.maxCoolingCycle))/100*this.coolingAdjuster,this.finalTemperature),this.animationPeriod=Math.ceil(this.initialAnimationPeriod*Math.sqrt(this.coolingFactor))}if(this.isTreeGrowing){if(this.growTreeIterations%10==0)if(this.prunedNodesAll.length>0){this.graphManager.updateBounds(),this.updateGrid(),this.growTree(this.prunedNodesAll),this.graphManager.resetAllNodesToApplyGravitation();var k=new Set(this.getAllNodes()),L=this.nodesWithGravity.filter(function(M){return k.has(M)});this.graphManager.setAllNodesToApplyGravitation(L),this.graphManager.updateBounds(),this.updateGrid(),m.PURE_INCREMENTAL?this.coolingFactor=y.DEFAULT_COOLING_FACTOR_INCREMENTAL/2:this.coolingFactor=y.DEFAULT_COOLING_FACTOR_INCREMENTAL}else this.isTreeGrowing=!1,this.isGrowthFinished=!0;this.growTreeIterations++}if(this.isGrowthFinished){if(this.isConverged())return!0;this.afterGrowthIterations%10==0&&(this.graphManager.updateBounds(),this.updateGrid()),m.PURE_INCREMENTAL?this.coolingFactor=y.DEFAULT_COOLING_FACTOR_INCREMENTAL/2*((100-this.afterGrowthIterations)/100):this.coolingFactor=y.DEFAULT_COOLING_FACTOR_INCREMENTAL*((100-this.afterGrowthIterations)/100),this.afterGrowthIterations++}var R=!this.isTreeGrowing&&!this.isGrowthFinished,O=this.growTreeIterations%10==1&&this.isTreeGrowing||this.afterGrowthIterations%10==1&&this.isGrowthFinished;return this.totalDisplacement=0,this.graphManager.updateBounds(),this.calcSpringForces(),this.calcRepulsionForces(R,O),this.calcGravitationalForces(),this.moveNodes(),this.animate(),!1},I.prototype.getPositionsData=function(){for(var k=this.graphManager.getAllNodes(),L={},R=0;R0&&this.updateDisplacements();for(var R=0;R0&&(O.fixedNodeWeight=B)}}if(this.constraints.relativePlacementConstraint){var F=new Map,P=new Map;if(this.dummyToNodeForVerticalAlignment=new Map,this.dummyToNodeForHorizontalAlignment=new Map,this.fixedNodesOnHorizontal=new Set,this.fixedNodesOnVertical=new Set,this.fixedNodeSet.forEach(function(le){k.fixedNodesOnHorizontal.add(le),k.fixedNodesOnVertical.add(le)}),this.constraints.alignmentConstraint){if(this.constraints.alignmentConstraint.vertical)for(var z=this.constraints.alignmentConstraint.vertical,R=0;R=2*le.length/3;X--)he=Math.floor(Math.random()*(X+1)),K=le[X],le[X]=le[he],le[he]=K;return le},this.nodesInRelativeHorizontal=[],this.nodesInRelativeVertical=[],this.nodeToRelativeConstraintMapHorizontal=new Map,this.nodeToRelativeConstraintMapVertical=new Map,this.nodeToTempPositionMapHorizontal=new Map,this.nodeToTempPositionMapVertical=new Map,this.constraints.relativePlacementConstraint.forEach(function(le){if(le.left){var he=F.has(le.left)?F.get(le.left):le.left,K=F.has(le.right)?F.get(le.right):le.right;k.nodesInRelativeHorizontal.includes(he)||(k.nodesInRelativeHorizontal.push(he),k.nodeToRelativeConstraintMapHorizontal.set(he,[]),k.dummyToNodeForVerticalAlignment.has(he)?k.nodeToTempPositionMapHorizontal.set(he,k.idToNodeMap.get(k.dummyToNodeForVerticalAlignment.get(he)[0]).getCenterX()):k.nodeToTempPositionMapHorizontal.set(he,k.idToNodeMap.get(he).getCenterX())),k.nodesInRelativeHorizontal.includes(K)||(k.nodesInRelativeHorizontal.push(K),k.nodeToRelativeConstraintMapHorizontal.set(K,[]),k.dummyToNodeForVerticalAlignment.has(K)?k.nodeToTempPositionMapHorizontal.set(K,k.idToNodeMap.get(k.dummyToNodeForVerticalAlignment.get(K)[0]).getCenterX()):k.nodeToTempPositionMapHorizontal.set(K,k.idToNodeMap.get(K).getCenterX())),k.nodeToRelativeConstraintMapHorizontal.get(he).push({right:K,gap:le.gap}),k.nodeToRelativeConstraintMapHorizontal.get(K).push({left:he,gap:le.gap})}else{var X=P.has(le.top)?P.get(le.top):le.top,te=P.has(le.bottom)?P.get(le.bottom):le.bottom;k.nodesInRelativeVertical.includes(X)||(k.nodesInRelativeVertical.push(X),k.nodeToRelativeConstraintMapVertical.set(X,[]),k.dummyToNodeForHorizontalAlignment.has(X)?k.nodeToTempPositionMapVertical.set(X,k.idToNodeMap.get(k.dummyToNodeForHorizontalAlignment.get(X)[0]).getCenterY()):k.nodeToTempPositionMapVertical.set(X,k.idToNodeMap.get(X).getCenterY())),k.nodesInRelativeVertical.includes(te)||(k.nodesInRelativeVertical.push(te),k.nodeToRelativeConstraintMapVertical.set(te,[]),k.dummyToNodeForHorizontalAlignment.has(te)?k.nodeToTempPositionMapVertical.set(te,k.idToNodeMap.get(k.dummyToNodeForHorizontalAlignment.get(te)[0]).getCenterY()):k.nodeToTempPositionMapVertical.set(te,k.idToNodeMap.get(te).getCenterY())),k.nodeToRelativeConstraintMapVertical.get(X).push({bottom:te,gap:le.gap}),k.nodeToRelativeConstraintMapVertical.get(te).push({top:X,gap:le.gap})}});else{var H=new Map,Q=new Map;this.constraints.relativePlacementConstraint.forEach(function(le){if(le.left){var he=F.has(le.left)?F.get(le.left):le.left,K=F.has(le.right)?F.get(le.right):le.right;H.has(he)?H.get(he).push(K):H.set(he,[K]),H.has(K)?H.get(K).push(he):H.set(K,[he])}else{var X=P.has(le.top)?P.get(le.top):le.top,te=P.has(le.bottom)?P.get(le.bottom):le.bottom;Q.has(X)?Q.get(X).push(te):Q.set(X,[te]),Q.has(te)?Q.get(te).push(X):Q.set(te,[X])}});var j=o(function(he,K){var X=[],te=[],J=new _,se=new Set,ue=0;return he.forEach(function(Z,Se){if(!se.has(Se)){X[ue]=[],te[ue]=!1;var ce=Se;for(J.push(ce),se.add(ce),X[ue].push(ce);J.length!=0;){ce=J.shift(),K.has(ce)&&(te[ue]=!0);var ae=he.get(ce);ae.forEach(function(Oe){se.has(Oe)||(J.push(Oe),se.add(Oe),X[ue].push(Oe))})}ue++}}),{components:X,isFixed:te}},"constructComponents"),ie=j(H,k.fixedNodesOnHorizontal);this.componentsOnHorizontal=ie.components,this.fixedComponentsOnHorizontal=ie.isFixed;var ne=j(Q,k.fixedNodesOnVertical);this.componentsOnVertical=ne.components,this.fixedComponentsOnVertical=ne.isFixed}}},I.prototype.updateDisplacements=function(){var k=this;if(this.constraints.fixedNodeConstraint&&this.constraints.fixedNodeConstraint.forEach(function(ne){var le=k.idToNodeMap.get(ne.nodeId);le.displacementX=0,le.displacementY=0}),this.constraints.alignmentConstraint){if(this.constraints.alignmentConstraint.vertical)for(var L=this.constraints.alignmentConstraint.vertical,R=0;R1){var P;for(P=0;PO&&(O=Math.floor(F.y)),B=Math.floor(F.x+m.DEFAULT_COMPONENT_SEPERATION)}this.transform(new b(v.WORLD_CENTER_X-F.x/2,v.WORLD_CENTER_Y-F.y/2))},I.radialLayout=function(k,L,R){var O=Math.max(this.maxDiagonalInTree(k),m.DEFAULT_RADIAL_SEPARATION);I.branchRadialLayout(L,null,0,359,0,O);var M=A.calculateBounds(k),B=new S;B.setDeviceOrgX(M.getMinX()),B.setDeviceOrgY(M.getMinY()),B.setWorldOrgX(R.x),B.setWorldOrgY(R.y);for(var F=0;F1;){var X=K[0];K.splice(0,1);var te=j.indexOf(X);te>=0&&j.splice(te,1),le--,ie--}L!=null?he=(j.indexOf(K[0])+1)%le:he=0;for(var J=Math.abs(O-R)/ie,se=he;ne!=ie;se=++se%le){var ue=j[se].getOtherEnd(k);if(ue!=L){var Z=(R+ne*J)%360,Se=(Z+J)%360;I.branchRadialLayout(ue,k,Z,Se,M+B,B),ne++}}},I.maxDiagonalInTree=function(k){for(var L=T.MIN_VALUE,R=0;RL&&(L=M)}return L},I.prototype.calcRepulsionRange=function(){return 2*(this.level+1)*this.idealEdgeLength},I.prototype.groupZeroDegreeMembers=function(){var k=this,L={};this.memberGroups={},this.idToDummyNode={};for(var R=[],O=this.graphManager.getAllNodes(),M=0;M"u"&&(L[P]=[]),L[P]=L[P].concat(B)}Object.keys(L).forEach(function(z){if(L[z].length>1){var $="DummyCompound_"+z;k.memberGroups[$]=L[z];var H=L[z][0].getParent(),Q=new d(k.graphManager);Q.id=$,Q.paddingLeft=H.paddingLeft||0,Q.paddingRight=H.paddingRight||0,Q.paddingBottom=H.paddingBottom||0,Q.paddingTop=H.paddingTop||0,k.idToDummyNode[$]=Q;var j=k.getGraphManager().add(k.newGraph(),Q),ie=H.getChild();ie.add(Q);for(var ne=0;neM?(O.rect.x-=(O.labelWidth-M)/2,O.setWidth(O.labelWidth),O.labelMarginLeft=(O.labelWidth-M)/2):O.labelPosHorizontal=="right"&&O.setWidth(M+O.labelWidth)),O.labelHeight&&(O.labelPosVertical=="top"?(O.rect.y-=O.labelHeight,O.setHeight(B+O.labelHeight),O.labelMarginTop=O.labelHeight):O.labelPosVertical=="center"&&O.labelHeight>B?(O.rect.y-=(O.labelHeight-B)/2,O.setHeight(O.labelHeight),O.labelMarginTop=(O.labelHeight-B)/2):O.labelPosVertical=="bottom"&&O.setHeight(B+O.labelHeight))}})},I.prototype.repopulateCompounds=function(){for(var k=this.compoundOrder.length-1;k>=0;k--){var L=this.compoundOrder[k],R=L.id,O=L.paddingLeft,M=L.paddingTop,B=L.labelMarginLeft,F=L.labelMarginTop;this.adjustLocations(this.tiledMemberPack[R],L.rect.x,L.rect.y,O,M,B,F)}},I.prototype.repopulateZeroDegreeMembers=function(){var k=this,L=this.tiledZeroDegreePack;Object.keys(L).forEach(function(R){var O=k.idToDummyNode[R],M=O.paddingLeft,B=O.paddingTop,F=O.labelMarginLeft,P=O.labelMarginTop;k.adjustLocations(L[R],O.rect.x,O.rect.y,M,B,F,P)})},I.prototype.getToBeTiled=function(k){var L=k.id;if(this.toBeTiled[L]!=null)return this.toBeTiled[L];var R=k.getChild();if(R==null)return this.toBeTiled[L]=!1,!1;for(var O=R.getNodes(),M=0;M0)return this.toBeTiled[L]=!1,!1;if(B.getChild()==null){this.toBeTiled[B.id]=!1;continue}if(!this.getToBeTiled(B))return this.toBeTiled[L]=!1,!1}return this.toBeTiled[L]=!0,!0},I.prototype.getNodeDegree=function(k){for(var L=k.id,R=k.getEdges(),O=0,M=0;MH&&(H=j.rect.height)}R+=H+k.verticalPadding}},I.prototype.tileCompoundMembers=function(k,L){var R=this;this.tiledMemberPack=[],Object.keys(k).forEach(function(O){var M=L[O];if(R.tiledMemberPack[O]=R.tileNodes(k[O],M.paddingLeft+M.paddingRight),M.rect.width=R.tiledMemberPack[O].width,M.rect.height=R.tiledMemberPack[O].height,M.setCenter(R.tiledMemberPack[O].centerX,R.tiledMemberPack[O].centerY),M.labelMarginLeft=0,M.labelMarginTop=0,m.NODE_DIMENSIONS_INCLUDE_LABELS){var B=M.rect.width,F=M.rect.height;M.labelWidth&&(M.labelPosHorizontal=="left"?(M.rect.x-=M.labelWidth,M.setWidth(B+M.labelWidth),M.labelMarginLeft=M.labelWidth):M.labelPosHorizontal=="center"&&M.labelWidth>B?(M.rect.x-=(M.labelWidth-B)/2,M.setWidth(M.labelWidth),M.labelMarginLeft=(M.labelWidth-B)/2):M.labelPosHorizontal=="right"&&M.setWidth(B+M.labelWidth)),M.labelHeight&&(M.labelPosVertical=="top"?(M.rect.y-=M.labelHeight,M.setHeight(F+M.labelHeight),M.labelMarginTop=M.labelHeight):M.labelPosVertical=="center"&&M.labelHeight>F?(M.rect.y-=(M.labelHeight-F)/2,M.setHeight(M.labelHeight),M.labelMarginTop=(M.labelHeight-F)/2):M.labelPosVertical=="bottom"&&M.setHeight(F+M.labelHeight))}})},I.prototype.tileNodes=function(k,L){var R=this.tileNodesByFavoringDim(k,L,!0),O=this.tileNodesByFavoringDim(k,L,!1),M=this.getOrgRatio(R),B=this.getOrgRatio(O),F;return BP&&(P=ne.getWidth())});var z=B/M,$=F/M,H=Math.pow(R-O,2)+4*(z+O)*($+R)*M,Q=(O-R+Math.sqrt(H))/(2*(z+O)),j;L?(j=Math.ceil(Q),j==Q&&j++):j=Math.floor(Q);var ie=j*(z+O)-O;return P>ie&&(ie=P),ie+=O*2,ie},I.prototype.tileNodesByFavoringDim=function(k,L,R){var O=m.TILING_PADDING_VERTICAL,M=m.TILING_PADDING_HORIZONTAL,B=m.TILING_COMPARE_BY,F={rows:[],rowWidth:[],rowHeight:[],width:0,height:L,verticalPadding:O,horizontalPadding:M,centerX:0,centerY:0};B&&(F.idealRowWidth=this.calcIdealRowWidth(k,R));var P=o(function(le){return le.rect.width*le.rect.height},"getNodeArea"),z=o(function(le,he){return P(he)-P(le)},"areaCompareFcn");k.sort(function(ne,le){var he=z;return F.idealRowWidth?(he=B,he(ne.id,le.id)):he(ne,le)});for(var $=0,H=0,Q=0;Q0&&(F+=k.horizontalPadding),k.rowWidth[R]=F,k.width0&&(P+=k.verticalPadding);var z=0;P>k.rowHeight[R]&&(z=k.rowHeight[R],k.rowHeight[R]=P,z=k.rowHeight[R]-z),k.height+=z,k.rows[R].push(L)},I.prototype.getShortestRowIndex=function(k){for(var L=-1,R=Number.MAX_VALUE,O=0;OR&&(L=O,R=k.rowWidth[O]);return L},I.prototype.canAddHorizontal=function(k,L,R){if(k.idealRowWidth){var O=k.rows.length-1,M=k.rowWidth[O];return M+L+k.horizontalPadding<=k.idealRowWidth}var B=this.getShortestRowIndex(k);if(B<0)return!0;var F=k.rowWidth[B];if(F+k.horizontalPadding+L<=k.width)return!0;var P=0;k.rowHeight[B]0&&(P=R+k.verticalPadding-k.rowHeight[B]);var z;k.width-F>=L+k.horizontalPadding?z=(k.height+P)/(F+L+k.horizontalPadding):z=(k.height+P)/k.width,P=R+k.verticalPadding;var $;return k.widthB&&L!=R){O.splice(-1,1),k.rows[R].push(M),k.rowWidth[L]=k.rowWidth[L]-B,k.rowWidth[R]=k.rowWidth[R]+B,k.width=k.rowWidth[instance.getLongestRowIndex(k)];for(var F=Number.MIN_VALUE,P=0;PF&&(F=O[P].height);L>0&&(F+=k.verticalPadding);var z=k.rowHeight[L]+k.rowHeight[R];k.rowHeight[L]=F,k.rowHeight[R]0)for(var ie=M;ie<=B;ie++)j[0]+=this.grid[ie][F-1].length+this.grid[ie][F].length-1;if(B0)for(var ie=F;ie<=P;ie++)j[3]+=this.grid[M-1][ie].length+this.grid[M][ie].length-1;for(var ne=T.MAX_VALUE,le,he,K=0;K{var u=l(551).FDLayoutNode,h=l(551).IMath;function f(p,m,g,y){u.call(this,p,m,g,y)}o(f,"CoSENode"),f.prototype=Object.create(u.prototype);for(var d in u)f[d]=u[d];f.prototype.calculateDisplacement=function(){var p=this.graphManager.getLayout();this.getChild()!=null&&this.fixedNodeWeight?(this.displacementX+=p.coolingFactor*(this.springForceX+this.repulsionForceX+this.gravitationForceX)/this.fixedNodeWeight,this.displacementY+=p.coolingFactor*(this.springForceY+this.repulsionForceY+this.gravitationForceY)/this.fixedNodeWeight):(this.displacementX+=p.coolingFactor*(this.springForceX+this.repulsionForceX+this.gravitationForceX)/this.noOfChildren,this.displacementY+=p.coolingFactor*(this.springForceY+this.repulsionForceY+this.gravitationForceY)/this.noOfChildren),Math.abs(this.displacementX)>p.coolingFactor*p.maxNodeDisplacement&&(this.displacementX=p.coolingFactor*p.maxNodeDisplacement*h.sign(this.displacementX)),Math.abs(this.displacementY)>p.coolingFactor*p.maxNodeDisplacement&&(this.displacementY=p.coolingFactor*p.maxNodeDisplacement*h.sign(this.displacementY)),this.child&&this.child.getNodes().length>0&&this.propogateDisplacementToChildren(this.displacementX,this.displacementY)},f.prototype.propogateDisplacementToChildren=function(p,m){for(var g=this.getChild().getNodes(),y,v=0;v{function u(g){if(Array.isArray(g)){for(var y=0,v=Array(g.length);y0){var ct=0;Ue.forEach(function(ot){xe=="horizontal"?(we.set(ot,x.has(ot)?b[x.get(ot)]:pe.get(ot)),ct+=we.get(ot)):(we.set(ot,x.has(ot)?w[x.get(ot)]:pe.get(ot)),ct+=we.get(ot))}),ct=ct/Ue.length,st.forEach(function(ot){q.has(ot)||we.set(ot,ct)})}else{var We=0;st.forEach(function(ot){xe=="horizontal"?We+=x.has(ot)?b[x.get(ot)]:pe.get(ot):We+=x.has(ot)?w[x.get(ot)]:pe.get(ot)}),We=We/st.length,st.forEach(function(ot){we.set(ot,We)})}});for(var qe=o(function(){var Ue=De.shift(),ct=V.get(Ue);ct.forEach(function(We){if(we.get(We.id)ot&&(ot=yt),ntYt&&(Yt=nt)}}catch(At){Mt=!0,xt=At}finally{try{!bt&&ut.return&&ut.return()}finally{if(Mt)throw xt}}var dn=(ct+ot)/2-(We+Yt)/2,Tt=!0,On=!1,tn=void 0;try{for(var _r=st[Symbol.iterator](),Dr;!(Tt=(Dr=_r.next()).done);Tt=!0){var Pn=Dr.value;we.set(Pn,we.get(Pn)+dn)}}catch(At){On=!0,tn=At}finally{try{!Tt&&_r.return&&_r.return()}finally{if(On)throw tn}}})}return we},"findAppropriatePositionForRelativePlacement"),D=o(function(V){var xe=0,q=0,pe=0,ve=0;if(V.forEach(function(Ve){Ve.left?b[x.get(Ve.left)]-b[x.get(Ve.right)]>=0?xe++:q++:w[x.get(Ve.top)]-w[x.get(Ve.bottom)]>=0?pe++:ve++}),xe>q&&pe>ve)for(var Pe=0;Peq)for(var _e=0;_eve)for(var we=0;we1)y.fixedNodeConstraint.forEach(function(oe,V){O[V]=[oe.position.x,oe.position.y],M[V]=[b[x.get(oe.nodeId)],w[x.get(oe.nodeId)]]}),B=!0;else if(y.alignmentConstraint)(function(){var oe=0;if(y.alignmentConstraint.vertical){for(var V=y.alignmentConstraint.vertical,xe=o(function(we){var Ve=new Set;V[we].forEach(function(at){Ve.add(at)});var De=new Set([].concat(u(Ve)).filter(function(at){return P.has(at)})),qe=void 0;De.size>0?qe=b[x.get(De.values().next().value)]:qe=_(Ve).x,V[we].forEach(function(at){O[oe]=[qe,w[x.get(at)]],M[oe]=[b[x.get(at)],w[x.get(at)]],oe++})},"_loop2"),q=0;q0?qe=b[x.get(De.values().next().value)]:qe=_(Ve).y,pe[we].forEach(function(at){O[oe]=[b[x.get(at)],qe],M[oe]=[b[x.get(at)],w[x.get(at)]],oe++})},"_loop3"),Pe=0;PeQ&&(Q=H[ie].length,j=ie);if(Q<$.size/2)D(y.relativePlacementConstraint),B=!1,F=!1;else{var ne=new Map,le=new Map,he=[];H[j].forEach(function(oe){z.get(oe).forEach(function(V){V.direction=="horizontal"?(ne.has(oe)?ne.get(oe).push(V):ne.set(oe,[V]),ne.has(V.id)||ne.set(V.id,[]),he.push({left:oe,right:V.id})):(le.has(oe)?le.get(oe).push(V):le.set(oe,[V]),le.has(V.id)||le.set(V.id,[]),he.push({top:oe,bottom:V.id}))})}),D(he),F=!1;var K=I(ne,"horizontal"),X=I(le,"vertical");H[j].forEach(function(oe,V){M[V]=[b[x.get(oe)],w[x.get(oe)]],O[V]=[],K.has(oe)?O[V][0]=K.get(oe):O[V][0]=b[x.get(oe)],X.has(oe)?O[V][1]=X.get(oe):O[V][1]=w[x.get(oe)]}),B=!0}}if(B){for(var te=void 0,J=d.transpose(O),se=d.transpose(M),ue=0;ue0){var ze={x:0,y:0};y.fixedNodeConstraint.forEach(function(oe,V){var xe={x:b[x.get(oe.nodeId)],y:w[x.get(oe.nodeId)]},q=oe.position,pe=S(q,xe);ze.x+=pe.x,ze.y+=pe.y}),ze.x/=y.fixedNodeConstraint.length,ze.y/=y.fixedNodeConstraint.length,b.forEach(function(oe,V){b[V]+=ze.x}),w.forEach(function(oe,V){w[V]+=ze.y}),y.fixedNodeConstraint.forEach(function(oe){b[x.get(oe.nodeId)]=oe.position.x,w[x.get(oe.nodeId)]=oe.position.y})}if(y.alignmentConstraint){if(y.alignmentConstraint.vertical)for(var He=y.alignmentConstraint.vertical,$e=o(function(V){var xe=new Set;He[V].forEach(function(ve){xe.add(ve)});var q=new Set([].concat(u(xe)).filter(function(ve){return P.has(ve)})),pe=void 0;q.size>0?pe=b[x.get(q.values().next().value)]:pe=_(xe).x,xe.forEach(function(ve){P.has(ve)||(b[x.get(ve)]=pe)})},"_loop4"),Re=0;Re0?pe=w[x.get(q.values().next().value)]:pe=_(xe).y,xe.forEach(function(ve){P.has(ve)||(w[x.get(ve)]=pe)})},"_loop5"),W=0;W{a.exports=t}},r={};function n(a){var s=r[a];if(s!==void 0)return s.exports;var l=r[a]={exports:{}};return e[a](l,l.exports,n),l.exports}o(n,"__webpack_require__");var i=n(45);return i})()})});var Pve=Mi((E4,gF)=>{"use strict";o(function(e,r){typeof E4=="object"&&typeof gF=="object"?gF.exports=r(mF()):typeof define=="function"&&define.amd?define(["cose-base"],r):typeof E4=="object"?E4.cytoscapeFcose=r(mF()):e.cytoscapeFcose=r(e.coseBase)},"webpackUniversalModuleDefinition")(E4,function(t){return(()=>{"use strict";var e={658:a=>{a.exports=Object.assign!=null?Object.assign.bind(Object):function(s){for(var l=arguments.length,u=Array(l>1?l-1:0),h=1;h{var u=function(){function d(p,m){var g=[],y=!0,v=!1,x=void 0;try{for(var b=p[Symbol.iterator](),w;!(y=(w=b.next()).done)&&(g.push(w.value),!(m&&g.length===m));y=!0);}catch(C){v=!0,x=C}finally{try{!y&&b.return&&b.return()}finally{if(v)throw x}}return g}return o(d,"sliceIterator"),function(p,m){if(Array.isArray(p))return p;if(Symbol.iterator in Object(p))return d(p,m);throw new TypeError("Invalid attempt to destructure non-iterable instance")}}(),h=l(140).layoutBase.LinkedList,f={};f.getTopMostNodes=function(d){for(var p={},m=0;m0&&B.merge($)});for(var F=0;F1){w=x[0],C=w.connectedEdges().length,x.forEach(function(M){M.connectedEdges().length0&&g.set("dummy"+(g.size+1),A),S},f.relocateComponent=function(d,p,m){if(!m.fixedNodeConstraint){var g=Number.POSITIVE_INFINITY,y=Number.NEGATIVE_INFINITY,v=Number.POSITIVE_INFINITY,x=Number.NEGATIVE_INFINITY;if(m.quality=="draft"){var b=!0,w=!1,C=void 0;try{for(var T=p.nodeIndexes[Symbol.iterator](),E;!(b=(E=T.next()).done);b=!0){var A=E.value,S=u(A,2),_=S[0],I=S[1],D=m.cy.getElementById(_);if(D){var k=D.boundingBox(),L=p.xCoords[I]-k.w/2,R=p.xCoords[I]+k.w/2,O=p.yCoords[I]-k.h/2,M=p.yCoords[I]+k.h/2;Ly&&(y=R),Ox&&(x=M)}}}catch($){w=!0,C=$}finally{try{!b&&T.return&&T.return()}finally{if(w)throw C}}var B=d.x-(y+g)/2,F=d.y-(x+v)/2;p.xCoords=p.xCoords.map(function($){return $+B}),p.yCoords=p.yCoords.map(function($){return $+F})}else{Object.keys(p).forEach(function($){var H=p[$],Q=H.getRect().x,j=H.getRect().x+H.getRect().width,ie=H.getRect().y,ne=H.getRect().y+H.getRect().height;Qy&&(y=j),iex&&(x=ne)});var P=d.x-(y+g)/2,z=d.y-(x+v)/2;Object.keys(p).forEach(function($){var H=p[$];H.setCenter(H.getCenterX()+P,H.getCenterY()+z)})}}},f.calcBoundingBox=function(d,p,m,g){for(var y=Number.MAX_SAFE_INTEGER,v=Number.MIN_SAFE_INTEGER,x=Number.MAX_SAFE_INTEGER,b=Number.MIN_SAFE_INTEGER,w=void 0,C=void 0,T=void 0,E=void 0,A=d.descendants().not(":parent"),S=A.length,_=0;_w&&(y=w),vT&&(x=T),b{var u=l(548),h=l(140).CoSELayout,f=l(140).CoSENode,d=l(140).layoutBase.PointD,p=l(140).layoutBase.DimensionD,m=l(140).layoutBase.LayoutConstants,g=l(140).layoutBase.FDLayoutConstants,y=l(140).CoSEConstants,v=o(function(b,w){var C=b.cy,T=b.eles,E=T.nodes(),A=T.edges(),S=void 0,_=void 0,I=void 0,D={};b.randomize&&(S=w.nodeIndexes,_=w.xCoords,I=w.yCoords);var k=o(function($){return typeof $=="function"},"isFn"),L=o(function($,H){return k($)?$(H):$},"optFn"),R=u.calcParentsWithoutChildren(C,T),O=o(function z($,H,Q,j){for(var ie=H.length,ne=0;ne0){var J=void 0;J=Q.getGraphManager().add(Q.newGraph(),K),z(J,he,Q,j)}}},"processChildrenList"),M=o(function($,H,Q){for(var j=0,ie=0,ne=0;ne0?y.DEFAULT_EDGE_LENGTH=g.DEFAULT_EDGE_LENGTH=j/ie:k(b.idealEdgeLength)?y.DEFAULT_EDGE_LENGTH=g.DEFAULT_EDGE_LENGTH=50:y.DEFAULT_EDGE_LENGTH=g.DEFAULT_EDGE_LENGTH=b.idealEdgeLength,y.MIN_REPULSION_DIST=g.MIN_REPULSION_DIST=g.DEFAULT_EDGE_LENGTH/10,y.DEFAULT_RADIAL_SEPARATION=g.DEFAULT_EDGE_LENGTH)},"processEdges"),B=o(function($,H){H.fixedNodeConstraint&&($.constraints.fixedNodeConstraint=H.fixedNodeConstraint),H.alignmentConstraint&&($.constraints.alignmentConstraint=H.alignmentConstraint),H.relativePlacementConstraint&&($.constraints.relativePlacementConstraint=H.relativePlacementConstraint)},"processConstraints");b.nestingFactor!=null&&(y.PER_LEVEL_IDEAL_EDGE_LENGTH_FACTOR=g.PER_LEVEL_IDEAL_EDGE_LENGTH_FACTOR=b.nestingFactor),b.gravity!=null&&(y.DEFAULT_GRAVITY_STRENGTH=g.DEFAULT_GRAVITY_STRENGTH=b.gravity),b.numIter!=null&&(y.MAX_ITERATIONS=g.MAX_ITERATIONS=b.numIter),b.gravityRange!=null&&(y.DEFAULT_GRAVITY_RANGE_FACTOR=g.DEFAULT_GRAVITY_RANGE_FACTOR=b.gravityRange),b.gravityCompound!=null&&(y.DEFAULT_COMPOUND_GRAVITY_STRENGTH=g.DEFAULT_COMPOUND_GRAVITY_STRENGTH=b.gravityCompound),b.gravityRangeCompound!=null&&(y.DEFAULT_COMPOUND_GRAVITY_RANGE_FACTOR=g.DEFAULT_COMPOUND_GRAVITY_RANGE_FACTOR=b.gravityRangeCompound),b.initialEnergyOnIncremental!=null&&(y.DEFAULT_COOLING_FACTOR_INCREMENTAL=g.DEFAULT_COOLING_FACTOR_INCREMENTAL=b.initialEnergyOnIncremental),b.tilingCompareBy!=null&&(y.TILING_COMPARE_BY=b.tilingCompareBy),b.quality=="proof"?m.QUALITY=2:m.QUALITY=0,y.NODE_DIMENSIONS_INCLUDE_LABELS=g.NODE_DIMENSIONS_INCLUDE_LABELS=m.NODE_DIMENSIONS_INCLUDE_LABELS=b.nodeDimensionsIncludeLabels,y.DEFAULT_INCREMENTAL=g.DEFAULT_INCREMENTAL=m.DEFAULT_INCREMENTAL=!b.randomize,y.ANIMATE=g.ANIMATE=m.ANIMATE=b.animate,y.TILE=b.tile,y.TILING_PADDING_VERTICAL=typeof b.tilingPaddingVertical=="function"?b.tilingPaddingVertical.call():b.tilingPaddingVertical,y.TILING_PADDING_HORIZONTAL=typeof b.tilingPaddingHorizontal=="function"?b.tilingPaddingHorizontal.call():b.tilingPaddingHorizontal,y.DEFAULT_INCREMENTAL=g.DEFAULT_INCREMENTAL=m.DEFAULT_INCREMENTAL=!0,y.PURE_INCREMENTAL=!b.randomize,m.DEFAULT_UNIFORM_LEAF_NODE_SIZES=b.uniformNodeDimensions,b.step=="transformed"&&(y.TRANSFORM_ON_CONSTRAINT_HANDLING=!0,y.ENFORCE_CONSTRAINTS=!1,y.APPLY_LAYOUT=!1),b.step=="enforced"&&(y.TRANSFORM_ON_CONSTRAINT_HANDLING=!1,y.ENFORCE_CONSTRAINTS=!0,y.APPLY_LAYOUT=!1),b.step=="cose"&&(y.TRANSFORM_ON_CONSTRAINT_HANDLING=!1,y.ENFORCE_CONSTRAINTS=!1,y.APPLY_LAYOUT=!0),b.step=="all"&&(b.randomize?y.TRANSFORM_ON_CONSTRAINT_HANDLING=!0:y.TRANSFORM_ON_CONSTRAINT_HANDLING=!1,y.ENFORCE_CONSTRAINTS=!0,y.APPLY_LAYOUT=!0),b.fixedNodeConstraint||b.alignmentConstraint||b.relativePlacementConstraint?y.TREE_REDUCTION_ON_INCREMENTAL=!1:y.TREE_REDUCTION_ON_INCREMENTAL=!0;var F=new h,P=F.newGraphManager();return O(P.addRoot(),u.getTopMostNodes(E),F,b),M(F,P,A),B(F,b),F.runLayout(),D},"coseLayout");a.exports={coseLayout:v}},212:(a,s,l)=>{var u=function(){function b(w,C){for(var T=0;T0)if(M){var P=d.getTopMostNodes(T.eles.nodes());if(k=d.connectComponents(E,T.eles,P),k.forEach(function(ce){var ae=ce.boundingBox();L.push({x:ae.x1+ae.w/2,y:ae.y1+ae.h/2})}),T.randomize&&k.forEach(function(ce){T.eles=ce,S.push(m(T))}),T.quality=="default"||T.quality=="proof"){var z=E.collection();if(T.tile){var $=new Map,H=[],Q=[],j=0,ie={nodeIndexes:$,xCoords:H,yCoords:Q},ne=[];if(k.forEach(function(ce,ae){ce.edges().length==0&&(ce.nodes().forEach(function(Oe,ge){z.merge(ce.nodes()[ge]),Oe.isParent()||(ie.nodeIndexes.set(ce.nodes()[ge].id(),j++),ie.xCoords.push(ce.nodes()[0].position().x),ie.yCoords.push(ce.nodes()[0].position().y))}),ne.push(ae))}),z.length>1){var le=z.boundingBox();L.push({x:le.x1+le.w/2,y:le.y1+le.h/2}),k.push(z),S.push(ie);for(var he=ne.length-1;he>=0;he--)k.splice(ne[he],1),S.splice(ne[he],1),L.splice(ne[he],1)}}k.forEach(function(ce,ae){T.eles=ce,D.push(y(T,S[ae])),d.relocateComponent(L[ae],D[ae],T)})}else k.forEach(function(ce,ae){d.relocateComponent(L[ae],S[ae],T)});var K=new Set;if(k.length>1){var X=[],te=A.filter(function(ce){return ce.css("display")=="none"});k.forEach(function(ce,ae){var Oe=void 0;if(T.quality=="draft"&&(Oe=S[ae].nodeIndexes),ce.nodes().not(te).length>0){var ge={};ge.edges=[],ge.nodes=[];var ze=void 0;ce.nodes().not(te).forEach(function(He){if(T.quality=="draft")if(!He.isParent())ze=Oe.get(He.id()),ge.nodes.push({x:S[ae].xCoords[ze]-He.boundingbox().w/2,y:S[ae].yCoords[ze]-He.boundingbox().h/2,width:He.boundingbox().w,height:He.boundingbox().h});else{var $e=d.calcBoundingBox(He,S[ae].xCoords,S[ae].yCoords,Oe);ge.nodes.push({x:$e.topLeftX,y:$e.topLeftY,width:$e.width,height:$e.height})}else D[ae][He.id()]&&ge.nodes.push({x:D[ae][He.id()].getLeft(),y:D[ae][He.id()].getTop(),width:D[ae][He.id()].getWidth(),height:D[ae][He.id()].getHeight()})}),ce.edges().forEach(function(He){var $e=He.source(),Re=He.target();if($e.css("display")!="none"&&Re.css("display")!="none")if(T.quality=="draft"){var Ie=Oe.get($e.id()),be=Oe.get(Re.id()),W=[],de=[];if($e.isParent()){var re=d.calcBoundingBox($e,S[ae].xCoords,S[ae].yCoords,Oe);W.push(re.topLeftX+re.width/2),W.push(re.topLeftY+re.height/2)}else W.push(S[ae].xCoords[Ie]),W.push(S[ae].yCoords[Ie]);if(Re.isParent()){var oe=d.calcBoundingBox(Re,S[ae].xCoords,S[ae].yCoords,Oe);de.push(oe.topLeftX+oe.width/2),de.push(oe.topLeftY+oe.height/2)}else de.push(S[ae].xCoords[be]),de.push(S[ae].yCoords[be]);ge.edges.push({startX:W[0],startY:W[1],endX:de[0],endY:de[1]})}else D[ae][$e.id()]&&D[ae][Re.id()]&&ge.edges.push({startX:D[ae][$e.id()].getCenterX(),startY:D[ae][$e.id()].getCenterY(),endX:D[ae][Re.id()].getCenterX(),endY:D[ae][Re.id()].getCenterY()})}),ge.nodes.length>0&&(X.push(ge),K.add(ae))}});var J=O.packComponents(X,T.randomize).shifts;if(T.quality=="draft")S.forEach(function(ce,ae){var Oe=ce.xCoords.map(function(ze){return ze+J[ae].dx}),ge=ce.yCoords.map(function(ze){return ze+J[ae].dy});ce.xCoords=Oe,ce.yCoords=ge});else{var se=0;K.forEach(function(ce){Object.keys(D[ce]).forEach(function(ae){var Oe=D[ce][ae];Oe.setCenter(Oe.getCenterX()+J[se].dx,Oe.getCenterY()+J[se].dy)}),se++})}}}else{var B=T.eles.boundingBox();if(L.push({x:B.x1+B.w/2,y:B.y1+B.h/2}),T.randomize){var F=m(T);S.push(F)}T.quality=="default"||T.quality=="proof"?(D.push(y(T,S[0])),d.relocateComponent(L[0],D[0],T)):d.relocateComponent(L[0],S[0],T)}var ue=o(function(ae,Oe){if(T.quality=="default"||T.quality=="proof"){typeof ae=="number"&&(ae=Oe);var ge=void 0,ze=void 0,He=ae.data("id");return D.forEach(function(Re){He in Re&&(ge={x:Re[He].getRect().getCenterX(),y:Re[He].getRect().getCenterY()},ze=Re[He])}),T.nodeDimensionsIncludeLabels&&(ze.labelWidth&&(ze.labelPosHorizontal=="left"?ge.x+=ze.labelWidth/2:ze.labelPosHorizontal=="right"&&(ge.x-=ze.labelWidth/2)),ze.labelHeight&&(ze.labelPosVertical=="top"?ge.y+=ze.labelHeight/2:ze.labelPosVertical=="bottom"&&(ge.y-=ze.labelHeight/2))),ge==null&&(ge={x:ae.position("x"),y:ae.position("y")}),{x:ge.x,y:ge.y}}else{var $e=void 0;return S.forEach(function(Re){var Ie=Re.nodeIndexes.get(ae.id());Ie!=null&&($e={x:Re.xCoords[Ie],y:Re.yCoords[Ie]})}),$e==null&&($e={x:ae.position("x"),y:ae.position("y")}),{x:$e.x,y:$e.y}}},"getPositions");if(T.quality=="default"||T.quality=="proof"||T.randomize){var Z=d.calcParentsWithoutChildren(E,A),Se=A.filter(function(ce){return ce.css("display")=="none"});T.eles=A.not(Se),A.nodes().not(":parent").not(Se).layoutPositions(C,T,ue),Z.length>0&&Z.forEach(function(ce){ce.position(ue(ce))})}else console.log("If randomize option is set to false, then quality option must be 'default' or 'proof'.")},"run")}]),b}();a.exports=x},657:(a,s,l)=>{var u=l(548),h=l(140).layoutBase.Matrix,f=l(140).layoutBase.SVD,d=o(function(m){var g=m.cy,y=m.eles,v=y.nodes(),x=y.nodes(":parent"),b=new Map,w=new Map,C=new Map,T=[],E=[],A=[],S=[],_=[],I=[],D=[],k=[],L=void 0,R=void 0,O=1e8,M=1e-9,B=m.piTol,F=m.samplingType,P=m.nodeSeparation,z=void 0,$=o(function(){for(var xe=0,q=0,pe=!1;q=Pe;){we=ve[Pe++];for(var st=T[we],Ue=0;Ueqe&&(qe=_[We],at=We)}return at},"BFS"),Q=o(function(xe){var q=void 0;if(xe){q=Math.floor(Math.random()*R),L=q;for(var ve=0;ve=1)break;qe=De}for(var st=0;st=1)break;qe=De}for(var ct=0;ct0&&(q.isParent()?T[xe].push(C.get(q.id())):T[xe].push(q.id()))})});var Z=o(function(xe){var q=w.get(xe),pe=void 0;b.get(xe).forEach(function(ve){g.getElementById(ve).isParent()?pe=C.get(ve):pe=ve,T[q].push(pe),T[w.get(pe)].push(xe)})},"_loop"),Se=!0,ce=!1,ae=void 0;try{for(var Oe=b.keys()[Symbol.iterator](),ge;!(Se=(ge=Oe.next()).done);Se=!0){var ze=ge.value;Z(ze)}}catch(V){ce=!0,ae=V}finally{try{!Se&&Oe.return&&Oe.return()}finally{if(ce)throw ae}}R=w.size;var He=void 0;if(R>2){z=R{var u=l(212),h=o(function(d){d&&d("layout","fcose",u)},"register");typeof cytoscape<"u"&&h(cytoscape),a.exports=h},140:a=>{a.exports=t}},r={};function n(a){var s=r[a];if(s!==void 0)return s.exports;var l=r[a]={exports:{}};return e[a](l,l.exports,n),l.exports}o(n,"__webpack_require__");var i=n(579);return i})()})});var dy,Zp,yF=N(()=>{"use strict";tu();dy=o(t=>`${t}`,"wrapIcon"),Zp={prefix:"mermaid-architecture",height:80,width:80,icons:{database:{body:dy('')},server:{body:dy('')},disk:{body:dy('')},internet:{body:dy('')},cloud:{body:dy('')},unknown:OC,blank:{body:dy("")}}}});var Bve,Fve,$ve,zve,Gve=N(()=>{"use strict";tu();zt();to();w4();yF();oC();Bve=o(async function(t,e){let r=Li("padding"),n=Li("iconSize"),i=n/2,a=n/6,s=a/2;await Promise.all(e.edges().map(async l=>{let{source:u,sourceDir:h,sourceArrow:f,sourceGroup:d,target:p,targetDir:m,targetArrow:g,targetGroup:y,label:v}=sC(l),{x,y:b}=l[0].sourceEndpoint(),{x:w,y:C}=l[0].midpoint(),{x:T,y:E}=l[0].targetEndpoint(),A=r+4;if(d&&(Ha(h)?x+=h==="L"?-A:A:b+=h==="T"?-A:A+18),y&&(Ha(m)?T+=m==="L"?-A:A:E+=m==="T"?-A:A+18),!d&&Qp.getNode(u)?.type==="junction"&&(Ha(h)?x+=h==="L"?i:-i:b+=h==="T"?i:-i),!y&&Qp.getNode(p)?.type==="junction"&&(Ha(m)?T+=m==="L"?i:-i:E+=m==="T"?i:-i),l[0]._private.rscratch){let S=t.insert("g");if(S.insert("path").attr("d",`M ${x},${b} L ${w},${C} L${T},${E} `).attr("class","edge"),f){let _=Ha(h)?v4[h](x,a):x-s,I=Zc(h)?v4[h](b,a):b-s;S.insert("polygon").attr("points",cF[h](a)).attr("transform",`translate(${_},${I})`).attr("class","arrow")}if(g){let _=Ha(m)?v4[m](T,a):T-s,I=Zc(m)?v4[m](E,a):E-s;S.insert("polygon").attr("points",cF[m](a)).attr("transform",`translate(${_},${I})`).attr("class","arrow")}if(v){let _=x4(h,m)?"XY":Ha(h)?"X":"Y",I=0;_==="X"?I=Math.abs(x-T):_==="Y"?I=Math.abs(b-E)/1.5:I=Math.abs(x-T)/2;let D=S.append("g");if(await Hn(D,v,{useHtmlLabels:!1,width:I,classes:"architecture-service-label"},me()),D.attr("dy","1em").attr("alignment-baseline","middle").attr("dominant-baseline","middle").attr("text-anchor","middle"),_==="X")D.attr("transform","translate("+w+", "+C+")");else if(_==="Y")D.attr("transform","translate("+w+", "+C+") rotate(-90)");else if(_==="XY"){let k=b4(h,m);if(k&&Sve(k)){let L=D.node().getBoundingClientRect(),[R,O]=Ave(k);D.attr("dominant-baseline","auto").attr("transform",`rotate(${-1*R*O*45})`);let M=D.node().getBoundingClientRect();D.attr("transform",` + translate(${w}, ${C-L.height/2}) + translate(${R*M.width/2}, ${O*M.height/2}) + rotate(${-1*R*O*45}, 0, ${L.height/2}) + `)}}}}}))},"drawEdges"),Fve=o(async function(t,e){let n=Li("padding")*.75,i=Li("fontSize"),s=Li("iconSize")/2;await Promise.all(e.nodes().map(async l=>{let u=Ff(l);if(u.type==="group"){let{h,w:f,x1:d,y1:p}=l.boundingBox();t.append("rect").attr("x",d+s).attr("y",p+s).attr("width",f).attr("height",h).attr("class","node-bkg");let m=t.append("g"),g=d,y=p;if(u.icon){let v=m.append("g");v.html(`${await wo(u.icon,{height:n,width:n,fallbackPrefix:Zp.prefix})}`),v.attr("transform","translate("+(g+s+1)+", "+(y+s+1)+")"),g+=n,y+=i/2-1-2}if(u.label){let v=m.append("g");await Hn(v,u.label,{useHtmlLabels:!1,width:f,classes:"architecture-service-label"},me()),v.attr("dy","1em").attr("alignment-baseline","middle").attr("dominant-baseline","start").attr("text-anchor","start"),v.attr("transform","translate("+(g+s+4)+", "+(y+s+2)+")")}}}))},"drawGroups"),$ve=o(async function(t,e,r){for(let n of r){let i=e.append("g"),a=Li("iconSize");if(n.title){let h=i.append("g");await Hn(h,n.title,{useHtmlLabels:!1,width:a*1.5,classes:"architecture-service-label"},me()),h.attr("dy","1em").attr("alignment-baseline","middle").attr("dominant-baseline","middle").attr("text-anchor","middle"),h.attr("transform","translate("+a/2+", "+a+")")}let s=i.append("g");if(n.icon)s.html(`${await wo(n.icon,{height:a,width:a,fallbackPrefix:Zp.prefix})}`);else if(n.iconText){s.html(`${await wo("blank",{height:a,width:a,fallbackPrefix:Zp.prefix})}`);let d=s.append("g").append("foreignObject").attr("width",a).attr("height",a).append("div").attr("class","node-icon-text").attr("style",`height: ${a}px;`).append("div").html(n.iconText),p=parseInt(window.getComputedStyle(d.node(),null).getPropertyValue("font-size").replace(/\D/g,""))??16;d.attr("style",`-webkit-line-clamp: ${Math.floor((a-2)/p)};`)}else s.append("path").attr("class","node-bkg").attr("id","node-"+n.id).attr("d",`M0 ${a} v${-a} q0,-5 5,-5 h${a} q5,0 5,5 v${a} H0 Z`);i.attr("class","architecture-service");let{width:l,height:u}=i._groups[0][0].getBBox();n.width=l,n.height=u,t.setElementForId(n.id,i)}return 0},"drawServices"),zve=o(function(t,e,r){r.forEach(n=>{let i=e.append("g"),a=Li("iconSize");i.append("g").append("rect").attr("id","node-"+n.id).attr("fill-opacity","0").attr("width",a).attr("height",a),i.attr("class","architecture-junction");let{width:l,height:u}=i._groups[0][0].getBBox();i.width=l,i.height=u,t.setElementForId(n.id,i)})},"drawJunctions")});function Srt(t,e){t.forEach(r=>{e.add({group:"nodes",data:{type:"service",id:r.id,icon:r.icon,label:r.title,parent:r.in,width:Li("iconSize"),height:Li("iconSize")},classes:"node-service"})})}function Crt(t,e){t.forEach(r=>{e.add({group:"nodes",data:{type:"junction",id:r.id,parent:r.in,width:Li("iconSize"),height:Li("iconSize")},classes:"node-junction"})})}function Art(t,e){e.nodes().map(r=>{let n=Ff(r);if(n.type==="group")return;n.x=r.position().x,n.y=r.position().y,t.getElementById(n.id).attr("transform","translate("+(n.x||0)+","+(n.y||0)+")")})}function _rt(t,e){t.forEach(r=>{e.add({group:"nodes",data:{type:"group",id:r.id,icon:r.icon,label:r.title,parent:r.in},classes:"node-group"})})}function Drt(t,e){t.forEach(r=>{let{lhsId:n,rhsId:i,lhsInto:a,lhsGroup:s,rhsInto:l,lhsDir:u,rhsDir:h,rhsGroup:f,title:d}=r,p=x4(r.lhsDir,r.rhsDir)?"segments":"straight",m={id:`${n}-${i}`,label:d,source:n,sourceDir:u,sourceArrow:a,sourceGroup:s,sourceEndpoint:u==="L"?"0 50%":u==="R"?"100% 50%":u==="T"?"50% 0":"50% 100%",target:i,targetDir:h,targetArrow:l,targetGroup:f,targetEndpoint:h==="L"?"0 50%":h==="R"?"100% 50%":h==="T"?"50% 0":"50% 100%"};e.add({group:"edges",data:m,classes:p})})}function Lrt(t,e,r){let n=o((l,u)=>Object.entries(l).reduce((h,[f,d])=>{let p=0,m=Object.entries(d);if(m.length===1)return h[f]=m[0][1],h;for(let g=0;g{let u={},h={};return Object.entries(l).forEach(([f,[d,p]])=>{let m=t.getNode(f)?.in??"default";u[p]??={},u[p][m]??=[],u[p][m].push(f),h[d]??={},h[d][m]??=[],h[d][m].push(f)}),{horiz:Object.values(n(u,"horizontal")).filter(f=>f.length>1),vert:Object.values(n(h,"vertical")).filter(f=>f.length>1)}}),[a,s]=i.reduce(([l,u],{horiz:h,vert:f})=>[[...l,...h],[...u,...f]],[[],[]]);return{horizontal:a,vertical:s}}function Rrt(t){let e=[],r=o(i=>`${i[0]},${i[1]}`,"posToStr"),n=o(i=>i.split(",").map(a=>parseInt(a)),"strToPos");return t.forEach(i=>{let a=Object.fromEntries(Object.entries(i).map(([h,f])=>[r(f),h])),s=[r([0,0])],l={},u={L:[-1,0],R:[1,0],T:[0,1],B:[0,-1]};for(;s.length>0;){let h=s.shift();if(h){l[h]=1;let f=a[h];if(f){let d=n(h);Object.entries(u).forEach(([p,m])=>{let g=r([d[0]+m[0],d[1]+m[1]]),y=a[g];y&&!l[g]&&(s.push(g),e.push({[lF[p]]:y,[lF[Eve(p)]]:f,gap:1.5*Li("iconSize")}))})}}}}),e}function Nrt(t,e,r,n,i,{spatialMaps:a,groupAlignments:s}){return new Promise(l=>{let u=Ge("body").append("div").attr("id","cy").attr("style","display:none"),h=rl({container:document.getElementById("cy"),style:[{selector:"edge",style:{"curve-style":"straight",label:"data(label)","source-endpoint":"data(sourceEndpoint)","target-endpoint":"data(targetEndpoint)"}},{selector:"edge.segments",style:{"curve-style":"segments","segment-weights":"0","segment-distances":[.5],"edge-distances":"endpoints","source-endpoint":"data(sourceEndpoint)","target-endpoint":"data(targetEndpoint)"}},{selector:"node",style:{"compound-sizing-wrt-labels":"include"}},{selector:"node[label]",style:{"text-valign":"bottom","text-halign":"center","font-size":`${Li("fontSize")}px`}},{selector:".node-service",style:{label:"data(label)",width:"data(width)",height:"data(height)"}},{selector:".node-junction",style:{width:"data(width)",height:"data(height)"}},{selector:".node-group",style:{padding:`${Li("padding")}px`}}]});u.remove(),_rt(r,h),Srt(t,h),Crt(e,h),Drt(n,h);let f=Lrt(i,a,s),d=Rrt(a),p=h.layout({name:"fcose",quality:"proof",styleEnabled:!1,animate:!1,nodeDimensionsIncludeLabels:!1,idealEdgeLength(m){let[g,y]=m.connectedNodes(),{parent:v}=Ff(g),{parent:x}=Ff(y);return v===x?1.5*Li("iconSize"):.5*Li("iconSize")},edgeElasticity(m){let[g,y]=m.connectedNodes(),{parent:v}=Ff(g),{parent:x}=Ff(y);return v===x?.45:.001},alignmentConstraint:f,relativePlacementConstraint:d});p.one("layoutstop",()=>{function m(g,y,v,x){let b,w,{x:C,y:T}=g,{x:E,y:A}=y;w=(x-T+(C-v)*(T-A)/(C-E))/Math.sqrt(1+Math.pow((T-A)/(C-E),2)),b=Math.sqrt(Math.pow(x-T,2)+Math.pow(v-C,2)-Math.pow(w,2));let S=Math.sqrt(Math.pow(E-C,2)+Math.pow(A-T,2));b=b/S;let _=(E-C)*(x-T)-(A-T)*(v-C);switch(!0){case _>=0:_=1;break;case _<0:_=-1;break}let I=(E-C)*(v-C)+(A-T)*(x-T);switch(!0){case I>=0:I=1;break;case I<0:I=-1;break}return w=Math.abs(w)*_,b=b*I,{distances:w,weights:b}}o(m,"getSegmentWeights"),h.startBatch();for(let g of Object.values(h.edges()))if(g.data?.()){let{x:y,y:v}=g.source().position(),{x,y:b}=g.target().position();if(y!==x&&v!==b){let w=g.sourceEndpoint(),C=g.targetEndpoint(),{sourceDir:T}=sC(g),[E,A]=Zc(T)?[w.x,C.y]:[C.x,w.y],{weights:S,distances:_}=m(w,C,E,A);g.style("segment-distances",_),g.style("segment-weights",S)}}h.endBatch(),p.run()}),p.run(),h.ready(m=>{Y.info("Ready",m),l(h)})})}var Vve,Mrt,Uve,Hve=N(()=>{"use strict";tu();kB();Vve=Sa(Pve(),1);dr();vt();Vc();Ei();w4();yF();oC();Gve();P4([{name:Zp.prefix,icons:Zp}]);rl.use(Vve.default);o(Srt,"addServices");o(Crt,"addJunctions");o(Art,"positionNodes");o(_rt,"addGroups");o(Drt,"addEdges");o(Lrt,"getAlignments");o(Rrt,"getRelativeConstraints");o(Nrt,"layoutArchitecture");Mrt=o(async(t,e,r,n)=>{let i=n.db,a=i.getServices(),s=i.getJunctions(),l=i.getGroups(),u=i.getEdges(),h=i.getDataStructures(),f=sa(e),d=f.append("g");d.attr("class","architecture-edges");let p=f.append("g");p.attr("class","architecture-services");let m=f.append("g");m.attr("class","architecture-groups"),await $ve(i,p,a),zve(i,p,s);let g=await Nrt(a,s,l,u,i,h);await Bve(d,g),await Fve(m,g),Art(i,g),Ao(void 0,f,Li("padding"),Li("useMaxWidth"))},"draw"),Uve={draw:Mrt}});var Wve={};hr(Wve,{diagram:()=>Irt});var Irt,qve=N(()=>{"use strict";Mve();w4();Ove();Hve();Irt={parser:Nve,db:Qp,renderer:Uve,styles:Ive}});var bnt={};hr(bnt,{default:()=>xnt});tu();PC();Xf();var YX="c4",PCe=o(t=>/^\s*C4Context|C4Container|C4Component|C4Dynamic|C4Deployment/.test(t),"detector"),BCe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(qX(),WX));return{id:YX,diagram:t}},"loader"),FCe={id:YX,detector:PCe,loader:BCe},XX=FCe;var Xie="flowchart",xOe=o((t,e)=>e?.flowchart?.defaultRenderer==="dagre-wrapper"||e?.flowchart?.defaultRenderer==="elk"?!1:/^\s*graph/.test(t),"detector"),bOe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(ak(),ik));return{id:Xie,diagram:t}},"loader"),wOe={id:Xie,detector:xOe,loader:bOe},jie=wOe;var Kie="flowchart-v2",TOe=o((t,e)=>e?.flowchart?.defaultRenderer==="dagre-d3"?!1:(e?.flowchart?.defaultRenderer==="elk"&&(e.layout="elk"),/^\s*graph/.test(t)&&e?.flowchart?.defaultRenderer==="dagre-wrapper"?!0:/^\s*flowchart/.test(t)),"detector"),kOe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(ak(),ik));return{id:Kie,diagram:t}},"loader"),EOe={id:Kie,detector:TOe,loader:kOe},Qie=EOe;var sae="er",DOe=o(t=>/^\s*erDiagram/.test(t),"detector"),LOe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(aae(),iae));return{id:sae,diagram:t}},"loader"),ROe={id:sae,detector:DOe,loader:LOe},oae=ROe;var uue="gitGraph",tze=o(t=>/^\s*gitGraph/.test(t),"detector"),rze=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(cue(),lue));return{id:uue,diagram:t}},"loader"),nze={id:uue,detector:tze,loader:rze},hue=nze;var Gue="gantt",Hze=o(t=>/^\s*gantt/.test(t),"detector"),Wze=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(zue(),$ue));return{id:Gue,diagram:t}},"loader"),qze={id:Gue,detector:Hze,loader:Wze},Vue=qze;var Que="info",Zze=o(t=>/^\s*info/.test(t),"detector"),Jze=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(Kue(),jue));return{id:Que,diagram:t}},"loader"),Zue={id:Que,detector:Zze,loader:Jze};var lhe="pie",fGe=o(t=>/^\s*pie/.test(t),"detector"),dGe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(ohe(),she));return{id:lhe,diagram:t}},"loader"),che={id:lhe,detector:fGe,loader:dGe};var The="quadrantChart",RGe=o(t=>/^\s*quadrantChart/.test(t),"detector"),NGe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(whe(),bhe));return{id:The,diagram:t}},"loader"),MGe={id:The,detector:RGe,loader:NGe},khe=MGe;var Khe="xychart",jGe=o(t=>/^\s*xychart-beta/.test(t),"detector"),KGe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(jhe(),Xhe));return{id:Khe,diagram:t}},"loader"),QGe={id:Khe,detector:jGe,loader:KGe},Qhe=QGe;var sfe="requirement",tVe=o(t=>/^\s*requirement(Diagram)?/.test(t),"detector"),rVe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(afe(),ife));return{id:sfe,diagram:t}},"loader"),nVe={id:sfe,detector:tVe,loader:rVe},ofe=nVe;var Afe="sequence",zVe=o(t=>/^\s*sequenceDiagram/.test(t),"detector"),GVe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(Cfe(),Sfe));return{id:Afe,diagram:t}},"loader"),VVe={id:Afe,detector:zVe,loader:GVe},_fe=VVe;var Ife="class",XVe=o((t,e)=>e?.class?.defaultRenderer==="dagre-wrapper"?!1:/^\s*classDiagram/.test(t),"detector"),jVe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(Mfe(),Nfe));return{id:Ife,diagram:t}},"loader"),KVe={id:Ife,detector:XVe,loader:jVe},Ofe=KVe;var Ffe="classDiagram",ZVe=o((t,e)=>/^\s*classDiagram/.test(t)&&e?.class?.defaultRenderer==="dagre-wrapper"?!0:/^\s*classDiagram-v2/.test(t),"detector"),JVe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(Bfe(),Pfe));return{id:Ffe,diagram:t}},"loader"),eUe={id:Ffe,detector:ZVe,loader:JVe},$fe=eUe;var Ede="state",LUe=o((t,e)=>e?.state?.defaultRenderer==="dagre-wrapper"?!1:/^\s*stateDiagram/.test(t),"detector"),RUe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(kde(),Tde));return{id:Ede,diagram:t}},"loader"),NUe={id:Ede,detector:LUe,loader:RUe},Sde=NUe;var _de="stateDiagram",IUe=o((t,e)=>!!(/^\s*stateDiagram-v2/.test(t)||/^\s*stateDiagram/.test(t)&&e?.state?.defaultRenderer==="dagre-wrapper"),"detector"),OUe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(Ade(),Cde));return{id:_de,diagram:t}},"loader"),PUe={id:_de,detector:IUe,loader:OUe},Dde=PUe;var Wde="journey",nHe=o(t=>/^\s*journey/.test(t),"detector"),iHe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(Hde(),Ude));return{id:Wde,diagram:t}},"loader"),aHe={id:Wde,detector:nHe,loader:iHe},qde=aHe;vt();Vc();Ei();var sHe=o((t,e,r)=>{Y.debug(`rendering svg for syntax error +`);let n=sa(e),i=n.append("g");n.attr("viewBox","0 0 2412 512"),vn(n,100,512,!0),i.append("path").attr("class","error-icon").attr("d","m411.313,123.313c6.25-6.25 6.25-16.375 0-22.625s-16.375-6.25-22.625,0l-32,32-9.375,9.375-20.688-20.688c-12.484-12.5-32.766-12.5-45.25,0l-16,16c-1.261,1.261-2.304,2.648-3.31,4.051-21.739-8.561-45.324-13.426-70.065-13.426-105.867,0-192,86.133-192,192s86.133,192 192,192 192-86.133 192-192c0-24.741-4.864-48.327-13.426-70.065 1.402-1.007 2.79-2.049 4.051-3.31l16-16c12.5-12.492 12.5-32.758 0-45.25l-20.688-20.688 9.375-9.375 32.001-31.999zm-219.313,100.687c-52.938,0-96,43.063-96,96 0,8.836-7.164,16-16,16s-16-7.164-16-16c0-70.578 57.422-128 128-128 8.836,0 16,7.164 16,16s-7.164,16-16,16z"),i.append("path").attr("class","error-icon").attr("d","m459.02,148.98c-6.25-6.25-16.375-6.25-22.625,0s-6.25,16.375 0,22.625l16,16c3.125,3.125 7.219,4.688 11.313,4.688 4.094,0 8.188-1.563 11.313-4.688 6.25-6.25 6.25-16.375 0-22.625l-16.001-16z"),i.append("path").attr("class","error-icon").attr("d","m340.395,75.605c3.125,3.125 7.219,4.688 11.313,4.688 4.094,0 8.188-1.563 11.313-4.688 6.25-6.25 6.25-16.375 0-22.625l-16-16c-6.25-6.25-16.375-6.25-22.625,0s-6.25,16.375 0,22.625l15.999,16z"),i.append("path").attr("class","error-icon").attr("d","m400,64c8.844,0 16-7.164 16-16v-32c0-8.836-7.156-16-16-16-8.844,0-16,7.164-16,16v32c0,8.836 7.156,16 16,16z"),i.append("path").attr("class","error-icon").attr("d","m496,96.586h-32c-8.844,0-16,7.164-16,16 0,8.836 7.156,16 16,16h32c8.844,0 16-7.164 16-16 0-8.836-7.156-16-16-16z"),i.append("path").attr("class","error-icon").attr("d","m436.98,75.605c3.125,3.125 7.219,4.688 11.313,4.688 4.094,0 8.188-1.563 11.313-4.688l32-32c6.25-6.25 6.25-16.375 0-22.625s-16.375-6.25-22.625,0l-32,32c-6.251,6.25-6.251,16.375-0.001,22.625z"),i.append("text").attr("class","error-text").attr("x",1440).attr("y",250).attr("font-size","150px").style("text-anchor","middle").text("Syntax error in text"),i.append("text").attr("class","error-text").attr("x",1250).attr("y",400).attr("font-size","100px").style("text-anchor","middle").text(`mermaid version ${r}`)},"draw"),fP={draw:sHe},Yde=fP;var oHe={db:{},renderer:fP,parser:{parse:o(()=>{},"parse")}},Xde=oHe;var jde="flowchart-elk",lHe=o((t,e={})=>/^\s*flowchart-elk/.test(t)||/^\s*flowchart|graph/.test(t)&&e?.flowchart?.defaultRenderer==="elk"?(e.layout="elk",!0):!1,"detector"),cHe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(ak(),ik));return{id:jde,diagram:t}},"loader"),uHe={id:jde,detector:lHe,loader:cHe},Kde=uHe;var Tpe="timeline",DHe=o(t=>/^\s*timeline/.test(t),"detector"),LHe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(wpe(),bpe));return{id:Tpe,diagram:t}},"loader"),RHe={id:Tpe,detector:DHe,loader:LHe},kpe=RHe;var e1e="mindmap",cJe=o(t=>/^\s*mindmap/.test(t),"detector"),uJe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(Jge(),Zge));return{id:e1e,diagram:t}},"loader"),hJe={id:e1e,detector:cJe,loader:uJe},t1e=hJe;var d1e="kanban",AJe=o(t=>/^\s*kanban/.test(t),"detector"),_Je=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(f1e(),h1e));return{id:d1e,diagram:t}},"loader"),DJe={id:d1e,detector:AJe,loader:_Je},p1e=DJe;var j1e="sankey",ZJe=o(t=>/^\s*sankey-beta/.test(t),"detector"),JJe=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(X1e(),Y1e));return{id:j1e,diagram:t}},"loader"),eet={id:j1e,detector:ZJe,loader:JJe},K1e=eet;var sye="packet",pet=o(t=>/^\s*packet-beta/.test(t),"detector"),met=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(aye(),iye));return{id:sye,diagram:t}},"loader"),oye={id:sye,detector:pet,loader:met};var vye="radar",Fet=o(t=>/^\s*radar-beta/.test(t),"detector"),$et=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(yye(),gye));return{id:vye,diagram:t}},"loader"),xye={id:vye,detector:Fet,loader:$et};var Tve="block",srt=o(t=>/^\s*block-beta/.test(t),"detector"),ort=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(wve(),bve));return{id:Tve,diagram:t}},"loader"),lrt={id:Tve,detector:srt,loader:ort},kve=lrt;var Yve="architecture",Ort=o(t=>/^\s*architecture/.test(t),"detector"),Prt=o(async()=>{let{diagram:t}=await Promise.resolve().then(()=>(qve(),Wve));return{id:Yve,diagram:t}},"loader"),Brt={id:Yve,detector:Ort,loader:Prt},Xve=Brt;Xf();zt();var jve=!1,py=o(()=>{jve||(jve=!0,ad("error",Xde,t=>t.toLowerCase().trim()==="error"),ad("---",{db:{clear:o(()=>{},"clear")},styles:{},renderer:{draw:o(()=>{},"draw")},parser:{parse:o(()=>{throw new Error("Diagrams beginning with --- are not valid. If you were trying to use a YAML front-matter, please ensure that you've correctly opened and closed the YAML front-matter with un-indented `---` blocks")},"parse")},init:o(()=>null,"init")},t=>t.toLowerCase().trimStart().startsWith("---")),z4(XX,p1e,$fe,Ofe,oae,Vue,Zue,che,ofe,_fe,Kde,Qie,jie,t1e,kpe,hue,Dde,Sde,qde,khe,K1e,oye,Qhe,kve,Xve,xye))},"addDiagrams");vt();Xf();zt();var Kve=o(async()=>{Y.debug("Loading registered diagrams");let e=(await Promise.allSettled(Object.entries(Yf).map(async([r,{detector:n,loader:i}])=>{if(i)try{jy(r)}catch{try{let{diagram:a,id:s}=await i();ad(s,a,n)}catch(a){throw Y.error(`Failed to load external diagram with key ${r}. Removing from detectors.`),delete Yf[r],a}}}))).filter(r=>r.status==="rejected");if(e.length>0){Y.error(`Failed to load ${e.length} external diagrams`);for(let r of e)Y.error(r);throw new Error(`Failed to load ${e.length} external diagrams`)}},"loadRegisteredDiagrams");vt();dr();var lC="comm",cC="rule",uC="decl";var Qve="@import";var Zve="@namespace",Jve="@keyframes";var e2e="@layer";var vF=Math.abs,S4=String.fromCharCode;function hC(t){return t.trim()}o(hC,"trim");function C4(t,e,r){return t.replace(e,r)}o(C4,"replace");function t2e(t,e,r){return t.indexOf(e,r)}o(t2e,"indexof");function $f(t,e){return t.charCodeAt(e)|0}o($f,"charat");function zf(t,e,r){return t.slice(e,r)}o(zf,"substr");function vo(t){return t.length}o(vo,"strlen");function r2e(t){return t.length}o(r2e,"sizeof");function my(t,e){return e.push(t),t}o(my,"append");var fC=1,gy=1,n2e=0,il=0,Ri=0,vy="";function dC(t,e,r,n,i,a,s,l){return{value:t,root:e,parent:r,type:n,props:i,children:a,line:fC,column:gy,length:s,return:"",siblings:l}}o(dC,"node");function i2e(){return Ri}o(i2e,"char");function a2e(){return Ri=il>0?$f(vy,--il):0,gy--,Ri===10&&(gy=1,fC--),Ri}o(a2e,"prev");function al(){return Ri=il2||yy(Ri)>3?"":" "}o(l2e,"whitespace");function c2e(t,e){for(;--e&&al()&&!(Ri<48||Ri>102||Ri>57&&Ri<65||Ri>70&&Ri<97););return pC(t,A4()+(e<6&&rh()==32&&al()==32))}o(c2e,"escaping");function xF(t){for(;al();)switch(Ri){case t:return il;case 34:case 39:t!==34&&t!==39&&xF(Ri);break;case 40:t===41&&xF(t);break;case 92:al();break}return il}o(xF,"delimiter");function u2e(t,e){for(;al()&&t+Ri!==57;)if(t+Ri===84&&rh()===47)break;return"/*"+pC(e,il-1)+"*"+S4(t===47?t:al())}o(u2e,"commenter");function h2e(t){for(;!yy(rh());)al();return pC(t,il)}o(h2e,"identifier");function p2e(t){return o2e(gC("",null,null,null,[""],t=s2e(t),0,[0],t))}o(p2e,"compile");function gC(t,e,r,n,i,a,s,l,u){for(var h=0,f=0,d=s,p=0,m=0,g=0,y=1,v=1,x=1,b=0,w="",C=i,T=a,E=n,A=w;v;)switch(g=b,b=al()){case 40:if(g!=108&&$f(A,d-1)==58){t2e(A+=C4(mC(b),"&","&\f"),"&\f",vF(h?l[h-1]:0))!=-1&&(x=-1);break}case 34:case 39:case 91:A+=mC(b);break;case 9:case 10:case 13:case 32:A+=l2e(g);break;case 92:A+=c2e(A4()-1,7);continue;case 47:switch(rh()){case 42:case 47:my(Frt(u2e(al(),A4()),e,r,u),u),(yy(g||1)==5||yy(rh()||1)==5)&&vo(A)&&zf(A,-1,void 0)!==" "&&(A+=" ");break;default:A+="/"}break;case 123*y:l[h++]=vo(A)*x;case 125*y:case 59:case 0:switch(b){case 0:case 125:v=0;case 59+f:x==-1&&(A=C4(A,/\f/g,"")),m>0&&(vo(A)-d||y===0&&g===47)&&my(m>32?d2e(A+";",n,r,d-1,u):d2e(C4(A," ","")+";",n,r,d-2,u),u);break;case 59:A+=";";default:if(my(E=f2e(A,e,r,h,f,i,l,w,C=[],T=[],d,a),a),b===123)if(f===0)gC(A,e,E,E,C,a,d,l,T);else{switch(p){case 99:if($f(A,3)===110)break;case 108:if($f(A,2)===97)break;default:f=0;case 100:case 109:case 115:}f?gC(t,E,E,n&&my(f2e(t,E,E,0,0,i,l,w,i,C=[],d,T),T),i,T,d,l,n?C:T):gC(A,E,E,E,[""],T,0,l,T)}}h=f=m=0,y=x=1,w=A="",d=s;break;case 58:d=1+vo(A),m=g;default:if(y<1){if(b==123)--y;else if(b==125&&y++==0&&a2e()==125)continue}switch(A+=S4(b),b*y){case 38:x=f>0?1:(A+="\f",-1);break;case 44:l[h++]=(vo(A)-1)*x,x=1;break;case 64:rh()===45&&(A+=mC(al())),p=rh(),f=d=vo(w=A+=h2e(A4())),b++;break;case 45:g===45&&vo(A)==2&&(y=0)}}return a}o(gC,"parse");function f2e(t,e,r,n,i,a,s,l,u,h,f,d){for(var p=i-1,m=i===0?a:[""],g=r2e(m),y=0,v=0,x=0;y0?m[b]+" "+w:C4(w,/&\f/g,m[b])))&&(u[x++]=C);return dC(t,e,r,i===0?cC:l,u,h,f,d)}o(f2e,"ruleset");function Frt(t,e,r,n){return dC(t,e,r,lC,S4(i2e()),zf(t,2,-2),0,n)}o(Frt,"comment");function d2e(t,e,r,n,i){return dC(t,e,r,uC,zf(t,0,n),zf(t,n+1,-1),n,i)}o(d2e,"declaration");function yC(t,e){for(var r="",n=0;n{v2e.forEach(t=>{t()}),v2e=[]},"attachFunctions");vt();var b2e=o(t=>t.replace(/^\s*%%(?!{)[^\n]+\n?/gm,"").trimStart(),"cleanupComments");$4();Ew();function w2e(t){let e=t.match(F4);if(!e)return{text:t,metadata:{}};let r=cm(e[1],{schema:lm})??{};r=typeof r=="object"&&!Array.isArray(r)?r:{};let n={};return r.displayMode&&(n.displayMode=r.displayMode.toString()),r.title&&(n.title=r.title.toString()),r.config&&(n.config=r.config),{text:t.slice(e[0].length),metadata:n}}o(w2e,"extractFrontMatter");ir();var zrt=o(t=>t.replace(/\r\n?/g,` +`).replace(/<(\w+)([^>]*)>/g,(e,r,n)=>"<"+r+n.replace(/="([^"]*)"/g,"='$1'")+">"),"cleanupText"),Grt=o(t=>{let{text:e,metadata:r}=w2e(t),{displayMode:n,title:i,config:a={}}=r;return n&&(a.gantt||(a.gantt={}),a.gantt.displayMode=n),{title:i,config:a,text:e}},"processFrontmatter"),Vrt=o(t=>{let e=Gt.detectInit(t)??{},r=Gt.detectDirective(t,"wrap");return Array.isArray(r)?e.wrap=r.some(({type:n})=>n==="wrap"):r?.type==="wrap"&&(e.wrap=!0),{text:IX(t),directive:e}},"processDirectives");function bF(t){let e=zrt(t),r=Grt(e),n=Vrt(r.text),i=Fi(r.config,n.directive);return t=b2e(n.text),{code:t,title:r.title,config:i}}o(bF,"preprocessDiagram");tA();q4();ir();function T2e(t){let e=new TextEncoder().encode(t),r=Array.from(e,n=>String.fromCodePoint(n)).join("");return btoa(r)}o(T2e,"toBase64");var Urt=5e4,Hrt="graph TB;a[Maximum text size in diagram exceeded];style a fill:#faa",Wrt="sandbox",qrt="loose",Yrt="http://www.w3.org/2000/svg",Xrt="http://www.w3.org/1999/xlink",jrt="http://www.w3.org/1999/xhtml",Krt="100%",Qrt="100%",Zrt="border:0;margin:0;",Jrt="margin:0",ent="allow-top-navigation-by-user-activation allow-popups",tnt='The "iframe" tag is not supported by your browser.',rnt=["foreignobject"],nnt=["dominant-baseline"];function C2e(t){let e=bF(t);return Ly(),W$(e.config??{}),e}o(C2e,"processAndSetConfigs");async function int(t,e){py();try{let{code:r,config:n}=C2e(t);return{diagramType:(await A2e(r)).type,config:n}}catch(r){if(e?.suppressErrors)return!1;throw r}}o(int,"parse");var k2e=o((t,e,r=[])=>` +.${t} ${e} { ${r.join(" !important; ")} !important; }`,"cssImportantStyles"),ant=o((t,e=new Map)=>{let r="";if(t.themeCSS!==void 0&&(r+=` +${t.themeCSS}`),t.fontFamily!==void 0&&(r+=` +:root { --mermaid-font-family: ${t.fontFamily}}`),t.altFontFamily!==void 0&&(r+=` +:root { --mermaid-alt-font-family: ${t.altFontFamily}}`),e instanceof Map){let s=t.htmlLabels??t.flowchart?.htmlLabels?["> *","span"]:["rect","polygon","ellipse","circle","path"];e.forEach(l=>{ur(l.styles)||s.forEach(u=>{r+=k2e(l.id,u,l.styles)}),ur(l.textStyles)||(r+=k2e(l.id,"tspan",(l?.textStyles||[]).map(u=>u.replace("color","fill"))))})}return r},"createCssStyles"),snt=o((t,e,r,n)=>{let i=ant(t,r),a=zG(e,i,t.themeVariables);return yC(p2e(`${n}{${a}}`),m2e)},"createUserStyles"),ont=o((t="",e,r)=>{let n=t;return!r&&!e&&(n=n.replace(/marker-end="url\([\d+./:=?A-Za-z-]*?#/g,'marker-end="url(#')),n=na(n),n=n.replace(/
    /g,"
    "),n},"cleanUpSvgCode"),lnt=o((t="",e)=>{let r=e?.viewBox?.baseVal?.height?e.viewBox.baseVal.height+"px":Qrt,n=T2e(`${t}`);return``},"putIntoIFrame"),E2e=o((t,e,r,n,i)=>{let a=t.append("div");a.attr("id",r),n&&a.attr("style",n);let s=a.append("svg").attr("id",e).attr("width","100%").attr("xmlns",Yrt);return i&&s.attr("xmlns:xlink",i),s.append("g"),t},"appendDivSvgG");function S2e(t,e){return t.append("iframe").attr("id",e).attr("style","width: 100%; height: 100%;").attr("sandbox","")}o(S2e,"sandboxedIframe");var cnt=o((t,e,r,n)=>{t.getElementById(e)?.remove(),t.getElementById(r)?.remove(),t.getElementById(n)?.remove()},"removeExistingElements"),unt=o(async function(t,e,r){py();let n=C2e(e);e=n.code;let i=cr();Y.debug(i),e.length>(i?.maxTextSize??Urt)&&(e=Hrt);let a="#"+t,s="i"+t,l="#"+s,u="d"+t,h="#"+u,f=o(()=>{let L=Ge(p?l:h).node();L&&"remove"in L&&L.remove()},"removeTempElements"),d=Ge("body"),p=i.securityLevel===Wrt,m=i.securityLevel===qrt,g=i.fontFamily;if(r!==void 0){if(r&&(r.innerHTML=""),p){let k=S2e(Ge(r),s);d=Ge(k.nodes()[0].contentDocument.body),d.node().style.margin=0}else d=Ge(r);E2e(d,t,u,`font-family: ${g}`,Xrt)}else{if(cnt(document,t,u,s),p){let k=S2e(Ge("body"),s);d=Ge(k.nodes()[0].contentDocument.body),d.node().style.margin=0}else d=Ge("body");E2e(d,t,u)}let y,v;try{y=await xy.fromText(e,{title:n.title})}catch(k){if(i.suppressErrorRendering)throw f(),k;y=await xy.fromText("error"),v=k}let x=d.select(h).node(),b=y.type,w=x.firstChild,C=w.firstChild,T=y.renderer.getClasses?.(e,y),E=snt(i,b,T,a),A=document.createElement("style");A.innerHTML=E,w.insertBefore(A,C);try{await y.renderer.draw(e,t,vb.version,y)}catch(k){throw i.suppressErrorRendering?f():Yde.draw(e,t,vb.version),k}let S=d.select(`${h} svg`),_=y.db.getAccTitle?.(),I=y.db.getAccDescription?.();fnt(b,S,_,I),d.select(`[id="${t}"]`).selectAll("foreignobject > *").attr("xmlns",jrt);let D=d.select(h).node().innerHTML;if(Y.debug("config.arrowMarkerAbsolute",i.arrowMarkerAbsolute),D=ont(D,p,fr(i.arrowMarkerAbsolute)),p){let k=d.select(h+" svg").node();D=lnt(D,k)}else m||(D=ch.sanitize(D,{ADD_TAGS:rnt,ADD_ATTR:nnt,HTML_INTEGRATION_POINTS:{foreignobject:!0}}));if(x2e(),v)throw v;return f(),{diagramType:b,svg:D,bindFunctions:y.db.bindFunctions}},"render");function hnt(t={}){let e=Gn({},t);e?.fontFamily&&!e.themeVariables?.fontFamily&&(e.themeVariables||(e.themeVariables={}),e.themeVariables.fontFamily=e.fontFamily),V$(e),e?.theme&&e.theme in To?e.themeVariables=To[e.theme].getThemeVariables(e.themeVariables):e&&(e.themeVariables=To.default.getThemeVariables(e.themeVariables));let r=typeof e=="object"?t7(e):r7();wy(r.logLevel),py()}o(hnt,"initialize");var A2e=o((t,e={})=>{let{code:r}=bF(t);return xy.fromText(r,e)},"getDiagramFromText");function fnt(t,e,r,n){g2e(e,t),y2e(e,r,n,e.attr("id"))}o(fnt,"addA11yInfo");var Gf=Object.freeze({render:unt,parse:int,getDiagramFromText:A2e,initialize:hnt,getConfig:cr,setConfig:X4,getSiteConfig:r7,updateSiteConfig:U$,reset:o(()=>{Ly()},"reset"),globalReset:o(()=>{Ly(lh)},"globalReset"),defaultConfig:lh});wy(cr().logLevel);Ly(cr());Yd();ir();var dnt=o((t,e,r)=>{Y.warn(t),Z9(t)?(r&&r(t.str,t.hash),e.push({...t,message:t.str,error:t})):(r&&r(t),t instanceof Error&&e.push({str:t.message,message:t.message,hash:t.name,error:t}))},"handleError"),_2e=o(async function(t={querySelector:".mermaid"}){try{await pnt(t)}catch(e){if(Z9(e)&&Y.error(e.str),nh.parseError&&nh.parseError(e),!t.suppressErrors)throw Y.error("Use the suppressErrors option to suppress these errors"),e}},"run"),pnt=o(async function({postRenderCallback:t,querySelector:e,nodes:r}={querySelector:".mermaid"}){let n=Gf.getConfig();Y.debug(`${t?"":"No "}Callback function found`);let i;if(r)i=r;else if(e)i=document.querySelectorAll(e);else throw new Error("Nodes and querySelector are both undefined");Y.debug(`Found ${i.length} diagrams`),n?.startOnLoad!==void 0&&(Y.debug("Start On Load: "+n?.startOnLoad),Gf.updateSiteConfig({startOnLoad:n?.startOnLoad}));let a=new Gt.InitIDGenerator(n.deterministicIds,n.deterministicIDSeed),s,l=[];for(let u of Array.from(i)){Y.info("Rendering diagram: "+u.id);if(u.getAttribute("data-processed"))continue;u.setAttribute("data-processed","true");let h=`mermaid-${a.next()}`;s=u.innerHTML,s=B4(Gt.entityDecode(s)).trim().replace(//gi,"
    ");let f=Gt.detectInit(s);f&&Y.debug("Detected early reinit: ",f);try{let{svg:d,bindFunctions:p}=await N2e(h,s,u);u.innerHTML=d,t&&await t(h),p&&p(u)}catch(d){dnt(d,l,nh.parseError)}}if(l.length>0)throw l[0]},"runThrowsErrors"),D2e=o(function(t){Gf.initialize(t)},"initialize"),mnt=o(async function(t,e,r){Y.warn("mermaid.init is deprecated. Please use run instead."),t&&D2e(t);let n={postRenderCallback:r,querySelector:".mermaid"};typeof e=="string"?n.querySelector=e:e&&(e instanceof HTMLElement?n.nodes=[e]:n.nodes=e),await _2e(n)},"init"),gnt=o(async(t,{lazyLoad:e=!0}={})=>{py(),z4(...t),e===!1&&await Kve()},"registerExternalDiagrams"),L2e=o(function(){if(nh.startOnLoad){let{startOnLoad:t}=Gf.getConfig();t&&nh.run().catch(e=>Y.error("Mermaid failed to initialize",e))}},"contentLoaded");if(typeof document<"u"){window.addEventListener("load",L2e,!1)}var ynt=o(function(t){nh.parseError=t},"setParseErrorHandler"),vC=[],wF=!1,R2e=o(async()=>{if(!wF){for(wF=!0;vC.length>0;){let t=vC.shift();if(t)try{await t()}catch(e){Y.error("Error executing queue",e)}}wF=!1}},"executeQueue"),vnt=o(async(t,e)=>new Promise((r,n)=>{let i=o(()=>new Promise((a,s)=>{Gf.parse(t,e).then(l=>{a(l),r(l)},l=>{Y.error("Error parsing",l),nh.parseError?.(l),s(l),n(l)})}),"performCall");vC.push(i),R2e().catch(n)}),"parse"),N2e=o((t,e,r)=>new Promise((n,i)=>{let a=o(()=>new Promise((s,l)=>{Gf.render(t,e,r).then(u=>{s(u),n(u)},u=>{Y.error("Error parsing",u),nh.parseError?.(u),l(u),i(u)})}),"performCall");vC.push(a),R2e().catch(i)}),"render"),nh={startOnLoad:!0,mermaidAPI:Gf,parse:vnt,render:N2e,init:mnt,run:_2e,registerExternalDiagrams:gnt,registerLayoutLoaders:vR,initialize:D2e,parseError:void 0,contentLoaded:L2e,setParseErrorHandler:ynt,detectType:a0,registerIconPacks:P4},xnt=nh;return V2e(bnt);})(); +/*! Check if previously processed */ +/*! + * Wait for document loaded before starting the execution + */ +/*! Bundled license information: + +dompurify/dist/purify.es.mjs: + (*! @license DOMPurify 3.2.4 | (c) Cure53 and other contributors | Released under the Apache license 2.0 and Mozilla Public License 2.0 | github.com/cure53/DOMPurify/blob/3.2.4/LICENSE *) + +js-yaml/dist/js-yaml.mjs: + (*! js-yaml 4.1.0 https://github.com/nodeca/js-yaml @license MIT *) + +lodash-es/lodash.js: + (** + * @license + * Lodash (Custom Build) + * Build: `lodash modularize exports="es" -o ./` + * Copyright OpenJS Foundation and other contributors + * Released under MIT license + * Based on Underscore.js 1.8.3 + * Copyright Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors + *) + +cytoscape/dist/cytoscape.esm.mjs: + (*! + Embeddable Minimum Strictly-Compliant Promises/A+ 1.1.1 Thenable + Copyright (c) 2013-2014 Ralf S. Engelschall (http://engelschall.com) + Licensed under The MIT License (http://opensource.org/licenses/MIT) + *) + (*! + Event object based on jQuery events, MIT license + + https://jquery.org/license/ + https://tldrlegal.com/license/mit-license + https://github.com/jquery/jquery/blob/master/src/event.js + *) + (*! Bezier curve function generator. Copyright Gaetan Renaudeau. MIT License: http://en.wikipedia.org/wiki/MIT_License *) + (*! Runge-Kutta spring physics function generator. Adapted from Framer.js, copyright Koen Bok. MIT License: http://en.wikipedia.org/wiki/MIT_License *) +*/ +globalThis.mermaid = globalThis.__esbuild_esm_mermaid.default; \ No newline at end of file diff --git a/docs/user-guide/src/SUMMARY.md b/docs/user-guide/src/SUMMARY.md new file mode 100644 index 0000000000..b467c53e7e --- /dev/null +++ b/docs/user-guide/src/SUMMARY.md @@ -0,0 +1,74 @@ +# Summary + +[Introduction](./introduction.md) + +# Getting Started + +- [Installation](./getting_started/installation.md) +- [Quick Start: Reading a Table](./getting_started/quick_start_read.md) +- [Quick Start: Writing a Table](./getting_started/quick_start_write.md) + +# Core Concepts + +- [Architecture Overview](./concepts/architecture.md) +- [The Engine Trait](./concepts/engine_trait.md) +- [Schemas and Data Types](./concepts/schema_and_types.md) +- [Feature Flags](./concepts/feature_flags.md) + +# Reading Tables + +- [Building a Scan](./reading/building_a_scan.md) +- [Column Selection](./reading/column_selection.md) +- [Filter Pushdown and File Skipping](./reading/filter_pushdown.md) +- [Advanced Reads with scan_metadata()](./reading/scan_metadata.md) +- [Distributed Log Replay](./reading/parallel_scan_metadata.md) +- [Time Travel and Snapshot Management](./reading/time_travel.md) +- [Incremental Scans](./reading/incremental_scan.md) +- [Reading Change Data Feed](./reading/change_data_feed.md) + +# Writing Tables + +- [Creating a Table](./writing/create_table.md) +- [Appending Data](./writing/append.md) +- [Writing to Partitioned Tables](./writing/partitioned_writes.md) +- [Removing Data](./writing/removing_files.md) +- [Domain Metadata](./writing/domain_metadata.md) +- [Idempotent Writes](./writing/idempotent_writes.md) +- [Altering a Table](./writing/alter_table.md) + +# Maintenance Operations + +- [Checkpointing](./maintenance/checkpointing.md) +- [Version Checksums](./maintenance/version_checksums.md) + +# Building a Connector + +- [Overview](./connector/overview.md) +- [Implementing the Engine Trait](./connector/implementing_engine.md) +- [The EngineData Trait](./connector/engine_data.md) + +# Catalog-Managed Tables + +- [Overview](./catalog_managed/overview.md) +- [Implementing a Catalog Committer](./catalog_managed/committer.md) +- [Reads](./catalog_managed/reading.md) +- [Writes](./catalog_managed/writing.md) + +# Unity Catalog Integration + +- [Overview](./unity_catalog/overview.md) +- [Creating UC Tables](./unity_catalog/creating_tables.md) +- [Reading UC Tables](./unity_catalog/reading.md) +- [Writing to UC Tables](./unity_catalog/writing.md) + +# Storage Configuration + +- [Configuring Storage](./storage/configuring_storage.md) + +# Observability + +- [Metrics and Monitoring](./observability/observability.md) + +# FFI (C/C++ Integration) + +- [Overview](./ffi/overview.md) diff --git a/docs/user-guide/src/catalog_managed/committer.md b/docs/user-guide/src/catalog_managed/committer.md new file mode 100644 index 0000000000..0181f3044d --- /dev/null +++ b/docs/user-guide/src/catalog_managed/committer.md @@ -0,0 +1,295 @@ +# Implementing a catalog committer + +To commit transactions on catalog-managed tables, you implement the `Committer` trait with +your catalog's staging and ratification logic. For filesystem-managed tables, the built-in +`FileSystemCommitter` writes delta files directly via PUT-if-absent. For catalog-managed +tables, you provide your own `Committer` that routes commits through your catalog. + +> [!WARNING] +> Kernel rejects `FileSystemCommitter` on a catalog-managed table at `txn.commit()` +> time. You must provide a catalog committer before commit runs. + +Before reading this page, make sure you understand [Catalog-managed tables](./overview.md). + +## The Committer trait + +```rust,ignore +pub trait Committer: Send { + fn commit( + &self, + engine: &dyn Engine, + actions: DeltaResultIterator<'_, FilteredEngineData>, + commit_metadata: CommitMetadata, + ) -> DeltaResult; + + fn is_catalog_committer(&self) -> bool; + + fn publish( + &self, + engine: &dyn Engine, + publish_metadata: PublishMetadata, + ) -> DeltaResult<()>; +} +``` + +The trait has three methods. Two (`commit()` and `publish()`) carry the real +logic; the third (`is_catalog_committer()`) is a one-line method that returns a +constant: + +1. **`commit()`** atomically commits the given actions at the version specified in + `CommitMetadata`. Returns `CommitResponse::Committed` on success or + `CommitResponse::Conflict { version }` if another writer already committed this + version. + +2. **`is_catalog_committer()`** returns `true` for catalog committers. Kernel checks + this flag on both commit and publish paths and enforces the pairing in both + directions: it rejects a `FileSystemCommitter` on a catalog-managed table, and + it rejects a catalog committer on a filesystem-managed table. + +3. **`publish()`** copies ratified catalog commits from `_staged_commits/` to the main + `_delta_log/` directory as published delta files. Some catalogs publish + server-side, in which case `publish()` only notifies the catalog; others use + PUT-if-absent copies from the client. + +## CommitMetadata + +Kernel constructs `CommitMetadata` and passes it to your `commit()` method. Key methods: + +```rust,ignore +impl CommitMetadata { + pub fn published_commit_path(&self) -> DeltaResult; + pub fn staged_commit_path(&self) -> DeltaResult; // unique UUID each call + pub fn version(&self) -> Version; + pub fn commit_type(&self) -> CommitType; + pub fn in_commit_timestamp(&self) -> i64; + pub fn max_published_version(&self) -> Option; + pub fn table_root(&self) -> &Url; +} +``` + +This block is a subset of `CommitMetadata`'s public surface. See the rustdoc for +the full list. + +- `published_commit_path()` returns the final delta log path (e.g., + `s3://bucket/table/_delta_log/00000000000000000001.json`). +- `staged_commit_path()` returns a unique staged commit path (e.g., + `s3://bucket/table/_delta_log/_staged_commits/00000000000000000001..json`). + Kernel generates a fresh UUID on every call; do not substitute a catalog-supplied + UUID. +- `commit_type()` returns a `CommitType` variant indicating whether this is a table + creation or a write, and whether the table is catalog-managed. +- `in_commit_timestamp()` returns the timestamp (milliseconds since the Unix epoch, + UTC) Kernel will record in the `CommitInfo` action as `inCommitTimestamp`. Kernel + also writes a fresh `txnId` (UUID) on every `CommitInfo` it generates; the Delta + spec requires `txnId` on catalog-managed commits, and Kernel emits it + unconditionally. +- `max_published_version()` returns the highest version already published to the delta + log, if any. Your catalog may need this to determine which commits still require + publishing. + +> [!WARNING] +> Call `staged_commit_path()` exactly once per `commit()` invocation. Because each +> call produces a different UUID, a second call yields a different path: only one +> path gets written, and if you use the other to tell the catalog, the catalog +> points at a missing file. + +## CommitResponse + +```rust,ignore +pub enum CommitResponse { + Committed { file_meta: FileMeta }, + Conflict { version: Version }, +} +``` + +Return `Committed` with the `FileMeta` of the staged commit file on success. Return +`Conflict` with the attempted version if another writer already committed that version. + +## Implementing a catalog committer + +The typical implementation follows four steps. Steps 1 and 2 form the body of +`commit()`. Step 3 is the `is_catalog_committer()` flag. Step 4 is `publish()`. + +### Step 1: Stage the commit + +Write the actions to a staged commit file in `_staged_commits/`: + +```rust,ignore +fn commit( + &self, + engine: &dyn Engine, + actions: DeltaResultIterator<'_, FilteredEngineData>, + commit_metadata: CommitMetadata, +) -> DeltaResult { + // Write actions to _staged_commits/..json. `actions` is + // already a Box>, so pass it directly (do not re-box). + let staged_path = commit_metadata.staged_commit_path()?; + engine.json_handler().write_json_file(&staged_path, actions, false)?; + // ... +``` + +### Step 2: Ratify through the catalog + +Call your catalog's commit API to ratify the staged commit. The exact arguments +vary by catalog; Unity Catalog's `CommitRequest`, for example, carries the table +id, commit version, staged filename, in-commit timestamp, and the maximum +published version. Your catalog's API may look different. Here is the general +shape: + +```rust,ignore + // Tell the catalog about the staged commit. + // Replace this with your catalog's ratification API. Forward the + // commit_metadata.in_commit_timestamp() value so the catalog records the + // same timestamp Kernel writes into the CommitInfo action. + self.catalog_client.ratify_commit( + &self.table_id, + commit_metadata.version(), + &staged_path, + commit_metadata.in_commit_timestamp(), + commit_metadata.max_published_version(), + )?; + + // Return the staged file metadata on success. HEAD the file via + // engine.storage_handler().head() to get the real byte size, and use + // the in-commit timestamp as the logical commit time (not the filesystem + // mtime, which reflects when the file was written rather than when the + // commit took effect). + let staged_file = engine.storage_handler().head(&staged_path)?; + Ok(CommitResponse::Committed { + file_meta: FileMeta::new( + staged_path, + commit_metadata.in_commit_timestamp(), + staged_file.size, + ), + }) +} +``` + +Map your catalog's "another writer won this version" error to +`CommitResponse::Conflict { version: commit_metadata.version() }` rather than +propagating it as an `Err`. Return other errors as `Err(...)`. Kernel classifies +only `Error::IOError` as retryable (surfaced as +`CommitResult::RetryableTransaction`); return `IOError` for transient storage +failures and other variants for everything else. Do not disguise non-I/O errors +as `IOError` to opt into retry semantics. + +### Step 3: Mark as catalog committer + +```rust,ignore +fn is_catalog_committer(&self) -> bool { + true +} +``` + +### Step 4: Implement publish + +Publishing copies staged commits from `_staged_commits/` to the main `_delta_log/`. +Kernel passes the commits to publish as a contiguous ascending batch via +`PublishMetadata::commits_to_publish()`. Your implementation must: + +- Copy the commits **in the order Kernel provides** (version `v-1` before version `v`). + Do not reorder. +- Be **idempotent**. Treat `FileAlreadyExists` as success; a previous publish may have + already copied some entries. + +```rust,ignore +use delta_kernel::Error; + +fn publish( + &self, + engine: &dyn Engine, + publish_metadata: PublishMetadata, +) -> DeltaResult<()> { + for catalog_commit in publish_metadata.commits_to_publish() { + let src = catalog_commit.location(); // _staged_commits/..json + let dest = catalog_commit.published_location(); // _delta_log/.json + match engine.storage_handler().copy_atomic(src, dest) { + Ok(()) | Err(Error::FileAlreadyExists(_)) => (), // already published + Err(e) => return Err(e), + } + } + Ok(()) +} +``` + +## Putting it all together + +Here is the full skeleton for a catalog committer. Replace `MyCatalogClient` with your +catalog's client type and fill in the ratification logic: + +```rust,ignore +// Imports elided for brevity. In addition to the ones below, you will need +// Committer, CommitMetadata, CommitResponse, PublishMetadata, DeltaResult, +// FilteredEngineData, and Engine from delta_kernel. +use delta_kernel::{Error, FileMeta}; + +pub struct MyCatalogCommitter { + catalog_client: Arc, + table_id: String, +} + +impl Committer for MyCatalogCommitter { + fn commit( + &self, + engine: &dyn Engine, + actions: DeltaResultIterator<'_, FilteredEngineData>, + commit_metadata: CommitMetadata, + ) -> DeltaResult { + // 1. Stage: write actions to _staged_commits/ + let staged_path = commit_metadata.staged_commit_path()?; + engine.json_handler().write_json_file(&staged_path, actions, false)?; + + // 2. Ratify: register the staged commit with the catalog. ratify_commit + // is an imagined example API; your catalog's signature will differ. + self.catalog_client.ratify_commit( + &self.table_id, + commit_metadata.version(), + &staged_path, + commit_metadata.in_commit_timestamp(), + commit_metadata.max_published_version(), + )?; + + // 3. Return success. HEAD the staged file to get the real byte size, + // and use the in-commit timestamp as the logical commit time (not + // the filesystem mtime). + let staged_file = engine.storage_handler().head(&staged_path)?; + Ok(CommitResponse::Committed { + file_meta: FileMeta::new( + staged_path, + commit_metadata.in_commit_timestamp(), + staged_file.size, + ), + }) + } + + fn is_catalog_committer(&self) -> bool { + true + } + + fn publish( + &self, + engine: &dyn Engine, + publish_metadata: PublishMetadata, + ) -> DeltaResult<()> { + for catalog_commit in publish_metadata.commits_to_publish() { + let src = catalog_commit.location(); + let dest = catalog_commit.published_location(); + match engine.storage_handler().copy_atomic(src, dest) { + Ok(()) | Err(Error::FileAlreadyExists(_)) => (), + Err(e) => return Err(e), + } + } + Ok(()) + } +} +``` + +For a complete Unity Catalog implementation, see +[Unity Catalog integration](../unity_catalog/overview.md). + +## What's next + +- [Reading catalog-managed tables](./reading.md) covers how the catalog client + provides commits for snapshot construction. +- [Writing: the commit and publish flow](./writing.md) walks through the complete write + lifecycle including publishing. diff --git a/docs/user-guide/src/catalog_managed/overview.md b/docs/user-guide/src/catalog_managed/overview.md new file mode 100644 index 0000000000..9a797d217e --- /dev/null +++ b/docs/user-guide/src/catalog_managed/overview.md @@ -0,0 +1,233 @@ +# Catalog-managed tables + +A **catalog-managed table** is a Delta table whose commits go through a catalog +instead of being written directly to the filesystem. This matters because it +shifts the source of truth from the filesystem to the catalog, enabling +centralized governance, enforceable constraints, and multi-table coordination. + +Before reading this page, make sure you understand +[Architecture overview](../concepts/architecture.md) and +[The Engine trait](../concepts/engine_trait.md). + +## Filesystem-managed vs. catalog-managed + +By default, Delta tables are **filesystem-managed**: every commit is written +directly to `_delta_log/` as a JSON file, and readers discover table state by +listing that directory. Atomicity comes from the filesystem's PUT-if-absent +semantics. + +A **catalog-managed table** changes this model. The `catalogManaged` +reader-writer table feature makes the catalog the **source of truth** for +commits: + +- Writers must commit through the catalog, not directly to the filesystem. +- Readers must contact the catalog to discover recent (possibly unpublished) + commits. +- The catalog decides whether a commit attempt succeeds, not the filesystem. +- Path-based access is not supported. Tables must be accessed through the + catalog. + +## Benefits of catalog-managed tables + +In filesystem-managed tables, the filesystem is the ultimate authority on table +state. Catalog-managed tables change that by putting the catalog on the critical +path for all reads and writes: + +- **Governance across engines.** Every read and write goes through the catalog, + ensuring all engines enforce the same permissions, constraints, lineage, and + auditing. +- **Enforceable constraints.** The catalog can reject invalid schema or + constraint changes (for example, blocking a NOT NULL drop on a column + referenced by a foreign key). +- **Multi-table atomic updates.** The catalog can coordinate commits spanning + multiple tables without custom coordination services. +- **Faster query planning.** The catalog can serve table metadata directly, + skipping cloud storage LIST operations that add 100+ ms of latency. + +## Terminology: commit types + +Catalog-managed tables have several kinds of commits: + +| Commit type | Description | +|------------|-------------| +| **Staged commit** | Written to `_delta_log/_staged_commits/..json`. Has the same format as a normal delta file. The catalog records which staged commit won each version. | +| **Ratified commit** | A staged commit that the catalog has accepted as the winner for a given version. May or may not be published yet. | +| **Published commit** | A ratified commit that has been copied to `_delta_log/.json` as a normal delta file. Once published, it is discoverable through filesystem listing. | + +Writing a staged commit file does **not** reserve a version. Multiple writers may stage +different `..json` files for the same version; the catalog decides which +one wins. Staged commits that lose the race become orphans on disk and are cleaned up +out of band (typically by a catalog-driven sweep). + +> [!NOTE] +> The Delta protocol also defines **inline commits**, where commit content is +> sent directly to the catalog rather than written to disk. Kernel does not yet +> support inline commits. + +## Protocol requirements + +The `catalogManaged` reader-writer table feature requires `inCommitTimestamp` +to be enabled. A spec-compliant catalog-managed table therefore carries: + +- `catalogManaged` in both `readerFeatures` and `writerFeatures` in the + `Protocol` action (it is a reader-writer feature). +- `inCommitTimestamp` enabled as a writer-only feature. + +Kernel constructs this protocol automatically during the `create_table` call; +you do not hand-craft the `Protocol` or `Metadata` actions. The list above +describes what ends up in the commit, not steps you write by hand. + +The Delta spec also requires every `CommitInfo` action on a catalog-managed +table to carry a unique `txnId` field. Kernel writes a fresh `txnId` (UUID) on +every `CommitInfo` it generates, so this requirement is met automatically. +Custom committers **must** write the full iterator of actions Kernel supplies +to the staged commit file. They do not pick and choose which actions to +include, and they do not synthesize replacement `CommitInfo` actions. + +## How Kernel fits in + +Kernel's design philosophy for catalog-managed tables is straightforward: +**Kernel doesn't know about catalogs.** It doesn't know about table names, table +IDs, catalog APIs, or catalog servers. Instead: + +- The **catalog client** (a client-side library you write or import) works + together with your connector to resolve table names to paths, fetch + credentials, and retrieve recent ratified commits from the catalog. +- The catalog client translates the catalog's response into a `Vec` log + tail and a max catalog version. Your connector passes these to `SnapshotBuilder` + to load the table. +- For writes, the catalog client provides a **`Committer`** that knows how to + stage, ratify, and publish commits through the catalog. +- Kernel does its job (protocol compliance, log replay, data skipping, schema + enforcement) without knowing or caring whether a catalog exists. + +```text ++-----------------------------------------------------------+ +| Compute Engine | +| (Spark, Flink, DuckDB, Polars, ...) | ++---------------------------+-------------------------------+ + | table name + v ++-----------------------------------------------------------+ +| Catalog Client | +| | +| 1. Resolves table name -> path + credentials | +| 2. Calls catalog API to get ratified commits | +| 3. Translates commits into LogPath entries (log tail) | +| 4. Provides a Committer for writing | ++---------------------------+-------------------------------+ + | path, log tail, committer + v ++-----------------------------------------------------------+ +| Your Delta Connector | +| | +| Uses Kernel APIs to read and write: | +| Snapshot::builder_for(path) | +| .with_log_tail(commits) | +| .with_max_catalog_version(version) | +| .build(&engine) | +| snapshot.transaction(committer, &engine) | ++---------------------------+-------------------------------+ + | calls Kernel APIs + v ++-----------------------------------------------------------+ +| Delta Kernel | +| | +| Knows nothing about catalogs, table names, or IDs. | +| Sees: a path, log files, and a Committer trait. | +| Handles: log replay, data skipping, protocol compliance | ++-----------------------------------------------------------+ +``` + +### Key Kernel APIs for catalog-managed tables + +Four Kernel APIs form the integration surface between the catalog client and +Kernel: + +- **`SnapshotBuilder::with_log_tail(Vec)`** accepts a contiguous run of + commits from version `M` to version `N` inclusive (published or staged). The + sequence must be ascending, gap-free, and duplicate-free. See + [Reading catalog-managed tables](./reading.md#building-a-log-tail) for the exact + rules, including what `M` and `N` must equal with and without time travel. + +- **`SnapshotBuilder::with_max_catalog_version(Version)`** caps the snapshot + version at the catalog's latest ratified version. Kernel requires this to be set + for every catalog-managed snapshot, so readers come with an explicit + catalog-ratified upper bound rather than silently reading whatever happens to be + on the filesystem. + +- **`Committer` trait** defines how transactions are committed. A catalog + committer implements `commit()` to stage and ratify commits through the + catalog API, and `publish()` to copy ratified commits to the Delta log. + +- **`Snapshot::publish()`** publishes all unpublished catalog commits at the + current snapshot version. Published commits become visible to filesystem-based + readers and enable maintenance operations like checkpointing. If the snapshot + has no unpublished commits, `publish()` is a no-op and returns the same + snapshot reference without invoking the committer. If unpublished commits + exist, both the snapshot and the committer must be catalog-managed; otherwise + `publish()` errors. + +## Example: the read flow + +The following pseudocode shows how a catalog client typically integrates with +Kernel for reading. The exact API calls depend on your catalog. + +```rust,ignore +// 1. Resolve table name through catalog API +// catalog_client.get_table("catalog.schema.table") -> table_id, path, credentials + +// 2. Get ratified commits from catalog +// catalog_client.get_commits(table_id) -> commits, max_version + +// 3. Convert catalog response to LogPath entries. If your catalog returns +// (version, uuid, ...) tuples, format the filename as {version:020}.{uuid}.json +// first. See "Building a log tail" in reading.md. +// let log_tail: Vec = commits.into_iter() +// .map(|c| LogPath::staged_commit(table_root.clone(), &c.filename, c.timestamp, c.size)) +// .collect::>()?; + +// 4. Build Engine with vended credentials +// let engine = build_engine_with_credentials(path, credentials); + +// 5. Build Snapshot with log tail from catalog +// let snapshot = Snapshot::builder_for(path) +// .with_log_tail(log_tail) +// .with_max_catalog_version(max_version) +// .build(&engine)?; + +// 6. Read the table using standard Kernel APIs +// let scan = snapshot.scan_builder().build()?; +// for data in scan.execute(Arc::new(engine))? { ... } +``` + +The key insight: the catalog client handles all catalog-specific logic +(authentication, API calls, response parsing). By the time Kernel sees the request, +it is a standard `Snapshot::builder_for()` call with an extra log tail and version +cap. + +## Maintenance operations + +Catalog-managed tables restrict which maintenance operations a client may run. +Two layers enforce the rules: + +- **Kernel enforces** that checkpoints and log compaction run only on + **published** versions. Publish commits before running either. Version + checksum (CRC) files, by contrast, can be written for unpublished versions + as well. +- **The managing catalog enforces** which maintenance operations a client may + request. + +## See also + +- [Unity Catalog integration](../unity_catalog/overview.md) for a concrete + implementation using Unity Catalog. + +## What's next + +- [Implementing a catalog committer](./committer.md) describes how to implement + the `Committer` trait for your catalog. +- [Reading catalog-managed tables](./reading.md) covers how to provide catalog + commits to Kernel for snapshot construction. +- [Writing: the commit and publish flow](./writing.md) walks through the full + commit, ratify, and publish lifecycle. diff --git a/docs/user-guide/src/catalog_managed/reading.md b/docs/user-guide/src/catalog_managed/reading.md new file mode 100644 index 0000000000..769319f2a0 --- /dev/null +++ b/docs/user-guide/src/catalog_managed/reading.md @@ -0,0 +1,116 @@ +# Reading catalog-managed tables + +To read a catalog-managed table, you provide Kernel with a **log tail**: the set of +recent ratified commits your catalog client knows about. Kernel merges these with whatever +it finds on the filesystem to build a complete [Snapshot](../concepts/architecture.md). + +Before reading this page, make sure you understand +[Catalog-managed tables](./overview.md). + +## Table resolution is outside Kernel + +Kernel doesn't know about table names, table IDs, or catalog APIs. Before you can use +any Kernel API, the catalog client must: + +1. **Resolve the table name to a storage path** (e.g. `unity.my_schema.my_table` -> + `s3://bucket/path/to/table/`) +2. **Fetch storage credentials** (e.g. temporary AWS credentials for the table's S3 + location) +3. **Get the latest ratified commits** from the catalog (the log tail) + +Each catalog has its own APIs for these steps. + +## Why a log tail? + +A catalog-managed table can have a mix of published commits (already in `_delta_log/`) +and unpublished ratified commits (in `_staged_commits/` or inline in the catalog). +Kernel lists `_delta_log/` to find published commits and checkpoints, but it cannot +discover unpublished commits on its own. The log tail bridges this gap. + +For example, suppose versions 0-7 are published and versions 8-9 are staged: + +```text +_delta_log/ + 00000000000000000000.json (published) + ... + 00000000000000000007.json (published) + _staged_commits/ + 00000000000000000008..json (ratified, not published) + 00000000000000000009..json (ratified, not published) +``` + +Without a log tail, Kernel would only see versions 0-7. The log tail tells Kernel about +versions 8 and 9. + +## Building a log tail + +The catalog client calls the catalog's commits API to get the latest ratified +commits, then translates each one into a `LogPath`: + +```rust,ignore +use delta_kernel::LogPath; + +let log_path = LogPath::staged_commit( + table_url.clone(), // table root URL (must end with '/') + "00000000000000000008..json", // staged filename: ..json + 1234567890, // last modified timestamp + 4096, // file size in bytes +)?; +``` + +The filename follows the Delta staged-commit convention: a zero-padded 20-digit version, +the UUID assigned when the commit was staged, and a `.json` extension. Kernel mints the +UUID inside `CommitMetadata::staged_commit_path()` at write time; the writer passes the +resulting path to the catalog, and the catalog returns that same path on reads. If your +catalog returns commits as `(version, uuid, ...)` tuples, format the filename as +`{version:020}.{uuid}.json` before calling `LogPath::staged_commit`. Published commit +files use `{version:020}.json` and are loaded via `LogPath::try_new`, not +`LogPath::staged_commit`. The `table_url` argument must end with `/`, otherwise +`LogPath::staged_commit` returns an error. + +Then pass the resulting `Vec` to `SnapshotBuilder` along with the maximum +catalog-ratified version: + +```rust,ignore +use delta_kernel::Snapshot; + +let snapshot = Snapshot::builder_for(table_url) + .with_max_catalog_version(latest_version) // the latest version the catalog ratified + .with_log_tail(log_paths) // the commits the catalog returned + .build(&engine)?; +``` + +The log tail must be a **contiguous sequence** `M..=N`: ascending by version, no gaps, +no duplicates. The first version `M` can be any value `>= 0` (typical catalog clients +return a suffix of recent commits, not the full history). The last version `N` depends +on whether you are time-traveling: + +- Without time travel, `N` must equal `max_catalog_version`. +- With `.at_version(v)`, `N` must be `>= v`, and `max_catalog_version` must still be + set. Both constraints apply simultaneously. + +The log tail can overlap with published commits already in `_delta_log/`. For any +version present in both, Kernel uses the log-tail entry, as the Delta spec requires +readers to prefer the catalog-supplied commit when the catalog provides one. + +Call `.with_max_catalog_version(N)` to tell Kernel the highest version the catalog has +ratified. This prevents Kernel from loading filesystem commits beyond what the catalog +knows about. + +> [!NOTE] +> To time-travel to a specific version within a catalog-managed table, call both +> `.at_version(v)` and `.with_max_catalog_version(N)` on the builder. Call order does +> not matter. The requested version must not exceed `N`. + +For a complete Unity Catalog example, see +[Reading Unity Catalog tables](../unity_catalog/reading.md). + +After building the snapshot, reading works the same as for a filesystem-managed table: +build a scan, apply predicates, and read data. See [Building a scan](../reading/building_a_scan.md). + +## What's next + +- [Writing: the commit and publish flow](./writing.md) covers how writes work for + catalog-managed tables. +- [Implementing a catalog committer](./committer.md) describes the `Committer` trait in + detail. diff --git a/docs/user-guide/src/catalog_managed/writing.md b/docs/user-guide/src/catalog_managed/writing.md new file mode 100644 index 0000000000..765e285669 --- /dev/null +++ b/docs/user-guide/src/catalog_managed/writing.md @@ -0,0 +1,193 @@ +# Writing: the commit and publish flow + +To write to a catalog-managed table, you follow the same transaction pattern as a +filesystem-managed table. The difference is that your catalog `Committer` stages, ratifies, +and publishes commits through the catalog instead of writing directly to the filesystem. + +Before reading this page, make sure you understand [Catalog-managed tables](./overview.md). + +## The write lifecycle + +A catalog-managed write has four phases: + +```text +1. LOAD SNAPSHOT + Catalog client resolves the table and fetches recent commits. + Build snapshot with Snapshot::builder_for(path).with_log_tail(commits).build() + +2. COMMIT + Transaction generates actions. Committer stages them to _staged_commits/. + Committer calls the catalog API to ratify the staged commit. + +3. HANDLE RESULT + Committed: get post-commit snapshot, proceed to publish. + Conflict: rebase and retry. + Retryable: transient I/O error, retry. + +4. PUBLISH + Copy staged commits from _staged_commits/ to _delta_log/ as published delta files. + Returns a new snapshot with all commits marked as published. +``` + +## Phase 1: Load a snapshot + +Your catalog client resolves the table, fetches credentials and recent commits, then +builds a snapshot. See [Reading catalog-managed tables](./reading.md) for the full +details. + +```rust,ignore +// Catalog client resolves table name -> path + credentials + commits +let (path, credentials, commits, latest_version) = catalog_client.load_table(table_name)?; + +// Build snapshot with log tail and cap at the catalog-ratified version +let snapshot = Snapshot::builder_for(path) + .with_max_catalog_version(latest_version) + .with_log_tail(commits) + .build(&engine)?; +``` + +## Phase 2: Create a transaction and commit + +To begin a write, create a transaction with your catalog's `Committer`, add files, and +call `commit()`: + +```rust,ignore +// transaction() moves the Box into the Transaction, and commit() +// consumes the Transaction, so the boxed committer is gone by the time you need +// to publish. Construct a second committer for publish() in Phase 3 and clone +// any catalog-client state you need to keep in scope across both calls. +let committer = Box::new(MyCatalogCommitter::new( + catalog_client.clone(), + table_id.clone(), +)); +let mut txn = snapshot + .transaction(committer, &engine)? + .with_operation("INSERT".to_string()); + +// Drive your Parquet writer from the write context, then hand the resulting +// add-file metadata batch to the transaction. See the +// [Appending data](../writing/append.md) how-to for the full Parquet-writing flow. +let write_context = txn.unpartitioned_write_context()?; +// ... use write_context to produce add_metadata: Box +// matching txn.add_files_schema() ... +txn.add_files(add_metadata); + +// Commit the transaction. Kernel invokes committer.commit() internally; Phase 3 +// handles the result. +let commit_result = txn.commit(&engine)?; +``` + +The `?` on `txn.commit(&engine)?` only propagates non-recoverable errors. Successful +commits, conflicts, and retryable I/O errors arrive as the three variants of +`CommitResult` in the match below. Everything else (auth errors, catalog protocol +errors, etc.) bubbles out directly. + +When `txn.commit()` runs, Kernel: + +1. Assembles the actions: a leading `CommitInfo`, any Protocol/Metadata updates + this transaction makes (absent on steady-state appends), and Add/Remove file + actions. +2. Verifies that a catalog committer is being used. Kernel rejects `FileSystemCommitter` + for catalog-managed tables. +3. Calls `committer.commit(engine, actions, commit_metadata)`. + +Your committer then: + +1. Writes the actions to `_staged_commits/..json`. +2. Calls the catalog API to ratify the staged commit. +3. Returns `CommitResponse::Committed` or `CommitResponse::Conflict`. + +## Phase 3: Handle the result + +On success, `CommittedTransaction` provides the commit version and a **post-commit +snapshot** that reflects the newly committed state: + +```rust,ignore +use delta_kernel::transaction::CommitResult; + +match commit_result { + CommitResult::CommittedTransaction(committed) => { + let version = committed.commit_version(); + // post_commit_snapshot() returns an Option. For catalog-managed + // commits today, Kernel returns Some. The Option exists for + // incremental-development paths (e.g., table creation). Treat a + // None as an error rather than silently skipping publish, so the + // problem surfaces loudly if the invariant ever changes. + let post_commit = committed + .post_commit_snapshot() + .ok_or_else(|| Error::generic("missing post-commit snapshot"))?; + + // commit() consumed the Box from Phase 2. publish() only + // needs &dyn Committer, so construct a fresh instance here. This moves + // catalog_client and table_id; clone them if you need them for a retry + // loop around the whole write. + let publish_committer = + MyCatalogCommitter::new(catalog_client, table_id); + + // Proceed to publish (Phase 4). + let published_snapshot = post_commit.publish(&engine, &publish_committer)?; + } + CommitResult::ConflictedTransaction(conflicted) => { + // Another writer already committed at this version. + // `conflicted.conflict_version()` returns the version this transaction + // attempted. Rebase onto the new table state and retry. + } + CommitResult::RetryableTransaction(retryable) => { + // Transient I/O error. `retryable.error` gives the underlying cause; + // `retryable.transaction` is the original transaction you can retry + // without rebasing. Kernel reaches this arm only for `Error::IOError` + // variants; return other error kinds as-is rather than disguising + // them as IOError to force retry. + } +} +``` + +## Phase 4: Publish + +Call `Snapshot::publish()` on the post-commit snapshot (shown in Phase 3 above) to +make ratified commits visible as normal delta files and to unlock maintenance +operations. `Snapshot::publish()`: + +1. Finds all unpublished catalog commits in the snapshot's log segment. +2. Validates that the table is catalog-managed and the committer is a catalog committer. +3. Builds a `PublishMetadata` containing the list of `CatalogCommit` entries to publish. +4. Delegates to `committer.publish(engine, publish_metadata)`, which copies each staged + commit from `_staged_commits/` to `_delta_log/.json`. +5. Returns a **new snapshot** where all commits are marked as published. + +Publishing matters because: + +- It makes commits visible to filesystem-based readers. +- It enables maintenance operations such as checkpointing, which can only + operate on published versions. +- It reduces the number of commits the catalog needs to store and serve. + +## Maintenance operations + +Once commits are published, you can perform maintenance operations on the published +snapshot: + +```rust,ignore +// Checkpoint the published snapshot. checkpoint() returns +// (CheckpointWriteResult, SnapshotRef): the first element is an enum reporting +// whether a checkpoint was written or one already existed, and the second is +// the snapshot with the updated log segment. +let (_checkpoint_result, _post_checkpoint_snapshot) = + published_snapshot.checkpoint(&engine)?; +``` + +> [!NOTE] +> Maintenance operations on catalog-managed tables are subject to extra rules: +> Kernel requires published versions, and the managing catalog controls which +> operations a client may run. + +For a complete Unity Catalog example, see +[Writing to Unity Catalog tables](../unity_catalog/writing.md). + +## What's next + +- [Implementing a catalog committer](./committer.md): how to build a `Committer` for + your catalog. +- [Reading catalog-managed tables](./reading.md): how to provide catalog commits for + snapshot construction. +- [Appending data](../writing/append.md): the basics of writing data with Kernel. diff --git a/docs/user-guide/src/concepts/architecture.md b/docs/user-guide/src/concepts/architecture.md new file mode 100644 index 0000000000..8c0d39b3ca --- /dev/null +++ b/docs/user-guide/src/concepts/architecture.md @@ -0,0 +1,290 @@ +# Architecture overview + +Delta Kernel is a Rust library that encapsulates the Delta Lake protocol so you can build +connectors without understanding protocol internals. This matters because the Delta protocol +is complex and evolving. Kernel absorbs that complexity and exposes a small, stable API +surface for reading and writing Delta tables. + +## Terminology + +Before diving in, here are the key terms used throughout this guide: + +- **Compute engine** is a data processing framework like Apache Spark, Apache Flink, DuckDB, + Polars, or DataFusion. Each defines its own DataSource API for reading and writing tables + (e.g., Spark's DataSourceV2, Flink's Source/Sink APIs). + +- **Connector** is the integration layer between a compute engine and Delta tables. It + implements the compute engine's DataSource API (Table, Scan, Writer, etc.) and uses Delta + Kernel to fulfill those interfaces. For example, the Delta Spark connector implements + Spark's `Table`, `ScanBuilder`, `Scan`, and `PartitionReader` interfaces, delegating the + actual Delta protocol work to Kernel. + +- **Delta Kernel** is the library (this project). It implements the Delta protocol and + exposes APIs (`Snapshot`, `Scan`, `Transaction`) that connectors use. It never does I/O + directly. + +- **Engine trait** is the I/O and compute abstraction that Kernel calls into. It has four + required handlers (`StorageHandler`, `ParquetHandler`, `JsonHandler`, + `EvaluationHandler`) and one optional handler (`MetricsReporter`). + A `DefaultEngine` is provided. Connectors can implement their own for better performance + with their native data formats and I/O. + +## Layered design + +Delta Kernel is organized into layers. Each layer has a clear responsibility and a well-defined +interface to the layer above and below it. + +```text + ┌────────────────────────────────────────────┐ + │ Compute Engine │ + │ (Spark, Flink, DuckDB, Polars, ...) │ + └──────────────────┬─────────────────────────┘ + │ calls DataSource API + ┌──────────────────▼──────────────────────────┐ + │ Your Delta Connector │ + │ │ + │ Implements the compute engine's DataSource │ + │ API and uses Kernel to fulfill it │ + └──────────────────┬──────────────────────────┘ + │ calls Kernel APIs + ┌──────────────────▼──────────────────────────┐ + │ Delta Kernel (core logic) │ + │ │ + │ Snapshot · Scan · Transaction · Log Replay │ + │ Data Skipping · Predicate Pushdown │ + │ Protocol Compliance · Table Features │ + └──────────────────┬──────────────────────────┘ + │ calls into + ┌──────────────────▼──────────────────────────┐ + │ Engine trait (abstraction) │ + │ │ + │ EvaluationHandler · StorageHandler │ + │ JsonHandler · ParquetHandler │ + │ MetricsReporter (optional) │ + └──────────────────┬─────────────────────────┘ + │ implemented by + ┌──────────────────▼──────────────────────────┐ + │ DefaultEngine (or your custom engine) │ + │ │ + │ Arrow-based evaluation · object_store I/O │ + └──────────────────┬──────────────────────────┘ + │ + ┌──────────────────▼──────────────────────────┐ + │ Storage │ + │ (Local FS, S3, GCS, Azure, HDFS, ...) │ + └─────────────────────────────────────────────┘ +``` + +**Kernel** contains all Delta protocol logic: log replay, data skipping, +schema enforcement, table features, and transaction coordination. It never does I/O directly. + +**Engine** is a trait that the kernel calls whenever it needs I/O or expression evaluation. +You can use the built-in `DefaultEngine` (Arrow + `object_store`) or implement your own. +See [The Engine Trait](./engine_trait.md). + +**Your connector** implements your compute engine's DataSource API and calls kernel APIs +(`Snapshot`, `Scan`, `Transaction`) to do the Delta work. The kernel handles the protocol; +your connector controls execution, distribution, and data flow. See +[Building a Connector](../connector/overview.md) for details. + +## Key types + +### Snapshot + +A `Snapshot` is an immutable view of a Delta table at a specific version. It is the entry point +for everything: reading, writing, and inspecting table metadata. + +```rust,ignore +let snapshot = Snapshot::builder_for("/path/to/table") + .build(&engine)?; // returns Arc + +let snapshot_v5 = Snapshot::builder_for("/path/to/table") + .at_version(5) + .build(&engine)?; + +println!("Version: {}", snapshot.version()); +println!("Schema: {:?}", snapshot.schema()); +``` + +From a snapshot you can: +- Read the table schema and properties +- Build a `Scan` to read data +- Start a `Transaction` to write data +- Create a checkpoint + +### Scan + +A `Scan` reads data from a table. It is built from a snapshot via `ScanBuilder`: + +```rust,ignore +let scan = snapshot + .scan_builder() + .with_schema(my_schema) // column selection (optional) + .with_predicate(my_predicate) // filter pushdown (optional) + .build()?; +``` + +There are two ways to execute a scan: + +**Simple path.** `execute()` does everything for you: + +```rust,ignore +for data in scan.execute(engine)? { + let batch = data?.try_into_record_batch()?; + // process batch +} +``` + +**Advanced path.** `scan_metadata()` gives you control over parallelism: + +```rust,ignore +for metadata in scan.scan_metadata(engine)? { + let metadata = metadata?; + // Each ScanMetadata contains the files to read + // and per-file transforms to apply. + // You can distribute these across threads or workers. +} +``` + +The advanced path is how you build a distributed connector. See +[Building a Connector](../connector/overview.md) for details. + +### Transaction + +A `Transaction` writes data to a table. It is built from a snapshot: + +```rust,ignore +let mut txn = snapshot // Arc + .transaction(Box::new(FileSystemCommitter::new()), &engine)? + .with_operation("INSERT".to_string()) + .with_data_change(true); + +// Write Parquet files, then register their metadata. +txn.add_files(file_metadata); + +// Commit atomically +match txn.commit(&engine)? { + CommitResult::CommittedTransaction(c) => println!("v{}", c.commit_version()), + CommitResult::ConflictedTransaction(_) => { /* handle conflict */ } + CommitResult::RetryableTransaction(_) => { /* retry */ } +} +``` + +The `file_metadata` argument is an `EngineData` batch that matches +`txn.add_files_schema()`, not raw file paths. See +[Appending data](../writing/append.md) for the full pattern, including how to +build that batch from Parquet write results. + +> [!NOTE] +> `with_operation` applies to update transactions (append, delete, etc.). For +> create-table transactions the operation is fixed to `"CREATE TABLE"` and +> cannot be overridden. + +For schema evolution, start from the snapshot's `alter_table()` builder +instead of `transaction()`. See +[Altering a Table](../writing/alter_table.md). + +See [Quick Start: Writing a Table](../getting_started/quick_start_write.md) for a complete example. + +## How a read works + +When you call `scan.execute(engine)`, here is what happens internally: + +```text +1. LOG REPLAY (kernel) + Read delta log commits via engine.json_handler() and checkpoint + parquet files (including parquet sidecars) via engine.parquet_handler(). + Determine the set of active files for this table version. + +2. DATA SKIPPING (kernel) + If a predicate was provided, evaluate file-level statistics + (min/max values, null counts) to skip files that cannot match. + This happens via engine.evaluation_handler(). + +3. FILE READING (engine) + For each remaining file, read the Parquet data via + engine.parquet_handler().read_parquet_files(). + +4. TRANSFORM (kernel + engine) + Apply per-file transformations: partition value injection, + column mapping, deletion vectors. Produces the final logical + data that your connector consumes. +``` + +The kernel handles steps 1, 2, and 4. The engine handles step 3. This separation means +the kernel never touches raw bytes. It works purely with metadata and delegates all I/O. + +## How a write works + +```text +1. START TRANSACTION (kernel) + Create a Transaction from a snapshot. The snapshot pins the + table version you're writing against. + +2. WRITE DATA (engine / your code) + Write Parquet files using the engine. Collect file metadata + (path, size, statistics) and register it with the transaction + via add_files(). + +3. COMMIT (kernel + committer) + The kernel assembles the commit actions (CommitInfo, Add files, + etc.) and hands them to the Committer. For filesystem tables, + the Committer writes a JSON delta file atomically. For + catalog-managed tables, it goes through the catalog. + +4. HANDLE RESULT + CommittedTransaction: success. + ConflictedTransaction: another writer committed first. + RetryableTransaction: transient I/O error, safe to retry. +``` + +## EngineData: staying engine-agnostic + +The kernel never assumes your data is Arrow. Instead, it uses the `EngineData` trait, an +opaque interface that any engine can implement. The kernel accesses data through visitor +callbacks, not by inspecting columns directly. + +```text + Kernel Engine + ────── ────── + "I need columns [path, size]" + ──────────────────> + "Here are GetData accessors + for those columns" + <────────────────── + Visits rows via GetData +``` + +The `DefaultEngine` implements `EngineData` with `ArrowEngineData` (wrapping Arrow +`RecordBatch`). If you use the default engine, you can convert back to `RecordBatch` +with the `EngineDataArrowExt` trait. If you build a custom engine, you implement +`EngineData` for your own columnar format. + +See [The Engine Trait](./engine_trait.md) for more on how this works. + +## Crate structure + +The project is organized into several crates: + +| Crate | Description | +|-------|-------------| +| `delta_kernel` | Core library: protocol logic, table operations, trait definitions, default engine | +| `delta_kernel_ffi` | C/C++ Foreign Function Interface for cross-language integration | +| `delta_kernel_derive` | Procedural macros for internal code generation | +| `delta_kernel_unity_catalog` | Unity Catalog integration for catalog-managed tables (see [Unity Catalog Integration](../unity_catalog/overview.md)) | +| `unity_catalog_delta_client_api` | Trait definitions for Unity Catalog client implementations | +| `unity_catalog_delta_rest_client` | REST-based Unity Catalog client built on the client API | +| `acceptance` | Delta Acceptance Tests (DAT) validation suite | +| `test_utils` | Shared test utilities | +| `feature_tests` | Feature flag compatibility tests | + +## What's next + +- [The Engine Trait](./engine_trait.md) explains the `Engine` abstraction and its handlers in detail. +- [Building a Scan](../reading/building_a_scan.md) walks through reading data from a Delta table. +- [Building a Connector](../connector/overview.md) describes how to integrate Kernel with a compute engine. + +## See also + +- [Schema and Types](./schema_and_types.md) for Kernel's type system. +- [Appending Data](../writing/append.md) for a complete write example. diff --git a/docs/user-guide/src/concepts/engine_trait.md b/docs/user-guide/src/concepts/engine_trait.md new file mode 100644 index 0000000000..a2d8eab2ed --- /dev/null +++ b/docs/user-guide/src/concepts/engine_trait.md @@ -0,0 +1,304 @@ +# The Engine trait + +The `Engine` trait is the integration point between Delta Kernel and your connector. Kernel +implements the Delta protocol and gives you scan and write APIs, but it needs help with the +mechanics: listing files, reading and writing JSON and Parquet, and evaluating expressions for +data skipping and logical-to-physical transformations. Kernel never does any of this directly. +Instead, it calls into the `Engine` trait, which your connector implements. + +This matters because it lets Kernel stay format-agnostic and runtime-agnostic. You control +how I/O happens, what columnar format you use, and how expressions are evaluated. + +A [DefaultEngine](#the-default-engine) is provided that you can use out of the box. If you +need better performance or want to use your own data formats, you can build a custom engine. +See [Building a Connector](../connector/overview.md) for details. + +## The trait + +```rust,ignore +trait Engine: AsAny { + fn evaluation_handler(&self) -> Arc; + fn storage_handler(&self) -> Arc; + fn json_handler(&self) -> Arc; + fn parquet_handler(&self) -> Arc; +} +``` + +Kernel calls these methods whenever it needs to interact with the outside world. Each returns +a handler trait object that Kernel uses for a specific category of work. The four handlers +cover storage, JSON, Parquet, and expression evaluation. For observability, see +[Observability](../observability/observability.md). + +## The four handlers + +### StorageHandler + +File system operations. Kernel calls this to list and read files from the Delta log, and to +write commit and checkpoint files. + +| Method | Purpose | +|--------|---------| +| `list_from(path)` | List files lexicographically after `path` in the same directory | +| `read_files(files)` | Read byte ranges from one or more files | +| `copy_atomic(src, dst)` | Atomically copy a file (used for publishing commits) | +| `put(path, data, overwrite)` | Write raw bytes to a path (fails if `overwrite` is false and file exists) | +| `head(path)` | Get file metadata (size, modification time) without reading content | + +### JsonHandler + +Reads and writes JSON. Kernel uses this for Delta log commits (the `_delta_log/*.json` +files) and the JSON checkpoint manifest that references parquet sidecars. The sidecar +files themselves are parquet and are read by the `ParquetHandler` below. + +| Method | Purpose | +|--------|---------| +| `parse_json(strings, schema)` | Parse JSON strings into columnar `EngineData` | +| `read_json_files(files, schema, predicate)` | Read JSON files and return `EngineData` (predicate is an optional hint) | +| `read_json_files_with_cancellation(files, schema, predicate, token)` | As above, but a cooperative engine can abandon the read when `token` fires. Optional to override — see [Cancellation](#cancellation) | +| `write_json_file(path, data, overwrite)` | Atomically write a stream of `FilteredEngineData` rows as a newline-delimited JSON file (one JSON object per row, nulls omitted) | + +### ParquetHandler + +Reads and writes Parquet. Kernel uses this for checkpoint files (including parquet +checkpoint sidecars) and for reading data files during scans. + +| Method | Purpose | +|--------|---------| +| `read_parquet_files(files, schema, predicate)` | Read Parquet files into `EngineData` | +| `read_parquet_files_with_cancellation(files, schema, predicate, token)` | As above, but a cooperative engine can abandon the read when `token` fires — see [Cancellation](#cancellation) | +| `write_parquet_file(url, data)` | Write a stream of `EngineData` batches as a single Parquet file at the given URL | +| `read_parquet_footer(file)` | Read file footer metadata (schema, field IDs) without reading data | +| `read_parquet_footer_with_cancellation(file, token)` | As above, but a cooperative engine can abandon the footer read when `token` fires — see [Cancellation](#cancellation) | + +The Parquet handler also supports metadata columns (row index, file path) and field-ID-based +column matching for column mapping. + +### Cancellation + +Reading the Delta log for a large table can touch many commit and checkpoint files, and a +connector often wants to abandon that work when the request behind it goes away — a cancelled +query, a dropped RPC, a closed session. Kernel supports this cooperatively through an optional +[`CancellationToken`] that a caller attaches to a scan (see +[Advanced reads with scan_metadata()](../reading/scan_metadata.md#cancelling-a-scan)). + +Cancellation reaches the handlers in two ways, and an engine can honor either or both: + +- **Between batches.** Kernel polls the token at each action-batch boundary regardless of what the + handler does, so even a handler that ignores the token entirely still stops promptly between + reads. +- **During a read.** The `read_*_with_cancellation` variants above hand the token down into the + read itself. Overriding them lets a cooperative engine interrupt an I/O that is already in + flight — the case that matters when one file read blocks for a long time — instead of waiting for + it to finish before the next batch-boundary check. + +These variants are optional. Their default implementations return early if the token is already +cancelled and otherwise delegate to the plain `read_parquet_files` / `read_json_files` / +`read_parquet_footer`, so an existing handler keeps working unchanged and simply doesn't interrupt +mid-read. Whenever cancellation is observed, the read surfaces `Error::Cancelled` as a terminal +error — never a short or empty result that could be mistaken for a complete one. + +For how to implement the cancellation-aware variants, see +[Implementing the Engine Trait](../connector/implementing_engine.md#cancellation-aware-reads). + +[`CancellationToken`]: https://docs.rs/delta_kernel/latest/delta_kernel/cancellation/trait.CancellationToken.html + +### EvaluationHandler + +Expression evaluation. Kernel uses this for data skipping (evaluating predicates against +file statistics) and for applying per-file transformations (partition value injection, column +mapping). + +| Method | Purpose | +|--------|---------| +| `new_expression_evaluator(schema, expr, output_type)` | Create a reusable evaluator for an expression | +| `new_predicate_evaluator(schema, predicate)` | Create a reusable evaluator for a boolean predicate | +| `null_row(output_schema)` | Create a single-row, all-null `EngineData` with the given schema | +| `create_many(schema, rows)` | Create a multi-row `EngineData` from scalar values | + +The expression and predicate evaluators are reusable objects that you can call repeatedly on +different batches of `EngineData`. + +## The Default Engine + +The `DefaultEngine` is a batteries-included implementation that works out of the box: + +- Uses **Apache Arrow** as the in-memory data format +- Uses **`object_store`** for I/O (supports local FS, S3, GCS, Azure) +- Runs async I/O on a **Tokio** thread pool +- Supports multiple Arrow versions (see [Feature Flags](./feature_flags.md)) + +To construct one, create an object store and pass it to the builder: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# extern crate url; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::DeltaResult; +# fn example() -> DeltaResult<()> { +let url = url::Url::parse("file:///path/to/table")?; +let store = store_from_url(&url)?; +let engine = DefaultEngine::builder(store).build(); +# Ok(()) +# } +``` + +`DefaultEngine` also provides a convenience method for writing Parquet files: + +```rust,ignore +let engine_data = engine + .write_parquet(&data, &write_context) + .await?; +``` + +This is not part of the `Engine` trait itself. It's a helper on `DefaultEngine` that +orchestrates the lower-level `ParquetHandler` methods with a `WriteContext` (write directory, +schema, stats columns, partition values). + +## Configuring the Default Engine + +`DefaultEngine` uses a builder pattern that lets you customize the task executor and plug in a +metrics reporter. The builder starts from `DefaultEngine::builder(store)` and chains optional +configuration before calling `build()`: + +```rust,ignore +let engine = DefaultEngine::builder(store) + .with_metrics_reporter(reporter) + .with_task_executor(executor) + .build(); +``` + +All builder methods are optional. Calling `DefaultEngine::builder(store).build()` gives you a +fully functional engine with sensible defaults. + +### The TaskExecutor trait + +`DefaultEngine` uses asynchronous I/O internally, but Kernel's public APIs are synchronous. +The `TaskExecutor` trait bridges this gap by defining how async work gets scheduled and +awaited. It has four methods: + +| Method | Purpose | +|--------|---------| +| `block_on(future)` | Run an async future to completion and return the result. Must not panic when called inside an async context. | +| `spawn(future)` | Run a future in the background without waiting for its result. | +| `spawn_blocking(closure)` | Run a blocking closure on a thread where blocking is safe, returning a future of the result. | +| `enter()` | Enter the executor's runtime context, returning a guard. While the guard is held, `tokio::runtime::Handle::current()` resolves to the executor's runtime. | + +You don't need to implement `TaskExecutor` yourself unless you have a non-Tokio async runtime. +Kernel ships two Tokio-based implementations behind the `tokio` feature flag. + +### TokioBackgroundExecutor + +`TokioBackgroundExecutor` is the default executor. It spawns a **dedicated background thread** +running a single-threaded Tokio runtime. All async work is dispatched to that thread over a +channel. + +```rust,ignore +use delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor; + +let executor = TokioBackgroundExecutor::new(); +``` + +This is the right choice when: + +- You don't already have a Tokio runtime in your application. +- You want Kernel's I/O isolated from the rest of your process. +- You're building a standalone connector or CLI tool. + +Because it owns its runtime, `TokioBackgroundExecutor` works even when no external Tokio +runtime is active. On drop, it shuts down the background thread cleanly. + +### TokioMultiThreadExecutor + +`TokioMultiThreadExecutor` runs async work on a multi-threaded Tokio runtime. It comes in two +flavors. + +**Share an existing runtime.** If your application already has a Tokio runtime (for example, +a web server or a query engine), pass its handle so Kernel's I/O shares the same thread pool: + +```rust,ignore +use delta_kernel_default_engine::executor::tokio::TokioMultiThreadExecutor; + +let handle = tokio::runtime::Handle::current(); +let executor = TokioMultiThreadExecutor::new(handle); +``` + +The handle must come from a multi-threaded runtime. Passing a current-thread handle causes a +panic. + +**Own a dedicated runtime.** If you want a multi-threaded runtime that Kernel manages, use +`new_owned_runtime`. You can optionally set the number of worker threads and the maximum number +of blocking threads. Pass `None` for either to use Tokio's defaults: + +```rust,ignore +let executor = TokioMultiThreadExecutor::new_owned_runtime( + Some(4), // 4 worker threads + Some(64), // up to 64 blocking threads +)?; +``` + +> [!WARNING] +> Deeply nested `block_on` calls can exhaust Tokio's blocking thread pool and deadlock. If you +> set a small `max_blocking_threads`, keep nesting depth low. + +### Choosing an executor + +| Scenario | Executor | +|----------|----------| +| No existing Tokio runtime | `TokioBackgroundExecutor` (the default) | +| You have an existing multi-threaded Tokio runtime you want to share | `TokioMultiThreadExecutor::new(handle)` | +| You want a dedicated multi-threaded pool with custom sizing | `TokioMultiThreadExecutor::new_owned_runtime(workers, blocking)` | +| You need to isolate Kernel I/O from other async work | `TokioBackgroundExecutor` | + +To use a custom executor, pass it to the builder with `with_task_executor`: + +```rust,ignore +let executor = Arc::new(TokioMultiThreadExecutor::new(handle)); +let engine = DefaultEngine::builder(store) + .with_task_executor(executor) + .build(); +``` + +### Entering the runtime context + +Some libraries (for example, `object_store` or `reqwest`) require an active Tokio runtime +context to construct clients or resolve configuration. If you call such code outside of an +async function, `tokio::runtime::Handle::current()` will panic because no runtime is active. + +`DefaultEngine::enter()` solves this by entering the executor's runtime context: + +```rust,ignore +let guard = engine.enter(); +// Code here can call Handle::current() safely. +// The guard must be dropped before acquiring another. +drop(guard); +``` + +The returned guard keeps the runtime context active until it is dropped. If you acquire +multiple guards, you must drop them in reverse order. Dropping out of order causes a panic. + +## When to implement your own Engine + +You should implement `Engine` if: + +- You have your own columnar data format (not Arrow) +- You need custom I/O (e.g. your own distributed file system client) +- You want to use your own expression evaluation engine +- You need to control parallelism or resource usage beyond what `DefaultEngine` offers + +You do **not** need a custom engine to use different storage (S3, Azure, etc.). The +`DefaultEngine` supports all `object_store` backends. See +[Configuring Storage](../storage/configuring_storage.md). + +For a guide on implementing `Engine`, see +[Implementing the Engine Trait](../connector/implementing_engine.md). + +## What's next + +- [Building a Connector](../connector/overview.md) explains the role of a connector and how + the `Engine` fits into the bigger picture. +- [Implementing the Engine Trait](../connector/implementing_engine.md) walks through building + a custom `Engine` from scratch. +- [Configuring Storage](../storage/configuring_storage.md) shows how to point `DefaultEngine` + at S3, GCS, or Azure storage. diff --git a/docs/user-guide/src/concepts/feature_flags.md b/docs/user-guide/src/concepts/feature_flags.md new file mode 100644 index 0000000000..ce6b6299d8 --- /dev/null +++ b/docs/user-guide/src/concepts/feature_flags.md @@ -0,0 +1,77 @@ +# Feature flags + +The Delta Kernel ships as two crates with Cargo feature flags to keep each lightweight. The +`delta_kernel` crate has no required runtime dependencies beyond the Rust standard library. +The `delta_kernel_default_engine` crate adds Arrow, Tokio, and `object_store`. Everything else +is opt-in. + +## Recommended starting point + +For most connectors that use the built-in engine with Arrow: + +```toml +[dependencies] +delta_kernel = "0.23" +delta_kernel_default_engine = { version = "0.23", features = ["rustls"] } +``` + +## Complete feature reference + +### `delta_kernel_default_engine` features + +`delta_kernel_default_engine` provides out-of-the-box support for reading and writing Delta +tables on local filesystems and cloud object stores. + +| Feature | Description | +|---------|-------------| +| `rustls` | TLS via `rustls`. Recommended for most users because it requires no native dependency. | +| `native-tls` | TLS via your platform's native library (OpenSSL on Linux, Schannel on Windows, Secure Transport on macOS). | +| `arrow` | Build against the latest supported Arrow version (currently 59). | +| `arrow-59` | Pin to Arrow 59 (with `parquet` 59 and `object_store` 0.13). | +| `arrow-58` | Pin to Arrow 58 (with `parquet` 58 and `object_store` 0.13). | + +Pick exactly one of `rustls` or `native-tls`. Picking an `arrow-*` version on the default +engine automatically activates the same version on `delta_kernel` (the two crates must agree). + +### `delta_kernel` features + +| Feature | Description | +|---------|-------------| +| `arrow-conversion` | Convert between Kernel schema types and Arrow types (`TryIntoArrow`, `TryFromArrow`). | +| `arrow-expression` | Evaluate Kernel expressions over Arrow data. | +| `default-engine-base` | Shared Arrow modules used by the default engine. Pulled in automatically by `delta_kernel_default_engine`. | +| `arrow-59` / `arrow-58` | Pin the Arrow version used by Kernel's arrow modules. | +| `schema-diff` | Experimental schema diffing. | +| `internal-api` | Expose additional APIs that aren't yet stabilized. Some examples in this guide need this. | +| `prettyprint` | Arrow pretty-print helpers. Useful for debugging and examples. | +| `test-utils` | Test-only constructors for downstream crate tests. Pulls in `prettyprint`. Not for production use. | +| `integration-test` | Heavy integration tests (e.g., HDFS via `hdfs-native-object-store`). | + +> [!TIP] +> Each `arrow-*` version feature pulls in the matching `parquet` and `object_store` crate +> versions. If your connector already depends on a specific Arrow version, pin the matching +> feature on both crates to avoid duplicate transitive dependencies. + +## Common combinations + +**Read and write with the default engine:** + +```toml +delta_kernel = "0.23" +delta_kernel_default_engine = { version = "0.23", features = ["rustls"] } +``` + +**Custom engine using Arrow (no default engine):** + +```toml +delta_kernel = { version = "0.23", features = ["arrow-conversion", "arrow-expression"] } +``` + +**Minimal custom engine with no Arrow dependency at all:** + +```toml +delta_kernel = "0.23" +``` + +That gives you only the core Kernel types and traits. You implement `Engine` and `EngineData` +entirely in your own data format. diff --git a/docs/user-guide/src/concepts/schema_and_types.md b/docs/user-guide/src/concepts/schema_and_types.md new file mode 100644 index 0000000000..a9f0a13c58 --- /dev/null +++ b/docs/user-guide/src/concepts/schema_and_types.md @@ -0,0 +1,235 @@ +# Schemas and Data Types + +Kernel defines its own type system that mirrors the Delta protocol specification. This +type system is independent of any engine's type system (such as Arrow). Your engine +converts to and from Kernel types as needed. + +## Data types + +The `DataType` enum represents all types supported by the Delta protocol: + +### Primitive types + +| Constant | Rust equivalent | Description | +|----------|-----------------|-------------| +| `DataType::BOOLEAN` | `bool` | True or false | +| `DataType::BYTE` | `i8` | 8-bit signed integer | +| `DataType::SHORT` | `i16` | 16-bit signed integer | +| `DataType::INTEGER` | `i32` | 32-bit signed integer | +| `DataType::LONG` | `i64` | 64-bit signed integer | +| `DataType::FLOAT` | `f32` | 32-bit IEEE 754 float | +| `DataType::DOUBLE` | `f64` | 64-bit IEEE 754 float | +| `DataType::STRING` | `String` | UTF-8 string | +| `DataType::BINARY` | `Vec` | Arbitrary bytes | +| `DataType::DATE` | N/A | Calendar date (days since epoch) | +| `DataType::TIMESTAMP` | N/A | Microsecond precision, adjusted to UTC | +| `DataType::TIMESTAMP_NTZ` | N/A | Microsecond precision, no timezone | + +Both timestamp types are microsecond precision. A Parquet file can store timestamps at a coarser physical precision, such as millisecond, when it was written by a non-kernel writer. Kernel always expects microsecond values, so the reader must rescale them to microseconds. The default engine does this for you. If you implement your own engine, your handlers must perform this conversion themselves when reading both data files and metadata, such as checkpoint statistics. + +#### Decimal + +Decimals have a precision (1 to 38 inclusive) and a scale (0 to precision inclusive): + +```rust,no_run +# extern crate delta_kernel; +# use delta_kernel::DeltaResult; +# use delta_kernel::schema::DataType; +# fn main() -> DeltaResult<()> { +let price_type = DataType::decimal(18, 2)?; +# Ok(()) +# } +``` + +### Complex types + +#### Array + +An ordered sequence of elements, all of the same type: + +```rust,no_run +# extern crate delta_kernel; +# use delta_kernel::schema::{ArrayType, DataType}; +// Array of nullable strings +let array_type = DataType::from(ArrayType::new(DataType::STRING, true)); +``` + +The `contains_null` parameter indicates whether elements can be null. + +#### Map + +A collection of key-value pairs: + +```rust,no_run +# extern crate delta_kernel; +# use delta_kernel::schema::{DataType, MapType}; +// Map from string keys to nullable integer values +let map_type = DataType::from(MapType::new(DataType::STRING, DataType::INTEGER, true)); +``` + +Map keys are never null. The `value_contains_null` parameter controls whether values +can be null. + +#### Struct + +A named collection of fields (see [Schemas](#schemas) below). Structs can be nested: + +```rust,no_run +# extern crate delta_kernel; +# use delta_kernel::schema::{DataType, StructField, StructType}; +# use delta_kernel::DeltaResult; +# fn main() -> DeltaResult<()> { +let address_type = StructType::try_new([ + StructField::nullable("street", DataType::STRING), + StructField::nullable("city", DataType::STRING), + StructField::nullable("zip", DataType::STRING), +])?; + +let person_type = StructType::try_new([ + StructField::not_null("name", DataType::STRING), + StructField::nullable("address", address_type), +])?; +# Ok(()) +# } +``` + +#### Variant + +A semi-structured type that can hold any value. The physical representation uses a struct +with `metadata` and `value` fields (both binary). To create an unshredded variant column: + +```rust,no_run +# extern crate delta_kernel; +# use delta_kernel::schema::DataType; +let variant_type = DataType::unshredded_variant(); +``` + +## Schemas + +A schema is a `StructType`, an ordered collection of named, typed fields. The type +aliases `Schema` and `SchemaRef` (`Arc`) are used throughout the API. + +### Creating a schema + +```rust,no_run +# extern crate delta_kernel; +# use delta_kernel::DeltaResult; +# use delta_kernel::schema::{DataType, StructField, StructType}; +# fn main() -> DeltaResult<()> { +let schema = StructType::try_new([ + StructField::not_null("id", DataType::LONG), + StructField::nullable("name", DataType::STRING), + StructField::nullable("score", DataType::DOUBLE), +])?; +# Ok(()) +# } +``` + +`try_new` returns an error if the schema contains duplicate field names +(case-insensitive, since Delta column names are case-insensitive). + +`StructType::builder()` provides a builder for incremental construction: + +```rust,no_run +# extern crate delta_kernel; +# use delta_kernel::DeltaResult; +# use delta_kernel::schema::{DataType, StructField, StructType}; +# fn main() -> DeltaResult<()> { +let schema = StructType::builder() + .add_field(StructField::not_null("id", DataType::LONG)) + .add_field(StructField::nullable("name", DataType::STRING)) + .build()?; +# Ok(()) +# } +``` + +### Querying a schema + +```rust,ignore +// Look up a field by name +if let Some(field) = schema.field("name") { + println!("{}: {:?}, nullable={}", field.name(), field.data_type(), field.is_nullable()); +} + +// Check if a field exists +assert!(schema.contains("id")); + +// Get the positional index of a field +let idx = schema.index_of("name"); // Some(1) + +// Iterate over all fields +for field in schema.fields() { + println!("{}", field.name()); +} + +// Number of fields +let n = schema.num_fields(); +``` + +### Projecting a schema + +`project()` creates a new schema with a subset of fields. The output preserves the +order you specify: + +```rust,ignore +// Table schema: [id, name, email, created_at] +// Select only [email, id] in that order +let projected = schema.project(&["email", "id"])?; +``` + +See [Column Selection](../reading/column_selection.md) for how this is used in scans. + +## Fields + +A `StructField` has a name, data type, nullability flag, and optional metadata: + +```rust,ignore +// Non-nullable field +let id = StructField::not_null("id", DataType::LONG); + +// Nullable field +let name = StructField::nullable("name", DataType::STRING); + +// Field with explicit nullability +let score = StructField::new("score", DataType::DOUBLE, true); +``` + +### Field metadata + +Fields can carry arbitrary key-value metadata: + +```rust,ignore +let field = StructField::nullable("price", DataType::decimal(18, 2)?) + .with_metadata([("description", "Unit price in USD")]); +``` + +Metadata is stored as a `HashMap`. The `MetadataValue` enum +supports strings, numbers (`i64`), booleans, and arbitrary JSON. + +## Reading a table's schema + +Every `Snapshot` exposes the table's schema: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn main() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let engine = DefaultEngine::builder(store_from_url(&url)?).build(); +let snapshot = Snapshot::builder_for(url).build(&engine)?; +let schema = snapshot.schema(); + +for field in schema.fields() { + println!("{}: {:?}", field.name(), field.data_type()); +} +# Ok(()) +# } +``` + +## What's next + +- [Column Selection](../reading/column_selection.md): projecting columns in a scan +- [Creating a Table](../writing/create_table.md): defining a schema for a new table diff --git a/docs/user-guide/src/connector/engine_data.md b/docs/user-guide/src/connector/engine_data.md new file mode 100644 index 0000000000..2f09991441 --- /dev/null +++ b/docs/user-guide/src/connector/engine_data.md @@ -0,0 +1,282 @@ +# The EngineData trait + +The `EngineData` trait is the kernel's interface for columnar data. Any data that flows between the +kernel and your connector is represented as `EngineData`. + +If you use the `DefaultEngine`, you get `ArrowEngineData`, a wrapper around an arrow `RecordBatch` +that implements the `EngineData` trait, and don't need to implement this trait. This page is for +connector builders who want to use a different columnar data format. + +## The trait + +```rust,ignore +pub trait EngineData: AsAny { + fn visit_rows( + &self, + column_names: &[ColumnName], + visitor: &mut dyn RowVisitor, + ) -> DeltaResult<()>; + + fn len(&self) -> usize; + + fn is_empty(&self) -> bool { + self.len() == 0 + } + + fn append_columns( + &self, + schema: SchemaRef, + columns: Vec, + ) -> DeltaResult>; + + fn apply_selection_vector( + self: Box, + selection_vector: Vec, + ) -> DeltaResult>; + + fn has_field(&self, name: &ColumnName) -> bool; +} +``` + +The five required methods to implement are `visit_rows`, `len`, `append_columns`, +`apply_selection_vector`, and `has_field`. The `is_empty` method has a default +implementation that delegates to `len`. + +`has_field` returns `true` if a field at the given (possibly nested) path exists in the +data's schema. For a top-level field named `"foo"`, pass `column_name!("foo")`. For +nested fields, each non-leaf element of the path must be a struct field at that level. + +## visit_rows and the visitor pattern + +Kernel uses a visitor pattern to access the actual data that's inside an `EngineData`. This pattern +means the connector can call into kernel with a reference to the data, which makes reasoning about +data lifetimes simpler. In particular, engines don't need to worry about keeping data alive past the +invocation of the visitor. + +`visit_rows` is the core data extraction method. The kernel never inspects your columns +directly. Instead, it passes a `RowVisitor` that knows which columns it needs, and your +implementation provides typed accessors (`GetData`) for those columns. + +The flow: + +```text +Kernel Your EngineData +------ --------------- +"I need columns [path, size]" + visit_rows(column_names, visitor) + 1. Look up the requested columns + 2. Create a GetData accessor per column + 3. Call visitor.visit(row_count, &getters) + +Kernel (inside the visitor) + for row in 0..row_count: + path = getters[0].get_str(row, "path")? + size = getters[1].get_long(row, "size")? +``` + +### GetData + +`GetData` is the typed accessor trait. Each accessor handles one column. The kernel calls +the appropriate typed method based on the column's data type: + +| Method | Return type | Delta type | +|--------|-------------|------------| +| `get_bool(row, name)` | `bool` | BOOLEAN | +| `get_byte(row, name)` | `i8` | BYTE | +| `get_short(row, name)` | `i16` | SHORT | +| `get_int(row, name)` | `i32` | INTEGER | +| `get_long(row, name)` | `i64` | LONG | +| `get_float(row, name)` | `f32` | FLOAT | +| `get_double(row, name)` | `f64` | DOUBLE | +| `get_date(row, name)` | `i32` | DATE | +| `get_timestamp(row, name)` | `i64` | TIMESTAMP | +| `get_decimal(row, name)` | `i128` | DECIMAL | +| `get_str(row, name)` | `&str` | STRING | +| `get_binary(row, name)` | `&[u8]` | BINARY | +| `get_list(row, name)` | `ListItem` | ARRAY (of strings) | +| `get_map(row, name)` | `MapItem` | MAP (string keys and values) | + +All methods return `DeltaResult>`. A `None` value means the field is null. By +default, every method returns an "unexpected type" error, so you only need to implement the +one that matches your column's type. + +`ListItem` provides access to a row's list of strings. Call `get(index)` to retrieve a +single element, or `materialize()` to collect all elements into a `Vec`. + +`MapItem` provides access to a row's string-to-string map. Call `get(key)` to look up a +value by key, or `materialize()` to collect all entries into a `HashMap`. +If a value is null, `get` returns `None` and `materialize` drops that entry. + +Not every possible data-type is covered (i.e. no `Map`). The trait only covers the data +types the kernel needs to fuction, and no more. + +### TypedGetData + +The `TypedGetData` trait provides a convenience wrapper which is automatically implemented for most +types that implement `GetData`. Instead of calling the specific `get_*` method for each type, you +can write generic code that dispatches based on the Rust type: + +```rust,ignore +// Without TypedGetData: explicit method per type +let path: Option<&str> = getters[0].get_str(row, "path")?; +let size: Option = getters[1].get_long(row, "size")?; + +// With TypedGetData: type-driven dispatch +let path: Option = getters[0].get_opt(row, "path")?; +let size: Option = getters[1].get_opt(row, "size")?; +``` + +`TypedGetData` also provides a `get` method that returns `DeltaResult` (without +`Option`), returning an error if the value is null. + +### RowVisitor + +Engines don't need to implement `RowVisitor`. The kernel provides its own visitors. Your +`visit_rows` implementation needs construct the correct getters array and then call +`visitor.visit(row_count, &getters)` once. + +The visitor declares which columns and types it expects via `selected_column_names_and_types()`, +which returns `(&'static [ColumnName], &'static [DataType])`. Your implementation should validate +that the requested columns exist and have compatible types before creating getters. + +`RowVisitor` also provides a convenience method `visit_rows_of(&mut self, data: &dyn EngineData)`. +This calls `data.visit_rows(...)` with the column names from +`selected_column_names_and_types()`, saving you from extracting them manually. + +## append_columns + +The kernel calls `append_columns` to add new columns to your data. This happens when committing to +the table, as the kernel needs to build up the commit log entries. For example, row-tracking +information is added to the data that needs to be written into the delta log this way. + +```rust,ignore +fn append_columns( + &self, + schema: SchemaRef, + columns: Vec, +) -> DeltaResult>; +``` + +- **`schema`** describes only the new columns being appended (not the full result schema) +- **`columns`** contains the data as `ArrayData`, the kernel's generic columnar + representation that you will need to convert to your engine's format. +- Returns a new `EngineData` with the original columns plus the appended columns +- The row count of the new columns must match the existing data + +## apply_selection_vector + +The kernel calls `apply_selection_vector` to filter rows. This is used for +deletion vector support and other row-level filtering. + +```rust,ignore +fn apply_selection_vector( + self: Box, + selection_vector: Vec, +) -> DeltaResult>; +``` + +- Within the selection_vector, `true` means keep the row, `false` means remove it +- If the selection vector is shorter than the data, remaining rows are kept +- This consumes the `EngineData` (`self: Box`), so you can implement this in-place if your + format supports it + +## FilteredEngineData + +The kernel pairs `EngineData` with a selection vector in `FilteredEngineData`. This is just a +convenience type since the two appear together in many situations. For example, this is used in +write paths (e.g., `JsonHandler::write_json_file` receives an iterator of `FilteredEngineData`). + +Construct a `FilteredEngineData` with `try_new`, which validates that the selection vector +is not longer than the data: + +```rust,ignore +let filtered = FilteredEngineData::try_new(data, selection_vector)?; +``` + +You can also convert a `Box` directly with `.into()`, which selects all +rows (equivalent to an empty selection vector): + +```rust,ignore +let filtered: FilteredEngineData = engine_data.into(); +``` + +Key methods: + +| Method | Purpose | +|--------|---------| +| `try_new(data, selection_vector)` | Construct with validation (errors if vector is longer than data) | +| `with_all_rows_selected(data)` | Wrap data with an empty selection vector (all rows kept) | +| `data()` | Access the underlying `EngineData` | +| `selection_vector()` | Get the boolean selection vector as `&[bool]` | +| `apply_selection_vector()` | Applies the filter, keeping only selected rows and consuming self | +| `into_parts()` | Decompose into `(Box, Vec)`. | + +> [!WARNING] +> If you call `into_parts()`, you MUST keep the returned parts paired. That is, the `Box EngineData>` _is not_ valid for every row, only for those rows as desribed below. + +In the selection vector, `true` means keep the row, `false` meands remove it. If the selection +vector is shorter than the data, uncovered rows are implicitly selected. If the selection vector is +longer than the data, `try_new` returns an error. + +### FilteredRowVisitor + +The `FilteredRowVisitor` trait processes `FilteredEngineData` with automatic row filtering. +Instead of receiving a raw row count, your visitor gets a `RowIndexIterator` that yields +only the indices of selected rows: + +```rust,ignore +pub trait FilteredRowVisitor { + fn selected_column_names_and_types(&self) -> (&'static [ColumnName], &'static [DataType]); + + fn visit_filtered<'a>( + &mut self, + getters: &[&'a dyn GetData<'a>], + rows: RowIndexIterator<'_>, + ) -> DeltaResult<()>; + + fn visit_rows_of(&mut self, data: &FilteredEngineData) -> DeltaResult<()>; +} +``` + +The default `visit_rows_of` method handles all the plumbing: extracting the selection +vector, building the bridge to `RowVisitor`, and calling `visit_rows`. Your implementation +only needs to provide `selected_column_names_and_types` and `visit_filtered`. + +Call `rows.num_rows()` inside `visit_filtered` to get the total row count (including +deselected rows), which is useful for sizing output vectors. + +## ArrowEngineData (the default) + +The `DefaultEngine` uses `ArrowEngineData`, which wraps an Arrow `RecordBatch`: + +```rust,ignore +use delta_kernel::engine::arrow_data::ArrowEngineData; + +// Wrap a RecordBatch +let engine_data = ArrowEngineData::new(record_batch); + +// Access the underlying RecordBatch by reference +let batch_ref = engine_data.record_batch(); + +// Convert back from a Box (requires the arrow-related default features) +use delta_kernel::engine::arrow_data::EngineDataArrowExt; +let batch = engine_data_box.try_into_record_batch()?; +``` + +`ArrowEngineData` also provides `try_from_engine_data(engine_data)` to downcast a +`Box` to `Box`, returning an error if the underlying type +is not `ArrowEngineData`. + +If you use the default engine, you work with `ArrowEngineData` and never need to implement +`EngineData` yourself. `EngineDataArrowExt` provides `try_into_record_batch()` for +converting the opaque `EngineData` trait object back to an Arrow `RecordBatch` for your +connector's use. This trait is implemented for both `Box` and +`DeltaResult>`, so you can call it directly on scan results. + +## What's next + +- [Implementing the Engine trait](./implementing_engine.md) covers the four handler traits + your engine must provide +- [Building a Scan](../reading/building_a_scan.md) shows how scan results flow through + `EngineData` diff --git a/docs/user-guide/src/connector/implementing_engine.md b/docs/user-guide/src/connector/implementing_engine.md new file mode 100644 index 0000000000..024c81f787 --- /dev/null +++ b/docs/user-guide/src/connector/implementing_engine.md @@ -0,0 +1,322 @@ +# Implementing the Engine trait + +The `Engine` trait is the main integration point between your connector and Delta Kernel. For +background on what the Engine trait is and when you need a custom one, see the +[Connector Overview](./overview.md) and [The Engine Trait](../concepts/engine_trait.md). + +## The Engine trait + +The `Engine` trait has four required methods, each returning a handler: + +```rust,ignore +pub trait Engine { + fn evaluation_handler(&self) -> Arc; + fn storage_handler(&self) -> Arc; + fn json_handler(&self) -> Arc; + fn parquet_handler(&self) -> Arc; +} +``` + +You don't have to implement all four handlers from scratch. A common approach is to start +with `DefaultEngine` and selectively replace handlers. For example, you might provide a +custom `ParquetHandler` that reads into your engine's native columnar format while reusing +the default handlers for everything else. + +Many of the `Engine` handlers take or return `EngineData`. See [EngineData](engine_data.md) for more +information about this type. + +## StorageHandler + +`StorageHandler` provides file system operations. The kernel calls this to list and read +files (as bytes) from storage. + +```rust,ignore +pub trait StorageHandler { + fn list_from(&self, path: &Url) + -> DeltaResult>>>; + + fn read_files(&self, files: Vec) + -> DeltaResult>>>; + + fn copy_atomic(&self, src: &Url, dest: &Url) -> DeltaResult<()>; + + fn put(&self, path: &Url, data: Bytes, overwrite: bool) -> DeltaResult<()>; + + fn head(&self, path: &Url) -> DeltaResult; +} +``` + +### Key contracts + +- **`list_from`**: Results must be sorted lexicographically by path. If the path ends with + `/`, list all files in that directory. Otherwise, list files lexicographically greater than + the given path in the same directory. + +- **`copy_atomic`**: Must fail with `Error::FileAlreadyExists` if the destination exists. + This is used for commit publishing in catalog-managed tables. + +- **`put`**: Writes raw bytes to the given path. If `overwrite` is false and the file already + exists, must fail with `Error::FileAlreadyExists`. + +- **`head`**: Must return `Error::FileNotFound` if the file doesn't exist. + +- **`read_files`**: Each `FileSlice` is a `(Url, Option>)`. When the range is + `None`, read the entire file. + +### Default implementation + +The `DefaultEngine` uses [`object_store`](https://docs.rs/object_store) for storage, which supports +local filesystem, S3, GCS, and Azure out of the box. + +## JsonHandler + +`JsonHandler` reads and writes JSON. The kernel uses this for Delta log commits +(`_delta_log/*.json`) and checkpoint sidecars. + +```rust,ignore +pub trait JsonHandler { + fn parse_json( + &self, + json_strings: Box, + output_schema: SchemaRef, + ) -> DeltaResult>; + + fn read_json_files( + &self, + files: &[FileMeta], + physical_schema: SchemaRef, + predicate: Option, + ) -> DeltaResult; + + fn write_json_file( + &self, + path: &Url, + data: DeltaResultIterator<'_, FilteredEngineData>, + overwrite: bool, + ) -> DeltaResult<()>; +} +``` + +### Key contracts + +- **`parse_json`**: Input is a single-column batch of strings (JSON objects). Output + columns match the `output_schema`. Missing fields should produce nulls for nullable columns. + +- **`read_json_files`**: Data must be returned in file order (same order as the `files` slice + argument), and rows within a file must be in source order. The predicate is an optional hint that + the engine may ignore. If applied, evaluate exact row values conservatively; unsupported + expressions and missing references remain unknown. + +- **`write_json_file`**: Must write newline-delimited JSON (one JSON object per line). Null + columns should be omitted from the output to save space. The write must be atomic. If + `overwrite` is false and the file exists, fail with an error. + +### Default implementation + +The `DefaultEngine` uses `arrow_json` for parsing and the `object_store` crate for I/O. + +## ParquetHandler + +`ParquetHandler` reads and writes Parquet files. This is typically the most important +handler to customize, since it's on the critical path for data reading performance. + +```rust,ignore +pub trait ParquetHandler { + fn read_parquet_files( + &self, + files: &[FileMeta], + physical_schema: SchemaRef, + predicate: Option, + ) -> DeltaResult; + + fn write_parquet_file( + &self, + location: Url, + data: DeltaResultIteratorStatic>, + ) -> DeltaResult<()>; + + fn read_parquet_footer(&self, file: &FileMeta) -> DeltaResult; +} +``` + +### Key contracts for `read_parquet_files` + +**Column resolution**: When reading, the handler must resolve columns from the Parquet file +to the `physical_schema`: + +1. If a `StructField` in the schema has a field ID (via `ColumnMetadataKey::ParquetFieldId` + metadata), match by field ID first. +2. Otherwise, fall back to matching by column name. +3. If no match is found: return nulls for nullable columns, or an error for non-nullable + columns. + +**Column Ordering**: Columns must be returned in the order specified in the `physical_schema` +argument, which is _not_ necessarily the order they may be specified in the parquet file itself. + +**Missing Columns**: If a column is specified in the schema, and is nullable, the parquet reader +must return a column of all nulls. + +**Ordering**: Like `JsonHandler`, data must be returned in file order, and rows within a +file must be in source order. + +**Predicate hint**: If applied, the complete predicate may discard data only when conservative +evaluation proves it cannot match. Footer min/max may be cast only when the cast preserves their +bounds; unsupported expressions and missing references remain unknown. + +**Metadata columns**: The handler must support two virtual metadata columns that are not +stored in the Parquet file but generated at read time: + +| Metadata column | How to detect | Type | Values | +|-----------------|---------------|------|--------| +| Row index | `StructField` created with `MetadataColumnSpec::RowIndex` | `LONG`, non-nullable | Sequential 0-based position within the file | +| File name | `StructField` has reserved field ID `2147483646` | `STRING`, non-nullable | Full file path/URL | + +**Footer reading**: `read_parquet_footer` reads only the Parquet metadata (no data). If the +file has field IDs (column mapping), they must be preserved in the returned schema's +`StructField` metadata under the `ParquetFieldId` key. + +### Default implementation + +The `DefaultEngine` uses the Apache Arrow Parquet reader/writer with support for column +projection, predicate pushdown, metadata columns, and field-ID-based column matching. + +## Cancellation-aware reads + +When a caller attaches a `CancellationToken` to a scan (see +[Cancelling a scan](../reading/scan_metadata.md#cancelling-a-scan)), Kernel threads it down to the +`JsonHandler` and `ParquetHandler` reads through three optional methods: + +```rust,ignore +fn read_json_files_with_cancellation( + &self, + files: &[FileMeta], + physical_schema: SchemaRef, + predicate: Option, + cancellation_token: Option, +) -> DeltaResult; + +fn read_parquet_files_with_cancellation(/* same shape as above */) -> DeltaResult; + +fn read_parquet_footer_with_cancellation( + &self, + file: &FileMeta, + cancellation_token: Option, +) -> DeltaResult; +``` + +You do not have to implement these. Each has a default that returns `Error::Cancelled` if the +token is already cancelled and otherwise delegates to its plain counterpart. So an engine that +never overrides them still reads correctly and still stops between batches (Kernel polls the token +at every action-batch boundary on its own) — it just won't interrupt a read that is already in +flight. + +Override them when a single file read can block long enough that waiting for it defeats the point +of cancelling — a large checkpoint over slow storage, say. The `DefaultEngine` does this by racing +each read against the token: + +```rust,ignore +fn read_parquet_files_with_cancellation( + &self, + files: &[FileMeta], + physical_schema: SchemaRef, + predicate: Option, + cancellation_token: Option, +) -> DeltaResult { + // Kick off the async read as usual, then poll the read future and the token's + // `cancelled_future()` together. If cancellation wins the race, drop the in-flight + // work and yield `Err(Error::Cancelled)` as the iterator's terminal item. +} +``` + +### The contract you must uphold + +- **Surface cancellation as `Error::Cancelled`, never as a short read.** A cancelled read must not + return fewer rows, an empty iterator, or a bare `None` — anything Kernel could mistake for a + complete result. Emit `Error::Cancelled` as the terminal item so a partial log replay can never + look like a finished one. +- **Kernel already handles the pre-read check.** The default bodies fail fast on an + already-cancelled token before delegating, so an override can skip that and focus on interrupting + I/O once it has started. + +### The CancellationToken trait + +The token a caller supplies implements this trait: + +```rust,ignore +pub trait CancellationToken: Send + Sync { + fn is_cancelled(&self) -> bool; + fn cancelled_future(&self) -> CancelledFuture<'_>; +} +``` + +Kernel and your engine only *consume* a token; the caller creates and fires it. `is_cancelled` is +the cheap synchronous poll Kernel uses between batches. `cancelled_future` returns a future that +resolves once the token fires — this is what an async engine selects against to wake a read that is +blocked in I/O. Back it with your runtime's own notification primitive (for example, wrapping +`tokio_util::sync::CancellationToken`); it cannot be synthesized from `is_cancelled` alone without +busy-polling. + +## EvaluationHandler + +`EvaluationHandler` creates reusable evaluators for expressions and predicates. The kernel +uses this for data skipping (evaluating predicates against file statistics) and for per-file +transformations (partition value injection, row tracking). + +```rust,ignore +pub trait EvaluationHandler { + fn new_expression_evaluator( + &self, + input_schema: SchemaRef, + expression: ExpressionRef, + output_type: DataType, + ) -> DeltaResult>; + + fn new_predicate_evaluator( + &self, + input_schema: SchemaRef, + predicate: PredicateRef, + ) -> DeltaResult>; + + fn null_row(&self, output_schema: SchemaRef) + -> DeltaResult>; + + fn create_many( + &self, + schema: SchemaRef, + rows: &[&[Scalar]], + ) -> DeltaResult>; +} +``` + +The returned evaluators are reusable objects. The kernel creates them once and calls +`evaluate()` on multiple batches: + +```rust,ignore +pub trait ExpressionEvaluator { + fn evaluate(&self, batch: &dyn EngineData) -> DeltaResult>; +} + +pub trait PredicateEvaluator { + fn evaluate(&self, batch: &dyn EngineData) -> DeltaResult>; +} +``` + +### Key contracts + +- **Expression evaluators** produce one output row per input row. If `output_type` is a + struct, its fields describe the output columns. Otherwise, the output is a single column. + +- **Predicate evaluators** produce a single nullable boolean column. `true` means the row + matches, `false` or `null` means it doesn't. + +- **`null_row`** creates a single-row `EngineData` with all null values. The kernel uses + this internally for partition column construction. + +- **`create_many`** creates a multi-row `EngineData` by applying the given schema to + multiple rows of `Scalar` values. Each element in `rows` contains one scalar per top-level + field in the schema. Returns an error if any row's scalar count doesn't match the schema's + field count, or if a scalar value's type doesn't match its corresponding field. + +### Default implementation + +The `DefaultEngine` uses Arrow compute kernels for expression evaluation. diff --git a/docs/user-guide/src/connector/overview.md b/docs/user-guide/src/connector/overview.md new file mode 100644 index 0000000000..fcb8190da5 --- /dev/null +++ b/docs/user-guide/src/connector/overview.md @@ -0,0 +1,172 @@ +# Building a connector: overview + +A **connector** is the adapter layer between a **compute engine** (Spark, Flink, DuckDB, +Polars, DataFusion, or your own query engine) and Delta tables. This matters because +compute engines know nothing about the Delta protocol. They expose their own DataSource +APIs, and the connector translates those calls into Delta Kernel operations. + +## The big picture + +Every compute engine defines its own interfaces for reading and writing data. For example: + +- **Apache Spark** has [DataSourceV2](https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/connector/catalog/Table.html): + `Table`, `ScanBuilder`, `Scan`, `Batch`, `PartitionReader`, `WriteBuilder`, and others. +- **Apache Flink** has its Source/Sink APIs: + `Source`, `SplitEnumerator`, `SourceReader`, `Sink`, `SinkWriter`, and others. +- **DuckDB** has its [Extension API](https://duckdb.org/docs/extensions/overview.html): + `TableFunction`, `TableFunctionBindInput`, and others. + +Building a Delta connector means implementing these compute-engine-specific interfaces and +using Delta Kernel to fulfill them: + +```text +┌─────────────────────────────────────────────────────────┐ +│ Compute Engine │ +│ (Spark, Flink, DuckDB, Polars, ...) │ +│ │ +│ "I need a Table, a Scan, a Writer..." │ +└───────────────────────┬─────────────────────────────────┘ + │ calls your DataSource API impl + ▼ +┌─────────────────────────────────────────────────────────┐ +│ Your Delta Connector │ +│ │ +│ Implements the compute engine's DataSource interfaces │ +│ (e.g. Spark's Table/ScanBuilder/Scan/Batch/Writer) │ +│ │ +│ Uses Kernel to fulfill those interfaces: │ +│ Snapshot -> table metadata, schema, version │ +│ Scan -> which files to read, data skipping │ +│ Transaction -> write files, commit atomically │ +└───────────────────────┬─────────────────────────────────┘ + │ calls Kernel APIs + ▼ +┌─────────────────────────────────────────────────────────┐ +│ Delta Kernel │ +│ │ +│ Protocol logic, log replay, data skipping, │ +│ schema enforcement, transaction coordination │ +│ │ +│ Calls into Engine trait for I/O and compute │ +└───────────────────────┬─────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────┐ +│ Engine trait implementation │ +│ │ +│ DefaultEngine (Arrow + object_store) │ +│ ...or your custom Engine with native I/O & formats │ +└─────────────────────────────────────────────────────────┘ +``` + +## Example: how the Spark connector works + +To make this concrete, here's how the +[Delta Spark connector](https://github.com/delta-io/delta/tree/master/spark) maps Spark's +DataSourceV2 interfaces to Kernel APIs: + +| Spark DataSourceV2 interface | Connector class | Uses Kernel... | +|------------------------------|----------------|----------------| +| `Table` (entry point, schema, capabilities) | `SparkTable` | `Snapshot` for schema and metadata | +| `ScanBuilder` (filter pushdown, column pruning) | `SparkScanBuilder` | `ScanBuilder` with predicates and column selection | +| `Scan` (plan which files to read) | `SparkScan` | `Scan.getScanFiles()` to get the list of data files | +| `Batch` (partition files for parallel execution) | `SparkBatch` | Partitions scan files across Spark tasks | +| `PartitionReader` (read data from files) | `SparkPartitionReader` | Reads Parquet files assigned to this task | + +The pattern is always the same: the compute engine asks for something through its DataSource +API, and the connector translates that into the corresponding Kernel call. + +## The Snapshot is the center of everything + +All connector operations start from a `Snapshot`, an immutable view of the table at a +specific version: + +```rust,ignore +let snapshot = Snapshot::builder_for(table_url).build(&engine)?; +``` + +From a snapshot, you can: + +| Operation | Kernel API | Typical DataSource equivalent | +|-----------|-----------|-------------------------------| +| Get schema and metadata | `snapshot.schema()`, `snapshot.table_properties()` | Table schema discovery | +| Read data | `snapshot.scan_builder()` | Scan / PartitionReader | +| Write data | `snapshot.transaction(committer, &engine)` | Writer / Committer | +| Checkpoint | `snapshot.create_checkpoint_writer(&engine)` | Maintenance task | + +## What your connector does vs. what Kernel does + +The key principle: **Kernel handles the Delta protocol, your connector handles execution.** + +**Kernel handles:** +- Log replay (figuring out which files are active at a version) +- Data skipping (pruning files using statistics and predicates) +- Schema enforcement and column mapping +- Transaction conflict detection +- Protocol compliance (table features, reader/writer requirements) + +**Your connector handles:** +- Implementing the compute engine's DataSource API +- Controlling parallelism and distribution (how many threads/tasks, which worker reads which file) +- Data format conversion (Kernel's `EngineData` to your engine's native format) +- Query planning integration (pushing filters and projections down to Kernel) +- Resource management (memory budgets, connection pooling) + +## The Engine trait: pluggable I/O and compute + +Kernel never does I/O directly. When it needs to list files, read Parquet, parse JSON, or +evaluate expressions, it calls into the `Engine` trait. This is where you can plug in your +engine's native implementations for maximum performance. + +Kernel provides a `DefaultEngine` (Arrow + `object_store` + Tokio). Many connectors +start here and only replace specific handlers when they need better performance. See +[The Engine trait](../concepts/engine_trait.md) for details on the four required handlers +(`StorageHandler`, `JsonHandler`, `ParquetHandler`, `EvaluationHandler`) and the optional +`MetricsReporter`. + +You need a custom `Engine` when: + +- **Your engine has its own columnar data format** (not Arrow) and you want to avoid + conversion overhead +- **Your engine has its own I/O layer** (e.g. a distributed file system client, encrypted + storage, or custom caching) +- **You want engine-native expression evaluation** (e.g. vectorized execution, JIT + compilation) + +To use different cloud storage (S3, Azure, GCS), you do not need a custom engine. +`DefaultEngine` supports all `object_store` backends without modification. + +## Getting started + +Building a connector typically involves these steps: + +1. **Choose or implement an Engine.** Start with `DefaultEngine` unless you have a reason + not to. See [Implementing the Engine trait](./implementing_engine.md). + +2. **Implement your DataSource's read interfaces** using Kernel's Scan API: + - Create a `Snapshot` to discover the table schema and version + - Use `ScanBuilder` to push down filters and column projections + - Use `Scan` to get the list of files to read + - Distribute those files across your engine's execution model + - Read and transform data using the Engine + - See [Building a scan](../reading/building_a_scan.md) and + [Filter pushdown](../reading/filter_pushdown.md) + +3. **Implement your DataSource's write interfaces** using Kernel's Transaction API: + - Create a `Transaction` from a `Snapshot` + - Write Parquet files and register them with the transaction + - Commit atomically, handling conflicts and retries + - See [Creating a table](../writing/create_table.md) and + [Appending data](../writing/append.md) + +4. **Handle distribution** (if your engine is distributed): + - Serialize scan metadata and transaction state to ship to workers + - Collect results from workers and commit on the driver + - See [Advanced reads with scan_metadata()](../reading/scan_metadata.md) + +## What's next + +- [Implementing the Engine trait](./implementing_engine.md): when and how to customize the + Engine +- [Building a scan](../reading/building_a_scan.md): the Kernel read API +- [Appending data](../writing/append.md): the Kernel write API diff --git a/docs/user-guide/src/ffi/overview.md b/docs/user-guide/src/ffi/overview.md new file mode 100644 index 0000000000..8cb56b574d --- /dev/null +++ b/docs/user-guide/src/ffi/overview.md @@ -0,0 +1,458 @@ +# FFI (C/C++ integration) + +The `delta_kernel_ffi` crate exposes delta-kernel-rs to C and C++ through a stable FFI +boundary. It uses [cbindgen](https://github.com/mozilla/cbindgen) to generate header +files (`.h` and `.hpp`) at build time. This matters because it lets you build a +[connector](../connector/overview.md) in any language that can call C functions, not +only Rust. + +## Building the FFI crate + +The crate can be built as a shared library (`cdylib`) or static library (`staticlib`): + +```bash +# Shared library (e.g. libdelta_kernel_ffi.so / .dylib / .dll) +cargo build -p delta_kernel_ffi --release + +# Generated headers are written to target/ffi-headers/ +# - delta_kernel_ffi.h (C) +# - delta_kernel_ffi.hpp (C++) +``` + +### Feature flags + +| Feature | Default | Description | +|---------|---------|-------------| +| `default-engine-rustls` | yes | Includes the `DefaultEngine` with rustls TLS | +| `default-engine-native-tls` | no | Includes the `DefaultEngine` with native TLS (instead of rustls) | +| `arrow` | yes | Enables Arrow integration (selects `arrow-59` by default) | +| `arrow-59` | yes | Pin to Arrow 59 explicitly (enabled transitively by `arrow`) | +| `arrow-58` | no | Pin to Arrow 58 explicitly | +| `delta-kernel-unity-catalog` | no | Enables Unity Catalog integration for catalog-managed tables | +| `tracing` | no | Enables tracing/logging support via `tracing-subscriber` | + +> **Note:** You must enable exactly one of `default-engine-rustls` or +> `default-engine-native-tls`. The `default-engine-base` feature contains +> shared implementation details and is not meant to be enabled directly. + +## The handle system + +Objects that cross the FFI boundary are wrapped in **handles**. These are opaque pointers +that carry ownership semantics. There are two kinds: + +- **Mutable handles** (`Box`-like) represent exclusive ownership. Dropping the handle + drops the underlying object. These are neither `Copy` nor `Clone`. +- **Shared handles** (`Arc`-like) represent shared ownership. Dropping the handle only + drops the underlying object if it was the last reference. + +Every handle has a corresponding `free_*` function that you must call to release it. +For example, `free_engine`, `free_snapshot`, `free_scan`, `free_transaction`. + +Several FFI functions _consume_ their handle argument and return a new handle. After +calling such a function, you must not use the old handle. The function documentation +notes this with "CONSUMES the handle." + +## Core API surface + +The FFI mirrors the Rust API. A typical read flow looks like (no transaction +is needed for reads): + +```text +get_default_engine() -> Handle + | +get_snapshot_builder() -> Handle + | +snapshot_builder_build() -> Handle + | + scan() -> Handle + | +scan_metadata_iter_init() -> Handle + | + (read parquet, apply transforms, apply selection vectors) +``` + +A typical write flow: + +```text +get_default_engine() -> Handle + | + transaction() -> Handle + | + with_engine_info() -> Handle + | + add_files() + | + commit() -> ExternResult (committed version) +``` + +For more control over scans, you can use the scan builder API instead of the +convenience `scan()` function: + +```text +scan_builder() -> Handle + | +scan_builder_with_predicate() -> Handle + | +scan_builder_with_schema() -> Handle + | +scan_builder_with_stats() -> Handle + | +scan_builder_with_partition_values() + -> Handle + | +scan_builder_build() -> Handle +``` + +### Public FFI functions + +The tables below group the stable FFI functions by purpose. Unless noted, +each function is available with the default feature flags. For the full, +authoritative list and signatures, consult the generated +`delta_kernel_ffi.h` header. + +**Engine creation** + +| Function | Purpose | +|----------|---------| +| `get_default_engine` | Create an engine from a table path with default options | +| `get_engine_builder` / `set_builder_option` / `builder_build` | Create an engine with custom storage options | +| `set_builder_with_multithreaded_executor` | Configure the builder to use a multi-threaded tokio executor | +| `set_builder_with_io_concurrency` | Configure read-path I/O concurrency (buffer size and batch size) for the JSON and Parquet handlers | +| `free_engine` | Release the engine handle | + +**Snapshots** + +| Function | Purpose | +|----------|---------| +| `get_snapshot_builder` | Create a snapshot builder from a table path | +| `get_snapshot_builder_from` | Create a snapshot builder incrementally from an existing snapshot | +| `snapshot_builder_set_version` | Pin the snapshot to a specific table version | +| `snapshot_builder_set_log_tail` | Provide a log tail for catalog-managed tables | +| `snapshot_builder_set_max_catalog_version` | Bound the snapshot to the version the catalog has ratified | +| `snapshot_builder_build` | Consume the builder and produce the snapshot | +| `free_snapshot_builder` / `free_snapshot` | Release snapshot-related handles | + +**Snapshot inspection** + +Use these on a `Handle` to read table metadata, protocol, and +partition columns without building a scan. + +| Function | Purpose | +|----------|---------| +| `version` | Return the version number of a snapshot | +| `snapshot_timestamp` | Return the snapshot's commit timestamp (milliseconds since epoch) | +| `snapshot_table_root` | Return the table root URL as an engine-allocated string | +| `logical_schema` | Return the table's logical schema as a `Handle` | +| `snapshot_get_metadata` | Clone the table metadata into a `Handle` | +| `snapshot_get_protocol` | Clone the protocol into a `Handle` | +| `get_partition_column_count` / `get_partition_columns` | Count partition columns and iterate their names as a `StringSliceIterator` | +| `string_slice_next` / `free_string_slice_data` | Iterate and release a `StringSliceIterator` (e.g. returned by `get_partition_columns`) | +| `get_app_id_version` | Look up the last committed transaction version for an `app_id` (the read side of [idempotent writes](../writing/idempotent_writes.md)) | +| `free_metadata` / `free_protocol` / `free_schema` | Release the corresponding handles | + +**Schema and metadata visitors** + +Kernel exposes schemas, protocols, and metadata through visitor callbacks so +the engine can materialize them into its own types without Kernel allocating +engine-owned memory. + +| Function | Purpose | +|----------|---------| +| `visit_schema` | Walk a `SharedSchema` by invoking per-field callbacks on an `EngineSchemaVisitor` | +| `visit_protocol` | Invoke a `visit_versions` callback, then a `visit_feature` callback per reader/writer feature | +| `visit_metadata` | Invoke a single callback with `(id, name, description, format_provider, has_created_time, created_time_ms)` | +| `visit_metadata_configuration` | Iterate the `configuration` key/value map (takes a snapshot handle, not a metadata handle) | +| `visit_string_map` / `get_from_string_map` | Iterate or look up entries in an opaque `CStringMap` (used by both metadata and scan-metadata surfaces) | + +See [Visitor callbacks](#visitor-callbacks) below for the pattern. + +**Schema construction (projection pushdown)** + +The build-side counterpart to `visit_schema`: per-field callbacks that let the +engine construct a Kernel `StructType` from its own type system (for example, +to pass to `scan_builder_with_schema`). + +| Function | Purpose | +|----------|---------| +| `visit_field_byte` / `visit_field_short` / `visit_field_integer` / `visit_field_long` / `visit_field_float` / `visit_field_double` / `visit_field_boolean` | Build a numeric or boolean primitive `StructField` | +| `visit_field_string` / `visit_field_binary` / `visit_field_date` / `visit_field_timestamp` / `visit_field_timestamp_ntz` | Build a string, binary, or date/time primitive `StructField` | +| `visit_field_decimal` | Build a decimal `StructField` with explicit precision and scale | +| `visit_field_struct` / `visit_field_array` / `visit_field_map` / `visit_field_variant` | Build a complex `StructField` (struct, array, map, or variant) from previously created field or struct IDs | + +**Reading (scans)** + +| Function | Purpose | +|----------|---------| +| `scan` | Create a scan with optional predicate and projection (convenience function) | +| `scan_builder` / `scan_builder_with_predicate` / `scan_builder_with_schema` / `scan_builder_with_stats` / `scan_builder_with_partition_values` / `scan_builder_build` | Build a scan incrementally and configure predicate, projection, stats, and partition-value output | +| `scan_logical_schema` / `scan_physical_schema` / `scan_table_root` | Inspect the scan's logical/physical read schemas and table root | +| `scan_metadata_iter_init` / `scan_metadata_next` | Iterate over scan metadata (per-file lists, deletion vectors, transforms) | +| `scan_metadata_next_arrow` / `free_scan_metadata_arrow_result` | Pull the next scan-metadata batch as an Arrow `RecordBatch` and release it (requires `default-engine-base`) | +| `visit_scan_metadata` | Invoke a callback for each scan file in a `SharedScanMetadata` batch | +| `selection_vector_from_scan_metadata` | Materialize the per-row selection bitmap from a scan-metadata batch | +| `selection_vector_from_dv` / `row_indexes_from_dv` | Materialize a selection bitmap or row-index array from a `DvInfo` | +| `get_transform_for_row` | Look up the per-file transform expression for a given row in a scan-metadata batch | +| `free_scan` / `free_scan_builder` / `free_scan_metadata` / `free_scan_metadata_iter` / `free_bool_slice` / `free_row_indexes` | Release scan-related handles and allocations | + +**Engine data and Arrow interop** + +Rows marked `(requires default-engine-base)` are only compiled when that +feature is enabled; the rest are always available. + +| Function | Purpose | +|----------|---------| +| `engine_data_length` | Return the row count of an `ExclusiveEngineData` batch | +| `get_engine_data` | Import Arrow C Data Interface array + schema into an `ExclusiveEngineData` (requires `default-engine-base`) | +| `get_raw_arrow_data` | Export an `ExclusiveEngineData` batch as Arrow C Data Interface structs (requires `default-engine-base`) | +| `read_result_next` / `free_read_result_iter` | Iterate a scan's parquet read iterator and release it | +| `free_engine_data` | Drop a single `ExclusiveEngineData` batch | +| `read_parquet_file` | Directly read a single parquet file via the engine's parquet handler | + +> **Warning:** `get_raw_engine_data` is always exported (regardless of feature +> flags) but unimplemented. It calls `todo!()` and will panic. Do not use it. + +**Writing (transactions)** + +| Function | Purpose | +|----------|---------| +| `transaction` | Start a write transaction on the latest snapshot | +| `transaction_with_committer` | Start a transaction with a custom committer | +| `with_engine_info` | Record a free-form engine identifier on the transaction (consumes and returns a new handle) | +| `with_transaction_id` | Set an `(app_id, version)` pair for idempotent writes (consumes and returns a new handle; see [Idempotent Writes](../writing/idempotent_writes.md)) | +| `with_domain_metadata` / `with_domain_metadata_removed` | Attach or remove a domain-metadata entry (each consumes and returns a new handle) | +| `add_files` | Append file-level write metadata to the transaction | +| `set_data_change` | Toggle the transaction's data-change flag (does not consume the handle) | +| `remove_files` | Register Remove actions for the files selected by a scan-metadata batch | +| `commit` | Commit the transaction and return the new version number | +| `free_transaction` | Release the transaction handle without committing | + +**Write context and file writing** + +Use a `WriteContext` to learn where to write parquet files and what schema to +write. For unpartitioned writes, one context serves the whole transaction. +Partitioned writes (which would use one context per partition) are tracked in +[#2355](https://github.com/delta-io/delta-kernel-rs/issues/2355). + +Engines must append their own `.parquet` filename (and any subdirectory +layout) onto the returned table root. For partitioned tables, use +[`get_write_dir`](https://docs.rs/delta_kernel_ffi/latest/delta_kernel_ffi/fn.get_write_dir.html) +for the kernel-recommended subdirectory (Hive-style partition paths when column mapping is off, or +a random 2-char prefix when column mapping is on). `get_write_path` returns the table root for +unpartitioned writes. + +| Function | Purpose | +|----------|---------| +| `get_unpartitioned_write_context` | Get a `SharedWriteContext` covering all rows in the transaction | +| `get_partitioned_write_context` | Get a `SharedWriteContext` for one partition (requires a `PartitionValueMap`) | +| `get_write_dir` | Return the recommended write subdirectory for a partitioned `SharedWriteContext` | +| `get_write_path` | Return the table root URL from a `SharedWriteContext` (engines append their own subdirectory and filename) | +| `get_write_schema` | Return the logical (user-facing) write schema from a `SharedWriteContext` | +| `free_write_context` | Release the write-context handle | + +**Domain metadata** + +| Function | Purpose | +|----------|---------| +| `get_domain_metadata` | Look up the configuration string for a specific domain on a snapshot | +| `visit_domain_metadata` | Iterate all domain metadata entries on a snapshot | + +**Table creation** + +| Function | Purpose | +|----------|---------| +| `get_create_table_builder` | Create a builder for a new Delta table with a schema | +| `create_table_builder_with_table_property` | Add a table property to the builder | +| `create_table_builder_build` | Consume the builder and produce a create-table transaction using the default (filesystem) committer | +| `create_table_builder_build_with_committer` | Consume the builder and produce a create-table transaction with a custom committer | +| `create_table_with_engine_info` | Attach a free-form engine identifier to a create-table transaction (consumes and returns a new handle) | +| `create_table_set_data_change` | Toggle the data-change flag on a create-table transaction (does not consume the handle) | +| `create_table_get_unpartitioned_write_context` | Get a `WriteContext` to stage initial data files during table creation | +| `create_table_add_files` | Register file metadata for initial data being written alongside the CREATE TABLE commit | +| `create_table_commit` | Commit the create-table transaction | +| `free_create_table_builder` | Release a create-table builder handle (before it is consumed by `create_table_builder_build*`) | +| `create_table_free_transaction` | Release a create-table transaction handle (after build, before commit) | + +**Change data feed (table changes)** + +Incremental log reads for change data feed. Mirrors the Rust +[`TableChanges`](../reading/change_data_feed.md) API. The entire `table_changes` +module is gated behind the `default-engine-base` feature. + +| Function | Purpose | +|----------|---------| +| `table_changes_from_version` | Open a `TableChanges` from `start_version` to the latest | +| `table_changes_between_versions` | Open a `TableChanges` between `start_version` and `end_version` (inclusive) | +| `table_changes_start_version` / `table_changes_end_version` | Read the requested start/end versions | +| `table_changes_schema` / `table_changes_table_root` | Inspect the CDF schema and table root | +| `table_changes_scan` | Apply an optional predicate and produce a `SharedTableChangesScan` | +| `table_changes_scan_logical_schema` / `table_changes_scan_physical_schema` / `table_changes_scan_table_root` | Inspect the resulting scan | +| `table_changes_scan_execute` | Produce an iterator of CDF data | +| `scan_table_changes_next` | Pull the next `*mut ArrowFFIData` batch from the CDF iterator; the engine must release each non-null batch via `free_arrow_ffi_data` | +| `free_table_changes` / `free_table_changes_scan` / `free_scan_table_changes_iter` | Release CDF-related handles | + +> [!WARNING] +> `scan_table_changes_next` returns `*mut ArrowFFIData` (a heap-allocated +> Arrow C Data Interface batch). C callers must release each non-null result +> with `free_arrow_ffi_data` exactly once. This is an ABI-breaking change +> from earlier releases that used a different return shape, so connectors +> upgrading across that boundary need to update both the type and the +> cleanup path. + +**Checkpointing** + +| Function | Purpose | +|----------|---------| +| `checkpoint_snapshot` | Write a checkpoint for the given snapshot | + +**Unity Catalog integration** + +Requires the `delta-kernel-unity-catalog` feature. Lets the engine provide a +catalog-aware committer without implementing the committer trait from scratch. + +| Function | Purpose | +|----------|---------| +| `get_uc_commit_client` | Wrap an engine-provided `CCommit` callback in a `SharedFfiUCCommitClient` | +| `get_uc_committer` | Produce a `MutableCommitter` bound to a specific `table_id`, ready to pass to `transaction_with_committer` | +| `free_uc_commit_client` / `free_uc_committer` | Release the corresponding handles | + +**Expressions and predicates** + +Kernel's expression system is exposed through two parallel surfaces: + +- **Build** (engine AST -> Kernel expression): the engine calls + `visit_engine_expression` / `visit_engine_predicate`, providing an + engine-side iterator. From inside that callback, the engine uses the + `visit_expression_*` / `visit_predicate_*` builder functions to append + Kernel-side nodes (columns, literals, operators) into an internal + `KernelExpressionVisitorState`. The final handle is a `SharedExpression` or + `SharedPredicate`. +- **Walk** (Kernel expression -> engine type): the engine calls + `visit_expression` / `visit_predicate`, supplying an `EngineExpressionVisitor` + struct whose function pointers Kernel invokes as it traverses the tree. + +Most engines need only one of the two surfaces. + +| Function | Purpose | +|----------|---------| +| `visit_engine_expression` / `visit_engine_predicate` | Build a Kernel `SharedExpression` / `SharedPredicate` from an engine-side AST (Build surface) | +| `visit_expression` / `visit_predicate` | Walk a Kernel expression or predicate with an `EngineExpressionVisitor` (Walk surface) | +| `visit_expression_ref` / `visit_predicate_ref` | Walk a pre-interned expression or predicate reference (Walk surface) | +| `visit_expression_column` / `visit_expression_struct` / `visit_expression_plus` / `visit_expression_literal_*` / ... | Builder functions the engine calls from inside `visit_engine_expression` to construct Kernel expression nodes; see `delta_kernel_ffi.h` for the full list | +| `visit_predicate_eq` / `visit_predicate_and` / `visit_predicate_or` / ... | Builder functions the engine calls from inside `visit_engine_predicate` to construct Kernel predicate nodes | +| `visit_expression_unknown` / `visit_predicate_unknown` | Builder helpers that bridge an opaque engine operator through Kernel unchanged | +| `visit_kernel_opaque_expression_op_name` / `visit_kernel_opaque_predicate_op_name` | Inspect the name of an opaque op carried through by the above | +| `new_expression_evaluator` / `evaluate_expression` / `free_expression_evaluator` | Compile and invoke an expression against an `EngineData` batch | +| `expressions_are_equal` / `predicates_are_equal` | Compare two expressions or predicates for structural equality | +| `free_kernel_expression` / `free_kernel_predicate` / `free_kernel_opaque_expression_op` / `free_kernel_opaque_predicate_op` | Release the corresponding handles | + +**Tracing** + +Enable Kernel's internal `tracing` instrumentation. Requires the `tracing` +feature flag. Call at most one of these during a process lifetime; later +calls return `false`. + +| Function | Purpose | +|----------|---------| +| `enable_log_line_tracing` | Forward each log line to a `TracingLogLineFn` callback | +| `enable_formatted_log_line_tracing` | Forward fully formatted log lines to a `TracingLogLineFn` callback | +| `enable_event_tracing` | Forward structured `TracingEvent` records instead of log lines | + +**Common utilities** + +| Function | Purpose | +|----------|---------| +| `allocate_kernel_string` | Create a Kernel-owned string from a `KernelStringSlice` | +| `allocate_kernel_bytes` | Create a Kernel-owned byte buffer from a `KernelBytesSlice` | + +## Visitor callbacks + +Several FFI entry points take a visitor struct (for example, +`EngineSchemaVisitor` for `visit_schema`, `EngineExpressionVisitor` for +`visit_expression`) or individual callbacks (as in `visit_metadata`). The +pattern is the same in every case: + +1. You allocate a context (a `NullableCvoid` you own) that the callbacks can + write into. Kernel does not interpret this value. +2. You fill in one callback per node kind. Kernel invokes the callbacks + passing the context plus node-specific arguments (names, types, literal + values, child handles). For tree-structured inputs (schemas, expressions, + predicates), callbacks fire in depth-first order; for flat inputs (e.g. + `visit_metadata`), the callback fires once. +3. When `visit_*` returns, the context holds your engine-side representation. + +Callbacks run synchronously on the same thread that called `visit_*`. Strings +passed to callbacks (`KernelStringSlice`) are borrowed for the duration of the +call; copy them if you need to retain them beyond the callback. + +## Error handling + +Kernel functions that can fail return an `ExternResult`, which is a tagged union: + +```c +// C representation (simplified) +typedef enum { Ok, Err } ExternResultTag; +typedef struct { + ExternResultTag tag; + union { + T ok; + EngineError* err; + }; +} ExternResult; +``` + +You provide an `allocate_error` callback when creating the engine. Kernel calls this +callback to allocate error objects in your memory space whenever an operation fails. +Because the engine allocates these errors, the engine is also responsible for freeing +them. Kernel returns the error pointer immediately and does not retain it. + +The `EngineError` struct contains a `KernelError` enum that classifies the error type +(e.g., `GenericError`, `FileNotFoundError`, `InvalidUrlError`). The error message +string passed to `allocate_error` is only valid for the duration of the callback, so +you must copy it if you need to keep it. + +## C examples + +The repository ships four runnable C examples under +[`ffi/examples/`](https://github.com/delta-io/delta-kernel-rs/tree/main/ffi/examples). +Each is a complete program that links against `delta_kernel_ffi` and +exercises a different slice of the API. + +| Example | Demonstrates | +|---------|--------------| +| [`read-table`](https://github.com/delta-io/delta-kernel-rs/tree/main/ffi/examples/read-table) | The full read path: schema visiting, scan-metadata iteration, and Arrow data handling. Pass `-a` to switch from the callback-based scan-metadata path to the Arrow batch-mode path (`scan_metadata_next_arrow`). | +| [`read-table-changes`](https://github.com/delta-io/delta-kernel-rs/tree/main/ffi/examples/read-table-changes) | Reading a [change data feed](../reading/change_data_feed.md) using `table_changes_*` and consuming `ArrowFFIData` batches from `scan_table_changes_next`. | +| [`create-table`](https://github.com/delta-io/delta-kernel-rs/tree/main/ffi/examples/create-table) | Creating a new Delta table via the `get_create_table_builder` / `create_table_builder_build` / `create_table_commit` flow. | +| [`write-table`](https://github.com/delta-io/delta-kernel-rs/tree/main/ffi/examples/write-table) | Appending data to an existing table via the `transaction` / `add_files` / `commit` flow. | + +The high-level flow in the `read-table` example: + +```c +// 1. Create an engine +ExternResultHandleSharedExternEngine engine_res = + get_default_engine(table_path, allocate_error); + +// 2. Build a snapshot +ExternResultHandleMutableFfiSnapshotBuilder builder_res = + get_snapshot_builder(table_path, engine); +ExternResultHandleSharedSnapshot snap_res = + snapshot_builder_build(builder); + +// 3. Create a scan +ExternResultHandleSharedScan scan_res = + scan(snap, engine, NULL, NULL); + +// 4. Iterate over scan metadata and read data +// ... (see the full example for details) + +// 5. Clean up +free_scan(the_scan); +free_snapshot(snap); +free_engine(engine); +``` + +## What's next + +- [Building a Connector](../connector/overview.md): implementing a custom engine in + Rust +- [The Engine Trait](../concepts/engine_trait.md): understanding the engine interface + that the FFI wraps diff --git a/docs/user-guide/src/getting_started/installation.md b/docs/user-guide/src/getting_started/installation.md new file mode 100644 index 0000000000..50a15fdfcb --- /dev/null +++ b/docs/user-guide/src/getting_started/installation.md @@ -0,0 +1,99 @@ +# Installation + +The Delta Kernel ships as two crates on crates.io: + +- [`delta_kernel`](https://crates.io/crates/delta_kernel): the core library. No I/O, no Arrow. +- [`delta_kernel_default_engine`](https://crates.io/crates/delta_kernel_default_engine): the default + Arrow + Tokio implementation of the `Engine` trait. + +Cargo feature flags keep both crates dependency-light. + +## Requirements + +- **Rust edition**: 2021 +- **Minimum Rust version**: 1.88 + +## Adding the dependency + +For the common case (use the default engine), add both crates: + +```toml +[dependencies] +delta_kernel = "0.23" +delta_kernel_default_engine = { version = "0.23", features = ["rustls"] } +``` + +That gives you Kernel plus a default engine that handles I/O and expression evaluation for you, +backed by Arrow with `rustls` for TLS. + +If you're building a custom engine and don't need the default, depend on just `delta_kernel` +and enable whatever Arrow interop flags you want: + +```toml +[dependencies] +delta_kernel = { version = "0.23", features = ["arrow-conversion", "arrow-expression"] } +``` + +## Feature flags + +You only pay for what you enable. + +### `delta_kernel_default_engine` features + +| Feature | Description | +|---------|-------------| +| `rustls` | Default engine with `rustls` for TLS. **Recommended for most users.** | +| `native-tls` | Default engine using your platform's native TLS (OpenSSL on Linux, Schannel on Windows, Secure Transport on macOS). Use this if `rustls` doesn't work in your environment. | +| `arrow` | Use the latest Arrow version Kernel supports. Currently maps to Arrow 59. | + +You need exactly one of `rustls` or `native-tls`. See +[Building a Connector](../connector/overview.md) for when a custom engine makes sense instead. + +### Arrow version pinning + +If you need a specific Arrow version (e.g. to match your existing Arrow dependency), pin it +explicitly on both crates: + +| Feature | Arrow version | +|---------|---------------| +| `arrow-59` | Arrow 59 (current default) | +| `arrow-58` | Arrow 58 | + +For more details on managing Arrow version compatibility, see +[Feature Flags](../concepts/feature_flags.md). + +### `delta_kernel` features + +| Feature | Description | +|---------|-------------| +| `arrow-conversion` | Convert between kernel types and Arrow types | +| `arrow-expression` | Evaluate kernel expressions over Arrow data | +| `internal-api` | Expose additional APIs that aren't yet stabilized. Some examples in this guide need this. | +| `schema-diff` | Experimental schema diffing | + +The `arrow-conversion` and `arrow-expression` flags are pulled in automatically by +`delta_kernel_default_engine`, so you typically only set them when building a custom Arrow-based +engine yourself. + +## Example `Cargo.toml` + +A typical project using Kernel: + +```toml +[package] +name = "my-delta-reader" +version = "0.1.0" +edition = "2021" + +[dependencies] +delta_kernel = "0.23" +delta_kernel_default_engine = { version = "0.23", features = ["rustls"] } + +# Kernel re-exports arrow, but you can also depend on it directly: +# arrow = "59" +``` + +## What's next + +With the dependencies added, head to [Quick Start: Reading a Table](./quick_start_read.md) to +read your first Delta table. diff --git a/docs/user-guide/src/getting_started/quick_start_read.md b/docs/user-guide/src/getting_started/quick_start_read.md new file mode 100644 index 0000000000..a2cd099c53 --- /dev/null +++ b/docs/user-guide/src/getting_started/quick_start_read.md @@ -0,0 +1,166 @@ +# Quick Start: Reading a table + +In this tutorial, you will read a Delta table and get the data as +[Arrow](https://arrow.apache.org) `RecordBatch`es using the default engine. + +## Create a new project + +```sh +cargo new delta_read_example +cd delta_read_example +``` + +Add the Kernel and default-engine dependencies (see [Installation](./installation.md) for details): + +```sh +cargo add delta_kernel -F internal-api +cargo add delta_kernel_default_engine -F rustls +``` + +Your `Cargo.toml` should look like: + +```toml +[dependencies] +delta_kernel = { version = "0.23", features = ["internal-api"] } +delta_kernel_default_engine = { version = "0.23", features = ["rustls"] } +``` + +> [!NOTE] +> The `internal-api` feature exposes `try_parse_uri`, a convenience function used in this tutorial +> and throughout the guide. This feature flag may be removed in a future release once the API +> stabilizes. + +## Write the code + +Replace `src/main.rs` with the following. We'll walk through each piece below. + +Filename: src/main.rs + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +use std::sync::Arc; + +use delta_kernel::arrow::util::pretty::print_batches; +use delta_kernel::engine::arrow_data::EngineDataArrowExt as _; +use delta_kernel_default_engine::storage::store_from_url; +use delta_kernel_default_engine::DefaultEngine; +use delta_kernel::{DeltaResult, Snapshot}; + +fn main() -> DeltaResult<()> { + // 1. Parse the table location + let table_path = std::env::args().nth(1).expect("usage: delta_read_example "); + let url = delta_kernel::try_parse_uri(&table_path)?; + + // 2. Build an object store and engine + let store = store_from_url(&url)?; + let engine = DefaultEngine::builder(store).build(); + + // 3. Get a snapshot of the table at the latest version + let snapshot = Snapshot::builder_for(url).build(&engine)?; + println!("Table version: {}", snapshot.version()); + println!("Schema:\n{}", snapshot.schema()); + + // 4. Build and execute a scan + let scan = snapshot.scan_builder().build()?; + let mut batches = vec![]; + for data in scan.execute(Arc::new(engine))? { + let record_batch: delta_kernel::arrow::record_batch::RecordBatch = + data?.try_into_record_batch()?; + batches.push(record_batch); + } + + // 5. Print the results + print_batches(&batches)?; + Ok(()) +} +``` + +## Step by step + +### 1. Parse the table location + +```rust,ignore +let url = delta_kernel::try_parse_uri(&table_path)?; +``` + +`try_parse_uri` converts a path string (local path or URI like `s3://bucket/path`) into a `Url`. + +### 2. Build an object store and engine + +```rust,ignore +let store = store_from_url(&url)?; +let engine = DefaultEngine::builder(store).build(); +``` + +`store_from_url` creates an object store from the URL. For cloud storage with custom credentials, +use `store_from_url_opts` instead. See [Configuring Storage](../storage/configuring_storage.md) +for S3, Azure, and GCS options. + +`DefaultEngine::builder(store).build()` constructs the default engine, which handles all I/O +(Parquet, JSON, file listing) and expression evaluation using Arrow. + +### 3. Get a snapshot + +```rust,ignore +let snapshot = Snapshot::builder_for(url).build(&engine)?; +``` + +A `Snapshot` is an immutable view of the table at a specific version. Without calling +`.at_version(v)` on the builder, this gives you the latest version. + +The snapshot gives you access to the table's schema and properties, and serves as the entry point +for scanning and writing. + +### 4. Build and execute a scan + +```rust,ignore +let scan = snapshot.scan_builder().build()?; +for data in scan.execute(Arc::new(engine))? { + let record_batch = data?.try_into_record_batch()?; + batches.push(record_batch); +} +``` + +`scan_builder()` returns a `ScanBuilder` which you can configure with column selection +(`.with_schema()`) or filter predicates (`.with_predicate()`). Here we use the defaults: all +columns, no filter. + +`execute()` returns an iterator of `EngineData` results. Since we're using the default engine, +each item is backed by an Arrow `RecordBatch`. The `try_into_record_batch()` method (from the +`EngineDataArrowExt` extension trait) unwraps it. + +## Run it + +If you have the delta-kernel-rs repo checked out locally, you can test with one of its test tables: + +```sh +cargo run -- /path/to/delta-kernel-rs/kernel/tests/data/basic_partitioned/ +``` + +Expected output: + +```text +Table version: 1 +Schema: +struct: +├─letter: string (is nullable: true, metadata: {}) +├─number: long (is nullable: true, metadata: {}) +└─a_float: double (is nullable: true, metadata: {}) + ++--------+--------+---------+ +| letter | number | a_float | ++--------+--------+---------+ +| | 6 | 6.6 | +| a | 4 | 4.4 | +| e | 5 | 5.5 | +| a | 1 | 1.1 | +| b | 2 | 2.2 | +| c | 3 | 3.3 | ++--------+--------+---------+ +``` + +## What's next + +- [Quick Start: Writing a Table](./quick_start_write.md) covers creating a table and writing data. +- [Building a Scan](../reading/building_a_scan.md) explores column selection, filter pushdown, and more. diff --git a/docs/user-guide/src/getting_started/quick_start_write.md b/docs/user-guide/src/getting_started/quick_start_write.md new file mode 100644 index 0000000000..454fab167f --- /dev/null +++ b/docs/user-guide/src/getting_started/quick_start_write.md @@ -0,0 +1,225 @@ +# Quick Start: Writing a Table + +In this tutorial, you will create a new Delta table, write data to it, and read it back. +It builds on the concepts from [Quick Start: Reading a Table](./quick_start_read.md). + +## Setup + +Create a new project and add dependencies: + +```sh +cargo new delta_write_example +cd delta_write_example +``` + +Writing data requires `tokio` because the default engine's Parquet writer is async: + +```sh +cargo add delta_kernel -F internal-api +cargo add delta_kernel_default_engine -F rustls +cargo add tokio -F rt-multi-thread -F macros +``` + +## Write the code + +Replace `src/main.rs` with the following: + +Filename: src/main.rs + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# extern crate tokio; +use std::sync::Arc; + +use delta_kernel::arrow::array::{Int32Array, RecordBatch, StringArray}; +use delta_kernel::arrow::util::pretty::print_batches; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::engine::arrow_conversion::TryIntoArrow; +use delta_kernel::engine::arrow_data::{ArrowEngineData, EngineDataArrowExt as _}; +use delta_kernel_default_engine::storage::store_from_url; +use delta_kernel_default_engine::DefaultEngine; +use delta_kernel::schema::{DataType, StructField, StructType}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::transaction::CommitResult; +use delta_kernel::{DeltaResult, Snapshot}; + +#[tokio::main] +async fn main() -> DeltaResult<()> { + let table_path = std::env::args() + .nth(1) + .expect("usage: delta_write_example "); + let url = delta_kernel::try_parse_uri(&table_path)?; + + // Build the engine + let engine = DefaultEngine::builder(store_from_url(&url)?).build(); + + // 1. Create the table + let schema = Arc::new(StructType::try_new(vec![ + StructField::not_null("id", DataType::INTEGER), + StructField::nullable("name", DataType::STRING), + ])?); + + create_table(url.as_str(), schema.clone(), "quick-start/1.0") + .build(&engine, Box::new(FileSystemCommitter::new()))? + .commit(&engine)?; + println!("Created table at {url}"); + + // 2. Write data + let snapshot = Snapshot::builder_for(url.clone()).build(&engine)?; + + let mut txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), &engine)? + .with_operation("INSERT".to_string()) + .with_engine_info("quick-start/1.0") + .with_data_change(true); + + // Build an Arrow RecordBatch + let arrow_schema: delta_kernel::arrow::datatypes::Schema = + schema.as_ref().try_into_arrow()?; + let batch = RecordBatch::try_new( + Arc::new(arrow_schema), + vec![ + Arc::new(Int32Array::from(vec![1, 2, 3])), + Arc::new(StringArray::from(vec!["Alice", "Bob", "Charlie"])), + ], + )?; + + // Write Parquet and add file metadata to the transaction + let write_context = Arc::new(txn.unpartitioned_write_context()?); + let data = ArrowEngineData::new(batch); + let file_metadata = engine + .write_parquet(&data, write_context.as_ref()) + .await?; + txn.add_files(file_metadata); + + // Commit + match txn.commit(&engine)? { + CommitResult::CommittedTransaction(committed) => { + println!("Committed version {}", committed.commit_version()); + } + CommitResult::ConflictedTransaction(_) => { + panic!("unexpected conflict on a brand new table"); + } + CommitResult::RetryableTransaction(retry) => { + panic!("commit failed with retryable error: {}", retry.error); + } + } + + // 3. Read it back + let snapshot = Snapshot::builder_for(url).build(&engine)?; + let scan = snapshot.scan_builder().build()?; + let batches: Vec = scan + .execute(Arc::new(engine))? + .map(|data| -> DeltaResult { + Ok(data?.try_into_record_batch()?) + }) + .collect::>>()?; + print_batches(&batches)?; + + Ok(()) +} +``` + +## Step by step + +### 1. Create the table + +```rust,ignore +let schema = Arc::new(StructType::try_new(vec![ + StructField::not_null("id", DataType::INTEGER), + StructField::nullable("name", DataType::STRING), +])?); + +create_table(url.as_str(), schema.clone(), "quick-start/1.0") + .build(&engine, Box::new(FileSystemCommitter::new()))? + .commit(&engine)?; +``` + +`create_table` returns a builder. You provide: +- The table path +- A schema (using kernel's `StructType`) +- An engine info string (identifies your application) + +`.build()` takes the engine and a `Committer`. For local filesystem tables, use +`FileSystemCommitter`. For catalog-managed tables, you provide your own committer. +[Catalog-Managed Tables](../catalog_managed/overview.md) covers that topic. + +`.commit()` writes version 0 of the table (the initial Protocol and Metadata actions). + +### 2. Write data + +The write flow has four parts: + +**Start a transaction:** +```rust,ignore +let mut txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), &engine)? + .with_operation("INSERT".to_string()) + .with_data_change(true); +``` + +**Build your data as an Arrow RecordBatch and wrap it:** +```rust,ignore +let arrow_schema: delta_kernel::arrow::datatypes::Schema = + schema.as_ref().try_into_arrow()?; +let batch = RecordBatch::try_new( + Arc::new(arrow_schema), + vec![ + Arc::new(Int32Array::from(vec![1, 2, 3])), + Arc::new(StringArray::from(vec!["Alice", "Bob", "Charlie"])), + ], +)?; +let data = ArrowEngineData::new(batch); +``` + +**Write the Parquet file and collect file metadata:** +```rust,ignore +let write_context = Arc::new(txn.unpartitioned_write_context()?); +let file_metadata = engine + .write_parquet(&data, write_context.as_ref()) + .await?; +txn.add_files(file_metadata); +``` + +`unpartitioned_write_context()` creates a `WriteContext` with the target directory, schema, and stats configuration. +`write_parquet` writes a Parquet file and returns metadata (path, size, stats) that the +transaction needs. `add_files` registers that metadata with the transaction. + +**Commit:** +```rust,ignore +match txn.commit(&engine)? { + CommitResult::CommittedTransaction(committed) => { /* success */ } + CommitResult::ConflictedTransaction(_) => { /* another writer won */ } + CommitResult::RetryableTransaction(retry) => { /* transient error, retry */ } +} +``` + +`commit()` returns a `CommitResult` with three variants. For blind appends to a table with no +concurrent writers, you'll always get `CommittedTransaction`. + +## Run it + +```sh +mkdir -p /tmp/my_delta_table +cargo run -- /tmp/my_delta_table +``` + +Expected output: + +```text +Created table at file:///tmp/my_delta_table +Committed version 1 ++----+---------+ +| id | name | ++----+---------+ +| 1 | Alice | +| 2 | Bob | +| 3 | Charlie | ++----+---------+ +``` + +## What's next + +- [Creating a Table](../writing/create_table.md) covers table properties, partition columns, and more. +- [Appending Data](../writing/append.md) covers writing to existing tables and retry logic. diff --git a/docs/user-guide/src/images/ferris-delta-small.png b/docs/user-guide/src/images/ferris-delta-small.png new file mode 100644 index 0000000000..cbff70b046 Binary files /dev/null and b/docs/user-guide/src/images/ferris-delta-small.png differ diff --git a/docs/user-guide/src/images/flow.svg b/docs/user-guide/src/images/flow.svg new file mode 100644 index 0000000000..02bf65a65b --- /dev/null +++ b/docs/user-guide/src/images/flow.svg @@ -0,0 +1,107 @@ + + + + + + + + + + + + +Connectors(Query Engines, Analytics Tools, etc.)Rust Bindings(Native Rust API)FFI Bindings(C/C++ API)Delta KernelDelta Table + + + \ No newline at end of file diff --git a/docs/user-guide/src/introduction.md b/docs/user-guide/src/introduction.md new file mode 100644 index 0000000000..5c1626e996 --- /dev/null +++ b/docs/user-guide/src/introduction.md @@ -0,0 +1,131 @@ +# Delta Kernel Rust User Guide + + + The Delta Kernel Logo + + +Delta Kernel is a Rust library for building Delta Lake connectors. It handles the +Delta protocol so you don't have to. Connectors read and write Delta tables through +Kernel's API without needing to understand the protocol internals. When the protocol +evolves, updating your Kernel dependency is all that's needed to pick up new features. + +Kernel is query-engine agnostic. It provides a native Rust API and a C/C++ FFI layer, +making it usable from virtually any language. + +> [!NOTE] +> This guide is a work in progress. + +## Architecture at a glance + +```text + ┌──────────────────────────────────────────┐ + │ Connectors │ + │ (Query Engines, Analytics Tools, etc.) │ + └───────────┬──────────────────────┬───────┘ + │ │ + ┌───────────▼─────────┐ ┌────────▼───────┐ + │ Rust Bindings │ │ FFI Bindings │ + │ (Native Rust API) │ │ (C/C++ API) │ + └────────────────┬────┘ └─┬──────────────┘ + │ │ + ┌───▼──────────▼─┐ + │ Delta Kernel │ + │ (core logic) │ + └───────┬────────┘ + │ calls into + ┌───────▼────────┐ + │ Engine trait │ + │ (abstraction) │ + └───────┬────────┘ + │ implemented by + ┌───────▼────────┐ + │ DefaultEngine │ + │ (or custom) │ + └───────┬────────┘ + │ + ┌───────▼────────┐ + │ Delta Table │ + │ (storage) │ + └────────────────┘ +``` + +The **Engine trait** is the boundary between Kernel and your connector. Kernel defines +_what_ needs to happen (read JSON, read Parquet, evaluate expressions); the engine +defines _how_. A batteries-included `DefaultEngine` is provided for common use cases. +See [Architecture Overview](./concepts/architecture.md) for details. + +## Key APIs + +**Snapshot** is a point-in-time view of a Delta table. Every operation starts here: +reading the schema, scanning data, or starting a Transaction. + +**Scan** reads data from a table. It supports predicate pushdown for file skipping and +column projection. See [Building a Scan](./reading/building_a_scan.md). + +**Transaction** writes data to a table. It supports creating tables, blind appends, and +committing changes atomically. See [Creating a Table](./writing/create_table.md) and +[Appending Data](./writing/append.md). + +**CheckpointWriter** compacts the transaction log into a checkpoint for faster reads. +See [Checkpointing](./maintenance/checkpointing.md). + +### Data types and schema + +Kernel defines its own protocol-compliant type system, independent of any engine's type +system. This includes primitive types (integers, strings, timestamps, decimals, etc.) and +complex types (structs, arrays, maps). The Kernel schema is the source of truth for a +table's structure. Your engine converts to and from it as needed. + +## FFI layer + +The `delta_kernel_ffi` crate exposes the full Kernel API to C and C++ via a stable FFI +boundary. Headers (`.h` and `.hpp`) are generated automatically at build time using +cbindgen. Rust objects cross the boundary as opaque **handles** with clear ownership +semantics, and every fallible function returns a structured error type. + +This means you can build a Delta connector in C, C++, or any language with a C FFI +without writing any Rust. See the [FFI overview](./ffi/overview.md) for details. + +## Design principles + +1. **Protocol abstraction.** Kernel encapsulates the Delta protocol. Connectors pick up + new protocol features by updating their Kernel dependency. +2. **Engine-agnostic.** Kernel defines _what_ to do; engines define _how_. The `Engine` + trait is the only integration point. +3. **Feature flag modularity.** Core functionality works without optional dependencies. + Pay only for what you use via Cargo feature flags. +4. **Clear I/O boundaries.** APIs clearly indicate when I/O operations occur, giving + connectors control over scheduling and parallelism. + +## Crate structure + +| Crate | Purpose | +|-------|---------| +| `delta_kernel` | Core library: protocol logic, table operations, trait definitions, default engine | +| `delta_kernel_ffi` | C/C++ FFI bindings ([overview](./ffi/overview.md)) | +| `delta_kernel_derive` | Procedural macros for internal code generation | +| `acceptance` | Delta Acceptance Tests (DAT) validation suite | +| `benchmarks` | Performance benchmarks for the core library | +| `delta-kernel-unity-catalog` | Unity Catalog integration ([overview](./unity_catalog/overview.md)) | +| `unity-catalog-delta-client-api` | Transport-agnostic client traits and wire models for the Unity Catalog Delta Tables API | +| `unity-catalog-delta-rest-client` | REST/HTTP client for the Unity Catalog Delta Tables API | + +## Getting started + +For Rust projects, add to `Cargo.toml`: + +```toml +delta_kernel = "0.23" +delta_kernel_default_engine = { version = "0.23", features = ["rustls"] } +``` + +For C/C++ projects, build the FFI crate and link against it. See the +[FFI overview](./ffi/overview.md). + +Then follow the quick starts to see Kernel in action. + +## What's next + +- [Quick Start: Reading a Table](./getting_started/quick_start_read.md) +- [Quick Start: Writing a Table](./getting_started/quick_start_write.md) +- [Architecture Overview](./concepts/architecture.md) diff --git a/docs/user-guide/src/maintenance/checkpointing.md b/docs/user-guide/src/maintenance/checkpointing.md new file mode 100644 index 0000000000..d4bbc46a66 --- /dev/null +++ b/docs/user-guide/src/maintenance/checkpointing.md @@ -0,0 +1,191 @@ +# Checkpointing + +To compact a Delta table's transaction log into a single Parquet file, you write a +**checkpoint**. A checkpoint allows readers to skip old JSON commit files and start from +the checkpoint instead, which speeds up table discovery for tables with many commits. + +## When to checkpoint + +Kernel does not checkpoint automatically. Your application decides when to trigger one. +The table property `delta.checkpointInterval` controls the recommended frequency. You can +read it via `snapshot.table_properties().checkpoint_interval`, which returns an +`Option>` representing the number of commits between checkpoints (e.g., 10 +means checkpoint every 10 commits). + +## The simple path: `Snapshot::checkpoint()` + +The easiest way to write a checkpoint is the convenience method on `Snapshot`: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::snapshot::CheckpointWriteResult; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn main() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let engine = DefaultEngine::builder(store_from_url(&url)?).build(); +let snapshot = Snapshot::builder_for(url).build(&engine)?; +let (result, new_snapshot) = snapshot.checkpoint(&engine, None)?; +match result { + CheckpointWriteResult::Written => println!("Checkpoint written"), + CheckpointWriteResult::AlreadyExists => println!("Checkpoint already exists"), +} +# Ok(()) +# } +``` + +`checkpoint()` takes a `&SnapshotRef` (i.e., `&Arc`) and returns a +`DeltaResult<(CheckpointWriteResult, SnapshotRef)>`. The returned `SnapshotRef` reflects +the new checkpoint when `Written`, or the original snapshot when `AlreadyExists`. + +This method handles everything in one call: reads the log, reconciles actions, writes the +checkpoint Parquet file, and updates the `_last_checkpoint` hint file. + +> [!NOTE] +> `checkpoint()` calls `ParquetHandler::write_parquet_file` to write the checkpoint and +> `StorageHandler::head` to retrieve its metadata. The `DefaultEngine` implements both. +> If you use a custom engine, make sure it provides working implementations of both. + +## After a commit + +The most common time to checkpoint is right after a successful commit. The post-commit +snapshot gives you a checkpoint-ready `SnapshotRef`: + +```rust,ignore +let committed = match txn.commit(&engine)? { + CommitResult::CommittedTransaction(c) => c, + _ => panic!("unexpected result"), +}; + +if let Some(snapshot) = committed.post_commit_snapshot() { + let (_result, _new_snapshot) = snapshot.clone().checkpoint(&engine, None)?; +} +``` + +> [!WARNING] +> For catalog-managed tables, you must +> [publish](../catalog_managed/writing.md) the commit before checkpointing. +> `create_checkpoint_writer(&engine)` returns an error if the snapshot is not published. + +## The custom path: `CheckpointWriter` + +If you need control over how the checkpoint file is written, use the lower-level +`CheckpointWriter` API. `create_checkpoint_writer(&engine)` consumes an `Arc`: + +```rust,ignore +// 1. Create a CheckpointWriter from a snapshot (consumes the Arc) +let writer = snapshot.create_checkpoint_writer(&engine)?; + +// 2. Get the path where the checkpoint should be written +let checkpoint_path = writer.checkpoint_path()?; + +// 3. Get the checkpoint data as an ActionReconciliationIterator +let checkpoint_data = writer.checkpoint_data(&engine)?; +let state = checkpoint_data.state(); + +// 4. Write data to storage (engine-specific). +// You must fully consume the iterator and write all data. +let lazy_data = checkpoint_data + .map(|r| r.and_then(|f| f.apply_selection_vector())); +// ... write lazy_data to checkpoint_path in your own way ... + +// 5. Get file metadata (size, modification time, etc.) +let file_meta = engine.storage_handler().head(&checkpoint_path)?; + +// 6. Build LastCheckpointHintStats from the now-exhausted iterator state. +// Use 0 for num_sidecars on V1 checkpoints or V2 checkpoints without sidecars. +let state = Arc::into_inner(state) + .ok_or_else(|| Error::internal_error("checkpoint state Arc still has other references"))?; +let last_checkpoint_stats = LastCheckpointHintStats::from_reconciliation_state( + state, + file_meta.size, + 0, // num_sidecars +)?; + +// 7. Finalize. Writes the _last_checkpoint hint file. +writer.finalize(&engine, &last_checkpoint_stats)?; +``` + +The key requirement is that the data iterator must be **fully consumed** before building +the `LastCheckpointHintStats`. `LastCheckpointHintStats::from_reconciliation_state` returns +an error if the iterator has not been exhausted. Because `finalize()` reads the action +counts from the stats struct rather than from the iterator, you build the stats first and +then pass them to `finalize()`. + +## Checkpoint format + +Kernel automatically selects the checkpoint format based on table features: + +| Table feature | Checkpoint format | +|---------------|-------------------| +| No `v2Checkpoints` | Classic V1 single-file checkpoint | +| `v2Checkpoints` enabled | Classic-named V2 checkpoint (with `CheckpointMetadata` action) | + +You do not need to manage this yourself. `CheckpointWriter` handles format selection and +includes the appropriate actions. + +## Deciding when to checkpoint + +After a successful commit, `CommittedTransaction` exposes a `PostCommitStats` struct through +`post_commit_stats()`. You can use these stats to decide whether to trigger a checkpoint or +other maintenance operations. + +`PostCommitStats` has two fields: + +| Field | Meaning | +|-------|---------| +| `commits_since_checkpoint` | Number of commits since the last checkpoint. Commit 0 counts as a checkpoint. | +| `commits_since_log_compaction` | Number of commits since the last log compaction or checkpoint. A checkpoint resets this counter too. | + +The typical pattern is to check `commits_since_checkpoint` against the table's +`delta.checkpointInterval` property and checkpoint when the threshold is reached: + +```rust,ignore +let committed = match txn.commit(&engine)? { + CommitResult::CommittedTransaction(c) => c, + _ => panic!("unexpected result"), +}; + +let stats = committed.post_commit_stats(); +let interval = committed + .post_commit_snapshot() + .map(|s| { + s.table_properties() + .checkpoint_interval + .map(|n| n.get()) + .unwrap_or(10) + }) + .unwrap_or(10); + +if stats.commits_since_checkpoint >= interval { + if let Some(snapshot) = committed.post_commit_snapshot() { + let (_result, _new_snapshot) = snapshot.clone().checkpoint(&engine, None)?; + } +} +``` + +See [Version Checksums](./version_checksums.md) for the recommended full +post-commit pattern that includes writing a CRC file before checkpointing. + +## Log compaction + +**Log compaction** aggregates multiple commit JSON files into a single compacted file, +reducing the number of files Kernel must read during log replay. The API follows a similar +pattern to the `CheckpointWriter` API: create a `LogCompactionWriter` from a `Snapshot`, +retrieve the compaction path and data, then write the data to storage. + +> [!NOTE] +> Log compaction is currently disabled on both reads and writes due to insufficient +> integration test coverage. See [issue #2337](https://github.com/delta-io/delta-kernel-rs/issues/2337) +> for tracking. The `Snapshot::log_compaction_writer()` method exists but returns an error +> if called. This section will be updated when the feature is re-enabled. + +## What's next + +- [Version Checksums](./version_checksums.md) for writing CRC files after every commit +- [Appending Data](../writing/append.md) to learn about writing data and committing +- [Catalog-Managed Tables](../catalog_managed/overview.md) to understand publishing + commits for catalog-managed tables diff --git a/docs/user-guide/src/maintenance/version_checksums.md b/docs/user-guide/src/maintenance/version_checksums.md new file mode 100644 index 0000000000..85c543f3d9 --- /dev/null +++ b/docs/user-guide/src/maintenance/version_checksums.md @@ -0,0 +1,154 @@ +# Version checksums + +To speed up snapshot loading and enable table state validation, you write a +**version checksum** (CRC file) after each commit. A CRC file records a compact +summary of the table state at a given version. When a CRC file exists, Kernel +can skip expensive log replay on the next snapshot load. + +Version checksums are independent of [checkpoints](./checkpointing.md). +Checkpoints compact the transaction log into Parquet. Checksums are small JSON +files that validate and accelerate snapshot construction. You may write both +after every commit. + +## Writing a checksum + +Call `write_checksum()` on the post-commit snapshot after every successful +commit. The method writes a small JSON CRC file into the `_delta_log/` +directory and returns a `ChecksumWriteResult` indicating whether the file was +written or already existed. + +```rust,ignore +use delta_kernel::snapshot::ChecksumWriteResult; + +let committed = match txn.commit(&engine)? { + CommitResult::CommittedTransaction(c) => c, + _ => panic!("unexpected result"), +}; + +if let Some(snapshot) = committed.post_commit_snapshot() { + let (result, new_snapshot) = snapshot.write_checksum(&engine)?; + match result { + ChecksumWriteResult::Written => println!("CRC file written"), + ChecksumWriteResult::AlreadyExists => println!("CRC file already exists"), + } +} +``` + +`write_checksum()` is safe to call unconditionally. If another writer already +wrote the CRC file for this version, it returns +`ChecksumWriteResult::AlreadyExists` without error. Per the Delta protocol, +writers must not overwrite existing checksum files. + +## ChecksumWriteResult + +The return type is `DeltaResult<(ChecksumWriteResult, SnapshotRef)>`: + +| Variant | Meaning | +|---------|---------| +| `ChecksumWriteResult::Written` | The CRC file was created. The returned `SnapshotRef` includes the new CRC file in its log segment. | +| `ChecksumWriteResult::AlreadyExists` | A CRC file already exists at this version. The original snapshot is returned unchanged. | + +Use the returned `SnapshotRef` for subsequent operations (checkpointing, +publishing) so that downstream code sees the CRC file. + +> [!NOTE] +> `write_checksum()` currently requires a post-commit snapshot with +> pre-computed CRC information in memory. Calling it on a snapshot loaded from +> disk (without a pre-computed CRC) returns an +> `Error::ChecksumWriteUnsupported` error. In practice, this means you call +> `write_checksum()` on the snapshot returned by +> `CommittedTransaction::post_commit_snapshot()`. + +## Recommended post-commit pattern + +After every commit, write the checksum first, then decide whether to +checkpoint: + +```rust,ignore +let committed = match txn.commit(&engine)? { + CommitResult::CommittedTransaction(c) => c, + _ => panic!("unexpected result"), +}; + +if let Some(snapshot) = committed.post_commit_snapshot() { + // 1. Write the version checksum + let (_, snapshot) = snapshot.write_checksum(&engine)?; + + // 2. Check whether it is time to checkpoint + let stats = committed.post_commit_stats(); + let interval = snapshot + .table_properties() + .checkpoint_interval + .map(|n| n.get()) + .unwrap_or(10); + + if stats.commits_since_checkpoint >= interval { + let (_result, _new_snapshot) = snapshot.checkpoint(&engine)?; + } +} +``` + +This pattern keeps your connector's maintenance logic in one place: commit, +write the CRC file, then conditionally checkpoint based on the table's +configured interval. + +## Reading file stats from a checksum + +When Kernel loads a snapshot whose version has a CRC file, it parses the +checksum's table-level statistics and caches them on the snapshot. Call +`Snapshot::get_file_stats_if_present()` to read them back: + +```rust,ignore +if let Some(stats) = snapshot.get_file_stats_if_present() { + println!("num files: {}", stats.num_files); + println!("table size bytes: {}", stats.table_size_bytes); +} +``` + +`get_file_stats_if_present()` returns `None` when this snapshot has no CRC at +its version. It never triggers a CRC read. It only returns stats that were +already materialized during snapshot construction, so the call is cheap and +always synchronous. + +### File size histogram + +When the writer that produced the CRC file recorded a file size histogram, +Kernel exposes it alongside the scalar totals. The histogram groups the +snapshot's live files into size bins: + +```rust,ignore +if let Some(stats) = snapshot.get_file_stats_if_present() { + if let Some(histogram) = &stats.file_size_histogram { + for ((bin_start, count), bytes) in histogram + .sorted_bin_boundaries + .iter() + .zip(&histogram.file_counts) + .zip(&histogram.total_bytes) + { + println!(">= {bin_start:>12} B: {count} files, {bytes} bytes"); + } + } +} +``` + +`sorted_bin_boundaries` gives the inclusive lower bound of each bin. The next +boundary is the exclusive upper bound. `file_counts` and `total_bytes` are +parallel arrays with the same length. The histogram is `None` when the writer +did not populate it. + +Use these stats for lightweight planning (cost estimation, compaction +heuristics, monitoring) without replaying the log. When +`get_file_stats_if_present()` returns `None`, either fall back to aggregating +`ScanFile.size` and `ScanFile.stats` via `visit_scan_files`, or write a CRC +file on the next commit so subsequent snapshots have stats available. + +> [!NOTE] +> `file_size_histogram` is an optional field in the Delta protocol. Kernel +> surfaces whatever the CRC file contains. A `None` value means the histogram +> was omitted, not that the table has no files. + +## What's next + +- [Checkpointing](./checkpointing.md) for compacting the transaction log into + Parquet +- [Appending Data](../writing/append.md) for the full write and commit flow diff --git a/docs/user-guide/src/observability/observability.md b/docs/user-guide/src/observability/observability.md new file mode 100644 index 0000000000..f3b7e0a769 --- /dev/null +++ b/docs/user-guide/src/observability/observability.md @@ -0,0 +1,320 @@ +# Metrics and monitoring + +To observe what Kernel is doing at runtime, install a `tracing-subscriber` layer +that converts Kernel's spans and events into `MetricEvent` values. Kernel +instruments snapshot loading, scanning, and storage I/O with the +[`tracing`](https://docs.rs/tracing) crate, so you can also receive raw spans +and events in any subscriber you already use for logging. + +## How metrics flow + +Kernel emits `tracing` spans at key milestones (snapshot build, log segment +load, scan metadata replay, storage I/O). A `ReportGeneratorLayer`, attached +to your subscriber, watches those spans and forwards `MetricEvent` values to a +`MetricsReporter` you provide. Your reporter can do whatever you want with +each event: print it, push it to Prometheus, increment a counter, or fan it +out to several destinations. + +```text +Kernel code + | tracing span / event + v +tracing-subscriber Registry + +-- ReportGeneratorLayer ---> MetricsReporter (your impl) + +-- fmt::layer (logs) + +-- EnvFilter, etc. +``` + +This means metrics aren't tied to your `Engine`. You wire them up once, at +process startup, alongside any other tracing layers you want. + +## Enabling metrics + +To enable metrics, build a tracing subscriber, add the metrics layer with +`WithMetricsReporterLayer::with_metrics_reporter_layer`, and call `init()`. +Kernel ships `LoggingMetricsReporter` as a built-in reporter that logs each +event at a tracing level you choose. + +Filename: src/main.rs + +```rust,no_run +# extern crate delta_kernel; +# extern crate tracing; +# extern crate tracing_subscriber; +use std::sync::Arc; +use delta_kernel::metrics::{LoggingMetricsReporter, WithMetricsReporterLayer}; +use tracing_subscriber::layer::SubscriberExt as _; +use tracing_subscriber::util::SubscriberInitExt as _; + +fn main() { + tracing_subscriber::registry() + .with(tracing_subscriber::fmt::layer()) + .with_metrics_reporter_layer( + Arc::new(LoggingMetricsReporter::new(tracing::Level::INFO)), + ) + .init(); + + // ... build engine and read tables as usual +} +``` + +If you don't install the layer, no `MetricEvent` values are produced. The +underlying spans still exist, so other tracing layers (logging, distributed +tracing) keep working. + +> [!NOTE] +> The `MetricsReporter` trait, `WithMetricsReporterLayer` extension, and +> `LoggingMetricsReporter` live under `delta_kernel::metrics`. You also need +> the `tracing` and `tracing-subscriber` crates as direct dependencies of +> your connector. + +## Implementing a custom MetricsReporter + +The `MetricsReporter` trait has a single method: + +```rust,ignore +pub trait MetricsReporter: Send + Sync + std::fmt::Debug { + fn report(&self, event: MetricEvent); +} +``` + +Your reported must be `Send + Sync` because the layer can call `report` from any thread +that produces a span. Keep `report` cheap. If your destination is slow, push +the event onto a channel and drain it from a worker. + +Filename: src/reporter.rs + +```rust,ignore +use delta_kernel::metrics::{MetricEvent, MetricsReporter}; + +#[derive(Debug)] +struct StdoutReporter; + +impl MetricsReporter for StdoutReporter { + fn report(&self, event: MetricEvent) { + println!("[kernel-metrics] {event}"); + } +} +``` + +Wire it in the same way as `LoggingMetricsReporter`: + +```rust,ignore +use std::sync::Arc; +use delta_kernel::metrics::WithMetricsReporterLayer; +use tracing_subscriber::layer::SubscriberExt as _; +use tracing_subscriber::util::SubscriberInitExt as _; + +tracing_subscriber::registry() + .with(tracing_subscriber::fmt::layer()) + .with_metrics_reporter_layer(Arc::new(StdoutReporter)) + .init(); +``` + +When a snapshot loads, you'll see output like: + +```text +[kernel-metrics] LogSegmentLoadSuccess(id=a1b2c3d4-..., duration=12.34ms, commits=5, checkpoints=1, compactions=0, has_latest_crc=true) +[kernel-metrics] ProtocolMetadataLoadSuccess(id=a1b2c3d4-..., duration=3.21ms) +[kernel-metrics] SnapshotBuildSuccess(id=a1b2c3d4-..., version=5, duration=15.55ms) +``` + +## Metric events + +Every callback receives a `MetricEvent` enum value. Events fall into three +categories: snapshot lifecycle, scan metadata, and storage/file I/O. + +### Snapshot lifecycle events + +These events track the process of loading a [Snapshot](../concepts/architecture.md) +from the Delta log. Each carries an `operation_id` (`MetricId`) that ties all +events from the same snapshot load together. + +| Event | Fields | What it measures | +|-------|--------|------------------| +| `LogSegmentLoadSuccess` | `operation_id`, `duration`, `num_commit_files`, `num_checkpoint_files`, `num_compaction_files`, `has_latest_crc_file` | Time to list and organize log files into a log segment. | +| `LogSegmentLoadFailure` | `operation_id` | Log segment load failed. | +| `ProtocolMetadataLoadSuccess` | `operation_id`, `duration` | Time to read protocol and metadata actions from the log. | +| `ProtocolMetadataLoadFailure` | `operation_id` | Protocol/metadata load failed. | +| `SnapshotBuildSuccess` | `operation_id`, `version`, `duration` | End-to-end snapshot creation, including the table version that was loaded. | +| `SnapshotBuildFailure` | `operation_id` | Snapshot creation failed. Use this to track error rates. | + +### Scan metadata events + +`ScanMetadataCompleted` is emitted when a scan metadata iterator is fully +consumed. It provides detailed statistics about the log replay process: + +| Field | Meaning | +|-------|---------| +| `operation_id` | Unique ID for this scan, useful for correlation. | +| `scan_type` | Which scan path produced the event (see below). | +| `duration` | Wall-clock time from scan start to iterator exhaustion. | +| `num_add_files_seen` | Add files that entered deduplication. This normally excludes data-skipped files; parse-error fallback includes data-skipped files during retry. | +| `num_active_add_files` | Add files that survived log replay. These are the files your connector reads. | +| `num_remove_files_seen` | Remove actions encountered in commit files. | +| `num_non_file_actions` | Non-file actions (protocol, metadata, etc.) seen during replay. | +| `num_predicate_filtered` | Files eliminated by predicate evaluation (data skipping and partition pruning). | +| `peak_hash_set_size` | Peak size of the internal deduplication set. Indicates memory pressure during log replay. | +| `dedup_visitor_time_ns` | Nanoseconds spent in the deduplication visitor. | +| `predicate_eval_time_ns` | Nanoseconds spent evaluating predicates. | + +#### The ScanType enum + +The `scan_type` field tells you which scan execution path produced the event: + +| Variant | Source | +|---------|--------| +| `ScanType::Full` | Produced by `Scan::scan_metadata()`. The entire log replay happened in one pass. | +| `ScanType::SequentialPhase` | The sequential phase of `Scan::parallel_scan_metadata()`. | +| `ScanType::ParallelPhase` | The parallel phase of `Scan::parallel_scan_metadata()`. | + +If you use `parallel_scan_metadata`, you'll receive two `ScanMetadataCompleted` +events per scan: one for each phase. + +### Storage and file I/O events + +These events track low-level I/O operations. The default storage, JSON, and +Parquet handlers emit them automatically when the metrics layer is installed. +Unlike snapshot events, these don't carry an `operation_id` because a single +storage call may serve multiple higher-level operations. + +| Event | Fields | What it measures | +|-------|--------|------------------| +| `StorageListCompleted` | `duration`, `num_files` | A storage list call (e.g., listing the `_delta_log` directory). | +| `StorageReadCompleted` | `duration`, `num_files`, `bytes_read` | A storage read call. `bytes_read` is the total on-disk size. | +| `StorageCopyCompleted` | `duration` | A storage copy/rename call. | +| `JsonReadCompleted` | `num_files`, `bytes_read` | One `JsonHandler::read_json_files` call completed. `bytes_read` is the sum of on-disk file sizes. | +| `ParquetReadCompleted` | `num_files`, `bytes_read` | One `ParquetHandler::read_parquet_files` call completed. `bytes_read` is the sum of on-disk file sizes. | +| `CrcReadSuccess` | `duration`, `bytes_read` | One CRC file read and parsed successfully. `bytes_read` is the raw byte count from storage. | +| `CrcReadFailure` | none | A CRC file read or parse failed. The caller falls back to log replay. | + +> [!NOTE] +> If you implement a custom `JsonHandler` or `ParquetHandler`, call +> `delta_kernel::metrics::emit_json_read_completed` or +> `emit_parquet_read_completed` once per read call so connectors that install +> the metrics layer still see those events from your handler. + +## Correlating events with MetricId + +Several events include an `operation_id` field of type `MetricId`. This is a +UUID that uniquely identifies an operation instance. All events from the same +snapshot load share the same `MetricId`, so you can group them to reconstruct +a timeline: + +1. `LogSegmentLoadSuccess` or `LogSegmentLoadFailure` (listing the log segment) +2. `ProtocolMetadataLoadSuccess` or `ProtocolMetadataLoadFailure` (reading protocol and metadata) +3. `SnapshotBuildSuccess` or `SnapshotBuildFailure` (final outcome and total duration) + +Each step emits a success or failure variant carrying the shared `operation_id`. When a +step fails, the snapshot build fails too, so the terminal `SnapshotBuildFailure` carries +the same id and you can see which step broke from the buffered group. + +You can store the `MetricId` in your monitoring system as a trace ID or +correlation key. Because everything flows through `tracing`, you can also +correlate Kernel's events with your own application spans by attaching them +to the same parent span. + +## Example: correlating snapshot events by operation ID + +To aggregate timing data per operation, store intermediate events keyed by +`MetricId` and compute totals when the terminal event arrives: + +Filename: src/correlating_reporter.rs + +```rust,ignore +use std::collections::HashMap; +use std::sync::Mutex; +use delta_kernel::metrics::{MetricEvent, MetricId, MetricsReporter}; + +#[derive(Debug)] +struct CorrelatingReporter { + pending: Mutex>>, +} + +impl CorrelatingReporter { + fn new() -> Self { + Self { pending: Mutex::new(HashMap::new()) } + } + + // Buffer an intermediate step under its operation_id. + fn buffer(&self, operation_id: MetricId, event: MetricEvent) { + self.pending.lock().unwrap().entry(operation_id).or_default().push(event); + } + + // Drain the buffered steps for a finished operation. + fn drain(&self, operation_id: MetricId) -> Vec { + self.pending.lock().unwrap().remove(&operation_id).unwrap_or_default() + } +} + +impl MetricsReporter for CorrelatingReporter { + fn report(&self, event: MetricEvent) { + match event { + // 1. Buffer each intermediate step, success OR failure, under its operation_id. + // `ref e` borrows the id so `event` can still move into the buffer. + MetricEvent::LogSegmentLoadSuccess(ref e) => self.buffer(e.operation_id, event), + MetricEvent::LogSegmentLoadFailure(ref e) => self.buffer(e.operation_id, event), + MetricEvent::ProtocolMetadataLoadSuccess(ref e) => self.buffer(e.operation_id, event), + MetricEvent::ProtocolMetadataLoadFailure(ref e) => self.buffer(e.operation_id, event), + + // 2. The terminal event drains the group. The buffered steps reconstruct the + // timeline; on failure, the last step shows where the build broke. + MetricEvent::SnapshotBuildSuccess(e) => { + let steps = self.drain(e.operation_id); + println!( + "Snapshot v{} completed in {:?} ({} sub-events)", + e.version, + e.duration, + steps.len() + ); + } + MetricEvent::SnapshotBuildFailure(e) => { + let steps = self.drain(e.operation_id); + println!( + "Snapshot {} failed after {} sub-events; last step: {:?}", + e.operation_id, + steps.len(), + steps.last() + ); + } + // Storage, scan, and post-build loads don't participate in snapshot correlation. + _ => {} + } + } +} +``` + +## Sending metrics to multiple destinations + +To report to more than one system (for example, a logger and Prometheus), +create a composite reporter that fans out to multiple inner reporters: + +Filename: src/composite_reporter.rs + +```rust,ignore +use std::sync::Arc; +use delta_kernel::metrics::{MetricEvent, MetricsReporter}; + +#[derive(Debug)] +struct CompositeReporter { + reporters: Vec>, +} + +impl MetricsReporter for CompositeReporter { + fn report(&self, event: MetricEvent) { + for reporter in &self.reporters { + reporter.report(event.clone()); + } + } +} +``` + +`MetricEvent` implements `Clone`, so each inner reporter receives its own copy. +You only install one `ReportGeneratorLayer` on your subscriber. The composite +fans out from there. + +## See also + +- [`tracing`](https://docs.rs/tracing) and + [`tracing-subscriber`](https://docs.rs/tracing-subscriber) crate + documentation, for layer composition, filters, and other subscribers. diff --git a/docs/user-guide/src/reading/building_a_scan.md b/docs/user-guide/src/reading/building_a_scan.md new file mode 100644 index 0000000000..de6b517d5b --- /dev/null +++ b/docs/user-guide/src/reading/building_a_scan.md @@ -0,0 +1,166 @@ +# Building a scan + +To read data from a Delta table, you build a `Scan` from a `Snapshot`, optionally +configure column selection and filter predicates, and then execute it. + +## The basic pattern + +Every scan follows the same pattern: + +1. Get a `Snapshot` of the table +2. Call `snapshot.scan_builder()` to get a `ScanBuilder` +3. Configure the builder (optional) +4. Call `.build()` to create the `Scan` +5. Execute the scan + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +let snapshot = Snapshot::builder_for(url).build(&engine)?; + +let scan = snapshot + .scan_builder() + .build()?; +# Ok(()) +# } +``` + +Without any configuration, this scans all columns with no filter. It's equivalent to +`SELECT * FROM table`. + +## Configuring a scan + +`ScanBuilder` supports two main configuration options: + +### Column selection with `with_schema` + +Pass a schema containing only the columns you want to read: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::schema::{DataType, StructField, StructType}; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +# let snapshot = Snapshot::builder_for(url).build(&engine)?; +let read_schema = Arc::new(StructType::try_new([ + StructField::nullable("name", DataType::STRING), + StructField::nullable("age", DataType::INTEGER), +])?); + +let scan = snapshot + .scan_builder() + .with_schema(read_schema) + .build()?; +# Ok(()) +# } +``` + +The schema you provide must be a subset of the table's schema. Kernel only reads the +columns you specify from each Parquet file. + +For more details, see [Column Selection](./column_selection.md). + +### Filter pushdown with `with_predicate` + +Pass a predicate expression to skip files that cannot contain matching rows: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::expressions::{col, lit, Predicate}; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +# let snapshot = Snapshot::builder_for(url).build(&engine)?; +let predicate = Arc::new( + Predicate::lt(col!("age"), lit(30)) +); + +let scan = snapshot + .scan_builder() + .with_predicate(predicate) + .build()?; +# Ok(()) +# } +``` + +Kernel uses the predicate to evaluate file-level statistics (min/max values) and skip +entire files that cannot match. This is called **data skipping** and can significantly +reduce the amount of data read. + +> [!NOTE] +> Filtering is **best-effort**. The scan may still include rows that don't match the +> predicate. Your connector should apply the filter to the returned data for exact +> results. + +For more details, see [Filter Pushdown and File Skipping](./filter_pushdown.md). + +## Executing a scan + +There are two ways to execute a scan: the **simple path** for single-process use, and the +**advanced path** for when you need control over parallelism. + +### Simple path: `execute()` + +`execute()` handles log replay, data skipping, file reading, and physical-to-logical +transformations. It returns an iterator of `EngineData` results. + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel::engine::arrow_data::EngineDataArrowExt as _; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +# let snapshot = Snapshot::builder_for(url).build(&engine)?; +let scan = snapshot.scan_builder().build()?; + +let mut batches = vec![]; +for data in scan.execute(Arc::new(engine))? { + let record_batch: delta_kernel::arrow::record_batch::RecordBatch = + data?.try_into_record_batch()?; + batches.push(record_batch); +} +# Ok(()) +# } +``` + +`execute()` takes an `Arc` (not a reference) because it needs to keep the +engine alive for the lifetime of the returned iterator. + +If you're using the default engine, each `EngineData` is backed by an Arrow `RecordBatch`. +The `try_into_record_batch()` method (from the `EngineDataArrowExt` trait) unwraps it. + +This is the right choice when you're running in a single process and don't need to control +how files are distributed across threads. + +## What's next + +- [Column Selection](./column_selection.md): projecting specific columns +- [Advanced Reads with scan_metadata()](./scan_metadata.md): distributed and multi-threaded reads +- [Filter Pushdown and File Skipping](./filter_pushdown.md): predicate-based optimization diff --git a/docs/user-guide/src/reading/change_data_feed.md b/docs/user-guide/src/reading/change_data_feed.md new file mode 100644 index 0000000000..97334b09a3 --- /dev/null +++ b/docs/user-guide/src/reading/change_data_feed.md @@ -0,0 +1,176 @@ +# Reading change data feed + +To read a row-level changelog of what changed between two versions of a Delta table, you +use `TableChanges` and `TableChangesScan`. This gives you every insert, update, and delete +that occurred in the specified version range, with metadata columns that identify the type +of change and the commit it came from. + +Before reading this page, make sure you understand +[Building a Scan](./building_a_scan.md). + +## Prerequisites + +**Change Data Feed** (CDF) is a Delta feature that records row-level changes (inserts, +updates, deletes) as part of each commit. The table must have the `delta.enableChangeDataFeed` +table property set to `true` for every version in the range you want to read. If CDF is +disabled for any version in the range, `TableChanges::try_new` returns an error. + +## Creating a TableChanges + +`TableChanges::try_new` takes a table URL, an `Engine` reference, a start version, and an +optional end version. It validates that CDF is enabled and that the schema is compatible +across the requested range. + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::table_changes::TableChanges; +# use delta_kernel::DeltaResult; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/my-table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +// Read changes from version 0 through version 5 (inclusive) +let table_changes = TableChanges::try_new(url, &engine, 0, Some(5))?; +# Ok(()) +# } +``` + +If you omit the end version by passing `None`, Kernel defaults to the latest version of the +table at the time of the call. + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::table_changes::TableChanges; +# use delta_kernel::DeltaResult; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/my-table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +// Read all changes from version 0 to the latest version +let table_changes = TableChanges::try_new(url, &engine, 0, None)?; +# Ok(()) +# } +``` + +`try_new` performs several validation checks before returning: + +- CDF must be enabled at both the start and end versions. +- The table schema at the start and end versions must be identical. +- No unsupported reader features (other than deletion vectors) are enabled. + +If any check fails, it returns an `Error`. + +## CDF metadata columns + +The schema of `TableChanges` is the table's schema at the end version plus three additional +columns: + +| Column | Type | Description | +|--------|------|-------------| +| `_change_type` | `STRING` (non-nullable) | One of `insert`, `delete`, `update_preimage`, or `update_postimage` | +| `_commit_version` | `LONG` (non-nullable) | The table version where this change was committed | +| `_commit_timestamp` | `TIMESTAMP` (non-nullable) | The timestamp of the commit. When In-Commit Timestamps are enabled, this is the ICT value; otherwise it falls back to the log file's modification time. | + +You can access the full schema (table columns plus CDF columns) via +`table_changes.schema()`. You can include any of the CDF columns in a projection, the same +way you would project regular table columns. + +## Building and executing a scan + +`TableChangesScanBuilder` works like `ScanBuilder` for regular scans. You can optionally +project columns with `with_schema` and filter rows with `with_predicate`. + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::expressions::{col, lit}; +# use delta_kernel::table_changes::TableChanges; +# use delta_kernel::{DeltaResult, Predicate}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/my-table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +# let table_changes = TableChanges::try_new(url, &engine, 0, Some(5))?; +// 1. Project only the columns you need (including CDF metadata) +let schema = table_changes + .schema() + .project(&["name", "age", "_change_type", "_commit_version"])?; + +// 2. Build a predicate on table columns +let predicate = Arc::new(Predicate::gt(col!("age"), lit(25))); + +// 3. Build and execute the scan +let scan = table_changes + .into_scan_builder() + .with_schema(schema) + .with_predicate(predicate) + .build()?; + +for data in scan.execute(Arc::new(engine))? { + let batch = data?; + println!("Got {} rows of change data", batch.len()); +} +# Ok(()) +# } +``` + +If you skip `with_schema`, the scan returns all table columns plus all three CDF columns. +If you skip `with_predicate`, no filtering is applied. + +> [!NOTE] +> Predicates on the CDF metadata columns (`_change_type`, `_commit_version`, +> `_commit_timestamp`) are accepted but have no effect. Kernel doesn't apply them during +> file pruning because CDF metadata is synthesized during scan execution, not read from +> data files. Apply such filters in your connector code after the scan returns. See +> [issue #525](https://github.com/delta-io/delta-kernel-rs/issues/525). + +> [!NOTE] +> Like regular scans, CDF filtering is best-effort. The scan may return rows that do not +> match your predicate. Apply row-level filtering in your connector if exact results are +> required. + +## Using `Arc` + +If you need to retain ownership of the `TableChanges` (for example, to inspect its schema +after building the scan), use `scan_builder` on an `Arc` instead of +`into_scan_builder`: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::table_changes::TableChanges; +# use delta_kernel::DeltaResult; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/my-table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +let table_changes = Arc::new( + TableChanges::try_new(url, &engine, 0, Some(5))? +); + +let scan = table_changes.clone().scan_builder().build()?; + +// You can still access table_changes after building the scan +println!("CDF range: version {} to {}", table_changes.start_version(), table_changes.end_version()); +# Ok(()) +# } +``` + +## What's next + +- [Building a Scan](./building_a_scan.md): the core scan pattern that CDF scanning builds on +- [Filter Pushdown and File Skipping](./filter_pushdown.md): predicate construction and + data skipping behavior +- [Column Selection](./column_selection.md): projecting specific columns from a scan diff --git a/docs/user-guide/src/reading/column_selection.md b/docs/user-guide/src/reading/column_selection.md new file mode 100644 index 0000000000..8d3ede85af --- /dev/null +++ b/docs/user-guide/src/reading/column_selection.md @@ -0,0 +1,96 @@ +# Column Selection + +By default a scan reads all columns from a table. You can select a subset of columns +(projection pushdown) so the engine only reads the data you need. + +## Projecting columns + +Use `Schema::project()` to create a schema containing only the columns you want, then +pass it to `ScanBuilder::with_schema()`: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn main() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let engine = DefaultEngine::builder(store_from_url(&url)?).build(); +# let snapshot = Snapshot::builder_for(url).build(&engine)?; +// Table has columns [id, name, email, created_at] +// Select only id and name +let projected_schema = snapshot.schema().project(&["id", "name"])?; + +let scan = snapshot + .scan_builder() + .with_schema(projected_schema) + .build()?; +# Ok(()) +# } +``` + +The returned data will contain only the projected columns, in the order you specified. +Requesting a column that does not exist in the table schema returns an error. + +## Reordering columns + +`project()` returns columns in the order you provide, which can differ from the table +schema order: + +```rust,ignore +// Table schema is [id, name, email] +// Return [email, id] +let reordered = snapshot.schema().project(&["email", "id"])?; +``` + +## Metadata columns + +You can request metadata columns that are not part of the table data but provide +information about each row's origin. Add them to your scan schema with +`Schema::add_metadata_column()`: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::schema::MetadataColumnSpec; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn main() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let engine = DefaultEngine::builder(store_from_url(&url)?).build(); +# let snapshot = Snapshot::builder_for(url).build(&engine)?; +// Start with a projection +let schema = snapshot.schema().project_as_struct(&["id", "name"])?; + +// Add a row index metadata column +let schema = schema.add_metadata_column("row_idx", MetadataColumnSpec::RowIndex)?; + +let scan = snapshot + .scan_builder() + .with_schema(Arc::new(schema)) + .build()?; +# Ok(()) +# } +``` + +The available metadata columns: + +| Spec | Data type | Description | +|------|-----------|-------------| +| `MetadataColumnSpec::FilePath` | `STRING` | Path of the Parquet file containing the row | +| `MetadataColumnSpec::RowIndex` | `LONG` | Zero-based row position within the Parquet file | +| `MetadataColumnSpec::RowId` | `LONG` | Stable row identifier (requires row tracking on the table) | +| `MetadataColumnSpec::RowCommitVersion` | `LONG` | Commit version that last wrote or updated the row (requires row tracking on the table) | + +You choose the column name when calling `add_metadata_column()`. Only one metadata +column of each type is allowed per scan. + +## What's next + +- [Building a Scan](./building_a_scan.md) covers the full scan API. +- [Filter Pushdown and File Skipping](./filter_pushdown.md) covers predicate-based + optimization. diff --git a/docs/user-guide/src/reading/filter_pushdown.md b/docs/user-guide/src/reading/filter_pushdown.md new file mode 100644 index 0000000000..1c92a3ead5 --- /dev/null +++ b/docs/user-guide/src/reading/filter_pushdown.md @@ -0,0 +1,344 @@ +# Filter pushdown and file skipping + +To reduce the amount of data your connector reads from storage, you can provide a predicate +to a scan. Kernel uses the predicate for **data skipping**, evaluating file-level statistics +to skip entire Parquet files that cannot contain matching rows. + +Before reading this page, make sure you understand +[Building a Scan](./building_a_scan.md). + +## Building predicates + +A **predicate** is a boolean expression that describes which rows you want. Kernel +evaluates predicates against file-level statistics to skip files before reading them. + +Predicates are built from the `Predicate` type and `Expression` values. The simplest way +is to use the `col!` macro for column references and `lit()` for literal values. + +### Comparison operators + +```rust,no_run +# extern crate delta_kernel; +use delta_kernel::expressions::{col, lit, Predicate}; + +// age < 30 +let pred = Predicate::lt(col!("age"), lit(30)); + +// price >= 100.0 +let pred = Predicate::ge(col!("price"), lit(100.0_f64)); + +// name == "Alice" +let pred = Predicate::eq(col!("name"), lit("Alice")); + +// status != "deleted" +let pred = Predicate::ne(col!("status"), lit("deleted")); +``` + +The full set of comparison constructors: + +| Constructor | SQL equivalent | +|-------------|----------------| +| `Predicate::eq(a, b)` | `a = b` | +| `Predicate::ne(a, b)` | `a != b` | +| `Predicate::lt(a, b)` | `a < b` | +| `Predicate::le(a, b)` | `a <= b` | +| `Predicate::gt(a, b)` | `a > b` | +| `Predicate::ge(a, b)` | `a >= b` | +| `Predicate::distinct(a, b)` | `a IS DISTINCT FROM b` | + +`distinct` is a NULL-safe inequality: it returns true when `a` and `b` differ, even when +one or both are NULL. The other comparisons follow SQL NULL semantics and produce NULL when +either input is NULL. + +Each constructor takes `impl Into` for both arguments, so you can pass +`col!()` / `lit()` results, `Scalar` values, or any `Expression` directly. + +### Combining predicates + +Use `and`, `or`, and `not` to build compound predicates: + +```rust,no_run +# extern crate delta_kernel; +use delta_kernel::expressions::{col, lit, Predicate}; + +// age >= 18 AND age < 65 +let pred = Predicate::and( + Predicate::ge(col!("age"), lit(18)), + Predicate::lt(col!("age"), lit(65)), +); + +// status == "active" OR status == "pending" +let pred = Predicate::or( + Predicate::eq(col!("status"), lit("active")), + Predicate::eq(col!("status"), lit("pending")), +); + +// NOT (archived) +let pred = Predicate::not( + Predicate::eq(col!("archived"), lit(true)), +); +``` + +For combining more than two predicates, use `and_from` or `or_from`: + +```rust,no_run +# extern crate delta_kernel; +use delta_kernel::expressions::{col, lit, Predicate}; + +// age >= 18 AND country == "US" AND active == true +let pred = Predicate::and_from([ + Predicate::ge(col!("age"), lit(18)), + Predicate::eq(col!("country"), lit("US")), + Predicate::eq(col!("active"), lit(true)), +]); +``` + +### NULL checks + +```rust,no_run +# extern crate delta_kernel; +use delta_kernel::expressions::{col, lit, Predicate}; + +// email IS NULL +let pred = Predicate::is_null(col!("email")); + +// email IS NOT NULL +let pred = Predicate::is_not_null(col!("email")); +``` + +### Nested columns + +The `col!` macro supports dot-separated paths for nested struct fields: + +```rust,no_run +# extern crate delta_kernel; +use delta_kernel::expressions::{col, lit, Predicate}; + +// address.city == "Seattle" +let pred = Predicate::eq( + col!("address.city"), + lit("Seattle"), +); +``` + +### Method syntax + +You can also build predicates using method syntax on `Expression`: + +```rust,no_run +# extern crate delta_kernel; +use delta_kernel::expressions::{col, lit}; + +// age < 30 +let pred = col!("age").lt(lit(30)); + +// name == "Alice" +let pred = col!("name").eq(lit("Alice")); + +// email IS NOT NULL +let pred = col!("email").is_not_null(); +``` + +## Applying a predicate to a scan + +Pass the predicate to `ScanBuilder::with_predicate`: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::expressions::{col, lit, Predicate}; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +# let snapshot = Snapshot::builder_for(url).build(&engine)?; +let predicate = Arc::new( + Predicate::and( + Predicate::ge(col!("age"), lit(18)), + Predicate::lt(col!("age"), lit(65)), + ) +); + +let scan = snapshot + .scan_builder() + .with_predicate(predicate) + .build()?; +# Ok(()) +# } +``` + +`with_predicate` takes `impl Into>`, so you can pass an +`Arc` directly. + +## How data skipping works + +When a scan has a predicate, Kernel applies it in two stages to eliminate files before +your connector reads them. + +**File skipping using statistics.** Each Parquet file in a Delta table has associated +statistics: minimum and maximum values per column, null counts, and row counts. Kernel +rewrites your predicate into a data skipping predicate that evaluates against these +statistics. For example, given the predicate `age < 30`, if a file's minimum value for +`age` is 35, Kernel knows no rows in that file can match and skips it entirely. If a +file's minimum is 10 and maximum is 50, the file *might* contain matching rows, so +Kernel keeps it. + +**Partition pruning.** For partitioned tables, partition column values are stored in the +Delta log metadata rather than in the Parquet files. Kernel evaluates predicates on +partition columns directly against these metadata values, which is even cheaper than +statistics-based skipping because no file I/O is required. + +## Filtering is best-effort + +> [!WARNING] +> Data skipping is an optimization, not a guarantee. The scan may return rows that do +> not match your predicate. Your connector must apply row-level filtering after reading +> the data if exact results are required. + +This happens for several reasons: + +- **Statistics are at the file level**, not the row level. A file whose min/max range + overlaps the predicate may still contain non-matching rows. +- **Not all columns have statistics.** Delta tables have a configurable limit on how + many columns collect statistics (default: 32). +- **Kernel may not fully evaluate complex predicates.** It skips what it can and passes + through the rest. + + + +## Controlling statistics + +By default, Kernel reads file-level statistics from the transaction log +and uses them internally for data skipping, but does not expose those statistics to your +connector. To override the default, call `ScanBuilder::with_stats` with a +`StatsOptions` value. The named constructors cover the common shapes; you can also +build the struct directly for any combination. + +### Disabling data skipping entirely + +If your compute engine performs its own data skipping, you can tell Kernel to skip reading +statistics altogether. This avoids the cost of parsing statistics from checkpoint files. + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::scan::StatsOptions; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +# let snapshot = Snapshot::builder_for(url).build(&engine)?; +let scan = snapshot + .scan_builder() + .with_stats(StatsOptions::none()) + .build()?; +# Ok(()) +# } +``` + +With `StatsOptions::none()`: + +- Kernel skips the stats column entirely during checkpoint reads. +- No statistics-based or partition-value-based file pruning occurs (row-level + partition filtering still applies). +- The `stats` field on each `ScanFile` is `None`. + +Use this when your connector or compute engine already handles file pruning and you want to +avoid the overhead of parsing statistics you won't use. + +### Including all statistics in scan metadata + +To receive pre-parsed statistics (min/max values, null counts, row counts) for every file in your +scan metadata, pass `StatsOptions::all_struct()` (structured stats without JSON synthesis) or +`StatsOptions::all()` (both structured stats and the legacy JSON `stats` column): + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::scan::StatsOptions; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +# let snapshot = Snapshot::builder_for(url).build(&engine)?; +let scan = snapshot + .scan_builder() + .with_stats(StatsOptions::all_struct()) + .build()?; +# Ok(()) +# } +``` + +The statistics appear in a `stats_parsed` column in the scan metadata. Which columns have +statistics depends on the table's configuration (`delta.dataSkippingStatsColumns` or +`delta.dataSkippingNumIndexedCols`). + +For compatible checkpoints, `all_struct` leaves `stats` null and avoids reading or synthesizing +JSON stats. JSON commits and fallback checkpoints preserve existing JSON. + +You can combine this with `with_predicate`. When both are set, Kernel performs its own data +skipping internally and exposes the parsed statistics so your connector can apply +additional pruning logic. + +### Including statistics for specific columns + +To receive statistics for only a subset of columns, pass `StatsOptions::struct_columns`: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::expressions::column_name; +# use delta_kernel::scan::StatsOptions; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +# let snapshot = Snapshot::builder_for(url).build(&engine)?; +let scan = snapshot + .scan_builder() + .with_stats(StatsOptions::struct_columns(vec![ + column_name!("age"), + column_name!("city"), + ])) + .build()?; +# Ok(()) +# } +``` + +The named columns always appear in `stats_parsed`. When the scan also has a predicate, +predicate-referenced columns may appear as well because Kernel can retain the statistics it uses +for data skipping. Connectors should treat the named columns as a minimum projection and ignore +additional columns they do not need. + +### Choosing the right mode + +| Goal | Constructor | +|------|-------------| +| Default behavior (Kernel skips files internally, no stats exposed) | No call needed (or `StatsOptions::json_only()`) | +| Disable all stats reading for performance | `StatsOptions::none()` | +| Expose all structured stats without JSON synthesis | `StatsOptions::all_struct()` | +| Expose both struct stats and the JSON `stats` column | `StatsOptions::all()` | +| Expose selected structured stats without JSON synthesis | `StatsOptions::struct_columns(cols)` | + +`with_stats` takes a single `StatsOptions` value, so each call fully replaces any prior +configuration. There is no "last call wins" composition to track. + +## What's next + +- [Column Selection](./column_selection.md) covers projecting specific columns to further + reduce the data you read. +- [Scan Metadata](./scan_metadata.md) explains how to access per-file scan information, + including partition values and deletion vectors. diff --git a/docs/user-guide/src/reading/incremental_scan.md b/docs/user-guide/src/reading/incremental_scan.md new file mode 100644 index 0000000000..57cf6f1f82 --- /dev/null +++ b/docs/user-guide/src/reading/incremental_scan.md @@ -0,0 +1,291 @@ +# Incremental scans + +To advance a cached file listing from one table version to a newer one without replaying the +whole log, you use an incremental scan. It streams only the file-level changes (added and +removed files) between a version you already know about and a newer `Snapshot`, so you can +patch your cache instead of rebuilding it from scratch. + +Before reading this page, make sure you understand [Building a Scan](./building_a_scan.md). + +An incremental scan is the right tool when you maintain a long-lived, cached listing of a +table's active files and poll for updates. If you only need the current file list once, use a +full scan (see [Building a Scan](./building_a_scan.md)). If you need row-level changes (the +individual inserts, updates, and deletes), use [Change Data Feed](./change_data_feed.md) +instead. An incremental scan reports file-level changes only. + +## The range model + +An incremental scan covers the half-open range `(base_version, target_version]`: + +- `base_version` is the version you already have cached. It is **excluded**. +- `target_version` is the version of the `Snapshot` you build the scan from. It is + **included**. + +So `base_version` is the last version whose state your cache reflects, and the scan tells you +what changed to reach the target `Snapshot`. The range must be non-empty: +`base_version < target_version`, or `build()` returns an error. + +## Building the scan + +Call `incremental_scan_builder(base_version)` on a `Snapshot`, then `build(engine)`: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +// The cache already reflects version 5. +let base_version = 5; + +// Build a Snapshot at the target version you want to advance to. +let target = Snapshot::builder_for(url).build(&engine)?; + +// Scan the range (5, target.version()]. +let maybe_stream = target + .incremental_scan_builder(base_version) + .build(&engine)?; +# Ok(()) +# } +``` + +`build()` returns `Option`, not the stream directly. The next section +explains why. + +### Handle `None`: fall back to a full scan + +`build()` returns `None` when the target `Snapshot` doesn't retain commit `base_version + 1`, +the first version in the range. Typically the `Snapshot` loaded from a checkpoint above +`base_version`, so its retained commit list begins past the first version you need. The commit +files may still exist on storage; the `Snapshot` just doesn't retain them. `None` is not an +error. It's the signal that an incremental advance isn't possible, so you rebuild your cache +from a full scan: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +# use delta_kernel::incremental_scan::IncrementalScanStream; +# fn consume(_: IncrementalScanStream) {} +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +# let base_version = 5; +# let target = Snapshot::builder_for(url).build(&engine)?; +match target.clone().incremental_scan_builder(base_version).build(&engine)? { + Some(stream) => { + // The range is covered. Consume the diff and patch your cache. + consume(stream); + } + None => { + // The Snapshot no longer retains commits back to the base. Rebuild from a full scan. + let _scan = target.scan_builder().build()?; + } +} +# Ok(()) +# } +``` + +> [!WARNING] +> Never unwrap the `Option`. `None` is a normal outcome whenever the `Snapshot` no longer +> retains commits back to your `base_version`. Treating it as an error strands your cache with +> no way to advance. + +## Consuming the stream + +`IncrementalScanStream` yields the live added files as `FilteredEngineData` batches, newest +commit first. A `FilteredEngineData` pairs a data batch with a selection vector, so respect +the selection vector when reading. Each batch is a raw commit action batch, one row per action. +Only the rows the selection vector marks active are live Adds. Every other row is present but +inactive: Removes, Adds that a later commit in the range cancelled, and non-file actions. + +Each batch corresponds to a source commit's surviving Adds. A commit whose Adds were all +cancelled by a later Remove in the range produces no batch at all, and one commit's rows can +span more than one batch. + +After you've read the batches you care about, call a terminal method to recover the file-key +sets that describe the diff. `into_summary()` drains any batches you didn't read and returns +the live added and removed file keys: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +# let base_version = 5; +# let target = Snapshot::builder_for(url).build(&engine)?; +# let Some(stream) = target.incremental_scan_builder(base_version).build(&engine)? else { +# return Ok(()); +# }; +let mut added_row_count = 0; +let mut stream = stream; +for batch in stream.by_ref() { + let batch = batch?; + // Only the rows selected by the selection vector are live Adds. + added_row_count += batch.selection_vector().iter().filter(|&&s| s).count(); +} + +// Recover the file-key sets after reading the batches. +let summary = stream.into_summary()?; +println!( + "range ({}, {}]: {} live adds, {} removes, {} added rows", + summary.base_version, + summary.target_version, + summary.live_adds.len(), + summary.removes.len(), + added_row_count, +); +# Ok(()) +# } +``` + +If you'd rather buffer the whole diff at once, `into_listing()` collects every Add batch into +a `Vec` alongside the summary. Use it when the diff fits comfortably in memory. + +### File keys identify files by path _and_ deletion vector + +The `live_adds` and `removes` sets hold `FileActionKey` values, each a `(path, dv_unique_id)` +pair. **Delta tables use deletion vectors to mark rows as deleted without rewriting the data +file**, so the same path can appear with different deletion-vector ids. Match on the whole key, +never on the path alone. + +For example, replacing a file's deletion vector adds `(P, dv=new)` and removes `(P, dv=old)`. +Those are two distinct logical files. Collapsing them to the path `P` would treat the add and +the remove as the same file and corrupt your listing. + +## Advancing a cached listing + +When your cache is a listing layered on top of a base version, an incremental scan gives you +exactly the delta to apply. Append the streamed Add batches to your delta layer, then mask the +base with the files that are no longer current. + +The mask is the union of two sets, so use `into_summary_against_base_closure` (or its `_iter` +variant) rather than `into_summary`. That terminal method takes your base listing and returns +both `removes` and `duplicate_adds`: + +- `removes` masks files that left the table. +- `duplicate_adds` masks the stale base entry of each file the range re-added with new + metadata. Compaction (OPTIMIZE) and liquid clustering re-tag files this way: the file key is + already in your base, and the range re-adds it, so the base's old row must be evicted in + favor of the freshly streamed one. + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::collections::HashSet; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +# use delta_kernel::log_replay::FileActionKey; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +# let base_version = 5; +# let target = Snapshot::builder_for(url).build(&engine)?; +# let Some(stream) = target.incremental_scan_builder(base_version).build(&engine)? else { +# return Ok(()); +# }; +// Your cache exposes a `contains` lookup keyed by (path, dv_unique_id). +let base: HashSet = HashSet::new(); + +let summary = stream.into_summary_against_base_closure(|key| base.contains(key))?; + +// Evict every base entry that either left the table or was re-added with new metadata. +let mask: HashSet<&FileActionKey> = + summary.removes.iter().chain(summary.duplicate_adds.iter()).collect(); +# let _ = mask; +# Ok(()) +# } +``` + +> [!WARNING] +> Apply both `removes` and `duplicate_adds` to the base. Dropping `duplicate_adds` leaves stale +> rows in your listing for every file that compaction or clustering re-tagged, so your cache +> keeps pointing at data files that are no longer current. + +> [!TIP] +> Pick the terminal variant by your cache's shape. Use `_closure` when your cache supports a +> `contains` lookup (`HashSet`, `HashMap`, `BTreeMap`, or a sorted `Vec` with `binary_search`). +> It probes the live-Add set, which is usually much smaller than the base. Use `_iter` only +> when your cache can't do membership lookup and can only stream its keys. + +## Correctness requirements + +A few properties of the stream, if violated, produce a silently wrong listing rather than an +error. + +### Errors are sticky + +Once `next()` yields an error, the stream is dead. Every later call returns `None`, and the +terminal methods return an error instead of a partial summary. The dedup state is incomplete +after an error, so a partial result would be wrong. To retry, drop the stream and build a fresh +one. + +### A vacuumed commit means fall back to a full scan + +If consumption fails with a file-not-found error, a commit in the range was vacuumed between +`build()` and draining the stream. Build a fresh incremental scan. It will most likely return +`None` (the commits are no longer available), which is your signal to fall back to a full scan. + +### Catalog-managed tables need the log tail on the `Snapshot` + +For **catalog-managed tables** (tables whose commits a catalog ratifies), the incremental scan +walks the `Snapshot`'s already-validated commit list and deliberately does not re-list the +`_delta_log/` directory. Staged commits that the catalog has ratified but not yet published +appear only if you passed them to the `Snapshot` when you built it. Build the target `Snapshot` +with the catalog's log tail so the range includes those commits. A fresh storage listing would +miss them. See [Reading Catalog-Managed Tables](../catalog_managed/reading.md). + +## Current limitations + +The incremental scan is narrower than a full scan. Know these limits before you build on it. + +### No predicate or column pushdown + +Unlike `scan_builder`, the incremental scan builder takes no predicate and no projection. It +reports every added and removed file in the range, so you can't push data skipping or column +selection into it. Apply any file-level or row-level filtering in your connector after the scan +returns. + +### In-range metadata changes aren't surfaced + +`build()` validates reader features against the target `Snapshot` only. If the table's schema, +column-mapping mode, or reader features changed _within_ the range, the scan doesn't tell you. +Kernel decodes only the file path and deletion-vector fields, and those stay correct across +such a change. But if your connector reads pass-through fields off each row (`stats`, +`partitionValues`, `baseRowId`), interpret each row against the protocol at that row's commit +version, not against the target `Snapshot`. Surfacing in-range metadata changes is tracked in +[issue #2552](https://github.com/delta-io/delta-kernel-rs/issues/2552). + +> [!WARNING] +> If the range may span a metadata change and your connector reads pass-through fields, decode +> each row against its own commit's protocol. Decoding every row against the target `Snapshot` +> silently misreads rows written under the older metadata. + +## What's next + +- [Time Travel and Snapshot Management](./time_travel.md): building and updating the `Snapshot` + you scan from +- [Advanced Reads with scan_metadata()](./scan_metadata.md): full-scan file listing when an + incremental advance isn't possible +- [Reading Change Data Feed](./change_data_feed.md): row-level changes instead of file-level + changes + +## See also + +- [Reading Catalog-Managed Tables](../catalog_managed/reading.md): building a `Snapshot` with a + log tail for staged commits diff --git a/docs/user-guide/src/reading/parallel_scan_metadata.md b/docs/user-guide/src/reading/parallel_scan_metadata.md new file mode 100644 index 0000000000..f437281711 --- /dev/null +++ b/docs/user-guide/src/reading/parallel_scan_metadata.md @@ -0,0 +1,124 @@ +# Distributed log replay with parallel_scan_metadata() + +To distribute Delta log replay across multiple threads or machines, use +`parallel_scan_metadata()`. The standard `scan_metadata()` method processes the entire log +sequentially on a single node. For large tables with V2 checkpoints (which use sidecars or +multi-part checkpoint files), that sequential replay can become a bottleneck. +`parallel_scan_metadata()` splits replay into two phases so that the expensive checkpoint +processing can run in parallel. + +## Why two phases? + +A Delta table's transaction log consists of: + +- **Commit files** (JSON), which must be processed sequentially in version order +- **Checkpoint files**, which can be large but whose constituent parts (sidecars or + multi-part checkpoint files) can be processed independently + +`parallel_scan_metadata()` exploits this structure. The sequential phase handles commits and +the checkpoint manifest. The parallel phase then distributes the checkpoint leaf files +across workers. + +## Sequential phase + +The sequential phase processes commits and the checkpoint manifest. It returns an iterator +of `ScanMetadata` (the same type that `scan_metadata()` yields): + +```rust,ignore +use delta_kernel::scan::{ + AfterSequentialScanMetadata, ParallelScanMetadata, ParallelState, + SequentialScanMetadata, +}; + +let scan = snapshot.scan_builder().build()?; +let mut sequential = scan.parallel_scan_metadata(engine.clone())?; + +// Process the sequential phase: commits and checkpoint manifest +for result in sequential.by_ref() { + let scan_metadata = result?; + // Process scan metadata (same as scan_metadata())... +} +``` + +After exhausting the iterator, call `finish()` to find out whether a parallel phase is +needed: + +```rust,ignore +match sequential.finish()? { + AfterSequentialScanMetadata::Done => { + // All log replay completed in the sequential phase. + // No checkpoint sidecars or multi-part files to process. + } + AfterSequentialScanMetadata::Parallel { state, files } => { + // Parallel phase needed. `files` contains the checkpoint leaf files + // (sidecars or multi-part checkpoint parts) to process in parallel. + } +} +``` + +## Parallel phase + +If `finish()` returns `Parallel`, partition the `files` across workers and create a +`ParallelScanMetadata` iterator per partition: + +```rust,ignore +AfterSequentialScanMetadata::Parallel { state, files } => { + // Unbox and wrap in Arc for sharing across workers + let state = Arc::new(*state); + + // Distribute files across workers (one file per worker, or batched) + for file in files { + let parallel = ParallelScanMetadata::try_new( + engine.clone(), + state.clone(), + vec![file], + )?; + for result in parallel { + let scan_metadata = result?; + // Process scan metadata (same as the sequential phase)... + } + } +} +``` + +Each `ParallelScanMetadata` reads its assigned checkpoint files and processes them through +the shared `ParallelState`, which handles deduplication (filtering out files already seen in +the sequential phase). + +## Serializing state across the network + +For distributed engines where workers run on different machines, the `ParallelState` can be +serialized to bytes and shipped over the network: + +```rust,ignore +AfterSequentialScanMetadata::Parallel { state, files } => { + // On the driver: serialize the state to bytes after the sequential phase + let serialized_bytes = state.into_bytes()?; + + // Ship `serialized_bytes` and `files` to remote workers... + + // On each worker: reconstruct state from bytes and create a parallel iterator + let state = Arc::new(ParallelState::from_bytes(engine.as_ref(), &serialized_bytes)?); + let parallel = ParallelScanMetadata::try_new(engine.clone(), state, my_files)?; +} +``` + +> [!WARNING] +> The serialized state may only be deserialized by the same binary version of +> `delta-kernel-rs`. Using different versions for serialization and deserialization leads to +> undefined behavior. + +## When to use this + +Use `parallel_scan_metadata()` instead of `scan_metadata()` when: + +- The table uses V2 checkpoints with sidecars or multi-part checkpoint files +- Log replay is a bottleneck (large tables with many files) +- Your engine can distribute work across multiple threads or machines + +For most use cases, `scan_metadata()` is sufficient and simpler. + +## What's next + +- [Advanced reads with scan_metadata()](./scan_metadata.md) for the standard sequential API +- [Building a Scan](./building_a_scan.md) for the basics of building and executing a scan diff --git a/docs/user-guide/src/reading/scan_metadata.md b/docs/user-guide/src/reading/scan_metadata.md new file mode 100644 index 0000000000..d97a360a82 --- /dev/null +++ b/docs/user-guide/src/reading/scan_metadata.md @@ -0,0 +1,349 @@ +# Advanced reads with scan_metadata() + +To control when and how data files are read, you can use `scan.scan_metadata()` instead of +`scan.execute()`. This method gives you access to the file list and metadata without reading the +actual Parquet files, so you can decide where and how to perform the I/O. This API provides a +path for building connectors for distributed compute engines (Spark, Flink, etc.) where data reads +may happen in parallel. + +Before reading this page, make sure you understand [Building a Scan](./building_a_scan.md). + +> [!NOTE] +> `scan_metadata()` parallelizes data reads but not log replay. To parallelize log replay as +> well, see [Distributed Log Replay](./parallel_scan_metadata.md). + +## Example pattern + +The execution flow below illustrates how one might implement parallel data reads using +`scan_metadata()`: + +```mermaid +sequenceDiagram + participant D as Driver + participant W as Workers + + D->>D: 1. Create Snapshot + Scan + D->>D: 2. Call scan_metadata() + D->>D: 3. Extract ScanFiles + D->>W: 4. Distribute files to workers + W->>W: 5. Read Parquet files + W->>W: 6. Transform physical → logical + W->>W: 7. Apply deletion vectors + W->>D: 8. Return logical data +``` + +For a complete working example of multi-threaded reads, see the +[read-table-multi-threaded](https://github.com/delta-io/delta-kernel-rs/tree/main/kernel/examples/read-table-multi-threaded) +example in the kernel repository. + +## Enumerating ScanFiles + +`scan_metadata()` returns an iterator of `ScanMetadata`. Each `ScanMetadata` represents a batch of +files that need to be read for the scan. You can then use `visit_scan_files` on each `ScanMetadata` +to extract each file (represented as a `ScanFile`) in the batch. + +For example: +```rust,no_run +# extern crate delta_kernel; +# use delta_kernel::scan::state::ScanFile; +# use delta_kernel::{DeltaResult, Engine}; +# fn perform_read(_chunk: &[ScanFile]) {} +# fn example(scan: &delta_kernel::scan::Scan, engine: &dyn Engine) -> DeltaResult<()> { +fn collect_files(files: &mut Vec, file: ScanFile) { + files.push(file); +} + +let mut all_files: Vec = vec![]; +for metadata in scan.scan_metadata(engine)? { + let metadata = metadata?; + all_files = metadata.visit_scan_files(all_files, collect_files)?; +} + +// Now distribute these files across threads, tasks, or remote workers +let num_workers = 4; +let chunk_size = (all_files.len() / num_workers).max(1); +for chunk in all_files.chunks(chunk_size) { + perform_read(chunk); +} +# Ok(()) +# } +``` + +It is preferred to use `visit_scan_files` to iterate through each `ScanFile`. However if you need +raw access to the set of files as an `EngineData`, each `ScanMetadata` contains: +- **`scan_files`**: a `FilteredEngineData` with one row per file to read, plus a selection + vector indicating which rows are active (rows excluded by **data skipping** are marked + inactive, so that files which can't match the scan's predicate will be filtered out). +- **`scan_file_transforms`**: a `Vec>` with a per-file transformation + expression. If present, this expression must be applied to the physical data to produce the + correct logical output (e.g., injecting partition column values, applying column mapping). + +## ScanFile + +A `ScanFile` contains everything needed to read one data file: + +```rust,ignore +pub struct ScanFile { + pub path: String, // relative to table root + pub size: i64, // file size in bytes + pub modification_time: i64, // millis since epoch + pub stats: Option, // parsed statistics + pub dv_info: DvInfo, // deletion vector info + pub transform: Option, // physical -> logical transform + pub partition_values: HashMap, // partition column values +} +``` + +In a distributed engine, for example, you could serialize the `ScanFile` data along with +the scan's `physical_schema()` and `logical_schema()`, then ship them to workers. Or in +a multi-threaded engine, you can send them through in-memory channels. + +## Reading and transforming data + +Once a worker has a `ScanFile`, it should read the Parquet file and apply transformations to +produce logical data. + +### Reading the Parquet file + +Resolve the file path against the table root and read with the physical schema: + +```rust,ignore +let file_url = scan.table_root().join(&scan_file.path)?; +let size: u64 = scan_file.size.try_into().map_err(|_| Error::generic("negative file size"))?; +let file_meta = FileMeta::new(file_url, scan_file.modification_time, size); + +let read_results = engine + .parquet_handler() + .read_parquet_files(&[file_meta], physical_schema.clone(), None)?; +``` + +### Transforming physical data to logical + +Each file may need a per-file transform to convert the physical Parquet data into the +logical schema that the scan requested. The transform handles: + +- **Partition value injection**: partition columns are not stored in the Parquet file. + The transform adds them from the file's metadata. +- **Column mapping**: if the table uses column mapping, the transform renames physical + columns to their logical names. +- **Schema evolution**: if the file was written with an older schema, the transform fills + in missing columns with nulls. + +Use `transform_to_logical`: + +```rust,ignore +use delta_kernel::scan::state::transform_to_logical; + +let physical_schema = scan.physical_schema(); +let logical_schema = scan.logical_schema(); + +for batch in read_results { + let physical_data = batch?; + let logical_data = transform_to_logical( + &engine, + physical_data, + physical_schema, + logical_schema, + scan_file.transform.clone(), + )?; + // logical_data now has the columns the scan requested +} +``` + +> [!WARNING] +> If `ScanFile.transform` is present, you must apply it before returning data. Omitting the +> transform produces incorrect output — missing partition columns, wrong logical names, or nulls +> where data should appear. If `transform` is `None`, the physical data already matches the +> logical schema and no transformation is needed. + +### Applying deletion vectors + +Delta tables can use **deletion vectors** to mark rows as logically deleted without +rewriting entire data files. If a file has a deletion vector, you must filter out those +rows. Call `DvInfo::get_selection_vector()` on the `ScanFile.dv_info` to get a boolean mask: + +```rust,ignore +let selection_vector = scan_file + .dv_info + .get_selection_vector(&engine, scan.table_root())?; + +let filtered = if let Some(sv) = selection_vector { + logical_data.apply_selection_vector(sv)? +} else { + // No deletion vector, so all rows are valid + logical_data +}; +``` + +#### Getting deleted row indexes + +If your engine works with row indexes rather than boolean masks, `DvInfo` also provides +`get_row_indexes()`. This method returns a `Vec` containing the indexes of rows that +should be _removed_ from the result set: + +```rust,ignore +let deleted_rows = scan_file + .dv_info + .get_row_indexes(&engine, scan.table_root())?; + +if let Some(indexes) = deleted_rows { + // indexes contains the positions of deleted rows (e.g., [2, 17, 42]) + // Use these to filter rows out of the result set +} +``` + +Choose `get_selection_vector()` when your engine applies boolean masks directly (for example, +Arrow's `filter` kernel). Choose `get_row_indexes()` when your engine removes rows by +position, or when the deletion vector is sparse and you want to avoid allocating a boolean +vector with one entry per row. + +## Accessing scan schemas + +When you read Parquet files yourself (instead of using `scan.execute()`), you need to know +the schemas and predicate to pass to the Parquet reader. `Scan` exposes three methods for this. + +### Physical schema + +`Scan::physical_schema()` returns the schema of the underlying data files. This is the +schema you pass to the Parquet reader when opening files. It can differ from the logical +schema because partition columns are stored in the Delta log metadata, not in the Parquet +files themselves. + +```rust,ignore +let physical_schema = scan.physical_schema(); +// Pass this to engine.parquet_handler().read_parquet_files(...) +``` + +### Logical schema + +`Scan::logical_schema()` returns the output schema of the scan — the schema your engine +sees after all transforms have been applied. Pass this to `transform_to_logical` and +serialize it alongside `physical_schema` and each `ScanFile` when distributing work to +remote workers. + +```rust,ignore +let logical_schema = scan.logical_schema(); +// Pass this to transform_to_logical(...) +``` + +### Physical predicate + +`Scan::physical_predicate()` returns the scan's predicate rewritten in terms of physical +column names. If the table uses column mapping, logical column names in the original +predicate are translated to the physical names stored in the Parquet files. This is the +predicate you can push down into the Parquet reader for row-group filtering. + +```rust,ignore +if let Some(predicate) = scan.physical_predicate() { + // Push this predicate into the Parquet reader for row-group skipping +} +``` + +If the scan has no predicate, this returns `None`. + +> [!TIP] +> When distributing work to remote workers, serialize the physical schema, logical schema, +> physical predicate, and the `ScanFile` data together. Workers need all four to read files +> correctly. + +## Typed partition values + +Kernel reads each file's partition values from the Delta log and exposes them on its +`ScanFile` as a raw string map. `transform_to_logical` materializes them as typed columns. A +connector that assembles output rows itself instead of using that transform would otherwise need +to parse the string map per file. + +To have Kernel hand you the typed values directly, opt in with `with_partition_values`: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::scan::PartitionValuesOptions; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +# let snapshot = Snapshot::builder_for(url).build(&engine)?; +let scan = snapshot + .scan_builder() + .with_partition_values( + PartitionValuesOptions::with_struct() + .with_timestamp_timezone("America/Los_Angeles"), + ) + .build()?; +# Ok(()) +# } +``` + +Scan metadata output then gains a top-level `partitionValues_parsed` column with one typed, +nullable field per partition column (by physical name). You read it as a typed column instead +of parsing the string map per file. The raw string map is still present, so this option only +adds the typed column. + +By default, offset-less `TIMESTAMP` partition strings are interpreted in UTC. Use +`with_timestamp_timezone` with an IANA timezone or fixed offset when the reader uses another +timezone. An explicit offset in a partition value takes precedence. This setting affects typed +`scan_metadata` output, internal partition pruning, and the partition-column row transforms used +by `Scan::execute`. Incremental scans expose the raw partition-value map. For daylight-saving +transitions, ambiguous local times use the earlier instant, and nonexistent local times use the +offset from before the transition. + +> [!TIP] +> When the checkpoint already stores typed partition values, Kernel reads that column directly +> unless a reader timezone requires zoned `TIMESTAMP` values to be reinterpreted. In that case, +> Kernel reparses the raw partition map. + +## Cancelling a scan + +Log replay for a large table can read many commit and checkpoint files before it yields a single +`ScanMetadata`. If the request driving the scan goes away — a cancelled query, a dropped +connection — you can tell Kernel to stop instead of reading to the end. Attach a +`CancellationToken` to the scan with `with_cancellation_token`: + +```rust,ignore +use std::sync::Arc; +use delta_kernel::CancellationTokenRef; + +// `token` is your connector's cancellation handle, implementing `CancellationToken`. +// It is commonly a thin wrapper over your runtime's primitive (e.g. +// `tokio_util::sync::CancellationToken`). See "Cancellation-aware reads" in +// Implementing the Engine Trait for how to build one. +let token: CancellationTokenRef = my_request_token(); + +let scan = snapshot + .scan_builder() + .with_cancellation_token(token) + .build()?; + +for metadata in scan.scan_metadata(engine)? { + let metadata = metadata?; // yields Err(Error::Cancelled) once the token fires + // ... process the batch ... +} +``` + +Cancellation is cooperative and always visible as an error, so a cancelled scan can never be +mistaken for a complete one: + +- Kernel polls the token at each action-batch boundary, so replay stops between reads even if the + engine ignores the token. +- If the engine implements the [cancellation-aware read + variants](../connector/implementing_engine.md#cancellation-aware-reads), an I/O already in flight + can be interrupted rather than running to completion — this is what lets a scan stuck on one slow + file read stop promptly. +- Either way, the outcome surfaces as `Error::Cancelled`: either returned directly from + `scan_metadata()` (when the token is already cancelled before replay begins) or as the terminal + item of its iterator. It is never a short or empty result. + +Without a token, the scan is not cancellable and runs to completion as usual. Cancellation applies +to the lazy `scan_metadata()` path; [`parallel_scan_metadata()`](./parallel_scan_metadata.md) does +not support it and rejects a token rather than silently ignoring it. + +## What's next + +- [Column Selection](./column_selection.md): projecting specific columns +- [Distributed Log Replay](./parallel_scan_metadata.md): distributing log replay for large tables +- [Filter Pushdown and File Skipping](./filter_pushdown.md): predicate-based optimization diff --git a/docs/user-guide/src/reading/time_travel.md b/docs/user-guide/src/reading/time_travel.md new file mode 100644 index 0000000000..dbf24be325 --- /dev/null +++ b/docs/user-guide/src/reading/time_travel.md @@ -0,0 +1,269 @@ +# Time travel and snapshot management + +To read a Delta table at a specific version or refresh an existing snapshot to +pick up new commits, you use Kernel's `SnapshotBuilder` API. The builder +supports both full construction from a table URL and incremental updates from an +existing `Snapshot`. + +Before reading this page, make sure you understand +[Building a Scan](./building_a_scan.md). + +## Reading a table at a specific version + +Every Delta table maintains a versioned +transaction log. Each commit creates a new version. +By default, `Snapshot::builder_for` loads the latest version. To read a +specific historical version, chain `.at_version()` onto the builder. + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +// Read the table at version 5 +let snapshot = Snapshot::builder_for(&url) + .at_version(5) + .build(&engine)?; + +println!("Loaded version: {}", snapshot.version()); +# Ok(()) +# } +``` + +The `at_version` method accepts a `u64` version number. If the requested +version does not exist in the transaction log, `build` returns an error. + +To read the latest version, omit `at_version`: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +let snapshot = Snapshot::builder_for(&url) + .build(&engine)?; +# Ok(()) +# } +``` + +## Refreshing an existing snapshot + +If you already hold a `Snapshot` (which is wrapped in an `Arc` as +`SnapshotRef`) and want to check for newer commits, use +`Snapshot::builder_from` instead of rebuilding from scratch. This performs +an incremental update: Kernel reads only the new commits since the existing +snapshot's version, avoiding a full log replay. + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +// Build an initial snapshot +let snapshot = Snapshot::builder_for(&url) + .build(&engine)?; +println!("Initial version: {}", snapshot.version()); + +// Later, refresh to pick up new commits +let updated = Snapshot::builder_from(snapshot) + .build(&engine)?; +println!("Updated version: {}", updated.version()); +# Ok(()) +# } +``` + +The incremental update follows these rules: + +1. If you call `.at_version()` with the same version the existing snapshot + already holds, Kernel returns the existing snapshot without doing any work. +2. If the table has not advanced since the existing snapshot, Kernel returns + the existing snapshot. +3. If new commits exist, Kernel replays only the commits after the existing + snapshot's version. +4. You cannot refresh backward. Requesting a version older than the existing + snapshot produces an error. + +You can also refresh to a specific newer version by combining `builder_from` +with `at_version`: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +# let snapshot = Snapshot::builder_for(&url).at_version(3).build(&engine)?; +// Refresh from version 3 to exactly version 7 +let updated = Snapshot::builder_from(snapshot) + .at_version(7) + .build(&engine)?; +# Ok(()) +# } +``` + +## Getting the timestamp of a version + +`Snapshot` exposes `get_timestamp`, which returns the timestamp of the +snapshot's version in milliseconds since the Unix epoch. When the table has +In-Commit Timestamps (ICT) enabled, the method returns the ICT value stored +inside the commit. Otherwise, it falls back to the filesystem's last-modified +time on the commit file. + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +let snapshot = Snapshot::builder_for(&url) + .at_version(5) + .build(&engine)?; + +let timestamp_ms = snapshot.get_timestamp(&engine)?; +println!("Version {} was committed at {} ms", snapshot.version(), timestamp_ms); +# Ok(()) +# } +``` + +> [!NOTE] +> `get_timestamp` requires an `&dyn Engine` because it may need to read the +> commit file from storage when In-Commit Timestamps are enabled. + +## Resolving timestamps to versions + +To time travel by timestamp rather than by version number, use the +`history_manager` module. It exposes three helpers that translate timestamps +(in milliseconds since the Unix epoch) into the version numbers you can pass +to `at_version`. + +| Function | Returns | +|----------|---------| +| `latest_version_as_of(snapshot, engine, timestamp, resolved_commit_type)` | The `Commit` for the latest version with a timestamp at or before `timestamp`. | +| `first_version_after(snapshot, engine, timestamp, resolved_commit_type)` | The `Commit` for the first version with a timestamp at or after `timestamp`. | +| `timestamp_range_to_versions(snapshot, engine, start, end)` | A `(start_version, end_version)` pair covering the timestamp range. | + +Each helper takes a `Snapshot` reference, which defines the searchable +version range. You typically pass the latest snapshot so the search covers +the table's full history. `timestamp_range_to_versions` is the most +common of the three because it translates a `[start_ts, end_ts]` window +into the version range that change data feed (CDF) queries need to read. + +The helper function also take a `resolved_commit_type`, controlling which +versions are eligible. Pass `HistoryCommitType::Recreatable` to restrict +the result to versions whose table state you can actually load. +Pass `HistoryCommitType::Published` to allow any version that still has a commit file. + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +# use delta_kernel::history_manager::{latest_version_as_of, HistoryCommitType}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +// 1. Load the latest snapshot to define the search range. +let latest = Snapshot::builder_for(&url).build(&engine)?; + +// 2. Resolve a timestamp (Jan 1, 2024 UTC) to a matching commit. +let timestamp_ms = 1_704_067_200_000; +let commit = latest_version_as_of(&latest, &engine, timestamp_ms, HistoryCommitType::Recreatable)?; + +// 3. Time travel to the resolved version. +let snapshot = Snapshot::builder_for(&url) + .at_version(commit.version) + .build(&engine)?; +println!( + "Resolved timestamp {timestamp_ms} to version {} (committed at {} ms)", + commit.version, commit.timestamp +); +# Ok(()) +# } +``` + +`first_version_after` is the symmetric variant. It returns a `CommitAt` for the +earliest version whose timestamp is at or after the requested timestamp, which +is useful for picking up changes that happened after a known point in time. + +To resolve a timestamp range, call `timestamp_range_to_versions`. This is +the primary entry point for CDF queries, which need to translate a user's +timestamp window into the start and end versions to read. The end +timestamp is optional. Pass `None` to indicate no upper bound. + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +use delta_kernel::history_manager::timestamp_range_to_versions; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let store = store_from_url(&url)?; +# let engine = DefaultEngine::builder(store).build(); +# let latest = Snapshot::builder_for(&url).build(&engine)?; +let start_ms = 1_704_067_200_000; // Jan 1, 2024 UTC +let end_ms = 1_706_745_600_000; // Feb 1, 2024 UTC + +let (start_version, end_version) = + timestamp_range_to_versions(&latest, &engine, start_ms, Some(end_ms))?; +# Ok(()) +# } +``` + +These helpers return errors when the timestamp falls outside the range of +commits visible from the snapshot, or when the requested range is empty. +See the `LogHistoryError` variants for the specific failure modes. + +## When to use `builder_for` vs. `builder_from` + +| Scenario | Method | Why | +|----------|--------|-----| +| First read of a table | `Snapshot::builder_for(url)` | You have no existing snapshot to update from. | +| Time travel to a known version | `Snapshot::builder_for(url).at_version(v)` | You want a specific historical version and have no nearby snapshot. | +| Polling for new commits | `Snapshot::builder_from(existing)` | Reuses the existing snapshot's state. Kernel reads only new commits. | +| Advancing to a specific newer version | `Snapshot::builder_from(existing).at_version(v)` | Combines incremental update with a target version. | + +The key difference is cost. `builder_for` replays the transaction log from the +most recent checkpoint. `builder_from` replays only the commits after the +existing snapshot's version. For long-lived connectors that periodically check +for updates, `builder_from` avoids redundant log replay. + +## What's next + +- [Column Selection](./column_selection.md) to read specific columns from a + historical snapshot. +- [Filter Pushdown and File Skipping](./filter_pushdown.md) to apply predicates + that reduce I/O. +- [Advanced Reads with scan_metadata()](./scan_metadata.md) for distributed + scan execution. diff --git a/docs/user-guide/src/storage/configuring_storage.md b/docs/user-guide/src/storage/configuring_storage.md new file mode 100644 index 0000000000..1b12116e9d --- /dev/null +++ b/docs/user-guide/src/storage/configuring_storage.md @@ -0,0 +1,128 @@ +# Configuring storage + +To configure how `DefaultEngine` accesses your Delta tables, you create an object store +from a URL and pass it to the engine builder. The `DefaultEngine` uses the +[`object_store`](https://docs.rs/object_store) crate for all storage I/O, supporting +local files, S3, GCS, and Azure out of the box. + +Before reading this page, make sure you understand +[The Engine Trait](../concepts/engine_trait.md). + +> [!NOTE] +> The storage APIs on this page come from the `delta_kernel_default_engine` crate. Add it +> to your `Cargo.toml` with either the `rustls` or `native-tls` feature. +> See [Feature Flags](../concepts/feature_flags.md) for details. + +Kernel provides several paths to construct an object store, depending on how +much control you need: + +```mermaid +flowchart TD + A[Need a DefaultEngine] --> B{How do you get
    your ObjectStore?} + B -->|"Standard URL"| C["store_from_url(&url)"] + B -->|"URL + options"| D["store_from_url_opts(&url, opts)"] + B -->|"Custom URL scheme"| E["insert_url_handler(scheme, handler)\nthen store_from_url"] + B -->|"Bring your own"| F["Build ObjectStore directly"] + C --> G["Arc#lt;dyn ObjectStore#gt;"] + D --> G + E --> G + F --> G + G --> H["DefaultEngine::builder(store).build()"] + + click C href "#standard-url" + click D href "#url-with-options" + click E href "#custom-url-schemes" + click F href "#bringing-your-own-object-store" +``` + +## Standard URL + +`store_from_url` creates an object store from a URL. The `object_store` crate detects +the storage backend from the URL scheme: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# extern crate url; +# use std::sync::Arc; +# use url::Url; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::DeltaResult; +# fn main() -> DeltaResult<()> { +let url = Url::parse("file:///path/to/table")?; +let store = store_from_url(&url)?; +let engine = DefaultEngine::builder(store).build(); +# Ok(()) +# } +``` + +## URL with options + +To pass provider-specific options (credentials, region, endpoint, etc.), use +`store_from_url_opts`. These options are forwarded directly to the `object_store` crate: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# extern crate url; +# use std::collections::HashMap; +# use url::Url; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url_opts; +# use delta_kernel::DeltaResult; +# fn main() -> DeltaResult<()> { +let url = Url::parse("s3://my-bucket/path/to/table")?; +let options = HashMap::from([ + ("region", "us-west-2"), + ("access_key_id", "AKIA..."), + ("secret_access_key", "..."), +]); +let store = store_from_url_opts(&url, options)?; +let engine = DefaultEngine::builder(store).build(); +# Ok(()) +# } +``` + +See the [`object_store` documentation](https://docs.rs/object_store) for the full list +of supported options per storage provider. + +## Custom URL schemes + +If you need to support a URL scheme that `object_store` doesn't handle natively (e.g. +`hdfs://`), register a handler with `insert_url_handler`: + +```rust,ignore +use std::sync::Arc; +use delta_kernel_default_engine::storage::insert_url_handler; + +insert_url_handler("hdfs", Arc::new(|url, options| { + // Build your custom ObjectStore from the URL and options + let store = build_hdfs_store(url, &options)?; + let path = object_store::path::Path::parse(url.path())?; + Ok((Box::new(store), path)) +}))?; + +// Now store_from_url uses your handler for hdfs:// URLs +let store = store_from_url(&url)?; +``` + +The handler closure receives a `&Url` and a `HashMap` of options, and +returns a `Result<(Box, Path), Error>`. + +## Bringing your own object store + +To bypass URL-based construction entirely, build an `ObjectStore` instance directly and +pass it to the engine builder: + +```rust,ignore +use std::sync::Arc; +use object_store::local::LocalFileSystem; +use delta_kernel_default_engine::DefaultEngine; + +let store = Arc::new(LocalFileSystem::new()); +let engine = DefaultEngine::builder(store).build(); +``` + +This is useful when you need full control over the store configuration or want to use a +store implementation that isn't reachable via URL parsing. diff --git a/docs/user-guide/src/unity_catalog/creating_tables.md b/docs/user-guide/src/unity_catalog/creating_tables.md new file mode 100644 index 0000000000..e42967df0e --- /dev/null +++ b/docs/user-guide/src/unity_catalog/creating_tables.md @@ -0,0 +1,186 @@ +# Creating Unity Catalog tables + + + + +To create a new Unity Catalog-managed Delta table, you register the table with +your UC server to obtain a table ID and storage location, write a version 0 +Delta log commit with the required catalog-managed properties, and then send a +second set of properties back to UC to finalize registration. + +Before reading this page, make sure you understand +[Creating a Table](../writing/create_table.md) and the +[Unity Catalog Integration overview](./overview.md). + +## Prerequisites + +- A three-part table name, Delta schema, and target storage location. +- A `UCClient` (from `unity-catalog-delta-rest-client`). + +## Step 1: Reserve the table in Unity Catalog + +Call `UCClient::create_staging_table`. UC allocates a table ID and storage location and returns +temporary credentials for the initial commit. + +```rust,ignore +use unity_catalog_delta_client_api::CreateStagingTableRequest; + +let staging_info = uc_client + .create_staging_table("my_catalog", "my_schema", CreateStagingTableRequest { + name: "my_table".to_string(), + }) + .await?; +let table_id = staging_info.table_id; +let table_uri = staging_info.location; +``` + +## Step 2: Collect the disk-bound properties + +```rust,ignore +use delta_kernel_unity_catalog::get_required_properties_for_disk; + +let disk_props = get_required_properties_for_disk(&table_id); +``` + +The returned map has exactly eight entries: four `delta.feature.*` signals that enable the required +table features, three companion config properties that kernel does not write itself, and the UC +table ID. + +| Key | Value | +|-----|-------| +| `delta.feature.catalogManaged` | `supported` | +| `delta.feature.vacuumProtocolCheck` | `supported` | +| `delta.feature.v2Checkpoint` | `supported` | +| `delta.feature.deletionVectors` | `supported` | +| `delta.enableDeletionVectors` | `true` | +| `delta.checkpoint.writeStatsAsStruct` | `true` | +| `delta.checkpoint.writeStatsAsJson` | `true` | +| `io.unitycatalog.tableId` | the UC-assigned table ID | + +> [!NOTE] +> The map intentionally omits the `inCommitTimestamp` feature and +> `delta.enableInCommitTimestamps=true`. Kernel's `create_table()` auto-enables +> both when it sees the `catalogManaged` feature. + +## Step 3: Build and commit the version 0 transaction + +```rust,ignore +use std::sync::Arc; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::transaction::CommitResult; +use delta_kernel_unity_catalog::UCCommitter; +use unity_catalog_delta_client_api::{Operation, TableIdentifier}; +use unity_catalog_delta_rest_client::{ClientConfig, UCClient, UCUpdateTableRestClient}; + +let config = ClientConfig::build(&endpoint, &token).with_additional_user_agent([("MyConnector", "1.0")]).build()?; +let update_client = Arc::new(UCUpdateTableRestClient::new(config)?); + +// Build the engine over the staging storage location using the staging credentials +// (`staging_info.storage_credentials`). `build_engine_with_credentials` is a connector-owned +// helper; see Step 4 of [Reading UC Tables](./reading.md#step-4-build-an-engine-with-vended-credentials). +let engine = build_engine_with_credentials(&table_uri, &staging_info.storage_credentials)?; + +// Build the create-table transaction with the disk-bound properties. `UCCommitter::new` takes the +// commit client plus the table's UC-assigned id and its three-part name. +let committer = Box::new(UCCommitter::new( + update_client.clone(), + table_id.clone(), + TableIdentifier::new("main", "default", "my_table"), +)); +let create_txn = create_table(table_uri.as_str(), Arc::new(schema), "MyApp/1.0") + .with_table_properties(disk_props) + .build(&engine, committer)?; + +let post_commit_snapshot = match create_txn.commit(&engine)? { + CommitResult::CommittedTransaction(committed) => committed + .post_commit_snapshot() + .cloned() + .expect("post-commit snapshot is always populated for create table"), + CommitResult::ConflictedTransaction(_) => { + // Another writer created the table first. Delete the UC reservation + // and fail, or fall through to read the existing table. + return Err("table already exists".into()); + } + CommitResult::RetryableTransaction(_) => { + return Err("version 0 commit failed with a transient error; retry".into()); + } +}; +``` + +On version 0, `UCCommitter` writes `_delta_log/00000000000000000000.json` +directly and skips the `update_table` API. + +## Step 4: Build the create-table request for UC + +`build_uc_create_table_request` reads the post-commit version 0 snapshot and produces a typed +`CreateTableRequest` to send to UC's create-table endpoint. Each part of the request maps to a +distinct typed field, so the same information is never duplicated across fields: + +```rust,ignore +use delta_kernel_unity_catalog::build_uc_create_table_request; + +let create_req = build_uc_create_table_request(&post_commit_snapshot, &engine, "my_table")?; +``` + +- `columns` carries the serialized table schema, and `partition_columns` the partition column names. +- `protocol` is a typed field holding the min reader and writer versions and the reader and writer + feature names (for a freshly created UC table this is at least `catalogManaged`, + `vacuumProtocolCheck`, and `inCommitTimestamp`). Features are not flattened into `properties`. +- `domain_metadata` carries the `delta.clustering` and `delta.rowTracking` domains verbatim when + present. UC ignores any other domain. +- `properties` is the table's metadata configuration as-is, such as `io.unitycatalog.tableId`, + `delta.enableInCommitTimestamps`, and any user-supplied custom properties. Protocol and clustering + data live in their own fields above, not here. + +> [!NOTE] +> `build_uc_create_table_request` requires a version 0 snapshot with an in-commit timestamp. The +> `post_commit_snapshot` from Step 3 satisfies both. +> +> The companion config properties from `get_required_properties_for_disk` +> (`delta.enableDeletionVectors`, `delta.checkpoint.writeStatsAsStruct`, +> `delta.checkpoint.writeStatsAsJson`) round-trip from the committed metadata into `properties` +> automatically. `build_uc_create_table_request` also sets `delta.checkpointPolicy=v2` in the request +> body, which kernel's CREATE TABLE rejects as a table property and so cannot round-trip from disk. + +## Step 5: Finalize the table in Unity Catalog + +Call `UCClient::create_table` with the request from Step 4 to register the table. It returns the +registered table as a `LoadTableResponse`. + +```rust,ignore +uc_client + .create_table("my_catalog", "my_schema", create_req) + .await?; +``` + +## Clustered tables + +Chain `with_data_layout` on the create-table builder: + +```rust,ignore +use delta_kernel::transaction::data_layout::DataLayout; + +let create_txn = create_table(table_uri.as_str(), Arc::new(schema), "MyApp/1.0") + .with_table_properties(disk_props) + .with_data_layout(DataLayout::clustered(["region"])) + .build(&engine, committer)?; +``` + +`build_uc_create_table_request` forwards the committed `delta.clustering` domain verbatim into the +request's domain metadata. Its `clusteringColumns` paths are physical column names when column +mapping is enabled, matching what the table committed. + +## What's next + +- [Writing to UC Tables](./writing.md) for the version >= 1 write flow that + follows table creation. +- [Reading UC Tables](./reading.md) for the read flow. +- [Creating a Table](../writing/create_table.md) for the generic Kernel + create-table APIs. + +## See also + +- [Catalog-Managed Tables: Overview](../catalog_managed/overview.md) for + staged / ratified / published commit terminology. +- [Implementing a Catalog Committer](../catalog_managed/committer.md) for the + `Committer` trait behind `UCCommitter`. diff --git a/docs/user-guide/src/unity_catalog/overview.md b/docs/user-guide/src/unity_catalog/overview.md new file mode 100644 index 0000000000..c79e3e6558 --- /dev/null +++ b/docs/user-guide/src/unity_catalog/overview.md @@ -0,0 +1,294 @@ +# Unity Catalog Integration + + + +**Unity Catalog (UC) integration** is a set of crates that connect Delta Kernel +to [Unity Catalog](https://www.unitycatalog.io/), a multi-engine governance +layer for data and AI assets. This matters because UC-managed Delta tables +require all reads and writes to go through the catalog, and these crates handle +that coordination so your connector doesn't have to implement the UC REST +protocol from scratch. + +Before reading this page, make sure you understand the general concepts in +[Catalog-Managed Tables: Overview](../catalog_managed/overview.md). + +## What Unity Catalog provides + +Unity Catalog acts as the source of truth for catalog-managed tables. When a +table has the `catalogManaged` table feature enabled, your connector can no +longer read or write it by accessing the +transaction log on disk alone. Instead, the +connector must: + +1. **Load** the table's metadata and recent commits from UC in one call. These + commits may not yet be published to disk. +2. **Obtain credentials** from UC to access the table's cloud storage. +3. **Commit through UC** rather than writing directly to `_delta_log/`. + +The UC integration crates handle steps 1 through 3 while Kernel handles +everything else: log replay, data skipping, schema enforcement, and protocol +compliance. + +## The three crates + +The UC integration is split across three crates, each with a distinct +responsibility. + +### `unity-catalog-delta-client-api`: transport-agnostic traits + +This crate defines the **API contract** for communicating with Unity Catalog. It +contains no HTTP code or network dependencies. The key types are: + +- **`UpdateTableClient`** trait: commits a new version to a UC-managed table. + Your implementation calls the UC commits API to ratify a staged commit. +- **`LoadTableResponse`** / **`TableMetadata`** / **`Commit`**: the response + models for `load_table`. A single response carries the table metadata, the + inline commits that form the log tail, and the latest ratified version. +- **`CredentialsResponse`** / **`StorageCredential`** / **`Operation`**: + credential vending models. Each `StorageCredential` is scoped to a storage + `prefix` and carries a `config` map of cloud-specific key-value pairs. + `Operation` distinguishes `Read` and `ReadWrite` access. +- **`InMemoryUpdateTableClient`**: a test-only implementation (behind the + `test-utils` feature flag) that stores commits in memory. Useful for unit + testing your connector without a live UC server. + +Because this crate is transport-agnostic, you can swap in any backend (REST, +gRPC, or in-memory) without changing the code that depends on these traits. + +### `unity-catalog-delta-rest-client`: HTTP implementation + +This crate provides the concrete REST-over-HTTP implementations: + +- **`UCClient`**: calls the UC Delta-Tables APIs. `load_table` returns a table's + metadata plus its inline log tail in one call, `get_table_credentials` vends + temporary cloud credentials scoped to the table's storage, and + `create_staging_table` / `create_table` reserve and register a new + catalog-managed table. +- **`UCUpdateTableRestClient`**: implements `UpdateTableClient` over HTTP. It + submits staged commits to the UC `update_table` endpoint, where Unity Catalog + ratifies them. +- **`ClientConfig`** / **`ClientConfigBuilder`**: configuration for the HTTP + clients, including the workspace URL and authentication token. + +### `delta-kernel-unity-catalog`: the Kernel integration layer + +This crate connects the UC client layer to Kernel's APIs. It depends on both +`unity-catalog-delta-client-api` and `delta_kernel`. The key items are: + +- **`snapshot_builder_from_load_table()`**: turns a `load_table` response into a + `SnapshotBuilder` with the log tail and catalog version already applied. You + call `build()` on the result (or `at_version()` first, to time-travel) to get a + Snapshot that includes unpublished commits. +- **`log_tail_from_commits()`**: converts the inline commits from a `load_table` + response into a `Vec` log tail. Use it directly when you need to assemble + the `SnapshotBuilder` yourself. +- **`UCCommitter`**: implements Kernel's `Committer` trait + for UC tables. Version 0 (table creation) writes `00000000000000000000.json` + directly and skips `update_table`. For version >= 1, it writes a staged commit + to `_delta_log/_staged_commits/`, then calls the UC `update_table` API to ratify + it. The `publish()` method copies ratified staged commits to `_delta_log/` as + published commits. +- **`get_required_properties_for_disk()`**: returns the table properties you + must include when creating a UC-managed table (the `catalogManaged`, + `vacuumProtocolCheck`, `v2Checkpoint`, and `deletionVectors` feature signals, + the companion config properties kernel does not write itself, plus the + `io.unitycatalog.tableId`). Kernel's `create_table()` consumes these as table + properties on the version 0 commit. +- **`build_uc_create_table_request()`**: reads the post-creation version 0 + Snapshot and produces a typed `CreateTableRequest` whose schema, partition + columns, protocol, domain metadata, and metadata-config properties are separate + typed fields. You send it to your UC server's table-registration endpoint to + finalize the table. + +See [Creating UC Tables](./creating_tables.md) for the end-to-end creation +flow and how these two utilities fit together. + +> [!NOTE] +> `UCCommitter` requires a multi-threaded tokio runtime. The default Kernel +> Engine uses tokio, so this is compatible. If you use a custom Engine, ensure +> your runtime is multi-threaded. + +## How the crates map to catalog-managed concepts + +The [Catalog-Managed Tables: Overview](../catalog_managed/overview.md) +describes the generic architecture: a catalog client-side component that +resolves tables, fetches commits, and provides a `Committer`. Here is how the +UC crates fill those roles: + +| Generic concept | UC implementation | +|-----------------|-------------------| +| Load table metadata + log tail | `UCClient::load_table()` | +| Obtain storage credentials | `UCClient::get_table_credentials()` | +| Build Snapshot with catalog commits | `snapshot_builder_from_load_table()`, then `build()` | +| Commit through catalog | `UCCommitter` implements `Committer`: stages a commit, submits it via `UpdateTableClient::update_table()` for UC to ratify, then publishes | +| Publish staged commits | `UCCommitter::publish()` copies staged files to `_delta_log/` | + +## Architecture + +The following diagram shows how data flows through the three crates when your +connector reads or writes a UC-managed table. + +```text + ┌───────────────────────────────────────────────────────────┐ + │ Your Connector │ + │ │ + │ 1. UCClient::load_table("cat", "schema", "table") │ + │ 2. UCClient::get_table_credentials(.., ReadWrite) │ + │ 3. snapshot_builder_from_load_table(&resp)?.build(..) │ + │ 4. snapshot.transaction(UCCommitter).commit(engine)? │ + └──────┬─────────────────────────┬──────────────────────────┘ + │ │ + ▼ ▼ + ┌──────────────────┐ ┌─────────────────────────────────────┐ + │ unity-catalog- │ │ delta-kernel-unity-catalog │ + │ delta-rest- │ │ │ + │ client │ │ snapshot_builder_from_load_table() │ + │ │ │ load_table resp -> builder │ + │ UCClient │ │ │ + │ load_table │ │ UCCommitter │ + │ get_table_ │ │ implements Committer trait │ + │ credentials │ │ stages a commit, submits via │ + │ UCUpdateTable │ │ update_table() for UC to ratify │ + │ RestClient │ │ │ + │ impl UpdateTable│ │ │ + │ Client │ │ │ + └──────┬───────────┘ └──────────┬──────────────────────────┘ + │ │ + ▼ ▼ + ┌──────────────────┐ ┌─────────────────────────────────────┐ + │ unity-catalog- │ │ delta_kernel │ + │ delta-client- │ │ │ + │ api │ │ Snapshot, Scan, Transaction │ + │ │ │ Committer trait │ + │ UpdateTable │ │ LogPath, SnapshotBuilder │ + │ Client (trait)│ │ │ + │ wire models │ │ Knows nothing about UC. │ + └──────────────────┘ └─────────────────────────────────────┘ +``` + +The diagram shows the steady-state commit flow for an existing table. Version 0 +(table creation) takes a different path: `UCCommitter` writes the published commit +directly and skips `update_table`. See [Creating UC Tables](./creating_tables.md) +for the creation flow. + +## Dependencies and feature flags + +To use the UC integration, add the following to your `Cargo.toml`: + +```toml +[dependencies] +delta-kernel-unity-catalog = { version = "..." } +unity-catalog-delta-rest-client = { version = "..." } +``` + +Depend on `unity-catalog-delta-client-api` whenever you import types from it +directly (including `Operation` and `UpdateTableClient`). The +REST client crate does not re-export these. You also need the client-api crate +when implementing a custom backend, such as a gRPC client. + +The `delta-kernel-unity-catalog` crate has the following feature flags: + +| Feature | Default | Description | +|---------|---------|-------------| +| `arrow` | Yes | Enables Arrow integration (currently delegates to `arrow-59`) | +| `arrow-59` | Via `arrow` | Uses Arrow version 59 | +| `arrow-58` | No | Uses Arrow version 58 | + +The `unity-catalog-delta-client-api` crate has one feature flag: + +| Feature | Default | Description | +|---------|---------|-------------| +| `test-utils` | No | Enables `InMemoryUpdateTableClient` for unit testing | + +> [!TIP] +> The `unity-catalog-delta-rest-client` crate also exposes a `test-utils` +> feature that enables the `test-utils` feature on the client API crate +> transitively. Add it to your `[dev-dependencies]` to get the in-memory +> client for tests. + +## Identifying your client (User-Agent) + +Some catalogs allowlist requests by `User-Agent` and reject callers they don't +recognize. The client always identifies itself as +`Unity-Catalog-Delta-Rest-Rust-Client/`. Use `with_additional_user_agent` +to add the other versions relevant to your setup. Providing them is voluntary, +and which ones apply depends on the caller. + +Versions worth including, when they apply to your setup: + +- **compute engine**: the external query system, if any (e.g. Spark, Flink). +- **connector**: your code integrating an engine with Delta and UC. +- **client**: `Unity-Catalog-Delta-Rest-Rust-Client`, added for you. +- **kernel**: `Delta-Kernel-Rust`, if your connector uses Kernel. + +```rust,ignore +let config = ClientConfig::build(&endpoint, &token) + .with_additional_user_agent([ + ("MyEngine", "1.0.0"), + ("MyConnector", "1.0.0"), + ("Delta-Kernel-Rust", "0.26.0"), + ]) + .build()?; +// User-Agent: Unity-Catalog-Delta-Rest-Rust-Client/ MyEngine/1.0.0 MyConnector/1.0.0 Delta-Kernel-Rust/0.26.0 +``` + +## Client configuration and retries + +`ClientConfigBuilder` exposes the following tuning knobs. The defaults are +sensible for most workloads, but long-running reads and bursty write paths +often benefit from raising timeouts or the retry budget. + +| Method | Default | Description | +|--------|---------|-------------| +| `with_timeout(Duration)` | 30 seconds | Per-request timeout for UC REST calls. | +| `with_connect_timeout(Duration)` | 10 seconds | TCP connect timeout. | +| `with_max_retries(u32)` | 3 | Maximum retry attempts for a single request. | +| `with_retry_delays(base, max)` | 500 ms base, 10 s max | Linear backoff bounds between retries. | + +```rust,ignore +use std::time::Duration; +use unity_catalog_delta_rest_client::ClientConfig; + +let config = ClientConfig::build(&endpoint, &token) + .with_timeout(Duration::from_secs(60)) + .with_max_retries(5) + .with_retry_delays(Duration::from_millis(200), Duration::from_secs(5)) + .build()?; +``` + +The REST client automatically retries requests that fail with server errors +(HTTP 5xx) or transient network errors, using linear backoff bounded by +`retry_base_delay` and `retry_max_delay`. Successful 2xx and client errors +(HTTP 4xx) are not retried. These retries apply to transport-level failures +only. Transaction-level conflicts (another writer won the version) must be +handled by the connector through the `CommitResult::ConflictedTransaction` +branch. See [Writing to UC Tables](./writing.md) for the full retry model. + +## When not to use this + +If your tables are not registered in Unity Catalog, you don't need these crates. +Standard filesystem-managed Delta tables work with Kernel directly. See +[Building a Scan](../reading/building_a_scan.md) and +[Appending Data](../writing/append.md) for the non-catalog path. + +If you use a different catalog (Hive Metastore, AWS Glue, Polaris), you need a +different catalog client-side component. The generic +[catalog-managed overview](../catalog_managed/overview.md) explains the +extension points. + +## What's next + +- [Creating UC Tables](./creating_tables.md): how to create a new UC-managed + table using `get_required_properties_for_disk` and + `build_uc_create_table_request`. +- [Reading UC Tables](./reading.md): how to load a Snapshot and read data from + a UC-managed table. +- [Writing to UC Tables](./writing.md): how to commit and publish writes through + Unity Catalog. + +## See also + +- [Catalog-Managed Tables: Overview](../catalog_managed/overview.md): the + generic catalog-managed concepts that the UC crates implement. +- [Implementing a Catalog Committer](../catalog_managed/committer.md): how the + `Committer` trait works under the hood. diff --git a/docs/user-guide/src/unity_catalog/reading.md b/docs/user-guide/src/unity_catalog/reading.md new file mode 100644 index 0000000000..972f44d0da --- /dev/null +++ b/docs/user-guide/src/unity_catalog/reading.md @@ -0,0 +1,258 @@ +# Reading Unity Catalog tables + + + + +To read a Unity Catalog-managed Delta table, you load the table through the UC +Delta-Tables API, fetch temporary storage credentials, build a Snapshot from the +catalog's log tail, and then build a Scan exactly as you would for a +filesystem-managed table. + +Before reading this page, make sure you understand +[Catalog-Managed Tables](../catalog_managed/overview.md) and the +[Unity Catalog Integration overview](./overview.md). + +> [!NOTE] +> This example uses the `delta-kernel-unity-catalog` and +> `unity-catalog-delta-rest-client` crates, which are not part of the +> `delta_kernel` crate itself. Add them as dependencies alongside `delta_kernel`. + +## Dependencies + +Add the following to your `Cargo.toml`: + +```toml +[dependencies] +delta_kernel = "..." +delta_kernel_default_engine = { version = "...", features = ["rustls"] } +delta-kernel-unity-catalog = "..." +unity-catalog-delta-rest-client = "..." +unity-catalog-delta-client-api = "..." +url = "2" +tokio = { version = "1", features = ["full"] } +``` + +Use `rustls` on `delta_kernel_default_engine` for a pure-Rust TLS stack or +`native-tls` to link against the system's TLS implementation. You need +`unity-catalog-delta-client-api` directly to import types like `Operation`, since +the REST client crate does not re-export them. + +## Step 1: Build the UC client + +`UCClient` calls the UC Delta-Tables API. It shares a `ClientConfig` with the +rest of the UC integration. + +```rust,ignore +use unity_catalog_delta_rest_client::{ClientConfig, UCClient}; + +let config = ClientConfig::build(&endpoint, &token) + .with_additional_user_agent([("MyConnector", "1.0")]) + .build()?; +let uc_client = UCClient::new(config)?; +``` + +The `endpoint` is your UC workspace URL (e.g. `"my-workspace.cloud.databricks.com"`), +and `token` is a valid authentication token. + +## Step 2: Load the table + +Call `load_table` with the three-part table name split into catalog, schema, and +table. A single call returns the table's metadata (including its storage +`location`), the inline unpublished commits that form the log tail, and the +latest version the catalog has ratified. + +```rust,ignore +let resp = uc_client + .load_table("my_catalog", "my_schema", "my_table") + .await?; +let table_uri = url::Url::parse(&resp.metadata.location)?; +``` + +The `LoadTableResponse` also carries `commits` (the log tail, covered in Step 5) +and `latest_table_version` (the catalog's current ratified version). + +## Step 3: Fetch temporary credentials + +UC vends short-lived cloud storage credentials scoped to the table's storage +location. For a read operation, request `Operation::Read`: + +```rust,ignore +use unity_catalog_delta_client_api::Operation; + +let creds = uc_client + .get_table_credentials("my_catalog", "my_schema", "my_table", Operation::Read) + .await?; +``` + +The returned `CredentialsResponse` holds a list of `StorageCredential`s, each +scoped to a storage `prefix`. Every credential carries a `config` map whose keys +are provider-namespaced (`s3.access-key-id`, `azure.sas-token`, +`gcs.oauth-token`, and so on), so the same response shape works across clouds. + +> [!WARNING] +> Vended credentials can expire. Each `StorageCredential` carries an +> `expiration_time_ms` (or `None` when the server omits it, as it does for local +> `file://` and static credentials). If your scan outlives the credential +> lifetime, you need to fetch fresh credentials and rebuild the Engine before +> continuing. See [Credential refresh](#credential-refresh-for-long-running-operations) +> below. + +## Step 4: Build an Engine with vended credentials + +The `config` map matches the option shape `object_store` expects. Translate the +provider-namespaced keys into the `object_store` option names, then wrap the +resulting store in a `DefaultEngineBuilder`. Mapping the keys is connector-owned +work, because only your connector knows which cloud it targets. + +```rust,ignore +use std::sync::Arc; +use delta_kernel_default_engine::DefaultEngineBuilder; +use delta_kernel::object_store; + +// Map the vended `config` keys to object_store option names. +let mut options: Vec<(String, String)> = Vec::new(); +for cred in &creds.storage_credentials { + for (key, value) in &cred.config { + let mapped = match key.as_str() { + "s3.access-key-id" => "access_key_id", + "s3.secret-access-key" => "secret_access_key", + "s3.session-token" => "session_token", + other => other, + }; + options.push((mapped.to_string(), value.clone())); + } +} +options.push(("region".to_string(), "us-west-2".to_string())); + +let (store, _path) = object_store::parse_url_opts(&table_uri, options)?; +let engine = DefaultEngineBuilder::new(store.into()).build(); +``` + +The `.into()` converts the `Box` returned by `parse_url_opts` +into the `Arc` that `DefaultEngineBuilder::new` expects. Set the +`region` to match the bucket's actual AWS region. + +## Step 5: Build a Snapshot from the load_table response + +The `load_table` response already contains the log tail. Pass it to +`snapshot_builder_from_load_table` to get a `SnapshotBuilder` with the log tail +and max catalog version already applied, then call `build`. + +```rust,ignore +use delta_kernel_unity_catalog::snapshot_builder_from_load_table; + +let snapshot = snapshot_builder_from_load_table(&resp)?.build(&engine)?; + +println!("Table version: {}", snapshot.version()); +``` + +The builder pins the Snapshot to the max catalog version, so it reflects commits +that haven't been published to `_delta_log/` yet. To read an earlier version, add +`.at_version(5)` before `.build()`. The requested version must not exceed the max +catalog version. + +> [!NOTE] +> If you need the log tail directly (for example, to assemble the builder +> yourself), use the lower-level `log_tail_from_commits`, which converts the +> response's `commits` into a `Vec`. + +## Step 6: Build and execute a Scan + +From this point on, reading works identically to a filesystem-managed table. +Build a Scan from the Snapshot and iterate over the results: + +```rust,ignore +let scan = snapshot.scan_builder().build()?; +for data in scan.execute(Arc::new(engine))? { + let batch = data?; + // process each batch of data +} +``` + +You can use all the same Scan features: [column selection](../reading/column_selection.md), +[filter pushdown](../reading/filter_pushdown.md), and +[scan metadata](../reading/scan_metadata.md) for distributed reads. + +## Complete example + +This example puts all the steps together. It connects to Unity Catalog, loads a +table, fetches credentials, builds a Snapshot from the log tail, and reads the +data. + +```rust,ignore +use std::sync::Arc; + +use delta_kernel::object_store; +use delta_kernel_default_engine::DefaultEngineBuilder; +use delta_kernel_unity_catalog::snapshot_builder_from_load_table; +use unity_catalog_delta_client_api::Operation; +use unity_catalog_delta_rest_client::{ClientConfig, UCClient}; + +#[tokio::main] +async fn main() -> Result<(), Box> { + // 1. Build the UC client + let config = ClientConfig::build(&endpoint, &token).with_additional_user_agent([("MyConnector", "1.0")]).build()?; + let uc_client = UCClient::new(config)?; + + // 2. Load the table (metadata + inline log tail) + let resp = uc_client + .load_table("my_catalog", "my_schema", "my_table") + .await?; + let table_uri = url::Url::parse(&resp.metadata.location)?; + + // 3. Fetch temporary read credentials + let creds = uc_client + .get_table_credentials("my_catalog", "my_schema", "my_table", Operation::Read) + .await?; + + // 4. Build an Engine from the vended credentials + let mut options: Vec<(String, String)> = Vec::new(); + for cred in &creds.storage_credentials { + for (key, value) in &cred.config { + let mapped = match key.as_str() { + "s3.access-key-id" => "access_key_id", + "s3.secret-access-key" => "secret_access_key", + "s3.session-token" => "session_token", + other => other, + }; + options.push((mapped.to_string(), value.clone())); + } + } + options.push(("region".to_string(), "us-west-2".to_string())); + let (store, _) = object_store::parse_url_opts(&table_uri, options)?; + let engine = DefaultEngineBuilder::new(store.into()).build(); + + // 5. Build a Snapshot from the load_table response + let snapshot = snapshot_builder_from_load_table(&resp)?.build(&engine)?; + println!("Loaded table at version {}", snapshot.version()); + + // 6. Build and execute the Scan + let scan = snapshot.scan_builder().build()?; + for data in scan.execute(Arc::new(engine))? { + let batch = data?; + // process each batch of data + } + + Ok(()) +} +``` + +## Credential refresh for long-running operations + +UC vends temporary credentials with a limited lifetime. For short-lived scans, +you don't need to worry about expiration. For long-running operations (large +table scans, streaming reads), check each credential's `expiration_time_ms` +before starting a new phase of work. + +If credentials have expired, call `get_table_credentials` again and rebuild the +Engine with the fresh credentials before continuing. Kernel does not manage +credential lifecycle for you. + +## What's next + +- [Writing to UC Tables](./writing.md): how to append data and commit through + Unity Catalog +- [Building a Scan](../reading/building_a_scan.md): scan configuration options + that apply to both filesystem and catalog-managed tables +- [Filter Pushdown and File Skipping](../reading/filter_pushdown.md): reduce + the data you read with predicates diff --git a/docs/user-guide/src/unity_catalog/writing.md b/docs/user-guide/src/unity_catalog/writing.md new file mode 100644 index 0000000000..1206158287 --- /dev/null +++ b/docs/user-guide/src/unity_catalog/writing.md @@ -0,0 +1,326 @@ +# Writing to Unity Catalog tables + + + + +To write to a Unity Catalog-managed Delta table, you create a `UCCommitter`, +pass it to a Kernel transaction, and then publish the staged commit to make it +visible in `_delta_log/`. + +Before reading this page, make sure you understand the generic +[catalog-managed write lifecycle](../catalog_managed/writing.md) and the +[Unity Catalog Integration overview](./overview.md). + +> [!NOTE] +> This page uses the `delta-kernel-unity-catalog` and +> `unity-catalog-delta-rest-client` crates. All code examples use +> `rust,ignore` because they require these external crates. + +## Set up clients and resolve the table + +Use `UCClient` to load the table and fetch read-write credentials, and build a +`UCUpdateTableRestClient` for the commit path. Both clients share a +`ClientConfig`. + +```rust,ignore +use std::sync::Arc; +use unity_catalog_delta_client_api::Operation; +use unity_catalog_delta_rest_client::{ClientConfig, UCClient, UCUpdateTableRestClient}; + +let config = ClientConfig::build("my-workspace.cloud.databricks.com", token).with_additional_user_agent([("MyConnector", "1.0")]).build()?; +let uc_client = UCClient::new(config.clone())?; +let update_client = Arc::new(UCUpdateTableRestClient::new(config)?); + +// Load the table: metadata + inline log tail +let resp = uc_client + .load_table("my_catalog", "my_schema", "my_table") + .await?; +let table_uri = url::Url::parse(&resp.metadata.location)?; + +// Fetch read-write credentials for the table's cloud storage +let creds = uc_client + .get_table_credentials("my_catalog", "my_schema", "my_table", Operation::ReadWrite) + .await?; +``` + +For the full details on building an engine from vended credentials, see +[Reading UC Tables](./reading.md). + +## Load a snapshot from the load_table response + +Build the Snapshot from the `load_table` response. Pass it to +`snapshot_builder_from_load_table` to get a `SnapshotBuilder` with the log tail +and max catalog version already applied, then call `build`. + +```rust,ignore +use delta_kernel_unity_catalog::snapshot_builder_from_load_table; + +let snapshot = snapshot_builder_from_load_table(&resp)?.build(&engine)?; +``` + +The returned `Snapshot` reflects all ratified commits the catalog knows about, +including those that haven't been published to `_delta_log/` yet. + +## Create a transaction with `UCCommitter` + +`UCCommitter` implements Kernel's `Committer` trait. It stages the commit to +`_staged_commits/`, then submits it to the UC `update_table` API for Unity +Catalog to ratify. Construct it with the commit client and the table's +three-part name plus its table ID. + +```rust,ignore +use delta_kernel_unity_catalog::UCCommitter; +use unity_catalog_delta_client_api::TableIdentifier; + +let table_id = resp.metadata.table_uuid.clone(); +let committer = Box::new(UCCommitter::new( + update_client.clone(), + table_id.clone(), + TableIdentifier::new("my_catalog", "my_schema", "my_table"), +)); +let mut txn = snapshot.clone().transaction(committer, &engine)? + .with_operation("INSERT".to_string()); +``` + +`UCCommitter` requires a multi-threaded tokio runtime. The default Kernel +engine already uses tokio, so this is compatible as long as you use the +multi-threaded runtime (the default for `#[tokio::main]`). + +> [!WARNING] +> `UCCommitter` validates that every commit targets a catalog-managed table +> with in-commit timestamps enabled. It rejects the commit if the table is +> missing the `catalogManaged`, `vacuumProtocolCheck`, or `inCommitTimestamp` +> writer features, if `delta.enableInCommitTimestamps` is not `"true"`, or if +> `io.unitycatalog.tableId` does not match the committer's `table_id`. Tables +> created through [Creating UC Tables](./creating_tables.md) satisfy all of +> these automatically. + +## Write data + +From this point, writing data works the same as any Kernel transaction. Get the +write context, write Parquet files to the table's storage location, and add the +resulting file metadata to the transaction. + +```rust,ignore +let write_context = txn.unpartitioned_write_context()?; +// ... write Parquet files using write_context ... +txn.add_files(file_metadata); +``` + +See [Appending Data](../writing/append.md) for the full details on writing +Parquet files and registering file metadata. + +## Commit and handle the result + +Call `txn.commit()` to stage the commit and ratify it through UC. + +```rust,ignore +use delta_kernel::transaction::CommitResult; + +match txn.commit(&engine)? { + CommitResult::CommittedTransaction(committed) => { + let version = committed.commit_version(); + let post_commit_snapshot = committed + .post_commit_snapshot() + .expect("post-commit snapshot"); + // Proceed to publish (next step) + } + CommitResult::ConflictedTransaction(conflicted) => { + // Another writer committed this version first. Rebase onto the new + // snapshot and retry. UCCommitter does not retry at this level. + } + CommitResult::RetryableTransaction(_retryable) => { + // Transient I/O or server error after the UC HTTP client's own retry + // budget was exhausted. Retry the commit from scratch. + } +} +``` + +The REST client automatically retries transport-level failures (HTTP 5xx, +connection errors) according to the retry knobs on `ClientConfigBuilder`. See +[Client configuration and retries](./overview.md#client-configuration-and-retries). +Once that budget is exhausted, `UCCommitter` surfaces the failure as +`CommitResult::RetryableTransaction`. Transaction-level retries (including +rebasing after a `ConflictedTransaction`) are the connector's responsibility; +`UCCommitter` does not retry commits itself. + +Under the hood, `UCCommitter::commit` does two things for versions >= 1: + +1. Writes the transaction's actions to + `_delta_log/_staged_commits/..json` +2. Submits the staged commit to the UC `update_table` API, where Unity Catalog + ratifies it + +Version 0 (table creation) takes a different path: `UCCommitter` writes +`_delta_log/00000000000000000000.json` directly and skips the `update_table` API. +See [Creating UC Tables](./creating_tables.md) for the creation flow and +[the catalog-managed write lifecycle](../catalog_managed/writing.md) for the +generic ratification flow. + +> [!WARNING] +> `UCCommitter` does not support ALTER TABLE operations (protocol changes, +> metadata changes, or clustering column changes). It also rejects attempts to +> upgrade a path-based table to catalog-managed or downgrade a catalog-managed +> table to path-based. Attempting any of these returns an error. + +## Publish staged commits + +After a successful commit, publish the staged commit so it becomes visible as a +normal delta file in `_delta_log/`. Without publishing, only catalog-aware +readers can see the commit. + +```rust,ignore +use delta_kernel_unity_catalog::UCCommitter; +use unity_catalog_delta_client_api::TableIdentifier; + +// Build a committer to drive publishing +let committer: Box = Box::new(UCCommitter::new( + update_client.clone(), + table_id.clone(), + TableIdentifier::new("my_catalog", "my_schema", "my_table"), +)); + +let published_snapshot = post_commit_snapshot + .publish(&engine, committer.as_ref())?; +``` + +Publishing copies each staged commit from `_staged_commits/..json` +to `_delta_log/.json`. If a published file already exists (from a +previous publish attempt), the copy is silently skipped. + +## Report commit metrics + +After a successful commit, report commit metrics to the catalog. For a +catalog-managed table, the catalog owns table maintenance, not your connector. +Unity Catalog uses the per-commit metrics you report to decide when to schedule +OPTIMIZE and VACUUM. + +> [!NOTE] +> Support for the metrics endpoint depends on the catalog. Some catalogs accept +> reported metrics; others, including the open-source Unity Catalog server, do +> not implement it yet and return a 404. Treat the report as best-effort: a +> failure never affects commit correctness. + +You supply the counts the write engine observed. The row counts and the +file-size histogram are only known to your engine. File and byte counts are +derivable from the commit's add and remove actions. + +The catalog requires the commit version on every report, and reads it from +`file_size_histogram.commit_version`. Set it to the version your commit +produced. When you have no size distribution to report, send a histogram with +empty bins that carries only the version. + +```rust,ignore +use unity_catalog_delta_client_api::{CommitReport, FileSizeHistogram}; + +let report = CommitReport { + num_files_added: 3, + num_bytes_added: 4096, + num_rows_inserted: Some(1000), + file_size_histogram: FileSizeHistogram { + commit_version, + ..Default::default() + }, + ..Default::default() +}; + +// table_id comes from the load_table response (metadata.table_uuid). +if let Err(e) = uc_client + .report_metrics("my_catalog", "my_schema", "my_table", &table_id, report) + .await +{ + eprintln!("metrics report failed (non-fatal): {e}"); +} +``` + +## Post-publish maintenance + +Once commits are published, you can checkpoint the table: + +```rust,ignore +published_snapshot.checkpoint(&engine, None)?; +``` + +Checkpointing requires published commits. If you skip the publish step, +checkpointing fails because it can only operate on published versions. + +## Complete example + +```rust,ignore +use std::sync::Arc; +use delta_kernel::transaction::CommitResult; +use delta_kernel_unity_catalog::{snapshot_builder_from_load_table, UCCommitter}; +use unity_catalog_delta_client_api::{Operation, TableIdentifier}; +use unity_catalog_delta_rest_client::{ClientConfig, UCClient, UCUpdateTableRestClient}; + +// 1. Set up clients +let config = ClientConfig::build("my-workspace.cloud.databricks.com", token).with_additional_user_agent([("MyConnector", "1.0")]).build()?; +let uc_client = UCClient::new(config.clone())?; +let update_client = Arc::new(UCUpdateTableRestClient::new(config)?); + +// 2. Load the table and fetch read-write credentials +let resp = uc_client + .load_table("my_catalog", "my_schema", "my_table") + .await?; +let table_uri = url::Url::parse(&resp.metadata.location)?; +let table_id = resp.metadata.table_uuid.clone(); +let creds = uc_client + .get_table_credentials("my_catalog", "my_schema", "my_table", Operation::ReadWrite) + .await?; + +// 3. Build engine with vended credentials. `build_engine_with_credentials` is +// a connector-owned helper, not part of the library. See Step 4 of +// [Reading UC Tables](./reading.md) for the full expansion. +let engine = build_engine_with_credentials(&table_uri, &creds)?; + +// 4. Build a Snapshot from the load_table response +let snapshot = snapshot_builder_from_load_table(&resp)?.build(&engine)?; + +// 5. Create transaction with UCCommitter +let committer = Box::new(UCCommitter::new( + update_client.clone(), + table_id.clone(), + TableIdentifier::new("my_catalog", "my_schema", "my_table"), +)); +let mut txn = snapshot.clone().transaction(committer, &engine)? + .with_operation("INSERT".to_string()); + +// 6. Write data +let write_context = txn.unpartitioned_write_context()?; +// ... write Parquet files using write_context ... +txn.add_files(file_metadata); + +// 7. Commit, publish, and checkpoint +let committer_for_publish: Box = Box::new(UCCommitter::new( + update_client.clone(), + table_id.clone(), + TableIdentifier::new("my_catalog", "my_schema", "my_table"), +)); + +match txn.commit(&engine)? { + CommitResult::CommittedTransaction(committed) => { + let post_commit_snapshot = committed + .post_commit_snapshot() + .expect("post-commit snapshot"); + + // Publish staged commits to _delta_log/ + let published_snapshot = post_commit_snapshot + .publish(&engine, committer_for_publish.as_ref())?; + + // Checkpoint the published snapshot + published_snapshot.checkpoint(&engine, None)?; + } + CommitResult::ConflictedTransaction(_) => { /* rebase and retry */ } + CommitResult::RetryableTransaction(_) => { /* retry the commit */ } +} +``` + +## What's next + +- [Creating UC Tables](./creating_tables.md) for creating a brand-new + UC-managed table before you can write to it +- [Catalog-managed write lifecycle](../catalog_managed/writing.md) for the + generic commit and publish flow +- [Appending Data](../writing/append.md) for the details of writing Parquet + files and collecting file metadata +- [Reading UC Tables](./reading.md) for resolving tables and loading snapshots diff --git a/docs/user-guide/src/writing/alter_table.md b/docs/user-guide/src/writing/alter_table.md new file mode 100644 index 0000000000..e68ee8f624 --- /dev/null +++ b/docs/user-guide/src/writing/alter_table.md @@ -0,0 +1,142 @@ +# Altering a table + +To add a column to an existing Delta table, you build an `AlterTableTransaction` +from a `Snapshot`, queue one or more schema operations, and commit. The result +is a metadata-only commit that updates the table's schema without rewriting any +data files. + +Before reading this page, make sure you understand +[Creating a Table](./create_table.md) and +[Appending Data](./append.md). + +## When to use alter table + +Use `alter_table()` when you need to evolve a table's schema in place. The +common case today is adding a new column to a table that already has data, +without rewriting the existing files. Existing rows read back `NULL` for the +new column. Subsequent writes can populate it. + +Schema evolution is a metadata-only change. The transaction emits an updated +`Metadata` action and commits with `data_change: false`. No `Add` or `Remove` +file actions are produced. This matters because readers can apply the new +schema to existing files without scanning them, and writers that are only +concerned with data changes can ignore the commit. + +> [!NOTE] +> The first supported operation is `add_column()`. Other schema operations +> (drop column, rename, type changes) are not yet available through the +> `AlterTableTransaction` API. + +## Adding a column + +Suppose your table has the canonical schema `name STRING, age INTEGER, city +STRING` with rows for Alice, Bob, and Carol, and you want to add a `country` +column. The flow is: + +1. Load a `Snapshot` of the table. +2. Call `snapshot.alter_table()` to get an `AlterTableTransactionBuilder`. +3. Call `add_column()` with the new field. +4. Call `build()` to produce an `AlterTableTransaction`. +5. Call `commit()` to atomically apply the schema change. + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use delta_kernel::committer::FileSystemCommitter; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::schema::{DataType, StructField}; +# use delta_kernel::transaction::CommitResult; +# use delta_kernel::{DeltaResult, Snapshot}; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let engine = DefaultEngine::builder(store_from_url(&url)?).build(); +// 1. Load a snapshot of the existing table. +let snapshot = Snapshot::builder_for(url).build(&engine)?; + +// 2. Build and commit an alter-table transaction that adds a new column. +let result = snapshot + .alter_table() + .add_column(StructField::nullable("country", DataType::STRING)) + .build(&engine, Box::new(FileSystemCommitter::new()))? + .with_engine_info("my-app/1.0") + .commit(&engine)?; + +match result { + CommitResult::CommittedTransaction(committed) => { + println!("Schema evolved at version {}", committed.commit_version()); + } + _ => eprintln!("alter table did not succeed"), +} +# Ok(()) +# } +``` + +After this commit, the table schema has four fields. Existing rows for Alice, +Bob, and Carol read back `NULL` for `country`. New writes can populate the +column by including it in the `RecordBatch` they pass to +`engine.write_parquet()`. + +## Validation rules + +`add_column()` checks the new field at `build()` time. If any rule is violated, +`build()` returns an error and no commit is attempted. + +| Rule | Why | +|------|-----| +| The field name must not already exist (case-insensitive) | Delta column names are unique within a struct. | +| The field must be nullable | Existing files do not contain the new column. They read back `NULL`, which would violate a `NOT NULL` constraint. | +| The table must not have column mapping enabled | The current implementation supports add-column only on tables without column mapping. | +| The table must support writes | Tables with unsupported writer features cannot be altered. | +| The evolved schema must not require protocol features the table does not enable | For example, adding a `TIMESTAMP_NTZ` column to a table without the `timestampNtz` feature fails. | + +> [!NOTE] +> The column-mapping limitation applies to add-column only. If your table uses +> column mapping (`delta.columnMapping.mode = "name"` or `"id"`), you cannot +> currently add a column through `alter_table()`. This restriction is expected +> to be lifted as the alter-table framework grows. + +## Chaining multiple operations + +`add_column()` can be called more than once to add several columns in a single +commit. The operations are applied in order, and the resulting schema is +validated as a whole before the commit is constructed: + +```rust,ignore +let result = snapshot + .alter_table() + .add_column(StructField::nullable("country", DataType::STRING)) + .add_column(StructField::nullable("postal_code", DataType::STRING)) + .build(&engine, Box::new(FileSystemCommitter::new()))? + .commit(&engine)?; +``` + +The builder uses a type-state pattern to enforce that at least one operation is +queued before `build()` is callable. Calling `.build()` directly on +`snapshot.alter_table()` without first calling `add_column()` is a compile +error. + +## What you cannot do on an alter-table transaction + +`AlterTableTransaction` is a metadata-only transaction. It does not implement +`SupportsDataFiles`, so the data-file methods are not available at compile +time. In particular, the following are not callable on an `AlterTableTransaction`: + +| Method | Used for | +|--------|----------| +| `unpartitioned_write_context()` / `partitioned_write_context()` | Obtaining a `WriteContext` to write Parquet files | +| `add_files()` | Registering newly written data files | +| `stats_schema()` | Retrieving the statistics schema for written files | + +If you need to add data and evolve the schema, run two transactions: an +alter-table transaction first, then a write transaction against the post-commit +snapshot. See [Appending Data](./append.md) for the write flow. + +## What's next + +- [Appending Data](./append.md) walks through writing data to the evolved + table. +- [Creating a Table](./create_table.md) covers creating a new table with the + schema you want from the start. +- [Schemas and Data Types](../concepts/schema_and_types.md) describes + Kernel's type system in detail. diff --git a/docs/user-guide/src/writing/append.md b/docs/user-guide/src/writing/append.md new file mode 100644 index 0000000000..2874e74c8b --- /dev/null +++ b/docs/user-guide/src/writing/append.md @@ -0,0 +1,351 @@ +# Appending Data + +To append data to an existing Delta table, you create a `Transaction` from a +[Snapshot](../concepts/architecture.md#snapshot), write Parquet files through the engine, register them, +and commit. For a quick end-to-end example that creates a table and writes data, see +[Quick Start: Writing a Table](../getting_started/quick_start_write.md). + +## The write flow + +Appending data to a Delta table follows these steps: + +1. Get a `Snapshot` of the table +2. Create a `Transaction` from the snapshot +3. Get the `WriteContext` from the transaction +4. [Write Parquet files](#writing-parquet-files) using the engine and `WriteContext` +5. Register the written files with the transaction via `add_files` +6. Commit the transaction + +The following example assumes you already have an `engine: DefaultEngine`, which provides +an async `write_parquet` helper. If you use a custom `Engine`, the step 4 +may differ. + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# extern crate tokio; +# use std::sync::Arc; +# use delta_kernel::arrow::array::{Int32Array, RecordBatch, StringArray}; +# use delta_kernel::committer::FileSystemCommitter; +# use delta_kernel::engine::arrow_conversion::TryIntoArrow; +# use delta_kernel::engine::arrow_data::ArrowEngineData; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::transaction::CommitResult; +# use delta_kernel::{DeltaResult, Snapshot}; +# #[tokio::main] +# async fn main() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let engine = DefaultEngine::builder(store_from_url(&url)?).build(); +// 1. Get a snapshot +let snapshot = Snapshot::builder_for(url).build(&engine)?; + +// 2. Create a transaction +let mut txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), &engine)? + .with_operation("INSERT".to_string()) + .with_engine_info("my-app/1.0") + .with_data_change(true); + +// 3. Get write context +let write_context = Arc::new(txn.unpartitioned_write_context()?); + +// 4. Write Parquet file(s) +// Assumes the table schema is: name (STRING), age (INTEGER), city (STRING) +let batch = RecordBatch::try_new( + Arc::new(write_context.logical_schema().as_ref().try_into_arrow()?), + vec![ + Arc::new(StringArray::from(vec!["Dave", "Eve", "Frank"])), + Arc::new(Int32Array::from(vec![4, 5, 6])), + Arc::new(StringArray::from(vec!["Austin", "Boston", "Chicago"])), + ], +)?; +let data = ArrowEngineData::new(batch); +let file_metadata = engine + .write_parquet(&data, write_context.as_ref()) + .await?; + +// 5. Register the files +txn.add_files(file_metadata); + +// 6. Commit +match txn.commit(&engine)? { + CommitResult::CommittedTransaction(committed) => { + println!("Committed version {}", committed.commit_version()); + } + _ => eprintln!("commit did not succeed"), +} +# Ok(()) +# } +``` + +## Creating a transaction + +A transaction is created from a snapshot. The snapshot pins the table version you are +writing against: + +```rust,ignore +let mut txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), &engine)? + .with_operation("INSERT".to_string()) + .with_engine_info("my-app/1.0") + .with_data_change(true); +``` + +The builder methods: + +| Method | Purpose | +|--------|---------| +| `with_operation(String)` | Operation name stored in the commit log (e.g. `"INSERT"`, `"MERGE"`) | +| `with_engine_info(impl Into)` | Identifies your application in the commit log | +| `with_data_change(bool)` | Whether this commit materially changes data (`true`) or just reorganizes it (`false`, e.g. OPTIMIZE) | + +## The WriteContext + +Before writing data, obtain a `WriteContext`. A `WriteContext` bundles everything +needed to correctly write Parquet files: + +```rust,ignore +// For unpartitioned tables +let write_context = txn.unpartitioned_write_context()?; + +// For partitioned tables, pass the partition values for this file +let write_context = txn.partitioned_write_context(partition_values)?; +``` + +For partitioned tables, see +[Writing to Partitioned Tables](./partitioned_writes.md). + +`WriteContext` provides: + +| Method | Returns | Purpose | +|--------|---------|---------| +| `table_root_dir()` | `&Url` | The table root URL | +| `write_dir()` | `Url` | The URL for writing files | +| `logical_schema()` | `&SchemaRef` | The schema your logical data should conform to | +| `physical_schema()` | `&SchemaRef` | The schema for the on-disk physical data | +| `logical_to_physical()` | `ExpressionRef` | Expression that transforms logical data to physical | +| `column_mapping_mode()` | `ColumnMappingMode` | The column mapping mode for this table | +| `stats_columns()` | `&[ColumnName]` | Columns that should have statistics collected | +| `physical_partition_values()` | `&HashMap>` | Serialized partition values keyed by physical column name | + +## Writing Parquet files + +Start with the logical `data: EngineData` you want to write. Its schema should conform to +`write_context.logical_schema()`. + +### Using `DefaultEngine` + +The `DefaultEngine` provides an async helper that does everything for you: + +```rust,ignore +let file_metadata = engine + .write_parquet(&data, write_context.as_ref()) + .await?; +``` + +- **`data`**: An `ArrowEngineData` wrapping a `RecordBatch` matching the logical schema +- **`write_context`**: From `unpartitioned_write_context()` or `partitioned_write_context()` + +`DefaultEngine::write_parquet` handles the logical-to-physical transformation, generates a unique filename, +writes the file, collects statistics, and returns file metadata that you pass to +`txn.add_files()`. + +### Using a custom engine + +If you do not use `DefaultEngine`, write the files yourself. The expected flow is: + +1. Evaluate `write_context.logical_to_physical()` to transform your logical data into + physical data. +2. Write the physical data under `write_context.write_dir()`, then pass the corresponding + add-file metadata to `txn.add_files()`. + +```rust,ignore +// Assume: data: Box (logical), engine: impl Engine, +// write_context: WriteContext. + +// 1. Transform logical data into physical data +let evaluator = engine.evaluation_handler().new_expression_evaluator( + write_context.logical_schema().clone(), + write_context.logical_to_physical(), + write_context.physical_schema().clone().into(), +)?; +let physical_data = evaluator.evaluate(data.as_ref())?; + +// 2. Write `physical_data` under `write_context.write_dir()` in whatever way your +// engine writes Parquet, producing `add_file_metadata` for the written file. + +txn.add_files(add_file_metadata); +``` + +You can call `add_files` multiple times to write multiple files in one transaction. + +> [!NOTE] +> Methods that produce or register data files (`unpartitioned_write_context`, +> `partitioned_write_context`, `add_files`, `stats_schema`) are gated by the +> `SupportsDataFiles` trait bound and are available on standard write transactions but not +> on metadata-only transaction states (such as a future `AlterTable`). + +## Committing + +`commit()` consumes the transaction and returns a `CommitResult`: + +```rust,ignore +match txn.commit(&engine)? { + CommitResult::CommittedTransaction(committed) => { + println!("Committed version {}", committed.commit_version()); + } + _ => { + eprintln!("commit did not succeed"); + } +} +``` + +> [!NOTE] +> `commit()` returns a `CommitResult` with three variants: `CommittedTransaction` on success, +> `ConflictedTransaction` if another writer committed first, and `RetryableTransaction` for +> transient IO errors. Automatic conflict resolution is not yet supported. A blind append to +> a table with no concurrent writers always succeeds. + +## Blind appends + +A **blind append** is a write that adds new files without reading or depending on existing +table state. To mark a transaction as a blind append, call `with_blind_append()` during +construction: + +```rust,ignore +let txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), &engine)? + .with_operation("INSERT".to_string()) + .with_blind_append(); +``` + +Kernel records `isBlindAppend: true` in the commit's `commitInfo` action. This flag +enables conflict resolution optimizations: two blind appends to the same table can never +conflict with each other, because neither depends on the other's output. + +Kernel validates the following rules at commit time. If any rule is violated, `commit()` +returns an error: + +| Rule | Rationale | +|------|-----------| +| The transaction must add at least one file | A blind append with no data is meaningless | +| `data_change` must be `true` | Blind appends are logical data additions, not reorganizations | +| The transaction must not remove any files | Removing files means the write depends on existing state | +| The transaction must not update deletion vectors | Deletion vector updates depend on existing state | +| The transaction must not be a create-table transaction | Table creation is not an append | + +> [!TIP] +> Mark your transaction as a blind append whenever you are inserting new data without reading +> the table first. This gives the committer the information it needs to resolve conflicts +> safely in multi-writer scenarios. + +## Custom commit info + +Kernel always writes a `commitInfo` action for every commit. To include your own fields in +that action, call `with_commit_info()` with your custom data and its schema: + +```rust,ignore +let txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), &engine)? + .with_operation("INSERT".to_string()) + .with_commit_info(engine_commit_info, commit_info_schema); +``` + +The `engine_commit_info` argument is a `Box` containing the fields you want +to add, and `commit_info_schema` is the corresponding `SchemaRef`. Kernel merges your fields +into the final `commitInfo` action. + +Kernel reserves certain fields and overrides them regardless of what you provide. Do not set +these fields in your custom commit info: + +| Field | Set by Kernel to | +|-------|------------------| +| `timestamp` | The transaction's commit timestamp | +| `inCommitTimestamp` | The in-commit timestamp (if ICT is enabled on the table) | +| `operation` | The value from `with_operation()` | +| `operationParameters` | Operation parameters (if any) | +| `kernelVersion` | The Kernel library version | +| `isBlindAppend` | `true` if `with_blind_append()` was called, omitted otherwise | +| `engineInfo` | The value from `with_engine_info()` | +| `txnId` | A unique transaction identifier | + +Any field in your custom data that shares a name with a Kernel-reserved field is replaced +with Kernel's value. Fields with names that do not collide are preserved as-is in the final +`commitInfo`. + +## After committing + +A successful commit returns a `CommittedTransaction` with access to a post-commit +snapshot and post-commit statistics: + +```rust,ignore +let committed = match txn.commit(&engine)? { + CommitResult::CommittedTransaction(c) => c, + _ => panic!("unexpected result"), +}; + +// The version that was committed +let version = committed.commit_version(); + +// Post-commit statistics help you decide when to run maintenance +let stats = committed.post_commit_stats(); +println!("Commits since last checkpoint: {}", stats.commits_since_checkpoint); +println!("Commits since last log compaction: {}", stats.commits_since_log_compaction); + +// The post-commit snapshot reflects the table state after this commit. +// Use it for maintenance operations like publishing, checkpointing, and checksums. +if let Some(snapshot) = committed.post_commit_snapshot() { + // Write a checksum file for this version + let (result, snapshot) = snapshot.write_checksum(&engine)?; + + // For catalog-managed tables: snapshot.publish(engine, committer)? + // For checkpointing: snapshot.checkpoint(&engine)? +} +``` + +### Post-commit statistics + +`CommittedTransaction::post_commit_stats()` returns a `PostCommitStats` struct with two +fields: + +| Field | Meaning | +|-------|---------| +| `commits_since_checkpoint` | Number of commits since the last checkpoint. Commit 0 counts as a checkpoint. | +| `commits_since_log_compaction` | Number of commits since the last log compaction or checkpoint. A checkpoint resets this counter too. | + +Use these values to decide when to trigger maintenance. For example, you might checkpoint +every 10 commits or compact the log every 50 commits. + +### Writing a checksum + +You can call `write_checksum()` on the post-commit snapshot to write a checksum file (CRC +file), which enables faster snapshot loading and table state validation: + +```rust,ignore +if let Some(snapshot) = committed.post_commit_snapshot() { + let (checksum_result, updated_snapshot) = snapshot.write_checksum(&engine)?; + // Use updated_snapshot for subsequent operations (it includes the new CRC file) +} +``` + +`write_checksum()` returns a `ChecksumWriteResult` and an updated `SnapshotRef`. +`ChecksumWriteResult::Written` means the CRC file was created successfully. +`ChecksumWriteResult::AlreadyExists` means a CRC file already exists at this version, and +the original snapshot is returned unchanged. Per the Delta protocol, writers must not +overwrite existing checksum files. + +> [!NOTE] +> `write_checksum()` requires in-memory CRC information, which is only available on +> post-commit snapshots. Calling it on a snapshot loaded from disk (without a pre-computed +> CRC) returns a `ChecksumWriteUnsupported` error. + +The post-commit snapshot is the entry point for maintenance operations that should happen +after a successful write. See [Checkpointing](../maintenance/checkpointing.md) and +[Catalog-Managed Tables](../catalog_managed/overview.md) for details. + +## What's next + +- [Writing to Partitioned Tables](./partitioned_writes.md) covers writing data with partition values. +- [Creating a Table](./create_table.md) covers creating a new table from scratch. +- [Checkpointing](../maintenance/checkpointing.md) explains when and how to write checkpoints. diff --git a/docs/user-guide/src/writing/create_table.md b/docs/user-guide/src/writing/create_table.md new file mode 100644 index 0000000000..9592916cc5 --- /dev/null +++ b/docs/user-guide/src/writing/create_table.md @@ -0,0 +1,386 @@ +# Creating a Table + +To create a new Delta table, you configure a `CreateTableTransactionBuilder` with your +schema and table options, then commit it. For a quick end-to-end example that creates a +table, writes data, and reads it back, see +[Quick Start: Writing a Table](../getting_started/quick_start_write.md). + +## Basic table creation + +The `create_table` function returns a builder that you configure and then commit: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel::committer::FileSystemCommitter; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::schema::{DataType, StructField, StructType}; +# use delta_kernel::transaction::create_table::create_table; +# use delta_kernel::DeltaResult; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let engine = DefaultEngine::builder(store_from_url(&url)?).build(); +let schema = Arc::new(StructType::try_new([ + StructField::not_null("id", DataType::INTEGER), + StructField::nullable("name", DataType::STRING), + StructField::nullable("age", DataType::INTEGER), +])?); + +create_table(url.as_str(), schema, "my-app/1.0") + .build(&engine, Box::new(FileSystemCommitter::new()))? + .commit(&engine)?; +# Ok(()) +# } +``` + +The three required arguments are: +- **`path`**: Where to create the table (local path or URI like `s3://bucket/path`) +- **`schema`**: The table's column definitions as a `StructType` +- **`engine_info`**: A string identifying your application (stored in the commit log) + +`.build()` validates the inputs and creates a `CreateTableTransaction`. `.commit()` writes +version 0 of the table, producing the initial Protocol and Metadata actions. + +## Defining a schema + +Schemas are built from `StructField`s, each with a name, data type, and nullability: + +```rust,no_run +# extern crate delta_kernel; +# use delta_kernel::DeltaResult; +# fn example() -> DeltaResult<()> { +use std::sync::Arc; +use delta_kernel::schema::{ArrayType, DataType, MapType, StructField, StructType}; + +let schema = Arc::new(StructType::try_new([ + // Non-nullable integer + StructField::not_null("id", DataType::LONG), + + // Nullable string + StructField::nullable("name", DataType::STRING), + + // Nested struct + StructField::nullable( + "address", + StructType::try_new([ + StructField::nullable("street", DataType::STRING), + StructField::nullable("city", DataType::STRING), + ])?, + ), + + // Array of integers + StructField::nullable( + "scores", + ArrayType::new(DataType::INTEGER, true), // true = elements are nullable + ), + + // Map from string to double + StructField::nullable( + "metrics", + MapType::new(DataType::STRING, DataType::DOUBLE, true), // true = values are nullable + ), +])?); +# Ok(()) +# } +``` + +For the full list of supported data types, see [Schemas and Data Types](../concepts/schema_and_types.md). + +## Table properties + +You can set custom application properties on the table: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel::committer::FileSystemCommitter; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::schema::{DataType, StructField, StructType}; +# use delta_kernel::transaction::create_table::create_table; +# use delta_kernel::DeltaResult; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let engine = DefaultEngine::builder(store_from_url(&url)?).build(); +# let schema = Arc::new(StructType::try_new([ +# StructField::not_null("id", DataType::INTEGER), +# ])?); +create_table(url.as_str(), schema, "my-app/1.0") + .with_table_properties([ + ("myapp.version", "2.0"), + ("myapp.owner", "data-team"), + ]) + .build(&engine, Box::new(FileSystemCommitter::new()))? + .commit(&engine)?; +# Ok(()) +# } +``` + +Custom properties (those not starting with `delta.`) are always allowed. Delta properties +(`delta.*`) are validated against an allow list. Kernel only permits properties for +features it supports. + +## Clustered tables + +You can create a clustered table using `with_data_layout`. Clustering optimizes data file +layout for queries that filter on the clustering columns: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel::committer::FileSystemCommitter; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::schema::{DataType, StructField, StructType}; +# use delta_kernel::transaction::create_table::create_table; +# use delta_kernel::transaction::data_layout::DataLayout; +# use delta_kernel::DeltaResult; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let engine = DefaultEngine::builder(store_from_url(&url)?).build(); +let schema = Arc::new(StructType::try_new([ + StructField::not_null("id", DataType::INTEGER), + StructField::nullable("region", DataType::STRING), + StructField::nullable("timestamp", DataType::TIMESTAMP), +])?); + +create_table(url.as_str(), schema, "my-app/1.0") + .with_data_layout(DataLayout::clustered(["region", "timestamp"])) + .build(&engine, Box::new(FileSystemCommitter::new()))? + .commit(&engine)?; +# Ok(()) +# } +``` + +Clustering columns can be top-level or nested. The `DataLayout::clustered([...])` helper +treats each string as a single top-level column name. To cluster on a nested column, +construct the `DataLayout::Clustered` variant directly with a multi-segment `ColumnName`: + +```rust,ignore +use delta_kernel::expressions::column_name; +use delta_kernel::transaction::data_layout::DataLayout; + +let layout = DataLayout::Clustered { + columns: vec![ + column_name!("region"), + column_name!("address", "city"), + ], +}; +``` + +For nested clustering columns, every intermediate path component must be a struct field, +and the leaf column must have a stats-eligible primitive type. + +The kernel automatically enables the required table features (`DomainMetadata` and +`ClusteredTable`) when clustering is specified. + +## Partitioned tables + +You can create a partitioned table using `DataLayout::partitioned()`. Partitioning organizes +data files into directories based on partition column values, which allows readers to skip +entire directories when filtering on those columns. + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# use std::sync::Arc; +# use delta_kernel::committer::FileSystemCommitter; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::schema::{DataType, StructField, StructType}; +# use delta_kernel::transaction::create_table::create_table; +# use delta_kernel::transaction::data_layout::DataLayout; +# use delta_kernel::DeltaResult; +# fn example() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let engine = DefaultEngine::builder(store_from_url(&url)?).build(); +let schema = Arc::new(StructType::try_new([ + StructField::not_null("id", DataType::INTEGER), + StructField::nullable("name", DataType::STRING), + StructField::not_null("year", DataType::INTEGER), + StructField::not_null("month", DataType::INTEGER), +])?); + +create_table(url.as_str(), schema, "my-app/1.0") + .with_data_layout(DataLayout::partitioned(["year", "month"])) + .build(&engine, Box::new(FileSystemCommitter::new()))? + .commit(&engine)?; +# Ok(()) +# } +``` + +### Validation rules + +`build()` validates partition columns against these rules: + +- **Top-level only**: partition columns must be top-level fields in the schema. Nested paths + like `address.city` are not supported. +- **Primitive types only**: each partition column must have a primitive type (`STRING`, + `INTEGER`, `LONG`, `DATE`, `TIMESTAMP`, etc.). Struct, array, and map types are rejected. +- **At least one non-partition column**: the schema must contain at least one column that is + not a partition column. A table with every column marked as a partition column is invalid. +- **No duplicates**: the same column cannot appear more than once in the partition column list. +- **Schema presence**: every partition column must exist in the schema. +- **At least one partition column**: if `DataLayout::partitioned(...)` is used, the column + list cannot be empty. + +### Where partition values live + +Partition column values are stored in the `partitionValues` field of each `add` action in +the transaction log rather than inside the data files, although specific table +features may additionally materialize them into the data files; see +[Writing to Partitioned Tables](./partitioned_writes.md). + +> [!NOTE] +> Partitioning and clustering are mutually exclusive. You can call `with_data_layout()` with +> either `DataLayout::partitioned()` or `DataLayout::clustered()`, but not both. Only the +> last `with_data_layout()` call takes effect. + +## The Committer + +The `build()` method takes a `Box` that controls how the commit is +persisted: + +- **`FileSystemCommitter`**: For standalone filesystem-based tables. Writes commit files + directly to `_delta_log/` using atomic put-if-absent. This is the default for most use + cases. + +- **Custom `Committer`**: For catalog-managed tables (e.g. Unity Catalog), you implement + the `Committer` trait to route commits through the catalog. See + [Catalog-Managed Tables](../catalog_managed/overview.md). + +## Handling the result + +`commit()` returns a `CommitResult`: + +```rust,ignore +match txn.commit(&engine)? { + CommitResult::CommittedTransaction(committed) => { + println!("Created table at version {}", committed.commit_version()); + } + CommitResult::ConflictedTransaction(_) => { + // Another writer created the table concurrently + } + CommitResult::RetryableTransaction(retry) => { + // Transient I/O error. Safe to retry. + println!("Retryable error: {}", retry.error); + } +} +``` + +For table creation, `CommittedTransaction` is the expected result (version 0). +`ConflictedTransaction` means another process created the table between your existence +check and commit. `RetryableTransaction` indicates a transient error. + +## Validations + +`build()` performs these checks before creating the transaction: + +**Path and existence:** +- The path is a valid URI +- No Delta table already exists at that path + +**Schema:** +- The schema has at least one field +- Column names are non-empty +- When column mapping is disabled, column names cannot contain Parquet special characters + (space, comma, semicolon, braces, parentheses, tab, newline, or `=`). Enable column + mapping to use these characters. +- When column mapping is enabled, column names cannot contain newlines +- No duplicate column names, case-insensitive, across all nested paths +- Non-null columns (fields with `nullable: false`) are accepted. When the schema contains + any non-null field anywhere in the tree, Kernel auto-enables the `invariants` writer + feature on the new table. This matches Delta-Spark, which treats `nullable: false` as an + implicit column invariant. + +**Table properties:** +- Custom properties (not starting with `delta.`) are always allowed +- `delta.*` properties are checked against an allow list +- Feature signals (`delta.feature.*`) are checked against an allow list +- Catalog-managed tables cannot set `delta.enableInCommitTimestamps=false`. Kernel + auto-enables ICT for catalog-managed tables. + +**Data layout:** +- Clustering columns (if specified) exist in the schema. Both top-level and nested paths + are supported. Each leaf column must have a stats-eligible primitive type. +- Partition columns (if specified) exist in the schema, are top-level, have primitive + types, and leave at least one non-partition column. The partition column list cannot + be empty. + +## Auto-enabled features + +`build()` auto-enables certain table features based on the schema, properties, and data +layout, so you do not need to set them manually. The triggers fall into four groups. +Each enabled feature is either a reader/writer feature (bumps both the reader and writer +protocol) or a writer-only feature (bumps the writer protocol only). + +### Table-property-derived + +Features enabled from the values of `delta.*` properties you set with +`with_table_properties()` or from the feature signals that appear in that property bag. + +| Trigger | Reader/Writer feature | Writer feature | Properties set | +|---------|-----------------------|----------------|----------------| +| `delta.columnMapping.mode=name` or `id` | `columnMapping` (assigns physical names and IDs) | — | `delta.columnMapping.maxColumnId` | +| `delta.enableDeletionVectors=true` | `deletionVectors` | — | — | +| `delta.enableTypeWidening=true` | `typeWidening` | — | — | +| `delta.feature.catalogManaged=supported` | `catalogManaged` | `inCommitTimestamp` | `delta.enableInCommitTimestamps=true` | +| `delta.enableInCommitTimestamps=true` | — | `inCommitTimestamp` | — | +| `delta.enableRowTracking=true` | — | `rowTracking`, `domainMetadata` | `delta.rowTracking.materializedRowIdColumnName`, `delta.rowTracking.materializedRowCommitVersionColumnName` | +| `delta.enableChangeDataFeed=true` | — | `changeDataFeed` | — | +| `delta.appendOnly=true` | — | `appendOnly` | — | + +### Data-layout-derived + +Features enabled by the data layout you pass to `with_data_layout()`. + +| Trigger | Reader/Writer feature | Writer feature | Properties set | +|---------|-----------------------|----------------|----------------| +| `DataLayout::clustered(...)` (or the `DataLayout::Clustered` variant) | — | `domainMetadata`, `clusteredTable` | — | + +### Schema-derived + +Features enabled when specific column types or annotations appear in the schema. + +| Trigger | Reader/Writer feature | Writer feature | Properties set | +|---------|-----------------------|----------------|----------------| +| `VARIANT` type in schema | `variantType` | — | — | +| `TIMESTAMP_NTZ` type in schema | `timestampNtz` | — | — | +| Any field with `nullable: false` in schema | — | `invariants` | — | + +### Feature-signal only + +Features enabled only when you explicitly include the feature signal in +`with_table_properties()`. Kernel does not derive these from the schema or from other +properties. + +| Trigger | Reader/Writer feature | Writer feature | Properties set | +|---------|-----------------------|----------------|----------------| +| `delta.feature.v2Checkpoint=supported` | `v2Checkpoint` | — | — | +| `delta.feature.vacuumProtocolCheck=supported` | `vacuumProtocolCheck` | — | — | + +> [!NOTE] +> Auto-enabled features cannot always be set explicitly via `delta.feature.*=supported`. +> Kernel enforces that the protocol and metadata stay consistent with the schema and data +> layout, so some features are controllable only through their driving condition: +> +> - `variantType` and `timestampNtz` are enabled only when the schema uses those types. +> Explicit feature signals for them are rejected. +> - `clusteredTable` is enabled only by passing clustering columns to `with_data_layout()`. +> `delta.feature.clustering=supported` is rejected. +> - For `catalogManaged` tables, `delta.enableInCommitTimestamps=false` is rejected because +> ICT is required. +> +> Other auto-enabled features (such as `columnMapping`, `inCommitTimestamp`) can still +> be enabled explicitly via `delta.feature.X=supported` if you want to pre-enable them +> before the schema or property trigger applies. + +## What's next + +- [Appending Data](./append.md) to write rows to an existing table +- [Writing to Partitioned Tables](./partitioned_writes.md) to write data with partition values +- [Quick Start: Writing a Table](../getting_started/quick_start_write.md) for a complete end-to-end example diff --git a/docs/user-guide/src/writing/domain_metadata.md b/docs/user-guide/src/writing/domain_metadata.md new file mode 100644 index 0000000000..9b9cc86060 --- /dev/null +++ b/docs/user-guide/src/writing/domain_metadata.md @@ -0,0 +1,177 @@ +# Domain metadata + +To store application-specific configuration alongside a Delta table, you use +**domain metadata**. Each domain is a named key-value pair persisted in the +Delta transaction log. Your connector can write, read, and remove domain +metadata through the Kernel API without affecting table data. + +Domain metadata is useful for tracking connector-specific state, feature +flags, or custom configuration that should travel with the table and survive +across sessions. + +Before reading this page, make sure you understand +[Appending Data](./append.md) and how transactions work. + +## Writing domain metadata + +To attach domain metadata to a commit, call `with_domain_metadata()` on the +transaction. This method is available on both create-table and existing-table +transactions. + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# extern crate tokio; +# use std::sync::Arc; +# use delta_kernel::committer::FileSystemCommitter; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +# #[tokio::main] +# async fn main() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let engine = DefaultEngine::builder(store_from_url(&url)?).build(); +# let snapshot = Snapshot::builder_for(url).build(&engine)?; +let txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), &engine)? + .with_domain_metadata( + "myConnector.settings".to_string(), + r#"{"version": 1, "compress": true}"#.to_string(), + ) + .with_operation("UPDATE METADATA".to_string()); + +txn.commit(&engine)?; +# Ok(()) +# } +``` + +The `with_domain_metadata` signature takes two `String` arguments: + +```rust,ignore +pub fn with_domain_metadata(self, domain: String, configuration: String) -> Self +``` + +The `domain` identifies the metadata namespace and `configuration` holds the +value. The configuration is an opaque string. You can store JSON, plain text, +or any format your connector understands. + +You can set metadata for multiple distinct domains in the same transaction by +calling `with_domain_metadata` more than once with different domain names. + +## Removing domain metadata + +To remove a domain from an existing table, call `with_domain_metadata_removed()` +on an existing-table transaction. This method is not available on create-table +transactions because there is no metadata to remove from a table that does not +exist yet. + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# extern crate tokio; +# use std::sync::Arc; +# use delta_kernel::committer::FileSystemCommitter; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +# #[tokio::main] +# async fn main() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let engine = DefaultEngine::builder(store_from_url(&url)?).build(); +# let snapshot = Snapshot::builder_for(url).build(&engine)?; +let txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), &engine)? + .with_domain_metadata_removed("myConnector.settings".to_string()) + .with_operation("REMOVE METADATA".to_string()); + +txn.commit(&engine)?; +# Ok(()) +# } +``` + +If the domain does not exist in the log, the removal is a no-op. Kernel +handles this gracefully during commit. + +## Reading domain metadata + +To read domain metadata from a table, call `get_domain_metadata()` on a +`Snapshot`. + +> [!TIP] +> Reading domain data from a snapshot is efficient if a CRC file was present when loading +> the snapshot. Otherwise determining domain metadata requires performing a log replay. +> Connectors that heavily rely on `get_domain_metadata()` should ensure checksum CRC files are +> written with each commit. + + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# extern crate tokio; +# use std::sync::Arc; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::{DeltaResult, Snapshot}; +# #[tokio::main] +# async fn main() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let engine = DefaultEngine::builder(store_from_url(&url)?).build(); +let snapshot = Snapshot::builder_for(url).build(&engine)?; + +// Returns Ok(Some(config)) if the domain exists, Ok(None) if it does not +let config: Option = snapshot.get_domain_metadata("myConnector.settings", &engine)?; + +match config { + Some(value) => println!("Domain config: {value}"), + None => println!("No metadata found for this domain"), +} +# Ok(()) +# } +``` + +> [!WARNING] +> `get_domain_metadata()` rejects domain names that start with `delta.` and +> returns an error. The `delta.*` namespace is reserved for Kernel's internal +> use (row tracking, clustering). You can only read user-defined domains +> through this API. + +## Constraints and validation + +Kernel validates domain metadata operations at commit time. The following rules +apply: + +- **One domain per transaction.** Each domain name can appear at most once per + transaction. You cannot set and remove the same domain in a single commit, + and you cannot set the same domain twice. If you include a duplicate domain, + the commit fails with an error. + +- **Reserved prefix.** Domain names starting with `delta.` are reserved for + Kernel's internal use (e.g., clustering metadata). Attempting to read, write, + or remove a `delta.` domain through the public API returns an error. + +- **Feature requirement.** Domain metadata operations require the + `domainMetadata` writer feature to be enabled on the table (writer version 7). + If the feature is not enabled, the commit fails. + +- **No removals on create-table.** The `with_domain_metadata_removed()` method + is only available on existing-table transactions. The Rust type system + prevents calling it on a create-table transaction at compile time. + +- **Multiple domains allowed.** Although each domain can appear only once, you + can set or remove metadata for multiple _distinct_ domains in the same + transaction. + +> [!NOTE] +> Validation is deferred until `commit()`. The builder methods +> `with_domain_metadata()` and `with_domain_metadata_removed()` do not check +> for duplicates or reserved prefixes eagerly. Errors surface when you call +> `commit()`. + +## What's next + +- [Idempotent Writes](./idempotent_writes.md) covers `SetTransaction` actions + for at-most-once write guarantees +- [Appending Data](./append.md) explains the full write flow for adding data + files to a table +- [Creating a Table](./create_table.md) shows how to set domain metadata during + table creation diff --git a/docs/user-guide/src/writing/idempotent_writes.md b/docs/user-guide/src/writing/idempotent_writes.md new file mode 100644 index 0000000000..fb1a563b0c --- /dev/null +++ b/docs/user-guide/src/writing/idempotent_writes.md @@ -0,0 +1,96 @@ +# Idempotent writes + +To guarantee at-most-once semantics for retryable write pipelines (e.g., a streaming job +or a queue consumer), you attach a **transaction identifier** to each commit. If the same +logical write is attempted twice, the second attempt becomes a no-op. Delta supports this +through `SetTransaction` actions recorded in the transaction log. + +## How it works + +Each transaction can carry a `SetTransaction` action containing: + +- **`app_id`**: a string identifying the application (e.g., `"my-streaming-job"`) +- **`version`**: an application-defined integer. The Delta protocol leaves its semantics + up to the connector. A common pattern is a monotonically increasing counter (batch + number, Kafka offset, etc.), but other schemes are valid. The protocol only guarantees + that the latest committed `version` per `app_id` is retrievable. + +Before writing, check the table for the latest committed version for your `app_id`. If +that version indicates the write has already been applied, skip it. The comparison logic +is up to you. + +## Writing with a transaction ID + +Call `with_transaction_id()` on the transaction and check `get_app_id_version()` before +committing: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# extern crate tokio; +# use std::sync::Arc; +# use delta_kernel::committer::FileSystemCommitter; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::transaction::CommitResult; +# use delta_kernel::{DeltaResult, Snapshot}; +# #[tokio::main] +# async fn main() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/table")?; +# let engine = DefaultEngine::builder(store_from_url(&url)?).build(); +# let snapshot = Snapshot::builder_for(url).build(&engine)?; +let app_id = "my-streaming-job"; +let batch_version: i64 = 42; + +// Check if this batch was already committed +if let Some(committed_version) = snapshot.get_app_id_version(app_id, &engine)? { + if committed_version >= batch_version { + println!("batch {batch_version} already committed (table has {committed_version})"); + return Ok(()); + } +} + +// Not yet committed. Proceed with the write. +let txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), &engine)? + .with_transaction_id(app_id.to_string(), batch_version) + .with_operation("STREAMING UPDATE".to_string()); + +// ... write data, add files, commit ... +# Ok(()) +# } +``` + +The `>=` check above assumes the application uses monotonically increasing versions. A +connector with a different scheme (for example, tracking a set of known batch IDs) would +compare differently. + +The `SetTransaction` action is written to the commit log alongside the data actions. On +the next run, `get_app_id_version()` will find it and the duplicate write is skipped. + +## Reading the latest transaction version + +`Snapshot::get_app_id_version()` scans the log for the latest `SetTransaction` with the +given `app_id`: + +```rust,ignore +let latest: Option = snapshot.get_app_id_version("my-app", &engine)?; +``` + +Returns `None` if no transaction with that `app_id` has been committed. + +## Transaction retention + +By default, `SetTransaction` actions are retained indefinitely. You can configure a +retention duration via the `delta.setTransactionRetentionDuration` table property (e.g. +`"interval 30 days"`). When set, transactions whose `lastUpdated` timestamp is older than +the retention window are filtered out by `get_app_id_version()`. + +> [!NOTE] +> Kernel automatically sets `lastUpdated` to the commit timestamp when you call +> `with_transaction_id()`. You don't need to set it manually. + +## What's next + +- [Appending data](./append.md) covers the basics of writing data with Kernel +- [Checkpointing](../maintenance/checkpointing.md) explains how to write checkpoints after commits diff --git a/docs/user-guide/src/writing/partitioned_writes.md b/docs/user-guide/src/writing/partitioned_writes.md new file mode 100644 index 0000000000..d5d090735d --- /dev/null +++ b/docs/user-guide/src/writing/partitioned_writes.md @@ -0,0 +1,163 @@ +# Writing to partitioned tables + +To write data to a partitioned table, you create a `WriteContext` for each distinct +set of partition values, write Parquet files through the engine for each one, and +commit. Kernel validates partition values, serializes them per the Delta protocol, +and constructs the correct directory paths. + +Before reading this page, make sure you understand +[Appending Data](./append.md) and +[Creating a Table](./create_table.md#partitioned-tables). + +## How partitioned writes differ + +For unpartitioned tables, you create one `WriteContext` and write all data through it. +For partitioned tables, you create one `WriteContext` per distinct partition value +combination. Partition values are file-constant metadata baked into the `WriteContext` +at creation time, not data columns passed at write time. + +```text +Unpartitioned: 1 WriteContext --> write all data +Partitioned: 1 WriteContext per distinct partition --> write that partition's data +``` + +## The write flow + +The pattern for partitioned writes is: **group your data by partition values, create a +`WriteContext` per group, and write each group**. + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# extern crate tokio; +# use std::collections::HashMap; +# use delta_kernel::arrow::array::RecordBatch; +# use delta_kernel::committer::FileSystemCommitter; +# use delta_kernel::engine::arrow_data::ArrowEngineData; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::expressions::Scalar; +# use delta_kernel::{DeltaResult, Snapshot}; +# #[tokio::main] +# async fn main() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/partitioned_table")?; +# let engine = DefaultEngine::builder(store_from_url(&url)?).build(); +let snapshot = Snapshot::builder_for(url).build(&engine)?; +let mut txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), &engine)? + .with_operation("INSERT".to_string()) + .with_data_change(true); + +// Suppose you have data grouped by partition values already. +// For each partition, create a WriteContext and write. +let partitions: Vec<(HashMap, RecordBatch)> = todo!("group your data"); + +for (partition_values, batch) in partitions { + // 1. Create a WriteContext for this partition + let wc = txn.partitioned_write_context(partition_values)?; + + // 2. Write the data (the logical write schema excludes partition columns) + let data = ArrowEngineData::new(batch); + let file_metadata = engine.write_parquet(&data, &wc).await?; + + // 3. Register the written file + txn.add_files(file_metadata); +} + +// Commit all partitions in a single transaction +txn.commit(&engine)?; +# Ok(()) +# } +``` + +Each `partitioned_write_context` call takes a `HashMap` mapping logical +partition column names to typed values: + +```rust,ignore +let partition_values = HashMap::from([ + ("year".to_string(), Scalar::Integer(2024)), + ("month".to_string(), Scalar::Integer(3)), +]); +let wc = txn.partitioned_write_context(partition_values)?; +``` + +Key points: + +- **Typed values**: partition values are `Scalar` values, not strings. Pass + `Scalar::Integer(2024)` for an integer column, not `Scalar::String("2024".into())`. + Kernel rejects type mismatches. +- **Case-insensitive keys**: `"YEAR"` matches schema column `"year"`. Kernel normalizes + to the schema case. +- **No partition columns in your logical data**: your data batches should follow the logical write + schema (`wc.logical_schema()`), which excludes partition columns. +- **Materialization is automatic**: some table features (such as + `materializePartitionColumns` and `icebergCompatV3`) require partition values to also be + written into the data files as regular columns. Kernel handles this through the + logical-to-physical expression. + +> [!TIP] +> To get the partition column names at runtime, call `txn.logical_partition_columns()`. +> This is useful when your connector handles arbitrary tables and does not know the +> partition columns in advance. + +## Grouping data by partition values + +How you group data by partition values is up to your connector. Kernel's contract is +that each `partitioned_write_context` call receives a `HashMap` for one +distinct partition, and the corresponding data files contain only that partition's rows. + +Kernel provides `serialize_partition_value` as a public utility for building hashable +group keys from `Scalar` values. It returns a `DeltaResult>` per value, +which you can collect into a `Vec>` group key for use in a `HashMap`. + +## Partition value validation + +`partitioned_write_context` validates the provided values before creating the +`WriteContext`: + +| Check | Example | +|-------|---------| +| Missing partition column | Table has `["year", "month"]` but only `"year"` provided | +| Extra key | `"region"` provided but is not a partition column | +| Type mismatch | `Scalar::String("2024")` for an `INTEGER` column | +| Duplicate after case normalization | Both `"YEAR"` and `"year"` provided | + +If validation fails, `partitioned_write_context` returns an error before any data reaches +disk. + +## What Kernel handles + +When you call `partitioned_write_context`, Kernel performs the following steps internally. +Your connector does not need to implement any of this: + +1. **Key validation**: all partition columns present, no extra keys +2. **Case normalization**: keys matched case-insensitively against the schema +3. **Type checking**: each `Scalar`'s type must match the partition column's schema type +4. **Value serialization**: `Scalar` values converted to protocol-compliant strings + (for example, `Scalar::Date(19723)` becomes `"2024-01-01"`) +5. **Key translation**: logical column names translated to physical names when + column mapping is enabled + +`write_dir()` on the resulting `WriteContext` returns the directory where data files +should be written. Without column mapping, this is a Hive-style path like +`/year=2024/month=3/`. With column mapping enabled, this is a +random two-character prefix directory like `/aB/` that avoids +exposing physical column names and distributes files across object store +prefixes to prevent S3 hotspots. + +> [!NOTE] +> Hive partition path segments are URI-encoded on top of Hive escaping, matching the +> on-disk layout produced by Delta-Spark and Delta-Kernel-Java. For example, a partition +> value `2025-03-31T15:30:00Z` produces the path segment +> `p=2025-03-31T15%253A30%253A00Z/` (the `:` is Hive-escaped to `%3A`, and the `%` is then +> URI-encoded to `%25`). The same encoded path is recorded in the `add.path` field of the +> commit. Callers writing through the filesystem must URI-decode `write_dir()` once before +> using it as an OS path. + +## What's next + +- [Appending Data](./append.md) covers the general write flow, commit handling, and + blind appends. +- [Creating a Table](./create_table.md#partitioned-tables) covers creating partitioned + tables. +- [Idempotent Writes](./idempotent_writes.md) for at-most-once write guarantees. diff --git a/docs/user-guide/src/writing/removing_files.md b/docs/user-guide/src/writing/removing_files.md new file mode 100644 index 0000000000..6387b13d25 --- /dev/null +++ b/docs/user-guide/src/writing/removing_files.md @@ -0,0 +1,220 @@ +# Removing data + +To remove data files from an existing Delta table, you scan the table's file +metadata, select which files to remove, and commit those removals as a +[Transaction](../concepts/architecture.md#transaction). Kernel tracks removes at +file-level granularity. Each removed file produces a **remove action** in the +Delta **transaction log**. + +Before reading this page, make sure you understand +[Appending Data](./append.md) and +[Advanced Reads with scan_metadata()](../reading/scan_metadata.md). + +## The remove flow + +Removing files follows these steps: + +1. Get a `Snapshot` of the table +2. Create a `Transaction` from the snapshot +3. Build a `Scan` and call `scan_metadata()` to get file-level metadata +4. Modify the selection vector to mark files for removal +5. Pass the modified `FilteredEngineData` to `Transaction::remove_files()` +6. Commit the transaction + +```text +Snapshot ──> Transaction + │ +Snapshot ──> Scan ──> scan_metadata() + │ + ScanMetadata { scan_files: FilteredEngineData, ... } + │ + Modify selection vector + │ + txn.remove_files(modified_scan_files) + │ + txn.commit(engine) +``` + +## Getting file metadata with scan_metadata() + +`Scan::scan_metadata()` returns an iterator of `ScanMetadata`. Each +`ScanMetadata` contains a `scan_files` field of type `FilteredEngineData`. +This `FilteredEngineData` holds one row per file in the table, along with a +selection vector that indicates which rows (files) are active. + +The underlying data conforms to the schema returned by `scan_row_schema()`: + +```text +{ + path: string, + size: long, + modificationTime: long, + stats: string, + deletionVector: { + storageType: string, + pathOrInlineDv: string, + offset: int, + sizeInBytes: int, + cardinality: long, + }, + fileConstantValues: { + partitionValues: map, + tags: map, + baseRowId: long, + defaultRowCommitVersion: long, + clusteringProvider: string, + } +} +``` + +You don't need to construct this data yourself. The scan produces it for you. +For full details on working with `scan_metadata()`, see +[Advanced Reads with scan_metadata()](../reading/scan_metadata.md). + +> [!NOTE] +> If your scan was built with a partition predicate (`Scan::with_predicate`), the +> scan-row schema also includes a `partitionValues_parsed` field. Kernel drops that +> field internally when transforming the rows for `remove_files()`, so you don't need +> to handle it. + +## Selecting files for removal + +`FilteredEngineData` pairs engine data with a boolean selection vector. Each +`true` entry marks a row (file) as selected. When you pass a +`FilteredEngineData` to `remove_files()`, Kernel removes every file whose +corresponding selection vector entry is `true`. + +To control which files are removed, decompose the `FilteredEngineData` with +`into_parts()`, modify the selection vector, and reassemble with +`FilteredEngineData::try_new()`: + +```rust,ignore +let (data, mut selection_vector) = scan_files.into_parts(); + +// Example: deselect all files, then select only the ones you want to remove +for entry in selection_vector.iter_mut() { + *entry = false; +} +// Mark specific files for removal (e.g., based on your own logic) +selection_vector[0] = true; + +let files_to_remove = FilteredEngineData::try_new(data, selection_vector)?; +``` + +## Calling remove_files() + +Once you have a `FilteredEngineData` with the correct selection vector, pass +it to the transaction: + +```rust,ignore +txn.remove_files(files_to_remove); +``` + +The signature is: + +```rust,ignore +pub fn remove_files(&mut self, remove_metadata: FilteredEngineData) +``` + +You can call `remove_files()` multiple times to remove files from different +`scan_metadata()` batches. Each call appends to the transaction's list of +pending removals. + +> [!NOTE] +> `remove_files()` is available on transaction states that produce data files (gated by +> the `SupportsDataFiles` trait bound). Metadata-only transaction states cannot register +> file removals. + +## Full example + +This example removes the first file from a filesystem-backed table: + +```rust,no_run +# extern crate delta_kernel; +# extern crate delta_kernel_default_engine; +# extern crate tokio; +# use std::sync::Arc; +# use delta_kernel::committer::FileSystemCommitter; +# use delta_kernel_default_engine::DefaultEngine; +# use delta_kernel_default_engine::storage::store_from_url; +# use delta_kernel::engine_data::FilteredEngineData; +# use delta_kernel::transaction::CommitResult; +# use delta_kernel::{DeltaResult, Snapshot}; +# #[tokio::main] +# async fn main() -> DeltaResult<()> { +# let url = delta_kernel::try_parse_uri("/tmp/my_table")?; +# let engine = DefaultEngine::builder(store_from_url(&url)?).build(); +// 1. Get a snapshot +let snapshot = Snapshot::builder_for(url).build(&engine)?; + +// 2. Create a transaction +let mut txn = snapshot + .clone() + .transaction(Box::new(FileSystemCommitter::new()), &engine)? + .with_operation("DELETE".to_string()); + +// 3. Build a scan and get file metadata +let scan = snapshot.scan_builder().build()?; + +for metadata in scan.scan_metadata(&engine)? { + let metadata = metadata?; + + // 4. Modify the selection vector to pick files for removal + let (data, mut selection_vector) = metadata.scan_files.into_parts(); + + // Deselect everything, then select only the first file + for entry in selection_vector.iter_mut() { + *entry = false; + } + if !selection_vector.is_empty() { + selection_vector[0] = true; + } + + let files_to_remove = FilteredEngineData::try_new(data, selection_vector)?; + + // 5. Register the files for removal + txn.remove_files(files_to_remove); +} + +// 6. Commit the transaction +match txn.commit(&engine)? { + CommitResult::CommittedTransaction(committed) => { + println!("Committed version {}", committed.commit_version()); + } + _ => eprintln!("commit did not succeed"), +} +# Ok(()) +# } +``` + +> [!TIP] +> In practice, you would inspect file statistics or partition values to decide +> which files to remove rather than selecting by index. Use a visitor on the +> `FilteredEngineData` to read per-file metadata before modifying the +> selection vector. + +## Change Data Feed restriction + +> [!WARNING] +> If the table has Change Data Feed enabled +> (`delta.enableChangeDataFeed = true`), you cannot add _and_ remove files in +> the same transaction. Kernel does not yet support writing the CDC files that +> Delta requires for DML operations that both add and remove data. If you +> attempt this, `commit()` returns an error. Use separate transactions: one to +> add files, another to remove files. + +## Blind appends and remove_files() + +A transaction marked with `with_blind_append()` cannot remove files. Blind +appends are optimized for the append-only case and reject any removals at +commit time. If your transaction removes files, do not call +`with_blind_append()`. + +## What's next + +- [Idempotent Writes](./idempotent_writes.md) covers how to make writes + safely repeatable. +- [Domain Metadata](./domain_metadata.md) explains how to attach custom + metadata to a commit. +- [Advanced Reads with scan_metadata()](../reading/scan_metadata.md) covers + the full `scan_metadata()` API used to obtain `FilteredEngineData`. diff --git a/docs/user-guide/theme/catppuccin.css b/docs/user-guide/theme/catppuccin.css new file mode 100644 index 0000000000..c6baaba224 --- /dev/null +++ b/docs/user-guide/theme/catppuccin.css @@ -0,0 +1,844 @@ +/* https://highlightjs.readthedocs.io/en/latest/css-classes-reference.html */ +.latte.hljs { + color: #4c4f69; + background: #eff1f5; +} +.latte .hljs-keyword { + color: #8839ef; +} +.latte .hljs-built_in { + color: #d20f39; +} +.latte .hljs-type { + color: #df8e1d; +} +.latte .hljs-literal { + color: #fe640b; +} +.latte .hljs-number { + color: #fe640b; +} +.latte .hljs-operator { + color: #04a5e5; +} +.latte .hljs-punctuation { + color: #5c5f77; +} +.latte .hljs-property { + color: #179299; +} +.latte .hljs-regexp { + color: #ea76cb; +} +.latte .hljs-string { + color: #40a02b; +} +.latte .hljs-char.escape_ { + color: #40a02b; +} +.latte .hljs-subst { + color: #6c6f85; +} +.latte .hljs-symbol { + color: #dd7878; +} +.latte .hljs-variable { + color: #8839ef; +} +.latte .hljs-variable.language_ { + color: #8839ef; +} +.latte .hljs-variable.constant_ { + color: #fe640b; +} +.latte .hljs-title { + color: #1e66f5; +} +.latte .hljs-title.class_ { + color: #df8e1d; +} +.latte .hljs-title.function_ { + color: #1e66f5; +} +.latte .hljs-params { + color: #4c4f69; +} +.latte .hljs-comment { + color: #7c7f93; +} +.latte .hljs-doctag { + color: #d20f39; +} +.latte .hljs-meta { + color: #fe640b; +} +.latte .hljs-section { + color: #1e66f5; +} +.latte .hljs-tag { + color: #179299; +} +.latte .hljs-name { + color: #8839ef; +} +.latte .hljs-attr { + color: #1e66f5; +} +.latte .hljs-attribute { + color: #40a02b; +} +.latte .hljs-bullet { + color: #179299; +} +.latte .hljs-code { + color: #40a02b; +} +.latte .hljs-emphasis { + color: #d20f39; + font-style: italic; +} +.latte .hljs-strong { + color: #d20f39; + font-weight: bold; +} +.latte .hljs-formula { + color: #179299; +} +.latte .hljs-link { + color: #209fb5; + font-style: italic; +} +.latte .hljs-quote { + color: #40a02b; + font-style: italic; +} +.latte .hljs-selector-tag { + color: #df8e1d; +} +.latte .hljs-selector-id { + color: #1e66f5; +} +.latte .hljs-selector-class { + color: #179299; +} +.latte .hljs-selector-attr { + color: #8839ef; +} +.latte .hljs-selector-pseudo { + color: #179299; +} +.latte .hljs-template-tag { + color: #dd7878; +} +.latte .hljs-template-variable { + color: #dd7878; +} +.latte .hljs-addition { + color: #40a02b; + background: rgba(64, 160, 43, 0.15); +} +.latte .hljs-deletion { + color: #d20f39; + background: rgba(210, 15, 57, 0.15); +} +.latte :is(h1, h2, h3, h4, h5, h6) a code { + color: #4c4f69; +} +.latte a code { + color: #1e66f5; +} +.latte code { + color: #4c4f69; + background: #e6e9ef; +} +.latte blockquote blockquote { + border-top: 0.1em solid #acb0be; + border-bottom: 0.1em solid #acb0be; +} +.latte hr { + border-color: #acb0be; + border-style: solid; +} +.latte del { + color: #7c7f93; +} +.latte .ace_gutter { + color: #8c8fa1; + background: #e6e9ef; +} +.latte .ace_gutter-active-line.ace_gutter-cell { + color: #ea76cb; + background: #e6e9ef; +} +.latte .tooltiptext { + background: #e6e9ef; + color: #4c4f69; +} + +.frappe.hljs { + color: #c6d0f5; + background: #303446; +} +.frappe .hljs-keyword { + color: #ca9ee6; +} +.frappe .hljs-built_in { + color: #e78284; +} +.frappe .hljs-type { + color: #e5c890; +} +.frappe .hljs-literal { + color: #ef9f76; +} +.frappe .hljs-number { + color: #ef9f76; +} +.frappe .hljs-operator { + color: #99d1db; +} +.frappe .hljs-punctuation { + color: #b5bfe2; +} +.frappe .hljs-property { + color: #81c8be; +} +.frappe .hljs-regexp { + color: #f4b8e4; +} +.frappe .hljs-string { + color: #a6d189; +} +.frappe .hljs-char.escape_ { + color: #a6d189; +} +.frappe .hljs-subst { + color: #a5adce; +} +.frappe .hljs-symbol { + color: #eebebe; +} +.frappe .hljs-variable { + color: #ca9ee6; +} +.frappe .hljs-variable.language_ { + color: #ca9ee6; +} +.frappe .hljs-variable.constant_ { + color: #ef9f76; +} +.frappe .hljs-title { + color: #8caaee; +} +.frappe .hljs-title.class_ { + color: #e5c890; +} +.frappe .hljs-title.function_ { + color: #8caaee; +} +.frappe .hljs-params { + color: #c6d0f5; +} +.frappe .hljs-comment { + color: #949cbb; +} +.frappe .hljs-doctag { + color: #e78284; +} +.frappe .hljs-meta { + color: #ef9f76; +} +.frappe .hljs-section { + color: #8caaee; +} +.frappe .hljs-tag { + color: #81c8be; +} +.frappe .hljs-name { + color: #ca9ee6; +} +.frappe .hljs-attr { + color: #8caaee; +} +.frappe .hljs-attribute { + color: #a6d189; +} +.frappe .hljs-bullet { + color: #81c8be; +} +.frappe .hljs-code { + color: #a6d189; +} +.frappe .hljs-emphasis { + color: #e78284; + font-style: italic; +} +.frappe .hljs-strong { + color: #e78284; + font-weight: bold; +} +.frappe .hljs-formula { + color: #81c8be; +} +.frappe .hljs-link { + color: #85c1dc; + font-style: italic; +} +.frappe .hljs-quote { + color: #a6d189; + font-style: italic; +} +.frappe .hljs-selector-tag { + color: #e5c890; +} +.frappe .hljs-selector-id { + color: #8caaee; +} +.frappe .hljs-selector-class { + color: #81c8be; +} +.frappe .hljs-selector-attr { + color: #ca9ee6; +} +.frappe .hljs-selector-pseudo { + color: #81c8be; +} +.frappe .hljs-template-tag { + color: #eebebe; +} +.frappe .hljs-template-variable { + color: #eebebe; +} +.frappe .hljs-addition { + color: #a6d189; + background: rgba(166, 209, 137, 0.15); +} +.frappe .hljs-deletion { + color: #e78284; + background: rgba(231, 130, 132, 0.15); +} +.frappe :is(h1, h2, h3, h4, h5, h6) a code { + color: #c6d0f5; +} +.frappe a code { + color: #8caaee; +} +.frappe code { + color: #c6d0f5; + background: #292c3c; +} +.frappe blockquote blockquote { + border-top: 0.1em solid #626880; + border-bottom: 0.1em solid #626880; +} +.frappe hr { + border-color: #626880; + border-style: solid; +} +.frappe del { + color: #949cbb; +} +.frappe .ace_gutter { + color: #838ba7; + background: #292c3c; +} +.frappe .ace_gutter-active-line.ace_gutter-cell { + color: #f4b8e4; + background: #292c3c; +} +.frappe .tooltiptext { + background: #292c3c; + color: #c6d0f5; +} + +.macchiato.hljs { + color: #cad3f5; + background: #24273a; +} +.macchiato .hljs-keyword { + color: #c6a0f6; +} +.macchiato .hljs-built_in { + color: #ed8796; +} +.macchiato .hljs-type { + color: #eed49f; +} +.macchiato .hljs-literal { + color: #f5a97f; +} +.macchiato .hljs-number { + color: #f5a97f; +} +.macchiato .hljs-operator { + color: #91d7e3; +} +.macchiato .hljs-punctuation { + color: #b8c0e0; +} +.macchiato .hljs-property { + color: #8bd5ca; +} +.macchiato .hljs-regexp { + color: #f5bde6; +} +.macchiato .hljs-string { + color: #a6da95; +} +.macchiato .hljs-char.escape_ { + color: #a6da95; +} +.macchiato .hljs-subst { + color: #a5adcb; +} +.macchiato .hljs-symbol { + color: #f0c6c6; +} +.macchiato .hljs-variable { + color: #c6a0f6; +} +.macchiato .hljs-variable.language_ { + color: #c6a0f6; +} +.macchiato .hljs-variable.constant_ { + color: #f5a97f; +} +.macchiato .hljs-title { + color: #8aadf4; +} +.macchiato .hljs-title.class_ { + color: #eed49f; +} +.macchiato .hljs-title.function_ { + color: #8aadf4; +} +.macchiato .hljs-params { + color: #cad3f5; +} +.macchiato .hljs-comment { + color: #939ab7; +} +.macchiato .hljs-doctag { + color: #ed8796; +} +.macchiato .hljs-meta { + color: #f5a97f; +} +.macchiato .hljs-section { + color: #8aadf4; +} +.macchiato .hljs-tag { + color: #8bd5ca; +} +.macchiato .hljs-name { + color: #c6a0f6; +} +.macchiato .hljs-attr { + color: #8aadf4; +} +.macchiato .hljs-attribute { + color: #a6da95; +} +.macchiato .hljs-bullet { + color: #8bd5ca; +} +.macchiato .hljs-code { + color: #a6da95; +} +.macchiato .hljs-emphasis { + color: #ed8796; + font-style: italic; +} +.macchiato .hljs-strong { + color: #ed8796; + font-weight: bold; +} +.macchiato .hljs-formula { + color: #8bd5ca; +} +.macchiato .hljs-link { + color: #7dc4e4; + font-style: italic; +} +.macchiato .hljs-quote { + color: #a6da95; + font-style: italic; +} +.macchiato .hljs-selector-tag { + color: #eed49f; +} +.macchiato .hljs-selector-id { + color: #8aadf4; +} +.macchiato .hljs-selector-class { + color: #8bd5ca; +} +.macchiato .hljs-selector-attr { + color: #c6a0f6; +} +.macchiato .hljs-selector-pseudo { + color: #8bd5ca; +} +.macchiato .hljs-template-tag { + color: #f0c6c6; +} +.macchiato .hljs-template-variable { + color: #f0c6c6; +} +.macchiato .hljs-addition { + color: #a6da95; + background: rgba(166, 218, 149, 0.15); +} +.macchiato .hljs-deletion { + color: #ed8796; + background: rgba(237, 135, 150, 0.15); +} +.macchiato :is(h1, h2, h3, h4, h5, h6) a code { + color: #cad3f5; +} +.macchiato a code { + color: #8aadf4; +} +.macchiato code { + color: #cad3f5; + background: #1e2030; +} +.macchiato blockquote blockquote { + border-top: 0.1em solid #5b6078; + border-bottom: 0.1em solid #5b6078; +} +.macchiato hr { + border-color: #5b6078; + border-style: solid; +} +.macchiato del { + color: #939ab7; +} +.macchiato .ace_gutter { + color: #8087a2; + background: #1e2030; +} +.macchiato .ace_gutter-active-line.ace_gutter-cell { + color: #f5bde6; + background: #1e2030; +} +.macchiato .tooltiptext { + background: #1e2030; + color: #cad3f5; +} + +.mocha.hljs { + color: #cdd6f4; + background: #1e1e2e; +} +.mocha .hljs-keyword { + color: #cba6f7; +} +.mocha .hljs-built_in { + color: #f38ba8; +} +.mocha .hljs-type { + color: #f9e2af; +} +.mocha .hljs-literal { + color: #fab387; +} +.mocha .hljs-number { + color: #fab387; +} +.mocha .hljs-operator { + color: #89dceb; +} +.mocha .hljs-punctuation { + color: #bac2de; +} +.mocha .hljs-property { + color: #94e2d5; +} +.mocha .hljs-regexp { + color: #f5c2e7; +} +.mocha .hljs-string { + color: #a6e3a1; +} +.mocha .hljs-char.escape_ { + color: #a6e3a1; +} +.mocha .hljs-subst { + color: #a6adc8; +} +.mocha .hljs-symbol { + color: #f2cdcd; +} +.mocha .hljs-variable { + color: #cba6f7; +} +.mocha .hljs-variable.language_ { + color: #cba6f7; +} +.mocha .hljs-variable.constant_ { + color: #fab387; +} +.mocha .hljs-title { + color: #89b4fa; +} +.mocha .hljs-title.class_ { + color: #f9e2af; +} +.mocha .hljs-title.function_ { + color: #89b4fa; +} +.mocha .hljs-params { + color: #cdd6f4; +} +.mocha .hljs-comment { + color: #9399b2; +} +.mocha .hljs-doctag { + color: #f38ba8; +} +.mocha .hljs-meta { + color: #fab387; +} +.mocha .hljs-section { + color: #89b4fa; +} +.mocha .hljs-tag { + color: #94e2d5; +} +.mocha .hljs-name { + color: #cba6f7; +} +.mocha .hljs-attr { + color: #89b4fa; +} +.mocha .hljs-attribute { + color: #a6e3a1; +} +.mocha .hljs-bullet { + color: #94e2d5; +} +.mocha .hljs-code { + color: #a6e3a1; +} +.mocha .hljs-emphasis { + color: #f38ba8; + font-style: italic; +} +.mocha .hljs-strong { + color: #f38ba8; + font-weight: bold; +} +.mocha .hljs-formula { + color: #94e2d5; +} +.mocha .hljs-link { + color: #74c7ec; + font-style: italic; +} +.mocha .hljs-quote { + color: #a6e3a1; + font-style: italic; +} +.mocha .hljs-selector-tag { + color: #f9e2af; +} +.mocha .hljs-selector-id { + color: #89b4fa; +} +.mocha .hljs-selector-class { + color: #94e2d5; +} +.mocha .hljs-selector-attr { + color: #cba6f7; +} +.mocha .hljs-selector-pseudo { + color: #94e2d5; +} +.mocha .hljs-template-tag { + color: #f2cdcd; +} +.mocha .hljs-template-variable { + color: #f2cdcd; +} +.mocha .hljs-addition { + color: #a6e3a1; + background: rgba(166, 227, 161, 0.15); +} +.mocha .hljs-deletion { + color: #f38ba8; + background: rgba(243, 139, 168, 0.15); +} +.mocha :is(h1, h2, h3, h4, h5, h6) a code { + color: #cdd6f4; +} +.mocha a code { + color: #89b4fa; +} +.mocha code { + color: #cdd6f4; + background: #181825; +} +.mocha blockquote blockquote { + border-top: 0.1em solid #585b70; + border-bottom: 0.1em solid #585b70; +} +.mocha hr { + border-color: #585b70; + border-style: solid; +} +.mocha del { + color: #9399b2; +} +.mocha .ace_gutter { + color: #7f849c; + background: #181825; +} +.mocha .ace_gutter-active-line.ace_gutter-cell { + color: #f5c2e7; + background: #181825; +} +.mocha .tooltiptext { + background: #181825; + color: #cdd6f4; +} + +.latte { + --bg: #eff1f5; + --fg: #4c4f69; + --sidebar-bg: #e6e9ef; + --sidebar-fg: #4c4f69; + --sidebar-non-existant: #9ca0b0; + --sidebar-active: #1e66f5; + --sidebar-spacer: #9ca0b0; + --scrollbar: #9ca0b0; + --icons: #9ca0b0; + --icons-hover: #7c7f93; + --links: #1e66f5; + --inline-code-color: #4c4f69; + --theme-popup-bg: #e6e9ef; + --theme-popup-border: #9ca0b0; + --theme-hover: #ccd0da; + --quote-bg: #e6e9ef; + --quote-border: #dce0e8; + --table-border-color: #ccd0da; + --table-header-bg: #e6e9ef; + --table-alternate-bg: #e6e9ef; + --searchbar-border-color: #ccd0da; + --searchbar-bg: #e6e9ef; + --searchbar-fg: #4c4f69; + --searchbar-shadow-color: #dce0e8; + --searchresults-header-fg: #4c4f69; + --searchresults-border-color: #ccd0da; + --searchresults-li-bg: #eff1f5; + --search-mark-bg: #fe640b; + --warning-border: #fe640b; + --color-scheme: light; + --copy-button-filter: brightness(0) saturate(100%) invert(47%) sepia(6%) saturate(1263%) hue-rotate(195deg) brightness(90%) contrast(81%); + --copy-button-filter-hover: brightness(0) saturate(100%) invert(30%) sepia(80%) saturate(1850%) hue-rotate(209deg) brightness(94%) contrast(105%); +} + +.frappe { + --bg: #303446; + --fg: #c6d0f5; + --sidebar-bg: #292c3c; + --sidebar-fg: #c6d0f5; + --sidebar-non-existant: #737994; + --sidebar-active: #8caaee; + --sidebar-spacer: #737994; + --scrollbar: #737994; + --icons: #737994; + --icons-hover: #949cbb; + --links: #8caaee; + --inline-code-color: #c6d0f5; + --theme-popup-bg: #292c3c; + --theme-popup-border: #737994; + --theme-hover: #414559; + --quote-bg: #292c3c; + --quote-border: #232634; + --table-border-color: #414559; + --table-header-bg: #292c3c; + --table-alternate-bg: #292c3c; + --searchbar-border-color: #414559; + --searchbar-bg: #292c3c; + --searchbar-fg: #c6d0f5; + --searchbar-shadow-color: #232634; + --searchresults-header-fg: #c6d0f5; + --searchresults-border-color: #414559; + --searchresults-li-bg: #303446; + --search-mark-bg: #ef9f76; + --warning-border: #ef9f76; + --color-scheme: dark; + --copy-button-filter: brightness(0) saturate(100%) invert(82%) sepia(6%) saturate(1287%) hue-rotate(192deg) brightness(86%) contrast(85%); + --copy-button-filter-hover: brightness(0) saturate(100%) invert(68%) sepia(16%) saturate(1070%) hue-rotate(185deg) brightness(96%) contrast(95%); +} + +.macchiato { + --bg: #24273a; + --fg: #cad3f5; + --sidebar-bg: #1e2030; + --sidebar-fg: #cad3f5; + --sidebar-non-existant: #6e738d; + --sidebar-active: #8aadf4; + --sidebar-spacer: #6e738d; + --scrollbar: #6e738d; + --icons: #6e738d; + --icons-hover: #939ab7; + --links: #8aadf4; + --inline-code-color: #cad3f5; + --theme-popup-bg: #1e2030; + --theme-popup-border: #6e738d; + --theme-hover: #363a4f; + --quote-bg: #1e2030; + --quote-border: #181926; + --table-border-color: #363a4f; + --table-header-bg: #1e2030; + --table-alternate-bg: #1e2030; + --searchbar-border-color: #363a4f; + --searchbar-bg: #1e2030; + --searchbar-fg: #cad3f5; + --searchbar-shadow-color: #181926; + --searchresults-header-fg: #cad3f5; + --searchresults-border-color: #363a4f; + --searchresults-li-bg: #24273a; + --search-mark-bg: #f5a97f; + --warning-border: #f5a97f; + --color-scheme: dark; + --copy-button-filter: brightness(0) saturate(100%) invert(75%) sepia(18%) saturate(361%) hue-rotate(190deg) brightness(91%) contrast(86%); + --copy-button-filter-hover: brightness(0) saturate(100%) invert(67%) sepia(17%) saturate(1007%) hue-rotate(183deg) brightness(99%) contrast(94%); +} + +.mocha { + --bg: #1e1e2e; + --fg: #cdd6f4; + --sidebar-bg: #181825; + --sidebar-fg: #cdd6f4; + --sidebar-non-existant: #6c7086; + --sidebar-active: #89b4fa; + --sidebar-spacer: #6c7086; + --scrollbar: #6c7086; + --icons: #6c7086; + --icons-hover: #9399b2; + --links: #89b4fa; + --inline-code-color: #cdd6f4; + --theme-popup-bg: #181825; + --theme-popup-border: #6c7086; + --theme-hover: #313244; + --quote-bg: #181825; + --quote-border: #11111b; + --table-border-color: #313244; + --table-header-bg: #181825; + --table-alternate-bg: #181825; + --searchbar-border-color: #313244; + --searchbar-bg: #181825; + --searchbar-fg: #cdd6f4; + --searchbar-shadow-color: #11111b; + --searchresults-header-fg: #cdd6f4; + --searchresults-border-color: #313244; + --searchresults-li-bg: #1e1e2e; + --search-mark-bg: #fab387; + --warning-border: #fab387; + --color-scheme: dark; + --copy-button-filter: brightness(0) saturate(100%) invert(84%) sepia(9%) saturate(767%) hue-rotate(192deg) brightness(84%) contrast(84%); + --copy-button-filter-hover: brightness(0) saturate(100%) invert(68%) sepia(18%) saturate(951%) hue-rotate(180deg) brightness(98%) contrast(100%); +} diff --git a/docs/user-guide/theme/index.hbs b/docs/user-guide/theme/index.hbs new file mode 100644 index 0000000000..ecc9306733 --- /dev/null +++ b/docs/user-guide/theme/index.hbs @@ -0,0 +1,366 @@ + + + + + + {{ title }} + {{#if is_print }} + + {{/if}} + {{#if base_url}} + + {{/if}} + + + + {{> head}} + + + + + + {{#if favicon_svg}} + + {{/if}} + {{#if favicon_png}} + + {{/if}} + + + + {{#if print_enable}} + + {{/if}} + + + + + + + + + + + {{#each additional_css}} + + {{/each}} + + {{#if mathjax_support}} + + + {{/if}} + + + + + + + +
    +
    +

    Keyboard shortcuts

    +
    +

    Press ← or → to navigate between chapters

    + {{#if search_enabled}} +

    Press S or / to search in the book

    + {{/if}} +

    Press ? to show this help

    +

    Press Esc to hide this help

    +
    +
    +
    +
    + + + + + + + + + + + + + +
    + +
    + {{> header}} +
    +     + + {{#if search_enabled}} + + {{/if}} + + + + +
    +
    + {{{ content }}} +
    + + +
    +
    + + + +
    + + + + + + + + {{#if live_reload_endpoint}} + + + {{/if}} + + {{#if playground_line_numbers}} + + {{/if}} + + {{#if playground_copyable}} + + {{/if}} + + {{#if playground_js}} + + + + + + {{/if}} + + {{#if search_js}} + + + + {{/if}} + + + + + + + {{#each additional_js}} + + {{/each}} + + {{#if is_print}} + {{#if mathjax_support}} + + {{else}} + + {{/if}} + {{/if}} + + {{#if fragment_map}} + + {{/if}} + +
    + + diff --git a/feature-tests/Cargo.toml b/feature-tests/Cargo.toml index 58984cba47..7983e4edbf 100644 --- a/feature-tests/Cargo.toml +++ b/feature-tests/Cargo.toml @@ -13,8 +13,16 @@ version.workspace = true release = false [dependencies] -delta_kernel = { path = "../kernel", features = ["arrow"] } +delta_kernel = { path = "../kernel" } +delta_kernel_default_engine = { path = "../default-engine", default-features = false, optional = true } +# Direct deps for TLS crypto provider regression tests (see lib.rs). +reqwest = { version = "0.13", default-features = false, optional = true } +rustls = { version = "0.23", default-features = false, optional = true } [features] -default-engine-native-tls= [ "delta_kernel/default-engine-native-tls" ] -default-engine-rustls = [ "delta_kernel/default-engine-rustls" ] +default = ["arrow"] +arrow = ["arrow-59"] +arrow-59 = ["delta_kernel/arrow-59"] +arrow-58 = ["delta_kernel/arrow-58"] +default-engine-native-tls = ["delta_kernel/default-engine-base", "dep:delta_kernel_default_engine", "delta_kernel_default_engine/arrow", "delta_kernel_default_engine/native-tls", "dep:rustls", "reqwest/native-tls"] +default-engine-rustls = ["delta_kernel/default-engine-base", "dep:delta_kernel_default_engine", "delta_kernel_default_engine/arrow", "delta_kernel_default_engine/rustls", "dep:rustls", "reqwest/rustls"] diff --git a/feature-tests/src/lib.rs b/feature-tests/src/lib.rs index 46355e5410..f78397ff17 100644 --- a/feature-tests/src/lib.rs +++ b/feature-tests/src/lib.rs @@ -3,7 +3,7 @@ /// /// Run (from workspace root) with: /// 1. `cargo b -p feature_tests --features default-engine-rustls` -/// 2. `cargo b -p feature_tests --features default-engine` +/// 2. `cargo b -p feature_tests --features default-engine-native-tls` /// /// These run in our build CI. pub fn test_default_engine_feature_flags() { @@ -13,6 +13,54 @@ pub fn test_default_engine_feature_flags() { ))] { #[allow(unused_imports)] - use delta_kernel::engine::default::DefaultEngine; + use delta_kernel_default_engine::DefaultEngine; + } +} + +/// Regression tests for rustls crypto provider conflicts. +/// +/// rustls 0.23 panics at runtime if both `aws-lc-rs` and `ring` features are active and no +/// provider is explicitly installed. object_store always brings `ring` transitively, so kernel +/// must avoid adding `aws-lc-rs` to the same rustls instance. +/// +/// Two APIs are tested because they behave differently: +/// - `rustls::ClientConfig::builder()` relies on auto-detection and panics on dual providers. +/// - `reqwest::Client::new()` explicitly constructs its provider, so it always succeeds. +#[cfg(test)] +mod tests { + // Verifies that `default-engine-native-tls` does not leak aws-lc-rs into the rustls + // feature set. If this panics, kernel's reqwest is pulling in `default-tls` again. + // + // Excluded when `default-engine-rustls` is also active (e.g. --all-features) because + // that feature inherently adds aws-lc-rs, making the dual-provider panic unavoidable. + #[test] + #[cfg(all( + feature = "default-engine-native-tls", + not(feature = "default-engine-rustls") + ))] + fn test_native_tls_rustls_builder_no_dual_provider_panic() { + let _config = rustls::ClientConfig::builder(); + } + + #[test] + #[cfg(feature = "default-engine-native-tls")] + fn test_native_tls_reqwest_client_no_panic() { + let _client = reqwest::Client::new(); + } + + // `default-engine-rustls` has an inherent dual-provider conflict: reqwest's `rustls` + // feature brings aws-lc-rs while object_store brings ring. This is an upstream limitation. + // If this test stops panicking, the upstream issue is fixed and the annotation can go. + #[test] + #[cfg(feature = "default-engine-rustls")] + #[should_panic(expected = "Could not automatically determine the process-level CryptoProvider")] + fn test_rustls_rustls_builder_has_dual_provider_panic() { + let _config = rustls::ClientConfig::builder(); + } + + #[test] + #[cfg(feature = "default-engine-rustls")] + fn test_rustls_reqwest_client_no_panic() { + let _client = reqwest::Client::new(); } } diff --git a/ffi/CLAUDE.md b/ffi/CLAUDE.md index 57dcac3ce8..1361398025 100644 --- a/ffi/CLAUDE.md +++ b/ffi/CLAUDE.md @@ -29,19 +29,155 @@ the caller's memory space. - `src/handle.rs` -- opaque handle system for passing Rust objects across FFI - `src/scan.rs` -- scan FFI interface - `src/schema_visitor.rs` -- visitor pattern for schema traversal +- `src/ffi_tracing.rs` -- log/tracing and metrics callback registration (`#[cfg(feature = "tracing")]`) +- `src/ffi_metrics.rs` -- `repr(C)` mirror of kernel `MetricEvent` types (`#[cfg(feature = "tracing")]`) +- `src/alloc_stats.rs` -- `peak_alloc` global allocator and native-heap FFI getters + (`alloc-tracking`) ## Read Flow ``` -get_default_engine() -> snapshot() -> scan() -> scan_metadata() -> read + transform +get_default_engine() -> get_snapshot_builder() -> snapshot_builder_build() -> scan() -> scan_metadata() -> read + transform ``` +Snapshot builder API (`ffi/src/lib.rs`): +- `get_snapshot_builder(path, engine)` -- fresh snapshot from a table path +- `get_snapshot_builder_from(old_snapshot, engine)` -- incremental update reusing an existing snapshot (avoids re-reading the log) +- `snapshot_builder_set_version(builder, version)` -- optional: pin to a specific version +- `snapshot_builder_set_log_tail(builder, log_tail)` -- optional: set log tail (for catalog-managed tables) +- `snapshot_builder_set_max_catalog_version(builder, version)` -- optional: set max catalog version (for catalog-managed tables) +- `snapshot_builder_build(builder)` -- consume the builder and produce a `SharedSnapshot` +- `free_snapshot_builder(builder)` -- discard without building (e.g. on error paths) + +The caller owns the returned builder handle and must call either `snapshot_builder_build` or `free_snapshot_builder`. + +Snapshot accessors (`ffi/src/lib.rs`) read a built `SharedSnapshot` without I/O -- e.g. `version`, +`snapshot_timestamp`, and `snapshot_file_stats`, which returns `OptionalValue` (scalar +`num_files` / `table_size_bytes` from the CRC; `None` when the snapshot has no CRC, or its CRC lacks +complete file stats). + +Domain-metadata reads live in `ffi/src/domain_metadata.rs`: `get_domain_metadata` / +`visit_domain_metadata` for user domains, and `visit_clustering_columns`, which reports one +descriptor per clustering column -- logical name, physical name (what per-file stats are keyed on), +and a type tag -- without exposing the guarded `delta.*` domain JSON directly. It returns +`OptionalValue`: `None` means not clustered, `Some(0)` means clustered on no columns. The +type tag reuses the `visit_expression_literal_null` encoding, with 255 for types that don't fit a +compact tag (struct, array, map, variant, void, and geometry/geography). + +## Commit Range Flow + +A `CommitRange` describes a contiguous range of a table's commits. Build one via the commit range +builder (`ffi/src/commit_range.rs`): + +``` +commit_range_builder_for(path, start_version, engine) + -> commit_range_builder_set_end_version(builder, end_version) // optional; else latest version + -> commit_range_builder_build(builder) // -> SharedCommitRange, always consume builder + -> commit_range_commits(range, engine, actions, actions_len) // -> SharedCommitActionsIterator + // or commit_range_commits_with_snapshot(range, engine, start_snapshot, actions, actions_len) + -> commit_range_commits_next(iter, ctx, visitor) // visitor receives a SharedCommitAction + // in the visitor: commit_action_version / commit_action_timestamp + // commit_action_get_actions(action, engine) -> ExclusiveFileReadResultIterator + // -> read_result_next(...) -> free_read_result_iter(...) +``` + +The caller owns the builder and must call either `commit_range_builder_build` or +`free_commit_range_builder`. Release the range with `free_commit_range` and the commits iterator +with `free_commit_actions_iter`. Each `SharedCommitAction` handed to the visitor must be released +with `free_commit_action`. + +## Incremental Scan Flow + +An incremental scan streams the file-action diff between a base version and a target snapshot +(`ffi/src/incremental_scan.rs`): + +``` +snapshot_incremental_scan_builder(snapshot, base_version, engine) + -> incremental_scan_builder_with_predicate(builder, engine, predicate) // optional; prunes live Adds + -> incremental_scan_builder_build(builder) // -> OptionalValue; None => full-scan fallback + -> incremental_scan_stream_next_arrow(stream)* // optional: pull live-Add batches as Arrow + -> incremental_scan_stream_into_summary(stream) // live-Add / Remove key sets; consumes the stream +``` + +The module-level docs in `incremental_scan.rs` are the source of truth for the error contract +(any `next_arrow` error kills the stream), the `OptionalValue::None` full-scan-fallback signal, +the pass-through-field version caveat (kernel issue #2552), and handle release. The Arrow batch +reuses `ScanMetadataArrowResult` with null `transforms`. + ## Write Flow ``` -get_default_engine() -> transaction() -> with_engine_info() -> add_files() -> commit() +get_default_engine() -> transaction() -> with_engine_info() -> with_operation() -> add_files() -> commit() + | + v + committed_transaction_version / committed_transaction_post_commit_snapshot + | + v + free_committed_transaction ``` +`commit()` and `create_table_commit()` return a `Handle` that the caller can read via `committed_transaction_version` and `committed_transaction_post_commit_snapshot`, then must release with `free_committed_transaction`. The post-commit snapshot, when present, is a separate `SharedSnapshot` handle that must be freed with `free_snapshot`. + +Write context: `get_unpartitioned_write_context` covers unpartitioned tables. For partitioned tables, build a `PartitionValueMap` (`partition_value_map_new` + the typed `partition_value_map_insert_*` functions, one entry per partition column keyed by logical name) and pass it to `get_partitioned_write_context` (consumes the map). Then use `get_write_dir` for the partition's target directory (Hive-style prefix or random prefix), `visit_partition_values` to read the physical `partitionValues` to record in each Add action, and `resolve_file_path` to turn a written file's URL into its relative `add.path`. The `create_table_*` variants apply the same flow to a create-table transaction whose partition columns were declared with `create_table_builder_with_partition_columns`. + +Catalog-managed publish flow (after a catalog committer stages commits): + +``` +committed_transaction_post_commit_snapshot() + -> snapshot_publish_with_committer(snapshot, committer, engine) + // borrows snapshot; consumes committer (do not free) + -> use returned snapshot for subsequent transaction_with_committer / checkpoint + // mint a fresh get_uc_committer for that transaction -- it also consumes + -> free_snapshot (returned snapshot) when done + -> free_snapshot (post-commit input snapshot) +``` + +`snapshot_publish_with_committer` mirrors kernel `Snapshot::publish`: it copies ratified staged +commits into `_delta_log/` via the catalog committer's `publish()` implementation. The input +snapshot is borrowed; the committer is consumed (do not free). The caller owns the returned +snapshot handle. The returned snapshot carries the published watermark (`max_published_version`) +needed for the next catalog commit; do not continue from the pre-publish post-commit snapshot. + +Deletion vector update flow: + +``` +transaction() + -> dv_descriptor_map_new() + -> dv_descriptor_new() + -> dv_descriptor_map_insert() + -> scan() -> scan_metadata_iter_init() + -> transaction_update_deletion_vectors() + -> commit() +``` + +The engine authors the DV file and passes descriptor fields to `dv_descriptor_new`. The +descriptor map and scan iterator are both consumed by `transaction_update_deletion_vectors`; +descriptor handles are consumed by `dv_descriptor_map_insert` only on success and must be +freed by the caller on error. DV updates require both the `deletionVectors` reader/writer +feature and `delta.enableDeletionVectors=true`. + +## Tracing & Metrics + +Gated behind the `tracing` feature. A single global `tracing` subscriber backs both logging and +metrics; it is installed lazily the first time any `enable_*` function below is called. The +subscriber has two reloadable slots: a logging layer (swapped wholesale between event-based and +log-line formats) and a metrics layer (a fixed `ReportGeneratorLayer` toggled on/off via a +reloadable level filter). + +Logging registration (each re-callable to replace the active callback, format, and level): +- `enable_event_tracing(callback, max_level)` -- structured `Event`s; the engine formats them +- `enable_log_line_tracing(callback, max_level)` -- pre-formatted log lines, default options +- `enable_formatted_log_line_tracing(callback, max_level, format, ansi, with_time, with_level, with_target)` + -- pre-formatted log lines with explicit formatting options + +Metrics registration: +- `enable_metrics_reporting(callback)` -- forwards each kernel `MetricEvent` to the callback as a + `repr(C)` `MetricEvent` (see `src/ffi_metrics.rs`). Re-calling replaces the callback. + +The `MetricEvent` and any `KernelStringSlice` it carries are only valid for the duration of the +callback. Durations are `u64`, suffixed `_ns` (nanoseconds) or `_ms` (milliseconds). Operation ids +are the raw 16 bytes of the kernel UUID (`MetricId`). + ## Building ```bash @@ -49,4 +185,56 @@ cargo build -p delta_kernel_ffi --release # Headers written to target/ffi-headers/ ``` -Feature flags: `default-engine-rustls` (default), `default-engine-native-tls`, `catalog-managed`, `tracing`. +Feature flags: +- `default-engine-rustls` (default) +- `default-engine-native-tls` +- `arrow` (default; currently maps to `arrow-59`) +- `arrow-59`, `arrow-58` +- `delta-kernel-unity-catalog` +- `tracing` +- `alloc-tracking` -- installs `peak_alloc` as the tracking global allocator; enables meaningful + `*_native_bytes` / `alloc_tracking_enabled` getters (cdylib only; conflicts with + another `#[global_allocator]` if linked as an rlib) + +## Testing under Miri + +CI runs this crate's tests under Miri (the `miri` job in `build.yml`) to catch undefined +behavior in the `unsafe` FFI boundary: raw-pointer reads/writes, `unsafe impl Send/Sync`, +`Handle` conversions (`as_ref` / `clone_as_arc` / `into_inner`), and `free_*`. Miri is a MIR +interpreter, so it runs 10x-100x slower than native and is billed by the interpreted +instruction, not wall-clock work. + +The consequence: a test is expensive under Miri in proportion to how much *code it executes*, +not how much it asserts. Tests that construct heavyweight but **safe** machinery -- a +reqwest/rustls client (crypto init), a multi-threaded tokio runtime, an Arrow/parquet write -- +cost minutes under the interpreter while exercising none of our `unsafe`. That time buys no +UB detection. + +Guidance for adding or triaging FFI tests: + +- **Keep under Miri** any test that executes `unsafe` whose correctness Miri can check. This is + the reason the job exists; do not skip these for speed. +- **Never add new `unsafe` to a Miri-ignored test.** An ignored test is invisible to Miri, so + `unsafe` introduced in one is never checked for undefined behavior, and nothing in CI reports + the gap. When a change needs new `unsafe` in an ignored test, either exercise that `unsafe` + from a test that runs under Miri, or un-skip the test. +- **`#[cfg_attr(miri, ignore)]` is legitimate for two reasons, and only these two:** + 1. Miri cannot run it (e.g. an unsupported foreign function). Before accepting this, check + whether the blocker is avoidable: local-filesystem storage reaches `std::fs::hard_link` + (`linkat`), which Miri rejects, but in-memory storage does not. Prefer an in-memory store. + 2. The test executes no `unsafe`, OR only `unsafe` that a kept test already covers, AND it is + expensive under Miri. Skip only with the coverage argument; skipping for cost alone drops + UB coverage. +- When you skip under reason 2, **prove the coverage is preserved**: the kept tests' set of + `unsafe` FFI functions must be a superset of the skipped test's. Name the covering test in the + `ignore` reason or a nearby comment so a future reader can re-check it. +- Miri's leak check also flags handles a test itself forgot to free. Triage before assuming a bug + in the code under test: a missing `free_*` in the test is a test fix, while an unjoined + background thread at teardown can be an artifact of how the test ends. +- Prefer picking the **cheapest** test that crosses a given `unsafe` path over keeping several + that cross the same path with more safe work each. +- `rest_engine` and the checkpoint tests in `lib.rs` document worked examples of this split. +- `-Zmiri-provenance-gc=1000000` on the Miri step is a pure speed knob (GC frequency) and does + not weaken detection. Do NOT add `-Zmiri-disable-stacked-borrows`, `-disable-validation`, + `-disable-data-race-detector`, or `-Zmiri-preemption-rate=0`: the first three are unsound, and + the last reduces data-race schedule exploration. diff --git a/ffi/Cargo.toml b/ffi/Cargo.toml index beb0867aa4..b2b8e02057 100644 --- a/ffi/Cargo.toml +++ b/ffi/Cargo.toml @@ -18,16 +18,20 @@ release = false crate-type = ["lib", "cdylib", "staticlib"] [dependencies] +bytes = "1.10" +prost = { version = "0.13", optional = true } tracing = "0.1" tracing-core = { version = "0.1", optional = true } tracing-subscriber = { version = "0.3", optional = true, features = [ "json" ] } url = "2" +peak_alloc = { version = "0.3.0", optional = true } delta_kernel = { path = "../kernel", default-features = false, features = [ "internal-api", ] } -delta_kernel_ffi_macros = { path = "../ffi-proc-macros", version = "0.20.0" } -uc-client = { path = "../uc-client", optional = true } -uc-catalog = { path = "../uc-catalog", optional = true } +delta_kernel_default_engine = { path = "../default-engine", default-features = false, optional = true } +delta_kernel_ffi_macros = { path = "../ffi-proc-macros", version = "0.27.1" } +delta-kernel-unity-catalog = { path = "../delta-kernel-unity-catalog", optional = true } +unity-catalog-delta-client-api = { path = "../unity-catalog-delta-client-api", optional = true } [build-dependencies] cbindgen = "0.29.2" @@ -38,24 +42,45 @@ paste = "1.0" rand = "0.9.2" serde = "1.0.219" serde_json = "1.0.142" -test_utils = { path = "../test-utils" } +test_utils = { path = "../test-utils", default-features = false } tokio = { version = "1.47" } trybuild = "1.0" tempfile = "3.20.0" itertools = "0.14.0" +rstest = "0.23" [features] -default = ["default-engine-rustls"] -default-engine-native-tls = ["delta_kernel/default-engine-native-tls", "default-engine-base"] -default-engine-rustls = ["delta_kernel/default-engine-rustls", "default-engine-base"] -catalog-managed = ["delta_kernel/catalog-managed"] -uc-catalog = [ "dep:uc-catalog", "dep:uc-client" ] +default = ["default-engine-rustls", "arrow"] +default-engine-native-tls = ["delta_kernel/default-engine-base", "dep:delta_kernel_default_engine", "delta_kernel_default_engine/native-tls", "default-engine-base"] +default-engine-rustls = ["delta_kernel/default-engine-base", "dep:delta_kernel_default_engine", "delta_kernel_default_engine/rustls", "default-engine-base"] +delta-kernel-unity-catalog = [ "dep:delta-kernel-unity-catalog", "dep:unity-catalog-delta-client-api" ] +arrow = ["arrow-59"] +arrow-59 = ["delta_kernel/arrow-59"] +arrow-58 = ["delta_kernel/arrow-58"] -# This is an 'internal' feature flag which has all the shared bits from default-engine-native-tls and -# default-engine-rustls. There is a check in kernel/lib.rs to ensure you have enabled one of -# default-engine-native-tls or default-engine-rustls, so default-engine-base will not work by itself -default-engine-base = ["delta_kernel/default-engine-base", "delta_kernel/arrow"] +# Internal feature flag with the shared bits between default-engine-native-tls and +# default-engine-rustls. Enable one of those two on the FFI crate instead to also pick a TLS backend. +default-engine-base = ["delta_kernel/default-engine-base", "dep:delta_kernel_default_engine"] + +# Declarative plan execution via PlanExecutor trait (experimental). +# +# Requires `arrow` because iterators returned from plan execution use Arrow to pass along data +# batches, and requires `default-engine-base` because the `PlanBasedEngine` FFI constructor pairs +# the plan executor with `ArrowEvaluationHandler` +declarative-plans = ["delta_kernel/declarative-plans", "dep:prost", "arrow", "default-engine-base"] + +vendored-protoc = ["delta_kernel/vendored-protoc"] tracing = [ "tracing-core", "tracing-subscriber" ] internal-api = [] test-ffi = [] + +# Experimental, in-development feature flag gating in-progress geospatial +# (geometry/geography) type support. TODO(#2949): remove once full geo support ships. +geo-type-in-dev = ["delta_kernel/geo-type-in-dev"] + +# Tracking global allocator for process-wide native live/peak bytes, readable via the +# `*_native_bytes` FFI getters. Off by default. Only enable when building the cdylib: +# this crate is also an rlib, so `#[global_allocator]` applies to any Rust binary that +# links it and conflicts with a binary that installs its own (jemalloc, dhat, ...). +alloc-tracking = ["dep:peak_alloc"] diff --git a/ffi/build.rs b/ffi/build.rs index 40f9410b04..8bc1efddc7 100644 --- a/ffi/build.rs +++ b/ffi/build.rs @@ -1,9 +1,10 @@ extern crate cbindgen; -use cbindgen::{Config, Language}; use std::env; use std::path::{Path, PathBuf}; +use cbindgen::{Config, Language}; + fn get_target_dir(manifest_dir: &str) -> PathBuf { if let Ok(target) = env::var("CARGO_TARGET_DIR") { // allow a CARGO_TARGET_DIR var that could be set via e.g. cmake to override where we put diff --git a/ffi/cbindgen.toml b/ffi/cbindgen.toml index 13e11b3f4d..ea964dc134 100644 --- a/ffi/cbindgen.toml +++ b/ffi/cbindgen.toml @@ -11,7 +11,6 @@ namespace = "ffi" cpp_compat = true [defines] -"feature = default-engine" = "DEFINE_DEFAULT_ENGINE" "feature = default-engine-rustls" = "DEFINE_DEFAULT_ENGINE_RUSTLS" "feature = default-engine-base" = "DEFINE_DEFAULT_ENGINE_BASE" diff --git a/ffi/examples/checkpoint-table/CMakeLists.txt b/ffi/examples/checkpoint-table/CMakeLists.txt new file mode 100644 index 0000000000..d68f1eed87 --- /dev/null +++ b/ffi/examples/checkpoint-table/CMakeLists.txt @@ -0,0 +1,33 @@ +cmake_minimum_required(VERSION 3.12) +project(checkpoint_table) + +add_executable(checkpoint_table checkpoint_table.c ../common/kernel_utils.c) +target_compile_definitions(checkpoint_table PUBLIC DEFINE_DEFAULT_ENGINE_BASE) +target_include_directories(checkpoint_table PUBLIC + "${CMAKE_CURRENT_SOURCE_DIR}/../../../target/ffi-headers" + "${CMAKE_CURRENT_SOURCE_DIR}/../common") +target_link_directories(checkpoint_table PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/../../../target/debug") +target_link_libraries(checkpoint_table PUBLIC delta_kernel_ffi) + +if(WIN32) + set(CMAKE_C_FLAGS_DEBUG "/MT") + target_link_libraries(checkpoint_table PUBLIC ws2_32 userenv bcrypt ncrypt crypt32 secur32 ntdll RuntimeObject) +endif(WIN32) + +if(MSVC) + target_compile_options(checkpoint_table PRIVATE /W4 /WX) +else() + target_compile_options(checkpoint_table PRIVATE -Wall -Wextra -Wpedantic -Werror -g -fsanitize=address) + target_link_options(checkpoint_table PRIVATE -g -fsanitize=address) +endif() + +# Testing +include(CTest) +set(TestRunner "../../../tests/checkpoint-table-testing/run_test.sh") +set(ExpectedPath "../../../tests/checkpoint-table-testing/expected-data") +add_test(NAME checkpoint_table_default + COMMAND ${TestRunner} default ${ExpectedPath}/default.expected) +add_test(NAME checkpoint_table_v2_no_sidecar + COMMAND ${TestRunner} v2_no_sidecar ${ExpectedPath}/v2_no_sidecar.expected) +add_test(NAME checkpoint_table_v2_with_sidecars + COMMAND ${TestRunner} v2_with_sidecars ${ExpectedPath}/v2_with_sidecars.expected) diff --git a/ffi/examples/checkpoint-table/README.md b/ffi/examples/checkpoint-table/README.md new file mode 100644 index 0000000000..442fac827d --- /dev/null +++ b/ffi/examples/checkpoint-table/README.md @@ -0,0 +1,42 @@ +# checkpoint-table + +C FFI example for the checkpoint write surface. Demonstrates the +`checkpoint_snapshot(snapshot, engine, spec)` function and its +`FfiCheckpointSpec` discriminated union (`V1` / `V2NoSidecar` / +`V2WithSidecar { file_actions_per_sidecar_hint: OptionalValue }`) plus the +`FfiCheckpointWriteResult` return type (`Written` / `AlreadyExists`, +each carrying an owned `Handle`). + +## Building + +```bash +# from repo root +$ cargo build -p delta_kernel_ffi +# from this directory +$ mkdir build && cd build && cmake .. && make +$ ./checkpoint_table /path/to/existing/table +``` + +The `` argument selects one of three demos: + +- `default` -- pass `spec = NULL`. The kernel auto-picks V1 or V2 from the + table's protocol features and writes a single-file checkpoint with no sidecars. +- `v2_no_sidecar` -- `FfiCheckpointSpec::V2NoSidecar` requests a V2 manifest + with all file actions inlined (no sidecar files). Requires the table to + declare the `v2Checkpoint` reader/writer feature. +- `v2_with_sidecars` -- `FfiCheckpointSpec::V2WithSidecar` with + `file_actions_per_sidecar_hint = Some(2)` requests a V2 manifest that emits + sidecar parquet files. The hint is the suggested upper bound of file actions + per sidecar parquet (the example passes `2` so multiple sidecars are emitted + even for the small fixture); pass `None` to use the kernel default. + Requires the table to declare the `v2Checkpoint` feature. + +## Test harness + +The paired `ffi/tests/checkpoint-table-testing/run_test.sh` seeds a temporary +table by copying one of the kernel-bundled fixtures +(`kernel/tests/data/app-txn-no-checkpoint/` for `default`, +`kernel/tests/data/v2-parquet-sidecars-struct-stats-only/` for the V2 sub-flows +with all pre-existing checkpoint artifacts and `_last_checkpoint` scrubbed), +runs this binary against it, and diffs the output against the matching +`expected-data/.expected` file. \ No newline at end of file diff --git a/ffi/examples/checkpoint-table/checkpoint_table.c b/ffi/examples/checkpoint-table/checkpoint_table.c new file mode 100644 index 0000000000..45924bf6f6 --- /dev/null +++ b/ffi/examples/checkpoint-table/checkpoint_table.c @@ -0,0 +1,145 @@ +#include +#include +#include +#include + +#include "delta_kernel_ffi.h" +#include "kernel_utils.h" + +// ============================================================================ +// checkpoint_table -- C FFI example for the checkpoint write surface +// ============================================================================ +// +// Usage: +// ./checkpoint_table /path/to/existing/table +// +// where is one of: +// default -- spec = NULL; kernel auto-picks V1/V2 from table protocol. +// v2_no_sidecar -- FfiCheckpointSpec::V2NoSidecar; V2 manifest, no sidecars. +// v2_with_sidecars -- FfiCheckpointSpec::V2WithSidecar { hint = Some(2) }; V2 + sidecars. +// +// Demonstrates the checkpoint FFI surface: +// - checkpoint_snapshot(snapshot, engine, spec) -> ExternResultFfiCheckpointWriteResult +// - FfiCheckpointSpec discriminated-union construction (V1 / V2NoSidecar / V2WithSidecar) +// - FfiCheckpointWriteResult discriminator inspection (Written vs AlreadyExists) +// - free_snapshot on the returned snapshot handle from either variant + +static int run_sub_flow(HandleSharedSnapshot snapshot, + SharedExternEngine* engine, + const char* sub_flow) { + FfiCheckpointSpec spec; + FfiCheckpointSpec* spec_ptr; + + if (strcmp(sub_flow, "default") == 0) { + // Spec = NULL => kernel auto-picks V1 or V2 based on table protocol features. + spec_ptr = NULL; + } else if (strcmp(sub_flow, "v2_no_sidecar") == 0) { + spec.tag = FfiCheckpointSpecV2NoSidecar; + spec_ptr = &spec; + } else if (strcmp(sub_flow, "v2_with_sidecars") == 0) { + // Pass a small hint (2) so the fixture's handful of file actions splits across + // multiple sidecars, making the on-disk shape observably different from the + // V2-no-sidecar sub-flow. Pass `Noneusize` instead to use the kernel default hint. + spec.tag = FfiCheckpointSpecV2WithSidecar; + spec.v2_with_sidecar.file_actions_per_sidecar_hint.tag = Someusize; + spec.v2_with_sidecar.file_actions_per_sidecar_hint.some = 2; + spec_ptr = &spec; + } else { + fprintf(stderr, "Unknown sub-flow: %s\n", sub_flow); + return 1; + } + + ExternResultFfiCheckpointWriteResult build_res = + checkpoint_snapshot(snapshot, engine, spec_ptr); + if (build_res.tag != OkFfiCheckpointWriteResult) { + print_error("checkpoint_snapshot failed.", (Error*)build_res.err); + free_error((Error*)build_res.err); + return 1; + } + + FfiCheckpointWriteResult result = build_res.ok; + HandleSharedSnapshot returned; + const char* discrim_str; + switch (result.tag) { + case FfiCheckpointWriteResultWritten: + returned = result.written; + discrim_str = "Written"; + break; + case FfiCheckpointWriteResultAlreadyExists: + returned = result.already_exists; + discrim_str = "AlreadyExists"; + break; + default: + fprintf(stderr, "Unexpected FfiCheckpointWriteResult tag: %d\n", result.tag); + return 1; + } + + uint64_t returned_version = version(returned); + printf("sub_flow=%s result=%s version=%" PRIu64 "\n", + sub_flow, discrim_str, returned_version); + free_snapshot(returned); + return 0; +} + +int main(int argc, char* argv[]) { + if (argc != 3) { + fprintf(stderr, + "Usage: %s /path/to/existing/table \n" + " : default | v2_no_sidecar | v2_with_sidecars\n", + argv[0]); + return 1; + } + char* table_path = argv[1]; + char* sub_flow = argv[2]; + KernelStringSlice table_path_slice = { table_path, strlen(table_path) }; + + // === Build engine === + ExternResultEngineBuilder engine_builder_res = + get_engine_builder(table_path_slice, allocate_error); + if (engine_builder_res.tag != OkEngineBuilder) { + print_error("Could not get engine builder.", (Error*)engine_builder_res.err); + free_error((Error*)engine_builder_res.err); + return 1; + } + // For DefaultEngine, Snapshot::checkpoint performs async I/O (read commit JSONs + write parquet checkpoint / + // sidecars) and drives it via the engine's task executor with nested `block_on` calls. The + // default single-threaded background executor would deadlock on that nesting: its one worker + // thread is parked on the outer `block_on`, so the inner task can never be scheduled. Use a + // small multithreaded executor (2 workers, default blocking threads) so the nested `block_on` + // can make progress. + set_builder_with_multithreaded_executor(engine_builder_res.ok, + /*worker_threads*/ 2, + /*max_blocking_threads*/ 0); + ExternResultHandleSharedExternEngine engine_res = builder_build(engine_builder_res.ok); + if (engine_res.tag != OkHandleSharedExternEngine) { + print_error("Failed to build engine.", (Error*)engine_res.err); + free_error((Error*)engine_res.err); + return 1; + } + SharedExternEngine* engine = engine_res.ok; + + // === Build snapshot === + ExternResultHandleMutableFfiSnapshotBuilder snapshot_builder_res = + get_snapshot_builder(table_path_slice, engine); + if (snapshot_builder_res.tag != OkHandleMutableFfiSnapshotBuilder) { + print_error("Failed to get snapshot builder.", (Error*)snapshot_builder_res.err); + free_error((Error*)snapshot_builder_res.err); + free_engine(engine); + return 1; + } + ExternResultHandleSharedSnapshot snap_res = + snapshot_builder_build(snapshot_builder_res.ok); + if (snap_res.tag != OkHandleSharedSnapshot) { + print_error("Failed to load snapshot.", (Error*)snap_res.err); + free_error((Error*)snap_res.err); + free_engine(engine); + return 1; + } + HandleSharedSnapshot snapshot = snap_res.ok; + + int rc = run_sub_flow(snapshot, engine, sub_flow); + + free_snapshot(snapshot); + free_engine(engine); + return rc; +} diff --git a/ffi/examples/common/kernel_utils.c b/ffi/examples/common/kernel_utils.c index 002c3bfc1e..4d04afc84f 100644 --- a/ffi/examples/common/kernel_utils.c +++ b/ffi/examples/common/kernel_utils.c @@ -1,4 +1,5 @@ #include +#include #include #include #include "kernel_utils.h" @@ -75,3 +76,256 @@ bool set_builder_opt(EngineBuilder* engine_builder, char* key, char* val) return true; } +// utility to print out a metric id as a uuid +void print_metric_id(const char* name, MetricId id) { + const uint8_t* b = id.bytes; + printf(" %s: " + "%02x%02x%02x%02x-%02x%02x-%02x%02x-%02x%02x-%02x%02x%02x%02x%02x%02x\n", + name, + b[0], b[1], b[2], b[3], b[4], b[5], b[6], b[7], + b[8], b[9], b[10], b[11], b[12], b[13], b[14], b[15]); +} + +// utility to print out a table type +void print_table_type(TableType tt) { + printf(" table_type:"); + switch (tt) { + case TableTypePathBased: printf(" PathBased,\n"); break; + case TableTypeCatalogManaged: printf(" CatalogManaged,\n"); break; + } +} + +// utility to print out a log-segment load type +void print_load_type(LogSegmentLoadType lt) { + printf(" load_type:"); + switch (lt) { + case LogSegmentLoadTypeFull: printf(" Full,\n"); break; + case LogSegmentLoadTypeIncremental: printf(" Incremental,\n"); break; + case LogSegmentLoadTypeUnknown: printf(" Unknown,\n"); break; + } +} + +#define PM_START(name) printf("\nMetric " #name " {\n") +#define PM_END printf("}\n\n"); +#define PM_ID(s, f) print_metric_id(#f, (s).f) +#define PM_U64(s, f) printf(" " #f ": %" PRIu64 ",\n", (uint64_t)(s).f) +#define PM_BOOL(s, f) printf(" " #f ": %s,\n", (s).f ? "true" : "false") +#define PM_SLICE(s, f) (s.f.len>0?printf(" " #f ": %.*s,\n", (int)(s).f.len, (s).f.ptr):printf(" " #f ": \"\",\n")) + +// print out metrics as json(ish) objects +void print_metric(MetricEvent event) { + switch (event.tag) { + case MetricEventLogSegmentLoadSuccess: + PM_START(LogSegmentLoadSuccess); + LogSegmentLoadSuccess lsls = event.log_segment_load_success; + PM_ID(lsls, operation_id); + PM_SLICE(lsls, correlation_id); + print_table_type(lsls.table_type); + print_load_type(lsls.load_type); + PM_U64(lsls, duration_ns); + PM_U64(lsls, num_commit_files); + PM_U64(lsls, num_checkpoint_files); + PM_U64(lsls, num_compaction_files); + PM_BOOL(lsls, has_crc); + PM_U64(lsls, crc_versions_behind); + PM_END; + return; + + case MetricEventLogSegmentLoadFailure: + PM_START(LogSegmentLoadFailure); + LogSegmentLoadFailure lslf = event.log_segment_load_failure; + PM_ID(lslf, operation_id); + PM_SLICE(lslf, correlation_id); + print_table_type(lslf.table_type); + print_load_type(lslf.load_type); + PM_END; + return; + + case MetricEventProtocolMetadataLoadSuccess: + PM_START(ProtocolMetadataLoadSuccess); + ProtocolMetadataLoadSuccess pmls = event.protocol_metadata_load_success; + PM_ID(pmls, operation_id); + PM_SLICE(pmls, correlation_id); + print_table_type(pmls.table_type); + print_load_type(pmls.load_type); + PM_U64(pmls, duration_ns); + PM_END; + return; + + case MetricEventProtocolMetadataLoadFailure: + PM_START(ProtocolMetadataLoadFailure); + ProtocolMetadataLoadFailure pmlf = event.protocol_metadata_load_failure; + PM_ID(pmlf, operation_id); + PM_SLICE(pmlf, correlation_id); + print_table_type(pmlf.table_type); + print_load_type(pmlf.load_type); + PM_END; + return; + + case MetricEventSnapshotBuildSuccess: + PM_START(SnapshotBuildSuccess); + SnapshotBuildSuccess sbs = event.snapshot_build_success; + PM_ID(sbs, operation_id); + PM_SLICE(sbs, correlation_id); + print_table_type(sbs.table_type); + print_load_type(sbs.load_type); + PM_U64(sbs, version); + PM_U64(sbs, duration_ns); + PM_END; + return; + + case MetricEventSnapshotBuildFailure: + PM_START(SnapshotBuildFailure); + SnapshotBuildFailure slf = event.snapshot_build_failure; + PM_ID(slf, operation_id); + PM_SLICE(slf, correlation_id); + print_table_type(slf.table_type); + print_load_type(slf.load_type); + PM_END; + return; + + case MetricEventTransactionCommitSuccess: + PM_START(TransactionCommitSuccess); + TransactionCommitSuccess tcs = event.transaction_commit_success; + PM_ID(tcs, operation_id); + PM_SLICE(tcs, correlation_id); + print_table_type(tcs.table_type); + PM_U64(tcs, commit_version); + PM_U64(tcs, num_add_files); + PM_U64(tcs, num_remove_files); + PM_U64(tcs, num_dv_updates); + PM_U64(tcs, add_files_bytes); + PM_U64(tcs, remove_files_bytes); + PM_BOOL(tcs, is_blind_append); + PM_BOOL(tcs, data_change); + printf("operation: %.*s,\n", (int)tcs.operation.len, tcs.operation.ptr); + PM_U64(tcs, prepare_duration_ns); + PM_U64(tcs, committer_duration_ns); + PM_U64(tcs, total_duration_ns); + PM_END; + return; + + case MetricEventTransactionCommitFailure: + PM_START(TransactionCommitFailure); + TransactionCommitFailure tcf = event.transaction_commit_failure; + PM_ID(tcf, operation_id); + PM_SLICE(tcf, correlation_id); + print_table_type(tcf.table_type); + printf(" reason:"); + switch (tcf.reason) { + case CommitFailureReasonConflict: printf(" Conflict,\n"); break; + case CommitFailureReasonRetryableIo: printf(" Retryable IO,\n"); break; + case CommitFailureReasonError: printf(" Error,\n"); break; + } + PM_END; + return; + + case MetricEventDomainMetadataLoadSuccess: + PM_START(DomainMetadataLoadSuccess); + DomainMetadataLoadSuccess dmls = event.domain_metadata_load_success; + PM_BOOL(dmls, from_cache); + PM_U64(dmls, num_domains_returned); + PM_U64(dmls, duration_ns); + PM_END; + return; + + case MetricEventDomainMetadataLoadFailure: + PM_START(DomainMetadataLoadFailure); + PM_END; + return; + + case MetricEventSetTransactionLoadSuccess: + PM_START(SetTransactionLoadSuccess); + SetTransactionLoadSuccess stls = event.set_transaction_load_success; + PM_BOOL(stls, from_cache); + PM_BOOL(stls, found); + PM_U64(stls, duration_ns); + PM_END; + return; + + case MetricEventSetTransactionLoadFailure: + PM_START(SetTransactionLoadFailure); + PM_END; + return; + + case MetricEventCrcReadSuccess: + PM_START(CrcReadSuccess); + CrcReadSuccess crs = event.crc_read_success; + PM_U64(crs, bytes_read); + PM_U64(crs, duration_ns); + PM_END; + return; + + case MetricEventCrcReadFailure: + PM_START(CrcReadFailure); + PM_END; + return; + + case MetricEventJsonReadCompleted: + PM_START(MetricEventJsonReadCompleted); + JsonReadCompleted jrc = event.json_read_completed; + PM_U64(jrc, num_files); + PM_U64(jrc, bytes_read); + PM_END; + return; + + case MetricEventParquetReadCompleted: + PM_START(MetricEventParquetReadCompleted); + ParquetReadCompleted prc = event.parquet_read_completed; + PM_U64(prc, num_files); + PM_U64(prc, bytes_read); + PM_END; + return; + + case MetricEventScanMetadataCompleted: + PM_START(ScanMetadataCompleted); + ScanMetadataCompleted smc = event.scan_metadata_completed; + PM_ID(smc, operation_id); + PM_SLICE(smc, correlation_id); + print_table_type(smc.table_type); + printf(" scan_type:"); + switch (smc.scan_type) { + case ScanTypeSequentialPhase: printf(" Sequential Phase,\n"); break; + case ScanTypeParallelPhase: printf(" Parallel Phase,\n"); break; + case ScanTypeFull: printf(" Full,\n"); break; + } + PM_U64(smc, duration_ns); + PM_U64(smc, num_add_files_seen); + PM_U64(smc, num_active_add_files); + PM_U64(smc, active_add_files_bytes); + PM_U64(smc, num_remove_files_seen); + PM_U64(smc, num_non_file_actions); + PM_U64(smc, num_predicate_filtered); + PM_U64(smc, peak_hash_set_size); + PM_U64(smc, dedup_visitor_time_ns); + PM_U64(smc, predicate_eval_time_ns); + PM_END; + return; + + case MetricEventStorageListCompleted: + PM_START(MetricEventStorageListCompleted); + StorageListCompleted slc = event.storage_list_completed; + PM_U64(slc, duration_ns); + PM_U64(slc, num_files); + PM_END; + return; + + case MetricEventStorageReadCompleted: + PM_START(MetricEventStorageReadCompleted); + StorageReadCompleted src = event.storage_read_completed; + PM_U64(src, duration_ns); + PM_U64(src, num_files); + PM_U64(src, bytes_read); + PM_END; + return; + + case MetricEventStorageCopyCompleted: + PM_START(MetricEventStorageCopyCompleted); + StorageCopyCompleted scc = event.storage_copy_completed; + PM_U64(scc, duration_ns); + PM_END; + return; + + default: printf("\n\n WARNING: Got an unknown metric \n\n"); return; + } +} diff --git a/ffi/examples/common/kernel_utils.h b/ffi/examples/common/kernel_utils.h index eca6eccd97..dfe35e0789 100644 --- a/ffi/examples/common/kernel_utils.h +++ b/ffi/examples/common/kernel_utils.h @@ -24,3 +24,5 @@ EngineError* allocate_error(KernelError etype, const KernelStringSlice msg); // utility function to convert key/val into slices and set them on a builder // returns false on failure bool set_builder_opt(EngineBuilder* engine_builder, char* key, char* val); +// print out metric info +void print_metric(MetricEvent event); diff --git a/ffi/examples/create-table/CMakeLists.txt b/ffi/examples/create-table/CMakeLists.txt new file mode 100644 index 0000000000..70e55011f3 --- /dev/null +++ b/ffi/examples/create-table/CMakeLists.txt @@ -0,0 +1,28 @@ +cmake_minimum_required(VERSION 3.12) +project(create_table) + +add_executable(create_table create_table.c ../common/kernel_utils.c) +target_compile_definitions(create_table PUBLIC DEFINE_DEFAULT_ENGINE_BASE) +target_include_directories(create_table PUBLIC + "${CMAKE_CURRENT_SOURCE_DIR}/../../../target/ffi-headers" + "${CMAKE_CURRENT_SOURCE_DIR}/../common") +target_link_directories(create_table PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/../../../target/debug") +target_link_libraries(create_table PUBLIC delta_kernel_ffi) + +if(WIN32) + set(CMAKE_C_FLAGS_DEBUG "/MT") + target_link_libraries(create_table PUBLIC ws2_32 userenv bcrypt ncrypt crypt32 secur32 ntdll RuntimeObject) +endif(WIN32) + +if(MSVC) + target_compile_options(create_table PRIVATE /W4 /WX) +else() + target_compile_options(create_table PRIVATE -Wall -Wextra -Wpedantic -Werror -g -fsanitize=address) + target_link_options(create_table PRIVATE -g -fsanitize=address) +endif() + +# Testing +include(CTest) +set(TestRunner "../../../tests/create-table-testing/run_test.sh") +set(ExpectedPath "../../../tests/create-table-testing/expected-data") +add_test(NAME create_table_basic COMMAND ${TestRunner} ${ExpectedPath}/basic.expected) diff --git a/ffi/examples/create-table/README.md b/ffi/examples/create-table/README.md new file mode 100644 index 0000000000..2571717cde --- /dev/null +++ b/ffi/examples/create-table/README.md @@ -0,0 +1,26 @@ +create-table +============ + +C FFI example for CREATE TABLE. Demonstrates the schema-visitor handoff +(`KernelSchemaVisitorState` + `visit_field_*`) used by `get_create_table_builder`, the +consuming-and-returning handle pattern of `create_table_builder_with_table_property` / +`create_table_with_engine_info`, and the final `create_table_commit`. + +# Building + +```bash +# from repo root +$ cargo build -p delta_kernel_ffi +# from this directory +$ mkdir build && cd build && cmake .. && make +$ ./create_table /path/to/new/table +``` + +# Notes + +- The example does not stage any initial data files (`create_table_add_files` is + omitted). Initial-data staging requires building an Arrow batch that matches + `Transaction::add_files_schema`, which needs arrow-glib (or a similar C-level Arrow + builder) on the C side. A shared writer helper is planned as a follow-up. +- The table is created with `delta.enableChangeDataFeed=true` to exercise + `create_table_builder_with_table_property`. diff --git a/ffi/examples/create-table/create_table.c b/ffi/examples/create-table/create_table.c new file mode 100644 index 0000000000..1bfe4367df --- /dev/null +++ b/ffi/examples/create-table/create_table.c @@ -0,0 +1,209 @@ +#include +#include +#include +#include + +#include "delta_kernel_ffi.h" +#include "kernel_utils.h" + +// ============================================================================ +// create_table -- C FFI example for CREATE TABLE +// ============================================================================ +// +// Usage: +// ./create_table /path/to/new/table +// +// Demonstrates: +// - Building a schema via the KernelSchemaVisitorState API (engine-side -> kernel-side +// schema conversion): visit_field_long, visit_field_string, visit_field_struct. +// - get_create_table_builder with a static schema spec. +// - create_table_builder_with_table_property to set `delta.enableChangeDataFeed`. +// - create_table_builder_build -> create_table_commit. +// - Opening a snapshot on the freshly-created table to confirm the commit landed. +// +// Note: get_create_table_builder takes engine_info and stores it on the builder, so the +// transaction is already labelled by the time we call create_table_commit. The example +// therefore does NOT call create_table_with_engine_info -- that function exists for +// engines that want to override the engine_info after building (and is exercised by the +// existing-table write path in write-table). +// +// The example does not stage any initial files. The +// add_files flow requires constructing an Arrow batch matching Transaction::add_files_schema, +// which would pull in arrow-glib (or a similar C-level Arrow builder). + +// === Schema visitor === +// +// The engine provides `struct EngineSchema { void* schema; uintptr_t (*visitor)(...); }`. +// Our schema description is a simple array of fields (no nesting). The visitor calls the +// kernel's visit_field_* functions to register each field, then a final visit_field_struct +// to register the top-level struct. + +enum field_type { + FIELD_LONG, + FIELD_STRING, +}; + +struct field_spec { + const char* name; + enum field_type type; + bool nullable; +}; + +struct schema_spec { + const struct field_spec* fields; + size_t field_count; +}; + +static uintptr_t build_schema(void* data, KernelSchemaVisitorState* state) { + const struct schema_spec* spec = data; + uintptr_t* child_ids = malloc(spec->field_count * sizeof(uintptr_t)); + for (size_t i = 0; i < spec->field_count; i++) { + const struct field_spec* f = &spec->fields[i]; + KernelStringSlice name = { f->name, strlen(f->name) }; + ExternResultusize r; + switch (f->type) { + case FIELD_LONG: + r = visit_field_long(state, name, f->nullable, allocate_error); + break; + case FIELD_STRING: + r = visit_field_string(state, name, f->nullable, allocate_error); + break; + default: + fprintf(stderr, "Unknown field type %d\n", f->type); + free(child_ids); + return 0; + } + if (r.tag != Okusize) { + print_error("visit_field_* failed", (Error*)r.err); + free_error((Error*)r.err); + free(child_ids); + return 0; + } + child_ids[i] = r.ok; + } + // Register the top-level struct. The name of the top-level struct is not used by kernel; + // pass a stable placeholder. + KernelStringSlice root_name = { "root", 4 }; + ExternResultusize root = visit_field_struct( + state, root_name, child_ids, spec->field_count, /*nullable*/ false, allocate_error); + free(child_ids); + if (root.tag != Okusize) { + print_error("visit_field_struct failed for top-level", (Error*)root.err); + free_error((Error*)root.err); + return 0; + } + return root.ok; +} + +int main(int argc, char* argv[]) { + if (argc != 2) { + fprintf(stderr, "Usage: %s /path/to/new/table\n", argv[0]); + return 1; + } + char* table_path = argv[1]; + printf("Creating table at %s\n", table_path); + KernelStringSlice table_path_slice = { table_path, strlen(table_path) }; + + // === Build engine === + ExternResultEngineBuilder engine_builder_res = + get_engine_builder(table_path_slice, allocate_error); + if (engine_builder_res.tag != OkEngineBuilder) { + print_error("Could not get engine builder.", (Error*)engine_builder_res.err); + free_error((Error*)engine_builder_res.err); + return 1; + } + ExternResultHandleSharedExternEngine engine_res = builder_build(engine_builder_res.ok); + if (engine_res.tag != OkHandleSharedExternEngine) { + print_error("Failed to build engine.", (Error*)engine_res.err); + free_error((Error*)engine_res.err); + return 1; + } + SharedExternEngine* engine = engine_res.ok; + + // === Describe the schema === + // NOTE: Delta requires columns to be nullable unless the `invariants` writer feature is + // enabled. + static const struct field_spec fields[] = { + { "id", FIELD_LONG, /*nullable*/ true }, + { "name", FIELD_STRING, /*nullable*/ true }, + }; + struct schema_spec spec = { fields, sizeof(fields) / sizeof(fields[0]) }; + EngineSchema engine_schema = { .schema = &spec, .visitor = build_schema }; + + // === Get create-table builder === + const char* engine_info_str = "create_table_example"; + KernelStringSlice engine_info_slice = { engine_info_str, strlen(engine_info_str) }; + ExternResultHandleExclusiveCreateTableBuilder builder_res = + get_create_table_builder(table_path_slice, &engine_schema, engine_info_slice, engine); + if (builder_res.tag != OkHandleExclusiveCreateTableBuilder) { + print_error("Failed to get create-table builder.", (Error*)builder_res.err); + free_error((Error*)builder_res.err); + free_engine(engine); + return 1; + } + ExclusiveCreateTableBuilder* builder = builder_res.ok; + + // === Chain a table property (CONSUMES and RETURNS the builder handle) === + const char* prop_key = "delta.enableChangeDataFeed"; + const char* prop_val = "true"; + KernelStringSlice prop_key_slice = { prop_key, strlen(prop_key) }; + KernelStringSlice prop_val_slice = { prop_val, strlen(prop_val) }; + ExternResultHandleExclusiveCreateTableBuilder prop_res = create_table_builder_with_table_property( + builder, prop_key_slice, prop_val_slice, engine); + if (prop_res.tag != OkHandleExclusiveCreateTableBuilder) { + // IMPORTANT: the old builder handle is consumed unconditionally, including on error. + print_error("Failed to set table property.", (Error*)prop_res.err); + free_error((Error*)prop_res.err); + free_engine(engine); + return 1; + } + builder = prop_res.ok; + + // === Build -> produces a create-table transaction === + ExternResultHandleExclusiveCreateTransaction txn_res = + create_table_builder_build(builder, engine); + if (txn_res.tag != OkHandleExclusiveCreateTransaction) { + print_error("create_table_builder_build failed.", (Error*)txn_res.err); + free_error((Error*)txn_res.err); + free_engine(engine); + return 1; + } + ExclusiveCreateTransaction* txn = txn_res.ok; + + // === Commit === + ExternResultHandleExclusiveCommittedTransaction commit_res = create_table_commit(txn, engine); + if (commit_res.tag != OkHandleExclusiveCommittedTransaction) { + print_error("create_table_commit failed.", (Error*)commit_res.err); + free_error((Error*)commit_res.err); + free_engine(engine); + return 1; + } + HandleExclusiveCommittedTransaction committed = commit_res.ok; + printf("Committed version: %" PRIu64 "\n", committed_transaction_version(&committed)); + free_committed_transaction(committed); + + // === Open a snapshot on the new table to confirm it landed === + ExternResultHandleMutableFfiSnapshotBuilder snapshot_builder_res = + get_snapshot_builder(table_path_slice, engine); + if (snapshot_builder_res.tag != OkHandleMutableFfiSnapshotBuilder) { + print_error("Failed to get snapshot builder.", (Error*)snapshot_builder_res.err); + free_error((Error*)snapshot_builder_res.err); + free_engine(engine); + return 1; + } + ExternResultHandleSharedSnapshot snapshot_res = + snapshot_builder_build(snapshot_builder_res.ok); + if (snapshot_res.tag != OkHandleSharedSnapshot) { + print_error("Failed to load snapshot of created table.", (Error*)snapshot_res.err); + free_error((Error*)snapshot_res.err); + free_engine(engine); + return 1; + } + SharedSnapshot* snapshot = snapshot_res.ok; + printf("Snapshot version after create: %" PRIu64 "\n", version(snapshot)); + printf("Partition columns: %" PRIuPTR "\n", get_partition_column_count(snapshot)); + + free_snapshot(snapshot); + free_engine(engine); + return 0; +} diff --git a/ffi/examples/delta-kernel-unity-catalog-example/CMakeLists.txt b/ffi/examples/delta-kernel-unity-catalog-example/CMakeLists.txt new file mode 100644 index 0000000000..d371209bb5 --- /dev/null +++ b/ffi/examples/delta-kernel-unity-catalog-example/CMakeLists.txt @@ -0,0 +1,27 @@ +cmake_minimum_required(VERSION 3.12) +project(delta_kernel_unity_catalog_example) + +add_executable(delta_kernel_unity_catalog_example delta_kernel_unity_catalog_example.c ../common/kernel_utils.c) +target_compile_definitions(delta_kernel_unity_catalog_example PUBLIC DEFINE_DEFAULT_ENGINE_BASE) +target_include_directories(delta_kernel_unity_catalog_example PUBLIC + "${CMAKE_CURRENT_SOURCE_DIR}/../../../target/ffi-headers" + "${CMAKE_CURRENT_SOURCE_DIR}/../common") +target_link_directories(delta_kernel_unity_catalog_example PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/../../../target/debug") +target_link_libraries(delta_kernel_unity_catalog_example PUBLIC delta_kernel_ffi) + +if(WIN32) + set(CMAKE_C_FLAGS_DEBUG "/MT") + target_link_libraries(delta_kernel_unity_catalog_example PUBLIC ws2_32 userenv bcrypt ncrypt crypt32 secur32 ntdll RuntimeObject) +endif(WIN32) + +if(MSVC) + target_compile_options(delta_kernel_unity_catalog_example PRIVATE /W3 /WX) +else() + target_compile_options(delta_kernel_unity_catalog_example PRIVATE -Wall -Wextra -Wpedantic -Werror -g -fsanitize=address) + target_link_options(delta_kernel_unity_catalog_example PRIVATE -g -fsanitize=address) +endif() + +# Testing +include(CTest) +set(ExprTestRunner "../../../tests/test-delta-kernel-unity-catalog-ffi/run_test.sh") +add_test(NAME test_delta_kernel_unity_catalog_ffi COMMAND ${ExprTestRunner}) diff --git a/ffi/examples/delta-kernel-unity-catalog-example/README.md b/ffi/examples/delta-kernel-unity-catalog-example/README.md new file mode 100644 index 0000000000..58222690fb --- /dev/null +++ b/ffi/examples/delta-kernel-unity-catalog-example/README.md @@ -0,0 +1,43 @@ +delta-kernel-unity-catalog example +=================================== + +Simple example showing how to use the delta-kernel-unity-catalog FFI surface -- namely +`get_uc_commit_client`, `get_uc_committer`, and `transaction_with_committer` -- to run a +commit against a catalog-managed table using a custom commit callback. + +# Building + +This example is built with [cmake]. Instructions below assume you start in the directory +containing this README. + +Before building this example, you must build `delta_kernel_ffi` with **all features** enabled +so that the Unity Catalog bindings are present in `delta_kernel_ffi.h`. See +[the FFI readme](../../README.md) for details. TLDR: + +```bash +# from repo root +$ cargo build -p delta_kernel_ffi [--release] --all-features +# or, from the ffi/ dir +$ cargo build [--release] --all-features +``` + +## Linux / MacOS + +``` +$ mkdir build +$ cd build +$ cmake .. +$ make +$ ./delta_kernel_unity_catalog_example [path/to/catalog-managed-table] +``` + +The included ctest (`test_delta_kernel_unity_catalog_ffi`) runs the binary against the +catalog-managed fixture at +`delta-kernel-unity-catalog/tests/data/catalog_managed_0`, so you generally don't need to +supply your own table to see it work: + +``` +$ ctest --output-on-failure +``` + +[cmake]: https://cmake.org/ diff --git a/ffi/examples/uc-catalog-example/uc_catalog_example.c b/ffi/examples/delta-kernel-unity-catalog-example/delta_kernel_unity_catalog_example.c similarity index 65% rename from ffi/examples/uc-catalog-example/uc_catalog_example.c rename to ffi/examples/delta-kernel-unity-catalog-example/delta_kernel_unity_catalog_example.c index ad81e6fb14..79c3bf880e 100644 --- a/ffi/examples/uc-catalog-example/uc_catalog_example.c +++ b/ffi/examples/delta-kernel-unity-catalog-example/delta_kernel_unity_catalog_example.c @@ -9,24 +9,31 @@ #include "kernel_utils.h" +// strlen("_delta_log/_staged_commits/") + 1 for the trailing NUL, the fixed prefix joined onto +// the table URI and staged file name when building the staged commit path. +#define STAGED_COMMITS_PREFIX_LEN 28 + // Context struct to hold any state needed by our client // This can hold connection info, auth tokens, etc. typedef struct UCContext { int call_count; const char* base_url; + // Table root (with trailing slash) the committer was created for. The UC API does not carry a + // table URI in the commit request, so the callback uses this to locate the staged commit file. + const char* table_root; } UCContext; // Check that a staging file matches what the commit info says, then remove it -void validate_and_clean_staging_file(char* table_uri, char* file_name, Commit *commit_info) { - char* uri = table_uri; +void validate_and_clean_staging_file(const char* table_uri, char* file_name, Commit *commit_info) { + const char* uri = table_uri; // strip 'file://' if it's present if(strncmp(table_uri, "file://", 7) == 0) { uri = uri + 7; } - int path_len = strlen(uri) + strlen(file_name) + 28; + int path_len = strlen(uri) + strlen(file_name) + STAGED_COMMITS_PREFIX_LEN; char path[path_len]; snprintf(path, path_len, "%s_delta_log/_staged_commits/%s", uri, file_name); printf("Checking that staging file at %s is valid\n", path); @@ -76,16 +83,17 @@ OptionalValueHandleExclusiveRustString commit_callback( // Extract request information char table_id[256]; - char table_uri[1024]; snprintf(table_id, sizeof(table_id), "%.*s", (int)request.table_id.len, request.table_id.ptr); - snprintf(table_uri, sizeof(table_uri), "%.*s", (int)request.table_uri.len, request.table_uri.ptr); printf("Committing to table ID: %s\n", table_id); - printf("Table URI: %s\n", table_uri); if (request.commit_info.tag == SomeCommit) { Commit commit_info = request.commit_info.some; char* file_name = allocate_string(commit_info.file_name); + if (file_name == NULL) { + perror("Failed to allocate file name"); + exit(-1); + } printf("Commit info:\n"); printf(" Version: %" PRId64 "\n", commit_info.version); @@ -94,7 +102,11 @@ OptionalValueHandleExclusiveRustString commit_callback( printf(" File size: %" PRId64 "\n", commit_info.file_size); printf(" File mod time: %" PRId64 "\n\n", commit_info.file_modification_timestamp); - validate_and_clean_staging_file(table_uri, file_name, &commit_info); + // The UC API commit request carries no table URI, so use the table root the committer + // was created for (stored in the context). + const char* staging_root = + (context != NULL && context->table_root != NULL) ? context->table_root : ""; + validate_and_clean_staging_file(staging_root, file_name, &commit_info); free(file_name); } @@ -118,22 +130,35 @@ int main(int argc, char* argv[]) char* table_path = argv[1]; + // Table root with a trailing slash, used by the commit callback to locate the staged commit + // file (the UC API does not echo the table URI back in the commit request). + char table_root[1200]; + snprintf(table_root, sizeof(table_root), "%s/", table_path); + // Initialize our UC context UCContext uc_context = { .call_count = 0, - .base_url = "https://uc-catalog.example.com/api/v1" + .base_url = "https://uc-catalog.example.com/api/v1", + .table_root = table_root }; // Create a UC commit client NullableCvoid context = (void*)&uc_context; HandleSharedFfiUCCommitClient uc_client = get_uc_commit_client(context, commit_callback); - // Create a UC committer for a specific table - const char* table_id = "my_catalog.my_schema.my_table"; + // Create a UC committer for a specific table, identified by its UUID and + // catalog/schema/table name. + const char* table_id = "64dcd182-b3b4-4ee0-88e0-63c159a4121c"; KernelStringSlice table_id_slice = { .ptr = table_id, .len = strlen(table_id) }; + const char* catalog = "my_catalog"; + KernelStringSlice catalog_slice = { .ptr = catalog, .len = strlen(catalog) }; + const char* schema = "my_schema"; + KernelStringSlice schema_slice = { .ptr = schema, .len = strlen(schema) }; + const char* table_name = "my_table"; + KernelStringSlice table_name_slice = { .ptr = table_name, .len = strlen(table_name) }; - ExternResultHandleMutableCommitter committer_res = - get_uc_committer(uc_client, table_id_slice, allocate_error); + ExternResultHandleMutableCommitter committer_res = get_uc_committer( + uc_client, table_id_slice, catalog_slice, schema_slice, table_name_slice, allocate_error); if (committer_res.tag != OkHandleMutableCommitter) { print_error("Failed to create UC committer", (Error*)committer_res.err); @@ -152,6 +177,7 @@ int main(int argc, char* argv[]) if (engine_builder_res.tag != OkEngineBuilder) { print_error("Could not get engine builder", (Error*)engine_builder_res.err); free_error((Error*)engine_builder_res.err); + free_uc_committer(uc_committer); free_uc_commit_client(uc_client); return -1; } @@ -162,16 +188,32 @@ int main(int argc, char* argv[]) if (engine_res.tag != OkHandleSharedExternEngine) { print_error("Failed to build engine", (Error*)engine_res.err); free_error((Error*)engine_res.err); + free_uc_committer(uc_committer); free_uc_commit_client(uc_client); return -1; } SharedExternEngine* engine = engine_res.ok; - ExternResultHandleSharedSnapshot snapshot_res = snapshot(table_path_slice, engine); + ExternResultHandleMutableFfiSnapshotBuilder snapshot_builder_res = get_snapshot_builder(table_path_slice, engine); + if (snapshot_builder_res.tag != OkHandleMutableFfiSnapshotBuilder) { + print_error("Failed to get snapshot builder.", (Error*)snapshot_builder_res.err); + free_error((Error*)snapshot_builder_res.err); + free_engine(engine); + free_uc_committer(uc_committer); + free_uc_commit_client(uc_client); + return -1; + } + // The test table is catalog-managed, so we must set the max catalog version. + // Version 0 is the only commit on disk (staged commits are not loaded here). + snapshot_builder_set_max_catalog_version(&snapshot_builder_res.ok, 0); + ExternResultHandleSharedSnapshot snapshot_res = snapshot_builder_build(snapshot_builder_res.ok); if (snapshot_res.tag != OkHandleSharedSnapshot) { print_error("Failed to create snapshot.", (Error*)snapshot_res.err); free_error((Error*)snapshot_res.err); + free_engine(engine); + free_uc_committer(uc_committer); + free_uc_commit_client(uc_client); return -1; } @@ -186,6 +228,7 @@ int main(int argc, char* argv[]) free_error((Error*)txn_res.err); free_engine(engine); free_uc_commit_client(uc_client); + free_snapshot(snapshot); return -1; } @@ -205,22 +248,27 @@ int main(int argc, char* argv[]) free_error((Error*)txn_with_info_res.err); free_engine(engine); free_uc_commit_client(uc_client); + free_snapshot(snapshot); return -1; } HandleExclusiveTransaction txn_with_info = txn_with_info_res.ok; // calling commit here will end up calling our callback - ExternResultu64 commit_res = commit(txn_with_info, engine); + ExternResultHandleExclusiveCommittedTransaction commit_res = commit(txn_with_info, engine); - if (commit_res.tag != Oku64) { + if (commit_res.tag != OkHandleExclusiveCommittedTransaction) { print_error("Commit failed", (Error*)commit_res.err); free_error((Error*)commit_res.err); free_engine(engine); free_uc_commit_client(uc_client); + free_snapshot(snapshot); return -1; } - printf("\nCommitted version: %lu\n", (unsigned long)commit_res.ok); + HandleExclusiveCommittedTransaction committed = commit_res.ok; + printf("\nCommitted version: %lu\n", + (unsigned long)committed_transaction_version(&committed)); + free_committed_transaction(committed); // Cleanup // Note: txn_with_info was consumed by commit(), so we don't free it diff --git a/ffi/examples/read-table-changes/CMakeLists.txt b/ffi/examples/read-table-changes/CMakeLists.txt new file mode 100644 index 0000000000..fe2753b5dd --- /dev/null +++ b/ffi/examples/read-table-changes/CMakeLists.txt @@ -0,0 +1,33 @@ +cmake_minimum_required(VERSION 3.12) +project(read_table_changes) + +add_executable(read_table_changes read_table_changes.c ../common/kernel_utils.c) +target_compile_definitions(read_table_changes PUBLIC DEFINE_DEFAULT_ENGINE_BASE) +target_include_directories(read_table_changes PUBLIC + "${CMAKE_CURRENT_SOURCE_DIR}/../../../target/ffi-headers" + "${CMAKE_CURRENT_SOURCE_DIR}/../common") +target_link_directories(read_table_changes PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/../../../target/debug") +target_link_libraries(read_table_changes PUBLIC delta_kernel_ffi) + +if(WIN32) + set(CMAKE_C_FLAGS_DEBUG "/MT") + target_link_libraries(read_table_changes PUBLIC ws2_32 userenv bcrypt ncrypt crypt32 secur32 ntdll RuntimeObject) +endif(WIN32) + +if(MSVC) + target_compile_options(read_table_changes PRIVATE /W4 /WX) +else() + target_compile_options(read_table_changes PRIVATE -Wall -Wextra -Wpedantic -Werror -g -fsanitize=address) + target_link_options(read_table_changes PRIVATE -g -fsanitize=address) +endif() + +# Testing +include(CTest) +set(TestRunner "../../../tests/read-table-changes-testing/run_test.sh") +set(DatPath "../../../../acceptance/tests/dat/out/reader_tests/generated") +set(ExpectedPath "../../../tests/read-table-changes-testing/expected-data") +add_test(NAME read_cdf_full COMMAND ${TestRunner} ${DatPath}/cdf/delta/ ${ExpectedPath}/cdf.expected) +add_test(NAME read_cdf_from_v1 COMMAND ${TestRunner} ${DatPath}/cdf/delta/ ${ExpectedPath}/cdf-from-v1.expected -s 1) +add_test(NAME read_cdf_v1_only COMMAND ${TestRunner} ${DatPath}/cdf/delta/ ${ExpectedPath}/cdf-v1-only.expected -s 1 -e 1) +# v2 is the DELETE commit (4 rows of CDC data); exercises the CDC-file-only path. +add_test(NAME read_cdf_v2_only COMMAND ${TestRunner} ${DatPath}/cdf/delta/ ${ExpectedPath}/cdf-v2-only.expected -s 2 -e 2) diff --git a/ffi/examples/read-table-changes/README.md b/ffi/examples/read-table-changes/README.md new file mode 100644 index 0000000000..03d343f1d4 --- /dev/null +++ b/ffi/examples/read-table-changes/README.md @@ -0,0 +1,27 @@ +read-table-changes +================== + +C FFI example for the Change Data Feed (CDF) surface. + +Exercises the full `table_changes_*` and `scan_table_changes_*` flow: + +- `table_changes_from_version` / `table_changes_between_versions` +- `table_changes_{schema,table_root,start_version,end_version}` +- `table_changes_scan` + `table_changes_scan_execute` +- `scan_table_changes_next` iteration -> `free_arrow_ffi_data` + +For each returned Arrow batch the example reads the row count directly off the +`FFI_ArrowArray` and prints a per-batch summary, then releases the batch via +`free_arrow_ffi_data`. A real engine would instead hand the inner +`FFI_ArrowArray` + `FFI_ArrowSchema` to its own arrow layer (e.g. arrow-glib's +`garrow_record_batch_import`, mirroring `read-table/arrow.c`) before freeing. + +# Building + +```bash +# from repo root +$ cargo build -p delta_kernel_ffi +# from this directory +$ mkdir build && cd build && cmake .. && make +$ ./read_table_changes [-s start_version] [-e end_version] /path/to/table +``` diff --git a/ffi/examples/read-table-changes/read_table_changes.c b/ffi/examples/read-table-changes/read_table_changes.c new file mode 100644 index 0000000000..cc375ef9a6 --- /dev/null +++ b/ffi/examples/read-table-changes/read_table_changes.c @@ -0,0 +1,203 @@ +#include +#include +#include +#include +#include +#include + +#include "delta_kernel_ffi.h" +#include "kernel_utils.h" + +// ============================================================================ +// read_table_changes -- C FFI example for the Change Data Feed (CDF) surface +// ============================================================================ +// +// Usage: +// ./read_table_changes [-s start_version] [-e end_version] table/path +// +// Exercises the TableChanges FFI surface: +// - table_changes_from_version / table_changes_between_versions +// - table_changes_{schema,table_root,start_version,end_version} +// - table_changes_scan + table_changes_scan_execute +// - scan_table_changes_next iteration -> free_arrow_ffi_data +// +// For each returned Arrow batch we read the row count directly off the +// FFI_ArrowArray and print a summary, then release the batch via +// free_arrow_ffi_data. A real engine would instead hand the inner +// FFI_ArrowArray + FFI_ArrowSchema to its own arrow layer (e.g. arrow-glib's +// garrow_record_batch_import, mirroring read-table/arrow.c) before freeing. + +// Parse a non-negative integer from optarg into *out. Returns true on success. +static bool parse_version(const char* arg, uint64_t* out) { + errno = 0; + char* end = NULL; + unsigned long long v = strtoull(arg, &end, 10); + if (errno != 0 || end == arg || *end != '\0') { + return false; + } + *out = (uint64_t)v; + return true; +} + +int main(int argc, char* argv[]) { + uint64_t start_version = 0; + bool have_end_version = false; + uint64_t end_version = 0; + + int c; + while ((c = getopt(argc, argv, "s:e:")) != -1) { + switch (c) { + case 's': + if (!parse_version(optarg, &start_version)) { + fprintf(stderr, "Invalid -s value: %s\n", optarg); + return 1; + } + break; + case 'e': + if (!parse_version(optarg, &end_version)) { + fprintf(stderr, "Invalid -e value: %s\n", optarg); + return 1; + } + have_end_version = true; + break; + default: + fprintf(stderr, + "Usage: %s [-s start_version] [-e end_version] table/path\n", + argv[0]); + return 1; + } + } + if (optind != argc - 1) { + fprintf(stderr, + "Usage: %s [-s start_version] [-e end_version] table/path\n", + argv[0]); + return 1; + } + + char* table_path = argv[optind]; + printf("Reading change data feed from %s\n", table_path); + printf(" requested start_version: %" PRIu64 "\n", start_version); + if (have_end_version) { + printf(" requested end_version: %" PRIu64 "\n", end_version); + } else { + printf(" requested end_version: \n"); + } + + KernelStringSlice table_path_slice = { table_path, strlen(table_path) }; + + ExternResultEngineBuilder engine_builder_res = + get_engine_builder(table_path_slice, allocate_error); + if (engine_builder_res.tag != OkEngineBuilder) { + print_error("Could not get engine builder.", (Error*)engine_builder_res.err); + free_error((Error*)engine_builder_res.err); + return 1; + } + ExternResultHandleSharedExternEngine engine_res = builder_build(engine_builder_res.ok); + if (engine_res.tag != OkHandleSharedExternEngine) { + print_error("Failed to build engine.", (Error*)engine_res.err); + free_error((Error*)engine_res.err); + return 1; + } + SharedExternEngine* engine = engine_res.ok; + + // === Create a TableChanges === + ExternResultHandleExclusiveTableChanges tc_res; + if (have_end_version) { + tc_res = table_changes_between_versions(table_path_slice, engine, start_version, end_version); + } else { + tc_res = table_changes_from_version(table_path_slice, engine, start_version); + } + if (tc_res.tag != OkHandleExclusiveTableChanges) { + print_error("Failed to construct TableChanges.", (Error*)tc_res.err); + free_error((Error*)tc_res.err); + free_engine(engine); + return 1; + } + ExclusiveTableChanges* table_changes = tc_res.ok; + + // === Inspect metadata accessors (these do not consume the handle) === + char* tc_table_root = table_changes_table_root(table_changes, allocate_string); + uint64_t tc_start = table_changes_start_version(table_changes); + uint64_t tc_end = table_changes_end_version(table_changes); + printf("TableChanges:\n"); + printf(" table_root: %s\n", tc_table_root); + printf(" start_version: %" PRIu64 "\n", tc_start); + printf(" end_version: %" PRIu64 "\n", tc_end); + free(tc_table_root); + + // table_changes_schema() clones the schema; we free it immediately since this example + // does not pretty-print schemas (read-table's schema.h already demonstrates that). + SharedSchema* cdf_schema = table_changes_schema(table_changes); + free_schema(cdf_schema); + + // === Build and execute a scan === + // + // table_changes_scan() consumes the ExclusiveTableChanges handle regardless of success or + // failure, so we must not free table_changes after this call. + ExternResultHandleSharedTableChangesScan scan_res = + table_changes_scan(table_changes, engine, NULL); + if (scan_res.tag != OkHandleSharedTableChangesScan) { + print_error("Failed to build table_changes scan.", (Error*)scan_res.err); + free_error((Error*)scan_res.err); + free_engine(engine); + return 1; + } + SharedTableChangesScan* scan = scan_res.ok; + + char* scan_root = table_changes_scan_table_root(scan, allocate_string); + printf("Scan:\n"); + printf(" table_root: %s\n", scan_root); + free(scan_root); + + // Logical vs physical schema: for CDF, the logical schema is the user-facing schema plus + // the system columns _change_type, _commit_version, _commit_timestamp. + SharedSchema* logical = table_changes_scan_logical_schema(scan); + SharedSchema* physical = table_changes_scan_physical_schema(scan); + free_schema(logical); + free_schema(physical); + + ExternResultHandleSharedScanTableChangesIterator iter_res = + table_changes_scan_execute(scan, engine); + if (iter_res.tag != OkHandleSharedScanTableChangesIterator) { + print_error("Failed to execute table_changes scan.", (Error*)iter_res.err); + free_error((Error*)iter_res.err); + free_table_changes_scan(scan); + free_engine(engine); + return 1; + } + SharedScanTableChangesIterator* iter = iter_res.ok; + + // === Iterate batches === + // + // scan_table_changes_next returns Ok(non-null) for a batch, Ok(null) when the iterator + // is exhausted, or Err. The handle is borrowed (not consumed), so we keep using `iter` + // across calls. Each non-null batch must be released via free_arrow_ffi_data exactly + // once. + printf("CDF batches:\n"); + int exit_code = 0; + int batch_idx = 0; + int64_t total_rows = 0; + for (;;) { + ExternResultArrowFFIData next_res = scan_table_changes_next(iter); + if (next_res.tag != OkArrowFFIData) { + print_error("scan_table_changes_next failed.", (Error*)next_res.err); + free_error((Error*)next_res.err); + exit_code = 1; + break; + } + if (next_res.ok == NULL) { + break; + } + ArrowFFIData* batch = next_res.ok; + printf(" batch %d: %" PRId64 " rows\n", batch_idx, batch->array.length); + batch_idx++; + total_rows += batch->array.length; + free_arrow_ffi_data(batch); + } + printf("Total: %d batches, %" PRId64 " rows\n", batch_idx, total_rows); + + free_scan_table_changes_iter(iter); + free_table_changes_scan(scan); + free_engine(engine); + return exit_code; +} diff --git a/ffi/examples/read-table/CMakeLists.txt b/ffi/examples/read-table/CMakeLists.txt index 050ed4b5be..4508b76af5 100644 --- a/ffi/examples/read-table/CMakeLists.txt +++ b/ffi/examples/read-table/CMakeLists.txt @@ -23,6 +23,15 @@ add_test(NAME read_and_print_nested COMMAND ${TestRunner} ${DatPath}/nested_type add_test(NAME read_and_print_nested_cols COMMAND ${TestRunner} ${DatPath}/nested_types/delta/ ${ExpectedPath}/nested-types-cols.expected -cmap,struct,array) add_test(NAME read_and_print_basic_partitioned COMMAND ${TestRunner} ${DatPath}/basic_partitioned/delta/ ${ExpectedPath}/basic-partitioned.expected) add_test(NAME read_and_print_with_dv_small COMMAND ${TestRunner} ${KernelTestPath}/table-with-dv-small/ ${ExpectedPath}/table-with-dv-small.expected) +add_test(NAME read_and_print_with_short_dv COMMAND ${TestRunner} ${KernelTestPath}/with-short-dv/ ${ExpectedPath}/with-short-dv.expected) +# Exercises the Arrow batch-mode scan metadata path (scan_metadata_next_arrow). The arrow path +# does not call read_parquet_file, so the data section is "[No data]"; this test still +# requires PRINT_DATA=ON because iterate_scan_metadata_arrow is gated behind it. +add_test(NAME read_arrow_metadata_basic COMMAND ${TestRunner} ${DatPath}/basic_partitioned/delta/ ${ExpectedPath}/basic-partitioned-arrow-metadata.expected -a) +# Exercises typed field metadata across the FFI: the column_mapping table's fields carry +# `delta.columnMapping.id`, which the schema visitor reads back via get_from_metadata_map and +# prints per field. +add_test(NAME read_column_mapping_metadata COMMAND ${TestRunner} ${DatPath}/column_mapping/delta/ ${ExpectedPath}/column-mapping-arrow-metadata.expected -a) if(WIN32) set(CMAKE_C_FLAGS_DEBUG "/MT") @@ -48,6 +57,14 @@ if(PRINT_DATA) target_include_directories(read_table PUBLIC ${ARROW_GLIB_INCLUDE_DIRS}) target_link_directories(read_table PUBLIC ${ARROW_GLIB_LIBRARY_DIRS}) target_link_libraries(read_table PUBLIC ${ARROW_GLIB_LIBRARIES}) - target_compile_options(read_table PUBLIC ${ARROW_GLIB_CFLAGS_OTHER}) + # arrow-glib's .pc file (as of v19) leaks `-std=c++11` into CFLAGS; that's a C++ flag + # which the C compiler rejects under -Werror. Drop any -std= flags before forwarding. + set(ARROW_GLIB_CFLAGS_FILTERED "") + foreach(flag ${ARROW_GLIB_CFLAGS_OTHER}) + if(NOT flag MATCHES "^-std=") + list(APPEND ARROW_GLIB_CFLAGS_FILTERED ${flag}) + endif() + endforeach() + target_compile_options(read_table PUBLIC ${ARROW_GLIB_CFLAGS_FILTERED}) target_compile_definitions(read_table PUBLIC PRINT_ARROW_DATA) endif(PRINT_DATA) diff --git a/ffi/examples/read-table/arrow.c b/ffi/examples/read-table/arrow.c index 08515a4ec5..04e77cd68d 100644 --- a/ffi/examples/read-table/arrow.c +++ b/ffi/examples/read-table/arrow.c @@ -51,6 +51,31 @@ static GArrowRecordBatch* get_record_batch(FFI_ArrowArray* array, GArrowSchema* return record_batch; } +// We might need to extend the filter array to match the size of the actual data, since kernel +// doesn't promise that the number of rows in the filter are equal to the number of rows in the +// data. Anything beyond the filter is implicitly "true", meaning keep the row. Returns the passed +// array if it is already the correct length, otherwise builds a new array. +static GArrowBooleanArray* get_full_dv(gint64 batch_len, GArrowBooleanArray* dv_array) { + gint64 filter_len = garrow_array_get_length((GArrowArray*) dv_array); + if (filter_len >= batch_len) { + return dv_array; + } + GArrowBooleanArrayBuilder* builder = garrow_boolean_array_builder_new(); + GError* error = NULL; + for (gint64 i = 0; i < batch_len; i++) { + gboolean val = (i < filter_len) ? garrow_boolean_array_get_value(dv_array, i) : TRUE; + garrow_boolean_array_builder_append_value(builder, val, &error); + if (report_g_error("Can't append to filter builder", error)) { + g_object_unref(builder); + return NULL; + } + } + GArrowBooleanArray* extended_filter = (GArrowBooleanArray*)garrow_array_builder_finish( + (GArrowArrayBuilder*)builder, &error); + g_object_unref(builder); + return extended_filter; +} + // append a batch to our context static void add_batch_to_context( ArrowContext* context, @@ -61,9 +86,16 @@ static void add_batch_to_context( g_object_unref(schema); if (context->cur_filter != NULL) { GArrowRecordBatch* unfiltered = record_batch; - record_batch = garrow_record_batch_filter(unfiltered, context->cur_filter, NULL, NULL); - // unref the old batch and filter since we don't need them anymore + gint64 batch_len = garrow_record_batch_get_n_rows(unfiltered); + GArrowBooleanArray* filter = get_full_dv(batch_len, context->cur_filter); + GError* filter_error = NULL; + record_batch = garrow_record_batch_filter(unfiltered, filter, NULL, &filter_error); + report_g_error("Can't filter record batch", filter_error); g_object_unref(unfiltered); + if (filter != context->cur_filter) { + // if the pointers aren't the same, we allocated + g_object_unref(filter); + } g_object_unref(context->cur_filter); context->cur_filter = NULL; } @@ -160,13 +192,19 @@ static void visit_read_data(void* vcontext, ExclusiveEngineData* data) } ArrowFFIData* arrow_data = arrow_res.ok; add_batch_to_context(context->arrow_context, arrow_data); - free(arrow_data); // just frees the struct, the data and schema are freed/owned by add_batch_to_context + // The inner FFI_ArrowArray/FFI_ArrowSchema have already been imported by arrow-glib + // inside add_batch_to_context (which moves their release callbacks), so this call only + // drops the outer Box. It's still necessary to use the kernel-provided free because the + // struct was allocated by Rust's global allocator. + free_arrow_ffi_data(arrow_data); } // We call this for each file we get called back to read in read_table.c::visit_callback void c_read_parquet_file( struct EngineContext* context, const KernelStringSlice path, + int64_t size, + int64_t mod_time, const KernelBoolSlice selection_vector, const Expression* transform) { @@ -177,6 +215,8 @@ void c_read_parquet_file( KernelStringSlice path_slice = { full_path, full_len }; FileMeta meta = { .path = path_slice, + .size = (uintptr_t)size, + .last_modified = mod_time, }; ExternResultHandleExclusiveFileReadResultIterator read_res = read_parquet_file(context->engine, &meta, context->physical_schema); diff --git a/ffi/examples/read-table/arrow.h b/ffi/examples/read-table/arrow.h index 8f34cdd4fd..14c170d06e 100644 --- a/ffi/examples/read-table/arrow.h +++ b/ffi/examples/read-table/arrow.h @@ -22,6 +22,8 @@ ArrowContext* init_arrow_context(void); void c_read_parquet_file( struct EngineContext* context, const KernelStringSlice path, + int64_t size, + int64_t mod_time, const KernelBoolSlice selection_vector, const Expression* transform); void print_arrow_context(ArrowContext* context); diff --git a/ffi/examples/read-table/kernel_schema_visitor.h b/ffi/examples/read-table/kernel_schema_visitor.h index 4aea1918a4..e3ed2e892b 100644 --- a/ffi/examples/read-table/kernel_schema_visitor.h +++ b/ffi/examples/read-table/kernel_schema_visitor.h @@ -36,6 +36,10 @@ uintptr_t visit_schema_item(SchemaItem* item, KernelSchemaVisitorState *state, C visit_res = visit_field_timestamp(state, name, item->is_nullable, allocate_error); } else if (strcmp(item->type, "timestamp_ntz") == 0) { visit_res = visit_field_timestamp_ntz(state, name, item->is_nullable, allocate_error); + } else if (strcmp(item->type, "interval year to month") == 0) { + visit_res = visit_field_interval_year_month(state, name, item->is_nullable, allocate_error); + } else if (strcmp(item->type, "interval day to second") == 0) { + visit_res = visit_field_interval_day_time(state, name, item->is_nullable, allocate_error); } else if (strncmp(item->type, "decimal", 7) == 0) { unsigned int precision; int scale; diff --git a/ffi/examples/read-table/read_table.c b/ffi/examples/read-table/read_table.c index f66e39ef10..4a68c90d87 100644 --- a/ffi/examples/read-table/read_table.c +++ b/ffi/examples/read-table/read_table.c @@ -63,8 +63,8 @@ void scan_row_callback( const Expression* transform, const CStringMap* partition_values) { - (void)mod_time; // not using this at the moment #ifndef PRINT_ARROW_DATA + (void)mod_time; // only used when PRINT_ARROW_DATA is defined (void)transform; // only used when PRINT_ARROW_DATA is defined #endif struct EngineContext* context = engine_context; @@ -98,7 +98,7 @@ void scan_row_callback( context->partition_values = partition_values; print_partition_info(context, partition_values); #ifdef PRINT_ARROW_DATA - c_read_parquet_file(context, path, selection_vector, transform); + c_read_parquet_file(context, path, size, mod_time, selection_vector, transform); #endif free_bool_slice(selection_vector); context->partition_values = NULL; @@ -132,6 +132,91 @@ void do_visit_scan_metadata(void* engine_context, HandleSharedScanMetadata scan_ free_scan_metadata(scan_metadata); } +// === Arrow batch-mode scan metadata === +// +// Alternative to the callback-based scan_metadata_next + visit_scan_metadata path above. +// Returns scan file metadata as an Arrow RecordBatch (via C Data Interface) plus a selection +// vector and per-row transforms, avoiding per-row FFI overhead. +// +// This is useful for engines that can process Arrow data natively -- they can extract file +// paths, sizes, deletion vector descriptors, and partition values directly from batch columns +// rather than receiving them one-at-a-time via callbacks. +#ifdef PRINT_ARROW_DATA +void iterate_scan_metadata_arrow( + struct EngineContext* context, + SharedScanMetadataIterator* data_iter) +{ + for (;;) { + ExternResultScanMetadataArrowResult res = + scan_metadata_next_arrow(data_iter, context->engine); + if (res.tag != OkScanMetadataArrowResult) { + print_error("Failed to get arrow scan metadata.", (Error*)res.err); + free_error((Error*)res.err); + return; + } + ScanMetadataArrowResult* result = res.ok; + if (!result) { + print_diag("Arrow scan metadata iterator done\n"); + break; + } + + // Import the Arrow batch via C Data Interface + GError* error = NULL; + GArrowSchema* schema = + garrow_schema_import((gpointer)&result->arrow_data.schema, &error); + if (!schema) { + printf("Failed to import arrow schema: %s\n", error->message); + g_error_free(error); + free_scan_metadata_arrow_result(result); + return; + } + GArrowRecordBatch* batch = + garrow_record_batch_import((gpointer)&result->arrow_data.array, schema, &error); + if (!batch) { + printf("Failed to import arrow batch: %s\n", error->message); + g_error_free(error); + g_object_unref(schema); + free_scan_metadata_arrow_result(result); + return; + } + + // Count selected rows from the selection vector + int64_t num_rows = garrow_record_batch_get_n_rows(batch); + uintptr_t selected_count = 0; + for (uintptr_t i = 0; i < result->selection_vector.len; i++) { + if (result->selection_vector.ptr[i]) { + selected_count++; + } + } + print_diag("Arrow scan metadata batch: %" PRId64 " rows, %" PRIuPTR " selected (files to scan)\n", + num_rows, selected_count); + + // Access per-row transforms -- an engine would apply these when reading each file + for (uintptr_t i = 0; i < result->selection_vector.len; i++) { + if (!result->selection_vector.ptr[i]) { + continue; + } + struct OptionalValueHandleSharedExpression transform = + get_transform_for_row(i, result->transforms); + if (transform.tag == SomeHandleSharedExpression) { + print_diag(" row %" PRIuPTR ": has transform\n", i); + free_kernel_expression(transform.some); + } else { + print_diag(" row %" PRIuPTR ": no transform needed\n", i); + } + } + + // A real engine would extract file paths, sizes, DV info, and partition values + // from the batch columns (path, size, modificationTime, stats, deletionVector, + // fileConstantValues) and use read_parquet_file() to read each file's data. + + g_object_unref(batch); + g_object_unref(schema); + free_scan_metadata_arrow_result(result); + } +} +#endif // PRINT_ARROW_DATA + // Called for each element of the partition StringSliceIterator. We just turn the slice into a // `char*` and append it to our list. We knew the total number of partitions up front, so this // assumes that `list->cols` has been allocated with enough space to store the pointer. @@ -234,9 +319,18 @@ void log_line_callback(KernelStringSlice line) { int main(int argc, char* argv[]) { char* requested_cols = NULL; + bool use_arrow_metadata = false; int c; - while ((c = getopt (argc, argv, "c:")) != -1) { + while ((c = getopt (argc, argv, "ac:")) != -1) { switch (c) { + case 'a': + // Use the Arrow batch-mode scan metadata path (scan_metadata_next_arrow) instead of + // the callback-based path (scan_metadata_next + visit_scan_metadata). This path + // returns scan-file metadata as Arrow C Data Interface batches; engines that already + // process Arrow natively can extract path/size/stats/DV/partition-values directly + // from the batch columns rather than via per-row callbacks. + use_arrow_metadata = true; + break; case 'c': requested_cols = optarg; break; @@ -259,9 +353,16 @@ int main(int argc, char* argv[]) } if (optind != (argc - 1)) { - printf("Usage: %s [-c top_level_column1,top_level_column2] table/path\n", argv[0]); + printf("Usage: %s [-a] [-c top_level_column1,top_level_column2] table/path\n", argv[0]); + return -1; + } + +#ifndef PRINT_ARROW_DATA + if (use_arrow_metadata) { + fprintf(stderr, "-a (arrow-batch metadata) requires building with PRINT_DATA=ON\n"); return -1; } +#endif char* table_path = argv[optind]; printf("Reading table at %s\n", table_path); @@ -270,6 +371,9 @@ int main(int argc, char* argv[]) enable_event_tracing(tracing_callback, TRACE); // we could also do something like this if we want less control over formatting // enable_formatted_log_line_tracing(log_line_callback, TRACE, FULL, true, true, false, false); + + // also enable printing metrics + enable_metrics_reporting(print_metric); #else enable_event_tracing(tracing_callback, WARN); #endif @@ -284,7 +388,10 @@ int main(int argc, char* argv[]) return -1; } - // an example of using a builder to set options when building an engine + // Example of using the builder to set object-store options before building the engine. The + // keys accepted here come from object_store's configuration vocabulary (e.g. "aws_region", + // "aws_access_key_id"). They are object-store-specific and only meaningful when the table URL + // points at that backend -- for a local file:// table the setters have no effect. EngineBuilder* engine_builder = engine_builder_res.ok; if (!set_builder_opt(engine_builder, "aws_region", "us-west-2")) { return -1; @@ -299,17 +406,27 @@ int main(int argc, char* argv[]) // get_default_engine(table_path_slice, NULL); if (engine_res.tag != OkHandleSharedExternEngine) { - print_error("File to get engine", (Error*)engine_builder_res.err); - free_error((Error*)engine_builder_res.err); + print_error("Failed to get engine", (Error*)engine_res.err); + free_error((Error*)engine_res.err); return -1; } SharedExternEngine* engine = engine_res.ok; - ExternResultHandleSharedSnapshot snapshot_res = snapshot(table_path_slice, engine); + ExternResultHandleMutableFfiSnapshotBuilder snapshot_builder_res = get_snapshot_builder(table_path_slice, engine); + if (snapshot_builder_res.tag != OkHandleMutableFfiSnapshotBuilder) { + print_error("Failed to get snapshot builder.", (Error*)snapshot_builder_res.err); + free_error((Error*)snapshot_builder_res.err); + free_engine(engine); + return -1; + } + // snapshot_builder_build consumes the builder handle whether it succeeds or fails, so there + // is nothing to free for the builder here. + ExternResultHandleSharedSnapshot snapshot_res = snapshot_builder_build(snapshot_builder_res.ok); if (snapshot_res.tag != OkHandleSharedSnapshot) { print_error("Failed to create snapshot.", (Error*)snapshot_res.err); free_error((Error*)snapshot_res.err); + free_engine(engine); return -1; } @@ -386,6 +503,14 @@ int main(int argc, char* argv[]) if (data_iter_res.tag != OkHandleSharedScanMetadataIterator) { print_error("Failed to construct scan metadata iterator.", (Error*)data_iter_res.err); free_error((Error*)data_iter_res.err); + free_scan(scan); + free_schema(logical_schema); + free_schema(physical_schema); + free_snapshot(snapshot); + free_engine(engine); + free(context.table_root); + free(scan_table_path); + free_partition_list(context.partition_cols); return -1; } @@ -393,17 +518,30 @@ int main(int argc, char* argv[]) print_diag("\nIterating scan metadata\n"); - // iterate scan files - for (;;) { - ExternResultbool ok_res = - scan_metadata_next(data_iter, &context, do_visit_scan_metadata); - if (ok_res.tag != Okbool) { - print_error("Failed to iterate scan metadata.", (Error*)ok_res.err); - free_error((Error*)ok_res.err); - return -1; - } else if (!ok_res.ok) { - print_diag("Scan metadata iterator done\n"); - break; + int exit_code = 0; +#ifdef PRINT_ARROW_DATA + if (use_arrow_metadata) { + // Arrow batch-mode: hand each metadata batch to iterate_scan_metadata_arrow, which + // imports it via arrow-glib for inspection. This path does NOT call read_parquet_file; + // a real engine would walk the batch columns and read parquet itself. + iterate_scan_metadata_arrow(&context, data_iter); + } else +#endif + { + // Callback-mode: kernel calls back into do_visit_scan_metadata for each metadata batch, + // which then walks each scan file via visit_scan_metadata + scan_row_callback. + for (;;) { + ExternResultbool ok_res = + scan_metadata_next(data_iter, &context, do_visit_scan_metadata); + if (ok_res.tag != Okbool) { + print_error("Failed to iterate scan metadata.", (Error*)ok_res.err); + free_error((Error*)ok_res.err); + exit_code = -1; + break; + } else if (!ok_res.ok) { + print_diag("Scan metadata iterator done\n"); + break; + } } } @@ -425,5 +563,5 @@ int main(int argc, char* argv[]) free(scan_table_path); free_partition_list(context.partition_cols); - return 0; + return exit_code; } diff --git a/ffi/examples/read-table/schema.h b/ffi/examples/read-table/schema.h index 13f8a1252b..3edddad084 100644 --- a/ffi/examples/read-table/schema.h +++ b/ffi/examples/read-table/schema.h @@ -37,6 +37,7 @@ typedef struct char* type; bool is_nullable; uintptr_t children; + char* column_mapping_id; } SchemaItem; typedef struct SchemaItemList @@ -66,6 +67,7 @@ SchemaItem* add_to_list(SchemaItemList* list, char* name, char* type, bool is_nu list->list[idx].name = name; list->list[idx].type = type; list->list[idx].is_nullable = is_nullable; + list->list[idx].column_mapping_id = NULL; list->len++; return &list->list[idx]; } @@ -87,6 +89,9 @@ void print_list(SchemaBuilder* builder, uintptr_t list_id, int indent, int paren SchemaItem* item = &list->list[i]; char* prefix = is_last ? "└" : "├"; printf("%s─ %s: %s", prefix, item->name, item->type); + if (item->column_mapping_id) { + printf(" (column mapping id: %s)", item->column_mapping_id); + } if (strcmp(item->type, "array") == 0) { SchemaItemList child_list = builder->lists[item->children]; if (child_list.len != 1) { @@ -110,29 +115,28 @@ void print_list(SchemaBuilder* builder, uintptr_t list_id, int indent, int paren } } -void print_physical_name(const char *name, const CStringMap* metadata, SharedExternEngine* engine) +// Read the column-mapping id back out of a field's typed metadata, across the FFI. Returns a +// freshly-allocated string (owned by the caller) when the field carries `delta.columnMapping.id`, +// or NULL when it does not. This exercises the typed metadata path end-to-end: the value crosses +// as a `MetadataNumber`, and we assert that kind before rendering it. +char* read_column_mapping_id(const CMetadataMap* metadata, SharedExternEngine* engine) { -#ifdef VERBOSE - char* key_str = "delta.columnMapping.physicalName"; + char* key_str = "delta.columnMapping.id"; KernelStringSlice key = { key_str, strlen(key_str) }; - ExternResultNullableCvoid res = get_from_string_map(metadata, key, allocate_string, engine); + CMetadataValueKind kind; + ExternResultNullableCvoid res = get_from_metadata_map(metadata, key, &kind, allocate_string, engine); if (res.tag != OkNullableCvoid) { - printf("Failed to get physical name\n"); free_error((Error*)res.err); - return; + return NULL; } char* value = res.ok; - if (value) { - printf("Physical name of %s is %s\n", name, value); + if (value && kind != MetadataNumber) { + // The kernel types column-mapping ids as numbers; a different kind means the FFI lost the type. + printf("Unexpected kind %d for delta.columnMapping.id\n", (int)kind); free(value); - } else { - printf("No physical name\n"); + return NULL; } -#else - (void)name; - (void)metadata; - (void)engine; -#endif + return value; } // declare all our visitor methods @@ -159,15 +163,15 @@ void visit_struct( uintptr_t sibling_list_id, struct KernelStringSlice name, bool is_nullable, - const CStringMap * metadata, + const CMetadataMap * metadata, uintptr_t child_list_id) { SchemaBuilder* builder = data; char* name_ptr = allocate_string(name); PRINT_CHILD_VISIT("struct", name_ptr, sibling_list_id, "Children", child_list_id); - print_physical_name(name_ptr, metadata, builder->engine); SchemaItem* struct_item = add_to_list(&builder->lists[sibling_list_id], name_ptr, "struct", is_nullable); struct_item->children = child_list_id; + struct_item->column_mapping_id = read_column_mapping_id(metadata, builder->engine); } void visit_array( @@ -175,15 +179,15 @@ void visit_array( uintptr_t sibling_list_id, struct KernelStringSlice name, bool is_nullable, - const CStringMap * metadata, + const CMetadataMap * metadata, uintptr_t child_list_id) { SchemaBuilder* builder = data; char* name_ptr = allocate_string(name); - print_physical_name(name_ptr, metadata, builder->engine); PRINT_CHILD_VISIT("array", name_ptr, sibling_list_id, "Types", child_list_id); SchemaItem* array_item = add_to_list(&builder->lists[sibling_list_id], name_ptr, "array", is_nullable); array_item->children = child_list_id; + array_item->column_mapping_id = read_column_mapping_id(metadata, builder->engine); } void visit_map( @@ -191,15 +195,15 @@ void visit_map( uintptr_t sibling_list_id, struct KernelStringSlice name, bool is_nullable, - const CStringMap * metadata, + const CMetadataMap * metadata, uintptr_t child_list_id) { SchemaBuilder* builder = data; char* name_ptr = allocate_string(name); - print_physical_name(name_ptr, metadata, builder->engine); PRINT_CHILD_VISIT("map", name_ptr, sibling_list_id, "Types", child_list_id); SchemaItem* map_item = add_to_list(&builder->lists[sibling_list_id], name_ptr, "map", is_nullable); map_item->children = child_list_id; + map_item->column_mapping_id = read_column_mapping_id(metadata, builder->engine); } void visit_decimal( @@ -207,7 +211,7 @@ void visit_decimal( uintptr_t sibling_list_id, struct KernelStringSlice name, bool is_nullable, - const CStringMap * metadata, + const CMetadataMap * metadata, uint8_t precision, uint8_t scale) { @@ -215,9 +219,9 @@ void visit_decimal( char* name_ptr = allocate_string(name); char* type = malloc(19 * sizeof(char)); snprintf(type, 19, "decimal(%u)(%d)", precision, scale); - print_physical_name(name_ptr, metadata, builder->engine); PRINT_NO_CHILD_VISIT(type, name_ptr, sibling_list_id); - add_to_list(&builder->lists[sibling_list_id], name_ptr, type, is_nullable); + SchemaItem* item = add_to_list(&builder->lists[sibling_list_id], name_ptr, type, is_nullable); + item->column_mapping_id = read_column_mapping_id(metadata, builder->engine); } void visit_simple_type( @@ -225,18 +229,18 @@ void visit_simple_type( uintptr_t sibling_list_id, struct KernelStringSlice name, bool is_nullable, - const CStringMap * metadata, + const CMetadataMap * metadata, char* type) { SchemaBuilder* builder = data; char* name_ptr = allocate_string(name); - print_physical_name(name_ptr, metadata, builder->engine); PRINT_NO_CHILD_VISIT(type, name_ptr, sibling_list_id); - add_to_list(&builder->lists[sibling_list_id], name_ptr, type, is_nullable); + SchemaItem* item = add_to_list(&builder->lists[sibling_list_id], name_ptr, type, is_nullable); + item->column_mapping_id = read_column_mapping_id(metadata, builder->engine); } #define DEFINE_VISIT_SIMPLE_TYPE(typename) \ - void visit_##typename(void* data, uintptr_t sibling_list_id, struct KernelStringSlice name, bool is_nullable, const CStringMap * metadata)\ + void visit_##typename(void* data, uintptr_t sibling_list_id, struct KernelStringSlice name, bool is_nullable, const CMetadataMap * metadata)\ { \ visit_simple_type(data, sibling_list_id, name, is_nullable, metadata, #typename); \ } @@ -253,6 +257,26 @@ DEFINE_VISIT_SIMPLE_TYPE(binary) DEFINE_VISIT_SIMPLE_TYPE(date) DEFINE_VISIT_SIMPLE_TYPE(timestamp) DEFINE_VISIT_SIMPLE_TYPE(timestamp_ntz) +DEFINE_VISIT_SIMPLE_TYPE(void) +DEFINE_VISIT_SIMPLE_TYPE(variant) + +void visit_interval_year_month(void* data, + uintptr_t sibling_list_id, + struct KernelStringSlice name, + bool is_nullable, + const CMetadataMap * metadata) +{ + visit_simple_type(data, sibling_list_id, name, is_nullable, metadata, "interval year to month"); +} + +void visit_interval_day_time(void* data, + uintptr_t sibling_list_id, + struct KernelStringSlice name, + bool is_nullable, + const CMetadataMap * metadata) +{ + visit_simple_type(data, sibling_list_id, name, is_nullable, metadata, "interval day to second"); +} // free all the data in the builder and the builder itself void free_builder(SchemaBuilder* builder) @@ -262,6 +286,7 @@ void free_builder(SchemaBuilder* builder) for (uint32_t j = 0; j < list->len; j++) { SchemaItem* item = list->list + j; free(item->name); + free(item->column_mapping_id); // NULL when the field carried no column-mapping id; free(NULL) is a no-op // don't free item->type, those are static strings if (!strncmp(item->type, "decimal", 7)) { // except decimal types, we malloc'd those :) @@ -307,6 +332,10 @@ CSchema* get_cschema(SharedSnapshot* snapshot, SharedExternEngine* engine) .visit_date = visit_date, .visit_timestamp = visit_timestamp, .visit_timestamp_ntz = visit_timestamp_ntz, + .visit_interval_year_month = visit_interval_year_month, + .visit_interval_day_time = visit_interval_day_time, + .visit_void = visit_void, + .visit_variant = visit_variant, }; SharedSchema* schema = logical_schema(snapshot); uintptr_t schema_list_id = visit_schema(schema, &visitor); diff --git a/ffi/examples/uc-catalog-example/CMakeLists.txt b/ffi/examples/uc-catalog-example/CMakeLists.txt deleted file mode 100644 index 66935f199e..0000000000 --- a/ffi/examples/uc-catalog-example/CMakeLists.txt +++ /dev/null @@ -1,28 +0,0 @@ -cmake_minimum_required(VERSION 3.12) -project(uc_catalog_example) - -add_executable(uc_catalog_example uc_catalog_example.c ../common/kernel_utils.c) -target_compile_definitions(uc_catalog_example PUBLIC DEFINE_DEFAULT_ENGINE_BASE) -target_include_directories(uc_catalog_example PUBLIC - "${CMAKE_CURRENT_SOURCE_DIR}/../../../target/ffi-headers" - "${CMAKE_CURRENT_SOURCE_DIR}/../common") -target_link_directories(uc_catalog_example PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/../../../target/debug") -target_link_libraries(uc_catalog_example PUBLIC delta_kernel_ffi) - -if(WIN32) - set(CMAKE_C_FLAGS_DEBUG "/MT") - target_link_libraries(uc_catalog_example PUBLIC ws2_32 userenv bcrypt ncrypt crypt32 secur32 ntdll RuntimeObject) -endif(WIN32) - -if(MSVC) - target_compile_options(uc_catalog_example PRIVATE /W3 /WX) -else() - target_compile_options(uc_catalog_example PRIVATE -Wall -Wextra -Wpedantic -Werror -g -fsanitize=address) - target_link_options(uc_catalog_example PRIVATE -g -fsanitize=address) -endif() - -# Testing -include(CTest) -set(ExprTestRunner "../../../tests/test-uc-catalog-ffi/run_test.sh") -set(ExprExpectedPath "../../../tests/test-uc-catalog-ffi/expected.txt") -add_test(NAME test_uc_catalog_ffi COMMAND ${ExprTestRunner} ${ExprExpectedPath}) diff --git a/ffi/examples/uc-catalog-example/README.md b/ffi/examples/uc-catalog-example/README.md deleted file mode 100644 index ca7beca1b0..0000000000 --- a/ffi/examples/uc-catalog-example/README.md +++ /dev/null @@ -1,35 +0,0 @@ -uc catalog example -========== - -Simple example to show how to use the uc-catalog ffi features - -# Building - -This example is built with [cmake]. Instructions below assume you start in the directory containing this README. - -Note that prior to building these examples you must build `delta_kernel_ffi` with all feature enabled (see [the FFI readme] for details). TLDR: -```bash -# from repo root -$ cargo build -p delta_kernel_ffi [--release] --all-features -# from ffi/ dir -$ cargo build [--release] --all-features -``` - -There are two configurations that can currently be configured in cmake: -```bash -# turn on VERBOSE mode (default is off) - print more diagnostics -$ cmake -DVERBOSE=yes .. -# turn off PRINT_DATA (default is on) - see below -$ cmake -DPRINT_DATA=no .. -``` - -## Linux / MacOS - -Most likely something like this should work: -``` -$ mkdir build -$ cd build -$ cmake .. -$ make -$ ./uc_catalog_example [path/to/table] -``` diff --git a/ffi/examples/update-dv/CMakeLists.txt b/ffi/examples/update-dv/CMakeLists.txt new file mode 100644 index 0000000000..a32ce2914f --- /dev/null +++ b/ffi/examples/update-dv/CMakeLists.txt @@ -0,0 +1,22 @@ +cmake_minimum_required(VERSION 3.12) +project(update_dv) + +add_executable(update_dv update_dv.c ../common/kernel_utils.c) +target_compile_definitions(update_dv PUBLIC DEFINE_DEFAULT_ENGINE_BASE) +target_include_directories(update_dv PUBLIC + "${CMAKE_CURRENT_SOURCE_DIR}/../../../target/ffi-headers" + "${CMAKE_CURRENT_SOURCE_DIR}/../common") +target_link_directories(update_dv PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/../../../target/debug") +target_link_libraries(update_dv PUBLIC delta_kernel_ffi) + +if(WIN32) + set(CMAKE_C_FLAGS_DEBUG "/MT") + target_link_libraries(update_dv PUBLIC ws2_32 userenv bcrypt ncrypt crypt32 secur32 ntdll RuntimeObject) +endif(WIN32) + +if(MSVC) + target_compile_options(update_dv PRIVATE /W4 /WX) +else() + target_compile_options(update_dv PRIVATE -Wall -Wextra -Wpedantic -Werror -g -fsanitize=address) + target_link_options(update_dv PRIVATE -g -fsanitize=address) +endif() diff --git a/ffi/examples/update-dv/README.md b/ffi/examples/update-dv/README.md new file mode 100644 index 0000000000..5b6ff9fb3b --- /dev/null +++ b/ffi/examples/update-dv/README.md @@ -0,0 +1,44 @@ +update-dv +========= + +C FFI example for connector-authored deletion vector updates. The example assumes the +connector has already written a DV file (or built inline DV bytes) and knows the descriptor +fields to install in the Delta log. + +It demonstrates: + +- `dv_descriptor_map_new` +- `dv_descriptor_new` +- `dv_descriptor_map_insert` +- `scan_metadata_iter_init` +- `transaction_update_deletion_vectors` +- `commit` + +# Building + +```bash +# from repo root +$ cargo build -p delta_kernel_ffi +# from this directory +$ mkdir build && cd build && cmake .. && make +``` + +# Running + +```bash +$ ./update_dv /path/to/table data-file.parquet p file:///tmp/table/dv.bin 1 36 2 +``` + +Arguments: + +1. Table path +2. Data file path exactly as it appears in scan metadata / the Add action +3. Storage type: `u` (persisted relative), `i` (inline), or `p` (persisted absolute) +4. `pathOrInlineDv` +5. Offset, or `-` to omit it +6. `sizeInBytes` +7. Cardinality + +The table must have the `deletionVectors` reader/writer feature and +`delta.enableDeletionVectors=true`. The example does not write the DV file itself; it only +installs the descriptor and lets the kernel stage the matching remove/add action pair. diff --git a/ffi/examples/update-dv/update_dv.c b/ffi/examples/update-dv/update_dv.c new file mode 100644 index 0000000000..2b3e0af446 --- /dev/null +++ b/ffi/examples/update-dv/update_dv.c @@ -0,0 +1,245 @@ +#include +#include +#include +#include +#include + +#include "delta_kernel_ffi.h" +#include "kernel_utils.h" + +// ============================================================================ +// update_dv -- C FFI example for connector-authored deletion vector updates +// ============================================================================ +// +// Usage: +// ./update_dv /path/to/table data-file.parquet p file:///tmp/table/dv.bin 1 36 2 +// +// The target table must already exist, have the deletionVectors reader/writer feature, +// and set delta.enableDeletionVectors=true. The DV file/bytes must already exist; this +// example only installs the descriptor into the Delta log. + +static int storage_type_code(const char* storage_type) { + if (strcmp(storage_type, "u") == 0) { + return 0; // PersistedRelative + } + if (strcmp(storage_type, "i") == 0) { + return 1; // Inline + } + if (strcmp(storage_type, "p") == 0) { + return 2; // PersistedAbsolute + } + return -1; +} + +static bool parse_i32(const char* text, int32_t* out) { + char* end = NULL; + long value = strtol(text, &end, 10); + if (end == text || *end != '\0' || value < INT32_MIN || value > INT32_MAX) { + return false; + } + *out = (int32_t)value; + return true; +} + +static bool parse_i64(const char* text, int64_t* out) { + char* end = NULL; + long long value = strtoll(text, &end, 10); + if (end == text || *end != '\0') { + return false; + } + *out = (int64_t)value; + return true; +} + +static void usage(const char* program) { + fprintf(stderr, + "Usage: %s /path/to/table data-file-path storage-type path-or-inline-dv " + "offset-or-- size-in-bytes cardinality\n\n" + "storage-type: u (persisted relative), i (inline), or p (persisted absolute)\n" + "offset-or--: byte offset for persisted DVs, or '-' to omit it\n", + program); +} + +int main(int argc, char* argv[]) { + if (argc != 8) { + usage(argv[0]); + return 1; + } + + char* table_path = argv[1]; + char* data_file_path = argv[2]; + int storage_type = storage_type_code(argv[3]); + char* path_or_inline_dv = argv[4]; + bool has_offset = strcmp(argv[5], "-") != 0; + int32_t offset = 0; + int32_t size_in_bytes = 0; + int64_t cardinality = 0; + + if (storage_type < 0) { + fprintf(stderr, "Invalid storage type: %s\n", argv[3]); + usage(argv[0]); + return 1; + } + if (has_offset && !parse_i32(argv[5], &offset)) { + fprintf(stderr, "Invalid offset: %s\n", argv[5]); + return 1; + } + if (!parse_i32(argv[6], &size_in_bytes)) { + fprintf(stderr, "Invalid size_in_bytes: %s\n", argv[6]); + return 1; + } + if (!parse_i64(argv[7], &cardinality)) { + fprintf(stderr, "Invalid cardinality: %s\n", argv[7]); + return 1; + } + + KernelStringSlice table_path_slice = { table_path, strlen(table_path) }; + KernelStringSlice data_file_path_slice = { data_file_path, strlen(data_file_path) }; + KernelStringSlice dv_slice = { path_or_inline_dv, strlen(path_or_inline_dv) }; + + // Owned handles start NULL and are released by the cleanup ladder below. Each FFI call + // that consumes a handle (e.g. `dv_descriptor_map_insert`, `snapshot_builder_build`, + // `transaction_update_deletion_vectors`, `commit`) is paired with an explicit NULL + // assignment so the cleanup block does not double-free. + int rc = 1; + Error* err = NULL; + HandleSharedExternEngine engine = NULL; + HandleExclusiveTransaction txn = NULL; + HandleExclusiveDvDescriptorMap map = NULL; + HandleExclusiveDvDescriptor descriptor = NULL; + HandleMutableFfiSnapshotBuilder snapshot_builder = NULL; + HandleSharedSnapshot snapshot = NULL; + HandleSharedScan scan_handle = NULL; + HandleSharedScanMetadataIterator scan_iter = NULL; + HandleExclusiveCommittedTransaction committed = NULL; + + // === Build engine === + ExternResultEngineBuilder engine_builder_res = + get_engine_builder(table_path_slice, allocate_error); + if (engine_builder_res.tag != OkEngineBuilder) { + err = (Error*)engine_builder_res.err; + print_error("Could not get engine builder.", err); + goto cleanup; + } + ExternResultHandleSharedExternEngine engine_res = builder_build(engine_builder_res.ok); + if (engine_res.tag != OkHandleSharedExternEngine) { + err = (Error*)engine_res.err; + print_error("Failed to build engine.", err); + goto cleanup; + } + engine = engine_res.ok; + + // === Start transaction === + ExternResultHandleExclusiveTransaction txn_res = transaction(table_path_slice, engine); + if (txn_res.tag != OkHandleExclusiveTransaction) { + err = (Error*)txn_res.err; + print_error("Failed to start transaction.", err); + goto cleanup; + } + txn = txn_res.ok; + + const char* engine_info = "update_dv_example"; + KernelStringSlice engine_info_slice = { engine_info, strlen(engine_info) }; + ExternResultHandleExclusiveTransaction with_info_res = + with_engine_info(txn, engine_info_slice, engine); + txn = NULL; // consumed by with_engine_info regardless of result + if (with_info_res.tag != OkHandleExclusiveTransaction) { + err = (Error*)with_info_res.err; + print_error("with_engine_info failed.", err); + goto cleanup; + } + txn = with_info_res.ok; + + // === Build descriptor map === + map = dv_descriptor_map_new(); + ExternResultHandleExclusiveDvDescriptor descriptor_res = dv_descriptor_new( + storage_type, dv_slice, has_offset, offset, size_in_bytes, cardinality, engine); + if (descriptor_res.tag != OkHandleExclusiveDvDescriptor) { + err = (Error*)descriptor_res.err; + print_error("dv_descriptor_new failed.", err); + goto cleanup; + } + descriptor = descriptor_res.ok; + + ExternResultbool insert_res = + dv_descriptor_map_insert(map, data_file_path_slice, descriptor, engine); + if (insert_res.tag != Okbool) { + err = (Error*)insert_res.err; + print_error("dv_descriptor_map_insert failed.", err); + goto cleanup; + } + descriptor = NULL; // consumed by dv_descriptor_map_insert on success + + // === Build a fresh scan metadata iterator for the update call === + ExternResultHandleMutableFfiSnapshotBuilder snapshot_builder_res = + get_snapshot_builder(table_path_slice, engine); + if (snapshot_builder_res.tag != OkHandleMutableFfiSnapshotBuilder) { + err = (Error*)snapshot_builder_res.err; + print_error("Failed to get snapshot builder.", err); + goto cleanup; + } + snapshot_builder = snapshot_builder_res.ok; + + ExternResultHandleSharedSnapshot snapshot_res = snapshot_builder_build(snapshot_builder); + snapshot_builder = NULL; // consumed by snapshot_builder_build + if (snapshot_res.tag != OkHandleSharedSnapshot) { + err = (Error*)snapshot_res.err; + print_error("Failed to load snapshot.", err); + goto cleanup; + } + snapshot = snapshot_res.ok; + + ExternResultHandleSharedScan scan_res = scan(snapshot, engine, NULL, NULL); + if (scan_res.tag != OkHandleSharedScan) { + err = (Error*)scan_res.err; + print_error("Failed to build scan.", err); + goto cleanup; + } + scan_handle = scan_res.ok; + + ExternResultHandleSharedScanMetadataIterator iter_res = + scan_metadata_iter_init(engine, scan_handle); + if (iter_res.tag != OkHandleSharedScanMetadataIterator) { + err = (Error*)iter_res.err; + print_error("scan_metadata_iter_init failed.", err); + goto cleanup; + } + scan_iter = iter_res.ok; + + ExternResultbool update_res = + transaction_update_deletion_vectors(txn, map, scan_iter, engine); + // map and scan_iter are consumed by transaction_update_deletion_vectors even on error. + map = NULL; + scan_iter = NULL; + if (update_res.tag != Okbool) { + err = (Error*)update_res.err; + print_error("transaction_update_deletion_vectors failed.", err); + goto cleanup; + } + + // === Commit === + ExternResultHandleExclusiveCommittedTransaction commit_res = commit(txn, engine); + txn = NULL; // consumed by commit + if (commit_res.tag != OkHandleExclusiveCommittedTransaction) { + err = (Error*)commit_res.err; + print_error("commit failed.", err); + goto cleanup; + } + committed = commit_res.ok; + printf("Committed DV update at version: %" PRIu64 "\n", + committed_transaction_version(&committed)); + rc = 0; + +cleanup: + if (err) free_error(err); + if (committed) free_committed_transaction(committed); + if (scan_iter) free_scan_metadata_iter(scan_iter); + if (scan_handle) free_scan(scan_handle); + if (snapshot) free_snapshot(snapshot); + if (snapshot_builder) free_snapshot_builder(snapshot_builder); + if (descriptor) free_dv_descriptor(descriptor); + if (map) free_dv_descriptor_map(map); + if (txn) free_transaction(txn); + if (engine) free_engine(engine); + return rc; +} diff --git a/ffi/examples/visit-expression/CMakeLists.txt b/ffi/examples/visit-expression/CMakeLists.txt index ca863e579f..fb965447a2 100644 --- a/ffi/examples/visit-expression/CMakeLists.txt +++ b/ffi/examples/visit-expression/CMakeLists.txt @@ -11,7 +11,7 @@ target_link_libraries(visit_expression PUBLIC delta_kernel_ffi) target_compile_options(visit_expression PUBLIC) if(MSVC) - target_compile_options(read_table PRIVATE /W4 /WX) + target_compile_options(visit_expression PRIVATE /W4 /WX) else() # no-strict-prototypes because arrow headers have fn defs without prototypes target_compile_options(visit_expression PRIVATE -Wall -Wextra -Wpedantic -Werror -Wno-strict-prototypes -g -fsanitize=address) diff --git a/ffi/examples/visit-expression/engine_to_kernel_expression.h b/ffi/examples/visit-expression/engine_to_kernel_expression.h index 87d3bbb5a8..e8a7034a21 100644 --- a/ffi/examples/visit-expression/engine_to_kernel_expression.h +++ b/ffi/examples/visit-expression/engine_to_kernel_expression.h @@ -56,6 +56,10 @@ uintptr_t convert_engine_to_kernel_literal( lit->value.long_data); case Date: return visit_expression_literal_date(state, lit->value.integer_data); + case IntervalYearMonth: + return visit_expression_literal_interval_year_month(state, lit->value.integer_data); + case IntervalDayTime: + return visit_expression_literal_interval_day_time(state, lit->value.long_data); case Binary: { return visit_expression_literal_binary( state, lit->value.binary.buf, lit->value.binary.len); @@ -72,7 +76,9 @@ uintptr_t convert_engine_to_kernel_literal( return result.ok; } case Null: { - ExternResultusize result = visit_expression_literal_null(state, allocate_error); + struct NullTypeInfo* nt = &lit->value.null_type; + ExternResultusize result = visit_expression_literal_null( + state, nt->type_tag, nt->precision, nt->scale, allocate_error); if (result.tag == Errusize) { print_error("visit_expression_literal_null failed", (Error*)result.err); free_error((Error*)result.err); @@ -247,8 +253,8 @@ uintptr_t convert_engine_to_kernel_expression_item( state, m2s->child_expr.list[0]); return visit_expression_map_to_struct(state, child); } - case Transform: - case FieldTransform: + case StructPatch: + case FieldPatch: case OpaqueExpression: case OpaquePredicate: fprintf(stderr, @@ -339,4 +345,3 @@ SharedPredicate* convert_engine_to_kernel_predicate( } } - diff --git a/ffi/examples/visit-expression/expression.h b/ffi/examples/visit-expression/expression.h index 920b0d54da..ded43e7b7c 100644 --- a/ffi/examples/visit-expression/expression.h +++ b/ffi/examples/visit-expression/expression.h @@ -46,6 +46,8 @@ enum LitType { Timestamp, TimestampNtz, Date, + IntervalYearMonth, + IntervalDayTime, Binary, Decimal, Null, @@ -59,8 +61,8 @@ enum ExpressionType { Literal, Unary, Column, - Transform, - FieldTransform, + StructPatch, + FieldPatch, OpaqueExpression, OpaquePredicate, Unknown, @@ -70,6 +72,8 @@ enum VariadicType { And, Or, StructExpression, + Coalesce, + ArrayConstructor, }; enum UnaryType { Not, IsNull }; typedef struct { @@ -92,14 +96,17 @@ struct Unary { enum UnaryType type; ExpressionItemList sub_expr; }; -struct TransformExpression { +struct StructPatchExpression { ExpressionItemList input_path; - ExpressionItemList field_transforms; + ExpressionItemList prepended_fields; + ExpressionItemList field_patches; + ExpressionItemList appended_fields; }; -struct FieldTransform { +struct FieldPatch { char* field_name; - ExpressionItemList exprs; - bool is_replace; + ExpressionItemList insertions; + bool keep_input; + bool optional; }; struct OpaqueExpression { HandleSharedOpaqueExpressionOp op; @@ -140,6 +147,11 @@ struct MapData { ExpressionItemList keys; ExpressionItemList vals; }; +struct NullTypeInfo { + uint8_t type_tag; + uint8_t precision; + uint8_t scale; +}; struct Literal { enum LitType type; union LiteralValue { @@ -156,6 +168,7 @@ struct Literal { struct MapData map_data; struct BinaryData binary; struct Decimal decimal; + struct NullTypeInfo null_type; } value; }; @@ -170,6 +183,7 @@ void put_expr_item(void* data, size_t sibling_list_id, void* ref, enum Expressio } ExpressionItemList get_expr_list(void* data, size_t list_id) { ExpressionBuilder* data_ptr = (ExpressionBuilder*)data; + assert(list_id != 0); assert(list_id < data_ptr->list_count); return data_ptr->lists[list_id]; } @@ -223,6 +237,16 @@ DEFINE_SIMPLE_SCALAR(visit_expr_boolean_literal, Boolean, _Bool, boolean_data); DEFINE_SIMPLE_SCALAR(visit_expr_timestamp_literal, Timestamp, int64_t, long_data); DEFINE_SIMPLE_SCALAR(visit_expr_timestamp_ntz_literal, TimestampNtz, int64_t, long_data); DEFINE_SIMPLE_SCALAR(visit_expr_date_literal, Date, int32_t, integer_data); +DEFINE_SIMPLE_SCALAR( + visit_expr_interval_year_month_literal, + IntervalYearMonth, + int32_t, + integer_data); +DEFINE_SIMPLE_SCALAR( + visit_expr_interval_day_time_literal, + IntervalDayTime, + int64_t, + long_data); #undef DEFINE_SIMPLE_SCALAR void visit_expr_string_literal(void* data, uintptr_t sibling_list_id, KernelStringSlice string) { @@ -269,9 +293,17 @@ void visit_expr_struct_literal(void* data, struct_data->values = get_expr_list(data, child_value_list_id); put_expr_item(data, sibling_list_id, literal, Literal); } -void visit_expr_null_literal(void* data, uintptr_t sibling_id_list) { +void visit_expr_null_literal( + void* data, + uintptr_t sibling_id_list, + uint8_t type_tag, + uint8_t precision, + uint8_t scale) { struct Literal* literal = malloc(sizeof(struct Literal)); literal->type = Null; + literal->value.null_type.type_tag = type_tag; + literal->value.null_type.precision = precision; + literal->value.null_type.scale = scale; put_expr_item(data, sibling_id_list, literal, Literal); } @@ -295,23 +327,17 @@ void visit_expr_variadic(void* data, DEFINE_VARIADIC(visit_expr_and, And) DEFINE_VARIADIC(visit_expr_or, Or) DEFINE_VARIADIC(visit_expr_struct_expr, StructExpression) +DEFINE_VARIADIC(visit_expr_coalesce, Coalesce) +DEFINE_VARIADIC(visit_expr_array, ArrayConstructor) #undef DEFINE_VARIADIC // Sort by field name, breaking ties by pointer address to ensure stability. -int transform_op_cmp(const void* a, const void* b) { - const struct FieldTransform* op_a = ((ExpressionItem*)a)->ref; - const struct FieldTransform* op_b = ((ExpressionItem*)b)->ref; - if (op_a->field_name == NULL && op_b->field_name == NULL) { - // break tie below - } else if (op_a->field_name == NULL) { - return -1; - } else if (op_b->field_name == NULL) { - return 1; - } else { - int cmp = strcmp(op_a->field_name, op_b->field_name); - if (cmp != 0) { - return cmp; - } // else break tie below +int patch_op_cmp(const void* a, const void* b) { + const struct FieldPatch* op_a = ((ExpressionItem*)a)->ref; + const struct FieldPatch* op_b = ((ExpressionItem*)b)->ref; + int cmp = strcmp(op_a->field_name, op_b->field_name); + if (cmp != 0) { + return cmp; } if (op_a < op_b) { return -1; @@ -322,37 +348,43 @@ int transform_op_cmp(const void* a, const void* b) { } } -void visit_transform_expr( +void visit_struct_patch_expr( void* data, uintptr_t sibling_list_id, uintptr_t input_path_list_id, - uintptr_t child_list_id) + uintptr_t prepended_field_list_id, + uintptr_t field_patch_list_id, + uintptr_t appended_field_list_id) { - struct TransformExpression* transform = malloc(sizeof(struct TransformExpression)); - transform->input_path = get_expr_list(data, input_path_list_id); - transform->field_transforms = get_expr_list(data, child_list_id); + struct StructPatchExpression* patch = malloc(sizeof(struct StructPatchExpression)); + patch->input_path = get_expr_list(data, input_path_list_id); + patch->prepended_fields = get_expr_list(data, prepended_field_list_id); + patch->field_patches = get_expr_list(data, field_patch_list_id); + patch->appended_fields = get_expr_list(data, appended_field_list_id); // stable sort the ops by field name to ensure deterministic output qsort( - transform->field_transforms.list, - transform->field_transforms.len, + patch->field_patches.list, + patch->field_patches.len, sizeof(ExpressionItem), - transform_op_cmp); + patch_op_cmp); - put_expr_item(data, sibling_list_id, transform, Transform); + put_expr_item(data, sibling_list_id, patch, StructPatch); } -void visit_field_transform( +void visit_field_patch( void* data, uintptr_t sibling_list_id, - const KernelStringSlice* field_name, - uintptr_t child_list_id, - bool is_replace) + KernelStringSlice field_name, + uintptr_t insertion_expr_list_id, + bool keep_input, + bool optional) { - struct FieldTransform* field_transform = malloc(sizeof(struct FieldTransform)); - field_transform->field_name = field_name? allocate_string(*field_name) : NULL; - field_transform->exprs = get_expr_list(data, child_list_id); - field_transform->is_replace = is_replace; - put_expr_item(data, sibling_list_id, field_transform, FieldTransform); + struct FieldPatch* field_patch = malloc(sizeof(struct FieldPatch)); + field_patch->field_name = allocate_string(field_name); + field_patch->insertions = get_expr_list(data, insertion_expr_list_id); + field_patch->keep_input = keep_input; + field_patch->optional = optional; + put_expr_item(data, sibling_list_id, field_patch, FieldPatch); } void visit_opaque_expr( void *data, @@ -456,8 +488,7 @@ uintptr_t make_field_list(void* data, uintptr_t reserve) { } ExpressionItemList construct_expression(SharedExpression* expression) { - ExpressionBuilder data = { 0 }; - make_field_list(&data, 0); // list id 0 is the invalid/missing/empty list + ExpressionBuilder data = { .list_count = 1 }; EngineExpressionVisitor visitor = { .data = &data, @@ -472,6 +503,8 @@ ExpressionItemList construct_expression(SharedExpression* expression) { .visit_literal_timestamp = visit_expr_timestamp_literal, .visit_literal_timestamp_ntz = visit_expr_timestamp_ntz_literal, .visit_literal_date = visit_expr_date_literal, + .visit_literal_interval_year_month = visit_expr_interval_year_month_literal, + .visit_literal_interval_day_time = visit_expr_interval_day_time_literal, .visit_literal_binary = visit_expr_binary_literal, .visit_literal_null = visit_expr_null_literal, .visit_literal_decimal = visit_expr_decimal_literal, @@ -494,12 +527,14 @@ ExpressionItemList construct_expression(SharedExpression* expression) { .visit_divide = visit_expr_divide, .visit_column = visit_expr_column, .visit_struct_expr = visit_expr_struct_expr, - .visit_transform_expr = visit_transform_expr, - .visit_field_transform = visit_field_transform, + .visit_struct_patch_expr = visit_struct_patch_expr, + .visit_field_patch = visit_field_patch, .visit_opaque_pred = visit_opaque_pred, .visit_opaque_expr = visit_opaque_expr, .visit_unknown = visit_unknown, .visit_map_to_struct = visit_map_to_struct_expr, + .visit_coalesce = visit_expr_coalesce, + .visit_array = visit_expr_array, }; uintptr_t top_level_id = visit_expression(&expression, &visitor); ExpressionItemList top_level_expr = data.lists[top_level_id]; @@ -508,7 +543,7 @@ ExpressionItemList construct_expression(SharedExpression* expression) { } ExpressionItemList construct_predicate(SharedPredicate* predicate) { - ExpressionBuilder data = { 0 }; + ExpressionBuilder data = { .list_count = 1 }; EngineExpressionVisitor visitor = { .data = &data, .make_field_list = make_field_list, @@ -522,6 +557,8 @@ ExpressionItemList construct_predicate(SharedPredicate* predicate) { .visit_literal_timestamp = visit_expr_timestamp_literal, .visit_literal_timestamp_ntz = visit_expr_timestamp_ntz_literal, .visit_literal_date = visit_expr_date_literal, + .visit_literal_interval_year_month = visit_expr_interval_year_month_literal, + .visit_literal_interval_day_time = visit_expr_interval_day_time_literal, .visit_literal_binary = visit_expr_binary_literal, .visit_literal_null = visit_expr_null_literal, .visit_literal_decimal = visit_expr_decimal_literal, @@ -547,6 +584,8 @@ ExpressionItemList construct_predicate(SharedPredicate* predicate) { .visit_opaque_expr = visit_opaque_expr, .visit_unknown = visit_unknown, .visit_map_to_struct = visit_map_to_struct_expr, + .visit_coalesce = visit_expr_coalesce, + .visit_array = visit_expr_array, }; uintptr_t top_level_id = visit_predicate(&predicate, &visitor); ExpressionItemList top_level_expr = data.lists[top_level_id]; @@ -569,18 +608,20 @@ void free_expression_item(ExpressionItem ref) { free(var); break; }; - case Transform: { - struct TransformExpression* transform = ref.ref; - free_expression_list(transform->input_path); - free_expression_list(transform->field_transforms); - free(transform); + case StructPatch: { + struct StructPatchExpression* patch = ref.ref; + free_expression_list(patch->input_path); + free_expression_list(patch->prepended_fields); + free_expression_list(patch->field_patches); + free_expression_list(patch->appended_fields); + free(patch); break; } - case FieldTransform: { - struct FieldTransform* field_transform = ref.ref; - free(field_transform->field_name); - free_expression_list(field_transform->exprs); - free(field_transform); + case FieldPatch: { + struct FieldPatch* field_patch = ref.ref; + free(field_patch->field_name); + free_expression_list(field_patch->insertions); + free(field_patch); break; } case OpaqueExpression: { @@ -642,6 +683,8 @@ void free_expression_item(ExpressionItem ref) { case Timestamp: case TimestampNtz: case Date: + case IntervalYearMonth: + case IntervalDayTime: case Decimal: case Null: break; diff --git a/ffi/examples/visit-expression/expression_print.h b/ffi/examples/visit-expression/expression_print.h index 44aa491837..311cd43e51 100644 --- a/ffi/examples/visit-expression/expression_print.h +++ b/ffi/examples/visit-expression/expression_print.h @@ -18,6 +18,22 @@ void print_expression_item_list(ExpressionItemList list, int depth) { print_tree_helper(list.list[i], depth); } } +void print_expression_item_list_field(const char* field_name, ExpressionItemList list, int depth) { + print_n_spaces(depth); + printf("%s\n", field_name); + print_expression_item_list(list, depth + 1); +} +void print_non_empty_expression_item_list_field( + const char* field_name, ExpressionItemList list, int depth) { + if (list.len == 0) { + return; + } + print_expression_item_list_field(field_name, list, depth); +} +void print_bool_field(const char* field_name, bool value, int depth) { + print_n_spaces(depth); + printf("%s: %s\n", field_name, value ? "true" : "false"); +} void print_opaque_op_name(void* op_type, KernelStringSlice name) { int len = name.len & 0x7fffffff; // truncate to 31 bits to ensure a positive value printf("%s(%.*s)\n", (char*) op_type, len, name.ptr); @@ -80,29 +96,38 @@ void print_tree_helper(ExpressionItem ref, int depth) { case StructExpression: printf("StructExpression\n"); break; + case Coalesce: + printf("Coalesce\n"); + break; + case ArrayConstructor: + printf("ArrayConstructor\n"); + break; } print_expression_item_list(var->exprs, depth + 1); break; } - case Transform: { - struct TransformExpression* transform = ref.ref; - printf("Transform\n"); - print_expression_item_list(transform->input_path, depth + 1); - print_expression_item_list(transform->field_transforms, depth + 1); + case StructPatch: { + struct StructPatchExpression* patch = ref.ref; + printf("StructPatch\n"); + print_non_empty_expression_item_list_field( + "input_path", patch->input_path, depth + 1); + print_non_empty_expression_item_list_field( + "prepended_fields", patch->prepended_fields, depth + 1); + print_non_empty_expression_item_list_field( + "field_patches", patch->field_patches, depth + 1); + print_non_empty_expression_item_list_field( + "appended_fields", patch->appended_fields, depth + 1); break; } - case FieldTransform: { - struct FieldTransform* field_transform = ref.ref; - if (!field_transform->field_name) { - printf("Prepend\n"); - } else if (!field_transform->is_replace) { - printf("Insert(%s)\n", field_transform->field_name); - } else if (!field_transform->exprs.len) { - printf("Drop(%s)\n", field_transform->field_name); - } else { - printf("Replace(%s)\n", field_transform->field_name); - } - print_expression_item_list(field_transform->exprs, depth + 1); + case FieldPatch: { + struct FieldPatch* field_patch = ref.ref; + printf("FieldPatch\n"); + print_n_spaces(depth + 1); + printf("field_name: %s\n", field_patch->field_name); + print_non_empty_expression_item_list_field( + "insertions", field_patch->insertions, depth + 1); + print_bool_field("keep_input", field_patch->keep_input, depth + 1); + print_bool_field("optional", field_patch->optional, depth + 1); break; } case OpaqueExpression: { @@ -159,6 +184,12 @@ void print_tree_helper(ExpressionItem ref, int depth) { case Date: printf("Date(%d)\n", lit->value.integer_data); break; + case IntervalYearMonth: + printf("IntervalYearMonth(%d)\n", lit->value.integer_data); + break; + case IntervalDayTime: + printf("IntervalDayTime(%lld)\n", (long long)lit->value.long_data); + break; case Binary: { printf("Binary("); for (size_t i = 0; i < lit->value.binary.len; i++) { @@ -176,9 +207,23 @@ void print_tree_helper(ExpressionItem ref, int depth) { dec->scale); break; } - case Null: - printf("Null\n"); + case Null: { + static const char* null_type_names[] = { + "Boolean", "Byte", "Short", "Integer", "Long", "Float", + "Double", "String", "Binary", "Date", "Timestamp", "TimestampNtz", + "Decimal", "IntervalYearMonth", "IntervalDayTime", + }; + const size_t null_type_count = sizeof(null_type_names) / sizeof(null_type_names[0]); + struct NullTypeInfo* nt = &lit->value.null_type; + if (nt->type_tag == 12) { + printf("Null(Decimal(%d,%d))\n", nt->precision, nt->scale); + } else if (nt->type_tag < null_type_count) { + printf("Null(%s)\n", null_type_names[nt->type_tag]); + } else { + printf("Null(tag=%d)\n", nt->type_tag); + } break; + } case Struct: printf("Struct\n"); struct Struct* struct_data = &lit->value.struct_data; diff --git a/ffi/examples/visit-expression/visit_expression.c b/ffi/examples/visit-expression/visit_expression.c index 1f56b9ba3b..96fc41b063 100644 --- a/ffi/examples/visit-expression/visit_expression.c +++ b/ffi/examples/visit-expression/visit_expression.c @@ -91,7 +91,7 @@ int main() { "This test demonstrates the full range of expression types.\n" "Some types are not yet supported in round-trip reconstruction:\n" " - Struct/Array/Map literals (nested data structures)\n" - " - Transform expressions (schema evolution operations)\n" + " - StructPatch expressions (struct transformation operations)\n" " - Opaque expressions (custom user-defined expressions)" }, { @@ -102,8 +102,8 @@ int main() { .description = "This test validates expressions/predicates with full support.\n" "Supported types: primitives (int, long, float, double, bool, " - "string),\n temporal (date, timestamp, timestamp_ntz), binary, " - "decimal, null,\n binary operations (+, -, *, /), struct " + "string),\n temporal (date, timestamp, timestamp_ntz), intervals, " + "binary, decimal, null,\n binary operations (+, -, *, /), struct " "expressions, predicates (eq, ne, lt, le,\n gt, ge, distinct, " "is_null, is_not_null, not, and, or)" } diff --git a/ffi/examples/write-table/CMakeLists.txt b/ffi/examples/write-table/CMakeLists.txt new file mode 100644 index 0000000000..3a08d47dd8 --- /dev/null +++ b/ffi/examples/write-table/CMakeLists.txt @@ -0,0 +1,30 @@ +cmake_minimum_required(VERSION 3.12) +project(write_table) + +add_executable(write_table write_table.c ../common/kernel_utils.c) +target_compile_definitions(write_table PUBLIC DEFINE_DEFAULT_ENGINE_BASE) +target_include_directories(write_table PUBLIC + "${CMAKE_CURRENT_SOURCE_DIR}/../../../target/ffi-headers" + "${CMAKE_CURRENT_SOURCE_DIR}/../common") +target_link_directories(write_table PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/../../../target/debug") +target_link_libraries(write_table PUBLIC delta_kernel_ffi) + +if(WIN32) + set(CMAKE_C_FLAGS_DEBUG "/MT") + target_link_libraries(write_table PUBLIC ws2_32 userenv bcrypt ncrypt crypt32 secur32 ntdll RuntimeObject) +endif(WIN32) + +if(MSVC) + target_compile_options(write_table PRIVATE /W4 /WX) +else() + target_compile_options(write_table PRIVATE -Wall -Wextra -Wpedantic -Werror -g -fsanitize=address) + target_link_options(write_table PRIVATE -g -fsanitize=address) +endif() + +# Testing +include(CTest) +set(TestRunner "../../../tests/write-table-testing/run_test.sh") +set(ExpectedPath "../../../tests/write-table-testing/expected-data") +set(CreateTableBinary "${CMAKE_CURRENT_SOURCE_DIR}/../create-table/build/create_table") +add_test(NAME write_table_empty_commit + COMMAND ${TestRunner} ${CreateTableBinary} ${ExpectedPath}/empty-commit.expected) diff --git a/ffi/examples/write-table/README.md b/ffi/examples/write-table/README.md new file mode 100644 index 0000000000..1f92e39748 --- /dev/null +++ b/ffi/examples/write-table/README.md @@ -0,0 +1,30 @@ +write-table +=========== + +C FFI example for the write transaction surface. Demonstrates `transaction`, +`with_engine_info`, `get_unpartitioned_write_context`, the four write-context accessors +(`get_write_schema`, `get_physical_write_schema`, `get_logical_to_physical`, `get_write_path`), +`set_data_change`, and `commit` against an existing table. + +# Building + +```bash +# from repo root +$ cargo build -p delta_kernel_ffi +# from this directory +$ mkdir build && cd build && cmake .. && make +$ ./write_table /path/to/existing/table +``` + +# Limitations + +This example currently commits **empty** transactions. Staging new parquet files requires +building an Arrow batch that matches `Transaction::add_files_schema` (`path`, +`partitionValues`, `size`, `modificationTime`, `stats`) and handing it to `add_files` via +`get_engine_data`. Constructing that batch from C needs arrow-glib (or a similar C-level +Arrow builder). A shared `ffi/examples/common/` arrow-glib writer helper is planned as a +follow-up; once it lands, this example should grow an `add_files` flow alongside a +`with_domain_metadata` / `with_domain_metadata_removed` demo (the `domainMetadata` writer +feature can be enabled today via the existing `create_table_builder_with_table_property` +API by setting `delta.feature.domainMetadata=supported`). + diff --git a/ffi/examples/write-table/write_table.c b/ffi/examples/write-table/write_table.c new file mode 100644 index 0000000000..6bcaad9997 --- /dev/null +++ b/ffi/examples/write-table/write_table.c @@ -0,0 +1,183 @@ +#include +#include +#include +#include + +#include "delta_kernel_ffi.h" +#include "kernel_utils.h" + +// ============================================================================ +// write_table -- C FFI example for the write transaction surface +// ============================================================================ +// +// Usage: +// ./write_table /path/to/existing/table +// +// The target table must already exist (e.g. created by the `create-table` example). +// +// Demonstrates the write-path FFI surface: +// - transaction(path, engine) to start an existing-table transaction +// - with_engine_info(txn, "...", engine) to set commitInfo.engineInfo +// - get_unpartitioned_write_context(txn, engine) plus the four +// write-context accessors an engine needs when writing parquet files itself: +// - get_write_schema -- logical (user-facing) schema +// - get_physical_write_schema -- on-disk parquet schema (carries +// parquet.field.id under column mapping) +// - get_logical_to_physical -- transform to apply per batch +// - get_write_path -- table root URL (partitioned write +// directory support tracked by #2355) +// - set_data_change(txn, false) because this empty commit does not add data +// - commit(txn, engine) to produce an empty commit, returning a CommittedTransaction handle +// - committed_transaction_version + committed_transaction_post_commit_snapshot to read the +// version and the post-commit snapshot directly from the result, avoiding a fresh +// snapshot load +// - free_committed_transaction to release the result handle +// +// NOTE: This example does NOT call add_files. Staging new files requires building an Arrow +// RecordBatch that matches Transaction::add_files_schema (path, partitionValues, size, +// modificationTime, stats), which needs arrow-glib (or equivalent) on the C side to +// construct. That flow is tracked as a follow-up; once the shared arrow-glib writer helper +// lands in ffi/examples/common/, this example should be extended to stage a real parquet +// file and exercise the full add_files -> commit path. + +int main(int argc, char* argv[]) { + if (argc != 2) { + fprintf(stderr, "Usage: %s /path/to/existing/table\n", argv[0]); + return 1; + } + char* table_path = argv[1]; + printf("Writing empty commit to %s\n", table_path); + KernelStringSlice table_path_slice = { table_path, strlen(table_path) }; + + // === Build engine === + ExternResultEngineBuilder engine_builder_res = + get_engine_builder(table_path_slice, allocate_error); + if (engine_builder_res.tag != OkEngineBuilder) { + print_error("Could not get engine builder.", (Error*)engine_builder_res.err); + free_error((Error*)engine_builder_res.err); + return 1; + } + ExternResultHandleSharedExternEngine engine_res = builder_build(engine_builder_res.ok); + if (engine_res.tag != OkHandleSharedExternEngine) { + print_error("Failed to build engine.", (Error*)engine_res.err); + free_error((Error*)engine_res.err); + return 1; + } + SharedExternEngine* engine = engine_res.ok; + + // === Start a transaction on the latest snapshot === + ExternResultHandleExclusiveTransaction txn_res = transaction(table_path_slice, engine); + if (txn_res.tag != OkHandleExclusiveTransaction) { + print_error("Failed to start transaction.", (Error*)txn_res.err); + free_error((Error*)txn_res.err); + free_engine(engine); + return 1; + } + ExclusiveTransaction* txn = txn_res.ok; + + // set_data_change does not consume the handle. This commit has no data, so dataChange=false. + // Setting it here (before with_engine_info) is fine: the consume-and-return chain below + // preserves staged transaction state across handle handoffs. + set_data_change(txn, false); + + // Attach engine_info. CONSUMES and returns the transaction handle. + const char* engine_info = "write_table_example"; + KernelStringSlice engine_info_slice = { engine_info, strlen(engine_info) }; + ExternResultHandleExclusiveTransaction with_info_res = + with_engine_info(txn, engine_info_slice, engine); + if (with_info_res.tag != OkHandleExclusiveTransaction) { + print_error("with_engine_info failed.", (Error*)with_info_res.err); + free_error((Error*)with_info_res.err); + free_engine(engine); + return 1; + } + txn = with_info_res.ok; + + // === Inspect the unpartitioned write context === + // + // The WriteContext carries the schema an engine's parquet writer should use plus the table + // root URL it should write under. This example does not actually write any files, but we + // print these so users see the shape of the information they'd consume in a real engine. + ExternResultHandleSharedWriteContext wc_res = get_unpartitioned_write_context(txn, engine); + if (wc_res.tag != OkHandleSharedWriteContext) { + print_error("get_unpartitioned_write_context failed.", (Error*)wc_res.err); + free_error((Error*)wc_res.err); + free_transaction(txn); + free_engine(engine); + return 1; + } + SharedWriteContext* write_context = wc_res.ok; + + // SharedSchema and SharedExpression are opaque in the C API. Schemas are walked with + // visit_schema (see read-table/schema.h); expressions are consumed by passing them to + // new_expression_evaluator. A real engine feeds (logical_schema, logical_to_physical, + // physical_schema) into new_expression_evaluator and applies the resulting evaluator to + // each batch before handing the rewritten data to its parquet writer. + SharedSchema* logical_schema = get_write_schema(write_context); + SharedSchema* physical_schema = get_physical_write_schema(write_context); + SharedExpression* logical_to_physical = get_logical_to_physical(write_context); + printf("Write context:\n"); + printf(" logical_schema: %s\n", logical_schema ? "" : ""); + printf(" physical_schema: %s\n", physical_schema ? "" : ""); + printf(" logical_to_physical: %s\n", logical_to_physical ? "" : ""); + char* write_path = get_write_path(write_context, allocate_string); + if (write_path) { + printf(" write_path: %s\n", write_path); + free(write_path); + } else { + printf(" write_path: \n"); + } + free_kernel_expression(logical_to_physical); + free_schema(physical_schema); + free_schema(logical_schema); + free_write_context(write_context); + + // === Commit === + ExternResultHandleExclusiveCommittedTransaction commit_res = commit(txn, engine); + if (commit_res.tag != OkHandleExclusiveCommittedTransaction) { + print_error("commit failed.", (Error*)commit_res.err); + free_error((Error*)commit_res.err); + free_engine(engine); + return 1; + } + HandleExclusiveCommittedTransaction committed = commit_res.ok; + printf("Committed version: %" PRIu64 "\n", committed_transaction_version(&committed)); + + // === Read post-commit snapshot directly from the CommittedTransaction === + // Avoids a fresh snapshot load: the kernel hands back the already-built snapshot + // for the post-commit version. + struct OptionalValueHandleSharedSnapshot post_commit = + committed_transaction_post_commit_snapshot(&committed); + if (post_commit.tag == SomeHandleSharedSnapshot) { + HandleSharedSnapshot snap = post_commit.some; + printf("Post-commit snapshot version: %" PRIu64 "\n", version(snap)); + free_snapshot(snap); + } else { + printf("No post-commit snapshot available; loading via snapshot builder.\n"); + ExternResultHandleMutableFfiSnapshotBuilder snapshot_builder_res = + get_snapshot_builder(table_path_slice, engine); + if (snapshot_builder_res.tag != OkHandleMutableFfiSnapshotBuilder) { + print_error("Failed to get snapshot builder.", (Error*)snapshot_builder_res.err); + free_error((Error*)snapshot_builder_res.err); + free_committed_transaction(committed); + free_engine(engine); + return 1; + } + ExternResultHandleSharedSnapshot snap_res = + snapshot_builder_build(snapshot_builder_res.ok); + if (snap_res.tag != OkHandleSharedSnapshot) { + print_error("Failed to load snapshot after commit.", (Error*)snap_res.err); + free_error((Error*)snap_res.err); + free_committed_transaction(committed); + free_engine(engine); + return 1; + } + HandleSharedSnapshot loaded = snap_res.ok; + printf("Snapshot version after commit: %" PRIu64 "\n", version(loaded)); + free_snapshot(loaded); + } + + free_committed_transaction(committed); + free_engine(engine); + return 0; +} diff --git a/ffi/src/alloc_stats.rs b/ffi/src/alloc_stats.rs new file mode 100644 index 0000000000..0b76987556 --- /dev/null +++ b/ffi/src/alloc_stats.rs @@ -0,0 +1,121 @@ +//! FFI access to `peak_alloc`'s process-wide Rust allocation counters. +//! +//! # Scope +//! +//! Counters record requested allocation sizes only, so they are a lower bound on process RSS: +//! C/`mmap` allocations and allocator overhead are not included. Values are process-global and +//! advisory, rather than measurements for a single operation. `peak_alloc` uses relaxed atomic +//! operations to update its counters. Its peak reset is not coordinated with allocation, so callers +//! must serialize a reset against native work when they require a meaningful post-reset peak. + +#[cfg(feature = "alloc-tracking")] +use peak_alloc::PeakAlloc; + +#[cfg(feature = "alloc-tracking")] +#[global_allocator] +static GLOBAL_ALLOC: PeakAlloc = PeakAlloc; + +/// Whether this library was built with allocation tracking. +/// +/// The `*_native_bytes` getters return zero when this is false. +#[no_mangle] +pub extern "C" fn alloc_tracking_enabled() -> bool { + cfg!(feature = "alloc-tracking") +} + +/// Reported peak simultaneously live native bytes since the library was loaded or reset. +/// +/// The value is process-wide, not per-operation, and counts requested Rust allocation sizes only. +/// Returns zero when built without `alloc-tracking`. +#[no_mangle] +pub extern "C" fn peak_native_bytes() -> u64 { + #[cfg(feature = "alloc-tracking")] + { + GLOBAL_ALLOC.peak_usage() as u64 + } + #[cfg(not(feature = "alloc-tracking"))] + { + 0 + } +} + +/// Native bytes currently live (allocated but not yet freed). +/// +/// The value is process-wide and counts requested Rust allocation sizes only. Returns zero when +/// built without `alloc-tracking`. +#[no_mangle] +pub extern "C" fn current_native_bytes() -> u64 { + #[cfg(feature = "alloc-tracking")] + { + GLOBAL_ALLOC.current_usage() as u64 + } + #[cfg(not(feature = "alloc-tracking"))] + { + 0 + } +} + +/// Resets the peak to the current live total and returns the peak sampled immediately before reset. +/// +/// This is process-global and cannot provide a per-task baseline while allocation is concurrent. +/// `peak_alloc` samples and resets separately, so concurrent allocation can cause the returned +/// value to understate the cleared peak and leave the reported peak lower than the current total. +/// Returns zero when built without `alloc-tracking`. +#[no_mangle] +pub extern "C" fn reset_peak_native_bytes() -> u64 { + #[cfg(feature = "alloc-tracking")] + { + let previous_peak = GLOBAL_ALLOC.peak_usage() as u64; + GLOBAL_ALLOC.reset_peak_usage(); + previous_peak + } + #[cfg(not(feature = "alloc-tracking"))] + { + 0 + } +} + +#[cfg(all(test, not(feature = "alloc-tracking")))] +mod disabled_tests { + use super::{ + alloc_tracking_enabled, current_native_bytes, peak_native_bytes, reset_peak_native_bytes, + }; + + #[test] + fn getters_report_disabled_tracking() { + assert!(!alloc_tracking_enabled()); + assert_eq!(peak_native_bytes(), 0); + assert_eq!(current_native_bytes(), 0); + assert_eq!(reset_peak_native_bytes(), 0); + } +} + +#[cfg(all(test, feature = "alloc-tracking"))] +mod global_allocator_tests { + use super::{ + alloc_tracking_enabled, current_native_bytes, peak_native_bytes, reset_peak_native_bytes, + }; + + // Far above incidental harness allocation, so the bounds below cannot be met by noise. + const N: usize = 8 * 1024 * 1024; + + #[test] + fn installed_global_allocator_accounts_a_large_allocation() { + assert!(alloc_tracking_enabled()); + let _ = reset_peak_native_bytes(); + let before = current_native_bytes(); + + let buf = vec![0u8; N]; + let during = current_native_bytes(); + assert!( + during >= before + N as u64, + "alloc not tracked: {before} -> {during}" + ); + assert!(peak_native_bytes() >= during); + + drop(buf); + let previous_peak = reset_peak_native_bytes(); + assert!(previous_peak >= during); + assert!(peak_native_bytes() >= current_native_bytes()); + } +} diff --git a/ffi/src/commit_range.rs b/ffi/src/commit_range.rs new file mode 100644 index 0000000000..57d2658c7f --- /dev/null +++ b/ffi/src/commit_range.rs @@ -0,0 +1,769 @@ +use std::sync::{Arc, Mutex, MutexGuard}; + +use delta_kernel::commit_range::{CommitAction, CommitRange, DeltaAction as KernelDeltaAction}; +use delta_kernel::snapshot::SnapshotRef; +use delta_kernel::{DeltaResult, DeltaResultIteratorStatic, Error, Version}; +use delta_kernel_ffi_macros::handle_descriptor; +use url::Url; + +use crate::engine_funcs::{ExclusiveFileReadResultIterator, FileReadResultIterator}; +use crate::handle::Handle; +use crate::{ + unwrap_and_parse_path_as_url, ExternEngine, ExternResult, IntoExternResult, KernelStringSlice, + NullableCvoid, SharedExternEngine, SharedSnapshot, +}; + +/// An opaque, shared handle owning a [`CommitRange`] produced by +/// [`commit_range_builder_build`]. The caller owns the handle and must release it with +/// [`free_commit_range`]. +#[handle_descriptor(target=CommitRange, mutable=false, sized=true)] +pub struct SharedCommitRange; + +/// Opaque builder for constructing a [`CommitRange`] from a table path. +/// +/// Create with [`commit_range_builder_for`]. Optionally pin the end of the range with +/// [`commit_range_builder_set_end_version`]. Finally, call [`commit_range_builder_build`] to +/// consume the builder and obtain the range. To discard the builder without building, call +/// [`free_commit_range_builder`]. +pub struct FfiCommitRangeBuilder { + engine: Arc, + table_root: Url, + start_version: Version, + end_version: Option, +} + +/// An opaque handle with exclusive (Box-like) ownership of a [`FfiCommitRangeBuilder`]. +#[handle_descriptor(target=FfiCommitRangeBuilder, mutable=true, sized=true)] +pub struct MutableFfiCommitRangeBuilder; + +fn make_commit_range_builder( + table_root: Url, + start_version: Version, + engine: Arc, +) -> Handle { + Box::new(FfiCommitRangeBuilder { + engine, + table_root, + start_version, + end_version: None, + }) + .into() +} + +/// Get a builder for constructing a [`CommitRange`] from a table path, starting at +/// `start_version`. +/// +/// The caller owns the returned handle and must eventually call either +/// [`commit_range_builder_build`] to produce a [`CommitRange`], or [`free_commit_range_builder`] +/// to drop it without building. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid path and engine handle. +#[no_mangle] +pub unsafe extern "C" fn commit_range_builder_for( + path: KernelStringSlice, + start_version: Version, + engine: Handle, +) -> ExternResult> { + let engine_ref = unsafe { engine.as_ref() }; + let engine_arc = unsafe { engine.clone_as_arc() }; + let url = unsafe { unwrap_and_parse_path_as_url(path) }; + url.map(|url| make_commit_range_builder(url, start_version, engine_arc)) + .into_extern_result(&engine_ref) +} + +/// Pin the end version of the range. When omitted, the range extends to the latest committed +/// version observed at build time. +/// +/// # Safety +/// +/// Caller must pass a valid builder pointer. +#[no_mangle] +pub unsafe extern "C" fn commit_range_builder_set_end_version( + builder: &mut Handle, + end_version: Version, +) { + unsafe { builder.as_mut() }.end_version = Some(end_version); +} + +/// Consume the builder and return a [`CommitRange`]. After calling, the builder pointer is no +/// longer valid. The builder is always freed by this call, whether or not it succeeds. +/// +/// Returns an error if the resolved version range is invalid (start > end), the listed commits +/// are non-contiguous, or the requested start version is not present. +/// +/// # Safety +/// +/// Caller must pass a valid builder pointer and must not use it again after this call. +#[no_mangle] +pub unsafe extern "C" fn commit_range_builder_build( + builder: Handle, +) -> ExternResult> { + let builder_box = unsafe { builder.into_inner() }; + let engine = builder_box.engine.clone(); + commit_range_builder_build_impl(*builder_box).into_extern_result(&engine.as_ref()) +} + +fn commit_range_builder_build_impl( + builder: FfiCommitRangeBuilder, +) -> DeltaResult> { + let engine = builder.engine.engine(); + let mut kernel_builder = CommitRange::builder_for(builder.table_root, builder.start_version); + if let Some(end_version) = builder.end_version { + kernel_builder = kernel_builder.with_end_version(end_version); + } + let range = kernel_builder.build(engine.as_ref())?; + Ok(Arc::new(range).into()) +} + +/// Free a commit range builder without building a range (e.g. on an error path). +/// +/// # Safety +/// +/// Caller must pass a valid builder pointer and must not use it again after this call. +#[no_mangle] +pub unsafe extern "C" fn free_commit_range_builder(builder: Handle) { + builder.drop_handle(); +} + +/// Free a [`CommitRange`]. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid handle. +#[no_mangle] +pub unsafe extern "C" fn free_commit_range(commit_range: Handle) { + commit_range.drop_handle(); +} + +/// FFI-safe mirror of the kernel's [`KernelDeltaAction`]: the Delta log action kinds a caller can +/// request from [`commit_range_commits`]. +#[repr(C)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum DeltaAction { + AddAction = 0, + RemoveAction = 1, + MetadataAction = 2, + ProtocolAction = 3, + CommitInfoAction = 4, + CdcAction = 5, + DomainMetadataAction = 6, + SetTxnAction = 7, + CheckpointMetadataAction = 8, + SidecarAction = 9, +} + +impl From for KernelDeltaAction { + fn from(value: DeltaAction) -> Self { + match value { + DeltaAction::AddAction => KernelDeltaAction::Add, + DeltaAction::RemoveAction => KernelDeltaAction::Remove, + DeltaAction::MetadataAction => KernelDeltaAction::Metadata, + DeltaAction::ProtocolAction => KernelDeltaAction::Protocol, + DeltaAction::CommitInfoAction => KernelDeltaAction::CommitInfo, + DeltaAction::CdcAction => KernelDeltaAction::Cdc, + DeltaAction::DomainMetadataAction => KernelDeltaAction::DomainMetadata, + DeltaAction::SetTxnAction => KernelDeltaAction::SetTxn, + DeltaAction::CheckpointMetadataAction => KernelDeltaAction::CheckpointMetadata, + DeltaAction::SidecarAction => KernelDeltaAction::Sidecar, + } + } +} + +impl From for DeltaAction { + fn from(value: KernelDeltaAction) -> Self { + match value { + KernelDeltaAction::Add => DeltaAction::AddAction, + KernelDeltaAction::Remove => DeltaAction::RemoveAction, + KernelDeltaAction::Metadata => DeltaAction::MetadataAction, + KernelDeltaAction::Protocol => DeltaAction::ProtocolAction, + KernelDeltaAction::CommitInfo => DeltaAction::CommitInfoAction, + KernelDeltaAction::Cdc => DeltaAction::CdcAction, + KernelDeltaAction::DomainMetadata => DeltaAction::DomainMetadataAction, + KernelDeltaAction::SetTxn => DeltaAction::SetTxnAction, + KernelDeltaAction::CheckpointMetadata => DeltaAction::CheckpointMetadataAction, + KernelDeltaAction::Sidecar => DeltaAction::SidecarAction, + } + } +} + +/// Copy a C array of [`DeltaAction`] into a `Vec`. A null pointer or zero +/// length yields an empty vector. +/// +/// # Safety +/// +/// `ptr` must point to `len` valid [`DeltaAction`] values that remain valid for this call. +unsafe fn parse_actions(ptr: *const DeltaAction, len: usize) -> Vec { + if ptr.is_null() || len == 0 { + return Vec::new(); + } + let slice = unsafe { std::slice::from_raw_parts(ptr, len) }; + slice.iter().map(|action| (*action).into()).collect() +} + +/// An opaque, shared handle to a single [`CommitAction`] yielded by [`commit_range_commits_next`]. +/// The engine owns each handle passed to its visitor and must release it with +/// [`free_commit_action`]. +#[handle_descriptor(target=CommitAction, mutable=false, sized=true)] +pub struct SharedCommitAction; + +/// The commit version of this commit action. +/// +/// # Safety +/// +/// Caller must pass a valid commit action handle. +#[no_mangle] +pub unsafe extern "C" fn commit_action_version(commit_action: Handle) -> u64 { + let commit_action = unsafe { commit_action.as_ref() }; + commit_action.version() +} + +/// The commit timestamp (milliseconds since epoch) of this commit action. +/// +/// # Safety +/// +/// Caller must pass a valid commit action handle. +#[no_mangle] +pub unsafe extern "C" fn commit_action_timestamp(commit_action: Handle) -> i64 { + let commit_action = unsafe { commit_action.as_ref() }; + commit_action.timestamp() +} + +/// Get an iterator over this commit's action batches, projected to the read schema requested when +/// the iterator was created (the `actions` passed to [`commit_range_commits`]). Each batch is the +/// raw actions recorded in the commit JSON, with no column-mapping translation applied. +/// +/// The caller owns the returned iterator: drain it with [`read_result_next`] and release it with +/// [`free_read_result_iter`]. +/// +/// # Safety +/// +/// Caller is responsible for passing valid commit action and engine handles. +/// +/// [`read_result_next`]: crate::engine_funcs::read_result_next +/// [`free_read_result_iter`]: crate::engine_funcs::free_read_result_iter +#[no_mangle] +pub unsafe extern "C" fn commit_action_get_actions( + commit_action: Handle, + engine: Handle, +) -> ExternResult> { + let engine_ref = unsafe { engine.as_ref() }; + let commit_action = unsafe { commit_action.as_ref() }; + let engine_arc = unsafe { engine.clone_as_arc() }; + commit_action_get_actions_impl(commit_action, engine_arc).into_extern_result(&engine_ref) +} + +fn commit_action_get_actions_impl( + commit_action: &CommitAction, + engine: Arc, +) -> DeltaResult> { + let actions = commit_action.get_actions(engine.engine().as_ref())?; + Ok(FileReadResultIterator::into_handle(actions, engine)) +} + +/// Free a [`CommitAction`]. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid handle. +#[no_mangle] +pub unsafe extern "C" fn free_commit_action(commit_action: Handle) { + commit_action.drop_handle(); +} + +type CommitActionIter = DeltaResultIteratorStatic; + +/// Iterator handle returned by [`commit_range_commits`]. Holds the boxed kernel iterator behind a +/// mutex (so it is safe to share across threads) plus an engine reference for error allocation. +pub struct FfiCommitActionsIterator { + data: Mutex, + engine: Arc, +} + +impl FfiCommitActionsIterator { + fn lock_iter(&self) -> DeltaResult> { + self.data + .lock() + .map_err(|_| Error::generic("poisoned commit-actions iterator mutex")) + } +} + +#[handle_descriptor(target=FfiCommitActionsIterator, mutable=false, sized=true)] +pub struct SharedCommitActionsIterator; + +/// Get an iterator over the commits in `commit_range`, yielding one [`CommitAction`] per commit. +/// Use [`commit_range_commits_with_snapshot`] to seed the iterator from a start snapshot instead. +/// +/// - `engine`: performs the per-commit JSON reads and allocates errors. +/// - `actions` / `actions_len`: the action kinds to project into each commit's read schema. +/// +/// Returns an error if `actions` is empty or contains duplicate kinds. The caller owns the +/// returned iterator and must release it with [`free_commit_actions_iter`]. +/// +/// # Safety +/// +/// Caller is responsible for passing valid handles, and an `actions` pointer to `actions_len` +/// valid [`DeltaAction`] values. +#[no_mangle] +pub unsafe extern "C" fn commit_range_commits( + commit_range: Handle, + engine: Handle, + actions: *const DeltaAction, + actions_len: usize, +) -> ExternResult> { + let engine_ref = unsafe { engine.as_ref() }; + let commit_range = unsafe { commit_range.as_ref() }; + let engine_arc = unsafe { engine.clone_as_arc() }; + let actions = unsafe { parse_actions(actions, actions_len) }; + commit_range_commits_impl(commit_range, engine_arc, None, actions) + .into_extern_result(&engine_ref) +} + +/// Like [`commit_range_commits`], but seeds the iterator's protocol/metadata from `start_snapshot`. +/// +/// The snapshot's version must match the range's anchor (the start version for ascending ranges, +/// the end version for descending ranges). +/// +/// Returns an error if `actions` is empty or contains duplicate kinds, or if `start_snapshot` +/// belongs to a different table, its version does not match the range anchor, or its table does +/// not support scanning. The caller owns the returned iterator and must release it with +/// [`free_commit_actions_iter`]. +/// +/// # Safety +/// +/// Caller is responsible for passing valid handles, and an `actions` pointer to `actions_len` +/// valid [`DeltaAction`] values. +#[no_mangle] +pub unsafe extern "C" fn commit_range_commits_with_snapshot( + commit_range: Handle, + engine: Handle, + start_snapshot: Handle, + actions: *const DeltaAction, + actions_len: usize, +) -> ExternResult> { + let engine_ref = unsafe { engine.as_ref() }; + let commit_range = unsafe { commit_range.as_ref() }; + let engine_arc = unsafe { engine.clone_as_arc() }; + let start_snapshot = unsafe { start_snapshot.clone_as_arc() }; + let actions = unsafe { parse_actions(actions, actions_len) }; + commit_range_commits_impl(commit_range, engine_arc, Some(start_snapshot), actions) + .into_extern_result(&engine_ref) +} + +fn commit_range_commits_impl( + commit_range: &CommitRange, + engine: Arc, + start_snapshot: Option, + actions: Vec, +) -> DeltaResult> { + let inner = commit_range.commits(engine.engine(), start_snapshot, &actions)?; + let boxed: CommitActionIter = Box::new(inner); + let iter = FfiCommitActionsIterator { + data: Mutex::new(boxed), + engine, + }; + Ok(Arc::new(iter).into()) +} + +/// Call `engine_visitor` with the next [`CommitAction`] in the range, returning `true` if an item +/// was yielded and `false` once the iterator is exhausted. The visitor receives a +/// [`SharedCommitAction`] must free via [`free_commit_action`]. Per-commit protocol validation +/// runs here, so a malformed commit surfaces as an error from this call. +/// +/// # Safety +/// +/// The iterator must be valid (returned by [`commit_range_commits`]) and not yet freed by +/// [`free_commit_actions_iter`]. The visitor function pointer must be non-null. +#[no_mangle] +pub unsafe extern "C" fn commit_range_commits_next( + data: Handle, + engine_context: NullableCvoid, + engine_visitor: extern "C" fn( + engine_context: NullableCvoid, + commit_action: Handle, + ), +) -> ExternResult { + let data = unsafe { data.as_ref() }; + commit_range_commits_next_impl(data, engine_context, engine_visitor) + .into_extern_result(&data.engine.as_ref()) +} + +fn commit_range_commits_next_impl( + data: &FfiCommitActionsIterator, + engine_context: NullableCvoid, + engine_visitor: extern "C" fn( + engine_context: NullableCvoid, + commit_action: Handle, + ), +) -> DeltaResult { + let mut iter = data.lock_iter()?; + match iter.next().transpose()? { + Some(commit_action) => { + (engine_visitor)(engine_context, Arc::new(commit_action).into()); + Ok(true) + } + None => Ok(false), + } +} + +/// Free a commit-actions iterator. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid pointer. +#[no_mangle] +pub unsafe extern "C" fn free_commit_actions_iter(iter: Handle) { + iter.drop_handle(); +} + +#[cfg(test)] +mod tests { + use std::ptr::NonNull; + use std::sync::Arc; + + use delta_kernel::object_store::memory::InMemory; + use delta_kernel_default_engine::DefaultEngineBuilder; + use rstest::rstest; + use test_utils::{actions_to_string, add_commit, TestAction}; + + use super::*; + use crate::engine_data::engine_data_length; + use crate::engine_funcs::{free_read_result_iter, read_result_next}; + use crate::error::KernelError; + use crate::ffi_test_utils::{ + allocate_err, assert_extern_result_error_with_message, assert_timestamp_is_recent, + build_snapshot, ok_or_panic, + }; + use crate::{ + engine_to_handle, free_engine, free_engine_data, free_snapshot, kernel_string_slice, + ExclusiveEngineData, + }; + + /// Build an in-memory engine handle pre-loaded with a metadata-only commit at each version in + /// `0..=last_version`. Returns `(engine_handle, table_root)`; the caller frees the engine. + async fn setup_engine_with_commits( + last_version: u64, + ) -> (Handle, &'static str) { + let table_root = "memory:///test_table/"; + let storage = Arc::new(InMemory::new()); + for version in 0..=last_version { + add_commit( + table_root, + storage.as_ref(), + version, + actions_to_string(vec![TestAction::Metadata]), + ) + .await + .unwrap(); + } + let engine = DefaultEngineBuilder::new(storage.clone()).build(); + (engine_to_handle(Arc::new(engine), allocate_err), table_root) + } + + /// Build a `[start_version, end_version]` commit range handle via the FFI builder API. + unsafe fn build_range( + table_root: &str, + start_version: u64, + end_version: u64, + engine: Handle, + ) -> Handle { + let mut builder = unsafe { + ok_or_panic(commit_range_builder_for( + kernel_string_slice!(table_root), + start_version, + engine, + )) + }; + unsafe { commit_range_builder_set_end_version(&mut builder, end_version) }; + unsafe { ok_or_panic(commit_range_builder_build(builder)) } + } + + /// Engine visitor that reads each commit action's version, assert its timestamp is recent, + /// then frees the handle. + extern "C" fn collect_versions_and_assert_timestamp( + ctx: NullableCvoid, + commit_action: Handle, + ) { + let versions = unsafe { &mut *(ctx.unwrap().as_ptr() as *mut Vec) }; + let version = unsafe { commit_action_version(commit_action.shallow_copy()) }; + let timestamp = unsafe { commit_action_timestamp(commit_action.shallow_copy()) }; + assert_timestamp_is_recent(timestamp); + versions.push(version); + unsafe { free_commit_action(commit_action) } + } + + /// Drain a commit-actions iterator, returning the versions yielded in order. + unsafe fn drain_versions(iter: Handle) -> Vec { + let mut versions = Vec::::new(); + let ctx = Some(unsafe { NonNull::new_unchecked(&mut versions) }.cast()); + while unsafe { + ok_or_panic(commit_range_commits_next( + iter.shallow_copy(), + ctx, + collect_versions_and_assert_timestamp, + )) + } {} + versions + } + + #[rstest] + #[case::with_end_version(Some(2), 2)] + #[case::without_end_version(None, 3)] + #[tokio::test] + async fn test_commit_range_builder_build_succeeds( + #[case] builder_end_version: Option, + #[case] expected_end_version: Version, + ) -> Result<(), Box> { + let (engine, table_root) = setup_engine_with_commits(3).await; + + let mut builder = unsafe { + ok_or_panic(commit_range_builder_for( + kernel_string_slice!(table_root), + 0, + engine.shallow_copy(), + )) + }; + if let Some(end_version) = builder_end_version { + unsafe { commit_range_builder_set_end_version(&mut builder, end_version) }; + } + let range = unsafe { ok_or_panic(commit_range_builder_build(builder)) }; + + assert_eq!(unsafe { range.as_ref() }.start_version(), 0); + assert_eq!( + unsafe { range.as_ref() }.end_version(), + expected_end_version + ); + + unsafe { free_commit_range(range) } + unsafe { free_engine(engine) } + Ok(()) + } + + #[tokio::test] + async fn test_commit_range_builder_for_invalid_path_errors( + ) -> Result<(), Box> { + let (engine, _) = setup_engine_with_commits(0).await; + + let invalid_path = "not a valid url!"; + let result = unsafe { + commit_range_builder_for(kernel_string_slice!(invalid_path), 0, engine.shallow_copy()) + }; + assert_extern_result_error_with_message( + result, + KernelError::InvalidTableLocationError, + None, + ); + + unsafe { free_engine(engine) } + Ok(()) + } + + #[tokio::test] + async fn test_free_commit_range_builder() -> Result<(), Box> { + let (engine, table_root) = setup_engine_with_commits(3).await; + + let builder = unsafe { + ok_or_panic(commit_range_builder_for( + kernel_string_slice!(table_root), + 0, + engine.shallow_copy(), + )) + }; + unsafe { free_commit_range_builder(builder) } + + unsafe { free_engine(engine) } + Ok(()) + } + + #[tokio::test] + async fn test_commit_range_builder_build_errors_on_missing_start_version( + ) -> Result<(), Box> { + // Table has only v=0, but we request a range starting at v=5. + let (engine, table_root) = setup_engine_with_commits(0).await; + + let builder = unsafe { + ok_or_panic(commit_range_builder_for( + kernel_string_slice!(table_root), + 5, + engine.shallow_copy(), + )) + }; + let result = unsafe { commit_range_builder_build(builder) }; + assert_extern_result_error_with_message(result, KernelError::GenericError, None); + + unsafe { free_engine(engine) } + Ok(()) + } + + #[rstest] + #[case(DeltaAction::AddAction, KernelDeltaAction::Add)] + #[case(DeltaAction::RemoveAction, KernelDeltaAction::Remove)] + #[case(DeltaAction::MetadataAction, KernelDeltaAction::Metadata)] + #[case(DeltaAction::ProtocolAction, KernelDeltaAction::Protocol)] + #[case(DeltaAction::CommitInfoAction, KernelDeltaAction::CommitInfo)] + #[case(DeltaAction::CdcAction, KernelDeltaAction::Cdc)] + #[case(DeltaAction::DomainMetadataAction, KernelDeltaAction::DomainMetadata)] + #[case(DeltaAction::SetTxnAction, KernelDeltaAction::SetTxn)] + #[case( + DeltaAction::CheckpointMetadataAction, + KernelDeltaAction::CheckpointMetadata + )] + #[case(DeltaAction::SidecarAction, KernelDeltaAction::Sidecar)] + fn test_delta_action_conversion_round_trips( + #[case] ffi_action: DeltaAction, + #[case] kernel_action: KernelDeltaAction, + ) { + assert_eq!(KernelDeltaAction::from(ffi_action), kernel_action); + assert_eq!(DeltaAction::from(kernel_action), ffi_action); + } + + #[rstest] + #[case::no_snapshot_yields_each_commit(1, false, 0, 1, Some(vec![0, 1]))] + #[case::no_snapshot_multi_commit_yields_each_commit(3, false, 0, 3, Some(vec![0, 1, 2, 3]))] + #[case::snapshot_matched_anchor_yields_commit(1, true, 1, 1, Some(vec![1]))] + #[case::snapshot_mismatched_anchor_errors(1, true, 0, 0, None)] + #[tokio::test] + async fn test_commit_range_commits( + #[case] last_version: u64, + #[case] with_snapshot: bool, + #[case] range_start: u64, + #[case] range_end: u64, + #[case] expected: Option>, + ) -> Result<(), Box> { + let (engine, table_root) = setup_engine_with_commits(last_version).await; + let range = + unsafe { build_range(table_root, range_start, range_end, engine.shallow_copy()) }; + let actions = [DeltaAction::MetadataAction, DeltaAction::ProtocolAction]; + + let snapshot = with_snapshot.then(|| unsafe { + build_snapshot(kernel_string_slice!(table_root), engine.shallow_copy()) + }); + let result = unsafe { + match &snapshot { + Some(snapshot) => commit_range_commits_with_snapshot( + range.shallow_copy(), + engine.shallow_copy(), + snapshot.shallow_copy(), + actions.as_ptr(), + actions.len(), + ), + None => commit_range_commits( + range.shallow_copy(), + engine.shallow_copy(), + actions.as_ptr(), + actions.len(), + ), + } + }; + + match expected { + Some(expected_versions) => { + let iter = ok_or_panic(result); + let versions = unsafe { drain_versions(iter.shallow_copy()) }; + assert_eq!(versions, expected_versions); + unsafe { free_commit_actions_iter(iter) } + } + None => { + assert_extern_result_error_with_message(result, KernelError::GenericError, None); + } + } + + if let Some(snapshot) = snapshot { + unsafe { free_snapshot(snapshot) } + } + unsafe { free_commit_range(range) } + unsafe { free_engine(engine) } + Ok(()) + } + + #[tokio::test] + async fn test_commit_range_commits_errors_on_empty_actions( + ) -> Result<(), Box> { + let (engine, table_root) = setup_engine_with_commits(1).await; + let range = unsafe { build_range(table_root, 0, 1, engine.shallow_copy()) }; + + let result = unsafe { + commit_range_commits( + range.shallow_copy(), + engine.shallow_copy(), + std::ptr::null(), + 0, + ) + }; + assert_extern_result_error_with_message(result, KernelError::GenericError, None); + + unsafe { free_commit_range(range) } + unsafe { free_engine(engine) } + Ok(()) + } + + /// Context for [`drain_commit_actions`]: the engine handle used to fetch each commit's action + /// batches, plus a running total of action rows read across all commits. + struct GetActionsCtx { + engine: Handle, + total_rows: usize, + } + + /// Engine visitor that calculuates `EngineData` batch's row to an accumulator, then free the + /// batch. + extern "C" fn count_rows(ctx: NullableCvoid, data: Handle) { + let total = unsafe { &mut *(ctx.unwrap().as_ptr() as *mut usize) }; + let mut data = data; + *total += unsafe { engine_data_length(&mut data) }; + unsafe { free_engine_data(data) } + } + + /// Engine visitor that read one commit's action batches via [`commit_action_get_actions`], + /// accumulating their row count, then free the read-result iterator and the commit action. + extern "C" fn drain_commit_actions(ctx: NullableCvoid, action: Handle) { + let ctx = unsafe { &mut *(ctx.unwrap().as_ptr() as *mut GetActionsCtx) }; + let iter = unsafe { + ok_or_panic(commit_action_get_actions( + action.shallow_copy(), + ctx.engine.shallow_copy(), + )) + }; + let rows_ctx = Some(unsafe { NonNull::new_unchecked(&mut ctx.total_rows) }.cast()); + while unsafe { ok_or_panic(read_result_next(iter.shallow_copy(), rows_ctx, count_rows)) } {} + unsafe { free_read_result_iter(iter) } + unsafe { free_commit_action(action) } + } + + #[tokio::test] + async fn test_commit_action_get_actions_yields_action_batches( + ) -> Result<(), Box> { + let (engine, table_root) = setup_engine_with_commits(1).await; + let range = unsafe { build_range(table_root, 0, 1, engine.shallow_copy()) }; + + let actions = [DeltaAction::MetadataAction]; + let iter = unsafe { + ok_or_panic(commit_range_commits( + range.shallow_copy(), + engine.shallow_copy(), + actions.as_ptr(), + actions.len(), + )) + }; + + let mut ctx = GetActionsCtx { + engine: engine.shallow_copy(), + total_rows: 0, + }; + let ctx_ptr = Some(unsafe { NonNull::new_unchecked(&mut ctx) }.cast()); + while unsafe { + ok_or_panic(commit_range_commits_next( + iter.shallow_copy(), + ctx_ptr, + drain_commit_actions, + )) + } {} + + assert_eq!(ctx.total_rows, 6); + + unsafe { free_commit_actions_iter(iter) } + unsafe { free_commit_range(range) } + unsafe { free_engine(engine) } + Ok(()) + } +} diff --git a/ffi/src/uc_catalog.rs b/ffi/src/delta_kernel_unity_catalog.rs similarity index 68% rename from ffi/src/uc_catalog.rs rename to ffi/src/delta_kernel_unity_catalog.rs index 5dd3b793be..9b935d017d 100644 --- a/ffi/src/uc_catalog.rs +++ b/ffi/src/delta_kernel_unity_catalog.rs @@ -1,26 +1,26 @@ -//! FFI hooks that enable constructing a UCCommitClient. +//! FFI hooks that enable constructing a UpdateTableClient. -use crate::error::{ExternResult, IntoExternResult as _}; -use crate::{error::AllocateErrorFn, transaction::MutableCommitter}; -use crate::{ExclusiveRustString, NullableCvoid}; use std::sync::Arc; use delta_kernel::committer::Committer; -use delta_kernel::engine::default::executor::tokio::{ - TokioBackgroundExecutor, TokioMultiThreadExecutor, -}; -use delta_kernel::engine::default::DefaultEngine; use delta_kernel::DeltaResult; -use delta_kernel_ffi::{ - handle::Handle, kernel_string_slice, KernelStringSlice, OptionalValue, TryFromStringSlice, +use delta_kernel_default_engine::executor::tokio::{ + TokioBackgroundExecutor, TokioMultiThreadExecutor, }; +use delta_kernel_default_engine::DefaultEngine; +use delta_kernel_ffi::handle::Handle; +use delta_kernel_ffi::{kernel_string_slice, KernelStringSlice, OptionalValue, TryFromStringSlice}; use delta_kernel_ffi_macros::handle_descriptor; -use uc_catalog::UCCommitter; - -use uc_client::models::CommitRequest as ClientCommitRequest; -use uc_client::UCCommitClient; - +use delta_kernel_unity_catalog::UCCommitter; use tracing::debug; +use unity_catalog_delta_client_api::{ + Result as ApiResult, TableIdentifier, UpdateTableClient, + UpdateTableRequest as ClientUpdateTableRequest, +}; + +use crate::error::{AllocateErrorFn, ExternResult, IntoExternResult as _}; +use crate::transaction::MutableCommitter; +use crate::{ExclusiveRustString, NullableCvoid}; /// Data representing a commit. #[repr(C)] @@ -37,7 +37,6 @@ pub struct Commit { #[repr(C)] pub struct CommitRequest { pub table_id: KernelStringSlice, - pub table_uri: KernelStringSlice, pub commit_info: OptionalValue, pub latest_backfilled_version: OptionalValue, /// json serialized version of the metadata @@ -47,7 +46,7 @@ pub struct CommitRequest { } /// The callback that will be called when the client wants to commit. Return `None` on success, or -/// `Some("error description")` if an error occured. +/// `Some(error)` if an error occurred. // Note, it doesn't make sense to return an ExternResult here because that can't hold the string // error msg pub type CCommit = extern "C" fn( @@ -65,11 +64,27 @@ pub struct FfiUCCommitClient { unsafe impl Send for FfiUCCommitClient {} unsafe impl Sync for FfiUCCommitClient {} -impl uc_client::UCCommitClient for FfiUCCommitClient { +impl UpdateTableClient for FfiUCCommitClient { /// Commit a new version to the table. - async fn commit(&self, request: ClientCommitRequest) -> uc_client::Result<()> { - let table_id = request.table_id; - let table_uri = request.table_uri; + /// + /// `target` is unused: the FFI CommitRequest is addressed by table UUID (from the request's + /// AssertTableUuid requirement), not the three-part identifier. + async fn update_table( + &self, + _target: &TableIdentifier, + request: ClientUpdateTableRequest, + ) -> ApiResult<()> { + let table_id = request + .table_uuid() + .ok_or_else(|| { + unity_catalog_delta_client_api::Error::UnsupportedOperation( + "FFI UC commit client requires an assert-table-uuid requirement".to_string(), + ) + })? + .to_string(); + + let latest_backfilled_version = request.latest_backfilled_version(); + let add_commit = request.staged_commit().cloned(); // there is a subtle issue here where we need to ensure that the string we use to refer to // the commit_info.file_name stays in scope until _after_ the callback returns, so that the @@ -77,13 +92,11 @@ impl uc_client::UCCommitClient for FfiUCCommitClient { // request.commit_info.map to build an Option. Rather we just use a closure to hold // the common code and call it from a scope where the string remains valid until after the // closure finishes - - let send_request = |commit_info| -> uc_client::Result<()> { + let send_request = |commit_info| -> ApiResult<()> { let c_commit_request = CommitRequest { table_id: kernel_string_slice!(table_id), - table_uri: kernel_string_slice!(table_uri), commit_info, - latest_backfilled_version: request.latest_backfilled_version.into(), + latest_backfilled_version: latest_backfilled_version.into(), metadata: None.into(), protocol: None.into(), }; @@ -92,13 +105,13 @@ impl uc_client::UCCommitClient for FfiUCCommitClient { OptionalValue::Some(e) => { let boxed_str = unsafe { e.into_inner() }; // get the string back into Box let s: String = *boxed_str; // move back onto the stack - uc_client::Result::Err(uc_client::Error::Generic(s)) + Err(unity_catalog_delta_client_api::Error::Generic(s)) } - OptionalValue::None => uc_client::Result::Ok(()), + OptionalValue::None => Ok(()), } }; - if let Some(client_commit_info) = request.commit_info { + if let Some(client_commit_info) = add_commit { let file_name = client_commit_info.file_name; let commit_info = Some(Commit { version: client_commit_info.version, @@ -150,11 +163,11 @@ pub unsafe extern "C" fn free_uc_commit_client(commit_client: Handle { +struct FfiUCCommitter { inner: UCCommitter, } -impl Committer for FfiUCCommitter { +impl Committer for FfiUCCommitter { fn commit( &self, engine: &dyn delta_kernel::Engine, @@ -201,32 +214,48 @@ impl Committer for FfiUCCommitter { /// # Safety /// /// Caller is responsible for passing a valid pointer to a SharedFfiUCCommitClient, obtained via -/// calling [`get_uc_commit_client`], a valid KernelStringSlice as the table_id, and a valid error -/// function pointer. +/// calling [`get_uc_commit_client`], valid KernelStringSlices for `table_id`, `catalog`, +/// `schema`, and `table_name`, and a valid error function pointer. #[no_mangle] pub unsafe extern "C" fn get_uc_committer( commit_client: Handle, table_id: KernelStringSlice, + catalog: KernelStringSlice, + schema: KernelStringSlice, + table_name: KernelStringSlice, error_fn: AllocateErrorFn, ) -> ExternResult> { - get_uc_committer_impl(commit_client, table_id).into_extern_result(&error_fn) + get_uc_committer_impl(commit_client, table_id, catalog, schema, table_name) + .into_extern_result(&error_fn) } fn get_uc_committer_impl( commit_client: Handle, table_id: KernelStringSlice, + catalog: KernelStringSlice, + schema: KernelStringSlice, + table_name: KernelStringSlice, ) -> DeltaResult> { let client: Arc = unsafe { commit_client.clone_as_arc() }; let table_id_str: String = unsafe { TryFromStringSlice::try_from_slice(&table_id) }?; + let catalog_str: String = unsafe { TryFromStringSlice::try_from_slice(&catalog) }?; + let schema_str: String = unsafe { TryFromStringSlice::try_from_slice(&schema) }?; + let table_name_str: String = unsafe { TryFromStringSlice::try_from_slice(&table_name) }?; let committer: Box = Box::new(FfiUCCommitter { - inner: UCCommitter::new(client, table_id_str), + inner: UCCommitter::new( + client, + table_id_str, + TableIdentifier::new(catalog_str, schema_str, table_name_str), + ), }); Ok(committer.into()) } -/// Free a committer obtained via get_uc_committer. Warning! Normally the value returned here will -/// be consumed when creating a transaction via [`crate::transaction::transaction_with_committer`] -/// and will NOT need to be freed. +/// Free a committer obtained via [`get_uc_committer`] that was not passed to a consuming API. +/// +/// APIs that take ownership of the committer; `transaction_with_committer`, +/// `create_table_builder_build_with_committer`, and `snapshot_publish_with_committer` +/// consume the handle (do not free; it will not need to be freed afterward). /// /// # Safety /// @@ -239,18 +268,22 @@ pub unsafe extern "C" fn free_uc_committer(commit_client: Handle, + pub(crate) last_commit_table_id: Option, pub(crate) last_staged_filename: Option, pub(crate) should_fail_commit: bool, } @@ -258,7 +291,7 @@ pub(crate) mod tests { pub(crate) fn get_test_context(should_fail_commit: bool) -> NullableCvoid { let context = Box::new(TestContext { commit_called: false, - last_commit_request: None, + last_commit_table_id: None, last_staged_filename: None, should_fail_commit, }); @@ -286,7 +319,6 @@ pub(crate) mod tests { context.commit_called = true; let table_id = unsafe { String::try_from_slice(&request.table_id).unwrap() }; - let table_uri = unsafe { String::try_from_slice(&request.table_uri).unwrap() }; if let OptionalValue::Some(commit_info) = request.commit_info { let file_name = unsafe { @@ -294,7 +326,7 @@ pub(crate) mod tests { }; context.last_staged_filename = Some(file_name); } - context.last_commit_request = Some((table_id.clone(), table_uri.clone())); + context.last_commit_table_id = Some(table_id.clone()); if context.should_fail_commit { let error_msg = "Test commit failure"; let error_str = unsafe { @@ -325,31 +357,32 @@ pub(crate) mod tests { let client_arc: Arc = unsafe { client.clone_as_arc() }; - let request = ClientCommitRequest { - table_id: "test_table_id".to_string(), - table_uri: "s3://bucket/path".to_string(), - commit_info: Some(ClientCommit { - version: 10, - timestamp: 2000000000, - file_name: "_staged_commits/00000000000000000010.uuid.json".to_string(), - file_size: 1024, - file_modification_timestamp: 2000000100, - }), - latest_backfilled_version: None, - metadata: None, - protocol: None, - }; - - let result: uc_client::Result<()> = client_arc.commit(request).await; + let request = ClientUpdateTableRequest::new( + vec![ApiRequirement::AssertTableUuid { + uuid: "test_table_id".to_string(), + }], + vec![ApiUpdate::AddCommit { + commit: ClientCommit { + version: 10, + timestamp: 2000000000, + file_name: "_staged_commits/00000000000000000010.uuid.json".to_string(), + file_size: 1024, + file_modification_timestamp: 2000000100, + }, + }], + ) + .unwrap(); + + let target = TableIdentifier::new("test_catalog", "test_schema", "test_table"); + let result: ApiResult<()> = client_arc.update_table(&target, request).await; assert!(result.is_ok()); let context = recover_test_context(context).unwrap(); assert!(context.commit_called); - let (table_id, table_uri) = context.last_commit_request.unwrap(); + let table_id = context.last_commit_table_id.unwrap(); assert_eq!(table_id, "test_table_id"); - assert_eq!(table_uri, "s3://bucket/path"); assert_eq!( context.last_staged_filename.unwrap(), "_staged_commits/00000000000000000010.uuid.json" @@ -366,22 +399,24 @@ pub(crate) mod tests { let client_arc: Arc = unsafe { client.clone_as_arc() }; - let request = ClientCommitRequest { - table_id: "test_table_id".to_string(), - table_uri: "s3://bucket/path".to_string(), - commit_info: Some(ClientCommit { - version: 10, - timestamp: 2000000000, - file_name: "00000000000000000010.uuid.json".to_string(), - file_size: 1024, - file_modification_timestamp: 2000000100, - }), - latest_backfilled_version: None, - metadata: None, - protocol: None, - }; - - let result: uc_client::Result<()> = client_arc.commit(request).await; + let request = ClientUpdateTableRequest::new( + vec![ApiRequirement::AssertTableUuid { + uuid: "test_table_id".to_string(), + }], + vec![ApiUpdate::AddCommit { + commit: ClientCommit { + version: 10, + timestamp: 2000000000, + file_name: "00000000000000000010.uuid.json".to_string(), + file_size: 1024, + file_modification_timestamp: 2000000100, + }, + }], + ) + .unwrap(); + + let target = TableIdentifier::new("test_catalog", "test_schema", "test_table"); + let result: ApiResult<()> = client_arc.update_table(&target, request).await; assert!(result.is_err()); @@ -390,8 +425,8 @@ pub(crate) mod tests { assert!(context.commit_called); let error = result.unwrap_err(); - assert!(matches!(error, uc_client::Error::Generic(_))); - if let uc_client::Error::Generic(msg) = error { + assert!(matches!(error, ApiError::Generic(_))); + if let unity_catalog_delta_client_api::Error::Generic(msg) = error { assert_eq!(msg, "Test commit failure"); } @@ -403,10 +438,16 @@ pub(crate) mod tests { let client = unsafe { get_uc_commit_client(None, test_commit_callback) }; let table_id = "test_table_id"; + let catalog = "test_catalog"; + let schema = "test_schema"; + let table_name = "test_table"; let committer = unsafe { ok_or_panic(get_uc_committer( client.shallow_copy(), kernel_string_slice!(table_id), + kernel_string_slice!(catalog), + kernel_string_slice!(schema), + kernel_string_slice!(table_name), allocate_err, )) }; diff --git a/ffi/src/domain_metadata.rs b/ffi/src/domain_metadata.rs index 29555a7524..8f7e0a774f 100644 --- a/ffi/src/domain_metadata.rs +++ b/ffi/src/domain_metadata.rs @@ -1,13 +1,16 @@ +use delta_kernel::snapshot::Snapshot; +use delta_kernel::DeltaResult; + use crate::error::{ExternResult, IntoExternResult}; +use crate::expressions::kernel_visitor::NullTypeTag; use crate::handle::Handle; use crate::{ kernel_string_slice, AllocateStringFn, ExternEngine, KernelStringSlice, NullableCvoid, - SharedExternEngine, SharedSnapshot, TryFromStringSlice, + OptionalValue, SharedExternEngine, SharedSnapshot, TryFromStringSlice, }; -use delta_kernel::snapshot::Snapshot; -use delta_kernel::DeltaResult; -/// Get the domain metadata as an optional string allocated by `AllocatedStringFn` for a specific domain in this snapshot +/// Get the domain metadata as an optional string allocated by `AllocatedStringFn` for a specific +/// domain in this snapshot /// /// # Safety /// @@ -34,10 +37,93 @@ fn get_domain_metadata_impl( ) -> DeltaResult { Ok(snapshot .get_domain_metadata(&domain?, extern_engine.engine().as_ref())? - .and_then(|config: String| allocate_fn(kernel_string_slice!(config)))) + .and_then(|config| allocate_fn(kernel_string_slice!(config)))) } -/// Get the domain metadata as an optional string allocated by `AllocatedStringFn` for a specific domain in this snapshot +/// Signature of the callback invoked once per clustering column by +/// [`visit_clustering_columns`], in the order the columns appear in the `delta.clustering` +/// domain. Each invocation describes one column: +/// - `logical_column`: the column path resolved against the table schema. When column mapping is +/// enabled this is the logical name, not the physical identifier stored in the domain. +/// - `physical_column`: the column path as stored in the domain. Use this to correlate a +/// clustering column with per-file statistics, which are keyed on physical names. +/// - `type_tag`: the column's data type, encoded with the same tag values as +/// `visit_expression_literal_null`. Complex types (struct, array, map, variant) and void report +/// the non-primitive sentinel (255); use a schema visitor for their full details. +/// - `precision` / `scale`: meaningful only when `type_tag` is the `decimal` tag (12); zero +/// otherwise. +/// +/// Both column paths are dotted strings with individual fields backtick-escaped when they +/// contain special characters, losslessly parseable back into a multi-part column reference. +/// Note that column-mapping physical identifiers (e.g. `col-`) contain hyphens and so +/// arrive backtick-quoted; parse the path rather than comparing the raw string. +/// +/// Resolving a domain-listed column against the schema matches names case-sensitively, so a +/// domain entry whose casing differs from the schema's is reported as an error rather than +/// resolved. Java kernel matches case-insensitively here. +pub type ClusteringColumnVisitor = extern "C" fn( + engine_context: NullableCvoid, + logical_column: KernelStringSlice, + physical_column: KernelStringSlice, + type_tag: u8, + precision: u8, + scale: u8, +); + +/// Visit the table's clustering columns, invoking `visitor` once per column. See +/// [`ClusteringColumnVisitor`] for what each invocation reports. +/// +/// Returns the number of columns visited when the table has the `clustering` feature with a +/// current `delta.clustering` domain entry -- `Some(0)` means the table is clustered but the +/// entry lists no columns. Returns `None` when the feature is absent or the domain has no +/// current entry; the visitor does not run in that case. This distinction mirrors kernel's +/// `Option>`: `None` is "not clustered", `Some(0)` is "clustered on nothing". +/// +/// # Safety +/// +/// Caller is responsible for passing in a valid snapshot and engine handle, a valid +/// `engine_context` opaque pointer forwarded to each `visitor` invocation, and a valid +/// `visitor` function pointer. +#[no_mangle] +pub unsafe extern "C" fn visit_clustering_columns( + snapshot: Handle, + engine: Handle, + engine_context: NullableCvoid, + visitor: ClusteringColumnVisitor, +) -> ExternResult> { + let snapshot = unsafe { snapshot.as_ref() }; + let engine = unsafe { engine.as_ref() }; + visit_clustering_columns_impl(snapshot, engine, engine_context, visitor) + .into_extern_result(&engine) +} + +fn visit_clustering_columns_impl( + snapshot: &Snapshot, + extern_engine: &dyn ExternEngine, + engine_context: NullableCvoid, + visitor: ClusteringColumnVisitor, +) -> DeltaResult> { + let Some(infos) = snapshot.get_clustering_column_infos(extern_engine.engine().as_ref())? else { + return Ok(OptionalValue::None); + }; + for info in &infos { + let logical = info.logical_column.to_string(); + let physical = info.physical_column.to_string(); + let (tag, precision, scale) = NullTypeTag::from_data_type(&info.data_type); + visitor( + engine_context, + kernel_string_slice!(logical), + kernel_string_slice!(physical), + tag as u8, + precision, + scale, + ); + } + Ok(OptionalValue::Some(infos.len())) +} + +/// Get the domain metadata as an optional string allocated by `AllocatedStringFn` for a specific +/// domain in this snapshot /// /// # Safety /// @@ -86,22 +172,25 @@ fn visit_domain_metadata_impl( #[cfg(test)] mod tests { - use super::*; - use crate::error::KernelError; - use crate::ffi_test_utils::{ - allocate_err, allocate_str, assert_extern_result_error_with_message, ok_or_panic, - recover_string, - }; - use crate::{engine_to_handle, free_engine, free_snapshot, kernel_string_slice, snapshot}; - use delta_kernel::engine::default::DefaultEngineBuilder; - use delta_kernel::object_store::memory::InMemory; - use delta_kernel::DeltaResult; - use serde_json::json; use std::collections::HashMap; use std::ptr::NonNull; use std::sync::Arc; + + use delta_kernel::object_store::memory::InMemory; + use delta_kernel::DeltaResult; + use delta_kernel_default_engine::DefaultEngineBuilder; + use rstest::rstest; + use serde_json::json; use test_utils::add_commit; + use super::*; + use crate::error::KernelError; + use crate::ffi_test_utils::{ + allocate_err, allocate_str, assert_extern_result_error_with_message, build_snapshot, + ok_or_panic, recover_string, + }; + use crate::{engine_to_handle, free_engine, free_snapshot, kernel_string_slice}; + #[tokio::test] async fn test_domain_metadata() -> DeltaResult<()> { let storage = Arc::new(InMemory::new()); @@ -113,7 +202,7 @@ mod tests { // commit0 // - domain1: not removed // - domain2: not removed - let commit = [ + let commit = join_actions(&[ json!({ "protocol": { "minReaderVersion": 1, @@ -144,11 +233,9 @@ mod tests { "removed": false } }), - ] - .map(|json| json.to_string()) - .join("\n"); + ]); - add_commit(table_root, storage.clone().as_ref(), 0, commit) + add_commit(table_root, storage.as_ref(), 0, commit) .await .unwrap(); @@ -156,7 +243,7 @@ mod tests { // - domain1: removed // - domain2: not-removed // - internal domain - let commit = [ + let commit = join_actions(&[ json!({ "domainMetadata": { "domain": "domain1", @@ -178,20 +265,14 @@ mod tests { "removed": false } }), - ] - .map(|json| json.to_string()) - .join("\n"); + ]); add_commit(table_root, storage.as_ref(), 1, commit) .await .unwrap(); - let snapshot = unsafe { - ok_or_panic(snapshot( - kernel_string_slice!(table_root), - engine.shallow_copy(), - )) - }; + let snapshot = + unsafe { build_snapshot(kernel_string_slice!(table_root), engine.shallow_copy()) }; let get_domain_metadata_helper = |domain: &str| unsafe { get_domain_metadata( @@ -214,7 +295,7 @@ mod tests { let domain3 = "delta.domain3"; let res = get_domain_metadata_helper(domain3); - assert_extern_result_error_with_message(res, KernelError::GenericError, "Generic delta kernel error: User DomainMetadata are not allowed to use system-controlled 'delta.*' domain"); + assert_extern_result_error_with_message(res, KernelError::GenericError, Some("Generic delta kernel error: User DomainMetadata are not allowed to use system-controlled 'delta.*' domain")); // Secondly, we visit the entire domain metadata @@ -264,4 +345,475 @@ mod tests { Ok(()) } + + // === visit_clustering_columns tests === + + fn protocol_action(clustering: bool) -> serde_json::Value { + if clustering { + json!({ + "protocol": { + "minReaderVersion": 1, + "minWriterVersion": 7, + "writerFeatures": ["domainMetadata", "clustering"] + } + }) + } else { + json!({ + "protocol": { + "minReaderVersion": 1, + "minWriterVersion": 1 + } + }) + } + } + + fn metadata_action(schema_json: &str) -> serde_json::Value { + json!({ + "metaData": { + "id": "5fba94ed-9794-4965-ba6e-6ee3c0d22af9", + "format": { "provider": "parquet", "options": {} }, + "schemaString": schema_json, + "partitionColumns": [], + "configuration": {}, + "createdTime": 1587968585495i64 + } + }) + } + + // Schema with two top-level columns (`id: integer`, `val: string`) plus a nested + // `addr: struct` so clustering on `["addr", "city"]` resolves. + fn clustering_test_schema_json() -> String { + let addr = json!({ + "type": "struct", + "fields": [ + {"name": "city", "type": "string", "nullable": true, "metadata": {}} + ] + }); + json!({ + "type": "struct", + "fields": [ + {"name": "id", "type": "integer", "nullable": true, "metadata": {}}, + {"name": "val", "type": "string", "nullable": true, "metadata": {}}, + {"name": "addr", "type": addr, "nullable": true, "metadata": {}} + ] + }) + .to_string() + } + + fn clustering_domain_action(config: &str, removed: bool) -> serde_json::Value { + json!({ + "domainMetadata": { + "domain": "delta.clustering", + "configuration": config, + "removed": removed + } + }) + } + + fn join_actions(actions: &[serde_json::Value]) -> String { + actions + .iter() + .map(|j| j.to_string()) + .collect::>() + .join("\n") + } + + /// One visited clustering column: `(logical, physical, type_tag, precision, scale)`. + type VisitedColumn = (String, String, u8, u8, u8); + + /// Drive `visit_clustering_columns`, collecting every visited column. Returns the count the + /// FFI symbol reported alongside the visited descriptors. + unsafe fn collect_clustering_columns( + snapshot: &crate::Handle, + engine: &crate::Handle, + ) -> (Option, Vec) { + let collected: Box> = Box::default(); + let collected_ptr = Box::into_raw(collected); + + extern "C" fn visitor( + state: NullableCvoid, + logical_column: KernelStringSlice, + physical_column: KernelStringSlice, + type_tag: u8, + precision: u8, + scale: u8, + ) { + let mut columns = + unsafe { Box::from_raw(state.unwrap().as_ptr() as *mut Vec) }; + let logical: DeltaResult = + unsafe { TryFromStringSlice::try_from_slice(&logical_column) }; + let physical: DeltaResult = + unsafe { TryFromStringSlice::try_from_slice(&physical_column) }; + columns.push(( + logical.unwrap(), + physical.unwrap(), + type_tag, + precision, + scale, + )); + Box::leak(columns); + } + + let count: Option = ok_or_panic(unsafe { + visit_clustering_columns( + snapshot.shallow_copy(), + engine.shallow_copy(), + Some(NonNull::new_unchecked(collected_ptr).cast()), + visitor, + ) + }) + .into(); + + let columns = *unsafe { Box::from_raw(collected_ptr) }; + (count, columns) + } + + async fn build_clustering_snapshot( + table_root: &str, + clustering_feature: bool, + domain_config: Option<&str>, + follow_up_commits: &[serde_json::Value], + ) -> ( + crate::Handle, + crate::Handle, + ) { + let storage = Arc::new(InMemory::new()); + let engine = DefaultEngineBuilder::new(storage.clone()).build(); + let engine = engine_to_handle(Arc::new(engine), allocate_err); + + let mut actions = vec![ + protocol_action(clustering_feature), + metadata_action(&clustering_test_schema_json()), + ]; + if let Some(config) = domain_config { + actions.push(clustering_domain_action(config, false)); + } + add_commit(table_root, storage.as_ref(), 0, join_actions(&actions)) + .await + .unwrap(); + + for (version, commit_actions) in follow_up_commits.iter().enumerate() { + add_commit( + table_root, + storage.as_ref(), + version as u64 + 1, + commit_actions.to_string(), + ) + .await + .unwrap(); + } + + let snapshot = + unsafe { build_snapshot(kernel_string_slice!(table_root), engine.shallow_copy()) }; + (engine, snapshot) + } + + #[tokio::test] + async fn test_get_clustering_domain_metadata_user_facing_path_still_rejects() -> DeltaResult<()> + { + let table_root = "memory:///test_clustering_user_facing_rejected/"; + let (engine, snapshot) = build_clustering_snapshot( + table_root, + true, + Some(r#"{"clusteringColumns":[["id"]]}"#), + &[], + ) + .await; + + let domain = "delta.clustering"; + let rejected = unsafe { + get_domain_metadata( + snapshot.shallow_copy(), + kernel_string_slice!(domain), + engine.shallow_copy(), + allocate_str, + ) + }; + assert_extern_result_error_with_message( + rejected, + KernelError::GenericError, + Some( + "Generic delta kernel error: User DomainMetadata are not allowed to use \ + system-controlled 'delta.*' domain", + ), + ); + + unsafe { free_snapshot(snapshot) } + unsafe { free_engine(engine) } + + Ok(()) + } + + // Type tags mirror `visit_expression_literal_null`'s encoding. + const TAG_INTEGER: u8 = 3; + const TAG_STRING: u8 = 7; + + // `follow_up` is an optional second `delta.clustering` write `(config, removed)`, letting + // one case exercise tombstone and latest-write-wins reconciliation. Expectations are + // `(logical, physical, type_tag)`; without column mapping logical == physical. + #[rstest] + #[case::clustered_single_column( + true, + Some(r#"{"clusteringColumns":[["id"]]}"#), + None, + Some(vec![("id", "id", TAG_INTEGER)]), + "memory:///test_clustering_single/" + )] + #[case::clustered_multi_column( + true, + Some(r#"{"clusteringColumns":[["id"],["val"]]}"#), + None, + Some(vec![("id", "id", TAG_INTEGER), ("val", "val", TAG_STRING)]), + "memory:///test_clustering_multi/" + )] + // Nested paths render as dotted, backtick-escaped `ColumnName` strings. + #[case::clustered_nested_path( + true, + Some(r#"{"clusteringColumns":[["addr","city"]]}"#), + None, + Some(vec![("addr.city", "addr.city", TAG_STRING)]), + "memory:///test_clustering_nested/" + )] + // Domain entry present but lists no columns: clustered on nothing -- Some(0), no visits. + #[case::clustered_empty_column_list( + true, + Some(r#"{"clusteringColumns":[]}"#), + None, + Some(vec![]), + "memory:///test_clustering_empty_list/" + )] + // Not clustered: None, and the visitor never runs. + #[case::unclustered_returns_none( + false, + None, + None, + None, + "memory:///test_clustering_unclustered/" + )] + #[case::clustered_feature_with_no_domain_entry( + true, + None, + None, + None, + "memory:///test_clustering_feature_no_entry/" + )] + #[case::no_clustering_feature_but_domain_present( + false, + Some(r#"{"clusteringColumns":[["id"]]}"#), + None, + None, + "memory:///test_clustering_no_feature_domain_present/" + )] + // A tombstone (`removed: true`) clears the domain: None, and the visitor never runs. + #[case::tombstoned_returns_none( + true, + Some(r#"{"clusteringColumns":[["id"]]}"#), + Some((r#"{"clusteringColumns":[["id"]]}"#, true)), + None, + "memory:///test_clustering_tombstoned/" + )] + // A later non-removed write wins over the earlier one. + #[case::latest_write_wins( + true, + Some(r#"{"clusteringColumns":[["id"]]}"#), + Some((r#"{"clusteringColumns":[["val"]]}"#, false)), + Some(vec![("val", "val", TAG_STRING)]), + "memory:///test_clustering_latest_wins/" + )] + #[tokio::test] + async fn test_visit_clustering_columns( + #[case] clustering_feature: bool, + #[case] domain_config: Option<&str>, + #[case] follow_up: Option<(&str, bool)>, + #[case] expected: Option>, + #[case] table_root: &str, + ) -> DeltaResult<()> { + let follow_up_commits: Vec = follow_up + .into_iter() + .map(|(config, removed)| clustering_domain_action(config, removed)) + .collect(); + let (engine, snapshot) = build_clustering_snapshot( + table_root, + clustering_feature, + domain_config, + &follow_up_commits, + ) + .await; + + let (count, columns) = unsafe { collect_clustering_columns(&snapshot, &engine) }; + + match expected { + Some(want) => { + // Non-decimal types report zero precision/scale. + let want: Vec = want + .into_iter() + .map(|(logical, physical, tag)| { + (logical.to_string(), physical.to_string(), tag, 0, 0) + }) + .collect(); + assert_eq!(count, Some(want.len())); + assert_eq!(columns, want); + } + None => { + assert_eq!(count, None); + assert!(columns.is_empty()); + } + } + + unsafe { free_snapshot(snapshot) } + unsafe { free_engine(engine) } + + Ok(()) + } + + // Decimal is the only tag carrying a payload, so pin precision/scale explicitly. + #[tokio::test] + async fn test_visit_clustering_columns_decimal_reports_precision_and_scale() -> DeltaResult<()> + { + const TAG_DECIMAL: u8 = 12; + let storage = Arc::new(InMemory::new()); + let engine = DefaultEngineBuilder::new(storage.clone()).build(); + let engine = engine_to_handle(Arc::new(engine), allocate_err); + let table_root = "memory:///test_clustering_decimal/"; + + let schema = json!({ + "type": "struct", + "fields": [ + {"name": "amt", "type": "decimal(12,3)", "nullable": true, "metadata": {}} + ] + }) + .to_string(); + let commit = join_actions(&[ + protocol_action(true), + metadata_action(&schema), + clustering_domain_action(r#"{"clusteringColumns":[["amt"]]}"#, false), + ]); + add_commit(table_root, storage.as_ref(), 0, commit) + .await + .unwrap(); + + let snapshot = + unsafe { build_snapshot(kernel_string_slice!(table_root), engine.shallow_copy()) }; + + let (count, columns) = unsafe { collect_clustering_columns(&snapshot, &engine) }; + assert_eq!(count, Some(1)); + assert_eq!( + columns, + vec![("amt".to_string(), "amt".to_string(), TAG_DECIMAL, 12, 3)] + ); + + unsafe { free_snapshot(snapshot) } + unsafe { free_engine(engine) } + + Ok(()) + } + + // With column mapping the domain stores physical identifiers, so the visitor must report the + // schema-resolved logical name alongside the physical one the domain (and per-file stats) use. + #[tokio::test] + async fn test_visit_clustering_columns_column_mapping_reports_both_names() -> DeltaResult<()> { + let storage = Arc::new(InMemory::new()); + let engine = DefaultEngineBuilder::new(storage.clone()).build(); + let engine = engine_to_handle(Arc::new(engine), allocate_err); + let table_root = "memory:///test_clustering_column_mapping/"; + + // `region` is stored physically as `col-region-phys`; clustering names the physical id. + let schema = json!({ + "type": "struct", + "fields": [{ + "name": "region", + "type": "string", + "nullable": true, + "metadata": { + "delta.columnMapping.id": 1, + "delta.columnMapping.physicalName": "col-region-phys" + } + }] + }) + .to_string(); + let commit = join_actions(&[ + json!({ + "protocol": { + "minReaderVersion": 2, + "minWriterVersion": 7, + "writerFeatures": ["domainMetadata", "clustering", "columnMapping"] + } + }), + json!({ + "metaData": { + "id": "5fba94ed-9794-4965-ba6e-6ee3c0d22af9", + "format": { "provider": "parquet", "options": {} }, + "schemaString": schema, + "partitionColumns": [], + "configuration": { "delta.columnMapping.mode": "name" }, + "createdTime": 1587968585495i64 + } + }), + clustering_domain_action(r#"{"clusteringColumns":[["col-region-phys"]]}"#, false), + ]); + add_commit(table_root, storage.as_ref(), 0, commit) + .await + .unwrap(); + + let snapshot = + unsafe { build_snapshot(kernel_string_slice!(table_root), engine.shallow_copy()) }; + + let (count, columns) = unsafe { collect_clustering_columns(&snapshot, &engine) }; + assert_eq!(count, Some(1)); + // Physical identifiers contain hyphens, which `ColumnName` renders backtick-quoted. + assert_eq!( + columns, + vec![( + "region".to_string(), + "`col-region-phys`".to_string(), + TAG_STRING, + 0, + 0 + )] + ); + + unsafe { free_snapshot(snapshot) } + unsafe { free_engine(engine) } + + Ok(()) + } + + // A clustering column absent from the schema cannot resolve to a logical name, so the + // symbol surfaces an `ExternResult::Err` rather than reporting the table as unclustered. + #[tokio::test] + async fn test_visit_clustering_columns_column_not_in_schema_errors() -> DeltaResult<()> { + let table_root = "memory:///test_clustering_col_missing/"; + // "ghost" is not one of id/val/addr in clustering_test_schema_json(). + let (engine, snapshot) = build_clustering_snapshot( + table_root, + true, + Some(r#"{"clusteringColumns":[["ghost"]]}"#), + &[], + ) + .await; + + extern "C" fn visitor( + _: NullableCvoid, + _: KernelStringSlice, + _: KernelStringSlice, + _: u8, + _: u8, + _: u8, + ) { + } + let res = unsafe { + visit_clustering_columns( + snapshot.shallow_copy(), + engine.shallow_copy(), + None, + visitor, + ) + }; + assert_extern_result_error_with_message(res, KernelError::GenericError, None); + + unsafe { free_snapshot(snapshot) } + unsafe { free_engine(engine) } + + Ok(()) + } } diff --git a/ffi/src/engine_data.rs b/ffi/src/engine_data.rs index 621ea7197b..96479dc1ca 100644 --- a/ffi/src/engine_data.rs +++ b/ffi/src/engine_data.rs @@ -1,4 +1,6 @@ //! EngineData related ffi code +use std::ffi::c_void; + #[cfg(feature = "default-engine-base")] use delta_kernel::arrow; #[cfg(feature = "default-engine-base")] @@ -11,16 +13,14 @@ use delta_kernel::engine::arrow_data::{ArrowEngineData, EngineDataArrowExt as _} #[cfg(feature = "default-engine-base")] use delta_kernel::DeltaResult; use delta_kernel::EngineData; -use std::ffi::c_void; +use super::handle::Handle; #[cfg(feature = "default-engine-base")] use crate::error::AllocateErrorFn; use crate::ExclusiveEngineData; #[cfg(feature = "default-engine-base")] use crate::{ExternResult, IntoExternResult, SharedExternEngine}; -use super::handle::Handle; - /// Get the number of rows in an engine data /// /// # Safety @@ -31,13 +31,13 @@ pub unsafe extern "C" fn engine_data_length(data: &mut Handle) -> DeltaResult { + let record_batch = data.try_into_record_batch()?; + Self::try_from_record_batch(record_batch) + } + + /// Export a `RecordBatch` as Arrow C Data Interface structs. + pub fn try_from_record_batch(batch: RecordBatch) -> DeltaResult { + let sa: StructArray = batch.into(); + let array_data: ArrayData = sa.into(); + let array = FFI_ArrowArray::new(&array_data); + let schema = FFI_ArrowSchema::try_from(array_data.data_type())?; + Ok(Self { array, schema }) + } } // TODO: This should use a callback to avoid having to have the engine free the struct /// Get an [`ArrowFFIData`] to allow binding to the arrow [C Data /// Interface](https://arrow.apache.org/docs/format/CDataInterface.html). This includes the data and -/// the schema. If this function returns an `Ok` variant the _engine_ must free the returned struct. +/// the schema. If this function returns an `Ok` variant the _engine_ must free the returned struct +/// via [`free_arrow_ffi_data`] exactly once. /// /// # Safety /// data_handle must be a valid ExclusiveEngineData as read by the -/// [`delta_kernel::engine::default::DefaultEngine`] obtained from `get_default_engine`. +/// [`delta_kernel_default_engine::DefaultEngine`] obtained from `get_default_engine`. #[cfg(feature = "default-engine-base")] #[no_mangle] pub unsafe extern "C" fn get_raw_arrow_data( @@ -92,17 +112,36 @@ pub unsafe extern "C" fn get_raw_arrow_data( get_raw_arrow_data_impl(data).into_extern_result(&engine.as_ref()) } -// TODO: This method leaks the returned pointer memory. How will the engine free it? #[cfg(feature = "default-engine-base")] fn get_raw_arrow_data_impl(data: Box) -> DeltaResult<*mut ArrowFFIData> { - let record_batch = data.try_into_record_batch()?; - let sa: StructArray = record_batch.into(); - let array_data: ArrayData = sa.into(); - // these call `clone`. is there a way to not copy anything and what exactly are they cloning? - let array = FFI_ArrowArray::new(&array_data); - let schema = FFI_ArrowSchema::try_from(array_data.data_type())?; - let ret_data = Box::new(ArrowFFIData { array, schema }); - Ok(Box::leak(ret_data)) + Ok(Box::into_raw(Box::new(ArrowFFIData::try_from_engine_data( + data, + )?))) +} + +/// Free an [`ArrowFFIData`] pointer produced by a kernel FFI function (e.g. +/// [`get_raw_arrow_data`] or [`crate::table_changes::scan_table_changes_next`]). +/// +/// If the consumer has already imported the inner `FFI_ArrowArray` / `FFI_ArrowSchema` via a +/// foreign Arrow layer (e.g. arrow-glib's `garrow_record_batch_import`), that import has +/// moved ownership of the release callbacks out of the structs; dropping the `Box` here is +/// then a cheap no-op on the arrays. If the consumer has not imported them, the structs' +/// `Drop` impls will call their release callbacks so no memory is leaked. +/// +/// A null pointer is a no-op, matching the convention used by +/// [`crate::scan::free_scan_metadata_arrow_result`]. +/// +/// # Safety +/// +/// `result` must be either null, or a pointer returned by a kernel FFI function that produces +/// `*mut ArrowFFIData`. Must be called at most once per non-null pointer. +#[cfg(feature = "default-engine-base")] +#[no_mangle] +pub unsafe extern "C" fn free_arrow_ffi_data(result: *mut ArrowFFIData) { + if result.is_null() { + return; + } + let _ = unsafe { Box::from_raw(result) }; } /// Creates engine data from Arrow C Data Interface array and schema. @@ -136,3 +175,66 @@ unsafe fn get_engine_data_impl( let engine_data: Box = Box::new(arrow_engine_data); Ok(engine_data.into()) } + +#[cfg(all(test, feature = "default-engine-base"))] +mod tests { + use std::sync::Arc; + + use delta_kernel::arrow::array::{Array, Int32Array, StructArray}; + use delta_kernel::arrow::datatypes::{DataType, Field}; + use delta_kernel::arrow::ffi::to_ffi; + + use super::*; + + /// Build a one-column `ArrowFFIData` we can use as a free-target. Mirrors the layout + /// produced by [`ArrowFFIData::try_from_engine_data`] (a struct array exported via + /// [`to_ffi`]) without going through the engine. + fn make_arrow_ffi_data() -> ArrowFFIData { + let column = Arc::new(Int32Array::from(vec![1, 2, 3])); + let field = Arc::new(Field::new("x", DataType::Int32, true)); + let struct_array = StructArray::from(vec![(field, column as _)]); + let array_data = struct_array.into_data(); + let (array, schema) = to_ffi(&array_data).expect("to_ffi"); + ArrowFFIData { array, schema } + } + + #[test] + fn free_null_is_safe() { + // Mirrors the convention enforced by `free_scan_metadata_arrow_result`. + unsafe { free_arrow_ffi_data(std::ptr::null_mut()) }; + } + + #[test] + fn empty_arrow_ffi_data_carries_no_release_callback() { + // The invariant `import_ffi_array` relies on to detect an unpopulated result slot. + assert!(ArrowFFIData::empty().array.is_released()); + } + + #[test] + fn free_drops_box_and_invokes_release_callbacks() { + // Verifies the common path: kernel allocates, engine immediately frees. The Drop + // impls on FFI_ArrowArray / FFI_ArrowSchema invoke their release callbacks; this + // test passes if there is no double-free / leak under sanitizers. + let ptr = Box::into_raw(Box::new(make_arrow_ffi_data())); + unsafe { free_arrow_ffi_data(ptr) }; + } + + #[test] + fn free_after_consumer_imported_arrays() { + // Mirrors the read-table example flow: an engine imports the Arrow arrays via + // `from_ffi` (which moves the release callbacks out, leaving the inner structs in + // a release-callback-less state), then frees the now-empty box. The Drop impls + // become no-ops on the consumed structs, but the Box itself still needs reclaiming. + let mut data = make_arrow_ffi_data(); + // Move the array+schema out via from_ffi -- this is what an engine does to import + // the data into its own arrow runtime. After this, `data.array` and `data.schema` + // have been replaced with empty stand-ins whose Drop impls are no-ops. + let array = std::mem::replace(&mut data.array, FFI_ArrowArray::empty()); + let schema = std::mem::replace(&mut data.schema, FFI_ArrowSchema::empty()); + // Consume them to release the underlying arrow allocations. + let _ = unsafe { arrow::array::ffi::from_ffi(array, &schema) }.expect("from_ffi"); + // Now free the now-empty box. + let ptr = Box::into_raw(Box::new(data)); + unsafe { free_arrow_ffi_data(ptr) }; + } +} diff --git a/ffi/src/engine_funcs.rs b/ffi/src/engine_funcs.rs index e1c76a5f89..b8d3af442a 100644 --- a/ffi/src/engine_funcs.rs +++ b/ffi/src/engine_funcs.rs @@ -11,13 +11,12 @@ use delta_kernel_ffi_macros::handle_descriptor; use tracing::debug; use url::Url; +use super::handle::Handle; use crate::{ ExclusiveEngineData, ExternEngine, ExternResult, IntoExternResult, KernelStringSlice, NullableCvoid, SharedExternEngine, SharedSchema, TryFromStringSlice, }; -use super::handle::Handle; - #[repr(C)] pub struct FileMeta { pub path: KernelStringSlice, @@ -46,6 +45,17 @@ impl Drop for FileReadResultIterator { } } +impl FileReadResultIterator { + /// Wrap a kernel `EngineData` iterator (paired with the engine that produced it) into a handle + /// the engine can drain via [`read_result_next`] and release with [`free_read_result_iter`]. + pub(crate) fn into_handle( + data: FileDataReadResultIterator, + engine: Arc, + ) -> Handle { + Box::new(FileReadResultIterator { data, engine }).into() + } +} + /// Call the engine back with the next `EngineData` batch read by Parquet/Json handler. The /// _engine_ "owns" the data that is passed into the `engine_visitor`, since it is allocated by the /// `Engine` being used for log-replay. If the engine wants the kernel to free this data, it _must_ @@ -216,14 +226,16 @@ fn evaluate_expression_impl( #[cfg(test)] mod tests { + use std::sync::Arc; + + use delta_kernel::schema::{DataType, StructField, StructType}; + use delta_kernel::Expression; + use super::{free_expression_evaluator, new_expression_evaluator}; use crate::ffi_test_utils::ok_or_panic; - use crate::{free_engine, handle::Handle, tests::get_default_engine, SharedSchema}; - use delta_kernel::{ - schema::{DataType, StructField, StructType}, - Expression, - }; - use std::sync::Arc; + use crate::handle::Handle; + use crate::tests::get_default_engine; + use crate::{free_engine, SharedSchema}; #[test] fn test_new_expression_evaluator() { diff --git a/ffi/src/error.rs b/ffi/src/error.rs index 9fcf3f04b1..3b032bbd64 100644 --- a/ffi/src/error.rs +++ b/ffi/src/error.rs @@ -1,6 +1,8 @@ use delta_kernel::{DeltaResult, Error}; +use tracing::warn; -use crate::{kernel_string_slice, ExternEngine, KernelStringSlice}; +use crate::handle::Handle; +use crate::{kernel_string_slice, ExclusiveRustString, ExternEngine, KernelStringSlice}; // We explicitly assign integer values to the error codes here because C and Rust are inconsistent // about values for "typedefed" features. Rust reserves the numbers for them regardless, so @@ -67,6 +69,9 @@ pub enum KernelError { LiteralExpressionTransformError = 40, CheckpointWriteError = 41, SchemaError = 42, + LogHistoryError = 43, + RowTrackingChangeFeedUnsupported = 44, + CancelledError = 45, } impl From for KernelError { @@ -80,6 +85,8 @@ impl From for KernelError { Error::Extract(..) => KernelError::ExtractError, Error::Generic(_) => KernelError::GenericError, Error::GenericError { .. } => KernelError::GenericError, + Error::MaxCatalogVersion(_) => KernelError::GenericError, + Error::LogTailVersionsNotContiguous { .. } => KernelError::GenericError, Error::IOError(_) => KernelError::IOErrorError, #[cfg(feature = "default-engine-base")] Error::Parquet(_) => KernelError::ParquetError, @@ -120,6 +127,9 @@ impl From for KernelError { Error::Unsupported(_) => KernelError::UnsupportedError, Error::ParseIntervalError(_) => KernelError::ParseIntervalError, Error::ChangeDataFeedUnsupported(_) => KernelError::ChangeDataFeedUnsupported, + Error::RowTrackingChangeFeedUnsupported(_) => { + KernelError::RowTrackingChangeFeedUnsupported + } Error::ChangeDataFeedIncompatibleSchema(_, _) => { KernelError::ChangeDataFeedIncompatibleSchema } @@ -128,6 +138,8 @@ impl From for KernelError { KernelError::LiteralExpressionTransformError } Error::Schema(_) => KernelError::SchemaError, + Error::LogHistory(_) => KernelError::LogHistoryError, + Error::Cancelled => KernelError::CancelledError, _ => KernelError::UnknownError, } } @@ -230,3 +242,177 @@ impl IntoExternResult for DeltaResult { } } } + +/// An error that can be returned from engine-side execution (e.g during an upcall). +/// +/// This is intended to be a kernel-allocated error which Engines can return TO kernel. It is the +/// inverse of [`EngineError`] (which is engine-allocated, and returned FROM kernel). +/// +/// The message is an [`ExclusiveRustString`] handle, which means the engine must +/// downcall to [`allocate_kernel_string`](crate::allocate_kernel_string) to construct it. Kernel +/// can then take ownership and free it appropriately after receiving the error. +#[repr(C)] +pub struct EngineExecError { + // TODO: we re-use KernelError for convenience, but we should ideally split this into a + // separate enum, containing only error types that make sense for the engine to return. + pub etype: KernelError, + pub message: Handle, +} + +/// Generic wrapper around an EngineExecError, representing the result of an engine upcall. +/// +/// Typically, engines will populate an out pointer with this result type. We include an `Uninit` +/// variant to signal that the engine returned without writing to the out pointer. Kernel should +/// always initialize such an out pointer to `Uninit` before handing it to an engine upcall. +/// +/// The variants are deliberately named `Success`/`Failure` rather than `Ok`/`Err` to avoid a +/// conflict with [`ExternResult`]. This is due to an issue in cbindgen, where generic types sharing +/// the same variant names causes failures during monomorphization (). +#[repr(C)] +pub enum EngineExecResult { + Success(T), + Failure(EngineExecError), + Uninit, +} + +/// Maps the given KernelError code to the given Error variant. Logs a warning if the associated +/// error message is non-empty. Useful for mapping kernel errors to error variants that don't +/// carry a message, but for some reason the engine still provided one. +fn messageless_error(code: KernelError, message: String, error: Error) -> Error { + if !message.is_empty() { + warn!("Discarding message for engine execution error ({code:?}): {message}"); + } + error +} + +impl From for Error { + /// Converts an [`EngineExecError`] into a [`delta_kernel::Error`], translating the + /// [`KernelError`] code back into its matching kernel error variant and consuming (and thereby + /// freeing) the message handle. + fn from(err: EngineExecError) -> Self { + let EngineExecError { etype, message } = err; + // SAFETY: `message` is an `ExclusiveRustString` handle that kernel owns and has not yet + // consumed. It is produced by the engine downcalling `allocate_kernel_string` and is + // consumed exactly once, here. + let message = *unsafe { message.into_inner() }; + match etype { + KernelError::CheckpointWriteError => Error::CheckpointWrite(message), + KernelError::EngineDataTypeError => Error::EngineDataType(message), + KernelError::GenericError => Error::Generic(message), + KernelError::InternalError => Error::InternalError(message), + KernelError::FileNotFoundError => Error::FileNotFound(message), + KernelError::MissingColumnError => Error::MissingColumn(message), + KernelError::UnexpectedColumnTypeError => Error::UnexpectedColumnType(message), + KernelError::MissingDataError => Error::MissingData(message), + KernelError::DeletionVectorError => Error::DeletionVector(message), + KernelError::InvalidProtocolError => Error::InvalidProtocol(message), + KernelError::JoinFailureError => Error::JoinFailure(message), + KernelError::InvalidColumnMappingModeError => Error::InvalidColumnMappingMode(message), + KernelError::InvalidTableLocationError => Error::InvalidTableLocation(message), + KernelError::InvalidDecimalError => Error::InvalidDecimal(message), + KernelError::InvalidStructDataError => Error::InvalidStructData(message), + KernelError::InvalidExpression => Error::InvalidExpressionEvaluation(message), + KernelError::InvalidLogPath => Error::InvalidLogPath(message), + KernelError::FileAlreadyExists => Error::FileAlreadyExists(message), + KernelError::UnsupportedError => Error::Unsupported(message), + KernelError::InvalidCheckpoint => Error::InvalidCheckpoint(message), + KernelError::SchemaError => Error::Schema(message), + code @ KernelError::MissingVersionError => { + messageless_error(code, message, Error::MissingVersion) + } + code @ KernelError::MissingMetadataError => { + messageless_error(code, message, Error::MissingMetadata) + } + code @ KernelError::MissingProtocolError => { + messageless_error(code, message, Error::MissingProtocol) + } + code @ KernelError::MissingMetadataAndProtocolError => { + messageless_error(code, message, Error::MissingMetadataAndProtocol) + } + code @ KernelError::CancelledError => { + messageless_error(code, message, Error::Cancelled) + } + + // These codes have no well-defined equivalent (e.g they wrap a foreign error type, + // carry a non-string payload, etc), so just map them to a generic error and + // preserve the code + message in the error string. + code @ (KernelError::UnknownError + | KernelError::FFIError + | KernelError::ExtractError + | KernelError::IOErrorError + | KernelError::InvalidUrlError + | KernelError::MalformedJsonError + | KernelError::ParseError + | KernelError::Utf8Error + | KernelError::ParseIntError + | KernelError::ParseIntervalError + | KernelError::ChangeDataFeedUnsupported + | KernelError::ChangeDataFeedIncompatibleSchema + | KernelError::RowTrackingChangeFeedUnsupported + | KernelError::LiteralExpressionTransformError + | KernelError::LogHistoryError) => { + Error::generic(format!("engine execution error ({code:?}): {message}")) + } + #[cfg(feature = "default-engine-base")] + code @ (KernelError::ArrowError + | KernelError::ParquetError + | KernelError::ObjectStoreError + | KernelError::ObjectStorePathError + | KernelError::ReqwestError) => { + Error::generic(format!("engine execution error ({code:?}): {message}")) + } + } + } +} + +#[cfg(test)] +mod error_code_tests { + use super::*; + + #[test] + fn row_tracking_change_feed_error_has_stable_ffi_mapping() { + assert_eq!( + KernelError::from(Error::RowTrackingChangeFeedUnsupported(7)), + KernelError::RowTrackingChangeFeedUnsupported + ); + assert_eq!(KernelError::RowTrackingChangeFeedUnsupported as i32, 44); + } +} + +#[cfg(all(test, feature = "declarative-plans"))] +mod tests { + use rstest::rstest; + + use super::*; + + fn exec_error(etype: KernelError, message: &str) -> EngineExecError { + let message: Handle = Box::new(message.to_string()).into(); + EngineExecError { etype, message } + } + + /// Each code should translate into its matching kernel error variant (preserving the message), + /// unit variants drop the message, and unmapped codes fall back to a generic error that retains + /// both the original code and message. + #[rstest] + #[case::file_not_found(KernelError::FileNotFoundError, "File not found: boom")] + #[case::schema(KernelError::SchemaError, "Schema error: boom")] + #[case::unsupported(KernelError::UnsupportedError, "Unsupported: boom")] + #[case::generic(KernelError::GenericError, "Generic delta kernel error: boom")] + #[case::invalid_expr(KernelError::InvalidExpression, "Invalid expression evaluation: boom")] + #[case::unit_missing_version(KernelError::MissingVersionError, "No table version found.")] + #[case::fallback_io( + KernelError::IOErrorError, + "Generic delta kernel error: engine execution error (IOErrorError): boom" + )] + #[case::fallback_row_tracking( + KernelError::RowTrackingChangeFeedUnsupported, + "Generic delta kernel error: engine execution error (RowTrackingChangeFeedUnsupported): boom" + )] + fn engine_exec_error_maps_kernel_error_code( + #[case] etype: KernelError, + #[case] expected: &str, + ) { + let err: Error = exec_error(etype, "boom").into(); + assert_eq!(err.to_string(), expected); + } +} diff --git a/ffi/src/expressions/arrow_eval.rs b/ffi/src/expressions/arrow_eval.rs new file mode 100644 index 0000000000..03b3f9e15c --- /dev/null +++ b/ffi/src/expressions/arrow_eval.rs @@ -0,0 +1,1480 @@ +//! `FfiOpaquePredicateOp`: an engine-defined opaque predicate evaluated through callbacks. +//! +//! Kernel's `evaluate_predicate` recurses through compound expressions (Binary, Junction, etc.) +//! natively. This module is the leaf: when kernel hits an opaque predicate, we evaluate each arg +//! via `evaluate_expression`, bundle the resulting `ArrayRef`s as a `RecordBatch`, export through +//! Arrow C Data Interface, and hand them to the engine's callback. Engine receives pre-computed +//! columns, never the AST. +//! +//! Available only under `default-engine-base` (requires the kernel-side arrow expression +//! evaluator). + +use std::sync::Arc; + +use delta_kernel::arrow::array::ffi::from_ffi; +use delta_kernel::arrow::array::{ + make_array, Array, ArrayRef, BooleanArray, RecordBatch, StructArray, +}; +use delta_kernel::arrow::datatypes::{Field, Fields, Schema}; +use delta_kernel::engine::arrow_expression::evaluate_expression::evaluate_expression; +use delta_kernel::engine::arrow_expression::opaque::{ + ArrowOpaquePredicate as _, ArrowOpaquePredicateOp, +}; +use delta_kernel::expressions::{Expression, ExpressionRef, Scalar, ScalarExpressionEvaluator}; +use delta_kernel::kernel_predicates::{ + DirectDataSkippingPredicateEvaluator, DirectPredicateEvaluator, + IndirectDataSkippingPredicateEvaluator, KernelPredicateEvaluator, +}; +use delta_kernel::schema::DataType; +use delta_kernel::{DeltaResult, Error, Predicate}; + +use super::opaque_eval::{COpaqueEvalCallbacks, FfiOpaqueEvalCallbacks}; +use crate::engine_data::ArrowFFIData; +use crate::error::EngineExecResult; +use crate::kernel_string_slice; + +// === FfiOpaquePredicateOp =================================================== + +/// Which engine callback an opaque op dispatches to (set by the stats rewrite; internal to the +/// FFI layer, not part of the ABI). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum EvalMode { + /// Row-time evaluation -> `eval_pred_rows`. + RowMode, + /// Stats-based evaluation (file data skipping) -> `eval_pred_stats`. + StatsMode, +} + +/// Engine-defined opaque predicate identified by a name (e.g. `STARTS_WITH`, `LIKE`). Kernel +/// routes evaluation through the engine's [`COpaqueEvalCallbacks`]: the engine receives +/// pre-evaluated args as Arrow arrays and returns one verdict per row (or per file, in stats +/// mode). +/// +/// # Data skipping +/// +/// `as_data_skipping_predicate` rewrites each `Column` arg into a +/// `Struct[min, max, nullcount, rowcount]` and tags the op with `EvalMode::StatsMode`. Literals +/// pass through; `Predicate` children recurse. This is the only mode that prunes files -- RowMode +/// evaluates per data row and never drops files, and the scalar / partition-pruning paths always +/// abstain. See [`EngineEvalRowsFn`] / [`EngineEvalStatsFn`] for the per-callback arg shapes. +/// +/// Inverted predicates (`NOT op`) are rewritten too: the inversion is recorded on the op and +/// forwarded to the engine callback's `inverted` flag, which must reason about the negated op (see +/// [`EngineEvalStatsFn`]). The StatsMode rewrite abstains (keeps all files) when a `Column` arg has +/// no sibling literal to infer its type from, or an arg is some kind other than `Column`, +/// `Literal`, or `Predicate`. +/// +/// [`EngineEvalRowsFn`]: super::opaque_eval::EngineEvalRowsFn +/// [`EngineEvalStatsFn`]: super::opaque_eval::EngineEvalStatsFn +#[derive(Debug, Clone)] +pub(crate) struct FfiOpaquePredicateOp { + name: String, + callbacks: Arc, + /// Which engine callback eval dispatches to. + mode: EvalMode, + /// Whether the op is negated. Set by the StatsMode rewrite so the eval callback can reason + /// about the negated op (see [`EngineEvalStatsFn`]); XORed with the eval-time `inverted`. + /// + /// [`EngineEvalStatsFn`]: super::opaque_eval::EngineEvalStatsFn + inverted: bool, +} + +impl FfiOpaquePredicateOp { + /// Build a RowMode, non-inverted op identified by `name`, evaluated via `callbacks`. + pub(crate) fn new(name: impl Into, callbacks: Arc) -> Self { + Self { + name: name.into(), + callbacks, + mode: EvalMode::RowMode, + inverted: false, + } + } + + /// Consume `self` and return it with `mode` overridden. + fn with_mode(mut self, mode: EvalMode) -> Self { + self.mode = mode; + self + } + + /// Consume `self` and return it with `inverted` overridden. + fn with_inverted(mut self, inverted: bool) -> Self { + self.inverted = inverted; + self + } +} + +impl PartialEq for FfiOpaquePredicateOp { + fn eq(&self, other: &Self) -> bool { + // Mode, inversion, and callbacks all change how the op evaluates, so equality includes + // them. Callbacks compare by `Arc` pointer identity (not otherwise comparable). + self.name == other.name + && self.mode == other.mode + && self.inverted == other.inverted + && Arc::ptr_eq(&self.callbacks, &other.callbacks) + } +} + +impl Eq for FfiOpaquePredicateOp {} + +/// Pre-evaluate each arg into an `ArrayRef`, bundle them as a `RecordBatch` keyed `arg0..argN`. +/// +/// `Expression::Struct` args (produced by the stats-mode rewrite) get a dedicated path because +/// kernel's evaluator needs a `DataType::Struct` result type to name fields, which we don't have. +fn evaluate_args(args: &[Expression], batch: &RecordBatch) -> DeltaResult { + // Zero-arg ops (e.g. NOW(), RAND()): empty-schema batch with explicit row count so the + // engine knows how many rows to emit. + if args.is_empty() { + let n_rows = batch.num_rows(); + return RecordBatch::try_new_with_options( + Arc::new(Schema::empty()), + vec![], + &delta_kernel::arrow::array::RecordBatchOptions::new().with_row_count(Some(n_rows)), + ) + .map_err(|e| Error::Generic(format!("zero-arg opaque eval batch construction: {e}"))); + } + + let arrays: Vec = args + .iter() + .map(|arg| match arg { + Expression::Struct(fields, _nullability) => evaluate_struct_arg(fields, batch), + _ => evaluate_expression(arg, batch, None), + }) + .collect::>()?; + + let fields: Vec = arrays + .iter() + .enumerate() + .map(|(i, a)| Field::new(format!("arg{i}"), a.data_type().clone(), true)) + .collect(); + let schema = Arc::new(Schema::new(fields)); + + RecordBatch::try_new(schema, arrays) + .map_err(|e| Error::Generic(format!("opaque eval batch construction: {e}"))) +} + +/// Lift a column type hint from the first `Literal` arg (kernel-native lifts the same way from +/// the right-hand `Scalar` in `col < val`). Returns `None` when there's no literal to borrow. +/// +/// The hint can be wrong: an opaque op's literal need not match the column's type (e.g. +/// `STR_LEN_GT(5, col("string_field"))` lifts `INTEGER` for a string column). The type only gates +/// min/max eligibility and the ref resolves to the real column type, so a wrong hint loses min/max +/// only when that type is itself min/max-ineligible. +fn pick_column_type_hint(exprs: &[Expression]) -> Option { + exprs.iter().find_map(|e| match e { + Expression::Literal(scalar) => Some(scalar.data_type()), + _ => None, + }) +} + +/// Rewrite a single opaque-predicate arg into its stats-mode form. `Column` -> a positional +/// `Struct[min, max, nullcount, rowcount]`; `Literal` passes through unchanged; `Predicate` +/// recurses through kernel's standard data-skipping evaluator. +/// +/// Returns `None` (abstaining the whole predicate) when the arg can't be expressed against stats: +/// an unsupported expression kind, or a `Column` with no sibling literal to infer its type. +fn rewrite_stat_arg( + evaluator: &IndirectDataSkippingPredicateEvaluator<'_>, + arg: &Expression, + type_hint: Option<&DataType>, +) -> Option { + let null_long = || Expression::literal(Scalar::Null(DataType::LONG)); + match arg { + Expression::Column(col) => { + // The two column kinds go through the same call: a partition column resolves to its + // exact value (the type is ignored), a data column to its min/max refs. The data + // case needs the type, so abstain when no sibling literal supplies one. + let ty = type_hint?; + let min = evaluator.get_min_stat(col, ty).unwrap_or_else(null_long); + let max = evaluator.get_max_stat(col, ty).unwrap_or_else(null_long); + let nullcount = evaluator.get_nullcount_stat(col).unwrap_or_else(null_long); + let rowcount = evaluator.get_rowcount_stat().unwrap_or_else(null_long); + Some(Expression::struct_from([min, max, nullcount, rowcount])) + } + Expression::Literal(_) => Some(arg.clone()), + Expression::Predicate(p) => Some(Expression::Predicate(Box::new( + evaluator.eval_pred(p, false)?, + ))), + _ => None, + } +} + +/// Materialize a stats-mode struct arg. Its children are the stat references the rewrite produced +/// (`minValues.`, `maxValues.`, nullcount, rowcount); evaluating them against the stats +/// batch resolves each to the actual per-file values. The results are packed into a `StructArray` +/// with positional field names (`f0`, `f1`, ...) since the engine reads the struct by index. +fn evaluate_struct_arg(fields: &[ExpressionRef], batch: &RecordBatch) -> DeltaResult { + let arrays: Vec = fields + .iter() + .map(|f| evaluate_expression(f, batch, None)) + .collect::>()?; + let arrow_fields: Fields = arrays + .iter() + .enumerate() + .map(|(i, a)| Field::new(format!("f{i}"), a.data_type().clone(), true)) + .collect(); + StructArray::try_new(arrow_fields, arrays, None) + .map(|sa| Arc::new(sa) as ArrayRef) + .map_err(|e| Error::Generic(format!("struct arg construction: {e}"))) +} + +/// Import an engine-produced `ArrowFFIData` into an `ArrayRef`, consuming the Arrow C Data +/// Interface handles. +fn import_ffi_array(ffi: ArrowFFIData) -> DeltaResult { + // A released (empty) array means the engine reported success without populating the result + // slot. This check is load-bearing: `from_ffi` asserts on the empty structs' null pointers, + // and kernel must never panic -- so reject the unpopulated case with an error up front. + if ffi.array.is_released() { + return Err(Error::Generic( + "engine callback returned success but no result array".into(), + )); + } + + let ArrowFFIData { array, schema } = ffi; + + // SAFETY: the engine promised these structs are valid Arrow C Data + // Interface payloads it produced and handed to us. + let array_data = unsafe { from_ffi(array, &schema) } + .map_err(|e| Error::Generic(format!("from_ffi: {e}")))?; + Ok(make_array(array_data)) +} + +fn require_boolean_array(arr: ArrayRef, expected_rows: usize) -> DeltaResult { + if arr.len() != expected_rows { + return Err(Error::Generic(format!( + "opaque predicate eval_pred returned {} rows, expected {expected_rows}", + arr.len() + ))); + } + arr.as_any() + .downcast_ref::() + .cloned() + .ok_or_else(|| { + Error::Generic(format!( + "opaque predicate eval_pred returned non-boolean array of type {:?}", + arr.data_type() + )) + }) +} + +fn call_eval_pred( + cb: &COpaqueEvalCallbacks, + op_name: &str, + args_batch: RecordBatch, + mode: EvalMode, + inverted: bool, +) -> DeltaResult { + let num_rows = args_batch.num_rows(); + let args_ffi = ArrowFFIData::try_from_record_batch(args_batch)?; + + // The args batch transfers to the engine by value; the engine owns it and must release it (by + // importing via `from_ffi`, or otherwise) on every path. The result uses the out-pointer + // convention: kernel pre-initializes the slot to Uninit, and the engine writes Success(result) + // -- transferring ownership of the written Arrow C Data Interface structs -- or Failure(err). + let mut result = EngineExecResult::Uninit; + + // SAFETY: callback was supplied by the engine. `args_ffi` is moved into the call (engine owns + // it now), and `result` is a writable slot valid for the duration of the call. + unsafe { + let name = kernel_string_slice!(op_name); + let eval = match mode { + EvalMode::RowMode => cb.eval_pred_rows, + EvalMode::StatsMode => cb.eval_pred_stats, + }; + eval(cb.engine_state, name, args_ffi, inverted, &mut result); + } + let result_ffi = match result { + EngineExecResult::Success(result_ffi) => result_ffi, + EngineExecResult::Failure(err) => return Err(err.into()), + EngineExecResult::Uninit => { + return Err(Error::Generic(format!( + "engine opaque-eval callback for `{op_name}` returned without writing a result" + ))) + } + }; + + let arr = import_ffi_array(result_ffi)?; + require_boolean_array(arr, num_rows) +} + +// === ArrowOpaquePredicateOp ==================================================== + +impl ArrowOpaquePredicateOp for FfiOpaquePredicateOp { + fn name(&self) -> &str { + &self.name + } + + fn eval_pred( + &self, + args: &[Expression], + batch: &RecordBatch, + inverted: bool, + ) -> DeltaResult { + // Materialize the args batch. StatsMode pruning is best-effort: if the rewrite referenced a + // stats column the batch doesn't carry, abstain (keep every file) rather than abort the + // scan. RowMode has no safe abstain, so its errors propagate. + let args_batch = match evaluate_args(args, batch) { + Ok(args_batch) => args_batch, + Err(e) if self.mode == EvalMode::StatsMode => { + tracing::warn!( + "opaque predicate `{}`: cannot materialize stats args ({e}); keeping all files", + self.name + ); + return Ok(BooleanArray::from(vec![true; batch.num_rows()])); + } + Err(e) => return Err(e), + }; + // The StatsMode rewrite records inversion on the op (the stats predicate is then evaluated + // non-inverted by kernel); RowMode carries it via the eval-time flag. XOR composes both. + let inverted = inverted ^ self.inverted; + call_eval_pred( + &self.callbacks.inner, + &self.name, + args_batch, + self.mode, + inverted, + ) + } + + fn eval_pred_scalar( + &self, + _eval_expr: &ScalarExpressionEvaluator<'_>, + _eval_pred: &DirectPredicateEvaluator<'_>, + _exprs: &[Expression], + _inverted: bool, + ) -> DeltaResult> { + // Abstains from scalar evaluation (e.g. partition pruning). + // TODO: support it by invoking the engine callback with a one-row stats batch. + Ok(None) + } + + fn eval_as_data_skipping_predicate( + &self, + _evaluator: &DirectDataSkippingPredicateEvaluator<'_>, + _exprs: &[Expression], + _inverted: bool, + ) -> Option { + // Row-group skipping: abstain. Opaque predicates prune at the file level instead -- via the + // `as_data_skipping_predicate` rewrite, then `evaluate_predicate` -> `eval_pred`. + // TODO: support row-group skipping by invoking the engine callback with a one-row stats + // batch. + None + } + + fn as_data_skipping_predicate( + &self, + evaluator: &IndirectDataSkippingPredicateEvaluator<'_>, + exprs: &[Expression], + inverted: bool, + ) -> Option { + // Each `Column` arg becomes a `Struct[min, max, nullcount, rowcount]`; literals pass + // through; `Predicate` children recurse. If any arg can't be rewritten (e.g. a data + // column with no type hint), the whole predicate abstains and kernel keeps every file. + let type_hint = pick_column_type_hint(exprs); + let rewritten = exprs + .iter() + .map(|arg| rewrite_stat_arg(evaluator, arg, type_hint.as_ref())) + .collect::>>()?; + + // Record inversion on the op, not as a surrounding `Predicate::not` (kernel would flip the + // verdict, unsound for stats). Use arrow_opaque so dispatch reaches our callback. + let stats_op = self + .clone() + .with_mode(EvalMode::StatsMode) + .with_inverted(inverted); + Some(Predicate::arrow_opaque(stats_op, rewritten)) + } +} + +#[cfg(test)] +mod tests { + #![allow(clippy::unwrap_used, clippy::panic)] + + use std::ffi::c_void; + use std::ptr; + use std::sync::atomic::{AtomicUsize, Ordering as AtomicOrdering}; + use std::sync::Arc; + + use delta_kernel::arrow::array::ffi::{from_ffi, FFI_ArrowArray, FFI_ArrowSchema}; + use delta_kernel::arrow::array::{ + ArrayRef, BooleanArray, Int32Array, RecordBatch, StringArray, StructArray, + }; + use delta_kernel::arrow::datatypes::{DataType as ArrowDataType, Field, Schema}; + use delta_kernel::engine::arrow_expression::evaluate_expression::evaluate_predicate; + use delta_kernel::engine::arrow_expression::opaque::ArrowOpaquePredicate as _; + use delta_kernel::expressions::{col, lit, Expression, Predicate}; + + use super::*; + use crate::expressions::opaque_eval::EngineEvalRowsFn; + use crate::KernelStringSlice; + + // === Engine-side test stubs =================================================== + // + // Tiny "engine" written in Rust that demonstrates composite opaque predicates. + // `STARTS_WITH_LOWER(col, prefix)` folds the LOWER step into the predicate + // callback, so no `Expression::Opaque` ever appears in the tree. + + /// Import the by-value args batch, consuming its FFI handles (their release callbacks fire + /// when the imported data drops). + unsafe fn take_ffi_record_batch(args_in: ArrowFFIData) -> RecordBatch { + let ArrowFFIData { array, schema } = args_in; + let array_data = unsafe { from_ffi(array, &schema) }.unwrap(); + let sa = StructArray::from(array_data); + sa.into() + } + + /// Write an `ArrayRef` into kernel's result slot as Arrow C Data Interface structs, the way a + /// real engine returns its result from `eval_pred` (out-pointer convention). + unsafe fn write_result(out: *mut EngineExecResult, arr: ArrayRef) { + let array_data = arr.to_data(); + let ffi = ArrowFFIData { + array: FFI_ArrowArray::new(&array_data), + schema: FFI_ArrowSchema::try_from(array_data.data_type()).unwrap(), + }; + unsafe { *out = EngineExecResult::Success(ffi) }; + } + + /// Composite opaque predicate: `STARTS_WITH(LOWER(args[0]), args[1])`. Both + /// args arrive as pre-evaluated String columns; the engine applies LOWER + /// then prefix-match per row. + unsafe extern "C" fn engine_starts_with_lower( + _state: *mut c_void, + _op_name: KernelStringSlice, + args_in: ArrowFFIData, + inverted: bool, + out: *mut EngineExecResult, + ) { + let batch = unsafe { take_ffi_record_batch(args_in) }; + assert_eq!(batch.num_columns(), 2); + let col = batch + .column(0) + .as_any() + .downcast_ref::() + .expect("arg0 should be StringArray"); + let prefix = batch + .column(1) + .as_any() + .downcast_ref::() + .expect("arg1 should be StringArray"); + let verdicts: BooleanArray = (0..batch.num_rows()) + .map(|i| match (col.is_valid(i), prefix.is_valid(i)) { + (true, true) => { + let m = col.value(i).to_lowercase().starts_with(prefix.value(i)); + Some(if inverted { !m } else { m }) + } + _ => None, + }) + .collect(); + unsafe { write_result(out, Arc::new(verdicts)) } + } + + /// Simpler opaque predicate: `STARTS_WITH(args[0], args[1])` with no LOWER. + /// Used by the composition test where we need two ops sharing the same op + /// name function. + unsafe extern "C" fn engine_starts_with( + _state: *mut c_void, + _op_name: KernelStringSlice, + args_in: ArrowFFIData, + inverted: bool, + out: *mut EngineExecResult, + ) { + let batch = unsafe { take_ffi_record_batch(args_in) }; + let lhs = batch + .column(0) + .as_any() + .downcast_ref::() + .expect("arg0 should be StringArray"); + let rhs = batch + .column(1) + .as_any() + .downcast_ref::() + .expect("arg1 should be StringArray"); + let verdicts: BooleanArray = (0..batch.num_rows()) + .map(|i| match (lhs.is_valid(i), rhs.is_valid(i)) { + (true, true) => { + let m = lhs.value(i).starts_with(rhs.value(i)); + Some(if inverted { !m } else { m }) + } + _ => None, + }) + .collect(); + unsafe { write_result(out, Arc::new(verdicts)) } + } + + unsafe extern "C" fn noop_free(_state: *mut c_void) {} + + fn callbacks_for(eval_pred: EngineEvalRowsFn) -> Arc { + // Each test exercises one mode; the unused slot is never invoked, so wire the same stub to + // both row and stats callbacks. + Arc::new(FfiOpaqueEvalCallbacks::new(COpaqueEvalCallbacks { + engine_state: ptr::null_mut(), + eval_pred_rows: eval_pred, + eval_pred_stats: eval_pred, + free_state: noop_free, + })) + } + + fn batch_with_col(values: Vec>) -> RecordBatch { + let schema = Arc::new(Schema::new(vec![Field::new( + "col", + ArrowDataType::Utf8, + true, + )])); + let arr: ArrayRef = Arc::new(StringArray::from(values)); + RecordBatch::try_new(schema, vec![arr]).unwrap() + } + + // === Op identity ============================================================= + + #[test] + fn op_equality_includes_name_mode_inversion_and_callback_identity() { + let cb = callbacks_for(engine_starts_with); + let op = FfiOpaquePredicateOp::new("LIKE", cb.clone()); + + // Same name + same callbacks Arc -> equal. + assert_eq!(op, FfiOpaquePredicateOp::new("LIKE", cb.clone())); + // Different name -> not equal. + assert_ne!(op, FfiOpaquePredicateOp::new("OTHER", cb.clone())); + // Mode and inversion change how the op evaluates -> not equal. + assert_ne!( + op, + FfiOpaquePredicateOp::new("LIKE", cb.clone()).with_mode(EvalMode::StatsMode) + ); + assert_ne!( + op, + FfiOpaquePredicateOp::new("LIKE", cb).with_inverted(true) + ); + // Distinct callback Arcs -> not equal (pointer identity). + assert_ne!( + op, + FfiOpaquePredicateOp::new("LIKE", callbacks_for(engine_starts_with)) + ); + } + + // === End-to-end: STARTS_WITH(LOWER(col), "foo") as one composite op ========= + + #[test] + fn composite_starts_with_lower_predicate_round_trips() { + let pred = Predicate::arrow_opaque( + FfiOpaquePredicateOp::new("STARTS_WITH_LOWER", callbacks_for(engine_starts_with_lower)), + [col!("col"), lit("foo")], + ); + + let batch = batch_with_col(vec![Some("FOOBAR"), Some("Apple"), None, Some("foozzz")]); + let result = evaluate_predicate(&pred, &batch, false).unwrap(); + + assert_eq!(result.len(), 4); + assert!(result.value(0), "FOOBAR -> foobar starts_with foo"); + assert!(!result.value(1), "Apple -> apple does not start_with foo"); + assert!(result.is_null(2), "null col -> null verdict"); + assert!(result.value(3), "foozzz starts_with foo"); + } + + #[test] + fn starts_with_is_case_sensitive() { + // Plain op (no LOWER) matches exactly, so case matters here -- unlike the LOWER variant. + let pred = Predicate::arrow_opaque( + FfiOpaquePredicateOp::new("STARTS_WITH", callbacks_for(engine_starts_with)), + [col!("col"), lit("foo")], + ); + let batch = batch_with_col(vec![Some("foobar"), Some("FOOBAR")]); + let result = evaluate_predicate(&pred, &batch, false).unwrap(); + assert!(result.value(0), "foobar starts with foo"); + assert!(!result.value(1), "FOOBAR does not start with foo"); + } + + #[test] + fn inversion_flips_verdicts() { + let pred = Predicate::arrow_opaque( + FfiOpaquePredicateOp::new("STARTS_WITH_LOWER", callbacks_for(engine_starts_with_lower)), + [col!("col"), lit("foo")], + ); + let batch = batch_with_col(vec![Some("FOOBAR"), Some("Apple")]); + let result = evaluate_predicate(&pred, &batch, true).unwrap(); + assert!(!result.value(0), "inverted: FOOBAR no longer matches"); + assert!(result.value(1), "inverted: Apple now matches"); + } + + // === Composition: AND of two opaque predicates ============================== + + #[test] + fn composition_and_of_two_opaque_predicates() { + let cb = callbacks_for(engine_starts_with); + let lhs = Predicate::arrow_opaque( + FfiOpaquePredicateOp::new("STARTS_WITH", cb.clone()), + [col!("col"), lit("F")], + ); + let rhs = Predicate::arrow_opaque( + FfiOpaquePredicateOp::new("STARTS_WITH", cb), + [col!("col"), lit("FO")], + ); + let and_pred = Predicate::and_from([lhs, rhs]); + + let batch = batch_with_col(vec![Some("FOOBAR"), Some("FBAR"), Some("BAR")]); + let result = evaluate_predicate(&and_pred, &batch, false).unwrap(); + assert!(result.value(0), "FOOBAR starts with F AND FO"); + assert!(!result.value(1), "FBAR starts with F but not FO"); + assert!(!result.value(2), "BAR starts with neither"); + } + + // === Negative paths ========================================================== + + #[test] + fn engine_leaving_result_uninit_surfaces_error() { + unsafe extern "C" fn engine_noop( + _state: *mut c_void, + _op_name: KernelStringSlice, + _args_in: ArrowFFIData, + _inverted: bool, + _out: *mut EngineExecResult, + ) { + // Leave *out as the pre-initialized `Uninit`: the engine returned without writing. + } + let pred = Predicate::arrow_opaque( + FfiOpaquePredicateOp::new("WRITES_NOTHING", callbacks_for(engine_noop)), + [col!("col")], + ); + let batch = batch_with_col(vec![Some("x")]); + let err = evaluate_predicate(&pred, &batch, false).unwrap_err(); + assert!( + format!("{err}").contains("returned without writing a result"), + "got: {err}" + ); + } + + #[test] + fn engine_failure_surfaces_engine_error_message() { + unsafe extern "C" fn engine_fail( + _state: *mut c_void, + _op_name: KernelStringSlice, + args_in: ArrowFFIData, + _inverted: bool, + out: *mut EngineExecResult, + ) { + let _ = unsafe { take_ffi_record_batch(args_in) }; + let err = crate::error::EngineExecError { + etype: crate::error::KernelError::GenericError, + message: Box::new("boom from engine".to_string()).into(), + }; + unsafe { *out = EngineExecResult::Failure(err) }; + } + let pred = Predicate::arrow_opaque( + FfiOpaquePredicateOp::new("FAILS", callbacks_for(engine_fail)), + [col!("col")], + ); + let batch = batch_with_col(vec![Some("x")]); + let err = evaluate_predicate(&pred, &batch, false).unwrap_err(); + assert!( + format!("{err}").contains("boom from engine"), + "engine-provided message should surface; got: {err}" + ); + } + + #[test] + fn engine_returning_non_boolean_array_errors() { + unsafe extern "C" fn engine_int( + _state: *mut c_void, + _op_name: KernelStringSlice, + args_in: ArrowFFIData, + _inverted: bool, + out: *mut EngineExecResult, + ) { + let _ = unsafe { take_ffi_record_batch(args_in) }; + let arr: ArrayRef = Arc::new(Int32Array::from(vec![1, 2, 3])); + unsafe { write_result(out, arr) } + } + let pred = Predicate::arrow_opaque( + FfiOpaquePredicateOp::new("RETURNS_INT", callbacks_for(engine_int)), + [col!("col")], + ); + let batch = batch_with_col(vec![Some("x"), Some("y"), Some("z")]); + let err = evaluate_predicate(&pred, &batch, false).unwrap_err(); + assert!(format!("{err}").contains("non-boolean array"), "got: {err}"); + } + + #[test] + fn engine_returning_wrong_length_array_errors() { + unsafe extern "C" fn engine_short( + _state: *mut c_void, + _op_name: KernelStringSlice, + args_in: ArrowFFIData, + _inverted: bool, + out: *mut EngineExecResult, + ) { + let _ = unsafe { take_ffi_record_batch(args_in) }; + let arr: ArrayRef = Arc::new(BooleanArray::from(vec![true])); + unsafe { write_result(out, arr) } + } + let pred = Predicate::arrow_opaque( + FfiOpaquePredicateOp::new("RETURNS_SHORT", callbacks_for(engine_short)), + [col!("col")], + ); + let batch = batch_with_col(vec![Some("x"), Some("y"), Some("z")]); + let err = evaluate_predicate(&pred, &batch, false).unwrap_err(); + assert!( + format!("{err}").contains("returned 1 rows, expected 3"), + "got: {err}" + ); + } + + #[test] + fn engine_returning_success_without_result_errors() { + unsafe extern "C" fn engine_empty( + _state: *mut c_void, + _op_name: KernelStringSlice, + args_in: ArrowFFIData, + _inverted: bool, + out: *mut EngineExecResult, + ) { + // Consume args but hand back an empty (unpopulated) result. + let _ = unsafe { take_ffi_record_batch(args_in) }; + unsafe { *out = EngineExecResult::Success(ArrowFFIData::empty()) }; + } + let pred = Predicate::arrow_opaque( + FfiOpaquePredicateOp::new("EMPTY", callbacks_for(engine_empty)), + [col!("col")], + ); + let batch = batch_with_col(vec![Some("x")]); + let err = evaluate_predicate(&pred, &batch, false).unwrap_err(); + assert!(format!("{err}").contains("no result array"), "got: {err}"); + } + + // === Zero-arg opaque predicate =============================================== + + #[test] + fn zero_arg_opaque_predicate_propagates_row_count() { + unsafe extern "C" fn engine_always_true( + _state: *mut c_void, + _op_name: KernelStringSlice, + args_in: ArrowFFIData, + _inverted: bool, + out: *mut EngineExecResult, + ) { + let batch = unsafe { take_ffi_record_batch(args_in) }; + assert_eq!(batch.num_columns(), 0, "zero-arg => no columns"); + let n = batch.num_rows(); + let arr: ArrayRef = Arc::new(BooleanArray::from(vec![true; n])); + unsafe { write_result(out, arr) } + } + let pred = Predicate::arrow_opaque( + FfiOpaquePredicateOp::new("ZERO_ARG", callbacks_for(engine_always_true)), + [] as [Expression; 0], + ); + let batch = batch_with_col(vec![Some("a"); 5]); + let result = evaluate_predicate(&pred, &batch, false).unwrap(); + assert_eq!(result.len(), 5); + assert!((0..5).all(|i| result.value(i))); + } + + /// `call_eval_pred` must route by mode: a RowMode op hits `eval_pred_rows`, a StatsMode op hits + /// `eval_pred_stats`. Every other test wires the same stub into both slots, so a swapped match + /// arm would pass them -- this pins the routing with two distinct stubs. + #[test] + fn dispatch_routes_each_mode_to_its_callback() { + static ROWS: AtomicUsize = AtomicUsize::new(0); + static STATS: AtomicUsize = AtomicUsize::new(0); + + unsafe extern "C" fn rows( + _: *mut c_void, + _: KernelStringSlice, + args_in: ArrowFFIData, + _: bool, + out: *mut EngineExecResult, + ) { + ROWS.fetch_add(1, AtomicOrdering::SeqCst); + let batch = unsafe { take_ffi_record_batch(args_in) }; + unsafe { + write_result( + out, + Arc::new(BooleanArray::from(vec![true; batch.num_rows()])), + ) + } + } + unsafe extern "C" fn stats( + _: *mut c_void, + _: KernelStringSlice, + args_in: ArrowFFIData, + _: bool, + out: *mut EngineExecResult, + ) { + STATS.fetch_add(1, AtomicOrdering::SeqCst); + let batch = unsafe { take_ffi_record_batch(args_in) }; + unsafe { + write_result( + out, + Arc::new(BooleanArray::from(vec![true; batch.num_rows()])), + ) + } + } + let callbacks = || { + Arc::new(FfiOpaqueEvalCallbacks::new(COpaqueEvalCallbacks { + engine_state: ptr::null_mut(), + eval_pred_rows: rows, + eval_pred_stats: stats, + free_state: noop_free, + })) + }; + let batch = batch_with_col(vec![Some("x")]); + + // RowMode (default) routes to eval_pred_rows only. + let row_pred = + Predicate::arrow_opaque(FfiOpaquePredicateOp::new("OP", callbacks()), [col!("col")]); + evaluate_predicate(&row_pred, &batch, false).unwrap(); + assert_eq!(ROWS.load(AtomicOrdering::SeqCst), 1); + assert_eq!( + STATS.load(AtomicOrdering::SeqCst), + 0, + "row mode must not hit the stats slot" + ); + + // StatsMode routes to eval_pred_stats only. + let stats_pred = Predicate::arrow_opaque( + FfiOpaquePredicateOp::new("OP", callbacks()).with_mode(EvalMode::StatsMode), + [col!("col")], + ); + evaluate_predicate(&stats_pred, &batch, false).unwrap(); + assert_eq!( + ROWS.load(AtomicOrdering::SeqCst), + 1, + "stats mode must not hit the rows slot" + ); + assert_eq!(STATS.load(AtomicOrdering::SeqCst), 1); + } + + // === Data-skipping rewrite ================================================== + + mod rewrite { + use std::cmp::Ordering; + use std::sync::atomic::{AtomicUsize, Ordering as AtomicOrdering}; + + use delta_kernel::arrow::array::Int64Array; + use delta_kernel::arrow::datatypes::DataType as ArrowDataType; + use delta_kernel::engine::arrow_expression::opaque::ArrowOpaquePredicateOp as _; + use delta_kernel::expressions::{ + col, column_name, joined_column_expr, lit, BinaryExpressionOp, BinaryPredicateOp, + CastOptions, ColumnName, Expression, JunctionPredicateOp, OpaquePredicateOpRef, Scalar, + }; + use delta_kernel::kernel_predicates::DataSkippingPredicateEvaluator; + use delta_kernel::schema::DataType; + use delta_kernel::Predicate; + + use super::*; + + /// Stub evaluator mirroring `DataSkippingPredicateCreator`: data columns produce + /// `stats_parsed.{min,max}Values.` refs, partition columns produce + /// `partitionValues_parsed.`. + struct StatsRewriter { + has_stats: bool, + partition: bool, + } + + fn stats_col(prefix: &str, col: &ColumnName) -> Expression { + Expression::from( + column_name!("stats_parsed") + .join(&ColumnName::new([prefix])) + .join(col), + ) + } + + fn partition_col(col: &ColumnName) -> Expression { + joined_column_expr!("partitionValues_parsed", col) + } + + fn stats_col_numrecords() -> Expression { + col!("stats_parsed.numRecords") + } + + /// Single-file stats batch: `stats_parsed { minValues.col=1, maxValues.col=10, + /// numRecords=1 }`. Shared by the rewrite-dispatch tests below. + fn single_col_int64_stats_batch() -> RecordBatch { + let col_field = Arc::new(Field::new("col", ArrowDataType::Int64, true)); + let min_struct = StructArray::from(vec![( + col_field.clone(), + Arc::new(Int64Array::from(vec![1])) as ArrayRef, + )]); + let max_struct = StructArray::from(vec![( + col_field, + Arc::new(Int64Array::from(vec![10])) as ArrayRef, + )]); + let num_records: ArrayRef = Arc::new(Int64Array::from(vec![1i64])); + let stats_parsed = StructArray::from(vec![ + ( + Arc::new(Field::new( + "minValues", + min_struct.data_type().clone(), + true, + )), + Arc::new(min_struct) as ArrayRef, + ), + ( + Arc::new(Field::new( + "maxValues", + max_struct.data_type().clone(), + true, + )), + Arc::new(max_struct) as ArrayRef, + ), + ( + Arc::new(Field::new("numRecords", ArrowDataType::Int64, true)), + num_records, + ), + ]); + let schema = Arc::new(Schema::new(vec![Field::new( + "stats_parsed", + stats_parsed.data_type().clone(), + true, + )])); + RecordBatch::try_new(schema, vec![Arc::new(stats_parsed) as ArrayRef]).unwrap() + } + + impl DataSkippingPredicateEvaluator for StatsRewriter { + type Output = Predicate; + type ColumnStat = Expression; + + fn get_min_stat(&self, col: &ColumnName, _: &DataType) -> Option { + if self.partition { + return Some(partition_col(col)); + } + self.has_stats.then(|| stats_col("minValues", col)) + } + fn get_max_stat(&self, col: &ColumnName, _: &DataType) -> Option { + if self.partition { + return Some(partition_col(col)); + } + self.has_stats.then(|| stats_col("maxValues", col)) + } + fn get_nullcount_stat(&self, _: &ColumnName) -> Option { + None + } + fn get_rowcount_stat(&self) -> Option { + Some(stats_col_numrecords()) + } + fn eval_pred_scalar(&self, val: &Scalar, inverted: bool) -> Option { + if let Scalar::Boolean(b) = val { + Some(Predicate::literal(*b != inverted)) + } else { + None + } + } + fn eval_pred_scalar_is_null(&self, _: &Scalar, _: bool) -> Option { + None + } + fn eval_pred_is_null(&self, _: &ColumnName, _: bool) -> Option { + None + } + fn eval_pred_binary_scalars( + &self, + _: BinaryPredicateOp, + _: &Scalar, + _: &Scalar, + _: bool, + ) -> Option { + None + } + fn eval_pred_cast( + &self, + _: BinaryPredicateOp, + _: &ColumnName, + _: &DataType, + _: &CastOptions, + _: &Scalar, + _: bool, + ) -> Option { + None + } + fn eval_pred_opaque( + &self, + _: &OpaquePredicateOpRef, + _: &[Expression], + _: bool, + ) -> Option { + None + } + fn finish_eval_pred_junction( + &self, + op: JunctionPredicateOp, + preds: &mut dyn Iterator>, + inverted: bool, + ) -> Option { + let collected: Vec = preds.collect::>>()?; + let pred = Predicate::junction(op, collected); + Some(if inverted { Predicate::not(pred) } else { pred }) + } + fn eval_partial_cmp( + &self, + ord: Ordering, + col: Expression, + val: &Scalar, + inverted: bool, + ) -> Option { + let base_op = match ord { + Ordering::Less => BinaryPredicateOp::LessThan, + Ordering::Greater => BinaryPredicateOp::GreaterThan, + Ordering::Equal => BinaryPredicateOp::Equal, + }; + let pred = Predicate::binary(base_op, col, Expression::literal(val.clone())); + Some(if inverted { Predicate::not(pred) } else { pred }) + } + } + + fn rewriter() -> StatsRewriter { + StatsRewriter { + has_stats: true, + partition: false, + } + } + + fn no_stats_rewriter() -> StatsRewriter { + StatsRewriter { + has_stats: false, + partition: false, + } + } + + fn partition_rewriter() -> StatsRewriter { + StatsRewriter { + has_stats: true, + partition: true, + } + } + + fn op_with_callbacks(name: &str) -> FfiOpaquePredicateOp { + FfiOpaquePredicateOp::new(name, callbacks_for(engine_starts_with)) + } + + fn rewrite( + op: &FfiOpaquePredicateOp, + args: &[Expression], + rewriter: &StatsRewriter, + inverted: bool, + ) -> Option { + op.as_data_skipping_predicate( + rewriter + as &dyn DataSkippingPredicateEvaluator< + Output = Predicate, + ColumnStat = Expression, + >, + args, + inverted, + ) + } + + #[test] + fn inverted_range_op_prunes_complement() { + // Range op `GE(col, t)`: non-inverted keeps a file iff some value could be `>= t` + // (max >= t); inverted (`col < t`) keeps iff some value could be `< t` (min < t). + // Same stats, different computation -- not a verdict flip. + unsafe extern "C" fn engine_ge( + _state: *mut c_void, + _op_name: KernelStringSlice, + args_in: ArrowFFIData, + inverted: bool, + out: *mut EngineExecResult, + ) { + let batch = unsafe { take_ffi_record_batch(args_in) }; + let stats = batch + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + let min = stats + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + let max = stats + .column(1) + .as_any() + .downcast_ref::() + .unwrap(); + let target = batch + .column(1) + .as_any() + .downcast_ref::() + .unwrap(); + let keep: BooleanArray = (0..batch.num_rows()) + .map(|i| { + let t = target.value(i); + Some(if inverted { + min.value(i) < t + } else { + max.value(i) >= t + }) + }) + .collect(); + unsafe { write_result(out, Arc::new(keep)) } + } + + // Stats batch: a single file with id in [1, 10]. Target 11 is above the whole range. + let op = FfiOpaquePredicateOp::new("GE", callbacks_for(engine_ge)); + let args = [col!("col"), lit(11i64)]; + let stats_batch = single_col_int64_stats_batch(); + + // GE(col, 11): max 10 < 11 -> prune. + let non_inverted = + rewrite(&op, &args, &rewriter(), false).expect("rewrite should succeed"); + let r = evaluate_predicate(&non_inverted, &stats_batch, false).unwrap(); + assert!(!r.value(0), "GE(col,11): max 10 < 11 prunes the file"); + + // NOT GE(col, 11) == col < 11: min 1 < 11 -> keep. Inverted is forwarded, not + // abstained. + let inverted = + rewrite(&op, &args, &rewriter(), true).expect("inverted rewrite should succeed"); + let r = evaluate_predicate(&inverted, &stats_batch, false).unwrap(); + assert!( + r.value(0), + "NOT GE(col,11) == col < 11: min 1 < 11 keeps the file" + ); + } + + #[test] + fn wraps_column_in_min_max_nullcount_rowcount_struct() { + let op = op_with_callbacks("STARTS_WITH"); + let args = [col!("col"), lit("foo")]; + let pred = rewrite(&op, &args, &rewriter(), false).expect("rewrite should succeed"); + + let Predicate::Opaque(opaque) = pred else { + panic!("expected Opaque, got {pred:?}"); + }; + assert_eq!(opaque.exprs.len(), 2); + let Expression::Struct(fields, _) = &opaque.exprs[0] else { + panic!("expected Struct arg, got {:?}", opaque.exprs[0]); + }; + assert_eq!(fields.len(), 4); + assert_eq!(*fields[0], stats_col("minValues", &column_name!("col"))); + assert_eq!(*fields[1], stats_col("maxValues", &column_name!("col"))); + assert_eq!( + *fields[2], + Expression::literal(Scalar::Null(DataType::LONG)) + ); + assert_eq!(*fields[3], stats_col_numrecords()); + assert_eq!(opaque.exprs[1], Expression::literal("foo")); + } + + /// A data column with no sibling literal gives no type hint, so min/max stats can't be + /// read. We abstain for the whole predicate rather than emit a partial wrapper. + #[test] + fn abstains_when_data_column_has_no_type_hint() { + let op = op_with_callbacks("IS_DEFINED"); + let args = [col!("col")]; + assert!(rewrite(&op, &args, &rewriter(), false).is_none()); + } + + #[test] + fn passes_literals_through_unchanged() { + let op = op_with_callbacks("OP"); + let args = [Expression::literal("foo")]; + let pred = rewrite(&op, &args, &rewriter(), false).expect("rewrite should succeed"); + let Predicate::Opaque(opaque) = pred else { + panic!("expected Opaque"); + }; + assert_eq!(opaque.exprs.len(), 1, "arity preserved"); + assert_eq!(opaque.exprs[0], Expression::literal("foo")); + } + + #[test] + fn recursively_rewrites_predicate_child() { + let op = op_with_callbacks("OP"); + let inner = Predicate::lt(col!("col"), lit(5i64)); + let args = [Expression::Predicate(Box::new(inner))]; + + let pred = rewrite(&op, &args, &rewriter(), false).expect("rewrite should succeed"); + let Predicate::Opaque(opaque) = pred else { + panic!("expected Opaque"); + }; + let Expression::Predicate(rewritten) = &opaque.exprs[0] else { + panic!("expected Predicate arg, got {:?}", opaque.exprs[0]); + }; + // The inner `col < 5` rewrites through `eval_partial_cmp` to a stats column ref. + let serialized = format!("{rewritten:?}"); + assert!( + serialized.contains("minValues") || serialized.contains("maxValues"), + "predicate child should reference stats columns; got: {serialized}" + ); + } + + #[test] + fn abstains_on_unsupported_expr_kinds() { + let op = op_with_callbacks("OP"); + let args = [Expression::binary( + BinaryExpressionOp::Plus, + col!("col"), + lit(1i64), + )]; + assert!(rewrite(&op, &args, &rewriter(), false).is_none()); + } + + /// Column with no min/max stats still produces a wrapper -- the engine sees nulls for + /// min/max but can still reason about rowcount (e.g. small-file filtering). + #[test] + fn column_without_stats_falls_back_to_rowcount_only() { + let op = op_with_callbacks("OP"); + let args = [col!("col"), lit(1i64)]; + let pred = + rewrite(&op, &args, &no_stats_rewriter(), false).expect("rewrite should succeed"); + let Predicate::Opaque(opaque) = pred else { + panic!("expected Opaque"); + }; + let Expression::Struct(fields, _) = &opaque.exprs[0] else { + panic!("expected Struct arg"); + }; + assert_eq!( + *fields[0], + Expression::literal(Scalar::Null(DataType::LONG)) + ); + assert_eq!( + *fields[1], + Expression::literal(Scalar::Null(DataType::LONG)) + ); + assert_eq!( + *fields[2], + Expression::literal(Scalar::Null(DataType::LONG)) + ); + assert_eq!(*fields[3], stats_col_numrecords()); + } + + #[test] + fn preserves_arity_for_mixed_args() { + let op = op_with_callbacks("OP"); + let inner = Predicate::lt(col!("y"), lit(5i64)); + let args = [ + col!("col"), + lit(1i64), + Expression::Predicate(Box::new(inner)), + ]; + + let pred = rewrite(&op, &args, &rewriter(), false).expect("rewrite should succeed"); + let Predicate::Opaque(opaque) = pred else { + panic!("expected Opaque"); + }; + assert_eq!( + opaque.exprs.len(), + 3, + "arity should be preserved (3 in, 3 out)" + ); + assert!(matches!(opaque.exprs[0], Expression::Struct(_, _))); + assert!(matches!(opaque.exprs[1], Expression::Literal(_))); + assert!(matches!(opaque.exprs[2], Expression::Predicate(_))); + } + + /// End-to-end: rewrite -> evaluate against a synthesized stats batch -> callback fires. + /// Bare `Predicate::opaque` would fail the adapter downcast and silently skip the callback. + #[test] + fn rewritten_predicate_dispatches_to_engine_callback() { + static CALL_COUNT: AtomicUsize = AtomicUsize::new(0); + + unsafe extern "C" fn engine_count( + _state: *mut c_void, + _op_name: KernelStringSlice, + args_in: ArrowFFIData, + _inverted: bool, + out: *mut EngineExecResult, + ) { + let batch = unsafe { take_ffi_record_batch(args_in) }; + CALL_COUNT.fetch_add(1, AtomicOrdering::SeqCst); + let arr: ArrayRef = Arc::new(BooleanArray::from(vec![true; batch.num_rows()])); + unsafe { write_result(out, arr) } + } + + let op = FfiOpaquePredicateOp::new("OP", callbacks_for(engine_count)); + let args = [col!("col"), lit(5i64)]; + let rewritten = + rewrite(&op, &args, &rewriter(), false).expect("rewrite should succeed"); + + let stats_batch = single_col_int64_stats_batch(); + + CALL_COUNT.store(0, AtomicOrdering::SeqCst); + let result = evaluate_predicate(&rewritten, &stats_batch, false).unwrap(); + assert_eq!(result.len(), 1); + assert!(result.value(0)); + assert_eq!( + CALL_COUNT.load(AtomicOrdering::SeqCst), + 1, + "callback must fire exactly once" + ); + } + + /// Engine inspects slot 2 (nullcount), sees all-null (stub returns None), + /// and falls back to keep-the-file. + #[test] + fn engine_handles_missing_nullcount_via_null_placeholder() { + static SAW_NULLCOUNT: AtomicUsize = AtomicUsize::new(0); + + unsafe extern "C" fn engine_check_nullcount( + _state: *mut c_void, + _op_name: KernelStringSlice, + args_in: ArrowFFIData, + _inverted: bool, + out: *mut EngineExecResult, + ) { + let batch = unsafe { take_ffi_record_batch(args_in) }; + let col = batch + .column(0) + .as_any() + .downcast_ref::() + .expect("arg0 should be StructArray"); + assert_eq!( + col.num_columns(), + 4, + "wrapper must be [min, max, nullcount, rowcount]" + ); + let nullcount = col.column(2); + let usable = (0..nullcount.len()).any(|i| !nullcount.is_null(i)); + if !usable { + SAW_NULLCOUNT.fetch_add(1, AtomicOrdering::SeqCst); + } + // Without nullcount, we can't prune -- keep every file. + let arr: ArrayRef = Arc::new(BooleanArray::from(vec![true; batch.num_rows()])); + unsafe { write_result(out, arr) } + } + + let op = + FfiOpaquePredicateOp::new("NEEDS_NULLCOUNT", callbacks_for(engine_check_nullcount)); + let args = [col!("col"), lit(5i64)]; + let rewritten = + rewrite(&op, &args, &rewriter(), false).expect("rewrite should succeed"); + + // Stats batch carries only min/max; the rewriter's literal-null nullcount/rowcount + // resolve in-expression without needing schema columns for them. + let stats_batch = single_col_int64_stats_batch(); + + SAW_NULLCOUNT.store(0, AtomicOrdering::SeqCst); + let result = evaluate_predicate(&rewritten, &stats_batch, false).unwrap(); + assert!(result.value(0)); + assert_eq!( + SAW_NULLCOUNT.load(AtomicOrdering::SeqCst), + 1, + "engine should observe nullcount as all-null and fall back to keep" + ); + } + + /// A row with no collected stats rewrites to null min/max, and a conservative engine + /// keeps it -- pins the keep-on-null-bounds contract that pruning accuracy relies on. + /// Log-replay safety does not depend on it: kernel's `OR(NOT is_add, ...)` guard keeps + /// Remove rows regardless of the verdict (see + /// `misbehaving_stats_callback_cannot_resurrect_removed_file`). + #[test] + fn null_bounds_keep_file_for_remove_row_safety() { + unsafe extern "C" fn engine_keep_on_null_bounds( + _state: *mut c_void, + _op_name: KernelStringSlice, + args_in: ArrowFFIData, + _inverted: bool, + out: *mut EngineExecResult, + ) { + let batch = unsafe { take_ffi_record_batch(args_in) }; + let stats = batch + .column(0) + .as_any() + .downcast_ref::() + .expect("arg0 should be StructArray"); + let (min, max) = (stats.column(0), stats.column(1)); + let keep: BooleanArray = (0..batch.num_rows()) + // Null bounds can't prove the predicate impossible, so keep the file (true). + .map(|i| Some(min.is_null(i) || max.is_null(i))) + .collect(); + unsafe { write_result(out, Arc::new(keep)) } + } + + let op = FfiOpaquePredicateOp::new("OP", callbacks_for(engine_keep_on_null_bounds)); + let args = [col!("col"), lit(5i64)]; + // `no_stats_rewriter` models a row with no collected stats (e.g. a Remove): null + // min/max. + let rewritten = rewrite(&op, &args, &no_stats_rewriter(), false) + .expect("rewrite should succeed with a type hint"); + let stats_batch = single_col_int64_stats_batch(); + let result = evaluate_predicate(&rewritten, &stats_batch, false).unwrap(); + assert!( + result.value(0), + "row with null bounds must be kept, not pruned" + ); + } + + /// A partition column goes through the same path as a data column: it resolves to its + /// exact value as both min and max, with the uniform `numRecords` rowcount ref (no + /// partition special-case). + #[test] + fn partition_column_rewrites_to_exact_value_bounds() { + let op = op_with_callbacks("OP"); + let args = [col!("part_col"), lit("X")]; + let pred = rewrite(&op, &args, &partition_rewriter(), false) + .expect("rewrite should succeed for partition column"); + let Predicate::Opaque(opaque) = pred else { + panic!("expected Opaque"); + }; + let Expression::Struct(fields, _) = &opaque.exprs[0] else { + panic!("expected Struct arg, got {:?}", opaque.exprs[0]); + }; + assert_eq!(fields.len(), 4); + // Exact partition value serves as both bounds. + assert_eq!(*fields[0], partition_col(&column_name!("part_col"))); + assert_eq!(*fields[1], partition_col(&column_name!("part_col"))); + // Rowcount is the same `numRecords` ref as for data columns. + assert_eq!(*fields[3], stats_col_numrecords()); + } + + #[test] + fn type_hint_picks_first_literal_when_multiple_present() { + let exprs = [col!("c"), lit("s"), lit(1i64)]; + assert_eq!( + super::super::pick_column_type_hint(&exprs), + Some(DataType::STRING), + "the first literal's type wins" + ); + } + + /// A min/max-ineligible column paired with an eligible literal makes the rewrite emit a + /// min/max stats ref the batch doesn't carry. Materializing the args fails, so StatsMode + /// abstains (keeps every file) instead of aborting the scan -- and the engine callback + /// never fires. + #[test] + fn stats_eval_abstains_when_min_max_ref_is_unbacked() { + static CALLS: AtomicUsize = AtomicUsize::new(0); + + unsafe extern "C" fn engine_count( + _state: *mut c_void, + _op_name: KernelStringSlice, + args_in: ArrowFFIData, + _inverted: bool, + out: *mut EngineExecResult, + ) { + let batch = unsafe { take_ffi_record_batch(args_in) }; + CALLS.fetch_add(1, AtomicOrdering::SeqCst); + unsafe { + write_result( + out, + Arc::new(BooleanArray::from(vec![true; batch.num_rows()])), + ) + } + } + + let op = FfiOpaquePredicateOp::new("OP", callbacks_for(engine_count)); + let args = [col!("col"), lit(5i64)]; + let rewritten = + rewrite(&op, &args, &rewriter(), false).expect("rewrite should succeed"); + + // Stats batch carries only numRecords -- no minValues/maxValues -- so materializing the + // rewritten min/max refs fails and StatsMode abstains. + let num_records: ArrayRef = Arc::new(Int64Array::from(vec![1i64])); + let stats_parsed = StructArray::from(vec![( + Arc::new(Field::new("numRecords", ArrowDataType::Int64, true)), + num_records, + )]); + let schema = Arc::new(Schema::new(vec![Field::new( + "stats_parsed", + stats_parsed.data_type().clone(), + true, + )])); + let stats_batch = + RecordBatch::try_new(schema, vec![Arc::new(stats_parsed) as ArrayRef]).unwrap(); + + CALLS.store(0, AtomicOrdering::SeqCst); + let result = evaluate_predicate(&rewritten, &stats_batch, false).unwrap(); + assert!(result.value(0), "abstain keeps the file"); + assert_eq!( + CALLS.load(AtomicOrdering::SeqCst), + 0, + "callback must not fire when the args can't be materialized" + ); + } + } +} diff --git a/ffi/src/expressions/engine_visitor.rs b/ffi/src/expressions/engine_visitor.rs index 61d8db4a50..63179a9e12 100644 --- a/ffi/src/expressions/engine_visitor.rs +++ b/ffi/src/expressions/engine_visitor.rs @@ -1,20 +1,22 @@ //! Defines [`EngineExpressionVisitor`]. This is a visitor that can be used to convert the kernel's //! [`Expression`] or [`Predicate`] to an engine's native expression format. -use crate::expressions::{ - SharedExpression, SharedOpaqueExpressionOp, SharedOpaquePredicateOp, SharedPredicate, -}; -use crate::{handle::Handle, kernel_string_slice, KernelStringSlice, SharedSchema}; +use std::ffi::c_void; use delta_kernel::expressions::{ ArrayData, BinaryExpression, BinaryExpressionOp, BinaryPredicate, BinaryPredicateOp, - ColumnName, Expression, ExpressionRef, JunctionPredicate, JunctionPredicateOp, MapData, - MapToStructExpression, OpaqueExpression, OpaqueExpressionOpRef, OpaquePredicate, - OpaquePredicateOpRef, ParseJsonExpression, Predicate, Scalar, StructData, Transform, + ColumnName, Expression, ExpressionRef, ExpressionStructPatch, JunctionPredicate, + JunctionPredicateOp, MapData, MapToStructExpression, OpaqueExpression, OpaqueExpressionOpRef, + OpaquePredicate, OpaquePredicateOpRef, ParseJsonExpression, Predicate, Scalar, StructData, UnaryExpression, UnaryExpressionOp, UnaryPredicate, UnaryPredicateOp, VariadicExpression, VariadicExpressionOp, }; -use std::ffi::c_void; +use super::kernel_visitor::NullTypeTag; +use crate::expressions::{ + SharedExpression, SharedOpaqueExpressionOp, SharedOpaquePredicateOp, SharedPredicate, +}; +use crate::handle::Handle; +use crate::{kernel_string_slice, KernelStringSlice, SharedSchema}; type VisitLiteralFn = extern "C" fn(data: *mut c_void, sibling_list_id: usize, value: T); type VisitUnaryFn = extern "C" fn(data: *mut c_void, sibling_list_id: usize, child_list_id: usize); @@ -39,10 +41,10 @@ type VisitParseJsonFn = extern "C" fn( /// future. /// /// Every expression the kernel visits belongs to some list of "sibling" elements. The schema -/// itself is a list of schema elements, and every complex type (struct expression, array, junction, etc) -/// contains a list of "child" elements. -/// 1. Before visiting any complex expression type, the kernel asks the engine to allocate a list to -/// hold its children +/// itself is a list of schema elements, and every complex type (struct expression, array, junction, +/// etc) contains a list of "child" elements. +/// 1. Before visiting any complex expression type, the kernel asks the engine to allocate a list +/// to hold its children /// 2. When visiting any expression element, the kernel passes its parent's "child list" as the /// "sibling list" the element should be appended to: /// - For a struct literal, first visit each struct field and visit each value @@ -51,15 +53,15 @@ type VisitParseJsonFn = extern "C" fn( /// - For a junction `and` or `or` expression, visit each sub-expression. /// - For a binary operator expression, visit the left and right operands. /// - For a unary `is null` or `not` expression, visit the sub-expression. -/// 3. When visiting a complex expression, the kernel also passes the "child list" containing -/// that element's (already-visited) children. +/// 3. When visiting a complex expression, the kernel also passes the "child list" containing that +/// element's (already-visited) children. /// 4. The [`visit_expression`] method returns the id of the list of top-level columns /// /// WARNING: The visitor MUST NOT retain internal references to string slices or binary data passed /// to visitor methods -/// TODO: Visit type information in struct field and null. This will likely involve using the schema -/// visitor. Note that struct literals are currently in flux, and may change significantly. Here is -/// the relevant issue: +/// TODO: Visit type information in struct field. This will likely involve using the schema visitor. +/// Note that struct literals are currently in flux, and may change significantly. Here is the +/// relevant issue: #[repr(C)] pub struct EngineExpressionVisitor { /// An opaque engine state pointer @@ -88,9 +90,13 @@ pub struct EngineExpressionVisitor { /// Visit a 64bit timestamp belonging to the list identified by `sibling_list_id`. /// The timestamp is microsecond precision with no timezone. pub visit_literal_timestamp_ntz: VisitLiteralFn, - /// Visit a 32bit integer `date` representing days since UNIX epoch 1970-01-01. The `date` belongs - /// to the list identified by `sibling_list_id`. + /// Visit a 32bit integer `date` representing days since UNIX epoch 1970-01-01. The `date` + /// belongs to the list identified by `sibling_list_id`. pub visit_literal_date: VisitLiteralFn, + /// Visit an `interval year to month` literal as a signed month count. + pub visit_literal_interval_year_month: VisitLiteralFn, + /// Visit an `interval day to second` literal as a signed microsecond count. + pub visit_literal_interval_day_time: VisitLiteralFn, /// Visit binary data at the `buffer` with length `len` belonging to the list identified by /// `sibling_list_id`. pub visit_literal_binary: @@ -128,8 +134,18 @@ pub struct EngineExpressionVisitor { key_list_id: usize, value_list_id: usize, ), - /// Visits a null value belonging to the list identified by `sibling_list_id. - pub visit_literal_null: extern "C" fn(data: *mut c_void, sibling_list_id: usize), + /// Visits a typed null value belonging to the list identified by `sibling_list_id`. + /// + /// The `type_tag` identifies reconstructible primitive types with tags 0-14. Decimal nulls + /// use tag 12 with `precision` and `scale`; interval year-month and day-time nulls use tags + /// 13 and 14. Non-primitive types and `void` use the sentinel tag 255. + pub visit_literal_null: extern "C" fn( + data: *mut c_void, + sibling_list_id: usize, + type_tag: u8, + precision: u8, + scale: u8, + ), /// Visits an `and` expression belonging to the list identified by `sibling_list_id`. /// The sub-expressions of the array are in a list identified by `child_list_id` pub visit_and: VisitJunctionFn, @@ -143,27 +159,33 @@ pub struct EngineExpressionVisitor { /// The sub-expression will be in a _one_ item list identified by `child_list_id` pub visit_is_null: VisitUnaryFn, /// Visits the `ToJson` unary operator belonging to the list identified by `sibling_list_id`. - /// The sub-expression will be in a _one_ item list identified by `child_list_id` + /// The sub-expression will be in a _one_ item list identified by `child_list_id`. + /// See [`UnaryExpressionOp::ToJson`] for the encoding the implementation must produce; in + /// particular, timestamps carry exactly three fractional digits, truncated. pub visit_to_json: VisitUnaryFn, /// Visits the `ParseJson` expression belonging to the list identified by `sibling_list_id`. - /// The sub-expression (JSON string) will be in a _one_ item list identified by `child_list_id`. - /// The `output_schema` handle specifies the schema to parse the JSON into. + /// The sub-expression (JSON string) will be in a _one_ item list identified by + /// `child_list_id`. The `output_schema` handle specifies the schema to parse the JSON + /// into. pub visit_parse_json: VisitParseJsonFn, /// Visits the `MapToStruct` expression belonging to the list identified by `sibling_list_id`. /// The sub-expression (map column) will be in a _one_ item list identified by `child_list_id`. /// The output struct schema is determined by the evaluator's result type. pub visit_map_to_struct: VisitUnaryFn, - /// Visits the `LessThan` binary operator belonging to the list identified by `sibling_list_id`. - /// The operands will be in a _two_ item list identified by `child_list_id` + /// Visits the `LessThan` binary operator belonging to the list identified by + /// `sibling_list_id`. The operands will be in a _two_ item list identified by + /// `child_list_id` pub visit_lt: VisitBinaryFn, - /// Visits the `GreaterThan` binary operator belonging to the list identified by `sibling_list_id`. - /// The operands will be in a _two_ item list identified by `child_list_id` + /// Visits the `GreaterThan` binary operator belonging to the list identified by + /// `sibling_list_id`. The operands will be in a _two_ item list identified by + /// `child_list_id` pub visit_gt: VisitBinaryFn, /// Visits the `Equal` binary operator belonging to the list identified by `sibling_list_id`. /// The operands will be in a _two_ item list identified by `child_list_id` pub visit_eq: VisitBinaryFn, - /// Visits the `Distinct` binary operator belonging to the list identified by `sibling_list_id`. - /// The operands will be in a _two_ item list identified by `child_list_id` + /// Visits the `Distinct` binary operator belonging to the list identified by + /// `sibling_list_id`. The operands will be in a _two_ item list identified by + /// `child_list_id` pub visit_distinct: VisitBinaryFn, /// Visits the `In` binary operator belonging to the list identified by `sibling_list_id`. /// The operands will be in a _two_ item list identified by `child_list_id` @@ -174,15 +196,20 @@ pub struct EngineExpressionVisitor { /// Visits the `Minus` binary operator belonging to the list identified by `sibling_list_id`. /// The operands will be in a _two_ item list identified by `child_list_id` pub visit_minus: VisitBinaryFn, - /// Visits the `Multiply` binary operator belonging to the list identified by `sibling_list_id`. - /// The operands will be in a _two_ item list identified by `child_list_id` + /// Visits the `Multiply` binary operator belonging to the list identified by + /// `sibling_list_id`. The operands will be in a _two_ item list identified by + /// `child_list_id` pub visit_multiply: VisitBinaryFn, /// Visits the `Divide` binary operator belonging to the list identified by `sibling_list_id`. /// The operands will be in a _two_ item list identified by `child_list_id` pub visit_divide: VisitBinaryFn, - /// Visits the `Coalesce` variadic operator belonging to the list identified by `sibling_list_id`. - /// The operands will be in a list identified by `child_list_id` + /// Visits the `Coalesce` variadic operator belonging to the list identified by + /// `sibling_list_id`. The operands will be in a list identified by `child_list_id` pub visit_coalesce: VisitVariadicFn, + /// Visits the `Array` variadic constructor belonging to the list identified by + /// `sibling_list_id`. The element expressions will be in a list identified by + /// `child_list_id`. + pub visit_array: VisitVariadicFn, /// Visits the `column` belonging to the list identified by `sibling_list_id`. pub visit_column: extern "C" fn(data: *mut c_void, sibling_list_id: usize, name: KernelStringSlice), @@ -190,48 +217,35 @@ pub struct EngineExpressionVisitor { /// The sub-expressions (fields) of the struct are in a list identified by `child_list_id` pub visit_struct_expr: extern "C" fn(data: *mut c_void, sibling_list_id: usize, child_list_id: usize), - /// Visits a `Transform` expression belonging to the list identified by `sibling_list_id`. The - /// `input_path_list_id` is a single-item list containing transform's input path as a column - /// reference (0 = no path). The `field_transform_list_id` identifies the list of field - /// transforms to apply (0 = identity transform). See also [`Self::visit_field_transform`]. - pub visit_transform_expr: extern "C" fn( + /// Visits a `StructPatch` expression belonging to the list identified by `sibling_list_id`. + /// The `input_path_list_id` is a zero-or-one item list containing the patch's input path as a + /// column reference. The `prepended_field_list_id` and `appended_field_list_id` identify + /// expression lists to emit before and after the named input fields. The + /// `field_patch_list_id` identifies the list of named field patches to apply. See also + /// [`Self::visit_field_patch`]. + pub visit_struct_patch_expr: extern "C" fn( data: *mut c_void, sibling_list_id: usize, input_path_list_id: usize, - field_transform_list_id: usize, + prepended_field_list_id: usize, + field_patch_list_id: usize, + appended_field_list_id: usize, ), - /// Visits one field transform of a `Transform` expression that owns the list identified by - /// `sibling_list_id`. Each field transform has a different insertion point (no duplicates). - /// - /// A field transform is modeled as the triple `(field_name, expr_list, is_replace)`, as - /// described by the truth table below. The `expr_list_id` identifies the list of expressions - /// the field transform should emit. The field name (if present) always references a field of - /// the input struct. Both the field name and the expression list are optional: - /// - /// |field_name? |expr_list? |is_replace? |meaning| - /// |-|-|-|-| - /// | NO | NO | * | NO-OP (prepend an empty list of expressions to the output) - /// | NO | YES | * | Prepend a list of expressions to the output - /// | YES | NO | NO | NO-OP (insert an empty list of expressions after the named input field) - /// | YES | NO | YES | Drop the named input field - /// | YES | YES | NO | Insert a list of expressions after the named input field - /// | YES | YES | YES | Replace the named input field with a list of expressions - /// - /// NOTE: Treating list id 0 as an empty list yields a simplified truth table: - /// - /// |field_name? |is_replace? |meaning| - /// |-|-|-| - /// | NO | * | Prepend a (possibly empty) list of expressions to the output - /// | YES | NO | Insert a (possibly empty) list of expressions after the named input field - /// | YES | YES | Replace the named input field with a (possibly empty) list of expressions + /// Visits one named field patch of a `StructPatch` expression that owns the list identified by + /// `sibling_list_id`. /// - /// NOTE: The expressions of each field transform must be emitted in order at the insertion point. - pub visit_field_transform: extern "C" fn( + /// The `insertion_expr_list_id` identifies expressions to emit after this field's output + /// position. If `keep_input` is true, the original input field is emitted before these + /// insertions. If `keep_input` is false, the original input field is omitted and the first + /// insertion, if present, occupies the input field's output position. The `optional` flag + /// indicates that the patch is silently ignored when the input field does not exist. + pub visit_field_patch: extern "C" fn( data: *mut c_void, sibling_list_id: usize, - field_name: *const KernelStringSlice, - expr_list_id: usize, - is_replace: bool, + field_name: KernelStringSlice, + insertion_expr_list_id: usize, + keep_input: bool, + optional: bool, ), /// Visits the operator (`op`) and children (`child_list_id`) of an opaque expression belonging /// to the list identified by `sibling_list_id`. @@ -249,8 +263,10 @@ pub struct EngineExpressionVisitor { op: Handle, child_list_id: usize, ), - /// Visits the name of an `Expression::Unknown` or `Predicate::Unknown` belonging to the - /// list identified by `sibling_list_id`. + /// Visits a node represented as unknown across FFI, belonging to `sibling_list_id`. + /// + /// Casts use `cast_to_`, or `cast_with_timestamp_timezone_to_` when + /// configured. pub visit_unknown: extern "C" fn(data: *mut c_void, sibling_list_id: usize, name: KernelStringSlice), } @@ -402,77 +418,58 @@ fn visit_expression_struct( exprs: &[ExpressionRef], sibling_list_id: usize, ) { + let child_list_id = visit_expression_list(visitor, exprs); + call!(visitor, visit_struct_expr, sibling_list_id, child_list_id) +} + +fn visit_expression_list(visitor: &mut EngineExpressionVisitor, exprs: &[ExpressionRef]) -> usize { let child_list_id = call!(visitor, make_field_list, exprs.len()); for expr in exprs { visit_expression_impl(visitor, expr, child_list_id); } - call!(visitor, visit_struct_expr, sibling_list_id, child_list_id) + child_list_id } -fn visit_expression_transform( +fn visit_expression_struct_patch( visitor: &mut EngineExpressionVisitor, - transform: &Transform, + patch: &ExpressionStructPatch, sibling_list_id: usize, ) { - let Transform { - input_path, - field_transforms, - prepended_fields, - } = transform; - - // Treat the input path like a column expression - let mut path_list_id = 0; - if let Some(ref column_name) = input_path { - path_list_id = call!(visitor, make_field_list, 1); + // Treat the input path like a zero-or-one column expression list. + let path_len = usize::from(patch.input_path.is_some()); + let path_list_id = call!(visitor, make_field_list, path_len); + if let Some(ref column_name) = patch.input_path { visit_expression_column(visitor, column_name, path_list_id); }; - // Emit one field transform for each named input field (ignoring no-ops), plus one more for the - // prepended field list (if any). - let prepended_count = if prepended_fields.is_empty() { 0 } else { 1 }; - let field_transform_count = prepended_count + field_transforms.len(); - let field_transform_list_id = call!(visitor, make_field_list, field_transform_count); + let prepended_field_list_id = visit_expression_list(visitor, &patch.prepended_fields); + let appended_field_list_id = visit_expression_list(visitor, &patch.appended_fields); - // Process the prepend first (if any) - if !prepended_fields.is_empty() { - let child_list_id = call!(visitor, make_field_list, prepended_fields.len()); - for expr in prepended_fields { - visit_expression_impl(visitor, expr, child_list_id); - } + // Process each named field patch in turn. Field patch order is not semantically meaningful; + // engines should apply field patches according to input schema order. + let field_patch_list_id = call!(visitor, make_field_list, patch.field_patches.len()); + for (field_name, field_patch) in &patch.field_patches { + let insertion_expr_list_id = visit_expression_list(visitor, &field_patch.insertions); call!( visitor, - visit_field_transform, - field_transform_list_id, - std::ptr::null(), - child_list_id, - false // doesn't matter, no field name + visit_field_patch, + field_patch_list_id, + kernel_string_slice!(field_name), + insertion_expr_list_id, + field_patch.keep_input, + field_patch.optional ); } - // Process each field transform in turn - for (field_name, field_transform) in field_transforms { - let child_list_id = call!(visitor, make_field_list, field_transform.exprs.len()); - for expr in &field_transform.exprs { - visit_expression_impl(visitor, expr, child_list_id); - } - - call!( - visitor, - visit_field_transform, - field_transform_list_id, - &kernel_string_slice!(field_name), - child_list_id, - field_transform.is_replace - ); - } - - // Attach the field transforms to the parent transform + // Attach the field patches to the parent struct patch. call!( visitor, - visit_transform_expr, + visit_struct_patch_expr, sibling_list_id, path_list_id, - field_transform_list_id + prepended_field_list_id, + field_patch_list_id, + appended_field_list_id ); } @@ -587,7 +584,33 @@ fn visit_expression_scalar( v.scale() ) } - Scalar::Null(_) => call!(visitor, visit_literal_null, sibling_list_id), + Scalar::Null(data_type) => { + let (tag, precision, scale) = NullTypeTag::from_data_type(data_type); + call!( + visitor, + visit_literal_null, + sibling_list_id, + tag as u8, + precision, + scale + ) + } + Scalar::IntervalYearMonth(val) => { + call!( + visitor, + visit_literal_interval_year_month, + sibling_list_id, + *val + ) + } + Scalar::IntervalDayTime(val) => { + call!( + visitor, + visit_literal_interval_day_time, + sibling_list_id, + *val + ) + } Scalar::Struct(struct_data) => { visit_expression_struct_literal(visitor, struct_data, sibling_list_id) } @@ -605,8 +628,8 @@ fn visit_expression_impl( Expression::Literal(scalar) => visit_expression_scalar(visitor, scalar, sibling_list_id), Expression::Column(name) => visit_expression_column(visitor, name, sibling_list_id), Expression::Struct(exprs, _) => visit_expression_struct(visitor, exprs, sibling_list_id), - Expression::Transform(transform) => { - visit_expression_transform(visitor, transform, sibling_list_id) + Expression::StructPatch(patch) => { + visit_expression_struct_patch(visitor, patch, sibling_list_id) } Expression::Predicate(pred) => visit_predicate_impl(visitor, pred, sibling_list_id), Expression::Unary(UnaryExpression { op, expr }) => { @@ -636,6 +659,7 @@ fn visit_expression_impl( } let visit_fn = match op { VariadicExpressionOp::Coalesce => visitor.visit_coalesce, + VariadicExpressionOp::Array => visitor.visit_array, }; visit_fn(visitor.data, sibling_list_id, child_list_id); } @@ -657,11 +681,34 @@ fn visit_expression_impl( schema_handle ); } - Expression::MapToStruct(MapToStructExpression { map_expr }) => { + Expression::MapToStruct(map_to_struct) + if map_to_struct.options.timestamp_timezone().is_some() => + { + visit_unknown( + visitor, + sibling_list_id, + "map_to_struct_with_timestamp_timezone", + ) + } + Expression::MapToStruct(MapToStructExpression { map_expr, .. }) => { let child_list_id = call!(visitor, make_field_list, 1); visit_expression_impl(visitor, map_expr, child_list_id); call!(visitor, visit_map_to_struct, sibling_list_id, child_list_id); } + // The FFI expression visitor has no ElementAt callback. Surface an explicit unknown node + // so engines cannot silently interpret it as another expression. + Expression::ElementAt(_) => visit_unknown(visitor, sibling_list_id, "element_at"), + Expression::Cast(cast) if cast.options.timestamp_timezone().is_some() => visit_unknown( + visitor, + sibling_list_id, + &format!("cast_with_timestamp_timezone_to_{}", cast.target), + ), + // TODO(#2975): Add a dedicated visitor callback for cast expressions. + Expression::Cast(cast) => visit_unknown( + visitor, + sibling_list_id, + &format!("cast_to_{}", cast.target), + ), Expression::Unknown(name) => visit_unknown(visitor, sibling_list_id, name), } } @@ -723,3 +770,280 @@ fn visit_predicate_internal(predicate: &Predicate, visitor: &mut EngineExpressio visit_predicate_impl(visitor, predicate, top_level); top_level } + +#[cfg(test)] +mod tests { + use delta_kernel::expressions::{CastOptions, Expression, MapToStructOptions, Scalar}; + use delta_kernel::schema::DataType; + use rstest::rstest; + + use super::*; + use crate::TryFromStringSlice; + + #[derive(Debug, PartialEq, Eq)] + enum LiteralEvent { + IntervalYearMonth { + sibling_list_id: usize, + value: i32, + }, + IntervalDayTime { + sibling_list_id: usize, + value: i64, + }, + Unknown { + sibling_list_id: usize, + name: String, + }, + } + + #[derive(Default)] + struct TestExpressionBuilder { + next_list_id: usize, + events: Vec, + } + + extern "C" fn make_field_list(data: *mut c_void, _reserve: usize) -> usize { + let builder = unsafe { &mut *(data as *mut TestExpressionBuilder) }; + let list_id = builder.next_list_id; + builder.next_list_id += 1; + list_id + } + + extern "C" fn visit_literal_interval_year_month( + data: *mut c_void, + sibling_list_id: usize, + value: i32, + ) { + let builder = unsafe { &mut *(data as *mut TestExpressionBuilder) }; + builder.events.push(LiteralEvent::IntervalYearMonth { + sibling_list_id, + value, + }); + } + + extern "C" fn visit_literal_interval_day_time( + data: *mut c_void, + sibling_list_id: usize, + value: i64, + ) { + let builder = unsafe { &mut *(data as *mut TestExpressionBuilder) }; + builder.events.push(LiteralEvent::IntervalDayTime { + sibling_list_id, + value, + }); + } + + extern "C" fn visit_unknown_name( + data: *mut c_void, + sibling_list_id: usize, + name: KernelStringSlice, + ) { + let builder = unsafe { &mut *(data as *mut TestExpressionBuilder) }; + let name = unsafe { String::try_from_slice(&name) }.unwrap(); + builder.events.push(LiteralEvent::Unknown { + sibling_list_id, + name, + }); + } + + macro_rules! ignore_fn { + ($fn_name:ident $(, $arg_type:ty)*) => { + extern "C" fn $fn_name( + _data: *mut c_void, + _sibling_list_id: usize, + $(_: $arg_type),* + ) { + } + }; + } + + ignore_fn!(ignore_i32, i32); + ignore_fn!(ignore_i64, i64); + ignore_fn!(ignore_i16, i16); + ignore_fn!(ignore_i8, i8); + ignore_fn!(ignore_f32, f32); + ignore_fn!(ignore_f64, f64); + ignore_fn!(ignore_bool, bool); + ignore_fn!(ignore_string_slice, KernelStringSlice); + ignore_fn!(ignore_binary, *const u8, usize); + ignore_fn!(ignore_decimal, i64, u64, u8, u8); + ignore_fn!(ignore_struct_literal, usize, usize); + ignore_fn!(ignore_child_list, usize); + ignore_fn!(ignore_map_literal, usize, usize); + ignore_fn!(ignore_null, u8, u8, u8); + ignore_fn!(ignore_parse_json, usize, Handle); + ignore_fn!(ignore_column, KernelStringSlice); + ignore_fn!(ignore_struct_patch, usize, usize, usize, usize); + ignore_fn!(ignore_field_patch, KernelStringSlice, usize, bool, bool); + ignore_fn!(ignore_opaque_expr, Handle, usize); + ignore_fn!(ignore_opaque_pred, Handle, usize); + + fn test_visitor(builder: &mut TestExpressionBuilder) -> EngineExpressionVisitor { + EngineExpressionVisitor { + data: builder as *mut _ as *mut c_void, + make_field_list, + visit_literal_int: ignore_i32, + visit_literal_long: ignore_i64, + visit_literal_short: ignore_i16, + visit_literal_byte: ignore_i8, + visit_literal_float: ignore_f32, + visit_literal_double: ignore_f64, + visit_literal_string: ignore_string_slice, + visit_literal_bool: ignore_bool, + visit_literal_timestamp: ignore_i64, + visit_literal_timestamp_ntz: ignore_i64, + visit_literal_date: ignore_i32, + visit_literal_interval_year_month, + visit_literal_interval_day_time, + visit_literal_binary: ignore_binary, + visit_literal_decimal: ignore_decimal, + visit_literal_struct: ignore_struct_literal, + visit_literal_array: ignore_child_list, + visit_literal_map: ignore_map_literal, + visit_literal_null: ignore_null, + visit_and: ignore_child_list, + visit_or: ignore_child_list, + visit_not: ignore_child_list, + visit_is_null: ignore_child_list, + visit_to_json: ignore_child_list, + visit_parse_json: ignore_parse_json, + visit_map_to_struct: ignore_child_list, + visit_lt: ignore_child_list, + visit_gt: ignore_child_list, + visit_eq: ignore_child_list, + visit_distinct: ignore_child_list, + visit_in: ignore_child_list, + visit_add: ignore_child_list, + visit_minus: ignore_child_list, + visit_multiply: ignore_child_list, + visit_divide: ignore_child_list, + visit_coalesce: ignore_child_list, + visit_array: ignore_child_list, + visit_column: ignore_column, + visit_struct_expr: ignore_child_list, + visit_struct_patch_expr: ignore_struct_patch, + visit_field_patch: ignore_field_patch, + visit_opaque_expr: ignore_opaque_expr, + visit_opaque_pred: ignore_opaque_pred, + visit_unknown: visit_unknown_name, + } + } + + #[rstest] + #[case( + Expression::literal(Scalar::IntervalYearMonth(26)), + LiteralEvent::IntervalYearMonth { sibling_list_id: 0, value: 26 } + )] + #[case( + Expression::literal(Scalar::IntervalDayTime(987_654)), + LiteralEvent::IntervalDayTime { sibling_list_id: 0, value: 987_654 } + )] + #[case( + Expression::literal(Scalar::IntervalYearMonth(-13)), + LiteralEvent::IntervalYearMonth { sibling_list_id: 0, value: -13 } + )] + #[case( + Expression::literal(Scalar::IntervalYearMonth(i32::MIN)), + LiteralEvent::IntervalYearMonth { sibling_list_id: 0, value: i32::MIN } + )] + #[case( + Expression::literal(Scalar::IntervalYearMonth(i32::MAX)), + LiteralEvent::IntervalYearMonth { sibling_list_id: 0, value: i32::MAX } + )] + #[case( + Expression::literal(Scalar::IntervalDayTime(-86_400_000_000)), + LiteralEvent::IntervalDayTime { sibling_list_id: 0, value: -86_400_000_000 } + )] + #[case( + Expression::literal(Scalar::IntervalDayTime(i64::MIN)), + LiteralEvent::IntervalDayTime { sibling_list_id: 0, value: i64::MIN } + )] + #[case( + Expression::literal(Scalar::IntervalDayTime(i64::MAX)), + LiteralEvent::IntervalDayTime { sibling_list_id: 0, value: i64::MAX } + )] + fn visit_expression_uses_interval_literal_callbacks( + #[case] expression: Expression, + #[case] expected: LiteralEvent, + ) { + let mut builder = TestExpressionBuilder::default(); + let mut visitor = test_visitor(&mut builder); + + let top_level_id = visit_expression_internal(&expression, &mut visitor); + + assert_eq!(top_level_id, 0); + assert_eq!(builder.events, vec![expected]); + } + + #[test] + fn timezone_aware_map_to_struct_visits_unknown() { + let expression = Expression::map_to_struct( + Expression::column(["partitionValues"]), + MapToStructOptions::default().with_timestamp_timezone("America/Los_Angeles"), + ); + let mut builder = TestExpressionBuilder::default(); + let mut visitor = test_visitor(&mut builder); + + let top_level_id = visit_expression_internal(&expression, &mut visitor); + + assert_eq!(top_level_id, 0); + assert_eq!( + builder.events, + vec![LiteralEvent::Unknown { + sibling_list_id: 0, + name: "map_to_struct_with_timestamp_timezone".to_string(), + }] + ); + } + + #[rstest] + #[case::plain(None, "cast_to_timestamp")] + #[case::timezone( + Some("America/Los_Angeles"), + "cast_with_timestamp_timezone_to_timestamp" + )] + fn cast_visits_unknown(#[case] timezone: Option<&str>, #[case] expected_name: &str) { + let options = timezone.map_or_else(CastOptions::default, |timezone| { + CastOptions::default().with_timestamp_timezone(timezone) + }); + let expression = Expression::cast( + Expression::column(["partitionValue"]), + DataType::TIMESTAMP, + options, + ); + let mut builder = TestExpressionBuilder::default(); + let mut visitor = test_visitor(&mut builder); + + let top_level_id = visit_expression_internal(&expression, &mut visitor); + + assert_eq!(top_level_id, 0); + assert_eq!( + builder.events, + vec![LiteralEvent::Unknown { + sibling_list_id: 0, + name: expected_name.to_string(), + }] + ); + } + + #[test] + fn element_at_visits_unknown() { + let expression = Expression::element_at( + Expression::column(["partitionValues"]), + Expression::literal("key"), + ); + let mut builder = TestExpressionBuilder::default(); + let mut visitor = test_visitor(&mut builder); + + let top_level_id = visit_expression_internal(&expression, &mut visitor); + + assert_eq!(top_level_id, 0); + assert_eq!( + builder.events, + vec![LiteralEvent::Unknown { + sibling_list_id: 0, + name: "element_at".to_string(), + }] + ); + } +} diff --git a/ffi/src/expressions/kernel_visitor.rs b/ffi/src/expressions/kernel_visitor.rs index 5102092032..8d7d5b4c49 100644 --- a/ffi/src/expressions/kernel_visitor.rs +++ b/ffi/src/expressions/kernel_visitor.rs @@ -2,12 +2,19 @@ //! engine's native expressions into kernel's [`Expression`] and [`Predicate`] types. use std::sync::Arc; +#[cfg(feature = "default-engine-base")] +use delta_kernel::engine::arrow_expression::opaque::ArrowOpaquePredicate; use delta_kernel::expressions::{ - BinaryExpressionOp, BinaryPredicateOp, ColumnName, Expression, Predicate, Scalar, - UnaryPredicateOp, + BinaryExpressionOp, BinaryPredicateOp, ColumnName, Expression, JunctionPredicateOp, + MapToStructOptions, Predicate, Scalar, UnaryPredicateOp, }; +use delta_kernel::schema::{DataType, PrimitiveType}; use delta_kernel::DeltaResult; +#[cfg(feature = "default-engine-base")] +use crate::expressions::opaque_eval::{COpaqueEvalCallbacks, FfiOpaqueEvalCallbacks}; +#[cfg(feature = "default-engine-base")] +use crate::expressions::FfiOpaquePredicateOp; use crate::expressions::{SharedExpression, SharedPredicate}; use crate::handle::Handle; use crate::scan::{EngineExpression, EnginePredicate}; @@ -346,7 +353,8 @@ pub extern "C" fn visit_expression_literal_timestamp( wrap_expression(state, Expression::literal(Scalar::Timestamp(value))) } -/// visit a timestamp_ntz literal expression 'value' (i64 representing microseconds since unix epoch) +/// visit a timestamp_ntz literal expression 'value' (i64 representing microseconds since unix +/// epoch) #[no_mangle] pub extern "C" fn visit_expression_literal_timestamp_ntz( state: &mut KernelExpressionVisitorState, @@ -355,6 +363,24 @@ pub extern "C" fn visit_expression_literal_timestamp_ntz( wrap_expression(state, Expression::literal(Scalar::TimestampNtz(value))) } +/// Visit an interval year-month literal (signed month count). +#[no_mangle] +pub extern "C" fn visit_expression_literal_interval_year_month( + state: &mut KernelExpressionVisitorState, + value: i32, +) -> usize { + wrap_expression(state, Expression::literal(Scalar::IntervalYearMonth(value))) +} + +/// Visit an interval day-time literal (signed microsecond count). +#[no_mangle] +pub extern "C" fn visit_expression_literal_interval_day_time( + state: &mut KernelExpressionVisitorState, + value: i64, +) -> usize { + wrap_expression(state, Expression::literal(Scalar::IntervalDayTime(value))) +} + /// visit a binary literal expression /// /// # Safety @@ -401,19 +427,200 @@ fn visit_expression_literal_decimal_impl( Ok(wrap_expression(state, Expression::literal(decimal))) } -/// Visit a null literal expression. +/// Type tag for null literal construction via FFI. Identifies the data type of a typed null. /// -/// Returns an error because NULL literal reconstruction is not supported - type information -/// is lost when converting from kernel to engine format, so we cannot faithfully reconstruct -/// the original NULL literal. +/// Most primitive types use fixed discriminants 0-14. Decimal uses 12 and requires additional +/// precision/scale parameters. Non-primitive types (struct, array, map, variant) and the `void` +/// primitive use the [`NonPrimitive`](Self::NonPrimitive) sentinel because they cannot be +/// reconstructed from a type tag alone. +/// +/// NOTE: These values are part of the FFI contract. Changing existing discriminants is a breaking +/// change. +#[repr(u8)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum NullTypeTag { + /// Null of type `boolean`. + Boolean = 0, + /// Null of type `byte` (8-bit signed integer). + Byte = 1, + /// Null of type `short` (16-bit signed integer). + Short = 2, + /// Null of type `integer` (32-bit signed integer). + Integer = 3, + /// Null of type `long` (64-bit signed integer). + Long = 4, + /// Null of type `float` (32-bit IEEE 754). + Float = 5, + /// Null of type `double` (64-bit IEEE 754). + Double = 6, + /// Null of type `string`. + String = 7, + /// Null of type `binary`. + Binary = 8, + /// Null of type `date` (days since epoch). + Date = 9, + /// Null of type `timestamp` (microseconds since epoch, UTC-adjusted). + Timestamp = 10, + /// Null of type `timestamp_ntz` (microseconds since epoch, no timezone). + TimestampNtz = 11, + /// Null of type `decimal`. Requires valid `precision` and `scale` parameters. + /// + /// WARNING: This variant MUST remain `= 12`. It is the only tag with special handling + /// (precision/scale parameters), and C consumers key on the value `12` directly. + Decimal = 12, + /// Null of type `interval year to month` (signed month count). + IntervalYearMonth = 13, + /// Null of type `interval day to second` (signed microsecond duration). + IntervalDayTime = 14, + /// Sentinel for non-primitive null types (struct, array, map, variant) and void. Emitted by + /// the kernel-to-engine visitor when the null's type cannot be reconstructed from a compact + /// tag. Engines that receive this tag should use opaque expressions or a schema visitor to + /// obtain full type details. + /// + /// Passing this tag to [`visit_expression_literal_null`] returns an error because the + /// original complex type cannot be reconstructed from a tag alone. + NonPrimitive = 255, +} + +impl TryFrom for NullTypeTag { + type Error = delta_kernel::Error; + + fn try_from(value: u8) -> Result { + match value { + 0 => Ok(Self::Boolean), + 1 => Ok(Self::Byte), + 2 => Ok(Self::Short), + 3 => Ok(Self::Integer), + 4 => Ok(Self::Long), + 5 => Ok(Self::Float), + 6 => Ok(Self::Double), + 7 => Ok(Self::String), + 8 => Ok(Self::Binary), + 9 => Ok(Self::Date), + 10 => Ok(Self::Timestamp), + 11 => Ok(Self::TimestampNtz), + 12 => Ok(Self::Decimal), + 13 => Ok(Self::IntervalYearMonth), + 14 => Ok(Self::IntervalDayTime), + 255 => Ok(Self::NonPrimitive), + other => Err(delta_kernel::Error::generic(format!( + "Unrecognized null type tag: {other}" + ))), + } + } +} + +impl NullTypeTag { + /// Convert a [`DataType`] to its null type tag and decimal parameters. + /// + /// Returns `(tag, precision, scale)`. The `precision` and `scale` are only meaningful when + /// `tag` is [`NullTypeTag::Decimal`]; they are zero for all other types. + pub(crate) fn from_data_type(data_type: &DataType) -> (Self, u8, u8) { + match data_type { + DataType::Primitive(p) => match p { + PrimitiveType::Boolean => (Self::Boolean, 0, 0), + PrimitiveType::Byte => (Self::Byte, 0, 0), + PrimitiveType::Short => (Self::Short, 0, 0), + PrimitiveType::Integer => (Self::Integer, 0, 0), + PrimitiveType::Long => (Self::Long, 0, 0), + PrimitiveType::Float => (Self::Float, 0, 0), + PrimitiveType::Double => (Self::Double, 0, 0), + PrimitiveType::String => (Self::String, 0, 0), + PrimitiveType::Binary => (Self::Binary, 0, 0), + PrimitiveType::Date => (Self::Date, 0, 0), + PrimitiveType::Timestamp => (Self::Timestamp, 0, 0), + PrimitiveType::TimestampNtz => (Self::TimestampNtz, 0, 0), + PrimitiveType::IntervalYearMonth => (Self::IntervalYearMonth, 0, 0), + PrimitiveType::IntervalDayTime => (Self::IntervalDayTime, 0, 0), + PrimitiveType::Decimal(dt) => (Self::Decimal, dt.precision(), dt.scale()), + // Void has no dedicated FFI tag. The current predicate-construction path is + // not expected to produce a void-typed literal null; if one ever reaches this + // code we want it represented as a non-primitive null rather than a missing + // case, so this arm is a defensive fallback rather than part of the normal + // void-column path. + PrimitiveType::Void => (Self::NonPrimitive, 0, 0), + // A geometry/geography's coordinate reference system string doesn't fit the + // (tag, u8, u8) payload, so map to the NonPrimitive sentinel. See #2949 for + // real geo FFI support. + #[cfg(feature = "geo-type-in-dev")] + PrimitiveType::Geometry(_) | PrimitiveType::Geography(_) => { + (Self::NonPrimitive, 0, 0) + } + }, + _ => (Self::NonPrimitive, 0, 0), + } + } + + /// Convert a null type tag back to a [`DataType`]. + /// + /// For [`Decimal`](Self::Decimal), `precision` and `scale` specify the decimal parameters. + /// For all other primitive types, `precision` and `scale` are ignored (callers should + /// pass 0). + /// + /// Returns an error for [`NonPrimitive`](Self::NonPrimitive) since complex types cannot be + /// reconstructed from a type tag alone. + pub(crate) fn to_data_type(self, precision: u8, scale: u8) -> DeltaResult { + match self { + Self::Boolean => Ok(DataType::BOOLEAN), + Self::Byte => Ok(DataType::BYTE), + Self::Short => Ok(DataType::SHORT), + Self::Integer => Ok(DataType::INTEGER), + Self::Long => Ok(DataType::LONG), + Self::Float => Ok(DataType::FLOAT), + Self::Double => Ok(DataType::DOUBLE), + Self::String => Ok(DataType::STRING), + Self::Binary => Ok(DataType::BINARY), + Self::Date => Ok(DataType::DATE), + Self::Timestamp => Ok(DataType::TIMESTAMP), + Self::TimestampNtz => Ok(DataType::TIMESTAMP_NTZ), + Self::IntervalYearMonth => Ok(DataType::INTERVAL_YEAR_MONTH), + Self::IntervalDayTime => Ok(DataType::INTERVAL_DAY_TIME), + Self::Decimal => Ok(DataType::Primitive(PrimitiveType::decimal( + precision, scale, + )?)), + Self::NonPrimitive => Err(delta_kernel::Error::generic( + "Non-primitive null types (struct, array, map, variant) cannot be reconstructed \ + from a type tag. Use opaque expressions or a schema visitor instead.", + )), + } + } +} + +/// Visit a typed null literal expression. +/// +/// The `type_tag` identifies the data type using the `NullTypeTag` encoding. For decimal nulls +/// (`type_tag == 12`), `precision` and `scale` specify the decimal type parameters; for all +/// other types, callers should pass 0 for both. +/// +/// Returns an error if the type tag is unrecognized, if the tag is `NonPrimitive` (255), or +/// if the decimal precision/scale is invalid. #[no_mangle] pub extern "C" fn visit_expression_literal_null( - _state: &mut KernelExpressionVisitorState, + state: &mut KernelExpressionVisitorState, + type_tag: u8, + precision: u8, + scale: u8, allocate_error: AllocateErrorFn, ) -> ExternResult { - let err = delta_kernel::Error::generic("NULL literal reconstruction is not supported"); // SAFETY: The allocate_error function pointer is provided by the engine and assumed valid. - unsafe { Err(err).into_extern_result(&allocate_error) } + unsafe { + visit_expression_literal_null_impl(state, type_tag, precision, scale) + .into_extern_result(&allocate_error) + } +} + +fn visit_expression_literal_null_impl( + state: &mut KernelExpressionVisitorState, + type_tag: u8, + precision: u8, + scale: u8, +) -> DeltaResult { + let tag = NullTypeTag::try_from(type_tag)?; + let data_type = tag.to_data_type(precision, scale)?; + Ok(wrap_expression( + state, + Expression::literal(Scalar::Null(data_type)), + )) } #[no_mangle] @@ -439,7 +646,6 @@ pub extern "C" fn visit_predicate_or( state: &mut KernelExpressionVisitorState, children: &mut EngineIterator, ) -> usize { - use delta_kernel::expressions::JunctionPredicateOp; let result = Predicate::junction( JunctionPredicateOp::Or, children.flat_map(|child| unwrap_kernel_predicate(state, child as usize)), @@ -465,7 +671,10 @@ pub extern "C" fn visit_expression_map_to_struct( child_expr: usize, ) -> usize { unwrap_kernel_expression(state, child_expr).map_or(0, |expr| { - wrap_expression(state, Expression::map_to_struct(expr)) + wrap_expression( + state, + Expression::map_to_struct(expr, MapToStructOptions::default()), + ) }) } @@ -528,3 +737,838 @@ fn visit_engine_predicate_impl( Ok(Arc::new(pred).into()) } + +// === Opaque predicate / expression builders ==================================== + +/// Drain `children` and resolve each ID to an [`Expression`]. Returns `None` +/// if any ID is invalid (already consumed, never created, or zero); on +/// failure all remaining valid IDs are still drained from state. +fn resolve_opaque_children>( + state: &mut KernelExpressionVisitorState, + child_ids: I, +) -> Option> { + // Drain ALL ids first so the ReferenceSet is consistently emptied, + // even when an earlier id is invalid. + let resolved: Vec> = child_ids + .into_iter() + .map(|id| unwrap_kernel_expression(state, id)) + .collect(); + resolved.into_iter().collect() +} + +/// Build a placeholder for an engine-defined predicate that kernel cannot evaluate: a NULL +/// boolean literal, which abstains from all pruning (even under `NOT`) while sibling predicates +/// prune normally. `children` are drained and discarded. +/// +/// Use [`visit_predicate_opaque_with_eval`] to attach engine eval callbacks so the node itself +/// can participate in file pruning. +/// +/// Returns 0 if any child ID is invalid. +/// +/// # Safety +/// `name` must be valid UTF-8 for the duration of the call. +#[no_mangle] +pub unsafe extern "C" fn visit_predicate_opaque( + state: &mut KernelExpressionVisitorState, + name: KernelStringSlice, + children: &mut EngineIterator, + allocate_error: AllocateErrorFn, +) -> ExternResult { + let name = unsafe { String::try_from_slice(&name) }; + visit_predicate_opaque_impl(state, name, children).into_extern_result(&allocate_error) +} + +fn visit_predicate_opaque_impl( + state: &mut KernelExpressionVisitorState, + name: DeltaResult, + children: &mut EngineIterator, +) -> DeltaResult { + let name = name?; + if resolve_opaque_children(state, children.map(|c| c as usize)).is_none() { + return Ok(0); + } + tracing::info!("opaque predicate `{name}`: no eval callbacks; kernel will not prune on it"); + Ok(wrap_predicate(state, Predicate::NULL)) +} + +/// Build an opaque predicate over `FfiOpaquePredicateOp(name, callbacks)` and +/// `children`, routed through the default engine's Arrow batch evaluator (via +/// `Predicate::arrow_opaque`). Kernel pre-evaluates each child arg recursively +/// via its standard `evaluate_expression`, exports the resulting columns as a +/// single `RecordBatch` over Arrow C Data Interface, and invokes the engine's +/// eval callback. Engine never walks the AST. +/// +/// `callbacks` is passed by value; ownership of its `engine_state` transfers to +/// kernel, which invokes `free_state` exactly once -- even when this call fails +/// or returns 0. Engines attaching the same logical state to multiple opaque +/// ops must pass independently freeable state per call. +/// +/// Returns 0 if any child ID is invalid. +/// +/// # Safety +/// `name` must be valid UTF-8 for the duration of the call; the function +/// pointers in `callbacks` must remain valid for the lifetime of the built +/// predicate. +#[cfg(feature = "default-engine-base")] +#[no_mangle] +pub unsafe extern "C" fn visit_predicate_opaque_with_eval( + state: &mut KernelExpressionVisitorState, + name: KernelStringSlice, + children: &mut EngineIterator, + callbacks: COpaqueEvalCallbacks, + allocate_error: AllocateErrorFn, +) -> ExternResult { + let name = unsafe { String::try_from_slice(&name) }; + // Wrap immediately so free_state fires exactly once on every exit path. + let callbacks = Arc::new(FfiOpaqueEvalCallbacks::new(callbacks)); + visit_predicate_opaque_with_eval_impl(state, name, children, callbacks) + .into_extern_result(&allocate_error) +} + +#[cfg(feature = "default-engine-base")] +fn visit_predicate_opaque_with_eval_impl( + state: &mut KernelExpressionVisitorState, + name: DeltaResult, + children: &mut EngineIterator, + callbacks: Arc, +) -> DeltaResult { + let name = name?; + let Some(exprs) = resolve_opaque_children(state, children.map(|c| c as usize)) else { + return Ok(0); + }; + let op = FfiOpaquePredicateOp::new(name, callbacks); + Ok(wrap_predicate(state, Predicate::arrow_opaque(op, exprs))) +} + +#[cfg(test)] +mod tests { + #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] + + use delta_kernel::expressions::{col, lit, Expression, Scalar}; + use delta_kernel::schema::{ArrayType, DataType, MapType, StructField, StructType}; + use rstest::rstest; + + use super::*; + + // ============================================================================ + // NullTypeTag::from_data_type + // ============================================================================ + + #[rstest] + #[case(DataType::BOOLEAN, NullTypeTag::Boolean, 0, 0)] + #[case(DataType::BYTE, NullTypeTag::Byte, 0, 0)] + #[case(DataType::SHORT, NullTypeTag::Short, 0, 0)] + #[case(DataType::INTEGER, NullTypeTag::Integer, 0, 0)] + #[case(DataType::LONG, NullTypeTag::Long, 0, 0)] + #[case(DataType::FLOAT, NullTypeTag::Float, 0, 0)] + #[case(DataType::DOUBLE, NullTypeTag::Double, 0, 0)] + #[case(DataType::STRING, NullTypeTag::String, 0, 0)] + #[case(DataType::BINARY, NullTypeTag::Binary, 0, 0)] + #[case(DataType::DATE, NullTypeTag::Date, 0, 0)] + #[case(DataType::TIMESTAMP, NullTypeTag::Timestamp, 0, 0)] + #[case(DataType::TIMESTAMP_NTZ, NullTypeTag::TimestampNtz, 0, 0)] + #[case(DataType::INTERVAL_YEAR_MONTH, NullTypeTag::IntervalYearMonth, 0, 0)] + #[case(DataType::INTERVAL_DAY_TIME, NullTypeTag::IntervalDayTime, 0, 0)] + fn from_data_type_primitive( + #[case] dt: DataType, + #[case] expected_tag: NullTypeTag, + #[case] expected_precision: u8, + #[case] expected_scale: u8, + ) { + let (tag, p, s) = NullTypeTag::from_data_type(&dt); + assert_eq!(tag, expected_tag); + assert_eq!(p, expected_precision); + assert_eq!(s, expected_scale); + } + + #[test] + fn from_data_type_decimal() { + let dt = DataType::decimal(18, 5).unwrap(); + let (tag, p, s) = NullTypeTag::from_data_type(&dt); + assert_eq!(tag, NullTypeTag::Decimal); + assert_eq!(p, 18); + assert_eq!(s, 5); + } + + #[test] + fn from_data_type_non_primitive_produces_sentinel() { + let struct_type = DataType::from( + StructType::try_new(vec![StructField::not_null("a", DataType::INTEGER)]).unwrap(), + ); + assert_eq!( + NullTypeTag::from_data_type(&struct_type), + (NullTypeTag::NonPrimitive, 0, 0) + ); + + let array_type = DataType::from(ArrayType::new(DataType::INTEGER, false)); + assert_eq!( + NullTypeTag::from_data_type(&array_type), + (NullTypeTag::NonPrimitive, 0, 0) + ); + + let map_type = DataType::from(MapType::new(DataType::STRING, DataType::INTEGER, false)); + assert_eq!( + NullTypeTag::from_data_type(&map_type), + (NullTypeTag::NonPrimitive, 0, 0) + ); + + let variant_type = DataType::unshredded_variant(); + assert_eq!( + NullTypeTag::from_data_type(&variant_type), + (NullTypeTag::NonPrimitive, 0, 0) + ); + } + + // ============================================================================ + // NullTypeTag::to_data_type + // ============================================================================ + + #[rstest] + #[case(NullTypeTag::Boolean, DataType::BOOLEAN)] + #[case(NullTypeTag::Byte, DataType::BYTE)] + #[case(NullTypeTag::Short, DataType::SHORT)] + #[case(NullTypeTag::Integer, DataType::INTEGER)] + #[case(NullTypeTag::Long, DataType::LONG)] + #[case(NullTypeTag::Float, DataType::FLOAT)] + #[case(NullTypeTag::Double, DataType::DOUBLE)] + #[case(NullTypeTag::String, DataType::STRING)] + #[case(NullTypeTag::Binary, DataType::BINARY)] + #[case(NullTypeTag::Date, DataType::DATE)] + #[case(NullTypeTag::Timestamp, DataType::TIMESTAMP)] + #[case(NullTypeTag::TimestampNtz, DataType::TIMESTAMP_NTZ)] + #[case(NullTypeTag::IntervalYearMonth, DataType::INTERVAL_YEAR_MONTH)] + #[case(NullTypeTag::IntervalDayTime, DataType::INTERVAL_DAY_TIME)] + fn to_data_type_primitive(#[case] tag: NullTypeTag, #[case] expected: DataType) { + assert_eq!(tag.to_data_type(0, 0).unwrap(), expected); + } + + #[test] + fn to_data_type_decimal() { + let dt = NullTypeTag::Decimal.to_data_type(10, 2).unwrap(); + assert_eq!(dt, DataType::decimal(10, 2).unwrap()); + } + + #[test] + fn to_data_type_decimal_invalid_precision_returns_error() { + assert!(NullTypeTag::Decimal.to_data_type(0, 0).is_err()); + } + + #[test] + fn to_data_type_decimal_scale_exceeds_precision_returns_error() { + assert!(NullTypeTag::Decimal.to_data_type(5, 10).is_err()); + } + + #[test] + fn to_data_type_non_primitive_returns_error() { + let result = NullTypeTag::NonPrimitive.to_data_type(0, 0); + assert!(result.is_err()); + let msg = result.unwrap_err().to_string(); + assert!(msg.contains("Non-primitive"), "unexpected error: {msg}"); + } + + // ============================================================================ + // TryFrom + // ============================================================================ + + #[rstest] + #[case(0, NullTypeTag::Boolean)] + #[case(1, NullTypeTag::Byte)] + #[case(2, NullTypeTag::Short)] + #[case(3, NullTypeTag::Integer)] + #[case(4, NullTypeTag::Long)] + #[case(5, NullTypeTag::Float)] + #[case(6, NullTypeTag::Double)] + #[case(7, NullTypeTag::String)] + #[case(8, NullTypeTag::Binary)] + #[case(9, NullTypeTag::Date)] + #[case(10, NullTypeTag::Timestamp)] + #[case(11, NullTypeTag::TimestampNtz)] + #[case(12, NullTypeTag::Decimal)] + #[case(13, NullTypeTag::IntervalYearMonth)] + #[case(14, NullTypeTag::IntervalDayTime)] + #[case(255, NullTypeTag::NonPrimitive)] + fn try_from_u8_valid(#[case] value: u8, #[case] expected: NullTypeTag) { + assert_eq!(NullTypeTag::try_from(value).unwrap(), expected); + } + + #[rstest] + #[case(15)] + #[case(42)] + #[case(254)] + fn try_from_u8_invalid(#[case] value: u8) { + let result = NullTypeTag::try_from(value); + assert!(result.is_err()); + let msg = result.unwrap_err().to_string(); + assert!( + msg.contains("Unrecognized null type tag"), + "unexpected error: {msg}" + ); + } + + // ============================================================================ + // visit_expression_literal_null_impl (error paths) + // ============================================================================ + + #[test] + fn visit_null_decimal_invalid_precision_returns_error() { + let mut state = KernelExpressionVisitorState::default(); + assert!(visit_expression_literal_null_impl(&mut state, 12, 0, 0).is_err()); + } + + #[test] + fn visit_null_unrecognized_tag_returns_error() { + let mut state = KernelExpressionVisitorState::default(); + assert!(visit_expression_literal_null_impl(&mut state, 15, 0, 0).is_err()); + } + + #[test] + fn visit_null_non_primitive_tag_returns_error() { + let mut state = KernelExpressionVisitorState::default(); + assert!(visit_expression_literal_null_impl(&mut state, 255, 0, 0).is_err()); + } + + // ============================================================================ + // Round-trip: from_data_type -> (tag as u8, p, s) -> visit_expression_literal_null_impl + // ============================================================================ + + #[rstest] + #[case(DataType::BOOLEAN)] + #[case(DataType::BYTE)] + #[case(DataType::SHORT)] + #[case(DataType::INTEGER)] + #[case(DataType::LONG)] + #[case(DataType::FLOAT)] + #[case(DataType::DOUBLE)] + #[case(DataType::STRING)] + #[case(DataType::BINARY)] + #[case(DataType::DATE)] + #[case(DataType::TIMESTAMP)] + #[case(DataType::TIMESTAMP_NTZ)] + #[case(DataType::INTERVAL_YEAR_MONTH)] + #[case(DataType::INTERVAL_DAY_TIME)] + fn null_type_round_trips_through_tag_encoding(#[case] data_type: DataType) { + let (tag, precision, scale) = NullTypeTag::from_data_type(&data_type); + let mut state = KernelExpressionVisitorState::default(); + let id = + visit_expression_literal_null_impl(&mut state, tag as u8, precision, scale).unwrap(); + let expr = unwrap_kernel_expression(&mut state, id).unwrap(); + assert_eq!(expr, Expression::literal(Scalar::Null(data_type)),); + } + + #[test] + fn decimal_null_round_trips_through_tag_encoding() { + let dt = DataType::decimal(20, 3).unwrap(); + let (tag, precision, scale) = NullTypeTag::from_data_type(&dt); + assert_eq!(tag, NullTypeTag::Decimal); + let mut state = KernelExpressionVisitorState::default(); + let id = + visit_expression_literal_null_impl(&mut state, tag as u8, precision, scale).unwrap(); + let expr = unwrap_kernel_expression(&mut state, id).unwrap(); + assert_eq!(expr, Expression::literal(Scalar::Null(dt))); + } + + #[rstest] + #[case(Scalar::IntervalYearMonth(17))] + #[case(Scalar::IntervalDayTime(1_234_567))] + #[case(Scalar::IntervalYearMonth(-13))] + #[case(Scalar::IntervalYearMonth(i32::MIN))] + #[case(Scalar::IntervalYearMonth(i32::MAX))] + #[case(Scalar::IntervalDayTime(-86_400_000_000))] + #[case(Scalar::IntervalDayTime(i64::MIN))] + #[case(Scalar::IntervalDayTime(i64::MAX))] + fn interval_literal_visitors_build_interval_scalars(#[case] expected: Scalar) { + let mut state = KernelExpressionVisitorState::default(); + let id = match expected { + Scalar::IntervalYearMonth(value) => { + visit_expression_literal_interval_year_month(&mut state, value) + } + Scalar::IntervalDayTime(value) => { + visit_expression_literal_interval_day_time(&mut state, value) + } + _ => unreachable!(), + }; + let expr = unwrap_kernel_expression(&mut state, id).unwrap(); + assert_eq!(expr, Expression::literal(expected)); + } + + // ============================================================================ + // Opaque-op builders (visit_predicate_opaque_impl and friends) + // + // These tests drive the `_impl` functions directly so we can construct + // `EngineIterator` from a plain Vec without going through the C FFI. + // ============================================================================ + + use std::os::raw::c_void; + use std::ptr::NonNull; + + /// Backing buffer for a test `EngineIterator`: yields each id as a pointer (consumer casts back + /// via `c as usize`). An id of 0 reads as end-of-iteration, fine since 0 is kernel's "invalid + /// id". + struct IterState { + ids: Vec, + idx: usize, + } + + extern "C" fn iter_next_direct(data: NonNull) -> *const c_void { + // SAFETY: `data` was set up by `make_iter` to point at a live + // `IterState`; the iterator is single-threaded and never re-entered. + let state = unsafe { &mut *(data.as_ptr() as *mut IterState) }; + if state.idx >= state.ids.len() { + return std::ptr::null(); + } + let id = state.ids[state.idx]; + state.idx += 1; + id as *const c_void + } + + /// RAII wrapper that frees the heap-allocated `IterState` backing an `EngineIterator`. Held as + /// a raw pointer via `Box::into_raw` so stacked borrows don't retag it while `it` holds the + /// pointer. + struct IterStateBox(*mut IterState); + + impl Drop for IterStateBox { + fn drop(&mut self) { + // SAFETY: produced by Box::into_raw in make_iter; reclaimed exactly once on drop. + drop(unsafe { Box::from_raw(self.0) }); + } + } + + fn make_iter(ids: Vec) -> (IterStateBox, EngineIterator) { + let raw = Box::into_raw(Box::new(IterState { ids, idx: 0 })); + let it = EngineIterator { + data: NonNull::new(raw as *mut c_void).unwrap(), + get_next: iter_next_direct, + }; + (IterStateBox(raw), it) + } + + fn make_two_literal_ids(state: &mut KernelExpressionVisitorState) -> (usize, usize) { + let a = wrap_expression(state, Expression::literal(1i32)); + let b = wrap_expression(state, Expression::literal(2i32)); + (a, b) + } + + #[test] + fn visit_predicate_opaque_impl_builds_null_literal_placeholder() { + let mut state = KernelExpressionVisitorState::default(); + let (a, b) = make_two_literal_ids(&mut state); + let (_keep, mut it) = make_iter(vec![a, b]); + let id = visit_predicate_opaque_impl(&mut state, Ok("MY_OP".to_string()), &mut it).unwrap(); + assert_ne!(id, 0); + let pred = unwrap_kernel_predicate(&mut state, id).unwrap(); + // No eval callbacks => NULL boolean literal, which abstains everywhere (even under NOT). + assert_eq!(pred, Predicate::NULL); + // Children are drained from the visitor state even though they're discarded. + assert!(state.inflight_ids.is_empty()); + } + + #[test] + fn visit_predicate_opaque_returns_zero_on_invalid_child() { + // 9999 is a never-issued id, so the builder bails without wrapping. + let mut state = KernelExpressionVisitorState::default(); + let (_keep, mut it) = make_iter(vec![9999usize]); + let id = visit_predicate_opaque_impl(&mut state, Ok("OP".to_string()), &mut it).unwrap(); + assert_eq!(id, 0, "invalid child should produce id=0 (no node created)"); + } + + // ============================================================================ + // Top-level `extern "C"` entry points. These drive the real FFI symbols (with + // an engine-supplied `allocate_error` and the handle alloc/dealloc dance) so + // miri can validate the unsafe boundary, not just the `_impl` helpers above. + // ============================================================================ + + use crate::ffi_test_utils::{allocate_err, ok_or_panic}; + use crate::kernel_string_slice; + + #[test] + fn visit_predicate_opaque_ffi_builds_null_literal_placeholder() { + let mut state = KernelExpressionVisitorState::default(); + let (a, b) = make_two_literal_ids(&mut state); + let (_keep, mut it) = make_iter(vec![a, b]); + let name = "MY_OP"; + let result = unsafe { + visit_predicate_opaque( + &mut state, + kernel_string_slice!(name), + &mut it, + allocate_err, + ) + }; + let id = ok_or_panic(result); + assert_ne!(id, 0); + let pred = unwrap_kernel_predicate(&mut state, id).unwrap(); + assert_eq!(pred, Predicate::NULL); + } + + /// Drives `visit_predicate_opaque_with_eval` end to end: pass the callbacks struct by value, + /// build the predicate through the FFI symbol, then verify the engine's `free_state` fires + /// exactly once when the predicate drops -- and also fires when the build bails early (invalid + /// child), since ownership of `engine_state` transfers with the call. + #[cfg(feature = "default-engine-base")] + #[test] + fn visit_predicate_opaque_with_eval_ffi_builds_and_frees() { + use std::ffi::c_void; + use std::sync::atomic::{AtomicUsize, Ordering}; + + use crate::engine_data::ArrowFFIData; + use crate::error::EngineExecResult; + + static FREED: AtomicUsize = AtomicUsize::new(0); + + unsafe extern "C" fn stub_eval( + _: *mut c_void, + _: KernelStringSlice, + _: ArrowFFIData, + _: bool, + _: *mut EngineExecResult, + ) { + } + unsafe extern "C" fn counting_free(_: *mut c_void) { + FREED.fetch_add(1, Ordering::SeqCst); + } + fn callbacks() -> COpaqueEvalCallbacks { + COpaqueEvalCallbacks { + engine_state: std::ptr::null_mut(), + eval_pred_rows: stub_eval, + eval_pred_stats: stub_eval, + free_state: counting_free, + } + } + + FREED.store(0, Ordering::SeqCst); + let mut state = KernelExpressionVisitorState::default(); + let (a, b) = make_two_literal_ids(&mut state); + let (_keep, mut it) = make_iter(vec![a, b]); + let name = "MY_EVAL_OP"; + let result = unsafe { + visit_predicate_opaque_with_eval( + &mut state, + kernel_string_slice!(name), + &mut it, + callbacks(), + allocate_err, + ) + }; + let id = ok_or_panic(result); + assert_ne!(id, 0); + assert_eq!( + FREED.load(Ordering::SeqCst), + 0, + "predicate still holds the callbacks" + ); + + // Dropping the predicate drops the last ref, firing the engine's free_state exactly once. + let pred = unwrap_kernel_predicate(&mut state, id).unwrap(); + drop(pred); + assert_eq!(FREED.load(Ordering::SeqCst), 1, "free_state must fire once"); + + // Bail-early path: an invalid child returns id 0, but ownership already transferred, so + // free_state still fires. + let (_keep, mut bad_it) = make_iter(vec![9999usize]); + let id = ok_or_panic(unsafe { + visit_predicate_opaque_with_eval( + &mut state, + kernel_string_slice!(name), + &mut bad_it, + callbacks(), + allocate_err, + ) + }); + assert_eq!(id, 0); + assert_eq!( + FREED.load(Ordering::SeqCst), + 2, + "free_state must fire on the bail-early path too" + ); + } + + /// End to end: build an opaque `IN_RANGE(id, 25)` predicate through the + /// `visit_predicate_opaque_with_eval` FFI symbol, then run a real `DefaultEngine` scan over a + /// 3-file in-memory table with disjoint `id` ranges. The engine callback prunes files whose + /// [min, max] excludes 25, so only the file covering [20, 30] survives -- proving the symbol -> + /// rewrite -> StatsMode dispatch -> file pruning chain works against a real log. + // Gated on a TLS backend (not bare `default-engine-base`): this test builds a real + // `DefaultEngineBuilder`, and kernel cannot compile with `default-engine-base` alone. + #[cfg(any( + feature = "default-engine-rustls", + feature = "default-engine-native-tls" + ))] + #[tokio::test(flavor = "multi_thread")] + async fn opaque_predicate_prunes_files_end_to_end() { + use std::ffi::c_void; + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::sync::Arc; + + use delta_kernel::arrow::array::ffi::{from_ffi, FFI_ArrowArray, FFI_ArrowSchema}; + use delta_kernel::arrow::array::{ + Array, ArrayRef, BooleanArray, Int64Array, RecordBatch, StructArray, + }; + use delta_kernel::object_store::memory::InMemory; + use delta_kernel::scan::state::ScanFile; + use delta_kernel::Snapshot; + use delta_kernel_default_engine::DefaultEngineBuilder; + use test_utils::add_commit; + + use crate::engine_data::ArrowFFIData; + use crate::error::EngineExecResult; + + static CALLS: AtomicUsize = AtomicUsize::new(0); + + // StatsMode `IN_RANGE`: keep a file iff the target (arg1) falls within the column's + // [min, max] (arg0 slots 0 and 1); null bounds keep the file. + unsafe extern "C" fn engine_in_range( + _state: *mut c_void, + _op_name: KernelStringSlice, + args_in: ArrowFFIData, + _inverted: bool, + out: *mut EngineExecResult, + ) { + CALLS.fetch_add(1, Ordering::SeqCst); + + let ArrowFFIData { array, schema } = args_in; + let data = unsafe { from_ffi(array, &schema) }.unwrap(); + let batch: RecordBatch = StructArray::from(data).into(); + + let stats = batch + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + let min = stats + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + let max = stats + .column(1) + .as_any() + .downcast_ref::() + .unwrap(); + let target = batch + .column(1) + .as_any() + .downcast_ref::() + .unwrap(); + + let keep: BooleanArray = (0..batch.num_rows()) + .map(|i| { + if min.is_null(i) || max.is_null(i) { + return Some(true); + } + let t = target.value(i); + Some(min.value(i) <= t && t <= max.value(i)) + }) + .collect(); + + let arr: ArrayRef = Arc::new(keep); + let array_data = arr.to_data(); + let ffi = ArrowFFIData { + array: FFI_ArrowArray::new(&array_data), + schema: FFI_ArrowSchema::try_from(array_data.data_type()).unwrap(), + }; + unsafe { *out = EngineExecResult::Success(ffi) }; + } + unsafe extern "C" fn noop_free(_: *mut c_void) {} + + // Build the predicate through the FFI symbol, then extract the kernel `Predicate`. + let mut state = KernelExpressionVisitorState::default(); + let col_id = wrap_expression(&mut state, col!("id")); + let target = wrap_expression(&mut state, lit(25i64)); + let (_keep, mut it) = make_iter(vec![col_id, target]); + let name = "IN_RANGE"; + let id = ok_or_panic(unsafe { + visit_predicate_opaque_with_eval( + &mut state, + kernel_string_slice!(name), + &mut it, + COpaqueEvalCallbacks { + engine_state: std::ptr::null_mut(), + eval_pred_rows: engine_in_range, + eval_pred_stats: engine_in_range, + free_state: noop_free, + }, + allocate_err, + ) + }); + let predicate = Arc::new(unwrap_kernel_predicate(&mut state, id).unwrap()); + + // 3-file table: A=[0,10], B=[20,30], C=[100,110]. Only B covers 25. + let table_root = "memory:///"; + let storage = Arc::new(InMemory::new()); + let schema = r#"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]}"#; + let add = |path: &str, lo: i64, hi: i64| { + format!( + r#"{{"add":{{"path":"{path}","partitionValues":{{}},"size":262,"modificationTime":1587968586000,"dataChange":true,"stats":"{{\"numRecords\":10,\"nullCount\":{{\"id\":0}},\"minValues\":{{\"id\":{lo}}},\"maxValues\":{{\"id\":{hi}}}}}"}}}}"# + ) + }; + let commit = [ + r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#.to_string(), + format!( + r#"{{"metaData":{{"id":"t","format":{{"provider":"parquet","options":{{}}}},"schemaString":"{schema}","partitionColumns":[],"configuration":{{}},"createdTime":1587968586000}}}}"# + ), + add("a.parquet", 0, 10), + add("b.parquet", 20, 30), + add("c.parquet", 100, 110), + ] + .join("\n"); + add_commit(table_root, storage.as_ref(), 0, commit) + .await + .unwrap(); + + let engine = DefaultEngineBuilder::new(storage).build(); + let snapshot = Snapshot::builder_for(table_root).build(&engine).unwrap(); + let scan = snapshot + .scan_builder() + .with_predicate(predicate) + .build() + .unwrap(); + + fn push_path(paths: &mut Vec, scan_file: ScanFile) { + paths.push(scan_file.path); + } + let mut paths: Vec = Vec::new(); + for sm in scan.scan_metadata(&engine).unwrap() { + paths = sm.unwrap().visit_scan_files(paths, push_path).unwrap(); + } + + assert_eq!( + paths, + vec!["b.parquet".to_string()], + "only [20,30] covers 25" + ); + assert!( + CALLS.load(Ordering::SeqCst) >= 1, + "engine callback must fire in StatsMode" + ); + } + + /// A misbehaving engine that violates the keep-on-null-bounds contract (returns `false` for + /// rows with null stats, as checkpoint Remove rows present) must not corrupt Add/Remove + /// reconciliation: kernel's `OR(NOT is_add, ...)` guard keeps Remove rows regardless of the + /// verdict, so the tombstone still suppresses the earlier Add instead of resurrecting it. + #[cfg(any( + feature = "default-engine-rustls", + feature = "default-engine-native-tls" + ))] + #[tokio::test(flavor = "multi_thread")] + async fn misbehaving_stats_callback_cannot_resurrect_removed_file() { + use std::ffi::c_void; + use std::sync::Arc; + + use delta_kernel::arrow::array::ffi::{from_ffi, FFI_ArrowArray, FFI_ArrowSchema}; + use delta_kernel::arrow::array::{Array, ArrayRef, BooleanArray, RecordBatch, StructArray}; + use delta_kernel::object_store::memory::InMemory; + use delta_kernel::scan::state::ScanFile; + use delta_kernel::Snapshot; + use delta_kernel_default_engine::DefaultEngineBuilder; + use test_utils::add_commit; + + use crate::engine_data::ArrowFFIData; + use crate::error::EngineExecResult; + + // Contract violation: skip (false) any row whose min/max bounds are null instead of + // keeping it. Remove rows carry null stats, so without a kernel-side guard this would + // drop tombstones from log replay. + unsafe extern "C" fn engine_skip_null_bounds( + _state: *mut c_void, + _op_name: KernelStringSlice, + args_in: ArrowFFIData, + _inverted: bool, + out: *mut EngineExecResult, + ) { + let ArrowFFIData { array, schema } = args_in; + let data = unsafe { from_ffi(array, &schema) }.unwrap(); + let batch: RecordBatch = StructArray::from(data).into(); + + let stats = batch + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + let (min, max) = (stats.column(0), stats.column(1)); + let keep: BooleanArray = (0..batch.num_rows()) + .map(|i| Some(!min.is_null(i) && !max.is_null(i))) + .collect(); + + let arr: ArrayRef = Arc::new(keep); + let array_data = arr.to_data(); + let ffi = ArrowFFIData { + array: FFI_ArrowArray::new(&array_data), + schema: FFI_ArrowSchema::try_from(array_data.data_type()).unwrap(), + }; + unsafe { *out = EngineExecResult::Success(ffi) }; + } + unsafe extern "C" fn noop_free(_: *mut c_void) {} + + let mut state = KernelExpressionVisitorState::default(); + let col_id = wrap_expression(&mut state, col!("id")); + let target = wrap_expression(&mut state, lit(25i64)); + let (_keep, mut it) = make_iter(vec![col_id, target]); + let name = "IN_RANGE"; + let id = ok_or_panic(unsafe { + visit_predicate_opaque_with_eval( + &mut state, + kernel_string_slice!(name), + &mut it, + COpaqueEvalCallbacks { + engine_state: std::ptr::null_mut(), + eval_pred_rows: engine_skip_null_bounds, + eval_pred_stats: engine_skip_null_bounds, + free_state: noop_free, + }, + allocate_err, + ) + }); + let predicate = Arc::new(unwrap_kernel_predicate(&mut state, id).unwrap()); + + // Version 0 adds two files (both with stats the engine keeps); version 1 removes + // b.parquet. Log replay must surface only a.parquet. + let table_root = "memory:///"; + let storage = Arc::new(InMemory::new()); + let schema = r#"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]}"#; + let add = |path: &str, lo: i64, hi: i64| { + format!( + r#"{{"add":{{"path":"{path}","partitionValues":{{}},"size":262,"modificationTime":1587968586000,"dataChange":true,"stats":"{{\"numRecords\":10,\"nullCount\":{{\"id\":0}},\"minValues\":{{\"id\":{lo}}},\"maxValues\":{{\"id\":{hi}}}}}"}}}}"# + ) + }; + let commit0 = [ + r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#.to_string(), + format!( + r#"{{"metaData":{{"id":"t","format":{{"provider":"parquet","options":{{}}}},"schemaString":"{schema}","partitionColumns":[],"configuration":{{}},"createdTime":1587968586000}}}}"# + ), + add("a.parquet", 20, 30), + add("b.parquet", 20, 30), + ] + .join("\n"); + add_commit(table_root, storage.as_ref(), 0, commit0) + .await + .unwrap(); + let commit1 = r#"{"remove":{"path":"b.parquet","deletionTimestamp":1587968586001,"dataChange":true}}"#.to_string(); + add_commit(table_root, storage.as_ref(), 1, commit1) + .await + .unwrap(); + + let engine = DefaultEngineBuilder::new(storage).build(); + let snapshot = Snapshot::builder_for(table_root).build(&engine).unwrap(); + let scan = snapshot + .scan_builder() + .with_predicate(predicate) + .build() + .unwrap(); + + fn push_path(paths: &mut Vec, scan_file: ScanFile) { + paths.push(scan_file.path); + } + let mut paths: Vec = Vec::new(); + for sm in scan.scan_metadata(&engine).unwrap() { + paths = sm.unwrap().visit_scan_files(paths, push_path).unwrap(); + } + + assert_eq!( + paths, + vec!["a.parquet".to_string()], + "the Remove tombstone must survive a skip-on-null-bounds verdict; \ + b.parquet reappearing means a resurrected deleted file" + ); + } +} diff --git a/ffi/src/expressions/mod.rs b/ffi/src/expressions/mod.rs index 9e3150d02d..55b4429613 100644 --- a/ffi/src/expressions/mod.rs +++ b/ffi/src/expressions/mod.rs @@ -1,16 +1,29 @@ //! This module holds functionality for moving expressions across the FFI boundary, both from //! engine to kernel, and from kernel to engine. +//! +//! This FFI exposes only opaque predicates, not opaque expressions; see the `opaque_eval` module +//! docs for why (expression-level functions fold into composite opaque predicates). use std::ffi::c_void; use delta_kernel::expressions::{OpaqueExpressionOp, OpaquePredicateOp}; use delta_kernel::{Expression, Predicate}; use delta_kernel_ffi_macros::handle_descriptor; -use crate::{handle::Handle, kernel_string_slice, KernelStringSlice}; +use crate::handle::Handle; +use crate::{kernel_string_slice, KernelStringSlice}; pub mod engine_visitor; pub mod kernel_visitor; +#[cfg(feature = "default-engine-base")] +pub mod opaque_eval; + +#[cfg(feature = "default-engine-base")] +mod arrow_eval; + +#[cfg(feature = "default-engine-base")] +pub(crate) use arrow_eval::FfiOpaquePredicateOp; + #[handle_descriptor(target=Expression, mutable=false, sized=true)] pub struct SharedExpression; diff --git a/ffi/src/expressions/opaque_eval.rs b/ffi/src/expressions/opaque_eval.rs new file mode 100644 index 0000000000..77ef965bd6 --- /dev/null +++ b/ffi/src/expressions/opaque_eval.rs @@ -0,0 +1,199 @@ +//! Engine callback framework for opaque-predicate evaluation. +//! +//! Engines pass a [`COpaqueEvalCallbacks`] struct by value to +//! [`crate::expressions::kernel_visitor::visit_predicate_opaque_with_eval`]. Kernel does all +//! recursive evaluation natively through its standard `evaluate_expression` / `evaluate_predicate` +//! paths; the engine is called only for the opaque-predicate node itself, and receives args as +//! pre-computed Arrow arrays. +//! +//! Available only under `default-engine-base`: args and results cross as Arrow C Data Interface +//! batches, and kernel pre-evaluates the args with its Arrow expression evaluator. +//! TODO: support a non-arrow opaque-eval path so engines without the Arrow evaluator can use this. +//! +//! Mirrors how kernel-native opaque ops work (see `OpaqueLessThanOp` in +//! `kernel/src/engine/arrow_expression/tests.rs`): the op gets the raw args, calls +//! `evaluate_expression` on each to produce ArrayRefs, then combines them with op-specific logic. + +use std::ffi::c_void; + +use crate::engine_data::ArrowFFIData; +use crate::error::EngineExecResult; +use crate::KernelStringSlice; + +/// Engine callback for **row-time** evaluation of an opaque predicate. +/// +/// Kernel pre-evaluates the predicate's args into a `RecordBatch` (one field per arg) and passes it +/// by value as `args_in` (an `ArrowFFIData`); the engine owns it and must release both Arrow C Data +/// Interface handles on every path (including `Failure`/`Uninit`). The batch's columns line up with +/// the predicate's args by position: +/// - a `Column` arg arrives as its evaluated values +/// - a `Literal` arg as the constant repeated for every row +/// - a `Predicate` arg as a `BooleanArray` of its verdicts +/// +/// The engine returns one bool per row. +/// +/// The result uses the out-pointer convention: kernel pre-initializes `*out` to +/// `EngineExecResult::Uninit`. On success the engine writes `EngineExecResult::Success` holding the +/// result `BooleanArray` as Arrow C Data Interface structs, transferring their ownership to kernel. +/// On failure it writes `EngineExecResult::Failure` carrying a `KernelError` code and a message +/// handle (built via `allocate_kernel_string`); leaving `*out` as `Uninit` is also treated as an +/// error. When `inverted`, evaluate `NOT op`. +/// +/// # Safety +/// `out` is valid only for the duration of the call; the engine must not retain it. The callback +/// must not panic or unwind across the FFI boundary. +pub type EngineEvalRowsFn = unsafe extern "C" fn( + engine_state: *mut c_void, + op_name: KernelStringSlice, + args_in: ArrowFFIData, + inverted: bool, + out: *mut EngineExecResult, +); + +/// Engine callback for **stats-based** evaluation of an opaque predicate, for file data skipping. +/// +/// Like [`EngineEvalRowsFn`], but each `Column` arg arrives as a per-file 4-field struct indexed by +/// position: `0`=min, `1`=max (column type; null if not collected), `2`=nullcount, `3`=rowcount +/// (`Int64`; null if not collected). `Literal`/`Predicate` args are unchanged. The engine returns +/// one bool per file: `false` = skip, `true`/`null` = keep. +/// +/// Stats are *conservative*: `min`/`max` only bound the values and `nullcount`/`rowcount` may +/// overcount, so skip a file only when the predicate *cannot* hold for any value in `[min, max]`. +/// Keep (`true`/`null`) whenever the stats can't prove the predicate impossible -- in particular +/// on null/absent bounds, where skipping silently drops live files from the scan. Log-replay +/// soundness does not rest on the engine, though: kernel wraps the rewritten predicate with an +/// `OR(NOT is_add, ...)` guard, so Remove rows (which carry null stats) are kept regardless of +/// the verdict and a misbehaving callback cannot resurrect deleted files. +/// When `inverted`, evaluate `NOT op` -- not `!verdict`; if you can't reason soundly about the +/// negated op, keep every file. Result/out-pointer/panic contract matches [`EngineEvalRowsFn`]. +pub type EngineEvalStatsFn = unsafe extern "C" fn( + engine_state: *mut c_void, + op_name: KernelStringSlice, + args_in: ArrowFFIData, + inverted: bool, + out: *mut EngineExecResult, +); + +/// Destructor for the engine's state pointer. +/// +/// Must not panic or unwind across the FFI boundary. +pub type EngineFreeStateFn = unsafe extern "C" fn(engine_state: *mut c_void); + +/// Bundle of engine callbacks for opaque-predicate evaluation, passed by value to +/// [`visit_predicate_opaque_with_eval`]. Ownership of `engine_state` transfers to kernel with the +/// call: `free_state` is invoked exactly once when the predicate built from it (including any +/// data-skipping clones kernel derives) is dropped. Engines attaching the same logical state to +/// multiple opaque ops must pass independently freeable state per call. +/// +/// # Thread safety +/// +/// Kernel may invoke the eval callbacks -- and ultimately `free_state` -- from any thread, +/// potentially concurrently: the predicate is shared across whatever threads drive the scan, and +/// the last reference may drop on any of them. The engine must hand over only an `engine_state` +/// and callbacks that tolerate this; any synchronization is the engine's responsibility. +/// +/// [`visit_predicate_opaque_with_eval`]: crate::expressions::kernel_visitor::visit_predicate_opaque_with_eval +#[repr(C)] +pub struct COpaqueEvalCallbacks { + /// Opaque engine state; passed back as the first argument to each + /// callback. + pub engine_state: *mut c_void, + /// Row-time evaluation: one verdict per data row. + pub eval_pred_rows: EngineEvalRowsFn, + /// Stats-based evaluation for file data skipping: one verdict per file. + pub eval_pred_stats: EngineEvalStatsFn, + /// Destructor for `engine_state`. Called exactly once; may run on any + /// kernel thread. + pub free_state: EngineFreeStateFn, +} + +impl std::fmt::Debug for COpaqueEvalCallbacks { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("COpaqueEvalCallbacks") + .field("engine_state", &self.engine_state) + .finish_non_exhaustive() + } +} + +/// Rust-side adapter owning a [`COpaqueEvalCallbacks`]: invokes the engine's `free_state` exactly +/// once on drop. Shared across an op and its data-skipping clones via `Arc`. +#[derive(Debug)] +pub(crate) struct FfiOpaqueEvalCallbacks { + pub(crate) inner: COpaqueEvalCallbacks, +} + +impl FfiOpaqueEvalCallbacks { + pub(crate) fn new(callbacks: COpaqueEvalCallbacks) -> Self { + Self { inner: callbacks } + } +} + +// SAFETY: values of this type exist only via the engine handover in +// `visit_predicate_opaque_with_eval`, whose thread-safety contract (see `COpaqueEvalCallbacks`) +// obligates the engine to pass only state and callbacks that are safe to use from any thread, +// concurrently. Kernel adds no thread-affine state of its own, so sharing the adapter across +// threads (behind the op's `Arc`) is sound. +unsafe impl Send for FfiOpaqueEvalCallbacks {} +unsafe impl Sync for FfiOpaqueEvalCallbacks {} + +impl Drop for FfiOpaqueEvalCallbacks { + fn drop(&mut self) { + // SAFETY: engine_state was handed over by value alongside the free_state callback; we + // promise to call it exactly once when the last Arc reference drops. free_state must not + // panic: drop can run while kernel is already unwinding, and unwinding into kernel aborts. + unsafe { (self.inner.free_state)(self.inner.engine_state) }; + } +} + +#[cfg(test)] +pub(crate) mod tests { + use std::ptr; + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::sync::Arc; + + use super::*; + + pub(crate) static TEST_FREES: AtomicUsize = AtomicUsize::new(0); + + pub(crate) unsafe extern "C" fn noop_eval_pred( + _state: *mut c_void, + _op_name: KernelStringSlice, + _args_in: ArrowFFIData, + _inverted: bool, + _out: *mut EngineExecResult, + ) { + } + + pub(crate) unsafe extern "C" fn counting_free_state(_state: *mut c_void) { + TEST_FREES.fetch_add(1, Ordering::SeqCst); + } + + fn counting_callbacks() -> COpaqueEvalCallbacks { + COpaqueEvalCallbacks { + engine_state: ptr::null_mut(), + eval_pred_rows: noop_eval_pred, + eval_pred_stats: noop_eval_pred, + free_state: counting_free_state, + } + } + + #[test] + fn drop_invokes_free_state_once() { + TEST_FREES.store(0, Ordering::SeqCst); + let wrapper = FfiOpaqueEvalCallbacks::new(counting_callbacks()); + assert_eq!(TEST_FREES.load(Ordering::SeqCst), 0); + drop(wrapper); + assert_eq!(TEST_FREES.load(Ordering::SeqCst), 1); + } + + #[test] + fn drop_via_arc_clone_fires_free_only_once() { + TEST_FREES.store(0, Ordering::SeqCst); + let arc1 = Arc::new(FfiOpaqueEvalCallbacks::new(counting_callbacks())); + let arc2 = arc1.clone(); + drop(arc1); + assert_eq!(TEST_FREES.load(Ordering::SeqCst), 0); + drop(arc2); + assert_eq!(TEST_FREES.load(Ordering::SeqCst), 1); + } +} diff --git a/ffi/src/ffi_metrics.rs b/ffi/src/ffi_metrics.rs new file mode 100644 index 0000000000..1a0bdea852 --- /dev/null +++ b/ffi/src/ffi_metrics.rs @@ -0,0 +1,860 @@ +//! FFI-safe representations of kernel [`MetricEvent`]s. +//! +//! [`MetricEvent`] is a `repr(C)` version of [`delta_kernel::metrics::MetricEvent`], to allow +//! engines to receive structured metrics across the FFI boundary. To enable metrics reporting call +//! [`crate::ffi_tracing::enable_metrics_reporting`]. +//! +//! - Durations are reported as `u64`, either nanoseconds or milliseconds, indicated by an `ns` or +//! `ms` suffix +//! - Operation ids are UUIDs in kernel (see [`MetricId`]). Here we report the raw 16 bytes of the +//! UUID. + +use delta_kernel::metrics as kernel; + +use crate::{kernel_string_slice, KernelStringSlice}; + +/// The 16 raw bytes of an operation's UUID, used to correlate events from the same operation. +#[repr(C)] +pub struct MetricId { + pub bytes: [u8; 16], +} + +impl From for MetricId { + fn from(id: kernel::MetricId) -> Self { + Self { + bytes: id.as_bytes(), + } + } +} + +// we add cbindgen:prefix-with-name=true below otherwise we get name conflicts. For example +// CommitFailureReason::Error would become just `Error`, which conflicts with the type in +// error.rs. Not all variants cause conflicts, but since they refer to common kernel concepts we +// prefix all enums to avoid future issues. + +/// Which scan execution path produced a [`ScanMetadataCompleted`] event. +/// +/// cbindgen:prefix-with-name=true +#[repr(C)] +pub enum ScanType { + SequentialPhase, + ParallelPhase, + Full, +} + +impl From for ScanType { + fn from(t: kernel::ScanType) -> Self { + match t { + kernel::ScanType::SequentialPhase => Self::SequentialPhase, + kernel::ScanType::ParallelPhase => Self::ParallelPhase, + kernel::ScanType::Full => Self::Full, + } + } +} + +/// Why a transaction commit did not succeed. +/// +/// cbindgen:prefix-with-name=true +#[repr(C)] +pub enum CommitFailureReason { + Conflict, + RetryableIo, + Error, +} + +impl From for CommitFailureReason { + fn from(r: kernel::CommitFailureReason) -> Self { + match r { + kernel::CommitFailureReason::Conflict => Self::Conflict, + kernel::CommitFailureReason::RetryableIo => Self::RetryableIo, + kernel::CommitFailureReason::Error => Self::Error, + } + } +} + +/// Whether a table is path-based or catalog-managed. +/// +/// cbindgen:prefix-with-name=true +#[repr(C)] +pub enum TableType { + PathBased, + CatalogManaged, +} + +impl From for TableType { + fn from(t: kernel::TableType) -> Self { + match t { + kernel::TableType::PathBased => Self::PathBased, + kernel::TableType::CatalogManaged => Self::CatalogManaged, + } + } +} + +/// The kind of log-segment load: a full listing from the base up to the target, or an incremental +/// listing of the commits above an existing segment. +/// +/// cbindgen:prefix-with-name=true +#[repr(C)] +pub enum LogSegmentLoadType { + Full, + Incremental, + Unknown, +} + +impl From for LogSegmentLoadType { + fn from(t: kernel::LogSegmentLoadType) -> Self { + match t { + kernel::LogSegmentLoadType::Full => Self::Full, + kernel::LogSegmentLoadType::Incremental => Self::Incremental, + _ => Self::Unknown, + } + } +} + +/// A log segment was loaded. +#[repr(C)] +pub struct LogSegmentLoadSuccess { + pub operation_id: MetricId, + pub correlation_id: KernelStringSlice, + pub table_type: TableType, + pub load_type: LogSegmentLoadType, + pub duration_ns: u64, + pub num_commit_files: u64, + pub num_checkpoint_files: u64, + pub num_compaction_files: u64, + /// Whether the segment had a CRC file. When false, `crc_versions_behind` is 0 and meaningless. + pub has_crc: bool, + /// Versions the latest CRC file is behind the loaded version (0 when at the loaded version). + /// Only meaningful when `has_crc` is true. + pub crc_versions_behind: u64, +} + +/// Listing the log segment failed. +#[repr(C)] +pub struct LogSegmentLoadFailure { + pub operation_id: MetricId, + pub correlation_id: KernelStringSlice, + pub table_type: TableType, + pub load_type: LogSegmentLoadType, +} + +/// How a snapshot load resolved Protocol and Metadata. +/// +/// cbindgen:prefix-with-name=true +#[repr(C)] +pub enum ProtocolMetadataSource { + CrcAtTarget, + CrcAdvancedByReplay, + CrcSeededPmOnlyReplay, + FullReplay, + Unknown, +} + +impl From for ProtocolMetadataSource { + fn from(s: kernel::ProtocolMetadataSource) -> Self { + match s { + kernel::ProtocolMetadataSource::CrcAtTarget => Self::CrcAtTarget, + kernel::ProtocolMetadataSource::CrcAdvancedByReplay => Self::CrcAdvancedByReplay, + kernel::ProtocolMetadataSource::CrcSeededPmOnlyReplay => Self::CrcSeededPmOnlyReplay, + kernel::ProtocolMetadataSource::FullReplay => Self::FullReplay, + _ => Self::Unknown, + } + } +} + +/// Protocol and metadata actions were read from the log. +#[repr(C)] +pub struct ProtocolMetadataLoadSuccess { + pub operation_id: MetricId, + pub correlation_id: KernelStringSlice, + pub table_type: TableType, + pub load_type: LogSegmentLoadType, + pub source: ProtocolMetadataSource, + pub duration_ns: u64, +} + +/// Reading protocol and metadata from the log failed. +#[repr(C)] +pub struct ProtocolMetadataLoadFailure { + pub operation_id: MetricId, + pub correlation_id: KernelStringSlice, + pub table_type: TableType, + pub load_type: LogSegmentLoadType, +} + +/// A snapshot was built successfully. +#[repr(C)] +pub struct SnapshotBuildSuccess { + pub operation_id: MetricId, + pub correlation_id: KernelStringSlice, + pub table_type: TableType, + pub load_type: LogSegmentLoadType, + pub version: u64, + pub duration_ns: u64, +} + +/// Building a snapshot failed. +#[repr(C)] +pub struct SnapshotBuildFailure { + pub operation_id: MetricId, + pub correlation_id: KernelStringSlice, + pub table_type: TableType, + pub load_type: LogSegmentLoadType, +} + +/// A transaction was committed successfully. `operation` and `correlation_id` are slices into +/// engine-supplied data that are only valid for the duration of the callback; an empty slice means +/// the value was unset. +#[repr(C)] +pub struct TransactionCommitSuccess { + pub operation_id: MetricId, + pub correlation_id: KernelStringSlice, + pub table_type: TableType, + pub commit_version: u64, + pub num_add_files: u64, + pub num_remove_files: u64, + pub num_dv_updates: u64, + pub add_files_bytes: u64, + pub remove_files_bytes: u64, + pub is_blind_append: bool, + pub data_change: bool, + pub operation: KernelStringSlice, + pub prepare_duration_ns: u64, + pub committer_duration_ns: u64, + pub total_duration_ns: u64, +} + +/// A transaction commit did not succeed; `reason` distinguishes conflict, retryable IO, and +/// terminal errors. +#[repr(C)] +pub struct TransactionCommitFailure { + pub operation_id: MetricId, + pub correlation_id: KernelStringSlice, + pub table_type: TableType, + pub reason: CommitFailureReason, +} + +/// A domain metadata load completed. +#[repr(C)] +pub struct DomainMetadataLoadSuccess { + pub from_cache: bool, + pub num_domains_returned: u64, + pub duration_ns: u64, +} + +/// A `SetTransaction` (app id) load completed. +#[repr(C)] +pub struct SetTransactionLoadSuccess { + pub from_cache: bool, + pub found: bool, + pub duration_ns: u64, +} + +/// A CRC file was read and parsed successfully. +#[repr(C)] +pub struct CrcReadSuccess { + pub bytes_read: u64, + pub duration_ns: u64, +} + +/// A `JsonHandler::read_json_files` call completed. +#[repr(C)] +pub struct JsonReadCompleted { + pub num_files: u64, + pub bytes_read: u64, +} + +/// A `ParquetHandler::read_parquet_files` call completed. +#[repr(C)] +pub struct ParquetReadCompleted { + pub num_files: u64, + pub bytes_read: u64, +} + +/// A scan metadata operation completed. See [`delta_kernel::metrics::ScanMetadataCompleted`] for +/// field semantics. +#[repr(C)] +pub struct ScanMetadataCompleted { + pub operation_id: MetricId, + pub correlation_id: KernelStringSlice, + pub table_type: TableType, + pub scan_type: ScanType, + pub duration_ns: u64, + pub num_add_files_seen: u64, + pub num_active_add_files: u64, + pub active_add_files_bytes: u64, + pub num_remove_files_seen: u64, + pub num_non_file_actions: u64, + pub num_predicate_filtered: u64, + pub peak_hash_set_size: u64, + pub dedup_visitor_time_ns: u64, + pub predicate_eval_time_ns: u64, +} + +/// A storage list operation completed. +#[repr(C)] +pub struct StorageListCompleted { + pub duration_ns: u64, + pub num_files: u64, +} + +/// A storage read operation completed. +#[repr(C)] +pub struct StorageReadCompleted { + pub duration_ns: u64, + pub num_files: u64, + pub bytes_read: u64, +} + +/// A storage copy or rename operation completed. +#[repr(C)] +pub struct StorageCopyCompleted { + pub duration_ns: u64, +} + +/// C version of [`delta_kernel::metrics::MetricEvent`]. Passed by value to a callback registered +/// via [`crate::ffi_tracing::enable_metrics_reporting`]. +/// +/// cbindgen:prefix-with-name=true +#[repr(C)] +pub enum MetricEvent { + LogSegmentLoadSuccess(LogSegmentLoadSuccess), + LogSegmentLoadFailure(LogSegmentLoadFailure), + ProtocolMetadataLoadSuccess(ProtocolMetadataLoadSuccess), + ProtocolMetadataLoadFailure(ProtocolMetadataLoadFailure), + SnapshotBuildSuccess(SnapshotBuildSuccess), + SnapshotBuildFailure(SnapshotBuildFailure), + TransactionCommitSuccess(TransactionCommitSuccess), + TransactionCommitFailure(TransactionCommitFailure), + DomainMetadataLoadSuccess(DomainMetadataLoadSuccess), + DomainMetadataLoadFailure, + SetTransactionLoadSuccess(SetTransactionLoadSuccess), + SetTransactionLoadFailure, + CrcReadSuccess(CrcReadSuccess), + CrcReadFailure, + JsonReadCompleted(JsonReadCompleted), + ParquetReadCompleted(ParquetReadCompleted), + ScanMetadataCompleted(ScanMetadataCompleted), + StorageListCompleted(StorageListCompleted), + StorageReadCompleted(StorageReadCompleted), + StorageCopyCompleted(StorageCopyCompleted), +} + +/// Borrow the caller-supplied correlation id as a [`KernelStringSlice`], or an empty slice when +/// unset. The slice is valid only as long as the source kernel event is alive. +fn correlation_id_slice(correlation_id: Option<&str>) -> KernelStringSlice { + match correlation_id { + // Safety: the borrowed str lives as long as the source event, which outlives both the FFI + // event and the callback it is passed to. + Some(id) => unsafe { KernelStringSlice::new_unsafe(id) }, + None => KernelStringSlice::empty(), + } +} + +impl MetricEvent { + /// Build an FFI event from a kernel event. Note that this event is only valid as long as the + /// passed argument is still alive, since we might make `KernelStringSlices` out of some + /// fields. + pub(crate) fn from_kernel(event: &kernel::MetricEvent) -> Self { + use kernel::MetricEvent as K; + let ns = |d: std::time::Duration| d.as_nanos() as u64; + match event { + K::LogSegmentLoadSuccess(kernel::LogSegmentLoadSuccess { + operation_id, + correlation_id, + table_type, + load_type, + num_commit_files, + num_checkpoint_files, + num_compaction_files, + crc_versions_behind, + duration, + }) => Self::LogSegmentLoadSuccess(LogSegmentLoadSuccess { + operation_id: (*operation_id).into(), + correlation_id: correlation_id_slice(correlation_id.as_deref()), + table_type: (*table_type).into(), + load_type: (*load_type).into(), + duration_ns: ns(*duration), + num_commit_files: *num_commit_files, + num_checkpoint_files: *num_checkpoint_files, + num_compaction_files: *num_compaction_files, + // C has no Option: `has_crc` gates `crc_versions_behind` (0 when absent). + has_crc: crc_versions_behind.is_some(), + crc_versions_behind: crc_versions_behind.unwrap_or(0), + }), + K::LogSegmentLoadFailure(kernel::LogSegmentLoadFailure { + operation_id, + correlation_id, + table_type, + load_type, + }) => Self::LogSegmentLoadFailure(LogSegmentLoadFailure { + operation_id: (*operation_id).into(), + correlation_id: correlation_id_slice(correlation_id.as_deref()), + table_type: (*table_type).into(), + load_type: (*load_type).into(), + }), + K::ProtocolMetadataLoadSuccess(kernel::ProtocolMetadataLoadSuccess { + operation_id, + correlation_id, + table_type, + load_type, + source, + duration, + }) => Self::ProtocolMetadataLoadSuccess(ProtocolMetadataLoadSuccess { + operation_id: (*operation_id).into(), + correlation_id: correlation_id_slice(correlation_id.as_deref()), + table_type: (*table_type).into(), + load_type: (*load_type).into(), + source: (*source).into(), + duration_ns: ns(*duration), + }), + K::ProtocolMetadataLoadFailure(kernel::ProtocolMetadataLoadFailure { + operation_id, + correlation_id, + table_type, + load_type, + }) => Self::ProtocolMetadataLoadFailure(ProtocolMetadataLoadFailure { + operation_id: (*operation_id).into(), + correlation_id: correlation_id_slice(correlation_id.as_deref()), + table_type: (*table_type).into(), + load_type: (*load_type).into(), + }), + K::SnapshotBuildSuccess(kernel::SnapshotBuildSuccess { + operation_id, + correlation_id, + table_type, + load_type, + version, + duration, + }) => Self::SnapshotBuildSuccess(SnapshotBuildSuccess { + operation_id: (*operation_id).into(), + correlation_id: correlation_id_slice(correlation_id.as_deref()), + table_type: (*table_type).into(), + load_type: (*load_type).into(), + version: *version, + duration_ns: ns(*duration), + }), + K::SnapshotBuildFailure(kernel::SnapshotBuildFailure { + operation_id, + correlation_id, + table_type, + load_type, + }) => Self::SnapshotBuildFailure(SnapshotBuildFailure { + operation_id: (*operation_id).into(), + correlation_id: correlation_id_slice(correlation_id.as_deref()), + table_type: (*table_type).into(), + load_type: (*load_type).into(), + }), + K::TransactionCommitSuccess(kernel::TransactionCommitSuccess { + operation_id, + correlation_id, + table_type, + commit_version, + num_add_files, + num_remove_files, + num_dv_updates, + add_files_bytes, + remove_files_bytes, + is_blind_append, + data_change, + operation, + prepare_duration, + committer_duration, + total_duration, + }) => { + let operation = match operation.as_ref() { + Some(o) => kernel_string_slice!(o), + None => KernelStringSlice::empty(), + }; + Self::TransactionCommitSuccess(TransactionCommitSuccess { + operation_id: (*operation_id).into(), + correlation_id: correlation_id_slice(correlation_id.as_deref()), + table_type: (*table_type).into(), + commit_version: *commit_version, + num_add_files: *num_add_files, + num_remove_files: *num_remove_files, + num_dv_updates: *num_dv_updates, + add_files_bytes: *add_files_bytes, + remove_files_bytes: *remove_files_bytes, + is_blind_append: *is_blind_append, + data_change: *data_change, + operation, + prepare_duration_ns: ns(*prepare_duration), + committer_duration_ns: ns(*committer_duration), + total_duration_ns: ns(*total_duration), + }) + } + K::TransactionCommitFailure(kernel::TransactionCommitFailure { + operation_id, + correlation_id, + table_type, + reason, + }) => Self::TransactionCommitFailure(TransactionCommitFailure { + operation_id: (*operation_id).into(), + correlation_id: correlation_id_slice(correlation_id.as_deref()), + table_type: (*table_type).into(), + reason: (*reason).into(), + }), + K::DomainMetadataLoadSuccess(kernel::DomainMetadataLoadSuccess { + from_cache, + num_domains_returned, + duration, + }) => Self::DomainMetadataLoadSuccess(DomainMetadataLoadSuccess { + from_cache: *from_cache, + num_domains_returned: *num_domains_returned, + duration_ns: ns(*duration), + }), + K::DomainMetadataLoadFailure => Self::DomainMetadataLoadFailure, + K::SetTransactionLoadSuccess(kernel::SetTransactionLoadSuccess { + from_cache, + found, + duration, + }) => Self::SetTransactionLoadSuccess(SetTransactionLoadSuccess { + from_cache: *from_cache, + found: *found, + duration_ns: ns(*duration), + }), + K::SetTransactionLoadFailure => Self::SetTransactionLoadFailure, + K::CrcReadSuccess(kernel::CrcReadSuccess { + bytes_read, + duration, + }) => Self::CrcReadSuccess(CrcReadSuccess { + bytes_read: *bytes_read, + duration_ns: ns(*duration), + }), + K::CrcReadFailure => Self::CrcReadFailure, + K::JsonReadCompleted(kernel::JsonReadCompleted { + num_files, + bytes_read, + }) => Self::JsonReadCompleted(JsonReadCompleted { + num_files: *num_files, + bytes_read: *bytes_read, + }), + K::ParquetReadCompleted(kernel::ParquetReadCompleted { + num_files, + bytes_read, + }) => Self::ParquetReadCompleted(ParquetReadCompleted { + num_files: *num_files, + bytes_read: *bytes_read, + }), + K::ScanMetadataCompleted(kernel::ScanMetadataCompleted { + operation_id, + correlation_id, + table_type, + scan_type, + duration, + num_add_files_seen, + num_active_add_files, + active_add_files_bytes, + num_remove_files_seen, + num_non_file_actions, + num_predicate_filtered, + peak_hash_set_size, + dedup_visitor_time, + predicate_eval_time, + }) => Self::ScanMetadataCompleted(ScanMetadataCompleted { + operation_id: (*operation_id).into(), + correlation_id: correlation_id_slice(correlation_id.as_deref()), + table_type: (*table_type).into(), + scan_type: (*scan_type).into(), + duration_ns: ns(*duration), + num_add_files_seen: *num_add_files_seen, + num_active_add_files: *num_active_add_files, + active_add_files_bytes: *active_add_files_bytes, + num_remove_files_seen: *num_remove_files_seen, + num_non_file_actions: *num_non_file_actions, + num_predicate_filtered: *num_predicate_filtered, + peak_hash_set_size: *peak_hash_set_size as u64, // note usize -> u64 cast + dedup_visitor_time_ns: ns(*dedup_visitor_time), + predicate_eval_time_ns: ns(*predicate_eval_time), + }), + K::StorageListCompleted(kernel::StorageListCompleted { + duration, + num_files, + }) => Self::StorageListCompleted(StorageListCompleted { + duration_ns: ns(*duration), + num_files: *num_files, + }), + K::StorageReadCompleted(kernel::StorageReadCompleted { + duration, + num_files, + bytes_read, + }) => Self::StorageReadCompleted(StorageReadCompleted { + duration_ns: ns(*duration), + num_files: *num_files, + bytes_read: *bytes_read, + }), + K::StorageCopyCompleted(kernel::StorageCopyCompleted { duration }) => { + Self::StorageCopyCompleted(StorageCopyCompleted { + duration_ns: ns(*duration), + }) + } + } + } +} + +/// Invoke `f` with the FFI [`MetricEvent`] built from `event`. +pub(crate) fn with_ffi_event( + event: &kernel::MetricEvent, + f: impl FnOnce(MetricEvent) -> R, +) -> R { + f(MetricEvent::from_kernel(event)) +} + +#[cfg(test)] +mod tests { + use std::mem::discriminant; + use std::time::Duration; + + use super::*; + use crate::TryFromStringSlice; + + #[test] + fn from_kernel_transaction_commit_success_carries_operation_and_id() { + let id = kernel::MetricId::new(); + let event = + kernel::MetricEvent::TransactionCommitSuccess(kernel::TransactionCommitSuccess { + operation_id: id, + correlation_id: Some("req-7".into()), + table_type: kernel::TableType::CatalogManaged, + commit_version: 7, + num_add_files: 2, + num_remove_files: 1, + num_dv_updates: 4, + add_files_bytes: 100, + remove_files_bytes: 50, + is_blind_append: true, + data_change: false, + operation: Some("WRITE".to_string()), + prepare_duration: Duration::from_nanos(10), + committer_duration: Duration::from_nanos(20), + total_duration: Duration::from_nanos(30), + }); + with_ffi_event(&event, |ffi| { + let MetricEvent::TransactionCommitSuccess(e) = ffi else { + panic!("expected TransactionCommitSuccess"); + }; + assert_eq!(e.operation_id.bytes, id.as_bytes()); + assert!(matches!(e.table_type, TableType::CatalogManaged)); + assert_eq!(e.commit_version, 7); + assert_eq!(e.num_dv_updates, 4); + assert_eq!(e.add_files_bytes, 100); + assert!(e.is_blind_append); + assert_eq!(e.prepare_duration_ns, 10); + assert_eq!(e.total_duration_ns, 30); + let op: &str = unsafe { TryFromStringSlice::try_from_slice(&e.operation).unwrap() }; + assert_eq!(op, "WRITE"); + let cid: &str = + unsafe { TryFromStringSlice::try_from_slice(&e.correlation_id).unwrap() }; + assert_eq!(cid, "req-7"); + }); + } + + #[test] + fn from_kernel_transaction_commit_success_unset_operation_is_empty_slice() { + let event = + kernel::MetricEvent::TransactionCommitSuccess(kernel::TransactionCommitSuccess { + operation_id: kernel::MetricId::new(), + correlation_id: None, + table_type: kernel::TableType::PathBased, + commit_version: 1, + num_add_files: 0, + num_remove_files: 0, + num_dv_updates: 0, + add_files_bytes: 0, + remove_files_bytes: 0, + is_blind_append: false, + data_change: true, + operation: None, + prepare_duration: Duration::from_nanos(0), + committer_duration: Duration::from_nanos(0), + total_duration: Duration::from_nanos(0), + }); + with_ffi_event(&event, |ffi| { + let MetricEvent::TransactionCommitSuccess(e) = ffi else { + panic!("expected TransactionCommitSuccess"); + }; + assert!(matches!(e.table_type, TableType::PathBased)); + let op: &str = unsafe { TryFromStringSlice::try_from_slice(&e.operation).unwrap() }; + assert_eq!(op, ""); + let cid: &str = + unsafe { TryFromStringSlice::try_from_slice(&e.correlation_id).unwrap() }; + assert_eq!(cid, ""); + }); + } + + #[test] + fn from_kernel_maps_failure_reason() { + let id = kernel::MetricId::new(); + let failure = + kernel::MetricEvent::TransactionCommitFailure(kernel::TransactionCommitFailure { + operation_id: id, + correlation_id: Some("req-fail".into()), + table_type: kernel::TableType::CatalogManaged, + reason: kernel::CommitFailureReason::Conflict, + }); + with_ffi_event(&failure, |ffi| { + let MetricEvent::TransactionCommitFailure(e) = ffi else { + panic!("expected TransactionCommitFailure"); + }; + assert!(matches!(e.reason, CommitFailureReason::Conflict)); + assert!(matches!(e.table_type, TableType::CatalogManaged)); + assert_eq!(e.operation_id.bytes, id.as_bytes()); + let cid: &str = + unsafe { TryFromStringSlice::try_from_slice(&e.correlation_id).unwrap() }; + assert_eq!(cid, "req-fail"); + }); + } + + #[test] + fn from_kernel_scan_metadata_completed_maps_all_fields() { + let id = kernel::MetricId::new(); + let event = kernel::MetricEvent::ScanMetadataCompleted(kernel::ScanMetadataCompleted { + operation_id: id, + correlation_id: Some("scan-req".into()), + table_type: kernel::TableType::CatalogManaged, + scan_type: kernel::ScanType::ParallelPhase, + duration: Duration::from_nanos(13), + num_add_files_seen: 17, + num_active_add_files: 19, + active_add_files_bytes: 23, + num_remove_files_seen: 29, + num_non_file_actions: 31, + num_predicate_filtered: 37, + peak_hash_set_size: 41, + dedup_visitor_time: Duration::from_nanos(43), + predicate_eval_time: Duration::from_nanos(47), + }); + with_ffi_event(&event, |ffi| { + let MetricEvent::ScanMetadataCompleted(e) = ffi else { + panic!("expected ScanMetadataCompleted"); + }; + assert_eq!(e.operation_id.bytes, id.as_bytes()); + assert!(matches!(e.table_type, TableType::CatalogManaged)); + let cid: &str = + unsafe { TryFromStringSlice::try_from_slice(&e.correlation_id).unwrap() }; + assert_eq!(cid, "scan-req"); + assert_eq!( + discriminant(&e.scan_type), + discriminant(&ScanType::ParallelPhase) + ); + assert_eq!(e.duration_ns, 13); + assert_eq!(e.num_add_files_seen, 17); + assert_eq!(e.num_active_add_files, 19); + assert_eq!(e.active_add_files_bytes, 23); + assert_eq!(e.num_remove_files_seen, 29); + assert_eq!(e.num_non_file_actions, 31); + assert_eq!(e.num_predicate_filtered, 37); + assert_eq!(e.peak_hash_set_size, 41); + assert_eq!(e.dedup_visitor_time_ns, 43); + assert_eq!(e.predicate_eval_time_ns, 47); + }); + } + + #[test] + fn from_kernel_protocol_metadata_load_success_maps_source() { + let id = kernel::MetricId::new(); + let event = + kernel::MetricEvent::ProtocolMetadataLoadSuccess(kernel::ProtocolMetadataLoadSuccess { + operation_id: id, + correlation_id: Some("pm-req".into()), + table_type: kernel::TableType::CatalogManaged, + load_type: kernel::LogSegmentLoadType::Incremental, + source: kernel::ProtocolMetadataSource::CrcAdvancedByReplay, + duration: Duration::from_nanos(53), + }); + with_ffi_event(&event, |ffi| { + let MetricEvent::ProtocolMetadataLoadSuccess(e) = ffi else { + panic!("expected ProtocolMetadataLoadSuccess"); + }; + assert_eq!(e.operation_id.bytes, id.as_bytes()); + assert!(matches!(e.table_type, TableType::CatalogManaged)); + assert!(matches!(e.load_type, LogSegmentLoadType::Incremental)); + let cid: &str = + unsafe { TryFromStringSlice::try_from_slice(&e.correlation_id).unwrap() }; + assert_eq!(cid, "pm-req"); + assert_eq!( + discriminant(&e.source), + discriminant(&ProtocolMetadataSource::CrcAdvancedByReplay) + ); + assert_eq!(e.duration_ns, 53); + }); + } + + #[test] + fn from_kernel_log_segment_load_success_carries_correlation_id_and_table_type() { + let id = kernel::MetricId::new(); + let event = kernel::MetricEvent::LogSegmentLoadSuccess(kernel::LogSegmentLoadSuccess { + operation_id: id, + correlation_id: Some("req-42".into()), + table_type: kernel::TableType::CatalogManaged, + load_type: kernel::LogSegmentLoadType::Full, + num_commit_files: 3, + num_checkpoint_files: 1, + num_compaction_files: 2, + crc_versions_behind: Some(5), + duration: Duration::from_nanos(61), + }); + with_ffi_event(&event, |ffi| { + let MetricEvent::LogSegmentLoadSuccess(e) = ffi else { + panic!("expected LogSegmentLoadSuccess"); + }; + assert_eq!(e.operation_id.bytes, id.as_bytes()); + assert!(matches!(e.table_type, TableType::CatalogManaged)); + assert_eq!(e.duration_ns, 61); + assert!(matches!(e.load_type, LogSegmentLoadType::Full)); + assert_eq!(e.num_commit_files, 3); + assert_eq!(e.num_checkpoint_files, 1); + assert_eq!(e.num_compaction_files, 2); + assert!(e.has_crc); + assert_eq!(e.crc_versions_behind, 5); + let cid: &str = + unsafe { TryFromStringSlice::try_from_slice(&e.correlation_id).unwrap() }; + assert_eq!(cid, "req-42"); + }); + } + + #[test] + fn from_kernel_log_segment_load_success_unset_correlation_id_is_empty_slice() { + let event = kernel::MetricEvent::LogSegmentLoadSuccess(kernel::LogSegmentLoadSuccess { + operation_id: kernel::MetricId::new(), + correlation_id: None, + table_type: kernel::TableType::PathBased, + load_type: kernel::LogSegmentLoadType::Full, + num_commit_files: 0, + num_checkpoint_files: 0, + num_compaction_files: 0, + crc_versions_behind: None, + duration: Duration::from_nanos(0), + }); + with_ffi_event(&event, |ffi| { + let MetricEvent::LogSegmentLoadSuccess(e) = ffi else { + panic!("expected LogSegmentLoadSuccess"); + }; + assert!(matches!(e.table_type, TableType::PathBased)); + assert!(!e.has_crc); + let cid: &str = + unsafe { TryFromStringSlice::try_from_slice(&e.correlation_id).unwrap() }; + assert_eq!(cid, ""); + }); + } + + #[test] + fn from_kernel_set_transaction_load_success_distinguishes_bools() { + // from_cache and found are adjacent bools that are trivial to swap; assert distinct values. + let event = + kernel::MetricEvent::SetTransactionLoadSuccess(kernel::SetTransactionLoadSuccess { + from_cache: true, + found: false, + duration: Duration::from_nanos(53), + }); + with_ffi_event(&event, |ffi| { + let MetricEvent::SetTransactionLoadSuccess(e) = ffi else { + panic!("expected SetTransactionLoadSuccess"); + }; + assert!(e.from_cache); + assert!(!e.found); + assert_eq!(e.duration_ns, 53); + }); + } +} diff --git a/ffi/src/ffi_test_utils.rs b/ffi/src/ffi_test_utils.rs index 67e960b259..3e9a28d6bc 100644 --- a/ffi/src/ffi_test_utils.rs +++ b/ffi/src/ffi_test_utils.rs @@ -1,9 +1,24 @@ //! Utility functions used for tests in this crate. -use crate::error::{EngineError, ExternResult, KernelError}; -use crate::{KernelStringSlice, NullableCvoid, TryFromStringSlice}; use std::os::raw::c_void; use std::ptr::NonNull; +#[cfg(test)] +use std::sync::Arc; + +#[cfg(test)] +use delta_kernel::object_store::memory::InMemory; +#[cfg(test)] +use delta_kernel_default_engine::DefaultEngineBuilder; +#[cfg(test)] +use test_utils::add_commit; + +use crate::error::{EngineError, ExternResult, KernelError}; +#[cfg(test)] +use crate::{ + engine_to_handle, get_snapshot_builder, kernel_string_slice, snapshot_builder_build, + SharedExternEngine, SharedSnapshot, +}; +use crate::{KernelStringSlice, NullableCvoid, TryFromStringSlice}; // Used to allocate EngineErrors with test information from Rust tests #[cfg(test)] @@ -56,27 +71,87 @@ pub(crate) fn ok_or_panic(result: ExternResult) -> T { } } +/// Build a latest-version snapshot via the FFI builder API. Panics on error. +#[cfg(test)] +pub(crate) unsafe fn build_snapshot( + path: KernelStringSlice, + engine: crate::handle::Handle, +) -> crate::handle::Handle { + let builder = ok_or_panic(get_snapshot_builder(path, engine)); + ok_or_panic(snapshot_builder_build(builder)) +} + +/// Wrap an already-seeded object store in an engine handle. Caller must free it. +/// +/// Needed because `get_default_engine` resolves the store from the URL, so a `memory://` path +/// builds a fresh (empty) `InMemory` rather than the seeded one. +#[cfg(test)] +pub(crate) fn engine_handle_for_store( + store: Arc, +) -> crate::handle::Handle { + let engine = DefaultEngineBuilder::new(store).build(); + engine_to_handle(Arc::new(engine), allocate_err) +} + +/// Create an in-memory engine and snapshot from the given commit data. Returns +/// `(engine_handle, snapshot_handle)` -- the caller must free both when done. +#[cfg(test)] +pub(crate) async fn setup_snapshot( + commit_data: String, +) -> Result< + ( + crate::handle::Handle, + crate::handle::Handle, + ), + Box, +> { + let table_root = "memory:///"; + let storage = Arc::new(InMemory::new()); + add_commit(table_root, storage.as_ref(), 0, commit_data).await?; + let engine = DefaultEngineBuilder::new(storage.clone()).build(); + let engine = engine_to_handle(Arc::new(engine), allocate_err); + let snap = unsafe { build_snapshot(kernel_string_slice!(table_root), engine.shallow_copy()) }; + Ok((engine, snap)) +} + /// Check error type and message while also recovering the error to prevent leaks pub(crate) fn assert_extern_result_error_with_message( res: ExternResult, expected_etype: KernelError, - expected_message: &str, + opt_message: Option<&str>, ) { match res { ExternResult::Err(e) => { let error = unsafe { recover_error(e) }; assert_eq!(error.etype, expected_etype); - assert_eq!(error.message, expected_message); + if let Some(expected_message) = opt_message { + assert_eq!(error.message, expected_message); + } } - _ => panic!("Expected error of type '{expected_etype:?}' and message '{expected_message}'"), + _ => panic!("Expected error of type '{expected_etype:?}' and message '{opt_message:?}'"), } } +/// Assert that `timestamp` (milliseconds since the Unix epoch) was written within the last day. +#[cfg(test)] +pub(crate) fn assert_timestamp_is_recent(timestamp: i64) { + let now_ms = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_millis() as i64; + let one_day_ms = 24 * 60 * 60 * 1000; + assert!( + (now_ms - one_day_ms..=now_ms).contains(×tamp), + "commit timestamp {timestamp} not within one day of now {now_ms}" + ); +} + #[cfg(test)] mod tests { - use super::*; use std::panic; + use super::*; + #[test] fn test_ok_or_panic_with_error() { // Create a test error diff --git a/ffi/src/ffi_tracing.rs b/ffi/src/ffi_tracing.rs index 231d59332f..0789b9a036 100644 --- a/ffi/src/ffi_tracing.rs +++ b/ffi/src/ffi_tracing.rs @@ -1,20 +1,46 @@ -//! FFI functions to allow engines to receive log and tracing events from kernel - -use std::sync::LazyLock; -use std::sync::{Arc, Mutex}; +//! FFI functions to allow engines to receive log, tracing, and metrics events from kernel. +//! +//! We use a single global tracing subscriber, registered the first time any of +//! [`enable_event_tracing`], [`enable_log_line_tracing`], [`enable_formatted_log_line_tracing`], or +//! [`enable_metrics_reporting`] is called. The subscriber has two layers, one for log events, and +//! one for metric report events: +//! +//! - The logging layer is a type-erased [`Layer`] that an `enable_*_tracing` call swaps in (either +//! event-based or formatted log-line) +//! - The metrics slot is a [`ReportGeneratorLayer`] that is `OFF` (zero overhead) until +//! [`enable_metrics_reporting`] turns it on. +//! +//! Both are reloadable so they can be swapped. + +use std::sync::{Arc, LazyLock, Mutex}; use std::{fmt, io}; -use delta_kernel::{DeltaResult, Error}; -use tracing::{error, warn}; -use tracing::{ - field::{Field as TracingField, Visit}, - Event as TracingEvent, Subscriber, +use delta_kernel::metrics::{ + MetricEvent as KernelMetricEvent, MetricsReporter, ReportGeneratorLayer, }; +use delta_kernel::{DeltaResult, Error}; +use tracing::field::{Field as TracingField, Visit}; +use tracing::{error, Event as TracingEvent, Subscriber}; +use tracing_core::Dispatch; +use tracing_subscriber::filter::LevelFilter; use tracing_subscriber::fmt::MakeWriter; -use tracing_subscriber::{filter::LevelFilter, layer::Context, registry::LookupSpan, Layer}; +use tracing_subscriber::layer::{Context, Identity, SubscriberExt}; +use tracing_subscriber::registry::LookupSpan; +use tracing_subscriber::{reload, Layer, Registry}; +use crate::ffi_metrics::{with_ffi_event, MetricEvent}; use crate::{kernel_string_slice, KernelStringSlice}; +/// A type-erased logging layer that lives in the reloadable logging slot of the global subscriber. +/// cbindgen:ignore +type BoxedLayer = Box + Send + Sync>; +/// Reload handle for swapping the boxed logging layer. +/// cbindgen:ignore +type LayerHandle = reload::Handle; +/// Reload handle for changing a slot's level filter. +/// cbindgen:ignore +type FilterHandle = reload::Handle; + /// Definitions of level verbosity. Verbose Levels are "greater than" less verbose ones. So /// Level::ERROR is the lowest, and Level::TRACE the highest. #[repr(C)] @@ -70,22 +96,22 @@ pub struct Event { target: KernelStringSlice, /// source file line number where the event occurred, or 0 (zero) if unknown line: u32, - /// file where the event occurred. If unknown the slice `ptr` will be null and the len will be 0 + /// file where the event occurred. If unknown the slice `ptr` will be null and the len will be + /// 0 file: KernelStringSlice, } pub type TracingEventFn = extern "C" fn(event: Event); /// Enable getting called back for tracing (logging) events in the kernel. `max_level` specifies -/// that only events `<=` to the specified level should be reported. More verbose Levels are "greater -/// than" less verbose ones. So Level::ERROR is the lowest, and Level::TRACE the highest. +/// that only events `<=` to the specified level should be reported. More verbose Levels are +/// "greater than" less verbose ones. So Level::ERROR is the lowest, and Level::TRACE the highest. /// -/// Note that setting up such a call back can only be done ONCE. Calling any of -/// `enable_event_tracing`, `enable_log_line_tracing`, or `enable_formatted_log_line_tracing` more -/// than once is a no-op. +/// Calling `enable_event_tracing`, `enable_log_line_tracing`, or +/// `enable_formatted_log_line_tracing` again replaces the active logging layer and its level, +/// including switching between event-based and log-line formats. /// -/// Returns `true` if the callback was setup successfully, false on failure (i.e. if called a second -/// time) +/// Returns `true` if the callback was setup successfully, `false` on failure. /// /// Event-based tracing gives an engine maximal flexibility in formatting event log /// lines. Kernel can also format events for the engine. If this is desired call @@ -128,7 +154,8 @@ pub enum LogLineFormat { /// structured logs are consumed as JSON by analysis and viewing tools. The JSON output is not /// optimized for human readability. /// Example: - /// `{"timestamp":"2022-02-15T18:47:10.821315Z","level":"INFO","fields":{"message":"preparing to shave yaks","number_of_yaks":3},"target":"fmt_json"}` + /// `{"timestamp":"2022-02-15T18:47:10.821315Z","level":"INFO","fields":{"message":"preparing + /// to shave yaks","number_of_yaks":3},"target":"fmt_json"}` JSON, } @@ -145,12 +172,11 @@ pub enum LogLineFormat { /// /// Log lines passed to the callback will already have a newline at the end. /// -/// Note that setting up such a call back can only be done ONCE. Calling any of -/// `enable_event_tracing`, `enable_log_line_tracing`, or `enable_formatted_log_line_tracing` more -/// than once is a no-op. +/// Calling `enable_event_tracing`, `enable_log_line_tracing`, or +/// `enable_formatted_log_line_tracing` again replaces the active logging layer and its level, +/// including switching between event-based and log-line formats. /// -/// Returns `true` if the callback was setup successfully, false on failure (i.e. if called a second -/// time) +/// Returns `true` if the callback was setup successfully, `false` on failure. /// /// Log line based tracing is simple for an engine as it can just log the passed string, but does /// not provide flexibility for an engine to format events. If the engine wants to use a specific @@ -179,12 +205,11 @@ pub unsafe extern "C" fn enable_log_line_tracing( /// formatting options for the log lines. See [`enable_log_line_tracing`] for general info on /// getting called back for log lines. /// -/// Note that setting up such a call back can only be done ONCE. Calling any of -/// `enable_event_tracing`, `enable_log_line_tracing`, or `enable_formatted_log_line_tracing` more -/// than once is a no-op. +/// Calling `enable_event_tracing`, `enable_log_line_tracing`, or +/// `enable_formatted_log_line_tracing` again replaces the active logging layer and its level, +/// including switching between event-based and log-line formats. /// -/// Returns `true` if the callback was setup successfully, false on failure (i.e. if called a second -/// time) +/// Returns `true` if the callback was setup successfully, `false` on failure. /// /// Options that can be set: /// - `format`: see [`LogLineFormat`] @@ -244,7 +269,7 @@ impl Visit for MessageFieldVisitor { } struct EventLayer { - callback: Arc>, + callback: TracingEventFn, } impl Layer for EventLayer @@ -269,149 +294,16 @@ where line: metadata.line().unwrap_or(0), file: kernel_string_slice!(file), }; - if let Ok(cb) = self.callback.lock() { - (cb)(event); - } else { - error!("Failed to lock event callback (mutex poisoned)."); - } + (self.callback)(event); } } } -struct GlobalTracingState { - dispatch: Option, - reload_handle: - Option>, - /// callback for event subscriber - event_callback: Option>>, - /// callback for log line subscriber - log_line_callback: Option>>, -} - -impl GlobalTracingState { - fn uninitialized() -> Self { - GlobalTracingState { - dispatch: None, - reload_handle: None, - event_callback: None, - log_line_callback: None, - } - } - - fn register_event_callback( - &mut self, - callback: TracingEventFn, - max_level: Level, - ) -> DeltaResult<()> { - if !max_level.is_valid() { - return Err(Error::generic("max_level out of range")); - } - - if let (Some(reload), Some(event_cb)) = (&self.reload_handle, &self.event_callback) { - let mut event_cb = event_cb.lock().map_err(|_e| { - Error::generic("Failed to acquire lock for event callback (mutex poisoned).") - })?; - *event_cb = callback; - return reload.reload(LevelFilter::from(max_level)).map_err(|e| { - warn!("Failed to reload tracing level: {e}"); - Error::generic(format!("Unable to reload subscriber: {e}")) - }); - } - - let (dispatch, reload_handle, event_callback) = create_event_dispatch(callback, max_level); - set_global_default(dispatch.clone())?; - self.dispatch = Some(dispatch); - self.reload_handle = Some(reload_handle); - self.event_callback = Some(event_callback); - Ok(()) - } - - #[allow(clippy::too_many_arguments)] - fn register_log_line_callback( - &mut self, - callback: TracingLogLineFn, - max_level: Level, - format: LogLineFormat, - ansi: bool, - with_time: bool, - with_level: bool, - with_target: bool, - ) -> DeltaResult<()> { - if !max_level.is_valid() { - return Err(Error::generic("max_level out of range")); - } - - if let (Some(reload), Some(log_cb)) = (&self.reload_handle, &self.log_line_callback) { - let mut log_cb = log_cb.lock().map_err(|_e| { - Error::generic("Failed to acquire lock for log callback (mutex poisoned).") - })?; - *log_cb = callback; - return reload.reload(LevelFilter::from(max_level)).map_err(|e| { - warn!("Failed to reload log level: {e}"); - Error::generic(format!("Unable to reload subscriber: {e}")) - }); - } - - let (dispatch, reload_handle, log_line_callback) = create_log_line_dispatch( - callback, - max_level, - format, - ansi, - with_time, - with_level, - with_target, - ); - set_global_default(dispatch.clone())?; - self.dispatch = Some(dispatch); - self.reload_handle = Some(reload_handle); - self.log_line_callback = Some(log_line_callback); - Ok(()) - } -} - -static TRACING_STATE: LazyLock> = - LazyLock::new(|| Mutex::new(GlobalTracingState::uninitialized())); - -fn create_event_dispatch( - callback: TracingEventFn, - max_level: Level, -) -> ( - tracing_core::Dispatch, - tracing_subscriber::reload::Handle, - Arc>, -) { - use tracing_subscriber::{layer::SubscriberExt, registry::Registry}; - - let callback_arc = Arc::new(Mutex::new(callback)); - let (filter_layer, reload_handle) = - tracing_subscriber::reload::Layer::new(LevelFilter::from(max_level)); - let event_layer = EventLayer { - callback: callback_arc.clone(), - } - .with_filter(filter_layer); - - let subscriber = Registry::default().with(event_layer); - ( - tracing_core::Dispatch::new(subscriber), - reload_handle, - callback_arc, - ) -} - -fn setup_event_subscriber(callback: TracingEventFn, max_level: Level) -> DeltaResult<()> { - let mut state = TRACING_STATE - .lock() - .map_err(|_e| Error::generic("Poisoned mutex while setting up event subscriber"))?; - state.register_event_callback(callback, max_level) -} - -// utility code below for setting up the tracing subscriber for log lines - type SharedBuffer = Arc>>; struct TriggerLayer { buf: SharedBuffer, - callback: Arc>, + callback: TracingLogLineFn, } impl Layer for TriggerLayer @@ -423,21 +315,13 @@ where Ok(mut buf) => { let message = String::from_utf8_lossy(&buf); let message = kernel_string_slice!(message); - if let Ok(cb) = self.callback.lock() { - (cb)(message); - } else { - error!("Failed to lock event callback (mutex poisoned)."); - } + (self.callback)(message); buf.clear(); } Err(_) => { let message = "INTERNAL KERNEL ERROR: Could not lock message buffer."; let message = kernel_string_slice!(message); - if let Ok(cb) = self.callback.lock() { - (cb)(message); - } else { - error!("Failed to lock event callback (mutex poisoned)."); - } + (self.callback)(message); } } } @@ -472,68 +356,208 @@ impl<'a> MakeWriter<'a> for BufferedMessageWriter { } } -fn create_log_line_dispatch( +/// Callback an engine registers via [`enable_metrics_reporting`] to receive kernel +/// [`MetricEvent`]s. This is invoked synchronously on the thread that emitted the event. Note that +/// the `event`, and any [`KernelStringSlice`] it carries, are only valid for the duration of the +/// call. +pub type MetricsEventFn = extern "C" fn(event: MetricEvent); + +/// A [`MetricsReporter`] that forwards each kernel [`MetricEvent`] to a registered FFI callback +#[derive(Debug)] +struct FfiMetricsReporter { + callback: Arc>>, +} + +impl MetricsReporter for FfiMetricsReporter { + fn report(&self, event: KernelMetricEvent) { + let callback = match self.callback.lock() { + Ok(guard) => *guard, + Err(_) => { + error!("Failed to lock metrics callback (mutex poisoned)."); + return; + } + }; + if let Some(callback) = callback { + with_ffi_event(&event, |ffi_event| callback(ffi_event)); + } + } +} + +fn build_event_layer(callback: TracingEventFn) -> BoxedLayer { + Box::new(EventLayer { callback }) +} + +fn build_log_line_layer( callback: TracingLogLineFn, - max_level: Level, format: LogLineFormat, ansi: bool, with_time: bool, with_level: bool, with_target: bool, -) -> ( - tracing_core::Dispatch, - tracing_subscriber::reload::Handle, - Arc>, -) { - use tracing_subscriber::{layer::SubscriberExt, registry::Registry}; - - let buffer = Arc::new(Mutex::new(vec![])); +) -> BoxedLayer { + let buffer: SharedBuffer = Arc::new(Mutex::new(vec![])); let writer = BufferedMessageWriter { current_buffer: buffer.clone(), }; - let fmt_layer = tracing_subscriber::fmt::layer() .with_writer(writer) .with_ansi(ansi) .with_level(with_level) .with_target(with_target); - - let (filter_layer, reload_handle) = - tracing_subscriber::reload::Layer::new(LevelFilter::from(max_level)); - - let callback_arc = Arc::new(Mutex::new(callback)); - let tracking_layer = TriggerLayer { - buf: buffer.clone(), - callback: callback_arc.clone(), + let trigger = TriggerLayer { + buf: buffer, + callback, }; - macro_rules! setup_subscriber { + // The fmt layer formats the line into the shared buffer; the trigger layer then flushes it to + // the callback. fmt must run before trigger, so it must be the first item in the vec. + macro_rules! boxed_log_line { ($($transform:ident()).*) => {{ - let subscriber = Registry::default() - .with(filter_layer) - .with(fmt_layer) - .with(tracking_layer); - ( - tracing_core::Dispatch::new(subscriber), - reload_handle, - callback_arc.clone(), - ) + let fmt: BoxedLayer = Box::new(fmt_layer $(.$transform())*); + let trigger: BoxedLayer = Box::new(trigger); + Box::new(vec![fmt, trigger]) as BoxedLayer }}; } use LogLineFormat::*; match (format, with_time) { - (FULL, true) => setup_subscriber!(), - (FULL, false) => setup_subscriber!(without_time()), - (COMPACT, true) => setup_subscriber!(compact()), - (COMPACT, false) => setup_subscriber!(compact().without_time()), - (PRETTY, true) => setup_subscriber!(pretty()), - (PRETTY, false) => setup_subscriber!(pretty().without_time()), - (JSON, true) => setup_subscriber!(json()), - (JSON, false) => setup_subscriber!(json().without_time()), + (FULL, true) => boxed_log_line!(), + (FULL, false) => boxed_log_line!(without_time()), + (COMPACT, true) => boxed_log_line!(compact()), + (COMPACT, false) => boxed_log_line!(compact().without_time()), + (PRETTY, true) => boxed_log_line!(pretty()), + (PRETTY, false) => boxed_log_line!(pretty().without_time()), + (JSON, true) => boxed_log_line!(json()), + (JSON, false) => boxed_log_line!(json().without_time()), } } +struct GlobalTracingState { + installed: bool, + /// Swaps the active logging layer (event-based vs log-line). + logging_layer: Option, + /// Sets the level filter gating the logging layer. + logging_filter: Option, + /// Toggles the metrics layer on (`TRACE`) or off (`OFF`). + metrics_filter: Option, + /// Shared callback the metrics layer's reporter forwards events to. + metrics_callback: Arc>>, +} + +impl GlobalTracingState { + fn uninitialized() -> Self { + GlobalTracingState { + installed: false, + logging_layer: None, + logging_filter: None, + metrics_filter: None, + metrics_callback: Arc::new(Mutex::new(None)), + } + } + + /// If the global subscriber hasn't been installed yet, this installs it and sets up all the + /// logging layers. When called again, this is a no-op. + fn ensure_installed(&mut self) -> DeltaResult<()> { + if self.installed { + return Ok(()); + } + + let noop: BoxedLayer = Box::new(Identity::new()); + let (logging_slot, logging_layer) = reload::Layer::new(noop); + let (logging_filter_layer, logging_filter) = reload::Layer::new(LevelFilter::OFF); + let logging: BoxedLayer = Box::new(logging_slot.with_filter(logging_filter_layer)); + + let (metrics_filter_layer, metrics_filter) = reload::Layer::new(LevelFilter::OFF); + let reporter = Arc::new(FfiMetricsReporter { + callback: self.metrics_callback.clone(), + }); + let metrics: BoxedLayer = + Box::new(ReportGeneratorLayer::new(reporter).with_filter(metrics_filter_layer)); + + let subscriber = Registry::default().with(vec![logging, metrics]); + set_global_default(Dispatch::new(subscriber))?; + + self.logging_layer = Some(logging_layer); + self.logging_filter = Some(logging_filter); + self.metrics_filter = Some(metrics_filter); + self.installed = true; + Ok(()) + } + + /// Make `layer` the active logging layer and set its level filter. + fn reload_logging(&self, layer: BoxedLayer, max_level: Level) -> DeltaResult<()> { + let reload_err = |e| Error::generic(format!("Unable to reload logging subscriber: {e}")); + self.logging_layer + .as_ref() + .ok_or_else(|| Error::generic("logging slot not installed"))? + .reload(layer) + .map_err(reload_err)?; + self.logging_filter + .as_ref() + .ok_or_else(|| Error::generic("logging filter not installed"))? + .reload(LevelFilter::from(max_level)) + .map_err(reload_err) + } + + fn register_event_callback( + &mut self, + callback: TracingEventFn, + max_level: Level, + ) -> DeltaResult<()> { + if !max_level.is_valid() { + return Err(Error::generic("max_level out of range")); + } + self.ensure_installed()?; + self.reload_logging(build_event_layer(callback), max_level) + } + + #[allow(clippy::too_many_arguments)] + fn register_log_line_callback( + &mut self, + callback: TracingLogLineFn, + max_level: Level, + format: LogLineFormat, + ansi: bool, + with_time: bool, + with_level: bool, + with_target: bool, + ) -> DeltaResult<()> { + if !max_level.is_valid() { + return Err(Error::generic("max_level out of range")); + } + self.ensure_installed()?; + let layer = + build_log_line_layer(callback, format, ansi, with_time, with_level, with_target); + self.reload_logging(layer, max_level) + } + + /// Set the metrics callback and turn the metrics slot's filter on. Metric spans are emitted at + /// `INFO`. + fn register_metrics_callback(&mut self, callback: MetricsEventFn) -> DeltaResult<()> { + self.ensure_installed()?; + *self + .metrics_callback + .lock() + .map_err(|_| Error::generic("Failed to lock metrics callback (mutex poisoned)."))? = + Some(callback); + self.metrics_filter + .as_ref() + .ok_or_else(|| Error::generic("metrics filter not installed"))? + .reload(LevelFilter::INFO) + .map_err(|e| Error::generic(format!("Unable to reload metrics subscriber: {e}"))) + } +} + +static TRACING_STATE: LazyLock> = + LazyLock::new(|| Mutex::new(GlobalTracingState::uninitialized())); + +fn setup_event_subscriber(callback: TracingEventFn, max_level: Level) -> DeltaResult<()> { + let mut state = TRACING_STATE + .lock() + .map_err(|_e| Error::generic("Poisoned mutex while setting up event subscriber"))?; + state.register_event_callback(callback, max_level) +} + fn setup_log_line_subscriber( callback: TracingLogLineFn, max_level: Level, @@ -557,24 +581,73 @@ fn setup_log_line_subscriber( ) } +/// Enable getting called back with structured kernel metric events. `callback` receives a +/// [`MetricEvent`] each time kernel emits a report. (See the [`delta_kernel::metrics`] module). +/// +/// Calling this replaces any previously set callback. +/// +/// Returns `true` if reporting was enabled successfully, `false` on failure. +/// +/// # Safety +/// Caller must pass a valid function pointer for the callback. +#[no_mangle] +pub unsafe extern "C" fn enable_metrics_reporting(callback: MetricsEventFn) -> bool { + setup_metrics_reporter(callback).is_ok() +} + +fn setup_metrics_reporter(callback: MetricsEventFn) -> DeltaResult<()> { + let mut state = TRACING_STATE + .lock() + .map_err(|_e| Error::generic("Poisoned mutex while setting up metrics reporter"))?; + state.register_metrics_callback(callback) +} + #[cfg(test)] mod tests { use std::sync::LazyLock; - use tracing::debug; - use tracing::info; - use tracing::trace; + use tracing::{debug, info, trace}; use tracing_subscriber::fmt::time::FormatTime; - use crate::TryFromStringSlice; - use super::*; + use crate::TryFromStringSlice; // Because we have to access a global messages buffer, we have to force tests to run one at a // time static TEST_LOCK: LazyLock> = LazyLock::new(|| Mutex::new(())); static MESSAGES: Mutex>> = Mutex::new(None); + // Local dispatch builders used with `with_default` so tests can exercise each layer in + // isolation without consuming the once-only global default subscriber. + fn create_event_dispatch(callback: TracingEventFn, max_level: Level) -> Dispatch { + let layer = build_event_layer(callback).with_filter(LevelFilter::from(max_level)); + Dispatch::new(Registry::default().with(layer)) + } + + #[allow(clippy::too_many_arguments)] + fn create_log_line_dispatch( + callback: TracingLogLineFn, + max_level: Level, + format: LogLineFormat, + ansi: bool, + with_time: bool, + with_level: bool, + with_target: bool, + ) -> Dispatch { + let layer = + build_log_line_layer(callback, format, ansi, with_time, with_level, with_target) + .with_filter(LevelFilter::from(max_level)); + Dispatch::new(Registry::default().with(layer)) + } + + fn create_metrics_dispatch(callback: MetricsEventFn) -> Dispatch { + let reporter = Arc::new(FfiMetricsReporter { + callback: Arc::new(Mutex::new(Some(callback))), + }); + let layer = ReportGeneratorLayer::new(reporter).with_filter(LevelFilter::TRACE); + Dispatch::new(Registry::default().with(layer)) + } + fn record_callback_with_filter(line: KernelStringSlice, expected_log_lines: Vec<&str>) { let line_str: &str = unsafe { TryFromStringSlice::try_from_slice(&line).unwrap() }; let line_str = line_str.to_string(); @@ -590,7 +663,8 @@ mod tests { } } - // Note: record callbacks must be extern "C". Thus we cannot construct test callback closures in runtime. + // Note: record callbacks must be extern "C". Thus we cannot construct test callback closures in + // runtime. extern "C" fn record_callback_with_filter_1(line: KernelStringSlice) { record_callback_with_filter(line, vec!["Testing 1\n", "Another line\n"]) } @@ -731,7 +805,7 @@ mod tests { fn info_logs_with_formatted_log_line_tracing() { let _lock = TEST_LOCK.lock().unwrap(); setup_messages(); - let (dispatch, _, _) = create_log_line_dispatch( + let dispatch = create_log_line_dispatch( record_callback_with_filter_1, Level::INFO, LogLineFormat::COMPACT, @@ -840,8 +914,7 @@ mod tests { fn trace_event_tracking() { let _lock = TEST_LOCK.lock().unwrap(); setup_events(); - let (dispatch, _filter, _) = - create_event_dispatch(event_callback_with_filter_1, Level::TRACE); + let dispatch = create_event_dispatch(event_callback_with_filter_1, Level::TRACE); tracing_core::dispatcher::with_default(&dispatch, || { let lines = ["Testing 1", "Another line"]; for line in lines { @@ -852,7 +925,8 @@ mod tests { } #[test] - #[ignore] // We cannot run this test if test_enable_log_line_tracing was run before - see comment there, however this test works if run individually. + #[ignore] // We cannot run this test if test_enable_log_line_tracing was run before - see comment there, + // however this test works if run individually. fn test_enable_event_tracing() { let _lock = TEST_LOCK.lock().unwrap(); setup_events(); @@ -904,4 +978,37 @@ mod tests { let error: Level = (&tracing::Level::ERROR).into(); assert_eq!(error, Level::ERROR); } + + static METRIC_EVENTS: Mutex>> = Mutex::new(None); + + extern "C" fn capture_metric_event(event: MetricEvent) { + let desc = match event { + MetricEvent::JsonReadCompleted(e) => format!("json:{}:{}", e.num_files, e.bytes_read), + MetricEvent::ParquetReadCompleted(e) => { + format!("parquet:{}:{}", e.num_files, e.bytes_read) + } + _ => "other".to_string(), + }; + if let Ok(mut lock) = METRIC_EVENTS.lock() { + if let Some(events) = lock.as_mut() { + events.push(desc); + } + } + } + + #[test] + fn metrics_reporting_delivers_structured_events() { + let _lock = TEST_LOCK.lock().unwrap(); + *METRIC_EVENTS.lock().unwrap() = Some(vec![]); + let dispatch = create_metrics_dispatch(capture_metric_event); + tracing_core::dispatcher::with_default(&dispatch, || { + delta_kernel::metrics::emit_json_read_completed(3, 100); + delta_kernel::metrics::emit_parquet_read_completed(2, 50); + }); + let lock = METRIC_EVENTS.lock().unwrap(); + assert_eq!( + lock.as_deref(), + Some(["json:3:100".to_string(), "parquet:2:50".to_string()].as_slice()) + ); + } } diff --git a/ffi/src/handle.rs b/ffi/src/handle.rs index 7af31de905..9d2c2bc023 100644 --- a/ffi/src/handle.rs +++ b/ffi/src/handle.rs @@ -3,11 +3,11 @@ //! //! Creating a [`Handle`] always implies some kind of ownership transfer. A mutable handle takes //! ownership of the object itself (analogous to [`Box`]), while a non-mutable (shared) handle -//! takes ownership of a shared reference to the object (analogous to [`std::sync::Arc`]). Thus, a created -//! handle remains [valid][Handle#Validity], and its underlying object remains accessible, until the -//! handle is explicitly dropped or consumed. Dropping a mutable handle always drops the underlying -//! object as well; dropping a shared handle only drops the underlying object if the handle was the -//! last reference to that object. +//! takes ownership of a shared reference to the object (analogous to [`std::sync::Arc`]). Thus, +//! a created handle remains [valid][Handle#Validity], and its underlying object remains accessible, +//! until the handle is explicitly dropped or consumed. Dropping a mutable handle always drops the +//! underlying object as well; dropping a shared handle only drops the underlying object if the +//! handle was the last reference to that object. //! //! Because handles carry ownership semantics, and lifetime information is not preserved across the //! FFI boundary, handles are always opaque types to avoid confusiong them with normal references @@ -39,7 +39,6 @@ /// /// #[handle_descriptor(target = dyn Bar, mutable = false)] /// pub struct SharedBar; -/// /// ``` /// /// NOTE: For obscure rust generic type constraint reasons, a `HandleDescriptor` must specifically @@ -194,8 +193,8 @@ mod private { /// * The handle is [valid][Handle#Validity]. /// /// * No other references (neither shared nor mutable) can overlap with the returned - /// reference. In particular, this means the caller must ensure no other thread can access this - /// handle while the returned reference is still alive. + /// reference. In particular, this means the caller must ensure no other thread can access + /// this handle while the returned reference is still alive. pub unsafe fn as_mut(&mut self) -> &mut T { H::as_mut(self.ptr.cast().as_ptr()) } @@ -497,9 +496,10 @@ pub use private::{Boolean, False, Handle, True, Unconstructable}; mod tests { use std::sync::Arc; - use super::*; use delta_kernel_ffi_macros::handle_descriptor; + use super::*; + #[allow(dead_code)] #[derive(Debug)] pub struct Foo { diff --git a/ffi/src/incremental_scan.rs b/ffi/src/incremental_scan.rs new file mode 100644 index 0000000000..a3f82b1415 --- /dev/null +++ b/ffi/src/incremental_scan.rs @@ -0,0 +1,1117 @@ +//! FFI bindings for the incremental scan API. +//! +//! An incremental scan streams the file-action diff between a base version and a target +//! [`SharedSnapshot`]. The flow mirrors the Rust API: +//! +//! ```text +//! snapshot_incremental_scan_builder(snapshot, base_version, engine) +//! -> incremental_scan_builder_with_predicate(builder, engine, predicate) // optional; prunes live Adds +//! -> incremental_scan_builder_build(builder) // -> OptionalValue; None => full-scan fallback +//! ``` +//! +//! Pull the diff's live-Add batches as Arrow with repeated +//! [`incremental_scan_stream_next_arrow`] calls (newest-commit first), then consume the stream +//! with [`incremental_scan_stream_into_summary`] to recover the live-Add and Remove key sets. +//! `into_summary` is the terminal consumer: it drains any batches `next_arrow` left behind, then +//! frees the stream. Any error from `next_arrow` kills the stream, so a later `next_arrow` or +//! `into_summary` call fails; rebuild the stream to retry. +//! +//! [`incremental_scan_builder_build`] returns [`OptionalValue::None`] (not an error) when the +//! target snapshot's commit list can't cover the range, which is the signal to fall back to a +//! full scan. +//! +//! The Arrow batch step (`incremental_scan_stream_next_arrow`, which reuses +//! [`crate::scan::ScanMetadataArrowResult`]) requires the `default-engine-base` feature; the +//! builder, summary, and visitor entry points are always available. +//! +//! Pass-through fields decoded from a batch (`stats`, `partitionValues`, `baseRowId`) must be +//! interpreted against the protocol at that row's own commit version, not against the target +//! snapshot: in-range protocol/metadata changes are not yet surfaced (see +//! ). For example, a `columnMapping` +//! change within the range shifts the physical column names keying `stats` and `partitionValues`, +//! so decoding older-commit rows against the target snapshot's names reads them wrong. + +use std::collections::HashSet; +use std::sync::{Arc, Mutex}; + +use delta_kernel::incremental_scan::{IncrementalScanStream, IncrementalScanSummary}; +use delta_kernel::log_replay::FileActionKey; +use delta_kernel::snapshot::SnapshotRef; +use delta_kernel::{DeltaResult, Error, PredicateRef, Version}; +use delta_kernel_ffi_macros::handle_descriptor; + +#[cfg(feature = "default-engine-base")] +use crate::engine_data::ArrowFFIData; +use crate::handle::Handle; +use crate::scan::{decode_engine_predicate, EnginePredicate}; +#[cfg(feature = "default-engine-base")] +use crate::scan::{CTransforms, ScanMetadataArrowResult}; +use crate::{ + kernel_string_slice, ExternEngine, ExternResult, IntoExternResult, KernelStringSlice, + NullableCvoid, OptionalValue, SharedExternEngine, SharedSnapshot, +}; + +/// Opaque builder for constructing an [`IncrementalScanStream`] from a snapshot. +/// +/// Create with [`snapshot_incremental_scan_builder`]. Call [`incremental_scan_builder_build`] to +/// consume the builder and obtain the stream, or [`free_incremental_scan_builder`] to drop it +/// without building. +pub struct FfiIncrementalScanBuilder { + engine: Arc, + target_snapshot: SnapshotRef, + base_version: Version, + predicate: Option, +} + +/// An opaque handle with exclusive (Box-like) ownership of a [`FfiIncrementalScanBuilder`]. +#[handle_descriptor(target=FfiIncrementalScanBuilder, mutable=true, sized=true)] +pub struct MutableFfiIncrementalScanBuilder; + +/// An incremental scan stream, guarded by a mutex so it can cross the FFI boundary as a shared +/// handle. The stream itself is single-consumer; the mutex serializes concurrent `next` calls. +/// +/// The engine `Arc` is retained so the JSON reader's runtime outlives the stream even if the +/// caller drops its own engine handle first. +pub struct FfiIncrementalScanStream { + stream: Mutex>, + engine: Arc, +} + +/// An opaque, shared handle owning an [`FfiIncrementalScanStream`]. Release with +/// [`free_incremental_scan_stream`], or consume it with +/// [`incremental_scan_stream_into_summary`]. +#[handle_descriptor(target=FfiIncrementalScanStream, mutable=false, sized=true)] +pub struct SharedIncrementalScanStream; + +/// An opaque, shared handle owning an [`IncrementalScanSummary`]. Release with +/// [`free_incremental_scan_summary`]. +#[handle_descriptor(target=IncrementalScanSummary, mutable=false, sized=true)] +pub struct SharedIncrementalScanSummary; + +/// Get a builder for an incremental scan over the range `(base_version, snapshot.version()]`. +/// +/// The caller owns the returned handle and must eventually call either +/// [`incremental_scan_builder_build`] to produce a stream, or [`free_incremental_scan_builder`] +/// to drop it without building. Does not consume the snapshot handle. +/// +/// # Safety +/// +/// Caller must pass a valid snapshot handle and engine handle. +#[no_mangle] +pub unsafe extern "C" fn snapshot_incremental_scan_builder( + snapshot: Handle, + base_version: Version, + engine: Handle, +) -> Handle { + let target_snapshot = unsafe { snapshot.clone_as_arc() }; + let engine = unsafe { engine.clone_as_arc() }; + Box::new(FfiIncrementalScanBuilder { + engine, + target_snapshot, + base_version, + predicate: None, + }) + .into() +} + +/// Apply a predicate to an incremental scan builder to prune streamed live Adds by their file +/// stats (data-skipping). Removes are never pruned; skipping is conservative, so the engine must +/// still re-apply the predicate at read time. +/// +/// Consumes the `builder` handle and returns a new handle with the predicate applied. The +/// `builder` handle must not be used after this call. A later call replaces any previously +/// applied predicate. Returns an error if the engine's predicate visitor fails to produce a valid +/// predicate (i.e. returns an invalid expression ID); on error, the builder is dropped. +/// +/// # Safety +/// +/// `builder` and `engine` must be valid handles. The `builder` handle must not be used after this +/// call. `predicate` must be a valid, non-null [`EnginePredicate`] whose `visitor` and `predicate` +/// fields are safe to call and read. +#[no_mangle] +pub unsafe extern "C" fn incremental_scan_builder_with_predicate( + builder: Handle, + engine: Handle, + predicate: &mut EnginePredicate, +) -> ExternResult> { + let engine = unsafe { engine.as_ref() }; + let builder = unsafe { builder.into_inner() }; + incremental_scan_builder_with_predicate_impl(*builder, predicate).into_extern_result(&engine) +} + +fn incremental_scan_builder_with_predicate_impl( + mut builder: FfiIncrementalScanBuilder, + predicate: &mut EnginePredicate, +) -> DeltaResult> { + builder.predicate = Some(Arc::new(decode_engine_predicate(predicate)?)); + Ok(Box::new(builder).into()) +} + +/// Consume the builder and return the incremental scan stream, or [`OptionalValue::None`] when +/// the target snapshot's commit list can't cover `(base_version, target_version]`. +/// +/// [`OptionalValue::None`] is not an error. It means an incremental advance isn't possible +/// (the target snapshot no longer retains commit `base_version + 1`, the first version in the +/// range, typically because a checkpoint above `base_version` truncated it), so the caller +/// should fall back to a full scan. The builder is always freed by this call, whether or not it +/// succeeds. +/// +/// Returns an error if `base_version >= target_version`, the target snapshot's protocol has an +/// unsupported reader feature, a [`incremental_scan_builder_with_predicate`] predicate references +/// a column absent from the table schema, or the engine fails to open the commit stream. +/// +/// # Safety +/// +/// Caller must pass a valid builder pointer and must not use it again after this call. +#[no_mangle] +pub unsafe extern "C" fn incremental_scan_builder_build( + builder: Handle, +) -> ExternResult>> { + let builder = unsafe { builder.into_inner() }; + let engine = builder.engine.clone(); + incremental_scan_builder_build_impl(*builder).into_extern_result(&engine.as_ref()) +} + +fn incremental_scan_builder_build_impl( + builder: FfiIncrementalScanBuilder, +) -> DeltaResult>> { + let engine = builder.engine.engine(); + let maybe_stream = builder + .target_snapshot + .incremental_scan_builder(builder.base_version) + .with_predicate(builder.predicate) + .build(engine.as_ref())?; + let handle = maybe_stream.map(|stream| { + Arc::new(FfiIncrementalScanStream { + stream: Mutex::new(Some(stream)), + engine: builder.engine, + }) + .into() + }); + Ok(handle.into()) +} + +/// Free an incremental scan builder without building (e.g. on an error path). +/// +/// # Safety +/// +/// Caller must pass a valid builder pointer and must not use it again after this call. +#[no_mangle] +pub unsafe extern "C" fn free_incremental_scan_builder( + builder: Handle, +) { + builder.drop_handle(); +} + +/// Get the next live-Add batch from the stream as Arrow via the C Data Interface. +/// +/// Returns `Ok(non-null)` with the next [`ScanMetadataArrowResult`] (an Arrow batch paired with a +/// boolean selection vector; its `transforms` are always null, as an incremental scan never +/// rewrites rows), `Ok(null)` when the stream is exhausted, or `Err` on failure. Only rows +/// selected by the vector are live Adds; batches are yielded newest-commit first. The engine must +/// free each non-null result with [`crate::scan::free_scan_metadata_arrow_result`]. +/// +/// Any error kills the stream: later [`incremental_scan_stream_next_arrow`] and +/// [`incremental_scan_stream_into_summary`] calls both fail, so no partial summary is exposed after +/// a failure. Rebuild the stream to retry. +/// +/// Decode pass-through fields against each row's own commit protocol, not the target snapshot; +/// see the module-level note on . +/// +/// # Safety +/// +/// Caller must pass a valid stream handle. +#[cfg(feature = "default-engine-base")] +#[no_mangle] +pub unsafe extern "C" fn incremental_scan_stream_next_arrow( + stream: Handle, +) -> ExternResult<*mut ScanMetadataArrowResult> { + let stream = unsafe { stream.as_ref() }; + incremental_scan_stream_next_arrow_impl(stream).into_extern_result(&stream.engine.as_ref()) +} + +#[cfg(feature = "default-engine-base")] +fn incremental_scan_stream_next_arrow_impl( + stream: &FfiIncrementalScanStream, +) -> DeltaResult<*mut ScanMetadataArrowResult> { + let mut guard = lock_stream(stream)?; + let Some(inner) = guard.as_mut() else { + // The stream was already consumed by `into_summary` or dropped by a prior error. + return Err(Error::generic( + "incremental scan stream was already consumed", + )); + }; + let result = next_arrow_batch(inner); + // Kill the stream on any error so a later next_arrow / into_summary can't return a partial + // result; the caller must rebuild to retry. + if result.is_err() { + guard.take(); + } + result +} + +#[cfg(feature = "default-engine-base")] +fn next_arrow_batch( + stream: &mut IncrementalScanStream, +) -> DeltaResult<*mut ScanMetadataArrowResult> { + let Some(filtered) = stream.next().transpose()? else { + return Ok(std::ptr::null_mut()); + }; + let (engine_data, selection_vector) = filtered.into_parts(); + let arrow_data = ArrowFFIData::try_from_engine_data(engine_data)?; + let result = Box::new(ScanMetadataArrowResult { + arrow_data, + selection_vector: selection_vector.into(), + transforms: Box::into_raw(Box::new(CTransforms::empty())), + }); + Ok(Box::into_raw(result)) +} + +/// Drain any unread batches, then consume the stream and return its summary of live Add and +/// Remove file keys for the range. +/// +/// Consumes the stream handle: the pointer is no longer valid after this call, whether or not it +/// succeeds. Returns an error if a prior [`incremental_scan_stream_next_arrow`] call errored, or +/// if draining the remaining batches fails. +/// +/// `live_adds` are the file keys still present at the target version; `removes` are those removed +/// within the range. A file re-tagged in place (e.g. by OPTIMIZE or clustering) keeps its `(path, +/// dv_unique_id)` key, so it appears in `live_adds` but not `removes`. +/// +/// # Safety +/// +/// Caller must pass a valid stream handle and must not use it again after this call. +#[no_mangle] +pub unsafe extern "C" fn incremental_scan_stream_into_summary( + stream: Handle, +) -> ExternResult> { + let inner_stream = unsafe { stream.into_inner() }; + let engine = inner_stream.engine.clone(); + incremental_scan_stream_into_summary_impl(&inner_stream).into_extern_result(&engine.as_ref()) +} + +fn incremental_scan_stream_into_summary_impl( + stream: &FfiIncrementalScanStream, +) -> DeltaResult> { + let inner = lock_stream(stream)? + .take() + .ok_or_else(|| Error::generic("incremental scan stream was already consumed"))?; + let summary = inner.into_summary()?; + Ok(Arc::new(summary).into()) +} + +fn lock_stream( + stream: &FfiIncrementalScanStream, +) -> DeltaResult>> { + stream + .stream + .lock() + .map_err(|_| Error::generic("poisoned incremental scan stream mutex")) +} + +/// The base (exclusive lower bound) version of the scanned range. +/// +/// # Safety +/// +/// Caller must pass a valid summary handle. +#[no_mangle] +pub unsafe extern "C" fn incremental_scan_summary_base_version( + summary: Handle, +) -> Version { + unsafe { summary.as_ref() }.base_version +} + +/// The target (inclusive upper bound) version of the scanned range. +/// +/// # Safety +/// +/// Caller must pass a valid summary handle. +#[no_mangle] +pub unsafe extern "C" fn incremental_scan_summary_target_version( + summary: Handle, +) -> Version { + unsafe { summary.as_ref() }.target_version +} + +/// Visit each live-Add file key in the summary, invoking `callback` once per key with its path +/// and (nullable) deletion-vector unique id. +/// +/// Match on the whole `(path, dv_unique_id)` key, not the path alone: the same path with different +/// DV ids refers to distinct logical files. `dv_unique_id` is passed as a zero-length slice when +/// the file has no deletion vector. +/// +/// The slices passed to the callback are only valid for the duration of the callback. +/// +/// # Safety +/// +/// Caller must pass a valid summary handle and a non-null `callback` function pointer. +#[no_mangle] +pub unsafe extern "C" fn incremental_scan_summary_visit_live_adds( + summary: Handle, + engine_context: NullableCvoid, + callback: FileKeyCallback, +) { + let summary = unsafe { summary.as_ref() }; + visit_file_keys(&summary.live_adds, engine_context, callback); +} + +/// Visit each Remove file key in the summary, invoking `callback` once per key with its path and +/// (nullable) deletion-vector unique id. +/// +/// See [`incremental_scan_summary_visit_live_adds`] for the callback contract. +/// +/// # Safety +/// +/// Caller must pass a valid summary handle and a non-null `callback` function pointer. +#[no_mangle] +pub unsafe extern "C" fn incremental_scan_summary_visit_removes( + summary: Handle, + engine_context: NullableCvoid, + callback: FileKeyCallback, +) { + let summary = unsafe { summary.as_ref() }; + visit_file_keys(&summary.removes, engine_context, callback); +} + +/// Callback invoked once per file key by the summary visitors. `dv_unique_id` is a zero-length +/// slice when the file carries no deletion vector. Both slices are only valid for the duration +/// of the call. +pub type FileKeyCallback = extern "C" fn( + engine_context: NullableCvoid, + path: KernelStringSlice, + dv_unique_id: KernelStringSlice, +); + +fn visit_file_keys( + keys: &HashSet, + engine_context: NullableCvoid, + callback: FileKeyCallback, +) { + for key in keys { + let path = key.path(); + let dv = key.dv_unique_id().unwrap_or(""); + callback( + engine_context, + kernel_string_slice!(path), + kernel_string_slice!(dv), + ); + } +} + +/// Free an incremental scan stream. Use this when abandoning a stream without draining it to a +/// summary. After [`incremental_scan_stream_into_summary`] consumes the stream, do not call this. +/// +/// # Safety +/// +/// Caller must pass a valid stream handle and must not use it again after this call. +#[no_mangle] +pub unsafe extern "C" fn free_incremental_scan_stream(stream: Handle) { + stream.drop_handle(); +} + +/// Free an incremental scan summary. +/// +/// # Safety +/// +/// Caller must pass a valid summary handle and must not use it again after this call. +#[no_mangle] +pub unsafe extern "C" fn free_incremental_scan_summary( + summary: Handle, +) { + summary.drop_handle(); +} + +#[cfg(test)] +mod tests { + use std::cell::RefCell; + use std::ffi::c_void; + use std::sync::Arc; + + use delta_kernel::object_store::memory::InMemory; + use delta_kernel_default_engine::DefaultEngineBuilder; + use test_utils::{actions_to_string, add_commit, TestAction}; + + use super::*; + use crate::error::KernelError; + use crate::expressions::kernel_visitor::{ + visit_expression_column, visit_expression_literal_int, visit_predicate_gt, + KernelExpressionVisitorState, + }; + use crate::ffi_test_utils::{ + allocate_err, assert_extern_result_error_with_message, ok_or_panic, recover_error, + }; + #[cfg(feature = "default-engine-base")] + use crate::scan::free_scan_metadata_arrow_result; + use crate::{ + engine_to_handle, free_engine, free_snapshot, get_snapshot_builder, kernel_string_slice, + snapshot_builder_build, snapshot_builder_set_version, NullableCvoid, TryFromStringSlice, + }; + + /// Build an in-memory engine handle and a snapshot pinned to the last version, from a v0 + /// metadata commit (`v0_metadata`) followed by `commits` (raw JSON) at v1.. Returns + /// `(engine_handle, snapshot_handle)`; the caller frees both. + async fn setup_from_raw( + v0_metadata: String, + commits: Vec, + ) -> (Handle, Handle) { + let table_root = "memory:///"; + let storage = Arc::new(InMemory::new()); + add_commit(table_root, storage.as_ref(), 0, v0_metadata) + .await + .unwrap(); + let target_version = commits.len() as u64; + for (idx, body) in commits.into_iter().enumerate() { + add_commit(table_root, storage.as_ref(), (idx + 1) as u64, body) + .await + .unwrap(); + } + let engine = DefaultEngineBuilder::new(storage.clone()).build(); + let engine = engine_to_handle(Arc::new(engine), allocate_err); + let mut builder = unsafe { + ok_or_panic(get_snapshot_builder( + kernel_string_slice!(table_root), + engine.shallow_copy(), + )) + }; + unsafe { snapshot_builder_set_version(&mut builder, target_version) }; + let snapshot = unsafe { ok_or_panic(snapshot_builder_build(builder)) }; + (engine, snapshot) + } + + /// Standard `id`/`val` metadata commit plus `TestAction` commit bodies at v1.. + async fn setup( + commits: Vec>, + ) -> (Handle, Handle) { + let v0 = actions_to_string(vec![TestAction::Metadata]); + setup_from_raw(v0, commits.into_iter().map(actions_to_string).collect()).await + } + + // Protocol + metadata commit enabling the deletionVectors feature, so DV-bearing Adds + // parse. `TestAction` can't emit a `deletionVector` field, so DV tests write raw JSON. + const DV_METADATA: &str = "{\"metaData\":{\"id\":\"test-id\",\"format\":{\"provider\":\"parquet\",\"options\":{}},\"schemaString\":\"{\\\"type\\\":\\\"struct\\\",\\\"fields\\\":[]}\",\"partitionColumns\":[],\"configuration\":{},\"createdTime\":1700000000000}}\n{\"protocol\":{\"minReaderVersion\":3,\"minWriterVersion\":7,\"readerFeatures\":[\"deletionVectors\"],\"writerFeatures\":[\"deletionVectors\"]}}"; + + fn add_with_dv(path: &str, storage_type: &str, path_or_inline: &str, offset: i32) -> String { + format!( + "{{\"add\":{{\"path\":\"{path}\",\"partitionValues\":{{}},\"size\":100,\"modificationTime\":1700000000000,\"dataChange\":true,\"stats\":null,\"deletionVector\":{{\"storageType\":\"{storage_type}\",\"pathOrInlineDv\":\"{path_or_inline}\",\"offset\":{offset},\"sizeInBytes\":10,\"cardinality\":1}}}}}}" + ) + } + + /// Build an engine + snapshot from raw commit-JSON strings at v1.., with a DV-enabling + /// metadata commit at v0. + async fn setup_raw( + commits: Vec, + ) -> (Handle, Handle) { + setup_from_raw(DV_METADATA.to_string(), commits).await + } + + // Collect the (path, dv_unique_id) pairs a summary visitor reports. `RefCell` because the + // C callback takes `&self` context; the visit is single-threaded. + extern "C" fn collect_key( + engine_context: NullableCvoid, + path: KernelStringSlice, + dv_unique_id: KernelStringSlice, + ) { + let keys = + engine_context.unwrap().as_ptr() as *const RefCell)>>; + let path = unsafe { String::try_from_slice(&path) }.unwrap(); + let dv = unsafe { String::try_from_slice(&dv_unique_id) }.unwrap(); + let dv = if dv.is_empty() { None } else { Some(dv) }; + unsafe { &*keys }.borrow_mut().push((path, dv)); + } + + fn visit_live_adds( + summary: &Handle, + ) -> Vec<(String, Option)> { + let keys = RefCell::new(Vec::new()); + let ctx = std::ptr::NonNull::new(&keys as *const _ as *mut std::ffi::c_void); + unsafe { + incremental_scan_summary_visit_live_adds(summary.shallow_copy(), ctx, collect_key) + }; + keys.into_inner() + } + + fn visit_removes( + summary: &Handle, + ) -> Vec<(String, Option)> { + let keys = RefCell::new(Vec::new()); + let ctx = std::ptr::NonNull::new(&keys as *const _ as *mut std::ffi::c_void); + unsafe { incremental_scan_summary_visit_removes(summary.shallow_copy(), ctx, collect_key) }; + keys.into_inner() + } + + /// Build a stream over `(base_version, target]`, asserting the range is covered. Tests that + /// exercise the builder-error or `OptionalValue::None` paths call the FFI directly instead. + fn build_stream( + snapshot: &Handle, + engine: &Handle, + base_version: Version, + ) -> Handle { + let builder = unsafe { + snapshot_incremental_scan_builder( + snapshot.shallow_copy(), + base_version, + engine.shallow_copy(), + ) + }; + let maybe_stream: Option> = + unsafe { ok_or_panic(incremental_scan_builder_build(builder)) }.into(); + maybe_stream.expect("range covered, expected Some(stream)") + } + + #[tokio::test] + async fn build_and_summary_reports_live_adds_and_removes( + ) -> Result<(), Box> { + // Range (0, 3]: A added at v1, removed at v3; B added at v2 stays live; C added at v3. + let (engine, snapshot) = setup(vec![ + vec![TestAction::Add("A".to_string())], + vec![TestAction::Add("B".to_string())], + vec![ + TestAction::Add("C".to_string()), + TestAction::Remove("A".to_string()), + ], + ]) + .await; + + let stream = build_stream(&snapshot, &engine, 0); + + let summary = unsafe { ok_or_panic(incremental_scan_stream_into_summary(stream)) }; + + assert_eq!( + unsafe { incremental_scan_summary_base_version(summary.shallow_copy()) }, + 0 + ); + assert_eq!( + unsafe { incremental_scan_summary_target_version(summary.shallow_copy()) }, + 3 + ); + + let mut live_adds: Vec<_> = visit_live_adds(&summary) + .into_iter() + .map(|(p, _)| p) + .collect(); + live_adds.sort(); + assert_eq!(live_adds, vec!["B".to_string(), "C".to_string()]); + + let removes: Vec<_> = visit_removes(&summary) + .into_iter() + .map(|(p, _)| p) + .collect(); + assert_eq!(removes, vec!["A".to_string()]); + + unsafe { free_incremental_scan_summary(summary) }; + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + #[cfg(feature = "default-engine-base")] + #[tokio::test] + async fn next_arrow_drains_then_null_then_summary() -> Result<(), Box> { + let (engine, snapshot) = setup(vec![ + vec![TestAction::Add("A".to_string())], + vec![TestAction::Add("B".to_string())], + ]) + .await; + + let stream = build_stream(&snapshot, &engine, 0); + + let mut batches = 0; + loop { + let ptr = + unsafe { ok_or_panic(incremental_scan_stream_next_arrow(stream.shallow_copy())) }; + if ptr.is_null() { + break; + } + batches += 1; + unsafe { free_scan_metadata_arrow_result(ptr) }; + } + assert_eq!(batches, 2, "one live-Add batch per commit"); + + // The stream is drained; into_summary still recovers the key sets. + let summary = unsafe { ok_or_panic(incremental_scan_stream_into_summary(stream)) }; + let mut live_adds: Vec<_> = visit_live_adds(&summary) + .into_iter() + .map(|(p, _)| p) + .collect(); + live_adds.sort(); + assert_eq!(live_adds, vec!["A".to_string(), "B".to_string()]); + + unsafe { free_incremental_scan_summary(summary) }; + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + #[tokio::test] + async fn build_errors_when_base_not_below_target() -> Result<(), Box> { + // Target snapshot is v2; base_version 2 makes the range empty. + let (engine, snapshot) = setup(vec![ + vec![TestAction::Add("A".to_string())], + vec![TestAction::Add("B".to_string())], + ]) + .await; + + let builder = unsafe { + snapshot_incremental_scan_builder(snapshot.shallow_copy(), 2, engine.shallow_copy()) + }; + let result = unsafe { incremental_scan_builder_build(builder) }; + assert_extern_result_error_with_message(result, KernelError::GenericError, None); + + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + #[tokio::test] + async fn free_builder_without_building() -> Result<(), Box> { + let (engine, snapshot) = setup(vec![vec![TestAction::Add("A".to_string())]]).await; + + let builder = unsafe { + snapshot_incremental_scan_builder(snapshot.shallow_copy(), 0, engine.shallow_copy()) + }; + unsafe { free_incremental_scan_builder(builder) }; + + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + // The whole point of the (path, dv_unique_id) key is that the same path with different DV + // ids is a distinct file. Assert the FFI visitor passes the full key through, not just the + // path: v1 adds (X, dv=uabc@1), v2 adds (X, dv=uxyz@2) -> two distinct live Adds. + #[tokio::test] + async fn visit_reports_full_dv_unique_id() -> Result<(), Box> { + let (engine, snapshot) = setup_raw(vec![ + add_with_dv("X.parquet", "u", "abc", 1), + add_with_dv("X.parquet", "u", "xyz", 2), + ]) + .await; + + let stream = build_stream(&snapshot, &engine, 0); + let summary = unsafe { ok_or_panic(incremental_scan_stream_into_summary(stream)) }; + + let mut live_adds = visit_live_adds(&summary); + live_adds.sort(); + assert_eq!( + live_adds, + vec![ + ("X.parquet".to_string(), Some("uabc@1".to_string())), + ("X.parquet".to_string(), Some("uxyz@2".to_string())), + ], + ); + + unsafe { free_incremental_scan_summary(summary) }; + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + // into_summary consumes the stream by taking the inner Option. When a second Arc reference + // to the same stream survives (via clone_handle, which bumps the refcount), a further + // terminal call hits the taken-Option guard and errors cleanly instead of double-consuming. + // (A single-refcount handle reused after into_summary is use-after-free per the # Safety + // contract, not this path; this test holds a real second reference.) + #[cfg(feature = "default-engine-base")] + #[tokio::test] + async fn terminal_calls_after_into_summary_error() -> Result<(), Box> { + let (engine, snapshot) = setup(vec![vec![TestAction::Add("A".to_string())]]).await; + + let stream = build_stream(&snapshot, &engine, 0); + + // A second owning reference keeps the object alive after into_summary consumes it. + let surviving = unsafe { stream.clone_handle() }; + let summary = unsafe { ok_or_panic(incremental_scan_stream_into_summary(stream)) }; + + let next = unsafe { incremental_scan_stream_next_arrow(surviving.shallow_copy()) }; + assert_extern_result_error_with_message(next, KernelError::GenericError, None); + let again = unsafe { incremental_scan_stream_into_summary(surviving) }; + assert_extern_result_error_with_message(again, KernelError::GenericError, None); + + unsafe { free_incremental_scan_summary(summary) }; + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + // A malformed `deletionVector` makes the kernel stream yield an error when the batch is + // processed. The first `next_arrow` surfaces it, and the error kills the stream: a second + // `next_arrow` and a following `into_summary` both fail rather than exposing a partial summary. + #[cfg(feature = "default-engine-base")] + #[tokio::test] + async fn next_arrow_error_kills_stream() -> Result<(), Box> { + // A DV with `storageType` set but `pathOrInlineDv` null fails the required-field read + // when the stream extracts the file key. + let bad_dv = "{\"add\":{\"path\":\"X.parquet\",\"partitionValues\":{},\"size\":100,\"modificationTime\":1700000000000,\"dataChange\":true,\"stats\":null,\"deletionVector\":{\"storageType\":\"u\",\"pathOrInlineDv\":null,\"offset\":1,\"sizeInBytes\":10,\"cardinality\":1}}}"; + let (engine, snapshot) = setup_raw(vec![bad_dv.to_string()]).await; + + let stream = build_stream(&snapshot, &engine, 0); + + // The first call surfaces the read error; later calls hit the taken-Option guard and report + // GenericError. Either way, no partial result leaks through. + let first = unsafe { incremental_scan_stream_next_arrow(stream.shallow_copy()) }; + let ExternResult::Err(e) = first else { + panic!("malformed DV must error"); + }; + drop(unsafe { recover_error(e) }); + + let again = unsafe { incremental_scan_stream_next_arrow(stream.shallow_copy()) }; + assert_extern_result_error_with_message(again, KernelError::GenericError, None); + let summary = unsafe { incremental_scan_stream_into_summary(stream) }; + assert_extern_result_error_with_message(summary, KernelError::GenericError, None); + + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + // Abandoning a never-drained stream must free the retained engine Arc and undrained kernel + // stream without leak or panic (the symmetric counterpart to free_builder_without_building). + #[tokio::test] + async fn free_stream_without_draining() -> Result<(), Box> { + let (engine, snapshot) = setup(vec![vec![TestAction::Add("A".to_string())]]).await; + + let stream = build_stream(&snapshot, &engine, 0); + unsafe { free_incremental_scan_stream(stream) }; + + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + // into_summary must drain the batches the caller didn't read. Read exactly one of two + // batches, then assert the summary still reports both commits' Adds. + #[cfg(feature = "default-engine-base")] + #[tokio::test] + async fn into_summary_drains_unread_batches() -> Result<(), Box> { + let (engine, snapshot) = setup(vec![ + vec![TestAction::Add("A".to_string())], + vec![TestAction::Add("B".to_string())], + ]) + .await; + + let stream = build_stream(&snapshot, &engine, 0); + + // Read exactly one batch, leaving the other for into_summary to drain. + let ptr = unsafe { ok_or_panic(incremental_scan_stream_next_arrow(stream.shallow_copy())) }; + assert!(!ptr.is_null()); + unsafe { free_scan_metadata_arrow_result(ptr) }; + + let summary = unsafe { ok_or_panic(incremental_scan_stream_into_summary(stream)) }; + let mut live_adds: Vec<_> = visit_live_adds(&summary) + .into_iter() + .map(|(p, _)| p) + .collect(); + live_adds.sort(); + assert_eq!(live_adds, vec!["A".to_string(), "B".to_string()]); + + unsafe { free_incremental_scan_summary(summary) }; + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + // Metadata declaring a single `id: integer` column, so `id` predicates are eligible for data + // skipping against `add.stats`. + const ID_METADATA: &str = "{\"metaData\":{\"id\":\"test-id\",\"format\":{\"provider\":\"parquet\",\"options\":{}},\"schemaString\":\"{\\\"type\\\":\\\"struct\\\",\\\"fields\\\":[{\\\"name\\\":\\\"id\\\",\\\"type\\\":\\\"integer\\\",\\\"nullable\\\":true,\\\"metadata\\\":{}}]}\",\"partitionColumns\":[],\"configuration\":{},\"createdTime\":1700000000000}}\n{\"protocol\":{\"minReaderVersion\":1,\"minWriterVersion\":2}}"; + + // Raw Add carrying `id` stats over the closed range `[id_min, id_max]`. + fn add_with_id_stats(path: &str, id_min: i32, id_max: i32) -> String { + let stats = format!( + "{{\\\"numRecords\\\":10,\\\"nullCount\\\":{{\\\"id\\\":0}},\\\"minValues\\\":{{\\\"id\\\":{id_min}}},\\\"maxValues\\\":{{\\\"id\\\":{id_max}}}}}" + ); + format!( + "{{\"add\":{{\"path\":\"{path}\",\"partitionValues\":{{}},\"size\":100,\"modificationTime\":1700000000000,\"dataChange\":true,\"stats\":\"{stats}\"}}}}" + ) + } + + /// Build an engine + snapshot from raw commit-JSON strings at v1.., with an `id`-column + /// metadata commit at v0. + async fn setup_with_id_metadata( + commits: Vec, + ) -> (Handle, Handle) { + setup_from_raw(ID_METADATA.to_string(), commits).await + } + + /// Predicate visitor that constructs `id > 25`. + extern "C" fn visit_id_gt_25( + _pred_ptr: *mut c_void, + state: &mut KernelExpressionVisitorState, + ) -> usize { + let id = "id"; + let col = unsafe { + ok_or_panic(visit_expression_column( + state, + kernel_string_slice!(id), + allocate_err, + )) + }; + let lit = visit_expression_literal_int(state, 25); + visit_predicate_gt(state, col, lit) + } + + // A predicate applied through the builder prunes streamed live Adds by their `add.stats`: + // `id > 25` drops the [0, 9] file and keeps the [20, 30] file. Removes are unaffected. + #[tokio::test] + async fn with_predicate_prunes_live_adds() -> Result<(), Box> { + let (engine, snapshot) = setup_with_id_metadata(vec![[ + add_with_id_stats("low.parquet", 0, 9), + add_with_id_stats("high.parquet", 20, 30), + ] + .join("\n")]) + .await; + + let builder = unsafe { + snapshot_incremental_scan_builder(snapshot.shallow_copy(), 0, engine.shallow_copy()) + }; + let mut predicate = EnginePredicate { + predicate: std::ptr::null_mut(), + visitor: visit_id_gt_25, + }; + let builder = unsafe { + ok_or_panic(incremental_scan_builder_with_predicate( + builder, + engine.shallow_copy(), + &mut predicate, + )) + }; + let maybe_stream: Option> = + unsafe { ok_or_panic(incremental_scan_builder_build(builder)) }.into(); + let stream = maybe_stream.expect("range covered, expected Some(stream)"); + let summary = unsafe { ok_or_panic(incremental_scan_stream_into_summary(stream)) }; + + let live_adds: Vec<_> = visit_live_adds(&summary) + .into_iter() + .map(|(p, _)| p) + .collect(); + assert_eq!( + live_adds, + vec!["high.parquet".to_string()], + "only the file whose stats overlap `id > 25` survives" + ); + + unsafe { free_incremental_scan_summary(summary) }; + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + // A predicate referencing a column absent from the table schema fails at build (fail-fast), + // surfacing the unresolved column rather than silently keeping every Add. + #[tokio::test] + async fn with_predicate_unknown_column_errors_at_build( + ) -> Result<(), Box> { + let (engine, snapshot) = + setup_with_id_metadata(vec![add_with_id_stats("a.parquet", 0, 9)]).await; + + let builder = unsafe { + snapshot_incremental_scan_builder(snapshot.shallow_copy(), 0, engine.shallow_copy()) + }; + // `id > 25` is fine to decode, but we point the visitor at a missing column below. + let mut predicate = EnginePredicate { + predicate: std::ptr::null_mut(), + visitor: visit_nonexistent_gt_25, + }; + let builder = unsafe { + ok_or_panic(incremental_scan_builder_with_predicate( + builder, + engine.shallow_copy(), + &mut predicate, + )) + }; + let result = unsafe { incremental_scan_builder_build(builder) }; + assert_extern_result_error_with_message(result, KernelError::MissingColumnError, None); + + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + /// Predicate visitor that constructs `nonexistent > 25` over a column not in the schema. + extern "C" fn visit_nonexistent_gt_25( + _pred_ptr: *mut c_void, + state: &mut KernelExpressionVisitorState, + ) -> usize { + let missing = "nonexistent"; + let col = unsafe { + ok_or_panic(visit_expression_column( + state, + kernel_string_slice!(missing), + allocate_err, + )) + }; + let lit = visit_expression_literal_int(state, 25); + visit_predicate_gt(state, col, lit) + } + + // The predicate's selection vector must line up with the drained Arrow batch, not just the + // summary: two Adds land in one batch, `id > 25` prunes the low file, and only the high file's + // row is selected. Guards the FFI-owned "only `true` rows are live Adds" contract for + // `next_arrow` (the summary path is covered separately by `with_predicate_prunes_live_adds`). + #[cfg(feature = "default-engine-base")] + #[tokio::test] + async fn next_arrow_selection_vector_reflects_predicate_pruning( + ) -> Result<(), Box> { + use delta_kernel::arrow::array::{Array, RecordBatch, StringArray, StructArray}; + use delta_kernel::arrow::ffi::from_ffi; + + let (engine, snapshot) = setup_with_id_metadata(vec![[ + add_with_id_stats("low.parquet", 0, 9), + add_with_id_stats("high.parquet", 20, 30), + ] + .join("\n")]) + .await; + + let builder = unsafe { + snapshot_incremental_scan_builder(snapshot.shallow_copy(), 0, engine.shallow_copy()) + }; + let mut predicate = EnginePredicate { + predicate: std::ptr::null_mut(), + visitor: visit_id_gt_25, + }; + let builder = unsafe { + ok_or_panic(incremental_scan_builder_with_predicate( + builder, + engine.shallow_copy(), + &mut predicate, + )) + }; + let maybe_stream: Option> = + unsafe { ok_or_panic(incremental_scan_builder_build(builder)) }.into(); + let stream = maybe_stream.expect("range covered, expected Some(stream)"); + + let ptr = unsafe { ok_or_panic(incremental_scan_stream_next_arrow(stream.shallow_copy())) }; + assert!(!ptr.is_null(), "expected one batch for the single commit"); + let ScanMetadataArrowResult { + arrow_data, + selection_vector, + transforms, + } = unsafe { *Box::from_raw(ptr) }; + // Incremental scan never rewrites rows, but the transforms box is always allocated. + assert!(!transforms.is_null()); + drop(unsafe { Box::from_raw(transforms) }); + let array_data = unsafe { from_ffi(arrow_data.array, &arrow_data.schema) }?; + let batch: RecordBatch = StructArray::from(array_data).into(); + let sv = unsafe { selection_vector.into_vec() }; + + // The batch carries every action row read; only the row whose `add.path` is the high file + // is selected. Inspect the column rather than assuming row order. + let add_paths = batch + .column_by_name("add") + .and_then(|c| c.as_any().downcast_ref::()) + .and_then(|s| s.column_by_name("path")) + .and_then(|c| c.as_any().downcast_ref::()) + .expect("add.path column"); + let selected: Vec<&str> = (0..batch.num_rows()) + .filter(|&i| sv[i] && !add_paths.is_null(i)) + .map(|i| add_paths.value(i)) + .collect(); + assert_eq!( + selected, + vec!["high.parquet"], + "only the high file is selected" + ); + + let ptr = unsafe { ok_or_panic(incremental_scan_stream_next_arrow(stream.shallow_copy())) }; + assert!(ptr.is_null(), "stream exhausted after one batch"); + + unsafe { free_incremental_scan_stream(stream) }; + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + // `build` returns `OptionalValue::None` (not an error) when the target snapshot's commit list + // can't cover `base_version + 1` -- the signal to fall back to a full scan. A checkpoint above + // the base truncates the retained commit JSONs so version 1 is no longer listed; a base of 0 + // then requires commit 1 and the range is uncovered. Needs a multi-thread runtime because + // `checkpoint_snapshot` issues nested `block_on` calls. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn build_returns_none_when_range_not_covered() -> Result<(), Box> { + use delta_kernel_default_engine::executor::tokio::TokioMultiThreadExecutor; + + use crate::{checkpoint_snapshot, version, FfiCheckpointWriteResult}; + + let table_root = "memory:///"; + let storage = Arc::new(InMemory::new()); + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await + .unwrap(); + for v in 1..=3u64 { + add_commit( + table_root, + storage.as_ref(), + v, + actions_to_string(vec![TestAction::Add(format!("file{v}.parquet"))]), + ) + .await + .unwrap(); + } + + let executor = Arc::new(TokioMultiThreadExecutor::new( + tokio::runtime::Handle::current(), + )); + let engine = engine_to_handle( + Arc::new( + DefaultEngineBuilder::new(storage.clone()) + .with_task_executor(executor) + .build(), + ), + allocate_err, + ); + + // Checkpoint at v2 so the log segment for a later snapshot starts at the checkpoint and + // drops commit 1's JSON. + let mut builder_v2 = unsafe { + ok_or_panic(get_snapshot_builder( + kernel_string_slice!(table_root), + engine.shallow_copy(), + )) + }; + unsafe { snapshot_builder_set_version(&mut builder_v2, 2) }; + let snapshot_v2 = unsafe { ok_or_panic(snapshot_builder_build(builder_v2)) }; + // checkpoint_snapshot borrows its handle (clone_as_arc) but does not consume it, so pass a + // shallow copy and free the original separately. + let checkpointed = match unsafe { + ok_or_panic(checkpoint_snapshot( + snapshot_v2.shallow_copy(), + engine.shallow_copy(), + None, + )) + } { + FfiCheckpointWriteResult::Written(s) | FfiCheckpointWriteResult::AlreadyExists(s) => s, + }; + assert_eq!(unsafe { version(checkpointed.shallow_copy()) }, 2); + unsafe { free_snapshot(checkpointed) }; + unsafe { free_snapshot(snapshot_v2) }; + + // Fresh snapshot at v3: its log segment starts at the v2 checkpoint, so commit 1 is gone. + let mut builder_v3 = unsafe { + ok_or_panic(get_snapshot_builder( + kernel_string_slice!(table_root), + engine.shallow_copy(), + )) + }; + unsafe { snapshot_builder_set_version(&mut builder_v3, 3) }; + let snapshot = unsafe { ok_or_panic(snapshot_builder_build(builder_v3)) }; + + // base_version 0 needs commit 1, which the checkpoint truncated => None, not an error. + let builder = unsafe { + snapshot_incremental_scan_builder(snapshot.shallow_copy(), 0, engine.shallow_copy()) + }; + let maybe_stream: Option> = + unsafe { ok_or_panic(incremental_scan_builder_build(builder)) }.into(); + assert!( + maybe_stream.is_none(), + "range not covered => None (full-scan fallback signal), not an error" + ); + + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + Ok(()) + } +} diff --git a/ffi/src/lib.rs b/ffi/src/lib.rs index de44a9748f..0ded362264 100644 --- a/ffi/src/lib.rs +++ b/ffi/src/lib.rs @@ -10,19 +10,28 @@ use std::default::Default; use std::os::raw::{c_char, c_void}; use std::ptr::NonNull; use std::sync::Arc; + +use delta_kernel::actions::{Metadata, Protocol}; +use delta_kernel::checkpoint::{CheckpointSpec, V2CheckpointConfig}; +use delta_kernel::history_manager::{ + first_version_after as kernel_first_version_after, + get_earliest_commit as kernel_get_earliest_commit, + latest_version_as_of as kernel_latest_version_as_of, CommitAt, HistoryCommitType, +}; +#[cfg(feature = "default-engine-base")] +use delta_kernel::object_store::ObjectStore; +use delta_kernel::schema::Schema; +use delta_kernel::snapshot::{CheckpointWriteResult, Snapshot, SnapshotRef}; +use delta_kernel::{DeltaResult, Engine, EngineData, FileStats, LogPath, Version}; +use delta_kernel_ffi_macros::handle_descriptor; use tracing::debug; use url::Url; #[cfg(feature = "default-engine-base")] use { - delta_kernel::engine::default::executor::tokio::TokioMultiThreadExecutor, - std::collections::HashMap, + delta_kernel_default_engine::executor::tokio::TokioMultiThreadExecutor, + std::collections::HashMap, std::num::NonZero, }; -use delta_kernel::schema::Schema; -use delta_kernel::snapshot::Snapshot; -use delta_kernel::{DeltaResult, Engine, EngineData, LogPath, Version}; -use delta_kernel_ffi_macros::handle_descriptor; - // cbindgen doesn't understand our use of feature flags here, and by default it parses `mod handle` // twice. So we tell it to ignore one of the declarations to avoid double-definition errors. /// cbindgen:ignore @@ -38,30 +47,40 @@ use handle::Handle; // relies on `crate::` extern crate self as delta_kernel_ffi; +mod alloc_stats; + +pub mod commit_range; mod domain_metadata; pub use domain_metadata::get_domain_metadata; pub mod engine_data; pub mod engine_funcs; pub mod error; #[cfg(feature = "default-engine-base")] +pub mod rest_engine; +#[cfg(feature = "default-engine-base")] pub mod table_changes; use error::{AllocateError, AllocateErrorFn, ExternResult, IntoExternResult}; +#[cfg(feature = "delta-kernel-unity-catalog")] +pub mod delta_kernel_unity_catalog; pub mod expressions; #[cfg(feature = "tracing")] +pub mod ffi_metrics; +#[cfg(feature = "tracing")] pub mod ffi_tracing; -#[cfg(feature = "catalog-managed")] +pub mod incremental_scan; pub mod log_path; +#[cfg(feature = "declarative-plans")] +pub mod plans; pub mod scan; pub mod schema; pub mod schema_visitor; -#[cfg(feature = "uc-catalog")] -pub mod uc_catalog; #[cfg(test)] mod ffi_test_utils; #[cfg(feature = "test-ffi")] pub mod test_ffi; pub mod transaction; +use transaction::MutableCommitter; pub(crate) type NullableCvoid = Option>; @@ -142,6 +161,38 @@ impl KernelStringSlice { len: source.len(), } } + + #[cfg(feature = "tracing")] + pub(crate) fn empty() -> Self { + KernelStringSlice { + ptr: NonNull::::dangling().as_ptr().cast(), + len: 0, + } + } +} + +/// A non-owned slice of raw bytes, intended for arg-passing between kernel and engine. +/// +/// Like [`KernelStringSlice`], the pointed-to data must outlive the slice itself, and the slice +/// must not be retained beyond the foreign function call it was passed into. +#[repr(C)] +pub struct KernelBytesSlice { + ptr: *const u8, + len: usize, +} + +impl KernelBytesSlice { + /// Creates a new bytes slice from a source byte slice. + /// + /// # Safety + /// Caller must guarantee that the source will outlive the created KernelBytesSlice. + #[cfg(feature = "declarative-plans")] + pub(crate) unsafe fn new_unsafe(source: &[u8]) -> Self { + Self { + ptr: source.as_ptr(), + len: source.len(), + } + } } /// FFI-safe implementation for Rust's `Option` @@ -161,6 +212,15 @@ impl From> for OptionalValue { } } +impl From> for Option { + fn from(value: OptionalValue) -> Self { + match value { + OptionalValue::Some(value) => Some(value), + OptionalValue::None => None, + } + } +} + /// Creates a new [`KernelStringSlice`] from a string reference (which must be an identifier, to /// ensure it is not immediately dropped). This is the safest way to create a kernel string slice. /// @@ -183,6 +243,23 @@ macro_rules! kernel_string_slice { } pub(crate) use kernel_string_slice; +/// Similar to [`kernel_string_slice!`](kernel_string_slice), this macro provides a safer way to +/// construct a KernelBytesSlice. +/// +/// Refer to [`kernel_string_slice!`](kernel_string_slice) for safety and implementation +/// notes. +#[cfg(feature = "declarative-plans")] +macro_rules! kernel_bytes_slice { + ( $source:ident ) => {{ + fn do_it(b: &[u8]) -> $crate::KernelBytesSlice { + unsafe { $crate::KernelBytesSlice::new_unsafe(b) } + } + do_it(&$source) + }}; +} +#[cfg(feature = "declarative-plans")] +pub(crate) use kernel_bytes_slice; + trait TryFromStringSlice<'a>: Sized { unsafe fn try_from_slice(slice: &'a KernelStringSlice) -> DeltaResult; } @@ -244,6 +321,33 @@ fn allocate_kernel_string_impl( Ok(Box::new(s).into()) } +/// A kernel-allocated type representing an owned byte buffer. This can be obtained by +/// calling [`allocate_kernel_bytes`] with a [`KernelBytesSlice`]. Kernel takes ownership of the +/// handle when it consumes the bytes and the engine must not use the handle afterwards. +#[cfg(feature = "declarative-plans")] +#[handle_descriptor(target=Vec, mutable=true, sized=true)] +pub struct ExclusiveRustBytes; + +/// Allow engines to create an opaque pointer to [`ExclusiveRustBytes`] by copying the provided +/// `bytes` into kernel-owned memory. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid [`KernelBytesSlice`] whose pointer references at least +/// `len` readable bytes. The slice only needs to remain valid until after this call returns. +#[cfg(feature = "declarative-plans")] +#[no_mangle] +pub unsafe extern "C" fn allocate_kernel_bytes( + bytes: KernelBytesSlice, +) -> Handle { + let copied = if bytes.len == 0 { + Vec::new() + } else { + unsafe { std::slice::from_raw_parts(bytes.ptr, bytes.len) }.to_vec() + }; + Box::new(copied).into() +} + // Put KernelBoolSlice in a sub-module, with non-public members, so rust code cannot instantiate it // directly. It can only be created by converting `From>`. mod private { @@ -266,6 +370,58 @@ mod private { len: usize, } + /// An owned byte buffer allocated by the kernel. Any time the engine receives a + /// `KernelOwnedBytes` as a return value from a kernel method, the engine owns the buffer and + /// must free it by calling [super::free_kernel_bytes] exactly once. + #[cfg(feature = "declarative-plans")] + #[repr(C)] + pub struct KernelOwnedBytes { + ptr: NonNull, + len: usize, + } + + #[cfg(feature = "declarative-plans")] + impl KernelOwnedBytes { + /// Converts this buffer back into a `Vec`. + /// + /// # Safety + /// + /// The buffer must have been originally created `From>`, and must not have already + /// been consumed by a previous call to this method. + pub unsafe fn into_vec(self) -> Vec { + if self.len == 0 { + Default::default() + } else { + unsafe { Vec::from_raw_parts(self.ptr.as_ptr(), self.len, self.len) } + } + } + } + + #[cfg(feature = "declarative-plans")] + impl From> for KernelOwnedBytes { + fn from(val: Vec) -> Self { + let len = val.len(); + let boxed = val.into_boxed_slice(); + let leaked_ptr = Box::leak(boxed).as_mut_ptr(); + // safety: Box::leak always returns a valid, non-null pointer + let ptr = unsafe { NonNull::new_unchecked(leaked_ptr) }; + KernelOwnedBytes { ptr, len } + } + } + + /// # Safety + /// + /// The engine assumes ownership of the buffer memory when kernel passes it a + /// [KernelOwnedBytes], but must only free it by calling [super::free_kernel_bytes]. Since the + /// global allocator is threadsafe, it doesn't matter which engine thread invokes that method. + #[cfg(feature = "declarative-plans")] + unsafe impl Send for KernelOwnedBytes {} + /// # Safety + /// + /// If engine chooses to leverage concurrency, engine is responsible to prevent data races. + #[cfg(feature = "declarative-plans")] + unsafe impl Sync for KernelOwnedBytes {} + impl KernelBoolSlice { /// Creates an empty slice. pub fn empty() -> KernelBoolSlice { @@ -373,8 +529,9 @@ mod private { } } } -pub use private::KernelBoolSlice; -pub use private::KernelRowIndexArray; +#[cfg(feature = "declarative-plans")] +pub use private::KernelOwnedBytes; +pub use private::{KernelBoolSlice, KernelRowIndexArray}; /// # Safety /// @@ -393,8 +550,20 @@ pub unsafe extern "C" fn free_row_indexes(slice: KernelRowIndexArray) { let _ = slice.into_vec(); } -// TODO: Do we want this handle at all? Perhaps we should just _always_ pass raw *mut c_void pointers -// that are the engine data? Even if we want the type, should it be a shared handle instead? +/// Free a [`KernelOwnedBytes`] buffer obtained from kernel. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid buffer, and must not use it again afterwards. +#[cfg(feature = "declarative-plans")] +#[no_mangle] +pub unsafe extern "C" fn free_kernel_bytes(bytes: KernelOwnedBytes) { + let _ = unsafe { bytes.into_vec() }; +} + +// TODO: Do we want this handle at all? Perhaps we should just _always_ pass raw *mut c_void +// pointers that are the engine data? Even if we want the type, should it be a shared handle +// instead? /// an opaque struct that encapsulates data read by an engine. this handle can be passed back into /// some kernel calls to operate on the data, or can be converted into the raw data as read by the /// [`delta_kernel::Engine`] by calling [`get_raw_engine_data`] @@ -473,19 +642,50 @@ unsafe fn unwrap_and_parse_path_as_url(path: KernelStringSlice) -> DeltaResult), +} + +/// A builder that allows setting options on the `Engine` before actually building it. +/// +/// For a normal object store backend, `url` is the table storage location (`s3://…`, `file://…`). +/// For REST, call [`set_builder_rest_object_store`] with a [`rest_engine::CRestEndpointConfig`] +/// and set `url` to the REST service base URL; see [`rest_engine`] for TLS and auth options. #[cfg(feature = "default-engine-base")] pub struct EngineBuilder { url: Url, allocate_fn: AllocateErrorFn, options: HashMap, + object_store_backend: ObjectStoreBackend, /// Configuration for multithreaded executor. If Some, use a multi-threaded executor /// If None, use the default single-threaded background executor. multithreaded_executor_config: Option, + /// Read-path I/O concurrency config for the JSON and Parquet handlers. `None` fields fall back + /// to the engine's defaults. + io_config: IoConcurrencyConfig, } #[cfg(feature = "default-engine-base")] -struct MultithreadedExecutorConfig { +#[derive(Default, Clone)] +pub(crate) struct IoConcurrencyConfig { + /// Maximum number of files read concurrently (file-level readahead depth). `None` uses the + /// engine default. + buffer_size: Option>, + /// Maximum number of rows per yielded batch. `None` uses the engine default. + batch_size: Option>, +} + +#[cfg(feature = "default-engine-base")] +#[derive(Clone)] +pub(crate) struct MultithreadedExecutorConfig { /// Number of worker threads for the tokio runtime. `None` uses Tokio's default. worker_threads: Option, /// Maximum number of threads for blocking operations. `None` uses Tokio's default. @@ -524,7 +724,9 @@ fn get_engine_builder_impl( url: url?, allocate_fn, options: HashMap::default(), + object_store_backend: ObjectStoreBackend::default(), multithreaded_executor_config: None, + io_config: IoConcurrencyConfig::default(), }); Ok(Box::into_raw(builder)) } @@ -582,6 +784,75 @@ pub unsafe extern "C" fn set_builder_with_multithreaded_executor( }); } +/// Configure read-path I/O concurrency for the engine's JSON and Parquet handlers. +/// +/// These control the `read_*_files` paths used during log replay and scans. Returned data ordering +/// is preserved regardless of these values. +/// +/// # Parameters +/// - `builder`: The engine builder to configure. +/// - `buffer_size`: Maximum number of files read concurrently (file-level readahead depth). Higher +/// values overlap more object-store requests to hide latency, at the cost of more in-flight +/// memory. Pass 0 to use the engine default. +/// - `batch_size`: Maximum number of rows per yielded batch. Pass 0 to use the engine default. +/// Overall read memory usage is roughly proportional to `buffer_size * batch_size`. +/// +/// # Safety +/// +/// Caller must pass a valid EngineBuilder pointer. +#[cfg(feature = "default-engine-base")] +#[no_mangle] +pub unsafe extern "C" fn set_builder_with_io_concurrency( + builder: &mut EngineBuilder, + buffer_size: usize, + batch_size: usize, +) { + // `NonZero::new` maps 0 -> `None`, which the engine reads as "use the default". + builder.io_config = IoConcurrencyConfig { + buffer_size: NonZero::new(buffer_size), + batch_size: NonZero::new(batch_size), + }; +} + +/// Select a REST-backed object store. See [`rest_engine`] for setup, option keys, and callbacks. +/// +/// # Safety +/// +/// Caller must pass a valid builder pointer and a non-null `endpoint_config`. When `callback` is +/// non-null, `context` must remain valid for the engine lifetime and the callback must be safe to +/// invoke from any thread concurrently (see [`rest_engine::CAuthHeaderCallback`]). +#[cfg(feature = "default-engine-base")] +#[no_mangle] +pub unsafe extern "C" fn set_builder_rest_object_store( + builder: &mut EngineBuilder, + endpoint_config: *const rest_engine::CRestEndpointConfig, + callback: Option, + context: NullableCvoid, +) -> ExternResult { + set_builder_rest_object_store_impl(builder, endpoint_config, callback, context) + .into_extern_result(&builder.allocate_fn) +} + +#[cfg(feature = "default-engine-base")] +fn set_builder_rest_object_store_impl( + builder: &mut EngineBuilder, + endpoint_config: *const rest_engine::CRestEndpointConfig, + callback: Option, + context: NullableCvoid, +) -> DeltaResult { + // SAFETY: caller guarantees a non-null, valid `endpoint_config` for the duration of the call. + let endpoint_config = unsafe { endpoint_config.as_ref() } + .ok_or_else(|| delta_kernel::Error::generic("null CRestEndpointConfig pointer"))?; + builder.object_store_backend = + ObjectStoreBackend::Rest(Box::new(rest_engine::rest_builder_state_from_ffi( + endpoint_config, + callback, + context, + builder.allocate_fn, + )?)); + Ok(true) +} + /// Consume the builder and return a `default` engine. After calling, the passed pointer is _no /// longer valid_. Note that this _consumes_ and frees the builder, so there is no need to /// drop/free it afterwards. @@ -599,7 +870,9 @@ pub unsafe extern "C" fn builder_build( get_default_engine_impl( builder_box.url, builder_box.options, + builder_box.object_store_backend, builder_box.multithreaded_executor_config, + builder_box.io_config, builder_box.allocate_fn, ) .into_extern_result(&builder_box.allocate_fn) @@ -624,7 +897,14 @@ fn get_default_default_engine_impl( url: DeltaResult, allocate_error: AllocateErrorFn, ) -> DeltaResult> { - get_default_engine_impl(url?, Default::default(), None, allocate_error) + get_default_engine_impl( + url?, + Default::default(), + ObjectStoreBackend::default(), + None, + IoConcurrencyConfig::default(), + allocate_error, + ) } /// Safety @@ -650,26 +930,59 @@ fn engine_to_handle( fn get_default_engine_impl( url: Url, options: HashMap, + object_store_backend: ObjectStoreBackend, executor_config: Option, + io_config: IoConcurrencyConfig, allocate_error: AllocateErrorFn, ) -> DeltaResult> { - use delta_kernel::engine::default::storage::store_from_url_opts; - use delta_kernel::engine::default::DefaultEngineBuilder; + use delta_kernel_default_engine::storage::store_from_url_opts; - let store = store_from_url_opts(&url, options)?; + let store = match object_store_backend { + ObjectStoreBackend::UrlScheme => store_from_url_opts(&url, options)?, + ObjectStoreBackend::Rest(rest) => { + rest_engine::build_rest_object_store(&url, &options, rest.as_ref())? + } + }; + build_engine_from_store(store, executor_config, io_config, allocate_error) +} + +/// Assemble a default engine from a pre-built [`ObjectStore`], applying executor and read-path I/O +/// tuning. Shared by the URL-scheme and REST engine builder paths. +#[cfg(feature = "default-engine-base")] +pub(crate) fn build_engine_from_store( + store: Arc, + executor_config: Option, + io_config: IoConcurrencyConfig, + allocate_error: AllocateErrorFn, +) -> DeltaResult> { + use delta_kernel_default_engine::DefaultEngineBuilder; + + // The builder is generic over the executor type, so apply the shared I/O config via a generic + // helper to both branches without naming the concrete builder type. + fn apply_io_config( + builder: DefaultEngineBuilder, + io_config: &IoConcurrencyConfig, + ) -> DefaultEngineBuilder { + let builder = match io_config.buffer_size { + Some(buffer_size) => builder.with_buffer_size(buffer_size), + None => builder, + }; + match io_config.batch_size { + Some(batch_size) => builder.with_batch_size(batch_size), + None => builder, + } + } let engine: Arc = if let Some(config) = executor_config { let executor = TokioMultiThreadExecutor::new_owned_runtime( config.worker_threads, config.max_blocking_threads, )?; - Arc::new( - DefaultEngineBuilder::new(store) - .with_task_executor(Arc::new(executor)) - .build(), - ) + let builder = DefaultEngineBuilder::new(store).with_task_executor(Arc::new(executor)); + Arc::new(apply_io_config(builder, &io_config).build()) } else { - Arc::new(DefaultEngineBuilder::new(store).build()) + let builder = apply_io_config(DefaultEngineBuilder::new(store), &io_config); + Arc::new(builder.build()) }; Ok(engine_to_handle(engine, allocate_error)) @@ -690,110 +1003,270 @@ pub struct SharedSchema; #[handle_descriptor(target=Snapshot, mutable=false, sized=true)] pub struct SharedSnapshot; -/// Get the latest snapshot from the specified table +/// Outcome of a checkpoint write performed via [`checkpoint_snapshot`]. +/// +/// `Written` indicates the kernel wrote a new checkpoint at this version and returns an +/// updated snapshot whose log segment reflects the new checkpoint. `AlreadyExists` indicates +/// a checkpoint at this version was already present (either pre-existed or was written by a +/// concurrent writer) and returns the original snapshot unchanged. +/// +/// Both variants carry an owned `Handle` that the caller must release via +/// [`free_snapshot`]. +/// +/// cbindgen:prefix-with-name=true +#[repr(C)] +pub enum FfiCheckpointWriteResult { + Written(Handle), + AlreadyExists(Handle), +} + +impl FfiCheckpointWriteResult { + fn from_kernel(result: CheckpointWriteResult, snapshot: SnapshotRef) -> Self { + match result { + CheckpointWriteResult::Written => Self::Written(snapshot.into()), + CheckpointWriteResult::AlreadyExists => Self::AlreadyExists(snapshot.into()), + } + } +} + +/// Checkpoint write configuration for [`checkpoint_snapshot`]. Mirrors the kernel's +/// [`CheckpointSpec`] and [`V2CheckpointConfig`] across the C ABI; see those types for the +/// authoritative semantics and constraints. +/// +/// Pass `Option<&FfiCheckpointSpec>::None` to [`checkpoint_snapshot`] to let the kernel +/// auto-pick V1 or V2 based on the table's protocol features. +/// +/// cbindgen:prefix-with-name=true +#[repr(C)] +pub enum FfiCheckpointSpec { + /// See [`CheckpointSpec::V1`]. + V1, + /// See [`V2CheckpointConfig::NoSidecar`]. + V2NoSidecar, + /// See [`V2CheckpointConfig::WithSidecar`]. + V2WithSidecar { + /// See [`V2CheckpointConfig::WithSidecar`]. `None` selects the kernel default hint. + file_actions_per_sidecar_hint: OptionalValue, + }, +} + +impl FfiCheckpointSpec { + fn to_kernel(&self) -> CheckpointSpec { + match self { + FfiCheckpointSpec::V1 => CheckpointSpec::V1, + FfiCheckpointSpec::V2NoSidecar => CheckpointSpec::V2(V2CheckpointConfig::NoSidecar), + FfiCheckpointSpec::V2WithSidecar { + file_actions_per_sidecar_hint, + } => { + // Forward the hint verbatim (including `Some(0)`) so the kernel applies the same + // validation as a native `V2CheckpointConfig::WithSidecar`; `None` => kernel + // default. + let file_actions_per_sidecar_hint = match file_actions_per_sidecar_hint { + OptionalValue::Some(hint) => Some(*hint), + OptionalValue::None => None, + }; + CheckpointSpec::V2(V2CheckpointConfig::WithSidecar { + file_actions_per_sidecar_hint, + }) + } + } + } +} + +#[handle_descriptor(target=Protocol, mutable=false, sized=true)] +pub struct SharedProtocol; + +#[handle_descriptor(target=Metadata, mutable=false, sized=true)] +pub struct SharedMetadata; + +/// Opaque builder for constructing a [`SharedSnapshot`]. +/// +/// Create with [`get_snapshot_builder`] (from a table path) or [`get_snapshot_builder_from`] +/// (incrementally from an existing snapshot). Configure with [`snapshot_builder_set_version`], +/// [`snapshot_builder_set_log_tail`], and [`snapshot_builder_set_max_catalog_version`] (for +/// catalog-managed tables). Finally, call [`snapshot_builder_build`] to consume the builder and +/// obtain the snapshot. If you need to discard the builder without building, call +/// [`free_snapshot_builder`]. +pub struct FfiSnapshotBuilder { + engine: Arc, + source: FfiSnapshotBuilderSource, + version: Option, + log_tail: Vec, + max_catalog_version: Option, +} + +/// An opaque handle with exclusive (Box-like) ownership of a [`FfiSnapshotBuilder`]. +#[handle_descriptor(target=FfiSnapshotBuilder, mutable=true, sized=true)] +pub struct MutableFfiSnapshotBuilder; + +enum FfiSnapshotBuilderSource { + TableRoot(Url), + ExistingSnapshot(SnapshotRef), +} + +fn make_snapshot_builder( + source: FfiSnapshotBuilderSource, + engine: Arc, +) -> DeltaResult> { + Ok(Box::new(FfiSnapshotBuilder { + engine, + source, + version: None, + log_tail: Vec::new(), + max_catalog_version: None, + }) + .into()) +} + +/// Get a builder for creating a [`SharedSnapshot`] from a table path. +/// +/// Use [`snapshot_builder_set_version`] to pin a specific version, then call +/// [`snapshot_builder_build`] to obtain the snapshot. The caller owns the returned handle and must +/// eventually call either [`snapshot_builder_build`] to produce a [`SharedSnapshot`], or +/// [`free_snapshot_builder`] to drop it without building. /// /// # Safety /// -/// Caller is responsible for passing valid handles and path pointer. +/// Caller is responsible for passing a valid path and engine handle. #[no_mangle] -pub unsafe extern "C" fn snapshot( +pub unsafe extern "C" fn get_snapshot_builder( path: KernelStringSlice, engine: Handle, -) -> ExternResult> { +) -> ExternResult> { + let engine_ref = unsafe { engine.as_ref() }; + let engine_arc = unsafe { engine.clone_as_arc() }; let url = unsafe { unwrap_and_parse_path_as_url(path) }; - let engine = unsafe { engine.as_ref() }; - snapshot_impl(url, engine, None, Vec::new()).into_extern_result(&engine) + let source = match url { + Ok(url) => FfiSnapshotBuilderSource::TableRoot(url), + Err(e) => return DeltaResult::Err(e).into_extern_result(&engine_ref), + }; + make_snapshot_builder(source, engine_arc).into_extern_result(&engine_ref) } -/// Get the latest snapshot from the specified table with optional log tail +/// Get a builder for incrementally updating an existing snapshot. +/// +/// This avoids re-reading the full log. Use [`snapshot_builder_set_version`] to target a specific +/// version, then call [`snapshot_builder_build`] to obtain the updated snapshot. The caller owns +/// the returned handle and must eventually call either [`snapshot_builder_build`] to produce a +/// [`SharedSnapshot`], or [`free_snapshot_builder`] to drop it without building. /// /// # Safety /// -/// Caller is responsible for passing valid handles and path pointer. -/// The log_paths array and its contents must remain valid for the duration of this call. -#[cfg(feature = "catalog-managed")] +/// Caller is responsible for passing valid handles. #[no_mangle] -pub unsafe extern "C" fn snapshot_with_log_tail( - path: KernelStringSlice, +pub unsafe extern "C" fn get_snapshot_builder_from( + prev_snapshot: Handle, engine: Handle, - log_paths: log_path::LogPathArray, -) -> ExternResult> { - let url = unsafe { unwrap_and_parse_path_as_url(path) }; +) -> ExternResult> { let engine_ref = unsafe { engine.as_ref() }; - - // Convert LogPathArray to Vec - let log_tail = match unsafe { log_paths.log_paths() } { - Ok(paths) => paths, - Err(err) => return Err(err).into_extern_result(&engine_ref), - }; - - snapshot_impl(url, engine_ref, None, log_tail).into_extern_result(&engine_ref) + let engine_arc = unsafe { engine.clone_as_arc() }; + let snapshot_arc = unsafe { prev_snapshot.clone_as_arc() }; + make_snapshot_builder( + FfiSnapshotBuilderSource::ExistingSnapshot(snapshot_arc), + engine_arc, + ) + .into_extern_result(&engine_ref) } -/// Get the snapshot from the specified table at a specific version. Note this is only safe for -/// non-catalog-managed tables. +/// Set the target version on a snapshot builder. When omitted, the snapshot is created at the +/// latest version of the table. /// /// # Safety /// -/// Caller is responsible for passing valid handles and path pointer. +/// Caller must pass a valid builder pointer. #[no_mangle] -pub unsafe extern "C" fn snapshot_at_version( - path: KernelStringSlice, - engine: Handle, +pub unsafe extern "C" fn snapshot_builder_set_version( + builder: &mut Handle, version: Version, -) -> ExternResult> { - let url = unsafe { unwrap_and_parse_path_as_url(path) }; - let engine = unsafe { engine.as_ref() }; - snapshot_impl(url, engine, version.into(), Vec::new()).into_extern_result(&engine) +) { + unsafe { builder.as_mut() }.version = Some(version); } -/// Get the snapshot from the specified table at a specific version with log tail. +/// Set the log tail on a snapshot builder for catalog-managed tables. /// /// # Safety /// -/// Caller is responsible for passing valid handles and path pointer. -/// The log_tail array and its contents must remain valid for the duration of this call. -#[cfg(feature = "catalog-managed")] +/// Caller must pass a valid builder pointer. The log_tail array and its contents must remain valid +/// for the duration of this call. #[no_mangle] -pub unsafe extern "C" fn snapshot_at_version_with_log_tail( - path: KernelStringSlice, - engine: Handle, - version: Version, +pub unsafe extern "C" fn snapshot_builder_set_log_tail( + builder: &mut Handle, log_tail: log_path::LogPathArray, -) -> ExternResult> { - let url = unsafe { unwrap_and_parse_path_as_url(path) }; - let engine_ref = unsafe { engine.as_ref() }; +) -> ExternResult { + let builder_mut = unsafe { builder.as_mut() }; + let engine_arc = builder_mut.engine.clone(); + let engine_ref = engine_arc.as_ref(); + snapshot_builder_set_log_tail_impl(builder_mut, log_tail).into_extern_result(&engine_ref) +} - // Convert LogPathArray to Vec - let log_tail = match unsafe { log_tail.log_paths() } { - Ok(paths) => paths, - Err(err) => return Err(err).into_extern_result(&engine_ref), - }; +unsafe fn snapshot_builder_set_log_tail_impl( + builder: &mut FfiSnapshotBuilder, + log_tail: log_path::LogPathArray, +) -> DeltaResult { + builder.log_tail = unsafe { log_tail.log_paths() }?; + Ok(true) +} - snapshot_impl(url, engine_ref, version.into(), log_tail).into_extern_result(&engine_ref) +/// Set the max catalog version on a snapshot builder for catalog-managed tables. This bounds the +/// snapshot version to what the catalog has ratified. +/// +/// # Safety +/// +/// Caller must pass a valid builder pointer. +#[no_mangle] +pub unsafe extern "C" fn snapshot_builder_set_max_catalog_version( + builder: &mut Handle, + max_catalog_version: Version, +) { + unsafe { builder.as_mut() }.max_catalog_version = Some(max_catalog_version); } -fn snapshot_impl( - url: DeltaResult, - extern_engine: &dyn ExternEngine, - version: Option, - #[allow(unused_variables)] log_tail: Vec, -) -> DeltaResult> { - let mut builder = Snapshot::builder_for(url?); +/// Consume the builder and return a snapshot. After calling, the builder pointer is _no longer +/// valid_. The builder is always freed by this call, whether or not it succeeds. +/// +/// # Safety +/// +/// Caller must pass a valid builder pointer and must not use it again after this call. +#[no_mangle] +pub unsafe extern "C" fn snapshot_builder_build( + mut builder: Handle, +) -> ExternResult> { + // Clone the engine Arc before consuming the handle so we can still use it for error reporting + let engine_arc = unsafe { builder.as_mut() }.engine.clone(); + let engine_ref = engine_arc.as_ref(); + let builder_box = unsafe { builder.into_inner() }; + snapshot_builder_build_impl(*builder_box).into_extern_result(&engine_ref) +} - if let Some(v) = version { - builder = builder.at_version(v); +fn snapshot_builder_build_impl(builder: FfiSnapshotBuilder) -> DeltaResult> { + let engine = builder.engine.engine(); + let mut rust_builder = match builder.source { + FfiSnapshotBuilderSource::TableRoot(url) => Snapshot::builder_for(url), + FfiSnapshotBuilderSource::ExistingSnapshot(snap) => Snapshot::builder_from(snap), + }; + if let Some(v) = builder.version { + rust_builder = rust_builder.at_version(v); } - - #[cfg(feature = "catalog-managed")] - if !log_tail.is_empty() { - builder = builder.with_log_tail(log_tail); + if !builder.log_tail.is_empty() { + rust_builder = rust_builder.with_log_tail(builder.log_tail); } - - let snapshot = builder.build(extern_engine.engine().as_ref())?; + if let Some(mcv) = builder.max_catalog_version { + rust_builder = rust_builder.with_max_catalog_version(mcv); + } + let snapshot = rust_builder.build(engine.as_ref())?; Ok(snapshot.into()) } +/// Free a snapshot builder without building a snapshot (e.g. on an error path). +/// +/// # Safety +/// +/// Caller must pass a valid builder pointer and must not use it again after this call. +#[no_mangle] +pub unsafe extern "C" fn free_snapshot_builder(builder: Handle) { + builder.drop_handle(); +} + /// # Safety /// /// Caller is responsible for passing a valid handle. @@ -803,29 +1276,65 @@ pub unsafe extern "C" fn free_snapshot(snapshot: Handle) { snapshot.drop_handle(); } -/// Perform a full checkpoint of the specified snapshot using the supplied engine. +/// Perform a checkpoint write on the given snapshot. Mirrors the kernel's +/// [`delta_kernel::snapshot::Snapshot::checkpoint`] API across the C ABI. +/// +/// Pass `Option<&FfiCheckpointSpec>::None` to let the kernel auto-pick V1 or V2 based on the +/// table's protocol features and emit an inline checkpoint with no sidecars. Pass `Some(&spec)` +/// to force a specific shape; see [`FfiCheckpointSpec`] for the available options. /// -/// This writes the checkpoint parquet file and the `_last_checkpoint` file. +/// Returns [`FfiCheckpointWriteResult::Written`] with the post-checkpoint snapshot (whose log +/// segment records the new checkpoint), or [`FfiCheckpointWriteResult::AlreadyExists`] with the +/// original snapshot when a checkpoint at this version was already present. In both branches the +/// caller owns the returned snapshot handle and must release it via [`free_snapshot`]. /// /// # Safety /// -/// Caller is responsible for passing valid handles. +/// Caller is responsible for passing valid handles. The `spec` pointer, if non-null, must point +/// to a valid `FfiCheckpointSpec` for the duration of the call. #[no_mangle] pub unsafe extern "C" fn checkpoint_snapshot( snapshot: Handle, engine: Handle, -) -> ExternResult { + spec: Option<&FfiCheckpointSpec>, +) -> ExternResult { let engine_ref = unsafe { engine.as_ref() }; - let snapshot = unsafe { snapshot.clone_as_arc() }; - snapshot_checkpoint_impl(snapshot, engine_ref).into_extern_result(&engine_ref) + let snapshot_ref: SnapshotRef = unsafe { snapshot.clone_as_arc() }; + let kernel_spec = spec.map(|s| s.to_kernel()); + snapshot_ref + .checkpoint(engine_ref.engine().as_ref(), kernel_spec.as_ref()) + .map(|(result, updated)| FfiCheckpointWriteResult::from_kernel(result, updated)) + .into_extern_result(&engine_ref) } -fn snapshot_checkpoint_impl( - snapshot: Arc, - extern_engine: &dyn ExternEngine, -) -> DeltaResult { - snapshot.checkpoint(extern_engine.engine().as_ref())?; - Ok(true) +/// Publish unpublished catalog commits on a catalog-managed snapshot. +/// +/// No-op (still returns a caller-owned snapshot handle at the same version) when there is +/// nothing to publish. Errors when unpublished commits exist but the table is not +/// catalog-managed or the committer is not a catalog committer. +/// +/// Caller owns the returned handle ([`free_snapshot`]). The input snapshot is borrowed; the +/// committer is consumed (do not free). The returned snapshot carries the published watermark +/// used by subsequent catalog commits -- use it for the next `transaction_with_committer` / +/// checkpoint. +/// +/// # Safety +/// +/// Caller must pass valid snapshot, committer, and engine handles. The committer handle is +/// consumed and must not be used or freed afterward. +#[no_mangle] +pub unsafe extern "C" fn snapshot_publish_with_committer( + snapshot: Handle, + committer: Handle, + engine: Handle, +) -> ExternResult> { + let engine_ref = unsafe { engine.as_ref() }; + let snapshot_ref: SnapshotRef = unsafe { snapshot.clone_as_arc() }; + let committer = unsafe { committer.into_inner() }; + snapshot_ref + .publish(engine_ref.engine().as_ref(), committer.as_ref()) + .map(|updated| updated.into()) + .into_extern_result(&engine_ref) } /// Get the version of the specified snapshot @@ -839,31 +1348,244 @@ pub unsafe extern "C" fn version(snapshot: Handle) -> u64 { snapshot.version() } -/// Get the logical schema of the specified snapshot +/// Get the timestamp of the specified snapshot in milliseconds since the Unix epoch. +/// +/// When In-Commit Timestamp (ICT) is enabled, returns the ICT value from the commit's +/// `CommitInfo` action. Otherwise, falls back to the filesystem last-modified time of +/// the latest commit file. +/// +/// Returns an error if the commit file is missing, the ICT configuration is invalid, or the +/// ICT value cannot be read. /// /// # Safety /// -/// Caller is responsible for passing a valid snapshot handle. +/// Caller is responsible for passing valid snapshot handle and engine handle. #[no_mangle] -pub unsafe extern "C" fn logical_schema(snapshot: Handle) -> Handle { +pub unsafe extern "C" fn snapshot_timestamp( + snapshot: Handle, + engine: Handle, +) -> ExternResult { + let engine_ref = unsafe { engine.as_ref() }; let snapshot = unsafe { snapshot.as_ref() }; - snapshot.schema().into() + snapshot + .get_timestamp(engine_ref.engine().as_ref()) + .into_extern_result(&engine_ref) } -/// Free a schema +/// File-level statistics for a snapshot, sourced from the snapshot's CRC. /// -/// # Safety -/// Engine is responsible for providing a valid schema handle. -#[no_mangle] -pub unsafe extern "C" fn free_schema(schema: Handle) { - schema.drop_handle(); +/// Pass-by-value mirror of the scalar fields of kernel's [`FileStats`]. +// TODO: expose kernel's `FileStats` file-size histogram once it has a C-ABI-friendly +// representation. It is a variable-length structure, so it needs a different accessor shape. +#[derive(Debug, PartialEq, Eq)] +#[repr(C)] +pub struct FfiFileStats { + /// Number of active `Add` file actions in this table version. + pub num_files: i64, + /// Total size of the table in bytes (sum of all active `Add` file sizes). + pub table_size_bytes: i64, } -/// Get the resolved root of the table. This should be used in any future calls that require -/// constructing a path -/// -/// # Safety -/// +impl From for FfiFileStats { + fn from(stats: FileStats) -> Self { + Self { + num_files: stats.num_files(), + table_size_bytes: stats.table_size_bytes(), + } + } +} + +/// Get the file-level statistics ([`FfiFileStats`]) for this snapshot from its CRC. +/// +/// Returns [`OptionalValue::None`] when no CRC is resolved at this version (none on disk, a read +/// failure, or the nearest on-disk CRC is stale), or its CRC does not carry complete file stats. +/// Performs no I/O. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid snapshot handle. +#[no_mangle] +pub unsafe extern "C" fn snapshot_file_stats( + snapshot: Handle, +) -> OptionalValue { + let snapshot = unsafe { snapshot.as_ref() }; + snapshot + .get_file_stats_if_present() + .map(FfiFileStats::from) + .into() +} + +/// Selects which commit type to return for the history_manager query. FFI-safe mirror of +/// [`HistoryCommitType`]. +#[repr(C)] +pub enum FfiHistoryCommitType { + /// Maps to [`HistoryCommitType::Published`]. + Published = 0, + /// Maps to [`HistoryCommitType::Recreatable`]. + Recreatable = 1, +} + +impl From for HistoryCommitType { + fn from(commit_type: FfiHistoryCommitType) -> Self { + match commit_type { + FfiHistoryCommitType::Published => Self::Published, + FfiHistoryCommitType::Recreatable => Self::Recreatable, + } + } +} + +/// Get the earliest commit version available in the table's `_delta_log/` directory. +/// +/// # Parameters +/// - `engine`: engine handle used to list the log directory. +/// - `log_root`: URL of the table's `_delta_log/` directory (must end with `/`). +/// - `earliest_ratified_commit_version`: for catalog-managed tables, the earliest version the +/// catalog has ratified a commit at; pass `OptionalValue::None` for filesystem-only tables. +/// - `commit_type`: selects the query. [`FfiHistoryCommitType::Published`] returns the earliest +/// commit; [`FfiHistoryCommitType::Recreatable`] returns the earliest fully reconstructable +/// version. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid `log_root` string slice and a valid engine handle. +#[no_mangle] +pub unsafe extern "C" fn get_earliest_commit( + engine: Handle, + log_root: KernelStringSlice, + earliest_ratified_commit_version: OptionalValue, + commit_type: FfiHistoryCommitType, +) -> ExternResult { + let engine_ref = unsafe { engine.as_ref() }; + let log_root = unsafe { unwrap_and_parse_path_as_url(log_root) }; + get_earliest_commit_impl( + engine_ref, + log_root, + earliest_ratified_commit_version, + commit_type, + ) + .into_extern_result(&engine_ref) +} + +fn get_earliest_commit_impl( + extern_engine: &dyn ExternEngine, + log_root: DeltaResult, + earliest_ratified_commit_version: OptionalValue, + commit_type: FfiHistoryCommitType, +) -> DeltaResult { + kernel_get_earliest_commit( + extern_engine.engine().as_ref(), + &log_root?, + earliest_ratified_commit_version.into(), + commit_type.into(), + ) +} + +/// A commit located by a timestamp query: a commit version paired with its timestamp. FFI-safe +/// mirror of [`CommitAt`]. +#[repr(C)] +pub struct FfiCommitAt { + /// The commit version. + pub version: Version, + /// Timestamp (milliseconds since the Unix epoch) associated with this commit. This is the + /// commit's in-commit timestamp (ICT) when ICT is enabled, otherwise the commit's file + /// modification time. + pub timestamp: i64, +} + +impl From for FfiCommitAt { + fn from(commit: CommitAt) -> Self { + Self { + version: commit.version, + timestamp: commit.timestamp, + } + } +} + +/// Get the latest commit (version and timestamp) within `snapshot`'s version range with a +/// timestamp at or before `timestamp` (milliseconds since the Unix epoch). `commit_type` selects +/// whether to return any version present in the log ([`FfiHistoryCommitType::Published`]) or only a +/// version whose table can be fully reconstructed ([`FfiHistoryCommitType::Recreatable`]). +/// +/// Returns an error if no version exists at or before `timestamp`, +/// or when there is an error resolving the commit based on the [`FfiHistoryCommitType`]. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid snapshot handle and a valid engine handle. +#[no_mangle] +pub unsafe extern "C" fn latest_version_as_of( + snapshot: Handle, + engine: Handle, + timestamp: i64, + commit_type: FfiHistoryCommitType, +) -> ExternResult { + let engine_ref = unsafe { engine.as_ref() }; + let snapshot = unsafe { snapshot.as_ref() }; + kernel_latest_version_as_of( + snapshot, + engine_ref.engine().as_ref(), + timestamp, + commit_type.into(), + ) + .map(FfiCommitAt::from) + .into_extern_result(&engine_ref) +} + +/// Get the first commit (version and timestamp) within `snapshot`'s version range with a +/// timestamp at or after `timestamp` (milliseconds since the Unix epoch). `commit_type` selects +/// whether to return any version present in the log ([`FfiHistoryCommitType::Published`]) or only a +/// version whose table can be fully reconstructed ([`FfiHistoryCommitType::Recreatable`]). +/// +/// Returns an error if no version exists at or after `timestamp`, +/// or when there is an error resolving the commit based on the [`FfiHistoryCommitType`]. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid snapshot handle and a valid engine handle. +#[no_mangle] +pub unsafe extern "C" fn first_version_after( + snapshot: Handle, + engine: Handle, + timestamp: i64, + commit_type: FfiHistoryCommitType, +) -> ExternResult { + let engine_ref = unsafe { engine.as_ref() }; + let snapshot = unsafe { snapshot.as_ref() }; + kernel_first_version_after( + snapshot, + engine_ref.engine().as_ref(), + timestamp, + commit_type.into(), + ) + .map(FfiCommitAt::from) + .into_extern_result(&engine_ref) +} + +/// Get the logical schema of the specified snapshot +/// +/// # Safety +/// +/// Caller is responsible for passing a valid snapshot handle. +#[no_mangle] +pub unsafe extern "C" fn logical_schema(snapshot: Handle) -> Handle { + let snapshot = unsafe { snapshot.as_ref() }; + snapshot.schema().into() +} + +/// Free a schema +/// +/// # Safety +/// Engine is responsible for providing a valid schema handle. +#[no_mangle] +pub unsafe extern "C" fn free_schema(schema: Handle) { + schema.drop_handle(); +} + +/// Get the resolved root of the table. This should be used in any future calls that require +/// constructing a path +/// +/// # Safety +/// /// Caller is responsible for passing a valid snapshot handle. #[no_mangle] pub unsafe extern "C" fn snapshot_table_root( @@ -882,7 +1604,10 @@ pub unsafe extern "C" fn snapshot_table_root( #[no_mangle] pub unsafe extern "C" fn get_partition_column_count(snapshot: Handle) -> usize { let snapshot = unsafe { snapshot.as_ref() }; - snapshot.table_configuration().partition_columns().len() + snapshot + .table_configuration() + .logical_partition_columns() + .len() } /// Get an iterator of the list of partition columns for this snapshot. @@ -895,11 +1620,188 @@ pub unsafe extern "C" fn get_partition_columns( ) -> Handle { let snapshot = unsafe { snapshot.as_ref() }; // NOTE: Clippy doesn't like it, but we need to_vec+into_iter to decouple lifetimes - let partition_columns = snapshot.table_configuration().partition_columns().to_vec(); + let partition_columns = snapshot + .table_configuration() + .logical_partition_columns() + .to_vec(); let iter: Box = Box::new(partition_columns.into_iter()); iter.into() } +/// Visit each metadata configuration (key/value pair) for the specified snapshot by invoking the +/// provided `visitor` callback once per entry. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid snapshot handle, a valid `engine_context` as an +/// opaque pointer passed to each `visitor` invocation, and a valid `visitor` function pointer. +#[no_mangle] +pub unsafe extern "C" fn visit_metadata_configuration( + snapshot: Handle, + engine_context: NullableCvoid, + visitor: extern "C" fn( + engine_context: NullableCvoid, + key: KernelStringSlice, + value: KernelStringSlice, + ), +) { + let snapshot = unsafe { snapshot.as_ref() }; + snapshot + .table_configuration() + .metadata() + .configuration() + .iter() + .for_each(|(key, value)| { + visitor( + engine_context, + kernel_string_slice!(key), + kernel_string_slice!(value), + ); + }); +} +// === Protocol handle FFI === + +/// Get the protocol for this snapshot. The returned handle must be freed with [`free_protocol`]. +/// +/// # Safety +/// Caller is responsible for providing a valid snapshot handle. +#[no_mangle] +pub unsafe extern "C" fn snapshot_get_protocol( + snapshot: Handle, +) -> Handle { + let snapshot = unsafe { snapshot.as_ref() }; + Arc::new(snapshot.table_configuration().protocol().clone()).into() +} + +/// Free a protocol handle obtained from [`snapshot_get_protocol`]. +/// +/// # Safety +/// Caller is responsible for providing a valid, non-freed protocol handle. +#[no_mangle] +pub unsafe extern "C" fn free_protocol(protocol: Handle) { + protocol.drop_handle(); +} + +/// Visit all fields of the protocol in a single FFI call. The caller provides: +/// - `visit_versions`: called once with `(context, min_reader_version, min_writer_version)` +/// - `visit_feature`: called once per feature with `(context, is_reader, feature_name)`. +/// `is_reader` is `true` for reader features, `false` for writer features. If the protocol uses +/// legacy versioning (no explicit feature lists), the `visit_feature` callback will not fire. +/// +/// # Safety +/// Caller is responsible for providing a valid protocol handle, a valid `context` pointer, and +/// valid function pointers for `visit_versions` and `visit_feature`. +#[no_mangle] +pub unsafe extern "C" fn visit_protocol( + protocol: Handle, + context: NullableCvoid, + visit_versions: extern "C" fn(context: NullableCvoid, min_reader: i32, min_writer: i32), + visit_feature: extern "C" fn( + context: NullableCvoid, + is_reader: bool, + feature: KernelStringSlice, + ), +) { + let protocol = unsafe { protocol.as_ref() }; + visit_versions( + context, + protocol.min_reader_version(), + protocol.min_writer_version(), + ); + if let Some(features) = protocol.reader_features() { + for f in features { + let name = f.as_ref(); + visit_feature(context, true, kernel_string_slice!(name)); + } + } + if let Some(features) = protocol.writer_features() { + for f in features { + let name = f.as_ref(); + visit_feature(context, false, kernel_string_slice!(name)); + } + } +} + +// === Metadata handle FFI === + +/// Get the metadata for this snapshot. The returned handle must be freed with [`free_metadata`]. +/// +/// # Safety +/// Caller is responsible for providing a valid snapshot handle. +#[no_mangle] +pub unsafe extern "C" fn snapshot_get_metadata( + snapshot: Handle, +) -> Handle { + let snapshot = unsafe { snapshot.as_ref() }; + Arc::new(snapshot.table_configuration().metadata().clone()).into() +} + +/// Free a metadata handle obtained from [`snapshot_get_metadata`]. +/// +/// # Safety +/// Caller is responsible for providing a valid, non-freed metadata handle. +#[no_mangle] +pub unsafe extern "C" fn free_metadata(metadata: Handle) { + metadata.drop_handle(); +} + +/// Visit all fields of the metadata in a single FFI call. String fields are passed as +/// [`KernelStringSlice`] references that borrow from the metadata handle -- they are only valid +/// for the duration of the callback. +/// +/// The visitor receives: +/// - `id`: always present +/// - `name`: `OptionalValue::None` if not set +/// - `description`: `OptionalValue::None` if not set +/// - `format_provider`: always present +/// - `has_created_time`: whether `created_time_ms` is meaningful +/// - `created_time_ms`: milliseconds since epoch (only valid when `has_created_time` is true) +/// +/// # Safety +/// Caller is responsible for providing a valid metadata handle, a valid `context` pointer, and +/// a valid `visit_metadata_fields` function pointer. String slices must not be retained past +/// the callback return. +#[no_mangle] +pub unsafe extern "C" fn visit_metadata( + metadata: Handle, + context: NullableCvoid, + visit_metadata_fields: extern "C" fn( + context: NullableCvoid, + id: KernelStringSlice, + name: OptionalValue, + description: OptionalValue, + format_provider: KernelStringSlice, + has_created_time: bool, + created_time_ms: i64, + ), +) { + let metadata = unsafe { metadata.as_ref() }; + let id_str = metadata.id(); + let id = kernel_string_slice!(id_str); + let name = metadata.name().map(|s| kernel_string_slice!(s)).into(); + let description = metadata + .description() + .map(|s| kernel_string_slice!(s)) + .into(); + let fp_str = metadata.format_provider(); + let format_provider = kernel_string_slice!(fp_str); + let (has_created_time, created_time_ms) = match metadata.created_time() { + Some(t) => (true, t), + None => (false, 0), + }; + visit_metadata_fields( + context, + id, + name, + description, + format_provider, + has_created_time, + created_time_ms, + ); +} + +// === Snapshot-level computed property FFI === + type StringIter = dyn Iterator + Send; #[handle_descriptor(target=StringIter, mutable=true, sized=false)] @@ -996,20 +1898,29 @@ impl Default for ReferenceSet { #[cfg(test)] mod tests { - use super::*; - use crate::error::{EngineError, KernelError}; - use crate::ffi_test_utils::{ - allocate_err, allocate_str, assert_extern_result_error_with_message, ok_or_panic, - recover_string, - }; - use delta_kernel::engine::default::executor::tokio::TokioMultiThreadExecutor; - use delta_kernel::engine::default::DefaultEngineBuilder; + use std::collections::HashMap; + use delta_kernel::object_store::memory::InMemory; use delta_kernel::object_store::path::Path; - use delta_kernel::object_store::ObjectStore; + use delta_kernel::object_store::{DynObjectStore, ObjectStore as _, ObjectStoreExt as _}; + use delta_kernel::schema::StructType; + use delta_kernel_default_engine::executor::tokio::TokioMultiThreadExecutor; + use delta_kernel_default_engine::DefaultEngineBuilder; + use rstest::rstest; use serde_json::Value; use test_utils::{ - actions_to_string, actions_to_string_partitioned, add_commit, TestAction, METADATA, + actions_to_string, actions_to_string_catalog_managed, actions_to_string_partitioned, + actions_to_string_with_metadata, add_commit, add_staged_commit, create_table, TestAction, + METADATA, METADATA_WITH_FEATURES, METADATA_WITH_TABLE_PROPERTIES, + TEST_ICT_ENABLEMENT_TIMESTAMP, + }; + use url::Url; + + use super::*; + use crate::error::{EngineError, KernelError}; + use crate::ffi_test_utils::{ + allocate_err, allocate_str, assert_extern_result_error_with_message, build_snapshot, + ok_or_panic, recover_string, setup_snapshot, }; #[no_mangle] @@ -1032,6 +1943,88 @@ mod tests { } } + /// Create an in-memory table with a single version-0 metadata commit, returning the storage, + /// engine handle, and a snapshot at version 0. The caller is responsible for freeing the + /// engine and snapshot handles. + async fn make_engine_and_v0_snapshot( + path: &str, + ) -> Result< + ( + Arc, + Handle, + Handle, + ), + Box, + > { + let storage = Arc::new(InMemory::new()); + add_commit( + path, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + let engine = engine_to_handle( + Arc::new(DefaultEngineBuilder::new(storage.clone()).build()), + allocate_err, + ); + let snap = unsafe { build_snapshot(kernel_string_slice!(path), engine.shallow_copy()) }; + Ok((storage, engine, snap)) + } + + /// Like [`make_engine_and_v0_snapshot`] but creates a catalog-managed table. The returned + /// snapshot is built with `max_catalog_version = 0`. + async fn make_catalog_managed_engine_and_v0_snapshot( + path: &str, + ) -> Result< + ( + Arc, + Handle, + Handle, + ), + Box, + > { + let storage = Arc::new(InMemory::new()); + add_commit( + path, + storage.as_ref(), + 0, + actions_to_string_catalog_managed(vec![TestAction::Metadata]), + ) + .await?; + let engine = engine_to_handle( + Arc::new(DefaultEngineBuilder::new(storage.clone()).build()), + allocate_err, + ); + let snap = unsafe { + let mut ptr = ok_or_panic(get_snapshot_builder( + kernel_string_slice!(path), + engine.shallow_copy(), + )); + snapshot_builder_set_max_catalog_version(&mut ptr, 0); + ok_or_panic(snapshot_builder_build(ptr)) + }; + Ok((storage, engine, snap)) + } + + /// Build a snapshot with version, log tail, and optional max catalog version via the FFI + /// builder API. Returns the raw `ExternResult` so callers can test error cases. + unsafe fn snapshot_at_version_with_log_tail( + path: KernelStringSlice, + engine: Handle, + version: Version, + log_tail: log_path::LogPathArray, + max_catalog_version: Option, + ) -> ExternResult> { + let mut ptr = ok_or_panic(get_snapshot_builder(path, engine)); + snapshot_builder_set_version(&mut ptr, version); + ok_or_panic(snapshot_builder_set_log_tail(&mut ptr, log_tail)); + if let Some(mcv) = max_catalog_version { + snapshot_builder_set_max_catalog_version(&mut ptr, mcv); + } + snapshot_builder_build(ptr) + } + pub(crate) fn get_default_engine(path: &str) -> Handle { let path = kernel_string_slice!(path); let builder = unsafe { ok_or_panic(get_engine_builder(path, allocate_err)) }; @@ -1048,44 +2041,35 @@ mod tests { #[tokio::test] async fn test_snapshot() -> Result<(), Box> { - let storage = Arc::new(InMemory::new()); let table_root = "memory:///test_table/"; - add_commit( - table_root, - storage.as_ref(), - 0, - actions_to_string(vec![TestAction::Metadata]), - ) - .await?; - let engine = DefaultEngineBuilder::new(storage.clone()).build(); - let engine = engine_to_handle(Arc::new(engine), allocate_err); + let (_, engine, snapshot1) = make_engine_and_v0_snapshot(table_root).await?; // Test getting latest snapshot - let snapshot1 = unsafe { - ok_or_panic(snapshot( - kernel_string_slice!(table_root), - engine.shallow_copy(), - )) - }; let version1 = unsafe { version(snapshot1.shallow_copy()) }; assert_eq!(version1, 0); // Test getting snapshot at version let snapshot2 = unsafe { - ok_or_panic(snapshot_at_version( + let mut ptr = ok_or_panic(get_snapshot_builder( kernel_string_slice!(table_root), engine.shallow_copy(), - 0, - )) + )); + snapshot_builder_set_version(&mut ptr, 0); + ok_or_panic(snapshot_builder_build(ptr)) }; let version2 = unsafe { version(snapshot2.shallow_copy()) }; assert_eq!(version2, 0); // Test getting non-existent snapshot let snapshot_at_non_existent_version = unsafe { - snapshot_at_version(kernel_string_slice!(table_root), engine.shallow_copy(), 1) + let mut ptr = ok_or_panic(get_snapshot_builder( + kernel_string_slice!(table_root), + engine.shallow_copy(), + )); + snapshot_builder_set_version(&mut ptr, 1); + snapshot_builder_build(ptr) }; - assert_extern_result_error_with_message(snapshot_at_non_existent_version, KernelError::GenericError, "Generic delta kernel error: LogSegment end version 0 not the same as the specified end version 1"); + assert_extern_result_error_with_message(snapshot_at_non_existent_version, KernelError::GenericError, Some("Generic delta kernel error: LogSegment end version 0 not the same as the specified end version 1")); let snapshot_table_root_str = unsafe { snapshot_table_root(snapshot1.shallow_copy(), allocate_str) }; @@ -1099,89 +2083,746 @@ mod tests { Ok(()) } - // NOTE: Snapshot::checkpoint requires a multi-threaded tokio task executor to avoid deadlocks. - #[tokio::test(flavor = "multi_thread", worker_threads = 2)] - async fn test_snapshot_checkpoint() -> Result<(), Box> { - let storage = Arc::new(InMemory::new()); - let table_root = "memory:///"; + #[test] + fn test_snapshot_file_stats_present() -> Result<(), Box> { + // The crc-full fixture has a CRC at version 0 with complete file stats. + let table_path = std::fs::canonicalize("../kernel/tests/data/crc-full/")?; + let table_root = Url::from_directory_path(&table_path) + .map_err(|()| delta_kernel::Error::generic("invalid table path"))? + .to_string(); + + let engine = get_default_engine(&table_root); + let snapshot = + unsafe { build_snapshot(kernel_string_slice!(table_root), engine.shallow_copy()) }; + + assert_eq!( + unsafe { snapshot_file_stats(snapshot.shallow_copy()) }, + OptionalValue::Some(FfiFileStats { + num_files: 10, + table_size_bytes: 5259 + }), + ); - // Create a minimal table history: initial metadata+protocol (no commitInfo), then some - // add/remove commits. - let protocol_and_metadata = METADATA - .lines() - .skip(1) // skip commitInfo - .collect::>() - .join("\n"); - add_commit(table_root, storage.as_ref(), 0, protocol_and_metadata).await?; - add_commit( - table_root, - storage.as_ref(), - 1, - actions_to_string(vec![ - TestAction::Add("file1.parquet".into()), - TestAction::Add("file2.parquet".into()), - ]), - ) - .await?; - add_commit( - table_root, - storage.as_ref(), - 2, - actions_to_string(vec![ - TestAction::Add("file3.parquet".into()), - TestAction::Remove("file1.parquet".into()), - ]), - ) - .await?; + unsafe { free_snapshot(snapshot) } + unsafe { free_engine(engine) } + Ok(()) + } - let executor = Arc::new(TokioMultiThreadExecutor::new( - tokio::runtime::Handle::current(), - )); - let engine = DefaultEngineBuilder::new(storage.clone()) - .with_task_executor(executor) - .build(); - let engine = engine_to_handle(Arc::new(engine), allocate_err); + #[tokio::test] + async fn test_snapshot_file_stats_absent_without_crc() -> Result<(), Box> + { + // A metadata-only table has no CRC, so file stats are absent. + let table_root = "memory:///test_file_stats_no_crc/"; + let (_storage, engine, snapshot) = make_engine_and_v0_snapshot(table_root).await?; + + assert_eq!( + unsafe { snapshot_file_stats(snapshot.shallow_copy()) }, + OptionalValue::None + ); - let snapshot = unsafe { - ok_or_panic(snapshot( - kernel_string_slice!(table_root), - engine.shallow_copy(), - )) - }; + unsafe { free_snapshot(snapshot) } + unsafe { free_engine(engine) } + Ok(()) + } - let did_checkpoint = unsafe { - ok_or_panic(checkpoint_snapshot( - snapshot.shallow_copy(), + enum EarliestCommitTableSetupScenario { + FilesystemV0Commit, + CatalogManagedV0Commit, + FilesystemV4CheckpointWithEarliestCommitAtV2, + NoCommits, + } + + async fn setup_earliest_commit_table( + setup: &EarliestCommitTableSetupScenario, + table_root: &str, + ) -> Handle { + match setup { + EarliestCommitTableSetupScenario::FilesystemV0Commit => { + let (_, engine, snapshot) = make_engine_and_v0_snapshot(table_root).await.unwrap(); + unsafe { free_snapshot(snapshot) }; + engine + } + EarliestCommitTableSetupScenario::CatalogManagedV0Commit => { + let (_, engine, snapshot) = make_catalog_managed_engine_and_v0_snapshot(table_root) + .await + .unwrap(); + unsafe { free_snapshot(snapshot) }; + engine + } + EarliestCommitTableSetupScenario::FilesystemV4CheckpointWithEarliestCommitAtV2 => { + let storage = Arc::new(InMemory::new()); + for version in 2..=4 { + add_commit( + table_root, + storage.as_ref(), + version, + actions_to_string(vec![TestAction::Metadata]), + ) + .await + .unwrap(); + } + let checkpoint_version = 4; + let checkpoint = + format!("_delta_log/{:020}.checkpoint.parquet", checkpoint_version); + let table_url = Url::parse(table_root).unwrap(); + let checkpoint_path = + Path::from_url_path(table_url.join(&checkpoint).unwrap().path()).unwrap(); + storage + .put(&checkpoint_path, "x".to_string().into()) + .await + .unwrap(); + engine_to_handle( + Arc::new(DefaultEngineBuilder::new(storage).build()), + allocate_err, + ) + } + EarliestCommitTableSetupScenario::NoCommits => get_default_engine(table_root), + } + } + + #[rstest] + #[case::fs_published( + EarliestCommitTableSetupScenario::FilesystemV0Commit, + OptionalValue::None, + FfiHistoryCommitType::Published, + Ok(0) + )] + #[case::fs_recreatable( + EarliestCommitTableSetupScenario::FilesystemV0Commit, + OptionalValue::None, + FfiHistoryCommitType::Recreatable, + Ok(0) + )] + #[case::cm_ratified_zero( + EarliestCommitTableSetupScenario::CatalogManagedV0Commit, + OptionalValue::Some(0), + FfiHistoryCommitType::Published, + Ok(0) + )] + #[case::cm_with_empty_delta_log( + EarliestCommitTableSetupScenario::NoCommits, + OptionalValue::Some(0), + FfiHistoryCommitType::Published, + Err(KernelError::GenericError) + )] + #[case::empty_log_errors( + EarliestCommitTableSetupScenario::NoCommits, + OptionalValue::None, + FfiHistoryCommitType::Published, + Err(KernelError::LogHistoryError) + )] + #[case::checkpoint_published( + EarliestCommitTableSetupScenario::FilesystemV4CheckpointWithEarliestCommitAtV2, + OptionalValue::None, + FfiHistoryCommitType::Published, + Ok(2) + )] + #[case::checkpoint_recreatable( + EarliestCommitTableSetupScenario::FilesystemV4CheckpointWithEarliestCommitAtV2, + OptionalValue::None, + FfiHistoryCommitType::Recreatable, + Ok(4) + )] + #[tokio::test] + async fn test_get_earliest_commit_cases( + #[case] setup: EarliestCommitTableSetupScenario, + #[case] earliest_ratified: OptionalValue, + #[case] commit_type: FfiHistoryCommitType, + #[case] expected: Result, + ) -> Result<(), Box> { + let table_root = "memory:///earliest_commit/"; + let log_root = "memory:///earliest_commit/_delta_log/"; + let engine = setup_earliest_commit_table(&setup, table_root).await; + + let result = unsafe { + get_earliest_commit( engine.shallow_copy(), - )) + kernel_string_slice!(log_root), + earliest_ratified, + commit_type, + ) }; - assert!(did_checkpoint); - // Verify `_last_checkpoint` exists and looks sane. + match expected { + Ok(version) => assert_eq!(ok_or_panic(result), version), + Err(kind) => assert_extern_result_error_with_message(result, kind, None), + } + + unsafe { free_engine(engine) } + Ok(()) + } + + // TODO: (PR #2307) will introduce a helper function for setting up storage, engine. + // The test will need to refactor to use the helper function. + #[tokio::test] + async fn test_snapshot_timestamp_no_ict() -> Result<(), Box> { + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///test_table/"; + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + + let engine = DefaultEngineBuilder::new(storage.clone()).build(); + let engine = engine_to_handle(Arc::new(engine), allocate_err); + let snap = + unsafe { build_snapshot(kernel_string_slice!(table_root), engine.shallow_copy()) }; + + let ts = unsafe { + ok_or_panic(snapshot_timestamp( + snap.shallow_copy(), + engine.shallow_copy(), + )) + }; + // ICT is not enabled -- falls back to commit file mtime (written "now"). + let now_ms = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_millis() as i64; + let two_days_ms = 2 * 24 * 60 * 60 * 1000_i64; + assert!( + (now_ms - two_days_ms..=now_ms).contains(&ts), + "timestamp {ts} not within 2 days of now {now_ms}" + ); + + unsafe { free_snapshot(snap) } + unsafe { free_engine(engine) } + Ok(()) + } + + // TODO: (PR #2307) will introduce a helper function for setting up storage, engine. + // The test will need to refactor to use the helper function. + #[tokio::test] + async fn test_snapshot_timestamp_ict_enabled() -> Result<(), Box> { + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///test_table/"; + + // create_table with "inCommitTimestamp" in writer_features sets up: + // - protocol v3.7 with writerFeatures=["inCommitTimestamp"] + // - metadata config: enableInCommitTimestamps=true, enablement version/timestamp + // - commitInfo with inCommitTimestamp=TEST_ICT_ENABLEMENT_TIMESTAMP + create_table( + storage.clone(), + Url::parse(table_root)?, + Arc::new(StructType::try_new([]).unwrap()), + &[], + true, + vec![], + vec!["inCommitTimestamp"], + ) + .await?; + + let engine = DefaultEngineBuilder::new(storage.clone()).build(); + let engine = engine_to_handle(Arc::new(engine), allocate_err); + let snap = + unsafe { build_snapshot(kernel_string_slice!(table_root), engine.shallow_copy()) }; + + let ts = unsafe { + ok_or_panic(snapshot_timestamp( + snap.shallow_copy(), + engine.shallow_copy(), + )) + }; + assert_eq!(ts, TEST_ICT_ENABLEMENT_TIMESTAMP); + + unsafe { free_snapshot(snap) } + unsafe { free_engine(engine) } + Ok(()) + } + + type HistoryQueryFn = unsafe extern "C" fn( + Handle, + Handle, + i64, + FfiHistoryCommitType, + ) -> ExternResult; + + #[rstest] + #[case::latest_version_query_at_ict(latest_version_as_of, TEST_ICT_ENABLEMENT_TIMESTAMP, Ok((0, TEST_ICT_ENABLEMENT_TIMESTAMP)))] + #[case::first_version_after_query_at_ict(first_version_after, TEST_ICT_ENABLEMENT_TIMESTAMP, Ok((0, TEST_ICT_ENABLEMENT_TIMESTAMP)))] + #[case::latest_version_query_out_of_range(latest_version_as_of, TEST_ICT_ENABLEMENT_TIMESTAMP - 1, Err(KernelError::LogHistoryError))] + #[case::first_version_after_query_out_of_range(first_version_after, TEST_ICT_ENABLEMENT_TIMESTAMP + 1, Err(KernelError::LogHistoryError))] + #[tokio::test] + async fn test_snapshot_version_at_timestamp_cases( + #[case] query: HistoryQueryFn, + #[case] timestamp: i64, + #[case] expected: Result<(Version, i64), KernelError>, + #[values(FfiHistoryCommitType::Published, FfiHistoryCommitType::Recreatable)] + commit_type: FfiHistoryCommitType, + ) -> Result<(), Box> { + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///test_table/"; + + create_table( + storage.clone(), + Url::parse(table_root)?, + Arc::new(StructType::try_new([]).unwrap()), + &[], + true, + vec![], + vec!["inCommitTimestamp"], + ) + .await?; + + let engine = DefaultEngineBuilder::new(storage.clone()).build(); + let engine = engine_to_handle(Arc::new(engine), allocate_err); + let snap = + unsafe { build_snapshot(kernel_string_slice!(table_root), engine.shallow_copy()) }; + + let result = unsafe { + query( + snap.shallow_copy(), + engine.shallow_copy(), + timestamp, + commit_type, + ) + }; + + match expected { + Ok((version, ts)) => { + let commit = ok_or_panic(result); + assert_eq!(commit.version, version); + assert_eq!(commit.timestamp, ts); + } + Err(kind) => assert_extern_result_error_with_message(result, kind, None), + } + + unsafe { free_snapshot(snap) } + unsafe { free_engine(engine) } + Ok(()) + } + + #[rstest] + #[case( + METADATA_WITH_TABLE_PROPERTIES, + HashMap::from([ + (String::from("delta.appendOnly"), String::from("true")), + (String::from("custom.key"), String::from("custom_value")), + ]) + )] + #[case(METADATA, HashMap::new())] + #[tokio::test] + async fn test_visit_metadata_configuration( + #[case] metadata: &str, + #[case] expected: HashMap, + ) -> Result<(), Box> { + let table_root = "memory:///"; + let storage = Arc::new(InMemory::new()); + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string_with_metadata(vec![TestAction::Metadata], metadata), + ) + .await?; + + let engine = DefaultEngineBuilder::new(storage.clone()).build(); + let engine = engine_to_handle(Arc::new(engine), allocate_err); + + let snap = + unsafe { build_snapshot(kernel_string_slice!(table_root), engine.shallow_copy()) }; + + extern "C" fn collect_property( + engine_context: NullableCvoid, + key: KernelStringSlice, + value: KernelStringSlice, + ) { + let map = + unsafe { &mut *(engine_context.unwrap().as_ptr() as *mut HashMap) }; + let k = unsafe { String::try_from_slice(&key) }.unwrap(); + let v = unsafe { String::try_from_slice(&value) }.unwrap(); + map.insert(k, v); + } + + let mut collected: HashMap = HashMap::new(); + let ctx = NonNull::new(&mut collected as *mut _ as *mut c_void); + unsafe { visit_metadata_configuration(snap.shallow_copy(), ctx, collect_property) }; + + assert_eq!(collected, expected); + + unsafe { free_snapshot(snap) } + unsafe { free_engine(engine) } + Ok(()) + } + + // V2 protocol + metadata declaring the `v2Checkpoint` reader+writer feature. + // + // Two columns (id INT, val STRING), no partitions, no extra table properties. Used by every + // test below that needs sidecar emission or a V2 checkpoint to succeed. + const V2_CHECKPOINT_PROTOCOL_AND_METADATA: &str = concat!( + r#"{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["v2Checkpoint"],"writerFeatures":["v2Checkpoint"]}}"#, + "\n", + r#"{"metaData":{"id":"5fba94ed-9794-4965-ba6e-6ee3c0d22af9","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"val\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1587968585495}}"#, + ); + + /// Seed a table at version 0 from [`test_utils::METADATA`] (protocol + metadata, plus its + /// leading `commitInfo` which the checkpoint writer ignores), then append `num_add_actions` + /// single-add commits at versions 1..=num_add_actions. + async fn seed_v1_table( + storage: &DynObjectStore, + table_root: &str, + num_add_actions: usize, + ) -> Result<(), Box> { + add_commit( + table_root, + storage, + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + for i in 0..num_add_actions { + add_commit( + table_root, + storage, + (i + 1) as u64, + actions_to_string(vec![TestAction::Add(format!("file{i}.parquet"))]), + ) + .await?; + } + Ok(()) + } + + /// Like `seed_v1_table` but uses [`V2_CHECKPOINT_PROTOCOL_AND_METADATA`] so the + /// table declares the `v2Checkpoint` reader+writer feature. + async fn seed_v2_table( + storage: &InMemory, + table_root: &str, + num_add_actions: usize, + ) -> Result<(), Box> { + add_commit( + table_root, + storage, + 0, + V2_CHECKPOINT_PROTOCOL_AND_METADATA.to_string(), + ) + .await?; + for i in 0..num_add_actions { + add_commit( + table_root, + storage, + (i + 1) as u64, + actions_to_string_with_metadata( + vec![TestAction::Add(format!("file{i}.parquet"))], + V2_CHECKPOINT_PROTOCOL_AND_METADATA, + ), + ) + .await?; + } + Ok(()) + } + + /// Count the entries in a given object_store prefix. + async fn count_objects_with_prefix(storage: &InMemory, prefix: &str) -> usize { + let prefix_path = Path::from(prefix); + storage + .list_with_delimiter(Some(&prefix_path)) + .await + .expect("list_with_delimiter should succeed") + .objects + .len() + } + + /// Extract the `Handle` from the `Written` variant; panics on + /// `AlreadyExists`. Caller owns the returned handle and must `free_snapshot` it. + fn unwrap_written(result: FfiCheckpointWriteResult) -> Handle { + match result { + FfiCheckpointWriteResult::Written(snap) => snap, + FfiCheckpointWriteResult::AlreadyExists(_) => { + panic!("expected Written, got AlreadyExists") + } + } + } + + /// Common setup for the checkpoint tests: seed an in-memory table (V1 or V2) with + /// `num_add_actions` single-add commits, then build a `DefaultEngine` and a latest-version + /// snapshot. + /// + /// The engine is wired to a multi-threaded task executor on purpose: `Snapshot::checkpoint` + /// drives nested `block_on` calls that would deadlock on the default single-threaded + /// background executor. + /// + /// Returns the backing `storage` (for post-checkpoint assertions) plus the owned `engine` + /// and `snapshot` handles; the caller must `free_snapshot` and `free_engine` them. + async fn setup_checkpoint_test( + is_v2: bool, + num_add_actions: usize, + ) -> Result< + ( + Arc, + Handle, + Handle, + ), + Box, + > { + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///"; + if is_v2 { + seed_v2_table(storage.as_ref(), table_root, num_add_actions).await?; + } else { + seed_v1_table(storage.as_ref(), table_root, num_add_actions).await?; + } + + let executor = Arc::new(TokioMultiThreadExecutor::new( + tokio::runtime::Handle::current(), + )); + let engine = engine_to_handle( + Arc::new( + DefaultEngineBuilder::new(storage.clone()) + .with_task_executor(executor) + .build(), + ), + allocate_err, + ); + let snapshot = + unsafe { build_snapshot(kernel_string_slice!(table_root), engine.shallow_copy()) }; + Ok((storage, engine, snapshot)) + } + + // Checkpoint sidecar shape is driven by (table kind, spec). Each case writes a checkpoint and + // asserts the resulting `_delta_log/_sidecars/` object count. + // + // NOTE: Snapshot::checkpoint requires a multi-threaded tokio task executor to avoid deadlocks. + #[rstest] + // non-V2 table, spec = None => kernel auto-picks V1, inline, no sidecars. + #[case::none_spec_v1(false, 2, None, 0)] + // V2 table, explicit no-sidecar => V2 manifest with file actions inline. + #[case::v2_no_sidecar(true, 2, Some(FfiCheckpointSpec::V2NoSidecar), 0)] + // V2 table, default hint (50k) => 3 actions fit in a single sidecar. + #[case::v2_default_hint( + true, + 3, + Some(FfiCheckpointSpec::V2WithSidecar { + file_actions_per_sidecar_hint: OptionalValue::None, + }), + 1 + )] + // V2 table, hint=2 over 8 actions => ceil(8/2)=4 sidecars. + #[case::v2_hint_2( + true, + 8, + Some(FfiCheckpointSpec::V2WithSidecar { + file_actions_per_sidecar_hint: OptionalValue::Some(2), + }), + 4 + )] + // Skipped under Miri: writes checkpoint parquet (minutes of safe work under the interpreter). + // Its unsafe (checkpoint_snapshot) is unconditional in `spec`, so the Some(V2WithSidecar) path + // adds no unsafe over test_checkpoint_snapshot_v2_with_sidecars_zero_hint_returns_error (runs + // under Miri); the checkpoint/free handles are covered by test_setting_multithread_executor. + // Sidecar-shape is safe kernel logic. Runs (all cases) under normal cargo test / nextest. + // + // DO NOT ADD NEW `unsafe` HERE (unsafe not already run by a Miri test): Miri never runs this + // test, so unsafe unique to it goes unchecked for undefined behavior. Put it in the anchor. + #[cfg_attr( + miri, + ignore = "writes checkpoint parquet (no unique unsafe); minutes under Miri" + )] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_checkpoint_snapshot_sidecar_shape( + #[case] is_v2: bool, + #[case] num_add_actions: usize, + #[case] spec: Option, + #[case] expected_sidecars: usize, + ) -> Result<(), Box> { + let (storage, engine, snapshot) = setup_checkpoint_test(is_v2, num_add_actions).await?; + + let result = unsafe { + ok_or_panic(checkpoint_snapshot( + snapshot.shallow_copy(), + engine.shallow_copy(), + spec.as_ref(), + )) + }; + let written_snap = unwrap_written(result); + + let sidecar_count = + count_objects_with_prefix(storage.as_ref(), "_delta_log/_sidecars/").await; + assert_eq!(sidecar_count, expected_sidecars); + + unsafe { free_snapshot(written_snap) } + unsafe { free_snapshot(snapshot) } + unsafe { free_engine(engine) } + Ok(()) + } + + // A zero sidecar hint is invalid: the kernel requires `file_actions_per_sidecar_hint > 0`, + // surfaced through the FFI as `CheckpointWriteError`. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_checkpoint_snapshot_v2_with_sidecars_zero_hint_returns_error( + ) -> Result<(), Box> { + let (_storage, engine, snapshot) = setup_checkpoint_test(true, 3).await?; + + let spec = FfiCheckpointSpec::V2WithSidecar { + file_actions_per_sidecar_hint: OptionalValue::Some(0), + }; + let extern_result = unsafe { + checkpoint_snapshot(snapshot.shallow_copy(), engine.shallow_copy(), Some(&spec)) + }; + assert_extern_result_error_with_message( + extern_result, + KernelError::CheckpointWriteError, + Some("Error writing checkpoint: file_actions_per_sidecar_hint must be greater than 0"), + ); + + unsafe { free_snapshot(snapshot) } + unsafe { free_engine(engine) } + Ok(()) + } + + // Checkpoint on V1 table with V2 spec => `KernelError::CheckpointWriteError`. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_checkpoint_snapshot_v2_on_non_v2_table_returns_checkpoint_write_error( + ) -> Result<(), Box> { + let (_storage, engine, snapshot) = setup_checkpoint_test(false, 1).await?; + + let spec = FfiCheckpointSpec::V2NoSidecar; + let extern_result = unsafe { + checkpoint_snapshot(snapshot.shallow_copy(), engine.shallow_copy(), Some(&spec)) + }; + assert_extern_result_error_with_message( + extern_result, + KernelError::CheckpointWriteError, + Some("Error writing checkpoint: CheckpointSpec::V2 requires the v2Checkpoint table feature to be supported"), + ); + + unsafe { free_snapshot(snapshot) } + unsafe { free_engine(engine) } + Ok(()) + } + + // Calling `checkpoint_snapshot` twice on the SAME (stale) input snapshot returns `Written` + // both times. The `AlreadyExists` short-circuit keys off the snapshot's own log segment + // (`checkpoint_version == end_version`), which never updates for the original handle, and the + // parquet writer silently overwrites on disk -- so neither call can observe `AlreadyExists`. + // Each returned snapshot reports the input version. (The `AlreadyExists` path is covered by + // `test_checkpoint_snapshot_written_snapshot_is_usable`, which chains the snapshot returned by + // the first checkpoint.) + // + // NOTE: Snapshot::checkpoint requires a multi-threaded tokio task executor to avoid deadlocks. + // Skipped under Miri: writes checkpoint parquet (minutes under the interpreter). Its unsafe FFI + // (checkpoint_snapshot, free_snapshot) is covered by test_setting_multithread_executor, and + // version()/SharedSnapshot::as_ref by test_snapshot. AlreadyExists/overwrite is safe kernel + // logic. Runs under normal cargo test / nextest. + // DO NOT ADD NEW `unsafe` HERE (unsafe not already run by a Miri test): Miri never runs this + // test, so unsafe unique to it goes unchecked for undefined behavior. Put it in the anchor. + #[cfg_attr( + miri, + ignore = "writes checkpoint parquet (no unique unsafe); minutes under Miri" + )] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_checkpoint_snapshot_second_call_returns_consistent_snapshot( + ) -> Result<(), Box> { + let (_storage, engine, snapshot) = setup_checkpoint_test(false, 2).await?; + let input_version = unsafe { version(snapshot.shallow_copy()) }; + + // First call on the original snapshot: writes the checkpoint => `Written`. + let first_snap = unsafe { + unwrap_written(ok_or_panic(checkpoint_snapshot( + snapshot.shallow_copy(), + engine.shallow_copy(), + None, + ))) + }; + let first_version = unsafe { version(first_snap.shallow_copy()) }; + assert_eq!(first_version, input_version); + unsafe { free_snapshot(first_snap) }; + + // Second call on the same (stale) input snapshot: overwrites => `Written` again. + let second_snap = unsafe { + unwrap_written(ok_or_panic(checkpoint_snapshot( + snapshot.shallow_copy(), + engine.shallow_copy(), + None, + ))) + }; + let second_version = unsafe { version(second_snap.shallow_copy()) }; + assert_eq!(second_version, input_version); + unsafe { free_snapshot(second_snap) }; + + unsafe { free_snapshot(snapshot) } + unsafe { free_engine(engine) } + Ok(()) + } + + // The `Written` snapshot is usable downstream: `version` matches input, and a chained + // `checkpoint_snapshot` returns `AlreadyExists` (same table version). Also validates the + // `_delta_log/_last_checkpoint` content (version, numOfAddFiles, size, sizeInBytes), which + // `test_checkpoint_snapshot_sidecar_shape` doesn't cover. + // Skipped under Miri: writes checkpoint parquet (minutes under the interpreter). Its unsafe FFI + // (checkpoint_snapshot, free_snapshot) is covered by test_setting_multithread_executor, and + // version()/SharedSnapshot::as_ref by test_snapshot. _last_checkpoint content is safe kernel + // logic. Runs under normal cargo test / nextest. + // DO NOT ADD NEW `unsafe` HERE (unsafe not already run by a Miri test): Miri never runs this + // test, so unsafe unique to it goes unchecked for undefined behavior. Put it in the anchor. + #[cfg_attr( + miri, + ignore = "writes checkpoint parquet (no unique unsafe); minutes under Miri" + )] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_checkpoint_snapshot_written_snapshot_is_usable( + ) -> Result<(), Box> { + let (storage, engine, snapshot) = setup_checkpoint_test(false, 2).await?; + let input_version = unsafe { version(snapshot.shallow_copy()) }; + + let result = unsafe { + ok_or_panic(checkpoint_snapshot( + snapshot.shallow_copy(), + engine.shallow_copy(), + None, + )) + }; + let written_snap = unwrap_written(result); + + // (a) version() on the returned snapshot matches the input version. + let returned_version = unsafe { version(written_snap.shallow_copy()) }; + assert_eq!(returned_version, input_version); + + // (b) `_last_checkpoint` exists and describes the checkpoint we just wrote. let last_checkpoint = storage .get(&Path::from("_delta_log/_last_checkpoint")) .await?; - let last_checkpoint_bytes = last_checkpoint.bytes().await?; - let v: Value = serde_json::from_slice(last_checkpoint_bytes.as_ref())?; + let v: Value = serde_json::from_slice(last_checkpoint.bytes().await?.as_ref())?; assert_eq!(v["version"].as_u64(), Some(2)); - // Here file1 was removed, so only file2 and - // file3 remain. + // 2 adds at versions 1..=2, no removes => 2 live add files. assert_eq!(v["numOfAddFiles"].as_u64(), Some(2)); - // size = 1 protocol + 1 metadata + 2 live adds + // size = 1 protocol + 1 metadata + 2 live adds. assert_eq!(v["size"].as_u64(), Some(4)); - - // Cross-check checkpoint file size against `_last_checkpoint.sizeInBytes`. + // Cross-check the checkpoint parquet file size against `_last_checkpoint.sizeInBytes`. let checkpoint_path = Path::from("_delta_log/00000000000000000002.checkpoint.parquet"); let checkpoint_size = storage.head(&checkpoint_path).await?.size; assert_eq!(v["sizeInBytes"].as_u64(), Some(checkpoint_size)); + // (c) chain a second checkpoint_snapshot(returned_snap, engine, None) => + // AlreadyExists. + let chained = unsafe { + ok_or_panic(checkpoint_snapshot( + written_snap.shallow_copy(), + engine.shallow_copy(), + None, + )) + }; + match chained { + FfiCheckpointWriteResult::AlreadyExists(snap) => unsafe { free_snapshot(snap) }, + FfiCheckpointWriteResult::Written(_) => panic!("expected AlreadyExists"), + } + + unsafe { free_snapshot(written_snap) } unsafe { free_snapshot(snapshot) } unsafe { free_engine(engine) } Ok(()) } // Test checkpoint using FFI engine builder APIs with multithreaded executor. - // NOTE: We made this a sync test to simulate the expected case: C code calling FFI APIs to build engine without existing tokio runtime. + // NOTE: We made this a sync test to simulate the expected case: C code calling FFI APIs to + // build engine without existing tokio runtime. + // + // Miri anchor for checkpoint FFI: covers the success-path checkpoint_snapshot -> Written + // handle -> free_snapshot that the skipped checkpoint tests share, and uniquely covers + // set_builder_with_multithreaded_executor (no other test calls it). Skipping it drops that + // coverage under Miri. #[cfg(feature = "default-engine-base")] #[test] fn test_setting_multithread_executor() -> Result<(), Box> { @@ -1195,15 +2836,75 @@ mod tests { .ok_or_else(|| delta_kernel::Error::generic("Invalid path"))?; let storage = Arc::new(LocalFileSystem::new()); - // Create a minimal table history: initial metadata+protocol (no commitInfo), then some - // add/remove commits. + // Use a temporary runtime for async setup, then drop it before the FFI calls so the engine + // built below drives all async work through its own (FFI-configured) executor. + { + let rt = tokio::runtime::Runtime::new()?; + rt.block_on(seed_v1_table(storage.as_ref(), table_root, 2))?; + } // runtime dropped here, before FFI calls + + // Build engine using FFI APIs + let builder = unsafe { + ok_or_panic(get_engine_builder( + kernel_string_slice!(table_root), + allocate_err, + )) + }; + unsafe { set_builder_with_multithreaded_executor(builder.as_mut().unwrap(), 2, 0) }; + let engine = unsafe { ok_or_panic(builder_build(builder)) }; + + let snapshot = + unsafe { build_snapshot(kernel_string_slice!(table_root), engine.shallow_copy()) }; + + let did_checkpoint = unsafe { + match ok_or_panic(checkpoint_snapshot( + snapshot.shallow_copy(), + engine.shallow_copy(), + None, + )) { + FfiCheckpointWriteResult::Written(snap) => { + free_snapshot(snap); + true + } + FfiCheckpointWriteResult::AlreadyExists(snap) => { + free_snapshot(snap); + true + } + } + }; + assert!(did_checkpoint); + + unsafe { free_snapshot(snapshot) } + unsafe { free_engine(engine) } + Ok(()) + } + + // Test building an engine via the FFI builder with custom read-path I/O concurrency, then + // exercising the read path (snapshot) to confirm the configured engine works. + #[cfg(feature = "default-engine-base")] + #[rstest] + #[case(4, 8)] + #[case(0, 0)] + fn test_setting_io_concurrency( + #[case] buffer_size: usize, + #[case] batch_size: usize, + ) -> Result<(), Box> { + use delta_kernel::object_store::local::LocalFileSystem; + use tempfile::tempdir; + + let tmp_dir = tempdir()?; + let tmp_path = tmp_dir.path(); + let table_root = tmp_path + .to_str() + .ok_or_else(|| delta_kernel::Error::generic("Invalid path"))?; + let storage = Arc::new(LocalFileSystem::new()); + let protocol_and_metadata = METADATA .lines() .skip(1) // skip commitInfo .collect::>() .join("\n"); - // Use a temporary runtime for async setup, then drop it before FFI calls { let rt = tokio::runtime::Runtime::new()?; rt.block_on(async { @@ -1218,44 +2919,25 @@ mod tests { ]), ) .await?; - add_commit( - &table_root, - storage.as_ref(), - 2, - actions_to_string(vec![ - TestAction::Add("file3.parquet".into()), - TestAction::Remove("file1.parquet".into()), - ]), - ) - .await?; Ok::<_, Box>(()) })?; - } // runtime dropped here, before FFI calls + } - // Build engine using FFI APIs let builder = unsafe { ok_or_panic(get_engine_builder( kernel_string_slice!(table_root), allocate_err, )) }; - unsafe { set_builder_with_multithreaded_executor(builder.as_mut().unwrap(), 2, 0) }; - let engine = unsafe { ok_or_panic(builder_build(builder)) }; - - let snapshot = unsafe { - ok_or_panic(snapshot( - kernel_string_slice!(table_root), - engine.shallow_copy(), - )) + unsafe { + set_builder_with_io_concurrency(builder.as_mut().unwrap(), buffer_size, batch_size) }; + let engine = unsafe { ok_or_panic(builder_build(builder)) }; - let did_checkpoint = unsafe { - ok_or_panic(checkpoint_snapshot( - snapshot.shallow_copy(), - engine.shallow_copy(), - )) - }; - assert!(did_checkpoint); + let snapshot = + unsafe { build_snapshot(kernel_string_slice!(table_root), engine.shallow_copy()) }; + let version = unsafe { version(snapshot.shallow_copy()) }; + assert_eq!(version, 1); unsafe { free_snapshot(snapshot) } unsafe { free_engine(engine) } @@ -1277,12 +2959,8 @@ mod tests { let engine = DefaultEngineBuilder::new(storage.clone()).build(); let engine = engine_to_handle(Arc::new(engine), allocate_err); - let snapshot = unsafe { - ok_or_panic(snapshot( - kernel_string_slice!(table_root), - engine.shallow_copy(), - )) - }; + let snapshot = + unsafe { build_snapshot(kernel_string_slice!(table_root), engine.shallow_copy()) }; let partition_count = unsafe { get_partition_column_count(snapshot.shallow_copy()) }; assert_eq!(partition_count, 1, "Should have one partition"); @@ -1321,7 +2999,12 @@ mod tests { // Get a non-existent snapshot, this will call allocate_null_err let snapshot_at_non_existent_version = unsafe { - snapshot_at_version(kernel_string_slice!(table_root), engine.shallow_copy(), 1) + let mut ptr = ok_or_panic(get_snapshot_builder( + kernel_string_slice!(table_root), + engine.shallow_copy(), + )); + snapshot_builder_set_version(&mut ptr, 1); + snapshot_builder_build(ptr) }; assert!(snapshot_at_non_existent_version.is_err()); @@ -1329,20 +3012,12 @@ mod tests { Ok(()) } - #[cfg(feature = "catalog-managed")] #[tokio::test] async fn test_snapshot_log_tail() -> Result<(), Box> { - use test_utils::add_staged_commit; - let storage = Arc::new(InMemory::new()); let table_root = "memory:///test_table/"; - - add_commit( - table_root, - storage.as_ref(), - 0, - actions_to_string(vec![TestAction::Metadata]), - ) - .await?; + let (storage, engine, snap) = + make_catalog_managed_engine_and_v0_snapshot(table_root).await?; + unsafe { free_snapshot(snap) }; let commit1 = add_staged_commit( table_root, storage.as_ref(), @@ -1350,8 +3025,6 @@ mod tests { actions_to_string(vec![TestAction::Add("path1".into())]), ) .await?; - let engine = DefaultEngineBuilder::new(storage.clone()).build(); - let engine = engine_to_handle(Arc::new(engine), allocate_err); let commit1_path = format!( "{}_delta_log/_staged_commits/{}", @@ -1366,25 +3039,49 @@ mod tests { len: log_tail.len(), }; let snapshot = unsafe { - ok_or_panic(snapshot_with_log_tail( + let mut ptr = ok_or_panic(get_snapshot_builder( kernel_string_slice!(table_root), engine.shallow_copy(), - log_tail.clone(), - )) + )); + ok_or_panic(snapshot_builder_set_log_tail(&mut ptr, log_tail.clone())); + snapshot_builder_set_max_catalog_version(&mut ptr, 1); + ok_or_panic(snapshot_builder_build(ptr)) }; let snapshot_version = unsafe { version(snapshot.shallow_copy()) }; assert_eq!(snapshot_version, 1); + // Test for failure when not passing in max_catalog_version + let invalid_snapshot = unsafe { + snapshot_at_version_with_log_tail( + kernel_string_slice!(table_root), + engine.shallow_copy(), + 1, + log_tail.clone(), + None, // max_catalog_version + ) + }; + assert_extern_result_error_with_message( + invalid_snapshot, + KernelError::GenericError, + Some(concat!( + "Max catalog version error: Max catalog version is required when providing ", + "staged commits in the log tail. ", + "Use with_max_catalog_version()." + )), + ); + // Test getting snapshot at version let snapshot2 = unsafe { - ok_or_panic(snapshot_at_version_with_log_tail( + let mut ptr = ok_or_panic(get_snapshot_builder( kernel_string_slice!(table_root), engine.shallow_copy(), - 1, - log_tail, - )) + )); + snapshot_builder_set_version(&mut ptr, 1); + ok_or_panic(snapshot_builder_set_log_tail(&mut ptr, log_tail)); + snapshot_builder_set_max_catalog_version(&mut ptr, 1); + ok_or_panic(snapshot_builder_build(ptr)) }; - let snapshot_version = unsafe { version(snapshot.shallow_copy()) }; + let snapshot_version = unsafe { version(snapshot2.shallow_copy()) }; assert_eq!(snapshot_version, 1); unsafe { free_snapshot(snapshot) } @@ -1392,4 +3089,529 @@ mod tests { unsafe { free_engine(engine) } Ok(()) } + + #[tokio::test] + async fn test_builder_from_existing_snapshot_advances_to_latest_and_pinned_version( + ) -> Result<(), Box> { + let path = "memory:///"; + let (storage, engine, snapshot_at_v0) = make_engine_and_v0_snapshot(path).await?; + assert_eq!(unsafe { version(snapshot_at_v0.shallow_copy()) }, 0); + + add_commit( + path, + storage.as_ref(), + 1, + actions_to_string(vec![TestAction::Add("file1.parquet".into())]), + ) + .await?; + add_commit( + path, + storage.as_ref(), + 2, + actions_to_string(vec![TestAction::Add("file2.parquet".into())]), + ) + .await?; + + let snapshot_at_v2 = unsafe { + let ptr = ok_or_panic(get_snapshot_builder_from( + snapshot_at_v0.shallow_copy(), + engine.shallow_copy(), + )); + ok_or_panic(snapshot_builder_build(ptr)) + }; + assert_eq!(unsafe { version(snapshot_at_v2.shallow_copy()) }, 2); + + let snapshot_at_v1 = unsafe { + let mut ptr = ok_or_panic(get_snapshot_builder_from( + snapshot_at_v0.shallow_copy(), + engine.shallow_copy(), + )); + snapshot_builder_set_version(&mut ptr, 1); + ok_or_panic(snapshot_builder_build(ptr)) + }; + assert_eq!(unsafe { version(snapshot_at_v1.shallow_copy()) }, 1); + + unsafe { free_snapshot(snapshot_at_v2) } + unsafe { free_snapshot(snapshot_at_v1) } + unsafe { free_snapshot(snapshot_at_v0) } + unsafe { free_engine(engine) } + Ok(()) + } + + #[tokio::test] + async fn test_builder_from_existing_snapshot_rejects_earlier_version( + ) -> Result<(), Box> { + let path = "memory:///"; + let (storage, engine, snapshot_at_v0) = make_engine_and_v0_snapshot(path).await?; + + add_commit( + path, + storage.as_ref(), + 1, + actions_to_string(vec![TestAction::Add("file1.parquet".into())]), + ) + .await?; + add_commit( + path, + storage.as_ref(), + 2, + actions_to_string(vec![TestAction::Add("file2.parquet".into())]), + ) + .await?; + + // build a v2 snapshot to use as the base + let snapshot_at_v2 = unsafe { + let ptr = ok_or_panic(get_snapshot_builder_from( + snapshot_at_v0.shallow_copy(), + engine.shallow_copy(), + )); + ok_or_panic(snapshot_builder_build(ptr)) + }; + assert_eq!(unsafe { version(snapshot_at_v2.shallow_copy()) }, 2); + + // pinning to a version older than the hint snapshot is rejected + let result = unsafe { + let mut ptr = ok_or_panic(get_snapshot_builder_from( + snapshot_at_v2.shallow_copy(), + engine.shallow_copy(), + )); + snapshot_builder_set_version(&mut ptr, 1); + snapshot_builder_build(ptr) + }; + assert_extern_result_error_with_message( + result, + KernelError::GenericError, + Some("Generic delta kernel error: Requested snapshot version 1 is older than snapshot hint version 2"), + ); + + unsafe { free_snapshot(snapshot_at_v2) } + unsafe { free_snapshot(snapshot_at_v0) } + unsafe { free_engine(engine) } + Ok(()) + } + + #[tokio::test] + async fn test_snapshot_with_prev_snapshot_and_log_tail( + ) -> Result<(), Box> { + let path = "memory:///"; + let (storage, engine, snapshot_at_v0) = + make_catalog_managed_engine_and_v0_snapshot(path).await?; + assert_eq!(unsafe { version(snapshot_at_v0.shallow_copy()) }, 0); + + // Add staged commit (version 1) + let commit1 = add_staged_commit( + path, + storage.as_ref(), + 1, + actions_to_string(vec![TestAction::Add("path1.parquet".into())]), + ) + .await?; + + // Add another staged commit (version 2) + let commit2 = add_staged_commit( + path, + storage.as_ref(), + 2, + actions_to_string(vec![TestAction::Add("path2.parquet".into())]), + ) + .await?; + + // Build log tail with both commits + let commit1_path = format!( + "{}_delta_log/_staged_commits/{}", + path, + commit1.filename().unwrap() + ); + let commit2_path = format!( + "{}_delta_log/_staged_commits/{}", + path, + commit2.filename().unwrap() + ); + let log_path1 = + log_path::FfiLogPath::new(kernel_string_slice!(commit1_path), 123456789, 100); + let log_path2 = + log_path::FfiLogPath::new(kernel_string_slice!(commit2_path), 123456790, 101); + let log_tail = [log_path1, log_path2]; + let log_tail_array = log_path::LogPathArray { + ptr: log_tail.as_ptr(), + len: log_tail.len(), + }; + + let snapshot_at_v2 = unsafe { + let mut ptr = ok_or_panic(get_snapshot_builder_from( + snapshot_at_v0.shallow_copy(), + engine.shallow_copy(), + )); + ok_or_panic(snapshot_builder_set_log_tail( + &mut ptr, + log_tail_array.clone(), + )); + snapshot_builder_set_max_catalog_version(&mut ptr, 2); + ok_or_panic(snapshot_builder_build(ptr)) + }; + assert_eq!(unsafe { version(snapshot_at_v2.shallow_copy()) }, 2); + + let snapshot_at_v1 = unsafe { + let mut ptr = ok_or_panic(get_snapshot_builder_from( + snapshot_at_v0.shallow_copy(), + engine.shallow_copy(), + )); + snapshot_builder_set_version(&mut ptr, 1); + ok_or_panic(snapshot_builder_set_log_tail(&mut ptr, log_tail_array)); + snapshot_builder_set_max_catalog_version(&mut ptr, 2); + ok_or_panic(snapshot_builder_build(ptr)) + }; + assert_eq!(unsafe { version(snapshot_at_v1.shallow_copy()) }, 1); + + unsafe { free_snapshot(snapshot_at_v2) } + unsafe { free_snapshot(snapshot_at_v1) } + unsafe { free_snapshot(snapshot_at_v0) } + unsafe { free_engine(engine) } + Ok(()) + } + + #[tokio::test] + async fn test_builder_from_table_path_builds_latest_version( + ) -> Result<(), Box> { + let storage = Arc::new(InMemory::new()); + let path = "memory:///"; + add_commit( + path, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + add_commit( + path, + storage.as_ref(), + 1, + actions_to_string(vec![TestAction::Add("file1.parquet".into())]), + ) + .await?; + let engine = engine_to_handle( + Arc::new(DefaultEngineBuilder::new(storage).build()), + allocate_err, + ); + + let snapshot_at_v1 = unsafe { + let ptr = ok_or_panic(get_snapshot_builder( + kernel_string_slice!(path), + engine.shallow_copy(), + )); + ok_or_panic(snapshot_builder_build(ptr)) + }; + assert_eq!(unsafe { version(snapshot_at_v1.shallow_copy()) }, 1); + + unsafe { free_snapshot(snapshot_at_v1) } + unsafe { free_engine(engine) } + Ok(()) + } + + #[tokio::test] + async fn test_free_snapshot_builder_without_building() -> Result<(), Box> + { + let path = "memory:///"; + let (_, engine, snap) = make_engine_and_v0_snapshot(path).await?; + unsafe { free_snapshot(snap) }; + + let ptr = unsafe { + ok_or_panic(get_snapshot_builder( + kernel_string_slice!(path), + engine.shallow_copy(), + )) + }; + + unsafe { free_snapshot_builder(ptr) }; + unsafe { free_engine(engine) } + Ok(()) + } + + // === Shared visitor state and callbacks for protocol/metadata tests === + + struct ProtocolVisitState { + min_reader: i32, + min_writer: i32, + reader_features: Vec, + writer_features: Vec, + } + + impl ProtocolVisitState { + fn new() -> Self { + Self { + min_reader: 0, + min_writer: 0, + reader_features: Vec::new(), + writer_features: Vec::new(), + } + } + } + + extern "C" fn protocol_version_cb(ctx: NullableCvoid, min_reader: i32, min_writer: i32) { + let state = unsafe { &mut *(ctx.unwrap().as_ptr() as *mut ProtocolVisitState) }; + state.min_reader = min_reader; + state.min_writer = min_writer; + } + + extern "C" fn protocol_feature_cb( + ctx: NullableCvoid, + is_reader: bool, + feature: KernelStringSlice, + ) { + let state = unsafe { &mut *(ctx.unwrap().as_ptr() as *mut ProtocolVisitState) }; + let name = unsafe { String::try_from_slice(&feature) }.unwrap(); + if is_reader { + state.reader_features.push(name); + } else { + state.writer_features.push(name); + } + } + + /// Visit protocol on a snapshot and return the collected state. + fn collect_protocol_state(snap: &handle::Handle) -> ProtocolVisitState { + let proto = unsafe { snapshot_get_protocol(snap.shallow_copy()) }; + let mut state = ProtocolVisitState::new(); + let ctx = NonNull::new(&mut state as *mut ProtocolVisitState as *mut c_void); + unsafe { + visit_protocol( + proto.shallow_copy(), + ctx, + protocol_version_cb, + protocol_feature_cb, + ) + }; + unsafe { free_protocol(proto) }; + state + } + + struct MetadataVisitState { + id: Option, + name: Option, + description: Option, + format_provider: Option, + has_created_time: bool, + created_time_ms: i64, + } + + impl MetadataVisitState { + fn new() -> Self { + Self { + id: None, + name: None, + description: None, + format_provider: None, + has_created_time: false, + created_time_ms: 0, + } + } + } + + /// Convert a [`KernelStringSlice`] to a [`String`] (test-only helper). + fn slice_to_string(slice: KernelStringSlice) -> String { + unsafe { String::try_from_slice(&slice) }.unwrap() + } + + extern "C" fn metadata_visit_cb( + ctx: NullableCvoid, + id: KernelStringSlice, + name: OptionalValue, + description: OptionalValue, + format_provider: KernelStringSlice, + has_created_time: bool, + created_time_ms: i64, + ) { + let state = unsafe { &mut *(ctx.unwrap().as_ptr() as *mut MetadataVisitState) }; + state.id = Some(slice_to_string(id)); + state.name = match name { + OptionalValue::Some(s) => Some(slice_to_string(s)), + OptionalValue::None => None, + }; + state.description = match description { + OptionalValue::Some(s) => Some(slice_to_string(s)), + OptionalValue::None => None, + }; + state.format_provider = Some(slice_to_string(format_provider)); + state.has_created_time = has_created_time; + state.created_time_ms = created_time_ms; + } + + /// Visit metadata on a snapshot and return the collected state. + fn collect_metadata_state(snap: &handle::Handle) -> MetadataVisitState { + let meta = unsafe { snapshot_get_metadata(snap.shallow_copy()) }; + let mut state = MetadataVisitState::new(); + let ctx = NonNull::new(&mut state as *mut MetadataVisitState as *mut c_void); + unsafe { visit_metadata(meta.shallow_copy(), ctx, metadata_visit_cb) }; + unsafe { free_metadata(meta) }; + state + } + + // === visit_protocol tests === + + #[tokio::test] + async fn test_visit_protocol_legacy() -> Result<(), Box> { + let (engine, snap) = setup_snapshot(METADATA.to_string()).await?; + let state = collect_protocol_state(&snap); + + assert_eq!(state.min_reader, 1); + assert_eq!(state.min_writer, 2); + assert!(state.reader_features.is_empty()); + assert!(state.writer_features.is_empty()); + + unsafe { free_snapshot(snap) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + #[tokio::test] + async fn test_builder_with_nonexistent_path_returns_error( + ) -> Result<(), Box> { + let storage = Arc::new(InMemory::new()); + let engine = engine_to_handle( + Arc::new(DefaultEngineBuilder::new(storage).build()), + allocate_err, + ); + + let result = unsafe { + let invalid_path = "not a valid url!"; + get_snapshot_builder(kernel_string_slice!(invalid_path), engine.shallow_copy()) + }; + assert_extern_result_error_with_message( + result, + KernelError::InvalidTableLocationError, + None, + ); + + unsafe { free_engine(engine) } + Ok(()) + } + + #[tokio::test] + async fn test_builder_at_nonexistent_version_returns_error( + ) -> Result<(), Box> { + let path = "memory:///"; + let (_, engine, snap) = make_engine_and_v0_snapshot(path).await?; + unsafe { free_snapshot(snap) }; + + let result = unsafe { + let mut ptr = ok_or_panic(get_snapshot_builder( + kernel_string_slice!(path), + engine.shallow_copy(), + )); + snapshot_builder_set_version(&mut ptr, 99); + snapshot_builder_build(ptr) + }; + assert_extern_result_error_with_message(result, KernelError::GenericError, None); + + unsafe { free_engine(engine) } + Ok(()) + } + + #[tokio::test] + async fn test_visit_protocol_with_features() -> Result<(), Box> { + let (engine, snap) = setup_snapshot(METADATA_WITH_FEATURES.to_string()).await?; + let state = collect_protocol_state(&snap); + + assert_eq!(state.min_reader, 3); + assert_eq!(state.min_writer, 7); + assert_eq!(state.reader_features, vec!["columnMapping"]); + let mut wf = state.writer_features.clone(); + wf.sort(); + assert_eq!(wf, vec!["columnMapping", "domainMetadata", "rowTracking"]); + + unsafe { free_snapshot(snap) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + #[tokio::test] + async fn test_visit_metadata_default() -> Result<(), Box> { + let (engine, snap) = setup_snapshot(METADATA.to_string()).await?; + let state = collect_metadata_state(&snap); + + assert_eq!( + state.id.as_deref(), + Some("5fba94ed-9794-4965-ba6e-6ee3c0d22af9") + ); + assert!( + state.name.is_none(), + "name should be None for default metadata" + ); + assert!( + state.description.is_none(), + "description should be None for default metadata" + ); + assert_eq!(state.format_provider.as_deref(), Some("parquet")); + assert!(state.has_created_time); + assert_eq!(state.created_time_ms, 1587968585495); + + unsafe { free_snapshot(snap) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + #[tokio::test] + async fn test_visit_metadata_with_name() -> Result<(), Box> { + let (engine, snap) = setup_snapshot(METADATA_WITH_FEATURES.to_string()).await?; + let state = collect_metadata_state(&snap); + + assert_eq!( + state.id.as_deref(), + Some("deadbeef-1234-5678-abcd-000000000000") + ); + assert_eq!(state.name.as_deref(), Some("test_table")); + assert!(state.description.is_none(), "description should be None"); + assert_eq!(state.format_provider.as_deref(), Some("parquet")); + assert!(state.has_created_time); + assert_eq!(state.created_time_ms, 1234567890000); + + unsafe { free_snapshot(snap) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + #[tokio::test] + async fn test_visit_metadata_with_description() -> Result<(), Box> { + let metadata_with_desc = concat!( + r#"{"commitInfo":{"timestamp":1587968586154,"operation":"WRITE","operationParameters":{},"isBlindAppend":true}}"#, + "\n", + r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#, + "\n", + r#"{"metaData":{"id":"5fba94ed-9794-4965-ba6e-6ee3c0d22af9","name":"my_table","description":"A test table","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[]}","partitionColumns":[],"configuration":{},"createdTime":1587968585495}}"#, + ); + let (engine, snap) = setup_snapshot(metadata_with_desc.to_string()).await?; + let state = collect_metadata_state(&snap); + + assert_eq!(state.name.as_deref(), Some("my_table")); + assert_eq!(state.description.as_deref(), Some("A test table")); + assert_eq!(state.format_provider.as_deref(), Some("parquet")); + assert!(state.has_created_time); + + unsafe { free_snapshot(snap) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + #[tokio::test] + async fn test_visit_metadata_without_created_time() -> Result<(), Box> { + let metadata_no_time = concat!( + r#"{"commitInfo":{"timestamp":1587968586154,"operation":"WRITE","operationParameters":{},"isBlindAppend":true}}"#, + "\n", + r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#, + "\n", + r#"{"metaData":{"id":"5fba94ed-9794-4965-ba6e-6ee3c0d22af9","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[]}","partitionColumns":[],"configuration":{}}}"#, + ); + let (engine, snap) = setup_snapshot(metadata_no_time.to_string()).await?; + let state = collect_metadata_state(&snap); + + assert_eq!( + state.id.as_deref(), + Some("5fba94ed-9794-4965-ba6e-6ee3c0d22af9") + ); + assert!(!state.has_created_time); + assert_eq!(state.created_time_ms, 0); + + unsafe { free_snapshot(snap) }; + unsafe { free_engine(engine) }; + Ok(()) + } } diff --git a/ffi/src/log_path.rs b/ffi/src/log_path.rs index cec6ae4178..6e2715fabb 100644 --- a/ffi/src/log_path.rs +++ b/ffi/src/log_path.rs @@ -13,8 +13,8 @@ use crate::{KernelStringSlice, TryFromStringSlice}; #[repr(C)] #[derive(Debug, Clone)] pub struct LogPathArray { - /// Pointer to the first element of the FfiLogPath array. If len is 0, this pointer may be null, - /// otherwise it must be non-null. + /// Pointer to the first element of the FfiLogPath array. If len is 0, this pointer may be + /// null, otherwise it must be non-null. pub ptr: *const FfiLogPath, /// Number of elements in the array pub len: usize, diff --git a/ffi/src/plans/executor.rs b/ffi/src/plans/executor.rs new file mode 100644 index 0000000000..22ecf3800a --- /dev/null +++ b/ffi/src/plans/executor.rs @@ -0,0 +1,356 @@ +//! This module provides an FFI-backed implementation of the [`PlanExecutor`] trait (allowing plan +//! execution to happen outside of Rust). +use std::sync::Arc; + +use delta_kernel::plans::proto::schema as proto_schema; +use delta_kernel::schema::StructType; +use delta_kernel::{DeltaResult, Error, Operation, ParquetFooter, PlanExecutor, PlanResult}; +use delta_kernel_ffi_macros::handle_descriptor; +use prost::Message as _; + +use crate::error::EngineExecResult; +use crate::plans::iter::{FfiBytesIter, FfiEngineDataIter, FfiFileMetaIter}; +use crate::plans::result::{CParquetFooter, CPlanResult}; +use crate::{kernel_bytes_slice, KernelBytesSlice, NullableCvoid}; + +/// A shared (`Arc`-like) handle to an [`PlanExecutor`]. +#[handle_descriptor(target=dyn PlanExecutor, mutable=false)] +pub struct SharedPlanExecutor; + +/// C callback, provided by the Engine, for executing an [`Operation`]. +/// +/// `context` - an opaque pointer, originally passed to +/// [`get_plan_executor`](super::get_plan_executor). +/// `plan_proto` - a byte slice containing the proto-serialized representation of an [`Operation`] +/// `out` - an out pointer into which the engine writes the result. +/// +/// Since the out result is written to caller (Kernel) provided memory, the kernel will also be +/// responsible for freeing it. Kernel will pre-initialize the out pointer to +/// [`EngineExecResult::Uninit`] before handing it to the engine upcall. +pub type CExecuteOpFn = extern "C" fn( + context: NullableCvoid, + plan_proto: KernelBytesSlice, + out: *mut EngineExecResult, +); + +/// A [`PlanExecutor`] implementation that forwards each [`Operation`] to a C callback. +/// +/// Constructed and managed via [`get_plan_executor`](super::get_plan_executor) and +/// [`free_plan_executor`](super::free_plan_executor). +pub struct FfiPlanExecutor { + context: NullableCvoid, + callback: CExecuteOpFn, +} + +impl FfiPlanExecutor { + /// Construct an [`FfiPlanExecutor`] from a context pointer and C callback. + /// + /// The `context` pointer and `callback` must satisfy the thread-safety and lifetime + /// requirements documented on [`get_plan_executor`](super::get_plan_executor). + pub(crate) fn new(context: NullableCvoid, callback: CExecuteOpFn) -> Self { + Self { context, callback } + } +} + +// SAFETY: the engine is expected to provide thread-safe context + callback function pointers. +unsafe impl Send for FfiPlanExecutor {} +unsafe impl Sync for FfiPlanExecutor {} + +impl PlanExecutor for FfiPlanExecutor { + fn execute_op(&self, op: Operation) -> DeltaResult { + let plan_proto_bytes = op.to_proto_bytes(); + let plan_proto_slice = kernel_bytes_slice!(plan_proto_bytes); + + let mut out = EngineExecResult::Uninit; + (self.callback)(self.context, plan_proto_slice, &mut out); + let plan_result = + match out { + EngineExecResult::Success(plan) => plan, + EngineExecResult::Failure(err) => return Err(err.into()), + EngineExecResult::Uninit => return Err(Error::internal_error( + "FFI engine returned from execute_op upcall without writing the plan result", + )), + }; + match plan_result { + CPlanResult::Unit => Ok(PlanResult::Unit), + CPlanResult::Data(it) => Ok(PlanResult::Data(Box::new(FfiEngineDataIter::new(it)))), + CPlanResult::FileMeta(it) => { + Ok(PlanResult::FileMeta(Box::new(FfiFileMetaIter::new(it)))) + } + CPlanResult::Bytes(it) => Ok(PlanResult::Bytes(Box::new(FfiBytesIter::new(it)))), + CPlanResult::ParquetFooter(footer) => { + Ok(PlanResult::ParquetFooter(decode_parquet_footer(footer)?)) + } + } + } +} + +/// Convert a [`CParquetFooter`] into a kernel [`ParquetFooter`]. +/// +/// Consumes the embedded [`ExclusiveRustBytes`](crate::ExclusiveRustBytes) handle carrying +/// the proto-serialized schema, returning an error if the bytes are not a valid schema proto +/// message. +fn decode_parquet_footer(footer: CParquetFooter) -> DeltaResult { + let CParquetFooter { schema_proto } = footer; + // SAFETY: ExclusiveRustBytes should only have a single owner, so consuming here is safe. + let bytes = *unsafe { schema_proto.into_inner() }; + let proto = proto_schema::StructType::decode(bytes.as_slice()).map_err(Error::generic_err)?; + let schema = Arc::new(StructType::try_from(proto)?); + Ok(ParquetFooter { schema }) +} + +#[cfg(test)] +mod tests { + use std::ffi::c_void; + use std::ptr::NonNull; + use std::sync::{Arc, Mutex}; + + use delta_kernel::arrow::array::ffi::FFI_ArrowArray; + use delta_kernel::plans::proto::{operation as proto, schema as proto_schema}; + use delta_kernel::schema::{DataType as KernelDataType, StructField, StructType}; + use delta_kernel::Error; + use prost::Message; + use url::Url; + + use super::*; + use crate::error::{EngineExecError, KernelError}; + use crate::handle::Handle; + use crate::plans::get_plan_executor; + use crate::plans::iter::{CBytesIterator, CEngineDataIterator, CFileMetaIterator}; + use crate::plans::result::CParquetFooter; + use crate::{ + allocate_kernel_bytes, kernel_bytes_slice, ExclusiveEngineData, ExclusiveRustString, + OptionalValue, + }; + + extern "C" fn noop_free(_state: NullableCvoid) {} + + /// Mock callback that pulls a pre-configured `CPlanResult` out of a `Mutex>` + /// stashed in `context` and writes it into the out pointer as an `EngineExecResult::Success`. + extern "C" fn mock_execute_op( + context: NullableCvoid, + _plan_proto: KernelBytesSlice, + out: *mut EngineExecResult, + ) { + let cell = unsafe { &*(context.unwrap().as_ptr() as *const Mutex>) }; + let plan = cell + .lock() + .unwrap() + .take() + .expect("mock_execute_op invoked more than once"); + unsafe { out.write(EngineExecResult::Success(plan)) }; + } + + /// Executes a dummy plan operation against a `PlanExecutor` whose callback returns the given + /// `expected_plan_result`, returning the raw result of `execute_op`. + fn try_execute_dummy_op(expected_plan_result: CPlanResult) -> DeltaResult { + let cell: Mutex> = Mutex::new(Some(expected_plan_result)); + let context = NonNull::new(&cell as *const Mutex> as *mut c_void); + let executor = unsafe { get_plan_executor(context, mock_execute_op) }; + let plan_executor: Arc = unsafe { executor.into_inner() }; + + // Construct a dummy op because we don't actually care about the plan bytes here. + let url = Url::parse("memory:///table/").unwrap(); + let op = Operation::IoOperation(delta_kernel::IoOperation::file_listing(url)); + + plan_executor.execute_op(op) + } + + /// Like [`try_execute_dummy_op`], but asserts the operation succeeds and returns the + /// resulting `PlanResult` for further validation. + fn execute_dummy_op(expected_plan_result: CPlanResult) -> PlanResult { + try_execute_dummy_op(expected_plan_result).expect("execute_op succeeds") + } + + #[test] + fn execute_op_unit_variant() { + execute_dummy_op(CPlanResult::Unit) + .into_unit() + .expect("Unit variant"); + } + + /// A callback that writes an `EngineExecResult::Failure` must surface as the matching kernel + /// error, preserving the engine-supplied message. + #[test] + fn execute_op_surfaces_engine_failure() { + extern "C" fn fail_execute_op( + _context: NullableCvoid, + _plan_proto: KernelBytesSlice, + out: *mut EngineExecResult, + ) { + // Mirror the engine downcalling `allocate_kernel_string` to build the message handle. + let message: Handle = Box::new("kaboom".to_string()).into(); + let err = EngineExecError { + etype: KernelError::UnsupportedError, + message, + }; + unsafe { out.write(EngineExecResult::Failure(err)) }; + } + + let executor = unsafe { get_plan_executor(None, fail_execute_op) }; + let plan_executor: Arc = unsafe { executor.into_inner() }; + + let url = Url::parse("memory:///table/").unwrap(); + let op = Operation::IoOperation(delta_kernel::IoOperation::file_listing(url)); + + let Err(err) = plan_executor.execute_op(op) else { + panic!("execute_op should surface the engine failure"); + }; + assert!( + matches!(err, Error::Unsupported(ref msg) if msg == "kaboom"), + "expected Error::Unsupported(\"kaboom\"), got {err:?}" + ); + } + + /// A callback that returns without writing the out pointer must surface as an internal error + #[test] + fn execute_op_surfaces_uninitialized_out() { + extern "C" fn noop_execute_op( + _context: NullableCvoid, + _plan_proto: KernelBytesSlice, + _out: *mut EngineExecResult, + ) { + } + + let executor = unsafe { get_plan_executor(None, noop_execute_op) }; + let plan_executor: Arc = unsafe { executor.into_inner() }; + + let url = Url::parse("memory:///table/").unwrap(); + let op = Operation::IoOperation(delta_kernel::IoOperation::file_listing(url)); + + let Err(err) = plan_executor.execute_op(op) else { + panic!("execute_op should surface an error when the engine does not write the result"); + }; + assert!( + err.to_string().contains( + "FFI engine returned from execute_op upcall without writing the plan result" + ), + "expected the engine-did-not-write message, got {err}" + ); + } + + #[test] + fn execute_op_data_variant() { + extern "C" fn empty_data_next( + _state: NullableCvoid, + out: *mut OptionalValue>>, + ) { + unsafe { out.write(OptionalValue::None) }; + } + + let iter = CEngineDataIterator { + state: None, + next: empty_data_next, + free: noop_free, + }; + let mut data_iter = execute_dummy_op(CPlanResult::Data(iter)) + .into_data() + .expect("Data variant"); + assert!(data_iter.next().is_none(), "empty data iterator"); + } + + #[test] + fn execute_op_file_meta_variant() { + extern "C" fn empty_file_meta_next( + _state: NullableCvoid, + out: *mut OptionalValue>, + ) { + unsafe { out.write(OptionalValue::None) }; + } + + let iter = CFileMetaIterator { + state: None, + next: empty_file_meta_next, + free: noop_free, + }; + let mut file_meta_iter = execute_dummy_op(CPlanResult::FileMeta(iter)) + .into_file_meta() + .expect("FileMeta variant"); + assert!(file_meta_iter.next().is_none(), "empty file meta iterator"); + } + + #[test] + fn execute_op_bytes_variant() { + extern "C" fn empty_bytes_next( + _state: NullableCvoid, + out: *mut OptionalValue>, + ) { + unsafe { out.write(OptionalValue::None) }; + } + + let iter = CBytesIterator { + state: None, + next: empty_bytes_next, + free: noop_free, + }; + let mut bytes_iter = execute_dummy_op(CPlanResult::Bytes(iter)) + .into_bytes() + .expect("Bytes variant"); + assert!(bytes_iter.next().is_none(), "empty bytes iterator"); + } + + #[test] + fn execute_op_parquet_footer_variant() { + let schema = + StructType::try_new(vec![StructField::nullable("id", KernelDataType::INTEGER)]) + .unwrap(); + let bytes = proto_schema::StructType::from(&schema).encode_to_vec(); + let schema_proto = unsafe { allocate_kernel_bytes(kernel_bytes_slice!(bytes)) }; + let footer = CParquetFooter { schema_proto }; + let footer = execute_dummy_op(CPlanResult::ParquetFooter(footer)) + .into_parquet_footer() + .expect("ParquetFooter variant"); + assert_eq!(footer.schema.fields().count(), 1); + let id_field = footer.schema.field("id").expect("id field"); + assert_eq!(id_field.data_type(), &KernelDataType::INTEGER); + assert!(id_field.is_nullable()); + } + + #[test] + fn execute_op_parquet_footer_rejects_invalid_schema_bytes() { + // Bytes that are not a valid proto-encoded `StructType` message. + let bytes = vec![0xffu8; 8]; + let schema_proto = unsafe { allocate_kernel_bytes(kernel_bytes_slice!(bytes)) }; + let footer = CParquetFooter { schema_proto }; + + let Err(err) = try_execute_dummy_op(CPlanResult::ParquetFooter(footer)) else { + panic!("invalid schema proto bytes should fail to decode"); + }; + assert!( + matches!(err, Error::GenericError { .. }), + "expected a proto decode error, got {err:?}" + ); + } + + #[test] + fn execute_op_passes_serialized_operation_bytes() { + extern "C" fn validate_op( + _context: NullableCvoid, + plan_proto: KernelBytesSlice, + out: *mut EngineExecResult, + ) { + let bytes = unsafe { std::slice::from_raw_parts(plan_proto.ptr, plan_proto.len) }; + let op = proto::Operation::decode(bytes).expect("decode Operation proto"); + let Some(proto::operation::Op::Io(io)) = op.op else { + panic!("expected an IoOperation"); + }; + let Some(proto::io_operation::Op::FileListing(file_listing)) = io.op else { + panic!("expected a FileListing"); + }; + assert_eq!(file_listing.url, "memory:///table/"); + unsafe { out.write(EngineExecResult::Success(CPlanResult::Unit)) }; + } + + let executor = unsafe { get_plan_executor(None, validate_op) }; + let plan_executor: Arc = unsafe { executor.into_inner() }; + + let url = Url::parse("memory:///table/").unwrap(); + let op = Operation::IoOperation(delta_kernel::IoOperation::file_listing(url)); + + plan_executor + .execute_op(op) + .expect("execute_op succeeds") + .into_unit() + .expect("Unit variant"); + } +} diff --git a/ffi/src/plans/iter.rs b/ffi/src/plans/iter.rs new file mode 100644 index 0000000000..0b0603c283 --- /dev/null +++ b/ffi/src/plans/iter.rs @@ -0,0 +1,764 @@ +//! This module contains types needed to implement Engine-owned iterators returned from plan +//! execution. +//! +//! Each `C*Iterator` type contains an opaque engine `state` pointer, a `next` function pointer that +//! yields one iterator item per call, and a `free` function pointer that releases the iterator +//! state. +//! +//! # `next()` protocol +//! +//! Each `next` callback writes its result as an `OptionalValue>` (mirroring +//! Rust's `Option>`) into a caller-provided out pointer. The out pointer is +//! pre-initialized to `Some(EngineExecResult::Uninit)`. +//! +//! This allows us to implement `DeltaResultIterator` semantics: +//! - The outer Option represents whether iteration is complete (`None` = done) +//! - The inner [`EngineExecResult`] represents the item (`Success`) or an engine-side error +//! (`Failure`) +//! +//! # Iterator Memory Management +//! There are 3 aspects to iterator memory management: +//! +//! 1. Iterator State - owned by the engine and freed by the kernel via the iterator's own `free` +//! function pointer. Each `Ffi*Iter` adapter invokes `free` exactly once when it is dropped. +//! +//! 2. Iterator Items - each data item (e.g EngineData, Bytes, FileMeta) yielded by the iterator has +//! its own separate lifetime and may have its own memory management rules. However we generally +//! use FFI_ArrowArray as a carrier for the item (which internally manages its own release +//! callback). +//! +//! 3. Errors - each [`EngineExecError`](crate::error::EngineExecError) carries a kernel-allocated +//! `ExclusiveRustString` message handle. Kernel takes ownership of the message and frees it when +//! converting the error into a kernel error (via `From for Error`). +//! +//! # Safety +//! The engine is responsible for ensuring that all `state`, `next`, and `free` pointers +//! are safe to send between threads. + +use bytes::Bytes; +use delta_kernel::arrow::array::ffi::{self as arrow_ffi, FFI_ArrowArray, FFI_ArrowSchema}; +use delta_kernel::arrow::array::{ + self as arrow_array, Array, BinaryArray, Int64Array, StringArray, StructArray, UInt64Array, +}; +use delta_kernel::arrow::datatypes::{DataType as ArrowDataType, Field as ArrowField, Fields}; +use delta_kernel::{DeltaResult, EngineData, Error}; +use url::Url; + +use crate::error::EngineExecResult; +use crate::handle::Handle; +use crate::{ExclusiveEngineData, NullableCvoid, OptionalValue}; + +// ============================================================================ +// repr(C) iterator types +// ============================================================================ + +/// Function pointer an engine iterator uses to yield its next item into `out`. +/// +/// See "`next()` protocol" in the module docs. +pub type CIterNextFn = + extern "C" fn(state: NullableCvoid, out: *mut OptionalValue>); + +/// Function pointer that releases an engine iterator's `state`. Invoked exactly once by kernel once +/// iteration is complete. +pub type CIterFreeFn = extern "C" fn(state: NullableCvoid); + +/// An engine-owned iterator of [`EngineData`] batches. +/// +/// See module level docs for memory management and safety requirements. +#[repr(C)] +pub struct CEngineDataIterator { + pub state: NullableCvoid, + pub next: CIterNextFn>, + pub free: CIterFreeFn, +} + +/// An engine-owned iterator of byte buffers. +/// +/// See module level docs for memory management and safety requirements. +/// +/// Each byte array is transferred across the FFI boundary as an [`FFI_ArrowArray`]. +/// Invocation of `next` MUST yield an Arrow array that is a `BinaryArray` containing a single +/// row of non-null bytes. Kernel assumes the schema is [`ArrowDataType::Binary`] (the schema is NOT +/// passed along through FFI). +/// +/// FFI_ArrowArray serves as a convenient container for receiving bytes because it internally +/// manages its own memory release callback. +/// +/// TODO: ArrowDataType::Binary uses i32 offsets, so each byte array is limited to 2 GiB. If this is +/// a problem, we need to support ArrowDataType::LargeBinary instead. +#[repr(C)] +pub struct CBytesIterator { + pub state: NullableCvoid, + pub next: CIterNextFn, + pub free: CIterFreeFn, +} + +/// An engine-owned iterator of [`FileMeta`](delta_kernel::FileMeta) batches. +/// +/// See module level docs for memory management and safety requirements. +/// +/// Similar to `CBytesIterator`, CFileMetaIterator uses `FFI_ArrowArray` as a convenient container +/// for passing FileMeta entries across the FFI boundary. Each `next` invocation MUST yield a batch +/// of one or more FileMeta rows as a `StructArray` whose fields are +/// `{location: Utf8, last_modified: Int64, size: UInt64}`, all non-null. Kernel assumes this fixed +/// schema when converting into FileMeta (the schema is NOT passed along through FFI). +/// Empty or otherwise invalid batches surface as errors and permanently terminate iteration +/// (kernel will not call `next` again). +#[repr(C)] +pub struct CFileMetaIterator { + pub state: NullableCvoid, + pub next: CIterNextFn, + pub free: CIterFreeFn, +} + +// ============================================================================ +// Rust Iterator adapters +// ============================================================================ + +/// Helper function for invokeing an engine iterator's `next` callback and normalizing its +/// out-pointer result into the next raw item. +fn next_item(next: CIterNextFn, state: NullableCvoid) -> Option> { + let mut out = OptionalValue::Some(EngineExecResult::Uninit); + next(state, &mut out); + match out { + OptionalValue::None => None, + OptionalValue::Some(EngineExecResult::Success(item)) => Some(Ok(item)), + OptionalValue::Some(EngineExecResult::Failure(err)) => Some(Err(err.into())), + OptionalValue::Some(EngineExecResult::Uninit) => Some(Err(Error::internal_error( + "FFI engine iterator returned from next upcall without writing an item", + ))), + } +} + +/// An abstraction for invoking an engine iterator's `free` callback exactly once, when dropped. +/// +/// Embedding this in every `Ffi*Iter` adapter provides a single shared mechanism for ensuring the +/// iterator is dropped correctly. +pub(crate) struct IterCleanup { + state: NullableCvoid, + free: CIterFreeFn, +} + +impl IterCleanup { + fn new(state: NullableCvoid, free: CIterFreeFn) -> Self { + Self { state, free } + } + + /// The opaque engine state pointer, forwarded to each `next` call. + fn state(&self) -> NullableCvoid { + self.state + } +} + +impl Drop for IterCleanup { + fn drop(&mut self) { + (self.free)(self.state); + } +} + +/// Provides the Rust `Iterator` trait for [`EngineData`] batches on top of a +/// [`CEngineDataIterator`]. +pub(crate) struct FfiEngineDataIter { + next: CIterNextFn>, + cleanup: IterCleanup, +} + +impl FfiEngineDataIter { + pub(crate) fn new(iter: CEngineDataIterator) -> Self { + let CEngineDataIterator { state, next, free } = iter; + Self { + next, + cleanup: IterCleanup::new(state, free), + } + } +} + +impl Iterator for FfiEngineDataIter { + type Item = DeltaResult>; + + fn next(&mut self) -> Option { + // into_inner transfers ownership of the EngineData to kernel, and kernel will free it when + // the yielded EngineData is dropped. + next_item(self.next, self.cleanup.state()) + .map(|item| item.map(|handle| unsafe { handle.into_inner() })) + } +} + +// # Safety +// The engine must ensure that all raw pointers sent are Send-safe (see module level docs) +unsafe impl Send for FfiEngineDataIter {} + +/// Provides the Rust `Iterator` trait for [`Bytes`] buffers on top of a [`CBytesIterator`]. +pub(crate) struct FfiBytesIter { + next: CIterNextFn, + cleanup: IterCleanup, +} + +impl FfiBytesIter { + pub(crate) fn new(iter: CBytesIterator) -> Self { + let CBytesIterator { state, next, free } = iter; + Self { + next, + cleanup: IterCleanup::new(state, free), + } + } + + fn arrow_array_to_bytes(array: FFI_ArrowArray) -> DeltaResult { + let schema = FFI_ArrowSchema::try_from(&ArrowDataType::Binary)?; + let array_data = unsafe { arrow_ffi::from_ffi(array, &schema) }?; + let array = arrow_array::make_array(array_data); + + let Some(binary) = array.as_any().downcast_ref::() else { + return Err(Error::generic(format!( + "CBytesIterator must yield BinaryArray, got {:?}", + array.data_type() + ))); + }; + if binary.len() != 1 { + return Err(Error::generic(format!( + "CBytesIterator array must contain exactly one row, got {}", + binary.len() + ))); + } + if binary.is_null(0) { + return Err(Error::generic("CBytesIterator array row must not be null")); + } + + // TODO: this copies the payload bytes, but could be made zero-copy by + // using `Bytes::from_owner` on a custom struct that implements `AsRef<[u8]>` + Ok(Bytes::copy_from_slice(binary.value(0))) + } +} + +impl Iterator for FfiBytesIter { + type Item = DeltaResult; + + fn next(&mut self) -> Option { + next_item(self.next, self.cleanup.state()) + .map(|item| item.and_then(Self::arrow_array_to_bytes)) + } +} + +// # Safety +// The engine must ensure that all raw pointers sent are Send-safe (see module level docs) +unsafe impl Send for FfiBytesIter {} + +/// Provides the Rust `Iterator` trait for [`FileMeta`] entries on top of a +/// [`CFileMetaIterator`]. +/// +/// Buffers each engine-yielded batch and surfaces one row per `Iterator::next` call. Permanantly +/// terminates if the engine returns None or a batch fails to decode. Errors surfaced from +/// the engine `next` callback are considered transient and do NOT terminate iteration. +pub(crate) struct FfiFileMetaIter { + next: CIterNextFn, + cleanup: IterCleanup, + // Decoded rows from the most recent engine batch + buffer: std::vec::IntoIter, + terminated: bool, +} + +impl FfiFileMetaIter { + pub(crate) fn new(iter: CFileMetaIterator) -> Self { + let CFileMetaIterator { state, next, free } = iter; + Self { + next, + cleanup: IterCleanup::new(state, free), + buffer: Vec::new().into_iter(), + terminated: false, + } + } + + /// The fixed Arrow type expected by [`CFileMetaIterator`]: a non-null struct of + /// `{location: Utf8, last_modified: Int64, size: UInt64}`, all non-null. + fn arrow_schema() -> DeltaResult { + let schema = ArrowDataType::Struct(Fields::from(vec![ + ArrowField::new("location", ArrowDataType::Utf8, false), + ArrowField::new("last_modified", ArrowDataType::Int64, false), + ArrowField::new("size", ArrowDataType::UInt64, false), + ])); + Ok(FFI_ArrowSchema::try_from(&schema)?) + } + + /// Decodes a single engine batch into a `Vec`. + fn arrow_array_to_file_metas( + array: FFI_ArrowArray, + ) -> DeltaResult> { + let schema = Self::arrow_schema()?; + let array_data = unsafe { arrow_ffi::from_ffi(array, &schema) }?; + let array = arrow_array::make_array(array_data); + + let Some(struct_array) = array.as_any().downcast_ref::() else { + return Err(Error::generic(format!( + "CFileMetaIterator must yield StructArray, got {:?}", + array.data_type() + ))); + }; + if struct_array.is_empty() { + return Err(Error::generic( + "CFileMetaIterator batch must contain at least one row", + )); + } + + // The fixed schema guarantees three columns in this order, all non-null. `from_ffi` + // above already validated the per-column data types match what we synthesized, so the + // downcasts cannot fail; the `ok_or_else` arms exist only to keep the converter + // panic-free. + let location_col = struct_array + .column(0) + .as_any() + .downcast_ref::() + .ok_or_else(|| Error::generic("CFileMetaIterator: location column is not Utf8"))?; + let last_modified_col = struct_array + .column(1) + .as_any() + .downcast_ref::() + .ok_or_else(|| { + Error::generic("CFileMetaIterator: last_modified column is not Int64") + })?; + let size_col = struct_array + .column(2) + .as_any() + .downcast_ref::() + .ok_or_else(|| Error::generic("CFileMetaIterator: size column is not UInt64"))?; + + // The fixed schema declares every column non-null; reject any batch that violates that + // contract up front so the per-row decode below can safely call `value(i)`. + if location_col.null_count() != 0 + || last_modified_col.null_count() != 0 + || size_col.null_count() != 0 + { + return Err(Error::generic( + "CFileMetaIterator batch must not contain null fields", + )); + } + + (0..struct_array.len()) + .map(|i| { + Ok(delta_kernel::FileMeta { + location: Url::parse(location_col.value(i))?, + last_modified: last_modified_col.value(i), + size: size_col.value(i), + }) + }) + .collect() + } +} + +impl Iterator for FfiFileMetaIter { + type Item = DeltaResult; + + fn next(&mut self) -> Option { + loop { + if self.terminated { + return None; + } + + if let Some(meta) = self.buffer.next() { + return Some(Ok(meta)); + } + + // Buffer drained and iteration is still live; pull the next batch from the engine. + let array = match next_item(self.next, self.cleanup.state()) { + None => { + self.terminated = true; + return None; + } + // Engine-reported errors (and a missing result) are transient and do not terminate + // iteration. + Some(Err(err)) => return Some(Err(err)), + Some(Ok(array)) => array, + }; + match Self::arrow_array_to_file_metas(array) { + Ok(batch) => self.buffer = batch.into_iter(), + Err(e) => { + // Decoder errors (including empty batches) indicate corrupt data and are fatal. + self.terminated = true; + return Some(Err(e)); + } + } + } + } +} + +// # Safety +// The engine must ensure that all raw pointers sent are Send-safe (see module level docs) +unsafe impl Send for FfiFileMetaIter {} + +#[cfg(test)] +mod tests { + use std::collections::VecDeque; + use std::ffi::c_void; + use std::ptr::NonNull; + use std::sync::atomic::{AtomicBool, Ordering}; + use std::sync::{Arc, Mutex}; + + use delta_kernel::arrow::array::{ + ffi as arrow_ffi, ArrayRef, BinaryArray, Int32Array, Int64Array, RecordBatch, StringArray, + StructArray, UInt64Array, + }; + use delta_kernel::arrow::datatypes::{ + DataType as ArrowDataType, Field as ArrowField, Schema as ArrowSchema, + }; + use delta_kernel::engine::arrow_data::ArrowEngineData; + + use super::*; + use crate::error::{EngineExecError, KernelError}; + use crate::ExclusiveRustString; + + // === Shared Test Helpers === + + struct MockIter { + items: Mutex>>>, + cleanup_called: Arc, + } + + impl Drop for MockIter { + fn drop(&mut self) { + let previously_set = self.cleanup_called.swap(true, Ordering::SeqCst); + assert!(!previously_set, "MockIter dropped more than once"); + } + } + + // Makes a mock iterator with the given items and returns the iterator + a cleanup flag for + // verifying that the iterator was dropped properly. + // + // Can be dropped using `free_mock_iter`. + fn make_mock_iter( + items: Vec>>, + ) -> (Box>, Arc) { + let cleanup_called = Arc::new(AtomicBool::new(false)); + let iter = Box::new(MockIter { + items: Mutex::new(VecDeque::from(items)), + cleanup_called: Arc::clone(&cleanup_called), + }); + (iter, cleanup_called) + } + + extern "C" fn free_mock_iter(state: NullableCvoid) { + let _ = unsafe { Box::from_raw(state.unwrap().as_ptr() as *mut MockIter) }; + } + + // Builds an engine execution error whose message is a kernel-allocated `ExclusiveRustString` + // handle (mirroring the engine downcalling `allocate_kernel_string`). + fn make_exec_error(etype: KernelError, message: &str) -> EngineExecError { + let message: Handle = Box::new(message.to_string()).into(); + EngineExecError { etype, message } + } + + // === FfiEngineDataIter Test === + + /// Helper for building each yielded EngineData batch + fn make_data_handle(rows: usize) -> Handle { + let schema = Arc::new(ArrowSchema::new(vec![ArrowField::new( + "x", + ArrowDataType::Int32, + true, + )])); + let values: Vec = (0..rows as i32).collect(); + let batch = RecordBatch::try_new(schema, vec![Arc::new(Int32Array::from(values))]).unwrap(); + let data: Box = Box::new(ArrowEngineData::new(batch)); + data.into() + } + + #[test] + fn ffi_engine_data_iter_drains_and_runs_cleanup() { + let (mock_iter, cleanup_called) = make_mock_iter::>(vec![ + OptionalValue::Some(EngineExecResult::Success(make_data_handle(3))), + OptionalValue::Some(EngineExecResult::Failure(make_exec_error( + KernelError::GenericError, + "boom", + ))), + OptionalValue::Some(EngineExecResult::Success(make_data_handle(5))), + ]); + let state = NonNull::new(Box::into_raw(mock_iter) as *mut c_void); + + extern "C" fn next( + state: NullableCvoid, + out: *mut OptionalValue>>, + ) { + let mock_iter = unsafe { + &*(state.unwrap().as_ptr() as *const MockIter>) + }; + let item = mock_iter + .items + .lock() + .unwrap() + .pop_front() + .unwrap_or(OptionalValue::None); + unsafe { out.write(item) }; + } + + let mut iter = FfiEngineDataIter::new(CEngineDataIterator { + state, + next, + free: free_mock_iter::>, + }); + + let first = iter.next().expect("first item").expect("first ok"); + assert_eq!(first.len(), 3); + + let Some(Err(err)) = iter.next() else { + panic!("second item should be Err"); + }; + assert!( + err.to_string().contains("boom"), + "engine error message must propagate, got {err}" + ); + + let third = iter.next().expect("third item").expect("third ok"); + assert_eq!(third.len(), 5); + + assert!(iter.next().is_none(), "iterator should be exhausted"); + assert!( + !cleanup_called.load(Ordering::SeqCst), + "cleanup must not run before the iterator is dropped" + ); + + drop(iter); + assert!( + cleanup_called.load(Ordering::SeqCst), + "cleanup must run when the iterator is dropped" + ); + } + + // === FfiBytesIter Test === + + // Helper for building each yielded Arrow bytes array + fn make_binary_ffi_array(payload: &[u8]) -> FFI_ArrowArray { + let array = BinaryArray::from(vec![payload]); + let (ffi_array, _ffi_schema) = arrow_ffi::to_ffi(&array.into_data()).unwrap(); + ffi_array + } + + #[test] + fn ffi_bytes_iter_drains_and_runs_cleanup() { + let (mock_iter, cleanup_called) = make_mock_iter::(vec![ + OptionalValue::Some(EngineExecResult::Success(make_binary_ffi_array(b"hello"))), + OptionalValue::Some(EngineExecResult::Failure(make_exec_error( + KernelError::GenericError, + "boom", + ))), + OptionalValue::Some(EngineExecResult::Success(make_binary_ffi_array(b"world!"))), + ]); + let state = NonNull::new(Box::into_raw(mock_iter) as *mut c_void); + + extern "C" fn next( + state: NullableCvoid, + out: *mut OptionalValue>, + ) { + let mock_iter = + unsafe { &*(state.unwrap().as_ptr() as *const MockIter) }; + let item = mock_iter + .items + .lock() + .unwrap() + .pop_front() + .unwrap_or(OptionalValue::None); + unsafe { out.write(item) }; + } + + let mut iter = FfiBytesIter::new(CBytesIterator { + state, + next, + free: free_mock_iter::, + }); + + let first = iter.next().expect("first item").expect("first ok"); + assert_eq!(first.as_ref(), b"hello"); + + let Some(Err(err)) = iter.next() else { + panic!("second item should be Err"); + }; + assert!( + err.to_string().contains("boom"), + "engine error message must propagate, got {err}" + ); + + let third = iter.next().expect("third item").expect("third ok"); + assert_eq!(third.as_ref(), b"world!"); + + assert!(iter.next().is_none(), "iterator should be exhausted"); + assert!(!cleanup_called.load(Ordering::SeqCst)); + + drop(iter); + assert!(cleanup_called.load(Ordering::SeqCst)); + } + + // === FfiFileMetaIter Test === + + /// Helper for building a yielded FileMeta batch. `rows` may be empty to construct a malformed + /// batch for negative tests. + fn make_file_meta_ffi_array(rows: &[(&str, i64, u64)]) -> FFI_ArrowArray { + let locations: Vec<&str> = rows.iter().map(|(l, _, _)| *l).collect(); + let last_modifieds: Vec = rows.iter().map(|(_, t, _)| *t).collect(); + let sizes: Vec = rows.iter().map(|(_, _, s)| *s).collect(); + let struct_array = StructArray::from(vec![ + ( + Arc::new(ArrowField::new("location", ArrowDataType::Utf8, false)), + Arc::new(StringArray::from(locations)) as ArrayRef, + ), + ( + Arc::new(ArrowField::new( + "last_modified", + ArrowDataType::Int64, + false, + )), + Arc::new(Int64Array::from(last_modifieds)) as ArrayRef, + ), + ( + Arc::new(ArrowField::new("size", ArrowDataType::UInt64, false)), + Arc::new(UInt64Array::from(sizes)) as ArrayRef, + ), + ]); + let (ffi_array, _ffi_schema) = arrow_ffi::to_ffi(&struct_array.into_data()).unwrap(); + ffi_array + } + + extern "C" fn file_meta_next( + state: NullableCvoid, + out: *mut OptionalValue>, + ) { + let mock_iter = unsafe { &*(state.unwrap().as_ptr() as *const MockIter) }; + let item = mock_iter + .items + .lock() + .unwrap() + .pop_front() + .unwrap_or(OptionalValue::None); + unsafe { out.write(item) }; + } + + /// Exercises batched yields with a multi-row batch, a transient engine error (which does not + /// terminate iteration), and a single-row batch. Verifies the Rust-side adapter buffers each + /// batch and surfaces one row per `Iterator::next` call. + #[test] + fn ffi_file_meta_iter_drains_and_runs_cleanup() { + let (mock_iter, cleanup_called) = make_mock_iter::(vec![ + OptionalValue::Some(EngineExecResult::Success(make_file_meta_ffi_array(&[ + ("file:///a.parquet", 1, 100), + ("file:///b.parquet", 2, 200), + ]))), + OptionalValue::Some(EngineExecResult::Failure(make_exec_error( + KernelError::GenericError, + "boom", + ))), + OptionalValue::Some(EngineExecResult::Success(make_file_meta_ffi_array(&[( + "file:///c.parquet", + 3, + 300, + )]))), + ]); + let state = NonNull::new(Box::into_raw(mock_iter) as *mut c_void); + + let mut iter = FfiFileMetaIter::new(CFileMetaIterator { + state, + next: file_meta_next, + free: free_mock_iter::, + }); + + // First batch contributes two rows. + let first = iter.next().expect("first item").expect("first ok"); + assert_eq!(first.location.as_str(), "file:///a.parquet"); + assert_eq!(first.last_modified, 1); + assert_eq!(first.size, 100); + + let second = iter.next().expect("second item").expect("second ok"); + assert_eq!(second.location.as_str(), "file:///b.parquet"); + assert_eq!(second.last_modified, 2); + assert_eq!(second.size, 200); + + // After draining the first batch the adapter pulls the transient engine error. + let Some(Err(err)) = iter.next() else { + panic!("third item should be Err"); + }; + assert!( + err.to_string().contains("boom"), + "engine error message must propagate, got {err}" + ); + + // Transient engine errors do not terminate iteration; the fourth batch is still returned. + let fourth = iter.next().expect("fourth item").expect("fourth ok"); + assert_eq!(fourth.location.as_str(), "file:///c.parquet"); + assert_eq!(fourth.last_modified, 3); + assert_eq!(fourth.size, 300); + + assert!(iter.next().is_none(), "iterator should be exhausted"); + assert!(!cleanup_called.load(Ordering::SeqCst)); + + drop(iter); + assert!(cleanup_called.load(Ordering::SeqCst)); + } + + /// Verifies that an empty batch surfaces as an error and permanently terminates iteration + #[test] + fn ffi_file_meta_iter_rejects_empty_batch_and_terminates() { + let (mock_iter, cleanup_called) = make_mock_iter::(vec![ + OptionalValue::Some(EngineExecResult::Success(make_file_meta_ffi_array(&[]))), + OptionalValue::Some(EngineExecResult::Success(make_file_meta_ffi_array(&[( + "file:///trap.parquet", + 99, + 999, + )]))), + ]); + let state = NonNull::new(Box::into_raw(mock_iter) as *mut c_void); + + // Helper fn for checking the queue length from `state` pointer + let queue_len = || { + let mock_iter = + unsafe { &*(state.unwrap().as_ptr() as *const MockIter) }; + mock_iter.items.lock().unwrap().len() + }; + + let mut iter = FfiFileMetaIter::new(CFileMetaIterator { + state, + next: file_meta_next, + free: free_mock_iter::, + }); + + let first = iter.next().expect("first item"); + assert!(first.is_err(), "empty batch must surface as Err"); + + // The final entry is still queued. + assert_eq!(queue_len(), 1, "final entry must remain unconsumed"); + + // Further polls return `None` without re-invoking the engine. + assert!(iter.next().is_none(), "iterator should be terminated"); + assert!(iter.next().is_none(), "iterator should stay terminated"); + assert_eq!( + queue_len(), + 1, + "final entry must still be unconsumed after extra polls" + ); + + assert!(!cleanup_called.load(Ordering::SeqCst)); + drop(iter); + assert!(cleanup_called.load(Ordering::SeqCst)); + } + + /// A `next` callback that returns without writing the out pointer must surface as an internal + /// error (from the pre-initialized `Uninit` sentinel) rather than reading uninitialized memory. + #[test] + fn ffi_iter_uninitialized_out_surfaces_internal_error() { + extern "C" fn noop_next( + _state: NullableCvoid, + _out: *mut OptionalValue>>, + ) { + } + extern "C" fn noop_free(_state: NullableCvoid) {} + + let mut iter = FfiEngineDataIter::new(CEngineDataIterator { + state: None, + next: noop_next, + free: noop_free, + }); + + let Some(Err(err)) = iter.next() else { + panic!("expected an error when the engine does not initialize the out pointer"); + }; + assert!( + err.to_string() + .contains("FFI engine iterator returned from next upcall without writing an item"), + "expected the engine-did-not-write message, got {err}" + ); + } +} diff --git a/ffi/src/plans/mod.rs b/ffi/src/plans/mod.rs new file mode 100644 index 0000000000..377958c7bc --- /dev/null +++ b/ffi/src/plans/mod.rs @@ -0,0 +1,184 @@ +//! This module contains all types and functions needed to support declarative plan execution +//! over the FFI boundary. + +use std::sync::Arc; + +use delta_kernel::engine::plans::PlanBasedEngine; +use delta_kernel::{Engine, PlanExecutor}; + +use crate::handle::Handle; +use crate::plans::executor::{CExecuteOpFn, FfiPlanExecutor, SharedPlanExecutor}; +use crate::{engine_to_handle, AllocateErrorFn, NullableCvoid, OptionalValue, SharedExternEngine}; + +pub mod executor; +pub mod iter; +pub mod result; + +/// Build a [`PlanExecutor`] backed by an engine-provided C callback. +/// +/// # Safety +/// The `context` pointer MUST be thread-safe (Send + Sync) and MUST remain valid for as long as the +/// executor is used. It is valid to pass NULL as the context. +#[no_mangle] +pub unsafe extern "C" fn get_plan_executor( + context: NullableCvoid, + callback: CExecuteOpFn, +) -> Handle { + let executor: Arc = Arc::new(FfiPlanExecutor::new(context, callback)); + executor.into() +} + +/// Free a plan executor obtained from [`get_plan_executor`]. +/// +/// Normally the handle is consumed by [`get_plan_based_engine`] and need not be explicitly freed by +/// the caller. Use this only when discarding the executor without wrapping it in PlanBasedEngine. +/// +/// # Safety +/// +/// Caller must pass a valid handle previously obtained from [`get_plan_executor`] and must not use +/// it again afterwards. +#[no_mangle] +pub unsafe extern "C" fn free_plan_executor(executor: Handle) { + executor.drop_handle(); +} + +/// Construct a [`PlanBasedEngine`] backed by `plan_executor`, optionally delegating operations not +/// yet implemented on the plan-execution path to `fallback_engine`. +/// +/// This method consumes the [`SharedPlanExecutor`] handle. It does NOT consume `fallback_engine`: +/// the caller retains ownership of the fallback engine handle and must free it separately via +/// `free_engine`. The returned plan-based engine clones the handlers it needs from the fallback, +/// so it remains valid independently of when the caller frees its fallback handle. When +/// `fallback_engine` is [`OptionalValue::None`], unimplemented operations return an unsupported +/// error. +/// +/// # Safety +/// +/// Caller must pass a valid [`SharedPlanExecutor`] handle obtained from [`get_plan_executor`] and a +/// valid [`AllocateErrorFn`]. When `fallback_engine` is [`OptionalValue::Some`], it must contain a +/// valid [`SharedExternEngine`] handle. +#[no_mangle] +pub unsafe extern "C" fn get_plan_based_engine( + plan_executor: Handle, + fallback_engine: OptionalValue>, + allocate_error: AllocateErrorFn, +) -> Handle { + let executor: Arc = unsafe { plan_executor.into_inner() }; + let fallback = match fallback_engine { + OptionalValue::Some(engine) => Some(unsafe { engine.as_ref() }.engine()), + OptionalValue::None => None, + }; + let engine: Arc = Arc::new(PlanBasedEngine::new(fallback, executor)); + engine_to_handle(engine, allocate_error) +} + +#[cfg(test)] +mod tests { + use delta_kernel::arrow::array::{Array, RecordBatch, StringArray}; + use delta_kernel::engine::arrow_data::ArrowEngineData; + use delta_kernel::engine_data::FilteredEngineData; + use delta_kernel::object_store::memory::InMemory; + use delta_kernel_default_engine::DefaultEngineBuilder; + use url::Url; + + use super::*; + use crate::error::EngineExecResult; + use crate::ffi_test_utils::allocate_err; + use crate::free_engine; + use crate::plans::result::CPlanResult; + + extern "C" fn unreachable_callback( + _context: NullableCvoid, + _plan_proto: crate::KernelBytesSlice, + _out: *mut EngineExecResult, + ) { + unreachable!("callback should not run -- this test only constructs the engine"); + } + + /// Assert that the given engine handle wraps a `PlanBasedEngine` (not e.g. a DefaultEngine). + fn assert_is_plan_based_engine(engine_handle: &Handle) { + let extern_engine = unsafe { engine_handle.as_ref() }; + let engine = extern_engine.engine(); + assert!( + engine.any_ref().downcast_ref::().is_some(), + "engine handle must wrap a PlanBasedEngine", + ); + } + + #[test] + fn plan_based_engine_works_after_fallback_handle_is_freed() { + let executor = unsafe { get_plan_executor(None, unreachable_callback) }; + + // The caller retains ownership of the fallback engine handle. `shallow_copy` mimics C + // passing the pointer by value, so we can still free our own handle afterward. + let fallback = DefaultEngineBuilder::new(Arc::new(InMemory::new())).build(); + let fallback_handle = engine_to_handle(Arc::new(fallback), allocate_err); + + let engine_handle = unsafe { + get_plan_based_engine( + executor, + OptionalValue::Some(fallback_handle.shallow_copy()), + allocate_err, + ) + }; + + assert_is_plan_based_engine(&engine_handle); + + unsafe { free_engine(fallback_handle) }; + + let batch = RecordBatch::try_from_iter(vec![( + "value", + Arc::new(StringArray::from(vec!["one"])) as Arc, + )]) + .unwrap(); + let data = Box::new(ArrowEngineData::new(batch)); + let location = Url::parse("memory:///table/_delta_log/00000000000000000001.json").unwrap(); + unsafe { engine_handle.as_ref() } + .engine() + .json_handler() + .write_json_file( + &location, + Box::new(std::iter::once(Ok( + FilteredEngineData::with_all_rows_selected(data), + ))), + false, + ) + .expect("plan-based engine retains its fallback json handler"); + + unsafe { free_engine(engine_handle) }; + } + + #[test] + fn get_plan_based_engine_without_fallback_returns_plan_based_engine() { + let executor = unsafe { get_plan_executor(None, unreachable_callback) }; + + let engine_handle = + unsafe { get_plan_based_engine(executor, OptionalValue::None, allocate_err) }; + + assert_is_plan_based_engine(&engine_handle); + + unsafe { free_engine(engine_handle) }; + } + + /// Without a fallback, an operation the plan-execution path does not implement reports that + /// rather than delegating. + #[test] + fn without_fallback_unimplemented_operations_are_unsupported() { + let executor = unsafe { get_plan_executor(None, unreachable_callback) }; + let engine_handle = + unsafe { get_plan_based_engine(executor, OptionalValue::None, allocate_err) }; + + let engine = unsafe { engine_handle.as_ref() }.engine(); + let location = Url::parse("memory:///table/_delta_log/00000000000000000001.json").unwrap(); + let err = engine + .json_handler() + .write_json_file(&location, Box::new(std::iter::empty()), false) + .expect_err("write_json_file has no plan-execution path and no fallback"); + assert!( + matches!(err, delta_kernel::Error::Unsupported(_)), + "expected an unsupported error, got: {err:?}", + ); + + unsafe { free_engine(engine_handle) }; + } +} diff --git a/ffi/src/plans/result.rs b/ffi/src/plans/result.rs new file mode 100644 index 0000000000..eabfd50c16 --- /dev/null +++ b/ffi/src/plans/result.rs @@ -0,0 +1,30 @@ +//! FFI types for mimicking the kernel's `PlanResult` enum. + +use super::iter::{CBytesIterator, CEngineDataIterator, CFileMetaIterator}; +use crate::handle::Handle; +use crate::ExclusiveRustBytes; + +/// C-compatible equivalent of the kernel's `ParquetFooter` struct. +/// +/// The schema is delivered as a proto-serialized `delta.kernel.schema.StructType` message. The +/// engine should call [`allocate_kernel_bytes`](crate::allocate_kernel_bytes) to copy those bytes +/// into a kernel-owned [`ExclusiveRustBytes`] handle that can then be embedded into this struct. +#[repr(C)] +pub struct CParquetFooter { + pub schema_proto: Handle, +} + +/// C-compatible equivalent of the kernel's `PlanResult` enum. +/// +/// We instruct cbindgen to prefix enum variants with enum name (e.g. `CPlanResult_Unit`) +/// so they don't collide with other identifiers (e.g. with the `FileMeta` struct) +/// +/// cbindgen:prefix-with-name=true +#[repr(C)] +pub enum CPlanResult { + Unit, + Data(CEngineDataIterator), + FileMeta(CFileMetaIterator), + Bytes(CBytesIterator), + ParquetFooter(CParquetFooter), +} diff --git a/ffi/src/rest_engine.rs b/ffi/src/rest_engine.rs new file mode 100644 index 0000000000..7b585ccf0a --- /dev/null +++ b/ffi/src/rest_engine.rs @@ -0,0 +1,720 @@ +//! REST [`RestObjectStore`] wiring for [`EngineBuilder`](crate::EngineBuilder). +//! +//! Call [`set_builder_rest_object_store`](crate::set_builder_rest_object_store) with a +//! [`CRestEndpointConfig`] to select the REST backend. The builder `url` must be the REST service +//! base URL, not a Delta table path. +//! +//! TLS, retries, and static auth use [`set_builder_option`](crate::set_builder_option) with the +//! `REST_BUILDER_OPTION_*` keys below. Pass a [`CAuthHeaderCallback`] when headers expire; with +//! `callback = NULL`, set static headers via `header.` options instead. See +//! [`CAuthHeaderCallback`] for the callback contract. + +use std::collections::HashMap; +use std::mem::MaybeUninit; +use std::ptr; +use std::sync::Arc; +use std::time::Duration; + +use delta_kernel::object_store::{Error as ObjectStoreError, ObjectStore}; +use delta_kernel::{DeltaResult, Error}; +use delta_kernel_default_engine::rest_store::{ + build_rest_client, headers_from_pairs, AuthHeaderProvider, HeaderMap, RefreshingHeaderProvider, + RestClientOptions, RestEndpointConfig, RestObjectStore, StaticHeaderProvider, +}; +use url::Url; + +use crate::error::AllocateErrorFn; +use crate::handle::Handle; +use crate::{ExclusiveRustString, KernelStringSlice, NullableCvoid, TryFromStringSlice}; + +/// Max `(name, value)` pairs in a [`CAuthHeaders`] struct. +pub const AUTH_MAX_NUM_HEADERS: usize = 8; + +// === `set_builder_option` keys for REST engines === +// +// Each `REST_BUILDER_OPTION_*` is the Rust `&str` key. The matching `*_KEY` static is a +// null-terminated byte array exported to C via cbindgen (`extern const uint8_t ...`). + +/// Prefix for static auth header options: `header.` (append the HTTP header name). +/// +/// Example: `header.Authorization` with value `Bearer `. Ignored when a +/// [`CAuthHeaderCallback`] is passed to +/// [`set_builder_rest_object_store`](crate::set_builder_rest_object_store). +pub const REST_BUILDER_OPTION_HEADER_PREFIX: &str = "header."; +#[no_mangle] +pub static REST_BUILDER_OPTION_HEADER_PREFIX_KEY: [u8; 8] = *b"header.\0"; + +/// PEM client-certificate path. mTLS is enabled only when cert, key, and CA paths are all set. +pub const REST_BUILDER_OPTION_TLS_CERT_PATH: &str = "tls.cert_path"; +#[no_mangle] +pub static REST_BUILDER_OPTION_TLS_CERT_PATH_KEY: [u8; 14] = *b"tls.cert_path\0"; + +/// PEM client-private-key path (required with [`REST_BUILDER_OPTION_TLS_CERT_PATH`] and +/// [`REST_BUILDER_OPTION_TLS_CA_PATH`] for mTLS). +pub const REST_BUILDER_OPTION_TLS_KEY_PATH: &str = "tls.key_path"; +#[no_mangle] +pub static REST_BUILDER_OPTION_TLS_KEY_PATH_KEY: [u8; 13] = *b"tls.key_path\0"; + +/// PEM CA-bundle path used to verify the server (required with cert and key for mTLS). +pub const REST_BUILDER_OPTION_TLS_CA_PATH: &str = "tls.ca_path"; +#[no_mangle] +pub static REST_BUILDER_OPTION_TLS_CA_PATH_KEY: [u8; 12] = *b"tls.ca_path\0"; + +/// Comma-separated DNS overrides as `host=ip:port` entries (bypasses system resolver; SNI +/// unchanged). +pub const REST_BUILDER_OPTION_TLS_DNS_OVERRIDE: &str = "tls.dns_override"; +#[no_mangle] +pub static REST_BUILDER_OPTION_TLS_DNS_OVERRIDE_KEY: [u8; 17] = *b"tls.dns_override\0"; + +/// HTTP request timeout in seconds; unset uses the reqwest default. +pub const REST_BUILDER_OPTION_TLS_TIMEOUT_SECS: &str = "tls.timeout_secs"; +#[no_mangle] +pub static REST_BUILDER_OPTION_TLS_TIMEOUT_SECS_KEY: [u8; 17] = *b"tls.timeout_secs\0"; + +/// Maximum automatic retries for transient failures (5xx, connect/timeout) on idempotent REST +/// requests; defaults to `0` when unset. Ambiguous `Create` puts use +/// [`REST_BUILDER_OPTION_PUT_VERIFY_ON_AMBIGUOUS`] instead. +pub const REST_BUILDER_OPTION_RETRY_MAX_RETRIES: &str = "retry.max_retries"; +#[no_mangle] +pub static REST_BUILDER_OPTION_RETRY_MAX_RETRIES_KEY: [u8; 18] = *b"retry.max_retries\0"; + +/// After an ambiguous PUT, verify object existence before treating the write as successful. +/// Value: `true` or `false`; defaults to `false` when unset. +pub const REST_BUILDER_OPTION_PUT_VERIFY_ON_AMBIGUOUS: &str = "put.verify_on_ambiguous"; +#[no_mangle] +pub static REST_BUILDER_OPTION_PUT_VERIFY_ON_AMBIGUOUS_KEY: [u8; 24] = + *b"put.verify_on_ambiguous\0"; + +/// REST file API dialect passed to +/// [`set_builder_rest_object_store`](crate::set_builder_rest_object_store). +/// +/// Each [`KernelStringSlice`] must remain valid for the duration of that call; the kernel copies +/// the strings into the built engine. Optional fields (the prefixes and `entry_strip_prefix`) may +/// be empty, which the kernel treats as unset. +#[repr(C)] +pub struct CRestEndpointConfig { + pub files_prefix: KernelStringSlice, + pub directories_prefix: KernelStringSlice, + pub page_token_param: KernelStringSlice, + pub start_from_param: KernelStringSlice, + pub recursive_param: KernelStringSlice, + pub overwrite_param: KernelStringSlice, + pub contents_field: KernelStringSlice, + pub next_page_token_field: KernelStringSlice, + pub entry_path_field: KernelStringSlice, + pub entry_size_field: KernelStringSlice, + pub entry_is_directory_field: KernelStringSlice, + pub entry_last_modified_field: KernelStringSlice, + pub entry_strip_prefix: KernelStringSlice, +} + +/// One HTTP header name/value pair produced by a [`CAuthHeaderCallback`]. +/// +/// Set each field with [`allocate_kernel_string`](crate::allocate_kernel_string); ownership +/// transfers to the kernel when the callback returns. +#[repr(C)] +pub struct CAuthHeaderPair { + pub name: Handle, + pub value: Handle, +} + +/// Output buffer for a [`CAuthHeaderCallback`] invocation. +/// +/// Set `count`, fill `headers[0..count]` with +/// [`allocate_kernel_string`](crate::allocate_kernel_string) handles, and optionally set `ttl_ms`. +#[repr(C)] +pub struct CAuthHeaders { + pub count: u32, + pub headers: [CAuthHeaderPair; AUTH_MAX_NUM_HEADERS], + /// How long (in milliseconds) the headers stay valid. `0` means refresh on every request. + pub ttl_ms: u64, +} + +/// Supplies auth headers for a REST-backed engine. +/// +/// The kernel invokes this when it needs fresh headers. For each entry in `headers[0..count]`, +/// set `name` and `value` via [`allocate_kernel_string`](crate::allocate_kernel_string) using the +/// supplied `allocate_error` callback (the same allocator the engine registered on +/// [`get_engine_builder`](crate::get_engine_builder)). Set `ttl_ms` to report how long those +/// headers stay valid. With a non-zero TTL the kernel caches them and re-invokes near expiry; with +/// `ttl_ms = 0` it invokes the callback on every request. +/// +/// `context` is the opaque pointer registered via +/// [`set_builder_rest_object_store`](crate::set_builder_rest_object_store). `allocate_error` is +/// forwarded on each invocation so the callback can pass it to +/// [`allocate_kernel_string`](crate::allocate_kernel_string) without the engine having to stash it +/// separately. +pub type CAuthHeaderCallback = + extern "C" fn(context: NullableCvoid, out: *mut CAuthHeaders, allocate_error: AllocateErrorFn); + +/// State for [`crate::ObjectStoreBackend::Rest`], stored on [`EngineBuilder`](crate::EngineBuilder) +/// after [`set_builder_rest_object_store`](crate::set_builder_rest_object_store). +pub(crate) struct RestBuilderState { + endpoint_config: RestEndpointConfig, + auth_callback: Option, +} + +/// Upcalls a [`CAuthHeaderCallback`] whenever the REST client needs fresh auth headers. +/// +/// Registered via [`set_builder_rest_object_store`](crate::set_builder_rest_object_store). +#[derive(Clone, Copy)] +pub(crate) struct FfiAuthHeaderProvider { + callback: CAuthHeaderCallback, + context: NullableCvoid, + allocate_error: AllocateErrorFn, +} +// SAFETY: see [`set_builder_rest_object_store`](crate::set_builder_rest_object_store): `context` +// and `callback` must be safe to invoke from any thread concurrently. +unsafe impl Send for FfiAuthHeaderProvider {} +unsafe impl Sync for FfiAuthHeaderProvider {} + +impl FfiAuthHeaderProvider { + pub(crate) fn new( + callback: CAuthHeaderCallback, + context: NullableCvoid, + allocate_error: AllocateErrorFn, + ) -> Self { + Self { + callback, + context, + allocate_error, + } + } + + fn collect(&self) -> DeltaResult<(HeaderMap, Option)> { + let mut headers = MaybeUninit::::uninit(); + let out = headers.as_mut_ptr(); + // SAFETY: `out` is valid for writes here. The callback initializes `headers[0..count]`; + // only those slots are read below. + unsafe { + (*out).count = 0; + (*out).ttl_ms = 0; + (self.callback)(self.context, out, self.allocate_error); + let ttl = ((*out).ttl_ms != 0).then(|| Duration::from_millis((*out).ttl_ms)); + let pairs = take_auth_pairs_from_c(out)?; + Ok((headers_from_pairs(pairs)?, ttl)) + } + } +} + +/// Build REST builder state from FFI endpoint config and optional auth callback. +pub(crate) fn rest_builder_state_from_ffi( + endpoint_config: &CRestEndpointConfig, + callback: Option, + context: NullableCvoid, + allocate_error: AllocateErrorFn, +) -> DeltaResult { + Ok(RestBuilderState { + endpoint_config: rest_endpoint_config_from_c(endpoint_config)?, + auth_callback: callback.map(|cb| FfiAuthHeaderProvider::new(cb, context, allocate_error)), + }) +} + +/// Take ownership of initialized `(name, value)` handles in `headers[0..count]`. +/// +/// # Safety +/// +/// `(*headers).count` must not exceed [`AUTH_MAX_NUM_HEADERS`], and every slot in +/// `headers[0..count]` must hold handles produced by +/// [`allocate_kernel_string`](crate::allocate_kernel_string). Padding slots are never read or +/// dropped. +unsafe fn take_auth_pairs_from_c(headers: *mut CAuthHeaders) -> DeltaResult> { + let count = (*headers).count as usize; + if count > AUTH_MAX_NUM_HEADERS { + return Err(Error::generic(format!( + "auth header count {count} exceeds max {AUTH_MAX_NUM_HEADERS}" + ))); + } + + let mut pairs = Vec::with_capacity(count); + for i in 0..count { + let slot = ptr::read(ptr::addr_of_mut!((*headers).headers[i])); + // SAFETY: each handle is moved out of `CAuthHeaders` exactly once. + let name = *slot.name.into_inner(); + let value = *slot.value.into_inner(); + pairs.push((name, value)); + } + Ok(pairs) +} + +/// Copy a [`CRestEndpointConfig`] into an owned [`RestEndpointConfig`]. +pub(crate) fn rest_endpoint_config_from_c( + config: &CRestEndpointConfig, +) -> DeltaResult { + Ok(RestEndpointConfig { + files_prefix: copy_optional_string(&config.files_prefix)?, + directories_prefix: copy_optional_string(&config.directories_prefix)?, + page_token_param: copy_required_string(&config.page_token_param, "page_token_param")?, + start_from_param: copy_required_string(&config.start_from_param, "start_from_param")?, + recursive_param: copy_required_string(&config.recursive_param, "recursive_param")?, + overwrite_param: copy_required_string(&config.overwrite_param, "overwrite_param")?, + contents_field: copy_required_string(&config.contents_field, "contents_field")?, + next_page_token_field: copy_required_string( + &config.next_page_token_field, + "next_page_token_field", + )?, + entry_path_field: copy_required_string(&config.entry_path_field, "entry_path_field")?, + entry_size_field: copy_required_string(&config.entry_size_field, "entry_size_field")?, + entry_is_directory_field: copy_required_string( + &config.entry_is_directory_field, + "entry_is_directory_field", + )?, + entry_last_modified_field: copy_required_string( + &config.entry_last_modified_field, + "entry_last_modified_field", + )?, + entry_strip_prefix: { + let prefix = copy_optional_string(&config.entry_strip_prefix)?; + (!prefix.is_empty()).then_some(prefix) + }, + }) +} + +fn copy_optional_string(slice: &KernelStringSlice) -> DeltaResult { + // SAFETY: caller keeps slice memory valid until `set_builder_rest_object_store` returns. + unsafe { String::try_from_slice(slice) } +} + +fn copy_required_string(slice: &KernelStringSlice, field: &str) -> DeltaResult { + let value = copy_optional_string(slice)?; + if value.is_empty() { + return Err(Error::generic(format!("`{field}` must be non-empty"))); + } + Ok(value) +} + +/// Build a [`RestObjectStore`] from an engine builder's URL, options, and REST state. +pub(crate) fn build_rest_object_store( + base_url: &Url, + options: &HashMap, + rest: &RestBuilderState, +) -> DeltaResult> { + let config = rest.endpoint_config.clone(); + + let auth: Arc = match rest.auth_callback { + Some(cb) => { + let provider = cb; + Arc::new(RefreshingHeaderProvider::new(move || { + provider.collect().map_err(|e| ObjectStoreError::Generic { + store: "RestObjectStore", + source: e.into(), + }) + })) + } + None => { + let header_pairs = options.iter().filter_map(|(k, v)| { + k.strip_prefix(REST_BUILDER_OPTION_HEADER_PREFIX) + .map(|name| (name.to_string(), v.clone())) + }); + Arc::new(StaticHeaderProvider::from_pairs(header_pairs)?) + } + }; + + let tls = RestClientOptions { + cert_path: options.get(REST_BUILDER_OPTION_TLS_CERT_PATH).cloned(), + key_path: options.get(REST_BUILDER_OPTION_TLS_KEY_PATH).cloned(), + ca_path: options.get(REST_BUILDER_OPTION_TLS_CA_PATH).cloned(), + dns_overrides: options + .get(REST_BUILDER_OPTION_TLS_DNS_OVERRIDE) + .map(|s| s.split(',').map(str::to_string).collect()) + .unwrap_or_default(), + timeout_secs: options + .get(REST_BUILDER_OPTION_TLS_TIMEOUT_SECS) + .map(|s| { + s.parse::().map_err(|e| { + Error::generic(format!( + "invalid {} `{s}`: {e}", + REST_BUILDER_OPTION_TLS_TIMEOUT_SECS + )) + }) + }) + .transpose()?, + }; + let max_retries = options + .get(REST_BUILDER_OPTION_RETRY_MAX_RETRIES) + .map(|s| { + s.parse::().map_err(|e| { + Error::generic(format!( + "invalid {} `{s}`: {e}", + REST_BUILDER_OPTION_RETRY_MAX_RETRIES + )) + }) + }) + .transpose()? + .unwrap_or(0); + let verify = parse_bool_option( + REST_BUILDER_OPTION_PUT_VERIFY_ON_AMBIGUOUS, + options.get(REST_BUILDER_OPTION_PUT_VERIFY_ON_AMBIGUOUS), + )?; + + // Validate every option before building the client: constructing the reqwest/rustls client + // initializes a crypto provider, which is minutes of work under the Miri interpreter. + let client = build_rest_client(&tls)?; + + Ok(Arc::new( + RestObjectStore::new(base_url.to_string(), client, auth, Arc::new(config)) + .with_max_retries(max_retries) + .with_verify_on_ambiguous(verify), + )) +} + +fn parse_bool_option(key: &str, value: Option<&String>) -> DeltaResult { + match value { + None => Ok(false), + Some(v) => match v.as_str() { + "true" => Ok(true), + "false" => Ok(false), + other => Err(Error::generic(format!( + "invalid {key} `{other}`: expected `true` or `false`" + ))), + }, + } +} + +#[cfg(test)] +mod tests { + use url::Url; + + use super::*; + use crate::ffi_test_utils::allocate_err; + use crate::kernel_string_slice; + + // Miri policy for this module. See ffi/CLAUDE.md "Testing under Miri" for the policy; + // tests below are grouped by their relationship to `unsafe`, in file order: + // + // 1. Pure-logic tests: no `unsafe`. Run under Miri. + // 2. Unsafe-FFI tests: exercise this module's `unsafe` directly. Run under Miri. + // 3. Client-build tests: reach `build_rest_client`. Their `unsafe` is a subset of what groups + // 1-2 run under Miri, so the slow ones are `#[cfg_attr(miri, ignore)]`. + + fn test_allocate_error() -> AllocateErrorFn { + allocate_err + } + + fn test_auth_header_pair(name: &str, value: &str) -> CAuthHeaderPair { + CAuthHeaderPair { + name: Box::new(name.to_string()).into(), + value: Box::new(value.to_string()).into(), + } + } + + fn option_key_bytes_match_const(key: &[u8], expected: &str) { + let without_nul = &key[..key.len() - 1]; + assert_eq!( + std::str::from_utf8(without_nul).expect("option key bytes should be valid UTF-8"), + expected + ); + } + + fn test_base_url() -> Url { + Url::parse("http://localhost/").unwrap() + } + + static EMPTY: &str = ""; + + fn test_c_rest_endpoint_config() -> CRestEndpointConfig { + let page_token = "page_token"; + let start_from = "start_from"; + let recursive = "recursive"; + let overwrite = "overwrite"; + let contents = "contents"; + let next_page_token = "next_page_token"; + let path = "path"; + let size = "size"; + let is_directory = "is_directory"; + let last_modified = "last_modified"; + CRestEndpointConfig { + files_prefix: kernel_string_slice!(EMPTY), + directories_prefix: kernel_string_slice!(EMPTY), + page_token_param: kernel_string_slice!(page_token), + start_from_param: kernel_string_slice!(start_from), + recursive_param: kernel_string_slice!(recursive), + overwrite_param: kernel_string_slice!(overwrite), + contents_field: kernel_string_slice!(contents), + next_page_token_field: kernel_string_slice!(next_page_token), + entry_path_field: kernel_string_slice!(path), + entry_size_field: kernel_string_slice!(size), + entry_is_directory_field: kernel_string_slice!(is_directory), + entry_last_modified_field: kernel_string_slice!(last_modified), + entry_strip_prefix: kernel_string_slice!(EMPTY), + } + } + + fn test_rest_builder_state() -> RestBuilderState { + rest_builder_state_from_ffi( + &test_c_rest_endpoint_config(), + None, + None, + test_allocate_error(), + ) + .unwrap() + } + + fn write_test_auth_headers(pairs: &[(&str, &str)], out: *mut CAuthHeaders) { + unsafe { + (*out).count = pairs.len() as u32; + (*out).ttl_ms = 0; + for (i, (name, value)) in pairs.iter().enumerate() { + (*out).headers[i] = test_auth_header_pair(name, value); + } + } + } + + // === Group 1: pure-logic tests (no unsafe; run under Miri, cheap) === + + #[test] + fn rest_builder_option_keys_match_c_exports() { + option_key_bytes_match_const( + &REST_BUILDER_OPTION_HEADER_PREFIX_KEY, + REST_BUILDER_OPTION_HEADER_PREFIX, + ); + option_key_bytes_match_const( + &REST_BUILDER_OPTION_TLS_CERT_PATH_KEY, + REST_BUILDER_OPTION_TLS_CERT_PATH, + ); + option_key_bytes_match_const( + &REST_BUILDER_OPTION_TLS_KEY_PATH_KEY, + REST_BUILDER_OPTION_TLS_KEY_PATH, + ); + option_key_bytes_match_const( + &REST_BUILDER_OPTION_TLS_CA_PATH_KEY, + REST_BUILDER_OPTION_TLS_CA_PATH, + ); + option_key_bytes_match_const( + &REST_BUILDER_OPTION_TLS_DNS_OVERRIDE_KEY, + REST_BUILDER_OPTION_TLS_DNS_OVERRIDE, + ); + option_key_bytes_match_const( + &REST_BUILDER_OPTION_TLS_TIMEOUT_SECS_KEY, + REST_BUILDER_OPTION_TLS_TIMEOUT_SECS, + ); + option_key_bytes_match_const( + &REST_BUILDER_OPTION_RETRY_MAX_RETRIES_KEY, + REST_BUILDER_OPTION_RETRY_MAX_RETRIES, + ); + option_key_bytes_match_const( + &REST_BUILDER_OPTION_PUT_VERIFY_ON_AMBIGUOUS_KEY, + REST_BUILDER_OPTION_PUT_VERIFY_ON_AMBIGUOUS, + ); + } + + #[test] + fn rest_endpoint_config_from_c_copies_fields() { + let config = rest_endpoint_config_from_c(&test_c_rest_endpoint_config()).unwrap(); + assert_eq!(config.page_token_param, "page_token"); + assert_eq!(config.entry_path_field, "path"); + assert!(config.entry_strip_prefix.is_none()); + } + + #[test] + fn rest_endpoint_config_from_c_rejects_empty_required_field() { + let mut config = test_c_rest_endpoint_config(); + config.page_token_param = kernel_string_slice!(EMPTY); + assert!(rest_endpoint_config_from_c(&config).is_err()); + } + + // === Group 2: unsafe-FFI tests (kept under Miri) === + // + // These exercise `take_auth_pairs_from_c` (raw-pointer reads of a C-filled `*mut CAuthHeaders`) + // and `FfiAuthHeaderProvider::collect` (which upcalls the C callback and dereferences its + // output): the undefined-behavior surface Miri checks here. + + #[test] + fn auth_pairs_from_c_takes_handles() { + let mut storage = MaybeUninit::::uninit(); + write_test_auth_headers( + &[ + ("Authorization", "Bearer t"), + ("X-Databricks-Org-Id", "123"), + ], + storage.as_mut_ptr(), + ); + let pairs = unsafe { take_auth_pairs_from_c(storage.as_mut_ptr()).unwrap() }; + assert_eq!( + pairs, + vec![ + ("Authorization".to_string(), "Bearer t".to_string()), + ("X-Databricks-Org-Id".to_string(), "123".to_string()), + ] + ); + } + + #[test] + fn auth_pairs_from_c_rejects_excess_count() { + let mut storage = MaybeUninit::::uninit(); + unsafe { + (*storage.as_mut_ptr()).count = (AUTH_MAX_NUM_HEADERS + 1) as u32; + } + assert!(unsafe { take_auth_pairs_from_c(storage.as_mut_ptr()).is_err() }); + } + + extern "C" fn fill_auth_direct(_: NullableCvoid, out: *mut CAuthHeaders, _: AllocateErrorFn) { + write_test_auth_headers(&[("authorization", "Bearer t")], out); + unsafe { + (*out).ttl_ms = 60_000; + } + } + + #[test] + fn auth_callback_collects_direct_fill_and_ttl() { + let provider = FfiAuthHeaderProvider { + callback: fill_auth_direct, + context: None, + allocate_error: test_allocate_error(), + }; + let (headers, ttl) = provider.collect().unwrap(); + assert_eq!(headers.get("authorization").unwrap(), "Bearer t"); + assert_eq!(headers.len(), 1); + assert_eq!(ttl, Some(Duration::from_millis(60_000))); + } + + // === Group 3: client-build tests (call build_rest_object_store) === + // + // These execute no `unsafe` beyond what groups 1-2 already run under Miri. + // `build_rest_object_store` has no `unsafe`; the `try_from_slice` in their config setup + // (via `rest_endpoint_config_from_c`) is covered by group 1, and + // `FfiAuthHeaderProvider::collect` by group 2 (it sits behind a per-request closure these + // build-only tests never fire). Their only Miri-relevant cost is `build_rest_client`, which + // builds the reqwest/rustls client and initializes the crypto provider: minutes of safe + // arithmetic under the interpreter. + // + // Split accordingly: + // + // 3a. Rejected cheaply, before rustls crypto init. Kept under Miri. + // + // 3b. Client built fully (the success cases). Marked `#[cfg_attr(miri, ignore)]`: skipping + // drops no coverage (groups 1-2 already cover their `unsafe`). They still run under + // `cargo test` / nextest. + + // === 3a: rejected before rustls crypto init (cheap; run under Miri) === + // + // `build_rest_object_store` validates every option before building the client, so a bad option + // rejects without paying crypto init. Partial mTLS rejects inside `build_rest_client` itself, + // but before `builder.build()`, so no crypto stack is built either. + + #[test] + fn build_rejects_invalid_timeout() { + let mut options = HashMap::new(); + options.insert(REST_BUILDER_OPTION_TLS_TIMEOUT_SECS.into(), "nope".into()); + assert!( + build_rest_object_store(&test_base_url(), &options, &test_rest_builder_state()) + .is_err() + ); + } + + #[test] + fn build_rejects_invalid_header_value() { + let mut options = HashMap::new(); + options.insert( + format!("{}X", REST_BUILDER_OPTION_HEADER_PREFIX), + "bad\nvalue".into(), + ); + assert!( + build_rest_object_store(&test_base_url(), &options, &test_rest_builder_state()) + .is_err() + ); + } + + #[test] + fn build_rejects_partial_mtls() { + let mut options = HashMap::new(); + options.insert( + REST_BUILDER_OPTION_TLS_CERT_PATH.into(), + "/x/cert.pem".into(), + ); + assert!( + build_rest_object_store(&test_base_url(), &options, &test_rest_builder_state()) + .is_err() + ); + } + + #[test] + fn build_rejects_invalid_max_retries() { + let mut options = HashMap::new(); + options.insert(REST_BUILDER_OPTION_RETRY_MAX_RETRIES.into(), "nope".into()); + assert!( + build_rest_object_store(&test_base_url(), &options, &test_rest_builder_state()) + .is_err() + ); + } + + #[test] + fn build_rejects_invalid_verify_on_ambiguous() { + let mut options = HashMap::new(); + options.insert( + REST_BUILDER_OPTION_PUT_VERIFY_ON_AMBIGUOUS.into(), + "nope".into(), + ); + assert!( + build_rest_object_store(&test_base_url(), &options, &test_rest_builder_state()) + .is_err() + ); + } + + // === 3b: build the reqwest/rustls client (~minutes under Miri, so skipped) === + // + // Every test below is `#[cfg_attr(miri, ignore)]` because it builds the client and pays the + // full rustls crypto-init cost under the interpreter. Their `unsafe` is already run under Miri + // by groups 1-2, so skipping drops no coverage. See the group-3 banner above. + // + // DO NOT ADD NEW `unsafe` TO A TEST BELOW (unsafe groups 1-2 don't already run). Miri never + // runs them, so unsafe unique to them goes unchecked for undefined behavior and nothing in CI + // reports the gap. Put it in a group 1-2 test. + + #[test] + #[cfg_attr( + miri, + ignore = "builds reqwest/rustls client (no unsafe); crypto init is minutes under Miri" + )] + fn build_succeeds_with_refreshing_auth_callback() { + let rest = rest_builder_state_from_ffi( + &test_c_rest_endpoint_config(), + Some(fill_auth_direct), + None, + test_allocate_error(), + ) + .unwrap(); + assert!(build_rest_object_store(&test_base_url(), &HashMap::new(), &rest).is_ok()); + } + + #[test] + #[cfg_attr( + miri, + ignore = "builds reqwest/rustls client (no unsafe); crypto init is minutes under Miri" + )] + fn build_succeeds_with_header_options_fallback() { + let mut options = HashMap::new(); + options.insert( + format!("{}Authorization", REST_BUILDER_OPTION_HEADER_PREFIX), + "Bearer x".into(), + ); + assert!( + build_rest_object_store(&test_base_url(), &options, &test_rest_builder_state()).is_ok() + ); + } + + #[test] + #[cfg_attr( + miri, + ignore = "builds reqwest/rustls client (no unsafe); crypto init is minutes under Miri" + )] + fn build_succeeds_with_prefixes_and_resilience_options() { + let prefix = "/TablesById/u"; + let mut config = test_c_rest_endpoint_config(); + config.files_prefix = kernel_string_slice!(prefix); + config.entry_strip_prefix = kernel_string_slice!(prefix); + let rest = rest_builder_state_from_ffi(&config, None, None, test_allocate_error()).unwrap(); + + let mut options = HashMap::new(); + options.insert( + format!("{}Authorization", REST_BUILDER_OPTION_HEADER_PREFIX), + "Bearer x".into(), + ); + options.insert(REST_BUILDER_OPTION_RETRY_MAX_RETRIES.into(), "3".into()); + options.insert( + REST_BUILDER_OPTION_PUT_VERIFY_ON_AMBIGUOUS.into(), + "true".into(), + ); + assert!(build_rest_object_store(&test_base_url(), &options, &rest).is_ok()); + } +} diff --git a/ffi/src/scan.rs b/ffi/src/scan.rs index 6c64d6044b..86759f307e 100644 --- a/ffi/src/scan.rs +++ b/ffi/src/scan.rs @@ -5,13 +5,17 @@ use std::ffi::c_void; use std::sync::{Arc, Mutex}; use delta_kernel::scan::state::{DvInfo, ScanFile}; -use delta_kernel::scan::{Scan, ScanMetadata}; +use delta_kernel::scan::{PartitionValuesOptions, Scan, ScanBuilder, ScanMetadata, StatsOptions}; +use delta_kernel::schema::MetadataValue; use delta_kernel::snapshot::SnapshotRef; -use delta_kernel::{DeltaResult, Error, Expression, ExpressionRef}; +use delta_kernel::{DeltaResult, DeltaResultIteratorStatic, Error, Expression, ExpressionRef}; use delta_kernel_ffi_macros::handle_descriptor; use tracing::debug; use url::Url; +use super::handle::Handle; +#[cfg(feature = "default-engine-base")] +use crate::engine_data::ArrowFFIData; use crate::expressions::kernel_visitor::{unwrap_kernel_predicate, KernelExpressionVisitorState}; use crate::expressions::SharedExpression; use crate::schema_visitor::{extract_kernel_schema, KernelSchemaVisitorState}; @@ -22,27 +26,81 @@ use crate::{ TryFromStringSlice, }; -use super::handle::Handle; - -// TODO: Why do we even need to expose a scan, when the only thing an engine can do with it is -// handit back to the kernel by calling `scan_metadata_iter_init`? There isn't even an FFI method to -// drop it! #[handle_descriptor(target=Scan, mutable=false, sized=true)] pub struct SharedScan; #[handle_descriptor(target=ScanMetadata, mutable=false, sized=true)] pub struct SharedScanMetadata; +/// Boxed scan metadata iterator stored behind [`ScanMetadataIterator`]'s mutex. +type ScanMetadataIter = DeltaResultIteratorStatic; + +/// An opaque, exclusive handle owning a [`ScanBuilder`]. +/// +/// The caller must eventually either call [`scan_builder_build`] (which consumes the handle +/// and produces a [`SharedScan`]) or [`free_scan_builder`] (which drops it without building). +#[handle_descriptor(target=ScanBuilder, mutable=true, sized=true)] +pub struct ExclusiveScanBuilder; + +/// Configuration for the stats output requested from scan metadata. +/// +/// The default behavior is `JsonOnly`. +/// +/// cbindgen:prefix-with-name=true +#[repr(C)] +pub enum FfiStatsOptions { + /// Emit JSON stats only. + JsonOnly, + /// Emit all indexed struct stats without synthesizing JSON stats. + AllStruct, + /// Emit both JSON stats and all indexed struct stats. + All, + /// Emit no stats and disable kernel data skipping. + None, + // TODO: Expose StructColumns variant (matching `StatsOptions::struct_columns`) +} + +impl From for StatsOptions { + fn from(options: FfiStatsOptions) -> Self { + match options { + FfiStatsOptions::JsonOnly => Self::json_only(), + FfiStatsOptions::AllStruct => Self::all_struct(), + FfiStatsOptions::All => Self::all(), + FfiStatsOptions::None => Self::none(), + } + } +} + +/// Configuration for the partition-value output requested from scan metadata. +/// +/// The default behavior is `StringMapOnly`. +/// +/// cbindgen:prefix-with-name=true +#[repr(C)] +pub enum FfiPartitionValuesOptions { + /// Emit only the raw string map. + StringMapOnly, + /// Also emit the typed `partitionValues_parsed` struct. + WithStruct, +} + +impl From for PartitionValuesOptions { + fn from(options: FfiPartitionValuesOptions) -> Self { + match options { + FfiPartitionValuesOptions::StringMapOnly => Self::string_map_only(), + FfiPartitionValuesOptions::WithStruct => Self::with_struct(), + } + } +} + /// A predicate that can be used to skip data when scanning. /// -/// When invoking [`scan`], The engine provides a pointer to the (engine's native) predicate, along -/// with a visitor function that can be invoked to recursively visit the predicate. This engine -/// state must be valid until the call to [`scan`] returns. Inside that method, the kernel allocates -/// visitor state, which becomes the second argument to the predicate visitor invocation along with -/// the engine-provided predicate pointer. The visitor state is valid for the lifetime of the -/// predicate visitor invocation. Thanks to this double indirection, engine and kernel each retain -/// ownership of their respective objects, with no need to coordinate memory lifetimes with the -/// other. +/// Used by [`scan`] and [`scan_builder_with_predicate`]. The engine provides a pointer to its +/// native predicate along with a visitor function that recursively visits it. This engine state +/// must remain valid for the duration of the call. The kernel allocates visitor state internally, +/// which becomes the second argument to the visitor invocation. Thanks to this double indirection, +/// engine and kernel each retain ownership of their respective objects with no need to coordinate +/// memory lifetimes. #[repr(C)] pub struct EnginePredicate { pub predicate: *mut c_void, @@ -50,17 +108,16 @@ pub struct EnginePredicate { extern "C" fn(predicate: *mut c_void, state: &mut KernelExpressionVisitorState) -> usize, } -/// A schema for columns to select from the snapshot. +/// An engine-provided schema along with a visitor function to convert it to a kernel schema. /// -/// This allows engines to specify which columns they want to read for projection pushdown or to -/// specify metadata columns. The engine provides a pointer to its native schema representation -/// along with a visitor function. The kernel uses this to build a kernel -/// [`delta_kernel::schema::Schema`] that specifies the projection. Inside [`scan`] the kernel -/// allocates visitor state, which becomes the second argument to the schema visitor invocation -/// along with the engine-provided schema pointer. The visitor state is valid for the lifetime of -/// the schema visitor invocation. Thanks to this double indirection, engine and kernel each retain -/// ownership of their respective objects, with no need to coordinate memory lifetimes with the -/// other. +/// Used by [`scan`] and [`scan_builder_with_schema`] for projection pushdown, and by +/// [`get_create_table_builder`] to specify the table schema at creation time. The engine +/// provides a pointer to its native schema representation along with a visitor function. The +/// kernel allocates visitor state internally, which becomes the second argument to the schema +/// visitor invocation. Thanks to this double indirection, engine and kernel each retain +/// ownership of their respective objects with no need to coordinate memory lifetimes. +/// +/// [`get_create_table_builder`]: crate::transaction::get_create_table_builder #[repr(C)] pub struct EngineSchema { pub schema: *mut c_void, @@ -131,38 +188,215 @@ pub unsafe extern "C" fn scan( snapshot: Handle, engine: Handle, predicate: Option<&mut EnginePredicate>, - schema: Option<&mut EngineSchema>, + schema: Option<&EngineSchema>, ) -> ExternResult> { let snapshot = unsafe { snapshot.clone_as_arc() }; scan_impl(snapshot, predicate, schema).into_extern_result(&engine.as_ref()) } +/// Decode an [`EnginePredicate`] into a kernel [`Predicate`] by running the engine's visitor. +/// +/// Returns an error if the visitor fails to produce a valid predicate (i.e. returns an invalid +/// expression ID). A `None` result from the visitor indicates the engine-side predicate +/// construction failed, which would silently widen to a full scan if ignored. +pub(crate) fn decode_engine_predicate( + predicate: &mut EnginePredicate, +) -> DeltaResult { + let mut visitor_state = KernelExpressionVisitorState::default(); + let pred_id = (predicate.visitor)(predicate.predicate, &mut visitor_state); + unwrap_kernel_predicate(&mut visitor_state, pred_id).ok_or_else(|| { + delta_kernel::Error::generic( + "engine predicate visitor returned an invalid expression ID; \ + predicate could not be decoded", + ) + }) +} + +/// Decode an [`EnginePredicate`] and apply it to a [`ScanBuilder`]. +/// +/// Returns an error if the engine's visitor fails to produce a valid predicate; see +/// [`decode_engine_predicate`]. +fn apply_predicate( + builder: ScanBuilder, + predicate: &mut EnginePredicate, +) -> DeltaResult { + let predicate = decode_engine_predicate(predicate)?; + debug!("Got predicate: {:#?}", predicate); + Ok(builder.with_predicate(Some(Arc::new(predicate)))) +} + +/// Decode an [`EngineSchema`] and apply it as a column projection to a [`ScanBuilder`]. +/// +/// Returns an error if the schema visitor produces an invalid schema. +fn apply_schema(builder: ScanBuilder, schema: &EngineSchema) -> DeltaResult { + let mut visitor_state = KernelSchemaVisitorState::default(); + let schema_id = (schema.visitor)(schema.schema, &mut visitor_state); + let schema = extract_kernel_schema(&mut visitor_state, schema_id)?; + debug!("FFI scan projection schema: {:#?}", schema); + Ok(builder.with_schema(Arc::new(schema))) +} + fn scan_impl( snapshot: SnapshotRef, predicate: Option<&mut EnginePredicate>, - schema: Option<&mut EngineSchema>, + schema: Option<&EngineSchema>, ) -> DeltaResult> { let mut scan_builder = snapshot.scan_builder(); - if let Some(predicate) = predicate { - let mut visitor_state = KernelExpressionVisitorState::default(); - let pred_id = (predicate.visitor)(predicate.predicate, &mut visitor_state); - let predicate = unwrap_kernel_predicate(&mut visitor_state, pred_id); - debug!("Got predicate: {:#?}", predicate); - scan_builder = scan_builder.with_predicate(predicate.map(Arc::new)); + scan_builder = apply_predicate(scan_builder, predicate)?; } - if let Some(schema) = schema { - let mut visitor_state = KernelSchemaVisitorState::default(); - let schema_id = (schema.visitor)(schema.schema, &mut visitor_state); - let schema = extract_kernel_schema(&mut visitor_state, schema_id)?; - debug!("FFI scan projection schema: {:#?}", schema); - scan_builder = scan_builder.with_schema(Arc::new(schema)); + scan_builder = apply_schema(scan_builder, schema)?; } - Ok(Arc::new(scan_builder.build()?).into()) } +/// Create a [`ScanBuilder`] for the given snapshot. +/// +/// The caller owns the returned handle and must eventually call either +/// [`scan_builder_build`] to produce a [`SharedScan`], or [`free_scan_builder`] to drop it +/// without building. +/// +/// This function is infallible; constructing a [`ScanBuilder`] from a snapshot always succeeds. +/// +/// # Safety +/// +/// `snapshot` must be a valid [`SharedSnapshot`] handle. +#[no_mangle] +pub unsafe extern "C" fn scan_builder( + snapshot: Handle, +) -> Handle { + let snapshot = unsafe { snapshot.clone_as_arc() }; + Box::new(snapshot.scan_builder()).into() +} + +/// Apply a predicate to an [`ExclusiveScanBuilder`] for data skipping and row-level filtering. +/// +/// Consumes the `builder` handle and returns a new handle with the predicate applied. The +/// `builder` handle must not be used after this call. Returns an error if the engine's predicate +/// visitor fails to produce a valid predicate (i.e. returns an invalid expression ID). On error, +/// the builder is dropped. +/// +/// # Safety +/// +/// `builder` and `engine` must be valid handles. The `builder` handle must not be used after this +/// call. `predicate` must be a valid, non-null [`EnginePredicate`] whose `visitor` and `predicate` +/// fields are safe to call and read. +#[no_mangle] +pub unsafe extern "C" fn scan_builder_with_predicate( + builder: Handle, + engine: Handle, + predicate: &mut EnginePredicate, +) -> ExternResult> { + let engine = unsafe { engine.as_ref() }; + let builder = unsafe { builder.into_inner() }; + apply_predicate(*builder, predicate) + .map(|b| Box::new(b).into()) + .into_extern_result(&engine) +} + +/// Apply a column projection schema to an [`ExclusiveScanBuilder`]. +/// +/// Consumes the `builder` handle and returns a new handle with the schema applied. The `builder` +/// handle must not be used after this call. Returns an error if the schema visitor produces an +/// invalid schema, such as a non-struct root or unconsumed field IDs. On error, the builder is +/// dropped. +/// +/// # Safety +/// +/// `builder` and `engine` must be valid handles. The `builder` handle must not be used after this +/// call. `schema` must be a valid, non-null [`EngineSchema`] whose `visitor` and `schema` fields +/// are safe to call and read. +#[no_mangle] +pub unsafe extern "C" fn scan_builder_with_schema( + builder: Handle, + engine: Handle, + schema: &EngineSchema, +) -> ExternResult> { + let engine = unsafe { engine.as_ref() }; + scan_builder_with_schema_impl(builder, schema).into_extern_result(&engine) +} + +fn scan_builder_with_schema_impl( + builder: Handle, + schema: &EngineSchema, +) -> DeltaResult> { + let builder = unsafe { builder.into_inner() }; + Ok(Box::new(apply_schema(*builder, schema)?).into()) +} + +/// Configure how stats are included in the resulting scan metadata. +/// +/// Consumes the `builder` handle and returns a new handle with `options` applied. The caller must +/// replace its builder handle with the returned handle. The old handle must not be reused or +/// freed. Calls replace the previous stats options. +/// +/// # Safety +/// +/// `builder` must be a valid handle that has not been consumed or freed. +#[no_mangle] +pub unsafe extern "C" fn scan_builder_with_stats( + builder: Handle, + options: FfiStatsOptions, +) -> Handle { + let builder = unsafe { builder.into_inner() }; + Box::new(builder.with_stats(options.into())).into() +} + +/// Configure how partition values are included in the resulting scan metadata. +/// +/// Consumes the `builder` handle and returns a new handle with `options` applied. The caller must +/// replace its builder handle with the returned handle. The old handle must not be reused or +/// freed. Calls replace the previous partition-value options. +/// +/// # Safety +/// +/// `builder` must be a valid handle that has not been consumed or freed. +#[no_mangle] +pub unsafe extern "C" fn scan_builder_with_partition_values( + builder: Handle, + options: FfiPartitionValuesOptions, +) -> Handle { + let builder = unsafe { builder.into_inner() }; + Box::new(builder.with_partition_values(options.into())).into() +} + +/// Consume an [`ExclusiveScanBuilder`] and produce a [`SharedScan`]. +/// +/// The `builder` handle is consumed and must not be used afterward. On error, the builder is +/// dropped and an error is returned. It is the responsibility of the caller to free the returned +/// scan handle by calling [`free_scan`]. +/// +/// # Safety +/// +/// `builder` and `engine` must be valid handles. The `builder` handle must not be used after +/// this call. +#[no_mangle] +pub unsafe extern "C" fn scan_builder_build( + builder: Handle, + engine: Handle, +) -> ExternResult> { + let engine = unsafe { engine.as_ref() }; + let builder = unsafe { builder.into_inner() }; + builder + .build() + .map(|scan| Arc::new(scan).into()) + .into_extern_result(&engine) +} + +/// Free an [`ExclusiveScanBuilder`] without building a scan. +/// +/// Only call this if you will not call [`scan_builder_build`]. If you have already called +/// [`scan_builder_build`], the builder handle was consumed and this must not be called. +/// +/// # Safety +/// +/// `builder` must be a valid handle that has not been previously consumed or freed. +#[no_mangle] +pub unsafe extern "C" fn free_scan_builder(builder: Handle) { + builder.drop_handle(); +} + /// Get the table root of a scan. /// /// # Safety @@ -199,6 +433,48 @@ pub unsafe extern "C" fn scan_physical_schema(scan: Handle) -> Handl scan.physical_schema().clone().into() } +/// Build the declarative metadata-scan [`Plan`](delta_kernel::plans::ir::plan::Plan) for a scan and +/// return it as proto-serialized [`Operation`](delta_kernel::Operation) bytes. +/// +/// On success, returns an [`OptionalValue`]: +/// - [`OptionalValue::Some`] wraps a [`KernelOwnedBytes`](crate::KernelOwnedBytes) buffer holding +/// the proto-serialized `delta.kernel.operation.Operation` message (a `QueryPlan`). The engine +/// owns the buffer and must free it with [`free_kernel_bytes`](crate::free_kernel_bytes). +/// - [`OptionalValue::None`] means there is no plan to execute because the scan's predicate +/// statically skips all files. +/// +/// # Safety +/// +/// Caller must pass a valid [`SharedScan`] handle and a valid [`SharedExternEngine`] handle. This +/// function borrows but does not consume either handle; the caller retains ownership of both. +#[cfg(feature = "declarative-plans")] +#[no_mangle] +pub unsafe extern "C" fn scan_declarative_metadata_plan( + scan: Handle, + engine: Handle, +) -> ExternResult> { + let scan = unsafe { scan.as_ref() }; + let extern_engine = unsafe { engine.as_ref() }; + let inner_engine = extern_engine.engine(); + scan_declarative_metadata_plan_impl(scan, inner_engine.as_ref()) + .into_extern_result(&extern_engine) +} + +#[cfg(feature = "declarative-plans")] +fn scan_declarative_metadata_plan_impl( + scan: &Scan, + engine: &dyn delta_kernel::Engine, +) -> DeltaResult> { + let plan = scan.declarative_metadata_scan_plan(engine)?; + Ok(plan + .map(|plan| { + delta_kernel::Operation::QueryPlan(plan) + .to_proto_bytes() + .into() + }) + .into()) +} + // Intentionally opaque to the engine. // // TODO: This approach liberates the engine from having to worry about mutual exclusion, but that @@ -207,16 +483,24 @@ pub unsafe extern "C" fn scan_physical_schema(scan: Handle) -> Handl // conceivable risk, if this interacts poorly with engine's mutual exclusion mechanism. pub struct ScanMetadataIterator { // Mutex -> Allow the iterator to be accessed safely by multiple threads. - // Box -> Wrap its unsized content this struct is fixed-size with thin pointers. - // Item = DeltaResult - data: Mutex> + Send>>, + data: Mutex, - // Also keep a reference to the external engine for its error allocator. The default Parquet and - // Json handlers don't hold any reference to the tokio reactor they rely on, so the iterator - // terminates early if the last engine goes out of scope. + // Also keep a reference to the external engine for its error allocator. The default Parquet + // and Json handlers don't hold any reference to the tokio reactor they rely on, so the + // iterator terminates early if the last engine goes out of scope. engine: Arc, } +impl ScanMetadataIterator { + /// Acquire the iterator's mutex, returning a guard the caller can drain. While the + /// guard is alive, concurrent `scan_metadata_next` calls on the same handle block. + pub(crate) fn lock_iter(&self) -> DeltaResult> { + self.data + .lock() + .map_err(|_| Error::generic("poisoned scan-metadata iterator mutex")) + } +} + #[handle_descriptor(target=ScanMetadataIterator, mutable=false, sized=true)] pub struct SharedScanMetadataIterator; @@ -255,10 +539,10 @@ fn scan_metadata_iter_init_impl( Ok(Arc::new(data).into()) } -/// Call the provided `engine_visitor` on the next scan metadata item. The visitor will be provided with -/// a [`SharedScanMetadata`], which contains the actual scan files and the associated selection vector. It is the -/// responsibility of the _engine_ to free the associated resources after use by calling -/// [`free_engine_data`] and [`free_bool_slice`] respectively. +/// Call the provided `engine_visitor` on the next scan metadata item. The visitor will be provided +/// with a [`SharedScanMetadata`], which contains the actual scan files and the associated selection +/// vector. It is the responsibility of the _engine_ to free the associated resources after use by +/// calling [`free_engine_data`] and [`free_bool_slice`] respectively. /// /// # Safety /// @@ -288,10 +572,7 @@ fn scan_metadata_next_impl( scan_metadata: Handle, ), ) -> DeltaResult { - let mut data = data - .data - .lock() - .map_err(|_| Error::generic("poisoned mutex"))?; + let mut data = data.lock_iter()?; if let Some(scan_metadata) = data.next().transpose()? { (engine_visitor)(engine_context, Arc::new(scan_metadata).into()); Ok(true) @@ -338,7 +619,8 @@ pub struct CDvInfo<'a> { /// This callback will be invoked for each valid file that needs to be read for a scan. /// /// The arguments to the callback are: -/// * `context`: a `void*` context this can be anything that engine needs to pass through to each call +/// * `context`: a `void*` context this can be anything that engine needs to pass through to each +/// call /// * `path`: a `KernelStringSlice` which is the path to the file /// * `size`: an `i64` which is the size of the file /// * `mod_time`: an `i64` which is the time the file was created, as milliseconds since the epoch @@ -421,6 +703,112 @@ pub unsafe extern "C" fn visit_string_map( } } +// === Typed field metadata === + +/// The type of a [`MetadataValue`] carried across the FFI boundary. Each value is delivered as a +/// string (via `Display`); this tag tells the engine how to interpret it: a signed 64-bit integer, +/// a plain string, a boolean value, or arbitrary JSON text. +#[repr(C)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum CMetadataValueKind { + MetadataNumber = 0, + MetadataString = 1, + MetadataBoolean = 2, + MetadataJson = 3, +} + +impl From<&MetadataValue> for CMetadataValueKind { + fn from(val: &MetadataValue) -> Self { + match val { + MetadataValue::Number(_) => CMetadataValueKind::MetadataNumber, + MetadataValue::String(_) => CMetadataValueKind::MetadataString, + MetadataValue::Boolean(_) => CMetadataValueKind::MetadataBoolean, + MetadataValue::Other(_) => CMetadataValueKind::MetadataJson, + } + } +} + +/// A field-metadata map that preserves each value's [`MetadataValue`] type. Used for schema field +/// metadata, where the kernel knows each value's type; the engine recovers that type via +/// [`CMetadataValueKind`] rather than inferring it from the key name. +#[derive(Default)] +pub struct CMetadataMap { + values: HashMap, +} + +impl From> for CMetadataMap { + fn from(values: HashMap) -> Self { + Self { values } + } +} + +/// Probe a [`CMetadataMap`] for a single key. If the key is present, kernel calls `allocate_fn` +/// with its value rendered to a string, writes the value's [`CMetadataValueKind`] to `kind_out` +/// (unless `kind_out` is null), and returns the pointer `allocate_fn` produced. If the key is +/// absent, returns NULL and leaves `kind_out` untouched. +/// +/// # Safety +/// +/// The engine is responsible for providing a valid [`CMetadataMap`] pointer, [`KernelStringSlice`], +/// and (when non-null) a valid `kind_out` pointer. +#[no_mangle] +pub unsafe extern "C" fn get_from_metadata_map( + map: &CMetadataMap, + key: KernelStringSlice, + kind_out: *mut CMetadataValueKind, + allocate_fn: AllocateStringFn, + engine: Handle, +) -> ExternResult { + let engine = unsafe { engine.as_ref() }; + get_from_metadata_map_impl(map, key, kind_out, allocate_fn).into_extern_result(&engine) +} +fn get_from_metadata_map_impl( + map: &CMetadataMap, + key: KernelStringSlice, + kind_out: *mut CMetadataValueKind, + allocate_fn: AllocateStringFn, +) -> DeltaResult { + let string_key = unsafe { TryFromStringSlice::try_from_slice(&key) }?; + let Some(val) = map.values.get(string_key) else { + return Ok(None); + }; + if !kind_out.is_null() { + unsafe { *kind_out = CMetadataValueKind::from(val) }; + } + let value = val.to_string(); + Ok(allocate_fn(kernel_string_slice!(value))) +} + +/// Visit all entries in a [`CMetadataMap`]. The callback is invoked once per entry with the key, +/// the value's [`CMetadataValueKind`], and the value rendered to a string. The engine uses the +/// kind to parse the string back into a typed value. +/// +/// # Safety +/// +/// The engine is responsible for providing a valid [`CMetadataMap`] pointer and callback. +#[no_mangle] +pub unsafe extern "C" fn visit_metadata_map( + map: &CMetadataMap, + engine_context: NullableCvoid, + visitor: extern "C" fn( + engine_context: NullableCvoid, + key: KernelStringSlice, + kind: CMetadataValueKind, + value: KernelStringSlice, + ), +) { + for (key, val) in &map.values { + let kind = CMetadataValueKind::from(val); + let value = val.to_string(); + visitor( + engine_context, + kernel_string_slice!(key), + kind, + kernel_string_slice!(value), + ); + } +} + /// Transformation expressions that need to be applied to each row `i` in ScanMetadata. You can use /// [`get_transform_for_row`] to get the transform for a particular row. If that returns an /// associated expression, it _must_ be applied to the data read from the file specified by the @@ -434,6 +822,17 @@ pub struct CTransforms { transforms: Vec>, } +impl CTransforms { + /// A [`CTransforms`] carrying no per-row transforms, so [`get_transform_for_row`] returns + /// `NULL` for every row. Used by scan paths (e.g. incremental scan) that never rewrite rows. + #[cfg(feature = "default-engine-base")] + pub(crate) fn empty() -> Self { + Self { + transforms: Vec::new(), + } + } +} + #[no_mangle] /// Allow getting the transform for a particular row. If the requested row is outside the range of /// the passed `CTransforms` returns `NULL`, otherwise returns the element at the index of the @@ -541,8 +940,8 @@ struct ContextWrapper { callback: CScanCallback, } -/// Shim for ffi to call visit_scan_metadata. This will generally be called when iterating through scan -/// data which provides the [`SharedScanMetadata`] as each element in the iterator. +/// Shim for ffi to call visit_scan_metadata. This will generally be called when iterating through +/// scan data which provides the [`SharedScanMetadata`] as each element in the iterator. /// /// # Safety /// engine is responsible for passing a valid [`SharedScanMetadata`]. @@ -570,9 +969,826 @@ fn visit_scan_metadata_impl( Ok(true) } +// === Arrow batch-mode scan metadata === + +/// Result of [`scan_metadata_next_arrow`]: an Arrow C Data Interface batch, a selection +/// vector, and per-row transformation expressions. +/// +/// The engine must free this by calling [`free_scan_metadata_arrow_result`] exactly once. +#[cfg(feature = "default-engine-base")] +#[repr(C)] +pub struct ScanMetadataArrowResult { + /// Arrow C Data Interface batch containing scan file metadata (path, size, stats, etc.). + pub arrow_data: ArrowFFIData, + /// Boolean selection vector indicating active rows. Length equals the batch row count; + /// `true` at index `i` means row `i` should be processed. + pub selection_vector: KernelBoolSlice, + /// Opaque pointer to per-row transformation expressions. Use [`get_transform_for_row`] + /// with a row index to retrieve the transform for that row. If non-null, the transform + /// must be applied to data read from the file to produce the correct logical schema. + /// Owned by this struct and freed by [`free_scan_metadata_arrow_result`]. + pub transforms: *mut CTransforms, +} + +/// Get the next scan metadata batch as Arrow via the C Data Interface. +/// +/// Advances the iterator by one batch and returns a [`ScanMetadataArrowResult`] containing: +/// - An Arrow RecordBatch with scan row schema columns (path, size, modificationTime, stats, +/// deletionVector, fileConstantValues) +/// - A boolean selection vector indicating active rows (true = selected) +/// - Per-row transformation expressions (use [`get_transform_for_row`] to access) +/// +/// Returns `Ok(non-null)` with the next batch, `Ok(null)` when the iterator is exhausted, +/// or `Err` if an error occurred during iteration. +/// +/// This is an alternative to the callback-based [`scan_metadata_next`] + +/// [`visit_scan_metadata`] path, avoiding per-row FFI overhead. +/// +/// # Safety +/// +/// `data` must be a valid [`SharedScanMetadataIterator`] handle. +/// `engine` must be a valid [`SharedExternEngine`] handle. +#[cfg(feature = "default-engine-base")] +#[no_mangle] +pub unsafe extern "C" fn scan_metadata_next_arrow( + data: Handle, + engine: Handle, +) -> ExternResult<*mut ScanMetadataArrowResult> { + let data = unsafe { data.as_ref() }; + let engine = unsafe { engine.as_ref() }; + scan_metadata_next_arrow_impl(data).into_extern_result(&engine) +} + +#[cfg(feature = "default-engine-base")] +fn scan_metadata_next_arrow_impl( + data: &ScanMetadataIterator, +) -> DeltaResult<*mut ScanMetadataArrowResult> { + let mut iter = data + .data + .lock() + .map_err(|_| Error::generic("poisoned mutex"))?; + + match iter.next().transpose()? { + Some(scan_metadata) => { + let (engine_data, selection_vector) = scan_metadata.scan_files.into_parts(); + let arrow_data = ArrowFFIData::try_from_engine_data(engine_data)?; + let result = Box::new(ScanMetadataArrowResult { + arrow_data, + selection_vector: selection_vector.into(), + transforms: Box::into_raw(Box::new(CTransforms { + transforms: scan_metadata.scan_file_transforms, + })), + }); + Ok(Box::into_raw(result)) + } + None => Ok(std::ptr::null_mut()), + } +} + +/// Free a [`ScanMetadataArrowResult`] returned by [`scan_metadata_next_arrow`]. +/// +/// # Safety +/// +/// `result` must be a valid pointer returned by [`scan_metadata_next_arrow`], or null. +/// Must be called at most once per result. +#[cfg(feature = "default-engine-base")] +#[no_mangle] +pub unsafe extern "C" fn free_scan_metadata_arrow_result(result: *mut ScanMetadataArrowResult) { + if result.is_null() { + return; + } + let ScanMetadataArrowResult { + arrow_data, + selection_vector, + transforms, + } = unsafe { *Box::from_raw(result) }; + // KernelBoolSlice is a leaked Vec; reconstitute and drop to free + let _ = unsafe { selection_vector.into_vec() }; + // ArrowFFIData's FFI_ArrowArray/FFI_ArrowSchema have Drop impls that call + // their release callbacks if non-null. If the consumer already imported the + // data (calling release), the pointers are null and drop is a no-op. + drop(arrow_data); + // CTransforms was heap-allocated; reconstitute the Box and drop it + drop(unsafe { Box::from_raw(transforms) }); +} + +#[cfg(test)] +mod scan_builder_tests { + #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] + + use std::ffi::c_void; + + use test_utils::{actions_to_string, TestAction}; + + use super::{ + free_scan, free_scan_builder, scan_builder, scan_builder_build, + scan_builder_with_predicate, scan_builder_with_schema, scan_logical_schema, + EnginePredicate, EngineSchema, + }; + use crate::error::KernelError; + use crate::expressions::kernel_visitor::{ + visit_expression_column, visit_expression_literal_int, visit_predicate_lt, + KernelExpressionVisitorState, + }; + use crate::ffi_test_utils::{allocate_err, ok_or_panic, recover_error, setup_snapshot}; + use crate::schema_visitor::{ + visit_field_integer, visit_field_struct, KernelSchemaVisitorState, + }; + use crate::{free_engine, free_schema, free_snapshot, kernel_string_slice, ExternResult}; + + /// Schema visitor that produces `{id: integer (nullable)}` -- a single-column projection of + /// the standard test table schema. + extern "C" fn visit_id_only_schema( + _schema_ptr: *mut c_void, + state: &mut KernelSchemaVisitorState, + ) -> usize { + let id = "id"; + let id_field_id = unsafe { + ok_or_panic(visit_field_integer( + state, + kernel_string_slice!(id), + true, + allocate_err, + )) + }; + let field_ids = [id_field_id]; + let schema = "schema"; + unsafe { + ok_or_panic(visit_field_struct( + state, + kernel_string_slice!(schema), + field_ids.as_ptr(), + 1, + false, + allocate_err, + )) + } + } + + /// Predicate visitor that constructs `id < 10`. + extern "C" fn visit_id_lt_10( + _pred_ptr: *mut c_void, + state: &mut KernelExpressionVisitorState, + ) -> usize { + let id = "id"; + let col = unsafe { + ok_or_panic(visit_expression_column( + state, + kernel_string_slice!(id), + allocate_err, + )) + }; + let lit = visit_expression_literal_int(state, 10); + visit_predicate_lt(state, col, lit) + } + + #[tokio::test] + async fn test_scan_builder_no_pushdown() { + let (engine, snapshot) = setup_snapshot(actions_to_string(vec![TestAction::Metadata])) + .await + .unwrap(); + let builder = unsafe { scan_builder(snapshot.shallow_copy()) }; + let scan = unsafe { ok_or_panic(scan_builder_build(builder, engine.shallow_copy())) }; + // Full schema: both `id` and `val` columns + let schema = unsafe { scan_logical_schema(scan.shallow_copy()) }; + let schema_ref = unsafe { schema.as_ref() }; + assert_eq!(schema_ref.fields().count(), 2); + unsafe { free_schema(schema) }; + unsafe { free_scan(scan) }; + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + } + + #[tokio::test] + async fn test_scan_builder_with_predicate() { + let (engine, snapshot) = setup_snapshot(actions_to_string(vec![TestAction::Metadata])) + .await + .unwrap(); + let builder = unsafe { scan_builder(snapshot.shallow_copy()) }; + let mut predicate = EnginePredicate { + predicate: std::ptr::null_mut(), + visitor: visit_id_lt_10, + }; + let builder = unsafe { + ok_or_panic(scan_builder_with_predicate( + builder, + engine.shallow_copy(), + &mut predicate, + )) + }; + let scan = unsafe { ok_or_panic(scan_builder_build(builder, engine.shallow_copy())) }; + // Predicate does not reduce columns -- full schema is still returned + let schema = unsafe { scan_logical_schema(scan.shallow_copy()) }; + let schema_ref = unsafe { schema.as_ref() }; + assert_eq!(schema_ref.fields().count(), 2); + unsafe { free_schema(schema) }; + unsafe { free_scan(scan) }; + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + } + + #[tokio::test] + async fn test_scan_builder_with_schema() { + let (engine, snapshot) = setup_snapshot(actions_to_string(vec![TestAction::Metadata])) + .await + .unwrap(); + let builder = unsafe { scan_builder(snapshot.shallow_copy()) }; + let schema_arg = EngineSchema { + schema: std::ptr::null_mut(), + visitor: visit_id_only_schema, + }; + let builder = unsafe { + ok_or_panic(scan_builder_with_schema( + builder, + engine.shallow_copy(), + &schema_arg, + )) + }; + let scan = unsafe { ok_or_panic(scan_builder_build(builder, engine.shallow_copy())) }; + // Projection to `{id}` -- only one column in the logical schema + let schema = unsafe { scan_logical_schema(scan.shallow_copy()) }; + let schema_ref = unsafe { schema.as_ref() }; + assert_eq!(schema_ref.fields().count(), 1); + assert!(schema_ref.field("id").is_some()); + unsafe { free_schema(schema) }; + unsafe { free_scan(scan) }; + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + } + + #[tokio::test] + async fn test_scan_builder_with_predicate_and_schema() { + let (engine, snapshot) = setup_snapshot(actions_to_string(vec![TestAction::Metadata])) + .await + .unwrap(); + let builder = unsafe { scan_builder(snapshot.shallow_copy()) }; + let mut predicate = EnginePredicate { + predicate: std::ptr::null_mut(), + visitor: visit_id_lt_10, + }; + let builder = unsafe { + ok_or_panic(scan_builder_with_predicate( + builder, + engine.shallow_copy(), + &mut predicate, + )) + }; + let schema_arg = EngineSchema { + schema: std::ptr::null_mut(), + visitor: visit_id_only_schema, + }; + let builder = unsafe { + ok_or_panic(scan_builder_with_schema( + builder, + engine.shallow_copy(), + &schema_arg, + )) + }; + let scan = unsafe { ok_or_panic(scan_builder_build(builder, engine.shallow_copy())) }; + // Predicate + projection: only `id` in logical schema + let schema = unsafe { scan_logical_schema(scan.shallow_copy()) }; + let schema_ref = unsafe { schema.as_ref() }; + assert_eq!(schema_ref.fields().count(), 1); + assert!(schema_ref.field("id").is_some()); + unsafe { free_schema(schema) }; + unsafe { free_scan(scan) }; + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + } + + /// Schema visitor that returns a bare integer field ID (not wrapped in a struct root). + /// `extract_kernel_schema` requires a struct root, so this produces an error. + extern "C" fn visit_invalid_schema_not_struct( + _schema_ptr: *mut c_void, + state: &mut KernelSchemaVisitorState, + ) -> usize { + let bare_field = "bare_field"; + unsafe { + ok_or_panic(visit_field_integer( + state, + kernel_string_slice!(bare_field), + true, + allocate_err, + )) + } + } + + #[tokio::test] + async fn test_scan_builder_with_schema_then_predicate() { + // Verify that applying schema before predicate produces the same result as the reverse + let (engine, snapshot) = setup_snapshot(actions_to_string(vec![TestAction::Metadata])) + .await + .unwrap(); + let builder = unsafe { scan_builder(snapshot.shallow_copy()) }; + let schema_arg = EngineSchema { + schema: std::ptr::null_mut(), + visitor: visit_id_only_schema, + }; + let builder = unsafe { + ok_or_panic(scan_builder_with_schema( + builder, + engine.shallow_copy(), + &schema_arg, + )) + }; + let mut predicate = EnginePredicate { + predicate: std::ptr::null_mut(), + visitor: visit_id_lt_10, + }; + let builder = unsafe { + ok_or_panic(scan_builder_with_predicate( + builder, + engine.shallow_copy(), + &mut predicate, + )) + }; + let scan = unsafe { ok_or_panic(scan_builder_build(builder, engine.shallow_copy())) }; + // Projection to `{id}` regardless of application order + let schema = unsafe { scan_logical_schema(scan.shallow_copy()) }; + let schema_ref = unsafe { schema.as_ref() }; + assert_eq!(schema_ref.fields().count(), 1); + assert!(schema_ref.field("id").is_some()); + unsafe { free_schema(schema) }; + unsafe { free_scan(scan) }; + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + } + + #[tokio::test] + async fn test_scan_builder_with_schema_error_propagates() { + // An invalid schema (bare primitive field, no struct root) must return ExternResult::Err + let (engine, snapshot) = setup_snapshot(actions_to_string(vec![TestAction::Metadata])) + .await + .unwrap(); + let builder = unsafe { scan_builder(snapshot.shallow_copy()) }; + let schema_arg = EngineSchema { + schema: std::ptr::null_mut(), + visitor: visit_invalid_schema_not_struct, + }; + let result = + unsafe { scan_builder_with_schema(builder, engine.shallow_copy(), &schema_arg) }; + assert!( + matches!(result, ExternResult::Err(_)), + "expected ExternResult::Err for invalid schema" + ); + if let ExternResult::Err(e) = result { + let err = unsafe { recover_error(e) }; + assert_eq!(err.etype, KernelError::SchemaError); + } + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + } + + #[tokio::test] + async fn test_free_scan_builder_without_build() { + let (engine, snapshot) = setup_snapshot(actions_to_string(vec![TestAction::Metadata])) + .await + .unwrap(); + let builder = unsafe { scan_builder(snapshot.shallow_copy()) }; + // Drop without building -- must not panic or leak + unsafe { free_scan_builder(builder) }; + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(engine) }; + } +} + +#[cfg(all(test, feature = "default-engine-base"))] +mod scan_metadata_arrow_tests { + #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] + + use delta_kernel::arrow::array::ffi::from_ffi; + use delta_kernel::arrow::array::{ + Array, Int32Array, Int64Array, RecordBatch, StringArray, StructArray, + }; + use rstest::rstest; + use test_utils::{actions_to_string, TestAction}; + + use super::{ + free_scan, free_scan_metadata_arrow_result, free_scan_metadata_iter, get_transform_for_row, + scan_builder, scan_builder_build, scan_builder_with_partition_values, + scan_builder_with_stats, scan_metadata_iter_init, scan_metadata_next_arrow_impl, + CTransforms, FfiPartitionValuesOptions, FfiStatsOptions, ScanMetadataArrowResult, + SharedScan, SharedScanMetadataIterator, + }; + use crate::engine_data::ArrowFFIData; + use crate::ffi_test_utils::{ok_or_panic, setup_snapshot}; + use crate::{free_engine, free_snapshot}; + + /// Sets up engine, snapshot, scan, and scan metadata iterator from the given test actions. + async fn setup_scan_iter( + actions: Vec, + ) -> ( + crate::handle::Handle, + crate::handle::Handle, + crate::handle::Handle, + crate::handle::Handle, + ) { + let (engine, snapshot) = setup_snapshot(actions_to_string(actions)).await.unwrap(); + let builder = unsafe { scan_builder(snapshot.shallow_copy()) }; + let scan = unsafe { ok_or_panic(scan_builder_build(builder, engine.shallow_copy())) }; + let iter = unsafe { + ok_or_panic(scan_metadata_iter_init( + engine.shallow_copy(), + scan.shallow_copy(), + )) + }; + (engine, snapshot, scan, iter) + } + + /// Consume a non-null `ScanMetadataArrowResult` pointer, import the Arrow FFI data back + /// to a `RecordBatch`, and return the batch, selection vector, and transforms. + unsafe fn import_arrow_result( + ptr: *mut ScanMetadataArrowResult, + ) -> (RecordBatch, Vec, Box) { + let ScanMetadataArrowResult { + arrow_data, + selection_vector, + transforms, + } = *Box::from_raw(ptr); + let ArrowFFIData { array, schema } = arrow_data; + let array_data = from_ffi(array, &schema).unwrap(); + let batch: RecordBatch = StructArray::from(array_data).into(); + let sv = selection_vector.into_vec(); + (batch, sv, Box::from_raw(transforms)) + } + + fn only_selected_row(selection_vector: &[bool]) -> usize { + let selected_rows = selection_vector + .iter() + .enumerate() + .filter_map(|(index, selected)| selected.then_some(index)) + .collect::>(); + assert_eq!(selected_rows.len(), 1); + selected_rows[0] + } + + #[tokio::test] + async fn returns_batch_then_null_on_exhaustion() { + let (engine, snapshot, scan, iter) = setup_scan_iter(vec![ + TestAction::Metadata, + TestAction::Add("file1.parquet".into()), + ]) + .await; + + let iter_ref = unsafe { iter.as_ref() }; + + // First call returns a non-null batch + let ptr = scan_metadata_next_arrow_impl(iter_ref).unwrap(); + assert!(!ptr.is_null()); + unsafe { free_scan_metadata_arrow_result(ptr) }; + + // Subsequent calls return null (exhausted) + let ptr = scan_metadata_next_arrow_impl(iter_ref).unwrap(); + assert!(ptr.is_null()); + let ptr = scan_metadata_next_arrow_impl(iter_ref).unwrap(); + assert!(ptr.is_null()); + + unsafe { + free_scan_metadata_iter(iter); + free_scan(scan); + free_snapshot(snapshot); + free_engine(engine); + } + } + + #[tokio::test] + async fn arrow_data_round_trips_with_correct_schema() { + let (engine, snapshot, scan, iter) = setup_scan_iter(vec![ + TestAction::Metadata, + TestAction::Add("file1.parquet".into()), + ]) + .await; + + let iter_ref = unsafe { iter.as_ref() }; + let ptr = scan_metadata_next_arrow_impl(iter_ref).unwrap(); + assert!(!ptr.is_null()); + + let (batch, sv, _transforms) = unsafe { import_arrow_result(ptr) }; + + // The batch includes all log entries (commitInfo, protocol, metadata, add) but only + // the add file row is selected. Verify schema columns match SCAN_ROW_SCHEMA. + let batch_schema = batch.schema(); + assert!(batch_schema.field_with_name("path").is_ok()); + assert!(batch_schema.field_with_name("size").is_ok()); + assert!(batch_schema.field_with_name("modificationTime").is_ok()); + assert!(batch_schema.field_with_name("stats").is_ok()); + assert!(batch_schema.field_with_name("deletionVector").is_ok()); + assert!(batch_schema.field_with_name("fileConstantValues").is_ok()); + assert!(batch_schema.field_with_name("stats_parsed").is_err()); + assert!(batch_schema + .field_with_name("partitionValues_parsed") + .is_err()); + + // Selection vector length matches batch rows; exactly 1 row selected (the add file) + assert_eq!(sv.len(), batch.num_rows()); + assert_eq!(sv.iter().filter(|&&v| v).count(), 1); + + unsafe { + free_scan_metadata_iter(iter); + free_scan(scan); + free_snapshot(snapshot); + free_engine(engine); + } + } + + #[rstest] + #[case::json_only(FfiStatsOptions::JsonOnly, false, true)] + #[case::all_struct(FfiStatsOptions::AllStruct, true, true)] + #[case::all(FfiStatsOptions::All, true, true)] + #[case::none(FfiStatsOptions::None, false, false)] + #[tokio::test] + async fn stats_options_control_arrow_metadata( + #[case] options: FfiStatsOptions, + #[case] expect_parsed: bool, + #[case] expect_json: bool, + ) { + let (engine, snapshot) = setup_snapshot(actions_to_string(vec![ + TestAction::Metadata, + TestAction::Add("file1.parquet".into()), + ])) + .await + .unwrap(); + let builder = unsafe { scan_builder(snapshot.shallow_copy()) }; + let builder = unsafe { scan_builder_with_stats(builder, options) }; + let scan = unsafe { ok_or_panic(scan_builder_build(builder, engine.shallow_copy())) }; + let iter = unsafe { + ok_or_panic(scan_metadata_iter_init( + engine.shallow_copy(), + scan.shallow_copy(), + )) + }; + + let ptr = scan_metadata_next_arrow_impl(unsafe { iter.as_ref() }).unwrap(); + assert!(!ptr.is_null()); + let (batch, selection_vector, _transforms) = unsafe { import_arrow_result(ptr) }; + let row = only_selected_row(&selection_vector); + + // Verify schema columns match SCAN_ROW_SCHEMA. + let schema = batch.schema(); + assert!(schema.field_with_name("stats").is_ok()); + assert_eq!( + schema.field_with_name("stats_parsed").is_ok(), + expect_parsed + ); + + // Verify stats JSON is present if requested. + let stats_json = batch + .column_by_name("stats") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(!stats_json.is_null(row), expect_json); + + // Verify stats_parsed is present if requested. + let stats = batch.column_by_name("stats_parsed"); + assert_eq!(stats.is_some(), expect_parsed); + if let Some(stats) = stats { + let stats = stats.as_any().downcast_ref::().unwrap(); + assert!(!stats.is_null(row)); + + let num_records = stats + .column_by_name("numRecords") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + let min_values = stats + .column_by_name("minValues") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + let max_values = stats + .column_by_name("maxValues") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + let min_id = min_values + .column_by_name("id") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + let max_id = max_values + .column_by_name("id") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + + assert_eq!(num_records.value(row), 2); + assert_eq!(min_id.value(row), 1); + assert_eq!(max_id.value(row), 3); + } + + unsafe { + free_scan_metadata_iter(iter); + free_scan(scan); + free_snapshot(snapshot); + free_engine(engine); + } + } + + #[rstest] + #[case::string_map_only(FfiPartitionValuesOptions::StringMapOnly, false)] + #[case::with_struct(FfiPartitionValuesOptions::WithStruct, true)] + #[tokio::test] + async fn partition_values_options_control_arrow_metadata( + #[case] options: FfiPartitionValuesOptions, + #[case] expect_parsed: bool, + ) { + let actions = format!( + "{}\n{}", + test_utils::METADATA_WITH_PARTITION_COLS, + r#"{"add":{"path":"val=a/file1.parquet","partitionValues":{"val":"a"},"size":262,"modificationTime":1587968586000,"dataChange":true}}"# + ); + let (engine, snapshot) = setup_snapshot(actions).await.unwrap(); + let builder = unsafe { scan_builder(snapshot.shallow_copy()) }; + let builder = unsafe { scan_builder_with_partition_values(builder, options) }; + let scan = unsafe { ok_or_panic(scan_builder_build(builder, engine.shallow_copy())) }; + let iter = unsafe { + ok_or_panic(scan_metadata_iter_init( + engine.shallow_copy(), + scan.shallow_copy(), + )) + }; + + let ptr = scan_metadata_next_arrow_impl(unsafe { iter.as_ref() }).unwrap(); + assert!(!ptr.is_null()); + let (batch, selection_vector, _transforms) = unsafe { import_arrow_result(ptr) }; + let row = only_selected_row(&selection_vector); + + // Verify schema columns match SCAN_ROW_SCHEMA. + let schema = batch.schema(); + assert!(schema.field_with_name("fileConstantValues").is_ok()); + assert_eq!( + schema.field_with_name("partitionValues_parsed").is_ok(), + expect_parsed + ); + + // Verify partitionValues_parsed is present if requested. + let partition_values = batch.column_by_name("partitionValues_parsed"); + assert_eq!(partition_values.is_some(), expect_parsed); + if let Some(partition_values) = partition_values { + let partition_values = partition_values + .as_any() + .downcast_ref::() + .unwrap(); + assert!(!partition_values.is_null(row)); + let val = partition_values + .column_by_name("val") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!(val.value(row), "a"); + } + + unsafe { + free_scan_metadata_iter(iter); + free_scan(scan); + free_snapshot(snapshot); + free_engine(engine); + } + } + + #[tokio::test] + async fn multiple_add_files_all_selected() { + let (engine, snapshot, scan, iter) = setup_scan_iter(vec![ + TestAction::Metadata, + TestAction::Add("a.parquet".into()), + TestAction::Add("b.parquet".into()), + TestAction::Add("c.parquet".into()), + ]) + .await; + + let iter_ref = unsafe { iter.as_ref() }; + let mut total_selected = 0usize; + + loop { + let ptr = scan_metadata_next_arrow_impl(iter_ref).unwrap(); + if ptr.is_null() { + break; + } + let (batch, sv, _transforms) = unsafe { import_arrow_result(ptr) }; + assert_eq!(sv.len(), batch.num_rows()); + total_selected += sv.iter().filter(|&&v| v).count(); + } + + assert_eq!(total_selected, 3, "3 add files -> 3 selected rows"); + + unsafe { + free_scan_metadata_iter(iter); + free_scan(scan); + free_snapshot(snapshot); + free_engine(engine); + } + } + + #[tokio::test] + async fn empty_table_returns_null_immediately() { + let (engine, snapshot, scan, iter) = setup_scan_iter(vec![TestAction::Metadata]).await; + + let iter_ref = unsafe { iter.as_ref() }; + let ptr = scan_metadata_next_arrow_impl(iter_ref).unwrap(); + assert!(ptr.is_null(), "table with no data files -> immediate null"); + + unsafe { + free_scan_metadata_iter(iter); + free_scan(scan); + free_snapshot(snapshot); + free_engine(engine); + } + } + + #[tokio::test] + async fn simple_table_has_no_transforms() { + let (engine, snapshot, scan, iter) = setup_scan_iter(vec![ + TestAction::Metadata, + TestAction::Add("file1.parquet".into()), + ]) + .await; + + let iter_ref = unsafe { iter.as_ref() }; + let ptr = scan_metadata_next_arrow_impl(iter_ref).unwrap(); + assert!(!ptr.is_null()); + + let (_batch, _sv, transforms) = unsafe { import_arrow_result(ptr) }; + + // Simple table (no partitions, no column mapping) has an empty transforms vec. + // get_transform_for_row returns None for any row index. + assert!(transforms.transforms.is_empty()); + let result = unsafe { get_transform_for_row(0, &transforms) }; + assert!(matches!(result, crate::OptionalValue::None)); + + unsafe { + free_scan_metadata_iter(iter); + free_scan(scan); + free_snapshot(snapshot); + free_engine(engine); + } + } + + #[tokio::test] + async fn partitioned_table_has_transforms() { + let (engine, snapshot) = setup_snapshot(test_utils::actions_to_string_partitioned(vec![ + TestAction::Metadata, + TestAction::Add("val=a/file1.parquet".into()), + ])) + .await + .unwrap(); + + let builder = unsafe { scan_builder(snapshot.shallow_copy()) }; + let scan = unsafe { ok_or_panic(scan_builder_build(builder, engine.shallow_copy())) }; + let iter = unsafe { + ok_or_panic(scan_metadata_iter_init( + engine.shallow_copy(), + scan.shallow_copy(), + )) + }; + + let iter_ref = unsafe { iter.as_ref() }; + let ptr = scan_metadata_next_arrow_impl(iter_ref).unwrap(); + assert!(!ptr.is_null()); + + let (batch, sv, transforms) = unsafe { import_arrow_result(ptr) }; + + // Partitioned table produces per-row transforms. The transforms vec has one + // entry per batch row. Selected (add-file) rows should have Some transform. + assert_eq!(transforms.transforms.len(), batch.num_rows()); + for (i, &selected) in sv.iter().enumerate() { + let result = unsafe { get_transform_for_row(i, &transforms) }; + if selected { + // get_transform_for_row clones the Arc into a Handle; must free it + match result { + crate::OptionalValue::Some(handle) => unsafe { handle.drop_handle() }, + crate::OptionalValue::None => { + panic!("selected row {i} should have a transform for partitioned table") + } + } + } + } + + unsafe { + free_scan_metadata_iter(iter); + free_scan(scan); + free_snapshot(snapshot); + free_engine(engine); + } + } + + #[test] + fn free_null_is_safe() { + unsafe { free_scan_metadata_arrow_result(std::ptr::null_mut()) }; + } +} + #[cfg(test)] mod tests { - use std::{collections::HashMap, ptr::NonNull}; + use std::collections::HashMap; + use std::ptr::NonNull; use crate::{KernelStringSlice, NullableCvoid, TryFromStringSlice}; @@ -607,4 +1823,207 @@ mod tests { let final_map: HashMap = *unsafe { Box::from_raw(map_ptr) }; assert_eq!(test_map, final_map); } + + #[cfg(feature = "declarative-plans")] + mod declarative_metadata_plan { + use std::ffi::c_void; + + use delta_kernel::plans::proto::operation as proto_op; + use prost::Message as _; + use test_utils::{actions_to_string, TestAction}; + + use super::super::{ + free_scan, scan_builder, scan_builder_build, scan_builder_with_predicate, + scan_declarative_metadata_plan, EnginePredicate, + }; + use crate::error::EngineExecResult; + use crate::expressions::kernel_visitor::{ + visit_expression_literal_bool, KernelExpressionVisitorState, + }; + use crate::ffi_test_utils::{allocate_err, ok_or_panic, setup_snapshot}; + use crate::handle::Handle; + use crate::plans::result::CPlanResult; + use crate::plans::{get_plan_based_engine, get_plan_executor}; + use crate::{ + free_engine, free_snapshot, KernelBytesSlice, NullableCvoid, OptionalValue, + SharedExternEngine, + }; + + extern "C" fn visit_false( + _predicate: *mut c_void, + state: &mut KernelExpressionVisitorState, + ) -> usize { + visit_expression_literal_bool(state, false) + } + + extern "C" fn unreachable_executor( + _context: NullableCvoid, + _plan_proto: KernelBytesSlice, + _out: *mut EngineExecResult, + ) { + unreachable!("plan executor does not run: these tables have no checkpoint"); + } + + /// Wrap a fallback engine in a `PlanBasedEngine` so it exposes a `PlanExecutor`. + unsafe fn plan_based_engine( + fallback: &Handle, + ) -> Handle { + let executor = unsafe { get_plan_executor(None, unreachable_executor) }; + unsafe { + get_plan_based_engine( + executor, + OptionalValue::Some(fallback.shallow_copy()), + allocate_err, + ) + } + } + + #[tokio::test] + async fn returns_query_plan_bytes() { + let (engine, snapshot) = setup_snapshot(actions_to_string(vec![ + TestAction::Metadata, + TestAction::Add("part-1.parquet".to_string()), + ])) + .await + .unwrap(); + + let builder = unsafe { scan_builder(snapshot.shallow_copy()) }; + let scan = unsafe { ok_or_panic(scan_builder_build(builder, engine.shallow_copy())) }; + let plan_engine = unsafe { plan_based_engine(&engine) }; + + let result = unsafe { + scan_declarative_metadata_plan(scan.shallow_copy(), plan_engine.shallow_copy()) + }; + let bytes = match ok_or_panic(result) { + OptionalValue::Some(bytes) => unsafe { bytes.into_vec() }, + OptionalValue::None => panic!("expected a plan for a scan with no static skip"), + }; + + let op = proto_op::Operation::decode(bytes.as_slice()).expect("decode succeeds"); + assert!( + matches!(op.op, Some(proto_op::operation::Op::QueryPlan(_))), + "expected a QueryPlan operation", + ); + + unsafe { free_scan(scan) }; + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(plan_engine) }; + unsafe { free_engine(engine) }; + } + + #[tokio::test] + async fn returns_none_when_statically_skipped() { + let (engine, snapshot) = setup_snapshot(actions_to_string(vec![ + TestAction::Metadata, + TestAction::Add("part-1.parquet".to_string()), + ])) + .await + .unwrap(); + + let builder = unsafe { scan_builder(snapshot.shallow_copy()) }; + let mut predicate = EnginePredicate { + predicate: std::ptr::null_mut(), + visitor: visit_false, + }; + let builder = unsafe { + ok_or_panic(scan_builder_with_predicate( + builder, + engine.shallow_copy(), + &mut predicate, + )) + }; + let scan = unsafe { ok_or_panic(scan_builder_build(builder, engine.shallow_copy())) }; + let plan_engine = unsafe { plan_based_engine(&engine) }; + + let result = unsafe { + scan_declarative_metadata_plan(scan.shallow_copy(), plan_engine.shallow_copy()) + }; + assert!( + matches!(ok_or_panic(result), OptionalValue::None), + "expected None for a statically-skipped scan", + ); + + unsafe { free_scan(scan) }; + unsafe { free_snapshot(snapshot) }; + unsafe { free_engine(plan_engine) }; + unsafe { free_engine(engine) }; + } + } + + extern "C" fn visit_metadata_entry( + engine_context: NullableCvoid, + key: KernelStringSlice, + kind: super::CMetadataValueKind, + value: KernelStringSlice, + ) { + let map_ptr: *mut HashMap = + engine_context.unwrap().as_ptr().cast(); + let key = unsafe { String::try_from_slice(&key).unwrap() }; + let value = unsafe { String::try_from_slice(&value).unwrap() }; + unsafe { + (*map_ptr).insert(key, (kind, value)); + } + } + + #[test] + fn visit_metadata_map_maps_kind_and_renders_each_variant() { + use delta_kernel::schema::MetadataValue; + + use super::CMetadataValueKind::*; + + // `Number` is i64-only, so a JSON number with a fractional part lands in `Other` -- cover + // both so the "parse as i64" contract for `MetadataNumber` stays honest. + let test_map = HashMap::from([ + ("num".to_string(), MetadataValue::Number(-7)), + ("str".to_string(), MetadataValue::String("hi".into())), + ("bool".to_string(), MetadataValue::Boolean(true)), + ( + "obj".to_string(), + MetadataValue::Other(serde_json::json!({"a": 1})), + ), + ( + "float".to_string(), + MetadataValue::Other(serde_json::json!(1.5)), + ), + ]); + let cmap: super::CMetadataMap = test_map.into(); + + let ctx: Box> = Box::default(); + let ctx_ptr = Box::into_raw(ctx); + unsafe { + let ptr = NonNull::new_unchecked(ctx_ptr.cast()); + super::visit_metadata_map(&cmap, Some(ptr), visit_metadata_entry); + } + let out = *unsafe { Box::from_raw(ctx_ptr) }; + + assert_eq!(out["num"], (MetadataNumber, "-7".to_string())); + assert_eq!(out["str"], (MetadataString, "hi".to_string())); + assert_eq!(out["bool"], (MetadataBoolean, "true".to_string())); + assert_eq!(out["obj"], (MetadataJson, "{\"a\":1}".to_string())); + assert_eq!(out["float"], (MetadataJson, "1.5".to_string())); + } + + #[test] + fn visit_metadata_map_empty_never_invokes_callback() { + let cmap = super::CMetadataMap::default(); + let ctx: Box> = Box::default(); + let ctx_ptr = Box::into_raw(ctx); + unsafe { + let ptr = NonNull::new_unchecked(ctx_ptr.cast()); + super::visit_metadata_map(&cmap, Some(ptr), visit_metadata_entry); + } + let out = *unsafe { Box::from_raw(ctx_ptr) }; + assert!(out.is_empty()); + } + + #[test] + fn metadata_value_kind_discriminants_are_stable() { + // These integers are the FFI ABI contract -- C consumers switch on them. Reordering the + // enum silently renumbers, so pin them here. + use super::CMetadataValueKind::*; + assert_eq!(MetadataNumber as i32, 0); + assert_eq!(MetadataString as i32, 1); + assert_eq!(MetadataBoolean as i32, 2); + assert_eq!(MetadataJson as i32, 3); + } } diff --git a/ffi/src/schema.rs b/ffi/src/schema.rs index e891aaff3a..5545a3d71e 100644 --- a/ffi/src/schema.rs +++ b/ffi/src/schema.rs @@ -1,9 +1,10 @@ use std::os::raw::c_void; +use delta_kernel::schema::{ArrayType, DataType, MapType, PrimitiveType, StructType}; + use crate::handle::Handle; -use crate::scan::CStringMap; +use crate::scan::CMetadataMap; use crate::{kernel_string_slice, KernelStringSlice, SharedSchema}; -use delta_kernel::schema::{ArrayType, DataType, MapType, PrimitiveType, StructType}; /// The `EngineSchemaVisitor` defines a visitor system to allow engines to build their own /// representation of a schema from a particular schema within kernel. @@ -30,7 +31,8 @@ use delta_kernel::schema::{ArrayType, DataType, MapType, PrimitiveType, StructTy /// 3. When visiting a complex schema element, the kernel also passes the "child list" containing /// that element's (already-visited) children. /// 4. The [`visit_schema`] method returns the id of the list of top-level columns -// WARNING: the visitor MUST NOT retain internal references to the string slices passed to visitor methods +// WARNING: the visitor MUST NOT retain internal references to the string slices passed to visitor +// methods #[repr(C)] pub struct EngineSchemaVisitor { /// opaque state pointer @@ -46,7 +48,7 @@ pub struct EngineSchemaVisitor { sibling_list_id: usize, name: KernelStringSlice, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, child_list_id: usize, ), @@ -57,7 +59,7 @@ pub struct EngineSchemaVisitor { sibling_list_id: usize, name: KernelStringSlice, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, child_list_id: usize, ), @@ -69,7 +71,7 @@ pub struct EngineSchemaVisitor { sibling_list_id: usize, name: KernelStringSlice, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, child_list_id: usize, ), @@ -79,7 +81,7 @@ pub struct EngineSchemaVisitor { sibling_list_id: usize, name: KernelStringSlice, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, precision: u8, scale: u8, ), @@ -90,7 +92,7 @@ pub struct EngineSchemaVisitor { sibling_list_id: usize, name: KernelStringSlice, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, ), /// Visit a `long` belonging to the list identified by `sibling_list_id`. @@ -99,7 +101,7 @@ pub struct EngineSchemaVisitor { sibling_list_id: usize, name: KernelStringSlice, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, ), /// Visit an `integer` belonging to the list identified by `sibling_list_id`. @@ -108,7 +110,7 @@ pub struct EngineSchemaVisitor { sibling_list_id: usize, name: KernelStringSlice, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, ), /// Visit a `short` belonging to the list identified by `sibling_list_id`. @@ -117,7 +119,7 @@ pub struct EngineSchemaVisitor { sibling_list_id: usize, name: KernelStringSlice, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, ), /// Visit a `byte` belonging to the list identified by `sibling_list_id`. @@ -126,7 +128,7 @@ pub struct EngineSchemaVisitor { sibling_list_id: usize, name: KernelStringSlice, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, ), /// Visit a `float` belonging to the list identified by `sibling_list_id`. @@ -135,7 +137,7 @@ pub struct EngineSchemaVisitor { sibling_list_id: usize, name: KernelStringSlice, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, ), /// Visit a `double` belonging to the list identified by `sibling_list_id`. @@ -144,7 +146,7 @@ pub struct EngineSchemaVisitor { sibling_list_id: usize, name: KernelStringSlice, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, ), /// Visit a `boolean` belonging to the list identified by `sibling_list_id`. @@ -153,7 +155,7 @@ pub struct EngineSchemaVisitor { sibling_list_id: usize, name: KernelStringSlice, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, ), /// Visit `binary` belonging to the list identified by `sibling_list_id`. @@ -162,7 +164,7 @@ pub struct EngineSchemaVisitor { sibling_list_id: usize, name: KernelStringSlice, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, ), /// Visit a `date` belonging to the list identified by `sibling_list_id`. @@ -171,7 +173,7 @@ pub struct EngineSchemaVisitor { sibling_list_id: usize, name: KernelStringSlice, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, ), /// Visit a `timestamp` belonging to the list identified by `sibling_list_id`. @@ -180,7 +182,7 @@ pub struct EngineSchemaVisitor { sibling_list_id: usize, name: KernelStringSlice, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, ), /// Visit a `timestamp` with no timezone belonging to the list identified by `sibling_list_id`. @@ -189,7 +191,34 @@ pub struct EngineSchemaVisitor { sibling_list_id: usize, name: KernelStringSlice, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, + ), + + /// Visit an `interval year to month` belonging to the list identified by `sibling_list_id`. + pub visit_interval_year_month: extern "C" fn( + data: *mut c_void, + sibling_list_id: usize, + name: KernelStringSlice, + is_nullable: bool, + metadata: &CMetadataMap, + ), + + /// Visit an `interval day to second` belonging to the list identified by `sibling_list_id`. + pub visit_interval_day_time: extern "C" fn( + data: *mut c_void, + sibling_list_id: usize, + name: KernelStringSlice, + is_nullable: bool, + metadata: &CMetadataMap, + ), + + /// Visit a `void` belonging to the list identified by `sibling_list_id`. + pub visit_void: extern "C" fn( + data: *mut c_void, + sibling_list_id: usize, + name: KernelStringSlice, + is_nullable: bool, + metadata: &CMetadataMap, ), /// Visit a `variant` belonging to the list identified by `sibling_list_id`. @@ -198,7 +227,7 @@ pub struct EngineSchemaVisitor { sibling_list_id: usize, name: KernelStringSlice, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, ), } @@ -228,7 +257,7 @@ fn visit_schema_impl(schema: &StructType, visitor: &mut EngineSchemaVisitor) -> field.name(), field.data_type(), field.is_nullable(), - &field.metadata_with_string_values().into(), + &field.metadata().clone().into(), visitor, child_list_id, ); @@ -242,7 +271,7 @@ fn visit_schema_impl(schema: &StructType, visitor: &mut EngineSchemaVisitor) -> contains_null: bool, ) -> usize { let child_list_id = (visitor.make_field_list)(visitor.data, 1); - let metadata = CStringMap::default(); + let metadata = CMetadataMap::default(); visit_schema_item( "array_element", &at.element_type, @@ -260,7 +289,7 @@ fn visit_schema_impl(schema: &StructType, visitor: &mut EngineSchemaVisitor) -> value_contains_null: bool, ) -> usize { let child_list_id = (visitor.make_field_list)(visitor.data, 2); - let metadata = CStringMap::default(); + let metadata = CMetadataMap::default(); visit_schema_item( "map_key", &mt.key_type, @@ -285,7 +314,7 @@ fn visit_schema_impl(schema: &StructType, visitor: &mut EngineSchemaVisitor) -> name: &str, data_type: &DataType, is_nullable: bool, - metadata: &CStringMap, + metadata: &CMetadataMap, visitor: &EngineSchemaVisitor, sibling_list_id: usize, ) { @@ -328,8 +357,229 @@ fn visit_schema_impl(schema: &StructType, visitor: &mut EngineSchemaVisitor) -> &DataType::DATE => call!(visit_date), &DataType::TIMESTAMP => call!(visit_timestamp), &DataType::TIMESTAMP_NTZ => call!(visit_timestamp_ntz), + &DataType::INTERVAL_YEAR_MONTH => call!(visit_interval_year_month), + &DataType::INTERVAL_DAY_TIME => call!(visit_interval_day_time), + &DataType::VOID => call!(visit_void), + #[cfg(feature = "geo-type-in-dev")] + DataType::Primitive(PrimitiveType::Geometry(_)) + | DataType::Primitive(PrimitiveType::Geography(_)) => { + // TODO(#2949): add visit_geometry / visit_geography callbacks carrying the CRS; + // skipping silently drops the column + tracing::warn!("Skipping unsupported geo field '{name}' in FFI schema visit"); + } } } visit_struct_fields(visitor, schema) } + +#[cfg(test)] +mod tests { + #![allow(clippy::unwrap_used, clippy::panic)] + + use delta_kernel::schema::{ArrayType, StructField}; + + use super::*; + use crate::TryFromStringSlice; + + #[derive(Debug, PartialEq, Eq)] + struct VisitedField { + name: String, + data_type: &'static str, + is_nullable: bool, + children: Option, + } + + #[derive(Default)] + struct TestSchemaBuilder { + lists: Vec>, + } + + extern "C" fn make_field_list(data: *mut c_void, reserve: usize) -> usize { + let builder = unsafe { &mut *(data as *mut TestSchemaBuilder) }; + let list_id = builder.lists.len(); + builder.lists.push(Vec::with_capacity(reserve)); + list_id + } + + fn add_field( + data: *mut c_void, + sibling_list_id: usize, + name: KernelStringSlice, + is_nullable: bool, + data_type: &'static str, + children: Option, + ) { + let builder = unsafe { &mut *(data as *mut TestSchemaBuilder) }; + builder.lists[sibling_list_id].push(VisitedField { + name: unsafe { String::try_from_slice(&name) }.unwrap(), + data_type, + is_nullable, + children, + }); + } + + macro_rules! visit_nested_type { + ($fn_name:ident, $type_name:expr) => { + extern "C" fn $fn_name( + data: *mut c_void, + sibling_list_id: usize, + name: KernelStringSlice, + is_nullable: bool, + _metadata: &CMetadataMap, + child_list_id: usize, + ) { + add_field( + data, + sibling_list_id, + name, + is_nullable, + $type_name, + Some(child_list_id), + ); + } + }; + } + + visit_nested_type!(visit_struct, "struct"); + visit_nested_type!(visit_array, "array"); + visit_nested_type!(visit_map, "map"); + + extern "C" fn visit_decimal( + data: *mut c_void, + sibling_list_id: usize, + name: KernelStringSlice, + is_nullable: bool, + _metadata: &CMetadataMap, + _precision: u8, + _scale: u8, + ) { + add_field(data, sibling_list_id, name, is_nullable, "decimal", None); + } + + macro_rules! visit_simple_type { + ($fn_name:ident, $type_name:expr) => { + extern "C" fn $fn_name( + data: *mut c_void, + sibling_list_id: usize, + name: KernelStringSlice, + is_nullable: bool, + _metadata: &CMetadataMap, + ) { + add_field(data, sibling_list_id, name, is_nullable, $type_name, None); + } + }; + } + + visit_simple_type!(visit_string, "string"); + visit_simple_type!(visit_long, "long"); + visit_simple_type!(visit_integer, "integer"); + visit_simple_type!(visit_short, "short"); + visit_simple_type!(visit_byte, "byte"); + visit_simple_type!(visit_float, "float"); + visit_simple_type!(visit_double, "double"); + visit_simple_type!(visit_boolean, "boolean"); + visit_simple_type!(visit_binary, "binary"); + visit_simple_type!(visit_date, "date"); + visit_simple_type!(visit_timestamp, "timestamp"); + visit_simple_type!(visit_timestamp_ntz, "timestamp_ntz"); + visit_simple_type!(visit_interval_year_month, "interval year to month"); + visit_simple_type!(visit_interval_day_time, "interval day to second"); + visit_simple_type!(visit_void, "void"); + visit_simple_type!(visit_variant, "variant"); + + fn test_visitor(builder: &mut TestSchemaBuilder) -> EngineSchemaVisitor { + EngineSchemaVisitor { + data: builder as *mut _ as *mut c_void, + make_field_list, + visit_struct, + visit_array, + visit_map, + visit_decimal, + visit_string, + visit_long, + visit_integer, + visit_short, + visit_byte, + visit_float, + visit_double, + visit_boolean, + visit_binary, + visit_date, + visit_timestamp, + visit_timestamp_ntz, + visit_interval_year_month, + visit_interval_day_time, + visit_void, + visit_variant, + } + } + + #[test] + fn visit_schema_preserves_interval_fields() { + let schema = StructType::try_new(vec![ + StructField::nullable("ym", DataType::INTERVAL_YEAR_MONTH), + StructField::not_null("dt", DataType::INTERVAL_DAY_TIME), + StructField::nullable( + "nested", + StructType::try_new(vec![StructField::nullable( + "inner_ym", + DataType::INTERVAL_YEAR_MONTH, + )]) + .unwrap(), + ), + StructField::nullable( + "intervals", + ArrayType::new(DataType::INTERVAL_DAY_TIME, false), + ), + ]) + .unwrap(); + + let mut builder = TestSchemaBuilder::default(); + let mut visitor = test_visitor(&mut builder); + let top_level_id = visit_schema_impl(&schema, &mut visitor); + + assert_eq!(top_level_id, 0); + assert_eq!(builder.lists[0].len(), 4); + assert_eq!( + builder.lists[0][0], + VisitedField { + name: "ym".to_string(), + data_type: "interval year to month", + is_nullable: true, + children: None, + } + ); + assert_eq!( + builder.lists[0][1], + VisitedField { + name: "dt".to_string(), + data_type: "interval day to second", + is_nullable: false, + children: None, + } + ); + + let nested_child_list_id = builder.lists[0][2].children.unwrap(); + assert_eq!( + builder.lists[nested_child_list_id][0], + VisitedField { + name: "inner_ym".to_string(), + data_type: "interval year to month", + is_nullable: true, + children: None, + } + ); + + let array_child_list_id = builder.lists[0][3].children.unwrap(); + assert_eq!( + builder.lists[array_child_list_id][0], + VisitedField { + name: "array_element".to_string(), + data_type: "interval day to second", + is_nullable: false, + children: None, + } + ); + } +} diff --git a/ffi/src/schema_visitor.rs b/ffi/src/schema_visitor.rs index 82ce0d9c5a..5b0a4cefbb 100644 --- a/ffi/src/schema_visitor.rs +++ b/ffi/src/schema_visitor.rs @@ -20,18 +20,18 @@ //! IDs are consumed when used. Each element takes ownership of its referenced child //! elements. Trying to pass an ID more than once to a complex field visitor will result in an //! error. -//! -use crate::{ - AllocateErrorFn, ExternResult, IntoExternResult, KernelStringSlice, ReferenceSet, - TryFromStringSlice, -}; use delta_kernel::schema::{ ArrayType, DataType, DecimalType, MapType, PrimitiveType, StructField, StructType, }; use delta_kernel::{DeltaResult, Error}; use tracing::warn; +use crate::{ + AllocateErrorFn, ExternResult, IntoExternResult, KernelStringSlice, ReferenceSet, + TryFromStringSlice, +}; + #[derive(Default)] pub struct KernelSchemaVisitorState { elements: ReferenceSet, @@ -90,7 +90,8 @@ fn visit_field_primitive_impl( Ok(wrap_field(state, field)) } -// TODO: turn all the primitive visitors below into a macro once cbindgen can run on macro expanded code +// TODO: turn all the primitive visitors below into a macro once cbindgen can run on macro expanded +// code /// Visit a string field. Strings can hold arbitrary UTF-8 text data. /// /// # Safety @@ -307,7 +308,44 @@ pub unsafe extern "C" fn visit_field_timestamp_ntz( .into_extern_result(&allocate_error) } -/// Visit a decimal field. Decimal fields store fixed-precision decimal numbers with specified precision and scale. +/// Visit an interval year-month field. Values store signed month counts. +/// +/// # Safety +/// +/// Caller is responsible for providing a valid `state`, `name` slice with valid UTF-8 data, +/// and `allocate_error` function pointer. +#[no_mangle] +pub unsafe extern "C" fn visit_field_interval_year_month( + state: &mut KernelSchemaVisitorState, + name: KernelStringSlice, + nullable: bool, + allocate_error: AllocateErrorFn, +) -> ExternResult { + let name_str = unsafe { TryFromStringSlice::try_from_slice(&name) }; + visit_field_primitive_impl(state, name_str, PrimitiveType::IntervalYearMonth, nullable) + .into_extern_result(&allocate_error) +} + +/// Visit an interval day-time field. Values store signed microsecond durations. +/// +/// # Safety +/// +/// Caller is responsible for providing a valid `state`, `name` slice with valid UTF-8 data, +/// and `allocate_error` function pointer. +#[no_mangle] +pub unsafe extern "C" fn visit_field_interval_day_time( + state: &mut KernelSchemaVisitorState, + name: KernelStringSlice, + nullable: bool, + allocate_error: AllocateErrorFn, +) -> ExternResult { + let name_str = unsafe { TryFromStringSlice::try_from_slice(&name) }; + visit_field_primitive_impl(state, name_str, PrimitiveType::IntervalDayTime, nullable) + .into_extern_result(&allocate_error) +} + +/// Visit a decimal field. Decimal fields store fixed-precision decimal numbers with specified +/// precision and scale. /// /// # Safety /// @@ -391,7 +429,7 @@ fn create_struct_data_type( .collect::>>()?; let struct_type = StructType::try_new(field_vec)?; - Ok(DataType::Struct(Box::new(struct_type))) + Ok(DataType::from(struct_type)) } fn visit_field_struct_impl( @@ -455,8 +493,8 @@ fn visit_field_array_impl( /// /// # Safety /// -/// Caller is responsible for providing valid `state`, `name` slice, `key_type_id` and `value_type_id` -/// from previous `visit_data_type_*` calls, and `allocate_error` function pointer. +/// Caller is responsible for providing valid `state`, `name` slice, `key_type_id` and +/// `value_type_id` from previous `visit_data_type_*` calls, and `allocate_error` function pointer. #[no_mangle] pub unsafe extern "C" fn visit_field_map( state: &mut KernelSchemaVisitorState, @@ -551,13 +589,15 @@ fn create_variant_data_type( #[cfg(test)] mod tests { + use delta_kernel::schema::{DataType, PrimitiveType}; + use super::*; use crate::error::{EngineError, KernelError}; use crate::ffi_test_utils::ok_or_panic; use crate::KernelStringSlice; - use delta_kernel::schema::{DataType, PrimitiveType}; - // Error allocator for tests that panics when invoked. It is used in tests where we don't expect errors. + // Error allocator for tests that panics when invoked. It is used in tests where we don't expect + // errors. #[no_mangle] extern "C" fn test_allocate_error( etype: KernelError, @@ -751,6 +791,8 @@ mod tests { // col_date: date, // col_timestamp: timestamp, // col_timestamp_ntz: timestamp_ntz, + // col_interval_year_month: interval year to month, + // col_interval_day_time: interval day to second, // col_decimal: decimal(10,2), // col_array: array, // col_map: map, @@ -773,6 +815,10 @@ mod tests { let col_date = visit_field!(date, state, "col_date", false); let col_timestamp = visit_field!(timestamp, state, "col_timestamp", false); let col_timestamp_ntz = visit_field!(timestamp_ntz, state, "col_timestamp_ntz", false); + let col_interval_year_month = + visit_field!(interval_year_month, state, "col_interval_year_month", false); + let col_interval_day_time = + visit_field!(interval_day_time, state, "col_interval_day_time", false); let col_decimal = visit_field!(decimal, state, "col_decimal", 10, 2, false); // Create array @@ -817,6 +863,8 @@ mod tests { col_date, col_timestamp, col_timestamp_ntz, + col_interval_year_month, + col_interval_day_time, col_decimal, col_array, col_map, @@ -837,7 +885,7 @@ mod tests { // Verify the schema let schema = extract_kernel_schema(&mut state, schema_id).unwrap(); let fields: Vec<_> = schema.fields().collect(); - assert_eq!(fields.len(), 17); + assert_eq!(fields.len(), 19); // Validate the primitive fields let primitive_field_expectations = [ @@ -853,6 +901,8 @@ mod tests { ("col_date", PrimitiveType::Date), ("col_timestamp", PrimitiveType::Timestamp), ("col_timestamp_ntz", PrimitiveType::TimestampNtz), + ("col_interval_year_month", PrimitiveType::IntervalYearMonth), + ("col_interval_day_time", PrimitiveType::IntervalDayTime), ]; for (index, (expected_name, expected_type)) in @@ -866,25 +916,25 @@ mod tests { assert!(!fields[index].is_nullable()); } - assert_eq!(fields[12].name(), "col_decimal"); - let DataType::Primitive(PrimitiveType::Decimal(decimal_type)) = fields[12].data_type() + assert_eq!(fields[14].name(), "col_decimal"); + let DataType::Primitive(PrimitiveType::Decimal(decimal_type)) = fields[14].data_type() else { panic!("Field col_decimal is not a decimal type"); }; assert_eq!(decimal_type.precision(), 10); assert_eq!(decimal_type.scale(), 2); - assert_eq!(fields[13].name(), "col_array"); - assert_array(fields[13], DataType::STRING, false); + assert_eq!(fields[15].name(), "col_array"); + assert_array(fields[15], DataType::STRING, false); - assert_eq!(fields[14].name(), "col_map"); - assert_map(fields[14], DataType::STRING, DataType::LONG, false); + assert_eq!(fields[16].name(), "col_map"); + assert_map(fields[16], DataType::STRING, DataType::LONG, false); - assert_eq!(fields[15].name(), "col_struct"); - assert_struct(fields[15], DataType::STRING, false); + assert_eq!(fields[17].name(), "col_struct"); + assert_struct(fields[17], DataType::STRING, false); - assert_eq!(fields[16].name(), "col_variant"); - let DataType::Variant(variant_type) = fields[16].data_type() else { + assert_eq!(fields[18].name(), "col_variant"); + let DataType::Variant(variant_type) = fields[18].data_type() else { panic!("Expected variant type for col_variant"); }; let variant_fields: Vec<_> = variant_type.fields().collect(); @@ -915,7 +965,8 @@ mod tests { // struct< // col_nested: 1.array<2.map<2a.struct, 2b.struct< // inner_arrays: 3.array<4.struct< - // deep_maps: 4a.map<4a1.variant, 4a2.array>, + // deep_maps: 4a.map<4a1.variant, + // 4a2.array>, // variant_data: 4b.variant, // nested_struct: 4c.struct< // final_array: 5.array<6.map<6a.struct, 6b.double>> @@ -1297,7 +1348,8 @@ mod tests { #[test] fn cannot_use_nullable_as_map_keys() { - // Error allocator for tests that panics when invoked. It is used in tests where we don't expect errors. + // Error allocator for tests that panics when invoked. It is used in tests where we don't + // expect errors. #[no_mangle] extern "C" fn ensure_map_err( _etype: KernelError, diff --git a/ffi/src/table_changes.rs b/ffi/src/table_changes.rs index ee2b2cfc55..4fb62e9859 100644 --- a/ffi/src/table_changes.rs +++ b/ffi/src/table_changes.rs @@ -1,22 +1,18 @@ //! TableChanges related ffi code -use std::sync::Arc; -use std::sync::Mutex; +use std::sync::{Arc, Mutex}; use delta_kernel::arrow::array::{Array, ArrayData, StructArray}; use delta_kernel::arrow::ffi::to_ffi; use delta_kernel::engine::arrow_data::EngineDataArrowExt; use delta_kernel::table_changes::scan::TableChangesScan; use delta_kernel::table_changes::TableChanges; -use delta_kernel::EngineData; -use delta_kernel::Error; -use delta_kernel::{DeltaResult, Version}; +use delta_kernel::{DeltaResult, DeltaResultIteratorStatic, EngineData, Error, Version}; use delta_kernel_ffi_macros::handle_descriptor; use tracing::debug; - -use super::handle::Handle; use url::Url; +use super::handle::Handle; use crate::engine_data::ArrowFFIData; use crate::expressions::kernel_visitor::{unwrap_kernel_predicate, KernelExpressionVisitorState}; use crate::scan::EnginePredicate; @@ -33,8 +29,8 @@ pub struct ExclusiveTableChanges; /// /// - `table_root`: url pointing at the table root (where `_delta_log` folder is located) /// - `engine`: Implementation of `Engine` apis. -/// - `start_version`: The start version of the change data feed -/// End version will be the newest table version. +/// - `start_version`: The start version of the change data feed End version will be the newest +/// table version. /// /// # Safety /// @@ -154,8 +150,8 @@ pub unsafe extern "C" fn table_changes_end_version( pub struct SharedTableChangesScan; /// Get a [`TableChangesScan`] over the table specified by the passed table changes. -/// It is the responsibility of the _engine_ to free this scan when complete by calling [`free_table_changes_scan`]. -/// Consumes TableChanges. +/// It is the responsibility of the _engine_ to free this scan when complete by calling +/// [`free_table_changes_scan`]. Consumes TableChanges. /// /// # Safety /// @@ -237,7 +233,7 @@ pub unsafe extern "C" fn table_changes_scan_physical_schema( table_changes_scan.physical_schema().clone().into() } -type TableChangesData = Mutex>> + Send>>; +type TableChangesData = Mutex>>; pub struct ScanTableChangesIterator { data: TableChangesData, @@ -297,26 +293,30 @@ pub unsafe extern "C" fn free_scan_table_changes_iter( /// Get next batch of data from the table changes iterator. /// +/// Returns `Ok(non-null)` with a heap-allocated [`ArrowFFIData`] containing the next batch, +/// `Ok(null)` when the iterator is exhausted, or `Err` on failure. A non-null pointer must +/// be freed by the engine via [`crate::engine_data::free_arrow_ffi_data`] exactly once. +/// /// # Safety /// -/// The iterator must be valid (returned by [table_changes_scan_execute]) and not yet freed by -/// [`free_scan_table_changes_iter`]. +/// The iterator must be valid (returned by [`table_changes_scan_execute`]) and not yet freed +/// by [`free_scan_table_changes_iter`]. #[no_mangle] pub unsafe extern "C" fn scan_table_changes_next( data: Handle, -) -> ExternResult { +) -> ExternResult<*mut ArrowFFIData> { let data = unsafe { data.as_ref() }; scan_table_changes_next_impl(data).into_extern_result(&data.engine.as_ref()) } -fn scan_table_changes_next_impl(data: &ScanTableChangesIterator) -> DeltaResult { +fn scan_table_changes_next_impl(data: &ScanTableChangesIterator) -> DeltaResult<*mut ArrowFFIData> { let mut data = data .data .lock() .map_err(|_| Error::generic("poisoned scan table changes iterator mutex"))?; let Some(data) = data.next().transpose()? else { - return Ok(ArrowFFIData::empty()); + return Ok(std::ptr::null_mut()); }; let record_batch = data.try_into_record_batch()?; @@ -324,17 +324,15 @@ fn scan_table_changes_next_impl(data: &ScanTableChangesIterator) -> DeltaResult< let batch_struct_array: StructArray = record_batch.into(); let array_data: ArrayData = batch_struct_array.into_data(); let (out_array, out_schema) = to_ffi(&array_data)?; - Ok(ArrowFFIData { + Ok(Box::into_raw(Box::new(ArrowFFIData { array: out_array, schema: out_schema, - }) + }))) } #[cfg(test)] mod tests { - use super::*; - use crate::ffi_test_utils::{allocate_err, allocate_str, ok_or_panic, recover_string}; - use crate::{engine_to_handle, free_engine, free_schema, kernel_string_slice}; + use std::sync::Arc; use delta_kernel::arrow::array::{ArrayRef, Int32Array, StringArray}; use delta_kernel::arrow::datatypes::{Field, Schema}; @@ -343,18 +341,23 @@ mod tests { use delta_kernel::arrow::util::pretty::pretty_format_batches; use delta_kernel::engine::arrow_conversion::TryIntoArrow as _; use delta_kernel::engine::arrow_data::ArrowEngineData; - use delta_kernel::engine::default::DefaultEngineBuilder; - use delta_kernel::object_store::{memory::InMemory, path::Path, DynObjectStore}; + use delta_kernel::object_store::memory::InMemory; + use delta_kernel::object_store::path::Path; + use delta_kernel::object_store::{DynObjectStore, ObjectStoreExt as _}; use delta_kernel::schema::{DataType, StructField, StructType}; use delta_kernel::Engine; + use delta_kernel_default_engine::DefaultEngineBuilder; use delta_kernel_ffi::engine_data::get_engine_data; use itertools::Itertools; - use std::sync::Arc; use test_utils::{ actions_to_string_with_metadata, add_commit, generate_batch, record_batch_to_bytes, IntoArray as _, TestAction, }; + use super::*; + use crate::ffi_test_utils::{allocate_err, allocate_str, ok_or_panic, recover_string}; + use crate::{engine_to_handle, free_engine, free_schema, kernel_string_slice}; + const PARQUET_FILE1: &str = "part-00000-a72b1fb3-f2df-41fe-a8f0-e65b746382dd-c000.snappy.parquet"; const PARQUET_FILE2: &str = @@ -462,8 +465,11 @@ mod tests { pub fn generate_batch_with_id(start_i: i32) -> Result { generate_batch(vec![ - ("id", vec![start_i, start_i + 1, start_i + 2].into_array()), - ("val", vec!["a", "b", "c"].into_array()), + ( + "id", + vec![start_i, start_i + 1, start_i + 2].into_arrow_array(), + ), + ("val", vec!["a", "b", "c"].into_arrow_array()), ]) } @@ -703,14 +709,16 @@ mod tests { let mut i: i32 = 0; loop { i += 1; - let data = ok_or_panic(unsafe { + let data_ptr = ok_or_panic(unsafe { scan_table_changes_next(table_changes_scan_iter_result.shallow_copy()) }); - if data.array.is_empty() { + if data_ptr.is_null() { break; } - let engine_data = - ok_or_panic(unsafe { get_engine_data(data.array, &data.schema, allocate_err) }); + // Take ownership of the boxed ArrowFFIData; the inner array/schema are moved + // into get_engine_data which takes ownership of the FFI_ArrowArray. + let ArrowFFIData { array, schema } = *unsafe { Box::from_raw(data_ptr) }; + let engine_data = ok_or_panic(unsafe { get_engine_data(array, &schema, allocate_err) }); let record_batch = unsafe { engine_data.into_inner().try_into_record_batch() }?; println!("Batch ({i}) num rows {:?}", record_batch.num_rows()); diff --git a/ffi/src/test_ffi.rs b/ffi/src/test_ffi.rs index c5ef94292b..a4022cf684 100644 --- a/ffi/src/test_ffi.rs +++ b/ffi/src/test_ffi.rs @@ -4,12 +4,11 @@ use std::sync::Arc; -use crate::expressions::{SharedExpression, SharedPredicate}; -use crate::handle::Handle; use delta_kernel::expressions::{ - column_expr, column_name, column_pred, ArrayData, BinaryExpressionOp, BinaryPredicateOp, - Expression as Expr, MapData, OpaqueExpressionOp, OpaquePredicateOp, Predicate as Pred, Scalar, - ScalarExpressionEvaluator, StructData, Transform, + col, column_name, column_pred, lit, ArrayData, BinaryExpressionOp, BinaryPredicateOp, + Expression as Expr, ExpressionStructPatchBuilder, MapData, MapToStructOptions, + OpaqueExpressionOp, OpaquePredicateOp, Predicate as Pred, Scalar, ScalarExpressionEvaluator, + StructData, }; use delta_kernel::kernel_predicates::{ DirectDataSkippingPredicateEvaluator, DirectPredicateEvaluator, @@ -18,6 +17,9 @@ use delta_kernel::kernel_predicates::{ use delta_kernel::schema::{ArrayType, DataType, MapType, StructField, StructType}; use delta_kernel::DeltaResult; +use crate::expressions::{SharedExpression, SharedPredicate}; +use crate::handle::Handle; + #[derive(Debug, PartialEq)] struct OpaqueTestOp(String); @@ -68,8 +70,8 @@ impl OpaquePredicateOp for OpaqueTestOp { } } -/// Constructs a kernel expression that is passed back as a [`SharedExpression`] handle. The expected -/// output expression can be found in `ffi/tests/test_expression_visitor/expected.txt`. +/// Constructs a kernel expression that is passed back as a [`SharedExpression`] handle. The +/// expected output expression can be found in `ffi/tests/test_expression_visitor/expected.txt`. /// /// # Safety /// The caller is responsible for freeing the returned memory, either by calling @@ -102,32 +104,32 @@ pub unsafe extern "C" fn get_testing_kernel_expression() -> Handle, Expr::literal("prepended").into()) - .with_inserted_field(Some("a".to_string()), Expr::literal("first").into()) - .with_inserted_field( - Some("a".to_string()), - Expr::transform(nested_transform).into(), - ) - .with_inserted_field(Some("a".to_string()), Expr::literal("third").into()); + // NOTE: This convoluted example cannot directly use nested builder helpers, because the fields + // of `foo.bar.baz` are hoisted up as the new top-level columns while the original top-level + // struct becomes a child of `foo.bar.baz`, inserted after `a`. Which means a hypothetical child + // `t` of `foo.bar.baz` will appear twice in the output (as `foo.bar.baz.t` and also as `t`). + let nested_patch = ExpressionStructPatchBuilder::new() + .drop("gone") + .replace("stub", lit("replaced")) + .insert_after("x", lit(true)) + .insert_after("y", lit(false)); + let top_level_patch = ExpressionStructPatchBuilder::new_nested(column_name!("foo.bar.baz")) + .drop("dropme") + .replace("replaceme", lit(42)) + .prepend(lit("prepended")) + .insert_after("a", lit("first")) + .insert_after("a", Expr::struct_patch(nested_patch).unwrap()) + .insert_after("a", lit("third")) + .append(lit("appended")); + let empty_nested_patch = ExpressionStructPatchBuilder::new_nested(column_name!("empty.nested")); let mut sub_exprs = vec![ - column_expr!("col"), + col!("col"), Expr::literal(i8::MAX), Expr::literal(i8::MIN), Expr::literal(f32::MAX), @@ -149,7 +151,9 @@ pub unsafe extern "C" fn get_testing_kernel_expression() -> Handle Handle Handle Handle Handle Handle { let sub_exprs = vec![ - column_expr!("simple_col"), + col!("simple_col"), Expr::literal(42i32), Expr::literal(100i64), Expr::literal(2.5f64), // Using 2.5 to avoid clippy::approx_constant warning @@ -249,6 +255,10 @@ pub unsafe extern "C" fn get_simple_testing_kernel_expression() -> Handle Handle Handle Handle { let sub_preds = vec![ column_pred!("pred_col"), - Pred::literal(true), - Pred::literal(false), + Pred::TRUE, + Pred::FALSE, Pred::eq(Expr::literal(10), Expr::literal(10)), Pred::ne(Expr::literal(5), Expr::literal(10)), Pred::lt(Expr::literal(5), Expr::literal(10)), @@ -297,9 +307,9 @@ pub unsafe extern "C" fn get_simple_testing_kernel_predicate() -> Handle, +} + +/// Mutable handle for a deletion vector descriptor map. +#[handle_descriptor(target=DvDescriptorMap, mutable=true, sized=true)] +pub struct ExclusiveDvDescriptorMap; + +/// Mutable handle for a [`DeletionVectorDescriptor`] crossing the FFI boundary. +#[handle_descriptor(target=DeletionVectorDescriptor, mutable=true, sized=true)] +pub struct ExclusiveDvDescriptor; + +/// Allocate an empty deletion vector descriptor map. The returned handle must be released +/// either by [`free_dv_descriptor_map`] or by [`transaction_update_deletion_vectors`] +/// (which consumes the map). +#[no_mangle] +pub extern "C" fn dv_descriptor_map_new() -> Handle { + Box::new(DvDescriptorMap { + inner: HashMap::new(), + }) + .into() +} + +/// Free a deletion vector descriptor map handle. +/// +/// # Safety +/// +/// Caller must pass a valid handle previously returned by [`dv_descriptor_map_new`]. +#[no_mangle] +pub unsafe extern "C" fn free_dv_descriptor_map(map: Handle) { + map.drop_handle(); +} + +/// Free a deletion vector descriptor handle. Only call this if the descriptor has not +/// been moved into a map via [`dv_descriptor_map_insert`]. +/// +/// # Safety +/// +/// Caller must pass a valid descriptor handle. +#[no_mangle] +pub unsafe extern "C" fn free_dv_descriptor(descriptor: Handle) { + descriptor.drop_handle(); +} + +// ============================================================================= +// DV descriptor construction +// ============================================================================= + +/// Storage type for a [`DeletionVectorDescriptor`], mirroring the protocol-level +/// `storageType` field. Values match the protocol's single-character codes: +/// - `PersistedRelative` (`'u'`): the DV is stored on disk; path is reconstructed from the prefix + +/// z85-encoded UUID held in `path_or_inline_dv` +/// - `Inline` (`'i'`): the deletion vector is stored inline in the log +/// - `PersistedAbsolute` (`'p'`): the DV is stored at the absolute URL given by `path_or_inline_dv` +#[repr(C)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum KernelDvStorageType { + /// `'u'` -- persisted DV with a relative `` reference. + PersistedRelative = 0, + /// `'i'` -- inline DV stored directly in the log. + Inline = 1, + /// `'p'` -- persisted DV with an absolute URL reference. + PersistedAbsolute = 2, +} + +impl From for DeletionVectorStorageType { + fn from(value: KernelDvStorageType) -> Self { + match value { + KernelDvStorageType::PersistedRelative => Self::PersistedRelative, + KernelDvStorageType::Inline => Self::Inline, + KernelDvStorageType::PersistedAbsolute => Self::PersistedAbsolute, + } + } +} + +impl TryFrom for KernelDvStorageType { + type Error = Error; + + fn try_from(value: c_int) -> DeltaResult { + match value { + 0 => Ok(Self::PersistedRelative), + 1 => Ok(Self::Inline), + 2 => Ok(Self::PersistedAbsolute), + _ => Err(Error::generic(format!( + "invalid deletion vector storage type: {value}" + ))), + } + } +} + +/// Construct a [`DeletionVectorDescriptor`] from raw fields, for engines that author DV +/// files themselves and want to install them via [`transaction_update_deletion_vectors`]. +/// +/// Field validation (storage-type rules, non-negative size/cardinality/offset, etc.) is +/// performed by [`DeletionVectorDescriptor::try_new`]; see its docs for the full contract. +/// Pass `has_offset = false` to omit the offset. +/// +/// For persisted DVs, `offset` is the byte offset within the DV file at which the DV's +/// 4-byte big-endian size prefix begins. For a single-DV file, this is usually `1` +/// (skipping the version byte). Omitting the offset is only appropriate for single-DV files +/// where the size prefix begins immediately after the version byte. +/// +/// # Safety +/// +/// Caller must pass valid string slice and engine handle. +#[no_mangle] +pub unsafe extern "C" fn dv_descriptor_new( + storage_type: c_int, + path_or_inline_dv: KernelStringSlice, + has_offset: bool, + offset: i32, + size_in_bytes: i32, + cardinality: i64, + engine: Handle, +) -> ExternResult> { + let engine = unsafe { engine.as_ref() }; + let path = unsafe { TryFromStringSlice::try_from_slice(&path_or_inline_dv) }; + let result = KernelDvStorageType::try_from(storage_type).and_then(|storage_type| { + dv_descriptor_new_impl( + storage_type, + path, + has_offset, + offset, + size_in_bytes, + cardinality, + ) + }); + result.into_extern_result(&engine) +} + +fn dv_descriptor_new_impl( + storage_type: KernelDvStorageType, + path: DeltaResult<&str>, + has_offset: bool, + offset: i32, + size_in_bytes: i32, + cardinality: i64, +) -> DeltaResult> { + let descriptor = DeletionVectorDescriptor::try_new( + storage_type.into(), + path?, + has_offset.then_some(offset), + size_in_bytes, + cardinality, + )?; + Ok(Box::new(descriptor).into()) +} + +/// Insert a deletion vector descriptor into the map under the given data file path. +/// Consumes the descriptor handle on success. On error (e.g. invalid `data_file_path`), +/// the descriptor handle is left untouched and must still be released by the caller via +/// [`free_dv_descriptor`]. +/// +/// `data_file_path` must be the data-file path exactly as it appears in the scan +/// metadata produced by the kernel (the Add file action's `path` field). The kernel +/// matches against this string when applying the DV update; a typo causes +/// [`transaction_update_deletion_vectors`] to return an error. +/// Re-inserting a descriptor for an existing path replaces the previous descriptor. +/// +/// # Safety +/// +/// Caller must pass valid handles. The descriptor handle is consumed only on success. +#[no_mangle] +pub unsafe extern "C" fn dv_descriptor_map_insert( + mut map: Handle, + data_file_path: KernelStringSlice, + descriptor: Handle, + engine: Handle, +) -> ExternResult { + let map_ref = unsafe { map.as_mut() }; + let engine_ref = unsafe { engine.as_ref() }; + // Parse the path BEFORE taking ownership of the descriptor: if parsing fails the + // descriptor must remain valid so the caller can free it (otherwise we get a UAF + // when they retry or clean up). + let path_result = unsafe { TryFromStringSlice::try_from_slice(&data_file_path) }; + dv_descriptor_map_insert_impl(map_ref, path_result, descriptor) + .map(|_| true) + .into_extern_result(&engine_ref) +} + +fn dv_descriptor_map_insert_impl( + map: &mut DvDescriptorMap, + data_file_path: DeltaResult<&str>, + descriptor: Handle, +) -> DeltaResult<()> { + let path = data_file_path?; + let owned_path = path.to_string(); + let descriptor = unsafe { descriptor.into_inner() }; + map.inner.insert(owned_path, *descriptor); + Ok(()) +} + +// ============================================================================= +// Apply DV updates to a transaction +// ============================================================================= + +/// Stage deletion-vector update actions on the transaction. For every entry in `dv_map` +/// the kernel emits a Remove + Add action pair on commit (the Add carries the new DV +/// descriptor; row-level statistics from the original Add are preserved). +/// Every descriptor path must match a selected scan row. Matched scan metadata must have an +/// accurate `numRecords` statistic and valid AddFile fields, including a non-empty path, +/// non-negative size, and exact physical partition keys. +/// +/// Consumes both `dv_map` and `scan_iter`. The engine should pass an iterator that covers +/// at least every file path mentioned in the map; extra files are ignored. If the map +/// references a path that does not appear in the iterator, the call returns an error and +/// leaves the transaction unchanged. +/// +/// This stages data-changing DV updates by default. Call +/// [`crate::transaction::set_data_change`] first for maintenance operations that should commit +/// with `dataChange = false`. +/// +/// # Safety +/// +/// Caller must pass valid handles. The transaction handle is borrowed in place and remains +/// valid after this call; the caller is expected to follow with `commit` (or +/// `free_transaction`) on the same handle. The DV map and scan iterator handles are +/// consumed and must not be used or freed after this call. +#[no_mangle] +pub unsafe extern "C" fn transaction_update_deletion_vectors( + mut txn: Handle, + dv_map: Handle, + scan_iter: Handle, + engine: Handle, +) -> ExternResult { + let txn_ref = unsafe { txn.as_mut() }; + let dv_map = unsafe { dv_map.into_inner() }; + let scan_iter = unsafe { scan_iter.into_inner() }; + let engine_ref = unsafe { engine.as_ref() }; + transaction_update_deletion_vectors_impl(txn_ref, *dv_map, &scan_iter) + .map(|_| true) + .into_extern_result(&engine_ref) +} + +fn transaction_update_deletion_vectors_impl( + txn: &mut Transaction, + dv_map: DvDescriptorMap, + scan_iter: &crate::scan::ScanMetadataIterator, +) -> DeltaResult<()> { + let mut guard = scan_iter.lock_iter()?; + let files_iter = Transaction::scan_metadata_to_engine_data(&mut **guard); + txn.update_deletion_vectors(dv_map.inner, files_iter) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn kernel_dv_storage_type_maps_to_kernel_storage_type() { + let cases = [ + ( + KernelDvStorageType::PersistedRelative, + DeletionVectorStorageType::PersistedRelative, + ), + ( + KernelDvStorageType::Inline, + DeletionVectorStorageType::Inline, + ), + ( + KernelDvStorageType::PersistedAbsolute, + DeletionVectorStorageType::PersistedAbsolute, + ), + ]; + for (ffi, kernel) in cases { + assert_eq!(DeletionVectorStorageType::from(ffi), kernel); + } + } + + #[test] + fn kernel_dv_storage_type_rejects_invalid_discriminant() { + let err = KernelDvStorageType::try_from(42).expect_err("expected error"); + assert!( + err.to_string() + .contains("invalid deletion vector storage type"), + "unexpected: {err}" + ); + } + + #[test] + fn dv_descriptor_new_round_trips_fields() { + let handle = dv_descriptor_new_impl( + KernelDvStorageType::PersistedAbsolute, + Ok("file:///tmp/dv.bin"), + true, + 7, + 42, + 9, + ) + .expect("descriptor should be valid"); + let descriptor = unsafe { handle.into_inner() }; + + assert_eq!( + descriptor.storage_type, + DeletionVectorStorageType::PersistedAbsolute + ); + assert_eq!(descriptor.path_or_inline_dv, "file:///tmp/dv.bin"); + assert_eq!(descriptor.offset, Some(7)); + assert_eq!(descriptor.size_in_bytes, 42); + assert_eq!(descriptor.cardinality, 9); + } + + #[test] + fn dv_descriptor_new_surfaces_kernel_validation_error() { + let err = dv_descriptor_new_impl(KernelDvStorageType::Inline, Ok("ABC"), true, 0, 4, 1) + .err() + .expect("expected validation error"); + assert!(err.to_string().contains("inline"), "unexpected: {err}"); + } + + #[test] + fn dv_descriptor_map_insert_error_leaves_descriptor_freeable() { + let mut map = DvDescriptorMap { + inner: HashMap::new(), + }; + let descriptor = + dv_descriptor_new_impl(KernelDvStorageType::Inline, Ok("ABC"), false, 0, 4, 1) + .expect("descriptor should be valid"); + + let result = dv_descriptor_map_insert_impl( + &mut map, + Err(Error::generic("bad data file path")), + descriptor.shallow_copy(), + ); + + assert!( + result.is_err(), + "insert should fail before consuming descriptor" + ); + assert!(map.inner.is_empty()); + unsafe { free_dv_descriptor(descriptor) }; + } +} diff --git a/ffi/src/transaction/mod.rs b/ffi/src/transaction/mod.rs index 499170efbb..601862f37f 100644 --- a/ffi/src/transaction/mod.rs +++ b/ffi/src/transaction/mod.rs @@ -1,27 +1,69 @@ //! This module holds functionality for managing transactions. +mod deletion_vector; +mod partition_value; mod transaction_id; mod write_context; use std::sync::Arc; -use crate::error::{ExternResult, IntoExternResult}; -use crate::handle::Handle; -use crate::{unwrap_and_parse_path_as_url, TryFromStringSlice}; -use crate::{DeltaResult, ExternEngine, Snapshot, Url}; -use crate::{ExclusiveEngineData, SharedExternEngine}; -use crate::{KernelStringSlice, SharedSnapshot}; +pub use deletion_vector::{ + dv_descriptor_map_insert, dv_descriptor_map_new, dv_descriptor_new, free_dv_descriptor, + free_dv_descriptor_map, transaction_update_deletion_vectors, ExclusiveDvDescriptor, + ExclusiveDvDescriptorMap, KernelDvStorageType, +}; use delta_kernel::committer::{Committer, FileSystemCommitter}; -use delta_kernel::transaction::{CommitResult, Transaction}; +use delta_kernel::engine_data::FilteredEngineData; +use delta_kernel::transaction::create_table::{ + CreateTableTransaction, CreateTableTransactionBuilder, +}; +use delta_kernel::transaction::data_layout::DataLayout; +use delta_kernel::transaction::{CommitResult, CommittedTransaction, Transaction}; use delta_kernel_ffi_macros::handle_descriptor; +pub use partition_value::{ + free_partition_value_map, partition_value_map_insert_binary, partition_value_map_insert_bool, + partition_value_map_insert_byte, partition_value_map_insert_date, + partition_value_map_insert_decimal, partition_value_map_insert_double, + partition_value_map_insert_float, partition_value_map_insert_int, + partition_value_map_insert_long, partition_value_map_insert_null, + partition_value_map_insert_short, partition_value_map_insert_string, + partition_value_map_insert_timestamp, partition_value_map_insert_timestamp_ntz, + partition_value_map_new, ExclusivePartitionValueMap, +}; -/// A handle representing an exclusive transaction on a Delta table. (Similar to a Box<_>) +use crate::error::{ExternResult, IntoExternResult}; +use crate::handle::Handle; +use crate::scan::EngineSchema; +use crate::schema_visitor::{extract_kernel_schema, KernelSchemaVisitorState}; +use crate::{ + unwrap_and_parse_path_as_url, DeltaResult, ExclusiveEngineData, ExternEngine, + KernelStringSlice, OptionalValue, SharedExternEngine, SharedSnapshot, Snapshot, + TryFromStringSlice, Url, +}; + +/// A handle for an existing-table transaction (`Transaction`). /// -/// This struct provides a safe wrapper around the underlying `Transaction` type, -/// ensuring exclusive access to transaction operations. The transaction can be used -/// to stage changes and commit them atomically to the Delta table. +/// Returned by [`transaction`] and [`transaction_with_committer`]. Supports all transaction +/// operations including existing-table-only operations like blind append and file removal. #[handle_descriptor(target=Transaction, mutable=true, sized=true)] pub struct ExclusiveTransaction; +/// A handle for a create-table transaction (`Transaction`). +/// +/// Returned by [`create_table_builder_build`]. Only supports operations valid during table +/// creation: adding files, setting data change, engine info, and committing. Operations like +/// file removal, blind append, and deletion vector updates are not available. +#[handle_descriptor(target=CreateTableTransaction, mutable=true, sized=true)] +pub struct ExclusiveCreateTransaction; + +/// A handle for a [`CommittedTransaction`]. +/// +/// Returned by [`commit`] and [`create_table_commit`]. Carries the committed version and, +/// when available, the post-commit snapshot. Use [`committed_transaction_version`] and +/// [`committed_transaction_post_commit_snapshot`] to read the contents, then release with +/// [`free_committed_transaction`]. +#[handle_descriptor(target=CommittedTransaction, mutable=true, sized=true)] +pub struct ExclusiveCommittedTransaction; + /// Handle for a mutable boxed committer that can be passed across FFI #[handle_descriptor(target = dyn Committer, mutable = true, sized = false)] pub struct MutableCommitter; @@ -80,6 +122,36 @@ fn transaction_with_committer_impl( Ok(Box::new(transaction?).into()) } +/// Convert a [`CommitResult`] into a [`CommittedTransaction`] handle, or an error if the commit +/// was not successful. +/// +/// The returned handle owns the [`CommittedTransaction`] and must be freed with +/// [`free_committed_transaction`]. +/// +/// TODO: expose the full `CommitResult` enum through FFI for conflict resolution. +fn commit_result_to_committed_handle( + result: DeltaResult>, +) -> DeltaResult> { + match result? { + CommitResult::CommittedTransaction(committed) => Ok(Box::new(committed).into()), + CommitResult::RetryableTransaction(_) => Err(delta_kernel::Error::unsupported( + "commit failed: retryable transaction not supported in FFI (yet)", + )), + CommitResult::ConflictedTransaction(conflicted) => { + Err(delta_kernel::Error::Generic(format!( + "commit conflict at version {}", + conflicted.conflict_version() + ))) + } + } +} + +// ============================================================================ +// Existing-table transaction FFI functions +// ============================================================================ + +/// Free an existing-table transaction handle without committing. +/// /// # Safety /// /// Caller is responsible for passing a valid handle. @@ -88,12 +160,11 @@ pub unsafe extern "C" fn free_transaction(txn: Handle) { txn.drop_handle(); } -/// Attaches commit information to a transaction. The commit info contains metadata about the -/// transaction that will be written to the log during commit. +/// Attaches engine info to an existing-table transaction. /// /// # Safety /// -/// Caller is responsible for passing a valid handle. CONSUMES TRANSACTION and commit info +/// Caller is responsible for passing a valid handle. CONSUMES the transaction handle. #[no_mangle] pub unsafe extern "C" fn with_engine_info( txn: Handle, @@ -102,7 +173,6 @@ pub unsafe extern "C" fn with_engine_info( ) -> ExternResult> { let txn = unsafe { txn.into_inner() }; let engine = unsafe { engine.as_ref() }; - with_engine_info_impl(*txn, engine_info).into_extern_result(&engine) } @@ -110,9 +180,99 @@ fn with_engine_info_impl( txn: Transaction, engine_info: KernelStringSlice, ) -> DeltaResult> { - let info_string: DeltaResult<&str> = - unsafe { TryFromStringSlice::try_from_slice(&engine_info) }; - Ok(Box::new(txn.with_engine_info(info_string?)).into()) + let info: &str = unsafe { TryFromStringSlice::try_from_slice(&engine_info) }?; + Ok(Box::new(txn.with_engine_info(info)).into()) +} + +/// Set the operation that this transaction is performing. This string will be persisted in the +/// commit and visible to anyone who describes the table history. This CONSUMES the transaction +/// handle and returns a new handle for the updated transaction. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid handle. CONSUMES the transaction handle. +#[no_mangle] +pub unsafe extern "C" fn with_operation( + txn: Handle, + operation: KernelStringSlice, + engine: Handle, +) -> ExternResult> { + let txn = unsafe { txn.into_inner() }; + let engine = unsafe { engine.as_ref() }; + with_operation_impl(*txn, operation).into_extern_result(&engine) +} + +fn with_operation_impl( + txn: Transaction, + operation: KernelStringSlice, +) -> DeltaResult> { + let operation: String = unsafe { TryFromStringSlice::try_from_slice(&operation) }?; + Ok(Box::new(txn.with_operation(operation)).into()) +} + +/// Add domain metadata to the transaction. The domain metadata will be written to the Delta log +/// as a `domainMetadata` action when the transaction is committed. +/// +/// `domain` identifies the metadata domain (e.g. `"myApp"`). `configuration` is an arbitrary +/// string value associated with the domain (typically JSON). +/// +/// Each domain can only appear once per transaction. Setting metadata for multiple distinct +/// domains is allowed. Duplicate domains or setting and removing the same domain in a single +/// transaction will cause the commit to fail. +/// +/// # Safety +/// +/// Caller is responsible for passing valid handles. CONSUMES the transaction handle and returns +/// a new one. +#[no_mangle] +pub unsafe extern "C" fn with_domain_metadata( + txn: Handle, + domain: KernelStringSlice, + configuration: KernelStringSlice, + engine: Handle, +) -> ExternResult> { + let txn = unsafe { txn.into_inner() }; + let engine = unsafe { engine.as_ref() }; + with_domain_metadata_impl(*txn, domain, configuration).into_extern_result(&engine) +} + +fn with_domain_metadata_impl( + txn: Transaction, + domain: KernelStringSlice, + configuration: KernelStringSlice, +) -> DeltaResult> { + let domain = unsafe { TryFromStringSlice::try_from_slice(&domain) }?; + let configuration = unsafe { TryFromStringSlice::try_from_slice(&configuration) }?; + Ok(Box::new(txn.with_domain_metadata(domain, configuration)).into()) +} + +/// Remove domain metadata from the table in this transaction. A tombstone action with +/// `removed: true` will be written to the Delta log when the transaction is committed. +/// +/// The caller does not need to provide a configuration value -- the existing value is +/// automatically preserved in the tombstone. +/// +/// # Safety +/// +/// Caller is responsible for passing valid handles. CONSUMES the transaction handle and returns +/// a new one. +#[no_mangle] +pub unsafe extern "C" fn with_domain_metadata_removed( + txn: Handle, + domain: KernelStringSlice, + engine: Handle, +) -> ExternResult> { + let txn = unsafe { txn.into_inner() }; + let engine = unsafe { engine.as_ref() }; + with_domain_metadata_removed_impl(*txn, domain).into_extern_result(&engine) +} + +fn with_domain_metadata_removed_impl( + txn: Transaction, + domain: KernelStringSlice, +) -> DeltaResult> { + let domain = unsafe { TryFromStringSlice::try_from_slice(&domain) }?; + Ok(Box::new(txn.with_domain_metadata_removed(domain)).into()) } /// Add file metadata to the transaction for files that have been written. The metadata contains @@ -132,7 +292,6 @@ pub unsafe extern "C" fn add_files( txn.add_files(write_metadata); } -/// /// Mark the transaction as having data changes or not (these are recorded at the file level). /// /// # Safety @@ -144,8 +303,12 @@ pub unsafe extern "C" fn set_data_change(mut txn: Handle, underlying_txn.set_data_change(data_change); } -/// Attempt to commit a transaction to the table. Returns version number if successful. -/// Returns error if the commit fails. +/// Attempt to commit a transaction to the table. On success, returns a handle to the +/// [`CommittedTransaction`] from which the caller can read the version and the optional +/// post-commit snapshot. The returned handle must be freed with [`free_committed_transaction`]. +/// +/// Returns an error if the commit fails. The FFI surfaces conflicted and retryable +/// `CommitResult` variants as errors today (see TODO on `commit_result_to_committed_handle`). /// /// # Safety /// @@ -155,67 +318,560 @@ pub unsafe extern "C" fn set_data_change(mut txn: Handle, pub unsafe extern "C" fn commit( txn: Handle, engine: Handle, -) -> ExternResult { +) -> ExternResult> { let txn = unsafe { txn.into_inner() }; let extern_engine = unsafe { engine.as_ref() }; let engine = extern_engine.engine(); - // TODO: for now this removes the enum, which prevents doing any conflict resolution. We should fix - // this by making the commit function return the enum somehow. - match txn.commit(engine.as_ref()) { - Ok(CommitResult::CommittedTransaction(committed)) => Ok(committed.commit_version()), - Ok(CommitResult::RetryableTransaction(_)) => Err(delta_kernel::Error::unsupported( - "commit failed: retryable transaction not supported in FFI (yet)", - )), - Ok(CommitResult::ConflictedTransaction(conflicted)) => { - Err(delta_kernel::Error::Generic(format!( - "commit conflict at version {}", - conflicted.conflict_version() - ))) - } - Err(e) => Err(e), + commit_result_to_committed_handle(txn.commit(engine.as_ref())) + .into_extern_result(&extern_engine) +} + +// ============================================================================ +// Create-table transaction FFI functions +// ============================================================================ + +/// Free a create-table transaction handle without committing. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid handle. +#[no_mangle] +pub unsafe extern "C" fn create_table_free_transaction(txn: Handle) { + txn.drop_handle(); +} + +/// Attaches engine info to a create-table transaction. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid handle. CONSUMES the transaction handle. +#[no_mangle] +pub unsafe extern "C" fn create_table_with_engine_info( + txn: Handle, + engine_info: KernelStringSlice, + engine: Handle, +) -> ExternResult> { + let txn = unsafe { txn.into_inner() }; + let engine = unsafe { engine.as_ref() }; + create_table_with_engine_info_impl(*txn, engine_info).into_extern_result(&engine) +} + +fn create_table_with_engine_info_impl( + txn: CreateTableTransaction, + engine_info: KernelStringSlice, +) -> DeltaResult> { + let info: &str = unsafe { TryFromStringSlice::try_from_slice(&engine_info) }?; + Ok(Box::new(txn.with_engine_info(info)).into()) +} + +/// Add file metadata to a create-table transaction for files that have been written. The metadata +/// contains information about files written during the transaction that will be added to the +/// Delta log during commit. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid handle. Consumes write_metadata. +#[no_mangle] +pub unsafe extern "C" fn create_table_add_files( + mut txn: Handle, + write_metadata: Handle, +) { + let txn = unsafe { txn.as_mut() }; + let write_metadata = unsafe { write_metadata.into_inner() }; + txn.add_files(write_metadata); +} + +/// Mark the create-table transaction as having data changes or not (these are recorded at the +/// file level). +/// +/// # Safety +/// +/// Caller is responsible for passing a valid handle. +#[no_mangle] +pub unsafe extern "C" fn create_table_set_data_change( + mut txn: Handle, + data_change: bool, +) { + let underlying_txn = unsafe { txn.as_mut() }; + underlying_txn.set_data_change(data_change); +} + +/// Attempt to commit a create-table transaction. On success, returns a handle to the +/// [`CommittedTransaction`] from which the caller can read the version and the optional +/// post-commit snapshot. The returned handle must be freed with [`free_committed_transaction`]. +/// +/// Returns an error if the commit fails. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid handle. And MUST NOT USE transaction after this +/// method is called. +#[no_mangle] +pub unsafe extern "C" fn create_table_commit( + txn: Handle, + engine: Handle, +) -> ExternResult> { + let txn = unsafe { txn.into_inner() }; + let extern_engine = unsafe { engine.as_ref() }; + let engine = extern_engine.engine(); + commit_result_to_committed_handle(txn.commit(engine.as_ref())) + .into_extern_result(&extern_engine) +} + +// ============================================================================ +// Committed transaction accessors +// ============================================================================ + +// TODO: expose CommittedTransaction::post_commit_stats through FFI. + +/// Free a [`CommittedTransaction`] handle. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid handle and must not use it after this call. +#[no_mangle] +pub unsafe extern "C" fn free_committed_transaction(txn: Handle) { + txn.drop_handle(); +} + +/// Read the committed version from a [`CommittedTransaction`] handle. +/// +/// Does not consume the handle; the caller still owns it and must eventually pass it to +/// [`free_committed_transaction`]. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid handle. +#[no_mangle] +pub unsafe extern "C" fn committed_transaction_version( + txn: &Handle, +) -> u64 { + unsafe { txn.as_ref() }.commit_version() +} + +/// Reads the post-commit snapshot, if available. +/// +/// Returns `Some` with a fresh [`SharedSnapshot`] handle if the committed transaction has an +/// associated post-commit snapshot. Returns `None` otherwise. +/// +/// Not every commit path produces a post-commit snapshot (see +/// [`CommittedTransaction::post_commit_snapshot`] for the kernel-side rationale); callers +/// can fall back to building a snapshot via [`get_snapshot_builder`](crate::get_snapshot_builder) +/// in that case. +/// +/// Each `Some` result contains an independent handle that the caller must eventually free with +/// [`free_snapshot`](crate::free_snapshot). Does not consume the input handle; the caller must +/// eventually pass it to [`free_committed_transaction`]. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid handle. +#[no_mangle] +pub unsafe extern "C" fn committed_transaction_post_commit_snapshot( + txn: &Handle, +) -> OptionalValue> { + unsafe { txn.as_ref() } + .post_commit_snapshot() + .map(|snap| Arc::clone(snap).into()) + .into() +} + +// ============================================================================ +// Create Table DDL +// ============================================================================ + +/// A handle representing an exclusive [`CreateTableTransactionBuilder`]. +/// +/// The caller must eventually either call [`create_table_builder_build`] (which consumes the +/// handle and returns a transaction) or [`free_create_table_builder`] (which drops it without +/// creating anything). +#[handle_descriptor(target=CreateTableTransactionBuilder, mutable=true, sized=true)] +pub struct ExclusiveCreateTableBuilder; + +/// Collect `num_columns` column-name string slices into owned `String`s. +/// +/// Returns an empty `Vec` when `num_columns == 0` without dereferencing `columns`, so a null +/// pointer is sound in the empty case. Returns `Err` if any slice is not valid UTF-8. +/// +/// # Safety +/// +/// When `num_columns > 0`, `columns` must point to `num_columns` contiguous, valid +/// [`KernelStringSlice`] values whose backing bytes are readable for the duration of the call. +unsafe fn collect_create_table_columns( + columns: *const KernelStringSlice, + num_columns: usize, +) -> DeltaResult> { + if num_columns == 0 { + return Ok(Vec::new()); } - .into_extern_result(&extern_engine) + let slices = unsafe { std::slice::from_raw_parts(columns, num_columns) }; + slices + .iter() + .map(|slice| { + unsafe { TryFromStringSlice::try_from_slice(slice) }.map(|s: &str| s.to_string()) + }) + .collect() +} + +/// Set a clustered data layout on a [`CreateTableTransactionBuilder`] from an array of top-level +/// clustering column names (in order). Clustering and partitioning are mutually exclusive; the +/// last data-layout call wins. Column validation (existence, stats-eligible types, duplicates) +/// happens later at [`create_table_builder_build`]. +/// +/// This consumes the builder handle and returns a new one. The caller MUST replace their handle +/// pointer with the returned handle. On error, the old builder handle is consumed and gone -- +/// do not free or reuse it. There is no new handle to free either. +/// +/// Only top-level columns are supported through this entry point (each slice is one column name); +/// nested clustering columns must be set on the Rust builder directly. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid builder handle and a valid `engine`. When +/// `num_columns > 0`, `columns` must point to `num_columns` contiguous, valid `KernelStringSlice` +/// values whose backing bytes are readable for the duration of the call; `columns` may be null +/// when `num_columns == 0`. CONSUMES the builder handle unconditionally (even on error). +#[no_mangle] +pub unsafe extern "C" fn create_table_builder_with_clustering_columns( + builder: Handle, + columns: *const KernelStringSlice, + num_columns: usize, + engine: Handle, +) -> ExternResult> { + let engine = unsafe { engine.as_ref() }; + let builder = unsafe { *builder.into_inner() }; + let columns = unsafe { collect_create_table_columns(columns, num_columns) }; + create_table_builder_with_data_layout_impl(builder, columns.map(DataLayout::clustered)) + .into_extern_result(&engine) +} + +/// Set a partitioned data layout on a [`CreateTableTransactionBuilder`] from an array of top-level +/// partition column names (in order). Clustering and partitioning are mutually exclusive; the last +/// data-layout call wins. Column validation (existence, primitive types, subset of schema) happens +/// later at [`create_table_builder_build`]. +/// +/// This consumes the builder handle and returns a new one. The caller MUST replace their handle +/// pointer with the returned handle. On error, the old builder handle is consumed and gone -- +/// do not free or reuse it. There is no new handle to free either. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid builder handle and a valid `engine`. When +/// `num_columns > 0`, `columns` must point to `num_columns` contiguous, valid `KernelStringSlice` +/// values whose backing bytes are readable for the duration of the call; `columns` may be null +/// when `num_columns == 0`. CONSUMES the builder handle unconditionally (even on error). +#[no_mangle] +pub unsafe extern "C" fn create_table_builder_with_partition_columns( + builder: Handle, + columns: *const KernelStringSlice, + num_columns: usize, + engine: Handle, +) -> ExternResult> { + let engine = unsafe { engine.as_ref() }; + let builder = unsafe { *builder.into_inner() }; + let columns = unsafe { collect_create_table_columns(columns, num_columns) }; + create_table_builder_with_data_layout_impl(builder, columns.map(DataLayout::partitioned)) + .into_extern_result(&engine) +} + +/// Shared lowering for the data-layout FFI entry points, extracted from the `unsafe extern` +/// wrappers so it can be unit-tested. `layout` is a `DeltaResult` so a column-parse failure +/// short-circuits here, dropping the already-consumed builder rather than producing a layout. +fn create_table_builder_with_data_layout_impl( + builder: CreateTableTransactionBuilder, + layout: DeltaResult, +) -> DeltaResult> { + Ok(Box::new(builder.with_data_layout(layout?)).into()) +} + +/// Create a new [`CreateTableTransactionBuilder`] for creating a Delta table at the given path. +/// +/// The schema is provided via the engine's visitor callback pattern ([`EngineSchema`]): the +/// kernel allocates a [`KernelSchemaVisitorState`], calls the engine's visitor function to +/// populate it via `visit_field_*` downcalls, then extracts the final schema. +/// +/// The returned builder can be configured with [`create_table_builder_with_table_property`] +/// before building with [`create_table_builder_build`]. The engine is only used for error +/// reporting at this stage. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid `path`, `schema`, `engine_info`, and `engine`. +#[no_mangle] +pub unsafe extern "C" fn get_create_table_builder( + path: KernelStringSlice, + schema: &EngineSchema, + engine_info: KernelStringSlice, + engine: Handle, +) -> ExternResult> { + let engine = unsafe { engine.as_ref() }; + let path = unsafe { TryFromStringSlice::try_from_slice(&path) }; + let info = unsafe { TryFromStringSlice::try_from_slice(&engine_info) }; + get_create_table_builder_impl(path, schema, info).into_extern_result(&engine) +} + +fn get_create_table_builder_impl( + path: DeltaResult<&str>, + schema: &EngineSchema, + engine_info: DeltaResult<&str>, +) -> DeltaResult> { + let mut visitor_state = KernelSchemaVisitorState::default(); + let schema_id = (schema.visitor)(schema.schema, &mut visitor_state); + let schema = extract_kernel_schema(&mut visitor_state, schema_id)?; + let builder = delta_kernel::transaction::create_table::create_table( + path?, + Arc::new(schema), + engine_info?.to_string(), + ); + Ok(Box::new(builder).into()) +} + +/// Add a single table property to a [`CreateTableTransactionBuilder`]. +/// +/// This consumes the builder handle and returns a new one. The caller MUST replace their handle +/// pointer with the returned handle. On error, the old builder handle is consumed and gone -- +/// do not free or reuse it. There is no new handle to free either. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid builder handle, `key`, `value`, and `engine`. +/// CONSUMES the builder handle unconditionally (even on error). +#[no_mangle] +pub unsafe extern "C" fn create_table_builder_with_table_property( + builder: Handle, + key: KernelStringSlice, + value: KernelStringSlice, + engine: Handle, +) -> ExternResult> { + let engine = unsafe { engine.as_ref() }; + let builder = unsafe { *builder.into_inner() }; + let key = unsafe { TryFromStringSlice::try_from_slice(&key) }; + let value = unsafe { TryFromStringSlice::try_from_slice(&value) }; + create_table_builder_with_table_property_impl(builder, key, value).into_extern_result(&engine) +} + +fn create_table_builder_with_table_property_impl( + builder: CreateTableTransactionBuilder, + key: DeltaResult, + value: DeltaResult, +) -> DeltaResult> { + let builder = builder.with_table_properties([(key?, value?)]); + Ok(Box::new(builder).into()) +} + +/// Build a create-table transaction using the default [`FileSystemCommitter`]. Returns a +/// create-table transaction handle that can be used with [`create_table_add_files`], +/// [`create_table_set_data_change`], [`create_table_with_engine_info`], and +/// [`create_table_commit`] to optionally stage initial data before committing. +/// +/// # Safety +/// +/// Caller is responsible for passing valid builder and engine handles. +/// CONSUMES the builder handle -- caller must not use it after this call. +#[no_mangle] +pub unsafe extern "C" fn create_table_builder_build( + builder: Handle, + engine: Handle, +) -> ExternResult> { + let builder = unsafe { *builder.into_inner() }; + let extern_engine = unsafe { engine.as_ref() }; + let committer = Box::new(FileSystemCommitter::new()); + create_table_builder_build_impl(builder, committer, extern_engine) + .into_extern_result(&extern_engine) +} + +/// Build a create-table transaction with a custom committer. Same as +/// [`create_table_builder_build`] but uses the provided committer instead of the default. +/// +/// # Safety +/// +/// Caller is responsible for passing valid handles. +/// CONSUMES both the builder and committer handles -- caller must not use them after this call. +#[no_mangle] +pub unsafe extern "C" fn create_table_builder_build_with_committer( + builder: Handle, + committer: Handle, + engine: Handle, +) -> ExternResult> { + let builder = unsafe { *builder.into_inner() }; + let committer = unsafe { committer.into_inner() }; + let extern_engine = unsafe { engine.as_ref() }; + create_table_builder_build_impl(builder, committer, extern_engine) + .into_extern_result(&extern_engine) +} + +fn create_table_builder_build_impl( + builder: CreateTableTransactionBuilder, + committer: Box, + extern_engine: &dyn ExternEngine, +) -> DeltaResult> { + let engine = extern_engine.engine(); + let create_txn = builder.build(engine.as_ref(), committer)?; + Ok(Box::new(create_txn).into()) +} + +/// Free a [`CreateTableTransactionBuilder`] without building. +/// +/// Use this on failure paths when the builder will not be built. +/// +/// # Safety +/// +/// Caller is responsible for passing a valid handle. +#[no_mangle] +pub unsafe extern "C" fn free_create_table_builder(builder: Handle) { + builder.drop_handle(); +} + +// ============================================================================ +// Remove Files DML +// ============================================================================ + +/// Remove files from a transaction using engine data and a selection vector. +/// +/// The `data` handle is consumed. The selection vector indicates which rows in `data` represent +/// files to remove: nonzero means the row is selected for removal, `0` means it is skipped. +/// If `selection_vector` is null or `selection_vector_len` is 0, all rows are selected. When +/// `selection_vector_len` is 0, the `selection_vector` pointer is not accessed and may be null +/// or any arbitrary value. +/// +/// The `data` and `selection_vector` should be derived from +/// [`scan_metadata_next`](crate::scan::scan_metadata_next): `data` is the engine data batch and +/// `selection_vector` is the scan's selection vector, modified to select only the rows (files) to +/// remove. Selecting rows that were not active in the original scan selection vector produces +/// invalid Remove actions in the commit log. +/// +/// Note: Unlike [`add_files`], this function takes an `engine` handle and returns +/// [`ExternResult`] because the selection vector validation can fail. Returns `true` on +/// success (the value itself is not meaningful). +/// +/// # Safety +/// +/// Caller is responsible for passing valid handles. The `selection_vector` pointer must be valid +/// for `selection_vector_len` bytes, or null. Consumes the `data` handle. Does NOT consume +/// the `txn` handle. +#[no_mangle] +pub unsafe extern "C" fn remove_files( + mut txn: Handle, + data: Handle, + selection_vector: *const u8, + selection_vector_len: usize, + engine: Handle, +) -> ExternResult { + let engine = unsafe { engine.as_ref() }; + let data = unsafe { data.into_inner() }; + let txn = unsafe { txn.as_mut() }; + // empty sv = all rows selected (per FilteredEngineData contract) + let sv = if selection_vector.is_null() || selection_vector_len == 0 { + vec![] + } else { + let raw = unsafe { std::slice::from_raw_parts(selection_vector, selection_vector_len) }; + raw.iter().map(|&b| b != 0).collect() + }; + let result: DeltaResult = (|| { + let filtered = FilteredEngineData::try_new(data, sv)?; + txn.remove_files(filtered); + Ok(true) + })(); + result.into_extern_result(&engine) } #[cfg(test)] mod tests { - use delta_kernel::schema::{DataType, StructField, StructType}; + use std::os::raw::c_void; + use std::sync::Arc; use delta_kernel::arrow::array::{Array, ArrayRef, Int32Array, StringArray, StructArray}; use delta_kernel::arrow::datatypes::Schema as ArrowSchema; use delta_kernel::arrow::ffi::to_ffi; use delta_kernel::arrow::json::reader::ReaderBuilder; use delta_kernel::arrow::record_batch::RecordBatch; - use delta_kernel::engine::arrow_conversion::TryIntoArrow; use delta_kernel::engine::arrow_data::ArrowEngineData; use delta_kernel::object_store::path::Path; - use delta_kernel::object_store::ObjectStore; + use delta_kernel::object_store::{DynObjectStore, ObjectStoreExt as _}; use delta_kernel::parquet::arrow::arrow_writer::ArrowWriter; use delta_kernel::parquet::file::properties::WriterProperties; - - use delta_kernel_ffi::engine_data::get_engine_data; - use delta_kernel_ffi::engine_data::ArrowFFIData; - - use delta_kernel_ffi::ffi_test_utils::{allocate_str, ok_or_panic, recover_string}; + use delta_kernel::schema::{ + schema_ref, try_schema, ColumnMetadataKey, DataType, MetadataValue, SchemaRef, StructField, + StructType, + }; + use delta_kernel::table_features::TableFeature; + use delta_kernel_ffi::engine_data::{get_engine_data, ArrowFFIData}; + use delta_kernel_ffi::error::KernelError; + use delta_kernel_ffi::ffi_test_utils::{ + allocate_err, allocate_str, assert_extern_result_error_with_message, build_snapshot, + engine_handle_for_store, ok_or_panic, recover_error, recover_string, + }; use delta_kernel_ffi::tests::get_default_engine; - - use crate::{free_engine, free_schema, kernel_string_slice}; - use write_context::{free_write_context, get_write_context, get_write_path, get_write_schema}; - - use test_utils::{set_json_value, setup_test_tables, test_read}; - use itertools::Itertools; - use serde_json::json; - use serde_json::Deserializer; + use rstest::rstest; + use serde_json::{json, Deserializer}; + use tempfile::tempdir; + use test_utils::delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor; + use test_utils::delta_kernel_default_engine::DefaultEngine; + use test_utils::{set_json_value, setup_test_tables, test_read}; + use write_context::{ + create_table_get_unpartitioned_write_context, free_write_context, get_logical_to_physical, + get_partitioned_write_context, get_physical_write_schema, get_unpartitioned_write_context, + get_write_dir, get_write_path, get_write_schema, resolve_file_path, visit_partition_values, + SharedWriteContext, + }; - use std::sync::Arc; + use super::*; + use crate::engine_funcs::{free_expression_evaluator, new_expression_evaluator}; + use crate::expressions::free_kernel_expression; + use crate::schema_visitor::{ + visit_field_integer, visit_field_long, visit_field_string, visit_field_struct, + }; + use crate::{ + free_engine, free_schema, free_snapshot, kernel_string_slice, logical_schema, version, + KernelStringSlice, NullableCvoid, OptionalValue, + }; const ZERO_UUID: &str = "00000000-0000-0000-0000-000000000000"; - use super::*; + type LocalTestTables = Vec<( + Url, + DefaultEngine, + Arc, + &'static str, + )>; + + /// Create v1.1 and v3.7 test tables on the local filesystem under a fresh temp directory. + /// + /// Keep the returned [`tempfile::TempDir`] alive for the duration of the test. + async fn setup_local_test_tables( + schema: SchemaRef, + partition_columns: &[&str], + table_base_name: &str, + ) -> Result<(tempfile::TempDir, LocalTestTables), Box> { + let tmp_test_dir = tempdir()?; + let tmp_dir_url = Url::from_directory_path(tmp_test_dir.path()).unwrap(); + let tables = setup_test_tables( + schema, + partition_columns, + Some(&tmp_dir_url), + table_base_name, + ) + .await?; + Ok((tmp_test_dir, tables)) + } - use tempfile::tempdir; + /// Reads the committed version from a [`Handle`] and + /// frees the handle. + /// + /// Useful for tests that only need to assert on the version. Tests that also need + /// the post-commit snapshot should call the accessors directly. + /// + /// # Safety + /// + /// Caller asserts the handle is valid (i.e. produced by [`commit`] / + /// [`create_table_commit`] and not previously freed). + unsafe fn version_and_free(committed: Handle) -> u64 { + let version = unsafe { committed_transaction_version(&committed) }; + unsafe { free_committed_transaction(committed) }; + version + } fn check_txn_id_exists(commit_info: &serde_json::Value) { commit_info["txnId"] @@ -276,8 +932,33 @@ mod tests { // writer must be closed to write footer let res = writer.close().unwrap(); - let file_size_bytes = std::fs::metadata(&full_path)?.len(); + let num_rows = res.file_metadata().num_rows(); + create_file_metadata(file_path, file_size_bytes, num_rows, metadata_schema) + } + + /// Write `batch` as parquet into `store` at `file_path` and return its Add-action metadata. + /// + /// The store-based counterpart to [`write_parquet_file`], for tests that must avoid + /// `std::fs` so they can run under Miri. + async fn put_parquet_file( + store: &Arc, + table_url: &Url, + file_path: &str, + batch: &RecordBatch, + metadata_schema: ArrowSchema, + ) -> Result> { + let mut buf = Vec::new(); + let props = WriterProperties::builder().build(); + let mut writer = ArrowWriter::try_new(&mut buf, batch.schema(), Some(props))?; + writer.write(batch)?; + let res = writer.close()?; + + let file_size_bytes = buf.len() as u64; + let url = table_url.join(file_path)?; + let path = Path::from_url_path(url.path())?; + store.put(&path, buf.into()).await?; + create_file_metadata( file_path, file_size_bytes, @@ -287,28 +968,24 @@ mod tests { } #[tokio::test] - #[cfg_attr(miri, ignore)] // FIXME: re-enable miri (can't call foreign function `linkat` on OS `linux`) + // Keeps local storage: `canonicalize` on the FFI-returned write path has no in-memory + // equivalent. + // DO NOT ADD NEW `unsafe` HERE (unsafe not already run by a Miri test): Miri never runs this + // test, so unsafe unique to it goes unchecked for undefined behavior. Put it in a test that + // runs under Miri. + #[cfg_attr( + miri, + ignore = "local-filesystem commit calls `linkat`, unsupported under Miri" + )] async fn test_basic_append() -> Result<(), Box> { let schema = Arc::new(StructType::try_new(vec![ StructField::nullable("number", DataType::INTEGER), StructField::nullable("string", DataType::STRING), ])?); - // Create a temporary local directory for use during this test - let tmp_test_dir = tempdir()?; - let tmp_dir_local_url = Url::from_directory_path(tmp_test_dir.path()).unwrap(); - // TODO: test with partitions - let partition_columns = vec![]; - - for (table_url, _engine, store, _table_name) in setup_test_tables( - schema, - &partition_columns, - Some(&tmp_dir_local_url), - "test_table", - ) - .await? - { + let (_tmp_test_dir, tables) = setup_local_test_tables(schema, &[], "test_table").await?; + for (table_url, _engine, store, _table_name) in tables { let table_path = table_url.to_file_path().unwrap(); let table_path_str = table_path.to_str().unwrap(); let engine = get_default_engine(table_path_str); @@ -322,7 +999,7 @@ mod tests { // Add engine info let engine_info = "default_engine"; let engine_info_kernel_string = kernel_string_slice!(engine_info); - let txn_with_engine_info = unsafe { + let txn = unsafe { ok_or_panic(with_engine_info( txn, engine_info_kernel_string, @@ -330,7 +1007,20 @@ mod tests { )) }; - let write_context = unsafe { get_write_context(txn_with_engine_info.shallow_copy()) }; + // Add the operation + let operation = "WRITE"; + let operation_kernel_string = kernel_string_slice!(operation); + let txn = unsafe { + ok_or_panic(with_operation( + txn, + operation_kernel_string, + engine.shallow_copy(), + )) + }; + + let write_context = ok_or_panic(unsafe { + get_unpartitioned_write_context(txn.shallow_copy(), engine.shallow_copy()) + }); // Ensure we get the correct schema let write_schema = unsafe { get_write_schema(write_context.shallow_copy()) }; @@ -371,12 +1061,7 @@ mod tests { ), ]) .unwrap(); - let parquet_schema = unsafe { - txn_with_engine_info - .shallow_copy() - .as_ref() - .add_files_schema() - }; + let parquet_schema = unsafe { txn.shallow_copy().as_ref().add_files_schema() }; let file_info = write_parquet_file( table_path_str, "my_file.parquet", @@ -385,16 +1070,13 @@ mod tests { )?; let file_info_engine_data = ok_or_panic(unsafe { - get_engine_data( - file_info.array, - &file_info.schema, - crate::ffi_test_utils::allocate_err, - ) + get_engine_data(file_info.array, &file_info.schema, allocate_err) }); - unsafe { add_files(txn_with_engine_info.shallow_copy(), file_info_engine_data) }; + unsafe { add_files(txn.shallow_copy(), file_info_engine_data) }; - ok_or_panic(unsafe { commit(txn_with_engine_info, engine.shallow_copy()) }); + let committed = ok_or_panic(unsafe { commit(txn, engine.shallow_copy()) }); + unsafe { free_committed_transaction(committed) }; // Confirm that our commit is what we expect let commit1_url = table_url @@ -421,7 +1103,7 @@ mod tests { "commitInfo": { "timestamp": 0, "engineInfo": "default_engine", - "operation": "UNKNOWN", + "operation": "WRITE", "kernelVersion": format!("v{}", env!("CARGO_PKG_VERSION")), "operationParameters": {}, "txnId": ZERO_UUID @@ -439,30 +1121,649 @@ mod tests { }), ]; - assert_eq!(parsed_commits, expected_commit); + assert_eq!(parsed_commits, expected_commit); + + // Confirm that the data matches what we appended + let test_batch = ArrowEngineData::from(batch); + test_read(&test_batch, &table_url, unsafe { engine.as_ref().engine() })?; + + unsafe { free_schema(write_schema) }; + unsafe { free_write_context(write_context) }; + unsafe { free_engine(engine) }; + } + + Ok(()) + } + + /// Callback for [`visit_partition_values`] that collects each entry into a + /// `Vec<(key, value, is_null)>` passed via the engine context pointer. + extern "C" fn collect_partition_value( + engine_context: NullableCvoid, + key: KernelStringSlice, + value: KernelStringSlice, + is_null: bool, + ) { + let collected = + unsafe { &mut *(engine_context.unwrap().as_ptr() as *mut Vec<(String, String, bool)>) }; + let key = unsafe { String::try_from_slice(&key) }.unwrap(); + let value = unsafe { String::try_from_slice(&value) }.unwrap(); + collected.push((key, value, is_null)); + } + + #[tokio::test] + // Keeps local storage: the test creates the Hive partition directory on disk, which an object + // store does not have. + // DO NOT ADD NEW `unsafe` HERE (unsafe not already run by a Miri test): Miri never runs this + // test, so unsafe unique to it goes unchecked for undefined behavior. Put it in a test that + // runs under Miri. + #[cfg_attr( + miri, + ignore = "local-filesystem commit calls `linkat`, unsupported under Miri" + )] + async fn test_partitioned_append() -> Result<(), Box> { + // Partition column `part` is listed last in the schema; the physical write schema must + // exclude it (CM=none, partition columns are not materialized). + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("number", DataType::INTEGER), + StructField::nullable("string", DataType::STRING), + StructField::nullable("part", DataType::INTEGER), + ])?); + + let (_tmp_test_dir, tables) = + setup_local_test_tables(schema, &["part"], "test_partitioned_table").await?; + for (table_url, _engine, store, _table_name) in tables { + let table_path = table_url.to_file_path().unwrap(); + let table_path_str = table_path.to_str().unwrap(); + let engine = get_default_engine(table_path_str); + + let txn = ok_or_panic(unsafe { + transaction(kernel_string_slice!(table_path_str), engine.shallow_copy()) + }); + unsafe { set_data_change(txn.shallow_copy(), true) }; + let engine_info = "default_engine"; + let txn = unsafe { + ok_or_panic(with_engine_info( + txn, + kernel_string_slice!(engine_info), + engine.shallow_copy(), + )) + }; + let operation = "WRITE"; + let txn = unsafe { + ok_or_panic(with_operation( + txn, + kernel_string_slice!(operation), + engine.shallow_copy(), + )) + }; + + // Build the partition values map: part = 100. + let part_name = "part"; + let partition_values = partition_value_map_new(); + let inserted = ok_or_panic(unsafe { + partition_value_map_insert_int( + partition_values.shallow_copy(), + kernel_string_slice!(part_name), + 100, + engine.shallow_copy(), + ) + }); + assert!(inserted); + + let write_context = ok_or_panic(unsafe { + get_partitioned_write_context( + txn.shallow_copy(), + partition_values, + engine.shallow_copy(), + ) + }); + + // Physical write schema excludes the partition column. + let write_schema = unsafe { get_physical_write_schema(write_context.shallow_copy()) }; + let write_schema_ref = unsafe { write_schema.as_ref() }; + assert_eq!(write_schema_ref.num_fields(), 2); + assert_eq!(write_schema_ref.field_at_index(0).unwrap().name, "number"); + assert_eq!(write_schema_ref.field_at_index(1).unwrap().name, "string"); + + // The write directory carries the Hive-style partition prefix. + let write_dir_str = recover_string( + unsafe { get_write_dir(write_context.shallow_copy(), allocate_str) }.unwrap(), + ); + let write_dir_url = Url::parse(&write_dir_str)?; + assert!( + write_dir_url.path().ends_with("part=100/"), + "unexpected write dir: {write_dir_str}" + ); + + // The serialized partition values surface the physical name and value. + let mut collected: Vec<(String, String, bool)> = Vec::new(); + let ctx = std::ptr::NonNull::new( + &mut collected as *mut Vec<(String, String, bool)> as *mut std::ffi::c_void, + ); + unsafe { + visit_partition_values(write_context.shallow_copy(), ctx, collect_partition_value) + }; + assert_eq!( + collected, + vec![("part".to_string(), "100".to_string(), false)] + ); + + // Write the parquet data (partition column excluded) into the partition directory. + let batch = RecordBatch::try_from_iter(vec![ + ( + "number", + Arc::new(Int32Array::from(vec![1, 2, 3])) as ArrayRef, + ), + ( + "string", + Arc::new(StringArray::from(vec!["a", "b", "c"])) as ArrayRef, + ), + ]) + .unwrap(); + let dir_path = write_dir_url.to_file_path().unwrap(); + std::fs::create_dir_all(&dir_path)?; + let file_name = "my_file.parquet"; + let file_fs_path = dir_path.join(file_name); + let parquet_file = std::fs::File::create(&file_fs_path)?; + let mut writer = ArrowWriter::try_new( + parquet_file, + batch.schema(), + Some(WriterProperties::builder().build()), + )?; + writer.write(&batch)?; + let parquet_meta = writer.close()?; + let num_rows = parquet_meta.file_metadata().num_rows(); + let size = std::fs::metadata(&file_fs_path)?.len(); + + // Resolve the relative add.path from the absolute file URL. + let file_url = write_dir_url.join(file_name)?; + let file_url_str = file_url.as_str(); + let add_path = recover_string( + ok_or_panic(unsafe { + resolve_file_path( + write_context.shallow_copy(), + kernel_string_slice!(file_url_str), + allocate_str, + engine.shallow_copy(), + ) + }) + .unwrap(), + ); + assert_eq!(add_path, "part=100/my_file.parquet"); + + // Build the add metadata with the populated partitionValues. + let metadata_json = format!( + r#"{{"path":"{add_path}", "partitionValues": {{"part":"100"}}, "size": {size}, "modificationTime": 0, "stats": {{"numRecords": {num_rows}}}}}"#, + ); + let metadata_schema = unsafe { txn.shallow_copy().as_ref().add_files_schema() } + .as_ref() + .try_into_arrow()?; + let file_info = create_arrow_ffi_from_json(metadata_schema, &metadata_json)?; + let file_info_engine_data = ok_or_panic(unsafe { + get_engine_data(file_info.array, &file_info.schema, allocate_err) + }); + unsafe { add_files(txn.shallow_copy(), file_info_engine_data) }; + + let committed = ok_or_panic(unsafe { commit(txn, engine.shallow_copy()) }); + unsafe { free_committed_transaction(committed) }; + + // The commit's add action records the partition values and Hive-style path. + let commit1_url = table_url + .join("_delta_log/00000000000000000001.json") + .unwrap(); + let commit1 = store + .get(&Path::from_url_path(commit1_url.path()).unwrap()) + .await?; + let parsed_commits: Vec<_> = Deserializer::from_slice(&commit1.bytes().await?) + .into_iter::() + .try_collect()?; + let add = &parsed_commits[1]["add"]; + assert_eq!(add["path"], json!("part=100/my_file.parquet")); + assert_eq!(add["partitionValues"], json!({ "part": "100" })); + + // The read reconstructs the partition column for every row. + let expected = RecordBatch::try_from_iter(vec![ + ( + "number", + Arc::new(Int32Array::from(vec![1, 2, 3])) as ArrayRef, + ), + ( + "string", + Arc::new(StringArray::from(vec!["a", "b", "c"])) as ArrayRef, + ), + ( + "part", + Arc::new(Int32Array::from(vec![100, 100, 100])) as ArrayRef, + ), + ]) + .unwrap(); + let expected_engine_data = ArrowEngineData::from(expected); + test_read(&expected_engine_data, &table_url, unsafe { + engine.as_ref().engine() + })?; + + unsafe { free_schema(write_schema) }; + unsafe { free_write_context(write_context) }; + unsafe { free_engine(engine) }; + } + + Ok(()) + } + + #[tokio::test] + async fn test_partitioned_write_context_rejects_unpartitioned_table( + ) -> Result<(), Box> { + let schema = Arc::new(StructType::try_new(vec![StructField::nullable( + "number", + DataType::INTEGER, + )])?); + let tables = setup_test_tables(schema, &[], None, "test_unpartitioned").await?; + + for (table_url, _engine, store, _table_name) in tables { + let table_url_str = table_url.as_str(); + let engine = engine_handle_for_store(store); + let txn = ok_or_panic(unsafe { + transaction(kernel_string_slice!(table_url_str), engine.shallow_copy()) + }); + + // Supplying partition values for a non-partitioned table is an error, and the call + // still consumes the map handle. + let partition_values = partition_value_map_new(); + let name = "nope"; + let _ = ok_or_panic(unsafe { + partition_value_map_insert_int( + partition_values.shallow_copy(), + kernel_string_slice!(name), + 1, + engine.shallow_copy(), + ) + }); + let result = unsafe { + get_partitioned_write_context( + txn.shallow_copy(), + partition_values, + engine.shallow_copy(), + ) + }; + let err = match result { + ExternResult::Err(e) => unsafe { recover_error(e) }, + ExternResult::Ok(_) => panic!("expected error for unpartitioned table"), + }; + assert!( + err.message.contains("not partitioned"), + "unexpected error: {}", + err.message + ); + unsafe { free_transaction(txn) }; + unsafe { free_engine(engine) }; + } + + Ok(()) + } + + #[tokio::test] + async fn test_visit_partition_values_surfaces_null() -> Result<(), Box> { + // A null partition value must surface across the visitor as `is_null = true` with an + // empty value slice (the documented C contract). + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("number", DataType::INTEGER), + StructField::nullable("part", DataType::INTEGER), + ])?); + let tables = setup_test_tables(schema, &["part"], None, "test_null_partition").await?; + + for (table_url, _engine, store, _table_name) in tables { + let table_url_str = table_url.as_str(); + let engine = engine_handle_for_store(store); + let txn = ok_or_panic(unsafe { + transaction(kernel_string_slice!(table_url_str), engine.shallow_copy()) + }); + + let partition_values = partition_value_map_new(); + let part_name = "part"; + // NullTypeTag::Integer == 3. + let _ = ok_or_panic(unsafe { + partition_value::partition_value_map_insert_null( + partition_values.shallow_copy(), + kernel_string_slice!(part_name), + crate::expressions::kernel_visitor::NullTypeTag::Integer as u8, + 0, + 0, + engine.shallow_copy(), + ) + }); + let write_context = ok_or_panic(unsafe { + get_partitioned_write_context( + txn.shallow_copy(), + partition_values, + engine.shallow_copy(), + ) + }); + + let mut collected: Vec<(String, String, bool)> = Vec::new(); + let ctx = std::ptr::NonNull::new( + &mut collected as *mut Vec<(String, String, bool)> as *mut std::ffi::c_void, + ); + unsafe { + visit_partition_values(write_context.shallow_copy(), ctx, collect_partition_value) + }; + assert_eq!(collected, vec![("part".to_string(), String::new(), true)]); + + unsafe { free_write_context(write_context) }; + unsafe { free_transaction(txn) }; + unsafe { free_engine(engine) }; + } + + Ok(()) + } + + #[tokio::test] + async fn test_visit_partition_values_is_sorted_by_key() -> Result<(), Box> + { + // Multiple partition columns must be visited in deterministic (sorted) key order, + // regardless of insertion order or the underlying HashMap layout. + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("number", DataType::INTEGER), + StructField::nullable("region", DataType::STRING), + StructField::nullable("year", DataType::INTEGER), + ])?); + let tables = + setup_test_tables(schema, &["year", "region"], None, "test_multi_partition").await?; + + for (table_url, _engine, store, _table_name) in tables { + let table_url_str = table_url.as_str(); + let engine = engine_handle_for_store(store); + let txn = ok_or_panic(unsafe { + transaction(kernel_string_slice!(table_url_str), engine.shallow_copy()) + }); - // Confirm that the data matches what we appended - let test_batch = ArrowEngineData::from(batch); - test_read(&test_batch, &table_url, unsafe { engine.as_ref().engine() })?; + let partition_values = partition_value_map_new(); + // Insert in a different order than the expected sorted output. + let year = "year"; + let _ = ok_or_panic(unsafe { + partition_value_map_insert_int( + partition_values.shallow_copy(), + kernel_string_slice!(year), + 2024, + engine.shallow_copy(), + ) + }); + let region = "region"; + let region_val = "US"; + let _ = ok_or_panic(unsafe { + partition_value_map_insert_string( + partition_values.shallow_copy(), + kernel_string_slice!(region), + kernel_string_slice!(region_val), + engine.shallow_copy(), + ) + }); + let write_context = ok_or_panic(unsafe { + get_partitioned_write_context( + txn.shallow_copy(), + partition_values, + engine.shallow_copy(), + ) + }); + + let mut collected: Vec<(String, String, bool)> = Vec::new(); + let ctx = std::ptr::NonNull::new( + &mut collected as *mut Vec<(String, String, bool)> as *mut std::ffi::c_void, + ); + unsafe { + visit_partition_values(write_context.shallow_copy(), ctx, collect_partition_value) + }; + let keys: Vec<&str> = collected.iter().map(|(k, _, _)| k.as_str()).collect(); + assert_eq!(keys, vec!["region", "year"]); - unsafe { free_schema(write_schema) }; unsafe { free_write_context(write_context) }; + unsafe { free_transaction(txn) }; unsafe { free_engine(engine) }; } Ok(()) } - #[cfg(feature = "uc-catalog")] + /// Read the commit log at `version` and return the `domainMetadata` action JSON. + async fn read_domain_metadata_action( + store: &DynObjectStore, + table_url: &Url, + version: u64, + ) -> serde_json::Value { + let path = format!("_delta_log/{version:020}.json"); + let commit_url = table_url.join(&path).unwrap(); + let data = store + .get(&Path::from_url_path(commit_url.path()).unwrap()) + .await + .unwrap(); + let actions: Vec = + Deserializer::from_slice(&data.bytes().await.unwrap()) + .into_iter::() + .try_collect() + .unwrap(); + actions + .into_iter() + .find(|a| a.get("domainMetadata").is_some()) + .expect("commit should contain a domainMetadata action") + } + + /// Create a table with the `domainMetadata` writer feature enabled and return the table + /// URL, object store, and FFI engine handle. + async fn setup_domain_metadata_table( + name: &str, + ) -> Result<(Url, Arc, Handle), Box> + { + let schema = Arc::new(try_schema! { nullable "id": INTEGER }?); + let (store, _test_engine, table_location) = test_utils::engine_store_setup(name, None); + let table_url = test_utils::create_table( + store.clone(), + table_location, + schema, + &[], + true, + vec![], + vec!["domainMetadata"], + ) + .await?; + let engine = engine_handle_for_store(Arc::clone(&store)); + Ok((table_url, store, engine)) + } + + #[tokio::test] + async fn test_domain_metadata_add_and_remove() -> Result<(), Box> { + let (table_url, store, engine) = setup_domain_metadata_table("test_dm").await?; + let table_path_str = table_url.as_str(); + + // === Transaction 1: add domain metadata === + let txn = ok_or_panic(unsafe { + transaction(kernel_string_slice!(table_path_str), engine.shallow_copy()) + }); + unsafe { set_data_change(txn.shallow_copy(), false) }; + + let domain = "testDomain"; + let configuration = r#"{"key": "value"}"#; + let txn = ok_or_panic(unsafe { + with_domain_metadata( + txn, + kernel_string_slice!(domain), + kernel_string_slice!(configuration), + engine.shallow_copy(), + ) + }); + + let committed = ok_or_panic(unsafe { commit(txn, engine.shallow_copy()) }); + let version = unsafe { version_and_free(committed) }; + assert_eq!(version, 1); + + let dm = read_domain_metadata_action(&*store, &table_url, 1).await; + assert_eq!(dm["domainMetadata"]["domain"], "testDomain"); + assert_eq!(dm["domainMetadata"]["configuration"], r#"{"key": "value"}"#); + assert_eq!(dm["domainMetadata"]["removed"], false); + + // === Transaction 2: remove domain metadata === + let txn = ok_or_panic(unsafe { + transaction(kernel_string_slice!(table_path_str), engine.shallow_copy()) + }); + unsafe { set_data_change(txn.shallow_copy(), false) }; + + let txn = ok_or_panic(unsafe { + with_domain_metadata_removed(txn, kernel_string_slice!(domain), engine.shallow_copy()) + }); + + let committed = ok_or_panic(unsafe { commit(txn, engine.shallow_copy()) }); + let version = unsafe { version_and_free(committed) }; + assert_eq!(version, 2); + + let dm = read_domain_metadata_action(&*store, &table_url, 2).await; + assert_eq!(dm["domainMetadata"]["domain"], "testDomain"); + assert_eq!(dm["domainMetadata"]["removed"], true); + assert_eq!(dm["domainMetadata"]["configuration"], r#"{"key": "value"}"#); + + unsafe { free_engine(engine) }; + Ok(()) + } + + #[tokio::test] + async fn test_domain_metadata_system_domain_rejected_at_commit( + ) -> Result<(), Box> { + let (table_url, _store, engine) = setup_domain_metadata_table("test_dm_sys").await?; + let table_path_str = table_url.as_str(); + + // with_domain_metadata succeeds (validation is lazy), but commit should fail + let txn = ok_or_panic(unsafe { + transaction(kernel_string_slice!(table_path_str), engine.shallow_copy()) + }); + unsafe { set_data_change(txn.shallow_copy(), false) }; + + let sys_domain = "delta.system"; + let config = "config"; + let txn = ok_or_panic(unsafe { + with_domain_metadata( + txn, + kernel_string_slice!(sys_domain), + kernel_string_slice!(config), + engine.shallow_copy(), + ) + }); + + let result = unsafe { commit(txn, engine.shallow_copy()) }; + assert_extern_result_error_with_message( + result, + KernelError::GenericError, + Some("Generic delta kernel error: Cannot modify domains that start with 'delta.' as those are system controlled"), + ); + + unsafe { free_engine(engine) }; + Ok(()) + } + + #[tokio::test] + async fn test_domain_metadata_duplicate_domain_rejected_at_commit( + ) -> Result<(), Box> { + let (table_url, _store, engine) = setup_domain_metadata_table("test_dm_dup").await?; + let table_path_str = table_url.as_str(); + + // Adding the same domain twice should cause commit to fail + let txn = ok_or_panic(unsafe { + transaction(kernel_string_slice!(table_path_str), engine.shallow_copy()) + }); + unsafe { set_data_change(txn.shallow_copy(), false) }; + + let dup_domain = "dup"; + let config_a = "a"; + let config_b = "b"; + let txn = ok_or_panic(unsafe { + with_domain_metadata( + txn, + kernel_string_slice!(dup_domain), + kernel_string_slice!(config_a), + engine.shallow_copy(), + ) + }); + let txn = ok_or_panic(unsafe { + with_domain_metadata( + txn, + kernel_string_slice!(dup_domain), + kernel_string_slice!(config_b), + engine.shallow_copy(), + ) + }); + + let result = unsafe { commit(txn, engine.shallow_copy()) }; + assert_extern_result_error_with_message( + result, + KernelError::GenericError, + Some("Generic delta kernel error: Metadata for domain dup already specified in this transaction"), + ); + + unsafe { free_engine(engine) }; + Ok(()) + } + + #[tokio::test] + async fn test_domain_metadata_rejected_without_feature( + ) -> Result<(), Box> { + let tmp_test_dir = tempdir()?; + let tmp_dir_url = Url::from_directory_path(tmp_test_dir.path()).unwrap(); + + // Create a table WITHOUT the domainMetadata writer feature (v1/v1 protocol) + let schema = Arc::new(try_schema! { nullable "id": INTEGER }?); + let (store, _test_engine, table_location) = + test_utils::engine_store_setup("test_dm_no_feature", Some(&tmp_dir_url)); + let table_url = test_utils::create_table( + store.clone(), + table_location, + schema, + &[], + false, + vec![], + vec![], + ) + .await?; + let table_path = table_url.to_file_path().unwrap(); + let table_path_str = table_path.to_str().unwrap(); + let engine = get_default_engine(table_path_str); + + let txn = ok_or_panic(unsafe { + transaction(kernel_string_slice!(table_path_str), engine.shallow_copy()) + }); + unsafe { set_data_change(txn.shallow_copy(), false) }; + + let domain = "myDomain"; + let config = "config"; + let txn = ok_or_panic(unsafe { + with_domain_metadata( + txn, + kernel_string_slice!(domain), + kernel_string_slice!(config), + engine.shallow_copy(), + ) + }); + + let result = unsafe { commit(txn, engine.shallow_copy()) }; + assert_extern_result_error_with_message( + result, + KernelError::UnsupportedError, + Some("Unsupported: Domain metadata operations require writer version 7 and the 'domainMetadata' writer feature"), + ); + + unsafe { free_engine(engine) }; + Ok(()) + } + + #[cfg(feature = "delta-kernel-unity-catalog")] #[tokio::test(flavor = "multi_thread", worker_threads = 2)] - #[cfg_attr(miri, ignore)] async fn test_transaction_with_uc_committer() -> Result<(), Box> { - use crate::uc_catalog::{ - free_uc_commit_client, get_uc_commit_client, get_uc_committer, - tests::{cast_test_context, get_test_context, recover_test_context}, - CommitRequest, + use delta_kernel_ffi::{ + get_snapshot_builder, snapshot_builder_build, snapshot_builder_set_max_catalog_version, + }; + + use crate::delta_kernel_unity_catalog::tests::{ + cast_test_context, get_test_context, recover_test_context, + }; + use crate::delta_kernel_unity_catalog::{ + free_uc_commit_client, get_uc_commit_client, get_uc_committer, CommitRequest, }; - use crate::{Handle, NullableCvoid, OptionalValue}; + use crate::{snapshot_publish_with_committer, Handle, NullableCvoid, OptionalValue}; #[no_mangle] extern "C" fn test_uc_commit( @@ -473,9 +1774,8 @@ mod tests { context.commit_called = true; let table_id = unsafe { String::try_from_slice(&request.table_id).unwrap() }; - let table_uri = unsafe { String::try_from_slice(&request.table_uri).unwrap() }; - context.last_commit_request = Some((table_id.clone(), table_uri.clone())); + context.last_commit_table_id = Some(table_id.clone()); // Capture the staged commit file name if present if let OptionalValue::Some(commit_info) = request.commit_info { @@ -491,43 +1791,57 @@ mod tests { StructField::nullable("string", DataType::STRING), ])); - let tmp_test_dir = tempdir()?; - let tmp_dir_local_url = Url::from_directory_path(tmp_test_dir.path()).unwrap(); - let partition_columns = vec![]; - - for (table_url, _engine, store, _table_name) in setup_test_tables( + // Create a catalog-managed table so UCCommitter (a catalog committer) is allowed. + let (store, _test_engine, table_location) = + test_utils::engine_store_setup("test_uc_table", None); + let table_url = test_utils::create_table( + store.clone(), + table_location, schema, - &partition_columns, - Some(&tmp_dir_local_url), - "test_uc_table", + &[], + true, // use v3/v7 protocol + vec!["catalogManaged", "vacuumProtocolCheck"], + vec!["inCommitTimestamp", "catalogManaged", "vacuumProtocolCheck"], ) - .await? + .await?; + { - let table_path = table_url.to_file_path().unwrap(); - let table_path_str = table_path.to_str().unwrap(); - let engine = get_default_engine(table_path_str); + let table_path_str = table_url.as_str(); + let engine = engine_handle_for_store(Arc::clone(&store)); let snapshot = unsafe { - ok_or_panic(crate::snapshot( + let mut ptr = ok_or_panic(get_snapshot_builder( kernel_string_slice!(table_path_str), engine.shallow_copy(), - )) + )); + snapshot_builder_set_max_catalog_version(&mut ptr, 0); + ok_or_panic(snapshot_builder_build(ptr)) }; let context = get_test_context(false); let uc_client = unsafe { get_uc_commit_client(context, test_uc_commit) }; - let table_id = "foo"; + let table_id = "test_id"; + let catalog = "test_catalog"; + let schema = "test_schema"; + let table_name = "test_table"; let uc_committer = unsafe { ok_or_panic(get_uc_committer( uc_client.shallow_copy(), kernel_string_slice!(table_id), - crate::ffi_test_utils::allocate_err, + kernel_string_slice!(catalog), + kernel_string_slice!(schema), + kernel_string_slice!(table_name), + allocate_err, )) }; let txn = ok_or_panic(unsafe { - transaction_with_committer(snapshot, engine.shallow_copy(), uc_committer) + transaction_with_committer( + snapshot.shallow_copy(), + engine.shallow_copy(), + uc_committer, + ) }); unsafe { set_data_change(txn.shallow_copy(), false) }; @@ -541,7 +1855,12 @@ mod tests { )) }; - let write_context = unsafe { get_write_context(txn_with_engine_info.shallow_copy()) }; + let write_context = ok_or_panic(unsafe { + get_unpartitioned_write_context( + txn_with_engine_info.shallow_copy(), + engine.shallow_copy(), + ) + }); let batch = RecordBatch::try_from_iter(vec![ ( @@ -561,19 +1880,17 @@ mod tests { .as_ref() .add_files_schema() }; - let file_info = write_parquet_file( - table_path_str, + let file_info = put_parquet_file( + &store, + &table_url, "uc_test_file.parquet", &batch, parquet_schema.as_ref().try_into_arrow()?, - )?; + ) + .await?; let file_info_engine_data = ok_or_panic(unsafe { - get_engine_data( - file_info.array, - &file_info.schema, - crate::ffi_test_utils::allocate_err, - ) + get_engine_data(file_info.array, &file_info.schema, allocate_err) }); unsafe { add_files(txn_with_engine_info.shallow_copy(), file_info_engine_data) }; @@ -581,7 +1898,87 @@ mod tests { let commit_result = unsafe { commit(txn_with_engine_info, engine.shallow_copy()) }; // UC committer returns success from our mock callback - assert!(commit_result.is_ok(), "Commit should succeed"); + let committed = ok_or_panic(commit_result); + let post_commit_snapshot = + match unsafe { committed_transaction_post_commit_snapshot(&committed) } { + OptionalValue::Some(snapshot) => snapshot, + OptionalValue::None => { + panic!("UC commit should produce a post-commit snapshot") + } + }; + assert_eq!(unsafe { version(post_commit_snapshot.shallow_copy()) }, 1); + + let publish_committer = unsafe { + ok_or_panic(get_uc_committer( + uc_client.shallow_copy(), + kernel_string_slice!(table_id), + kernel_string_slice!(catalog), + kernel_string_slice!(schema), + kernel_string_slice!(table_name), + allocate_err, + )) + }; + let published_snapshot = ok_or_panic(unsafe { + snapshot_publish_with_committer( + post_commit_snapshot.shallow_copy(), + publish_committer, + engine.shallow_copy(), + ) + }); + assert_eq!(unsafe { version(published_snapshot.shallow_copy()) }, 1); + assert!( + !std::ptr::eq(unsafe { post_commit_snapshot.as_ref() }, unsafe { + published_snapshot.as_ref() + },), + "first publish should return a new snapshot carrying published state" + ); + + let published_commit_url = table_url + .join("_delta_log/00000000000000000001.json") + .unwrap(); + let published_commit = store + .get(&Path::from_url_path(published_commit_url.path()).unwrap()) + .await?; + let published_content = published_commit.bytes().await?; + let published_actions: Vec<_> = Deserializer::from_slice(&published_content) + .into_iter::() + .try_collect()?; + assert!( + published_actions.iter().any(|a| a.get("add").is_some()), + "Published commit should contain the add action" + ); + + // Second publish is a no-op when all staged commits are already published. + // Publish consumes the committer, so mint a fresh one for the second call. + let republish_committer = unsafe { + ok_or_panic(get_uc_committer( + uc_client.shallow_copy(), + kernel_string_slice!(table_id), + kernel_string_slice!(catalog), + kernel_string_slice!(schema), + kernel_string_slice!(table_name), + allocate_err, + )) + }; + let republished_snapshot = ok_or_panic(unsafe { + snapshot_publish_with_committer( + published_snapshot.shallow_copy(), + republish_committer, + engine.shallow_copy(), + ) + }); + assert!( + std::ptr::eq(unsafe { published_snapshot.as_ref() }, unsafe { + republished_snapshot.as_ref() + },), + "second publish should short-circuit and return the same snapshot" + ); + + unsafe { free_snapshot(republished_snapshot) }; + unsafe { free_snapshot(published_snapshot) }; + unsafe { free_snapshot(post_commit_snapshot) }; + unsafe { free_snapshot(snapshot) }; + unsafe { free_committed_transaction(committed) }; let context = recover_test_context(context).unwrap(); @@ -592,9 +1989,9 @@ mod tests { { // scope so we don't hold mutex across the await lower down - let (last_table_id, _) = context.last_commit_request.unwrap(); + let last_table_id = context.last_commit_table_id.unwrap(); assert_eq!( - last_table_id, "foo", + last_table_id, "test_id", "Table ID should match the one passed to UCCommitter" ); } @@ -660,4 +2057,1122 @@ mod tests { Ok(()) } + + /// Schema visitor callback for tests: encodes the schema stored in the `schema` pointer + /// (which points to a `Vec`) into the kernel's `KernelSchemaVisitorState`. + /// Only supports primitive fields (no nested structs/arrays/maps) -- sufficient for tests. + extern "C" fn visit_test_schema( + schema_ptr: *mut c_void, + state: &mut KernelSchemaVisitorState, + ) -> usize { + let fields = unsafe { &*(schema_ptr as *const Vec) }; + let field_ids: Vec = fields + .iter() + .map(|field| { + let name = field.name.as_str(); + let nullable = field.nullable; + unsafe { + ok_or_panic(match field.data_type { + DataType::INTEGER => visit_field_integer( + state, + kernel_string_slice!(name), + nullable, + allocate_err, + ), + DataType::STRING => visit_field_string( + state, + kernel_string_slice!(name), + nullable, + allocate_err, + ), + DataType::LONG => visit_field_long( + state, + kernel_string_slice!(name), + nullable, + allocate_err, + ), + _ => panic!("Unsupported test field type: {:?}", field.data_type), + }) + } + }) + .collect(); + let root = "schema"; + unsafe { + ok_or_panic(visit_field_struct( + state, + kernel_string_slice!(root), + field_ids.as_ptr(), + field_ids.len(), + false, + allocate_err, + )) + } + } + + /// Create a [`CreateTableTransactionBuilder`] handle via the FFI, using the given schema + /// fields. Returns `(table_path, engine_handle, builder_handle)`. The caller is responsible + /// for freeing/consuming the engine and builder handles. + fn create_table_builder( + store: &Arc, + table_url: &Url, + fields: Vec, + ) -> ( + String, + Handle, + Handle, + ) { + let table_path = table_url.to_string(); + let engine = engine_handle_for_store(Arc::clone(store)); + let engine_info = "test-engine/1.0"; + let schema_arg = EngineSchema { + schema: &fields as *const Vec as *mut c_void, + visitor: visit_test_schema, + }; + let builder = ok_or_panic(unsafe { + get_create_table_builder( + kernel_string_slice!(table_path), + &schema_arg, + kernel_string_slice!(engine_info), + engine.shallow_copy(), + ) + }); + (table_path, engine, builder) + } + + /// Build and commit a create-table builder with default (null) committer, asserting that + /// the committed version is 0. + fn build_and_commit( + builder: Handle, + engine: &Handle, + ) -> u64 { + let txn = + ok_or_panic(unsafe { create_table_builder_build(builder, engine.shallow_copy()) }); + let committed = ok_or_panic(unsafe { create_table_commit(txn, engine.shallow_copy()) }); + let version = unsafe { version_and_free(committed) }; + assert_eq!(version, 0); + version + } + + #[tokio::test] + async fn test_create_table_basic() -> Result<(), Box> { + let (store, _test_engine, table_url) = + test_utils::engine_store_setup("test_create_table", None); + let (table_path, engine, builder) = create_table_builder( + &store, + &table_url, + vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("name", DataType::STRING), + ], + ); + let table_path_str = table_path.as_str(); + + build_and_commit(builder, &engine); + + // Verify by opening a snapshot of the created table + let snap = + unsafe { build_snapshot(kernel_string_slice!(table_path_str), engine.shallow_copy()) }; + let snap_version = unsafe { version(snap.shallow_copy()) }; + assert_eq!(snap_version, 0); + + // Verify schema + let snap_schema = unsafe { logical_schema(snap.shallow_copy()) }; + let snap_schema_ref = unsafe { snap_schema.as_ref() }; + assert_eq!(snap_schema_ref.num_fields(), 2); + assert_eq!(snap_schema_ref.field_at_index(0).unwrap().name, "id"); + assert_eq!(snap_schema_ref.field_at_index(1).unwrap().name, "name"); + + unsafe { free_schema(snap_schema) }; + unsafe { free_snapshot(snap) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + /// Reads the v0 commit file (`_delta_log/00..00.json`) of a freshly created table. + async fn read_v0_commit(store: &Arc, table_url: &Url) -> String { + let path = table_url + .join("_delta_log/00000000000000000000.json") + .expect("valid commit url"); + let path = Path::from_url_path(path.path()).expect("valid object store path"); + let bytes = store + .get(&path) + .await + .expect("v0 commit file should exist") + .bytes() + .await + .expect("v0 commit body"); + String::from_utf8(bytes.to_vec()).expect("v0 commit is utf-8") + } + + #[tokio::test] + async fn test_create_table_with_clustering_columns() -> Result<(), Box> { + let (store, _test_engine, table_url) = + test_utils::engine_store_setup("test_create_table", None); + let (_table_path, engine, builder) = create_table_builder( + &store, + &table_url, + vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("name", DataType::STRING), + ], + ); + + let col = "id"; + let columns = [kernel_string_slice!(col)]; + let builder = ok_or_panic(unsafe { + create_table_builder_with_clustering_columns( + builder, + columns.as_ptr(), + columns.len(), + engine.shallow_copy(), + ) + }); + build_and_commit(builder, &engine); + + // A clustered create records the `delta.clustering` domain metadata (listing the + // clustering columns, JSON-escaped inside the configuration string) and adds the + // `clustering` writer feature to the protocol. + let log = read_v0_commit(&store, &table_url).await; + assert!( + log.contains("delta.clustering"), + "missing clustering domain metadata: {log}" + ); + assert!( + log.contains("clusteringColumns") && log.contains(r#"[[\"id\"]]"#), + "missing clustering columns: {log}" + ); + assert!( + log.contains(r#""clustering""#), + "missing clustering writer feature: {log}" + ); + + unsafe { free_engine(engine) }; + Ok(()) + } + + #[tokio::test] + async fn test_create_table_with_partition_columns() -> Result<(), Box> { + let (store, _test_engine, table_url) = + test_utils::engine_store_setup("test_create_table", None); + // Partitioning requires at least one non-partition column, so partition on `date` + // while keeping `id` as data. + let (_table_path, engine, builder) = create_table_builder( + &store, + &table_url, + vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("date", DataType::STRING), + ], + ); + + let col = "date"; + let columns = [kernel_string_slice!(col)]; + let builder = ok_or_panic(unsafe { + create_table_builder_with_partition_columns( + builder, + columns.as_ptr(), + columns.len(), + engine.shallow_copy(), + ) + }); + build_and_commit(builder, &engine); + + // A partitioned create records the partition columns in the table metadata. + let log = read_v0_commit(&store, &table_url).await; + assert!( + log.contains(r#""partitionColumns":["date"]"#), + "missing partitionColumns in metadata: {log}" + ); + + unsafe { free_engine(engine) }; + Ok(()) + } + + #[tokio::test] + async fn test_create_table_with_multiple_clustering_columns( + ) -> Result<(), Box> { + let (store, _test_engine, table_url) = + test_utils::engine_store_setup("test_create_table", None); + let (_table_path, engine, builder) = create_table_builder( + &store, + &table_url, + vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("name", DataType::STRING), + ], + ); + + // Two columns: order must be preserved in the serialized clustering domain metadata. + let c0 = "id"; + let c1 = "name"; + let columns = [kernel_string_slice!(c0), kernel_string_slice!(c1)]; + let builder = ok_or_panic(unsafe { + create_table_builder_with_clustering_columns( + builder, + columns.as_ptr(), + columns.len(), + engine.shallow_copy(), + ) + }); + build_and_commit(builder, &engine); + + let log = read_v0_commit(&store, &table_url).await; + assert!( + log.contains(r#"[[\"id\"],[\"name\"]]"#), + "clustering column order not preserved: {log}" + ); + + unsafe { free_engine(engine) }; + Ok(()) + } + + #[tokio::test] + async fn test_create_table_data_layout_last_call_wins() -> Result<(), Box> + { + let (store, _test_engine, table_url) = + test_utils::engine_store_setup("test_create_table", None); + let (_table_path, engine, builder) = create_table_builder( + &store, + &table_url, + vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("date", DataType::STRING), + ], + ); + + // Set clustering, then partition: the layouts are mutually exclusive and the last call + // wins, so the committed table must be partitioned with no clustering domain metadata. + let c_col = "id"; + let c_cols = [kernel_string_slice!(c_col)]; + let builder = ok_or_panic(unsafe { + create_table_builder_with_clustering_columns( + builder, + c_cols.as_ptr(), + c_cols.len(), + engine.shallow_copy(), + ) + }); + let p_col = "date"; + let p_cols = [kernel_string_slice!(p_col)]; + let builder = ok_or_panic(unsafe { + create_table_builder_with_partition_columns( + builder, + p_cols.as_ptr(), + p_cols.len(), + engine.shallow_copy(), + ) + }); + build_and_commit(builder, &engine); + + let log = read_v0_commit(&store, &table_url).await; + assert!( + log.contains(r#""partitionColumns":["date"]"#), + "partition layout should win: {log}" + ); + assert!( + !log.contains("delta.clustering"), + "stale clustering layout leaked: {log}" + ); + + unsafe { free_engine(engine) }; + Ok(()) + } + + #[test] + fn test_collect_create_table_columns_empty_is_null_safe() { + // num_columns == 0 must short-circuit before from_raw_parts, so a null pointer is sound. + let columns = unsafe { collect_create_table_columns(std::ptr::null(), 0) }; + assert_eq!(columns.unwrap(), Vec::::new()); + } + + #[test] + fn test_with_data_layout_impl_propagates_column_error() { + // A column-collection error must short-circuit the shared lowering and drop the consumed + // builder, never producing a layout or a dangling handle. + let (store, _test_engine, table_url) = + test_utils::engine_store_setup("test_create_table", None); + let (_table_path, engine, builder_handle) = create_table_builder( + &store, + &table_url, + vec![StructField::nullable("id", DataType::INTEGER)], + ); + let builder = unsafe { *builder_handle.into_inner() }; + let layout: DeltaResult = Err(delta_kernel::Error::generic("bad column")); + let result = create_table_builder_with_data_layout_impl(builder, layout); + assert!(result.is_err()); + unsafe { free_engine(engine) }; + } + + /// CREATE TABLE: the committed transaction must expose a post-commit snapshot at version 0 + /// without re-listing the log. + #[tokio::test] + async fn test_post_commit_snapshot_create_table() -> Result<(), Box> { + let (store, _test_engine, table_url) = + test_utils::engine_store_setup("test_create_table", None); + let (_table_path, engine, builder) = create_table_builder( + &store, + &table_url, + vec![StructField::nullable("id", DataType::INTEGER)], + ); + + let txn = + ok_or_panic(unsafe { create_table_builder_build(builder, engine.shallow_copy()) }); + let committed = ok_or_panic(unsafe { create_table_commit(txn, engine.shallow_copy()) }); + + assert_eq!(unsafe { committed_transaction_version(&committed) }, 0); + + let snap = match unsafe { committed_transaction_post_commit_snapshot(&committed) } { + OptionalValue::Some(snap) => snap, + OptionalValue::None => { + panic!("create_table commit should produce a post-commit snapshot") + } + }; + assert_eq!(unsafe { version(snap.shallow_copy()) }, 0); + + unsafe { free_snapshot(snap) }; + unsafe { free_committed_transaction(committed) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + /// Existing-table commit: the committed transaction's post-commit snapshot must be at the + /// just-committed version. Also verifies that calling the accessor a second time yields an + /// independent handle (Arc clone). + #[tokio::test] + async fn test_post_commit_snapshot_existing_table() -> Result<(), Box> { + let (store, _test_engine, table_url) = + test_utils::engine_store_setup("test_post_commit_existing", None); + // create_table_with_one_file commits v0 (create) and v1 (file add). + let (table_path, engine) = create_table_with_one_file(&store, &table_url).await?; + + // Blind no-op commit on top of v1 -> v2. + let txn = ok_or_panic(unsafe { + transaction(kernel_string_slice!(table_path), engine.shallow_copy()) + }); + unsafe { set_data_change(txn.shallow_copy(), false) }; + let engine_info = "test-engine/1.0"; + let txn = ok_or_panic(unsafe { + with_engine_info( + txn, + kernel_string_slice!(engine_info), + engine.shallow_copy(), + ) + }); + + let committed = ok_or_panic(unsafe { commit(txn, engine.shallow_copy()) }); + let v = unsafe { committed_transaction_version(&committed) }; + assert_eq!(v, 2); + + let snap = match unsafe { committed_transaction_post_commit_snapshot(&committed) } { + OptionalValue::Some(snap) => snap, + OptionalValue::None => { + panic!("existing-table commit should produce a post-commit snapshot") + } + }; + assert_eq!(unsafe { version(snap.shallow_copy()) }, v); + + // Calling the accessor a second time must yield an independent handle (Arc clone). + let snap2 = match unsafe { committed_transaction_post_commit_snapshot(&committed) } { + OptionalValue::Some(snap) => snap, + OptionalValue::None => { + panic!("existing-table commit should produce a second post-commit snapshot") + } + }; + assert_eq!(unsafe { version(snap2.shallow_copy()) }, v); + + // Free the CommittedTransaction handle first to verify the post-commit snapshot + // remains valid afterwards (per Arc::clone semantics in + // committed_transaction_post_commit_snapshot). + unsafe { free_committed_transaction(committed) }; + assert_eq!(unsafe { version(snap.shallow_copy()) }, v); + assert_eq!(unsafe { version(snap2.shallow_copy()) }, v); + unsafe { free_snapshot(snap2) }; + unsafe { free_snapshot(snap) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + #[test] + fn test_create_table_with_properties() { + let (store, _test_engine, table_url) = + test_utils::engine_store_setup("test_create_table", None); + let (table_path, engine, builder) = create_table_builder( + &store, + &table_url, + vec![StructField::nullable("id", DataType::INTEGER)], + ); + + // Set properties + let prop_key1 = "delta.appendOnly"; + let prop_val1 = "true"; + let builder = ok_or_panic(unsafe { + create_table_builder_with_table_property( + builder, + kernel_string_slice!(prop_key1), + kernel_string_slice!(prop_val1), + engine.shallow_copy(), + ) + }); + let prop_key2 = "custom.key"; + let prop_val2 = "custom_value"; + let builder = ok_or_panic(unsafe { + create_table_builder_with_table_property( + builder, + kernel_string_slice!(prop_key2), + kernel_string_slice!(prop_val2), + engine.shallow_copy(), + ) + }); + + build_and_commit(builder, &engine); + + // Verify properties via snapshot + let table_path_str = table_path.as_str(); + let snap = + unsafe { build_snapshot(kernel_string_slice!(table_path_str), engine.shallow_copy()) }; + let snap_ref = unsafe { snap.as_ref() }; + + // Verify parsed table properties + let props = snap_ref.table_properties(); + assert_eq!(props.append_only, Some(true)); + assert_eq!( + props + .unknown_properties + .get("custom.key") + .map(|s| s.as_str()), + Some("custom_value") + ); + + // Verify feature is enabled in protocol (property set + protocol supports it) + let config = snap_ref.table_configuration(); + assert!(config.is_feature_enabled(&TableFeature::AppendOnly)); + + unsafe { free_snapshot(snap) }; + unsafe { free_engine(engine) }; + } + + /// Builds a create-table transaction and its unpartitioned write context, optionally + /// enabling column mapping. Returns handles the caller must free. + fn cm_table_write_context( + store: &Arc, + table_url: &Url, + cm_mode: Option<&str>, + ) -> ( + Handle, + Handle, + Handle, + ) { + let (_table_path, engine, mut builder) = create_table_builder( + store, + table_url, + vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("name", DataType::STRING), + ], + ); + if let Some(mode) = cm_mode { + let cm_key = "delta.columnMapping.mode"; + builder = ok_or_panic(unsafe { + create_table_builder_with_table_property( + builder, + kernel_string_slice!(cm_key), + kernel_string_slice!(mode), + engine.shallow_copy(), + ) + }); + } + let txn = + ok_or_panic(unsafe { create_table_builder_build(builder, engine.shallow_copy()) }); + let write_context = ok_or_panic(unsafe { + create_table_get_unpartitioned_write_context(txn.shallow_copy(), engine.shallow_copy()) + }); + (engine, txn, write_context) + } + + #[rstest] + #[case::no_column_mapping(None)] + #[case::name_mode(Some("name"))] + #[case::id_mode(Some("id"))] + fn test_write_context_accessors(#[case] cm_mode: Option<&str>) { + let (store, _test_engine, table_url) = + test_utils::engine_store_setup("test_create_table", None); + let (engine, txn, write_context) = cm_table_write_context(&store, &table_url, cm_mode); + + let logical = unsafe { get_write_schema(write_context.shallow_copy()) }; + let physical = unsafe { get_physical_write_schema(write_context.shallow_copy()) }; + let l2p = unsafe { get_logical_to_physical(write_context.shallow_copy()) }; + + let logical_ref = unsafe { logical.as_ref() }; + let physical_ref = unsafe { physical.as_ref() }; + assert_eq!(logical_ref.num_fields(), 2); + assert_eq!(physical_ref.num_fields(), 2); + assert_eq!(logical_ref.field_at_index(0).unwrap().name, "id"); + assert_eq!(logical_ref.field_at_index(1).unwrap().name, "name"); + + let field_id_key = ColumnMetadataKey::ParquetFieldId.as_ref(); + for (i, logical_name) in ["id", "name"].iter().enumerate() { + let field = physical_ref.field_at_index(i).unwrap(); + let field_id = field.metadata.get(field_id_key); + match cm_mode { + Some(_) => { + // Column mapping rewrites each logical name to a fresh `col-` per + // the Delta protocol's column-mapping section, and stamps a numeric + // `parquet.field.id` so the field id reaches the parquet file. + assert!( + field.name.starts_with("col-"), + "field {i}: expected `col-` physical name, got {:?}", + field.name, + ); + assert_ne!(&field.name, logical_name); + assert!( + matches!(field_id, Some(MetadataValue::Number(_))), + "field {i}: parquet.field.id should be a Number, got {field_id:?}", + ); + } + None => { + assert_eq!(&field.name, logical_name); + assert!( + field_id.is_none(), + "field {i}: parquet.field.id should be absent without column mapping, got {field_id:?}", + ); + } + } + } + + // Smoke-check that (logical, l2p, physical) compose into a valid evaluator. + let evaluator = ok_or_panic(unsafe { + new_expression_evaluator( + engine.shallow_copy(), + logical.shallow_copy(), + l2p.as_ref(), + physical.shallow_copy(), + ) + }); + + unsafe { + free_expression_evaluator(evaluator); + free_kernel_expression(l2p); + free_schema(physical); + free_schema(logical); + free_write_context(write_context); + create_table_free_transaction(txn); + free_engine(engine); + } + } + + #[tokio::test] + async fn test_create_table_already_exists() -> Result<(), Box> { + let (store, _test_engine, table_url) = + test_utils::engine_store_setup("test_create_table", None); + + // Create the table first time -- should succeed + let (_table_path, engine, builder) = create_table_builder( + &store, + &table_url, + vec![StructField::nullable("id", DataType::INTEGER)], + ); + build_and_commit(builder, &engine); + + // Try to create the same table again -- build should error (table already exists) + let (_, engine2, builder2) = create_table_builder( + &store, + &table_url, + vec![StructField::nullable("id", DataType::INTEGER)], + ); + let result = unsafe { create_table_builder_build(builder2, engine2.shallow_copy()) }; + match result { + ExternResult::Err(e) => { + // Clean up the error to prevent leaks + let error = unsafe { recover_error(e) }; + assert!( + error.message.contains("already exists"), + "Expected 'already exists' error, got: {}", + error.message + ); + } + ExternResult::Ok(txn) => { + unsafe { create_table_free_transaction(txn) }; + panic!("Expected error for table that already exists"); + } + } + + unsafe { free_engine(engine2) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + #[test] + fn test_free_create_table_builder() { + let schema = schema_ref! { nullable "id": INTEGER }; + let builder = + delta_kernel::transaction::create_table::create_table("memory:///test", schema, "test"); + let handle: Handle = Box::new(builder).into(); + // Should not panic or leak + unsafe { free_create_table_builder(handle) }; + } + + #[tokio::test] + async fn test_create_table_build_with_empty_schema_succeeds( + ) -> Result<(), Box> { + let (store, _test_engine, table_url) = + test_utils::engine_store_setup("test_create_table", None); + // CREATE TABLE with no columns is permitted by the Delta protocol; users may + // populate the schema later via ALTER TABLE ADD COLUMN. + let (_table_path, engine, builder) = create_table_builder(&store, &table_url, vec![]); + + let txn = + ok_or_panic(unsafe { create_table_builder_build(builder, engine.shallow_copy()) }); + unsafe { create_table_free_transaction(txn) }; + + unsafe { free_engine(engine) }; + Ok(()) + } + + #[tokio::test] + async fn test_create_table_with_custom_committer() -> Result<(), Box> { + let (store, _test_engine, table_url) = + test_utils::engine_store_setup("test_create_table", None); + let (table_path, engine, builder) = create_table_builder( + &store, + &table_url, + vec![StructField::nullable("id", DataType::INTEGER)], + ); + let table_path_str = table_path.as_str(); + + // Create a FileSystemCommitter handle and pass it to build_with_committer + let committer: Box = + Box::new(FileSystemCommitter::new()); + let committer_handle: Handle = committer.into(); + + let txn = ok_or_panic(unsafe { + create_table_builder_build_with_committer( + builder, + committer_handle, + engine.shallow_copy(), + ) + }); + let committed = ok_or_panic(unsafe { create_table_commit(txn, engine.shallow_copy()) }); + let committed_version = unsafe { version_and_free(committed) }; + assert_eq!(committed_version, 0); + + // Verify the table was created + let snap = + unsafe { build_snapshot(kernel_string_slice!(table_path_str), engine.shallow_copy()) }; + assert_eq!(unsafe { version(snap.shallow_copy()) }, 0); + + unsafe { free_snapshot(snap) }; + unsafe { free_engine(engine) }; + Ok(()) + } + + /// Helper: create a table, write one parquet file, and return (table_path, engine_handle). + /// The caller is responsible for freeing the engine handle. + async fn create_table_with_one_file( + store: &Arc, + table_url: &Url, + ) -> Result<(String, Handle), Box> { + let table_path = table_url.as_str(); + let fields = vec![ + StructField::nullable("number", DataType::INTEGER), + StructField::nullable("value", DataType::STRING), + ]; + + let engine = engine_handle_for_store(Arc::clone(store)); + + // Create the table + let engine_info = "test-engine/1.0"; + let schema_arg = EngineSchema { + schema: &fields as *const Vec as *mut c_void, + visitor: visit_test_schema, + }; + let builder = ok_or_panic(unsafe { + get_create_table_builder( + kernel_string_slice!(table_path), + &schema_arg, + kernel_string_slice!(engine_info), + engine.shallow_copy(), + ) + }); + build_and_commit(builder, &engine); + + // Write a parquet file and commit it + let batch = RecordBatch::try_from_iter(vec![ + ( + "number", + Arc::new(Int32Array::from(vec![1, 2, 3])) as ArrayRef, + ), + ( + "value", + Arc::new(StringArray::from(vec!["a", "b", "c"])) as ArrayRef, + ), + ])?; + + let txn = ok_or_panic(unsafe { + transaction(kernel_string_slice!(table_path), engine.shallow_copy()) + }); + let engine_info = "test-engine/1.0"; + let txn = ok_or_panic(unsafe { + with_engine_info( + txn, + kernel_string_slice!(engine_info), + engine.shallow_copy(), + ) + }); + + let parquet_schema = unsafe { txn.shallow_copy().as_ref().add_files_schema() }; + let file_info = put_parquet_file( + store, + table_url, + "file1.parquet", + &batch, + parquet_schema.as_ref().try_into_arrow()?, + ) + .await?; + let file_info_engine_data = ok_or_panic(unsafe { + get_engine_data( + file_info.array, + &file_info.schema, + crate::ffi_test_utils::allocate_err, + ) + }); + unsafe { add_files(txn.shallow_copy(), file_info_engine_data) }; + let committed = ok_or_panic(unsafe { commit(txn, engine.shallow_copy()) }); + unsafe { free_committed_transaction(committed) }; + + Ok((table_path.to_string(), engine)) + } + + fn assert_no_active_files( + kernel_engine: &Arc, + table_path: &str, + ) -> Result<(), Box> { + let snapshot = + delta_kernel::snapshot::Snapshot::builder_for(delta_kernel::try_parse_uri(table_path)?) + .build(kernel_engine.as_ref())?; + let scan = snapshot.scan_builder().build()?; + let scan_meta: Vec<_> = scan + .scan_metadata(kernel_engine.as_ref())? + .collect::>()?; + let total_selected: usize = scan_meta + .iter() + .map(|m| { + let sv = m.scan_files.selection_vector(); + let data_len = m.scan_files.data().len(); + if sv.is_empty() { + data_len + } else { + sv.iter().filter(|&&b| b).count() + } + }) + .sum(); + assert_eq!(total_selected, 0, "Expected 0 files after removal"); + Ok(()) + } + + /// Helper: create a table with one file, build a snapshot, extract scan metadata, and + /// return the components needed for remove_files tests. Caller must free the engine handle + /// (and txn if not committed). + #[allow(clippy::type_complexity)] + async fn setup_remove_files_test( + store: &Arc, + table_url: &Url, + ) -> Result< + ( + Box, + Vec, + Handle, + Handle, + Arc, + String, + ), + Box, + > { + let (table_path, engine) = create_table_with_one_file(store, table_url).await?; + let table_path_str = table_path.as_str(); + + let kernel_engine = unsafe { engine.as_ref() }.engine(); + let snapshot = delta_kernel::snapshot::Snapshot::builder_for(delta_kernel::try_parse_uri( + table_path_str, + )?) + .build(kernel_engine.as_ref())?; + + let scan = snapshot.scan_builder().build()?; + let scan_meta_items: Vec<_> = scan + .scan_metadata(kernel_engine.as_ref())? + .collect::>()?; + assert_eq!(scan_meta_items.len(), 1); + + let scan_meta = scan_meta_items.into_iter().next().unwrap(); + let (data, sv) = scan_meta.scan_files.into_parts(); + + let txn = ok_or_panic(unsafe { + transaction(kernel_string_slice!(table_path_str), engine.shallow_copy()) + }); + let engine_info = "test-engine/1.0"; + let txn = ok_or_panic(unsafe { + with_engine_info( + txn, + kernel_string_slice!(engine_info), + engine.shallow_copy(), + ) + }); + + Ok((data, sv, txn, engine, kernel_engine, table_path)) + } + + #[tokio::test] + async fn test_remove_files_with_null_sv_commits_and_removes_all( + ) -> Result<(), Box> { + let (store, _test_engine, table_url) = + test_utils::engine_store_setup("test_remove_files", None); + let (data, sv, txn, engine, kernel_engine, table_path) = + setup_remove_files_test(&store, &table_url).await?; + let data_handle: Handle = data.into(); + + // Pass the original SV as u8 values + let sv_u8: Vec = sv.iter().map(|&b| b as u8).collect(); + let sv_ptr = if sv_u8.is_empty() { + std::ptr::null() + } else { + sv_u8.as_ptr() + }; + ok_or_panic(unsafe { + remove_files( + txn.shallow_copy(), + data_handle, + sv_ptr, + sv_u8.len(), + engine.shallow_copy(), + ) + }); + + let committed = ok_or_panic(unsafe { commit(txn, engine.shallow_copy()) }); + unsafe { free_committed_transaction(committed) }; + assert_no_active_files(&kernel_engine, table_path.as_str())?; + + unsafe { free_engine(engine) }; + Ok(()) + } + + #[test] + fn test_empty_sv_creates_filtered_data_selecting_all_rows() { + // Verifies the FilteredEngineData contract that an empty selection vector means + // "all rows selected". The remove_files FFI wrapper normalizes null/zero-length + // pointers to an empty vec before constructing FilteredEngineData. + let batch = RecordBatch::try_from_iter(vec![( + "id", + Arc::new(Int32Array::from(vec![1, 2, 3])) as ArrayRef, + )]) + .unwrap(); + let data: Box = Box::new(ArrowEngineData::from(batch)); + + let filtered = FilteredEngineData::try_new(data, vec![]).unwrap(); + assert_eq!(filtered.selection_vector().len(), 0); + assert_eq!(filtered.data().len(), 3); + } + + /// End-to-end FFI round trip for connector-authored deletion vector updates. + #[tokio::test] + async fn test_dv_update_round_trip_via_ffi() -> Result<(), Box> { + use delta_kernel::actions::deletion_vector_writer::{ + KernelDeletionVector, StreamingDeletionVectorWriter, + }; + use delta_kernel::object_store::path::Path as ObjectStorePath; + use delta_kernel_ffi::scan::{free_scan, scan, scan_metadata_iter_init}; + use delta_kernel_ffi::transaction::{ + dv_descriptor_map_insert, dv_descriptor_map_new, dv_descriptor_new, + transaction_update_deletion_vectors, KernelDvStorageType, + }; + use test_utils::record_batch_to_bytes; + + // Build a DV-enabled table; create_table sets delta.enableDeletionVectors for the + // writer feature. + let schema = Arc::new(try_schema! { nullable "id": INTEGER }?); + let (store, _test_engine, table_location) = + test_utils::engine_store_setup("test_dv_ffi", None); + let table_url = test_utils::create_table( + store.clone(), + table_location.clone(), + schema.clone(), + &[], + true, + vec!["deletionVectors"], + vec!["deletionVectors"], + ) + .await?; + + // Write a 4-row parquet file directly into the table. + let batch = RecordBatch::try_from_iter(vec![( + "id", + Arc::new(Int32Array::from(vec![10, 20, 30, 40])) as ArrayRef, + )])?; + let parquet_bytes = record_batch_to_bytes(&batch); + let parquet_len = parquet_bytes.len(); + let data_file_path = "data_file.parquet".to_string(); + let data_url = table_url.join(&data_file_path)?; + store + .put( + &ObjectStorePath::from_url_path(data_url.path())?, + parquet_bytes.into(), + ) + .await?; + + // Add the file via a kernel transaction so the table has something to scan. + // Build the kernel engine over the seeded store: `create_default_engine` resolves a fresh + // store from the URL, which for `memory://` would not see this table. + let kernel_engine: Arc = Arc::new( + delta_kernel_default_engine::DefaultEngineBuilder::new(Arc::clone(&store)).build(), + ); + let snapshot = delta_kernel::snapshot::Snapshot::builder_for(table_url.clone()) + .build(kernel_engine.as_ref())?; + let mut add_txn = snapshot + .clone() + .transaction(Box::new(FileSystemCommitter::new()), kernel_engine.as_ref())? + .with_engine_info("test-engine/1.0") + .with_operation("WRITE".to_string()); + let add_files_schema = add_txn.add_files_schema(); + let add_metadata = test_utils::create_add_files_metadata( + add_files_schema, + vec![(&data_file_path, parquet_len as i64, 1_000_000, Some(4))], + )?; + add_txn.add_files(add_metadata); + let _ = add_txn.commit(kernel_engine.as_ref())?.unwrap_committed(); + + // Build and write a connector-authored DV file deleting rows 1 and 2 (ids 20, 30). + let mut dv = KernelDeletionVector::new(); + dv.add_deleted_row_indexes([1u64, 2]); + let mut dv_bytes = Vec::new(); + let mut dv_writer = StreamingDeletionVectorWriter::new(&mut dv_bytes); + let dv_write_result = dv_writer.write_deletion_vector(dv)?; + dv_writer.finalize()?; + let dv_url = table_url.join("connector_dv.bin")?; + store + .put( + &ObjectStorePath::from_url_path(dv_url.path())?, + dv_bytes.into(), + ) + .await?; + + // === FFI surface under test === + let table_path_str = table_url.as_str(); + let engine = engine_handle_for_store(Arc::clone(&store)); + let txn = ok_or_panic(unsafe { + transaction(kernel_string_slice!(table_path_str), engine.shallow_copy()) + }); + let engine_info = "test-engine/1.0"; + let txn = ok_or_panic(unsafe { + with_engine_info( + txn, + kernel_string_slice!(engine_info), + engine.shallow_copy(), + ) + }); + + // Build a descriptor from the connector-authored DV file metadata. + let dv_url_string = dv_url.to_string(); + let descriptor = ok_or_panic(unsafe { + dv_descriptor_new( + KernelDvStorageType::PersistedAbsolute as i32, + kernel_string_slice!(dv_url_string), + true, + dv_write_result.offset, + dv_write_result.size_in_bytes, + dv_write_result.cardinality, + engine.shallow_copy(), + ) + }); + + // Build the DV map and insert. + let map = dv_descriptor_map_new(); + ok_or_panic(unsafe { + dv_descriptor_map_insert( + map.shallow_copy(), + kernel_string_slice!(data_file_path), + descriptor, + engine.shallow_copy(), + ) + }); + + // Build a scan + iterator handle to feed the update call. + let snap_handle = + unsafe { build_snapshot(kernel_string_slice!(table_path_str), engine.shallow_copy()) }; + let scan_handle = ok_or_panic(unsafe { + scan( + snap_handle.shallow_copy(), + engine.shallow_copy(), + None, + None, + ) + }); + let scan_iter = ok_or_panic(unsafe { + scan_metadata_iter_init(engine.shallow_copy(), scan_handle.shallow_copy()) + }); + + ok_or_panic(unsafe { + transaction_update_deletion_vectors( + txn.shallow_copy(), + map, + scan_iter, + engine.shallow_copy(), + ) + }); + + let committed = ok_or_panic(unsafe { commit(txn, engine.shallow_copy()) }); + let committed_version = unsafe { version_and_free(committed) }; + assert_eq!(committed_version, 2); + + // Verify: scan now returns 2 rows (10 and 40). + let snapshot = delta_kernel::snapshot::Snapshot::builder_for(table_url.clone()) + .build(kernel_engine.as_ref())?; + let scan_after = snapshot.scan_builder().build()?; + let total: usize = scan_after + .execute(kernel_engine.clone())? + .map(|r| Ok::<_, delta_kernel::Error>(r?.len())) + .sum::>()?; + assert_eq!(total, 2, "expected 2 surviving rows"); + + unsafe { + free_scan(scan_handle); + free_snapshot(snap_handle); + free_engine(engine); + } + + Ok(()) + } + + #[tokio::test] + async fn test_remove_files_with_non_empty_sv_exercises_from_raw_parts( + ) -> Result<(), Box> { + // Exercises the from_raw_parts code path in the remove_files FFI wrapper by passing + // a non-null selection vector pointer with non-zero length. The null-SV test always + // passes a null pointer because scan_metadata for a single-file table returns an + // empty selection vector. + let (store, _test_engine, table_url) = + test_utils::engine_store_setup("test_remove_files_sv", None); + let (data, _sv, txn, engine, _kernel_engine, _table_path) = + setup_remove_files_test(&store, &table_url).await?; + let num_rows = data.len(); + assert!(num_rows > 0); + let data_handle: Handle = data.into(); + + // Construct a non-empty all-true SV selecting every row. Uses u8 values to match + // the FFI signature (nonzero = selected). + let sv: Vec = vec![1; num_rows]; + ok_or_panic(unsafe { + remove_files( + txn.shallow_copy(), + data_handle, + sv.as_ptr(), + sv.len(), + engine.shallow_copy(), + ) + }); + + // Don't commit -- the from_raw_parts path has been exercised. Committing with + // a non-empty SV triggers a different code path in generate_remove_actions that + // requires additional scan row fields not present in this test setup. + unsafe { free_transaction(txn) }; + unsafe { free_engine(engine) }; + Ok(()) + } } diff --git a/ffi/src/transaction/partition_value.rs b/ffi/src/transaction/partition_value.rs new file mode 100644 index 0000000000..233b7b14aa --- /dev/null +++ b/ffi/src/transaction/partition_value.rs @@ -0,0 +1,648 @@ +//! FFI surface for building the partition values of a partitioned write. +//! +//! Engines build a [`PartitionValueMap`] by inserting one typed value per partition column +//! (keyed by the column's logical name), then pass it to `get_partitioned_write_context`. The +//! typed inserts mirror the `visit_expression_literal_*` family: one entry point per scalar type. +//! The kernel validates the supplied values against the table's partition schema and serializes +//! them per the Delta protocol when the write context is built. + +use std::collections::HashMap; + +use delta_kernel::expressions::Scalar; +use delta_kernel::DeltaResult; +use delta_kernel_ffi_macros::handle_descriptor; + +use crate::error::{ExternResult, IntoExternResult}; +use crate::expressions::kernel_visitor::NullTypeTag; +use crate::handle::Handle; +use crate::{KernelStringSlice, SharedExternEngine, TryFromStringSlice}; + +/// Owns a map from a partition column's logical name to its [`Scalar`] value. Engines build the +/// map with the `partition_value_map_insert_*` functions and pass it to +/// `get_partitioned_write_context` (or its create-table counterpart), which consumes it. +pub struct PartitionValueMap { + pub(crate) inner: HashMap, +} + +/// Mutable handle for a `PartitionValueMap`. +#[handle_descriptor(target=PartitionValueMap, mutable=true, sized=true)] +pub struct ExclusivePartitionValueMap; + +/// Allocate an empty partition value map. The returned handle must be released either by +/// [`free_partition_value_map`] or by `get_partitioned_write_context` (which consumes the map). +#[no_mangle] +pub extern "C" fn partition_value_map_new() -> Handle { + Box::new(PartitionValueMap { + inner: HashMap::new(), + }) + .into() +} + +/// Free a partition value map handle. +/// +/// # Safety +/// +/// Caller must pass a valid handle previously returned by [`partition_value_map_new`] that has +/// not already been consumed by a write-context call. +#[no_mangle] +pub unsafe extern "C" fn free_partition_value_map(map: Handle) { + map.drop_handle(); +} + +// ============================================================================= +// Typed inserts +// ============================================================================= +// +// Each insert keys on the partition column's *logical* name (the kernel translates to physical +// names). Re-inserting the same name replaces the previous value. Inserts can fail when the `name` +// slice is not valid UTF-8 (and, for decimal/null, when the type parameters are invalid), hence the +// `ExternResult` return; the engine handle is used to allocate any error. The map handle is +// borrowed, never consumed, so the caller keeps ownership across inserts. These are written out +// per-type rather than macro-generated because cbindgen scans source syntactically and cannot see +// `extern "C" fn`s produced by macro expansion, so they would be absent from the generated header. +// See issue #255 for more info. + +/// Common insertion path: validate the name, then store the scalar under it. Returns `true` on +/// success. +fn partition_value_map_insert_impl( + map: &mut PartitionValueMap, + name: DeltaResult<&str>, + scalar: DeltaResult, +) -> DeltaResult { + map.inner.insert(name?.to_string(), scalar?); + Ok(true) +} + +/// Insert a `string` partition value under `name`. +/// +/// # Safety +/// +/// Caller must pass valid handles and string slices. +#[no_mangle] +pub unsafe extern "C" fn partition_value_map_insert_string( + mut map: Handle, + name: KernelStringSlice, + value: KernelStringSlice, + engine: Handle, +) -> ExternResult { + let map = unsafe { map.as_mut() }; + let engine = unsafe { engine.as_ref() }; + let name = unsafe { TryFromStringSlice::try_from_slice(&name) }; + let value = unsafe { String::try_from_slice(&value) }; + partition_value_map_insert_impl(map, name, value.map(Scalar::from)).into_extern_result(&engine) +} + +/// Insert an `integer` (32-bit) partition value under `name`. +/// +/// # Safety +/// +/// Caller must pass valid handles and a valid `name` slice. +#[no_mangle] +pub unsafe extern "C" fn partition_value_map_insert_int( + mut map: Handle, + name: KernelStringSlice, + value: i32, + engine: Handle, +) -> ExternResult { + let map = unsafe { map.as_mut() }; + let engine = unsafe { engine.as_ref() }; + let name = unsafe { TryFromStringSlice::try_from_slice(&name) }; + partition_value_map_insert_impl(map, name, Ok(Scalar::from(value))).into_extern_result(&engine) +} + +/// Insert a `long` (64-bit) partition value under `name`. +/// +/// # Safety +/// +/// Caller must pass valid handles and a valid `name` slice. +#[no_mangle] +pub unsafe extern "C" fn partition_value_map_insert_long( + mut map: Handle, + name: KernelStringSlice, + value: i64, + engine: Handle, +) -> ExternResult { + let map = unsafe { map.as_mut() }; + let engine = unsafe { engine.as_ref() }; + let name = unsafe { TryFromStringSlice::try_from_slice(&name) }; + partition_value_map_insert_impl(map, name, Ok(Scalar::from(value))).into_extern_result(&engine) +} + +/// Insert a `short` (16-bit) partition value under `name`. +/// +/// # Safety +/// +/// Caller must pass valid handles and a valid `name` slice. +#[no_mangle] +pub unsafe extern "C" fn partition_value_map_insert_short( + mut map: Handle, + name: KernelStringSlice, + value: i16, + engine: Handle, +) -> ExternResult { + let map = unsafe { map.as_mut() }; + let engine = unsafe { engine.as_ref() }; + let name = unsafe { TryFromStringSlice::try_from_slice(&name) }; + partition_value_map_insert_impl(map, name, Ok(Scalar::from(value))).into_extern_result(&engine) +} + +/// Insert a `byte` (8-bit) partition value under `name`. +/// +/// # Safety +/// +/// Caller must pass valid handles and a valid `name` slice. +#[no_mangle] +pub unsafe extern "C" fn partition_value_map_insert_byte( + mut map: Handle, + name: KernelStringSlice, + value: i8, + engine: Handle, +) -> ExternResult { + let map = unsafe { map.as_mut() }; + let engine = unsafe { engine.as_ref() }; + let name = unsafe { TryFromStringSlice::try_from_slice(&name) }; + partition_value_map_insert_impl(map, name, Ok(Scalar::from(value))).into_extern_result(&engine) +} + +/// Insert a `float` (32-bit) partition value under `name`. +/// +/// # Safety +/// +/// Caller must pass valid handles and a valid `name` slice. +#[no_mangle] +pub unsafe extern "C" fn partition_value_map_insert_float( + mut map: Handle, + name: KernelStringSlice, + value: f32, + engine: Handle, +) -> ExternResult { + let map = unsafe { map.as_mut() }; + let engine = unsafe { engine.as_ref() }; + let name = unsafe { TryFromStringSlice::try_from_slice(&name) }; + partition_value_map_insert_impl(map, name, Ok(Scalar::from(value))).into_extern_result(&engine) +} + +/// Insert a `double` (64-bit) partition value under `name`. +/// +/// # Safety +/// +/// Caller must pass valid handles and a valid `name` slice. +#[no_mangle] +pub unsafe extern "C" fn partition_value_map_insert_double( + mut map: Handle, + name: KernelStringSlice, + value: f64, + engine: Handle, +) -> ExternResult { + let map = unsafe { map.as_mut() }; + let engine = unsafe { engine.as_ref() }; + let name = unsafe { TryFromStringSlice::try_from_slice(&name) }; + partition_value_map_insert_impl(map, name, Ok(Scalar::from(value))).into_extern_result(&engine) +} + +/// Insert a `boolean` partition value under `name`. +/// +/// # Safety +/// +/// Caller must pass valid handles and a valid `name` slice. +#[no_mangle] +pub unsafe extern "C" fn partition_value_map_insert_bool( + mut map: Handle, + name: KernelStringSlice, + value: bool, + engine: Handle, +) -> ExternResult { + let map = unsafe { map.as_mut() }; + let engine = unsafe { engine.as_ref() }; + let name = unsafe { TryFromStringSlice::try_from_slice(&name) }; + partition_value_map_insert_impl(map, name, Ok(Scalar::from(value))).into_extern_result(&engine) +} + +/// Insert a `date` partition value (`value` = days since the Unix epoch) under `name`. +/// +/// # Safety +/// +/// Caller must pass valid handles and a valid `name` slice. +#[no_mangle] +pub unsafe extern "C" fn partition_value_map_insert_date( + mut map: Handle, + name: KernelStringSlice, + value: i32, + engine: Handle, +) -> ExternResult { + let map = unsafe { map.as_mut() }; + let engine = unsafe { engine.as_ref() }; + let name = unsafe { TryFromStringSlice::try_from_slice(&name) }; + partition_value_map_insert_impl(map, name, Ok(Scalar::Date(value))).into_extern_result(&engine) +} + +/// Insert a `timestamp` partition value (`value` = microseconds since the Unix epoch, UTC) under +/// `name`. +/// +/// # Safety +/// +/// Caller must pass valid handles and a valid `name` slice. +#[no_mangle] +pub unsafe extern "C" fn partition_value_map_insert_timestamp( + mut map: Handle, + name: KernelStringSlice, + value: i64, + engine: Handle, +) -> ExternResult { + let map = unsafe { map.as_mut() }; + let engine = unsafe { engine.as_ref() }; + let name = unsafe { TryFromStringSlice::try_from_slice(&name) }; + partition_value_map_insert_impl(map, name, Ok(Scalar::Timestamp(value))) + .into_extern_result(&engine) +} + +/// Insert a `timestamp_ntz` partition value (`value` = microseconds since the Unix epoch, no +/// timezone) under `name`. +/// +/// # Safety +/// +/// Caller must pass valid handles and a valid `name` slice. +#[no_mangle] +pub unsafe extern "C" fn partition_value_map_insert_timestamp_ntz( + mut map: Handle, + name: KernelStringSlice, + value: i64, + engine: Handle, +) -> ExternResult { + let map = unsafe { map.as_mut() }; + let engine = unsafe { engine.as_ref() }; + let name = unsafe { TryFromStringSlice::try_from_slice(&name) }; + partition_value_map_insert_impl(map, name, Ok(Scalar::TimestampNtz(value))) + .into_extern_result(&engine) +} + +/// Insert a `binary` partition value under `name`, copying `len` bytes from `value`. +/// +/// # Safety +/// +/// Caller must pass valid handles, a valid `name` slice, and (when `len > 0`) a `value` pointer to +/// at least `len` readable bytes. An empty value may be passed as `(null, 0)`. +#[no_mangle] +pub unsafe extern "C" fn partition_value_map_insert_binary( + mut map: Handle, + name: KernelStringSlice, + value: *const u8, + len: usize, + engine: Handle, +) -> ExternResult { + let map = unsafe { map.as_mut() }; + let engine = unsafe { engine.as_ref() }; + let name = unsafe { TryFromStringSlice::try_from_slice(&name) }; + // `from_raw_parts` requires a non-null pointer even when `len == 0`, so guard the empty case. + let bytes = if value.is_null() || len == 0 { + &[] + } else { + unsafe { std::slice::from_raw_parts(value, len) } + }; + partition_value_map_insert_impl(map, name, Ok(Scalar::Binary(bytes.to_vec()))) + .into_extern_result(&engine) +} + +/// Insert a `decimal` partition value under `name`. The unscaled 128-bit value is supplied as two +/// 64-bit halves (`value_hi << 64 | value_lo`). Returns an error if the precision/scale +/// combination is invalid. +/// +/// # Safety +/// +/// Caller must pass valid handles and a valid `name` slice. +#[no_mangle] +pub unsafe extern "C" fn partition_value_map_insert_decimal( + mut map: Handle, + name: KernelStringSlice, + value_hi: u64, + value_lo: u64, + precision: u8, + scale: u8, + engine: Handle, +) -> ExternResult { + let map = unsafe { map.as_mut() }; + let engine = unsafe { engine.as_ref() }; + let name = unsafe { TryFromStringSlice::try_from_slice(&name) }; + let value = ((value_hi as i128) << 64) | (value_lo as i128); + partition_value_map_insert_impl(map, name, Scalar::decimal(value, precision, scale)) + .into_extern_result(&engine) +} + +/// Insert a typed `null` partition value under `name`. The `type_tag` identifies the column's +/// data type using the same `NullTypeTag` encoding as `visit_expression_literal_null` in +/// [`kernel_visitor`](crate::expressions::kernel_visitor). For the decimal tag, `precision` and +/// `scale` specify the decimal parameters; for all other types, pass 0 for both. +/// +/// Returns an error if the tag is unrecognized, is the non-primitive sentinel (255), or carries an +/// invalid decimal precision/scale. A null partition value is only legal for a nullable partition +/// column; the kernel rejects it otherwise when the write context is built. +/// +/// # Safety +/// +/// Caller must pass valid handles and a valid `name` slice. +#[no_mangle] +pub unsafe extern "C" fn partition_value_map_insert_null( + mut map: Handle, + name: KernelStringSlice, + type_tag: u8, + precision: u8, + scale: u8, + engine: Handle, +) -> ExternResult { + let map = unsafe { map.as_mut() }; + let engine = unsafe { engine.as_ref() }; + let name = unsafe { TryFromStringSlice::try_from_slice(&name) }; + let scalar = NullTypeTag::try_from(type_tag) + .and_then(|tag| tag.to_data_type(precision, scale)) + .map(Scalar::Null); + partition_value_map_insert_impl(map, name, scalar).into_extern_result(&engine) +} + +#[cfg(test)] +mod tests { + use delta_kernel::Error; + + use super::*; + + #[test] + fn insert_stores_typed_scalars_under_logical_names() { + let mut map = PartitionValueMap { + inner: HashMap::new(), + }; + partition_value_map_insert_impl(&mut map, Ok("year"), Ok(Scalar::from(2024i32))).unwrap(); + partition_value_map_insert_impl(&mut map, Ok("region"), Ok(Scalar::from("US"))).unwrap(); + + assert_eq!(map.inner.len(), 2); + assert_eq!(map.inner.get("year"), Some(&Scalar::Integer(2024))); + assert_eq!(map.inner.get("region"), Some(&Scalar::String("US".into()))); + } + + #[test] + fn insert_replaces_existing_value_for_same_name() { + let mut map = PartitionValueMap { + inner: HashMap::new(), + }; + partition_value_map_insert_impl(&mut map, Ok("p"), Ok(Scalar::from(1i32))).unwrap(); + partition_value_map_insert_impl(&mut map, Ok("p"), Ok(Scalar::from(2i32))).unwrap(); + + assert_eq!(map.inner.len(), 1); + assert_eq!(map.inner.get("p"), Some(&Scalar::Integer(2))); + } + + #[test] + fn insert_propagates_name_error_without_mutating_map() { + let mut map = PartitionValueMap { + inner: HashMap::new(), + }; + let result = partition_value_map_insert_impl( + &mut map, + Err(Error::generic("bad name")), + Ok(Scalar::from(1i32)), + ); + assert!(result.is_err()); + assert!(map.inner.is_empty()); + } + + #[test] + fn insert_propagates_scalar_error_without_mutating_map() { + let mut map = PartitionValueMap { + inner: HashMap::new(), + }; + // precision 0 is invalid for a decimal. + let result = partition_value_map_insert_impl(&mut map, Ok("d"), Scalar::decimal(1, 0, 0)); + assert!(result.is_err()); + assert!(map.inner.is_empty()); + } + + #[test] + fn null_insert_builds_typed_null_scalar() { + let mut map = PartitionValueMap { + inner: HashMap::new(), + }; + let scalar = NullTypeTag::try_from(NullTypeTag::Integer as u8) + .and_then(|tag| tag.to_data_type(0, 0)) + .map(Scalar::Null); + partition_value_map_insert_impl(&mut map, Ok("p"), scalar).unwrap(); + assert_eq!( + map.inner.get("p"), + Some(&Scalar::Null(delta_kernel::schema::DataType::INTEGER)) + ); + } + + // The following tests drive the typed `extern "C"` wrappers through their real signatures + // (which require an engine handle for error allocation), guarding against the easy + // copy-paste hazards: a wrong `Scalar` variant in a wrapper, a botched decimal hi/lo + // recombination, or a mis-dispatched null type tag. They build a throwaway in-memory engine. + #[cfg(feature = "default-engine-base")] + mod ffi { + use delta_kernel::schema::{DataType, PrimitiveType}; + + use super::*; + use crate::error::{ExternResult, KernelError}; + use crate::ffi_test_utils::{ok_or_panic, recover_error}; + use crate::kernel_string_slice; + use crate::tests::get_default_engine; + + #[test] + fn typed_wrappers_construct_expected_scalars() { + let engine = get_default_engine("memory:///partition_value_test/"); + let map = partition_value_map_new(); + let binary = [1u8, 2, 3]; + // A decimal value whose magnitude exceeds 64 bits, so the high word must be used. + let big = 1i128 << 64; + + unsafe { + let n = "i"; + ok_or_panic(partition_value_map_insert_int( + map.shallow_copy(), + kernel_string_slice!(n), + 9, + engine.shallow_copy(), + )); + let n = "l"; + ok_or_panic(partition_value_map_insert_long( + map.shallow_copy(), + kernel_string_slice!(n), + 10, + engine.shallow_copy(), + )); + let n = "sh"; + ok_or_panic(partition_value_map_insert_short( + map.shallow_copy(), + kernel_string_slice!(n), + 11, + engine.shallow_copy(), + )); + let n = "by"; + ok_or_panic(partition_value_map_insert_byte( + map.shallow_copy(), + kernel_string_slice!(n), + 12, + engine.shallow_copy(), + )); + let n = "f"; + ok_or_panic(partition_value_map_insert_float( + map.shallow_copy(), + kernel_string_slice!(n), + 1.5, + engine.shallow_copy(), + )); + let n = "d"; + ok_or_panic(partition_value_map_insert_double( + map.shallow_copy(), + kernel_string_slice!(n), + 2.5, + engine.shallow_copy(), + )); + let n = "b"; + ok_or_panic(partition_value_map_insert_bool( + map.shallow_copy(), + kernel_string_slice!(n), + true, + engine.shallow_copy(), + )); + let n = "dt"; + ok_or_panic(partition_value_map_insert_date( + map.shallow_copy(), + kernel_string_slice!(n), + 19723, + engine.shallow_copy(), + )); + let n = "ts"; + ok_or_panic(partition_value_map_insert_timestamp( + map.shallow_copy(), + kernel_string_slice!(n), + 1000, + engine.shallow_copy(), + )); + let n = "tsntz"; + ok_or_panic(partition_value_map_insert_timestamp_ntz( + map.shallow_copy(), + kernel_string_slice!(n), + 2000, + engine.shallow_copy(), + )); + let n = "bin"; + ok_or_panic(partition_value_map_insert_binary( + map.shallow_copy(), + kernel_string_slice!(n), + binary.as_ptr(), + binary.len(), + engine.shallow_copy(), + )); + // An empty binary value may arrive as (null, 0); it must not deref the pointer. + let n = "bin_empty"; + ok_or_panic(partition_value_map_insert_binary( + map.shallow_copy(), + kernel_string_slice!(n), + std::ptr::null(), + 0, + engine.shallow_copy(), + )); + let n = "dec"; + ok_or_panic(partition_value_map_insert_decimal( + map.shallow_copy(), + kernel_string_slice!(n), + 1, // value_hi + 0, // value_lo + 38, + 0, + engine.shallow_copy(), + )); + let n = "s"; + let v = "hello"; + ok_or_panic(partition_value_map_insert_string( + map.shallow_copy(), + kernel_string_slice!(n), + kernel_string_slice!(v), + engine.shallow_copy(), + )); + } + + let inner = &unsafe { map.as_ref() }.inner; + assert_eq!(inner.get("i"), Some(&Scalar::Integer(9))); + assert_eq!(inner.get("l"), Some(&Scalar::Long(10))); + assert_eq!(inner.get("sh"), Some(&Scalar::Short(11))); + assert_eq!(inner.get("by"), Some(&Scalar::Byte(12))); + assert_eq!(inner.get("f"), Some(&Scalar::Float(1.5))); + assert_eq!(inner.get("d"), Some(&Scalar::Double(2.5))); + assert_eq!(inner.get("b"), Some(&Scalar::Boolean(true))); + assert_eq!(inner.get("dt"), Some(&Scalar::Date(19723))); + assert_eq!(inner.get("ts"), Some(&Scalar::Timestamp(1000))); + assert_eq!(inner.get("tsntz"), Some(&Scalar::TimestampNtz(2000))); + assert_eq!(inner.get("bin"), Some(&Scalar::Binary(vec![1, 2, 3]))); + assert_eq!(inner.get("bin_empty"), Some(&Scalar::Binary(vec![]))); + assert_eq!( + inner.get("dec"), + Some(&Scalar::decimal(big, 38, 0).unwrap()) + ); + assert_eq!(inner.get("s"), Some(&Scalar::String("hello".into()))); + + unsafe { + free_partition_value_map(map); + crate::free_engine(engine); + } + } + + #[test] + fn null_wrapper_dispatches_type_tags_and_rejects_invalid() { + let engine = get_default_engine("memory:///partition_value_null_test/"); + let map = partition_value_map_new(); + + // Decimal null carries precision/scale through to the data type. + let n = "dec_null"; + ok_or_panic(unsafe { + partition_value_map_insert_null( + map.shallow_copy(), + kernel_string_slice!(n), + NullTypeTag::Decimal as u8, + 10, + 2, + engine.shallow_copy(), + ) + }); + let expected = + Scalar::Null(DataType::Primitive(PrimitiveType::decimal(10, 2).unwrap())); + assert_eq!( + unsafe { map.as_ref() }.inner.get("dec_null"), + Some(&expected) + ); + + // The non-primitive sentinel and unrecognized tags are rejected. + let expect_err = |result: ExternResult| match result { + ExternResult::Err(e) => unsafe { recover_error(e) }.etype, + ExternResult::Ok(_) => panic!("expected error"), + }; + let n = "bad"; + let err = expect_err(unsafe { + partition_value_map_insert_null( + map.shallow_copy(), + kernel_string_slice!(n), + NullTypeTag::NonPrimitive as u8, + 0, + 0, + engine.shallow_copy(), + ) + }); + assert_eq!(err, KernelError::GenericError); + let err = expect_err(unsafe { + partition_value_map_insert_null( + map.shallow_copy(), + kernel_string_slice!(n), + 200, // unrecognized tag + 0, + 0, + engine.shallow_copy(), + ) + }); + assert_eq!(err, KernelError::GenericError); + // Failed inserts did not add the "bad" key. + assert!(!unsafe { map.as_ref() }.inner.contains_key("bad")); + + unsafe { + free_partition_value_map(map); + crate::free_engine(engine); + } + } + } +} diff --git a/ffi/src/transaction/transaction_id.rs b/ffi/src/transaction/transaction_id.rs index 32b234cade..8b6c0b9e17 100644 --- a/ffi/src/transaction/transaction_id.rs +++ b/ffi/src/transaction/transaction_id.rs @@ -1,5 +1,8 @@ use std::sync::Arc; +use delta_kernel::transaction::Transaction; +use delta_kernel::{DeltaResult, Snapshot}; + use crate::error::ExternResult; use crate::handle::Handle; use crate::transaction::ExclusiveTransaction; @@ -8,17 +11,15 @@ use crate::{ SharedSnapshot, TryFromStringSlice, }; -use delta_kernel::transaction::Transaction; -use delta_kernel::{DeltaResult, Snapshot}; - -/// Associates an app_id and version with a transaction. These will be applied to the table on commit. +/// Associates an app_id and version with a transaction. These will be applied to the table on +/// commit. /// /// # Returns /// A new handle to the transaction that will set the `app_id` version to `version` on commit /// /// # Safety -/// Caller is responsible for passing [valid][Handle#Validity] handles. The `app_id` string slice must be valid. -/// CONSUMES TRANSACTION +/// Caller is responsible for passing [valid][Handle#Validity] handles. The `app_id` string slice +/// must be valid. CONSUMES TRANSACTION #[no_mangle] pub unsafe extern "C" fn with_transaction_id( txn: Handle, @@ -46,8 +47,8 @@ fn with_transaction_id_impl( /// The version number if found, or an error of type `MissingDataError` when the app_id was not set /// /// # Safety -/// Caller must ensure [valid][Handle#Validity] handles are provided for snapshot and engine. The `app_id` -/// string slice must be valid. +/// Caller must ensure [valid][Handle#Validity] handles are provided for snapshot and engine. The +/// `app_id` string slice must be valid. #[no_mangle] pub unsafe extern "C" fn get_app_id_version( snapshot: Handle, @@ -73,43 +74,35 @@ fn get_app_id_version_impl( #[cfg(test)] mod tests { - use super::*; + use std::sync::Arc; - use crate::ffi_test_utils::ok_or_panic; - use crate::kernel_string_slice; - use crate::tests::get_default_engine; - use crate::transaction::{commit, transaction}; use delta_kernel::schema::{DataType, StructField, StructType}; use delta_kernel::Snapshot; - use std::sync::Arc; - use tempfile::tempdir; use test_utils::setup_test_tables; - use url::Url; + + use super::*; + use crate::ffi_test_utils::{engine_handle_for_store, ok_or_panic}; + use crate::transaction::{commit, free_committed_transaction, transaction}; + use crate::{free_engine, free_snapshot, kernel_string_slice}; #[cfg(feature = "default-engine-base")] #[tokio::test] - #[cfg_attr(miri, ignore)] // FIXME: re-enable miri (can't call foreign function `linkat` on OS `linux`) async fn test_write_txn_actions() -> Result<(), Box> { - // Create a temporary local directory for use during this test - let tmp_test_dir = tempdir()?; - let tmp_dir_local_url = Url::from_directory_path(tmp_test_dir.path()).unwrap(); - // create a simple table: one int column named 'number' let schema = Arc::new( StructType::try_new(vec![StructField::nullable("number", DataType::INTEGER)]).unwrap(), ); - for (table_url, engine, _store, _table_name) in - setup_test_tables(schema, &[], Some(&tmp_dir_local_url), "test_table").await? + for (table_url, engine, store, _table_name) in + setup_test_tables(schema, &[], None, "test_table").await? { - let table_path = table_url.to_file_path().unwrap(); - let table_path_str = table_path.to_str().unwrap(); - let default_engine_handle = get_default_engine(table_path_str); + let table_url_str = table_url.as_str(); + let default_engine_handle = engine_handle_for_store(store); // Start the transaction let txn = ok_or_panic(unsafe { transaction( - kernel_string_slice!(table_path_str), + kernel_string_slice!(table_url_str), default_engine_handle.shallow_copy(), ) }); @@ -135,7 +128,9 @@ mod tests { }); // commit! - ok_or_panic(unsafe { commit(txn, default_engine_handle.shallow_copy()) }); + let committed = + ok_or_panic(unsafe { commit(txn, default_engine_handle.shallow_copy()) }); + unsafe { free_committed_transaction(committed) }; let snapshot: Arc = Snapshot::builder_for(table_url.clone()) .at_version(1) @@ -146,10 +141,12 @@ mod tests { assert_eq!(snapshot.get_app_id_version("app_id2", &engine)?, Some(2)); assert_eq!(snapshot.get_app_id_version("app_id3", &engine)?, None); - // Check versions through ffi handles + // Check versions through ffi handles. `get_app_id_version` borrows the handle, so + // one handle serves all three calls and is freed once at the end. + let snapshot_handle: Handle = snapshot.clone().into(); let version1 = ok_or_panic(unsafe { get_app_id_version( - Handle::from(snapshot.clone()), + snapshot_handle.shallow_copy(), kernel_string_slice!(app_id1), default_engine_handle.shallow_copy(), ) @@ -158,7 +155,7 @@ mod tests { let version2 = ok_or_panic(unsafe { get_app_id_version( - Handle::from(snapshot.clone()), + snapshot_handle.shallow_copy(), kernel_string_slice!(app_id2), default_engine_handle.shallow_copy(), ) @@ -168,12 +165,15 @@ mod tests { let app_id3 = "app_id3"; let version3 = ok_or_panic(unsafe { get_app_id_version( - Handle::from(snapshot.clone()), + snapshot_handle.shallow_copy(), kernel_string_slice!(app_id3), default_engine_handle.shallow_copy(), ) }); assert_eq!(version3, OptionalValue::None); + + unsafe { free_snapshot(snapshot_handle) }; + unsafe { free_engine(default_engine_handle) }; } Ok(()) } diff --git a/ffi/src/transaction/write_context.rs b/ffi/src/transaction/write_context.rs index d41666baac..0d0b69558a 100644 --- a/ffi/src/transaction/write_context.rs +++ b/ffi/src/transaction/write_context.rs @@ -1,11 +1,20 @@ -use crate::handle::Handle; -use crate::{kernel_string_slice, AllocateStringFn, NullableCvoid, SharedSchema}; +use std::collections::HashMap; +use std::sync::Arc; + +use delta_kernel::expressions::Scalar; use delta_kernel::transaction::WriteContext; +use delta_kernel::{DeltaResult, Error}; use delta_kernel_ffi_macros::handle_descriptor; -use std::sync::Arc; - -use super::ExclusiveTransaction; +use super::partition_value::{ExclusivePartitionValueMap, PartitionValueMap}; +use super::{ExclusiveCreateTransaction, ExclusiveTransaction}; +use crate::error::{ExternResult, IntoExternResult}; +use crate::expressions::SharedExpression; +use crate::handle::Handle; +use crate::{ + kernel_string_slice, AllocateStringFn, KernelStringSlice, NullableCvoid, SharedExternEngine, + SharedSchema, TryFromStringSlice, Url, +}; /// A [`WriteContext`] that provides schema and path information needed for writing data. /// This is a shared reference that can be cloned and used across multiple consumers. @@ -14,18 +23,105 @@ use super::ExclusiveTransaction; #[handle_descriptor(target=WriteContext, mutable=false, sized=true)] pub struct SharedWriteContext; -/// Gets the write context from a transaction. The write context provides schema and path information -/// needed for writing data. +/// Gets the write context from a transaction for an unpartitioned table. The write context +/// provides schema and path information needed for writing data. +/// +/// For partitioned tables, use [`get_partitioned_write_context`] instead. Returns an error if the +/// table is partitioned. +/// +/// # Safety +/// +/// Caller is responsible for passing a [valid][Handle#Validity] transaction handle and engine. +#[no_mangle] +pub unsafe extern "C" fn get_unpartitioned_write_context( + txn: Handle, + engine: Handle, +) -> ExternResult> { + let txn = unsafe { txn.as_ref() }; + let engine = unsafe { engine.as_ref() }; + txn.unpartitioned_write_context() + .map(|wc| Arc::new(wc).into()) + .into_extern_result(&engine) +} + +/// Gets the write context from a create-table transaction for an unpartitioned table. +/// +/// For partitioned tables, use [`create_table_get_partitioned_write_context`] instead. Returns an +/// error if the table is partitioned. +/// +/// # Safety +/// +/// Caller is responsible for passing a [valid][Handle#Validity] transaction handle and engine. +#[no_mangle] +pub unsafe extern "C" fn create_table_get_unpartitioned_write_context( + txn: Handle, + engine: Handle, +) -> ExternResult> { + let txn = unsafe { txn.as_ref() }; + let engine = unsafe { engine.as_ref() }; + txn.unpartitioned_write_context() + .map(|wc| Arc::new(wc).into()) + .into_extern_result(&engine) +} + +/// Gets the write context from a transaction for a partitioned table, for the partition described +/// by `partition_values`. A separate write context (and write directory) is needed per partition, +/// so call this once per distinct set of partition values. +/// +/// `partition_values` maps each partition column's logical name to its value; build it with +/// [`partition_value_map_new`](super::partition_value::partition_value_map_new) and the +/// `partition_value_map_insert_*` functions. The map must contain exactly the table's partition +/// columns (the kernel validates completeness and value types and rejects extras). This function +/// consumes the map handle on both success and error; do not use or free it afterward. +/// +/// Returns an error if the table is not partitioned (use [`get_unpartitioned_write_context`] +/// instead) or if the partition values are invalid for the table's partition schema. /// /// # Safety /// -/// Caller is responsible for passing a [valid][Handle#Validity] transaction handle. +/// Caller is responsible for passing a [valid][Handle#Validity] transaction handle, partition +/// value map handle, and engine. #[no_mangle] -pub unsafe extern "C" fn get_write_context( +pub unsafe extern "C" fn get_partitioned_write_context( txn: Handle, -) -> Handle { + partition_values: Handle, + engine: Handle, +) -> ExternResult> { let txn = unsafe { txn.as_ref() }; - Arc::new(txn.get_write_context()).into() + let partition_values = unsafe { partition_values.into_inner() }; + let engine = unsafe { engine.as_ref() }; + partitioned_write_context_impl(|pv| txn.partitioned_write_context(pv), *partition_values) + .into_extern_result(&engine) +} + +/// Gets the write context from a create-table transaction for a partitioned table. See +/// [`get_partitioned_write_context`] for the contract; this is the create-table counterpart. +/// +/// # Safety +/// +/// Caller is responsible for passing a [valid][Handle#Validity] transaction handle, partition +/// value map handle, and engine. +#[no_mangle] +pub unsafe extern "C" fn create_table_get_partitioned_write_context( + txn: Handle, + partition_values: Handle, + engine: Handle, +) -> ExternResult> { + let txn = unsafe { txn.as_ref() }; + let partition_values = unsafe { partition_values.into_inner() }; + let engine = unsafe { engine.as_ref() }; + partitioned_write_context_impl(|pv| txn.partitioned_write_context(pv), *partition_values) + .into_extern_result(&engine) +} + +/// Shared body for the partitioned write-context entry points: hand the owned partition values to +/// the transaction's `partitioned_write_context` and wrap the result in a shared handle. +fn partitioned_write_context_impl( + build: impl FnOnce(HashMap) -> DeltaResult, + partition_values: PartitionValueMap, +) -> DeltaResult> { + let wc = build(partition_values.inner)?; + Ok(Arc::new(wc).into()) } #[no_mangle] @@ -33,8 +129,11 @@ pub unsafe extern "C" fn free_write_context(write_context: Handle`) and each field has a `parquet.field.id` metadata entry per the Delta +/// column-mapping spec; otherwise it matches the logical schema. Partition columns are +/// excluded unless the `materializePartitionColumns` writer feature or `IcebergCompatV3` is +/// enabled. +/// +/// Use this as the parquet writer schema and as the output schema of the evaluator built +/// from [`get_logical_to_physical`]. +/// +/// The returned schema must be freed via [`crate::free_schema`]. +/// +/// # Safety +/// Engine is responsible for providing a valid WriteContext pointer +#[no_mangle] +pub unsafe extern "C" fn get_physical_write_schema( + write_context: Handle, +) -> Handle { + let write_context = unsafe { write_context.as_ref() }; + write_context.physical_schema().clone().into() +} + +/// Returns the logical-to-physical expression from a [`WriteContext`] handle. Engines apply +/// it via an [`ExpressionEvaluator`] to each batch of logical data before writing parquet. +/// The logical data batches must not contain partition columns. The column rename itself is encoded +/// in the physical schema (the evaluator matches input columns to output fields by position), not +/// in this expression. +/// +/// To build the evaluator, pass the schema of the partition-free input data as the input, this +/// value as the expression to evaluate, and [`get_physical_write_schema`] as the output. See +/// [`crate::engine_funcs::new_expression_evaluator`]. +/// +/// The returned expression must be freed via [`crate::expressions::free_kernel_expression`]. +/// +/// # Safety +/// Engine is responsible for providing a valid WriteContext pointer +/// +/// [`ExpressionEvaluator`]: delta_kernel::ExpressionEvaluator +#[no_mangle] +pub unsafe extern "C" fn get_logical_to_physical( + write_context: Handle, +) -> Handle { + let write_context = unsafe { write_context.as_ref() }; + write_context.logical_to_physical().into() +} + +/// Get the table root URL from a WriteContext handle. Returns the table root, not the +/// recommended write directory (which may include Hive-style partition paths or random +/// prefixes); use [`get_write_dir`] for the latter. /// /// # Safety /// Engine is responsible for providing a valid WriteContext pointer @@ -56,6 +203,101 @@ pub unsafe extern "C" fn get_write_path( allocate_fn: AllocateStringFn, ) -> NullableCvoid { let write_context = unsafe { write_context.as_ref() }; - let write_path = write_context.target_dir().to_string(); + let write_path = write_context.table_root_dir().to_string(); allocate_fn(kernel_string_slice!(write_path)) } + +/// Get the recommended directory URL for writing data files from a WriteContext handle. +/// Connectors should write files as `/.parquet`. For a partitioned write context +/// this includes the Hive-style partition prefix (e.g. `year=2024/`) when column mapping is off, or +/// a random prefix when column mapping or `delta.randomizeFilePrefixes` is on. +/// +/// The returned URL is URI-encoded. Engines that write to a local filesystem must URI-decode it +/// once before using it as a path; the still-encoded URL (plus the file name) is what +/// [`resolve_file_path`] expects to produce the `add.path` recorded in the Delta log. +/// +/// A fresh random prefix is generated on each call when column mapping or random prefixes are +/// enabled, so call this once per file batch and reuse the result. +/// +/// # Safety +/// Engine is responsible for providing a valid WriteContext pointer +#[no_mangle] +pub unsafe extern "C" fn get_write_dir( + write_context: Handle, + allocate_fn: AllocateStringFn, +) -> NullableCvoid { + let write_context = unsafe { write_context.as_ref() }; + let write_dir = write_context.write_dir().to_string(); + allocate_fn(kernel_string_slice!(write_dir)) +} + +/// Visit the serialized partition values of a WriteContext handle by invoking `visitor` once per +/// partition column. Keys are *physical* column names (column-mapping applied) and values are the +/// protocol-serialized strings the engine must record in each Add action's `partitionValues`. When +/// a partition value is null, `is_null` is `true` and `value` is an empty slice. For an +/// unpartitioned write context, `visitor` is never called. Entries are visited in sorted key order +/// so the callback sequence is deterministic across runs. +/// +/// # Safety +/// Engine is responsible for providing a valid WriteContext pointer, a valid `engine_context` +/// pointer passed through to each `visitor` invocation, and a valid `visitor` function pointer. +#[no_mangle] +pub unsafe extern "C" fn visit_partition_values( + write_context: Handle, + engine_context: NullableCvoid, + visitor: extern "C" fn( + engine_context: NullableCvoid, + key: KernelStringSlice, + value: KernelStringSlice, + is_null: bool, + ), +) { + let write_context = unsafe { write_context.as_ref() }; + let values = write_context.physical_partition_values(); + let mut keys: Vec<&String> = values.keys().collect(); + keys.sort(); + for key in keys { + let value = &values[key]; + let value_str = value.as_deref().unwrap_or(""); + visitor( + engine_context, + kernel_string_slice!(key), + kernel_string_slice!(value_str), + value.is_none(), + ); + } +} + +/// Compute the relative `add.path` for the Delta log from the absolute URL of a data file the +/// engine has written. `file_url` is the full (URI-encoded) URL of the written file, typically +/// formed by appending the file name to [`get_write_dir`]'s result. +/// +/// Returns an error if `file_url` is not a valid URL or does not live under the table root. +/// +/// # Safety +/// Engine is responsible for providing a valid WriteContext pointer, a valid `file_url` slice, and +/// a valid engine handle. +#[no_mangle] +pub unsafe extern "C" fn resolve_file_path( + write_context: Handle, + file_url: KernelStringSlice, + allocate_fn: AllocateStringFn, + engine: Handle, +) -> ExternResult { + let write_context = unsafe { write_context.as_ref() }; + let engine = unsafe { engine.as_ref() }; + let file_url: DeltaResult<&str> = unsafe { TryFromStringSlice::try_from_slice(&file_url) }; + resolve_file_path_impl(write_context, file_url) + .map(|path| allocate_fn(kernel_string_slice!(path))) + .into_extern_result(&engine) +} + +fn resolve_file_path_impl( + write_context: &WriteContext, + file_url: DeltaResult<&str>, +) -> DeltaResult { + let url = Url::parse(file_url?).map_err(|e| { + Error::generic(format!("invalid file URL passed to resolve_file_path: {e}")) + })?; + write_context.resolve_file_path(&url) +} diff --git a/ffi/tests/checkpoint-table-testing/expected-data/default.expected b/ffi/tests/checkpoint-table-testing/expected-data/default.expected new file mode 100644 index 0000000000..f5e7f16d9e --- /dev/null +++ b/ffi/tests/checkpoint-table-testing/expected-data/default.expected @@ -0,0 +1 @@ +sub_flow=default result=Written version=1 diff --git a/ffi/tests/checkpoint-table-testing/expected-data/v2_no_sidecar.expected b/ffi/tests/checkpoint-table-testing/expected-data/v2_no_sidecar.expected new file mode 100644 index 0000000000..fdaea1d9f3 --- /dev/null +++ b/ffi/tests/checkpoint-table-testing/expected-data/v2_no_sidecar.expected @@ -0,0 +1 @@ +sub_flow=v2_no_sidecar result=Written version=5 diff --git a/ffi/tests/checkpoint-table-testing/expected-data/v2_with_sidecars.expected b/ffi/tests/checkpoint-table-testing/expected-data/v2_with_sidecars.expected new file mode 100644 index 0000000000..c9d8afded2 --- /dev/null +++ b/ffi/tests/checkpoint-table-testing/expected-data/v2_with_sidecars.expected @@ -0,0 +1 @@ +sub_flow=v2_with_sidecars result=Written version=5 diff --git a/ffi/tests/checkpoint-table-testing/run_test.sh b/ffi/tests/checkpoint-table-testing/run_test.sh new file mode 100755 index 0000000000..341c8af91e --- /dev/null +++ b/ffi/tests/checkpoint-table-testing/run_test.sh @@ -0,0 +1,74 @@ +#!/bin/bash + +set -euxo pipefail + +# Usage: run_test.sh +# +# Seeds a temp dir by copying the appropriate kernel-bundled fixture, scrubs any +# pre-existing checkpoint state (V1 + V2 manifest, sidecars, _last_checkpoint hint) +# so the C binary exercises checkpoint EMISSION rather than READING, runs +# ./checkpoint_table against it, then diffs output against the expected file. +# +# Fixtures: +# default -> kernel/tests/data/app-txn-no-checkpoint/ (non-V2, no checkpoint) +# v2_no_sidecar -> kernel/tests/data/v2-parquet-sidecars-struct-stats-only/ +# v2_with_sidecars -> kernel/tests/data/v2-parquet-sidecars-struct-stats-only/ + +SUB_FLOW="$1" +EXPECTED="$2" +shift 2 + +# Resolve the kernel-data root relative to this script. The runner lives at +# ffi/tests/checkpoint-table-testing/; the fixtures live at kernel/tests/data/. +SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd -P) +KERNEL_ROOT=$(cd "$SCRIPT_DIR/../../.." && pwd -P) + +case "$SUB_FLOW" in + default) + FIXTURE="$KERNEL_ROOT/kernel/tests/data/app-txn-no-checkpoint" + ;; + v2_no_sidecar|v2_with_sidecars) + FIXTURE="$KERNEL_ROOT/kernel/tests/data/v2-parquet-sidecars-struct-stats-only" + ;; + *) + echo "ERROR: unknown sub-flow $SUB_FLOW" >&2 + exit 1 + ;; +esac + +if [ ! -d "$FIXTURE" ]; then + echo "ERROR: fixture not found at $FIXTURE" >&2 + exit 1 +fi + +TABLE_DIR=$(mktemp -d "${TMPDIR:-/tmp}/checkpoint_table_example.XXXXXX") +# macOS resolves /var -> /private/var when canonicalising paths inside the kernel; match the +# write-table example's pattern. +TABLE_DIR=$(cd "$TABLE_DIR" && pwd -P) +OUT_FILE=$(mktemp) +trap 'rm -rf "$TABLE_DIR" "$OUT_FILE"' EXIT + +# Seed by copying the entire fixture. +cp -R "$FIXTURE"/. "$TABLE_DIR/" + +# Scrub any pre-existing checkpoint artifacts so the test exercises EMISSION, not READING. +# Cover both V1-shape (`.checkpoint.parquet`) and V2-shape +# (`.checkpoint..parquet`) manifest filenames, plus the sidecar directory and +# the _last_checkpoint hint file. The V1 glob is harmless on V2 fixtures; the V2 glob is +# harmless on V1 fixtures; both are kept for shape-agnostic correctness. +rm -f "$TABLE_DIR"/_delta_log/*.checkpoint.parquet +rm -f "$TABLE_DIR"/_delta_log/*.checkpoint.*.parquet +rm -rf "$TABLE_DIR"/_delta_log/_sidecars/ +rm -f "$TABLE_DIR"/_delta_log/_last_checkpoint + +# Sanity check: scrub left no checkpoint artifacts behind. +if ls "$TABLE_DIR"/_delta_log/ | grep -q checkpoint; then + echo "ERROR: scrub failed -- checkpoint artifacts remain in $TABLE_DIR/_delta_log/" >&2 + ls -la "$TABLE_DIR"/_delta_log/ >&2 + exit 1 +fi + +./checkpoint_table "$TABLE_DIR" "$SUB_FLOW" \ + | sed -E "s|$TABLE_DIR||g" \ + | tee "$OUT_FILE" +diff "$OUT_FILE" "$EXPECTED" diff --git a/ffi/tests/create-table-testing/expected-data/basic.expected b/ffi/tests/create-table-testing/expected-data/basic.expected new file mode 100644 index 0000000000..ca40eaebc2 --- /dev/null +++ b/ffi/tests/create-table-testing/expected-data/basic.expected @@ -0,0 +1,4 @@ +Creating table at
    +Committed version: 0 +Snapshot version after create: 0 +Partition columns: 0 diff --git a/ffi/tests/create-table-testing/run_test.sh b/ffi/tests/create-table-testing/run_test.sh new file mode 100755 index 0000000000..375f341de7 --- /dev/null +++ b/ffi/tests/create-table-testing/run_test.sh @@ -0,0 +1,22 @@ +#!/bin/bash + +set -euxo pipefail + +# Usage: run_test.sh +# +# Creates a fresh temp directory, runs ./create_table against it, and diffs output against +# the expected file. The temp dir is cleaned up regardless of success. +EXPECTED="$1" +shift + +TABLE_DIR=$(mktemp -d "${TMPDIR:-/tmp}/create_table_example.XXXXXX") +# Resolve symlinks (e.g. macOS /var -> /private/var) so the sed substitution matches paths +# the kernel canonicalises in any printed output. `cd && pwd -P` is portable to BSD/GNU. +TABLE_DIR=$(cd "$TABLE_DIR" && pwd -P) +OUT_FILE=$(mktemp) +trap 'rm -rf "$TABLE_DIR" "$OUT_FILE"' EXIT + +./create_table "$TABLE_DIR" \ + | sed -E "s|$TABLE_DIR|
    |g" \ + | tee "$OUT_FILE" +diff "$OUT_FILE" "$EXPECTED" diff --git a/ffi/tests/invalid-handle-code/mut-clone-handle.stderr b/ffi/tests/invalid-handle-code/mut-clone-handle.stderr index 7e09c5581b..a7838bc1ac 100644 --- a/ffi/tests/invalid-handle-code/mut-clone-handle.stderr +++ b/ffi/tests/invalid-handle-code/mut-clone-handle.stderr @@ -2,13 +2,13 @@ error[E0599]: the method `clone_as_arc` exists for struct `Handle`, but --> tests/invalid-handle-code/mut-clone-handle.rs:13:15 | 8 | pub struct MutFoo; - | ----------------- doesn't satisfy `::Mutable = False` + | ----------------- doesn't satisfy `<_ as HandleDescriptor>::Mutable = False` ... 13 | let r = h.clone_as_arc(); | ^^^^^^^^^^^^ method cannot be called on `Handle` due to unsatisfied trait bounds | = note: the following trait bounds were not satisfied: - `::Mutable = False` + `::Mutable = delta_kernel_ffi::handle::False` error[E0271]: type mismatch resolving `::Mutable == False` --> tests/invalid-handle-code/mut-clone-handle.rs:12:41 @@ -16,7 +16,7 @@ error[E0271]: type mismatch resolving `::Mutable == 12 | let h: Handle = Arc::new(s).into(); | ^^^^ type mismatch resolving `::Mutable == False` | -note: expected this to be `False` +note: expected this to be `delta_kernel_ffi::handle::False` --> tests/invalid-handle-code/mut-clone-handle.rs:7:1 | 7 | #[handle_descriptor(target=Foo, mutable=true, sized=true)] @@ -29,10 +29,10 @@ error[E0599]: the method `clone_handle` exists for struct `Handle`, but --> tests/invalid-handle-code/mut-clone-handle.rs:14:15 | 8 | pub struct MutFoo; - | ----------------- doesn't satisfy `::Mutable = False` + | ----------------- doesn't satisfy `<_ as HandleDescriptor>::Mutable = False` ... 14 | let h = h.clone_handle(); | ^^^^^^^^^^^^ | = note: the following trait bounds were not satisfied: - `::Mutable = False` + `::Mutable = delta_kernel_ffi::handle::False` diff --git a/ffi/tests/invalid-handle-code/shared-as-mut.stderr b/ffi/tests/invalid-handle-code/shared-as-mut.stderr index d1b72ffe0d..7f82045a02 100644 --- a/ffi/tests/invalid-handle-code/shared-as-mut.stderr +++ b/ffi/tests/invalid-handle-code/shared-as-mut.stderr @@ -2,13 +2,13 @@ error[E0599]: the method `as_mut` exists for struct `Handle`, but its --> tests/invalid-handle-code/shared-as-mut.rs:12:15 | 7 | pub struct SharedFoo; - | -------------------- doesn't satisfy `::Mutable = True` + | -------------------- doesn't satisfy `<_ as HandleDescriptor>::Mutable = True` ... 12 | let r = h.as_mut(); | ^^^^^^ method cannot be called on `Handle` due to unsatisfied trait bounds | = note: the following trait bounds were not satisfied: - `::Mutable = True` + `::Mutable = delta_kernel_ffi::handle::True` error[E0271]: type mismatch resolving `::Mutable == True` --> tests/invalid-handle-code/shared-as-mut.rs:11:44 @@ -16,7 +16,7 @@ error[E0271]: type mismatch resolving `::Mutable 11 | let h: Handle = Box::new(s).into(); | ^^^^ type mismatch resolving `::Mutable == True` | -note: expected this to be `True` +note: expected this to be `delta_kernel_ffi::handle::True` --> tests/invalid-handle-code/shared-as-mut.rs:6:1 | 6 | #[handle_descriptor(target=Foo, mutable=false, sized=true)] diff --git a/ffi/tests/invalid-handle-code/shared-not-sync.stderr b/ffi/tests/invalid-handle-code/shared-not-sync.stderr index a3f4b5e8d9..1ddf9356ee 100644 --- a/ffi/tests/invalid-handle-code/shared-not-sync.stderr +++ b/ffi/tests/invalid-handle-code/shared-not-sync.stderr @@ -9,7 +9,7 @@ error[E0599]: the method `clone_as_arc` exists for struct `Handle | = note: the following trait bounds were not satisfied: `*mut u32: Sync` - which is required by `SharedNotSync: handle::private::SharedHandleOps` + which is required by `SharedNotSync: handle::private::SharedHandleOps` error[E0277]: `*mut u32` cannot be shared between threads safely --> tests/invalid-handle-code/shared-not-sync.rs:14:48 @@ -23,7 +23,7 @@ note: required because it appears within the type `NotSync` | 5 | pub struct NotSync(*mut u32); | ^^^^^^^ - = note: required for `SharedNotSync` to implement `handle::private::SharedHandleOps` + = note: required for `SharedNotSync` to implement `handle::private::SharedHandleOps` = note: required for `Handle` to implement `From>` = note: required for `Arc` to implement `Into>` @@ -38,4 +38,4 @@ error[E0599]: the method `clone_handle` exists for struct `Handle | = note: the following trait bounds were not satisfied: `*mut u32: Sync` - which is required by `SharedNotSync: handle::private::SharedHandleOps` + which is required by `SharedNotSync: handle::private::SharedHandleOps` diff --git a/ffi/tests/read-table-changes-testing/expected-data/cdf-from-v1.expected b/ffi/tests/read-table-changes-testing/expected-data/cdf-from-v1.expected new file mode 100644 index 0000000000..8030c0a157 --- /dev/null +++ b/ffi/tests/read-table-changes-testing/expected-data/cdf-from-v1.expected @@ -0,0 +1,13 @@ +Reading change data feed from ../../../../acceptance/tests/dat/out/reader_tests/generated/cdf/delta/ + requested start_version: 1 + requested end_version: +TableChanges: + table_root: file:///out/reader_tests/generated/cdf/delta/ + start_version: 1 + end_version: 2 +Scan: + table_root: file:///out/reader_tests/generated/cdf/delta/ +CDF batches: + batch 0: 5 rows + batch 1: 4 rows +Total: 2 batches, 9 rows diff --git a/ffi/tests/read-table-changes-testing/expected-data/cdf-v1-only.expected b/ffi/tests/read-table-changes-testing/expected-data/cdf-v1-only.expected new file mode 100644 index 0000000000..95d2d8d322 --- /dev/null +++ b/ffi/tests/read-table-changes-testing/expected-data/cdf-v1-only.expected @@ -0,0 +1,12 @@ +Reading change data feed from ../../../../acceptance/tests/dat/out/reader_tests/generated/cdf/delta/ + requested start_version: 1 + requested end_version: 1 +TableChanges: + table_root: file:///out/reader_tests/generated/cdf/delta/ + start_version: 1 + end_version: 1 +Scan: + table_root: file:///out/reader_tests/generated/cdf/delta/ +CDF batches: + batch 0: 5 rows +Total: 1 batches, 5 rows diff --git a/ffi/tests/read-table-changes-testing/expected-data/cdf-v2-only.expected b/ffi/tests/read-table-changes-testing/expected-data/cdf-v2-only.expected new file mode 100644 index 0000000000..a4f12756f8 --- /dev/null +++ b/ffi/tests/read-table-changes-testing/expected-data/cdf-v2-only.expected @@ -0,0 +1,12 @@ +Reading change data feed from ../../../../acceptance/tests/dat/out/reader_tests/generated/cdf/delta/ + requested start_version: 2 + requested end_version: 2 +TableChanges: + table_root: file:///out/reader_tests/generated/cdf/delta/ + start_version: 2 + end_version: 2 +Scan: + table_root: file:///out/reader_tests/generated/cdf/delta/ +CDF batches: + batch 0: 4 rows +Total: 1 batches, 4 rows diff --git a/ffi/tests/read-table-changes-testing/expected-data/cdf.expected b/ffi/tests/read-table-changes-testing/expected-data/cdf.expected new file mode 100644 index 0000000000..c0e10aee4c --- /dev/null +++ b/ffi/tests/read-table-changes-testing/expected-data/cdf.expected @@ -0,0 +1,13 @@ +Reading change data feed from ../../../../acceptance/tests/dat/out/reader_tests/generated/cdf/delta/ + requested start_version: 0 + requested end_version: +TableChanges: + table_root: file:///out/reader_tests/generated/cdf/delta/ + start_version: 0 + end_version: 2 +Scan: + table_root: file:///out/reader_tests/generated/cdf/delta/ +CDF batches: + batch 0: 5 rows + batch 1: 4 rows +Total: 2 batches, 9 rows diff --git a/ffi/tests/read-table-changes-testing/run_test.sh b/ffi/tests/read-table-changes-testing/run_test.sh new file mode 100755 index 0000000000..141a10801e --- /dev/null +++ b/ffi/tests/read-table-changes-testing/run_test.sh @@ -0,0 +1,18 @@ +#!/bin/bash + +set -euxo pipefail + +# Usage: run_test.sh [extra read_table_changes args...] +TABLE_PATH="$1" +EXPECTED="$2" +shift 2 + +OUT_FILE=$(mktemp) +trap 'rm -f "$OUT_FILE"' EXIT + +# table_root lines contain machine-specific absolute paths. Replace everything up to and +# including ".../acceptance/tests/dat/" with "" so expected files are portable. +./read_table_changes "$@" "$TABLE_PATH" \ + | sed -E 's|file://.*/acceptance/tests/dat/|file:///|g' \ + | tee "$OUT_FILE" +diff "$OUT_FILE" "$EXPECTED" diff --git a/ffi/tests/read-table-testing/expected-data/basic-partitioned-arrow-metadata.expected b/ffi/tests/read-table-testing/expected-data/basic-partitioned-arrow-metadata.expected new file mode 100644 index 0000000000..87e77ef7f0 --- /dev/null +++ b/ffi/tests/read-table-testing/expected-data/basic-partitioned-arrow-metadata.expected @@ -0,0 +1,9 @@ +Reading table at ../../../../acceptance/tests/dat/out/reader_tests/generated/basic_partitioned/delta/ +version: 1 + +Schema: +├─ letter: string +├─ number: long +└─ a_float: double + +[No data] diff --git a/ffi/tests/read-table-testing/expected-data/column-mapping-arrow-metadata.expected b/ffi/tests/read-table-testing/expected-data/column-mapping-arrow-metadata.expected new file mode 100644 index 0000000000..bd140c7a08 --- /dev/null +++ b/ffi/tests/read-table-testing/expected-data/column-mapping-arrow-metadata.expected @@ -0,0 +1,9 @@ +Reading table at ../../../../acceptance/tests/dat/out/reader_tests/generated/column_mapping/delta/ +version: 3 + +Schema: +├─ letter: string (column mapping id: 1) +├─ new_int: long (column mapping id: 2) +└─ date: date (column mapping id: 3) + +[No data] diff --git a/ffi/tests/read-table-testing/expected-data/with-short-dv.expected b/ffi/tests/read-table-testing/expected-data/with-short-dv.expected new file mode 100644 index 0000000000..b1c4a9ec40 --- /dev/null +++ b/ffi/tests/read-table-testing/expected-data/with-short-dv.expected @@ -0,0 +1,45 @@ +Reading table at ../../../../kernel/tests/data/with-short-dv/ +version: 0 + +Schema: +├─ id: long +├─ value: string +├─ timestamp: timestamp +└─ rand: double + +id: [ + 3, + 4, + 5, + 6, + 7, + 8, + 9 +] +value: [ + "3", + "4", + "5", + "6", + "7", + "8", + "9" +] +timestamp: [ + 2023-05-31 18:58:33.633000Z, + 2023-05-31 18:58:33.633000Z, + 2023-05-31 18:58:33.633000Z, + 2023-05-31 18:58:33.633000Z, + 2023-05-31 18:58:33.633000Z, + 2023-05-31 18:58:33.633000Z, + 2023-05-31 18:58:33.633000Z +] +rand: [ + 0.7918174793484931, + 0.9281049271981882, + 0.27796520310701633, + 0.15263801464228832, + 0.1981143710215575, + 0.3069439236599195, + 0.5175919190815845 +] diff --git a/ffi/tests/test-uc-catalog-ffi/run_test.sh b/ffi/tests/test-delta-kernel-unity-catalog-ffi/run_test.sh similarity index 60% rename from ffi/tests/test-uc-catalog-ffi/run_test.sh rename to ffi/tests/test-delta-kernel-unity-catalog-ffi/run_test.sh index 26c7a4760e..64113b45e8 100755 --- a/ffi/tests/test-uc-catalog-ffi/run_test.sh +++ b/ffi/tests/test-delta-kernel-unity-catalog-ffi/run_test.sh @@ -4,9 +4,9 @@ set -euxo pipefail OUT_FILE=$(mktemp "${TMPDIR:-/tmp}/catalog_test.out.XXXX") TMP_TABLE_DIR=$(mktemp -d "${TMPDIR:-/tmp}/catalog_test.table.XXXX") -cp -r ../../../../kernel/tests/data/table-with-dv-small "$TMP_TABLE_DIR" +cp -r ../../../../delta-kernel-unity-catalog/tests/data/catalog_managed_0 "$TMP_TABLE_DIR" -./uc_catalog_example "$TMP_TABLE_DIR/table-with-dv-small" +./delta_kernel_unity_catalog_example "$TMP_TABLE_DIR/catalog_managed_0" CATALOG_EXIT_CODE=$? rm "$OUT_FILE" diff --git a/ffi/tests/test-expression-visitor/expected.txt b/ffi/tests/test-expression-visitor/expected.txt index ce11c4beef..6bd4e6fc96 100644 --- a/ffi/tests/test-expression-visitor/expected.txt +++ b/ffi/tests/test-expression-visitor/expected.txt @@ -2,7 +2,7 @@ This test demonstrates the full range of expression types. Some types are not yet supported in round-trip reconstruction: - Struct/Array/Map literals (nested data structures) - - Transform expressions (schema evolution operations) + - StructPatch expressions (struct transformation operations) - Opaque expressions (custom user-defined expressions) StructExpression @@ -25,7 +25,7 @@ StructExpression Date(32) Binary(0000deadbeefcafe) Decimal(1,1,20,3) - Null + Null(Short) Struct Field: top Struct @@ -35,24 +35,59 @@ StructExpression Array Short(5) Short(0) - Transform - Column(foo.bar.baz) - Prepend + StructPatch + input_path + Column(foo.bar.baz) + prepended_fields String(prepended) - Insert(a) - String(first) - Transform - Drop(gone) - Replace(stub) - String(replaced) - Insert(x) - Boolean(1) - Insert(y) - Boolean(0) - String(third) - Drop(dropme) - Replace(replaceme) - Integer(42) + field_patches + FieldPatch + field_name: a + insertions + String(first) + StructPatch + field_patches + FieldPatch + field_name: gone + keep_input: false + optional: false + FieldPatch + field_name: stub + insertions + String(replaced) + keep_input: false + optional: false + FieldPatch + field_name: x + insertions + Boolean(1) + keep_input: true + optional: false + FieldPatch + field_name: y + insertions + Boolean(0) + keep_input: true + optional: false + String(third) + keep_input: true + optional: false + FieldPatch + field_name: dropme + keep_input: false + optional: false + FieldPatch + field_name: replaceme + insertions + Integer(42) + keep_input: false + optional: false + appended_fields + String(appended) + StructPatch + StructPatch + input_path + Column(empty.nested) Array Short(5) Short(0) @@ -68,6 +103,12 @@ StructExpression Unknown(mystery) MapToStruct Column(pv) + Coalesce + Column(col) + Integer(0) + ArrayConstructor + Integer(1) + Integer(2) Divide Integer(0) Integer(0) @@ -138,7 +179,7 @@ And === Simple Round-trip Test === This test validates expressions/predicates with full support. Supported types: primitives (int, long, float, double, bool, string), - temporal (date, timestamp, timestamp_ntz), binary, decimal, null, + temporal (date, timestamp, timestamp_ntz), intervals, binary, decimal, null, binary operations (+, -, *, /), struct expressions, predicates (eq, ne, lt, le, gt, ge, distinct, is_null, is_not_null, not, and, or) @@ -153,6 +194,10 @@ StructExpression Date(19000) Timestamp(1234567890) TimestampNtz(9876543210) + IntervalYearMonth(-13) + IntervalDayTime(9876543210) + Null(Integer) + Null(Decimal(10,5)) Add Integer(10) Integer(20) diff --git a/ffi/tests/write-table-testing/expected-data/empty-commit.expected b/ffi/tests/write-table-testing/expected-data/empty-commit.expected new file mode 100644 index 0000000000..f266341c73 --- /dev/null +++ b/ffi/tests/write-table-testing/expected-data/empty-commit.expected @@ -0,0 +1,8 @@ +Writing empty commit to
    +Write context: + logical_schema: + physical_schema: + logical_to_physical: + write_path: file://
    / +Committed version: 1 +Post-commit snapshot version: 1 diff --git a/ffi/tests/write-table-testing/run_test.sh b/ffi/tests/write-table-testing/run_test.sh new file mode 100755 index 0000000000..9f55a51816 --- /dev/null +++ b/ffi/tests/write-table-testing/run_test.sh @@ -0,0 +1,35 @@ +#!/bin/bash + +set -euxo pipefail + +# Usage: run_test.sh +# +# Seeds a temp dir with a freshly-created Delta table using the create-table example, +# runs ./write_table against it, then diffs output against the expected file. +CREATE_BIN="$1" +EXPECTED="$2" +shift 2 + +if [ ! -x "$CREATE_BIN" ]; then + echo "ERROR: create_table binary not found at $CREATE_BIN" >&2 + echo "Build the create-table example first (it seeds the test table)." >&2 + exit 1 +fi + +TABLE_DIR=$(mktemp -d "${TMPDIR:-/tmp}/write_table_example.XXXXXX") +# macOS resolves /var -> /private/var when canonicalising paths inside the kernel, so the +# write_path the binary prints looks like file:///private/var/... while $TABLE_DIR is still +# /var/.... Resolve TABLE_DIR up front (and feed the resolved form to both binaries) so the +# sed substitution below has a literal match. `cd && pwd -P` is portable to both BSD and GNU. +TABLE_DIR=$(cd "$TABLE_DIR" && pwd -P) +OUT_FILE=$(mktemp) +trap 'rm -rf "$TABLE_DIR" "$OUT_FILE"' EXIT + +# Seed with an empty Delta table (v0). We discard create_table's output -- this example's +# expected output covers only what write_table prints. +"$CREATE_BIN" "$TABLE_DIR" > /dev/null + +./write_table "$TABLE_DIR" \ + | sed -E "s|$TABLE_DIR|
    |g" \ + | tee "$OUT_FILE" +diff "$OUT_FILE" "$EXPECTED" diff --git a/integration-tests/Cargo.toml b/integration-tests/Cargo.toml index cfd2913ac1..70da764a99 100644 --- a/integration-tests/Cargo.toml +++ b/integration-tests/Cargo.toml @@ -6,4 +6,4 @@ edition = "2021" [workspace] [dependencies] -delta_kernel = { path = "../kernel", features = ["default-engine-rustls"] } +delta_kernel = { path = "../kernel", features = ["arrow-conversion"] } diff --git a/kernel/Cargo.toml b/kernel/Cargo.toml index dd4fda5637..6af66a2de0 100644 --- a/kernel/Cargo.toml +++ b/kernel/Cargo.toml @@ -39,109 +39,131 @@ pre-release-hook = [ ] [dependencies] -delta_kernel_derive = { path = "../derive-macros", version = "0.20.0" } +delta_kernel_derive = { path = "../derive-macros", version = "0.27.1" } bytes = "1.10" chrono = "0.4.41" +chrono-tz = "0.10.4" crc = "3.2.2" indexmap = "2.10.0" itertools = "0.14" +percent-encoding = "2" +rand = "0.9" roaring = "0.11.2" serde = { version = "1", features = ["derive", "rc"] } serde_json = "1" -strum = { version = "0.27", features = ["derive"] } +strum = { version = "0.28", features = ["derive"] } thiserror = "2" -# only for structured logging +prost = { version = "0.13", optional = true } tracing = { version = "0.1", features = ["log"] } +tracing-subscriber = "0.3" url = "2" uuid = { version = "1.18.0", features = ["v4", "fast-rng"] } z85 = "3.0.6" # optional deps -futures = { version = "0.3", optional = true } -# Used for fetching direct urls (like pre-signed urls) -reqwest = { version = "0.13", optional = true } -# optionally used with default engine (though not required) -tokio = { version = "1.47", optional = true, features = ["rt-multi-thread"] } +# `reqwest` is only needed for the gated `Error::Reqwest` variant; the actual HTTP client +# lives in the `delta_kernel_default_engine` crate now. +reqwest = { version = "0.13", default-features = false, optional = true } # both arrow versions below are optional and require object_store -object_store_12 = { package = "object_store", version = "0.12.3", optional = true, features = ["aws", "azure", "gcp", "http"] } +object_store_13 = { package = "object_store", version = "0.13.2", optional = true, features = ["aws", "azure", "gcp", "http"] } -# arrow 56 -[dependencies.arrow_56] +# arrow 58 +[dependencies.arrow_58] package = "arrow" -version = "56" -features = ["chrono-tz", "ffi", "json", "prettyprint"] +version = "58" +features = ["chrono-tz", "ffi", "json"] optional = true -[dependencies.parquet_56] +[dependencies.parquet_58] package = "parquet" -version = "56" +version = "58.1" features = ["async", "object_store"] optional = true -# arrow 57 -[dependencies.arrow_57] +# arrow 59 +[dependencies.arrow_59] package = "arrow" -version = "57" -features = ["chrono-tz", "ffi", "json", "prettyprint"] +version = "59" +features = ["chrono-tz", "ffi", "json"] optional = true -[dependencies.parquet_57] +[dependencies.parquet_59] package = "parquet" -version = "57" +version = "59" features = ["async", "object_store"] optional = true [features] # no default features default = [] -# internal-api will make everything marked #[internal_api] public +# internal-api will make everything marked #[internal_api] public. internal-api = [] +# prettyprint enables Arrow pretty-print helpers (test/example oriented). +prettyprint = ["arrow_58?/prettyprint", "arrow_59?/prettyprint"] # test-utils exposes test-only constructors for downstream crate tests -test-utils = [] +test-utils = ["prettyprint"] # integration-test turns on a particularly heavy test for hdfs-object-store integration-test = ["hdfs-native-object-store/integration-test"] # The default versions for arrow/parquet/object_store -arrow = ["arrow-57"] # latest arrow version +arrow = ["arrow-59"] # latest arrow version need-arrow = [] # need-arrow is a marker that the feature needs arrow dep -arrow-56 = ["dep:arrow_56", "dep:parquet_56", "dep:object_store_12"] -arrow-57 = ["dep:arrow_57", "dep:parquet_57", "dep:object_store_12"] +arrow-58 = ["dep:arrow_58", "dep:parquet_58", "dep:object_store_13"] +arrow-59 = ["dep:arrow_59", "dep:parquet_59", "dep:object_store_13"] arrow-conversion = ["need-arrow"] arrow-expression = ["need-arrow"] -# WARNING: experimental feature, still under active development -# enables new experimental catalog-managed tables support -catalog-managed = [] - # Schema diffing functionality (experimental) schema-diff = [] -# this is an 'internal' feature flag which has all the shared bits from default-engine and -# default-engine-rustls +# Declarative plan IR (experimental). +declarative-plans = ["internal-api", "dep:prost", "dep:prost-build"] + +# Sources a prebuilt `protoc` for the codegen `declarative-plans` runs, so the build needs no +# system protobuf install. Leave it off to pass `protoc` through the `PROTOC` env var instead; +# a build that pins a hermetic toolchain cannot use a downloaded binary. +vendored-protoc = ["dep:protoc-bin-vendored"] + +# Experimental, temporary, test-only feature flag guarding the in-progress check-constraints +# work: the SQL predicate parser and, later, `checkConstraints` write enforcement. Remove once +# full check-constraints support ships. +check-constraints-in-dev = [] +# Experimental, temporary, test-only feature flag guarding the in-progress adaptiveMetadata +# (Iceberg V4 adaptive metadata tree) work. TODO(#2866): remove once full support ships. +adaptive-metadata-in-dev = [] +# Experimental, temporary, test-only feature flag guarding the in-progress geospatial +# (geometry/geography) type work. TODO(#2949): remove once full geo support ships. +geo-type-in-dev = [] + +# Enables shared Arrow-based engine support modules (arrow_data, arrow_expression, +# ensure_data_types, parquet_row_group_skipping). Used by the delta_kernel_default_engine crate +# and the SyncEngine. Keeps `reqwest` only for the gated `Error::Reqwest` variant. default-engine-base = [ "arrow-conversion", "arrow-expression", - "futures", "need-arrow", - "tokio", -] -# the default-engine-native-tls use the reqwest crate with default features which uses native-tls. if you want -# to instead use rustls, use 'default-engine-rustls' which has no native-tls dependency -default-engine-native-tls = ["default-engine-base", "reqwest/native-tls"] -default-engine-rustls = [ - "default-engine-base", - "reqwest/http2", + "reqwest", ] [build-dependencies] rustc_version = "0.4.1" +# Proto codegen, only pulled in / run under the `declarative-plans` feature. +prost-build = { version = "0.13", optional = true } +# Prebuilt `protoc` for that codegen, only pulled in under the `vendored-protoc` feature. +protoc-bin-vendored = { version = "3.2", optional = true } [dev-dependencies] -delta_kernel = { path = ".", features = ["arrow", "catalog-managed", "default-engine-rustls", "internal-api", "test-utils"] } -test_utils = { path = "../test-utils" } +delta_kernel = { path = ".", features = ["test-utils"] } +# NOTE: test_utils forcibly pulls the default engine and arrow-58. By default it ALSO pulls +# arrow-59, which takes precedence and would make it impossible to test against arrow-58. +test_utils = { path = "../test-utils", default-features = false } +# Used by the test-only SyncEngine to drive object_store calls via `futures::executor::block_on`. +futures = "0.3" +# Used by integration tests that bridge object_store's async API in test setup. +tokio = { version = "1.47", features = ["rt-multi-thread", "macros"] } criterion = "0.5" # Used for testing parse_url_opts extensibility -hdfs-native-object-store = { version = "0.15.0" } -hdfs-native = "0.12.2" +hdfs-native-object-store = { version = "0.16.0" } +hdfs-native = "0.13.4" walkdir = { version = "2.5.0" } async-trait = "0.1" # only used for our custom SlowGetStore ObjectStore implementation paste = "1.0" @@ -152,6 +174,7 @@ tracing-subscriber = { version = "0.3", default-features = false, features = [ "fmt", ] } rstest = "0.23" +rstest_reuse = "0.7" [[bench]] name = "metadata_bench" harness = false @@ -159,3 +182,16 @@ harness = false [[bench]] name = "expression_bench" harness = false + +# Single integration test binary. Linking each integration test as a separate +# binary is expensive (every binary re-links Arrow, Parquet, and the default +# engine), so all integration tests share one binary whose entry point is +# `tests/integration/main.rs` and whose topics live as submodules. +[[test]] +name = "integration" +path = "tests/integration/main.rs" + +# Declare as an example for access to test_utils +[[example]] +name = "metrics-tester" +path = "tests/integration/metrics_main.rs" diff --git a/kernel/benches/expression_bench.rs b/kernel/benches/expression_bench.rs index 6ef8ee7569..fea46cdcf6 100644 --- a/kernel/benches/expression_bench.rs +++ b/kernel/benches/expression_bench.rs @@ -16,14 +16,13 @@ use std::hint::black_box; use std::sync::Arc; +use criterion::{criterion_group, criterion_main, BenchmarkId, Criterion, Throughput}; use delta_kernel::arrow::array::{ ArrayRef, BooleanBuilder, Float64Builder, Int32Builder, StringBuilder, StructArray, }; use delta_kernel::arrow::datatypes::{DataType, Field, Fields}; use delta_kernel::engine::arrow_expression::evaluate_expression::to_json; -use criterion::{criterion_group, criterion_main, BenchmarkId, Criterion, Throughput}; - /// Creates a test struct array with realistic data for benchmarking. fn create_test_struct_array(num_rows: usize) -> StructArray { let mut id_builder = Int32Builder::with_capacity(num_rows); diff --git a/kernel/benches/metadata_bench.rs b/kernel/benches/metadata_bench.rs index db974af0f7..d84b3d0ca2 100644 --- a/kernel/benches/metadata_bench.rs +++ b/kernel/benches/metadata_bench.rs @@ -19,15 +19,13 @@ use std::sync::Arc; -use delta_kernel::engine::default::executor::tokio::TokioBackgroundExecutor; -use delta_kernel::engine::default::{DefaultEngine, DefaultEngineBuilder}; +use criterion::{criterion_group, criterion_main, Criterion}; use delta_kernel::snapshot::Snapshot; use delta_kernel::try_parse_uri; - -use test_utils::load_test_data; - -use criterion::{criterion_group, criterion_main, Criterion}; use tempfile::TempDir; +use test_utils::delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor; +use test_utils::delta_kernel_default_engine::{DefaultEngine, DefaultEngineBuilder}; +use test_utils::load_test_data; use url::Url; // force scan metadata bench to use smaller sample size so test runs faster (100 -> 20) @@ -40,7 +38,7 @@ fn setup() -> (TempDir, Url, Arc>) { let table_path = tempdir.path().join(table); let url = try_parse_uri(table_path.to_str().unwrap()).expect("Failed to parse table path"); // TODO: use multi-threaded executor - use delta_kernel::engine::default::storage::store_from_url; + use test_utils::delta_kernel_default_engine::storage::store_from_url; let store = store_from_url(&url).expect("Failed to create store"); let engine = DefaultEngineBuilder::new(store).build(); @@ -68,22 +66,32 @@ fn scan_metadata_benchmark(c: &mut Criterion) { let mut group = c.benchmark_group("scan_metadata"); group.sample_size(SCAN_METADATA_BENCH_SAMPLE_SIZE); - group.bench_function("scan_metadata", |b| { - b.iter(|| { - let scan = snapshot - .clone() // arc - .scan_builder() - .build() - .expect("Failed to build scan"); - let metadata_iter = scan - .scan_metadata(engine.as_ref()) - .expect("Failed to get scan metadata"); - // kernel scans are lazy, we must consume iterator to do the work we want to test - for result in metadata_iter { - result.expect("Failed to process scan metadata"); - } - }) - }); + // Benchmark both the default path (kernel builds a per-file transform expression for this + // partitioned table) and the `without_row_transforms` opt-out (which skips that build and the + // per-row partition-value parse that feeds it). + for without_row_transforms in [false, true] { + let id = if without_row_transforms { + "scan_metadata_without_row_transforms" + } else { + "scan_metadata" + }; + group.bench_function(id, |b| { + b.iter(|| { + let mut builder = snapshot.clone().scan_builder(); + if without_row_transforms { + builder = builder.without_row_transforms(); + } + let scan = builder.build().expect("Failed to build scan"); + let metadata_iter = scan + .scan_metadata(engine.as_ref()) + .expect("Failed to get scan metadata"); + // kernel scans are lazy, we must consume iterator to do the work we want to test + for result in metadata_iter { + result.expect("Failed to process scan metadata"); + } + }) + }); + } group.finish(); } diff --git a/kernel/build.rs b/kernel/build.rs index a8f413dfad..36cf6bad8f 100644 --- a/kernel/build.rs +++ b/kernel/build.rs @@ -6,4 +6,50 @@ fn main() { if let Channel::Nightly = version_meta().unwrap().channel { println!("cargo:rustc-cfg=NIGHTLY_CHANNEL"); } + + // Generate prost bindings for the declarative-plans proto schema only when the feature is + // enabled. Off-by-default consumers don't pay the protoc / codegen cost and don't pull in + // prost-build. + #[cfg(feature = "declarative-plans")] + compile_proto_definitions(); +} + +#[cfg(feature = "declarative-plans")] +fn compile_proto_definitions() { + let proto_dir = "proto"; + let proto_files = [ + "schema.proto", + "expressions.proto", + "plan.proto", + "operation.proto", + ]; + + for file in &proto_files { + println!("cargo:rerun-if-changed={proto_dir}/{file}"); + } + + let files: Vec = proto_files + .iter() + .map(|f| format!("{proto_dir}/{f}")) + .collect(); + + #[cfg(feature = "vendored-protoc")] + set_vendored_protoc(); + + // Don't propagate `.proto` comments into the generated code as doc comments: they contain + // angle-bracket generics (`Vec<...>`, `Option<...>`) that rustdoc would parse as unclosed + // HTML tags. The `.proto` files stay the canonical reference for the wire format. + prost_build::Config::new() + .disable_comments(["."]) + .compile_protos(&files, &[proto_dir]) + .expect("failed to compile .proto files"); +} + +#[cfg(all(feature = "declarative-plans", feature = "vendored-protoc"))] +fn set_vendored_protoc() { + // Leave a caller-supplied `protoc` alone, so a pinned toolchain wins over the vendored binary. + if std::env::var_os("PROTOC").is_none() { + let protoc = protoc_bin_vendored::protoc_bin_path().expect("vendored protoc binary"); + std::env::set_var("PROTOC", protoc); + } } diff --git a/kernel/examples/checkpoint-table/Cargo.toml b/kernel/examples/checkpoint-table/Cargo.toml index 9a9f75faca..60ca3b5d36 100644 --- a/kernel/examples/checkpoint-table/Cargo.toml +++ b/kernel/examples/checkpoint-table/Cargo.toml @@ -4,15 +4,21 @@ version = "0.1.0" edition = "2021" publish = false +[features] +default = ["arrow"] +arrow = ["arrow-59"] +arrow-59 = ["common/arrow-59"] +arrow-58 = ["common/arrow-58"] + [dependencies] clap = { version = "4.5", features = ["derive"] } chrono = { version = "0.4", features = ["serde"] } -common = { path = "../common" } +common = { path = "../common", default-features = false } delta_kernel = { path = "../../../kernel", features = [ - "arrow", - "default-engine-rustls", + "default-engine-base", "internal-api", ] } +delta_kernel_default_engine = { path = "../../../default-engine", features = ["rustls"] } env_logger = "0.11.8" tokio = { version = "1.0", features = ["full"] } url = "2" diff --git a/kernel/examples/checkpoint-table/src/main.rs b/kernel/examples/checkpoint-table/src/main.rs index a846f1f853..579a09b04e 100644 --- a/kernel/examples/checkpoint-table/src/main.rs +++ b/kernel/examples/checkpoint-table/src/main.rs @@ -4,15 +4,14 @@ use std::sync::Arc; use clap::Parser; use common::{LocationArgs, ParseWithExamples}; use delta_kernel::arrow::array::RecordBatch; +use delta_kernel::engine::arrow_data::EngineDataArrowExt; use delta_kernel::parquet::arrow::async_writer::AsyncFileWriter; use delta_kernel::parquet::arrow::AsyncArrowWriter; use delta_kernel::parquet::errors::Result as ParquetResult; -use futures::future::{BoxFuture, FutureExt}; - -use delta_kernel::engine::arrow_data::EngineDataArrowExt; -use delta_kernel::engine::default::executor::tokio::TokioMultiThreadExecutor; -use delta_kernel::engine::default::DefaultEngineBuilder; use delta_kernel::{ActionReconciliationIterator, DeltaResult, Error, Snapshot}; +use delta_kernel_default_engine::executor::tokio::TokioMultiThreadExecutor; +use delta_kernel_default_engine::DefaultEngineBuilder; +use futures::future::{BoxFuture, FutureExt}; /// An example program that checkpoints a table. /// !!!WARNING!!!: This doesn't use put-if-absent, or a catalog based commit, so it is UNSAFE. @@ -26,9 +25,9 @@ struct Cli { #[command(flatten)] location_args: LocationArgs, - /// This program doesn't use put-if-absent, or a catalog based commit, so it is UNSAFE. As such - /// you need to pass --unsafe-i-know-what-im-doing as an argument to get this to actually write - /// the checkpoint + /// This program doesn't use put-if-absent, or a catalog based commit, so it is UNSAFE. As + /// such you need to pass --unsafe-i-know-what-im-doing as an argument to get this to + /// actually write the checkpoint #[arg(long)] unsafe_i_know_what_im_doing: bool, } @@ -65,7 +64,7 @@ async fn try_main() -> DeltaResult<()> { let url = delta_kernel::try_parse_uri(&cli.location_args.path)?; println!("Checkpointing Delta table at: {url}"); - use delta_kernel::engine::default::storage::store_from_url; + use delta_kernel_default_engine::storage::store_from_url; let store = store_from_url(&url)?; let executor = Arc::new(TokioMultiThreadExecutor::new( tokio::runtime::Handle::current(), @@ -76,11 +75,11 @@ async fn try_main() -> DeltaResult<()> { let snapshot = Snapshot::builder_for(url).build(&engine)?; if cli.unsafe_i_know_what_im_doing { - snapshot.checkpoint(&engine)?; + snapshot.checkpoint(&engine, None)?; println!("Table checkpointed"); } else { // first we create a checkpoint writer - let writer = snapshot.create_checkpoint_writer()?; + let writer = snapshot.create_checkpoint_writer(&engine)?; // this tells us the path where we should write the checkpoint file let checkpoint_path = writer.checkpoint_path()?; @@ -93,12 +92,12 @@ async fn try_main() -> DeltaResult<()> { let Some(first) = first else { return Err(Error::generic("No batches in checkpoint data")); }; - // Note that with `FilteredEngineData` it's important to `apply_selection_vector` to remove any - // filtered out rows. It's also possible to use `into_parts` to get the unfiltered batch and the - // selection vector individually, such that an engine could write only the selected rows out - // without having to allocate a new engine data. - // NB: Unselected rows MUST NOT be written to the checkpoint! Doing so will create an invalid - // checkpoint + // Note that with `FilteredEngineData` it's important to `apply_selection_vector` to remove + // any filtered out rows. It's also possible to use `into_parts` to get the + // unfiltered batch and the selection vector individually, such that an engine could + // write only the selected rows out without having to allocate a new engine data. + // NB: Unselected rows MUST NOT be written to the checkpoint! Doing so will create an + // invalid checkpoint let first_data = first?.apply_selection_vector()?; let first_batch = first_data.try_into_record_batch()?; diff --git a/kernel/examples/common/Cargo.toml b/kernel/examples/common/Cargo.toml index 5c8f75f94b..30f260d143 100644 --- a/kernel/examples/common/Cargo.toml +++ b/kernel/examples/common/Cargo.toml @@ -10,11 +10,17 @@ version.workspace = true [package.metadata.release] release = false +[features] +default = ["arrow"] +arrow = ["arrow-59"] +arrow-59 = ["delta_kernel/arrow-59"] +arrow-58 = ["delta_kernel/arrow-58"] + [dependencies] clap = { version = "4.5", features = ["derive"] } delta_kernel = { path = "../../../kernel", features = [ - "arrow", - "default-engine-rustls", + "default-engine-base", "internal-api", ] } +delta_kernel_default_engine = { path = "../../../default-engine", features = ["rustls"] } url = "2" diff --git a/kernel/examples/common/src/lib.rs b/kernel/examples/common/src/lib.rs index c847489cf7..9a4bb5b5fd 100644 --- a/kernel/examples/common/src/lib.rs +++ b/kernel/examples/common/src/lib.rs @@ -1,19 +1,20 @@ //! Common code to be shared between all examples. Mostly argument parsing, and a few other //! utilities -use std::{collections::HashMap, sync::Arc}; +use std::collections::HashMap; +use std::sync::Arc; use clap::{Args, CommandFactory, FromArgMatches}; -use delta_kernel::{ - arrow::array::RecordBatch, engine::default::executor::tokio::TokioBackgroundExecutor, - engine::default::storage::store_from_url_opts, engine::default::DefaultEngine, - engine::default::DefaultEngineBuilder, scan::Scan, schema::MetadataColumnSpec, DeltaResult, - SnapshotRef, -}; - -use delta_kernel::object_store::{ - aws::AmazonS3Builder, azure::MicrosoftAzureBuilder, gcp::GoogleCloudStorageBuilder, - DynObjectStore, ObjectStoreScheme, -}; +use delta_kernel::arrow::array::RecordBatch; +use delta_kernel::object_store::aws::AmazonS3Builder; +use delta_kernel::object_store::azure::MicrosoftAzureBuilder; +use delta_kernel::object_store::gcp::GoogleCloudStorageBuilder; +use delta_kernel::object_store::{DynObjectStore, ObjectStoreScheme}; +use delta_kernel::scan::Scan; +use delta_kernel::schema::MetadataColumnSpec; +use delta_kernel::{DeltaResult, SnapshotRef}; +use delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor; +use delta_kernel_default_engine::storage::store_from_url_opts; +use delta_kernel_default_engine::{DefaultEngine, DefaultEngineBuilder}; use url::Url; #[derive(Args)] @@ -44,8 +45,8 @@ pub struct LocationArgs { pub env_creds: bool, /// Specify that the table is "public" (i.e. no cloud credentials are needed). This is required - /// for things like s3 public buckets, otherwise the kernel will try and authenticate by talking - /// to the aws metadata server, which will fail unless you're on an ec2 instance. + /// for things like s3 public buckets, otherwise the kernel will try and authenticate by + /// talking to the aws metadata server, which will fail unless you're on an ec2 instance. #[arg(long)] pub public: bool, } diff --git a/kernel/examples/inspect-table/Cargo.toml b/kernel/examples/inspect-table/Cargo.toml index 97d3d0f150..bb923cc2fc 100644 --- a/kernel/examples/inspect-table/Cargo.toml +++ b/kernel/examples/inspect-table/Cargo.toml @@ -4,16 +4,24 @@ version = "0.1.0" edition = "2021" publish = false +[features] +default = ["arrow"] +arrow = ["arrow-59"] +arrow-59 = ["common/arrow-59"] +arrow-58 = ["common/arrow-58"] + [dependencies] chrono = "0.4.42" clap = { version = "4.5", features = ["derive"] } -common = { path = "../common" } +common = { path = "../common", default-features = false } delta_kernel = { path = "../../../kernel", features = [ - "arrow", - "default-engine-rustls", + "default-engine-base", "internal-api", ] } +delta_kernel_default_engine = { path = "../../../default-engine", features = ["rustls"] } env_logger = "0.11.8" +tracing = "0.1" +tracing-subscriber = { version="0.3.22", features = [ "env-filter" ] } # for cargo-release [package.metadata.release] diff --git a/kernel/examples/inspect-table/src/main.rs b/kernel/examples/inspect-table/src/main.rs index ba1552ae19..407235733d 100644 --- a/kernel/examples/inspect-table/src/main.rs +++ b/kernel/examples/inspect-table/src/main.rs @@ -1,3 +1,8 @@ +use std::collections::HashMap; +use std::process::ExitCode; +use std::sync::{Arc, LazyLock}; + +use clap::{Parser, Subcommand}; use common::{LocationArgs, ParseWithExamples}; use delta_kernel::actions::visitors::{ visit_metadata_at, visit_protocol_at, AddVisitor, CdcVisitor, RemoveVisitor, @@ -9,16 +14,13 @@ use delta_kernel::actions::{ }; use delta_kernel::engine_data::{GetData, RowVisitor, TypedGetData as _}; use delta_kernel::expressions::ColumnName; +use delta_kernel::metrics::{LoggingMetricsReporter, WithMetricsReporterLayer}; use delta_kernel::scan::state::ScanFile; use delta_kernel::scan::ScanBuilder; use delta_kernel::schema::{ColumnNamesAndTypes, DataType}; use delta_kernel::{DeltaResult, Error, Snapshot}; - -use std::collections::HashMap; -use std::process::ExitCode; -use std::sync::LazyLock; - -use clap::{Parser, Subcommand}; +use tracing_subscriber::layer::SubscriberExt as _; +use tracing_subscriber::util::SubscriberInitExt as _; #[derive(Parser)] #[command(author, version, about, long_about = None)] @@ -50,7 +52,14 @@ enum Commands { } fn main() -> ExitCode { - env_logger::init(); + tracing_subscriber::registry() + .with( + tracing_subscriber::fmt::layer() + .with_span_events(tracing_subscriber::fmt::format::FmtSpan::NONE), + ) + .with_metrics_reporter_layer(Arc::new(LoggingMetricsReporter::new(tracing::Level::INFO))) + .with(tracing_subscriber::EnvFilter::from_default_env()) + .init(); match try_main() { Ok(()) => ExitCode::SUCCESS, Err(e) => { diff --git a/kernel/examples/read-table-changes/Cargo.toml b/kernel/examples/read-table-changes/Cargo.toml index 17c57ef574..ce981c982c 100644 --- a/kernel/examples/read-table-changes/Cargo.toml +++ b/kernel/examples/read-table-changes/Cargo.toml @@ -8,12 +8,19 @@ publish = false [package.metadata.release] release = false +[features] +default = ["arrow"] +arrow = ["arrow-59"] +arrow-59 = ["common/arrow-59"] +arrow-58 = ["common/arrow-58"] + [dependencies] clap = { version = "4.5", features = ["derive"] } -common = { path = "../common" } +common = { path = "../common", default-features = false } delta_kernel = { path = "../../../kernel", features = [ - "arrow", - "default-engine-rustls", + "default-engine-base", "internal-api", + "prettyprint", ] } +delta_kernel_default_engine = { path = "../../../default-engine", features = ["rustls"] } itertools = "0.14" diff --git a/kernel/examples/read-table-multi-threaded/Cargo.toml b/kernel/examples/read-table-multi-threaded/Cargo.toml index 3bef1eaeda..01b9ab7f13 100644 --- a/kernel/examples/read-table-multi-threaded/Cargo.toml +++ b/kernel/examples/read-table-multi-threaded/Cargo.toml @@ -4,14 +4,21 @@ version = "0.1.0" edition = "2021" publish = false +[features] +default = ["arrow"] +arrow = ["arrow-59"] +arrow-59 = ["common/arrow-59"] +arrow-58 = ["common/arrow-58"] + [dependencies] clap = { version = "4.5", features = ["derive"] } -common = { path = "../common" } +common = { path = "../common", default-features = false } delta_kernel = { path = "../../../kernel", features = [ - "arrow", - "default-engine-rustls", + "default-engine-base", "internal-api", + "prettyprint", ] } +delta_kernel_default_engine = { path = "../../../default-engine", features = ["rustls"] } env_logger = "0.11.8" spmc = "0.3.0" url = "2" diff --git a/kernel/examples/read-table-multi-threaded/src/main.rs b/kernel/examples/read-table-multi-threaded/src/main.rs index 1ff0c42b85..b261c5c13d 100644 --- a/kernel/examples/read-table-multi-threaded/src/main.rs +++ b/kernel/examples/read-table-multi-threaded/src/main.rs @@ -3,6 +3,7 @@ use std::sync::mpsc::Sender; use std::sync::{mpsc, Arc}; use std::thread; +use clap::Parser; use common::{LocationArgs, ParseWithExamples, ScanArgs}; use delta_kernel::actions::deletion_vector::split_vector; use delta_kernel::arrow::compute::filter_record_batch; @@ -12,8 +13,6 @@ use delta_kernel::engine::arrow_data::EngineDataArrowExt as _; use delta_kernel::scan::state::{transform_to_logical, DvInfo, ScanFile}; use delta_kernel::schema::SchemaRef; use delta_kernel::{DeltaResult, Engine, ExpressionRef, FileMeta, Snapshot}; - -use clap::Parser; use url::Url; /// An example program that reads a table using multiple threads. This shows the use of the @@ -123,8 +122,8 @@ fn try_main() -> DeltaResult<()> { }); }); - // have handed out all copies needed, drop so record_batch_rx will exit when the last thread is - // done sending + // have handed out all copies needed, drop so record_batch_rx will exit when the last thread + // is done sending drop(record_batch_tx); for res in scan_metadata { diff --git a/kernel/examples/read-table-single-threaded/Cargo.toml b/kernel/examples/read-table-single-threaded/Cargo.toml index 9c5a481b18..de53273d2e 100644 --- a/kernel/examples/read-table-single-threaded/Cargo.toml +++ b/kernel/examples/read-table-single-threaded/Cargo.toml @@ -4,14 +4,21 @@ version = "0.1.0" edition = "2021" publish = false +[features] +default = ["arrow"] +arrow = ["arrow-59"] +arrow-59 = ["common/arrow-59"] +arrow-58 = ["common/arrow-58"] + [dependencies] clap = { version = "4.5", features = ["derive"] } -common = { path = "../common" } +common = { path = "../common", default-features = false } delta_kernel = { path = "../../../kernel", features = [ - "arrow", - "default-engine-rustls", + "default-engine-base", "internal-api", + "prettyprint", ] } +delta_kernel_default_engine = { path = "../../../default-engine", features = ["rustls"] } env_logger = "0.11.8" itertools = "0.14" diff --git a/kernel/examples/read-table-single-threaded/src/main.rs b/kernel/examples/read-table-single-threaded/src/main.rs index 207c792361..22bd8f9055 100644 --- a/kernel/examples/read-table-single-threaded/src/main.rs +++ b/kernel/examples/read-table-single-threaded/src/main.rs @@ -1,13 +1,12 @@ use std::process::ExitCode; use std::sync::Arc; +use clap::Parser; use common::{LocationArgs, ParseWithExamples, ScanArgs}; use delta_kernel::arrow::record_batch::RecordBatch; use delta_kernel::arrow::util::pretty::print_batches; use delta_kernel::engine::arrow_data::EngineDataArrowExt; use delta_kernel::{DeltaResult, Snapshot}; - -use clap::Parser; use itertools::Itertools; /// An example program that dumps out the data of a delta table. @@ -57,7 +56,8 @@ fn try_main() -> DeltaResult<()> { }; let batch_rows = batch.num_rows(); let result = match cli.scan_args.limit { - Some(limit) if **rows_so_far >= limit => return None, // over the limit, stop iteration + // over the limit, stop iteration + Some(limit) if **rows_so_far >= limit => return None, Some(limit) => { let batch = if **rows_so_far + batch_rows > limit { common::truncate_batch(batch, limit - **rows_so_far) diff --git a/kernel/examples/write-table/Cargo.toml b/kernel/examples/write-table/Cargo.toml index 61f94386e5..22d60ca3c8 100644 --- a/kernel/examples/write-table/Cargo.toml +++ b/kernel/examples/write-table/Cargo.toml @@ -4,14 +4,21 @@ version = "0.1.0" edition = "2021" publish = false +[features] +default = ["arrow"] +arrow = ["arrow-59"] +arrow-59 = ["common/arrow-59"] +arrow-58 = ["common/arrow-58"] + [dependencies] clap = { version = "4.5", features = ["derive"] } -common = { path = "../common" } +common = { path = "../common", default-features = false } delta_kernel = { path = "../../../kernel", features = [ - "arrow", - "default-engine-rustls", + "default-engine-base", "internal-api", + "prettyprint", ] } +delta_kernel_default_engine = { path = "../../../default-engine", features = ["rustls"] } env_logger = "0.11.8" itertools = "0.14" url = "2" diff --git a/kernel/examples/write-table/src/main.rs b/kernel/examples/write-table/src/main.rs index e29786b207..5221f3eac2 100644 --- a/kernel/examples/write-table/src/main.rs +++ b/kernel/examples/write-table/src/main.rs @@ -1,4 +1,3 @@ -use std::collections::HashMap; use std::fs::create_dir_all; use std::path::Path; use std::process::ExitCode; @@ -8,21 +7,20 @@ use clap::Parser; use common::{LocationArgs, ParseWithExamples}; use delta_kernel::arrow::array::{ ArrayRef, BooleanArray, Float64Array, Int32Array, Int64Array, RecordBatch, StringArray, + TimestampMicrosecondArray, }; use delta_kernel::arrow::util::pretty::print_batches; -use itertools::Itertools; -use url::Url; - -use delta_kernel::arrow::array::TimestampMicrosecondArray; use delta_kernel::committer::FileSystemCommitter; use delta_kernel::engine::arrow_conversion::TryIntoArrow; use delta_kernel::engine::arrow_data::{ArrowEngineData, EngineDataArrowExt}; -use delta_kernel::engine::default::executor::tokio::TokioBackgroundExecutor; -use delta_kernel::engine::default::{DefaultEngine, DefaultEngineBuilder}; use delta_kernel::schema::{DataType, SchemaRef, StructField, StructType}; use delta_kernel::transaction::create_table::create_table as create_delta_table; use delta_kernel::transaction::{CommitResult, RetryableTransaction}; use delta_kernel::{DeltaResult, Engine, Error, Snapshot, SnapshotRef}; +use delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor; +use delta_kernel_default_engine::{DefaultEngine, DefaultEngineBuilder}; +use itertools::Itertools; +use url::Url; /// An example program that writes to a Delta table and creates it if necessary. #[derive(Parser)] @@ -44,7 +42,8 @@ struct Cli { #[arg(long, short, default_value = "10")] num_rows: usize, // TODO: Support providing input data from a JSON file instead of generating random data - // TODO: Support specifying whether the transaction should overwrite, append, or error if the table already exists + // TODO: Support specifying whether the transaction should overwrite, append, or error if the + // table already exists } #[tokio::main] @@ -68,7 +67,7 @@ async fn try_main() -> DeltaResult<()> { create_dir_all(&cli.location_args.path).map_err(|e| { Error::generic(format!( "Failed to create directory {}: {e}", - &cli.location_args.path + cli.location_args.path )) })?; } @@ -77,7 +76,7 @@ async fn try_main() -> DeltaResult<()> { println!("Using Delta table at: {url}"); // Get the engine for local filesystem - use delta_kernel::engine::default::storage::store_from_url; + use delta_kernel_default_engine::storage::store_from_url; let engine = DefaultEngineBuilder::new(store_from_url(&url)?).build(); // Create or get the table @@ -95,9 +94,9 @@ async fn try_main() -> DeltaResult<()> { .with_data_change(true); // Write the data using the engine - let write_context = Arc::new(txn.get_write_context()); + let write_context = Arc::new(txn.unpartitioned_write_context()?); let file_metadata = engine - .write_parquet(&sample_data, write_context.as_ref(), HashMap::new()) + .write_parquet(&sample_data, write_context.as_ref()) .await?; // Add the file metadata to the transaction diff --git a/kernel/proto/expressions.proto b/kernel/proto/expressions.proto new file mode 100644 index 0000000000..4beaae5415 --- /dev/null +++ b/kernel/proto/expressions.proto @@ -0,0 +1,270 @@ +// Proto mirror of `kernel/src/expressions/mod.rs` + `scalars.rs` (the Rust IR is the source +// of truth for these shapes). +// +// Opaque{Expression,Predicate} and Unknown are escape hatches: kernel-rs can't serialise the +// opaque trait objects, and Unknown is the "do not silently interpret" marker. They appear +// here so the grammar is total, but engines MUST error on them, not produce NULL. + +syntax = "proto3"; + +package delta.kernel.expressions; + +// Java package for generated bindings. Ignored by prost (Rust codegen). +option java_package = "io.delta.kernel.proto.expressions"; + +import "schema.proto"; + +// ============================================================================ +// Operator enums +// ============================================================================ + +enum UnaryPredicateOp { + UNARY_PREDICATE_OP_UNSPECIFIED = 0; + UNARY_PREDICATE_OP_IS_NULL = 1; +} + +enum BinaryPredicateOp { + BINARY_PREDICATE_OP_UNSPECIFIED = 0; + BINARY_PREDICATE_OP_LESS_THAN = 1; + BINARY_PREDICATE_OP_GREATER_THAN = 2; + BINARY_PREDICATE_OP_EQUAL = 3; + BINARY_PREDICATE_OP_DISTINCT = 4; + BINARY_PREDICATE_OP_IN = 5; +} + +enum UnaryExpressionOp { + UNARY_EXPRESSION_OP_UNSPECIFIED = 0; + UNARY_EXPRESSION_OP_TO_JSON = 1; +} + +enum BinaryExpressionOp { + BINARY_EXPRESSION_OP_UNSPECIFIED = 0; + BINARY_EXPRESSION_OP_PLUS = 1; + BINARY_EXPRESSION_OP_MINUS = 2; + BINARY_EXPRESSION_OP_MULTIPLY = 3; + BINARY_EXPRESSION_OP_DIVIDE = 4; +} + +enum VariadicExpressionOp { + VARIADIC_EXPRESSION_OP_UNSPECIFIED = 0; + VARIADIC_EXPRESSION_OP_COALESCE = 1; + VARIADIC_EXPRESSION_OP_ARRAY = 2; +} + +enum JunctionPredicateOp { + JUNCTION_PREDICATE_OP_UNSPECIFIED = 0; + JUNCTION_PREDICATE_OP_AND = 1; + JUNCTION_PREDICATE_OP_OR = 2; +} + +// ============================================================================ +// Scalar payloads +// ============================================================================ + +// `bits` is the unscaled integer, big-endian two's-complement; pair with `decimal_type` to +// materialise a language-native decimal. +message DecimalData { + bytes bits = 1; + delta.kernel.schema.DecimalType decimal_type = 2; +} + +message ArrayData { + delta.kernel.schema.ArrayType array_type = 1; + repeated Scalar elements = 2; +} + +// Modeled as repeated key/value pairs because proto map keys must be scalar, but Delta map +// keys can be any `Scalar` (including structs). +message MapEntry { + Scalar key = 1; + Scalar value = 2; +} +message MapData { + delta.kernel.schema.MapType map_type = 1; + repeated MapEntry pairs = 2; +} + +// `fields` and `values` are pairwise indexed: `values[i]` is the value for `fields[i]`. +message StructData { + repeated delta.kernel.schema.StructField fields = 1; + repeated Scalar values = 2; +} + +// `Null` carries a `DataType` so the evaluator can produce a NULL of the right type. +message Scalar { + oneof value { + int32 integer = 1; + int64 long = 2; + int32 short = 3; // i16 narrowed at decode (proto has no i16) + int32 byte = 4; // i8 narrowed at decode (proto has no i8) + float float = 5; + double double = 6; + string string = 7; + bool boolean = 8; + int64 timestamp = 9; // micros since epoch, UTC + int64 timestamp_ntz = 10; // micros since epoch, no tz + int32 date = 11; // days since epoch + bytes binary = 12; + DecimalData decimal = 13; + delta.kernel.schema.DataType null = 14; + StructData struct = 15; + ArrayData array = 16; + MapData map = 17; + int32 interval_year_month = 18; // year-month interval: signed month count + int64 interval_day_time = 19; // day-time interval: signed microsecond count + } +} + +message ColumnName { + repeated string path = 1; +} + +// ============================================================================ +// Composite expression bodies +// ============================================================================ + +message UnaryExpression { + UnaryExpressionOp op = 1; + Expression expr = 2; +} + +message BinaryExpression { + BinaryExpressionOp op = 1; + Expression left = 2; + Expression right = 3; +} + +message VariadicExpression { + VariadicExpressionOp op = 1; + repeated Expression exprs = 2; +} + +message IfExpression { + Predicate condition = 1; + Expression then_expr = 2; + Expression else_expr = 3; +} + +message ParseJsonExpression { + Expression json_expr = 1; + delta.kernel.schema.StructType output_schema = 2; +} + +message MapToStructOptions { + // Optional IANA timezone or fixed offset for interpreting offset-less TIMESTAMP values. An + // explicit offset in a value takes precedence; absence preserves the default UTC behavior. + optional string timestamp_timezone = 1; +} + +message MapToStructExpression { + Expression map_expr = 1; + MapToStructOptions options = 2; +} + +message ElementAtExpression { + Expression map_expr = 1; + Expression key_expr = 2; +} + +message CastOptions { + // Optional IANA timezone or fixed offset for interpreting offset-less strings cast to + // TIMESTAMP. An explicit offset in a value takes precedence; absence preserves the default UTC + // behavior. + optional string timestamp_timezone = 1; +} + +message CastExpression { + Expression expr = 1; + delta.kernel.schema.DataType target = 2; + CastOptions options = 3; +} + +// `nullability_predicate` is optional: when set and it evaluates to false/null, the whole +// struct is null. +message StructExpression { + repeated Expression exprs = 1; + Expression nullability_predicate = 2; +} + +message FieldTransform { + repeated Expression exprs = 1; + bool is_replace = 2; + bool optional = 3; +} + +message Transform { + // `input_path` is optional: absent means a top-level transform (no input path). + ColumnName input_path = 1; + map field_transforms = 2; + repeated Expression prepended_fields = 3; + repeated Expression appended_fields = 4; +} + +// The opaque op carries only its `name()` because the Rust trait object can't be serialised; +// engines resolve `name` against a local op registry or hard-error (never NULL). +message OpaqueExpression { + string name = 1; + repeated Expression exprs = 2; +} + +message OpaquePredicate { + string name = 1; + repeated Expression exprs = 2; +} + +// ============================================================================ +// Predicate +// ============================================================================ + +message UnaryPredicate { + UnaryPredicateOp op = 1; + Expression expr = 2; +} + +message BinaryPredicate { + BinaryPredicateOp op = 1; + Expression left = 2; + Expression right = 3; +} + +message JunctionPredicate { + JunctionPredicateOp op = 1; + repeated Predicate preds = 2; +} + +message Predicate { + oneof kind { + Expression boolean_expression = 1; + Predicate not = 2; + UnaryPredicate unary = 3; + BinaryPredicate binary = 4; + JunctionPredicate junction = 5; + OpaquePredicate opaque = 6; + string unknown = 7; + } +} + +// ============================================================================ +// Expression +// ============================================================================ + +message Expression { + oneof kind { + Scalar literal = 1; + ColumnName column = 2; + Predicate predicate = 3; + StructExpression struct_expr = 4; + Transform transform = 5; + UnaryExpression unary = 6; + BinaryExpression binary = 7; + VariadicExpression variadic = 8; + // TODO: IfExpression is not yet available in Kernel but will be needed in the future. + IfExpression if_expr = 9; + OpaqueExpression opaque = 10; + ParseJsonExpression parse_json = 11; + MapToStructExpression map_to_struct = 12; + string unknown = 13; + ElementAtExpression element_at = 14; + CastExpression cast = 15; + } +} diff --git a/kernel/proto/operation.proto b/kernel/proto/operation.proto new file mode 100644 index 0000000000..2b9f5a52de --- /dev/null +++ b/kernel/proto/operation.proto @@ -0,0 +1,71 @@ +// Proto mirror of `kernel/src/plans/ir/operation.rs` (the Rust IR is the source of truth). + +syntax = "proto3"; + +package delta.kernel.operation; + +// Java package for generated bindings. Ignored by prost (Rust codegen). +option java_package = "io.delta.kernel.proto.operation"; + +import "plan.proto"; + +// ============================================================================ +// Operation / IoOperation (top-level types) +// ============================================================================ + +message Operation { + oneof op { + IoOperation io = 1; + delta.kernel.plan.Plan query_plan = 2; + } +} + +message IoOperation { + oneof op { + FileListing file_listing = 1; + ReadBytes read_bytes = 2; + WriteBytes write_bytes = 3; + HeadFile head_file = 4; + AtomicCopy atomic_copy = 5; + ParquetFooter parquet_footer = 6; + } +} + +// ============================================================================ +// IoOperation payloads +// ============================================================================ + +// A file URL plus an optional half-open byte range `[range_start, range_end)`. +// `range_start` and `range_end` are set together when a range is present (or both absent otherwise). +message FileSlice { + string url = 1; + optional uint64 range_start = 2; + optional uint64 range_end = 3; +} + +message FileListing { + string url = 1; +} + +message ReadBytes { + repeated FileSlice files = 1; +} + +message WriteBytes { + string url = 1; + bytes data = 2; + bool overwrite = 3; +} + +message HeadFile { + string url = 1; +} + +message AtomicCopy { + string source = 1; + string destination = 2; +} + +message ParquetFooter { + delta.kernel.plan.FileMeta file = 1; +} diff --git a/kernel/proto/plan.proto b/kernel/proto/plan.proto new file mode 100644 index 0000000000..2017520cdf --- /dev/null +++ b/kernel/proto/plan.proto @@ -0,0 +1,170 @@ +// Proto mirror of `kernel/src/plans/ir/{plan,nodes}.rs` (the Rust IR is the source of truth). +// +// A plan is a DAG of plan nodes: `Plan` is a flat list of `PlanNode`s, and each node wires +// into the DAG by referencing other nodes' positions (indices into `Plan.nodes`) in its +// `inputs`. A node is identified by its index; the terminal node is the last entry. Field +// numbers are never reused once published, so the wire stays compatible. + +syntax = "proto3"; + +package delta.kernel.plan; + +// Java package for generated bindings. Ignored by prost (Rust codegen). +option java_package = "io.delta.kernel.proto.plan"; + +import "expressions.proto"; +import "schema.proto"; + +message FileMeta { + string location = 1; + uint64 size = 2; + int64 last_modified = 3; // milliseconds since epoch +} + +// ============================================================================ +// Source payloads (0 inputs) +// ============================================================================ + +message ScanFile { + FileMeta meta = 1; + repeated delta.kernel.expressions.Scalar file_constants = 2; +} + +message ScanParquetNode { + repeated ScanFile files = 1; + repeated string file_constant_columns = 2; + delta.kernel.schema.StructType schema = 3; +} + +message ScanJsonNode { + repeated ScanFile files = 1; + repeated string file_constant_columns = 2; + delta.kernel.schema.StructType schema = 3; +} + +message ValuesRow { + repeated delta.kernel.expressions.Scalar values = 1; +} +message ValuesNode { + delta.kernel.schema.StructType schema = 1; + repeated ValuesRow rows = 2; +} + +// ============================================================================ +// Transform payloads (1+ inputs) +// ============================================================================ + +message ProjectNode { + delta.kernel.expressions.Expression expr = 1; + delta.kernel.schema.StructType schema = 2; +} + +message FilterNode { + delta.kernel.expressions.Predicate predicate = 1; +} + +message UnionAllNode {} + +enum FileType { + FILE_TYPE_UNSPECIFIED = 0; + FILE_TYPE_PARQUET = 1; + FILE_TYPE_JSON = 2; +} + +message DynamicScanNode { + delta.kernel.schema.StructType schema = 1; + FileType file_type = 2; + // Hierarchical base URL ending in `/` that relative file paths resolve against. + string base_url = 3; + repeated string file_constant_columns = 4; + // Required input columns: path is STRING; size and last-modified are LONG. + // Each column and every ancestor in its path must be non-nullable. + delta.kernel.expressions.ColumnName path_column = 5; + delta.kernel.expressions.ColumnName file_size_column = 6; + delta.kernel.expressions.ColumnName last_modified_column = 7; + delta.kernel.expressions.ColumnName dv_column = 8; +} + +message MinAgg { + delta.kernel.expressions.ColumnName value = 1; +} +message MaxAgg { + delta.kernel.expressions.ColumnName value = 1; +} +message SumAgg { + delta.kernel.expressions.ColumnName value = 1; +} +message CountAgg { + delta.kernel.expressions.ColumnName value = 1; +} +message CountStarAgg {} +message MinNonNullByAgg { + delta.kernel.expressions.ColumnName value = 1; + delta.kernel.expressions.ColumnName null_sentinel = 2; + delta.kernel.expressions.ColumnName key = 3; +} +message MaxNonNullByAgg { + delta.kernel.expressions.ColumnName value = 1; + delta.kernel.expressions.ColumnName null_sentinel = 2; + delta.kernel.expressions.ColumnName key = 3; +} + +message Agg { + oneof func { + MinAgg min = 1; + MaxAgg max = 2; + MinNonNullByAgg min_non_null_by = 3; + MaxNonNullByAgg max_non_null_by = 4; + SumAgg sum = 5; + CountAgg count = 6; + CountStarAgg count_star = 7; + } +} + +message AggregateNode { + repeated delta.kernel.expressions.ColumnName group_by = 1; + repeated Agg aggs = 2; + delta.kernel.schema.StructType schema = 3; +} + +message SemiJoinNode { + bool inverted = 1; + repeated delta.kernel.expressions.ColumnName probe_keys = 2; + repeated delta.kernel.expressions.ColumnName build_keys = 3; +} + +// ============================================================================ +// Operator +// ============================================================================ + +message Operator { + oneof op { + // Sources (0 inputs) + ScanParquetNode scan_parquet = 1; + ScanJsonNode scan_json = 2; + ValuesNode values = 3; + // Transforms (1+ inputs) + ProjectNode project = 4; + FilterNode filter = 5; + UnionAllNode union_all = 6; + DynamicScanNode dynamic_scan = 7; + AggregateNode aggregate = 8; + SemiJoinNode semi_join = 9; + } +} + +// ============================================================================ +// PlanNode / Plan +// ============================================================================ + +message PlanNode { + Operator op = 1; + // Indices into `Plan.nodes` of this node's upstream inputs. Each index is strictly less than + // this node's own index, so `nodes` can be evaluated in order. + repeated uint32 inputs = 2; +} + +// The plan's terminal node is its last entry; the engine streams that node's rows. +message Plan { + repeated PlanNode nodes = 1; +} diff --git a/kernel/proto/schema.proto b/kernel/proto/schema.proto new file mode 100644 index 0000000000..76011e7684 --- /dev/null +++ b/kernel/proto/schema.proto @@ -0,0 +1,112 @@ +// Proto mirror of `kernel/src/schema/mod.rs` + +syntax = "proto3"; + +package delta.kernel.schema; + +// Java package for generated bindings. Ignored by prost (Rust codegen). +option java_package = "io.delta.kernel.proto.schema"; + +// ============================================================================ +// Primitive types +// ============================================================================ + +enum SimplePrimitiveType { + SIMPLE_PRIMITIVE_TYPE_UNSPECIFIED = 0; + SIMPLE_PRIMITIVE_TYPE_STRING = 1; + SIMPLE_PRIMITIVE_TYPE_LONG = 2; + SIMPLE_PRIMITIVE_TYPE_INTEGER = 3; + SIMPLE_PRIMITIVE_TYPE_SHORT = 4; + SIMPLE_PRIMITIVE_TYPE_BYTE = 5; + SIMPLE_PRIMITIVE_TYPE_FLOAT = 6; + SIMPLE_PRIMITIVE_TYPE_DOUBLE = 7; + SIMPLE_PRIMITIVE_TYPE_BOOLEAN = 8; + SIMPLE_PRIMITIVE_TYPE_BINARY = 9; + SIMPLE_PRIMITIVE_TYPE_DATE = 10; + SIMPLE_PRIMITIVE_TYPE_TIMESTAMP = 11; + SIMPLE_PRIMITIVE_TYPE_TIMESTAMP_NTZ = 12; + SIMPLE_PRIMITIVE_TYPE_VOID = 13; + SIMPLE_PRIMITIVE_TYPE_INTERVAL_YEAR_MONTH = 14; + SIMPLE_PRIMITIVE_TYPE_INTERVAL_DAY_TIME = 15; +} + +// Rust uses `u8` (precision in [1,38], scale in [0,precision]); widened to uint32 because +// proto has no smaller integer type, so consumers must validate the range. +message DecimalType { + uint32 precision = 1; + uint32 scale = 2; +} + +// Edge interpolation algorithm for geography type. +enum EdgeInterpolationAlgorithm { + EDGE_INTERPOLATION_ALGORITHM_UNSPECIFIED = 0; + EDGE_INTERPOLATION_ALGORITHM_SPHERICAL = 1; + EDGE_INTERPOLATION_ALGORITHM_VINCENTY = 2; + EDGE_INTERPOLATION_ALGORITHM_THOMAS = 3; + EDGE_INTERPOLATION_ALGORITHM_ANDOYER = 4; + EDGE_INTERPOLATION_ALGORITHM_KARNEY = 5; +} + +// Geometry column type carrying a coordinate reference system (CRS) in +// AUTHORITY:CODE form. +message GeometryType { + string crs = 1; +} + +// Geography column type carrying a CRS and an edge interpolation algorithm. +message GeographyType { + string crs = 1; + EdgeInterpolationAlgorithm algorithm = 2; +} + +message PrimitiveType { + oneof kind { + SimplePrimitiveType simple = 1; + DecimalType decimal = 2; + GeometryType geometry = 3; + GeographyType geography = 4; + } +} + +message ArrayType { + DataType element_type = 1; + bool contains_null = 2; +} + +message MapType { + DataType key_type = 1; + DataType value_type = 2; + bool value_contains_null = 3; +} + +message StructType { + repeated StructField fields = 1; +} + +message VariantType {} + +message MetadataValue { + oneof value { + int64 number = 1; + string string = 2; + bool boolean = 3; + string other_json = 4; + } +} + +message StructField { + string name = 1; + DataType data_type = 2; + bool nullable = 3; + map metadata = 4; +} + +message DataType { + oneof kind { + PrimitiveType primitive = 1; + ArrayType array = 2; + StructType struct = 3; + MapType map = 4; + VariantType variant = 5; + } +} diff --git a/kernel/src/action_reconciliation/log_replay.rs b/kernel/src/action_reconciliation/log_replay.rs index 4a1baccf36..db0522dedf 100644 --- a/kernel/src/action_reconciliation/log_replay.rs +++ b/kernel/src/action_reconciliation/log_replay.rs @@ -8,41 +8,40 @@ //! //! This processor applies several filtering and deduplication steps to each batch of log actions: //! -//! 1. **Protocol and Metadata**: Retains exactly one of each - keeping only the latest protocol -//! and metadata actions. -//! 2. **Txn Actions**: Keeps exactly one `txn` action for each unique app ID, always selecting -//! the latest one encountered. -//! 3. **File Actions**: Resolves file actions to produce the latest state of the table, keeping -//! the most recent valid add actions and unexpired remove actions (tombstones) that are newer -//! than `minimum_file_retention_timestamp`. +//! 1. **Protocol and Metadata**: Retains exactly one of each - keeping only the latest protocol and +//! metadata actions. +//! 2. **Txn Actions**: Keeps exactly one `txn` action for each unique app ID, always selecting the +//! latest one encountered. +//! 3. **File Actions**: Resolves file actions to produce the latest state of the table, keeping the +//! most recent valid add actions and unexpired remove actions (tombstones) that are newer than +//! `minimum_file_retention_timestamp`. //! //! ## Architecture //! -//! - [`ActionReconciliationVisitor`]: Implements [`RowVisitor`] to examine each action in a batch and -//! determine if it should be included. It maintains state for deduplication across multiple actions -//! in a batch and efficiently handles all filtering rules. +//! - [`ActionReconciliationVisitor`]: Implements [`RowVisitor`] to examine each action in a batch +//! and determine if it should be included. It maintains state for deduplication across multiple +//! actions in a batch and efficiently handles all filtering rules. //! -//! - [`ActionReconciliationProcessor`]: Implements the [`LogReplayProcessor`] trait and orchestrates -//! the overall process. For each batch of log actions, it: +//! - [`ActionReconciliationProcessor`]: Implements the [`LogReplayProcessor`] trait and +//! orchestrates the overall process. For each batch of log actions, it: //! 1. Creates a visitor with the current deduplication state //! 2. Applies the visitor to filter actions in the batch //! 3. Tracks state for deduplication across batches -//! 4. Produces a [`ActionReconciliationBatch`] result which includes both the filtered data and counts of -//! actions selected -//! +//! 4. Produces a [`ActionReconciliationBatch`] result which includes both the filtered data and +//! counts of actions selected +use std::collections::HashSet; +use std::sync::atomic::{AtomicBool, AtomicI64, Ordering}; +use std::sync::{Arc, LazyLock}; + use crate::engine_data::{FilteredEngineData, GetData, RowVisitor, TypedGetData as _}; -use crate::log_replay::deduplicator::Deduplicator as _; +use crate::log_replay::deduplicator::{Deduplicator as _, FileActionInfo}; use crate::log_replay::{ ActionsBatch, FileActionDeduplicator, FileActionKey, HasSelectionVector, LogReplayProcessor, }; use crate::scan::data_skipping::DataSkippingFilter; use crate::schema::{column_name, ColumnName, ColumnNamesAndTypes, DataType}; use crate::utils::require; -use crate::{DeltaResult, Error}; - -use std::collections::HashSet; -use std::sync::atomic::{AtomicBool, AtomicI64, Ordering}; -use std::sync::{Arc, LazyLock}; +use crate::{DeltaResult, DeltaResultIteratorStatic, Error}; /// The [`ActionReconciliationProcessor`] is an implementation of the [`LogReplayProcessor`] /// trait that filters log segment actions. @@ -113,6 +112,16 @@ impl ActionReconciliationIteratorState { pub fn is_exhausted(&self) -> bool { self.is_exhausted.load(Ordering::Acquire) } + + /// Test helper that produces a state with the given counts and marked exhausted. + #[cfg(test)] + pub(crate) fn new_exhausted(actions_count: i64, add_actions_count: i64) -> Self { + Self { + actions_count: AtomicI64::new(actions_count), + add_actions_count: AtomicI64::new(add_actions_count), + is_exhausted: AtomicBool::new(true), + } + } } /// Iterator over action reconciliation data. @@ -120,15 +129,13 @@ impl ActionReconciliationIteratorState { /// This iterator yields a stream of [`FilteredEngineData`] items while, tracking action /// counts. Used by both checkpoint and log compaction workflows. pub struct ActionReconciliationIterator { - inner: Box> + Send>, + inner: DeltaResultIteratorStatic, state: Arc, } impl ActionReconciliationIterator { /// Create a new iterator with counters initialized to 0 - pub(crate) fn new( - inner: Box> + Send>, - ) -> Self { + pub(crate) fn new(inner: DeltaResultIteratorStatic) -> Self { Self { inner, state: Arc::new(ActionReconciliationIteratorState::default()), @@ -185,9 +192,9 @@ impl LogReplayProcessor for ActionReconciliationProcessor { /// and returns a [`ActionReconciliationBatch`], which contains the filtered actions, /// along with statistics about the included actions. /// - /// This method delegates the filtering logic to the [`ActionReconciliationVisitor`], which implements - /// the deduplication rules described in the module documentation. The method tracks - /// statistics about processed actions (total count, add actions count) and maintains + /// This method delegates the filtering logic to the [`ActionReconciliationVisitor`], which + /// implements the deduplication rules described in the module documentation. The method + /// tracks statistics about processed actions (total count, add actions count) and maintains /// state for cross-batch deduplication. fn process_actions_batch(&mut self, actions_batch: ActionsBatch) -> DeltaResult { let ActionsBatch { @@ -223,7 +230,8 @@ impl LogReplayProcessor for ActionReconciliationProcessor { }) } - /// We never do data skipping for action reconciliation log replay (entire table state is always reproduced) + /// We never do data skipping for action reconciliation log replay (entire table state is always + /// reproduced) fn data_skipping_filter(&self) -> Option<&DataSkippingFilter> { None } @@ -255,9 +263,8 @@ impl ActionReconciliationProcessor { /// # File Action Filtering Rules: /// Kept Actions: /// - The first (newest) add action for each unique (path, dvId) pair -/// - The first (newest) remove action for each unique (path, dvId) pair, but only if -/// its deletionTimestamp > minimumFileRetentionTimestamp -/// Omitted Actions: +/// - The first (newest) remove action for each unique (path, dvId) pair, but only if its +/// deletionTimestamp > minimumFileRetentionTimestamp Omitted Actions: /// - Any file action (add/remove) with the same (path, dvId) as a previously processed action /// - All remove actions with deletionTimestamp ≤ minimumFileRetentionTimestamp /// - All remove actions with missing deletionTimestamp (defaults to 0) @@ -272,12 +279,12 @@ impl ActionReconciliationProcessor { /// - Keeps only the first domainMetadata action for each unique domain name /// /// # Excluded Actions -/// - CommitInfo, CDC, and CheckpointMetadata actions should not appear in the action -/// batches processed by this visitor, as they are excluded by the schema used to -/// read the log files upstream. If present, they will be ignored by the visitor. -/// - Sidecar actions should also be excluded—when encountered in the log, the -/// corresponding sidecar files are read to extract the referenced file actions, -/// which are then included directly in the action stream instead of the sidecar actions themselves. +/// - CommitInfo, CDC, and CheckpointMetadata actions should not appear in the action batches +/// processed by this visitor, as they are excluded by the schema used to read the log files +/// upstream. If present, they will be ignored by the visitor. +/// - Sidecar actions should also be excluded—when encountered in the log, the corresponding sidecar +/// files are read to extract the referenced file actions, which are then included directly in the +/// action stream instead of the sidecar actions themselves. /// - The CheckpointMetadata action is included down the wire when writing a V2 spec checkpoint. /// /// # Memory Usage @@ -290,7 +297,8 @@ impl ActionReconciliationProcessor { pub(crate) struct ActionReconciliationVisitor<'seen> { // Deduplicates file actions (applies logic to filter Adds with corresponding Removes, // and keep unexpired Removes). This deduplicator builds a set of seen file actions. - // This set has O(M) memory usage where M = number of file actions with unique (path, dvId) pairs + // This set has O(M) memory usage where M = number of file actions with unique (path, dvId) + // pairs deduplicator: FileActionDeduplicator<'seen>, // Tracks which rows to include in the final output selection_vector: Vec, @@ -310,37 +318,58 @@ pub(crate) struct ActionReconciliationVisitor<'seen> { // This set has O(N) memory usage where N = number of txn actions with unique appIds seen_txns: &'seen mut HashSet, // Set of domain names to deduplicate domainMetadata by domain - // This set has O(D) memory usage where D = number of domainMetadata actions with unique domains + // This set has O(D) memory usage where D = number of domainMetadata actions with unique + // domains seen_domains: &'seen mut HashSet, /// Transaction expiration timestamp for filtering old transactions txn_expiration_timestamp: Option, } +/// A projected column used by `ActionReconciliationVisitor`. +/// +/// `index` is the position in the `getters: &[&dyn GetData]` slice. +/// `name` is the fully-qualified field path used when calling `get_*` (and appears in errors). +/// +/// Invariant: these constants must match the order in +/// `ActionReconciliationVisitor::selected_column_names_and_types()`. +#[derive(Debug, Copy, Clone)] +struct GetterColumn { + index: usize, + name: &'static str, +} + +impl GetterColumn { + const fn new(index: usize, name: &'static str) -> Self { + GetterColumn { index, name } + } +} + #[allow(unused)] impl ActionReconciliationVisitor<'_> { - // TODO(#1717): Combine index and field name constants into a single struct - // These index positions correspond to the order of columns defined in - // `selected_column_names_and_types()` - const ADD_PATH_INDEX: usize = 0; // Position of "add.path" in getters - const ADD_DV_START_INDEX: usize = 1; // Start position of add deletion vector columns - const REMOVE_PATH_INDEX: usize = 4; // Position of "remove.path" in getters - const REMOVE_DELETION_TIMESTAMP_INDEX: usize = 5; // Position of "remove.deletionTimestamp" in getters - const REMOVE_DV_START_INDEX: usize = 6; // Start position of remove deletion vector columns - const METADATA_ID_INDEX: usize = 9; - const PROTOCOL_MIN_READER_VERSION_INDEX: usize = 10; - const TXN_APP_ID_INDEX: usize = 11; - const TXN_LAST_UPDATED_INDEX: usize = 12; - const DOMAIN_METADATA_DOMAIN_INDEX: usize = 13; - const DOMAIN_METADATA_REMOVED_INDEX: usize = 14; - - // These are the column names used to access the data in the getters - const REMOVE_DELETION_TIMESTAMP: &'static str = "remove.deletionTimestamp"; - const PROTOCOL_MIN_READER_VERSION: &'static str = "protocol.minReaderVersion"; - const METADATA_ID: &'static str = "metaData.id"; - const TXN_APP_ID: &'static str = "txn.appId"; - const TXN_LAST_UPDATED: &'static str = "txn.lastUpdated"; - const DOMAIN_METADATA_DOMAIN: &'static str = "domainMetadata.domain"; - const DOMAIN_METADATA_REMOVED: &'static str = "domainMetadata.removed"; + // Projected columns in the same order as `selected_column_names_and_types()`. + // DV columns are defined individually for completeness, even when accessed via a start index. + const ADD_PATH: GetterColumn = GetterColumn::new(0, "add.path"); + const ADD_SIZE: GetterColumn = GetterColumn::new(1, "add.size"); + const ADD_DV_STORAGE_TYPE: GetterColumn = + GetterColumn::new(2, "add.deletionVector.storageType"); + const ADD_DV_PATH_OR_INLINE_DV: GetterColumn = + GetterColumn::new(3, "add.deletionVector.pathOrInlineDv"); + const ADD_DV_OFFSET: GetterColumn = GetterColumn::new(4, "add.deletionVector.offset"); + const REMOVE_PATH: GetterColumn = GetterColumn::new(5, "remove.path"); + const REMOVE_DELETION_TIMESTAMP: GetterColumn = + GetterColumn::new(6, "remove.deletionTimestamp"); + const REMOVE_DV_STORAGE_TYPE: GetterColumn = + GetterColumn::new(7, "remove.deletionVector.storageType"); + const REMOVE_DV_PATH_OR_INLINE_DV: GetterColumn = + GetterColumn::new(8, "remove.deletionVector.pathOrInlineDv"); + const REMOVE_DV_OFFSET: GetterColumn = GetterColumn::new(9, "remove.deletionVector.offset"); + const METADATA_ID: GetterColumn = GetterColumn::new(10, "metaData.id"); + const PROTOCOL_MIN_READER_VERSION: GetterColumn = + GetterColumn::new(11, "protocol.minReaderVersion"); + const TXN_APP_ID: GetterColumn = GetterColumn::new(12, "txn.appId"); + const TXN_LAST_UPDATED: GetterColumn = GetterColumn::new(13, "txn.lastUpdated"); + const DOMAIN_METADATA_DOMAIN: GetterColumn = GetterColumn::new(14, "domainMetadata.domain"); + const DOMAIN_METADATA_REMOVED: GetterColumn = GetterColumn::new(15, "domainMetadata.removed"); #[allow(clippy::too_many_arguments)] pub(crate) fn new<'seen>( @@ -358,10 +387,11 @@ impl ActionReconciliationVisitor<'_> { deduplicator: FileActionDeduplicator::new( seen_file_keys, is_log_batch, - Self::ADD_PATH_INDEX, - Self::REMOVE_PATH_INDEX, - Self::ADD_DV_START_INDEX, - Self::REMOVE_DV_START_INDEX, + Self::ADD_PATH.index, + Self::ADD_SIZE.index, + Self::REMOVE_PATH.index, + Self::ADD_DV_STORAGE_TYPE.index, + Self::REMOVE_DV_STORAGE_TYPE.index, ), selection_vector, actions_count: 0, @@ -377,9 +407,9 @@ impl ActionReconciliationVisitor<'_> { /// Determines if a remove action tombstone has expired and should be excluded. /// - /// A remove action includes a deletion_timestamp indicating when the deletion occurred. Physical - /// files are deleted lazily after a user-defined expiration time. Remove actions are kept to allow - /// concurrent readers to read snapshots at older versions. + /// A remove action includes a deletion_timestamp indicating when the deletion occurred. + /// Physical files are deleted lazily after a user-defined expiration time. Remove actions + /// are kept to allow concurrent readers to read snapshots at older versions. /// /// Tombstone expiration rules: /// - If deletion_timestamp <= minimum_file_retention_timestamp: Expired (exclude) @@ -390,7 +420,7 @@ impl ActionReconciliationVisitor<'_> { // Spark and the Java Kernel. // Note: When remove.deletion_timestamp is not present (defaulting to 0), the remove action // will be excluded as it will be treated as expired. - let deletion_timestamp = getter.get_opt(i, "remove.deletionTimestamp")?; + let deletion_timestamp = getter.get_opt(i, Self::REMOVE_DELETION_TIMESTAMP.name)?; let deletion_timestamp = deletion_timestamp.unwrap_or(0i64); Ok(deletion_timestamp <= self.minimum_file_retention_timestamp) @@ -399,9 +429,10 @@ impl ActionReconciliationVisitor<'_> { /// Processes a potential file action to determine if it should be included. /// /// Returns `Ok(Some(true))` if the row contains a valid file action to be included. - /// Returns `Ok(Some(false))` if the row contains a file action but it's suppressed (duplicate/expired). - /// Returns `Ok(None)` if the row doesn't contain a file action (continue checking other action types). - /// Returns `Err(...)` if there was an error processing the action. + /// Returns `Ok(Some(false))` if the row contains a file action but it's suppressed + /// (duplicate/expired). Returns `Ok(None)` if the row doesn't contain a file action + /// (continue checking other action types). Returns `Err(...)` if there was an error + /// processing the action. /// /// Note: This function handles both add and remove actions, applying deduplication logic and /// tombstone expiration rules as needed. @@ -411,7 +442,11 @@ impl ActionReconciliationVisitor<'_> { getters: &[&'a dyn GetData<'a>], ) -> DeltaResult> { // Extract the file action and handle errors immediately - let Some((file_key, is_add)) = self.deduplicator.extract_file_action(i, getters, false)? + let Some(FileActionInfo { + key: file_key, + is_add, + .. + }) = self.deduplicator.extract_file_action(i, getters, false)? else { return Ok(None); // No file action found, continue checking other types }; @@ -424,7 +459,7 @@ impl ActionReconciliationVisitor<'_> { true } else { // Expired remove actions are not valid - !self.is_expired_tombstone(i, getters[Self::REMOVE_DELETION_TIMESTAMP_INDEX])? + !self.is_expired_tombstone(i, getters[Self::REMOVE_DELETION_TIMESTAMP.index])? }; Ok(Some(is_valid)) } @@ -432,18 +467,20 @@ impl ActionReconciliationVisitor<'_> { /// Processes a potential protocol action to determine if it should be included. /// /// Returns `Ok(Some(true))` if the row contains a valid protocol action. - /// Returns `Ok(Some(false))` if the row contains a protocol action but it's suppressed (duplicate). - /// Returns `Ok(None)` if the row doesn't contain a protocol action (continue checking other action types). - /// Returns `Err(...)` if there was an error processing the action. + /// Returns `Ok(Some(false))` if the row contains a protocol action but it's suppressed + /// (duplicate). Returns `Ok(None)` if the row doesn't contain a protocol action (continue + /// checking other action types). Returns `Err(...)` if there was an error processing the + /// action. fn check_protocol_action<'a>( &mut self, i: usize, getter: &'a dyn GetData<'a>, ) -> DeltaResult> { - // minReaderVersion is a required field, so we check for its presence to determine if this is a protocol action. - // Only return the first (newest) protocol action we see, ignoring other types + // minReaderVersion is a required field, so we check for its presence to determine if this + // is a protocol action. Only return the first (newest) protocol action we see, + // ignoring other types let result = getter - .get_int(i, Self::PROTOCOL_MIN_READER_VERSION)? + .get_int(i, Self::PROTOCOL_MIN_READER_VERSION.name)? .is_some() .then(|| !std::mem::replace(&mut self.seen_protocol, true)); Ok(result) @@ -452,18 +489,20 @@ impl ActionReconciliationVisitor<'_> { /// Processes a potential metadata action to determine if it should be included. /// /// Returns `Ok(Some(true))` if the row contains a valid metadata action. - /// Returns `Ok(Some(false))` if the row contains a metadata action but it's suppressed (duplicate). - /// Returns `Ok(None)` if the row doesn't contain a metadata action (continue checking other action types). - /// Returns `Err(...)` if there was an error processing the action. + /// Returns `Ok(Some(false))` if the row contains a metadata action but it's suppressed + /// (duplicate). Returns `Ok(None)` if the row doesn't contain a metadata action (continue + /// checking other action types). Returns `Err(...)` if there was an error processing the + /// action. fn check_metadata_action<'a>( &mut self, i: usize, getter: &'a dyn GetData<'a>, ) -> DeltaResult> { - // id is a required field, so we check for its presence to determine if this is a metadata action. - // Only return the first (newest) metadata action we see, ignoring other types + // id is a required field, so we check for its presence to determine if this is a metadata + // action. Only return the first (newest) metadata action we see, ignoring other + // types let result = getter - .get_str(i, Self::METADATA_ID)? + .get_str(i, Self::METADATA_ID.name)? .is_some() .then(|| !std::mem::replace(&mut self.seen_metadata, true)); Ok(result) @@ -472,35 +511,41 @@ impl ActionReconciliationVisitor<'_> { /// Processes a potential txn action to determine if it should be included. /// /// Returns `Ok(Some(true))` if the row contains a valid txn action. - /// Returns `Ok(Some(false))` if the row contains a txn action but it's suppressed (duplicate/expired). - /// Returns `Ok(None)` if the row doesn't contain a txn action (continue checking other action types). - /// Returns `Err(...)` if there was an error processing the action. + /// Returns `Ok(Some(false))` if the row contains a txn action but it's suppressed + /// (duplicate/expired). Returns `Ok(None)` if the row doesn't contain a txn action + /// (continue checking other action types). Returns `Err(...)` if there was an error + /// processing the action. fn check_txn_action<'a>( &mut self, i: usize, getters: &[&'a dyn GetData<'a>], ) -> DeltaResult> { - let Some(app_id) = getters[Self::TXN_APP_ID_INDEX].get_str(i, Self::TXN_APP_ID)? else { + let Some(app_id) = getters[Self::TXN_APP_ID.index].get_str(i, Self::TXN_APP_ID.name)? + else { return Ok(None); // Not a txn action, continue checking other types }; - // Check retention if last_updated is present + // Replay is newest-to-oldest, so the first txn seen for an app_id is the winner. Record it + // before checking retention: an expired winner must still suppress older txns for the same + // app_id rather than let one of them survive. + if !self.seen_txns.insert(app_id.to_string()) { + return Ok(Some(false)); // superseded by a newer txn for this app_id + } + + // Exclude the winner when retention has expired it. A txn without last_updated never + // expires (kept for backward compatibility). if let Some(retention_ts) = self.txn_expiration_timestamp { if let Some(last_updated) = - getters[Self::TXN_LAST_UPDATED_INDEX].get_opt(i, Self::TXN_LAST_UPDATED)? + getters[Self::TXN_LAST_UPDATED.index].get_opt(i, Self::TXN_LAST_UPDATED.name)? { let last_updated: i64 = last_updated; if last_updated <= retention_ts { - // Transaction is old, exclude it return Ok(Some(false)); } } - // Note: transactions without last_updated are kept for backward compatibility } - // If the app ID already exists in the set, the insertion will return false, - // indicating that this is a duplicate. - Ok(Some(self.seen_txns.insert(app_id.to_string()))) + Ok(Some(true)) } /// Processes a potential domainMetadata action to determine if it should be included. @@ -508,30 +553,36 @@ impl ActionReconciliationVisitor<'_> { /// Returns `Ok(Some(true))` if the row contains a valid domainMetadata action. /// Returns `Ok(Some(false))` if the row contains a domainMetadata action but it's suppressed /// (duplicate or tombstone with removed=true). - /// Returns `Ok(None)` if the row doesn't contain a domainMetadata action (continue checking other action types). - /// Returns `Err(...)` if there was an error processing the action. + /// Returns `Ok(None)` if the row doesn't contain a domainMetadata action (continue checking + /// other action types). Returns `Err(...)` if there was an error processing the action. fn check_domain_metadata_action<'a>( &mut self, i: usize, getters: &[&'a dyn GetData<'a>], ) -> DeltaResult> { - let Some(domain) = - getters[Self::DOMAIN_METADATA_DOMAIN_INDEX].get_str(i, Self::DOMAIN_METADATA_DOMAIN)? + let Some(domain) = getters[Self::DOMAIN_METADATA_DOMAIN.index] + .get_str(i, Self::DOMAIN_METADATA_DOMAIN.name)? else { return Ok(None); // Not a domainMetadata action, continue checking other types }; - // Exclude tombstones (removed=true) from checkpoint per protocol spec - let removed: bool = getters[Self::DOMAIN_METADATA_REMOVED_INDEX] - .get_opt(i, Self::DOMAIN_METADATA_REMOVED)? + // Record the domain as seen first so older versions are deduplicated + // even when a newer version is a tombstone. Log replay walks newest-to-oldest, + // so a tombstone at a later version must still mask earlier versions of the + // same domain in the checkpoint. + if !self.seen_domains.insert(domain.to_string()) { + return Ok(Some(false)); // duplicate - older version of a domain we've already seen + } + + // Exclude tombstones (removed=true) from the checkpoint per protocol spec. + let removed: bool = getters[Self::DOMAIN_METADATA_REMOVED.index] + .get_opt(i, Self::DOMAIN_METADATA_REMOVED.name)? .unwrap_or(false); if removed { return Ok(Some(false)); } - // If the domain already exists in the set, the insertion will return false, - // indicating that this is a duplicate. - Ok(Some(self.seen_domains.insert(domain.to_string()))) + Ok(Some(true)) } /// Determines if a row in the batch should be included. @@ -562,11 +613,11 @@ impl ActionReconciliationVisitor<'_> { } else if let Some(result) = self.check_domain_metadata_action(i, getters)? { result } else if let Some(result) = - self.check_protocol_action(i, getters[Self::PROTOCOL_MIN_READER_VERSION_INDEX])? + self.check_protocol_action(i, getters[Self::PROTOCOL_MIN_READER_VERSION.index])? { result } else { - self.check_metadata_action(i, getters[Self::METADATA_ID_INDEX])? + self.check_metadata_action(i, getters[Self::METADATA_ID.index])? .unwrap_or_default() }; @@ -596,6 +647,7 @@ impl RowVisitor for ActionReconciliationVisitor<'_> { let types_and_names = vec![ // File action columns (STRING, column_name!("add.path")), + (LONG, column_name!("add.size")), (STRING, column_name!("add.deletionVector.storageType")), (STRING, column_name!("add.deletionVector.pathOrInlineDv")), (INTEGER, column_name!("add.deletionVector.offset")), @@ -620,9 +672,9 @@ impl RowVisitor for ActionReconciliationVisitor<'_> { fn visit<'a>(&mut self, row_count: usize, getters: &[&'a dyn GetData<'a>]) -> DeltaResult<()> { require!( - getters.len() == 15, + getters.len() == 16, Error::InternalError(format!( - "Wrong number of visitor getters: {}", + "Wrong number of visitor getters for ActionReconciliationVisitor: {}", getters.len() )) ); @@ -638,13 +690,13 @@ impl RowVisitor for ActionReconciliationVisitor<'_> { mod tests { use std::collections::HashSet; + use itertools::Itertools; + use super::*; use crate::arrow::array::StringArray; - use crate::utils::test_utils::{action_batch, parse_json_batch}; + use crate::unit_test_utils::{action_batch, parse_json_batch}; use crate::Error; - use itertools::Itertools; - /// Helper function to create test batches from JSON strings fn create_batch(json_strings: Vec<&str>) -> DeltaResult { let actions = parse_json_batch(StringArray::from(json_strings)); @@ -712,7 +764,8 @@ mod tests { /// Tests the boundary conditions for tombstone expiration logic. /// Specifically checks: - /// - Remove actions with deletionTimestamp == minimumFileRetentionTimestamp (should be excluded) + /// - Remove actions with deletionTimestamp == minimumFileRetentionTimestamp (should be + /// excluded) /// - Remove actions with deletionTimestamp < minimumFileRetentionTimestamp (should be excluded) /// - Remove actions with deletionTimestamp > minimumFileRetentionTimestamp (should be included) /// - Remove actions with missing deletionTimestamp (defaults to 0, should be excluded) @@ -1057,14 +1110,51 @@ mod tests { visitor.visit_rows_of(batch.as_ref())?; - // app1 and app4 should be filtered out (too old) - // app2 and app3 should be kept + // app1 and app4 are excluded (expired); app2 and app3 are emitted. All four app_ids are + // recorded as seen, since recording precedes the retention check. let expected = vec![false, true, true, false]; assert_eq!(visitor.selection_vector, expected); assert_eq!(visitor.actions_count, 2); - assert_eq!(visitor.seen_txns.len(), 2); - assert!(visitor.seen_txns.contains("app2")); - assert!(visitor.seen_txns.contains("app3")); + assert_eq!(visitor.seen_txns.len(), 4); + + Ok(()) + } + + // Replay is newest-to-oldest. When an app_id's newest txn is expired but an older one is not, + // the app_id must be dropped entirely: the expired newest suppresses the older duplicate, and + // neither reaches the checkpoint. Guards against resurrecting the older txn (a stale winner). + #[test] + fn test_action_reconciliation_expired_newest_txn_suppresses_older_txn_for_same_app( + ) -> DeltaResult<()> { + let json_strings: StringArray = vec![ + // Newest for "app" (visited first), expired. + r#"{"txn":{"appId":"app","version":2,"lastUpdated":500}}"#, + // Older for "app", not expired. Must NOT survive. + r#"{"txn":{"appId":"app","version":1,"lastUpdated":2000}}"#, + ] + .into(); + let batch = parse_json_batch(json_strings); + + let mut seen_file_keys = HashSet::new(); + let mut seen_txns = HashSet::new(); + let mut seen_domains = HashSet::new(); + let mut visitor = ActionReconciliationVisitor::new( + &mut seen_file_keys, + true, + vec![true; 2], + 0, + false, + false, + &mut seen_txns, + &mut seen_domains, + Some(1000), + ); + + visitor.visit_rows_of(batch.as_ref())?; + + assert_eq!(visitor.selection_vector, vec![false, false]); + assert_eq!(visitor.actions_count, 0); + assert_eq!(visitor.seen_txns.len(), 1); Ok(()) } @@ -1222,7 +1312,7 @@ mod tests { error_field: &'static str, error_type: &'static str, ) -> Vec { - (0..15) + (0..16) .map(|i| { if i == error_index { MockErrorGetData::new(error_field, error_type) @@ -1253,14 +1343,14 @@ mod tests { // Test 2: Basic type mismatch errors using parameterized approach let test_cases = [ (0, "add.path", "str", "add.path is not of type str"), - (9, "metaData.id", "str", "metaData.id is not of type str"), + (10, "metaData.id", "str", "metaData.id is not of type str"), ( - 10, + 11, "protocol.minReaderVersion", "int", "protocol.minReaderVersion is not of type i32", ), - (11, "txn.appId", "str", "txn.appId is not of type str"), + (12, "txn.appId", "str", "txn.appId is not of type str"), ]; for (getter_index, field_name, error_type, expected_error_text) in test_cases { @@ -1293,7 +1383,7 @@ mod tests { &mut seen_domains, Some(1000), ); - let defaults = (0..11) + let defaults = (0..12) .map(|_| MockErrorGetData::default()) .collect::>(); let error_mock = FlexibleMock { @@ -1309,10 +1399,8 @@ mod tests { getters.push(&domain_removed_default); // domainMetadata.removed let result = visitor.visit(1, &getters); assert!(result.is_err()); - assert!(result - .unwrap_err() - .to_string() - .contains("lastUpdated is not of type i64")); + let err_str = result.unwrap_err().to_string(); + assert!(err_str.contains("lastUpdated is not of type i64")); // Test remove.deletionTimestamp let mut seen_file_keys = HashSet::new(); @@ -1320,7 +1408,7 @@ mod tests { let mut seen_domains = HashSet::new(); let mut visitor = create_test_visitor(&mut seen_file_keys, &mut seen_txns, &mut seen_domains, None); - let defaults = (0..4) + let defaults = (0..5) .map(|_| MockErrorGetData::default()) .collect::>(); let error_mock = FlexibleMock { @@ -1353,8 +1441,8 @@ mod tests { let batch = ActionsBatch::new(actions, true); // Create a processor and try to process the batch - // We can't easily trigger an error in the normal flow since parse_json_batch creates valid data - // But this test ensures the error propagation path exists and is tested + // We can't easily trigger an error in the normal flow since parse_json_batch creates valid + // data But this test ensures the error propagation path exists and is tested let mut processor = ActionReconciliationProcessor::new(0, None); let result = processor.process_actions_batch(batch); diff --git a/kernel/src/action_reconciliation/mod.rs b/kernel/src/action_reconciliation/mod.rs index ccc824f1a0..7c287b0537 100644 --- a/kernel/src/action_reconciliation/mod.rs +++ b/kernel/src/action_reconciliation/mod.rs @@ -6,13 +6,14 @@ //! //! ## Log Replay for Action Reconciliation //! -//! The [`log_replay`] module provides specialized log replay functionality for action reconciliation, -//! including checkpoint creation. It processes log files in reverse chronological order and selects -//! the appropriate actions to include based on deduplication and retention rules. +//! The [`log_replay`] module provides specialized log replay functionality for action +//! reconciliation, including checkpoint creation. It processes log files in reverse chronological +//! order and selects the appropriate actions to include based on deduplication and retention rules. //! //! ## Retention and Cleanup //! -//! This module provides utilities for calculating retention timestamps used during action reconciliation: +//! This module provides utilities for calculating retention timestamps used during action +//! reconciliation: //! //! - **Deleted File Retention**: Determines when `remove` actions can be excluded from checkpoints //! - **Transaction Retention**: Calculates when expired app ids can be cleaned up @@ -87,8 +88,8 @@ pub(crate) trait RetentionCalculator { /// # Parameters /// - `retention_duration`: The duration to retain deleted files. The table property /// `deleted_file_retention_duration` is passed here. If `None`, defaults to 7 days. -/// - `now_duration`: The current time as a [`Duration`]. This allows for testing with -/// a specific time instead of using `SystemTime::now()`. +/// - `now_duration`: The current time as a [`Duration`]. This allows for testing with a specific +/// time instead of using `SystemTime::now()`. /// /// # Returns: The timestamp in milliseconds since epoch pub(crate) fn deleted_file_retention_timestamp_with_time( @@ -130,9 +131,10 @@ pub(crate) fn calculate_transaction_expiration_timestamp( #[cfg(test)] mod tests { - use super::*; use std::time::Duration; + use super::*; + #[test] fn test_deleted_file_retention_timestamp_with_time() -> DeltaResult<()> { // Test with default retention (7 days) diff --git a/kernel/src/actions/deletion_vector.rs b/kernel/src/actions/deletion_vector.rs index 21f9287443..f3d1639874 100644 --- a/kernel/src/actions/deletion_vector.rs +++ b/kernel/src/actions/deletion_vector.rs @@ -5,13 +5,12 @@ use std::str::FromStr; use std::sync::Arc; use bytes::Bytes; +use crc::{Crc, CRC_32_ISO_HDLC}; use delta_kernel::schema::derive_macro_utils::ToDataType; -use delta_kernel_derive::ToSchema; +use delta_kernel_derive::{internal_api, ToSchema}; use roaring::RoaringTreemap; use url::Url; -use crc::{Crc, CRC_32_ISO_HDLC}; - use crate::schema::DataType; use crate::utils::require; use crate::{DeltaResult, Error, StorageHandler}; @@ -136,23 +135,23 @@ pub struct DeletionVectorDescriptor { pub storage_type: DeletionVectorStorageType, /// Three format options are currently proposed: - /// - If `storageType = 'u'` then ``: - /// The deletion vector is stored in a file with a path relative to the data - /// directory of this Delta table, and the file name can be reconstructed from - /// the UUID. See Derived Fields for how to reconstruct the file name. The random - /// prefix is recovered as the extra characters before the (20 characters fixed length) uuid. - /// - If `storageType = 'i'` then ``: The deletion vector - /// is stored inline in the log. The format used is the `RoaringBitmapArray` - /// format also used when the DV is stored on disk and described in [Deletion Vector Format]. + /// - If `storageType = 'u'` then ``: The + /// deletion vector is stored in a file with a path relative to the data directory of this + /// Delta table, and the file name can be reconstructed from the UUID. See Derived Fields for + /// how to reconstruct the file name. The random prefix is recovered as the extra characters + /// before the (20 characters fixed length) uuid. + /// - If `storageType = 'i'` then ``: The deletion vector is stored + /// inline in the log. The format used is the `RoaringBitmapArray` format also used when the + /// DV is stored on disk and described in [Deletion Vector Format]. /// - If `storageType = 'p'` then ``: The DV is stored in a file with an - /// absolute path given by this path, which has the same format as the `path` field - /// in the `add`/`remove` actions. + /// absolute path given by this path, which has the same format as the `path` field in the + /// `add`/`remove` actions. /// /// [Deletion Vector Format]: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Deletion-Vector-Format pub path_or_inline_dv: String, - /// Start of the data for this DV in number of bytes from the beginning of the file it is stored in. - /// Always None (absent in JSON) when `storageType = 'i'`. + /// Start of the data for this DV in number of bytes from the beginning of the file it is + /// stored in. Always None (absent in JSON) when `storageType = 'i'`. pub offset: Option, /// Size of the serialized DV in bytes (raw data size, i.e. before base85 encoding, if inline). @@ -163,6 +162,80 @@ pub struct DeletionVectorDescriptor { } impl DeletionVectorDescriptor { + /// Construct a validated [`DeletionVectorDescriptor`] from its raw fields. + /// + /// Validates the protocol-level invariants from the "Deletion Vector Descriptor Schema" + /// section of the Delta protocol: + /// - `size_in_bytes` and `cardinality` must be non-negative. + /// - If `offset` is present, it must be non-negative. + /// - `Inline` descriptors must not carry an offset. + /// - `PersistedRelative` paths carry an optional random prefix followed by a 20-character + /// z85-encoded UUID, so they must be at least 20 characters long. + /// - `PersistedAbsolute` paths must parse as a URL. + /// + /// `Inline` payload bytes are accepted verbatim; the framing of the embedded RoaringBitmap + /// is only checked when the DV is later read via [`Self::read`]. + pub fn try_new( + storage_type: DeletionVectorStorageType, + path_or_inline_dv: impl Into, + offset: Option, + size_in_bytes: i32, + cardinality: i64, + ) -> DeltaResult { + require!( + size_in_bytes >= 0, + Error::deletion_vector("size_in_bytes must be non-negative") + ); + require!( + cardinality >= 0, + Error::deletion_vector("cardinality must be non-negative") + ); + require!( + offset.is_none_or(|o| o >= 0), + Error::deletion_vector("offset must be non-negative") + ); + let path_or_inline_dv = path_or_inline_dv.into(); + match storage_type { + DeletionVectorStorageType::Inline => require!( + offset.is_none(), + Error::deletion_vector("inline deletion vectors must not carry an offset") + ), + DeletionVectorStorageType::PersistedRelative => { + // Byte-slice rather than char-slice: z85 is ASCII-only, and string slicing + // would panic if a non-ASCII byte boundary fell inside the trailing 20-byte + // window. `z85::decode` accepts `&[u8]` and rejects non-z85 bytes. + let bytes = path_or_inline_dv.as_bytes(); + require!( + bytes.len() >= 20, + Error::deletion_vector(format!( + "persisted-relative DV path must be at least 20 bytes, got {}", + bytes.len() + )) + ); + let suffix = &bytes[bytes.len() - 20..]; + z85::decode(suffix).map_err(|_| { + Error::deletion_vector( + "persisted-relative DV path must end with a z85-encoded UUID", + ) + })?; + } + DeletionVectorStorageType::PersistedAbsolute => { + Url::parse(&path_or_inline_dv).map_err(|e| { + Error::deletion_vector(format!( + "persisted-absolute DV path must parse as a URL: {e}" + )) + })?; + } + } + Ok(Self { + storage_type, + path_or_inline_dv, + offset, + size_in_bytes, + cardinality, + }) + } + pub fn unique_id(&self) -> String { Self::unique_id_from_parts( &self.storage_type.to_string(), @@ -181,21 +254,48 @@ impl DeletionVectorDescriptor { } } + /// Decodes a `PersistedRelative` path to its relative-path form + /// (`/deletion_vector_.bin`). + /// + /// The encoded path is an optional random prefix followed by a fixed-length 20-character z85 + /// UUID; the prefix becomes a directory component, and is omitted entirely when absent: + /// + /// ```text + /// "ab^-aqEH.-t@S}K{vb[*k^" -> "ab/deletion_vector_d2c639aa-8816-431a-aaf6-d3fe2512ff61.bin" + /// "vBn[lx{q8@P<9BNH/isA" -> "deletion_vector_61d16c75-6994-46b7-a15b-8b538852e50e.bin" + /// ``` + /// + /// Errors if called on a non-`PersistedRelative` descriptor, if the encoded path is shorter + /// than the 20-character z85 UUID suffix, or if that suffix fails to decode into a UUID. + pub(crate) fn relative_path(&self) -> DeltaResult { + require!( + self.storage_type == DeletionVectorStorageType::PersistedRelative, + Error::DeletionVector(format!( + "relative_path is only valid for PersistedRelative, got {:?}", + self.storage_type + )) + ); + // Byte-slice rather than char-slice: z85 is ASCII-only, and string slicing would panic if + // a non-ASCII byte boundary fell inside the trailing 20-byte window. Mirrors `try_new`. + let bytes = self.path_or_inline_dv.as_bytes(); + require!( + bytes.len() >= 20, + Error::DeletionVector(format!("Invalid length {}, must be >= 20", bytes.len())) + ); + let prefix_len = bytes.len() - 20; + let decoded = z85::decode(&bytes[prefix_len..]) + .map_err(|_| Error::deletion_vector("Failed to decode DV uuid"))?; + let uuid = uuid::Uuid::from_slice(&decoded) + .map_err(|err| Error::DeletionVector(err.to_string()))?; + let prefix = std::str::from_utf8(&bytes[..prefix_len]) + .map_err(|_| Error::deletion_vector("DV path prefix is not valid UTF-8"))?; + Ok(DeletionVectorPath::relative_path(prefix, &uuid)) + } + pub fn absolute_path(&self, parent: &Url) -> DeltaResult> { match self.storage_type { DeletionVectorStorageType::PersistedRelative => { - let path_len = self.path_or_inline_dv.len(); - require!( - path_len >= 20, - Error::DeletionVector(format!("Invalid length {path_len}, must be >= 20")) - ); - let prefix_len = path_len - 20; - let decoded = z85::decode(&self.path_or_inline_dv[prefix_len..]) - .map_err(|_| Error::deletion_vector("Failed to decode DV uuid"))?; - let uuid = uuid::Uuid::from_slice(&decoded) - .map_err(|err| Error::DeletionVector(err.to_string()))?; - let dv_suffix = - DeletionVectorPath::relative_path(&self.path_or_inline_dv[..prefix_len], &uuid); + let dv_suffix = self.relative_path()?; let dv_path = parent .join(&dv_suffix) .map_err(|_| Error::DeletionVector(format!("invalid path: {dv_suffix}")))?; @@ -293,8 +393,8 @@ impl DeletionVectorDescriptor { // bitmap_start = this_dv_start + 4 (dv_size field) + 4 (magic field) let bitmap_start = this_dv_start + 8; // crc_start = this_dv_start + 4 (dv_size field) + dv_size (magic field + bitmap) - // Safety: size_in_bytes is checked to fit in u32 which for all known platforms should - // fix in usize range. + // Safety: size_in_bytes is checked to fit in u32 which for all known platforms + // should fix in usize range. let crc_start = this_dv_start + 4 + (size_in_bytes as usize); require!( this_dv_start < dv_data_len, @@ -401,18 +501,21 @@ fn slice_to_u32(buf: &[u8], endian: Endian) -> DeltaResult { /// helper function to convert a treemap into a boolean vector where, for index i, if the bit is /// set, the vector will be false, and otherwise at index i the vector will be true +#[internal_api] pub(crate) fn deletion_treemap_to_bools(treemap: RoaringTreemap) -> Vec { treemap_to_bools_with(treemap, false) } /// helper function to convert a treemap into a boolean vector where, for index i, if the bit is /// set, the vector will be true, and otherwise at index i the vector will be false +#[internal_api] pub(crate) fn selection_treemap_to_bools(treemap: RoaringTreemap) -> Vec { treemap_to_bools_with(treemap, true) } /// helper function to generate vectors of bools from treemap. If `set_bit` is `true`, this is /// [`selection_treemap_to_bools`]. If `set_bit` is false, this is [`deletion_treemap_to_bools`] +#[internal_api] fn treemap_to_bools_with(treemap: RoaringTreemap, set_bit: bool) -> Vec { fn combine(high_bits: u32, low_bits: u32) -> usize { ((u64::from(high_bits) << 32) | u64::from(low_bits)) as usize @@ -474,10 +577,9 @@ mod tests { use roaring::RoaringTreemap; - use crate::{engine::sync::SyncEngine, Engine}; - - use super::DeletionVectorDescriptor; - use super::*; + use super::{DeletionVectorDescriptor, *}; + use crate::engine::sync::SyncEngine; + use crate::Engine; fn dv_relative() -> DeletionVectorDescriptor { DeletionVectorDescriptor { @@ -550,6 +652,74 @@ mod tests { assert_eq!(dv_url, example.absolute_path(&parent).unwrap().unwrap()); } + fn dv_with_path( + storage_type: DeletionVectorStorageType, + path_or_inline_dv: &str, + ) -> DeletionVectorDescriptor { + DeletionVectorDescriptor { + storage_type, + path_or_inline_dv: path_or_inline_dv.to_string(), + offset: Some(1), + size_in_bytes: 36, + cardinality: 2, + } + } + + #[rstest::rstest] + // z85 UUID with a random prefix -> `/deletion_vector_.bin`. + #[case( + "ab^-aqEH.-t@S}K{vb[*k^", + "ab/deletion_vector_d2c639aa-8816-431a-aaf6-d3fe2512ff61.bin" + )] + // Bare 20-char z85 UUID, no prefix. + #[case( + "vBn[lx{q8@P<9BNH/isA", + "deletion_vector_61d16c75-6994-46b7-a15b-8b538852e50e.bin" + )] + fn test_relative_path_decodes(#[case] encoded: &str, #[case] expected: &str) { + let dv = dv_with_path(DeletionVectorStorageType::PersistedRelative, encoded); + assert_eq!(dv.relative_path().unwrap(), expected); + } + + #[rstest::rstest] + // Non-relative storage types are rejected by the guard. + #[case( + DeletionVectorStorageType::PersistedAbsolute, + "s3://mytable/deletion_vector_d2c639aa-8816-431a-aaf6-d3fe2512ff61.bin", + "only valid for PersistedRelative" + )] + #[case( + DeletionVectorStorageType::Inline, + "^Bg9^0rr910000000000iXQKl0rr91000f55c8Xg0@@D72lkbi5=-{L", + "only valid for PersistedRelative" + )] + // Path shorter than the 20-byte z85 UUID suffix. + #[case( + DeletionVectorStorageType::PersistedRelative, + "short", + "Invalid length" + )] + // A non-ASCII byte straddling the trailing-20-byte window must error, not panic. `é` is two + // bytes, so this 21-byte path leaves `prefix_len == 1` inside the multi-byte `é`; byte-slicing + // (vs the pre-fix char-slicing) errors cleanly instead of panicking. + #[case( + DeletionVectorStorageType::PersistedRelative, + "éaaaaaaaaaaaaaaaaaaa", + "Failed to decode DV uuid" + )] + fn test_relative_path_errors( + #[case] storage_type: DeletionVectorStorageType, + #[case] path_or_inline_dv: &str, + #[case] expected_error: &str, + ) { + let dv = dv_with_path(storage_type, path_or_inline_dv); + let err = dv.relative_path().unwrap_err().to_string(); + assert!( + err.contains(expected_error), + "error {err:?} did not contain {expected_error:?}" + ); + } + #[test] fn test_magic_number_constants() { assert_eq!(ROARING_BITMAP_PORTABLE_MAGIC, 1681511377); @@ -618,8 +788,9 @@ mod tests { assert_eq!(found, expected) } - // this test is ignored by default as it's expensive to allocate such big vecs full of `true`. you can run it via: - // cargo test actions::deletion_vector::tests::test_dv_to_bools -- --ignored + // this test is ignored by default as it's expensive to allocate such big vecs full of `true`. + // you can run it via: cargo test actions::deletion_vector::tests::test_dv_to_bools -- + // --ignored #[test] #[ignore] fn test_dv_to_bools() { @@ -641,9 +812,9 @@ mod tests { assert_eq!(bools, expected); } - // Unlike [`test_dv_to_bools`], this test is not ignored because the large zero-initialized selection vector is fast to allocate. - // It just gets a bunch of empty pages from the OS. [`tet_dv_to_bools`] is slow because we must - // set every element to `true`. + // Unlike [`test_dv_to_bools`], this test is not ignored because the large zero-initialized + // selection vector is fast to allocate. It just gets a bunch of empty pages from the OS. + // [`tet_dv_to_bools`] is slow because we must set every element to `true`. #[test] fn test_sv_to_bools() { let mut rb = RoaringTreemap::new(); @@ -789,7 +960,8 @@ mod tests { "file:///tmp/test_table/dv/deletion_vector_550e8400-e29b-41d4-a716-446655440000.bin"; assert_eq!(abs_path.as_str(), expected_path); - // Verify the encoded_relative_path is exactly as expected (prefix + z85 encoded UUID: 20 chars) + // Verify the encoded_relative_path is exactly as expected (prefix + z85 encoded UUID: 20 + // chars) let encoded = dv_path.encoded_relative_path(); assert_eq!(encoded, "dvrsTVZ&*Sl-RXRWjryu/!"); } @@ -813,4 +985,94 @@ mod tests { let encoded = dv_path.encoded_relative_path(); assert_eq!(encoded, "5:JjlQ/G/]6C<1m"); } + + #[rstest::rstest] + #[case::inline_with_offset(DeletionVectorStorageType::Inline, "ABC", Some(0), 4, 1, "inline")] + #[case::persisted_relative_short_path( + DeletionVectorStorageType::PersistedRelative, + "short", + Some(1), + 4, + 1, + "20 bytes" + )] + #[case::persisted_relative_invalid_z85( + DeletionVectorStorageType::PersistedRelative, + // 20 bytes but `_` is outside the z85 alphabet, so z85::decode fails. + "____________________", + Some(1), + 4, + 1, + "z85" + )] + #[case::persisted_relative_non_ascii( + DeletionVectorStorageType::PersistedRelative, + // 22 bytes (`euro` U+20AC is 3 bytes) with `len - 20 = 2` landing inside the + // multi-byte codepoint; byte-slicing keeps this from panicking and z85 rejects + // the non-ASCII payload. + "a\u{20ac}aaaaaaaaaaaaaaaaaa", + Some(1), + 4, + 1, + "z85" + )] + #[case::persisted_absolute_non_url( + DeletionVectorStorageType::PersistedAbsolute, + "not a url", + Some(1), + 4, + 1, + "URL" + )] + #[case::negative_size( + DeletionVectorStorageType::Inline, "ABC", None, -1, 0, "size_in_bytes" + )] + #[case::negative_cardinality( + DeletionVectorStorageType::Inline, "ABC", None, 4, -1, "cardinality" + )] + #[case::negative_offset( + DeletionVectorStorageType::PersistedAbsolute, "file:///tmp/dv.bin", Some(-1), 4, 1, "offset" + )] + fn dv_descriptor_try_new_rejects_invalid( + #[case] storage_type: DeletionVectorStorageType, + #[case] path: &str, + #[case] offset: Option, + #[case] size_in_bytes: i32, + #[case] cardinality: i64, + #[case] expected_substr: &str, + ) { + let err = DeletionVectorDescriptor::try_new( + storage_type, + path, + offset, + size_in_bytes, + cardinality, + ) + .expect_err("expected validation error"); + assert!( + err.to_string().contains(expected_substr), + "expected error containing {expected_substr:?}, got: {err}" + ); + } + + #[test] + fn dv_descriptor_try_new_round_trips_fields() { + let descriptor = DeletionVectorDescriptor::try_new( + DeletionVectorStorageType::PersistedAbsolute, + "file:///tmp/dv.bin", + Some(7), + 42, + 9, + ) + .expect("valid descriptor"); + + assert_eq!( + descriptor.storage_type, + DeletionVectorStorageType::PersistedAbsolute + ); + assert_eq!(descriptor.path_or_inline_dv, "file:///tmp/dv.bin"); + assert_eq!(descriptor.offset, Some(7)); + assert_eq!(descriptor.size_in_bytes, 42); + assert_eq!(descriptor.cardinality, 9); + } } diff --git a/kernel/src/actions/deletion_vector_writer.rs b/kernel/src/actions/deletion_vector_writer.rs index 6121adfb81..0ed3c9c7bd 100644 --- a/kernel/src/actions/deletion_vector_writer.rs +++ b/kernel/src/actions/deletion_vector_writer.rs @@ -52,8 +52,9 @@ pub trait DeletionVector: Sized { /// Serialize the deletion vector into bytes. /// /// This serializes the deletion vector in the format expected by the Delta Lake protocol. - /// it may be overridden for more efficient serialization if the implementation already has the data in a suitable format. - /// But generally, only do this if you fully understand the the format requirements. + /// it may be overridden for more efficient serialization if the implementation already has the + /// data in a suitable format. But generally, only do this if you fully understand the the + /// format requirements. fn serialize(self) -> DeltaResult { let treemap: RoaringTreemap = self.into_iter().collect(); let mut serialized = Vec::new(); @@ -85,7 +86,8 @@ pub struct DeletionVectorWriteResult { impl DeletionVectorWriteResult { /// Convert the write result to a deletion vector descriptor. /// - /// As an implementation detail, this method will always use the persisted relative storage type. + /// As an implementation detail, this method will always use the persisted relative storage + /// type. /// /// # Arguments /// @@ -182,7 +184,7 @@ impl DeletionVector for KernelDeletionVector { /// The writer produces deletion vector files in the Delta Lake format: /// - The first byte of the file is a version byte (currently 1) /// - Each DV is prefixed with a 4-byte size (big-endian) of the serialized data -/// - Followed by a 4-byte magic number (0x64485871, little-endian) +/// - Followed by a 4-byte magic number (0x6439d3d1, little-endian) /// - Followed by the serialized 64-bit Roaring Bitmap /// - Followed by a 4-byte CRC32 checksum (big-endian) of the serialized data /// @@ -262,6 +264,10 @@ impl<'a, W: Write> StreamingDeletionVectorWriter<'a, W> { &mut self, deletion_vector: impl DeletionVector, ) -> DeltaResult { + // Serialize first so a failure leaves the writer state and output untouched. + let cardinality = deletion_vector.cardinality(); + let serialized = deletion_vector.serialize()?; + // Write version byte on first write if self.current_offset == 0 { // Write header. @@ -271,10 +277,6 @@ impl<'a, W: Write> StreamingDeletionVectorWriter<'a, W> { self.current_offset = 1; } - let cardinality = deletion_vector.cardinality(); - // Serialize the deletion vector to bytes - let serialized = deletion_vector.serialize()?; - // Calculate sizes // The size field contains the size of data + magic(4) (doesn't include CRC) @@ -367,9 +369,10 @@ impl<'a, W: Write> StreamingDeletionVectorWriter<'a, W> { #[cfg(test)] mod tests { - use super::*; use std::io::Cursor; + use super::*; + #[test] fn test_kernel_deletion_vector_new() { let dv = KernelDeletionVector::new(); @@ -433,11 +436,13 @@ mod tests { #[test] fn test_streaming_writer_empty_dv() { - use crate::Engine; use std::fs::File; + use tempfile::tempdir; use url::Url; + use crate::Engine; + // Create a temporary directory and file let temp_dir = tempdir().unwrap(); let table_url = Url::from_directory_path(temp_dir.path()).unwrap(); @@ -567,11 +572,13 @@ mod tests { #[test] fn test_array_based_deletion_vector() { - use crate::Engine; use std::fs::File; + use tempfile::tempdir; use url::Url; + use crate::Engine; + // Custom DeletionVector implementation that wraps an array of u64 struct ArrayDeletionVector { deleted_rows: Vec, @@ -722,11 +729,13 @@ mod tests { #[test] fn test_multiple_deletion_vectors_roundtrip_with_descriptor() { - use crate::Engine; use std::fs::File; + use tempfile::tempdir; use url::Url; + use crate::Engine; + // Create a temporary directory and file let temp_dir = tempdir().unwrap(); let table_url = Url::from_directory_path(temp_dir.path()).unwrap(); @@ -770,8 +779,8 @@ mod tests { // Now read back each deletion vector using the descriptors for (write_result, expected_indexes) in descriptors.iter().zip(&test_data) { - // Create a new DeletionVectorPath for each DV (they would have different UUIDs normally, - // but for this test we're writing multiple to the same file) + // Create a new DeletionVectorPath for each DV (they would have different UUIDs + // normally, but for this test we're writing multiple to the same file) let descriptor = write_result.clone().to_descriptor(&dv_path); // Read the deletion vector back using the descriptor diff --git a/kernel/src/actions/mod.rs b/kernel/src/actions/mod.rs index 8884931b40..e5f152ccd4 100644 --- a/kernel/src/actions/mod.rs +++ b/kernel/src/actions/mod.rs @@ -2,29 +2,36 @@ //! specification](https://github.com/delta-io/delta/blob/master/PROTOCOL.md) use std::collections::HashMap; -use std::sync::{Arc, LazyLock}; +use std::sync::LazyLock; + +use delta_kernel_derive::{ + internal_api, IntoEngineData, IntoStructData, ToSchema, TryFromStructData, +}; +use serde::{Deserialize, Serialize}; +use tracing::warn; +use url::Url; +use visitors::{MetadataVisitor, ProtocolVisitor}; use self::deletion_vector::DeletionVectorDescriptor; -use crate::expressions::{MapData, Scalar, StructData}; -use crate::schema::{DataType, MapType, SchemaRef, StructField, StructType, ToSchema as _}; +use crate::schema::{ + is_unsupported_delta_type_error, lazy_schema_ref, schema_ref, SchemaRef, StructField, + StructType, ToSchema as _, +}; +#[cfg(feature = "adaptive-metadata-in-dev")] +use crate::schema::{schema, ArrayType}; use crate::table_features::{ - FeatureType, IntoTableFeature, TableFeature, TABLE_FEATURES_MIN_READER_VERSION, - TABLE_FEATURES_MIN_WRITER_VERSION, + FeatureType, TableFeature, LEGACY_READER_FEATURES, MIN_VALID_RW_VERSION, + TABLE_FEATURES_MIN_READER_VERSION, TABLE_FEATURES_MIN_WRITER_VERSION, }; use crate::table_properties::TableProperties; use crate::utils::require; use crate::{ - DeltaResult, Engine, EngineData, Error, EvaluationHandlerExtension as _, FileMeta, + DeltaResult, Engine, EngineData, Error, EvaluationHandlerExtension as _, FileMeta, FileSize, IntoEngineData, RowVisitor as _, }; -use url::Url; -use visitors::{MetadataVisitor, ProtocolVisitor}; - -use delta_kernel_derive::{internal_api, IntoEngineData, ToSchema}; -use serde::{Deserialize, Serialize}; - const KERNEL_VERSION: &str = env!("CARGO_PKG_VERSION"); +const SERDE_JSON_RECURSION_LIMIT_ERROR_PREFIX: &str = "recursion limit exceeded"; const UNKNOWN_OPERATION: &str = "UNKNOWN"; pub mod deletion_vector; @@ -57,73 +64,191 @@ pub(crate) const SIDECAR_NAME: &str = "sidecar"; pub(crate) const CHECKPOINT_METADATA_NAME: &str = "checkpointMetadata"; #[internal_api] pub(crate) const DOMAIN_METADATA_NAME: &str = "domainMetadata"; +#[cfg(feature = "adaptive-metadata-in-dev")] +#[internal_api] +pub(crate) const CHECKPOINT_ACTION_NAME: &str = "checkpoint"; +#[cfg(feature = "adaptive-metadata-in-dev")] +#[internal_api] +pub(crate) const CONTENT_ROOT_NAME: &str = "contentRoot"; pub(crate) const INTERNAL_DOMAIN_PREFIX: &str = "delta."; -static COMMIT_SCHEMA: LazyLock = LazyLock::new(|| { - Arc::new(StructType::new_unchecked([ - StructField::nullable(ADD_NAME, Add::to_schema()), - StructField::nullable(REMOVE_NAME, Remove::to_schema()), - StructField::nullable(METADATA_NAME, Metadata::to_schema()), - StructField::nullable(PROTOCOL_NAME, Protocol::to_schema()), - StructField::nullable(SET_TRANSACTION_NAME, SetTransaction::to_schema()), - StructField::nullable(COMMIT_INFO_NAME, CommitInfo::to_schema()), - StructField::nullable(CDC_NAME, Cdc::to_schema()), - StructField::nullable(DOMAIN_METADATA_NAME, DomainMetadata::to_schema()), - ])) +/// Returns the required leaf used to identify rows containing `action_name`. +/// +/// Returns `None` when `action_name` is unknown or has no required identifying leaf. +pub(crate) fn action_presence_leaf(action_name: &str) -> Option<&'static str> { + match action_name { + ADD_NAME | REMOVE_NAME | CDC_NAME | SIDECAR_NAME => Some("path"), + METADATA_NAME => Some("id"), + PROTOCOL_NAME => Some("minReaderVersion"), + SET_TRANSACTION_NAME => Some("appId"), + DOMAIN_METADATA_NAME => Some("domain"), + CHECKPOINT_METADATA_NAME => Some("version"), + _ => None, + } +} + +// === Sub-fields of an AddFile's `stats` struct === +// See the Delta protocol spec, "Per-file Statistics", and `expected_stats_schema` in +// `scan/data_skipping/stats_schema/mod.rs` for the full semantics. +/// Logical (post-DV) row count, stored as a `long`. +#[internal_api] +pub(crate) const NUM_RECORDS: &str = "numRecords"; +/// Per-column null counts, as a nested struct mirroring the table schema. +#[internal_api] +pub(crate) const NULL_COUNT: &str = "nullCount"; +/// Per-column lower bounds, as a nested struct mirroring the table schema. +#[internal_api] +pub(crate) const MIN_VALUES: &str = "minValues"; +/// Per-column upper bounds, as a nested struct mirroring the table schema. +#[internal_api] +pub(crate) const MAX_VALUES: &str = "maxValues"; +/// Whether the min/max/nullCount stats are tight or wide. Defaults to `true` when absent. +#[internal_api] +pub(crate) const TIGHT_BOUNDS: &str = "tightBounds"; + +/// Struct-encoded per-file statistics column (checkpoints with `writeStatsAsStruct=true`). +#[internal_api] +pub(crate) const STATS_PARSED: &str = "stats_parsed"; + +pub(crate) static ADD_SCHEMA: LazyLock = LazyLock::new(Add::to_schema); + +pub(crate) static ADD_FIELD: LazyLock = + LazyLock::new(|| StructField::nullable(ADD_NAME, ADD_SCHEMA.clone())); +pub(crate) static REMOVE_FIELD: LazyLock = + LazyLock::new(|| StructField::nullable(REMOVE_NAME, Remove::to_schema())); +pub(crate) static METADATA_FIELD: LazyLock = + LazyLock::new(|| StructField::nullable(METADATA_NAME, Metadata::to_schema())); +pub(crate) static PROTOCOL_FIELD: LazyLock = + LazyLock::new(|| StructField::nullable(PROTOCOL_NAME, Protocol::to_schema())); +pub(crate) static SET_TRANSACTION_FIELD: LazyLock = + LazyLock::new(|| StructField::nullable(SET_TRANSACTION_NAME, SetTransaction::to_schema())); +pub(crate) static COMMIT_INFO_FIELD: LazyLock = + LazyLock::new(|| StructField::nullable(COMMIT_INFO_NAME, CommitInfo::to_schema())); +pub(crate) static CDC_FIELD: LazyLock = + LazyLock::new(|| StructField::nullable(CDC_NAME, Cdc::to_schema())); +pub(crate) static DOMAIN_METADATA_FIELD: LazyLock = + LazyLock::new(|| StructField::nullable(DOMAIN_METADATA_NAME, DomainMetadata::to_schema())); +pub(crate) static CHECKPOINT_METADATA_FIELD: LazyLock = LazyLock::new(|| { + StructField::nullable(CHECKPOINT_METADATA_NAME, CheckpointMetadata::to_schema()) +}); +pub(crate) static SIDECAR_FIELD: LazyLock = + LazyLock::new(|| StructField::nullable(SIDECAR_NAME, Sidecar::to_schema())); + +#[cfg(feature = "adaptive-metadata-in-dev")] +pub(crate) static CONTENT_ROOT_FIELD: LazyLock = + LazyLock::new(|| StructField::nullable(CONTENT_ROOT_NAME, ContentRoot::to_schema())); + +/// A `sidecar` element inside a `checkpoint` action array. Unlike the V2-checkpoint [`Sidecar`] +/// (which references spilled file actions), this references spilled user `txn` / `domainMetadata` +/// entries, discriminated by its `type` field (`"txn"` or `"domainMetadata"`). Its shape is a +/// [`Sidecar`] prefixed with that `type` column, so the schema is composed from +/// [`Sidecar::to_schema`] here rather than duplicated: `type` cannot be produced by [`ToSchema`] +/// (which stringifies the Rust identifier, and `r#type` stringifies to `"r#type"`). +#[cfg(feature = "adaptive-metadata-in-dev")] +static CONTENT_SIDECAR_FIELD: LazyLock = LazyLock::new(|| { + StructField::nullable( + SIDECAR_NAME, + schema! { + not_null "type": STRING, + ..(Sidecar::to_schema().into_fields()), + }, + ) }); -static ALL_ACTIONS_SCHEMA: LazyLock = LazyLock::new(|| { - Arc::new(StructType::new_unchecked( - get_commit_schema().fields().cloned().chain([ - StructField::nullable(CHECKPOINT_METADATA_NAME, CheckpointMetadata::to_schema()), - StructField::nullable(SIDECAR_NAME, Sidecar::to_schema()), - ]), - )) +/// The `checkpoint` action serializes as an array whose elements are each one of the metadata +/// actions embedded in an adaptiveMetadata manifest commit. This schema is the union of every +/// element type that may appear in that array (per the adaptiveMetadata RFC, delta-io/delta#6978): +/// `checkpointMetadata`, `contentRoot`, `protocol`, `metaData`, `domainMetadata`, `txn`, `sidecar`. +#[cfg(feature = "adaptive-metadata-in-dev")] +static CHECKPOINT_ACTION_ELEMENT_SCHEMA: LazyLock = lazy_schema_ref! { + (&CHECKPOINT_METADATA_FIELD), + (&CONTENT_ROOT_FIELD), + (&PROTOCOL_FIELD), + (&METADATA_FIELD), + (&DOMAIN_METADATA_FIELD), + (&SET_TRANSACTION_FIELD), + (&CONTENT_SIDECAR_FIELD), +}; + +#[cfg(feature = "adaptive-metadata-in-dev")] +pub(crate) static CHECKPOINT_ACTION_FIELD: LazyLock = LazyLock::new(|| { + StructField::nullable( + CHECKPOINT_ACTION_NAME, + ArrayType::new(CHECKPOINT_ACTION_ELEMENT_SCHEMA.clone(), false), + ) }); +/// The `checkpoint` action field, present only under the `adaptive-metadata-in-dev` feature; +/// otherwise an empty iterator. +fn checkpoint_action_field() -> impl IntoIterator { + #[cfg(feature = "adaptive-metadata-in-dev")] + { + Some(&*CHECKPOINT_ACTION_FIELD) + } + #[cfg(not(feature = "adaptive-metadata-in-dev"))] + { + None::<&'static StructField> + } +} + +static COMMIT_SCHEMA: LazyLock = lazy_schema_ref! { + (&ADD_FIELD), + (&REMOVE_FIELD), + (&METADATA_FIELD), + (&PROTOCOL_FIELD), + (&SET_TRANSACTION_FIELD), + (&COMMIT_INFO_FIELD), + (&CDC_FIELD), + (&DOMAIN_METADATA_FIELD), + ..(checkpoint_action_field()), +}; + +static ALL_ACTIONS_SCHEMA: LazyLock = lazy_schema_ref! { + (&ADD_FIELD), + (&REMOVE_FIELD), + (&METADATA_FIELD), + (&PROTOCOL_FIELD), + (&SET_TRANSACTION_FIELD), + (&COMMIT_INFO_FIELD), + (&CDC_FIELD), + (&DOMAIN_METADATA_FIELD), + ..(checkpoint_action_field()), + (&CHECKPOINT_METADATA_FIELD), + (&SIDECAR_FIELD), +}; + /// Schema for Add actions in the Delta log. /// Wraps the Add action schema in a top-level struct with "add" field name. -static LOG_ADD_SCHEMA: LazyLock = LazyLock::new(|| { - Arc::new(StructType::new_unchecked([StructField::nullable( - ADD_NAME, - Add::to_schema(), - )])) -}); +#[internal_api] +pub(crate) static LOG_ADD_SCHEMA: LazyLock = lazy_schema_ref! { (&ADD_FIELD) }; /// Schema for Remove actions in the Delta log. /// Wraps the Remove action schema in a top-level struct with "remove" field name. -static LOG_REMOVE_SCHEMA: LazyLock = LazyLock::new(|| { - Arc::new(StructType::new_unchecked([StructField::nullable( - REMOVE_NAME, - Remove::to_schema(), - )])) -}); +#[internal_api] +pub(crate) static LOG_REMOVE_SCHEMA: LazyLock = lazy_schema_ref! { (&REMOVE_FIELD) }; + +#[internal_api] +pub(crate) static LOG_METADATA_SCHEMA: LazyLock = lazy_schema_ref! { (&METADATA_FIELD) }; + +#[internal_api] +pub(crate) static LOG_PROTOCOL_SCHEMA: LazyLock = lazy_schema_ref! { (&PROTOCOL_FIELD) }; /// Schema for CommitInfo actions in the Delta log. /// Wraps the CommitInfo schema in a top-level struct with "commitInfo" field name. -static LOG_COMMIT_INFO_SCHEMA: LazyLock = LazyLock::new(|| { - Arc::new(StructType::new_unchecked([StructField::nullable( - COMMIT_INFO_NAME, - CommitInfo::to_schema(), - )])) -}); +#[internal_api] +pub(crate) static LOG_COMMIT_INFO_SCHEMA: LazyLock = + lazy_schema_ref! { (&COMMIT_INFO_FIELD) }; /// Schema for transaction (txn) actions in the Delta log. /// Wraps the SetTransaction schema in a top-level struct with "txn" field name. -static LOG_TXN_SCHEMA: LazyLock = LazyLock::new(|| { - Arc::new(StructType::new_unchecked([StructField::nullable( - SET_TRANSACTION_NAME, - SetTransaction::to_schema(), - )])) -}); +#[internal_api] +pub(crate) static LOG_TXN_SCHEMA: LazyLock = + lazy_schema_ref! { (&SET_TRANSACTION_FIELD) }; -static LOG_DOMAIN_METADATA_SCHEMA: LazyLock = LazyLock::new(|| { - Arc::new(StructType::new_unchecked([StructField::nullable( - DOMAIN_METADATA_NAME, - DomainMetadata::to_schema(), - )])) -}); +#[internal_api] +pub(crate) static LOG_DOMAIN_METADATA_SCHEMA: LazyLock = + lazy_schema_ref! { (&DOMAIN_METADATA_FIELD) }; #[internal_api] /// Gets the schema for all actions that can appear in commits @@ -139,27 +264,6 @@ pub(crate) fn get_all_actions_schema() -> &'static SchemaRef { &ALL_ACTIONS_SCHEMA } -#[internal_api] -pub(crate) fn get_log_add_schema() -> &'static SchemaRef { - &LOG_ADD_SCHEMA -} - -pub(crate) fn get_log_remove_schema() -> &'static SchemaRef { - &LOG_REMOVE_SCHEMA -} - -pub(crate) fn get_log_commit_info_schema() -> &'static SchemaRef { - &LOG_COMMIT_INFO_SCHEMA -} - -pub(crate) fn get_log_txn_schema() -> &'static SchemaRef { - &LOG_TXN_SCHEMA -} - -pub(crate) fn get_log_domain_metadata_schema() -> &'static SchemaRef { - &LOG_DOMAIN_METADATA_SCHEMA -} - /// Returns true if the schema contains file actions (add or remove) /// columns. #[internal_api] @@ -171,14 +275,14 @@ pub(crate) fn schema_contains_file_actions(schema: &SchemaRef) -> bool { /// /// This is useful for JSON conversion, as it allows us to wrap a dynamically maintained add action /// schema in a top-level "add" struct. -pub(crate) fn as_log_add_schema(schema: SchemaRef) -> SchemaRef { - Arc::new(StructType::new_unchecked([StructField::nullable( - ADD_NAME, schema, - )])) +pub(crate) fn as_log_add_schema(add_schema: SchemaRef) -> SchemaRef { + schema_ref! { nullable ADD_NAME: (add_schema) } } // Serde derives are needed for CRC file deserialization (see `crc::reader`). -#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +#[derive( + Debug, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema, IntoStructData, TryFromStructData, +)] #[serde(rename_all = "camelCase")] #[internal_api] pub(crate) struct Format { @@ -197,24 +301,12 @@ impl Default for Format { } } -impl TryFrom for Scalar { - type Error = Error; - - fn try_from(format: Format) -> DeltaResult { - let provider = Scalar::from(format.provider); - let options = MapData::try_new( - MapType::new(DataType::STRING, DataType::STRING, false), - format.options, - ) - .map(Scalar::Map)?; - Ok(Scalar::Struct(StructData::try_new( - Format::to_schema().into_fields().collect(), - vec![provider, options], - )?)) - } -} - // Serde derives are needed for CRC file deserialization (see `crc::reader`). +// +// TODO(#2446): `Metadata` stores the schema only as a JSON string. Callers that already hold +// a parsed `SchemaRef` (e.g. CREATE TABLE) serialize into `schema_string` and then re-parse +// downstream in `TableConfiguration::try_new` via `parse_schema()`. Caching the parsed schema +// on `Metadata` would eliminate the round-trip. #[derive(Debug, Default, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)] #[serde(rename_all = "camelCase")] #[internal_api] @@ -243,9 +335,7 @@ impl Metadata { /// # Errors /// /// Returns an error if there are any metadata columns in the schema. - // TODO: remove allow(dead_code) after we use this API in CREATE TABLE, etc. #[internal_api] - #[allow(dead_code)] pub(crate) fn try_new( name: Option, description: Option, @@ -313,19 +403,50 @@ impl Metadata { } #[internal_api] - #[allow(dead_code)] pub(crate) fn configuration(&self) -> &HashMap { &self.configuration } + #[internal_api] + #[allow(dead_code)] + pub(crate) fn format_provider(&self) -> &str { + &self.format.provider + } + #[internal_api] pub(crate) fn schema_string(&self) -> &String { &self.schema_string } + /// Parses the table schema from its JSON representation. + /// + /// # Errors + /// + /// Returns [`Error::Schema`] when the schema exceeds the supported decoding depth or + /// declares a type the kernel doesn't support, or [`Error::MalformedJson`] for other + /// JSON decoding failures. #[internal_api] pub(crate) fn parse_schema(&self) -> DeltaResult { - Ok(serde_json::from_str(&self.schema_string)?) + // TODO(#1896): Increase the supported nesting depth or use non-recursive schema decoding. + serde_json::from_str(&self.schema_string).map_err(|error| { + // serde_json keeps ErrorCode::RecursionLimitExceeded private, so we use string + // matching. + if error.is_syntax() + && error + .to_string() + .starts_with(SERDE_JSON_RECURSION_LIMIT_ERROR_PREFIX) + { + Error::schema(format!( + "Table schema is too deeply nested: decoding metaData.schemaString exceeded \ + serde_json's recursion limit: {error}" + )) + .with_backtrace() + } else if is_unsupported_delta_type_error(&error) { + Error::schema(error.to_string()).with_backtrace() + } else { + error.into() + } + }) } #[internal_api] @@ -341,6 +462,29 @@ impl Metadata { TableProperties::from(self.configuration.iter()) } + /// Returns a new Metadata with the schema replaced, preserving all other fields. + /// + /// # Errors + /// + /// Returns an error if schema serialization fails. + pub(crate) fn with_schema(self, schema: SchemaRef) -> DeltaResult { + Ok(Self { + schema_string: serde_json::to_string(&schema)?, + ..self + }) + } + + /// Returns a new Metadata with a single configuration entry inserted (or replaced), + /// preserving all other configuration entries and metadata fields. + pub(crate) fn with_configuration_entry( + mut self, + key: impl Into, + value: impl Into, + ) -> Self { + self.configuration.insert(key.into(), value.into()); + self + } + #[cfg(test)] #[allow(clippy::too_many_arguments)] pub(crate) fn new_unchecked( @@ -380,11 +524,11 @@ impl IntoEngineData for Metadata { self.name.into(), self.description.into(), self.format.provider.into(), - self.format.options.try_into()?, + self.format.options.into(), self.schema_string.into(), - self.partition_columns.try_into()?, + self.partition_columns.into(), self.created_time.into(), - self.configuration.try_into()?, + self.configuration.into(), ]; engine.evaluation_handler().create_one(schema, &values) @@ -394,7 +538,10 @@ impl IntoEngineData for Metadata { #[derive( Default, Debug, Clone, PartialEq, Eq, ToSchema, Serialize, Deserialize, IntoEngineData, )] -#[serde(rename_all = "camelCase")] +// Deserialization goes through `ProtocolRaw` so every serde entry point (e.g. CRC files) is +// validated by `try_new`, like the JSON-replay path. Otherwise a CRC file could load a malformed +// feature shape that log replay would reject. +#[serde(rename_all = "camelCase", try_from = "ProtocolRaw")] #[internal_api] // TODO move to another module so that we disallow constructing this struct without using the // try_new function. @@ -415,20 +562,45 @@ pub(crate) struct Protocol { writer_features: Option>, } +/// Raw, unvalidated form of [`Protocol`] that serde reads before validation. Deserialize-only +/// (never serialized): `Protocol`'s `#[serde(try_from)]` converts it via [`Protocol::try_new`], +/// so every deserialization is validated. +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct ProtocolRaw { + min_reader_version: i32, + min_writer_version: i32, + reader_features: Option>, + writer_features: Option>, +} + +impl TryFrom for Protocol { + type Error = Error; + + fn try_from(protocol: ProtocolRaw) -> DeltaResult { + Protocol::try_new( + protocol.min_reader_version, + protocol.min_writer_version, + protocol.reader_features, + protocol.writer_features, + ) + } +} + /// Parse a list of feature identifiers into TableFeatures. Returns `None` for `None` input; /// otherwise infallible (unrecognized names become `TableFeature::Unknown`). fn parse_features( - features: Option>, + features: Option>>, ) -> Option> { - let features = features?.into_iter().map(|f| f.into_table_feature()); + let features = features?.into_iter().map(Into::into); Some(features.collect()) } impl Protocol { /// Try to create a new modern Protocol instance with the given table feature lists pub(crate) fn try_new_modern( - reader_features: impl IntoIterator, - writer_features: impl IntoIterator, + reader_features: impl IntoIterator>, + writer_features: impl IntoIterator>, ) -> DeltaResult { Self::try_new( TABLE_FEATURES_MIN_READER_VERSION, @@ -456,13 +628,27 @@ impl Protocol { pub(crate) fn try_new( min_reader_version: i32, min_writer_version: i32, - reader_features: Option>, - writer_features: Option>, + reader_features: Option>>, + writer_features: Option>>, ) -> DeltaResult { + require!( + min_reader_version >= MIN_VALID_RW_VERSION, + Error::InvalidProtocol(format!( + "min_reader_version must be >= {MIN_VALID_RW_VERSION}, got {min_reader_version}" + )) + ); + require!( + min_writer_version >= MIN_VALID_RW_VERSION, + Error::InvalidProtocol(format!( + "min_writer_version must be >= {MIN_VALID_RW_VERSION}, got {min_writer_version}" + )) + ); + let reader_features = parse_features(reader_features); let writer_features = parse_features(writer_features); - // The protocol states that Reader features may be present if and only if the min_reader_version is 3 + // The protocol states that Reader features may be present if and only if the + // min_reader_version is 3 if min_reader_version == TABLE_FEATURES_MIN_READER_VERSION { require!( reader_features.is_some(), @@ -479,7 +665,8 @@ impl Protocol { ); } - // The protocol states that Writer features may be present if and only if the min_writer_version is 7 + // The protocol states that Writer features may be present if and only if the + // min_writer_version is 7 if min_writer_version == TABLE_FEATURES_MIN_WRITER_VERSION { require!( writer_features.is_some(), @@ -500,57 +687,90 @@ impl Protocol { match (&reader_features, &writer_features) { (Some(reader_features), Some(writer_features)) => { // Check all reader features are ReaderWriter and present in writer features. - // Unknown features are treated as potentially ReaderWriter for forward compatibility. - let check_r = reader_features.iter().all(|feature| { - matches!( + // Unknown features are treated as potentially ReaderWriter for forward + // compatibility. + if let Some(offending) = reader_features.iter().find(|feature| { + !matches!( feature.feature_type(), FeatureType::ReaderWriter | FeatureType::Unknown - ) && writer_features.contains(feature) - }); - require!( - check_r, - Error::invalid_protocol( - "Reader features must contain only ReaderWriter features that are also listed in writer features" - ) - ); - - // Check all writer features that are ReaderWriter must also be in reader features - // Unknown features are treated as potentially Writer-only for forward compatibility. - let check_w = writer_features - .iter() - .all(|feature| match feature.feature_type() { - FeatureType::WriterOnly | FeatureType::Unknown => true, - FeatureType::ReaderWriter => reader_features.contains(feature), - }); - require!( - check_w, - Error::invalid_protocol( - "Writer features must be Writer-only or also listed in reader features" - ) - ); + ) || !writer_features.contains(*feature) + }) { + return Err(Error::invalid_protocol(format!( + "Reader features must contain only ReaderWriter features that are also \ + listed in writer features, but {offending:?} is not \ + (readerFeatures={reader_features:?}, writerFeatures={writer_features:?}, \ + minReaderVersion={min_reader_version}, minWriterVersion={min_writer_version})" + ))); + } + + // Every ReaderWriter feature in writerFeatures must also appear in readerFeatures. + // Unknown features are treated as potentially Writer-only for forward + // compatibility. + // + // Accept the legacy writer-list-only shape for delta-spark compatibility: a + // past delta-spark bug produced (3, 7) tables with ColumnMapping in writerFeatures + // only and an empty readerFeatures. Such tables still read correctly because the + // mode comes from writerFeatures, and rejecting them would break existing + // production tables. See #3110 to tighten this once such tables are migrated. + // + // Validate the whole writer list before warning: a non-legacy orphan rejects the + // protocol outright, so we must not emit an acceptance warning for a legacy orphan + // seen earlier in the list only to fail on a later one. + let mut legacy_orphans = Vec::new(); + for feature in writer_features.iter() { + let orphaned_reader_writer_feature = feature.feature_type() + == FeatureType::ReaderWriter + && !reader_features.contains(feature); + if !orphaned_reader_writer_feature { + continue; + } + if LEGACY_READER_FEATURES.contains(feature) { + legacy_orphans.push(feature); + } else { + return Err(Error::invalid_protocol(format!( + "Writer features must be Writer-only or also listed in reader features, \ + but ReaderWriter feature {feature:?} is listed in writerFeatures and \ + missing from readerFeatures \ + (readerFeatures={reader_features:?}, \ + writerFeatures={writer_features:?}, \ + minReaderVersion={min_reader_version}, \ + minWriterVersion={min_writer_version})" + ))); + } + } + // Reached only once the whole writer list is known valid. + for feature in legacy_orphans { + warn!( + "ReaderWriter feature {feature:?} is listed in writerFeatures but \ + missing from readerFeatures at minReaderVersion={min_reader_version}; \ + treating it as reader-enabled (malformed protocol)" + ); + } Ok(()) } (None, None) => Ok(()), (None, Some(writer_features)) => { // Special case: reader version 2 implies ColumnMapping support. - // All other ReaderWriter features require explicit reader_features list (reader version 3). - // Unknown features are treated as potentially Writer-only for forward compatibility. - let is_valid = writer_features.iter().all(|feature| { + // All other ReaderWriter features require explicit reader_features list (reader + // version 3). Unknown features are treated as potentially + // Writer-only for forward compatibility. + if let Some(offending) = writer_features.iter().find(|feature| { match feature.feature_type() { - FeatureType::WriterOnly | FeatureType::Unknown => true, + FeatureType::WriterOnly | FeatureType::Unknown => false, FeatureType::ReaderWriter => { // ColumnMapping is allowed when reader version is 2 (implied support) - min_reader_version == 2 && feature == &TableFeature::ColumnMapping + !(min_reader_version == 2 && *feature == &TableFeature::ColumnMapping) } } - }); - - require!( - is_valid, - Error::invalid_protocol( - "Writer features must be Writer-only or also listed in reader features" - ) - ); + }) { + return Err(Error::invalid_protocol(format!( + "Writer features must be Writer-only or also listed in reader features, \ + but ReaderWriter feature {offending:?} is listed in writerFeatures with \ + no reader features present \ + (writerFeatures={writer_features:?}, minReaderVersion={min_reader_version}, \ + minWriterVersion={min_writer_version})" + ))); + } Ok(()) } (Some(_), None) => Err(Error::invalid_protocol( @@ -688,8 +908,8 @@ pub(crate) struct Add { /// [RFC 2396 URI Generic Syntax]: https://www.ietf.org/rfc/rfc2396.txt pub(crate) path: String, - /// A map from partition column to value for this logical file. This map can contain null in the - /// values meaning a partition is null. We drop those values from this map, due to the + /// A map from partition column to value for this logical file. This map can contain null in + /// the values meaning a partition is null. We drop those values from this map, due to the /// `allow_null_container_values` annotation allowing them and because [`materialize`] drops /// null values. This means an engine can assume that if a partition is found in /// [`Metadata::partition_columns`] but not in this map, its value is null. @@ -708,7 +928,8 @@ pub(crate) struct Add { /// in the added file must be contained in one or more remove actions in the same version. pub(crate) data_change: bool, - /// Contains [statistics] (e.g., count, min/max values for columns) about the data in this logical file encoded as a JSON string. + /// Contains [statistics] (e.g., count, min/max values for columns) about the data in this + /// logical file encoded as a JSON string. /// /// [statistics]: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Per-file-Statistics #[cfg_attr(test, serde(skip_serializing_if = "Option::is_none"))] @@ -773,7 +994,14 @@ pub(crate) struct Remove { #[cfg_attr(test, serde(skip_serializing_if = "Option::is_none"))] pub(crate) extended_file_metadata: Option, - /// A map from partition column to value for this logical file. + /// A map from partition column to value for this logical file. This map can contain null in + /// the values meaning a partition is null. We drop those values from this map, due to the + /// `allow_null_container_values` annotation allowing them and because [`materialize`] drops + /// null values. This means an engine can assume that if a partition is found in + /// [`Metadata::partition_columns`] but not in this map, its value is null. + /// + /// [`materialize`]: crate::engine_data::EngineMap::materialize + #[allow_null_container_values] #[cfg_attr(test, serde(skip_serializing_if = "Option::is_none"))] pub(crate) partition_values: Option>, @@ -781,13 +1009,15 @@ pub(crate) struct Remove { #[cfg_attr(test, serde(skip_serializing_if = "Option::is_none"))] pub(crate) size: Option, - /// Contains [statistics] (e.g., count, min/max values for columns) about the data in this logical file encoded as a JSON string. + /// Contains [statistics] (e.g., count, min/max values for columns) about the data in this + /// logical file encoded as a JSON string. /// /// [statistics]: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#Per-file-Statistics #[cfg_attr(test, serde(skip_serializing_if = "Option::is_none"))] pub stats: Option, - /// Map containing metadata about this logical file. + /// Map containing metadata about this logical file. Values can be null. + #[allow_null_container_values] #[cfg_attr(test, serde(skip_serializing_if = "Option::is_none"))] pub(crate) tags: Option>, @@ -817,8 +1047,8 @@ pub(crate) struct Cdc { /// [RFC 2396 URI Generic Syntax]: https://www.ietf.org/rfc/rfc2396.txt pub path: String, - /// A map from partition column to value for this logical file. This map can contain null in the - /// values meaning a partition is null. We drop those values from this map, due to the + /// A map from partition column to value for this logical file. This map can contain null in + /// the values meaning a partition is null. We drop those values from this map, due to the /// `allow_null_container_values` annotation allowing them and because [`materialize`] drops /// null values. This means an engine can assume that if a partition is found in /// [`Metadata::partition_columns`] but not in this map, its value is null. @@ -837,7 +1067,8 @@ pub(crate) struct Cdc { /// data of the table pub data_change: bool, - /// Map containing metadata about this logical file. + /// Map containing metadata about this logical file. Values can be null. + #[allow_null_container_values] pub tags: Option>, } @@ -863,13 +1094,202 @@ impl SetTransaction { last_updated, } } + + /// Whether this transaction is expired: `last_updated <= expiration_timestamp` with both + /// present. A `None` `last_updated` (no timestamp recorded) or a `None` `expiration_timestamp` + /// (no retention duration configured) never expires. + pub(crate) fn is_expired(&self, expiration_timestamp: Option) -> bool { + matches!( + (expiration_timestamp, self.last_updated), + (Some(exp_ts), Some(lu)) if lu <= exp_ts + ) + } + + /// This transaction's `version`, unless it is expired under `expiration_timestamp`. + pub(crate) fn non_expired_version(&self, expiration_timestamp: Option) -> Option { + (!self.is_expired(expiration_timestamp)).then_some(self.version) + } +} + +/// Reference to a root of an adaptive metadata tree. +/// +/// Contains the path, size, and version of the root manifest file. +#[cfg(feature = "adaptive-metadata-in-dev")] +#[derive(Debug, Clone, PartialEq, Eq, ToSchema)] +#[internal_api] +#[cfg_attr( + test, + derive(Serialize, Deserialize, Default), + serde(rename_all = "camelCase") +)] +pub(crate) struct ContentRoot { + /// Path to the root manifest file. It is absolute if it begins with an [RFC 3986] URI scheme + /// (e.g. `s3://bucket/...`); otherwise it is relative and resolved against the table root by + /// concatenation with a `/` separator, matching the [Iceberg V4 relative paths specification]. + /// Unlike [`Add`]/[`Remove`] paths, this is not RFC 2396 percent-encoded. + /// + /// [RFC 3986]: https://datatracker.ietf.org/doc/html/rfc3986#section-3.1 + /// [Iceberg V4 relative paths specification]: https://iceberg.apache.org/spec/#paths-in-metadata + pub(crate) path: String, + /// Size of the root manifest file in bytes. Not exposed directly -- use + /// [`ContentRoot::to_filemeta`] to get a validated [`FileMeta`]. + size_in_bytes: i64, + /// The table version the root manifest reflects. Per the adaptiveMetadata RFC this is + /// `<= checkpointMetadata.version`: equal in a manifest commit, and strictly less in a + /// standalone checkpoint (where inline file actions cover the gap up to the checkpoint + /// version). Distinct from [`CheckpointAction::version`], which is + /// `checkpointMetadata.version`. + version: i64, +} + +/// The checkpoint action embeds metadata tree state in a Delta log entry. +/// +/// When a manifest commit occurs, the Delta log entry contains a `checkpoint` action that +/// references a root manifest file. The `version` field indicates the table version up to +/// which the checkpoint is complete. For manifest commits, the checkpoint action also contains +/// the table protocol and metadata, making the commit self-contained with respect to P+M. +/// +/// +/// [adaptiveMetadata RFC]: https://github.com/delta-io/delta/pull/6978 +/// +/// Example manifest-commit JSON: +/// ```json +/// { "checkpoint": [ +/// { "checkpointMetadata": { "version": 42 } }, +/// { "contentRoot": { "path": "...", "sizeInBytes": 1024, "version": 42 } }, +/// { "protocol": { ... } }, +/// { "metaData": { ... } }, +/// { "txn": { ... } }, +/// { "domainMetadata": { ... } }, +/// { "sidecar": { "type": "txn", "path": "...", "sizeInBytes": 1024, "modificationTime": 0 } } +/// ] +/// } +/// ``` +#[cfg(feature = "adaptive-metadata-in-dev")] +#[derive(Debug, Clone, PartialEq, Eq)] +#[internal_api] +pub(crate) struct CheckpointAction { + /// The table version up to which the checkpoint is complete, sourced from the wire + /// `checkpointMetadata.version`. May be less than or equal to the commit version containing + /// this checkpoint action, and is `>= content_root.version` (see [`ContentRoot::version`]). + pub(crate) version: i64, + /// Reference to the root manifest file. + pub(crate) content_root: ContentRoot, + /// The table protocol at the checkpoint version. + pub(crate) protocol: Protocol, + /// The table metadata at the checkpoint version. + pub(crate) metadata: Metadata, + /// Inline `txn` ([`SetTransaction`]) entries carried in the checkpoint array. + pub(crate) transactions: Vec, + /// Inline `domainMetadata` ([`DomainMetadata`]) entries carried in the checkpoint array. + pub(crate) domain_metadata: Vec, + /// `sidecar` entries of type `txn`, referencing spilled [`SetTransaction`] actions. + pub(crate) txn_sidecars: Vec, + /// `sidecar` entries of type `domainMetadata`, referencing spilled [`DomainMetadata`] actions. + pub(crate) domain_metadata_sidecars: Vec, +} + +/// Returns whether `location` begins with a URI scheme, per [RFC 3986 section 3.1]: +/// `scheme = ALPHA *( ALPHA / DIGIT / "+" / "-" / "." )`, terminated by `:`. +/// +/// [RFC 3986 section 3.1]: https://datatracker.ietf.org/doc/html/rfc3986#section-3.1 +#[cfg(feature = "adaptive-metadata-in-dev")] +fn has_scheme(location: &str) -> bool { + for (position, ch) in location.char_indices() { + if ch == ':' { + return position > 0; + } + if !is_scheme_char(ch, position) { + return false; + } + } + false +} + +/// Returns whether `ch` is allowed at `position` in a URI scheme, per [RFC 3986 section 3.1]: +/// the first character must be `ALPHA`; subsequent characters may also be `DIGIT`, `+`, `-`, or +/// `.`. Schemes are restricted to US-ASCII, so non-ASCII letters are rejected. +/// +/// [RFC 3986 section 3.1]: https://datatracker.ietf.org/doc/html/rfc3986#section-3.1 +#[cfg(feature = "adaptive-metadata-in-dev")] +fn is_scheme_char(ch: char, position: usize) -> bool { + if ch.is_ascii_alphabetic() { + return true; + } + position > 0 && (ch.is_ascii_digit() || ch == '+' || ch == '-' || ch == '.') +} + +#[cfg(feature = "adaptive-metadata-in-dev")] +impl ContentRoot { + /// Convert this root manifest reference into a [`FileMeta`] for engine I/O. + /// + /// A `path` with a URI scheme is absolute and used as-is; otherwise it is resolved relative to + /// `table_root` by concatenation with a single `/` separator, matching Iceberg V4's + /// [relative paths specification]. + /// + /// Returns an error if the resolved location fails to parse as a [`Url`], or if the size does + /// not fit a [`crate::FileSize`]. + /// + /// [relative paths specification]: https://iceberg.apache.org/spec/#paths-in-metadata + #[internal_api] + pub(crate) fn to_filemeta(&self, table_root: &Url) -> DeltaResult { + let path = &self.path; + let location = if has_scheme(path) { + // A URI scheme means the path is absolute and used as-is. + Url::parse(path).map_err(|e| { + Error::generic(format!( + "Failed to parse absolute checkpoint contentRoot path {path:?}: {e}" + )) + })? + } else { + // Otherwise the path is relative and concatenated onto `table_root` with a single `/`. + let mut base = table_root.as_str().to_string(); + if !base.ends_with('/') { + base.push('/'); + } + Url::parse(&format!("{base}{path}")).map_err(|e| { + Error::generic(format!( + "Failed to resolve checkpoint contentRoot path {path:?} against table \ + root {base}: {e}" + )) + })? + }; + Ok(FileMeta { + location, + last_modified: i64::MAX, + size: to_file_size(self.size_in_bytes, "checkpoint contentRoot")?, + }) + } +} + +#[cfg(feature = "adaptive-metadata-in-dev")] +impl CheckpointAction { + /// Path to the root manifest file (delegates to the nested [`ContentRoot`]). + #[internal_api] + pub(crate) fn path(&self) -> &str { + &self.content_root.path + } + + /// Get the checkpoint version. + #[internal_api] + pub(crate) fn version(&self) -> i64 { + self.version + } + + /// Convert the referenced root manifest into a [`FileMeta`] for engine I/O (delegates to + /// [`ContentRoot::to_filemeta`]). + #[internal_api] + pub(crate) fn root_filemeta(&self, table_root: &Url) -> DeltaResult { + self.content_root.to_filemeta(table_root) + } } /// The sidecar action references a sidecar file which provides some of the checkpoint's /// file actions. This action is only allowed in checkpoints following the V2 spec. /// /// [More info]: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#sidecar-file-information -#[derive(ToSchema, Debug, PartialEq)] +#[derive(ToSchema, Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] #[internal_api] pub(crate) struct Sidecar { /// A path to a sidecar file that can be either: @@ -887,10 +1307,21 @@ pub(crate) struct Sidecar { /// The time this logical file was created, as milliseconds since the epoch. pub modification_time: i64, - /// A map containing any additional metadata about the logicial file. + /// A map containing any additional metadata about the logical file. Values can be null. + #[allow_null_container_values] pub tags: Option>, } +/// Convert an `i64` byte count from a log action into a [`FileSize`], erroring with `context` (a +/// short action name, e.g. `"sidecar"`) and the offending value when it is negative. +fn to_file_size(bytes: i64, context: &str) -> DeltaResult { + bytes.try_into().map_err(|_| { + Error::generic(format!( + "Failed to convert {context} size {bytes} to FileSize" + )) + }) +} + impl Sidecar { /// Convert a Sidecar record to a FileMeta. /// @@ -900,20 +1331,17 @@ impl Sidecar { Ok(FileMeta { location: log_root.join("_sidecars/")?.join(&self.path)?, last_modified: self.modification_time, - size: self.size_in_bytes.try_into().map_err(|_| { - Error::generic(format!( - "Failed to convert sidecar size {} to usize", - self.size_in_bytes - )) - })?, + size: to_file_size(self.size_in_bytes, "sidecar")?, }) } } -/// The CheckpointMetadata action describes details about a checkpoint following the V2 specification. +/// The CheckpointMetadata action describes details about a checkpoint following the V2 +/// specification. /// /// [More info]: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#checkpoint-metadata -#[derive(Debug, Clone, PartialEq, Eq, ToSchema)] +#[derive(Debug, Clone, PartialEq, Eq, ToSchema, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] #[internal_api] pub(crate) struct CheckpointMetadata { /// The version of the V2 spec checkpoint. @@ -924,7 +1352,8 @@ pub(crate) struct CheckpointMetadata { /// See issue #786 for tracking progress. pub(crate) version: i64, - /// Map containing any additional metadata about the V2 spec checkpoint. + /// Map containing any additional metadata about the V2 spec checkpoint. Values can be null. + #[allow_null_container_values] pub(crate) tags: Option>, } @@ -988,21 +1417,42 @@ impl DomainMetadata { #[cfg(test)] mod tests { + use std::sync::Arc; + + use rstest::rstest; + use serde_json::json; + use super::*; + use crate::arrow::array::{ + Array, BooleanArray, Int32Array, Int64Array, ListArray, ListBuilder, MapBuilder, + MapFieldNames, RecordBatch, StringArray, StringBuilder, StructArray, + }; + use crate::arrow::datatypes::{DataType as ArrowDataType, Field, Schema}; + use crate::arrow::json::ReaderBuilder; + use crate::engine::arrow_data::EngineDataArrowExt as _; + use crate::engine::arrow_expression::ArrowEvaluationHandler; + use crate::expressions::Scalar; + use crate::schema::{schema_ref, ArrayType, DataType, MapType, StructField}; + use crate::unit_test_utils::assert_result_error_with_message; use crate::{ - arrow::{ - array::{ - Array, BooleanArray, Int32Array, Int64Array, ListArray, ListBuilder, MapBuilder, - MapFieldNames, RecordBatch, StringArray, StringBuilder, StructArray, - }, - datatypes::{DataType as ArrowDataType, Field, Schema}, - json::ReaderBuilder, - }, - engine::{arrow_data::EngineDataArrowExt as _, arrow_expression::ArrowEvaluationHandler}, - schema::{ArrayType, DataType, MapType, StructField}, Engine, EvaluationHandler, IntoEngineData, JsonHandler, ParquetHandler, StorageHandler, }; - use serde_json::json; + + #[rstest] + #[case::add(ADD_NAME, Some("path"))] + #[case::remove(REMOVE_NAME, Some("path"))] + #[case::metadata(METADATA_NAME, Some("id"))] + #[case::protocol(PROTOCOL_NAME, Some("minReaderVersion"))] + #[case::transaction(SET_TRANSACTION_NAME, Some("appId"))] + #[case::cdc(CDC_NAME, Some("path"))] + #[case::domain_metadata(DOMAIN_METADATA_NAME, Some("domain"))] + #[case::checkpoint_metadata(CHECKPOINT_METADATA_NAME, Some("version"))] + #[case::sidecar(SIDECAR_NAME, Some("path"))] + #[case::witnessless_action(COMMIT_INFO_NAME, None)] + #[case::unknown_action("futureAction", None)] + fn test_action_presence_leaf(#[case] action_name: &str, #[case] expected_leaf: Option<&str>) { + assert_eq!(action_presence_leaf(action_name), expected_leaf); + } // duplicated struct ExprEngine(Arc); @@ -1050,6 +1500,26 @@ mod tests { )) } + #[rstest] + #[case::no_expiration_configured(None, Some(1000), false)] + #[case::null_last_updated_never_expires(Some(5000), None, false)] + #[case::both_none(None, None, false)] + #[case::last_updated_before_expiration(Some(2000), Some(1000), true)] + #[case::last_updated_at_expiration(Some(1000), Some(1000), true)] + #[case::last_updated_after_expiration(Some(2000), Some(3000), false)] + fn test_set_transaction_expiration( + #[case] expiration_timestamp: Option, + #[case] last_updated: Option, + #[case] expired: bool, + ) { + let txn = SetTransaction::new("app".to_string(), 7, last_updated); + assert_eq!(txn.is_expired(expiration_timestamp), expired); + assert_eq!( + txn.non_expired_version(expiration_timestamp), + (!expired).then_some(7) + ); + } + #[test] fn test_metadata_schema() { let schema = get_commit_schema() @@ -1084,6 +1554,105 @@ mod tests { assert_eq!(schema, expected); } + #[rstest] + #[case::supported(41, false)] + #[case::exceeded(42, true)] + fn parse_schema_nesting_boundary(#[case] depth: usize, #[case] exceeds_limit: bool) { + let metadata = Metadata { + schema_string: serde_json::to_string(&nested_schema(depth)).unwrap(), + ..Default::default() + }; + + let result = metadata.parse_schema(); + if exceeds_limit { + assert_result_error_with_message( + result.as_ref(), + concat!( + "Schema error: Table schema is too deeply nested: decoding ", + "metaData.schemaString exceeded serde_json's ", + "recursion limit: recursion limit exceeded" + ), + ); + let error = match result.unwrap_err() { + Error::Backtraced { source, .. } => *source, + error => error, + }; + assert!(matches!(error, Error::Schema(_))); + } else { + result.unwrap(); + } + } + + #[rstest] + // Syntax error -> MalformedJson. + #[case::malformed_syntax("{", "MalformedJson")] + // Data error lacking the unsupported-type prefix (invalid decimal) -> MalformedJson, NOT + // Schema: the reclassification must not fire for every is_data error. + #[case::malformed_bad_decimal( + r#"{"type":"struct","fields":[{"name":"t","type":"decimal(nope)","nullable":true,"metadata":{}}]}"#, + "MalformedJson" + )] + // Regression guard: a well-formed schema whose wrong-typed field value echoes the prefix must + // stay MalformedJson. `nullable` is a bool, so a string value is an is_data error whose message + // *contains* the prefix; matching by `starts_with` keeps it out of the Schema arm. + #[case::malformed_value_echoes_prefix( + r#"{"type":"struct","fields":[{"name":"t","type":"string","nullable":"Unsupported Delta table type","metadata":{}}]}"#, + "MalformedJson" + )] + // Unsupported primitive types -> Schema. + #[case::unsupported_time( + r#"{"type":"struct","fields":[{"name":"t","type":"time(6)","nullable":true,"metadata":{}}]}"#, + "Schema" + )] + #[case::unsupported_interval( + r#"{"type":"struct","fields":[{"name":"t","type":"interval week","nullable":true,"metadata":{}}]}"#, + "Schema" + )] + // Unsupported primitive nested inside a struct field -> Schema (reclassification is + // position-agnostic, not limited to top-level columns). + #[case::unsupported_nested( + r#"{"type":"struct","fields":[{"name":"t","type":{"type":"struct","fields":[{"name":"inner","type":"time(6)","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}}]}"#, + "Schema" + )] + // Unknown complex type -> Schema. + #[case::unsupported_complex( + r#"{"type":"struct","fields":[{"name":"t","type":{"type":"matrix"},"nullable":true,"metadata":{}}]}"#, + "Schema" + )] + fn parse_schema_error_classification( + #[case] schema_string: &str, + #[case] expected_error: &str, + ) { + let metadata = Metadata { + schema_string: schema_string.to_string(), + ..Default::default() + }; + // Error conversion captures a backtrace only when enabled, so normalize both forms before + // checking the underlying error. + let error = match metadata.parse_schema().unwrap_err() { + Error::Backtraced { source, .. } => *source, + error => error, + }; + match expected_error { + "MalformedJson" => { + assert!(matches!(error, Error::MalformedJson(_)), "got: {error:?}") + } + "Schema" => { + assert!(matches!(error, Error::Schema(_)), "got: {error:?}") + } + other => panic!("unknown expected_error discriminant: {other}"), + } + } + + fn nested_schema(depth: usize) -> StructType { + (0..depth).fold( + StructType::new_unchecked([StructField::nullable("leaf", DataType::INTEGER)]), + |schema, depth| { + StructType::new_unchecked([StructField::nullable(format!("level_{depth}"), schema)]) + }, + ) + } + #[test] fn test_add_schema() { let schema = get_commit_schema() @@ -1118,14 +1687,14 @@ mod tests { fn tags_field() -> StructField { StructField::nullable( "tags", - MapType::new(DataType::STRING, DataType::STRING, false), + MapType::new(DataType::STRING, DataType::STRING, true), ) } fn partition_values_field() -> StructField { StructField::nullable( "partitionValues", - MapType::new(DataType::STRING, DataType::STRING, false), + MapType::new(DataType::STRING, DataType::STRING, true), ) } @@ -1313,6 +1882,30 @@ mod tests { } } + #[rstest] + #[case(0, 1)] + #[case(1, 0)] + #[case(-1, 2)] + #[case(1, -1)] + fn reject_protocol_version_below_minimum(#[case] rv: i32, #[case] wv: i32) { + let expected = if rv < 1 { + format!("Invalid protocol action in the delta log: min_reader_version must be >= 1, got {rv}") + } else { + format!("Invalid protocol action in the delta log: min_writer_version must be >= 1, got {wv}") + }; + assert_result_error_with_message( + Protocol::try_new(rv, wv, TableFeature::NO_LIST, TableFeature::NO_LIST), + &expected, + ); + } + + #[test] + fn accept_min_versions() { + let p = Protocol::try_new_legacy(1, 1).unwrap(); + assert_eq!(p.min_reader_version(), 1); + assert_eq!(p.min_writer_version(), 1); + } + #[test] fn test_validate_table_features_invalid() { // (reader_feature, writer_feature) @@ -1352,12 +1945,14 @@ mod tests { for (reader_features, writer_features, error_msg) in invalid_features { let res = Protocol::try_new_modern(reader_features, writer_features); + // The error message is enriched with the offending feature and the parsed + // feature lists, so match on a prefix rather than the whole string. assert!( matches!( &res, - Err(Error::InvalidProtocol(error)) if error.to_string().eq(error_msg) + Err(Error::InvalidProtocol(error)) if error.to_string().contains(error_msg) ), - "Expected:\t{error_msg}\nBut got:{res:?}\n" + "Expected message containing:\t{error_msg}\nBut got:{res:?}\n" ); } } @@ -1586,10 +2181,7 @@ mod tests { #[test] fn test_metadata_try_new() { - let schema = Arc::new(StructType::new_unchecked([StructField::not_null( - "id", - DataType::INTEGER, - )])); + let schema = schema_ref! { not_null "id": INTEGER }; let config = HashMap::from([("key1".to_string(), "value1".to_string())]); let metadata = Metadata::try_new( @@ -1614,10 +2206,7 @@ mod tests { #[test] fn test_metadata_try_new_default() { - let schema = Arc::new(StructType::new_unchecked([StructField::not_null( - "id", - DataType::INTEGER, - )])); + let schema = schema_ref! { not_null "id": INTEGER }; let metadata = Metadata::try_new(None, None, schema, vec![], 0, HashMap::new()).unwrap(); assert!(!metadata.id.is_empty()); @@ -1627,42 +2216,43 @@ mod tests { #[test] fn test_metadata_unique_ids() { - let schema = Arc::new(StructType::new_unchecked([StructField::not_null( - "id", - DataType::INTEGER, - )])); + let schema = schema_ref! { not_null "id": INTEGER }; let m1 = Metadata::try_new(None, None, schema.clone(), vec![], 0, HashMap::new()).unwrap(); let m2 = Metadata::try_new(None, None, schema, vec![], 0, HashMap::new()).unwrap(); assert_ne!(m1.id, m2.id); } - #[test] - fn test_format_try_from_scalar() { - let options = HashMap::from([ - ("path".to_string(), "/delta/table".to_string()), - ("compressionType".to_string(), "snappy".to_string()), - ]); + #[rstest] + #[case::typical(HashMap::from([ + ("path".to_string(), "/delta/table".to_string()), + ("compressionType".to_string(), "snappy".to_string()), + ]))] + #[case::empty(HashMap::new())] + #[case::special_characters(HashMap::from([ + ("path".to_string(), "/path/with spaces".to_string()), + ("unicode".to_string(), "测试🎉".to_string()), + ("empty".to_string(), String::new()), + ]))] + fn test_format_scalar_round_trip(#[case] options: HashMap) { let format = Format { provider: "parquet".to_string(), - options, - }; - let scalar = Scalar::try_from(format).unwrap(); - - let Scalar::Struct(struct_data) = scalar else { - panic!("Expected struct scalar"); + options: options.clone(), }; - assert_eq!(struct_data.fields()[0].name(), "provider"); - assert_eq!(struct_data.fields()[1].name(), "options"); + let scalar = Scalar::from(format.clone()); - let Scalar::String(provider) = &struct_data.values()[0] else { - panic!("Expected string provider"); + let Scalar::Struct(struct_data) = &scalar else { + panic!("Expected struct scalar, got {scalar}"); }; - assert_eq!(provider, "parquet"); + let field_names: Vec<_> = struct_data.fields().iter().map(|f| f.name()).collect(); + assert_eq!(field_names, ["provider", "options"]); + assert_eq!(struct_data.values()[0], Scalar::from("parquet")); let Scalar::Map(map_data) = &struct_data.values()[1] else { panic!("Expected map options"); }; - assert_eq!(map_data.pairs().len(), 2); + assert_eq!(map_data.pairs().len(), options.len()); + + assert_eq!(Format::try_from(scalar).unwrap(), format); } #[test] @@ -1675,52 +2265,10 @@ mod tests { assert_eq!(format, expected); } - #[test] - fn test_format_empty_options() { - let format = Format { - provider: "parquet".to_string(), - options: HashMap::new(), - }; - let scalar = Scalar::try_from(format).unwrap(); - - let Scalar::Struct(struct_data) = scalar else { - panic!("Expected struct"); - }; - let Scalar::Map(map_data) = &struct_data.values()[1] else { - panic!("Expected map"); - }; - assert!(map_data.pairs().is_empty()); - } - - #[test] - fn test_format_special_characters() { - let options = HashMap::from([ - ("path".to_string(), "/path/with spaces".to_string()), - ("unicode".to_string(), "测试🎉".to_string()), - ("empty".to_string(), "".to_string()), - ]); - let format = Format { - provider: "custom".to_string(), - options, - }; - let scalar = Scalar::try_from(format).unwrap(); - - let Scalar::Struct(struct_data) = scalar else { - panic!("Expected struct"); - }; - let Scalar::Map(map_data) = &struct_data.values()[1] else { - panic!("Expected map"); - }; - assert_eq!(map_data.pairs().len(), 3); - } - #[test] fn test_metadata_into_engine_data() { let engine = ExprEngine::new(); - let schema = Arc::new(StructType::new_unchecked([StructField::not_null( - "id", - DataType::INTEGER, - )])); + let schema = schema_ref! { not_null "id": INTEGER }; let test_metadata = Metadata::try_new( Some("test".to_string()), @@ -1768,10 +2316,7 @@ mod tests { #[test] fn test_metadata_with_log_schema() { let engine = ExprEngine::new(); - let schema = Arc::new(StructType::new_unchecked([StructField::not_null( - "id", - DataType::INTEGER, - )])); + let schema = schema_ref! { not_null "id": INTEGER }; let metadata = Metadata::try_new( Some("table".to_string()), @@ -1786,7 +2331,7 @@ mod tests { let metadata_id = metadata.id.clone(); // test with the full log schema that wraps metadata in a "metaData" field - let commit_schema = get_commit_schema().project(&[METADATA_NAME]).unwrap(); + let commit_schema = LOG_METADATA_SCHEMA.clone(); let actual = metadata .into_engine_data(commit_schema, &engine) .unwrap() @@ -1876,7 +2421,7 @@ mod tests { assert_eq!(record_batch, expected); // test with the full log schema that wraps protocol in a "protocol" field - let commit_schema = get_commit_schema().project(&[PROTOCOL_NAME]).unwrap(); + let commit_schema = LOG_PROTOCOL_SCHEMA.clone(); let engine_data = protocol.into_engine_data(commit_schema, &engine); let schema = Arc::new(Schema::new(vec![Field::new( @@ -2010,7 +2555,7 @@ mod tests { #[test] fn test_schema_contains_file_actions_empty_schema() { - let schema = Arc::new(StructType::new_unchecked([])); + let schema = schema_ref! {}; assert!(!schema_contains_file_actions(&schema)); } @@ -2051,4 +2596,169 @@ mod tests { Some(&Some("1677811178336000".to_string())) ); } + + #[cfg(feature = "adaptive-metadata-in-dev")] + #[test] + fn test_checkpoint_action_schema() { + let schema = get_commit_schema() + .project(&[CHECKPOINT_ACTION_NAME]) + .unwrap(); + + // The `checkpoint` action serializes as an array whose elements are a union of the + // embedded metadata actions. + let checkpoint_field = schema.field(CHECKPOINT_ACTION_NAME).unwrap(); + assert!(checkpoint_field.is_nullable()); + let array = match checkpoint_field.data_type() { + DataType::Array(array) => array, + other => panic!("Expected array, got {other:?}"), + }; + assert!(!array.contains_null()); + let element = match array.element_type() { + DataType::Struct(s) => s, + other => panic!("Expected struct element, got {other:?}"), + }; + let field_names: Vec<&str> = element.fields().map(|f| f.name.as_str()).collect(); + assert_eq!( + field_names, + vec![ + CHECKPOINT_METADATA_NAME, + CONTENT_ROOT_NAME, + PROTOCOL_NAME, + METADATA_NAME, + DOMAIN_METADATA_NAME, + SET_TRANSACTION_NAME, + SIDECAR_NAME, + ] + ); + // `commitInfo` must NOT be a checkpoint-array element; the RFC routes it to the top-level + // Delta log. + assert!(!field_names.contains(&COMMIT_INFO_NAME)); + + // Every element type is an optional (union member) struct. + for field in element.fields() { + assert!(field.is_nullable(), "{} should be nullable", field.name); + assert!(matches!(field.data_type(), DataType::Struct(_))); + } + } + + #[cfg(feature = "adaptive-metadata-in-dev")] + #[test] + fn test_content_sidecar_is_type_prefixed_sidecar() { + // The content-sidecar schema is composed from `Sidecar::to_schema()` with a `type` + // discriminator prepended, so it must equal exactly `type` followed by `Sidecar`'s fields. + let DataType::Struct(content_sidecar) = CONTENT_SIDECAR_FIELD.data_type() else { + panic!("content sidecar should be a struct"); + }; + let expected: Vec = + std::iter::once(StructField::not_null("type", DataType::STRING)) + .chain(Sidecar::to_schema().into_fields()) + .collect(); + let actual: Vec = content_sidecar.fields().cloned().collect(); + assert_eq!(actual, expected); + } + + #[cfg(feature = "adaptive-metadata-in-dev")] + #[rstest] + #[case::relative_path( + "memory:///table/", + "metadata/root.parquet", + 2048, + "memory:///table/metadata/root.parquet", + Ok(2048) + )] + #[case::absolute_path( + "memory:///table/", + "s3://bucket/table/metadata/root.parquet", + 2048, + "s3://bucket/table/metadata/root.parquet", + Ok(2048) + )] + #[case::negative_size( + "memory:///table/", + "metadata/root.parquet", + -1, + "memory:///table/metadata/root.parquet", + Err("Failed to convert checkpoint contentRoot size -1") + )] + #[case::table_root_without_trailing_slash_gets_one( + "memory:///table", + "metadata/root.parquet", + 2048, + "memory:///table/metadata/root.parquet", + Ok(2048) + )] + #[case::single_char_scheme_treated_as_absolute( + "memory:///table/", + "c:/foo/root.parquet", + 2048, + "c:/foo/root.parquet", + Ok(2048) + )] + // A colon inside a relative path segment is not a scheme delimiter (a `/` precedes it), so + // the path stays relative. + #[case::colon_in_relative_segment_stays_relative( + "memory:///table/", + "metadata/snap-123:456.parquet", + 2048, + "memory:///table/metadata/snap-123:456.parquet", + Ok(2048) + )] + // RFC 3986 requires the first scheme char to be ALPHA; a leading digit is not a scheme. + #[case::leading_digit_scheme_treated_as_relative( + "memory:///table/", + "3com/root.parquet", + 2048, + "memory:///table/3com/root.parquet", + Ok(2048) + )] + // A non-ASCII leading letter (Greek alpha, U+03B1) is not a valid scheme char. + #[case::non_ascii_scheme_treated_as_relative( + "memory:///table/", + "\u{03b1}scheme/root.parquet", + 2048, + "memory:///table/%CE%B1scheme/root.parquet", + Ok(2048) + )] + // A multi-char, non-alphanumeric scheme (`git+ssh`) is absolute and used as-is. + #[case::compound_scheme_treated_as_absolute( + "memory:///table/", + "git+ssh://host/repo/root.parquet", + 2048, + "git+ssh://host/repo/root.parquet", + Ok(2048) + )] + fn test_checkpoint_action_root_filemeta( + #[case] table_root: &str, + #[case] path: &str, + #[case] size_in_bytes: i64, + #[case] expected_location: &str, + #[case] expected: Result, + ) { + let table_root = Url::parse(table_root).unwrap(); + let checkpoint_action = CheckpointAction { + version: 1, + content_root: ContentRoot { + path: path.to_string(), + size_in_bytes, + version: 1, + }, + protocol: Protocol::new_unchecked(1, 2, None, None), + metadata: Metadata::default(), + transactions: Vec::new(), + domain_metadata: Vec::new(), + txn_sidecars: Vec::new(), + domain_metadata_sidecars: Vec::new(), + }; + + let result = checkpoint_action.root_filemeta(&table_root); + match expected { + Ok(expected_size) => { + let file_meta = result.unwrap(); + assert_eq!(file_meta.location.as_str(), expected_location); + assert_eq!(file_meta.size, expected_size); + assert_eq!(file_meta.last_modified, i64::MAX); + } + Err(expected_message) => assert_result_error_with_message(result, expected_message), + } + } } diff --git a/kernel/src/actions/set_transaction.rs b/kernel/src/actions/set_transaction.rs index 3845e602da..a94d336119 100644 --- a/kernel/src/actions/set_transaction.rs +++ b/kernel/src/actions/set_transaction.rs @@ -1,12 +1,16 @@ -use crate::actions::get_log_txn_schema; +pub(crate) use crate::actions::visitors::SetTransactionMap; use crate::actions::visitors::SetTransactionVisitor; -use crate::actions::SetTransaction; +use crate::actions::{SetTransaction, LOG_TXN_SCHEMA}; use crate::log_replay::ActionsBatch; use crate::log_segment::LogSegment; -use crate::{DeltaResult, Engine, RowVisitor as _}; - -pub(crate) use crate::actions::visitors::SetTransactionMap; - +use crate::{DeltaResult, Engine, RowVisitor as _, Version}; + +/// Resolves the latest `txn` action per application id via log replay, where the newest action in +/// log order wins. +/// +/// Every method returns the resolved `txn` as-is. Applying retention is the caller's +/// responsibility: use [`SetTransaction::non_expired_version`] on the resolved result. Filtering +/// during replay instead would drop an expired newest `txn` and resolve to an older one. pub(crate) struct SetTransactionScanner {} impl SetTransactionScanner { @@ -18,28 +22,42 @@ impl SetTransactionScanner { log_segment: &LogSegment, application_id: &str, engine: &dyn Engine, - expiration_timestamp: Option, ) -> DeltaResult> { - let mut transactions = scan_application_transactions( - log_segment, - Some(application_id), - engine, - expiration_timestamp, - )?; + let mut transactions = + scan_application_transactions(log_segment, Some(application_id), engine)?; Ok(transactions.remove(application_id)) } - /// Scan the Delta Log to obtain the all of the latest `txn` actions. - /// - /// This performs log replay and populates the `SetTransactionMap` with the latest `txn` action - /// found for each app_id. + /// Scan the Delta Log for the latest `txn` action of every application id. #[allow(unused)] pub(crate) fn get_all( log_segment: &LogSegment, engine: &dyn Engine, - expiration_timestamp: Option, ) -> DeltaResult { - scan_application_transactions(log_segment, None, engine, expiration_timestamp) + scan_application_transactions(log_segment, None, engine) + } + + /// Fetch the latest `txn` action for `application_id`, rooted in an authoritative (`Complete`) + /// but stale CRC's `base_active` map at `base_version`, scanning ONLY the commits in + /// `(base_version, log_segment.end_version]`. + /// + /// The checkpoint and every commit at/below `base_version` are skipped via + /// [`LogSegment::segment_after_version`]. When the tail holds a `txn` for `application_id`, its + /// newest wins; otherwise the result is that id's entry in `base_active`, the value the CRC + /// recorded at `base_version`. + pub(crate) fn get_one_rooted_in_crc( + log_segment: &LogSegment, + application_id: &str, + base_active: &SetTransactionMap, + base_version: Version, + engine: &dyn Engine, + ) -> DeltaResult> { + let tail = Self::get_one( + &log_segment.segment_after_version(base_version), + application_id, + engine, + )?; + Ok(tail.or_else(|| base_active.get(application_id).cloned())) } } @@ -50,10 +68,8 @@ fn scan_application_transactions( log_segment: &LogSegment, application_id: Option<&str>, engine: &dyn Engine, - expiration_timestamp: Option, ) -> DeltaResult { - let mut visitor = - SetTransactionVisitor::new(application_id.map(|s| s.to_owned()), expiration_timestamp); + let mut visitor = SetTransactionVisitor::new(application_id.map(|s| s.to_owned())); // If a specific id is requested then we can terminate log replay early as soon as it was // found. If all ids are requested then we are forced to replay the entire log. for maybe_data in replay_for_app_ids(log_segment, engine)? { @@ -73,22 +89,21 @@ fn replay_for_app_ids( log_segment: &LogSegment, engine: &dyn Engine, ) -> DeltaResult> + Send> { - let txn_schema = get_log_txn_schema(); - log_segment.read_actions(engine, txn_schema.clone()) + log_segment.read_actions(engine, LOG_TXN_SCHEMA.clone()) } #[cfg(test)] mod tests { use std::path::PathBuf; + use itertools::Itertools; + use super::*; + use crate::arrow::array::StringArray; use crate::engine::sync::SyncEngine; - use crate::utils::test_utils::parse_json_batch; + use crate::unit_test_utils::parse_json_batch; use crate::Snapshot; - use crate::arrow::array::StringArray; - use itertools::Itertools; - fn get_latest_transactions( path: &str, app_id: &str, @@ -101,8 +116,8 @@ mod tests { let log_segment = snapshot.log_segment(); ( - SetTransactionScanner::get_all(log_segment, &engine, None).unwrap(), - SetTransactionScanner::get_one(log_segment, app_id, &engine, None).unwrap(), + SetTransactionScanner::get_all(log_segment, &engine).unwrap(), + SetTransactionScanner::get_one(log_segment, app_id, &engine).unwrap(), ) } @@ -159,7 +174,7 @@ mod tests { } #[test] - fn test_txn_retention_filtering() { + fn test_get_all_returns_every_txn_unfiltered() { let path = std::fs::canonicalize(PathBuf::from("./tests/data/app-txn-with-last-updated/")); let url = url::Url::from_directory_path(path.unwrap()).unwrap(); let engine = SyncEngine::new(); @@ -167,34 +182,25 @@ mod tests { let snapshot = Snapshot::builder_for(url).build(&engine).unwrap(); let log_segment = snapshot.log_segment(); - // Test with no retention (should get all transactions) - let all_txns = SetTransactionScanner::get_all(log_segment, &engine, None).unwrap(); + // The scanner returns every app_id regardless of `lastUpdated`; callers apply retention. + let all_txns = SetTransactionScanner::get_all(log_segment, &engine).unwrap(); assert_eq!(all_txns.len(), 4); - - // Test with retention that filters out old transactions - let expiration_timestamp = Some(100); // Very old timestamp - let filtered_txns = - SetTransactionScanner::get_all(log_segment, &engine, expiration_timestamp).unwrap(); - - // Exact count depends on test data - assert!(filtered_txns.len() <= all_txns.len()); } #[test] - fn test_visitor_retention_with_null_last_updated() { + fn test_visitor_keeps_newest_by_log_order_regardless_of_last_updated() { + // Batches are visited in reverse log order, so the first row wins per app_id. let json_strings: StringArray = vec![ - r#"{"txn":{"appId":"app_with_time","version":1,"lastUpdated":100}}"#, - r#"{"txn":{"appId":"app_without_time","version":2}}"#, + r#"{"txn":{"appId":"app","version":2,"lastUpdated":100}}"#, + r#"{"txn":{"appId":"app","version":1,"lastUpdated":999}}"#, ] .into(); let batch = parse_json_batch(json_strings); - let mut visitor = SetTransactionVisitor::new(None, Some(1000)); + let mut visitor = SetTransactionVisitor::new(None); visitor.visit_rows_of(batch.as_ref()).unwrap(); - // app_with_last_updated should be filtered out (100 < 1000) - // app_without_last_updated should be kept assert_eq!(visitor.set_transactions.len(), 1); - assert!(visitor.set_transactions.contains_key("app_without_time")); + assert_eq!(visitor.set_transactions.get("app").unwrap().version, 2); } } diff --git a/kernel/src/actions/visitors.rs b/kernel/src/actions/visitors.rs index 22d966bfc8..c6551cbb63 100644 --- a/kernel/src/actions/visitors.rs +++ b/kernel/src/actions/visitors.rs @@ -7,14 +7,16 @@ use std::sync::{Arc, LazyLock}; use delta_kernel_derive::internal_api; +use super::deletion_vector::DeletionVectorDescriptor; +use super::*; use crate::engine_data::{GetData, RowVisitor, TypedGetData as _}; +use crate::log_segment::DomainMetadataMap; use crate::schema::{column_name, ColumnName, ColumnNamesAndTypes, DataType, Schema, StructField}; use crate::utils::require; use crate::{DeltaResult, Error}; -use super::deletion_vector::DeletionVectorDescriptor; -use super::*; -use crate::log_segment::DomainMetadataMap; +pub(crate) static METADATA_LEAVES: LazyLock = + LazyLock::new(|| Metadata::to_schema().leaves(METADATA_NAME)); #[derive(Default)] #[internal_api] @@ -24,9 +26,7 @@ pub(crate) struct MetadataVisitor { impl RowVisitor for MetadataVisitor { fn selected_column_names_and_types(&self) -> (&'static [ColumnName], &'static [DataType]) { - static NAMES_AND_TYPES: LazyLock = - LazyLock::new(|| Metadata::to_schema().leaves(METADATA_NAME)); - NAMES_AND_TYPES.as_ref() + METADATA_LEAVES.as_ref() } fn visit<'a>(&mut self, row_count: usize, getters: &[&'a dyn GetData<'a>]) -> DeltaResult<()> { @@ -43,6 +43,7 @@ impl RowVisitor for MetadataVisitor { #[derive(Default)] pub(crate) struct SelectionVectorVisitor { pub(crate) selection_vector: Vec, + pub(crate) num_filtered: u64, } /// A single non-nullable BOOL column @@ -61,13 +62,19 @@ impl RowVisitor for SelectionVectorVisitor { )) ); for i in 0..row_count { - self.selection_vector - .push(getters[0].get(i, "selectionvector.output")?); + let selected: bool = getters[0].get(i, "selectionvector.output")?; + if !selected { + self.num_filtered += 1; + } + self.selection_vector.push(selected); } Ok(()) } } +pub(crate) static PROTOCOL_LEAVES: LazyLock = + LazyLock::new(|| Protocol::to_schema().leaves(PROTOCOL_NAME)); + #[derive(Default)] #[internal_api] pub(crate) struct ProtocolVisitor { @@ -76,9 +83,7 @@ pub(crate) struct ProtocolVisitor { impl RowVisitor for ProtocolVisitor { fn selected_column_names_and_types(&self) -> (&'static [ColumnName], &'static [DataType]) { - static NAMES_AND_TYPES: LazyLock = - LazyLock::new(|| Protocol::to_schema().leaves(PROTOCOL_NAME)); - NAMES_AND_TYPES.as_ref() + PROTOCOL_LEAVES.as_ref() } fn visit<'a>(&mut self, row_count: usize, getters: &[&'a dyn GetData<'a>]) -> DeltaResult<()> { for i in 0..row_count { @@ -297,24 +302,19 @@ pub(crate) type SetTransactionMap = HashMap; /// `application_id` can be set. This bounds the memory required for the /// visitor to at most one entry and reduces the amount of processing /// required. -/// #[derive(Default, Debug)] #[internal_api] pub(crate) struct SetTransactionVisitor { pub(crate) set_transactions: SetTransactionMap, pub(crate) application_id: Option, - /// Minimum timestamp for transaction retention. Transactions with last_updated - /// older than or equal to this timestamp will be filtered out. None means no filtering. - expiration_timestamp: Option, } impl SetTransactionVisitor { /// Create a new visitor. When application_id is set then bookkeeping is only for that id only - pub(crate) fn new(application_id: Option, expiration_timestamp: Option) -> Self { + pub(crate) fn new(application_id: Option) -> Self { SetTransactionVisitor { set_transactions: HashMap::default(), application_id, - expiration_timestamp, } } @@ -359,14 +359,6 @@ impl RowVisitor for SetTransactionVisitor { .is_none_or(|requested| requested.eq(&app_id)) { let txn = SetTransactionVisitor::visit_txn(i, app_id, getters)?; - // Check retention: filter out transactions that are old - // If last_updated is None, the transaction never expires - match self.expiration_timestamp.zip(txn.last_updated) { - Some((expiration_ts, last_updated)) if last_updated <= expiration_ts => { - continue - } - _ => (), - } if !self.set_transactions.contains_key(&txn.app_id) { self.set_transactions.insert(txn.app_id.clone(), txn); } @@ -440,7 +432,8 @@ pub(crate) struct DomainMetadataVisitor { impl DomainMetadataVisitor { /// Create a new visitor. When domain_filter is set then we only retain domain metadata for - /// domains in the provided set, enabling early termination once all requested domains are found. + /// domains in the provided set, enabling early termination once all requested domains are + /// found. pub(crate) fn new(domain_filter: Option>) -> Self { DomainMetadataVisitor { domain_filter, @@ -478,9 +471,15 @@ impl DomainMetadataVisitor { .is_some_and(|filter| self.domain_metadatas.len() == filter.len()) } - pub(crate) fn into_domain_metadatas(mut self) -> DomainMetadataMap { - // note that the resulting visitor.domain_metadatas includes removed domains, so we need to filter - self.domain_metadatas.retain(|_, dm| !dm.removed); + pub(crate) fn into_domain_metadatas(self) -> DomainMetadataMap { + let mut domain_metadatas = self.domain_metadatas; + domain_metadatas.retain(|_, dm| !dm.removed); + domain_metadatas + } + + /// The newest-wins domain-metadata map, retaining tombstones (`removed == true`). + /// [`Self::into_domain_metadatas`] returns the same map with tombstones stripped. + pub(crate) fn into_domain_metadatas_including_tombstones(self) -> DomainMetadataMap { self.domain_metadatas } } @@ -694,13 +693,14 @@ impl RowVisitor for InCommitTimestampVisitor { #[cfg(test)] mod tests { use super::*; - - use crate::arrow::array::StringArray; - + use crate::arrow::array::{BooleanArray, StringArray}; + use crate::arrow::datatypes::{DataType, Field, Schema as ArrowSchema}; + use crate::arrow::record_batch::RecordBatch; + use crate::engine::arrow_data::ArrowEngineData; use crate::engine::sync::SyncEngine; use crate::expressions::{column_expr_ref, Expression}; use crate::table_features::TableFeature; - use crate::utils::test_utils::{action_batch, parse_json_batch}; + use crate::unit_test_utils::{action_batch, parse_json_batch}; use crate::Engine; #[test] @@ -838,7 +838,7 @@ mod tests { }; let expected = vec![add1, add2, add3]; assert_eq!(add_visitor.adds.len(), expected.len()); - for (add, expected) in add_visitor.adds.into_iter().zip(expected.into_iter()) { + for (add, expected) in add_visitor.adds.into_iter().zip(expected) { assert_eq!(add, expected); } } @@ -1243,9 +1243,9 @@ mod tests { assert!(visitor.into_domain_metadatas().is_empty()); } - /************************************* - * In-commit timestamp visitor tests * - **************************************/ + // ------------------------------------------------------------ + // In-commit timestamp visitor tests + // ------------------------------------------------------------ fn add_action() -> &'static str { r#"{"add":{"path":"file1","partitionValues":{"c1":"6","c2":"a"},"size":452,"modificationTime":1670892998137,"dataChange":true}}"# @@ -1299,4 +1299,29 @@ mod tests { Some(1677811178585), // Retrieved ICT ); } + + // Helper to create a boolean batch for SelectionVectorVisitor tests + fn create_boolean_batch(values: Vec) -> Box { + let array = BooleanArray::from(values); + let arrow_schema = ArrowSchema::new(vec![Field::new("output", DataType::Boolean, false)]); + let batch = RecordBatch::try_new(Arc::new(arrow_schema), vec![Arc::new(array)]).unwrap(); + Box::new(ArrowEngineData::new(batch)) + } + + #[rstest::rstest] + #[case::empty_batch(vec![], 0, "empty batch should have no filtered rows")] + #[case::all_selected(vec![true, true, true, true], 0, "all selected should have no filtered rows")] + #[case::all_filtered(vec![false, false, false, false, false], 5, "all filtered should count all rows")] + #[case::mixed_selection(vec![true, false, true, false, false, true], 3, "mixed selection should count false values")] + fn selection_vector_visitor_counter_accuracy( + #[case] input: Vec, + #[case] expected_filtered: u64, + #[case] _description: &str, + ) { + let batch = create_boolean_batch(input.clone()); + let mut visitor = SelectionVectorVisitor::default(); + visitor.visit_rows_of(batch.as_ref()).unwrap(); + assert_eq!(visitor.selection_vector, input); + assert_eq!(visitor.num_filtered, expected_filtered); + } } diff --git a/kernel/src/arrow_compat.rs b/kernel/src/arrow_compat.rs index 08e2d0e8ab..b39a7e6ca1 100644 --- a/kernel/src/arrow_compat.rs +++ b/kernel/src/arrow_compat.rs @@ -1,28 +1,36 @@ //! This module re-exports the different versions of arrow, parquet, and object_store we support. -#[cfg(feature = "arrow-57")] +#[cfg(feature = "arrow-59")] mod arrow_compat_shims { - pub use arrow_57 as arrow; - pub use object_store_12 as object_store; - pub use parquet_57 as parquet; + pub use arrow_59 as arrow; + pub use parquet_59 as parquet; + + pub mod object_store { + pub use object_store_13::*; + } } -#[cfg(all(feature = "arrow-56", not(feature = "arrow-57"),))] +#[cfg(all(feature = "arrow-58", not(feature = "arrow-59")))] mod arrow_compat_shims { - pub use arrow_56 as arrow; - pub use object_store_12 as object_store; - pub use parquet_56 as parquet; + pub use arrow_58 as arrow; + pub use parquet_58 as parquet; + + pub mod object_store { + pub use object_store_13::*; + } } // if nothing is enabled but we need arrow because of some other feature flag, throw compile-time // error #[cfg(all( feature = "need-arrow", - not(feature = "arrow-56"), - not(feature = "arrow-57") + not(feature = "arrow-58"), + not(feature = "arrow-59") ))] -compile_error!("Requested a feature that needs arrow without enabling arrow. Please enable the `arrow-56`, or `arrow-57` feature"); +compile_error!( + "Requested a feature that needs arrow without enabling arrow. Please enable the `arrow-58` or `arrow-59` feature" +); -#[cfg(any(feature = "arrow-56", feature = "arrow-57"))] +#[cfg(any(feature = "arrow-58", feature = "arrow-59"))] #[doc(hidden)] pub use arrow_compat_shims::*; diff --git a/kernel/src/cancellation.rs b/kernel/src/cancellation.rs new file mode 100644 index 0000000000..07f63e9f07 --- /dev/null +++ b/kernel/src/cancellation.rs @@ -0,0 +1,208 @@ +//! Cooperative cancellation for long-running Kernel reads. +//! +//! Kernel never does I/O itself and owns no async runtime, so cancellation is *cooperative* and +//! *runtime-agnostic*: a caller supplies a [`CancellationToken`] (via +//! [`ScanBuilder::with_cancellation_token`](crate::scan::ScanBuilder::with_cancellation_token)), +//! Kernel polls it at action-batch boundaries, and cancellation-aware [`Engine`](crate::Engine) +//! reads may race their I/O against it. Cancellation is always surfaced as +//! [`Error::Cancelled`] -- never as normal iterator exhaustion -- so a partial listing can never be +//! mistaken for a complete one. + +use std::future::Future; +use std::pin::Pin; +use std::sync::Arc; + +use crate::{DeltaResult, Error}; + +/// A shared, thread-safe cancellation token. Held as an `Arc` because the lazy scan iterator and +/// the engine reads it drives can outlive the builder call and run on other threads. +pub type CancellationTokenRef = Arc; + +/// A future that resolves when a [`CancellationToken`] is cancelled. Runtime-neutral: it is a +/// plain boxed [`Future`], so an engine can `select!` it against its own async reads without +/// Kernel taking on any async-runtime dependency. +pub type CancelledFuture<'a> = Pin + Send + 'a>>; + +/// Returns `Err(Error::Cancelled)` if `token` is present and already cancelled, else `Ok(())`. +/// +/// Used to fail fast before starting a setup/read operation (e.g. a footer read or a sidecar +/// listing) so cancelled work is not begun. +pub(crate) fn check_cancelled(token: Option<&CancellationTokenRef>) -> DeltaResult<()> { + match token { + Some(t) if t.is_cancelled() => Err(Error::Cancelled), + _ => Ok(()), + } +} + +/// A cooperative cancellation signal supplied by a caller. +/// +/// Implementors wrap whatever their runtime provides (e.g. `tokio_util::sync::CancellationToken`). +/// Kernel and cancellation-aware engines only *consume* it: Kernel polls [`is_cancelled`] between +/// action batches, while an async engine may await [`cancelled_future`] to wake blocked I/O. +/// +/// [`is_cancelled`]: CancellationToken::is_cancelled +/// [`cancelled_future`]: CancellationToken::cancelled_future +pub trait CancellationToken: Send + Sync { + /// Returns `true` once cancellation has been requested. Cheap, synchronous, and monotonic: + /// once it returns `true` it must never return `false` again. + fn is_cancelled(&self) -> bool; + + /// Returns a future that resolves when the token is cancelled (immediately if it already is). + /// + /// There is no default implementation: a correct notification cannot be synthesized from + /// [`is_cancelled`](Self::is_cancelled) alone without either busy-polling or a runtime, and + /// Kernel has neither. Implementors back this with their own notification primitive. + fn cancelled_future(&self) -> CancelledFuture<'_>; +} + +/// Wraps a fallible iterator so that cancellation terminates it with a single +/// [`Error::Cancelled`] rather than silent truncation. +/// +/// Before each pull, the token is polled: if cancelled, one `Err(Error::Cancelled)` is yielded +/// and every subsequent call returns `None` (the iterator is fused). An `Err(Error::Cancelled)` +/// arriving from the inner iterator (e.g. a cancellation-aware engine interrupting a read) fuses +/// it the same way, so a token shared with the engine still yields exactly one terminal error. +/// With no token, or before cancellation, items pass through unchanged. This is deliberately +/// **not** `take_while`, which would end the iterator with `None` and make a cancelled listing +/// look complete. +pub(crate) struct CancellableIterator { + inner: I, + token: Option, + /// Set once cancellation has been observed and the terminal error emitted, fusing the + /// iterator to `None` thereafter. + done: bool, +} + +impl CancellableIterator { + pub(crate) fn new(inner: I, token: Option) -> Self { + Self { + inner, + token, + done: false, + } + } +} + +impl Iterator for CancellableIterator +where + I: Iterator>, +{ + type Item = DeltaResult; + + fn next(&mut self) -> Option { + if self.done { + return None; + } + if self.token.as_ref().is_some_and(|t| t.is_cancelled()) { + self.done = true; + return Some(Err(Error::Cancelled)); + } + let item = self.inner.next(); + // A cancellation-aware engine can itself surface `Err(Cancelled)` from an interrupted + // read. Fuse on it so the composed pipeline still yields exactly one terminal error + // rather than this layer re-injecting a second one on the next poll. + if matches!(item, Some(Err(Error::Cancelled))) { + self.done = true; + } + item + } +} + +#[cfg(test)] +mod tests { + use std::future::ready; + use std::sync::atomic::{AtomicBool, Ordering}; + + use super::*; + + /// Minimal [`CancellationToken`] backed by an [`AtomicBool`], for tests. + #[derive(Default)] + struct TestToken(AtomicBool); + + impl TestToken { + fn cancel(&self) { + self.0.store(true, Ordering::SeqCst); + } + } + + impl CancellationToken for TestToken { + fn is_cancelled(&self) -> bool { + self.0.load(Ordering::SeqCst) + } + fn cancelled_future(&self) -> CancelledFuture<'_> { + // Tests only drive `is_cancelled`; a resolved/pending future is enough here. + Box::pin(ready(())) + } + } + + fn ok_iter(n: usize) -> impl Iterator> { + (0..n).map(Ok) + } + + #[test] + fn no_token_passes_through_unchanged() { + let out: Vec<_> = CancellableIterator::new(ok_iter(3), None) + .map(Result::unwrap) + .collect(); + assert_eq!(out, vec![0, 1, 2]); + } + + #[test] + fn uncancelled_token_passes_through_unchanged() { + let token: CancellationTokenRef = Arc::new(TestToken::default()); + let out: Vec<_> = CancellableIterator::new(ok_iter(3), Some(token)) + .map(Result::unwrap) + .collect(); + assert_eq!(out, vec![0, 1, 2]); + } + + #[test] + fn pre_cancelled_yields_one_error_then_ends() { + let token = Arc::new(TestToken::default()); + token.cancel(); + let mut iter = CancellableIterator::new(ok_iter(3), Some(token as CancellationTokenRef)); + assert!(matches!(iter.next(), Some(Err(Error::Cancelled)))); + // Fused: never a `Some(Ok(..))` after cancellation, and no infinite error stream. + assert!(iter.next().is_none()); + assert!(iter.next().is_none()); + } + + #[test] + fn mid_stream_cancellation_yields_error_not_silent_truncation() { + let token = Arc::new(TestToken::default()); + let ct: CancellationTokenRef = token.clone(); + let mut iter = CancellableIterator::new(ok_iter(5), Some(ct)); + assert!(matches!(iter.next(), Some(Ok(0)))); + assert!(matches!(iter.next(), Some(Ok(1)))); + token.cancel(); + // The terminal item is an error, so a cancelled listing can't look complete (which a + // bare `None` / `take_while` would). + assert!(matches!(iter.next(), Some(Err(Error::Cancelled)))); + assert!(iter.next().is_none()); + } + + // An `Err(Cancelled)` from the inner iterator (as a cancellation-aware engine emits) must + // fuse this layer, so a token shared between engine and kernel yields exactly ONE terminal + // error, not two. Regression guard for the double-emit the layered pipeline would otherwise + // produce. The token is left uncancelled so the fuse comes solely from the inner error. + #[test] + fn inner_cancelled_error_fuses_without_double_emit() { + let token: CancellationTokenRef = Arc::new(TestToken::default()); + let inner = vec![Ok(0), Err(Error::Cancelled), Ok(99)].into_iter(); + let mut iter = CancellableIterator::new(inner, Some(token)); + assert!(matches!(iter.next(), Some(Ok(0)))); + assert!(matches!(iter.next(), Some(Err(Error::Cancelled)))); + // Fused on the inner error: the trailing Ok is never yielded, and no second error. + assert!(iter.next().is_none()); + } + + #[test] + fn check_cancelled_reports_state() { + let token = Arc::new(TestToken::default()); + let ct: CancellationTokenRef = token.clone(); + assert!(check_cancelled(Some(&ct)).is_ok()); + assert!(check_cancelled(None).is_ok()); + token.cancel(); + assert!(matches!(check_cancelled(Some(&ct)), Err(Error::Cancelled))); + } +} diff --git a/kernel/src/checkpoint/checkpoint_shape.rs b/kernel/src/checkpoint/checkpoint_shape.rs new file mode 100644 index 0000000000..95d6adf457 --- /dev/null +++ b/kernel/src/checkpoint/checkpoint_shape.rs @@ -0,0 +1,540 @@ +//! Resolves a checkpoint shape. This falls into the following cases: no checkpoint, +//! leaf (file actions inline, including multi-part), or manifest (which references sidecar files). +//! When schemas are requested, also reports whether the checkpoint has compatible parsed stats +//! and partition values. +//! Driven through a [`PlanExecutor`]. + +use url::Url; + +use crate::actions::visitors::SidecarVisitor; +use crate::actions::SIDECAR_NAME; +use crate::engine_data::RowVisitor; +use crate::log_segment::LogSegment; +use crate::plans::ir::nodes::FileType; +use crate::plans::{Operation, PlanBuilder, PlanExecutor}; +use crate::schema::SchemaRef; +use crate::snapshot::Snapshot; +use crate::{DeltaResult, FileMeta}; + +/// Topology of a checkpoint: where the `add` / `remove` actions live. +#[derive(Clone, Debug, PartialEq)] +pub(crate) enum CheckpointType { + /// No checkpoint files. + None, + /// File actions inline. Classic V1, inline V2, and multi-part V1. + Leaf, + /// V2 manifest: checkpoint references sidecar files holding the file actions. + Manifest, +} + +/// A snapshot's resolved checkpoint type and compatible parsed schemas. +#[derive(Clone, Debug, PartialEq)] +pub(crate) struct CheckpointShape { + /// What kind of checkpoint this is. + pub(crate) checkpoint_type: CheckpointType, + /// The requested stats schema when the checkpoint has a compatible `add.stats_parsed` struct + /// to read it from; `None` when stats were not requested or no compatible parsed stats exist. + pub(crate) parsed_stats_schema: Option, + /// The requested partition schema when the checkpoint has a compatible + /// `add.partitionValues_parsed` struct; `None` when partition values were not requested or no + /// compatible parsed partition values exist. + pub(crate) parsed_partition_schema: Option, +} + +impl CheckpointShape { + /// Resolve `snapshot`'s checkpoint shape and compatible parsed stats and partition schemas. + pub(crate) fn try_new( + exec: &dyn PlanExecutor, + snapshot: &Snapshot, + stats_schema: Option<&SchemaRef>, + partition_schema: Option<&SchemaRef>, + ) -> DeltaResult { + let segment = snapshot.log_segment(); + + let (root_checkpoint, file_type) = match segment.listed.checkpoint_parts.first() { + Some(checkpoint) if checkpoint.is_json() => (&checkpoint.location, FileType::Json), + Some(checkpoint) => (&checkpoint.location, FileType::Parquet), + None => { + return Ok(CheckpointShape { + checkpoint_type: CheckpointType::None, + parsed_stats_schema: None, + parsed_partition_schema: None, + }) + } + }; + + // Classify from a V2 checkpoint's `_last_checkpoint` hint when possible, else inspect the + // file. + if let Some(shape) = Self::from_v2_checkpoint_hint( + exec, + segment, + root_checkpoint, + file_type, + stats_schema, + partition_schema, + )? { + return Ok(shape); + } + + // A checkpoint with sidecars is a manifest, one without is a leaf. + match file_type { + FileType::Parquet => { + let cp_schema = match segment.checkpoint_hint_schema() { + Some(schema) => schema, + None => exec.read_parquet_footer(root_checkpoint.clone())?.schema, + }; + // No `sidecar` column means the file actions are inline, so this is a leaf. + if !cp_schema.contains(SIDECAR_NAME) { + return Ok(Self::try_new_leaf( + Some(cp_schema), + stats_schema, + partition_schema, + )); + } + // The `sidecar` column may still be all-null (not a manifest), so scan it to + // confirm whether a sidecar is actually present. + match collect_single_sidecar(exec, root_checkpoint, file_type, &segment.log_root)? { + Some(sidecar) => { + Self::try_new_manifest(exec, sidecar, stats_schema, partition_schema) + } + None => Ok(Self::try_new_leaf( + Some(cp_schema), + stats_schema, + partition_schema, + )), + } + } + // A JSON checkpoint has no footer schema to inspect, so try to collect a sidecar to + // decide if it is a manifest or a leaf. A JSON leaf has no readable schema, + // hence no parsed stats. + FileType::Json => { + match collect_single_sidecar(exec, root_checkpoint, file_type, &segment.log_root)? { + Some(sidecar) => { + Self::try_new_manifest(exec, sidecar, stats_schema, partition_schema) + } + None => Ok(Self::try_new_leaf(None, stats_schema, partition_schema)), + } + } + } + } + + /// Classify the checkpoint from its `_last_checkpoint` sidecar hint, without reading the + /// checkpoint file. Returns `None` when the hint is absent (or was trimmed away), leaving the + /// caller to inspect the file. A non-empty sidecar list is a manifest; an empty list is a leaf + /// (the writer emits an empty list only for a leaf, and trims an oversized manifest to absent, + /// never to empty). + fn from_v2_checkpoint_hint( + exec: &dyn PlanExecutor, + segment: &LogSegment, + root_checkpoint: &FileMeta, + file_type: FileType, + stats_schema: Option<&SchemaRef>, + partition_schema: Option<&SchemaRef>, + ) -> DeltaResult> { + match segment.checkpoint_hint_sidecars().map(Vec::as_slice) { + Some([sidecar, ..]) => { + let sidecar_meta = sidecar.to_filemeta(&segment.log_root)?; + let result = + Self::try_new_manifest(exec, sidecar_meta, stats_schema, partition_schema)?; + Ok(Some(result)) + } + Some([]) => { + // A parquet leaf's parsed stats and partition values live in its own schema; read + // it when either schema is requested. A JSON leaf has no readable schema. + let leaf_schema = match file_type { + FileType::Parquet if stats_schema.is_some() || partition_schema.is_some() => { + Some(match segment.checkpoint_hint_schema() { + Some(schema) => schema, + None => exec.read_parquet_footer(root_checkpoint.clone())?.schema, + }) + } + _ => None, + }; + Ok(Some(Self::try_new_leaf( + leaf_schema, + stats_schema, + partition_schema, + ))) + } + None => Ok(None), + } + } + + /// Build the shape for a manifest checkpoint. File actions live in sidecars, so probe the first + /// sidecar footer when stats or partition schemas are requested; all sidecars share one schema. + fn try_new_manifest( + exec: &dyn PlanExecutor, + sidecar: FileMeta, + stats_schema: Option<&SchemaRef>, + partition_schema: Option<&SchemaRef>, + ) -> DeltaResult { + let footer_schema = if stats_schema.is_some() || partition_schema.is_some() { + Some(exec.read_parquet_footer(sidecar)?.schema) + } else { + None + }; + let parsed_stats_schema = stats_schema.filter(|stats_schema| { + footer_schema.as_ref().is_some_and(|footer_schema| { + LogSegment::schema_has_compatible_stats_parsed(footer_schema, stats_schema) + }) + }); + let parsed_partition_schema = partition_schema.filter(|partition_schema| { + footer_schema.as_ref().is_some_and(|footer_schema| { + LogSegment::schema_has_compatible_partition_values_parsed( + footer_schema, + partition_schema, + ) + }) + }); + Ok(CheckpointShape { + checkpoint_type: CheckpointType::Manifest, + parsed_stats_schema: parsed_stats_schema.cloned(), + parsed_partition_schema: parsed_partition_schema.cloned(), + }) + } + + /// Build the shape for a leaf checkpoint. Its file actions are inline, so `leaf_schema` carries + /// their stats (`None` for a JSON leaf, which has no readable footer schema). + fn try_new_leaf( + leaf_schema: Option, + stats_schema: Option<&SchemaRef>, + partition_schema: Option<&SchemaRef>, + ) -> CheckpointShape { + let parsed_stats_schema = stats_schema.filter(|stats_schema| { + leaf_schema.as_ref().is_some_and(|leaf_schema| { + LogSegment::schema_has_compatible_stats_parsed(leaf_schema.as_ref(), stats_schema) + }) + }); + let parsed_partition_schema = partition_schema.filter(|partition_schema| { + leaf_schema.as_ref().is_some_and(|leaf_schema| { + LogSegment::schema_has_compatible_partition_values_parsed( + leaf_schema, + partition_schema, + ) + }) + }); + CheckpointShape { + checkpoint_type: CheckpointType::Leaf, + parsed_stats_schema: parsed_stats_schema.cloned(), + parsed_partition_schema: parsed_partition_schema.cloned(), + } + } +} + +/// Read the checkpoint `file`'s `sidecar` column, returning the first referenced sidecar's +/// [`FileMeta`] (enough to classify and probe; not a full enumeration). +fn collect_single_sidecar( + exec: &dyn PlanExecutor, + file: &FileMeta, + file_format: FileType, + log_root: &Url, +) -> DeltaResult> { + let read_schema = LogSegment::sidecar_read_schema(); + // No file-constant columns: the sidecar column is read directly from each file. + let plan = match file_format { + FileType::Parquet => PlanBuilder::scan_parquet([file.clone()], &[], read_schema), + FileType::Json => PlanBuilder::scan_json([file.clone()], &[], read_schema), + }? + .build()?; + let data = exec.execute_op(Operation::QueryPlan(plan))?.into_data()?; + + let mut visitor = SidecarVisitor::default(); + for batch in data { + visitor.visit_rows_of(batch?.as_ref())?; + if !visitor.sidecars.is_empty() { + break; + } + } + match visitor.sidecars.first() { + Some(sidecar) => Ok(Some(sidecar.to_filemeta(log_root)?)), + None => Ok(None), + } +} + +#[cfg(test)] +mod tests { + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::sync::Arc; + + use rstest::rstest; + + use super::*; + use crate::actions::{MAX_VALUES, MIN_VALUES, NUM_RECORDS}; + use crate::engine::sync::plan::SyncPlanExecutor; + use crate::plans::{IoOperation, PlanResult}; + use crate::schema::{DataType, StructField, StructType}; + use crate::unit_test_utils::load_test_table; + + /// Counts ops by kind and delegates to `SyncPlanExecutor`, to assert which I/O the fast path + /// performs. + struct CountingExecutor { + inner: SyncPlanExecutor, + query_scans: AtomicUsize, + footer_reads: AtomicUsize, + } + + impl CountingExecutor { + fn new() -> Self { + Self { + inner: SyncPlanExecutor::default(), + query_scans: AtomicUsize::new(0), + footer_reads: AtomicUsize::new(0), + } + } + } + + impl PlanExecutor for CountingExecutor { + fn execute_op(&self, op: Operation) -> DeltaResult { + match &op { + Operation::QueryPlan(_) => _ = self.query_scans.fetch_add(1, Ordering::Relaxed), + Operation::IoOperation(IoOperation::ParquetFooter { .. }) => { + _ = self.footer_reads.fetch_add(1, Ordering::Relaxed) + } + _ => {} + } + self.inner.execute_op(op) + } + } + + /// Resolves checkpoint shape and, when requested, parsed-stats availability across fixtures. + #[rstest] + #[case::no_checkpoint("app-txn-no-checkpoint", CheckpointType::None, None)] + #[case::no_checkpoint_with_stats("app-txn-no-checkpoint", CheckpointType::None, Some(false))] + #[case::leaf_parquet("with_checkpoint_no_last_checkpoint", CheckpointType::Leaf, None)] + #[case::manifest_parquet( + "v2-checkpoints-parquet-with-sidecars", + CheckpointType::Manifest, + Some(false) + )] + #[case::manifest_json("v2-checkpoints-json-with-sidecars", CheckpointType::Manifest, None)] + #[case::leaf_json_inline( + "v2-checkpoints-json-without-sidecars", + CheckpointType::Leaf, + Some(false) + )] + // Regression: an all-null `sidecar` column is still a leaf. + #[case::leaf_parquet_inline( + "v2-checkpoints-parquet-without-sidecars", + CheckpointType::Leaf, + None + )] + #[case::leaf_multipart("v1-multi-part-struct-stats-only", CheckpointType::Leaf, Some(true))] + #[case::json_stats_classic( + "v2-classic-checkpoint-parquet", + CheckpointType::Manifest, + Some(false) + )] + #[case::struct_stats_leaf( + "v2-classic-parquet-struct-stats-only", + CheckpointType::Leaf, + Some(true) + )] + #[case::struct_stats_json_manifest( + "v2-json-sidecars-struct-stats-only", + CheckpointType::Manifest, + Some(true) + )] + #[case::struct_stats_parquet_manifest( + "v2-parquet-sidecars-struct-stats-only", + CheckpointType::Manifest, + Some(true) + )] + fn resolve_checkpoint_and_stats( + #[case] table: &str, + #[case] expected_checkpoint: CheckpointType, + #[case] expect_parsed: Option, + ) { + let (_engine, snapshot, _tempdir) = load_test_table(table).unwrap(); + let exec = SyncPlanExecutor::default(); + let stats_schema = expect_parsed.map(|_| probe_stats_schema()); + + let shape = CheckpointShape::try_new(&exec, snapshot.as_ref(), stats_schema.as_ref(), None) + .unwrap(); + + assert_eq!( + shape.checkpoint_type, expected_checkpoint, + "{table}: checkpoint type" + ); + + match expect_parsed { + // Stats not requested: no parsed-stats schema regardless of the checkpoint. + None => assert!( + shape.parsed_stats_schema.is_none(), + "{table}: stats not requested" + ), + // Requested with compatible parsed stats: the requested schema is echoed back. + Some(true) => assert_eq!( + shape.parsed_stats_schema.as_ref(), + stats_schema.as_ref(), + "{table}: parsed stats available, schema echoed" + ), + // Requested but no compatible parsed stats: `None`. + Some(false) => assert!( + shape.parsed_stats_schema.is_none(), + "{table}: no compatible parsed stats" + ), + } + } + + /// Requested stats schema for the `*-struct-stats-only` fixtures (`id: long`, `value: string`), + /// so compatibility does real per-column matching. + fn probe_stats_schema() -> SchemaRef { + let columns = || { + StructType::new_unchecked([ + StructField::nullable("id", DataType::LONG), + StructField::nullable("value", DataType::STRING), + ]) + }; + Arc::new(StructType::new_unchecked([ + StructField::nullable(NUM_RECORDS, DataType::LONG), + StructField::nullable(MIN_VALUES, columns()), + StructField::nullable(MAX_VALUES, columns()), + ])) + } + + /// Fast path on a manifest hint: one sidecar footer read, no drain (`query_scans == 0`). Guards + /// against the optimization silently not firing (result-only checks pass via the drain too). + #[test] + fn fast_path_skips_checkpoint_drain_when_hint_lists_sidecars() { + let (_engine, snapshot, _tempdir) = + load_test_table("v2-checkpoints-parquet-with-sidecars").unwrap(); + let exec = CountingExecutor::new(); + + let shape = + CheckpointShape::try_new(&exec, snapshot.as_ref(), Some(&probe_stats_schema()), None) + .unwrap(); + + assert_eq!(shape.checkpoint_type, CheckpointType::Manifest); + assert_eq!( + exec.query_scans.load(Ordering::Relaxed), + 0, + "fast path must not drain the checkpoint sidecar column" + ); + assert_eq!( + exec.footer_reads.load(Ordering::Relaxed), + 1, + "fast path footer-reads exactly the hint's first sidecar" + ); + } + + /// With the hint removed, a JSON manifest must drain the `sidecar` column (`query_scans >= 1`). + #[test] + fn resolve_json_manifest_via_drain_without_hint() { + let (engine, snapshot, _tempdir) = + load_test_table("v2-checkpoints-json-with-sidecars").unwrap(); + // Remove the hint so resolve must drain. + let hint = snapshot + .log_segment() + .log_root + .join("_last_checkpoint") + .unwrap(); + std::fs::remove_file(hint.to_file_path().unwrap()).unwrap(); + let table_root = snapshot + .table_configuration() + .table_root() + .as_str() + .to_string(); + let snapshot = Snapshot::builder_for(&table_root) + .build(engine.as_ref()) + .unwrap(); + + let exec = CountingExecutor::new(); + let shape = + CheckpointShape::try_new(&exec, snapshot.as_ref(), Some(&probe_stats_schema()), None) + .unwrap(); + + assert_eq!(shape.checkpoint_type, CheckpointType::Manifest); + assert!( + exec.query_scans.load(Ordering::Relaxed) >= 1, + "must drain, not fast-path" + ); + } + + /// Builds a `LogSegment` whose applicable hint carries `v2Checkpoint.sidecarFiles == Some([])` + /// (empty, not absent) for a checkpoint with the given extension. No real fixture carries an + /// empty sidecar list, so this synthetic segment is the only way to exercise the leaf fast + /// path. + fn segment_with_empty_sidecars_hint(extension: &str) -> LogSegment { + use crate::last_checkpoint_hint::{LastCheckpointHint, LastCheckpointV2}; + use crate::log_segment_files::LogSegmentFiles; + use crate::unit_test_utils::{create_log_path, create_log_path_with_size}; + + let (_store, log_root) = crate::checkpoint::tests::new_in_memory_store(); + let selected = format!( + "00000000000000000001.checkpoint.11111111-1111-1111-1111-111111111111.{extension}" + ); + let checkpoint_file = log_root.join(&selected).unwrap().to_string(); + let commit = create_log_path(log_root.join("00000000000000000002.json").unwrap().as_str()); + LogSegment::try_new( + LogSegmentFiles { + checkpoint_parts: vec![create_log_path_with_size(&checkpoint_file, 1)], + ascending_commit_files: vec![commit.clone()], + latest_commit_file: Some(commit), + ..Default::default() + }, + log_root, + None, + Some(LastCheckpointHint { + version: 1, + v2_checkpoint: Some(LastCheckpointV2 { + path: selected, + sidecar_files: Some(vec![]), + ..Default::default() + }), + ..Default::default() + }), + ) + .unwrap() + } + + /// An empty-sidecars hint (`Some([])`) classifies as a leaf without inspecting the checkpoint. + /// A JSON leaf never reports parsed stats; a parquet leaf reads its schema only when stats are + /// requested. `None` here means the hint short-circuited (a fall-through would return `Some`). + #[rstest] + // JSON leaf: no readable schema, so never any parsed stats, and never any I/O. + #[case::json_no_stats("json", false, None, 0)] + #[case::json_with_stats("json", true, None, 0)] + // Parquet leaf: no stats requested -> no schema read, no parsed stats. + #[case::parquet_no_stats("parquet", false, None, 0)] + fn empty_sidecars_hint_classifies_leaf_without_drain( + #[case] extension: &str, + #[case] request_stats: bool, + #[case] expect_parsed: Option, + #[case] expected_footer_reads: usize, + ) { + let segment = segment_with_empty_sidecars_hint(extension); + let root = &segment.listed.checkpoint_parts[0].location; + let file_type = match extension { + "json" => FileType::Json, + _ => FileType::Parquet, + }; + let stats_schema = request_stats.then(probe_stats_schema); + let exec = CountingExecutor::new(); + + let shape = CheckpointShape::from_v2_checkpoint_hint( + &exec, + &segment, + root, + file_type, + stats_schema.as_ref(), + None, + ) + .unwrap() + .expect("an empty-sidecars hint must classify without falling through"); + + assert_eq!(shape.checkpoint_type, CheckpointType::Leaf); + assert_eq!( + shape.parsed_stats_schema.is_some(), + expect_parsed == Some(true) + ); + assert_eq!( + exec.query_scans.load(Ordering::Relaxed), + 0, + "empty-sidecars leaf must never drain the checkpoint" + ); + assert_eq!( + exec.footer_reads.load(Ordering::Relaxed), + expected_footer_reads + ); + } +} diff --git a/kernel/src/checkpoint/checkpoint_transform.rs b/kernel/src/checkpoint/checkpoint_transform.rs index 95c1db997a..b442c5f875 100644 --- a/kernel/src/checkpoint/checkpoint_transform.rs +++ b/kernel/src/checkpoint/checkpoint_transform.rs @@ -15,14 +15,15 @@ use std::sync::{Arc, LazyLock}; -use crate::actions::ADD_NAME; -use crate::expressions::{Expression, ExpressionRef, Transform, UnaryExpressionOp}; -use crate::schema::{DataType, SchemaRef, StructField, StructType}; +use crate::actions::{ADD_NAME, STATS_PARSED as STATS_PARSED_FIELD}; +use crate::expressions::{col, Expression, ExpressionRef, MapToStructOptions, UnaryExpressionOp}; +use crate::schema::{DataType, SchemaRef, SchemaStructPatchBuilder, StructField, StructType}; +use crate::struct_patch::ProjectionStructPatchBuilder; use crate::table_properties::TableProperties; +use crate::utils::FoldWithOption as _; use crate::{DeltaResult, Error}; pub(crate) const STATS_FIELD: &str = "stats"; -pub(crate) const STATS_PARSED_FIELD: &str = "stats_parsed"; pub(crate) const PARTITION_VALUES_FIELD: &str = "partitionValues"; pub(crate) const PARTITION_VALUES_PARSED_FIELD: &str = "partitionValues_parsed"; @@ -58,38 +59,43 @@ impl StatsTransformConfig { /// /// # Arguments /// -/// * `stats_schema` - The expected schema for parsed file statistics, typically generated -/// by [`expected_stats_schema`]. This schema has the following structure: -/// ```ignore +/// * `stats_schema` - The expected schema for parsed file statistics, typically generated by +/// [`expected_stats_schema`]. This schema has the following structure: +/// +/// ```text /// { -/// numRecords: long, -/// nullCount: , -/// minValues: , -/// maxValues: , +/// numRecords: long, +/// nullCount: , +/// minValues: , +/// maxValues: , /// } /// ``` +/// /// The schema is derived from the table's physical file schema and table properties -/// (`dataSkippingNumIndexedCols`, `dataSkippingStatsColumns`). Only columns eligible -/// for data skipping are included in `minValues`/`maxValues`. +/// (`dataSkippingNumIndexedCols`, `dataSkippingStatsColumns`). Only columns eligible for data +/// skipping are included in `minValues`/`maxValues`. /// -/// * `partition_schema` - The physical partition schema for `partitionValues_parsed`. `None` -/// for non-partitioned tables. +/// * `partition_schema` - The physical partition schema for `partitionValues_parsed`. `None` for +/// non-partitioned tables. /// /// [`expected_stats_schema`]: crate::scan::data_skipping::stats_schema::expected_stats_schema pub(crate) fn build_checkpoint_transform( config: &StatsTransformConfig, + read_schema: &StructType, stats_schema: &SchemaRef, partition_schema: Option<&SchemaRef>, -) -> ExpressionRef { - let mut add_transform = Transform::new_nested([ADD_NAME]); +) -> DeltaResult<(SchemaRef, ExpressionRef)> { + let mut patch_builder = ProjectionStructPatchBuilder::new(read_schema); // Handle stats field if config.write_stats_as_json { - // Populate stats from stats_parsed if needed (for old checkpoints that only had stats_parsed) - add_transform = add_transform.with_replaced_field(STATS_FIELD, STATS_JSON_EXPR.clone()); + // Populate stats from stats_parsed if needed (for old checkpoints that only had + // stats_parsed) + patch_builder = + patch_builder.replace_expr_at([ADD_NAME], STATS_FIELD, STATS_JSON_EXPR.clone()); } else { // Drop stats field when not writing as JSON - add_transform = add_transform.with_dropped_field(STATS_FIELD); + patch_builder = patch_builder.drop_at([ADD_NAME], STATS_FIELD); } // Handle stats_parsed field @@ -98,30 +104,29 @@ pub(crate) fn build_checkpoint_transform( if config.write_stats_as_struct { // Populate stats_parsed from JSON stats (for commits that only have JSON stats) let stats_parsed_expr = build_stats_parsed_expr(stats_schema); - add_transform = add_transform.with_replaced_field(STATS_PARSED_FIELD, stats_parsed_expr); + patch_builder = + patch_builder.replace_expr_at([ADD_NAME], STATS_PARSED_FIELD, stats_parsed_expr); } else { // Drop stats_parsed field when not writing as struct - add_transform = add_transform.with_dropped_field(STATS_PARSED_FIELD); + patch_builder = patch_builder.drop_at([ADD_NAME], STATS_PARSED_FIELD); } // Handle partitionValues_parsed field (only for partitioned tables) if partition_schema.is_some() { if config.write_stats_as_struct { let pv_parsed_expr = build_partition_values_parsed_expr(); - add_transform = - add_transform.with_replaced_field(PARTITION_VALUES_PARSED_FIELD, pv_parsed_expr); + patch_builder = patch_builder.replace_expr_at( + [ADD_NAME], + PARTITION_VALUES_PARSED_FIELD, + pv_parsed_expr, + ); } else { // Drop partitionValues_parsed since it was added to read schema - add_transform = add_transform.with_dropped_field(PARTITION_VALUES_PARSED_FIELD); + patch_builder = patch_builder.drop_at([ADD_NAME], PARTITION_VALUES_PARSED_FIELD); } } - // Wrap the nested Add transform in a top-level transform that replaces the Add field - let add_transform_expr: ExpressionRef = Arc::new(Expression::transform(add_transform)); - let outer_transform = - Transform::new_top_level().with_replaced_field(ADD_NAME, add_transform_expr); - - Arc::new(Expression::transform(outer_transform)) + patch_builder.build() } /// Builds a read schema that includes `stats_parsed` and optionally `partitionValues_parsed` @@ -155,43 +160,18 @@ pub(crate) fn build_checkpoint_read_schema( "partitionValues_parsed field already exists in Add schema", )); } - let mut result = add_struct.clone().with_field_inserted_after( - Some(STATS_FIELD), - StructField::nullable( - STATS_PARSED_FIELD, - DataType::Struct(Box::new(stats_schema.clone())), - ), - )?; - if let Some(pv_schema) = partition_schema { - result = result.with_field_inserted_after( - Some(PARTITION_VALUES_FIELD), - StructField::nullable( - PARTITION_VALUES_PARSED_FIELD, - DataType::Struct(Box::new(pv_schema.clone())), - ), - )?; - } - Ok(result) - }) -} - -/// Builds the output schema based on configuration. -/// -/// The output schema determines which fields are included in the checkpoint: -/// - If `writeStatsAsJson=false`: `stats` field is excluded -/// - If `writeStatsAsStruct=true`: `stats_parsed` and `partitionValues_parsed` fields are included -/// -/// # Errors -/// -/// Returns an error if the `add` field is not found or is not a struct type. -pub(crate) fn build_checkpoint_output_schema( - config: &StatsTransformConfig, - base_schema: &StructType, - stats_schema: &StructType, - partition_schema: Option<&StructType>, -) -> DeltaResult { - transform_add_schema(base_schema, |add_struct| { - build_add_output_schema(config, add_struct, stats_schema, partition_schema) + SchemaStructPatchBuilder::new() + .insert_after( + STATS_FIELD, + StructField::nullable(STATS_PARSED_FIELD, stats_schema.clone()), + ) + .fold_with(partition_schema, |patch, pv_schema| { + patch.insert_after( + PARTITION_VALUES_FIELD, + StructField::nullable(PARTITION_VALUES_PARSED_FIELD, pv_schema.clone()), + ) + }) + .build(add_struct) }) } @@ -205,15 +185,11 @@ pub(crate) fn build_checkpoint_output_schema( /// If `stats_parsed` is non-null, the data originated from a checkpoint (commits only /// contain JSON stats, so `stats_parsed` will be null for commit-sourced rows). /// -/// Column paths are relative to the full batch (not the nested Add struct), so we use -/// ["add", "stats"] instead of just ["stats"]. +/// Column paths are relative to the full batch, not the nested Add struct. fn build_stats_parsed_expr(stats_schema: &SchemaRef) -> ExpressionRef { Arc::new(Expression::coalesce([ - Expression::column([ADD_NAME, STATS_PARSED_FIELD]), - Expression::parse_json( - Expression::column([ADD_NAME, STATS_FIELD]), - stats_schema.clone(), - ), + col!(ADD_NAME, STATS_PARSED_FIELD), + Expression::parse_json(col!(ADD_NAME, STATS_FIELD), stats_schema.clone()), ])) } @@ -226,26 +202,32 @@ fn build_stats_parsed_expr(stats_schema: &SchemaRef) -> ExpressionRef { /// itself carries no schema, so the expression evaluator uses the expected output type to /// parse each string value into the correct native type. /// -/// Column paths are relative to the full batch (not the nested Add struct), so we use -/// `["add", "partitionValues"]` instead of just `["partitionValues"]`. +/// The fallback uses the same `MAP_TO_STRUCT` the scan applies, so an empty-string partition value +/// (only ever a foreign `""`, since kernel serializes its own empty and null partition values to +/// JSON null on write) reconstructs into the checkpoint identically to how the scan reconstructs it +/// from a commit. +/// +/// Column paths are relative to the full batch, not the nested Add struct. fn build_partition_values_parsed_expr() -> ExpressionRef { Arc::new(Expression::coalesce([ - Expression::column([ADD_NAME, PARTITION_VALUES_PARSED_FIELD]), - Expression::map_to_struct(Expression::column([ADD_NAME, PARTITION_VALUES_FIELD])), + col!(ADD_NAME, PARTITION_VALUES_PARSED_FIELD), + Expression::map_to_struct( + col!(ADD_NAME, PARTITION_VALUES_FIELD), + MapToStructOptions::default(), + ), ])) } /// Static expression: `stats = COALESCE(stats, ToJson(stats_parsed))` /// /// This expression prefers existing JSON stats, falling back to converting stats_parsed. -/// Column paths are relative to the full batch (not the nested Add struct), so we use -/// ["add", "stats"] instead of just ["stats"]. +/// Column paths are relative to the full batch, not the nested Add struct. static STATS_JSON_EXPR: LazyLock = LazyLock::new(|| { Arc::new(Expression::coalesce([ - Expression::column([ADD_NAME, STATS_FIELD]), + col!(ADD_NAME, STATS_FIELD), Expression::unary( UnaryExpressionOp::ToJson, - Expression::column([ADD_NAME, STATS_PARSED_FIELD]), + col!(ADD_NAME, STATS_PARSED_FIELD), ), ])) }); @@ -255,9 +237,6 @@ static STATS_JSON_EXPR: LazyLock = LazyLock::new(|| { /// This helper applies a transformation function to the Add struct and returns /// a new schema with the modified Add field. /// -// TODO(https://github.com/delta-io/delta-kernel-rs/issues/1820): Replace manual field -// iteration with StructType helper methods (e.g., with_field_inserted, with_field_removed). -/// /// # Errors /// /// Returns an error if: @@ -280,55 +259,23 @@ fn transform_add_schema( }; let modified_add = transform_fn(add_struct)?; - let new_schema = base_schema.clone().with_field_replaced( - ADD_NAME, - StructField { - name: ADD_NAME.to_string(), - data_type: DataType::Struct(Box::new(modified_add)), - nullable: add_field.nullable, - metadata: add_field.metadata.clone(), - }, - )?; + let new_add_field = StructField::new(ADD_NAME, modified_add, add_field.nullable) + .with_metadata(add_field.metadata.clone()); + let new_schema = SchemaStructPatchBuilder::new() + .replace(ADD_NAME, new_add_field) + .build(base_schema)?; Ok(Arc::new(new_schema)) } -fn build_add_output_schema( - config: &StatsTransformConfig, - add_schema: &StructType, - stats_schema: &StructType, - partition_schema: Option<&StructType>, -) -> DeltaResult { - let mut new_schema = add_schema.clone(); - if config.write_stats_as_struct { - new_schema = new_schema.with_field_inserted_after( - Some(STATS_FIELD), - StructField::nullable( - STATS_PARSED_FIELD, - DataType::Struct(Box::new(stats_schema.clone())), - ), - )?; - if let Some(pv_schema) = partition_schema { - new_schema = new_schema.with_field_inserted_after( - Some(PARTITION_VALUES_FIELD), - StructField::nullable( - PARTITION_VALUES_PARSED_FIELD, - DataType::Struct(Box::new(pv_schema.clone())), - ), - )?; - } - } - - if config.write_stats_as_json { - Ok(new_schema) - } else { - Ok(new_schema.with_field_removed(STATS_FIELD)) - } -} - #[cfg(test)] mod tests { + use rstest::rstest; + use super::*; + use crate::actions::NUM_RECORDS; + use crate::expressions::ExpressionStructPatch; + use crate::schema::MapType; #[test] fn test_config_defaults() { @@ -350,76 +297,103 @@ mod tests { assert!(config.write_stats_as_struct); } - /// Helper to extract the outer and inner transforms from a stats transform expression. - /// Returns (outer_transform, inner_transform). - fn extract_transforms(expr: &Expression) -> (&Transform, &Transform) { - let Expression::Transform(outer) = expr else { - panic!("Expected outer Transform expression"); + /// Helper to extract the outer and inner patches from a stats transform expression. + /// Returns (outer_patch, inner_patch). + fn extract_patches(expr: &Expression) -> (&ExpressionStructPatch, &ExpressionStructPatch) { + let Expression::StructPatch(outer) = expr else { + panic!("Expected outer StructPatch expression"); }; // Outer should be top-level (no input path) assert!( outer.input_path.is_none(), - "Outer transform should be top-level" + "Outer patch should be top-level" ); // Outer should replace "add" field - let add_field_transform = outer - .field_transforms + let add_field_patch = outer + .field_patches .get(ADD_NAME) - .expect("Outer transform should have 'add' field transform"); - assert!(add_field_transform.is_replace, "Should replace 'add' field"); - assert_eq!( - add_field_transform.exprs.len(), - 1, - "Should have exactly one replacement expression" + .expect("Outer patch should have 'add' field patch"); + assert!( + !add_field_patch.keep_input && !add_field_patch.insertions.is_empty(), + "Should replace 'add' field" ); - // Extract inner transform - let Expression::Transform(inner) = add_field_transform.exprs[0].as_ref() else { - panic!("Expected inner Transform expression for 'add' field"); + // Extract inner patch + let Expression::StructPatch(inner) = add_field_patch.insertions[0].as_ref() else { + panic!("Expected inner StructPatch expression for 'add' field"); }; // Inner should target "add" path assert_eq!( inner.input_path.as_ref().map(|p| p.to_string()), Some("add".to_string()), - "Inner transform should target 'add' path" + "Inner patch should target 'add' path" ); (outer, inner) } - /// Helper to check if a field transform is a drop (replace with nothing). - fn is_drop(transform: &Transform, field: &str) -> bool { - transform - .field_transforms + /// Helper to check if a field patch is a drop (replace with nothing). + fn is_drop(patch: &ExpressionStructPatch, field: &str) -> bool { + patch + .field_patches .get(field) - .map(|ft| ft.is_replace && ft.exprs.is_empty()) - .unwrap_or(false) + .is_some_and(|ft| !ft.keep_input && ft.insertions.is_empty()) } - /// Helper to check if a field transform is a replacement with an expression. - fn is_replacement(transform: &Transform, field: &str) -> bool { - transform - .field_transforms + /// Helper to check if a field patch is a replacement with an expression. + fn is_replacement(patch: &ExpressionStructPatch, field: &str) -> bool { + patch + .field_patches .get(field) - .map(|ft| ft.is_replace && ft.exprs.len() == 1) - .unwrap_or(false) + .is_some_and(|ft| !ft.keep_input && !ft.insertions.is_empty()) + } + + fn build_checkpoint_transform( + config: &StatsTransformConfig, + stats_schema: &SchemaRef, + partition_schema: Option<&SchemaRef>, + ) -> (SchemaRef, ExpressionRef) { + let base_schema = StructType::new_unchecked([StructField::nullable( + ADD_NAME, + add_schema(partition_schema.is_some()), + )]); + let read_schema = build_checkpoint_read_schema( + &base_schema, + stats_schema.as_ref(), + partition_schema.map(AsRef::as_ref), + ) + .expect("build checkpoint read schema should produce a valid schema"); + super::build_checkpoint_transform(config, &read_schema, stats_schema, partition_schema) + .expect("build checkpoint transform should produce a valid schema and expression") + } + + fn add_schema(with_partition_schema: bool) -> StructType { + let fields = [ + Some(StructField::not_null("path", DataType::STRING)), + with_partition_schema.then(|| { + let partition_values = MapType::new(DataType::STRING, DataType::STRING, true); + StructField::nullable(PARTITION_VALUES_FIELD, partition_values) + }), + Some(StructField::nullable(STATS_FIELD, DataType::STRING)), + ]; + StructType::new_unchecked(fields.into_iter().flatten()) } #[test] fn test_build_transform_with_json_only() { // writeStatsAsJson=true, writeStatsAsStruct=false (default) - // Inner transform: stats=COALESCE, stats_parsed=drop + // Inner patch: stats=COALESCE, stats_parsed=drop let config = StatsTransformConfig { write_stats_as_json: true, write_stats_as_struct: false, }; let stats_schema = Arc::new(StructType::new_unchecked([])); - let transform_expr = build_checkpoint_transform(&config, &stats_schema, None); + let (_, transform_expr) = build_checkpoint_transform(&config, &stats_schema, None); - let (_, inner) = extract_transforms(&transform_expr); + let (_, inner) = extract_patches(&transform_expr); // stats should be replaced with COALESCE expression assert!( @@ -435,40 +409,21 @@ mod tests { } #[test] - fn test_build_transform_drops_both_when_false() { - // writeStatsAsJson=false, writeStatsAsStruct=false - // Inner transform: stats=drop, stats_parsed=drop - let config = StatsTransformConfig { - write_stats_as_json: false, - write_stats_as_struct: false, - }; - let stats_schema = Arc::new(StructType::new_unchecked([])); - let transform_expr = build_checkpoint_transform(&config, &stats_schema, None); - - let (_, inner) = extract_transforms(&transform_expr); - - // Both fields should be dropped - assert!(is_drop(inner, STATS_FIELD), "stats should be dropped"); - assert!( - is_drop(inner, STATS_PARSED_FIELD), - "stats_parsed should be dropped" - ); - } - - #[test] - fn test_build_transform_with_both_enabled() { - // writeStatsAsJson=true, writeStatsAsStruct=true - // Inner transform: stats=COALESCE, stats_parsed=COALESCE + fn test_build_transform_populates_struct_stats_and_partition_values() { let config = StatsTransformConfig { write_stats_as_json: true, write_stats_as_struct: true, }; let stats_schema = Arc::new(StructType::new_unchecked([])); - let transform_expr = build_checkpoint_transform(&config, &stats_schema, None); + let pv_schema = Arc::new(StructType::new_unchecked([ + StructField::nullable("year", DataType::INTEGER), + StructField::nullable("month", DataType::INTEGER), + ])); + let (_, transform_expr) = + build_checkpoint_transform(&config, &stats_schema, Some(&pv_schema)); - let (_, inner) = extract_transforms(&transform_expr); + let (_, inner) = extract_patches(&transform_expr); - // Both fields should be replaced with COALESCE expressions assert!( is_replacement(inner, STATS_FIELD), "stats should be replaced" @@ -477,266 +432,77 @@ mod tests { is_replacement(inner, STATS_PARSED_FIELD), "stats_parsed should be replaced" ); - } - - #[test] - fn test_build_transform_struct_only() { - // writeStatsAsJson=false, writeStatsAsStruct=true - // Inner transform: stats=drop, stats_parsed=COALESCE - let config = StatsTransformConfig { - write_stats_as_json: false, - write_stats_as_struct: true, - }; - let stats_schema = Arc::new(StructType::new_unchecked([])); - let transform_expr = build_checkpoint_transform(&config, &stats_schema, None); - - let (_, inner) = extract_transforms(&transform_expr); - - // stats should be dropped - assert!(is_drop(inner, STATS_FIELD), "stats should be dropped"); - - // stats_parsed should be replaced with COALESCE expression - assert!( - is_replacement(inner, STATS_PARSED_FIELD), - "stats_parsed should be replaced" - ); - } - - #[test] - fn test_build_transform_with_partition_values() { - // writeStatsAsStruct=true with partitioned table - let config = StatsTransformConfig { - write_stats_as_json: true, - write_stats_as_struct: true, - }; - let stats_schema = Arc::new(StructType::new_unchecked([])); - let pv_schema = Arc::new(StructType::new_unchecked([ - StructField::nullable("year", DataType::INTEGER), - StructField::nullable("month", DataType::INTEGER), - ])); - let transform_expr = build_checkpoint_transform(&config, &stats_schema, Some(&pv_schema)); - - let (_, inner) = extract_transforms(&transform_expr); - - // partitionValues_parsed should be replaced with COALESCE expression assert!( is_replacement(inner, PARTITION_VALUES_PARSED_FIELD), "partitionValues_parsed should be replaced" ); } + /// The checkpoint falls back to `MAP_TO_STRUCT` over `partitionValues` when no native + /// `partitionValues_parsed` column is present, so a checkpoint reconstructs the same typed + /// struct the scan reads and the two can never disagree on a value. #[test] - fn test_build_transform_no_partition_values_when_struct_disabled() { - // writeStatsAsStruct=false with partitioned table - let config = StatsTransformConfig { - write_stats_as_json: true, - write_stats_as_struct: false, + fn build_partition_values_parsed_expr_falls_back_to_map_to_struct() { + let expr = build_partition_values_parsed_expr(); + let Expression::Variadic(coalesce) = expr.as_ref() else { + panic!("expected a COALESCE expression"); }; - let stats_schema = Arc::new(StructType::new_unchecked([])); - let pv_schema = Arc::new(StructType::new_unchecked([StructField::nullable( - "year", - DataType::INTEGER, - )])); - let transform_expr = build_checkpoint_transform(&config, &stats_schema, Some(&pv_schema)); - - let (_, inner) = extract_transforms(&transform_expr); - - // partitionValues_parsed should be dropped + let has_map_to_struct_fallback = coalesce + .exprs + .iter() + .any(|e| matches!(e, Expression::MapToStruct(_))); assert!( - is_drop(inner, PARTITION_VALUES_PARSED_FIELD), - "partitionValues_parsed should be dropped" + has_map_to_struct_fallback, + "checkpoint partitionValues_parsed must reconstruct via MAP_TO_STRUCT" ); } - #[test] - fn test_build_transform_non_partitioned_table() { - // Non-partitioned table: no partitionValues_parsed handling at all + #[rstest] + #[case::no_stats(false, false, false, &["path"])] + #[case::json_only(true, false, false, &["path", "stats"])] + #[case::struct_only(false, true, false, &["path", "stats_parsed"])] + #[case::json_and_struct(true, true, false, &["path", "stats", "stats_parsed"])] + #[case::with_partition_values( + true, + true, + true, + &["path", "partitionValues", "partitionValues_parsed", "stats", "stats_parsed"])] + #[case::partition_values_struct_disabled( + true, + false, + true, + &["path", "partitionValues", "stats"])] + fn test_build_add_output_schema( + #[case] write_stats_as_json: bool, + #[case] write_stats_as_struct: bool, + #[case] with_partition_schema: bool, + #[case] expected_names: &[&str], + ) { let config = StatsTransformConfig { - write_stats_as_json: true, - write_stats_as_struct: true, + write_stats_as_json, + write_stats_as_struct, }; - let stats_schema = Arc::new(StructType::new_unchecked([])); - let transform_expr = build_checkpoint_transform(&config, &stats_schema, None); - - let (_, inner) = extract_transforms(&transform_expr); - - // No partitionValues_parsed transform should exist - assert!( - !inner - .field_transforms - .contains_key(PARTITION_VALUES_PARSED_FIELD), - "non-partitioned table should not have partitionValues_parsed transform" - ); - } - - #[test] - fn test_field_inserted_after_in_add_schema() { - let add_schema = StructType::new_unchecked([ - StructField::not_null("path", DataType::STRING), - StructField::nullable("stats", DataType::STRING), - StructField::nullable("tags", DataType::STRING), - ]); - - let injected_schema = - StructType::new_unchecked([StructField::nullable("numRecords", DataType::LONG)]); - - let result = add_schema - .with_field_inserted_after( - Some(STATS_FIELD), - StructField::nullable( - STATS_PARSED_FIELD, - DataType::Struct(Box::new(injected_schema)), - ), - ) - .expect("inserting stats_parsed should succeed"); - - // Should have 4 fields: path, stats, stats_parsed, tags - assert_eq!(result.fields().count(), 4); - - let field_names: Vec<&str> = result.fields().map(|f| f.name.as_str()).collect(); - assert_eq!(field_names, vec!["path", "stats", "stats_parsed", "tags"]); - } - - #[test] - fn test_build_add_output_schema_json_only() { - let config = StatsTransformConfig { - write_stats_as_json: true, - write_stats_as_struct: false, + let num_records = StructField::nullable(NUM_RECORDS, DataType::LONG); + let stats_schema = Arc::new(StructType::new_unchecked([num_records])); + let pv_schema = with_partition_schema.then(|| { + Arc::new(StructType::new_unchecked([ + StructField::nullable("year", DataType::INTEGER), + StructField::nullable("month", DataType::INTEGER), + ])) + }); + + let (output_schema, _) = + build_checkpoint_transform(&config, &stats_schema, pv_schema.as_ref()); + let add_field = output_schema + .field(ADD_NAME) + .expect("Expected output schema to contain add field"); + let DataType::Struct(add_schema) = add_field.data_type() else { + panic!("Expected output add field to be a struct"); }; - - let add_schema = StructType::new_unchecked([ - StructField::not_null("path", DataType::STRING), - StructField::nullable("stats", DataType::STRING), - ]); - - let stats_schema = StructType::new_unchecked([]); - - let result = build_add_output_schema(&config, &add_schema, &stats_schema, None) - .expect("build add output schema should produce a valid schema"); - - // Should have path and stats, no stats_parsed - let field_names: Vec<&str> = result.fields().map(|f| f.name.as_str()).collect(); - assert_eq!(field_names, vec!["path", "stats"]); - } - - #[test] - fn test_build_add_output_schema_struct_only() { - let config = StatsTransformConfig { - write_stats_as_json: false, - write_stats_as_struct: true, - }; - - let add_schema = StructType::new_unchecked([ - StructField::not_null("path", DataType::STRING), - StructField::nullable("stats", DataType::STRING), - ]); - - let stats_schema = - StructType::new_unchecked([StructField::nullable("numRecords", DataType::LONG)]); - - let result = build_add_output_schema(&config, &add_schema, &stats_schema, None) - .expect("build add output schema should produce a valid schema"); - - // Should have path and stats_parsed (stats dropped) - let field_names: Vec<&str> = result.fields().map(|f| f.name.as_str()).collect(); - assert_eq!(field_names, vec!["path", "stats_parsed"]); - } - - #[test] - fn test_build_add_output_schema_both() { - let config = StatsTransformConfig { - write_stats_as_json: true, - write_stats_as_struct: true, - }; - - let add_schema = StructType::new_unchecked([ - StructField::not_null("path", DataType::STRING), - StructField::nullable("stats", DataType::STRING), - ]); - - let stats_schema = - StructType::new_unchecked([StructField::nullable("numRecords", DataType::LONG)]); - - let result = build_add_output_schema(&config, &add_schema, &stats_schema, None) - .expect("build add output schema should produce a valid schema"); - - // Should have path, stats, and stats_parsed - let field_names: Vec<&str> = result.fields().map(|f| f.name.as_str()).collect(); - assert_eq!(field_names, vec!["path", "stats", "stats_parsed"]); - } - - #[test] - fn test_build_add_output_schema_with_partition_values() { - let config = StatsTransformConfig { - write_stats_as_json: true, - write_stats_as_struct: true, - }; - - let add_schema = StructType::new_unchecked([ - StructField::not_null("path", DataType::STRING), - StructField::nullable( - "partitionValues", - DataType::Map(Box::new(crate::schema::MapType::new( - DataType::STRING, - DataType::STRING, - true, - ))), - ), - StructField::nullable("stats", DataType::STRING), - ]); - - let stats_schema = - StructType::new_unchecked([StructField::nullable("numRecords", DataType::LONG)]); - let pv_schema = StructType::new_unchecked([ - StructField::nullable("year", DataType::INTEGER), - StructField::nullable("month", DataType::INTEGER), - ]); - - let result = build_add_output_schema(&config, &add_schema, &stats_schema, Some(&pv_schema)) - .expect("build add output schema should produce a valid schema"); - - let field_names: Vec<&str> = result.fields().map(|f| f.name.as_str()).collect(); - assert_eq!( - field_names, - vec![ - "path", - "partitionValues", - "partitionValues_parsed", - "stats", - "stats_parsed" - ] - ); - } - - #[test] - fn test_build_add_output_schema_no_partition_values_when_struct_disabled() { - let config = StatsTransformConfig { - write_stats_as_json: true, - write_stats_as_struct: false, - }; - - let add_schema = StructType::new_unchecked([ - StructField::not_null("path", DataType::STRING), - StructField::nullable( - "partitionValues", - DataType::Map(Box::new(crate::schema::MapType::new( - DataType::STRING, - DataType::STRING, - true, - ))), - ), - StructField::nullable("stats", DataType::STRING), - ]); - - let stats_schema = StructType::new_unchecked([]); - let pv_schema = - StructType::new_unchecked([StructField::nullable("year", DataType::INTEGER)]); - - let result = build_add_output_schema(&config, &add_schema, &stats_schema, Some(&pv_schema)) - .expect("build add output schema should produce a valid schema"); - - let field_names: Vec<&str> = result.fields().map(|f| f.name.as_str()).collect(); - // partitionValues_parsed should NOT be present when writeStatsAsStruct=false - assert_eq!(field_names, vec!["path", "partitionValues", "stats"]); + let field_names: Vec<_> = add_schema + .fields() + .map(|field| field.name.as_str()) + .collect(); + assert_eq!(field_names, expected_names); } } diff --git a/kernel/src/checkpoint/mod.rs b/kernel/src/checkpoint/mod.rs index 6533afa753..5c177968db 100644 --- a/kernel/src/checkpoint/mod.rs +++ b/kernel/src/checkpoint/mod.rs @@ -1,14 +1,16 @@ -//! This module implements the API for writing single-file checkpoints. +//! This module implements the API for **customized** Delta checkpoint writes, where the +//! caller drives the write themselves. The entry point is [`Snapshot::create_checkpoint_writer`]. //! -//! The entry point for this API is [`Snapshot::create_checkpoint_writer`]. +//! If you want an all-in-one API that handles writing the checkpoint, use +//! [`Snapshot::checkpoint`] instead. //! //! ## Checkpoint Types and Selection Logic //! This API supports two checkpoint types, selected based on table features: //! -//! | Table Feature | Resulting Checkpoint Type | Description | -//! |------------------|-------------------------------|-----------------------------------------------------------------------------| -//! | No v2Checkpoints | Single-file Classic-named V1 | Follows V1 specification without [`CheckpointMetadata`] action | -//! | v2Checkpoints | Single-file Classic-named V2 | Follows V2 specification with [`CheckpointMetadata`] action while maintaining backward compatibility via classic naming | +//! | Table Feature | Resulting Checkpoint Type | Description | +//! |------------------|---------------------------------------------|-------------------------------------------------------------------------------------------------------------------------| +//! | No v2Checkpoints | Single-file Classic-named V1 | Follows V1 specification without [`CheckpointMetadata`] action | +//! | v2Checkpoints | Classic-named V2 (with or without sidecars) | Follows V2 specification with [`CheckpointMetadata`] action while maintaining backward compatibility via classic naming | //! //! For more information on the V1/V2 specifications, see the following protocol section: //! @@ -24,10 +26,11 @@ //! //! 1. Create a [`CheckpointWriter`] using [`Snapshot::create_checkpoint_writer`] //! 2. Get the checkpoint path from [`CheckpointWriter::checkpoint_path`] -//! 2. Get the checkpoint data from [`CheckpointWriter::checkpoint_data`] -//! 3. Write the data to the path in object storage (engine-specific) -//! 4. Collect metadata ([`FileMeta`]) from the write operation -//! 5. Pass the metadata and exhausted data iterator to [`CheckpointWriter::finalize`] +//! 3. Get the checkpoint data from [`CheckpointWriter::checkpoint_data`] +//! 4. Write the data to the path in object storage (engine-specific) +//! 5. Collect metadata ([`FileMeta`]) from the write operation +//! 6. Build a [`LastCheckpointHintStats`] from the exhausted iterator state +//! 7. Pass the [`LastCheckpointHintStats`] to [`CheckpointWriter::finalize`] //! //! ```no_run //! # use std::sync::Arc; @@ -40,7 +43,7 @@ //! # use delta_kernel::Error; //! # use delta_kernel::FileMeta; //! # use url::Url; -//! fn write_checkpoint_file(path: Url, data: &mut ActionReconciliationIterator) -> DeltaResult { +//! fn write_checkpoint_file(path: Url, data: ActionReconciliationIterator) -> DeltaResult { //! todo!() /* engine-specific logic to write data to object storage*/ //! } //! @@ -51,22 +54,32 @@ //! let snapshot = Snapshot::builder_for(url).build(engine)?; //! //! // Create a checkpoint writer from the snapshot -//! let mut writer = snapshot.create_checkpoint_writer()?; +//! let writer = snapshot.create_checkpoint_writer(engine)?; //! //! // Get the checkpoint path and data //! let checkpoint_path = writer.checkpoint_path()?; //! let checkpoint_data = writer.checkpoint_data(engine)?; //! -//! // Get the iterator state +//! // Get the iterator state before consuming the data //! let state = checkpoint_data.state(); //! //! // Write the checkpoint data to the object store and collect metadata -//! let metadata: FileMeta = write_checkpoint_file(checkpoint_path, &mut checkpoint_data)?; -//! +//! // The write function consumes the iterator, dropping its Arc reference to the state. +//! let metadata: FileMeta = write_checkpoint_file(checkpoint_path, checkpoint_data)?; //! /* IMPORTANT: All data must be written before finalizing the checkpoint */ //! -//! // Finalize the checkpoint by passing the metadata and state handle -//! writer.finalize(engine, &metadata, &state)?; +//! // Build the [`LastCheckpointHintStats`] from the exhausted iterator state +//! let state = std::sync::Arc::into_inner(state) +//! .ok_or(Error::internal_error("checkpoint state Arc still has other references"))?; +//! let last_checkpoint_stats = +//! delta_kernel::checkpoint::LastCheckpointHintStats::from_reconciliation_state( +//! state, +//! metadata.size, +//! 0, /* num_sidecars */ +//! )?; +//! +//! // Finalize the checkpoint by passing the stats +//! writer.finalize(engine, &last_checkpoint_stats)?; //! //! # Ok::<_, Error>(()) //! ``` @@ -82,12 +95,14 @@ //! in the future, we can revisit this decision. //! //! [`CheckpointMetadata`]: crate::actions::CheckpointMetadata +//! [`FileMeta`]: crate::FileMeta //! [`LastCheckpointHint`]: crate::last_checkpoint_hint::LastCheckpointHint +//! [`Snapshot::checkpoint`]: crate::Snapshot::checkpoint //! [`Snapshot::create_checkpoint_writer`]: crate::Snapshot::create_checkpoint_writer -// Future extensions: -// - TODO(#837): Multi-file V2 checkpoints are not supported yet. The API is designed to be extensible for future -// multi-file support, but the current implementation only supports single-file checkpoints. -use std::sync::{Arc, LazyLock}; +use std::sync::{Arc, LazyLock, Mutex}; + +use tracing::info; +use url::Url; use crate::action_reconciliation::log_replay::{ ActionReconciliationBatch, ActionReconciliationProcessor, @@ -96,33 +111,194 @@ use crate::action_reconciliation::{ ActionReconciliationIterator, ActionReconciliationIteratorState, RetentionCalculator, }; use crate::actions::{ - Add, DomainMetadata, Metadata, Protocol, Remove, SetTransaction, Sidecar, ADD_NAME, - CHECKPOINT_METADATA_NAME, DOMAIN_METADATA_NAME, METADATA_NAME, PROTOCOL_NAME, REMOVE_NAME, - SET_TRANSACTION_NAME, SIDECAR_NAME, + ADD_FIELD, CHECKPOINT_METADATA_NAME, DOMAIN_METADATA_FIELD, METADATA_FIELD, PROTOCOL_FIELD, + REMOVE_FIELD, SET_TRANSACTION_FIELD, SIDECAR_FIELD, }; use crate::engine_data::FilteredEngineData; -use crate::expressions::{Expression, Scalar, StructData, Transform}; +use crate::expressions::{ + lit, Expression, ExpressionRef, ExpressionStructPatchBuilder, Scalar, StructData, +}; use crate::last_checkpoint_hint::LastCheckpointHint; use crate::log_replay::LogReplayProcessor; -use crate::path::ParsedLogPath; -use crate::schema::{DataType, SchemaRef, StructField, StructType, ToSchema as _}; +use crate::path::{self, ParsedLogPath}; +use crate::schema::{lazy_schema_ref, DataType, SchemaRef, StructField, StructType}; use crate::snapshot::SnapshotRef; use crate::table_features::TableFeature; use crate::table_properties::TableProperties; -use crate::{DeltaResult, Engine, EngineData, Error, EvaluationHandlerExtension, FileMeta}; - -use url::Url; +use crate::{ + version_as_i64, DeltaResult, DeltaResultIteratorStatic, Engine, EngineData, Error, + EvaluationHandlerExtension, FileMeta, Version, +}; +#[cfg(feature = "declarative-plans")] +mod checkpoint_shape; mod checkpoint_transform; +mod sidecar; +#[cfg(feature = "declarative-plans")] +#[allow(unused_imports)] +pub(crate) use checkpoint_shape::{CheckpointShape, CheckpointType}; use checkpoint_transform::{ - build_checkpoint_output_schema, build_checkpoint_read_schema, build_checkpoint_transform, - StatsTransformConfig, + build_checkpoint_read_schema, build_checkpoint_transform, StatsTransformConfig, }; - +use sidecar::{create_sidecar_action_batch, SidecarSplitter, SingleSidecarDataIterator}; #[cfg(test)] mod tests; +/// Information about a freshly-written checkpoint. Pass it to +/// [`CheckpointWriter::finalize`] to produce the `_last_checkpoint` hint file. +/// +/// Unlike [`LastCheckpointHint`], which is used on the read path and has many optional +/// fields, this struct focuses on the write path and all fields are required. Note this +/// is a kernel requirement; the Delta protocol itself marks some of these fields as +/// optional in the `_last_checkpoint` hint. See the [Last Checkpoint File Schema] for +/// more details. +/// +/// Construct via [`LastCheckpointHintStats::from_reconciliation_state`]. +/// +/// # Note +/// This is intended for sophisticated connectors that customize how checkpoints are +/// written. If you are not deeply familiar with the Delta protocol, you may use +/// [`Snapshot::checkpoint`](crate::snapshot::Snapshot::checkpoint), which handles +/// checkpoint writing end-to-end. +/// +/// [Last Checkpoint File Schema]: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#last-checkpoint-file-schema +#[derive(Debug)] +pub struct LastCheckpointHintStats { + /// The total number of actions stored in the checkpoint (including actions created + /// outside the reconciliation iterator, e.g. sidecar actions). + num_actions: i64, + /// The total size in bytes of the checkpoint. For V2 checkpoint with sidecars, + /// this is the sum of the main checkpoint file size and all sidecar file sizes. + size_in_bytes: i64, + /// The number of Add-file actions in the checkpoint. + num_of_add_files: i64, +} + +impl LastCheckpointHintStats { + /// Constructs a `LastCheckpointHintStats` from the fully-exhausted reconciliation state. + /// + /// # Parameters + /// - `state`: fully-exhausted reconciliation iterator state. All data must have been written to + /// storage before calling this. + /// - `size_in_bytes`: total byte size of the checkpoint. For V2 checkpoints with sidecars, the + /// caller should include the main checkpoint file size plus all sidecar file sizes. + /// - `num_sidecars`: number of sidecar actions. Use `0` for V1 checkpoints or V2 checkpoints + /// without sidecars. + /// + /// # Errors + /// - If the reconciliation iterator has not been fully exhausted. + /// - If `size_in_bytes` exceeds `i64::MAX`. + /// - If `num_sidecars` exceeds `i64::MAX`. + /// - If `state.actions_count() + num_sidecars` overflows `i64`. + pub fn from_reconciliation_state( + state: ActionReconciliationIteratorState, + size_in_bytes: u64, + num_sidecars: u64, + ) -> DeltaResult { + if !state.is_exhausted() { + return Err(Error::checkpoint_write( + "Cannot build LastCheckpointHintStats: the reconciliation iterator must be fully \ + consumed and all data written to storage before finalizing", + )); + } + let size_in_bytes = i64::try_from(size_in_bytes).map_err(|e| { + Error::checkpoint_write(format!("size_in_bytes {size_in_bytes} exceeds i64: {e}")) + })?; + let num_sidecars_i64 = i64::try_from(num_sidecars).map_err(|e| { + Error::checkpoint_write(format!("num_sidecars {num_sidecars} exceeds i64: {e}")) + })?; + let num_actions = state + .actions_count() + .checked_add(num_sidecars_i64) + .ok_or_else(|| { + Error::checkpoint_write(format!( + "checkpoint action count overflowed i64: {} + {num_sidecars}", + state.actions_count() + )) + })?; + Ok(Self { + num_actions, + size_in_bytes, + num_of_add_files: state.add_actions_count(), + }) + } +} + +#[derive(Debug)] +pub(crate) struct WrittenCheckpointInfo { + /// Metadata of the main checkpoint file. + pub(crate) file_meta: FileMeta, + /// Stats for the `_last_checkpoint` hint. + pub(crate) last_checkpoint_stats: LastCheckpointHintStats, +} + +/// Schemas and configs needed for building the checkpoint read/output schemas. +struct CheckpointSchemaContext { + stats_config: StatsTransformConfig, + /// The checkpoint schema before table-specific fields like + /// `stats_parsed` and `partitionValues_parsed` are injected. + checkpoint_base_schema: SchemaRef, + stats_schema: SchemaRef, + partition_schema: Option, + is_v2: bool, +} + +/// Default value for [`V2CheckpointConfig::WithSidecar::file_actions_per_sidecar_hint`]. +/// It's the suggested upper bound of file actions (`add` and `remove`) per sidecar file when +/// the caller does not provide an explicit hint. +pub const DEFAULT_FILE_ACTIONS_PER_SIDECAR_HINT: usize = 50_000; + +/// Specifies the checkpoint format and behavior. +#[derive(Debug)] +pub enum CheckpointSpec { + /// Write a checkpoint following the V1 spec, the original checkpoint format, without + /// sidecar files or checkpoint metadata. See [V1 spec] for more details. + /// + /// [V1 spec]: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#v1-spec + V1, + /// Write a checkpoint following the V2 spec, which allows putting file actions (`add` + /// and `remove`) in sidecar files. Requires the `v2Checkpoint` reader/writer feature. + /// See [V2 spec] and [`V2CheckpointConfig::WithSidecar`] for more details. + /// + /// [V2 spec]: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#v2-spec + V2(V2CheckpointConfig), +} + +/// Configuration for V2 checkpoints. +/// +/// Note: "File actions" here means `add` and `remove` actions. "Non-file actions" means +/// the rest (`protocol`, `metaData`, `txn`, etc.). +#[derive(Debug)] +pub enum V2CheckpointConfig { + /// Write a V2 checkpoint without sidecar files. + NoSidecar, + /// Write a V2 checkpoint with file actions split into [Sidecar Files]. A main + /// checkpoint file is written, with one `sidecar` action pointing to each sidecar file. + /// + /// # Benefits of Sidecars + /// - **Read parallelism**: readers can fetch sidecars in parallel. + /// - **Smaller main checkpoint**: callers that only need non-file actions (e.g. `protocol`, + /// `metaData`) can skip the sidecars entirely. + /// + /// # Note + /// Sidecars add extra write cost (one parquet file per sidecar). + /// + /// [Sidecar Files]: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#sidecar-files + WithSidecar { + /// Suggested number of file actions per sidecar file. When there are X file actions, + /// the number of sidecars will roughly be `X / file_actions_per_sidecar_hint`. + /// + /// This is a hint, not a strict limit, because file actions are stored in `EngineData` + /// batches that cannot be split. For example, if the hint is 99 but a single + /// `EngineData` batch contains 100 file actions, all 100 will be written to one sidecar. + /// + /// When `None`, kernel uses [`DEFAULT_FILE_ACTIONS_PER_SIDECAR_HINT`] (50,000) as the + /// default. + file_actions_per_sidecar_hint: Option, + }, +} + /// Schema of the `_last_checkpoint` file /// We cannot use `LastCheckpointInfo::to_schema()` as it would include the 'checkpoint_schema' /// field, which is only known at runtime. @@ -138,21 +314,21 @@ static LAST_CHECKPOINT_SCHEMA: LazyLock = LazyLock::new(|| { }); /// Action fields shared by V1 and V2 checkpoint schemas. -fn base_checkpoint_action_fields() -> Vec { - vec![ - StructField::nullable(ADD_NAME, Add::to_schema()), - StructField::nullable(REMOVE_NAME, Remove::to_schema()), - StructField::nullable(METADATA_NAME, Metadata::to_schema()), - StructField::nullable(PROTOCOL_NAME, Protocol::to_schema()), - StructField::nullable(SET_TRANSACTION_NAME, SetTransaction::to_schema()), - StructField::nullable(DOMAIN_METADATA_NAME, DomainMetadata::to_schema()), - StructField::nullable(SIDECAR_NAME, Sidecar::to_schema()), +fn base_checkpoint_action_fields() -> [&'static LazyLock; 7] { + [ + &ADD_FIELD, + &REMOVE_FIELD, + &METADATA_FIELD, + &PROTOCOL_FIELD, + &SET_TRANSACTION_FIELD, + &DOMAIN_METADATA_FIELD, + &SIDECAR_FIELD, ] } /// Schema for V1 checkpoints (without checkpointMetadata action) static CHECKPOINT_ACTIONS_SCHEMA_V1: LazyLock = - LazyLock::new(|| Arc::new(StructType::new_unchecked(base_checkpoint_action_fields()))); + lazy_schema_ref! { ..(base_checkpoint_action_fields()) }; /// Schema for the checkpointMetadata field in V2 checkpoints. /// We cannot use `CheckpointMetadata::to_schema()` as it would include the 'tags' field which @@ -165,11 +341,10 @@ fn checkpoint_metadata_field() -> StructField { } /// Schema for V2 checkpoints (includes checkpointMetadata action) -static CHECKPOINT_ACTIONS_SCHEMA_V2: LazyLock = LazyLock::new(|| { - let mut fields = base_checkpoint_action_fields(); - fields.push(checkpoint_metadata_field()); - Arc::new(StructType::new_unchecked(fields)) -}); +static CHECKPOINT_ACTIONS_SCHEMA_V2: LazyLock = lazy_schema_ref! { + ..(base_checkpoint_action_fields()), + (checkpoint_metadata_field()), +}; /// Orchestrates the process of creating a checkpoint for a table. /// @@ -178,8 +353,8 @@ static CHECKPOINT_ACTIONS_SCHEMA_V2: LazyLock = LazyLock::new(|| { /// supports the `v2Checkpoints` reader/writer feature. /// /// # Warning -/// The checkpoint data must be fully written to storage before calling [`CheckpointWriter::finalize`]. -/// Failing to do so may result in data loss or corruption. +/// The checkpoint data must be fully written to storage before calling +/// [`CheckpointWriter::finalize`]. Failing to do so may result in data loss or corruption. /// /// # See Also /// See the [module-level documentation](self) for the complete checkpoint workflow @@ -192,6 +367,11 @@ pub struct CheckpointWriter { /// Note: Although the version is stored as a u64 in the snapshot, it is stored as an i64 /// field here to avoid multiple type conversions. version: i64, + + is_v2: bool, + read_schema: SchemaRef, + output_schema: SchemaRef, + transform_expr: ExpressionRef, } impl RetentionCalculator for CheckpointWriter { @@ -202,21 +382,34 @@ impl RetentionCalculator for CheckpointWriter { impl CheckpointWriter { /// Creates a new [`CheckpointWriter`] for the given snapshot. - pub(crate) fn try_new(snapshot: SnapshotRef) -> DeltaResult { - let version = i64::try_from(snapshot.version()).map_err(|e| { - Error::CheckpointWrite(format!( - "Failed to convert checkpoint version from u64 {} to i64: {}", - snapshot.version(), - e - )) - })?; - + pub(crate) fn try_new(snapshot: SnapshotRef, engine: &dyn Engine) -> DeltaResult { // We disallow checkpointing if the Snapshot is not published. If we didn't, this could // create gaps in the version history, thereby breaking old readers. snapshot.log_segment().validate_published()?; - Ok(Self { snapshot, version }) + let schema_context = Self::checkpoint_schema_context(&snapshot, engine)?; + let read_schema = build_checkpoint_read_schema( + &schema_context.checkpoint_base_schema, + &schema_context.stats_schema, + schema_context.partition_schema.as_deref(), + )?; + let (output_schema, transform_expr) = build_checkpoint_transform( + &schema_context.stats_config, + &read_schema, + &schema_context.stats_schema, + schema_context.partition_schema.as_ref(), + )?; + + Ok(Self { + version: version_as_i64(snapshot.version())?, + snapshot, + is_v2: schema_context.is_v2, + read_schema, + output_schema, + transform_expr, + }) } + /// Returns the URL where the checkpoint file should be written. /// /// This method generates the checkpoint path based on the table's root and the version @@ -234,6 +427,7 @@ impl CheckpointWriter { ) .map(|parsed| parsed.location) } + /// Returns the checkpoint data to be written to the checkpoint file. /// /// This method reads actions from the log segment, processes them for checkpoint creation, @@ -243,7 +437,7 @@ impl CheckpointWriter { /// /// The returned [`ActionReconciliationIterator`] yields [`FilteredEngineData`] batches with /// stats transforms already applied. Use [`ActionReconciliationIterator::state`] to get the - /// shared state for passing to [`CheckpointWriter::finalize`]. + /// shared state for building a [`LastCheckpointHintStats`] after the iterator is exhausted. /// /// # Engine Usage /// @@ -254,7 +448,12 @@ impl CheckpointWriter { /// let data = batch?.apply_selection_vector()?; /// parquet_writer.write(&data).await?; /// } - /// writer.finalize(&engine, &metadata, &state)?; + /// drop(checkpoint_data); + /// let state = Arc::into_inner(state) + /// .ok_or(Error::internal_error("checkpoint state Arc still has other references"))?; + /// let last_checkpoint_stats = + /// LastCheckpointHintStats::from_reconciliation_state(state, size_in_bytes, 0)?; + /// writer.finalize(&engine, &last_checkpoint_stats)?; /// ``` // Implementation overview: // 1. Determines whether to write a V1 or V2 checkpoint based on `v2Checkpoints` feature @@ -266,54 +465,11 @@ impl CheckpointWriter { &self, engine: &dyn Engine, ) -> DeltaResult { - let config = StatsTransformConfig::from_table_properties(self.snapshot.table_properties()); - - // Get clustering columns so they are always included in stats per the Delta protocol. - let tc = self.snapshot.table_configuration(); - let clustering_columns_physical = self.snapshot.get_clustering_columns_physical(engine)?; - - // Get stats schema from table configuration. - // This already excludes partition columns and applies column mapping. - let stats_schema = tc - .build_expected_stats_schemas(clustering_columns_physical.as_deref(), None)? - .physical; - - // Select schema based on V2 checkpoint support - let is_v2_checkpoints_supported = self - .snapshot - .table_configuration() - .is_feature_supported(&TableFeature::V2Checkpoint); - - let base_schema = if is_v2_checkpoints_supported { - &CHECKPOINT_ACTIONS_SCHEMA_V2 - } else { - &CHECKPOINT_ACTIONS_SCHEMA_V1 - }; - - // Build partition schema for partitionValues_parsed (None for non-partitioned tables) - let partition_schema = self - .snapshot - .table_configuration() - .build_partition_values_parsed_schema(); - - // The read schema and output schema differ because the transform needs access to - // both stats formats as input, but may only write one format as output. - // - // read_schema: Always includes both `stats` and `stats_parsed` fields in the Add - // action, so COALESCE expressions can read from either source. For commit files, - // `stats_parsed` doesn't exist and is read as nulls. For partitioned tables, - // `partitionValues_parsed` is also included. - // - // output_schema: Only includes the stats fields that the table config requests - // (e.g., only `stats` if writeStatsAsJson=true and writeStatsAsStruct=false). - let read_schema = - build_checkpoint_read_schema(base_schema, &stats_schema, partition_schema.as_deref())?; - // Read actions from log segment let actions = self .snapshot .log_segment() - .read_actions(engine, read_schema.clone())?; + .read_actions(engine, self.read_schema.clone())?; // Process actions through reconciliation let checkpoint_data = ActionReconciliationProcessor::new( @@ -322,21 +478,12 @@ impl CheckpointWriter { ) .process_actions_iter(actions); - let output_schema = build_checkpoint_output_schema( - &config, - base_schema, - &stats_schema, - partition_schema.as_deref(), - )?; - - // Build transform expression and create expression evaluator. + // Create the expression evaluator for the checkpoint transform. // The transform is applied to reconciled action batches only (not checkpoint metadata). - let transform_expr = - build_checkpoint_transform(&config, &stats_schema, partition_schema.as_ref()); let evaluator = engine.evaluation_handler().new_expression_evaluator( - read_schema, - transform_expr, - output_schema.clone().into(), + self.read_schema.clone(), + self.transform_expr.clone(), + self.output_schema.clone().into(), )?; // Apply stats transform to each reconciled batch @@ -354,8 +501,9 @@ impl CheckpointWriter { // For V2 checkpoints, chain the checkpoint metadata batch after the transformed // action stream. The metadata batch is created with the output schema directly, // bypassing the stats transform (it has no add actions to transform). - let checkpoint_metadata = is_v2_checkpoints_supported - .then(|| self.create_checkpoint_metadata_batch(engine, &output_schema)); + let checkpoint_metadata = self + .is_v2 + .then(|| self.create_checkpoint_metadata_batch(engine, &self.output_schema)); Ok(ActionReconciliationIterator::new(Box::new( transformed.chain(checkpoint_metadata), @@ -370,41 +518,39 @@ impl CheckpointWriter { /// 2. All data has been successfully written to object storage /// /// # Parameters - /// - `engine`: Implementation of [`Engine`] apis. - /// - `metadata`: The metadata of the written checkpoint file - /// - `checkpoint_iter_state`: The state of the checkpoint data iterator + /// - `engine`: Implementation of [`Engine`] APIs. + /// - `last_checkpoint_stats`: The [`LastCheckpointHintStats`] containing fields needed to write + /// the `_last_checkpoint` file. /// /// # Returns: `Ok` if the checkpoint was successfully finalized // Internally, this method: - // 1. Validates that the checkpoint data iterator is fully exhausted - // 2. Creates the `_last_checkpoint` data with `create_last_checkpoint_data` - // 3. Writes the `_last_checkpoint` data to the `_last_checkpoint` file in the delta log + // 1. Creates the `_last_checkpoint` data with `create_last_checkpoint_data` + // 2. Writes the `_last_checkpoint` data to the `_last_checkpoint` file in the delta log pub fn finalize( self, engine: &dyn Engine, - metadata: &FileMeta, - checkpoint_iter_state: &ActionReconciliationIteratorState, + last_checkpoint_stats: &LastCheckpointHintStats, ) -> DeltaResult<()> { - // Ensure the checkpoint data iterator is fully exhausted - if !checkpoint_iter_state.is_exhausted() { - return Err(Error::checkpoint_write( - "The checkpoint data iterator must be fully consumed and written to storage before calling finalize" - )); + // Skip writing `_last_checkpoint` if the existing hint already points to a newer + // checkpoint, to avoid regressing the hint. + let checkpoint_version = self.snapshot.version(); + if let Some(hint_version) = self.snapshot.log_segment().last_checkpoint_version() { + if hint_version > checkpoint_version { + info!( + hint_version, + checkpoint_version, + "Skipping _last_checkpoint write: existing hint is newer than checkpoint" + ); + return Ok(()); + } } - let size_in_bytes = i64::try_from(metadata.size).map_err(|e| { - Error::CheckpointWrite(format!( - "Failed to convert checkpoint size in bytes from u64 {} to i64: {}, when writing _last_checkpoint", - metadata.size, e - )) - })?; - let data = create_last_checkpoint_data( engine, self.version, - checkpoint_iter_state.actions_count(), - checkpoint_iter_state.add_actions_count(), - size_in_bytes, + last_checkpoint_stats.num_actions, + last_checkpoint_stats.num_of_add_files, + last_checkpoint_stats.size_in_bytes, ); let last_checkpoint_path = LastCheckpointHint::path(&self.snapshot.log_segment().log_root)?; @@ -420,6 +566,103 @@ impl CheckpointWriter { Ok(()) } + pub(crate) fn write_v2_checkpoint_with_sidecars( + &self, + engine: &dyn Engine, + file_actions_per_sidecar_hint: usize, + ) -> DeltaResult { + let data_iter = self.checkpoint_data(engine)?; + let iter_state = data_iter.state(); + + let splitter = SidecarSplitter::new_mut_shared( + data_iter, + engine.evaluation_handler().as_ref(), + self.output_schema.clone(), + )?; + + // Write sidecar files + let mut sidecar_metas: Vec<(String, FileMeta)> = Vec::new(); + loop { + if let Some(entry) = write_single_sidecar( + engine, + &splitter, + file_actions_per_sidecar_hint, + self.snapshot.table_root(), + self.snapshot.version(), + )? { + sidecar_metas.push(entry); + } + let is_exhausted = splitter + .lock() + .map_err(|e| Error::internal_error(format!("sidecar splitter lock poisoned: {e}")))? + .is_exhausted(); + if is_exhausted { + break; + } + } + + // Collect non-file action batches(e.g., `protocol`, `metaData`, `txn`, etc.) + let non_file_batches = Arc::into_inner(splitter) + .ok_or_else(|| { + Error::internal_error("sidecar splitter Arc should have no other references") + })? + .into_inner() + .map_err(|e| Error::internal_error(format!("sidecar splitter lock poisoned: {e}")))? + .into_non_file_batches(); + + // Create sidecar action rows for the main checkpoint file. Each row populates only + // the `sidecar` column, e.g. `sidecar: { path: ".parquet", sizeInBytes: + // , modificationTime: ` in the parquet schema. + min_index: Option, + /// Index of `add.stats_parsed.maxValues.` in the parquet schema. + max_index: Option, + /// Index of `add.stats_parsed.nullCount.` in the parquet schema. + nullcount_index: Option, +} + +/// A [`ParquetStatsProvider`] for row group skipping in checkpoint and sidecar parquet files. +/// +/// Unlike [`RowGroupFilter`] which evaluates a predicate whose column references directly match +/// the parquet schema, this filter evaluates a predicate with physical column names (e.g. +/// `x > 10`, or `col-abc-123 > 10` under column mapping) against +/// a checkpoint file where statistics are nested under `add.stats_parsed.{minValues,maxValues, +/// nullCount}.` and partition values under `add.partitionValues_parsed.`. +/// +/// A data-stat leaf is null when an Add lacks that statistic or when the row is not an Add. Because +/// parquet footer min/max ignore those nulls, the filter treats a stat as unavailable whenever its +/// footer null count is nonzero. +/// +/// Partition columns read their footer min/max of `add.partitionValues_parsed.` directly, +/// without a null guard. Footer min/max ignore nulls, so a non-matching comparison may prune a row +/// group containing a null partition leaf. This is sound for scan checkpoint reads because non-Add +/// rows do not participate in replay and an Add with a null partition value cannot match the +/// comparison. +/// +/// For example, consider a row group with an Add for partition `p = "A"` whose data statistics are +/// all null, plus a Remove whose entire `add` struct is null. The partition leaf still contains +/// `["A", null]`, even though both rows have null data-stat leaves. Its footer min/max is therefore +/// `"A"`, because parquet ignores the Remove's null. For `p = "B"`, pruning drops the non-matching +/// Add and the irrelevant Remove. The null data-stat leaves remain guarded and cannot cause +/// pruning. +#[allow(dead_code)] +pub(crate) struct CheckpointRowGroupFilter<'a> { + row_group: &'a RowGroupMetaData, + /// Maps each predicate data column to its stats column indices in the checkpoint parquet file. + stats_column_indices: HashMap, + /// Maps each predicate partition column to its `add.partitionValues_parsed.` field index. + partition_column_indices: HashMap, + /// Physical names of table partition columns. + partition_columns: &'a HashSet, +} + +#[allow(dead_code)] +impl<'a> CheckpointRowGroupFilter<'a> { + /// Creates a new checkpoint row group filter. The `predicate` uses physical column names + /// (e.g. `x > 10`, or `col-abc-123 > 10` under column mapping), and `partition_columns` + /// are the physical names of the table's partition columns. + pub(crate) fn new( + row_group: &'a RowGroupMetaData, + predicate: &Predicate, + partition_columns: &'a HashSet, + ) -> Self { + let (stats_column_indices, partition_column_indices) = compute_checkpoint_field_indices( + row_group.schema_descr().columns(), + predicate, + partition_columns, + ); + Self { + row_group, + stats_column_indices, + partition_column_indices, + partition_columns, + } + } + + /// Applies the predicate to a checkpoint row group. Returns `false` if the row group can be + /// safely skipped (none of its add file rows can match the predicate). + pub(crate) fn apply( + row_group: &'a RowGroupMetaData, + predicate: &Predicate, + partition_columns: &'a HashSet, + ) -> bool { + use crate::kernel_predicates::KernelPredicateEvaluator as _; + CheckpointRowGroupFilter::new(row_group, predicate, partition_columns) + .eval_sql_where(predicate) + != Some(false) + } + + /// Returns the footer statistics for a parquet column at the given index. + fn get_stats_at(&self, index: usize) -> Option<&Statistics> { + self.row_group.column(index).statistics() + } + + /// Retrieves a stat value for a data column, but only if the stat column has no null values + /// in this row group. A null in the stat column means some file is missing that statistic, + /// making the footer aggregate unreliable. + fn get_guarded_stat( + &self, + col: &ColumnName, + data_type: &DataType, + get_index: impl FnOnce(&StatsColumnIndices) -> Option, + extract: impl FnOnce(&DataType, &Statistics) -> Option, + ) -> Option { + let indices = self.stats_column_indices.get(col)?; + let stat_index = get_index(indices)?; + // If the stat column has any nulls, some files are missing this stat and the footer + // aggregate is unreliable. + if column_has_nulls(self.row_group, stat_index) { + return None; + } + extract(data_type, self.get_stats_at(stat_index)?) + } +} + +impl ParquetStatsProvider for CheckpointRowGroupFilter<'_> { + fn get_parquet_min_stat(&self, col: &ColumnName, data_type: &DataType) -> Option { + if is_top_level_partition_column(col, self.partition_columns) { + let &idx = self.partition_column_indices.get(col)?; + return extract_min_scalar(data_type, self.get_stats_at(idx)?); + } + self.get_guarded_stat(col, data_type, |i| i.min_index, extract_min_scalar) + } - // Parquet nullcount stats are always u64, so we can directly return the value instead of - // wrapping it in a Scalar. We can safely cast it from u64 to i64 because the nullcount can - // never be larger than the rowcount and the parquet rowcount stat is i64. - Some(nullcount? as i64) + fn get_parquet_max_stat(&self, col: &ColumnName, data_type: &DataType) -> Option { + if is_top_level_partition_column(col, self.partition_columns) { + let &idx = self.partition_column_indices.get(col)?; + return extract_max_scalar(data_type, self.get_stats_at(idx)?); + } + let max = self.get_guarded_stat(col, data_type, |i| i.max_index, extract_max_scalar)?; + Some(adjust_stats_for_truncation(max)) + } + + fn get_parquet_nullcount_stat(&self, col: &ColumnName) -> Option { + if is_top_level_partition_column(col, self.partition_columns) { + let &idx = self.partition_column_indices.get(col)?; + return extract_nullcount(self.get_stats_at(idx)); + } + let indices = self.stats_column_indices.get(col)?; + let nullcount_index = indices.nullcount_index?; + // If the nullCount stat column itself has nulls, some files are missing nullCount + // stats, making the aggregate unreliable. + if column_has_nulls(self.row_group, nullcount_index) { + return None; + } + // Return the MAX of the nullCount column from the footer. If any file has null values, + // max(nullCount) > 0, so IS NULL predicates won't falsely prune the row group. + let stats = self.get_stats_at(nullcount_index)?; + extract_max_i64(stats) } - fn get_parquet_rowcount_stat(&self) -> i64 { - self.row_group.num_rows() + fn get_parquet_rowcount_stat(&self) -> Option { + // The footer row count is the number of add file _rows_ in this row group, not the + // sum of data file row counts. Returning `None` makes IS NOT NULL evaluate to `None` + // (can't decide), which prevents it from ever pruning checkpoint row groups. This also + // allows the IS NOT NULL guard in `eval_pred_sql_where` to pass through for + // null-intolerant binary comparisons, letting the actual comparison decide. + None + } +} + +/// Adjusts a max stat value to account for millisecond truncation in JSON-serialized stats. +/// `stats_parsed` inherits from JSON stats which truncate timestamps to millisecond precision: +/// `stored_max <= actual_max <= stored_max + 999us`. Adding 999us ensures we never falsely +/// prune files whose actual max exceeds the truncated value. Non-timestamp values pass through +/// unchanged. +/// +/// See also [`DataSkippingPredicateCreator::adjust_scalar_for_max_stat_truncation`] in +/// `scan/data_skipping.rs`, which handles the same truncation issue from the predicate side +/// (subtracting 999us from the comparison value instead of adding to the stat). +#[allow(dead_code)] +fn adjust_stats_for_truncation(val: Scalar) -> Scalar { + match val { + Scalar::Timestamp(us) => Scalar::Timestamp(us.saturating_add(999)), + Scalar::TimestampNtz(us) => Scalar::TimestampNtz(us.saturating_add(999)), + other => other, + } +} + +/// Extracts the maximum value as i64 from parquet statistics. Used for reading checkpoint +/// nullCount stats where the footer max represents the largest per-file null count. +#[allow(dead_code)] +fn extract_max_i64(stats: &Statistics) -> Option { + match stats { + Statistics::Int64(s) => Some(*s.max_opt()?), + Statistics::Int32(s) => Some(i64::from(*s.max_opt()?)), + _ => None, } } @@ -250,3 +502,66 @@ pub(crate) fn compute_field_indices( }) .collect() } + +/// Returns `true` if the column is a top-level partition column. +/// Delta partition columns are always top-level (no nested partition columns). +#[allow(dead_code)] +fn is_top_level_partition_column(col: &ColumnName, partition_columns: &HashSet) -> bool { + let path = col.path(); + path.len() == 1 && partition_columns.contains(path[0].as_str()) +} + +/// Builds column index mappings for checkpoint row group skipping. Maps each predicate column to +/// its corresponding stats column indices +/// (`add.stats_parsed.{minValues,maxValues,nullCount}.`) or partition column index +/// (`add.partitionValues_parsed.`). +#[allow(dead_code)] +fn compute_checkpoint_field_indices( + fields: &[ColumnDescPtr], + predicate: &Predicate, + partition_columns: &HashSet, +) -> ( + HashMap, + HashMap, +) { + let referenced_columns = predicate.references(); + let mut stats_indices: HashMap = HashMap::new(); + let mut partition_indices: HashMap = HashMap::new(); + + for (i, field) in fields.iter().enumerate() { + let parts = field.path().parts(); + + // Match add.partitionValues_parsed.+ // Delta partition columns are always top-level (no nested partition columns). + if parts.len() == 3 && parts[0] == "add" && parts[1] == "partitionValues_parsed" { + let col_name = ColumnName::new([&parts[2]]); + if referenced_columns.contains(&col_name) + && is_top_level_partition_column(&col_name, partition_columns) + { + partition_indices.insert(col_name, i); + } + continue; + } + + // Match add.stats_parsed.{minValues,maxValues,nullCount}. + if parts.len() >= 4 && parts[0] == "add" && parts[1] == "stats_parsed" { + let stat_type = parts[2].as_str(); + let col_name = ColumnName::new(&parts[3..]); + // Skip partition columns (they use partitionValues_parsed, not stats_parsed) + if !referenced_columns.contains(&col_name) + || is_top_level_partition_column(&col_name, partition_columns) + { + continue; + } + let entry = stats_indices.entry(col_name).or_default(); + match stat_type { + MIN_VALUES => entry.min_index = Some(i), + MAX_VALUES => entry.max_index = Some(i), + NULL_COUNT => entry.nullcount_index = Some(i), + _ => {} + } + } + } + + (stats_indices, partition_indices) +} diff --git a/kernel/src/engine/parquet_row_group_skipping/tests.rs b/kernel/src/engine/parquet_row_group_skipping/tests.rs index 20707f697b..6d2170e6bc 100644 --- a/kernel/src/engine/parquet_row_group_skipping/tests.rs +++ b/kernel/src/engine/parquet_row_group_skipping/tests.rs @@ -1,9 +1,29 @@ +use std::cmp::Ordering; +use std::collections::HashSet; +use std::fs::File; +use std::sync::{Arc, LazyLock}; + use super::*; -use crate::expressions::{column_name, column_pred}; -use crate::kernel_predicates::DataSkippingPredicateEvaluator as _; +use crate::arrow::array::{Int64Array, RecordBatch, StringArray, StructArray}; +use crate::arrow::datatypes::{DataType as ArrowDataType, Field, Fields, Schema as ArrowSchema}; +use crate::expressions::{ + col, column_name, column_pred, lit, Expression, OpaquePredicateOp, ScalarExpressionEvaluator, +}; +use crate::kernel_predicates::{ + DataSkippingPredicateEvaluator as _, DirectDataSkippingPredicateEvaluator, + DirectPredicateEvaluator, IndirectDataSkippingPredicateEvaluator, + KernelPredicateEvaluator as _, +}; use crate::parquet::arrow::arrow_reader::ArrowReaderMetadata; -use crate::Predicate; -use std::fs::File; +use crate::parquet::arrow::ArrowWriter; +use crate::parquet::data_type::{ByteArray, FixedLenByteArray}; +use crate::parquet::file::properties::WriterProperties; +use crate::parquet::file::reader::FileReader; +use crate::parquet::file::serialized_reader::SerializedFileReader; +use crate::{DeltaResult, Predicate}; + +/// Empty partition column set for tests that don't need partition columns. +static NO_PARTITIONS: LazyLock> = LazyLock::new(HashSet::new); /// Performs an exhaustive set of reads against a specially crafted parquet file. /// @@ -67,10 +87,10 @@ fn test_get_stat_values() { Some(3i64.into()) ); - // Should be Some(0), but https://github.com/apache/arrow-rs/issues/9451 + // No nulls -> exact zero count (parquet 58.1+, arrow-rs#9451). assert_eq!( filter.get_nullcount_stat(&column_name!("varlen.utf8")), - None // Some(0i64.into()) + Some(0i64.into()) ); assert_eq!( @@ -437,3 +457,741 @@ fn test_get_stat_values() { ) ); } + +// A missing footer null count decodes to `None`; an exact zero is preserved as `Some(0)`. Parquet +// 58.1+ distinguishes the two (arrow-rs#9451). +#[test] +fn test_extract_nullcount_distinguishes_missing_from_zero() { + let stats = |null_count| Statistics::int64(Some(1), Some(2), None, null_count, false); + assert_eq!(extract_nullcount(None), None); + assert_eq!(extract_nullcount(Some(&stats(None))), None); + assert_eq!(extract_nullcount(Some(&stats(Some(0)))), Some(0)); + assert_eq!(extract_nullcount(Some(&stats(Some(3)))), Some(3)); +} + +// A zero null count lets IS NULL prune a column with no nulls; a positive count keeps it. The +// fixture's `varlen.utf8` has 0 nulls, `bool` has 3. +#[test] +fn test_row_group_filter_is_null_prunes_when_nullcount_is_zero() { + let file = File::open("./tests/data/parquet_row_group_skipping/part-00000-b92e017a-50ba-4676-8322-48fc371c2b59-c000.snappy.parquet").unwrap(); + let metadata = ArrowReaderMetadata::load(&file, Default::default()).unwrap(); + let row_group = metadata.metadata().row_group(0); + + assert!(!RowGroupFilter::apply( + row_group, + &Predicate::is_null(col!("varlen.utf8")) + )); + assert!(RowGroupFilter::apply( + row_group, + &Predicate::is_null(col!("bool")) + )); +} + +// Intervals are unsupported for skipping under any footer encoding, so extraction returns None. +#[test] +fn test_interval_skipping_unsupported_for_any_footer_stats() { + let interval_flba = FixedLenByteArray::from(ByteArray::from(vec![0u8; 12])); + let variants = [ + Statistics::int32(Some(0), Some(1), None, Some(0), false), + Statistics::int64(Some(0), Some(1), None, Some(0), false), + Statistics::fixed_len_byte_array( + Some(interval_flba.clone()), + Some(interval_flba), + None, + Some(0), + false, + ), + ]; + for dt in [DataType::INTERVAL_YEAR_MONTH, DataType::INTERVAL_DAY_TIME] { + for stats in &variants { + // Call the extractor directly: the fixture has no interval column for get_min_stat + assert_eq!(extract_min_scalar(&dt, stats), None); + assert_eq!(extract_max_scalar(&dt, stats), None); + } + } +} + +/// Wraps an Int64 leaf array in nested StructArrays matching the given column path. +/// +/// For `col_path = &["a", "b"]` and a leaf array `[10, 20]`, produces the Arrow structure: +/// `a: Struct { b: Int64 }` with values `[{b: 10}, {b: 20}]`. +/// +/// Returns the outermost `(field, array)` pair for embedding in a parent struct. +fn wrap_in_nested_struct( + col_path: &[&str], + values: Arc, +) -> (Arc, Arc) { + assert!(!col_path.is_empty()); + let mut field = Arc::new(Field::new( + *col_path.last().unwrap(), + ArrowDataType::Int64, + true, + )); + let mut array: Arc = values; + for &name in col_path[..col_path.len() - 1].iter().rev() { + let struct_array = StructArray::from(vec![(field.clone(), array)]); + field = Arc::new(Field::new( + name, + ArrowDataType::Struct(Fields::from(vec![field])), + true, + )); + array = Arc::new(struct_array); + } + (field, array) +} + +/// Builds a `(field, array)` pair for a single stat type (e.g. `minValues.`). +fn build_stat_column( + stat_name: &str, + col_path: &[&str], + values: Arc, +) -> (Arc, Arc) { + let (col_field, col_array) = wrap_in_nested_struct(col_path, values); + let stat_struct = StructArray::from(vec![(col_field.clone(), col_array)]); + let stat_field = Arc::new(Field::new( + stat_name, + ArrowDataType::Struct(Fields::from(vec![col_field])), + true, + )); + (stat_field, Arc::new(stat_struct)) +} + +/// Writes a checkpoint-like parquet file with the given per-file statistics. +/// +/// Schema: `add.stats_parsed.{minValues,maxValues,nullCount}.` (INT64) +/// + optionally `add.partitionValues_parsed.part_col` (STRING) +/// +/// `col_path` supports nested columns (e.g. `&["a", "b"]` produces `...minValues.a.b`). +/// Each array index represents one data file row. Use `None` to simulate missing statistics. +/// When `part_values` is `Some`, a `partitionValues_parsed.part_col` column is included. +fn write_checkpoint_parquet( + min_values: &[Option], + max_values: &[Option], + null_counts: &[Option], + col_path: &[&str], + part_values: Option<&[Option<&str>]>, +) -> tempfile::NamedTempFile { + let (min_f, min_a) = build_stat_column( + MIN_VALUES, + col_path, + Arc::new(Int64Array::from(min_values.to_vec())), + ); + let (max_f, max_a) = build_stat_column( + MAX_VALUES, + col_path, + Arc::new(Int64Array::from(max_values.to_vec())), + ); + let (nc_f, nc_a) = build_stat_column( + NULL_COUNT, + col_path, + Arc::new(Int64Array::from(null_counts.to_vec())), + ); + + let stats_struct = StructArray::from(vec![ + (min_f.clone(), min_a), + (max_f.clone(), max_a), + (nc_f.clone(), nc_a), + ]); + let stats_parsed_field = Arc::new(Field::new( + "stats_parsed", + ArrowDataType::Struct(Fields::from(vec![min_f, max_f, nc_f])), + true, + )); + + let mut add_children: Vec<(Arc, Arc)> = + vec![(stats_parsed_field, Arc::new(stats_struct))]; + + if let Some(part_values) = part_values { + let part_col_field = Arc::new(Field::new("part_col", ArrowDataType::Utf8, true)); + let pv_parsed_field = Arc::new(Field::new( + "partitionValues_parsed", + ArrowDataType::Struct(Fields::from(vec![part_col_field.clone()])), + true, + )); + let part_col = Arc::new(StringArray::from(part_values.to_vec())); + let pv_struct = StructArray::from(vec![(part_col_field.clone(), part_col as _)]); + add_children.push((pv_parsed_field, Arc::new(pv_struct))); + } + + let add_struct = StructArray::from(add_children.clone()); + let add_fields: Fields = add_children.iter().map(|(f, _)| f.clone()).collect(); + let add_field = Arc::new(Field::new("add", ArrowDataType::Struct(add_fields), true)); + let schema = Arc::new(ArrowSchema::new(vec![add_field])); + + let batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(add_struct)]).unwrap(); + + let tmp = tempfile::NamedTempFile::new().unwrap(); + let file = tmp.as_file().try_clone().unwrap(); + let mut writer = ArrowWriter::try_new(file, schema, None).unwrap(); + writer.write(&batch).unwrap(); + writer.close().unwrap(); + tmp +} + +fn checkpoint_row_group_metadata( + tmp: &tempfile::NamedTempFile, +) -> crate::parquet::file::metadata::ParquetMetaData { + let file = File::open(tmp.path()).unwrap(); + let reader = SerializedFileReader::new(file).unwrap(); + reader.metadata().clone() +} + +#[test] +fn checkpoint_filter_returns_stats_when_no_nulls_in_stat_columns() { + let tmp = write_checkpoint_parquet( + &[Some(10), Some(20)], + &[Some(100), Some(200)], + &[Some(2), Some(0)], + &["x"], + Some(&[Some("a"), Some("b")]), + ); + let metadata = checkpoint_row_group_metadata(&tmp); + let row_group = metadata.row_group(0); + + let predicate = Predicate::gt(col!("x"), lit(50i64)); + let filter = CheckpointRowGroupFilter::new(row_group, &predicate, &NO_PARTITIONS); + + // All stat columns are non-null, so stats should be available. + // min(minValues.x) across rows = 10, max(maxValues.x) = 200 + assert_eq!( + filter.get_min_stat(&column_name!("x"), &DataType::LONG), + Some(10i64.into()) + ); + assert_eq!( + filter.get_max_stat(&column_name!("x"), &DataType::LONG), + Some(200i64.into()) + ); + // max(nullCount.x) = 2 (conservative: at least one file has nulls) + assert_eq!( + filter.get_nullcount_stat(&column_name!("x")), + Some(2i64.into()) + ); + // Row count is None for checkpoint files (footer rowcount is not meaningful) + assert_eq!(filter.get_rowcount_stat(), None); +} + +#[test] +fn checkpoint_filter_returns_none_when_stat_column_has_nulls() { + // Row 1 has null stats (file missing statistics) + let tmp = write_checkpoint_parquet( + &[Some(10), None, Some(20)], + &[Some(100), None, Some(200)], + &[Some(2), None, Some(0)], + &["x"], + Some(&[Some("a"), Some("b"), Some("b")]), + ); + let metadata = checkpoint_row_group_metadata(&tmp); + let row_group = metadata.row_group(0); + + let predicate = Predicate::gt(col!("x"), lit(50i64)); + let filter = CheckpointRowGroupFilter::new(row_group, &predicate, &NO_PARTITIONS); + + // Stat columns have nulls (row 1), so footer aggregates are unreliable. + assert_eq!( + filter.get_min_stat(&column_name!("x"), &DataType::LONG), + None + ); + assert_eq!( + filter.get_max_stat(&column_name!("x"), &DataType::LONG), + None + ); + assert_eq!(filter.get_nullcount_stat(&column_name!("x")), None); +} + +#[test] +fn checkpoint_filter_partition_columns_always_available() { + // Row 1 has null stats, but partition values are always present. + let tmp = write_checkpoint_parquet( + &[Some(10), None], + &[Some(100), None], + &[Some(2), None], + &["x"], + Some(&[Some("a"), Some("b")]), + ); + let metadata = checkpoint_row_group_metadata(&tmp); + let row_group = metadata.row_group(0); + + let partition_columns: HashSet = ["part_col".to_string()].into(); + let predicate = Predicate::and( + Predicate::gt(col!("x"), lit(50i64)), + Predicate::eq(col!("part_col"), lit("a")), + ); + let filter = CheckpointRowGroupFilter::new(row_group, &predicate, &partition_columns); + + // Partition column stats should always be available (not null-guarded). + assert_eq!( + filter.get_min_stat(&column_name!("part_col"), &DataType::STRING), + Some("a".into()) + ); + assert_eq!( + filter.get_max_stat(&column_name!("part_col"), &DataType::STRING), + Some("b".into()) + ); + + // Data column stats should still be None (stat columns have nulls). + assert_eq!( + filter.get_min_stat(&column_name!("x"), &DataType::LONG), + None + ); +} + +#[test] +fn checkpoint_filter_apply_keeps_row_group_with_missing_stats() { + // Row 1 has null stats -- row group must NOT be pruned even though + // the non-null footer max is 100 < 500. + let tmp = write_checkpoint_parquet( + &[Some(10), None], + &[Some(100), None], + &[Some(0), None], + &["x"], + Some(&[Some("a"), Some("b")]), + ); + let metadata = checkpoint_row_group_metadata(&tmp); + let row_group = metadata.row_group(0); + + let predicate = Predicate::gt(col!("x"), lit(500i64)); + // Without null guarding, footer max=100 < 500 would falsely prune this row group. + assert!(CheckpointRowGroupFilter::apply( + row_group, + &predicate, + &NO_PARTITIONS + )); +} + +#[test] +fn checkpoint_filter_apply_prunes_row_group_with_all_stats_present() { + // All stats present, max=200 < 500, so row group can be pruned. + let tmp = write_checkpoint_parquet( + &[Some(10), Some(20)], + &[Some(100), Some(200)], + &[Some(0), Some(0)], + &["x"], + Some(&[Some("a"), Some("b")]), + ); + let metadata = checkpoint_row_group_metadata(&tmp); + let row_group = metadata.row_group(0); + + let predicate = Predicate::gt(col!("x"), lit(500i64)); + assert!(!CheckpointRowGroupFilter::apply( + row_group, + &predicate, + &NO_PARTITIONS + )); +} + +#[test] +fn checkpoint_filter_is_null_with_all_stats_present() { + // All nullCount values present. max(nullCount.x) = 5 > 0, so IS NULL should keep. + let tmp = write_checkpoint_parquet( + &[Some(10), Some(20)], + &[Some(100), Some(200)], + &[Some(5), Some(0)], + &["x"], + Some(&[Some("a"), Some("b")]), + ); + let metadata = checkpoint_row_group_metadata(&tmp); + let row_group = metadata.row_group(0); + + // IS NULL(x): at least one file has nullCount > 0, so keep the row group. + let predicate = Predicate::is_null(col!("x")); + assert!(CheckpointRowGroupFilter::apply( + row_group, + &predicate, + &NO_PARTITIONS + )); +} + +#[test] +fn checkpoint_filter_is_null_all_zero_nullcounts() { + // All nullCount values are 0, so max(nullCount) = 0. The data skipping evaluator checks + // nullcount != 0 for IS NULL. Since nullcount == 0, no files have nulls and IS NULL prunes. + let tmp = write_checkpoint_parquet( + &[Some(10), Some(20)], + &[Some(100), Some(200)], + &[Some(0), Some(0)], + &["x"], + Some(&[Some("a"), Some("b")]), + ); + let metadata = checkpoint_row_group_metadata(&tmp); + let row_group = metadata.row_group(0); + + let predicate = Predicate::is_null(col!("x")); + let filter = CheckpointRowGroupFilter::new(row_group, &predicate, &NO_PARTITIONS); + + // All nullCount entries are present (no nulls in the stat column), so the null guard + // passes. max(nullCount.x) = 0, meaning no files have nulls for x. + // The data skipping evaluator checks nullcount != 0 for IS NULL. Since nullcount == 0, + // it means no files have nulls, so IS NULL can prune. + let result = filter.eval_sql_where(&predicate); + // eval_sql_where returns Some(false) -> can skip. Semantics: no file has null values for x. + assert_eq!(result, Some(false)); +} + +#[test] +fn checkpoint_filter_is_not_null_never_prunes() { + // IS NOT NULL can never prune checkpoint row groups because get_rowcount_stat() returns None. + let tmp = write_checkpoint_parquet( + &[Some(10), Some(20)], + &[Some(100), Some(200)], + &[Some(5), Some(3)], + &["x"], + Some(&[Some("a"), Some("b")]), + ); + let metadata = checkpoint_row_group_metadata(&tmp); + let row_group = metadata.row_group(0); + + let predicate = Predicate::is_not_null(col!("x")); + let filter = CheckpointRowGroupFilter::new(row_group, &predicate, &NO_PARTITIONS); + + // IS NOT NULL checks nullcount != rowcount. Since rowcount is None, the evaluator + // short-circuits and returns None (can't decide), which always keeps the row group. + let result = filter.eval_sql_where(&predicate); + assert_eq!(result, None); +} + +#[test] +fn checkpoint_filter_timestamp_max_widened() { + // Timestamp max stats are widened by 999us to account for millisecond truncation + // in JSON-serialized stats (which stats_parsed inherits). + let tmp = write_checkpoint_parquet( + &[Some(10), Some(20)], + &[Some(100), Some(200)], + &[Some(0), Some(0)], + &["x"], + Some(&[Some("a"), Some("b")]), + ); + let metadata = checkpoint_row_group_metadata(&tmp); + let row_group = metadata.row_group(0); + + let predicate = column_pred!("x"); + let filter = CheckpointRowGroupFilter::new(row_group, &predicate, &NO_PARTITIONS); + + // max(maxValues.x) = 200, widened to 200 + 999 = 1199 + assert_eq!( + filter.get_max_stat(&column_name!("x"), &DataType::TIMESTAMP), + Some(Scalar::Timestamp(1199)) + ); + assert_eq!( + filter.get_max_stat(&column_name!("x"), &DataType::TIMESTAMP_NTZ), + Some(Scalar::TimestampNtz(1199)) + ); + + // Non-timestamp types are not widened. + assert_eq!( + filter.get_max_stat(&column_name!("x"), &DataType::LONG), + Some(200i64.into()) + ); +} + +#[test] +fn checkpoint_filter_unknown_column_returns_none() { + let tmp = write_checkpoint_parquet( + &[Some(10)], + &[Some(100)], + &[Some(0)], + &["x"], + Some(&[Some("a")]), + ); + let metadata = checkpoint_row_group_metadata(&tmp); + let row_group = metadata.row_group(0); + + // Predicate references column "y" which doesn't exist in the checkpoint. + let predicate = Predicate::gt(col!("y"), lit(50i64)); + let filter = CheckpointRowGroupFilter::new(row_group, &predicate, &NO_PARTITIONS); + + assert_eq!( + filter.get_min_stat(&column_name!("y"), &DataType::LONG), + None + ); + assert_eq!( + filter.get_max_stat(&column_name!("y"), &DataType::LONG), + None + ); + assert_eq!(filter.get_nullcount_stat(&column_name!("y")), None); +} + +#[test] +fn checkpoint_filter_mixed_partition_and_data_predicate() { + // Row 1 has null data stats but valid partition values. + // Predicate: part_col = "a" AND x > 500 + // The partition predicate should still work, but the data predicate can't prune. + let tmp = write_checkpoint_parquet( + &[Some(10), None], + &[Some(100), None], + &[Some(0), None], + &["x"], + Some(&[Some("a"), Some("b")]), + ); + let metadata = checkpoint_row_group_metadata(&tmp); + let row_group = metadata.row_group(0); + + let partition_columns: HashSet = ["part_col".to_string()].into(); + + // x > 500: data stats have nulls, can't prune. Overall AND can't prune. + let predicate = Predicate::and( + Predicate::eq(col!("part_col"), lit("a")), + Predicate::gt(col!("x"), lit(500i64)), + ); + assert!(CheckpointRowGroupFilter::apply( + row_group, + &predicate, + &partition_columns + )); + + // part_col = "c": partition stats show min="a", max="b", so "c" is out of range. + // But x stats are unreliable. AND("c" not in [a,b], x unknown) -- the partition arm + // is false, so the AND is false -> can prune! + let predicate = Predicate::and( + Predicate::eq(col!("part_col"), lit("c")), + Predicate::gt(col!("x"), lit(5i64)), + ); + assert!(!CheckpointRowGroupFilter::apply( + row_group, + &predicate, + &partition_columns, + )); +} + +/// An opaque "less than" predicate op that supports data skipping. Used to verify that +/// `CheckpointRowGroupFilter` correctly delegates opaque predicates through the +/// `ParquetStatsProvider` trait, where null guarding is applied by the provider. +#[derive(Debug, PartialEq)] +struct OpaqueLessThanOp; + +impl OpaquePredicateOp for OpaqueLessThanOp { + fn name(&self) -> &str { + "less_than" + } + + fn eval_pred_scalar( + &self, + _eval_expr: &ScalarExpressionEvaluator<'_>, + _evaluator: &DirectPredicateEvaluator<'_>, + _exprs: &[Expression], + _inverted: bool, + ) -> DeltaResult> { + unimplemented!("not needed for data skipping tests") + } + + fn eval_as_data_skipping_predicate( + &self, + evaluator: &DirectDataSkippingPredicateEvaluator<'_>, + exprs: &[Expression], + inverted: bool, + ) -> Option { + let (col, val, ord) = match exprs { + [Expression::Column(col), Expression::Literal(val)] => (col, val, Ordering::Less), + [Expression::Literal(val), Expression::Column(col)] => (col, val, Ordering::Greater), + _ => return None, + }; + evaluator.partial_cmp_min_stat(col, val, ord, inverted) + } + + fn as_data_skipping_predicate( + &self, + _evaluator: &IndirectDataSkippingPredicateEvaluator<'_>, + _exprs: &[Expression], + _inverted: bool, + ) -> Option { + unimplemented!("not needed for data skipping tests") + } +} + +#[test] +fn checkpoint_filter_opaque_predicate_with_null_guarded_stats() { + // All stats present: opaque predicate should participate in skipping. + let tmp = write_checkpoint_parquet( + &[Some(10), Some(20)], + &[Some(100), Some(200)], + &[Some(0), Some(0)], + &["x"], + Some(&[Some("a"), Some("b")]), + ); + let metadata = checkpoint_row_group_metadata(&tmp); + let row_group = metadata.row_group(0); + + // OpaqueLessThanOp(x, 5) -> "x < 5". Data skipping checks min(x) < 5. + // min(x) = 10, so 10 < 5 is false -> can skip the row group. + let predicate = Predicate::opaque(OpaqueLessThanOp, vec![col!("x"), lit(5i64)]); + assert!(!CheckpointRowGroupFilter::apply( + row_group, + &predicate, + &NO_PARTITIONS + )); + + // OpaqueLessThanOp(x, 50) -> "x < 50". min(x) = 10, so 10 < 50 is true -> keep. + let predicate = Predicate::opaque(OpaqueLessThanOp, vec![col!("x"), lit(50i64)]); + assert!(CheckpointRowGroupFilter::apply( + row_group, + &predicate, + &NO_PARTITIONS + )); +} + +#[test] +fn checkpoint_filter_opaque_predicate_with_missing_stats() { + // Row 1 has null stats: opaque predicate must not cause false pruning. + let tmp = write_checkpoint_parquet( + &[Some(10), None], + &[Some(100), None], + &[Some(0), None], + &["x"], + Some(&[Some("a"), Some("b")]), + ); + let metadata = checkpoint_row_group_metadata(&tmp); + let row_group = metadata.row_group(0); + + // OpaqueLessThanOp(x, 5) -> "x < 5". The stat column has nulls, so the null-guarded + // provider returns None for min(x). The opaque op gets None and returns None, + // which means the row group cannot be pruned. This is the safe behavior. + let predicate = Predicate::opaque(OpaqueLessThanOp, vec![col!("x"), lit(5i64)]); + assert!(CheckpointRowGroupFilter::apply( + row_group, + &predicate, + &NO_PARTITIONS + )); +} + +#[test] +fn checkpoint_filter_partition_is_null_prunes_when_all_values_present() { + // All partition values non-null -> exact zero null count, so IS NULL matches nothing and + // prunes. + let tmp = write_checkpoint_parquet( + &[Some(10), Some(20)], + &[Some(100), Some(200)], + &[Some(0), Some(0)], + &["x"], + Some(&[Some("a"), Some("b")]), + ); + let metadata = checkpoint_row_group_metadata(&tmp); + let row_group = metadata.row_group(0); + + let partition_columns: HashSet = ["part_col".to_string()].into(); + let predicate = Predicate::is_null(col!("part_col")); + let filter = CheckpointRowGroupFilter::new(row_group, &predicate, &partition_columns); + + assert_eq!( + filter.get_nullcount_stat(&column_name!("part_col")), + Some(0i64.into()) + ); + assert_eq!(filter.eval_sql_where(&predicate), Some(false)); +} + +#[test] +fn checkpoint_filter_multi_row_group_skipping() { + // Build schema: add.stats_parsed.{minValues,maxValues,nullCount}.x (INT64) + let col_field = Arc::new(Field::new("x", ArrowDataType::Int64, true)); + let min_field = Arc::new(Field::new( + MIN_VALUES, + ArrowDataType::Struct(Fields::from(vec![col_field.clone()])), + true, + )); + let max_field = Arc::new(Field::new( + MAX_VALUES, + ArrowDataType::Struct(Fields::from(vec![col_field.clone()])), + true, + )); + let nc_field = Arc::new(Field::new( + NULL_COUNT, + ArrowDataType::Struct(Fields::from(vec![col_field.clone()])), + true, + )); + let stats_field = Arc::new(Field::new( + "stats_parsed", + ArrowDataType::Struct(Fields::from(vec![ + min_field.clone(), + max_field.clone(), + nc_field.clone(), + ])), + true, + )); + let add_field = Arc::new(Field::new( + "add", + ArrowDataType::Struct(Fields::from(vec![stats_field.clone()])), + true, + )); + let schema = Arc::new(ArrowSchema::new(vec![add_field])); + + let make_batch = |mins: &[i64], maxs: &[i64], ncs: &[i64]| { + let min_arr = Arc::new(Int64Array::from(mins.to_vec())); + let max_arr = Arc::new(Int64Array::from(maxs.to_vec())); + let nc_arr = Arc::new(Int64Array::from(ncs.to_vec())); + let min_s = StructArray::from(vec![(col_field.clone(), min_arr as _)]); + let max_s = StructArray::from(vec![(col_field.clone(), max_arr as _)]); + let nc_s = StructArray::from(vec![(col_field.clone(), nc_arr as _)]); + let stats_s = StructArray::from(vec![ + (min_field.clone(), Arc::new(min_s) as _), + (max_field.clone(), Arc::new(max_s) as _), + (nc_field.clone(), Arc::new(nc_s) as _), + ]); + let add_s = StructArray::from(vec![(stats_field.clone(), Arc::new(stats_s) as _)]); + RecordBatch::try_new(schema.clone(), vec![Arc::new(add_s)]).unwrap() + }; + + // RG0: x in [10, 100] -> pruned by "x > 500" + // RG1: x in [400, 600] -> kept by "x > 500" + let tmp = tempfile::NamedTempFile::new().unwrap(); + let file = tmp.as_file().try_clone().unwrap(); + #[allow(deprecated)] // renamed to set_max_row_group_row_count in newer parquet versions + let props = WriterProperties::builder() + .set_max_row_group_size(2) + .build(); + let mut writer = ArrowWriter::try_new(file, schema.clone(), Some(props)).unwrap(); + writer + .write(&make_batch(&[10, 20], &[50, 100], &[0, 0])) + .unwrap(); + writer + .write(&make_batch(&[400, 450], &[500, 600], &[0, 0])) + .unwrap(); + writer.close().unwrap(); + + let file = File::open(tmp.path()).unwrap(); + let builder = + crate::parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder::try_new(file) + .unwrap(); + assert_eq!(builder.metadata().num_row_groups(), 2); + + let predicate = Predicate::gt(col!("x"), lit(500i64)); + let builder = builder.with_checkpoint_row_group_filter(&predicate, &NO_PARTITIONS, None); + + // Only RG1 (x in [400, 600]) survives: max(x) = 600 > 500. + let reader = builder.build().unwrap(); + let batches: Vec<_> = reader.into_iter().collect::>().unwrap(); + assert_eq!(batches.len(), 1); + assert_eq!(batches[0].num_rows(), 2); +} + +#[test] +fn checkpoint_filter_nested_struct_column_stats() { + // Per-file statistics mirror the data schema structure (Delta protocol spec, "Per-file + // Statistics" section). For a nested column `a.b`, stats are stored as: + // add.stats_parsed.{minValues,maxValues,nullCount}.a.b (INT64) + let tmp = write_checkpoint_parquet( + &[Some(10), Some(20)], + &[Some(100), Some(200)], + &[Some(0), Some(0)], + &["a", "b"], + None, + ); + let metadata = checkpoint_row_group_metadata(&tmp); + let row_group = metadata.row_group(0); + + let col = column_name!("a.b"); + let predicate = Predicate::gt(col.clone(), Scalar::from(500i64)); + let filter = CheckpointRowGroupFilter::new(row_group, &predicate, &NO_PARTITIONS); + + assert_eq!( + filter.get_min_stat(&col, &DataType::LONG), + Some(10i64.into()) + ); + assert_eq!( + filter.get_max_stat(&col, &DataType::LONG), + Some(200i64.into()) + ); + // max(x) = 200 < 500 -> can prune. + assert!(!CheckpointRowGroupFilter::apply( + row_group, + &predicate, + &NO_PARTITIONS + )); +} diff --git a/kernel/src/engine/plans/json.rs b/kernel/src/engine/plans/json.rs new file mode 100644 index 0000000000..b316572782 --- /dev/null +++ b/kernel/src/engine/plans/json.rs @@ -0,0 +1,234 @@ +//! A [`JsonHandler`] implementation backed by a [`PlanExecutor`]. + +use std::sync::Arc; + +use tracing::debug; +use url::Url; + +use crate::engine::arrow_utils; +use crate::plans::{Operation, PlanBuilder, PlanExecutor}; +use crate::schema::SchemaRef; +use crate::{ + DeltaResult, DeltaResultIterator, EngineData, Error, FileDataReadResultIterator, FileMeta, + FilteredEngineData, JsonHandler, PredicateRef, +}; + +/// A [`JsonHandler`] that delegates to a [`PlanExecutor`]. +/// +/// Operations not yet implemented on the plan-execution path delegate to `fallback` when one is +/// configured, and otherwise return an unsupported error. +pub struct PlanBasedJsonHandler { + executor: Arc, + fallback: Option>, +} + +impl PlanBasedJsonHandler { + /// Construct a handler that delegates not-yet-implemented operations to `fallback`, or returns + /// an unsupported error for them when `fallback` is `None`. + pub fn new( + plan_executor: Arc, + fallback: Option>, + ) -> Self { + Self { + executor: plan_executor, + fallback, + } + } +} + +impl JsonHandler for PlanBasedJsonHandler { + fn parse_json( + &self, + json_strings: Box, + output_schema: SchemaRef, + ) -> DeltaResult> { + arrow_utils::parse_json(json_strings, output_schema) + } + + fn read_json_files( + &self, + files: &[FileMeta], + physical_schema: SchemaRef, + _predicate: Option, + ) -> DeltaResult { + // TODO: `_predicate` is dropped. Re-apply it as a Filter node over the scan; the + // single-node executor can then match the filter -> scan shape. + let query = PlanBuilder::scan_json(files.to_vec(), &[], physical_schema)?.build()?; + self.executor + .execute_op(Operation::QueryPlan(query))? + .into_data() + } + + fn write_json_file( + &self, + path: &Url, + data: DeltaResultIterator<'_, FilteredEngineData>, + overwrite: bool, + ) -> DeltaResult<()> { + let Some(fallback) = &self.fallback else { + return Err(Error::unsupported( + "PlanBasedJsonHandler does not support write_json_file yet, and no fallback \ + handler is configured", + )); + }; + debug!(%path, "PlanBasedJsonHandler delegating write_json_file to fallback handler"); + fallback.write_json_file(path, data, overwrite) + } +} + +#[cfg(test)] +mod tests { + // TODO(#2618): Refactor and share a test suite with sync engine tests. + + use std::io::Write as _; + use std::sync::Arc; + + use rstest::rstest; + use tempfile::{tempdir, NamedTempFile}; + use url::Url; + + use super::PlanBasedJsonHandler; + use crate::arrow::array::{Array, Int32Array, RecordBatch, StringArray}; + use crate::engine::arrow_data::ArrowEngineData; + use crate::engine::plans::parquet::PlanBasedParquetHandler; + use crate::engine::sync::plan::SyncPlanExecutor; + use crate::engine::sync::SyncEngine; + use crate::engine_data::FilteredEngineData; + use crate::schema::{DataType, SchemaRef, StructField, StructType}; + use crate::{ + DeltaResult, Engine as _, EngineData, FileDataReadResultIterator, FileMeta, + JsonHandler as _, ParquetHandler as _, + }; + + fn make_handler() -> PlanBasedJsonHandler { + PlanBasedJsonHandler::new( + Arc::new(SyncPlanExecutor::default()), + Some(SyncEngine::new().json_handler()), + ) + } + + fn single_column_data(values: Vec<&str>) -> Box { + let batch = RecordBatch::try_from_iter(vec![( + "x", + Arc::new(StringArray::from(values)) as Arc, + )]) + .unwrap(); + Box::new(ArrowEngineData::new(batch)) + } + + #[test] + fn test_write_json_file_delegates_to_fallback() { + let dir = tempdir().unwrap(); + let path = dir.path().join("out.json"); + let url = Url::from_file_path(&path).unwrap(); + let filtered = Ok(FilteredEngineData::with_all_rows_selected( + single_column_data(vec!["a", "b"]), + )); + make_handler() + .write_json_file(&url, Box::new(std::iter::once(filtered)), false) + .unwrap(); + let contents = std::fs::read_to_string(&path).unwrap(); + assert_eq!(contents, "{\"x\":\"a\"}\n{\"x\":\"b\"}\n"); + } + + fn make_parquet_handler() -> PlanBasedParquetHandler { + PlanBasedParquetHandler::new( + Arc::new(SyncPlanExecutor::default()), + Some(SyncEngine::new().parquet_handler()), + ) + } + + fn temp_json_file(lines: &[&str]) -> (NamedTempFile, FileMeta) { + let mut temp_file = NamedTempFile::new().unwrap(); + for line in lines { + writeln!(temp_file, "{line}").unwrap(); + } + let path = temp_file.path(); + let file_url = Url::from_file_path(path).unwrap(); + let size = std::fs::metadata(path).unwrap().len(); + let file_meta = FileMeta { + location: file_url, + last_modified: 0, + size, + }; + (temp_file, file_meta) + } + + #[test] + fn test_read_json_files() { + let (_temp, file_meta) = temp_json_file(&[r#"{"x": 1}"#, r#"{"x": 2}"#, r#"{"x": 3}"#]); + let schema = + Arc::new(StructType::try_new([StructField::not_null("x", DataType::INTEGER)]).unwrap()); + + let mut iter = make_handler() + .read_json_files(&[file_meta], schema, None) + .unwrap(); + + let batch = iter.next().expect("expected at least one batch").unwrap(); + let record_batch: RecordBatch = + ArrowEngineData::try_from_engine_data(batch).unwrap().into(); + assert_eq!(record_batch.num_rows(), 3); + assert_eq!(record_batch.num_columns(), 1); + assert_eq!( + record_batch + .column(0) + .as_any() + .downcast_ref::() + .unwrap() + .values(), + &[1, 2, 3] + ); + assert!(iter.next().is_none(), "expected exactly one batch"); + } + + fn test_schema() -> SchemaRef { + Arc::new(StructType::try_new([StructField::not_null("x", DataType::INTEGER)]).unwrap()) + } + + /// No files -> an absent plan -> a zero-row result (no rows, no error), for either handler. + #[rstest] + #[case(make_handler().read_json_files(&[], test_schema(), None))] + #[case(make_parquet_handler().read_parquet_files(&[], test_schema(), None))] + fn empty_input_yields_no_rows(#[case] res: DeltaResult) { + let rows: usize = res.unwrap().map(|batch| batch.unwrap().len()).sum(); + assert_eq!(rows, 0); + } + + #[test] + fn test_parse_json() { + let json_strings = StringArray::from(vec![r#"{"x": 1}"#, r#"{"x": 2}"#, r#"{"x": 3}"#]); + let input_batch = RecordBatch::try_from_iter(vec![( + "json", + Arc::new(json_strings) as Arc, + )]) + .unwrap(); + let input: Box = Box::new(ArrowEngineData::new(input_batch)); + + let output_schema = + Arc::new(StructType::try_new([StructField::not_null("x", DataType::INTEGER)]).unwrap()); + + let parsed = make_handler().parse_json(input, output_schema).unwrap(); + let record_batch: RecordBatch = ArrowEngineData::try_from_engine_data(parsed) + .unwrap() + .into(); + assert_eq!(record_batch.num_rows(), 3); + assert_eq!(record_batch.num_columns(), 1); + assert_eq!( + record_batch + .column(0) + .as_any() + .downcast_ref::() + .unwrap() + .values(), + &[1, 2, 3] + ); + } + + // TODO(#2618): Restore once `PlanBasedJsonHandler` moves to delta_kernel_default_engine and + // can use the test_utils::engine_contract helpers without the kernel-cfg-test cycle issue. + // + // #[test] + // fn test_file_path_contract() { + // test_json_handler_file_path_contract(&make_handler()); + // } +} diff --git a/kernel/src/engine/plans/mod.rs b/kernel/src/engine/plans/mod.rs new file mode 100644 index 0000000000..6d44d0e115 --- /dev/null +++ b/kernel/src/engine/plans/mod.rs @@ -0,0 +1,205 @@ +//! This module contains an implementation of the Engine trait that is +//! backed by a PlanExecutor. The engine delegates handler operations +//! (e.g. storage, JSON, parquet) to declarative plan execution rather than implementing +//! each handler independently. +//! +//! This allows a PlanExecutor to become the single surface for connector optimizations, +//! while still allowing kernel to use existing Engine trait APIs. +//! +//! ### Arrow Requirement: +//! The PlanBasedEngine implementation assumes the use of ArrowEngineData during JSON parsing, so it +//! is only compatible with `PlanExecutor`'s which return ArrowEngineData. + +use std::sync::Arc; + +pub mod json; +pub mod parquet; +pub mod storage; + +use json::PlanBasedJsonHandler; +use parquet::PlanBasedParquetHandler; +use storage::PlanBasedStorageHandler; + +use crate::engine::arrow_expression::ArrowEvaluationHandler; +use crate::plans::PlanExecutor; +use crate::{Engine, EvaluationHandler, JsonHandler, ParquetHandler, StorageHandler}; + +/// An [`Engine`] that routes operations through a [`PlanExecutor`]. +/// +/// Storage, JSON file reads, and Parquet file reads are converted into +/// [`Operation`](crate::plans::Operation)s and delegated to the plan executor. +/// +/// Operations not yet implemented on the plan-execution path (e.g. `write_json_file`, +/// `write_parquet_file`) are delegated to `fallback` when one is configured, and otherwise return +/// an unsupported error. A fallback is optional because a connector may be unable to construct one: +/// an FFI-backed engine resolves table paths itself, which fails for a path only the connector can +/// resolve. +pub struct PlanBasedEngine { + executor: Arc, + evaluation: Arc, + storage: Arc, + json: Arc, + parquet: Arc, +} + +impl PlanBasedEngine { + /// Construct a `PlanBasedEngine` backed by `plan_executor`, delegating operations not yet + /// implemented on the plan-execution path to `fallback` when it is provided. + /// + /// When `fallback` is `None`, those operations return an unsupported error. Evaluation uses the + /// fallback's [`EvaluationHandler`] when one is available and otherwise uses the Arrow + /// implementation. + pub fn new(fallback: Option>, plan_executor: Arc) -> Self { + let evaluation: Arc = fallback.as_ref().map_or_else( + || Arc::new(ArrowEvaluationHandler) as Arc, + |engine| engine.evaluation_handler(), + ); + Self { + evaluation, + storage: Arc::new(PlanBasedStorageHandler::new(plan_executor.clone())), + json: Arc::new(PlanBasedJsonHandler::new( + plan_executor.clone(), + fallback.as_ref().map(|engine| engine.json_handler()), + )), + parquet: Arc::new(PlanBasedParquetHandler::new( + plan_executor.clone(), + fallback.as_ref().map(|engine| engine.parquet_handler()), + )), + executor: plan_executor, + } + } +} + +impl Engine for PlanBasedEngine { + fn evaluation_handler(&self) -> Arc { + self.evaluation.clone() + } + + fn storage_handler(&self) -> Arc { + self.storage.clone() + } + + fn json_handler(&self) -> Arc { + self.json.clone() + } + + fn parquet_handler(&self) -> Arc { + self.parquet.clone() + } + + fn plan_executor(&self) -> Option> { + Some(self.executor.clone()) + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use tempfile::tempdir; + use url::Url; + + use super::PlanBasedEngine; + use crate::arrow::array::{Array, Int64Array, RecordBatch, StringArray}; + use crate::engine::arrow_data::ArrowEngineData; + use crate::engine::arrow_expression::ArrowEvaluationHandler; + use crate::engine::sync::plan::SyncPlanExecutor; + use crate::engine::sync::SyncEngine; + use crate::engine_data::FilteredEngineData; + use crate::{Engine as _, EngineData, Error}; + + fn plan_engine(fallback: Option>) -> PlanBasedEngine { + PlanBasedEngine::new(fallback, Arc::new(SyncPlanExecutor::default())) + } + + #[test] + fn uses_fallback_evaluation_handler_when_available() { + let fallback: Arc = Arc::new(SyncEngine::new()); + let expected_evaluation_handler = fallback.evaluation_handler(); + let engine = plan_engine(Some(fallback)); + + assert!(Arc::ptr_eq( + &engine.evaluation_handler(), + &expected_evaluation_handler + )); + } + + #[test] + fn uses_arrow_evaluation_handler_without_fallback() { + let engine = plan_engine(None); + + assert!(engine + .evaluation_handler() + .as_ref() + .any_ref() + .is::()); + } + + #[test] + fn unimplemented_writes_delegate_to_fallback() { + let dir = tempdir().unwrap(); + let engine = plan_engine(Some(Arc::new(SyncEngine::new()))); + let json_location = Url::from_file_path(dir.path().join("out.json")).unwrap(); + let json_batch = RecordBatch::try_from_iter(vec![( + "value", + Arc::new(StringArray::from(vec!["one"])) as Arc, + )]) + .unwrap(); + let json_data: Box = Box::new(ArrowEngineData::new(json_batch)); + + engine + .json_handler() + .write_json_file( + &json_location, + Box::new(std::iter::once(Ok( + FilteredEngineData::with_all_rows_selected(json_data), + ))), + false, + ) + .unwrap(); + assert_eq!( + std::fs::read_to_string(json_location.to_file_path().unwrap()).unwrap(), + "{\"value\":\"one\"}\n" + ); + + let parquet_location = Url::from_file_path(dir.path().join("out.parquet")).unwrap(); + let parquet_batch = RecordBatch::try_from_iter(vec![( + "value", + Arc::new(Int64Array::from(vec![1])) as Arc, + )]) + .unwrap(); + let parquet_data: Box = Box::new(ArrowEngineData::new(parquet_batch)); + + engine + .parquet_handler() + .write_parquet_file( + parquet_location.clone(), + Box::new(std::iter::once(Ok(parquet_data))), + ) + .unwrap(); + assert!( + std::fs::metadata(parquet_location.to_file_path().unwrap()) + .unwrap() + .len() + > 0 + ); + } + + #[test] + fn unimplemented_writes_are_unsupported_without_fallback() { + let engine = plan_engine(None); + let location = Url::parse("memory:///table/out").unwrap(); + + let json_err = engine + .json_handler() + .write_json_file(&location, Box::new(std::iter::empty()), false) + .expect_err("no fallback is configured"); + assert!(matches!(json_err, Error::Unsupported(_))); + + let parquet_err = engine + .parquet_handler() + .write_parquet_file(location, Box::new(std::iter::empty())) + .expect_err("no fallback is configured"); + assert!(matches!(parquet_err, Error::Unsupported(_))); + } +} diff --git a/kernel/src/engine/plans/parquet.rs b/kernel/src/engine/plans/parquet.rs new file mode 100644 index 0000000000..0179c5563e --- /dev/null +++ b/kernel/src/engine/plans/parquet.rs @@ -0,0 +1,230 @@ +//! A [`ParquetHandler`] implementation backed by a [`PlanExecutor`]. + +use std::sync::Arc; + +use tracing::debug; +use url::Url; + +use crate::plans::{IoOperation, Operation, PlanBuilder, PlanExecutor}; +use crate::schema::SchemaRef; +use crate::{ + DeltaResult, DeltaResultIteratorStatic, EngineData, Error, FileDataReadResultIterator, + FileMeta, ParquetFooter, ParquetHandler, PredicateRef, +}; + +/// A [`ParquetHandler`] that delegates to a [`PlanExecutor`]. +/// +/// Operations not yet implemented on the plan-execution path delegate to `fallback` when one is +/// configured, and otherwise return an unsupported error. +pub struct PlanBasedParquetHandler { + executor: Arc, + fallback: Option>, +} + +impl PlanBasedParquetHandler { + /// Construct a handler that delegates not-yet-implemented operations to `fallback`, or returns + /// an unsupported error for them when `fallback` is `None`. + pub fn new( + plan_executor: Arc, + fallback: Option>, + ) -> Self { + Self { + executor: plan_executor, + fallback, + } + } +} + +impl ParquetHandler for PlanBasedParquetHandler { + fn read_parquet_files( + &self, + files: &[FileMeta], + physical_schema: SchemaRef, + _predicate: Option, + ) -> DeltaResult { + // TODO: `_predicate` is dropped. Re-apply it as a Filter node over the scan; the + // single-node executor can then match the filter -> scan shape. + let query = PlanBuilder::scan_parquet(files.to_vec(), &[], physical_schema)?.build()?; + self.executor + .execute_op(Operation::QueryPlan(query))? + .into_data() + } + + fn write_parquet_file( + &self, + location: Url, + data: DeltaResultIteratorStatic>, + ) -> DeltaResult<()> { + let Some(fallback) = &self.fallback else { + return Err(Error::unsupported( + "PlanBasedParquetHandler does not support write_parquet_file yet, and no fallback \ + handler is configured", + )); + }; + debug!(%location, "PlanBasedParquetHandler delegating write_parquet_file to fallback handler"); + fallback.write_parquet_file(location, data) + } + + fn read_parquet_footer(&self, file: &FileMeta) -> DeltaResult { + let op = IoOperation::parquet_footer(file.clone()); + self.executor + .execute_op(Operation::IoOperation(op))? + .into_parquet_footer() + } +} + +#[cfg(test)] +mod tests { + // TODO(#2618): Refactor and share a test suite with sync engine tests. + + use std::path::Path; + use std::sync::Arc; + + use tempfile::tempdir; + use url::Url; + + use super::PlanBasedParquetHandler; + use crate::arrow::array::{Array, Int64Array, RecordBatch}; + use crate::engine::arrow_conversion::TryIntoKernel as _; + use crate::engine::arrow_data::ArrowEngineData; + use crate::engine::sync::plan::SyncPlanExecutor; + use crate::engine::sync::SyncEngine; + use crate::parquet::arrow::arrow_writer::ArrowWriter; + use crate::schema::{DataType, SchemaRef, StructField, StructType}; + use crate::{Engine as _, EngineData, FileMeta, ParquetHandler as _}; + + fn make_handler() -> PlanBasedParquetHandler { + PlanBasedParquetHandler::new( + Arc::new(SyncPlanExecutor::default()), + Some(SyncEngine::new().parquet_handler()), + ) + } + + fn single_column_batch() -> RecordBatch { + RecordBatch::try_from_iter(vec![( + "value", + Arc::new(Int64Array::from(vec![1, 2, 3])) as Arc, + )]) + .unwrap() + } + + #[test] + fn test_write_parquet_file_delegates_to_fallback() { + let temp_dir = tempdir().unwrap(); + let path = temp_dir.path().join("out.parquet"); + let url = Url::from_file_path(&path).unwrap(); + let data: Box = Box::new(ArrowEngineData::new(single_column_batch())); + make_handler() + .write_parquet_file(url.clone(), Box::new(std::iter::once(Ok(data)))) + .unwrap(); + + // Read it back to confirm the fallback actually wrote the rows. + let schema: SchemaRef = Arc::new( + StructType::try_new([StructField::nullable("value", DataType::LONG)]).unwrap(), + ); + let file_meta = FileMeta { + location: url, + last_modified: 0, + size: std::fs::metadata(&path).unwrap().len(), + }; + let rows: usize = make_handler() + .read_parquet_files(&[file_meta], schema, None) + .unwrap() + .map(|batch| batch.unwrap().len()) + .sum(); + assert_eq!(rows, 3); + } + + fn file_meta_for(path: &Path) -> FileMeta { + let url = Url::from_file_path(path).unwrap(); + let size = std::fs::metadata(path).unwrap().len(); + FileMeta { + location: url, + last_modified: 0, + size, + } + } + + fn make_test_parquet_file(path: &Path, batch: &RecordBatch) -> (FileMeta, SchemaRef) { + let mut writer = + ArrowWriter::try_new(std::fs::File::create(path).unwrap(), batch.schema(), None) + .unwrap(); + writer.write(batch).unwrap(); + writer.close().unwrap(); + + let file_meta = file_meta_for(path); + let schema = Arc::new(batch.schema().as_ref().try_into_kernel().unwrap()); + (file_meta, schema) + } + + #[test] + fn test_read_parquet_files() { + let temp_dir = tempdir().unwrap(); + let file_path = temp_dir.path().join("data.parquet"); + let batch = RecordBatch::try_from_iter(vec![( + "value", + Arc::new(Int64Array::from(vec![10, 20, 30])) as Arc, + )]) + .unwrap(); + let (file_meta, schema) = make_test_parquet_file(&file_path, &batch); + + let batches: Vec = make_handler() + .read_parquet_files(&[file_meta], schema, None) + .unwrap() + .map(|r| { + ArrowEngineData::try_from_engine_data(r.unwrap()) + .unwrap() + .into() + }) + .collect(); + + assert_eq!(batches.len(), 1); + assert_eq!(batches[0].num_rows(), 3); + assert_eq!( + batches[0] + .column(0) + .as_any() + .downcast_ref::() + .unwrap() + .values(), + &[10, 20, 30] + ); + } + + /// No files -> an absent plan -> a zero-row result (no rows, no error). + #[test] + fn test_read_parquet_files_empty_is_empty() { + let schema: SchemaRef = Arc::new( + StructType::try_new([StructField::nullable("value", DataType::LONG)]).unwrap(), + ); + let rows: usize = make_handler() + .read_parquet_files(&[], schema, None) + .unwrap() + .map(|batch| batch.unwrap().len()) + .sum(); + assert_eq!(rows, 0); + } + + // TODO(#2618): Restore once `PlanBasedParquetHandler` moves to delta_kernel_default_engine + // and can use the test_utils::engine_contract helpers without the kernel-cfg-test cycle issue. + // + // #[test] + // fn test_parquet_handler_reads_contract() { + // test_parquet_handler_reads_file_with_arrow_schema(&make_handler()); + // } + // + // #[test] + // fn test_read_parquet_footer_contract() { + // test_parquet_handler_reads_footer(&make_handler()); + // } + // + // #[test] + // fn test_read_parquet_footer_missing_file_contract() { + // test_parquet_handler_footer_errors_on_missing_file(&make_handler()); + // } + // + // #[test] + // fn test_read_parquet_footer_preserves_field_ids_contract() { + // test_parquet_handler_footer_preserves_field_ids(&make_handler()); + // } +} diff --git a/kernel/src/engine/plans/storage.rs b/kernel/src/engine/plans/storage.rs new file mode 100644 index 0000000000..9e565ab4f8 --- /dev/null +++ b/kernel/src/engine/plans/storage.rs @@ -0,0 +1,174 @@ +//! A [`StorageHandler`] implementation backed by a [`PlanExecutor`]. + +use std::sync::Arc; + +use bytes::Bytes; +use itertools::Itertools as _; +use url::Url; + +use crate::plans::{IoOperation, Operation, PlanExecutor, PlanResult}; +use crate::{DeltaResult, Error, FileMeta, FileSlice, StorageHandler}; + +/// A [`StorageHandler`] that delegates to a [`PlanExecutor`]. +pub struct PlanBasedStorageHandler { + executor: Arc, +} + +impl PlanBasedStorageHandler { + pub fn new(executor: Arc) -> Self { + Self { executor } + } + + fn execute_io(&self, op: IoOperation) -> DeltaResult { + self.executor.execute_op(Operation::IoOperation(op)) + } +} + +impl StorageHandler for PlanBasedStorageHandler { + fn list_from( + &self, + path: &Url, + ) -> DeltaResult>>> { + Ok(self + .execute_io(IoOperation::file_listing(path.clone()))? + .into_file_meta()?) + } + + fn read_files( + &self, + files: Vec, + ) -> DeltaResult>>> { + Ok(self + .execute_io(IoOperation::read_bytes(files))? + .into_bytes()?) + } + + fn copy_atomic(&self, src: &Url, dest: &Url) -> DeltaResult<()> { + self.execute_io(IoOperation::atomic_copy(src.clone(), dest.clone()))? + .into_unit() + } + + fn put(&self, path: &Url, data: Bytes, overwrite: bool) -> DeltaResult<()> { + self.execute_io(IoOperation::write_bytes(path.clone(), data, overwrite))? + .into_unit() + } + + fn head(&self, path: &Url) -> DeltaResult { + self.execute_io(IoOperation::head_file(path.clone()))? + .into_file_meta()? + .exactly_one() + .map_err(|e| Error::generic(format!("Expected exactly one file meta: {e}")))? + } + + fn delete(&self, _path: &Url) -> DeltaResult<()> { + // TODO(#2820): implement here once supported as IoOperation. + // Intentionally do not use a fallback because we expect this SHOULD be implemented via + // plan-execution. + Err(Error::unsupported( + "PlanBasedStorageHandler does not yet implement delete", + )) + } +} + +#[cfg(test)] +mod tests { + // TODO(#2618): Refactor and share a test suite with sync engine tests. + + use std::fs::File; + use std::io::Write as _; + use std::sync::Arc; + + use itertools::Itertools as _; + use tempfile::tempdir; + use url::Url; + + use super::PlanBasedStorageHandler; + use crate::engine::sync::plan::SyncPlanExecutor; + use crate::{Error, StorageHandler as _}; + + fn make_handler() -> PlanBasedStorageHandler { + PlanBasedStorageHandler::new(Arc::new(SyncPlanExecutor::default())) + } + + #[test] + fn test_list_from() { + let tmp = tempdir().unwrap(); + for i in 0..3 { + let path = tmp.path().join(format!("{i:020}.json")); + let mut f = File::create(&path).unwrap(); + writeln!(f, "null").unwrap(); + } + + let storage = make_handler(); + + // Listing from a file URL is exclusive of the offset itself, so listing from index 1 + // returns only index 2. + let from = Url::from_file_path(tmp.path().join("00000000000000000001.json")).unwrap(); + let files: Vec<_> = storage.list_from(&from).unwrap().try_collect().unwrap(); + assert_eq!(files.len(), 1); + let expected = Url::from_file_path(tmp.path().join("00000000000000000002.json")).unwrap(); + assert_eq!(files[0].location, expected); + + // Listing the directory returns all 3 files. + let dir = Url::from_directory_path(tmp.path()).unwrap(); + let all: Vec<_> = storage.list_from(&dir).unwrap().try_collect().unwrap(); + assert_eq!(all.len(), 3); + } + + #[test] + fn test_put_then_read_round_trip() { + let tmp = tempdir().unwrap(); + let path = tmp.path().join("blob.bin"); + let url = Url::from_file_path(&path).unwrap(); + + let storage = make_handler(); + + storage + .put(&url, bytes::Bytes::from_static(b"hello"), false) + .unwrap(); + + let files = vec![(url.clone(), None)]; + let bytes: Vec<_> = storage.read_files(files).unwrap().try_collect().unwrap(); + assert_eq!(bytes.len(), 1); + assert_eq!(bytes[0].as_ref(), b"hello"); + } + + #[test] + fn test_put_without_overwrite() { + let tmp = tempdir().unwrap(); + let path = tmp.path().join("blob.bin"); + let url = Url::from_file_path(&path).unwrap(); + + let storage = make_handler(); + + storage + .put(&url, bytes::Bytes::from_static(b"first"), false) + .unwrap(); + let err = storage + .put(&url, bytes::Bytes::from_static(b"second"), false) + .unwrap_err(); + assert!(matches!(err, Error::FileAlreadyExists(_))); + + // With `overwrite = true`, the second write succeeds. + storage + .put(&url, bytes::Bytes::from_static(b"second"), true) + .unwrap(); + } + + #[test] + fn test_head() { + let tmp = tempdir().unwrap(); + let path = tmp.path().join("file.json"); + std::fs::write(&path, b"hello").unwrap(); + let url = Url::from_file_path(&path).unwrap(); + + let meta = make_handler().head(&url).unwrap(); + assert_eq!(meta.location, url); + assert_eq!(meta.size, 5); + + // Errors on missing file + let url = Url::from_file_path(tmp.path().join("missing.json")).unwrap(); + let err = make_handler().head(&url).unwrap_err(); + assert!(matches!(err, Error::FileNotFound(_))); + } +} diff --git a/kernel/src/engine/sync/aggs.rs b/kernel/src/engine/sync/aggs.rs new file mode 100644 index 0000000000..ff2b1f1085 --- /dev/null +++ b/kernel/src/engine/sync/aggs.rs @@ -0,0 +1,759 @@ +//! Aggregate evaluation for the synchronous plan executor. + +use std::any::Any; +use std::collections::HashMap; +use std::sync::Arc; + +use itertools::Itertools; + +use super::plan::encode_keys_as_rows; +use crate::arrow::array::{new_null_array, Array, ArrayRef, Int64Array, RecordBatch}; +use crate::arrow::compute::interleave; +use crate::arrow::datatypes::DataType as ArrowDataType; +use crate::arrow::row::OwnedRow; +use crate::engine::arrow_conversion::TryIntoArrow as _; +use crate::engine::arrow_expression::evaluate_expression::{extract_column, extract_column_ref}; +use crate::expressions::ColumnName; +use crate::plans::ir::nodes::{Agg, Aggregate, NonNullByOperands}; +use crate::schema::DataType; +use crate::{DeltaResult, Error}; + +/// A specific row of input: `(batch index, row index)`. +type InputRow = (usize, usize); + +/// Batch-scoped row updater produced by [`BoundAggregate::prepare`]. +trait AggUpdater { + /// Applies `row` to the agg's `state` + fn update(&self, state: &mut dyn Any, row: InputRow) -> DeltaResult<()>; +} + +/// Bound aggregate operator: init/prepare/materialize with type-erased per-group state. +trait BoundAggregate { + /// Creates a new agg state (i.e. NULL for MIN, 0 for COUNT) + fn init_state(&self) -> Box; + /// Creates an `AggUpdater` for processing the rows of `batch` + fn prepare<'a>(&'a self, batch: &'a RecordBatch) -> DeltaResult>; + /// Finalize agg states and write them to a single output column + fn finalize(&self, states: &[&dyn Any], input: &[RecordBatch]) -> DeltaResult; +} + +/// Evaluates an [`Aggregate`]. +/// Currently supports Min, Max, Sum, Count, CountStar, MinNonNullBy, and MaxNonNullBy. Comparison +/// and arithmetic operands are LONG-only; Count accepts an arbitrary column or `COUNT(*)`. +pub(super) fn eval_aggregate( + aggregate: &Aggregate, + input: &[RecordBatch], +) -> DeltaResult> { + let ops: Vec> = aggregate + .aggs + .iter() + .zip(aggregate.schema.fields().skip(aggregate.group_by.len())) + .map(|(agg, field)| bind_aggregate(agg, field.data_type())) + .try_collect()?; + + // Each group tracks a representative input row and per-agg states as we route rows to it. + let mut groups = HashMap::>)>::new(); + let initial_aggs = || ops.iter().map(|op| op.init_state()).collect(); + for (batch_idx, batch) in input.iter().enumerate() { + let updaters: Vec<_> = ops.iter().map(|op| op.prepare(batch)).try_collect()?; + let group_keys = encode_keys_as_rows(batch, &aggregate.group_by)?; + for (row_idx, group_key) in group_keys.into_iter().enumerate() { + let row = (batch_idx, row_idx); + let (_, aggs) = groups + .entry(group_key) + .or_insert_with(|| (row, initial_aggs())); + for (updater, agg_state) in updaters.iter().zip(aggs) { + updater.update(agg_state.as_mut(), row)?; + } + } + } + + // Handle empty input. Grouped: no rows. Ungrouped: one synthetic group of initial agg states. + let output_schema = Arc::new(aggregate.schema.as_ref().try_into_arrow()?); + let (reps, mut aggs_by_group): (Vec<_>, Vec<_>) = groups.into_values().unzip(); + if aggs_by_group.is_empty() { + if !aggregate.group_by.is_empty() { + return Ok(vec![]); + } + aggs_by_group.push(initial_aggs()); + } + + // Materialize the grouping key columns, followed by agg columns + let mut output_arrays = Vec::with_capacity(aggregate.schema.fields().len()); + for group_by in &aggregate.group_by { + let arrays = extract_column_values(input, group_by)?; + output_arrays.push(interleave_column_values(&arrays, &reps)?); + } + for (agg_idx, op) in ops.iter().enumerate() { + let states = Vec::from_iter(aggs_by_group.iter().map(|aggs| aggs[agg_idx].as_ref())); + output_arrays.push(op.finalize(&states, input)?); + } + + Ok(vec![RecordBatch::try_new(output_schema, output_arrays)?]) +} + +// Extracts the named column from each of the input batches +fn extract_column_values(input: &[RecordBatch], name: &ColumnName) -> DeltaResult> { + input + .iter() + .map(|batch| extract_column(batch, name)) + .try_collect() +} + +// Thin wrapper around arrow `interleave` that converts our `&[ArrayRef]` into `&[&dyn Array]` +fn interleave_column_values(arrays: &[ArrayRef], indices: &[InputRow]) -> DeltaResult { + let refs = Vec::from_iter(arrays.iter().map(|c| c.as_ref())); + Ok(interleave(&refs, indices)?) +} + +fn bind_aggregate(agg: &Agg, output_type: &DataType) -> DeltaResult> { + match agg { + Agg::Min(value) => { + let op = LongAccumulator::MinMax(Comparison::Min); + LongAccumulatorAgg::try_new(value, output_type, op) + } + Agg::Max(value) => { + let op = LongAccumulator::MinMax(Comparison::Max); + LongAccumulatorAgg::try_new(value, output_type, op) + } + Agg::Sum(value) => LongAccumulatorAgg::try_new(value, output_type, LongAccumulator::Sum), + Agg::Count(value) => CountAgg::try_new(Some(value), output_type), + Agg::CountStar => CountAgg::try_new(None, output_type), + Agg::MinNonNullBy(ops) => NonNullByAgg::try_new(ops, output_type, Comparison::Min), + Agg::MaxNonNullBy(ops) => NonNullByAgg::try_new(ops, output_type, Comparison::Max), + } +} + +// Used by both Min/Max and Min/MaxNonNullBy +#[derive(Clone, Copy)] +enum Comparison { + Min, + Max, +} + +impl Comparison { + fn replaces(self, winner: Option, candidate: i64) -> bool { + match self { + Self::Min => winner.is_none_or(|best| candidate < best), + Self::Max => winner.is_none_or(|best| candidate > best), + } + } +} + +#[derive(Clone, Copy)] +enum LongAccumulator { + MinMax(Comparison), + Sum, +} + +struct LongAccumulatorAgg { + value: ColumnName, + op: LongAccumulator, +} + +impl LongAccumulatorAgg { + fn try_new( + value: &ColumnName, + output_type: &DataType, + op: LongAccumulator, + ) -> DeltaResult> { + if output_type != &DataType::LONG { + return Err(Error::unsupported( + "SyncPlanExecutor min/max/sum aggregate with non-LONG value", + )); + } + Ok(Box::new(Self { + value: value.clone(), + op, + })) + } +} + +struct LongAccumulatorState(Option); + +impl BoundAggregate for LongAccumulatorAgg { + fn init_state(&self) -> Box { + Box::new(LongAccumulatorState(None)) + } + + fn prepare<'a>(&'a self, batch: &'a RecordBatch) -> DeltaResult> { + Ok(Box::new(LongAccumulatorUpdater { + values: extract_long_column(batch, &self.value)?, + op: self.op, + })) + } + + fn finalize(&self, states: &[&dyn Any], _input: &[RecordBatch]) -> DeltaResult { + let values = states + .iter() + .map(|state| Ok(downcast_state::(*state)?.0)) + .collect::>>()?; + Ok(Arc::new(Int64Array::from(values))) + } +} + +struct LongAccumulatorUpdater<'a> { + values: &'a Int64Array, + op: LongAccumulator, +} + +impl AggUpdater for LongAccumulatorUpdater<'_> { + fn update(&self, state: &mut dyn Any, (_, row_idx): InputRow) -> DeltaResult<()> { + if self.values.is_valid(row_idx) { + let state = downcast_state_mut::(state)?; + let candidate = self.values.value(row_idx); + match self.op { + LongAccumulator::MinMax(cmp) => { + if cmp.replaces(state.0, candidate) { + state.0 = Some(candidate); + } + } + LongAccumulator::Sum => { + state.0 = Some(match state.0 { + None => candidate, + Some(sum) => i64::checked_add(sum, candidate).ok_or_else(|| { + Error::generic("SyncPlanExecutor SUM aggregate overflowed i64") + })?, + }); + } + } + } + Ok(()) + } +} + +/// COUNT non-null values of Some column, else count rows as COUNT(*) +struct CountAgg(Option); + +impl CountAgg { + fn try_new( + value: Option<&ColumnName>, + output_type: &DataType, + ) -> DeltaResult> { + if output_type != &DataType::LONG { + return Err(Error::unsupported( + "SyncPlanExecutor count aggregate with non-LONG output", + )); + } + Ok(Box::new(Self(value.cloned()))) + } +} + +struct CountState(i64); + +impl BoundAggregate for CountAgg { + fn init_state(&self) -> Box { + Box::new(CountState(0)) + } + + fn prepare<'a>(&'a self, batch: &'a RecordBatch) -> DeltaResult> { + Ok(Box::new(CountUpdater(match &self.0 { + Some(name) => Some(extract_column_ref(batch, name)?), + None => None, + }))) + } + + fn finalize(&self, states: &[&dyn Any], _input: &[RecordBatch]) -> DeltaResult { + let values = states + .iter() + .map(|state| Ok(downcast_state::(*state)?.0)) + .collect::>>()?; + Ok(Arc::new(Int64Array::from(values))) + } +} + +struct CountUpdater<'a>(Option<&'a ArrayRef>); + +impl AggUpdater for CountUpdater<'_> { + fn update(&self, state: &mut dyn Any, (_, row_idx): InputRow) -> DeltaResult<()> { + if self.0.is_none_or(|values| values.is_valid(row_idx)) { + let state = downcast_state_mut::(state)?; + state.0 = i64::checked_add(state.0, 1) + .ok_or_else(|| Error::generic("SyncPlanExecutor COUNT aggregate overflowed i64"))?; + } + Ok(()) + } +} + +/// Min/max-by selector: compare LONG keys among rows with a non-null sentinel, gather `value`. +struct NonNullByAgg { + value: ColumnName, + null_sentinel: ColumnName, + key: ColumnName, + comparison: Comparison, + output_type: ArrowDataType, +} + +impl NonNullByAgg { + fn try_new( + operands: &NonNullByOperands, + output_type: &DataType, + comparison: Comparison, + ) -> DeltaResult> { + Ok(Box::new(Self { + value: operands.value.clone(), + null_sentinel: operands.null_sentinel.clone(), + key: operands.key.clone(), + comparison, + output_type: output_type.try_into_arrow()?, + })) + } +} + +struct NonNullByState(Option<(InputRow, i64)>); + +impl BoundAggregate for NonNullByAgg { + fn init_state(&self) -> Box { + Box::new(NonNullByState(None)) + } + + fn prepare<'a>(&'a self, batch: &'a RecordBatch) -> DeltaResult> { + Ok(Box::new(NonNullByUpdater { + null_sentinels: extract_column_ref(batch, &self.null_sentinel)?, + keys: extract_long_column(batch, &self.key)?, + comparison: self.comparison, + })) + } + + fn finalize(&self, states: &[&dyn Any], input: &[RecordBatch]) -> DeltaResult { + let mut arrays = extract_column_values(input, &self.value)?; + arrays.push(new_null_array(&self.output_type, 1)); + + // Groups with no winner interleave from the one-row null array we appended above. + let initial_row = (input.len(), 0); + let rows: Vec<_> = states + .iter() + .map(|state| -> DeltaResult<_> { + let winner = downcast_state::(*state)?.0; + Ok(winner.map_or(initial_row, |(row, _)| row)) + }) + .try_collect()?; + + interleave_column_values(&arrays, &rows) + } +} + +struct NonNullByUpdater<'a> { + null_sentinels: &'a ArrayRef, + keys: &'a Int64Array, + comparison: Comparison, +} + +impl AggUpdater for NonNullByUpdater<'_> { + fn update(&self, state: &mut dyn Any, (batch_idx, row_idx): InputRow) -> DeltaResult<()> { + if self.null_sentinels.is_valid(row_idx) && self.keys.is_valid(row_idx) { + let state = downcast_state_mut::(state)?; + let best = state.0.map(|(_, best)| best); + let candidate = self.keys.value(row_idx); + if self.comparison.replaces(best, candidate) { + state.0 = Some(((batch_idx, row_idx), candidate)); + } + } + Ok(()) + } +} + +fn extract_long_column<'a>( + batch: &'a RecordBatch, + name: &ColumnName, +) -> DeltaResult<&'a Int64Array> { + let array = extract_column_ref(batch, name)?; + array.as_any().downcast_ref::().ok_or_else(|| { + Error::unsupported(format!( + "SyncPlanExecutor aggregate operand `{name}` has non-LONG type" + )) + }) +} + +fn downcast_state(state: &dyn Any) -> DeltaResult<&T> { + state.downcast_ref().ok_or_else(|| { + Error::generic(format!( + "Aggregate state is not a {}", + std::any::type_name::() + )) + }) +} + +fn downcast_state_mut(state: &mut dyn Any) -> DeltaResult<&mut T> { + state.downcast_mut().ok_or_else(|| { + Error::generic(format!( + "Aggregate state is not a {}", + std::any::type_name::() + )) + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::arrow::array::{BooleanArray, StringArray, StructArray}; + use crate::arrow::datatypes::Field; + use crate::arrow::util::pretty::pretty_format_batches; + use crate::expressions::column_name; + use crate::schema::schema_ref; + + /// Asserts `batches` pretty-print equal to `expected`, sorting data rows so HashMap group order + /// does not matter. `expected` is the full pretty table (header, body, footer). + fn assert_batches_eq(batches: &[RecordBatch], expected: &str) { + let formatted = pretty_format_batches(batches).unwrap().to_string(); + let sort_body = |s: &str| -> String { + let mut lines: Vec<_> = s.trim().lines().map(str::to_string).collect(); + let len = lines.len(); + if len > 3 { + lines[2..len - 1].sort_unstable(); + } + lines.join("\n") + }; + assert_eq!(sort_body(&formatted), sort_body(expected)); + } + + #[test] + fn grouped_aggregate_routes_rows_to_every_agg() -> DeltaResult<()> { + let input = RecordBatch::try_from_iter([ + ( + "group", + Arc::new(StringArray::from(vec!["a", "a", "b"])) as ArrayRef, + ), + ( + "value", + Arc::new(StringArray::from(vec!["low", "high", "none"])), + ), + ( + "sentinel", + Arc::new(BooleanArray::from(vec![Some(true), Some(true), None])), + ), + ( + "key", + Arc::new(Int64Array::from(vec![Some(1), Some(3), None])), + ), + ])?; + let aggregate = Aggregate { + group_by: vec![column_name!("group")], + aggs: vec![ + Agg::max_non_null_by( + column_name!("value"), + column_name!("sentinel"), + column_name!("key"), + ), + Agg::min_non_null_by( + column_name!("value"), + column_name!("sentinel"), + column_name!("key"), + ), + Agg::max(column_name!("key")), + Agg::min(column_name!("key")), + Agg::sum(column_name!("key")), + Agg::count(column_name!("sentinel")), + Agg::count_star(), + ], + schema: schema_ref! { + not_null "group": STRING, + nullable "maximum": STRING, + nullable "minimum": STRING, + nullable "max_key": LONG, + nullable "min_key": LONG, + nullable "sum_key": LONG, + not_null "qualified": LONG, + not_null "rows": LONG, + }, + }; + + assert_batches_eq( + &eval_aggregate(&aggregate, &[input])?, + "\ ++-------+---------+---------+---------+---------+---------+-----------+------+ +| group | maximum | minimum | max_key | min_key | sum_key | qualified | rows | ++-------+---------+---------+---------+---------+---------+-----------+------+ +| a | high | low | 3 | 1 | 4 | 2 | 2 | +| b | | | | | | 0 | 1 | ++-------+---------+---------+---------+---------+---------+-----------+------+", + ); + Ok(()) + } + + #[test] + fn non_null_by_selects_across_batches() -> DeltaResult<()> { + let aggregate = Aggregate { + group_by: vec![], + aggs: vec![ + Agg::min_non_null_by( + column_name!("value"), + column_name!("sentinel"), + column_name!("key"), + ), + Agg::max_non_null_by( + column_name!("value"), + column_name!("sentinel"), + column_name!("key"), + ), + ], + schema: schema_ref! { + nullable "minimum": STRING, + nullable "maximum": STRING, + }, + }; + let first = RecordBatch::try_from_iter([ + ( + "value", + Arc::new(StringArray::from(vec!["low"])) as ArrayRef, + ), + ("sentinel", Arc::new(BooleanArray::from(vec![true]))), + ("key", Arc::new(Int64Array::from(vec![1]))), + ])?; + let second = RecordBatch::try_from_iter([ + ( + "value", + Arc::new(StringArray::from(vec!["high"])) as ArrayRef, + ), + ("sentinel", Arc::new(BooleanArray::from(vec![true]))), + ("key", Arc::new(Int64Array::from(vec![2]))), + ])?; + + assert_batches_eq( + &eval_aggregate(&aggregate, &[first, second])?, + "\ ++---------+---------+ +| minimum | maximum | ++---------+---------+ +| low | high | ++---------+---------+", + ); + Ok(()) + } + + #[test] + fn empty_input_distinguishes_grouped_and_ungrouped_aggregates() -> DeltaResult<()> { + let ungrouped = Aggregate { + group_by: vec![], + aggs: vec![ + Agg::min(column_name!("key")), + Agg::max(column_name!("key")), + Agg::sum(column_name!("key")), + Agg::count(column_name!("sentinel")), + Agg::count_star(), + Agg::min_non_null_by( + column_name!("value"), + column_name!("sentinel"), + column_name!("key"), + ), + Agg::max_non_null_by( + column_name!("value"), + column_name!("sentinel"), + column_name!("key"), + ), + ], + schema: schema_ref! { + nullable "minimum": LONG, + nullable "maximum": LONG, + nullable "total": LONG, + not_null "qualified": LONG, + not_null "rows": LONG, + nullable "first": STRING, + nullable "last": STRING, + }, + }; + assert_batches_eq( + &eval_aggregate(&ungrouped, &[])?, + "\ ++---------+---------+-------+-----------+------+-------+------+ +| minimum | maximum | total | qualified | rows | first | last | ++---------+---------+-------+-----------+------+-------+------+ +| | | | 0 | 0 | | | ++---------+---------+-------+-----------+------+-------+------+", + ); + + let grouped = Aggregate { + group_by: vec![column_name!("group")], + aggs: vec![Agg::max(column_name!("key"))], + schema: schema_ref! { + not_null "group": STRING, + nullable "max_key": LONG, + }, + }; + assert!(eval_aggregate(&grouped, &[])?.is_empty()); + Ok(()) + } + + #[rstest::rstest] + #[case::all_null( + &[None, None], + "\ ++-------+-----------+------+ +| total | non_nulls | rows | ++-------+-----------+------+ +| | 0 | 2 | ++-------+-----------+------+" + )] + #[case::some_null( + &[None, Some(3), Some(1)], + "\ ++-------+-----------+------+ +| total | non_nulls | rows | ++-------+-----------+------+ +| 4 | 2 | 3 | ++-------+-----------+------+" + )] + #[case::all_non_null( + &[Some(3), Some(1), Some(2)], + "\ ++-------+-----------+------+ +| total | non_nulls | rows | ++-------+-----------+------+ +| 6 | 3 | 3 | ++-------+-----------+------+" + )] + fn sum_and_count_null_patterns( + #[case] values: &[Option], + #[case] expected: &str, + ) -> DeltaResult<()> { + let input = RecordBatch::try_from_iter([( + "value", + Arc::new(Int64Array::from(values.to_vec())) as ArrayRef, + )])?; + let aggregate = Aggregate { + group_by: vec![], + aggs: vec![ + Agg::sum(column_name!("value")), + Agg::count(column_name!("value")), + Agg::count_star(), + ], + schema: schema_ref! { + nullable "total": LONG, + not_null "non_nulls": LONG, + not_null "rows": LONG, + }, + }; + assert_batches_eq(&eval_aggregate(&aggregate, &[input])?, expected); + Ok(()) + } + + #[test] + fn count_over_struct_counts_non_null_structs() -> DeltaResult<()> { + let nested = Arc::new(StructArray::from(vec![( + Arc::new(Field::new("x", ArrowDataType::Int64, true)), + Arc::new(Int64Array::from(vec![Some(1), None, Some(3)])) as ArrayRef, + )])); + // Make the middle struct itself NULL while keeping child arrays aligned. + let validity = crate::arrow::buffer::BooleanBuffer::from(vec![true, false, true]); + let structs = Arc::new(StructArray::new( + nested.fields().clone(), + nested.columns().to_vec(), + Some(crate::arrow::buffer::NullBuffer::new(validity)), + )); + let input = RecordBatch::try_from_iter([("payload", structs as ArrayRef)])?; + let aggregate = Aggregate { + group_by: vec![], + aggs: vec![Agg::count(column_name!("payload")), Agg::count_star()], + schema: schema_ref! { + not_null "payloads": LONG, + not_null "rows": LONG, + }, + }; + + assert_batches_eq( + &eval_aggregate(&aggregate, &[input])?, + "\ ++----------+------+ +| payloads | rows | ++----------+------+ +| 2 | 3 | ++----------+------+", + ); + Ok(()) + } + + #[test] + fn aggregate_resolves_nested_columns_over_top_level_traps() -> DeltaResult<()> { + let outer = StructArray::from(vec![ + ( + Arc::new(Field::new("group", ArrowDataType::Utf8, false)), + Arc::new(StringArray::from(vec!["nested_group"])) as ArrayRef, + ), + ( + Arc::new(Field::new("value", ArrowDataType::Utf8, false)), + Arc::new(StringArray::from(vec!["nested_value"])) as ArrayRef, + ), + ( + Arc::new(Field::new("sentinel", ArrowDataType::Boolean, true)), + Arc::new(BooleanArray::from(vec![Some(true)])) as ArrayRef, + ), + ( + Arc::new(Field::new("key", ArrowDataType::Int64, true)), + Arc::new(Int64Array::from(vec![Some(7)])) as ArrayRef, + ), + ]); + let input = RecordBatch::try_from_iter([ + ( + "group", + Arc::new(StringArray::from(vec!["trap_group"])) as ArrayRef, + ), + ("value", Arc::new(StringArray::from(vec!["trap_value"]))), + ("sentinel", Arc::new(BooleanArray::from(vec![None]))), + ("key", Arc::new(Int64Array::from(vec![Some(99)]))), + ("outer", Arc::new(outer)), + ])?; + let aggregate = Aggregate { + group_by: vec![column_name!("outer.group")], + aggs: vec![ + Agg::min(column_name!("outer.key")), + Agg::count(column_name!("outer.sentinel")), + Agg::max_non_null_by( + column_name!("outer.value"), + column_name!("outer.sentinel"), + column_name!("outer.key"), + ), + ], + schema: schema_ref! { + not_null "group": STRING, + nullable "minimum": LONG, + not_null "qualified": LONG, + nullable "winner": STRING, + }, + }; + + assert_batches_eq( + &eval_aggregate(&aggregate, &[input])?, + "\ ++--------------+---------+-----------+--------------+ +| group | minimum | qualified | winner | ++--------------+---------+-----------+--------------+ +| nested_group | 7 | 1 | nested_value | ++--------------+---------+-----------+--------------+", + ); + Ok(()) + } + + #[test] + fn non_null_by_can_select_a_null_value() -> DeltaResult<()> { + let input = RecordBatch::try_from_iter([ + ( + "value", + Arc::new(StringArray::from(vec![Some("fallback"), None])) as ArrayRef, + ), + ("sentinel", Arc::new(BooleanArray::from(vec![true, true]))), + ("key", Arc::new(Int64Array::from(vec![1, 2]))), + ])?; + let aggregate = Aggregate { + group_by: vec![], + aggs: vec![Agg::max_non_null_by( + column_name!("value"), + column_name!("sentinel"), + column_name!("key"), + )], + schema: schema_ref! { + nullable "winner": STRING, + }, + }; + + assert_batches_eq( + &eval_aggregate(&aggregate, &[input])?, + "\ ++--------+ +| winner | ++--------+ +| | ++--------+", + ); + Ok(()) + } +} diff --git a/kernel/src/engine/sync/json.rs b/kernel/src/engine/sync/json.rs index cc4b2965f9..ce5a3743d4 100644 --- a/kernel/src/engine/sync/json.rs +++ b/kernel/src/engine/sync/json.rs @@ -1,39 +1,45 @@ +use std::io::{BufReader, Cursor}; use std::sync::Arc; -use std::{fs::File, io::BufReader, io::Write}; -use crate::arrow::json::ReaderBuilder; -use tempfile::NamedTempFile; +use bytes::Bytes; use url::Url; -use super::read_files; +use super::{put_bytes, read_files_arrow}; +use crate::arrow::json::ReaderBuilder; use crate::engine::arrow_data::ArrowEngineData; use crate::engine::arrow_utils::{ build_json_reorder_indices, fixup_json_read, json_arrow_schema, parse_json as arrow_parse_json, to_json_bytes, }; use crate::engine_data::FilteredEngineData; +use crate::object_store::DynObjectStore; use crate::schema::SchemaRef; use crate::{ - DeltaResult, EngineData, Error, FileDataReadResultIterator, FileMeta, JsonHandler, PredicateRef, + DeltaResult, DeltaResultIterator, EngineData, Error, FileDataReadResultIterator, FileMeta, + JsonHandler, PredicateRef, }; -pub(crate) struct SyncJsonHandler; +pub(crate) struct SyncJsonHandler { + store: Option>, +} + +impl SyncJsonHandler { + pub(crate) fn new(store: Option>) -> Self { + Self { store } + } +} -fn try_create_from_json( - file: File, +pub(super) fn try_create_from_json( + data: Bytes, schema: SchemaRef, _predicate: Option, file_location: String, ) -> DeltaResult>> { - // Build Arrow schema from only the real JSON columns, omitting any metadata columns - // (e.g. FilePath) that the JSON reader cannot populate from the file content. let json_schema = Arc::new(json_arrow_schema(&schema)?); - // Build the reorder index vec once; apply it to every batch to re-insert synthesized metadata - // columns (e.g. file path) at their schema positions. let reorder_indices = build_json_reorder_indices(&schema)?; let json = ReaderBuilder::new(json_schema) .with_coerce_primitive(true) - .build(BufReader::new(file))? + .build(BufReader::new(Cursor::new(data)))? .map(move |data| fixup_json_read(data?, &reorder_indices, &file_location)); Ok(json) } @@ -45,7 +51,14 @@ impl JsonHandler for SyncJsonHandler { schema: SchemaRef, predicate: Option, ) -> DeltaResult { - read_files(files, schema, predicate, try_create_from_json) + let iter = read_files_arrow( + self.store.as_ref(), + files, + schema, + predicate, + try_create_from_json, + ); + Ok(Box::new(iter.map(|data| Ok(Box::new(data?) as _)))) } fn parse_json( @@ -56,64 +69,30 @@ impl JsonHandler for SyncJsonHandler { arrow_parse_json(json_strings, output_schema) } - // For sync writer we write data to a tmp file then atomically rename it to the final path. - // This is highly OS-dependent and for now relies on the atomicity of tempfile's - // `persist_noclobber`. fn write_json_file( &self, path: &Url, - data: Box> + Send + '_>, + data: DeltaResultIterator<'_, FilteredEngineData>, overwrite: bool, ) -> DeltaResult<()> { - let path = path - .to_file_path() - .map_err(|_| crate::Error::generic("sync client can only read local files"))?; - let Some(parent) = path.parent() else { - return Err(crate::Error::generic(format!( - "no parent found for {path:?}" - ))); - }; - - if !parent.exists() { - std::fs::create_dir_all(parent)?; - } - - // write data to tmp file - let mut tmp_file = NamedTempFile::new_in(parent)?; let buf = to_json_bytes(data)?; - tmp_file.write_all(&buf)?; - tmp_file.flush()?; - - let persist_result = if overwrite { - tmp_file.persist(path.clone()) - } else { - // use 'persist_noclobber' to atomically rename tmp file to final path - tmp_file.persist_noclobber(path.clone()) - }; - - // Map errors (handling AlreadyExists only in non-overwrite mode). - persist_result.map_err(|e| { - if !overwrite && e.error.kind() == std::io::ErrorKind::AlreadyExists { - Error::FileAlreadyExists(path.to_string_lossy().to_string()) - } else { - Error::IOError(e.into()) - } - })?; - - Ok(()) + put_bytes(self.store.as_ref(), path, buf.into(), overwrite) } } + #[cfg(test)] mod tests { - use super::*; - use crate::arrow::array::{RecordBatch, StringArray}; - use crate::arrow::datatypes::{DataType as ArrowDataType, Field, Schema as ArrowSchema}; - use serde_json::json; use std::path::Path; use std::sync::Arc; + + use serde_json::json; use tempfile::TempDir; use url::Url; + use super::*; + use crate::arrow::array::{RecordBatch, StringArray}; + use crate::arrow::datatypes::{DataType as ArrowDataType, Field, Schema as ArrowSchema}; + // Helper function to create test data fn create_test_data(values: Vec<&str>) -> DeltaResult> { let schema = Arc::new(ArrowSchema::new(vec![Field::new( @@ -149,7 +128,7 @@ mod tests { fn do_test_write_json_file(overwrite: bool) -> DeltaResult<()> { let test_dir = TempDir::new().unwrap(); let path = test_dir.path().join("00000000000000000001.json"); - let handler = SyncJsonHandler; + let handler = SyncJsonHandler::new(None); let url = Url::from_file_path(&path).unwrap(); // First write with no existing file @@ -176,14 +155,17 @@ mod tests { assert_eq!(json, vec![json!({"dog": "seb"}), json!({"dog": "tia"})]); } else { // Verify the second write fails with FileAlreadyExists error - match result { - Err(Error::FileAlreadyExists(err_path)) => { - assert_eq!(err_path, path.to_string_lossy().to_string()); - } - _ => panic!("Expected FileAlreadyExists error, got: {result:?}"), - } + assert!(matches!(result, Err(Error::FileAlreadyExists(_)))); } Ok(()) } + + // TODO(#2618): Restore once the engine contract helpers move to test_utils and SyncEngine can + // call them without the kernel-cfg-test cycle issue. + // + // #[test] + // fn json_handler_file_path_contract() { + // test_json_handler_file_path_contract(&SyncJsonHandler::new(None)); + // } } diff --git a/kernel/src/engine/sync/mod.rs b/kernel/src/engine/sync/mod.rs index 15a7e8a1b7..0efdbc62e6 100644 --- a/kernel/src/engine/sync/mod.rs +++ b/kernel/src/engine/sync/mod.rs @@ -1,37 +1,78 @@ -//! A simple, single threaded, test-only [`Engine`] that can only read from the local filesystem +//! A simple, single threaded, test-only `Engine`. +//! +//! All I/O goes through an `ObjectStore`. `SyncEngine::new` uses a [`LocalFileSystem`] +//! built lazily per-URL (rooted at the URL's drive on Windows, or `/` on Unix), so any +//! `file://` URL is supported. `SyncEngine::new_with_store` takes any other store +//! (e.g. `InMemory`) and uses it directly for all URLs. +//! +//! Async work is driven via [`futures::executor::block_on`], which is sufficient for stores +//! that do not require a tokio reactor (`LocalFileSystem`, `InMemory`). Cloud-backed stores +//! are NOT supported here -- they would deadlock because `block_on` parks the calling thread +//! with no reactor to wake it. That's acceptable for this test-only engine; production code +//! should use `DefaultEngine` from the `delta_kernel_default_engine` crate instead. +//! +//! [`LocalFileSystem`]: crate::object_store::local::LocalFileSystem + +use std::sync::Arc; + +use bytes::Bytes; +use itertools::Itertools; +use tracing::debug; +use url::Url; use super::arrow_expression::ArrowEvaluationHandler; use crate::engine::arrow_data::ArrowEngineData; +use crate::object_store::local::LocalFileSystem; +use crate::object_store::path::Path; +use crate::object_store::{DynObjectStore, ObjectStoreExt as _}; use crate::{ - DeltaResult, Engine, Error, EvaluationHandler, FileDataReadResultIterator, FileMeta, - JsonHandler, ParquetHandler, PredicateRef, SchemaRef, StorageHandler, + DeltaResult, Engine, Error, EvaluationHandler, FileMeta, JsonHandler, ParquetHandler, + PredicateRef, SchemaRef, StorageHandler, }; -use itertools::Itertools; -use std::fs::File; -use std::sync::Arc; -use tracing::debug; - pub(crate) mod json; mod parquet; -pub(crate) use parquet::SyncParquetHandler; mod storage; -/// This is a simple (test-only) implementation of [`Engine`]. It only supports reading data from -/// the local filesystem, and internally represents data using `Arrow`. +#[cfg(feature = "declarative-plans")] +mod aggs; + +#[cfg(feature = "declarative-plans")] +pub(crate) mod plan; + +#[cfg(feature = "declarative-plans")] +use plan::SyncPlanExecutor; + +/// A simple (test-only) implementation of [`Engine`]. See module docs for supported stores. pub(crate) struct SyncEngine { storage_handler: Arc, json_handler: Arc, parquet_handler: Arc, evaluation_handler: Arc, + #[cfg(feature = "declarative-plans")] + plan_executor: Arc, } impl SyncEngine { + /// Create a SyncEngine that reads from the local filesystem via [`LocalFileSystem`]. pub(crate) fn new() -> Self { + Self::new_inner(None) + } + + /// Create a SyncEngine backed by `store`. All I/O is performed synchronously via + /// [`futures::executor::block_on`]. See module docs for the deadlock caveat on + /// reactor-dependent stores. + pub(crate) fn new_with_store(store: Arc) -> Self { + Self::new_inner(Some(store)) + } + + fn new_inner(store: Option>) -> Self { SyncEngine { - storage_handler: Arc::new(storage::SyncStorageHandler {}), - json_handler: Arc::new(json::SyncJsonHandler {}), - parquet_handler: Arc::new(parquet::SyncParquetHandler {}), + storage_handler: Arc::new(storage::SyncStorageHandler::new(store.clone())), + #[cfg(feature = "declarative-plans")] + plan_executor: Arc::new(plan::SyncPlanExecutor::new(store.clone())), + json_handler: Arc::new(json::SyncJsonHandler::new(store.clone())), + parquet_handler: Arc::new(parquet::SyncParquetHandler::new(store)), evaluation_handler: Arc::new(ArrowEvaluationHandler {}), } } @@ -46,7 +87,6 @@ impl Engine for SyncEngine { self.storage_handler.clone() } - /// Get the connector provided [`ParquetHandler`]. fn parquet_handler(&self) -> Arc { self.parquet_handler.clone() } @@ -54,57 +94,175 @@ impl Engine for SyncEngine { fn json_handler(&self) -> Arc { self.json_handler.clone() } + + #[cfg(feature = "declarative-plans")] + fn plan_executor(&self) -> Option> { + Some(self.plan_executor.clone()) + } } -fn read_files( +/// Resolve the store, base URL, and path for `url`. +/// +/// When `default_store` is `Some`, the user-provided store is returned with `url`'s decoded +/// path. The base URL is `url` with its path replaced by `/`, suitable for re-joining +/// listed [`Path`]s back into URLs. +/// +/// When `default_store` is `None`, only `file://` URLs are accepted: a [`LocalFileSystem`] +/// is constructed rooted at the URL's resolved directory (the URL itself if it ends with +/// `/`, otherwise the parent), and the returned path is the simple filename (or empty for +/// directory URLs). Rooting at the parent avoids a url-crate quirk where path segments +/// containing reserved-looking characters (e.g. `:` in Windows drive letters or `~` in 8.3 +/// short names like `RUNNER~1`) get URL-encoded by `path_segments_mut.extend(...)` but not +/// decoded back by `to_file_path`. By keeping such characters inside the store's base URL +/// (which is built once from a `PathBuf` and not re-processed), only simple filenames flow +/// through the broken round-trip. +pub(super) fn resolve_scope( + default_store: Option<&Arc>, + url: &Url, +) -> DeltaResult<(Arc, Url, Path)> { + if let Some(store) = default_store { + let mut base_url = url.clone(); + base_url.set_path("/"); + let path = Path::from_url_path(url.path())?; + return Ok((store.clone(), base_url, path)); + } + if url.scheme() != "file" { + return Err(Error::generic(format!( + "SyncEngine without an explicit store can only access file:// URLs, got: {url}" + ))); + } + let file_path = url + .to_file_path() + .map_err(|()| Error::generic(format!("Invalid file URL: {url}")))?; + // Use the deepest existing ancestor of the URL's directory as the store prefix: + // `LocalFileSystem::new_with_prefix` canonicalizes its argument (which requires the path + // to exist), and operating on a non-existent path is a valid case for `list_from` on a + // not-yet-written `_delta_log/` directory or `head` on a missing file. + let target_dir = if url.path().ends_with('/') { + file_path.clone() + } else { + file_path + .parent() + .ok_or_else(|| Error::generic(format!("File URL has no parent: {url}")))? + .to_path_buf() + }; + let mut prefix = target_dir.as_path(); + while !prefix.exists() { + prefix = prefix + .parent() + .ok_or_else(|| Error::generic(format!("No existing ancestor for {target_dir:?}")))?; + } + let prefix = prefix.to_path_buf(); + let relative = file_path + .strip_prefix(&prefix) + .map_err(|e| Error::generic(format!("Failed to strip prefix: {e}")))?; + let path = Path::from_iter(relative.components().filter_map(|c| match c { + std::path::Component::Normal(s) => s.to_str().map(String::from), + _ => None, + })); + let base_url = Url::from_directory_path(&prefix) + .map_err(|()| Error::generic(format!("Could not URL-encode prefix {prefix:?}")))?; + let store: Arc = Arc::new(LocalFileSystem::new_with_prefix(&prefix)?); + Ok((store, base_url, path)) +} + +/// Fetch the contents of a file via [`resolve_scope`] and return them as bytes. +pub(super) fn get_bytes( + default_store: Option<&Arc>, + location: &Url, +) -> DeltaResult { + let (store, _, path) = resolve_scope(default_store, location)?; + let get_result = futures::executor::block_on(store.get(&path))?; + Ok(futures::executor::block_on(get_result.bytes())?) +} + +/// Write `data` to `location` via [`resolve_scope`]. +/// +/// For `file://` URLs the parent directory is created if missing (matching the local FS +/// behavior callers expect; `LocalFileSystem::put` itself does not create parents). When +/// `overwrite` is false, an existing file at `location` produces [`Error::FileAlreadyExists`]. +pub(super) fn put_bytes( + default_store: Option<&Arc>, + location: &Url, + data: Bytes, + overwrite: bool, +) -> DeltaResult<()> { + if location.scheme() == "file" { + if let Ok(file_path) = location.to_file_path() { + if let Some(parent) = file_path.parent() { + if !parent.exists() { + std::fs::create_dir_all(parent)?; + } + } + } + } + let (store, _, object_path) = resolve_scope(default_store, location)?; + let opts = if overwrite { + crate::object_store::PutOptions::default() + } else { + crate::object_store::PutOptions { + mode: crate::object_store::PutMode::Create, + ..Default::default() + } + }; + futures::executor::block_on(store.put_opts(&object_path, data.into(), opts)).map_err(|e| { + match e { + crate::object_store::Error::AlreadyExists { .. } => { + Error::FileAlreadyExists(location.to_string()) + } + other => Error::generic(other.to_string()), + } + })?; + Ok(()) +} + +/// Read each file as bytes and feed it to `try_create_from_bytes` to produce data batches. +fn read_files_arrow( + store: Option<&Arc>, files: &[FileMeta], schema: SchemaRef, predicate: Option, - mut try_create_from_file: F, -) -> DeltaResult + mut try_create_from_bytes: F, +) -> impl Iterator> + Send + 'static where I: Iterator> + Send + 'static, - F: FnMut(File, SchemaRef, Option, String) -> DeltaResult + Send + 'static, + F: FnMut(Bytes, SchemaRef, Option, String) -> DeltaResult + Send + 'static, { debug!("Reading files: {files:#?} with schema {schema:#?} and predicate {predicate:#?}"); - if files.is_empty() { - return Ok(Box::new(std::iter::empty())); - } - let files = files.to_vec(); - let result = files + let files = files.to_vec(); // Clone for static iterator (clippy hates chained to_vec+into_iter) + let store = store.cloned(); + files .into_iter() - // Produces Iterator>>> .map(move |file| { let location_string = file.location.to_string(); - let location = file.location; - debug!("Reading {location:#?} with schema {schema:#?} and predicate {predicate:#?}"); - let path = location - .to_file_path() - .map_err(|_| Error::generic("can only read local files"))?; - try_create_from_file( - File::open(path)?, - schema.clone(), - predicate.clone(), - location_string, - ) + let bytes = get_bytes(store.as_ref(), &file.location)?; + try_create_from_bytes(bytes, schema.clone(), predicate.clone(), location_string) }) - // Flatten to Iterator>> .flatten_ok() - // Double unpack and map Iterator>> - .map(|data| Ok(Box::new(ArrowEngineData::new(data??.into())) as _)); - Ok(Box::new(result)) + .map(|data| data?) } -#[cfg(test)] -mod tests { - use super::*; - use crate::engine::tests::test_arrow_engine; - - #[test] - fn test_sync_engine() { - let tmp = tempfile::tempdir().unwrap(); - let url = url::Url::from_directory_path(tmp.path()).unwrap(); - let engine = SyncEngine::new(); - test_arrow_engine(&engine, &url); - } -} +// TODO(#2618): Restore once the engine contract helpers move to test_utils and SyncEngine can +// call them without the kernel-cfg-test cycle issue. +// +// #[cfg(test)] +// mod tests { +// use super::*; +// use crate::engine::tests::test_arrow_engine; +// +// #[test] +// fn test_sync_engine() { +// let tmp = tempfile::tempdir().unwrap(); +// let url = Url::from_directory_path(tmp.path()).unwrap(); +// let engine = SyncEngine::new(); +// test_arrow_engine(&engine, &url); +// } +// +// #[test] +// fn test_sync_engine_with_store() { +// let store = Arc::new(crate::object_store::memory::InMemory::new()); +// let engine = SyncEngine::new_with_store(store); +// let url = Url::parse("memory:///test/").unwrap(); +// test_arrow_engine(&engine, &url); +// } +// } diff --git a/kernel/src/engine/sync/parquet.rs b/kernel/src/engine/sync/parquet.rs index 2d183be81c..67612435e7 100644 --- a/kernel/src/engine/sync/parquet.rs +++ b/kernel/src/engine/sync/parquet.rs @@ -1,49 +1,53 @@ -use std::fs::File; use std::sync::Arc; -use crate::engine::default::parquet::{reader_options, writer_options}; -use crate::parquet::arrow::arrow_reader::{ArrowReaderMetadata, ParquetRecordBatchReaderBuilder}; +use bytes::Bytes; +use url::Url; -use super::read_files; -use crate::engine::arrow_conversion::{TryFromArrow as _, TryIntoArrow as _}; +use super::{get_bytes, put_bytes, read_files_arrow}; +use crate::engine::arrow_conversion::TryFromArrow as _; use crate::engine::arrow_data::ArrowEngineData; use crate::engine::arrow_utils::{ fixup_parquet_read, generate_mask, get_requested_indices, ordering_needs_row_indexes, RowIndexBuilder, }; use crate::engine::parquet_row_group_skipping::ParquetRowGroupSkipping; +use crate::engine::{reader_options, writer_options}; +use crate::object_store::DynObjectStore; +use crate::parquet::arrow::arrow_reader::{ArrowReaderMetadata, ParquetRecordBatchReaderBuilder}; use crate::parquet::arrow::arrow_writer::ArrowWriter; use crate::schema::{SchemaRef, StructType}; use crate::{ - DeltaResult, Error, FileDataReadResultIterator, FileMeta, ParquetFooter, ParquetHandler, - PredicateRef, + DeltaResult, DeltaResultIteratorStatic, EngineData, FileDataReadResultIterator, FileMeta, + ParquetFooter, ParquetHandler, PredicateRef, }; -use url::Url; +pub(crate) struct SyncParquetHandler { + store: Option>, +} -pub(crate) struct SyncParquetHandler; +impl SyncParquetHandler { + pub(crate) fn new(store: Option>) -> Self { + Self { store } + } +} -fn try_create_from_parquet( - file: File, +pub(super) fn try_create_from_parquet( + data: Bytes, schema: SchemaRef, predicate: Option, file_location: String, ) -> DeltaResult>> { - let arrow_schema = Arc::new(schema.as_ref().try_into_arrow()?); - let reader_options = reader_options(); - let metadata = ArrowReaderMetadata::load(&file, reader_options.clone())?; + let metadata = ArrowReaderMetadata::load(&data, reader_options())?; let parquet_schema = metadata.schema(); - let mut builder = ParquetRecordBatchReaderBuilder::try_new_with_options(file, reader_options)?; + let mut builder = ParquetRecordBatchReaderBuilder::new_with_metadata(data, metadata.clone()); let (indices, requested_ordering) = get_requested_indices(&schema, parquet_schema)?; if let Some(mask) = generate_mask(&schema, parquet_schema, builder.parquet_schema(), &indices) { builder = builder.with_projection(mask); } - // Only create RowIndexBuilder if row indexes are actually needed let mut row_indexes = ordering_needs_row_indexes(&requested_ordering) .then(|| RowIndexBuilder::new(builder.metadata().row_groups())); - // Filter row groups and row indexes if a predicate is provided if let Some(predicate) = predicate { builder = builder.with_row_group_filter(predicate.as_ref(), row_indexes.as_mut()); } @@ -56,7 +60,7 @@ fn try_create_from_parquet( &requested_ordering, row_indexes.as_mut(), Some(&file_location), - Some(&arrow_schema), + Some(&schema), ) })) } @@ -68,94 +72,85 @@ impl ParquetHandler for SyncParquetHandler { schema: SchemaRef, predicate: Option, ) -> DeltaResult { - read_files(files, schema, predicate, try_create_from_parquet) + let iter = read_files_arrow( + self.store.as_ref(), + files, + schema, + predicate, + try_create_from_parquet, + ); + Ok(Box::new(iter.map(|data| Ok(Box::new(data?) as _)))) } /// Writes engine data to a Parquet file at the specified location. /// - /// This implementation uses synchronous file I/O to write the Parquet file. + /// Buffers the entire file in memory and `put`s it to the underlying [`ObjectStore`]. /// If a file already exists at the given location, it will be overwritten. /// /// # Parameters /// - /// - `location` - The full URL path where the Parquet file should be written - /// (e.g., `file:///path/to/file.parquet`). + /// - `location` - The full URL path where the Parquet file should be written (e.g. `file:///path/to/file.parquet`). /// - `data` - An iterator of engine data to be written to the Parquet file. - /// - /// # Returns - /// - /// A [`DeltaResult`] indicating success or failure. fn write_parquet_file( &self, location: Url, - mut data: Box>> + Send>, + mut data: DeltaResultIteratorStatic>, ) -> DeltaResult<()> { - // Convert URL to file path - let path = location - .to_file_path() - .map_err(|_| crate::Error::generic(format!("Invalid file URL: {location}")))?; - - let mut file = File::create(&path)?; - - // Get first batch to initialize writer with schema let first_batch = data.next().ok_or_else(|| { crate::Error::generic("Cannot write parquet file with empty data iterator") })??; let first_arrow = ArrowEngineData::try_from_engine_data(first_batch)?; let first_record_batch: crate::arrow::array::RecordBatch = (*first_arrow).into(); + let mut buf = Vec::new(); let mut writer = ArrowWriter::try_new_with_options( - &mut file, + &mut buf, first_record_batch.schema(), writer_options(), )?; writer.write(&first_record_batch)?; - - // Write remaining batches for result in data { let engine_data = result?; let arrow_data = ArrowEngineData::try_from_engine_data(engine_data)?; let batch: crate::arrow::array::RecordBatch = (*arrow_data).into(); writer.write(&batch)?; } + writer.close()?; - writer.close()?; // writer must be closed to write footer - - Ok(()) + put_bytes(self.store.as_ref(), &location, buf.into(), true) } fn read_parquet_footer(&self, file: &FileMeta) -> DeltaResult { - let path = file - .location - .to_file_path() - .map_err(|_| Error::generic("SyncEngine can only read local files"))?; - let file = File::open(path)?; - let metadata = ArrowReaderMetadata::load(&file, reader_options())?; - let schema = StructType::try_from_arrow(metadata.schema().as_ref()) - .map(Arc::new) - .map_err(Error::Arrow)?; - Ok(ParquetFooter { schema }) + parquet_footer(self.store.as_ref(), file) } } +/// Read the [`ParquetFooter`] (schema) of `file`. +pub(super) fn parquet_footer( + store: Option<&Arc>, + file: &FileMeta, +) -> DeltaResult { + let data = get_bytes(store, &file.location)?; + let metadata = ArrowReaderMetadata::load(&data, reader_options())?; + let schema = Arc::new(StructType::try_from_arrow(metadata.schema().as_ref())?); + Ok(ParquetFooter { schema }) +} + #[cfg(test)] mod tests { - use super::*; - use crate::arrow::array::{Array, Int64Array, RecordBatch, StringArray}; - use crate::engine::arrow_conversion::TryIntoKernel as _; + use std::fs::File; use std::sync::Arc; + use tempfile::tempdir; use url::Url; - #[test] - fn test_sync_write_parquet_file() { - let handler = SyncParquetHandler; - let temp_dir = tempdir().unwrap(); - let file_path = temp_dir.path().join("test.parquet"); - let url = Url::from_file_path(&file_path).unwrap(); + use super::*; + use crate::arrow::array::{Array, Int64Array, RecordBatch, StringArray}; + use crate::engine::arrow_conversion::TryIntoKernel as _; + use crate::EngineData; - // Create test data - let engine_data: Box = Box::new(ArrowEngineData::new( + fn test_data_iter() -> DeltaResultIteratorStatic> { + let engine_data: Box = Box::new(ArrowEngineData::new( RecordBatch::try_from_iter(vec![ ( "id", @@ -168,102 +163,22 @@ mod tests { ]) .unwrap(), )); - - // Create iterator with single batch - let data_iter: Box< - dyn Iterator>> + Send, - > = Box::new(std::iter::once(Ok(engine_data))); - - // Write the file - handler.write_parquet_file(url.clone(), data_iter).unwrap(); - - // Verify the file exists - assert!(file_path.exists()); - - // Read it back to verify - let file = File::open(&file_path).unwrap(); - let reader = - crate::parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder::try_new(file) - .unwrap(); - let schema = reader.schema().clone(); - let file_size = std::fs::metadata(&file_path).unwrap().len(); - let file_meta = FileMeta { - location: url, - last_modified: 0, - size: file_size, - }; - - let mut result = handler - .read_parquet_files( - &[file_meta], - Arc::new(schema.try_into_kernel().unwrap()), - None, - ) - .unwrap(); - - let engine_data = result.next().unwrap().unwrap(); - let batch = ArrowEngineData::try_from_engine_data(engine_data).unwrap(); - let record_batch = batch.record_batch(); - - // Verify shape - assert_eq!(record_batch.num_rows(), 3); - assert_eq!(record_batch.num_columns(), 2); - - // Verify content - id column - let id_col = record_batch - .column(0) - .as_any() - .downcast_ref::() - .unwrap(); - assert_eq!(id_col.values(), &[1, 2, 3]); - - // Verify content - name column - let name_col = record_batch - .column(1) - .as_any() - .downcast_ref::() - .unwrap(); - assert_eq!(name_col.value(0), "a"); - assert_eq!(name_col.value(1), "b"); - assert_eq!(name_col.value(2), "c"); - - assert!(result.next().is_none()); + Box::new(std::iter::once(Ok(engine_data))) } #[test] - fn test_sync_write_parquet_file_with_filter() { - let handler = SyncParquetHandler; + fn test_sync_write_parquet_file() { + let handler = SyncParquetHandler::new(None); let temp_dir = tempdir().unwrap(); - let file_path = temp_dir.path().join("test_filtered.parquet"); + let file_path = temp_dir.path().join("test.parquet"); let url = Url::from_file_path(&file_path).unwrap(); - // Create test data with only filtered rows: 1, 3, 5 - let engine_data: Box = Box::new(ArrowEngineData::new( - RecordBatch::try_from_iter(vec![ - ( - "id", - Arc::new(Int64Array::from(vec![1, 3, 5])) as Arc, - ), - ( - "name", - Arc::new(StringArray::from(vec!["a", "c", "e"])) as Arc, - ), - ]) - .unwrap(), - )); - - // Create iterator with single batch - let data_iter: Box< - dyn Iterator>> + Send, - > = Box::new(std::iter::once(Ok(engine_data))); - - // Write the file - handler.write_parquet_file(url.clone(), data_iter).unwrap(); - - // Verify the file exists + handler + .write_parquet_file(url.clone(), test_data_iter()) + .unwrap(); assert!(file_path.exists()); - // Read it back to verify only filtered rows are present + // Read it back to verify let file = File::open(&file_path).unwrap(); let reader = crate::parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder::try_new(file) @@ -288,39 +203,35 @@ mod tests { let batch = ArrowEngineData::try_from_engine_data(engine_data).unwrap(); let record_batch = batch.record_batch(); - // Verify shape - should only have 3 rows (filtered from 5) assert_eq!(record_batch.num_rows(), 3); assert_eq!(record_batch.num_columns(), 2); - // Verify content - id column should have values 1, 3, 5 let id_col = record_batch .column(0) .as_any() .downcast_ref::() .unwrap(); - assert_eq!(id_col.values(), &[1, 3, 5]); + assert_eq!(id_col.values(), &[1, 2, 3]); - // Verify content - name column should have values "a", "c", "e" let name_col = record_batch .column(1) .as_any() .downcast_ref::() .unwrap(); assert_eq!(name_col.value(0), "a"); - assert_eq!(name_col.value(1), "c"); - assert_eq!(name_col.value(2), "e"); + assert_eq!(name_col.value(1), "b"); + assert_eq!(name_col.value(2), "c"); assert!(result.next().is_none()); } #[test] fn test_sync_write_parquet_file_multiple_batches() { - let handler = SyncParquetHandler; + let handler = SyncParquetHandler::new(None); let temp_dir = tempdir().unwrap(); let file_path = temp_dir.path().join("test_multi_batch.parquet"); let url = Url::from_file_path(&file_path).unwrap(); - // Create multiple batches let batch1: Box = Box::new(ArrowEngineData::new( RecordBatch::try_from_iter(vec![( "value", @@ -343,19 +254,13 @@ mod tests { .unwrap(), )); - // Create iterator with multiple batches let batches = vec![Ok(batch1), Ok(batch2), Ok(batch3)]; - let data_iter: Box< - dyn Iterator>> + Send, - > = Box::new(batches.into_iter()); + let data_iter: DeltaResultIteratorStatic> = + Box::new(batches.into_iter()); - // Write the file handler.write_parquet_file(url.clone(), data_iter).unwrap(); - - // Verify the file exists assert!(file_path.exists()); - // Read it back to verify all batches were written let file = File::open(&file_path).unwrap(); let reader = crate::parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder::try_new(file) @@ -380,7 +285,6 @@ mod tests { let batch = ArrowEngineData::try_from_engine_data(engine_data).unwrap(); let record_batch = batch.record_batch(); - // Verify we have all 9 rows from 3 batches assert_eq!(record_batch.num_rows(), 9); let value_col = record_batch .column(0) @@ -391,4 +295,75 @@ mod tests { assert!(result.next().is_none()); } + + #[test] + fn write_parquet_creates_parent_directories() { + let handler = SyncParquetHandler::new(None); + let temp_dir = tempdir().unwrap(); + let file_path = temp_dir.path().join("a/b/c/test.parquet"); + let url = Url::from_file_path(&file_path).unwrap(); + + handler.write_parquet_file(url, test_data_iter()).unwrap(); + assert!(file_path.exists()); + } + + /// Ensures `write_parquet_file` and `read_parquet_footer` work end-to-end with an + /// `ObjectStore` backend. The local path is exercised by the other tests in this module. + #[test] + fn parquet_store_write_and_footer_roundtrip() { + let store = Arc::new(crate::object_store::memory::InMemory::new()); + let handler = SyncParquetHandler::new(Some(store)); + let url = Url::parse("memory:///t/data.parquet").unwrap(); + + handler + .write_parquet_file(url.clone(), test_data_iter()) + .unwrap(); + + let footer = handler + .read_parquet_footer(&FileMeta { + location: url, + last_modified: 0, + size: 0, + }) + .unwrap(); + let field_names: Vec<_> = footer + .schema + .fields() + .map(|f| f.name().to_string()) + .collect(); + assert_eq!(field_names, vec!["id".to_string(), "name".to_string()]); + } + + // TODO(#2618): Restore once the engine contract helpers move to test_utils and SyncEngine can + // call them without the kernel-cfg-test cycle issue. + // + // #[test] + // fn parquet_handler_reads_footer() { + // test_parquet_handler_reads_footer(&SyncParquetHandler::new(None)); + // } + // + // #[test] + // fn parquet_handler_footer_errors_on_missing_file() { + // test_parquet_handler_footer_errors_on_missing_file(&SyncParquetHandler::new(None)); + // } + // + // #[test] + // fn parquet_handler_footer_preserves_field_ids() { + // test_parquet_handler_footer_preserves_field_ids(&SyncParquetHandler::new(None)); + // } + // + // #[test] + // fn parquet_handler_write_always_overwrites() { + // test_parquet_handler_write_always_overwrites(&SyncParquetHandler::new(None)); + // } + // + // #[test] + // fn parquet_handler_write_omits_arrow_schema() { + // test_parquet_handler_write_omits_arrow_schema(&SyncParquetHandler::new(None)); + // } + // + // #[test] + // fn parquet_handler_reads_file_with_arrow_schema() { + // test_parquet_handler_reads_file_with_arrow_schema(&SyncParquetHandler::new(None)); + // } } diff --git a/kernel/src/engine/sync/plan.rs b/kernel/src/engine/sync/plan.rs new file mode 100644 index 0000000000..59b00df6bd --- /dev/null +++ b/kernel/src/engine/sync/plan.rs @@ -0,0 +1,695 @@ +//! A synchronous, test-only [`PlanExecutor`] backed by [`SyncEngine`] handlers. +//! +//! Wired into [`SyncEngine::plan_executor`]. The query path evaluates a [`Plan`] eagerly, +//! materializing every node's output in full before its consumers run. Streaming is not needed +//! because this executor only serves tests, which never approach memory limits. +//! +//! [`SyncEngine`]: super::SyncEngine +//! [`SyncEngine::plan_executor`]: super::SyncEngine +// +// TODO: The `IoOperation` paths will eventually be used to replace SyncEngine with an +// PlanBasedEngine (backed by this PlanExecutor) + +use std::collections::HashSet; +use std::sync::Arc; + +use bytes::Bytes; +use itertools::Itertools; + +use super::aggs::eval_aggregate; +use super::json::try_create_from_json; +use super::parquet::{parquet_footer, try_create_from_parquet}; +use super::read_files_arrow; +use super::storage::SyncStorageHandler; +use crate::arrow::array::{ + Array, ArrayRef, BooleanArray, Int64Array, ListArray, RecordBatch, StringArray, +}; +use crate::arrow::compute::filter_record_batch; +use crate::arrow::datatypes::Schema as ArrowSchema; +use crate::arrow::row::{OwnedRow, RowConverter, SortField}; +use crate::engine::arrow_conversion::{TryFromArrow as _, TryIntoArrow as _}; +use crate::engine::arrow_data::{ArrowEngineData, EngineDataArrowExt}; +use crate::engine::arrow_expression::evaluate_expression::extract_column_ref; +use crate::engine::arrow_expression::{extract_column, ArrowEvaluationHandler}; +use crate::engine::arrow_utils::coerce_columns_to_schema; +use crate::expressions::{ArrayData, ColumnName, PredicateRef, Scalar}; +use crate::object_store::DynObjectStore; +use crate::plans::ir::nodes::{ + DynamicScan, FileType, Operator, Project, ScanFile, ScanJson, ScanParquet, SemiJoin, Values, +}; +use crate::plans::ir::plan::{Plan, PlanNode}; +use crate::plans::{IoOperation, Operation, PlanExecutor, PlanResult}; +use crate::schema::{ArrayType, DataType, SchemaRef, StructType}; +use crate::{DeltaResult, Error, EvaluationHandler as _, FileMeta, StorageHandler as _}; + +/// A synchronous, test-only [`PlanExecutor`]. +/// +/// Scans read files directly through the sync module's Arrow read core ([`read_files_arrow`], +/// [`parquet_footer`]) that backs the [`JsonHandler`] / [`ParquetHandler`] traits, so those +/// handlers can eventually be retired in favor of declarative plans. [`IoOperation`]s still +/// delegate to [`SyncStorageHandler`]. +/// +/// All I/O is performed synchronously via [`futures::executor::block_on`]; cloud-backed stores are +/// not supported (see [`super`] module docs). +/// +/// [`JsonHandler`]: crate::JsonHandler +/// [`ParquetHandler`]: crate::ParquetHandler +pub(crate) struct SyncPlanExecutor { + storage: SyncStorageHandler, +} + +impl SyncPlanExecutor { + /// Create a `SyncPlanExecutor` over `store`, or over a per-URL [`LocalFileSystem`] when `None`. + /// + /// [`LocalFileSystem`]: crate::object_store::local::LocalFileSystem + pub(crate) fn new(store: Option>) -> Self { + let storage = SyncStorageHandler::new(store); + Self { storage } + } +} + +// Convenience constructor for tests that don't customize the object store. +impl Default for SyncPlanExecutor { + fn default() -> Self { + Self::new(None) + } +} + +impl PlanExecutor for SyncPlanExecutor { + fn execute_op(&self, op: Operation) -> DeltaResult { + match op { + Operation::IoOperation(io_op) => self.execute_io(io_op), + Operation::QueryPlan(query) => self.execute_query(query), + } + } +} + +impl SyncPlanExecutor { + fn execute_io(&self, op: IoOperation) -> DeltaResult { + match op { + IoOperation::FileListing { url } => { + // `StorageHandler::list_from` returns a non-`Send` iterator, so we collect into + // a `Vec` first to convert into a `Send` iterator. + // TODO(#2619): Evaluate whether StorageHandler should just return `Send` iterators + let metas: Vec> = self.storage.list_from(&url)?.collect(); + Ok(PlanResult::FileMeta(Box::new(metas.into_iter()))) + } + IoOperation::ReadBytes { files } => { + // `StorageHandler::read_files` returns a non-`Send` iterator, so we collect into + // a `Vec` first to convert into a `Send` iterator. + // TODO(#2619): Evaluate whether StorageHandler should just return `Send` iterators + let bytes: Vec> = self.storage.read_files(files)?.collect(); + Ok(PlanResult::Bytes(Box::new(bytes.into_iter()))) + } + IoOperation::WriteBytes { + url, + data, + overwrite, + } => { + self.storage.put(&url, data, overwrite)?; + Ok(PlanResult::Unit) + } + IoOperation::HeadFile { url } => { + let meta = self.storage.head(&url)?; + Ok(PlanResult::FileMeta(Box::new(std::iter::once(Ok(meta))))) + } + IoOperation::AtomicCopy { + source, + destination, + } => { + self.storage.copy_atomic(&source, &destination)?; + Ok(PlanResult::Unit) + } + IoOperation::ParquetFooter { file } => { + let footer = parquet_footer(self.storage.store(), &file)?; + Ok(PlanResult::ParquetFooter(footer)) + } + } + } + + /// Evaluates `query` by materializing each node's output in slice (topological) order, then + /// streams the terminal (last) node's batches to the caller. + fn execute_query(&self, query: Plan) -> DeltaResult { + let mut outputs: Vec> = Vec::with_capacity(query.nodes.len()); + for node in query.nodes { + let output = self.eval_node(node, &outputs)?; + outputs.push(output); + } + let terminal = outputs + .pop() + .ok_or_else(|| Error::generic("plan has no nodes"))?; + let batches = terminal + .into_iter() + .map(|batch| Ok(Box::new(ArrowEngineData::new(batch)) as _)); + Ok(PlanResult::Data(Box::new(batches))) + } + + /// Evaluates a single plan node. `node.inputs` are indices into `outputs`, the + /// already-materialized results of every prior node, in the node's declared input order. + fn eval_node( + &self, + node: PlanNode, + results: &[Vec], + ) -> DeltaResult> { + let PlanNode { op, inputs } = node; + match op { + Operator::ScanJson(ScanJson { + files, + file_constant_columns, + schema, + }) => self.eval_scan(FileType::Json, files, file_constant_columns, schema), + Operator::ScanParquet(ScanParquet { + files, + file_constant_columns, + schema, + }) => self.eval_scan(FileType::Parquet, files, file_constant_columns, schema), + Operator::Values(values) => Ok(vec![values_to_record_batch(values)?]), + Operator::UnionAll(_) => Ok(Vec::from_iter( + inputs.iter().flat_map(|&i| results[i].iter().cloned()), + )), + Operator::Project(project) => eval_project(project, &results[inputs[0]]), + Operator::Filter(filter) => eval_filter(filter.predicate, &results[inputs[0]]), + Operator::DynamicScan(dynamic_scan) => { + self.eval_dynamic_scan(dynamic_scan, &results[inputs[0]]) + } + Operator::Aggregate(aggregate) => eval_aggregate(&aggregate, &results[inputs[0]]), + Operator::SemiJoin(join) => { + eval_semi_join(join, &results[inputs[0]], &results[inputs[1]]) + } + } + } + + /// Reads `files` as `file_type`, broadcasting each file's [`ScanFile::file_constants`] into the + /// output columns named by `file_constant_columns` (see [`ScanParquet`]). Columns not sourced + /// from a file constant are read from the file itself. + /// + /// Files are read one at a time so each batch stays associated with the file whose constants + /// must be broadcast onto it. + fn eval_scan( + &self, + file_type: FileType, + files: Vec, + file_constant_columns: Vec, + schema: SchemaRef, + ) -> DeltaResult> { + // The engine reads only the non-constant columns; constants are spliced in afterwards. + let read_fields = schema + .fields() + .filter(|f| !file_constant_columns.contains(f.name())) + .cloned(); + let read_schema = Arc::new(StructType::try_new(read_fields)?); + let output_schema: Arc = Arc::new(schema.as_ref().try_into_arrow()?); + + let store = self.storage.store(); + let mut batches = Vec::new(); + for file in files { + let metas = [file.meta.clone()]; + let read_schema = read_schema.clone(); + // The two constructors have distinct `impl Iterator` types, so box to unify the arms. + let data: Box>> = match file_type { + FileType::Json => Box::new(read_files_arrow( + store, + &metas, + read_schema, + None, + try_create_from_json, + )), + FileType::Parquet => Box::new(read_files_arrow( + store, + &metas, + read_schema, + None, + try_create_from_parquet, + )), + }; + for batch in data { + let batch: RecordBatch = batch?.into(); + let columns = splice_file_constants( + batch, + &schema, + &file_constant_columns, + &file.file_constants, + )?; + // Reconcile writer-chosen map/list field names to `output_schema`'s before + // `try_new` asserts the schema. See `coerce_columns_to_schema`. + let columns = coerce_columns_to_schema(columns, &output_schema)?; + batches.push(RecordBatch::try_new(output_schema.clone(), columns)?); + } + } + Ok(batches) + } + + /// Reads files named by `input` rows. + fn eval_dynamic_scan( + &self, + dynamic_scan: DynamicScan, + input: &[RecordBatch], + ) -> DeltaResult> { + let files = dynamic_scan_files(&dynamic_scan, input)?; + self.eval_scan( + dynamic_scan.file_type, + files, + dynamic_scan.file_constant_columns, + dynamic_scan.schema, + ) + } +} + +fn dynamic_scan_files( + dynamic_scan: &DynamicScan, + input: &[RecordBatch], +) -> DeltaResult> { + let mut files = Vec::new(); + for batch in input { + let path = extract_column(batch, dynamic_scan.path_column.path())?; + let path = path.as_any().downcast_ref::().ok_or_else(|| { + Error::generic(format!( + "Expected STRING Load path, got {:?}", + path.data_type() + )) + })?; + let size = extract_column(batch, dynamic_scan.file_size_column.path())?; + let size = size.as_any().downcast_ref::().ok_or_else(|| { + Error::generic(format!( + "Expected LONG Load file size, got {:?}", + size.data_type() + )) + })?; + let last_modified = extract_column_ref(batch, dynamic_scan.last_modified_column.path())?; + let last_modified = last_modified + .as_any() + .downcast_ref::() + .ok_or_else(|| { + Error::generic(format!( + "Expected LONG Load last modified, got {:?}", + last_modified.data_type() + )) + })?; + let dv_path = dynamic_scan.dv_column.path(); + let dv = extract_column(batch, dv_path)?; + let dv_ancestors: Vec<_> = (1..dv_path.len()) + .map(|len| extract_column(batch, &dv_path[..len])) + .try_collect()?; + + for row in 0..batch.num_rows() { + if path.is_null(row) { + return Err(Error::generic("DynamicScan path must not be null")); + } + if dv.is_valid(row) && dv_ancestors.iter().all(|ancestor| ancestor.is_valid(row)) { + return Err(Error::unsupported( + "SyncPlanExecutor DynamicScan with deletion vectors", + )); + } + let path = path.value(row); + let location = dynamic_scan.base_url.join(path)?; + if size.is_null(row) { + return Err(Error::generic("DynamicScan file size must not be null")); + } + let size = size.value(row); + if size <= 0 { + return Err(Error::generic("DynamicScan file size must be positive")); + } + let size = u64::try_from(size) + .map_err(|_| Error::generic("DynamicScan file size must fit in a u64"))?; + if last_modified.is_null(row) { + return Err(Error::generic( + "DynamicScan last-modified time must not be null", + )); + } + let last_modified = last_modified.value(row); + let file_constants = dynamic_scan + .file_constant_columns + .iter() + .map(|name| scalar_value(extract_column(batch, &[name])?.as_ref(), row)) + .try_collect()?; + files.push(ScanFile { + meta: FileMeta { + location, + last_modified, + size, + }, + file_constants, + }); + } + } + Ok(files) +} + +fn eval_project(project: Project, input: &[RecordBatch]) -> DeltaResult> { + let Some(first_batch) = input.first() else { + return Ok(vec![]); + }; + let input_schema = Arc::new(StructType::try_from_arrow(first_batch.schema().as_ref())?); + let evaluator = ArrowEvaluationHandler.new_expression_evaluator( + input_schema, + project.expr, + project.schema.as_ref().clone().into(), + )?; + input + .iter() + .map(|batch| { + evaluator + .evaluate(&ArrowEngineData::new(batch.clone()))? + .try_into_record_batch() + }) + .collect() +} + +fn eval_filter(predicate: PredicateRef, input: &[RecordBatch]) -> DeltaResult> { + let Some(first_batch) = input.first() else { + return Ok(vec![]); + }; + let input_schema = Arc::new(StructType::try_from_arrow(first_batch.schema().as_ref())?); + let evaluator = ArrowEvaluationHandler.new_predicate_evaluator(input_schema, predicate)?; + input + .iter() + .map(|batch| { + let mask = evaluator + .evaluate(&ArrowEngineData::new(batch.clone()))? + .try_into_record_batch()?; + let mask = mask + .column(0) + .as_any() + .downcast_ref::() + .ok_or_else(|| { + Error::generic("Filter predicate did not produce a boolean array") + })?; + Ok(filter_record_batch(batch, mask)?) + }) + .collect() +} + +fn eval_semi_join( + join: SemiJoin, + probe: &[RecordBatch], + build: &[RecordBatch], +) -> DeltaResult> { + let mut build_keys = HashSet::new(); + for batch in build { + build_keys.extend(encode_keys_as_rows(batch, &join.build_keys)?); + } + + probe + .iter() + .map(|batch| { + let keep = encode_keys_as_rows(batch, &join.probe_keys)? + .into_iter() + .map(|key| join.inverted != build_keys.contains(&key)); + Ok(filter_record_batch(batch, &BooleanArray::from_iter(keep))?) + }) + .collect() +} + +/// Builds output columns in `schema` order: each column named in `file_constant_columns` is +/// broadcast from `constants` (at the matching slot), and every other column is drained in turn +/// from `batch`, which holds exactly the non-constant columns in schema order. +fn splice_file_constants( + batch: RecordBatch, + schema: &SchemaRef, + file_constant_columns: &[String], + constants: &[Scalar], +) -> DeltaResult> { + let (_, read_columns, rows) = batch.into_parts(); + let mut read_columns = read_columns.into_iter(); + schema + .fields() + .map( + |field| match file_constant_columns.iter().position(|c| c == field.name()) { + Some(slot) => constants[slot].to_array(rows), + None => read_columns + .next() + .ok_or_else(|| Error::generic("scan output has fewer columns than schema")), + }, + ) + .collect() +} + +/// Encodes `columns` of `batch` as one comparable/hashable [`OwnedRow`] key per input row. +/// +/// Empty `columns` (e.g. ungrouped aggregation) yields a vec of empty keys that self-compare equal +pub(super) fn encode_keys_as_rows( + batch: &RecordBatch, + columns: &[ColumnName], +) -> DeltaResult> { + if columns.is_empty() { + let key = RowConverter::new(vec![])?.parser().parse(&[]).owned(); + return Ok(vec![key; batch.num_rows()]); + } + let arrays: Vec<_> = columns + .iter() + .map(|name| extract_column(batch, name)) + .try_collect()?; + // Constructing RowConverter requires a `SortField`. We initialize default, unsorted field for + // each column. + let sort_fields = arrays + .iter() + .map(|array| SortField::new(array.data_type().clone())) + .collect(); + let converter = RowConverter::new(sort_fields)?; + let rows = converter.convert_columns(&arrays)?; + Ok(rows.iter().map(|row| row.owned()).collect()) +} + +fn scalar_value(array: &dyn Array, row: usize) -> DeltaResult { + if array.is_null(row) { + return Ok(Scalar::Null(DataType::try_from_arrow(array.data_type())?)); + } + if let Some(strings) = array.as_any().downcast_ref::() { + return Ok(Scalar::String(strings.value(row).to_string())); + } + if let Some(longs) = array.as_any().downcast_ref::() { + return Ok(Scalar::Long(longs.value(row))); + } + Err(Error::unsupported(format!( + "Scalar conversion from array type {:?}", + array.data_type() + ))) +} + +/// Materialize a [`Values`] node's literal rows into a [`RecordBatch`]. An empty relation (the +/// [`PlanBuilder::build`] output for an absent input) yields zero-row data. +/// +/// [`PlanBuilder::build`]: crate::plans::PlanBuilder::build +fn values_to_record_batch(values: Values) -> DeltaResult { + let Values { schema, rows } = values; + let columns: Vec = schema + .fields() + .enumerate() + .map(|(col, field)| -> DeltaResult { + let element_type = ArrayType::new(field.data_type().clone(), true); + let column = ArrayData::try_new(element_type, rows.iter().map(|row| row[col].clone()))?; + // This produces a single array row. The array contains n elements, one for each + // attribute of the column. + let list = Scalar::Array(column).to_array(1)?; + let list = list.as_any().downcast_ref::().ok_or_else(|| { + Error::generic("Values: Scalar::Array did not lower to a ListArray") + })?; + let (_field, _offsets, values, _nulls) = list.clone().into_parts(); + Ok(values) + }) + .try_collect()?; + let schema = Arc::new(schema.as_ref().try_into_arrow()?); + Ok(RecordBatch::try_new(schema, columns)?) +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + use url::Url; + + use super::*; + use crate::actions::deletion_vector::DeletionVectorDescriptor; + use crate::arrow::array::StructArray; + use crate::arrow::buffer::{BooleanBuffer, NullBuffer}; + use crate::expressions::{column_name, StructData}; + use crate::schema::{StructField, ToSchema as _}; + + #[test] + fn encode_keys_as_rows_synthesizes_empty_keys_when_ungrouped() -> DeltaResult<()> { + let batch = RecordBatch::try_from_iter([( + "x", + Arc::new(Int64Array::from(vec![1, 2, 3])) as ArrayRef, + )])?; + let keys = encode_keys_as_rows(&batch, &[])?; + assert_eq!(keys.len(), 3); + assert!(keys.iter().all(|key| key == &keys[0])); + Ok(()) + } + + fn null_dv() -> Scalar { + Scalar::Null(DataType::from(DeletionVectorDescriptor::to_schema())) + } + + fn present_dv() -> Scalar { + Scalar::Struct( + StructData::try_new( + DeletionVectorDescriptor::to_schema() + .fields() + .cloned() + .collect(), + vec![ + Scalar::from("i"), + Scalar::from("inline"), + Scalar::Null(DataType::INTEGER), + Scalar::from(1_i32), + Scalar::from(1_i64), + ], + ) + .unwrap(), + ) + } + + fn dynamic_scan_files_result( + path: Scalar, + size: Scalar, + last_modified: Scalar, + dv: Scalar, + ) -> DeltaResult> { + let input_schema = Arc::new(StructType::new_unchecked([ + StructField::nullable("path", DataType::STRING), + StructField::nullable("size", DataType::LONG), + StructField::nullable("filemod", DataType::LONG), + StructField::nullable("dv", DeletionVectorDescriptor::to_schema()), + ])); + let input = values_to_record_batch(Values::new( + input_schema, + vec![vec![path, size, last_modified, dv]], + )) + .unwrap(); + let dynamic_scan = DynamicScan { + schema: Arc::new(StructType::new_unchecked(Vec::::new())), + file_type: FileType::Parquet, + base_url: Url::parse("memory:///").unwrap(), + file_constant_columns: vec![], + path_column: column_name!("path"), + file_size_column: column_name!("size"), + last_modified_column: column_name!("filemod"), + dv_column: column_name!("dv"), + }; + + dynamic_scan_files(&dynamic_scan, &[input]) + } + + #[test] + fn dynamic_scan_executor_rejects_non_null_deletion_vector() { + let err = dynamic_scan_files_result( + "file.parquet".into(), + 1_i64.into(), + 0_i64.into(), + present_dv(), + ) + .unwrap_err(); + assert!(err.to_string().contains("with deletion vectors"), "{err}"); + } + + #[rstest] + #[case::path( + Scalar::Null(DataType::STRING), + 1_i64.into(), + 0_i64.into(), + "path must not be null" + )] + #[case::size( + "file.parquet".into(), + Scalar::Null(DataType::LONG), + 0_i64.into(), + "file size must not be null" + )] + #[case::last_modified( + "file.parquet".into(), + 1_i64.into(), + Scalar::Null(DataType::LONG), + "last-modified time must not be null" + )] + fn dynamic_scan_executor_rejects_null_required_field( + #[case] path: Scalar, + #[case] size: Scalar, + #[case] last_modified: Scalar, + #[case] needle: &str, + ) { + let err = dynamic_scan_files_result(path, size, last_modified, null_dv()).unwrap_err(); + assert!(err.to_string().contains(needle), "{err}"); + } + + #[rstest] + #[case::zero(0_i64.into(), "file size must be positive")] + #[case::negative((-1_i64).into(), "file size must be positive")] + fn dynamic_scan_executor_rejects_invalid_size(#[case] size: Scalar, #[case] needle: &str) { + let err = dynamic_scan_files_result("file.parquet".into(), size, 0_i64.into(), null_dv()) + .unwrap_err(); + assert!(err.to_string().contains(needle), "{err}"); + } + + #[test] + fn dynamic_scan_executor_accepts_i64_max_file_size() { + let err = dynamic_scan_files_result( + "file.parquet".into(), + i64::MAX.into(), + Scalar::Null(DataType::LONG), + null_dv(), + ) + .unwrap_err(); + assert!( + err.to_string() + .contains("last-modified time must not be null"), + "got a size validation error before last-modified validation: {err}" + ); + } + + #[test] + fn dynamic_scan_threads_last_modified_into_file_meta() { + let files = dynamic_scan_files_result( + "file.parquet".into(), + 1_i64.into(), + 1_700_000_000_123_i64.into(), + null_dv(), + ) + .unwrap(); + + assert_eq!(files[0].meta.last_modified, 1_700_000_000_123); + } + + #[test] + fn dynamic_scan_treats_dv_under_null_ancestor_as_null() { + let dv_field = StructField::nullable("dv", DeletionVectorDescriptor::to_schema()); + let metadata_type = StructType::new_unchecked([dv_field]); + let input_schema = Arc::new(StructType::new_unchecked([ + StructField::not_null("path", DataType::STRING), + StructField::not_null("size", DataType::LONG), + StructField::not_null("filemod", DataType::LONG), + StructField::nullable("metadata", metadata_type.clone()), + ])); + let metadata_schema: ArrowSchema = (&metadata_type).try_into_arrow().unwrap(); + let metadata = StructArray::new( + metadata_schema.fields().clone(), + vec![present_dv().to_array(1).unwrap()], + Some(NullBuffer::new(BooleanBuffer::from(vec![false]))), + ); + let arrow_schema = Arc::new(input_schema.as_ref().try_into_arrow().unwrap()); + let input = RecordBatch::try_new( + arrow_schema, + vec![ + Arc::new(StringArray::from(vec!["file.parquet"])), + Arc::new(Int64Array::from(vec![1_i64])), + Arc::new(Int64Array::from(vec![0_i64])), + Arc::new(metadata), + ], + ) + .unwrap(); + let dynamic_scan = DynamicScan { + schema: Arc::new(StructType::new_unchecked(Vec::::new())), + file_type: FileType::Parquet, + base_url: Url::parse("memory:///").unwrap(), + file_constant_columns: vec![], + path_column: column_name!("path"), + file_size_column: column_name!("size"), + last_modified_column: column_name!("filemod"), + dv_column: column_name!("metadata.dv"), + }; + + assert_eq!( + dynamic_scan_files(&dynamic_scan, &[input]).unwrap().len(), + 1 + ); + } +} diff --git a/kernel/src/engine/sync/storage.rs b/kernel/src/engine/sync/storage.rs index bf876a629e..a8c8ee301e 100644 --- a/kernel/src/engine/sync/storage.rs +++ b/kernel/src/engine/sync/storage.rs @@ -1,96 +1,115 @@ +use std::sync::Arc; + use bytes::Bytes; -use itertools::Itertools; +use futures::StreamExt as _; use url::Url; +use super::{put_bytes, resolve_scope}; +use crate::object_store::path::Path; +use crate::object_store::{DynObjectStore, ObjectStoreExt as _}; use crate::{DeltaResult, Error, FileMeta, FileSlice, StorageHandler}; -pub(crate) struct SyncStorageHandler; +pub(crate) struct SyncStorageHandler { + store: Option>, +} + +impl SyncStorageHandler { + pub(crate) fn new(store: Option>) -> Self { + Self { store } + } + + /// The backing store, or `None` for the per-URL [`LocalFileSystem`] fallback. + /// + /// [`LocalFileSystem`]: crate::object_store::local::LocalFileSystem + #[cfg(feature = "declarative-plans")] + pub(super) fn store(&self) -> Option<&Arc> { + self.store.as_ref() + } +} impl StorageHandler for SyncStorageHandler { - /// List the paths in the same directory that are lexicographically greater or equal to - /// (UTF-8 sorting) the given `path`. The result is sorted by the file name. fn list_from( &self, url_path: &Url, ) -> DeltaResult>>> { - if url_path.scheme() == "file" { - let path = url_path - .to_file_path() - .map_err(|_| Error::Generic(format!("Invalid path for list_from: {url_path:?}")))?; - - let (path_to_read, min_file_name) = if path.is_dir() { - // passed path is an existing dir, don't strip anything and don't filter the results - (path, None) - } else { - // path doesn't exist, or is not a dir, assume the final part is a filename. strip - // that and use it as the min_file_name to return - let parent = path - .parent() - .ok_or_else(|| Error::Generic(format!("Invalid path for list_from: {path:?}")))? - .to_path_buf(); - let file_name = path.file_name().ok_or_else(|| { - Error::Generic(format!("Invalid path for list_from: {path:?}")) - })?; - (parent, Some(file_name)) - }; - - let all_ents: Vec<_> = std::fs::read_dir(path_to_read)? - .filter(|ent_res| { - match (ent_res, min_file_name) { - (Ok(ent), Some(min_file_name)) => ent.file_name() > *min_file_name, - _ => true, // Keep unfiltered and/or error entries - } - }) - .try_collect()?; - let it = all_ents - .into_iter() - .sorted_by_key(|ent| ent.path()) - .map(TryFrom::try_from); - Ok(Box::new(it)) + let (store, base_url, offset) = resolve_scope(self.store.as_ref(), url_path)?; + + // For directory URLs, prefix == offset and the offset acts as a lower bound that still + // includes everything underneath (since `dir/a` > `dir` lexicographically). For file + // URLs, the prefix is the parent directory and the offset filters out files at-or-below. + let prefix = if url_path.path().ends_with('/') { + offset.clone() } else { - Err(Error::generic("Can only read local filesystem")) - } + let mut parts: Vec<_> = offset.parts().collect(); + parts.pop(); + Path::from_iter(parts) + }; + + // LocalFileSystem and InMemory do not return sorted listings, so we collect and sort + // to give callers a deterministic order. + let mut metas: Vec<_> = futures::executor::block_on( + store + .list_with_offset(Some(&prefix), &offset) + .collect::>(), + ) + .into_iter() + .collect::>()?; + metas.sort_unstable_by(|a, b| a.location.cmp(&b.location)); + + let iter = metas.into_iter().map(move |meta| { + let location = base_url + .join(meta.location.as_ref()) + .map_err(|e| Error::generic(format!("Failed to construct URL: {e}")))?; + Ok(FileMeta { + location, + last_modified: meta.last_modified.timestamp_millis(), + size: meta.size, + }) + }); + Ok(Box::new(iter)) } - /// Read data specified by the start and end offset from the file. fn read_files( &self, files: Vec, ) -> DeltaResult>>> { - let iter = files.into_iter().map(|(url, _range_opt)| { - if url.scheme() == "file" { - if let Ok(file_path) = url.to_file_path() { - let bytes_vec_res = std::fs::read(file_path); - let bytes: std::io::Result = - bytes_vec_res.map(|bytes_vec| bytes_vec.into()); - return bytes.map_err(|_| Error::file_not_found(url.path())); - } - } - Err(Error::generic("Can only read local filesystem")) - }); - Ok(Box::new(iter)) + let store = self.store.clone(); + let results: Vec> = files + .into_iter() + .map(|(url, _range_opt)| { + let (s, _, path) = resolve_scope(store.as_ref(), &url)?; + let get_result = futures::executor::block_on(s.get(&path))?; + Ok(futures::executor::block_on(get_result.bytes())?) + }) + .collect(); + Ok(Box::new(results.into_iter())) } fn put(&self, path: &Url, data: Bytes, overwrite: bool) -> DeltaResult<()> { - if path.scheme() != "file" { - return Err(Error::generic("Can only write to local filesystem")); - } - let file_path = path - .to_file_path() - .map_err(|_| Error::generic(format!("Invalid path for put: {path:?}")))?; - if !overwrite && file_path.exists() { - return Err(Error::FileAlreadyExists(file_path.to_string_lossy().into())); - } - std::fs::write(&file_path, &data) - .map_err(|e| Error::generic(format!("Failed to write {}: {e}", file_path.display()))) + put_bytes(self.store.as_ref(), path, data, overwrite) } fn copy_atomic(&self, _src: &Url, _dest: &Url) -> DeltaResult<()> { unimplemented!("SyncStorageHandler does not implement copy"); } - fn head(&self, _path: &Url) -> DeltaResult { - unimplemented!("head is not implemented for SyncStorageHandler") + fn head(&self, url: &Url) -> DeltaResult { + let (store, _, path) = resolve_scope(self.store.as_ref(), url)?; + let meta = futures::executor::block_on(store.head(&path))?; + Ok(FileMeta { + location: url.clone(), + last_modified: meta.last_modified.timestamp_millis(), + size: meta.size, + }) + } + + fn delete(&self, url: &Url) -> DeltaResult<()> { + let (store, _, path) = resolve_scope(self.store.as_ref(), url)?; + match futures::executor::block_on(store.delete(&path)) { + Ok(()) => Ok(()), + Err(crate::object_store::Error::NotFound { .. }) => Ok(()), + Err(e) => Err(e.into()), + } } } @@ -105,8 +124,10 @@ mod tests { use url::Url; use super::SyncStorageHandler; + use crate::object_store::memory::InMemory; + use crate::object_store::ObjectStoreExt as _; use crate::utils::current_time_duration; - use crate::StorageHandler; + use crate::{Error, StorageHandler}; /// generate json filenames that follow the spec (numbered padded to 20 chars) fn get_json_filename(index: usize) -> String { @@ -115,7 +136,7 @@ mod tests { #[test] fn test_file_meta_is_correct() -> Result<(), Box> { - let storage = SyncStorageHandler; + let storage = SyncStorageHandler::new(None); let tmp_dir = tempfile::tempdir().unwrap(); let begin_time = current_time_duration()?; @@ -139,7 +160,7 @@ mod tests { #[test] fn test_list_from() -> Result<(), Box> { - let storage = SyncStorageHandler; + let storage = SyncStorageHandler::new(None); let tmp_dir = tempfile::tempdir().unwrap(); let mut expected = vec![]; for i in 0..3 { @@ -162,8 +183,9 @@ mod tests { } assert_eq!(file_count, 1); - let url_path = tmp_dir.path().join(""); - let url = Url::from_file_path(url_path).unwrap(); + // Listing a directory URL: the URL must end with `/` so list_from recognizes it as a + // directory rather than as a file whose parent should be listed. + let url = Url::from_directory_path(tmp_dir.path()).unwrap(); let list = storage.list_from(&url)?; file_count = list.count(); assert_eq!(file_count, 3); @@ -178,7 +200,7 @@ mod tests { #[test] fn test_read_files() -> Result<(), Box> { - let storage = SyncStorageHandler; + let storage = SyncStorageHandler::new(None); let tmp_dir = tempfile::tempdir().unwrap(); let path = tmp_dir.path().join(get_json_filename(1)); let mut f = File::create(path.clone())?; @@ -198,4 +220,141 @@ mod tests { assert_eq!(file_count, 1); Ok(()) } + + /// `list_from` against an [`ObjectStore`] must walk subdirectories, matching the local FS + /// implementation that uses `read_dir` recursively. + #[tokio::test] + async fn list_from_store_is_recursive() { + let store = std::sync::Arc::new(InMemory::new()); + store + .put( + &crate::object_store::path::Path::from("a/file1.json"), + bytes::Bytes::from("x").into(), + ) + .await + .unwrap(); + store + .put( + &crate::object_store::path::Path::from("a/sub/file2.json"), + bytes::Bytes::from("y").into(), + ) + .await + .unwrap(); + + let storage = SyncStorageHandler::new(Some(store)); + let url = Url::parse("memory:///a/").unwrap(); + let listed: Vec<_> = storage + .list_from(&url) + .unwrap() + .map(Result::unwrap) + .collect(); + + let names: Vec<_> = listed + .iter() + .map(|f| f.location.path().to_string()) + .collect(); + assert_eq!(names, vec!["/a/file1.json", "/a/sub/file2.json"]); + } + + #[test] + fn head_local_returns_metadata() { + let storage = SyncStorageHandler::new(None); + let tmp_dir = tempfile::tempdir().unwrap(); + let path = tmp_dir.path().join("file.json"); + std::fs::write(&path, b"hello").unwrap(); + let url = Url::from_file_path(&path).unwrap(); + + let meta = storage.head(&url).unwrap(); + assert_eq!(meta.location, url); + assert_eq!(meta.size, 5); + } + + #[test] + fn head_local_missing_file_errors() { + let storage = SyncStorageHandler::new(None); + let tmp_dir = tempfile::tempdir().unwrap(); + let url = Url::from_file_path(tmp_dir.path().join("missing.json")).unwrap(); + assert!(matches!( + storage.head(&url).unwrap_err(), + Error::FileNotFound(_) + )); + } + + #[tokio::test] + async fn head_store_returns_metadata() { + let store = std::sync::Arc::new(InMemory::new()); + let path = crate::object_store::path::Path::from("file.json"); + store + .put(&path, bytes::Bytes::from("hello").into()) + .await + .unwrap(); + + let storage = SyncStorageHandler::new(Some(store)); + let url = Url::parse("memory:///file.json").unwrap(); + let meta = storage.head(&url).unwrap(); + assert_eq!(meta.location, url); + assert_eq!(meta.size, 5); + } + + #[tokio::test] + async fn put_store_creates_and_blocks_overwrite() { + let store = std::sync::Arc::new(InMemory::new()); + let storage = SyncStorageHandler::new(Some(store.clone())); + let url = Url::parse("memory:///file.json").unwrap(); + + storage + .put(&url, bytes::Bytes::from("first"), false) + .unwrap(); + let path = crate::object_store::path::Path::from("file.json"); + let bytes = futures::executor::block_on(async { + store.get(&path).await.unwrap().bytes().await.unwrap() + }); + assert_eq!(bytes.as_ref(), b"first"); + + // Without overwrite, a second put must fail with FileAlreadyExists. + let err = storage + .put(&url, bytes::Bytes::from("second"), false) + .unwrap_err(); + assert!(matches!(err, Error::FileAlreadyExists(_))); + + // With overwrite, it should succeed. + storage + .put(&url, bytes::Bytes::from("third"), true) + .unwrap(); + let bytes = futures::executor::block_on(async { + store.get(&path).await.unwrap().bytes().await.unwrap() + }); + assert_eq!(bytes.as_ref(), b"third"); + } + + // TODO(#2872): add a double-delete test to exercise idempotency on the in-memory store. + #[tokio::test] + async fn delete_store_removes_file() { + let store = std::sync::Arc::new(InMemory::new()); + let storage = SyncStorageHandler::new(Some(store)); + let url = Url::parse("memory:///file.json").unwrap(); + + storage + .put(&url, bytes::Bytes::from("hello"), false) + .unwrap(); + storage.delete(&url).unwrap(); + + assert!(matches!( + storage.head(&url).unwrap_err(), + Error::FileNotFound(_) + )); + } + + #[tokio::test] + async fn delete_missing_is_ok() { + let store = std::sync::Arc::new(InMemory::new()); + let storage = SyncStorageHandler::new(Some(store)); + let url = Url::parse("memory:///missing.json").unwrap(); + + assert!(matches!( + storage.head(&url).unwrap_err(), + Error::FileNotFound(_) + )); + storage.delete(&url).unwrap(); + } } diff --git a/kernel/src/engine/test_delegating.rs b/kernel/src/engine/test_delegating.rs new file mode 100644 index 0000000000..3d6f6cc57f --- /dev/null +++ b/kernel/src/engine/test_delegating.rs @@ -0,0 +1,102 @@ +//! Test [`Engine`] that delegates handlers to an inner engine. + +use std::sync::Arc; + +#[cfg(feature = "declarative-plans")] +use crate::plans::PlanExecutor; +use crate::{Engine, EvaluationHandler, JsonHandler, ParquetHandler, StorageHandler}; + +/// A test [`Engine`] that forwards every handler to `inner`, except those overridden via the +/// `with_*` setters. Lets a test swap a single handler without re-typing the forwards for the +/// rest. +pub(crate) struct DelegatingEngine { + inner: Arc, + storage: Option>, + json: Option>, + parquet: Option>, + #[cfg(feature = "declarative-plans")] + plan_executor: PlanExecutorOverride, +} + +/// How [`DelegatingEngine`] resolves `plan_executor`. Unlike the other handlers, `None` is +/// itself a valid override (forcing an engine to report no executor), so it needs a distinct +/// variant from inheriting the inner engine's executor. +#[cfg(feature = "declarative-plans")] +enum PlanExecutorOverride { + Inherit, + None, + Some(Arc), +} + +impl DelegatingEngine { + pub(crate) fn new(inner: Arc) -> Self { + Self { + inner, + storage: None, + json: None, + parquet: None, + #[cfg(feature = "declarative-plans")] + plan_executor: PlanExecutorOverride::Inherit, + } + } + + pub(crate) fn with_storage_handler(mut self, handler: Arc) -> Self { + self.storage = Some(handler); + self + } + + pub(crate) fn with_json_handler(mut self, handler: Arc) -> Self { + self.json = Some(handler); + self + } + + pub(crate) fn with_parquet_handler(mut self, handler: Arc) -> Self { + self.parquet = Some(handler); + self + } + + #[cfg(feature = "declarative-plans")] + pub(crate) fn with_plan_executor(mut self, executor: Arc) -> Self { + self.plan_executor = PlanExecutorOverride::Some(executor); + self + } + + #[cfg(feature = "declarative-plans")] + pub(crate) fn without_plan_executor(mut self) -> Self { + self.plan_executor = PlanExecutorOverride::None; + self + } +} + +impl Engine for DelegatingEngine { + fn evaluation_handler(&self) -> Arc { + self.inner.evaluation_handler() + } + + fn storage_handler(&self) -> Arc { + self.storage + .clone() + .unwrap_or_else(|| self.inner.storage_handler()) + } + + fn json_handler(&self) -> Arc { + self.json + .clone() + .unwrap_or_else(|| self.inner.json_handler()) + } + + fn parquet_handler(&self) -> Arc { + self.parquet + .clone() + .unwrap_or_else(|| self.inner.parquet_handler()) + } + + #[cfg(feature = "declarative-plans")] + fn plan_executor(&self) -> Option> { + match &self.plan_executor { + PlanExecutorOverride::Inherit => self.inner.plan_executor(), + PlanExecutorOverride::None => None, + PlanExecutorOverride::Some(executor) => Some(executor.clone()), + } + } +} diff --git a/kernel/src/engine_data.rs b/kernel/src/engine_data.rs index 8ce4220951..ba34cd9a7c 100644 --- a/kernel/src/engine_data.rs +++ b/kernel/src/engine_data.rs @@ -5,23 +5,27 @@ use std::ops::Range; use tracing::debug; +use crate::actions::visitors::SelectionVectorVisitor; use crate::expressions::ArrayData; use crate::log_replay::HasSelectionVector; use crate::schema::{ColumnName, DataType, SchemaRef}; +use crate::utils::require; use crate::{AsAny, DeltaResult, Error}; /// Engine data paired with a selection vector indicating which rows are logically selected. /// -/// A value of `true` in the selection vector means the corresponding row is selected (i.e., not deleted), -/// while `false` means the row is logically deleted and should be ignored. If the selection vector is shorter -/// than the number of rows in `data` then all rows not covered by the selection vector are assumed to be selected. +/// A value of `true` in the selection vector means the corresponding row is selected (i.e., not +/// deleted), while `false` means the row is logically deleted and should be ignored. If the +/// selection vector is shorter than the number of rows in `data` then all rows not covered by the +/// selection vector are assumed to be selected. /// /// Interpreting unselected (`false`) rows will result in incorrect/undefined behavior. pub struct FilteredEngineData { // The underlying engine data data: Box, // The selection vector where `true` marks rows to include in results. N.B. this selection - // vector may be less then `data.len()` and any gaps represent rows that are assumed to be selected. + // vector may be less then `data.len()` and any gaps represent rows that are assumed to be + // selected. selection_vector: Vec, } @@ -66,11 +70,10 @@ impl FilteredEngineData { } } - /// Apply the contained selection vector and return an engine data with only the valid rows - /// included. This consumes the `FilteredEngineData` + /// Apply the contained selection vector and return an engine data with only the selected rows + /// included. This consumes the `FilteredEngineData`. pub fn apply_selection_vector(self) -> DeltaResult> { - self.data - .apply_selection_vector(self.selection_vector.clone()) + self.data.apply_selection_vector(self.selection_vector) } } @@ -192,6 +195,13 @@ impl<'a> MapItem<'a> { self.values.is_valid(idx).then(|| self.values.value(idx)) } + /// Returns the map's keys in storage order, including keys whose value is null. Unlike + /// [`MapItem::materialize`], which drops null-valued entries, this exposes the full key set. + pub(crate) fn keys(&self) -> impl Iterator + 'a { + let keys = self.keys; + self.offsets.clone().map(move |idx| keys.value(idx)) + } + pub fn materialize(&self) -> HashMap { let mut ret = HashMap::with_capacity(self.offsets.len()); for idx in self.offsets.clone() { @@ -225,6 +235,8 @@ macro_rules! impl_default_get { pub trait GetData<'a> { impl_default_get!( (get_bool, bool), + (get_byte, i8), + (get_short, i16), (get_int, i32), (get_long, i64), (get_float, f32), @@ -252,6 +264,8 @@ macro_rules! impl_null_get { impl<'a> GetData<'a> for () { impl_null_get!( (get_bool, bool), + (get_byte, i8), + (get_short, i16), (get_int, i32), (get_long, i64), (get_float, f32), @@ -295,6 +309,8 @@ macro_rules! impl_typed_get_data { // Use get_date/get_timestamp directly instead of through TypedGetData. impl_typed_get_data!( (get_bool, bool), + (get_byte, i8), + (get_short, i16), (get_int, i32), (get_long, i64), (get_float, f32), @@ -335,6 +351,105 @@ impl<'a> TypedGetData<'a, HashMap> for dyn GetData<'a> + '_ { } } +/// An iterator over the indices of selected rows in an engine-data batch. +/// +/// Each call to [`Iterator::next`] returns the index of the next selected row. +/// +/// Constructed internally and passed (alongside the column getters) to +/// [`FilteredRowVisitor::visit_filtered`]. +pub struct RowIndexIterator<'sv> { + sv_pos: usize, + selection_vector: &'sv [bool], + row_count: usize, +} + +impl<'sv> RowIndexIterator<'sv> { + pub(crate) fn new(row_count: usize, selection_vector: &'sv [bool]) -> Self { + Self { + sv_pos: 0, + selection_vector, + row_count, + } + } + + /// Returns the total number of rows in the batch (selected and deselected). + pub fn num_rows(&self) -> usize { + self.row_count + } +} + +impl<'sv> Iterator for RowIndexIterator<'sv> { + type Item = usize; + + fn next(&mut self) -> Option { + while self.sv_pos < self.row_count { + let pos = self.sv_pos; + self.sv_pos += 1; + if pos >= self.selection_vector.len() || self.selection_vector[pos] { + return Some(pos); + } + } + None + } +} + +/// A visitor that processes [`FilteredEngineData`] with automatic row filtering. +/// +/// Implementors provide [`visit_filtered`] which receives the column getters and a +/// [`RowIndexIterator`] that yields the index of each selected row. +/// The default [`visit_rows_of`] method handles all the plumbing: extracting the selection +/// vector, building the bridge, and calling [`EngineData::visit_rows`]. +/// +/// [`visit_filtered`]: FilteredRowVisitor::visit_filtered +/// [`visit_rows_of`]: FilteredRowVisitor::visit_rows_of +pub trait FilteredRowVisitor { + fn selected_column_names_and_types(&self) -> (&'static [ColumnName], &'static [DataType]); + + /// Process this batch. `getters` contains one [`GetData`] item per requested column. + /// Iterate `rows` to receive the index of each selected row. Use + /// [`RowIndexIterator::num_rows`] to get the total row count (for padding output + /// vectors with null values for deselected rows). + fn visit_filtered<'a>( + &mut self, + getters: &[&'a dyn GetData<'a>], + rows: RowIndexIterator<'_>, + ) -> DeltaResult<()>; + + /// Visit the rows of a [`FilteredEngineData`], automatically respecting the selection vector. + /// + /// Extracts the selection vector and passes a [`RowIndexIterator`] of selected row indices + /// to [`FilteredRowVisitor::visit_filtered`]. + fn visit_rows_of(&mut self, data: &FilteredEngineData) -> DeltaResult<()> + where + Self: Sized, + { + // column_names is 'static so this borrow ends immediately, before bridge borrows self + let column_names = self.selected_column_names_and_types().0; + let mut bridge = FilteredVisitorBridge { + visitor: self, + selection_vector: data.selection_vector(), + }; + data.data().visit_rows(column_names, &mut bridge) + } +} + +/// Private bridge that implements [`RowVisitor`] and forwards to a [`FilteredRowVisitor`]. +struct FilteredVisitorBridge<'bridge, V: FilteredRowVisitor> { + visitor: &'bridge mut V, + selection_vector: &'bridge [bool], +} + +impl RowVisitor for FilteredVisitorBridge<'_, V> { + fn selected_column_names_and_types(&self) -> (&'static [ColumnName], &'static [DataType]) { + self.visitor.selected_column_names_and_types() + } + + fn visit<'a>(&mut self, row_count: usize, getters: &[&'a dyn GetData<'a>]) -> DeltaResult<()> { + let rows = RowIndexIterator::new(row_count, self.selection_vector); + self.visitor.visit_filtered(getters, rows) + } +} + /// A `RowVisitor` can be called back to visit extracted data. Aside from calling /// [`RowVisitor::visit`] on the visitor passed to [`EngineData::visit_rows`], engines do /// not need to worry about this trait. @@ -394,7 +509,10 @@ pub trait RowVisitor { /// todo!() // convert `SchemaRef` and `ArrayData` into local representation and append them /// } /// fn apply_selection_vector(self: Box, selection_vector: Vec) -> DeltaResult> { -/// todo!() // filter out unselected rows and return the new set of data +/// todo!() // filter out unselected rows; rows beyond the selection vector's end are selected +/// } +/// fn has_field(&self, name: &ColumnName) -> bool { +/// todo!() // determine whether the field exists in the data /// } /// } /// ``` @@ -422,9 +540,10 @@ pub trait EngineData: AsAny { /// with the provided new columns. The original data remains unchanged. /// /// # Parameters - /// - `schema`: The schema of the columns being appended (not the entire resulting schema). - /// This schema must describe exactly the columns being added in the `columns` parameter. - /// - `columns`: The column data to append. Each [`ArrayData`] corresponds to one field in the schema. + /// - `schema`: The schema of the columns being appended (not the entire resulting schema). This + /// schema must describe exactly the columns being added in the `columns` parameter. + /// - `columns`: The column data to append. Each [`ArrayData`] corresponds to one field in the + /// schema. /// /// # Returns /// A new `EngineData` instance containing both the original columns and the appended columns. @@ -442,24 +561,58 @@ pub trait EngineData: AsAny { columns: Vec, ) -> DeltaResult>; - /// Apply a selection vector to the data and return a data where only the valid rows are + /// Apply a selection vector to the data and return a data where only the selected rows are /// included. This consumes the EngineData, allowing engines to implement this "in place" if - /// desired + /// desired. + /// + /// The selection vector may be shorter than the data; rows beyond its end are selected and + /// must be retained (see [`FilteredEngineData`]). An empty selection vector selects all rows. + /// A selection vector longer than the data is invalid and must be rejected, e.g. with + /// [`Error::InvalidSelectionVector`]. fn apply_selection_vector( self: Box, selection_vector: Vec, ) -> DeltaResult>; + + /// Returns `true` if a field at the given (possibly nested) path exists in this data's schema. + /// + /// For a top-level field named `"foo"`, use `column_name!("foo")`. For nested fields, + /// each non-leaf element of the path must be a struct field at that level. + fn has_field(&self, name: &ColumnName) -> bool; +} + +/// Evaluates a predicate on the batch, extracts the resulting selection vector, and applies +/// it to filter out rows that don't satisfy the predicate. +pub(crate) fn filter_by_predicate( + filter: &dyn crate::PredicateEvaluator, + batch: Box, +) -> DeltaResult> { + let predicate_result = filter.evaluate(batch.as_ref())?; + let mut visitor = SelectionVectorVisitor::default(); + visitor.visit_rows_of(predicate_result.as_ref())?; + require!( + visitor.selection_vector.len() == batch.len(), + Error::internal_error(format!( + "predicate output length {} != batch length {}", + visitor.selection_vector.len(), + batch.len() + )) + ); + batch.apply_selection_vector(visitor.selection_vector) } #[cfg(test)] mod tests { + use std::sync::Arc; + + use rstest::rstest; + use super::*; use crate::arrow::array::{RecordBatch, StringArray}; use crate::arrow::datatypes::{ DataType as ArrowDataType, Field as ArrowField, Schema as ArrowSchema, }; use crate::engine::arrow_data::ArrowEngineData; - use std::sync::Arc; fn get_engine_data(rows: usize) -> Box { let schema = Arc::new(ArrowSchema::new(vec![ArrowField::new( @@ -565,7 +718,6 @@ mod tests { } #[test] - fn test_get_binary_some_value() { use crate::arrow::array::BinaryArray; @@ -668,4 +820,56 @@ mod tests { let data = filtered.apply_selection_vector().unwrap(); assert_eq!(data.len(), 3); } + + fn collect_indices(row_count: usize, selection: &[bool]) -> Vec { + RowIndexIterator::new(row_count, selection).collect() + } + + #[rstest] + #[case(0, &[], vec![])] + #[case(3, &[], vec![0, 1, 2])] + #[case(3, &[true, true, true], vec![0, 1, 2])] + #[case(3, &[false, false, false], vec![])] + #[case(5, &[true, false, false, true, true], vec![0, 3, 4])] + #[case(4, &[false, false, true, true], vec![2, 3])] + #[case(3, &[true, false, false], vec![0])] + // sv shorter than row_count: tail rows implicitly selected + #[case(4, &[false, true], vec![1, 2, 3])] + #[case(4, &[true, false], vec![0, 2, 3])] + #[case(4, &[false, true, false, true], vec![1, 3])] + fn row_index_iter( + #[case] row_count: usize, + #[case] selection: &[bool], + #[case] expected: Vec, + ) { + assert_eq!(collect_indices(row_count, selection), expected); + } + + #[test] + fn map_item_keys_includes_null_valued_keys() { + // Map { "a" => "1", "b" => null }: `keys()` must expose both keys, while `materialize()` + // (and `get`) drop the null-valued entry. + let keys = StringArray::from(vec!["a", "b"]); + let values = StringArray::from(vec![Some("1"), None]); + let map = MapItem::new(&keys, &values, 0..2); + + let mut seen: Vec<&str> = map.keys().collect(); + seen.sort_unstable(); + assert_eq!(seen, vec!["a", "b"]); + + assert_eq!(map.get("a"), Some("1")); + assert_eq!(map.get("b"), None); + assert_eq!( + map.materialize(), + HashMap::from([("a".to_string(), "1".to_string())]) + ); + } + + #[test] + fn map_item_keys_empty() { + let keys = StringArray::from(Vec::<&str>::new()); + let values = StringArray::from(Vec::<&str>::new()); + let map = MapItem::new(&keys, &values, 0..0); + assert_eq!(map.keys().count(), 0); + } } diff --git a/kernel/src/error.rs b/kernel/src/error.rs index 8e92ff445b..ba4269a11f 100644 --- a/kernel/src/error.rs +++ b/kernel/src/error.rs @@ -1,24 +1,96 @@ //! Definitions of errors that the delta kernel can encounter -use std::{ - backtrace::{Backtrace, BacktraceStatus}, - convert::Infallible, - num::ParseIntError, - str::Utf8Error, -}; - -use crate::schema::{DataType, StructType}; -use crate::table_properties::ParseIntervalError; -use crate::Version; +use std::backtrace::{Backtrace, BacktraceStatus}; +use std::convert::Infallible; +use std::num::ParseIntError; +use std::str::Utf8Error; #[cfg(feature = "default-engine-base")] use crate::arrow::error::ArrowError; #[cfg(feature = "default-engine-base")] use crate::object_store; +use crate::schema::{DataType, StructType}; +use crate::table_properties::ParseIntervalError; +use crate::Version; + +/// Details of a failed conversion from a scalar into a Rust value. +/// +/// Conversion code adds path elements as an error unwinds, producing a path from the outermost +/// value to the value that failed without carrying mutable path state through successful parsing. +#[derive(Debug)] +pub struct ScalarConversionError { + expected: String, + actual: String, + // Stored innermost-first because parent context is appended as conversion errors unwind. + path: Vec, +} + +impl ScalarConversionError { + pub(crate) fn new(expected: impl Into, actual: impl Into) -> Self { + Self { + expected: expected.into(), + actual: actual.into(), + path: Vec::new(), + } + } + + fn add_path_context(mut self, element: impl Into) -> Self { + self.path.push(element.into()); + self + } + + fn path_string(&self) -> String { + let mut path = String::new(); + for element in self.path.iter().rev() { + if !path.is_empty() && !element.starts_with('[') { + path.push('.'); + } + path.push_str(element); + } + path + } +} + +impl std::fmt::Display for ScalarConversionError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + let mut target = self.path_string(); + if target.is_empty() { + target.push_str("scalar"); + } + write!( + f, + "Cannot convert {target}: expected {}, found {}", + self.expected, self.actual + ) + } +} + +impl std::error::Error for ScalarConversionError {} + +/// Adds an outer path element to a scalar conversion error as nested conversion unwinds. +/// +/// Other error variants are returned unchanged: a field's `TryFrom` implementation may +/// report a failure unrelated to scalar shape, and this helper must not reclassify it. +pub(crate) fn add_scalar_path_context(error: Error, element: impl Into) -> Error { + match error { + Error::ScalarConversion(error) => Error::ScalarConversion(error.add_path_context(element)), + other => other, + } +} /// A [`std::result::Result`] that has the kernel [`Error`] as the error variant pub type DeltaResult = std::result::Result; +/// A boxed, `Send` iterator of [`DeltaResult`] items. +/// +/// Convenience alias for the common pattern of returning a streaming, fallible iterator from +/// kernel APIs. +pub type DeltaResultIterator<'a, T> = Box> + Send + 'a>; + +/// `'static` counterpart to [`DeltaResultIterator`] for cases where the iterator does not +/// reference borrowed data. +pub type DeltaResultIteratorStatic = DeltaResultIterator<'static, T>; + /// All the types of errors that the kernel can run into #[non_exhaustive] #[derive(thiserror::Error, Debug)] @@ -49,6 +121,10 @@ pub enum Error { #[error("Error extracting type {0}: {1}")] Extract(&'static str, &'static str), + /// A scalar could not be converted into the requested Rust value. + #[error(transparent)] + ScalarConversion(#[from] ScalarConversionError), + /// A generic error with a message #[error("Generic delta kernel error: {0}")] Generic(String), @@ -60,6 +136,19 @@ pub enum Error { source: Box, }, + /// An error involving the maximum catalog-ratified version when building a snapshot. + #[error("Max catalog version error: {0}")] + MaxCatalogVersion(String), + + /// The supplied log tail contains adjacent versions that are not contiguous. + #[error("Log tail versions {first_version} and {second_version} are not contiguous")] + LogTailVersionsNotContiguous { + /// Earlier version in the invalid adjacent pair. + first_version: Version, + /// Later version in the invalid adjacent pair. + second_version: Version, + }, + /// Some kind of [`std::io::Error`] #[error(transparent)] IOError(std::io::Error), @@ -97,6 +186,11 @@ pub enum Error { #[error("{0}")] MissingColumn(String), + /// The connector-provided partition values are invalid (missing/extra/duplicate keys, + /// or a value type does not match the schema column type). + #[error("Invalid partition values: {0}")] + InvalidPartitionValues(String), + /// A column was specified with a specific type, but it is not of that type #[error("Expected column type: {0}")] UnexpectedColumnType(String), @@ -172,6 +266,10 @@ pub enum Error { #[error("Invalid decimal: {0}")] InvalidDecimal(String), + /// Invalid CRS or other parameter for a Geometry / Geography type + #[error("Invalid geo parameters: {0}")] + InvalidGeoParams(String), + /// Inconsistent data passed to struct scalar #[error("Invalid struct data: {0}")] InvalidStructData(String), @@ -203,6 +301,14 @@ pub enum Error { #[error("Change data feed is unsupported for the table at version {0}")] ChangeDataFeedUnsupported(Version), + /// Row tracking (`delta.enableRowTracking`) must be enabled for the entire version range of a + /// row-tracking change feed, but it is not enabled at the given version. + #[error( + "Row tracking (delta.enableRowTracking) must be enabled for the entire row-tracking change \ + feed range, but it is not enabled at version {0}" + )] + RowTrackingChangeFeedUnsupported(Version), + #[error("Change data feed encountered incompatible schema. Expected {0}, got {1}")] ChangeDataFeedIncompatibleSchema(String, String), @@ -216,17 +322,42 @@ pub enum Error { #[from] crate::expressions::literal_expression_transform::Error, ), - /// Schema mismatch has occurred or invalid schema used somewhere + /// Schema mismatch has occurred or invalid/not-kernel-supported schema used somewhere #[error("Schema error: {0}")] Schema(String), /// Validation error for file statistics (e.g., missing required clustering column stats) #[error("Stats validation error: {0}")] StatsValidation(String), + + /// Error during log history operations (timestamp queries, version lookups) + #[error(transparent)] + LogHistory(#[from] Box), + + #[cfg(feature = "declarative-plans")] + #[error("Declarative plan execution yielded the incorrect type: expected PlanResult::{expected}, got PlanResult::{actual}")] + PlanResultTypeMismatch { + expected: &'static str, + actual: &'static str, + }, + + /// The operation was cancelled via a [`CancellationToken`](crate::CancellationToken). + /// + /// Surfaced by cancellation-aware reads as a terminal error, distinct from normal iterator + /// exhaustion. See [`CancellableIterator`](crate::cancellation) for the enforced contract. + #[error("Operation cancelled")] + Cancelled, } // Convenience constructors for Error types that take a String argument impl Error { + pub(crate) fn scalar_conversion( + expected: impl Into, + actual: impl Into, + ) -> Self { + ScalarConversionError::new(expected, actual).into() + } + pub(crate) fn checkpoint_write(msg: impl ToString) -> Self { Self::CheckpointWrite(msg.to_string()) } @@ -248,6 +379,9 @@ impl Error { pub fn unexpected_column_type(name: impl ToString) -> Self { Self::UnexpectedColumnType(name.to_string()) } + pub fn invalid_partition_values(msg: impl ToString) -> Self { + Self::InvalidPartitionValues(msg.to_string()) + } pub fn missing_data(name: impl ToString) -> Self { Self::MissingData(name.to_string()) } @@ -269,6 +403,10 @@ impl Error { pub fn invalid_decimal(msg: impl ToString) -> Self { Self::InvalidDecimal(msg.to_string()) } + #[cfg(feature = "geo-type-in-dev")] + pub fn invalid_geo_params(msg: impl ToString) -> Self { + Self::InvalidGeoParams(msg.to_string()) + } pub fn invalid_struct_data(msg: impl ToString) -> Self { Self::InvalidStructData(msg.to_string()) } @@ -297,11 +435,28 @@ impl Error { pub fn change_data_feed_unsupported(version: impl Into) -> Self { Self::ChangeDataFeedUnsupported(version.into()) } + /// Creates an [`Error::RowTrackingChangeFeedUnsupported`] for the given version, used when row + /// tracking is not enabled at some point in a row-tracking change feed's version range. + pub(crate) fn row_tracking_change_feed_unsupported(version: impl Into) -> Self { + Self::RowTrackingChangeFeedUnsupported(version.into()) + } pub(crate) fn change_data_feed_incompatible_schema( expected: &StructType, actual: &StructType, ) -> Self { - Self::ChangeDataFeedIncompatibleSchema(format!("{expected:?}"), format!("{actual:?}")) + Self::ChangeDataFeedIncompatibleSchema(expected.to_string(), actual.to_string()) + } + + /// Creates an incompatible-schema error that identifies the version of `actual`. + pub(crate) fn change_data_feed_incompatible_schema_at_version( + expected: &StructType, + actual: &StructType, + version: Version, + ) -> Self { + Self::ChangeDataFeedIncompatibleSchema( + expected.to_string(), + format!("schema at version {version}: {actual}"), + ) } pub fn invalid_checkpoint(msg: impl ToString) -> Self { @@ -316,6 +471,11 @@ impl Error { Self::StatsValidation(msg.to_string()) } + #[cfg(feature = "declarative-plans")] + pub fn plan_result_type_mismatch(expected: &'static str, actual: &'static str) -> Self { + Self::PlanResultTypeMismatch { expected, actual } + } + // Capture a backtrace when the error is constructed. #[must_use] pub fn with_backtrace(self) -> Self { diff --git a/kernel/src/expressions/column_names.rs b/kernel/src/expressions/column_names.rs index 6565f8e9d1..b2f24cae94 100644 --- a/kernel/src/expressions/column_names.rs +++ b/kernel/src/expressions/column_names.rs @@ -1,11 +1,12 @@ -use crate::{DeltaResult, Error}; - use std::borrow::Borrow; use std::fmt::{Display, Formatter}; use std::hash::{Hash, Hasher}; use std::iter::Peekable; use std::ops::Deref; +use crate::utils::CollectInto; +use crate::{DeltaResult, Error}; + /// A (possibly nested) column name. #[derive(Debug, Clone, Default, PartialEq, PartialOrd, Eq, Ord, Serialize, Deserialize)] pub struct ColumnName { @@ -15,11 +16,8 @@ pub struct ColumnName { impl ColumnName { /// Creates a new column name from input satisfying `FromIterator for ColumnName`. The provided /// field names are concatenated into a single path. - pub fn new(iter: impl IntoIterator) -> Self - where - Self: FromIterator, - { - iter.into_iter().collect() + pub fn new(iter: impl CollectInto) -> Self { + iter.collect_into() } /// Naively splits a string at dots to create a column name. @@ -236,7 +234,7 @@ impl Display for ColumnName { } // Simple column names contain only simple chars, and do not need to be wrapped in backticks. -fn is_simple_char(c: char) -> bool { +pub(crate) fn is_simple_char(c: char) -> bool { c.is_ascii_alphanumeric() || c == '_' } @@ -343,8 +341,12 @@ fn parse_simple_field_name(chars: &mut Chars<'_>) -> DeltaResult { Ok(name) } -/// Parses a field name escaped with backticks, e.g. "`ab``c``d`". -fn parse_escaped_field_name(chars: &mut Chars<'_>) -> DeltaResult { +/// Parses a field name escaped with backticks, e.g. "`ab``c``d`", returning its unescaped logical +/// name. The caller must have already consumed the opening backtick. Shared with the +/// check-constraint tokenizer ([`crate::expressions::sql`]) so backtick-quoted column references +/// parse identically. +/// Examples: `col` -> col; `ab `` -> ``ab``. Returns an error if there is no closing backtick. +pub(crate) fn parse_escaped_field_name(chars: &mut Chars<'_>) -> DeltaResult { let mut name = String::new(); loop { match chars.next() { @@ -360,37 +362,81 @@ fn parse_escaped_field_name(chars: &mut Chars<'_>) -> DeltaResult { Ok(name) } +/// Validates a single column segment at compile time, returning it unchanged when valid. Used by +/// the `column_name!` proc macro for non-literal segment expressions; the macro unwraps the result +/// in a const context so an invalid segment surfaces as a compile error. +#[doc(hidden)] +pub const fn __require_valid_simple_column_segment(s: &str) -> Option<&str> { + let bytes = s.as_bytes(); + if bytes.is_empty() { + return None; + } + let mut i = 0; + while i < bytes.len() { + let b = bytes[i]; + if !b.is_ascii_alphanumeric() && b != b'_' { + return None; + } + i += 1; + } + Some(s) +} + /// Creates a nested column name whose field names are all simple column names (containing only -/// alphanumeric characters and underscores), delimited by dots. This macro is provided as a -/// convenience for the common case where the caller knows the column name contains only simple -/// field names and that splitting by periods is safe: +/// alphanumeric characters and underscores). +/// +/// Each **string-literal** argument is treated as a dot-separated path and split into segments, so +/// multiple literals concatenate into a single path: /// /// ``` /// # use delta_kernel::expressions::{column_name, ColumnName}; /// assert_eq!(column_name!("a.b.c"), ColumnName::new(["a", "b", "c"])); +/// assert_eq!(column_name!("a.b", "c.d"), ColumnName::new(["a", "b", "c", "d"])); +/// ``` +/// +/// Every **constant** argument is taken as a single segment (never split on `.`; its value is not +/// visible at the call site, so a `.` is rejected): +/// +/// ``` +/// # use delta_kernel::expressions::{column_name, ColumnName}; +/// const VERSION: &str = "version"; +/// assert_eq!(column_name!(VERSION), ColumnName::new(["version"])); +/// assert_eq!(column_name!(VERSION, "a.b"), ColumnName::new(["version", "a", "b"])); /// ``` /// -/// To avoid accidental misuse, the argument must be a string literal, so the compiler can validate -/// the safety conditions. Thus, the following uses would fail to compile: +/// The following would fail to compile: /// /// ```fail_compile /// # use delta_kernel::expressions::column_name; /// let s = "a.b"; -/// let name = column_name!(s); // not a string literal +/// let name = column_name!(s); // not a constant +/// ``` +/// +/// ```fail_compile +/// # use delta_kernel::expressions::column_name; +/// const BAD: &str = "a.b"; +/// let name = column_name!(BAD); // dots not allowed in constant segments +/// ``` +/// +/// ```fail_compile +/// # use delta_kernel::expressions::column_name; +/// let name = column_name!("a b"); // non-alphanumeric character in path /// ``` /// /// ```fail_compile -/// # use delta_kernel::expressions::simple_column_name; -/// let name = simple_column_name!("a b"); // non-alphanumeric character +/// # use delta_kernel::expressions::column_name; +/// let name = column_name!("a..b"); // empty segment /// ``` // NOTE: Macros are only public if exported, which defines them at the root of the crate. But we // don't want it there. So, we export a hidden macro and pub use it here where we actually want it. #[macro_export] #[doc(hidden)] macro_rules! __column_name { - ( $($name:tt)* ) => { - $crate::expressions::ColumnName::new($crate::delta_kernel_derive::parse_column_name!($($name)*)) - }; + ( $($segments:tt)+ ) => {{ + const SEGMENTS: &[&str] = + $crate::delta_kernel_derive::column_name_segments!($($segments)+); + $crate::expressions::ColumnName::new(SEGMENTS.iter().copied()) + }}; } #[doc(inline)] pub use __column_name as column_name; @@ -430,6 +476,19 @@ macro_rules! __joined_column_name { #[doc(inline)] pub use __joined_column_name as joined_column_name; +/// Convenience macro that builds an [`Expression`](crate::expressions::Expression) column reference +/// by forwarding all args to [`column_name!`]: +/// +/// ``` +/// # use delta_kernel::expressions::{col, ColumnName, Expression}; +/// assert_eq!(col!("a.b.c"), Expression::Column(ColumnName::new(["a", "b", "c"]))); +/// +/// const VERSION: &str = "version"; +/// assert_eq!( +/// col!(VERSION, "a.b"), +/// Expression::Column(ColumnName::new(["version", "a", "b"])) +/// ); +/// ``` #[macro_export] #[doc(hidden)] macro_rules! __column_expr { @@ -437,8 +496,10 @@ macro_rules! __column_expr { $crate::expressions::Expression::from($crate::__column_name!($($name)*)) }; } -#[doc(inline)] +#[doc(hidden)] pub use __column_expr as column_expr; +#[doc(inline)] +pub use __column_expr as col; #[macro_export] #[doc(hidden)] @@ -474,7 +535,6 @@ use serde::{Deserialize, Serialize}; #[cfg(test)] mod test { use super::*; - use delta_kernel_derive::parse_column_name; impl ColumnName { fn empty() -> Self { @@ -482,14 +542,8 @@ mod test { } } - #[test] - fn test_parse_column_name_macros() { - assert_eq!(parse_column_name!("a"), ["a"]); - - assert_eq!(parse_column_name!("a"), ["a"]); - assert_eq!(parse_column_name!("a.b"), ["a", "b"]); - assert_eq!(parse_column_name!("a.b.c"), ["a", "b", "c"]); - } + const TEST_ADD: &str = "add"; + const TEST_STATS: &str = "stats"; #[test] fn test_column_name_macros() { @@ -500,7 +554,27 @@ mod test { assert_eq!(column_name!("a.b"), ColumnName::new(["a", "b"])); assert_eq!(column_name!("a.b.c"), ColumnName::new(["a", "b", "c"])); - assert_eq!(joined_column_name!("a", "b"), ColumnName::new(["a", "b"])); + // Every string literal is split on dots, so multiple literals concatenate into one path. + assert_eq!( + column_name!("a.b", "c.d"), + ColumnName::new(["a", "b", "c", "d"]) + ); + + assert_eq!(column_name!(TEST_ADD), ColumnName::new(["add"])); + assert_eq!( + column_name!(TEST_ADD, TEST_STATS), + ColumnName::new(["add", "stats"]) + ); + assert_eq!( + column_name!(TEST_ADD, "parsed"), + ColumnName::new(["add", "parsed"]) + ); + // A constant segment mixed with a dotted literal path. + assert_eq!( + column_name!(TEST_ADD, "a.b"), + ColumnName::new(["add", "a", "b"]) + ); + assert_eq!(joined_column_name!("a", "b"), ColumnName::new(["a", "b"])); assert_eq!( diff --git a/kernel/src/expressions/literal_expression_transform.rs b/kernel/src/expressions/literal_expression_transform.rs index 04edbfc199..d30f9228f7 100644 --- a/kernel/src/expressions/literal_expression_transform.rs +++ b/kernel/src/expressions/literal_expression_transform.rs @@ -1,14 +1,11 @@ -//! The [`LiteralExpressionTransform`] is a [`SchemaTransform`] that transforms a [`Schema`] and an -//! ordered list of leaf values (scalars) into an [`Expression`] with a literal value for each leaf. +//! Transforms a [`StructType`] and an ordered list of leaf values (scalars) into an +//! [`Expression`] with a literal value for each leaf. -use std::borrow::Cow; use std::ops::Deref as _; -use tracing::debug; - use crate::expressions::{Expression, Scalar}; -use crate::schema::{ArrayType, DataType, MapType, PrimitiveType, StructField, StructType}; -use crate::transforms::SchemaTransform; +use crate::schema::{ArrayType, DataType, MapType, PrimitiveType, StructType}; +use crate::transforms::{transform_output_type, SchemaTransform}; use crate::DeltaResult; /// [`SchemaTransform`] that will transform a [`Schema`] and an ordered list of leaf values @@ -19,8 +16,6 @@ struct LiteralExpressionTransform<'a, T: Iterator> { /// A stack of built Expressions. After visiting children, we pop them off to /// build the parent container, then push the parent back on. stack: Vec, - /// Since schema transforms are infallible we keep track of errors here - error: Result<(), Error>, } /// Any error for [`LiteralExpressionTransform`] @@ -55,77 +50,52 @@ pub(crate) fn literal_expression_transform<'a>( let mut transform = LiteralExpressionTransform { scalars: scalars.into_iter(), stack: Vec::new(), - error: Ok(()), }; - let _ = transform.transform_struct(schema); - transform.error?; - - if let Some(s) = transform.scalars.next() { - return Err(Error::ExcessScalars(s.clone()).into()); - } - - transform.stack.pop().ok_or(Error::EmptyStack.into()) -} - -impl<'a, I: Iterator> LiteralExpressionTransform<'a, I> { - fn set_error(&mut self, error: Error) { - // Only set when the error not yet set - if let Err(ref existing_error) = self.error { - debug!("Trying to overwrite an existing error: {existing_error:?} with {error:?}"); - } else { - self.error = Err(error); - } + transform.transform_struct(schema)?; + match transform.scalars.next() { + Some(s) => Err(Error::ExcessScalars(s.clone()).into()), + None => transform.stack.pop().ok_or(Error::EmptyStack.into()), } } // All leaf types (primitive, array, map) share the same "shape" of transformation logic macro_rules! transform_leaf { ($self:ident, $type_variant:path, $type:ident) => {{ - // first always check error to terminate early if possible - $self.error.as_ref().ok()?; - let Some(scalar) = $self.scalars.next() else { - $self.set_error(Error::InsufficientScalars); - return None; + return Err(Error::InsufficientScalars); }; // NOTE: Grab a reference here so code below can leverage the blanket impl Deref for &T let $type_variant(ref scalar_type) = scalar.data_type() else { - $self.set_error(Error::Schema(format!( + return Err(Error::Schema(format!( "Mismatched scalar type while creating Expression: expected {}({:?}), got {:?}", stringify!($type_variant), $type, scalar.data_type() ))); - return None; }; // NOTE: &T and &Box both deref to &T if scalar_type.deref() != $type { - $self.set_error(Error::Schema(format!( + return Err(Error::Schema(format!( "Mismatched scalar type while creating Expression: expected {:?}, got {:?}", $type, scalar_type ))); - return None; } $self.stack.push(Expression::Literal(scalar.clone())); - None + Ok(()) }}; } impl<'a, T: Iterator> SchemaTransform<'a> for LiteralExpressionTransform<'a, T> { - fn transform_primitive( - &mut self, - prim_type: &'a PrimitiveType, - ) -> Option> { + transform_output_type!(|'a, U| Result<(), Error>); + + fn transform_primitive(&mut self, prim_type: &'a PrimitiveType) -> Result<(), Error> { transform_leaf!(self, DataType::Primitive, prim_type) } - fn transform_struct(&mut self, struct_type: &'a StructType) -> Option> { - // first always check error to terminate early if possible - self.error.as_ref().ok()?; - + fn transform_struct(&mut self, struct_type: &'a StructType) -> Result<(), Error> { // Only consume newly-added entries (if any). There could be fewer than expected if // the recursion encountered an error. let mark = self.stack.len(); @@ -134,8 +104,7 @@ impl<'a, T: Iterator> SchemaTransform<'a> for LiteralExpressi let fields = struct_type.fields(); if field_exprs.len() != fields.len() { - self.set_error(Error::InsufficientScalars); - return None; + return Err(Error::InsufficientScalars); } let mut found_non_nullable_null = false; @@ -153,10 +122,9 @@ impl<'a, T: Iterator> SchemaTransform<'a> for LiteralExpressi let struct_expr = if found_non_nullable_null { if !all_null { // we found a non_nullable NULL, but other siblings are non-null: error - self.set_error(Error::Schema( + return Err(Error::Schema( "NULL value for non-nullable struct field with non-NULL siblings".to_string(), )); - return None; } Expression::null_literal(struct_type.clone().into()) } else { @@ -164,43 +132,34 @@ impl<'a, T: Iterator> SchemaTransform<'a> for LiteralExpressi }; self.stack.push(struct_expr); - None - } - - fn transform_struct_field(&mut self, field: &'a StructField) -> Option> { - // first always check error to terminate early if possible - self.error.as_ref().ok()?; - - self.recurse_into_struct_field(field); - Some(Cow::Borrowed(field)) + Ok(()) } // arrays treated as leaves - fn transform_array(&mut self, array_type: &'a ArrayType) -> Option> { + fn transform_array(&mut self, array_type: &'a ArrayType) -> Result<(), Error> { transform_leaf!(self, DataType::Array, array_type) } // maps treated as leaves - fn transform_map(&mut self, map_type: &'a MapType) -> Option> { + fn transform_map(&mut self, map_type: &'a MapType) -> Result<(), Error> { transform_leaf!(self, DataType::Map, map_type) } + + // NOTE: No support for variant scalar values yet, so nothing to transform. } #[cfg(test)] mod tests { - use super::*; - use std::sync::Arc; - use crate::expressions::{ArrayData, MapData}; - use crate::schema::SchemaRef; - use crate::schema::StructType; - use crate::DataType as DeltaDataTypes; - use paste::paste; - use Expression as Expr; + use super::*; + use crate::expressions::{ArrayData, MapData}; + use crate::schema::{schema_ref, SchemaRef, StructField, StructType}; + use crate::DataType as DeltaDataTypes; + // helper to take values/schema to pass to `create_one` and assert the result = expected fn assert_single_row_transform( values: &[Scalar], @@ -218,17 +177,11 @@ mod tests { fn test_create_one_top_level_null() { let values = &[Scalar::Null(DeltaDataTypes::INTEGER)]; - let schema = Arc::new(StructType::new_unchecked([StructField::not_null( - "col_1", - DeltaDataTypes::INTEGER, - )])); + let schema = schema_ref! { not_null "col_1": INTEGER }; let expected = Expr::null_literal(schema.clone().into()); assert_single_row_transform(values, schema, Ok(expected)); - let schema = Arc::new(StructType::new_unchecked([StructField::nullable( - "col_1", - DeltaDataTypes::INTEGER, - )])); + let schema = schema_ref! { nullable "col_1": INTEGER }; let expected = Expr::struct_from(vec![Expr::null_literal(DeltaDataTypes::INTEGER)]); assert_single_row_transform(values, schema, Ok(expected)); } @@ -267,22 +220,16 @@ mod tests { #[test] fn test_many_structs() { let values: &[Scalar] = &[1.into(), 2.into(), 3.into(), 4.into()]; - let schema = Arc::new(StructType::new_unchecked([ - StructField::nullable( - "x", - DeltaDataTypes::struct_type_unchecked([ - StructField::not_null("a", DeltaDataTypes::INTEGER), - StructField::nullable("b", DeltaDataTypes::INTEGER), - ]), - ), - StructField::nullable( - "y", - DeltaDataTypes::struct_type_unchecked([ - StructField::not_null("c", DeltaDataTypes::INTEGER), - StructField::nullable("d", DeltaDataTypes::INTEGER), - ]), - ), - ])); + let schema = schema_ref! { + nullable "x": { + not_null "a": INTEGER, + nullable "b": INTEGER, + }, + nullable "y": { + not_null "c": INTEGER, + nullable "d": INTEGER, + }, + }; let expected = Expr::struct_from(vec![ Expr::struct_from(vec![Expr::literal(1), Expr::literal(2)]), Expr::struct_from(vec![Expr::literal(3), Expr::literal(4)]), @@ -301,8 +248,8 @@ mod tests { Scalar::Array(array_data.clone()), ]; let schema = Arc::new(StructType::new_unchecked([ - StructField::nullable("map", DeltaDataTypes::Map(Box::new(map_type))), - StructField::nullable("array", DeltaDataTypes::Array(Box::new(array_type))), + StructField::nullable("map", map_type), + StructField::nullable("array", array_type), ])); let expected = Expr::struct_from(vec![ Expr::literal(Scalar::Map(map_data)), diff --git a/kernel/src/expressions/mod.rs b/kernel/src/expressions/mod.rs index 6b72ab20c4..50be3b0308 100644 --- a/kernel/src/expressions/mod.rs +++ b/kernel/src/expressions/mod.rs @@ -1,15 +1,16 @@ //! Definitions and functions to create and manipulate kernel expressions -use std::borrow::Cow; -use std::collections::{HashMap, HashSet}; +use std::collections::HashSet; use std::fmt::{Display, Formatter}; use std::sync::Arc; use itertools::Itertools; use serde::{de, ser, Deserialize, Deserializer, Serialize, Serializer}; +#[doc(hidden)] +pub use self::column_names::{__require_valid_simple_column_segment, column_expr}; pub use self::column_names::{ - column_expr, column_expr_ref, column_name, column_pred, joined_column_expr, joined_column_name, + col, column_expr_ref, column_name, column_pred, joined_column_expr, joined_column_name, ColumnName, }; pub use self::scalars::{ArrayData, DecimalData, MapData, Scalar, StructData}; @@ -18,17 +19,39 @@ use crate::kernel_predicates::{ IndirectDataSkippingPredicateEvaluator, }; use crate::schema::SchemaRef; -use crate::transforms::ExpressionTransform; -use crate::{DataType, DeltaResult, DynPartialEq}; +pub use crate::struct_patch::{ExpressionFieldPatch, ExpressionStructPatch}; +use crate::transforms::{transform_output_type, ExpressionTransform}; +use crate::utils::CollectInto; +use crate::{DataType, DeltaResult, DynPartialEq, Error}; mod column_names; pub(crate) mod literal_expression_transform; pub(crate) use literal_expression_transform::literal_expression_transform; mod scalars; +mod sql; +pub(crate) use self::sql::parse_sql; pub type ExpressionRef = std::sync::Arc; pub type PredicateRef = std::sync::Arc; +/// Build an [`Expression::Literal`] from anything that converts into a [`Scalar`]. +/// +/// Concise alternative to [`Expression::literal`] for plan builders. Accepts the same value +/// types [`Scalar`] does (`i32`, `i64`, `&str`, `bool`, ...). +/// +/// ``` +/// use delta_kernel::expressions::lit; +/// let _zero = lit(0i64); +/// ``` +pub fn lit(value: impl Into) -> Expression { + Expression::literal(value) +} + +/// A [`StructPatchBuilder`](crate::struct_patch::StructPatchBuilder) whose emitted items are +/// expressions, lowered into an [`ExpressionStructPatch`] that can be embedded in an +/// [`Expression`]. +pub type ExpressionStructPatchBuilder = crate::struct_patch::StructPatchBuilder; + //////////////////////////////////////////////////////////////////////// // Operators //////////////////////////////////////////////////////////////////////// @@ -36,58 +59,127 @@ pub type PredicateRef = std::sync::Arc; /// A unary predicate operator. #[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)] pub enum UnaryPredicateOp { - /// Unary Is Null + /// SQL `expr IS NULL`: `true` when the input is null and `false` otherwise, never null itself. + /// A wrapping `NOT` inverts it to `IS NOT NULL`. IsNull, } /// A binary predicate operator. +/// +/// The ordering and equality comparisons follow SQL three-valued logic, so a NULL operand yields +/// NULL rather than `false`. [`Distinct`](Self::Distinct) and [`In`](Self::In) instead tolerate +/// NULL and always answer `true` or `false`. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] pub enum BinaryPredicateOp { - /// Comparison Less Than + /// `left < right`. LessThan, - /// Comparison Greater Than + /// `left > right`. GreaterThan, - /// Comparison Equal + /// `left = right`. Equal, - /// Distinct + /// SQL `left IS DISTINCT FROM right`: null-safe inequality, treating NULL as an ordinary + /// value. `DISTINCT(1, 1)` and `DISTINCT(NULL, NULL)` are `false`; `DISTINCT(NULL, 1)` is + /// `true`. Distinct, - /// IN + /// SQL `left IN (elements)`: returns `false` when `left` is NULL. Otherwise, it returns `true` + /// when `left` equals any element and `false` when none match. NULL elements never match. + /// `left` must be a literal, and the elements are either a list-typed column or an + /// [`Expression::Literal`] holding a [`Scalar::Array`], never a list of expressions or a + /// subquery: + /// + /// ```sql + /// 2 IN (1, 2, 3) -- literal elements + /// 2 IN (SELECT ...) -- unsupported: no subquery form + /// col IN (1, 2, 3) -- unsupported: the left operand must be a literal + /// ``` + /// + /// Testing a literal against a list column is the shape data skipping uses, and it is the + /// reason the operands sit this way around rather than the more familiar + /// column-on-the-left form. In, } /// A unary expression operator. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] pub enum UnaryExpressionOp { - /// Convert struct data to JSON-encoded strings + /// SQL `to_json(expr)`: encode a struct as a JSON object string, one string per row. The input + /// must be a struct and the output is STRING. A NULL input row produces a NULL string rather + /// than `"null"`. This is the inverse of [`ParseJsonExpression`] for every type except + /// timestamps, whose sub-millisecond precision this operator discards (see below). + /// + /// Nested structs and arrays encode as JSON objects and arrays. Binary encodes as lowercase + /// hex rather than base64, two digits per byte in the order the bytes appear, so + /// `{ b: 0xABCD, l: [1, 2], n: { z: 7 } }` becomes: + /// + /// ```text + /// {"b":"abcd","l":[1,2],"n":{"z":7}} + /// ``` + /// + /// Timestamps must encode with exactly three fractional digits, truncated toward negative + /// infinity, because kernel writes `add.stats` with this operator and [Per-file Statistics] + /// truncates timestamp statistics down to milliseconds. TIMESTAMP takes a literal `Z` suffix + /// and TIMESTAMP_NTZ takes no offset, so `{ ts: 2026-07-02T15:55:55.298677Z }` becomes + /// `{"ts":"2026-07-02T15:55:55.298Z"}`. Emitting more digits, or rounding up, makes readers + /// prune files that hold matching rows. + /// + /// [Per-file Statistics]: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#per-file-statistics ToJson, } -/// A binary expression operator. +/// A binary arithmetic operator over two numeric operands. +/// +/// Both operands share a numeric type and the result takes that type. Kernel inserts no implicit +/// casts, so widening a result (decimal precision or scale, for instance) needs an explicit +/// [`Expression::cast`] to match a declared output type. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] pub enum BinaryExpressionOp { - /// Arithmetic Plus + /// `left + right`. Plus, - /// Arithmetic Minus + /// `left - right`. Minus, - /// Arithmetic Multiply + /// `left * right`. Multiply, - /// Arithmetic Divide + /// `left / right`. A zero divisor never yields NULL. + /// + /// Integer operands divide truncating toward zero, and a zero divisor fails. Float operands + /// follow IEEE 754: `+/-inf` for a non-zero numerator, `NaN` for `0.0 / 0.0`. In a dialect + /// whose `/` is always fractional, the integer case is the other division operator: + /// + /// ```sql + /// 7 DIV 2 -- 3, this operator over integers + /// 7 / 2 -- 3.5, NOT this operator + /// ``` Divide, } /// A variadic expression operator. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] pub enum VariadicExpressionOp { - /// Collapse multiple values into one by taking the first non-null value + /// SQL `COALESCE(exprs...)`: the first non-null value, or null when every input is null. All + /// inputs share one type, which is also the result type. Requires at least one input. Coalesce, + /// SQL `ARRAY(exprs...)`: an array built by evaluating each input per row, so + /// `ARRAY(1, 1 + 2, my_int_col)` yields `[1, 3, ]`. All inputs must share + /// the same element type. Requires at least one element; the element type is inferred from + /// the inputs. + /// + /// For static array literals whose elements are all compile-time constants, use + /// [`Scalar::Array`] instead. The difference is that `Array` is evaluated at runtime, while + /// `Scalar::Array` is evaluated at compile time. + Array, } -/// A junction (AND/OR) predicate operator. +/// A junction (AND/OR) predicate operator over N child predicates. +/// +/// Both use SQL three-valued (Kleene) logic, treating a NULL child as unknown rather than false: a +/// decisive child wins over a NULL sibling, so `AND` is `false` and `OR` is `true` despite the +/// NULL, and only an undecided junction yields NULL. [`Predicate::junction`] folds an empty +/// junction to its operator's identity, `true` for `AND` and `false` for `OR`. #[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)] pub enum JunctionPredicateOp { - /// Conjunction + /// SQL `a AND b AND ...`: true when every child is true. And, - /// Disjunction + /// SQL `a OR b OR ...`: true when any child is true. Or, } @@ -237,9 +329,28 @@ pub struct VariadicExpression { pub exprs: Vec, } -/// An expression that parses a JSON string into a struct with the given schema. -/// This is the inverse of `ToJson` - it converts a JSON-encoded string column into a -/// struct column. +/// An expression that parses a JSON string column into a struct column of `output_schema`, the +/// inverse of [`UnaryExpressionOp::ToJson`] except for the sub-millisecond timestamp precision +/// that operator discards. +/// +/// Unparseable input must degrade to NULL rather than fail the query, because kernel parses +/// `add.stats` with this operator and data skipping reads null stats as "include the file". The +/// required part is that it does not error; whether a given row comes back as a null struct or as a +/// struct of null fields is unspecified, since data skipping treats the two alike. +/// +/// An empty string is not valid JSON here, so it is unparseable. This operator does not share +/// [`MapToStructExpression`]'s empty-string-to-NULL behavior. It is SQL `from_json(json_expr, +/// output_schema)` in a dialect whose `from_json` is permissive rather than strict. +/// +/// # Default engine behavior +/// +/// `arrow-json`'s typed decoders reject a whole batch when one cell fails to parse. The default +/// engine works around that for the leaf types that fail most often (timestamp, date, decimal) by +/// decoding them as strings and safe-casting back, so a bad value in one of those degrades to a +/// NULL for that field alone. Anything the workaround does not cover, namely structurally invalid +/// JSON and a type mismatch on any other leaf, falls back to nulling the entire batch rather than +/// the offending row. A NULL input decodes as `{}`, leaving every field NULL without disturbing the +/// rest of the batch. #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] pub struct ParseJsonExpression { /// The expression that evaluates to a STRING column containing JSON objects. @@ -248,6 +359,84 @@ pub struct ParseJsonExpression { pub output_schema: SchemaRef, } +/// Looks up one value in a `Map` for each input row. +/// +/// Missing keys, null maps, and null keys evaluate to NULL. Duplicate keys are resolved by taking +/// the rightmost entry. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +pub struct ElementAtExpression { + /// The expression that evaluates to a `Map` column. + pub map_expr: Box, + + /// The expression that evaluates to a STRING key column. + pub key_expr: Box, +} + +#[derive(Clone, Debug, Default, Eq, Hash, PartialEq, Serialize, Deserialize)] +struct TimestampInterpretation { + #[serde(default, skip_serializing_if = "Option::is_none")] + timestamp_timezone: Option, +} + +impl TimestampInterpretation { + fn with_timestamp_timezone(mut self, timestamp_timezone: impl Into) -> Self { + self.timestamp_timezone = Some(timestamp_timezone.into()); + self + } + + fn timestamp_timezone(&self) -> Option<&str> { + self.timestamp_timezone.as_deref() + } + + fn is_default(&self) -> bool { + self.timestamp_timezone.is_none() + } +} + +/// Options controlling how a [`CastExpression`] evaluates values. +#[derive(Clone, Debug, Default, Eq, Hash, PartialEq, Serialize, Deserialize)] +pub struct CastOptions { + #[serde(flatten)] + timestamp: TimestampInterpretation, +} + +impl CastOptions { + /// Interpret offset-less strings cast to `TIMESTAMP` in `timestamp_timezone`. + /// + /// The timezone may be an IANA name or a fixed offset such as `+12:45:30`. Explicit offsets in + /// input strings take precedence. Ambiguous local times use the earlier instant, while + /// nonexistent local times use the pre-transition offset. Invalid timezones are reported + /// during evaluation. + pub fn with_timestamp_timezone(mut self, timestamp_timezone: impl Into) -> Self { + self.timestamp = self.timestamp.with_timestamp_timezone(timestamp_timezone); + self + } + + /// Returns the timezone used to interpret offset-less strings cast to `TIMESTAMP`. + pub fn timestamp_timezone(&self) -> Option<&str> { + self.timestamp.timestamp_timezone() + } + + fn is_default(&self) -> bool { + self.timestamp.is_default() + } +} + +/// An expression that casts a child expression to a target type, following SQL `CAST` semantics: a +/// value that cannot be represented in the target type evaluates to NULL rather than erroring. +/// Offset-less strings cast to `TIMESTAMP` use the timezone in [`CastOptions`], or UTC when no +/// timezone is configured. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +pub struct CastExpression { + /// The expression whose value is cast. + pub expr: Box, + /// The type the value is cast to. + pub target: DataType, + /// Options controlling cast evaluation. + #[serde(default, skip_serializing_if = "CastOptions::is_default")] + pub options: CastOptions, +} + #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] pub struct JunctionPredicate { /// The operator. @@ -335,111 +524,6 @@ where Err(de::Error::custom("Cannot deserialize an Opaque Expression")) } -/// A transformation affecting a single field (one pieces of a [`Transform`]). The transformation -/// could insert 0+ new fields after the target, or could replace the target with 0+ a new fields). -#[derive(Debug, Clone, PartialEq, Default, Serialize, Deserialize)] -pub struct FieldTransform { - /// The list of expressions this field transform emits at the target location. - pub exprs: Vec, - /// If true, the output expressions replace the input field instead of following after it. - pub is_replace: bool, - /// If true, this transform is silently ignored when the target field does not exist in the - /// input. Otherwise, a missing target field produces an error. - pub optional: bool, -} - -/// A transformation that efficiently represents sparse modifications to struct schemas. -/// -/// `Transform` achieves `O(changes)` space complexity instead of `O(schema_width)` by only -/// specifying those fields that actually change (inserted, replaced, or deleted). Any input field -/// not specifically mentioned by the transform is passed through, unmodified and with the same -/// relative field ordering. This is particularly useful for wide schemas where only a few columns -/// need to be modified and/or dropped, or where a small number of columns need to be injected. -#[derive(Debug, Clone, PartialEq, Default, Serialize, Deserialize)] -pub struct Transform { - /// The path to the nested input struct this transform operates on (if any). If no path is - /// given, the transform operates directly on top-level columns. - pub input_path: Option, - /// A mapping from named input fields to the transform to be performed on each field. - pub field_transforms: HashMap, - /// A list of new fields to emit before processing the first input field. - pub prepended_fields: Vec, -} - -impl Transform { - /// Creates a new top-level identity transform. The various `with_xxx` helper methods can be - /// used to add specific field transforms. - pub fn new_top_level() -> Self { - Self::default() - } - - /// Creates a new identity transform that operates on fields of a nested struct identified by - /// `path`. The various `with_xxx` helper methods can be used to add specific field transforms. - pub fn new_nested(path: impl IntoIterator) -> Self - where - ColumnName: FromIterator, - { - Self { - input_path: Some(ColumnName::new(path)), - ..Default::default() - } - } - - /// Specifies a field to drop. - pub fn with_dropped_field(mut self, name: impl Into) -> Self { - let field_transform = self.field_transform(name); - field_transform.is_replace = true; - self - } - - /// Like [`Self::with_dropped_field`], but silently ignored if the field does not exist. - pub fn with_dropped_field_if_exists(mut self, name: impl Into) -> Self { - let field_transform = self.field_transform(name); - field_transform.is_replace = true; - field_transform.optional = true; - self - } - - /// Specifies an expression to replace a field with. - pub fn with_replaced_field(mut self, name: impl Into, expr: ExpressionRef) -> Self { - let field_transform = self.field_transform(name); - field_transform.exprs.push(expr); - field_transform.is_replace = true; - self - } - - /// Specifies an expression to insert after an optional predecessor (None = prepend, emit the - /// expression before the first input field). Multiple fields can be inserted after the same - /// predecessor, and they will be emitted in the same order they were registered. - pub fn with_inserted_field( - mut self, - after: Option>, - expr: ExpressionRef, - ) -> Self { - match after { - Some(field_name) => self.field_transform(field_name).exprs.push(expr), - None => self.prepended_fields.push(expr), - } - self - } - - /// True if this is the identity transform (all input fields pass through unchanged, with no new - /// fields inserted). - pub fn is_identity(&self) -> bool { - self.prepended_fields.is_empty() && self.field_transforms.is_empty() - } - - /// None, if this is a top-level transform. Otherwise, the path of this nested transform. - pub fn input_path(&self) -> Option<&ColumnName> { - self.input_path.as_ref() - } - - // Gets or creates the field transform for a named input field - fn field_transform(&mut self, field_name: impl Into) -> &mut FieldTransform { - self.field_transforms.entry(field_name.into()).or_default() - } -} - /// A SQL expression. /// /// These expressions do not track or validate data types, other than the type @@ -449,16 +533,31 @@ impl Transform { pub enum Expression { /// A literal value. Literal(Scalar), - /// A column reference by name. + /// A column reference by name. A [`ColumnName`] is a path, so a multi-segment name like + /// `add.stats.numRecords` descends one nested struct field per segment, matching by name. Column(ColumnName), /// A predicate treated as a boolean expression Predicate(Box), // should this be Arc? - /// A struct computed from a Vec of expressions. - /// The optional nullability predicate, if provided and evaluates to false/null, makes the entire struct null. + /// A struct computed from one expression per output field, in field order. + /// + /// Field names and nullability come from the surrounding output schema (the evaluator's + /// `result_type`, such as a [`Project`]'s target field), and each field expression's type is + /// validated against the schema, so building a struct takes both. The expression count must + /// equal the output field count. + /// + /// The optional nullability predicate says when to *keep* the struct: a row survives only + /// where it is true, and nulls entirely where it is false or null. + /// + /// ```sql + /// CASE WHEN keep_pred THEN struct(expr1, expr2, ...) END + /// ``` + /// + /// [`Project`]: crate::plans::ir::nodes::Project Struct(Vec, Option), - /// A sparse transformation of a struct schema. More efficient than `Struct` for wide schemas + /// A sparse patch of a struct. More efficient than `Struct` for wide schemas /// where only a few fields change, achieving O(changes) instead of O(schema_width) complexity. - Transform(Transform), + #[serde(alias = "Transform")] + StructPatch(ExpressionStructPatch), /// An expression that takes one expression as input. Unary(UnaryExpression), /// An expression that takes two expressions as input. @@ -479,11 +578,17 @@ pub enum Expression { /// all rows -- almost certainly NOT what the query author intended. Use `Expression::Opaque` /// for expressions kernel doesn't understand but which engine can still evaluate. Unknown(String), - /// Parse a JSON string expression into a struct with the given schema. + /// Parse a JSON string expression into a struct with the given schema. Unparseable input, + /// which includes an empty string, must yield NULL rather than error; see + /// [`ParseJsonExpression`]. ParseJson(ParseJsonExpression), - /// Extract keys from a `Map` and parse values into a typed struct using - /// Delta's partition value serialization rules. + /// Extract keys from a `Map` and parse values into a typed struct. See + /// [`MapToStructExpression`] for how values are parsed. MapToStruct(MapToStructExpression), + /// Looks up a STRING value in a `Map`. + ElementAt(ElementAtExpression), + /// Cast a child expression to a target type. See [`CastExpression`]. + Cast(CastExpression), } /// A SQL predicate. @@ -514,13 +619,13 @@ pub enum Predicate { #[serde(deserialize_with = "fail_deserialize_opaque_predicate")] Opaque(OpaquePredicate), /// An unknown predicate (i.e. one that neither kernel nor engine attempts to evaluate). For - /// data skipping purposes, kernel treats unknown predicates as if they were literal NULL values - /// (which may disable skipping if it "poisons" the predicate), but engines MUST NOT attempt to - /// interpret them as NULL when evaluating query filters because it could produce incorrect - /// results. For example, converting `WHERE ` to `WHERE NULL` is - /// equivalent to `WHERE FALSE` and would filter out all rows -- almost certainly NOT what the - /// query author intended. Use `Predicate::Opaque` for predicates kernel doesn't understand - /// but which engine can still evaluate. + /// data skipping purposes, kernel treats unknown predicates as if they were literal NULL + /// values (which may disable skipping if it "poisons" the predicate), but engines MUST NOT + /// attempt to interpret them as NULL when evaluating query filters because it could + /// produce incorrect results. For example, converting `WHERE ` to + /// `WHERE NULL` is equivalent to `WHERE FALSE` and would filter out all rows -- almost + /// certainly NOT what the query author intended. Use `Predicate::Opaque` for predicates + /// kernel doesn't understand but which engine can still evaluate. Unknown(String), } @@ -606,27 +711,80 @@ impl ParseJsonExpression { } } +/// Options controlling how a [`MapToStructExpression`] parses map values. +#[derive(Clone, Debug, Default, Eq, Hash, PartialEq, Serialize, Deserialize)] +pub struct MapToStructOptions { + #[serde(flatten)] + timestamp: TimestampInterpretation, +} + +impl MapToStructOptions { + /// Interpret offset-less `TIMESTAMP` values in `timestamp_timezone`. + /// + /// The timezone may be an IANA name or a fixed offset such as `+12:45:30`. Explicit offsets in + /// values take precedence. Ambiguous local times use the earlier instant, while nonexistent + /// local times use the pre-transition offset. Invalid timezones are reported during + /// evaluation. + pub fn with_timestamp_timezone(mut self, timestamp_timezone: impl Into) -> Self { + self.timestamp = self.timestamp.with_timestamp_timezone(timestamp_timezone); + self + } + + /// Returns the timezone used to interpret offset-less `TIMESTAMP` values. + pub fn timestamp_timezone(&self) -> Option<&str> { + self.timestamp.timestamp_timezone() + } + + fn is_default(&self) -> bool { + self.timestamp.is_default() + } +} + +impl ElementAtExpression { + pub(crate) fn new(map_expr: impl Into, key_expr: impl Into) -> Self { + Self { + map_expr: Box::new(map_expr.into()), + key_expr: Box::new(key_expr.into()), + } + } +} + /// Transforms a `Map` column into a struct whose schema is provided by the /// evaluator's output type (via `result_type`). Each row in the map column becomes one row in /// the output struct column: a `key` -> `value` mapping in the map means the struct field named -/// `key` receives `value`, parsed into the field's target type using Delta's partition value -/// serialization rules ([`PrimitiveType::parse_scalar`]). +/// `key` receives `value`, parsed into the field's target type. Offset-less `TIMESTAMP` values use +/// the timezone in [`MapToStructOptions`], or UTC when no timezone is configured. An empty-string +/// value is the exception (aligning with Spark): it casts to itself for string, to empty bytes for +/// binary, and to null for every other type. This empty-string rule is specific to this operator; +/// [`ParseJsonExpression`] does not share it. /// /// - Missing keys produce null values -/// - Parse errors are propagated (indicating a broken table) +/// - A value that cannot be parsed as its target field type returns [`Error::ParseError`] /// - Duplicate map keys are resolved by taking the rightmost entry -/// -/// [`PrimitiveType::parse_scalar`]: crate::schema::PrimitiveType::parse_scalar #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] pub struct MapToStructExpression { /// The expression that evaluates to a `Map` column. pub map_expr: Box, + /// Options controlling value parsing. + #[serde(default, skip_serializing_if = "MapToStructOptions::is_default")] + pub options: MapToStructOptions, } impl MapToStructExpression { - pub(crate) fn new(map_expr: impl Into) -> Self { + pub(crate) fn new(map_expr: impl Into, options: MapToStructOptions) -> Self { Self { map_expr: Box::new(map_expr.into()), + options, + } + } +} + +impl CastExpression { + pub(crate) fn new(expr: impl Into, target: DataType, options: CastOptions) -> Self { + Self { + expr: Box::new(expr.into()), + target, + options, } } } @@ -641,15 +799,12 @@ impl Expression { /// Returns a set of columns referenced by this expression. pub fn references(&self) -> HashSet<&ColumnName> { let mut references = GetColumnReferences::default(); - let _ = references.transform_expr(self); + references.transform_expr(self); references.0 } /// Create a new column name expression from input satisfying `FromIterator for ColumnName`. - pub fn column(field_names: impl IntoIterator) -> Expression - where - ColumnName: FromIterator, - { + pub fn column(field_names: impl CollectInto) -> Expression { ColumnName::new(field_names).into() } @@ -695,9 +850,21 @@ impl Expression { ) } - /// Create a new transform expression - pub fn transform(transform: Transform) -> Self { - Self::Transform(transform) + /// Creates a new struct patch expression from a raw patch or patch builder. + /// + /// Returns an expression that applies the supplied sparse patch to an input struct. Passing a + /// raw [`ExpressionStructPatch`] is infallible; passing an [`ExpressionStructPatchBuilder`] + /// validates and lowers the recorded operations before constructing the expression. + /// + /// # Errors + /// + /// Returns an error if the supplied patch builder contains conflicting operations. + pub fn struct_patch

    (patch: P) -> DeltaResult + where + P: TryInto, + Error: From, + { + Ok(Self::StructPatch(patch.try_into()?)) } /// Create a new predicate `self IS NULL` @@ -775,6 +942,11 @@ impl Expression { Self::variadic(VariadicExpressionOp::Coalesce, exprs) } + /// Creates a new Array constructor expression. See [`VariadicExpressionOp::Array`]. + pub fn array(exprs: impl IntoIterator>) -> Self { + Self::variadic(VariadicExpressionOp::Array, exprs) + } + /// Creates a new opaque expression pub fn opaque( op: impl OpaqueExpressionOp, @@ -789,41 +961,62 @@ impl Expression { } /// Creates a new ParseJson expression that parses a JSON string column into a struct. - /// This is the inverse of `ToJson` - it converts a JSON-encoded string into a struct. + /// This is the inverse of [`UnaryExpressionOp::ToJson`] - it converts a JSON-encoded string + /// into a struct. Sub-millisecond timestamp precision does not survive the round trip, since + /// `ToJson` truncates it. pub fn parse_json(json_expr: impl Into, output_schema: SchemaRef) -> Self { Self::ParseJson(ParseJsonExpression::new(json_expr, output_schema)) } - /// Extracts keys from a `Map` and parses values into a typed struct using - /// Delta's partition value serialization rules. The output struct schema is determined by the - /// evaluator's `result_type`. - pub fn map_to_struct(map_expr: impl Into) -> Self { - Self::MapToStruct(MapToStructExpression::new(map_expr)) + /// Extracts keys from a `Map` and parses values into a typed struct according + /// to `options`. The output struct schema is determined by the evaluator's `result_type`. + /// [`MapToStructOptions::default`] interprets offset-less timestamps in UTC. An empty-string + /// value is the exception (aligning with Spark): it casts to itself for string, to empty bytes + /// for binary, and to null for every other type. See [`MapToStructExpression`] for the full + /// contract. + pub fn map_to_struct(map_expr: impl Into, options: MapToStructOptions) -> Self { + Self::MapToStruct(MapToStructExpression::new(map_expr, options)) + } + + /// Looks up a STRING value in a `Map` for each input row. + pub fn element_at(map_expr: impl Into, key_expr: impl Into) -> Self { + Self::ElementAt(ElementAtExpression::new(map_expr, key_expr)) + } + + /// Casts `expr` to `target` according to `options`, following SQL `CAST` semantics + /// (unrepresentable values become NULL). [`CastOptions::default`] retains the standard cast + /// behavior and interprets offset-less timestamps in UTC. See [`CastExpression`]. + pub fn cast(expr: impl Into, target: DataType, options: CastOptions) -> Self { + Self::Cast(CastExpression::new(expr, target, options)) } } impl Predicate { + /// Literal boolean true + pub const TRUE: Self = Self::literal(true); + /// Literal boolean false + pub const FALSE: Self = Self::literal(false); + /// NULL boolean literal + pub const NULL: Self = Self::null_literal(); + /// Returns a set of columns referenced by this predicate. pub fn references(&self) -> HashSet<&ColumnName> { let mut references = GetColumnReferences::default(); - let _ = references.transform_pred(self); + references.transform_pred(self); references.0 } /// Creates a new boolean column reference. See also [`Expression::column`]. - pub fn column(field_names: impl IntoIterator) -> Predicate - where - ColumnName: FromIterator, - { + pub fn column(field_names: impl CollectInto) -> Self { Self::from_expr(ColumnName::new(field_names)) } - /// Create a new literal boolean value + /// Create a new literal boolean value. See also [`Self::TRUE`] and [`Self::FALSE`]. pub const fn literal(value: bool) -> Self { Self::BooleanExpression(Expression::Literal(Scalar::Boolean(value))) } - /// Creates a NULL literal boolean value + /// Creates a NULL literal boolean value. Prefer [`Self::NULL`]. pub const fn null_literal() -> Self { Self::BooleanExpression(Expression::Literal(Scalar::Null(DataType::BOOLEAN))) } @@ -842,12 +1035,12 @@ impl Predicate { } /// Create a new predicate `self IS NULL` - pub fn is_null(expr: impl Into) -> Predicate { + pub fn is_null(expr: impl Into) -> Self { Self::unary(UnaryPredicateOp::IsNull, expr) } /// Create a new predicate `self IS NOT NULL` - pub fn is_not_null(expr: impl Into) -> Predicate { + pub fn is_not_null(expr: impl Into) -> Self { Self::not(Self::is_null(expr)) } @@ -929,8 +1122,8 @@ impl Predicate { /// Creates a new junction predicate OP(preds...). Normalizes degenerate cases: /// - /// - Empty junction returns the identity element (the value that has no effect when - /// combined with other predicates under the same operator): + /// - Empty junction returns the identity element (the value that has no effect when combined + /// with other predicates under the same operator): /// - `AND()` -> `true`, because `true AND p` == `p` for any predicate `p`. /// - `OR()` -> `false`, because `false OR p` == `p` for any predicate `p`. /// - Single-element junction unwraps the element: `AND(p)` / `OR(p)` -> `p`. @@ -1000,6 +1193,7 @@ impl Display for VariadicExpressionOp { use VariadicExpressionOp::*; match self { Coalesce => write!(f, "COALESCE"), + Array => write!(f, "ARRAY"), } } } @@ -1033,31 +1227,33 @@ impl Display for Expression { Column(name) => write!(f, "Column({name})"), Predicate(p) => write!(f, "{p}"), Struct(exprs, _) => write!(f, "Struct({})", format_child_list(exprs)), - Transform(transform) => { - write!(f, "Transform(")?; + StructPatch(patch) => { + write!(f, "StructPatch(")?; let mut sep = ""; - if !transform.prepended_fields.is_empty() { - let prepended_fields = format_child_list(&transform.prepended_fields); + if !patch.prepended_fields.is_empty() { + let prepended_fields = format_child_list(&patch.prepended_fields); write!(f, "prepend [{prepended_fields}]")?; sep = ", "; } - for (field_name, field_transform) in &transform.field_transforms { - let insertions = &field_transform.exprs; - if insertions.is_empty() { - if field_transform.is_replace { - write!(f, "{sep}drop {field_name}")?; - } else { - continue; // no-op; ignore it and don't change `sep` below - } - } else { - let insertions = format_child_list(insertions); - if field_transform.is_replace { - write!(f, "{sep}replace {field_name} with [{insertions}]")?; + for (field_name, field_patch) in &patch.field_patches { + if !field_patch.keep_input && field_patch.insertions.is_empty() { + write!(f, "{sep}drop {field_name}")?; + sep = ", "; + } + if !field_patch.insertions.is_empty() { + let insertions = format_child_list(&field_patch.insertions); + let action = if field_patch.keep_input { + "after" } else { - write!(f, "{sep}after {field_name} insert [{insertions}]")?; - } + "replace/after" + }; + write!(f, "{sep}{action} {field_name} insert [{insertions}]")?; + sep = ", "; } - sep = ", "; + } + if !patch.appended_fields.is_empty() { + let appended_fields = format_child_list(&patch.appended_fields); + write!(f, "{sep}append [{appended_fields}]")?; } write!(f, ")") } @@ -1078,7 +1274,21 @@ impl Display for Expression { p.output_schema.fields().len() ) } - MapToStruct(m) => write!(f, "MAP_TO_STRUCT({})", m.map_expr), + MapToStruct(m) => match m.options.timestamp_timezone() { + Some(timezone) => write!( + f, + "MAP_TO_STRUCT({}, timestamp_timezone=\"{}\")", + m.map_expr, timezone + ), + None => write!(f, "MAP_TO_STRUCT({})", m.map_expr), + }, + ElementAt(e) => write!(f, "ELEMENT_AT({}, {})", e.map_expr, e.key_expr), + Cast(c) => match c.options.timestamp_timezone() { + Some(timezone) => { + write!(f, "CAST({} AS {} USING \"{}\")", c.expr, c.target, timezone) + } + None => write!(f, "CAST({} AS {})", c.expr, c.target), + }, } } } @@ -1174,9 +1384,10 @@ impl> std::ops::Div for Expression { struct GetColumnReferences<'a>(HashSet<&'a ColumnName>); impl<'a> ExpressionTransform<'a> for GetColumnReferences<'a> { - fn transform_expr_column(&mut self, name: &'a ColumnName) -> Option> { + transform_output_type!(|'a, T| ()); + + fn transform_expr_column(&mut self, name: &'a ColumnName) { self.0.insert(name); - Some(Cow::Borrowed(name)) } } @@ -1187,7 +1398,10 @@ mod tests { use serde::de::DeserializeOwned; use serde::Serialize; - use super::{column_expr, column_pred, Expression as Expr, Predicate as Pred}; + use super::{ + col, column_pred, lit, CastOptions, DataType, Expression as Expr, MapToStructOptions, + Predicate as Pred, + }; /// Helper function to verify roundtrip serialization/deserialization fn assert_roundtrip(value: &T) { @@ -1199,15 +1413,23 @@ mod tests { #[test] fn test_expression_format() { let cases = [ - (column_expr!("x"), "Column(x)"), + (col!("x"), "Column(x)"), ( - (column_expr!("x") + Expr::literal(4)) / Expr::literal(10) * Expr::literal(42), + (col!("x") + lit(4)) / lit(10) * lit(42), "Column(x) + 4 / 10 * 42", ), ( - Expr::struct_from([column_expr!("x"), Expr::literal(2), Expr::literal(10)]), + Expr::struct_from([col!("x"), lit(2), lit(10)]), "Struct(Column(x), 2, 10)", ), + ( + Expr::array([col!("x"), col!("y"), lit(0)]), + "ARRAY(Column(x), Column(y), 0)", + ), + ( + Expr::cast(col!("x"), DataType::DATE, CastOptions::default()), + "CAST(Column(x) AS date)", + ), ]; for (expr, expected) in cases { @@ -1216,41 +1438,79 @@ mod tests { } } + #[test] + fn test_map_to_struct_options_state_and_format() { + let default = Expr::map_to_struct(col!("m"), MapToStructOptions::default()); + let Expr::MapToStruct(default_state) = &default else { + panic!("expected map-to-struct expression"); + }; + assert_eq!(default_state.options.timestamp_timezone(), None); + assert_eq!(format!("{default}"), "MAP_TO_STRUCT(Column(m))"); + + let configured = Expr::map_to_struct( + col!("m"), + MapToStructOptions::default().with_timestamp_timezone("America/Los_Angeles"), + ); + let Expr::MapToStruct(configured_state) = &configured else { + panic!("expected map-to-struct expression"); + }; + assert_eq!( + configured_state.options.timestamp_timezone(), + Some("America/Los_Angeles") + ); + assert_eq!( + format!("{configured}"), + "MAP_TO_STRUCT(Column(m), timestamp_timezone=\"America/Los_Angeles\")" + ); + } + + #[test] + fn test_cast_options_state_and_format() { + let cast_default = Expr::cast(col!("s"), DataType::TIMESTAMP, CastOptions::default()); + let Expr::Cast(cast_default_state) = &cast_default else { + panic!("expected cast expression"); + }; + assert_eq!(cast_default_state.options.timestamp_timezone(), None); + assert_eq!(format!("{cast_default}"), "CAST(Column(s) AS timestamp)"); + + let cast = Expr::cast( + col!("s"), + DataType::TIMESTAMP, + CastOptions::default().with_timestamp_timezone("+12:45:30"), + ); + let Expr::Cast(cast_state) = &cast else { + panic!("expected cast expression"); + }; + assert_eq!(cast_state.options.timestamp_timezone(), Some("+12:45:30")); + assert_eq!( + format!("{cast}"), + "CAST(Column(s) AS timestamp USING \"+12:45:30\")" + ); + } + #[test] fn test_predicate_format() { let cases = [ (column_pred!("x"), "Column(x)"), - (column_expr!("x").eq(Expr::literal(2)), "Column(x) = 2"), - ( - (column_expr!("x") - Expr::literal(4)).lt(Expr::literal(10)), - "Column(x) - 4 < 10", - ), + (col!("x").eq(lit(2)), "Column(x) = 2"), + ((col!("x") - lit(4)).lt(lit(10)), "Column(x) - 4 < 10"), ( - Pred::and( - column_expr!("x").ge(Expr::literal(2)), - column_expr!("x").le(Expr::literal(10)), - ), + Pred::and(col!("x").ge(lit(2)), col!("x").le(lit(10))), "AND(NOT(Column(x) < 2), NOT(Column(x) > 10))", ), ( Pred::and_from([ - column_expr!("x").ge(Expr::literal(2)), - column_expr!("x").le(Expr::literal(10)), - column_expr!("x").le(Expr::literal(100)), + col!("x").ge(lit(2)), + col!("x").le(lit(10)), + col!("x").le(lit(100)), ]), "AND(NOT(Column(x) < 2), NOT(Column(x) > 10), NOT(Column(x) > 100))", ), ( - Pred::or( - column_expr!("x").gt(Expr::literal(2)), - column_expr!("x").lt(Expr::literal(10)), - ), + Pred::or(col!("x").gt(lit(2)), col!("x").lt(lit(10))), "OR(Column(x) > 2, Column(x) < 10)", ), - ( - column_expr!("x").eq(Expr::literal("foo")), - "Column(x) = 'foo'", - ), + (col!("x").eq(lit("foo")), "Column(x) = 'foo'"), ]; for (pred, expected) in cases { @@ -1264,15 +1524,15 @@ mod tests { mod serde_tests { use std::sync::Arc; + use super::assert_roundtrip; use crate::expressions::scalars::{ArrayData, DecimalData, MapData, StructData}; use crate::expressions::{ - column_expr, column_name, BinaryExpressionOp, BinaryPredicateOp, ColumnName, - Expression, Predicate, Scalar, Transform, UnaryExpressionOp, + col, column_name, lit, BinaryExpressionOp, BinaryPredicateOp, CastOptions, ColumnName, + Expression, ExpressionStructPatchBuilder, MapToStructOptions, Predicate, Scalar, + UnaryExpressionOp, }; use crate::schema::{ArrayType, DataType, DecimalType, MapType, StructField}; - use crate::utils::test_utils::assert_result_error_with_message; - - use super::assert_roundtrip; + use crate::unit_test_utils::assert_result_error_with_message; // ==================== Expression::Literal Tests ==================== @@ -1357,11 +1617,8 @@ mod tests { #[test] fn test_column_expressions_roundtrip() { - let cases: Vec = vec![ - column_expr!("my_column"), - Expression::column(["parent", "child"]), - Expression::column(["a", "b", "c", "d"]), - ]; + let cases: Vec = + vec![col!("my_column"), col!("parent.child"), col!("a.b.c.d")]; for expr in &cases { assert_roundtrip(expr); @@ -1372,8 +1629,8 @@ mod tests { fn test_column_names_roundtrip() { let cases: Vec = vec![ column_name!("simple"), - ColumnName::new(["a", "b", "c"]), - ColumnName::new::<&str>([]), + column_name!("a.b.c"), + ColumnName::default(), ]; for col in &cases { @@ -1385,7 +1642,7 @@ mod tests { #[test] fn test_unary_expression_roundtrip() { - let expr = Expression::unary(UnaryExpressionOp::ToJson, column_expr!("data")); + let expr = Expression::unary(UnaryExpressionOp::ToJson, col!("data")); assert_roundtrip(&expr); } @@ -1399,47 +1656,71 @@ mod tests { ]; for op in ops { - let expr = Expression::binary(op, column_expr!("a"), Expression::literal(10)); + let expr = Expression::binary(op, col!("a"), lit(10)); assert_roundtrip(&expr); } } #[test] fn test_variadic_expression_roundtrip() { - let expr = Expression::coalesce([ - column_expr!("a"), - column_expr!("b"), - Expression::literal("default"), - ]); + let expr = Expression::coalesce([col!("a"), col!("b"), lit("default")]); + assert_roundtrip(&expr); + } + + #[rstest::rstest] + #[case::column(Expression::cast( + col!("part"), + DataType::DATE, + CastOptions::default(), + ))] + #[case::literal(Expression::cast( + lit("2025-01-01"), + DataType::DATE, + CastOptions::default(), + ))] + #[case::nested(Expression::cast( + Expression::cast(col!("part"), DataType::STRING, CastOptions::default()), + DataType::INTEGER, + CastOptions::default(), + ))] + #[case::timestamp_timezone(Expression::cast( + col!("part"), + DataType::TIMESTAMP, + CastOptions::default().with_timestamp_timezone("America/Los_Angeles"), + ))] + fn test_cast_expression_roundtrip(#[case] expr: Expression) { + assert_roundtrip(&expr); + } + + #[rstest::rstest] + #[case::array_single(Expression::array([lit(7i32)]))] + #[case::array_mixed(Expression::array([ + col!("a"), + col!("b"), + lit(42i64), + ]))] + fn test_array_expression_roundtrip(#[case] expr: Expression) { assert_roundtrip(&expr); } #[test] fn test_nested_arithmetic_expression_roundtrip() { // (a + b) * (c - d) / 2 - let left = Expression::binary( - BinaryExpressionOp::Plus, - column_expr!("a"), - column_expr!("b"), - ); - let right = Expression::binary( - BinaryExpressionOp::Minus, - column_expr!("c"), - column_expr!("d"), - ); + let left = Expression::binary(BinaryExpressionOp::Plus, col!("a"), col!("b")); + let right = Expression::binary(BinaryExpressionOp::Minus, col!("c"), col!("d")); let mul = Expression::binary(BinaryExpressionOp::Multiply, left, right); - let expr = Expression::binary(BinaryExpressionOp::Divide, mul, Expression::literal(2)); + let expr = Expression::binary(BinaryExpressionOp::Divide, mul, lit(2)); assert_roundtrip(&expr); } - // ==================== Expression::Struct/Transform/Other Tests ==================== + // ==================== Expression::Struct/StructPatch/Other Tests ==================== #[test] fn test_struct_expression_roundtrip() { let expr = Expression::struct_from([ - Arc::new(column_expr!("x")), - Arc::new(Expression::literal(42)), - Arc::new(Expression::literal("hello")), + Arc::new(col!("x")), + Arc::new(lit(42)), + Arc::new(lit("hello")), ]); assert_roundtrip(&expr); } @@ -1448,27 +1729,28 @@ mod tests { fn test_transform_expressions_roundtrip() { let cases: Vec = vec![ // Identity transform - Expression::transform(Transform::new_top_level()), + Expression::struct_patch(ExpressionStructPatchBuilder::new()).unwrap(), // Drop field - Expression::transform(Transform::new_top_level().with_dropped_field("old_column")), + Expression::struct_patch(ExpressionStructPatchBuilder::new().drop("old_column")) + .unwrap(), // Replace field - Expression::transform( - Transform::new_top_level() - .with_replaced_field("original", Arc::new(Expression::literal(0))), - ), + Expression::struct_patch( + ExpressionStructPatchBuilder::new().replace("original", lit(0)), + ) + .unwrap(), // Insert fields - Expression::transform( - Transform::new_top_level() - .with_inserted_field(Some("after_col"), Arc::new(column_expr!("new_col"))) - .with_inserted_field( - None::, - Arc::new(Expression::literal("prepended")), - ), - ), + Expression::struct_patch( + ExpressionStructPatchBuilder::new() + .insert_after("after_col", col!("new_col")) + .prepend(lit("prepended")) + .append(lit("appended")), + ) + .unwrap(), // Nested transform - Expression::transform( - Transform::new_nested(["parent", "child"]).with_dropped_field("to_drop"), - ), + Expression::struct_patch( + ExpressionStructPatchBuilder::new_nested(["parent", "child"]).drop("to_drop"), + ) + .unwrap(), ]; for expr in &cases { @@ -1478,7 +1760,7 @@ mod tests { #[test] fn test_expression_wrapping_predicate_roundtrip() { - let pred = Predicate::eq(column_expr!("x"), Expression::literal(10)); + let pred = Predicate::eq(col!("x"), lit(10)); let expr = Expression::from_pred(pred); assert_roundtrip(&expr); } @@ -1492,8 +1774,12 @@ mod tests { #[test] fn test_map_to_struct_expression_roundtrip() { let cases: Vec = vec![ - Expression::map_to_struct(column_expr!("pv")), - Expression::map_to_struct(Expression::literal("ignored")), + Expression::map_to_struct(col!("pv"), MapToStructOptions::default()), + Expression::map_to_struct(lit("ignored"), MapToStructOptions::default()), + Expression::map_to_struct( + col!("pv"), + MapToStructOptions::default().with_timestamp_timezone("America/Los_Angeles"), + ), ]; for expr in &cases { @@ -1501,28 +1787,31 @@ mod tests { } } + #[test] + fn test_element_at_expression_roundtrip() { + let expr = Expression::element_at(col!("partitionValues"), col!("partitionKey")); + assert_roundtrip(&expr); + } + // ==================== Predicate Tests ==================== #[test] fn test_predicate_basics_roundtrip() { let cases: Vec = vec![ // Boolean expression - Predicate::from_expr(column_expr!("is_active")), + Predicate::from_expr(col!("is_active")), // Literals - Predicate::literal(true), - Predicate::literal(false), + Predicate::TRUE, + Predicate::FALSE, // NOT - Predicate::not(Predicate::from_expr(column_expr!("x"))), + Predicate::not(Predicate::from_expr(col!("x"))), // Nested NOT - Predicate::not(Predicate::not(Predicate::gt( - column_expr!("x"), - Expression::literal(5), - ))), + Predicate::not(Predicate::not(Predicate::gt(col!("x"), lit(5)))), // Unknown Predicate::unknown("some_unknown_predicate()"), // Unary predicates - Predicate::is_null(column_expr!("nullable_col")), - Predicate::is_not_null(column_expr!("nullable_col")), + Predicate::is_null(col!("nullable_col")), + Predicate::is_not_null(col!("nullable_col")), ]; for pred in &cases { @@ -1532,20 +1821,19 @@ mod tests { #[test] fn test_predicate_null_literal_roundtrip() { - let pred = Predicate::null_literal(); - assert_roundtrip(&pred); + assert_roundtrip(&Predicate::NULL); } #[test] fn test_predicate_comparisons_roundtrip() { let cases: Vec = vec![ - Predicate::eq(column_expr!("x"), Expression::literal(42)), - Predicate::ne(column_expr!("status"), Expression::literal("active")), - Predicate::lt(column_expr!("age"), Expression::literal(18)), - Predicate::le(column_expr!("price"), Expression::literal(100)), - Predicate::gt(column_expr!("score"), Expression::literal(90)), - Predicate::ge(column_expr!("quantity"), Expression::literal(1)), - Predicate::distinct(column_expr!("a"), column_expr!("b")), + Predicate::eq(col!("x"), lit(42)), + Predicate::ne(col!("status"), lit("active")), + Predicate::lt(col!("age"), lit(18)), + Predicate::le(col!("price"), lit(100)), + Predicate::gt(col!("score"), lit(90)), + Predicate::ge(col!("quantity"), lit(1)), + Predicate::distinct(col!("a"), col!("b")), ]; for pred in &cases { @@ -1562,7 +1850,7 @@ mod tests { .unwrap(); let pred = Predicate::binary( BinaryPredicateOp::In, - column_expr!("x"), + col!("x"), Expression::Literal(Scalar::Array(array_data)), ); assert_roundtrip(&pred); @@ -1573,33 +1861,33 @@ mod tests { let cases: Vec = vec![ // Simple AND Predicate::and( - Predicate::gt(column_expr!("x"), Expression::literal(0)), - Predicate::lt(column_expr!("x"), Expression::literal(100)), + Predicate::gt(col!("x"), lit(0)), + Predicate::lt(col!("x"), lit(100)), ), // Simple OR Predicate::or( - Predicate::eq(column_expr!("status"), Expression::literal("active")), - Predicate::eq(column_expr!("status"), Expression::literal("pending")), + Predicate::eq(col!("status"), lit("active")), + Predicate::eq(col!("status"), lit("pending")), ), // Multiple AND Predicate::and_from([ - Predicate::gt(column_expr!("x"), Expression::literal(0)), - Predicate::lt(column_expr!("x"), Expression::literal(100)), - Predicate::is_not_null(column_expr!("x")), + Predicate::gt(col!("x"), lit(0)), + Predicate::lt(col!("x"), lit(100)), + Predicate::is_not_null(col!("x")), ]), // Multiple OR Predicate::or_from([ - Predicate::eq(column_expr!("type"), Expression::literal("A")), - Predicate::eq(column_expr!("type"), Expression::literal("B")), - Predicate::eq(column_expr!("type"), Expression::literal("C")), + Predicate::eq(col!("type"), lit("A")), + Predicate::eq(col!("type"), lit("B")), + Predicate::eq(col!("type"), lit("C")), ]), // Nested: (a > 0 AND b < 100) OR (c = 'special') Predicate::or( Predicate::and( - Predicate::gt(column_expr!("a"), Expression::literal(0)), - Predicate::lt(column_expr!("b"), Expression::literal(100)), + Predicate::gt(col!("a"), lit(0)), + Predicate::lt(col!("b"), lit(100)), ), - Predicate::eq(column_expr!("c"), Expression::literal("special")), + Predicate::eq(col!("c"), lit("special")), ), ]; @@ -1613,30 +1901,18 @@ mod tests { #[test] fn test_deeply_nested_structures_roundtrip() { // COALESCE(a + b, c * d, 0) > 100 - let add = Expression::binary( - BinaryExpressionOp::Plus, - column_expr!("a"), - column_expr!("b"), - ); - let mul = Expression::binary( - BinaryExpressionOp::Multiply, - column_expr!("c"), - column_expr!("d"), - ); - let coalesce = Expression::coalesce([add, mul, Expression::literal(0)]); - let pred = Predicate::gt(coalesce, Expression::literal(100)); + let add = Expression::binary(BinaryExpressionOp::Plus, col!("a"), col!("b")); + let mul = Expression::binary(BinaryExpressionOp::Multiply, col!("c"), col!("d")); + let coalesce = Expression::coalesce([add, mul, lit(0)]); + let pred = Predicate::gt(coalesce, lit(100)); assert_roundtrip(&pred); // Expression wrapping a predicate that references expressions let inner_pred = Predicate::and( - Predicate::eq(column_expr!("x"), Expression::literal(1)), + Predicate::eq(col!("x"), lit(1)), Predicate::gt( - Expression::binary( - BinaryExpressionOp::Plus, - column_expr!("y"), - column_expr!("z"), - ), - Expression::literal(10), + Expression::binary(BinaryExpressionOp::Plus, col!("y"), col!("z")), + lit(10), ), ); let expr = Expression::from_pred(inner_pred); @@ -1722,22 +1998,22 @@ mod tests { #[test] fn single_element_and_from_returns_unwrapped_predicate() { - let inner = Pred::gt(column_expr!("x"), Expr::literal(0)); + let inner = Pred::gt(col!("x"), lit(0)); let result = Pred::and_from([inner.clone()]); assert_eq!(result, inner); } #[test] fn single_element_or_from_returns_unwrapped_predicate() { - let inner = Pred::gt(column_expr!("x"), Expr::literal(0)); + let inner = Pred::gt(col!("x"), lit(0)); let result = Pred::or_from([inner.clone()]); assert_eq!(result, inner); } #[test] fn multi_element_and_from_returns_junction() { - let p1 = Pred::gt(column_expr!("x"), Expr::literal(0)); - let p2 = Pred::lt(column_expr!("x"), Expr::literal(100)); + let p1 = Pred::gt(col!("x"), lit(0)); + let p2 = Pred::lt(col!("x"), lit(100)); let result = Pred::and_from([p1.clone(), p2.clone()]); assert!(matches!(result, Pred::Junction(ref j) if j.preds.len() == 2)); assert_eq!(result, Pred::and(p1, p2)); @@ -1746,12 +2022,12 @@ mod tests { #[test] fn empty_and_from_returns_identity_literal() { let result = Pred::and_from(std::iter::empty()); - assert_eq!(result, Pred::literal(true)); + assert_eq!(result, Pred::TRUE); } #[test] fn empty_or_from_returns_identity_literal() { let result = Pred::or_from(std::iter::empty()); - assert_eq!(result, Pred::literal(false)); + assert_eq!(result, Pred::FALSE); } } diff --git a/kernel/src/expressions/scalars.rs b/kernel/src/expressions/scalars.rs index 993566dce6..711a44b0b6 100644 --- a/kernel/src/expressions/scalars.rs +++ b/kernel/src/expressions/scalars.rs @@ -1,16 +1,32 @@ use std::cmp::Ordering; use std::collections::HashMap; use std::fmt::{Display, Formatter}; +use std::hash::Hash; use chrono::{DateTime, NaiveDate, NaiveDateTime, TimeZone, Utc}; +use delta_kernel_derive::internal_api; use itertools::Itertools; use serde::{Deserialize, Serialize}; - -use crate::schema::derive_macro_utils::ToDataType; -use crate::schema::{ArrayType, DataType, DecimalType, MapType, PrimitiveType, StructField}; +use strum::AsRefStr; + +use crate::error::add_scalar_path_context; +use crate::schema::derive_macro_utils::{GetStructField, ToDataType}; +use crate::schema::{ + parse_interval_type, ArrayType, DataType, DecimalType, IntervalField, IntervalFieldRange, + MapType, PrimitiveType, StructField, StructType, +}; use crate::utils::require; use crate::{DeltaResult, Error}; +/// Pairs [`Into`] with [`ToDataType`] for infallible container conversions. +/// +/// `Option` is not itself `IntoScalar`: nullability belongs to the owning +/// struct/array/map rather than its value type. See [`ToDataType`] for the pairing contract. +#[internal_api] +pub(crate) trait IntoScalar: Into + ToDataType {} + +impl + ToDataType> IntoScalar for T {} + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] pub struct DecimalData { bits: i128, @@ -98,6 +114,39 @@ impl ArrayData { pub fn array_elements(&self) -> &[Scalar] { &self.elements } + + /// Consume this array and return its elements. + #[internal_api] + pub(crate) fn into_elements(self) -> Vec { + self.elements + } + + /// Infallible constructor used by `From>` and `From>` where we know the + /// resulting `Scalar::data_type` is `T:to_data_type`. + fn from_elements( + elements: impl IntoIterator>, + contains_null: bool, + ) -> Self { + Self { + tpe: ArrayType::new(T::to_data_type(), contains_null), + elements: elements.into_iter().map(Into::into).collect(), + } + } +} + +/// Builds [`ArrayData`] from a Rust `Vec` whose element type implements both `Into` and +/// [`ToSchema`](delta_kernel::schema::ToSchema). +impl From> for ArrayData { + fn from(vec: Vec) -> Self { + Self::from_elements::(vec, false) + } +} + +/// Like [`From> for ArrayData`], but elements may be null (`contains_null = true`). +impl From>> for ArrayData { + fn from(vec: Vec>) -> Self { + Self::from_elements::(vec, true) + } } #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] @@ -156,6 +205,39 @@ impl MapData { pub fn map_type(&self) -> &MapType { &self.data_type } + + /// Consume this map and return its key/value pairs. + #[internal_api] + pub(crate) fn into_pairs(self) -> Vec<(Scalar, Scalar)> { + self.pairs + } + + /// Infallible constructor used by `From>` / `From>>` + /// where key/value scalars are known to match `K`/`V`'s [`ToDataType`]. + fn from_pairs( + pairs: impl IntoIterator, impl Into)>, + value_contains_null: bool, + ) -> Self { + Self { + data_type: MapType::new(K::to_data_type(), V::to_data_type(), value_contains_null), + pairs: Vec::from_iter(pairs.into_iter().map(|(k, v)| (k.into(), v.into()))), + } + } +} + +/// Builds [`MapData`] from a Rust [`HashMap`] whose key/value types implement both `Into` +/// and [`ToSchema`](delta_kernel::schema::ToSchema). +impl From> for MapData { + fn from(map: HashMap) -> Self { + Self::from_pairs::(map, false) + } +} + +/// Like [`From> for MapData`], but values may be null (`value_contains_null = true`). +impl From>> for MapData { + fn from(map: HashMap>) -> Self { + Self::from_pairs::(map, true) + } } #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] @@ -205,6 +287,17 @@ impl StructData { Ok(Self { fields, values }) } + /// Infallible constructor used by `From` where `schema` and `values` are produced using + /// [`ToSchema`](delta_kernel::schema::ToSchema) and `Into`. Does not re-validate types + /// or lengths. + #[internal_api] + pub(crate) fn from_values_unchecked(schema: StructType, values: Vec) -> Self { + Self { + fields: schema.into_fields().collect(), + values, + } + } + pub fn fields(&self) -> &[StructField] { &self.fields } @@ -212,6 +305,12 @@ impl StructData { pub fn values(&self) -> &[Scalar] { &self.values } + + /// Consume this struct and return its fields and values. + #[internal_api] + pub(crate) fn into_parts(self) -> (Vec, Vec) { + (self.fields, self.values) + } } /// A single value, which can be null. Used for representing literal values @@ -219,7 +318,8 @@ impl StructData { /// /// NOTE: `PartialEq` uses physical (structural) comparison semantics. /// For SQL NULL semantics, use [`Scalar::logical_eq`] or [`Scalar::logical_partial_cmp`]. -#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, AsRefStr)] +#[strum(serialize_all = "snake_case")] pub enum Scalar { /// 32bit integer Integer(i32), @@ -241,6 +341,10 @@ pub enum Scalar { Timestamp(i64), /// Microsecond precision timestamp, with no timezone. TimestampNtz(i64), + /// Year-month interval, stored as a signed 32bit count of months (matches Spark Catalyst). + IntervalYearMonth(i32), + /// Day-time interval, stored as a signed 64bit count of microseconds (matches Spark Catalyst). + IntervalDayTime(i64), /// Date stored as a signed 32bit int days since UNIX epoch 1970-01-01 Date(i32), /// Binary data @@ -270,6 +374,8 @@ impl Scalar { Self::Boolean(_) => DataType::BOOLEAN, Self::Timestamp(_) => DataType::TIMESTAMP, Self::TimestampNtz(_) => DataType::TIMESTAMP_NTZ, + Self::IntervalYearMonth(_) => DataType::INTERVAL_YEAR_MONTH, + Self::IntervalDayTime(_) => DataType::INTERVAL_DAY_TIME, Self::Date(_) => DataType::DATE, Self::Binary(_) => DataType::BINARY, Self::Decimal(d) => DataType::from(*d.ty()), @@ -285,6 +391,17 @@ impl Scalar { matches!(self, Self::Null(_)) } + /// Constructs a null `Scalar` of the given type. Accepts anything convertible into a + /// [`DataType`], so container types like [`StructType`], [`ArrayType`], and [`MapType`] + /// can be passed directly without an explicit `DataType::from(...)` wrapper. + /// + /// [`StructType`]: crate::schema::StructType + /// [`ArrayType`]: crate::schema::ArrayType + /// [`MapType`]: crate::schema::MapType + pub fn null(data_type: impl Into) -> Self { + Self::Null(data_type.into()) + } + /// Constructs a Decimal value from raw parts pub fn decimal(bits: impl Into, precision: u8, scale: u8) -> DeltaResult { let dtype = DecimalType::try_new(precision, scale)?; @@ -292,6 +409,12 @@ impl Scalar { Ok(Self::Decimal(dval)) } + /// Error for a failed conversion of this scalar into the Rust type named by `target`. + #[internal_api] + pub(crate) fn conversion_error(&self, target: &str) -> Error { + Error::scalar_conversion(target, self.as_ref()) + } + /// Constructs a Scalar timestamp (in UTC) from an `i64` millisecond since unix epoch pub(crate) fn timestamp_from_millis(millis: i64) -> DeltaResult { let Some(timestamp) = DateTime::from_timestamp_millis(millis) else { @@ -341,7 +464,9 @@ impl Scalar { Some(result) } - /// Attempts to divide two scalars, returning None if they were incompatible. + /// Attempts to divide two scalars, truncating toward zero. Returns None if they were + /// incompatible, if `other` is zero, or on overflow. Only the integral types divide; a float or + /// decimal operand is treated as incompatible. pub fn try_div(&self, other: &Scalar) -> Option { use Scalar::*; let result = match (self, other) { @@ -368,6 +493,8 @@ impl Display for Scalar { Self::Boolean(b) => write!(f, "{b}"), Self::Timestamp(ts) => write!(f, "{ts}"), Self::TimestampNtz(ts) => write!(f, "{ts}"), + Self::IntervalYearMonth(months) => write!(f, "{months}"), + Self::IntervalDayTime(micros) => write!(f, "{micros}"), Self::Date(d) => write!(f, "{d}"), Self::Binary(b) => write!(f, "{b:?}"), Self::Decimal(d) => match d.scale().cmp(&0) { @@ -485,6 +612,10 @@ impl Scalar { (Timestamp(_), _) => None, (TimestampNtz(a), TimestampNtz(b)) => a.partial_cmp(b), (TimestampNtz(_), _) => None, + (IntervalYearMonth(a), IntervalYearMonth(b)) => a.partial_cmp(b), + (IntervalYearMonth(_), _) => None, + (IntervalDayTime(a), IntervalDayTime(b)) => a.partial_cmp(b), + (IntervalDayTime(_), _) => None, (Date(a), Date(b)) => a.partial_cmp(b), (Date(_), _) => None, (Binary(a), Binary(b)) => a.partial_cmp(b), @@ -493,10 +624,11 @@ impl Scalar { .then(|| d1.bits().partial_cmp(&d2.bits())) .flatten(), (Decimal(_), _) => None, - (Null(_), _) => None, // NOTE: NULL values are incomparable by definition (SQL NULL semantics) + // NOTE: NULL values are incomparable by definition (SQL NULL semantics) + (Null(_), _) => None, (Struct(_), _) => None, // TODO: Support Struct? - (Array(_), _) => None, // TODO: Support Array? - (Map(_), _) => None, // TODO: Support Map? + (Array(_), _) => None, // TODO: Support Array? + (Map(_), _) => None, // TODO: Support Map? } } } @@ -579,120 +711,226 @@ impl From for Scalar { } } -impl TryFrom> for Scalar +impl From> for Scalar where - T: Into + ToDataType, + Vec: Into, { - type Error = Error; - - fn try_from(vec: Vec) -> Result { - let array_type = ArrayType::new(T::to_data_type(), false); - let array_data = ArrayData::try_new(array_type, vec)?; - Ok(array_data.into()) + fn from(vec: Vec) -> Self { + Self::Array(vec.into()) } } -impl TryFrom>> for Scalar +impl From> for Scalar where - T: Into + ToDataType, + HashMap: Into, { - type Error = Error; - - fn try_from(vec: Vec>) -> Result { - let array_type = ArrayType::new(T::to_data_type(), true); - let array_data = ArrayData::try_new(array_type, vec)?; - Ok(array_data.into()) + fn from(map: HashMap) -> Self { + Self::Map(map.into()) } } -impl TryFrom>> for Scalar -where - T: Into + ToDataType, -{ - type Error = Error; - - fn try_from(opt: Option>) -> Result { - match opt { - Some(vec) => vec.try_into(), - None => Ok(Self::Null(ArrayType::new(T::to_data_type(), false).into())), +// NOTE: We "cheat" and use the macro support trait `ToDataType` +impl From> for Scalar { + fn from(t: Option) -> Self { + match t { + Some(t) => t.into(), + None => Self::Null(T::to_data_type()), } } } -impl TryFrom> for Scalar -where - K: Into + ToDataType, - V: Into + ToDataType, -{ - type Error = Error; - - fn try_from(map: HashMap) -> Result { - let map_type = MapType::new(K::to_data_type(), V::to_data_type(), false); - let map_data = MapData::try_new(map_type, map)?; - Ok(map_data.into()) +impl From for Scalar { + fn from(array_data: ArrayData) -> Self { + Self::Array(array_data) } } -impl TryFrom>> for Scalar -where - K: Into + ToDataType, - V: Into + ToDataType, -{ - type Error = Error; +impl From for Scalar { + fn from(map_data: MapData) -> Self { + Self::Map(map_data) + } +} - fn try_from(map: HashMap>) -> Result { - let map_type = MapType::new(K::to_data_type(), V::to_data_type(), true); - let map_data = MapData::try_new(map_type, map)?; - Ok(map_data.into()) +impl From for Scalar { + fn from(struct_data: StructData) -> Self { + Self::Struct(struct_data) } } -impl TryFrom>> for Scalar -where - K: Into + ToDataType, - V: Into + ToDataType, -{ - type Error = Error; +// ===== Scalar -> rust conversions, inverting the `From for Scalar` impls above ===== - fn try_from(opt: Option>) -> Result { - match opt { - Some(map) => map.try_into(), - None => Ok(Self::Null( - MapType::new(K::to_data_type(), V::to_data_type(), false).into(), - )), - } - } +/// Inverts the corresponding `From for Scalar` conversion. The generated `try_from` matches on +/// the single `Scalar` variant its forward counterpart produces and returns a conversion error for +/// every other variant. +/// +/// Matching is by variant, not by physical representation, so a target type is never produced from +/// a variant that merely shares its in-memory layout: `i64` accepts `Long` but rejects `Timestamp`, +/// `TimestampNtz`, and `IntervalDayTime`; `i32` accepts `Integer` but rejects `Date` and +/// `IntervalYearMonth`. This keeps `Scalar` <-> rust conversions type-preserving, so a value's data +/// type survives the round trip rather than collapsing onto whichever variant shares its layout. +macro_rules! impl_try_from_scalar { + ( $(($variant:ident, $rust_type:ty)),* $(,)? ) => { + $( + impl TryFrom for $rust_type { + type Error = Error; + + fn try_from(scalar: Scalar) -> DeltaResult { + match scalar { + Scalar::$variant(value) => Ok(value.into()), + other => Err(other.conversion_error(stringify!($rust_type))), + } + } + } + )* + }; } -// NOTE: We "cheat" and use the macro support trait `ToDataType` -impl + ToDataType> From> for Scalar { - fn from(t: Option) -> Self { - match t { - Some(t) => t.into(), - None => Self::Null(T::to_data_type()), +impl_try_from_scalar!( + (Byte, i8), + (Short, i16), + (Integer, i32), + (Long, i64), + (Float, f32), + (Double, f64), + (Boolean, bool), + (String, String), + (Binary, bytes::Bytes), + (Decimal, DecimalData), + (Array, ArrayData), + (Map, MapData), + (Struct, StructData), +); + +/// Null becomes `None` when its typed null matches `T::to_data_type`; anything else must convert +/// to `T`. +impl + ToDataType> TryFrom for Option { + type Error = Error; + + fn try_from(scalar: Scalar) -> DeltaResult { + match scalar { + Scalar::Null(data_type) => { + let expected = T::to_data_type(); + require!( + data_type == expected, + Error::scalar_conversion(expected.kind_name(), data_type.kind_name()) + ); + Ok(None) + } + other => Ok(Some(T::try_from(other)?)), } } } -impl From for Scalar { - fn from(array_data: ArrayData) -> Self { - Self::Array(array_data) +/// Extracts an array scalar's elements. Use `Vec>` for arrays that contain nulls. +impl TryFrom for Vec +where + T: GetStructField + TryFrom, +{ + type Error = Error; + + fn try_from(scalar: Scalar) -> DeltaResult { + let array: ArrayData = scalar.try_into()?; + let element = T::get_struct_field("element"); + let expected = ArrayType::new(element.data_type().clone(), element.is_nullable()); + require!( + array.array_type() == &expected, + Error::scalar_conversion( + format!( + "array<{}, contains_null={}>", + expected.element_type().kind_name(), + expected.contains_null() + ), + format!( + "array<{}, contains_null={}>", + array.array_type().element_type().kind_name(), + array.array_type().contains_null() + ), + ) + ); + array + .into_elements() + .into_iter() + .enumerate() + .map(|(index, value)| { + T::try_from(value) + .map_err(|error| add_scalar_path_context(error, format!("[{index}]"))) + }) + .try_collect() } } -impl From for Scalar { - fn from(map_data: MapData) -> Self { - Self::Map(map_data) +/// Extracts a map scalar's entries. Use `HashMap>` for maps with null values. +impl TryFrom for HashMap +where + K: TryFrom + Eq + Hash, + V: GetStructField + TryFrom, + K: ToDataType, +{ + type Error = Error; + + fn try_from(scalar: Scalar) -> DeltaResult { + let map: MapData = scalar.try_into()?; + let value = V::get_struct_field("value"); + let expected = MapType::new( + K::to_data_type(), + value.data_type().clone(), + value.is_nullable(), + ); + require!( + map.map_type() == &expected, + Error::scalar_conversion( + format!( + "map<{}, {}, value_contains_null={}>", + expected.key_type().kind_name(), + expected.value_type().kind_name(), + expected.value_contains_null() + ), + format!( + "map<{}, {}, value_contains_null={}>", + map.map_type().key_type().kind_name(), + map.map_type().value_type().kind_name(), + map.map_type().value_contains_null() + ), + ) + ); + map.into_pairs() + .into_iter() + .enumerate() + .map(|(index, (key, value))| { + let key = K::try_from(key) + .map_err(|error| add_scalar_path_context(error, format!("[{index}].key")))?; + let value = V::try_from(value) + .map_err(|error| add_scalar_path_context(error, format!("[{index}].value")))?; + Ok((key, value)) + }) + .try_collect() } } -// TODO: add more From impls - impl PrimitiveType { fn data_type(&self) -> DataType { DataType::Primitive(self.clone()) } + /// Parses a serialized string into a [`Scalar`] of this primitive type, per the Delta + /// protocol's [partition value serialization] rules. An empty string parses as + /// [`Scalar::Null`]. + /// + /// Note: the scan read path does not use this empty-string handling for partition values; it + /// applies a type-dependent cast instead (empty string on a string or binary column surfaces as + /// an empty value rather than null). + /// + /// Timestamp and TimestampNtz accept the space-separated form `{year}-{month}-{day} + /// {hour}:{minute}:{second}[.{fraction}]`. Timestamp (only) also accepts ISO 8601 / RFC 3339 + /// strings such as `1970-01-01T00:00:00.123456Z`; an explicit offset is honored and the value + /// is normalized to UTC. The spec stores timestamp partition values either without a zone + /// (interpreted in the writer's local time zone) or adjusted to UTC with a `Z` suffix. + /// + /// # Errors + /// + /// Returns [`Error::ParseError`] if `raw` is not a valid encoding of this type. + /// + /// [partition value serialization]: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#partition-value-serialization pub fn parse_scalar(&self, raw: &str) -> Result { use PrimitiveType::*; @@ -710,6 +948,7 @@ impl PrimitiveType { Long => self.parse_str_as_scalar(raw, Scalar::Long), Float => self.parse_str_as_scalar(raw, Scalar::Float), Double => self.parse_str_as_scalar(raw, Scalar::Double), + Void => Err(self.parse_error(raw)), Boolean => { if raw.eq_ignore_ascii_case("true") { Ok(Scalar::Boolean(true)) @@ -728,20 +967,18 @@ impl PrimitiveType { let days = date.signed_duration_since(DateTime::UNIX_EPOCH).num_days() as i32; Ok(Scalar::Date(days)) } - // NOTE: Timestamp and TimestampNtz are both parsed into microsecond since unix epoch. - // They may both have the format `{year}-{month}-{day} {hour}:{minute}:{second}`. - // Timestamps may additionally be encoded as a ISO 8601 formatted string such as - // `1970-01-01T00:00:00.123456Z`. - // - // The difference arises mostly in how they are to be handled on the engine side - i.e. timestampNTZ - // is not adjusted to UTC, this is just so we can (de-)serialize it as a date sting. - // https://github.com/delta-io/delta/blob/master/PROTOCOL.md#partition-value-serialization + // NOTE: Timestamp and TimestampNtz are both parsed into microseconds since unix + // epoch. The difference arises mostly in how they are to be handled on the engine + // side - i.e. timestampNTZ is not adjusted to UTC, this is just so we can + // (de-)serialize it as a date string. TimestampNtz | Timestamp => { let mut timestamp = NaiveDateTime::parse_from_str(raw, "%Y-%m-%d %H:%M:%S%.f"); if timestamp.is_err() && *self == Timestamp { - // Note: `%+` specifies the ISO 8601 / RFC 3339 format - timestamp = NaiveDateTime::parse_from_str(raw, "%+"); + // `%+` is chrono's relaxed ISO 8601 / RFC 3339 parser: unlike the stricter + // DateTime::parse_from_rfc3339, it also accepts a space or lowercase `t` + // separator and colon-less offsets (e.g. `+0530`). + timestamp = DateTime::parse_from_str(raw, "%+").map(|dt| dt.naive_utc()); } let timestamp = timestamp.map_err(|_| self.parse_error(raw))?; let timestamp = Utc.from_utc_datetime(×tamp); @@ -755,6 +992,37 @@ impl PrimitiveType { _ => unreachable!(), } } + IntervalYearMonth => parse_year_month_interval(raw) + .map(Scalar::IntervalYearMonth) + .ok_or_else(|| self.parse_error(raw)), + IntervalDayTime => parse_day_time_interval(raw) + .map(Scalar::IntervalDayTime) + .ok_or_else(|| self.parse_error(raw)), + // Kernel does not support parsing text into Geometry/Geography types yet. + #[cfg(feature = "geo-type-in-dev")] + Geometry(_) | Geography(_) => Err(Error::Unsupported(format!( + "parse_scalar is not supported for {self:?}" + ))), + } + } + + /// Casts an empty partition-value string to its target [`Scalar`], or `None` for a null value. + /// + /// An empty string is a value for a string or binary column (the empty string / empty bytes) + /// but has no representation for any other type, so it casts to null there. This aligns with + /// Spark, which reads the partition-value map as a non-ANSI SQL cast, and is the empty-string + /// rule the scan read path applies, in place of [`parse_scalar`]'s null-for-every-type + /// handling. + /// + /// A literal empty string only reaches this path from a foreign writer, since kernel + /// serializes its own empty and null partition values to JSON null on write. + /// + /// [`parse_scalar`]: PrimitiveType::parse_scalar + pub(crate) fn empty_string_partition_cast(&self) -> Option { + match self { + PrimitiveType::String => Some(Scalar::String(String::new())), + PrimitiveType::Binary => Some(Scalar::Binary(Vec::new())), + _ => None, } } @@ -765,7 +1033,8 @@ impl PrimitiveType { /// Parse a string as a scalar value, returning an error if the string is not parseable. /// /// The `f` function is used to convert the parsed value into a `Scalar`. - /// The desired type that `FromStr::parse` should parse into is inferred from the parameter type of `f`. + /// The desired type that `FromStr::parse` should parse into is inferred from the parameter type + /// of `f`. fn parse_str_as_scalar( &self, raw: &str, @@ -778,15 +1047,15 @@ impl PrimitiveType { } fn parse_decimal(raw: &str, dtype: DecimalType) -> Result { + let parse_error = || PrimitiveType::from(dtype).parse_error(raw); let (base, exp): (&str, i128) = match raw.find(['e', 'E']) { None => (raw, 0), // no 'e' or 'E', so there's no exponent Some(pos) => { let (base, exp) = raw.split_at(pos); // exp now has '[e/E][exponent]', strip the 'e/E' and parse it - (base, exp[1..].parse()?) + (base, exp[1..].parse().map_err(|_| parse_error())?) } }; - let parse_error = || PrimitiveType::from(dtype).parse_error(raw); require!(!base.is_empty(), parse_error()); // now split on any '.' and parse @@ -811,22 +1080,249 @@ impl PrimitiveType { let scale: u8 = scale.try_into().map_err(|_| parse_error())?; require!(scale == dtype.scale(), parse_error()); let int: i128 = match frac_part { - None => int_part.parse()?, - Some(frac_part) => format!("{int_part}{frac_part}").parse()?, + None => int_part.parse().map_err(|_| parse_error())?, + Some(frac_part) => format!("{int_part}{frac_part}") + .parse() + .map_err(|_| parse_error())?, }; - Ok(Scalar::Decimal(DecimalData::try_new(int, dtype)?)) + DecimalData::try_new(int, dtype) + .map(Scalar::Decimal) + .map_err(|_| parse_error()) + } +} + +// === ANSI interval literal parsing === +// +// Spark serializes interval partition values as ANSI interval literal strings, e.g. +// `INTERVAL '1-0' YEAR TO MONTH` (= 12 months) and +// `INTERVAL '1 12:30:45.000000' DAY TO SECOND`. Kernel represents intervals as their +// physical integer (i32 months / i64 microseconds), so these parsers convert literals to +// their physical representation. +// +// Currently, there's no DAT test for DAY TO SECOND partition values. +// TODO: Unify SQL literal parsing with `expressions/sql.rs`. + +/// Extracts the quoted body and field range of an ANSI interval literal. +fn extract_interval_literal(raw: &str) -> Option<(&str, IntervalFieldRange)> { + let trimmed = raw.trim(); + if !trimmed.get(..8)?.eq_ignore_ascii_case("INTERVAL") { + return None; + } + let rest = trimmed[8..].trim_start().strip_prefix('\'')?; + let (body, after) = rest.split_once('\'')?; + let field_range = after + .split_whitespace() + .map(str::to_ascii_lowercase) + .join(" "); + let interval_type = parse_interval_type(&format!("interval {field_range}"))?; + Some((body, interval_type)) +} + +fn interval_magnitude(body: &str) -> Option<(bool, &str)> { + let (negative, magnitude) = body.strip_prefix('-').map_or((false, body), |m| (true, m)); + (!magnitude.starts_with('-') && !magnitude.contains('+')).then_some((negative, magnitude)) +} + +fn signed_i32(magnitude: u32, negative: bool) -> Option { + let signed = if negative { + -(magnitude as i64) + } else { + magnitude as i64 + }; + i32::try_from(signed).ok() +} + +fn signed_i64(magnitude: u64, negative: bool) -> Option { + let signed = if negative { + -(magnitude as i128) + } else { + magnitude as i128 + }; + i64::try_from(signed).ok() +} + +fn parse_seconds(raw: &str) -> Option<(u64, u64)> { + match raw.split_once('.') { + Some((seconds, fraction)) => { + if fraction.is_empty() || !fraction.bytes().all(|byte| byte.is_ascii_digit()) { + return None; + } + // Fractional precision beyond microseconds is truncated. + let fraction: String = fraction + .chars() + .chain(std::iter::repeat('0')) + .take(6) + .collect(); + Some((seconds.parse().ok()?, fraction.parse().ok()?)) + } + None => Some((raw.parse().ok()?, 0)), } } +fn parse_clock(raw: &str, has_seconds: bool) -> Option<(u64, u64, u64, u64)> { + let mut parts = raw.split(':'); + let hours = parts.next()?.parse().ok()?; + let minutes = parts.next()?.parse::().ok()?; + let (seconds, micros) = if has_seconds { + parse_seconds(parts.next()?)? + } else { + (0, 0) + }; + (parts.next().is_none() && minutes < 60 && seconds < 60) + .then_some((hours, minutes, seconds, micros)) +} + +fn checked_day_time_micros( + days: u64, + hours: u64, + minutes: u64, + seconds: u64, + micros: u64, +) -> Option { + days.checked_mul(24)? + .checked_add(hours)? + .checked_mul(60)? + .checked_add(minutes)? + .checked_mul(60)? + .checked_add(seconds)? + .checked_mul(1_000_000)? + .checked_add(micros) +} + +/// Parses a year-month interval literal into a signed total month count. +fn parse_year_month_interval(raw: &str) -> Option { + use IntervalField::*; + + let (body, field_range) = extract_interval_literal(raw)?; + let (negative, magnitude) = interval_magnitude(body)?; + let total = match (field_range.start, field_range.end) { + (Year, Year) => magnitude.parse::().ok()?.checked_mul(12)?, + (Month, Month) => magnitude.parse().ok()?, + (Year, Month) => { + let (years, months) = magnitude.split_once('-')?; + let months = months.parse::().ok()?; + if months >= 12 { + return None; + } + years + .parse::() + .ok()? + .checked_mul(12)? + .checked_add(months)? + } + _ => return None, + }; + signed_i32(total, negative) +} + +/// Parses a day-time interval literal into a signed total microsecond count. +fn parse_day_time_interval(raw: &str) -> Option { + use IntervalField::*; + + let (body, field_range) = extract_interval_literal(raw)?; + let (negative, magnitude) = interval_magnitude(body)?; + let (days, hours, minutes, seconds, micros) = match (field_range.start, field_range.end) { + (Day, Day) => (magnitude.parse().ok()?, 0, 0, 0, 0), + (Hour, Hour) => (0, magnitude.parse().ok()?, 0, 0, 0), + (Minute, Minute) => (0, 0, magnitude.parse().ok()?, 0, 0), + (Second, Second) => { + let (seconds, micros) = parse_seconds(magnitude)?; + (0, 0, 0, seconds, micros) + } + (Day, Hour) => { + let (days, hours) = magnitude.split_once(' ')?; + let hours = hours.parse::().ok()?; + if hours >= 24 { + return None; + } + (days.parse().ok()?, hours, 0, 0, 0) + } + (Day, Minute) | (Day, Second) => { + let (days, time) = magnitude.split_once(' ')?; + let (hours, minutes, seconds, micros) = parse_clock(time, field_range.end == Second)?; + if hours >= 24 { + return None; + } + (days.parse().ok()?, hours, minutes, seconds, micros) + } + (Hour, Minute) | (Hour, Second) => { + let (hours, minutes, seconds, micros) = + parse_clock(magnitude, field_range.end == Second)?; + (0, hours, minutes, seconds, micros) + } + (Minute, Second) => { + let (minutes, seconds) = magnitude.split_once(':')?; + let (seconds, micros) = parse_seconds(seconds)?; + if seconds >= 60 { + return None; + } + (0, 0, minutes.parse().ok()?, seconds, micros) + } + _ => return None, + }; + signed_i64( + checked_day_time_micros(days, hours, minutes, seconds, micros)?, + negative, + ) +} + #[cfg(test)] mod tests { use std::f32::consts::PI; + use std::fmt::Debug; - use crate::expressions::{column_expr, BinaryPredicateOp}; - use crate::utils::test_utils::assert_result_error_with_message; - use crate::{Expression as Expr, Predicate as Pred}; + use bytes::Bytes; + use delta_kernel_derive::{IntoStructData, ToSchema, TryFromStructData}; + use rstest::rstest; use super::*; + use crate::expressions::{col, lit, BinaryPredicateOp}; + use crate::schema::ToSchema as _; + use crate::table_features::TableFeature; + use crate::unit_test_utils::assert_result_error_with_message; + use crate::Predicate as Pred; + + #[rstest] + #[case::truncates(Scalar::Integer(7), Scalar::Integer(2), Some(Scalar::Integer(3)))] + #[case::zero_divisor(Scalar::Integer(7), Scalar::Integer(0), None)] + #[case::floats_unsupported(Scalar::Double(7.0), Scalar::Double(2.0), None)] + fn test_try_div_truncates_and_returns_none_for_zero_divisor_and_floats( + #[case] left: Scalar, + #[case] right: Scalar, + #[case] expected: Option, + ) { + assert_eq!(left.try_div(&right), expected); + } + + #[test] + fn test_void_parse_scalar() { + // Empty string should produce Null (like all primitive types) + let scalar = PrimitiveType::Void.parse_scalar("").unwrap(); + assert_eq!(scalar, Scalar::Null(DataType::VOID)); + + // Non-empty string should fail + PrimitiveType::Void.parse_scalar("anything").unwrap_err(); + } + + #[cfg(feature = "geo-type-in-dev")] + #[rstest::rstest] + #[case(PrimitiveType::Geometry(Box::new( + crate::schema::GeometryType::try_new("EPSG:4326").unwrap() + )))] + #[case(PrimitiveType::Geography(Box::new( + crate::schema::GeographyType::try_new( + "EPSG:4326", + crate::schema::EdgeInterpolationAlgorithm::Spherical, + ) + .unwrap() + )))] + fn test_geo_parse_scalar_unsupported(#[case] ptype: PrimitiveType) { + let err = ptype.parse_scalar("anything").unwrap_err(); + assert!( + matches!(err, Error::Unsupported(_)), + "expected Unsupported, got: {err:?}" + ); + } #[test] fn test_bad_decimal() { @@ -835,6 +1331,7 @@ mod tests { PrimitiveType::parse_decimal("0.12345", dtype).expect_err("should have failed"); PrimitiveType::parse_decimal("12345", dtype).expect_err("should have failed"); } + #[test] fn test_decimal_display() { let s = Scalar::decimal(123456789, 9, 2).unwrap(); @@ -948,8 +1445,10 @@ mod tests { fn expect_fail_parse(raw: &str, prec: u8, scale: u8) { let s = PrimitiveType::decimal(prec, scale).unwrap(); - let res = s.parse_scalar(raw); - assert!(res.is_err(), "Fail on {raw}"); + match s.parse_scalar(raw) { + Err(Error::ParseError(..)) => {} + other => panic!("expected ParseError for {raw:?}, got {other:?}"), + } } #[test] @@ -986,18 +1485,13 @@ mod tests { elements: vec![Scalar::Integer(1), Scalar::Integer(2), Scalar::Integer(3)], }); - let column = column_expr!("item"); - let array_op = Pred::binary(BinaryPredicateOp::In, Expr::literal(10), array.clone()); - let array_not_op = Pred::not(Pred::binary( - BinaryPredicateOp::In, - Expr::literal(10), - array, - )); - let column_op = Pred::binary(BinaryPredicateOp::In, Expr::literal(PI), column.clone()); + let array_op = Pred::binary(BinaryPredicateOp::In, lit(10), array.clone()); + let array_not_op = Pred::not(Pred::binary(BinaryPredicateOp::In, lit(10), array)); + let column_op = Pred::binary(BinaryPredicateOp::In, lit(PI), col!("item")); let column_not_op = Pred::not(Pred::binary( BinaryPredicateOp::In, - Expr::literal("Cool"), - column, + lit("Cool"), + col!("item"), )); assert_eq!(&format!("{array_op}"), "10 IN (1, 2, 3)"); assert_eq!(&format!("{array_not_op}"), "NOT(10 IN (1, 2, 3))"); @@ -1054,48 +1548,58 @@ mod tests { ); } - #[test] - fn test_timestamp_parse() { - let assert_timestamp_eq = |scalar_string, micros| { - let scalar = PrimitiveType::Timestamp - .parse_scalar(scalar_string) - .unwrap(); - assert_eq!(scalar, Scalar::Timestamp(micros)); + #[rstest] + #[case::seconds("2011-01-11 13:06:07", 1294751167000000)] + #[case::fractional_seconds("2011-01-11 13:06:07.123456", 1294751167123456)] + #[case::epoch("1970-01-01 00:00:00", 0)] + fn test_timestamp_space_form_parse( + #[values(PrimitiveType::Timestamp, PrimitiveType::TimestampNtz)] p_type: PrimitiveType, + #[case] raw: &str, + #[case] micros: i64, + ) { + let expected = match p_type { + PrimitiveType::Timestamp => Scalar::Timestamp(micros), + PrimitiveType::TimestampNtz => Scalar::TimestampNtz(micros), + _ => unreachable!(), }; - assert_timestamp_eq("1971-07-22T03:06:40.678910Z", 49000000678910); - assert_timestamp_eq("1971-07-22T03:06:40Z", 49000000000000); - assert_timestamp_eq("2011-01-11 13:06:07", 1294751167000000); - assert_timestamp_eq("2011-01-11 13:06:07.123456", 1294751167123456); - assert_timestamp_eq("1970-01-01 00:00:00", 0); - } - - #[test] - fn test_timestamp_ntz_parse() { - let assert_timestamp_eq = |scalar_string, micros| { - let scalar = PrimitiveType::TimestampNtz - .parse_scalar(scalar_string) - .unwrap(); - assert_eq!(scalar, Scalar::TimestampNtz(micros)); - }; - assert_timestamp_eq("2011-01-11 13:06:07", 1294751167000000); - assert_timestamp_eq("2011-01-11 13:06:07.123456", 1294751167123456); - assert_timestamp_eq("1970-01-01 00:00:00", 0); - } - - #[test] - fn test_timestamp_parse_fails() { - let assert_timestamp_fails = |p_type: &PrimitiveType, scalar_string| { - let res = p_type.parse_scalar(scalar_string); - assert!(res.is_err()); - }; - - let p_type = PrimitiveType::TimestampNtz; - assert_timestamp_fails(&p_type, "1971-07-22T03:06:40.678910Z"); - assert_timestamp_fails(&p_type, "1971-07-22T03:06:40Z"); - assert_timestamp_fails(&p_type, "1971-07-22"); - - let p_type = PrimitiveType::Timestamp; - assert_timestamp_fails(&p_type, "1971-07-22"); + assert_eq!(p_type.parse_scalar(raw).unwrap(), expected); + } + + #[rstest] + #[case::z_fractional("1971-07-22T03:06:40.678910Z", 49000000678910)] + #[case::z_seconds("1971-07-22T03:06:40Z", 49000000000000)] + #[case::z("2024-06-15T14:30:00Z", 1718461800000000)] + #[case::lowercase_t_z("2024-06-15t14:30:00z", 1718461800000000)] + #[case::zero_offset("2024-06-15T14:30:00+00:00", 1718461800000000)] + #[case::negative_zero_offset("2024-06-15T14:30:00-00:00", 1718461800000000)] + #[case::positive_offset("2024-06-15T14:30:00+05:00", 1718443800000000)] // 09:30:00Z + #[case::negative_offset("2024-06-15T14:30:00-05:00", 1718479800000000)] // 19:30:00Z + #[case::half_hour_offset("2024-06-15T14:30:00+05:30", 1718442000000000)] // 09:00:00Z + #[case::colonless_offset("2024-06-15T14:30:00+0530", 1718442000000000)] // 09:00:00Z + #[case::fractional_with_offset("2024-06-15T14:30:00.456+05:00", 1718443800456000)] + #[case::space_separator_with_offset("2024-06-15 14:30:00+05:00", 1718443800000000)] + #[case::pre_epoch_after_normalization("1970-01-01T00:00:00+05:00", -18000000000)] + fn test_timestamp_iso8601_parse(#[case] raw: &str, #[case] micros: i64) { + let scalar = PrimitiveType::Timestamp.parse_scalar(raw).unwrap(); + assert_eq!(scalar, Scalar::Timestamp(micros)); + } + + #[rstest] + // TimestampNtz has no ISO 8601 fallback: rejects Z, offsets, and date-only + #[case::ntz_z_fractional(PrimitiveType::TimestampNtz, "1971-07-22T03:06:40.678910Z")] + #[case::ntz_z(PrimitiveType::TimestampNtz, "1971-07-22T03:06:40Z")] + #[case::ntz_offset(PrimitiveType::TimestampNtz, "2024-06-15T14:30:00+05:00")] + #[case::ntz_space_offset(PrimitiveType::TimestampNtz, "2024-06-15 14:30:00+05:00")] + #[case::ntz_date_only(PrimitiveType::TimestampNtz, "1971-07-22")] + // Timestamp rejects date-only and the T-form without a trailing Z or offset + #[case::date_only(PrimitiveType::Timestamp, "1971-07-22")] + #[case::zoneless_t_form(PrimitiveType::Timestamp, "2024-06-15T14:30:00")] + // out-of-range offsets and UTC normalizations that overflow chrono's datetime range + // error cleanly rather than being misinterpreted + #[case::offset_out_of_range(PrimitiveType::Timestamp, "2024-06-15T14:30:00+24:00")] + #[case::normalization_overflow(PrimitiveType::Timestamp, "-262143-01-01T00:00:00+05:00")] + fn test_timestamp_parse_fails(#[case] p_type: PrimitiveType, #[case] raw: &str) { + assert!(p_type.parse_scalar(raw).is_err()); } #[test] @@ -1131,15 +1635,47 @@ mod tests { assert!(!null.logical_eq(&null)); } + fn assert_into_scalar_matches_to_data_type(value: T) { + let scalar: Scalar = value.into(); + assert!(!scalar.is_null()); + assert_eq!(scalar.data_type(), T::to_data_type()); + } + + #[test] + fn into_scalar_matches_to_data_type() { + assert_into_scalar_matches_to_data_type(true); + assert_into_scalar_matches_to_data_type(1i8); + assert_into_scalar_matches_to_data_type(1i16); + assert_into_scalar_matches_to_data_type(1i32); + assert_into_scalar_matches_to_data_type(1i64); + assert_into_scalar_matches_to_data_type(1.0f32); + assert_into_scalar_matches_to_data_type(1.0f64); + assert_into_scalar_matches_to_data_type("x".to_string()); + assert_into_scalar_matches_to_data_type(Bytes::from_static(b"x")); + assert_into_scalar_matches_to_data_type(TableFeature::DeletionVectors); + + assert_into_scalar_matches_to_data_type(vec![1i8, 2, 3]); + assert_into_scalar_matches_to_data_type(vec![1, 2, 3]); + assert_into_scalar_matches_to_data_type(vec![Some(1i32), None]); + assert_into_scalar_matches_to_data_type(HashMap::from([ + ("key1".to_string(), 42i32), + ("key2".to_string(), 100i32), + ])); + assert_into_scalar_matches_to_data_type(HashMap::from([ + ("key1".to_string(), Some(42i32)), + ("key2".to_string(), None), + ])); + } + #[test] - fn test_hashmap_conversion() -> DeltaResult<()> { + fn test_hashmap_conversion() { // Create a simple HashMap with string keys and integer values let mut map = HashMap::new(); map.insert("key1".to_string(), 42i32); map.insert("key2".to_string(), 100i32); // Convert HashMap to Scalar - let scalar = Scalar::try_from(map)?; + let scalar = Scalar::from(map); // Verify the scalar is of Map type assert!(matches!(scalar, Scalar::Map(_))); @@ -1161,12 +1697,10 @@ mod tests { let entry2 = (Scalar::String("key2".to_string()), Scalar::Integer(100)); assert!(pairs.contains(&entry1), "Missing key1 -> 42 pair"); assert!(pairs.contains(&entry2), "Missing key2 -> 100 pair"); - - Ok(()) } #[test] - fn test_hashmap_conversion_with_nullable_values() -> DeltaResult<()> { + fn test_hashmap_conversion_with_nullable_values() { // Create a HashMap with string keys and optional integer values let mut map = HashMap::new(); map.insert("key1".to_string(), Some(42i32)); @@ -1174,7 +1708,7 @@ mod tests { map.insert("key3".to_string(), Some(100i32)); // Convert HashMap to Scalar - let scalar = Scalar::try_from(map)?; + let scalar = Scalar::from(map); // Verify the scalar is of Map type assert!(matches!(scalar, Scalar::Map(_))); @@ -1201,17 +1735,15 @@ mod tests { assert!(pairs.contains(&entry1), "Missing key1 -> 42 pair"); assert!(pairs.contains(&entry2), "Missing key2 -> null pair"); assert!(pairs.contains(&entry3), "Missing key3 -> 100 pair"); - - Ok(()) } #[test] - fn test_vec_conversion() -> DeltaResult<()> { + fn test_vec_conversion() { // Create a simple Vec with integer values let vec = vec![42i32, 100i32, 200i32]; // Convert Vec to Scalar - let scalar = Scalar::try_from(vec)?; + let scalar = Scalar::from(vec); // Verify the scalar is of Array type assert!(matches!(scalar, Scalar::Array(_))); @@ -1232,17 +1764,15 @@ mod tests { assert_eq!(elements[0], Scalar::Integer(42)); assert_eq!(elements[1], Scalar::Integer(100)); assert_eq!(elements[2], Scalar::Integer(200)); - - Ok(()) } #[test] - fn test_vec_conversion_with_nullable_values() -> DeltaResult<()> { + fn test_vec_conversion_with_nullable_values() { // Create a Vec with optional integer values let vec = vec![Some(42i32), None, Some(100i32)]; // Convert Vec to Scalar - let scalar = Scalar::try_from(vec)?; + let scalar = Scalar::from(vec); // Verify the scalar is of Array type assert!(matches!(scalar, Scalar::Array(_))); @@ -1264,15 +1794,13 @@ mod tests { assert_eq!(elements[0], Scalar::Integer(42)); assert!(elements[1].is_null()); assert_eq!(elements[2], Scalar::Integer(100)); - - Ok(()) } #[test] - fn test_vec_conversion_different_types() -> DeltaResult<()> { + fn test_vec_conversion_different_types() { // Test with string Vec let string_vec = vec!["hello".to_string(), "world".to_string()]; - let string_scalar = Scalar::try_from(string_vec)?; + let string_scalar = Scalar::from(string_vec); if let Scalar::Array(array_data) = string_scalar { let expected_array_type = ArrayType::new(DataType::STRING, false); @@ -1283,7 +1811,7 @@ mod tests { // Test with bool Vec let bool_vec = vec![true, false, true]; - let bool_scalar = Scalar::try_from(bool_vec)?; + let bool_scalar = Scalar::from(bool_vec); if let Scalar::Array(array_data) = bool_scalar { let expected_array_type = ArrayType::new(DataType::BOOLEAN, false); @@ -1291,8 +1819,6 @@ mod tests { } else { panic!("Expected Array scalar"); } - - Ok(()) } #[test] @@ -1325,4 +1851,433 @@ mod tests { panic!("Expected Binary scalar"); } } + + const INTERVAL_YM_LITERAL: &str = "INTERVAL '1-0' YEAR TO MONTH"; + const INTERVAL_YM_LITERAL_LARGER: &str = "INTERVAL '2-0' YEAR TO MONTH"; + const INTERVAL_DT_LITERAL: &str = "INTERVAL '0 01:00:00.000000' DAY TO SECOND"; + const INTERVAL_DT_LITERAL_LARGER: &str = "INTERVAL '0 02:00:00.000000' DAY TO SECOND"; + + #[test] + fn interval_parse_and_data_type() { + let ym = PrimitiveType::IntervalYearMonth + .parse_scalar(INTERVAL_YM_LITERAL) + .unwrap(); + assert_eq!(ym.data_type(), DataType::INTERVAL_YEAR_MONTH); + + let dt = PrimitiveType::IntervalDayTime + .parse_scalar(INTERVAL_DT_LITERAL) + .unwrap(); + assert_eq!(dt.data_type(), DataType::INTERVAL_DAY_TIME); + } + + #[test] + fn interval_logical_partial_cmp() { + let ym = PrimitiveType::IntervalYearMonth + .parse_scalar(INTERVAL_YM_LITERAL) + .unwrap(); + let ym_larger = PrimitiveType::IntervalYearMonth + .parse_scalar(INTERVAL_YM_LITERAL_LARGER) + .unwrap(); + assert_eq!(ym.logical_partial_cmp(&ym_larger), Some(Ordering::Less)); + assert_eq!(ym_larger.logical_partial_cmp(&ym), Some(Ordering::Greater)); + assert_eq!(ym.logical_partial_cmp(&ym), Some(Ordering::Equal)); + + let dt = PrimitiveType::IntervalDayTime + .parse_scalar(INTERVAL_DT_LITERAL) + .unwrap(); + let dt_larger = PrimitiveType::IntervalDayTime + .parse_scalar(INTERVAL_DT_LITERAL_LARGER) + .unwrap(); + assert_eq!(dt.logical_partial_cmp(&dt_larger), Some(Ordering::Less)); + + // Comparison is defined only within a family; everything else is incomparable. + assert_eq!(ym.logical_partial_cmp(&dt), None); + assert_eq!(ym.logical_partial_cmp(&Scalar::Integer(0)), None); + } + + #[test] + fn interval_logical_eq() { + let ym = PrimitiveType::IntervalYearMonth + .parse_scalar(INTERVAL_YM_LITERAL) + .unwrap(); + let ym_same = PrimitiveType::IntervalYearMonth + .parse_scalar(INTERVAL_YM_LITERAL) + .unwrap(); + assert!(ym.logical_eq(&ym_same)); + // SQL NULL semantics: NULL is not equal to anything, including a like-typed value. + assert!(!ym.logical_eq(&Scalar::null(DataType::INTERVAL_YEAR_MONTH))); + } + + #[test] + fn test_year_month_interval_literal_parse() { + // Values observed in the DAT intv_003 workload, plus negatives and zero. + let cases = [ + ("INTERVAL '1-0' YEAR TO MONTH", 12), + ("INTERVAL '2-6' YEAR TO MONTH", 30), + ("INTERVAL '0-0' YEAR TO MONTH", 0), + ("INTERVAL '0-11' YEAR TO MONTH", 11), + ("INTERVAL '-1-6' YEAR TO MONTH", -18), + ]; + for (literal, months) in cases { + assert_eq!( + PrimitiveType::IntervalYearMonth + .parse_scalar(literal) + .unwrap(), + Scalar::IntervalYearMonth(months), + "parsing {literal}" + ); + } + } + + #[test] + fn test_day_time_interval_literal_parse() { + let cases = [ + ("INTERVAL '0 00:00:00.000000' DAY TO SECOND", 0_i64), + ( + "INTERVAL '1 12:30:45.000000' DAY TO SECOND", + 131_445_000_000, + ), + ("INTERVAL '0 00:00:00.000005' DAY TO SECOND", 5), + ( + "INTERVAL '-1 00:00:00.000000' DAY TO SECOND", + -86_400_000_000, + ), + ]; + for (literal, micros) in cases { + assert_eq!( + PrimitiveType::IntervalDayTime + .parse_scalar(literal) + .unwrap(), + Scalar::IntervalDayTime(micros), + "parsing {literal}" + ); + } + // Fractional seconds shorter than 6 digits are right-padded to microseconds. + assert_eq!( + PrimitiveType::IntervalDayTime + .parse_scalar("INTERVAL '0 00:00:00.5' DAY TO SECOND") + .unwrap(), + Scalar::IntervalDayTime(500_000) + ); + } + + #[test] + fn test_interval_literal_parse_rejects_malformed() { + for bad in [ + "1-0", // missing INTERVAL/unit + "INTERVAL '1-0' DAY TO SECOND", // unit mismatch + "INTERVAL 'x-0' YEAR TO MONTH", // non-numeric + "INTERVAL '1' YEAR TO MONTH", // missing '-' + "INTERVAL '1-12' YEAR TO MONTH", // month field out of range + "INTERVAL '+5' YEAR", // leading plus sign + "INTERVAL '1-+5' YEAR TO MONTH", // embedded plus sign + "INTERVAL '1 12:30' DAY TO SECOND", // missing seconds field + "INTERVAL '0.+5' SECOND", // fractional plus sign + "INTERVAL '0 00:00:00.123456xyz' DAY TO SECOND", + "INTERVAL '5.' SECOND", + "INTERVAL '0.123456.789' SECOND", + ] { + assert!( + PrimitiveType::IntervalYearMonth.parse_scalar(bad).is_err() + && PrimitiveType::IntervalDayTime.parse_scalar(bad).is_err(), + "expected {bad} to fail parsing as both interval families" + ); + } + } + + #[rstest] + #[case("INTERVAL '1 24' DAY TO HOUR")] + #[case("INTERVAL '1 00:60' DAY TO MINUTE")] + #[case("INTERVAL '1 00:00:60' DAY TO SECOND")] + #[case("INTERVAL '00:60' HOUR TO MINUTE")] + #[case("INTERVAL '00:00:60' HOUR TO SECOND")] + #[case("INTERVAL '00:60' MINUTE TO SECOND")] + #[case("INTERVAL '1 02:03:04:05' DAY TO SECOND")] + fn test_day_time_interval_rejects_out_of_range_subordinate(#[case] bad: &str) { + assert!( + PrimitiveType::IntervalDayTime.parse_scalar(bad).is_err(), + "{bad}" + ); + } + + #[rstest] + #[case("INTERVAL '1' YEAR", Scalar::IntervalYearMonth(12))] + #[case("INTERVAL '-6' MONTH", Scalar::IntervalYearMonth(-6))] + #[case("INTERVAL '2-6' YEAR TO MONTH", Scalar::IntervalYearMonth(30))] + #[case("INTERVAL '1' DAY", Scalar::IntervalDayTime(86_400_000_000))] + #[case("INTERVAL '25' HOUR", Scalar::IntervalDayTime(90_000_000_000))] + #[case("INTERVAL '90' MINUTE", Scalar::IntervalDayTime(5_400_000_000))] + #[case("INTERVAL '1.5' SECOND", Scalar::IntervalDayTime(1_500_000))] + #[case("INTERVAL '1 02' DAY TO HOUR", Scalar::IntervalDayTime(93_600_000_000))] + #[case( + "INTERVAL '1 02:03' DAY TO MINUTE", + Scalar::IntervalDayTime(93_780_000_000) + )] + #[case( + "INTERVAL '1 02:03:04.5' DAY TO SECOND", + Scalar::IntervalDayTime(93_784_500_000) + )] + #[case( + "INTERVAL '25:30' HOUR TO MINUTE", + Scalar::IntervalDayTime(91_800_000_000) + )] + #[case( + "INTERVAL '25:30:45.25' HOUR TO SECOND", + Scalar::IntervalDayTime(91_845_250_000) + )] + #[case( + "INTERVAL '90:45.25' MINUTE TO SECOND", + Scalar::IntervalDayTime(5_445_250_000) + )] + fn test_narrowed_interval_literal_parse(#[case] literal: &str, #[case] expected: Scalar) { + assert_eq!( + expected + .data_type() + .as_primitive_opt() + .unwrap() + .parse_scalar(literal) + .unwrap(), + expected + ); + } + + #[test] + fn test_interval_literal_overflow_rejected() { + assert_eq!( + parse_year_month_interval("INTERVAL '178956970-8' YEAR TO MONTH"), + None + ); + assert_eq!( + parse_year_month_interval("INTERVAL '-178956970-9' YEAR TO MONTH"), + None + ); + assert_eq!( + parse_day_time_interval("INTERVAL '9223372036854.775808' SECOND"), + None + ); + assert_eq!( + parse_day_time_interval("INTERVAL '-9223372036854.775809' SECOND"), + None + ); + } + + #[test] + fn test_interval_literal_case_whitespace_and_fraction_truncation() { + assert_eq!( + parse_year_month_interval(" interval '1-0' year to month "), + Some(12) + ); + assert_eq!( + parse_day_time_interval("INTERVAL '0.1234567' SECOND"), + Some(123_456) + ); + } + + /// `TryFrom` must invert `Into` and produce the expected data type. + fn assert_round_trip(value: T, expected_type: impl Into) + where + T: Clone + Debug + PartialEq + Into + TryFrom, + { + let scalar: Scalar = value.clone().into(); + assert_eq!(scalar.data_type(), expected_type.into()); + assert_eq!(T::try_from(scalar).unwrap(), value); + } + + #[test] + fn scalar_conversions_round_trip() { + assert_round_trip(1i8, DataType::BYTE); + assert_round_trip(2i16, DataType::SHORT); + assert_round_trip(3i32, DataType::INTEGER); + assert_round_trip(4i64, DataType::LONG); + assert_round_trip(PI, DataType::FLOAT); + assert_round_trip(6.0f64, DataType::DOUBLE); + assert_round_trip(true, DataType::BOOLEAN); + assert_round_trip("seven".to_string(), DataType::STRING); + assert_round_trip(Bytes::from_static(b"eight"), DataType::BINARY); + let decimal_type = DecimalType::try_new(2, 1).unwrap(); + assert_round_trip(DecimalData::try_new(9, decimal_type).unwrap(), decimal_type); + assert_round_trip(vec![10i32, 11], ArrayType::new(DataType::INTEGER, false)); + assert_round_trip( + vec![Some(12i32), None], + ArrayType::new(DataType::INTEGER, true), + ); + assert_round_trip( + HashMap::from([("k".to_string(), "v".to_string())]), + MapType::new(DataType::STRING, DataType::STRING, false), + ); + assert_round_trip( + HashMap::from([("k".to_string(), None as Option)]), + MapType::new(DataType::STRING, DataType::STRING, true), + ); + assert_round_trip(Some(13i32), DataType::INTEGER); + assert_round_trip(None::, DataType::INTEGER); + } + + #[rstest] + #[case::long(Scalar::Long(1), "expected i32, found long")] + #[case::date(Scalar::Date(1), "expected i32, found date")] + #[case::interval_year_month( + Scalar::IntervalYearMonth(1), + "expected i32, found interval_year_month" + )] + #[case::string(Scalar::from("1"), "expected i32, found string")] + #[case::null(Scalar::null(DataType::INTEGER), "expected i32, found null")] + fn i32_conversion_rejects_other_variants(#[case] scalar: Scalar, #[case] expected: &str) { + assert_result_error_with_message(i32::try_from(scalar), expected); + } + + #[rstest] + #[case::timestamp(Scalar::Timestamp(1), "expected i64, found timestamp")] + #[case::timestamp_ntz(Scalar::TimestampNtz(1), "expected i64, found timestamp_ntz")] + #[case::interval_day_time(Scalar::IntervalDayTime(1), "expected i64, found interval_day_time")] + #[case::integer(Scalar::Integer(1), "expected i64, found integer")] + fn i64_conversion_rejects_other_variants(#[case] scalar: Scalar, #[case] expected: &str) { + assert_result_error_with_message(i64::try_from(scalar), expected); + } + + #[test] + fn null_option_requires_matching_element_data_type() { + assert_eq!( + Option::::try_from(Scalar::null(DataType::INTEGER)).unwrap(), + None + ); + assert_result_error_with_message( + Option::::try_from(Scalar::null(DataType::STRING)), + "expected integer, found string", + ); + } + + #[test] + fn array_with_nulls_requires_optional_element_type() { + let scalar = Scalar::from(vec![Some(1i32), None]); + assert_result_error_with_message( + Vec::::try_from(scalar), + "expected array, found array", + ); + } + + #[derive(Clone, Debug, PartialEq, ToSchema, IntoStructData, TryFromStructData)] + struct Address { + city: String, + zip: Option, + } + + #[derive(Clone, Debug, PartialEq, ToSchema, IntoStructData, TryFromStructData)] + struct Person { + id: i32, + address: Address, + display_names: Vec, + } + + fn test_person() -> Person { + Person { + id: 1, + address: Address { + city: "NYC".to_string(), + zip: None, + }, + display_names: vec!["ace".to_string()], + } + } + + #[test] + fn derived_struct_conversions_round_trip() { + assert_round_trip(test_person(), Person::to_schema()); + } + + #[rstest] + #[case::not_a_struct(Scalar::Long(1), "expected Person, found long")] + #[case::wrong_field_type( + Scalar::Struct(StructData::from_values_unchecked( + Person::to_schema(), + vec![ + Scalar::from("not an integer"), + Scalar::from(Address { city: "NYC".to_string(), zip: None }), + Scalar::from(vec!["ace".to_string()]), + ], + )), + "id: expected i32, found string" + )] + fn derived_struct_conversion_rejects_mismatched_scalars( + #[case] scalar: Scalar, + #[case] expected: &str, + ) { + assert_result_error_with_message(Person::try_from(scalar), expected); + } + + #[test] + fn derived_struct_conversion_requires_exact_field_count() { + let values = vec![Scalar::from(1)]; + let struct_data = StructData::from_values_unchecked(Person::to_schema(), values); + assert_result_error_with_message( + Person::try_from(struct_data), + "expected 3 struct values, found 1 struct values", + ); + } + + #[test] + fn derived_struct_conversion_matches_fields_by_name() { + let person = test_person(); + let Scalar::Struct(data) = Scalar::from(person.clone()) else { + unreachable!() + }; + let (mut fields, mut values) = data.into_parts(); + fields.swap(0, 2); + values.swap(0, 2); + let reordered = StructData::try_new(fields, values).unwrap(); + assert_eq!(Person::try_from(reordered).unwrap(), person); + } + + #[test] + fn derived_struct_conversion_builds_nested_error_path_while_unwinding() { + let address = StructData::from_values_unchecked( + Address::to_schema(), + vec![Scalar::from(7), Scalar::null(DataType::INTEGER)], + ); + let person = StructData::from_values_unchecked( + Person::to_schema(), + vec![ + Scalar::from(1), + Scalar::from(address), + Scalar::from(vec!["ace".to_string()]), + ], + ); + assert_result_error_with_message( + Person::try_from(person), + "address.city: expected String, found integer", + ); + } + + #[test] + fn container_conversion_adds_index_to_nested_error_path() { + let address = StructData::from_values_unchecked( + Address::to_schema(), + vec![Scalar::from(7), Scalar::null(DataType::INTEGER)], + ); + let array = ArrayData::try_new( + ArrayType::new(Address::to_schema(), false), + [Scalar::from(address)], + ) + .unwrap(); + assert_result_error_with_message( + Vec::

    ::try_from(Scalar::from(array)), + "[0].city: expected String, found integer", + ); + } + + #[test] + fn derived_struct_conversion_checks_null_field_data_type() { + // `Option::try_from` rejects a typed null whose data type does not match `T`. + let address = StructData::from_values_unchecked( + StructType::new_unchecked([ + StructField::not_null("city", DataType::STRING), + StructField::nullable("zip", DataType::STRING), + ]), + vec![Scalar::from("NYC"), Scalar::null(DataType::STRING)], + ); + assert_result_error_with_message( + Address::try_from(address), + "zip: expected integer, found string", + ); + } } diff --git a/kernel/src/expressions/sql.rs b/kernel/src/expressions/sql.rs new file mode 100644 index 0000000000..2cfae0d69d --- /dev/null +++ b/kernel/src/expressions/sql.rs @@ -0,0 +1,804 @@ +//! Parse a SQL string into a kernel [`Expression`]. +//! +//! Delta stores column defaults, check constraints, and generated column definitions as SQL +//! strings in table metadata. This module turns those strings into kernel [`Expression`] values +//! so the kernel can interpret them without depending on a full SQL parser. +//! +//! The grammar follows the Spark SQL standard: this parser implements a subset of Spark's SQL +//! grammar rather than defining a kernel-specific dialect, so the forms it accepts match what +//! Spark reads and writes. +//! +//! This is an intentionally light start: a small internal parser covering only the literal forms +//! Delta metadata contains today. If the supported SQL surface grows, options include moving +//! parsing behind the [`Engine`](crate::Engine) trait or adopting an existing SQL parser library. + +use crate::expressions::{Expression, Scalar}; +use crate::schema::{DataType, PrimitiveType}; +use crate::{DeltaResult, Error}; + +#[cfg(feature = "check-constraints-in-dev")] +mod parser; +#[cfg(feature = "check-constraints-in-dev")] +mod token; + +/// Parse a SQL string into an [`Expression`] that yields a value of the given [`DataType`] +/// (e.g. the type of the column whose default is being parsed). +/// +/// Leading and trailing whitespace are ignored. `NULL` (case-insensitive) is accepted for any +/// data type. All other input is parsed as a typed literal, which is supported only for primitive +/// types. +/// +/// # Examples +/// +/// The SQL comes from table metadata. A column declared `c DATE DEFAULT DATE '2024-01-01'` stores +/// the string `DATE '2024-01-01'` as its default, and `parse_sql("DATE '2024-01-01'", &DATE)` +/// parses it into `Expression::literal(Scalar::Date(..))`. The bare form `'2024-01-01'` is +/// equivalent; the `DATE` keyword is optional. +/// +/// # Errors +/// +/// Returns an error if the input is not a SQL form this parser accepts, or if the parsed value +/// is not compatible with `data_type` (incompatible type, out of range, etc.). +pub(crate) fn parse_sql(sql: &str, data_type: &DataType) -> DeltaResult { + let trimmed = sql.trim(); + if trimmed.is_empty() { + return Err(Error::generic("empty SQL literal")); + } + // NULL is valid for any data type, including non-primitive ones. + if trimmed.eq_ignore_ascii_case("null") { + return Ok(Expression::literal(Scalar::Null(data_type.clone()))); + } + // TODO(#2630): support SQL function calls (e.g. `current_date()`) when column defaults + // need them. + parse_literal(trimmed, data_type, sql) +} + +/// Dispatch a SQL literal to the per-type parser for its primitive `data_type`, then wrap the +/// resulting [`Scalar`] in an [`Expression`]. Errors on a non-primitive `data_type` (only `NULL`, +/// handled in [`parse_sql`], is valid for complex types). +/// +/// Typed-literal keywords (`DATE`, `TIMESTAMP`, `TIMESTAMP_LTZ`, `TIMESTAMP_NTZ`, `X`) and +/// `TRUE`/`FALSE` are case-insensitive. +fn parse_literal(trimmed: &str, data_type: &DataType, sql: &str) -> DeltaResult { + let DataType::Primitive(primitive) = data_type else { + return Err(Error::generic(format!( + "SQL literal parsing only supports primitive types, got {data_type:?}" + ))); + }; + let scalar = match primitive { + PrimitiveType::Binary => parse_binary_literal(trimmed)?, + PrimitiveType::String => parse_string_literal(trimmed)?, + PrimitiveType::Date => parse_date_literal(trimmed, sql)?, + PrimitiveType::Timestamp => parse_timestamp_ltz_literal(trimmed, sql)?, + PrimitiveType::TimestampNtz => parse_timestamp_ntz_literal(trimmed, sql)?, + PrimitiveType::Float | PrimitiveType::Double => { + parse_double_or_float(primitive, trimmed, sql)? + } + _ => primitive.parse_scalar(trimmed)?, + }; + Ok(Expression::literal(scalar)) +} + +/// Build a `Scalar::String` from a single-quoted body via [`unquote_string`] (e.g. `'it''s'` -> +/// `it's`). Bypasses `parse_scalar`, which maps an empty input to SQL NULL (partition-value +/// convention), so an empty literal `''` round-trips here as `Scalar::String("")`, distinct from +/// NULL. A backslash and double-quoted strings (`"foo"`) are rejected (see [`unquote_string`]). +fn parse_string_literal(trimmed: &str) -> DeltaResult { + Ok(Scalar::String(unquote_string(trimmed)?)) +} + +/// Build a `Scalar::Binary` from an `X'deadbeef'` literal (even number of hex digits) via +/// [`decode_binary_literal`]. Bypasses `parse_scalar` for the same empty-vs-NULL reason as +/// [`parse_string_literal`]. +fn parse_binary_literal(trimmed: &str) -> DeltaResult { + Ok(Scalar::Binary(decode_binary_literal(trimmed)?)) +} + +/// Parse a `Scalar::Date` from a trimmed string. +/// +/// Supported formats include `'2024-01-01'`, `DATE '2024-01-01'`, or `DATE'2024-01-01'`. The +/// `DATE` keyword is optional and may have 0 or more whitespace before the apostrophe. +fn parse_date_literal(trimmed: &str, sql: &str) -> DeltaResult { + let raw = unwrap_quoted_body(trimmed, &["DATE"], &PrimitiveType::Date, sql)?; + PrimitiveType::Date.parse_scalar(&raw) +} + +/// Parse a zoneless (wall-clock) `Scalar::TimestampNtz` from a trimmed string. +/// +/// Supported formats include `'2024-01-01 12:00:00[.fff]'` or `TIMESTAMP_NTZ '2024-01-01 +/// 12:00:00'`. The `TIMESTAMP_NTZ` keyword is optional and may have 0 or more whitespace before the +/// apostrophe; it must be `TIMESTAMP_NTZ`, not bare `TIMESTAMP` (which is LTZ). Carrying no zone, +/// it needs no UTC guard. +fn parse_timestamp_ntz_literal(trimmed: &str, sql: &str) -> DeltaResult { + let raw = unwrap_quoted_body( + trimmed, + &["TIMESTAMP_NTZ"], + &PrimitiveType::TimestampNtz, + sql, + )?; + PrimitiveType::TimestampNtz.parse_scalar(&raw) +} + +/// Parse a `Scalar::Timestamp` (local-time-zone) from a trimmed string in ISO 8601 / RFC 3339 form +/// with an explicit UTC `Z` suffix. +/// +/// Supported formats include `'1970-01-01T00:00:00.123Z'`, `TIMESTAMP '...Z'`, or +/// `TIMESTAMP_LTZ '...Z'`. The keyword is optional and may have 0 or more whitespace before the +/// apostrophe. Only the LTZ keywords are accepted: a mismatched keyword (e.g. an NTZ literal on an +/// LTZ column) carries different timezone semantics and must not be reused. `TIMESTAMP_LTZ` is an +/// explicit spelling of LTZ (== bare `TIMESTAMP`); both route through [`require_utc_z_suffix`]. +fn parse_timestamp_ltz_literal(trimmed: &str, sql: &str) -> DeltaResult { + let raw = unwrap_quoted_body( + trimmed, + &["TIMESTAMP", "TIMESTAMP_LTZ"], + &PrimitiveType::Timestamp, + sql, + )?; + require_utc_z_suffix(&raw, sql)?; + PrimitiveType::Timestamp.parse_scalar(&raw) +} + +/// Strip the typed-literal keyword prefix, if any, and return the inner literal value, unquoted and +/// trimmed: `DATE '2024-01-01'` and `DATE ' 2024-01-01 '` both return `2024-01-01`. +/// +/// An empty body is rejected: `parse_scalar` maps empty input to NULL (partition-value convention), +/// but an empty quoted body like `DATE ''` is invalid SQL, not NULL (`NULL` is accepted only as a +/// bare keyword, handled in [`parse_sql`]). +fn unwrap_quoted_body( + trimmed: &str, + keywords: &[&str], + primitive: &PrimitiveType, + sql: &str, +) -> DeltaResult { + let body = strip_typed_prefix_and_unquote(trimmed, keywords)?; + // Trim the inner body to match Spark's `stringToDate`/ `stringToTimestamp`. + let body = body.trim(); + if body.is_empty() { + return Err(Error::generic(format!( + "empty {primitive:?} literal: {sql}" + ))); + } + Ok(body.to_string()) +} + +/// Require a TIMESTAMP (LTZ) literal to pin an absolute instant with an explicit UTC `Z` suffix. +/// +/// TODO(#2733): `parse_scalar` only honors a `Z` suffix today, so until it reads explicit numeric +/// offsets we reject anything that is not UTC-`Z`: +/// - lowercase `t`/`z` (RFC 3339 permits, Spark rejects); +/// - zoneless literals (Spark resolves against the session timezone, unknown to kernel); +/// - numeric offsets (e.g. `+05:00`, even `+00:00`), which `parse_scalar` silently drops as UTC. +fn require_utc_z_suffix(raw: &str, sql: &str) -> DeltaResult<()> { + if raw.contains(['t', 'z']) { + return Err(Error::generic( + "TIMESTAMP literal must use uppercase 'T' and or 'Z'", + )); + } + if raw.ends_with('Z') { + return Ok(()); + } + // An explicit offset lives in the time part after the date/time separator (`T`/space). + let has_offset = raw + .split_once(['T', ' ']) + .is_some_and(|(_, time)| time.contains(['+', '-'])); + Err(if has_offset { + Error::generic(format!( + "TIMESTAMP literal with an explicit offset is not yet supported; use 'Z' (UTC): {sql}" + )) + } else { + Error::generic( + "zoneless TIMESTAMP literal is not yet supported; use an explicit 'Z' (UTC) suffix", + ) + }) +} + +/// Parse a bare FLOAT or DOUBLE literal, matching Spark's literal-typing + cast semantics: +/// - Reject a non-exponent literal whose implied precision exceeds 38 (see +/// [`exceeds_decimal_precision`]); only FLOAT/DOUBLE need this (Integer/Decimal already reject). +/// - For FLOAT, parse an exponent literal as f64 then narrow, matching Spark's DOUBLE-then-cast +/// (double rounding); a direct f32 parse single-rounds and can differ by 1 ULP. +/// - Reject non-finite results and fold a plain `-0.0` to `+0.0` (exponent forms keep their sign). +fn parse_double_or_float(primitive: &PrimitiveType, raw: &str, sql: &str) -> DeltaResult { + let has_exponent = raw.contains(['e', 'E']); + if !has_exponent && exceeds_decimal_precision(raw) { + return Err(Error::generic(format!( + "numeric literal exceeds maximum DECIMAL precision 38: {sql}" + ))); + } + let scalar = if *primitive == PrimitiveType::Float && has_exponent { + // f64 parse + `as f32` is the identical operation to Spark's Double.parseDouble + (float): + // both correctly-rounded decimal->f64, then IEEE round-to-nearest-even f64->f32. + let value: f64 = raw + .parse() + .map_err(|_| Error::generic(format!("invalid FLOAT literal: {sql}")))?; + Scalar::Float(value as f32) + } else { + primitive.parse_scalar(raw)? + }; + // Negative zero: `+ 0.0` folds a plain `-0.0` to `+0.0` (a no-op for every other value); + // exponent forms keep their sign, so skip them. + let normalize_neg_zero = !has_exponent; + let non_finite_error = || Error::generic("non-finite float literals are not supported"); + Ok(match scalar { + Scalar::Float(f) if !f.is_finite() => return Err(non_finite_error()), + Scalar::Double(d) if !d.is_finite() => return Err(non_finite_error()), + Scalar::Float(f) if normalize_neg_zero => Scalar::Float(f + 0.0), + Scalar::Double(d) if normalize_neg_zero => Scalar::Double(d + 0.0), + other => other, + }) +} + +/// Whether a bare non-exponent numeric literal exceeds Spark's DECIMAL precision cap of 38 and so +/// would be rejected by Spark's parser. +/// +/// The effective precision is `max(significant_digits, scale)`, where `significant_digits` +/// excludes only leading zeros (matching `java.math.BigDecimal.precision`, so embedded and +/// trailing zeros count) and `scale` is the fractional digit count. Callers must exclude exponent +/// forms, which Spark types as DOUBLE (no precision cap). +fn exceeds_decimal_precision(raw: &str) -> bool { + let unsigned = raw.strip_prefix(['+', '-']).unwrap_or(raw); + let scale = match unsigned.split_once('.') { + Some((_, frac)) => frac.chars().filter(|c| c.is_ascii_digit()).count(), + None => 0, + }; + let significant = unsigned + .chars() + .filter(|c| c.is_ascii_digit()) + .skip_while(|&c| c == '0') + .count(); + significant.max(scale) > 38 +} + +/// Unquote a SQL string literal: strip the surrounding single quotes and un-escape each `''` into +/// a single `'`. E.g. `'foo'` -> `foo`, `'it''s'` -> `it's`, `''''` -> `'`, `''` -> empty; `'''` +/// errors as unterminated (the `''` escapes a quote, leaving no closing quote). Errors if `input` +/// is not a properly terminated single-quoted string (missing closing quote or trailing chars), or +/// contains a backslash (Spark's `\n`/`\\` escapes are not yet supported). +fn unquote_string(input: &str) -> DeltaResult { + let body = input.strip_prefix('\'').ok_or_else(|| { + Error::generic(format!("expected a single-quoted SQL string, got: {input}")) + })?; + + // Walk the body after the opening quote. A `'` either escapes a literal quote (`''`) or + // closes the string; reaching the end without a closing quote is an unterminated literal. + let mut out = String::with_capacity(body.len()); + let mut chars = body.chars(); + while let Some(c) = chars.next() { + if c == '\\' { + return Err(Error::generic(format!( + "backslash escapes in SQL string literals are not yet supported: {input}" + ))); + } + if c != '\'' { + out.push(c); + continue; + } + match chars.next() { + None => return Ok(out), + Some('\'') => out.push('\''), + Some(_) => { + return Err(Error::generic(format!( + "unexpected characters after closing quote in SQL string literal: {input}" + ))) + } + } + } + Err(Error::generic(format!( + "unterminated SQL string literal: {input}" + ))) +} + +/// Strip an optional typed-literal keyword prefix (e.g. `DATE`, `TIMESTAMP`, `TIMESTAMP_NTZ`) and +/// unwrap the required `'...'` quoted body, returning the inner value without its surrounding +/// quotes: `DATE '2024-01-01'`, `DATE'2024-01-01'`, and bare `'2024-01-01'` all return +/// `2024-01-01`. Any of `keywords` may appear as the prefix (case-insensitive), separated from +/// the quote by optional whitespace. +/// +/// Unlike [`unwrap_quoted_body`], the inner body is returned verbatim, not trimmed: `DATE +/// ' 2024-01-01 '` returns ` 2024-01-01 ` (interior whitespace preserved). +fn strip_typed_prefix_and_unquote(input: &str, keywords: &[&str]) -> DeltaResult { + // Match a keyword only as a complete token: it must be followed by the opening quote or + // whitespace, never more identifier characters (so `DATEX '..'` is not read as `DATE`). + let body = keywords.iter().find_map(|kw| { + let prefix = input.get(..kw.len())?; + let rest = &input[kw.len()..]; + let is_token = rest.starts_with('\'') || rest.starts_with(char::is_whitespace); + (prefix.eq_ignore_ascii_case(kw) && is_token).then(|| rest.trim_start()) + }); + unquote_string(body.unwrap_or(input)) +} + +/// Decode a `X'hex'` SQL binary literal into a byte vector. The leading `X` is case-insensitive; +/// the body must be an even-length sequence of hex digits. +fn decode_binary_literal(input: &str) -> DeltaResult> { + let err = || { + Error::generic(format!( + "expected a SQL binary literal like X'..', got: {input}" + )) + }; + let hex = input + .strip_prefix(['x', 'X']) + .and_then(|rest| rest.strip_prefix('\'')) + .and_then(|rest| rest.strip_suffix('\'')) + .ok_or_else(err)?; + if !hex.len().is_multiple_of(2) { + return Err(Error::generic(format!( + "binary literal must contain an even number of hex digits: {input}" + ))); + } + hex.as_bytes() + .chunks_exact(2) + .map(|pair| { + let hi = (pair[0] as char) + .to_digit(16) + .ok_or_else(|| Error::generic(format!("invalid hex digit in {input}")))?; + let lo = (pair[1] as char) + .to_digit(16) + .ok_or_else(|| Error::generic(format!("invalid hex digit in {input}")))?; + Ok((hi << 4 | lo) as u8) + }) + .collect() +} + +#[cfg(test)] +mod tests { + use chrono::{DateTime, NaiveDate, NaiveDateTime, TimeZone, Utc}; + use rstest::rstest; + + use super::*; + use crate::expressions::{DecimalData, Expression}; + use crate::schema::{ArrayType, DataType, DecimalType, MapType, StructField}; + + fn date_days(year: i32, month: u32, day: u32) -> i32 { + let nd = NaiveDate::from_ymd_opt(year, month, day) + .unwrap() + .and_hms_opt(0, 0, 0) + .unwrap(); + Utc.from_utc_datetime(&nd) + .signed_duration_since(DateTime::UNIX_EPOCH) + .num_days() as i32 + } + + fn ts_micros(s: &str) -> i64 { + let ndt = NaiveDateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S%.f").unwrap(); + Utc.from_utc_datetime(&ndt) + .signed_duration_since(DateTime::UNIX_EPOCH) + .num_microseconds() + .unwrap() + } + + fn decimal_type(precision: u8, scale: u8) -> DataType { + DataType::Primitive(PrimitiveType::Decimal( + DecimalType::try_new(precision, scale).unwrap(), + )) + } + + #[rstest] + #[case("42", DataType::INTEGER, Scalar::Integer(42))] + #[case(" -7 ", DataType::INTEGER, Scalar::Integer(-7))] + #[case("+5", DataType::INTEGER, Scalar::Integer(5))] + #[case("127", DataType::BYTE, Scalar::Byte(127))] + #[case("-32768", DataType::SHORT, Scalar::Short(i16::MIN))] + #[case("9223372036854775807", DataType::LONG, Scalar::Long(i64::MAX))] + #[case("2.5", DataType::DOUBLE, Scalar::Double(2.5))] + #[case("0.5", DataType::FLOAT, Scalar::Float(0.5))] + #[case("1.5E3", DataType::DOUBLE, Scalar::Double(1500.0))] + #[case("1.5E3", DataType::FLOAT, Scalar::Float(1500.0))] + #[case("TRUE", DataType::BOOLEAN, Scalar::Boolean(true))] + #[case("false", DataType::BOOLEAN, Scalar::Boolean(false))] + #[case("'hello'", DataType::STRING, Scalar::String("hello".into()))] + #[case("' hi '", DataType::STRING, Scalar::String(" hi ".into()))] // strings are not trimmed + #[case("''", DataType::STRING, Scalar::String(String::new()))] + #[case("'it''s'", DataType::STRING, Scalar::String("it's".into()))] + #[case("'a''b''c'", DataType::STRING, Scalar::String("a'b'c".into()))] + #[case("'''hello'", DataType::STRING, Scalar::String("'hello".into()))] + #[case("'hello'''", DataType::STRING, Scalar::String("hello'".into()))] + #[case("'''bad'''", DataType::STRING, Scalar::String("'bad'".into()))] + #[case( + "1.23", + decimal_type(5, 2), + Scalar::Decimal(DecimalData::try_new(123, DecimalType::try_new(5, 2).unwrap()).unwrap()), + )] + #[case( + "-12345.67", + decimal_type(10, 2), + Scalar::Decimal( + DecimalData::try_new(-1234567, DecimalType::try_new(10, 2).unwrap()).unwrap() + ), + )] + fn parses_basic_literals(#[case] sql: &str, #[case] ty: DataType, #[case] expected: Scalar) { + let got = parse_sql(sql, &ty).unwrap(); + assert_eq!(got, Expression::literal(expected)); + } + + #[rstest] + #[case("'2024-01-01'", date_days(2024, 1, 1))] + #[case("DATE '2024-01-01'", date_days(2024, 1, 1))] + #[case("DATE'2024-01-01'", date_days(2024, 1, 1))] // keyword butted against the quote + #[case("date '1970-01-02'", date_days(1970, 1, 2))] + #[case("' 2024-01-01 '", date_days(2024, 1, 1))] // body is trimmed, matching Spark + #[case("DATE ' 2024-01-01 '", date_days(2024, 1, 1))] + fn parses_date_literals(#[case] sql: &str, #[case] expected_days: i32) { + let got = parse_sql(sql, &DataType::DATE).unwrap(); + assert_eq!(got, Expression::literal(Scalar::Date(expected_days))); + } + + #[rstest] + #[case("'2024-01-01 12:34:56'", "2024-01-01 12:34:56")] + #[case("TIMESTAMP_NTZ '2024-01-01 12:34:56'", "2024-01-01 12:34:56")] + #[case("TIMESTAMP_NTZ'2024-01-01 12:34:56'", "2024-01-01 12:34:56")] // keyword butted against quote + #[case("timestamp_ntz '2024-01-01 12:34:56.789'", "2024-01-01 12:34:56.789")] + #[case("' 2024-01-01 12:34:56 '", "2024-01-01 12:34:56")] // body is trimmed, matching Spark + fn parses_zoneless_timestamp_ntz_literals(#[case] sql: &str, #[case] equivalent: &str) { + let got = parse_sql(sql, &DataType::TIMESTAMP_NTZ).unwrap(); + assert_eq!( + got, + Expression::literal(Scalar::TimestampNtz(ts_micros(equivalent))) + ); + } + + #[rstest] + #[case("'2024-01-01 12:34:56'")] // zoneless + #[case("TIMESTAMP '2024-01-01 12:34:56.789'")] // zoneless, with keyword + #[case("TIMESTAMP'2024-01-01 12:34:56'")] // zoneless, keyword butted against quote + #[case("TIMESTAMP_LTZ '2024-01-01 12:34:56'")] // zoneless, explicit LTZ keyword + #[case("' 2024-01-01 12:34:56 '")] // zoneless, padded + #[case("'2024-06-15T14:30:00+05:00'")] // offset (parse_scalar drops it, see TODO(#2733)) + #[case("TIMESTAMP '2024-06-15T14:30:00-05:00'")] // offset + #[case("'2024-06-15T14:30:00+00:00'")] // +00:00 is UTC-valued but dropped, so still rejected + fn rejects_zoneless_and_offset_timestamp_ltz(#[case] sql: &str) { + let result = parse_sql(sql, &DataType::TIMESTAMP); + assert!( + result.is_err(), + "expected error for zoneless/offset TIMESTAMP {sql:?}, got {result:?}" + ); + } + + // The two LTZ rejections carry distinct messages; assert on the message so the + // zoneless-vs-offset split stays pinned (both otherwise just return an error). + #[rstest] + #[case("'2024-01-01 12:34:56'", "zoneless")] + #[case("TIMESTAMP '2024-01-01 12:34:56'", "zoneless")] + #[case("'2024-06-15T14:30:00+05:00'", "offset")] + #[case("'2024-06-15T14:30:00+00:00'", "offset")] // +00:00 is UTC-valued but dropped, so rejected + fn timestamp_ltz_rejection_distinguishes_zoneless_from_offset( + #[case] sql: &str, + #[case] needle: &str, + ) { + let err = parse_sql(sql, &DataType::TIMESTAMP) + .unwrap_err() + .to_string(); + assert!( + err.contains(needle), + "{sql:?} message missing {needle:?}: {err}" + ); + } + + #[rstest] + #[case("'2024-01-01t12:00:00Z'")] // lowercase t separator + #[case("'2024-01-01T12:00:00z'")] // lowercase z designator + #[case("'2024-01-01t12:00:00z'")] // both lowercase + #[case("TIMESTAMP '2024-01-01t12:00:00Z'")] // lowercase t, with keyword + #[case("TIMESTAMP_LTZ '2024-01-01T12:00:00z'")] // lowercase z, explicit LTZ keyword + fn rejects_lowercase_timestamp_separator_and_zone(#[case] sql: &str) { + let err = parse_sql(sql, &DataType::TIMESTAMP) + .unwrap_err() + .to_string(); + assert!( + err.contains("uppercase"), + "{sql:?} should be rejected for lowercase 't'/'z': {err}" + ); + } + + #[rstest] + #[case("TIMESTAMP_NTZ '1970-01-01T00:00:00Z'", DataType::TIMESTAMP)] // NTZ keyword, LTZ target + #[case("TIMESTAMP '2024-01-01 12:34:56'", DataType::TIMESTAMP_NTZ)] // LTZ keyword, NTZ target + #[case("TIMESTAMP_LTZ '1970-01-01T00:00:00Z'", DataType::TIMESTAMP_NTZ)] // LTZ keyword, NTZ target + fn rejects_mismatched_timestamp_keyword(#[case] sql: &str, #[case] ty: DataType) { + let result = parse_sql(sql, &ty); + assert!( + result.is_err(), + "expected error for mismatched timestamp keyword {sql:?} as {ty:?}, got {result:?}" + ); + } + + // `PrimitiveType::parse_scalar` falls back to ISO 8601 / RFC 3339 (`%+`) only for the + // `Timestamp` variant; `TimestampNtz` has no such fallback and rejects ISO form. Pin the + // asymmetry so a future refactor that unifies the two paths gets caught. + #[rstest] + #[case("'1970-01-01T00:00:00.123Z'", "1970-01-01 00:00:00.123")] + #[case("'2024-06-15T14:30:00Z'", "2024-06-15 14:30:00")] + #[case("TIMESTAMP '2024-06-15T14:30:00.456Z'", "2024-06-15 14:30:00.456")] + #[case("TIMESTAMP_LTZ '2024-06-15T14:30:00Z'", "2024-06-15 14:30:00")] // explicit LTZ spelling + #[case("TIMESTAMP_LTZ'1970-01-01T00:00:00.123Z'", "1970-01-01 00:00:00.123")] // butted against quote + fn iso_8601_form_accepted_only_for_timestamp(#[case] sql: &str, #[case] equivalent: &str) { + let got = parse_sql(sql, &DataType::TIMESTAMP).unwrap(); + assert_eq!( + got, + Expression::literal(Scalar::Timestamp(ts_micros(equivalent))) + ); + parse_sql(sql, &DataType::TIMESTAMP_NTZ).unwrap_err(); + } + + #[rstest] + #[case("X''", vec![])] + #[case("X'00'", vec![0x00])] + #[case("X'DeAdBeEf'", vec![0xde, 0xad, 0xbe, 0xef])] + #[case("x'01ff'", vec![0x01, 0xff])] + fn parses_binary_literals(#[case] sql: &str, #[case] expected: Vec) { + let got = parse_sql(sql, &DataType::BINARY).unwrap(); + assert_eq!(got, Expression::literal(Scalar::Binary(expected))); + } + + #[rstest] + #[case(DataType::INTEGER)] + #[case(DataType::STRING)] + #[case(DataType::BOOLEAN)] + #[case(DataType::DATE)] + #[case(DataType::BINARY)] + fn null_is_accepted_for_any_primitive(#[case] ty: DataType) { + let got = parse_sql("NULL", &ty).unwrap(); + assert_eq!(got, Expression::literal(Scalar::Null(ty.clone()))); + // also case-insensitive + let got_lower = parse_sql(" null ", &ty).unwrap(); + assert_eq!(got_lower, Expression::literal(Scalar::Null(ty))); + } + + /// As the parser grows new capabilities (typed numeric suffixes, CAST, foldable + /// function calls), cases should be removed from this list and moved into the positive tests + /// above. + #[rstest] + #[case("1L", DataType::LONG)] + #[case("1.23BD", decimal_type(5, 2))] + #[case("1.5F", DataType::FLOAT)] + #[case("CAST('2024-01-01' AS DATE)", DataType::DATE)] + #[case("CAST(NULL AS INT)", DataType::INTEGER)] + #[case("current_date()", DataType::DATE)] + #[case("current_timestamp()", DataType::TIMESTAMP)] + #[case("now()", DataType::TIMESTAMP)] + #[case("1 + 1", DataType::INTEGER)] + #[case("concat('a', 'b')", DataType::STRING)] + #[case("0", decimal_type(10, 2))] // parse_scalar requires the literal's scale to match exactly + #[case("1.2", decimal_type(5, 2))] + fn currently_unsupported_valid_sql(#[case] sql: &str, #[case] ty: DataType) { + let result = parse_sql(sql, &ty); + assert!( + result.is_err(), + "expected error for currently-unsupported SQL {sql:?} as {ty:?}, got {result:?}" + ); + } + + #[rstest] + #[case("", DataType::INTEGER)] + #[case(" ", DataType::INTEGER)] + #[case("'42'", DataType::INTEGER)] // quoted number for int + #[case("+", DataType::INTEGER)] // lone sign + #[case("UnknownFn()", DataType::INTEGER)] // function call + #[case("42", DataType::STRING)] // unquoted number for string + #[case("foo", DataType::STRING)] // unquoted string + #[case("'unterminated", DataType::STRING)] + #[case("'bad'quote'", DataType::STRING)] // characters after the closing quote + #[case("'''", DataType::STRING)] // unterminated: odd number of quotes + #[case("'''''", DataType::STRING)] // unterminated: odd number of quotes + #[case("'ab''", DataType::STRING)] // unterminated: trailing escaped quote, no close + #[case("nope", DataType::BOOLEAN)] + #[case("'TRUE'", DataType::BOOLEAN)] // quoted boolean + #[case("'2024-13-01'", DataType::DATE)] // bad month + #[case("not-a-date", DataType::DATE)] + #[case("''", DataType::DATE)] // empty quoted body must not parse as NULL + #[case("DATE ''", DataType::DATE)] + #[case("' '", DataType::DATE)] // whitespace-only body trims to empty + #[case("DATE ' '", DataType::DATE)] + #[case("DATEX '2024-01-01'", DataType::DATE)] // extra chars after keyword, not a token + #[case("DATEX'2024-01-01'", DataType::DATE)] // ditto, no space + #[case("''", DataType::TIMESTAMP)] + #[case("' '", DataType::TIMESTAMP)] + #[case("TIMESTAMP ''", DataType::TIMESTAMP)] + #[case("TIMESTAMPX '2024-01-01 12:34:56'", DataType::TIMESTAMP)] // extra chars after keyword + #[case("TIMESTAMP_NTZ ''", DataType::TIMESTAMP_NTZ)] + #[case("timestamp_ntza'2024-01-01 12:34:56'", DataType::TIMESTAMP_NTZ)] // extra chars, no space + #[case(" now() ", DataType::TIMESTAMP)] // function call with padding + #[case("X'0'", DataType::BINARY)] // odd number of hex digits + #[case("X'gg'", DataType::BINARY)] // non-hex chars + #[case("'deadbeef'", DataType::BINARY)] // missing X prefix + #[case("128", DataType::BYTE)] // out of range for i8 + #[case("2147483648", DataType::INTEGER)] // out of range for i32 + fn rejects_invalid_input(#[case] sql: &str, #[case] ty: DataType) { + let result = parse_sql(sql, &ty); + assert!( + result.is_err(), + "expected error for {sql:?} as {ty:?}, got {result:?}" + ); + } + + #[rstest] + fn rejects_bare_non_finite_floats( + #[values( + "NaN", + "nan", + "Infinity", + "infinity", + "inf", + "-inf", + "+inf", + "-Infinity", + "1e999", // overflows to infinity + "-1e999" + )] + sql: &str, + #[values(DataType::FLOAT, DataType::DOUBLE)] ty: DataType, + ) { + let result = parse_sql(sql, &ty); + assert!( + result.is_err(), + "expected error for bare non-finite literal {sql:?} as {ty:?}, got {result:?}" + ); + } + + // A plain `-0.0` is a Spark decimal that folds to `+0.0`, so the parser normalizes the sign. + // `assert_eq` on `Scalar` can't see it (`-0.0 == 0.0`), so check the sign bit directly. + #[rstest] + #[case("-0.0")] + #[case("-0")] + #[case("-0.00")] + fn normalizes_negative_zero_to_positive(#[case] sql: &str) { + let Expression::Literal(Scalar::Double(d)) = parse_sql(sql, &DataType::DOUBLE).unwrap() + else { + panic!("expected a Double literal for {sql:?}"); + }; + assert!( + d == 0.0 && d.is_sign_positive(), + "DOUBLE {sql:?} kept the sign: {d}" + ); + + let Expression::Literal(Scalar::Float(f)) = parse_sql(sql, &DataType::FLOAT).unwrap() + else { + panic!("expected a Float literal for {sql:?}"); + }; + assert!( + f == 0.0 && f.is_sign_positive(), + "FLOAT {sql:?} kept the sign: {f}" + ); + } + + // An exponent literal like `-0.0E0` is a Spark double literal that keeps its sign, so the + // parser must not normalize it. + #[rstest] + #[case("-0.0E0")] + #[case("-0E0")] + #[case("-0.0e10")] + fn preserves_negative_zero_with_exponent(#[case] sql: &str) { + let Expression::Literal(Scalar::Double(d)) = parse_sql(sql, &DataType::DOUBLE).unwrap() + else { + panic!("expected a Double literal for {sql:?}"); + }; + assert!( + d == 0.0 && d.is_sign_negative(), + "DOUBLE {sql:?} should keep the negative sign: {d}" + ); + + let Expression::Literal(Scalar::Float(f)) = parse_sql(sql, &DataType::FLOAT).unwrap() + else { + panic!("expected a Float literal for {sql:?}"); + }; + assert!( + f == 0.0 && f.is_sign_negative(), + "FLOAT {sql:?} should keep the negative sign: {f}" + ); + } + + // `7.038531E-26` is a Spark DOUBLE literal narrowed to FLOAT via a cast (decimal->f64->f32, + // double rounding). Parsing the string straight to f32 single-rounds and lands 1 ULP off, so + // the parser must route FLOAT exponent literals through f64 first to match Spark exactly. + #[test] + fn float_exponent_literal_double_rounds_to_match_spark() { + let got = parse_sql("7.038531E-26", &DataType::FLOAT).unwrap(); + let spark = "7.038531E-26".parse::().unwrap() as f32; + assert_eq!(got, Expression::literal(Scalar::Float(spark))); + assert_eq!(spark.to_bits(), 0x15ae_43fe); + // The value a direct decimal->f32 parse (single rounding) would have produced -- 1 ULP off. + assert_ne!( + spark.to_bits(), + "7.038531E-26".parse::().unwrap().to_bits() + ); + } + + // A bare non-exponent literal whose implied DECIMAL precision (`max(significant_digits, + // scale)`) exceeds 38 is a DECIMAL literal that Spark rejects at parse time. Only + // FLOAT/DOUBLE need an explicit guard (integer targets overflow, Decimal targets fail the + // column-precision check). + #[rstest] + #[case(DataType::DOUBLE)] + #[case(DataType::FLOAT)] + fn rejects_numeric_literal_over_decimal_precision_38(#[case] ty: DataType) { + let thirty_nine_fraction = format!("0.{}", "1".repeat(39)); // sig 39, scale 39 + let forty_digit_integer = "1".repeat(40); // sig 40, scale 0 + let scale_39_one_sig = format!("0.{}1", "0".repeat(38)); // sig 1, scale 39 + for sql in [ + &thirty_nine_fraction, + &forty_digit_integer, + &scale_39_one_sig, + ] { + let err = parse_sql(sql, &ty).unwrap_err().to_string(); + assert!( + err.contains("precision"), + "{sql:?} as {ty:?} not rejected: {err}" + ); + } + } + + // Precision exactly 38 is accepted. Only leading zeros are dropped from the significant-digit + // count, so embedded (`.10101010101`) and trailing (`1.230`) zeros stay significant. + #[rstest] + #[case(DataType::DOUBLE)] + #[case(DataType::FLOAT)] + fn accepts_numeric_literal_at_decimal_precision_boundary(#[case] ty: DataType) { + let thirty_eight_fraction = format!("0.{}", "1".repeat(38)); // sig 38, scale 38 + let thirty_eight_integer = "1".repeat(38); // sig 38, scale 0 + for sql in [ + thirty_eight_fraction.as_str(), + thirty_eight_integer.as_str(), + ".10101010101", // embedded zeros count: sig 11, scale 11 + "1.230", // trailing zero counts: sig 4, scale 3 + ] { + parse_sql(sql, &ty).unwrap_or_else(|e| panic!("{sql:?} as {ty:?} rejected: {e}")); + } + } + + #[rstest] + #[case(r"'a\nb'")] // would-be newline escape + #[case(r"'c:\temp'")] // literal backslash in a path + #[case(r"'\\'")] // doubled backslash + fn rejects_backslash_in_string_literal(#[case] sql: &str) { + let result = parse_sql(sql, &DataType::STRING); + assert!( + result.is_err(), + "expected error for backslash in string literal {sql:?}, got {result:?}" + ); + } + + // Spark accepts double-quoted strings; kernel does not yet. + #[rstest] + #[case(r#""foo""#)] + #[case(r#""it's""#)] // embedded single quote + fn rejects_double_quoted_string(#[case] sql: &str) { + let result = parse_sql(sql, &DataType::STRING); + assert!( + result.is_err(), + "expected error for double-quoted string {sql:?}, got {result:?}" + ); + } + + fn struct_ty() -> DataType { + DataType::try_struct_type([StructField::nullable("a", DataType::INTEGER)]).unwrap() + } + + fn array_ty() -> DataType { + DataType::Array(Box::new(ArrayType::new(DataType::INTEGER, true))) + } + + fn map_ty() -> DataType { + DataType::Map(Box::new(MapType::new( + DataType::STRING, + DataType::INTEGER, + true, + ))) + } + + #[rstest] + #[case::struct_target(struct_ty())] + #[case::array_target(array_ty())] + #[case::map_target(map_ty())] + fn rejects_non_primitive_target(#[case] ty: DataType) { + assert!(parse_sql("'foo'", &ty).is_err()); + } + + #[rstest] + #[case::struct_target(struct_ty())] + #[case::array_target(array_ty())] + #[case::map_target(map_ty())] + fn null_is_accepted_for_non_primitive_target(#[case] ty: DataType) { + let got = parse_sql("NULL", &ty).unwrap(); + assert_eq!(got, Expression::literal(Scalar::Null(ty))); + } +} diff --git a/kernel/src/expressions/sql/parser.rs b/kernel/src/expressions/sql/parser.rs new file mode 100644 index 0000000000..ba1bce2658 --- /dev/null +++ b/kernel/src/expressions/sql/parser.rs @@ -0,0 +1,291 @@ +//! Parser for a single SQL comparison. +//! +//! Consumes the [`Token`] stream from [`super::token`] and produces a [`Comparison`]: exactly one +//! `operand operand`. For now the grammar covers only a single comparison; junctions +//! (`AND`/`OR`/`NOT`), parentheses, and `IS [NOT] NULL` are not yet supported and surface as +//! errors. A later stage resolves the operands against a schema. + +// WIP feature behind `check-constraints-in-dev`; some items have no caller until enforcement lands. +// TODO(#2896): remove this allow once check-constraint enforcement wires up a caller. +#![allow(dead_code)] + +use std::iter::Peekable; +use std::vec; + +use super::token::Token; +use crate::expressions::ColumnName; +use crate::{DeltaResult, Error}; + +/// An operand of a comparison: a column reference (its as-written path) or a literal (raw source +/// text, e.g. `42`, `'foo'`, `NULL`). +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) enum Operand { + Column(ColumnName), + Literal(String), +} + +/// A comparison operator. Kernel has no native `<=`/`>=`/`!=` predicate op; lowering composes them +/// from the primitives it does have (`<`/`>`/`=`), e.g. `le` is `not(gt)`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) enum CmpOp { + Lt, + Le, + Gt, + Ge, + Eq, + Ne, + /// `<=>`, null-safe equality; lowering maps it to `not(distinct(..))`. + NullSafeEq, +} + +/// A single parsed comparison `left right`. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) struct Comparison { + pub(super) op: CmpOp, + pub(super) left: Operand, + pub(super) right: Operand, +} + +/// Parse a token stream into a single [`Comparison`]. The grammar has no junctions, so any tokens +/// left over after one comparison are an error. +pub(super) fn parse(tokens: Vec) -> DeltaResult { + let mut parser = Parser { + tokens: tokens.into_iter().peekable(), + }; + let comparison = parser.parse_comparison()?; + if parser.advance().is_some() { + return Err(Error::generic( + "unexpected trailing input: only a single comparison is supported", + )); + } + Ok(comparison) +} + +struct Parser { + tokens: Peekable>, +} + +impl Parser { + fn peek(&mut self) -> Option<&Token> { + self.tokens.peek() + } + + fn advance(&mut self) -> Option { + self.tokens.next() + } + + // comparison := operand cmp operand + fn parse_comparison(&mut self) -> DeltaResult { + let left = self.parse_operand()?; + let op = self.parse_cmp_op()?; + let right = self.parse_operand()?; + Ok(Comparison { op, left, right }) + } + + fn parse_cmp_op(&mut self) -> DeltaResult { + let op = match self.advance() { + Some(Token::Lt) => CmpOp::Lt, + Some(Token::Le) => CmpOp::Le, + Some(Token::Gt) => CmpOp::Gt, + Some(Token::Ge) => CmpOp::Ge, + Some(Token::Eq) => CmpOp::Eq, + Some(Token::Ne) => CmpOp::Ne, + Some(Token::NullSafeEq) => CmpOp::NullSafeEq, + other => return Err(expected("a comparison operator", other)), + }; + Ok(op) + } + + // operand := ('+' | '-')? Number | Literal | Ident ( '.' Ident )* + // + // A leading sign applies only to a number (the tokenizer emits it as a separate Plus/Minus + // operator): the sign is folded back into the number's raw text so the later lowering stage + // types it, e.g. `-234` -> Literal("-234"). A sign before anything else (a column, string/date + // literal, keyword) is not a single-comparison operand and errors: unary minus on a column and + // binary arithmetic are out of grammar. + fn parse_operand(&mut self) -> DeltaResult { + match self.advance() { + Some(sign @ (Token::Plus | Token::Minus)) => { + let sign = if sign == Token::Minus { '-' } else { '+' }; + match self.advance() { + Some(Token::Number(raw)) => Ok(Operand::Literal(format!("{sign}{raw}"))), + other => Err(expected(&format!("a number after '{sign}'"), other)), + } + } + Some(Token::Number(raw)) | Some(Token::Literal(raw)) => Ok(Operand::Literal(raw)), + Some(Token::Ident(first)) => self.parse_column_path(first), + other => Err(expected("a column or literal", other)), + } + } + + // Continue a dotted column path after its first `Ident`: `( '.' Ident )*`. + fn parse_column_path(&mut self, first: String) -> DeltaResult { + let mut path = vec![first]; + while self.peek() == Some(&Token::Dot) { + self.advance(); + match self.advance() { + Some(Token::Ident(segment)) => path.push(segment), + other => return Err(expected("an identifier after '.'", other)), + } + } + Ok(Operand::Column(ColumnName::new(path))) + } +} + +/// Build the `expected {what}, found {found:?}` error shared by the parser's operand and operator +/// sites. +fn expected(what: &str, found: Option) -> Error { + Error::generic(format!("expected {what}, found {found:?}")) +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::super::token::{tokenize, Keyword, Token}; + use super::{parse, CmpOp, Comparison, Operand}; + use crate::expressions::ColumnName; + + fn ident(s: &str) -> Token { + Token::Ident(s.to_string()) + } + + fn col(segments: &[&str]) -> Operand { + Operand::Column(ColumnName::new(segments.iter().map(|s| s.to_string()))) + } + + fn lit(s: &str) -> Operand { + Operand::Literal(s.to_string()) + } + + /// Each operator token maps to its `CmpOp`, with the two operands preserved in order. + #[rstest] + #[case(Token::Lt, CmpOp::Lt)] + #[case(Token::Le, CmpOp::Le)] + #[case(Token::Gt, CmpOp::Gt)] + #[case(Token::Ge, CmpOp::Ge)] + #[case(Token::Eq, CmpOp::Eq)] + #[case(Token::Ne, CmpOp::Ne)] + #[case(Token::NullSafeEq, CmpOp::NullSafeEq)] + fn parses_each_operator(#[case] tok: Token, #[case] expected: CmpOp) { + let got = parse(vec![ident("a"), tok, Token::Number("1".into())]).unwrap(); + assert_eq!( + got, + Comparison { + op: expected, + left: col(&["a"]), + right: Operand::Literal("1".into()), + } + ); + } + + /// Operands may be column-column, column-literal, or literal-column, and a column may be a + /// dotted path. + #[rstest] + #[case(vec![ident("a"), Token::Gt, ident("b")], col(&["a"]), col(&["b"]))] + #[case( + vec![Token::Number("0".into()), Token::Lt, ident("a")], + Operand::Literal("0".into()), + col(&["a"]) + )] + #[case( + vec![ident("a"), Token::Dot, ident("b"), Token::Eq, Token::Number("1".into())], + col(&["a", "b"]), + Operand::Literal("1".into()) + )] + // A three-segment path exercises loop re-entry in `parse_column_path` (two `.` iterations). + #[case( + vec![ + ident("a"), Token::Dot, ident("b"), Token::Dot, ident("c"), + Token::Eq, Token::Number("1".into()), + ], + col(&["a", "b", "c"]), + Operand::Literal("1".into()) + )] + #[case( + vec![ident("a"), Token::Gt, Token::Minus, Token::Number("234".into())], + col(&["a"]), + Operand::Literal("-234".into()) + )] + #[case( + vec![Token::Plus, Token::Number("5".into()), Token::Lt, ident("a")], + Operand::Literal("+5".into()), + col(&["a"]) + )] + // A sign folds into a leading-dot decimal too (`.5`). + #[case( + vec![ident("a"), Token::Gt, Token::Minus, Token::Number(".5".into())], + col(&["a"]), + Operand::Literal("-.5".into()) + )] + fn parses_operand_shapes( + #[case] tokens: Vec, + #[case] left: Operand, + #[case] right: Operand, + ) { + let got = parse(tokens).unwrap(); + assert_eq!(got.left, left); + assert_eq!(got.right, right); + } + + /// Run a raw constraint string through the real tokenizer and into the parser, the way a caller + /// consumes it. The other tests hand-build `Token` vectors, so they cannot catch a mismatch + /// between what the tokenizer emits and what the parser expects; these end-to-end cases do. + /// `a > -.5` in particular exercises the tokenizer's leading-dot decimal plus the parser's + /// sign reassembly (`Minus` + `Number(".5")` -> `Literal("-.5")`). + #[rstest] + #[case("a > -.5", CmpOp::Gt, col(&["a"]), lit("-.5"))] + #[case("amount >= -234", CmpOp::Ge, col(&["amount"]), lit("-234"))] + #[case("ratio == .25", CmpOp::Eq, col(&["ratio"]), lit(".25"))] + #[case("a.b.c <= 100", CmpOp::Le, col(&["a", "b", "c"]), lit("100"))] + #[case("x != y", CmpOp::Ne, col(&["x"]), col(&["y"]))] + #[case("n <=> NULL", CmpOp::NullSafeEq, col(&["n"]), lit("NULL"))] + #[case("+5 < a", CmpOp::Lt, lit("+5"), col(&["a"]))] + #[case("status = 'active'", CmpOp::Eq, col(&["status"]), lit("'active'"))] + fn tokenize_then_parse_yields_expected_comparison( + #[case] sql: &str, + #[case] op: CmpOp, + #[case] left: Operand, + #[case] right: Operand, + ) { + let got = parse(tokenize(sql).unwrap()).unwrap(); + assert_eq!(got, Comparison { op, left, right }); + } + + /// End-to-end rejections: inputs that fail the tokenizer (`(`) or parse to something outside a + /// single comparison (junctions, `IS [NOT] NULL`, a bare boolean column) both surface as an + /// error through the real pipeline. + #[rstest] + #[case::junction_and("a > 0 AND b < 9")] + #[case::junction_or("a > 0 OR b > 0")] + #[case::is_not_null("a IS NOT NULL")] + #[case::parens("(amount > 0)")] + #[case::bare_bool_column("is_active")] + fn tokenize_then_parse_rejects_non_single_comparison(#[case] sql: &str) { + assert!(tokenize(sql).and_then(parse).is_err()); + } + + /// Anything outside a single `operand op operand` is rejected. + #[rstest] + #[case::trailing(vec![ident("a"), Token::Gt, Token::Number("0".into()), ident("extra")])] + #[case::missing_operator(vec![ident("a"), ident("b")])] + #[case::dangling_dot(vec![ident("a"), Token::Dot, Token::Gt, Token::Number("0".into())])] + #[case::operator_first(vec![Token::Gt, Token::Number("0".into())])] + #[case::empty(vec![])] + #[case::keyword_operand(vec![ident("a"), Token::Gt, Token::Keyword(Keyword::And)])] + #[case::keyword_junction(vec![ + ident("a"), Token::Gt, Token::Number("0".into()), + Token::Keyword(Keyword::And), ident("b"), Token::Lt, Token::Number("9".into()), + ])] + #[case::sign_before_column(vec![Token::Minus, ident("a"), Token::Gt, Token::Number("0".into())])] + #[case::sign_before_string(vec![ + ident("a"), Token::Eq, Token::Minus, Token::Literal("'foo'".into()), + ])] + #[case::sign_then_eof(vec![ident("a"), Token::Gt, Token::Minus])] + #[case::double_sign(vec![ + ident("a"), Token::Gt, Token::Minus, Token::Minus, Token::Number("5".into()), + ])] + fn rejects_non_single_comparison(#[case] tokens: Vec) { + assert!(parse(tokens).is_err()); + } +} diff --git a/kernel/src/expressions/sql/token.rs b/kernel/src/expressions/sql/token.rs new file mode 100644 index 0000000000..36ecc3f941 --- /dev/null +++ b/kernel/src/expressions/sql/token.rs @@ -0,0 +1,449 @@ +//! A tokenizer for a subset of SQL. +//! +//! Splits a SQL string into [`Token`]s; what the tokens mean is up to the caller. The token set is +//! intentionally minimal and grows with the grammar the parser supports. Literal tokens keep their +//! raw source text so a caller can decode them however it needs. +//! +//! Example: `` `col1` > 5 `` tokenizes to `[Ident("col1"), Gt, Number("5")]`. + +// WIP feature behind `check-constraints-in-dev`; some items have no caller until enforcement lands. +// TODO(#2896): remove this allow once check-constraint enforcement wires up a caller. +#![allow(dead_code)] + +use std::iter::Peekable; +use std::str::Chars; + +use crate::expressions::column_names::{is_simple_char, parse_escaped_field_name}; +use crate::{DeltaResult, Error}; + +/// A peekable character stream over the source string. +type CharStream<'a> = Peekable>; + +/// A lexical token. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) enum Token { + Ident(String), + /// An unsigned numeric literal's raw source text, e.g. `42`, `.5`, `2e+1`. A leading sign is a + /// separate `Plus`/`Minus` token, never part of the number. Kept distinct from `Literal` so a + /// caller need not re-derive number-ness from the raw text. + Number(String), + /// A non-numeric literal's raw, undecoded source text (quotes and any typed-literal keyword + /// retained): quoted strings, `X'..'` binary, `TRUE`/`FALSE`/`NULL`, and typed + /// `DATE`/`TIMESTAMP`/ `TIMESTAMP_LTZ`/`TIMESTAMP_NTZ` literals. + Literal(String), + Dot, + Lt, + Le, + Gt, + Ge, + Eq, + /// `<>` or `!=` (both mean not-equal). + Ne, + /// `<=>`, Spark's null-safe equality (`NULL <=> NULL` is true). + NullSafeEq, + /// `+` and `-`. Emitted as standalone operators (matching Spark, where a leading sign is unary + /// minus/plus, not part of the number). A number is never lexed with a sign; the caller + /// reassembles a signed literal from `Minus`/`Plus` followed by a `Number`. + Plus, + Minus, + /// The `AND`/`OR`/`NOT`/`IS` keywords. Recognized here so a backtick-quoted column + /// named `` `AND` `` stays an `Ident`, distinct from the keyword. The current + /// parser has no grammar for them and rejects them. + Keyword(Keyword), +} + +/// A SQL keyword the tokenizer recognizes as a distinct token (see [`Token::Keyword`]). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) enum Keyword { + And, + Or, + Not, + Is, +} + +/// Tokenize `sql`, or return an error for any character/run outside the recognized token set (e.g. +/// parentheses, `*` or `/`, a lone `!`, or an unterminated string). +pub(super) fn tokenize(sql: &str) -> DeltaResult> { + let mut chars = sql.chars().peekable(); + let mut tokens = Vec::new(); + while let Some(&c) = chars.peek() { + match c { + c if c.is_whitespace() => { + chars.next(); + } + // A `.` starts a number when a digit follows (a leading-dot decimal like `.5`); + // otherwise it is a column-path separator. + '.' => { + if matches!(peek_second(&chars), Some(d) if d.is_ascii_digit()) { + tokens.push(Token::Number(take_number(&mut chars))); + } else { + chars.next(); + tokens.push(Token::Dot); + } + } + '=' => { + chars.next(); + // Accept `==` as an alias for `=` (Spark SQL allows both); consume the optional + // second `=`. + chars.next_if_eq(&'='); + tokens.push(Token::Eq); + } + '<' => { + chars.next(); + // Maximal munch: `<=` may extend to `<=>` (null-safe equal), so after consuming + // `=` check for a trailing `>` before settling on `Le`. + let tok = if chars.next_if_eq(&'=').is_some() { + if chars.next_if_eq(&'>').is_some() { + Token::NullSafeEq + } else { + Token::Le + } + } else if chars.next_if_eq(&'>').is_some() { + Token::Ne + } else { + Token::Lt + }; + tokens.push(tok); + } + '>' => { + chars.next(); + let tok = if chars.next_if_eq(&'=').is_some() { + Token::Ge + } else { + Token::Gt + }; + tokens.push(tok); + } + // `!=` is not-equal; `!>` and `!<` are Spark aliases for `<=` and `>=` respectively + // (Spark lexer: `LTE: '<=' | '!>'`, `GTE: '>=' | '!<'`). + '!' => { + chars.next(); + let tok = if chars.next_if_eq(&'=').is_some() { + Token::Ne + } else if chars.next_if_eq(&'>').is_some() { + Token::Le + } else if chars.next_if_eq(&'<').is_some() { + Token::Ge + } else { + return Err(unexpected('!', sql)); + }; + tokens.push(tok); + } + '\'' => tokens.push(Token::Literal(take_quoted_string(&mut chars, sql)?)), + // A backtick-quoted column field (`` `my col` ``, `` `a.b` ``). Reuse the crate's + // column-name field parser so quoting/escaping matches `ColumnName` exactly; it expects + // the opening backtick already consumed. + '`' => { + chars.next(); + tokens.push(Token::Ident(parse_escaped_field_name(&mut chars)?)); + } + // `+`/`-` are standalone operators (Spark treats a leading sign as unary +/-, not part + // of the number); the parser reassembles a signed literal from the sign + number. + // TODO(#2895): recognize `(` / `)` as LParen / RParen tokens once the grammar supports + // grouping. + '+' => { + chars.next(); + tokens.push(Token::Plus); + } + '-' => { + chars.next(); + tokens.push(Token::Minus); + } + c if c.is_ascii_digit() => tokens.push(Token::Number(take_number(&mut chars))), + c if c.is_ascii_alphabetic() || c == '_' => { + tokens.push(classify_word(&mut chars, sql)?) + } + _ => return Err(unexpected(c, sql)), + } + } + Ok(tokens) +} + +fn unexpected(c: char, sql: &str) -> Error { + Error::generic(format!("unexpected character '{c}' in {sql}")) +} + +/// Peek the second character of the stream (the one after the next), without consuming anything. +fn peek_second(chars: &CharStream<'_>) -> Option { + let mut lookahead = chars.clone(); + lookahead.next(); + lookahead.peek().copied() +} + +/// Consume a `'...'` string literal, returning its raw text *including* the surrounding quotes and +/// any doubled `''` escapes. +/// +/// Example: `'it''s'` returns `'it''s'` (verbatim). Returns an error if the input does not start +/// with a `'`. +fn take_quoted_string(chars: &mut CharStream<'_>, sql: &str) -> DeltaResult { + if chars.next_if_eq(&'\'').is_none() { + return Err(Error::generic(format!( + "string literal must start with a quote in {sql}" + ))); + } + let mut out = String::from('\''); + loop { + match chars.next() { + Some('\'') => { + out.push('\''); + // A doubled quote is an embedded single quote; keep scanning. Otherwise this was + // the closing quote. + match chars.next_if_eq(&'\'') { + Some(q) => out.push(q), + None => return Ok(out), + } + } + Some(c) => out.push(c), + None => { + return Err(Error::generic(format!( + "unterminated string literal in {sql}" + ))) + } + } + } +} + +/// Consume an unsigned numeric literal: a run of digits and `.` followed by an optional exponent. A +/// leading `+`/`-` is a separate operator token (see the `Plus`/`Minus` arms), not consumed here; +/// the exponent's own sign (`2e+1`) is part of the number. +/// +/// Examples: `3.14159` -> `3.14159`; `.2` -> `.2` (no leading digit needed); `2e+1` -> `2e+1`; +/// `1.2.3` -> `1.2.3` (malformed, emitted verbatim); `3/4` -> `3` (stops at `/`); `1+1` -> `1` +/// (stops before a second operator). +fn take_number(chars: &mut CharStream<'_>) -> String { + let mut out = String::new(); + while let Some(c) = chars.next_if(|c| c.is_ascii_digit() || *c == '.') { + out.push(c); + } + if let Some(e) = chars.next_if(|c| *c == 'e' || *c == 'E') { + out.push(e); + if let Some(sign) = chars.next_if(|c| *c == '+' || *c == '-') { + out.push(sign); + } + while let Some(c) = chars.next_if(|c| c.is_ascii_digit()) { + out.push(c); + } + } + out +} + +/// Scan a bare word and classify it: a keyword (`AND`/`OR`/`NOT`/`IS`), a boolean/null/typed/binary +/// literal, or an identifier. All classification is case-insensitive. +/// +/// Keywords are recognized here rather than downstream so a backtick-quoted `` `AND` `` (parsed as +/// an `Ident` by the caller of this function) stays distinct from the bareword keyword `AND`. +fn classify_word(chars: &mut CharStream<'_>, sql: &str) -> DeltaResult { + let mut word = String::new(); + while let Some(c) = chars.next_if(|c| is_simple_char(*c)) { + word.push(c); + } + let token = match word.to_ascii_uppercase().as_str() { + "AND" => Token::Keyword(Keyword::And), + "OR" => Token::Keyword(Keyword::Or), + "NOT" => Token::Keyword(Keyword::Not), + "IS" => Token::Keyword(Keyword::Is), + "NULL" | "TRUE" | "FALSE" => Token::Literal(word), + // `X'deadbeef'` binary literal: only when a quote immediately follows (no whitespace), + // otherwise `x` is a column named `x`. + "X" if chars.peek() == Some(&'\'') => { + let quoted = take_quoted_string(chars, sql)?; + Token::Literal(format!("{word}{quoted}")) + } + // Typed literal: the keyword followed (whitespace allowed) by a quoted string. Without the + // quote it is just a column named `date`/`timestamp`/etc. + "DATE" | "TIMESTAMP" | "TIMESTAMP_LTZ" | "TIMESTAMP_NTZ" if quote_follows(chars) => { + while chars.next_if(|c| c.is_whitespace()).is_some() {} + let quoted = take_quoted_string(chars, sql)?; + Token::Literal(format!("{word} {quoted}")) + } + _ => Token::Ident(word), + }; + Ok(token) +} + +/// Peek (without consuming) past any whitespace to see whether the next character is a `'`. Used to +/// tell a typed literal from a plain identifier: in `TIMESTAMP '..'` a quote follows (true), but a +/// bare `timestamp` column has none (false). +fn quote_follows(chars: &CharStream<'_>) -> bool { + let mut lookahead = chars.clone(); + while matches!(lookahead.peek(), Some(c) if c.is_whitespace()) { + lookahead.next(); + } + lookahead.peek() == Some(&'\'') +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::{tokenize, Keyword, Token}; + + fn ident(s: &str) -> Token { + Token::Ident(s.to_string()) + } + + fn lit(s: &str) -> Token { + Token::Literal(s.to_string()) + } + + fn num(s: &str) -> Token { + Token::Number(s.to_string()) + } + + /// `==` aliases `=`; `<>`/`!=` are `Ne`; `!>`/`!<` are Spark aliases for `<=`/`>=`. + #[rstest] + #[case("<", Token::Lt)] + #[case("<=", Token::Le)] + #[case(">", Token::Gt)] + #[case(">=", Token::Ge)] + #[case("=", Token::Eq)] + #[case("==", Token::Eq)] + #[case("!=", Token::Ne)] + #[case("<>", Token::Ne)] + #[case("<=>", Token::NullSafeEq)] + #[case("!>", Token::Le)] + #[case("!<", Token::Ge)] + #[case("+", Token::Plus)] + #[case("-", Token::Minus)] + fn tokenizes_each_operator(#[case] op: &str, #[case] expected: Token) { + assert_eq!( + tokenize(&format!("a {op} 1")).unwrap(), + [ident("a"), expected.clone(), num("1")] + ); + // Whitespace around the operator is optional. + assert_eq!( + tokenize(&format!("a{op}1")).unwrap(), + [ident("a"), expected, num("1")] + ); + } + + /// A leading `+`/`-` is a standalone operator, not part of the number: a signed number lexes as + /// two tokens (the parser reassembles it), and whitespace between the sign and the digits + /// (`- 5`) is irrelevant. The exponent's own sign stays inside the number. + #[rstest] + #[case("-5", &[Token::Minus, num("5")])] + #[case("- 5", &[Token::Minus, num("5")])] + #[case("+5", &[Token::Plus, num("5")])] + #[case("-.5", &[Token::Minus, num(".5")])] + #[case("-2e+1", &[Token::Minus, num("2e+1")])] + // `1+1` is three tokens (the sign never glues onto the following number), so the number after + // an operator is not swallowed into the previous one. + #[case("1+1", &[num("1"), Token::Plus, num("1")])] + fn tokenizes_signed_number_as_sign_then_number(#[case] sql: &str, #[case] expected: &[Token]) { + assert_eq!(tokenize(sql).unwrap(), expected); + } + + #[rstest] + #[case("42", num("42"))] + #[case(".5", num(".5"))] + #[case("1e3", num("1e3"))] + #[case("2e+1", num("2e+1"))] // the exponent sign is part of the number + fn tokenizes_number_as_single_raw_token(#[case] sql: &str, #[case] expected: Token) { + assert_eq!(tokenize(sql).unwrap(), [expected]); + } + + #[rstest] + #[case("'foo'", lit("'foo'"))] + #[case("'O''Brien'", lit("'O''Brien'"))] // doubled '' escape retained + #[case("NULL", lit("NULL"))] + #[case("TRUE", lit("TRUE"))] + #[case("false", lit("false"))] + #[case("DATE '1970-01-02'", lit("DATE '1970-01-02'"))] + #[case("DATE'1970-01-02'", lit("DATE '1970-01-02'"))] // butted quote, space normalized in output + #[case("X'01ff'", lit("X'01ff'"))] + fn tokenizes_literal_as_single_raw_token(#[case] sql: &str, #[case] expected: Token) { + assert_eq!(tokenize(sql).unwrap(), [expected]); + } + + /// Every typed-literal keyword tokenizes to exactly one literal, so dropping one from the + /// tokenizer's set (which would mis-lex it as an identifier) fails here. + #[rstest] + #[case("DATE '2024-01-01'")] + #[case("TIMESTAMP '2024-01-01T00:00:00Z'")] + #[case("TIMESTAMP_LTZ '2024-01-01T00:00:00Z'")] + #[case("TIMESTAMP_NTZ '2024-01-01 00:00:00'")] + fn every_typed_keyword_tokenizes_to_one_literal(#[case] sql: &str) { + assert!(matches!( + tokenize(sql).unwrap().as_slice(), + [Token::Literal(_)] + )); + } + + /// `AND`/`OR`/`NOT`/`IS` are keyword tokens (case-insensitive); a backtick-quoted `` `AND` `` + /// stays an `Ident`, distinct from the keyword. + #[rstest] + #[case("AND", Token::Keyword(Keyword::And))] + #[case("or", Token::Keyword(Keyword::Or))] + #[case("Not", Token::Keyword(Keyword::Not))] + #[case("IS", Token::Keyword(Keyword::Is))] + #[case("`AND`", ident("AND"))] + fn tokenizes_keyword_vs_quoted_identifier(#[case] sql: &str, #[case] expected: Token) { + assert_eq!(tokenize(sql).unwrap(), [expected]); + } + + #[test] + fn tokenizes_dotted_column_path() { + assert_eq!( + tokenize("a.b.c").unwrap(), + [ident("a"), Token::Dot, ident("b"), Token::Dot, ident("c")] + ); + } + + #[rstest] + #[case("")] + #[case(" ")] + #[case("\t\n ")] + fn tokenizes_empty_or_whitespace_only_input_to_no_tokens(#[case] sql: &str) { + assert_eq!(tokenize(sql).unwrap(), []); + } + + /// A malformed number is emitted verbatim as one raw `Number` token; the tokenizer does not + /// validate numeric structure. + #[rstest] + #[case("1e", num("1e"))] + #[case("1.2.3", num("1.2.3"))] + #[case("1E3", num("1E3"))] + #[case("5e-3", num("5e-3"))] + fn tokenizes_malformed_number_as_single_raw_token(#[case] sql: &str, #[case] expected: Token) { + assert_eq!(tokenize(sql).unwrap(), [expected]); + } + + /// A backtick-quoted field is one `Ident` carrying its unescaped name (spaces, dots, and + /// doubled-backtick escapes included), matching `ColumnName`. + #[rstest] + #[case("`my col`", ident("my col"))] + #[case("`a.b`", ident("a.b"))] + #[case("`a``b`", ident("a`b"))] + fn tokenizes_backtick_quoted_field(#[case] sql: &str, #[case] expected: Token) { + assert_eq!(tokenize(sql).unwrap(), [expected]); + } + + #[test] + fn tokenizes_backtick_field_in_comparison_and_path() { + assert_eq!( + tokenize("`my col` > 0").unwrap(), + [ident("my col"), Token::Gt, num("0")] + ); + assert_eq!( + tokenize("a.`b c`").unwrap(), + [ident("a"), Token::Dot, ident("b c")] + ); + assert!(tokenize("`unterminated").is_err()); + } + + #[rstest] + #[case("amount", ident("amount"))] + #[case("date", ident("date"))] // keyword not followed by a quote is a plain identifier + #[case("x", ident("x"))] + fn tokenizes_bareword_as_identifier(#[case] sql: &str, #[case] expected: Token) { + assert_eq!(tokenize(sql).unwrap(), [expected]); + } + + #[rstest] + #[case("a ! b")] // lone `!` + #[case("(a)")] // grouping not yet supported + #[case("'unterminated")] + fn rejects_ungrammatical_input(#[case] sql: &str) { + assert!(tokenize(sql).is_err(), "expected {sql:?} to be rejected"); + } +} diff --git a/kernel/src/history_manager/error.rs b/kernel/src/history_manager/error.rs new file mode 100644 index 0000000000..5ac2b80182 --- /dev/null +++ b/kernel/src/history_manager/error.rs @@ -0,0 +1,141 @@ +//! Error types for the history manager module. + +use url::Url; + +use super::search::Bound; +use super::Timestamp; +use crate::Version; + +/// The nearest retained timestamp on the side of the search bound that an +/// out-of-range search failed against. Engines surface this to users so the +/// error message can point at a valid timestamp. +/// +/// `Earliest` and `Latest` carry the boundary commit's timestamp. `Unknown` +/// means no boundary timestamp was available, e.g. empty log or a failure +/// reading the boundary commit's In-Commit Timestamp at the error site. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub enum NearestTimestamp { + /// The earliest retained commit's timestamp. Returned for a + /// `GreatestLower` search whose input fell below the retained range. + Earliest(Timestamp), + /// The latest retained commit's timestamp. Returned for a `LeastUpper` + /// search whose input fell above the retained range. + Latest(Timestamp), + /// No boundary timestamp could be determined. + Unknown, +} + +impl NearestTimestamp { + /// Tags `ts` with the variant matching `bound`. + pub(crate) fn from_boundary(bound: Bound, ts: Timestamp) -> Self { + match bound { + Bound::GreatestLower => Self::Earliest(ts), + Bound::LeastUpper => Self::Latest(ts), + } + } +} + +/// Represents errors that can occur when converting commit timestamps to versions. +#[derive(Debug, thiserror::Error)] +pub enum LogHistoryError { + /// No commit files were found in the log directory. + #[error("No commits found in log directory {log_root}")] + NoCommitsFound { + /// The log directory URL that was searched. + log_root: Url, + }, + /// The provided timestamp range is invalid (start > end). + #[error("Invalid timestamp range: ({start_timestamp}, {end_timestamp})")] + InvalidTimestampRange { + /// The start timestamp that was greater than the end timestamp. + start_timestamp: Timestamp, + /// The end timestamp that was less than the start timestamp. + end_timestamp: Timestamp, + }, + /// The timestamp range contains no commits - the entire range falls between two adjacent + /// versions. + #[error( + "There are no commits in the timestamp range ({start_timestamp}, {end_timestamp}). \ + The entire timestamp range falls between versions {between_version} and {}.", + .between_version + 1 + )] + EmptyTimestampRange { + /// The start of the empty timestamp range. + start_timestamp: Timestamp, + /// The end of the empty timestamp range. + end_timestamp: Timestamp, + /// The version immediately before the timestamp range (the next version is + /// `between_version + 1`). + between_version: Version, + }, + /// The timestamp is outside the range of available commits. + #[error("Timestamp {timestamp} is out of range: {reason}")] + TimestampOutOfRange { + /// The timestamp that was out of range. + timestamp: Timestamp, + /// Description of why the timestamp is out of range. + reason: &'static str, + /// The nearest retained commit's timestamp on the side of the search + /// bound. For example, given a retained range `[100, 500]` and a + /// `GreatestLower` search at timestamp `50`, this is + /// `NearestTimestamp::Earliest(100)`. + nearest_timestamp: NearestTimestamp, + }, + /// Commit files exist in the log but the table cannot be reconstructed: commit version 0 + /// is missing and no complete checkpoint is present to anchor the surviving commits. + #[error( + "No recreatable commits found at {log_root}: commits exist but version 0 is missing \ + and no complete checkpoint is present" + )] + NoRecreatableCommit { + /// The log root URL that was scanned. + log_root: Url, + }, + /// An internal error occurred during timestamp conversion. + #[error("{context}{}", source.as_ref().map(|e| format!(": {e}")).unwrap_or_default())] + Internal { + /// Description of the operation that failed. + context: &'static str, + /// The underlying error, if any. + #[source] + source: Option>, + }, +} + +impl LogHistoryError { + /// Creates an internal error with context and an underlying cause. + pub(crate) fn internal(context: &'static str, source: crate::Error) -> Self { + Self::Internal { + context, + source: Some(Box::new(source)), + } + } + + /// Creates an internal error with just a context message. + pub(crate) fn internal_message(context: &'static str) -> Self { + Self::Internal { + context, + source: None, + } + } + + /// Creates a `TimestampOutOfRange` error. The reason is derived from `bound`. + pub(crate) fn out_of_range( + timestamp: Timestamp, + bound: Bound, + nearest_timestamp: NearestTimestamp, + ) -> Self { + Self::TimestampOutOfRange { + timestamp, + reason: bound.out_of_range_reason(), + nearest_timestamp, + } + } +} + +impl From for crate::Error { + fn from(e: LogHistoryError) -> Self { + crate::Error::LogHistory(Box::new(e)) + } +} diff --git a/kernel/src/history_manager/mod.rs b/kernel/src/history_manager/mod.rs index 7c37877ba3..1cbb248046 100644 --- a/kernel/src/history_manager/mod.rs +++ b/kernel/src/history_manager/mod.rs @@ -1 +1,2303 @@ +//! This module provides functions for performing timestamp queries over the Delta Log, translating +//! between timestamps and Delta versions. +//! +//! # Usage +//! +//! Use this module to: +//! - Convert timestamps or timestamp ranges into Delta versions or version ranges +//! - Perform time travel queries +//! - Execute timestamp-based change data feed queries +//! +//! The history_manager module works with tables regardless of whether they have In-Commit +//! Timestamps (ICT) enabled. ICT is a Delta table feature that stores precise commit timestamps +//! in the commit metadata rather than relying on file modification times, which can be affected +//! by clock skew or filesystem limitations. +//! +//! # Limitations +//! +//! All timestamp queries are limited to the state captured in the provided [`Snapshot`]. For +//! tables without ICT enabled, timestamp queries rely on file modification timestamps, which are +//! not guaranteed to be monotonically increasing across commits. + +use std::cmp::Ordering; +use std::collections::{HashMap, HashSet}; +use std::num::NonZero; + +use error::{LogHistoryError, NearestTimestamp}; +use itertools::Itertools; +use search::{binary_search_by_key_with_bounds, Bound, SearchError}; +use tracing::{info, trace, warn}; +use url::Url; + +use crate::log_segment::LogSegment; +use crate::log_segment_files::{list_delta_log_from_storage, should_process_log_file}; +use crate::path::{LogPathFileType, ParsedLogPath}; +use crate::snapshot::Snapshot; +use crate::table_configuration::InCommitTimestampEnablement; +use crate::utils::require; +use crate::{DeltaResult, Engine, Error as DeltaError, Version}; + pub(crate) mod search; + +pub mod error; + +/// A timestamp representing milliseconds since the Unix epoch (1970-01-01 00:00:00 UTC). +/// +/// This type is used throughout the history_manager module for timestamp-to-version conversion. +/// All timestamp values should be specified in milliseconds, not seconds or nanoseconds. +pub type Timestamp = i64; + +/// A commit located by a timestamp query: the commit [`Version`] paired with its timestamp. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct CommitAt { + /// The commit version. + pub version: Version, + /// Timestamp (milliseconds since the Unix epoch) associated with this commit. + /// + /// This is the commit's in-commit timestamp (ICT) when ICT is enabled, otherwise the + /// commit's file modification time. + pub timestamp: Timestamp, +} + +impl CommitAt { + /// Creates a [`CommitAt`] pairing a commit `version` with its `timestamp`. + pub fn new(version: Version, timestamp: Timestamp) -> Self { + Self { version, timestamp } + } +} + +/// Determines the search strategy for timestamp-to-version conversion based on ICT enablement. +#[derive(Debug, PartialEq, Eq)] +enum TimestampSearchBounds { + /// The timestamp exactly matches the ICT enablement timestamp. + ExactMatch(Version), + /// Search over In-Commit Timestamps starting from the given index. + ICTSearchStartingFrom(usize), + /// Search over file modification timestamps (ICT not enabled). + FileModificationSearch, + /// Search over file modification timestamps up to the ICT enablement point. + FileModificationSearchUntil { + /// The index in the commit list where ICT begins. + index: usize, + /// The version at which ICT was enabled. + ict_enablement_version: Version, + /// The timestamp at which ICT was enabled. + ict_enablement_timestamp: Timestamp, + }, +} + +/// Determines the search strategy for timestamp-to-version conversion based on ICT enablement. +/// +/// Given a timestamp, this function determines which commit range to search and what timestamp +/// source to use (file modification time vs in-commit timestamp). A timestamp search may be +/// conducted over one of two version ranges: +/// 1) A range of commits whose timestamp is the file modification timestamp +/// 2) A range of commits whose timestamp is an in-commit timestamp +/// +/// # Returns +/// +/// - [`TimestampSearchBounds::FileModificationSearch`]: ICT not enabled; search all commits using +/// file modification timestamps. +/// - [`TimestampSearchBounds::ICTSearchStartingFrom`]: Search commits with ICT enabled using +/// in-commit timestamps (binary search). +/// - [`TimestampSearchBounds::FileModificationSearchUntil`]: Timestamp is before ICT enablement; +/// search pre-ICT commits using file modification timestamps. +/// - [`TimestampSearchBounds::ExactMatch`]: Timestamp exactly matches the ICT enablement timestamp. +/// +/// # Errors +/// +/// Returns [`LogHistoryError::Internal`] if: +/// - Failed to read table configuration +/// - ICT enablement version is beyond the log segment (indicates corrupted metadata) +#[tracing::instrument(skip(snapshot, log_segment), ret)] +fn get_timestamp_search_bounds( + snapshot: &Snapshot, + log_segment: &LogSegment, + timestamp: Timestamp, +) -> Result { + debug_assert!(log_segment.end_version == snapshot.version()); + let table_config = snapshot.table_configuration(); + + // Get the In-commit timestamp (ICT) enablement version and timestamp. If ICT is not + // supported or enabled, we perform a regular file modification search. + let ict_enablement = table_config + .in_commit_timestamp_enablement() + .map_err(|e| LogHistoryError::internal("failed to read table configuration", e))?; + + let (ict_enablement_version, ict_timestamp) = match ict_enablement { + InCommitTimestampEnablement::NotEnabled => { + return Ok(TimestampSearchBounds::FileModificationSearch) + } + InCommitTimestampEnablement::Enabled { + enablement: Some((v, t)), + } => (v, t), + InCommitTimestampEnablement::Enabled { enablement: None } => { + // ICT was enabled from table creation - all commits have ICT + return Ok(TimestampSearchBounds::ICTSearchStartingFrom(0)); + } + }; + + // Fast path: if the first commit is at or after ICT enablement (e.g., old commits cleaned + // up), all commits in the segment have ICT - skip binary search. + let first_version = log_segment.listed.ascending_commit_files[0].version; + if first_version >= ict_enablement_version { + return Ok(TimestampSearchBounds::ICTSearchStartingFrom(0)); + } + + // Get the index of the ICT enablement version in the log segment. If the version is not + // present, binary_search returns the insertion point (first commit at or after). + let commit_count = log_segment.listed.ascending_commit_files.len(); + let ict_enablement_idx = log_segment + .listed + .ascending_commit_files + .binary_search_by(|x| x.version.cmp(&ict_enablement_version)) + .unwrap_or_else(|idx| idx); + + // Invariant: ICT enablement version must be within the log segment. If it equals + // commit_count, the enablement version is beyond the table's latest version. + require!( + ict_enablement_idx < commit_count, + LogHistoryError::internal_message( + "ICT enablement version is beyond the table's latest version" + ) + ); + + // Per PROTOCOL.md section on In-Commit Timestamps: + // - ts >= enablementTs => consider only versions >= enablementVersion (ICT region) + // - ts < enablementTs => consider only versions < enablementVersion (file mod region) + let result = match timestamp.cmp(&ict_timestamp) { + // Equal: enablementTs IS the ICT of enablementVersion, so this version is the exact + // answer for both bounds (GreatestLower and LeastUpper). + Ordering::Equal => TimestampSearchBounds::ExactMatch(ict_enablement_version), + // Less: Restrict search to the pre-ICT region via FileModificationSearchUntil. The + // protocol guarantees enablementTs > prev_mod_time, so when the pre-ICT linear scan + // fails for LeastUpper, enablementVersion is the true least upper bound. + Ordering::Less => TimestampSearchBounds::FileModificationSearchUntil { + index: ict_enablement_idx, + ict_enablement_version, + ict_enablement_timestamp: ict_timestamp, + }, + // Greater: Restrict search to ICT commits only via ICTSearchStartingFrom. + Ordering::Greater => TimestampSearchBounds::ICTSearchStartingFrom(ict_enablement_idx), + }; + Ok(result) +} + +/// Performs a linear search over file modification timestamps with monotonization. +/// +/// File modification timestamps are not guaranteed to be monotonically increasing due to clock +/// skew or filesystem limitations. This function ensures correctness by monotonizing timestamps +/// as it scans: if a commit's timestamp is less than or equal to the previous commit's timestamp, +/// it is adjusted to `previous_timestamp + 1`. +/// +/// # Arguments +/// * `commits` - Slice of commits to search, ordered by version (ascending) +/// * `timestamp` - The target timestamp to search for +/// * `bound` - The type of bound to find (GreatestLower or LeastUpper) +/// +/// # Returns +/// The [`CommitAt`] matching the bound criteria, whose `timestamp` is the monotonized file +/// modification time, or an error if no match exists. +// +// NOTE: Monotonization trade-offs +// +// File modification timestamps can have clock skew. We monotonize over visible history +// only, which may produce incorrect results for commits near the retention boundary if +// metadata cleanup removed skewed commits that affected monotonization. +// +// Since ICT is the correct long-term solution for timestamps, we avoid over-investing +// in file modification timestamp handling. +// +// TODO: Today we read full commit history for timestamp conversion, which is expensive. +// A future backwards-listing optimization would avoid reading full history - at that +// point, monotonizing over full history becomes a bad trade-off and should be revisited. +fn linear_search_file_mod_timestamps( + commits: &[ParsedLogPath], + timestamp: Timestamp, + bound: Bound, +) -> Result { + if commits.is_empty() { + return Err(LogHistoryError::out_of_range( + timestamp, + bound, + NearestTimestamp::Unknown, + )); + } + + let lo_version = commits[0].version; + let hi_version = commits[commits.len() - 1].version; + info!(lo_version, hi_version, "File modification linear search"); + + let mut result: Option = None; + let mut prev_monotonic_ts = i64::MIN; + + for commit in commits { + let raw_ts = commit.location.last_modified; + // Monotonize: ensure each timestamp is strictly greater than the previous + let monotonic_ts = raw_ts.max(prev_monotonic_ts.saturating_add(1)); + if monotonic_ts != raw_ts { + trace!( + version = commit.version, + raw_ts, + monotonic_ts, + "Adjusted non-monotonic timestamp" + ); + } + + match bound { + // GreatestLower: update result until we surpass `timestamp` + Bound::GreatestLower if monotonic_ts <= timestamp => { + result = Some(CommitAt::new(commit.version, monotonic_ts)) + } + Bound::GreatestLower => break, + // LeastUpper: return version upon first match + Bound::LeastUpper if monotonic_ts >= timestamp => { + return Ok(CommitAt::new(commit.version, monotonic_ts)) + } + Bound::LeastUpper => {} + } + + prev_monotonic_ts = monotonic_ts; + } + + result.ok_or_else(|| { + // OOR with non-empty commits: under GreatestLower, `timestamp` is below + // the first commit's monotonic ts (which equals its raw `last_modified` + // since no prior commit affects it). Under LeastUpper, `timestamp` is + // above the last commit's monotonic ts (held in `prev_monotonic_ts`). + // Using the monotonized values means a re-query at the suggested + // `nearest` round-trips to the same boundary version. + let nearest_ts = match bound { + Bound::GreatestLower => commits[0].location.last_modified, + Bound::LeastUpper => prev_monotonic_ts, + }; + LogHistoryError::out_of_range( + timestamp, + bound, + NearestTimestamp::from_boundary(bound, nearest_ts), + ) + }) +} + +/// Binary search over commits with In-Commit Timestamps (ICT). +/// +/// ICT timestamps are guaranteed monotonic by protocol, so binary search is safe. +fn binary_search_ict_timestamps( + commits: &[ParsedLogPath], + timestamp: Timestamp, + bound: Bound, + engine: &dyn Engine, +) -> Result { + if commits.is_empty() { + return Err(LogHistoryError::out_of_range( + timestamp, + bound, + NearestTimestamp::Unknown, + )); + } + + let lo_version = commits[0].version; + let hi_version = commits[commits.len() - 1].version; + info!( + lo_version, + hi_version, "ICT binary search over version range" + ); + + let commit_to_ict = |commit: &ParsedLogPath| -> Result { + commit + .read_in_commit_timestamp(engine) + .map_err(|e| LogHistoryError::internal("failed to read in-commit timestamp", e)) + }; + + match binary_search_by_key_with_bounds(commits, timestamp, commit_to_ict, bound) { + Ok(idx) => { + let version = commits[idx].version; + // TODO(#2687): reading in_commit_timestamp is an expensive operation since it require + // an engine I/O. An optimization is to have the `binary_search_by_key_with_bounds` + // return the search result's index and key value to avoid calling commit_to_ict + let timestamp = commit_to_ict(&commits[idx])?; + Ok(CommitAt::new(version, timestamp)) + } + Err(SearchError::KeyFunctionError(error)) => Err(error), + Err(SearchError::OutOfRange) => { + // OutOfRange under `GreatestLower` means timestamp < commits[0]'s + // ICT; under `LeastUpper` it means timestamp > commits.last()'s + // ICT. The boundary timestamp is read via `commit_to_ict`; on + // engine failure we degrade to `Unknown` so the original + // out-of-range error is preserved. + let nearest = bound + .boundary_of(commits) + .and_then(|boundary| { + commit_to_ict(boundary) + .inspect_err(|e| { + warn!( + error = ?e, + version = boundary.version, + "failed to read boundary ICT for nearest_timestamp hint", + ); + }) + .ok() + .map(|ts| NearestTimestamp::from_boundary(bound, ts)) + }) + .unwrap_or(NearestTimestamp::Unknown); + Err(LogHistoryError::out_of_range(timestamp, bound, nearest)) + } + } +} + +/// Converts a timestamp to a [`CommitAt`] based on the specified bound type. +/// +/// This function finds the appropriate commit that corresponds to the given timestamp +/// according to the bound parameter: +/// +/// - `Bound::GreatestLower`: Returns the commit with the greatest timestamp that is less than or +/// equal to the given timestamp (the version immediately before or at the timestamp). +/// - `Bound::LeastUpper`: Returns the commit with the smallest timestamp that is greater than or +/// equal to the given timestamp (the version immediately at or after the timestamp). +/// +/// # Visual Example: +/// ```ignore +/// versions: v0 v1 v2 +/// |----------|----------|----------|----------| +/// timestamp: t1 t2 +/// ``` +/// +/// # Results Table: +/// | Timestamp | GreatestLower | LeastUpper | +/// |-----------|---------------|------------| +/// | t1 | v0 | v1 | +/// | t2 | v1 | v2 | +/// +/// # Errors +/// Returns [`LogHistoryError::TimestampOutOfRange`] if the timestamp is out of range. This can +/// happen in the following cases based on the bound: +/// - `Bound::GreatestLower`: There is no commit whose timestamp is less than or equal to the given +/// `timestamp`. +/// - `Bound::LeastUpper`: There is no commit whose timestamp is greater than or equal to the given +/// `timestamp`. +/// +/// Propagates a [`LogHistoryError`] from getting the earliest recreatable commit when +/// `resolved_commit_type` is [`HistoryCommitType::Recreatable`]. +pub(crate) fn timestamp_to_version( + snapshot: &Snapshot, + engine: &dyn Engine, + timestamp: Timestamp, + bound: Bound, + resolved_commit_type: HistoryCommitType, +) -> Result { + // Short-circuit: compare against snapshot's timestamp to avoid log segment rebuild. + // This optimization mirrors Delta Spark and Java Kernel behavior. + let snap_ts = snapshot + .get_timestamp(engine) + .map_err(|e| LogHistoryError::internal("failed to get snapshot timestamp", e))?; + match (timestamp.cmp(&snap_ts), bound) { + // Exact match: snapshot version satisfies both bounds + (Ordering::Equal, _) => return Ok(CommitAt::new(snapshot.version(), snap_ts)), + // Timestamp after snapshot: for GreatestLower, snapshot is the answer + (Ordering::Greater, Bound::GreatestLower) => { + return Ok(CommitAt::new(snapshot.version(), snap_ts)) + } + // Timestamp after snapshot: for LeastUpper, no version exists + (Ordering::Greater, Bound::LeastUpper) => { + return Err(LogHistoryError::out_of_range( + timestamp, + bound, + NearestTimestamp::Latest(snap_ts), + )); + } + // Timestamp before snapshot: need to search the log + _ => {} + } + + // Catalog-managed tables carry staged commits that exist only in the snapshot's log_tail, + // not on the filesystem. These commits carry ICT, so to support timestamp conversion we pass + // the log_tail in for catalog-managed tables. + let staged_log_tail: Vec = snapshot + .log_segment() + .listed + .staged_commits() + .cloned() + .collect(); + + let listing_limit = match resolved_commit_type { + HistoryCommitType::Published => None, + HistoryCommitType::Recreatable => { + let earliest = get_earliest_commit( + engine, + &snapshot.log_segment().log_root, + None, + HistoryCommitType::Recreatable, + ) + .map_err(|e| match e { + DeltaError::LogHistory(inner) => *inner, + _ => LogHistoryError::internal("failed to get earliest commit", e), + })?; + let limit = snapshot + .version() + .checked_sub(earliest) + .and_then(|diff| usize::try_from(diff + 1).ok()) + .and_then(NonZero::new) + .ok_or_else(|| { + LogHistoryError::internal( + "earliest recreatable version exceeds snapshot version", + DeltaError::generic(format!( + "earliest = {earliest}, snapshot version = {}", + snapshot.version() + )), + ) + })?; + Some(limit) + } + }; + + // Build a dedicated log segment for timestamp conversion. We cannot reuse + // snapshot.log_segment() because it may include a checkpoint prefix with gaps in + // the commit sequence. for_timestamp_conversion returns only contiguous commits, + // which is required for correct timestamp-to-version mapping. + let log_segment = LogSegment::for_timestamp_conversion( + engine.storage_handler().as_ref(), + snapshot.log_segment().log_root.clone(), + snapshot.version(), + listing_limit, + staged_log_tail, + ) + .map_err(|e| { + LogHistoryError::internal("failed to build log segment for timestamp conversion", e) + })?; + debug_assert!( + !log_segment.listed.ascending_commit_files.is_empty(), + "LogSegment should ensure that a segment is non-empty" + ); + debug_assert!(log_segment.end_version == snapshot.log_segment().end_version); + + // Determine the type of timestamp search. The search may either be over file modification + // timestamps or over In-Commit Timestamps. + let commits = &log_segment.listed.ascending_commit_files; + let search_bounds = get_timestamp_search_bounds(snapshot, &log_segment, timestamp)?; + + match search_bounds { + TimestampSearchBounds::ExactMatch(version) => Ok(CommitAt::new(version, timestamp)), + TimestampSearchBounds::ICTSearchStartingFrom(lo) => { + binary_search_ict_timestamps(&commits[lo..], timestamp, bound, engine) + } + TimestampSearchBounds::FileModificationSearch => { + linear_search_file_mod_timestamps(commits, timestamp, bound) + } + TimestampSearchBounds::FileModificationSearchUntil { + index, + ict_enablement_version, + ict_enablement_timestamp, + } => { + linear_search_file_mod_timestamps(&commits[..index], timestamp, bound).or_else(|e| { + // For LeastUpper, if no version found in pre-ICT region, the ICT enablement + // version is the answer (protocol guarantees enablementTs > prev_mod_time) + match (&e, bound) { + (LogHistoryError::TimestampOutOfRange { .. }, Bound::LeastUpper) => Ok( + CommitAt::new(ict_enablement_version, ict_enablement_timestamp), + ), + _ => Err(e), + } + }) + } + } +} + +/// Gets the latest [`CommitAt`] (version and timestamp) with a timestamp at or before `timestamp`. +/// +/// `resolved_commit_type` constrains the returned version: +/// - [`HistoryCommitType::Published`]: may return any version present in the log, even one whose +/// table cannot be reconstructed. +/// - [`HistoryCommitType::Recreatable`]: only returns a version whose table can be fully +/// reconstructed at the query time. +/// +/// # Errors +/// Returns [`LogHistoryError::TimestampOutOfRange`] if no version exists at or before +/// the given timestamp. +/// +/// # Examples +/// ```ignore +/// use delta_kernel::snapshot::Snapshot; +/// use test_utils::delta_kernel_default_engine::DefaultEngine; +/// use delta_kernel::history_manager::{latest_version_as_of, HistoryCommitType}; +/// +/// let engine = DefaultEngine::try_new(...)?; +/// let snapshot = Snapshot::builder_for(table_uri).build(&engine)?; +/// +/// // Get the latest commit as of January 1, 2023 +/// let timestamp = 1672531200000; // Milliseconds since epoch for 2023-01-01 +/// let commit = latest_version_as_of(&snapshot, &engine, timestamp, HistoryCommitType::Recreatable)?; +/// ``` +#[tracing::instrument(skip(snapshot, engine), ret, fields(latest_version = snapshot.version(), table_root = %snapshot.table_root()))] +pub fn latest_version_as_of( + snapshot: &Snapshot, + engine: &dyn Engine, + timestamp: Timestamp, + resolved_commit_type: HistoryCommitType, +) -> DeltaResult { + timestamp_to_version( + snapshot, + engine, + timestamp, + Bound::GreatestLower, + resolved_commit_type, + ) + .map_err(Into::into) +} + +/// Gets the first [`CommitAt`] (version and timestamp) with a timestamp at or after `timestamp`. +/// +/// `resolved_commit_type` constrains the returned version: +/// - [`HistoryCommitType::Published`]: may return any version present in the log, even one whose +/// table cannot be reconstructed. +/// - [`HistoryCommitType::Recreatable`]: only returns a version whose table can be fully +/// reconstructed at the query time. +/// +/// Returns [`LogHistoryError::TimestampOutOfRange`] if no version exists at or after +/// the given timestamp. +/// +/// # Examples +/// ```ignore +/// use delta_kernel::snapshot::Snapshot; +/// use test_utils::delta_kernel_default_engine::DefaultEngine; +/// use delta_kernel::history_manager::{first_version_after, HistoryCommitType}; +/// +/// let engine = DefaultEngine::try_new(...)?; +/// let snapshot = Snapshot::builder_for(table_uri).build(&engine)?; +/// +/// // Find the first commit that occurred at or after January 1, 2023 +/// let timestamp = 1672531200000; // Milliseconds since epoch for 2023-01-01 +/// let commit = first_version_after(&snapshot, &engine, timestamp, HistoryCommitType::Recreatable)?; +/// ``` +#[tracing::instrument(skip(snapshot, engine), ret, fields(latest_version = snapshot.version(), table_root = %snapshot.table_root()))] +pub fn first_version_after( + snapshot: &Snapshot, + engine: &dyn Engine, + timestamp: Timestamp, + resolved_commit_type: HistoryCommitType, +) -> DeltaResult { + timestamp_to_version( + snapshot, + engine, + timestamp, + Bound::LeastUpper, + resolved_commit_type, + ) + .map_err(Into::into) +} + +/// Converts a timestamp range to a corresponding version range. +/// +/// This function finds the version range that corresponds to the given timestamp range. +/// The returned tuple contains: +/// - The first (earliest) version with a timestamp greater than or equal to `start_timestamp` +/// - If `end_timestamp` is provided, the version with a timestamp less than or equal to +/// `end_timestamp`. +/// +/// # Arguments +/// * `snapshot` - The snapshot that defines the searchable version range +/// * `engine` - The engine used to access version history +/// * `start_timestamp` - The lower bound timestamp (inclusive), in milliseconds since Unix epoch +/// * `end_timestamp` - The optional upper bound timestamp (inclusive), or `None` to indicate no +/// upper bound +/// +/// # Returns +/// A tuple containing the start version and optional end version (inclusive) +/// +/// # Errors +/// Returns [`LogHistoryError::InvalidTimestampRange`] if `start_timestamp > end_timestamp`. +/// +/// Returns [`LogHistoryError::TimestampOutOfRange`] if: +/// - No version exists at or after `start_timestamp` +/// - `end_timestamp` is provided and no version exists at or before it +/// +/// Returns [`LogHistoryError::EmptyTimestampRange`] if the entire range falls between two commits. +/// +/// # Examples +/// ```ignore +/// use delta_kernel::snapshot::Snapshot; +/// use test_utils::delta_kernel_default_engine::DefaultEngine; +/// use delta_kernel::history_manager::timestamp_range_to_versions; +/// +/// let engine = DefaultEngine::try_new(...)?; +/// let snapshot = Snapshot::builder_for(table_uri).build(&engine)?; +/// +/// // Find versions between January 1, 2023 and March 1, 2023 +/// let start_timestamp = 1672531200000; // Jan 1, 2023 (milliseconds since epoch) +/// let end_timestamp = 1677628800000; // Mar 1, 2023 (milliseconds since epoch) +/// +/// let (start_version, end_version) = +/// timestamp_range_to_versions(&snapshot, &engine, start_timestamp, Some(end_timestamp))?; +/// ``` +#[tracing::instrument(skip(snapshot, engine), ret, fields(latest_version = snapshot.version(), table_root = %snapshot.table_root()))] +pub fn timestamp_range_to_versions( + snapshot: &Snapshot, + engine: &dyn Engine, + start_timestamp: Timestamp, + end_timestamp: Option, +) -> DeltaResult<(Version, Option)> { + if let Some(end_timestamp) = end_timestamp { + // The `start_timestamp` must be no greater than the `end_timestamp`. + require!( + start_timestamp <= end_timestamp, + LogHistoryError::InvalidTimestampRange { + start_timestamp, + end_timestamp + } + .into() + ); + } + + // TODO(#2449): Optimize by reusing the log segment between the two timestamp conversions. + // Currently, both first_version_after and latest_version_as_of each build their own + // log segment via listing. We could build the segment once and pass it to a shared + // helper (e.g., timestamp_to_version_in_segment) to cut listing time in half. + + // Convert the start timestamp to version + let start_version = first_version_after( + snapshot, + engine, + start_timestamp, + HistoryCommitType::Published, + )? + .version; + + // If the end timestamp is present, convert it to an end version + let end_version = end_timestamp + .map(|end| { + let end_version = + latest_version_as_of(snapshot, engine, end, HistoryCommitType::Published)?.version; + + // Verify that the start version is no greater than the end version. This can + // happen in the case that the entire timestamp range falls between two commits. + // Consider the following history: + // |-------------|--------------------|---------------| + // v4 start_timestamp end_timestamp v5 + // + // The latest version as of the end_timestamp is 4. The first version after the + // start_timestamp is 5. Thus in the case where end_version < start_version, we + // return an [`LogHistoryError::EmptyTimestampRange`]. + require!( + start_version <= end_version, + DeltaError::from(LogHistoryError::EmptyTimestampRange { + start_timestamp, + end_timestamp: end, + between_version: end_version, + }) + ); + + Ok(end_version) + }) + .transpose()?; + + Ok((start_version, end_version)) +} + +/// Returns the earliest commit version available on the file system for this table. +/// +/// The returned version is the version of the lowest-numbered `*.json` commit file present in +/// `log_root`. The returned version is not guaranteed to exist by the time the caller +/// acts on it: a concurrent log-cleanup operation may delete the file. +/// +/// # Parameters +/// - `engine`: kernel engine used to list `log_root`. +/// - `log_root`: URL of the table's `_delta_log/` directory (must end with `/`). +/// - `earliest_ratified_commit_version`: For catalog-managed tables, it is the earliest version the +/// catalog has ratified commit. Pass `None` for filesystem-only tables. +/// +/// # Errors +/// - Propagates any error from listing the log directory. +/// - [`LogHistoryError::NoCommitsFound`] when the log directory contains no commits. +/// - [`DeltaError::Generic`] when there is no publised file-system commit and the earliest ratified +/// CCv2 commit is v0. For a catalog-managed table, v0 must be a published file-system commit +/// before the catalog exposes the table. Otherwise a filesystem-only client could list an empty +/// `_delta_log/` and "create" a table at the same location. An empty listing here therefore +/// indicates a broken invariant rather than a normal missing version. +#[tracing::instrument(skip(engine), ret, err)] +fn get_earliest_published_commit_version( + engine: &dyn Engine, + log_root: &Url, + earliest_ratified_commit_version: Option, +) -> DeltaResult { + list_delta_log_from_storage(engine.storage_handler().as_ref(), log_root, 0, Version::MAX)? + .filter_ok(|f| f.file_type == LogPathFileType::Commit) + .next() + .transpose()? + .map(|f| f.version) + .ok_or_else(|| { + if earliest_ratified_commit_version == Some(0) { + return DeltaError::generic(format!( + "expected a published v0 commit for catalog-managed table {log_root}, \ + but the log listing returned no commits" + )); + } + DeltaError::from(LogHistoryError::NoCommitsFound { + log_root: log_root.clone(), + }) + }) +} + +/// Returns the earliest table version that can be fully reconstructed, and from which we can replay +/// forward in time to the current version `log_root`. This is either commit version 0 (if +/// `00...00.json` exists), or the version of the earliest complete checkpoint. The returned version +/// is not guaranteed to exist by the time the caller acts on it: a concurrent log-cleanup operation +/// may delete the file. This method assumes that the commits are contiguous. +/// +/// # Parameters +/// - `engine`: kernel engine used to list `log_root`. +/// - `log_root`: URL of the table's `_delta_log/` directory (must end with `/`). +/// - `earliest_ratified_commit_version`: For catalog-managed tables, it is the earliest version the +/// catalog has ratified commit. Pass `None` for filesystem-only tables. +/// +/// # Errors +/// - Propagate any error from listing the log directory. +/// - [`LogHistoryError::NoCommitsFound`] if the log contains no commit files at all +/// (empty directory, or only checkpoint files) -- unless `earliest_ratified_commit_version` +/// is `Some(0)`, in which case it returns a generic [`Error`](crate::Error) flagging the +/// broken CCv2 invariant (ratified commit 0 with no published filesystem commit). +/// - [`LogHistoryError::NoRecreatableCommit`] if commits exist but neither +/// `00...00.json` nor a complete checkpoint that anchors the smallest commit is present. +#[tracing::instrument(skip(engine), err, ret)] +fn get_earliest_recreatable_commit( + engine: &dyn Engine, + log_root: &Url, + earliest_ratified_commit_version: Option, +) -> DeltaResult { + let mut last_complete_checkpoint: Option = None; + // Tracks (version, num_parts) -> set of part numbers observed so far, for multi-part + // checkpoint completeness. + let mut multi_part_checkpoint_progress = HashMap::<(Version, u32), HashSet>::new(); + let mut earliest_commit_version: Option = None; + + let listing = + list_delta_log_from_storage(engine.storage_handler().as_ref(), log_root, 0, Version::MAX)?; + for parsed_result in listing { + let parsed_log_path = parsed_result?; + if !should_process_log_file(&parsed_log_path) { + continue; + } + match parsed_log_path.file_type { + LogPathFileType::Commit => { + if parsed_log_path.version == 0 { + return Ok(0); + } + + let earliest_version = + *earliest_commit_version.get_or_insert(parsed_log_path.version); + + if let Some(checkpoint_version) = last_complete_checkpoint { + if checkpoint_version >= earliest_version { + // Given the contiguity assumption of delta_log commits, + // when a full checkpoint has contiguous commits starting before or at + // checkpoint_version that table can be + // recreated at checkpoint_version. + return Ok(checkpoint_version); + } + } + } + LogPathFileType::ClassicCheckpoint | LogPathFileType::UuidCheckpoint => { + last_complete_checkpoint = Some(parsed_log_path.version); + } + LogPathFileType::MultiPartCheckpoint { + part_num, + num_parts, + } => { + let parts = multi_part_checkpoint_progress + .entry((parsed_log_path.version, num_parts)) + .or_default(); + parts.insert(part_num); + if parts.len() == num_parts as usize { + last_complete_checkpoint = Some(parsed_log_path.version); + } + } + LogPathFileType::StagedCommit + | LogPathFileType::CompactedCommit { .. } + | LogPathFileType::Crc + | LogPathFileType::Unknown => {} + } + } + + // Files are listed in ascending lexicography order, so any recreatable version, e.g commit 0, + // or a complete checkpoint immediately followed by its contiguous commit, is detected and + // returned inside the loop above. Reaching here therefore means no such version exists, + // which is always an error. + if earliest_commit_version.is_some() { + // Commits exist, but none is anchored by commit 0 or a complete checkpoint. + return Err(DeltaError::from(LogHistoryError::NoRecreatableCommit { + log_root: log_root.clone(), + })); + } + if earliest_ratified_commit_version == Some(0) { + // Broken CCv2 invariant: the catalog ratified commit 0, but no published commit exists. + return Err(DeltaError::generic(format!( + "expected a published v0 commit for catalog-managed table {log_root}, \ + but the log listing returned no commits" + ))); + } + Err(DeltaError::from(LogHistoryError::NoCommitsFound { + log_root: log_root.clone(), + })) +} + +/// Selects which commit the [`get_earliest_commit`] query returns. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum HistoryCommitType { + /// A filesystem or catalog commit present in the table's `_delta_log/`. Its presence does not + /// guarantee that the table can be reconstructed at that commit's version. + Published, + /// A commit whose version the table state can be fully reconstructed at and replayed forward + /// from to the latest version. + Recreatable, +} + +/// Returns the earliest table version available on the file system at `log_root`. The returned +/// version is not guaranteed to exist by the time the caller acts on it: a concurrent log-cleanup +/// operation may delete the underlying file. +/// +/// # Parameters +/// - `engine`: kernel engine used to list `log_root`. +/// - `log_root`: URL of the table's `_delta_log/` directory (must end with `/`). +/// - `earliest_ratified_commit_version`: For catalog-managed tables, the earliest version the +/// catalog has ratified a commit at. Pass `None` for filesystem-only tables. +/// - `commit_type`: selects the query. [`HistoryCommitType::Published`] returns the lowest-numbered +/// published commit; [`HistoryCommitType::Recreatable`] returns the earliest version whose state +/// can be fully reconstructed (commit 0 or the earliest complete checkpoint). +/// +/// # Errors +/// - Propagates any error from listing the log directory. +/// - [`LogHistoryError::NoCommitsFound`] when the log directory contains no commits and +/// `earliest_ratified_commit_version` is not `Some(0)`. +/// - [`LogHistoryError::NoRecreatableCommit`] when `commit_type` is +/// [`HistoryCommitType::Recreatable`], commits exist, but neither `00...00.json` nor a checkpoint +/// anchoring the smallest commit is present. +/// - [`DeltaError::Generic`] when the listing yields no commits and +/// `earliest_ratified_commit_version` is `Some(0)`, flagging a broken catalog-managed invariant +/// (ratified commit 0 with no published filesystem commit). +#[tracing::instrument(skip(engine), err, ret)] +pub fn get_earliest_commit( + engine: &dyn Engine, + log_root: &Url, + earliest_ratified_commit_version: Option, + commit_type: HistoryCommitType, +) -> DeltaResult { + match commit_type { + HistoryCommitType::Published => get_earliest_published_commit_version( + engine, + log_root, + earliest_ratified_commit_version, + ), + + HistoryCommitType::Recreatable => { + get_earliest_recreatable_commit(engine, log_root, earliest_ratified_commit_version) + } + } +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + use std::fs::{remove_file, OpenOptions}; + use std::ops::RangeInclusive; + use std::sync::Arc; + use std::time::{Duration, SystemTime}; + + use bytes::Bytes; + use test_utils::delta_path_for_version; + use url::Url; + use uuid::Uuid; + + use super::*; + use crate::actions::{CommitInfo, Metadata, Protocol}; + use crate::engine::sync::SyncEngine; + use crate::object_store::memory::InMemory; + use crate::schema::{DataType, SchemaRef, StructField, StructType}; + use crate::snapshot::{Snapshot, SnapshotBuilder}; + use crate::table_features::TableFeature; + use crate::unit_test_utils::{Action, LocalMockTable}; + use crate::utils::FoldWithOption as _; + use crate::Version; + + fn get_test_schema() -> SchemaRef { + Arc::new(StructType::new_unchecked([StructField::nullable( + "value", + DataType::INTEGER, + )])) + } + + fn set_mod_time(mock_table: &LocalMockTable, version: Version, timestamp: Timestamp) { + let file_name = delta_path_for_version(version, "json") + .filename() + .unwrap() + .to_string(); + let path = mock_table.table_root().join("_delta_log/").join(file_name); + let file = OpenOptions::new().write(true).open(path).unwrap(); + let time = SystemTime::UNIX_EPOCH + Duration::from_millis(timestamp.try_into().unwrap()); + file.set_modified(time).unwrap(); + } + + /// Creates a test table with specified timestamps. + /// + /// # Arguments + /// * `timestamps` - List of `(file_mod_ts, Option)` for each version + /// * `ict_enablement_version` - Version where ICT is enabled: + /// - `None`: ICT never enabled (file mod only) + /// - `Some(0)`: ICT enabled from creation + /// - `Some(n)`: ICT enabled at version n + async fn mock_table_with_timestamps( + timestamps: &[(Timestamp, Option)], + ict_enablement_version: Option, + ) -> LocalMockTable { + let mut mock_table = LocalMockTable::new(); + + for (version, (file_mod_ts, ict_ts)) in timestamps.iter().enumerate() { + let version = version as Version; + let is_first = version == 0; + let is_ict_enablement = ict_enablement_version == Some(version); + let ict_from_creation = ict_enablement_version == Some(0) && is_first; + + let mut actions: Vec = vec![]; + + // CommitInfo with optional ICT (must be first when ICT is present) + if ict_ts.is_some() { + actions.push(Action::CommitInfo(CommitInfo { + in_commit_timestamp: *ict_ts, + ..Default::default() + })); + } + + // Metadata: on first commit, or when enabling ICT + if is_first || is_ict_enablement { + let config = if ict_from_creation { + // ICT enabled from creation: no enablement version/timestamp + HashMap::from_iter([( + "delta.enableInCommitTimestamps".to_string(), + "true".to_string(), + )]) + } else if is_ict_enablement { + // ICT enabled at this version + HashMap::from_iter([ + ( + "delta.enableInCommitTimestamps".to_string(), + "true".to_string(), + ), + ( + "delta.inCommitTimestampEnablementVersion".to_string(), + version.to_string(), + ), + ( + "delta.inCommitTimestampEnablementTimestamp".to_string(), + ict_ts.unwrap().to_string(), + ), + ]) + } else { + HashMap::new() + }; + actions.push(Action::Metadata( + Metadata::try_new(None, None, get_test_schema(), vec![], 0, config).unwrap(), + )); + } + + // Protocol on first commit + if is_first { + let writer_features: Option> = if ict_enablement_version.is_some() + { + Some(vec![TableFeature::InCommitTimestamp]) + } else { + Some(vec![]) + }; + actions.push(Action::Protocol( + Protocol::try_new(3, 7, Some(Vec::::new()), writer_features).unwrap(), + )); + } + + // If no actions yet (non-first commit without ICT), add empty CommitInfo + if actions.is_empty() { + actions.push(Action::CommitInfo(CommitInfo::default())); + } + + mock_table.commit(actions).await; + set_mod_time(&mock_table, version, *file_mod_ts); + } + + mock_table + } + + /// Helper to build snapshot and log segment + async fn build_test_snapshot( + timestamps: &[(Timestamp, Option)], + ict_enablement: Option, + at_version: Option, + ) -> (LocalMockTable, SyncEngine, Arc, LogSegment) { + let table = mock_table_with_timestamps(timestamps, ict_enablement).await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path) + .fold_with(at_version, SnapshotBuilder::at_version) + .build(&engine) + .unwrap(); + let log_segment = LogSegment::for_timestamp_conversion( + engine.storage_handler().as_ref(), + snapshot.log_segment().log_root.clone(), + snapshot.version(), + None, + vec![], + ) + .unwrap(); + (table, engine, snapshot, log_segment) + } + + // ==================================================================================== + // Tests for low-level internal functions: get_timestamp_search_bounds, + // timestamp_to_version, linear_search_file_mod_timestamps + // ==================================================================================== + + // Table: v0=50, v1=150, v2=250 (file mod only), snapshot at v2 (no ICT) + #[rstest::rstest] + #[case::before_all(0)] + #[case::within_range(100)] + #[case::after_all(1000)] + #[tokio::test] + async fn test_search_bounds_no_ict(#[case] timestamp: Timestamp) { + let timestamps = [ + (50, None), + (150, None), + (250, None), + (350, Some(300)), + (450, Some(400)), + ]; + let (_table, _engine, snapshot, log_segment) = + build_test_snapshot(×tamps, Some(3), Some(2)).await; + + let res = get_timestamp_search_bounds(&snapshot, &log_segment, timestamp).unwrap(); + assert!( + matches!(res, TimestampSearchBounds::FileModificationSearch), + "{res:?}" + ); + } + + // Table: v0=50, v1=150, v2=250 (file mod), v3=300, v4=400 (ICT enabled at v3) + #[rstest::rstest] + #[case::file_mod_region(50, TimestampSearchBounds::FileModificationSearchUntil { index: 3, ict_enablement_version: 3, ict_enablement_timestamp: 300 })] + #[case::file_mod_between(60, TimestampSearchBounds::FileModificationSearchUntil { index: 3, ict_enablement_version: 3, ict_enablement_timestamp: 300 })] + #[case::file_mod_last(299, TimestampSearchBounds::FileModificationSearchUntil { index: 3, ict_enablement_version: 3, ict_enablement_timestamp: 300 })] + #[case::exact_enablement(300, TimestampSearchBounds::ExactMatch(3))] + #[case::ict_after_enablement(301, TimestampSearchBounds::ICTSearchStartingFrom(3))] + #[case::ict_far_future(1000, TimestampSearchBounds::ICTSearchStartingFrom(3))] + #[tokio::test] + async fn test_search_bounds_with_ict( + #[case] timestamp: Timestamp, + #[case] expected: TimestampSearchBounds, + ) { + let timestamps = [ + (50, None), + (150, None), + (250, None), + (350, Some(300)), + (450, Some(400)), + ]; + let (_table, _engine, snapshot, log_segment) = + build_test_snapshot(×tamps, Some(3), None).await; + + let res = get_timestamp_search_bounds(&snapshot, &log_segment, timestamp).unwrap(); + assert_eq!(res, expected); + } + + // Table: v0=100, v1=200, v2=300 (ICT from creation) + #[rstest::rstest] + #[case::before_all(50)] + #[case::at_v0(100)] + #[case::between_v0_v1(150)] + #[case::after_all(1000)] + #[tokio::test] + async fn test_search_bounds_ict_from_creation(#[case] timestamp: Timestamp) { + let timestamps = [(0, Some(100)), (0, Some(200)), (0, Some(300))]; + let (_table, _engine, snapshot, log_segment) = + build_test_snapshot(×tamps, Some(0), None).await; + + let res = get_timestamp_search_bounds(&snapshot, &log_segment, timestamp).unwrap(); + assert!( + matches!(res, TimestampSearchBounds::ICTSearchStartingFrom(0)), + "{res:?}" + ); + } + + /// Tests the fast path: when pre-ICT commits are cleaned up (e.g., via VACUUM), the first + /// commit in the log segment is at or after ICT enablement. All visible commits have ICT, + /// so we return ICTSearchStartingFrom(0) directly without binary search. + #[rstest::rstest] + #[case::before_all(50)] + #[case::at_ict_enablement(300)] + #[case::after_ict_enablement(350)] + #[case::after_all(1000)] + #[tokio::test] + async fn test_search_bounds_fast_path_pre_ict_commits_cleaned_up(#[case] timestamp: Timestamp) { + // Table: ICT enabled at v3 with ts=300, but only v3+ remain after cleanup + let timestamps = [ + (50, None), + (150, None), + (250, None), + (350, Some(300)), + (450, Some(400)), + ]; + let (table, engine, snapshot, _log_segment) = + build_test_snapshot(×tamps, Some(3), None).await; + + // Simulate cleanup: delete pre-ICT commit files (v0, v1, v2) + let log_dir = table.table_root().join("_delta_log"); + for version in 0..3 { + let file_name = format!("{:020}.json", version); + std::fs::remove_file(log_dir.join(&file_name)).unwrap(); + } + + // Build a new log segment that only sees v3+ (simulating post-cleanup state) + let cleaned_log_segment = LogSegment::for_timestamp_conversion( + engine.storage_handler().as_ref(), + snapshot.log_segment().log_root.clone(), + snapshot.version(), + None, + vec![], + ) + .unwrap(); + + // Verify the log segment only contains v3+ (first_version=3 >= ict_enablement_version=3) + assert_eq!( + cleaned_log_segment.listed.ascending_commit_files[0].version, + 3 + ); + + // The fast path should trigger: returns ICTSearchStartingFrom(0) for any timestamp + let res = get_timestamp_search_bounds(&snapshot, &cleaned_log_segment, timestamp).unwrap(); + assert!( + matches!(res, TimestampSearchBounds::ICTSearchStartingFrom(0)), + "Expected fast path ICTSearchStartingFrom(0), got {res:?}" + ); + } + + /// Tests reading timestamps from commits - both file modification and ICT. + #[tokio::test] + async fn test_reading_commit_timestamps() { + // Table: v0=50, v1=150, v2=250 (file mod), v3=300, v4=400 (ICT at v3) + let timestamps = [ + (50, None), + (150, None), + (250, None), + (350, Some(300)), + (450, Some(400)), + ]; + let (_table, engine, _snapshot, log_segment) = + build_test_snapshot(×tamps, Some(3), None).await; + let commits = &log_segment.listed.ascending_commit_files; + + // File modification timestamps are set correctly + assert_eq!(commits[0].location.last_modified, 50); + assert_eq!(commits[1].location.last_modified, 150); + assert_eq!(commits[2].location.last_modified, 250); + + // Reading ICT from file without ICT returns error + let res = commits[0].read_in_commit_timestamp(&engine); + assert!(res.is_err(), "Expected error for file without ICT: {res:?}"); + + // Reading ICT from non-existent file returns error + let mut fake_log_path = commits[0].clone(); + let failing_path = if cfg!(windows) { + "C:\\phony\\path" + } else { + "/phony/path" + }; + fake_log_path.location.location = Url::from_file_path(failing_path).unwrap(); + let res = fake_log_path.read_in_commit_timestamp(&engine); + assert!( + res.is_err(), + "Expected error for non-existent file: {res:?}" + ); + + // In-commit timestamps read correctly + assert_eq!(commits[3].read_in_commit_timestamp(&engine).unwrap(), 300); + assert_eq!(commits[4].read_in_commit_timestamp(&engine).unwrap(), 400); + } + + // Table: v0=50, v1=150, v2=250 (file mod), v3=300, v4=400 (ICT at v3) + #[rstest::rstest] + #[case::before_all_lub(0, Bound::LeastUpper, Some(CommitAt::new(0, 50)))] + #[case::before_all_glb(0, Bound::GreatestLower, None)] + #[case::negative_lub(-1, Bound::LeastUpper, Some(CommitAt::new(0, 50)))] + #[case::after_all_lub(1000, Bound::LeastUpper, None)] + #[case::after_all_glb(1000, Bound::GreatestLower, Some(CommitAt::new(4, 400)))] + #[case::exact_v0_lub(50, Bound::LeastUpper, Some(CommitAt::new(0, 50)))] + #[case::exact_v0_glb(50, Bound::GreatestLower, Some(CommitAt::new(0, 50)))] + #[case::exact_v1_lub(150, Bound::LeastUpper, Some(CommitAt::new(1, 150)))] + #[case::exact_v1_glb(150, Bound::GreatestLower, Some(CommitAt::new(1, 150)))] + #[case::exact_v3_ict_lub(300, Bound::LeastUpper, Some(CommitAt::new(3, 300)))] + #[case::exact_v3_ict_glb(300, Bound::GreatestLower, Some(CommitAt::new(3, 300)))] + #[case::between_v0_v1_lub(100, Bound::LeastUpper, Some(CommitAt::new(1, 150)))] + #[case::between_v0_v1_glb(100, Bound::GreatestLower, Some(CommitAt::new(0, 50)))] + #[case::just_after_last_file_mod_lub(251, Bound::LeastUpper, Some(CommitAt::new(3, 300)))] + #[case::just_after_last_file_mod_glb(251, Bound::GreatestLower, Some(CommitAt::new(2, 250)))] + #[case::just_before_ict_lub(299, Bound::LeastUpper, Some(CommitAt::new(3, 300)))] + #[case::just_before_ict_glb(299, Bound::GreatestLower, Some(CommitAt::new(2, 250)))] + #[case::just_after_ict_glb(301, Bound::GreatestLower, Some(CommitAt::new(3, 300)))] + #[case::just_after_ict_lub(301, Bound::LeastUpper, Some(CommitAt::new(4, 400)))] + #[case::between_ict_commits(350, Bound::GreatestLower, Some(CommitAt::new(3, 300)))] + #[case::between_ict_commits_lub(350, Bound::LeastUpper, Some(CommitAt::new(4, 400)))] + #[tokio::test] + async fn test_timestamp_to_version_standard_table( + #[case] timestamp: Timestamp, + #[case] bound: Bound, + #[case] expected: Option, + ) { + let timestamps = [ + (50, None), + (150, None), + (250, None), + (350, Some(300)), + (450, Some(400)), + ]; + let (_table, engine, snapshot, _log_segment) = + build_test_snapshot(×tamps, Some(3), None).await; + + let res = timestamp_to_version( + &snapshot, + &engine, + timestamp, + bound, + HistoryCommitType::Published, + ); + match expected { + Some(commit) => assert_eq!(res.unwrap(), commit), + None => assert!( + matches!(res, Err(LogHistoryError::TimestampOutOfRange { .. })), + "{res:?}" + ), + } + } + + // Table: v0=100, v1=200, v2=300 (ICT from creation) + #[rstest::rstest] + #[case::exact_v0_glb(100, Bound::GreatestLower, Some(CommitAt::new(0, 100)))] + #[case::exact_v0_lub(100, Bound::LeastUpper, Some(CommitAt::new(0, 100)))] + #[case::between_v0_v1_glb(150, Bound::GreatestLower, Some(CommitAt::new(0, 100)))] + #[case::between_v0_v1_lub(150, Bound::LeastUpper, Some(CommitAt::new(1, 200)))] + #[case::between_v1_v2_glb(250, Bound::GreatestLower, Some(CommitAt::new(1, 200)))] + #[case::between_v1_v2_lub(250, Bound::LeastUpper, Some(CommitAt::new(2, 300)))] + #[case::before_all_glb(50, Bound::GreatestLower, None)] + #[case::before_all_lub(50, Bound::LeastUpper, Some(CommitAt::new(0, 100)))] + #[tokio::test] + async fn test_ict_from_creation( + #[case] timestamp: Timestamp, + #[case] bound: Bound, + #[case] expected: Option, + ) { + let mock_table = + mock_table_with_timestamps(&[(0, Some(100)), (0, Some(200)), (0, Some(300))], Some(0)) + .await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(mock_table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + let res = timestamp_to_version( + &snapshot, + &engine, + timestamp, + bound, + HistoryCommitType::Published, + ); + match expected { + Some(commit) => assert_eq!(res.unwrap(), commit), + None => assert!( + matches!(res, Err(LogHistoryError::TimestampOutOfRange { .. })), + "{res:?}" + ), + } + } + + /// Single commit table: v0=100 (file mod only) + #[rstest::rstest] + #[case::exact_glb(100, Bound::GreatestLower, Some(CommitAt::new(0, 100)))] + #[case::exact_lub(100, Bound::LeastUpper, Some(CommitAt::new(0, 100)))] + #[case::before_glb(50, Bound::GreatestLower, None)] + #[case::before_lub(50, Bound::LeastUpper, Some(CommitAt::new(0, 100)))] + #[case::after_glb(150, Bound::GreatestLower, Some(CommitAt::new(0, 100)))] + #[case::after_lub(150, Bound::LeastUpper, None)] + #[tokio::test] + async fn test_single_commit( + #[case] timestamp: Timestamp, + #[case] bound: Bound, + #[case] expected: Option, + ) { + let mock_table = mock_table_with_timestamps(&[(100, None)], None).await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(mock_table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + let res = timestamp_to_version( + &snapshot, + &engine, + timestamp, + bound, + HistoryCommitType::Published, + ); + match expected { + Some(commit) => assert_eq!(res.unwrap(), commit), + None => assert!( + matches!(res, Err(LogHistoryError::TimestampOutOfRange { .. })), + "{res:?}" + ), + } + } + + /// Non-monotonic file mod timestamps (clock skew): + /// Raw: v0=100, v1=200, v2=150, v3=180, v4=300 + /// Monotonized: v0=100, v1=200, v2=201, v3=202, v4=300 + #[rstest::rstest] + #[case::exact_v0_glb(100, Bound::GreatestLower, Some(CommitAt::new(0, 100)))] + #[case::exact_v1_glb(200, Bound::GreatestLower, Some(CommitAt::new(1, 200)))] + #[case::monotonized_v2_glb(201, Bound::GreatestLower, Some(CommitAt::new(2, 201)))] + #[case::monotonized_v3_glb(202, Bound::GreatestLower, Some(CommitAt::new(3, 202)))] + #[case::between_v3_v4_glb(250, Bound::GreatestLower, Some(CommitAt::new(3, 202)))] + #[case::exact_v4_glb(300, Bound::GreatestLower, Some(CommitAt::new(4, 300)))] + #[case::raw_v2_maps_to_v1_lub(150, Bound::LeastUpper, Some(CommitAt::new(1, 200)))] + #[case::monotonized_v2_lub(201, Bound::LeastUpper, Some(CommitAt::new(2, 201)))] + #[case::after_v3_monotonized_lub(203, Bound::LeastUpper, Some(CommitAt::new(4, 300)))] + #[tokio::test] + async fn test_non_monotonic_timestamps( + #[case] timestamp: Timestamp, + #[case] bound: Bound, + #[case] expected: Option, + ) { + let mock_table = mock_table_with_timestamps( + &[ + (100, None), + (200, None), + (150, None), + (180, None), + (300, None), + ], + None, + ) + .await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(mock_table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + let res = timestamp_to_version( + &snapshot, + &engine, + timestamp, + bound, + HistoryCommitType::Published, + ); + match expected { + Some(commit) => assert_eq!(res.unwrap(), commit), + None => assert!( + matches!(res, Err(LogHistoryError::TimestampOutOfRange { .. })), + "{res:?}" + ), + } + } + + /// Verifies `nearest_timestamp` is populated on `TimestampOutOfRange`. Covers + /// the three reachable populate sites: linear file-mod search out-of-range, + /// ICT binary-search out-of-range, and the snapshot short-circuit. + #[rstest::rstest] + // Linear file-mod GLB: timestamp below earliest commit on the standard table. + // nearest = Earliest(commits[0].last_modified) = Earliest(50). + #[case::linear_glb_below_earliest( + &[(50, None), (150, None), (250, None), (350, Some(300)), (450, Some(400))], + Some(3), + 0, + Bound::GreatestLower, + NearestTimestamp::Earliest(50), + )] + // Snapshot short-circuit LUB: timestamp above the snapshot's ICT (snap_ts = 400). + #[case::short_circuit_lub_after_snapshot( + &[(50, None), (150, None), (250, None), (350, Some(300)), (450, Some(400))], + Some(3), + 1000, + Bound::LeastUpper, + NearestTimestamp::Latest(400), + )] + // ICT binary-search GLB: timestamp below earliest ICT on an ICT-from-creation + // table. nearest = Earliest(ICT(v0)) = Earliest(100). + #[case::ict_glb_below_earliest( + &[(0, Some(100)), (0, Some(200)), (0, Some(300))], + Some(0), + 50, + Bound::GreatestLower, + NearestTimestamp::Earliest(100), + )] + #[tokio::test] + async fn test_timestamp_out_of_range_nearest_timestamp( + #[case] timestamps: &[(Timestamp, Option)], + #[case] ict_enablement: Option, + #[case] timestamp: Timestamp, + #[case] bound: Bound, + #[case] expected_nearest: NearestTimestamp, + ) { + let (_table, engine, snapshot, _log_segment) = + build_test_snapshot(timestamps, ict_enablement, None).await; + + let err = timestamp_to_version( + &snapshot, + &engine, + timestamp, + bound, + HistoryCommitType::Published, + ) + .unwrap_err(); + let LogHistoryError::TimestampOutOfRange { + nearest_timestamp, .. + } = err + else { + panic!("expected TimestampOutOfRange, got {err:?}"); + }; + assert_eq!(nearest_timestamp, expected_nearest); + } + + /// ICT enabled at v3 but v4 is missing its ICT timestamp (error case). + /// Table: v0=50, v1=150, v2=250 (file mod), v3=300 (ICT), v4=450 (missing ICT) + #[tokio::test] + async fn test_missing_ict_after_enablement() { + // Create table where v4 is after ICT enablement but doesn't have an ICT + let timestamps = [ + (50, None), + (150, None), + (250, None), + (350, Some(300)), // ICT enabled at v3 + (450, None), // v4 missing ICT (bug in writer!) + ]; + let table = mock_table_with_timestamps(×tamps, Some(3)).await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + // Search for timestamp 350 which would need to search in ICT range (v3+) + // The search will try to read ICT from v4 which is missing + let res = timestamp_to_version( + &snapshot, + &engine, + 350, + Bound::LeastUpper, + HistoryCommitType::Published, + ); + assert!( + matches!(res, Err(LogHistoryError::Internal { .. })), + "Expected internal error for missing ICT: {res:?}" + ); + } + + // ==================================================================================== + // Tests for high-level public APIs: latest_version_as_of, first_version_after, + // timestamp_range_to_versions + // ==================================================================================== + + // Table: v0=50, v1=150, v2=250 (file mod), v3=300, v4=400 (ICT at v3) + // Matches test_timestamp_to_version_standard_table cases for GreatestLower bound + #[rstest::rstest] + #[case::before_all(0, None)] + #[case::exact_v0(50, Some(CommitAt::new(0, 50)))] + #[case::between_v0_v1(100, Some(CommitAt::new(0, 50)))] + #[case::exact_v1(150, Some(CommitAt::new(1, 150)))] + #[case::just_after_last_file_mod(251, Some(CommitAt::new(2, 250)))] + #[case::just_before_ict(299, Some(CommitAt::new(2, 250)))] + #[case::exact_ict_enablement(300, Some(CommitAt::new(3, 300)))] + #[case::just_after_ict(301, Some(CommitAt::new(3, 300)))] + #[case::between_ict_commits(350, Some(CommitAt::new(3, 300)))] + #[case::after_all(1000, Some(CommitAt::new(4, 400)))] + #[tokio::test] + async fn test_latest_version_as_of( + #[case] timestamp: Timestamp, + #[case] expected: Option, + ) { + let timestamps = [ + (50, None), + (150, None), + (250, None), + (350, Some(300)), + (450, Some(400)), + ]; + let table = mock_table_with_timestamps(×tamps, Some(3)).await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + let res = latest_version_as_of(&snapshot, &engine, timestamp, HistoryCommitType::Published); + match expected { + Some(commit) => assert_eq!(res.unwrap(), commit), + None => assert!(res.is_err(), "{res:?}"), + } + } + + // Table: v0=50, v1=150, v2=250 (file mod), v3=300, v4=400 (ICT at v3) + // Matches test_timestamp_to_version_standard_table cases for LeastUpper bound + #[rstest::rstest] + #[case::before_all(0, Some(CommitAt::new(0, 50)))] + #[case::exact_v0(50, Some(CommitAt::new(0, 50)))] + #[case::between_v0_v1(100, Some(CommitAt::new(1, 150)))] + #[case::exact_v1(150, Some(CommitAt::new(1, 150)))] + #[case::just_after_last_file_mod(251, Some(CommitAt::new(3, 300)))] + #[case::just_before_ict(299, Some(CommitAt::new(3, 300)))] + #[case::exact_ict_enablement(300, Some(CommitAt::new(3, 300)))] + #[case::just_after_ict(301, Some(CommitAt::new(4, 400)))] + #[case::between_ict_commits(350, Some(CommitAt::new(4, 400)))] + #[case::after_all(1000, None)] + #[tokio::test] + async fn test_first_version_after( + #[case] timestamp: Timestamp, + #[case] expected: Option, + ) { + let timestamps = [ + (50, None), + (150, None), + (250, None), + (350, Some(300)), + (450, Some(400)), + ]; + let table = mock_table_with_timestamps(×tamps, Some(3)).await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + let res = first_version_after(&snapshot, &engine, timestamp, HistoryCommitType::Published); + match expected { + Some(commit) => assert_eq!(res.unwrap(), commit), + None => assert!(res.is_err(), "{res:?}"), + } + } + + /// Table: v0=50, v1=150, v2=250 (file mod), v3=300, v4=400 (ICT at v3) + #[rstest::rstest] + #[case::basic_range(50, Some(300), Ok((0, Some(3))))] + #[case::no_end(100, None, Ok((1, None)))] + #[case::start_equals_end(150, Some(150), Ok((1, Some(1))))] + #[case::spanning_ict_boundary(100, Some(350), Ok((1, Some(3))))] + #[case::entire_table(0, Some(1000), Ok((0, Some(4))))] + #[case::exact_v0_to_v1(50, Some(150), Ok((0, Some(1))))] + #[case::exact_v1_to_v2(150, Some(250), Ok((1, Some(2))))] + #[case::exact_v3_ict_to_v4(300, Some(400), Ok((3, Some(4))))] + #[case::between_v0_v1_to_exact_v2(100, Some(250), Ok((1, Some(2))))] + #[case::just_after_last_file_mod_no_end(251, None, Ok((3, None)))] + #[case::just_before_ict_to_just_after(299, Some(301), Ok((3, Some(3))))] + #[case::exact_ict_to_after_all(300, Some(1000), Ok((3, Some(4))))] + #[case::between_ict_commits(350, Some(400), Ok((4, Some(4))))] + #[case::just_after_ict_to_end(301, Some(1000), Ok((4, Some(4))))] + #[case::end_before_all_fails(0, Some(40), Err(()))] + #[case::start_after_all_fails(500, Some(1000), Err(()))] + #[tokio::test] + async fn test_timestamp_range_to_versions( + #[case] start: Timestamp, + #[case] end: Option, + #[case] expected: Result<(Version, Option), ()>, + ) { + let timestamps = [ + (50, None), + (150, None), + (250, None), + (350, Some(300)), + (450, Some(400)), + ]; + let table = mock_table_with_timestamps(×tamps, Some(3)).await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + let res = timestamp_range_to_versions(&snapshot, &engine, start, end); + match expected { + Ok(v) => assert_eq!(res.unwrap(), v), + Err(()) => assert!(res.is_err(), "{res:?}"), + } + } + + #[tokio::test] + async fn test_timestamp_range_invalid_range() { + // Table: v0=50, v1=150, v2=250 (file mod), v3=300, v4=400 (ICT at v3) + let timestamps = [ + (50, None), + (150, None), + (250, None), + (350, Some(300)), + (450, Some(400)), + ]; + let table = mock_table_with_timestamps(×tamps, Some(3)).await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + let res = timestamp_range_to_versions(&snapshot, &engine, 300, Some(100)); + assert!( + matches!( + res, + Err(crate::Error::LogHistory(ref e)) + if matches!(**e, LogHistoryError::InvalidTimestampRange { .. }) + ), + "{res:?}" + ); + } + + #[tokio::test] + async fn test_timestamp_range_empty_range() { + // Table: v0=50, v1=150, v2=250 (file mod), v3=300, v4=400 (ICT at v3) + let timestamps = [ + (50, None), + (150, None), + (250, None), + (350, Some(300)), + (450, Some(400)), + ]; + let table = mock_table_with_timestamps(×tamps, Some(3)).await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + // Timestamps 51-149 fall between v0 (ts=50) and v1 (ts=150) + let res = timestamp_range_to_versions(&snapshot, &engine, 51, Some(149)); + assert!( + matches!( + res, + Err(crate::Error::LogHistory(ref e)) + if matches!(**e, LogHistoryError::EmptyTimestampRange { between_version: 0, .. }) + ), + "{res:?}" + ); + } + + // ==================================================================================== + // High-level API tests for additional table configurations + // ==================================================================================== + + /// File-mod-only table (no ICT): v0=100, v1=200, v2=300 + #[rstest::rstest] + #[case::exact_match(200, Some(CommitAt::new(1, 200)))] + #[case::between_commits(150, Some(CommitAt::new(0, 100)))] + #[case::after_all(500, Some(CommitAt::new(2, 300)))] + #[case::before_all(50, None)] + #[tokio::test] + async fn test_latest_version_as_of_file_mod_only( + #[case] timestamp: Timestamp, + #[case] expected: Option, + ) { + let table = + mock_table_with_timestamps(&[(100, None), (200, None), (300, None)], None).await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + let res = latest_version_as_of(&snapshot, &engine, timestamp, HistoryCommitType::Published); + match expected { + Some(commit) => assert_eq!(res.unwrap(), commit), + None => assert!(res.is_err(), "{res:?}"), + } + } + + /// File-mod-only table (no ICT): v0=100, v1=200, v2=300 + #[rstest::rstest] + #[case::exact_match(200, Some(CommitAt::new(1, 200)))] + #[case::between_commits(150, Some(CommitAt::new(1, 200)))] + #[case::before_all(50, Some(CommitAt::new(0, 100)))] + #[case::after_all(500, None)] + #[tokio::test] + async fn test_first_version_after_file_mod_only( + #[case] timestamp: Timestamp, + #[case] expected: Option, + ) { + let table = + mock_table_with_timestamps(&[(100, None), (200, None), (300, None)], None).await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + let res = first_version_after(&snapshot, &engine, timestamp, HistoryCommitType::Published); + match expected { + Some(commit) => assert_eq!(res.unwrap(), commit), + None => assert!(res.is_err(), "{res:?}"), + } + } + + /// ICT from creation: v0=100, v1=200, v2=300 (all ICT) + #[rstest::rstest] + #[case::exact_match(200, Some(CommitAt::new(1, 200)))] + #[case::between_commits(150, Some(CommitAt::new(0, 100)))] + #[case::after_all(500, Some(CommitAt::new(2, 300)))] + #[case::before_all(50, None)] + #[tokio::test] + async fn test_latest_version_as_of_ict_from_creation( + #[case] timestamp: Timestamp, + #[case] expected: Option, + ) { + let table = + mock_table_with_timestamps(&[(0, Some(100)), (0, Some(200)), (0, Some(300))], Some(0)) + .await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + let res = latest_version_as_of(&snapshot, &engine, timestamp, HistoryCommitType::Published); + match expected { + Some(commit) => assert_eq!(res.unwrap(), commit), + None => assert!(res.is_err(), "{res:?}"), + } + } + + /// ICT from creation: v0=100, v1=200, v2=300 (all ICT) + #[rstest::rstest] + #[case::exact_match(200, Some(CommitAt::new(1, 200)))] + #[case::between_commits(150, Some(CommitAt::new(1, 200)))] + #[case::before_all(50, Some(CommitAt::new(0, 100)))] + #[case::after_all(500, None)] + #[tokio::test] + async fn test_first_version_after_ict_from_creation( + #[case] timestamp: Timestamp, + #[case] expected: Option, + ) { + let table = + mock_table_with_timestamps(&[(0, Some(100)), (0, Some(200)), (0, Some(300))], Some(0)) + .await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + let res = first_version_after(&snapshot, &engine, timestamp, HistoryCommitType::Published); + match expected { + Some(commit) => assert_eq!(res.unwrap(), commit), + None => assert!(res.is_err(), "{res:?}"), + } + } + + /// Non-monotonic file mod timestamps (clock skew): + /// Raw: v0=100, v1=200, v2=150, v3=180, v4=300 + /// Monotonized: v0=100, v1=200, v2=201, v3=202, v4=300 + #[rstest::rstest] + #[case::exact_v1(200, Some(CommitAt::new(1, 200)))] + #[case::monotonized_v2(201, Some(CommitAt::new(2, 201)))] + #[case::monotonized_v3(202, Some(CommitAt::new(3, 202)))] + #[case::between_v3_v4(250, Some(CommitAt::new(3, 202)))] + #[case::raw_v2_between_v0_v1(150, Some(CommitAt::new(0, 100)))] + #[tokio::test] + async fn test_latest_version_as_of_non_monotonic( + #[case] timestamp: Timestamp, + #[case] expected: Option, + ) { + let table = mock_table_with_timestamps( + &[ + (100, None), + (200, None), + (150, None), // Clock skew - earlier than v1 + (180, None), // Clock skew - earlier than v1 + (300, None), + ], + None, + ) + .await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + let res = latest_version_as_of(&snapshot, &engine, timestamp, HistoryCommitType::Published); + match expected { + Some(commit) => assert_eq!(res.unwrap(), commit), + None => assert!(res.is_err(), "{res:?}"), + } + } + + /// Non-monotonic file mod timestamps (clock skew): + /// Raw: v0=100, v1=200, v2=150, v3=180, v4=300 + /// Monotonized: v0=100, v1=200, v2=201, v3=202, v4=300 + #[rstest::rstest] + #[case::exact_v1(200, Some(CommitAt::new(1, 200)))] + #[case::monotonized_v2(201, Some(CommitAt::new(2, 201)))] + #[case::monotonized_v3(202, Some(CommitAt::new(3, 202)))] + #[case::after_v3_monotonized(203, Some(CommitAt::new(4, 300)))] + #[case::raw_v2_maps_to_v1(150, Some(CommitAt::new(1, 200)))] + #[tokio::test] + async fn test_first_version_after_non_monotonic( + #[case] timestamp: Timestamp, + #[case] expected: Option, + ) { + let table = mock_table_with_timestamps( + &[ + (100, None), + (200, None), + (150, None), // Clock skew - earlier than v1 + (180, None), // Clock skew - earlier than v1 + (300, None), + ], + None, + ) + .await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + let res = first_version_after(&snapshot, &engine, timestamp, HistoryCommitType::Published); + match expected { + Some(commit) => assert_eq!(res.unwrap(), commit), + None => assert!(res.is_err(), "{res:?}"), + } + } + + /// Builds a table from `timestamps` (file-modification times, one per version starting at + /// v0, no ICT), then simulates log cleanup: writes an empty placeholder checkpoint at + /// `checkpoint_version` and deletes commit files for versions + /// `0..=log_cleanup_until_version`. + async fn build_mock_table_with_log_cleanup( + timestamps: &[Timestamp], + checkpoint_version: Version, + log_cleanup_until_version: Version, + ) -> (LocalMockTable, SyncEngine, Arc) { + let timestamps: Vec<_> = timestamps.iter().map(|&ts| (ts, None)).collect(); + let table = mock_table_with_timestamps(×tamps, None).await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + let log_dir = table.table_root().join("_delta_log"); + std::fs::write( + log_dir.join(format!("{checkpoint_version:020}.checkpoint.parquet")), + b"x", + ) + .unwrap(); + for version in 0..=log_cleanup_until_version { + remove_file(log_dir.join(format!("{version:020}.json"))).unwrap(); + } + + (table, engine, snapshot) + } + + #[rstest::rstest] + #[case::published_glb_orphan_v1( + HistoryCommitType::Published, + 15, + Bound::GreatestLower, + Some(CommitAt::new(1, 10)) + )] + #[case::recreatable_glb_orphan_v1( + HistoryCommitType::Recreatable, + 15, + Bound::GreatestLower, + None + )] + #[case::published_glb_orphan_v2( + HistoryCommitType::Published, + 25, + Bound::GreatestLower, + Some(CommitAt::new(2, 20)) + )] + #[case::recreatable_glb_orphan_v2( + HistoryCommitType::Recreatable, + 25, + Bound::GreatestLower, + None + )] + #[case::published_lub_orphan( + HistoryCommitType::Published, + 15, + Bound::LeastUpper, + Some(CommitAt::new(2, 20)) + )] + #[case::recreatable_lub_anchor( + HistoryCommitType::Recreatable, + 15, + Bound::LeastUpper, + Some(CommitAt::new(3, 30)) + )] + #[tokio::test] + async fn test_timestamp_to_version_with_commit_type( + #[case] commit_type: HistoryCommitType, + #[case] timestamp: Timestamp, + #[case] bound: Bound, + #[case] expected: Option, + ) { + let (_table, engine, snapshot) = + build_mock_table_with_log_cleanup(&[5, 10, 20, 30, 40, 50], 3, 0).await; + + let res = timestamp_to_version(&snapshot, &engine, timestamp, bound, commit_type); + match expected { + Some(commit) => assert_eq!(res.unwrap(), commit), + None => assert!( + matches!(res, Err(LogHistoryError::TimestampOutOfRange { .. })), + "{res:?}" + ), + } + } + + #[rstest::rstest] + #[tokio::test] + async fn test_timestamp_to_version_recreatable_matches_published_when_v0_present( + #[values(0, 50, 100, 250, 300, 350, 1000)] timestamp: Timestamp, + #[values(Bound::GreatestLower, Bound::LeastUpper)] bound: Bound, + ) { + let timestamps = [ + (50, None), + (150, None), + (250, None), + (350, Some(300)), + (450, Some(400)), + ]; + let (_table, engine, snapshot, _log_segment) = + build_test_snapshot(×tamps, Some(3), None).await; + + let published = timestamp_to_version( + &snapshot, + &engine, + timestamp, + bound, + HistoryCommitType::Published, + ); + let recreatable = timestamp_to_version( + &snapshot, + &engine, + timestamp, + bound, + HistoryCommitType::Recreatable, + ); + match (published, recreatable) { + (Ok(p), Ok(r)) => assert_eq!(p, r, "diverged at ts={timestamp} bound={bound:?}"), + (Err(_), Err(_)) => {} + (p, r) => { + panic!("Published and Recreatable diverged at ts={timestamp} bound={bound:?}: {p:?} vs {r:?}") + } + } + } + + #[rstest::rstest] + #[case::published_sees_orphan(HistoryCommitType::Published, 25, Some(CommitAt::new(2, 20)))] + #[case::recreatable_excludes_orphan(HistoryCommitType::Recreatable, 25, None)] + #[tokio::test] + async fn test_latest_version_as_of_with_commit_type( + #[case] commit_type: HistoryCommitType, + #[case] timestamp: Timestamp, + #[case] expected: Option, + ) { + let (_table, engine, snapshot) = + build_mock_table_with_log_cleanup(&[5, 10, 20, 30, 40, 50], 3, 0).await; + + let res = latest_version_as_of(&snapshot, &engine, timestamp, commit_type); + match expected { + Some(commit) => assert_eq!(res.unwrap(), commit), + None => assert!(res.is_err(), "{res:?}"), + } + } + + #[rstest::rstest] + #[case::published_sees_orphan(HistoryCommitType::Published, 15, CommitAt::new(2, 20))] + #[case::recreatable_skips_to_anchor(HistoryCommitType::Recreatable, 15, CommitAt::new(3, 30))] + #[tokio::test] + async fn test_first_version_after_with_commit_type( + #[case] commit_type: HistoryCommitType, + #[case] timestamp: Timestamp, + #[case] expected: CommitAt, + ) { + let (_table, engine, snapshot) = + build_mock_table_with_log_cleanup(&[5, 10, 20, 30, 40, 50], 3, 0).await; + + let res = first_version_after(&snapshot, &engine, timestamp, commit_type).unwrap(); + assert_eq!(res, expected); + } + + /// timestamp_range_to_versions with file-mod-only table (v0=100, v1=200, v2=300) + #[rstest::rstest] + #[case::basic_range(100, Some(200), Ok((0, Some(1))))] + #[case::entire_table(50, Some(500), Ok((0, Some(2))))] + #[case::no_end(150, None, Ok((1, None)))] + #[case::exact_v0_to_v2(100, Some(300), Ok((0, Some(2))))] + #[case::between_commits(150, Some(250), Ok((1, Some(1))))] + #[case::start_equals_end_exact(200, Some(200), Ok((1, Some(1))))] + #[case::end_before_all_fails(50, Some(80), Err(()))] + #[case::start_after_all_fails(400, Some(500), Err(()))] + #[tokio::test] + async fn test_timestamp_range_file_mod_only( + #[case] start: Timestamp, + #[case] end: Option, + #[case] expected: Result<(Version, Option), ()>, + ) { + let table = + mock_table_with_timestamps(&[(100, None), (200, None), (300, None)], None).await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + let res = timestamp_range_to_versions(&snapshot, &engine, start, end); + match expected { + Ok(v) => assert_eq!(res.unwrap(), v), + Err(()) => assert!(res.is_err(), "{res:?}"), + } + } + + /// timestamp_range_to_versions with ICT from creation (v0=100, v1=200, v2=300) + #[rstest::rstest] + #[case::basic_range(100, Some(200), Ok((0, Some(1))))] + #[case::entire_table(50, Some(500), Ok((0, Some(2))))] + #[case::no_end(150, None, Ok((1, None)))] + #[case::exact_v0_to_v2(100, Some(300), Ok((0, Some(2))))] + #[case::between_commits(150, Some(250), Ok((1, Some(1))))] + #[case::start_equals_end_exact(200, Some(200), Ok((1, Some(1))))] + #[case::end_before_all_fails(50, Some(80), Err(()))] + #[case::start_after_all_fails(400, Some(500), Err(()))] + #[tokio::test] + async fn test_timestamp_range_ict_from_creation( + #[case] start: Timestamp, + #[case] end: Option, + #[case] expected: Result<(Version, Option), ()>, + ) { + let table = + mock_table_with_timestamps(&[(0, Some(100)), (0, Some(200)), (0, Some(300))], Some(0)) + .await; + let engine = SyncEngine::new(); + let path = Url::from_directory_path(table.table_root()).unwrap(); + let snapshot = Snapshot::builder_for(path).build(&engine).unwrap(); + + let res = timestamp_range_to_versions(&snapshot, &engine, start, end); + match expected { + Ok(v) => assert_eq!(res.unwrap(), v), + Err(()) => assert!(res.is_err(), "{res:?}"), + } + } + + /// Builds an in-memory store with a mock entry per given log file path and returns + /// an engine wired to it plus the corresponding `_delta_log/` URL. Paths listed in + /// `empty_paths` are written as 0-byte files to exercise empty-file skipping. + fn engine_with_log_files(paths: &[String], empty_paths: &HashSet) -> (SyncEngine, Url) { + let engine = SyncEngine::new_with_store(Arc::new(InMemory::new())); + let storage = engine.storage_handler(); + for path in paths { + let bytes = if empty_paths.contains(path) { + Bytes::new() + } else { + Bytes::from_static(b"x") + }; + let url = Url::parse(&format!("memory:///{path}")).unwrap(); + storage.put(&url, bytes, false).expect("put log file"); + } + (engine, Url::parse("memory:///_delta_log/").unwrap()) + } + + fn commit_path(version_range: RangeInclusive) -> Vec { + version_range + .map(|v| delta_path_for_version(v, "json").to_string()) + .collect() + } + + fn single_part_checkpoint_path(v: Version) -> String { + format!("_delta_log/{v:020}.checkpoint.parquet") + } + + fn uuid_checkpoint_path(v: Version) -> String { + format!("_delta_log/{v:020}.checkpoint.{}.parquet", Uuid::new_v4()) + } + + fn multi_part_checkpoint_paths(v: Version, num_parts: u32) -> Vec { + (1..=num_parts) + .map(|part| format!("_delta_log/{v:020}.checkpoint.{part:010}.{num_parts:010}.parquet")) + .collect() + } + + /// Builds a "truncated log" layout: `checkpoint_paths` plus commits over `commit_range`. + fn truncated_log( + checkpoint_paths: Vec, + commit_range: RangeInclusive, + ) -> Vec { + let mut paths: Vec = checkpoint_paths; + paths.extend(commit_path(commit_range)); + paths + } + + #[rstest::rstest] + #[case::v0_exists(commit_path(0..=5), None, Expected::Version(0))] + #[case::truncated_single_part(truncated_log(vec![single_part_checkpoint_path(5)], 5..=9), None, Expected::Version(5))] + #[case::truncated_uuid(truncated_log(vec![uuid_checkpoint_path(5)], 5..=9), None, Expected::Version(5))] + #[case::truncated_multi_part(truncated_log(multi_part_checkpoint_paths(5, 3), 5..=9), None, Expected::Version(5))] + #[case::ratified_zero_with_v0_present(commit_path(0..=3), Some(0), Expected::Version(0))] + #[case::non_zero_ratified_falls_through_to_scan(commit_path(0..=3), Some(4), Expected::Version(0))] + #[case::commit_cleanup_before_checkpoint( + { + let mut p = commit_path(2..=4); + p.push(uuid_checkpoint_path(4)); + p + }, + None, + Expected::Version(4) + )] + #[case::picks_first_anchored_checkpoint( + { + let mut p = vec![single_part_checkpoint_path(3)]; + p.extend(multi_part_checkpoint_paths(7, 2)); + p.extend(commit_path(7..=8)); + p + }, + None, + Expected::Version(7), + )] + #[case::listing_with_multiple_full_checkpoints( + { + let mut p = vec![single_part_checkpoint_path(3)]; + p.extend(commit_path(3..=7)); + p.push(single_part_checkpoint_path(7)); + p + }, + None, + Expected::Version(3), + )] + #[case::listing_with_cleanup_checkpoints( + { + let mut p = multi_part_checkpoint_paths(3, 3); + p.remove(0); + p.extend(commit_path(3..=7)); + p.push(single_part_checkpoint_path(7)); + p + }, + None, + Expected::Version(7), + )] + #[case::listing_with_v0_and_full_checkpoint( + { + let mut p = commit_path(0..=5); + p.push(single_part_checkpoint_path(5)); + p + }, + None, + Expected::Version(0), + )] + #[case::checkpoint_before_smallest_commit_anchors( + { + let mut p = vec![single_part_checkpoint_path(5)]; + p.extend(commit_path(6..=8)); + p + }, + None, + Expected::NoRecreatableCommit, + )] + #[case::empty_log(vec![], None, Expected::NoCommitsFound)] + #[case::crc_only(vec![format!("_delta_log/{:020}.crc", 5)], None, Expected::NoCommitsFound)] + #[case::compacted_only(vec![format!("_delta_log/{:020}.{:020}.compacted.json", 0, 5)], None, Expected::NoCommitsFound)] + #[case::checkpoint_only(vec![single_part_checkpoint_path(5)], None, Expected::NoCommitsFound)] + #[case::non_zero_ratified_on_empty_log(vec![], Some(5), Expected::NoCommitsFound)] + #[case::ratified_zero_on_empty_log(vec![], Some(0), Expected::CCv2MissingV0FilesystemCommit)] + #[case::commits_have_no_anchor(commit_path(2..=3), None, Expected::NoRecreatableCommit)] + #[case::ratified_zero_commits_no_anchor(commit_path(2..=3), Some(0), Expected::NoRecreatableCommit)] + #[case::staged_commits_and_last_checkpoint_ignored( + { + let mut p = truncated_log(vec![single_part_checkpoint_path(5)], 5..=9); + p.push("_delta_log/_last_checkpoint".to_string()); + p.push(format!("_delta_log/_staged_commits/{:020}.{}.json", 9, Uuid::new_v4())); + p + }, + None, + Expected::Version(5) + )] + fn earliest_recreatable_returns_expected( + #[case] paths: Vec, + #[case] ratified: Option, + #[case] expected: Expected, + ) { + let (engine, log_root) = engine_with_log_files(&paths, &HashSet::new()); + let res = get_earliest_recreatable_commit(&engine, &log_root, ratified); + match expected { + Expected::Version(v) => assert_eq!(res.unwrap(), v), + Expected::NoCommitsFound => assert!( + matches!(&res, Err(DeltaError::LogHistory(e)) if matches!(**e, LogHistoryError::NoCommitsFound { .. })), + "expected NoCommitsFound error, got {res:?}" + ), + Expected::NoRecreatableCommit => assert!( + matches!(&res, Err(DeltaError::LogHistory(e)) if matches!(**e, LogHistoryError::NoRecreatableCommit { .. })), + "expected NoRecreatableCommit error, got {res:?}" + ), + Expected::CCv2MissingV0FilesystemCommit => { + assert!(matches!(res, Err(DeltaError::Generic(_))), "{res:?}") + } + } + } + + enum Expected { + Version(Version), + NoCommitsFound, + NoRecreatableCommit, + CCv2MissingV0FilesystemCommit, + } + + #[test] + fn test_empty_checkpoint_is_skipped() { + let mut paths = vec![single_part_checkpoint_path(5)]; + paths.extend(commit_path(5..=9)); + paths.push(single_part_checkpoint_path(8)); + + let mut empty_paths = HashSet::new(); + empty_paths.insert(single_part_checkpoint_path(5)); + + let (engine, log_root) = engine_with_log_files(&paths, &empty_paths); + let version = get_earliest_recreatable_commit(&engine, &log_root, None).unwrap(); + assert_eq!(version, 8); + } + + #[tokio::test] + #[rstest::rstest] + #[case::no_ratified_commit(3, None, None, Expected::Version(0))] + #[case::ratified_commit_version_0(3, Some(0), None, Expected::Version(0))] + #[case::ratified_commit_version_greater_than_0(3, Some(2), None, Expected::Version(0))] + #[case::log_listing_empty_no_ratified_commit(0, None, None, Expected::NoCommitsFound)] + #[case::log_listing_empty_ratified_commit(0, Some(2), None, Expected::NoCommitsFound)] + #[case::log_listing_empty_ratified_commit_v0( + 0, + Some(0), + None, + Expected::CCv2MissingV0FilesystemCommit + )] + async fn test_get_earliest_published_commit_version( + #[case] num_commits: usize, + #[case] earliest_ratified_commit_version: Option, + #[case] last_cleanup_version: Option, + #[case] expected: Expected, + ) { + let timestamps = (0..num_commits) + .map(|i| (i as i64, None)) + .collect::>(); + let table = mock_table_with_timestamps(×tamps, None).await; + let engine = SyncEngine::new(); + let log_root = Url::from_directory_path(table.table_root().join("_delta_log")).unwrap(); + + if let Some(last_cleanup_version) = last_cleanup_version { + for version in 0..=last_cleanup_version { + let filename = format!("{version:020}.json"); + remove_file(log_root.to_file_path().unwrap().join(filename)).unwrap(); + } + } + + let res = get_earliest_published_commit_version( + &engine, + &log_root, + earliest_ratified_commit_version, + ); + match expected { + Expected::Version(v) => assert_eq!(res.unwrap(), v), + Expected::NoCommitsFound => assert!( + matches!(res, Err(DeltaError::LogHistory(ref e)) if matches!(**e, LogHistoryError::NoCommitsFound { .. })), + "{res:?}" + ), + Expected::CCv2MissingV0FilesystemCommit => { + assert!(matches!(res, Err(DeltaError::Generic(_))), "{res:?}") + } + Expected::NoRecreatableCommit => { + unreachable!( + "get_earliest_published_commit_version never returns NoRecreatableCommit" + ) + } + } + } + + #[tokio::test] + async fn test_get_earliest_published_commit_version_ignores_staged_commits() { + let timestamps = vec![(0, None)]; + let table = mock_table_with_timestamps(×tamps, None).await; + let engine = SyncEngine::new(); + let log_dir = table.table_root().join("_delta_log"); + let log_root = Url::from_directory_path(&log_dir).unwrap(); + + // Write a staged commit at v1 with a fixed UUID. Staged commits should NOT influence + // the earliest-published-commit result. + let staged_dir = log_dir.join("_staged_commits"); + std::fs::create_dir_all(&staged_dir).unwrap(); + let staged_file = + staged_dir.join("00000000000000000001.01234567-89ab-cdef-0123-456789abcdef.json"); + std::fs::write(&staged_file, "{}").unwrap(); + + let res = get_earliest_published_commit_version(&engine, &log_root, None); + assert_eq!(res.unwrap(), 0); + } +} diff --git a/kernel/src/history_manager/search.rs b/kernel/src/history_manager/search.rs index 399555b364..d07c1fbfa9 100644 --- a/kernel/src/history_manager/search.rs +++ b/kernel/src/history_manager/search.rs @@ -7,21 +7,42 @@ use std::fmt::Debug; /// Defines the type of bound to return from a binary search operation. /// /// For a search operation over `values` using key `key`: -/// * [`Bound::LeastUpper`] - Finds the smallest index `i` such that `values[i] >= key`. -/// This represents the first element greater than or equal to the search key. +/// * [`Bound::LeastUpper`] - Finds the smallest index `i` such that `values[i] >= key`. This +/// represents the first element greater than or equal to the search key. /// -/// * [`Bound::GreatestLower`] - Finds the largest index `i` such that `values[i] <= key`. -/// This represents the last element less than or equal to the search key. -#[allow(unused)] -#[derive(Debug, Clone, Copy)] +/// * [`Bound::GreatestLower`] - Finds the largest index `i` such that `values[i] <= key`. This +/// represents the last element less than or equal to the search key. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(crate) enum Bound { + /// Find the smallest index with value >= key (first element at or after). LeastUpper, + /// Find the largest index with value <= key (last element at or before). GreatestLower, } +impl Bound { + /// Returns the error message for when a timestamp search is out of range. + pub(crate) fn out_of_range_reason(&self) -> &'static str { + match self { + Bound::LeastUpper => "no version exists at or after this timestamp", + Bound::GreatestLower => "no version exists at or before this timestamp", + } + } + + /// Returns the boundary element of `slice` on the side an out-of-range + /// search failed against: the first element for `GreatestLower` (search + /// key was below it) and the last for `LeastUpper` (search key was above + /// it). Returns `None` when `slice` is empty. + pub(crate) fn boundary_of<'a, T>(&self, slice: &'a [T]) -> Option<&'a T> { + match self { + Bound::GreatestLower => slice.first(), + Bound::LeastUpper => slice.last(), + } + } +} + /// Represents the errors that can occur when performing binary search using /// [`binary_search_by_key_with_bounds`]. -#[allow(unused)] #[derive(Debug)] pub(crate) enum SearchError { /// Error that occurs when a search goes out of range. The meaning of "out of range" depends on @@ -32,8 +53,8 @@ pub(crate) enum SearchError { /// to the provided key. OutOfRange, /// Error that occurs when the `key_fn` fails when retrieving the key in - /// [`binary_search_by_key_with_bounds`]. The error that occurred in the `key_fn` is returned in - /// this error variant. + /// [`binary_search_by_key_with_bounds`]. The error that occurred in the `key_fn` is returned + /// in this error variant. KeyFunctionError(T), } @@ -54,8 +75,8 @@ pub(crate) enum SearchError { /// /// * Ok(usize) - The index of the found element /// * [`SearchError::OutOfRange`] - If no suitable bound exists in the slice -/// * [`SearchError::KeyFunctionError`] - If the key function fails. This contains the error -/// emitted by the `key_fn`. +/// * [`SearchError::KeyFunctionError`] - If the key function fails. This contains the error emitted +/// by the `key_fn`. /// /// #Safety /// Assumes that the provided input is sorted by key using the `key_fn`. If the input is not @@ -129,7 +150,6 @@ pub(crate) enum SearchError { /// ); /// assert!(matches!(result, Err(SearchError::KeyFunctionError(_)))); /// ``` -#[allow(unused)] pub(crate) fn binary_search_by_key_with_bounds<'a, T, K: Ord + Debug, E: Error>( values: &'a [T], key: K, diff --git a/kernel/src/incremental_scan/mod.rs b/kernel/src/incremental_scan/mod.rs new file mode 100644 index 0000000000..13ea118fbb --- /dev/null +++ b/kernel/src/incremental_scan/mod.rs @@ -0,0 +1,697 @@ +//! Streams the file-action diff between two table versions. See +//! [`IncrementalScanBuilder`] for the entry point. + +use std::collections::HashSet; +use std::sync::{Arc, LazyLock}; + +use crate::engine_data::{FilteredEngineData, GetData, RowVisitor}; +use crate::expressions::{column_name, ColumnName}; +use crate::log_replay::deduplicator::{Deduplicator, FileActionInfo}; +use crate::log_replay::{FileActionDeduplicator, FileActionKey}; +use crate::scan::data_skipping::DataSkippingFilter; +use crate::scan::{PhysicalPredicate, COMMIT_READ_SCHEMA}; +use crate::schema::{ColumnNamesAndTypes, DataType}; +use crate::snapshot::SnapshotRef; +use crate::table_features::Operation; +use crate::utils::require; +use crate::{ + DeltaResult, Engine, EngineData, Error, FileDataReadResultIterator, FileMeta, PredicateRef, + Version, +}; + +/// Builder for an incremental scan over `(base_version, target_version]`. Construct via +/// [`crate::Snapshot::incremental_scan_builder`] and drive with +/// [`IncrementalScanBuilder::build`]. +#[derive(Debug)] +pub struct IncrementalScanBuilder { + target_snapshot: SnapshotRef, + base_version: Version, + predicate: Option, +} + +impl IncrementalScanBuilder { + /// Create a new builder for the range `(base_version, target_snapshot.version()]`. + pub(crate) fn new(target_snapshot: impl Into, base_version: Version) -> Self { + Self { + target_snapshot: target_snapshot.into(), + base_version, + predicate: None, + } + } + + /// Restrict the streamed live `Add` actions to those whose file stats do not provably + /// exclude `predicate`. `None` (the default) preserves the current behavior: every live + /// Add is streamed. + /// + /// Removes are unaffected: they are never pruned and must all be reported so consumers can + /// drop stale cached entries. Skipping is best-effort and conservative: a file is dropped + /// only when its stats prove no row can match, and files without stats are always kept. + /// + /// Skipping reuses the same stats-based filter the read path uses: data-column predicates + /// prune against file stats parsed from `add.stats`, and partition-column predicates prune + /// against the file's partition values. The surviving Add set matches what a full scan at the + /// target version with the same predicate would keep among these added files. Skipping never + /// drops a file a cold scan would keep, so consumers must re-apply the predicate at read time + /// regardless. + pub fn with_predicate(mut self, predicate: impl Into>) -> Self { + self.predicate = predicate.into(); + self + } + + /// Build the incremental scan stream, or `None` if the target snapshot's commit list + /// cannot serve `(base_version, target_version]` (consumers should fall back to a full + /// scan via [`crate::Snapshot::scan_builder`]). + /// + /// Does not re-list `_delta_log/`: walks the target snapshot's already-validated commit + /// list and clips it to `(base_version, target_version]`. For catalog-managed tables + /// that's important — the snapshot's `log_tail` carries staged commits that a fresh + /// storage listing would miss. + /// + /// Validates only the target's reader features via [`Operation::Scan`]. By the protocol's + /// feature-immutability rule, the target's `readerFeatures` is a superset of every + /// reader feature used in any commit in `(base_version, target_version]`. Of the fields + /// kernel itself decodes from each row (path + deletionVector.*), only + /// `deletionVector.*` is feature-gated, and pre-`deletionVectors`-enable commits cannot + /// populate it. Consumers that decode pass-through fields (stats, partitionValues, + /// baseRowId) should interpret each row against the protocol at that row's commit + /// version, not naively against the target snapshot. + /// + /// # Errors + /// - `Err` if `base_version >= target_snapshot.version()` (caller error). + /// - [`Error::MissingColumn`] if a [`with_predicate`](Self::with_predicate) predicate + /// references a column absent from the table schema. + /// - `Err` if the target snapshot's protocol contains an unsupported reader feature. + /// - `Err` if the engine fails to open the commit stream. + pub fn build(self, engine: &dyn Engine) -> DeltaResult> { + // TODO(#2493): the version validation, the snapshot-commit-list extraction and + // clipping below, and the "snapshot covers the range" check should all be replaced + // by a single `CommitRangeBuilder::from_snapshot(...).build(engine)` call once + // that primitive lands. https://github.com/delta-io/delta-kernel-rs/issues/2493 + let target_version = self.target_snapshot.version(); + require!( + self.base_version < target_version, + Error::generic(format!( + "IncrementalScanBuilder: base_version ({}) must be less than target_version ({})", + self.base_version, target_version + )) + ); + // Resolve the predicate into an Add-side skipping strategy up front so `build` can + // fail fast on a malformed predicate (e.g. references to unknown columns), rather + // than surfacing the error lazily from the stream. + let add_skipping = self.resolve_add_skipping(engine)?; + // `base_version < target_version` above guarantees `base_version < u64::MAX`, but use + // `checked_add` to make the dependency explicit and panic-free. + let start_version = self.base_version.checked_add(1).ok_or_else(|| { + Error::generic("IncrementalScanBuilder: base_version + 1 overflowed u64") + })?; + + // TODO(#2552): surface in-range protocol/metadata changes to the consumer. + self.target_snapshot + .table_configuration() + .ensure_operation_supported(Operation::Scan)?; + + // TODO(#2493): replace this block with `CommitRange` once it exists. + // Today we use the snapshot's already-validated commit list rather than re-listing + // storage. This is correct for catalog-managed tables: the snapshot's log_segment + // includes any staged/ratified-but-unpublished commits passed in via `with_log_tail`, + // which a fresh storage listing would silently miss. `CommitRange` will own this + // listing-and-validation responsibility centrally. + let snapshot_commits = &self + .target_snapshot + .log_segment() + .listed + .ascending_commit_files; + let snapshot_first_version = snapshot_commits.first().map(|c| c.version); + + // TODO(#2493): this "snapshot covers range" check becomes a typed error from + // `CommitRange::build` (e.g. `Error::CommitsUnavailable { missing: Range }`), + // pattern-matched here to return `None`. + if snapshot_first_version.is_none_or(|v| v > start_version) { + return Ok(None); + } + + // Collect in-range commit locations in newest-first order. Handing all of them to + // `read_json_files` in a single call lets the engine's prefetch (e.g. + // `DefaultEngine`'s `buffered(buffer_size)`) overlap object-store GETs across + // commits, instead of paying first-byte latency once per commit serially. + // Newest-first order gives `FileActionDeduplicator` newest-wins semantics across + // the range. + let commit_locations: Vec = snapshot_commits + .iter() + .filter(|c| c.version >= start_version && c.version <= target_version) + .rev() + .map(|c| c.location.clone()) + .collect(); + + let actions = engine.json_handler().read_json_files( + &commit_locations, + COMMIT_READ_SCHEMA.clone(), + None, + )?; + + Ok(Some(IncrementalScanStream { + base_version: self.base_version, + target_version, + actions, + add_skipping, + seen_file_keys: HashSet::new(), + live_adds: HashSet::new(), + removes: HashSet::new(), + errored: false, + })) + } + + /// Lowers `self.predicate` into the per-batch skipping strategy applied to live Adds. + /// + /// Returns [`AddSkipping::KeepAll`] when there is no predicate, when the predicate is + /// useless for data skipping ([`PhysicalPredicate::None`]), or when the filter cannot be + /// constructed. [`AddSkipping::SkipAll`] when the predicate statically excludes every file + /// ([`PhysicalPredicate::StaticSkipAll`]); the stream still reports Removes but yields no + /// live Adds. [`AddSkipping::Filter`] carries the reusable [`DataSkippingFilter`]. + fn resolve_add_skipping(&self, engine: &dyn Engine) -> DeltaResult { + let Some(predicate) = self.predicate.as_ref() else { + return Ok(AddSkipping::KeepAll); + }; + + let table_configuration = self.target_snapshot.table_configuration(); + let logical_schema = table_configuration.logical_schema(); + let column_mapping_mode = table_configuration.column_mapping_mode(); + match PhysicalPredicate::try_new(predicate, &logical_schema, column_mapping_mode)? { + PhysicalPredicate::StaticSkipAll => Ok(AddSkipping::SkipAll), + PhysicalPredicate::None => Ok(AddSkipping::KeepAll), + PhysicalPredicate::Some(physical_predicate, _referenced_schema) => { + // Reuse the same raw-action-batch filter the change-data-feed path builds: + // parse stats from `add.stats` JSON and partition values from + // `add.partitionValues`, then skip against the shared `DataSkippingFilter`. + // `None` (predicate ineligible for skipping) degrades to keep-all. + let filter = DataSkippingFilter::for_raw_action_batch( + engine, + physical_predicate, + table_configuration, + COMMIT_READ_SCHEMA.clone(), + ); + Ok(filter.map_or(AddSkipping::KeepAll, |f| AddSkipping::Filter(Arc::new(f)))) + } + } + } +} + +/// How the streamed live `Add` actions are pruned against the builder's predicate. +enum AddSkipping { + /// No predicate, or a predicate useless for data skipping: every live Add is streamed. + KeepAll, + /// The predicate statically excludes every file: no live Add is streamed (Removes are + /// still reported). + SkipAll, + /// Prune each Add batch through the stats-based filter, keeping only Adds the predicate + /// cannot provably exclude. + Filter(Arc), +} + +/// Streaming output of an incremental scan over `(base_version, target_version]`. +/// Yields live Add batches as [`FilteredEngineData`] in newest-first order via +/// [`Iterator::next`]; call [`into_summary`] or [`into_listing`] to terminate and recover +/// the live file-key sets. +/// +/// An Add is "live" at the target if, walking commits newest-first and keying by +/// `(path, dv_unique_id)`, it is the first occurrence of that key (first-seen-wins dedup) +/// and no later commit in the range contains a Remove for the same key — i.e. the file is +/// still in the table at `target_version`. +/// +/// On error, `next()` yields `Some(Err(_))` once and returns `None` on every subsequent +/// call; the stream's dedup state is incomplete, so terminal methods then return `Err` +/// rather than producing a partial summary. +/// +/// [`into_summary`]: Self::into_summary +/// [`into_listing`]: Self::into_listing +pub struct IncrementalScanStream { + base_version: Version, + target_version: Version, + actions: FileDataReadResultIterator, + add_skipping: AddSkipping, + seen_file_keys: HashSet, + live_adds: HashSet, + removes: HashSet, + errored: bool, +} + +impl Iterator for IncrementalScanStream { + type Item = DeltaResult; + + fn next(&mut self) -> Option { + if self.errored { + return None; + } + loop { + match self.actions.next()? { + Err(e) => { + self.errored = true; + return Some(Err(e)); + } + Ok(batch) => match process_batch( + batch, + &self.add_skipping, + &mut self.seen_file_keys, + &mut self.live_adds, + &mut self.removes, + ) { + Ok(Some(filtered)) => return Some(Ok(filtered)), + Ok(None) => continue, + Err(e) => { + self.errored = true; + return Some(Err(e)); + } + }, + } + } + } +} + +impl IncrementalScanStream { + /// Drain any unread batches, then return the live file-key sets without applying + /// cross-snapshot dedup. Consumers can intersect these against whatever base + /// data structure they prefer. + /// + /// # Errors + /// - [`Error::IOError`], [`Error::ObjectStore`], or [`Error::Reqwest`] on transient I/O while + /// reading commit JSONs. Retryable by rebuilding the stream. + /// - [`Error::FileNotFound`] if a commit was vacuumed between [`IncrementalScanBuilder::build`] + /// and stream consumption. Rebuilding will likely return `Ok(None)` (commits unavailable); + /// fall back to [`crate::Snapshot::scan_builder`]. + /// - [`Error::MalformedJson`] or [`Error::Arrow`] (default-engine) on commit JSON corruption. + /// Not retryable. + /// - [`Error::Generic`] on malformed `deletionVector` fields in a commit row, or on "cannot + /// finish a stream that previously errored" when a terminal method is called after a prior + /// `next()` returned `Err`. Rebuild to retry the latter; the former indicates table + /// corruption. + pub fn into_summary(mut self) -> DeltaResult { + if self.errored { + return Err(Error::generic( + "IncrementalScanStream: cannot finish a stream that previously errored", + )); + } + // Drain anything the consumer didn't pull. `self.next()` propagates errors via + // `Some(Err(_))` and sets `errored`; the `?` below surfaces them. + for item in self.by_ref() { + item?; + } + Ok(IncrementalScanSummary { + base_version: self.base_version, + target_version: self.target_version, + live_adds: self.live_adds, + removes: self.removes, + }) + } + + /// Eager helper: collect every live Add batch into a [`Vec`], then call + /// [`into_summary`] to recover the file-key sets. Use this when the diff fits in memory + /// and the consumer prefers the simpler eager shape over the iterator pattern. + /// + /// The returned `Vec` may contain multiple batches per source commit: the engine is + /// free to split a commit's rows across [`ActionsBatch`] yields (e.g. by JSON-reader + /// batch-size limits). One yielded batch produces at most one `Vec` entry, and a + /// commit whose Adds were all cancelled by later Removes produces no entry at all. + /// + /// # Errors + /// See [`into_summary`]. + /// + /// [`into_summary`]: Self::into_summary + /// [`ActionsBatch`]: crate::log_replay::ActionsBatch + pub fn into_listing(mut self) -> DeltaResult { + let mut add_files: Vec = Vec::new(); + for item in self.by_ref() { + add_files.push(item?); + } + let summary = self.into_summary()?; + Ok(IncrementalListing { summary, add_files }) + } + + /// Drain any unread batches, then intersect `base_keys` against the live-Add file-key + /// set to compute `duplicate_adds` (file keys in the consumer's base listing that the + /// range re-adds with new metadata, e.g. OPTIMIZE / liquid clustering re-tag). + /// + /// Streaming form: accepts borrowed keys from any source (`&HashSet`, `&Vec`, `&[…]`, + /// `map.keys()`, custom iterator). Kernel streams `base_keys` once and probes the + /// live-Add set per element. Work is `O(|base|)` with no call-site clones. Use this + /// when your cache holds keys in a streamable shape and you don't have a faster + /// `contains` lookup. + /// + /// Prefer [`into_summary_against_base_closure`] when your cache supports + /// `contains`-style lookup (HashMap, HashSet, BTreeMap, custom index). That form + /// iterates the (typically much smaller) live-Add set and probes your base via the + /// supplied closure, dropping work from `O(|base|)` to `O(|live_adds|)`. + /// + /// # Errors + /// See [`into_summary`]. + /// + /// [`into_summary`]: Self::into_summary + /// [`into_summary_against_base_closure`]: Self::into_summary_against_base_closure + pub fn into_summary_against_base_iter<'a>( + self, + base_keys: impl IntoIterator, + ) -> DeltaResult { + let summary = self.into_summary()?; + let duplicate_adds: HashSet = base_keys + .into_iter() + .filter(|k| summary.live_adds.contains(*k)) + .cloned() + .collect(); + Ok(IncrementalScanSummaryAgainstBase { + base_version: summary.base_version, + target_version: summary.target_version, + duplicate_adds, + removes: summary.removes, + }) + } + + /// Predicate form of [`into_summary_against_base_iter`]: pass a closure that answers + /// "is this key in your base?" Iterates the live-Add set (typically much smaller than + /// the base) and calls `base_contains` once per live-Add key. + /// + /// Works with any base shape that supports membership lookup: + /// - `HashMap` -> `|k| map.contains_key(k)` + /// - `HashSet` -> `|k| set.contains(k)` + /// - `BTreeMap` -> `|k| map.contains_key(k)` + /// - Sorted `Vec` -> `|k| vec.binary_search(k).is_ok()` + /// + /// # Errors + /// See [`into_summary`]. + /// + /// [`into_summary`]: Self::into_summary + /// [`into_summary_against_base_iter`]: Self::into_summary_against_base_iter + pub fn into_summary_against_base_closure( + self, + base_contains: impl Fn(&FileActionKey) -> bool, + ) -> DeltaResult { + let summary = self.into_summary()?; + let duplicate_adds: HashSet = summary + .live_adds + .iter() + .filter(|k| base_contains(k)) + .cloned() + .collect(); + Ok(IncrementalScanSummaryAgainstBase { + base_version: summary.base_version, + target_version: summary.target_version, + duplicate_adds, + removes: summary.removes, + }) + } + + /// Eager classified helper: collect every live Add batch and call + /// [`into_summary_against_base_iter`] against `base_keys`. Returns an + /// [`IncrementalListingAgainstBase`] with the classified summary. + /// + /// Prefer [`into_listing_against_base_closure`] when your base supports `contains`-style + /// lookup — see [`into_summary_against_base_closure`] for the rationale. + /// + /// # Errors + /// See [`into_summary`]. + /// + /// [`into_summary`]: Self::into_summary + /// [`into_summary_against_base_iter`]: Self::into_summary_against_base_iter + /// [`into_summary_against_base_closure`]: Self::into_summary_against_base_closure + /// [`into_listing_against_base_closure`]: Self::into_listing_against_base_closure + pub fn into_listing_against_base_iter<'a>( + mut self, + base_keys: impl IntoIterator, + ) -> DeltaResult { + let mut add_files: Vec = Vec::new(); + for item in self.by_ref() { + add_files.push(item?); + } + let summary = self.into_summary_against_base_iter(base_keys)?; + Ok(IncrementalListingAgainstBase { summary, add_files }) + } + + /// Predicate form of [`into_listing_against_base_iter`]; see + /// [`into_summary_against_base_closure`] for the rationale. + /// + /// # Errors + /// See [`into_summary`]. + /// + /// [`into_summary`]: Self::into_summary + /// [`into_listing_against_base_iter`]: Self::into_listing_against_base_iter + /// [`into_summary_against_base_closure`]: Self::into_summary_against_base_closure + pub fn into_listing_against_base_closure( + mut self, + base_contains: impl Fn(&FileActionKey) -> bool, + ) -> DeltaResult { + let mut add_files: Vec = Vec::new(); + for item in self.by_ref() { + add_files.push(item?); + } + let summary = self.into_summary_against_base_closure(base_contains)?; + Ok(IncrementalListingAgainstBase { summary, add_files }) + } +} + +impl std::fmt::Debug for IncrementalScanStream { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("IncrementalScanStream") + .field("base_version", &self.base_version) + .field("target_version", &self.target_version) + .field("live_adds", &self.live_adds.len()) + .field("removes", &self.removes.len()) + .field("errored", &self.errored) + .finish() + } +} + +/// Live file-key sets without cross-snapshot classification, returned by +/// [`IncrementalScanStream::into_summary`]. +/// +/// Each set element is a [`FileActionKey`] of `(path, dv_unique_id)`. Consumers should match +/// on the full key rather than just the path: the same path with different DV ids (e.g. an +/// `add(P, dv=new) + remove(P, dv=old)` DV-replacement pair) refers to distinct logical +/// files, and collapsing to path alone loses that distinction. +#[derive(Debug)] +#[non_exhaustive] +pub struct IncrementalScanSummary { + /// Exclusive lower bound of the scan range, as supplied to [`IncrementalScanBuilder`]. + pub base_version: Version, + /// Inclusive upper bound; equals the source snapshot's version. + pub target_version: Version, + /// File keys of live Add actions in `(base_version, target_version]` (i.e. Adds whose + /// `(path, dv_unique_id)` is still present at `target_version`). + pub live_adds: HashSet, + /// File keys of Remove actions in `(base_version, target_version]`, deduped + /// first-seen-wins by `(path, dv_unique_id)`. + pub removes: HashSet, +} + +/// Eager output of [`IncrementalScanStream::into_listing`]: the buffered Add +/// batches plus the summary (no cross-snapshot classification). +#[non_exhaustive] +pub struct IncrementalListing { + /// Live Add and Remove file-key sets for the range. + pub summary: IncrementalScanSummary, + /// All live Add batches in descending commit-version order. One entry per source + /// commit batch that produced any live Adds. + pub add_files: Vec, +} + +impl std::fmt::Debug for IncrementalListing { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("IncrementalListing") + .field("summary", &self.summary) + .field("add_files_batch_count", &self.add_files.len()) + .finish() + } +} + +/// Cross-snapshot-classified file-key sets, returned by +/// [`IncrementalScanStream::into_summary_against_base_iter`] (or its closure variant). +/// +/// To advance a delta-on-base file listing cache, append the streamed Add batches to +/// the delta layer and use the union `removes U duplicate_adds` as the remove-mask +/// against the base. Both sets are required: `removes` masks files that left the table; +/// `duplicate_adds` masks the stale base entry of each file the range re-added with new +/// metadata. Matching against the full `(path, dv_unique_id)` key (not path alone) keeps +/// distinct DV-revision entries separate. +#[derive(Debug)] +#[non_exhaustive] +pub struct IncrementalScanSummaryAgainstBase { + /// Exclusive lower bound of the scan range. + pub base_version: Version, + /// Inclusive upper bound; equals the source snapshot's version. + pub target_version: Version, + /// File keys from the live Add stream that also appear in the consumer's base + /// listing (metadata-only re-adds, e.g. OPTIMIZE / liquid clustering re-tag). + /// The corresponding rows are still in the streamed Adds. + pub duplicate_adds: HashSet, + /// File keys of Remove actions in the range. Consumers must union this with + /// `duplicate_adds` when masking the base. + pub removes: HashSet, +} + +/// Eager output of [`IncrementalScanStream::into_listing_against_base_iter`] (or its +/// closure variant): the buffered +/// Add batches plus the classified summary. +#[non_exhaustive] +pub struct IncrementalListingAgainstBase { + /// Classified file-key sets for the range; see [`IncrementalScanSummaryAgainstBase`]. + pub summary: IncrementalScanSummaryAgainstBase, + /// All live Add batches in descending commit-version order. + pub add_files: Vec, +} + +impl std::fmt::Debug for IncrementalListingAgainstBase { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("IncrementalListingAgainstBase") + .field("summary", &self.summary) + .field("add_files_batch_count", &self.add_files.len()) + .finish() + } +} + +/// Visit one commit-batch, updating dedup state and accumulators. Returns the filtered Add +/// rows for this batch, or `None` if no Adds survived dedup (and skipping) in this batch. +/// +/// When `add_skipping` carries a [`DataSkippingFilter`], a per-row stats mask is computed +/// once for the batch and AND-ed into the live-Add selection: an Add whose stats prove it +/// cannot match the predicate is dropped from both the emitted selection vector and the +/// `live_adds` set, keeping the streamed batches and the summary consistent. A skipped Add +/// still advances dedup state (its `(path, dv_unique_id)` key is recorded as seen) so an +/// older duplicate of the same file cannot leak through, matching newest-wins semantics. +/// Removes are never skipped. +fn process_batch( + batch: Box, + add_skipping: &AddSkipping, + seen_file_keys: &mut HashSet, + live_adds: &mut HashSet, + removes: &mut HashSet, +) -> DeltaResult> { + let row_count = batch.len(); + let mut adds_sel = vec![false; row_count]; + + // Per-row "keep this Add" mask derived from the predicate. `true` (the default) keeps the + // Add; `false` drops it. `SkipAll` drops every Add; `KeepAll` keeps every Add; `Filter` + // consults the stats-based skipper (which conservatively keeps rows it cannot disprove, + // including Removes and stats-less files). + let keep_add: Vec = match add_skipping { + AddSkipping::KeepAll => vec![true; row_count], + AddSkipping::SkipAll => vec![false; row_count], + AddSkipping::Filter(filter) => filter.apply(batch.as_ref())?, + }; + + let deduplicator = FileActionDeduplicator::new( + seen_file_keys, + true, // commit batches always update the seen set + ADD_PATH_INDEX, + ADD_SIZE_INDEX, + REMOVE_PATH_INDEX, + ADD_DV_START_INDEX, + REMOVE_DV_START_INDEX, + ); + let mut visitor = IncrementalDedupVisitor { + deduplicator, + keep_add: &keep_add, + adds_sel: &mut adds_sel, + live_adds, + removes, + }; + visitor.visit_rows_of(batch.as_ref())?; + + // No Adds in the batch survived dedup and skipping, e.g. a Removes-only commit, a commit + // whose Adds were all cancelled by later commits in the range, or one whose Adds were all + // pruned by the predicate. Skip yielding an empty batch; dedup state has still been + // advanced by the visit above. + if !adds_sel.iter().any(|s| *s) { + return Ok(None); + } + Ok(Some(FilteredEngineData::try_new(batch, adds_sel)?)) +} + +// Indices of the leaf columns visited per row; must match `selected_column_names_and_types`. +const ADD_PATH_INDEX: usize = 0; +const ADD_SIZE_INDEX: usize = 1; +const ADD_DV_START_INDEX: usize = 2; // .storageType, .pathOrInlineDv, .offset +const REMOVE_PATH_INDEX: usize = 5; +const REMOVE_DV_START_INDEX: usize = 6; // .storageType, .pathOrInlineDv, .offset +const NUM_GETTERS: usize = 9; // 5 add + 4 remove columns + +// We share `FileActionDeduplicator` with the scan path but not `AddRemoveDedupVisitor`: +// that visitor is wired to scan-only state (`StateInfo`, `ScanMetrics`, per-row transform +// expressions, partition value parsing, base row id) and only outputs Adds. Removes are +// folded into `seen_file_keys` as a side effect and never surfaced to the caller. The +// incremental scan needs both live Adds and Removes exposed (keyed by +// `(path, dv_unique_id)`), and it deliberately skips the scan-only per-row work, so a +// slimmer visitor is the right shape here. +struct IncrementalDedupVisitor<'a, 'seen> { + deduplicator: FileActionDeduplicator<'seen>, + /// Per-row predicate mask (`true` = keep the Add). Same length as the batch; indexed by + /// the visitor's row index. Removes ignore this mask. + keep_add: &'a [bool], + adds_sel: &'a mut [bool], + live_adds: &'a mut HashSet, + removes: &'a mut HashSet, +} + +impl RowVisitor for IncrementalDedupVisitor<'_, '_> { + fn selected_column_names_and_types(&self) -> (&'static [ColumnName], &'static [DataType]) { + // The DV columns are needed because [`FileActionDeduplicator`] keys on + // `(path, dv_unique_id)`, not path alone. Same path with different DVs (e.g. an + // `add(P, dv=new) + remove(P, dv=old)` DV-replacement pair) are distinct file + // identities and must be tracked separately so newer-wins dedup is correct. + static NAMES_AND_TYPES: LazyLock = LazyLock::new(|| { + const STRING: DataType = DataType::STRING; + const INTEGER: DataType = DataType::INTEGER; + const LONG: DataType = DataType::LONG; + let columns = vec![ + (STRING, column_name!("add.path")), + (LONG, column_name!("add.size")), + (STRING, column_name!("add.deletionVector.storageType")), + (STRING, column_name!("add.deletionVector.pathOrInlineDv")), + (INTEGER, column_name!("add.deletionVector.offset")), + (STRING, column_name!("remove.path")), + (STRING, column_name!("remove.deletionVector.storageType")), + (STRING, column_name!("remove.deletionVector.pathOrInlineDv")), + (INTEGER, column_name!("remove.deletionVector.offset")), + ]; + let (types, names) = columns.into_iter().unzip(); + (names, types).into() + }); + let (names, types) = NAMES_AND_TYPES.as_ref(); + (names, types) + } + + fn visit<'a>(&mut self, row_count: usize, getters: &[&'a dyn GetData<'a>]) -> DeltaResult<()> { + require!( + getters.len() == NUM_GETTERS, + Error::InternalError(format!( + "IncrementalDedupVisitor expected {NUM_GETTERS} getters, got {}", + getters.len() + )) + ); + + for i in 0..row_count { + let Some(FileActionInfo { key, is_add, .. }) = + self.deduplicator.extract_file_action(i, getters, false)? + else { + continue; + }; + + // Both Adds AND Removes update the seen set. A duplicate Add for a file + // removed earlier in the range (later chronologically, since we read newest + // first) must not leak through. + if self.deduplicator.check_and_record_seen(key.clone()) { + continue; + } + + if is_add { + // Record `seen` above regardless of the predicate so an older duplicate of a + // pruned file stays suppressed (newest-wins). Only inclusion in the output + // and `live_adds` is gated by the stats mask. + if self.keep_add[i] { + self.adds_sel[i] = true; + self.live_adds.insert(key); + } + } else { + self.removes.insert(key); + } + } + + Ok(()) + } +} diff --git a/kernel/src/kernel_predicates/mod.rs b/kernel/src/kernel_predicates/mod.rs index 620b662ba5..fa938c562e 100644 --- a/kernel/src/kernel_predicates/mod.rs +++ b/kernel/src/kernel_predicates/mod.rs @@ -3,17 +3,18 @@ //! and parquet row group filtering. The evaluation is normally performed over [`Scalar`] values, //! but data skipping "evaluation" actually produces a transformed predicate that replaces column //! references with stats column references, which log replay will instruct the engine to evaluate. +use std::cmp::Ordering; + +use tracing::{debug, warn}; + use crate::expressions::{ - BinaryExpression, BinaryExpressionOp, BinaryPredicate, BinaryPredicateOp, ColumnName, - Expression as Expr, JunctionPredicate, JunctionPredicateOp, OpaqueExpression, + BinaryExpression, BinaryExpressionOp, BinaryPredicate, BinaryPredicateOp, CastOptions, + ColumnName, Expression as Expr, JunctionPredicate, JunctionPredicateOp, OpaqueExpression, OpaqueExpressionOpRef, OpaquePredicate, OpaquePredicateOpRef, Predicate as Pred, Scalar, UnaryPredicate, UnaryPredicateOp, }; use crate::schema::DataType; -use std::cmp::Ordering; -use tracing::{debug, warn}; - pub(crate) mod parquet_stats_skipping; #[cfg(test)] @@ -26,7 +27,8 @@ mod tests; // below with generic lifetimes allows `&'r DynFoo<'a>` (again with `'a: 'r`). Unfortunately, // generic lifetimes cannot be hidden, so we end up with `&DynFoo<'_>` at every use site. -/// A predicate evaluator that directly evaluates predicates, resolving column references to scalar values. +/// A predicate evaluator that directly evaluates predicates, resolving column references to scalar +/// values. pub type DirectPredicateEvaluator<'a> = dyn KernelPredicateEvaluator + 'a; /// A data skipping predicate evaluator that directly applies data skipping, resolving column @@ -124,6 +126,24 @@ pub trait KernelPredicateEvaluator { inverted: bool, ) -> Option; + /// A (possibly inverted) comparison between `CAST(
    AS target)` and a scalar, e.g. + /// `CAST(AS DATE) < `. The scalar is always on the right (the caller commutes the + /// operator for ` op CAST()`). Unsupported by default so a cast never drives file + /// pruning. A per-row evaluator that resolves the column to its exact value can override it; a + /// range-based evaluator can only do so soundly when the cast preserves order over the range. + /// Implementations must return `None` for any [`CastOptions`] they cannot reproduce exactly. + fn eval_pred_cast( + &self, + _op: BinaryPredicateOp, + _col: &ColumnName, + _target: &DataType, + _options: &CastOptions, + _val: &Scalar, + _inverted: bool, + ) -> Option { + None + } + /// Dispatches an opaque predicate. fn eval_pred_opaque( &self, @@ -178,17 +198,20 @@ pub trait KernelPredicateEvaluator { self.eval_pred_expr_opaque(op, exprs, inverted) } Expr::Struct(..) - | Expr::Transform(_) + | Expr::StructPatch(_) | Expr::Unary(_) | Expr::Binary(_) | Expr::Variadic(_) | Expr::ParseJson(_) | Expr::MapToStruct(_) + | Expr::ElementAt(_) + | Expr::Cast(_) | Expr::Unknown(_) => None, } } - /// Dispatches a (possibly inverted) unary expression to each operator's specific implementation. + /// Dispatches a (possibly inverted) unary expression to each operator's specific + /// implementation. fn eval_pred_unary( &self, op: UnaryPredicateOp, @@ -206,13 +229,15 @@ pub trait KernelPredicateEvaluator { Expr::Column(col) => self.eval_pred_is_null(col, inverted), Expr::Predicate(_) | Expr::Struct(..) - | Expr::Transform(_) + | Expr::StructPatch(_) | Expr::Unary(_) | Expr::Binary(_) | Expr::Variadic(_) | Expr::Opaque(_) | Expr::ParseJson { .. } | Expr::MapToStruct(_) + | Expr::ElementAt(_) + | Expr::Cast(_) | Expr::Unknown(_) => { debug!("Unsupported operand: IS [NOT] NULL: {expr:?}"); None @@ -256,7 +281,8 @@ pub trait KernelPredicateEvaluator { None // TODO? } - /// Dispatches a (possibly inverted) binary expression to each operator's specific implementation. + /// Dispatches a (possibly inverted) binary expression to each operator's specific + /// implementation. /// /// NOTE: Only binary operators that produce boolean outputs are supported. fn eval_pred_binary( @@ -267,7 +293,7 @@ pub trait KernelPredicateEvaluator { inverted: bool, ) -> Option { use BinaryPredicateOp::*; - use Expr::{Column, Literal}; + use Expr::{Cast, Column, Literal}; match (left, right) { (Column(a), Column(b)) => self.eval_pred_binary_columns(op, a, b, inverted), @@ -287,6 +313,23 @@ pub trait KernelPredicateEvaluator { Distinct => self.eval_pred_distinct(col, val, inverted), In => None, // arg order is semantically important }, + (Cast(c), Literal(val)) => match c.expr.as_ref() { + Column(col) => self.eval_pred_cast(op, col, &c.target, &c.options, val, inverted), + _ => None, + }, + (Literal(val), Cast(c)) => match c.expr.as_ref() { + // Commute so the cast column is on the left. + Column(col) => { + let op = match op { + LessThan => GreaterThan, + GreaterThan => LessThan, + Equal | Distinct => op, + In => return None, // arg order is semantically important + }; + self.eval_pred_cast(op, col, &c.target, &c.options, val, inverted) + } + _ => None, + }, _ => { debug!("Unsupported binary operand(s): {left:?} {op:?} {right:?}"); None @@ -326,6 +369,10 @@ pub trait KernelPredicateEvaluator { /// Evaluates a (possibly inverted) predicate with SQL WHERE semantics. /// + /// NOTE: A NULL literal in a boolean position is treated as unknown (not false), because + /// callers like `build_actions_meta_predicate` use NULL as a sentinel for unsupported arms. + /// Treating it as false would let `AND(supported, NULL)` incorrectly prune files. + /// /// By default, [`Self::eval_pred`] behaves badly for comparisons involving NULL columns /// (e.g. `a < 10` when `a` is NULL), because the comparison correctly evaluates to NULL, but /// NULL values are interpreted as "stats missing" (= cannot skip). This ambiguity can "poison" @@ -430,7 +477,8 @@ pub trait KernelPredicateEvaluator { use Pred::*; match pred { Junction(JunctionPredicate { op, preds }) => { - // Recursively invoke `eval_pred_sql_where` instead of the usual `eval_pred` for AND/OR. + // Recursively invoke `eval_pred_sql_where` instead of the usual `eval_pred` for + // AND/OR. let mut preds = preds .iter() .map(|pred| self.eval_pred_sql_where(pred, inverted)); @@ -456,10 +504,6 @@ pub trait KernelPredicateEvaluator { .into_iter(); self.finish_eval_pred_junction(JunctionPredicateOp::And, &mut preds, false) } - BooleanExpression(Expr::Literal(val)) if val.is_null() => { - // AND(NULL IS NOT NULL, NULL) = AND(FALSE, NULL) = FALSE - self.eval_pred_scalar(&Scalar::from(false), false) - } BooleanExpression(Expr::Predicate(pred)) => self.eval_pred_sql_where(pred, inverted), // Process all remaining predicates normally, because they are not proven safe. Indeed, // predicates like DISTINCT and IS [NOT] NULL are known-unsafe under SQL semantics: @@ -480,7 +524,6 @@ pub trait KernelPredicateEvaluator { // = AND(TRUE, FALSE, TRUE) // = FALSE // ``` - // _ => self.eval_pred(pred, inverted), } } @@ -497,8 +540,8 @@ pub trait KernelPredicateEvaluator { } } -/// A collection of provided methods from the [`KernelPredicateEvaluator`] trait, factored out to allow -/// reuse by multiple bool-output predicate evaluator implementations. +/// A collection of provided methods from the [`KernelPredicateEvaluator`] trait, factored out to +/// allow reuse by multiple bool-output predicate evaluator implementations. pub struct KernelPredicateEvaluatorDefaults; impl KernelPredicateEvaluatorDefaults { /// Directly evaluates a boolean scalar. See [`KernelPredicateEvaluator::eval_pred_scalar`]. @@ -533,7 +576,8 @@ impl KernelPredicateEvaluatorDefaults { Some(matched != inverted) } - /// Directly evaluates a boolean comparison. See [`KernelPredicateEvaluator::eval_pred_binary_scalars`]. + /// Directly evaluates a boolean comparison. See + /// [`KernelPredicateEvaluator::eval_pred_binary_scalars`]. pub fn eval_pred_binary_scalars( op: BinaryPredicateOp, left: &Scalar, @@ -626,7 +670,7 @@ impl DefaultKernelPredicateEvaluator { Expr::Literal(value) => Some(value.clone()), Expr::Column(name) => self.resolve_column(name), Expr::Predicate(pred) => self.eval_pred(pred, false).map(Scalar::from), - Expr::Struct(..) | Expr::Transform(_) | Expr::Unary(_) => None, // TODO? + Expr::Struct(..) | Expr::StructPatch(_) | Expr::Unary(_) => None, // TODO? Expr::Binary(BinaryExpression { op, left, right }) => { let op_fn = match op { BinaryExpressionOp::Plus => Scalar::try_add, @@ -643,13 +687,42 @@ impl DefaultKernelPredicateEvaluator { warn!("Failed to evaluate {:?}: {err:?}", op.as_ref()); }) .ok(), - // ParseJson and MapToStruct produce structured output, not scalar values - Expr::ParseJson(_) | Expr::MapToStruct(_) => None, + Expr::Cast(c) if c.options == CastOptions::default() => { + cast_scalar(self.eval_expr(&c.expr)?, &c.target) + } + Expr::Cast(_) => None, + // ParseJson and MapToStruct produce structured output, and ElementAt is unsupported by + // this scalar evaluator. + Expr::ParseJson(_) | Expr::MapToStruct(_) | Expr::ElementAt(_) => None, Expr::Unknown(_) => None, } } } +/// Casts a scalar following SQL `CAST` semantics: a string source is parsed into a primitive target +/// via [`PrimitiveType::parse_scalar`](crate::schema::PrimitiveType::parse_scalar), and a source +/// already of the target type passes through. An unparseable string yields `Scalar::Null`. +/// +/// String-to-primitive, identity, and null-source conversions are handled; any other source/target +/// combination yields `None`, including numeric/temporal casts like `Long -> Int` that the arrow +/// path can perform. The accepted string formats are a strict subset of arrow's (e.g. arrow parses +/// the hyphen-less `20240115` as a date but `parse_scalar` does not), so this path is only ever +/// more conservative than arrow: a form it rejects becomes NULL, never a differing non-null value. +fn cast_scalar(value: Scalar, target: &DataType) -> Option { + if value.data_type() == *target { + return Some(value); + } + match (value, target) { + (Scalar::String(raw), DataType::Primitive(ptype)) => Some( + ptype + .parse_scalar(&raw) + .unwrap_or_else(|_| Scalar::Null(target.clone())), + ), + (Scalar::Null(_), _) => Some(Scalar::Null(target.clone())), + _ => None, + } +} + impl From for DefaultKernelPredicateEvaluator { fn from(resolver: R) -> Self { Self { resolver } @@ -690,6 +763,22 @@ impl KernelPredicateEvaluator for DefaultKernelPredica self.eval_pred_binary_scalars(BinaryPredicateOp::Equal, &col, val, inverted) } + fn eval_pred_cast( + &self, + op: BinaryPredicateOp, + col: &ColumnName, + target: &DataType, + options: &CastOptions, + val: &Scalar, + inverted: bool, + ) -> Option { + if options != &CastOptions::default() { + return None; + } + let col = cast_scalar(self.resolve_column(col)?, target)?; + self.eval_pred_binary_scalars(op, &col, val, inverted) + } + fn eval_pred_binary_scalars( &self, op: BinaryPredicateOp, @@ -805,6 +894,22 @@ pub trait DataSkippingPredicateEvaluator { inverted: bool, ) -> Option; + /// See [`KernelPredicateEvaluator::eval_pred_cast`]. + /// + /// Implementations may evaluate exact values or rewrite casts over exact-value references. A + /// bound-based consumer of a rewritten predicate must independently prove that the cast + /// preserves those bounds. Implementations must return `None` for any [`CastOptions`] they + /// cannot reproduce exactly. + fn eval_pred_cast( + &self, + op: BinaryPredicateOp, + col: &ColumnName, + target: &DataType, + options: &CastOptions, + val: &Scalar, + inverted: bool, + ) -> Option; + /// See [`KernelPredicateEvaluator::eval_pred_opaque`]. fn eval_pred_opaque( &self, @@ -832,7 +937,8 @@ pub trait DataSkippingPredicateEvaluator { ) -> Option; /// Performs a partial comparison against a column min-stat. See - /// [`KernelPredicateEvaluatorDefaults::partial_cmp_scalars`] for details of the comparison semantics. + /// [`KernelPredicateEvaluatorDefaults::partial_cmp_scalars`] for details of the comparison + /// semantics. fn partial_cmp_min_stat( &self, col: &ColumnName, @@ -845,7 +951,8 @@ pub trait DataSkippingPredicateEvaluator { } /// Performs a partial comparison against a column max-stat. See - /// [`KernelPredicateEvaluatorDefaults::partial_cmp_scalars`] for details of the comparison semantics. + /// [`KernelPredicateEvaluatorDefaults::partial_cmp_scalars`] for details of the comparison + /// semantics. fn partial_cmp_max_stat( &self, col: &ColumnName, @@ -971,6 +1078,20 @@ impl KernelPredicateEvaluator for T None } + fn eval_pred_cast( + &self, + op: BinaryPredicateOp, + col: &ColumnName, + target: &DataType, + options: &CastOptions, + val: &Scalar, + inverted: bool, + ) -> Option { + DataSkippingPredicateEvaluator::eval_pred_cast( + self, op, col, target, options, val, inverted, + ) + } + fn eval_pred_opaque( &self, op: &OpaquePredicateOpRef, diff --git a/kernel/src/kernel_predicates/parquet_stats_skipping.rs b/kernel/src/kernel_predicates/parquet_stats_skipping.rs index 73888c5b76..53e9a821a8 100644 --- a/kernel/src/kernel_predicates/parquet_stats_skipping.rs +++ b/kernel/src/kernel_predicates/parquet_stats_skipping.rs @@ -1,18 +1,18 @@ //! An implementation of data skipping that leverages parquet stats from the file footer. +use std::cmp::Ordering; + use crate::expressions::{ - BinaryPredicateOp, ColumnName, Expression, JunctionPredicateOp, OpaquePredicateOpRef, Scalar, + BinaryPredicateOp, CastOptions, ColumnName, Expression, JunctionPredicateOp, + OpaquePredicateOpRef, Scalar, }; use crate::kernel_predicates::{DataSkippingPredicateEvaluator, KernelPredicateEvaluatorDefaults}; use crate::schema::DataType; -use std::cmp::Ordering; - #[cfg(test)] mod tests; -/// A helper trait (mostly exposed for testing). It provides the four stats getters needed by -/// [`DataSkippingStatsProvider`]. From there, we can automatically derive a -/// [`DataSkippingPredicateEvaluator`]. +/// A helper trait (mostly exposed for testing). It provides the four stats getters needed to +/// derive a [`DataSkippingPredicateEvaluator`] via the blanket impl below. pub(crate) trait ParquetStatsProvider { /// The min-value stat for this column, if the column exists in this file, has the expected /// type, and the parquet footer provides stats for it. @@ -26,8 +26,10 @@ pub(crate) trait ParquetStatsProvider { /// type, and the parquet footer provides stats for it. fn get_parquet_nullcount_stat(&self, col: &ColumnName) -> Option; - /// The rowcount stat for this row group. It is always available in the parquet footer. - fn get_parquet_rowcount_stat(&self) -> i64; + /// The rowcount stat for this row group. Returns `None` if the rowcount is not meaningful + /// (e.g. in checkpoint files where the footer rowcount is the number of add file rows, not + /// the sum of data file row counts). + fn get_parquet_rowcount_stat(&self) -> Option; } // Blanket implementation for all types that impl ParquetStatsProvider. @@ -48,7 +50,7 @@ impl DataSkippingPredicateEvaluator for T { } fn get_rowcount_stat(&self) -> Option { - Some(Scalar::from(self.get_parquet_rowcount_stat())) + self.get_parquet_rowcount_stat().map(Scalar::from) } fn eval_partial_cmp( @@ -89,6 +91,18 @@ impl DataSkippingPredicateEvaluator for T { KernelPredicateEvaluatorDefaults::eval_pred_binary_scalars(op, left, right, inverted) } + fn eval_pred_cast( + &self, + _op: BinaryPredicateOp, + _col: &ColumnName, + _target: &DataType, + _options: &CastOptions, + _val: &Scalar, + _inverted: bool, + ) -> Option { + None + } + fn eval_pred_opaque( &self, op: &OpaquePredicateOpRef, diff --git a/kernel/src/kernel_predicates/parquet_stats_skipping/tests.rs b/kernel/src/kernel_predicates/parquet_stats_skipping/tests.rs index 44d79e23d0..d17a4ec352 100644 --- a/kernel/src/kernel_predicates/parquet_stats_skipping/tests.rs +++ b/kernel/src/kernel_predicates/parquet_stats_skipping/tests.rs @@ -1,5 +1,5 @@ use super::*; -use crate::expressions::{column_expr, Expression as Expr, Predicate as Pred}; +use crate::expressions::{col, lit, CastOptions, Expression as Expr, Predicate as Pred}; use crate::kernel_predicates::KernelPredicateEvaluator as _; use crate::DataType; @@ -35,11 +35,22 @@ impl ParquetStatsProvider for UnimplementedTestFilter { unimplemented!() } - fn get_parquet_rowcount_stat(&self) -> i64 { + fn get_parquet_rowcount_stat(&self) -> Option { unimplemented!() } } +#[test] +fn test_cast_stays_unknown_for_parquet_stats() { + let filter = UnimplementedTestFilter; + let pred = Pred::eq( + Expr::cast(col!("x"), DataType::DATE, CastOptions::default()), + Scalar::Date(19_723), + ); + + expect_eq!(filter.eval(&pred), NULL, "{pred:#?}"); +} + /// Tests apply_junction and apply_scalar #[test] fn test_junctions() { @@ -94,7 +105,7 @@ fn test_junctions() { .iter() .map(|val| match val { Some(v) => Pred::literal(*v), - None => Pred::null_literal(), + None => Pred::NULL, }) .collect(); @@ -148,7 +159,7 @@ impl ParquetStatsProvider for MinMaxTestFilter { unimplemented!() } - fn get_parquet_rowcount_stat(&self) -> i64 { + fn get_parquet_rowcount_stat(&self) -> Option { unimplemented!() } } @@ -161,12 +172,12 @@ fn test_eval_binary_comparisons() { const NULL_VAL: Scalar = Scalar::Null(DataType::INTEGER); let predicates = [ - Pred::lt(column_expr!("x"), Expr::literal(10)), - Pred::le(column_expr!("x"), Expr::literal(10)), - Pred::eq(column_expr!("x"), Expr::literal(10)), - Pred::ne(column_expr!("x"), Expr::literal(10)), - Pred::gt(column_expr!("x"), Expr::literal(10)), - Pred::ge(column_expr!("x"), Expr::literal(10)), + Pred::lt(col!("x"), lit(10)), + Pred::le(col!("x"), lit(10)), + Pred::eq(col!("x"), lit(10)), + Pred::ne(col!("x"), lit(10)), + Pred::gt(col!("x"), lit(10)), + Pred::ge(col!("x"), lit(10)), ]; let do_test = |min: Scalar, max: Scalar, expected: &[Option]| { @@ -221,17 +232,14 @@ impl ParquetStatsProvider for NullCountTestFilter { self.nullcount } - fn get_parquet_rowcount_stat(&self) -> i64 { - self.rowcount + fn get_parquet_rowcount_stat(&self) -> Option { + Some(self.rowcount) } } #[test] fn test_eval_is_null() { - let expressions = [ - Pred::is_null(column_expr!("x")), - Pred::is_not_null(column_expr!("x")), - ]; + let expressions = [Pred::is_null(col!("x")), Pred::is_not_null(col!("x"))]; let do_test = |nullcount: i64, expected: &[Option]| { let filter = NullCountTestFilter::new(Some(nullcount), 2); diff --git a/kernel/src/kernel_predicates/tests.rs b/kernel/src/kernel_predicates/tests.rs index bff413989e..54fa42d5de 100644 --- a/kernel/src/kernel_predicates/tests.rs +++ b/kernel/src/kernel_predicates/tests.rs @@ -1,15 +1,15 @@ +use std::collections::HashMap; + use super::*; use crate::expressions::{ - column_expr, column_name, column_pred, ArrayData, Expression as Expr, OpaqueExpressionOp, - OpaquePredicateOp, Predicate as Pred, ScalarExpressionEvaluator, StructData, + col, column_name, column_pred, lit, ArrayData, CastOptions, Expression as Expr, + OpaqueExpressionOp, OpaquePredicateOp, Predicate as Pred, ScalarExpressionEvaluator, + StructData, }; use crate::kernel_predicates::parquet_stats_skipping::ParquetStatsProvider; use crate::scan::data_skipping::as_data_skipping_predicate; use crate::schema::ArrayType; -use crate::DataType; -use crate::DeltaResult; - -use std::collections::HashMap; +use crate::{DataType, DeltaResult}; /// Helper trait to allow expect_eq! to work with both Option and Option trait LogicalEq { @@ -52,6 +52,47 @@ impl ResolveColumnAsScalar for Scalar { } } +/// Resolves the data-skipping rewrite's `is_add` guard column to a fixed boolean and every other +/// column to the wrapped scalar. Lets opaque-rewrite tests evaluate the guarded predicate +/// (`OR(NOT is_add, ...)`) for Add rows (`is_add = true`) and Remove rows (`is_add = false`). +struct IsAddResolver(Scalar, bool); + +impl ResolveColumnAsScalar for IsAddResolver { + fn resolve_column(&self, col: &ColumnName) -> Option { + if *col == column_name!("is_add") { + Some(Scalar::from(self.1)) + } else { + Some(self.0.clone()) + } + } +} + +#[rstest::rstest] +#[case::string_to_integer_succeeds( + Scalar::String("42".to_string()), + DataType::INTEGER, + Some(Scalar::Integer(42)) +)] +#[case::string_to_integer_failure_is_null( + Scalar::String("not an integer".to_string()), + DataType::INTEGER, + Some(Scalar::Null(DataType::INTEGER)) +)] +#[case::identity_passthrough(Scalar::Long(42), DataType::LONG, Some(Scalar::Long(42)))] +#[case::null_source_uses_target_type( + Scalar::Null(DataType::STRING), + DataType::INTEGER, + Some(Scalar::Null(DataType::INTEGER)) +)] +#[case::unsupported_pair(Scalar::Long(42), DataType::INTEGER, None)] +fn test_cast_scalar( + #[case] value: Scalar, + #[case] target: DataType, + #[case] expected: Option, +) { + assert_eq!(cast_scalar(value, &target), expected); +} + #[rstest::rstest] #[case::bool_true_not_inverted(Scalar::Boolean(true), false, Some(true))] #[case::bool_true_inverted(Scalar::Boolean(true), true, Some(false))] @@ -377,7 +418,8 @@ fn test_eval_binary_scalars() { } } -// NOTE: We're testing routing here -- the actual comparisons are already validated by test_eval_binary_scalars. +// NOTE: We're testing routing here -- the actual comparisons are already validated by +// test_eval_binary_scalars. #[test] fn test_eval_binary_columns() { let columns = HashMap::from_iter(vec![ @@ -385,8 +427,8 @@ fn test_eval_binary_columns() { (column_name!("y"), Scalar::from(10)), ]); let filter = DefaultKernelPredicateEvaluator::from(columns); - let x = column_expr!("x"); - let y = column_expr!("y"); + let x = col!("x"); + let y = col!("y"); for inverted in [true, false] { assert_eq!( filter.eval_pred_binary(BinaryPredicateOp::Equal, &x, &y, inverted), @@ -429,7 +471,7 @@ fn test_eval_junction() { .cloned() .map(|v| match v { Some(v) => Pred::literal(v), - None => Pred::null_literal(), + None => Pred::NULL, }) .collect(); for inverted in [true, false] { @@ -489,7 +531,7 @@ fn test_eval_not( #[test] fn test_eval_is_null() { use crate::expressions::UnaryPredicateOp::IsNull; - let expr = column_expr!("x"); + let expr = col!("x"); let filter = DefaultKernelPredicateEvaluator::from(Scalar::from(1)); expect_eq!( filter.eval_pred_unary(IsNull, &expr, true), @@ -576,14 +618,60 @@ fn test_eval_distinct() { ); } +#[test] +fn test_default_evaluator_resolves_column_then_casts_and_compares() { + let col = column_name!("x"); + let filter = DefaultKernelPredicateEvaluator::from(HashMap::from([( + col.clone(), + Scalar::String("10".to_string()), + )])); + + assert_eq!( + filter.eval_pred_cast( + BinaryPredicateOp::LessThan, + &col, + &DataType::INTEGER, + &CastOptions::default(), + &Scalar::Integer(20), + false, + ), + Some(true) + ); +} + +#[test] +fn default_evaluator_does_not_fold_cast_options_it_cannot_honor() { + let col = column_name!("x"); + let options = CastOptions::default().with_timestamp_timezone("America/Los_Angeles"); + let evaluator = DefaultKernelPredicateEvaluator::from(HashMap::from([( + col.clone(), + Scalar::String("2024-01-15 12:30:45".to_string()), + )])); + assert_eq!( + evaluator.eval_pred_cast( + BinaryPredicateOp::LessThan, + &col, + &DataType::TIMESTAMP, + &options, + &Scalar::Timestamp(1_705_350_645_000_000), + false, + ), + None + ); + + let expression = Expr::cast(lit("2024-01-15 12:30:45"), DataType::TIMESTAMP, options); + + assert_eq!(evaluator.eval_expr(&expression), None); +} + // NOTE: We're testing routing here -- the actual comparisons are already validated by // test_eval_binary_scalars. #[test] fn eval_binary() { use crate::expressions::BinaryPredicateOp; - let col = column_expr!("x"); - let val = Expr::literal(10); + let col = col!("x"); + let val = lit(10); let filter = DefaultKernelPredicateEvaluator::from(Scalar::from(1)); for inverted in [true, false] { @@ -631,6 +719,24 @@ fn eval_binary() { } } +#[rstest::rstest] +#[case::less_than(BinaryPredicateOp::LessThan, false)] +#[case::greater_than(BinaryPredicateOp::GreaterThan, true)] +fn test_eval_binary_commutes_literal_and_cast_column( + #[case] op: BinaryPredicateOp, + #[case] expected: bool, + #[values(false, true)] inverted: bool, +) { + let val = lit(10); + let cast_col = Expr::cast(col!("x"), DataType::INTEGER, CastOptions::default()); + let filter = DefaultKernelPredicateEvaluator::from(Scalar::String("1".to_string())); + + assert_eq!( + filter.eval_pred_binary(op, &val, &cast_col, inverted), + Some(expected != inverted) + ); +} + #[derive(Debug, PartialEq)] struct OpaqueLessThanOp; impl OpaqueLessThanOp { @@ -713,8 +819,8 @@ impl OpaquePredicateOp for OpaqueLessThanOp { _ => return None, }; - // NOTE: `evaluator.partial_cmp_min`_stat returns `Pred::Binary`. That's fine, because we have - // separate testing for the `eval_pred_scalar` path. + // NOTE: `evaluator.partial_cmp_min`_stat returns `Pred::Binary`. That's fine, because we + // have separate testing for the `eval_pred_scalar` path. evaluator.partial_cmp_min_stat(col, val, ord, inverted) } } @@ -734,31 +840,39 @@ impl ParquetStatsProvider for MinStatsValue { Some(0) } - fn get_parquet_rowcount_stat(&self) -> i64 { - 1 + fn get_parquet_rowcount_stat(&self) -> Option { + Some(1) } } #[test] fn test_eval_opaque_simple() { - let expr = Expr::opaque(OpaqueLessThanOp, vec![column_expr!("x"), Expr::literal(10)]); - let pred = Pred::opaque(OpaqueLessThanOp, vec![column_expr!("x"), Expr::literal(10)]); + let expr = Expr::opaque(OpaqueLessThanOp, vec![col!("x"), lit(10)]); + let pred = Pred::opaque(OpaqueLessThanOp, vec![col!("x"), lit(10)]); let skipping_pred = as_data_skipping_predicate(&pred).unwrap(); assert_eq!(expr, expr); assert_eq!(pred, pred); - // Test direct expression and predicate eval, and indirect data skipping + // Test direct expression and predicate eval, and indirect data skipping. The rewrite wraps + // opaque predicates with `OR(NOT is_add, ...)`, so skipping assertions resolve `is_add`. let filter = DefaultKernelPredicateEvaluator::from(Scalar::from(1)); assert_eq!(filter.eval_expr(&expr), Some(Scalar::from(true)), "x < 10"); assert_eq!(filter.eval(&pred), Some(true), "x < 10"); + let filter = DefaultKernelPredicateEvaluator::from(IsAddResolver(Scalar::from(1), true)); assert_eq!(filter.eval(&skipping_pred), Some(true), "x < 10"); let filter = DefaultKernelPredicateEvaluator::from(Scalar::from(100)); assert_eq!(filter.eval_expr(&expr), Some(Scalar::from(false)), "x < 10"); assert_eq!(filter.eval(&pred), Some(false), "x < 10"); + let filter = DefaultKernelPredicateEvaluator::from(IsAddResolver(Scalar::from(100), true)); assert_eq!(filter.eval(&skipping_pred), Some(false), "x < 10"); + // Remove-row guard: even when the stats would prune, a non-Add row must be kept -- the + // opaque rewrite cannot drop Removes from log replay. + let filter = DefaultKernelPredicateEvaluator::from(IsAddResolver(Scalar::from(100), false)); + assert_eq!(filter.eval(&skipping_pred), Some(true), "remove row kept"); + // Test direct data skipping let filter = MinStatsValue(Scalar::from(1)); assert_eq!(filter.eval(&pred), Some(true), "x < 10"); @@ -770,7 +884,7 @@ fn test_eval_opaque_simple() { let filter = DefaultKernelPredicateEvaluator::from(Scalar::from(1)); let pred = Pred::from_expr(Expr::from(Pred::opaque( OpaqueLessThanOp, - vec![column_expr!("x"), Expr::literal(10)], + vec![col!("x"), lit(10)], ))); assert_eq!(filter.eval(&pred), Some(true), "pred(expr(x < 10))"); } @@ -839,30 +953,53 @@ impl ParquetStatsProvider for OneStatsValue { Some(nullcount) } - fn get_parquet_rowcount_stat(&self) -> i64 { - 1 + fn get_parquet_rowcount_stat(&self) -> Option { + Some(1) } } #[test] fn test_eval_opaque_predicate() { - let pred = Pred::opaque(OpaqueAndOp, vec![column_expr!("x"), Expr::literal(true)]); + let pred = Pred::opaque(OpaqueAndOp, vec![col!("x"), lit(true)]); let skipping_pred = as_data_skipping_predicate(&pred).unwrap(); - // Test direct evaluation and indirect data skipping + // Direct evaluation works for any column type. let filter = DefaultKernelPredicateEvaluator::from(Scalar::from(true)); assert_eq!(filter.eval(&pred), Some(true), "AND(x, TRUE)"); - assert_eq!(filter.eval(&skipping_pred), Some(true), "AND(x, TRUE)"); let filter = DefaultKernelPredicateEvaluator::from(Scalar::from(false)); assert_eq!(filter.eval(&pred), Some(false), "AND(x, TRUE)"); - assert_eq!(filter.eval(&skipping_pred), Some(false), "AND(x, TRUE)"); let filter = DefaultKernelPredicateEvaluator::from(Scalar::Null(DataType::BOOLEAN)); assert_eq!(filter.eval(&pred), None, "AND(x, TRUE)"); - assert_eq!(filter.eval(&skipping_pred), None, "AND(x, TRUE)"); - // Test direct data skipping + // Indirect data skipping: `x` is Boolean and Boolean columns carry no min/max stats + // per the Delta protocol (see `is_skipping_eligible_datatype`). The rewrite therefore + // drops the `x` arm, leaving `AND(NULL, TRUE)` which evaluates to NULL for every filter. + // The rewrite's `OR(NOT is_add, ...)` guard is transparent for Add rows (`is_add = true`). + for value in [ + Scalar::from(true), + Scalar::from(false), + Scalar::Null(DataType::BOOLEAN), + ] { + assert_eq!( + DefaultKernelPredicateEvaluator::from(IsAddResolver(value, true)).eval(&skipping_pred), + None, + "AND(x, TRUE) -- Boolean min/max unsupported" + ); + } + + // Remove-row guard: non-Add rows are always kept by the rewritten predicate. + assert_eq!( + DefaultKernelPredicateEvaluator::from(IsAddResolver(Scalar::from(false), false)) + .eval(&skipping_pred), + Some(true), + "remove row kept" + ); + + // Direct evaluation through a `ParquetStatsProvider` still works because that path + // doesn't go through min/max rewriting -- `OpaqueAndOp::eval_pred_scalar` evaluates + // each arm directly against the provider. let filter = OneStatsValue(Scalar::from(true)); assert_eq!(filter.eval(&pred), Some(true), "AND(x, TRUE)"); @@ -877,12 +1014,12 @@ fn test_eval_opaque_predicate() { fn test_eval_opaque_complex() { // A contrived example that uses an opaque predicate that references an opaque expression let complex_pred = Pred::and( - Pred::lt(column_expr!("x"), Scalar::from(true)), + Pred::lt(col!("x"), lit(true)), Pred::opaque( OpaqueLessThanOp, vec![ - column_expr!("x"), - Expr::opaque(OpaqueLessThanOp, vec![Expr::literal(2), Expr::literal(5)]), + col!("x"), + Expr::opaque(OpaqueLessThanOp, vec![lit(2), lit(5)]), ], ), ); @@ -909,10 +1046,13 @@ fn test_eval_opaque_complex() { Some(false), "AND(x < TRUE, x < (2 < 5))" ); + // Post-fold, `x < TRUE` no longer produces a stats-based skipping arm because Boolean + // columns carry no min/max stats per the Delta protocol. Combined with the already-NULL + // opaque-expression arm, `complex_skipping_pred` evaluates to NULL for every filter. assert_eq!( filter.eval(&complex_skipping_pred), - Some(false), - "AND(x < TRUE, x < (2 < 5))" + None, + "AND(x < TRUE, x < (2 < 5)) -- Boolean min/max unsupported" ); } @@ -933,12 +1073,8 @@ impl ResolveColumnAsScalar for NullColumnResolver { #[test] fn test_sql_where() { - let col = &column_expr!("x"); let col_pred = &column_pred!("x"); const VAL: Expr = Expr::Literal(Scalar::Integer(1)); - const NULL: Pred = Pred::null_literal(); - const FALSE: Pred = Pred::literal(false); - const TRUE: Pred = Pred::literal(true); let null_filter = DefaultKernelPredicateEvaluator::from(NullColumnResolver); let empty_filter = DefaultKernelPredicateEvaluator::from(EmptyColumnResolver); @@ -966,7 +1102,7 @@ fn test_sql_where() { ); // SQL eval does not modify behavior of IS NULL - let pred = &Pred::is_null(col.clone()); + let pred = &col!("x").is_null(); expect_eq!(null_filter.eval_sql_where(pred), Some(true), "{pred}"); // NOT a gets skipped when NULL but not when missing @@ -975,52 +1111,54 @@ fn test_sql_where() { expect_eq!(empty_filter.eval_sql_where(pred), None, "{pred}"); // Injected NULL checks only short circuit if inputs are NULL - let pred = &Pred::lt(FALSE, TRUE); + let pred = &Pred::lt(Pred::FALSE, Pred::TRUE); expect_eq!(null_filter.eval_sql_where(pred), Some(true), "{pred}"); expect_eq!(empty_filter.eval_sql_where(pred), Some(true), "{pred}"); // Contrast normal vs SQL WHERE semantics - comparison - let pred = &Pred::lt(col.clone(), VAL); + let pred = &Pred::lt(col!("x"), VAL); expect_eq!(null_filter.eval(pred), None, "{pred}"); expect_eq!(null_filter.eval_sql_where(pred), Some(false), "{pred}"); // NULL check produces NULL due to missing column expect_eq!(empty_filter.eval_sql_where(pred), None, "{pred}"); - let pred = &Pred::lt(VAL, col.clone()); + let pred = &Pred::lt(VAL, col!("x")); expect_eq!(null_filter.eval(pred), None, "{pred}"); expect_eq!(null_filter.eval_sql_where(pred), Some(false), "{pred}"); expect_eq!(empty_filter.eval_sql_where(pred), None, "{pred}"); - let pred = &Pred::distinct(VAL, col.clone()); + let pred = &Pred::distinct(VAL, col!("x")); expect_eq!(null_filter.eval(pred), Some(true), "{pred}"); expect_eq!(null_filter.eval_sql_where(pred), Some(true), "{pred}"); expect_eq!(empty_filter.eval_sql_where(pred), None, "{pred}"); - let pred = &Pred::distinct(NULL, col.clone()); + let pred = &Pred::distinct(Pred::NULL, col!("x")); expect_eq!(null_filter.eval(pred), Some(false), "{pred}"); expect_eq!(null_filter.eval_sql_where(pred), Some(false), "{pred}"); expect_eq!(empty_filter.eval_sql_where(pred), None, "{pred}"); // Contrast normal vs SQL WHERE semantics - comparison inside AND - let pred = &Pred::and(TRUE, Pred::lt(col.clone(), VAL)); + let pred = &Pred::and(Pred::TRUE, Pred::lt(col!("x"), VAL)); expect_eq!(null_filter.eval(pred), None, "{pred}"); expect_eq!(null_filter.eval_sql_where(pred), Some(false), "{pred}"); expect_eq!(empty_filter.eval_sql_where(pred), None, "{pred}"); - // NULL allows static skipping under SQL semantics - let pred = &Pred::and(NULL, Pred::lt(col.clone(), VAL)); + // NULL literal is treated as unknown (not false) under eval_sql_where, so it does not + // force static skipping. This prevents incorrect pruning when indirect data skipping + // rewriters use NULL as a sentinel for unsupported predicate arms. + let pred = &Pred::and(Pred::NULL, Pred::lt(col!("x"), VAL)); expect_eq!(null_filter.eval(pred), None, "{pred}"); expect_eq!(null_filter.eval_sql_where(pred), Some(false), "{pred}"); - expect_eq!(empty_filter.eval_sql_where(pred), Some(false), "{pred}"); + expect_eq!(empty_filter.eval_sql_where(pred), None, "{pred}"); // Contrast normal vs. SQL WHERE semantics - comparison inside AND inside AND - let pred = &Pred::and(TRUE, Pred::and(TRUE, Pred::lt(col.clone(), VAL))); + let pred = &Pred::and(Pred::TRUE, Pred::and(Pred::TRUE, Pred::lt(col!("x"), VAL))); expect_eq!(null_filter.eval(pred), None, "{pred}"); expect_eq!(null_filter.eval_sql_where(pred), Some(false), "{pred}"); expect_eq!(empty_filter.eval_sql_where(pred), None, "{pred}"); // Ditto for comparison inside OR inside AND - let pred = &Pred::or(FALSE, Pred::and(TRUE, Pred::lt(col.clone(), VAL))); + let pred = &Pred::or(Pred::FALSE, Pred::and(Pred::TRUE, Pred::lt(col!("x"), VAL))); expect_eq!(null_filter.eval(pred), None, "{pred}"); expect_eq!(null_filter.eval_sql_where(pred), Some(false), "{pred}"); expect_eq!(empty_filter.eval_sql_where(pred), None, "{pred}"); diff --git a/kernel/src/last_checkpoint_hint.rs b/kernel/src/last_checkpoint_hint.rs index f0a8c49303..02d6384183 100644 --- a/kernel/src/last_checkpoint_hint.rs +++ b/kernel/src/last_checkpoint_hint.rs @@ -1,23 +1,35 @@ -//! Utities for reading the `_last_checkpoint` file. Maybe this file should instead go under +//! Utilities for reading the `_last_checkpoint` file. Maybe this file should instead go under //! log_segment module since it should only really be used there? as hint for listing? use std::collections::HashMap; -use crate::schema::SchemaRef; -use crate::{DeltaResult, Error, StorageHandler, Version}; use delta_kernel_derive::internal_api; - use serde::{Deserialize, Serialize}; -use tracing::{info, instrument, warn}; +use tracing::{debug, info, instrument, warn}; use url::Url; +use crate::actions::{ + CheckpointMetadata, DomainMetadata, Metadata, Protocol, SetTransaction, Sidecar, +}; +use crate::path::{CheckpointInstance, ParsedLogPath}; +use crate::schema::SchemaRef; +use crate::{DeltaResult, Error, FileMeta, StorageHandler, Version}; + /// Name of the _last_checkpoint file that provides metadata about the last checkpoint /// created for the table. This file is used as a hint for the engine to quickly locate /// the latest checkpoint without a full directory listing. const LAST_CHECKPOINT_FILE_NAME: &str = "_last_checkpoint"; +/// Per-field count cap on a retained hint's `sidecarFiles` / `nonFileActions`. Matches the +/// Delta-Spark defaults for `lastCheckpoint.{sidecars,nonFileActions}.threshold` (both 30), which +/// drop the whole field by count when it exceeds the cap: +/// +const LAST_CHECKPOINT_SIDECARS_THRESHOLD: usize = 30; +const LAST_CHECKPOINT_NON_FILE_ACTIONS_THRESHOLD: usize = 30; + // Note: Schema can not be derived because the checkpoint schema is only known at runtime. -#[derive(Debug, Deserialize, Serialize, PartialEq, Eq)] +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq, Eq)] +#[cfg_attr(test, derive(Default))] #[serde(rename_all = "camelCase")] #[internal_api] pub(crate) struct LastCheckpointHint { @@ -26,7 +38,8 @@ pub(crate) struct LastCheckpointHint { pub version: Version, /// The number of actions that are stored in the checkpoint. pub(crate) size: i64, - /// The number of fragments if the last checkpoint was written in multiple parts. + /// The number of fragments if the last checkpoint was written in multiple parts. `None` means + /// a single-part or classic checkpoint (i.e. `numParts == 1`). pub(crate) parts: Option, /// The number of bytes of the checkpoint. pub(crate) size_in_bytes: Option, @@ -38,10 +51,134 @@ pub(crate) struct LastCheckpointHint { pub(crate) checksum: Option, /// Additional metadata about the last checkpoint. pub(crate) tags: Option>, + /// For a V2 checkpoint, the embedded V2 checkpoint info. Identifies the specific checkpoint + /// file the hint describes. Absent for V1 / classic checkpoints. + pub(crate) v2_checkpoint: Option, +} + +/// The `v2Checkpoint` object embedded in a `_last_checkpoint` hint for a V2 checkpoint. +/// +/// Carries the V2 checkpoint file's identity and metadata plus the actions a reader would otherwise +/// read from the checkpoint itself -- its sidecar references and its non-file actions. Absent for +/// V1 / classic checkpoints. +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq, Eq)] +#[cfg_attr(test, derive(Default))] +#[serde(rename_all = "camelCase")] +#[internal_api] +pub(crate) struct LastCheckpointV2 { + /// Bare file name of the V2 checkpoint this hint describes, matched against the selected + /// checkpoint part's file name. Several V2 checkpoints can share a version, so this identifies + /// which one the hint's fields describe. + pub(crate) path: String, + + /// Size in bytes of the V2 checkpoint file named by `path`. + pub(crate) size_in_bytes: Option, + + /// Modification time of the V2 checkpoint file named by `path`, in milliseconds since the Unix + /// epoch. + pub(crate) modification_time: Option, + + /// The sidecar files this checkpoint references, for a manifest (non-leaf) V2 checkpoint. + /// Empty/absent for a leaf checkpoint that inlines its file actions. Also dropped to `None` by + /// [`LastCheckpointHint::drop_oversized_fields`] when the count exceeds the threshold, so + /// absence is a missing optimization, never a signal that the checkpoint is a leaf. + pub(crate) sidecar_files: Option>, + + /// The checkpoint's non-file actions (see [`HintAction`]), letting a reader obtain them + /// without reading the checkpoint file. Dropped to `None` by + /// [`LastCheckpointHint::drop_oversized_fields`] when the count exceeds the threshold. + pub(crate) non_file_actions: Option>, +} + +/// One element of [`LastCheckpointV2`]'s `non_file_actions`. A log action is exactly one action +/// type, so this is an externally-tagged enum keyed by the action name, reusing kernel's action +/// structs to yield the same types as log replay. An unrecognized action key fails the whole-hint +/// parse; `try_read` swallows that, so the reader falls back to reading the checkpoint. +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq, Eq)] +#[serde(rename_all = "camelCase")] +#[internal_api] +pub(crate) enum HintAction { + #[serde(rename = "metaData")] + Metadata(Metadata), + Protocol(Protocol), + Txn(SetTransaction), + DomainMetadata(DomainMetadata), + CheckpointMetadata(CheckpointMetadata), } impl LastCheckpointHint { - // Returns the path the last checkpoint file given the log root of a table. + /// Whether this hint describes the checkpoint a log segment selected, given that segment's + /// `checkpoint_parts`. Multiple checkpoints can share a version, so a matching version alone is + /// not enough: the hint's own identity must equal the selected checkpoint's. + /// + /// On a mismatch, callers read the checkpoint file itself instead of trusting the hint's + /// fields. + pub(crate) fn applies_to(&self, checkpoint_parts: &[ParsedLogPath]) -> bool { + let Some(selected) = checkpoint_parts.first() else { + return false; + }; + self.version == selected.version + && CheckpointInstance::of(selected) == self.implied_instance() + } + + /// The checkpoint identity this hint's fields describe, mirroring Delta-Spark's + /// `getFormatEnum`: a `v2Checkpoint` object means uuid-named, else `parts` means multi-part, + /// else classic-named. `None` if `parts` overflows `u32`, so no checkpoint can match it. + fn implied_instance(&self) -> Option { + Some(match (&self.v2_checkpoint, self.parts) { + (Some(v2), _) => CheckpointInstance::Uuid { + filename: v2.path.clone(), + }, + (None, Some(parts)) => CheckpointInstance::MultiPart { + num_parts: parts.try_into().ok()?, + }, + (None, None) => CheckpointInstance::Classic, + }) + } + + /// Parses a hint from raw `_last_checkpoint` bytes, dropping oversized fields so the retained + /// hint is always bounded. This is the only way to construct a hint from disk, so callers can + /// never hold an untrimmed one. + fn from_bytes_with_oversized_fields_dropped(bytes: &[u8]) -> serde_json::Result { + let hint: Self = serde_json::from_slice(bytes)?; + Ok(hint.drop_oversized_fields()) + } + + /// Drops `sidecarFiles` / `nonFileActions` over the threshold. Drops the whole field, never + /// truncates. Absent means info missing, so this only loses an optimization. + fn drop_oversized_fields(mut self) -> Self { + if let Some(v2) = &mut self.v2_checkpoint { + let version = self.version; + if let Some(count) = v2 + .sidecar_files + .as_ref() + .map(Vec::len) + .filter(|&n| n > LAST_CHECKPOINT_SIDECARS_THRESHOLD) + { + debug!( + version, + count, "dropping _last_checkpoint sidecarFiles above threshold" + ); + v2.sidecar_files = None; + } + if let Some(count) = v2 + .non_file_actions + .as_ref() + .map(Vec::len) + .filter(|&n| n > LAST_CHECKPOINT_NON_FILE_ACTIONS_THRESHOLD) + { + debug!( + version, + count, "dropping _last_checkpoint nonFileActions above threshold" + ); + v2.non_file_actions = None; + } + } + self + } + + /// Returns the path of the `_last_checkpoint` file given the log root of a table. + #[internal_api] pub(crate) fn path(log_root: &Url) -> DeltaResult { Ok(log_root.join(LAST_CHECKPOINT_FILE_NAME)?) } @@ -62,9 +199,10 @@ impl LastCheckpointHint { let file_path = Self::path(log_root)?; match storage.read_files(vec![(file_path, None)])?.next() { Some(Ok(data)) => { - let result: Option = serde_json::from_slice(&data) - .inspect_err(|e| warn!("invalid _last_checkpoint JSON: {e}")) - .ok(); + let result: Option = + Self::from_bytes_with_oversized_fields_dropped(&data) + .inspect_err(|e| warn!("invalid _last_checkpoint JSON: {e}")) + .ok(); info!(hint = result.as_ref().map(|h| h.summary())); Ok(result) } @@ -94,3 +232,490 @@ impl LastCheckpointHint { serde_json::to_vec(self).expect("Failed to convert LastCheckpointHint to JSON bytes") } } + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + use crate::schema::{DataType, StructField, StructType}; + use crate::table_features::TableFeature; + use crate::unit_test_utils::create_log_path; + + /// A real `_last_checkpoint` for a V2 checkpoint carries a `v2Checkpoint` object; we parse its + /// `path` and file metadata. An empty `sidecarFiles` (a leaf checkpoint) parses to `Some([])`, + /// distinct from an absent field. Guards the `camelCase` wire keys -- a rename would otherwise + /// silently parse to `None` (errors are swallowed in `try_read`) and disable the identity + /// filter. + #[test] + fn parses_v2_checkpoint_path_from_wire_json() { + let json = br#"{ + "version": 5, + "size": 10, + "v2Checkpoint": { + "path": "00000000000000000005.checkpoint.0190e8f5-uuid.parquet", + "sizeInBytes": 1234, + "modificationTime": 1700000000000, + "sidecarFiles": [] + } + }"#; + let hint: LastCheckpointHint = serde_json::from_slice(json).unwrap(); + let v2 = hint.v2_checkpoint.expect("v2Checkpoint present"); + assert_eq!( + v2.path, + "00000000000000000005.checkpoint.0190e8f5-uuid.parquet" + ); + assert_eq!(v2.size_in_bytes, Some(1234)); + assert_eq!(v2.modification_time, Some(1700000000000)); + assert_eq!(v2.sidecar_files, Some(vec![])); + assert_eq!(v2.non_file_actions, None); + } + + /// A manifest V2 checkpoint hint carries its sidecar references and non-file actions. Each + /// non-file action decodes to exactly one [`HintAction`] variant via its action key. + #[test] + fn parses_v2_checkpoint_sidecars_and_non_file_actions() { + let json = br#"{ + "version": 5, + "size": 10, + "v2Checkpoint": { + "path": "00000000000000000005.checkpoint.0190e8f5-uuid.parquet", + "sidecarFiles": [ + {"path": "sidecar-1.parquet", "sizeInBytes": 42, "modificationTime": 1700000000000} + ], + "nonFileActions": [ + {"protocol": {"minReaderVersion": 3, "minWriterVersion": 7, + "readerFeatures": [], "writerFeatures": []}}, + {"metaData": {"id": "table-id", "format": {"provider": "parquet", "options": {}}, + "schemaString": "{\"type\":\"struct\",\"fields\":[]}", + "partitionColumns": [], "configuration": {}}}, + {"txn": {"appId": "app", "version": 1}}, + {"domainMetadata": {"domain": "d", "configuration": "c", "removed": false}}, + {"checkpointMetadata": {"version": 5}} + ] + } + }"#; + let hint: LastCheckpointHint = serde_json::from_slice(json).unwrap(); + let v2 = hint.v2_checkpoint.expect("v2Checkpoint present"); + + let sidecars = v2.sidecar_files.expect("sidecarFiles present"); + assert_eq!(sidecars.len(), 1); + assert_eq!(sidecars[0].path, "sidecar-1.parquet"); + assert_eq!(sidecars[0].size_in_bytes, 42); + + let actions = v2.non_file_actions.expect("nonFileActions present"); + assert_eq!(actions.len(), 5); + assert!(matches!(&actions[0], HintAction::Protocol(p) if p.min_reader_version() == 3)); + assert!(matches!(&actions[1], HintAction::Metadata(m) if m.id() == "table-id")); + assert!(matches!(&actions[2], HintAction::Txn(t) if t.app_id == "app")); + assert!(matches!(&actions[3], HintAction::DomainMetadata(_))); + assert!(matches!(&actions[4], HintAction::CheckpointMetadata(c) if c.version == 5)); + } + + /// A `_last_checkpoint` without a `v2Checkpoint` object (V1 / classic) parses to `None`. + #[test] + fn v2_checkpoint_absent_parses_to_none() { + let json = br#"{"version": 5, "size": 10}"#; + let hint: LastCheckpointHint = serde_json::from_slice(json).unwrap(); + assert!(hint.v2_checkpoint.is_none()); + } + + /// A malformed `v2Checkpoint` -- missing the required `path`, or a type-mismatched field -- + /// fails the whole-hint parse. `try_read` swallows that to `None`, so the reader falls back to + /// a footer read rather than trusting a partially-parsed hint. + #[test] + fn malformed_v2_checkpoint_fails_whole_hint_parse() { + let missing_path = br#"{"version": 5, "size": 10, "v2Checkpoint": {"sizeInBytes": 1234}}"#; + assert!(serde_json::from_slice::(missing_path).is_err()); + + let bad_type = br#"{"version": 5, "size": 10, + "v2Checkpoint": {"path": "c.parquet", "sizeInBytes": "not-a-number"}}"#; + assert!(serde_json::from_slice::(bad_type).is_err()); + } + + /// `applies_to` accepts the hint only for the checkpoint a segment actually selected: same + /// version, and the identity the hint's fields imply must equal the selected checkpoint's. + #[test] + fn applies_to_matches_only_the_selected_checkpoint() { + let root = Url::parse("memory:///_delta_log/").unwrap(); + let part = |name: &str| create_log_path(root.join(name).unwrap().as_str()); + + let selected = + "00000000000000000001.checkpoint.11111111-1111-1111-1111-111111111111.parquet"; + let other = "00000000000000000001.checkpoint.22222222-2222-2222-2222-222222222222.parquet"; + + // V2 single-part: applies only when the version and the file name both match. + let v2 = LastCheckpointHint { + version: 1, + v2_checkpoint: Some(LastCheckpointV2 { + path: selected.to_string(), + ..Default::default() + }), + ..Default::default() + }; + assert!(v2.applies_to(&[part(selected)])); + assert!(!v2.applies_to(&[part(other)]), "wrong v2 path"); + let v2_wrong_version = LastCheckpointHint { + version: 2, + ..v2.clone() + }; + assert!( + !v2_wrong_version.applies_to(&[part(selected)]), + "wrong version" + ); + + // V1 multi-part: the part count comes from the selected part's file name. + let mp1 = "00000000000000000001.checkpoint.0000000001.0000000002.parquet"; + let mp2 = "00000000000000000001.checkpoint.0000000002.0000000002.parquet"; + let v1_multi = LastCheckpointHint { + version: 1, + parts: Some(2), + ..Default::default() + }; + assert!(v1_multi.applies_to(&[part(mp1), part(mp2)])); + let mp1_of_3 = "00000000000000000001.checkpoint.0000000001.0000000003.parquet"; + assert!(!v1_multi.applies_to(&[part(mp1_of_3)]), "wrong part count"); + + // V1 single-part: an absent `parts` means one part. + let v1_single = LastCheckpointHint { + version: 1, + ..Default::default() + }; + let classic = "00000000000000000001.checkpoint.parquet"; + assert!(v1_single.applies_to(&[part(classic)])); + + // Naming has to match, not just part count: a 1-of-1 multi-part and a uuid-named checkpoint + // each hold one part too. + let one_of_one = "00000000000000000001.checkpoint.0000000001.0000000001.parquet"; + assert!( + !v1_single.applies_to(&[part(one_of_one)]), + "single-file hint applied to a multi-part checkpoint" + ); + assert!( + !v1_single.applies_to(&[part(selected)]), + "single-file hint applied to a uuid-named checkpoint" + ); + // Nor can a multi-part hint describe a single whole file. + let v1_one_part = LastCheckpointHint { + version: 1, + parts: Some(1), + ..Default::default() + }; + assert!(v1_one_part.applies_to(&[part(one_of_one)])); + assert!( + !v1_one_part.applies_to(&[part(classic)]), + "multi-part hint applied to a classic checkpoint" + ); + + assert!(!v1_single.applies_to(&[]), "no checkpoint selected"); + } + + /// `sidecarFiles` / `nonFileActions` are dropped only when their count exceeds the threshold -- + /// independently, and the whole field at once (never truncated); within-threshold fields are + /// kept verbatim. + #[test] + fn drops_oversized_embedded_fields() { + let sidecar = Sidecar { + path: "s.parquet".to_string(), + size_in_bytes: 1, + modification_time: 0, + tags: None, + }; + let action = HintAction::Protocol(Protocol::default()); + let hint = |sidecars: usize, actions: usize| LastCheckpointHint { + version: 1, + v2_checkpoint: Some(LastCheckpointV2 { + path: "cp".to_string(), + sidecar_files: Some(vec![sidecar.clone(); sidecars]), + non_file_actions: Some(vec![action.clone(); actions]), + ..Default::default() + }), + ..Default::default() + }; + + // At the threshold: both fields kept. + let v2 = hint( + LAST_CHECKPOINT_SIDECARS_THRESHOLD, + LAST_CHECKPOINT_NON_FILE_ACTIONS_THRESHOLD, + ) + .drop_oversized_fields() + .v2_checkpoint + .unwrap(); + assert_eq!( + v2.sidecar_files.unwrap().len(), + LAST_CHECKPOINT_SIDECARS_THRESHOLD + ); + assert_eq!( + v2.non_file_actions.unwrap().len(), + LAST_CHECKPOINT_NON_FILE_ACTIONS_THRESHOLD + ); + + // Over threshold: each field dropped independently. + let v2 = hint(LAST_CHECKPOINT_SIDECARS_THRESHOLD + 1, 5) + .drop_oversized_fields() + .v2_checkpoint + .unwrap(); + assert!(v2.sidecar_files.is_none(), "oversized sidecarFiles dropped"); + assert_eq!(v2.non_file_actions.unwrap().len(), 5, "nonFileActions kept"); + + let v2 = hint(5, LAST_CHECKPOINT_NON_FILE_ACTIONS_THRESHOLD + 1) + .drop_oversized_fields() + .v2_checkpoint + .unwrap(); + assert_eq!(v2.sidecar_files.unwrap().len(), 5, "sidecarFiles kept"); + assert!( + v2.non_file_actions.is_none(), + "oversized nonFileActions dropped" + ); + } + + /// Returns the single `actions` element matching `extract`, asserting there is exactly one. + fn one_action<'a, T: 'a>( + actions: &'a [HintAction], + extract: impl Fn(&'a HintAction) -> Option<&'a T>, + ) -> &'a T { + let mut matching = actions.iter().filter_map(extract); + let found = matching.next().expect("expected a matching action"); + assert!( + matching.next().is_none(), + "expected exactly one matching action" + ); + found + } + + /// The `v2Checkpoint` hint parses from V2 checkpoint tables to its exact contents. Pins, per + /// table, the checkpoint version, file path, sidecar paths, metadata id, created time, and + /// configuration; and -- shared across these fixtures -- a `(3, 7)` protocol with + /// V2Checkpoint/AppendOnly/ Invariants features and an unpartitioned parquet `id: long` + /// schema. The non-file actions are exactly protocol + metadata + checkpointMetadata (no + /// txn, no domainMetadata). Also checks the identity gate exposes a matched hint's sidecars + /// but suppresses a mismatched one (`v2-classic-checkpoint-parquet`, whose hint names a + /// UUID checkpoint while the segment selects the classic-named one). + /// A table's expected `_last_checkpoint` identity and metadata for the case below. + struct ExpectedHint { + table: &'static str, + version: u64, + path: &'static str, + sidecars: &'static [&'static str], + metadata_id: &'static str, + created_time: i64, + config: &'static [(&'static str, &'static str)], + } + + #[rstest] + #[case::parquet_sidecars(ExpectedHint { + table: "v2-checkpoints-parquet-with-sidecars", + version: 6, + path: "00000000000000000006.checkpoint.f15b9025-707a-4c73-aac0-31dfcbd29aa6.parquet", + sidecars: &[ + "00000000000000000006.checkpoint.0000000001.0000000002.76931b15-ead3-480d-b86c-afe55a577fc3.parquet", + "00000000000000000006.checkpoint.0000000002.0000000002.4367b29c-0e87-447f-8e81-9814cc01ad1f.parquet", + ], + metadata_id: "5a5afdfe-7d40-4109-bb92-29b051257e4c", + created_time: 1739329708855, + config: &[("delta.checkpointInterval", "1"), ("delta.checkpointPolicy", "v2")], + })] + #[case::json_sidecars(ExpectedHint { + table: "v2-checkpoints-json-with-sidecars", + version: 6, + path: "00000000000000000006.checkpoint.2a15d0c6-8b11-4a98-bab4-957905d62f7f.json", + sidecars: &[ + "00000000000000000006.checkpoint.0000000001.0000000002.19af1366-a425-47f4-8fa6-8d6865625573.parquet", + "00000000000000000006.checkpoint.0000000002.0000000002.5008b69f-aa8a-4a66-9299-0733a56a7e63.parquet", + ], + metadata_id: "f571bf08-452e-4155-9f52-f793e630c55c", + created_time: 1739329697356, + config: &[("delta.checkpointInterval", "1"), ("delta.checkpointPolicy", "v2")], + })] + #[case::parquet_last_checkpoint(ExpectedHint { + table: "v2-checkpoints-parquet-with-last-checkpoint", + version: 0, + path: "00000000000000000000.checkpoint.8516aa94-7099-4e71-92a0-d6d7e7bb3b2c.parquet", + sidecars: &["00000000000000000000.checkpoint.0000000001.0000000001.c561300b-ad5f-49d4-a28d-9b3f4bb0331c.parquet"], + metadata_id: "d3b78022-27fc-470d-a4ea-1b8b47fcc143", + created_time: 1739329764309, + config: &[("delta.checkpointPolicy", "v2")], + })] + #[case::json_last_checkpoint(ExpectedHint { + table: "v2-checkpoints-json-with-last-checkpoint", + version: 0, + path: "00000000000000000000.checkpoint.0e42c15b-17cc-4918-990d-2ff76e918e4d.json", + sidecars: &["00000000000000000000.checkpoint.0000000001.0000000001.9167a758-dd93-4e52-8636-7cf5776eb10f.parquet"], + metadata_id: "f03ce383-0d09-4e1c-9446-8d80e1a59daa", + created_time: 1739329763101, + config: &[("delta.checkpointPolicy", "v2")], + })] + #[case::classic_parquet(ExpectedHint { + table: "v2-classic-checkpoint-parquet", + version: 1, + path: "00000000000000000001.checkpoint.bfe7499d-715e-4d64-82a4-e6cdd2fc37af.parquet", + sidecars: &["00000000000000000001.checkpoint.0000000001.0000000001.e2eb56f9-1c54-4a82-b122-de108e317c20.parquet"], + metadata_id: "541a194a-df83-4f46-9adf-032a1275e82b", + created_time: 1739329759409, + config: &[("delta.checkpointPolicy", "v2")], + })] + #[case::classic_json(ExpectedHint { + table: "v2-classic-checkpoint-json", + version: 1, + path: "00000000000000000001.checkpoint.6c750e24-bbc4-4618-8feb-7cd7d5b9e084.json", + sidecars: &["00000000000000000001.checkpoint.0000000001.0000000001.c1bacf45-f3a9-4846-bd44-87cdacd4620f.parquet"], + metadata_id: "29ef2045-59c5-4cf7-9d5d-2ba47e971d32", + created_time: 1739313200623, + config: &[("delta.checkpointPolicy", "v2")], + })] + fn v2_last_checkpoint_hint_contents(#[case] expected: ExpectedHint) -> DeltaResult<()> { + use crate::unit_test_utils::load_test_table; + + let ExpectedHint { + table, + version: expected_version, + path: expected_path, + sidecars: expected_sidecars, + metadata_id: expected_metadata_id, + created_time: expected_created_time, + config: expected_config, + } = expected; + + let (engine, snapshot, _tempdir) = load_test_table(table)?; + let seg = snapshot.log_segment(); + let hint = LastCheckpointHint::try_read(engine.storage_handler().as_ref(), &seg.log_root)? + .expect("table has a _last_checkpoint"); + let v2 = hint.v2_checkpoint.as_ref().expect("V2 checkpoint hint"); + + // Version, checkpoint file path, and sidecar paths are this table's exact identity. + assert_eq!(hint.version, expected_version, "{table}: version"); + assert_eq!(v2.path, expected_path, "{table}: v2 path"); + let sidecar_paths: Vec<&str> = v2 + .sidecar_files + .as_ref() + .expect("sidecarFiles present") + .iter() + .map(|s| s.path.as_str()) + .collect(); + assert_eq!( + sidecar_paths.as_slice(), + expected_sidecars, + "{table}: sidecar paths" + ); + + // The non-file actions are exactly one protocol, one metadata, and one checkpointMetadata + // -- no txn or domainMetadata. + let actions = v2 + .non_file_actions + .as_ref() + .expect("nonFileActions present"); + assert!( + !actions.iter().any(|a| matches!(a, HintAction::Txn(_))), + "{table}: no txn" + ); + assert!( + !actions + .iter() + .any(|a| matches!(a, HintAction::DomainMetadata(_))), + "{table}: no domain metadata" + ); + + // A (3, 7) protocol gated on the V2Checkpoint reader feature, with V2Checkpoint/AppendOnly/ + // Invariants on the writer side. + let protocol = one_action(actions, |a| match a { + HintAction::Protocol(p) => Some(p), + _ => None, + }); + assert_eq!( + (protocol.min_reader_version(), protocol.min_writer_version()), + (3, 7), + "{table}: protocol version" + ); + assert_eq!( + protocol.reader_features(), + Some([TableFeature::V2Checkpoint].as_slice()), + "{table}: reader features" + ); + assert_eq!( + protocol.writer_features(), + Some( + [ + TableFeature::V2Checkpoint, + TableFeature::AppendOnly, + TableFeature::Invariants + ] + .as_slice() + ), + "{table}: writer features" + ); + + // Metadata for an unnamed, unpartitioned parquet table with a single `id: long` column. + let metadata = one_action(actions, |a| match a { + HintAction::Metadata(m) => Some(m), + _ => None, + }); + assert_eq!(metadata.id(), expected_metadata_id, "{table}: metadata id"); + assert_eq!(metadata.name(), None, "{table}: metadata name"); + assert_eq!( + metadata.description(), + None, + "{table}: metadata description" + ); + assert_eq!(metadata.format_provider(), "parquet", "{table}: format"); + assert_eq!( + metadata.parse_schema()?, + StructType::new_unchecked([StructField::nullable("id", DataType::LONG)]), + "{table}: metadata schema" + ); + assert!( + metadata.partition_columns().is_empty(), + "{table}: unpartitioned" + ); + assert_eq!( + metadata.created_time(), + Some(expected_created_time), + "{table}: created time" + ); + let expected_config: HashMap = expected_config + .iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect(); + assert_eq!( + metadata.configuration(), + &expected_config, + "{table}: configuration" + ); + + // checkpointMetadata records this checkpoint's version. + let checkpoint_metadata = one_action(actions, |a| match a { + HintAction::CheckpointMetadata(c) => Some(c), + _ => None, + }); + assert_eq!( + checkpoint_metadata.version as u64, expected_version, + "{table}: checkpointMetadata.version" + ); + assert_eq!( + checkpoint_metadata.tags, None, + "{table}: checkpointMetadata.tags" + ); + + // Identity gate: a hint naming the selected checkpoint exposes its sidecars through the + // accessor; one naming a different same-version checkpoint is fully suppressed. + let selected = &seg + .listed + .checkpoint_parts + .first() + .expect("checkpoint present") + .filename; + if &v2.path == selected { + assert_eq!( + seg.checkpoint_hint_sidecars(), + v2.sidecar_files.as_ref(), + "{table}: matched hint exposes its sidecars" + ); + } else { + assert!( + seg.checkpoint_hint_schema().is_none() && seg.checkpoint_hint_sidecars().is_none(), + "{table}: mismatched hint ({}) must be suppressed", + v2.path + ); + } + Ok(()) + } +} diff --git a/kernel/src/lib.rs b/kernel/src/lib.rs index eb70aee74f..6d7777b681 100644 --- a/kernel/src/lib.rs +++ b/kernel/src/lib.rs @@ -74,10 +74,11 @@ extern crate self as delta_kernel; use std::any::Any; +use std::cmp::Ordering; use std::fs::DirEntry; +use std::ops::Range; use std::sync::Arc; use std::time::SystemTime; -use std::{cmp::Ordering, ops::Range}; use bytes::Bytes; use url::Url; @@ -86,46 +87,68 @@ use self::schema::{DataType, SchemaRef}; mod action_reconciliation; pub mod actions; +pub mod cancellation; pub mod checkpoint; +pub mod commit_range; pub mod committer; -// Public under test-utils so integration tests can inspect CRC state via Snapshot::get_current_crc_if_loaded_for_testing. -#[cfg(feature = "test-utils")] +#[cfg(feature = "adaptive-metadata-in-dev")] +mod content_tree; +#[cfg(feature = "internal-api")] pub mod crc; -#[cfg(not(feature = "test-utils"))] +#[cfg(not(feature = "internal-api"))] pub(crate) mod crc; pub mod engine_data; pub mod error; pub mod expressions; +pub mod incremental_scan; mod log_compaction; mod log_path; mod log_reader; pub mod metrics; +pub mod partition; +#[cfg(feature = "declarative-plans")] +pub mod plans; pub mod scan; pub mod schema; pub mod snapshot; +pub mod struct_patch; pub mod table_changes; pub mod table_configuration; pub mod table_features; pub mod table_properties; +mod timestamp_timezone; pub mod transaction; pub mod transforms; +pub use crc::{FileSizeHistogram, FileStats}; pub use log_path::LogPath; -mod row_tracking; +// Public under test-utils so integration tests can call get_high_water_mark via snapshot. +#[cfg(feature = "test-utils")] +pub mod row_tracking; +#[cfg(not(feature = "test-utils"))] +pub(crate) mod row_tracking; pub(crate) mod clustering; mod arrow_compat; -#[cfg(any(feature = "arrow-56", feature = "arrow-57"))] +#[cfg(any(feature = "arrow-58", feature = "arrow-59"))] pub use arrow_compat::*; +#[cfg(feature = "internal-api")] +pub mod column_trie; +#[cfg(not(feature = "internal-api"))] pub(crate) mod column_trie; pub mod kernel_predicates; +#[cfg(test)] +pub(crate) mod unit_test_utils; +#[cfg(feature = "internal-api")] +pub mod utils; +#[cfg(not(feature = "internal-api"))] pub(crate) mod utils; #[cfg(feature = "internal-api")] -pub use utils::try_parse_uri; +pub use utils::{try_parse_uri, CollectInto}; // for the below modules, we cannot introduce a macro to clean this up. rustfmt doesn't follow into // macros, and so will not format the files associated with these modules if we get too clever. see: @@ -151,12 +174,12 @@ pub mod last_checkpoint_hint; #[cfg(not(feature = "internal-api"))] pub(crate) mod last_checkpoint_hint; +#[cfg(feature = "internal-api")] +pub mod log_segment_files; +#[cfg(not(feature = "internal-api"))] pub(crate) mod log_segment_files; -#[cfg(feature = "internal-api")] pub mod history_manager; -#[cfg(not(feature = "internal-api"))] -pub(crate) mod history_manager; #[cfg(feature = "internal-api")] pub mod parallel; @@ -164,33 +187,38 @@ pub mod parallel; pub(crate) mod parallel; pub use action_reconciliation::{ActionReconciliationIterator, ActionReconciliationIteratorState}; +use cancellation::check_cancelled; +pub use cancellation::{CancellationToken, CancellationTokenRef, CancelledFuture}; pub use delta_kernel_derive; use delta_kernel_derive::internal_api; -pub use engine_data::{EngineData, FilteredEngineData, RowVisitor}; -pub use error::{DeltaResult, Error}; +pub use engine_data::{ + EngineData, FilteredEngineData, FilteredRowVisitor, GetData, RowIndexIterator, RowVisitor, +}; +pub use error::{DeltaResult, DeltaResultIterator, DeltaResultIteratorStatic, Error}; +use expressions::{literal_expression_transform, Scalar}; pub use expressions::{Expression, ExpressionRef, Predicate, PredicateRef}; pub use log_compaction::{should_compact, LogCompactionWriter}; -pub use metrics::MetricsReporter; -pub use snapshot::Snapshot; -pub use snapshot::SnapshotRef; - -use expressions::literal_expression_transform; -use expressions::Scalar; +#[cfg(feature = "declarative-plans")] +pub use plans::{IoOperation, Operation, PlanBuilder, PlanExecutor, PlanResult}; use schema::{StructField, StructType}; +pub use snapshot::{Snapshot, SnapshotRef}; #[cfg(any( - feature = "default-engine-native-tls", - feature = "default-engine-rustls", - feature = "arrow-conversion" + feature = "default-engine-base", + feature = "arrow-conversion", + feature = "declarative-plans" ))] pub mod engine; /// Delta table version is 8 byte unsigned int pub type Version = u64; -/// Sentinel version indicating a pre-commit state (table does not exist yet). -/// Used for create-table transactions before the first commit. -pub const PRE_COMMIT_VERSION: Version = u64::MAX; +/// Converts a [`Version`] to `i64`, returning an error if the version exceeds `i64::MAX`. +pub(crate) fn version_as_i64(version: Version) -> DeltaResult { + version + .try_into() + .map_err(|_| Error::generic(format!("Delta log version {version} exceeds i64::MAX"))) +} pub type FileSize = u64; pub type FileIndex = u64; @@ -202,8 +230,7 @@ pub type FileSlice = (Url, Option>); pub type FileDataReadResult = (FileMeta, Box); /// An iterator of data read from specified files -pub type FileDataReadResultIterator = - Box>> + Send>; +pub type FileDataReadResultIterator = DeltaResultIteratorStatic>; /// The metadata that describes an object. #[derive(Debug, Clone, PartialEq, Eq)] @@ -262,6 +289,12 @@ impl FileMeta { size, } } + + /// Casts `size` to `i64`. Errors if `size` exceeds `i64::MAX`. + pub(crate) fn size_as_i64(&self) -> DeltaResult { + i64::try_from(self.size) + .map_err(|_| Error::generic(format!("file size {} exceeds i64::MAX", self.size))) + } } /// Extension trait that makes it easier to work with traits objects that implement [`Any`], @@ -405,7 +438,8 @@ pub trait ExpressionEvaluator: AsAny { /// Evaluate the expression on a given EngineData. /// /// Produces one value for each row of the input. - /// The data type of the output is same as the type output of the expression this evaluator is using. + /// The data type of the output is same as the type output of the expression this evaluator is + /// using. fn evaluate(&self, batch: &dyn EngineData) -> DeltaResult>; } @@ -472,6 +506,37 @@ pub trait EvaluationHandler: AsAny { // NOTE: we should probably allow DataType instead of SchemaRef, but can expand that in the // future. fn null_row(&self, output_schema: SchemaRef) -> DeltaResult>; + + /// Create a multi-row [`EngineData`] by applying the given schema to multiple rows of values. + /// + /// Each element in `rows` represents one row of data, where each row is a slice of structured + /// scalar values (one scalar per top-level field in the schema). + /// + /// # Parameters + /// + /// - `schema`: Schema describing the structure of each row. + /// - `rows`: Slice of rows, where each row contains one structured scalar per top-level schema + /// field. + /// + /// # Returns + /// + /// A multi-row `EngineData` containing all rows. + /// + /// # Errors + /// + /// Returns an error if any row has a number of scalars that does not match the number of + /// top-level fields in `schema`, or if any scalar value cannot be appended to its corresponding + /// field's builder (e.g. due to a type mismatch). + /// + /// # Example + /// + /// For a schema with fields `[add: Struct, remove: Struct]`, each row should contain exactly 2 + /// scalars: one for the `add` field and one for the `remove` field. + fn create_many( + &self, + schema: SchemaRef, + rows: &[&[Scalar]], + ) -> DeltaResult>; } /// Internal trait to allow us to have a private `create_one` API that's implemented for all @@ -544,11 +609,20 @@ pub(crate) trait IntoEngineData { /// file system where the Delta table is present. Connector implementation of /// this trait can hide filesystem specific details from Delta Kernel. pub trait StorageHandler: AsAny { - /// List the paths in the same directory that are lexicographically greater than - /// (UTF-8 sorting) the given `path`. The result should also be sorted by the file name. - /// - /// If the path is directory-like (ends with '/'), the result should contain - /// all the files in the directory. + /// Recursively list files whose full path is lexicographically greater than (UTF-8 sorting) + /// the given `path`, restricted to descendants of `path`'s parent directory. The result must + /// be sorted by the full path (UTF-8 byte order). + /// + /// The listing is **recursive**: files in nested subdirectories are included, not just files + /// directly under the parent. For example, listing from `dir/0001.json` may return + /// `dir/0002.json`, `dir/sub/0003.json`, and `dir/sub/nested/0004.json`, all interleaved + /// in lexicographic order. + /// + /// The parent directory is derived from `path`: + /// - If `path` is directory-like (ends with `/`), the parent is `path` itself and the result + /// contains all files at or below that directory. + /// - Otherwise, the parent is the directory containing `path`, and only files (at any depth + /// under that parent) whose full path sorts strictly greater than `path` are returned. fn list_from(&self, path: &Url) -> DeltaResult>>>; @@ -572,29 +646,36 @@ pub trait StorageHandler: AsAny { /// /// If the file does not exist, this must return an `Err` with [`Error::FileNotFound`]. fn head(&self, path: &Url) -> DeltaResult; + + /// Delete the file at the given path. + /// + /// This operation is idempotent: deleting a path that does not exist should return `Ok(())`. + /// For any other error, this must propagate the corresponding error. + fn delete(&self, path: &Url) -> DeltaResult<()>; } /// Provides JSON handling functionality to Delta Kernel. /// -/// Delta Kernel can use this handler to parse JSON strings into Row or read content from JSON files. -/// Connectors can leverage this trait to provide their best implementation of the JSON parsing -/// capability to Delta Kernel. +/// Delta Kernel can use this handler to parse JSON strings into Row or read content from JSON +/// files. Connectors can leverage this trait to provide their best implementation of the JSON +/// parsing capability to Delta Kernel. pub trait JsonHandler: AsAny { - /// Parse the given json strings and return the fields requested by output schema as columns in [`EngineData`]. - /// json_strings MUST be a single column batch of engine data, and the column type must be string + /// Parse the given json strings and return the fields requested by output schema as columns in + /// [`EngineData`]. json_strings MUST be a single column batch of engine data, and the + /// column type must be string fn parse_json( &self, json_strings: Box, output_schema: SchemaRef, ) -> DeltaResult>; - /// Read and parse the JSON format file at given locations and return the data as EngineData with - /// the columns requested by physical schema. Note: The [`FileDataReadResultIterator`] must emit - /// data from files in the order that `files` is given. For example if files ["a", "b"] is provided, - /// then the engine data iterator must first return all the engine data from file "a", _then_ all - /// the engine data from file "b". Moreover, for a given file, all of its [`EngineData`] and - /// constituent rows must be in order that they occur in the file. Consider a file with rows - /// (1, 2, 3). The following are legal iterator batches: + /// Read and parse the JSON format file at given locations and return the data as EngineData + /// with the columns requested by physical schema. Note: The [`FileDataReadResultIterator`] + /// must emit data from files in the order that `files` is given. For example if files ["a", + /// "b"] is provided, then the engine data iterator must first return all the engine data + /// from file "a", _then_ all the engine data from file "b". Moreover, for a given file, all + /// of its [`EngineData`] and constituent rows must be in order that they occur in the file. + /// Consider a file with rows (1, 2, 3). The following are legal iterator batches: /// iter: [EngineData(1, 2), EngineData(3)] /// iter: [EngineData(1), EngineData(2, 3)] /// iter: [EngineData(1, 2, 3)] @@ -609,7 +690,12 @@ pub trait JsonHandler: AsAny { /// /// - `files` - File metadata for files to be read. /// - `physical_schema` - Select list of columns to read from the JSON file. - /// - `predicate` - Optional push-down predicate hint (engine is free to ignore it). + /// - `predicate` - Optional conservative push-down predicate. Implementations may ignore it. If + /// applied, a file or row may be omitted only when the predicate cannot evaluate to true for + /// any row in that unit. CAST conversion, failure, and NULL semantics must match final + /// filtering over exact row values. Unsupported or missing references remain unknown without + /// failing the read, including references synthesized as NULL columns during schema + /// reconciliation. Returned data is not guaranteed to satisfy the predicate. fn read_json_files( &self, files: &[FileMeta], @@ -617,9 +703,38 @@ pub trait JsonHandler: AsAny { predicate: Option, ) -> DeltaResult; - /// Atomically (!) write a single JSON file. Each row of the input data should be written as a - /// new JSON object appended to the file. this write must: - /// (1) serialize the data to newline-delimited json (each row is a json object literal) + /// Cancellation-aware variant of [`read_json_files`](Self::read_json_files). + /// + /// When `cancellation_token` is `Some`, an engine may race its I/O against the token and + /// terminate the returned iterator with [`Error::Cancelled`] once cancellation is observed, + /// rather than reading every file to completion. + /// + /// The default implementation returns [`Error::Cancelled`] if the token is already cancelled + /// and otherwise delegates to [`read_json_files`](Self::read_json_files), ignoring the token + /// for the rest of the read. So an engine that does not override this stays source-compatible + /// while still honoring an up-front cancellation; kernel additionally polls the token at + /// action-batch boundaries. An engine that overrides this may assume kernel has already + /// performed the pre-read check, and should focus on interrupting its in-flight I/O. + /// + /// [`Error::Cancelled`]: crate::Error::Cancelled + fn read_json_files_with_cancellation( + &self, + files: &[FileMeta], + physical_schema: SchemaRef, + predicate: Option, + cancellation_token: Option, + ) -> DeltaResult { + check_cancelled(cancellation_token.as_ref())?; + self.read_json_files(files, physical_schema, predicate) + } + + /// Atomically (!) write a single JSON file. Each selected row of the input data must be + /// written as a new JSON object appended to the file; rows not selected by a batch's + /// selection vector (see [`FilteredEngineData`]) must not be written. + /// [`FilteredEngineData::apply_selection_vector`] produces the selected-rows view for + /// implementations that do not filter during serialization. This write must: + /// (1) serialize the selected rows to newline-delimited json (each row is a json object + /// literal) /// (2) write the data to storage atomically (i.e. if the file already exists, fail unless the /// overwrite flag is set) /// @@ -634,15 +749,13 @@ pub trait JsonHandler: AsAny { /// # Parameters /// /// - `path` - URL specifying the location to write the JSON file - /// - `data` - Iterator of EngineData to write to the JSON file. Each row should be written as - /// a new JSON object appended to the file. (that is, the file is newline-delimited JSON, and - /// each row is a JSON object on a single line) + /// - `data` - Iterator of [`FilteredEngineData`] to write to the JSON file /// - `overwrite` - If true, overwrite the file if it exists. If false, the call must fail if /// the file exists. fn write_json_file( &self, path: &Url, - data: Box> + Send + '_>, + data: DeltaResultIterator<'_, FilteredEngineData>, overwrite: bool, ) -> DeltaResult<()>; } @@ -684,16 +797,27 @@ pub trait ParquetHandler: AsAny { /// must be in the same order as specified in `physical_schema`. /// /// Parquet columns are matched to `physical_schema` [`StructField`]s using the following rules: - /// 1. **Field ID**: If a [`StructField`] in `physical_schema` contains a field ID - /// (specified in [`ColumnMetadataKey::ParquetFieldId`] metadata), use the ID to - /// match the Parquet column's field id - /// 2. **Field Name**: If no field ID is present in the `physical_schema`'s [`StructField`] or no matching parquet field ID is found, - /// fall back to matching by column name + /// 1. **Field ID**: If a [`StructField`] in `physical_schema` contains a field ID (specified in + /// [`ColumnMetadataKey::ParquetFieldId`] metadata), use the ID to match the Parquet column's + /// field id + /// 2. **Field Name**: If no field ID is present in the `physical_schema`'s [`StructField`] or + /// no matching parquet field ID is found, fall back to matching by column name + /// + /// # Type coercion + /// + /// A matched Parquet column whose physical type differs from the `physical_schema` + /// [`StructField`] must be coerced to the requested type. In particular, timestamp columns MUST + /// be normalized to the protocol specified microsecond precision: a `TIMESTAMP(MILLIS)` (or + /// any other non-microsecond unit) column read into a `TIMESTAMP` / `TIMESTAMP_NTZ` field + /// must be rescaled to microseconds (a finer unit such as nanosecond is truncated). The + /// default engine does this via `arrow::compute::cast` while reordering columns to the + /// requested schema. /// /// # Metadata Columns /// /// The ParquetHandler must support virtual metadata columns that provide additional information - /// about each row. These columns are not stored in the Parquet file but are generated at read time. + /// about each row. These columns are not stored in the Parquet file but are generated at read + /// time. /// /// ## Row Index Column /// @@ -704,7 +828,8 @@ pub trait ParquetHandler: AsAny { /// - **Column name**: User-specified (commonly `"row_index"` or `"_metadata.row_index"`) /// - **Type**: `LONG` (non-nullable) /// - **Values**: Sequential integers starting at 0 for each file - /// - **Use case**: Track row positions for downstream processing, or internally used to compute Row IDs + /// - **Use case**: Track row positions for downstream processing, or internally used to compute + /// Row IDs /// /// Example: A file with 5 rows would have row_index values `[0, 1, 2, 3, 4]`. /// @@ -751,7 +876,8 @@ pub trait ParquetHandler: AsAny { /// ## Column Matching Examples /// /// Consider a `physical_schema` with the following fields: - /// - Column 0: `"i_logical"` (integer, non-null) with field ID 1 (via [`ColumnMetadataKey::ParquetFieldId`]) + /// - Column 0: `"i_logical"` (integer, non-null) with field ID 1 (via + /// [`ColumnMetadataKey::ParquetFieldId`]) /// - Column 1: `"s"` (string, nullable) with no field ID metadata /// - Column 2: `"i2"` (integer, nullable) with no field ID metadata /// @@ -774,12 +900,22 @@ pub trait ParquetHandler: AsAny { /// /// - `files` - File metadata for files to be read. /// - `physical_schema` - Select list and order of columns to read from the Parquet file. - /// - `predicate` - Optional push-down predicate hint (engine is free to ignore it). + /// - `predicate` - Optional conservative push-down predicate. Implementations may ignore it. A + /// file, row group, or row may be omitted only when the predicate cannot evaluate to true for + /// any row in that unit. CAST and NULL semantics must agree with the eventual query filter. + /// An evaluator using exact row values may apply the CAST directly. Footer min/max may be + /// cast only when the cast preserves those bounds; otherwise footer evaluation of the CAST + /// must remain unknown and must not fail the read. Other unsupported subexpressions and + /// references with no matching physical or generated value must also remain unknown without + /// failing the read. A missing reference remains unknown for pruning even if schema + /// reconciliation synthesizes a NULL output column. Returned data is not guaranteed to + /// satisfy the predicate. /// /// # Returns /// A [`DeltaResult`] containing a [`FileDataReadResultIterator`]. /// Each element of the iterator is a [`DeltaResult`] of [`EngineData`]. The [`EngineData`] - /// has the contents of `files` and must match the provided `physical_schema`. + /// contains rows from `files` after any predicate push-down and must match the provided + /// `physical_schema`. /// /// Note: The [`FileDataReadResultIterator`] must emit data from files in the order that `files` /// is given. For example if files ["a", "b"] is provided, then the engine data iterator must @@ -805,25 +941,72 @@ pub trait ParquetHandler: AsAny { predicate: Option, ) -> DeltaResult; + /// Cancellation-aware variant of [`read_parquet_files`](Self::read_parquet_files). + /// + /// When `cancellation_token` is `Some`, an engine may race its I/O against the token and + /// terminate the returned iterator with [`Error::Cancelled`] once cancellation is observed, + /// rather than reading every file to completion. + /// + /// The default implementation returns [`Error::Cancelled`] if the token is already cancelled + /// and otherwise delegates to [`read_parquet_files`](Self::read_parquet_files), ignoring the + /// token for the rest of the read. So an engine that does not override this stays + /// source-compatible while still honoring an up-front cancellation; kernel additionally polls + /// the token at action-batch boundaries. An engine that overrides this may assume kernel has + /// already performed the pre-read check, and should focus on interrupting its in-flight I/O. + /// + /// [`Error::Cancelled`]: crate::Error::Cancelled + fn read_parquet_files_with_cancellation( + &self, + files: &[FileMeta], + physical_schema: SchemaRef, + predicate: Option, + cancellation_token: Option, + ) -> DeltaResult { + check_cancelled(cancellation_token.as_ref())?; + self.read_parquet_files(files, physical_schema, predicate) + } + /// Write data to a Parquet file at the specified URL. /// /// This method writes the provided `data` to a Parquet file at the given `url`. /// - /// This will overwrite the file if it already exists. + /// This will overwrite the file if it already exists. For filesystem-backed + /// implementations, the parent directories must be created if they do not exist. + /// + /// # Parquet field IDs + /// + /// The engine must write a Parquet `field_id` correctly when the kernel + /// [`StructField`] carries a field-id related annotation, including: + /// - [`ColumnMetadataKey::ColumnMappingId`] / [`ColumnMetadataKey::ParquetFieldId`] + /// - [`ColumnMetadataKey::ColumnMappingNestedIds`] + /// + /// For how to use these keys, refer to the Delta protocol's [Column Mapping] and + /// [IcebergCompatV2] sections. + /// + /// **Non-compliance produces files with incorrect `field_id`s**, which may lead to + /// read failures when column mapping mode is `id` and to failures when converting + /// the table to Iceberg. /// /// # Parameters /// - /// - `url` - The full URL path where the Parquet file should be written - /// (e.g., `s3://bucket/path/file.parquet`). + /// - `url` - The full URL path where the Parquet file should be written (e.g., + /// `s3://bucket/path/file.parquet`). /// - `data` - An iterator of engine data to be written to the Parquet file. /// /// # Returns /// /// A [`DeltaResult`] indicating success or failure. + /// + /// [`StructField`]: crate::schema::StructField + /// [`ColumnMetadataKey::ColumnMappingId`]: crate::schema::ColumnMetadataKey::ColumnMappingId + /// [`ColumnMetadataKey::ParquetFieldId`]: crate::schema::ColumnMetadataKey::ParquetFieldId + /// [`ColumnMetadataKey::ColumnMappingNestedIds`]: crate::schema::ColumnMetadataKey::ColumnMappingNestedIds + /// [Column Mapping]: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#column-mapping + /// [IcebergCompatV2]: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#iceberg-compatibility-v2 fn write_parquet_file( &self, location: url::Url, - data: Box>> + Send>, + data: DeltaResultIteratorStatic>, ) -> DeltaResult<()>; /// Read the footer metadata from a Parquet file without reading the data. @@ -861,6 +1044,28 @@ pub trait ParquetHandler: AsAny { /// [`StructField::get_config_value`]: crate::schema::StructField::get_config_value /// [`ColumnMetadataKey::ParquetFieldId`]: crate::schema::ColumnMetadataKey::ParquetFieldId fn read_parquet_footer(&self, file: &FileMeta) -> DeltaResult; + + /// Cancellation-aware variant of [`read_parquet_footer`](Self::read_parquet_footer). + /// + /// When `cancellation_token` is `Some`, an engine may race the footer read against the token + /// and return [`Error::Cancelled`] once cancellation is observed. + /// + /// The default implementation returns [`Error::Cancelled`] if the token is already cancelled + /// and otherwise delegates to [`read_parquet_footer`](Self::read_parquet_footer), ignoring the + /// token for the rest of the read. So an engine that does not override this stays + /// source-compatible while still honoring an up-front cancellation. An engine that overrides + /// this may assume kernel has already performed the pre-read check, and should focus on + /// interrupting its in-flight I/O. + /// + /// [`Error::Cancelled`]: crate::Error::Cancelled + fn read_parquet_footer_with_cancellation( + &self, + file: &FileMeta, + cancellation_token: Option, + ) -> DeltaResult { + check_cancelled(cancellation_token.as_ref())?; + self.read_parquet_footer(file) + } } /// The `Engine` trait encapsulates all the functionality an engine or connector needs to provide @@ -881,34 +1086,58 @@ pub trait Engine: AsAny { /// Get the connector provided [`ParquetHandler`]. fn parquet_handler(&self) -> Arc; - /// Get the connector provided [`MetricsReporter`] for metrics collection. + /// Get the connector provided [`PlanExecutor`], or `None` if this engine provides none. /// - /// Returns an optional reporter that will receive metric events from Delta operations. - /// The default implementation returns None (no metrics reporting). - fn get_metrics_reporter(&self) -> Option> { + /// The default implementation returns `None`. A connector opts into plan-based execution by + /// overriding this to return its executor. + #[cfg(feature = "declarative-plans")] + fn plan_executor(&self) -> Option> { None } -} -// we have an 'internal' feature flag: default-engine-base, which is actually just the shared -// pieces of default-engine-native-tls and default-engine-rustls. the crate can't compile with _only_ -// default-engine-base, so we give a friendly error here. -#[cfg(all( - feature = "default-engine-base", - not(any( - feature = "default-engine-native-tls", - feature = "default-engine-rustls", - )) -))] -compile_error!( - "The default-engine-base feature flag is not meant to be used directly. \ - Please use either default-engine-native-tls or default-engine-rustls." -); + /// Get the connector provided [`PlanExecutor`], erroring if this engine provides none. + /// + /// # Errors + /// + /// Returns [`Error::Unsupported`] when [`plan_executor`](Self::plan_executor) is `None`. + #[cfg(feature = "declarative-plans")] + fn require_plan_executor(&self) -> DeltaResult> { + self.plan_executor() + .ok_or_else(|| Error::unsupported("this engine does not provide a PlanExecutor")) + } +} // Rustdoc's documentation tests can do some things that regular unit tests can't. Here we are // using doctests to test macros. Specifically, we are testing for failed macro invocations due -// to invalid input, not the macro output when the macro invocation is successful (which can/should be -// done in unit tests). This module is not exclusively for macro tests only so other doctests can also be added. -// https://doc.rust-lang.org/rustdoc/write-documentation/documentation-tests.html#include-items-only-when-collecting-doctests +// to invalid input, not the macro output when the macro invocation is successful (which can/should +// be done in unit tests). This module is not exclusively for macro tests only so other doctests can +// also be added. https://doc.rust-lang.org/rustdoc/write-documentation/documentation-tests.html#include-items-only-when-collecting-doctests #[cfg(doctest)] mod doctests; + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + + #[rstest] + #[case::zero(0, Some(0))] + #[case::one(1, Some(1))] + #[case::i64_max(i64::MAX as u64, Some(i64::MAX))] + #[case::just_over_i64_max(i64::MAX as u64 + 1, None)] + #[case::u64_max(u64::MAX, None)] + /// Tests `FileMeta::size_as_i64` for both success (size fits in i64) and error (size exceeds + /// i64::MAX) paths. + fn test_file_meta_size_as_i64(#[case] size: u64, #[case] expected: Option) { + let meta = FileMeta::new(Url::parse("file:///x").unwrap(), 0, size); + match expected { + Some(v) => assert_eq!(meta.size_as_i64().unwrap(), v), + None => assert!(meta + .size_as_i64() + .unwrap_err() + .to_string() + .contains("exceeds i64::MAX")), + } + } +} diff --git a/kernel/src/log_compaction/mod.rs b/kernel/src/log_compaction/mod.rs index c6f6064e05..810767e0cc 100644 --- a/kernel/src/log_compaction/mod.rs +++ b/kernel/src/log_compaction/mod.rs @@ -1,5 +1,8 @@ //! # Log Compaction //! +//! **NOTE:** Log compaction is currently disabled on both reads and writes due to +//! insufficient integration test coverage. See issue #2337 for re-enablement tracking. +//! //! This module provides an API for writing log compaction files that aggregate //! multiple commit JSON files into single compacted files. This improves performance //! by reducing the number of individual log files that need to be processed during @@ -7,18 +10,19 @@ //! //! ## Overview //! -//! Log compaction creates files with the naming pattern `{start_version}.{end_version}.compacted.json` -//! that contain the reconciled actions from all commit files in the specified version range. -//! Only commit/compaction files that intersect with [start_version, end_version] are processed. -//! Note that `end_version` must be greater than `start_version` (equal versions are not allowed). -//! This is similar to checkpoints but operates on a subset of versions rather than the entire table. +//! Log compaction creates files with the naming pattern +//! `{start_version}.{end_version}.compacted.json` that contain the reconciled actions from all +//! commit files in the specified version range. Only commit/compaction files that intersect with +//! [start_version, end_version] are processed. Note that `end_version` must be greater than +//! `start_version` (equal versions are not allowed). This is similar to checkpoints but operates on +//! a subset of versions rather than the entire table. //! //! ## Usage //! //! The log compaction API follows a similar pattern to the checkpoint API: //! -//! 1. Create a [`LogCompactionWriter`] using [`crate::Snapshot::log_compaction_writer`] to compact the log -//! from a given start_version to end_version (inclusive) +//! 1. Create a [`LogCompactionWriter`] using [`crate::Snapshot::log_compaction_writer`] to compact +//! the log from a given start_version to end_version (inclusive) //! 2. Get the compaction path from [`LogCompactionWriter::compaction_path`] //! 3. Get the compaction data from [`LogCompactionWriter::compaction_data`] //! 4. Write the data to the path in cloud storage (engine-specific) @@ -73,14 +77,13 @@ //! - **Log Compaction**: Aggregates only a specific range of commit files //! - Both use similar action reconciliation logic but serve different use cases -use std::sync::{Arc, LazyLock}; +use std::sync::LazyLock; use crate::actions::{ - Add, DomainMetadata, Metadata, Protocol, Remove, SetTransaction, Sidecar, ADD_NAME, - DOMAIN_METADATA_NAME, METADATA_NAME, PROTOCOL_NAME, REMOVE_NAME, SET_TRANSACTION_NAME, - SIDECAR_NAME, + ADD_FIELD, DOMAIN_METADATA_FIELD, METADATA_FIELD, PROTOCOL_FIELD, REMOVE_FIELD, + SET_TRANSACTION_FIELD, SIDECAR_FIELD, }; -use crate::schema::{SchemaRef, StructField, StructType, ToSchema as _}; +use crate::schema::{lazy_schema_ref, SchemaRef}; mod writer; @@ -91,14 +94,12 @@ mod tests; /// Schema for extracting relevant actions from log files for compaction. /// CommitInfo is excluded as it's not needed in compaction files. -static COMPACTION_ACTIONS_SCHEMA: LazyLock = LazyLock::new(|| { - Arc::new(StructType::new_unchecked([ - StructField::nullable(ADD_NAME, Add::to_schema()), - StructField::nullable(REMOVE_NAME, Remove::to_schema()), - StructField::nullable(METADATA_NAME, Metadata::to_schema()), - StructField::nullable(PROTOCOL_NAME, Protocol::to_schema()), - StructField::nullable(SET_TRANSACTION_NAME, SetTransaction::to_schema()), - StructField::nullable(DOMAIN_METADATA_NAME, DomainMetadata::to_schema()), - StructField::nullable(SIDECAR_NAME, Sidecar::to_schema()), - ])) -}); +static COMPACTION_ACTIONS_SCHEMA: LazyLock = lazy_schema_ref! { + (&ADD_FIELD), + (&REMOVE_FIELD), + (&METADATA_FIELD), + (&PROTOCOL_FIELD), + (&SET_TRANSACTION_FIELD), + (&DOMAIN_METADATA_FIELD), + (&SIDECAR_FIELD), +}; diff --git a/kernel/src/log_compaction/tests.rs b/kernel/src/log_compaction/tests.rs index 20ec0ceb97..e2a6f6a562 100644 --- a/kernel/src/log_compaction/tests.rs +++ b/kernel/src/log_compaction/tests.rs @@ -23,6 +23,7 @@ fn create_multi_version_snapshot() -> SnapshotRef { } #[test] +#[ignore = "log compaction disabled (#2337)"] fn test_log_compaction_writer_creation() { let snapshot = create_mock_snapshot(); let start_version = 0; @@ -37,6 +38,7 @@ fn test_log_compaction_writer_creation() { } #[test] +#[ignore = "log compaction disabled (#2337)"] fn test_invalid_version_range() { let start_version = 20; let end_version = 10; // Invalid: start > end @@ -51,6 +53,7 @@ fn test_invalid_version_range() { } #[test] +#[ignore = "log compaction disabled (#2337)"] fn test_equal_version_range_invalid() { let start_version = 5; let end_version = 5; // Invalid: start == end (must be start < end) @@ -65,6 +68,7 @@ fn test_equal_version_range_invalid() { } #[test] +#[ignore = "log compaction disabled (#2337)"] fn test_should_compact() { assert!(should_compact(9, 10)); assert!(!should_compact(5, 10)); @@ -74,6 +78,7 @@ fn test_should_compact() { } #[test] +#[ignore = "log compaction disabled (#2337)"] fn test_compaction_actions_schema_access() { let schema = &*COMPACTION_ACTIONS_SCHEMA; assert!(schema.fields().len() > 0); @@ -87,6 +92,7 @@ fn test_compaction_actions_schema_access() { } #[test] +#[ignore = "log compaction disabled (#2337)"] fn test_writer_debug_impl() { let snapshot = create_mock_snapshot(); let writer = LogCompactionWriter::try_new(snapshot, 1, 5).unwrap(); @@ -96,6 +102,7 @@ fn test_writer_debug_impl() { } #[test] +#[ignore = "log compaction disabled (#2337)"] fn test_compaction_data() { let snapshot = create_mock_snapshot(); let mut writer = LogCompactionWriter::try_new(snapshot, 0, 1).unwrap(); @@ -119,6 +126,7 @@ fn test_compaction_data() { } #[test] +#[ignore = "log compaction disabled (#2337)"] fn test_end_version_exceeds_snapshot_version() { let snapshot = create_mock_snapshot(); let snapshot_version = snapshot.version(); @@ -136,6 +144,7 @@ fn test_end_version_exceeds_snapshot_version() { } #[test] +#[ignore = "log compaction disabled (#2337)"] fn test_retention_calculator() { let snapshot = create_mock_snapshot(); let writer = LogCompactionWriter::try_new(snapshot.clone(), 0, 1).unwrap(); @@ -145,6 +154,7 @@ fn test_retention_calculator() { } #[test] +#[ignore = "log compaction disabled (#2337)"] fn test_compaction_data_with_actual_iterator() { let snapshot = create_multi_version_snapshot(); let mut writer = LogCompactionWriter::try_new(snapshot, 0, 1).unwrap(); @@ -174,6 +184,7 @@ fn test_compaction_data_with_actual_iterator() { } #[test] +#[ignore = "log compaction disabled (#2337)"] fn test_compaction_paths() { let snapshot = create_mock_snapshot(); @@ -207,6 +218,7 @@ fn test_compaction_paths() { } #[test] +#[ignore = "log compaction disabled (#2337)"] fn test_version_filtering() { let snapshot = create_multi_version_snapshot(); let engine = SyncEngine::new(); @@ -230,21 +242,25 @@ fn test_version_filtering() { } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_no_compaction_staged_commits() { + use std::sync::Arc; + use crate::actions::Add; - use crate::engine::default::DefaultEngineBuilder; - use crate::object_store::{memory::InMemory, path::Path, ObjectStore}; + use crate::engine::sync::SyncEngine; + use crate::object_store::memory::InMemory; + use crate::object_store::path::Path; + use crate::object_store::ObjectStoreExt as _; use crate::table_features::TableFeature; - use std::sync::Arc; // Set up in-memory store let store = Arc::new(InMemory::new()); - let engine = DefaultEngineBuilder::new(store.clone()).build(); + let engine = SyncEngine::new_with_store(store.clone()); // Create basic commits with proper metadata and protocol use crate::actions::{Metadata, Protocol}; use crate::schema::{DataType as KernelDataType, StructField, StructType}; - use crate::utils::test_utils::Action; + use crate::unit_test_utils::Action; let metadata = Action::Metadata( Metadata::try_new( @@ -304,3 +320,13 @@ async fn test_no_compaction_staged_commits() { // The validation in LogCompactionWriter is a safety check for edge cases // where staged commits might slip through the normal filtering } + +// === Tests for disabled log compaction (TODO(#2337): remove when re-enabled) === + +#[test] +fn test_should_compact_always_false() { + // These inputs would return true if compaction were enabled + assert!(!should_compact(9, 10)); + assert!(!should_compact(19, 10)); + assert!(!should_compact(99, 100)); +} diff --git a/kernel/src/log_compaction/writer.rs b/kernel/src/log_compaction/writer.rs index dcf8acbf97..034a66f18d 100644 --- a/kernel/src/log_compaction/writer.rs +++ b/kernel/src/log_compaction/writer.rs @@ -1,3 +1,6 @@ +// TODO(#2337): remove dead_code allows when log compaction is re-enabled +#![allow(dead_code, unused_imports)] + use url::Url; use super::COMPACTION_ACTIONS_SCHEMA; @@ -11,22 +14,32 @@ use crate::{DeltaResult, Engine, Error, SnapshotRef, Version}; /// Determine if log compaction should be performed based on the commit version and /// compaction interval. -pub fn should_compact(commit_version: Version, compaction_interval: Version) -> bool { - // Commits start at 0, so we add one to the commit version to check if we've hit the interval - compaction_interval > 0 - && commit_version > 0 - && ((commit_version + 1) % compaction_interval) == 0 +/// +/// Always returns `false` because log compaction is currently disabled. +pub fn should_compact(_commit_version: Version, _compaction_interval: Version) -> bool { + // TODO(#2337): re-enable log compaction once testing is sufficient + // + // // Commits start at 0, so we add one to the commit version to check if we've hit the interval + // compaction_interval > 0 + // && commit_version > 0 + // && (commit_version + 1).is_multiple_of(compaction_interval) + false } /// Writer for log compaction files /// /// This writer provides an API for creating log compaction files that aggregate actions /// from multiple commit files. +/// +/// Log compaction is currently disabled (#2337) #[derive(Debug)] pub struct LogCompactionWriter { /// Reference to the snapshot of the table being compacted snapshot: SnapshotRef, + // TODO(#2337): remove allow(dead_code) when log compaction is re-enabled + #[allow(dead_code)] start_version: Version, + #[allow(dead_code)] end_version: Version, /// Cached compaction file path compaction_path: Url, @@ -39,31 +52,39 @@ impl RetentionCalculator for LogCompactionWriter { } impl LogCompactionWriter { + // TODO(#2337): re-enable log compaction once testing is sufficient pub(crate) fn try_new( - snapshot: SnapshotRef, - start_version: Version, - end_version: Version, + _snapshot: SnapshotRef, + _start_version: Version, + _end_version: Version, ) -> DeltaResult { - if start_version >= end_version { - return Err(Error::generic(format!( - "Invalid version range: end_version {end_version} must be greater than start_version {start_version}" - ))); - } - - // We disallow log compaction if the Snapshot is not published. If we didn't, this could - // create gaps in the version history, thereby breaking old readers. - snapshot.log_segment().validate_published()?; - - // Compute the compaction path once during construction - let compaction_path = - ParsedLogPath::new_log_compaction(snapshot.table_root(), start_version, end_version)?; - - Ok(Self { - snapshot, - start_version, - end_version, - compaction_path: compaction_path.location, - }) + Err(Error::unsupported( + "Log compaction is not currently supported", + )) + // if start_version >= end_version { + // return Err(Error::generic(format!( + // "Invalid version range: end_version {end_version} must be greater than \ + // start_version {start_version}" + // ))); + // } + // + // // We disallow log compaction if the Snapshot is not published. If we didn't, this + // // could create gaps in the version history, thereby breaking old readers. + // snapshot.log_segment().validate_published()?; + // + // // Compute the compaction path once during construction + // let compaction_path = ParsedLogPath::new_log_compaction( + // snapshot.table_root(), + // start_version, + // end_version, + // )?; + // + // Ok(Self { + // snapshot, + // start_version, + // end_version, + // compaction_path: compaction_path.location, + // }) } /// Get the path where the compaction file will be written diff --git a/kernel/src/log_path.rs b/kernel/src/log_path.rs index fe468873c7..0ff50121df 100644 --- a/kernel/src/log_path.rs +++ b/kernel/src/log_path.rs @@ -1,11 +1,11 @@ //! Public-facing [`LogPath`] type for representing paths to delta log files. +use url::Url; + use crate::path::ParsedLogPath; use crate::utils::require; use crate::{DeltaResult, Error, FileMeta, FileSize}; -use url::Url; - /// A path to a valid delta log file. You can parse a given `FileMeta` into a `LogPath` using /// [`LogPath::try_new`]. /// @@ -70,10 +70,10 @@ impl LogPath { #[cfg(test)] mod test { - use super::*; - use std::str::FromStr; + use super::*; + #[test] fn test_staged_commit_path_creation() { let table_root = Url::from_str("s3://my-bucket/my-table/").unwrap(); diff --git a/kernel/src/log_reader/checkpoint_manifest.rs b/kernel/src/log_reader/checkpoint_manifest.rs index 3d2d59de02..f7e31e7e6e 100644 --- a/kernel/src/log_reader/checkpoint_manifest.rs +++ b/kernel/src/log_reader/checkpoint_manifest.rs @@ -6,19 +6,18 @@ use itertools::Itertools; use url::Url; use crate::actions::visitors::SidecarVisitor; -use crate::actions::{Add, Remove, Sidecar, ADD_NAME}; -use crate::actions::{REMOVE_NAME, SIDECAR_NAME}; +use crate::actions::{ADD_FIELD, REMOVE_FIELD, SIDECAR_FIELD}; use crate::log_replay::ActionsBatch; use crate::path::ParsedLogPath; -use crate::schema::{SchemaRef, StructField, StructType, ToSchema}; +use crate::schema::{lazy_schema_ref, SchemaRef}; use crate::utils::require; -use crate::{DeltaResult, Engine, Error, FileMeta, RowVisitor}; +use crate::{DeltaResult, DeltaResultIteratorStatic, Engine, Error, FileMeta, RowVisitor}; /// Phase that processes single-part checkpoint. This also treats the checkpoint as a manifest file /// and extracts the sidecar actions during iteration. #[allow(unused)] pub(crate) struct CheckpointManifestReader { - actions: Box> + Send>, + actions: DeltaResultIteratorStatic, sidecar_visitor: SidecarVisitor, log_root: Url, is_complete: bool, @@ -41,13 +40,11 @@ impl CheckpointManifestReader { manifest: &ParsedLogPath, log_root: Url, ) -> DeltaResult { - static MANIFEST_READ_SCHMEA: LazyLock = LazyLock::new(|| { - Arc::new(StructType::new_unchecked([ - StructField::nullable(ADD_NAME, Add::to_schema()), - StructField::nullable(REMOVE_NAME, Remove::to_schema()), - StructField::nullable(SIDECAR_NAME, Sidecar::to_schema()), - ])) - }); + static MANIFEST_READ_SCHMEA: LazyLock = lazy_schema_ref! { + (&ADD_FIELD), + (&REMOVE_FIELD), + (&SIDECAR_FIELD), + }; let actions = match manifest.extension.as_str() { "json" => engine.json_handler().read_json_files( @@ -118,15 +115,16 @@ impl Iterator for CheckpointManifestReader { #[cfg(test)] mod tests { + use std::sync::Arc; + + use itertools::Itertools; + use super::*; use crate::arrow::array::{Array, StringArray, StructArray}; use crate::engine::arrow_data::EngineDataArrowExt as _; - use crate::utils::test_utils::{assert_result_error_with_message, load_test_table}; + use crate::unit_test_utils::{assert_result_error_with_message, load_test_table}; use crate::SnapshotRef; - use itertools::Itertools; - use std::sync::Arc; - /// Helper function to test manifest phase with expected add paths and sidecars fn verify_manifest_phase( engine: Arc, diff --git a/kernel/src/log_reader/commit.rs b/kernel/src/log_reader/commit.rs index 34f128db1c..db5f79dc94 100644 --- a/kernel/src/log_reader/commit.rs +++ b/kernel/src/log_reader/commit.rs @@ -2,14 +2,15 @@ use itertools::Itertools; +use crate::cancellation::CancellationTokenRef; use crate::log_replay::ActionsBatch; use crate::log_segment::LogSegment; use crate::schema::SchemaRef; -use crate::{DeltaResult, Engine}; +use crate::{DeltaResult, DeltaResultIteratorStatic, Engine}; /// Phase that processes JSON commit files into [`ActionsBatch`]s pub(crate) struct CommitReader { - actions: Box> + Send>, + actions: DeltaResultIteratorStatic, } impl CommitReader { @@ -19,15 +20,22 @@ impl CommitReader { /// - `engine`: Engine for reading files /// - `log_segment`: The log segment to process /// - `schema`: The schema to read the json files + /// - `cancellation_token`: Optional token so a cancelled request can stop the commit read pub(crate) fn try_new( engine: &dyn Engine, log_segment: &LogSegment, schema: SchemaRef, + cancellation_token: Option<&CancellationTokenRef>, ) -> DeltaResult { let commit_files = log_segment.find_commit_cover(); let actions = engine .json_handler() - .read_json_files(&commit_files, schema, None)? + .read_json_files_with_cancellation( + &commit_files, + schema, + None, + cancellation_token.cloned(), + )? .map_ok(|batch| ActionsBatch::new(batch, true)); Ok(Self { @@ -46,13 +54,15 @@ impl Iterator for CommitReader { #[cfg(test)] mod tests { + use std::sync::Arc; + + use itertools::Itertools; + use super::*; use crate::arrow::array::{StringArray, StructArray}; use crate::engine::arrow_data::EngineDataArrowExt as _; use crate::scan::COMMIT_READ_SCHEMA; - use crate::utils::test_utils::load_test_table; - use itertools::Itertools; - use std::sync::Arc; + use crate::unit_test_utils::load_test_table; #[test] fn test_commit_phase_processes_commits() -> Result<(), Box> { @@ -60,7 +70,7 @@ mod tests { let log_segment = Arc::new(snapshot.log_segment().clone()); let schema = COMMIT_READ_SCHEMA.clone(); - let commit_phase = CommitReader::try_new(engine.as_ref(), &log_segment, schema)?; + let commit_phase = CommitReader::try_new(engine.as_ref(), &log_segment, schema, None)?; let mut file_paths = vec![]; for result in commit_phase { diff --git a/kernel/src/log_replay/deduplicator.rs b/kernel/src/log_replay/deduplicator.rs index e1e609daf4..727daf0a36 100644 --- a/kernel/src/log_replay/deduplicator.rs +++ b/kernel/src/log_replay/deduplicator.rs @@ -2,8 +2,8 @@ //! //! The [`Deduplicator`] trait supports two deduplication strategies: //! -//! - **JSON commit files** (`is_log_batch = true`): Tracks (path, dv_unique_id) and updates -//! the hashmap as files are seen. Implementation: [`FileActionDeduplicator`] +//! - **JSON commit files** (`is_log_batch = true`): Tracks (path, dv_unique_id) and updates the +//! hashmap as files are seen. Implementation: [`FileActionDeduplicator`] //! //! - **Checkpoint files** (`is_log_batch = false`): Uses (path, dv_unique_id) to filter actions //! using a read-only hashmap pre-populated from the commit log phase. Future implementation. @@ -12,13 +12,25 @@ use std::collections::HashSet; +use tracing::warn; + use crate::actions::deletion_vector::DeletionVectorDescriptor; use crate::engine_data::{GetData, TypedGetData}; use crate::log_replay::FileActionKey; use crate::DeltaResult; +/// Information we want to return to the add-dedup about file related actions +pub(crate) struct FileActionInfo { + /// A key that uniquely identifies the file, includes the path and dv info + pub(crate) key: FileActionKey, + /// The size of the file. Might be 0 for removes + pub(crate) size: u64, + /// If this action was an add + pub(crate) is_add: bool, +} + pub(crate) trait Deduplicator { - /// Extracts a file action key from the data. Returns `(key, is_add)` if found. + /// Extracts a file action key from the data. Returns a `FileActionInfo` if found. /// /// TODO: Remove the skip_removes field in the future. The caller is responsible for using the /// correct Deduplicator instance depending on whether the batch belongs to a commit or to a @@ -28,13 +40,14 @@ pub(crate) trait Deduplicator { i: usize, getters: &[&'a dyn GetData<'a>], skip_removes: bool, - ) -> DeltaResult>; + ) -> DeltaResult>; /// Checks if this file has been seen. When `is_log_batch() = true`, updates the hashmap /// to track new files. Returns `true` if the file should be filtered out. fn check_and_record_seen(&mut self, key: FileActionKey) -> bool; - /// Returns `true` for commit log batches (updates hashmap), `false` for checkpoints (read-only). + /// Returns `true` for commit log batches (updates hashmap), `false` for checkpoints + /// (read-only). fn is_log_batch(&self) -> bool; /// Extracts the deletion vector unique ID if it exists. @@ -42,7 +55,8 @@ pub(crate) trait Deduplicator { /// This function retrieves the necessary fields for constructing a deletion vector unique ID /// by accessing `getters` at `dv_start_index` and the following two indices. Specifically: /// - `dv_start_index` retrieves the storage type (`deletionVector.storageType`). - /// - `dv_start_index + 1` retrieves the path or inline deletion vector (`deletionVector.pathOrInlineDv`). + /// - `dv_start_index + 1` retrieves the path or inline deletion vector + /// (`deletionVector.pathOrInlineDv`). /// - `dv_start_index + 2` retrieves the optional offset (`deletionVector.offset`). fn extract_dv_unique_id<'a>( &self, @@ -77,6 +91,7 @@ pub(crate) trait Deduplicator { pub(crate) struct CheckpointDeduplicator<'a> { seen_file_keys: &'a HashSet, add_path_index: usize, + add_size_index: usize, add_dv_start_index: usize, } @@ -85,11 +100,13 @@ impl<'a> CheckpointDeduplicator<'a> { pub(crate) fn try_new( seen_file_keys: &'a HashSet, add_path_index: usize, + add_size_index: usize, add_dv_start_index: usize, ) -> DeltaResult { Ok(CheckpointDeduplicator { seen_file_keys, add_path_index, + add_size_index, add_dv_start_index, }) } @@ -102,12 +119,26 @@ impl Deduplicator for CheckpointDeduplicator<'_> { i: usize, getters: &[&'b dyn GetData<'b>], _skip_removes: bool, - ) -> DeltaResult> { + ) -> DeltaResult> { let Some(path) = getters[self.add_path_index].get_str(i, "add.path")? else { return Ok(None); }; let dv_unique_id = self.extract_dv_unique_id(i, getters, self.add_dv_start_index)?; - Ok(Some((FileActionKey::new(path, dv_unique_id), true))) + let size = match getters[self.add_size_index].get_long(i, "add.size")? { + Some(s) => u64::try_from(s).unwrap_or_else(|e| { + warn!("Could not convert add.size {s} to u64: {e}"); + 0 + }), + None => { + warn!("Add action without required size field"); + 0 + } + }; + Ok(Some(FileActionInfo { + key: FileActionKey::new(path, dv_unique_id), + size, + is_add: true, + })) } /// Read-only check against seen set. Returns `true` if file should be filtered out. diff --git a/kernel/src/log_replay.rs b/kernel/src/log_replay/mod.rs similarity index 82% rename from kernel/src/log_replay.rs rename to kernel/src/log_replay/mod.rs index 2073f3a89d..188355da7d 100644 --- a/kernel/src/log_replay.rs +++ b/kernel/src/log_replay/mod.rs @@ -13,17 +13,16 @@ //! This module provides structures for efficient batch processing, focusing on file action //! deduplication with `FileActionDeduplicator` which tracks unique files across log batches //! to minimize memory usage for tables with extensive history. -use crate::engine_data::GetData; -use crate::log_replay::deduplicator::Deduplicator; -use crate::scan::data_skipping::DataSkippingFilter; -use crate::{DeltaResult, EngineData}; - -use delta_kernel_derive::internal_api; - use std::collections::HashSet; use std::sync::Arc; -use tracing::debug; +use delta_kernel_derive::internal_api; +use tracing::{debug, warn}; + +use crate::engine_data::GetData; +use crate::log_replay::deduplicator::{Deduplicator, FileActionInfo}; +use crate::scan::data_skipping::DataSkippingFilter; +use crate::{DeltaResult, EngineData}; pub(crate) mod deduplicator; @@ -31,15 +30,28 @@ pub(crate) mod deduplicator; /// of adds and removes during log replay. #[derive(Debug, Hash, Eq, PartialEq, serde::Serialize, serde::Deserialize, Clone)] pub struct FileActionKey { - pub(crate) path: String, - pub(crate) dv_unique_id: Option, + path: String, + dv_unique_id: Option, } impl FileActionKey { - pub(crate) fn new(path: impl Into, dv_unique_id: Option) -> Self { + /// Build a key from a path and optional deletion-vector unique id. + pub fn new(path: impl Into, dv_unique_id: Option) -> Self { let path = path.into(); Self { path, dv_unique_id } } + + /// The data file path, as stored on the action. + pub fn path(&self) -> &str { + &self.path + } + + /// Deletion-vector unique id, or `None` if the action carries no deletion vector. Same + /// `path` with different `dv_unique_id` (e.g. an `add(P, dv=new) + remove(P, dv=old)` + /// DV-replacement pair) refers to distinct logical file identities. + pub fn dv_unique_id(&self) -> Option<&str> { + self.dv_unique_id.as_deref() + } } /// Maintains state and provides functionality for deduplicating file actions during log replay. @@ -64,6 +76,8 @@ pub(crate) struct FileActionDeduplicator<'seen> { is_log_batch: bool, /// Index of the getter containing the add.path column add_path_index: usize, + /// Index of the getter containing the add.size column + add_size_index: usize, /// Index of the getter containing the remove.path column remove_path_index: usize, /// Starting index for add action deletion vector columns @@ -77,6 +91,7 @@ impl<'seen> FileActionDeduplicator<'seen> { seen_file_keys: &'seen mut HashSet, is_log_batch: bool, add_path_index: usize, + add_size_index: usize, remove_path_index: usize, add_dv_start_index: usize, remove_dv_start_index: usize, @@ -85,6 +100,7 @@ impl<'seen> FileActionDeduplicator<'seen> { seen_file_keys, is_log_batch, add_path_index, + add_size_index, remove_path_index, add_dv_start_index, remove_dv_start_index, @@ -133,7 +149,8 @@ impl Deduplicator for FileActionDeduplicator<'_> { /// - `skip_removes`: Whether to skip remove actions when extracting file actions /// /// # Returns - /// - `Ok(Some((key, is_add)))`: When a file action is found, returns the key and whether it's an add operation + /// - `Ok(Some(FileActionInfo))`: When a file action is found, This contains the key whether + /// it's an add operation, and the size of the file /// - `Ok(None)`: When no file action is found /// - `Err(...)`: On any error during extraction fn extract_file_action<'a>( @@ -141,14 +158,29 @@ impl Deduplicator for FileActionDeduplicator<'_> { i: usize, getters: &[&'a dyn GetData<'a>], skip_removes: bool, - ) -> DeltaResult> { + ) -> DeltaResult> { // Try to extract an add action by the required path column if let Some(path) = getters[self.add_path_index].get_str(i, "add.path")? { + let size = match getters[self.add_size_index].get_long(i, "add.size")? { + Some(s) => u64::try_from(s).unwrap_or_else(|e| { + warn!("Could not convert add.size {s} to u64: {e}"); + 0 + }), + None => { + warn!("Add action without required size field"); + 0 + } + }; let dv_unique_id = self.extract_dv_unique_id(i, getters, self.add_dv_start_index)?; - return Ok(Some((FileActionKey::new(path, dv_unique_id), true))); + return Ok(Some(FileActionInfo { + key: FileActionKey::new(path, dv_unique_id), + size, + is_add: true, + })); } - // The AddRemoveDedupVisitor skips remove actions when extracting file actions from a checkpoint batch. + // The AddRemoveDedupVisitor skips remove actions when extracting file actions from a + // checkpoint batch. if skip_removes { return Ok(None); } @@ -156,7 +188,11 @@ impl Deduplicator for FileActionDeduplicator<'_> { // Try to extract a remove action by the required path column if let Some(path) = getters[self.remove_path_index].get_str(i, "remove.path")? { let dv_unique_id = self.extract_dv_unique_id(i, getters, self.remove_dv_start_index)?; - return Ok(Some((FileActionKey::new(path, dv_unique_id), false))); + return Ok(Some(FileActionInfo { + key: FileActionKey::new(path, dv_unique_id), + size: 0, + is_add: false, + })); } // No file action found @@ -186,8 +222,8 @@ impl ActionsBatch { /// /// # Parameters /// - `actions`: A boxed [`EngineData`] instance representing the actions batch. - /// - `is_log_batch`: A boolean indicating whether the batch is from a commit log (`true`) or - /// a checkpoint/CRC/elsewhere (`false`). + /// - `is_log_batch`: A boolean indicating whether the batch is from a commit log (`true`) or a + /// checkpoint/CRC/elsewhere (`false`). pub(crate) fn new(actions: Box, is_log_batch: bool) -> Self { Self { actions, @@ -223,55 +259,59 @@ where /// A trait for processing batches of actions from Delta transaction logs during log replay. /// -/// Log replay processors scan transaction logs in **reverse chronological order** (newest to oldest), -/// filtering and transforming action batches into specialized output types. These processors: +/// Log replay processors scan transaction logs in **reverse chronological order** (newest to +/// oldest), filtering and transforming action batches into specialized output types. These +/// processors: /// /// - **Track and deduplicate file actions** to apply appropriate `Remove` actions to corresponding /// `Add` actions (and omit the file from the log replay output) /// - **Maintain selection vectors** to indicate which actions in each batch should be included. -/// - **Apply custom filtering logic** based on the processor’s purpose (e.g., checkpointing, scanning). -/// - **Data skipping** filters are applied to the initial selection vector to reduce the number of rows -/// processed by the processor, (if a filter is provided). +/// - **Apply custom filtering logic** based on the processor’s purpose (e.g., checkpointing, +/// scanning). +/// - **Data skipping** filters are applied to the initial selection vector to reduce the number of +/// rows processed by the processor, (if a filter is provided). /// /// # Implementations /// -/// - [`ScanLogReplayProcessor`]: Used for table scans, this processor filters and selects deduplicated -/// `Add` actions from log batches to reconstruct the view of the table at a specific point in time. -/// Note that scans do not expose `Remove` actions. Data skipping may be applied when a predicate is -/// provided. +/// - [`ScanLogReplayProcessor`]: Used for table scans, this processor filters and selects +/// deduplicated `Add` actions from log batches to reconstruct the view of the table at a specific +/// point in time. Note that scans do not expose `Remove` actions. Data skipping may be applied +/// when a predicate is provided. /// -/// - [`ActionReconciliationProcessor`]: Used for action reconciliation (including checkpoint writing), -/// this processor filters and selects actions from log batches for inclusion in V1 spec checkpoint files. -/// Unlike scans, action reconciliation processing includes additional actions, such as `Remove`, `Metadata`, -/// and `Protocol`, required to fully reconstruct table state. Data skipping is not applied during action -/// reconciliation processing. +/// - [`ActionReconciliationProcessor`]: Used for action reconciliation (including checkpoint +/// writing), this processor filters and selects actions from log batches for inclusion in V1 spec +/// checkpoint files. Unlike scans, action reconciliation processing includes additional actions, +/// such as `Remove`, `Metadata`, and `Protocol`, required to fully reconstruct table state. Data +/// skipping is not applied during action reconciliation processing. /// /// [`ActionReconciliationProcessor`]: crate::action_reconciliation::log_replay::ActionReconciliationProcessor /// /// # Action Iterator Input /// -/// The [`LogReplayProcessor::process_actions_iter`] method is the entry point for log replay processing. -/// It takes as input an iterator of (actions batch, is_commit_batch flag) tuples and returns an iterator of -/// processor-specific output types with selection vectors. The is_commit_batch bool flag in each tuple -/// indicates whether the batch came from a commit log (`true`) or checkpoint (`false`). Action batches -/// **must** be sorted by the order of the actions in the log from most recent to oldest. +/// The [`LogReplayProcessor::process_actions_iter`] method is the entry point for log replay +/// processing. It takes as input an iterator of (actions batch, is_commit_batch flag) tuples and +/// returns an iterator of processor-specific output types with selection vectors. The +/// is_commit_batch bool flag in each tuple indicates whether the batch came from a commit log +/// (`true`) or checkpoint (`false`). Action batches **must** be sorted by the order of the actions +/// in the log from most recent to oldest. /// /// Each row that is selected in the returned output **must** be included in the processor's result /// (e.g., in scan results or checkpoint files), while non-selected rows **must** be ignored. /// /// # Output Types /// -/// The [`LogReplayProcessor::Output`] type represents the material result of log replay, and it must -/// implement the [`HasSelectionVector`] trait to allow filtering of irrelevant rows: +/// The [`LogReplayProcessor::Output`] type represents the material result of log replay, and it +/// must implement the [`HasSelectionVector`] trait to allow filtering of irrelevant rows: /// /// - For **scans**, the output type is [`ScanMetadata`], which contains the file actions (`Add` -/// actions) that need to be applied to build the table's view, accompanied by a -/// **selection vector** that identifies which rows should be included. A transform vector may -/// also be included to handle schema changes, such as renaming columns or modifying data types. +/// actions) that need to be applied to build the table's view, accompanied by a **selection +/// vector** that identifies which rows should be included. A transform vector may also be +/// included to handle schema changes, such as renaming columns or modifying data types. /// /// - For **checkpoints**, the output type is [`FilteredEngineData`], which includes the actions /// necessary to write to the checkpoint file (`Add`, `Remove`, `Metadata`, `Protocol` actions), -/// filtered by the **selection vector** to determine which rows are included in the final checkpoint. +/// filtered by the **selection vector** to determine which rows are included in the final +/// checkpoint. /// /// TODO: Refactor the Change Data Feed (CDF) processor to use this trait. #[allow(rustdoc::broken_intra_doc_links, rustdoc::private_intra_doc_links)] @@ -287,9 +327,11 @@ pub(crate) trait LogReplayProcessor: Sized { /// representing a batch of actions and a boolean flag indicating whether the batch originates /// from a commit log, `false` if from a checkpoint. /// - /// Returns a [`DeltaResult`] containing the processor’s output, which includes only selected actions. + /// Returns a [`DeltaResult`] containing the processor’s output, which includes only selected + /// actions. /// - /// Note: Since log replay is stateful, processing may update internal processor state (e.g., deduplication sets). + /// Note: Since log replay is stateful, processing may update internal processor state (e.g., + /// deduplication sets). fn process_actions_batch(&mut self, actions_batch: ActionsBatch) -> DeltaResult; /// Applies the processor to an actions iterator and filters out empty results. @@ -332,8 +374,8 @@ pub(crate) trait LogReplayProcessor: Sized { /// - `batch`: A reference to the batch of actions to be processed. /// /// # Returns - /// A `DeltaResult>`, where each boolean indicates if the corresponding row should be included. - /// If no filter is provided, all rows are selected. + /// A `DeltaResult>`, where each boolean indicates if the corresponding row should be + /// included. If no filter is provided, all rows are selected. fn build_selection_vector(&self, batch: &dyn EngineData) -> DeltaResult> { match self.data_skipping_filter() { Some(filter) => filter.apply(batch), @@ -357,16 +399,20 @@ pub(crate) trait HasSelectionVector { #[cfg(test)] mod tests { + use std::collections::{HashMap, HashSet}; + + use rstest::rstest; + use super::deduplicator::CheckpointDeduplicator; use super::*; use crate::engine_data::GetData; use crate::DeltaResult; - use std::collections::{HashMap, HashSet}; /// Mock GetData implementation for testing struct MockGetData { string_values: HashMap<(usize, String), String>, int_values: HashMap<(usize, String), i32>, + long_values: HashMap<(usize, String), i64>, errors: HashMap<(usize, String), String>, } @@ -375,6 +421,7 @@ mod tests { Self { string_values: HashMap::new(), int_values: HashMap::new(), + long_values: HashMap::new(), errors: HashMap::new(), } } @@ -387,6 +434,10 @@ mod tests { fn add_int(&mut self, row: usize, field: &str, value: i32) { self.int_values.insert((row, field.to_string()), value); } + + fn add_long(&mut self, row: usize, field: &str, value: i64) { + self.long_values.insert((row, field.to_string()), value); + } } impl<'a> GetData<'a> for MockGetData { @@ -409,6 +460,16 @@ mod tests { .get(&(row_index, field_name.to_string())) .cloned()) } + + fn get_long(&'a self, row_index: usize, field_name: &str) -> DeltaResult> { + if let Some(error_msg) = self.errors.get(&(row_index, field_name.to_string())) { + return Err(crate::Error::Generic(error_msg.clone())); + } + Ok(self + .long_values + .get(&(row_index, field_name.to_string())) + .cloned()) + } } /// Helper to create a FileActionDeduplicator with standard indices @@ -420,6 +481,7 @@ mod tests { seen, is_log_batch, 0, // add_path_index + 1, // add_size_index, 5, // remove_path_index 2, // add_dv_start_index 6, // remove_dv_start_index @@ -437,7 +499,7 @@ mod tests { let empty_ref = &*EMPTY; vec![ add_mock.unwrap_or(empty_ref), // 0: add.path - empty_ref, // 1: (unused) + add_mock.unwrap_or(empty_ref), // 1: add.size add_mock.unwrap_or(empty_ref), // 2: add.dv.storageType add_mock.unwrap_or(empty_ref), // 3: add.dv.pathOrInlineDv add_mock.unwrap_or(empty_ref), // 4: add.dv.offset @@ -448,21 +510,30 @@ mod tests { ] } - #[test] - fn test_extract_file_action_add() -> DeltaResult<()> { + #[rstest] + #[case::valid_size(Some(100), 100)] + #[case::missing_size_defaults_to_zero(None, 0)] + #[case::negative_size_defaults_to_zero(Some(-1), 0)] + fn test_extract_file_action_add( + #[case] raw_size: Option, + #[case] expected_size: u64, + ) -> DeltaResult<()> { let mut seen = HashSet::new(); let deduplicator = create_deduplicator(&mut seen, true); let mut mock_add = MockGetData::new(); mock_add.add_string(0, "add.path", "file1.parquet"); + if let Some(s) = raw_size { + mock_add.add_long(0, "add.size", s); + } let getters = create_getters_with_mocks(Some(&mock_add), None); let result = deduplicator.extract_file_action(0, &getters, false)?; - assert!(result.is_some()); - let (key, is_add) = result.unwrap(); + let FileActionInfo { key, size, is_add } = result.unwrap(); assert_eq!(key.path, "file1.parquet"); assert!(key.dv_unique_id.is_none()); assert!(is_add); + assert_eq!(size, expected_size); Ok(()) } @@ -478,7 +549,7 @@ mod tests { let result = deduplicator.extract_file_action(0, &getters, false)?; assert!(result.is_some()); - let (key, is_add) = result.unwrap(); + let FileActionInfo { key, is_add, .. } = result.unwrap(); assert_eq!(key.path, "file2.parquet"); assert!(!is_add); @@ -499,7 +570,7 @@ mod tests { let result = deduplicator.extract_file_action(0, &getters, false)?; assert!(result.is_some()); - let (key, is_add) = result.unwrap(); + let FileActionInfo { key, is_add, .. } = result.unwrap(); assert!(matches!( key.dv_unique_id.as_deref(), Some("s3path/to/dv@100") @@ -612,7 +683,7 @@ mod tests { #[test] fn test_checkpoint_extract_file_action_add() -> DeltaResult<()> { let seen = HashSet::new(); - let deduplicator = CheckpointDeduplicator::try_new(&seen, 0, 2)?; + let deduplicator = CheckpointDeduplicator::try_new(&seen, 0, 2, 3)?; let mut mock_add = MockGetData::new(); mock_add.add_string(0, "add.path", "checkpoint_file.parquet"); @@ -620,7 +691,7 @@ mod tests { let result = deduplicator.extract_file_action(0, &getters, false)?; assert!(result.is_some()); - let (key, is_add) = result.unwrap(); + let FileActionInfo { key, is_add, .. } = result.unwrap(); assert_eq!(key.path, "checkpoint_file.parquet"); assert!(key.dv_unique_id.is_none()); assert!(is_add); @@ -631,7 +702,7 @@ mod tests { #[test] fn test_checkpoint_extract_file_action_with_deletion_vector() -> DeltaResult<()> { let seen = HashSet::new(); - let deduplicator = CheckpointDeduplicator::try_new(&seen, 0, 2)?; + let deduplicator = CheckpointDeduplicator::try_new(&seen, 0, 1, 2)?; let mut mock_dv = MockGetData::new(); mock_dv.add_string(0, "add.path", "file_with_dv.parquet"); @@ -642,7 +713,7 @@ mod tests { let result = deduplicator.extract_file_action(0, &getters, false)?; assert!(result.is_some()); - let (key, is_add) = result.unwrap(); + let FileActionInfo { key, is_add, .. } = result.unwrap(); assert_eq!(key.path, "file_with_dv.parquet"); assert!(matches!( key.dv_unique_id.as_deref(), @@ -664,7 +735,7 @@ mod tests { Some("dv123".to_string()), )); - let mut deduplicator = CheckpointDeduplicator::try_new(&seen, 0, 2)?; + let mut deduplicator = CheckpointDeduplicator::try_new(&seen, 0, 2, 3)?; // File modified in commit - should be filtered from checkpoint let commit_modified = FileActionKey::new("modified_in_commit.parquet", None); diff --git a/kernel/src/log_segment/crc_replay.rs b/kernel/src/log_segment/crc_replay.rs new file mode 100644 index 0000000000..9532a90a99 --- /dev/null +++ b/kernel/src/log_segment/crc_replay.rs @@ -0,0 +1,1076 @@ +//! Reverse log replay for incremental CRC construction. +//! +//! Reverse-replays a log segment's commit files to produce a [`CrcDelta`] covering +//! commits `(X, Y]`. Per the incremental equation `Crc[X] + CrcDelta = Crc[Y]`, that +//! delta is applied to a stale base via [`Crc::apply`]. +//! +//! The base `Crc[X]` itself can also be built here: [`LogSegment::build_crc_from_checkpoint`] +//! reads a checkpoint into a Complete CRC, and [`LogSegment::build_crc_from_version_zero`] builds +//! one from a full reverse replay when there is neither a CRC nor a checkpoint to root at. +// +// TODO(#2615): support log compaction files. + +use std::collections::hash_map::Entry; +use std::sync::{Arc, LazyLock}; + +use tracing::{instrument, warn}; + +use super::LogSegment; +use crate::actions::visitors::{ + visit_metadata_at, visit_protocol_at, METADATA_LEAVES, PROTOCOL_LEAVES, +}; +use crate::actions::{ + DomainMetadata, SetTransaction, ADD_NAME, COMMIT_INFO_NAME, DOMAIN_METADATA_FIELD, + METADATA_FIELD, PROTOCOL_FIELD, REMOVE_NAME, SET_TRANSACTION_FIELD, +}; +use crate::crc::{ + is_incremental_safe_operation, read_crc_file_or_none, size_to_u64, Crc, CrcDelta, + FileSizeHistogram, FileStatsDelta, +}; +use crate::engine_data::{GetData, TypedGetData as _}; +use crate::metrics::ProtocolMetadataSource; +use crate::path::ParsedLogPath; +use crate::schema::{ + column_name, schema, ColumnName, ColumnNamesAndTypes, DataType, MetadataColumnSpec, SchemaRef, +}; +use crate::snapshot::IncrementalReplay; +use crate::utils::require; +use crate::{DeltaResult, Engine, Error, FileMeta, RowVisitor, Version}; + +#[allow(clippy::expect_used)] +static REPLAY_SCHEMA: LazyLock = LazyLock::new(|| { + let base = schema! { + // size is the only Add leaf the visitor reads, and it is required, so its presence marks + // an Add row. + nullable ADD_NAME: { not_null "size": LONG }, + // remove.size is optional, so we read remove.path (required) to know a row is a Remove + // before reading its size. + nullable REMOVE_NAME: { + not_null "path": STRING, + nullable "size": LONG, + }, + (&PROTOCOL_FIELD), + (&METADATA_FIELD), + (&SET_TRANSACTION_FIELD), + (&DOMAIN_METADATA_FIELD), + nullable COMMIT_INFO_NAME: { + nullable "operation": STRING, + nullable "inCommitTimestamp": LONG, + }, + }; + let with_file = base + .add_metadata_column("_file", MetadataColumnSpec::FilePath) + .expect("add _file metadata column"); + Arc::new(with_file) +}); + +impl LogSegment { + /// Try to build the CRC at this segment's `end_version` from the caller's resolved `base` CRC. + /// Handles three cases: + /// - Case 1: no base CRC available -> return None + /// - Case 2: base CRC at `end_version` -> return it as-is + /// - Case 3: stale base CRC older than `end_version` -> advance it to `end_version` when + /// `incremental_replay` permits, else fall back to normal log replay (return None) + pub(crate) fn try_build_crc_within_budget( + &self, + engine: &dyn Engine, + base: Option<&Arc>, + incremental_replay: IncrementalReplay, + ) -> DeltaResult, ProtocolMetadataSource)>> { + let Some(base) = base else { + return Ok(None); + }; + if base.version == self.end_version { + return Ok(Some((base.clone(), ProtocolMetadataSource::CrcAtTarget))); + } + if !incremental_replay.should_advance(base.version, self.end_version)? { + return Ok(None); + } + let advanced = self.build_crc_from_base(engine, base)?; + Ok(Some(( + Arc::new(advanced), + ProtocolMetadataSource::CrcAdvancedByReplay, + ))) + } + + /// Pick the latest CRC to use as an advance base: this segment's on-disk CRC or + /// `in_memory_base`, whichever is newer, falling back to `in_memory_base` on a failed on-disk + /// read. Drops a base below the checkpoint; returns None when no candidate remains. + pub(crate) fn pick_latest_base_crc( + &self, + engine: &dyn Engine, + in_memory_base: Option<&Arc>, + ) -> Option> { + let preferred_disk_crc = self + .listed + .latest_crc_file + .as_ref() + .filter(|f| in_memory_base.is_none_or(|m| f.version > m.version)); + preferred_disk_crc + .and_then(|f| read_crc_file_or_none(engine, f)) + .or_else(|| in_memory_base.cloned()) + .filter(|crc| { + self.checkpoint_version + .is_none_or(|ckpt| crc.version >= ckpt) + }) + } + + /// Read this segment's latest on-disk CRC (`latest_crc_file`), at whatever version it sits. + /// Returns None when there is no CRC file or the read fails. The returned CRC may be stale + /// (older than `end_version`). + pub(crate) fn read_latest_crc(&self, engine: &dyn Engine) -> Option> { + self.pick_latest_base_crc(engine, /* in_memory_base */ None) + } + + /// Produce a fresh `Crc` at `self.end_version` by reverse-replaying the commits in + /// `(base_crc.version, self.end_version]` and applying the resulting delta to + /// `base_crc` via [`Crc::apply`]. + #[instrument(name = "log_seg.build_crc_from_base", skip_all, err)] + pub(crate) fn build_crc_from_base( + &self, + engine: &dyn Engine, + base_crc: &Crc, + ) -> DeltaResult { + let seed_histogram = base_crc + .file_stats() + .and_then(|s| s.file_size_histogram()) + .map(|h| { + FileSizeHistogram::create_empty_with_boundaries(h.sorted_bin_boundaries().to_vec()) + }) + .transpose()?; + let delta = self.build_crc_delta_from_base(engine, base_crc.version, seed_histogram)?; + Ok(base_crc.clone().apply(delta, self.end_version)) + } + + /// Build a Complete base [`Crc`] at `checkpoint_version` from this segment's checkpoint files. + /// The tail commits are folded in separately by the caller as a [`CrcDelta`], so this reads + /// only the checkpoint. Returns `None` when the segment has no checkpoint, or if protocol or + /// metadata could not be recovered from it. + /// + /// File stats sum the checkpoint's AddFile actions (a reconciled checkpoint holds only live + /// files). Domain metadata and set transactions are + /// [`Complete`](DomainMetadataState::Complete) since a checkpoint is authoritative. + /// `in_commit_timestamp_opt` is left `None`: a checkpoint carries no `commitInfo`, so the + /// caller sets the ICT on the returned CRC afterward. + pub(crate) fn build_crc_from_checkpoint( + &self, + engine: &dyn Engine, + ) -> DeltaResult> { + let Some(version) = self.checkpoint_version else { + return Ok(None); + }; + // No commit boundaries here, so the delta stays incremental-safe. It covers the full + // table, which `into_complete_crc` turns into a Complete CRC. + let mut acc = CrcReplayAccumulator::new(Some(FileSizeHistogram::create_default())); + // Read only the checkpoint parquet plus any V2 sidecars via `create_checkpoint_stream`. + let batches = self + .create_checkpoint_stream( + engine, + CHECKPOINT_CRC_SCHEMA.clone(), + None, + None, + None, + None, + )? + .actions; + for batch in batches { + let batch = batch?; + let mut visitor = CheckpointCrcVisitor { acc: &mut acc }; + visitor.visit_rows_of(batch.actions())?; + } + Ok(acc.into_crc_delta().into_complete_crc(version)) + } + + /// Build a Complete [`Crc`] at `end_version` by reverse-replaying every commit in the segment, + /// for a segment with no CRC and no checkpoint to root at. The commits must run contiguously + /// from version 0. + /// + /// Returns `None` if protocol or metadata could not be recovered. File stats degrade to + /// [`Indeterminate`](FileStatsState::Indeterminate) if the replay is not incremental-safe. + pub(crate) fn build_crc_from_version_zero( + &self, + engine: &dyn Engine, + ) -> DeltaResult> { + require!( + self.checkpoint_version.is_none(), + Error::internal_error("build_crc_from_version_zero called with a checkpoint present") + ); + let Some(first) = self.listed.ascending_commit_files.first() else { + return Ok(None); + }; + // A log with no checkpoint must start at version 0; a higher first version means a table + // truncated without a checkpoint. + require!( + first.version == 0, + Error::generic(format!( + "Cannot build CRC: log has no checkpoint but its first commit is at version {} \ + (expected 0); the log appears truncated without a checkpoint", + first.version + )) + ); + let delta = self.replay_commits_into_crc_delta( + engine, + self.listed.ascending_commit_files.iter(), + Some(FileSizeHistogram::create_default()), + )?; + Ok(delta.into_complete_crc(self.end_version)) + } + + /// Build a `CrcDelta` covering commits `(base_version, self.end_version]` via reverse + /// log replay. `seed_histogram` is an empty histogram with the same bin boundaries as + /// the downstream base CRC, or `None` to skip histogram tracking on the delta. + /// + /// Errors if `base_version >= self.end_version` or if the segment is missing the + /// commit at `base_version + 1` (i.e. has a gap above `base_version`). + pub(crate) fn build_crc_delta_from_base( + &self, + engine: &dyn Engine, + base_version: Version, + seed_histogram: Option, + ) -> DeltaResult { + require!( + base_version < self.end_version, + Error::internal_error(format!( + "build_crc_delta_from_base: base_version ({}) must be strictly less \ + than end_version ({})", + base_version, self.end_version, + )) + ); + + let deltas: Vec<_> = self + .listed + .ascending_commit_files + .iter() + .filter(|c| c.version > base_version) + .collect(); + + let first_above = deltas.first().map(|c| c.version); + require!( + first_above == Some(base_version + 1), + Error::internal_error(format!( + "build_crc_delta_from_base: segment is missing commit {} \ + (lowest commit above base_version is {:?})", + base_version + 1, + first_above, + )) + ); + + self.replay_commits_into_crc_delta(engine, deltas.into_iter(), seed_histogram) + } + + /// Replay the given commits into a [`CrcDelta`]. The shared core of + /// [`Self::build_crc_delta_from_base`] and [`Self::build_crc_from_version_zero`]. + /// `ascending_commits` are taken oldest-first; `seed_histogram` is an empty histogram with the + /// downstream base's bin boundaries, or `None` to skip histogram tracking. + fn replay_commits_into_crc_delta<'a>( + &self, + engine: &dyn Engine, + ascending_commits: impl DoubleEndedIterator, + seed_histogram: Option, + ) -> DeltaResult { + // Replay newest-first: ICT capture reads from the newest commit only. + let locations: Vec = ascending_commits + .rev() + .map(|c| c.location.clone()) + .collect(); + let mut acc = CrcReplayAccumulator::new(seed_histogram); + let batches = + engine + .json_handler() + .read_json_files(&locations, REPLAY_SCHEMA.clone(), None)?; + + for batch_result in batches { + // Transient visitor borrows the shared accumulator for the duration of the + // batch; same pattern as `ActionReconciliationVisitor`. + let mut visitor = CommitCrcVisitor { acc: &mut acc }; + visitor.visit_rows_of(batch_result?.as_ref())?; + } + + // Run the per-commit invariant on the final (oldest) commit; no successor batch + // will trigger it. + acc.process_commit_file_end(); + + Ok(acc.into_crc_delta()) + } +} + +// ============================================================================ +// Accumulator +// ============================================================================ + +/// In-progress [`CrcDelta`] plus the scaffolding needed to build it correctly during reverse +/// replay. The visitor calls `process_batch_start` on each batch and the `on_*` methods on +/// each row. After all batches have been folded in and `process_commit_file_end` has run for +/// the final commit, [`Self::into_crc_delta`] returns the result. +struct CrcReplayAccumulator { + delta: CrcDelta, + + /// True while the visitor is still on the newest commit. Used to gate ICT capture + /// (only the newest commit contributes to [`CrcDelta::in_commit_timestamp`]). Cannot + /// be derived from `current_file_url` alone, since after the first batch the URL is + /// `Some` but we're still on the newest commit until a transition. + is_first_commit: bool, + + /// URL of the commit file currently being processed. Drives commit-boundary detection: + /// a different URL on a later batch means we've moved to an older commit. + current_file_url: Option, + + /// True if the current commit had at least one add/remove row. Combined with + /// `current_commit_saw_safe_op` for the per-commit invariant check. + current_commit_saw_file_action: bool, + + /// True if the current commit had a commitInfo row whose `operation` was in the + /// [`is_incremental_safe_operation`] safelist. If false at commit end and the commit + /// had file actions, we can't trust the file-stats delta and mark it unsafe. + current_commit_saw_safe_op: bool, +} + +impl CrcReplayAccumulator { + fn new(seed_histogram: Option) -> Self { + Self { + delta: CrcDelta { + is_incremental_safe: true, + file_stats: FileStatsDelta { + net_histogram: seed_histogram, + ..Default::default() + }, + ..Default::default() + }, + is_first_commit: true, + current_file_url: None, + current_commit_saw_file_action: false, + current_commit_saw_safe_op: false, + } + } + + fn process_batch_start(&mut self, batch_file_url: &str) { + if self.current_file_url.as_deref() == Some(batch_file_url) { + return; // same file, still inside the current commit + } + if self.current_file_url.is_some() { + // commit boundary: finalize the previous one, drop "newest" status + self.process_commit_file_end(); + self.is_first_commit = false; + } + self.current_file_url = Some(batch_file_url.to_owned()); + } + + fn process_commit_file_end(&mut self) { + // `is_incremental_safe` is one-way; once false, this check has nothing to set. + if !self.delta.is_incremental_safe { + return; + } + // File actions without a safe-classified operation: we can't trust file stats. + // Covers both "no commitInfo at all" and "commitInfo with no `operation` field". + if self.current_commit_saw_file_action && !self.current_commit_saw_safe_op { + warn!( + "CRC reverse-replay: commit at {} carried file actions but no safe-classified \ + operation; defaulting to non-incremental-safe", + self.current_file_url.as_deref().unwrap_or("?") + ); + self.delta.is_incremental_safe = false; + } + self.current_commit_saw_file_action = false; + self.current_commit_saw_safe_op = false; + } + + // ===== Row-level updates (also the seams used by `on_*` unit tests) ===== + + /// Called by the visitor once per commitInfo row (gated on operation or ict being + /// present). Handles both pieces: `operation` drives per-commit safety classification, + /// `ict` is captured into the delta from the newest commit only. + fn on_commit_info(&mut self, operation: Option<&str>, ict: Option) { + if let Some(op) = operation { + if is_incremental_safe_operation(op) { + self.current_commit_saw_safe_op = true; + } else { + warn!("CRC reverse-replay: non-incremental op {op}"); + self.delta.is_incremental_safe = false; + } + } + if self.is_first_commit { + self.delta.in_commit_timestamp = ict; + } + } + + fn on_add(&mut self, size: i64) -> DeltaResult<()> { + self.current_commit_saw_file_action = true; + // Once the delta is no longer incremental-safe, [`Crc::apply`] will transition the + // file-stats state to `Indeterminate` and discard the accumulated file stats and + // histogram. Stop accumulating; further math is wasted work. + if !self.delta.is_incremental_safe { + return Ok(()); + } + let fs = &mut self.delta.file_stats; + fs.gross_add_files += 1; + // TODO(#2676): a negative size errors here and fails the snapshot load; degrade to + // Indeterminate instead, like a missing remove size. + fs.gross_add_bytes += size_to_u64(size)?; + if let Some(hist) = fs.net_histogram.as_mut() { + hist.insert(size)?; + } + Ok(()) + } + + /// `size = None` means the remove row had a path but no size, which makes incremental + /// tracking impossible. + fn on_remove(&mut self, path: &str, size: Option) -> DeltaResult<()> { + self.current_commit_saw_file_action = true; + // Once the delta is no longer incremental-safe, [`Crc::apply`] will transition the + // file-stats state to `Indeterminate` and discard the accumulated file stats and + // histogram. Stop accumulating; further math is wasted work. + if !self.delta.is_incremental_safe { + return Ok(()); + } + match size { + Some(s) => { + let fs = &mut self.delta.file_stats; + fs.gross_remove_files += 1; + fs.gross_remove_bytes += size_to_u64(s)?; + if let Some(hist) = fs.net_histogram.as_mut() { + hist.remove(s)?; + } + } + None => { + warn!("CRC reverse-replay: remove action at {path} has missing size"); + self.delta.is_incremental_safe = false; + } + } + Ok(()) + } + + /// Tombstones (`removed=true`) are kept in the delta; [`Crc::apply`] consumes them as + /// removals from the base map. + fn on_domain_metadata(&mut self, domain: String, configuration: String, removed: bool) { + if let Entry::Vacant(e) = self.delta.domain_metadata.entry(domain.clone()) { + let dm = if removed { + DomainMetadata::remove(domain, configuration) + } else { + DomainMetadata::new(domain, configuration) + }; + e.insert(dm); + } + } + + fn on_set_transaction(&mut self, app_id: String, version: i64, last_updated: Option) { + if let Entry::Vacant(e) = self.delta.set_transactions.entry(app_id.clone()) { + e.insert(SetTransaction::new(app_id, version, last_updated)); + } + } + + /// Apply the shared columns (`shared`, laid out by [`shared_columns`] then the protocol and + /// metadata leaves) from row `i` to the delta. Each visitor reads its source-specific columns + /// (commit: `_file`/commitInfo/remove) and passes the trailing shared slice here so the shared + /// leaves live in one place. + fn apply_shared_columns<'a>( + &mut self, + i: usize, + shared: &[&'a dyn GetData<'a>], + ) -> DeltaResult<()> { + // `add.size` (required) marks an Add row. + if let Some(size) = shared[SHARED_COL_ADD_SIZE].get_opt(i, "add.size")? { + self.on_add(size)?; + } + if let Some(domain) = shared[SHARED_COL_DM_DOMAIN].get_opt(i, "domainMetadata.domain")? { + let configuration: String = + shared[SHARED_COL_DM_CONFIG].get(i, "domainMetadata.configuration")?; + let removed: bool = shared[SHARED_COL_DM_REMOVED].get(i, "domainMetadata.removed")?; + self.on_domain_metadata(domain, configuration, removed); + } + if let Some(app_id) = shared[SHARED_COL_TXN_APP_ID].get_opt(i, "txn.appId")? { + let version: i64 = shared[SHARED_COL_TXN_VERSION].get(i, "txn.version")?; + let last_updated: Option = + shared[SHARED_COL_TXN_LAST_UPDATED].get_opt(i, "txn.lastUpdated")?; + self.on_set_transaction(app_id, version, last_updated); + } + let leaves = &shared[N_SHARED_SINGLE_LEAF_COLS..]; + let n_protocol_leaves = PROTOCOL_LEAVES.as_ref().0.len(); + if self.delta.protocol.is_none() { + self.delta.protocol = visit_protocol_at(i, &leaves[..n_protocol_leaves])?; + } + if self.delta.metadata.is_none() { + self.delta.metadata = visit_metadata_at(i, &leaves[n_protocol_leaves..])?; + } + Ok(()) + } + + fn into_crc_delta(self) -> CrcDelta { + self.delta + } +} + +// ===== Shared column indices ===== +// Indices into the shared slice each visitor passes to +// [`CrcReplayAccumulator::apply_shared_columns`], laid out by [`shared_columns`]. +// The slice starts at the first shared column. +const SHARED_COL_ADD_SIZE: usize = 0; +const SHARED_COL_DM_DOMAIN: usize = 1; +const SHARED_COL_DM_CONFIG: usize = 2; +const SHARED_COL_DM_REMOVED: usize = 3; +const SHARED_COL_TXN_APP_ID: usize = 4; +const SHARED_COL_TXN_VERSION: usize = 5; +const SHARED_COL_TXN_LAST_UPDATED: usize = 6; +/// The single-leaf shared columns end here; the protocol and metadata leaves follow them in the +/// shared slice. +const N_SHARED_SINGLE_LEAF_COLS: usize = SHARED_COL_TXN_LAST_UPDATED + 1; + +/// The columns every source carries (add file, domain metadata, set transaction), in the order +/// [`CrcReplayAccumulator::apply_shared_columns`] assumes. Each visitor appends these to its +/// source-specific columns. +fn shared_columns() -> Vec<(DataType, ColumnName)> { + vec![ + (DataType::LONG, column_name!("add.size")), + (DataType::STRING, column_name!("domainMetadata.domain")), + ( + DataType::STRING, + column_name!("domainMetadata.configuration"), + ), + (DataType::BOOLEAN, column_name!("domainMetadata.removed")), + (DataType::STRING, column_name!("txn.appId")), + (DataType::LONG, column_name!("txn.version")), + (DataType::LONG, column_name!("txn.lastUpdated")), + ] +} + +/// Append the protocol and metadata leaf columns (in that order) to a visitor's fixed columns, +/// shared by both replay visitors' `selected_column_names_and_types`. +fn append_protocol_metadata_leaves( + fixed: Vec<(DataType, ColumnName)>, +) -> (Vec, Vec) { + let (mut types, mut names): (Vec<_>, Vec<_>) = fixed.into_iter().unzip(); + for leaves in [&*PROTOCOL_LEAVES, &*METADATA_LEAVES] { + let (leaf_names, leaf_types) = leaves.as_ref(); + names.extend_from_slice(leaf_names); + types.extend_from_slice(leaf_types); + } + (names, types) +} + +/// Validate that `getters` carries exactly the columns a replay visitor projects: `n_fixed` +/// fixed columns plus the protocol and metadata leaves. `visitor_name` names the visitor in the +/// error. +fn check_visitor_getters( + getters: &[&dyn GetData<'_>], + n_fixed: usize, + visitor_name: &str, +) -> DeltaResult<()> { + let n_protocol_leaves = PROTOCOL_LEAVES.as_ref().0.len(); + let n_metadata_leaves = METADATA_LEAVES.as_ref().0.len(); + require!( + getters.len() == n_fixed + n_protocol_leaves + n_metadata_leaves, + Error::internal_error(format!( + "Wrong number of {visitor_name} getters: {}", + getters.len() + )) + ); + Ok(()) +} + +// ============================================================================ +// Visitor +// ============================================================================ + +// ===== Visitor column indices ===== +// Indices into the column list returned by [`CommitCrcVisitor::selected_column_names_and_types`]. +const COL_FILE: usize = 0; +const COL_OP: usize = 1; +const COL_ICT: usize = 2; +const COL_REMOVE_PATH: usize = 3; +const COL_REMOVE_SIZE: usize = 4; +/// Source-specific columns end here; the shared columns follow. +const N_CRC_SPECIFIC_COLS: usize = COL_REMOVE_SIZE + 1; +const N_FIXED_COLS: usize = N_CRC_SPECIFIC_COLS + N_SHARED_SINGLE_LEAF_COLS; + +/// Thin shim that pulls leaf values from `getters` and forwards them to the accumulator's +/// `on_*` methods. All behavior lives in [`CrcReplayAccumulator`]. +struct CommitCrcVisitor<'a> { + acc: &'a mut CrcReplayAccumulator, +} + +impl RowVisitor for CommitCrcVisitor<'_> { + fn selected_column_names_and_types(&self) -> (&'static [ColumnName], &'static [DataType]) { + static NAMES_AND_TYPES: LazyLock = LazyLock::new(|| { + let mut fixed = vec![ + (DataType::STRING, column_name!("_file")), + (DataType::STRING, column_name!("commitInfo.operation")), + (DataType::LONG, column_name!("commitInfo.inCommitTimestamp")), + (DataType::STRING, column_name!("remove.path")), + (DataType::LONG, column_name!("remove.size")), + ]; + fixed.extend(shared_columns()); + append_protocol_metadata_leaves(fixed).into() + }); + NAMES_AND_TYPES.as_ref() + } + + fn visit<'a>(&mut self, row_count: usize, getters: &[&'a dyn GetData<'a>]) -> DeltaResult<()> { + check_visitor_getters(getters, N_FIXED_COLS, "CommitCrcVisitor")?; + if row_count == 0 { + return Ok(()); + } + // `_file` is constant across all rows of a batch per the JsonHandler contract. Read + // once from row 0 and signal a potential file (commit) transition. + let file_url: String = getters[COL_FILE].get(0, "_file")?; + self.acc.process_batch_start(&file_url); + + for i in 0..row_count { + let operation: Option = getters[COL_OP].get_opt(i, "commitInfo.operation")?; + let ict: Option = getters[COL_ICT].get_opt(i, "commitInfo.inCommitTimestamp")?; + if operation.is_some() || ict.is_some() { + self.acc.on_commit_info(operation.as_deref(), ict); + } + + let remove_path: Option = getters[COL_REMOVE_PATH].get_opt(i, "remove.path")?; + if let Some(path) = remove_path { + let remove_size: Option = + getters[COL_REMOVE_SIZE].get_opt(i, "remove.size")?; + self.acc.on_remove(&path, remove_size)?; + } + + self.acc + .apply_shared_columns(i, &getters[N_CRC_SPECIFIC_COLS..])?; + } + Ok(()) + } +} + +// ============================================================================ +// Checkpoint base construction +// ============================================================================ + +/// Action schema for reading a checkpoint into a base [`Crc`], projecting only the leaves the +/// accumulator needs. `add.size` is the only Add leaf read, and it is required, so its presence +/// marks an Add row (a checkpoint Add missing `size` errors at read time). A checkpoint has no +/// `remove` or `commitInfo` to project. +#[allow(clippy::expect_used)] +static CHECKPOINT_CRC_SCHEMA: LazyLock = LazyLock::new(|| { + Arc::new(schema! { + nullable ADD_NAME: { not_null "size": LONG }, + (&PROTOCOL_FIELD), + (&METADATA_FIELD), + (&SET_TRANSACTION_FIELD), + (&DOMAIN_METADATA_FIELD), + }) +}); + +// A checkpoint has no source-specific columns, so its projection is the shared columns. + +/// Pulls leaf values from a checkpoint batch into the shared [`CrcReplayAccumulator`]. +struct CheckpointCrcVisitor<'a> { + acc: &'a mut CrcReplayAccumulator, +} + +impl RowVisitor for CheckpointCrcVisitor<'_> { + fn selected_column_names_and_types(&self) -> (&'static [ColumnName], &'static [DataType]) { + static NAMES_AND_TYPES: LazyLock = + LazyLock::new(|| append_protocol_metadata_leaves(shared_columns()).into()); + NAMES_AND_TYPES.as_ref() + } + + fn visit<'a>(&mut self, row_count: usize, getters: &[&'a dyn GetData<'a>]) -> DeltaResult<()> { + check_visitor_getters(getters, N_SHARED_SINGLE_LEAF_COLS, "CheckpointCrcVisitor")?; + for i in 0..row_count { + self.acc.apply_shared_columns(i, getters)?; + } + Ok(()) + } +} + +// ============================================================================ +// Tests +// ============================================================================ + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + use std::sync::Arc; + + use test_utils::{add_commit, assert_result_error_with_message}; + + use super::*; + use crate::crc::{DomainMetadataState, FileStats, FileStatsState, SetTransactionState}; + use crate::engine::sync::SyncEngine; + use crate::object_store::memory::InMemory; + use crate::table_features::TableFeature; + + // ===== Unit tests on `on_*` methods ===== + + // ===== commitInfo ===== + + #[rstest::rstest] + #[case::safe("WRITE", true)] + #[case::unsafe_op("ANALYZE STATS", false)] + fn on_commit_info_classifies_operation(#[case] op: &str, #[case] is_safe: bool) { + let mut acc = CrcReplayAccumulator::new(None); + acc.on_commit_info(Some(op), None); + assert_eq!(acc.current_commit_saw_safe_op, is_safe); + assert_eq!(acc.delta.is_incremental_safe, is_safe); + } + + #[test] + fn on_commit_info_ict_only_no_operation_does_not_mark_saw_safe_op() { + let mut acc = CrcReplayAccumulator::new(None); + acc.on_commit_info(None, Some(1234)); + assert!(!acc.current_commit_saw_safe_op); + assert!(acc.delta.is_incremental_safe); + assert_eq!(acc.delta.in_commit_timestamp, Some(1234)); + } + + #[test] + fn on_commit_info_captures_ict_only_on_first_commit() { + let mut acc = CrcReplayAccumulator::new(None); + acc.process_batch_start("v2.json"); + acc.on_commit_info(Some("WRITE"), Some(2000)); + assert_eq!(acc.delta.in_commit_timestamp, Some(2000)); + acc.process_batch_start("v1.json"); + acc.on_commit_info(Some("WRITE"), Some(1000)); + assert_eq!(acc.delta.in_commit_timestamp, Some(2000)); + } + + #[test] + fn on_commit_info_first_commit_none_ict_does_not_get_overwritten_by_older() { + let mut acc = CrcReplayAccumulator::new(None); + acc.process_batch_start("v2.json"); + acc.on_commit_info(Some("WRITE"), None); + assert_eq!(acc.delta.in_commit_timestamp, None); + acc.process_batch_start("v1.json"); + acc.on_commit_info(Some("WRITE"), Some(1000)); + assert_eq!(acc.delta.in_commit_timestamp, None); + } + + // ===== add ===== + + #[test] + fn on_add_increments_files_and_bytes() { + let mut acc = CrcReplayAccumulator::new(None); + acc.on_add(100).unwrap(); + acc.on_add(200).unwrap(); + assert_eq!(acc.delta.file_stats.net_files(), 2); + assert_eq!(acc.delta.file_stats.net_bytes(), 300); + assert!(acc.delta.is_incremental_safe); + } + + // ===== remove ===== + + #[test] + fn on_remove_with_size_decrements_files_and_bytes() { + let mut acc = CrcReplayAccumulator::new(None); + acc.on_remove("p", Some(50)).unwrap(); + assert_eq!(acc.delta.file_stats.net_files(), -1); + assert_eq!(acc.delta.file_stats.net_bytes(), -50); + assert!(acc.delta.is_incremental_safe); + } + + #[test] + fn on_remove_missing_size_trips_is_incremental_safe() { + let mut acc = CrcReplayAccumulator::new(None); + acc.on_remove("p", None).unwrap(); + assert!(!acc.delta.is_incremental_safe); + assert!(acc.current_commit_saw_file_action); + } + + // ===== domainMetadata ===== + + #[test] + fn on_domain_metadata_first_seen_in_reverse_wins() { + let mut acc = CrcReplayAccumulator::new(None); + acc.on_domain_metadata("d".into(), "new".into(), false); + acc.on_domain_metadata("d".into(), "old".into(), false); + assert_eq!(acc.delta.domain_metadata["d"].configuration(), "new"); + assert!(!acc.delta.domain_metadata["d"].is_removed()); + } + + #[test] + fn on_domain_metadata_tombstone_is_kept() { + let mut acc = CrcReplayAccumulator::new(None); + acc.on_domain_metadata("d".into(), "v".into(), true); + assert!(acc.delta.domain_metadata["d"].is_removed()); + } + + // ===== txn ===== + + #[test] + fn on_set_transaction_first_seen_in_reverse_wins() { + let mut acc = CrcReplayAccumulator::new(None); + acc.on_set_transaction("a".into(), 99, Some(123)); + acc.on_set_transaction("a".into(), 1, Some(0)); + assert_eq!(acc.delta.set_transactions["a"].version, 99); + } + + // ===== auxiliary ===== + + #[test] + fn accumulator_with_seed_histogram_inserts_into_seeded_bin() { + let seed = FileSizeHistogram::create_empty_with_boundaries(vec![0, 200, 1000]).unwrap(); + let mut acc = CrcReplayAccumulator::new(Some(seed)); + acc.on_add(150).unwrap(); + let hist = acc.delta.file_stats.net_histogram.as_ref().unwrap(); + assert_eq!(hist.sorted_bin_boundaries(), &[0, 200, 1000]); + assert_eq!(hist.file_counts()[0], 1); + assert_eq!(hist.total_bytes()[0], 150); + } + + #[test] + fn accumulator_with_no_seed_histogram_keeps_delta_histogram_none() { + let mut acc = CrcReplayAccumulator::new(None); + acc.on_add(150).unwrap(); + assert!(acc.delta.file_stats.net_histogram.is_none()); + } + + #[test] + fn into_crc_delta_transfers_accumulated_state() { + let mut acc = CrcReplayAccumulator::new(None); + acc.on_add(42).unwrap(); + let delta = acc.into_crc_delta(); + assert_eq!(delta.file_stats.net_files(), 1); + assert_eq!(delta.file_stats.net_bytes(), 42); + } + + #[test] + fn visitor_schema_length_matches_column_indices() { + let mut acc = CrcReplayAccumulator::new(None); + let visitor = CommitCrcVisitor { acc: &mut acc }; + let (names, types) = visitor.selected_column_names_and_types(); + let expected = + N_FIXED_COLS + PROTOCOL_LEAVES.as_ref().0.len() + METADATA_LEAVES.as_ref().0.len(); + assert_eq!(names.len(), expected); + assert_eq!(types.len(), expected); + } + + #[test] + fn checkpoint_visitor_schema_length_matches_column_indices() { + let mut acc = CrcReplayAccumulator::new(None); + let visitor = CheckpointCrcVisitor { acc: &mut acc }; + let (names, types) = visitor.selected_column_names_and_types(); + let expected = N_SHARED_SINGLE_LEAF_COLS + + PROTOCOL_LEAVES.as_ref().0.len() + + METADATA_LEAVES.as_ref().0.len(); + assert_eq!(names.len(), expected); + assert_eq!(types.len(), expected); + } + + // ===== Commit-boundary state machine: direct accumulator tests ===== + // + // `SyncEngine` emits one batch per file, so multi-batch-per-file scenarios are only + // reachable by driving the accumulator directly. + + #[test] + fn per_commit_invariant_holds_when_file_action_and_commit_info_split_across_batches_of_one_file( + ) { + let mut acc = CrcReplayAccumulator::new(None); + acc.process_batch_start("v1.json"); + acc.on_add(0).unwrap(); + acc.process_batch_start("v1.json"); + acc.on_commit_info(Some("WRITE"), None); + acc.process_commit_file_end(); + assert!(acc.delta.is_incremental_safe); + } + + #[test] + fn per_commit_invariant_trips_when_file_action_has_no_safe_op_across_batches() { + let mut acc = CrcReplayAccumulator::new(None); + acc.process_batch_start("v1.json"); + acc.on_add(0).unwrap(); + acc.process_batch_start("v1.json"); + acc.process_commit_file_end(); + assert!(!acc.delta.is_incremental_safe); + } + + #[test] + fn per_commit_invariant_trips_when_file_action_has_commit_info_but_no_operation() { + let mut acc = CrcReplayAccumulator::new(None); + acc.process_batch_start("v1.json"); + acc.on_add(100).unwrap(); + acc.on_commit_info(None, Some(42)); + acc.process_commit_file_end(); + assert!(!acc.delta.is_incremental_safe); + } + + #[test] + fn is_first_commit_stays_true_across_batches_of_same_file() { + let mut acc = CrcReplayAccumulator::new(None); + acc.process_batch_start("v2.json"); + acc.process_batch_start("v2.json"); + acc.process_batch_start("v2.json"); + assert!(acc.is_first_commit); + } + + #[test] + fn is_first_commit_becomes_false_after_file_transition() { + let mut acc = CrcReplayAccumulator::new(None); + acc.process_batch_start("v2.json"); + acc.process_batch_start("v1.json"); + assert!(!acc.is_first_commit); + } + + // ===== End-to-end smoke ===== + // + // Tests all aspects of the full pipeline via the visitor. + + #[tokio::test] + async fn end_to_end_smoke_full_coverage() { + let store = Arc::new(InMemory::new()); + let engine = SyncEngine::new_with_store(store.clone()); + let root = "memory:///t/"; + + // v0: bootstrap. Protocol already supports `domainMetadata` and `inCommitTimestamp` + // so the v1 DM action and v2 commitInfo with ICT are well-formed. v0 is outside + // the replay range; the segment covers (0, 2]. + add_commit(root, store.as_ref(), 0, r#" +{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":[],"writerFeatures":["domainMetadata","inCommitTimestamp"]}} +{"metaData":{"id":"id-0","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[]}","partitionColumns":[],"configuration":{},"createdTime":0}} +"#.to_string()).await.unwrap(); + + // v1: DM add, txn add, four file adds spanning histogram bins 0 (sizes 100, 200), + // 1 (size 10000), and 2 (size 20000). + add_commit( + root, + store.as_ref(), + 1, + r#" +{"add":{"path":"a","partitionValues":{},"size":100,"modificationTime":1,"dataChange":true}} +{"add":{"path":"b","partitionValues":{},"size":200,"modificationTime":1,"dataChange":true}} +{"add":{"path":"c","partitionValues":{},"size":10000,"modificationTime":1,"dataChange":true}} +{"add":{"path":"d","partitionValues":{},"size":20000,"modificationTime":1,"dataChange":true}} +{"domainMetadata":{"domain":"keep","configuration":"cfg","removed":false}} +{"txn":{"appId":"app1","version":1,"lastUpdated":1}} +{"commitInfo":{"timestamp":1,"operation":"WRITE"}} +"# + .to_string(), + ) + .await + .unwrap(); + + // v2 (newest): protocol upgrade that adds `rowTracking` on top of v0's features, + // new metadata, DM tombstone, second txn, one remove (bin 0), commitInfo with ICT. + add_commit(root, store.as_ref(), 2, r#" +{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":[],"writerFeatures":["domainMetadata","inCommitTimestamp","rowTracking"]}} +{"metaData":{"id":"id-2","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[]}","partitionColumns":[],"configuration":{"delta.enableRowTracking":"true"},"createdTime":2}} +{"remove":{"path":"a","deletionTimestamp":2,"dataChange":true,"size":100}} +{"domainMetadata":{"domain":"drop","configuration":"","removed":true}} +{"txn":{"appId":"app2","version":5,"lastUpdated":2}} +{"commitInfo":{"timestamp":2,"operation":"WRITE","inCommitTimestamp":9999}} +"#.to_string()).await.unwrap(); + + let log_root = url::Url::parse(root).unwrap().join("_delta_log/").unwrap(); + let segment = LogSegment::for_snapshot_impl( + engine.storage_handler().as_ref(), + log_root, + vec![], + None, + Some(2), + ) + .unwrap(); + + // A seed histogram is required for the replay to track histogram bin updates. + let base = Crc { + file_stats_state: FileStatsState::Complete(FileStats { + num_files: 0, + table_size_bytes: 0, + file_size_histogram: Some(FileSizeHistogram::create_default()), + }), + domain_metadata_state: DomainMetadataState::Complete(HashMap::new()), + set_transaction_state: SetTransactionState::Complete(HashMap::new()), + ..Default::default() + }; + let crc = segment.build_crc_from_base(&engine, &base).unwrap(); + + // Newest-wins: v2's upgraded protocol (now carries `rowTracking` on top of v0's + // features), v2's metadata, v2's ICT. + assert_eq!(crc.version, 2); + assert_eq!(crc.protocol.min_writer_version(), 7); + assert!(crc.protocol.has_table_feature(&TableFeature::RowTracking)); + assert!(crc + .protocol + .has_table_feature(&TableFeature::InCommitTimestamp)); + assert!(crc + .protocol + .has_table_feature(&TableFeature::DomainMetadata)); + assert_eq!(crc.metadata.id(), "id-2"); + assert_eq!(crc.in_commit_timestamp_opt, Some(9999)); + + // DM: "keep" inserted from v1; "drop" tombstone applied (no-op on empty base). + let dm = crc.domain_metadata_state.expect_complete(); + assert_eq!(dm.get("keep").unwrap().configuration(), "cfg"); + assert!(!dm.contains_key("drop")); + + // Both txns upserted across the two commits. + let txn = crc.set_transaction_state.expect_complete(); + assert_eq!(txn.get("app1").unwrap().version, 1); + assert_eq!(txn.get("app2").unwrap().version, 5); + + // 4 adds spanning bins 0/1/2 minus 1 remove in bin 0: + // bin 0: +2 files (100 + 200 = 300 bytes) - 1 file (100 bytes) = 1 file, 200 bytes + // bin 1: +1 file (10000 bytes) + // bin 2: +1 file (20000 bytes) + // Totals: 3 files, 30200 bytes. + let stats = crc.file_stats().unwrap(); + assert_eq!(stats.num_files(), 3); + assert_eq!(stats.table_size_bytes(), 30_200); + let hist = stats.file_size_histogram().unwrap(); + assert_eq!(hist.file_counts()[0], 1); + assert_eq!(hist.total_bytes()[0], 200); + assert_eq!(hist.file_counts()[1], 1); + assert_eq!(hist.total_bytes()[1], 10_000); + assert_eq!(hist.file_counts()[2], 1); + assert_eq!(hist.total_bytes()[2], 20_000); + } + + #[tokio::test] + async fn build_crc_delta_from_base_errors_when_base_version_geq_end_version() { + let store = Arc::new(InMemory::new()); + let engine = SyncEngine::new_with_store(store.clone()); + let root = "memory:///t/"; + add_commit( + root, + store.as_ref(), + 0, + r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":1}}"#.to_string(), + ) + .await + .unwrap(); + let log_root = url::Url::parse(root).unwrap().join("_delta_log/").unwrap(); + let segment = LogSegment::for_snapshot_impl( + engine.storage_handler().as_ref(), + log_root, + vec![], + None, + Some(0), + ) + .unwrap(); + for base in [0, 5] { + assert_result_error_with_message( + segment.build_crc_delta_from_base(&engine, base, None), + "must be strictly less than end_version", + ); + } + } + + #[tokio::test] + async fn build_crc_from_version_zero_no_checkpoint_first_commit_nonzero_errors() { + let store = Arc::new(InMemory::new()); + let engine = SyncEngine::new_with_store(store.clone()); + let root = "memory:///t/"; + add_commit( + root, + store.as_ref(), + 1, + r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":1}}"#.to_string(), + ) + .await + .unwrap(); + let log_root = url::Url::parse(root).unwrap().join("_delta_log/").unwrap(); + let segment = LogSegment::for_snapshot_impl( + engine.storage_handler().as_ref(), + log_root, + vec![], + None, + Some(1), + ) + .unwrap(); + assert_result_error_with_message( + segment.build_crc_from_version_zero(&engine), + "log appears truncated without a checkpoint", + ); + } +} diff --git a/kernel/src/log_segment/crc_tests.rs b/kernel/src/log_segment/crc_tests.rs index 83fe958a6c..e08ea076f1 100644 --- a/kernel/src/log_segment/crc_tests.rs +++ b/kernel/src/log_segment/crc_tests.rs @@ -6,17 +6,23 @@ use std::collections::HashMap; use std::sync::Arc; -use serde_json::json; +use rstest::rstest; +use serde_json::{json, Value}; +use test_utils::{assert_result_error_with_message, delta_path_for_version}; use url::Url; -use crate::actions::{CommitInfo, Format, Metadata, Protocol}; -use crate::engine::default::executor::tokio::TokioBackgroundExecutor; -use crate::engine::default::{DefaultEngine, DefaultEngineBuilder}; +use super::LogSegment; +use crate::actions::{DomainMetadata, Format, Metadata, Protocol, SetTransaction}; +use crate::crc::{ + try_read_crc_file, Crc, DomainMetadataState, FileSizeHistogram, SetTransactionState, +}; +use crate::engine::sync::SyncEngine; use crate::object_store::memory::InMemory; -use crate::object_store::ObjectStore; -use crate::Snapshot; - -use test_utils::{assert_result_error_with_message, delta_path_for_version}; +use crate::object_store::ObjectStoreExt as _; +use crate::path::ParsedLogPath; +use crate::snapshot::{IncrementalReplay, SnapshotBuilder, SnapshotRef}; +use crate::utils::FoldWithOption as _; +use crate::{DeltaResult, Engine, Snapshot, Version}; // ============================================================================ // Expected values @@ -24,11 +30,14 @@ use test_utils::{assert_result_error_with_message, delta_path_for_version}; const SCHEMA_STRING: &str = r#"{"type":"struct","fields":[{"name":"id","type":"integer","nullable":true,"metadata":{}},{"name":"val","type":"string","nullable":true,"metadata":{}}]}"#; -fn protocol_v2() -> Protocol { +const COMMIT_INFO_TIMESTAMP: i64 = 1587968586154; +const DEFAULT_OPERATION: &str = "TEST_OP"; + +fn protocol_a() -> Protocol { Protocol::try_new_modern(["v2Checkpoint"], ["v2Checkpoint"]).unwrap() } -fn protocol_v2_dv() -> Protocol { +fn protocol_b() -> Protocol { Protocol::try_new_modern( ["v2Checkpoint", "deletionVectors"], ["v2Checkpoint", "deletionVectors"], @@ -36,7 +45,7 @@ fn protocol_v2_dv() -> Protocol { .unwrap() } -fn protocol_v2_dv_ntz() -> Protocol { +fn protocol_c() -> Protocol { Protocol::try_new_modern( ["v2Checkpoint", "deletionVectors", "timestampNtz"], ["v2Checkpoint", "deletionVectors", "timestampNtz"], @@ -44,14 +53,8 @@ fn protocol_v2_dv_ntz() -> Protocol { .unwrap() } -fn protocol_v2_ict() -> Protocol { - Protocol::try_new( - 3, - 7, - Some(["v2Checkpoint"]), - Some(["v2Checkpoint", "inCommitTimestamp"]), - ) - .unwrap() +fn protocol_ict() -> Protocol { + Protocol::try_new_modern(["v2Checkpoint"], ["v2Checkpoint", "inCommitTimestamp"]).unwrap() } fn metadata_a() -> Metadata { @@ -96,53 +99,116 @@ fn metadata_ict() -> Metadata { ) } -fn commit_info() -> CommitInfo { - CommitInfo { - timestamp: Some(1587968586154), - operation: Some("WRITE".to_string()), - ..Default::default() +// ============================================================================ +// Action JSON helpers +// ============================================================================ + +fn create_table_actions() -> Vec { + vec![ + commit_info("CREATE", None), + protocol(protocol_a()), + metadata(metadata_a()), + ] +} + +fn create_table_actions_with_ict() -> Vec { + vec![ + commit_info("CREATE", None), + protocol(protocol_ict()), + metadata(metadata_a()), + ] +} + +/// A commitInfo action with the given operation and optional in-commit timestamp. +fn commit_info(op: &str, ict: Option) -> serde_json::Value { + let mut obj = json!({"commitInfo": {"timestamp": COMMIT_INFO_TIMESTAMP, "operation": op}}); + if let Some(ict) = ict { + obj["commitInfo"]["inCommitTimestamp"] = json!(ict); } + obj } -// ============================================================================ -// JSON builders -// ============================================================================ +fn protocol(p: Protocol) -> serde_json::Value { + json!({"protocol": serde_json::to_value(&p).unwrap()}) +} -const V2_CKPT_SUFFIX: &str = "checkpoint.00000000-0000-0000-0000-000000000000.json"; +fn metadata(m: Metadata) -> serde_json::Value { + json!({"metaData": serde_json::to_value(&m).unwrap()}) +} -fn protocol_action_json(protocol: &Protocol) -> serde_json::Value { - json!({"protocol": serde_json::to_value(protocol).unwrap()}) +fn add(path: &str, size: i64) -> serde_json::Value { + json!({"add": { + "path": path, + "size": size, + "partitionValues": {}, + "modificationTime": 0, + "dataChange": true, + }}) } -fn metadata_action_json(metadata: &Metadata) -> serde_json::Value { - json!({"metaData": serde_json::to_value(metadata).unwrap()}) +fn remove(path: &str, size: Option) -> serde_json::Value { + let mut obj = json!({"remove": { + "path": path, + "dataChange": true, + "deletionTimestamp": 0, + }}); + if let Some(s) = size { + obj["remove"]["size"] = json!(s); + } + obj } -fn commit_info_json() -> serde_json::Value { - json!({"commitInfo": serde_json::to_value(commit_info()).unwrap()}) +fn domain_metadata(domain: &str, configuration: &str) -> serde_json::Value { + json!({"domainMetadata": { + "domain": domain, + "configuration": configuration, + "removed": false, + }}) } -fn commit_info_json_with_ict(ict: i64) -> serde_json::Value { - json!({"commitInfo": { - "timestamp": 1587968586154i64, - "operation": "WRITE", - "inCommitTimestamp": ict, +fn domain_metadata_tombstone(domain: &str) -> serde_json::Value { + json!({"domainMetadata": { + "domain": domain, + "configuration": "", + "removed": true, }}) } -fn crc_json(protocol: &Protocol, metadata: &Metadata, ict: Option) -> serde_json::Value { - let mut v = json!({ - "tableSizeBytes": 0, - "numFiles": 0, +fn set_txn(app_id: &str, version: i64, last_updated: Option) -> serde_json::Value { + let mut obj = json!({"txn": {"appId": app_id, "version": version}}); + if let Some(lu) = last_updated { + obj["txn"]["lastUpdated"] = json!(lu); + } + obj +} + +fn crc_json( + protocol: &Protocol, + metadata: &Metadata, + ict: Option, + file_sizes: &[i64], +) -> serde_json::Value { + let total_bytes: i64 = file_sizes.iter().sum(); + let num_files = file_sizes.len() as i64; + let mut obj = json!({ + "tableSizeBytes": total_bytes, + "numFiles": num_files, "numMetadata": 1, "numProtocol": 1, "metadata": serde_json::to_value(metadata).unwrap(), "protocol": serde_json::to_value(protocol).unwrap(), }); + if !file_sizes.is_empty() { + let mut hist = FileSizeHistogram::create_default(); + for &s in file_sizes { + hist.insert(s).unwrap(); + } + obj["fileSizeHistogram"] = serde_json::to_value(&hist).unwrap(); + } if let Some(ict) = ict { - v["inCommitTimestampOpt"] = json!(ict); + obj["inCommitTimestampOpt"] = json!(ict); } - v + obj } // ============================================================================ @@ -156,23 +222,20 @@ enum Op { protocol: Protocol, metadata: Metadata, }, - Delta(u64), - DeltaWithPM { - version: u64, - protocol: Option, - metadata: Option, - }, - DeltaWithIct { + Commit { version: u64, - ict: i64, + actions: Vec, }, Crc { version: u64, protocol: Protocol, metadata: Metadata, ict: Option, + file_sizes: Vec, + }, + CorruptCrc { + version: u64, }, - CorruptCrc(u64), } /// Declarative test builder: accumulate log operations, then build and assert. @@ -185,7 +248,6 @@ impl CrcReadTest { Self { ops: vec![] } } - /// Write a V2 checkpoint at the given version. fn v2_checkpoint(mut self, version: u64, protocol: Protocol, metadata: Metadata) -> Self { self.ops.push(Op::V2Checkpoint { version, @@ -195,133 +257,207 @@ impl CrcReadTest { self } - /// Write a plain delta (commitInfo only, no protocol or metadata). - fn delta(mut self, version: u64) -> Self { - self.ops.push(Op::Delta(version)); - self - } - - /// Write a delta with optional protocol and/or metadata overrides. - fn delta_with_p_m( + /// Write a commit file containing exactly the given actions, in order. + fn commit( mut self, version: u64, - protocol: impl Into>, - metadata: impl Into>, + actions: impl IntoIterator, ) -> Self { - self.ops.push(Op::DeltaWithPM { - version, - protocol: protocol.into(), - metadata: metadata.into(), - }); + let actions: Vec = actions.into_iter().collect(); + self.ops.push(Op::Commit { version, actions }); self } - /// Write a delta with an in-commit timestamp in commitInfo. - fn delta_with_ict(mut self, version: u64, ict: i64) -> Self { - self.ops.push(Op::DeltaWithIct { version, ict }); - self + fn crc( + self, + version: u64, + protocol: Protocol, + metadata: Metadata, + ict: impl Into>, + ) -> Self { + self.crc_with_files(version, protocol, metadata, ict, &[]) } - /// Write a CRC file with the given protocol, metadata, and optional ICT. - fn crc( + /// Write a CRC file whose on-disk `fileSizeHistogram` is built from `file_sizes`. + fn crc_with_files( mut self, version: u64, protocol: Protocol, metadata: Metadata, ict: impl Into>, + file_sizes: &[i64], ) -> Self { self.ops.push(Op::Crc { version, protocol, metadata, ict: ict.into(), + file_sizes: file_sizes.to_vec(), }); self } - /// Write a corrupt CRC file. fn corrupt_crc(mut self, version: u64) -> Self { - self.ops.push(Op::CorruptCrc(version)); + self.ops.push(Op::CorruptCrc { version }); self } /// Execute all operations, returning a built test that can be asserted against. async fn build(self) -> BuiltCrcTest { + Self::validate_ops(&self.ops); + let store = Arc::new(InMemory::new()); let url = Url::parse("memory:///").unwrap(); - let engine = DefaultEngineBuilder::new(store.clone()).build(); + let engine = SyncEngine::new_with_store(store.clone()); for op in self.ops { match op { - Op::V2Checkpoint { - version: v, - ref protocol, - ref metadata, - } => { - let content = format!( - "{}\n{}\n{}", - protocol_action_json(protocol), - metadata_action_json(metadata), - json!({"checkpointMetadata": {"version": 2}}) - ); - put(&store, v, V2_CKPT_SUFFIX, &content).await; - } - Op::Delta(v) => { - put(&store, v, "json", &commit_info_json().to_string()).await; - } - Op::DeltaWithPM { - version: v, - protocol, - metadata, - } => { - let mut lines = vec![commit_info_json().to_string()]; - if let Some(ref p) = protocol { - lines.push(protocol_action_json(p).to_string()); - } - if let Some(ref m) = metadata { - lines.push(metadata_action_json(m).to_string()); - } - put(&store, v, "json", &lines.join("\n")).await; - } - Op::DeltaWithIct { version: v, ict } => { - put( - &store, - v, - "json", - &commit_info_json_with_ict(ict).to_string(), - ) - .await; + Op::Commit { version, actions } => { + let content = actions + .iter() + .map(|a| a.to_string()) + .collect::>() + .join("\n"); + put(&store, version, "json", &content).await; } Op::Crc { - version: v, + version, ref protocol, ref metadata, ict, + ref file_sizes, } => { put( &store, - v, + version, "crc", - &crc_json(protocol, metadata, ict).to_string(), + &crc_json(protocol, metadata, ict, file_sizes).to_string(), ) .await; } - Op::CorruptCrc(v) => { - put(&store, v, "crc", "CORRUPT_CRC_DATA").await; + Op::CorruptCrc { version } => { + put(&store, version, "crc", "CORRUPT_CRC_DATA").await; + } + Op::V2Checkpoint { + version, + protocol: p, + metadata: m, + } => { + const V2_CKPT_SUFFIX: &str = + "checkpoint.00000000-0000-0000-0000-000000000000.json"; + let content = format!( + "{}\n{}\n{}", + protocol(p), + metadata(m), + json!({"checkpointMetadata": {"version": 2}}) + ); + put(&store, version, V2_CKPT_SUFFIX, &content).await; } } } BuiltCrcTest { engine, url } } + + /// Asserts the accumulated ops are valid before `build()` writes anything. + fn validate_ops(ops: &[Op]) { + let mut prev: Option = None; + for op in ops { + if let Op::Commit { version, actions } = op { + // Each commit must contain at least one action. + assert!( + !actions.is_empty(), + "commit(v{version}, []): a commit must contain at least one action", + ); + // Commit versions must be strictly increasing (and therefore unique). + if let Some(p) = prev { + assert!( + *version > p, + "commit(v{version}): commits must be added in strictly increasing \ + version order (previous commit was at v{p})", + ); + } + prev = Some(*version); + } + } + } } struct BuiltCrcTest { - engine: DefaultEngine, + engine: SyncEngine, url: Url, } impl BuiltCrcTest { + /// Construct a `LogSegment` directly from the store state (no `Snapshot`) and run + /// `build_crc_from_base` against `base`. + fn incrementally_build_crc(&self, base: &Crc) -> DeltaResult { + let storage = self.engine.storage_handler(); + let log_root = self.url.join("_delta_log/").unwrap(); + let log_segment = + LogSegment::for_snapshot_impl(storage.as_ref(), log_root, vec![], None, None)?; + log_segment.build_crc_from_base(&self.engine, base) + } + + /// Run `pick_latest_base_crc` against a directly-listed `LogSegment`, using `in_memory_base`. + fn pick_latest_base_crc( + &self, + in_memory_base: Option<&Arc>, + ) -> DeltaResult> { + let storage = self.engine.storage_handler(); + let log_root = self.url.join("_delta_log/").unwrap(); + let log_segment = + LogSegment::for_snapshot_impl(storage.as_ref(), log_root, vec![], None, None)?; + Ok(log_segment + .pick_latest_base_crc(&self.engine, in_memory_base) + .map(|c| c.version)) + } + + /// Read the on-disk CRC at `version` from this test's log. + fn read_crc_at(&self, version: u64) -> DeltaResult { + try_read_crc_file( + &self.engine, + &ParsedLogPath::create_parsed_crc(&self.url, version), + ) + } + + /// Build a snapshot at `version` (or latest if `None`) under `mode`, returning it + /// alongside a human-readable label like `"v2"` or `"latest"`. + fn snapshot_at_with_replay( + &self, + version: Option, + mode: IncrementalReplay, + ) -> (SnapshotRef, String) { + let snapshot = Snapshot::builder_for(self.url.clone()) + .with_incremental_crc_replay(mode) + .fold_with(version, SnapshotBuilder::at_version) + .build(&self.engine) + .unwrap(); + let label = version.map_or("latest".to_string(), |v| format!("v{v}")); + (snapshot, label) + } + + /// Build a snapshot at `version` with no incremental CRC replay (the `Disabled` default). + fn snapshot_at(&self, version: Option) -> (SnapshotRef, String) { + self.snapshot_at_with_replay(version, IncrementalReplay::Disabled) + } + + fn snapshot_latest_with_replay(&self, mode: IncrementalReplay) -> SnapshotRef { + self.snapshot_at_with_replay(None, mode).0 + } + + /// Build a snapshot at `from_version`, then incrementally advance it to latest under `mode`. + fn snapshot_from(&self, from_version: u64, mode: IncrementalReplay) -> SnapshotRef { + let base = Snapshot::builder_for(self.url.clone()) + .at_version(from_version) + .build(&self.engine) + .unwrap(); + Snapshot::builder_from(base) + .with_incremental_crc_replay(mode) + .build(&self.engine) + .unwrap() + } + fn assert_p_m( &self, version: impl Into>, @@ -329,38 +465,32 @@ impl BuiltCrcTest { expected_metadata: &Metadata, ) { let version = version.into(); - let mut builder = Snapshot::builder_for(self.url.clone()); - if let Some(v) = version { - builder = builder.at_version(v); + // Protocol and Metadata must resolve identically regardless of the CRC replay budget + // (no advance / bounded advance / unbounded advance), so assert across the spectrum. + for mode in [ + IncrementalReplay::Disabled, + IncrementalReplay::UpToCommits(2), + IncrementalReplay::Unlimited, + ] { + let (snapshot, label) = self.snapshot_at_with_replay(version, mode); + let table_config = snapshot.table_configuration(); + assert_eq!( + table_config.protocol(), + expected_protocol, + "Protocol mismatch at {label} ({mode:?})" + ); + assert_eq!( + table_config.metadata(), + expected_metadata, + "Metadata mismatch at {label} ({mode:?})" + ); } - let snapshot = builder.build(&self.engine).unwrap(); - let table_config = snapshot.table_configuration(); - - let version_label = version.map_or("latest".to_string(), |v| format!("v{v}")); - assert_eq!( - table_config.protocol(), - expected_protocol, - "Protocol mismatch at {version_label}" - ); - - assert_eq!( - table_config.metadata(), - expected_metadata, - "Metadata mismatch at {version_label}" - ); } fn assert_ict(&self, version: impl Into>, expected_ict: Option) { - let version = version.into(); - let mut builder = Snapshot::builder_for(self.url.clone()); - if let Some(v) = version { - builder = builder.at_version(v); - } - let snapshot = builder.build(&self.engine).unwrap(); + let (snapshot, label) = self.snapshot_at(version.into()); let ict = snapshot.get_in_commit_timestamp(&self.engine).unwrap(); - - let version_label = version.map_or("latest".to_string(), |v| format!("v{v}")); - assert_eq!(ict, expected_ict, "ICT mismatch at {version_label}"); + assert_eq!(ict, expected_ict, "ICT mismatch at {label}"); } } @@ -386,45 +516,65 @@ async fn put(store: &InMemory, version: u64, suffix: &str, content: &str) { #[tokio::test] async fn test_get_p_m_from_delta_no_checkpoint() { CrcReadTest::new() - .delta_with_p_m(0, protocol_v2(), metadata_a()) // <-- P & M from here - .delta(1) - .delta(2) + .commit( + 0, // <-- P & M from here + [ + commit_info(DEFAULT_OPERATION, None), + protocol(protocol_a()), + metadata(metadata_a()), + ], + ) + .commit(1, [commit_info(DEFAULT_OPERATION, None)]) + .commit(2, [commit_info(DEFAULT_OPERATION, None)]) .build() .await - .assert_p_m(None, &protocol_v2(), &metadata_a()); + .assert_p_m(None, &protocol_a(), &metadata_a()); } #[tokio::test] async fn test_get_p_and_m_from_different_deltas() { CrcReadTest::new() - .v2_checkpoint(0, protocol_v2(), metadata_a()) - .delta_with_p_m(1, protocol_v2_dv(), None) // <-- P from here - .delta_with_p_m(2, None, metadata_b()) // <-- M from here + .v2_checkpoint(0, protocol_a(), metadata_a()) + .commit( + 1, // <-- P from here + [commit_info(DEFAULT_OPERATION, None), protocol(protocol_b())], + ) + .commit( + 2, // <-- M from here + [commit_info(DEFAULT_OPERATION, None), metadata(metadata_b())], + ) .build() .await - .assert_p_m(None, &protocol_v2_dv(), &metadata_b()); + .assert_p_m(None, &protocol_b(), &metadata_b()); } #[tokio::test] async fn test_get_p_m_from_checkpoint() { CrcReadTest::new() - .v2_checkpoint(0, protocol_v2(), metadata_a()) // <-- P & M from here - .delta(1) - .delta(2) + .v2_checkpoint(0, protocol_a(), metadata_a()) // <-- P & M from here + .commit(1, [commit_info(DEFAULT_OPERATION, None)]) + .commit(2, [commit_info(DEFAULT_OPERATION, None)]) .build() .await - .assert_p_m(None, &protocol_v2(), &metadata_a()); + .assert_p_m(None, &protocol_a(), &metadata_a()); } #[tokio::test] async fn test_get_p_m_from_delta_after_checkpoint() { CrcReadTest::new() - .v2_checkpoint(0, protocol_v2(), metadata_a()) - .delta_with_p_m(1, protocol_v2_dv(), metadata_b()) // <-- P & M from here - .delta(2) + .v2_checkpoint(0, protocol_a(), metadata_a()) + .commit( + 1, // <-- P & M from here + [ + commit_info(DEFAULT_OPERATION, None), + protocol(protocol_b()), + metadata(metadata_b()), + ], + ) + .commit(2, [commit_info(DEFAULT_OPERATION, None)]) .build() .await - .assert_p_m(None, &protocol_v2_dv(), &metadata_b()); + .assert_p_m(None, &protocol_b(), &metadata_b()); } // ============================================================================ @@ -434,13 +584,13 @@ async fn test_get_p_m_from_delta_after_checkpoint() { #[tokio::test] async fn test_get_p_m_from_crc_at_target() { CrcReadTest::new() - .v2_checkpoint(0, protocol_v2(), metadata_a()) - .delta(1) - .delta(2) - .crc(2, protocol_v2_dv(), metadata_b(), None) // <-- P & M from here + .v2_checkpoint(0, protocol_a(), metadata_a()) + .commit(1, [commit_info(DEFAULT_OPERATION, None)]) + .commit(2, [commit_info(DEFAULT_OPERATION, None)]) + .crc(2, protocol_b(), metadata_b(), None) // <-- P & M from here .build() .await - .assert_p_m(None, &protocol_v2_dv(), &metadata_b()); + .assert_p_m(None, &protocol_b(), &metadata_b()); } #[tokio::test] @@ -448,25 +598,32 @@ async fn test_crc_preferred_over_delta_at_target() { // The P & M for the 002.crc and 002.json should NOT be different in practice. // We only do this for this test so we can differentiate which P & M is used. CrcReadTest::new() - .v2_checkpoint(0, protocol_v2(), metadata_a()) - .delta(1) - .delta_with_p_m(2, protocol_v2_dv(), metadata_a()) - .crc(2, protocol_v2_dv_ntz(), metadata_b(), None) // <-- P & M from here + .v2_checkpoint(0, protocol_a(), metadata_a()) + .commit(1, [commit_info(DEFAULT_OPERATION, None)]) + .commit( + 2, // <-- P from here + [ + commit_info(DEFAULT_OPERATION, None), + protocol(protocol_b()), + metadata(metadata_a()), + ], + ) + .crc(2, protocol_c(), metadata_b(), None) // <-- P & M from here .build() .await - .assert_p_m(None, &protocol_v2_dv_ntz(), &metadata_b()); + .assert_p_m(None, &protocol_c(), &metadata_b()); } #[tokio::test] async fn test_corrupt_crc_at_target_falls_back() { CrcReadTest::new() - .v2_checkpoint(0, protocol_v2(), metadata_a()) // <-- P & M from here - .delta(1) - .delta(2) + .v2_checkpoint(0, protocol_a(), metadata_a()) // <-- P & M from here + .commit(1, [commit_info(DEFAULT_OPERATION, None)]) + .commit(2, [commit_info(DEFAULT_OPERATION, None)]) .corrupt_crc(2) // <-- Corrupt! Fall back to replay. .build() .await - .assert_p_m(None, &protocol_v2(), &metadata_a()); + .assert_p_m(None, &protocol_a(), &metadata_a()); } #[tokio::test] @@ -474,27 +631,27 @@ async fn test_crc_wins_over_checkpoint() { // The P & M for the 002.crc and the v2 checkpoint should NOT be different in practice. // We only do this for this test so we can differentiate which P & M is used. CrcReadTest::new() - .v2_checkpoint(0, protocol_v2(), metadata_a()) - .delta(1) - .delta(2) - .v2_checkpoint(2, protocol_v2(), metadata_a()) - .crc(2, protocol_v2_dv(), metadata_b(), None) // <-- P & M from here + .v2_checkpoint(0, protocol_a(), metadata_a()) + .commit(1, [commit_info(DEFAULT_OPERATION, None)]) + .commit(2, [commit_info(DEFAULT_OPERATION, None)]) + .v2_checkpoint(2, protocol_a(), metadata_a()) + .crc(2, protocol_b(), metadata_b(), None) // <-- P & M from here .build() .await - .assert_p_m(None, &protocol_v2_dv(), &metadata_b()); + .assert_p_m(None, &protocol_b(), &metadata_b()); } #[tokio::test] async fn test_checkpoint_on_corrupt_crc() { CrcReadTest::new() - .v2_checkpoint(0, protocol_v2(), metadata_a()) - .delta(1) - .delta(2) - .v2_checkpoint(2, protocol_v2(), metadata_a()) // <-- P & M from here + .v2_checkpoint(0, protocol_a(), metadata_a()) + .commit(1, [commit_info(DEFAULT_OPERATION, None)]) + .commit(2, [commit_info(DEFAULT_OPERATION, None)]) + .v2_checkpoint(2, protocol_a(), metadata_a()) // <-- P & M from here .corrupt_crc(2) // <-- Corrupt! Fall back to replay. .build() .await - .assert_p_m(None, &protocol_v2(), &metadata_a()); + .assert_p_m(None, &protocol_a(), &metadata_a()); } // ============================================================================ @@ -504,62 +661,75 @@ async fn test_checkpoint_on_corrupt_crc() { #[tokio::test] async fn test_crc_at_earlier_version() { CrcReadTest::new() - .v2_checkpoint(0, protocol_v2(), metadata_a()) - .delta(1) - .crc(1, protocol_v2_dv(), metadata_b(), None) // <-- P & M from here - .delta(2) + .v2_checkpoint(0, protocol_a(), metadata_a()) + .commit(1, [commit_info(DEFAULT_OPERATION, None)]) + .crc(1, protocol_b(), metadata_b(), None) // <-- P & M from here + .commit(2, [commit_info(DEFAULT_OPERATION, None)]) .build() .await - .assert_p_m(None, &protocol_v2_dv(), &metadata_b()); + .assert_p_m(None, &protocol_b(), &metadata_b()); } #[tokio::test] async fn test_get_p_from_newer_delta_over_older_crc() { CrcReadTest::new() - .v2_checkpoint(0, protocol_v2(), metadata_a()) - .delta(1) - .crc(1, protocol_v2_dv(), metadata_b(), None) // <-- M from here - .delta_with_p_m(2, protocol_v2_dv_ntz(), None) // <-- P from here + .v2_checkpoint(0, protocol_a(), metadata_a()) + .commit(1, [commit_info(DEFAULT_OPERATION, None)]) + .crc(1, protocol_b(), metadata_b(), None) // <-- M from here + .commit( + 2, + [commit_info(DEFAULT_OPERATION, None), protocol(protocol_c())], + ) .build() .await - .assert_p_m(None, &protocol_v2_dv_ntz(), &metadata_b()); + .assert_p_m(None, &protocol_c(), &metadata_b()); } #[tokio::test] async fn test_get_m_from_newer_delta_over_older_crc() { CrcReadTest::new() - .v2_checkpoint(0, protocol_v2(), metadata_a()) - .delta(1) - .crc(1, protocol_v2_dv(), metadata_b(), None) // <-- P from here - .delta_with_p_m(2, None, metadata_a()) // <-- M from here + .v2_checkpoint(0, protocol_a(), metadata_a()) + .commit(1, [commit_info(DEFAULT_OPERATION, None)]) + .crc(1, protocol_b(), metadata_b(), None) // <-- P from here + .commit( + 2, // <-- M from here + [commit_info(DEFAULT_OPERATION, None), metadata(metadata_a())], + ) .build() .await - .assert_p_m(None, &protocol_v2_dv(), &metadata_a()); + .assert_p_m(None, &protocol_b(), &metadata_a()); } #[tokio::test] async fn test_corrupt_crc_at_non_target_version_falls_back() { CrcReadTest::new() - .v2_checkpoint(0, protocol_v2(), metadata_a()) // <-- P & M from here - .delta(1) + .v2_checkpoint(0, protocol_a(), metadata_a()) // <-- P & M from here + .commit(1, [commit_info(DEFAULT_OPERATION, None)]) .corrupt_crc(1) // <-- Corrupt! Fall back to replay. - .delta(2) + .commit(2, [commit_info(DEFAULT_OPERATION, None)]) .build() .await - .assert_p_m(None, &protocol_v2(), &metadata_a()); + .assert_p_m(None, &protocol_a(), &metadata_a()); } #[tokio::test] async fn test_crc_before_checkpoint_is_ignored() { CrcReadTest::new() - .delta_with_p_m(0, protocol_v2(), metadata_a()) - .delta(1) - .crc(1, protocol_v2_dv_ntz(), metadata_b(), None) - .v2_checkpoint(2, protocol_v2_dv(), metadata_a()) // <-- P & M from here - .delta(3) + .commit( + 0, + [ + commit_info(DEFAULT_OPERATION, None), + protocol(protocol_a()), + metadata(metadata_a()), + ], + ) + .commit(1, [commit_info(DEFAULT_OPERATION, None)]) + .crc(1, protocol_c(), metadata_b(), None) + .v2_checkpoint(2, protocol_b(), metadata_a()) // <-- P & M from here + .commit(3, [commit_info(DEFAULT_OPERATION, None)]) .build() .await - .assert_p_m(None, &protocol_v2_dv(), &metadata_a()); + .assert_p_m(None, &protocol_b(), &metadata_a()); } // ============================================================================ @@ -569,9 +739,9 @@ async fn test_crc_before_checkpoint_is_ignored() { #[tokio::test] async fn test_ict_from_crc_at_snapshot_version() { CrcReadTest::new() - .v2_checkpoint(0, protocol_v2_ict(), metadata_ict()) - .delta_with_ict(1, 2000) - .crc(1, protocol_v2_ict(), metadata_ict(), 1000) // <-- ICT from here + .v2_checkpoint(0, protocol_ict(), metadata_ict()) + .commit(1, [commit_info(DEFAULT_OPERATION, Some(2000))]) + .crc(1, protocol_ict(), metadata_ict(), 1000) // <-- ICT from here .build() .await .assert_ict(None, Some(1000)); @@ -580,16 +750,13 @@ async fn test_ict_from_crc_at_snapshot_version() { #[tokio::test] async fn test_ict_errors_when_crc_has_no_ict() { let setup = CrcReadTest::new() - .v2_checkpoint(0, protocol_v2_ict(), metadata_ict()) - .delta_with_ict(1, 2000) - .crc(1, protocol_v2_ict(), metadata_ict(), None) + .v2_checkpoint(0, protocol_ict(), metadata_ict()) + .commit(1, [commit_info(DEFAULT_OPERATION, Some(2000))]) + .crc(1, protocol_ict(), metadata_ict(), None) .build() .await; - let snapshot = Snapshot::builder_for(setup.url.clone()) - .build(&setup.engine) - .unwrap(); - + let (snapshot, _) = setup.snapshot_at(None); let result = snapshot.get_in_commit_timestamp(&setup.engine); assert_result_error_with_message( @@ -597,3 +764,565 @@ async fn test_ict_errors_when_crc_has_no_ict() { "In-Commit Timestamp not found in CRC file at version 1", ); } + +// ============================================================================ +// Tests: CrcReadTest framework itself +// ============================================================================ + +/// Build a `CrcReadTest` containing one commit per version, in the given order. +async fn build_with_commit_versions(versions: &[u64]) -> BuiltCrcTest { + let mut t = CrcReadTest::new(); + for &v in versions { + t = t.commit(v, [commit_info(DEFAULT_OPERATION, None)]); + } + t.build().await +} + +#[rstest::rstest] +#[case::single(&[0])] +#[case::consecutive(&[0, 1, 2])] +#[case::with_gaps(&[0, 5, 88, 1000])] +#[tokio::test] +async fn test_commit_versions_strictly_increasing_succeeds(#[case] versions: &[u64]) { + build_with_commit_versions(versions).await; +} + +#[rstest::rstest] +#[case::immediate_duplicate(&[0, 0])] +#[case::later_duplicate(&[0, 1, 1])] +#[case::immediate_decrease(&[5, 3])] +#[case::later_decrease(&[0, 5, 4])] +#[case::decrease_to_zero(&[2, 0])] +#[tokio::test] +#[should_panic(expected = "strictly increasing")] +async fn test_commit_versions_not_increasing_panics(#[case] versions: &[u64]) { + build_with_commit_versions(versions).await; +} + +#[tokio::test] +#[should_panic(expected = "at least one action")] +async fn test_commit_empty_actions_panics() { + CrcReadTest::new().commit(0, []).build().await; +} + +// ============================================================================ +// Integration tests for incremental CRC replay +// ============================================================================ + +fn crc_with_dm_and_txn_states(dm: DomainMetadataState, txn: SetTransactionState) -> Crc { + Crc { + domain_metadata_state: dm, + set_transaction_state: txn, + ..Default::default() + } +} + +fn crc_complete_empty_dm_set_txn() -> Crc { + crc_with_dm_and_txn_states( + DomainMetadataState::Complete(HashMap::new()), + SetTransactionState::Complete(HashMap::new()), + ) +} + +fn dm_entry(domain: &str, configuration: &str) -> (String, DomainMetadata) { + ( + domain.to_string(), + DomainMetadata::new(domain.to_string(), configuration.to_string()), + ) +} + +fn txn_entry(app_id: &str, version: i64, last_updated: Option) -> (String, SetTransaction) { + ( + app_id.to_string(), + SetTransaction::new(app_id.to_string(), version, last_updated), + ) +} + +// === Protocol / metadata === + +#[rstest] +#[case::newest_p_wins(vec![protocol(protocol_b()), commit_info("WRITE", None)], protocol_b(), metadata_a())] +#[case::newest_m_wins(vec![metadata(metadata_b()), commit_info("WRITE", None)], protocol_a(), metadata_b())] +#[case::base_preserved_when_segment_has_none(vec![commit_info("WRITE", None)], protocol_a(), metadata_a())] +#[tokio::test] +async fn test_p_m_propagation( + #[case] v1_actions: Vec, + #[case] expected_p: Protocol, + #[case] expected_m: Metadata, +) { + let base = Crc { + protocol: protocol_a(), + metadata: metadata_a(), + ..crc_complete_empty_dm_set_txn() + }; + let crc = CrcReadTest::new() + .commit(0, create_table_actions()) + .commit(1, v1_actions) + .build() + .await + .incrementally_build_crc(&base) + .unwrap(); + assert_eq!(crc.protocol, expected_p); + assert_eq!(crc.metadata, expected_m); +} + +// === ICT === + +#[rstest] +#[case::captures_newest(vec![commit_info("WRITE", Some(2000))], Some(2000))] +#[case::none_when_newest_has_no_commit_info(vec![set_txn("app", 1, None)], None)] +#[tokio::test] +async fn test_ict_from_newest_commit_only_replaces_base( + #[case] v2_actions: Vec, + #[case] expected_ict: Option, +) { + // Base carries an ICT so each case proves the delta's ICT replaces the base's + let base = Crc { + in_commit_timestamp_opt: Some(500), + ..crc_complete_empty_dm_set_txn() + }; + let crc = CrcReadTest::new() + .commit(0, create_table_actions_with_ict()) + .commit(1, [commit_info("WRITE", Some(1000))]) + .commit(2, v2_actions) + .build() + .await + .incrementally_build_crc(&base) + .unwrap(); + assert_eq!(crc.in_commit_timestamp_opt, expected_ict); +} + +// === Domain metadata === + +#[rstest] +#[case::complete_base(DomainMetadataState::Complete( + HashMap::from([dm_entry("a", "base_a"), dm_entry("b", "base_b")]) +))] +#[case::partial_base(DomainMetadataState::Partial( + HashMap::from([dm_entry("a", "base_a"), dm_entry("b", "base_b")]) +))] +#[tokio::test] +async fn test_dm_overrides_base_and_preserves_completeness(#[case] base_dm: DomainMetadataState) { + let base = crc_with_dm_and_txn_states(base_dm, SetTransactionState::Complete(HashMap::new())); + let crc = CrcReadTest::new() + .commit(0, create_table_actions()) + .commit( + 1, // <-- updates "a"; "b" stays at base value + [commit_info("WRITE", None), domain_metadata("a", "new_a")], + ) + .build() + .await + .incrementally_build_crc(&base) + .unwrap(); + // Variant preservation: `expect_*` panics if the base's variant did not survive. + let map = match &base.domain_metadata_state { + DomainMetadataState::Complete(_) => crc.domain_metadata_state.expect_complete(), + DomainMetadataState::Partial(_) => crc.domain_metadata_state.expect_partial(), + }; + assert_eq!(map["a"].configuration(), "new_a"); + assert_eq!(map["b"].configuration(), "base_b"); +} + +#[rstest] +#[case::complete_base(DomainMetadataState::Complete( + HashMap::from([dm_entry("a", "base_a"), dm_entry("b", "base_b")]) +))] +#[case::partial_base(DomainMetadataState::Partial( + HashMap::from([dm_entry("a", "base_a"), dm_entry("b", "base_b")]) +))] +#[tokio::test] +async fn test_dm_tombstone_removes_entry_from_base(#[case] base_dm: DomainMetadataState) { + let base = crc_with_dm_and_txn_states(base_dm, SetTransactionState::Complete(HashMap::new())); + let crc = CrcReadTest::new() + .commit(0, create_table_actions()) + .commit( + 1, // <-- tombstone removes "a" from the base map + [commit_info("WRITE", None), domain_metadata_tombstone("a")], + ) + .build() + .await + .incrementally_build_crc(&base) + .unwrap(); + let map = match &base.domain_metadata_state { + DomainMetadataState::Complete(_) => crc.domain_metadata_state.expect_complete(), + DomainMetadataState::Partial(_) => crc.domain_metadata_state.expect_partial(), + }; + assert!(!map.contains_key("a")); + assert_eq!(map["b"].configuration(), "base_b"); // <-- untouched entry preserved +} + +#[tokio::test] +async fn test_dm_newer_commit_wins_over_older_commit_for_same_domain() { + let crc = CrcReadTest::new() + .commit(0, create_table_actions()) + .commit(1, [commit_info("WRITE", None), domain_metadata("d", "old")]) + .commit( + 2, // <-- same domain in two commits; newest wins + [commit_info("WRITE", None), domain_metadata("d", "new")], + ) + .build() + .await + .incrementally_build_crc(&crc_complete_empty_dm_set_txn()) + .unwrap(); + let map = crc.domain_metadata_state.expect_complete(); + assert_eq!(map["d"].configuration(), "new"); +} + +// === SetTransaction === + +#[rstest] +#[case::complete_base(SetTransactionState::Complete(HashMap::from([ + txn_entry("app_a", 1, Some(100)), + txn_entry("app_b", 2, Some(200)), +])))] +#[case::partial_base(SetTransactionState::Partial(HashMap::from([ + txn_entry("app_a", 1, Some(100)), + txn_entry("app_b", 2, Some(200)), +])))] +#[tokio::test] +async fn test_txn_overrides_base_and_preserves_completeness(#[case] base_txn: SetTransactionState) { + let base = crc_with_dm_and_txn_states(DomainMetadataState::Complete(HashMap::new()), base_txn); + let crc = CrcReadTest::new() + .commit(0, create_table_actions()) + .commit( + 1, + [set_txn("app_a", 42, Some(999)), commit_info("WRITE", None)], + ) + .build() + .await + .incrementally_build_crc(&base) + .unwrap(); + let map = match &base.set_transaction_state { + SetTransactionState::Complete(_) => crc.set_transaction_state.expect_complete(), + SetTransactionState::Partial(_) => crc.set_transaction_state.expect_partial(), + }; + assert_eq!(map["app_a"].version, 42); + assert_eq!(map["app_a"].last_updated, Some(999)); + assert_eq!(map["app_b"].version, 2); + assert_eq!(map["app_b"].last_updated, Some(200)); +} + +#[tokio::test] +async fn test_txn_newer_commit_wins_over_older_commit_for_same_app() { + let crc = CrcReadTest::new() + .commit(0, create_table_actions()) + .commit( + 1, + [set_txn("app", 1, Some(100)), commit_info("WRITE", None)], + ) + .commit( + 2, // <-- same app in two commits; newest wins + [set_txn("app", 42, Some(999)), commit_info("WRITE", None)], + ) + .build() + .await + .incrementally_build_crc(&crc_complete_empty_dm_set_txn()) + .unwrap(); + let map = crc.set_transaction_state.expect_complete(); + assert_eq!(map["app"].version, 42); + assert_eq!(map["app"].last_updated, Some(999)); +} + +// === File stats (in-memory base) === + +#[tokio::test] +async fn test_adds_and_removes_accumulate() { + let crc = CrcReadTest::new() + .commit(0, create_table_actions()) + .commit(1, [commit_info("WRITE", None), add("a", 100)]) + .commit(2, [commit_info("WRITE", None), add("b", 200)]) + .commit(3, [commit_info("WRITE", None), add("c", 300)]) + .commit(4, [remove("a", Some(100)), commit_info("WRITE", None)]) + .commit(5, [commit_info("WRITE", None), add("d", 400)]) + .commit(6, [remove("b", Some(200)), commit_info("WRITE", None)]) + .build() + .await + .incrementally_build_crc(&crc_complete_empty_dm_set_txn()) + .unwrap(); + assert!(crc.file_stats_state().is_complete()); + let stats = crc.file_stats().unwrap(); + assert_eq!(stats.num_files(), 2); // a and b removed; c and d remain + assert_eq!(stats.table_size_bytes(), 700); // c (300) + d (400) +} + +// === Indeterminate trips === + +// Three distinct ways a single commit can lose incremental safety. +#[rstest] +#[case::remove_no_size(vec![commit_info("WRITE", None), remove("orphan", None)])] +#[case::add_with_unsafe_op(vec![commit_info("ANALYZE STATS", None), add("a", 100)])] +#[case::add_with_no_commit_info(vec![add("a", 100)])] +#[tokio::test] +async fn test_trips_indeterminate(#[case] v1_actions: Vec) { + // Base carries DM + txn entries so we can verify the indeterminate trip is scoped + // to file stats and leaves other state alone. + let base = crc_with_dm_and_txn_states( + DomainMetadataState::Complete(HashMap::from([dm_entry("d", "base_d")])), + SetTransactionState::Complete(HashMap::from([txn_entry("app", 5, Some(100))])), + ); + let crc = CrcReadTest::new() + .commit(0, create_table_actions()) + .commit(1, v1_actions) + .build() + .await + .incrementally_build_crc(&base) + .unwrap(); + assert!(crc.file_stats_state().is_indeterminate()); + assert_eq!( + crc.domain_metadata_state.expect_complete()["d"].configuration(), + "base_d", + ); + assert_eq!( + crc.set_transaction_state.expect_complete()["app"].version, + 5 + ); +} + +// === On-disk base CRC (exercises serde round-trip) === + +#[tokio::test] +async fn test_from_disk_advances_file_stats() { + let built = CrcReadTest::new() + .commit(0, create_table_actions()) + // Base CRC: a (100 B, bin 0) + b (10_000 B, bin 1). + .crc_with_files(0, protocol_a(), metadata_a(), None, &[100, 10_000]) + .commit(1, [commit_info("WRITE", None), add("c", 20_000)]) // c lands in bin 2 + .commit(2, [remove("a", Some(100)), commit_info("WRITE", None)]) // a removed from bin 0 + .build() + .await; + let base = built.read_crc_at(0).unwrap(); + let crc = built.incrementally_build_crc(&base).unwrap(); + assert!(crc.file_stats_state().is_complete()); + let stats = crc.file_stats().unwrap(); + assert_eq!(stats.num_files(), 2); + assert_eq!(stats.table_size_bytes(), 30_000); // b (10_000) + c (20_000) + + let hist = stats.file_size_histogram().unwrap(); + assert_eq!(hist.file_counts()[0], 0); // a was removed + assert_eq!(hist.file_counts()[1], 1); // b + assert_eq!(hist.total_bytes()[1], 10_000); + assert_eq!(hist.file_counts()[2], 1); // c + assert_eq!(hist.total_bytes()[2], 20_000); +} + +#[tokio::test] +async fn test_from_disk_no_histogram_means_no_result_histogram() { + // Empty file_sizes => no histogram persisted on disk. + let built = CrcReadTest::new() + .commit(0, create_table_actions()) + .crc(0, protocol_a(), metadata_a(), None) + .commit(1, [commit_info("WRITE", None), add("a", 100)]) + .build() + .await; + let base = built.read_crc_at(0).unwrap(); + let crc = built.incrementally_build_crc(&base).unwrap(); + assert!(crc.file_stats().unwrap().file_size_histogram().is_none()); +} + +#[tokio::test] +async fn test_from_disk_with_non_zero_crc_version() { + // Stale CRC is at v2, not v0. Replay covers (2, 3]. + let built = CrcReadTest::new() + .commit(0, create_table_actions()) + .commit(1, [commit_info("WRITE", None), add("a", 100)]) + .commit(2, [commit_info("WRITE", None), add("b", 200)]) + .crc_with_files(2, protocol_a(), metadata_a(), None, &[100, 200]) + .commit(3, [commit_info("WRITE", None), add("c", 300)]) + .build() + .await; + let base = built.read_crc_at(2).unwrap(); + let crc = built.incrementally_build_crc(&base).unwrap(); + assert!(crc.file_stats_state().is_complete()); + let stats = crc.file_stats().unwrap(); + // Base CRC at v=2 has a (100), b (200); replay over (2, 3] adds c (300). + assert_eq!(stats.num_files(), 3); // a, b, c + assert_eq!(stats.table_size_bytes(), 600); // a (100) + b (200) + c (300) +} + +#[tokio::test] +async fn test_full_replay_across_two_commits_propagates_all_state() { + // Realistic shape: a table evolves over two commits. v=1 initial activity; v=2 adds + // more, removes an older file, updates metadata, and stamps an ICT. The create_table_actions + // (and the base) use an ICT-enabled protocol so the v=2 ICT is well-formed. + let base = Crc { + protocol: protocol_ict(), + metadata: metadata_a(), + ..crc_complete_empty_dm_set_txn() + }; + let crc = CrcReadTest::new() + .commit(0, create_table_actions_with_ict()) + .commit( + 1, + [ + commit_info("WRITE", None), + add("a", 100), + domain_metadata("kept", "v1_value"), + set_txn("app_a", 1, Some(100)), + ], + ) + .commit( + 2, + [ + commit_info("WRITE", Some(9999)), + metadata(metadata_b()), + add("b", 200), + remove("a", Some(100)), + domain_metadata("d", "x"), + set_txn("app_b", 5, Some(200)), + ], + ) + .build() + .await + .incrementally_build_crc(&base) + .unwrap(); + + // Newest commit's M and ICT win; protocol stays at create_table_actions's ICT-enabled choice. + assert_eq!(crc.protocol, protocol_ict()); + assert_eq!(crc.metadata, metadata_b()); + assert_eq!(crc.in_commit_timestamp_opt, Some(9999)); + + // File stats accumulated across both commits: a added then removed; b remains. + assert!(crc.file_stats_state().is_complete()); + let stats = crc.file_stats().unwrap(); + assert_eq!(stats.num_files(), 1); // only b remains + assert_eq!(stats.table_size_bytes(), 200); // b (200) + + // DM and txn entries from BOTH commits land in the result map. + let dm = crc.domain_metadata_state.expect_complete(); + assert_eq!(dm["kept"].configuration(), "v1_value"); + assert_eq!(dm["d"].configuration(), "x"); + let txn = crc.set_transaction_state.expect_complete(); + assert_eq!(txn["app_a"].version, 1); + assert_eq!(txn["app_b"].version, 5); +} + +// ============================================================================ +// Tests: incremental-replay budget gate +// ============================================================================ + +// Commits 0-3 (target v3) with a CRC at `crc_version`, so distance = 3 - crc_version. +#[rstest] +// CRC already at the target (distance 0): used regardless of mode, even Disabled. +#[case::at_target_disabled(3, IncrementalReplay::Disabled, Some(3))] +// Stale CRC (distance >= 1): advancement depends on the budget. +#[case::disabled_stale(1, IncrementalReplay::Disabled, None)] +#[case::up_to_zero(1, IncrementalReplay::UpToCommits(0), None)] +#[case::interior(2, IncrementalReplay::UpToCommits(5), Some(3))] // distance 1 < 5 +#[case::at_budget(1, IncrementalReplay::UpToCommits(2), Some(3))] // distance 2 == 2 +#[case::over_budget(1, IncrementalReplay::UpToCommits(1), None)] // distance 2 > 1 +#[case::unlimited(0, IncrementalReplay::Unlimited, Some(3))] // distance 3 +#[tokio::test] +async fn test_stale_crc_advanced_only_within_replay_budget( + #[case] crc_version: u64, + #[case] mode: IncrementalReplay, + #[case] expected_crc_version: Option, +) { + let built = CrcReadTest::new() + .v2_checkpoint(0, protocol_a(), metadata_a()) + .commit(1, [commit_info(DEFAULT_OPERATION, None)]) + .commit(2, [commit_info(DEFAULT_OPERATION, None)]) + .commit(3, [commit_info(DEFAULT_OPERATION, None)]) + .crc(crc_version, protocol_a(), metadata_a(), None) + .build() + .await; + let snapshot = built.snapshot_latest_with_replay(mode); + assert_eq!(snapshot.version(), 3); + assert_eq!( + snapshot.crc_at_version().map(|c| c.version), + expected_crc_version + ); +} + +// A base exactly at the checkpoint version is kept (its tail is (ckpt, end]); a base below it is +// dropped. Guards the `>=` vs `>` seam in `pick_latest_base_crc`. +#[tokio::test] +async fn test_pick_latest_base_crc_keeps_base_at_checkpoint_drops_below() { + let built = CrcReadTest::new() + .v2_checkpoint(2, protocol_a(), metadata_a()) + .commit(3, [commit_info(DEFAULT_OPERATION, None)]) + .build() + .await; + let base_at_ckpt = Arc::new(Crc { + version: 2, + ..Default::default() + }); + let base_below_ckpt = Arc::new(Crc { + version: 1, + ..Default::default() + }); + assert_eq!( + built.pick_latest_base_crc(Some(&base_at_ckpt)).unwrap(), + Some(2), + "a base exactly at the checkpoint version must be retained" + ); + assert_eq!( + built.pick_latest_base_crc(Some(&base_below_ckpt)).unwrap(), + None, + "a base below the checkpoint version must be dropped" + ); +} + +// ============================================================================ +// Tests: incremental build (builder_from) reads P&M from the CRC +// ============================================================================ + +// `builder_from` (incremental snapshot) Protocol resolution. +// - CRC layouts: no new CRC / CRC below a json change / CRC above a json change / CRC at latest. +// - Replay modes: normal CRC replay / incremental CRC replay (Unlimited). +// The JSON protocol change is `protocol_b`; the CRC carries `protocol_c`, which matches no JSON +// version, so the result traces its source. The newer JSON change wins when it is above the CRC; +// otherwise the CRC supersedes it. +#[rstest] +#[case::no_crc(6, None, protocol_b())] // change found by replay +#[case::crc_below_change(8, Some(6), protocol_b())] // newer JSON change wins +#[case::crc_above_change(6, Some(8), protocol_c())] // CRC supersedes the change +#[case::crc_at_latest(6, Some(10), protocol_c())] // CRC at end version +#[tokio::test] +async fn test_incremental_build_from_reads_protocol_from_crc( + #[case] protocol_change_version: u64, + #[case] crc_version: Option, + #[case] expected_protocol: Protocol, + // Protocol & Metadata must resolve identically regardless of the CRC replay strategy or + // budget. + #[values( + IncrementalReplay::Disabled, + IncrementalReplay::UpToCommits(2), + IncrementalReplay::Unlimited + )] + mode: IncrementalReplay, +) { + // ====== GIVEN ===== + // v0 sets protocol_a; v5 is the "from" Snapshot version. + let mut setup = CrcReadTest::new().commit( + 0, + [ + commit_info(DEFAULT_OPERATION, None), + protocol(protocol_a()), + metadata(metadata_a()), + ], + ); + // `protocol_change_version` has a JSON commit changing the protocol to protocol_b. + for v in 1..=10 { + let mut actions = vec![commit_info(DEFAULT_OPERATION, None)]; + if v == protocol_change_version { + actions.push(protocol(protocol_b())); + } + setup = setup.commit(v, actions); + } + if let Some(v) = crc_version { + // Hack: the CRC carries protocol_c, matching no JSON version. + setup = setup.crc(v, protocol_c(), metadata_a(), None); + } + let built = setup.build().await; + + // ====== WHEN ===== + let snapshot = built.snapshot_from(5, mode); + + // ====== THEN ===== + assert_eq!(snapshot.version(), 10); + assert_eq!( + snapshot.table_configuration().protocol(), + &expected_protocol, + "protocol_change@{protocol_change_version} crc={crc_version:?} mode={mode:?}" + ); +} diff --git a/kernel/src/log_segment/domain_metadata_replay.rs b/kernel/src/log_segment/domain_metadata_replay.rs index b79a6319b4..58849bf7c7 100644 --- a/kernel/src/log_segment/domain_metadata_replay.rs +++ b/kernel/src/log_segment/domain_metadata_replay.rs @@ -1,19 +1,19 @@ //! Domain metadata replay logic for [`LogSegment`]. //! -//! This module contains the method that performs a log replay to extract the latest domain -//! metadata actions from a [`LogSegment`]. +//! Two entry points: [`LogSegment::scan_domain_metadatas`] replays the whole log for the latest +//! domain metadata, and [`LogSegment::scan_domain_metadatas_rooted_in_crc`] scans only the commits +//! after an authoritative stale CRC and reconciles them against its active-domain map. use std::collections::{HashMap, HashSet}; use tracing::instrument; -use crate::actions::get_log_domain_metadata_schema; +use super::LogSegment; use crate::actions::visitors::DomainMetadataVisitor; -use crate::actions::DomainMetadata; +use crate::actions::{DomainMetadata, LOG_DOMAIN_METADATA_SCHEMA}; +use crate::crc::merge_domain_metadata; use crate::log_replay::ActionsBatch; -use crate::{DeltaResult, Engine, RowVisitor as _}; - -use super::LogSegment; +use crate::{DeltaResult, Engine, RowVisitor as _, Version}; pub(crate) type DomainMetadataMap = HashMap; @@ -30,6 +30,75 @@ impl LogSegment { domains: Option<&HashSet<&str>>, engine: &dyn Engine, ) -> DeltaResult { + Ok(self + .visit_domain_metadatas(domains, engine)? + .into_domain_metadatas()) + } + + /// Answer a domain-metadata query rooted in an authoritative (`Complete`) but stale CRC's + /// `base_active` map at `base_version`, scanning ONLY the commits in + /// `(base_version, self.end_version]`. + /// + /// The checkpoint and every commit at/below `base_version` are skipped via + /// [`Self::segment_after_version`]. For each domain the newest action in the tail wins; a tail + /// tombstone means the domain is absent at `self.end_version` even when `base_active` holds it. + /// A domain the tail never mentions falls back to `base_active`. `domains == None` answers all + /// active domains; `Some(filter)` answers only the requested ones. Returned maps never contain + /// tombstones. + pub(crate) fn scan_domain_metadatas_rooted_in_crc( + &self, + base_version: Version, + base_active: &HashMap, + domains: Option<&HashSet<&str>>, + engine: &dyn Engine, + ) -> DeltaResult { + let tail = self + .segment_after_version(base_version) + .scan_tail_including_tombstones(domains, engine)?; + let reconciled = match domains { + // Filtered: the newest tail action per requested domain wins; a tombstone settles the + // answer as absent. Fall back to `base_active` only when the tail never mentions it. + Some(filter) => filter + .iter() + .filter_map(|&k| match tail.get(k) { + Some(dm) if dm.is_removed() => None, + Some(dm) => Some((k.to_string(), dm.clone())), + None => base_active.get(k).map(|dm| (k.to_string(), dm.clone())), + }) + .collect(), + // Unfiltered: merge the whole tail onto the base, dropping tombstoned domains. + None => { + let mut active = base_active.clone(); + merge_domain_metadata(&mut active, tail); + active + } + }; + Ok(reconciled) + } + + /// Reverse-replay this segment for domain metadata, keeping tombstones. Terminates early once + /// every requested domain is decided (a tombstone counts as decided). The CRC-rooted path keeps + /// tombstones so a removal in a newer commit can suppress a domain the base holds. + fn scan_tail_including_tombstones( + &self, + domains: Option<&HashSet<&str>>, + engine: &dyn Engine, + ) -> DeltaResult { + Ok(self + .visit_domain_metadatas(domains, engine)? + .into_domain_metadatas_including_tombstones()) + } + + /// Reverse-replay this segment, folding domain metadata actions into a + /// [`DomainMetadataVisitor`] (newest-first, first-seen-wins). A domain filter terminates + /// the scan early once every requested domain is found; without one the whole segment is + /// replayed. The caller chooses whether to keep or strip tombstones from the returned + /// visitor. + fn visit_domain_metadatas( + &self, + domains: Option<&HashSet<&str>>, + engine: &dyn Engine, + ) -> DeltaResult { let domain_filter = domains.map(|set| { set.iter() .map(|s| s.to_string()) @@ -47,8 +116,7 @@ impl LogSegment { break; } } - - Ok(visitor.into_domain_metadatas()) + Ok(visitor) } /// Read action batches from the log, projecting rows to only contain domain metadata columns. @@ -56,8 +124,7 @@ impl LogSegment { &self, engine: &dyn Engine, ) -> DeltaResult> + Send> { - let schema = get_log_domain_metadata_schema(); - self.read_actions(engine, schema.clone()) + self.read_actions(engine, LOG_DOMAIN_METADATA_SCHEMA.clone()) } } @@ -70,7 +137,7 @@ mod tests { use crate::actions::visitors::DomainMetadataVisitor; use crate::committer::FileSystemCommitter; - use crate::engine::default::DefaultEngineBuilder; + use crate::engine::sync::SyncEngine; use crate::object_store::memory::InMemory; use crate::schema::{DataType, StructField, StructType}; use crate::transaction::create_table::create_table as create_table_txn; @@ -84,7 +151,7 @@ mod tests { /// is the second batch. fn build_two_commit_log() -> (impl crate::Engine, std::sync::Arc) { let store = Arc::new(InMemory::new()); - let engine = DefaultEngineBuilder::new(store).build(); + let engine = SyncEngine::new_with_store(store); let url = Url::parse("memory:///").unwrap(); // Commit 0: CREATE TABLE (protocol + metadata) with "domainC" in the same commit. @@ -244,4 +311,39 @@ mod tests { assert_eq!(result["domainA"].configuration(), "cfgA"); assert_eq!(result["domainC"].configuration(), "cfgC"); } + + /// Adds a third commit to [`build_two_commit_log`] that tombstones "domainA". + fn build_log_with_tombstone() -> (impl crate::Engine, std::sync::Arc) { + let (engine, snapshot) = build_two_commit_log(); + let table_root = snapshot.table_root().clone(); + let _ = snapshot + .transaction(Box::new(FileSystemCommitter::new()), &engine) + .unwrap() + .with_domain_metadata_removed("domainA".to_string()) + .commit(&engine) + .unwrap(); + let snapshot = Snapshot::builder_for(table_root).build(&engine).unwrap(); + (engine, snapshot) + } + + #[tokio::test] + async fn test_scan_tail_keeps_tombstone_that_scan_domain_metadatas_drops() { + let (engine, snapshot) = build_log_with_tombstone(); + let log_segment = snapshot.log_segment(); + + // scan_domain_metadatas strips the tombstone: "domainA" is gone. + let active = log_segment.scan_domain_metadatas(None, &engine).unwrap(); + assert!(!active.contains_key("domainA")); + assert!(active.contains_key("domainB")); + assert!(active.contains_key("domainC")); + + // scan_tail_including_tombstones retains it as a tombstone so it can suppress a base + // domain. + let with_tombstones = log_segment + .scan_tail_including_tombstones(None, &engine) + .unwrap(); + assert!(with_tombstones["domainA"].is_removed()); + assert!(!with_tombstones["domainB"].is_removed()); + assert!(!with_tombstones["domainC"].is_removed()); + } } diff --git a/kernel/src/log_segment.rs b/kernel/src/log_segment/mod.rs similarity index 56% rename from kernel/src/log_segment.rs rename to kernel/src/log_segment/mod.rs index 06ecd16590..2171713a93 100644 --- a/kernel/src/log_segment.rs +++ b/kernel/src/log_segment/mod.rs @@ -3,36 +3,41 @@ use std::num::NonZero; use std::sync::{Arc, LazyLock}; -use std::time::Instant; +use delta_kernel_derive::internal_api; +use itertools::Itertools; +use tracing::{debug, info, warn}; +use url::Url; use crate::actions::visitors::SidecarVisitor; use crate::actions::{ - get_log_add_schema, schema_contains_file_actions, Sidecar, DOMAIN_METADATA_NAME, METADATA_NAME, - PROTOCOL_NAME, SET_TRANSACTION_NAME, SIDECAR_NAME, + action_presence_leaf, schema_contains_file_actions, Sidecar, LOG_ADD_SCHEMA, SIDECAR_NAME, }; +use crate::cancellation::{CancellableIterator, CancellationTokenRef}; use crate::committer::CatalogCommit; use crate::expressions::ColumnName; use crate::last_checkpoint_hint::LastCheckpointHint; use crate::log_reader::commit::CommitReader; use crate::log_replay::ActionsBatch; -use crate::metrics::{MetricEvent, MetricId, MetricsReporter}; +#[internal_api] +use crate::log_segment_files::LogSegmentFiles; +use crate::metrics::{ + emit_log_segment_load, emit_log_segment_load_failure, SnapshotLoadMetricContext, +}; +use crate::path::LogPathFileType::*; use crate::path::{LogPathFileType, ParsedLogPath}; -use crate::schema::{DataType, SchemaRef, StructField, StructType, ToSchema as _}; +#[cfg(feature = "declarative-plans")] +use crate::plans::ir::nodes::{FileType, ScanFile}; +use crate::schema::compare::SchemaComparison; +use crate::schema::{lazy_schema_ref, DataType, SchemaRef, StructField, StructType, ToSchema as _}; use crate::utils::require; +#[cfg(feature = "declarative-plans")] +use crate::Scalar; use crate::{ - DeltaResult, Engine, Error, Expression, FileMeta, Predicate, PredicateRef, RowVisitor, - StorageHandler, Version, PRE_COMMIT_VERSION, + DeltaResult, Engine, Error, FileMeta, Predicate, PredicateRef, RowVisitor, StorageHandler, + Version, }; -use delta_kernel_derive::internal_api; - -#[internal_api] -use crate::log_segment_files::LogSegmentFiles; -use crate::schema::compare::SchemaComparison; - -use itertools::Itertools; -use tracing::{debug, info, instrument, warn}; -use url::Url; +mod crc_replay; mod domain_metadata_replay; mod protocol_metadata_replay; @@ -72,7 +77,7 @@ impl CheckpointReadInfo { Self { has_stats_parsed: false, has_partition_values_parsed: false, - checkpoint_read_schema: get_log_add_schema().clone(), + checkpoint_read_schema: LOG_ADD_SCHEMA.clone(), } } } @@ -93,9 +98,9 @@ pub(crate) struct ActionsWithCheckpointInfo, pub log_root: Url, - /// Schema of the checkpoint file(s), if known from `_last_checkpoint` hint. - /// Used to determine if `stats_parsed` is available for data skipping. - pub checkpoint_schema: Option, /// The set of log files found during listing. pub listed: LogSegmentFiles, + + /// The retained `_last_checkpoint` hint, if one was read when this segment was built. The hint + /// may describe a different checkpoint than the one this segment selected, so for validated + /// access use [`Self::checkpoint_hint`] (and the [`Self::checkpoint_hint_schema`] / + /// [`Self::checkpoint_hint_sidecars`] accessors built on it). Read this field directly only + /// when the raw hint is wanted as-is -- e.g. re-threading it into a derived segment. + pub(crate) last_checkpoint_metadata: Option, } -/// Returns the identifying leaf column path for a known action type, used to build IS NOT NULL -/// predicates that enable row group skipping in checkpoint parquet files. -/// -/// For `txn`, this is effective because all app ids end up in a single checkpoint part when -/// partitioned by `add.path` as the Delta spec requires. Filtering by a specific app id is not -/// worthwhile since all app ids share one part with a large min/max range (typically UUIDs). -fn action_identifying_column(action_name: &str) -> Option { - match action_name { - METADATA_NAME => Some(ColumnName::new([METADATA_NAME, "id"])), - PROTOCOL_NAME => Some(ColumnName::new([PROTOCOL_NAME, "minReaderVersion"])), - SET_TRANSACTION_NAME => Some(ColumnName::new([SET_TRANSACTION_NAME, "appId"])), - DOMAIN_METADATA_NAME => Some(ColumnName::new([DOMAIN_METADATA_NAME, "domain"])), - _ => None, - } +/// Returns the column whose non-nullness identifies a row containing `action_name`. +fn action_presence_witness(action_name: &str) -> Option { + action_presence_leaf(action_name).map(|leaf| ColumnName::new([action_name, leaf])) } -/// Builds an IS NOT NULL predicate for row group skipping based on the action types in `schema`. -/// Returns `None` if any top-level field in the schema is not a recognized action type, since -/// an unknown type could have non-null rows in the same row group, making skipping unsafe. -fn schema_to_is_not_null_predicate(schema: &StructType) -> Option { - // Collect identifying columns for every field; short-circuit to None on any unknown field. +/// Builds a checkpoint predicate that retains every action requested by `schema`. +/// +/// Each requested action contributes an `IS NOT NULL` presence witness, and the witnesses are +/// joined with `OR`. Returns `None` if the schema is empty, or if any field lacks a reliable +/// witness (an action that could occupy a row group the resulting predicate would otherwise skip). +fn checkpoint_action_projection_predicate(schema: &StructType) -> Option { let columns: Vec = schema .fields() - .map(|f| action_identifying_column(f.name())) + .map(|field| action_presence_witness(field.name())) .collect::>()?; - let mut predicates = columns - .into_iter() - .map(|col| Expression::column(col).is_not_null()); + let mut predicates = columns.into_iter().map(Predicate::is_not_null); let first = predicates.next()?; Some(Arc::new(predicates.fold(first, Predicate::or))) } +fn combine_checkpoint_predicates( + projection_predicate: Option, + meta_predicate: Option, +) -> Option { + match (projection_predicate, meta_predicate) { + (None, predicate) | (predicate, None) => predicate, + (Some(a), Some(b)) => Some(Arc::new(Predicate::and((*a).clone(), (*b).clone()))), + } +} + impl LogSegment { - /// Creates a synthetic LogSegment for pre-commit transactions (e.g., create-table). - /// The sentinel version PRE_COMMIT_VERSION indicates no version exists yet on disk. - /// This is used to construct a pre-commit snapshot that provides table configuration - /// (protocol, metadata, schema) for operations like CTAS. - #[allow(dead_code)] // Used by create_table module - pub(crate) fn for_pre_commit(log_root: Url) -> Self { - use crate::PRE_COMMIT_VERSION; - Self { - end_version: PRE_COMMIT_VERSION, + /// Creates a LogSegment for a newly created table at version 0 from a single commit file. + /// + /// Normal log segments are built by listing files from storage and replaying them. For CREATE + /// TABLE, the table has no prior log. We construct the segment directly from the just created + /// commit file. + /// + /// # Errors + /// + /// Returns an `internal_error` if `commit_file` is not version 0 or not a commit file type. + pub(crate) fn new_for_version_zero( + log_root: Url, + commit_file: ParsedLogPath, + ) -> DeltaResult { + require!( + commit_file.version == 0, + crate::Error::internal_error(format!( + "new_for_version_zero called with version {}", + commit_file.version + )) + ); + require!( + commit_file.is_commit(), + crate::Error::internal_error(format!( + "new_for_version_zero called with non-commit file type: {:?}", + commit_file.file_type + )) + ); + Ok(Self { + end_version: commit_file.version, checkpoint_version: None, log_root, - checkpoint_schema: None, - listed: LogSegmentFiles::default(), - } + last_checkpoint_metadata: None, + listed: LogSegmentFiles { + max_published_version: Some(commit_file.version), + latest_commit_file: Some(commit_file.clone()), + ascending_commit_files: vec![commit_file], + ..Default::default() + }, + }) } #[internal_api] @@ -170,7 +202,7 @@ impl LogSegment { mut listed_files: LogSegmentFiles, log_root: Url, end_version: Option, - checkpoint_schema: Option, + last_checkpoint_metadata: Option, ) -> DeltaResult { validate_compaction_files(&listed_files.ascending_compaction_files)?; validate_checkpoint_parts(&listed_files.checkpoint_parts)?; @@ -195,12 +227,18 @@ impl LogSegment { &listed_files.checkpoint_parts, end_version, )?; + validate_latest_commit_file(&listed_files, effective_version)?; + validate_crc( + listed_files.latest_crc_file.as_ref(), + checkpoint_version, + effective_version, + )?; let log_segment = LogSegment { end_version: effective_version, checkpoint_version, log_root, - checkpoint_schema, + last_checkpoint_metadata, listed: listed_files, }; @@ -209,6 +247,42 @@ impl LogSegment { Ok(log_segment) } + /// Returns the checkpoint version from the `_last_checkpoint` hint + pub(crate) fn last_checkpoint_version(&self) -> Option { + self.last_checkpoint_metadata.as_ref().map(|m| m.version) + } + + /// The retained `_last_checkpoint` hint, but only when it describes the checkpoint this segment + /// selected (see `LastCheckpointHint::applies_to`), so the caller may trust its fields. + #[internal_api] + pub(crate) fn checkpoint_hint(&self) -> Option<&LastCheckpointHint> { + self.checkpoint_version?; + self.last_checkpoint_metadata + .as_ref() + .filter(|hint| hint.applies_to(&self.listed.checkpoint_parts)) + } + + /// The checkpoint schema from the `_last_checkpoint` hint, when the hint describes the selected + /// checkpoint (see [`Self::checkpoint_hint`]) and carried a `checkpointSchema`. `None` + /// otherwise -- the caller then reads the checkpoint footer instead. + pub(crate) fn checkpoint_hint_schema(&self) -> Option { + self.checkpoint_hint()?.checkpoint_schema.clone() + } + + /// The hint's sidecar references, when it describes the selected checkpoint (see + /// [`Self::checkpoint_hint`]). `None` if there is no matching hint or it omitted + /// `sidecarFiles`; `Some(empty)` if the hint listed none (a definitive inline leaf). A + /// non-empty list identifies a manifest checkpoint whose file actions live in those + /// sidecars. + #[allow(unused)] // consumed by the scan-shape checkpoint classifier + pub(crate) fn checkpoint_hint_sidecars(&self) -> Option<&Vec> { + self.checkpoint_hint()? + .v2_checkpoint + .as_ref()? + .sidecar_files + .as_ref() + } + /// Succinct summary string for logging purposes. fn summary(&self) -> String { format!( @@ -236,52 +310,49 @@ impl LogSegment { /// parts. All these parts will have the same checkpoint version. /// /// The options for constructing a LogSegment for Snapshot are as follows: - /// - `checkpoint_hint`: a `LastCheckpointHint` to start the log segment from (e.g. from reading the `last_checkpoint` file). + /// - `checkpoint_hint`: a `LastCheckpointHint` to start the log segment from (e.g. from reading + /// the `last_checkpoint` file). /// - `time_travel_version`: The version of the log that the Snapshot will be at. /// /// [`Snapshot`]: crate::snapshot::Snapshot /// - /// Reports metrics: `LogSegmentLoaded`. - #[instrument(name = "log_seg.for_snap", skip_all, err)] + /// Reports metrics: `LogSegmentLoadSuccess` or `LogSegmentLoadFailure`. #[internal_api] pub(crate) fn for_snapshot( storage: &dyn StorageHandler, log_root: Url, log_tail: Vec, time_travel_version: impl Into>, - reporter: Option<&Arc>, - operation_id: Option, + metric_context: SnapshotLoadMetricContext, ) -> DeltaResult { - let operation_id = operation_id.unwrap_or_default(); - let start = Instant::now(); - let time_travel_version = time_travel_version.into(); - let checkpoint_hint = LastCheckpointHint::try_read(storage, &log_root)?; - let result = Self::for_snapshot_impl( - storage, - log_root, - log_tail, - checkpoint_hint, - time_travel_version, - ); - let log_segment_loading_duration = start.elapsed(); - - match result { - Ok(log_segment) => { - reporter.inspect(|r| { - r.report(MetricEvent::LogSegmentLoaded { - operation_id, - duration: log_segment_loading_duration, - num_commit_files: log_segment.listed.ascending_commit_files.len() as u64, - num_checkpoint_files: log_segment.listed.checkpoint_parts.len() as u64, - num_compaction_files: log_segment.listed.ascending_compaction_files.len() - as u64, - }); - }); - Ok(log_segment) - } - Err(e) => Err(e), - } + let start = std::time::Instant::now(); + let build = || { + let checkpoint_hint = LastCheckpointHint::try_read(storage, &log_root)?; + Self::for_snapshot_impl( + storage, + log_root, + log_tail, + checkpoint_hint, + time_travel_version, + ) + }; + let log_segment = + build().inspect_err(|_| emit_log_segment_load_failure(&metric_context))?; + + emit_log_segment_load(&metric_context, &log_segment, start.elapsed()); + Ok(log_segment) + } + + /// How many versions behind `end_version` the latest on-disk CRC file is, for the + /// `LogSegmentLoad` metric: `None` if there is no CRC file, else `end_version - crc.version` + /// (`Some(0)` when a CRC sits at the end version). `try_new` enforces `crc.version <= + /// end_version`, so the subtraction never underflows. + pub(crate) fn crc_versions_behind(&self) -> Option { + self.listed + .latest_crc_file + .as_ref() + .map(|crc| self.end_version - crc.version) } // factored out for testing @@ -292,39 +363,53 @@ impl LogSegment { checkpoint_hint: Option, time_travel_version: Option, ) -> DeltaResult { - // Extract checkpoint schema from hint (already an Arc, no clone needed) - let checkpoint_schema = checkpoint_hint + // The end_version is the time_travel_version, if present + // TODO: When max catalog version is implemented, we would use that as end_version if + // time_travel_version is not present + let end_version = time_travel_version; + + // Keep the hint only if it points at or before end_version, or if there is no end_version + // bound + let usable_hint = checkpoint_hint .as_ref() - .and_then(|hint| hint.checkpoint_schema.clone()); - - let listed_files = match (checkpoint_hint, time_travel_version) { - (Some(cp), None) => { - LogSegmentFiles::list_with_checkpoint_hint(&cp, storage, &log_root, log_tail, None)? - } - (Some(cp), Some(end_version)) if cp.version <= end_version => { - LogSegmentFiles::list_with_checkpoint_hint( - &cp, - storage, - &log_root, - log_tail, - Some(end_version), - )? - } - _ => LogSegmentFiles::list(storage, &log_root, log_tail, None, time_travel_version)?, + .filter(|cp| end_version.is_none_or(|v| cp.version <= v)); + + // Cases: + // + // 1. usable_hint present, end_version is Some --> list_with_checkpoint_hint from + // hint.version TO end_version + // 2. usable_hint present, end_version is None --> list_with_checkpoint_hint from + // hint.version unbounded + // 3. no usable_hint, end_version is Some --> backward-scan for checkpoint before + // end_version, list from that checkpoint TO end_version (falls back to v0 if no + // checkpoint found) + // 4. no usable_hint, end_version is None --> list from v0 unbounded + + let listed_files = match (usable_hint, end_version) { + // Cases 1 and 2 + (Some(cp), end_version) => LogSegmentFiles::list_with_checkpoint_hint( + cp, + storage, + &log_root, + log_tail, + end_version, + )?, + // Case 3 + (None, Some(end)) => LogSegmentFiles::list_with_backward_checkpoint_scan( + storage, &log_root, log_tail, end, + )?, + // Case 4 + (None, None) => LogSegmentFiles::list(storage, &log_root, log_tail, None, None)?, }; - LogSegment::try_new( - listed_files, - log_root, - time_travel_version, - checkpoint_schema, - ) + LogSegment::try_new(listed_files, log_root, time_travel_version, checkpoint_hint) } - /// Constructs a [`LogSegment`] to be used for `TableChanges`. For a TableChanges between versions - /// `start_version` and `end_version`: Its LogSegment is made of zero checkpoints and all commits - /// between versions `start_version` (inclusive) and `end_version` (inclusive). If no `end_version` - /// is specified it will be the most recent version by default. + /// Constructs a [`LogSegment`] to be used for `TableChanges`. For a TableChanges between + /// versions `start_version` and `end_version`: Its LogSegment is made of zero checkpoints + /// and all commits between versions `start_version` (inclusive) and `end_version` + /// (inclusive). If no `end_version` is specified it will be the most recent version by + /// default. #[internal_api] pub(crate) fn for_table_changes( storage: &dyn StorageHandler, @@ -342,8 +427,15 @@ impl LogSegment { } // TODO: compactions? - let listed_files = - LogSegmentFiles::list_commits(storage, &log_root, Some(start_version), end_version)?; + // TODO(#2796): table-changes does not supply a log_tail yet. CDF over a catalog-managed + // table will need the catalog's commits passed here to see unbackfilled staged commits. + let listed_files = LogSegmentFiles::list_commits( + storage, + &log_root, + vec![], // log-tail + Some(start_version), + end_version, + )?; // - Here check that the start version is correct. // - [`LogSegment::try_new`] will verify that the `end_version` is correct if present. // - [`LogSegmentFiles::list_commits`] also checks that there are no gaps between commits. @@ -368,8 +460,9 @@ impl LogSegment { /// Constructs a [`LogSegment`] to be used for timestamp conversion. This [`LogSegment`] will /// consist only of contiguous commit files up to `end_version` (inclusive). If present, /// `limit` specifies the maximum length of the returned log segment. The log segment may be - /// shorter than `limit` if there are missing commits. - /// + /// shorter than `limit` if there are missing commits. `log_tail` supplies caller-provided + /// commits (e.g. catalog-managed staged commits) that take precedence over the filesystem + /// listing. // This lists all files starting from `end-limit` if `limit` is defined. For large tables, // listing with a `limit` can be a significant speedup over listing _all_ the files in the log. pub(crate) fn for_timestamp_conversion( @@ -377,6 +470,7 @@ impl LogSegment { log_root: Url, end_version: Version, limit: Option>, + log_tail: Vec, ) -> DeltaResult { // Compute the version to start listing from. let start_from = limit @@ -390,8 +484,13 @@ impl LogSegment { // this is a list of commits with possible gaps, we want to take the latest contiguous // chunk of commits - let mut listed_commits = - LogSegmentFiles::list_commits(storage, &log_root, start_from, Some(end_version))?; + let mut listed_commits = LogSegmentFiles::list_commits( + storage, + &log_root, + log_tail, + start_from, + Some(end_version), + )?; // remove gaps - return latest contiguous chunk of commits let commits = listed_commits.ascending_commit_files_mut(); @@ -447,6 +546,97 @@ impl LogSegment { Ok(new_log_segment) } + /// Creates a new LogSegment reflecting a checkpoint written at this segment's version. + /// The checkpoint must be at `end_version`. Kernel does not write multi-part checkpoints, + /// so the checkpoint must be a single file (classic parquet or V2 UUID). + /// + /// If the existing `latest_crc_file` is older than the new checkpoint version, it is + /// cleared to preserve the `LogSegmentFiles` invariant that `latest_crc_file.version >= + /// checkpoint version`. + pub(crate) fn try_new_with_checkpoint(&self, checkpoint: ParsedLogPath) -> DeltaResult { + require!( + matches!( + checkpoint.file_type, + LogPathFileType::ClassicCheckpoint | LogPathFileType::UuidCheckpoint + ), + Error::internal_error(format!( + "Cannot update LogSegment with checkpoint. Path is not a single-file \ + checkpoint. Path: {}, Type: {:?}.", + checkpoint.location.location, checkpoint.file_type + )) + ); + require!( + checkpoint.version == self.end_version, + Error::internal_error(format!( + "Cannot update LogSegment with checkpoint. Checkpoint version ({}) does not \ + equal LogSegment end_version ({}).", + checkpoint.version, self.end_version + )) + ); + + let mut new_log_segment = self.clone(); + new_log_segment.checkpoint_version = Some(checkpoint.version); + let checkpoint_version = checkpoint.version; + new_log_segment.listed.checkpoint_parts = vec![checkpoint]; + // A snapshot at version N only contains commits and compactions at versions <= N, + // so a checkpoint at N covers everything and we can clear them entirely. + new_log_segment.listed.ascending_commit_files.clear(); + new_log_segment.listed.ascending_compaction_files.clear(); + // Preserve the LogSegmentFiles invariant that `latest_crc_file.version >= checkpoint + // version`. A stale CRC is worse than missing: downstream P&M fallbacks (see + // `read_protocol_metadata_opt`) would load an older P&M and overwrite the current one. + new_log_segment + .listed + .latest_crc_file + .take_if(|crc| crc.version < checkpoint_version); + // TODO(#839): Once CheckpointWriter exposes the output schema, synthesize a + // LastCheckpointHint and thread it through here instead of None. Today the + // schema is computed inside checkpoint_data() but not returned. With None, the next + // scan will read the checkpoint parquet footer to determine the schema (e.g. + // whether stats_parsed or sidecar columns exist). + new_log_segment.last_checkpoint_metadata = None; + Ok(new_log_segment) + } + + /// Creates a new LogSegment with the given CRC file recorded as the latest. + /// The CRC file must be at `end_version`. + pub(crate) fn try_new_with_crc_file(&self, crc_file: ParsedLogPath) -> DeltaResult { + require!( + crc_file.file_type == LogPathFileType::Crc, + Error::internal_error(format!( + "Cannot update LogSegment with CRC. Path is not a CRC file. \ + Path: {}, Type: {:?}.", + crc_file.location, crc_file.file_type + )) + ); + require!( + crc_file.version == self.end_version, + Error::internal_error(format!( + "Cannot update LogSegment with CRC. CRC version ({}) does not \ + equal LogSegment end_version ({}).", + crc_file.version, self.end_version + )) + ); + // Convert to FileMeta with placeholder metadata (size=0, last_modified=0). + // Only the URL matters for CRC files: downstream code uses it for version + // tracking and reading CRC content via `try_read_crc_file`. Neither `size` + // nor `last_modified` is ever accessed. + let crc_file = ParsedLogPath { + location: FileMeta { + location: crc_file.location, + last_modified: 0, + size: 0, + }, + filename: crc_file.filename, + extension: crc_file.extension, + version: crc_file.version, + file_type: crc_file.file_type, + }; + let mut new_log_segment = self.clone(); + new_log_segment.listed.latest_crc_file = Some(crc_file); + Ok(new_log_segment) + } + pub(crate) fn new_as_published(&self) -> DeltaResult { // In the future, we can additionally convert the staged commit files to published commit // files. That would reqire faking their FileMeta locations. @@ -457,9 +647,7 @@ impl LogSegment { pub(crate) fn get_unpublished_catalog_commits(&self) -> DeltaResult> { self.listed - .ascending_commit_files - .iter() - .filter(|file| file.file_type == LogPathFileType::StagedCommit) + .staged_commits() .filter(|file| { self.listed .max_published_version @@ -491,11 +679,13 @@ impl LogSegment { /// /// Also returns `CheckpointReadInfo` with stats_parsed compatibility and the checkpoint schema. /// - /// `meta_predicate` is an optional expression for row group skipping in checkpoint parquet - /// files. It is _NOT_ the query's data predicate, but a hint for skipping irrelevant data. - /// IS NOT NULL predicates are automatically derived from `checkpoint_read_schema` and combined - /// (AND) with `meta_predicate`, so callers only need to supply query-based skipping predicates. + /// `meta_predicate` is an optional conservative predicate for checkpoint reads, not the query's + /// final data filter. When every projected action has a reliable presence witness, their + /// `IS NOT NULL` predicates are combined (OR), then combined (AND) with `meta_predicate`. + /// Otherwise projection-based pruning is disabled. Readers may ignore the resulting predicate, + /// and downstream log replay must tolerate rows that do not satisfy it. #[internal_api] + #[allow(clippy::too_many_arguments)] pub(crate) fn read_actions_with_projected_checkpoint_actions( &self, engine: &dyn Engine, @@ -504,27 +694,28 @@ impl LogSegment { meta_predicate: Option, stats_schema: Option<&StructType>, partition_schema: Option<&StructType>, + cancellation_token: Option<&CancellationTokenRef>, ) -> DeltaResult< ActionsWithCheckpointInfo> + Send>, > { - // Combine schema-derived IS NOT NULL predicate with any caller-supplied predicate so - // checkpoint parquet row groups without any relevant action type can be skipped. - // TODO: The semantics of `meta_predicate` will change in a follow-up PR. - let is_not_null_pred = schema_to_is_not_null_predicate(&checkpoint_read_schema); - let effective_predicate = match (is_not_null_pred, meta_predicate) { - (None, x) | (x, None) => x, - (Some(a), Some(b)) => Some(Arc::new(Predicate::and((*a).clone(), (*b).clone()))), - }; + // Combine the action-presence predicate with the caller's skipping predicate so readers + // can omit checkpoint data that cannot contain a relevant action. + let projection_predicate = checkpoint_action_projection_predicate(&checkpoint_read_schema); + let checkpoint_predicate = + combine_checkpoint_predicates(projection_predicate, meta_predicate); - // `replay` expects commit files to be sorted in descending order, so the return value here is correct - let commit_stream = CommitReader::try_new(engine, self, commit_read_schema)?; + // `replay` expects commit files to be sorted in descending order, so the return value here + // is correct + let commit_stream = + CommitReader::try_new(engine, self, commit_read_schema, cancellation_token)?; let checkpoint_result = self.create_checkpoint_stream( engine, checkpoint_read_schema, - effective_predicate, + checkpoint_predicate, stats_schema, partition_schema, + cancellation_token, )?; Ok(ActionsWithCheckpointInfo { @@ -549,6 +740,7 @@ impl LogSegment { None, None, None, + None, )?; Ok(result.actions) } @@ -559,6 +751,13 @@ impl LogSegment { /// that all files in `self.ascending_commit_files` and `self.ascending_compaction_files` are in /// range for this log segment. This invariant is maintained by our listing code. pub(crate) fn find_commit_cover(&self) -> Vec { + self.find_commit_cover_paths() + .into_iter() + .map(|path| path.location) + .collect() + } + + fn find_commit_cover_paths(&self) -> Vec { // Create an iterator sorted in ascending order by (initial version, end version), e.g. // [00.json, 00.09.compacted.json, 00.99.compacted.json, 01.json, 02.json, ..., 10.json, // 10.19.compacted.json, 11.json, ...] @@ -590,112 +789,199 @@ impl LogSegment { } debug!("Provisionally selecting {next:?}"); last_pushed = Some(next); - selected_files.push(next.location.clone()); + selected_files.push(next.clone()); } selected_files.reverse(); selected_files } + #[cfg(feature = "declarative-plans")] + fn version_tagged_scan_files<'a>( + paths: impl IntoIterator, + ) -> DeltaResult> { + paths + .into_iter() + .map(|path| { + Ok(ScanFile { + meta: path.location.clone(), + file_constants: vec![Scalar::Long(path.version_as_i64()?)], + }) + }) + .collect() + } + + /// Returns commit-cover files tagged with file-constant `version`. + #[cfg(feature = "declarative-plans")] + pub(crate) fn commit_cover_version_tagged_scan_files(&self) -> DeltaResult> { + Self::version_tagged_scan_files(&self.find_commit_cover_paths()) + } + + /// Returns the checkpoint parts tagged with file-constant `version`, paired with the single + /// [`FileType`] they share, or `None` when there is no checkpoint. A V2 checkpoint may be + /// UUID-named JSON, so the caller routes the parts to the matching scan operator by `FileType`. + /// + /// All checkpoint parts of a version share one format (`validate_checkpoint_parts` rejects a + /// mixed set), so a single `FileType` describes the whole group -- making a json/parquet mix + /// unrepresentable in the return type. + /// + /// [`FileType`]: crate::plans::ir::nodes::FileType + #[cfg(feature = "declarative-plans")] + pub(crate) fn checkpoint_version_tagged_scan_files( + &self, + ) -> DeltaResult)>> { + let parts = &self.listed.checkpoint_parts; + let Some(first) = parts.first() else { + return Ok(None); + }; + let file_type = if first.is_json() { + FileType::Json + } else { + FileType::Parquet + }; + Ok(Some((file_type, Self::version_tagged_scan_files(parts)?))) + } + + /// Returns sidecars from the applicable checkpoint hint, tagged with checkpoint version. + #[cfg(feature = "declarative-plans")] + pub(crate) fn checkpoint_hint_version_tagged_sidecar_scan_files( + &self, + ) -> DeltaResult>> { + let Some(sidecars) = self.checkpoint_hint_sidecars() else { + return Ok(None); + }; + let version = self.checkpoint_version.ok_or_else(|| { + Error::generic("Checkpoint hint sidecars require a selected checkpoint version") + })?; + let version = crate::version_as_i64(version)?; + sidecars + .iter() + .map(|sidecar| { + Ok(ScanFile { + meta: sidecar.to_filemeta(&self.log_root)?, + file_constants: vec![Scalar::Long(version)], + }) + }) + .collect::>>() + .map(Some) + } + /// Determines the file actions schema and extracts sidecar file references for checkpoints. /// /// This function analyzes the checkpoint to determine: - /// 1. The schema containing file actions (for future stats_parsed detection) + /// 1. The file actions schema (for stats_parsed / partitionValues_parsed detection) /// 2. Sidecar file references if this is a V2 checkpoint /// /// The logic is: - /// - JSON checkpoint: Always V2, extract sidecars and read first sidecar's schema - /// - Parquet checkpoint: Check hint/footer for sidecar column - /// - No sidecar column: V1, use footer schema - /// - Has sidecar column: V2, extract sidecars and read first sidecar's schema - /// - /// Note: `self.checkpoint_schema` from `_last_checkpoint` hint is the main checkpoint - /// parquet schema. For V1 this is what we want. For V2 we need the sidecar schema. + /// - No checkpoint parts: return (None, []) + /// - Multi-part (always V1, no sidecars): return checkpoint schema directly + /// - UUID-named JSON (always V2): extract sidecars, read first sidecar's schema + /// - Classic-named or UUID-named parquet (V1 or V2): read checkpoint schema from hint or + /// footer, then check for sidecar column to distinguish + /// - Has sidecar column (V2): extract sidecars, read first sidecar's schema + /// - No sidecar column (V1): use checkpoint schema directly fn get_file_actions_schema_and_sidecars( &self, engine: &dyn Engine, + cancellation_token: Option<&CancellationTokenRef>, ) -> DeltaResult<(Option, Vec)> { - // Only process single-part checkpoints (multi-part are always V1, no sidecars) - let checkpoint = match self.listed.checkpoint_parts.first() { - Some(cp) if self.listed.checkpoint_parts.len() == 1 => cp, - _ => return Ok((None, vec![])), - }; + // Hint schema from `_last_checkpoint` avoids footer reads when available. + let hint_schema = self.checkpoint_hint_schema(); - // Cached hint schema for determining V1 vs V2 without footer read. - // hint_schema is Option<&SchemaRef> where SchemaRef = Arc. - let hint_schema = self.checkpoint_schema.as_ref(); - - match checkpoint.extension.as_str() { - "json" => { - // JSON checkpoint is always V2, extract sidecars - let sidecar_files = self.extract_sidecar_refs(engine, checkpoint)?; + // All parts of a multi-part checkpoint belong to the same table version and follow + // the same V1 spec, so reading any one part's schema is sufficient. + let Some(checkpoint) = self.listed.checkpoint_parts.first() else { + return Ok((None, vec![])); + }; - // For V2, read first sidecar's schema (contains file actions) - let file_actions_schema = match sidecar_files.first() { - Some(first) => { - Some(engine.parquet_handler().read_parquet_footer(first)?.schema) - } - None => None, - }; - Ok((file_actions_schema, sidecar_files)) + match &checkpoint.file_type { + MultiPartCheckpoint { .. } => { + // Multi-part checkpoints are always V1 and never have sidecars. + let schema = Self::read_checkpoint_schema( + engine, + checkpoint, + hint_schema.as_ref(), + cancellation_token, + )?; + Ok((Some(schema), vec![])) } - "parquet" => { - // Check hint first to avoid unnecessary footer reads - let has_sidecars_in_hint = hint_schema.map(|s| s.field(SIDECAR_NAME).is_some()); - - match has_sidecars_in_hint { - Some(false) => { - // Hint says V1 checkpoint (no sidecars) - // Use hint schema as the file actions schema - Ok((hint_schema.cloned(), vec![])) - } - Some(true) => { - // Hint says V2 checkpoint, extract sidecars - let sidecar_files = self.extract_sidecar_refs(engine, checkpoint)?; - // For V2, read first sidecar's schema if sidecars exist. - // If no sidecars, V2 checkpoint may still have add actions in main file - // (like V1), so fall back to hint schema for stats_parsed check. - let file_actions_schema = match sidecar_files.first() { - Some(first) => { - Some(engine.parquet_handler().read_parquet_footer(first)?.schema) - } - None => hint_schema.cloned(), - }; - Ok((file_actions_schema, sidecar_files)) - } - None => { - // No hint, need to read parquet footer - let footer = engine - .parquet_handler() - .read_parquet_footer(&checkpoint.location)?; - - if footer.schema.field(SIDECAR_NAME).is_some() { - // V2 parquet checkpoint - let sidecar_files = self.extract_sidecar_refs(engine, checkpoint)?; - // For V2, read first sidecar's schema if sidecars exist. - // If no sidecars, V2 checkpoint may still have add actions in main file - // (like V1), so fall back to footer schema for stats_parsed check. - let file_actions_schema = match sidecar_files.first() { - Some(first) => Some( - engine.parquet_handler().read_parquet_footer(first)?.schema, - ), - None => Some(footer.schema), - }; - Ok((file_actions_schema, sidecar_files)) - } else { - // V1 parquet checkpoint - Ok((Some(footer.schema), vec![])) - } - } + UuidCheckpoint if checkpoint.extension.as_str() == "json" => { + // JSON checkpoint is always V2. No checkpoint schema is available since JSON + // checkpoints don't have a parquet footer to read. + self.read_sidecar_schema_and_files(engine, checkpoint, None, cancellation_token) + } + ClassicCheckpoint | UuidCheckpoint if checkpoint.extension.as_str() == "parquet" => { + // Parquet checkpoint (classic-named or UUID-named): either can be V1 or V2. + // Check for sidecar column to distinguish. + let checkpoint_schema = Self::read_checkpoint_schema( + engine, + checkpoint, + hint_schema.as_ref(), + cancellation_token, + )?; + if checkpoint_schema.field(SIDECAR_NAME).is_some() { + self.read_sidecar_schema_and_files( + engine, + checkpoint, + Some(&checkpoint_schema), + cancellation_token, + ) + } else { + Ok((Some(checkpoint_schema), vec![])) } } _ => Ok((None, vec![])), } } + /// Reads a parquet footer schema, threading the cancellation token so a cancelled request can + /// stop before or during the read (the read itself fails fast on an already-cancelled token). + fn read_footer_schema( + engine: &dyn Engine, + file: &FileMeta, + cancellation_token: Option<&CancellationTokenRef>, + ) -> DeltaResult { + Ok(engine + .parquet_handler() + .read_parquet_footer_with_cancellation(file, cancellation_token.cloned())? + .schema) + } + + /// Returns the checkpoint's parquet schema, using the hint from `_last_checkpoint` if + /// available or reading the parquet footer otherwise. + fn read_checkpoint_schema( + engine: &dyn Engine, + checkpoint: &ParsedLogPath, + hint_schema: Option<&SchemaRef>, + cancellation_token: Option<&CancellationTokenRef>, + ) -> DeltaResult { + match hint_schema { + Some(schema) => Ok(schema.clone()), + None => Self::read_footer_schema(engine, &checkpoint.location, cancellation_token), + } + } + + /// Extracts sidecar file references and reads the file actions schema from the first + /// sidecar's parquet footer. If no sidecars exist, falls back to `checkpoint_schema` + /// since V2 checkpoints may store add actions directly in the main file. + fn read_sidecar_schema_and_files( + &self, + engine: &dyn Engine, + checkpoint: &ParsedLogPath, + checkpoint_schema: Option<&SchemaRef>, + cancellation_token: Option<&CancellationTokenRef>, + ) -> DeltaResult<(Option, Vec)> { + let sidecar_files = self.extract_sidecar_refs(engine, checkpoint, cancellation_token)?; + let file_actions_schema = match sidecar_files.first() { + Some(first) => Some(Self::read_footer_schema(engine, first, cancellation_token)?), + None => checkpoint_schema.cloned(), + }; + Ok((file_actions_schema, sidecar_files)) + } + /// Returns an iterator over checkpoint data, processing sidecar files when necessary. /// - /// For single-part checkpoints that need file actions, this function: - /// 1. Determines the files actions schema (for future stats_parsed detection) + /// For checkpoints that need file actions, this function: + /// 1. Determines the file actions schema (for stats_parsed / partitionValues_parsed detection) /// 2. Extracts sidecar file references if present (V2 checkpoints) /// 3. Reads checkpoint and sidecar data using cached sidecar refs /// @@ -707,22 +993,18 @@ impl LogSegment { meta_predicate: Option, stats_schema: Option<&StructType>, partition_schema: Option<&StructType>, + cancellation_token: Option<&CancellationTokenRef>, ) -> DeltaResult< ActionsWithCheckpointInfo> + Send>, > { let need_file_actions = schema_contains_file_actions(&action_schema); - // Extract file actions schema and sidecar files - // Only process sidecars when: - // 1. We need file actions (add/remove) - sidecars only contain file actions - // 2. Single-part checkpoint - multi-part checkpoints are always V1 (no sidecars) let (file_actions_schema, sidecar_files) = if need_file_actions { - self.get_file_actions_schema_and_sidecars(engine)? + self.get_file_actions_schema_and_sidecars(engine, cancellation_token)? } else { (None, vec![]) }; - // Check if checkpoint has compatible stats_parsed and add it to the schema if so let has_stats_parsed = stats_schema .zip(file_actions_schema.as_ref()) @@ -734,10 +1016,19 @@ impl LogSegment { .zip(file_actions_schema.as_ref()) .is_some_and(|(ps, fs)| Self::schema_has_compatible_partition_values_parsed(fs, ps)); - // Build final schema with any additional fields needed - // (stats_parsed, partitionValues_parsed, sidecar) + // JSON checkpoint stats are required when structured stats cannot satisfy the scan schema. + let needs_json_stats_fallback = stats_schema.is_some() + && !has_stats_parsed + && action_schema.field("add").is_some_and(|field| { + let DataType::Struct(add) = field.data_type() else { + return false; + }; + add.field("stats").is_none() + }); + let needs_sidecar = need_file_actions && !sidecar_files.is_empty(); - let needs_add_augmentation = has_stats_parsed || has_partition_values_parsed; + let needs_add_augmentation = + needs_json_stats_fallback || has_stats_parsed || has_partition_values_parsed; let augmented_checkpoint_read_schema = if needs_add_augmentation || needs_sidecar { let mut new_fields: Vec = if let (true, Some(add_field)) = (needs_add_augmentation, action_schema.field("add")) @@ -749,18 +1040,16 @@ impl LogSegment { }; let mut add_fields: Vec = add_struct.fields().cloned().collect(); + if needs_json_stats_fallback { + add_fields.push(StructField::nullable("stats", DataType::STRING)); + } + if let (true, Some(ss)) = (has_stats_parsed, stats_schema) { - add_fields.push(StructField::nullable( - "stats_parsed", - DataType::Struct(Box::new(ss.clone())), - )); + add_fields.push(StructField::nullable("stats_parsed", ss.clone())); } if let (true, Some(ps)) = (has_partition_values_parsed, partition_schema) { - add_fields.push(StructField::nullable( - "partitionValues_parsed", - DataType::Struct(Box::new(ps.clone())), - )); + add_fields.push(StructField::nullable("partitionValues_parsed", ps.clone())); } // Rebuild schema with modified add field @@ -801,25 +1090,26 @@ impl LogSegment { .map(|f| f.location.clone()) .collect(); - let parquet_handler = engine.parquet_handler(); - // Historically, we had a shared file reader trait for JSON and Parquet handlers, // but it was removed to avoid unnecessary coupling. This is a concrete case // where it *could* have been useful, but for now, we're keeping them separate. // If similar patterns start appearing elsewhere, we should reconsider that decision. let actions = match self.listed.checkpoint_parts.first() { Some(parsed_log_path) if parsed_log_path.extension == "json" => { - engine.json_handler().read_json_files( + engine.json_handler().read_json_files_with_cancellation( &checkpoint_file_meta, augmented_checkpoint_read_schema.clone(), meta_predicate.clone(), + cancellation_token.cloned(), )? } - Some(parsed_log_path) if parsed_log_path.extension == "parquet" => parquet_handler - .read_parquet_files( + Some(parsed_log_path) if parsed_log_path.extension == "parquet" => engine + .parquet_handler() + .read_parquet_files_with_cancellation( &checkpoint_file_meta, augmented_checkpoint_read_schema.clone(), meta_predicate.clone(), + cancellation_token.cloned(), )?, Some(parsed_log_path) => { return Err(Error::generic(format!( @@ -837,11 +1127,14 @@ impl LogSegment { // Both checkpoint and sidecar parquet files share the same `add.stats_parsed.*` column // layout, so we reuse the same predicate for row group skipping. let sidecar_batches = if !sidecar_files.is_empty() { - parquet_handler.read_parquet_files( - &sidecar_files, - augmented_checkpoint_read_schema.clone(), - meta_predicate, - )? + engine + .parquet_handler() + .read_parquet_files_with_cancellation( + &sidecar_files, + augmented_checkpoint_read_schema.clone(), + meta_predicate, + cancellation_token.cloned(), + )? } else { Box::new(std::iter::empty()) }; @@ -869,22 +1162,32 @@ impl LogSegment { &self, engine: &dyn Engine, checkpoint: &ParsedLogPath, + cancellation_token: Option<&CancellationTokenRef>, ) -> DeltaResult> { // Read checkpoint with just the sidecar column let batches = match checkpoint.extension.as_str() { - "json" => engine.json_handler().read_json_files( - std::slice::from_ref(&checkpoint.location), - Self::sidecar_read_schema(), - None, - )?, - "parquet" => engine.parquet_handler().read_parquet_files( + "json" => engine.json_handler().read_json_files_with_cancellation( std::slice::from_ref(&checkpoint.location), Self::sidecar_read_schema(), None, + cancellation_token.cloned(), )?, + "parquet" => engine + .parquet_handler() + .read_parquet_files_with_cancellation( + std::slice::from_ref(&checkpoint.location), + Self::sidecar_read_schema(), + None, + cancellation_token.cloned(), + )?, _ => return Ok(vec![]), }; + // Unlike the checkpoint/commit reads that feed the wrapped scan-action stream, this loop + // consumes batches locally, so wrap it to poll the token between batches even against an + // engine whose reader ignores it. + let batches = CancellableIterator::new(batches, cancellation_token.cloned()); + // Extract sidecar file references let mut visitor = SidecarVisitor::default(); for batch_result in batches { @@ -900,19 +1203,25 @@ impl LogSegment { .try_collect() } - /// Creates a pruned LogSegment for replay *after* a CRC at `start_v_exclusive`. - /// - /// The CRC covers protocol, metadata, and checkpoint state, so this segment drops - /// checkpoint files, CRC files, and checkpoint schema. Only commits and compactions - /// in `(start_v_exclusive, end_version]` are retained. - pub(crate) fn segment_after_crc(&self, start_v_exclusive: Version) -> Self { + /// Creates a pruned LogSegment of only the commits and compactions in + /// `(start_v_exclusive, end_version]`, dropping checkpoint files, CRC files, and + /// last-checkpoint metadata. + pub(crate) fn segment_after_version(&self, start_v_exclusive: Version) -> Self { + // A checkpoint above start_v_exclusive would drop the commits between them, since the + // returned segment keeps only (start_v_exclusive, end] and no checkpoint. + debug_assert!( + self.checkpoint_version + .is_none_or(|ckpt| start_v_exclusive >= ckpt), + "segment_after_version: start_v_exclusive ({start_v_exclusive}) is below checkpoint {:?}", + self.checkpoint_version, + ); let (commits, compactions) = self.filtered_commits_and_compactions(Some(start_v_exclusive), self.end_version); LogSegment { end_version: self.end_version, checkpoint_version: None, log_root: self.log_root.clone(), - checkpoint_schema: None, + last_checkpoint_metadata: None, listed: LogSegmentFiles { ascending_commit_files: commits, ascending_compaction_files: compactions, @@ -924,31 +1233,6 @@ impl LogSegment { } } - /// Creates a pruned LogSegment for replay *before* a CRC at `end_v_inclusive`. - /// - /// Used as fallback when the CRC at `end_v_inclusive` fails to load. Falls back to - /// checkpoint-based replay, so checkpoint files and schema are preserved. Only commits - /// and compactions in `(checkpoint_version, end_v_inclusive]` are retained. Fields not - /// needed for this replay path (CRC file, latest commit file) are dropped. - pub(crate) fn segment_through_crc(&self, end_v_inclusive: Version) -> Self { - let (commits, compactions) = - self.filtered_commits_and_compactions(self.checkpoint_version, end_v_inclusive); - LogSegment { - end_version: self.end_version, - checkpoint_version: self.checkpoint_version, - log_root: self.log_root.clone(), - checkpoint_schema: self.checkpoint_schema.clone(), - listed: LogSegmentFiles { - ascending_commit_files: commits, - ascending_compaction_files: compactions, - checkpoint_parts: self.listed.checkpoint_parts.clone(), - latest_crc_file: None, - latest_commit_file: None, - max_published_version: None, - }, - } - } - /// Filters commits and compactions to those within `(lo_exclusive, hi_inclusive]`. /// If `lo_exclusive` is `None`, there is no lower bound. fn filtered_commits_and_compactions( @@ -981,11 +1265,7 @@ impl LogSegment { } /// How many commits since a checkpoint, according to this log segment. - /// Returns 0 for pre-commit snapshots (where end_version is PRE_COMMIT_VERSION). pub(crate) fn commits_since_checkpoint(&self) -> u64 { - if self.end_version == PRE_COMMIT_VERSION { - return 0; - } // we can use 0 as the checkpoint version if there is no checkpoint since `end_version - 0` // is the correct number of commits since a checkpoint if there are no checkpoints let checkpoint_version = self.checkpoint_version.unwrap_or(0); @@ -994,11 +1274,7 @@ impl LogSegment { } /// How many commits since a log-compaction or checkpoint, according to this log segment. - /// Returns 0 for pre-commit snapshots (where end_version is PRE_COMMIT_VERSION). pub(crate) fn commits_since_log_compaction_or_checkpoint(&self) -> u64 { - if self.end_version == PRE_COMMIT_VERSION { - return 0; - } // Annoyingly we have to search all the compaction files to determine this, because we only // sort by start version, so technically the max end version could be anywhere in the vec. // We can return 0 in the case there is no compaction since end_version - 0 is the correct @@ -1036,24 +1312,30 @@ impl LogSegment { } /// Schema to read just the sidecar column from a checkpoint file. - fn sidecar_read_schema() -> SchemaRef { - static SIDECAR_SCHEMA: LazyLock = LazyLock::new(|| { - Arc::new(StructType::new_unchecked([StructField::nullable( - SIDECAR_NAME, - Sidecar::to_schema(), - )])) - }); + pub(crate) fn sidecar_read_schema() -> SchemaRef { + static SIDECAR_SCHEMA: LazyLock = + lazy_schema_ref! { nullable SIDECAR_NAME: (Sidecar::to_schema()) }; SIDECAR_SCHEMA.clone() } + fn get_field_from_add<'a>( + checkpoint_schema: &'a StructType, + name: &str, + ) -> Option<&'a StructField> { + let DataType::Struct(add) = checkpoint_schema.field("add")?.data_type() else { + return None; + }; + add.field(name) + } + /// Checks if a checkpoint schema contains a usable `add.stats_parsed` field. /// /// This validates that: /// 1. The `add.stats_parsed` field exists in the checkpoint schema /// 2. The types in `stats_parsed` are compatible with the stats schema for data skipping /// - /// The `stats_schema` parameter contains only the columns referenced in the data skipping - /// predicate. This is built from the predicate and passed in by the caller. + /// `stats_schema` is the physical typed schema required by the scan. It includes requested + /// structured output fields and any data-skipping predicate fields. /// /// Both the checkpoint's `stats_parsed` schema and the `stats_schema` for data skipping /// use physical column names (not logical names), so direct name comparison is correct. @@ -1063,14 +1345,7 @@ impl LogSegment { checkpoint_schema: &StructType, stats_schema: &StructType, ) -> bool { - // Get add.stats_parsed from the checkpoint schema - let Some(stats_parsed) = checkpoint_schema - .field("add") - .and_then(|f| match f.data_type() { - DataType::Struct(s) => s.field("stats_parsed"), - _ => None, - }) - else { + let Some(stats_parsed) = Self::get_field_from_add(checkpoint_schema, "stats_parsed") else { debug!("stats_parsed not compatible: checkpoint schema does not contain add.stats_parsed field"); return false; }; @@ -1083,52 +1358,20 @@ impl LogSegment { return false; }; - // Check type compatibility for both minValues and maxValues structs. - // While these typically have the same schema, the protocol doesn't guarantee it, - // so we check both to be safe. - for field_name in ["minValues", "maxValues"] { - let Some(checkpoint_values_field) = stats_struct.field(field_name) else { - // stats_parsed exists but no minValues/maxValues - unusual but valid - continue; - }; - - // minValues/maxValues must be a Struct containing per-column statistics. - // If it exists but isn't a Struct, the schema is malformed and unusable. - let DataType::Struct(checkpoint_values) = checkpoint_values_field.data_type() else { - debug!( - "stats_parsed not compatible: stats_parsed.{} is not a Struct, got {:?}", - field_name, - checkpoint_values_field.data_type() - ); - return false; - }; - - // Get the corresponding field from stats_schema (e.g., stats_schema.minValues) - let Some(stats_values_field) = stats_schema.field(field_name) else { - // stats_schema doesn't have minValues/maxValues, skip this check - continue; - }; - let DataType::Struct(stats_values) = stats_values_field.data_type() else { - // stats_schema.minValues/maxValues isn't a struct - shouldn't happen but skip - continue; - }; - - // Check type compatibility recursively for nested structs. - // Only fields that exist in both schemas need compatible types. - // Extra fields in checkpoint are ignored; missing fields return null. - if !Self::structs_have_compatible_types(checkpoint_values, stats_values, field_name) { - return false; - } + if !Self::structs_have_compatible_types(stats_struct, stats_schema, "stats_parsed") { + return false; } debug!("Checkpoint schema has compatible stats_parsed for data skipping"); true } - /// Recursively checks if two struct types have compatible field types for stats parsing. + /// Recursively checks if two struct types have compatible field types. /// - /// For each field in `needed` (stats schema), if it exists in `available` (checkpoint): - /// - Primitive types: must be compatible via `can_read_as` (allows type widening) + /// Used by both `stats_parsed` and `partitionValues_parsed` compatibility checks. + /// For each field in `needed`, if it exists in `available` (checkpoint): + /// - Primitive types: must be compatible via [`PrimitiveType::is_stats_type_compatible_with`] + /// (allows type widening and Parquet physical type reinterpretation) /// - Nested structs: recursively check inner fields /// - Missing fields in checkpoint: OK (will return null when accessed) /// - Extra fields in checkpoint: OK (ignored) @@ -1139,7 +1382,6 @@ impl LogSegment { ) -> bool { for needed_field in needed.fields() { let Some(available_field) = available.field(needed_field.name()) else { - // Field missing in checkpoint - that's OK, it will be null continue; }; @@ -1155,14 +1397,21 @@ impl LogSegment { return false; } } - // Non-struct types: use can_read_as for type compatibility + // Non-struct types: use stats-specific rules for primitives and standard + // schema rules otherwise. (avail_type, need_type) => { - if avail_type.can_read_as(need_type).is_err() { + let compatible = match (avail_type, need_type) { + (DataType::Primitive(a), DataType::Primitive(b)) => { + a.is_stats_type_compatible_with(b) + } + (a, b) => a.can_read_as(b).is_ok(), + }; + if !compatible { debug!( - "stats_parsed not compatible: incompatible type for '{}' in {}: \ + "{} not compatible: incompatible type for '{}': \ checkpoint has {:?}, stats schema needs {:?}", - needed_field.name(), context, + needed_field.name(), avail_type, need_type ); @@ -1188,12 +1437,7 @@ impl LogSegment { partition_schema: &StructType, ) -> bool { let Some(partition_parsed) = - checkpoint_schema - .field("add") - .and_then(|f| match f.data_type() { - DataType::Struct(s) => s.field("partitionValues_parsed"), - _ => None, - }) + Self::get_field_from_add(checkpoint_schema, "partitionValues_parsed") else { debug!("partitionValues_parsed not compatible: checkpoint schema does not contain add.partitionValues_parsed field"); return false; @@ -1354,3 +1598,58 @@ fn validate_end_version( } Ok(effective_version) } + +/// Validates the `latest_commit_file` field of a [`LogSegmentFiles`]. Enforces: +/// +/// 1. If `ascending_commit_files` is non-empty, `latest_commit_file` must be `Some`. +/// 2. If `latest_commit_file` is `Some`, its version must equal `effective_version`. +fn validate_latest_commit_file( + listed: &LogSegmentFiles, + effective_version: Version, +) -> DeltaResult<()> { + require!( + listed.ascending_commit_files.is_empty() || listed.latest_commit_file.is_some(), + Error::internal_error( + "latest_commit_file must be Some when ascending_commit_files is non-empty" + ) + ); + if let Some(commit) = &listed.latest_commit_file { + require!( + commit.version == effective_version, + Error::internal_error(format!( + "latest_commit_file version {} does not match end_version {effective_version}", + commit.version, + )) + ); + } + Ok(()) +} + +/// A CRC describes the table state at its version, so it must sit within the segment: at or above +/// the checkpoint (when present) and at or below the end version. +fn validate_crc( + crc: Option<&ParsedLogPath>, + checkpoint_version: Option, + effective_version: Version, +) -> DeltaResult<()> { + let Some(crc) = crc else { + return Ok(()); + }; + if let Some(checkpoint_version) = checkpoint_version { + require!( + crc.version >= checkpoint_version, + Error::internal_error(format!( + "CRC file version {} is older than checkpoint version {checkpoint_version}", + crc.version + )) + ); + } + require!( + crc.version <= effective_version, + Error::internal_error(format!( + "CRC file version {} is newer than log segment version {effective_version}", + crc.version + )) + ); + Ok(()) +} diff --git a/kernel/src/log_segment/protocol_metadata_replay.rs b/kernel/src/log_segment/protocol_metadata_replay.rs index 6b0da90729..bd40ffb898 100644 --- a/kernel/src/log_segment/protocol_metadata_replay.rs +++ b/kernel/src/log_segment/protocol_metadata_replay.rs @@ -3,61 +3,69 @@ //! This module contains the methods that perform a lightweight log replay to extract the latest //! Protocol and Metadata actions from a [`LogSegment`]. -use crate::actions::{get_commit_schema, Metadata, Protocol, METADATA_NAME, PROTOCOL_NAME}; -use crate::crc::{CrcLoadResult, LazyCrc}; -use crate::log_replay::ActionsBatch; -use crate::{DeltaResult, Engine, Error}; +use std::sync::Arc; -use tracing::{info, instrument, warn}; +use tracing::{info, instrument}; use super::LogSegment; +use crate::actions::{Metadata, Protocol, METADATA_FIELD, PROTOCOL_FIELD}; +#[cfg(feature = "declarative-plans")] +use crate::actions::{METADATA_NAME, PROTOCOL_NAME}; +use crate::crc::Crc; +use crate::log_replay::ActionsBatch; +use crate::metrics::ProtocolMetadataSource; +#[cfg(feature = "declarative-plans")] +use crate::plans::ir::nodes::FileType; +#[cfg(feature = "declarative-plans")] +use crate::plans::{Operation, PlanBuilder, PlanExecutor}; +#[cfg(feature = "declarative-plans")] +use crate::schema::column_name; +use crate::schema::schema_ref; +use crate::{DeltaResult, Engine, Error}; impl LogSegment { /// Read the latest Protocol and Metadata from this log segment, using CRC when available. - /// Returns an error if either is missing. + /// Returns an error if either is missing, and the [`ProtocolMetadataSource`] describing how + /// P&M was resolved. /// - /// This is the checked variant of [`Self::read_protocol_metadata_unchecked`], used for - /// fresh snapshot creation where both Protocol and Metadata must exist. + /// This is the checked variant of [`Self::read_protocol_metadata_opt`], used for fresh + /// snapshot creation where both Protocol and Metadata must exist. pub(crate) fn read_protocol_metadata( &self, engine: &dyn Engine, - lazy_crc: &LazyCrc, - ) -> DeltaResult<(Metadata, Protocol)> { - match self.read_protocol_metadata_opt(engine, lazy_crc)? { - (Some(m), Some(p)) => Ok((m, p)), - (None, Some(_)) => Err(Error::MissingMetadata), - (Some(_), None) => Err(Error::MissingProtocol), - (None, None) => Err(Error::MissingMetadataAndProtocol), + crc: Option<&Arc>, + ) -> DeltaResult<(Metadata, Protocol, ProtocolMetadataSource)> { + match self.read_protocol_metadata_opt(engine, crc)? { + (Some(m), Some(p), source) => Ok((m, p, source)), + (None, Some(_), _) => Err(Error::MissingMetadata), + (Some(_), None, _) => Err(Error::MissingProtocol), + (None, None, _) => Err(Error::MissingMetadataAndProtocol), } } /// Read the latest Protocol and Metadata from this log segment, using CRC when available. /// Returns `None` for either if not found. /// - /// This is the unchecked variant of [`Self::read_protocol_metadata_checked`], used for - /// incremental snapshot updates where the caller can fall back to an existing snapshot's - /// Protocol and Metadata. + /// This is the unchecked variant of [`Self::read_protocol_metadata`], used for incremental + /// snapshot updates where the caller can fall back to an existing snapshot's Protocol and + /// Metadata. /// - /// The `lazy_crc` parameter allows the CRC to be loaded at most once and shared for - /// future use (domain metadata, in-commit timestamp, etc.). + /// The `crc` parameter is the CRC eagerly resolved by the caller; it is used to + /// short-circuit or seed the replay. #[instrument(name = "log_seg.load_p_m", skip_all, err)] pub(crate) fn read_protocol_metadata_opt( &self, engine: &dyn Engine, - lazy_crc: &LazyCrc, - ) -> DeltaResult<(Option, Option)> { - let crc_version = lazy_crc.crc_version(); - + crc: Option<&Arc>, + ) -> DeltaResult<(Option, Option, ProtocolMetadataSource)> { // Case 1: If CRC at target version, use it directly and exit early. - if crc_version == Some(self.end_version) { - if let CrcLoadResult::Loaded(crc) = lazy_crc.get_or_load(engine) { - info!("P&M from CRC at target version {}", self.end_version); - return Ok((Some(crc.metadata.clone()), Some(crc.protocol.clone()))); - } - warn!( - "CRC at target version {} failed to load, falling back to log replay", - self.end_version - ); + if let Some(crc) = crc.filter(|c| c.version == self.end_version) { + info!("P&M from CRC at target version {}", self.end_version); + return Ok(( + Some(crc.metadata.clone()), + Some(crc.protocol.clone()), + ProtocolMetadataSource::CrcAtTarget, + )); } // We didn't return above, so we need to do log replay to find P&M. @@ -66,59 +74,65 @@ impl LogSegment { // commits *after* the CRC version. // (a) If we find new P&M in the pruned replay, return it. // (b) If we don't find new P&M, fall back to the CRC. - // (c) If the CRC also fails, fall back to replaying the remaining segment - // (checkpoint + commits up through the CRC version). - // - // Case 3: CRC at target version failed to load => Full P&M log replay. // - // Case 4: No CRC exists at all => Full P&M log replay. + // Case 3: No CRC exists => Full P&M log replay. - if let Some(crc_v) = crc_version.filter(|&v| v < self.end_version) { + if let Some(crc) = crc.filter(|c| c.version < self.end_version) { // Case 2(a): Replay only commits after CRC version - info!("Pruning log segment to commits after CRC version {}", crc_v); - let pruned = self.segment_after_crc(crc_v); - let (metadata_opt, protocol_opt) = pruned.replay_for_pm(engine, None, None)?; + info!( + "Pruning log segment to commits after CRC version {}", + crc.version + ); + let pruned = self.segment_after_version(crc.version); + let (metadata_opt, protocol_opt) = pruned.replay_for_pm(engine)?; if metadata_opt.is_some() && protocol_opt.is_some() { info!("Found P&M from pruned log replay"); - return Ok((metadata_opt, protocol_opt)); - } - - // Case 2(b): P&M incomplete from pruned replay, try CRC. - // Use `or_else` so any newer P or M found in the pruned replay takes priority - // over the (older) CRC values. - if let CrcLoadResult::Loaded(crc) = lazy_crc.get_or_load(engine) { - info!("P&M fallback to CRC (no P&M changes after CRC version)"); return Ok(( - metadata_opt.or_else(|| Some(crc.metadata.clone())), - protocol_opt.or_else(|| Some(crc.protocol.clone())), + metadata_opt, + protocol_opt, + ProtocolMetadataSource::CrcSeededPmOnlyReplay, )); } - // Case 2(c): CRC failed to load. Replay the remaining segment (checkpoint + - // commits up through CRC version), carrying forward any partial results from the - // pruned replay above. - warn!( - "CRC at version {} failed to load, replaying remaining segment", - crc_v - ); - let remaining = self.segment_through_crc(crc_v); - return remaining.replay_for_pm(engine, metadata_opt, protocol_opt); + // Case 2(b): P&M incomplete from pruned replay, use the CRC. + // Use `or_else` so any newer P or M found in the pruned replay takes priority + // over the (older) CRC values. + info!("P&M fallback to CRC (no P&M changes after CRC version)"); + return Ok(( + metadata_opt.or_else(|| Some(crc.metadata.clone())), + protocol_opt.or_else(|| Some(crc.protocol.clone())), + ProtocolMetadataSource::CrcSeededPmOnlyReplay, + )); } - // Case 3 / Case 4: Full P&M log replay. - self.replay_for_pm(engine, None, None) + // Case 3: Full P&M log replay. + let (metadata_opt, protocol_opt) = self.replay_for_pm(engine)?; + Ok(( + metadata_opt, + protocol_opt, + ProtocolMetadataSource::FullReplay, + )) } - /// Replays the log segment for Protocol and Metadata, merging with any already-found values. - /// Stops early once both are found. + /// Replays the log segment for Protocol and Metadata, stopping early once both are found. fn replay_for_pm( &self, engine: &dyn Engine, - mut metadata_opt: Option, - mut protocol_opt: Option, ) -> DeltaResult<(Option, Option)> { - for actions_batch in self.read_pm_batches(engine)? { + // Providing a plan executor opts the engine into declarative P&M replay. + #[cfg(feature = "declarative-plans")] + let actions_batches = match engine.plan_executor() { + Some(executor) => self.read_pm_batches_via_plan(executor.as_ref())?, + None => Box::new(self.read_pm_batches(engine)?) as _, + }; + + #[cfg(not(feature = "declarative-plans"))] + let actions_batches = self.read_pm_batches(engine)?; + + let mut metadata_opt = None; + let mut protocol_opt = None; + for actions_batch in actions_batches { let actions = actions_batch?.actions; if metadata_opt.is_none() { metadata_opt = Metadata::try_new_from_data(actions.as_ref())?; @@ -133,12 +147,64 @@ impl LogSegment { Ok((metadata_opt, protocol_opt)) } + #[cfg(feature = "declarative-plans")] + fn read_pm_batches_via_plan( + &self, + executor: &dyn PlanExecutor, + ) -> DeltaResult> + Send>> { + let versioned_schema = schema_ref! { + (&PROTOCOL_FIELD), + (&METADATA_FIELD), + not_null "version": LONG, + }; + + let commit_files = self.commit_cover_version_tagged_scan_files()?; + let commits = PlanBuilder::scan_json(commit_files, &["version"], versioned_schema.clone())?; + + // A checkpoint's parts share one format; scan them with the matching operator. + let checkpoint = self + .checkpoint_version_tagged_scan_files()? + .map(|(file_type, checkpoint_files)| { + let scan = match file_type { + FileType::Json => PlanBuilder::scan_json, + FileType::Parquet => PlanBuilder::scan_parquet, + }; + scan(checkpoint_files, &["version"], versioned_schema.clone()) + }) + .transpose()?; + + let plan = PlanBuilder::union_all(std::iter::once(commits).chain(checkpoint))? + .aggregate_ungrouped(|a| { + a.max_non_null_by( + column_name!(PROTOCOL_NAME), + column_name!(PROTOCOL_NAME), + column_name!("version"), + ) + .max_non_null_by( + column_name!(METADATA_NAME), + column_name!(METADATA_NAME), + column_name!("version"), + ) + })? + .build()?; + + // NOTE: The plan dedupes all actions, so mark all results as coming from checkpoint + let batches = executor + .execute_op(Operation::QueryPlan(plan))? + .into_data()? + .map(|batch| Ok(ActionsBatch::new(batch?, true))); + Ok(Box::new(batches)) + } + // Replay the commit log, projecting rows to only contain Protocol and Metadata action columns. fn read_pm_batches( &self, engine: &dyn Engine, ) -> DeltaResult> + Send> { - let schema = get_commit_schema().project(&[PROTOCOL_NAME, METADATA_NAME])?; + let schema = schema_ref! { + (&PROTOCOL_FIELD), + (&METADATA_FIELD), + }; self.read_actions(engine, schema) } } @@ -146,12 +212,32 @@ impl LogSegment { #[cfg(test)] mod tests { use std::path::PathBuf; + #[cfg(feature = "declarative-plans")] + use std::sync::Arc; use itertools::Itertools; use test_log::test; use crate::engine::sync::SyncEngine; + #[cfg(feature = "declarative-plans")] + use crate::engine::test_delegating::DelegatingEngine; + #[cfg(feature = "declarative-plans")] + use crate::plans::{Operation, PlanExecutor, PlanResult}; use crate::Snapshot; + #[cfg(feature = "declarative-plans")] + use crate::{DeltaResult, Error}; + + // A [`PlanExecutor`] whose every operation fails, used to prove that a plan-path failure + // surfaces from P&M replay rather than falling back to legacy replay. + #[cfg(feature = "declarative-plans")] + struct FailingPlanExecutor; + + #[cfg(feature = "declarative-plans")] + impl PlanExecutor for FailingPlanExecutor { + fn execute_op(&self, _op: Operation) -> DeltaResult { + Err(Error::generic("plan executor deliberately failed")) + } + } // NOTE: In addition to testing the meta-predicate for metadata replay, this test also verifies // that the parquet reader properly infers nullcount = rowcount for missing columns. The two @@ -194,4 +280,60 @@ mod tests { // missing columns, but can still skip part 3 because has valid nullcount stats for P&M. assert_eq!(data.len(), 4); } + + // With the `declarative-plans` feature flag on, `SyncEngine` resolves P&M through the + // declarative plan. + // + // This fixture's checkpoint names its map entry fields `entries` where kernel expects + // `key_value`. Parquet takes that name from the writer's Arrow schema unless the writer sets + // `WriterProperties::coerce_types`, which is off by default, and Arrow's own + // `MapFieldNames::default()` is `entries`. So a writer that builds its maps from Arrow defaults + // produces a file kernel must translate on read. Spark and kernel both write `key_value`, + // covered by + // `scan_plan::execution_tests::declarative_metadata_reconciles_checkpoint_with_later_commits`. + #[test] + fn test_snapshot_build_via_plan_over_parquet_checkpoint_with_entries_named_maps() { + let path = + std::fs::canonicalize(PathBuf::from("./tests/data/app-txn-checkpoint/")).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + let engine = SyncEngine::new(); + + let snapshot = Snapshot::builder_for(url).build(&engine).unwrap(); + + assert_eq!(snapshot.version(), 1); + assert_eq!(snapshot.schema().fields().count(), 3); + } + + // The array counterpart of the test above. This fixture's checkpoint names its array element + // fields `item` where kernel expects `element`, so it covers the other half of the naming + // disagreement. `metaData.partitionColumns` is the array in question, and it is present in + // every `metaData` action, so its element name is checked on every P&M replay. + #[test] + fn test_snapshot_build_via_plan_over_parquet_checkpoint_with_item_named_arrays() { + let path = std::fs::canonicalize(PathBuf::from("./tests/data/parsed-stats/")).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + let engine = SyncEngine::new(); + + let snapshot = Snapshot::builder_for(url).build(&engine).unwrap(); + + assert_eq!(snapshot.version(), 5); + assert_eq!(snapshot.schema().fields().count(), 5); + } + + #[cfg(feature = "declarative-plans")] + #[test] + fn test_snapshot_build_via_failing_plan_executor_surfaces_error_without_fallback() { + let path = + std::fs::canonicalize(PathBuf::from("./tests/data/app-txn-checkpoint/")).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + let engine = DelegatingEngine::new(Arc::new(SyncEngine::new())) + .with_plan_executor(Arc::new(FailingPlanExecutor)); + + let result = Snapshot::builder_for(url).build(&engine); + + assert!( + result.is_err(), + "plan failure must surface, not fall back to legacy replay" + ); + } } diff --git a/kernel/src/log_segment/tests.rs b/kernel/src/log_segment/tests.rs index dc579ee6e1..b6912c07dd 100644 --- a/kernel/src/log_segment/tests.rs +++ b/kernel/src/log_segment/tests.rs @@ -1,48 +1,54 @@ -use std::sync::Arc; -use std::sync::LazyLock; +use std::sync::{Arc, LazyLock}; use itertools::Itertools; +use rstest::rstest; +use test_utils::{ + compacted_log_path_for_versions, delta_path_for_version, staged_commit_path_for_version, +}; use url::Url; -use crate::actions::visitors::AddVisitor; +use super::*; +use crate::actions::visitors::{AddVisitor, SidecarVisitor}; use crate::actions::{ - get_all_actions_schema, get_commit_schema, Add, Sidecar, ADD_NAME, DOMAIN_METADATA_NAME, - METADATA_NAME, PROTOCOL_NAME, REMOVE_NAME, SET_TRANSACTION_NAME, SIDECAR_NAME, + get_all_actions_schema, get_commit_schema, Add, Remove, Sidecar, ADD_NAME, COMMIT_INFO_NAME, + LOG_METADATA_SCHEMA, MAX_VALUES, METADATA_NAME, MIN_VALUES, NUM_RECORDS, REMOVE_NAME, + SIDECAR_NAME, }; use crate::arrow::array::StringArray; use crate::engine::arrow_data::ArrowEngineData; -use crate::engine::default::executor::tokio::TokioBackgroundExecutor; -use crate::engine::default::filesystem::ObjectStoreStorageHandler; -use crate::engine::default::DefaultEngineBuilder; use crate::engine::sync::json::SyncJsonHandler; use crate::engine::sync::SyncEngine; -use crate::expressions::ColumnName; -use crate::last_checkpoint_hint::LastCheckpointHint; +use crate::engine::test_delegating::DelegatingEngine; +use crate::expressions::{col, column_name}; +use crate::last_checkpoint_hint::{LastCheckpointHint, LastCheckpointV2}; use crate::log_replay::ActionsBatch; use crate::log_segment::LogSegment; use crate::log_segment_files::LogSegmentFiles; -use crate::object_store::{memory::InMemory, path::Path, ObjectStore}; +use crate::object_store::memory::InMemory; +use crate::object_store::path::Path; +use crate::object_store::ObjectStoreExt as _; use crate::parquet::arrow::ArrowWriter; +use crate::path::tests::multipart_checkpoint_name; use crate::path::{LogPathFileType, ParsedLogPath}; use crate::scan::test_utils::{ - add_batch_simple, add_batch_with_remove, sidecar_batch_with_given_paths, - sidecar_batch_with_given_paths_and_sizes, + add_batch_simple, add_batch_with_remove, adds_only_batch, remove_only_batch, + sidecar_batch_with_given_paths, sidecar_batch_with_given_paths_and_sizes, }; -use crate::schema::{DataType, StructField, StructType}; -use crate::utils::test_utils::string_array_to_engine_data; -use crate::utils::test_utils::{assert_batch_matches, assert_result_error_with_message, Action}; -use crate::{ - DeltaResult, Engine as _, EngineData, Expression, FileMeta, JsonHandler, Predicate, - PredicateRef, RowVisitor, StorageHandler, +use crate::scan::{ + CHECKPOINT_READ_SCHEMA, CHECKPOINT_READ_SCHEMA_NO_JSON_STATS, COMMIT_READ_SCHEMA, }; -use test_utils::{ - compacted_log_path_for_versions, delta_path_for_version, staged_commit_path_for_version, +use crate::schema::{ + schema, schema_ref, DataType, SchemaRef, SchemaStructPatchBuilder, StructField, StructType, +}; +use crate::unit_test_utils::{ + assert_batch_matches, assert_result_error_with_message, create_log_path, + create_log_path_with_size, string_array_to_engine_data, Action, +}; +use crate::{ + DeltaResult, DeltaResultIteratorStatic, EngineData, FileDataReadResultIterator, FileMeta, + JsonHandler, ParquetFooter, ParquetHandler, Predicate, PredicateRef, RowVisitor, + StorageHandler, }; - -use super::*; - -use crate::actions::visitors::SidecarVisitor; -use crate::ParquetHandler; /// Processes sidecar files for the given checkpoint batch. /// @@ -78,11 +84,11 @@ fn process_sidecars( )?)) } -// get an ObjectStore path for a checkpoint file, based on version, part number, and total number of parts +// get an ObjectStore path for a checkpoint file, based on version, part number, and total number of +// parts fn delta_path_for_multipart_checkpoint(version: u64, part_num: u32, num_parts: u32) -> Path { - let path = - format!("_delta_log/{version:020}.checkpoint.{part_num:010}.{num_parts:010}.parquet"); - Path::from(path.as_str()) + let name = multipart_checkpoint_name(version, part_num, num_parts); + Path::from(format!("_delta_log/{name}").as_str()) } // Utility method to build a log using a list of log paths and an optional checkpoint hint. The @@ -90,7 +96,7 @@ fn delta_path_for_multipart_checkpoint(version: u64, part_num: u32, num_parts: u async fn build_log_with_paths_and_checkpoint( paths: &[Path], checkpoint_metadata: Option<&LastCheckpointHint>, -) -> (Box, Url) { +) -> (Arc, Url) { let store = Arc::new(InMemory::new()); let data = bytes::Bytes::from("kernel-data"); @@ -114,12 +120,12 @@ async fn build_log_with_paths_and_checkpoint( .expect("Write _last_checkpoint"); } - let storage = - ObjectStoreStorageHandler::new(store, Arc::new(TokioBackgroundExecutor::new()), None); + let engine = SyncEngine::new_with_store(store); + let storage = engine.storage_handler(); let table_root = Url::parse("memory:///").expect("valid url"); let log_root = table_root.join("_delta_log/").unwrap(); - (Box::new(storage), log_root) + (storage, log_root) } // Create an in-memory store and return the store and the URL for the store's _delta_log directory. @@ -139,17 +145,7 @@ async fn write_parquet_to_store( path: String, data: Box, ) -> DeltaResult<()> { - let batch = ArrowEngineData::try_from_engine_data(data)?; - let record_batch = batch.record_batch(); - - let mut buffer = vec![]; - let mut writer = ArrowWriter::try_new(&mut buffer, record_batch.schema(), None)?; - writer.write(record_batch)?; - writer.close()?; - - store.put(&Path::from(path), buffer.into()).await?; - - Ok(()) + write_multi_row_group_parquet_to_store(store, vec![data], &path).await } /// Writes all actions to a _delta_log parquet checkpoint file in the store. @@ -163,6 +159,101 @@ pub(crate) async fn add_checkpoint_to_store( write_parquet_to_store(store, path, data).await } +/// Writes a Parquet file with one row group per batch. All batches must share the same Arrow +/// schema. +async fn write_multi_row_group_parquet_to_store( + store: &Arc, + row_groups: Vec>, + path: &str, +) -> DeltaResult<()> { + let batches = row_groups + .into_iter() + .map(ArrowEngineData::try_from_engine_data) + .collect::>>()?; + let schema = batches + .first() + .ok_or_else(|| Error::internal_error("at least one row group is required"))? + .record_batch() + .schema(); + + let mut buffer = vec![]; + let mut writer = ArrowWriter::try_new(&mut buffer, schema, None)?; + for batch in &batches { + writer.write(batch.record_batch())?; + writer.flush()?; + } + writer.close()?; + + store.put(&Path::from(path), buffer.into()).await?; + Ok(()) +} + +/// Returns the materialized row count and sorted paths of all materialized Add actions. +fn collect_materialized_adds( + actions: impl Iterator>, +) -> DeltaResult<(usize, Vec)> { + let mut rows = 0; + let mut add_paths: Vec = Vec::new(); + for batch in actions { + let batch = batch?.actions; + rows += batch.len(); + let mut visitor = AddVisitor::default(); + visitor.visit_rows_of(&*batch)?; + add_paths.extend(visitor.adds.into_iter().map(|add| add.path)); + } + add_paths.sort(); + Ok((rows, add_paths)) +} + +fn collect_projected_adds( + log_segment: &LogSegment, + engine: &dyn Engine, +) -> DeltaResult<(usize, Vec)> { + let actions = log_segment + .read_actions_with_projected_checkpoint_actions( + engine, + COMMIT_READ_SCHEMA.clone(), + CHECKPOINT_READ_SCHEMA.clone(), + None, + None, + None, + None, // cancellation_token + )? + .actions; + collect_materialized_adds(actions) +} + +struct IgnorePredicateParquetHandler(Arc); + +impl ParquetHandler for IgnorePredicateParquetHandler { + fn read_parquet_files( + &self, + files: &[FileMeta], + physical_schema: SchemaRef, + _predicate: Option, + ) -> DeltaResult { + self.0.read_parquet_files(files, physical_schema, None) + } + + fn write_parquet_file( + &self, + location: Url, + data: DeltaResultIteratorStatic>, + ) -> DeltaResult<()> { + self.0.write_parquet_file(location, data) + } + + fn read_parquet_footer(&self, file: &FileMeta) -> DeltaResult { + self.0.read_parquet_footer(file) + } +} + +fn ignore_predicate_engine(sync_engine: &Arc) -> DelegatingEngine { + DelegatingEngine::new(sync_engine.clone()).with_parquet_handler(Arc::new( + IgnorePredicateParquetHandler(sync_engine.parquet_handler()), + )) +} + /// Writes all actions to a _delta_log/_sidecars file in the store and return the [`FileMeta`]. /// This function formats the provided filename into the _sidecars subdirectory. async fn add_sidecar_to_store( @@ -202,18 +293,15 @@ async fn write_json_to_store( Ok(()) } -fn create_log_path(path: &str) -> ParsedLogPath { - create_log_path_with_size(path, 0) -} - -fn create_log_path_with_size(path: &str, size: u64) -> ParsedLogPath { - ParsedLogPath::try_from(FileMeta { - location: Url::parse(path).expect("Invalid file URL"), - last_modified: 0, - size, - }) - .unwrap() - .unwrap() +/// Builds a staged-commit log path (`ParsedLogPath`) for each version: a +/// `memory:///_delta_log/_staged_commits/..json` entry that parses to +/// `file_type: StagedCommit`. +fn staged_commit_log_paths(versions: &[Version]) -> Vec { + versions + .iter() + .map(|v| staged_commit_path_for_version(*v)) + .map(|path| create_log_path_with_size(&format!("memory:///{}", path.as_ref()), 100)) + .collect() } /// Gets the file size from the store for use in FileMeta @@ -299,6 +387,7 @@ async fn build_snapshot_with_uuid_checkpoint_json() { #[tokio::test] async fn build_snapshot_with_correct_last_uuid_checkpoint() { let checkpoint_metadata = LastCheckpointHint { + v2_checkpoint: None, version: 5, size: 10, parts: Some(1), @@ -394,6 +483,7 @@ async fn build_snapshot_with_multiple_incomplete_multipart_checkpoints() { #[tokio::test] async fn build_snapshot_with_out_of_date_last_checkpoint() { let checkpoint_metadata = LastCheckpointHint { + v2_checkpoint: None, version: 3, size: 10, parts: None, @@ -440,6 +530,7 @@ async fn build_snapshot_with_out_of_date_last_checkpoint() { #[tokio::test] async fn build_snapshot_with_correct_last_multipart_checkpoint() { let checkpoint_metadata = LastCheckpointHint { + v2_checkpoint: None, version: 5, size: 10, parts: Some(3), @@ -491,6 +582,7 @@ async fn build_snapshot_with_correct_last_multipart_checkpoint() { #[tokio::test] async fn build_snapshot_with_missing_checkpoint_part_from_hint_fails() { let checkpoint_metadata = LastCheckpointHint { + v2_checkpoint: None, version: 5, size: 10, parts: Some(3), @@ -534,12 +626,22 @@ async fn build_snapshot_with_missing_checkpoint_part_from_hint_fails() { ) } +/// v5 holds three complete checkpoints (classic, 2-part, 3-part), so the 3-part one always wins. +/// The hint's `parts` decides only whether the hint describes that winner. +#[rstest] +#[case::hint_names_winner(Some(3), true)] +#[case::hint_names_losing_multipart(Some(2), false)] +#[case::hint_names_losing_classic(None, false)] #[tokio::test] -async fn build_snapshot_with_bad_checkpoint_hint_fails() { +async fn build_snapshot_applies_checkpoint_hint_iff_it_names_the_selected_checkpoint( + #[case] hint_parts: Option, + #[case] expect_hint_applies: bool, +) { let checkpoint_metadata = LastCheckpointHint { + v2_checkpoint: None, version: 5, size: 10, - parts: Some(1), + parts: hint_parts, size_in_bytes: None, num_of_add_files: None, checkpoint_schema: None, @@ -556,8 +658,12 @@ async fn build_snapshot_with_bad_checkpoint_hint_fails() { delta_path_for_version(3, "checkpoint.parquet"), delta_path_for_version(3, "json"), delta_path_for_version(4, "json"), + delta_path_for_version(5, "checkpoint.parquet"), delta_path_for_multipart_checkpoint(5, 1, 2), delta_path_for_multipart_checkpoint(5, 2, 2), + delta_path_for_multipart_checkpoint(5, 1, 3), + delta_path_for_multipart_checkpoint(5, 2, 3), + delta_path_for_multipart_checkpoint(5, 3, 3), delta_path_for_version(5, "json"), delta_path_for_version(6, "json"), delta_path_for_version(7, "json"), @@ -572,12 +678,38 @@ async fn build_snapshot_with_bad_checkpoint_hint_fails() { vec![], // log_tail Some(checkpoint_metadata), None, - ); - assert_result_error_with_message( - log_segment, - "Invalid Checkpoint: _last_checkpoint indicated that checkpoint should have 1 parts, but \ - it has 2", ) + .unwrap(); + + assert_eq!(log_segment.checkpoint_version, Some(5)); + assert_eq!(log_segment.end_version, 7); + let checkpoint_filenames = log_segment + .listed + .checkpoint_parts + .iter() + .map(|p| p.filename.as_str()) + .collect_vec(); + assert_eq!( + checkpoint_filenames, + vec![ + "00000000000000000005.checkpoint.0000000001.0000000003.parquet", + "00000000000000000005.checkpoint.0000000002.0000000003.parquet", + "00000000000000000005.checkpoint.0000000003.0000000003.parquet", + ] + ); + + // The hint is always retained; describing some other checkpoint only makes its fields + // untrustworthy, and readers fall back to the checkpoint footer. + assert!(log_segment.last_checkpoint_metadata.is_some()); + assert_eq!(log_segment.checkpoint_hint().is_some(), expect_hint_applies); + + let commit_versions = log_segment + .listed + .ascending_commit_files + .iter() + .map(|c| c.version) + .collect_vec(); + assert_eq!(commit_versions, vec![6, 7]); } #[tokio::test] @@ -630,6 +762,7 @@ async fn build_snapshot_with_out_of_date_last_checkpoint_and_incomplete_recent_c // Snapshot should be made of the most recent complete checkpoint and the commit files that // follow it. let checkpoint_metadata = LastCheckpointHint { + v2_checkpoint: None, version: 3, size: 10, parts: None, @@ -741,6 +874,7 @@ async fn build_snapshot_without_checkpoints() { #[tokio::test] async fn build_snapshot_with_checkpoint_greater_than_time_travel_version() { let checkpoint_metadata = LastCheckpointHint { + v2_checkpoint: None, version: 5, size: 10, parts: None, @@ -789,6 +923,7 @@ async fn build_snapshot_with_checkpoint_greater_than_time_travel_version() { #[tokio::test] async fn build_snapshot_with_start_checkpoint_and_time_travel_version() { let checkpoint_metadata = LastCheckpointHint { + v2_checkpoint: None, version: 3, size: 10, parts: None, @@ -828,6 +963,63 @@ async fn build_snapshot_with_start_checkpoint_and_time_travel_version() { assert_eq!(log_segment.listed.ascending_commit_files[0].version, 4); } +#[rstest::rstest] +#[case::no_hint(None)] +#[case::stale_hint(Some(LastCheckpointHint { + v2_checkpoint: None, + version: 10, // stale: 10 > end_version 5, so it is discarded + size: 10, + parts: None, + size_in_bytes: None, + num_of_add_files: None, + checkpoint_schema: None, + checksum: None, + tags: None, +}))] +#[tokio::test] +async fn build_snapshot_time_travel_no_checkpoint_falls_back_to_v0( + #[case] hint: Option, +) { + let paths: Vec = (0..=5).map(|v| delta_path_for_version(v, "json")).collect(); + let (storage, log_root) = build_log_with_paths_and_checkpoint(&paths, None).await; + + let log_segment = + LogSegment::for_snapshot_impl(storage.as_ref(), log_root, vec![], hint, Some(5)).unwrap(); + + let commit_files = log_segment.listed.ascending_commit_files; + let checkpoint_parts = log_segment.listed.checkpoint_parts; + + assert_eq!(checkpoint_parts.len(), 0); + let versions = commit_files.into_iter().map(|x| x.version).collect_vec(); + assert_eq!(versions, vec![0, 1, 2, 3, 4, 5]); +} + +#[tokio::test] +async fn build_snapshot_time_travel_no_hint_checkpoint_at_end_version_included() { + let (storage, log_root) = build_log_with_paths_and_checkpoint( + &[ + delta_path_for_version(0, "json"), + delta_path_for_version(1, "json"), + delta_path_for_version(2, "json"), + delta_path_for_version(3, "json"), + delta_path_for_version(4, "json"), + delta_path_for_version(5, "json"), + delta_path_for_version(5, "checkpoint.parquet"), + ], + None, + ) + .await; + + let log_segment = + LogSegment::for_snapshot_impl(storage.as_ref(), log_root, vec![], None, Some(5)).unwrap(); + + let commit_files = log_segment.listed.ascending_commit_files; + let checkpoint_parts = log_segment.listed.checkpoint_parts; + assert_eq!(checkpoint_parts.len(), 1); + assert_eq!(checkpoint_parts[0].version, 5); + assert_eq!(commit_files.len(), 0); +} + #[tokio::test] async fn build_table_changes_with_commit_versions() { let (storage, log_root) = build_log_with_paths_and_checkpoint( @@ -997,7 +1189,7 @@ fn test_checkpoint_batch_with_no_sidecars_returns_none() -> DeltaResult<()> { #[tokio::test] async fn test_checkpoint_batch_with_sidecars_returns_sidecar_batches() -> DeltaResult<()> { let (store, log_root) = new_in_memory_store(); - let engine = DefaultEngineBuilder::new(store.clone()).build(); + let engine = SyncEngine::new_with_store(store.clone()); let read_schema = get_all_actions_schema().project(&[ADD_NAME, REMOVE_NAME, SIDECAR_NAME])?; let sidecar1_size = add_sidecar_to_store( @@ -1045,7 +1237,7 @@ async fn test_checkpoint_batch_with_sidecars_returns_sidecar_batches() -> DeltaR #[test] fn test_checkpoint_batch_with_sidecar_files_that_do_not_exist() -> DeltaResult<()> { let (store, log_root) = new_in_memory_store(); - let engine = DefaultEngineBuilder::new(store.clone()).build(); + let engine = SyncEngine::new_with_store(store.clone()); let checkpoint_batch = sidecar_batch_with_given_paths( vec!["sidecarfile1.parquet", "sidecarfile2.parquet"], @@ -1064,7 +1256,10 @@ fn test_checkpoint_batch_with_sidecar_files_that_do_not_exist() -> DeltaResult<( // Assert that an error is returned when trying to read sidecar files that do not exist let err = iter.next().unwrap(); - assert_result_error_with_message(err, "Arrow error: External: Object at location _delta_log/_sidecars/sidecarfile1.parquet not found: No data in memory found. Location: _delta_log/_sidecars/sidecarfile1.parquet"); + assert_result_error_with_message( + err, + "File not found: _delta_log/_sidecars/sidecarfile1.parquet", + ); Ok(()) } @@ -1072,7 +1267,7 @@ fn test_checkpoint_batch_with_sidecar_files_that_do_not_exist() -> DeltaResult<( #[tokio::test] async fn test_reading_sidecar_files_with_predicate() -> DeltaResult<()> { let (store, log_root) = new_in_memory_store(); - let engine = DefaultEngineBuilder::new(store.clone()).build(); + let engine = SyncEngine::new_with_store(store.clone()); let read_schema = get_all_actions_schema().project(&[ADD_NAME, REMOVE_NAME, SIDECAR_NAME])?; // Add a sidecar file with only add actions @@ -1090,11 +1285,8 @@ async fn test_reading_sidecar_files_with_predicate() -> DeltaResult<()> { ); // Filter out sidecar files that do not contain remove actions - let remove_predicate: LazyLock> = LazyLock::new(|| { - Some(Arc::new( - Expression::column([REMOVE_NAME, "path"]).is_not_null(), - )) - }); + let remove_predicate: LazyLock> = + LazyLock::new(|| Some(Arc::new(col!(REMOVE_NAME, "path").is_not_null()))); let mut iter = process_sidecars( engine.parquet_handler(), @@ -1116,10 +1308,11 @@ async fn test_reading_sidecar_files_with_predicate() -> DeltaResult<()> { async fn test_create_checkpoint_stream_returns_checkpoint_batches_as_is_if_schema_has_no_file_actions( ) -> DeltaResult<()> { let (store, log_root) = new_in_memory_store(); - let engine = DefaultEngineBuilder::new(store.clone()).build(); + let engine = SyncEngine::new_with_store(store.clone()); add_checkpoint_to_store( &store, - // Create a checkpoint batch with sidecar actions to verify that the sidecar actions are not read. + // Create a checkpoint batch with sidecar actions to verify that the sidecar actions are + // not read. sidecar_batch_with_given_paths(vec!["sidecar1.parquet"], get_commit_schema().clone()), "00000000000000000001.checkpoint.parquet", ) @@ -1129,7 +1322,7 @@ async fn test_create_checkpoint_stream_returns_checkpoint_batches_as_is_if_schem .join("00000000000000000001.checkpoint.parquet")? .to_string(); - let v2_checkpoint_read_schema = get_commit_schema().project(&[METADATA_NAME])?; + let v2_checkpoint_read_schema = LOG_METADATA_SCHEMA.clone(); let log_segment = LogSegment::try_new( LogSegmentFiles { @@ -1147,6 +1340,7 @@ async fn test_create_checkpoint_stream_returns_checkpoint_batches_as_is_if_schem None, // meta_predicate None, // stats_schema None, // partition_schema + None, // cancellation_token )?; let mut iter = checkpoint_result.actions; @@ -1169,7 +1363,7 @@ async fn test_create_checkpoint_stream_returns_checkpoint_batches_as_is_if_schem async fn test_create_checkpoint_stream_returns_checkpoint_batches_if_checkpoint_is_multi_part( ) -> DeltaResult<()> { let (store, log_root) = new_in_memory_store(); - let engine = DefaultEngineBuilder::new(store.clone()).build(); + let engine = SyncEngine::new_with_store(store.clone()); // Multi-part checkpoints should never contain sidecar actions. // This test intentionally includes batches with sidecar actions in multi-part checkpoints @@ -1193,16 +1387,19 @@ async fn test_create_checkpoint_stream_returns_checkpoint_batches_if_checkpoint_ ) .await?; + let cp1_size = get_file_size(&store, &format!("_delta_log/{checkpoint_part_1}")).await; + let cp2_size = get_file_size(&store, &format!("_delta_log/{checkpoint_part_2}")).await; + let checkpoint_one_file = log_root.join(checkpoint_part_1)?.to_string(); let checkpoint_two_file = log_root.join(checkpoint_part_2)?.to_string(); - let v2_checkpoint_read_schema = get_commit_schema().project(&[ADD_NAME])?; + let v2_checkpoint_read_schema = CHECKPOINT_READ_SCHEMA.clone(); let log_segment = LogSegment::try_new( LogSegmentFiles { checkpoint_parts: vec![ - create_log_path(&checkpoint_one_file), - create_log_path(&checkpoint_two_file), + create_log_path_with_size(&checkpoint_one_file, cp1_size), + create_log_path_with_size(&checkpoint_two_file, cp2_size), ], latest_commit_file: Some(create_log_path("file:///00000000000000000001.json")), ..Default::default() @@ -1217,6 +1414,7 @@ async fn test_create_checkpoint_stream_returns_checkpoint_batches_if_checkpoint_ None, // meta_predicate None, // stats_schema None, // partition_schema + None, // cancellation_token )?; let mut iter = checkpoint_result.actions; @@ -1244,7 +1442,7 @@ async fn test_create_checkpoint_stream_returns_checkpoint_batches_if_checkpoint_ async fn test_create_checkpoint_stream_reads_parquet_checkpoint_batch_without_sidecars( ) -> DeltaResult<()> { let (store, log_root) = new_in_memory_store(); - let engine = DefaultEngineBuilder::new(store.clone()).build(); + let engine = SyncEngine::new_with_store(store.clone()); add_checkpoint_to_store( &store, @@ -1282,6 +1480,7 @@ async fn test_create_checkpoint_stream_reads_parquet_checkpoint_batch_without_si None, // meta_predicate None, // stats_schema None, // partition_schema + None, // cancellation_token )?; let mut iter = checkpoint_result.actions; @@ -1297,11 +1496,220 @@ async fn test_create_checkpoint_stream_reads_parquet_checkpoint_batch_without_si Ok(()) } +#[rstest] +#[case::all_remove_part_skipped(vec![remove_only_batch(get_commit_schema().clone()) as _], 0, &[])] +// The single row group holds a live Add, so the whole batch is kept: all 4 rows are materialized +// (`add_batch_with_remove` = 1 remove + 2 adds + 1 metadata). +#[case::mixed_add_remove_part_kept( + vec![add_batch_with_remove(get_commit_schema().clone()) as _], + 4, + &[ + "part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet", + "part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c001.snappy.parquet", + ], +)] +#[case::all_remove_group_skipped_adjacent_add_group_kept( + vec![ + remove_only_batch(get_commit_schema().clone()) as _, + adds_only_batch(get_commit_schema().clone()) as _, + ], + 2, + &[ + "part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet", + "part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c002.snappy.parquet", + ], +)] +#[tokio::test] +async fn test_scan_checkpoint_read_handles_all_remove_row_groups( + #[case] row_groups: Vec>, + #[case] expected_rows_after_pruning: usize, + #[case] expected_add_paths: &[&str], + #[values(false, true)] ignore_predicate: bool, +) -> DeltaResult<()> { + let (store, log_root) = new_in_memory_store(); + let sync_engine = Arc::new(SyncEngine::new_with_store(store.clone())); + let ignore_predicate_engine = ignore_predicate_engine(&sync_engine); + let engine: &dyn Engine = if ignore_predicate { + &ignore_predicate_engine + } else { + sync_engine.as_ref() + }; + + let checkpoint_name = "00000000000000000001.checkpoint.parquet"; + // Real checkpoints store removes with the full file-action schema; the add column is null. + let total_rows: usize = row_groups.iter().map(|batch| batch.len()).sum(); + write_multi_row_group_parquet_to_store( + &store, + row_groups, + &format!("_delta_log/{checkpoint_name}"), + ) + .await?; + + let checkpoint_file = log_root.join(checkpoint_name)?.to_string(); + let checkpoint_size = get_file_size(&store, &format!("_delta_log/{checkpoint_name}")).await; + + let log_segment = LogSegment::try_new( + LogSegmentFiles { + checkpoint_parts: vec![create_log_path_with_size(&checkpoint_file, checkpoint_size)], + latest_commit_file: Some(create_log_path("file:///00000000000000000001.json")), + ..Default::default() + }, + log_root, + None, + None, + )?; + + // The projected checkpoint read derives `add.path IS NOT NULL`. Engines may use it to skip + // all-remove row groups or ignore it and return extra rows; both paths must surface the same + // Adds. + let (materialized_rows, add_paths) = collect_projected_adds(&log_segment, engine)?; + let expected_materialized_rows = if ignore_predicate { + total_rows + } else { + expected_rows_after_pruning + }; + assert_eq!(materialized_rows, expected_materialized_rows); + + let mut expected: Vec = expected_add_paths.iter().map(|s| s.to_string()).collect(); + expected.sort(); + assert_eq!( + add_paths, expected, + "predicate handling must surface every live Add and no others" + ); + + Ok(()) +} + +/// `SyncJsonHandler` ignores the checkpoint predicate, so replay must tolerate the returned remove +/// row while still surfacing the live Add. +#[tokio::test] +async fn test_scan_checkpoint_read_tolerates_unfiltered_json_rows() -> DeltaResult<()> { + let (store, log_root) = new_in_memory_store(); + let engine = SyncEngine::new_with_store(store.clone()); + + let checkpoint_name = + "00000000000000000001.checkpoint.3a0d65cd-4056-49b8-937b-95f9e3ee90e5.json"; + write_json_to_store( + &store, + vec![ + Action::Remove(Remove { + path: "part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c001.snappy.parquet".into(), + data_change: true, + ..Default::default() + }), + Action::Add(Add { + path: "part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet".into(), + data_change: true, + ..Default::default() + }), + ], + checkpoint_name, + ) + .await?; + + let checkpoint_file = log_root.join(checkpoint_name)?.to_string(); + let log_segment = LogSegment::try_new( + LogSegmentFiles { + checkpoint_parts: vec![create_log_path(&checkpoint_file)], + latest_commit_file: Some(create_log_path("file:///00000000000000000001.json")), + ..Default::default() + }, + log_root, + None, + None, + )?; + + let (materialized_rows, add_paths) = collect_projected_adds(&log_segment, &engine)?; + assert_eq!( + materialized_rows, 2, + "SyncJsonHandler should return the unfiltered checkpoint rows" + ); + assert_eq!( + add_paths, + vec!["part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet".to_string()], + ); + + Ok(()) +} + +#[rstest] +#[case::row_group_skipping(false, 2)] +#[case::predicate_ignored(true, 5)] +#[tokio::test] +async fn test_scan_checkpoint_read_handles_all_remove_sidecar_row_groups( + #[case] ignore_predicate: bool, + #[case] expected_materialized_rows: usize, +) -> DeltaResult<()> { + let (store, log_root) = new_in_memory_store(); + let sync_engine = Arc::new(SyncEngine::new_with_store(store.clone())); + let ignore_predicate_engine = ignore_predicate_engine(&sync_engine); + let engine: &dyn Engine = if ignore_predicate { + &ignore_predicate_engine + } else { + sync_engine.as_ref() + }; + + let sidecar_name = "sidecarfile.parquet"; + write_multi_row_group_parquet_to_store( + &store, + vec![ + remove_only_batch(get_commit_schema().clone()) as _, + adds_only_batch(get_commit_schema().clone()) as _, + ], + &format!("_delta_log/_sidecars/{sidecar_name}"), + ) + .await?; + let sidecar_size = get_file_size(&store, &format!("_delta_log/_sidecars/{sidecar_name}")).await; + + let checkpoint_name = "00000000000000000001.checkpoint.parquet"; + add_checkpoint_to_store( + &store, + sidecar_batch_with_given_paths_and_sizes( + vec![(sidecar_name, sidecar_size)], + get_all_actions_schema().clone(), + ), + checkpoint_name, + ) + .await?; + let checkpoint_file = log_root.join(checkpoint_name)?.to_string(); + let checkpoint_size = get_file_size(&store, &format!("_delta_log/{checkpoint_name}")).await; + + let log_segment = LogSegment::try_new( + LogSegmentFiles { + checkpoint_parts: vec![create_log_path_with_size(&checkpoint_file, checkpoint_size)], + latest_commit_file: Some(create_log_path("file:///00000000000000000001.json")), + ..Default::default() + }, + log_root, + None, + None, + )?; + + // Sidecar discovery reads the manifest without a predicate, so pruning its null-`add.path` + // rows from the projected action stream cannot hide sidecar references. + let (materialized_rows, add_paths) = collect_projected_adds(&log_segment, engine)?; + assert_eq!( + materialized_rows, expected_materialized_rows, + "materialized rows must reflect whether the engine applies the predicate" + ); + + assert_eq!( + add_paths, + vec![ + "part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet".to_string(), + "part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c002.snappy.parquet".to_string(), + ], + "the kept sidecar row group must surface every live Add and no others" + ); + + Ok(()) +} + #[tokio::test] async fn test_create_checkpoint_stream_reads_json_checkpoint_batch_without_sidecars( ) -> DeltaResult<()> { let (store, log_root) = new_in_memory_store(); - let engine = DefaultEngineBuilder::new(store.clone()).build(); + let engine = SyncEngine::new_with_store(store.clone()); let filename = "00000000000000000010.checkpoint.80a083e8-7026-4e79-81be-64bd76c43a11.json"; @@ -1323,7 +1731,6 @@ async fn test_create_checkpoint_stream_reads_json_checkpoint_batch_without_sidec let log_segment = LogSegment::try_new( LogSegmentFiles { checkpoint_parts: vec![create_log_path(&checkpoint_one_file)], - latest_commit_file: Some(create_log_path("file:///00000000000000000001.json")), ..Default::default() }, log_root, @@ -1336,6 +1743,7 @@ async fn test_create_checkpoint_stream_reads_json_checkpoint_batch_without_sidec None, // meta_predicate None, // stats_schema None, // partition_schema + None, // cancellation_token )?; let mut iter = checkpoint_result.actions; @@ -1365,12 +1773,12 @@ async fn test_create_checkpoint_stream_reads_json_checkpoint_batch_without_sidec async fn test_create_checkpoint_stream_reads_checkpoint_file_and_returns_sidecar_batches( ) -> DeltaResult<()> { let (store, log_root) = new_in_memory_store(); - let engine = DefaultEngineBuilder::new(store.clone()).build(); + let engine = SyncEngine::new_with_store(store.clone()); // Write sidecars first so we can get their actual sizes let sidecar1_size = add_sidecar_to_store( &store, - add_batch_simple(get_commit_schema().project(&[ADD_NAME, REMOVE_NAME])?), + add_batch_simple(COMMIT_READ_SCHEMA.clone()), "sidecarfile1.parquet", ) .await? @@ -1378,7 +1786,7 @@ async fn test_create_checkpoint_stream_reads_checkpoint_file_and_returns_sidecar let sidecar2_size = add_sidecar_to_store( &store, - add_batch_with_remove(get_commit_schema().project(&[ADD_NAME, REMOVE_NAME])?), + add_batch_with_remove(COMMIT_READ_SCHEMA.clone()), "sidecarfile2.parquet", ) .await? @@ -1428,6 +1836,7 @@ async fn test_create_checkpoint_stream_reads_checkpoint_file_and_returns_sidecar None, // meta_predicate None, // stats_schema None, // partition_schema + None, // cancellation_token )?; let mut iter = checkpoint_result.actions; @@ -1504,21 +1913,7 @@ async fn create_segment_for(segment: LogSegmentConfig<'_>) -> LogSegment { )); } let (storage, log_root) = build_log_with_paths_and_checkpoint(&paths, None).await; - let table_root = Url::parse("memory:///").expect("valid url"); - let staged_commits_log_tail: Vec = segment - .staged_commit_versions - .iter() - .map(|version| staged_commit_path_for_version(*version)) - .map(|path| { - ParsedLogPath::try_from(FileMeta { - location: table_root.join(path.as_ref()).unwrap(), - last_modified: 0, - size: 0, - }) - .unwrap() - .unwrap() - }) - .collect(); + let staged_commits_log_tail = staged_commit_log_paths(segment.staged_commit_versions); LogSegment::for_snapshot_impl( storage.as_ref(), log_root.clone(), @@ -1625,6 +2020,7 @@ async fn test_compaction_listing( } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_compaction_simple() { test_compaction_listing( &[0, 1, 2], @@ -1636,6 +2032,7 @@ async fn test_compaction_simple() { } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_compaction_in_version_range() { test_compaction_listing( &[0, 1, 2, 3], @@ -1647,6 +2044,7 @@ async fn test_compaction_in_version_range() { } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_compaction_out_of_version_range() { test_compaction_listing( &[0, 1, 2, 3, 4], @@ -1658,6 +2056,7 @@ async fn test_compaction_out_of_version_range() { } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_multi_compaction() { test_compaction_listing( &[0, 1, 2, 3, 4, 5], @@ -1669,6 +2068,7 @@ async fn test_multi_compaction() { } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_multi_compaction_one_out_of_range() { test_compaction_listing( &[0, 1, 2, 3, 4, 5], @@ -1680,6 +2080,7 @@ async fn test_multi_compaction_one_out_of_range() { } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_compaction_with_checkpoint() { test_compaction_listing( &[0, 1, 2, 4, 5], @@ -1691,6 +2092,7 @@ async fn test_compaction_with_checkpoint() { } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_compaction_to_early_with_checkpoint() { test_compaction_listing( &[0, 1, 2, 4, 5], @@ -1702,6 +2104,7 @@ async fn test_compaction_to_early_with_checkpoint() { } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_compaction_starts_at_checkpoint() { test_compaction_listing( &[0, 1, 2, 4, 5], @@ -1752,6 +2155,7 @@ async fn test_commit_cover( } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_commit_cover_one_compaction() { test_commit_cover( &[0, 1, 2], @@ -1764,6 +2168,7 @@ async fn test_commit_cover_one_compaction() { } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_commit_cover_in_version_range() { test_commit_cover( &[0, 1, 2, 3], @@ -1792,6 +2197,7 @@ async fn test_commit_cover_out_of_version_range() { } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_commit_cover_multi_compaction() { test_commit_cover( &[0, 1, 2, 3, 4, 5], @@ -1808,6 +2214,7 @@ async fn test_commit_cover_multi_compaction() { } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_commit_cover_multi_compaction_one_out_of_range() { test_commit_cover( &[0, 1, 2, 3, 4, 5], @@ -1825,6 +2232,7 @@ async fn test_commit_cover_multi_compaction_one_out_of_range() { } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_commit_cover_compaction_with_checkpoint() { test_commit_cover( &[0, 1, 2, 4, 5], @@ -1837,6 +2245,7 @@ async fn test_commit_cover_compaction_with_checkpoint() { } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_commit_cover_too_early_with_checkpoint() { test_commit_cover( &[0, 1, 2, 4, 5], @@ -1849,6 +2258,7 @@ async fn test_commit_cover_too_early_with_checkpoint() { } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_commit_cover_starts_at_checkpoint() { test_commit_cover( &[0, 1, 2, 4, 5], @@ -1861,6 +2271,7 @@ async fn test_commit_cover_starts_at_checkpoint() { } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_commit_cover_wider_range() { test_commit_cover( &Vec::from_iter(0..20), @@ -1895,6 +2306,7 @@ async fn test_commit_cover_no_compactions() { } #[tokio::test] +#[ignore = "log compaction disabled (#2337)"] async fn test_commit_cover_minimal_overlap() { test_commit_cover( &Vec::from_iter(0..6), @@ -1911,7 +2323,57 @@ async fn test_commit_cover_minimal_overlap() { .await; } +#[tokio::test] +async fn test_commit_cover_zero_byte_compaction_uses_commits() { + let store = Arc::new(InMemory::new()); + + let commit_data = bytes::Bytes::from("kernel-data"); + for v in 0..=4u64 { + let path = delta_path_for_version(v, "json"); + store + .put(&path, commit_data.clone().into()) + .await + .expect("put commit"); + } + // Write a 0-byte compaction file covering v0-v4 + let compaction_path = compacted_log_path_for_versions(0, 4, "json"); + store + .put(&compaction_path, bytes::Bytes::new().into()) + .await + .expect("put empty compaction"); + + let engine = SyncEngine::new_with_store(store); + let table_root = Url::parse("memory:///").expect("valid url"); + let log_root = table_root.join("_delta_log/").unwrap(); + + let log_segment = LogSegment::for_snapshot_impl( + engine.storage_handler().as_ref(), + log_root.clone(), + vec![], + None, + None, + ) + .unwrap(); + + assert_eq!( + log_segment.listed.ascending_compaction_files.len(), + 0, + "0-byte compaction should have been filtered at listing time" + ); + + let cover = log_segment.find_commit_cover(); + assert_eq!(cover.len(), 5); + for (i, file) in cover.iter().enumerate() { + let expected_version = 4 - i as u64; + let expected_url = log_root + .join(&format!("{expected_version:020}.json")) + .unwrap(); + assert_eq!(file.location, expected_url); + } +} + #[test] +#[ignore = "log compaction disabled (#2337)"] fn test_validate_listed_log_file_in_order_compaction_files() { let log_root = Url::parse("file:///_delta_log/").unwrap(); assert!(LogSegment::try_new( @@ -1937,6 +2399,7 @@ fn test_validate_listed_log_file_in_order_compaction_files() { } #[test] +#[ignore = "log compaction disabled (#2337)"] fn test_validate_listed_log_file_out_of_order_compaction_files() { let log_root = Url::parse("file:///_delta_log/").unwrap(); assert!(LogSegment::try_new( @@ -2002,20 +2465,92 @@ fn test_validate_listed_log_file_out_of_order_commit_files() { } #[test] -fn test_validate_listed_log_file_checkpoint_parts_contains_non_checkpoint() { +fn test_try_new_crc_at_end_version_is_ok() { let log_root = Url::parse("file:///_delta_log/").unwrap(); assert!(LogSegment::try_new( LogSegmentFiles { - checkpoint_parts: vec![create_log_path( - "file:///_delta_log/00000000000000000010.json", + ascending_commit_files: vec![create_log_path( + "file:///_delta_log/00000000000000000002.json", )], + latest_commit_file: Some(create_log_path( + "file:///_delta_log/00000000000000000002.json", + )), + latest_crc_file: Some(create_log_path( + "file:///_delta_log/00000000000000000002.crc" + )), ..Default::default() }, log_root, None, None, ) - .is_err()); + .is_ok()); +} + +#[test] +fn test_try_new_crc_newer_than_end_version_is_err() { + let log_root = Url::parse("file:///_delta_log/").unwrap(); + assert!(LogSegment::try_new( + LogSegmentFiles { + ascending_commit_files: vec![create_log_path( + "file:///_delta_log/00000000000000000002.json", + )], + latest_commit_file: Some(create_log_path( + "file:///_delta_log/00000000000000000002.json", + )), + latest_crc_file: Some(create_log_path( + "file:///_delta_log/00000000000000000003.crc" + )), + ..Default::default() + }, + log_root, + None, + None, + ) + .is_err()); +} + +#[test] +fn test_try_new_crc_older_than_checkpoint_is_err() { + let log_root = Url::parse("file:///_delta_log/").unwrap(); + assert!(LogSegment::try_new( + LogSegmentFiles { + ascending_commit_files: vec![create_log_path( + "file:///_delta_log/00000000000000000004.json", + )], + latest_commit_file: Some(create_log_path( + "file:///_delta_log/00000000000000000004.json", + )), + checkpoint_parts: vec![create_log_path( + "file:///_delta_log/00000000000000000003.checkpoint.parquet", + )], + latest_crc_file: Some(create_log_path( + "file:///_delta_log/00000000000000000002.crc" + )), + ..Default::default() + }, + log_root, + None, + None, + ) + .is_err()); +} + +#[test] +fn test_validate_listed_log_file_checkpoint_parts_contains_non_checkpoint() { + let log_root = Url::parse("file:///_delta_log/").unwrap(); + assert!(LogSegment::try_new( + LogSegmentFiles { + checkpoint_parts: vec![create_log_path( + "file:///_delta_log/00000000000000000010.json", + )], + ..Default::default() + }, + log_root, + None, + None, + ) + .is_err()); } #[test] @@ -2061,7 +2596,7 @@ fn test_validate_listed_log_file_single_multipart_checkpoint_num_parts_mismatch( #[test] fn test_validate_listed_log_file_multiple_single_part_checkpoints() { - // Two SinglePartCheckpoints at the same version: n=2 but neither is a MultiPartCheckpoint + // Two ClassicCheckpoints at the same version: n=2 but neither is a MultiPartCheckpoint let log_root = Url::parse("file:///_delta_log/").unwrap(); assert!(LogSegment::try_new( LogSegmentFiles { @@ -2096,6 +2631,7 @@ fn test_validate_listed_log_file_commit_files_contains_non_commit() { } #[test] +#[ignore = "log compaction disabled (#2337)"] fn test_validate_listed_log_file_compaction_files_contains_non_compaction() { let log_root = Url::parse("file:///_delta_log/").unwrap(); assert!(LogSegment::try_new( @@ -2116,6 +2652,7 @@ fn test_validate_listed_log_file_compaction_files_contains_non_compaction() { } #[test] +#[ignore = "log compaction disabled (#2337)"] fn test_validate_listed_log_file_compaction_start_exceeds_end() { // A compaction file where the start version is greater than the end version let log_root = Url::parse("file:///_delta_log/").unwrap(); @@ -2147,6 +2684,10 @@ async fn commits_since() { assert_eq!(log_segment.commits_since_checkpoint(), 4); assert_eq!(log_segment.commits_since_log_compaction_or_checkpoint(), 4); + // TODO(#2337): restore original expected values when log compaction is re-enabled. + // Compaction files are currently skipped during listing, so + // commits_since_log_compaction_or_checkpoint() equals commits_since_checkpoint(). + // with compaction, no checkpoint let log_segment = create_segment_for(LogSegmentConfig { published_commit_versions: &Vec::from_iter(0..=4), @@ -2155,7 +2696,7 @@ async fn commits_since() { }) .await; assert_eq!(log_segment.commits_since_checkpoint(), 4); - assert_eq!(log_segment.commits_since_log_compaction_or_checkpoint(), 2); + assert_eq!(log_segment.commits_since_log_compaction_or_checkpoint(), 4); // checkpoint, no compaction let log_segment = create_segment_for(LogSegmentConfig { @@ -2187,7 +2728,7 @@ async fn commits_since() { }) .await; assert_eq!(log_segment.commits_since_checkpoint(), 4); - assert_eq!(log_segment.commits_since_log_compaction_or_checkpoint(), 2); + assert_eq!(log_segment.commits_since_log_compaction_or_checkpoint(), 4); // multiple compactions let log_segment = create_segment_for(LogSegmentConfig { @@ -2197,7 +2738,7 @@ async fn commits_since() { }) .await; assert_eq!(log_segment.commits_since_checkpoint(), 6); - assert_eq!(log_segment.commits_since_log_compaction_or_checkpoint(), 2); + assert_eq!(log_segment.commits_since_log_compaction_or_checkpoint(), 6); // multiple compactions, out of order let log_segment = create_segment_for(LogSegmentConfig { @@ -2207,172 +2748,100 @@ async fn commits_since() { }) .await; assert_eq!(log_segment.commits_since_checkpoint(), 10); - assert_eq!(log_segment.commits_since_log_compaction_or_checkpoint(), 1); -} - -#[tokio::test] -async fn for_timestamp_conversion_gets_commit_range() { - let (storage, log_root) = build_log_with_paths_and_checkpoint( - &[ - delta_path_for_version(0, "json"), - delta_path_for_version(1, "json"), - delta_path_for_version(1, "checkpoint.parquet"), - delta_path_for_version(2, "json"), - delta_path_for_version(3, "json"), - delta_path_for_version(3, "checkpoint.parquet"), - delta_path_for_version(4, "json"), - delta_path_for_version(5, "json"), - delta_path_for_version(5, "checkpoint.parquet"), - delta_path_for_version(6, "json"), - delta_path_for_version(7, "json"), - ], - None, - ) - .await; - - let log_segment = - LogSegment::for_timestamp_conversion(storage.as_ref(), log_root.clone(), 7, None).unwrap(); - let commit_files = log_segment.listed.ascending_commit_files; - let checkpoint_parts = log_segment.listed.checkpoint_parts; - - assert!(checkpoint_parts.is_empty()); - - let versions = commit_files.iter().map(|x| x.version).collect_vec(); - assert_eq!(vec![0, 1, 2, 3, 4, 5, 6, 7], versions); -} - -#[tokio::test] -async fn for_timestamp_conversion_with_old_end_version() { - let (storage, log_root) = build_log_with_paths_and_checkpoint( - &[ - delta_path_for_version(0, "json"), - delta_path_for_version(1, "json"), - delta_path_for_version(1, "checkpoint.parquet"), - delta_path_for_version(2, "json"), - delta_path_for_version(3, "json"), - delta_path_for_version(3, "checkpoint.parquet"), - delta_path_for_version(4, "json"), - delta_path_for_version(5, "json"), - delta_path_for_version(5, "checkpoint.parquet"), - delta_path_for_version(6, "json"), - delta_path_for_version(7, "json"), - ], - None, - ) - .await; - - let log_segment = - LogSegment::for_timestamp_conversion(storage.as_ref(), log_root.clone(), 5, None).unwrap(); - let commit_files = log_segment.listed.ascending_commit_files; - let checkpoint_parts = log_segment.listed.checkpoint_parts; - - assert!(checkpoint_parts.is_empty()); - - let versions = commit_files.iter().map(|x| x.version).collect_vec(); - assert_eq!(vec![0, 1, 2, 3, 4, 5], versions); -} - -#[tokio::test] -async fn for_timestamp_conversion_only_contiguous_ranges() { - let (storage, log_root) = build_log_with_paths_and_checkpoint( - &[ - delta_path_for_version(0, "json"), - delta_path_for_version(1, "json"), - delta_path_for_version(1, "checkpoint.parquet"), - delta_path_for_version(2, "json"), - delta_path_for_version(3, "json"), - delta_path_for_version(3, "checkpoint.parquet"), - // version 4 is missing - delta_path_for_version(5, "json"), - delta_path_for_version(5, "checkpoint.parquet"), - delta_path_for_version(6, "json"), - delta_path_for_version(7, "json"), - ], - None, - ) - .await; - - let log_segment = - LogSegment::for_timestamp_conversion(storage.as_ref(), log_root.clone(), 7, None).unwrap(); - let commit_files = log_segment.listed.ascending_commit_files; - let checkpoint_parts = log_segment.listed.checkpoint_parts; - - assert!(checkpoint_parts.is_empty()); - - let versions = commit_files.iter().map(|x| x.version).collect_vec(); - assert_eq!(vec![5, 6, 7], versions); -} - -#[tokio::test] -async fn for_timestamp_conversion_with_limit() { - let (storage, log_root) = build_log_with_paths_and_checkpoint( - &[ - delta_path_for_version(0, "json"), - delta_path_for_version(1, "json"), - delta_path_for_version(1, "checkpoint.parquet"), - delta_path_for_version(2, "json"), - delta_path_for_version(3, "json"), - delta_path_for_version(3, "checkpoint.parquet"), - delta_path_for_version(4, "json"), - delta_path_for_version(5, "json"), - delta_path_for_version(5, "checkpoint.parquet"), - delta_path_for_version(6, "json"), - delta_path_for_version(7, "json"), - ], - None, - ) - .await; - - let log_segment = LogSegment::for_timestamp_conversion( - storage.as_ref(), - log_root.clone(), - 7, - Some(NonZero::new(3).unwrap()), - ) - .unwrap(); - let commit_files = log_segment.listed.ascending_commit_files; - let checkpoint_parts = log_segment.listed.checkpoint_parts; - - assert!(checkpoint_parts.is_empty()); - - let versions = commit_files.iter().map(|x| x.version).collect_vec(); - assert_eq!(vec![5, 6, 7], versions); -} - + assert_eq!(log_segment.commits_since_log_compaction_or_checkpoint(), 10); +} + +/// A log built from `published` commit versions, `checkpoints`, and catalog-supplied `staged` +/// commit versions, queried up to `end_version` (optionally bounded by `limit`), expecting +/// `expected` commit versions in the resulting segment. +struct TimestampConversionCase { + published: &'static [Version], + checkpoints: &'static [Version], + staged: &'static [Version], + end_version: Version, + limit: Option, + expected: &'static [Version], +} + +/// `for_timestamp_conversion` returns the latest contiguous run of commit files (no checkpoint +/// parts), merging any caller-provided `log_tail` over the filesystem listing. +#[rstest] +// Filesystem-only (no log_tail): +#[case::full_range(TimestampConversionCase { + published: &[0, 1, 2, 3, 4, 5, 6, 7], checkpoints: &[1, 3, 5], staged: &[], + end_version: 7, limit: None, expected: &[0, 1, 2, 3, 4, 5, 6, 7], +})] +#[case::old_end_version(TimestampConversionCase { + published: &[0, 1, 2, 3, 4, 5, 6, 7], checkpoints: &[1, 3, 5], staged: &[], + end_version: 5, limit: None, expected: &[0, 1, 2, 3, 4, 5], +})] +#[case::only_contiguous_ranges(TimestampConversionCase { + published: &[0, 1, 2, 3, 5, 6, 7], checkpoints: &[1, 3, 5], staged: &[], + end_version: 7, limit: None, expected: &[5, 6, 7], +})] +#[case::with_limit(TimestampConversionCase { + published: &[0, 1, 2, 3, 4, 5, 6, 7], checkpoints: &[1, 3, 5], staged: &[], + end_version: 7, limit: Some(3), expected: &[5, 6, 7], +})] +#[case::with_large_limit(TimestampConversionCase { + published: &[0, 1, 2, 3, 4, 5, 6, 7], checkpoints: &[1, 3, 5], staged: &[], + end_version: 7, limit: Some(20), expected: &[0, 1, 2, 3, 4, 5, 6, 7], +})] +// Catalog-managed staged commits supplied via the log_tail: +#[case::log_tail_joins_prefix(TimestampConversionCase { + published: &[0, 1], checkpoints: &[], staged: &[2, 3], + end_version: 3, limit: None, expected: &[0, 1, 2, 3], +})] +#[case::log_tail_gap_drops_prefix(TimestampConversionCase { + published: &[0], checkpoints: &[], staged: &[2, 3], + end_version: 3, limit: None, expected: &[2, 3], +})] +#[case::log_tail_across_checkpoint(TimestampConversionCase { + published: &[0, 1, 2], checkpoints: &[2], staged: &[3, 4], + end_version: 4, limit: None, expected: &[0, 1, 2, 3, 4], +})] +#[case::log_tail_with_limit(TimestampConversionCase { + published: &[0, 1, 2, 3], checkpoints: &[], staged: &[4, 5], + end_version: 5, limit: Some(3), expected: &[3, 4, 5], +})] #[tokio::test] -async fn for_timestamp_conversion_with_large_limit() { - let (storage, log_root) = build_log_with_paths_and_checkpoint( - &[ - delta_path_for_version(0, "json"), - delta_path_for_version(1, "json"), - delta_path_for_version(1, "checkpoint.parquet"), - delta_path_for_version(2, "json"), - delta_path_for_version(3, "json"), - delta_path_for_version(3, "checkpoint.parquet"), - delta_path_for_version(4, "json"), - delta_path_for_version(5, "json"), - delta_path_for_version(5, "checkpoint.parquet"), - delta_path_for_version(6, "json"), - delta_path_for_version(7, "json"), - ], - None, - ) - .await; +async fn for_timestamp_conversion_cases(#[case] case: TimestampConversionCase) { + let TimestampConversionCase { + published, + checkpoints, + staged, + end_version, + limit, + expected, + } = case; + let mut paths: Vec = published + .iter() + .map(|v| delta_path_for_version(*v, "json")) + .collect(); + paths.extend( + checkpoints + .iter() + .map(|v| delta_path_for_version(*v, "checkpoint.parquet")), + ); + let (storage, log_root) = build_log_with_paths_and_checkpoint(&paths, None).await; let log_segment = LogSegment::for_timestamp_conversion( storage.as_ref(), log_root.clone(), - 7, - Some(NonZero::new(20).unwrap()), + end_version, + limit.map(|l| NonZero::new(l).unwrap()), + staged_commit_log_paths(staged), ) .unwrap(); - let commit_files = log_segment.listed.ascending_commit_files; - let checkpoint_parts = log_segment.listed.checkpoint_parts; - assert!(checkpoint_parts.is_empty()); - - let versions = commit_files.iter().map(|x| x.version).collect_vec(); - assert_eq!(vec![0, 1, 2, 3, 4, 5, 6, 7], versions); + assert!(log_segment.listed.checkpoint_parts.is_empty()); + let versions = log_segment + .listed + .ascending_commit_files + .iter() + .map(|x| x.version) + .collect_vec(); + assert_eq!(expected, versions.as_slice()); } #[tokio::test] @@ -2383,7 +2852,8 @@ async fn for_timestamp_conversion_no_commit_files() { ) .await; - let res = LogSegment::for_timestamp_conversion(storage.as_ref(), log_root.clone(), 0, None); + let res = + LogSegment::for_timestamp_conversion(storage.as_ref(), log_root.clone(), 0, None, vec![]); assert_result_error_with_message(res, "Generic delta kernel error: No files in log segment"); } @@ -2404,9 +2874,14 @@ async fn test_latest_commit_file_field_is_captured() { ) .await; - let log_segment = - LogSegment::for_snapshot(storage.as_ref(), log_root.clone(), vec![], None, None, None) - .unwrap(); + let log_segment = LogSegment::for_snapshot( + storage.as_ref(), + log_root.clone(), + vec![], + None, + SnapshotLoadMetricContext::for_test(), + ) + .unwrap(); // The latest commit should be version 5 assert_eq!(log_segment.listed.latest_commit_file.unwrap().version, 5); @@ -2432,9 +2907,14 @@ async fn test_latest_commit_file_with_checkpoint_filtering() { ) .await; - let log_segment = - LogSegment::for_snapshot(storage.as_ref(), log_root.clone(), vec![], None, None, None) - .unwrap(); + let log_segment = LogSegment::for_snapshot( + storage.as_ref(), + log_root.clone(), + vec![], + None, + SnapshotLoadMetricContext::for_test(), + ) + .unwrap(); // The latest commit should be version 4 assert_eq!(log_segment.listed.latest_commit_file.unwrap().version, 4); @@ -2454,9 +2934,14 @@ async fn test_latest_commit_file_with_no_commits() { ) .await; - let log_segment = - LogSegment::for_snapshot(storage.as_ref(), log_root.clone(), vec![], None, None, None) - .unwrap(); + let log_segment = LogSegment::for_snapshot( + storage.as_ref(), + log_root.clone(), + vec![], + None, + SnapshotLoadMetricContext::for_test(), + ) + .unwrap(); // latest_commit_file should be None when there are no commits assert!(log_segment.listed.latest_commit_file.is_none()); @@ -2479,9 +2964,14 @@ async fn test_latest_commit_file_with_checkpoint_at_same_version() { ) .await; - let log_segment = - LogSegment::for_snapshot(storage.as_ref(), log_root.clone(), vec![], None, None, None) - .unwrap(); + let log_segment = LogSegment::for_snapshot( + storage.as_ref(), + log_root.clone(), + vec![], + None, + SnapshotLoadMetricContext::for_test(), + ) + .unwrap(); // The latest commit should be version 1 (saved before filtering) assert_eq!(log_segment.listed.latest_commit_file.unwrap().version, 1); @@ -2506,9 +2996,14 @@ async fn test_latest_commit_file_edge_case_commit_before_checkpoint() { ) .await; - let log_segment = - LogSegment::for_snapshot(storage.as_ref(), log_root.clone(), vec![], None, None, None) - .unwrap(); + let log_segment = LogSegment::for_snapshot( + storage.as_ref(), + log_root.clone(), + vec![], + None, + SnapshotLoadMetricContext::for_test(), + ) + .unwrap(); // latest_commit_file should be None since there's no commit at the checkpoint version assert!(log_segment.listed.latest_commit_file.is_none()); @@ -2532,6 +3027,9 @@ fn test_log_segment_contiguous_commit_files() { create_log_path("file:///_delta_log/00000000000000000002.json"), create_log_path("file:///_delta_log/00000000000000000003.json"), ], + latest_commit_file: Some(create_log_path( + "file:///_delta_log/00000000000000000003.json", + )), ..Default::default() }, log_root.clone(), @@ -2568,101 +3066,267 @@ fn test_log_segment_contiguous_commit_files() { ); } -/// Test that checkpoint_schema from _last_checkpoint hint is properly propagated to LogSegment -#[tokio::test] -async fn test_checkpoint_schema_propagation_from_hint() { - use crate::schema::{StructField, StructType}; +/// `checkpoint_sidecars()` distinguishes "the matched hint lists zero sidecars" (`Some(&[])`) from +/// "no applicable hint / no sidecar info" (`None`) -- the empty-vs-absent contract the accessor's +/// doc promises. Real V2 fixtures only carry non-empty sidecar lists, so this synthetic case is the +/// only place it is exercised. +#[test] +fn checkpoint_sidecars_distinguishes_empty_from_absent() -> DeltaResult<()> { + let (_store, log_root) = new_in_memory_store(); + let selected = "00000000000000000001.checkpoint.11111111-1111-1111-1111-111111111111.parquet"; + let checkpoint_file = log_root.join(selected)?.to_string(); + let commit = create_log_path(log_root.join("00000000000000000002.json")?.as_str()); + let log_segment = LogSegment::try_new( + LogSegmentFiles { + checkpoint_parts: vec![create_log_path_with_size(&checkpoint_file, 1)], + ascending_commit_files: vec![commit.clone()], + latest_commit_file: Some(commit), + ..Default::default() + }, + log_root, + None, + Some(LastCheckpointHint { + version: 1, + v2_checkpoint: Some(LastCheckpointV2 { + path: selected.to_string(), + sidecar_files: Some(vec![]), + ..Default::default() + }), + ..Default::default() + }), + )?; - // Create a sample schema that would be in _last_checkpoint - let sample_schema: SchemaRef = Arc::new(StructType::new_unchecked([ - StructField::nullable("add", StructType::new_unchecked([])), - StructField::nullable("remove", StructType::new_unchecked([])), - ])); + assert_eq!(log_segment.checkpoint_hint_sidecars(), Some(&vec![])); + Ok(()) +} - let checkpoint_metadata = LastCheckpointHint { - version: 5, - size: 10, - parts: Some(1), - size_in_bytes: None, - num_of_add_files: None, - checkpoint_schema: Some(sample_schema.clone()), - checksum: None, - tags: None, - }; +/// Checkpoint schema resolution uses the `_last_checkpoint` schema only when the hint's version +/// matches [`LogSegment::checkpoint_version`]. Otherwise the parquet footer is read. +#[rstest] +#[case::hint_matches_checkpoint(1, true)] +#[case::hint_newer_than_checkpoint(99, false)] +#[case::hint_older_than_checkpoint(0, false)] +#[tokio::test] +async fn test_get_file_actions_schema_v1_parquet_with_hint( + #[case] hint_version: u64, + #[case] expect_hint_schema_used: bool, +) -> DeltaResult<()> { + let (store, log_root) = new_in_memory_store(); + let engine = SyncEngine::new_with_store(store.clone()); - let (storage, log_root) = build_log_with_paths_and_checkpoint( - &[ - delta_path_for_version(0, "json"), - delta_path_for_version(5, "checkpoint.parquet"), - delta_path_for_version(5, "json"), - delta_path_for_version(6, "json"), - ], - Some(&checkpoint_metadata), + // Build a checkpoint with an initial v1 schema + let v1_schema = COMMIT_READ_SCHEMA.clone(); + add_checkpoint_to_store( + &store, + add_batch_simple(v1_schema.clone()), + "00000000000000000001.checkpoint.parquet", ) - .await; + .await?; - let log_segment = LogSegment::for_snapshot_impl( - storage.as_ref(), + let checkpoint_rel = "00000000000000000001.checkpoint.parquet"; + let checkpoint_file = log_root.join(checkpoint_rel)?.to_string(); + let cp_size = get_file_size(&store, &format!("_delta_log/{checkpoint_rel}")).await; + + let hint_schema: SchemaRef = schema_ref! { + nullable "metadata": {}, + }; + + // Build a commit that uses v1 checkpoint and a hint that describes a different schema + let commit_v2_path = log_root.join("00000000000000000002.json")?.to_string(); + let commit_v2 = create_log_path(&commit_v2_path); + let log_segment = LogSegment::try_new( + LogSegmentFiles { + checkpoint_parts: vec![create_log_path_with_size(&checkpoint_file, cp_size)], + ascending_commit_files: vec![commit_v2.clone()], + latest_commit_file: Some(commit_v2), + ..Default::default() + }, log_root, - vec![], // log_tail - Some(checkpoint_metadata), None, - ) - .unwrap(); + Some(LastCheckpointHint { + version: hint_version, + checkpoint_schema: Some(hint_schema.clone()), + ..Default::default() + }), + )?; - // Verify checkpoint_schema is propagated - assert!(log_segment.checkpoint_schema.is_some()); - assert_eq!(log_segment.checkpoint_schema.unwrap(), sample_schema); + // Verify that checkpoint_schema only returns schema if it is valid + assert_eq!(log_segment.checkpoint_version, Some(1)); + assert_eq!(log_segment.end_version, 2); + if expect_hint_schema_used { + assert_eq!( + log_segment.checkpoint_hint_schema().as_ref(), + Some(&hint_schema) + ); + } else { + assert!( + log_segment.checkpoint_hint_schema().is_none(), + "hint should not have been returned since version does not match checkpoint_version" + ); + } + + // Verify that get_file_actions_schema_and_sidecars returns appropriate schema based on hint + // version + let (schema, sidecars) = log_segment.get_file_actions_schema_and_sidecars(&engine, None)?; + let schema = schema.expect("V1 checkpoint should yield a file actions schema"); + if expect_hint_schema_used { + assert_eq!(schema, hint_schema, "should use hint when versions match"); + } else { + assert_eq!( + schema, v1_schema, + "should read schema from parquet footer when versions mismatch" + ); + } + assert!(sidecars.is_empty(), "V1 checkpoint should have no sidecars"); + + Ok(()) } -/// Test get_file_actions_schema_and_sidecars with V1 parquet checkpoint using hint schema -/// This verifies the optimization path where hint schema is used directly (avoiding footer read) +/// For a V2 (UUID-named) parquet checkpoint, `get_file_actions_schema_and_sidecars` uses the +/// `_last_checkpoint` hint schema only when the hint names the selected checkpoint; a hint that +/// names a different same-version V2 checkpoint is ignored and the footer is read instead. +#[rstest] +#[case::identity_matches(true)] +#[case::identity_mismatch(false)] #[tokio::test] -async fn test_get_file_actions_schema_v1_parquet_with_hint() -> DeltaResult<()> { - use crate::schema::{StructField, StructType}; +async fn test_get_file_actions_schema_v2_identity_filter( + #[case] identity_matches: bool, +) -> DeltaResult<()> { + let (store, log_root) = new_in_memory_store(); + let engine = SyncEngine::new_with_store(store.clone()); + + let selected = "00000000000000000001.checkpoint.3a0d65cd-4056-49b8-937b-95f9e3ee90e5.parquet"; + let other = "00000000000000000001.checkpoint.016ae953-37a9-438e-8683-9a9a4a79a395.parquet"; + + // Schema actually written to the selected (leaf, no-sidecar) V2 checkpoint footer. + let footer_schema = get_commit_schema().project(&[ADD_NAME, REMOVE_NAME])?; + add_checkpoint_to_store(&store, add_batch_simple(footer_schema.clone()), selected).await?; + let checkpoint_file = log_root.join(selected)?.to_string(); + let cp_size = get_file_size(&store, &format!("_delta_log/{selected}")).await; + // A distinct hint schema so we can tell whether the hint or the footer was used. + let hint_schema: SchemaRef = Arc::new(StructType::new_unchecked([StructField::nullable( + "metadata", + StructType::new_unchecked([]), + )])); + let hint_name = if identity_matches { selected } else { other }; + + let commit_v2_path = log_root.join("00000000000000000002.json")?.to_string(); + let commit_v2 = create_log_path(&commit_v2_path); + let log_segment = LogSegment::try_new( + LogSegmentFiles { + checkpoint_parts: vec![create_log_path_with_size(&checkpoint_file, cp_size)], + ascending_commit_files: vec![commit_v2.clone()], + latest_commit_file: Some(commit_v2), + ..Default::default() + }, + log_root, + None, + Some(LastCheckpointHint { + version: 1, + checkpoint_schema: Some(hint_schema.clone()), + v2_checkpoint: Some(LastCheckpointV2 { + path: hint_name.to_string(), + ..Default::default() + }), + ..Default::default() + }), + )?; + + let (schema, sidecars) = log_segment.get_file_actions_schema_and_sidecars(&engine, None)?; + let schema = schema.expect("leaf V2 checkpoint should yield a file actions schema"); + if identity_matches { + assert_eq!( + schema, hint_schema, + "matching hint identity -> use hint schema" + ); + } else { + assert_eq!( + schema, footer_schema, + "mismatched hint identity -> read footer schema, not the stale hint" + ); + } + assert!(sidecars.is_empty(), "leaf V2 checkpoint has no sidecars"); + Ok(()) +} + +// Multi-part V1 checkpoint returns file_actions_schema with stats_parsed from hint or footer. +#[rstest] +#[case::with_hint(true)] +#[case::without_hint(false)] +#[tokio::test] +async fn test_get_file_actions_schema_multi_part_v1(#[case] use_hint: bool) -> DeltaResult<()> { let (store, log_root) = new_in_memory_store(); - let engine = DefaultEngineBuilder::new(store.clone()).build(); + let engine = SyncEngine::new_with_store(store.clone()); + + let checkpoint_part_1 = "00000000000000000001.checkpoint.0000000001.0000000002.parquet"; + let checkpoint_part_2 = "00000000000000000001.checkpoint.0000000002.0000000002.parquet"; + + // Build a V1 checkpoint schema with stats_parsed containing an integer column. + let v1_schema = schema_ref! { + nullable (ADD_NAME): { + nullable "path": STRING, + nullable "stats_parsed": { + nullable (NUM_RECORDS): LONG, + nullable (MIN_VALUES): { nullable "id": LONG }, + nullable (MAX_VALUES): { nullable "id": LONG }, + }, + }, + nullable (REMOVE_NAME): { + nullable "path": STRING, + }, + }; - // Create a V1 checkpoint (without sidecar column) - let v1_schema = get_commit_schema().project(&[ADD_NAME, REMOVE_NAME])?; add_checkpoint_to_store( &store, add_batch_simple(v1_schema.clone()), - "00000000000000000001.checkpoint.parquet", + checkpoint_part_1, + ) + .await?; + add_checkpoint_to_store( + &store, + add_batch_simple(v1_schema.clone()), + checkpoint_part_2, ) .await?; - let checkpoint_file = log_root - .join("00000000000000000001.checkpoint.parquet")? - .to_string(); + let cp1_size = get_file_size(&store, &format!("_delta_log/{checkpoint_part_1}")).await; + let cp2_size = get_file_size(&store, &format!("_delta_log/{checkpoint_part_2}")).await; - // Create a hint schema without sidecar field (indicates V1) - let hint_schema: SchemaRef = Arc::new(StructType::new_unchecked([ - StructField::nullable("add", StructType::new_unchecked([])), - StructField::nullable("remove", StructType::new_unchecked([])), - ])); + let cp1_file = log_root.join(checkpoint_part_1)?.to_string(); + let cp2_file = log_root.join(checkpoint_part_2)?.to_string(); let log_segment = LogSegment::try_new( LogSegmentFiles { - checkpoint_parts: vec![create_log_path(&checkpoint_file)], - latest_commit_file: Some(create_log_path("file:///00000000000000000002.json")), + checkpoint_parts: vec![ + create_log_path_with_size(&cp1_file, cp1_size), + create_log_path_with_size(&cp2_file, cp2_size), + ], ..Default::default() }, log_root, None, - Some(hint_schema.clone()), // V1 hint schema (no sidecar field) + use_hint.then(|| LastCheckpointHint { + version: 1, + parts: Some(2), + checkpoint_schema: Some(v1_schema.clone()), + ..Default::default() + }), )?; - // With V1 hint, should use hint schema and avoid footer read - let (schema, sidecars) = log_segment.get_file_actions_schema_and_sidecars(&engine)?; - assert!(schema.is_some(), "Should return hint schema for V1"); - assert_eq!( - schema.unwrap(), - hint_schema, - "Should use hint schema directly" + let (schema, sidecars) = log_segment.get_file_actions_schema_and_sidecars(&engine, None)?; + let schema = schema.expect("Multi-part V1 should return file actions schema"); + + // Verify stats_parsed is detectable in the returned schema. + let add_field = schema.field(ADD_NAME).expect("should have add field"); + let DataType::Struct(add_struct) = add_field.data_type() else { + panic!("add field should be a struct type"); + }; + assert!( + add_struct.field("stats_parsed").is_some(), + "Returned schema should include stats_parsed for data skipping" ); - assert!(sidecars.is_empty(), "V1 checkpoint should have no sidecars"); + assert!(sidecars.is_empty(), "Multi-part V1 should have no sidecars"); Ok(()) } @@ -2777,45 +3441,149 @@ async fn test_max_published_version_checkpoint_only() { // Helper to create a checkpoint schema with stats_parsed for testing fn create_checkpoint_schema_with_stats_parsed(min_values_fields: Vec) -> StructType { - let stats_parsed = StructType::new_unchecked([ - StructField::nullable("numRecords", DataType::LONG), - StructField::nullable( - "minValues", - StructType::new_unchecked(min_values_fields.clone()), - ), - StructField::nullable("maxValues", StructType::new_unchecked(min_values_fields)), - ]); - - let add_schema = StructType::new_unchecked([ - StructField::nullable("path", DataType::STRING), - StructField::nullable("stats_parsed", stats_parsed), - ]); + schema! { + nullable "add": { + nullable "path": STRING, + nullable "stats_parsed": { + nullable (NUM_RECORDS): LONG, + nullable (MIN_VALUES): { ..(min_values_fields.clone()) }, + nullable (MAX_VALUES): { ..(min_values_fields) }, + }, + }, + } +} - StructType::new_unchecked([StructField::nullable("add", add_schema)]) +fn create_checkpoint_file_schema_with_stats_parsed( + min_values_fields: Vec, + include_json_stats: bool, +) -> DeltaResult { + let stats_parsed = StructField::nullable( + "stats_parsed", + schema! { + nullable (NUM_RECORDS): LONG, + nullable (MIN_VALUES): { ..(min_values_fields.clone()) }, + nullable (MAX_VALUES): { ..(min_values_fields) }, + }, + ); + let patch = SchemaStructPatchBuilder::new().append_at(["add"], stats_parsed); + let patch = if include_json_stats { + patch + } else { + patch.drop_at(["add"], "stats") + }; + Ok(Arc::new(patch.build(get_commit_schema().as_ref())?)) } // Helper to create a stats_schema with proper structure (numRecords, minValues, maxValues) fn create_stats_schema(column_fields: Vec) -> StructType { - StructType::new_unchecked([ - StructField::nullable("numRecords", DataType::LONG), - StructField::nullable( - "minValues", - StructType::new_unchecked(column_fields.clone()), - ), - StructField::nullable("maxValues", StructType::new_unchecked(column_fields)), - ]) + schema! { + nullable (NUM_RECORDS): LONG, + nullable (MIN_VALUES): { ..(column_fields.clone()) }, + nullable (MAX_VALUES): { ..(column_fields) }, + } } // Helper to create a checkpoint schema without stats_parsed fn create_checkpoint_schema_without_stats_parsed() -> StructType { - use crate::schema::StructType; + schema! { + nullable "add": { + nullable "path": STRING, + nullable "stats": STRING, + }, + } +} - let add_schema = StructType::new_unchecked([ - StructField::nullable("path", DataType::STRING), - StructField::nullable("stats", DataType::STRING), - ]); +#[rstest] +#[case::missing_with_json(false, true, false, true)] +#[case::partial_with_json(true, true, true, false)] +#[case::partial_without_json(true, false, true, false)] +#[tokio::test] +async fn test_checkpoint_stream_resolves_stats_projection( + #[case] include_parsed_stats: bool, + #[case] include_json_stats: bool, + #[case] expect_parsed_stats: bool, + #[case] expect_json_stats: bool, +) -> DeltaResult<()> { + let (store, log_root) = new_in_memory_store(); + let engine = SyncEngine::new_with_store(store.clone()); + let checkpoint_schema = if include_parsed_stats { + create_checkpoint_file_schema_with_stats_parsed( + vec![StructField::nullable("other", DataType::LONG)], + include_json_stats, + )? + } else { + get_commit_schema().clone() + }; + add_checkpoint_to_store( + &store, + add_batch_simple(checkpoint_schema), + "00000000000000000001.checkpoint.parquet", + ) + .await?; + + let checkpoint_file = log_root + .join("00000000000000000001.checkpoint.parquet")? + .to_string(); + let checkpoint_size = + get_file_size(&store, "_delta_log/00000000000000000001.checkpoint.parquet").await; + let log_segment = LogSegment::try_new( + LogSegmentFiles { + checkpoint_parts: vec![create_log_path_with_size(&checkpoint_file, checkpoint_size)], + latest_commit_file: Some(create_log_path("file:///00000000000000000001.json")), + ..Default::default() + }, + log_root, + None, + None, + )?; + let stats_schema = create_stats_schema(vec![StructField::nullable("id", DataType::LONG)]); + + let checkpoint_result = log_segment.create_checkpoint_stream( + &engine, + CHECKPOINT_READ_SCHEMA_NO_JSON_STATS.clone(), + None, // meta_predicate + Some(&stats_schema), + None, // partition_schema + None, // cancellation_token + )?; + + assert_eq!( + checkpoint_result.checkpoint_info.has_stats_parsed, + expect_parsed_stats + ); + let add_field = checkpoint_result + .checkpoint_info + .checkpoint_read_schema + .field("add") + .expect("checkpoint read schema must contain add"); + let DataType::Struct(add) = add_field.data_type() else { + panic!("checkpoint add field must be a struct"); + }; + assert_eq!(add.field("stats").is_some(), expect_json_stats); + assert_eq!(add.field("stats_parsed").is_some(), expect_parsed_stats); + + let read_schema = checkpoint_result + .checkpoint_info + .checkpoint_read_schema + .clone(); + let mut actions = checkpoint_result.actions; + let batch = actions + .next() + .expect("checkpoint stream must yield one batch")?; + assert!(!batch.is_log_batch); + assert_eq!( + batch.actions.has_field(&column_name!("add.stats")), + expect_json_stats, + "checkpoint batch JSON stats projection must match the resolved schema" + ); + if include_json_stats { + assert_batch_matches(batch.actions, add_batch_simple(read_schema)); + } else { + assert!(!batch.actions.is_empty()); + } + assert!(actions.next().is_none()); - StructType::new_unchecked([StructField::nullable("add", add_schema)]) + Ok(()) } #[test] @@ -2944,17 +3712,15 @@ fn test_schema_has_compatible_stats_parsed_multiple_columns() { #[test] fn test_schema_has_compatible_stats_parsed_missing_min_max_values() { // stats_parsed exists but has no minValues/maxValues fields - unusual but valid (continue case) - let stats_parsed = StructType::new_unchecked([ - StructField::nullable("numRecords", DataType::LONG), - // No minValues or maxValues fields - ]); - - let add_schema = StructType::new_unchecked([ - StructField::nullable("path", DataType::STRING), - StructField::nullable("stats_parsed", stats_parsed), - ]); - - let checkpoint_schema = StructType::new_unchecked([StructField::nullable("add", add_schema)]); + let checkpoint_schema = schema! { + nullable "add": { + nullable "path": STRING, + nullable "stats_parsed": { + nullable (NUM_RECORDS): LONG, + // No minValues or maxValues fields + }, + }, + }; let stats_schema = create_stats_schema(vec![StructField::nullable("id", DataType::INTEGER)]); @@ -2968,19 +3734,17 @@ fn test_schema_has_compatible_stats_parsed_missing_min_max_values() { #[test] fn test_schema_has_compatible_stats_parsed_min_values_not_struct() { // minValues/maxValues exist but are not Struct types - malformed schema (return false case) - let stats_parsed = StructType::new_unchecked([ - StructField::nullable("numRecords", DataType::LONG), - // minValues is a primitive type instead of a Struct - StructField::nullable("minValues", DataType::STRING), - StructField::nullable("maxValues", DataType::STRING), - ]); - - let add_schema = StructType::new_unchecked([ - StructField::nullable("path", DataType::STRING), - StructField::nullable("stats_parsed", stats_parsed), - ]); - - let checkpoint_schema = StructType::new_unchecked([StructField::nullable("add", add_schema)]); + let checkpoint_schema = schema! { + nullable "add": { + nullable "path": STRING, + nullable "stats_parsed": { + nullable (NUM_RECORDS): LONG, + // minValues/maxValues are primitives instead of Structs + nullable (MIN_VALUES): STRING, + nullable (MAX_VALUES): STRING, + }, + }, + }; let stats_schema = create_stats_schema(vec![StructField::nullable("id", DataType::INTEGER)]); @@ -3142,13 +3906,168 @@ fn test_schema_has_compatible_stats_parsed_deeply_nested_type_mismatch() { )); } +#[test] +fn test_schema_has_compatible_stats_parsed_long_to_timestamp() { + // Checkpoint stores timestamp stats as Int64 (no logical type annotation) + let checkpoint_schema = create_checkpoint_schema_with_stats_parsed(vec![ + StructField::nullable("ts_col", DataType::LONG), + StructField::nullable("ts_ntz_col", DataType::LONG), + ]); + + // Stats schema expects Timestamp and TimestampNtz types + let stats_schema = create_stats_schema(vec![ + StructField::nullable("ts_col", DataType::TIMESTAMP), + StructField::nullable("ts_ntz_col", DataType::TIMESTAMP_NTZ), + ]); + + // Long -> Timestamp/TimestampNtz reinterpretation should be accepted + assert!(LogSegment::schema_has_compatible_stats_parsed( + &checkpoint_schema, + &stats_schema + )); +} + +#[test] +fn test_schema_has_compatible_stats_parsed_timestamp_to_long_rejected() { + // Checkpoint has Timestamp-typed stats + let checkpoint_schema = + create_checkpoint_schema_with_stats_parsed(vec![StructField::nullable( + "ts_col", + DataType::TIMESTAMP, + )]); + + // Stats schema expects Long -- narrowing should be rejected + let stats_schema = create_stats_schema(vec![StructField::nullable("ts_col", DataType::LONG)]); + + assert!(!LogSegment::schema_has_compatible_stats_parsed( + &checkpoint_schema, + &stats_schema + )); +} + +#[test] +fn test_schema_has_compatible_stats_parsed_integer_to_date() { + // Checkpoint stores date stats as Int32 (no DATE logical annotation) + let checkpoint_schema = + create_checkpoint_schema_with_stats_parsed(vec![StructField::nullable( + "date_col", + DataType::INTEGER, + )]); + + // Stats schema expects Date type + let stats_schema = create_stats_schema(vec![StructField::nullable("date_col", DataType::DATE)]); + + // Integer -> Date reinterpretation should be accepted + assert!(LogSegment::schema_has_compatible_stats_parsed( + &checkpoint_schema, + &stats_schema + )); +} + +#[test] +fn test_schema_has_compatible_stats_parsed_date_to_integer_rejected() { + // Checkpoint has Date-typed stats + let checkpoint_schema = + create_checkpoint_schema_with_stats_parsed(vec![StructField::nullable( + "date_col", + DataType::DATE, + )]); + + // Stats schema expects Integer -- narrowing should be rejected + let stats_schema = + create_stats_schema(vec![StructField::nullable("date_col", DataType::INTEGER)]); + + assert!(!LogSegment::schema_has_compatible_stats_parsed( + &checkpoint_schema, + &stats_schema + )); +} + +// Type widening + checkpoint reinterpretation interaction scenarios. +// Verifies that schema evolution doesn't create false-positive type matches. +#[rstest] +// Standard widening: Integer -> Long in old checkpoint after column was widened +#[case::widening_integer_to_long(DataType::INTEGER, DataType::LONG, true)] +// Checkpoint reinterpretation: Int32 without DATE annotation -> Date +#[case::reinterpret_integer_to_date(DataType::INTEGER, DataType::DATE, true)] +// Checkpoint reinterpretation: Int64 without TIMESTAMP annotation -> Timestamp +#[case::reinterpret_long_to_timestamp(DataType::LONG, DataType::TIMESTAMP, true)] +// Compound: checkpoint dropped Date annotation (Int32) + column widened to Timestamp. +// Integer -> Timestamp is neither a widening nor reinterpretation rule. +#[case::reinterpret_plus_widen_integer_to_timestamp(DataType::INTEGER, DataType::TIMESTAMP, false)] +#[case::reinterpret_plus_widen_integer_to_timestamp_ntz( + DataType::INTEGER, + DataType::TIMESTAMP_NTZ, + false +)] +// Date -> Timestamp is a valid Delta type widening rule, but kernel's can_widen_to does not +// currently support it. This test documents the current behavior. +#[case::date_widened_to_timestamp(DataType::DATE, DataType::TIMESTAMP, false)] +fn test_stats_parsed_widening_and_reinterpretation_interaction( + #[case] checkpoint_type: DataType, + #[case] stats_type: DataType, + #[case] expected: bool, +) { + let checkpoint_schema = + create_checkpoint_schema_with_stats_parsed(vec![StructField::nullable( + "col", + checkpoint_type, + )]); + let stats_schema = create_stats_schema(vec![StructField::nullable("col", stats_type)]); + + assert_eq!( + LogSegment::schema_has_compatible_stats_parsed(&checkpoint_schema, &stats_schema), + expected + ); +} + +#[test] +fn test_stats_parsed_mixed_widening_and_reinterpretation() { + // Multiple columns with different compatibility paths should all pass. + let checkpoint_schema = create_checkpoint_schema_with_stats_parsed(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("ts_col", DataType::LONG), + StructField::nullable("date_col", DataType::INTEGER), + ]); + let stats_schema = create_stats_schema(vec![ + StructField::nullable("id", DataType::LONG), + StructField::nullable("ts_col", DataType::TIMESTAMP), + StructField::nullable("date_col", DataType::DATE), + ]); + + assert!(LogSegment::schema_has_compatible_stats_parsed( + &checkpoint_schema, + &stats_schema + )); +} + +#[test] +fn test_stats_parsed_mixed_with_one_incompatible_rejects_all() { + // One incompatible column (Integer -> Timestamp) rejects the whole schema. + let checkpoint_schema = create_checkpoint_schema_with_stats_parsed(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("ts_col", DataType::LONG), + StructField::nullable("bad_col", DataType::INTEGER), + ]); + let stats_schema = create_stats_schema(vec![ + StructField::nullable("id", DataType::LONG), + StructField::nullable("ts_col", DataType::TIMESTAMP), + StructField::nullable("bad_col", DataType::TIMESTAMP), + ]); + + assert!(!LogSegment::schema_has_compatible_stats_parsed( + &checkpoint_schema, + &stats_schema + )); +} + // ============================================================================ // create_checkpoint_stream: partitionValues_parsed schema augmentation tests // ============================================================================ /// Creates a checkpoint batch with `add.partitionValues_parsed` in the parquet schema. fn add_batch_with_partition_values_parsed(output_schema: SchemaRef) -> Box { - let handler = SyncJsonHandler {}; + let handler = SyncJsonHandler::new(None); let json_strings: StringArray = vec![ r#"{"add":{"path":"part-00000.parquet","partitionValues":{"id":"1"},"partitionValues_parsed":{"id":1},"size":635,"modificationTime":1677811178336,"dataChange":true}}"#, r#"{"metaData":{"id":"testId","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"value\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["id"],"configuration":{},"createdTime":1677811175819}}"#, @@ -3163,43 +4082,27 @@ fn add_batch_with_partition_values_parsed(output_schema: SchemaRef) -> Box DeltaResult<()> { let (store, log_root) = new_in_memory_store(); - let engine = DefaultEngineBuilder::new(store.clone()).build(); + let engine = SyncEngine::new_with_store(store.clone()); // Build a schema that includes add.partitionValues_parsed.id: integer - let partition_parsed_struct = - StructType::new_unchecked([StructField::nullable("id", DataType::INTEGER)]); - let add_struct = StructType::new_unchecked([ - StructField::nullable("path", DataType::STRING), - StructField::nullable( - "partitionValues", - crate::schema::MapType::new(DataType::STRING, DataType::STRING, true), - ), - StructField::nullable("partitionValues_parsed", partition_parsed_struct), - StructField::nullable("size", DataType::LONG), - StructField::nullable("modificationTime", DataType::LONG), - StructField::nullable("dataChange", DataType::BOOLEAN), - ]); - let metadata_struct = StructType::new_unchecked([ - StructField::nullable("id", DataType::STRING), - StructField::nullable( - "format", - StructType::new_unchecked([StructField::nullable("provider", DataType::STRING)]), - ), - StructField::nullable("schemaString", DataType::STRING), - StructField::nullable( - "partitionColumns", - crate::schema::ArrayType::new(DataType::STRING, false), - ), - StructField::nullable( - "configuration", - crate::schema::MapType::new(DataType::STRING, DataType::STRING, true), - ), - StructField::nullable("createdTime", DataType::LONG), - ]); - let checkpoint_schema: SchemaRef = Arc::new(StructType::new_unchecked([ - StructField::nullable("add", add_struct), - StructField::nullable("metaData", metadata_struct), - ])); + let checkpoint_schema: SchemaRef = schema_ref! { + nullable "add": { + nullable "path": STRING, + nullable "partitionValues": { STRING => nullable STRING }, + nullable "partitionValues_parsed": { nullable "id": INTEGER }, + nullable "size": LONG, + nullable "modificationTime": LONG, + nullable "dataChange": BOOLEAN, + }, + nullable "metaData": { + nullable "id": STRING, + nullable "format": { nullable "provider": STRING }, + nullable "schemaString": STRING, + nullable "partitionColumns": [ not_null STRING ], + nullable "configuration": { STRING => nullable STRING }, + nullable "createdTime": LONG, + }, + }; add_checkpoint_to_store( &store, @@ -3249,6 +4152,7 @@ async fn test_checkpoint_stream_sets_has_partition_values_parsed() -> DeltaResul None, // meta_predicate None, // stats_schema Some(&partition_schema), + None, // cancellation_token )?; // Verify that checkpoint_info reports partitionValues_parsed as available @@ -3276,7 +4180,7 @@ async fn test_checkpoint_stream_sets_has_partition_values_parsed() -> DeltaResul #[tokio::test] async fn test_checkpoint_stream_no_partition_values_parsed_when_incompatible() -> DeltaResult<()> { let (store, log_root) = new_in_memory_store(); - let engine = DefaultEngineBuilder::new(store.clone()).build(); + let engine = SyncEngine::new_with_store(store.clone()); // Write a checkpoint WITHOUT partitionValues_parsed add_checkpoint_to_store( @@ -3314,6 +4218,7 @@ async fn test_checkpoint_stream_no_partition_values_parsed_when_incompatible() - None, None, Some(&partition_schema), + None, // cancellation_token )?; // Verify it's false @@ -3577,6 +4482,160 @@ async fn test_new_with_commit_not_end_version_plus_one() { ); } +// ============================================================================ +// try_new_with_checkpoint tests +// ============================================================================ + +#[rstest] +#[case::non_checkpoint_file( + "file:///_delta_log/00000000000000000002.json", + "Path is not a single-file checkpoint" +)] +#[case::multi_part_checkpoint( + "file:///_delta_log/00000000000000000002.checkpoint.0000000001.0000000002.parquet", + "Path is not a single-file checkpoint" +)] +#[case::wrong_version( + "file:///_delta_log/00000000000000000005.checkpoint.parquet", + "Checkpoint version (5) does not equal LogSegment end_version (2)" +)] +#[tokio::test] +async fn test_try_new_with_checkpoint_rejects_invalid_path( + #[case] path: &str, + #[case] expected_error: &str, +) { + let log_segment = create_segment_for(LogSegmentConfig { + published_commit_versions: &[0, 1, 2], + ..Default::default() + }) + .await; + let result = log_segment.try_new_with_checkpoint(create_log_path(path)); + assert_result_error_with_message(result, expected_error); +} + +#[rstest] +#[case::no_crc(None, None)] +#[case::stale_crc_cleared(Some(1), None)] +#[case::crc_at_checkpoint_retained(Some(2), Some(2))] +#[tokio::test] +async fn test_try_new_with_checkpoint( + #[values( + "checkpoint.parquet", + "checkpoint.3a0d65cd-4056-49b8-937b-95f9e3ee90e5.parquet" + )] + ckpt_suffix: &str, + #[case] crc_version: Option, + #[case] expected_crc_version: Option, +) { + const CHECKPOINT_VERSION: u64 = 2; + let ckpt_url = format!("file:///_delta_log/{CHECKPOINT_VERSION:020}.{ckpt_suffix}"); + + let mut log_segment = create_segment_for(LogSegmentConfig { + published_commit_versions: &[0, 1, 2], + compaction_versions: &[(0, 2)], + ..Default::default() + }) + .await; + if let Some(v) = crc_version { + // Bypass try_new_with_crc_file's end_version check so we can plant a stale CRC. + log_segment.listed.latest_crc_file = + Some(create_log_path(&format!("file:///_delta_log/{v:020}.crc"))); + } + assert!(!log_segment.listed.ascending_commit_files.is_empty()); + // TODO(#2337): restore to assert !is_empty() when log compaction is re-enabled + assert!(log_segment.listed.ascending_compaction_files.is_empty()); + + let result = log_segment + .try_new_with_checkpoint(create_log_path(&ckpt_url)) + .unwrap(); + + assert_eq!(result.checkpoint_version, Some(CHECKPOINT_VERSION)); + assert_eq!(result.listed.checkpoint_parts.len(), 1); + assert_eq!( + result.listed.checkpoint_parts[0].version, + CHECKPOINT_VERSION + ); + assert!(result.listed.ascending_commit_files.is_empty()); + assert!(result.listed.ascending_compaction_files.is_empty()); + assert!(result.last_checkpoint_metadata.is_none()); + assert_eq!( + result.listed.latest_crc_file.as_ref().map(|c| c.version), + expected_crc_version + ); + + // latest_commit_file is preserved for ICT access even though commits are cleared + assert_eq!( + result.listed.latest_commit_file.as_ref().map(|f| f.version), + log_segment + .listed + .latest_commit_file + .as_ref() + .map(|f| f.version) + ); + + // Structural fields are preserved + assert_eq!(result.end_version, log_segment.end_version); + assert_eq!(result.log_root, log_segment.log_root); +} + +// ============================================================================ +// try_new_with_crc_file tests +// ============================================================================ + +#[rstest] +#[case::non_crc_file( + "file:///_delta_log/00000000000000000002.json", + "Path is not a CRC file" +)] +#[case::wrong_version( + "file:///_delta_log/00000000000000000005.crc", + "CRC version (5) does not equal LogSegment end_version (2)" +)] +#[tokio::test] +async fn test_try_new_with_crc_file_rejects_invalid_path( + #[case] path: &str, + #[case] expected_error: &str, +) { + let log_segment = create_segment_for(LogSegmentConfig { + published_commit_versions: &[0, 1, 2], + ..Default::default() + }) + .await; + let url = Url::parse(path).unwrap(); + let crc_path = ParsedLogPath::try_from(url).unwrap().unwrap(); + let result = log_segment.try_new_with_crc_file(crc_path); + assert_result_error_with_message(result, expected_error); +} + +#[tokio::test] +async fn test_try_new_with_crc_file_sets_crc_and_preserves_other_fields() { + let log_segment = create_segment_for(LogSegmentConfig { + published_commit_versions: &[0, 1, 2], + checkpoint_version: Some(1), + ..Default::default() + }) + .await; + let url = Url::parse("file:///_delta_log/00000000000000000002.crc").unwrap(); + let crc_path = ParsedLogPath::try_from(url).unwrap().unwrap(); + let result = log_segment.try_new_with_crc_file(crc_path).unwrap(); + + let crc_file = result.listed.latest_crc_file.as_ref().unwrap(); + assert_eq!(crc_file.version, 2); + + // Everything else is preserved + assert_eq!(result.end_version, log_segment.end_version); + assert_eq!(result.checkpoint_version, log_segment.checkpoint_version); + assert_eq!( + result.listed.ascending_commit_files.len(), + log_segment.listed.ascending_commit_files.len() + ); + assert_eq!( + result.listed.checkpoint_parts.len(), + log_segment.listed.checkpoint_parts.len() + ); + assert_eq!(result.log_root, log_segment.log_root); +} + // ============================================================================ // get_unpublished_catalog_commits tests // ============================================================================ @@ -3597,7 +4656,7 @@ async fn test_get_unpublished_catalog_commits() { } // ============================================================================ -// Tests: segment_after_crc / segment_through_crc +// Tests: segment_after_version // ============================================================================ fn extract_commit_versions(seg: &LogSegment) -> Vec { @@ -3626,8 +4685,6 @@ struct CrcPruningCase { crc_version: u64, after_commits: &'static [u64], after_compactions: &'static [(u64, u64)], - through_commits: &'static [u64], - through_compactions: &'static [(u64, u64)], } #[rstest::rstest] @@ -3635,7 +4692,6 @@ struct CrcPruningCase { // commits: x x x x x x x x x x // crc: | // after commits: x x x x x -// through commits: x x x x x #[case::only_deltas_no_checkpoint(CrcPruningCase { commits: &[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], compactions: &[], @@ -3643,15 +4699,12 @@ struct CrcPruningCase { crc_version: 4, after_commits: &[5, 6, 7, 8, 9], after_compactions: &[], - through_commits: &[0, 1, 2, 3, 4], - through_compactions: &[], })] // 0 1 2 3 4 5 6 7 8 9 // checkpoint: | // commits: x x x x x x x // crc: | // after commits: x x x x x -// through commits: x x #[case::only_deltas_with_checkpoint(CrcPruningCase { commits: &[3, 4, 5, 6, 7, 8, 9], compactions: &[], @@ -3659,8 +4712,6 @@ struct CrcPruningCase { crc_version: 4, after_commits: &[5, 6, 7, 8, 9], after_compactions: &[], - through_commits: &[3, 4], - through_compactions: &[], })] // 0 1 2 3 4 5 6 7 8 9 // commits: x x x x x x x x x x @@ -3668,24 +4719,19 @@ struct CrcPruningCase { // crc: | // after commits: x x x x x // after compactions: [-----] -// through commits: x x x x x #[case::compaction_after_crc(CrcPruningCase { commits: &[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], compactions: &[(5, 7)], checkpoint: None, crc_version: 4, after_commits: &[5, 6, 7, 8, 9], - after_compactions: &[(5, 7)], - through_commits: &[0, 1, 2, 3, 4], - through_compactions: &[], + after_compactions: &[], // TODO(#2337): restore to &[(5, 7)] when re-enabled })] // 0 1 2 3 4 5 6 7 8 9 // commits: x x x x x x x x x x // compactions: [-----------] // crc: | // after commits: x x x x x -// through commits: x x x x x -// through compactions: #[case::compaction_overlaps_crc(CrcPruningCase { commits: &[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], compactions: &[(2, 6)], @@ -3693,16 +4739,12 @@ struct CrcPruningCase { crc_version: 4, after_commits: &[5, 6, 7, 8, 9], after_compactions: &[], - through_commits: &[0, 1, 2, 3, 4], - through_compactions: &[], })] // 0 1 2 3 4 5 6 7 8 9 // commits: x x x x x x x x x x // compactions: [-----] // crc: | // after commits: x x x x x -// through commits: x x x x x -// through compactions: [-----] #[case::compaction_before_crc(CrcPruningCase { commits: &[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], compactions: &[(0, 2)], @@ -3710,8 +4752,6 @@ struct CrcPruningCase { crc_version: 4, after_commits: &[5, 6, 7, 8, 9], after_compactions: &[], - through_commits: &[0, 1, 2, 3, 4], - through_compactions: &[(0, 2)], })] #[tokio::test] async fn test_segment_crc_filtering(#[case] case: CrcPruningCase) { @@ -3723,83 +4763,266 @@ async fn test_segment_crc_filtering(#[case] case: CrcPruningCase) { }) .await; - let after = seg.segment_after_crc(case.crc_version); + let after = seg.segment_after_version(case.crc_version); assert_eq!(extract_commit_versions(&after), case.after_commits); assert_eq!(extract_compaction_ranges(&after), case.after_compactions); assert!(after.checkpoint_version.is_none()); assert!(after.listed.checkpoint_parts.is_empty()); - - let through = seg.segment_through_crc(case.crc_version); - assert_eq!(extract_commit_versions(&through), case.through_commits); - assert_eq!( - extract_compaction_ranges(&through), - case.through_compactions - ); - assert_eq!(through.checkpoint_version, case.checkpoint); } #[rstest::rstest] #[case::empty_schema(StructType::new_unchecked([]), None)] -#[case::metadata_field( - StructType::new_unchecked([StructField::nullable( - METADATA_NAME, - StructType::new_unchecked([]), - )]), +#[case::add_field( + schema! { nullable (ADD_NAME): {} }, Some(Arc::new( - Expression::column(ColumnName::new([METADATA_NAME, "id"])).is_not_null(), + col!(ADD_NAME, "path").is_not_null(), )), )] -#[case::protocol_field( - StructType::new_unchecked([StructField::nullable( - PROTOCOL_NAME, - StructType::new_unchecked([]), - )]), - Some(Arc::new( - Expression::column(ColumnName::new([PROTOCOL_NAME, "minReaderVersion"])).is_not_null(), - )), -)] -#[case::txn_field( - StructType::new_unchecked([StructField::nullable( - SET_TRANSACTION_NAME, - StructType::new_unchecked([]), - )]), - Some(Arc::new( - Expression::column(ColumnName::new([SET_TRANSACTION_NAME, "appId"])).is_not_null(), - )), -)] -#[case::domain_metadata_field( - StructType::new_unchecked([StructField::nullable( - DOMAIN_METADATA_NAME, - StructType::new_unchecked([]), - )]), +#[case::remove_field( + schema! { nullable (REMOVE_NAME): {} }, Some(Arc::new( - Expression::column(ColumnName::new([DOMAIN_METADATA_NAME, "domain"])).is_not_null(), + col!(REMOVE_NAME, "path").is_not_null(), )), )] -#[case::unknown_field_returns_none( - StructType::new_unchecked([StructField::nullable(ADD_NAME, StructType::new_unchecked([]))]), +#[case::action_without_required_leaf_returns_none( + schema! { nullable (COMMIT_INFO_NAME): {} }, None, )] -#[case::multiple_known_fields( +#[case::add_and_remove_fields( StructType::new_unchecked([ - StructField::nullable(METADATA_NAME, StructType::new_unchecked([])), - StructField::nullable(PROTOCOL_NAME, StructType::new_unchecked([])), + StructField::nullable(ADD_NAME, StructType::new_unchecked([])), + StructField::nullable(REMOVE_NAME, StructType::new_unchecked([])), ]), Some(Arc::new(Predicate::or( - Expression::column(ColumnName::new([METADATA_NAME, "id"])).is_not_null(), - Expression::column(ColumnName::new([PROTOCOL_NAME, "minReaderVersion"])).is_not_null(), + col!(ADD_NAME, "path").is_not_null(), + col!(REMOVE_NAME, "path").is_not_null(), ))), )] -#[case::known_and_unknown_field_returns_none( +#[case::witness_and_witnessless_field_returns_none( StructType::new_unchecked([ StructField::nullable(METADATA_NAME, StructType::new_unchecked([])), + StructField::nullable(COMMIT_INFO_NAME, StructType::new_unchecked([])), + ]), + None, +)] +#[case::known_and_unknown_field_returns_none( + StructType::new_unchecked([ StructField::nullable(ADD_NAME, StructType::new_unchecked([])), + StructField::nullable("futureAction", StructType::new_unchecked([])), ]), None, )] -fn test_schema_to_is_not_null_predicate( +fn test_checkpoint_action_projection_predicate( #[case] schema: StructType, #[case] expected: Option, ) { - assert_eq!(schema_to_is_not_null_predicate(&schema), expected); + assert_eq!(checkpoint_action_projection_predicate(&schema), expected); +} + +#[rstest] +#[case::neither(false, false)] +#[case::projection_only(true, false)] +#[case::metadata_only(false, true)] +#[case::both(true, true)] +fn test_combine_checkpoint_predicates( + #[case] include_projection: bool, + #[case] include_metadata: bool, +) { + let projection = Arc::new(col!(ADD_NAME, "path").is_not_null()); + let metadata = Arc::new(col!(ADD_NAME, "size").is_not_null()); + let expected = match (include_projection, include_metadata) { + (false, false) => None, + (true, false) => Some(projection.clone()), + (false, true) => Some(metadata.clone()), + (true, true) => Some(Arc::new(Predicate::and( + (*projection).clone(), + (*metadata).clone(), + ))), + }; + + assert_eq!( + combine_checkpoint_predicates( + include_projection.then_some(projection), + include_metadata.then_some(metadata), + ), + expected, + ); +} + +/// Verify that `read_actions` correctly handles null values in map fields across all +/// action types. The Delta protocol allows null values in `partitionValues` maps (a null +/// partition value means the partition column is null for that file) and in `tags` maps. +/// +/// Spark defaults all `Map[String, String]` types to `valueContainsNull = true`, and +/// checkpoint writing calls `schema.asNullable` which forces all maps nullable. The +/// schema must match this behavior. +/// +/// This test reads JSON actions through `DefaultEngine` + `InMemory` store + +/// `log_segment.read_actions()`, then re-validates the resulting Arrow `StructArray` with +/// `StructArray::try_new`. Without the fix, non-nullable map value fields cause: +/// "Found unmasked nulls for non-nullable StructArray field 'value'" +#[rstest] +// remove.partitionValues.month: null +#[case::remove_partition_values( + "remove", + "partitionValues", + r#"{"remove":{"path":"file.parquet","deletionTimestamp":1000,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{"year":"2024","month":null},"size":100}}"# +)] +// remove.tags.key2: null +#[case::remove_tags( + "remove", + "tags", + r#"{"remove":{"path":"file.parquet","deletionTimestamp":1000,"dataChange":true,"tags":{"key1":"val1","key2":null}}}"# +)] +// add.partitionValues.month: null +#[case::add_partition_values( + "add", + "partitionValues", + r#"{"add":{"path":"file.parquet","partitionValues":{"year":"2024","month":null},"size":100,"modificationTime":1000,"dataChange":true}}"# +)] +// add.tags.key2: null +#[case::add_tags( + "add", + "tags", + r#"{"add":{"path":"file.parquet","partitionValues":{},"size":100,"modificationTime":1000,"dataChange":true,"tags":{"key1":"val1","key2":null}}}"# +)] +// cdc.partitionValues.month: null +#[case::cdc_partition_values( + "cdc", + "partitionValues", + r#"{"cdc":{"path":"file.parquet","partitionValues":{"year":"2024","month":null},"size":100,"dataChange":false}}"# +)] +// cdc.tags.key2: null +#[case::cdc_tags( + "cdc", + "tags", + r#"{"cdc":{"path":"file.parquet","partitionValues":{},"size":100,"dataChange":false,"tags":{"key1":"val1","key2":null}}}"# +)] +// sidecar.tags.key2: null +#[case::sidecar_tags( + "sidecar", + "tags", + r#"{"sidecar":{"path":"sidecar.parquet","sizeInBytes":100,"modificationTime":1000,"tags":{"key1":"val1","key2":null}}}"# +)] +// checkpointMetadata.tags.key2: null +#[case::checkpoint_metadata_tags( + "checkpointMetadata", + "tags", + r#"{"checkpointMetadata":{"version":0,"tags":{"key1":"val1","key2":null}}}"# +)] +// Known issues: these map fields don't yet have #[allow_null_container_values]. +// commitInfo.operationParameters.description: null +#[should_panic(expected = "StructArray re-validation failed")] +#[case::commit_info_operation_parameters_known_issue( + "commitInfo", + "operationParameters", + r#"{"commitInfo":{"timestamp":1000,"operation":"WRITE","operationParameters":{"mode":"ErrorIfExists","description":null}}}"# +)] +// metaData.configuration.key2: null +#[should_panic(expected = "StructArray re-validation failed")] +#[case::metadata_configuration_known_issue( + "metaData", + "configuration", + r#"{"metaData":{"id":"test","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[]}","partitionColumns":[],"configuration":{"key1":"val1","key2":null},"createdTime":1000}}"# +)] +#[tokio::test] +async fn read_actions_with_null_map_values( + #[case] action_name: &str, + #[case] map_field: &str, + #[case] json_action: &str, +) { + use crate::arrow::array::{Array, AsArray, MapArray, StructArray}; + + let store = Arc::new(InMemory::new()); + let log_root = Url::parse("memory:///_delta_log/").unwrap(); + + // Write a single commit file with the action containing null map values. + store + .put( + &delta_path_for_version(0, "json"), + json_action.to_string().into(), + ) + .await + .unwrap(); + + // Build engine and read actions -- same as DeltaActionExtractor::get_actions. + let engine = SyncEngine::new_with_store(store); + let log_segment = + LogSegment::for_table_changes(engine.storage_handler().as_ref(), log_root, 0, Some(0)) + .unwrap(); + + // Use all_actions_schema to cover sidecar and checkpointMetadata (checkpoint-only actions). + let action_schema = get_all_actions_schema().clone(); + let action_batches = log_segment + .read_actions(&engine, action_schema) + .expect("read_actions should succeed"); + + // Iterate batches and verify the map value field is nullable. + let mut found = false; + for batch_result in action_batches { + let actions_batch = batch_result.expect("Iterating action batches should succeed"); + + let data_any = actions_batch.actions.into_any(); + let arrow_data = data_any + .downcast_ref::() + .expect("ArrowEngineData"); + let rb = arrow_data.record_batch(); + + let Some(action_col) = rb.column_by_name(action_name) else { + continue; + }; + let action_struct = action_col + .as_struct_opt() + .unwrap_or_else(|| panic!("{action_name} column should be a struct")); + let map_col = action_struct + .column_by_name(map_field) + .unwrap_or_else(|| panic!("{action_name}.{map_field} not found")); + let map_array = map_col + .as_any() + .downcast_ref::() + .unwrap_or_else(|| panic!("{action_name}.{map_field} should be a MapArray")); + // Re-validate the entries StructArray with its own schema, same as what Arrow's + // IPC deserializer does. Without the fix, this fails with: + // "Found unmasked nulls for non-nullable StructArray field 'value'" + let entries = map_array.entries(); + StructArray::try_new( + entries.fields().clone(), + entries.columns().to_vec(), + entries.nulls().cloned(), + ) + .unwrap_or_else(|e| { + panic!( + "{action_name}.{map_field} entries StructArray re-validation failed: {e}. \ + This means the schema has non-nullable value field but the data has nulls." + ) + }); + found = true; + } + assert!(found, "Should have found a {action_name} action batch"); +} + +#[test] +fn new_for_version_zero_creates_valid_log_segment() { + let log_root = Url::parse("memory:///_delta_log/").unwrap(); + let commit_path = create_log_path("memory:///_delta_log/00000000000000000000.json"); + let segment = super::LogSegment::new_for_version_zero(log_root.clone(), commit_path).unwrap(); + assert_eq!(segment.end_version, 0); + assert_eq!(segment.log_root, log_root); +} + +#[test] +fn new_for_version_zero_rejects_non_zero_version() { + let log_root = Url::parse("memory:///_delta_log/").unwrap(); + let commit_path = create_log_path("memory:///_delta_log/00000000000000000001.json"); + let err = super::LogSegment::new_for_version_zero(log_root, commit_path).unwrap_err(); + assert!(err.to_string().contains("version")); +} + +#[test] +fn new_for_version_zero_rejects_non_commit_file() { + let log_root = Url::parse("memory:///_delta_log/").unwrap(); + let checkpoint_path = + create_log_path("memory:///_delta_log/00000000000000000000.checkpoint.parquet"); + let err = super::LogSegment::new_for_version_zero(log_root, checkpoint_path).unwrap_err(); + assert!(err.to_string().contains("non-commit")); } diff --git a/kernel/src/log_segment_files.rs b/kernel/src/log_segment_files.rs deleted file mode 100644 index 8548d6f4c7..0000000000 --- a/kernel/src/log_segment_files.rs +++ /dev/null @@ -1,897 +0,0 @@ -//! [`LogSegmentFiles`] is a struct holding the result of listing the delta log. Currently, it -//! exposes three APIs for listing: -//! 1. [`list_commits`]: Lists all commit files between the provided start and end versions. -//! 2. [`list`]: Lists all commit and checkpoint files between the provided start and end versions. -//! 3. [`list_with_checkpoint_hint`]: Lists all commit and checkpoint files after the provided -//! checkpoint hint. -//! -//! After listing, one can leverage the [`LogSegmentFiles`] to construct a [`LogSegment`]. -//! -//! [`list_commits`]: Self::list_commits -//! [`list`]: Self::list -//! [`list_with_checkpoint_hint`]: Self::list_with_checkpoint_hint -//! [`LogSegment`]: crate::log_segment::LogSegment - -use std::collections::HashMap; - -use crate::last_checkpoint_hint::LastCheckpointHint; -use crate::path::{LogPathFileType, ParsedLogPath}; -use crate::{DeltaResult, Error, StorageHandler, Version}; - -use delta_kernel_derive::internal_api; - -use itertools::Itertools; -use tracing::{debug, info, instrument}; -use url::Url; - -/// Represents the set of log files found during a listing operation in the Delta log directory. -/// -/// - `ascending_commit_files`: All commit and staged commit files found, sorted by version. May contain gaps. -/// - `ascending_compaction_files`: All compaction commit files found, sorted by version. -/// - `checkpoint_parts`: All parts of the most recent complete checkpoint (all same version). Empty if no checkpoint found. -/// - `latest_crc_file`: The CRC file with the highest version, only if version >= checkpoint version. -/// - `latest_commit_file`: The commit file with the highest version, or `None` if no commits were found. -/// - `max_published_version`: The highest published commit file version, or `None` if no published commits were found. -#[derive(Debug, Default, Clone, PartialEq, Eq)] -#[internal_api] -pub(crate) struct LogSegmentFiles { - pub ascending_commit_files: Vec, - pub ascending_compaction_files: Vec, - pub checkpoint_parts: Vec, - pub latest_crc_file: Option, - pub latest_commit_file: Option, - pub max_published_version: Option, -} - -/// Returns a lazy iterator of [`ParsedLogPath`]s from the filesystem over versions -/// `[start_version, end_version]`. The iterator handles parsing, filtering out non-listable -/// files (e.g. staged commits, dot-prefixed files), and stopping at `end_version`. -/// -/// This is a thin wrapper around [`StorageHandler::list_from`] that provides the standard -/// Delta log file discovery pipeline. Callers are responsible for handling the `log_tail` -/// (catalog-provided commits) and tracking `max_published_version`. -fn list_from_storage( - storage: &dyn StorageHandler, - log_root: &Url, - start_version: Version, - end_version: Version, -) -> DeltaResult>> { - let start_from = log_root.join(&format!("{start_version:020}"))?; - let files = storage - .list_from(&start_from)? - .map(|meta| ParsedLogPath::try_from(meta?)) - // NOTE: this filters out .crc files etc which start with "." - some engines - // produce `.something.parquet.crc` corresponding to `something.parquet`. Kernel - // doesn't care about these files. Critically, note these are _different_ than - // normal `version.crc` files which are listed + captured normally. Additionally - // we likely aren't even 'seeing' these files since lexicographically the string - // "." comes before the string "0". - .filter_map_ok(|path_opt| path_opt.filter(|p| p.should_list())) - .take_while(move |path_res| match path_res { - // discard any path with too-large version; keep errors - Ok(path) => path.version <= end_version, - Err(_) => true, - }); - Ok(files) -} - -/// Groups all checkpoint parts according to the checkpoint they belong to. -/// -/// NOTE: There could be a single-part and/or any number of uuid-based checkpoints. They -/// are all equivalent, and this routine keeps only one of them (arbitrarily chosen). -fn group_checkpoint_parts(parts: Vec) -> HashMap> { - let mut checkpoints: HashMap> = HashMap::new(); - for part_file in parts { - use LogPathFileType::*; - match &part_file.file_type { - SinglePartCheckpoint - | UuidCheckpoint - | MultiPartCheckpoint { - part_num: 1, - num_parts: 1, - } => { - // All single-file checkpoints are equivalent, just keep one - checkpoints.insert(1, vec![part_file]); - } - MultiPartCheckpoint { - part_num: 1, - num_parts, - } => { - // Start a new multi-part checkpoint with at least 2 parts - checkpoints.insert(*num_parts, vec![part_file]); - } - MultiPartCheckpoint { - part_num, - num_parts, - } => { - // Continue a new multi-part checkpoint with at least 2 parts. - // Checkpoint parts are required to be in-order from log listing to build - // a multi-part checkpoint - if let Some(part_files) = checkpoints.get_mut(num_parts) { - // `part_num` is guaranteed to be non-negative and within `usize` range - if *part_num as usize == 1 + part_files.len() { - // Safe to append because all previous parts exist - part_files.push(part_file); - } - } - } - Commit | StagedCommit | CompactedCommit { .. } | Crc | Unknown => {} - } - } - checkpoints -} - -impl LogSegmentFiles { - pub(crate) fn ascending_commit_files(&self) -> &Vec { - &self.ascending_commit_files - } - - pub(crate) fn ascending_commit_files_mut(&mut self) -> &mut Vec { - &mut self.ascending_commit_files - } - - pub(crate) fn checkpoint_parts(&self) -> &Vec { - &self.checkpoint_parts - } - - pub(crate) fn latest_commit_file(&self) -> &Option { - &self.latest_commit_file - } - - /// List all commits between the provided `start_version` (inclusive) and `end_version` - /// (inclusive). All other types are ignored. - pub(crate) fn list_commits( - storage: &dyn StorageHandler, - log_root: &Url, - start_version: Option, - end_version: Option, - ) -> DeltaResult { - // TODO: plumb through a log_tail provided by our caller - let start = start_version.unwrap_or(0); - let end = end_version.unwrap_or(Version::MAX); - let fs_iter = list_from_storage(storage, log_root, start, end)?; - - let mut listed_commits = Vec::new(); - let mut max_published_version: Option = None; - - for file_result in fs_iter { - let file = file_result?; - if matches!(file.file_type, LogPathFileType::Commit) { - max_published_version = max_published_version.max(Some(file.version)); - listed_commits.push(file); - } - } - - let latest_commit_file = listed_commits.last().cloned(); - Ok(LogSegmentFiles { - ascending_commit_files: listed_commits, - latest_commit_file, - max_published_version, - ..Default::default() - }) - } - - /// List all commit and checkpoint files with versions above the provided `start_version` (inclusive). - /// If successful, this returns a `LogSegmentFiles`. - /// - /// The `log_tail` is an optional sequence of commits provided by the caller, e.g. via - /// [`SnapshotBuilder::with_log_tail`]. It may contain either published or staged commits. The - /// `log_tail` must strictly adhere to being a 'tail' — a contiguous cover of versions `X..=Y` - /// where `Y` is the latest version of the table. If it overlaps with commits listed from the - /// filesystem, the `log_tail` will take precedence for commits; non-commit files (CRC, - /// checkpoints, compactions) are always taken from the filesystem. - // TODO: encode some of these guarantees in the output types. e.g. we could have: - // - SortedCommitFiles: Vec, is_ascending: bool, end_version: Version - // - CheckpointParts: Vec, checkpoint_version: Version (guarantee all same version) - #[instrument(name = "log.list", skip_all, fields(start = ?start_version, end = ?end_version), err)] - pub(crate) fn list( - storage: &dyn StorageHandler, - log_root: &Url, - log_tail: Vec, - start_version: Option, - end_version: Option, - ) -> DeltaResult { - // check log_tail is only commits - // note that LogSegment checks no gaps/duplicates so we don't duplicate that here - debug_assert!( - log_tail.iter().all(|entry| entry.is_commit()), - "log_tail should only contain commits" - ); - - let start = start_version.unwrap_or(0); - let end = end_version.unwrap_or(Version::MAX); - let log_tail_start_version: Option = log_tail.first().map(|f| f.version); - - // Helper that accumulates and groups log files during listing. Each "group" consists of all - // files that share the same version number (e.g., commit, checkpoint parts, CRC files). - // - // We need to group by version because: - // 1. A version may have multiple checkpoint parts that must be collected before we can - // determine if the checkpoint is complete - // 2. If a complete checkpoint exists, we can discard all commits before it - // - // Groups are flushed (processed) when we encounter a file with a different version or - // reach EOF, at which point we check for complete checkpoints and update our state. - #[derive(Default)] - struct ListingAccumulator { - /// The result being built up - output: LogSegmentFiles, - /// Staging area for checkpoint parts at the current version group; always empty when iteration ends - new_checkpoint_parts: Vec, - /// End-version bound used in process_file() to filter CompactedCommit files - end_version: Option, - } - - impl ListingAccumulator { - fn process_file(&mut self, file: ParsedLogPath) { - use LogPathFileType::*; - match file.file_type { - Commit | StagedCommit => self.output.ascending_commit_files.push(file), - CompactedCommit { hi } if self.end_version.is_none_or(|end| hi <= end) => { - self.output.ascending_compaction_files.push(file); - } - CompactedCommit { .. } => (), // Failed the bounds check above - SinglePartCheckpoint | UuidCheckpoint | MultiPartCheckpoint { .. } => { - self.new_checkpoint_parts.push(file) - } - Crc => { - self.output.latest_crc_file.replace(file); - } - Unknown => { - // It is possible that there are other files being stashed away into - // _delta_log/ This is not necessarily forbidden, but something we - // want to know about in a debugging scenario - debug!( - "Found file {} with unknown file type {:?} at version {}", - file.filename, file.file_type, file.version - ); - } - } - } - - /// Called before processing each new file. If `file_version` differs from the current - /// `group_version`, finalizes the current group by calling `flush_checkpoint_group`, - /// then advances `group_version` to the new version. On the first call (when - /// `group_version` is `None`), simply initializes it. - fn maybe_flush_and_advance( - &mut self, - file_version: Version, - group_version: &mut Option, - ) { - match *group_version { - Some(gv) if file_version != gv => { - self.flush_checkpoint_group(gv); - *group_version = Some(file_version); - } - None => { - *group_version = Some(file_version); - } - _ => {} // same version, no flush needed - } - } - - // Group and find the first complete checkpoint for this version. - // All checkpoints for the same version are equivalent, so we only take one. - // - // If this version has a complete checkpoint, we can drop the existing commit and - // compaction files we collected so far -- except we must keep the latest commit. - fn flush_checkpoint_group(&mut self, version: Version) { - let new_checkpoint_parts = std::mem::take(&mut self.new_checkpoint_parts); - if let Some((_, complete_checkpoint)) = group_checkpoint_parts(new_checkpoint_parts) - .into_iter() - // `num_parts` is guaranteed to be non-negative and within `usize` range - .find(|(num_parts, part_files)| part_files.len() == *num_parts as usize) - { - self.output.checkpoint_parts = complete_checkpoint; - // Check if there's a commit file at the same version as this checkpoint. We pop - // the last element from ascending_commit_files (which is sorted by version) and - // set latest_commit_file to it only if it matches the checkpoint version. If it - // doesn't match, we set latest_commit_file to None to discard any older commits - // from before the checkpoint - self.output.latest_commit_file = self - .output - .ascending_commit_files - .pop() - .filter(|commit| commit.version == version); - // Log replay only uses commits/compactions after a complete checkpoint - self.output.ascending_commit_files.clear(); - self.output.ascending_compaction_files.clear(); - // Drop CRC file if older than checkpoint (CRC must be >= checkpoint version) - if self - .output - .latest_crc_file - .as_ref() - .is_some_and(|crc| crc.version < version) - { - self.output.latest_crc_file = None; - } - } - } - } - - let mut builder = ListingAccumulator { - end_version, - ..Default::default() - }; - let mut group_version: Option = None; - - // Phase 1: Stream filesystem files lazily (no collect). - // We always list from the filesystem even when the log_tail covers the entire commit - // range, because non-commit files (CRC, checkpoints, compactions) only exist on the - // filesystem — the log_tail only provides commit files. - let fs_iter = list_from_storage(storage, log_root, start, end)?; - for file_result in fs_iter { - let file = file_result?; - - // Track max published commit version from ALL filesystem Commit files, - // including those that will be skipped because log_tail takes precedence. - if matches!(file.file_type, LogPathFileType::Commit) { - builder.output.max_published_version = - builder.output.max_published_version.max(Some(file.version)); - } - - // Skip filesystem commits at versions covered by the log_tail (the log_tail - // is authoritative for commits). Non-commit files are always kept. - if file.is_commit() - && log_tail_start_version.is_some_and(|tail_start| file.version >= tail_start) - { - continue; - } - - builder.maybe_flush_and_advance(file.version, &mut group_version); - builder.process_file(file); - } - - // Phase 2: Process log_tail entries. We do this after Phase 1 because log_tail commits - // start at log_tail_start_version and are in ascending version order — they always extend - // (or overlap with, but supersede) the filesystem-listed commits. Processing them after - // Phase 1 maintains ascending version order throughout, which is required by the checkpoint - // grouping logic. Note that Phase 1 already skipped filesystem commits at log_tail - // versions, so there's no duplication here. - let filtered_log_tail = log_tail - .into_iter() - .filter(|entry| entry.version >= start && entry.version <= end); - for file in filtered_log_tail { - // Track max published version for published commits from the log_tail - if matches!(file.file_type, LogPathFileType::Commit) { - builder.output.max_published_version = - builder.output.max_published_version.max(Some(file.version)); - } - - builder.maybe_flush_and_advance(file.version, &mut group_version); - builder.process_file(file); - } - - // Flush the final group - if let Some(gv) = group_version { - builder.flush_checkpoint_group(gv); - } - - // Since ascending_commit_files is cleared at each checkpoint, if it's non-empty here - // it contains only commits after the most recent checkpoint. The last element is the - // highest version commit overall, so we update latest_commit_file to it. If it's empty, - // we keep the value set at the checkpoint (if a commit existed at the checkpoint version), - // or remains None. - if let Some(commit_file) = builder.output.ascending_commit_files.last() { - builder.output.latest_commit_file = Some(commit_file.clone()); - } - - Ok(LogSegmentFiles { ..builder.output }) - } - - /// List all commit and checkpoint files after the provided checkpoint. It is guaranteed that all - /// the returned [`ParsedLogPath`]s will have a version less than or equal to the `end_version`. - /// See [`list_log_files_with_version`] for details on the return type. - pub(crate) fn list_with_checkpoint_hint( - checkpoint_metadata: &LastCheckpointHint, - storage: &dyn StorageHandler, - log_root: &Url, - log_tail: Vec, - end_version: Option, - ) -> DeltaResult { - let listed_files = Self::list( - storage, - log_root, - log_tail, - Some(checkpoint_metadata.version), - end_version, - )?; - - let Some(latest_checkpoint) = listed_files.checkpoint_parts.last() else { - // TODO: We could potentially recover here - return Err(Error::invalid_checkpoint( - "Had a _last_checkpoint hint but didn't find any checkpoints", - )); - }; - if latest_checkpoint.version != checkpoint_metadata.version { - info!( - "_last_checkpoint hint is out of date. _last_checkpoint version: {}. Using actual most recent: {}", - checkpoint_metadata.version, - latest_checkpoint.version - ); - } else if listed_files.checkpoint_parts.len() != checkpoint_metadata.parts.unwrap_or(1) { - return Err(Error::InvalidCheckpoint(format!( - "_last_checkpoint indicated that checkpoint should have {} parts, but it has {}", - checkpoint_metadata.parts.unwrap_or(1), - listed_files.checkpoint_parts.len() - ))); - } - Ok(listed_files) - } -} - -#[cfg(test)] -mod list_log_files_with_log_tail_tests { - use std::sync::Arc; - - use url::Url; - - use crate::engine::default::executor::tokio::TokioBackgroundExecutor; - use crate::engine::default::filesystem::ObjectStoreStorageHandler; - use crate::object_store::{memory::InMemory, path::Path as ObjectPath, ObjectStore}; - use crate::FileMeta; - - use super::*; - - // size markers used to identify commit sources in tests - const FILESYSTEM_SIZE_MARKER: u64 = 10; - const CATALOG_SIZE_MARKER: u64 = 7; - - #[derive(Debug, Clone, Copy, PartialEq, Eq)] - enum CommitSource { - Filesystem, - Catalog, - } - - // create test storage given list of log files with custom data content - async fn create_storage( - log_files: Vec<(Version, LogPathFileType, CommitSource)>, - ) -> (Box, Url) { - let store = Arc::new(InMemory::new()); - let log_root = Url::parse("memory:///_delta_log/").unwrap(); - - for (version, file_type, source) in log_files { - let path = match file_type { - LogPathFileType::Commit => { - format!("_delta_log/{version:020}.json") - } - LogPathFileType::StagedCommit => { - let uuid = uuid::Uuid::new_v4(); - format!("_delta_log/_staged_commits/{version:020}.{uuid}.json") - } - LogPathFileType::SinglePartCheckpoint => { - format!("_delta_log/{version:020}.checkpoint.parquet") - } - LogPathFileType::MultiPartCheckpoint { - part_num, - num_parts, - } => { - format!( - "_delta_log/{version:020}.checkpoint.{part_num:010}.{num_parts:010}.parquet" - ) - } - LogPathFileType::Crc => { - format!("_delta_log/{version:020}.crc") - } - LogPathFileType::CompactedCommit { hi } => { - format!("_delta_log/{version:020}.{hi:020}.compacted.json") - } - LogPathFileType::UuidCheckpoint | LogPathFileType::Unknown => { - panic!("Unsupported file type in test: {file_type:?}") - } - }; - let data = match source { - CommitSource::Filesystem => bytes::Bytes::from("filesystem"), - CommitSource::Catalog => bytes::Bytes::from("catalog"), - }; - store - .put(&ObjectPath::from(path.as_str()), data.into()) - .await - .expect("Failed to put test file"); - } - - let executor = Arc::new(TokioBackgroundExecutor::new()); - let storage = Box::new(ObjectStoreStorageHandler::new(store, executor, None)); - (storage, log_root) - } - - // helper to create a ParsedLogPath with specific source marker - fn make_parsed_log_path_with_source( - version: Version, - file_type: LogPathFileType, - source: CommitSource, - ) -> ParsedLogPath { - let url = Url::parse(&format!("memory:///_delta_log/{version:020}.json")).unwrap(); - let mut filename_path_segments = url.path_segments().unwrap(); - let filename = filename_path_segments.next_back().unwrap().to_string(); - let extension = filename.split('.').next_back().unwrap().to_string(); - - let size = match source { - CommitSource::Filesystem => FILESYSTEM_SIZE_MARKER, - CommitSource::Catalog => CATALOG_SIZE_MARKER, - }; - - let location = FileMeta { - location: url, - last_modified: 0, - size, - }; - - ParsedLogPath { - location, - filename, - extension, - version, - file_type, - } - } - - fn assert_source(commit: &ParsedLogPath, expected_source: CommitSource) { - let expected_size = match expected_source { - CommitSource::Filesystem => FILESYSTEM_SIZE_MARKER, - CommitSource::Catalog => CATALOG_SIZE_MARKER, - }; - assert_eq!( - commit.location.size, expected_size, - "Commit version {} should be from {:?}, but size was {}", - commit.version, expected_source, commit.location.size - ); - } - - /// Helper to call `LogSegmentFiles::list()` and destructure the result for assertions. - /// Returns (ascending_commit_files, ascending_compaction_files, checkpoint_parts, - /// latest_crc_file, latest_commit_file, max_published_version). - #[allow(clippy::type_complexity)] - fn list_and_destructure( - storage: &dyn StorageHandler, - log_root: &Url, - log_tail: Vec, - start_version: Option, - end_version: Option, - ) -> ( - Vec, - Vec, - Vec, - Option, - Option, - Option, - ) { - let r = - LogSegmentFiles::list(storage, log_root, log_tail, start_version, end_version).unwrap(); - ( - r.ascending_commit_files, - r.ascending_compaction_files, - r.checkpoint_parts, - r.latest_crc_file, - r.latest_commit_file, - r.max_published_version, - ) - } - - #[tokio::test] - async fn test_empty_log_tail() { - let log_files = vec![ - (0, LogPathFileType::Commit, CommitSource::Filesystem), - (1, LogPathFileType::Commit, CommitSource::Filesystem), - (2, LogPathFileType::Commit, CommitSource::Filesystem), - ]; - let (storage, log_root) = create_storage(log_files).await; - - let (commits, _, _, _, latest_commit, max_pub) = - list_and_destructure(storage.as_ref(), &log_root, vec![], Some(1), Some(2)); - - assert_eq!(commits.len(), 2); - assert_eq!(commits[0].version, 1); - assert_eq!(commits[1].version, 2); - assert_source(&commits[0], CommitSource::Filesystem); - assert_source(&commits[1], CommitSource::Filesystem); - assert_eq!(latest_commit.unwrap().version, 2); - assert_eq!(max_pub, Some(2)); - } - - #[tokio::test] - async fn test_log_tail_has_latest_commit_files() { - // Filesystem has commits 0-2, log_tail has commits 3-5 (the latest) - let log_files = vec![ - (0, LogPathFileType::Commit, CommitSource::Filesystem), - (1, LogPathFileType::Commit, CommitSource::Filesystem), - (2, LogPathFileType::Commit, CommitSource::Filesystem), - ]; - let (storage, log_root) = create_storage(log_files).await; - - let log_tail = vec![ - make_parsed_log_path_with_source(3, LogPathFileType::Commit, CommitSource::Catalog), - make_parsed_log_path_with_source(4, LogPathFileType::Commit, CommitSource::Catalog), - make_parsed_log_path_with_source(5, LogPathFileType::Commit, CommitSource::Catalog), - ]; - - let (commits, _, _, _, latest_commit, max_pub) = - list_and_destructure(storage.as_ref(), &log_root, log_tail, Some(0), Some(5)); - - assert_eq!(commits.len(), 6); - // filesystem commits 0-2 - for (i, commit) in commits.iter().enumerate().take(3) { - assert_eq!(commit.version, i as u64); - assert_source(commit, CommitSource::Filesystem); - } - // catalog commits 3-5 - for (i, commit) in commits.iter().enumerate().skip(3) { - assert_eq!(commit.version, i as u64); - assert_source(commit, CommitSource::Catalog); - } - assert_eq!(latest_commit.unwrap().version, 5); - assert_eq!(max_pub, Some(5)); - } - - #[tokio::test] - async fn test_request_subset_with_log_tail() { - // Test requesting a subset when log_tail is the latest commits - let log_files = vec![ - (0, LogPathFileType::Commit, CommitSource::Filesystem), - (1, LogPathFileType::Commit, CommitSource::Filesystem), - ]; - let (storage, log_root) = create_storage(log_files).await; - - // log_tail represents versions 2-4 (latest commits) - let log_tail = vec![ - make_parsed_log_path_with_source(2, LogPathFileType::Commit, CommitSource::Catalog), - make_parsed_log_path_with_source(3, LogPathFileType::Commit, CommitSource::Catalog), - make_parsed_log_path_with_source(4, LogPathFileType::Commit, CommitSource::Catalog), - ]; - - // list for only versions 1-3 - let (commits, _, _, _, latest_commit, max_pub) = - list_and_destructure(storage.as_ref(), &log_root, log_tail, Some(1), Some(3)); - - assert_eq!(commits.len(), 3); - assert_eq!(commits[0].version, 1); - assert_eq!(commits[1].version, 2); - assert_eq!(commits[2].version, 3); - assert_source(&commits[0], CommitSource::Filesystem); - assert_source(&commits[1], CommitSource::Catalog); - assert_source(&commits[2], CommitSource::Catalog); - assert_eq!(latest_commit.unwrap().version, 3); - assert_eq!(max_pub, Some(3)); - } - - #[tokio::test] - async fn test_log_tail_defines_latest_version() { - // log_tail defines the latest version of the table: if there is file system files after log - // tail, they are ignored. But we still list all filesystem files to track max_published_version. - let log_files = vec![ - (0, LogPathFileType::Commit, CommitSource::Filesystem), - (1, LogPathFileType::Commit, CommitSource::Filesystem), - (2, LogPathFileType::Commit, CommitSource::Filesystem), // <-- max_published_version - ]; - let (storage, log_root) = create_storage(log_files).await; - - // log_tail is just [1], indicating version 1 is the latest - let log_tail = vec![make_parsed_log_path_with_source( - 1, - LogPathFileType::Commit, - CommitSource::Catalog, - )]; - - let (commits, _, _, _, latest_commit, max_pub) = - list_and_destructure(storage.as_ref(), &log_root, log_tail, Some(0), None); - - // expect only 0 from file system and 1 from log tail - assert_eq!(commits.len(), 2); - assert_eq!(commits[0].version, 0); - assert_eq!(commits[1].version, 1); - assert_source(&commits[0], CommitSource::Filesystem); - assert_source(&commits[1], CommitSource::Catalog); - assert_eq!(latest_commit.unwrap().version, 1); - // max_published_version should reflect the highest published commit on filesystem - assert_eq!(max_pub, Some(2)); - } - - #[test] - fn test_log_tail_covers_entire_range_empty_filesystem() { - // Test-only storage handler that returns an empty listing. - // When the log_tail covers the entire commit range, we still call list_from - // (to pick up non-commit files like CRC/checkpoints), but the filesystem may - // have nothing — e.g. a purely catalog-managed table. - struct EmptyStorageHandler; - impl StorageHandler for EmptyStorageHandler { - fn list_from( - &self, - _path: &Url, - ) -> DeltaResult>>> { - Ok(Box::new(std::iter::empty())) - } - fn read_files( - &self, - _files: Vec, - ) -> DeltaResult>>> { - panic!("read_files should not be called during listing"); - } - fn put(&self, _path: &Url, _data: bytes::Bytes, _overwrite: bool) -> DeltaResult<()> { - panic!("put should not be called during listing"); - } - fn copy_atomic(&self, _src: &Url, _dest: &Url) -> DeltaResult<()> { - panic!("copy_atomic should not be called during listing"); - } - fn head(&self, _path: &Url) -> DeltaResult { - panic!("head should not be called during listing"); - } - } - - // log_tail covers versions 0-2, the entire range - let log_tail = vec![ - make_parsed_log_path_with_source(0, LogPathFileType::Commit, CommitSource::Catalog), - make_parsed_log_path_with_source(1, LogPathFileType::Commit, CommitSource::Catalog), - make_parsed_log_path_with_source( - 2, - LogPathFileType::StagedCommit, - CommitSource::Catalog, - ), - ]; - - let storage = EmptyStorageHandler; - let url = Url::parse("memory:///anything/_delta_log/").unwrap(); - let (commits, _, _, _, latest_commit, max_pub) = - list_and_destructure(&storage, &url, log_tail, Some(0), Some(2)); - - // Only log_tail commits should appear (filesystem is empty) - assert_eq!(commits.len(), 3); - assert_eq!(commits[0].version, 0); - assert_eq!(commits[1].version, 1); - assert_eq!(commits[2].version, 2); - assert_source(&commits[0], CommitSource::Catalog); - assert_source(&commits[1], CommitSource::Catalog); - assert_source(&commits[2], CommitSource::Catalog); - assert_eq!(latest_commit.unwrap().version, 2); - // Only published (non-staged) commits from log_tail count for max_published_version - assert_eq!(max_pub, Some(1)); - } - - #[tokio::test] - async fn test_log_tail_covers_entire_range_with_crc() { - // When log_tail covers the entire requested range (starts at version 0), commit files - // from the filesystem should be excluded (log_tail is authoritative for commits), but - // non-commit files (CRC, checkpoints) should still be picked up from the filesystem. - let log_files = vec![ - (0, LogPathFileType::Commit, CommitSource::Filesystem), - (1, LogPathFileType::Commit, CommitSource::Filesystem), - (2, LogPathFileType::Crc, CommitSource::Filesystem), - ]; - let (storage, log_root) = create_storage(log_files).await; - - // log_tail covers versions 0-2, which includes the entire range we'll request - let log_tail = vec![ - make_parsed_log_path_with_source(0, LogPathFileType::Commit, CommitSource::Catalog), - make_parsed_log_path_with_source(1, LogPathFileType::Commit, CommitSource::Catalog), - make_parsed_log_path_with_source( - 2, - LogPathFileType::StagedCommit, - CommitSource::Catalog, - ), - ]; - - let (commits, _, _, latest_crc, latest_commit, max_pub) = - list_and_destructure(storage.as_ref(), &log_root, log_tail, Some(0), Some(2)); - - // 3 commits from log_tail: 0, 1, 2 - assert_eq!(commits.len(), 3); - assert_source(&commits[0], CommitSource::Catalog); - assert_source(&commits[1], CommitSource::Catalog); - assert_source(&commits[2], CommitSource::Catalog); - - // CRC at version 2 from filesystem is preserved - let crc = latest_crc.unwrap(); - assert_eq!(crc.version, 2); - assert!(matches!(crc.file_type, LogPathFileType::Crc)); - - assert_eq!(latest_commit.unwrap().version, 2); - // Only published commits count: filesystem 0,1 (skipped but tracked) + log_tail 0,1 - assert_eq!(max_pub, Some(1)); - } - - #[tokio::test] - async fn test_listing_omits_staged_commits() { - // note that in the presence of staged commits, we CANNOT trust listing to determine which - // to include in our listing/log segment. This is up to the catalog. (e.g. version - // 5.uuid1.json and 5.uuid2.json can both exist and only catalog can say which is the 'real' - // version 5). - - let log_files = vec![ - (0, LogPathFileType::Commit, CommitSource::Filesystem), - (1, LogPathFileType::Commit, CommitSource::Filesystem), // <-- max_published_version - (1, LogPathFileType::StagedCommit, CommitSource::Filesystem), - (2, LogPathFileType::StagedCommit, CommitSource::Filesystem), - ]; - - let (storage, log_root) = create_storage(log_files).await; - let (commits, _, _, _, latest_commit, max_pub) = - list_and_destructure(storage.as_ref(), &log_root, vec![], None, None); - - // we must only see two regular commits - assert_eq!(commits.len(), 2); - assert_eq!(commits[0].version, 0); - assert_eq!(commits[1].version, 1); - assert_source(&commits[0], CommitSource::Filesystem); - assert_source(&commits[1], CommitSource::Filesystem); - assert_eq!(latest_commit.unwrap().version, 1); - assert_eq!(max_pub, Some(1)); - } - - #[tokio::test] - async fn test_listing_with_large_end_version() { - let log_files = vec![ - (0, LogPathFileType::Commit, CommitSource::Filesystem), - (1, LogPathFileType::Commit, CommitSource::Filesystem), // <-- max_published_version - (2, LogPathFileType::StagedCommit, CommitSource::Filesystem), - ]; - - let (storage, log_root) = create_storage(log_files).await; - // note we let you request end version past the end of log. up to consumer to interpret - let (commits, _, _, _, latest_commit, max_pub) = - list_and_destructure(storage.as_ref(), &log_root, vec![], None, Some(3)); - - // we must only see two regular commits - assert_eq!(commits.len(), 2); - assert_eq!(commits[0].version, 0); - assert_eq!(commits[1].version, 1); - assert_eq!(latest_commit.unwrap().version, 1); - assert_eq!(max_pub, Some(1)); - } - - #[tokio::test] - async fn test_non_commit_files_at_log_tail_versions_are_preserved() { - // Filesystem has commits 0-5, a checkpoint at version 7, and a CRC at version 8. - // Log tail provides commits 6-10. The checkpoint and CRC are on the filesystem - // at versions covered by the log_tail and must NOT be filtered out. - // - // After processing through LogListingGroupBuilder, the checkpoint at version 7 - // causes commits before it to be cleared, keeping only commits after the checkpoint. - let log_files = vec![ - (0, LogPathFileType::Commit, CommitSource::Filesystem), - (1, LogPathFileType::Commit, CommitSource::Filesystem), - (2, LogPathFileType::Commit, CommitSource::Filesystem), - (3, LogPathFileType::Commit, CommitSource::Filesystem), - (4, LogPathFileType::Commit, CommitSource::Filesystem), - (5, LogPathFileType::Commit, CommitSource::Filesystem), - ( - 7, - LogPathFileType::SinglePartCheckpoint, - CommitSource::Filesystem, - ), - (8, LogPathFileType::Crc, CommitSource::Filesystem), - ]; - let (storage, log_root) = create_storage(log_files).await; - - let log_tail = vec![ - make_parsed_log_path_with_source(6, LogPathFileType::Commit, CommitSource::Catalog), - make_parsed_log_path_with_source(7, LogPathFileType::Commit, CommitSource::Catalog), - make_parsed_log_path_with_source(8, LogPathFileType::Commit, CommitSource::Catalog), - make_parsed_log_path_with_source(9, LogPathFileType::Commit, CommitSource::Catalog), - make_parsed_log_path_with_source(10, LogPathFileType::Commit, CommitSource::Catalog), - ]; - - let (commits, _, checkpoint_parts, latest_crc, latest_commit, max_pub) = - list_and_destructure(storage.as_ref(), &log_root, log_tail, Some(0), Some(10)); - - // Checkpoint at version 7 is preserved from filesystem - assert_eq!(checkpoint_parts.len(), 1); - assert_eq!(checkpoint_parts[0].version, 7); - assert!(checkpoint_parts[0].is_checkpoint()); - - // CRC at version 8 is preserved from filesystem - let crc = latest_crc.unwrap(); - assert_eq!(crc.version, 8); - assert!(matches!(crc.file_type, LogPathFileType::Crc)); - - // After checkpoint processing: commits before checkpoint are cleared, - // only log_tail commits 6-10 remain (added after checkpoint flush) - assert_eq!(commits.len(), 5); - for (i, commit) in commits.iter().enumerate() { - assert_eq!(commit.version, (i + 6) as u64); - assert_source(commit, CommitSource::Catalog); - } - assert_eq!(latest_commit.unwrap().version, 10); - - // max_published_version reflects all published commits seen (filesystem 0-5 + log_tail 6-10) - assert_eq!(max_pub, Some(10)); - } -} diff --git a/kernel/src/log_segment_files/mod.rs b/kernel/src/log_segment_files/mod.rs new file mode 100644 index 0000000000..5f227e4147 --- /dev/null +++ b/kernel/src/log_segment_files/mod.rs @@ -0,0 +1,695 @@ +//! [`LogSegmentFiles`] is a struct holding the result of listing the delta log. Currently, it +//! exposes four APIs for listing: +//! 1. `list_commits`: Lists all commit files between the provided start and end versions. +//! 2. `list`: Lists all commit and checkpoint files between the provided start and end versions. +//! 3. `list_with_checkpoint_hint`: Lists all commit and checkpoint files after the provided +//! checkpoint hint. +//! 4. `list_with_backward_checkpoint_scan`: Scans backward from an end version in 1000-version +//! windows until a complete checkpoint is found or the log is exhausted. +//! +//! After listing, one can leverage the [`LogSegmentFiles`] to construct a [`LogSegment`]. +//! +//! [`LogSegment`]: crate::log_segment::LogSegment + +use std::collections::HashMap; + +use delta_kernel_derive::internal_api; +use itertools::Itertools; +use tracing::{debug, info, instrument, warn}; +use url::Url; + +use crate::last_checkpoint_hint::LastCheckpointHint; +use crate::path::LogPathFileType::*; +use crate::path::{ + may_begin_listable_log_path, CheckpointInstance, LogPathFileType, ParsedLogPath, +}; +use crate::{DeltaResult, Error, StorageHandler, Version}; + +#[cfg(test)] +mod tests; + +/// Represents the set of log files found during a listing operation in the Delta log directory. +/// +/// - `ascending_commit_files`: All commit and staged commit files found, sorted by version. May +/// contain gaps. +/// - `ascending_compaction_files`: All compaction commit files found, sorted by version. +/// - `checkpoint_parts`: All parts of the most recent complete checkpoint (all same version). Empty +/// if no checkpoint found. A version can hold several complete checkpoints; see +/// [`group_checkpoint_parts`] for which one this is. +/// - `latest_crc_file`: The CRC file with the highest version, only if version >= checkpoint +/// version. +/// - `latest_commit_file`: The commit file with the highest version, or `None` if no commits were +/// found. This field may be present even when `ascending_commit_files` is empty, such as when a +/// checkpoint subsumes all commits. In that case, it is retained because downstream code (e.g. +/// In-Commit Timestamp reading) needs access to the commit file at the snapshot version. +/// - `max_published_version`: The highest published commit file version, or `None` if no published +/// commits were found. +#[derive(Debug, Default, Clone, PartialEq, Eq)] +#[internal_api] +pub(crate) struct LogSegmentFiles { + pub ascending_commit_files: Vec, + pub ascending_compaction_files: Vec, + pub checkpoint_parts: Vec, + pub latest_crc_file: Option, + pub latest_commit_file: Option, + pub max_published_version: Option, +} + +/// Returns a lazy iterator of [`ParsedLogPath`]s from the filesystem over versions +/// `[start_version, end_version]`. The iterator handles parsing, filtering out non-listable +/// files (e.g. dot-prefixed files), and stopping at `end_version`. It stops consuming the +/// underlying listing at the first path past the version-named region, so directories like +/// `_staged_commits/` and `_sidecars/` are never paged through. +/// +/// This is a thin wrapper around [`StorageHandler::list_from`] that provides the standard +/// Delta log file discovery pipeline. Callers are responsible for handling the `log_tail` +/// (catalog-provided commits) and tracking `max_published_version`. +#[internal_api] +pub(crate) fn list_delta_log_from_storage( + storage: &dyn StorageHandler, + log_root: &Url, + start_version: Version, + end_version: Version, +) -> DeltaResult>> { + let start_from = log_root.join(&format!("{start_version:020}"))?; + let log_root_str = log_root.to_string(); + let files = storage + .list_from(&start_from)? + // The listing is sorted by full path, so nothing relevant follows the first relative path + // past the version-named region (see `may_begin_listable_log_path`). Stopping there avoids + // paging through `_staged_commits/` and `_sidecars/`, which can hold thousands of files. + // A path that doesn't strip the log_root prefix is kept; parsing discards it. + // TODO(#2740): push the bound into the listing request itself. + .take_while(move |meta_res| match meta_res { + Ok(meta) => meta + .location + .as_str() + .strip_prefix(&log_root_str) + .is_none_or(may_begin_listable_log_path), + Err(_) => true, + }) + .map(|meta| ParsedLogPath::try_from(meta?)) + // NOTE: this filters out .crc files etc which start with "." - some engines + // produce `.something.parquet.crc` corresponding to `something.parquet`. Kernel + // doesn't care about these files. Critically, note these are _different_ than + // normal `version.crc` files which are listed + captured normally. Additionally + // we likely aren't even 'seeing' these files since lexicographically the string + // "." comes before the string "0". + .filter_map_ok(|path_opt| path_opt.filter(|p| p.should_list())) + .take_while(move |path_res| match path_res { + // discard any path with too-large version; keep errors + Ok(path) => path.version <= end_version, + Err(_) => true, + }); + Ok(files) +} + +/// Groups all checkpoint parts according to the checkpoint they belong to. +/// +/// Several _complete_ checkpoints can legitimately share a version, say two multi-part checkpoints +/// with different part counts, or two uuid-named ones. Each gets its own [`CheckpointInstance`] +/// key, so the caller can pick a winner deterministically (see +/// `ListingAccumulator::select_checkpoint_for_group`). +/// +/// `parts` must arrive in ascending file name order, which log listing provides: a multi-part +/// checkpoint only accumulates while its parts arrive in order. +#[internal_api] +fn group_checkpoint_parts( + parts: Vec, +) -> HashMap> { + debug_assert!( + parts.is_sorted_by_key(|p| &p.filename), + "checkpoint parts must arrive in ascending file name order" + ); + let mut checkpoints: HashMap> = HashMap::new(); + for part_file in parts { + match &part_file.file_type { + // A single-file checkpoint is complete on its own. Keying uuid-named ones on file name + // keeps two of them at the same version separate. + ClassicCheckpoint | UuidCheckpoint => { + if let Some(instance) = CheckpointInstance::of(&part_file) { + checkpoints.insert(instance, vec![part_file]); + } + } + MultiPartCheckpoint { + part_num: 1, + num_parts, + } => { + // Start a new multi-part checkpoint + checkpoints.insert( + CheckpointInstance::MultiPart { + num_parts: *num_parts, + }, + vec![part_file], + ); + } + MultiPartCheckpoint { + part_num, + num_parts, + } => { + // Continue a multi-part checkpoint. + // Checkpoint parts are required to be in-order from log listing to build + // a multi-part checkpoint + if let Some(part_files) = checkpoints.get_mut(&CheckpointInstance::MultiPart { + num_parts: *num_parts, + }) { + if *part_num as usize == 1 + part_files.len() { + // Safe to append because all previous parts exist + part_files.push(part_file); + } + } + } + Commit | StagedCommit | CompactedCommit { .. } | Crc | Unknown => {} + } + } + checkpoints +} + +/// Returns the version of the latest complete checkpoint in `files`, or `None` if no complete +/// checkpoint exists. Skips 0-byte checkpoint files so they don't count toward completeness. +fn find_complete_checkpoint_version(ascending_files: &[ParsedLogPath]) -> Option { + ascending_files + .iter() + .filter(|f| f.is_checkpoint() && should_process_log_file(f)) + .chunk_by(|f| f.version) + .into_iter() + .filter_map(|(version, parts)| { + let owned: Vec = parts.cloned().collect(); + group_checkpoint_parts(owned) + .iter() + .any(|(instance, part_files)| instance.is_complete(part_files)) + .then_some(version) + }) + .last() +} + +/// Validates a log file's size. Returns `true` to keep the file, `false` to skip it +/// (with a warning already emitted). +/// +/// Compaction and checkpoint files are skipped when empty -- they have fallbacks +/// (individual commits, older checkpoints). Commit and CRC files are kept even +/// if empty; the warning ensures the corrupt file is identifiable in logs. +#[internal_api] +pub(crate) fn should_process_log_file(file: &ParsedLogPath) -> bool { + if file.location.size > 0 { + return true; + } + match file.file_type { + // Commit files are kept even if 0 bytes -- the downstream JSON handler might + // error, but the warning here ensures the corrupt file is identifiable in logs. + // We don't skip commits because they are the source of truth for table state. + Commit | StagedCommit => { + warn!( + "{:?} file is empty (0 bytes): {}", + file.file_type, file.location.location, + ); + return true; + } + CompactedCommit { .. } => { + warn!( + "Skipping empty (0 byte) compacted log file {}, \ + falling back to individual commits", + file.location.location, + ); + } + ClassicCheckpoint | UuidCheckpoint | MultiPartCheckpoint { .. } => { + warn!( + "Skipping empty (0 byte) checkpoint file: {}", + file.location.location, + ); + } + // CRC files are optional and may report size 0 on some platforms. + // Keep them -- the CRC reader handles empty/invalid content. + Crc => { + warn!("CRC file is empty (0 bytes): {}", file.location.location,); + return true; + } + Unknown => return true, + } + false +} + +/// Accumulates and groups log files during listing. Each "group" consists of all files that +/// share the same version number (e.g., commit, checkpoint parts, CRC files). +/// +/// We need to group by version because: +/// 1. A version may have multiple checkpoint parts that must be collected before we can determine +/// if the checkpoint is complete +/// 2. If a complete checkpoint exists, we can discard all commits before it +/// +/// Groups are flushed (processed) when we encounter a file with a different version or +/// reach EOF, at which point we check for complete checkpoints and update our state. +#[derive(Default)] +struct ListingAccumulator { + /// The result being built up + output: LogSegmentFiles, + /// Staging area for checkpoint parts at the current version group; always empty when iteration + /// ends + pending_checkpoint_parts: Vec, + /// End-version bound used in process_file() to filter CompactedCommit files + // TODO(#2337): remove allow(dead_code) when log compaction is re-enabled + #[allow(dead_code)] + end_version: Option, + /// The version of the current group being accumulated + group_version: Option, +} + +impl ListingAccumulator { + fn process_file(&mut self, file: ParsedLogPath) { + if !should_process_log_file(&file) { + return; + } + match file.file_type { + Commit | StagedCommit => self.output.ascending_commit_files.push(file), + // TODO(#2337): re-enable log compaction once testing is sufficient + // CompactedCommit { hi } if self.end_version.is_none_or(|end| hi <= end) => { + // self.output.ascending_compaction_files.push(file); + // } + // CompactedCommit { .. } => (), // Failed the bounds check above + CompactedCommit { .. } => { + debug!( + "Skipping unsupported log compaction file: {:?}", + file.location + ); + } + ClassicCheckpoint | UuidCheckpoint | MultiPartCheckpoint { .. } => { + self.pending_checkpoint_parts.push(file) + } + Crc => { + self.output.latest_crc_file.replace(file); + } + Unknown => { + // It is possible that there are other files being stashed away into + // _delta_log/ This is not necessarily forbidden, but something we + // want to know about in a debugging scenario + debug!( + "Found file {} with unknown file type {:?} at version {}", + file.filename, file.file_type, file.version + ); + } + } + } + + /// Called before processing each new file. If `file_version` differs from the current + /// `group_version`, finalizes the current group by calling `select_checkpoint_for_group`, + /// then advances `group_version` to the new version. On the first call (when + /// `group_version` is `None`), simply initializes it. + fn maybe_flush_and_advance(&mut self, file_version: Version) { + match self.group_version { + Some(gv) if file_version != gv => { + self.select_checkpoint_for_group(gv); + self.group_version = Some(file_version); + } + None => { + self.group_version = Some(file_version); + } + _ => {} // same version, no flush needed + } + } + + /// Selects this version's checkpoint. Any of a version's complete checkpoints (see + /// [`group_checkpoint_parts`]) describes the same table state; the choice must be stable across + /// processes (matching Delta-Spark), so we take the greatest in [`CheckpointInstance`] order. + /// + /// When a complete checkpoint exists we drop the commits/compactions collected so far, keeping + /// only the latest commit. + fn select_checkpoint_for_group(&mut self, version: Version) { + let pending_checkpoint_parts = std::mem::take(&mut self.pending_checkpoint_parts); + if let Some((_, complete_checkpoint)) = group_checkpoint_parts(pending_checkpoint_parts) + .into_iter() + .filter(|(instance, part_files)| instance.is_complete(part_files)) + .max_by(|(a, _), (b, _)| a.cmp(b)) + { + self.output.checkpoint_parts = complete_checkpoint; + // Keep the commit at the checkpoint version (if any) before clearing all older commits. + self.output.latest_commit_file = self + .output + .ascending_commit_files + .last() + .filter(|c| c.version == version) + .cloned(); + // Log replay only uses commits/compactions after a complete checkpoint + self.output.ascending_commit_files.clear(); + self.output.ascending_compaction_files.clear(); + // Drop CRC file if older than checkpoint (CRC must be >= checkpoint version) + if self + .output + .latest_crc_file + .as_ref() + .is_some_and(|crc| crc.version < version) + { + self.output.latest_crc_file = None; + } + } + } +} + +/// Number of versions covered by each backward-scan window in +/// `LogSegmentFiles::list_with_backward_checkpoint_scan` +const BACKWARD_SCAN_WINDOW_SIZE: u64 = 1000; + +impl LogSegmentFiles { + /// Assembles a `LogSegmentFiles` from `fs_files` (an iterator of files + /// listed from storage) and `log_tail` (catalog-provided commits). + /// + /// - `fs_files`: files listed from storage in ascending version order + /// - `log_tail`: list of commits that takes precedence over the filesystem ones + /// - `start_version`: start version of the entire listing range provided; in practice, this is + /// the lower bound (inclusive) for log_tail entries included in the result + /// - `end_version`: upper bound (inclusive) on versions to include, `None` means no bound + pub(crate) fn build_log_segment_files( + fs_files: impl Iterator>, + log_tail: Vec, + start_version: Version, + end_version: Option, + ) -> DeltaResult { + // check log_tail is only commits + // note that LogSegment checks no gaps/duplicates so we don't duplicate that here + debug_assert!( + log_tail.iter().all(|entry| entry.is_commit()), + "log_tail should only contain commits" + ); + + let log_tail_start_version = log_tail.first().map(|f| f.version); + let end = end_version.unwrap_or(Version::MAX); + + let mut acc = ListingAccumulator { + end_version, + ..Default::default() + }; + + // Phase 1: Stream filesystem files lazily (no collect). + // We always list from the filesystem even when the log_tail covers the entire commit + // range, because non-commit files (CRC, checkpoints, compactions) only exist on the + // filesystem — the log_tail only provides commit files. + for file_result in fs_files { + let file = file_result?; + + // Track max published commit version from ALL filesystem Commit files, + // including those that will be skipped because log_tail takes precedence. + if matches!(file.file_type, LogPathFileType::Commit) { + acc.output.max_published_version = + acc.output.max_published_version.max(Some(file.version)); + } + + // Skip filesystem commits at versions covered by the log_tail (the log_tail + // is authoritative for commits). Non-commit files are always kept. + if file.is_commit() + && log_tail_start_version.is_some_and(|tail_start| file.version >= tail_start) + { + continue; + } + + acc.maybe_flush_and_advance(file.version); + acc.process_file(file); + } + + // Phase 2: Process log_tail entries. We do this after Phase 1 because log_tail commits + // start at log_tail_start_version and are in ascending version order — they always extend + // (or overlap with, but supersede) the filesystem-listed commits. Processing them after + // Phase 1 maintains ascending version order throughout, which is required by the checkpoint + // grouping logic. Note that Phase 1 already skipped filesystem commits at log_tail + // versions, so there's no duplication here. + // + // log_tail entries at versions before a checkpoint may still be included + // here - LogSegment::try_new is the safeguard that filters those out unconditionally + let filtered_log_tail = log_tail + .into_iter() + .filter(|entry| entry.version >= start_version && entry.version <= end); + for file in filtered_log_tail { + // Track max published version for published commits from the log_tail + if matches!(file.file_type, LogPathFileType::Commit) { + acc.output.max_published_version = + acc.output.max_published_version.max(Some(file.version)); + } + + acc.maybe_flush_and_advance(file.version); + acc.process_file(file); + } + + // Flush the final group + if let Some(gv) = acc.group_version { + acc.select_checkpoint_for_group(gv); + } + + // Since ascending_commit_files is cleared at each checkpoint, if it's non-empty here + // it contains only commits after the most recent checkpoint. The last element is the + // highest version commit overall, so we update latest_commit_file to it. If it's empty, + // we keep the value set at the checkpoint (if a commit existed at the checkpoint version), + // or remains None. + if let Some(commit_file) = acc.output.ascending_commit_files.last() { + acc.output.latest_commit_file = Some(commit_file.clone()); + } + + Ok(acc.output) + } + + pub(crate) fn ascending_commit_files(&self) -> &Vec { + &self.ascending_commit_files + } + + /// The staged (unpublished) commit files, in ascending version order. + pub(crate) fn staged_commits(&self) -> impl Iterator { + self.ascending_commit_files + .iter() + .filter(|f| f.file_type == LogPathFileType::StagedCommit) + } + + pub(crate) fn ascending_commit_files_mut(&mut self) -> &mut Vec { + &mut self.ascending_commit_files + } + + pub(crate) fn checkpoint_parts(&self) -> &Vec { + &self.checkpoint_parts + } + + pub(crate) fn latest_commit_file(&self) -> &Option { + &self.latest_commit_file + } + + /// Iterator over every listed log path across all fields. + pub(crate) fn iter_all_paths(&self) -> impl Iterator { + self.ascending_commit_files + .iter() + .chain(&self.ascending_compaction_files) + .chain(&self.checkpoint_parts) + .chain(&self.latest_crc_file) + .chain(&self.latest_commit_file) + } + + /// Estimated heap size in bytes, best-effort estimate. + pub(crate) fn estimated_heap_size_bytes(&self) -> usize { + let vec_buffer_bytes = (self.ascending_commit_files.capacity() + + self.ascending_compaction_files.capacity() + + self.checkpoint_parts.capacity()) + * size_of::(); + let path_bytes: usize = self + .iter_all_paths() + .map(ParsedLogPath::estimated_heap_size_bytes) + .sum(); + vec_buffer_bytes + path_bytes + } + + /// List all commits between the provided `start_version` (inclusive) and `end_version` + /// (inclusive). All other types are ignored. + /// + /// `log_tail` is a contiguous run of commits ending at the table's latest version. It takes + /// precedence over the filesystem listing, and is required for catalog-managed tables, whose + /// unbackfilled staged commits exist only here. + pub(crate) fn list_commits( + storage: &dyn StorageHandler, + log_root: &Url, + log_tail: Vec, + start_version: Option, + end_version: Option, + ) -> DeltaResult { + debug_assert!( + log_tail.iter().all(|entry| entry.is_commit()), + "log_tail should only contain commits" + ); + let start = start_version.unwrap_or(0); + let end = end_version.unwrap_or(Version::MAX); + let fs_iter = list_delta_log_from_storage(storage, log_root, start, end)?; + + let log_tail_start_version = log_tail.first().map(|f| f.version); + let mut listed_commits = Vec::new(); + let mut max_published_version: Option = None; + // Filesystem commits, skipping any covered by the log_tail. + for file_result in fs_iter { + let file = file_result?; + if file.file_type != LogPathFileType::Commit { + continue; + } + max_published_version = max_published_version.max(Some(file.version)); + if log_tail_start_version.is_some_and(|tail_start| file.version >= tail_start) { + continue; + } + should_process_log_file(&file); // warns if 0 bytes + listed_commits.push(file); + } + + // Log_tail commits, extending the filesystem prefix in ascending order. + for file in log_tail { + if file.version < start || file.version > end { + continue; + } + if file.file_type == LogPathFileType::Commit { + max_published_version = max_published_version.max(Some(file.version)); + } + listed_commits.push(file); + } + + let latest_commit_file = listed_commits.last().cloned(); + Ok(LogSegmentFiles { + ascending_commit_files: listed_commits, + latest_commit_file, + max_published_version, + ..Default::default() + }) + } + + /// List all commit and checkpoint files with versions above the provided `start_version` + /// (inclusive). If successful, this returns a `LogSegmentFiles`. + /// + /// The `log_tail` is an optional sequence of commits provided by the caller, e.g. via + /// [`SnapshotBuilder::with_log_tail`]. It may contain either published or staged commits. The + /// `log_tail` must strictly adhere to being a 'tail' — a contiguous cover of versions `X..=Y` + /// where `Y` is the latest version of the table. If it overlaps with commits listed from the + /// filesystem, the `log_tail` will take precedence for commits; non-commit files (CRC, + /// checkpoints, compactions) are always taken from the filesystem. + // TODO: encode some of these guarantees in the output types. e.g. we could have: + // - SortedCommitFiles: Vec, is_ascending: bool, end_version: Version + // - CheckpointParts: Vec, checkpoint_version: Version (guarantee all same + // version) + #[instrument(name = "log.list", skip_all, fields(start = ?start_version, end = ?end_version), err)] + pub(crate) fn list( + storage: &dyn StorageHandler, + log_root: &Url, + log_tail: Vec, + start_version: Option, + end_version: Option, + ) -> DeltaResult { + let start = start_version.unwrap_or(0); + let end = end_version.unwrap_or(Version::MAX); + let fs_iter = list_delta_log_from_storage(storage, log_root, start, end)?; + Self::build_log_segment_files(fs_iter, log_tail, start, end_version) + } + + /// List all commit and checkpoint files after the provided checkpoint. It is guaranteed that + /// all the returned [`ParsedLogPath`]s will have a version less than or equal to the + /// `end_version`. + /// + /// The hint only tells us where to start listing; it never influences which checkpoint is + /// selected at a version. A hint that turns out to describe a different checkpoint than the one + /// selected is logged and ignored, not an error. + pub(crate) fn list_with_checkpoint_hint( + checkpoint_metadata: &LastCheckpointHint, + storage: &dyn StorageHandler, + log_root: &Url, + log_tail: Vec, + end_version: Option, + ) -> DeltaResult { + let listed_files = Self::list( + storage, + log_root, + log_tail, + Some(checkpoint_metadata.version), + end_version, + )?; + + let Some(latest_checkpoint) = listed_files.checkpoint_parts.last() else { + // The hint names a checkpoint that no longer exists, and because the listing started + // at the hinted version, no checkpoint exists at or after it either. The log was + // modified out of band: the checkpoint was deleted without clearing the hint, or the + // table was dropped and recreated at the same path. + // + // Kernel fails rather than retrying the listing from version 0, because that recovery + // is unsound. Listing only checks that the commits it finds are contiguous, not that + // they start at version 0, so two distinct kinds of damage survive as a plausible but + // wrong snapshot instead of an error: + // - A deleted log prefix leaves a contiguous suffix with no checkpoint. If commits + // 0-40 and the checkpoint are removed but 41.. remain, listing from version 0 + // replays 41 as if it were the start of history, dropping every action before it. + // - A recreated table yields a segment mixing files from two different tables. A + // table with a checkpoint at commit 3 is dropped and recreated at the same path; + // the drop leaves commits 0-2 behind, the new table writes its own 0, 1, 2, ..., + // and listing from version 0 sees one contiguous sequence whose low versions belong + // to two different tables and replays it as a single history. + return Err(Error::invalid_checkpoint( + "Had a _last_checkpoint hint but didn't find any checkpoints", + )); + }; + if latest_checkpoint.version != checkpoint_metadata.version { + info!( + "_last_checkpoint hint is out of date. _last_checkpoint version: {}. Using actual most recent: {}", + checkpoint_metadata.version, + latest_checkpoint.version + ); + } else if !checkpoint_metadata.applies_to(&listed_files.checkpoint_parts) { + // Expected whenever a writer checkpoints a version another writer already checkpointed + // and leaves the hint alone. `applies_to` also makes `LogSegment::checkpoint_hint` + // yield `None`, so this logs exactly when the hint's fields get dropped. + debug!( + version = checkpoint_metadata.version, + hint_parts = checkpoint_metadata.parts.unwrap_or(1), + selected_parts = listed_files.checkpoint_parts.len(), + selected_checkpoint_part = %latest_checkpoint.filename, + "_last_checkpoint hint describes a different checkpoint than the one selected at \ + this version; using the checkpoint file's own fields" + ); + } + Ok(listed_files) + } + + /// Returns a [`LogSegmentFiles`] ending at `end_version`, rooted at the most recent complete + /// checkpoint at or before `end_version`, or rooted at version 0 if no checkpoint is found. + /// + /// To find the checkpoint without a full forward listing from version 0, this scans backward + /// from `end_version` in windows of size [`BACKWARD_SCAN_WINDOW_SIZE`], stopping as soon as + /// a complete checkpoint is found (or version 0 is reached). + /// Then, all files from the windows that were scanned are combined with `log_tail` to produce a + /// log segment rooted at the checkpoint version (or version 0 if no checkpoint) with all + /// commits after the checkpoint version. A log_tail commit at exactly the checkpoint + /// version may be included at this stage but will be filtered out by `LogSegment::try_new`. + /// + /// For example, given the desired end_version = 12500 and a checkpoint at v8900: + /// - Window 1 [11501, 12501): no checkpoint -> continue + /// - Window 2 [10501, 11501): no checkpoint -> continue + /// - Window 3 [9501, 10501): no checkpoint -> continue + /// - Window 4 [8501, 9501): checkpoint at v8900 found -> stop + /// All files from windows 1-4 are combined with `log_tail` to produce a log segment + /// rooted at the checkpoint at v8900 with all commits from v8901 to v12500. + #[instrument(name = "log.list_with_backward_checkpoint_scan", skip_all, fields(end = end_version), err)] + pub(crate) fn list_with_backward_checkpoint_scan( + storage: &dyn StorageHandler, + log_root: &Url, + log_tail: Vec, + end_version: Version, + ) -> DeltaResult { + // Scan backward in 1000-version windows, collecting ALL file types, until a complete + // checkpoint is found or the log is exhausted. + let mut windows: Vec> = Vec::new(); + let mut found_checkpoint_version: Option = None; + // upper is the exclusive upper bound of the next window; adding 1 includes end_version + // in the first window. The inclusive range passed to list_delta_log_from_storage is + // [lower, upper - 1]. + let mut upper = end_version + 1; + while upper > 0 { + let lower = upper.saturating_sub(BACKWARD_SCAN_WINDOW_SIZE); + let window_files: Vec<_> = + list_delta_log_from_storage(storage, log_root, lower, upper - 1)?.try_collect()?; + + found_checkpoint_version = find_complete_checkpoint_version(&window_files); + windows.push(window_files); + + if found_checkpoint_version.is_some() { + break; + } + upper = lower; + } + + let fs_iter = windows.into_iter().rev().flatten().map(Ok); + let start = found_checkpoint_version.unwrap_or(0); + Self::build_log_segment_files(fs_iter, log_tail, start, Some(end_version)) + } +} diff --git a/kernel/src/log_segment_files/tests.rs b/kernel/src/log_segment_files/tests.rs new file mode 100644 index 0000000000..62152f4d67 --- /dev/null +++ b/kernel/src/log_segment_files/tests.rs @@ -0,0 +1,1731 @@ +use std::sync::atomic::{AtomicU32, Ordering}; +use std::sync::Arc; + +use rstest::rstest; +use url::Url; + +use super::*; +use crate::engine::sync::SyncEngine; +use crate::object_store::memory::InMemory; +use crate::object_store::path::Path as ObjectPath; +use crate::object_store::ObjectStoreExt as _; +use crate::path::tests::multipart_checkpoint_name; +use crate::{Engine as _, FileMeta}; + +// size markers used to identify commit sources in tests +const FILESYSTEM_SIZE_MARKER: u64 = 10; +const CATALOG_SIZE_MARKER: u64 = 7; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum CommitSource { + Filesystem, + Catalog, +} + +fn log_path_for_file_type(version: Version, file_type: &LogPathFileType) -> String { + match file_type { + LogPathFileType::Commit => { + format!("_delta_log/{version:020}.json") + } + LogPathFileType::StagedCommit => { + let uuid = uuid::Uuid::new_v4(); + format!("_delta_log/_staged_commits/{version:020}.{uuid}.json") + } + LogPathFileType::ClassicCheckpoint => { + format!("_delta_log/{version:020}.checkpoint.parquet") + } + LogPathFileType::MultiPartCheckpoint { + part_num, + num_parts, + } => { + let name = multipart_checkpoint_name(version, *part_num, *num_parts); + format!("_delta_log/{name}") + } + LogPathFileType::Crc => { + format!("_delta_log/{version:020}.crc") + } + LogPathFileType::CompactedCommit { hi } => { + format!("_delta_log/{version:020}.{hi:020}.compacted.json") + } + LogPathFileType::UuidCheckpoint | LogPathFileType::Unknown => { + panic!("Unsupported file type in test: {file_type:?}") + } + } +} + +async fn create_storage( + log_files: Vec<(Version, LogPathFileType, CommitSource)>, +) -> (Arc, Url) { + create_storage_with_raw_paths(log_files, &[]).await +} + +/// Like [`create_storage`] but also writes `raw_paths` verbatim, for log entries that +/// [`log_path_for_file_type`] cannot produce (e.g. `_last_checkpoint`). +async fn create_storage_with_raw_paths( + log_files: Vec<(Version, LogPathFileType, CommitSource)>, + raw_paths: &[&str], +) -> (Arc, Url) { + let store = Arc::new(InMemory::new()); + let log_root = Url::parse("memory:///_delta_log/").unwrap(); + + for path in raw_paths { + store + .put(&ObjectPath::from(*path), bytes::Bytes::from("raw").into()) + .await + .expect("Failed to put raw test file"); + } + + for (version, file_type, source) in log_files { + let path = log_path_for_file_type(version, &file_type); + let data = match source { + CommitSource::Filesystem => bytes::Bytes::from("filesystem"), + CommitSource::Catalog => bytes::Bytes::from("catalog"), + }; + store + .put(&ObjectPath::from(path.as_str()), data.into()) + .await + .expect("Failed to put test file"); + } + + let engine = SyncEngine::new_with_store(store); + (engine.storage_handler(), log_root) +} + +// helper to create a ParsedLogPath with specific source marker +fn make_parsed_log_path_with_source( + version: Version, + file_type: LogPathFileType, + source: CommitSource, +) -> ParsedLogPath { + let url = Url::parse(&format!("memory:///_delta_log/{version:020}.json")).unwrap(); + let mut filename_path_segments = url.path_segments().unwrap(); + let filename = filename_path_segments.next_back().unwrap().to_string(); + let extension = filename.split('.').next_back().unwrap().to_string(); + + let size = match source { + CommitSource::Filesystem => FILESYSTEM_SIZE_MARKER, + CommitSource::Catalog => CATALOG_SIZE_MARKER, + }; + + let location = FileMeta { + location: url, + last_modified: 0, + size, + }; + + ParsedLogPath { + location, + filename, + extension, + version, + file_type, + } +} + +fn assert_source(commit: &ParsedLogPath, expected_source: CommitSource) { + let expected_size = match expected_source { + CommitSource::Filesystem => FILESYSTEM_SIZE_MARKER, + CommitSource::Catalog => CATALOG_SIZE_MARKER, + }; + assert_eq!( + commit.location.size, expected_size, + "Commit version {} should be from {:?}, but size was {}", + commit.version, expected_source, commit.location.size + ); +} + +/// A [`StorageHandler`] wrapper that counts the number of `list_from` calls and the number of +/// items consumed from the returned iterators. Used to verify that +/// `list_with_backward_checkpoint_scan` issues the expected number of storage listing requests, +/// and that listing terminates without consuming files past the version-named region. +struct CountingStorageHandler { + inner: Arc, + list_from_count: AtomicU32, + items_listed: Arc, +} + +impl CountingStorageHandler { + fn new(inner: Arc) -> Self { + Self { + inner, + list_from_count: AtomicU32::new(0), + items_listed: Arc::new(AtomicU32::new(0)), + } + } + + fn call_count(&self) -> u32 { + self.list_from_count.load(Ordering::Relaxed) + } + + fn items_listed(&self) -> u32 { + self.items_listed.load(Ordering::Relaxed) + } +} + +impl StorageHandler for CountingStorageHandler { + fn list_from( + &self, + path: &Url, + ) -> DeltaResult>>> { + self.list_from_count.fetch_add(1, Ordering::Relaxed); + let items_listed = self.items_listed.clone(); + let iter = self.inner.list_from(path)?; + Ok(Box::new(iter.inspect(move |_| { + items_listed.fetch_add(1, Ordering::Relaxed); + }))) + } + + fn read_files( + &self, + _files: Vec, + ) -> DeltaResult>>> { + panic!("read_files should not be called during listing"); + } + + fn put(&self, _path: &Url, _data: bytes::Bytes, _overwrite: bool) -> DeltaResult<()> { + panic!("put should not be called during listing"); + } + + fn copy_atomic(&self, _src: &Url, _dest: &Url) -> DeltaResult<()> { + panic!("copy_atomic should not be called during listing"); + } + + fn head(&self, _path: &Url) -> DeltaResult { + panic!("head should not be called during listing"); + } + + fn delete(&self, _path: &Url) -> DeltaResult<()> { + panic!("delete should not be called during listing"); + } +} + +/// Helper to call `LogSegmentFiles::list()` and destructure the result for assertions. +/// Returns (ascending_commit_files, ascending_compaction_files, checkpoint_parts, +/// latest_crc_file, latest_commit_file, max_published_version). +#[allow(clippy::type_complexity)] +fn list_and_destructure( + storage: &dyn StorageHandler, + log_root: &Url, + log_tail: Vec, + start_version: Option, + end_version: Option, +) -> ( + Vec, + Vec, + Vec, + Option, + Option, + Option, +) { + let r = LogSegmentFiles::list(storage, log_root, log_tail, start_version, end_version).unwrap(); + ( + r.ascending_commit_files, + r.ascending_compaction_files, + r.checkpoint_parts, + r.latest_crc_file, + r.latest_commit_file, + r.max_published_version, + ) +} + +// ===== list() tests ===== + +#[tokio::test] +async fn test_empty_log_tail() { + let log_files = vec![ + (0, LogPathFileType::Commit, CommitSource::Filesystem), + (1, LogPathFileType::Commit, CommitSource::Filesystem), + (2, LogPathFileType::Commit, CommitSource::Filesystem), + ]; + let (storage, log_root) = create_storage(log_files).await; + + let (commits, _, _, _, latest_commit, max_pub) = + list_and_destructure(storage.as_ref(), &log_root, vec![], Some(1), Some(2)); + + assert_eq!(commits.len(), 2); + assert_eq!(commits[0].version, 1); + assert_eq!(commits[1].version, 2); + assert_source(&commits[0], CommitSource::Filesystem); + assert_source(&commits[1], CommitSource::Filesystem); + assert_eq!(latest_commit.unwrap().version, 2); + assert_eq!(max_pub, Some(2)); +} + +#[tokio::test] +async fn test_log_tail_has_latest_commit_files() { + // Filesystem has commits 0-2, log_tail has commits 3-5 (the latest) + let log_files = vec![ + (0, LogPathFileType::Commit, CommitSource::Filesystem), + (1, LogPathFileType::Commit, CommitSource::Filesystem), + (2, LogPathFileType::Commit, CommitSource::Filesystem), + ]; + let (storage, log_root) = create_storage(log_files).await; + + let log_tail = vec![ + make_parsed_log_path_with_source(3, LogPathFileType::Commit, CommitSource::Catalog), + make_parsed_log_path_with_source(4, LogPathFileType::Commit, CommitSource::Catalog), + make_parsed_log_path_with_source(5, LogPathFileType::Commit, CommitSource::Catalog), + ]; + + let (commits, _, _, _, latest_commit, max_pub) = + list_and_destructure(storage.as_ref(), &log_root, log_tail, Some(0), Some(5)); + + assert_eq!(commits.len(), 6); + // filesystem commits 0-2 + for (i, commit) in commits.iter().enumerate().take(3) { + assert_eq!(commit.version, i as u64); + assert_source(commit, CommitSource::Filesystem); + } + // catalog commits 3-5 + for (i, commit) in commits.iter().enumerate().skip(3) { + assert_eq!(commit.version, i as u64); + assert_source(commit, CommitSource::Catalog); + } + assert_eq!(latest_commit.unwrap().version, 5); + assert_eq!(max_pub, Some(5)); +} + +#[tokio::test] +async fn test_request_subset_with_log_tail() { + // Test requesting a subset when log_tail is the latest commits + let log_files = vec![ + (0, LogPathFileType::Commit, CommitSource::Filesystem), + (1, LogPathFileType::Commit, CommitSource::Filesystem), + ]; + let (storage, log_root) = create_storage(log_files).await; + + // log_tail represents versions 2-4 (latest commits) + let log_tail = vec![ + make_parsed_log_path_with_source(2, LogPathFileType::Commit, CommitSource::Catalog), + make_parsed_log_path_with_source(3, LogPathFileType::Commit, CommitSource::Catalog), + make_parsed_log_path_with_source(4, LogPathFileType::Commit, CommitSource::Catalog), + ]; + + // list for only versions 1-3 + let (commits, _, _, _, latest_commit, max_pub) = + list_and_destructure(storage.as_ref(), &log_root, log_tail, Some(1), Some(3)); + + assert_eq!(commits.len(), 3); + assert_eq!(commits[0].version, 1); + assert_eq!(commits[1].version, 2); + assert_eq!(commits[2].version, 3); + assert_source(&commits[0], CommitSource::Filesystem); + assert_source(&commits[1], CommitSource::Catalog); + assert_source(&commits[2], CommitSource::Catalog); + assert_eq!(latest_commit.unwrap().version, 3); + assert_eq!(max_pub, Some(3)); +} + +#[tokio::test] +async fn test_log_tail_defines_latest_version() { + // log_tail defines the latest version of the table: if there is file system files after log + // tail, they are ignored. But we still list all filesystem files to track + // max_published_version. + let log_files = vec![ + (0, LogPathFileType::Commit, CommitSource::Filesystem), + (1, LogPathFileType::Commit, CommitSource::Filesystem), + (2, LogPathFileType::Commit, CommitSource::Filesystem), // <-- max_published_version + ]; + let (storage, log_root) = create_storage(log_files).await; + + // log_tail is just [1], indicating version 1 is the latest + let log_tail = vec![make_parsed_log_path_with_source( + 1, + LogPathFileType::Commit, + CommitSource::Catalog, + )]; + + let (commits, _, _, _, latest_commit, max_pub) = + list_and_destructure(storage.as_ref(), &log_root, log_tail, Some(0), None); + + // expect only 0 from file system and 1 from log tail + assert_eq!(commits.len(), 2); + assert_eq!(commits[0].version, 0); + assert_eq!(commits[1].version, 1); + assert_source(&commits[0], CommitSource::Filesystem); + assert_source(&commits[1], CommitSource::Catalog); + assert_eq!(latest_commit.unwrap().version, 1); + // max_published_version should reflect the highest published commit on filesystem + assert_eq!(max_pub, Some(2)); +} + +#[test] +fn test_log_tail_covers_entire_range_empty_filesystem() { + // Test-only storage handler that returns an empty listing. + // When the log_tail covers the entire commit range, we still call list_from + // (to pick up non-commit files like CRC/checkpoints), but the filesystem may + // have nothing — e.g. a purely catalog-managed table. + struct EmptyStorageHandler; + impl StorageHandler for EmptyStorageHandler { + fn list_from( + &self, + _path: &Url, + ) -> DeltaResult>>> { + Ok(Box::new(std::iter::empty())) + } + fn read_files( + &self, + _files: Vec, + ) -> DeltaResult>>> { + panic!("read_files should not be called during listing"); + } + fn put(&self, _path: &Url, _data: bytes::Bytes, _overwrite: bool) -> DeltaResult<()> { + panic!("put should not be called during listing"); + } + fn copy_atomic(&self, _src: &Url, _dest: &Url) -> DeltaResult<()> { + panic!("copy_atomic should not be called during listing"); + } + fn head(&self, _path: &Url) -> DeltaResult { + panic!("head should not be called during listing"); + } + fn delete(&self, _path: &Url) -> DeltaResult<()> { + panic!("delete should not be called during listing"); + } + } + + // log_tail covers versions 0-2, the entire range + let log_tail = vec![ + make_parsed_log_path_with_source(0, LogPathFileType::Commit, CommitSource::Catalog), + make_parsed_log_path_with_source(1, LogPathFileType::Commit, CommitSource::Catalog), + make_parsed_log_path_with_source(2, LogPathFileType::StagedCommit, CommitSource::Catalog), + ]; + + let storage = EmptyStorageHandler; + let url = Url::parse("memory:///anything/_delta_log/").unwrap(); + let (commits, _, _, _, latest_commit, max_pub) = + list_and_destructure(&storage, &url, log_tail, Some(0), Some(2)); + + // Only log_tail commits should appear (filesystem is empty) + assert_eq!(commits.len(), 3); + assert_eq!(commits[0].version, 0); + assert_eq!(commits[1].version, 1); + assert_eq!(commits[2].version, 2); + assert_source(&commits[0], CommitSource::Catalog); + assert_source(&commits[1], CommitSource::Catalog); + assert_source(&commits[2], CommitSource::Catalog); + assert_eq!(latest_commit.unwrap().version, 2); + // Only published (non-staged) commits from log_tail count for max_published_version + assert_eq!(max_pub, Some(1)); +} + +#[tokio::test] +async fn test_log_tail_covers_entire_range_with_crc() { + // When log_tail covers the entire requested range (starts at version 0), commit files + // from the filesystem should be excluded (log_tail is authoritative for commits), but + // non-commit files (CRC, checkpoints) should still be picked up from the filesystem. + let log_files = vec![ + (0, LogPathFileType::Commit, CommitSource::Filesystem), + (1, LogPathFileType::Commit, CommitSource::Filesystem), + (2, LogPathFileType::Crc, CommitSource::Filesystem), + ]; + let (storage, log_root) = create_storage(log_files).await; + + // log_tail covers versions 0-2, which includes the entire range we'll request + let log_tail = vec![ + make_parsed_log_path_with_source(0, LogPathFileType::Commit, CommitSource::Catalog), + make_parsed_log_path_with_source(1, LogPathFileType::Commit, CommitSource::Catalog), + make_parsed_log_path_with_source(2, LogPathFileType::StagedCommit, CommitSource::Catalog), + ]; + + let (commits, _, _, latest_crc, latest_commit, max_pub) = + list_and_destructure(storage.as_ref(), &log_root, log_tail, Some(0), Some(2)); + + // 3 commits from log_tail: 0, 1, 2 + assert_eq!(commits.len(), 3); + assert_source(&commits[0], CommitSource::Catalog); + assert_source(&commits[1], CommitSource::Catalog); + assert_source(&commits[2], CommitSource::Catalog); + + // CRC at version 2 from filesystem is preserved + let crc = latest_crc.unwrap(); + assert_eq!(crc.version, 2); + assert!(matches!(crc.file_type, LogPathFileType::Crc)); + + assert_eq!(latest_commit.unwrap().version, 2); + // Only published commits count: filesystem 0,1 (skipped but tracked) + log_tail 0,1 + assert_eq!(max_pub, Some(1)); +} + +#[tokio::test] +async fn test_listing_omits_staged_commits() { + // note that in the presence of staged commits, we CANNOT trust listing to determine which + // to include in our listing/log segment. This is up to the catalog. (e.g. version + // 5.uuid1.json and 5.uuid2.json can both exist and only catalog can say which is the 'real' + // version 5). + + let log_files = vec![ + (0, LogPathFileType::Commit, CommitSource::Filesystem), + (1, LogPathFileType::Commit, CommitSource::Filesystem), // <-- max_published_version + (1, LogPathFileType::StagedCommit, CommitSource::Filesystem), + (2, LogPathFileType::StagedCommit, CommitSource::Filesystem), + ]; + + let (storage, log_root) = create_storage(log_files).await; + let (commits, _, _, _, latest_commit, max_pub) = + list_and_destructure(storage.as_ref(), &log_root, vec![], None, None); + + // we must only see two regular commits + assert_eq!(commits.len(), 2); + assert_eq!(commits[0].version, 0); + assert_eq!(commits[1].version, 1); + assert_source(&commits[0], CommitSource::Filesystem); + assert_source(&commits[1], CommitSource::Filesystem); + assert_eq!(latest_commit.unwrap().version, 1); + assert_eq!(max_pub, Some(1)); +} + +#[tokio::test] +async fn test_listing_stops_at_first_staged_commit_without_consuming_the_rest() { + let mut log_files = vec![ + (0, LogPathFileType::Commit, CommitSource::Filesystem), + (1, LogPathFileType::Commit, CommitSource::Filesystem), + (2, LogPathFileType::Commit, CommitSource::Filesystem), + ]; + // Staged commits sort after every version-named file ('_' > '9'), so a sorted listing + // reaches them only after all relevant files. None should be consumed beyond the first. + log_files + .extend((0..100).map(|v| (v, LogPathFileType::StagedCommit, CommitSource::Filesystem))); + + let (storage, log_root) = create_storage(log_files).await; + let storage = CountingStorageHandler::new(storage); + + let (commits, _, _, _, latest_commit, max_pub) = + list_and_destructure(&storage, &log_root, vec![], None, None); + + assert_eq!(commits.len(), 3); + assert_eq!(latest_commit.unwrap().version, 2); + assert_eq!(max_pub, Some(2)); + // 3 commits plus the single staged commit that stops the listing + assert_eq!(storage.items_listed(), 4); +} + +// Any path past the version-named region stops the listing, not just `_staged_commits/`: +// checkpoint sidecars under `_sidecars/` and non-underscore names whose first byte sorts +// past '9' (e.g. 'Z'). Both sentinels sort before `_staged_commits/`, so no staged commit +// is ever consumed. +#[rstest] +#[case::sidecar("_delta_log/_sidecars/016ae953-37a9-438e-8683-9a9a4a79a395.parquet")] +#[case::non_underscore_sentinel("_delta_log/Zsentinel")] +#[tokio::test] +async fn test_listing_stops_at_first_non_version_named_path(#[case] sentinel_path: &str) { + let mut log_files = vec![ + (0, LogPathFileType::Commit, CommitSource::Filesystem), + (1, LogPathFileType::Commit, CommitSource::Filesystem), + (2, LogPathFileType::Commit, CommitSource::Filesystem), + ]; + log_files.extend((0..50).map(|v| (v, LogPathFileType::StagedCommit, CommitSource::Filesystem))); + + let (storage, log_root) = create_storage_with_raw_paths(log_files, &[sentinel_path]).await; + let storage = CountingStorageHandler::new(storage); + + let (commits, _, _, _, latest_commit, max_pub) = + list_and_destructure(&storage, &log_root, vec![], None, None); + + assert_eq!(commits.len(), 3); + assert_eq!(latest_commit.unwrap().version, 2); + assert_eq!(max_pub, Some(2)); + // 3 commits plus the sentinel that stops the listing + assert_eq!(storage.items_listed(), 4); +} + +#[tokio::test] +async fn test_listing_stops_at_last_checkpoint_marker() { + // In a real table `_last_checkpoint` sorts before `_staged_commits/` ('_la' < '_st'), so + // it is the path that stops the listing. + let mut log_files = vec![ + (0, LogPathFileType::Commit, CommitSource::Filesystem), + (1, LogPathFileType::Commit, CommitSource::Filesystem), + (2, LogPathFileType::Commit, CommitSource::Filesystem), + ( + 2, + LogPathFileType::ClassicCheckpoint, + CommitSource::Filesystem, + ), + (2, LogPathFileType::Crc, CommitSource::Filesystem), + ]; + log_files.extend((0..50).map(|v| (v, LogPathFileType::StagedCommit, CommitSource::Filesystem))); + + let (storage, log_root) = + create_storage_with_raw_paths(log_files, &["_delta_log/_last_checkpoint"]).await; + let storage = CountingStorageHandler::new(storage); + + let (commits, _, checkpoint_parts, latest_crc, latest_commit, max_pub) = + list_and_destructure(&storage, &log_root, vec![], None, None); + + // The checkpoint at version 2 subsumes all commits; only latest_commit_file is retained + assert!(commits.is_empty()); + assert_eq!(checkpoint_parts.len(), 1); + assert_eq!(checkpoint_parts[0].version, 2); + assert_eq!(latest_crc.unwrap().version, 2); + assert_eq!(latest_commit.unwrap().version, 2); + assert_eq!(max_pub, Some(2)); + // 5 version-named files plus the `_last_checkpoint` that stops the listing; no staged + // commit is ever consumed + assert_eq!(storage.items_listed(), 6); +} + +#[tokio::test] +async fn test_listing_with_large_end_version() { + let log_files = vec![ + (0, LogPathFileType::Commit, CommitSource::Filesystem), + (1, LogPathFileType::Commit, CommitSource::Filesystem), // <-- max_published_version + (2, LogPathFileType::StagedCommit, CommitSource::Filesystem), + ]; + + let (storage, log_root) = create_storage(log_files).await; + // note we let you request end version past the end of log. up to consumer to interpret + let (commits, _, _, _, latest_commit, max_pub) = + list_and_destructure(storage.as_ref(), &log_root, vec![], None, Some(3)); + + // we must only see two regular commits + assert_eq!(commits.len(), 2); + assert_eq!(commits[0].version, 0); + assert_eq!(commits[1].version, 1); + assert_eq!(latest_commit.unwrap().version, 1); + assert_eq!(max_pub, Some(1)); +} + +#[tokio::test] +async fn test_non_commit_files_at_log_tail_versions_are_preserved() { + // Filesystem has commits 0-5, a checkpoint at version 7, and a CRC at version 8. + // Log tail provides commits 6-10. The checkpoint and CRC are on the filesystem + // at versions covered by the log_tail and must NOT be filtered out. + // + // After processing through ListingAccumulator, the checkpoint at version 7 + // causes commits before it to be cleared, keeping only commits after the checkpoint. + let log_files = vec![ + (0, LogPathFileType::Commit, CommitSource::Filesystem), + (1, LogPathFileType::Commit, CommitSource::Filesystem), + (2, LogPathFileType::Commit, CommitSource::Filesystem), + (3, LogPathFileType::Commit, CommitSource::Filesystem), + (4, LogPathFileType::Commit, CommitSource::Filesystem), + (5, LogPathFileType::Commit, CommitSource::Filesystem), + ( + 7, + LogPathFileType::ClassicCheckpoint, + CommitSource::Filesystem, + ), + (8, LogPathFileType::Crc, CommitSource::Filesystem), + ]; + let (storage, log_root) = create_storage(log_files).await; + + let log_tail = vec![ + make_parsed_log_path_with_source(6, LogPathFileType::Commit, CommitSource::Catalog), + make_parsed_log_path_with_source(7, LogPathFileType::Commit, CommitSource::Catalog), + make_parsed_log_path_with_source(8, LogPathFileType::Commit, CommitSource::Catalog), + make_parsed_log_path_with_source(9, LogPathFileType::Commit, CommitSource::Catalog), + make_parsed_log_path_with_source(10, LogPathFileType::Commit, CommitSource::Catalog), + ]; + + let (commits, _, checkpoint_parts, latest_crc, latest_commit, max_pub) = + list_and_destructure(storage.as_ref(), &log_root, log_tail, Some(0), Some(10)); + + // Checkpoint at version 7 is preserved from filesystem + assert_eq!(checkpoint_parts.len(), 1); + assert_eq!(checkpoint_parts[0].version, 7); + assert!(checkpoint_parts[0].is_checkpoint()); + + // CRC at version 8 is preserved from filesystem + let crc = latest_crc.unwrap(); + assert_eq!(crc.version, 8); + assert!(matches!(crc.file_type, LogPathFileType::Crc)); + + // After checkpoint processing: commits before checkpoint are cleared, + // only log_tail commits 6-10 remain (added after checkpoint flush) + assert_eq!(commits.len(), 5); + for (i, commit) in commits.iter().enumerate() { + assert_eq!(commit.version, (i + 6) as u64); + assert_source(commit, CommitSource::Catalog); + } + assert_eq!(latest_commit.unwrap().version, 10); + + // max_published_version reflects all published commits seen (filesystem 0-5 + log_tail 6-10) + assert_eq!(max_pub, Some(10)); +} + +// ===== list_with_backward_checkpoint_scan() tests ===== + +// Log from v0 to v1005. Each case places an optional single-part checkpoint and +// verifies the expected commits, checkpoint version, and number of storage listings. +// +// Window boundaries (window size=1000, end_version=1005, exclusive upper): +// Window 1: [6, 1006) covers v6..=v1005 +// Window 2: [0, 6) covers v0..=v5 +// +// A checkpoint at v6+ is found in window 1 (1 listing); at v5 or lower in window 2 +// (2 listings). A checkpoint beyond end_version is never seen. +#[rstest] +// No checkpoint: scan exhausts both windows, all 1006 commits returned +#[case::no_checkpoint(None, 0..=1005, None, 2)] +// Checkpoint beyond end_version is never seen; same behavior as no checkpoint +#[case::checkpoint_beyond_end(Some(1006), 0..=1005, None, 2)] +// Checkpoint at end_version: found in window 1, no commits after it +#[case::checkpoint_at_end(Some(1005), 0..0, Some(1005), 1)] +// Checkpoint at v5: falls in window 2 -> 2 listings; commits 6..=1005 returned. +// Tests the inclusive window boundary: window 1 covers [6, 1006) or [6, 1005] (lower = 1006 - 1000 +// = 6), so v5 falls just outside it and requires a second listing, while v6 (next case) does not. +#[case::checkpoint_in_second_window(Some(5), 6..=1005, Some(5), 2)] +// Checkpoint at v6: falls in window 1 -> 1 listing; commits 7..=1005 returned +#[case::checkpoint_in_first_window(Some(6), 7..=1005, Some(6), 1)] +#[tokio::test] +async fn backward_scan_single_checkpoint_cases( + #[case] checkpoint_version: Option, + #[case] expected_commits: impl Iterator, + #[case] expected_checkpoint: Option, + #[case] expected_listings: u32, +) { + let mut log_files: Vec<(Version, LogPathFileType, CommitSource)> = (0u64..=1005) + .map(|v| (v, LogPathFileType::Commit, CommitSource::Filesystem)) + .collect(); + + if let Some(cp) = checkpoint_version { + log_files.push(( + cp, + LogPathFileType::ClassicCheckpoint, + CommitSource::Filesystem, + )); + } + + let (storage, log_root) = create_storage(log_files).await; + let counter = CountingStorageHandler::new(storage); + + let result = + LogSegmentFiles::list_with_backward_checkpoint_scan(&counter, &log_root, vec![], 1005) + .unwrap(); + + assert_eq!(counter.call_count(), expected_listings); + + assert_eq!( + result.checkpoint_parts.len(), + if expected_checkpoint.is_some() { 1 } else { 0 } + ); + if let Some(cp_version) = expected_checkpoint { + assert_eq!(result.checkpoint_parts[0].version, cp_version); + } + + assert!(result + .ascending_commit_files + .iter() + .map(|f| f.version) + .eq(expected_commits)); +} + +/// end_version=3000. Window 2 contains an incomplete 2-of-2 multipart checkpoint (only +/// part 1 present). find_complete_checkpoint_version must return None for window 2, causing +/// the scan to continue to window 3, where a complete single-part checkpoint at v500 is +/// found. Verifies that incomplete parts from window 2 are discarded and do not pollute +/// the result's checkpoint_parts. +/// +/// Window 1 [2001, 3001): commits v2001..=v3000, no checkpoint -> continue +/// Window 2 [1001, 2001): commits v1001..=v2000, v1500 (1-of-2 parts) incomplete -> continue +/// Window 3 [1, 1001): commits v1..=v1000, v500 (complete) -> checkpoint found -> break +fn files_incomplete_in_second_window_complete_in_third_window( +) -> Vec<(Version, LogPathFileType, CommitSource)> { + let mut log_files: Vec<(Version, LogPathFileType, CommitSource)> = (0u64..=3000) + .map(|v| (v, LogPathFileType::Commit, CommitSource::Filesystem)) + .collect(); + log_files.push(( + 500, + LogPathFileType::ClassicCheckpoint, + CommitSource::Filesystem, + )); + log_files.push(( + 1500, + LogPathFileType::MultiPartCheckpoint { + part_num: 1, + num_parts: 2, + }, + CommitSource::Filesystem, + )); + log_files +} +fn multipart_checkpoint_files() -> Vec<(Version, LogPathFileType, CommitSource)> { + // Log v0..=v52 with a complete 3-part checkpoint at v50. + // Single window [0, 53): checkpoint found -> stop. + let mut log_files: Vec<(Version, LogPathFileType, CommitSource)> = (0u64..=52) + .map(|v| (v, LogPathFileType::Commit, CommitSource::Filesystem)) + .collect(); + log_files.extend([ + ( + 50, + LogPathFileType::MultiPartCheckpoint { + part_num: 1, + num_parts: 3, + }, + CommitSource::Filesystem, + ), + ( + 50, + LogPathFileType::MultiPartCheckpoint { + part_num: 2, + num_parts: 3, + }, + CommitSource::Filesystem, + ), + ( + 50, + LogPathFileType::MultiPartCheckpoint { + part_num: 3, + num_parts: 3, + }, + CommitSource::Filesystem, + ), + ]); + log_files +} + +struct BackwardScanExpected { + listings: u32, + checkpoint_parts: usize, + checkpoint_version: Version, + commit_count: usize, + first_commit: Version, + last_commit: Version, +} + +// Case 1: complete 3-part checkpoint at v50, single window needed +// Case 2: incomplete 1-of-2 part at v1500 in window 2, complete checkpoint at v500 in window 3 +#[rstest] +#[case::multipart_checkpoint( + multipart_checkpoint_files(), + 52, + BackwardScanExpected { listings: 1, checkpoint_parts: 3, checkpoint_version: 50, commit_count: 2, first_commit: 51, last_commit: 52 } + )] +#[case::incomplete_in_second_window_complete_in_third( + files_incomplete_in_second_window_complete_in_third_window(), + 3000, + BackwardScanExpected { listings: 3, checkpoint_parts: 1, checkpoint_version: 500, commit_count: 2500, first_commit: 501, last_commit: 3000 } + )] +#[tokio::test] +async fn backward_scan_multipart_checkpoint_cases( + #[case] log_files: Vec<(Version, LogPathFileType, CommitSource)>, + #[case] end_version: Version, + #[case] expected: BackwardScanExpected, +) { + let BackwardScanExpected { + listings: expected_listings, + checkpoint_parts: expected_checkpoint_parts, + checkpoint_version: expected_checkpoint_version, + commit_count: expected_commit_count, + first_commit: expected_first_commit, + last_commit: expected_last_commit, + } = expected; + let (storage, log_root) = create_storage(log_files).await; + let counter = CountingStorageHandler::new(storage); + + let result = LogSegmentFiles::list_with_backward_checkpoint_scan( + &counter, + &log_root, + vec![], + end_version, + ) + .unwrap(); + + assert_eq!(counter.call_count(), expected_listings); + assert_eq!(result.checkpoint_parts.len(), expected_checkpoint_parts); + assert!(result + .checkpoint_parts + .iter() + .all(|p| p.version == expected_checkpoint_version)); + assert_eq!(result.ascending_commit_files.len(), expected_commit_count); + assert_eq!( + result.ascending_commit_files.first().unwrap().version, + expected_first_commit + ); + assert_eq!( + result.ascending_commit_files.last().unwrap().version, + expected_last_commit + ); + assert_eq!( + result.latest_commit_file.unwrap().version, + expected_last_commit + ); +} + +#[tokio::test] +async fn backward_scan_with_log_tail_derives_lower_bound_from_checkpoint() { + // FS: commits v0..=v7 + checkpoint at v5. log_tail: catalog commits v8..=v10. + // The checkpoint at v5 sets the lower bound to v6, so FS commits v6 and v7 plus all + // catalog entries v8..=v10 are included. + let mut log_files: Vec<(Version, LogPathFileType, CommitSource)> = (0u64..=7) + .map(|v| (v, LogPathFileType::Commit, CommitSource::Filesystem)) + .collect(); + log_files.push(( + 5, + LogPathFileType::ClassicCheckpoint, + CommitSource::Filesystem, + )); + let (storage, log_root) = create_storage(log_files).await; + + let log_tail: Vec<_> = (8u64..=10) + .map(|v| { + make_parsed_log_path_with_source(v, LogPathFileType::Commit, CommitSource::Catalog) + }) + .collect(); + + let result = LogSegmentFiles::list_with_backward_checkpoint_scan( + storage.as_ref(), + &log_root, + log_tail, + 10, + ) + .unwrap(); + + assert_eq!(result.checkpoint_parts.len(), 1); + assert_eq!(result.checkpoint_parts[0].version, 5); + + // FS commits v6, v7 after the checkpoint; catalog commits v8..=v10 + let expected = [ + (6, CommitSource::Filesystem), + (7, CommitSource::Filesystem), + (8, CommitSource::Catalog), + (9, CommitSource::Catalog), + (10, CommitSource::Catalog), + ]; + assert_eq!(result.ascending_commit_files.len(), expected.len()); + for (file, (version, source)) in result.ascending_commit_files.iter().zip(expected) { + assert_eq!(file.version, version); + assert_source(file, source); + } + assert_eq!(result.latest_commit_file.unwrap().version, 10); +} + +#[tokio::test] +async fn backward_scan_with_log_tail_starting_before_checkpoint() { + // FS: commits v0..=v5 + checkpoint at v5 + CRC at v6. log_tail: catalog commits v3..=v8, + // starting before the checkpoint. The checkpoint at v5 sets the lower bound to v5, so + // log_tail v3..=v4 are excluded. The log_tail commit at v5 passes through (it is at the + // checkpoint version). The CRC at v6 is preserved even though v6 is within the log_tail range. + let mut log_files: Vec<(Version, LogPathFileType, CommitSource)> = (0u64..=5) + .map(|v| (v, LogPathFileType::Commit, CommitSource::Filesystem)) + .collect(); + log_files.push(( + 5, + LogPathFileType::ClassicCheckpoint, + CommitSource::Filesystem, + )); + log_files.push((6, LogPathFileType::Crc, CommitSource::Filesystem)); + let (storage, log_root) = create_storage(log_files).await; + + let log_tail: Vec<_> = (3u64..=8) + .map(|v| { + make_parsed_log_path_with_source(v, LogPathFileType::Commit, CommitSource::Catalog) + }) + .collect(); + + let result = LogSegmentFiles::list_with_backward_checkpoint_scan( + storage.as_ref(), + &log_root, + log_tail, + 8, + ) + .unwrap(); + + assert_eq!(result.checkpoint_parts.len(), 1); + assert_eq!(result.checkpoint_parts[0].version, 5); + + // CRC at v6 is preserved even though v6 is within the log_tail range + let crc = result.latest_crc_file.unwrap(); + assert_eq!(crc.version, 6); + assert!(matches!(crc.file_type, LogPathFileType::Crc)); + + // v5 passes the start version filter (>= 5) and is included here + assert_eq!(result.ascending_commit_files.len(), 4); + for (i, commit) in result.ascending_commit_files.iter().enumerate() { + assert_eq!(commit.version, (i + 5) as u64); + assert_source(commit, CommitSource::Catalog); + } + assert_eq!(result.latest_commit_file.unwrap().version, 8); +} + +#[tokio::test] +async fn backward_scan_log_tail_defines_latest_version() { + // FS: commits v0..=v5. log_tail: catalog commit v4. end_version=5. + // FS v4 and v5 are filtered since log_tail_start=4. max_published_version is Some(5), + // the highest FS commit seen within end_version, even though v5 is not in + // ascending_commit_files. + let log_files: Vec<(Version, LogPathFileType, CommitSource)> = (0u64..=5) + .map(|v| (v, LogPathFileType::Commit, CommitSource::Filesystem)) + .collect(); + let (storage, log_root) = create_storage(log_files).await; + + let log_tail = vec![make_parsed_log_path_with_source( + 4, + LogPathFileType::Commit, + CommitSource::Catalog, + )]; + + let result = LogSegmentFiles::list_with_backward_checkpoint_scan( + storage.as_ref(), + &log_root, + log_tail, + 5, + ) + .unwrap(); + + let expected = [ + (0, CommitSource::Filesystem), + (1, CommitSource::Filesystem), + (2, CommitSource::Filesystem), + (3, CommitSource::Filesystem), + (4, CommitSource::Catalog), + ]; + assert_eq!(result.ascending_commit_files.len(), expected.len()); + for (file, (version, source)) in result.ascending_commit_files.iter().zip(expected) { + assert_eq!(file.version, version); + assert_source(file, source); + } + assert_eq!(result.latest_commit_file.unwrap().version, 4); + assert_eq!(result.max_published_version, Some(5)); +} + +// ===== empty (0-byte) log file detection tests ===== + +/// Creates storage where some files can be empty (0 bytes). Each entry is +/// (version, file_type, is_empty). Non-empty files get placeholder content. +async fn create_storage_with_empty_files( + log_files: Vec<(Version, LogPathFileType, bool)>, +) -> (Arc, Url) { + let store = Arc::new(InMemory::new()); + let log_root = Url::parse("memory:///_delta_log/").unwrap(); + + for (version, file_type, is_empty) in log_files { + let path = log_path_for_file_type(version, &file_type); + let data = if is_empty { + bytes::Bytes::new() + } else { + bytes::Bytes::from("placeholder") + }; + store + .put(&ObjectPath::from(path.as_str()), data.into()) + .await + .expect("Failed to put test file"); + } + + let engine = SyncEngine::new_with_store(store); + (engine.storage_handler(), log_root) +} + +// v2.json is 0 bytes -> kept in listing (commits are source of truth, not skipped) +#[tokio::test] +async fn test_zero_byte_commit_kept_in_listing() { + let log_files = vec![ + (0, LogPathFileType::Commit, false), + (1, LogPathFileType::Commit, false), + (2, LogPathFileType::Commit, true), // empty but still listed + ]; + let (storage, log_root) = create_storage_with_empty_files(log_files).await; + + let result = + LogSegmentFiles::list(storage.as_ref(), &log_root, vec![], Some(0), Some(2)).unwrap(); + assert_eq!(result.ascending_commit_files.len(), 3); + assert_eq!(result.ascending_commit_files[0].version, 0); + assert_eq!(result.ascending_commit_files[1].version, 1); + assert_eq!(result.ascending_commit_files[2].version, 2); + assert_eq!(result.ascending_commit_files[2].location.size, 0); +} + +// 0.4.compacted.json is 0 bytes -> skipped, individual commits v0-v4 used instead +#[rstest] +#[case::forward_list(false)] +#[case::backward_scan(true)] +#[tokio::test] +async fn test_zero_byte_compaction_skipped_commits_used(#[case] use_backward_scan: bool) { + let log_files = vec![ + (0, LogPathFileType::Commit, false), + (1, LogPathFileType::Commit, false), + (2, LogPathFileType::Commit, false), + (3, LogPathFileType::Commit, false), + (4, LogPathFileType::Commit, false), + ( + 0, + LogPathFileType::CompactedCommit { hi: 4 }, + true, // empty compaction + ), + ]; + let (storage, log_root) = create_storage_with_empty_files(log_files).await; + + let result = if use_backward_scan { + LogSegmentFiles::list_with_backward_checkpoint_scan(storage.as_ref(), &log_root, vec![], 4) + .unwrap() + } else { + LogSegmentFiles::list(storage.as_ref(), &log_root, vec![], Some(0), Some(4)).unwrap() + }; + + assert!( + result.ascending_compaction_files.is_empty(), + "0-byte compaction should have been skipped" + ); + assert_eq!(result.ascending_commit_files.len(), 5); + for (i, commit) in result.ascending_commit_files.iter().enumerate() { + assert_eq!(commit.version, i as u64); + } +} + +// v10.checkpoint.parquet is 0 bytes -> skipped, falls back to valid checkpoint at v5 +#[rstest] +#[case::forward_list(false)] +#[case::backward_scan(true)] +#[tokio::test] +async fn test_zero_byte_checkpoint_skipped_older_used(#[case] use_backward_scan: bool) { + let log_files = vec![ + (0, LogPathFileType::Commit, false), + (1, LogPathFileType::Commit, false), + (2, LogPathFileType::Commit, false), + (3, LogPathFileType::Commit, false), + (4, LogPathFileType::Commit, false), + (5, LogPathFileType::Commit, false), + (6, LogPathFileType::Commit, false), + (7, LogPathFileType::Commit, false), + (8, LogPathFileType::Commit, false), + (9, LogPathFileType::Commit, false), + (10, LogPathFileType::Commit, false), + (5, LogPathFileType::ClassicCheckpoint, false), // valid checkpoint + (10, LogPathFileType::ClassicCheckpoint, true), // empty checkpoint + ]; + let (storage, log_root) = create_storage_with_empty_files(log_files).await; + + let result = if use_backward_scan { + LogSegmentFiles::list_with_backward_checkpoint_scan(storage.as_ref(), &log_root, vec![], 10) + .unwrap() + } else { + LogSegmentFiles::list(storage.as_ref(), &log_root, vec![], Some(0), Some(10)).unwrap() + }; + + // Should fall back to checkpoint at v5 (the empty v10 checkpoint is skipped) + assert_eq!(result.checkpoint_parts.len(), 1); + assert_eq!(result.checkpoint_parts[0].version, 5); + + // Commits after checkpoint v5: v6 through v10 + assert_eq!(result.ascending_commit_files.len(), 5); + assert_eq!(result.ascending_commit_files[0].version, 6); + assert_eq!(result.ascending_commit_files[4].version, 10); +} + +// v2.crc is 0 bytes -> kept (CRC is optional, may report size 0 on some platforms) +#[tokio::test] +async fn test_zero_byte_crc_kept() { + let log_files = vec![ + (0, LogPathFileType::Commit, false), + (1, LogPathFileType::Commit, false), + (2, LogPathFileType::Commit, false), + (1, LogPathFileType::Crc, false), // valid CRC + (2, LogPathFileType::Crc, true), // empty CRC, still kept + ]; + let (storage, log_root) = create_storage_with_empty_files(log_files).await; + + let result = + LogSegmentFiles::list(storage.as_ref(), &log_root, vec![], Some(0), Some(2)).unwrap(); + + // The 0-byte CRC at v2 is kept (latest_crc_file tracks the highest version) + let crc = result.latest_crc_file.unwrap(); + assert_eq!(crc.version, 2); +} + +// v1005.checkpoint.parquet is 0 bytes in window 1 -> scan continues to window 2, +// finds valid checkpoint at v5. Verifies the fix in find_complete_checkpoint_version. +#[tokio::test] +async fn test_zero_byte_checkpoint_backward_scan_crosses_windows() { + // Commits v0..=1005, valid checkpoint at v5, 0-byte checkpoint at v1005. + // Window 1 [6, 1006): sees 0-byte checkpoint at v1005, must NOT stop. + // Window 2 [0, 6): finds valid checkpoint at v5 -> stop. + let mut log_files: Vec<(Version, LogPathFileType, bool)> = (0u64..=1005) + .map(|v| (v, LogPathFileType::Commit, false)) + .collect(); + log_files.push((5, LogPathFileType::ClassicCheckpoint, false)); + log_files.push((1005, LogPathFileType::ClassicCheckpoint, true)); + + let (storage, log_root) = create_storage_with_empty_files(log_files).await; + let counter = CountingStorageHandler::new(storage); + + let result = + LogSegmentFiles::list_with_backward_checkpoint_scan(&counter, &log_root, vec![], 1005) + .unwrap(); + + // Needed 2 windows because the 0-byte checkpoint at v1005 was skipped + assert_eq!(counter.call_count(), 2); + assert_eq!(result.checkpoint_parts.len(), 1); + assert_eq!(result.checkpoint_parts[0].version, 5); + // Commits after checkpoint v5: v6 through v1005 + assert_eq!(result.ascending_commit_files.len(), 1000); + assert_eq!(result.ascending_commit_files[0].version, 6); + assert_eq!(result.ascending_commit_files[999].version, 1005); +} + +// 0-byte commit in list_commits -> kept (commits are source of truth, not skipped) +#[tokio::test] +async fn test_list_commits_zero_byte_commit_kept() { + let log_files = vec![ + (0, LogPathFileType::Commit, false), + (1, LogPathFileType::Commit, false), + (2, LogPathFileType::Commit, true), // empty but still listed + ]; + let (storage, log_root) = create_storage_with_empty_files(log_files).await; + + let result = + LogSegmentFiles::list_commits(storage.as_ref(), &log_root, vec![], Some(0), Some(2)) + .unwrap(); + assert_eq!(result.ascending_commit_files.len(), 3); + assert_eq!(result.ascending_commit_files[2].version, 2); + assert_eq!(result.ascending_commit_files[2].location.size, 0); +} + +/// `list_commits` merges a caller-provided `log_tail` over the filesystem listing: the log_tail +/// supersedes filesystem commits at overlapping versions and is clipped to `[start, end]`, while +/// the published watermark counts only filesystem (published) commits. +#[rstest] +// log_tail supersedes filesystem at overlapping versions; filesystem commits still set the +// watermark. +#[case::supersedes_filesystem( + &[0, 1, 2], + &[1, 2], + Some(0), Some(2), + &[(0, CommitSource::Filesystem), (1, CommitSource::Catalog), (2, CommitSource::Catalog)], + Some(2), +)] +// staged tail extends the prefix but does not move the published watermark. +#[case::staged_tail_keeps_watermark( + &[0], + &[1, 2], + Some(0), Some(2), + &[(0, CommitSource::Filesystem), (1, CommitSource::Catalog), (2, CommitSource::Catalog)], + Some(0), +)] +// log_tail entries below `start` are dropped (filesystem v3 superseded by the tail). +#[case::drops_tail_below_start( + &[3], + &[2, 3, 4], + Some(3), Some(4), + &[(3, CommitSource::Catalog), (4, CommitSource::Catalog)], + Some(3), +)] +// log_tail entries above `end` are dropped. +#[case::drops_tail_above_end( + &[3], + &[2, 3, 4], + Some(3), Some(3), + &[(3, CommitSource::Catalog)], + Some(3), +)] +#[tokio::test] +async fn list_commits_merges_log_tail( + #[case] filesystem_commits: &[Version], + #[case] staged: &[Version], + #[case] start: Option, + #[case] end: Option, + #[case] expected: &[(Version, CommitSource)], + #[case] expected_max_published: Option, +) { + let (storage, log_root) = create_storage( + filesystem_commits + .iter() + .map(|v| (*v, LogPathFileType::Commit, CommitSource::Filesystem)) + .collect(), + ) + .await; + let log_tail: Vec = staged + .iter() + .map(|v| { + make_parsed_log_path_with_source( + *v, + LogPathFileType::StagedCommit, + CommitSource::Catalog, + ) + }) + .collect(); + + let result = + LogSegmentFiles::list_commits(storage.as_ref(), &log_root, log_tail, start, end).unwrap(); + + let commits = &result.ascending_commit_files; + assert_eq!(commits.len(), expected.len()); + for (commit, (version, source)) in commits.iter().zip(expected) { + assert_eq!(commit.version, *version); + assert_source(commit, *source); + } + assert_eq!(result.max_published_version, expected_max_published); + if let Some((last_version, _)) = expected.last() { + assert_eq!(result.latest_commit_file.unwrap().version, *last_version); + } +} + +#[tokio::test] +async fn test_list_commits_keeps_commits_across_checkpoint() { + let mut files: Vec<_> = (0..=5) + .map(|v| (v, LogPathFileType::Commit, CommitSource::Filesystem)) + .collect(); + files.push(( + 3, + LogPathFileType::ClassicCheckpoint, + CommitSource::Filesystem, + )); + let (storage, log_root) = create_storage(files).await; + + let result = + LogSegmentFiles::list_commits(storage.as_ref(), &log_root, vec![], Some(0), Some(5)) + .unwrap(); + let versions: Vec<_> = result + .ascending_commit_files + .iter() + .map(|c| c.version) + .collect(); + assert_eq!(versions, vec![0, 1, 2, 3, 4, 5]); +} + +// --------------------------------------------------------------------------- +// find_complete_checkpoint_version direct unit tests +// (other cases already covered by tests above) +// --------------------------------------------------------------------------- + +fn incomplete_then_complete_files() -> Vec { + // Commits v0..=10, an incomplete checkpoint at v5 (1 of 3 parts), and a complete + // checkpoint at v10. find_complete_checkpoint_version must continue past the failed group + // and find the complete one. + let mut files: Vec = (0..=10) + .map(|v| { + make_parsed_log_path_with_source(v, LogPathFileType::Commit, CommitSource::Filesystem) + }) + .collect(); + files.push(make_parsed_log_path_with_source( + 5, + LogPathFileType::MultiPartCheckpoint { + part_num: 1, + num_parts: 3, + }, + CommitSource::Filesystem, + )); + files.push(make_parsed_log_path_with_source( + 10, + LogPathFileType::ClassicCheckpoint, + CommitSource::Filesystem, + )); + files +} + +fn two_complete_checkpoints_files() -> Vec { + // Commits v0..=10, complete checkpoint at v5 and complete checkpoint at v10. + // The function must return the latest (v10), not the first (v5). + let mut files: Vec = (0..=10) + .map(|v| { + make_parsed_log_path_with_source(v, LogPathFileType::Commit, CommitSource::Filesystem) + }) + .collect(); + files.push(make_parsed_log_path_with_source( + 5, + LogPathFileType::ClassicCheckpoint, + CommitSource::Filesystem, + )); + files.push(make_parsed_log_path_with_source( + 10, + LogPathFileType::ClassicCheckpoint, + CommitSource::Filesystem, + )); + files +} + +#[rstest] +// Commits v0..=5, no checkpoint files +#[case::no_checkpoint( + (0u64..=5).map(|v| make_parsed_log_path_with_source(v, LogPathFileType::Commit, CommitSource::Filesystem)).collect(), + None + )] +// Commits v0..=10, incomplete checkpoint at v5, complete checkpoint at v10 +#[case::incomplete_then_complete(incomplete_then_complete_files(), Some(10))] +// Commits v0..=10, complete checkpoint at v5 and v10: must return v10 (latest) +#[case::two_complete(two_complete_checkpoints_files(), Some(10))] +fn find_complete_checkpoint_version_cases( + #[case] files: Vec, + #[case] expected: Option, +) { + assert_eq!(find_complete_checkpoint_version(&files), expected); +} + +/// [`crate::path::tests::parse_log_path`] stamped with this module's filesystem size marker. Unlike +/// [`make_parsed_log_path_with_source`], whose url is always `.json`, this works for +/// checkpoint paths, whose file name takes part in checkpoint selection. +fn parse_log_path(filename: &str) -> ParsedLogPath { + crate::path::tests::parse_log_path(filename, FILESYSTEM_SIZE_MARKER) +} + +/// A v5 `_last_checkpoint` hint describing the uuid-named (V2) checkpoint `filename`. +fn hint_naming_uuid(filename: &str) -> LastCheckpointHint { + LastCheckpointHint { + version: 5, + v2_checkpoint: Some(crate::last_checkpoint_hint::LastCheckpointV2 { + path: filename.to_string(), + ..Default::default() + }), + ..Default::default() + } +} + +#[test] +fn checkpoint_instance_orders_like_delta_spark() { + let classic = CheckpointInstance::Classic; + let uuid_a = CheckpointInstance::Uuid { + filename: "00000000000000000005.checkpoint.aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa.parquet" + .to_string(), + }; + let uuid_b = CheckpointInstance::Uuid { + filename: "00000000000000000005.checkpoint.bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb.parquet" + .to_string(), + }; + + assert!(classic < CheckpointInstance::MultiPart { num_parts: 2 }); + assert!( + CheckpointInstance::MultiPart { num_parts: 2 } + < CheckpointInstance::MultiPart { num_parts: 11 } + ); + assert!(classic < CheckpointInstance::MultiPart { num_parts: 11 }); + assert!(uuid_a > CheckpointInstance::MultiPart { num_parts: 11 }); + assert!(uuid_a > classic); + // Two uuid checkpoints at one version break the tie on file name. + assert!(uuid_a < uuid_b); +} + +/// The reported failure: two writers checkpointed one version with different part counts. +#[tokio::test] +async fn two_complete_checkpoints_at_one_version_selects_more_parts() { + let mut log_files: Vec<_> = (0..=6) + .map(|v| (v, LogPathFileType::Commit, CommitSource::Filesystem)) + .collect(); + for part_num in 1..=2 { + log_files.push(( + 4, + LogPathFileType::MultiPartCheckpoint { + part_num, + num_parts: 2, + }, + CommitSource::Filesystem, + )); + } + for part_num in 1..=11 { + log_files.push(( + 4, + LogPathFileType::MultiPartCheckpoint { + part_num, + num_parts: 11, + }, + CommitSource::Filesystem, + )); + } + let (storage, log_root) = create_storage(log_files).await; + + let (commit_files, _, checkpoint_parts, _, _, _) = + list_and_destructure(storage.as_ref(), &log_root, vec![], None, None); + + let selected: Vec<_> = checkpoint_parts.iter().map(|p| &p.filename).collect(); + let expected: Vec<_> = (1..=11) + .map(|p| multipart_checkpoint_name(4, p, 11)) + .collect(); + assert_eq!(selected, expected.iter().collect::>()); + let commit_versions: Vec<_> = commit_files.iter().map(|c| c.version).collect(); + assert_eq!(commit_versions, vec![5, 6]); +} + +/// Rank orders the candidates, but only complete checkpoints are candidates at all. +#[tokio::test] +async fn torn_higher_ranked_checkpoint_loses_to_complete_lower_ranked() { + let mut log_files: Vec<_> = (0..=6) + .map(|v| (v, LogPathFileType::Commit, CommitSource::Filesystem)) + .collect(); + for part_num in 1..=2 { + log_files.push(( + 4, + LogPathFileType::MultiPartCheckpoint { + part_num, + num_parts: 2, + }, + CommitSource::Filesystem, + )); + } + // A writer got 3 of its 11 parts out before dying: outranks the 2-part group, unusable. + for part_num in 1..=3 { + log_files.push(( + 4, + LogPathFileType::MultiPartCheckpoint { + part_num, + num_parts: 11, + }, + CommitSource::Filesystem, + )); + } + let (storage, log_root) = create_storage(log_files).await; + + let (commit_files, _, checkpoint_parts, _, _, _) = + list_and_destructure(storage.as_ref(), &log_root, vec![], None, None); + + let selected: Vec<_> = checkpoint_parts.iter().map(|p| &p.filename).collect(); + let expected: Vec<_> = (1..=2) + .map(|p| multipart_checkpoint_name(4, p, 2)) + .collect(); + assert_eq!(selected, expected.iter().collect::>()); + let commit_versions: Vec<_> = commit_files.iter().map(|c| c.version).collect(); + assert_eq!(commit_versions, vec![5, 6]); +} + +/// Same rule as above, but the surviving candidate is a single-file checkpoint. +#[tokio::test] +async fn torn_multipart_checkpoint_loses_to_classic_checkpoint() { + let mut log_files = vec![ + (1, LogPathFileType::Commit, CommitSource::Filesystem), + ( + 1, + LogPathFileType::ClassicCheckpoint, + CommitSource::Filesystem, + ), + ]; + for part_num in 1..=2 { + log_files.push(( + 1, + LogPathFileType::MultiPartCheckpoint { + part_num, + num_parts: 3, + }, + CommitSource::Filesystem, + )); + } + let (storage, log_root) = create_storage(log_files).await; + + let (_, _, checkpoint_parts, _, _, _) = + list_and_destructure(storage.as_ref(), &log_root, vec![], None, None); + + assert_eq!(checkpoint_parts.len(), 1); + assert_eq!( + checkpoint_parts[0].filename, + "00000000000000000001.checkpoint.parquet" + ); +} + +#[tokio::test] +async fn uuid_and_multipart_checkpoints_at_one_version_selects_uuid() { + let (storage, log_root) = create_storage_with_raw_paths( + vec![ + (1, LogPathFileType::Commit, CommitSource::Filesystem), + (2, LogPathFileType::Commit, CommitSource::Filesystem), + ], + &[ + &format!("_delta_log/{}", multipart_checkpoint_name(1, 1, 2)), + &format!("_delta_log/{}", multipart_checkpoint_name(1, 2, 2)), + "_delta_log/00000000000000000001.checkpoint.aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa.parquet", + ], + ) + .await; + + let (_, _, checkpoint_parts, _, _, _) = + list_and_destructure(storage.as_ref(), &log_root, vec![], None, None); + + assert_eq!(checkpoint_parts.len(), 1); + assert_eq!( + checkpoint_parts[0].filename, + "00000000000000000001.checkpoint.aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa.parquet" + ); +} + +/// Two uuid-named checkpoints at one version are distinct checkpoints, not parts of one, since each +/// writer picked its own uuid. +#[tokio::test] +async fn two_uuid_checkpoints_at_one_version_select_greater_filename() { + let (storage, log_root) = create_storage_with_raw_paths( + vec![(1, LogPathFileType::Commit, CommitSource::Filesystem)], + &[ + "_delta_log/00000000000000000001.checkpoint.11111111-1111-1111-1111-111111111111.parquet", + "_delta_log/00000000000000000001.checkpoint.22222222-2222-2222-2222-222222222222.parquet", + ], + ) + .await; + + let (_, _, checkpoint_parts, _, _, _) = + list_and_destructure(storage.as_ref(), &log_root, vec![], None, None); + + assert_eq!(checkpoint_parts.len(), 1); + assert_eq!( + checkpoint_parts[0].filename, + "00000000000000000001.checkpoint.22222222-2222-2222-2222-222222222222.parquet" + ); +} + +/// v5 holds a complete 2-part checkpoint and two uuid-named ones, so listing selects uuid `bbbb`. +/// Each case points the hint at a different candidate; only the one naming `bbbb` applies. Runs +/// `applies_to` against listed paths rather than hand-built ones. +#[rstest] +#[case::hint_names_winner_uuid( + hint_naming_uuid( + "00000000000000000005.checkpoint.bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb.parquet" + ), + true +)] +#[case::hint_names_loser_uuid( + hint_naming_uuid( + "00000000000000000005.checkpoint.aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa.parquet" + ), + false +)] +#[case::hint_names_losing_multipart( + LastCheckpointHint { version: 5, parts: Some(2), ..Default::default() }, + false +)] +#[case::hint_names_phantom_classic( + LastCheckpointHint { version: 5, ..Default::default() }, + false +)] +#[tokio::test] +async fn last_checkpoint_hint_applies_iff_it_names_the_selected_checkpoint( + #[case] hint: LastCheckpointHint, + #[case] expect_hint_applies: bool, +) { + let winner = "00000000000000000005.checkpoint.bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb.parquet"; + let (storage, log_root) = create_storage_with_raw_paths( + vec![ + (5, LogPathFileType::Commit, CommitSource::Filesystem), + (6, LogPathFileType::Commit, CommitSource::Filesystem), + ], + &[ + &format!("_delta_log/{}", multipart_checkpoint_name(5, 1, 2)), + &format!("_delta_log/{}", multipart_checkpoint_name(5, 2, 2)), + "_delta_log/00000000000000000005.checkpoint.aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa.parquet", + &format!("_delta_log/{winner}"), + ], + ) + .await; + + let listed = LogSegmentFiles::list_with_checkpoint_hint( + &hint, + storage.as_ref(), + &log_root, + vec![], + None, + ) + .unwrap(); + + // The winner is the same no matter what the hint names. + assert_eq!(listed.checkpoint_parts.len(), 1); + assert_eq!(listed.checkpoint_parts[0].filename, winner); + assert_eq!( + hint.applies_to(&listed.checkpoint_parts), + expect_hint_applies + ); +} + +#[tokio::test] +async fn uuid_checkpoint_beats_classic_checkpoint_at_one_version() { + let (storage, log_root) = create_storage_with_raw_paths( + vec![ + (1, LogPathFileType::Commit, CommitSource::Filesystem), + (1, LogPathFileType::ClassicCheckpoint, CommitSource::Filesystem), + ], + &["_delta_log/00000000000000000001.checkpoint.aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa.parquet"], + ) + .await; + + let (_, _, checkpoint_parts, _, _, _) = + list_and_destructure(storage.as_ref(), &log_root, vec![], None, None); + + assert_eq!(checkpoint_parts.len(), 1); + assert_eq!( + checkpoint_parts[0].filename, + "00000000000000000001.checkpoint.aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa.parquet" + ); +} + +/// A json checkpoint has no parquet footer, so selecting it routes schema discovery through the +/// sidecar path rather than a footer read (see `LogSegment::get_file_actions_schema_and_sidecars`). +#[tokio::test] +async fn json_uuid_checkpoint_beats_v1_checkpoints_at_one_version() { + let (storage, log_root) = create_storage_with_raw_paths( + vec![ + (1, LogPathFileType::Commit, CommitSource::Filesystem), + ( + 1, + LogPathFileType::ClassicCheckpoint, + CommitSource::Filesystem, + ), + ], + &[ + &format!("_delta_log/{}", multipart_checkpoint_name(1, 1, 2)), + &format!("_delta_log/{}", multipart_checkpoint_name(1, 2, 2)), + "_delta_log/00000000000000000001.checkpoint.aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa.json", + ], + ) + .await; + + let (_, _, checkpoint_parts, _, _, _) = + list_and_destructure(storage.as_ref(), &log_root, vec![], None, None); + + assert_eq!(checkpoint_parts.len(), 1); + assert_eq!( + checkpoint_parts[0].filename, + "00000000000000000001.checkpoint.aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa.json" + ); +} + +/// Both are complete single-file V1 checkpoints, so they replay to the same state either way. +#[tokio::test] +async fn one_of_one_multipart_checkpoint_beats_classic_checkpoint() { + let (storage, log_root) = create_storage(vec![ + (1, LogPathFileType::Commit, CommitSource::Filesystem), + ( + 1, + LogPathFileType::ClassicCheckpoint, + CommitSource::Filesystem, + ), + ( + 1, + LogPathFileType::MultiPartCheckpoint { + part_num: 1, + num_parts: 1, + }, + CommitSource::Filesystem, + ), + ]) + .await; + + let (_, _, checkpoint_parts, _, _, _) = + list_and_destructure(storage.as_ref(), &log_root, vec![], None, None); + + assert_eq!(checkpoint_parts.len(), 1); + assert_eq!( + checkpoint_parts[0].filename, + multipart_checkpoint_name(1, 1, 1) + ); +} + +#[rstest] +// A classic and a 1-of-1 multi-part checkpoint at one version each fill their own group. +#[case::single_and_one_of_one_same_version( + vec![ + parse_log_path(&multipart_checkpoint_name(5, 1, 1)), + parse_log_path("00000000000000000005.checkpoint.parquet"), + ], + Some(5) + )] +// A complete 2-part and a complete 11-part checkpoint at one version. +#[case::two_complete_multipart_same_version( + (1..=2).map(|p| parse_log_path(&multipart_checkpoint_name(10, p, 2))) + .chain((1..=11).map(|p| parse_log_path(&multipart_checkpoint_name(10, p, 11)))) + .sorted_by_key(|f| f.filename.clone()) + .collect(), + Some(10) + )] +// A torn 11-part group beside a complete 2-part one: still checkpointed, on the complete group. +#[case::torn_beside_complete_same_version( + (1..=2).map(|p| parse_log_path(&multipart_checkpoint_name(10, p, 2))) + .chain((1..=3).map(|p| parse_log_path(&multipart_checkpoint_name(10, p, 11)))) + .sorted_by_key(|f| f.filename.clone()) + .collect(), + Some(10) + )] +// Every group at the version is torn, so the version is not checkpointed. +#[case::all_torn_same_version( + (1..=1).map(|p| parse_log_path(&multipart_checkpoint_name(10, p, 2))) + .chain((1..=3).map(|p| parse_log_path(&multipart_checkpoint_name(10, p, 11)))) + .sorted_by_key(|f| f.filename.clone()) + .collect(), + None + )] +fn find_complete_checkpoint_version_same_version_cases( + #[case] files: Vec, + #[case] expected: Option, +) { + assert_eq!(find_complete_checkpoint_version(&files), expected); +} diff --git a/kernel/src/metrics/README.md b/kernel/src/metrics/README.md new file mode 100644 index 0000000000..9f4f98d65d --- /dev/null +++ b/kernel/src/metrics/README.md @@ -0,0 +1,35 @@ +# Kernel metrics + +Kernel emits metrics as `tracing` spans. A connector installs `ReportGeneratorLayer` +(`reporter.rs`), which turns each span carrying the `report` field into a `MetricEvent` +(`events.rs`) and hands it to the connector's `MetricsReporter`. Kernel does no aggregation. + +## Two mechanisms + +The difference is *when a metric's values are known*: + +- **Span-based** (`#[instrument]` on a function): values arrive over the function's life. Some at + the start, some mid-call (`Span::current().record(...)`), the duration at the end. +- **Emit-based** (an `emit_*` helper): all values are known at once, so the helper fires a single + `span!` carrying every field (including `duration_ns`) and drops it. Nothing arrives later. + +`from_attrs` builds the event from a span's fields. It reads them with a `Visit` (the trait whose +`record_*` methods receive one field at a time). Emit events run one `Visit` over the whole struct; +span-based events reuse small per-field helpers like `MetricId::from_attrs`, each a `Visit` inside. + +## Failure events + +Success and failure are two variants of one event. Emit the success span, then fire `error` on it +(`#[instrument(err)]` does this on `Err`; an emit helper calls `tracing::info!(error = ...)`). The +layer sees `error` and flips the event via `MetricEvent::into_failure`. + +## Adding an event + +1. Define the struct + `SPAN_NAME` in `events.rs`; add a `MetricEvent` variant and its + `into_failure` arm. +2. For emit, add an `emit_*` helper and a `from_attrs`; register the span name in `on_new_span` + (`reporter.rs`). +3. Make each label a strum enum (`serialize_all = "snake_case"`). + +The FFI mirror (`ffi/src/ffi_metrics.rs`) and connectors destructure these structs, so adding a +field or variant is a breaking change. diff --git a/kernel/src/metrics/events.rs b/kernel/src/metrics/events.rs index b9ebf40e14..3ed29b07e0 100644 --- a/kernel/src/metrics/events.rs +++ b/kernel/src/metrics/events.rs @@ -1,139 +1,2026 @@ -//! Metric event types and utilities. +//! Metric event types emitted during Delta Kernel operations. +//! +//! Each [`MetricEvent`] variant wraps a per-event struct that owns its fields, `Display` impl, +//! span name, and the small set of methods the `tracing` layer in [`crate::metrics::reporter`] +//! calls to construct and finalize the event. Per-event code is colocated in a single block +//! per type below. +//! +//! Enums carried in event payloads derive the full strum set (`EnumString`, `Display`, +//! `AsRefStr`, `IntoStaticStr`) with stable serialized names. `IntoStaticStr` in particular +//! lets connectors convert a value to its `&'static str` metric-label string via `.into()` +//! instead of maintaining their own variant-to-string `match`. +//! +//! Event construction is infallible: a malformed span field warns and falls back to a +//! default rather than failing the operation being observed. use std::fmt; +use std::str::FromStr; +use std::sync::Arc; use std::time::Duration; + +use delta_kernel_derive::internal_api; +use strum::{AsRefStr, Display as StrumDisplay, EnumString, IntoStaticStr}; +use tracing::field::{Field, Visit}; +use tracing::span::Attributes; +use tracing::warn; use uuid::Uuid; +use crate::log_segment::LogSegment; + +// ==================================================================== +// MetricId +// ==================================================================== + /// Unique identifier for a metrics operation. /// -/// Each operation (Snapshot, Transaction, Scan) gets a unique MetricId that -/// is used to correlate all events from that operation. +/// Each operation (Snapshot, Transaction, Scan) gets a unique `MetricId` that correlates all +/// events emitted from that operation. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] -pub struct MetricId(Uuid); +pub struct MetricId(pub(crate) Uuid); impl MetricId { - /// Generate a new unique MetricId. + /// Generate a new unique `MetricId`. + #[allow(clippy::new_without_default)] pub fn new() -> Self { Self(Uuid::new_v4()) } -} -impl Default for MetricId { - fn default() -> Self { - Self::new() + /// The nil id (all-zero UUID): the decode seed when a span omits `operation_id`, and the + /// sentinel for a malformed one. + pub(crate) fn nil() -> Self { + Self(Uuid::nil()) + } + + /// Return the 16 raw bytes of the underlying UUID. Useful for FFI consumers that want to + /// carry the id without allocating or parsing its string form. + pub fn as_bytes(&self) -> [u8; 16] { + *self.0.as_bytes() + } + + /// Extract the `operation_id` field from span attributes. Returns a nil id if the field is + /// absent, or warns and returns nil if the value is present but malformed. + pub(crate) fn from_attrs(attrs: &Attributes<'_>) -> Self { + #[derive(Default)] + struct V(Uuid); + impl Visit for V { + fn record_debug(&mut self, field: &Field, value: &dyn fmt::Debug) { + if field.name() == "operation_id" { + let s = format!("{value:?}"); + match Uuid::from_str(&s) { + Ok(u) => self.0 = u, + Err(e) => warn!("Invalid uuid '{s}' on span: {e}. Using default."), + } + } + } + } + let mut v = V::default(); + attrs.record(&mut v); + Self(v.0) } } impl fmt::Display for MetricId { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - write!(f, "{}", self.0) + self.0.fmt(f) } } +// ==================================================================== +// MetricEvent +// ==================================================================== + /// Metric events emitted during Delta Kernel operations. -/// -/// Some events include an `operation_id` (MetricId) that uniquely identifies the operation -/// instance. This allows correlating multiple events from the same operation. #[derive(Debug, Clone)] pub enum MetricEvent { - /// Log segment loading completed (listing and organizing log files). - LogSegmentLoaded { - operation_id: MetricId, - duration: Duration, - num_commit_files: u64, - num_checkpoint_files: u64, - num_compaction_files: u64, - }, - - /// Protocol and metadata loading completed. - ProtocolMetadataLoaded { - operation_id: MetricId, - duration: Duration, - }, - - /// Snapshot creation completed successfully. - SnapshotCompleted { - operation_id: MetricId, - version: u64, - total_duration: Duration, - }, - - /// Snapshot creation failed. - SnapshotFailed { - operation_id: MetricId, - duration: Duration, - }, - - /// Storage list operation completed. - /// These events track storage-level latencies and are emitted automatically - /// by the default storage handler implementation. - StorageListCompleted { duration: Duration, num_files: u64 }, - - /// Storage read operation completed. - StorageReadCompleted { - duration: Duration, - num_files: u64, - bytes_read: u64, - }, - - /// Storage copy operation completed. - StorageCopyCompleted { duration: Duration }, + LogSegmentLoadSuccess(LogSegmentLoadSuccess), + LogSegmentLoadFailure(LogSegmentLoadFailure), + ProtocolMetadataLoadSuccess(ProtocolMetadataLoadSuccess), + ProtocolMetadataLoadFailure(ProtocolMetadataLoadFailure), + SnapshotBuildSuccess(SnapshotBuildSuccess), + SnapshotBuildFailure(SnapshotBuildFailure), + TransactionCommitSuccess(TransactionCommitSuccess), + TransactionCommitFailure(TransactionCommitFailure), + DomainMetadataLoadSuccess(DomainMetadataLoadSuccess), + DomainMetadataLoadFailure, + SetTransactionLoadSuccess(SetTransactionLoadSuccess), + SetTransactionLoadFailure, + CrcReadSuccess(CrcReadSuccess), + CrcReadFailure, + JsonReadCompleted(JsonReadCompleted), + ParquetReadCompleted(ParquetReadCompleted), + ScanMetadataCompleted(ScanMetadataCompleted), + StorageListCompleted(StorageListCompleted), + StorageReadCompleted(StorageReadCompleted), + StorageCopyCompleted(StorageCopyCompleted), +} + +impl MetricEvent { + /// Set the wall-clock duration on lifecycle events. + pub(crate) fn set_duration_if_applicable(&mut self, d: Duration) { + match self { + // Lifecycle success: duration must be set by the tracing layer on span close. + Self::SnapshotBuildSuccess(e) => e.set_duration(d), + Self::TransactionCommitSuccess(e) => e.set_duration(d), + Self::DomainMetadataLoadSuccess(e) => e.set_duration(d), + Self::SetTransactionLoadSuccess(e) => e.set_duration(d), + Self::CrcReadSuccess(e) => e.set_duration(d), + + // Emit-based success events set their duration as a creation attribute, so the + // span-close hook must not overwrite it. + Self::LogSegmentLoadSuccess(_) + | Self::ProtocolMetadataLoadSuccess(_) + | Self::ScanMetadataCompleted(_) + | Self::StorageListCompleted(_) + | Self::StorageReadCompleted(_) + | Self::StorageCopyCompleted(_) => {} + + // Failure events carry no duration. + Self::LogSegmentLoadFailure(_) + | Self::ProtocolMetadataLoadFailure(_) + | Self::SnapshotBuildFailure(_) + | Self::TransactionCommitFailure(_) + | Self::DomainMetadataLoadFailure + | Self::SetTransactionLoadFailure + | Self::CrcReadFailure => {} + + // Read events have no duration field. + Self::JsonReadCompleted(_) | Self::ParquetReadCompleted(_) => {} + } + } + + pub(crate) fn record_u64(&mut self, name: &str, value: u64) -> Result<(), &'static str> { + match self { + // Variants with u64 fields set during span lifetime. + Self::SnapshotBuildSuccess(e) => e.record_u64(name, value), + Self::TransactionCommitSuccess(e) => e.record_u64(name, value), + Self::DomainMetadataLoadSuccess(e) => e.record_u64(name, value), + Self::CrcReadSuccess(e) => e.record_u64(name, value), + + Self::LogSegmentLoadSuccess(_) => Err(LogSegmentLoadSuccess::SPAN_NAME), + Self::ProtocolMetadataLoadSuccess(_) => Err(ProtocolMetadataLoadSuccess::SPAN_NAME), + Self::SetTransactionLoadSuccess(_) => Err(SetTransactionLoadSuccess::SPAN_NAME), + Self::ScanMetadataCompleted(_) => Err(ScanMetadataCompleted::SPAN_NAME), + Self::JsonReadCompleted(_) => Err(JsonReadCompleted::SPAN_NAME), + Self::ParquetReadCompleted(_) => Err(ParquetReadCompleted::SPAN_NAME), + Self::StorageListCompleted(_) + | Self::StorageReadCompleted(_) + | Self::StorageCopyCompleted(_) => Err(STORAGE_SPAN), + + // Failure events are built at span close, after any runtime records. + Self::LogSegmentLoadFailure(_) => Err(LogSegmentLoadSuccess::SPAN_NAME), + Self::ProtocolMetadataLoadFailure(_) => Err(ProtocolMetadataLoadSuccess::SPAN_NAME), + Self::SnapshotBuildFailure(_) => Err(SnapshotBuildSuccess::SPAN_NAME), + Self::TransactionCommitFailure(_) => Err(TransactionCommitSuccess::SPAN_NAME), + Self::DomainMetadataLoadFailure => Err(DomainMetadataLoadSuccess::SPAN_NAME), + Self::SetTransactionLoadFailure => Err(SetTransactionLoadSuccess::SPAN_NAME), + Self::CrcReadFailure => Err(CrcReadSuccess::SPAN_NAME), + } + } + + pub(crate) fn record_bool(&mut self, name: &str, value: bool) -> Result<(), &'static str> { + match self { + // Variants with bool fields set during span lifetime. + Self::TransactionCommitSuccess(e) => e.record_bool(name, value), + Self::DomainMetadataLoadSuccess(e) => e.record_bool(name, value), + Self::SetTransactionLoadSuccess(e) => e.record_bool(name, value), + + // No bool fields set during span lifetime — a runtime record() on these is a bug. + Self::LogSegmentLoadSuccess(_) => Err(LogSegmentLoadSuccess::SPAN_NAME), + Self::ProtocolMetadataLoadSuccess(_) => Err(ProtocolMetadataLoadSuccess::SPAN_NAME), + Self::SnapshotBuildSuccess(_) => Err(SnapshotBuildSuccess::SPAN_NAME), + Self::CrcReadSuccess(_) => Err(CrcReadSuccess::SPAN_NAME), + Self::ScanMetadataCompleted(_) => Err(ScanMetadataCompleted::SPAN_NAME), + Self::JsonReadCompleted(_) => Err(JsonReadCompleted::SPAN_NAME), + Self::ParquetReadCompleted(_) => Err(ParquetReadCompleted::SPAN_NAME), + Self::StorageListCompleted(_) + | Self::StorageReadCompleted(_) + | Self::StorageCopyCompleted(_) => Err(STORAGE_SPAN), + + // Failure events are built at span close, after any runtime records. + Self::LogSegmentLoadFailure(_) => Err(LogSegmentLoadSuccess::SPAN_NAME), + Self::ProtocolMetadataLoadFailure(_) => Err(ProtocolMetadataLoadSuccess::SPAN_NAME), + Self::SnapshotBuildFailure(_) => Err(SnapshotBuildSuccess::SPAN_NAME), + Self::TransactionCommitFailure(_) => Err(TransactionCommitSuccess::SPAN_NAME), + Self::DomainMetadataLoadFailure => Err(DomainMetadataLoadSuccess::SPAN_NAME), + Self::SetTransactionLoadFailure => Err(SetTransactionLoadSuccess::SPAN_NAME), + Self::CrcReadFailure => Err(CrcReadSuccess::SPAN_NAME), + } + } + + pub(crate) fn record_str(&mut self, name: &str, value: &str) -> Result<(), &'static str> { + if name == "failure_reason" { + if let Self::TransactionCommitSuccess(e) = self { + let operation_id = e.operation_id; + let table_type = e.table_type; + let correlation_id = e.correlation_id.take(); + *self = Self::TransactionCommitFailure(TransactionCommitFailure { + operation_id, + table_type, + correlation_id, + reason: value.parse().unwrap_or_else(|e| { + warn!("Invalid failure_reason '{value}' on span: {e}. Using Error."); + CommitFailureReason::Error + }), + }); + } + return Ok(()); + } + // Only TransactionCommitSuccess records string fields today. If other events need them, + // dispatch per-variant like `record_u64`/`record_bool` above instead of this catch-all. + match self { + Self::TransactionCommitSuccess(e) => e.record_str(name, value), + _ => Ok(()), + } + } + + /// Maps a lifecycle success event to its failure counterpart when the span errors. + pub(crate) fn into_failure(self) -> Self { + match self { + Self::LogSegmentLoadSuccess(e) => Self::LogSegmentLoadFailure(LogSegmentLoadFailure { + operation_id: e.operation_id, + table_type: e.table_type, + correlation_id: e.correlation_id, + load_type: e.load_type, + }), + Self::ProtocolMetadataLoadSuccess(e) => { + Self::ProtocolMetadataLoadFailure(ProtocolMetadataLoadFailure { + operation_id: e.operation_id, + table_type: e.table_type, + correlation_id: e.correlation_id, + load_type: e.load_type, + }) + } + Self::SnapshotBuildSuccess(e) => Self::SnapshotBuildFailure(SnapshotBuildFailure { + operation_id: e.operation_id, + table_type: e.table_type, + correlation_id: e.correlation_id, + load_type: e.load_type, + }), + Self::TransactionCommitSuccess(e) => { + Self::TransactionCommitFailure(TransactionCommitFailure { + operation_id: e.operation_id, + table_type: e.table_type, + correlation_id: e.correlation_id, + reason: CommitFailureReason::Error, + }) + } + Self::DomainMetadataLoadSuccess(_) => Self::DomainMetadataLoadFailure, + Self::SetTransactionLoadSuccess(_) => Self::SetTransactionLoadFailure, + Self::CrcReadSuccess(_) => Self::CrcReadFailure, + // Events with no failure form pass through unchanged. + // Note: here we list explicitly so adding a lifecycle event without a failure mapping + // fails to compile. + e @ (Self::LogSegmentLoadFailure(_) + | Self::ProtocolMetadataLoadFailure(_) + | Self::SnapshotBuildFailure(_) + | Self::TransactionCommitFailure(_) + | Self::DomainMetadataLoadFailure + | Self::SetTransactionLoadFailure + | Self::CrcReadFailure + | Self::JsonReadCompleted(_) + | Self::ParquetReadCompleted(_) + | Self::ScanMetadataCompleted(_) + | Self::StorageListCompleted(_) + | Self::StorageReadCompleted(_) + | Self::StorageCopyCompleted(_)) => e, + } + } } impl fmt::Display for MetricEvent { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { match self { - MetricEvent::LogSegmentLoaded { - operation_id, - duration, - num_commit_files, - num_checkpoint_files, - num_compaction_files, - } => write!( - f, - "LogSegmentLoaded(id={operation_id}, duration={duration:?}, commits={num_commit_files}, checkpoints={num_checkpoint_files}, compactions={num_compaction_files})" - ), - MetricEvent::ProtocolMetadataLoaded { - operation_id, - duration, - } => write!( - f, - "ProtocolMetadataLoaded(id={operation_id}, duration={duration:?})" - ), - MetricEvent::SnapshotCompleted { - operation_id, - version, - total_duration, - } => write!( - f, - "SnapshotCompleted(id={operation_id}, version={version}, duration={total_duration:?})" - ), - MetricEvent::SnapshotFailed { - operation_id, - duration, - } => write!( - f, - "SnapshotFailed(id={operation_id}, duration={duration:?})" - ), - MetricEvent::StorageListCompleted { - duration, - num_files, - } => write!( - f, - "StorageListCompleted(duration={duration:?}, files={num_files})" - ), - MetricEvent::StorageReadCompleted { - duration, - num_files, - bytes_read, - } => write!( - f, - "StorageReadCompleted(duration={duration:?}, files={num_files}, bytes={bytes_read})" - ), - MetricEvent::StorageCopyCompleted { duration } => write!( - f, - "StorageCopyCompleted(duration={duration:?})" - ), + Self::LogSegmentLoadSuccess(e) => e.fmt(f), + Self::LogSegmentLoadFailure(e) => e.fmt(f), + Self::ProtocolMetadataLoadSuccess(e) => e.fmt(f), + Self::ProtocolMetadataLoadFailure(e) => e.fmt(f), + Self::SnapshotBuildSuccess(e) => e.fmt(f), + Self::SnapshotBuildFailure(e) => e.fmt(f), + Self::TransactionCommitSuccess(e) => e.fmt(f), + Self::TransactionCommitFailure(e) => e.fmt(f), + Self::DomainMetadataLoadSuccess(e) => e.fmt(f), + Self::DomainMetadataLoadFailure => f.write_str("DomainMetadataLoadFailure"), + Self::SetTransactionLoadSuccess(e) => e.fmt(f), + Self::SetTransactionLoadFailure => f.write_str("SetTransactionLoadFailure"), + Self::CrcReadSuccess(e) => e.fmt(f), + Self::CrcReadFailure => f.write_str("CrcReadFailure"), + Self::JsonReadCompleted(e) => e.fmt(f), + Self::ParquetReadCompleted(e) => e.fmt(f), + Self::ScanMetadataCompleted(e) => e.fmt(f), + Self::StorageListCompleted(e) => e.fmt(f), + Self::StorageReadCompleted(e) => e.fmt(f), + Self::StorageCopyCompleted(e) => e.fmt(f), + } + } +} + +// ==================================================================== +// LogSegmentLoad +// ==================================================================== +// +// Canonical example for the per-event block pattern. Other events below follow the same +// shape; detailed `///` docs on `from_attrs` and `record_*` live here only. + +pub(crate) const LOG_SEGMENT_LOADED_SPAN: &str = "segment.for_snapshot"; + +/// The kind of log-segment load: a full listing from the base up to the target, or an +/// incremental listing of the commits above an existing segment. +#[derive( + Debug, Clone, Copy, PartialEq, Eq, Default, EnumString, StrumDisplay, AsRefStr, IntoStaticStr, +)] +#[strum(serialize_all = "snake_case")] +#[non_exhaustive] +pub enum LogSegmentLoadType { + /// The segment was listed from its base (a checkpoint, else version 0) up to the target. A + /// fresh snapshot build (`LogSegment::for_snapshot`) reads this way. + Full, + /// The segment was listed as a delta above an existing base. An incremental snapshot update + /// (`Snapshot::try_new_from`) reads only the commits above the existing snapshot. + Incremental, + /// Decode fell back here because the span's `load_type` field was unset or unrecognized. + /// Kernel never emits this deliberately. + #[default] + Unknown, +} + +impl LogSegmentLoadType { + fn parse_or_unknown(s: &str) -> Self { + if s.is_empty() { + return Self::Unknown; + } + Self::from_str(s).unwrap_or_else(|e| { + warn!("Invalid load_type '{s}': {e}. Using Unknown."); + Self::Unknown + }) + } +} + +/// A log segment was listed and assembled for a snapshot. +/// +/// All fields are set at emit time (creation attrs). +#[derive(Debug, Clone)] +pub struct LogSegmentLoadSuccess { + pub operation_id: MetricId, + /// Opaque, caller-supplied id for joining this operation's metric events to the caller's + /// own request or operation id. + pub correlation_id: Option>, + pub table_type: TableType, + pub load_type: LogSegmentLoadType, + pub num_commit_files: u64, + pub num_checkpoint_files: u64, + pub num_compaction_files: u64, + /// How many versions behind the segment's end version the latest on-disk CRC file is, or + /// `None` if there is no CRC file. + pub crc_versions_behind: Option, + pub duration: Duration, +} + +impl LogSegmentLoadSuccess { + pub(crate) const SPAN_NAME: &'static str = LOG_SEGMENT_LOADED_SPAN; + + /// Span field for CRC staleness. `None` (no CRC file) encodes as `-1`. + const CRC_VERSIONS_BEHIND_FIELD: &'static str = "crc_versions_behind"; + + fn encode_crc_versions_behind(v: Option) -> i64 { + v.map_or(-1, |n| n as i64) + } + + fn decode_crc_versions_behind(v: i64) -> Option { + u64::try_from(v).ok() + } + + /// The default seed [`Self::from_attrs`] populates from span fields. + fn empty() -> Self { + Self { + operation_id: MetricId::nil(), + correlation_id: None, + table_type: TableType::from_catalog_managed(false), + load_type: LogSegmentLoadType::default(), + num_commit_files: 0, + num_checkpoint_files: 0, + num_compaction_files: 0, + crc_versions_behind: None, + duration: Duration::ZERO, + } + } + + pub(crate) fn from_attrs(attrs: &Attributes<'_>) -> Self { + let mut event = Self::empty(); + attrs.record(&mut event); + event + } +} + +impl Visit for LogSegmentLoadSuccess { + fn record_bool(&mut self, field: &Field, value: bool) { + if field.name() == IS_CATALOG_MANAGED_FIELD { + self.table_type = TableType::from_catalog_managed(value); + } + } + + fn record_str(&mut self, field: &Field, value: &str) { + match field.name() { + CORRELATION_ID_FIELD if !value.is_empty() => self.correlation_id = Some(value.into()), + "load_type" => self.load_type = LogSegmentLoadType::parse_or_unknown(value), + _ => {} + } + } + + fn record_u64(&mut self, field: &Field, value: u64) { + match field.name() { + "duration_ns" => self.duration = Duration::from_nanos(value), + "num_commit_files" => self.num_commit_files = value, + "num_checkpoint_files" => self.num_checkpoint_files = value, + "num_compaction_files" => self.num_compaction_files = value, + _ => {} + } + } + + fn record_i64(&mut self, field: &Field, value: i64) { + if field.name() == Self::CRC_VERSIONS_BEHIND_FIELD { + self.crc_versions_behind = Self::decode_crc_versions_behind(value); + } + } + + fn record_debug(&mut self, field: &Field, value: &dyn fmt::Debug) { + if let Some(id) = record_operation_id(field, value, Self::SPAN_NAME) { + self.operation_id = id; + } + } +} + +impl fmt::Display for LogSegmentLoadSuccess { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let Self { + operation_id, + table_type, + correlation_id, + load_type, + duration, + num_commit_files, + num_checkpoint_files, + num_compaction_files, + crc_versions_behind, + } = self; + write!( + f, + "LogSegmentLoadSuccess(id={operation_id}, table_type={table_type}, \ + correlation_id={correlation_id:?}, load_type={load_type}, \ + duration={duration:?}, commits={num_commit_files}, \ + checkpoints={num_checkpoint_files}, compactions={num_compaction_files}, \ + crc_versions_behind={crc_versions_behind:?})" + ) + } +} + +/// Listing the log segment for a snapshot failed. +#[derive(Debug, Clone)] +pub struct LogSegmentLoadFailure { + pub operation_id: MetricId, + /// Opaque, caller-supplied id for joining this operation's metric events to the caller's + /// own request or operation id. + pub correlation_id: Option>, + pub table_type: TableType, + pub load_type: LogSegmentLoadType, +} + +impl fmt::Display for LogSegmentLoadFailure { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "LogSegmentLoadFailure(id={}, table_type={}, correlation_id={:?}, load_type={})", + self.operation_id, self.table_type, self.correlation_id, self.load_type + ) + } +} + +// ==================================================================== +// ProtocolMetadataLoad +// ==================================================================== + +pub(crate) const PROTOCOL_METADATA_LOADED_SPAN: &str = "segment.read_metadata"; + +/// How a snapshot load resolved Protocol and Metadata. +#[derive(Debug, Clone, Copy, PartialEq, Eq, EnumString, StrumDisplay, AsRefStr, IntoStaticStr)] +#[strum(serialize_all = "snake_case")] +#[non_exhaustive] +pub enum ProtocolMetadataSource { + /// A CRC already sat at the target version; used as-is with zero replay. + CrcAtTarget, + /// A stale CRC was advanced to the target version by reverse replay. + CrcAdvancedByReplay, + /// A stale CRC seeded a pruned P&M replay of the commits above it (falling back to the + /// CRC's own P&M when the pruned replay found no newer Protocol or Metadata). + CrcSeededPmOnlyReplay, + /// No CRC baseline; P&M came from log replay (full history on a fresh load, or the new + /// commits on an incremental load). + FullReplay, + /// Decode fell back here because the span's `pm_source` field was unset or unrecognized. + /// Kernel never emits this deliberately. + Unknown, +} + +impl ProtocolMetadataSource { + fn parse_or_unknown(s: &str) -> Self { + if s.is_empty() { + return Self::Unknown; + } + Self::from_str(s).unwrap_or_else(|e| { + warn!("Invalid pm_source '{s}': {e}. Using Unknown."); + Self::Unknown + }) + } +} + +/// Protocol and metadata actions were resolved for a snapshot. +/// +/// All fields are set at emit time (creation attrs). +#[derive(Debug, Clone)] +pub struct ProtocolMetadataLoadSuccess { + pub operation_id: MetricId, + /// Opaque, caller-supplied id for joining this operation's metric events to the caller's + /// own request or operation id. + pub correlation_id: Option>, + pub table_type: TableType, + pub load_type: LogSegmentLoadType, + pub source: ProtocolMetadataSource, + pub duration: Duration, +} + +impl ProtocolMetadataLoadSuccess { + pub(crate) const SPAN_NAME: &'static str = PROTOCOL_METADATA_LOADED_SPAN; + + /// The all-unset seed that [`Self::from_attrs`] fills in as span fields arrive. Each field + /// here is the value used when its span field is absent. + fn empty() -> Self { + Self { + operation_id: MetricId::nil(), + correlation_id: None, + table_type: TableType::from_catalog_managed(false), + load_type: LogSegmentLoadType::default(), + source: ProtocolMetadataSource::Unknown, + duration: Duration::ZERO, + } + } + + pub(crate) fn from_attrs(attrs: &Attributes<'_>) -> Self { + let mut event = Self::empty(); + attrs.record(&mut event); + event + } +} + +impl Visit for ProtocolMetadataLoadSuccess { + fn record_bool(&mut self, field: &Field, value: bool) { + if field.name() == IS_CATALOG_MANAGED_FIELD { + self.table_type = TableType::from_catalog_managed(value); + } + } + + fn record_str(&mut self, field: &Field, value: &str) { + match field.name() { + CORRELATION_ID_FIELD if !value.is_empty() => self.correlation_id = Some(value.into()), + "load_type" => self.load_type = LogSegmentLoadType::parse_or_unknown(value), + "pm_source" => self.source = ProtocolMetadataSource::parse_or_unknown(value), + _ => {} + } + } + + fn record_u64(&mut self, field: &Field, value: u64) { + if field.name() == "duration_ns" { + self.duration = Duration::from_nanos(value); + } + } + + fn record_debug(&mut self, field: &Field, value: &dyn fmt::Debug) { + if let Some(id) = record_operation_id(field, value, Self::SPAN_NAME) { + self.operation_id = id; + } + } +} + +impl fmt::Display for ProtocolMetadataLoadSuccess { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let Self { + operation_id, + table_type, + correlation_id, + load_type, + source, + duration, + } = self; + write!( + f, + "ProtocolMetadataLoadSuccess(id={operation_id}, table_type={table_type}, \ + correlation_id={correlation_id:?}, load_type={load_type}, source={source}, \ + duration={duration:?})" + ) + } +} + +/// Reading protocol and metadata from the log failed. +#[derive(Debug, Clone)] +pub struct ProtocolMetadataLoadFailure { + pub operation_id: MetricId, + /// Opaque, caller-supplied id for joining this operation's metric events to the caller's + /// own request or operation id. + pub correlation_id: Option>, + pub table_type: TableType, + pub load_type: LogSegmentLoadType, +} + +impl fmt::Display for ProtocolMetadataLoadFailure { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "ProtocolMetadataLoadFailure(id={}, table_type={}, correlation_id={:?}, \ + load_type={})", + self.operation_id, self.table_type, self.correlation_id, self.load_type + ) + } +} + +// ==================================================================== +// SnapshotBuild +// ==================================================================== + +pub(crate) const SNAPSHOT_COMPLETED_SPAN: &str = "snap.build"; + +/// A snapshot was built successfully. +#[derive(Debug, Clone)] +pub struct SnapshotBuildSuccess { + // === Set on span creation === + pub operation_id: MetricId, + /// Opaque, caller-supplied id for joining this operation's metric events to the caller's + /// own request or operation id. + pub correlation_id: Option>, + pub table_type: TableType, + pub load_type: LogSegmentLoadType, + + // === Set during span lifetime === + pub version: u64, + + // === Set on span close === + pub duration: Duration, +} + +impl SnapshotBuildSuccess { + pub(crate) const SPAN_NAME: &'static str = SNAPSHOT_COMPLETED_SPAN; + + pub(crate) fn from_attrs(attrs: &Attributes<'_>) -> Self { + Self { + operation_id: MetricId::from_attrs(attrs), + table_type: TableType::from_catalog_managed(read_is_catalog_managed(attrs)), + correlation_id: correlation_id_from_attrs(attrs), + load_type: load_type_from_attrs(attrs), + version: 0, + duration: Duration::default(), + } + } + + pub(crate) fn record_u64(&mut self, name: &str, value: u64) -> Result<(), &'static str> { + match name { + "version" => self.version = value, + _ => return Err(Self::SPAN_NAME), + } + Ok(()) + } + + pub(crate) fn set_duration(&mut self, d: Duration) { + self.duration = d; + } +} + +impl fmt::Display for SnapshotBuildSuccess { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let Self { + operation_id, + table_type, + correlation_id, + load_type, + version, + duration, + } = self; + write!( + f, + "SnapshotBuildSuccess(id={operation_id}, table_type={table_type}, \ + correlation_id={correlation_id:?}, load_type={load_type}, version={version}, \ + duration={duration:?})" + ) + } +} + +// ==================================================================== +// SnapshotBuildFailure +// ==================================================================== + +/// Building a snapshot failed. +#[derive(Debug, Clone)] +pub struct SnapshotBuildFailure { + pub operation_id: MetricId, + /// Opaque, caller-supplied id for joining this operation's metric events to the caller's + /// own request or operation id. + pub correlation_id: Option>, + pub table_type: TableType, + pub load_type: LogSegmentLoadType, +} + +impl fmt::Display for SnapshotBuildFailure { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "SnapshotBuildFailure(id={}, table_type={}, correlation_id={:?}, load_type={})", + self.operation_id, self.table_type, self.correlation_id, self.load_type + ) + } +} + +// ==================================================================== +// TransactionCommit +// ==================================================================== + +pub(crate) const TRANSACTION_COMMIT_SPAN: &str = "txn.commit"; + +/// A transaction was committed successfully. +#[derive(Debug, Clone)] +pub struct TransactionCommitSuccess { + // === Set on span creation === + pub operation_id: MetricId, + /// Opaque, caller-supplied id for joining this operation's metric events to the caller's + /// own request or operation id. + pub correlation_id: Option>, + pub table_type: TableType, + pub commit_version: u64, + + // === Set during span lifetime === + pub num_add_files: u64, + pub num_remove_files: u64, + pub num_dv_updates: u64, + pub add_files_bytes: u64, + pub remove_files_bytes: u64, + pub is_blind_append: bool, + pub data_change: bool, + pub operation: Option, + /// Time assembling and validating the commit, before the committer call. + pub prepare_duration: Duration, + /// Time in the committer's `commit()` call. + pub committer_duration: Duration, + + // === Set on span close === + pub total_duration: Duration, +} + +impl TransactionCommitSuccess { + pub(crate) const SPAN_NAME: &'static str = TRANSACTION_COMMIT_SPAN; + + pub(crate) fn from_attrs(attrs: &Attributes<'_>) -> Self { + let mut v = TransactionCommitAttrs::default(); + attrs.record(&mut v); + Self { + operation_id: MetricId::from_attrs(attrs), + table_type: TableType::from_catalog_managed(read_is_catalog_managed(attrs)), + correlation_id: correlation_id_from_attrs(attrs), + commit_version: v.commit_version, + num_add_files: 0, + num_remove_files: 0, + num_dv_updates: 0, + add_files_bytes: 0, + remove_files_bytes: 0, + is_blind_append: false, + data_change: false, + operation: None, + prepare_duration: Duration::default(), + committer_duration: Duration::default(), + total_duration: Duration::default(), + } + } + + pub(crate) fn record_u64(&mut self, name: &str, value: u64) -> Result<(), &'static str> { + match name { + "num_add_files" => self.num_add_files = value, + "num_remove_files" => self.num_remove_files = value, + "num_dv_updates" => self.num_dv_updates = value, + "add_files_bytes" => self.add_files_bytes = value, + "remove_files_bytes" => self.remove_files_bytes = value, + "prepare_duration_ns" => self.prepare_duration = Duration::from_nanos(value), + "committer_duration_ns" => self.committer_duration = Duration::from_nanos(value), + _ => return Err(Self::SPAN_NAME), + } + Ok(()) + } + + pub(crate) fn record_bool(&mut self, name: &str, value: bool) -> Result<(), &'static str> { + match name { + "is_blind_append" => self.is_blind_append = value, + "data_change" => self.data_change = value, + _ => return Err(Self::SPAN_NAME), + } + Ok(()) + } + + pub(crate) fn record_str(&mut self, name: &str, value: &str) -> Result<(), &'static str> { + match name { + "operation" => self.operation = Some(value.to_string()), + _ => return Err(Self::SPAN_NAME), + } + Ok(()) + } + + pub(crate) fn set_duration(&mut self, d: Duration) { + self.total_duration = d; + } +} + +impl fmt::Display for TransactionCommitSuccess { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let Self { + operation_id, + table_type, + correlation_id, + commit_version, + num_add_files, + num_remove_files, + num_dv_updates, + add_files_bytes, + remove_files_bytes, + is_blind_append, + data_change, + operation, + prepare_duration, + committer_duration, + total_duration, + } = self; + write!( + f, + "TransactionCommitSuccess(id={operation_id}, table_type={table_type}, \ + correlation_id={correlation_id:?}, version={commit_version}, \ + total_duration={total_duration:?}, prepare={prepare_duration:?}, committer={committer_duration:?}, \ + add_files={num_add_files}, remove_files={num_remove_files}, dv_updates={num_dv_updates}, \ + add_bytes={add_files_bytes}, remove_bytes={remove_files_bytes}, \ + is_blind_append={is_blind_append}, data_change={data_change}, operation={operation:?})" + ) + } +} + +/// Why a transaction commit did not succeed. +/// +/// Serializes to its `snake_case` name for the `failure_reason` span field (e.g. +/// `RetryableIo` -> `"retryable_io"`). +#[derive(Debug, Clone, Copy, PartialEq, Eq, EnumString, StrumDisplay, AsRefStr, IntoStaticStr)] +#[strum(serialize_all = "snake_case")] +pub enum CommitFailureReason { + /// The commit conflicted with a concurrently committed version. + Conflict, + /// A retryable IO error occurred during the commit. + RetryableIo, + /// A terminal (non-retryable) error occurred. + Error, +} + +/// A transaction commit did not succeed; `reason` distinguishes conflict, retryable IO, and +/// terminal errors. +#[derive(Debug, Clone)] +pub struct TransactionCommitFailure { + pub operation_id: MetricId, + /// Opaque, caller-supplied id for joining this operation's metric events to the caller's + /// own request or operation id. + pub correlation_id: Option>, + pub table_type: TableType, + pub reason: CommitFailureReason, +} + +impl fmt::Display for TransactionCommitFailure { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let Self { + operation_id, + table_type, + correlation_id, + reason, + } = self; + write!( + f, + "TransactionCommitFailure(id={operation_id}, table_type={table_type}, \ + correlation_id={correlation_id:?}, reason={reason})" + ) + } +} + +#[derive(Default)] +struct TransactionCommitAttrs { + commit_version: u64, +} + +impl Visit for TransactionCommitAttrs { + fn record_u64(&mut self, field: &Field, value: u64) { + if field.name() == "commit_version" { + self.commit_version = value; } } + + fn record_debug(&mut self, _field: &Field, _value: &dyn fmt::Debug) {} +} + +// ==================================================================== +// DomainMetadataLoad +// ==================================================================== + +pub(crate) const DOMAIN_METADATA_LOADED_SPAN: &str = "snap.get_domain_metadata"; + +/// Emitted once per domain metadata load, whether served from the CRC cache (`from_cache`) or +/// from a log replay. Covers connector-issued loads of user domains and kernel-internal loads of +/// system (`delta.*`) domains such as clustering or row tracking. +#[derive(Debug, Clone)] +pub struct DomainMetadataLoadSuccess { + // === Set during span lifetime === + pub from_cache: bool, + pub num_domains_returned: u64, + + // === Set on span close === + pub duration: Duration, +} + +impl DomainMetadataLoadSuccess { + pub(crate) const SPAN_NAME: &'static str = DOMAIN_METADATA_LOADED_SPAN; + + pub(crate) fn from_attrs(_attrs: &Attributes<'_>) -> Self { + Self { + from_cache: false, + num_domains_returned: 0, + duration: Duration::default(), + } + } + + pub(crate) fn record_u64(&mut self, name: &str, value: u64) -> Result<(), &'static str> { + match name { + "num_domains_returned" => self.num_domains_returned = value, + _ => return Err(Self::SPAN_NAME), + } + Ok(()) + } + + pub(crate) fn record_bool(&mut self, name: &str, value: bool) -> Result<(), &'static str> { + match name { + "from_cache" => self.from_cache = value, + _ => return Err(Self::SPAN_NAME), + } + Ok(()) + } + + pub(crate) fn set_duration(&mut self, d: Duration) { + self.duration = d; + } +} + +impl fmt::Display for DomainMetadataLoadSuccess { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let Self { + from_cache, + num_domains_returned, + duration, + } = self; + write!( + f, + "DomainMetadataLoadSuccess(duration={duration:?}, from_cache={from_cache}, \ + num_domains_returned={num_domains_returned})" + ) + } +} + +// ==================================================================== +// SetTransactionLoad +// ==================================================================== + +pub(crate) const SET_TRANSACTION_LOADED_SPAN: &str = "snap.get_app_id_version"; + +/// Emitted once per `SetTransaction` (app id) load, whether served from the CRC cache +/// (`from_cache`) or from a log replay. `found` is true when the app id has a committed +/// transaction version, false when none exists or the existing one is expired. +#[derive(Debug, Clone)] +pub struct SetTransactionLoadSuccess { + // === Set during span lifetime === + pub from_cache: bool, + pub found: bool, + + // === Set on span close === + pub duration: Duration, +} + +impl SetTransactionLoadSuccess { + pub(crate) const SPAN_NAME: &'static str = SET_TRANSACTION_LOADED_SPAN; + + pub(crate) fn from_attrs(_attrs: &Attributes<'_>) -> Self { + Self { + from_cache: false, + found: false, + duration: Duration::default(), + } + } + + pub(crate) fn record_bool(&mut self, name: &str, value: bool) -> Result<(), &'static str> { + match name { + "from_cache" => self.from_cache = value, + "found" => self.found = value, + _ => return Err(Self::SPAN_NAME), + } + Ok(()) + } + + pub(crate) fn set_duration(&mut self, d: Duration) { + self.duration = d; + } +} + +impl fmt::Display for SetTransactionLoadSuccess { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let Self { + from_cache, + found, + duration, + } = self; + write!( + f, + "SetTransactionLoadSuccess(duration={duration:?}, from_cache={from_cache}, found={found})" + ) + } +} + +// ==================================================================== +// CrcRead +// ==================================================================== + +pub(crate) const CRC_READ_COMPLETED_SPAN: &str = "crc_read_completed"; + +/// A CRC file was read and parsed successfully. `bytes_read` is the raw byte count from storage. +#[derive(Debug, Clone)] +pub struct CrcReadSuccess { + // === Set during span lifetime === + pub bytes_read: u64, + + // === Set on span close === + pub duration: Duration, +} + +impl CrcReadSuccess { + pub(crate) const SPAN_NAME: &'static str = CRC_READ_COMPLETED_SPAN; + + pub(crate) fn from_attrs(_attrs: &Attributes<'_>) -> Self { + Self { + bytes_read: 0, + duration: Duration::default(), + } + } + + pub(crate) fn record_u64(&mut self, name: &str, value: u64) -> Result<(), &'static str> { + match name { + "bytes_read" => self.bytes_read = value, + _ => return Err(Self::SPAN_NAME), + } + Ok(()) + } + + pub(crate) fn set_duration(&mut self, d: Duration) { + self.duration = d; + } +} + +impl fmt::Display for CrcReadSuccess { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let Self { + duration, + bytes_read, + } = self; + write!( + f, + "CrcReadSuccess(duration={duration:?}, bytes={bytes_read})" + ) + } +} + +// ==================================================================== +// JsonReadCompleted +// ==================================================================== + +/// Emitted once per `JsonHandler::read_json_files` call when the returned iterator is fully +/// consumed or dropped. `bytes_read` is the sum of on-disk `FileMeta::size`, not the +/// deserialized payload size. +#[derive(Debug, Clone)] +pub struct JsonReadCompleted { + // === Set on span creation === + pub num_files: u64, + pub bytes_read: u64, +} + +impl JsonReadCompleted { + pub(crate) const SPAN_NAME: &'static str = "json_read_completed"; + + pub(crate) fn from_attrs(attrs: &Attributes<'_>) -> Self { + let mut v = FileReadAttrs::default(); + attrs.record(&mut v); + Self { + num_files: v.num_files, + bytes_read: v.bytes_read, + } + } +} + +impl fmt::Display for JsonReadCompleted { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let Self { + num_files, + bytes_read, + } = self; + write!( + f, + "JsonReadCompleted(files={num_files}, bytes={bytes_read})" + ) + } +} + +// ==================================================================== +// ParquetReadCompleted +// ==================================================================== + +/// Emitted once per `ParquetHandler::read_parquet_files` call when the returned iterator is +/// fully consumed or dropped. `bytes_read` is the sum of on-disk `FileMeta::size`, not the +/// deserialized payload size. +#[derive(Debug, Clone)] +pub struct ParquetReadCompleted { + // === Set on span creation === + pub num_files: u64, + pub bytes_read: u64, +} + +impl ParquetReadCompleted { + pub(crate) const SPAN_NAME: &'static str = "parquet_read_completed"; + + pub(crate) fn from_attrs(attrs: &Attributes<'_>) -> Self { + let mut v = FileReadAttrs::default(); + attrs.record(&mut v); + Self { + num_files: v.num_files, + bytes_read: v.bytes_read, + } + } +} + +impl fmt::Display for ParquetReadCompleted { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let Self { + num_files, + bytes_read, + } = self; + write!( + f, + "ParquetReadCompleted(files={num_files}, bytes={bytes_read})" + ) + } +} + +/// Shared attribute decoder for `JsonReadCompleted` and `ParquetReadCompleted`. +#[derive(Default)] +struct FileReadAttrs { + num_files: u64, + bytes_read: u64, +} + +impl Visit for FileReadAttrs { + fn record_u64(&mut self, field: &Field, value: u64) { + match field.name() { + "num_files" => self.num_files = value, + "bytes_read" => self.bytes_read = value, + _ => {} + } + } + fn record_debug(&mut self, _field: &Field, _value: &dyn fmt::Debug) {} +} + +// ==================================================================== +// ScanMetadataCompleted +// ==================================================================== + +/// Identifies which scan execution path produced a scan metadata metrics event. +/// +/// Serializes to the explicit `serialize` name on each variant for the `scan_type` span field +/// (e.g. `SequentialPhase` -> `"sequential"`). +#[derive(Debug, Clone, Copy, PartialEq, Eq, EnumString, StrumDisplay, AsRefStr, IntoStaticStr)] +pub enum ScanType { + /// Sequential phase of [`crate::scan::Scan::parallel_scan_metadata`]. + #[strum(serialize = "sequential")] + SequentialPhase, + /// Parallel phase of [`crate::scan::Scan::parallel_scan_metadata`]. + #[strum(serialize = "parallel")] + ParallelPhase, + /// Scan metadata from [`crate::scan::Scan::scan_metadata`]. + #[strum(serialize = "full")] + Full, +} + +impl ScanType { + fn parse_lenient(s: &str) -> Self { + Self::from_str(s).unwrap_or_else(|e| { + warn!("Invalid scan_type '{s}' on span: {e}. Using Full."); + Self::Full + }) + } +} + +// ==================================================================== +// SnapshotLoadMetricContext and shared span fields +// ==================================================================== + +/// The `is_catalog_managed` bool span field carried by every event that records it. It is the +/// confirmed protocol value, except on snapshot-load events, which use the requested mode because +/// the on-disk protocol is not known that early (see `SnapshotBuilder::build`). +pub(crate) const IS_CATALOG_MANAGED_FIELD: &str = "is_catalog_managed"; + +/// Whether a table is path-based or catalog-managed. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, serde::Serialize, serde::Deserialize)] +pub enum TableType { + /// Loaded without catalog involvement; commits live directly in the Delta log. + #[default] + PathBased, + /// Backed by a managing catalog. + CatalogManaged, +} + +impl TableType { + #[internal_api] + pub(crate) fn from_catalog_managed(is_catalog_managed: bool) -> Self { + if is_catalog_managed { + Self::CatalogManaged + } else { + Self::PathBased + } + } + + pub(crate) fn is_catalog_managed(self) -> bool { + self == Self::CatalogManaged + } +} + +impl fmt::Display for TableType { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(match self { + Self::PathBased => "path_based", + Self::CatalogManaged => "catalog_managed", + }) + } +} + +/// Operation-scoped values threaded through the snapshot-load chain to label its metric events. +#[derive(Debug, Clone)] +pub struct SnapshotLoadMetricContext { + pub(crate) operation_id: MetricId, + pub(crate) correlation_id: Option>, + pub(crate) is_catalog_managed: bool, + pub(crate) load_type: LogSegmentLoadType, +} + +#[cfg(test)] +impl SnapshotLoadMetricContext { + /// A context seeded with a nil `operation_id` for tests that exercise the load chain without + /// asserting on the id. + pub(crate) fn for_test() -> Self { + Self { + operation_id: MetricId::nil(), + correlation_id: None, + is_catalog_managed: false, + load_type: LogSegmentLoadType::default(), + } + } +} + +/// Decode the `operation_id` debug-string span field into a [`MetricId`]. Returns `None` when the +/// field is not `operation_id`; returns the nil id (with a warning) when the value is malformed. +/// Shared by the self-visiting event decoders. +pub(crate) fn record_operation_id( + field: &Field, + value: &dyn fmt::Debug, + span_name: &str, +) -> Option { + (field.name() == "operation_id").then(|| { + let s = format!("{value:?}"); + Uuid::from_str(&s).map(MetricId).unwrap_or_else(|e| { + warn!("Invalid uuid '{s}' on {span_name}: {e}. Using nil."); + MetricId::nil() + }) + }) +} + +pub(crate) fn read_is_catalog_managed(attrs: &Attributes<'_>) -> bool { + #[derive(Default)] + struct V(bool); + impl Visit for V { + fn record_bool(&mut self, field: &Field, value: bool) { + if field.name() == IS_CATALOG_MANAGED_FIELD { + self.0 = value; + } + } + fn record_debug(&mut self, _field: &Field, _value: &dyn fmt::Debug) {} + } + let mut v = V::default(); + attrs.record(&mut v); + v.0 +} + +/// The `correlation_id` string span field carried by every event that records it. Empty means the +/// caller did not supply one. +pub(crate) const CORRELATION_ID_FIELD: &str = "correlation_id"; + +/// Extract the optional caller-supplied correlation id from span attributes; empty or absent +/// yields `None`. +pub(crate) fn correlation_id_from_attrs(attrs: &Attributes<'_>) -> Option> { + #[derive(Default)] + struct CorrelationIdVisitor(Option>); + impl Visit for CorrelationIdVisitor { + fn record_str(&mut self, field: &Field, value: &str) { + if field.name() == CORRELATION_ID_FIELD && !value.is_empty() { + self.0 = Some(value.into()); + } + } + fn record_debug(&mut self, _field: &Field, _value: &dyn fmt::Debug) {} + } + let mut v = CorrelationIdVisitor::default(); + attrs.record(&mut v); + v.0 +} + +pub(crate) fn load_type_from_attrs(attrs: &Attributes<'_>) -> LogSegmentLoadType { + #[derive(Default)] + struct V(LogSegmentLoadType); + impl Visit for V { + fn record_str(&mut self, field: &Field, value: &str) { + if field.name() == "load_type" { + self.0 = LogSegmentLoadType::parse_or_unknown(value); + } + } + fn record_debug(&mut self, _field: &Field, _value: &dyn fmt::Debug) {} + } + let mut v = V::default(); + attrs.record(&mut v); + v.0 +} + +/// A `parallel_scan_metadata` scan emits **two** events (one per phase) sharing the same +/// `operation_id`; `scan_metadata` emits one event with [`ScanType::Full`]. +#[derive(Debug, Clone)] +pub struct ScanMetadataCompleted { + // === Set on span creation === + /// Unique ID to correlate this scan with other events. + pub operation_id: MetricId, + /// Opaque, caller-supplied id for joining this operation's metric events to the caller's + /// own request or operation id. + pub correlation_id: Option>, + /// Whether the scanned table is path-based or catalog-managed. + pub table_type: TableType, + /// Which scan execution path produced this event. + pub scan_type: ScanType, + /// Wall-clock time from scan start to iterator exhaustion. + pub duration: Duration, + /// Add files that entered deduplication. This normally excludes add files filtered by data + /// skipping. During parse-error fallback, deduplication runs first, so add files filtered by + /// retry-time data skipping are included. See + /// [`LogReplayProcessor::process_actions_batch`](crate::log_replay::LogReplayProcessor::process_actions_batch). + pub num_add_files_seen: u64, + /// Add files that survived log replay (the files the connector reads). + pub num_active_add_files: u64, + /// Size in bytes of the files that survived log replay (files to read). + pub active_add_files_bytes: u64, + /// Remove files seen (from delta/commit files only). + pub num_remove_files_seen: u64, + /// Non-file actions seen (protocol, metadata, etc.). + pub num_non_file_actions: u64, + /// Files filtered by predicates (data skipping + partition pruning). + pub num_predicate_filtered: u64, + /// Peak size of the deduplication hash set. + pub peak_hash_set_size: usize, + /// Time spent in the deduplication visitor. + pub dedup_visitor_time: Duration, + /// Time spent evaluating predicates. + pub predicate_eval_time: Duration, +} + +impl ScanMetadataCompleted { + pub(crate) const SPAN_NAME: &'static str = "scan.metadata_completed"; + + pub(crate) fn from_attrs(attrs: &Attributes<'_>) -> Self { + let mut v = ScanMetadataCompletedAttrs::default(); + attrs.record(&mut v); + Self { + operation_id: MetricId(v.operation_id), + table_type: TableType::from_catalog_managed(v.is_catalog_managed), + correlation_id: v.correlation_id, + scan_type: ScanType::parse_lenient(&v.scan_type), + duration: Duration::from_nanos(v.duration_ns), + num_add_files_seen: v.num_add_files_seen, + num_active_add_files: v.num_active_add_files, + active_add_files_bytes: v.active_add_files_bytes, + num_remove_files_seen: v.num_remove_files_seen, + num_non_file_actions: v.num_non_file_actions, + num_predicate_filtered: v.num_predicate_filtered, + peak_hash_set_size: v.peak_hash_set_size as usize, + dedup_visitor_time: Duration::from_nanos(v.dedup_visitor_time_ns), + predicate_eval_time: Duration::from_nanos(v.predicate_eval_time_ns), + } + } +} + +impl fmt::Display for ScanMetadataCompleted { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let Self { + operation_id, + table_type, + correlation_id, + scan_type, + duration, + num_add_files_seen, + num_active_add_files, + active_add_files_bytes, + num_remove_files_seen, + num_non_file_actions, + num_predicate_filtered, + peak_hash_set_size, + dedup_visitor_time, + predicate_eval_time, + } = self; + write!( + f, + "ScanMetadataCompleted(id={operation_id}, table_type={table_type}, \ + correlation_id={correlation_id:?}, scan_type={scan_type}, duration={duration:?}, \ + add_files_seen={num_add_files_seen}, active_add_files={num_active_add_files}, \ + active_add_files_bytes={active_add_files_bytes}, \ + remove_files_seen={num_remove_files_seen}, non_file_actions={num_non_file_actions}, \ + predicate_filtered={num_predicate_filtered}, peak_hash_set_size={peak_hash_set_size}, \ + dedup_visitor_time={dedup_visitor_time:?}, predicate_eval_time={predicate_eval_time:?})" + ) + } +} + +#[derive(Default)] +struct ScanMetadataCompletedAttrs { + operation_id: Uuid, + is_catalog_managed: bool, + correlation_id: Option>, + scan_type: String, + duration_ns: u64, + num_add_files_seen: u64, + num_active_add_files: u64, + active_add_files_bytes: u64, + num_remove_files_seen: u64, + num_non_file_actions: u64, + num_predicate_filtered: u64, + peak_hash_set_size: u64, + dedup_visitor_time_ns: u64, + predicate_eval_time_ns: u64, +} + +impl Visit for ScanMetadataCompletedAttrs { + fn record_bool(&mut self, field: &Field, value: bool) { + if field.name() == IS_CATALOG_MANAGED_FIELD { + self.is_catalog_managed = value; + } + } + + fn record_str(&mut self, field: &Field, value: &str) { + if field.name() == CORRELATION_ID_FIELD && !value.is_empty() { + self.correlation_id = Some(value.into()); + } + } + + fn record_u64(&mut self, field: &Field, value: u64) { + match field.name() { + "duration_ns" => self.duration_ns = value, + "num_add_files_seen" => self.num_add_files_seen = value, + "num_active_add_files" => self.num_active_add_files = value, + "active_add_files_bytes" => self.active_add_files_bytes = value, + "num_remove_files_seen" => self.num_remove_files_seen = value, + "num_non_file_actions" => self.num_non_file_actions = value, + "num_predicate_filtered" => self.num_predicate_filtered = value, + "peak_hash_set_size" => self.peak_hash_set_size = value, + "dedup_visitor_time_ns" => self.dedup_visitor_time_ns = value, + "predicate_eval_time_ns" => self.predicate_eval_time_ns = value, + _ => {} + } + } + + fn record_debug(&mut self, field: &Field, value: &dyn fmt::Debug) { + if let Some(id) = record_operation_id(field, value, ScanMetadataCompleted::SPAN_NAME) { + self.operation_id = id.0; + } else if field.name() == "scan_type" { + self.scan_type = format!("{value:?}"); + } + } +} + +// ==================================================================== +// Storage events (List / Read / Copy) +// ==================================================================== +// +// All three storage events share the `STORAGE_SPAN` span name and distinguish themselves via +// a `name=` field carrying one of `::NAME`. The shared [`storage_metric_from_attrs`] +// helper inspects the `name` value and constructs the matching variant. + +pub(crate) const STORAGE_SPAN: &str = "storage"; + +/// Build the appropriate storage `MetricEvent` from the span attributes. Returns `None` if the +/// `name` field is missing or unknown. +pub(crate) fn storage_metric_from_attrs(attrs: &Attributes<'_>) -> Option { + let mut v = StorageAttrs::default(); + attrs.record(&mut v); + let duration = Duration::from_nanos(v.duration_ns); + match v.kind { + StorageKind::List => Some(MetricEvent::StorageListCompleted(StorageListCompleted { + duration, + num_files: v.num_files, + })), + StorageKind::Read => Some(MetricEvent::StorageReadCompleted(StorageReadCompleted { + duration, + num_files: v.num_files, + bytes_read: v.bytes_read, + })), + StorageKind::Copy => Some(MetricEvent::StorageCopyCompleted(StorageCopyCompleted { + duration, + })), + StorageKind::Unknown => None, + } +} + +#[derive(Default)] +enum StorageKind { + #[default] + Unknown, + List, + Read, + Copy, +} + +#[derive(Default)] +struct StorageAttrs { + kind: StorageKind, + num_files: u64, + bytes_read: u64, + duration_ns: u64, +} + +impl Visit for StorageAttrs { + fn record_str(&mut self, field: &Field, value: &str) { + if field.name() == "name" { + self.kind = match value { + StorageListCompleted::NAME => StorageKind::List, + StorageReadCompleted::NAME => StorageKind::Read, + StorageCopyCompleted::NAME => StorageKind::Copy, + _ => { + warn!("Storage span with unknown name: {value}"); + StorageKind::Unknown + } + }; + } + } + + fn record_u64(&mut self, field: &Field, value: u64) { + match field.name() { + "num_files" => self.num_files = value, + "bytes_read" => self.bytes_read = value, + "duration_ns" => self.duration_ns = value, + _ => {} + } + } + + fn record_debug(&mut self, _field: &Field, _value: &dyn fmt::Debug) {} +} + +// ============================ +// StorageListCompleted +// ============================ + +/// A storage list operation completed. +#[derive(Debug, Clone)] +pub struct StorageListCompleted { + // === Set on span creation === + pub duration: Duration, + pub num_files: u64, +} + +impl StorageListCompleted { + pub(crate) const NAME: &'static str = "list_completed"; +} + +impl fmt::Display for StorageListCompleted { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let Self { + duration, + num_files, + } = self; + write!( + f, + "StorageListCompleted(duration={duration:?}, files={num_files})" + ) + } +} + +// ============================ +// StorageReadCompleted +// ============================ + +/// A storage read operation completed. +#[derive(Debug, Clone)] +pub struct StorageReadCompleted { + // === Set on span creation === + pub duration: Duration, + pub num_files: u64, + pub bytes_read: u64, +} + +impl StorageReadCompleted { + pub(crate) const NAME: &'static str = "read_completed"; +} + +impl fmt::Display for StorageReadCompleted { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let Self { + duration, + num_files, + bytes_read, + } = self; + write!( + f, + "StorageReadCompleted(duration={duration:?}, files={num_files}, bytes={bytes_read})" + ) + } +} + +// ============================ +// StorageCopyCompleted +// ============================ + +/// A storage copy or rename operation completed. +#[derive(Debug, Clone)] +pub struct StorageCopyCompleted { + // === Set on span creation === + pub duration: Duration, +} + +impl StorageCopyCompleted { + pub(crate) const NAME: &'static str = "copy_completed"; +} + +impl fmt::Display for StorageCopyCompleted { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let Self { duration } = self; + write!(f, "StorageCopyCompleted(duration={duration:?})") + } +} + +// ==================================================================== +// emit_* helpers +// ==================================================================== +// +// Fire a tracing span carrying a pre-built event payload. Used by the scan metadata emission +// site (which constructs the event up front) and by connector-side `JsonHandler` / +// `ParquetHandler` implementations that want to report read metrics. + +/// Emit a [`MetricEvent::JsonReadCompleted`] via a tracing span. +/// +/// Call once per [`crate::JsonHandler::read_json_files`] invocation, at iterator exhaustion or +/// drop. +pub fn emit_json_read_completed(num_files: u64, bytes_read: u64) { + let _span = tracing::span!( + tracing::Level::INFO, + JsonReadCompleted::SPAN_NAME, + report = tracing::field::Empty, + num_files, + bytes_read, + ); +} + +/// Emit a [`MetricEvent::ParquetReadCompleted`] via a tracing span. +/// +/// Call once per [`crate::ParquetHandler::read_parquet_files`] invocation, at iterator exhaustion +/// or drop. +pub fn emit_parquet_read_completed(num_files: u64, bytes_read: u64) { + let _span = tracing::span!( + tracing::Level::INFO, + ParquetReadCompleted::SPAN_NAME, + report = tracing::field::Empty, + num_files, + bytes_read, + ); +} + +/// Emit a [`MetricEvent::ScanMetadataCompleted`] via a tracing span. Call when the scan metadata +/// iterator is exhausted or dropped. +pub(crate) fn emit_scan_metadata_completed(e: &ScanMetadataCompleted) { + let _span = tracing::span!( + tracing::Level::INFO, + ScanMetadataCompleted::SPAN_NAME, + report = tracing::field::Empty, + operation_id = %e.operation_id, + is_catalog_managed = e.table_type.is_catalog_managed(), + correlation_id = e.correlation_id.as_deref().unwrap_or(""), + scan_type = %e.scan_type, + duration_ns = e.duration.as_nanos() as u64, + num_add_files_seen = e.num_add_files_seen, + num_active_add_files = e.num_active_add_files, + active_add_files_bytes = e.active_add_files_bytes, + num_remove_files_seen = e.num_remove_files_seen, + num_non_file_actions = e.num_non_file_actions, + num_predicate_filtered = e.num_predicate_filtered, + peak_hash_set_size = e.peak_hash_set_size as u64, + dedup_visitor_time_ns = e.dedup_visitor_time.as_nanos() as u64, + predicate_eval_time_ns = e.predicate_eval_time.as_nanos() as u64, + ); +} + +/// Emit a [`MetricEvent::LogSegmentLoadSuccess`] for `segment`, reporting its file counts and CRC +/// staleness. Call once per log-segment load, on success; callers time the load and pass +/// `duration`. +pub(crate) fn emit_log_segment_load( + ctx: &SnapshotLoadMetricContext, + segment: &LogSegment, + duration: Duration, +) { + tracing::span!( + tracing::Level::INFO, + LogSegmentLoadSuccess::SPAN_NAME, + report = tracing::field::Empty, + operation_id = %ctx.operation_id, + is_catalog_managed = ctx.is_catalog_managed, + correlation_id = ctx.correlation_id.as_deref().unwrap_or(""), + load_type = ctx.load_type.as_ref(), + num_commit_files = segment.listed.ascending_commit_files.len() as u64, + num_checkpoint_files = segment.listed.checkpoint_parts.len() as u64, + num_compaction_files = segment.listed.ascending_compaction_files.len() as u64, + crc_versions_behind = + LogSegmentLoadSuccess::encode_crc_versions_behind(segment.crc_versions_behind()), + duration_ns = duration.as_nanos() as u64, + ); +} + +/// Emit a [`MetricEvent::LogSegmentLoadFailure`]. Call once per log-segment load, on failure. +pub(crate) fn emit_log_segment_load_failure(ctx: &SnapshotLoadMetricContext) { + let span = tracing::span!( + tracing::Level::INFO, + LogSegmentLoadSuccess::SPAN_NAME, + report = tracing::field::Empty, + operation_id = %ctx.operation_id, + is_catalog_managed = ctx.is_catalog_managed, + correlation_id = ctx.correlation_id.as_deref().unwrap_or(""), + load_type = ctx.load_type.as_ref(), + ); + let _enter = span.enter(); + tracing::info!(error = tracing::field::debug("log segment load failed")); +} + +/// Emit a [`MetricEvent::ProtocolMetadataLoadSuccess`]. Call once per snapshot load on the P&M +/// resolution path, on success. +pub(crate) fn emit_protocol_metadata_load( + ctx: &SnapshotLoadMetricContext, + source: ProtocolMetadataSource, + duration: Duration, +) { + let _span = tracing::span!( + tracing::Level::INFO, + ProtocolMetadataLoadSuccess::SPAN_NAME, + report = tracing::field::Empty, + operation_id = %ctx.operation_id, + is_catalog_managed = ctx.is_catalog_managed, + correlation_id = ctx.correlation_id.as_deref().unwrap_or(""), + load_type = ctx.load_type.as_ref(), + pm_source = source.as_ref(), + duration_ns = duration.as_nanos() as u64, + ); +} + +/// Emit a [`MetricEvent::ProtocolMetadataLoadFailure`]. Call once per snapshot load on the P&M +/// resolution path, on failure. +pub(crate) fn emit_protocol_metadata_load_failure(ctx: &SnapshotLoadMetricContext) { + let span = tracing::span!( + tracing::Level::INFO, + ProtocolMetadataLoadSuccess::SPAN_NAME, + report = tracing::field::Empty, + operation_id = %ctx.operation_id, + is_catalog_managed = ctx.is_catalog_managed, + correlation_id = ctx.correlation_id.as_deref().unwrap_or(""), + load_type = ctx.load_type.as_ref(), + ); + let _enter = span.enter(); + tracing::info!(error = tracing::field::debug("protocol/metadata load failed")); +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + + fn commit_success(operation_id: MetricId) -> TransactionCommitSuccess { + TransactionCommitSuccess { + operation_id, + table_type: TableType::PathBased, + correlation_id: None, + commit_version: 1, + num_add_files: 0, + num_remove_files: 0, + num_dv_updates: 0, + add_files_bytes: 0, + remove_files_bytes: 0, + is_blind_append: false, + data_change: false, + operation: None, + prepare_duration: Duration::default(), + committer_duration: Duration::default(), + total_duration: Duration::default(), + } + } + + #[rstest] + #[case::conflict("conflict", CommitFailureReason::Conflict)] + #[case::retryable_io("retryable_io", CommitFailureReason::RetryableIo)] + #[case::error("error", CommitFailureReason::Error)] + #[case::unknown_defaults_to_error("totally_unknown", CommitFailureReason::Error)] + fn record_str_failure_reason_flips_to_expected_reason( + #[case] value: &str, + #[case] expected: CommitFailureReason, + ) { + let id = MetricId::new(); + let mut event = MetricEvent::TransactionCommitSuccess(commit_success(id)); + event.record_str("failure_reason", value).unwrap(); + let MetricEvent::TransactionCommitFailure(failure) = event else { + panic!("expected TransactionCommitFailure"); + }; + assert_eq!(failure.operation_id, id); + assert_eq!(failure.reason, expected); + } + + #[test] + fn record_str_operation_sets_field_without_flipping() { + let mut event = MetricEvent::TransactionCommitSuccess(commit_success(MetricId::new())); + event.record_str("operation", "WRITE").unwrap(); + let MetricEvent::TransactionCommitSuccess(success) = event else { + panic!("expected TransactionCommitSuccess"); + }; + assert_eq!(success.operation.as_deref(), Some("WRITE")); + } + + #[test] + fn record_u64_num_dv_updates_sets_field() { + let mut event = MetricEvent::TransactionCommitSuccess(commit_success(MetricId::new())); + event.record_u64("num_dv_updates", 7).unwrap(); + let MetricEvent::TransactionCommitSuccess(success) = event else { + panic!("expected TransactionCommitSuccess"); + }; + assert_eq!(success.num_dv_updates, 7); + } + + #[rstest] + #[case::conflict(CommitFailureReason::Conflict, "conflict")] + #[case::retryable_io(CommitFailureReason::RetryableIo, "retryable_io")] + #[case::error(CommitFailureReason::Error, "error")] + fn commit_failure_reason_serializes_to_wire_name_and_parses_back( + #[case] reason: CommitFailureReason, + #[case] wire: &str, + ) { + let serialized: &'static str = reason.into(); + assert_eq!(serialized, wire); + assert_eq!(CommitFailureReason::from_str(wire).unwrap(), reason); + } + + #[rstest] + #[case::sequential(ScanType::SequentialPhase, "sequential")] + #[case::parallel(ScanType::ParallelPhase, "parallel")] + #[case::full(ScanType::Full, "full")] + fn scan_type_serializes_to_wire_name_and_parses_back( + #[case] scan_type: ScanType, + #[case] wire: &str, + ) { + let serialized: &'static str = scan_type.into(); + assert_eq!(serialized, wire); + assert_eq!(ScanType::from_str(wire).unwrap(), scan_type); + } + + #[rstest] + #[case::known("parallel", ScanType::ParallelPhase)] + #[case::unknown_defaults_to_full("totally_unknown", ScanType::Full)] + fn scan_type_parse_lenient_maps_unknown_to_full( + #[case] value: &str, + #[case] expected: ScanType, + ) { + assert_eq!(ScanType::parse_lenient(value), expected); + } + + #[rstest] + #[case::full(LogSegmentLoadType::Full, "full")] + #[case::incremental(LogSegmentLoadType::Incremental, "incremental")] + #[case::unknown(LogSegmentLoadType::Unknown, "unknown")] + fn log_segment_load_type_serializes_to_wire_name_and_parses_back( + #[case] load_type: LogSegmentLoadType, + #[case] wire: &str, + ) { + let serialized: &'static str = load_type.into(); + assert_eq!(serialized, wire); + assert_eq!(LogSegmentLoadType::from_str(wire).unwrap(), load_type); + } + + #[rstest] + #[case::known("incremental", LogSegmentLoadType::Incremental)] + #[case::empty_maps_to_unknown("", LogSegmentLoadType::Unknown)] + #[case::unrecognized_maps_to_unknown("totally_unknown", LogSegmentLoadType::Unknown)] + fn log_segment_load_type_parse_or_unknown( + #[case] value: &str, + #[case] expected: LogSegmentLoadType, + ) { + assert_eq!(LogSegmentLoadType::parse_or_unknown(value), expected); + } + + #[rstest] + #[case::none(None)] + #[case::zero(Some(0))] + #[case::some(Some(7))] + fn crc_versions_behind_sentinel_round_trips(#[case] v: Option) { + let encoded = LogSegmentLoadSuccess::encode_crc_versions_behind(v); + assert_eq!( + LogSegmentLoadSuccess::decode_crc_versions_behind(encoded), + v + ); + } + + #[rstest] + #[case::crc_at_target(ProtocolMetadataSource::CrcAtTarget, "crc_at_target")] + #[case::crc_advanced(ProtocolMetadataSource::CrcAdvancedByReplay, "crc_advanced_by_replay")] + #[case::crc_seeded( + ProtocolMetadataSource::CrcSeededPmOnlyReplay, + "crc_seeded_pm_only_replay" + )] + #[case::full_replay(ProtocolMetadataSource::FullReplay, "full_replay")] + #[case::unknown(ProtocolMetadataSource::Unknown, "unknown")] + fn protocol_metadata_source_serializes_to_wire_name_and_parses_back( + #[case] source: ProtocolMetadataSource, + #[case] wire: &str, + ) { + let serialized: &'static str = source.into(); + assert_eq!(serialized, wire); + assert_eq!(ProtocolMetadataSource::from_str(wire).unwrap(), source); + } + + #[rstest] + #[case::known("crc_at_target", ProtocolMetadataSource::CrcAtTarget)] + #[case::empty_maps_to_unknown("", ProtocolMetadataSource::Unknown)] + #[case::unrecognized_maps_to_unknown("totally_unknown", ProtocolMetadataSource::Unknown)] + fn protocol_metadata_source_parse_or_unknown( + #[case] value: &str, + #[case] expected: ProtocolMetadataSource, + ) { + assert_eq!(ProtocolMetadataSource::parse_or_unknown(value), expected); + } + + #[test] + fn into_failure_maps_commit_success_to_error_reason() { + let id = MetricId::new(); + let failure = MetricEvent::TransactionCommitSuccess(commit_success(id)).into_failure(); + let MetricEvent::TransactionCommitFailure(failure) = failure else { + panic!("expected TransactionCommitFailure"); + }; + assert_eq!(failure.operation_id, id); + assert_eq!(failure.reason, CommitFailureReason::Error); + } + + #[test] + fn record_str_failure_reason_flip_preserves_correlation_id() { + let mut success = commit_success(MetricId::new()); + success.correlation_id = Some("commit-req-1".into()); + let mut event = MetricEvent::TransactionCommitSuccess(success); + event.record_str("failure_reason", "conflict").unwrap(); + let MetricEvent::TransactionCommitFailure(failure) = event else { + panic!("expected TransactionCommitFailure"); + }; + assert_eq!(failure.correlation_id.as_deref(), Some("commit-req-1")); + } + + #[test] + fn into_failure_preserves_correlation_id() { + let mut success = commit_success(MetricId::new()); + success.correlation_id = Some("commit-req-2".into()); + let MetricEvent::TransactionCommitFailure(failure) = + MetricEvent::TransactionCommitSuccess(success).into_failure() + else { + panic!("expected TransactionCommitFailure"); + }; + assert_eq!(failure.correlation_id.as_deref(), Some("commit-req-2")); + + let snapshot = SnapshotBuildSuccess { + operation_id: MetricId::new(), + table_type: TableType::PathBased, + correlation_id: Some("snap-req-3".into()), + load_type: LogSegmentLoadType::Incremental, + version: 0, + duration: Duration::default(), + }; + let MetricEvent::SnapshotBuildFailure(failure) = + MetricEvent::SnapshotBuildSuccess(snapshot).into_failure() + else { + panic!("expected SnapshotBuildFailure"); + }; + assert_eq!(failure.correlation_id.as_deref(), Some("snap-req-3")); + assert_eq!(failure.load_type, LogSegmentLoadType::Incremental); + } } diff --git a/kernel/src/metrics/metered_engine.rs b/kernel/src/metrics/metered_engine.rs new file mode 100644 index 0000000000..262d259646 --- /dev/null +++ b/kernel/src/metrics/metered_engine.rs @@ -0,0 +1,223 @@ +//! [`MeteredDeltaEngine`] wraps any [`Engine`] so its `storage_handler`, +//! `json_handler`, and `parquet_handler` emit the kernel's standard handler-completion +//! tracing spans. `evaluation_handler` passes through unchanged. +//! +//! ```ignore +//! let inner: Arc = Arc::new(MyEngine::build()?); +//! let engine: Arc = Arc::new(MeteredDeltaEngine::new(inner)); +//! ``` + +use std::sync::Arc; + +use crate::metrics::{MeteredJsonHandler, MeteredParquetHandler, MeteredStorageHandler}; +use crate::{Engine, EvaluationHandler, JsonHandler, ParquetHandler, StorageHandler}; + +/// Decorator over any [`Engine`] that meters its storage, JSON, and Parquet handlers. +/// See module docs. +pub struct MeteredDeltaEngine { + inner: Arc, + storage: Arc, + json: Arc, + parquet: Arc, +} + +impl MeteredDeltaEngine { + /// Wrap `inner`. Debug-asserts that none of `inner`'s handlers are already metered + /// wrappers, so the resulting engine emits each span exactly once. + /// + /// The check is shallow: it inspects the immediate concrete type returned by each + /// handler accessor and does not walk intermediate wrapper types. Wrapping a metered + /// handler behind a non-metered wrapper before re-wrapping (e.g. + /// `Metered(Foo(Metered(...)))`) silently double-counts. + pub fn new(inner: Arc) -> Self { + let inner_storage = inner.storage_handler(); + debug_assert!( + !inner_storage.any_ref().is::(), + "MeteredDeltaEngine wraps an engine whose storage_handler is already a \ + MeteredStorageHandler; remove the outer wrap to avoid double-counting metrics", + ); + let inner_json = inner.json_handler(); + debug_assert!( + !inner_json.any_ref().is::(), + "MeteredDeltaEngine wraps an engine whose json_handler is already a \ + MeteredJsonHandler; remove the outer wrap to avoid double-counting metrics", + ); + let inner_parquet = inner.parquet_handler(); + debug_assert!( + !inner_parquet.any_ref().is::(), + "MeteredDeltaEngine wraps an engine whose parquet_handler is already a \ + MeteredParquetHandler; remove the outer wrap to avoid double-counting metrics", + ); + let storage = Arc::new(MeteredStorageHandler::new(inner_storage)); + let json = Arc::new(MeteredJsonHandler::new(inner_json)); + let parquet = Arc::new(MeteredParquetHandler::new(inner_parquet)); + Self { + inner, + storage, + json, + parquet, + } + } +} + +impl std::fmt::Debug for MeteredDeltaEngine { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("MeteredDeltaEngine").finish_non_exhaustive() + } +} + +impl Engine for MeteredDeltaEngine { + fn evaluation_handler(&self) -> Arc { + self.inner.evaluation_handler() + } + + fn storage_handler(&self) -> Arc { + Arc::clone(&self.storage) + } + + fn json_handler(&self) -> Arc { + Arc::clone(&self.json) + } + + fn parquet_handler(&self) -> Arc { + Arc::clone(&self.parquet) + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use bytes::Bytes; + use url::Url; + + use super::*; + use crate::engine::sync::SyncEngine; + use crate::engine::test_delegating::DelegatingEngine; + use crate::metrics::MetricEvent; + use crate::unit_test_utils::{install_thread_local_metrics_reporter, CapturingReporter}; + use crate::{DeltaResult, FileMeta, FileSlice}; + + /// Minimal storage handler used in tests: returns N preconfigured FileMeta items. + #[derive(Debug)] + struct StubStorageHandler { + list_results: Vec, + } + + impl StorageHandler for StubStorageHandler { + fn list_from( + &self, + _path: &Url, + ) -> DeltaResult>>> { + let results: Vec<_> = self.list_results.iter().cloned().map(Ok).collect(); + Ok(Box::new(results.into_iter())) + } + fn read_files( + &self, + _files: Vec, + ) -> DeltaResult>>> { + Ok(Box::new(std::iter::empty())) + } + fn copy_atomic(&self, _src: &Url, _dest: &Url) -> DeltaResult<()> { + Ok(()) + } + fn put(&self, _path: &Url, _data: Bytes, _overwrite: bool) -> DeltaResult<()> { + Ok(()) + } + fn head(&self, _path: &Url) -> DeltaResult { + unreachable!("not exercised") + } + fn delete(&self, _path: &Url) -> DeltaResult<()> { + Ok(()) + } + } + + fn stub_engine() -> DelegatingEngine { + DelegatingEngine::new(Arc::new(SyncEngine::new())).with_storage_handler(Arc::new( + StubStorageHandler { + list_results: vec![ + FileMeta { + location: Url::parse("memory:///_delta_log/00000000000000000000.json") + .unwrap(), + last_modified: 0, + size: 0, + }, + FileMeta { + location: Url::parse("memory:///_delta_log/00000000000000000001.json") + .unwrap(), + last_modified: 0, + size: 0, + }, + ], + }, + )) + } + + #[test] + fn storage_handler_emits_metered_spans() { + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(Arc::clone(&reporter) as _); + + let engine = MeteredDeltaEngine::new(Arc::new(stub_engine())); + let url = Url::parse("memory:///_delta_log/").unwrap(); + let iter = engine.storage_handler().list_from(&url).unwrap(); + let _: Vec<_> = iter.collect(); + + let listed = reporter + .events() + .iter() + .find(|e| matches!(e, MetricEvent::StorageListCompleted(_))) + .cloned() + .expect("expected StorageListCompleted via metering wrapper"); + let MetricEvent::StorageListCompleted(e) = listed else { + unreachable!(); + }; + assert_eq!(e.num_files, 2); + } + + #[test] + fn evaluation_handler_passes_through() { + let inner: Arc = Arc::new(stub_engine()); + let inner_eval = inner.evaluation_handler(); + + let engine = MeteredDeltaEngine::new(inner); + assert!(Arc::ptr_eq(&inner_eval, &engine.evaluation_handler())); + } + + #[test] + fn json_and_parquet_handlers_are_metered() { + let engine = MeteredDeltaEngine::new(Arc::new(stub_engine())); + assert!(engine.json_handler().any_ref().is::()); + assert!(engine + .parquet_handler() + .any_ref() + .is::()); + } + + #[test] + #[should_panic(expected = "storage_handler is already a MeteredStorageHandler")] + fn new_panics_when_inner_storage_already_metered() { + let inner = stub_engine(); + let metered = Arc::new(MeteredStorageHandler::new(inner.storage_handler())); + let engine = DelegatingEngine::new(Arc::new(inner)).with_storage_handler(metered); + let _ = MeteredDeltaEngine::new(Arc::new(engine)); + } + + #[test] + #[should_panic(expected = "json_handler is already a MeteredJsonHandler")] + fn new_panics_when_inner_json_already_metered() { + let inner = stub_engine(); + let metered = Arc::new(MeteredJsonHandler::new(inner.json_handler())); + let engine = DelegatingEngine::new(Arc::new(inner)).with_json_handler(metered); + let _ = MeteredDeltaEngine::new(Arc::new(engine)); + } + + #[test] + #[should_panic(expected = "parquet_handler is already a MeteredParquetHandler")] + fn new_panics_when_inner_parquet_already_metered() { + let inner = stub_engine(); + let metered = Arc::new(MeteredParquetHandler::new(inner.parquet_handler())); + let engine = DelegatingEngine::new(Arc::new(inner)).with_parquet_handler(metered); + let _ = MeteredDeltaEngine::new(Arc::new(engine)); + } +} diff --git a/kernel/src/metrics/metered_json.rs b/kernel/src/metrics/metered_json.rs new file mode 100644 index 0000000000..7872f82274 --- /dev/null +++ b/kernel/src/metrics/metered_json.rs @@ -0,0 +1,294 @@ +//! [`MeteredJsonHandler`] wraps any [`JsonHandler`] so its `read_json_files` emits the +//! kernel's standard `JsonReadCompleted` span, carrying `(num_files, bytes_read)` exactly +//! once when the returned iterator is exhausted or dropped. `parse_json` and +//! `write_json_file` pass through. + +use std::sync::Arc; + +use crate::metrics::events::emit_json_read_completed; +use crate::metrics::PrecountedMetricsIterator; +use crate::schema::SchemaRef; +use crate::{ + CancellationTokenRef, DeltaResult, DeltaResultIterator, EngineData, FileDataReadResultIterator, + FileMeta, FilteredEngineData, JsonHandler, PredicateRef, +}; + +/// Decorator over an engine-provided `Arc` that emits a +/// `JsonReadCompleted` span on every `read_json_files` call. `parse_json` and +/// `write_json_file` are pass-through and emit nothing. +pub struct MeteredJsonHandler { + inner: Arc, +} + +impl MeteredJsonHandler { + /// Wrap `inner`. Debug-asserts that `inner` is not already a [`MeteredJsonHandler`] + /// so spans are emitted exactly once. + pub fn new(inner: Arc) -> Self { + debug_assert!( + !inner.any_ref().is::(), + "MeteredJsonHandler wraps another MeteredJsonHandler; \ + remove the outer wrap to avoid double-counting metrics", + ); + Self { inner } + } +} + +impl std::fmt::Debug for MeteredJsonHandler { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("MeteredJsonHandler").finish_non_exhaustive() + } +} + +impl MeteredJsonHandler { + /// Wraps `inner` so it emits a `JsonReadCompleted` span carrying `num_files` and the summed + /// `bytes_read` of `files` when the iterator is exhausted or dropped. + fn meter(files: &[FileMeta], inner: FileDataReadResultIterator) -> FileDataReadResultIterator { + let num_files = files.len() as u64; + let bytes_read = files.iter().map(|f| f.size).sum(); + Box::new(PrecountedMetricsIterator::new( + inner, + num_files, + bytes_read, + emit_json_read_completed, + )) + } +} + +impl JsonHandler for MeteredJsonHandler { + fn parse_json( + &self, + json_strings: Box, + output_schema: SchemaRef, + ) -> DeltaResult> { + self.inner.parse_json(json_strings, output_schema) + } + + fn read_json_files( + &self, + files: &[FileMeta], + physical_schema: SchemaRef, + predicate: Option, + ) -> DeltaResult { + let inner = self + .inner + .read_json_files(files, physical_schema, predicate)?; + Ok(Self::meter(files, inner)) + } + + fn read_json_files_with_cancellation( + &self, + files: &[FileMeta], + physical_schema: SchemaRef, + predicate: Option, + cancellation_token: Option, + ) -> DeltaResult { + let inner = self.inner.read_json_files_with_cancellation( + files, + physical_schema, + predicate, + cancellation_token, + )?; + Ok(Self::meter(files, inner)) + } + + fn write_json_file( + &self, + path: &url::Url, + data: DeltaResultIterator<'_, FilteredEngineData>, + overwrite: bool, + ) -> DeltaResult<()> { + self.inner.write_json_file(path, data, overwrite) + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use url::Url; + + use super::*; + use crate::arrow::array::RecordBatch; + use crate::arrow::datatypes::Schema; + use crate::engine::arrow_data::ArrowEngineData; + use crate::metrics::MetricEvent; + use crate::schema::{DataType, StructField, StructType}; + use crate::unit_test_utils::{install_thread_local_metrics_reporter, CapturingReporter}; + + #[derive(Debug, Default)] + struct StubJsonHandler { + /// Set when the cancellation-aware read variant is invoked, so a test can assert the + /// metered wrapper forwards to it (rather than the plain read). + cancellation_read_called: std::sync::atomic::AtomicBool, + } + + fn empty_batch() -> Box { + Box::new(ArrowEngineData::new(RecordBatch::new_empty(Arc::new( + Schema::empty(), + )))) + } + + impl JsonHandler for StubJsonHandler { + fn parse_json( + &self, + _json_strings: Box, + _output_schema: SchemaRef, + ) -> DeltaResult> { + Ok(empty_batch()) + } + + fn read_json_files( + &self, + _files: &[FileMeta], + _physical_schema: SchemaRef, + _predicate: Option, + ) -> DeltaResult { + Ok(Box::new(std::iter::empty())) + } + + fn read_json_files_with_cancellation( + &self, + _files: &[FileMeta], + _physical_schema: SchemaRef, + _predicate: Option, + _cancellation_token: Option, + ) -> DeltaResult { + self.cancellation_read_called + .store(true, std::sync::atomic::Ordering::SeqCst); + Ok(Box::new(std::iter::empty())) + } + + fn write_json_file( + &self, + _path: &Url, + _data: DeltaResultIterator<'_, FilteredEngineData>, + _overwrite: bool, + ) -> DeltaResult<()> { + Ok(()) + } + } + + fn fake_file(name: &str, size: u64) -> FileMeta { + FileMeta { + location: Url::parse(&format!("memory:///_delta_log/{name}")).unwrap(), + last_modified: 0, + size, + } + } + + fn install_capture() -> (Arc, tracing::subscriber::DefaultGuard) { + let reporter = Arc::new(CapturingReporter::default()); + let guard = install_thread_local_metrics_reporter(reporter.clone()); + (reporter, guard) + } + + fn delta_schema() -> SchemaRef { + Arc::new(StructType::try_new([StructField::nullable("x", DataType::INTEGER)]).unwrap()) + } + + #[test] + fn read_json_files_emits_json_read_completed() { + let (reporter, _guard) = install_capture(); + let inner: Arc = Arc::new(StubJsonHandler::default()); + let handler = MeteredJsonHandler::new(inner); + + let files = vec![fake_file("0.json", 100), fake_file("1.json", 50)]; + let iter = handler + .read_json_files(&files, delta_schema(), None) + .unwrap(); + let _: Vec<_> = iter.collect(); + + let events = reporter.events(); + let read = events + .iter() + .find(|e| matches!(e, MetricEvent::JsonReadCompleted(_))) + .expect("expected JsonReadCompleted event"); + let MetricEvent::JsonReadCompleted(e) = read else { + unreachable!(); + }; + assert_eq!(e.num_files, 2); + assert_eq!(e.bytes_read, 150); + } + + #[test] + fn read_json_files_emits_on_drop_without_consumption() { + let (reporter, _guard) = install_capture(); + let inner: Arc = Arc::new(StubJsonHandler::default()); + let handler = MeteredJsonHandler::new(inner); + + let files = vec![fake_file("0.json", 100), fake_file("1.json", 50)]; + { + let _iter = handler + .read_json_files(&files, delta_schema(), None) + .unwrap(); + } + + let events = reporter.events(); + let read = events + .iter() + .find(|e| matches!(e, MetricEvent::JsonReadCompleted(_))) + .expect("expected JsonReadCompleted event on drop"); + let MetricEvent::JsonReadCompleted(e) = read else { + unreachable!(); + }; + assert_eq!(e.num_files, 2); + assert_eq!(e.bytes_read, 150); + } + + // The metered wrapper's cancellation-aware read must forward to the inner handler's + // cancellation-aware read (not the plain one) AND still emit the metrics span. + #[test] + fn metered_wrapper_forwards_cancellation_read_and_emits_metrics() { + let (reporter, _guard) = install_capture(); + let stub = Arc::new(StubJsonHandler::default()); + let handler = MeteredJsonHandler::new(stub.clone() as Arc); + + let files = vec![fake_file("0.json", 100), fake_file("1.json", 50)]; + let iter = handler + .read_json_files_with_cancellation(&files, delta_schema(), None, None) + .unwrap(); + let _: Vec<_> = iter.collect(); + + assert!( + stub.cancellation_read_called + .load(std::sync::atomic::Ordering::SeqCst), + "metered wrapper must forward to the inner cancellation-aware read" + ); + let events = reporter.events(); + assert!( + events + .iter() + .any(|e| matches!(e, MetricEvent::JsonReadCompleted(_))), + "cancellation-aware read must still emit JsonReadCompleted" + ); + } + + #[test] + fn read_json_files_emits_zero_event_for_empty_input() { + let (reporter, _guard) = install_capture(); + let inner: Arc = Arc::new(StubJsonHandler::default()); + let handler = MeteredJsonHandler::new(inner); + + let iter = handler.read_json_files(&[], delta_schema(), None).unwrap(); + let _: Vec<_> = iter.collect(); + + let events = reporter.events(); + let read = events + .iter() + .find(|e| matches!(e, MetricEvent::JsonReadCompleted(_))) + .expect("expected zero-valued JsonReadCompleted"); + let MetricEvent::JsonReadCompleted(e) = read else { + unreachable!(); + }; + assert_eq!(e.num_files, 0); + assert_eq!(e.bytes_read, 0); + } + + #[test] + #[should_panic(expected = "wraps another MeteredJsonHandler")] + fn new_panics_on_double_wrap() { + let inner: Arc = Arc::new(StubJsonHandler::default()); + let once: Arc = Arc::new(MeteredJsonHandler::new(inner)); + let _twice = MeteredJsonHandler::new(once); + } +} diff --git a/kernel/src/metrics/metered_parquet.rs b/kernel/src/metrics/metered_parquet.rs new file mode 100644 index 0000000000..addd00db22 --- /dev/null +++ b/kernel/src/metrics/metered_parquet.rs @@ -0,0 +1,287 @@ +//! [`MeteredParquetHandler`] wraps any [`ParquetHandler`] so its `read_parquet_files` +//! emits the kernel's standard `ParquetReadCompleted` span, carrying +//! `(num_files, bytes_read)` exactly once when the returned iterator is exhausted or +//! dropped. `read_parquet_footer` and `write_parquet_file` pass through. + +use std::sync::Arc; + +use crate::metrics::events::emit_parquet_read_completed; +use crate::metrics::PrecountedMetricsIterator; +use crate::schema::SchemaRef; +use crate::{ + CancellationTokenRef, DeltaResult, DeltaResultIteratorStatic, EngineData, + FileDataReadResultIterator, FileMeta, ParquetFooter, ParquetHandler, PredicateRef, +}; + +/// Decorator over an engine-provided `Arc` that emits a +/// `ParquetReadCompleted` span on every `read_parquet_files` call. +/// `read_parquet_footer` and `write_parquet_file` are pass-through and emit nothing. +pub struct MeteredParquetHandler { + inner: Arc, +} + +impl MeteredParquetHandler { + /// Wrap `inner`. Debug-asserts that `inner` is not already a [`MeteredParquetHandler`] + /// so spans are emitted exactly once. + pub fn new(inner: Arc) -> Self { + debug_assert!( + !inner.any_ref().is::(), + "MeteredParquetHandler wraps another MeteredParquetHandler; \ + remove the outer wrap to avoid double-counting metrics", + ); + Self { inner } + } +} + +impl std::fmt::Debug for MeteredParquetHandler { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("MeteredParquetHandler") + .finish_non_exhaustive() + } +} + +impl MeteredParquetHandler { + /// Wraps `inner` so it emits a `ParquetReadCompleted` span carrying `num_files` and the summed + /// `bytes_read` of `files` when the iterator is exhausted or dropped. + fn meter(files: &[FileMeta], inner: FileDataReadResultIterator) -> FileDataReadResultIterator { + let num_files = files.len() as u64; + let bytes_read = files.iter().map(|f| f.size).sum(); + Box::new(PrecountedMetricsIterator::new( + inner, + num_files, + bytes_read, + emit_parquet_read_completed, + )) + } +} + +impl ParquetHandler for MeteredParquetHandler { + fn read_parquet_files( + &self, + files: &[FileMeta], + physical_schema: SchemaRef, + predicate: Option, + ) -> DeltaResult { + let inner = self + .inner + .read_parquet_files(files, physical_schema, predicate)?; + Ok(Self::meter(files, inner)) + } + + fn read_parquet_files_with_cancellation( + &self, + files: &[FileMeta], + physical_schema: SchemaRef, + predicate: Option, + cancellation_token: Option, + ) -> DeltaResult { + let inner = self.inner.read_parquet_files_with_cancellation( + files, + physical_schema, + predicate, + cancellation_token, + )?; + Ok(Self::meter(files, inner)) + } + + fn write_parquet_file( + &self, + location: url::Url, + data: DeltaResultIteratorStatic>, + ) -> DeltaResult<()> { + self.inner.write_parquet_file(location, data) + } + + fn read_parquet_footer(&self, file: &FileMeta) -> DeltaResult { + self.inner.read_parquet_footer(file) + } + + fn read_parquet_footer_with_cancellation( + &self, + file: &FileMeta, + cancellation_token: Option, + ) -> DeltaResult { + self.inner + .read_parquet_footer_with_cancellation(file, cancellation_token) + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use url::Url; + + use super::*; + use crate::metrics::MetricEvent; + use crate::schema::{DataType, StructField, StructType}; + use crate::unit_test_utils::{install_thread_local_metrics_reporter, CapturingReporter}; + + #[derive(Debug, Default)] + struct StubParquetHandler { + /// Set when the cancellation-aware read variant is invoked, so a test can assert the + /// metered wrapper forwards to it (rather than the plain read). + cancellation_read_called: std::sync::atomic::AtomicBool, + } + + impl ParquetHandler for StubParquetHandler { + fn read_parquet_files( + &self, + _files: &[FileMeta], + _physical_schema: SchemaRef, + _predicate: Option, + ) -> DeltaResult { + Ok(Box::new(std::iter::empty())) + } + + fn read_parquet_files_with_cancellation( + &self, + _files: &[FileMeta], + _physical_schema: SchemaRef, + _predicate: Option, + _cancellation_token: Option, + ) -> DeltaResult { + self.cancellation_read_called + .store(true, std::sync::atomic::Ordering::SeqCst); + Ok(Box::new(std::iter::empty())) + } + + fn write_parquet_file( + &self, + _location: Url, + _data: DeltaResultIteratorStatic>, + ) -> DeltaResult<()> { + Ok(()) + } + + fn read_parquet_footer(&self, _file: &FileMeta) -> DeltaResult { + unreachable!("not exercised in these tests") + } + } + + fn fake_file(name: &str, size: u64) -> FileMeta { + FileMeta { + location: Url::parse(&format!("memory:///_delta_log/{name}")).unwrap(), + last_modified: 0, + size, + } + } + + fn install_capture() -> (Arc, tracing::subscriber::DefaultGuard) { + let reporter = Arc::new(CapturingReporter::default()); + let guard = install_thread_local_metrics_reporter(reporter.clone()); + (reporter, guard) + } + + fn delta_schema() -> SchemaRef { + Arc::new(StructType::try_new([StructField::nullable("x", DataType::INTEGER)]).unwrap()) + } + + #[test] + fn read_parquet_files_emits_parquet_read_completed() { + let (reporter, _guard) = install_capture(); + let inner: Arc = Arc::new(StubParquetHandler::default()); + let handler = MeteredParquetHandler::new(inner); + + let files = vec![fake_file("a.parquet", 256), fake_file("b.parquet", 1024)]; + let iter = handler + .read_parquet_files(&files, delta_schema(), None) + .unwrap(); + let _: Vec<_> = iter.collect(); + + let events = reporter.events(); + let read = events + .iter() + .find(|e| matches!(e, MetricEvent::ParquetReadCompleted(_))) + .expect("expected ParquetReadCompleted event"); + let MetricEvent::ParquetReadCompleted(e) = read else { + unreachable!(); + }; + assert_eq!(e.num_files, 2); + assert_eq!(e.bytes_read, 1280); + } + + #[test] + fn read_parquet_files_emits_on_drop_without_consumption() { + let (reporter, _guard) = install_capture(); + let inner: Arc = Arc::new(StubParquetHandler::default()); + let handler = MeteredParquetHandler::new(inner); + + let files = vec![fake_file("a.parquet", 256), fake_file("b.parquet", 1024)]; + { + let _iter = handler + .read_parquet_files(&files, delta_schema(), None) + .unwrap(); + } + + let events = reporter.events(); + let read = events + .iter() + .find(|e| matches!(e, MetricEvent::ParquetReadCompleted(_))) + .expect("expected ParquetReadCompleted event on drop"); + let MetricEvent::ParquetReadCompleted(e) = read else { + unreachable!(); + }; + assert_eq!(e.num_files, 2); + assert_eq!(e.bytes_read, 1280); + } + + // The metered wrapper's cancellation-aware read must forward to the inner handler's + // cancellation-aware read (not the plain one) AND still emit the metrics span. + #[test] + fn metered_wrapper_forwards_cancellation_read_and_emits_metrics() { + let (reporter, _guard) = install_capture(); + let stub = Arc::new(StubParquetHandler::default()); + let handler = MeteredParquetHandler::new(stub.clone() as Arc); + + let files = vec![fake_file("a.parquet", 256), fake_file("b.parquet", 1024)]; + let iter = handler + .read_parquet_files_with_cancellation(&files, delta_schema(), None, None) + .unwrap(); + let _: Vec<_> = iter.collect(); + + assert!( + stub.cancellation_read_called + .load(std::sync::atomic::Ordering::SeqCst), + "metered wrapper must forward to the inner cancellation-aware read" + ); + let events = reporter.events(); + assert!( + events + .iter() + .any(|e| matches!(e, MetricEvent::ParquetReadCompleted(_))), + "cancellation-aware read must still emit ParquetReadCompleted" + ); + } + + #[test] + fn read_parquet_files_emits_zero_event_for_empty_input() { + let (reporter, _guard) = install_capture(); + let inner: Arc = Arc::new(StubParquetHandler::default()); + let handler = MeteredParquetHandler::new(inner); + + let iter = handler + .read_parquet_files(&[], delta_schema(), None) + .unwrap(); + let _: Vec<_> = iter.collect(); + + let events = reporter.events(); + let read = events + .iter() + .find(|e| matches!(e, MetricEvent::ParquetReadCompleted(_))) + .expect("expected zero-valued ParquetReadCompleted"); + let MetricEvent::ParquetReadCompleted(e) = read else { + unreachable!(); + }; + assert_eq!(e.num_files, 0); + assert_eq!(e.bytes_read, 0); + } + + #[test] + #[should_panic(expected = "wraps another MeteredParquetHandler")] + fn new_panics_on_double_wrap() { + let inner: Arc = Arc::new(StubParquetHandler::default()); + let once: Arc = Arc::new(MeteredParquetHandler::new(inner)); + let _twice = MeteredParquetHandler::new(once); + } +} diff --git a/kernel/src/metrics/metered_storage.rs b/kernel/src/metrics/metered_storage.rs new file mode 100644 index 0000000000..798b989777 --- /dev/null +++ b/kernel/src/metrics/metered_storage.rs @@ -0,0 +1,235 @@ +//! [`MeteredStorageHandler`] wraps any [`StorageHandler`] so it emits the kernel's +//! standard `"storage"` tracing spans. Usually reached via [`MeteredDeltaEngine`]; +//! construct directly when wrapping a `StorageHandler` outside an `Engine`. +//! +//! [`MeteredDeltaEngine`]: crate::metrics::MeteredDeltaEngine + +use std::sync::Arc; +use std::time::Instant; + +use bytes::Bytes; +use url::Url; + +use crate::metrics::events::{StorageCopyCompleted, StorageListCompleted, StorageReadCompleted}; +use crate::metrics::{emit_storage_span, MetricsIterator}; +use crate::{DeltaResult, FileMeta, FileSlice, StorageHandler}; + +/// Decorator over an engine-provided `Arc` that emits the kernel's +/// standard `"storage"` spans on operations that produce metrics. `put`, `head`, and `delete` +/// are pass-through and emit nothing. +pub struct MeteredStorageHandler { + inner: Arc, +} + +impl MeteredStorageHandler { + /// Wrap `inner`. Debug-asserts that `inner` is not already a + /// [`MeteredStorageHandler`] so spans are emitted exactly once. + pub fn new(inner: Arc) -> Self { + debug_assert!( + !inner.any_ref().is::(), + "MeteredStorageHandler wraps another MeteredStorageHandler; \ + remove the outer wrap to avoid double-counting metrics", + ); + Self { inner } + } +} + +impl std::fmt::Debug for MeteredStorageHandler { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("MeteredStorageHandler") + .finish_non_exhaustive() + } +} + +impl StorageHandler for MeteredStorageHandler { + fn list_from( + &self, + path: &Url, + ) -> DeltaResult>>> { + let start = Instant::now(); + let inner = self.inner.list_from(path)?; + Ok(Box::new(MetricsIterator::<_, FileMeta>::new( + inner, + StorageListCompleted::NAME, + start, + ))) + } + + fn read_files( + &self, + files: Vec, + ) -> DeltaResult>>> { + let start = Instant::now(); + let inner = self.inner.read_files(files)?; + Ok(Box::new(MetricsIterator::<_, Bytes>::new( + inner, + StorageReadCompleted::NAME, + start, + ))) + } + + fn copy_atomic(&self, src: &Url, dest: &Url) -> DeltaResult<()> { + let start = Instant::now(); + let result = self.inner.copy_atomic(src, dest); + emit_storage_span(StorageCopyCompleted::NAME, start.elapsed(), 0, 0); + result + } + + fn put(&self, path: &Url, data: Bytes, overwrite: bool) -> DeltaResult<()> { + self.inner.put(path, data, overwrite) + } + + fn head(&self, path: &Url) -> DeltaResult { + self.inner.head(path) + } + + fn delete(&self, path: &Url) -> DeltaResult<()> { + self.inner.delete(path) + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use super::*; + use crate::metrics::MetricEvent; + use crate::unit_test_utils::{install_thread_local_metrics_reporter, CapturingReporter}; + + /// Storage handler that returns N preconfigured FileMeta / byte slices. + #[derive(Debug)] + struct StubStorageHandler { + list_results: Vec, + read_results: Vec, + } + + impl StorageHandler for StubStorageHandler { + fn list_from( + &self, + _path: &Url, + ) -> DeltaResult>>> { + let results: Vec<_> = self.list_results.iter().cloned().map(Ok).collect(); + Ok(Box::new(results.into_iter())) + } + + fn read_files( + &self, + _files: Vec, + ) -> DeltaResult>>> { + let results: Vec<_> = self.read_results.iter().cloned().map(Ok).collect(); + Ok(Box::new(results.into_iter())) + } + + fn copy_atomic(&self, _src: &Url, _dest: &Url) -> DeltaResult<()> { + Ok(()) + } + + fn put(&self, _path: &Url, _data: Bytes, _overwrite: bool) -> DeltaResult<()> { + Ok(()) + } + + fn head(&self, _path: &Url) -> DeltaResult { + unreachable!("not exercised in these tests") + } + + fn delete(&self, _path: &Url) -> DeltaResult<()> { + Ok(()) + } + } + + fn fake_url() -> Url { + Url::parse("memory:///_delta_log/").unwrap() + } + + fn fake_file_meta(name: &str) -> FileMeta { + FileMeta { + location: Url::parse(&format!("memory:///_delta_log/{name}")).unwrap(), + last_modified: 0, + size: 0, + } + } + + fn install_capture() -> (Arc, tracing::subscriber::DefaultGuard) { + let reporter = Arc::new(CapturingReporter::default()); + let guard = install_thread_local_metrics_reporter(reporter.clone()); + (reporter, guard) + } + + #[test] + fn list_from_emits_storage_list_completed() { + let (reporter, _guard) = install_capture(); + let inner: Arc = Arc::new(StubStorageHandler { + list_results: vec![ + fake_file_meta("00000000000000000000.json"), + fake_file_meta("00000000000000000001.json"), + ], + read_results: vec![], + }); + let storage = MeteredStorageHandler::new(inner); + + let iter = storage.list_from(&fake_url()).unwrap(); + let _: Vec<_> = iter.collect(); + + let events = reporter.events(); + let listed = events + .iter() + .find(|e| matches!(e, MetricEvent::StorageListCompleted(_))) + .expect("expected StorageListCompleted event"); + let MetricEvent::StorageListCompleted(e) = listed else { + unreachable!(); + }; + assert_eq!(e.num_files, 2); + } + + #[test] + fn read_files_emits_storage_read_completed() { + let (reporter, _guard) = install_capture(); + let inner: Arc = Arc::new(StubStorageHandler { + list_results: vec![], + read_results: vec![Bytes::from(vec![0u8; 32]), Bytes::from(vec![0u8; 8])], + }); + let storage = MeteredStorageHandler::new(inner); + + let iter = storage.read_files(vec![]).unwrap(); + let _: Vec<_> = iter.collect(); + + let events = reporter.events(); + let read = events + .iter() + .find(|e| matches!(e, MetricEvent::StorageReadCompleted(_))) + .expect("expected StorageReadCompleted event"); + let MetricEvent::StorageReadCompleted(e) = read else { + unreachable!(); + }; + assert_eq!(e.num_files, 2); + assert_eq!(e.bytes_read, 40); + } + + #[test] + fn copy_atomic_emits_storage_copy_completed() { + let (reporter, _guard) = install_capture(); + let inner: Arc = Arc::new(StubStorageHandler { + list_results: vec![], + read_results: vec![], + }); + let storage = MeteredStorageHandler::new(inner); + + storage.copy_atomic(&fake_url(), &fake_url()).unwrap(); + + let events = reporter.events(); + assert!(events + .iter() + .any(|e| matches!(e, MetricEvent::StorageCopyCompleted(_)))); + } + + #[test] + #[should_panic(expected = "wraps another MeteredStorageHandler")] + fn new_panics_on_double_wrap() { + let inner: Arc = Arc::new(StubStorageHandler { + list_results: vec![], + read_results: vec![], + }); + let once: Arc = Arc::new(MeteredStorageHandler::new(inner)); + let _twice = MeteredStorageHandler::new(once); + } +} diff --git a/kernel/src/metrics/mod.rs b/kernel/src/metrics/mod.rs index 5432dafe3b..dabe52d2b6 100644 --- a/kernel/src/metrics/mod.rs +++ b/kernel/src/metrics/mod.rs @@ -11,7 +11,6 @@ //! # Example: Implementing a Custom MetricsReporter //! //! ``` -//! use std::sync::Arc; //! use delta_kernel::metrics::{MetricsReporter, MetricEvent}; //! //! #[derive(Debug)] @@ -20,14 +19,14 @@ //! impl MetricsReporter for LoggingReporter { //! fn report(&self, event: MetricEvent) { //! match event { -//! MetricEvent::LogSegmentLoaded { operation_id, duration, num_commit_files, .. } => { -//! println!("Log segment loaded in {:?}: {} commits", duration, num_commit_files); +//! MetricEvent::LogSegmentLoadSuccess(e) => { +//! println!("Log segment loaded in {:?}: {} commits", e.duration, e.num_commit_files); //! } -//! MetricEvent::SnapshotCompleted { operation_id, version, total_duration } => { -//! println!("Snapshot completed: v{} in {:?}", version, total_duration); +//! MetricEvent::SnapshotBuildSuccess(e) => { +//! println!("Snapshot completed: v{} in {:?}", e.version, e.duration); //! } -//! MetricEvent::SnapshotFailed { operation_id, duration } => { -//! println!("Snapshot failed: {} after {:?}", operation_id, duration); +//! MetricEvent::SnapshotBuildFailure(e) => { +//! println!("Snapshot failed: {}", e.operation_id); //! } //! _ => {} //! } @@ -57,18 +56,125 @@ //! } //! ``` //! -//! # Storage Metrics +//! # Handler Metrics //! -//! Storage operations (list, read, copy) are automatically instrumented when using -//! `DefaultEngine` with a metrics reporter. The default storage handler implementation -//! emits `StorageListCompleted`, `StorageReadCompleted`, and `StorageCopyCompleted` -//! events that track latencies at the storage layer. +//! Storage, JSON, and Parquet handler operations emit one event per read or copy +//! operation, fired when the returned iterator is exhausted or dropped: +//! `StorageListCompleted` / `StorageReadCompleted` / `StorageCopyCompleted` for storage, +//! `JsonReadCompleted` for `JsonHandler::read_json_files`, and `ParquetReadCompleted` +//! for `ParquetHandler::read_parquet_files`. `DefaultEngine` wraps its three handlers +//! in [`MeteredStorageHandler`], [`MeteredJsonHandler`], and [`MeteredParquetHandler`] +//! at construction; any other engine gets the same coverage by wrapping its [`Engine`] +//! in [`MeteredDeltaEngine`]. //! -//! These metrics are standalone and track aggregate storage performance without +//! These metrics are standalone and track aggregate handler performance without //! correlating to specific Snapshot/Transaction operations. +//! +//! [`Engine`]: crate::Engine + +pub(crate) mod events; +mod metered_engine; +mod metered_json; +mod metered_parquet; +mod metered_storage; +mod precounted_metrics_iterator; +pub(crate) mod reporter; +mod streaming_metrics_iterator; + +use std::sync::Arc; + +pub use events::{ + emit_json_read_completed, emit_parquet_read_completed, CommitFailureReason, CrcReadSuccess, + DomainMetadataLoadSuccess, JsonReadCompleted, LogSegmentLoadFailure, LogSegmentLoadSuccess, + LogSegmentLoadType, MetricEvent, MetricId, ParquetReadCompleted, ProtocolMetadataLoadFailure, + ProtocolMetadataLoadSuccess, ProtocolMetadataSource, ScanMetadataCompleted, ScanType, + SetTransactionLoadSuccess, SnapshotBuildFailure, SnapshotBuildSuccess, + SnapshotLoadMetricContext, StorageCopyCompleted, StorageListCompleted, StorageReadCompleted, + TableType, TransactionCommitFailure, TransactionCommitSuccess, +}; +pub(crate) use events::{ + emit_log_segment_load, emit_log_segment_load_failure, emit_protocol_metadata_load, + emit_protocol_metadata_load_failure, +}; +pub use metered_engine::MeteredDeltaEngine; +pub use metered_json::MeteredJsonHandler; +pub use metered_parquet::MeteredParquetHandler; +pub use metered_storage::MeteredStorageHandler; +pub(crate) use precounted_metrics_iterator::PrecountedMetricsIterator; +pub use reporter::{LoggingMetricsReporter, MetricsReporter, ReportGeneratorLayer}; +pub(crate) use streaming_metrics_iterator::{emit_storage_span, MetricsIterator}; +use tracing::Subscriber; +use tracing_subscriber::layer::{Layered, SubscriberExt as _}; +use tracing_subscriber::registry::LookupSpan; -mod events; -mod reporter; +/// Extension trait that adds [`with_metrics_reporter_layer`] to any compatible tracing subscriber. +/// +/// Only implemented for subscribers that also implement [`LookupSpan`], which is required by +/// [`ReportGeneratorLayer`] to store and retrieve per-span state. +/// +/// [`with_metrics_reporter_layer`]: WithMetricsReporterLayer::with_metrics_reporter_layer +pub trait WithMetricsReporterLayer: Subscriber + for<'lookup> LookupSpan<'lookup> { + /// Wrap this subscriber with a [`ReportGeneratorLayer`] that converts tracing spans into + /// [`MetricEvent`]s and forwards them to `reporter`. + /// + /// # Example + /// + /// ``` + /// use std::sync::Arc; + /// use delta_kernel::metrics::{WithMetricsReporterLayer, LoggingMetricsReporter}; + /// use tracing_subscriber::prelude::*; + /// + /// tracing_subscriber::registry() + /// .with_metrics_reporter_layer( + /// Arc::new(LoggingMetricsReporter::new(tracing::Level::INFO)) + /// ); + /// ``` + /// + /// # Important: keep a real global default subscriber installed + /// + /// Prefer installing this subscriber as the *global* default via + /// [`tracing::dispatcher::set_global_default`] (or + /// [`tracing_subscriber::util::SubscriberInitExt::init`]). The global path rebuilds + /// the callsite interest cache internally and guarantees that every thread has a + /// real subscriber to consult the first time it hits a callsite. + /// + /// If you need thread-local isolation -- e.g. per-test + /// [`tracing_subscriber::util::SubscriberInitExt::set_default`] guards in a + /// multi-threaded test binary -- also install a bare global default subscriber + /// (e.g. `tracing_subscriber::registry().init()`) up front. Test code should + /// prefer `test_utils::install_thread_local_metrics_reporter`, which bundles + /// the global-subscriber install with the thread-local `set_default` so callers + /// cannot accidentally skip the first step. Several kernel metrics are emitted + /// from `Drop` impls (notably storage list/read completion in the default + /// engine), and those `Drop` sites can run on threads that have no subscriber + /// installed -- for example tokio worker threads owned by the `DefaultEngine`'s + /// background runtime. If a no-subscriber thread is the first to hit such a + /// callsite, tracing caches its `Interest` as `never` process-globally, + /// silently disabling that metric for the rest of the process. Keeping a real + /// global default active means every thread's "current dispatcher" is a real + /// subscriber, so the cached interest stays in the `always`/`sometimes` regime. + /// + /// [`tracing::callsite::rebuild_interest_cache`] is *not* a reliable substitute on + /// its own: it only re-evaluates callsites that are already registered, but + /// callsites can be registered for the first time at any point on no-subscriber + /// threads, re-poisoning the cache after the rebuild. + /// + /// [`tracing_subscriber::util::SubscriberInitExt::init`]: https://docs.rs/tracing-subscriber/latest/tracing_subscriber/util/trait.SubscriberInitExt.html#method.init + /// [`tracing_subscriber::util::SubscriberInitExt::set_default`]: https://docs.rs/tracing-subscriber/latest/tracing_subscriber/util/trait.SubscriberInitExt.html#method.set_default + fn with_metrics_reporter_layer( + self, + reporter: Arc, + ) -> Layered + where + Self: Sized, + { + self.with(ReportGeneratorLayer::new(reporter)) + } +} -pub use events::{MetricEvent, MetricId}; -pub use reporter::MetricsReporter; +impl WithMetricsReporterLayer for S +where + S: Subscriber, + for<'lookup> S: LookupSpan<'lookup>, +{ +} diff --git a/kernel/src/metrics/precounted_metrics_iterator.rs b/kernel/src/metrics/precounted_metrics_iterator.rs new file mode 100644 index 0000000000..ad82a2c16a --- /dev/null +++ b/kernel/src/metrics/precounted_metrics_iterator.rs @@ -0,0 +1,61 @@ +//! Iterator wrapper that emits a metric with pre-computed `(num_files, bytes_read)` +//! exactly once -- when the inner iterator is exhausted or the wrapper is dropped. +//! +//! Used by handlers that know their counts up-front from the input `&[FileMeta]` -- +//! today, JSON and Parquet `read_*_files` -- so there's no need to count items as +//! they flow. Pair with [`MetricsIterator`] (count-as-you-stream) for handlers whose +//! counts aren't known until the work happens. +//! +//! [`MetricsIterator`]: crate::metrics::MetricsIterator + +/// Emits a precomputed `(num_files, bytes_read)` payload exactly once via `emit_fn` +/// when the inner iterator finishes (either exhausted via `next` returning `None` +/// or dropped early). +pub(crate) struct PrecountedMetricsIterator { + inner: I, + num_files: u64, + bytes_read: u64, + emitted: bool, + emit_fn: fn(u64, u64), +} + +impl PrecountedMetricsIterator { + /// Wrap `inner` with the precomputed counts. `emit_fn` is called once on + /// exhaustion or drop with `(num_files, bytes_read)`; typically a thin + /// wrapper around `tracing::span!` so the kernel's `ReportGeneratorLayer` + /// picks it up. + pub(crate) fn new(inner: I, num_files: u64, bytes_read: u64, emit_fn: fn(u64, u64)) -> Self { + Self { + inner, + num_files, + bytes_read, + emitted: false, + emit_fn, + } + } + + fn emit_once(&mut self) { + if !self.emitted { + self.emitted = true; + (self.emit_fn)(self.num_files, self.bytes_read); + } + } +} + +impl Iterator for PrecountedMetricsIterator { + type Item = I::Item; + + fn next(&mut self) -> Option { + let item = self.inner.next(); + if item.is_none() { + self.emit_once(); + } + item + } +} + +impl Drop for PrecountedMetricsIterator { + fn drop(&mut self) { + self.emit_once(); + } +} diff --git a/kernel/src/metrics/reporter.rs b/kernel/src/metrics/reporter.rs index 5357aef0fe..72504adf8e 100644 --- a/kernel/src/metrics/reporter.rs +++ b/kernel/src/metrics/reporter.rs @@ -1,14 +1,446 @@ -//! Metrics reporter trait and implementations. +//! Metrics reporter trait and tracing-layer integration. -use super::MetricEvent; +use std::sync::Arc; +use std::time::Instant; -/// Trait for reporting metrics events from Delta operations. -/// -/// Implementations of this trait receive metric events as they occur during operations -/// and can forward them to monitoring systems like Prometheus, DataDog, etc. -/// -/// Events are emitted throughout an operation's lifecycle, allowing real-time monitoring. +use tracing::field::{Field, Visit}; +use tracing::span::{Attributes, Id, Record}; +use tracing::{event, warn, Level, Span, Subscriber}; +use tracing_subscriber::layer::Context; +use tracing_subscriber::Layer; + +use super::events::{ + storage_metric_from_attrs, CrcReadSuccess, DomainMetadataLoadSuccess, JsonReadCompleted, + LogSegmentLoadSuccess, MetricEvent, ParquetReadCompleted, ProtocolMetadataLoadSuccess, + ScanMetadataCompleted, SetTransactionLoadSuccess, SnapshotBuildSuccess, + TransactionCommitSuccess, STORAGE_SPAN, +}; + +// ==================================================================== +// MetricsReporter trait + LoggingMetricsReporter +// ==================================================================== + +/// Receives [`MetricEvent`]s as they occur during Delta operations and forwards them to a +/// monitoring system. Reporter implementations must be cheap to call on any thread. pub trait MetricsReporter: Send + Sync + std::fmt::Debug { /// Report a metric event. fn report(&self, event: MetricEvent); } + +/// A [`MetricsReporter`] that logs each event as a tracing event at the configured level. +#[derive(Debug)] +pub struct LoggingMetricsReporter { + level: Level, +} + +impl LoggingMetricsReporter { + /// Create a new reporter that logs each [`MetricEvent`] at the given tracing level. + pub fn new(level: Level) -> Self { + Self { level } + } +} + +impl MetricsReporter for LoggingMetricsReporter { + fn report(&self, event: MetricEvent) { + // event! needs a constant level. Detach from the parent span so the log line carries the + // event payload alone, not the span context that produced it. + match self.level { + Level::ERROR => event!(parent: Span::none(), Level::ERROR, "{}", event), + Level::WARN => event!(parent: Span::none(), Level::WARN, "{}", event), + Level::INFO => event!(parent: Span::none(), Level::INFO, "{}", event), + Level::DEBUG => event!(parent: Span::none(), Level::DEBUG, "{}", event), + Level::TRACE => event!(parent: Span::none(), Level::TRACE, "{}", event), + } + } +} + +// ==================================================================== +// ReportGeneratorLayer +// ==================================================================== + +/// A [`tracing_subscriber::Layer`] that converts kernel tracing spans into [`MetricEvent`]s and +/// forwards them to a registered [`MetricsReporter`]. +/// +/// Typically added to a subscriber via +/// [`super::WithMetricsReporterLayer::with_metrics_reporter_layer`]. +#[derive(Debug)] +pub struct ReportGeneratorLayer { + reporter: Arc, +} + +impl ReportGeneratorLayer { + /// Create a new layer that forwards metric events to the given reporter. + pub fn new(reporter: Arc) -> Self { + Self { reporter } + } + + /// Apply `record` to the visitor stashed on `span`, then drain its pending warnings. Warnings + /// must be emitted *after* the `extensions_mut` lock is released; calling `warn!()` while + /// holding it would re-enter the layer and deadlock. + fn drain_into_visitor( + span: Option>, + record: impl FnOnce(&mut EventVisitor), + ) where + S: Subscriber + for<'l> tracing_subscriber::registry::LookupSpan<'l>, + { + let warnings = span.and_then(|span| { + let mut extensions = span.extensions_mut(); + let visitor = extensions.get_mut::()?; + record(visitor); + Some(std::mem::take(&mut visitor.pending_warnings)) + }); + for w in warnings.unwrap_or_default() { + warn!("{w}"); + } + } +} + +impl Layer for ReportGeneratorLayer +where + S: Subscriber + for<'lookup> tracing_subscriber::registry::LookupSpan<'lookup>, +{ + // CONSTRUCTION-TIME CHANNEL. Each Type::from_attrs(attrs) extracts fields bound at span + // creation. + fn on_new_span(&self, attrs: &Attributes<'_>, id: &Id, ctx: Context<'_, S>) { + let Some(metadata) = ctx.metadata(id) else { + return; + }; + let event = match metadata.name() { + LogSegmentLoadSuccess::SPAN_NAME => Some(MetricEvent::LogSegmentLoadSuccess( + LogSegmentLoadSuccess::from_attrs(attrs), + )), + ProtocolMetadataLoadSuccess::SPAN_NAME => { + Some(MetricEvent::ProtocolMetadataLoadSuccess( + ProtocolMetadataLoadSuccess::from_attrs(attrs), + )) + } + SnapshotBuildSuccess::SPAN_NAME => Some(MetricEvent::SnapshotBuildSuccess( + SnapshotBuildSuccess::from_attrs(attrs), + )), + TransactionCommitSuccess::SPAN_NAME => Some(MetricEvent::TransactionCommitSuccess( + TransactionCommitSuccess::from_attrs(attrs), + )), + DomainMetadataLoadSuccess::SPAN_NAME => Some(MetricEvent::DomainMetadataLoadSuccess( + DomainMetadataLoadSuccess::from_attrs(attrs), + )), + SetTransactionLoadSuccess::SPAN_NAME => Some(MetricEvent::SetTransactionLoadSuccess( + SetTransactionLoadSuccess::from_attrs(attrs), + )), + CrcReadSuccess::SPAN_NAME => Some(MetricEvent::CrcReadSuccess( + CrcReadSuccess::from_attrs(attrs), + )), + JsonReadCompleted::SPAN_NAME => Some(MetricEvent::JsonReadCompleted( + JsonReadCompleted::from_attrs(attrs), + )), + ParquetReadCompleted::SPAN_NAME => Some(MetricEvent::ParquetReadCompleted( + ParquetReadCompleted::from_attrs(attrs), + )), + ScanMetadataCompleted::SPAN_NAME => Some(MetricEvent::ScanMetadataCompleted( + ScanMetadataCompleted::from_attrs(attrs), + )), + STORAGE_SPAN => storage_metric_from_attrs(attrs), + _ => None, + }; + + if let Some(span) = ctx.span(id) { + let mut extensions = span.extensions_mut(); + extensions.insert(EventVisitor::new(event)); + } + } + + // RUNTIME CHANNEL. on_event carries log lines, not metric writes; its only metric effect is the + // `#[instrument(err)]` failure signal, which FailureFlipVisitor turns into the event's failure + // counterpart. + fn on_event(&self, event: &tracing::Event<'_>, ctx: Context<'_, S>) { + let Some(span) = ctx.event_span(event) else { + return; + }; + let mut extensions = span.extensions_mut(); + if let Some(visitor) = extensions.get_mut::() { + event.record(&mut FailureFlipVisitor { + event: &mut visitor.event, + }); + } + } + + fn on_record(&self, id: &Id, values: &Record<'_>, ctx: Context<'_, S>) { + Self::drain_into_visitor(ctx.span(id), |v| values.record(v)); + } + + fn on_enter(&self, id: &Id, ctx: Context<'_, S>) { + // Stash the start time on first entry. Spans entered and exited multiple times (e.g. + // iterator adapters) keep the original timestamp so on_close measures wall time from the + // span's first entry. + if let Some(span) = ctx.span(id) { + let mut extensions = span.extensions_mut(); + if extensions.get_mut::().is_none() { + extensions.insert(Instant::now()); + } + } + } + + fn on_close(&self, id: Id, ctx: Context<'_, S>) { + let Some(metadata) = ctx.metadata(&id) else { + return; + }; + if metadata.fields().field("report").is_none() { + return; + } + let Some(span) = ctx.span(&id) else { return }; + let event = { + let mut extensions = span.extensions_mut(); + let duration = extensions.get_mut::().map(|s| s.elapsed()); + let Some(visitor) = extensions.get_mut::() else { + return; + }; + if let (Some(d), Some(event)) = (duration, visitor.event.as_mut()) { + event.set_duration_if_applicable(d); + } + visitor.event.take() + }; // unlock the extensions before reporting; the reporter is free to warn!() etc. + if let Some(event) = event { + self.reporter.report(event); + } + } +} + +// ==================================================================== +// EventVisitor: dispatches field updates to the inner MetricEvent struct +// ==================================================================== + +/// Per-span visitor stashed in the span's extensions. Responsible for: +/// +/// 1. Holding the partially-constructed `MetricEvent` while the span runs. +/// 2. Implementing [`Visit`] so the layer can route `record_*` callbacks to the underlying +/// `MetricEvent` state. +/// 3. Accumulating any deferred warnings to emit after locks are released. +struct EventVisitor { + event: Option, + pending_warnings: Vec, +} + +impl EventVisitor { + fn new(event: Option) -> Self { + Self { + event, + pending_warnings: vec![], + } + } + + fn warn_invalid(&mut self, field: &Field, span_name: &str) { + self.pending_warnings.push(format!( + "Invalid field '{}' recorded on {span_name} span", + field.name() + )); + } +} + +impl Visit for EventVisitor { + fn record_u64(&mut self, field: &Field, value: u64) { + let Some(event) = self.event.as_mut() else { + return; + }; + if let Err(span_name) = event.record_u64(field.name(), value) { + self.warn_invalid(field, span_name); + } + } + + fn record_bool(&mut self, field: &Field, value: bool) { + let Some(event) = self.event.as_mut() else { + return; + }; + if let Err(span_name) = event.record_bool(field.name(), value) { + self.warn_invalid(field, span_name); + } + } + + fn record_str(&mut self, field: &Field, value: &str) { + let Some(event) = self.event.as_mut() else { + return; + }; + if let Err(span_name) = event.record_str(field.name(), value) { + self.warn_invalid(field, span_name); + } + } + + fn record_debug(&mut self, _field: &Field, _value: &dyn std::fmt::Debug) { + // Kernel records every metric field as u64/bool/str (handled above). Anything arriving here + // is a debug-formatted log field (or a numeric type tracing forwards to record_debug), not + // a metric write. + } +} + +// ==================================================================== +// FailureFlipVisitor: the event channel's only metric effect +// ==================================================================== + +/// Event-path visitor for the synthetic event `#[instrument(err)]` emits on `Err` return. +/// +/// Flips the span's in-flight success event to its failure counterpart and ignores every other +/// field, so arbitrary log-line fields never reach the metric allowlist. +struct FailureFlipVisitor<'a> { + event: &'a mut Option, +} + +impl Visit for FailureFlipVisitor<'_> { + fn record_debug(&mut self, field: &Field, _value: &dyn std::fmt::Debug) { + // `#[instrument(err)]` emits a synthetic event with an `error` field (via Display, hence + // record_debug) when the wrapped fn returns Err. That field name is a tracing convention, + // not kernel's. + if field.name() == "error" { + *self.event = self.event.take().map(MetricEvent::into_failure); + } + } +} + +#[cfg(test)] +mod tests { + use std::io::Error; + use std::sync::{Arc, Mutex}; + + use test_utils::{ensure_metrics_compatible_global_subscriber, LogWriter}; + use tracing::field::Empty; + use tracing::subscriber::with_default; + use tracing::{info, info_span, instrument}; + use tracing_subscriber::fmt::layer; + use tracing_subscriber::layer::SubscriberExt as _; + use tracing_subscriber::registry; + use uuid::Uuid; + + use crate::metrics::events::SNAPSHOT_COMPLETED_SPAN; + use crate::metrics::{MetricEvent, WithMetricsReporterLayer as _}; + use crate::unit_test_utils::{install_thread_local_metrics_reporter, CapturingReporter}; + + /// Run `f` under a subscriber that layers the metrics layer over an fmt layer capturing to a + /// buffer, and return the captured log text -- so a test can assert on the warnings the metrics + /// layer emits. + fn capture_logs(f: impl FnOnce()) -> String { + ensure_metrics_compatible_global_subscriber(); + let logs = Arc::new(Mutex::new(Vec::::new())); + let logs_writer = Arc::clone(&logs); + let subscriber = registry() + .with( + layer() + .with_writer(move || LogWriter(Arc::clone(&logs_writer))) + .with_ansi(false), + ) + .with_metrics_reporter_layer(Arc::new(CapturingReporter::default())); + with_default(subscriber, f); + let bytes = logs.lock().unwrap().clone(); + String::from_utf8(bytes).unwrap() + } + + #[test] + fn log_event_field_inside_reporting_span_does_not_warn() { + // info_span! requires a string-literal name; assert it still matches the span const. + assert_eq!(SNAPSHOT_COMPLETED_SPAN, "snap.build"); + let logs = capture_logs(|| { + let span = info_span!( + "snap.build", + report = Empty, + operation_id = %Uuid::new_v4(), + ); + let _enter = span.enter(); + info!(log_tail_len = 5u64, "building snapshot"); + }); + assert!( + !logs.contains("Invalid field"), + "a log line's fields must not be judged as metric writes; got: {logs}" + ); + } + + #[test] + fn recorded_unknown_field_still_warns() { + assert_eq!(SNAPSHOT_COMPLETED_SPAN, "snap.build"); + let logs = capture_logs(|| { + let span = info_span!( + "snap.build", + report = Empty, + operation_id = %Uuid::new_v4(), + bogus = Empty, + ); + span.record("bogus", 7u64); + }); + assert!( + logs.contains("Invalid field 'bogus' recorded on snap.build span"), + "an unrecognized .record() field is a declared-field typo and must still warn; got: {logs}" + ); + } + + // A log line whose field name collides with a real metric field must NOT overwrite the value a + // genuine `span.record` wrote -- the core guarantee of narrowing on_event to + // FailureFlipVisitor. capture_logs discards the reporter, so this test installs a + // CapturingReporter to inspect the reported event. + #[test] + fn log_event_reusing_a_metric_field_name_does_not_overwrite_recorded_metric() { + assert_eq!(SNAPSHOT_COMPLETED_SPAN, "snap.build"); + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + { + let span = info_span!( + "snap.build", + report = Empty, + operation_id = %Uuid::new_v4(), + version = Empty, + ); + let _enter = span.enter(); + span.record("version", 7u64); // real metric write via the on_record channel + info!(version = 999u64, "noise"); // log line reusing the metric field name (on_event) + } // span close -> report + let events = reporter.events(); + let Some(MetricEvent::SnapshotBuildSuccess(success)) = events + .iter() + .find(|e| matches!(e, MetricEvent::SnapshotBuildSuccess(_))) + else { + panic!("expected a snapshot build success event; got: {events:?}"); + }; + assert_eq!( + success.version, 7, + "an on_event log field must not overwrite a span.record metric" + ); + } + + #[instrument(name = SNAPSHOT_COMPLETED_SPAN, fields(report, operation_id = %Uuid::new_v4()), err)] + fn failing_build() -> Result<(), Error> { + Err(Error::other("boom")) + } + + #[test] + fn instrument_err_event_still_flips_to_failure() { + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + let _ = failing_build(); + assert!( + reporter + .events() + .iter() + .any(|e| matches!(e, MetricEvent::SnapshotBuildFailure(_))), + "#[instrument(err)] must flip the success event to its failure form via on_event" + ); + } + + #[instrument(name = SNAPSHOT_COMPLETED_SPAN, fields(report, operation_id = %Uuid::new_v4()), ret)] + fn succeeding_build() -> Result<(), Error> { + Ok(()) + } + + #[test] + fn instrument_ret_event_does_not_flip_to_failure() { + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + let _ = succeeding_build(); + let events = reporter.events(); + assert!( + events + .iter() + .any(|e| matches!(e, MetricEvent::SnapshotBuildSuccess(_))), + "#[instrument(ret)]'s `return` field must NOT flip success to failure; got: {events:?}" + ); + assert!( + !events + .iter() + .any(|e| matches!(e, MetricEvent::SnapshotBuildFailure(_))), + "a successful #[instrument(ret)] must not emit a failure event; got: {events:?}" + ); + } +} diff --git a/kernel/src/metrics/streaming_metrics_iterator.rs b/kernel/src/metrics/streaming_metrics_iterator.rs new file mode 100644 index 0000000000..6b555bbd2c --- /dev/null +++ b/kernel/src/metrics/streaming_metrics_iterator.rs @@ -0,0 +1,111 @@ +//! Iterator wrapper that counts items as they stream and emits the kernel's standard +//! `"storage"` tracing span on drop. Used by [`MeteredStorageHandler`] for handlers +//! whose counts are only known as items flow (`list_from`, `read_files`). +//! +//! The span fires on the thread that drops the iterator (the caller's thread), +//! avoiding background-thread emission where no tracing subscriber may be installed. +//! +//! [`MeteredStorageHandler`]: crate::metrics::MeteredStorageHandler + +use std::marker::PhantomData; +use std::time::Instant; + +use bytes::Bytes; + +use crate::metrics::events::STORAGE_SPAN; +use crate::{DeltaResult, FileMeta}; + +/// Counts items observed and emits a `"storage"` span on drop. The type parameter +/// `T` selects whether bytes are counted: `Bytes` counts bytes, `FileMeta` does not. +pub(crate) struct MetricsIterator { + inner: I, + name: &'static str, + start: Instant, + num_files: u64, + bytes_read: u64, + _phantom: PhantomData, +} + +impl MetricsIterator { + pub(crate) fn new(inner: I, name: &'static str, start: Instant) -> Self { + Self { + inner, + name, + start, + num_files: 0, + bytes_read: 0, + _phantom: PhantomData, + } + } + + fn record_file(&mut self) { + self.num_files += 1; + } + + fn record_bytes(&mut self, n: u64) { + self.num_files += 1; + self.bytes_read += n; + } +} + +impl Drop for MetricsIterator { + fn drop(&mut self) { + emit_storage_span( + self.name, + self.start.elapsed(), + self.num_files, + self.bytes_read, + ); + } +} + +/// Emit a one-shot `"storage"` span describing a completed operation. Use when the +/// operation does not return an iterator (e.g. `copy_atomic`). +pub(crate) fn emit_storage_span( + name: &'static str, + elapsed: std::time::Duration, + num_files: u64, + bytes_read: u64, +) { + // Storage ops > 584 years (u64::MAX nanos) imply a bug, not a metric we care about. + let duration_ns = u64::try_from(elapsed.as_nanos()).unwrap_or(u64::MAX); + let _span = tracing::span!( + tracing::Level::INFO, + STORAGE_SPAN, + report = tracing::field::Empty, + name = name, + num_files = num_files, + bytes_read = bytes_read, + duration_ns = duration_ns, + ); +} + +impl Iterator for MetricsIterator +where + I: Iterator>, +{ + type Item = I::Item; + + fn next(&mut self) -> Option { + let item = self.inner.next(); + if matches!(item, Some(Ok(_))) { + self.record_file(); + } + item + } +} + +impl Iterator for MetricsIterator +where + I: Iterator>, +{ + type Item = I::Item; + + fn next(&mut self) -> Option { + let item = self.inner.next(); + if let Some(Ok(ref bytes)) = item { + self.record_bytes(bytes.len() as u64); + } + item + } +} diff --git a/kernel/src/parallel/parallel_phase.rs b/kernel/src/parallel/parallel_phase.rs index 114b8a91e2..0abcc64a9b 100644 --- a/kernel/src/parallel/parallel_phase.rs +++ b/kernel/src/parallel/parallel_phase.rs @@ -9,15 +9,12 @@ use std::sync::Arc; use delta_kernel_derive::internal_api; +use itertools::Itertools; -use crate::log_replay::ActionsBatch; -use crate::log_replay::ParallelLogReplayProcessor; +use crate::log_replay::{ActionsBatch, ParallelLogReplayProcessor}; use crate::scan::CHECKPOINT_READ_SCHEMA; use crate::schema::SchemaRef; -use crate::EngineData; -use crate::{DeltaResult, Engine, FileMeta}; - -use itertools::Itertools; +use crate::{DeltaResult, Engine, EngineData, FileMeta}; /// Processes checkpoint leaf files in parallel using a shared processor. /// @@ -124,28 +121,39 @@ impl Iterator for ParallelPhase

    { #[cfg(test)] mod tests { + use std::collections::HashSet; + use std::sync::Arc; + use std::thread; + + use tracing_subscriber::layer::SubscriberExt as _; + use tracing_subscriber::util::SubscriberInitExt as _; + use url::Url; + use super::*; - use crate::actions::get_log_add_schema; use crate::engine::arrow_data::ArrowEngineData; - use crate::engine::default::DefaultEngine; + use crate::engine::sync::SyncEngine; use crate::log_replay::FileActionKey; use crate::log_segment::CheckpointReadInfo; + use crate::metrics::{MetricEvent, ScanType, WithMetricsReporterLayer}; use crate::object_store::memory::InMemory; use crate::object_store::path::Path; - use crate::object_store::ObjectStore; - use crate::parallel::parallel_scan_metadata::AfterSequentialScanMetadata; - use crate::parallel::parallel_scan_metadata::{ParallelScanMetadata, ParallelState}; + use crate::object_store::ObjectStoreExt as _; + use crate::parallel::parallel_scan_metadata::{ + AfterSequentialScanMetadata, ParallelScanMetadata, ParallelState, + }; use crate::parquet::arrow::arrow_writer::ArrowWriter; - use crate::scan::log_replay::ScanLogReplayProcessor; + use crate::scan::log_replay::{ + ScanLogReplayProcessor, ScanPartitionValuesOptions, ScanStatsOptions, + }; use crate::scan::state::ScanFile; use crate::scan::state_info::tests::get_simple_state_info; + use crate::scan::{ScanBuilder, StatsOptions}; use crate::schema::{DataType, StructField, StructType}; - use crate::utils::test_utils::{load_test_table, parse_json_batch}; + use crate::unit_test_utils::{ + install_thread_local_metrics_reporter, load_test_table, parse_json_batch, CapturingReporter, + }; + use crate::utils::FoldWithOption as _; use crate::{PredicateRef, SnapshotRef}; - use std::collections::HashSet; - use std::sync::Arc; - use std::thread; - use url::Url; // ============================================================ // Test helpers for focused ParallelPhase tests @@ -203,7 +211,8 @@ mod tests { state_info, checkpoint_info, seen_file_keys, - false, + ScanStatsOptions::default(), + ScanPartitionValuesOptions::default(), ) } @@ -224,7 +233,7 @@ mod tests { ) -> DeltaResult<()> { let store = Arc::new(InMemory::new()); let url = Url::parse("memory:///")?; - let engine = DefaultEngine::builder(store.clone()).build(); + let engine = SyncEngine::new_with_store(store.clone()); // Create sidecar with add actions let json_adds = add_paths @@ -311,7 +320,7 @@ mod tests { // This test uses multiple sidecar files, so we need custom logic let store = Arc::new(InMemory::new()); let url = Url::parse("memory:///")?; - let engine = DefaultEngine::builder(store.clone()).build(); + let engine = SyncEngine::new_with_store(store.clone()); // Create two sidecars let sidecar1_data = parse_json_batch(vec![ @@ -377,11 +386,11 @@ mod tests { snapshot: &SnapshotRef, predicate: Option, ) -> DeltaResult> { - let mut builder = snapshot.clone().scan_builder(); - if let Some(pred) = predicate { - builder = builder.with_predicate(pred); - } - let scan = builder.build()?; + let scan = snapshot + .clone() + .scan_builder() + .fold_with(predicate, ScanBuilder::with_predicate) + .build()?; let mut scan_metadata_iter = scan.scan_metadata(engine)?; let mut paths = scan_metadata_iter.try_fold(Vec::new(), |acc, metadata_res| { @@ -398,16 +407,17 @@ mod tests { predicate: Option, with_serde: bool, one_file_per_worker: bool, + dispatcher: Option, ) -> DeltaResult<()> { let (engine, snapshot, _tempdir) = load_test_table(table_name)?; let expected_paths = get_expected_paths(engine.as_ref(), &snapshot, predicate.clone())?; - let mut builder = snapshot.scan_builder(); - if let Some(pred) = predicate { - builder = builder.with_predicate(pred); - } - let scan = builder.build()?; + let scan = snapshot + .clone() + .scan_builder() + .fold_with(predicate, ScanBuilder::with_predicate) + .build()?; let mut sequential = scan.parallel_scan_metadata(engine.clone())?; let mut all_paths = sequential.try_fold(Vec::new(), |acc, metadata_res| { @@ -442,8 +452,12 @@ mod tests { .map(|partition_files| { let engine = engine.clone(); let state = final_state.clone(); + let dispatcher = dispatcher.clone(); thread::spawn(move || -> DeltaResult> { + // Set the dispatcher in this thread to capture logs + let _guard = dispatcher.map(|d| tracing::dispatcher::set_default(&d)); + assert!(!partition_files.is_empty()); let mut parallel = ParallelScanMetadata::try_new( @@ -468,6 +482,9 @@ mod tests { let paths = handle.join().expect("Thread panicked")?; all_paths.extend(paths); } + + // Log metrics after all parallel workers complete + final_state.log_metrics(); } } @@ -480,66 +497,458 @@ mod tests { Ok(()) } - #[test] - fn test_parallel_with_json_sidecars() -> DeltaResult<()> { - for with_serde in [false, true] { - for one_file_per_worker in [false, true] { - verify_parallel_workflow( - "v2-checkpoints-json-with-sidecars", - None, - with_serde, - one_file_per_worker, - )?; - } + /// A caller-supplied correlation id reaches both the sequential and parallel phase + /// `ScanMetadataCompleted` events. The sequential event is emitted before any serialization so + /// it always carries the id. The parallel event carries it in-memory but loses it when + /// `ParallelState` is rebuilt from bytes, a documented limitation shared with `operation_id` + /// (tracked in #2736). Workers are driven inline (not on spawned threads) so every emission + /// stays on the thread holding the metrics reporter guard. + #[rstest::rstest] + #[case::in_memory(false, Some("scan-corr-xyz"))] + #[case::across_serde_boundary(true, None)] + fn parallel_scan_metadata_phases_carry_correlation_id( + #[case] with_serde: bool, + #[case] expected_parallel: Option<&str>, + ) -> DeltaResult<()> { + // This table has checkpoint sidecars, so the sequential phase yields a parallel phase. + let (engine, snapshot, _tempdir) = load_test_table("v2-checkpoints-json-with-sidecars")?; + + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let scan = snapshot + .scan_builder() + .with_correlation_id("scan-corr-xyz") + .build()?; + let mut sequential = scan.parallel_scan_metadata(engine.clone())?; + for sm in sequential.by_ref() { + sm?; + } + let AfterSequentialScanMetadata::Parallel { state, files } = sequential.finish()? else { + panic!("table with sidecars should require a parallel phase"); + }; + let state = if with_serde { + Arc::new(ParallelState::from_bytes( + engine.as_ref(), + &state.into_bytes()?, + )?) + } else { + Arc::new(*state) + }; + let mut parallel = ParallelScanMetadata::try_new(engine.clone(), state.clone(), files)?; + for sm in parallel.by_ref() { + sm?; } + state.log_metrics(); + + let correlation_for = |phase: ScanType| { + reporter.events().into_iter().find_map(|e| match e { + MetricEvent::ScanMetadataCompleted(s) if s.scan_type == phase => { + Some(s.correlation_id) + } + _ => None, + }) + }; + assert_eq!( + correlation_for(ScanType::SequentialPhase) + .expect("expected a sequential-phase event") + .as_deref(), + Some("scan-corr-xyz"), + ); + assert_eq!( + correlation_for(ScanType::ParallelPhase) + .expect("expected a parallel-phase event") + .as_deref(), + expected_parallel, + ); Ok(()) } - #[test] - fn test_parallel_with_parquet_sidecars() -> DeltaResult<()> { - for with_serde in [false, true] { - for one_file_per_worker in [false, true] { - verify_parallel_workflow( - "v2-checkpoints-parquet-with-sidecars", - None, - with_serde, - one_file_per_worker, - )?; - } - } - Ok(()) + /// Extract a metric value from logs by searching for "metric_name=value" + fn extract_metric(logs: &str, metric_name: &str) -> u64 { + let Some(pos) = logs.find(&format!("{}=", metric_name)) else { + panic!("Failed to find {} in logs", metric_name); + }; + let after = &logs[pos + metric_name.len() + 1..]; + // Find the end of the value (whitespace, comma, or closing paren) + let end_pos = after + .find(|c: char| c.is_whitespace() || c == ',' || c == ')') + .unwrap_or_else(|| panic!("Failed to find end of {} value", metric_name)); + let value_str = &after[..end_pos]; + value_str + .parse() + .unwrap_or_else(|_| panic!("Failed to parse {} value: {}", metric_name, value_str)) } - #[test] - fn test_no_parallel_phase_needed() -> DeltaResult<()> { - for with_serde in [false, true] { - for one_file_per_worker in [false, true] { - verify_parallel_workflow( - "table-without-dv-small", - None, - with_serde, - one_file_per_worker, - )?; - } - } - Ok(()) + /// Expected metric values for a phase (sequential or parallel) + #[derive(Debug, Clone)] + struct ExpectedMetrics { + add_files_seen: u64, + active_add_files: u64, + remove_files_seen: u64, + non_file_actions: u64, + predicate_filtered: u64, } - #[test] - fn test_parallel_with_dataskipping_predicate() -> DeltaResult<()> { - use crate::expressions::{column_expr, Expression as Expr}; - - let predicate = Arc::new(Expr::gt(column_expr!("id"), Expr::literal(20i64))); - for with_serde in [false, true] { - for one_file_per_worker in [false, true] { - verify_parallel_workflow( - "v2-checkpoints-json-with-sidecars", - Some(predicate.clone()), - with_serde, - one_file_per_worker, - )?; + /// Test case for parallel log replay workflow + struct ParallelLogReplayCase { + path: &'static str, + predicate: Option, + expected_sequential_metrics: ExpectedMetrics, + expected_parallel_metrics: Option, + } + + fn verify_metrics_in_logs( + logs: &str, + table_name: &str, + sequential_expected: &ExpectedMetrics, + parallel_expected: Option<&ExpectedMetrics>, + ) { + // Find the Sequential scan log line and extract metrics from it + let sequential_pos = logs + .find("Sequential scan metadata completed") + .unwrap_or_else(|| { + panic!( + "Expected Sequential completion log for table '{}'", + table_name + ) + }); + let sequential_logs = &logs[sequential_pos..]; + + // Extract and verify counter values from Phase 1 (sequential log line) + let add_files_seen = extract_metric(sequential_logs, "add_files_seen"); + let active_add_files = extract_metric(sequential_logs, "active_add_files"); + let remove_files_seen = extract_metric(sequential_logs, "remove_files_seen"); + let non_file_actions = extract_metric(sequential_logs, "non_file_actions"); + let predicate_filtered = extract_metric(sequential_logs, "predicate_filtered"); + + assert_eq!( + add_files_seen, sequential_expected.add_files_seen, + "Sequential add_files_seen mismatch" + ); + assert_eq!( + active_add_files, sequential_expected.active_add_files, + "Sequential active_add_files mismatch" + ); + assert_eq!( + remove_files_seen, sequential_expected.remove_files_seen, + "Sequential remove_files_seen mismatch" + ); + assert_eq!( + non_file_actions, sequential_expected.non_file_actions, + "Sequential non_file_actions mismatch", + ); + assert_eq!( + predicate_filtered, sequential_expected.predicate_filtered, + "Sequential predicate_filtered mismatch", + ); + + // Verify timing metrics are present and parseable (values may be 0 for fast operations) + let _dedup_time = extract_metric(sequential_logs, "dedup_visitor_time_ns"); + let _predicate_eval_time = extract_metric(sequential_logs, "predicate_eval_time_ns"); + + // Verify Parallel metrics if expected + if let Some(expected) = parallel_expected { + // Accumulate totals across all parallel logs + let mut total_add_files_seen = 0u64; + let mut total_active_add_files = 0u64; + let mut total_remove_files_seen = 0u64; + let mut total_non_file_actions = 0u64; + let mut total_predicate_filtered = 0u64; + let mut search_start = 0; + + while let Some(pos) = logs[search_start..].find("Parallel scan metadata completed") { + let absolute_pos = search_start + pos; + let remaining = &logs[absolute_pos..]; + + // Extract and accumulate metrics + total_add_files_seen += extract_metric(remaining, "add_files_seen"); + total_active_add_files += extract_metric(remaining, "active_add_files"); + total_remove_files_seen += extract_metric(remaining, "remove_files_seen"); + total_non_file_actions += extract_metric(remaining, "non_file_actions"); + total_predicate_filtered += extract_metric(remaining, "predicate_filtered"); + + // Verify timing metrics are present and parseable in parallel phase + let _dedup_time = extract_metric(remaining, "dedup_visitor_time_ns"); + let _predicate_eval_time = extract_metric(remaining, "predicate_eval_time_ns"); + + search_start = absolute_pos + 1; } + + // Verify accumulated totals match expected values + assert_eq!( + total_add_files_seen, expected.add_files_seen, + "Parallel add_files_seen mismatch" + ); + assert_eq!( + total_active_add_files, expected.active_add_files, + "Parallel active_add_files mismatch" + ); + assert_eq!( + total_remove_files_seen, expected.remove_files_seen, + "Parallel remove_files_seen mismatch" + ); + assert_eq!( + total_non_file_actions, expected.non_file_actions, + "Parallel non_file_actions mismatch" + ); + assert_eq!( + total_predicate_filtered, expected.predicate_filtered, + "Parallel predicate_filtered mismatch" + ); } + } + + /// Tests parallel workflow with sidecars and verifies metrics logging. + /// + /// This parameterized test covers both JSON and Parquet checkpoint sidecars, + /// with all combinations of serialization and worker configurations. + /// + /// Note: This test captures logs from spawned threads by sharing the tracing dispatcher. + /// If running with other tests in parallel causes flakiness, use `--test-threads=1`. + #[rstest::rstest] + #[case::json_sidecars(ParallelLogReplayCase { + path: "v2-checkpoints-json-with-sidecars", + predicate: None, + expected_sequential_metrics: ExpectedMetrics { + add_files_seen: 0, + active_add_files: 0, + remove_files_seen: 0, + non_file_actions: 5, + predicate_filtered: 0, + }, + expected_parallel_metrics: Some(ExpectedMetrics { + add_files_seen: 101, + active_add_files: 101, + remove_files_seen: 0, + non_file_actions: 0, + predicate_filtered: 0, + }), + })] + #[case::parquet_sidecars(ParallelLogReplayCase { + path: "v2-checkpoints-parquet-with-sidecars", + predicate: None, + expected_sequential_metrics: ExpectedMetrics { + add_files_seen: 0, + active_add_files: 0, + remove_files_seen: 0, + non_file_actions: 5, + predicate_filtered: 0, + }, + expected_parallel_metrics: Some(ExpectedMetrics { + add_files_seen: 101, + active_add_files: 101, + remove_files_seen: 0, + non_file_actions: 0, + predicate_filtered: 0, + }), + })] + #[case::data_skipping(ParallelLogReplayCase { + // Tests data skipping filtering based on column stats (min/max values) + path: "v2-checkpoints-json-with-sidecars", + predicate: Some({ + use crate::expressions::{col, lit, Expression as Expr}; + Arc::new(Expr::gt(col!("id"), lit(20i64))) + }), + expected_sequential_metrics: ExpectedMetrics { + add_files_seen: 0, + active_add_files: 0, + remove_files_seen: 0, + non_file_actions: 5, + predicate_filtered: 0, + }, + // Data skipping predicate filters 4 files (101 -> 97). + // add_files_seen counts files AFTER data skipping. + expected_parallel_metrics: Some(ExpectedMetrics { + add_files_seen: 97, + active_add_files: 97, + remove_files_seen: 0, + non_file_actions: 0, + predicate_filtered: 4, + }), + })] + #[case::partition_pruning(ParallelLogReplayCase { + // Tests partition pruning filtering based on partition column values. + // Table is partitioned by 'letter' with partitions: a, b, c, e, null. + // Predicate letter='a' prunes 4 files (b, c, e, null), leaving 2 letter=a files. + // All 4 non-matching files are pruned by the columnar DataSkippingFilter. The is_add + // guard (OR(NOT is_add, pred)) only protects Remove/non-file rows, not Adds with null + // partition values -- those are correctly filtered since is_add=true for them. + path: "basic_partitioned", + predicate: Some({ + use crate::expressions::{col, lit, Expression as Expr}; + Arc::new(Expr::eq(col!("letter"), lit("a"))) + }), + expected_sequential_metrics: ExpectedMetrics { + // Columnar filter prunes all 4 non-matching files (b, c, e, null) before the + // visitor. The is_add guard protects Removes but not null-partition Adds. + add_files_seen: 2, + active_add_files: 2, + remove_files_seen: 0, + non_file_actions: 4, + predicate_filtered: 4, + }, + // No parallel phase (no V2 checkpoint with sidecars) + expected_parallel_metrics: None, + })] + #[case::json_without_sidecars(ParallelLogReplayCase { + path: "v2-checkpoints-json-without-sidecars", + predicate: None, + expected_sequential_metrics: ExpectedMetrics { + add_files_seen: 3, + active_add_files: 3, + remove_files_seen: 0, + non_file_actions: 4, + predicate_filtered: 0, + }, + expected_parallel_metrics: None, + })] + #[case::json_with_last_checkpoint(ParallelLogReplayCase { + path: "v2-checkpoints-json-with-last-checkpoint", + predicate: None, + expected_sequential_metrics: ExpectedMetrics { + add_files_seen: 0, + active_add_files: 0, + remove_files_seen: 0, + non_file_actions: 4, + predicate_filtered: 0, + }, + expected_parallel_metrics: Some(ExpectedMetrics { + add_files_seen: 2, + active_add_files: 2, + remove_files_seen: 0, + non_file_actions: 0, + predicate_filtered: 0, + }), + })] + #[case::parquet_without_sidecars(ParallelLogReplayCase { + path: "v2-checkpoints-parquet-without-sidecars", + predicate: None, + expected_sequential_metrics: ExpectedMetrics { + add_files_seen: 3, + active_add_files: 3, + remove_files_seen: 0, + non_file_actions: 4, + predicate_filtered: 0, + }, + expected_parallel_metrics: None, + })] + #[case::parquet_with_last_checkpoint(ParallelLogReplayCase { + path: "v2-checkpoints-parquet-with-last-checkpoint", + predicate: None, + expected_sequential_metrics: ExpectedMetrics { + add_files_seen: 0, + active_add_files: 0, + remove_files_seen: 0, + non_file_actions: 4, + predicate_filtered: 0, + }, + expected_parallel_metrics: Some(ExpectedMetrics { + add_files_seen: 2, + active_add_files: 2, + remove_files_seen: 0, + non_file_actions: 0, + predicate_filtered: 0, + }), + })] + #[case::v2_classic_json(ParallelLogReplayCase { + path: "v2-classic-checkpoint-json", + predicate: None, + expected_sequential_metrics: ExpectedMetrics { + add_files_seen: 0, + active_add_files: 0, + remove_files_seen: 0, + non_file_actions: 4, + predicate_filtered: 0, + }, + expected_parallel_metrics: Some(ExpectedMetrics { + add_files_seen: 4, + active_add_files: 4, + remove_files_seen: 0, + non_file_actions: 0, + predicate_filtered: 0, + }), + })] + #[case::v2_classic_parquet(ParallelLogReplayCase { + path: "v2-classic-checkpoint-parquet", + predicate: None, + expected_sequential_metrics: ExpectedMetrics { + add_files_seen: 0, + active_add_files: 0, + remove_files_seen: 0, + non_file_actions: 4, + predicate_filtered: 0, + }, + expected_parallel_metrics: Some(ExpectedMetrics { + add_files_seen: 4, + active_add_files: 4, + remove_files_seen: 0, + non_file_actions: 0, + predicate_filtered: 0, + }), + })] + #[case::no_parallel_needed(ParallelLogReplayCase { + path: "table-without-dv-small", + predicate: None, + expected_sequential_metrics: ExpectedMetrics { + // This table has single-part checkpoint, completes in sequential phase + add_files_seen: 1, + active_add_files: 1, + remove_files_seen: 0, + non_file_actions: 3, + predicate_filtered: 0, + }, + // No parallel phase needed + expected_parallel_metrics: None, + })] + #[case::with_removes_deduplication(ParallelLogReplayCase { + // This table has removes that filter checkpoint adds, showing add_files_seen > active_add_files + path: "with_checkpoint_no_last_checkpoint", + predicate: None, + expected_sequential_metrics: ExpectedMetrics { + // Checkpoint 2 contains: add B (surviving state at v2), metadata/protocol + // Commit 3 (after checkpoint) has: add C, remove B + // Log replay: process commit 3 first (add C active, remove B recorded), + // then checkpoint (add B filtered by remove) + // Result: 2 adds seen, 1 active (only C), 1 remove seen, B filtered by dedup + add_files_seen: 2, + active_add_files: 1, + remove_files_seen: 1, + non_file_actions: 4, + predicate_filtered: 0, + }, + expected_parallel_metrics: None, + })] + fn test_parallel_workflow_with_metrics( + #[case] test_case: ParallelLogReplayCase, + #[values(false, true)] with_serde: bool, + #[values(false, true)] one_file_per_worker: bool, + ) -> DeltaResult<()> { + use test_utils::LoggingTest; + + // Set up log capture + let logging_test = LoggingTest::new(); + + // Capture the dispatcher to share with spawned threads + let dispatcher = tracing::dispatcher::get_default(|d| d.clone()); + + verify_parallel_workflow( + test_case.path, + test_case.predicate, + with_serde, + one_file_per_worker, + Some(dispatcher), + )?; + + // Verify metrics were logged + let logs = logging_test.logs(); + verify_metrics_in_logs( + &logs, + test_case.path, + &test_case.expected_sequential_metrics, + test_case.expected_parallel_metrics.as_ref(), + ); + Ok(()) } @@ -551,7 +960,7 @@ mod tests { let scan = snapshot .clone() .scan_builder() - .with_skip_stats(true) + .with_stats(StatsOptions::none()) .build()?; let mut single_node_iter = scan.scan_metadata(engine.as_ref())?; let mut expected_paths = single_node_iter.try_fold(Vec::new(), |acc, metadata_res| { @@ -566,7 +975,10 @@ mod tests { expected_paths.sort(); // Run parallel workflow with skip_stats=true - let scan = snapshot.scan_builder().with_skip_stats(true).build()?; + let scan = snapshot + .scan_builder() + .with_stats(StatsOptions::none()) + .build()?; let mut sequential = scan.parallel_scan_metadata(engine.clone())?; // Verify stats is None in sequential results and collect paths @@ -610,4 +1022,97 @@ mod tests { Ok(()) } + + /// Sequential-only tables (single-part checkpoint, no sidecars) emit exactly one + /// `ScanMetadataCompleted` event with `ScanType::SequentialPhase` when `finish()` is called. + #[test] + fn sequential_done_phase_emits_sequential_scan_metadata_completed_event() -> DeltaResult<()> { + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let (engine, snapshot, _tempdir) = load_test_table("table-without-dv-small")?; + let scan = snapshot.scan_builder().build()?; + let mut sequential = scan.parallel_scan_metadata(engine)?; + for result in sequential.by_ref() { + result?; + } + assert!(matches!( + sequential.finish()?, + AfterSequentialScanMetadata::Done + )); + + let events = reporter.events(); + let scan_events: Vec<&ScanType> = events + .iter() + .filter_map(|e| match e { + MetricEvent::ScanMetadataCompleted(s) => Some(&s.scan_type), + _ => None, + }) + .collect(); + + assert_eq!( + scan_events.len(), + 1, + "expected exactly one ScanMetadataCompleted event" + ); + assert_eq!(*scan_events[0], ScanType::SequentialPhase); + Ok(()) + } + + /// Tables with v2 checkpoint sidecars go through both phases and emit two + /// `ScanMetadataCompleted` events. The `operation_id` must be the same on both + /// events so callers can correlate them. + #[test] + fn parallel_scan_emits_correlated_sequential_and_parallel_events() -> DeltaResult<()> { + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let (engine, snapshot, _tempdir) = load_test_table("v2-checkpoints-json-with-sidecars")?; + let scan = snapshot.scan_builder().build()?; + let mut sequential = scan.parallel_scan_metadata(engine.clone())?; + for result in sequential.by_ref() { + result?; + } + + let AfterSequentialScanMetadata::Parallel { state, files } = sequential.finish()? else { + panic!("expected parallel phase for v2-checkpoints-json-with-sidecars"); + }; + + let seq_op_id = reporter + .events() + .into_iter() + .find_map(|e| match e { + MetricEvent::ScanMetadataCompleted(s) + if s.scan_type == ScanType::SequentialPhase => + { + Some(s.operation_id) + } + _ => None, + }) + .expect("expected SequentialPhase ScanMetadataCompleted event after finish()"); + + let final_state = Arc::new(*state); + let mut parallel = ParallelScanMetadata::try_new(engine, final_state.clone(), files)?; + for result in parallel.by_ref() { + result?; + } + final_state.log_metrics(); + + let par_op_id = reporter + .events() + .into_iter() + .find_map(|e| match e { + MetricEvent::ScanMetadataCompleted(s) if s.scan_type == ScanType::ParallelPhase => { + Some(s.operation_id) + } + _ => None, + }) + .expect("expected ParallelPhase ScanMetadataCompleted event after log_metrics()"); + + assert_eq!( + seq_op_id, par_op_id, + "sequential and parallel ScanMetadataCompleted events must share the same operation_id" + ); + Ok(()) + } } diff --git a/kernel/src/parallel/parallel_scan_metadata.rs b/kernel/src/parallel/parallel_scan_metadata.rs index a52cdc6887..d98c7b1ff7 100644 --- a/kernel/src/parallel/parallel_scan_metadata.rs +++ b/kernel/src/parallel/parallel_scan_metadata.rs @@ -1,8 +1,12 @@ use std::sync::Arc; +use std::time::Instant; use delta_kernel_derive::internal_api; +use tracing::{info_span, Span}; use crate::log_replay::{ActionsBatch, ParallelLogReplayProcessor}; +use crate::metrics::events::emit_scan_metadata_completed; +use crate::metrics::{MetricId, ScanType}; use crate::parallel::parallel_phase::ParallelPhase; use crate::parallel::sequential_phase::{AfterSequential, SequentialPhase}; use crate::scan::log_replay::{ScanLogReplayProcessor, SerializableScanState}; @@ -30,19 +34,68 @@ pub enum AfterSequentialScanMetadata { /// a distributed phase is needed. pub struct SequentialScanMetadata { pub(crate) sequential: SequentialPhase, + operation_id: MetricId, + /// Opaque, caller-supplied correlation id propagated to both phases' metric events. + correlation_id: Option>, + start: Instant, + span: Span, } impl SequentialScanMetadata { - pub(crate) fn new(sequential: SequentialPhase) -> Self { - Self { sequential } + pub(crate) fn new( + sequential: SequentialPhase, + correlation_id: Option>, + ) -> Self { + let operation_id = MetricId::new(); + Self { + sequential, + operation_id, + correlation_id, + start: Instant::now(), + // TODO: Associate a unique scan ID with this span to correlate sequential and parallel + // phases + span: info_span!("sequential_scan_metadata"), + } } pub fn finish(self) -> DeltaResult { + let _guard = self.span.enter(); match self.sequential.finish()? { - AfterSequential::Done(_) => Ok(AfterSequentialScanMetadata::Done), + AfterSequential::Done(processor) => { + let event = processor.get_metrics().to_event( + self.operation_id, + processor.is_catalog_managed(), + self.correlation_id, + ScanType::SequentialPhase, + self.start.elapsed(), + ); + processor + .get_metrics() + .log("Sequential scan metadata completed"); + emit_scan_metadata_completed(&event); + Ok(AfterSequentialScanMetadata::Done) + } AfterSequential::Parallel { processor, files } => { + let event = processor.get_metrics().to_event( + self.operation_id, + processor.is_catalog_managed(), + self.correlation_id.clone(), + ScanType::SequentialPhase, + self.start.elapsed(), + ); + processor + .get_metrics() + .log("Sequential scan metadata completed"); + emit_scan_metadata_completed(&event); + processor.get_metrics().reset_counters(); + Ok(AfterSequentialScanMetadata::Parallel { - state: Box::new(ParallelState { inner: processor }), + state: Box::new(ParallelState { + inner: processor, + operation_id: self.operation_id, + correlation_id: self.correlation_id, + parallel_start: Instant::now(), + }), files, }) } @@ -54,6 +107,7 @@ impl Iterator for SequentialScanMetadata { type Item = DeltaResult; fn next(&mut self) -> Option { + let _guard = self.span.enter(); self.sequential.next() } } @@ -64,6 +118,14 @@ impl Iterator for SequentialScanMetadata { /// in Arc and shared across threads for local parallel processing. pub struct ParallelState { inner: ScanLogReplayProcessor, + /// Operation ID inherited from the sequential phase for event correlation. + operation_id: MetricId, + /// Opaque, caller-supplied correlation id inherited from the sequential phase. Does not + /// survive the serialization boundary; a reconstructed state carries `None` (tracked in + /// #2736). + correlation_id: Option>, + /// Start time for the parallel phase, set when this state is created. + parallel_start: Instant, } impl ParallelLogReplayProcessor for Arc { @@ -75,6 +137,40 @@ impl ParallelLogReplayProcessor for Arc { } impl ParallelState { + /// Log the accumulated metrics from parallel processing. + /// + /// Call this after all parallel workers complete. The metrics will be logged + /// in the current tracing span context. + /// + /// # Example + /// + /// ```no_run + /// # use std::sync::Arc; + /// # use delta_kernel::scan::ParallelState; + /// # use tracing::instrument; + /// #[instrument(skip_all, name = "parallel_scan")] + /// async fn process(state: Arc) { + /// // ... spawn workers that share Arc ... + /// // ... wait for workers to complete ... + /// + /// // Log accumulated metrics + /// state.log_metrics(); + /// } + /// ``` + pub fn log_metrics(&self) { + let event = self.inner.get_metrics().to_event( + self.operation_id, + self.inner.is_catalog_managed(), + self.correlation_id.clone(), + ScanType::ParallelPhase, + self.parallel_start.elapsed(), + ); + self.inner + .get_metrics() + .log("Parallel scan metadata completed"); + emit_scan_metadata_completed(&event); + } + /// Get the schema to use for reading checkpoint files. /// /// Returns the checkpoint read schema which may have stats excluded @@ -101,7 +197,6 @@ impl ParallelState { /// # Parameters /// - `engine`: Engine for creating evaluators and filters /// - `state`: The serialized state from a previous `into_serializable_state()` call - /// - `scan_span`: The parent scan span for tracing #[internal_api] #[allow(unused)] pub(crate) fn from_serializable_state( @@ -109,7 +204,14 @@ impl ParallelState { state: SerializableScanState, ) -> DeltaResult { let inner = ScanLogReplayProcessor::from_serializable_state(engine, state)?; - Ok(Self { inner }) + Ok(Self { + inner, + // Neither operation_id nor correlation_id survive the serialization boundary today; a + // reconstructed state starts a fresh operation_id and has no correlation_id (#2736). + operation_id: MetricId::new(), + correlation_id: None, + parallel_start: Instant::now(), + }) } /// Serialize the processor state directly to bytes. @@ -135,7 +237,6 @@ impl ParallelState { /// # Parameters /// - `engine`: Engine for creating evaluators and filters /// - `bytes`: The serialized bytes from a previous `into_bytes()` call - /// - `scan_span`: The parent scan span for tracing #[allow(unused)] pub fn from_bytes(engine: &dyn Engine, bytes: &[u8]) -> DeltaResult { let state: SerializableScanState = @@ -146,6 +247,7 @@ impl ParallelState { pub struct ParallelScanMetadata { pub(crate) processor: ParallelPhase>, + span: Span, } impl ParallelScanMetadata { @@ -157,6 +259,8 @@ impl ParallelScanMetadata { let read_schema = state.file_read_schema(); Ok(Self { processor: ParallelPhase::try_new(engine, state, leaf_files, read_schema)?, + // TODO: Associate the same scan ID from sequential phase to correlate phases + span: info_span!("parallel_scan_metadata"), }) } @@ -166,6 +270,8 @@ impl ParallelScanMetadata { ) -> Self { Self { processor: ParallelPhase::new_from_iter(state.clone(), iter), + // TODO: Associate the same scan ID from sequential phase to correlate phases + span: info_span!("parallel_scan_metadata"), } } } @@ -174,6 +280,73 @@ impl Iterator for ParallelScanMetadata { type Item = DeltaResult; fn next(&mut self) -> Option { + let _guard = self.span.enter(); self.processor.next() } } + +#[cfg(test)] +mod tests { + use std::collections::HashSet; + use std::sync::Arc; + + use super::ParallelState; + use crate::engine::sync::SyncEngine; + use crate::log_segment::CheckpointReadInfo; + use crate::metrics::{MetricEvent, ScanType, TableType}; + use crate::scan::log_replay::{ + ScanLogReplayProcessor, ScanPartitionValuesOptions, ScanStatsOptions, + }; + use crate::scan::state_info::StateInfo; + use crate::scan::PhysicalPredicate; + use crate::schema::{DataType, SchemaRef, StructField, StructType}; + use crate::table_features::ColumnMappingMode; + use crate::unit_test_utils::{install_thread_local_metrics_reporter, CapturingReporter}; + + #[test] + fn test_parallel_state_log_metrics_carries_round_tripped_table_type() { + let engine = SyncEngine::new(); + let schema: SchemaRef = Arc::new(StructType::new_unchecked([StructField::new( + "id", + DataType::INTEGER, + true, + )])); + let state_info = Arc::new(StateInfo { + logical_schema: schema.clone(), + physical_schema: schema, + physical_predicate: PhysicalPredicate::None, + transform_spec: None, + column_mapping_mode: ColumnMappingMode::None, + physical_stats_schema: None, + physical_partition_schema: None, + physical_stats_columns: HashSet::new(), + is_catalog_managed: true, + skip_row_transforms: false, + }); + let processor = ScanLogReplayProcessor::new( + &engine, + state_info, + CheckpointReadInfo::without_stats_parsed(), + ScanStatsOptions::default(), + ScanPartitionValuesOptions::default(), + ) + .unwrap(); + let serialized = processor.into_serializable_state().unwrap(); + let state = ParallelState::from_serializable_state(&engine, serialized).unwrap(); + + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + state.log_metrics(); + + let event = reporter + .events() + .into_iter() + .find_map(|e| match e { + MetricEvent::ScanMetadataCompleted(c) => Some(c), + _ => None, + }) + .expect("ScanMetadataCompleted emitted"); + assert_eq!(event.table_type, TableType::CatalogManaged); + assert_eq!(event.scan_type, ScanType::ParallelPhase); + } +} diff --git a/kernel/src/parallel/sequential_phase.rs b/kernel/src/parallel/sequential_phase.rs index de395fc88e..ebb09756b5 100644 --- a/kernel/src/parallel/sequential_phase.rs +++ b/kernel/src/parallel/sequential_phase.rs @@ -11,6 +11,7 @@ use std::sync::Arc; +use delta_kernel_derive::internal_api; use itertools::Itertools; use crate::log_reader::checkpoint_manifest::CheckpointManifestReader; @@ -20,7 +21,6 @@ use crate::log_segment::LogSegment; use crate::scan::COMMIT_READ_SCHEMA; use crate::utils::require; use crate::{DeltaResult, Engine, Error, FileMeta}; -use delta_kernel_derive::internal_api; /// Sequential log replay processor for parallel execution. /// @@ -106,6 +106,7 @@ impl SequentialPhase

    { engine.as_ref(), log_segment, COMMIT_READ_SCHEMA.clone(), + None, )?); // Concurrently start reading the checkpoint manifest. Only create a checkpoint manifest @@ -141,8 +142,7 @@ impl SequentialPhase

    { /// /// # Returns /// - `Done`: All processing done sequentially - no parallel phase needed - /// - `Parallel`: Parallel phase needed. The resulting files may be processed - /// in parallel. + /// - `Parallel`: Parallel phase needed. The resulting files may be processed in parallel. /// /// # Errors /// Returns an error if called before iterator exhaustion. @@ -207,7 +207,7 @@ impl Iterator for SequentialPhase

    { mod tests { use super::*; use crate::scan::AfterSequentialScanMetadata; - use crate::utils::test_utils::{assert_result_error_with_message, load_test_table}; + use crate::unit_test_utils::{assert_result_error_with_message, load_test_table}; /// Core helper function to verify sequential processing with expected adds and sidecars. fn verify_sequential_processing( @@ -340,8 +340,8 @@ mod tests { fn test_sequential_checkpoint_no_commits() -> DeltaResult<()> { verify_sequential_processing( "with_checkpoint_no_last_checkpoint", - &["part-00000-70b1dcdf-0236-4f63-a072-124cdbafd8a0-c000.snappy.parquet"], // Add from commit 3 - &[], // No sidecars + &["part-00000-70b1dcdf-0236-4f63-a072-124cdbafd8a0-c000.snappy.parquet"], /* Add from commit 3 */ + &[], // No sidecars ) } } diff --git a/kernel/src/partition/hive.rs b/kernel/src/partition/hive.rs new file mode 100644 index 0000000000..73d7acc6ed --- /dev/null +++ b/kernel/src/partition/hive.rs @@ -0,0 +1,684 @@ +//! Hive-style partition path encoding. +//! +//! Percent-encodes partition column names and values for filesystem directory paths +//! (e.g., `region=US%2FEast/year=2024/`). Matches Hive's [`FileUtils.escapePathName`][hive] +//! and Spark's [`ExternalCatalogUtils.escapePathName`][spark]. +//! +//! ```text +//! Step 2 (serialization): Scalar::String("US/East") --> "US/East" (partitionValues) +//! Step 3 (THIS MODULE): "US/East" --> "US%2FEast" (directory name) +//! ``` +//! +//! # Encoding rules +//! +//! Encodes: ASCII control chars (0x00-0x1F), DEL (0x7F), +//! `"` `#` `%` `'` `*` `/` `:` `=` `?` `\` `{` `[` `]` `^`. +//! +//! NOT encoded: space (0x20), non-ASCII (>= 0x80), `}`. +//! +//! On Windows, space (0x20), `<`, `>`, and `|` are additionally encoded to match +//! Spark's platform-specific behavior in `ExternalCatalogUtils.escapePathName`. +//! +//! See the encoding tables in the [`super`] module for comprehensive examples. +//! +//! [hive]: https://github.com/apache/hive/blob/trunk/common/src/java/org/apache/hadoop/hive/common/FileUtils.java +//! [spark]: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala + +use std::borrow::Cow; + +use percent_encoding::{utf8_percent_encode, AsciiSet, CONTROLS}; + +const HEX_UPPER: &[u8; 16] = b"0123456789ABCDEF"; + +/// Characters encoded by [`uri_encode_path`]: `%`, space, and ASCII chars illegal in a +/// URI path, plus all ASCII controls via [`CONTROLS`]. Non-ASCII UTF-8 bytes are ALSO +/// percent-encoded by `utf8_percent_encode` (each byte of the UTF-8 sequence becomes +/// `%XX`), even though they are not in the [`AsciiSet`]. For example: +/// `uri_encode_path("München") == "M%C3%BCnchen"`, because `ü` is `0xC3 0xBC` in UTF-8. +/// +/// TODO(#2423): Delta-Spark leaves non-ASCII bytes raw in `add.path` (e.g. `München` +/// stays as `München`). Our current behavior diverges — revisit once kernel matches +/// Delta-Spark's `add.path` encoding for non-ASCII. +const HADOOP_URI_PATH_ENCODE_SET: &AsciiSet = &CONTROLS + .add(b' ') + .add(b'"') + .add(b'#') + .add(b'%') + .add(b'<') + .add(b'>') + .add(b'?') + .add(b'[') + .add(b'\\') + .add(b']') + .add(b'^') + .add(b'`') + .add(b'{') + .add(b'|') + .add(b'}'); + +/// URI percent-encodes a Hive-escaped path to produce the string that belongs in +/// `add.path`. After the Hive layer turns a partition value like `"a:b"` into `"a%3Ab"`, +/// this turns that into `"a%253Ab"` — a valid URI whose single RFC 2396 decode yields the +/// literal filesystem directory name `"a%3Ab"`. +/// +/// Matches Hadoop `Path.toUri().toString()` for the ASCII Hive-escaped inputs produced by +/// [`build_partition_path`], as used by kernel-java and Delta-Spark. Returns +/// [`Cow::Borrowed`] when no encoding is needed (zero allocation fast path). +/// See the module-level documentation in [`super`] for the full pipeline. +pub(crate) fn uri_encode_path(hive_escaped: &str) -> Cow<'_, str> { + utf8_percent_encode(hive_escaped, HADOOP_URI_PATH_ENCODE_SET).into() +} + +/// Placeholder for missing partition values in Hive-style directory paths. +/// +/// Used when a partition column is null, empty string, or empty binary. +/// +/// ```text +/// INSERT INTO t VALUES (1, NULL); --> p=__HIVE_DEFAULT_PARTITION__/... +/// INSERT INTO t VALUES (2, ''); --> p=__HIVE_DEFAULT_PARTITION__/... +/// INSERT INTO t VALUES (3, 'US'); --> p=US/... +/// ``` +pub(crate) const HIVE_DEFAULT_PARTITION: &str = "__HIVE_DEFAULT_PARTITION__"; +const HIVE_DEFAULT_PARTITION_LEN: usize = HIVE_DEFAULT_PARTITION.len(); + +/// Percent-encodes a string for use in a Hive-style partition path segment. +/// +/// Returns `Cow::Borrowed` when no encoding is needed (zero allocation fast path). +/// +/// Examples: `"US"` passes through unchanged, `"Serbia/srb%"` becomes +/// `"Serbia%2Fsrb%25"`, `"a=b"` becomes `"a%3Db"`. +pub(crate) fn escape_partition_value(s: &str) -> Cow<'_, str> { + let first = s.bytes().position(needs_escaping); + let Some(first) = first else { + return Cow::Borrowed(s); + }; + + // Each escaped byte expands from 1 byte to 3 (e.g., "/" -> "%2F"), adding 2 extra. + // One pass over the string gives us the exact allocation size. + let extra = s.bytes().filter(|b| needs_escaping(*b)).count() * 2; + let mut out = String::with_capacity(s.len() + extra); + out.push_str(&s[..first]); + + // All escapable bytes are ASCII (< 0x80). UTF-8 continuation bytes are >= 0x80, so + // iterating by char correctly preserves multi-byte sequences. For example, in + // "a/\u{00FC}" the slash is a single byte 0x2F (escaped), but \u{00FC} is two bytes + // [0xC3, 0xBC] which must be emitted together. Byte-level iteration would split them. + for c in s[first..].chars() { + if c.is_ascii() && needs_escaping(c as u8) { + let b = c as u8; + out.push('%'); + out.push(HEX_UPPER[(b >> 4) as usize] as char); + out.push(HEX_UPPER[(b & 0x0F) as usize] as char); + } else { + out.push(c); + } + } + Cow::Owned(out) +} + +/// Builds a Hive-style partition path prefix from column names and serialized values. +/// +/// Returns `col1=val1/col2=val2/` with both names and values percent-encoded. +/// Missing values (`None` or empty string) use [`HIVE_DEFAULT_PARTITION`]. +/// +/// ```ignore +/// use crate::partition::hive::build_partition_path; +/// +/// let path = build_partition_path(&[("country", Some("US")), ("year", Some("2025"))]); +/// assert_eq!(path, "country=US/year=2025/"); +/// +/// let null_path = build_partition_path(&[("col", None)]); +/// let empty_path = build_partition_path(&[("col", Some(""))]); +/// assert_eq!(null_path, empty_path); +/// assert_eq!(null_path, "col=__HIVE_DEFAULT_PARTITION__/"); +/// ``` +pub(crate) fn build_partition_path(columns: &[(&str, Option<&str>)]) -> String { + // Lower-bound capacity: exact when no escaping needed (the common case for + // partition names and most values like dates, integers, short strings). + let cap: usize = columns + .iter() + .map(|(n, v)| n.len() + 1 + v.map_or(HIVE_DEFAULT_PARTITION_LEN, str::len) + 1) + .sum(); + let mut path = String::with_capacity(cap); + for (name, value) in columns { + path.push_str(&escape_partition_value(name)); + path.push('='); + match value { + Some(v) if !v.is_empty() => path.push_str(&escape_partition_value(v)), + _ => path.push_str(HIVE_DEFAULT_PARTITION), + } + path.push('/'); + } + path +} + +// ============================================================================ +// Helpers +// ============================================================================ + +/// Returns true if the byte must be percent-encoded in a Hive partition path segment. +/// Matches the escape set from Hive's `FileUtils.escapePathName` and Spark's +/// `ExternalCatalogUtils.escapePathName`. +/// +/// `}` (0x7D) is NOT escaped, matching Hive. Only `{` is in the set. +/// +/// On Windows, additional characters are escaped. See the module-level encoding rules. +fn needs_escaping(b: u8) -> bool { + if matches!( + b, + 0x00..=0x1F + | b'"' + | b'#' + | b'%' + | b'\'' + | b'*' + | b'/' + | b':' + | b'=' + | b'?' + | b'\\' + | 0x7F + | b'{' + | b'[' + | b']' + | b'^' + ) { + return true; + } + #[cfg(target_os = "windows")] + if matches!(b, b' ' | b'<' | b'>' | b'|') { + return true; + } + false +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + + /// Inverse of `escape_partition_value` for round-trip testing. + /// Ported from Hive's `FileUtils.unescapePathName`. + fn unescape_path_name(path: &str) -> String { + let Some(first) = path.find('%') else { + return path.to_owned(); + }; + let bytes = path.as_bytes(); + let mut out = Vec::with_capacity(bytes.len()); + out.extend_from_slice(&bytes[..first]); + let mut i = first; + while i < bytes.len() { + if bytes[i] == b'%' && i + 2 < bytes.len() { + if let (Some(hi), Some(lo)) = (from_hex(bytes[i + 1]), from_hex(bytes[i + 2])) { + out.push(hi << 4 | lo); + i += 3; + continue; + } + } + out.push(bytes[i]); + i += 1; + } + String::from_utf8(out).expect("unescape produced invalid UTF-8") + } + + fn from_hex(b: u8) -> Option { + match b { + b'0'..=b'9' => Some(b - b'0'), + b'A'..=b'F' => Some(b - b'A' + 10), + b'a'..=b'f' => Some(b - b'a' + 10), + _ => None, + } + } + + // ============================================================================ + // escape_partition_value + // ============================================================================ + + /// Rows from the encoding table in `partition/mod.rs`. + /// Rows 50/52 (non-UTF-8 binary like X'DEADBEEF', X'00FF') cannot be tested here + /// because Rust's `&str` requires valid UTF-8, which prevents constructing those inputs. + /// Rows 49/65/66 (empty/null) are tested via `build_partition_path` below. + #[rstest] + // Rows 47-48: NUL bytes. Delta-Spark fails at mkdirs, but our encoding succeeds. + #[case("\x00", "%00")] + #[case("before\x00after", "before%00after")] + #[case("HELLO", "HELLO")] // 51 + #[case("/=%", "%2F%3D%25")] // 53 + #[case("a{b", "a%7Bb")] // 54 + #[case("a}b", "a}b")] // 55 + #[case("M\u{00FC}nchen", "M\u{00FC}nchen")] // 57 + #[case("\u{65E5}\u{672C}\u{8A9E}", "\u{65E5}\u{672C}\u{8A9E}")] // 58 + #[case("\u{1F3B5}\u{1F3B6}", "\u{1F3B5}\u{1F3B6}")] // 59 + #[case("a@b!c(d)", "a@b!c(d)")] // 61 + #[case("a&b+c$d;e,f", "a&b+c$d;e,f")] // 62 + #[case("Serbia/srb%", "Serbia%2Fsrb%25")] // 63 + #[case("100%25", "100%2525")] // 64 + fn test_escape_table_rows(#[case] input: &str, #[case] expected: &str) { + assert_eq!(escape_partition_value(input), expected); + } + + /// Individual chars from the Hive escape set. + #[rstest] + #[case("\x00", "%00")] + #[case("\x01", "%01")] + #[case("\t", "%09")] + #[case("\n", "%0A")] + #[case("\r", "%0D")] + #[case("\"", "%22")] + #[case("#", "%23")] + #[case("%", "%25")] + #[case("'", "%27")] + #[case("*", "%2A")] + #[case("/", "%2F")] + #[case(":", "%3A")] + #[case("=", "%3D")] + #[case("?", "%3F")] + #[case("\\", "%5C")] + #[case("[", "%5B")] + #[case("]", "%5D")] + #[case("^", "%5E")] + #[case("{", "%7B")] + #[case("\x7F", "%7F")] + fn test_escape_individual_special_chars(#[case] input: &str, #[case] expected: &str) { + assert_eq!(escape_partition_value(input), expected); + } + + /// Chars NOT in the escape set on any platform. + #[rstest] + #[case("", "")] + #[case("US", "US")] + #[case("2024-01-15", "2024-01-15")] + #[case("}", "}")] + #[case("~", "~")] + #[case("@", "@")] + #[case("!", "!")] + #[case("(", "(")] + #[case(")", ")")] + #[case("&", "&")] + #[case("+", "+")] + #[case("$", "$")] + #[case(";", ";")] + #[case(",", ",")] + fn test_escape_passthrough(#[case] input: &str, #[case] expected: &str) { + assert_eq!(escape_partition_value(input), expected); + } + + /// Mixed strings: mid-string encoding, index-zero encoding, double-encoding. + #[rstest] + #[case("a/b", "a%2Fb")] + #[case("a=b", "a%3Db")] + #[case("100%", "100%25")] + #[case("/abc", "%2Fabc")] + #[case("\x01abc", "%01abc")] + #[case("{}", "%7B}")] + #[case("a{b}c", "a%7Bb}c")] + #[case("%%", "%25%25")] + #[case("a%b%c", "a%25b%25c")] + #[case("%2F", "%252F")] + #[case("%3D", "%253D")] + #[case("region=us/east#1", "region%3Dus%2Feast%231")] + fn test_escape_mixed(#[case] input: &str, #[case] expected: &str) { + assert_eq!(escape_partition_value(input), expected); + } + + /// Non-ASCII preserved after escaped chars, including 2/3/4-byte UTF-8. + #[rstest] + #[case("a/\u{00FC}", "a%2F\u{00FC}")] + #[case("M\u{00FC}nchen/Bayern", "M\u{00FC}nchen%2FBayern")] + #[case("/\u{00FC}", "%2F\u{00FC}")] + #[case("/\u{20AC}", "%2F\u{20AC}")] + #[case("/\u{1D11E}", "%2F\u{1D11E}")] + #[case( + "\u{65E5}\u{672C}=\u{6771}\u{4EAC}", + "\u{65E5}\u{672C}%3D\u{6771}\u{4EAC}" + )] + #[case("\u{1F3B5}/\u{1F3B6}", "\u{1F3B5}%2F\u{1F3B6}")] + #[case("/\u{00E4}\u{00E4}", "%2F\u{00E4}\u{00E4}")] + fn test_escape_unicode_after_special(#[case] input: &str, #[case] expected: &str) { + assert_eq!(escape_partition_value(input), expected); + } + + /// Cow::Borrowed fast path when no encoding is needed. + #[rstest] + #[case("hello")] + #[case("\u{00FC}ber")] + fn test_escape_fast_path_returns_borrowed(#[case] input: &str) { + assert!(matches!(escape_partition_value(input), Cow::Borrowed(_))); + } + + // ============================================================================ + // Exhaustive / combinatorial + // ============================================================================ + + /// Every ASCII byte (0x00-0x7F) individually, verifying the exact Hive escape set. + #[test] + fn test_escape_every_ascii_byte() { + #[allow(unused_mut)] + let mut must_escape: Vec = vec![ + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, + 0x0E, 0x0F, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1A, 0x1B, + 0x1C, 0x1D, 0x1E, 0x1F, b'"', b'#', b'%', b'\'', b'*', b'/', b':', b'=', b'?', b'\\', + 0x7F, b'{', b'[', b']', b'^', + ]; + #[cfg(target_os = "windows")] + must_escape.extend_from_slice(b" <>|"); + for byte in 0x00..=0x7Fu8 { + let input = String::from(byte as char); + let result = escape_partition_value(&input); + if must_escape.contains(&byte) { + assert_eq!(result, format!("%{:02X}", byte), "byte 0x{byte:02X}"); + } else { + assert_eq!(result, input, "byte 0x{byte:02X}"); + } + } + } + + /// Every ASCII byte preceded by `/` (exercises slow path for all second bytes). + #[test] + fn test_escape_slash_followed_by_every_ascii() { + for second in 0x01..=0x7Fu8 { + let input = format!("/{}", second as char); + let result = escape_partition_value(&input); + assert!(result.starts_with("%2F"), "byte 0x{second:02X}: {result:?}"); + if needs_escaping(second) { + assert_eq!(result, format!("%2F%{:02X}", second), "byte 0x{second:02X}"); + } else { + assert_eq!( + result, + format!("%2F{}", second as char), + "byte 0x{second:02X}" + ); + } + } + } + + /// Non-ASCII (u-umlaut) after every escaped char. + #[test] + fn test_escape_non_ascii_after_every_special() { + for &byte in b"\"#%'*/:=?\\{[]^" { + let input = format!("{}\u{00FC}", byte as char); + let expected = format!("%{:02X}\u{00FC}", byte); + assert_eq!( + escape_partition_value(&input), + expected, + "byte 0x{byte:02X}" + ); + } + } + + // ============================================================================ + // Platform-sensitive chars (space, <, >, |) + // ============================================================================ + + /// On non-Windows, space//| pass through unescaped (rows 56, 60, 67, 68). + #[cfg(not(target_os = "windows"))] + #[test] + fn test_escape_platform_sensitive_chars_pass_through() { + assert_eq!(escape_partition_value(" "), " "); + assert_eq!(escape_partition_value(" "), " "); + assert_eq!(escape_partition_value("hello world"), "hello world"); + assert_eq!(escape_partition_value("<"), "<"); + assert_eq!(escape_partition_value(">"), ">"); + assert_eq!(escape_partition_value("|"), "|"); + assert_eq!(escape_partition_value("ac|d"), "ac|d"); + assert_eq!( + escape_partition_value("2024-01-15 12:30:45"), + "2024-01-15 12%3A30%3A45" + ); + } + + /// On non-Windows, build_partition_path preserves spaces (rows 56, 67, 68). + #[cfg(not(target_os = "windows"))] + #[test] + fn test_build_path_platform_sensitive_chars_pass_through() { + assert_eq!( + build_partition_path(&[("p", Some("hello world"))]), + "p=hello world/" + ); + assert_eq!(build_partition_path(&[("p", Some(" "))]), "p= /"); + assert_eq!(build_partition_path(&[("p", Some(" "))]), "p= /"); + } + + /// On Windows, space//| are escaped. + #[cfg(target_os = "windows")] + #[test] + fn test_escape_windows_special_chars_are_escaped() { + assert_eq!(escape_partition_value(" "), "%20"); + assert_eq!(escape_partition_value(" "), "%20%20"); + assert_eq!(escape_partition_value("hello world"), "hello%20world"); + assert_eq!(escape_partition_value("<"), "%3C"); + assert_eq!(escape_partition_value(">"), "%3E"); + assert_eq!(escape_partition_value("|"), "%7C"); + assert_eq!(escape_partition_value("ac|d"), "a%3Cb%3Ec%7Cd"); + assert_eq!( + escape_partition_value("2024-01-15 12:30:45"), + "2024-01-15%2012%3A30%3A45" + ); + } + + /// On Windows, build_partition_path escapes spaces. + #[cfg(target_os = "windows")] + #[test] + fn test_build_path_windows_escapes_spaces() { + assert_eq!( + build_partition_path(&[("p", Some("hello world"))]), + "p=hello%20world/" + ); + assert_eq!(build_partition_path(&[("p", Some(" "))]), "p=%20/"); + assert_eq!(build_partition_path(&[("p", Some(" "))]), "p=%20%20/"); + } + + // ============================================================================ + // Round-trip: unescape(escape(s)) == s + // ============================================================================ + + #[test] + fn test_round_trip_every_ascii_byte() { + for byte in 0x01..=0x7Fu8 { + let input = String::from(byte as char); + let rt = unescape_path_name(&escape_partition_value(&input)); + assert_eq!(rt, input, "byte 0x{byte:02X}"); + } + } + + #[test] + fn test_round_trip_mixed_strings() { + for input in [ + "hello", + "", + "a/b", + "key=value", + "100%", + "region=us/east#1", + "2024-01-15 12:30:45", + "Serbia/srb%", + "a\"b#c%d'e*f/g:h=i?j\\k", + "path/with spaces/and=equals", + " spaces ", + "tab\there", + "newline\nhere", + ] { + assert_eq!( + unescape_path_name(&escape_partition_value(input)), + input, + "{input:?}" + ); + } + } + + #[test] + fn test_round_trip_unicode() { + for input in [ + "\u{00FC}ber", + "\u{65E5}\u{672C}\u{8A9E}", + "M\u{00FC}nchen/Bayern", + "\u{65E5}\u{672C}=\u{6771}\u{4EAC}", + "\u{1F3B5}/\u{1F3B6}", + "a/\u{00FC}ber", + "/\u{20AC}100", + "caf\u{00E9}=\u{2615}", + ] { + assert_eq!( + unescape_path_name(&escape_partition_value(input)), + input, + "{input:?}" + ); + } + } + + #[test] + fn test_round_trip_two_char_combinations() { + for first in 0x01..=0x7Fu8 { + for second in 0x01..=0x7Fu8 { + let input = format!("{}{}", first as char, second as char); + let rt = unescape_path_name(&escape_partition_value(&input)); + assert_eq!(rt, input, "(0x{first:02X}, 0x{second:02X})"); + } + } + } + + /// Ported from Hive's `TestFileUtils.testPathEscapeChars`. + #[test] + fn test_round_trip_all_escaped_chars_combined() { + let mut input = String::new(); + for byte in 0x00..=0x7Fu8 { + if needs_escaping(byte) { + input.push(byte as char); + } + } + assert_eq!(unescape_path_name(&escape_partition_value(&input)), input); + } + + // ============================================================================ + // build_partition_path + // ============================================================================ + + /// Rows from the encoding table in `partition/mod.rs`. + #[rstest] + #[case("p", Some(""), "p=__HIVE_DEFAULT_PARTITION__/")] // 49: empty binary + #[case("p", Some("/=%"), "p=%2F%3D%25/")] // 53 + #[case("p", Some("a{b"), "p=a%7Bb/")] // 54 + #[case("p", Some("a}b"), "p=a}b/")] // 55 + #[case("p", Some("M\u{00FC}nchen"), "p=M\u{00FC}nchen/")] // 57 + #[case("p", Some("Serbia/srb%"), "p=Serbia%2Fsrb%25/")] // 63 + #[case("p", Some("100%25"), "p=100%2525/")] // 64 + #[case("p", None, "p=__HIVE_DEFAULT_PARTITION__/")] // 66: NULL + fn test_build_path_table_rows( + #[case] name: &str, + #[case] value: Option<&str>, + #[case] expected: &str, + ) { + assert_eq!(build_partition_path(&[(name, value)]), expected); + } + + /// Additional single-column cases. + #[rstest] + #[case("country", Some("US"), "country=US/")] + #[case("country", Some("US/A%B"), "country=US%2FA%25B/")] + #[case("col=name", Some("value"), "col%3Dname=value/")] + fn test_build_path_single_column( + #[case] name: &str, + #[case] value: Option<&str>, + #[case] expected: &str, + ) { + assert_eq!(build_partition_path(&[(name, value)]), expected); + } + + #[test] + fn test_build_path_empty_columns_returns_empty() { + assert_eq!(build_partition_path(&[]), ""); + } + + #[test] + fn test_build_path_multiple_columns_preserves_order() { + assert_eq!( + build_partition_path(&[("country", Some("US")), ("year", Some("2025"))]), + "country=US/year=2025/" + ); + } + + #[test] + fn test_build_path_mixed_null_and_values_preserves_order() { + assert_eq!( + build_partition_path(&[ + ("year", Some("2025")), + ("region", None), + ("country", Some("US")), + ]), + "year=2025/region=__HIVE_DEFAULT_PARTITION__/country=US/" + ); + } + + // ============================================================================ + // uri_encode_path + // ============================================================================ + + /// Every ASCII byte: chars in `HADOOP_URI_PATH_ENCODE_SET` (space, `"`, `#`, `%`, `<`, + /// `>`, `?`, `[`, `\`, `]`, `^`, backtick, `{`, `|`, `}`) plus all controls encode to + /// `%XX`; everything else passes through unchanged. + #[test] + fn test_uri_encode_path_every_ascii_byte() { + let must_encode: &[u8] = b" \"#%<>?[\\]^`{|}"; + for byte in 0x00..=0x7Fu8 { + let input = String::from(byte as char); + let result = uri_encode_path(&input); + let is_control = byte <= 0x1F || byte == 0x7F; + if must_encode.contains(&byte) || is_control { + assert_eq!(result, format!("%{:02X}", byte), "byte 0x{byte:02X}"); + } else { + assert_eq!(result, input, "byte 0x{byte:02X}"); + } + } + } + + /// Multi-char contexts for URI-only chars, pinning that surrounding unreserved chars + /// pass through while the target gets encoded. Per-byte encoding decisions are covered + /// exhaustively by `test_uri_encode_path_every_ascii_byte`; this guards against a + /// refactor that only fails at character boundaries. + #[rstest] + #[case::backtick("a`b", "a%60b")] + #[case::right_brace("a}b", "a%7Db")] + #[case::double_quote("a\"b", "a%22b")] + #[case::space("hello world", "hello%20world")] + #[case::less_than("ab", "a%3Eb")] + #[case::pipe("a|b", "a%7Cb")] + fn test_uri_encode_path_encodes_uri_only_set(#[case] input: &str, #[case] expected: &str) { + assert_eq!(uri_encode_path(input), expected); + } + + /// Composing Hive then URI encoding produces the `add.path` form. Pins the full + /// pipeline end-to-end, including mixed cases where a string contains both Hive- and + /// URI-encoded bytes after each step. + #[rstest] + #[case::colon("a:b", "a%3Ab", "a%253Ab")] + #[case::slash("US/East", "US%2FEast", "US%252FEast")] + #[case::percent("100%", "100%25", "100%2525")] + #[case::slash_percent("Serbia/srb%", "Serbia%2Fsrb%25", "Serbia%252Fsrb%2525")] + fn test_uri_encode_path_composes_with_hive( + #[case] raw: &str, + #[case] hive: &str, + #[case] add_path: &str, + ) { + let hive_escaped = escape_partition_value(raw); + assert_eq!(hive_escaped, hive); + assert_eq!(uri_encode_path(&hive_escaped), add_path); + } + + /// Unreserved strings pass through byte-identical AND via the `Cow::Borrowed` fast + /// path, so no allocation occurs on the common partition-path input (dates, integers, + /// short alnum strings, the Hive null-partition placeholder). + #[rstest] + #[case::empty("")] + #[case::alnum("hello")] + #[case::partition_path("p=abc/q=def/")] + #[case::hive_default_partition("__HIVE_DEFAULT_PARTITION__")] + #[case::alnum_unreserved("A-Za-z0-9_.~")] + #[case::date("2024-01-15")] + fn test_uri_encode_path_unreserved_borrows_and_passes_through(#[case] input: &str) { + let result = uri_encode_path(input); + assert_eq!(result, input); + assert!(matches!(result, Cow::Borrowed(_))); + } +} diff --git a/kernel/src/partition/mod.rs b/kernel/src/partition/mod.rs new file mode 100644 index 0000000000..bdecc7271f --- /dev/null +++ b/kernel/src/partition/mod.rs @@ -0,0 +1,167 @@ +//! Partition utilities for Delta table writes. +//! +//! A partition value goes through multiple transformations before it reaches the Delta +//! log. Each step is handled by a separate submodule. The differences between steps are +//! subtle and easy to get wrong, so this documentation is thorough with examples observed +//! from real Spark-written Delta tables. +//! +//! # Steps +//! +//! ```text +//! Scalar value (e.g., String("US/East")) +//! | +//! | Step 1: validation --> (type/key checks against table schema) +//! | Step 2: serialization --> "US/East" (add.partitionValues) +//! | Step 3: path encoding --> "US%2FEast" (filesystem directory name) +//! | Step 4: URI encoding --> "US%252FEast" (add.path in the Delta log) +//! v +//! AddFile { partitionValues: {"region": "US/East"}, path: "region=US%252FEast/..." } +//! ``` +//! +//! - **Step 1: Validation** (`validation` module): checks partition column key completeness and +//! value types against the table schema. Runs first to reject bad input before doing any +//! serialization work. +//! +//! - **Step 2: Serialization** (`serialization` module): converts validated typed `Scalar` values +//! to protocol-compliant strings for `add.partitionValues`. Null, empty string, and empty binary +//! all serialize as JSON `null`. +//! +//! - **Step 3: Path encoding** (`hive` module): percent-encodes serialized strings for Hive-style +//! filesystem directory names. Only a small set of ASCII characters is encoded; spaces and +//! non-ASCII pass through raw. Hive-style paths are optional; the Delta protocol does not require +//! them. Kernel uses them for partitioned tables without column mapping enabled. Happens lazily +//! in the write path. +//! +//! - **Step 4: URI encoding**: an additional layer of URI encoding per RFC 2396 applied when +//! constructing `add.path` in the Delta log. More aggressive than step 3: characters like space +//! (0x20), `<`, `>`, `{`, `}`, and `|` that survive Hive encoding get percent-encoded. Characters +//! already percent-encoded by step 3 (e.g., `%2F`) get double-encoded (e.g., `%252F`). +//! +//! # Encoding by type +//! +//! Shows how each data type serializes across all three forms. Numeric and date types are +//! straightforward (no special characters to encode), but are included here for +//! completeness since the string representation in `partitionValues` must match exactly. +//! +//! ```text +//! Row | Type | Input Value | Filesystem Dir | add.path | partitionValues.p +//! ----+----------------+------------------------------+-------------------------------------------+--------------------+---------------------------- +//! 1 | INT | 0 | p=0 | same | "0" +//! 2 | | -1 | p=-1 | same | "-1" +//! 3 | | 2147483647 (max) | p=2147483647 | same | "2147483647" +//! 4 | | -2147483648 (min) | p=-2147483648 | same | "-2147483648" +//! 5 | | NULL | p=__HIVE_DEFAULT_PARTITION__ | same | null +//! 6 | BIGINT | 9223372036854775807 (max) | p=9223372036854775807 | same | "9223372036854775807" +//! 7 | | -9223372036854775808 (min) | p=-9223372036854775808 | same | "-9223372036854775808" +//! 8 | | NULL | p=__HIVE_DEFAULT_PARTITION__ | same | null +//! 9 | TINYINT | 127 (max) | p=127 | same | "127" +//! 10 | | -128 (min) | p=-128 | same | "-128" +//! 11 | | NULL | p=__HIVE_DEFAULT_PARTITION__ | same | null +//! 12 | SMALLINT | 32767 (max) | p=32767 | same | "32767" +//! 13 | | -32768 (min) | p=-32768 | same | "-32768" +//! 14 | | NULL | p=__HIVE_DEFAULT_PARTITION__ | same | null +//! 15 | DOUBLE | 0.0 | p=0.0 | same | "0.0" +//! 16 | | -0.0 | p=-0.0* | same | "-0.0"* +//! 17 | | 1.7976931348623157E308 (max) | p=1.7976931348623157E308 | same | "1.7976931348623157E308" +//! 18 | | 5E-324 (min subnormal) | p=5.0E-324* | same | "5.0E-324"* +//! 19 | | NaN | p=NaN | same | "NaN" +//! 20 | | Infinity | p=Infinity | same | "Infinity" +//! 21 | | -Infinity | p=-Infinity | same | "-Infinity" +//! 22 | | NULL | p=__HIVE_DEFAULT_PARTITION__ | same | null +//! 23 | FLOAT | 0.0 | p=0.0 | same | "0.0" +//! 24 | | NaN | p=NaN | same | "NaN" +//! 25 | | Infinity | p=Infinity | same | "Infinity" +//! 26 | | -Infinity | p=-Infinity | same | "-Infinity" +//! 27 | | NULL | p=__HIVE_DEFAULT_PARTITION__ | same | null +//! 28 | BOOLEAN | true | p=true | same | "true" +//! 29 | | false | p=false | same | "false" +//! 30 | | NULL | p=__HIVE_DEFAULT_PARTITION__ | same | null +//! 31 | DECIMAL(38,18) | 0 | p=0.000000000000000000 | same | "0.000000000000000000" +//! 32 | | 1.23 | p=1.230000000000000000 | same | "1.230000000000000000" +//! 33 | | -1.23 | p=-1.230000000000000000 | same | "-1.230000000000000000" +//! 34 | | NULL | p=__HIVE_DEFAULT_PARTITION__ | same | null +//! 35 | DATE | 2024-01-01 | p=2024-01-01 | same | "2024-01-01" +//! 36 | | 1970-01-01 (epoch) | p=1970-01-01 | same | "1970-01-01" +//! 37 | | 0001-01-01 | p=0001-01-01 | same | "0001-01-01" +//! 38 | | 9999-12-31 | p=9999-12-31 | same | "9999-12-31" +//! 39 | | NULL | p=__HIVE_DEFAULT_PARTITION__ | same | null +//! 40 | TIMESTAMP | 2024-06-15 12:30:45 | p=2024-06-15 12%3A30%3A45 | p=...%2012%253A... | "2024-06-15T19:30:45.000000Z" +//! 41 | | 1970-01-01 00:00:00 | p=1970-01-01 00%3A00%3A00 | p=...%2000%253A... | "1970-01-01T08:00:00.000000Z" +//! 42 | | 2024-06-15 23:59:59.999999 | p=2024-06-15 23%3A59%3A59.999999 | p=...%2023%253A... | "2024-06-16T06:59:59.999999Z" +//! 43 | | NULL | p=__HIVE_DEFAULT_PARTITION__ | same | null +//! 44 | TIMESTAMP_NTZ | 2024-06-15 12:30:45 | p=2024-06-15 12%3A30%3A45 | p=...%2012%253A... | "2024-06-15 12:30:45.000000"* +//! 45 | | 1970-01-01 00:00:00 | p=1970-01-01 00%3A00%3A00 | p=...%2000%253A... | "1970-01-01 00:00:00.000000"* +//! 46 | | NULL | p=__HIVE_DEFAULT_PARTITION__ | same | null +//! ``` +//! +//! Notable behaviors: +//! - DECIMAL always includes the full scale (18 trailing digits for `DECIMAL(38,18)`). +//! - TIMESTAMP `partitionValues` are UTC ISO-8601 with microsecond precision. The filesystem dir +//! and `add.path` use local time without timezone. +//! - TIMESTAMP_NTZ `partitionValues` omit the timezone and use space-separated format. +//! +//! `*` = kernel diverges from Delta-Spark: +//! - Row 16: Delta-Spark normalizes `-0.0` to `"0.0"` at the SQL layer. Kernel preserves the sign, +//! producing `"-0.0"` (matching Java's `Double.toString(-0.0)`). +//! - Row 18: Delta-Spark formats the smallest subnormal as `"4.9E-324"`. Rust's shortest- +//! representation algorithm produces `"5.0E-324"`. Both parse to the same `f64`. +//! - Rows 44-45: Delta-Spark omits `.000000` when sub-seconds are zero. Kernel always includes +//! microsecond precision. Both formats round-trip through `parse_scalar`. +//! +//! # String and binary encoding +//! +//! String values are the interesting case because they can contain characters that +//! require encoding. Binary values are interpreted as raw bytes and treated as strings +//! for path construction. +//! +//! ```text +//! Row | Input Value | Filesystem Dir | add.path | partitionValues.p +//! ----+--------------------------+--------------------------------+------------------------------+------------------ +//! 47 | chr(0) (NUL) | WRITE FAILS* | N/A | N/A +//! 48 | before\0after | WRITE FAILS* | N/A | N/A +//! 49 | X'' (empty binary) | p=__HIVE_DEFAULT_PARTITION__ | same | null +//! 50 | X'DEADBEEF' | p= | same | ** +//! 51 | X'48454C4C4F' ("HELLO") | p=HELLO | same | "HELLO" +//! 52 | X'00FF' (has NUL) | WRITE FAILS* | N/A | N/A +//! 53 | X'2F3D25' (/=%) | p=%2F%3D%25 | p=%252F%253D%2525 | "/=%" +//! 54 | a{b | p=a%7Bb | p=a%257Bb | "a{b" +//! 55 | a}b | p=a}b | p=a%7Db | "a}b" +//! 56 | hello world | p=hello world | p=hello%20world | "hello world" +//! 57 | M\u{00FC}nchen | p=M\u{00FC}nchen | same | "M\u{00FC}nchen" +//! 58 | 日本語 | p=日本語 | same | "日本語" +//! 59 | 🎵🎶 | p=🎵🎶 | same | "🎵🎶" +//! 60 | ac|d | p=ac|d | p=a%3Cb%3Ec%7Cd | "ac|d" +//! 61 | a@b!c(d) | p=a@b!c(d) | same | "a@b!c(d)" +//! 62 | a&b+c$d;e,f | p=a&b+c$d;e,f | same | "a&b+c$d;e,f" +//! 63 | Serbia/srb% | p=Serbia%2Fsrb%25 | p=Serbia%252Fsrb%2525 | "Serbia/srb%" +//! 64 | 100%25 (literal) | p=100%2525 | p=100%252525 | "100%25" +//! 65 | '' (empty string) | p=__HIVE_DEFAULT_PARTITION__ | same | null +//! 66 | NULL | p=__HIVE_DEFAULT_PARTITION__ | same | null +//! 67 | ' ' (1 space) | p= (trailing space) | p=%20 | " " +//! 68 | ' ' (2 spaces) | p= (trailing spaces) | p=%20%20 | " " +//! ``` +//! +//! *WRITE FAILS: Delta-Spark fails at the filesystem `mkdirs` stage when the +//! Hive-encoded directory name contains NUL bytes. The encoding itself succeeds +//! (NUL encodes to `%00`), but the filesystem rejects the path. +//! +//! Key observations: +//! +//! - `add.path` applies an additional layer of URI encoding on top of the Hive-encoded directory +//! name. Characters that Hive leaves raw (space, `<`, `>`, `}`, `|`) get percent-encoded in +//! `add.path`. Characters that Hive already encoded (`/` -> `%2F`) get double-encoded in +//! `add.path` (`%2F` -> `%252F`). +//! - `partitionValues` stores the raw value with no encoding. Null and empty string are both +//! represented as JSON `null`. +//! - NUL bytes cause write failures at the filesystem level (`mkdirs` error). +//! - Non-empty binary values are interpreted as raw bytes. If the bytes happen to be valid UTF-8, +//! they work (e.g., `X'48454C4C4F'` = "HELLO"). +//! +//! **Spark bug: non-UTF-8 binary (e.g., `X'DEADBEEF'`) passes raw bytes through to the +//! filesystem path and `partitionValues` JSON, producing invalid UTF-8 in both. The JSON +//! becomes unparseable. In kernel, `escape_partition_value` takes `&str`, so Rust's type +//! system prevents non-UTF-8 bytes from reaching the encoding layer. + +pub(crate) mod hive; +pub mod serialization; +pub(crate) mod validation; diff --git a/kernel/src/partition/serialization.rs b/kernel/src/partition/serialization.rs new file mode 100644 index 0000000000..3d669831bd --- /dev/null +++ b/kernel/src/partition/serialization.rs @@ -0,0 +1,642 @@ +//! Partition value serialization for the Delta log. +//! +//! A partition value goes through several transformation steps before reaching the +//! Delta log (see the [`super`] module for the full pipeline and encoding tables). +//! This module converts typed [`Scalar`] values into the strings that appear in +//! `AddFile.partitionValues`. +//! +//! ```text +//! Step 2 (THIS MODULE): Scalar::String("US/East") -> Some("US/East") (partitionValues) +//! Step 3 (hive module): "US/East" -> "US%2FEast" (directory name) +//! ``` +//! +//! [`Scalar`]: crate::expressions::Scalar + +use chrono::{DateTime, NaiveDate, Utc}; + +use crate::expressions::{DecimalData, Scalar}; +use crate::{DeltaResult, Error}; + +/// The UNIX epoch (1970-01-01) expressed as a CE day number for chrono's +/// `NaiveDate::from_num_days_from_ce_opt`, which counts from 0001-01-01. +const UNIX_EPOCH_CE_DAYS: i32 = 719_163; + +// === Partition Value Serialization === +// +// Serialization rules per Delta protocol spec (Partition Value Serialization section). +// Float and Double formatting aligns with Java's Float.toString() / Double.toString() +// format, which is also what Delta-Spark uses (the reference table in mod.rs was +// generated from Delta-Spark). The only known divergence is -0.0: Delta-Spark +// normalizes it to "0.0" at the SQL layer, while Java's toString() returns "-0.0". +// +// Type Serialization Notes +// Null None +// String as-is; "" -> None +// Boolean "true" / "false" +// Byte/Short/Int to_string() +// Long to_string() +// Float Java Float.toString() format format_java_float! macro +// Double Java Double.toString() format format_java_float! macro +// Date "YYYY-MM-DD" +// Timestamp "YYYY-MM-DDTHH:MM:SS.ffffffZ" ISO 8601 with microseconds +// TimestampNtz "YYYY-MM-DD HH:MM:SS.ffffff" space separator, no Z +// Decimal "42.00", "-0.05" preserves scale +// Binary Raw UTF-8: String::from_utf8 strict, error on invalid +// Struct/Array/Map Error not valid partition types + +/// Whether a partition-value [`Scalar`] is equivalent to null under the Delta protocol's +/// partition value serialization rules. +/// +/// The protocol collapses three Scalar shapes onto JSON null in `AddFile.partitionValues`: +/// `Scalar::Null(_)`, empty `Scalar::String`, and empty `Scalar::Binary`. Partition +/// values are serialized according to protocol rules on write and interpreted against +/// the table schema on read, so all three forms must be treated as null before enforcing +/// NOT NULL constraints. +/// +/// Read paths do not need this predicate: the on-wire collapse to JSON null is undone at +/// JSON deserialization (null map entries are dropped before kernel sees a Scalar), so a +/// "null" partition value never appears in Scalar form on reads. +pub(crate) fn would_serialize_to_null(value: &Scalar) -> bool { + match value { + Scalar::Null(_) => true, + Scalar::String(s) if s.is_empty() => true, + Scalar::Binary(b) if b.is_empty() => true, + _ => false, + } +} + +/// Serializes a [`Scalar`] partition value to a protocol-compliant string for the +/// `partitionValues` map in Add actions. +/// +/// Returns `Ok(None)` for any value the Delta protocol treats as null in `partitionValues`: +/// `Scalar::Null`, empty `Scalar::String`, and empty `Scalar::Binary`. Non-null partition +/// values are serialized according to protocol rules; readers interpret the stored value +/// against the table schema. Returns `Err` for non-null values of types that cannot be +/// partition columns (Struct, Array, Map) or for binary values that are not valid UTF-8. +/// +/// The inverse of [`PrimitiveType::parse_scalar`]. +/// +/// [`PrimitiveType::parse_scalar`]: crate::schema::PrimitiveType::parse_scalar +pub fn serialize_partition_value(value: &Scalar) -> DeltaResult> { + match value { + Scalar::Null(_) => Ok(None), + Scalar::String(s) if s.is_empty() => Ok(None), + Scalar::String(s) => Ok(Some(s.clone())), + Scalar::Boolean(v) => Ok(Some(v.to_string())), + Scalar::Byte(v) => Ok(Some(v.to_string())), + Scalar::Short(v) => Ok(Some(v.to_string())), + Scalar::Integer(v) => Ok(Some(v.to_string())), + Scalar::Long(v) => Ok(Some(v.to_string())), + Scalar::Float(v) => Ok(Some(format_f32(*v))), + Scalar::Double(v) => Ok(Some(format_f64(*v))), + Scalar::Date(days) => Ok(Some(format_date(*days)?)), + Scalar::Timestamp(us) => Ok(Some(format_timestamp(*us)?)), + Scalar::TimestampNtz(us) => Ok(Some(format_timestamp_ntz(*us)?)), + Scalar::IntervalYearMonth(months) => Ok(Some(format_year_month_interval(*months))), + Scalar::IntervalDayTime(micros) => Ok(Some(format_day_time_interval(*micros))), + Scalar::Decimal(d) => Ok(Some(format_decimal(d))), + Scalar::Binary(b) if b.is_empty() => Ok(None), + Scalar::Binary(b) => Ok(Some(format_binary(b)?)), + Scalar::Struct(_) | Scalar::Array(_) | Scalar::Map(_) => Err(Error::generic(format!( + "cannot serialize partition value: type {:?} is not a valid partition column type", + value.data_type() + ))), + } +} + +// === Float/Double formatting === +// +// Matches Java's Float.toString() / Double.toString(): decimal notation for values +// in [1e-3, 1e7), scientific notation otherwise. Separate f32/f64 entry points +// preserve native precision (f32::MAX formats as "3.4028235E38" natively but +// "3.4028234663852886E38" when cast to f64). + +// This macro uses `return` and must only be used as the sole body of a function. +macro_rules! format_java_float { + ($v:expr) => {{ + let v = $v; + if v.is_nan() { + return "NaN".into(); + } + if v.is_infinite() { + return if v > 0.0 { + "Infinity".into() + } else { + "-Infinity".into() + }; + } + if v == 0.0 { + return if v.is_sign_negative() { + "-0.0".into() + } else { + "0.0".into() + }; + } + let abs = v.abs(); + if (1e-3..1e7).contains(&abs) { + let s = v.to_string(); + if s.contains('.') { + s + } else { + format!("{s}.0") + } + } else { + let s = format!("{v:e}").replace('e', "E"); + if s.contains('.') { + s + } else { + s.replacen('E', ".0E", 1) + } + } + }}; +} + +fn format_f32(v: f32) -> String { + format_java_float!(v) +} + +fn format_f64(v: f64) -> String { + format_java_float!(v) +} + +/// Formats a date value (days since UNIX epoch) as "YYYY-MM-DD". +fn format_date(days: i32) -> DeltaResult { + let ce_days = UNIX_EPOCH_CE_DAYS.checked_add(days).ok_or_else(|| { + Error::generic(format!("date value {days} days from epoch is out of range")) + })?; + NaiveDate::from_num_days_from_ce_opt(ce_days) + .map(|d| d.format("%Y-%m-%d").to_string()) + .ok_or_else(|| Error::generic(format!("date value {days} days from epoch is out of range"))) +} + +/// Converts microseconds since epoch to a [`DateTime`], returning an error if out of range. +fn micros_to_datetime(micros: i64, label: &str) -> DeltaResult> { + let secs = micros.div_euclid(1_000_000); + let subsec_nanos = (micros.rem_euclid(1_000_000) as u32) * 1000; + DateTime::from_timestamp(secs, subsec_nanos).ok_or_else(|| { + Error::generic(format!( + "{label} value {micros} microseconds from epoch is out of range" + )) + }) +} + +/// Formats a timestamp (microseconds since epoch) as ISO 8601: "YYYY-MM-DDTHH:MM:SS.ffffffZ". +fn format_timestamp(micros: i64) -> DeltaResult { + micros_to_datetime(micros, "timestamp") + .map(|dt| dt.format("%Y-%m-%dT%H:%M:%S%.6fZ").to_string()) +} + +/// Formats a timestamp without timezone (microseconds) as "YYYY-MM-DD HH:MM:SS.ffffff". +/// Space separator, no Z suffix (there is no timezone). +fn format_timestamp_ntz(micros: i64) -> DeltaResult { + micros_to_datetime(micros, "timestamp_ntz") + .map(|dt| dt.naive_utc().format("%Y-%m-%d %H:%M:%S%.6f").to_string()) +} + +fn format_year_month_interval(months: i32) -> String { + let sign = if months < 0 { "-" } else { "" }; + let abs = months.unsigned_abs(); + format!("INTERVAL '{sign}{}-{}' YEAR TO MONTH", abs / 12, abs % 12) +} + +fn format_day_time_interval(micros: i64) -> String { + let sign = if micros < 0 { "-" } else { "" }; + let abs = micros.unsigned_abs(); + let fraction = format!("{:06}", abs % 1_000_000); + let fraction = fraction.trim_end_matches('0'); + let separator = if fraction.is_empty() { "" } else { "." }; + let total_secs = abs / 1_000_000; + let (secs, total_mins) = (total_secs % 60, total_secs / 60); + let (mins, total_hours) = (total_mins % 60, total_mins / 60); + let (hours, days) = (total_hours % 24, total_hours / 24); + format!( + "INTERVAL '{sign}{days} {hours:02}:{mins:02}:{secs:02}{separator}{fraction}' DAY TO SECOND" + ) +} + +/// Formats a decimal value preserving scale (trailing zeros). +/// For example, decimal(10,2) with value 42 serializes as "42.00", not "42". +fn format_decimal(d: &DecimalData) -> String { + let scale = d.scale(); + if scale == 0 { + return d.bits().to_string(); + } + let sign = if d.bits() < 0 { "-" } else { "" }; + let abs = d.bits().unsigned_abs(); + let divisor = 10_u128.pow(scale as u32); + let int_part = abs / divisor; + let frac_part = abs % divisor; + format!( + "{sign}{int_part}.{frac_part:0>width$}", + width = scale as usize + ) +} + +/// Formats binary data as raw UTF-8, matching Java's `new String(bytes, UTF_8)`. +/// +/// Returns an error if the bytes are not valid UTF-8. For example, `[0x48, 0x49]` +/// ("HI") succeeds, but `[0xDE, 0xAD]` fails because those bytes are not valid UTF-8. +fn format_binary(bytes: &[u8]) -> DeltaResult { + std::str::from_utf8(bytes) + .map(|s| s.to_string()) + .map_err(|e| Error::generic(format!("binary partition value is not valid UTF-8: {e}"))) +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + use crate::expressions::{ArrayData, MapData, Scalar, StructData}; + use crate::schema::{ArrayType, DataType, MapType, PrimitiveType, StructField}; + + // ============================================================================ + // Spark reference: null and empty values + // ============================================================================ + + /// Null partition values return None regardless of type. + #[rstest] + #[case::row05_int(DataType::INTEGER)] + #[case::row08_bigint(DataType::LONG)] + #[case::row11_tinyint(DataType::BYTE)] + #[case::row14_smallint(DataType::SHORT)] + #[case::row22_double(DataType::DOUBLE)] + #[case::row27_float(DataType::FLOAT)] + #[case::row30_boolean(DataType::BOOLEAN)] + #[case::row34_decimal(DataType::decimal(38, 18).unwrap())] + #[case::row39_date(DataType::DATE)] + #[case::row43_timestamp(DataType::TIMESTAMP)] + #[case::row46_timestamp_ntz(DataType::TIMESTAMP_NTZ)] + #[case::row66_string(DataType::STRING)] + #[case::binary(DataType::BINARY)] + fn test_spark_ref_null_returns_none(#[case] dtype: DataType) { + assert_eq!( + serialize_partition_value(&Scalar::Null(dtype)).unwrap(), + None + ); + } + + /// Null serialization returns None even for complex types. Complex-typed partition + /// columns should be rejected by validation before reaching serialization. + #[test] + fn test_null_complex_type_returns_none() { + let val = Scalar::null(ArrayType::new(DataType::INTEGER, false)); + assert_eq!(serialize_partition_value(&val).unwrap(), None); + } + + /// Empty string (row 65) and empty binary (row 49) serialize as None, same as null. + #[rstest] + #[case::row65_empty_string(Scalar::String(String::new()))] + #[case::row49_empty_binary(Scalar::Binary(vec![]))] + fn test_spark_ref_empty_value_returns_none(#[case] input: Scalar) { + assert_eq!(serialize_partition_value(&input).unwrap(), None); + } + + /// Tests the canonical null-equivalence rule: `would_serialize_to_null` returns true for + /// exactly the three Scalar shapes that collapse to JSON null (`Scalar::Null`, empty + /// `Scalar::String`, empty `Scalar::Binary`) and false for everything else, including + /// non-empty strings/binary and primitive scalars. + #[rstest] + // Null-equivalent: all three shapes return true. + #[case::null_int(Scalar::Null(DataType::INTEGER), true)] + #[case::null_string(Scalar::Null(DataType::STRING), true)] + #[case::null_binary(Scalar::Null(DataType::BINARY), true)] + #[case::empty_string(Scalar::String(String::new()), true)] + #[case::empty_binary(Scalar::Binary(Vec::new()), true)] + // Non-empty strings/binary and primitive scalars return false. + #[case::nonempty_string(Scalar::String("a".into()), false)] + #[case::single_space(Scalar::String(" ".into()), false)] + #[case::nonempty_binary(Scalar::Binary(vec![0x00]), false)] + #[case::integer(Scalar::Integer(0), false)] + #[case::boolean_false(Scalar::Boolean(false), false)] + fn test_would_serialize_to_null(#[case] value: Scalar, #[case] expected: bool) { + assert_eq!(would_serialize_to_null(&value), expected); + } + + /// Cross-check: any value where `would_serialize_to_null` is true must also serialize + /// to `Ok(None)`, and vice versa for the cases above. Locks the predicate to the actual + /// serialization outcome so the helper cannot drift from `serialize_partition_value`. + #[rstest] + #[case(Scalar::Null(DataType::INTEGER))] + #[case(Scalar::String(String::new()))] + #[case(Scalar::Binary(Vec::new()))] + fn test_would_serialize_to_null_matches_serialize_result(#[case] value: Scalar) { + assert!(would_serialize_to_null(&value)); + assert_eq!(serialize_partition_value(&value).unwrap(), None); + } + + // ============================================================================ + // Spark reference: non-null serialization + // ============================================================================ + // + // Tests validate serialize_partition_value against the `partitionValues.p` column + // from the encoding table in partition/mod.rs, derived from real Spark-written + // Delta tables. Divergences from Spark are noted inline. + + /// INT (rows 1-4), BIGINT (rows 6-7), TINYINT (rows 9-10), SMALLINT (rows 12-13). + #[rstest] + #[case::row01_int_zero(Scalar::Integer(0), "0")] + #[case::row02_int_neg(Scalar::Integer(-1), "-1")] + #[case::row03_int_max(Scalar::Integer(i32::MAX), "2147483647")] + #[case::row04_int_min(Scalar::Integer(i32::MIN), "-2147483648")] + #[case::row06_bigint_max(Scalar::Long(i64::MAX), "9223372036854775807")] + #[case::row07_bigint_min(Scalar::Long(i64::MIN), "-9223372036854775808")] + #[case::row09_tinyint_max(Scalar::Byte(i8::MAX), "127")] + #[case::row10_tinyint_min(Scalar::Byte(i8::MIN), "-128")] + #[case::row12_smallint_max(Scalar::Short(i16::MAX), "32767")] + #[case::row13_smallint_min(Scalar::Short(i16::MIN), "-32768")] + fn test_spark_ref_integer_types(#[case] input: Scalar, #[case] expected: &str) { + assert_eq!( + serialize_partition_value(&input).unwrap(), + Some(expected.to_string()) + ); + } + + /// DOUBLE (rows 15-21). + /// + /// Row 16 divergence: Spark normalizes -0.0 to "0.0" at the SQL expression layer, + /// but Java's Double.toString(-0.0) returns "-0.0". We match Java's behavior since + /// the Delta spec references Java's toString format. + #[rstest] + #[case::row15_zero(0.0, "0.0")] + #[case::row16_neg_zero(-0.0, "-0.0")] // Spark: "0.0" (SQL-level normalization) + #[case::row17_max(f64::MAX, "1.7976931348623157E308")] + #[case::row17b_min_positive_normal(f64::MIN_POSITIVE, "2.2250738585072014E-308")] + // Row 18: Rust and Java use different shortest-decimal-string algorithms for + // this subnormal double, producing "5.0E-324" (kernel) vs "4.9E-324" (Delta-Spark). + // Both parse to the same f64 bits. See test_subnormal_double_kernel_and_spark_ + // representations_parse_to_same_value which validates against Delta-Spark. + #[case::row18_min_subnormal(5e-324f64, "5.0E-324")] + #[case::row19_nan(f64::NAN, "NaN")] + #[case::row20_inf(f64::INFINITY, "Infinity")] + #[case::row21_neg_inf(f64::NEG_INFINITY, "-Infinity")] + #[case::integer_valued(1.0, "1.0")] // exercises "no dot" branch in decimal range + fn test_spark_ref_double(#[case] input: f64, #[case] expected: &str) { + assert_eq!( + serialize_partition_value(&Scalar::Double(input)).unwrap(), + Some(expected.to_string()) + ); + } + + /// FLOAT (rows 23-26), plus edge cases for -0.0 and scientific notation. + #[rstest] + #[case::row23_zero(0.0f32, "0.0")] + #[case::row24_nan(f32::NAN, "NaN")] + #[case::row25_inf(f32::INFINITY, "Infinity")] + #[case::row26_neg_inf(f32::NEG_INFINITY, "-Infinity")] + #[case::neg_zero(-0.0f32, "-0.0")] + #[case::scientific_small(1e-4f32, "1.0E-4")] + #[case::scientific_large(1e8f32, "1.0E8")] + #[case::f32_max(f32::MAX, "3.4028235E38")] + fn test_spark_ref_float(#[case] input: f32, #[case] expected: &str) { + assert_eq!( + serialize_partition_value(&Scalar::Float(input)).unwrap(), + Some(expected.to_string()) + ); + } + + /// BOOLEAN (rows 28-29). + #[rstest] + #[case::row28_true(true, "true")] + #[case::row29_false(false, "false")] + fn test_spark_ref_boolean(#[case] input: bool, #[case] expected: &str) { + assert_eq!( + serialize_partition_value(&Scalar::Boolean(input)).unwrap(), + Some(expected.to_string()) + ); + } + + /// DECIMAL (rows 31-33), plus edge cases for various precision/scale combos. + #[rstest] + #[case::row31_zero(0, 38, 18, "0.000000000000000000")] + #[case::row32_positive(1_230_000_000_000_000_000, 38, 18, "1.230000000000000000")] + #[case::row33_negative(-1_230_000_000_000_000_000, 38, 18, "-1.230000000000000000")] + #[case::scale_zero(42, 5, 0, "42")] + #[case::trailing_zeros(4200, 5, 2, "42.00")] + #[case::neg_between_zero_and_one(-5, 3, 2, "-0.05")] + #[case::neg_with_scale(-12345, 5, 2, "-123.45")] + #[case::pos_with_scale(12345, 5, 2, "123.45")] + fn test_spark_ref_decimal( + #[case] bits: i128, + #[case] precision: u8, + #[case] scale: u8, + #[case] expected: &str, + ) { + let d = Scalar::decimal(bits, precision, scale).unwrap(); + assert_eq!( + serialize_partition_value(&d).unwrap(), + Some(expected.to_string()) + ); + } + + /// DATE (rows 35-38), plus pre-epoch edge case. + #[rstest] + #[case::row35_recent(19723, "2024-01-01")] + #[case::row36_epoch(0, "1970-01-01")] + #[case::row37_year_one(-719_162, "0001-01-01")] + #[case::row38_year_9999(2_932_896, "9999-12-31")] + #[case::pre_epoch(-1, "1969-12-31")] + fn test_spark_ref_date(#[case] days: i32, #[case] expected: &str) { + assert_eq!( + serialize_partition_value(&Scalar::Date(days)).unwrap(), + Some(expected.to_string()) + ); + } + + /// TIMESTAMP (rows 40-42), plus pre-epoch edge case that exercises + /// div_euclid/rem_euclid for negative microsecond values. + #[rstest] + #[case::row40_afternoon(1_718_479_845_000_000, "2024-06-15T19:30:45.000000Z")] + #[case::row41_epoch_offset(28_800_000_000, "1970-01-01T08:00:00.000000Z")] + #[case::row42_with_micros(1_718_521_199_999_999, "2024-06-16T06:59:59.999999Z")] + #[case::pre_epoch(-1, "1969-12-31T23:59:59.999999Z")] + fn test_spark_ref_timestamp(#[case] micros: i64, #[case] expected: &str) { + assert_eq!( + serialize_partition_value(&Scalar::Timestamp(micros)).unwrap(), + Some(expected.to_string()) + ); + } + + /// TIMESTAMP_NTZ (rows 44-45). + /// + /// Divergence: Spark omits ".000000" when sub-seconds are zero (e.g., + /// "2024-06-15 12:30:45"), but our implementation always includes microsecond + /// precision ("2024-06-15 12:30:45.000000"). Both formats are valid per the Delta + /// spec and round-trip correctly through parse_scalar. + #[rstest] + #[case::row44_afternoon(1_718_454_645_000_000, "2024-06-15 12:30:45.000000")] + #[case::row45_epoch(0, "1970-01-01 00:00:00.000000")] + fn test_spark_ref_timestamp_ntz(#[case] micros: i64, #[case] expected: &str) { + assert_eq!( + serialize_partition_value(&Scalar::TimestampNtz(micros)).unwrap(), + Some(expected.to_string()) + ); + } + + /// String values pass through as-is with no encoding (rows 47-48, 54-64, 67-68). + /// NUL bytes (rows 47-48) are valid string characters at the partition value layer; + /// the filesystem constraint that rejects them is a separate concern. + #[rstest] + #[case::row47_nul("\x00")] + #[case::row48_embedded_nul("before\x00after")] + #[case::row54_left_brace("a{b")] + #[case::row55_right_brace("a}b")] + #[case::row56_space("hello world")] + #[case::row57_umlaut("M\u{00FC}nchen")] + #[case::row58_cjk("\u{65E5}\u{672C}\u{8A9E}")] + #[case::row59_emoji("\u{1F3B5}\u{1F3B6}")] + #[case::row60_angle_pipe("ac|d")] + #[case::row61_at_bang_parens("a@b!c(d)")] + #[case::row62_special_ascii("a&b+c$d;e,f")] + #[case::row63_slash_percent("Serbia/srb%")] + #[case::row64_percent_literal("100%25")] + #[case::row67_single_space(" ")] + #[case::row68_double_space(" ")] + fn test_spark_ref_string_passthrough(#[case] input: &str) { + assert_eq!( + serialize_partition_value(&Scalar::String(input.to_string())).unwrap(), + Some(input.to_string()) + ); + } + + /// UTF-8 binary values (rows 51, 53). + #[rstest] + #[case::row51_hello(b"HELLO".to_vec(), "HELLO")] + #[case::row53_special_chars(vec![0x2F, 0x3D, 0x25], "/=%")] + fn test_spark_ref_binary_utf8(#[case] input: Vec, #[case] expected: &str) { + assert_eq!( + serialize_partition_value(&Scalar::Binary(input)).unwrap(), + Some(expected.to_string()) + ); + } + + // ============================================================================ + // Roundtrip tests: serialize then parse_scalar + // ============================================================================ + + /// Serialize then parse back for every partition-eligible type. Float cases are + /// limited to values that round-trip exactly through string representation. + #[rstest] + #[case::integer(Scalar::Integer(42), PrimitiveType::Integer)] + #[case::integer_neg(Scalar::Integer(-1), PrimitiveType::Integer)] + #[case::long(Scalar::Long(9_876_543_210), PrimitiveType::Long)] + #[case::byte(Scalar::Byte(42), PrimitiveType::Byte)] + #[case::short(Scalar::Short(-1000), PrimitiveType::Short)] + #[case::boolean_true(Scalar::Boolean(true), PrimitiveType::Boolean)] + #[case::boolean_false(Scalar::Boolean(false), PrimitiveType::Boolean)] + #[case::string(Scalar::String("hello".into()), PrimitiveType::String)] + #[case::binary(Scalar::Binary(b"HELLO".to_vec()), PrimitiveType::Binary)] + #[case::date(Scalar::Date(19723), PrimitiveType::Date)] + #[case::timestamp(Scalar::Timestamp(1_718_521_199_999_999), PrimitiveType::Timestamp)] + #[case::timestamp_ntz( + Scalar::TimestampNtz(1_718_454_645_000_000), + PrimitiveType::TimestampNtz + )] + #[case::float(Scalar::Float(3.125), PrimitiveType::Float)] + #[case::float_one(Scalar::Float(1.0), PrimitiveType::Float)] + #[case::float_scientific(Scalar::Float(1e-4), PrimitiveType::Float)] + #[case::double(Scalar::Double(3.125), PrimitiveType::Double)] + #[case::double_max(Scalar::Double(f64::MAX), PrimitiveType::Double)] + #[case::double_min_positive(Scalar::Double(f64::MIN_POSITIVE), PrimitiveType::Double)] + #[case::decimal( + Scalar::decimal(1_230_000_000_000_000_000i128, 38, 18).unwrap(), + PrimitiveType::decimal(38, 18).unwrap() + )] + #[case::interval_ym(Scalar::IntervalYearMonth(30), PrimitiveType::IntervalYearMonth)] + #[case::interval_ym_neg(Scalar::IntervalYearMonth(-18), PrimitiveType::IntervalYearMonth)] + #[case::interval_ym_zero(Scalar::IntervalYearMonth(0), PrimitiveType::IntervalYearMonth)] + #[case::interval_ym_min(Scalar::IntervalYearMonth(i32::MIN), PrimitiveType::IntervalYearMonth)] + #[case::interval_ym_max(Scalar::IntervalYearMonth(i32::MAX), PrimitiveType::IntervalYearMonth)] + #[case::interval_dt( + Scalar::IntervalDayTime(131_445_000_000), + PrimitiveType::IntervalDayTime + )] + #[case::interval_dt_neg(Scalar::IntervalDayTime(-5), PrimitiveType::IntervalDayTime)] + #[case::interval_dt_min(Scalar::IntervalDayTime(i64::MIN), PrimitiveType::IntervalDayTime)] + #[case::interval_dt_max(Scalar::IntervalDayTime(i64::MAX), PrimitiveType::IntervalDayTime)] + fn test_roundtrip(#[case] input: Scalar, #[case] ptype: PrimitiveType) { + let serialized = serialize_partition_value(&input).unwrap().unwrap(); + let parsed = ptype.parse_scalar(&serialized).unwrap(); + assert_eq!(parsed, input); + } + + /// Interval partition values serialize to Spark ANSI interval literal strings. The + /// year-month spelling is verified against the DAT `intv_003` workload. + #[rstest] + #[case(Scalar::IntervalYearMonth(12), "INTERVAL '1-0' YEAR TO MONTH")] + #[case(Scalar::IntervalYearMonth(30), "INTERVAL '2-6' YEAR TO MONTH")] + #[case( + Scalar::IntervalDayTime(131_445_000_000), + "INTERVAL '1 12:30:45' DAY TO SECOND" + )] + fn test_spark_ref_interval(#[case] input: Scalar, #[case] expected: &str) { + assert_eq!( + serialize_partition_value(&input).unwrap(), + Some(expected.to_string()) + ); + } + + /// Kernel formats the smallest subnormal double as "5.0E-324" while Delta-Spark + /// uses "4.9E-324" (Java's Double.MIN_VALUE.toString()). Verify both strings parse + /// to the identical f64 value, confirming that Spark can read kernel's representation + /// without data loss. Validated against Delta-Spark. + #[test] + fn test_subnormal_double_kernel_and_spark_representations_parse_to_same_value() { + let kernel_str = serialize_partition_value(&Scalar::Double(5e-324f64)) + .unwrap() + .unwrap(); + let spark_str = "4.9E-324"; + let from_kernel = PrimitiveType::Double.parse_scalar(&kernel_str).unwrap(); + let from_spark = PrimitiveType::Double.parse_scalar(spark_str).unwrap(); + assert_eq!(from_kernel, from_spark); + } + + // ============================================================================ + // Error tests + // ============================================================================ + + /// Out-of-range temporal values return an error. + #[rstest] + #[case::date_max(Scalar::Date(i32::MAX))] + #[case::date_min(Scalar::Date(i32::MIN))] + #[case::timestamp(Scalar::Timestamp(i64::MAX))] + #[case::timestamp_ntz(Scalar::TimestampNtz(i64::MAX))] + fn test_temporal_out_of_range_returns_error(#[case] input: Scalar) { + assert!(serialize_partition_value(&input).is_err()); + } + + /// Non-UTF-8 binary returns an error (rows 50, 52). + #[rstest] + #[case::row50_deadbeef(vec![0xDE, 0xAD, 0xBE, 0xEF])] + #[case::row52_nul_and_high_byte(vec![0x00, 0xFF])] + fn test_non_utf8_binary_returns_error(#[case] input: Vec) { + assert!(serialize_partition_value(&Scalar::Binary(input)).is_err()); + } + + /// Non-null Struct, Array, and Map values return an error. + #[test] + fn test_non_null_struct_returns_error() { + let data = StructData::try_new( + vec![StructField::new("x", DataType::INTEGER, true)], + vec![Scalar::Integer(1)], + ) + .unwrap(); + assert!(serialize_partition_value(&Scalar::Struct(data)).is_err()); + } + + #[test] + fn test_non_null_array_returns_error() { + let data = ArrayData::try_new(ArrayType::new(DataType::INTEGER, false), [1i32]).unwrap(); + assert!(serialize_partition_value(&Scalar::Array(data)).is_err()); + } + + #[test] + fn test_non_null_map_returns_error() { + let data = MapData::try_new( + MapType::new(DataType::STRING, DataType::INTEGER, false), + [("k".to_string(), 1i32)], + ) + .unwrap(); + assert!(serialize_partition_value(&Scalar::Map(data)).is_err()); + } +} diff --git a/kernel/src/partition/validation.rs b/kernel/src/partition/validation.rs new file mode 100644 index 0000000000..85be27bafd --- /dev/null +++ b/kernel/src/partition/validation.rs @@ -0,0 +1,502 @@ +//! Partition key and type validation. +//! +//! Validates that the connector provided the correct partition column names and value types +//! before serialization. +//! +//! ```text +//! Step 1 (THIS MODULE): {"YEAR": Integer(2024)} --> {"Year": Integer(2024)} +//! (case-normalize keys, check types against schema) +//! Step 2 (serialization): Integer(2024) --> "2024" +//! Step 3 (path encoding): "Year=2024/" +//! ``` +//! +//! See the encoding tables in the [`super`] module for the full set of partition-eligible +//! types and their expected serializations. +//! +//! The primary entry point is [`validate_partition_values`], which combines key validation +//! and type checking. The two phases are also exposed individually for testing: +//! +//! - [`validate_keys`]: checks key completeness (case-insensitive matching, normalizes to schema +//! case, detects post-normalization duplicates). +//! - [`validate_types`]: checks that each `Scalar`'s type matches the partition column's schema +//! type, and that non-null partition columns are never assigned a value that would serialize to a +//! null partition value. +use std::collections::HashMap; + +use crate::expressions::Scalar; +use crate::partition::serialization::would_serialize_to_null; +use crate::schema::{DataType, StructType}; +use crate::{DeltaResult, Error}; + +/// Validates and normalizes partition keys and value types against the table schema. +/// Returns the map re-keyed to schema case. +/// +/// This is the primary entry point for partition validation. It combines [`validate_keys`] +/// (key completeness and case normalization) with [`validate_types`] (scalar type checking). +/// +/// # Parameters +/// - `logical_partition_columns`: logical partition column names from kernel's table metadata. +/// - `logical_schema`: the table's logical schema. +/// - `logical_partition_values`: connector-provided map from logical column names (any case) to +/// typed values. +pub(crate) fn validate_partition_values( + logical_partition_columns: &[String], + logical_schema: &StructType, + logical_partition_values: HashMap, +) -> DeltaResult> { + let normalized = validate_keys(logical_partition_columns, logical_partition_values)?; + validate_types(logical_schema, &normalized)?; + Ok(normalized) +} + +/// Validates that a connector-provided partition value map contains exactly the expected +/// partition columns, with case-insensitive key matching. Returns the map re-keyed to +/// the logical schema case. +/// +/// Keys in the input map are logical column names provided by the connector, which may +/// use any casing (e.g., "YEAR" or "year" for a schema column named "Year"). The returned +/// map uses the exact logical names from the schema. +/// +/// # Parameters +/// - `logical_partition_columns`: logical partition column names from kernel's table metadata. +/// - `logical_partition_values`: connector-provided map from logical column names (any case) to +/// typed values. +/// +/// # Errors +/// - A partition column is missing from the map +/// - An extra key is present that is not a partition column +/// - Two keys collide after case normalization (e.g., "COL" and "col" both provided) +fn validate_keys( + logical_partition_columns: &[String], + logical_partition_values: HashMap, +) -> DeltaResult> { + let schema_lookup: HashMap = logical_partition_columns + .iter() + .map(|name| (name.to_lowercase(), name.as_str())) + .collect(); + + let mut normalized = HashMap::with_capacity(logical_partition_values.len()); + for (key, value) in logical_partition_values { + let lower_key = key.to_lowercase(); + let schema_name = schema_lookup.get(&lower_key).ok_or_else(|| { + Error::invalid_partition_values(format!( + "unknown partition column '{key}'. Expected one of: [{}]", + logical_partition_columns.join(", ") + )) + })?; + // Detect post-normalization duplicates (e.g., "COL" and "col" both provided). + if normalized.contains_key(*schema_name) { + return Err(Error::invalid_partition_values(format!( + "duplicate partition column '{key}' (normalized to same key as a previously provided entry)" + ))); + } + normalized.insert(schema_name.to_string(), value); + } + + for col in logical_partition_columns { + if !normalized.contains_key(col.as_str()) { + return Err(Error::invalid_partition_values(format!( + "missing partition column '{col}'. Provided: [{}]", + normalized.keys().cloned().collect::>().join(", ") + ))); + } + } + + Ok(normalized) +} + +/// Validates that each [`Scalar`] value's type is compatible with the corresponding +/// partition column's type in the table schema. Values that would serialize to a null +/// partition value skip the value type check, but a +/// null partition value is only legal when the schema field is nullable. +/// +/// Nullability is enforced before any non-null type check, so the rejection treats null +/// scalars and their serialization-equivalent forms (empty strings, empty binary) +/// uniformly -- otherwise an empty `Scalar::String` for a `nullable: false` column would +/// pass validation, then collapse to JSON null at serialization, landing a null value in +/// a NOT NULL column. +/// +/// # Parameters +/// - `logical_schema`: the table's logical schema. +/// - `logical_partition_values`: map from logical column names (schema case) to typed values. Keys +/// must use the exact logical names from the schema, as returned by [`validate_keys`]. This +/// function uses case-sensitive schema lookup. +/// +/// # Errors +/// - A partition column is not found in the table schema +/// - A partition column's schema type is not a primitive (struct, array, map are rejected) +/// - A null-equivalent value (null scalar, empty string, or empty binary) is provided for a +/// non-nullable partition column +/// - A non-null value's type does not match the partition column's schema type +fn validate_types( + logical_schema: &StructType, + logical_partition_values: &HashMap, +) -> DeltaResult<()> { + for (col_name, value) in logical_partition_values { + let field = logical_schema.field(col_name).ok_or_else(|| { + Error::invalid_partition_values(format!( + "partition column '{col_name}' not found in table schema" + )) + })?; + let expected_type = field.data_type(); + if matches!( + expected_type, + DataType::Struct(_) | DataType::Array(_) | DataType::Map(_) + ) { + return Err(Error::invalid_partition_values(format!( + "partition column '{col_name}' has non-primitive type {expected_type:?}. \ + Partition columns must be primitive types." + ))); + } + if would_serialize_to_null(value) { + if !field.nullable { + return Err(Error::invalid_partition_values(format!( + "partition column '{col_name}' is not nullable but received a value that \ + serializes to null (null scalar, empty string, or empty binary)" + ))); + } + continue; + } + let actual_type = value.data_type(); + if *expected_type != actual_type { + return Err(Error::invalid_partition_values(format!( + "partition column '{col_name}' has type {expected_type:?} but got \ + value of type {actual_type:?}" + ))); + } + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + use crate::expressions::Scalar; + use crate::schema::{ArrayType, DataType, MapType, StructField}; + + fn assert_type_ok(data_type: DataType, value: Scalar) { + let schema = StructType::new_unchecked(vec![StructField::not_null("p", data_type)]); + let values = HashMap::from([("p".to_string(), value)]); + validate_types(&schema, &values).unwrap(); + } + + fn assert_type_err(data_type: DataType, value: Scalar) -> String { + let schema = StructType::new_unchecked(vec![StructField::not_null("p", data_type)]); + let values = HashMap::from([("p".to_string(), value)]); + validate_types(&schema, &values).unwrap_err().to_string() + } + + /// Like [`assert_type_ok`] but uses a `nullable` schema field. Used by null-value cases, + /// since null scalars are only valid for nullable partition columns. + fn assert_type_ok_nullable(data_type: DataType, value: Scalar) { + let schema = StructType::new_unchecked(vec![StructField::nullable("p", data_type)]); + let values = HashMap::from([("p".to_string(), value)]); + validate_types(&schema, &values).unwrap(); + } + + // ============================================================================ + // validate_keys + // ============================================================================ + + #[test] + fn test_validate_partition_keys_matching_keys_returns_ok() { + let cols = vec!["year".to_string(), "region".to_string()]; + let values = HashMap::from([ + ("year".to_string(), Scalar::Integer(2024)), + ("region".to_string(), Scalar::String("US".into())), + ]); + let result = validate_keys(&cols, values).unwrap(); + assert_eq!(result.len(), 2); + assert_eq!(result.get("year"), Some(&Scalar::Integer(2024))); + assert_eq!(result.get("region"), Some(&Scalar::String("US".into()))); + } + + #[test] + fn test_validate_partition_keys_empty_columns_and_values_returns_ok() { + let cols: Vec = vec![]; + let values = HashMap::new(); + let result = validate_keys(&cols, values).unwrap(); + assert!(result.is_empty()); + } + + #[test] + fn test_validate_partition_keys_missing_key_returns_error() { + let cols = vec!["year".to_string(), "region".to_string()]; + let values = HashMap::from([("year".to_string(), Scalar::Integer(2024))]); + let result = validate_keys(&cols, values); + assert!(result.is_err()); + let err = result.unwrap_err().to_string(); + assert!(err.contains("missing partition column 'region'"), "{err}"); + } + + #[test] + fn test_validate_partition_keys_extra_key_returns_error() { + let cols = vec!["year".to_string()]; + let values = HashMap::from([ + ("year".to_string(), Scalar::Integer(2024)), + ("region".to_string(), Scalar::String("US".into())), + ]); + let result = validate_keys(&cols, values); + assert!(result.is_err()); + let err = result.unwrap_err().to_string(); + assert!(err.contains("unknown partition column 'region'"), "{err}"); + } + + #[test] + fn test_validate_partition_keys_case_normalizes_to_schema_case() { + let cols = vec!["Year".to_string()]; + let values = HashMap::from([("YEAR".to_string(), Scalar::Integer(2024))]); + let result = validate_keys(&cols, values).unwrap(); + assert!(result.contains_key("Year")); + assert!(!result.contains_key("YEAR")); + } + + #[test] + fn test_validate_partition_keys_duplicate_after_normalization_returns_error() { + let cols = vec!["col".to_string()]; + let values = HashMap::from([ + ("COL".to_string(), Scalar::Integer(1)), + ("col".to_string(), Scalar::Integer(2)), + ]); + let result = validate_keys(&cols, values); + assert!(result.is_err()); + let err = result.unwrap_err().to_string(); + assert!(err.contains("duplicate"), "{err}"); + } + + // ============================================================================ + // validate_types + // ============================================================================ + // + // Covers every partition-eligible type from the encoding tables in `partition/mod.rs`. + // Each row tests that the correct Scalar variant passes type validation against the + // corresponding schema DataType. + + /// Every non-null row from the "Encoding by type" table in `partition/mod.rs`. + /// Row numbers reference that table. Validation only checks types (not values), but + /// covering every row mirrors the reference table 1:1 for completeness. + #[rstest] + // INT (rows 1-4) + #[case(DataType::INTEGER, Scalar::Integer(0))] // row 1 + #[case(DataType::INTEGER, Scalar::Integer(-1))] // row 2 + #[case(DataType::INTEGER, Scalar::Integer(i32::MAX))] // row 3 + #[case(DataType::INTEGER, Scalar::Integer(i32::MIN))] // row 4 + // BIGINT (rows 6-7) + #[case(DataType::LONG, Scalar::Long(i64::MAX))] // row 6 + #[case(DataType::LONG, Scalar::Long(i64::MIN))] // row 7 + // TINYINT (rows 9-10) + #[case(DataType::BYTE, Scalar::Byte(127))] // row 9 + #[case(DataType::BYTE, Scalar::Byte(-128))] // row 10 + // SMALLINT (rows 12-13) + #[case(DataType::SHORT, Scalar::Short(32767))] // row 12 + #[case(DataType::SHORT, Scalar::Short(-32768))] // row 13 + // DOUBLE (rows 15-21) + #[case(DataType::DOUBLE, Scalar::Double(0.0))] // row 15 + #[case(DataType::DOUBLE, Scalar::Double(-0.0))] // row 16 + #[case(DataType::DOUBLE, Scalar::Double(f64::MAX))] // row 17 + #[case(DataType::DOUBLE, Scalar::Double(f64::MIN_POSITIVE))] // row 18 + #[case(DataType::DOUBLE, Scalar::Double(f64::NAN))] // row 19 + #[case(DataType::DOUBLE, Scalar::Double(f64::INFINITY))] // row 20 + #[case(DataType::DOUBLE, Scalar::Double(f64::NEG_INFINITY))] // row 21 + // FLOAT (rows 23-26) + #[case(DataType::FLOAT, Scalar::Float(0.0))] // row 23 + #[case(DataType::FLOAT, Scalar::Float(f32::NAN))] // row 24 + #[case(DataType::FLOAT, Scalar::Float(f32::INFINITY))] // row 25 + #[case(DataType::FLOAT, Scalar::Float(f32::NEG_INFINITY))] // row 26 + // BOOLEAN (rows 28-29) + #[case(DataType::BOOLEAN, Scalar::Boolean(true))] // row 28 + #[case(DataType::BOOLEAN, Scalar::Boolean(false))] // row 29 + // DECIMAL(38,18) (rows 31-33) + #[case(DataType::decimal(38, 18).unwrap(), Scalar::decimal(0, 38, 18).unwrap())] // row 31 + #[case(DataType::decimal(38, 18).unwrap(), Scalar::decimal(1_230_000_000_000_000_000i128, 38, 18).unwrap())] // row 32 + #[case(DataType::decimal(38, 18).unwrap(), Scalar::decimal(-1_230_000_000_000_000_000i128, 38, 18).unwrap())] // row 33 + // DATE (rows 35-38) + #[case(DataType::DATE, Scalar::Date(19723))] // row 35: 2024-01-01 + #[case(DataType::DATE, Scalar::Date(0))] // row 36: 1970-01-01 (epoch) + #[case(DataType::DATE, Scalar::Date(-719_162))] // row 37: 0001-01-01 + #[case(DataType::DATE, Scalar::Date(2_932_896))] // row 38: 9999-12-31 + // TIMESTAMP (rows 40-42) + #[case(DataType::TIMESTAMP, Scalar::Timestamp(1_718_451_045_000_000))] // row 40 + #[case(DataType::TIMESTAMP, Scalar::Timestamp(0))] // row 41 + #[case(DataType::TIMESTAMP, Scalar::Timestamp(1_718_499_599_999_999))] // row 42 + // TIMESTAMP_NTZ (rows 44-45) + #[case(DataType::TIMESTAMP_NTZ, Scalar::TimestampNtz(1_718_451_045_000_000))] // row 44 + #[case(DataType::TIMESTAMP_NTZ, Scalar::TimestampNtz(0))] // row 45 + fn test_validate_types_encoding_table_rows_return_ok( + #[case] data_type: DataType, + #[case] value: Scalar, + ) { + assert_type_ok(data_type, value); + } + + /// Every non-null row from the "String and binary encoding" table in + /// `partition/mod.rs`. Row numbers reference that table. + #[rstest] + // STRING rows. Rows 49 (empty binary) and 65 (empty string) are null-equivalent under + // the protocol and are covered by the null-into-nullable / null-into-non-nullable tests + // further down -- they are intentionally absent from this "type-match passes" group. + #[case(DataType::STRING, Scalar::String("\x00".into()))] // row 47 + #[case(DataType::STRING, Scalar::String("before\x00after".into()))] // row 48 + #[case(DataType::STRING, Scalar::String("a{b".into()))] // row 54 + #[case(DataType::STRING, Scalar::String("a}b".into()))] // row 55 + #[case(DataType::STRING, Scalar::String("hello world".into()))] // row 56 + #[case(DataType::STRING, Scalar::String("M\u{00FC}nchen".into()))] // row 57 + #[case(DataType::STRING, Scalar::String("\u{65E5}\u{672C}\u{8A9E}".into()))] // row 58 + #[case(DataType::STRING, Scalar::String("\u{1F3B5}\u{1F3B6}".into()))] // row 59 + #[case(DataType::STRING, Scalar::String("ac|d".into()))] // row 60 + #[case(DataType::STRING, Scalar::String("a@b!c(d)".into()))] // row 61 + #[case(DataType::STRING, Scalar::String("a&b+c$d;e,f".into()))] // row 62 + #[case(DataType::STRING, Scalar::String("Serbia/srb%".into()))] // row 63 + #[case(DataType::STRING, Scalar::String("100%25".into()))] // row 64 + #[case(DataType::STRING, Scalar::String(" ".into()))] // row 67 + #[case(DataType::STRING, Scalar::String(" ".into()))] // row 68 + // BINARY rows + #[case(DataType::BINARY, Scalar::Binary(vec![0xDE, 0xAD, 0xBE, 0xEF]))] // row 50 + #[case(DataType::BINARY, Scalar::Binary(vec![0x48, 0x45, 0x4C, 0x4C, 0x4F]))] // row 51 + #[case(DataType::BINARY, Scalar::Binary(vec![0x00, 0xFF]))] // row 52 + #[case(DataType::BINARY, Scalar::Binary(vec![0x2F, 0x3D, 0x25]))] // row 53 + fn test_validate_types_string_binary_table_rows_return_ok( + #[case] data_type: DataType, + #[case] value: Scalar, + ) { + assert_type_ok(data_type, value); + } + + /// Null-equivalent rows from the encoding table. Under the Delta protocol, three Scalar + /// shapes serialize to JSON null in `partitionValues`: `Scalar::Null`, empty `Scalar::String`, + /// and empty `Scalar::Binary`. All three are valid for every partition column type when the + /// schema field is nullable, regardless of the `Scalar::Null` inner type. + #[rstest] + #[case(DataType::INTEGER, Scalar::Null(DataType::INTEGER))] // row 5 + #[case(DataType::LONG, Scalar::Null(DataType::LONG))] // row 8 + #[case(DataType::BYTE, Scalar::Null(DataType::BYTE))] // row 11 + #[case(DataType::SHORT, Scalar::Null(DataType::SHORT))] // row 14 + #[case(DataType::DOUBLE, Scalar::Null(DataType::DOUBLE))] // row 22 + #[case(DataType::FLOAT, Scalar::Null(DataType::FLOAT))] // row 27 + #[case(DataType::BOOLEAN, Scalar::Null(DataType::BOOLEAN))] // row 30 + #[case(DataType::decimal(38, 18).unwrap(), Scalar::Null(DataType::decimal(38, 18).unwrap()))] // row 34 + #[case(DataType::DATE, Scalar::Null(DataType::DATE))] // row 39 + #[case(DataType::TIMESTAMP, Scalar::Null(DataType::TIMESTAMP))] // row 43 + #[case(DataType::TIMESTAMP_NTZ, Scalar::Null(DataType::TIMESTAMP_NTZ))] // row 46 + #[case(DataType::STRING, Scalar::Null(DataType::STRING))] // row 66 + #[case(DataType::BINARY, Scalar::Null(DataType::BINARY))] // (binary NULL) + #[case(DataType::STRING, Scalar::String(String::new()))] // row 65: empty string + #[case(DataType::BINARY, Scalar::Binary(Vec::new()))] // row 49: empty binary + fn test_validate_types_null_into_nullable_returns_ok( + #[case] data_type: DataType, + #[case] value: Scalar, + ) { + assert_type_ok_nullable(data_type, value); + } + + /// Null skips the value type check even when the `Scalar::Null` inner type does not match + /// the schema column type. This is intentional: null carries no concrete type to compare + /// against. + #[test] + fn test_validate_types_null_with_mismatched_inner_type_returns_ok() { + assert_type_ok_nullable(DataType::INTEGER, Scalar::Null(DataType::STRING)); + } + + /// A null-equivalent partition value (null scalar, empty string, or empty binary) for + /// a non-nullable partition column is rejected + #[rstest] + #[case(DataType::INTEGER, Scalar::Null(DataType::INTEGER))] + #[case(DataType::LONG, Scalar::Null(DataType::LONG))] + #[case(DataType::STRING, Scalar::Null(DataType::STRING))] + #[case(DataType::BINARY, Scalar::Null(DataType::BINARY))] + #[case(DataType::BOOLEAN, Scalar::Null(DataType::BOOLEAN))] + #[case(DataType::DATE, Scalar::Null(DataType::DATE))] + #[case(DataType::TIMESTAMP, Scalar::Null(DataType::TIMESTAMP))] + #[case(DataType::TIMESTAMP_NTZ, Scalar::Null(DataType::TIMESTAMP_NTZ))] + #[case(DataType::decimal(10, 2).unwrap(), Scalar::Null(DataType::decimal(10, 2).unwrap()))] + // Null with mismatched inner type is also rejected the schema's nullability is the + // only thing checked for null scalars. + #[case(DataType::INTEGER, Scalar::Null(DataType::STRING))] + // Empty string and empty binary serialize to JSON null per the Delta protocol; they + // must be rejected for NOT NULL partition columns alongside `Scalar::Null` so a + // serialization-equivalent value cannot bypass the nullability check. + #[case(DataType::STRING, Scalar::String(String::new()))] + #[case(DataType::BINARY, Scalar::Binary(Vec::new()))] + fn test_validate_types_null_into_non_nullable_returns_error( + #[case] data_type: DataType, + #[case] value: Scalar, + ) { + let err = assert_type_err(data_type, value); + assert!(err.contains("not nullable"), "{err}"); + assert!(err.contains("'p'"), "{err}"); + } + + /// Type mismatch: every non-null Scalar variant against the wrong schema type. + #[rstest] + #[case(DataType::STRING, Scalar::Integer(1))] + #[case(DataType::INTEGER, Scalar::String("x".into()))] + #[case(DataType::INTEGER, Scalar::Long(1))] + #[case(DataType::LONG, Scalar::Integer(1))] + #[case(DataType::DOUBLE, Scalar::Float(1.0))] + #[case(DataType::FLOAT, Scalar::Double(1.0))] + #[case(DataType::DATE, Scalar::Timestamp(0))] + #[case(DataType::TIMESTAMP, Scalar::TimestampNtz(0))] + #[case(DataType::STRING, Scalar::Binary(vec![0x41]))] + #[case(DataType::BINARY, Scalar::String("A".into()))] + #[case(DataType::BOOLEAN, Scalar::Integer(1))] + fn test_validate_types_mismatch_returns_error( + #[case] data_type: DataType, + #[case] value: Scalar, + ) { + let err = assert_type_err(data_type, value); + assert!(err.contains("p"), "{err}"); + } + + /// Complex types (struct, array, map) are rejected as partition column types. + #[rstest] + #[case( + DataType::from(StructType::new_unchecked(vec![StructField::not_null("x", DataType::INTEGER)])), + Scalar::Null(DataType::STRING) + )] + #[case( + DataType::from(ArrayType::new(DataType::INTEGER, false)), + Scalar::Null(DataType::STRING) + )] + #[case( + DataType::from(MapType::new(DataType::STRING, DataType::INTEGER, false)), + Scalar::Null(DataType::STRING) + )] + fn test_validate_types_complex_type_returns_error( + #[case] data_type: DataType, + #[case] value: Scalar, + ) { + let err = assert_type_err(data_type, value); + assert!(err.contains("non-primitive type"), "{err}"); + } + + #[test] + fn test_validate_types_column_not_in_schema_returns_error() { + let schema = + StructType::new_unchecked(vec![StructField::not_null("year", DataType::INTEGER)]); + let values = HashMap::from([("nonexistent".to_string(), Scalar::Integer(42))]); + let result = validate_types(&schema, &values); + assert!(result.is_err()); + let err = result.unwrap_err().to_string(); + assert!(err.contains("not found in table schema"), "{err}"); + } + + /// validate_partition_values normalizes keys to schema case and type-checks in one call. + #[test] + fn test_validate_partition_values_with_case_mismatch_succeeds() { + let partition_cols = vec!["Year".to_string(), "Region".to_string()]; + let schema = StructType::new_unchecked(vec![ + StructField::not_null("id", DataType::INTEGER), + StructField::not_null("Year", DataType::INTEGER), + StructField::nullable("Region", DataType::STRING), + ]); + let values = HashMap::from([ + ("YEAR".to_string(), Scalar::Integer(2024)), + ("region".to_string(), Scalar::String("US".into())), + ]); + let result = validate_partition_values(&partition_cols, &schema, values).unwrap(); + assert_eq!(result.get("Year"), Some(&Scalar::Integer(2024))); + assert_eq!(result.get("Region"), Some(&Scalar::String("US".into()))); + } +} diff --git a/kernel/src/path.rs b/kernel/src/path.rs index 31c61d5066..16b6f168db 100644 --- a/kernel/src/path.rs +++ b/kernel/src/path.rs @@ -3,14 +3,14 @@ use std::slice; use std::str::FromStr; +use delta_kernel_derive::internal_api; +use url::Url; +use uuid::Uuid; + use crate::actions::visitors::InCommitTimestampVisitor; use crate::engine_data::RowVisitor; use crate::utils::require; use crate::{DeltaResult, Engine, Error, FileMeta, Version}; -use delta_kernel_derive::internal_api; - -use url::Url; -use uuid::Uuid; /// How many characters a version tag has const VERSION_LEN: usize = 20; @@ -26,6 +26,8 @@ const DELTA_LOG_DIR: &str = "_delta_log"; const DELTA_LOG_DIR_WITH_SLASH: &str = "_delta_log/"; /// The subdirectory name within the delta log where staged commits reside const STAGED_COMMITS_DIR: &str = "_staged_commits/"; +/// The subdirectory name within the delta log where checkpoint sidecars reside +const SIDECAR_DIR_WITH_SLASH: &str = "_sidecars/"; #[derive(Debug, Clone, PartialEq, Eq)] #[internal_api] @@ -33,7 +35,13 @@ pub(crate) enum LogPathFileType { Commit, /// Staged commits are commits with UUID filenames, stored in _delta_log/_staged_commits dir. StagedCommit, - SinglePartCheckpoint, + /// A classic-named checkpoint, `.checkpoint.parquet`. The name is the file-naming + /// scheme, not the spec version: this file may hold a V1 checkpoint with its actions inline, + /// or a V2 checkpoint that references sidecars. + ClassicCheckpoint, + /// A uuid-named checkpoint, `.checkpoint..{parquet,json}`. Always V2, since + /// only the V2 spec writes this naming scheme. Each writer picks a fresh uuid, so several + /// can share a version. #[allow(unused)] UuidCheckpoint, // NOTE: Delta spec doesn't actually say, but checkpoint part numbers are effectively 31-bit @@ -52,6 +60,60 @@ pub(crate) enum LogPathFileType { Unknown, } +/// Identifies one checkpoint among those at a single version and orders it against its siblings. +/// +/// The variant is the naming scheme, read from the file name with no I/O. Naming scheme and +/// [checkpoint spec] are independent: only multi-part (always V1) and uuid (always V2) pin the +/// spec, so a `Classic` checkpoint follows either one and only its contents say which. +/// +/// Variant order is the rank and each payload breaks ties within a rank, so the derived [`Ord`] is +/// the whole comparison: `Uuid` > `MultiPart` > `Classic`, matching Delta-Spark. Reordering these +/// changes which checkpoint kernel selects. +/// +/// [checkpoint spec]: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#checkpoint-specs +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)] +#[internal_api] +pub(crate) enum CheckpointInstance { + /// `.checkpoint.parquet`. At most one per version, so nothing to break ties on. + Classic, + /// `.checkpoint...parquet`. More parts wins. + MultiPart { num_parts: u32 }, + /// `.checkpoint..{json,parquet}`. The greater file name wins. + Uuid { filename: String }, +} + +impl CheckpointInstance { + /// The instance a checkpoint part belongs to, or `None` for a non-checkpoint file. + pub(crate) fn of(part: &ParsedLogPath) -> Option { + match &part.file_type { + LogPathFileType::ClassicCheckpoint => Some(Self::Classic), + LogPathFileType::UuidCheckpoint => Some(Self::Uuid { + filename: part.filename.clone(), + }), + LogPathFileType::MultiPartCheckpoint { num_parts, .. } => Some(Self::MultiPart { + num_parts: *num_parts, + }), + _ => None, + } + } + + /// How many files this checkpoint spans. + pub(crate) fn num_parts(&self) -> usize { + match self { + Self::MultiPart { num_parts } => *num_parts as usize, + Self::Classic | Self::Uuid { .. } => 1, + } + } + + /// Whether `part_files` holds every part this checkpoint needs. + pub(crate) fn is_complete( + &self, + part_files: &[ParsedLogPath], + ) -> bool { + self.num_parts() == part_files.len() + } +} + /// A ParsedLogPath is a well-understood path to a file in the _delta_log directory. /// /// Note this includes things like checkpoints and commits (containing current table state), but @@ -106,8 +168,34 @@ fn path_contains_delta_log_dir(mut path_segments: std::str::Split<'_, char>) -> path_segments.any(|p| p == DELTA_LOG_DIR) } +/// Returns whether `rel_path`, a path relative to the `_delta_log/` directory, could still be +/// within the version-named region of a lexicographically sorted log listing. +/// +/// Every listable log file begins with a 20-digit version, so its first byte is an ASCII digit. +/// Paths like `_staged_commits/`, `_sidecars/`, and `_last_checkpoint` sort after every +/// version-named file because `'_'` (0x5F) > `'9'` (0x39), so a sorted listing can stop at the +/// first relative path whose first byte sorts past `'9'`. +/// +/// This is a scan bound, not a log-file filter, so it must not require a digit first byte: a +/// path sorting before `'0'` (e.g. a dot-prefixed `.{version}.json.crc` written by some engines) +/// can still be followed by version-named files, and stopping there would silently drop them. +/// Such paths are kept here and discarded by [`ParsedLogPath`] parsing instead. An empty +/// `rel_path` is conservatively kept. +pub(crate) fn may_begin_listable_log_path(rel_path: &str) -> bool { + rel_path.as_bytes().first().is_none_or(|b| *b <= b'9') +} + impl ParsedLogPath { - // NOTE: We can't actually impl TryFrom because Option is a foreign struct even if T is local. + /// Estimated heap size in bytes, best-effort estimate. + /// + /// The Url(self.location) is measured via `len()` because it doesn't expose the capacity of its + /// internal `serialization` String. Any String capacity slack on it is not counted. + pub(crate) fn estimated_heap_size_bytes(&self) -> usize { + self.filename.capacity() + self.extension.capacity() + self.location.as_url().as_str().len() + } + + // NOTE: We can't actually impl TryFrom because Option is a foreign struct even if T is + // local. #[internal_api] pub(crate) fn try_from(location: Location) -> DeltaResult>> { let url = location.as_url(); @@ -146,7 +234,8 @@ impl ParsedLogPath { None => return Ok(None), }; - // this check determines if we're in the delta log dir, or in the staged commits dir. The check is: + // this check determines if we're in the delta log dir, or in the staged commits dir. The + // check is: // 1. If the dir is named _staged_commits, check if the parent dir is _delta_log, and ensure // no higher level directories are _also_ named _delta_log. If those checks pass we're in // the staged_commits dir @@ -178,7 +267,7 @@ impl ParsedLogPath { } } ["crc"] if in_delta_log_dir => LogPathFileType::Crc, - ["checkpoint", "parquet"] if in_delta_log_dir => LogPathFileType::SinglePartCheckpoint, + ["checkpoint", "parquet"] if in_delta_log_dir => LogPathFileType::ClassicCheckpoint, ["checkpoint", uuid, "json" | "parquet"] if in_delta_log_dir => { let Some(_) = parse_path_part::(uuid, UUID_PART_LEN) else { return Ok(None); @@ -239,7 +328,7 @@ impl ParsedLogPath { pub(crate) fn should_list(&self) -> bool { match self.file_type { LogPathFileType::Commit - | LogPathFileType::SinglePartCheckpoint + | LogPathFileType::ClassicCheckpoint | LogPathFileType::UuidCheckpoint | LogPathFileType::MultiPartCheckpoint { .. } | LogPathFileType::CompactedCommit { .. } @@ -249,6 +338,12 @@ impl ParsedLogPath { } } + /// Convenience wrapper around [`version_as_i64`] for this parsed path's `version`. + #[cfg(feature = "declarative-plans")] + pub(crate) fn version_as_i64(&self) -> DeltaResult { + crate::version_as_i64(self.version) + } + #[internal_api] pub(crate) fn is_commit(&self) -> bool { matches!( @@ -259,12 +354,7 @@ impl ParsedLogPath { #[internal_api] pub(crate) fn is_checkpoint(&self) -> bool { - matches!( - self.file_type, - LogPathFileType::SinglePartCheckpoint - | LogPathFileType::MultiPartCheckpoint { .. } - | LogPathFileType::UuidCheckpoint - ) + CheckpointInstance::of(self).is_some() } #[internal_api] @@ -272,6 +362,13 @@ impl ParsedLogPath { pub(crate) fn is_unknown(&self) -> bool { matches!(self.file_type, LogPathFileType::Unknown) } + + /// Whether this log path's file extension is `json`. + #[internal_api] + #[allow(dead_code)] // not all cfgs exercise this + pub(crate) fn is_json(&self) -> bool { + self.extension == "json" + } } impl ParsedLogPath { @@ -283,6 +380,7 @@ impl ParsedLogPath { /// /// Returns the inCommitTimestamp value, or an error if ICT is not found or cannot be read. /// Callers should handle enablement version checks before calling this method. + #[tracing::instrument(skip(engine), ret, fields(version = self.version, path = %self.location.as_url()))] pub(crate) fn read_in_commit_timestamp(&self, engine: &dyn Engine) -> DeltaResult { // Only works on commit files if !self.is_commit() { @@ -340,7 +438,6 @@ impl ParsedLogPath { } /// Create a new ParsedCheckpointPath for a classic parquet checkpoint file - #[allow(dead_code)] // TODO: Remove this once we have a use case for it pub(crate) fn new_classic_parquet_checkpoint( table_root: &Url, version: Version, @@ -372,6 +469,7 @@ impl ParsedLogPath { } /// Create a new `ParsedLogPath` for a version checksum (CRC) file. + #[internal_api] pub(crate) fn new_crc(table_root: &Url, version: Version) -> DeltaResult { let filename = format!("{version:020}.crc"); let path = Self::create_path(table_root, filename)?; @@ -384,6 +482,8 @@ impl ParsedLogPath { } /// Create a new ParsedLogPath for a log compaction file + // TODO(#2337): remove allow(dead_code) when log compaction is re-enabled + #[allow(dead_code)] pub(crate) fn new_log_compaction( table_root: &Url, start_version: Version, @@ -400,6 +500,20 @@ impl ParsedLogPath { } } +/// A checkpoint sidecar is a uniquely-named parquet file: `{unique}.parquet` where `unique` is +/// some unique string such as a UUID. We use `.checkpoint..parquet` here. +/// +/// Sidecar paths should be URI-encoded. All characters in the filename here are Unreserved +/// Characters, so we can just retain them. Ref: +pub(crate) fn new_sidecar(table_root: &Url, version: Version) -> DeltaResult<(String, Url)> { + let filename = format!("{version:020}.checkpoint.{}.parquet", Uuid::new_v4()); + let url = table_root + .join(DELTA_LOG_DIR_WITH_SLASH)? + .join(SIDECAR_DIR_WITH_SLASH)? + .join(&filename)?; + Ok((filename, url)) +} + /// A wrapper around parsed log path to provide more structure/safety when handling /// table/log/commit paths. #[derive(Debug, Clone)] @@ -443,7 +557,6 @@ impl LogRoot { } /// Create a new staged commit path (absolute path) for the given version. - #[allow(unused)] // TODO: Remove this once we remove catalog-managed feature pub(crate) fn new_staged_commit_path( &self, version: Version, @@ -462,12 +575,36 @@ pub(crate) mod tests { use std::path::PathBuf; use std::sync::Arc; + use test_utils::add_commit; + use super::*; - use crate::engine::default::DefaultEngineBuilder; use crate::engine::sync::SyncEngine; use crate::object_store::memory::InMemory; - use crate::utils::test_utils::assert_result_error_with_message; - use test_utils::add_commit; + use crate::unit_test_utils::assert_result_error_with_message; + + /// Builds a `ParsedLogPath` by parsing a real log file name, so `filename`, `extension` and + /// `file_type` agree. `size` is a parameter because listing tests use it to mark where a file + /// came from. + pub(crate) fn parse_log_path(filename: &str, size: u64) -> ParsedLogPath { + let url = Url::parse(&format!("memory:///_delta_log/{filename}")).unwrap(); + ParsedLogPath::try_from(FileMeta { + location: url, + last_modified: 0, + size, + }) + .unwrap_or_else(|e| panic!("{filename} is not a log path: {e}")) + .unwrap_or_else(|| panic!("{filename} is not a log path")) + } + + /// One part of a multi-part checkpoint. Kernel never writes these, so there's no production + /// constructor to reuse. + pub(crate) fn multipart_checkpoint_name( + version: Version, + part_num: u32, + num_parts: u32, + ) -> String { + format!("{version:020}.checkpoint.{part_num:010}.{num_parts:010}.parquet") + } impl ParsedLogPath { pub(crate) fn create_parsed_published_commit(table_root: &Url, version: Version) -> Self { @@ -477,7 +614,7 @@ pub(crate) mod tests { .unwrap() .join(&filename) .unwrap(); - let parsed = ParsedLogPath::try_from(FileMeta::new(location, 0, 0)) + let parsed = ParsedLogPath::try_from(FileMeta::new(location, 0, 100)) .unwrap() .unwrap(); assert!(parsed.file_type == LogPathFileType::Commit); @@ -494,7 +631,7 @@ pub(crate) mod tests { .unwrap() .join(&filename) .unwrap(); - let parsed = ParsedLogPath::try_from(FileMeta::new(location, 0, 0)) + let parsed = ParsedLogPath::try_from(FileMeta::new(location, 0, 100)) .unwrap() .unwrap(); assert!(parsed.file_type == LogPathFileType::StagedCommit); @@ -508,7 +645,7 @@ pub(crate) mod tests { .unwrap() .join(&filename) .unwrap(); - let parsed = ParsedLogPath::try_from(FileMeta::new(location, 0, 0)) + let parsed = ParsedLogPath::try_from(FileMeta::new(location, 0, 100)) .unwrap() .unwrap(); assert!(parsed.file_type == LogPathFileType::Crc); @@ -534,6 +671,23 @@ pub(crate) mod tests { url } + #[test] + fn test_may_begin_listable_log_path() { + // version-named files, and anything sorting before them, keep the scan going + assert!(may_begin_listable_log_path("00000000000000000010.json")); + assert!(may_begin_listable_log_path( + ".00000000000000000010.json.crc" + )); + assert!(may_begin_listable_log_path("")); + // paths sorting past '9' end the version-named region + assert!(!may_begin_listable_log_path("_last_checkpoint")); + assert!(!may_begin_listable_log_path("_sidecars/3a0d65cd.parquet")); + assert!(!may_begin_listable_log_path( + "_staged_commits/00000000000000000010.3a0d65cd.json" + )); + assert!(!may_begin_listable_log_path("Zsentinel")); + } + #[test] fn test_unknown_invalid_patterns() { let table_log_dir = table_log_dir_url(); @@ -659,7 +813,7 @@ pub(crate) mod tests { assert_eq!(log_path.version, 2); assert!(matches!( log_path.file_type, - LogPathFileType::SinglePartCheckpoint + LogPathFileType::ClassicCheckpoint )); assert!(!log_path.is_commit()); assert!(log_path.is_checkpoint()); @@ -948,7 +1102,7 @@ pub(crate) mod tests { assert_eq!(log_path.extension, "parquet"); assert!(matches!( log_path.file_type, - LogPathFileType::SinglePartCheckpoint + LogPathFileType::ClassicCheckpoint )); assert_eq!(log_path.filename, "00000000000000000010.checkpoint.parquet"); } @@ -1012,7 +1166,7 @@ pub(crate) mod tests { for (file_type, should_list) in [ (LogPathFileType::Commit, true), (LogPathFileType::StagedCommit, false), - (LogPathFileType::SinglePartCheckpoint, true), + (LogPathFileType::ClassicCheckpoint, true), (LogPathFileType::UuidCheckpoint, true), ( LogPathFileType::MultiPartCheckpoint { @@ -1038,7 +1192,7 @@ pub(crate) mod tests { #[tokio::test] async fn test_read_in_commit_timestamp_success() { let store = Arc::new(InMemory::new()); - let engine = DefaultEngineBuilder::new(store.clone()).build(); + let engine = SyncEngine::new_with_store(store.clone()); let table_root = "memory://test/"; let table_url = url::Url::parse(table_root).unwrap(); @@ -1068,7 +1222,7 @@ pub(crate) mod tests { #[tokio::test] async fn test_read_in_commit_timestamp_missing_ict() { let store = Arc::new(InMemory::new()); - let engine = DefaultEngineBuilder::new(store.clone()).build(); + let engine = SyncEngine::new_with_store(store.clone()); let table_root = "memory://test/"; let table_url = url::Url::parse(table_root).unwrap(); @@ -1119,4 +1273,36 @@ pub(crate) mod tests { "read_in_commit_timestamp can only be called on commit files", ); } + + /// Verifies `new_sidecar` builds a `.checkpoint..parquet` filename + /// under `/_delta_log/_sidecars/`. + #[rstest::rstest] + #[case::version_zero(0)] + #[case::small_version(7)] + #[case::large_version(1_234_567_890)] + fn test_new_sidecar_path(#[case] version: Version) { + let table_root = Url::parse("memory:///table/").unwrap(); + let (filename, url) = new_sidecar(&table_root, version).unwrap(); + + // Filename: `.checkpoint..parquet` + let prefix = format!("{version:020}.checkpoint."); + assert!( + filename.starts_with(&prefix) && filename.ends_with(".parquet"), + "unexpected filename: {filename}" + ); + // The middle segment must be a valid UUID. + let uuid_part = filename + .strip_prefix(&prefix) + .and_then(|s| s.strip_suffix(".parquet")) + .unwrap(); + Uuid::parse_str(uuid_part).expect("middle segment must be a valid UUID"); + + // URL: `/_delta_log/_sidecars/` + let expected = table_root + .join("_delta_log/_sidecars/") + .unwrap() + .join(&filename) + .unwrap(); + assert_eq!(url, expected); + } } diff --git a/kernel/src/plans/builder.rs b/kernel/src/plans/builder.rs new file mode 100644 index 0000000000..00a2665c48 --- /dev/null +++ b/kernel/src/plans/builder.rs @@ -0,0 +1,1416 @@ +//! Fluent builder for declarative [`Plan`]s: a source, chained transforms, then +//! [`PlanBuilder::build`] / [`PlanBuilder::build_opt`]. Each transform consumes `self` and returns +//! a new builder; clone a builder (cheap) to feed it into more than one transform. Validating +//! methods return [`DeltaResult`] and surface errors at the call site. +//! +//! A source with no rows (e.g. a scan over no files) is the *absent* relation. The builder treats +//! it as dead code: transforms validate as usual but propagate absence, eliminating the parts of +//! the plan that cannot contribute rows (a union drops absent arms, an anti-join with an absent +//! build forwards its probe, etc.). [`build`](PlanBuilder::build) collapses the plan to its +//! minimal correct shape, emitting a single empty [`Values`] node when the whole relation is +//! absent, so it always yields a runnable plan; [`build_opt`](PlanBuilder::build_opt) yields +//! `None` for it. +//! +//! ``` +//! use std::sync::Arc; +//! use delta_kernel::PlanBuilder; +//! use delta_kernel::expressions::col; +//! use delta_kernel::plans::ir::nodes::Operator; +//! use delta_kernel::schema::{DataType, StructField, StructType}; +//! +//! let schema = Arc::new(StructType::try_new([StructField::not_null("id", DataType::INTEGER)])?); +//! +//! // A source with no rows is the *absent* relation: it produces no rows, so the builder +//! // eliminates it as dead code, collapsing the plan to its minimal correct shape. Downstream +//! // transforms still validate and chain exactly as they would over a populated source. +//! let scan = PlanBuilder::values(schema, vec![])? +//! .filter(col!("id").is_not_null())?; +//! +//! // `build_opt` reports the eliminated plan as `None` ... +//! assert!(scan.clone().build_opt()?.is_none()); +//! // ... while `build` keeps the plan runnable, replacing the absent relation with a single +//! // empty `Values` node. +//! let plan = scan.build()?; +//! let [node] = plan.nodes.as_slice() else { panic!("expected one node") }; +//! assert!(matches!(&node.op, Operator::Values(values) if values.rows.is_empty())); +//! # Ok::<(), delta_kernel::Error>(()) +//! ``` + +use std::collections::HashMap; +use std::sync::Arc; + +use delta_kernel_derive::internal_api; + +use super::ir::nodes::{ + Aggregate, AggregateBuilder, DynamicScan, FileType, Filter, Operator, Project, ScanFile, + ScanJson, ScanParquet, SemiJoin, UnionAll, Values, +}; +use super::ir::plan::{Plan, PlanNode}; +use crate::expressions::{ColumnName, ExpressionRef, PredicateRef, Scalar, StructData}; +use crate::schema::{SchemaRef, ToSchema}; +use crate::struct_patch::ProjectionStructPatchBuilder; +use crate::utils::CollectInto; +use crate::{DeltaResult, Error}; + +/// One node of a plan DAG: an operator, its inputs, and its output schema. Node identity is the +/// `Arc`'s address ([`PlanBuilder::build`] dedups shared subgraphs by pointer). Inputs are always +/// present nodes -- combinators drop or forward an absent input. +#[derive(Debug)] +struct BuilderNode { + op: Operator, + inputs: Vec, + schema: SchemaRef, +} + +type BuilderNodeRef = Arc; + +/// A present node, or the *absent* (uninhabited) relation, which still carries its output schema. +#[derive(Clone, Debug)] +enum PlanBuilderRoot { + Present(BuilderNodeRef), + Absent(SchemaRef), +} + +/// A [`Plan`] under construction. See the module docs. +#[derive(Clone, Debug)] +#[must_use] +pub struct PlanBuilder(PlanBuilderRoot); + +impl PlanBuilder { + /// A present relation applying `op` (output `schema`) over `inputs`. + fn present(schema: SchemaRef, op: impl Into, inputs: Vec) -> Self { + PlanBuilder(PlanBuilderRoot::Present(Arc::new(BuilderNode { + op: op.into(), + inputs, + schema, + }))) + } + + /// The uninhabited relation over `schema`. + fn absent(schema: SchemaRef) -> Self { + PlanBuilder(PlanBuilderRoot::Absent(schema)) + } + + /// The output schema of this relation -- carried whether present or absent. + fn schema(&self) -> &SchemaRef { + match &self.0 { + PlanBuilderRoot::Present(node) => &node.schema, + PlanBuilderRoot::Absent(schema) => schema, + } + } + + /// Apply a single-input transform: wrap `op` (output `schema`) over a present input, or stay + /// absent. Callers validate against [`Self::schema`] first. + fn unary_op_or_absent(self, schema: SchemaRef, op: impl Into) -> Self { + match self.0 { + PlanBuilderRoot::Present(node) => Self::present(schema, op, vec![node]), + PlanBuilderRoot::Absent(_) => Self::absent(schema), + } + } + + /// A Parquet scan source over `files` producing rows matching `schema`. + /// + /// `file_constant_columns` names the per-file-constant output columns (see [`ScanParquet`]); + /// pass `&[]` when there are none. An empty `files` yields the absent relation. + /// + /// Produces an error when any file's constant count differs from `file_constant_columns`'s + /// length, or a `file_constant_columns` entry is absent from `schema`. + pub fn scan_parquet( + files: impl IntoIterator>, + file_constant_columns: &[&str], + schema: impl Into, + ) -> DeltaResult { + Self::scan_source(FileType::Parquet, files, file_constant_columns, schema) + } + + /// A newline-delimited JSON scan source over `files` producing rows matching `schema`. See + /// [`ScanJson`]. Exhibits same behaviour as [`Self::scan_parquet`]. + pub fn scan_json( + files: impl IntoIterator>, + file_constant_columns: &[&str], + schema: impl Into, + ) -> DeltaResult { + Self::scan_source(FileType::Json, files, file_constant_columns, schema) + } + + /// Shared body of [`Self::scan_parquet`] / [`Self::scan_json`]. Normalizes and validates the + /// inputs, then yields the absent relation for an empty file set, else the scan source node for + /// `file_type`. + fn scan_source( + file_type: FileType, + files: impl IntoIterator>, + file_constant_columns: &[&str], + schema: impl Into, + ) -> DeltaResult { + let schema = schema.into(); + let files = Vec::from_iter(files.into_iter().map(Into::into)); + let cols = Vec::from_iter(file_constant_columns.iter().map(|&c| c.to_string())); + for (i, file) in files.iter().enumerate() { + if file.file_constants.len() != cols.len() { + return Err(Error::generic(format!( + "scan: file {i} has {} constant value(s) {:?} for {} file-constant column(s) {:?}", + file.file_constants.len(), + file.file_constants, + cols.len(), + cols, + ))); + } + } + check_file_constant_columns(&schema, &cols, "scan file_constant_columns")?; + if files.is_empty() { + return Ok(Self::absent(schema)); + } + let file_constant_columns = cols; + let op = match file_type { + FileType::Parquet => Operator::from(ScanParquet { + files, + file_constant_columns, + schema: Arc::clone(&schema), + }), + FileType::Json => Operator::from(ScanJson { + files, + file_constant_columns, + schema: Arc::clone(&schema), + }), + }; + Ok(Self::present(schema, op, vec![])) + } + + /// Inline literal rows. See [`Values`] for the row encoding. Empty `rows` yields the absent + /// relation. + /// + /// Produces an error when any row's width differs from `schema`'s top-level field count. + /// + /// # Example + /// ``` + /// # use std::sync::Arc; + /// # use delta_kernel::PlanBuilder; + /// # use delta_kernel::schema::{DataType, StructField, StructType}; + /// let schema = Arc::new(StructType::try_new([StructField::not_null("id", DataType::INTEGER)])?); + /// let plan = PlanBuilder::values(schema, vec![vec![1.into()], vec![2.into()]])?.build()?; + /// # Ok::<(), delta_kernel::Error>(()) + /// ``` + pub fn values(schema: impl Into, rows: Vec>) -> DeltaResult { + let schema = schema.into(); + let width = schema.fields().count(); + for (i, row) in rows.iter().enumerate() { + if row.len() != width { + return Err(Error::generic(format!( + "values: row {i} has {} value(s) {row:?} but schema has {width} field(s) {:?}", + row.len(), + Vec::from_iter(schema.fields().map(|f| f.name())), + ))); + } + } + Ok(Values::new(schema, rows).into()) + } + + /// Infallible sibling of [`Self::values`] containing rows of `T` converted to scalar data. + /// + /// Schema is [`ToSchema::to_schema`] for `T`. Each row converts via [`Into`] and is + /// peeled into top-level field scalars (nested fields remain [`Scalar::Struct`]); see + /// [`Values`]'s [`FromIterator`]. Empty `rows` yields the absent relation. + /// + /// # Example + /// ``` + /// # use delta_kernel::PlanBuilder; + /// # use delta_kernel::expressions::Scalar; + /// # use delta_kernel::plans::ir::nodes::Operator; + /// # use delta_kernel_derive::{IntoStructData, ToSchema}; + /// # + /// #[derive(ToSchema, IntoStructData)] + /// struct Row { + /// id: i32, + /// } + /// + /// let plan = PlanBuilder::values_from([Row { id: 1 }, Row { id: 2 }]).build()?; + /// let Operator::Values(values) = &plan.nodes[0].op else { panic!("expected Values") }; + /// assert_eq!( + /// values.rows, + /// vec![vec![Scalar::Integer(1)], vec![Scalar::Integer(2)]], + /// ); + /// assert!(PlanBuilder::values_from(std::iter::empty::()) + /// .build_opt()? + /// .is_none()); + /// # Ok::<(), delta_kernel::Error>(()) + /// ``` + #[internal_api] + pub(crate) fn values_from(rows: impl IntoIterator) -> Self + where + T: Into + ToSchema, + { + Values::from_iter(rows).into() + } + + /// Keep rows where `predicate` holds. Output schema is unchanged. See [`Filter`]. + /// + /// Produces an error when `predicate` references a column absent from the input schema. + /// + /// # Example + /// ``` + /// # use std::sync::Arc; + /// # use delta_kernel::PlanBuilder; + /// # use delta_kernel::expressions::col; + /// # use delta_kernel::schema::{DataType, StructField, StructType}; + /// let schema = Arc::new(StructType::try_new([StructField::not_null("id", DataType::INTEGER)])?); + /// let plan = PlanBuilder::values(schema, vec![vec![1.into()], vec![2.into()]])? + /// .filter(col!("id").is_not_null())? + /// .build()?; + /// # Ok::<(), delta_kernel::Error>(()) + /// ``` + pub fn filter(self, predicate: impl Into) -> DeltaResult { + let predicate = predicate.into(); + check_columns_resolve(self.schema(), predicate.references(), "filter")?; + let schema = Arc::clone(self.schema()); + Ok(self.unary_op_or_absent(schema, Filter { predicate })) + } + + /// Project `self` through `expr` into rows of the caller-declared `schema`. `expr` must be a + /// struct constructor / patch matching `schema`. See [`Project`]. + /// + /// Produces an error when `expr` references a column absent from the input schema. References + /// inside a `StructPatch` are validated by [`ProjectionStructPatchBuilder`] when the patch is + /// + /// [`ProjectionStructPatchBuilder`]: crate::struct_patch::ProjectionStructPatchBuilder + /// + /// # Example + /// ``` + /// # use std::sync::Arc; + /// # use delta_kernel::PlanBuilder; + /// # use delta_kernel::expressions::{col, Expression}; + /// # use delta_kernel::schema::{DataType, StructField, StructType}; + /// let input = Arc::new(StructType::try_new([ + /// StructField::not_null("id", DataType::INTEGER), + /// StructField::nullable("name", DataType::STRING), + /// ])?); + /// let out = Arc::new(StructType::try_new([StructField::not_null("id", DataType::INTEGER)])?); + /// // Keep only `id`. + /// let plan = PlanBuilder::values(input, vec![vec![1.into(), "a".into()]])? + /// .project(Expression::struct_from([col!("id")]), out)? + /// .build()?; + /// # Ok::<(), delta_kernel::Error>(()) + /// ``` + pub fn project( + self, + expr: impl Into, + schema: impl Into, + ) -> DeltaResult { + let schema = schema.into(); + let expr = expr.into(); + check_columns_resolve(self.schema(), expr.references(), "project")?; + Ok(self.unary_op_or_absent(Arc::clone(&schema), Project { expr, schema })) + } + + /// Project `self` by editing its columns. `edit` receives a [`ProjectionStructPatchBuilder`] + /// rooted at `self`'s schema and records field edits (replace/drop/append); the lowered patch + /// and the resulting output schema feed the [`Project`] node together. See [`Project`]. + /// + /// Produces an error when the patch fails to build -- e.g. a replaced or dropped field is + /// absent from the input schema, or a nested path does not resolve. + pub fn project_patch( + self, + patch: impl FnOnce(ProjectionStructPatchBuilder<'_>) -> ProjectionStructPatchBuilder<'_>, + ) -> DeltaResult { + let (out, expr) = patch(ProjectionStructPatchBuilder::new(self.schema())).build()?; + self.project(expr, out) + } + + /// Read data files named by `self`'s rows. Output schema is `dynamic_scan.schema`. See + /// [`DynamicScan`]. + /// + /// Applies [`DynamicScan::validate_input`]'s validation against `self`'s schema. + pub fn dynamic_scan(self, dynamic_scan: DynamicScan) -> DeltaResult { + dynamic_scan.validate_input(self.schema())?; + let schema = Arc::clone(&dynamic_scan.schema); + Ok(self.unary_op_or_absent(schema, dynamic_scan)) + } + + /// Aggregate `self` into `aggregate` (build one with [`Aggregate::group_by`]). The output + /// schema is the group keys followed by the aggregate columns. See [`Aggregate`]. + /// + /// Over an absent input the result is group-arity dependent: a **grouped** aggregate yields + /// zero groups and so is absent; a **global** aggregate (no group keys) still yields one row, + /// so it aggregates an empty [`Values`] input and lets the engine produce that row. + /// + /// Produces an error when building `aggregate` fails: a group key or an aggregate's operand + /// column is absent from its input schema, or two output columns would share a name. + pub fn aggregate(self, aggregate: impl TryInto) -> DeltaResult { + let aggregate = aggregate.try_into()?; + let schema = Arc::clone(&aggregate.schema); + match self.0 { + PlanBuilderRoot::Present(node) => Ok(Self::present(schema, aggregate, vec![node])), + // A grouped aggregate over the empty relation has zero groups, hence absent. + PlanBuilderRoot::Absent(_) if !aggregate.group_by.is_empty() => { + Ok(Self::absent(schema)) + } + // A global aggregate always emits one row, even over no input: aggregate an empty + // Values relation and let the engine produce that row. + PlanBuilderRoot::Absent(input_schema) => { + let input = Self::present( + Arc::clone(&input_schema), + Values::new(input_schema, vec![]), + vec![], + ); + Ok(input.unary_op_or_absent(schema, aggregate)) + } + } + } + + /// Aggregate `self`, grouped by `keys`. `aggs` receives an [`AggregateBuilder`] rooted at + /// `self`'s schema and adds the aggregate columns (see e.g. [`AggregateBuilder::max`]). + /// + /// Mirrors [`Self::aggregate`] but spares the caller from naming the input schema, just as + /// [`Self::project_patch`] does for [`Self::project`]. See [`Aggregate`]. + /// + /// Produces an error under the same conditions as [`Self::aggregate`]. + /// + /// # Example + /// ``` + /// # use std::sync::Arc; + /// # use delta_kernel::PlanBuilder; + /// # use delta_kernel::expressions::column_name; + /// # use delta_kernel::schema::{DataType, StructField, StructType}; + /// let schema = Arc::new(StructType::try_new([ + /// StructField::not_null("id", DataType::INTEGER), + /// StructField::nullable("version", DataType::LONG), + /// ])?); + /// // Latest `version` per `id`. + /// let plan = PlanBuilder::values(schema, vec![vec![1.into(), 7i64.into()]])? + /// .aggregate_by([column_name!("id")], |a| a.max(column_name!("version")))? + /// .build()?; + /// # Ok::<(), delta_kernel::Error>(()) + /// ``` + pub fn aggregate_by( + self, + keys: impl CollectInto>, + aggs: impl FnOnce(AggregateBuilder) -> AggregateBuilder, + ) -> DeltaResult { + let builder = Aggregate::group_by(Arc::clone(self.schema()), keys); + self.aggregate(aggs(builder)) + } + + /// Aggregate `self` without grouping. `aggs` receives an [`AggregateBuilder`] rooted at + /// `self`'s schema and adds aggregate columns (see e.g. [`AggregateBuilder::max`]). + /// + /// Mirrors [`Self::aggregate`] but infers the input schema and uses + /// [`Aggregate::ungrouped`] as the builder root. + /// + /// Produces an error under the same conditions as [`Self::aggregate`]. + /// + /// # Example + /// ``` + /// # use std::sync::Arc; + /// # use delta_kernel::PlanBuilder; + /// # use delta_kernel::expressions::column_name; + /// # use delta_kernel::schema::{DataType, StructField, StructType}; + /// let schema = Arc::new(StructType::try_new([ + /// StructField::not_null("id", DataType::INTEGER), + /// StructField::nullable("version", DataType::LONG), + /// ])?); + /// // Latest version across all rows. + /// let plan = PlanBuilder::values(schema, vec![vec![1.into(), 7i64.into()]])? + /// .aggregate_ungrouped(|a| a.max(column_name!("version")))? + /// .build()?; + /// # Ok::<(), delta_kernel::Error>(()) + /// ``` + pub fn aggregate_ungrouped( + self, + aggs: impl FnOnce(AggregateBuilder) -> AggregateBuilder, + ) -> DeltaResult { + let builder = Aggregate::ungrouped(Arc::clone(self.schema())); + self.aggregate(aggs(builder)) + } + + /// Semi join: emit the `self` (probe) rows that have a match in `build` on the join keys. + /// Output schema mirrors `self`. Inputs are recorded as `[probe, build]`. See [`SemiJoin`]. + /// + /// Keys are columns (e.g. [`column_name!`](crate::expressions::column_name)), matched pairwise. + /// + /// Produces an error when the probe and build key counts differ, or a probe/build key is absent + /// from its input schema. + /// + /// # Example + /// ``` + /// # use std::sync::Arc; + /// # use delta_kernel::PlanBuilder; + /// # use delta_kernel::expressions::column_name; + /// # use delta_kernel::schema::{DataType, StructField, StructType}; + /// let schema = Arc::new(StructType::try_new([StructField::not_null("id", DataType::INTEGER)])?); + /// let probe = PlanBuilder::values(Arc::clone(&schema), vec![vec![1.into()], vec![2.into()]])?; + /// let allow = PlanBuilder::values(schema, vec![vec![2.into()]])?; + /// // Probe rows whose `id` appears in `allow`. + /// let plan = probe.semi_join(allow, [column_name!("id")], [column_name!("id")])?.build()?; + /// # Ok::<(), delta_kernel::Error>(()) + /// ``` + pub fn semi_join( + self, + build: PlanBuilder, + probe_keys: impl IntoIterator, + build_keys: impl IntoIterator, + ) -> DeltaResult { + self.semi_join_impl(build, false, probe_keys, build_keys) + } + + /// Anti join: emit the `self` (probe) rows that have *no* match in `build` on the join keys. + /// An inverted [`SemiJoin`]; otherwise as [`Self::semi_join`]. + pub fn anti_join( + self, + build: PlanBuilder, + probe_keys: impl IntoIterator, + build_keys: impl IntoIterator, + ) -> DeltaResult { + self.semi_join_impl(build, true, probe_keys, build_keys) + } + + fn semi_join_impl( + self, + build: PlanBuilder, + inverted: bool, + probe_keys: impl IntoIterator, + build_keys: impl IntoIterator, + ) -> DeltaResult { + let probe_keys = Vec::from_iter(probe_keys); + let build_keys = Vec::from_iter(build_keys); + if probe_keys.len() != build_keys.len() { + return Err(Error::generic(format!( + "join: {} probe key(s) {probe_keys:?} but {} build key(s) {build_keys:?}; \ + they must match in length.", + probe_keys.len(), + build_keys.len(), + ))); + } + check_columns_resolve(self.schema(), &probe_keys, "join probe")?; + check_columns_resolve(build.schema(), &build_keys, "join build")?; + // An uninhabited probe always produces an uninhabited result; forward it unchanged. + let PlanBuilderRoot::Present(probe) = self.0 else { + return Ok(self); + }; + let build = match build.0 { + PlanBuilderRoot::Present(build) => build, + // An absent build side matches nothing, so an anti-join forwards an unfiltered + // probe side while a semi-join is uninhabited. + PlanBuilderRoot::Absent(_) if inverted => { + return Ok(PlanBuilder(PlanBuilderRoot::Present(probe))) + } + PlanBuilderRoot::Absent(_) => return Ok(Self::absent(Arc::clone(&probe.schema))), + }; + let node = SemiJoin { + inverted, + probe_keys, + build_keys, + }; + let schema = Arc::clone(&probe.schema); + Ok(Self::present(schema, node, vec![probe, build])) + } + + /// Unordered bag union of `inputs` into a [`UnionAll`]. All inputs must share the same schema; + /// absent inputs are dropped, and a lone present input is forwarded unchanged. + /// + /// Produces an error when `inputs` is empty, or two inputs have differing schemas. + /// + /// # Example + /// ``` + /// # use std::sync::Arc; + /// # use delta_kernel::PlanBuilder; + /// # use delta_kernel::schema::{DataType, StructField, StructType}; + /// let schema = Arc::new(StructType::try_new([StructField::not_null("id", DataType::INTEGER)])?); + /// let a = PlanBuilder::values(Arc::clone(&schema), vec![vec![1.into()]])?; + /// let b = PlanBuilder::values(schema, vec![vec![2.into()]])?; + /// let plan = PlanBuilder::union_all([a, b])?.build()?; + /// # Ok::<(), delta_kernel::Error>(()) + /// ``` + pub fn union_all(inputs: impl IntoIterator) -> DeltaResult { + let inputs = Vec::from_iter(inputs); + let Some(schema) = inputs.first().map(|b| Arc::clone(b.schema())) else { + return Err(Error::generic("union_all: requires at least one input")); + }; + if let Some(i) = inputs.iter().position(|b| b.schema() != &schema) { + return Err(Error::generic(format!( + "union_all: input {i} has a schema differing from input 0: {:?} vs {:?}", + inputs[i].schema(), + schema, + ))); + } + let mut present = Vec::from_iter(inputs.into_iter().filter_map(|b| match b.0 { + PlanBuilderRoot::Present(node) => Some(node), + PlanBuilderRoot::Absent(_) => None, + })); + Ok(match present.len() { + 0 => Self::absent(schema), + 1 => PlanBuilder(PlanBuilderRoot::Present(present.swap_remove(0))), + _ => Self::present(schema, UnionAll, present), + }) + } + + /// Linearize the DAG reachable from `self` into a [`Plan`]. An absent relation builds to a + /// single empty [`Values`] node carrying its schema, so the result is always a runnable plan. + pub fn build(&self) -> DeltaResult { + Ok(match &self.0 { + PlanBuilderRoot::Present(root) => Self::build_plan(root), + PlanBuilderRoot::Absent(schema) => Plan { + nodes: vec![PlanNode::new( + Values::new(Arc::clone(schema), vec![]), + vec![], + )], + }, + }) + } + + /// Like [`Self::build`], but yields `None` for an absent relation instead of an empty plan. + pub fn build_opt(&self) -> DeltaResult> { + Ok(match &self.0 { + PlanBuilderRoot::Present(root) => Some(Self::build_plan(root)), + PlanBuilderRoot::Absent(_) => None, + }) + } + + /// Linearize the DAG rooted at `root` into a topologically sorted [`Plan`] with `root` as the + /// last (terminal) node. A node's inputs always precede it, referenced by their index in + /// [`Plan::nodes`]. Shared subgraphs are emitted once; multiple nodes can reference them. + fn build_plan(root: &BuilderNodeRef) -> Plan { + // Maps a node's Arc pointer to its index, deduping shared subgraphs. Pointers are stable + // because `root` pins all reachable nodes for the lifetime of `build_plan`. + fn emit( + node: &BuilderNodeRef, + nodes: &mut Vec, + emitted: &mut HashMap<*const BuilderNode, usize>, + ) -> usize { + let key = Arc::as_ptr(node); + if let Some(&index) = emitted.get(&key) { + return index; + } + // Recurse before `entry` so the closure borrows only `nodes`, not `emitted`. + let inputs = Vec::from_iter(node.inputs.iter().map(|i| emit(i, nodes, emitted))); + *emitted.entry(key).or_insert_with(|| { + nodes.push(PlanNode::new(node.op.clone(), inputs)); + nodes.len() - 1 + }) + } + let mut nodes = Vec::new(); + emit(root, &mut nodes, &mut HashMap::new()); + Plan { nodes } + } +} + +/// A literal relation. Empty rows is the absent relation, which the builder eliminates as dead +/// code (see the module docs). +impl From for PlanBuilder { + fn from(values: Values) -> Self { + match values.rows.is_empty() { + true => Self::absent(values.schema), + false => Self::present(values.schema.clone(), values, vec![]), + } + } +} + +/// Error if any of `cols` fails to resolve against `schema` (nested paths supported). +fn check_columns_resolve<'a>( + schema: &SchemaRef, + cols: impl IntoIterator, + ctx: &str, +) -> DeltaResult<()> { + for col in cols { + schema.field_at(col).map_err(|_| { + Error::generic(format!( + "{ctx}: column `{col}` not found; schema has {:?}", + Vec::from_iter(schema.fields().map(|f| f.name())), + )) + })?; + } + Ok(()) +} + +/// Error unless each of `names` is a top-level, non-metadata field of `schema`. A file-constant is +/// one scalar broadcast into a top-level field, so nested paths and engine-generated metadata +/// columns are rejected. +fn check_file_constant_columns<'a>( + schema: &SchemaRef, + names: impl IntoIterator, + ctx: &str, +) -> DeltaResult<()> { + for name in names { + let Some(field) = schema.field(name.as_str()) else { + return Err(Error::generic(format!( + "{ctx}: column `{name}` not found; schema has {:?}", + Vec::from_iter(schema.fields().map(|f| f.name())), + ))); + }; + if field.is_metadata_column() { + return Err(Error::generic(format!( + "{ctx}: column `{name}` is a metadata column" + ))); + } + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use test_utils::assert_result_error_with_message; + + use super::*; + use crate::actions::deletion_vector::DeletionVectorDescriptor; + use crate::expressions::{col, column_name, lit, Expression}; + use crate::plans::ir::nodes::FileType; + use crate::schema::{DataType, MetadataColumnSpec, StructField, StructType}; + use crate::FileMeta; + + /// A single-file scan (present), no file-constant columns -- the trivial scan fixture. + fn scan(schema: SchemaRef) -> PlanBuilder { + PlanBuilder::scan_parquet([test_file("file:///a.parquet")], &[], schema).unwrap() + } + + fn test_file(path: &str) -> FileMeta { + FileMeta { + location: url::Url::parse(path).unwrap(), + last_modified: 0, + size: 0, + } + } + + fn id_schema() -> SchemaRef { + Arc::new(StructType::new_unchecked([StructField::nullable( + "id", + DataType::STRING, + )])) + } + + fn x_schema() -> SchemaRef { + Arc::new(StructType::new_unchecked([StructField::nullable( + "x", + DataType::LONG, + )])) + } + + /// A one-row (all-null) `Values` source over `schema` -- the common present-source fixture. + /// (Empty rows would collapse to absent.) + fn vals(schema: SchemaRef) -> PlanBuilder { + let row = Vec::from_iter(schema.fields().map(|f| Scalar::null(f.data_type().clone()))); + PlanBuilder::values(schema, vec![row]).unwrap() + } + + /// The absent (uninhabited) relation over `schema`. + fn absent_over(schema: SchemaRef) -> PlanBuilder { + PlanBuilder::values(schema, vec![]).unwrap() + } + + /// The absent (uninhabited) relation over `id_schema`. + fn absent_src() -> PlanBuilder { + absent_over(id_schema()) + } + + /// Build `builder` and assert its nodes match `expected` one-for-one in order, each described + /// by its `(inputs, op)`: the indices of the nodes it references and its operator + /// tag. A node is identified by its index in `nodes`. Returns the built [`Plan`] for further + /// inspection. + fn assert_plan(builder: PlanBuilder, expected: &[(&[usize], &str)]) -> Plan { + let plan = builder.build().unwrap(); + assert_eq!(plan.nodes.len(), expected.len(), "node count"); + for (i, (node, &(inputs, op))) in plan.nodes.iter().zip(expected).enumerate() { + assert_eq!(node.inputs.as_slice(), inputs, "node {i} inputs"); + assert_eq!(node.op.to_string(), op, "node {i} op"); + } + plan + } + + /// A source records its declared schema and builds to a single node. + #[test] + fn source_builds_single_node() { + let src = scan(id_schema()); + assert_eq!(src.schema(), &id_schema()); + assert_plan(src, &[(&[], "scan_parquet")]); + } + + /// `{ id, part }`, with `part` used as a file-constant column. + fn part_schema() -> SchemaRef { + Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::STRING), + StructField::nullable("part", DataType::STRING), + ])) + } + + /// A single-file scan with one file constant `"p1"` for the `part` column. + fn one_constant_file(path: &str) -> ScanFile { + ScanFile { + meta: test_file(path), + file_constants: vec!["p1".into()], + } + } + + /// Both scan sources carry file-constant columns and record them on the node. + #[test] + fn scans_record_file_constant_columns() -> DeltaResult<()> { + let part = vec!["part".to_string()]; + + let parquet = PlanBuilder::scan_parquet( + vec![one_constant_file("file:///a.parquet")], + &["part"], + part_schema(), + )?; + assert_eq!(parquet.schema(), &part_schema()); + let plan = assert_plan(parquet, &[(&[], "scan_parquet")]); + let Operator::ScanParquet(node) = &plan.nodes[0].op else { + panic!("expected ScanParquet"); + }; + assert_eq!(node.file_constant_columns, part); + + let json = PlanBuilder::scan_json( + vec![one_constant_file("file:///a.json")], + &["part"], + part_schema(), + )?; + let plan = assert_plan(json, &[(&[], "scan_json")]); + let Operator::ScanJson(node) = &plan.nodes[0].op else { + panic!("expected ScanJson"); + }; + assert_eq!(node.file_constant_columns, part); + Ok(()) + } + + /// Filter passes the input schema through unchanged. + #[test] + fn filter_preserves_schema() -> DeltaResult<()> { + let filtered = vals(id_schema()).filter(col!("id").is_not_null())?; + assert_eq!(filtered.schema(), &id_schema()); + Ok(()) + } + + /// A method chain linearizes in topological order with the terminal last. + #[test] + fn monadic_chain_is_topological() -> DeltaResult<()> { + assert_plan( + scan(id_schema()) + .filter(col!("id").is_not_null())? + .filter(col!("id").is_null())?, + &[(&[], "scan_parquet"), (&[0], "filter"), (&[1], "filter")], + ); + Ok(()) + } + + /// A value reused by two consumers (SSA style) is emitted exactly once: the shared source + /// node appears a single time and both consumers reference its index. + #[test] + fn shared_source_emitted_once() -> DeltaResult<()> { + let a = vals(id_schema()); + let b = a.clone().filter(col!("id").is_not_null())?; + let c = a.filter(col!("id").is_null())?; + + // `a` is emitted once as node 0; both filters reference it. + assert_plan( + PlanBuilder::union_all([b, c])?, + &[ + (&[], "values"), + (&[0], "filter"), + (&[0], "filter"), + (&[1, 2], "union_all"), + ], + ); + Ok(()) + } + + /// Nodes not reachable from the terminal are dropped (dead-code elimination). + #[test] + fn unreachable_nodes_are_dropped() -> DeltaResult<()> { + let src = vals(id_schema()); + let _dead = src.clone().filter(col!("id").is_null())?; // never reaches the terminal + let kept = src.filter(col!("id").is_not_null())?; + + // Only src(0) and the kept filter(1) survive. + assert_plan(kept, &[(&[], "values"), (&[0], "filter")]); + Ok(()) + } + + /// A shared *intermediate* (a node with its own input) feeding two consumers is emitted once, + /// and both consumers reference its single index. + #[test] + fn shared_intermediate_emitted_once_diamond() -> DeltaResult<()> { + let mid = vals(id_schema()).filter(col!("id").is_not_null())?; + let left = mid.clone().filter(col!("id").is_not_null())?; + let right = mid.filter(col!("id").is_null())?; + + // src(0) and mid(1) each emitted once; left(2) and right(3) both reference mid. + assert_plan( + PlanBuilder::union_all([left, right])?, + &[ + (&[], "values"), + (&[0], "filter"), + (&[1], "filter"), + (&[1], "filter"), + (&[2, 3], "union_all"), + ], + ); + Ok(()) + } + + /// `union_all` over two or more present inputs keeps `self`'s schema and records all inputs. + #[test] + fn union_all_records_present_inputs() -> DeltaResult<()> { + let inputs = Vec::from_iter((0..3).map(|_| vals(id_schema()))); + assert_plan( + PlanBuilder::union_all(inputs)?, + &[ + (&[], "values"), + (&[], "values"), + (&[], "values"), + (&[0, 1, 2], "union_all"), + ], + ); + Ok(()) + } + + /// `union_all` with a single present input forwards it unchanged -- no `UnionAll` node. + #[test] + fn union_all_of_one_forwards() -> DeltaResult<()> { + assert_plan( + PlanBuilder::union_all([vals(id_schema())])?, + &[(&[], "values")], + ); + Ok(()) + } + + // === Absent (uninhabited) relation ======================================= + + /// Empty sources and transforms over an absent input all collapse to the absent relation, which + /// `build_opt` reports as `None`. + #[rstest::rstest] + #[case::empty_parquet(PlanBuilder::scan_parquet(Vec::::new(), &[], id_schema()))] + #[case::empty_json(PlanBuilder::scan_json(Vec::::new(), &[], id_schema()))] + #[case::empty_values(PlanBuilder::values(id_schema(), vec![]))] + #[case::filter(absent_src().filter(col!("id").is_not_null()))] + #[case::project(absent_src().project(Expression::struct_from([col!("id")]), id_schema()))] + #[case::project_patch(absent_src().project_patch(|p| p.drop("id")))] + #[case::dynamic_scan({ + let input = dynamic_scan_input_schema(); + dynamic_scan_node(&input, dynamic_scan_output_schema()) + .and_then(|dynamic_scan| absent_over(input).dynamic_scan(dynamic_scan)) + })] + #[case::grouped_aggregate(absent_src().aggregate( + Aggregate::group_by(part_schema(), [column_name!("id")]).max(column_name!("part"))))] + #[case::semi_join_absent_build( + vals(id_schema()).semi_join(absent_src(), [column_name!("id")], [column_name!("id")]))] + #[case::semi_join_absent_probe( + absent_src().semi_join(vals(x_schema()), [column_name!("id")], [column_name!("x")]))] + #[case::anti_join_absent_probe( + absent_src().anti_join(vals(x_schema()), [column_name!("id")], [column_name!("x")]))] + #[case::union_all_of_absent(PlanBuilder::union_all([absent_src(), absent_src()]))] + fn collapses_to_absent(#[case] builder: DeltaResult) -> DeltaResult<()> { + assert!(builder?.build_opt()?.is_none()); // absent has no plan to run + Ok(()) + } + + /// An absent relation still carries its schema, and `build` materializes it as an empty + /// `Values` node. + #[test] + fn absent_carries_schema_and_builds_empty_values() -> DeltaResult<()> { + let absent = absent_src(); + assert_eq!(absent.schema(), &id_schema()); + let plan = assert_plan(absent, &[(&[], "values")]); + let Operator::Values(values) = &plan.nodes[0].op else { + panic!("expected Values"); + }; + assert!(values.rows.is_empty()); + assert_eq!(values.schema, id_schema()); + Ok(()) + } + + /// `union_all` drops absent arms, so a lone present arm survives and is forwarded unchanged -- + /// no `UnionAll` node. + #[test] + fn union_all_forwards_lone_present_arm() -> DeltaResult<()> { + assert_plan( + PlanBuilder::union_all([absent_src(), vals(id_schema())])?, + &[(&[], "values")], + ); + Ok(()) + } + + /// An anti join with an absent build forwards its probe unchanged: it subtracts nothing. + #[test] + fn anti_join_over_absent_build_forwards_probe() -> DeltaResult<()> { + let anti = vals(id_schema()).anti_join( + absent_src(), + [column_name!("id")], + [column_name!("id")], + )?; + assert_plan(anti, &[(&[], "values")]); + Ok(()) + } + + /// `aggregate` outputs the group keys followed by the aggregate columns. + #[test] + fn aggregate_outputs_keys_then_aggs() -> DeltaResult<()> { + // Group by `id`, take the latest `part` by version -- output is `{id, part}`. + let agg = vals(part_schema()).aggregate( + Aggregate::group_by(part_schema(), [column_name!("id")]).max_non_null_by( + column_name!("part"), + column_name!("part"), + column_name!("id"), + ), + )?; + assert_eq!(agg.schema(), &part_schema()); + assert_plan(agg, &[(&[], "values"), (&[0], "aggregate")]); + Ok(()) + } + + /// `aggregate_by` roots the `AggregateBuilder` at the input schema, yielding the same plan as + /// `aggregate` with an explicitly schema'd builder. + #[test] + fn aggregate_by_infers_input_schema() -> DeltaResult<()> { + let agg = vals(part_schema()).aggregate_by([column_name!("id")], |a| { + a.max_non_null_by( + column_name!("part"), + column_name!("part"), + column_name!("id"), + ) + })?; + assert_eq!(agg.schema(), &part_schema()); + assert_plan(agg, &[(&[], "values"), (&[0], "aggregate")]); + Ok(()) + } + + /// `aggregate_ungrouped` roots the `AggregateBuilder` at the input schema without grouping. + #[test] + fn aggregate_ungrouped_infers_input_schema() -> DeltaResult<()> { + let agg = vals(part_schema()).aggregate_ungrouped(|a| a.max(column_name!("part")))?; + let schema = agg.schema(); + assert_eq!(schema.fields().count(), 1); + assert!(schema.field("part").is_some()); + assert_plan(agg, &[(&[], "values"), (&[0], "aggregate")]); + Ok(()) + } + + /// A global (no group keys) aggregate over an absent input aggregates an empty `Values` + /// relation, so the engine produces the single output row. + #[test] + fn aggregate_global_over_absent_aggregates_empty_values() -> DeltaResult<()> { + let agg = absent_src().aggregate( + Aggregate::group_by(id_schema(), Vec::::new()).max(column_name!("id")), + )?; + let plan = assert_plan(agg, &[(&[], "values"), (&[0], "aggregate")]); + let Operator::Values(values) = &plan.nodes[0].op else { + panic!("expected Values input"); + }; + assert!(values.rows.is_empty()); + Ok(()) + } + + /// A global (no group keys) aggregate over a present input outputs just the aggregate columns. + #[test] + fn aggregate_global_over_present_outputs_aggs_only() -> DeltaResult<()> { + let agg = vals(part_schema()).aggregate( + Aggregate::group_by(part_schema(), Vec::::new()).max(column_name!("part")), + )?; + let schema = agg.schema(); + assert_eq!(schema.fields().count(), 1); + assert!(schema.field("part").is_some()); + let plan = assert_plan(agg, &[(&[], "values"), (&[0], "aggregate")]); + let Operator::Aggregate(node) = &plan.nodes[1].op else { + panic!("expected Aggregate"); + }; + assert!(node.group_by.is_empty()); + Ok(()) + } + + /// `{ outer: { a, b }, c }` -- a nested schema for exercising `project_patch`. + fn nested_ab_c() -> SchemaRef { + Arc::new(StructType::new_unchecked([ + StructField::nullable( + "outer", + StructType::new_unchecked([ + StructField::nullable("a", DataType::LONG), + StructField::nullable("b", DataType::STRING), + ]), + ), + StructField::nullable("c", DataType::LONG), + ])) + } + + /// `project_patch` lowers field edits and the output schema together: a nested replace, a + /// top-level append, and a drop are all reflected in the resulting schema. + #[test] + fn project_patch_edits_track_schema() -> DeltaResult<()> { + let patched = vals(nested_ab_c()).project_patch(|p| { + p.replace_at( + ["outer"], + "b", + StructField::nullable("b2", DataType::LONG), + lit(0i64), + ) + .append(StructField::nullable("d", DataType::LONG), lit(1i64)) + .drop("c") + })?; + + let expected: SchemaRef = Arc::new(StructType::new_unchecked([ + StructField::nullable( + "outer", + StructType::new_unchecked([ + StructField::nullable("a", DataType::LONG), + StructField::nullable("b2", DataType::LONG), + ]), + ), + StructField::nullable("d", DataType::LONG), + ])); + assert_eq!(patched.schema(), &expected); + assert_plan(patched, &[(&[], "values"), (&[0], "project")]); + Ok(()) + } + + /// A shared edit `fn` can be reused across multiple `project_patch` sites: it is higher-ranked + /// over the patch-builder lifetime, which a `let` closure cannot express. + #[test] + fn project_patch_accepts_shared_fn() -> DeltaResult<()> { + fn drop_c(p: ProjectionStructPatchBuilder<'_>) -> ProjectionStructPatchBuilder<'_> { + p.drop("c") + } + let a = vals(nested_ab_c()).project_patch(drop_c)?; + let b = vals(nested_ab_c()).project_patch(drop_c)?; + assert_eq!(a.schema(), b.schema()); + Ok(()) + } + + /// `project` records the caller's declared output schema verbatim. + #[test] + fn project_records_declared_schema() -> DeltaResult<()> { + let out = x_schema(); + let p = vals(id_schema()).project(Expression::struct_from([col!("id")]), out.clone())?; + assert_eq!(p.schema(), &out); + assert_plan(p, &[(&[], "values"), (&[0], "project")]); + Ok(()) + } + + /// `semi_join`/`anti_join` mirror the probe schema, record `[probe, build]` inputs in order, + /// and set `inverted` accordingly. + #[rstest::rstest] + fn join_mirrors_probe_and_orders_inputs( + #[values(false, true)] inverted: bool, + ) -> DeltaResult<()> { + let probe = vals(id_schema()); + let build = vals(x_schema()); + let joined = if inverted { + probe.anti_join(build, [column_name!("id")], [column_name!("x")])? + } else { + probe.semi_join(build, [column_name!("id")], [column_name!("x")])? + }; + assert_eq!(joined.schema(), &id_schema()); + + let plan = assert_plan( + joined, + &[(&[], "values"), (&[], "values"), (&[0, 1], "semi_join")], + ); + let Operator::SemiJoin(node) = &plan.nodes[2].op else { + panic!("expected SemiJoin"); + }; + assert_eq!(node.inverted, inverted); + Ok(()) + } + + fn dynamic_scan_node( + input_schema: &SchemaRef, + out_schema: SchemaRef, + ) -> DeltaResult { + DynamicScan::try_new( + input_schema, + out_schema, + FileType::Parquet, + url::Url::parse("memory:///").unwrap(), + ["version"], + column_name!("path"), + column_name!("size"), + column_name!("filemod"), + column_name!("dv"), + ) + } + + fn dynamic_scan_input_schema() -> SchemaRef { + Arc::new(StructType::new_unchecked([ + StructField::not_null("path", DataType::STRING), + StructField::not_null("size", DataType::LONG), + StructField::not_null("filemod", DataType::LONG), + StructField::nullable("dv", DeletionVectorDescriptor::to_schema()), + StructField::nullable("version", DataType::LONG), + ])) + } + + fn dynamic_scan_output_schema() -> SchemaRef { + Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::STRING), + StructField::nullable("version", DataType::LONG), + ])) + } + + #[test] + fn dynamic_scan_sets_output_schema_and_records_input() -> DeltaResult<()> { + let input = dynamic_scan_input_schema(); + let out = dynamic_scan_output_schema(); + let node = dynamic_scan_node(&input, out.clone())?; + let dynamic_scan = vals(input).dynamic_scan(node)?; + assert_eq!(dynamic_scan.schema(), &out); + assert_plan(dynamic_scan, &[(&[], "values"), (&[0], "dynamic_scan")]); + Ok(()) + } + + fn dynamic_scan_input_missing(omit: &str) -> SchemaRef { + let schema = dynamic_scan_input_schema(); + let fields = schema + .fields() + .filter(|field| field.name() != omit) + .cloned(); + Arc::new(StructType::new_unchecked(fields)) + } + + fn schema_with_field(schema: SchemaRef, replacement: StructField) -> SchemaRef { + let fields = schema.fields().map(|field| { + if field.name() == replacement.name() { + replacement.clone() + } else { + field.clone() + } + }); + Arc::new(StructType::new_unchecked(fields)) + } + + fn dynamic_scan_input_with(field: StructField) -> SchemaRef { + schema_with_field(dynamic_scan_input_schema(), field) + } + + fn construct_dynamic_scan(input: SchemaRef) -> DeltaResult { + dynamic_scan_node(&input, dynamic_scan_output_schema()) + } + + fn build_dynamic_scan_with_schemas( + input: SchemaRef, + output: SchemaRef, + ) -> DeltaResult { + let dynamic_scan = dynamic_scan_node(&input, output)?; + vals(input).dynamic_scan(dynamic_scan) + } + + #[rstest::rstest] + #[case::path("path", 0)] + #[case::size("size", 1)] + #[case::filemod("filemod", 2)] + fn dynamic_scan_validates_nested_metadata( + #[case] column: &str, + #[case] index: usize, + #[values(false, true)] parent_nullable: bool, + ) { + let base_schema = dynamic_scan_input_schema(); + let metadata = StructField::new( + "metadata", + StructType::new_unchecked([ + StructField::not_null("path", DataType::STRING), + StructField::not_null("size", DataType::LONG), + StructField::not_null("filemod", DataType::LONG), + ]), + parent_nullable, + ); + let input = Arc::new(StructType::new_unchecked( + base_schema.fields().cloned().chain([metadata]), + )); + let mut columns = [ + column_name!("path"), + column_name!("size"), + column_name!("filemod"), + ]; + columns[index] = ColumnName::new(["metadata", column]); + let [path_column, file_size_column, last_modified_column] = columns; + let result = DynamicScan::try_new( + &input, + dynamic_scan_output_schema(), + FileType::Parquet, + url::Url::parse("memory:///").unwrap(), + ["version"], + path_column, + file_size_column, + last_modified_column, + column_name!("dv"), + ) + .and_then(|dynamic_scan| vals(input).dynamic_scan(dynamic_scan)); + + if parent_nullable { + assert_result_error_with_message(result, &format!("`metadata.{column}` is nullable")); + } else { + let plan = result.unwrap(); + assert_eq!(plan.schema(), &dynamic_scan_output_schema()); + } + } + + #[rstest::rstest] + fn dynamic_scan_accepts_dv_with_nullable_ancestor( + #[values(false, true)] parent_nullable: bool, + ) { + let metadata = StructField::new( + "metadata", + StructType::new_unchecked([StructField::nullable( + "dv", + DeletionVectorDescriptor::to_schema(), + )]), + parent_nullable, + ); + let base_schema = dynamic_scan_input_schema(); + let input = Arc::new(StructType::new_unchecked( + base_schema.fields().cloned().chain([metadata]), + )); + + DynamicScan::try_new( + &input, + dynamic_scan_output_schema(), + FileType::Parquet, + url::Url::parse("memory:///").unwrap(), + ["version"], + column_name!("path"), + column_name!("size"), + column_name!("filemod"), + column_name!("metadata.dv"), + ) + .unwrap(); + } + + /// Every validating method surfaces its malformed input as an error at the call site, with a + /// message that names the offending column or mismatch. + #[rstest::rstest] + // sources + #[case::scan_file_constant_arity("constant value", + || PlanBuilder::scan_parquet(vec![one_constant_file("file:///a.parquet")], &[], part_schema()))] + #[case::scan_unknown_file_constant("`nope` not found", + || PlanBuilder::scan_parquet(vec![one_constant_file("file:///a.parquet")], &["nope"], part_schema()))] + #[case::values_row_width("schema has 1 field", + || PlanBuilder::values(id_schema(), vec![vec!["a".into(), "b".into()]]))] + // transforms referencing an absent column + #[case::filter_unknown_column("`nope` not found", + || vals(id_schema()).filter(col!("nope").is_not_null()))] + #[case::project_unknown_column("`nope` not found", + || vals(id_schema()).project(Expression::struct_from([col!("nope")]), id_schema()))] + #[case::project_patch_missing_field("does not exist", + || vals(id_schema()).project_patch(|p| p.drop("nope")))] + #[case::aggregate_unknown_value("not found in schema", + || vals(id_schema()).aggregate(Aggregate::group_by(id_schema(), Vec::::new()).max(column_name!("nope"))))] + #[case::aggregate_unknown_group_by("not found in schema", + || vals(id_schema()).aggregate(Aggregate::group_by(id_schema(), [column_name!("nope")]).max(column_name!("id"))))] + // union + #[case::union_schema_disagrees("differing", + || PlanBuilder::union_all([vals(id_schema()), vals(x_schema())]))] + #[case::union_empty("at least one input", || PlanBuilder::union_all([]))] + // semi-join keys + #[case::join_unknown_probe_key("join probe: column `nope`", + || vals(id_schema()).semi_join(vals(x_schema()), [column_name!("nope")], [column_name!("x")]))] + #[case::join_unknown_build_key("join build: column `nope`", + || vals(id_schema()).anti_join(vals(x_schema()), [column_name!("id")], [column_name!("nope")]))] + #[case::join_key_arity("must match", + || vals(id_schema()).semi_join(vals(x_schema()), [column_name!("id")], [column_name!("x"), column_name!("x")]))] + // dynamic scan file constants + #[case::dynamic_scan_missing_source("`version`", || build_dynamic_scan_with_schemas( + dynamic_scan_input_missing("version"), + dynamic_scan_output_schema(), + ))] + #[case::dynamic_scan_absent_from_output("`version`", || { + let input = dynamic_scan_input_schema(); + let dynamic_scan = dynamic_scan_node(&input, id_schema())?; + vals(input).dynamic_scan(dynamic_scan) + })] + #[case::dynamic_scan_type_mismatch("same type and nullability", || { + let output = schema_with_field( + dynamic_scan_output_schema(), + StructField::nullable("version", DataType::STRING), + ); + build_dynamic_scan_with_schemas(dynamic_scan_input_schema(), output) + })] + #[case::dynamic_scan_input_nullable("same type and nullability", || { + let output = schema_with_field( + dynamic_scan_output_schema(), + StructField::not_null("version", DataType::LONG), + ); + build_dynamic_scan_with_schemas(dynamic_scan_input_schema(), output) + })] + #[case::dynamic_scan_output_nullable("same type and nullability", || { + let input = dynamic_scan_input_with(StructField::not_null("version", DataType::LONG)); + build_dynamic_scan_with_schemas(input, dynamic_scan_output_schema()) + })] + #[case::dynamic_scan_metadata_source("metadata column", || { + let input = dynamic_scan_input_with( + StructField::create_metadata_column("version", MetadataColumnSpec::RowIndex), + ); + build_dynamic_scan_with_schemas(input, dynamic_scan_output_schema()) + })] + #[case::dynamic_scan_metadata_output("metadata column", || { + let output = schema_with_field( + dynamic_scan_output_schema(), + StructField::create_metadata_column("version", MetadataColumnSpec::RowIndex), + ); + build_dynamic_scan_with_schemas(dynamic_scan_input_schema(), output) + })] + fn rejects(#[case] needle: &str, #[case] make: impl Fn() -> DeltaResult) { + assert_result_error_with_message(make(), needle); + } + + #[rstest::rstest] + #[case::missing_path("path", || construct_dynamic_scan(dynamic_scan_input_missing("path")))] + #[case::missing_size("size", || construct_dynamic_scan(dynamic_scan_input_missing("size")))] + #[case::missing_filemod("filemod", || construct_dynamic_scan(dynamic_scan_input_missing("filemod")))] + #[case::nullable_path("required column `path` is nullable", || construct_dynamic_scan(dynamic_scan_input_with(StructField::nullable("path", DataType::STRING))))] + #[case::nullable_size("required column `size` is nullable", || construct_dynamic_scan(dynamic_scan_input_with(StructField::nullable("size", DataType::LONG))))] + #[case::nullable_filemod("required column `filemod` is nullable", || construct_dynamic_scan(dynamic_scan_input_with(StructField::nullable("filemod", DataType::LONG))))] + #[case::wrong_path_type("must have type string", || construct_dynamic_scan(dynamic_scan_input_with(StructField::not_null("path", DataType::BOOLEAN))))] + #[case::wrong_size_type("must have type long", || construct_dynamic_scan(dynamic_scan_input_with(StructField::not_null("size", DataType::STRING))))] + #[case::wrong_filemod_type("must have type long", || construct_dynamic_scan(dynamic_scan_input_with(StructField::not_null("filemod", DataType::STRING))))] + #[case::missing_dv("`dv`", || construct_dynamic_scan(dynamic_scan_input_missing("dv")))] + #[case::wrong_dv_type("deletion-vector column `dv` must have type", || construct_dynamic_scan(dynamic_scan_input_with(StructField::nullable("dv", DataType::STRING))))] + #[case::non_nullable_dv("deletion-vector column `dv` must be nullable", || construct_dynamic_scan(dynamic_scan_input_with(StructField::not_null("dv", DeletionVectorDescriptor::to_schema()))))] + #[case::opaque_base_url("must be hierarchical and end in `/`", || { + let input = dynamic_scan_input_schema(); + DynamicScan::try_new( + &input, + dynamic_scan_output_schema(), + FileType::Parquet, + url::Url::parse("mailto:user@example.com").unwrap(), + ["version"], + column_name!("path"), + column_name!("size"), + column_name!("filemod"), + column_name!("dv"), + ) + })] + #[case::base_url_without_trailing_slash("must be hierarchical and end in `/`", || { + let input = dynamic_scan_input_schema(); + DynamicScan::try_new( + &input, + dynamic_scan_output_schema(), + FileType::Parquet, + url::Url::parse("s3://bucket/table/data").unwrap(), + ["version"], + column_name!("path"), + column_name!("size"), + column_name!("filemod"), + column_name!("dv"), + ) + })] + fn dynamic_scan_constructor_rejects_invalid_configuration( + #[case] needle: &str, + #[case] make: impl Fn() -> DeltaResult, + ) { + assert_result_error_with_message(make(), needle); + } + + #[test] + fn dynamic_scan_attachment_revalidates_input_schema() { + let input = dynamic_scan_input_with(StructField::nullable("size", DataType::LONG)); + let valid_input = dynamic_scan_input_schema(); + let dynamic_scan = dynamic_scan_node(&valid_input, dynamic_scan_output_schema()).unwrap(); + assert_result_error_with_message( + vals(input).dynamic_scan(dynamic_scan), + "required column `size` is nullable", + ); + } + + #[test] + fn dynamic_scan_attachment_revalidates_base_url() { + let input = dynamic_scan_input_schema(); + let mut dynamic_scan = dynamic_scan_node(&input, dynamic_scan_output_schema()).unwrap(); + dynamic_scan.base_url = url::Url::parse("s3://bucket/table/data").unwrap(); + + assert_result_error_with_message( + vals(input).dynamic_scan(dynamic_scan), + "must be hierarchical and end in `/`", + ); + } +} diff --git a/kernel/src/plans/ir/mod.rs b/kernel/src/plans/ir/mod.rs new file mode 100644 index 0000000000..3547c696a6 --- /dev/null +++ b/kernel/src/plans/ir/mod.rs @@ -0,0 +1,12 @@ +//! Plan intermediate representation. +//! +//! - [`operation`] -- top-level [`Operation`] dispatch (I/O vs query) consumed by +//! [`PlanExecutor`](super::PlanExecutor). +//! - [`nodes`] -- the plan nodes: [`nodes::Operator`] and its payload structs. +//! - [`plan`] -- plan topology: [`plan::Plan`] holds a sequence of [`plan::PlanNode`]s wired into a +//! DAG by their input node indices. +pub mod nodes; +pub mod operation; +pub mod plan; + +pub use operation::{IoOperation, Operation}; diff --git a/kernel/src/plans/ir/nodes.rs b/kernel/src/plans/ir/nodes.rs new file mode 100644 index 0000000000..2d4531785b --- /dev/null +++ b/kernel/src/plans/ir/nodes.rs @@ -0,0 +1,1338 @@ +//! Plan node operator kinds and their payloads. +//! +//! [`Operator`] enumerates every operator. Each operator's payload struct is defined +//! below. + +use std::collections::HashMap; +use std::sync::{Arc, LazyLock}; + +use itertools::Itertools; +use strum::Display; +use url::Url; + +use crate::actions::deletion_vector::DeletionVectorDescriptor; +use crate::error::add_scalar_path_context; +use crate::expressions::{ColumnName, ExpressionRef, PredicateRef, Scalar, StructData}; +use crate::schema::{DataType, SchemaRef, StructField, StructType, ToSchema}; +use crate::utils::CollectInto; +use crate::{DeltaResult, Error, FileMeta}; + +// ============================================================================ +// Operator: enumerates every operator kind +// ============================================================================ + +/// Plan node operators, grouped below by input arity. Each variant wraps a payload struct +/// documenting that operator's semantics, invariants, and output shape. +/// +/// An operator that reshapes its rows (a source, projection, aggregation, or file scan) carries a +/// caller-declared `schema` field holding its output schema. The rest emit rows they were given, so +/// they inherit an input's schema; each payload's docs name which input. +#[derive(Debug, Clone, Display)] +#[strum(serialize_all = "snake_case")] +pub enum Operator { + // === Source operators (0 inputs) ========================================= + ScanParquet(ScanParquet), + ScanJson(ScanJson), + Values(Values), + + // === Unary operators (1 input) =========================================== + Project(Project), + Filter(Filter), + DynamicScan(DynamicScan), + Aggregate(Aggregate), + + // === Binary operators (2 inputs) ========================================= + SemiJoin(SemiJoin), + + // === N-ary operators (variable inputs) =================================== + UnionAll(UnionAll), +} + +/// Generate `From for Operator` for each listed variant, wrapping the payload in the +/// same-named [`Operator`] variant. Example: `Filter { .. }.into()` yields `Operator::Filter`). +macro_rules! impl_from_payload_for_operator { + ($($variant:ident),+ $(,)?) => { + $(impl From<$variant> for Operator { + fn from(payload: $variant) -> Self { + Operator::$variant(payload) + } + })+ + }; +} + +impl_from_payload_for_operator!( + ScanParquet, + ScanJson, + Values, + Project, + Filter, + DynamicScan, + Aggregate, + SemiJoin, + UnionAll, +); + +/// One file to scan plus literal values broadcast to every row read from that file. +/// +/// `file_constants` holds one [`Scalar`] per entry in the parent scan node's +/// [`ScanParquet::file_constant_columns`] / [`ScanJson::file_constant_columns`], in the +/// same order. +#[derive(Debug, Clone, PartialEq)] +pub struct ScanFile { + pub meta: FileMeta, + /// One [`Scalar`] per `file_constant_columns` on the enclosing scan node, same order. + pub file_constants: Vec, +} + +impl ScanFile { + /// A scan file with no file-constant column values. + pub fn new(meta: FileMeta) -> Self { + Self { + meta, + file_constants: Vec::new(), + } + } +} + +impl From for ScanFile { + fn from(meta: FileMeta) -> Self { + Self::new(meta) + } +} + +/// Reads Parquet `files` into row batches matching `schema`. The engine returns exactly the +/// columns named by `schema`, in schema order. +/// +/// Output row order is unspecified: the engine is free to read `files` in any order, in +/// parallel, and to interleave rows from different files. +/// +/// # Column resolution +/// +/// The engine iterates `schema`'s fields in order; for each field it produces one column of +/// output: +/// +/// 1. **Metadata columns**: if the field is annotated as a metadata column (e.g. via +/// [`StructField::create_metadata_column`] with [`MetadataColumnSpec::RowIndex`]), the engine +/// populates it from the read context rather than from the Parquet file. See [Metadata columns] +/// below. +/// 2. **File-constant columns**: if the field's name appears in [`Self::file_constant_columns`], +/// the engine broadcasts the corresponding entry from [`ScanFile::file_constants`] for the file +/// being read (not from Parquet bytes). See [File-constant columns] below. +/// 3. **Data columns**: otherwise the engine attempts to locate the field in the Parquet file, in +/// this order: +/// - **Field ID**: if the field carries a Parquet field ID via +/// [`ColumnMetadataKey::ParquetFieldId`] metadata, match it against the Parquet column with +/// the same field id. +/// - **Field name**: otherwise, or if no Parquet column has the requested field id, match by +/// column name. +/// - **No match**: the output column is filled with NULLs when the field is nullable, or an +/// error is returned when it is non-nullable. +/// +/// Parquet columns not referenced by any `schema` field are ignored. +/// +/// [Metadata columns]: #metadata-columns +/// [File-constant columns]: #file-constant-columns +/// [`StructField::create_metadata_column`]: crate::schema::StructField::create_metadata_column +/// [`MetadataColumnSpec::RowIndex`]: crate::schema::MetadataColumnSpec::RowIndex +/// [`ColumnMetadataKey::ParquetFieldId`]: crate::schema::ColumnMetadataKey::ParquetFieldId +/// +/// ## Example +/// +/// Consider a `schema` with the following fields (none of which are metadata columns): +/// - Column 0: `"i_logical"` (integer, non-null) with field ID 1 (via +/// [`ColumnMetadataKey::ParquetFieldId`]) +/// - Column 1: `"s"` (string, nullable) with no field ID metadata +/// - Column 2: `"i2"` (integer, nullable) with no field ID metadata +/// +/// And a Parquet file containing these columns: +/// - Column 0: `"i2"` (integer, nullable) with field ID 3 +/// - Column 1: `"i"` (integer, non-null) with field ID 1 +/// - No `"s"` column present +/// +/// Resolving each `schema` field in turn: +/// - `"i_logical"` matches `"i"` by field ID (both have ID 1). +/// - `"s"` has no matching Parquet column, so the output column is filled with NULLs. +/// - `"i2"` matches `"i2"` by column name (no field ID to match on). +/// +/// The returned data contains exactly 3 columns in schema order: +/// `{i_logical: parquet[1], s: NULL.., i2: parquet[0]}`. +/// +/// # Metadata columns +/// +/// A field marked as a row index metadata column (via [`StructField::create_metadata_column`] +/// with [`MetadataColumnSpec::RowIndex`]) is populated by the engine with the 0-based row +/// position within the file (`LONG`, non-nullable); a file with 5 rows yields `[0, 1, 2, 3, 4]`. +/// The column name is caller-chosen (commonly `"row_index"`). +/// +/// # File-constant columns +/// +/// [`Self::file_constant_columns`] names output fields whose values are identical for every row +/// in a given file (for example Delta partition columns or a table-changes `version`). Types and +/// nullability come from [`Self::schema`]; [`ScanFile::file_constants`] supplies the per-file +/// literals in the same order as `file_constant_columns`. +/// +/// File-constant columns are distinct from [metadata columns], which are engine-generated +/// (such as row index). [`DynamicScan::file_constant_columns`] is the same concept for the +/// [`DynamicScan`] node. +/// +/// # Invariants +/// +/// - `files[i].file_constants.len() == file_constant_columns.len()` for every `i`. +/// - Every name in `file_constant_columns` resolves to a field in `schema` that is not a metadata +/// column. +/// - Each scalar in `file_constants` is compatible with its schema field's type. +#[derive(Debug, Clone)] +pub struct ScanParquet { + pub files: Vec, + pub file_constant_columns: Vec, + pub schema: SchemaRef, +} + +/// Reads newline-delimited JSON `files` (one JSON object per line) into row batches matching +/// `schema`. +/// +/// Column resolution matches [`ScanParquet`]: metadata columns, then file-constant columns +/// (see [`Self::file_constant_columns`] and [`ScanFile::file_constants`]), then fields read from +/// each JSON line. Missing JSON fields produce NULL for nullable `schema` fields and an error for +/// non-nullable fields. +/// +/// Output row order is unspecified: the engine is free to read `files` in any order, in +/// parallel, and to interleave rows from different files. +/// +/// # File-constant columns +/// +/// Same contract as [`ScanParquet::file_constant_columns`]. +/// +/// # Invariants +/// +/// Same invariants as [`ScanParquet`]. +#[derive(Debug, Clone)] +pub struct ScanJson { + pub files: Vec, + pub file_constant_columns: Vec, + pub schema: SchemaRef, +} + +/// Inline literal rows. Each `rows[i]` carries one [`Scalar`] per **top-level** field +/// of `schema`, in field order; `rows[i].len() == schema.fields().count()` for every +/// row. Nested struct values are encoded as [`Scalar::Struct`], and array / map +/// values as [`Scalar::Array`] / [`Scalar::Map`]; nested leaves are not flattened +/// into the row vec. +/// +/// # Example (flat) +/// +/// Two rows over `{ id: int, active: bool }`: +/// +/// ```text +/// Values { +/// schema: { id: int, active: bool }, +/// rows: [ +/// [1, true], +/// [2, false], +/// ], +/// } +/// ``` +/// +/// produces: +/// +/// ```text +/// id | active +/// ---+-------- +/// 1 | true +/// 2 | false +/// ``` +/// +/// # Example (nested) +/// +/// Two rows over `{ id: int, address: { city: string, zip: int } }`. The `address` +/// field is one top-level slot in the row vec, populated with a single +/// `Scalar::Struct`: +/// +/// ```text +/// Values { +/// schema: { id: int, address: { city: string, zip: int } }, +/// rows: [ +/// [1, Scalar::Struct({ city: "NYC", zip: 10001 })], +/// [2, Scalar::Struct({ city: "SF", zip: 94102 })], +/// ], +/// } +/// ``` +/// +/// produces: +/// +/// ```text +/// id | address.city | address.zip +/// ---+--------------+------------ +/// 1 | NYC | 10001 +/// 2 | SF | 94102 +/// ``` +#[derive(Debug, Clone)] +pub struct Values { + pub schema: SchemaRef, + pub rows: Vec>, +} + +impl Values { + /// Literal `rows` matching `schema`. Empty `rows` is the uninhabited relation over `schema`. + pub fn new(schema: impl Into, rows: Vec>) -> Self { + Self { + schema: schema.into(), + rows, + } + } +} + +/// Collect rows of `T` into a [`Values`] node. +/// +/// Schema comes from [`ToSchema`]. Each row is converted via [`Into`] and peeled into +/// top-level field scalars (nested fields remain [`Scalar::Struct`]). +impl + ToSchema> FromIterator for Values { + fn from_iter>(iter: I) -> Self { + let rows = iter.into_iter().map(|row| row.into().into_parts().1); + Self::new(Arc::new(T::to_schema()), rows.collect()) + } +} + +/// Inverse of [`FromIterator for Values`]: rebuild each row as [`StructData`] and convert via +/// [`TryFrom`]. +impl TryFrom for Vec +where + T: TryFrom + ToSchema, +{ + type Error = Error; + + fn try_from(Values { schema, rows }: Values) -> DeltaResult { + rows.into_iter() + .enumerate() + .map(|(index, row)| { + let schema = schema.as_ref().clone(); + T::try_from(StructData::from_values_unchecked(schema, row)) + .map_err(|error| add_scalar_path_context(error, format!("[{index}]"))) + }) + .try_collect() + } +} + +/// Projects the input through `expr` into rows of `schema`. +/// +/// `expr` must be a struct constructor or struct patch whose fields match `schema`. It is +/// evaluated with `schema` as its output struct type: the struct's fields are the output +/// columns, `schema` supplies names and nullability, and any type or arity mismatch is an error. +/// Downstream nodes see the logical field names declared in `schema`. +/// +/// A struct patch carries the input struct through field by field, naming only the columns that +/// change -- replacing or dropping existing fields and injecting new ones -- while everything else +/// passes through unchanged, so it costs O(changes) rather than O(schema width). The patched +/// result still covers every field in `schema`. +/// +/// # Example +/// +/// Input `{ id, first, last, add: { path, size, stats_parsed: { numRecords } } }` projected to +/// `{ id, names, file_meta }`, showing passthrough, array construction, nested input access, and a +/// struct output column: +/// +/// ```text +/// Project { +/// expr: Expression::struct_from([ +/// col!("id"), +/// Expression::array([col!("first"), col!("last")]), +/// Expression::struct_from([ +/// col!("add.path"), +/// col!("add.size"), +/// col!("add.stats_parsed.numRecords"), +/// ]), +/// ]), +/// schema: { +/// id: int, +/// names: array, +/// file_meta: { path: string, size: long, num_records: long }, +/// }, +/// } +/// ``` +#[derive(Debug, Clone)] +pub struct Project { + pub expr: ExpressionRef, + pub schema: SchemaRef, +} + +/// Keeps input rows where `predicate` evaluates true (SQL null semantics). +/// Output schema is the input schema unchanged. +#[derive(Debug, Clone)] +pub struct Filter { + pub predicate: PredicateRef, +} + +/// File formats supported by [`DynamicScan`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum FileType { + Parquet, + Json, +} + +/// Reads data files from an upstream stream of file-metadata tuples, one input row per file. +/// For each row, the path, size, and last-modified columns describe the file; the engine resolves +/// its path against `base_url` (see below), opens it as `file_type`, and reads columns matching +/// `schema`. +/// +/// `file_constant_columns` lists upstream columns whose per-file values are broadcast onto +/// every emitted file row. This is file-constant metadata, the same concept as +/// [`ScanParquet::file_constant_columns`]. Each named input field must have the same type and +/// nullability as its output field. See the example below. +/// +/// `dv_column` names a nullable column on the upstream row holding a Delta +/// [`DeletionVectorDescriptor`] struct. The engine resolves it into a roaring bitmap +/// and drops file rows whose row index appears in the DV. A NULL value for a given +/// input row means "no DV for this file", so all file rows are emitted. +/// +/// [`DeletionVectorDescriptor`]: crate::actions::deletion_vector::DeletionVectorDescriptor +/// +/// Each path value is resolved against `base_url` via [`Url::join`]. URL-reference resolution need +/// not stay under `base_url`: a different-scheme absolute URL replaces the base, while a value +/// starting with `/` or `//` replaces its path or authority. +/// +/// Output row order is unspecified: the engine is free to read files in any order, in +/// parallel, and to interleave rows from different files. The relative order of upstream +/// rows is not preserved. +/// +/// # Example +/// +/// Given an upstream metadata stream and a `DynamicScan` configuration: +/// +/// ```text +/// upstream (metadata) +/// path | size | filemod | version | dv +/// -----------------+------+---------+---------+------ +/// part-0.parquet | 1024 | 100000 | 7 | NULL +/// part-1.parquet | 2048 | 200000 | 8 | NULL +/// ``` +/// ```text +/// DynamicScan { +/// schema: { id: int, name: string, version: long }, +/// file_type: Parquet, +/// base_url: "s3://table/", +/// file_constant_columns: ["version"], +/// path_column: "path", +/// file_size_column: "size", +/// last_modified_column: "filemod", +/// dv_column: "dv", +/// } +/// ``` +/// The engine opens `s3://table/part-0.parquet` and `s3://table/part-1.parquet`, reads +/// `{id, name}` from each, sees a NULL DV for each file so all rows survive, and +/// broadcasts the row's `version` onto every emitted file row. One possible output +/// (row order is not guaranteed): +/// ```text +/// | id | name | version +/// +----+------+-------- +/// | 3 | c | 8 +/// | 2 | b | 7 +/// | 4 | d | 8 +/// | 1 | a | 7 +/// ``` +#[derive(Debug, Clone)] +pub struct DynamicScan { + pub schema: SchemaRef, + pub file_type: FileType, + /// Hierarchical base URL ending in `/` against which per-row path values resolve. + pub base_url: Url, + pub file_constant_columns: Vec, + /// Non-nullable input column holding the per-row file path or URL fragment. + pub path_column: ColumnName, + /// Non-nullable input column with the file's total size in bytes. + pub file_size_column: ColumnName, + /// Non-nullable input column with the last-modified timestamp in milliseconds since epoch. + pub last_modified_column: ColumnName, + /// Nullable input column with the schema of [`DeletionVectorDescriptor`]. + pub dv_column: ColumnName, +} + +impl DynamicScan { + /// Constructs a [`DynamicScan`] whose emitted rows match `output_schema`. + /// + /// `input_schema` describes the upstream rows containing file metadata. The scan reads + /// `file_type` files relative to `base_url`. + /// + /// # Errors + /// + /// Returns an error when `base_url` is not hierarchical or does not end in `/`; when a required + /// metadata or deletion-vector column is absent from `input_schema`, has an incompatible type, + /// or has invalid nullability; or when a file-constant column is absent from either schema, is + /// a metadata column, or has different input and output types or nullability. + #[allow(clippy::too_many_arguments)] + pub fn try_new( + input_schema: &SchemaRef, + output_schema: impl Into, + file_type: FileType, + base_url: Url, + file_constant_columns: impl IntoIterator>, + path_column: ColumnName, + file_size_column: ColumnName, + last_modified_column: ColumnName, + dv_column: ColumnName, + ) -> DeltaResult { + let schema = output_schema.into(); + let file_constant_columns = file_constant_columns + .into_iter() + .map(Into::into) + .collect::>(); + let dynamic_scan = Self { + schema, + file_type, + base_url, + file_constant_columns, + path_column, + file_size_column, + last_modified_column, + dv_column, + }; + dynamic_scan.validate_input(input_schema)?; + Ok(dynamic_scan) + } + + /// Validates the columns consumed by this scan against an upstream `input_schema`. + /// + /// Returns `Ok(())` when the base URL is valid and every configured column resolves with the + /// required type and nullability. + /// + /// # Errors + /// + /// Returns an error when `base_url` is not hierarchical or does not end in `/`; when a required + /// metadata or deletion-vector column is absent, has an incompatible type, or has invalid + /// nullability; or when a file-constant column is absent from either schema, is a metadata + /// column, or has different input and output types or nullability. + pub fn validate_input(&self, input_schema: &SchemaRef) -> DeltaResult<()> { + static DELETION_VECTOR_DATA_TYPE: LazyLock = + LazyLock::new(|| DataType::from(DeletionVectorDescriptor::to_schema())); + + if self.base_url.cannot_be_a_base() || !self.base_url.path().ends_with('/') { + return Err(Error::generic(format!( + "dynamic scan: base URL `{}` must be hierarchical and end in `/`", + self.base_url + ))); + } + + Self::validate_required_column(input_schema, &self.path_column, &DataType::STRING)?; + Self::validate_required_column(input_schema, &self.file_size_column, &DataType::LONG)?; + Self::validate_required_column(input_schema, &self.last_modified_column, &DataType::LONG)?; + Self::validate_file_constant_columns( + input_schema, + &self.schema, + &self.file_constant_columns, + )?; + + let fields = input_schema + .fields_of_path(&self.dv_column) + .map_err(|err| { + Error::generic(format!( + "dynamic scan: deletion-vector column `{}` is invalid: {err}", + self.dv_column + )) + })?; + let Some((field, _ancestors)) = fields.split_last() else { + return Err(Error::internal_error("fields_of_path returned no fields")); + }; + let expected = &*DELETION_VECTOR_DATA_TYPE; + if field.data_type() != expected { + return Err(Error::generic(format!( + "dynamic scan: deletion-vector column `{}` must have type {expected}, found {}", + self.dv_column, + field.data_type() + ))); + } + if !field.is_nullable() { + return Err(Error::generic(format!( + "dynamic scan: deletion-vector column `{}` must be nullable", + self.dv_column + ))); + } + + Ok(()) + } + + fn validate_required_column( + schema: &SchemaRef, + column: &ColumnName, + expected_type: &DataType, + ) -> DeltaResult<()> { + let fields = schema.fields_of_path(column)?; + let Some((field, ancestors)) = fields.split_last() else { + return Err(Error::internal_error("fields_of_path returned no fields")); + }; + if field.data_type() != expected_type { + return Err(Error::generic(format!( + "dynamic scan: column `{column}` must have type {expected_type}, found {}", + field.data_type() + ))); + } + if field.is_nullable() || ancestors.iter().any(|field| field.is_nullable()) { + return Err(Error::generic(format!( + "dynamic scan: required column `{column}` is nullable" + ))); + } + Ok(()) + } + + fn validate_file_constant_columns( + input_schema: &SchemaRef, + output_schema: &SchemaRef, + file_constant_columns: &[String], + ) -> DeltaResult<()> { + for name in file_constant_columns { + let Some(input_field) = input_schema.field(name) else { + return Err(Error::generic(format!( + "dynamic scan file_constant source: column `{name}` not found; schema has \ + {:?}", + Vec::from_iter(input_schema.fields().map(|field| field.name())), + ))); + }; + if input_field.is_metadata_column() { + return Err(Error::generic(format!( + "dynamic scan file_constant source: column `{name}` is a metadata column" + ))); + } + let Some(output_field) = output_schema.field(name) else { + return Err(Error::generic(format!( + "dynamic scan file_constant: column `{name}` not found; schema has {:?}", + Vec::from_iter(output_schema.fields().map(|field| field.name())), + ))); + }; + if output_field.is_metadata_column() { + return Err(Error::generic(format!( + "dynamic scan file_constant: column `{name}` is a metadata column" + ))); + } + if input_field.data_type() != output_field.data_type() + || input_field.is_nullable() != output_field.is_nullable() + { + return Err(Error::generic(format!( + "dynamic scan file_constant: column `{name}` must have the same type and \ + nullability in input and output" + ))); + } + } + Ok(()) + } +} + +/// Groups input rows by `group_by` (a global aggregation over all rows when `group_by` is +/// empty) and computes one output column per [`Agg`] in `aggs`. The output `schema` lists the +/// group-by key columns first (in order), then the aggregate columns (in order). +/// +/// Build an `Aggregate` with [`Aggregate::group_by`], which derives `schema` from the input +/// schema -- including each output column's name, type, and nullability. +/// +/// # Output schema +/// +/// - **Group keys** pass through verbatim: each key column keeps its input type, nullability, and +/// metadata. +/// - **Aggregate columns**: name, type, and nullability come from each [`Agg`] (see per-function +/// docs); use [`AggregateBuilder::aggregate_as`] to override the name. Aggregates that preserve +/// the input type also preserve its field metadata; fixed-LONG aggregates emit a bare field. +/// +/// # SQL equivalent +/// +/// ```sql +/// SELECT +/// , +/// +/// FROM input +/// GROUP BY +/// ``` +/// +/// SQL grouping uses null-safe equals, so `(NULL, b)`, `(a, NULL)`, and `(NULL, NULL)` are all +/// different groups. +/// +/// # Example +/// +/// Each person's best and worst score across their bowling games: +/// +/// ```text +/// Aggregate { +/// group_by: [name], +/// aggs: [max(score) AS high, min(score) AS low], +/// schema: { name: string, high: long, low: long }, +/// } +/// ``` +/// +/// Input: +/// +/// ```text +/// name | score +/// --------+------ +/// Alice | 140 +/// Bob | 200 +/// Alice | 180 +/// Bob | 160 +/// Alice | 155 +/// Charlie| 175 +/// ``` +/// +/// Output: +/// +/// ```text +/// name | high | low +/// --------+------+----- +/// Bob | 200 | 160 +/// Charlie| 175 | 175 +/// Alice | 180 | 140 +/// ``` +/// +/// An ungrouped aggregate (`group_by` empty) always emits one row. Over empty input that row holds +/// each agg's initial value (i.e. NULL for [`Agg::min`] and `0` for [`Agg::count`]). See individual +/// [`Agg`] docs for per-function initial values and NULL-handling semantics. +#[derive(Debug, Clone)] +pub struct Aggregate { + /// Group-by key columns, emitted first in the output schema. Empty means a single global + /// group over all input rows. + pub group_by: Vec, + /// The aggregate columns, emitted after the group keys in the output schema. + pub aggs: Vec, + /// Output schema: group-by key columns followed by aggregate columns. + pub schema: SchemaRef, +} + +impl Aggregate { + /// Starts building an ungrouped [`Aggregate`] over `input_schema`. Add aggregators directly + /// with [`aggregate`](AggregateBuilder::aggregate) or using named helpers + /// (e.g. [`max`](AggregateBuilder::max)), and finalize the aggregate by calling + /// [`build`](AggregateBuilder::build). The output schema follows aggregator insertion order. + pub fn ungrouped(input_schema: SchemaRef) -> AggregateBuilder { + Self::group_by(input_schema, std::iter::empty::()) + } + + /// Starts building an [`Aggregate`] over `input_schema`, grouped by `grouping_keys`. Add + /// aggregators directly with [`aggregate`](AggregateBuilder::aggregate) or using named helpers + /// (e.g. [`max`](AggregateBuilder::max)), and finalize the aggregate by calling + /// [`build`](AggregateBuilder::build). Grouping keys are emitted first in the output schema, + /// followed by aggregators in insertion order. + pub fn group_by( + input_schema: SchemaRef, + grouping_keys: impl CollectInto>, + ) -> AggregateBuilder { + AggregateBuilder { + input_schema, + group_by: grouping_keys.collect_into(), + aggs: Vec::new(), + } + } +} + +/// An aggregate function and its operand column(s) within an [`Aggregate`] operator. +#[derive(Debug, Clone)] +pub enum Agg { + /// Operand for [`Agg::min`]. + Min(ColumnName), + /// Operand for [`Agg::max`]. + Max(ColumnName), + /// Operand for [`Agg::sum`]. + Sum(ColumnName), + /// Operand for [`Agg::count`]. + Count(ColumnName), + /// [`Agg::count_star`] has no operands. + CountStar, + /// Operands for [`Agg::min_non_null_by`]. + MinNonNullBy(NonNullByOperands), + /// Operands for [`Agg::max_non_null_by`]. + MaxNonNullBy(NonNullByOperands), +} + +/// Operands for [`Agg::min_non_null_by`] and [`Agg::max_non_null_by`]. +#[derive(Debug, Clone)] +pub struct NonNullByOperands { + pub value: ColumnName, + pub null_sentinel: ColumnName, + pub key: ColumnName, +} + +impl Agg { + /// Like [`max`](Self::max), but selects the least non-NULL value in each group. + /// + /// ```text + /// [3, NULL, 5, 1] -> 1 + /// [NULL, NULL] -> NULL + /// [] -> NULL + /// ``` + pub fn min(value: impl Into) -> Self { + Self::Min(value.into()) + } + + /// The greatest non-NULL value in each group, or NULL if the group has no non-NULL value. + /// The output is always nullable, with name and type matching `value`. + /// + /// ```text + /// [3, NULL, 5, 1] -> 5 + /// [NULL, NULL] -> NULL + /// [] -> NULL + /// ``` + pub fn max(value: impl Into) -> Self { + Self::Max(value.into()) + } + + /// The sum of non-NULL LONG values in each group, or NULL if the group has no non-NULL value. + /// The output is always a nullable LONG, with default name matching `value`. + /// + /// ```text + /// [3, NULL, 5, 1] -> 9 + /// [NULL, NULL] -> NULL + /// [] -> NULL + /// ``` + pub fn sum(value: impl Into) -> Self { + Self::Sum(value.into()) + } + + /// The number of non-NULL values in `value` for each group. The output is always a non-nullable + /// LONG, with default name matching `value`. + /// + /// ```text + /// [3, NULL, 5, 1] -> 3 + /// [NULL, NULL] -> 0 + /// [] -> 0 + /// ``` + pub fn count(value: impl Into) -> Self { + Self::Count(value.into()) + } + + /// The number of input rows in each group (`COUNT(*)`). The output is always a non-nullable + /// LONG named `count` by default. + /// + /// ```text + /// [3, NULL, 5, 1] -> 4 + /// [NULL, NULL] -> 2 + /// [] -> 0 + /// ``` + pub fn count_star() -> Self { + Self::CountStar + } + + /// Like [`max_non_null_by`](Self::max_non_null_by), but selects the `value` from the qualifying + /// row with the *least* `key`. + pub fn min_non_null_by( + value: impl Into, + null_sentinel: impl Into, + key: impl Into, + ) -> Self { + Self::MinNonNullBy(NonNullByOperands { + value: value.into(), + null_sentinel: null_sentinel.into(), + key: key.into(), + }) + } + + /// The `value` from a row with the greatest `key` where `null_sentinel` and `key` are both + /// non-NULL. Returns NULL if no qualifying row exists. A winning `value` may itself be NULL. It + /// is unspecified which of multiple rows with greatest `key` provides the winning `value`. The + /// output is always nullable, with name and type matching `value`. + /// + /// ```text + /// key | sentinel | value -> NULL + /// -----+----------+------ + /// 1 | present | a + /// 3 | present | c + /// 5 | present | NULL (greatest qualifying key; NULL value is retained) + /// 7 | NULL | d (ignored: NULL sentinel) + /// NULL | present | e (ignored: NULL key) + /// + /// (no rows) -> NULL + /// ``` + /// + /// Most systems with a native `max_by` only provide a two-arg form that considers all rows with + /// non-NULL keys. The sentinel check can be added manually in one of two ways: + /// + /// ```sql + /// -- FILTER that drops NULL-sentinel rows before aggregating + /// max_by(value, key) FILTER (WHERE sentinel IS NOT NULL) + /// + /// -- NULL out the key when sentinel is NULL, which max_by then ignores. Use this where + /// -- FILTER is unavailable, such as a DataFrame API with no filtered-aggregate form. + /// max_by(value, CASE WHEN sentinel IS NOT NULL THEN key END) + /// ``` + /// + /// In systems without `max_by`, it can also be expressed using window functions, with the + /// caveat that window functions don't work correctly for ungrouped aggs over empty input + /// (produces no rows when it should produce one row containing initial agg values): + /// + /// ```sql + /// SELECT + /// , + /// value + /// FROM ( + /// SELECT + /// value, + /// , + /// ROW_NUMBER() OVER ( + /// PARTITION BY + /// ORDER BY key DESC + /// ) AS rn + /// FROM input + /// WHERE key IS NOT NULL AND null_sentinel IS NOT NULL + /// ) WHERE rn = 1 + /// ``` + pub fn max_non_null_by( + value: impl Into, + null_sentinel: impl Into, + key: impl Into, + ) -> Self { + Self::MaxNonNullBy(NonNullByOperands { + value: value.into(), + null_sentinel: null_sentinel.into(), + key: key.into(), + }) + } + + /// Derives this aggregate's output [`StructField`] over `input_schema`, validating that every + /// operand column resolves. + fn output_field( + &self, + input_schema: &StructType, + alias: Option, + ) -> DeltaResult { + // `output_data_type: None` preserves the input field's type and metadata; `Some` overrides + // the type and strips metadata (new column). + let resolve = |value: &ColumnName, output_data_type: Option, nullable: bool| { + let field = input_schema.field_at(value)?; + let (data_type, metadata) = match output_data_type { + Some(data_type) => (data_type, HashMap::new()), + None => (field.data_type.clone(), field.metadata.clone()), + }; + Ok(StructField { + // Without clone, we capture `alias` by value and `CountStar` arm can't use it + name: alias.clone().unwrap_or_else(|| field.name.clone()), + data_type, + metadata, + nullable, + }) + }; + match self { + Agg::Min(value) | Agg::Max(value) => resolve(value, None, true), + Agg::Sum(value) => resolve(value, Some(DataType::LONG), true), + Agg::Count(value) => resolve(value, Some(DataType::LONG), false), + Agg::CountStar => Ok(StructField::not_null( + alias.unwrap_or_else(|| "count".to_string()), + DataType::LONG, + )), + Agg::MinNonNullBy(operands) | Agg::MaxNonNullBy(operands) => { + let _ = input_schema.field_at(&operands.key)?; + let _ = input_schema.field_at(&operands.null_sentinel)?; + resolve(&operands.value, None, true) + } + } + } +} + +/// Builds an [`Aggregate`] over an input schema, deriving the output schema from the group keys +/// and aggregators. +/// +/// Created by [`Aggregate::group_by`], which fixes the group keys. Aggregators are then collected +/// by the named helpers or [`aggregate`](Self::aggregate); [`build`](Self::build) resolves keys and +/// aggregators against the input schema; derives each output column's name, type and nullability +/// from its [`Agg`] or group-by column; and validates that all output column names are unique. +#[derive(Debug)] +pub struct AggregateBuilder { + input_schema: SchemaRef, + group_by: Vec, + aggs: Vec<(Agg, Option)>, +} + +impl AggregateBuilder { + /// Adds an aggregate column, emitted after the group keys in call order, using each [`Agg`]'s + /// default output name (see the per-function docs). Prefer the named helpers for the common + /// case (e.g. [`max`](Self::max); use [`aggregate_as`](Self::aggregate_as) to override the + /// output name. + pub fn aggregate(mut self, agg: Agg) -> Self { + self.aggs.push((agg, None)); + self + } + + /// Like [`aggregate`](Self::aggregate), but with the specified output name. + pub fn aggregate_as(mut self, agg: Agg, name: impl Into) -> Self { + self.aggs.push((agg, Some(name.into()))); + self + } + + /// Adds an unaliased [`Agg::min`] over `value`. + pub fn min(self, value: impl Into) -> Self { + self.aggregate(Agg::min(value)) + } + + /// Adds an unaliased [`Agg::max`] over `value`. + pub fn max(self, value: impl Into) -> Self { + self.aggregate(Agg::max(value)) + } + + /// Adds an unaliased [`Agg::sum`] over `value`. + pub fn sum(self, value: impl Into) -> Self { + self.aggregate(Agg::sum(value)) + } + + /// Adds an unaliased [`Agg::count`] over `value`. + pub fn count(self, value: impl Into) -> Self { + self.aggregate(Agg::count(value)) + } + + /// Adds an unaliased [`Agg::count_star`]. + pub fn count_star(self) -> Self { + self.aggregate(Agg::count_star()) + } + + /// Adds an unaliased [`Agg::min_non_null_by`] over `value`, qualifying rows with + /// `null_sentinel` and keyed on `key`. + pub fn min_non_null_by( + self, + value: impl Into, + null_sentinel: impl Into, + key: impl Into, + ) -> Self { + self.aggregate(Agg::min_non_null_by(value, null_sentinel, key)) + } + + /// Adds an unaliased [`Agg::max_non_null_by`] over `value`, qualifying rows with + /// `null_sentinel` and keyed on `key`. + pub fn max_non_null_by( + self, + value: impl Into, + null_sentinel: impl Into, + key: impl Into, + ) -> Self { + self.aggregate(Agg::max_non_null_by(value, null_sentinel, key)) + } + + /// Resolves group keys and aggregators against the input schema and builds the [`Aggregate`]. + /// + /// # Errors + /// + /// Returns an error if a group key or an aggregate's operand column is not found in the input + /// schema, or if two output columns would share a name (case-insensitive). + pub fn build(self) -> DeltaResult { + let mut fields = Vec::with_capacity(self.group_by.len() + self.aggs.len()); + for key in &self.group_by { + fields.push(self.input_schema.field_at(key)?.clone()); + } + let mut aggs = Vec::with_capacity(self.aggs.len()); + for (agg, alias) in self.aggs { + fields.push(agg.output_field(&self.input_schema, alias)?); + aggs.push(agg); + } + // NOTE: `StructType::try_new` rejects duplicate (case-insensitive) output column names. + Ok(Aggregate { + group_by: self.group_by, + aggs, + schema: Arc::new(StructType::try_new(fields)?), + }) + } +} + +impl TryFrom for Aggregate { + type Error = Error; + + fn try_from(builder: AggregateBuilder) -> DeltaResult { + builder.build() + } +} + +/// Performs a semi join between two inputs, `inputs.len() == 2`, the child +/// nodes are `[probe, build]` in this order. It emits a subset of probe rows; +/// the build side acts as a filter and never contributes columns. This is +/// analogous to a SQL `SEMI JOIN` (`inverted = false`) or `ANTI JOIN` +/// (`inverted = true`). A semi join finds all probe rows that are present in +/// the build side, and an anti join finds all probe rows **not** present in the +/// build side. This is analogous to set intersection and set difference, +/// respectively. +/// +/// The output schema is the same as the probe input's schema. +/// +/// # Example +/// +/// ```text +/// SemiJoin { probe_keys: ["path"], build_keys: ["path"] } +/// +/// probe build +/// path | version path +/// -----+-------- ---- +/// a | 1 b +/// b | 2 d +/// c | 3 +/// +/// output (inverted = false, semi join: probe rows whose path is in build): +/// path | version +/// -----+-------- +/// b | 2 +/// +/// output (inverted = true, anti join: probe rows whose path is not in build): +/// path | version +/// -----+-------- +/// a | 1 +/// c | 3 +/// ``` +#[derive(Debug, Clone)] +pub struct SemiJoin { + pub inverted: bool, + pub probe_keys: Vec, + pub build_keys: Vec, +} + +/// The unordered bag union of N input relations. All rows of all inputs appear in the +/// output, in arbitrary order. All input schemas must agree, and the output schema +/// is the common schema of the inputs. +/// +/// # Example +/// +/// `UnionAll` over two relations with schema `{ id: int }`: +/// +/// ```text +/// input 0: +/// id +/// -- +/// 1 +/// 2 +/// 3 +/// +/// input 1: +/// id +/// -- +/// 3 +/// 4 +/// 5 +/// +/// output (arbitrary order; bag semantics keep the duplicate 3): +/// id +/// -- +/// 4 +/// 1 +/// 3 +/// 2 +/// 5 +/// 3 +/// ``` +#[derive(Debug, Clone)] +pub struct UnionAll; + +#[cfg(test)] +mod tests { + use delta_kernel_derive::{IntoStructData, ToSchema, TryFromStructData}; + + use super::*; + use crate::expressions::column_name; + use crate::schema::{DataType, MetadataValue, StructField}; + use crate::unit_test_utils::assert_result_error_with_message; + + /// Builds a flat `LONG` schema from `(name, nullable)` pairs. + fn schema(fields: &[(&str, bool)]) -> SchemaRef { + Arc::new(StructType::new_unchecked(fields.iter().map( + |(name, nullable)| StructField::new(*name, DataType::LONG, *nullable), + ))) + } + + #[test] + fn output_lists_group_keys_then_aggregates_in_order() { + let input = schema(&[("g", false), ("a", true), ("b", true)]); + let agg = Aggregate::group_by(input, [column_name!("g")]) + .max(column_name!("a")) + .min(column_name!("b")) + .build() + .unwrap(); + let names: Vec<&str> = agg.schema.fields().map(|f| f.name().as_str()).collect(); + assert_eq!(names, ["g", "a", "b"]); + } + + /// Group keys and type-preserving aggregates keep input field metadata; only nullability + /// changes. Fixed-LONG aggregates build a fresh field and so carry no metadata. + #[test] + fn output_fields_preserve_input_field_metadata() { + let metadata = [("k", MetadataValue::Number(7))]; + let input = Arc::new(StructType::new_unchecked([ + StructField::not_null("g", DataType::LONG).with_metadata(metadata.clone()), + StructField::not_null("a", DataType::LONG).with_metadata(metadata.clone()), + StructField::not_null("s", DataType::LONG).with_metadata(metadata), + ])); + let agg = Aggregate::group_by(input, [column_name!("g")]) + .max(column_name!("a")) + .sum(column_name!("s")) + .build() + .unwrap(); + + let key = agg.schema.field("g").unwrap(); + assert!(!key.nullable); + assert_eq!(key.metadata()["k"], MetadataValue::Number(7)); + let max = agg.schema.field("a").unwrap(); + assert!(max.nullable); + assert_eq!(max.metadata()["k"], MetadataValue::Number(7)); + assert!(agg.schema.field("s").unwrap().metadata().is_empty()); + } + + /// Output nullability is fixed by the aggregate kind, independent of input nullability. + #[rstest::rstest] + #[case::min(Agg::min(column_name!("a")), "a", true)] + #[case::max(Agg::max(column_name!("a")), "a", true)] + #[case::sum(Agg::sum(column_name!("a")), "a", true)] + #[case::count(Agg::count(column_name!("a")), "a", false)] + #[case::count_star(Agg::count_star(), "count", false)] + #[case::min_non_null_by( + Agg::min_non_null_by(column_name!("a"), column_name!("s"), column_name!("v")), + "a", + true + )] + #[case::max_non_null_by( + Agg::max_non_null_by(column_name!("a"), column_name!("s"), column_name!("v")), + "a", + true + )] + fn agg_output_nullability( + #[case] agg: Agg, + #[case] name: &str, + #[case] nullable: bool, + #[values(true, false)] value_nullable: bool, + ) { + let input = schema(&[("a", value_nullable), ("s", true), ("v", true)]); + let built = Aggregate::ungrouped(input).aggregate(agg).build().unwrap(); + let field = built.schema.field(name).unwrap(); + assert_eq!(field.nullable, nullable); + assert_eq!(field.data_type(), &DataType::LONG); + } + + #[test] + fn alias_overrides_default_output_name() { + let input = schema(&[("a", true)]); + let agg = Aggregate::group_by(input, []) + .aggregate_as(Agg::max(column_name!("a")), "a_max") + .build() + .unwrap(); + assert!(agg.schema.field("a_max").is_some()); + assert!(agg.schema.field("a").is_none()); + } + + #[test] + fn duplicate_output_names_are_rejected() { + let input = schema(&[("a", true)]); + // min and max of the same column collide on the default name "a". + let result = Aggregate::group_by(input, []) + .min(column_name!("a")) + .max(column_name!("a")) + .build(); + assert_result_error_with_message(result, "Duplicate field name"); + } + + #[test] + fn distinct_aliases_resolve_min_max_collision() { + let input = schema(&[("a", true)]); + let agg = Aggregate::group_by(input, []) + .aggregate_as(Agg::min(column_name!("a")), "a_min") + .aggregate_as(Agg::max(column_name!("a")), "a_max") + .build() + .unwrap(); + let names: Vec<&str> = agg.schema.fields().map(|f| f.name().as_str()).collect(); + assert_eq!(names, ["a_min", "a_max"]); + } + + #[rstest::rstest] + #[case::missing_value_column(false)] + #[case::missing_group_key(true)] + fn build_rejects_missing_column(#[case] missing_in_key: bool) { + let input = schema(&[("a", true)]); + let (keys, value) = if missing_in_key { + (vec![column_name!("missing")], column_name!("a")) + } else { + (vec![], column_name!("missing")) + }; + let result = Aggregate::group_by(input, keys).max(value).build(); + assert_result_error_with_message(result, "missing"); + } + + /// A `*_non_null_by` aggregate whose key column is absent is rejected, even with no group keys + /// (i.e. the validation does not ride on the grouped/nullability path). + #[test] + fn build_rejects_missing_non_null_by_key() { + let input = schema(&[("a", true)]); + let result = Aggregate::group_by(input, []) + .max_non_null_by( + column_name!("a"), + column_name!("a"), + column_name!("missing"), + ) + .build(); + assert_result_error_with_message(result, "missing"); + } + + #[test] + fn build_rejects_missing_non_null_by_sentinel_column() { + let input = schema(&[("a", true), ("v", true)]); + let result = Aggregate::group_by(input, []) + .max_non_null_by( + column_name!("a"), + column_name!("missing"), + column_name!("v"), + ) + .build(); + assert_result_error_with_message(result, "missing"); + } + + #[derive(Clone, Debug, PartialEq, ToSchema, IntoStructData, TryFromStructData)] + struct Address { + city: String, + } + + #[derive(Clone, Debug, PartialEq, ToSchema, IntoStructData, TryFromStructData)] + struct Person { + id: i32, + address: Address, + } + + #[test] + fn values_from_iter_peels_top_level_and_keeps_nested_struct() { + let values = Values::from_iter([Person { + id: 1, + address: Address { city: "NYC".into() }, + }]); + + assert_eq!( + values + .schema + .fields() + .map(|f| f.name().as_str()) + .collect::>(), + ["id", "address"] + ); + assert_eq!(values.rows.len(), 1); + assert_eq!(values.rows[0].len(), 2); + assert_eq!(values.rows[0][0], Scalar::Integer(1)); + let Scalar::Struct(address) = &values.rows[0][1] else { + panic!("expected nested Struct for address"); + }; + assert_eq!(address.values(), &[Scalar::String("NYC".into())]); + } + + #[test] + fn values_from_iter_empty_still_carries_schema() { + let values: Values = std::iter::empty::().collect(); + assert!(values.rows.is_empty()); + assert_eq!(values.schema.num_fields(), 2); + } + + #[test] + fn values_round_trips_through_vec() { + let people = vec![ + Person { + id: 1, + address: Address { city: "NYC".into() }, + }, + Person { + id: 2, + address: Address { city: "SF".into() }, + }, + ]; + let values = Values::from_iter(people.clone()); + assert_eq!(Vec::::try_from(values).unwrap(), people); + } + + #[test] + fn values_conversion_adds_row_index_to_error_path() { + let mut values = Values::from_iter([Person { + id: 1, + address: Address { city: "NYC".into() }, + }]); + values.rows[0][0] = Scalar::from("not an integer"); + assert_result_error_with_message( + Vec::::try_from(values), + "[0].id: expected i32, found string", + ); + } +} diff --git a/kernel/src/plans/ir/operation.rs b/kernel/src/plans/ir/operation.rs new file mode 100644 index 0000000000..da9ecb7ed7 --- /dev/null +++ b/kernel/src/plans/ir/operation.rs @@ -0,0 +1,124 @@ +//! Top-level [`Operation`] dispatch -- I/O ops and relational queries. +//! +//! The executor receives an [`Operation`] and produces a +//! [`PlanResult`](crate::plans::PlanResult). + +use bytes::Bytes; +use url::Url; + +use super::plan::Plan; +use crate::{FileMeta, FileSlice}; + +/// Represents a set of instructions that the +/// [`PlanExecutor`](crate::plans::PlanExecutor) should perform. +/// +/// It can either be an IO operation or a declarative query. +#[derive(Debug)] +pub enum Operation { + /// A singular I/O operation that returns concretely typed data such as bytes or file metadata. + IoOperation(IoOperation), + /// A query on relational-like data, expressed as a [`Plan`]: a DAG of + /// [`PlanNode`](super::plan::PlanNode)s whose terminal (last) node produces the rows the + /// engine streams to the caller. + QueryPlan(Plan), +} + +impl Operation { + /// Serializes this [`Operation`] into its proto wire representation. + pub fn to_proto_bytes(&self) -> Vec { + use prost::Message as _; + crate::plans::proto::operation::Operation::from(self).encode_to_vec() + } +} + +/// A singular I/O operation that returns typed data such as raw bytes or file metadata. +/// +/// Each variant describes an operation and its parameters. The shape of the result it produces +/// is documented on the variant in terms of [`PlanResult`](crate::plans::PlanResult). +#[derive(Debug)] +pub enum IoOperation { + /// Recursively list files at the given URL. + /// + /// Should return a [`PlanResult::FileMeta`](crate::plans::PlanResult::FileMeta) with one entry + /// per file. See [`StorageHandler::list_from`] for more details on the ordering contract. + /// + /// [`StorageHandler::list_from`]: crate::StorageHandler::list_from + FileListing { url: Url }, + /// Read raw bytes from one or more files (or byte ranges within files). + /// + /// Each [`FileSlice`] specifies a file URL and an optional byte range. Results are returned + /// as [`PlanResult::Bytes`](crate::plans::PlanResult::Bytes) in the same order as the input + /// slices, with one Bytes buffer per file slice. + ReadBytes { files: Vec }, + /// Write raw bytes to a file at the given URL. + /// + /// Returns [`PlanResult::Unit`](crate::plans::PlanResult::Unit) on success. If `overwrite` + /// is false and the file already exists, the executor should return + /// [`Error::FileAlreadyExists`](crate::Error::FileAlreadyExists). + WriteBytes { + url: Url, + data: Bytes, + overwrite: bool, + }, + /// Retrieve metadata for a single file (HEAD request). + /// + /// Returns [`PlanResult::FileMeta`](crate::plans::PlanResult::FileMeta) with a single entry. + /// If the file does not exist, the executor should return an error. + HeadFile { url: Url }, + /// Atomically copy a file from `source` to `destination`. + /// + /// The copy must be atomic: if `destination` already exists, the executor should return + /// [`Error::FileAlreadyExists`](crate::Error::FileAlreadyExists) without modifying it. + /// Returns [`PlanResult::Unit`](crate::plans::PlanResult::Unit) on success. + AtomicCopy { source: Url, destination: Url }, + /// Read the footer of a Parquet file. + /// + /// Returns [`PlanResult::ParquetFooter`] containing the file's footer metadata (schema and + /// any future extensions). See [`ParquetHandler::read_parquet_footer`] for the full contract. + /// + /// [`PlanResult::ParquetFooter`]: crate::plans::PlanResult::ParquetFooter + /// [`ParquetHandler::read_parquet_footer`]: crate::ParquetHandler::read_parquet_footer + ParquetFooter { file: FileMeta }, + // TODO(#2820): add a `Delete { url: Url }` variant (plus an `IoOperation::delete` + // constructor) so `PlanBasedStorageHandler::delete` can route through the plan path + // instead of `unimplemented!()`. +} + +impl IoOperation { + /// Constructs an [`IoOperation::FileListing`] for the given URL. + pub fn file_listing(url: Url) -> Self { + Self::FileListing { url } + } + + /// Constructs an [`IoOperation::ReadBytes`] for the given file slices. + pub fn read_bytes(files: Vec) -> Self { + Self::ReadBytes { files } + } + + /// Constructs an [`IoOperation::WriteBytes`] for the given URL, data, and overwrite flag. + pub fn write_bytes(url: Url, data: Bytes, overwrite: bool) -> Self { + Self::WriteBytes { + url, + data, + overwrite, + } + } + + /// Constructs an [`IoOperation::HeadFile`] for the given URL. + pub fn head_file(url: Url) -> Self { + Self::HeadFile { url } + } + + /// Constructs an [`IoOperation::AtomicCopy`] for the given source and destination URLs. + pub fn atomic_copy(source: Url, destination: Url) -> Self { + Self::AtomicCopy { + source, + destination, + } + } + + /// Constructs an [`IoOperation::ParquetFooter`] for the given file metadata. + pub fn parquet_footer(file: FileMeta) -> Self { + Self::ParquetFooter { file } + } +} diff --git a/kernel/src/plans/ir/plan.rs b/kernel/src/plans/ir/plan.rs new file mode 100644 index 0000000000..eb1e669b96 --- /dev/null +++ b/kernel/src/plans/ir/plan.rs @@ -0,0 +1,96 @@ +//! Plan containers ([`Plan`], [`PlanNode`]). + +pub use super::nodes::Operator; + +// ============================================================================ +// Plan nodes +// ============================================================================ + +/// One node in a plan: an [`Operator`] and the indices of its input nodes. +/// +/// A node is identified by its position in [`Plan::nodes`]; `inputs` lists those indices for +/// the upstream nodes this operator reads from. `inputs` order is interpreted per [`Operator`] +/// (e.g. for `Operator::SemiJoin` the convention is `[probe, build]`). `Operator::UnionAll` +/// emits the rows of all inputs regardless of input order. +#[derive(Debug, Clone)] +pub struct PlanNode { + pub op: Operator, + pub inputs: Vec, +} + +impl PlanNode { + /// A node applying `op` over the nodes at `inputs` (indices into [`Plan::nodes`]). + pub fn new(op: impl Into, inputs: Vec) -> Self { + Self { + op: op.into(), + inputs, + } + } +} + +// ============================================================================ +// Plans +// ============================================================================ + +/// A plan: an ordered sequence of [`PlanNode`]s forming a dataflow DAG. +/// +/// A node is identified by its index in `nodes`. Each [`PlanNode`] pairs an operator with the +/// indices of its inputs: +/// +/// - `op` ([`Operator`]) is the operator: a source like `ScanParquet` or a transform like +/// `Project`. +/// - `inputs` is a `Vec` naming the indices of the upstream nodes the operator reads from. +/// +/// A node depends on another when one of its `inputs` is that node's index. `nodes` is stored in +/// topological order: every node appears after the nodes it consumes (each input index is +/// strictly less than the node's own index), so an engine can evaluate `nodes` in slice order; +/// each node's inputs are guaranteed bound by the time the node is reached. +/// +/// A well-formed `Plan` has at least one node. The **terminal node** is always the last entry in +/// `nodes`: no other node lists its index in `inputs`, and its rows are the value the engine +/// streams to the caller. +/// +/// # Optimization +/// +/// For the best performance, connectors are encouraged to run kernel-produced +/// plans through their query optimizer before execution (e.g. to fold adjacent +/// filters, merge scans over the same files, or choose physical join and scan +/// strategies). +/// +/// # Example +/// +/// A five-node plan: two independent scans, each filtered, then unioned. The `nodes` +/// `Vec`: +/// +/// ```text +/// Plan { +/// nodes: vec![ +/// PlanNode { op: ScanParquet(..), inputs: vec![] }, // node 0 +/// PlanNode { op: ScanParquet(..), inputs: vec![] }, // node 1 +/// PlanNode { op: Filter(..), inputs: vec![0] }, // node 2 +/// PlanNode { op: Filter(..), inputs: vec![1] }, // node 3 +/// PlanNode { op: UnionAll(..), inputs: vec![2, 3] }, // node 4 +/// ], +/// } +/// ``` +/// +/// The dataflow DAG this encodes: +/// +/// ```text +/// ScanParquet [0] ScanParquet [1] +/// | | +/// v v +/// Filter [2] Filter [3] +/// | | +/// +---------+----------+ +/// v +/// UnionAll [4] <-- terminal (last node) +/// ``` +/// +/// The engine evaluates the nodes in the order of the `nodes` vector: nodes `0` and `1` first +/// (sources), then `2` and `3`, then `4`. The engine streams the rows produced at the terminal +/// node to the caller. +#[derive(Debug, Clone)] +pub struct Plan { + pub nodes: Vec, +} diff --git a/kernel/src/plans/mod.rs b/kernel/src/plans/mod.rs new file mode 100644 index 0000000000..6def2b7b31 --- /dev/null +++ b/kernel/src/plans/mod.rs @@ -0,0 +1,159 @@ +//! Declarative plans: the kernel describes data work as a relational plan; the engine executes it. +//! +//! Kernel does no I/O or data processing itself. When an operation needs data work, kernel builds a +//! plan and hands it to the engine's [`PlanExecutor`], which compiles it into the engine's own +//! representation (a Spark or DataFusion logical plan, an iterator pipeline) and runs it. The +//! engine therefore applies its own optimizer, parallelism, and async I/O to all of kernel's data +//! work, not just leaf scans. +//! +//! # What a plan is +//! +//! A [`Plan`](ir::plan::Plan) is a DAG of relational operators ([`Operator`](ir::nodes::Operator)): +//! sources, transforms, and set combinators. Most map one-to-one onto a SQL operator, so a plan +//! reads like a query. The live-add metadata plan built in `scan::scan_plan`, for example, is +//! roughly: +//! +//! ```sql +//! -- commits: keep the newest action per file, then keep only the live adds +//! SELECT add FROM ( +//! SELECT max_by(action, version) AS add FROM commits GROUP BY file_key +//! ) WHERE add IS NOT NULL +//! UNION ALL +//! -- checkpoint adds that no newer commit superseded +//! SELECT c.add FROM checkpoint c +//! LEFT ANTI JOIN commit_keys k ON c.file_key = k.file_key +//! ``` +//! +//! # Writing an executor +//! +//! An executor implements [`PlanExecutor::execute_op`], dispatching on the [`Operation`] it +//! receives and returning the matching [`PlanResult`] variant: +//! +//! - [`Operation::IoOperation`] is a single I/O request; each [`IoOperation`] variant documents the +//! [`PlanResult`] it must return. +//! - [`Operation::QueryPlan`] is a [`Plan`](ir::plan::Plan), returning [`PlanResult::Data`]. Either +//! evaluate [`Plan::nodes`](ir::plan::Plan::nodes) in slice order, which is topologically sorted +//! so a node's inputs are already evaluated, or compile the DAG into the engine's own plan. +//! +//! Every operator, expression, and predicate a plan contains must be handled; returning an error +//! for an unsupported one is fine, and kernel surfaces it to the caller. The sync engine's +//! `SyncPlanExecutor` is a complete reference implementation. +//! +//! # Where to look +//! +//! - [`PlanBuilder`] builds plans through a fluent, schema-validating API, each method documenting +//! its operator with a runnable example. +//! - [`ir::nodes`] is the operator catalog: each [`Operator`](ir::nodes::Operator) variant's +//! payload struct carries its semantics, invariants, and worked examples. +//! - [`crate::expressions`] defines the expressions and predicates operators evaluate, including +//! the type and null semantics an executor must match. +//! +//! This module is opt-in behind the `declarative-plans` feature flag. +mod builder; +pub mod ir; +pub mod proto; + +pub use builder::PlanBuilder; +use bytes::Bytes; +pub use ir::{IoOperation, Operation}; + +use crate::{ + AsAny, DeltaResult, DeltaResultIteratorStatic, EngineData, Error, FileMeta, ParquetFooter, +}; + +/// Provides the ability to execute declarative plans to the Delta Kernel. +/// +/// This gives the kernel the ability to execute data-intensive operations by constructing a +/// declarative, relational plan algebra, without prescribing *how* to do it. +pub trait PlanExecutor: AsAny { + /// Executes the given declarative plan and returns the result. + fn execute_op(&self, op: Operation) -> DeltaResult; + + /// Reads a parquet file's footer (schema and, in future, row-group stats) via a + /// [`IoOperation::ParquetFooter`] op. + fn read_parquet_footer(&self, file: FileMeta) -> DeltaResult { + self.execute_op(Operation::IoOperation(IoOperation::parquet_footer(file)))? + .into_parquet_footer() + } +} + +/// The result of executing an [`Operation`]. +/// +/// Each variant describes a different shape of output that a plan can possibly produce. +pub enum PlanResult { + /// A stream of columnar data batches (as [`EngineData`]) produced by the plan. + Data(DeltaResultIteratorStatic>), + /// A stream of file metadata entries. + FileMeta(DeltaResultIteratorStatic), + /// A stream of raw byte buffers. + Bytes(DeltaResultIteratorStatic), + /// Metadata extracted from a Parquet file footer. + ParquetFooter(ParquetFooter), + /// Represents the successful completion of a plan, but with no return value. + Unit, +} + +impl PlanResult { + /// Consumes the PlanResult and extracts the inner iterator of EngineData (assuming that it is a + /// PlanResult::Data variant). Returns an error if the PlanResult is not the expected variant. + pub fn into_data(self) -> DeltaResult>> { + match self { + Self::Data(iter) => Ok(iter), + other => Err(other.type_mismatch("Data")), + } + } + + /// Consumes the PlanResult and extracts the inner iterator of FileMeta (assuming that it is a + /// PlanResult::FileMeta variant). Returns an error if the PlanResult is not the expected + /// variant. + pub fn into_file_meta(self) -> DeltaResult> { + match self { + Self::FileMeta(iter) => Ok(iter), + other => Err(other.type_mismatch("FileMeta")), + } + } + + /// Consumes the PlanResult and extracts the inner iterator of Bytes (assuming that it is a + /// PlanResult::Bytes variant). Returns an error if the PlanResult is not a PlanResult::Bytes + /// variant. + pub fn into_bytes(self) -> DeltaResult> { + match self { + Self::Bytes(iter) => Ok(iter), + other => Err(other.type_mismatch("Bytes")), + } + } + + /// Consumes the PlanResult and extracts the inner [`ParquetFooter`] (assuming that it is a + /// PlanResult::ParquetFooter variant). Returns an error if the PlanResult is not the expected + /// variant. + pub fn into_parquet_footer(self) -> DeltaResult { + match self { + Self::ParquetFooter(footer) => Ok(footer), + other => Err(other.type_mismatch("ParquetFooter")), + } + } + + /// Consumes the PlanResult, verifying that it is a PlanResult::Unit variant. Returns an error + /// if the PlanResult is not the expected variant. + pub fn into_unit(self) -> DeltaResult<()> { + match self { + Self::Unit => Ok(()), + other => Err(other.type_mismatch("Unit")), + } + } + + fn variant_name(&self) -> &'static str { + match self { + Self::Data(_) => "Data", + Self::FileMeta(_) => "FileMeta", + Self::Bytes(_) => "Bytes", + Self::ParquetFooter(_) => "ParquetFooter", + Self::Unit => "Unit", + } + } + + /// Build an [`Error::PlanResultTypeMismatch`] reporting `self`'s variant as the actual one. + fn type_mismatch(&self, expected: &'static str) -> Error { + Error::plan_result_type_mismatch(expected, self.variant_name()) + } +} diff --git a/kernel/src/plans/proto/convert.rs b/kernel/src/plans/proto/convert.rs new file mode 100644 index 0000000000..26c14f61b3 --- /dev/null +++ b/kernel/src/plans/proto/convert.rs @@ -0,0 +1,2523 @@ +//! Conversions from the kernel plan IR into the prost-generated proto wire types. + +use super::plan::agg as proto_agg; +use super::schema::data_type::Kind as DataTypeKind; +use super::schema::metadata_value::Value as MetadataValueKind; +use super::schema::primitive_type::Kind as PrimitiveTypeKind; +#[cfg(feature = "geo-type-in-dev")] +use super::schema::EdgeInterpolationAlgorithm as EdgeAlgo; +use super::schema::SimplePrimitiveType as Simple; +use super::{ + expressions as proto_expr, operation as proto_op, plan as proto_plan, schema as proto_schema, +}; +use crate::expressions::{ + ArrayData, BinaryExpression, BinaryExpressionOp, BinaryPredicate, BinaryPredicateOp, + CastExpression, CastOptions, ColumnName, DecimalData, ElementAtExpression, Expression, + ExpressionFieldPatch, ExpressionStructPatch, JunctionPredicate, JunctionPredicateOp, MapData, + MapToStructExpression, MapToStructOptions, OpaqueExpression, OpaquePredicate, + ParseJsonExpression, Predicate, Scalar, StructData, UnaryExpression, UnaryExpressionOp, + UnaryPredicate, UnaryPredicateOp, VariadicExpression, VariadicExpressionOp, +}; +use crate::plans::ir::nodes::{ + Agg, Aggregate, DynamicScan, FileType, Filter, Operator, Project, ScanFile, ScanJson, + ScanParquet, SemiJoin, Values, +}; +use crate::plans::ir::plan::{Plan, PlanNode}; +use crate::plans::{IoOperation, Operation}; +use crate::schema::{ + ArrayType, DataType, DecimalType, MapType, MetadataValue, PrimitiveType, StructField, + StructType, +}; +#[cfg(feature = "geo-type-in-dev")] +use crate::schema::{EdgeInterpolationAlgorithm, GeographyType, GeometryType}; +use crate::{DeltaResult, Error, FileMeta, FileSlice}; + +// === Helpers === + +/// Converts each element of `items` into proto via [`From`]. +fn convert_vec<'a, T, U>(items: &'a [T]) -> Vec +where + U: From<&'a T>, +{ + items.iter().map(U::from).collect() +} + +/// Converts a slice of [`Expression`] references into proto. +fn convert_expr_vec(items: &[E]) -> Vec +where + E: AsRef, +{ + items.iter().map(|e| e.as_ref().into()).collect() +} + +// === Operation / IoOperation to Proto === + +impl From<&Operation> for proto_op::Operation { + fn from(op: &Operation) -> Self { + let op = match op { + Operation::IoOperation(io) => proto_op::operation::Op::Io(io.into()), + Operation::QueryPlan(plan) => proto_op::operation::Op::QueryPlan(plan.into()), + }; + proto_op::Operation { op: Some(op) } + } +} + +impl From<&IoOperation> for proto_op::IoOperation { + fn from(io: &IoOperation) -> Self { + use proto_op::io_operation::Op; + let op = match io { + IoOperation::FileListing { url } => Op::FileListing(proto_op::FileListing { + url: url.to_string(), + }), + IoOperation::ReadBytes { files } => Op::ReadBytes(proto_op::ReadBytes { + files: convert_vec(files), + }), + IoOperation::WriteBytes { + url, + data, + overwrite, + } => Op::WriteBytes(proto_op::WriteBytes { + url: url.to_string(), + data: data.to_vec(), + overwrite: *overwrite, + }), + IoOperation::HeadFile { url } => Op::HeadFile(proto_op::HeadFile { + url: url.to_string(), + }), + IoOperation::AtomicCopy { + source, + destination, + } => Op::AtomicCopy(proto_op::AtomicCopy { + source: source.to_string(), + destination: destination.to_string(), + }), + IoOperation::ParquetFooter { file } => Op::ParquetFooter(proto_op::ParquetFooter { + file: Some(file.into()), + }), + }; + proto_op::IoOperation { op: Some(op) } + } +} + +impl From<&FileSlice> for proto_op::FileSlice { + fn from(slice: &FileSlice) -> Self { + let (url, range) = slice; + proto_op::FileSlice { + url: url.to_string(), + range_start: range.as_ref().map(|r| r.start), + range_end: range.as_ref().map(|r| r.end), + } + } +} + +impl From<&FileMeta> for proto_plan::FileMeta { + fn from(meta: &FileMeta) -> Self { + proto_plan::FileMeta { + location: meta.location.to_string(), + size: meta.size, + last_modified: meta.last_modified, + } + } +} + +// === Plan / nodes to Proto === + +impl From<&Plan> for proto_plan::Plan { + fn from(plan: &Plan) -> Self { + proto_plan::Plan { + nodes: convert_vec(&plan.nodes), + } + } +} + +impl From<&PlanNode> for proto_plan::PlanNode { + fn from(node: &PlanNode) -> Self { + proto_plan::PlanNode { + op: Some((&node.op).into()), + inputs: node.inputs.iter().map(|&i| i as u32).collect(), + } + } +} + +impl From<&Operator> for proto_plan::Operator { + fn from(op: &Operator) -> Self { + use proto_plan::operator::Op; + let op = match op { + Operator::ScanParquet(n) => Op::ScanParquet(n.into()), + Operator::ScanJson(n) => Op::ScanJson(n.into()), + Operator::Values(n) => Op::Values(n.into()), + Operator::Project(n) => Op::Project(n.into()), + Operator::Filter(n) => Op::Filter(n.into()), + Operator::DynamicScan(n) => Op::DynamicScan(n.into()), + Operator::Aggregate(n) => Op::Aggregate(n.into()), + Operator::SemiJoin(n) => Op::SemiJoin(n.into()), + Operator::UnionAll(_) => Op::UnionAll(proto_plan::UnionAllNode {}), + }; + proto_plan::Operator { op: Some(op) } + } +} + +impl From<&ScanFile> for proto_plan::ScanFile { + fn from(file: &ScanFile) -> Self { + proto_plan::ScanFile { + meta: Some((&file.meta).into()), + file_constants: convert_vec(&file.file_constants), + } + } +} + +impl From<&ScanParquet> for proto_plan::ScanParquetNode { + fn from(node: &ScanParquet) -> Self { + proto_plan::ScanParquetNode { + files: convert_vec(&node.files), + file_constant_columns: node.file_constant_columns.clone(), + schema: Some(node.schema.as_ref().into()), + } + } +} + +impl From<&ScanJson> for proto_plan::ScanJsonNode { + fn from(node: &ScanJson) -> Self { + proto_plan::ScanJsonNode { + files: convert_vec(&node.files), + file_constant_columns: node.file_constant_columns.clone(), + schema: Some(node.schema.as_ref().into()), + } + } +} + +impl From<&Values> for proto_plan::ValuesNode { + fn from(node: &Values) -> Self { + let rows = node + .rows + .iter() + .map(|row| proto_plan::ValuesRow { + values: convert_vec(row), + }) + .collect(); + proto_plan::ValuesNode { + schema: Some(node.schema.as_ref().into()), + rows, + } + } +} + +impl From<&Project> for proto_plan::ProjectNode { + fn from(node: &Project) -> Self { + proto_plan::ProjectNode { + expr: Some(node.expr.as_ref().into()), + schema: Some(node.schema.as_ref().into()), + } + } +} + +impl From<&Filter> for proto_plan::FilterNode { + fn from(node: &Filter) -> Self { + proto_plan::FilterNode { + predicate: Some(node.predicate.as_ref().into()), + } + } +} + +impl From<&DynamicScan> for proto_plan::DynamicScanNode { + fn from(node: &DynamicScan) -> Self { + proto_plan::DynamicScanNode { + schema: Some(node.schema.as_ref().into()), + file_type: proto_plan::FileType::from(node.file_type) as i32, + base_url: node.base_url.to_string(), + file_constant_columns: node.file_constant_columns.clone(), + path_column: Some((&node.path_column).into()), + file_size_column: Some((&node.file_size_column).into()), + last_modified_column: Some((&node.last_modified_column).into()), + dv_column: Some((&node.dv_column).into()), + } + } +} + +impl From<&Aggregate> for proto_plan::AggregateNode { + fn from(node: &Aggregate) -> Self { + proto_plan::AggregateNode { + group_by: convert_vec(&node.group_by), + aggs: convert_vec(&node.aggs), + schema: Some(node.schema.as_ref().into()), + } + } +} + +impl From<&Agg> for proto_plan::Agg { + fn from(agg: &Agg) -> Self { + let func = match agg { + Agg::Min(value) => proto_agg::Func::Min(proto_plan::MinAgg { + value: Some(value.into()), + }), + Agg::Max(value) => proto_agg::Func::Max(proto_plan::MaxAgg { + value: Some(value.into()), + }), + Agg::Sum(value) => proto_agg::Func::Sum(proto_plan::SumAgg { + value: Some(value.into()), + }), + Agg::Count(value) => proto_agg::Func::Count(proto_plan::CountAgg { + value: Some(value.into()), + }), + Agg::CountStar => proto_agg::Func::CountStar(proto_plan::CountStarAgg {}), + Agg::MinNonNullBy(operands) => { + proto_agg::Func::MinNonNullBy(proto_plan::MinNonNullByAgg { + value: Some((&operands.value).into()), + null_sentinel: Some((&operands.null_sentinel).into()), + key: Some((&operands.key).into()), + }) + } + Agg::MaxNonNullBy(operands) => { + proto_agg::Func::MaxNonNullBy(proto_plan::MaxNonNullByAgg { + value: Some((&operands.value).into()), + null_sentinel: Some((&operands.null_sentinel).into()), + key: Some((&operands.key).into()), + }) + } + }; + proto_plan::Agg { func: Some(func) } + } +} + +impl From<&SemiJoin> for proto_plan::SemiJoinNode { + fn from(node: &SemiJoin) -> Self { + proto_plan::SemiJoinNode { + inverted: node.inverted, + probe_keys: convert_vec(&node.probe_keys), + build_keys: convert_vec(&node.build_keys), + } + } +} + +impl From for proto_plan::FileType { + fn from(file_type: FileType) -> Self { + match file_type { + FileType::Parquet => proto_plan::FileType::Parquet, + FileType::Json => proto_plan::FileType::Json, + } + } +} + +// === Expressions to Proto === + +impl From<&Expression> for proto_expr::Expression { + fn from(expr: &Expression) -> Self { + use proto_expr::expression::Kind; + let kind = match expr { + Expression::Literal(scalar) => Kind::Literal(scalar.into()), + Expression::Column(column) => Kind::Column(column.into()), + Expression::Predicate(pred) => Kind::Predicate(Box::new(pred.as_ref().into())), + Expression::Struct(exprs, nullability) => { + let nullability_predicate = + nullability.as_ref().map(|n| Box::new(n.as_ref().into())); + Kind::StructExpr(Box::new(proto_expr::StructExpression { + exprs: convert_expr_vec(exprs), + nullability_predicate, + })) + } + Expression::StructPatch(patch) => Kind::Transform(patch.into()), + Expression::Unary(unary) => Kind::Unary(Box::new(unary.into())), + Expression::Binary(binary) => Kind::Binary(Box::new(binary.into())), + Expression::Variadic(variadic) => Kind::Variadic(variadic.into()), + Expression::Opaque(opaque) => Kind::Opaque(opaque.into()), + Expression::Unknown(name) => Kind::Unknown(name.clone()), + Expression::ParseJson(parse_json) => Kind::ParseJson(Box::new(parse_json.into())), + Expression::MapToStruct(map_to_struct) => { + Kind::MapToStruct(Box::new(map_to_struct.into())) + } + Expression::ElementAt(element_at) => Kind::ElementAt(Box::new(element_at.into())), + Expression::Cast(cast) => Kind::Cast(Box::new(cast.into())), + }; + proto_expr::Expression { kind: Some(kind) } + } +} + +impl From<&Predicate> for proto_expr::Predicate { + fn from(pred: &Predicate) -> Self { + use proto_expr::predicate::Kind; + let kind = match pred { + Predicate::BooleanExpression(expr) => Kind::BooleanExpression(Box::new(expr.into())), + Predicate::Not(inner) => Kind::Not(Box::new(inner.as_ref().into())), + Predicate::Unary(unary) => Kind::Unary(Box::new(unary.into())), + Predicate::Binary(binary) => Kind::Binary(Box::new(binary.into())), + Predicate::Junction(junction) => Kind::Junction(junction.into()), + Predicate::Opaque(opaque) => Kind::Opaque(opaque.into()), + Predicate::Unknown(name) => Kind::Unknown(name.clone()), + }; + proto_expr::Predicate { kind: Some(kind) } + } +} + +impl From<&ColumnName> for proto_expr::ColumnName { + fn from(column: &ColumnName) -> Self { + proto_expr::ColumnName { + path: column.path().to_vec(), + } + } +} + +impl From<&UnaryExpression> for proto_expr::UnaryExpression { + fn from(unary: &UnaryExpression) -> Self { + proto_expr::UnaryExpression { + op: proto_expr::UnaryExpressionOp::from(unary.op) as i32, + expr: Some(Box::new(unary.expr.as_ref().into())), + } + } +} + +impl From<&BinaryExpression> for proto_expr::BinaryExpression { + fn from(binary: &BinaryExpression) -> Self { + proto_expr::BinaryExpression { + op: proto_expr::BinaryExpressionOp::from(binary.op) as i32, + left: Some(Box::new(binary.left.as_ref().into())), + right: Some(Box::new(binary.right.as_ref().into())), + } + } +} + +impl From<&VariadicExpression> for proto_expr::VariadicExpression { + fn from(variadic: &VariadicExpression) -> Self { + proto_expr::VariadicExpression { + op: proto_expr::VariadicExpressionOp::from(variadic.op) as i32, + exprs: convert_vec(&variadic.exprs), + } + } +} + +impl From<&OpaqueExpression> for proto_expr::OpaqueExpression { + fn from(opaque: &OpaqueExpression) -> Self { + proto_expr::OpaqueExpression { + name: opaque.op.name().to_string(), + exprs: convert_vec(&opaque.exprs), + } + } +} + +impl From<&ParseJsonExpression> for proto_expr::ParseJsonExpression { + fn from(parse_json: &ParseJsonExpression) -> Self { + proto_expr::ParseJsonExpression { + json_expr: Some(Box::new(parse_json.json_expr.as_ref().into())), + output_schema: Some(parse_json.output_schema.as_ref().into()), + } + } +} + +impl From<&MapToStructExpression> for proto_expr::MapToStructExpression { + fn from(map_to_struct: &MapToStructExpression) -> Self { + proto_expr::MapToStructExpression { + map_expr: Some(Box::new(map_to_struct.map_expr.as_ref().into())), + options: (map_to_struct.options != MapToStructOptions::default()) + .then(|| (&map_to_struct.options).into()), + } + } +} + +impl From<&MapToStructOptions> for proto_expr::MapToStructOptions { + fn from(options: &MapToStructOptions) -> Self { + Self { + timestamp_timezone: options.timestamp_timezone().map(ToOwned::to_owned), + } + } +} + +impl From<&ElementAtExpression> for proto_expr::ElementAtExpression { + fn from(element_at: &ElementAtExpression) -> Self { + Self { + map_expr: Some(Box::new(element_at.map_expr.as_ref().into())), + key_expr: Some(Box::new(element_at.key_expr.as_ref().into())), + } + } +} + +impl From<&CastExpression> for proto_expr::CastExpression { + fn from(cast: &CastExpression) -> Self { + Self { + expr: Some(Box::new(cast.expr.as_ref().into())), + target: Some((&cast.target).into()), + options: (cast.options != CastOptions::default()).then(|| (&cast.options).into()), + } + } +} + +impl From<&CastOptions> for proto_expr::CastOptions { + fn from(options: &CastOptions) -> Self { + Self { + timestamp_timezone: options.timestamp_timezone().map(ToOwned::to_owned), + } + } +} + +impl From<&UnaryPredicate> for proto_expr::UnaryPredicate { + fn from(unary: &UnaryPredicate) -> Self { + proto_expr::UnaryPredicate { + op: proto_expr::UnaryPredicateOp::from(unary.op) as i32, + expr: Some(Box::new(unary.expr.as_ref().into())), + } + } +} + +impl From<&BinaryPredicate> for proto_expr::BinaryPredicate { + fn from(binary: &BinaryPredicate) -> Self { + proto_expr::BinaryPredicate { + op: proto_expr::BinaryPredicateOp::from(binary.op) as i32, + left: Some(Box::new(binary.left.as_ref().into())), + right: Some(Box::new(binary.right.as_ref().into())), + } + } +} + +impl From<&JunctionPredicate> for proto_expr::JunctionPredicate { + fn from(junction: &JunctionPredicate) -> Self { + proto_expr::JunctionPredicate { + op: proto_expr::JunctionPredicateOp::from(junction.op) as i32, + preds: convert_vec(&junction.preds), + } + } +} + +impl From<&OpaquePredicate> for proto_expr::OpaquePredicate { + fn from(opaque: &OpaquePredicate) -> Self { + proto_expr::OpaquePredicate { + name: opaque.op.name().to_string(), + exprs: convert_vec(&opaque.exprs), + } + } +} + +// The proto `Transform` struct mirrors `ExpressionStructPatch`. +impl From<&ExpressionStructPatch> for proto_expr::Transform { + fn from(patch: &ExpressionStructPatch) -> Self { + let field_transforms = patch + .field_patches + .iter() + .map(|(name, field_patch)| (name.clone(), field_patch.into())) + .collect(); + proto_expr::Transform { + input_path: patch.input_path.as_ref().map(Into::into), + field_transforms, + prepended_fields: convert_expr_vec(&patch.prepended_fields), + appended_fields: convert_expr_vec(&patch.appended_fields), + } + } +} + +// The proto `FieldTransform` struct mirrors `ExpressionFieldPatch`. +impl From<&ExpressionFieldPatch> for proto_expr::FieldTransform { + fn from(field_patch: &ExpressionFieldPatch) -> Self { + proto_expr::FieldTransform { + exprs: convert_expr_vec(&field_patch.insertions), + is_replace: !field_patch.keep_input, + optional: field_patch.optional, + } + } +} + +impl From for proto_expr::UnaryExpressionOp { + fn from(op: UnaryExpressionOp) -> Self { + match op { + UnaryExpressionOp::ToJson => proto_expr::UnaryExpressionOp::ToJson, + } + } +} + +impl From for proto_expr::BinaryExpressionOp { + fn from(op: BinaryExpressionOp) -> Self { + match op { + BinaryExpressionOp::Plus => proto_expr::BinaryExpressionOp::Plus, + BinaryExpressionOp::Minus => proto_expr::BinaryExpressionOp::Minus, + BinaryExpressionOp::Multiply => proto_expr::BinaryExpressionOp::Multiply, + BinaryExpressionOp::Divide => proto_expr::BinaryExpressionOp::Divide, + } + } +} + +impl From for proto_expr::VariadicExpressionOp { + fn from(op: VariadicExpressionOp) -> Self { + match op { + VariadicExpressionOp::Coalesce => proto_expr::VariadicExpressionOp::Coalesce, + VariadicExpressionOp::Array => proto_expr::VariadicExpressionOp::Array, + } + } +} + +impl From for proto_expr::UnaryPredicateOp { + fn from(op: UnaryPredicateOp) -> Self { + match op { + UnaryPredicateOp::IsNull => proto_expr::UnaryPredicateOp::IsNull, + } + } +} + +impl From for proto_expr::BinaryPredicateOp { + fn from(op: BinaryPredicateOp) -> Self { + match op { + BinaryPredicateOp::LessThan => proto_expr::BinaryPredicateOp::LessThan, + BinaryPredicateOp::GreaterThan => proto_expr::BinaryPredicateOp::GreaterThan, + BinaryPredicateOp::Equal => proto_expr::BinaryPredicateOp::Equal, + BinaryPredicateOp::Distinct => proto_expr::BinaryPredicateOp::Distinct, + BinaryPredicateOp::In => proto_expr::BinaryPredicateOp::In, + } + } +} + +impl From for proto_expr::JunctionPredicateOp { + fn from(op: JunctionPredicateOp) -> Self { + match op { + JunctionPredicateOp::And => proto_expr::JunctionPredicateOp::And, + JunctionPredicateOp::Or => proto_expr::JunctionPredicateOp::Or, + } + } +} + +// === Scalars to Proto === + +impl From<&Scalar> for proto_expr::Scalar { + fn from(scalar: &Scalar) -> Self { + use proto_expr::scalar::Value; + let value = match scalar { + Scalar::Integer(v) => Value::Integer(*v), + Scalar::Long(v) => Value::Long(*v), + Scalar::Short(v) => Value::Short(*v as i32), + Scalar::Byte(v) => Value::Byte(*v as i32), + Scalar::Float(v) => Value::Float(*v), + Scalar::Double(v) => Value::Double(*v), + Scalar::String(v) => Value::String(v.clone()), + Scalar::Boolean(v) => Value::Boolean(*v), + Scalar::Timestamp(v) => Value::Timestamp(*v), + Scalar::TimestampNtz(v) => Value::TimestampNtz(*v), + Scalar::IntervalYearMonth(v) => Value::IntervalYearMonth(*v), + Scalar::IntervalDayTime(v) => Value::IntervalDayTime(*v), + Scalar::Date(v) => Value::Date(*v), + Scalar::Binary(v) => Value::Binary(v.clone()), + Scalar::Decimal(decimal) => Value::Decimal(decimal.into()), + Scalar::Null(data_type) => Value::Null(data_type.into()), + Scalar::Struct(struct_data) => Value::Struct(struct_data.into()), + Scalar::Array(array_data) => Value::Array(array_data.into()), + Scalar::Map(map_data) => Value::Map(map_data.into()), + }; + proto_expr::Scalar { value: Some(value) } + } +} + +impl From<&DecimalData> for proto_expr::DecimalData { + fn from(decimal: &DecimalData) -> Self { + proto_expr::DecimalData { + bits: decimal.bits().to_be_bytes().to_vec(), + decimal_type: Some((*decimal.ty()).into()), + } + } +} + +impl From<&StructData> for proto_expr::StructData { + fn from(struct_data: &StructData) -> Self { + proto_expr::StructData { + fields: convert_vec(struct_data.fields()), + values: convert_vec(struct_data.values()), + } + } +} + +impl From<&ArrayData> for proto_expr::ArrayData { + fn from(array_data: &ArrayData) -> Self { + proto_expr::ArrayData { + array_type: Some(array_data.array_type().into()), + elements: convert_vec(array_data.array_elements()), + } + } +} + +impl From<&MapData> for proto_expr::MapData { + fn from(map_data: &MapData) -> Self { + let pairs = map_data + .pairs() + .iter() + .map(|(key, value)| proto_expr::MapEntry { + key: Some(key.into()), + value: Some(value.into()), + }) + .collect(); + proto_expr::MapData { + map_type: Some(map_data.map_type().into()), + pairs, + } + } +} + +// === Schema to Proto === + +impl From<&DataType> for proto_schema::DataType { + fn from(data_type: &DataType) -> Self { + let kind = match data_type { + DataType::Primitive(primitive) => DataTypeKind::Primitive(primitive.into()), + DataType::Array(array) => DataTypeKind::Array(Box::new(array.as_ref().into())), + DataType::Struct(struct_type) => DataTypeKind::Struct(struct_type.as_ref().into()), + DataType::Map(map) => DataTypeKind::Map(Box::new(map.as_ref().into())), + // The proto `VariantType` is intentionally empty: variants are opaque on the wire. + DataType::Variant(_) => DataTypeKind::Variant(proto_schema::VariantType {}), + }; + proto_schema::DataType { kind: Some(kind) } + } +} + +impl From<&PrimitiveType> for proto_schema::PrimitiveType { + fn from(primitive: &PrimitiveType) -> Self { + let kind = match primitive { + PrimitiveType::String => PrimitiveTypeKind::Simple(Simple::String as i32), + PrimitiveType::Long => PrimitiveTypeKind::Simple(Simple::Long as i32), + PrimitiveType::Integer => PrimitiveTypeKind::Simple(Simple::Integer as i32), + PrimitiveType::Short => PrimitiveTypeKind::Simple(Simple::Short as i32), + PrimitiveType::Byte => PrimitiveTypeKind::Simple(Simple::Byte as i32), + PrimitiveType::Float => PrimitiveTypeKind::Simple(Simple::Float as i32), + PrimitiveType::Double => PrimitiveTypeKind::Simple(Simple::Double as i32), + PrimitiveType::Boolean => PrimitiveTypeKind::Simple(Simple::Boolean as i32), + PrimitiveType::Binary => PrimitiveTypeKind::Simple(Simple::Binary as i32), + PrimitiveType::Date => PrimitiveTypeKind::Simple(Simple::Date as i32), + PrimitiveType::Timestamp => PrimitiveTypeKind::Simple(Simple::Timestamp as i32), + PrimitiveType::TimestampNtz => PrimitiveTypeKind::Simple(Simple::TimestampNtz as i32), + PrimitiveType::Decimal(decimal) => PrimitiveTypeKind::Decimal((*decimal).into()), + #[cfg(feature = "geo-type-in-dev")] + PrimitiveType::Geometry(geometry) => { + PrimitiveTypeKind::Geometry(geometry.as_ref().into()) + } + #[cfg(feature = "geo-type-in-dev")] + PrimitiveType::Geography(geography) => { + PrimitiveTypeKind::Geography(geography.as_ref().into()) + } + PrimitiveType::Void => PrimitiveTypeKind::Simple(Simple::Void as i32), + PrimitiveType::IntervalYearMonth => { + PrimitiveTypeKind::Simple(Simple::IntervalYearMonth as i32) + } + PrimitiveType::IntervalDayTime => { + PrimitiveTypeKind::Simple(Simple::IntervalDayTime as i32) + } + }; + proto_schema::PrimitiveType { kind: Some(kind) } + } +} + +impl From for proto_schema::DecimalType { + fn from(decimal: DecimalType) -> Self { + proto_schema::DecimalType { + precision: u32::from(decimal.precision()), + scale: u32::from(decimal.scale()), + } + } +} + +#[cfg(feature = "geo-type-in-dev")] +impl From<&GeometryType> for proto_schema::GeometryType { + fn from(geometry: &GeometryType) -> Self { + proto_schema::GeometryType { + crs: geometry.crs().to_string(), + } + } +} + +#[cfg(feature = "geo-type-in-dev")] +impl From<&GeographyType> for proto_schema::GeographyType { + fn from(geography: &GeographyType) -> Self { + proto_schema::GeographyType { + crs: geography.crs().to_string(), + algorithm: EdgeAlgo::from(geography.algorithm()) as i32, + } + } +} + +#[cfg(feature = "geo-type-in-dev")] +impl From<&EdgeInterpolationAlgorithm> for EdgeAlgo { + fn from(algorithm: &EdgeInterpolationAlgorithm) -> Self { + match algorithm { + EdgeInterpolationAlgorithm::Spherical => EdgeAlgo::Spherical, + EdgeInterpolationAlgorithm::Vincenty => EdgeAlgo::Vincenty, + EdgeInterpolationAlgorithm::Thomas => EdgeAlgo::Thomas, + EdgeInterpolationAlgorithm::Andoyer => EdgeAlgo::Andoyer, + EdgeInterpolationAlgorithm::Karney => EdgeAlgo::Karney, + } + } +} + +impl From<&ArrayType> for proto_schema::ArrayType { + fn from(array: &ArrayType) -> Self { + proto_schema::ArrayType { + element_type: Some(Box::new(array.element_type().into())), + contains_null: array.contains_null(), + } + } +} + +impl From<&MapType> for proto_schema::MapType { + fn from(map: &MapType) -> Self { + proto_schema::MapType { + key_type: Some(Box::new(map.key_type().into())), + value_type: Some(Box::new(map.value_type().into())), + value_contains_null: map.value_contains_null(), + } + } +} + +impl From<&StructType> for proto_schema::StructType { + fn from(struct_type: &StructType) -> Self { + proto_schema::StructType { + fields: struct_type.fields().map(Into::into).collect(), + } + } +} + +impl From<&StructField> for proto_schema::StructField { + fn from(field: &StructField) -> Self { + let metadata = field + .metadata + .iter() + .map(|(key, value)| (key.clone(), value.into())) + .collect(); + proto_schema::StructField { + name: field.name.clone(), + data_type: Some((&field.data_type).into()), + nullable: field.nullable, + metadata, + } + } +} + +impl From<&MetadataValue> for proto_schema::MetadataValue { + fn from(metadata: &MetadataValue) -> Self { + let value = match metadata { + MetadataValue::Number(n) => MetadataValueKind::Number(*n), + MetadataValue::String(s) => MetadataValueKind::String(s.clone()), + MetadataValue::Boolean(b) => MetadataValueKind::Boolean(*b), + MetadataValue::Other(json) => MetadataValueKind::OtherJson(json.to_string()), + }; + proto_schema::MetadataValue { value: Some(value) } + } +} + +// === Schema from Proto === + +impl TryFrom for StructType { + type Error = Error; + fn try_from(proto: proto_schema::StructType) -> DeltaResult { + let fields = proto + .fields + .into_iter() + .map(StructField::try_from) + .collect::>>()?; + StructType::try_new(fields) + } +} + +impl TryFrom for StructField { + type Error = Error; + + fn try_from(proto: proto_schema::StructField) -> DeltaResult { + let data_type = proto + .data_type + .ok_or_else(|| Error::schema("StructField proto missing data_type"))?; + let metadata = proto + .metadata + .into_iter() + .map(|(key, value)| Ok::<_, Error>((key, MetadataValue::try_from(value)?))) + .collect::>>()?; + Ok(StructField { + name: proto.name, + data_type: DataType::try_from(data_type)?, + nullable: proto.nullable, + metadata, + }) + } +} + +impl TryFrom for DataType { + type Error = Error; + fn try_from(proto: proto_schema::DataType) -> DeltaResult { + let kind = proto + .kind + .ok_or_else(|| Error::schema("DataType proto missing kind"))?; + let data_type = match kind { + DataTypeKind::Primitive(primitive) => DataType::Primitive(primitive.try_into()?), + DataTypeKind::Array(array) => DataType::Array(Box::new((*array).try_into()?)), + DataTypeKind::Struct(struct_type) => { + DataType::Struct(Box::new(struct_type.try_into()?)) + } + DataTypeKind::Map(map) => DataType::Map(Box::new((*map).try_into()?)), + // Kernel does not support shredded variants, so always decode as unshredded. + DataTypeKind::Variant(_) => DataType::unshredded_variant(), + }; + Ok(data_type) + } +} + +impl TryFrom for PrimitiveType { + type Error = Error; + fn try_from(proto: proto_schema::PrimitiveType) -> DeltaResult { + let kind = proto + .kind + .ok_or_else(|| Error::schema("PrimitiveType proto missing kind"))?; + let primitive = match kind { + PrimitiveTypeKind::Simple(simple) => { + let simple = Simple::try_from(simple).map_err(|_| { + Error::schema(format!("unknown SimplePrimitiveType value: {simple}")) + })?; + match simple { + Simple::String => PrimitiveType::String, + Simple::Long => PrimitiveType::Long, + Simple::Integer => PrimitiveType::Integer, + Simple::Short => PrimitiveType::Short, + Simple::Byte => PrimitiveType::Byte, + Simple::Float => PrimitiveType::Float, + Simple::Double => PrimitiveType::Double, + Simple::Boolean => PrimitiveType::Boolean, + Simple::Binary => PrimitiveType::Binary, + Simple::Date => PrimitiveType::Date, + Simple::Timestamp => PrimitiveType::Timestamp, + Simple::TimestampNtz => PrimitiveType::TimestampNtz, + Simple::Void => PrimitiveType::Void, + Simple::IntervalYearMonth => PrimitiveType::IntervalYearMonth, + Simple::IntervalDayTime => PrimitiveType::IntervalDayTime, + Simple::Unspecified => { + return Err(Error::schema("SimplePrimitiveType is unspecified")) + } + } + } + PrimitiveTypeKind::Decimal(decimal) => PrimitiveType::Decimal(decimal.try_into()?), + #[cfg(feature = "geo-type-in-dev")] + PrimitiveTypeKind::Geometry(geometry) => { + PrimitiveType::Geometry(Box::new(geometry.try_into()?)) + } + #[cfg(feature = "geo-type-in-dev")] + PrimitiveTypeKind::Geography(geography) => { + PrimitiveType::Geography(Box::new(geography.try_into()?)) + } + // The proto oneof always carries the geo variants, but kernel only knows how to decode + // them when the geo feature is enabled. + #[cfg(not(feature = "geo-type-in-dev"))] + PrimitiveTypeKind::Geometry(_) | PrimitiveTypeKind::Geography(_) => { + return Err(Error::schema( + "geometry/geography types require the 'geo-type-in-dev' feature", + )) + } + }; + Ok(primitive) + } +} + +impl TryFrom for DecimalType { + type Error = Error; + fn try_from(proto: proto_schema::DecimalType) -> DeltaResult { + let precision = u8::try_from(proto.precision).map_err(|_| { + Error::invalid_decimal(format!("precision out of range: {}", proto.precision)) + })?; + let scale = u8::try_from(proto.scale) + .map_err(|_| Error::invalid_decimal(format!("scale out of range: {}", proto.scale)))?; + DecimalType::try_new(precision, scale) + } +} + +#[cfg(feature = "geo-type-in-dev")] +impl TryFrom for GeometryType { + type Error = Error; + fn try_from(proto: proto_schema::GeometryType) -> DeltaResult { + GeometryType::try_new(&proto.crs) + } +} + +#[cfg(feature = "geo-type-in-dev")] +impl TryFrom for GeographyType { + type Error = Error; + fn try_from(proto: proto_schema::GeographyType) -> DeltaResult { + let algorithm = EdgeAlgo::try_from(proto.algorithm) + .map_err(|_| { + Error::invalid_geo_params(format!( + "unknown EdgeInterpolationAlgorithm value: {}", + proto.algorithm + )) + })? + .try_into()?; + GeographyType::try_new(&proto.crs, algorithm) + } +} + +#[cfg(feature = "geo-type-in-dev")] +impl TryFrom for EdgeInterpolationAlgorithm { + type Error = Error; + fn try_from(proto: EdgeAlgo) -> DeltaResult { + let algorithm = match proto { + EdgeAlgo::Spherical => EdgeInterpolationAlgorithm::Spherical, + EdgeAlgo::Vincenty => EdgeInterpolationAlgorithm::Vincenty, + EdgeAlgo::Thomas => EdgeInterpolationAlgorithm::Thomas, + EdgeAlgo::Andoyer => EdgeInterpolationAlgorithm::Andoyer, + EdgeAlgo::Karney => EdgeInterpolationAlgorithm::Karney, + EdgeAlgo::Unspecified => { + return Err(Error::invalid_geo_params( + "EdgeInterpolationAlgorithm is unspecified", + )) + } + }; + Ok(algorithm) + } +} + +impl TryFrom for ArrayType { + type Error = Error; + fn try_from(proto: proto_schema::ArrayType) -> DeltaResult { + let element_type = proto + .element_type + .ok_or_else(|| Error::schema("ArrayType proto missing element_type"))?; + Ok(ArrayType::new( + DataType::try_from(*element_type)?, + proto.contains_null, + )) + } +} + +impl TryFrom for MapType { + type Error = Error; + fn try_from(proto: proto_schema::MapType) -> DeltaResult { + let key_type = proto + .key_type + .ok_or_else(|| Error::schema("MapType proto missing key_type"))?; + let value_type = proto + .value_type + .ok_or_else(|| Error::schema("MapType proto missing value_type"))?; + Ok(MapType::new( + DataType::try_from(*key_type)?, + DataType::try_from(*value_type)?, + proto.value_contains_null, + )) + } +} + +impl TryFrom for MetadataValue { + type Error = Error; + fn try_from(proto: proto_schema::MetadataValue) -> DeltaResult { + let value = proto + .value + .ok_or_else(|| Error::schema("MetadataValue proto missing value"))?; + let metadata = match value { + MetadataValueKind::Number(n) => MetadataValue::Number(n), + MetadataValueKind::String(s) => MetadataValue::String(s), + MetadataValueKind::Boolean(b) => MetadataValue::Boolean(b), + MetadataValueKind::OtherJson(json) => { + MetadataValue::Other(serde_json::from_str(&json)?) + } + }; + Ok(metadata) + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use bytes::Bytes; + use rstest::rstest; + use url::Url; + + #[cfg(feature = "geo-type-in-dev")] + use super::EdgeAlgo; + use crate::actions::deletion_vector::DeletionVectorDescriptor; + use crate::expressions::{ + col, column_name, lit, ArrayData, BinaryExpressionOp, BinaryPredicateOp, CastOptions, + DecimalData, Expression, ExpressionStructPatchBuilder, JunctionPredicateOp, MapData, + MapToStructOptions, OpaqueExpressionOp, OpaquePredicateOp, Predicate, Scalar, + ScalarExpressionEvaluator, StructData, UnaryExpressionOp, UnaryPredicateOp, + VariadicExpressionOp, + }; + use crate::kernel_predicates::{ + DirectDataSkippingPredicateEvaluator, DirectPredicateEvaluator, + IndirectDataSkippingPredicateEvaluator, + }; + use crate::plans::ir::nodes::{ + Agg, Aggregate, DynamicScan, FileType, Filter, Operator, Project, ScanFile, ScanJson, + ScanParquet, SemiJoin, UnionAll, Values, + }; + use crate::plans::ir::plan::{Plan, PlanNode}; + use crate::plans::proto::{ + expressions as proto_expr, operation as proto_op, plan as proto_plan, + schema as proto_schema, + }; + use crate::plans::{IoOperation, Operation}; + use crate::schema::{ + ArrayType, DataType, DecimalType, MapType, MetadataValue, PrimitiveType, SchemaRef, + StructField, StructType, ToSchema as _, + }; + #[cfg(feature = "geo-type-in-dev")] + use crate::schema::{EdgeInterpolationAlgorithm, GeographyType, GeometryType}; + use crate::{DeltaResult, FileMeta, FileSlice}; + + // === Test helpers === + + #[derive(Debug, PartialEq)] + struct TestOpaqueExprOp; + + impl OpaqueExpressionOp for TestOpaqueExprOp { + fn name(&self) -> &str { + "test_opaque_expr" + } + fn eval_expr_scalar( + &self, + _eval_expr: &ScalarExpressionEvaluator<'_>, + _exprs: &[Expression], + ) -> DeltaResult { + Ok(Scalar::Integer(0)) + } + } + + #[derive(Debug, PartialEq)] + struct TestOpaquePredOp; + + impl OpaquePredicateOp for TestOpaquePredOp { + fn name(&self) -> &str { + "test_opaque_pred" + } + fn eval_pred_scalar( + &self, + _eval_expr: &ScalarExpressionEvaluator<'_>, + _eval_pred: &DirectPredicateEvaluator<'_>, + _exprs: &[Expression], + _inverted: bool, + ) -> DeltaResult> { + Ok(Some(true)) + } + fn eval_as_data_skipping_predicate( + &self, + _evaluator: &DirectDataSkippingPredicateEvaluator<'_>, + _exprs: &[Expression], + _inverted: bool, + ) -> Option { + None + } + fn as_data_skipping_predicate( + &self, + _evaluator: &IndirectDataSkippingPredicateEvaluator<'_>, + _exprs: &[Expression], + _inverted: bool, + ) -> Option { + None + } + } + + fn sample_file_meta() -> FileMeta { + FileMeta { + location: Url::parse("memory:///table/part-0.parquet").unwrap(), + last_modified: 123, + size: 456, + } + } + + fn sample_schema() -> SchemaRef { + Arc::new(StructType::try_new(vec![StructField::nullable("id", DataType::INTEGER)]).unwrap()) + } + + fn decode(op: &Operation) -> proto_op::Operation { + let bytes = op.to_proto_bytes(); + prost::Message::decode(bytes.as_slice()).expect("decode succeeds") + } + + fn io_op(op: &Operation) -> proto_op::io_operation::Op { + let Some(proto_op::operation::Op::Io(io)) = decode(op).op else { + panic!("expected an IoOperation"); + }; + io.op.expect("io op present") + } + + fn expr_kind_of(expr: Expression) -> proto_expr::expression::Kind { + proto_expr::Expression::from(&expr) + .kind + .expect("expression kind present") + } + + fn pred_kind_of(pred: Predicate) -> proto_expr::predicate::Kind { + proto_expr::Predicate::from(&pred) + .kind + .expect("predicate kind present") + } + + fn scalar_value_of(scalar: Scalar) -> proto_expr::scalar::Value { + proto_expr::Scalar::from(&scalar) + .value + .expect("scalar value present") + } + + // === Operation / IoOperation === + + #[rstest] + #[case( + Operation::IoOperation(IoOperation::head_file(Url::parse("memory:///h").unwrap())), + "io" + )] + #[case(Operation::QueryPlan(Plan { nodes: vec![] }), "query_plan")] + fn from_operation(#[case] op: Operation, #[case] expected: &str) { + use proto_op::operation::Op; + let kind = match decode(&op).op.unwrap() { + Op::Io(_) => "io", + Op::QueryPlan(_) => "query_plan", + }; + assert_eq!(kind, expected); + } + + #[test] + fn from_io_operation_file_listing() { + let url = "memory:///table/"; + let op = Operation::IoOperation(IoOperation::file_listing(Url::parse(url).unwrap())); + let proto_op::io_operation::Op::FileListing(file_listing) = io_op(&op) else { + panic!("expected FileListing"); + }; + assert_eq!(file_listing.url, url); + } + + #[test] + fn from_io_operation_read_bytes() { + let files = vec![ + (Url::parse("memory:///a").unwrap(), Some(0u64..10u64)), + (Url::parse("memory:///b").unwrap(), None), + ]; + let op = Operation::IoOperation(IoOperation::read_bytes(files)); + let proto_op::io_operation::Op::ReadBytes(read_bytes) = io_op(&op) else { + panic!("expected ReadBytes"); + }; + assert_eq!(read_bytes.files.len(), 2); + assert_eq!(read_bytes.files[0].url, "memory:///a"); + } + + #[test] + fn from_io_operation_write_bytes() { + let op = Operation::IoOperation(IoOperation::write_bytes( + Url::parse("memory:///out").unwrap(), + Bytes::from_static(b"hello"), + true, + )); + let proto_op::io_operation::Op::WriteBytes(write_bytes) = io_op(&op) else { + panic!("expected WriteBytes"); + }; + assert_eq!(write_bytes.url, "memory:///out"); + assert_eq!(write_bytes.data, b"hello"); + assert!(write_bytes.overwrite); + } + + #[test] + fn from_io_operation_head_file() { + let op = Operation::IoOperation(IoOperation::head_file( + Url::parse("memory:///head").unwrap(), + )); + let proto_op::io_operation::Op::HeadFile(head_file) = io_op(&op) else { + panic!("expected HeadFile"); + }; + assert_eq!(head_file.url, "memory:///head"); + } + + #[test] + fn from_io_operation_atomic_copy() { + let op = Operation::IoOperation(IoOperation::atomic_copy( + Url::parse("memory:///src").unwrap(), + Url::parse("memory:///dst").unwrap(), + )); + let proto_op::io_operation::Op::AtomicCopy(atomic_copy) = io_op(&op) else { + panic!("expected AtomicCopy"); + }; + assert_eq!(atomic_copy.source, "memory:///src"); + assert_eq!(atomic_copy.destination, "memory:///dst"); + } + + #[test] + fn from_io_operation_parquet_footer() { + let op = Operation::IoOperation(IoOperation::parquet_footer(sample_file_meta())); + let proto_op::io_operation::Op::ParquetFooter(parquet_footer) = io_op(&op) else { + panic!("expected ParquetFooter"); + }; + let file = parquet_footer.file.expect("file meta present"); + assert_eq!(file.location, "memory:///table/part-0.parquet"); + assert_eq!(file.size, 456); + assert_eq!(file.last_modified, 123); + } + + #[rstest] + #[case(Some(0u64..10u64), Some(0), Some(10))] + #[case(None, None, None)] + fn from_file_slice( + #[case] range: Option>, + #[case] expected_start: Option, + #[case] expected_end: Option, + ) { + let slice: FileSlice = (Url::parse("memory:///a").unwrap(), range); + let proto = proto_op::FileSlice::from(&slice); + assert_eq!(proto.url, "memory:///a"); + assert_eq!(proto.range_start, expected_start); + assert_eq!(proto.range_end, expected_end); + } + + #[test] + fn from_file_meta() { + let proto = proto_plan::FileMeta::from(&sample_file_meta()); + assert_eq!(proto.location, "memory:///table/part-0.parquet"); + assert_eq!(proto.size, 456); + assert_eq!(proto.last_modified, 123); + } + + // === Plan / nodes === + + #[test] + fn from_plan() { + let schema = Arc::new( + StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::not_null("name", DataType::STRING), + ]) + .unwrap(), + ); + let plan = Plan { + nodes: vec![ + PlanNode { + op: Operator::ScanParquet(ScanParquet { + files: vec![ScanFile::new(sample_file_meta())], + file_constant_columns: vec![], + schema: schema.clone(), + }), + inputs: vec![], + }, + PlanNode { + op: Operator::Filter(Filter { + predicate: Arc::new(Predicate::gt(col!("id"), lit(5i32))), + }), + inputs: vec![0], + }, + ], + }; + + let Some(proto_op::operation::Op::QueryPlan(plan)) = decode(&Operation::QueryPlan(plan)).op + else { + panic!("expected a QueryPlan"); + }; + assert_eq!(plan.nodes.len(), 2); + + // Source node: ScanParquet at index 0, no inputs. + let source = &plan.nodes[0]; + assert!(source.inputs.is_empty()); + let Some(proto_plan::operator::Op::ScanParquet(scan)) = &source.op.as_ref().unwrap().op + else { + panic!("expected ScanParquet"); + }; + assert_eq!(scan.files.len(), 1); + assert_eq!(scan.schema.as_ref().unwrap().fields.len(), 2); + + // Filter node at index 1: consumes node 0, carries a `id > 5` binary predicate. + let filter_node = &plan.nodes[1]; + assert_eq!(filter_node.inputs.len(), 1); + assert_eq!(filter_node.inputs[0], 0); + let Some(proto_plan::operator::Op::Filter(filter)) = &filter_node.op.as_ref().unwrap().op + else { + panic!("expected Filter"); + }; + let predicate = filter.predicate.as_ref().unwrap(); + let Some(proto_expr::predicate::Kind::Binary(binary)) = &predicate.kind else { + panic!("expected a binary predicate"); + }; + assert_eq!(binary.op, proto_expr::BinaryPredicateOp::GreaterThan as i32); + assert!(matches!( + binary.left.as_ref().unwrap().kind, + Some(proto_expr::expression::Kind::Column(_)) + )); + } + + #[rstest] + #[case( + Operator::ScanParquet(ScanParquet { + files: vec![], + file_constant_columns: vec![], + schema: sample_schema(), + }), + "scan_parquet" + )] + #[case( + Operator::ScanJson(ScanJson { + files: vec![], + file_constant_columns: vec![], + schema: sample_schema(), + }), + "scan_json" + )] + #[case(Operator::Values(Values { schema: sample_schema(), rows: vec![] }), "values")] + #[case( + Operator::Project(Project { + expr: Arc::new(Expression::struct_from([lit(1)])), + schema: sample_schema(), + }), + "project" + )] + #[case(Operator::Filter(Filter { predicate: Arc::new(Predicate::TRUE) }), "filter")] + #[case( + Operator::DynamicScan(DynamicScan { + schema: sample_schema(), + file_type: FileType::Parquet, + base_url: Url::parse("memory:///").unwrap(), + file_constant_columns: vec![], + path_column: column_name!("path"), + file_size_column: column_name!("size"), + last_modified_column: column_name!("filemod"), + dv_column: column_name!("dv"), + }), + "dynamic_scan" + )] + #[case( + Operator::Aggregate(Aggregate { + group_by: vec![], + aggs: vec![], + schema: sample_schema(), + }), + "aggregate" + )] + #[case( + Operator::SemiJoin(SemiJoin { inverted: false, probe_keys: vec![], build_keys: vec![] }), + "semi_join" + )] + #[case(Operator::UnionAll(UnionAll), "union_all")] + fn from_operator(#[case] op: Operator, #[case] expected: &str) { + use proto_plan::operator::Op; + let kind = match proto_plan::Operator::from(&op).op.unwrap() { + Op::ScanParquet(_) => "scan_parquet", + Op::ScanJson(_) => "scan_json", + Op::Values(_) => "values", + Op::Project(_) => "project", + Op::Filter(_) => "filter", + Op::DynamicScan(_) => "dynamic_scan", + Op::Aggregate(_) => "aggregate", + Op::SemiJoin(_) => "semi_join", + Op::UnionAll(_) => "union_all", + }; + assert_eq!(kind, expected); + } + + #[test] + fn from_scan_file() { + let scan_file = ScanFile { + meta: sample_file_meta(), + file_constants: vec![Scalar::Integer(1), Scalar::String("x".into())], + }; + let proto = proto_plan::ScanFile::from(&scan_file); + assert!(proto.meta.is_some()); + assert_eq!(proto.file_constants.len(), 2); + } + + #[test] + fn from_scan_parquet() { + let node = ScanParquet { + files: vec![ScanFile::new(sample_file_meta())], + file_constant_columns: vec!["c".to_string()], + schema: sample_schema(), + }; + let proto = proto_plan::ScanParquetNode::from(&node); + assert_eq!(proto.files.len(), 1); + assert_eq!(proto.file_constant_columns.len(), 1); + assert!(proto.schema.is_some()); + } + + #[test] + fn from_scan_json() { + let node = ScanJson { + files: vec![ScanFile::new(sample_file_meta())], + file_constant_columns: vec!["c".to_string()], + schema: sample_schema(), + }; + let proto = proto_plan::ScanJsonNode::from(&node); + assert_eq!(proto.files.len(), 1); + assert_eq!(proto.file_constant_columns.len(), 1); + assert!(proto.schema.is_some()); + } + + #[test] + fn from_values() { + let node = Values { + schema: sample_schema(), + rows: vec![vec![Scalar::Integer(1)], vec![Scalar::Integer(2)]], + }; + let proto = proto_plan::ValuesNode::from(&node); + assert!(proto.schema.is_some()); + assert_eq!(proto.rows.len(), 2); + assert_eq!(proto.rows[0].values.len(), 1); + } + + #[test] + fn from_project() { + let node = Project { + expr: Arc::new(Expression::struct_from([lit(1)])), + schema: sample_schema(), + }; + let proto = proto_plan::ProjectNode::from(&node); + assert!(proto.expr.is_some()); + assert!(proto.schema.is_some()); + } + + #[test] + fn from_filter() { + let node = Filter { + predicate: Arc::new(Predicate::TRUE), + }; + let proto = proto_plan::FilterNode::from(&node); + assert!(proto.predicate.is_some()); + } + + fn sample_dynamic_scan_input_schema() -> SchemaRef { + Arc::new(StructType::new_unchecked([ + StructField::not_null("path", DataType::STRING), + StructField::not_null("size", DataType::LONG), + StructField::not_null("filemod", DataType::LONG), + StructField::nullable("dv", DeletionVectorDescriptor::to_schema()), + StructField::nullable("c", DataType::INTEGER), + ])) + } + + fn sample_dynamic_scan_output_schema() -> SchemaRef { + Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("c", DataType::INTEGER), + ])) + } + + #[rstest] + #[case( + FileType::Json, + Url::parse("memory:///base/").unwrap(), + "memory:///base/" + )] + #[case( + FileType::Parquet, + Url::parse("file:///table/").unwrap(), + "file:///table/" + )] + fn from_dynamic_scan( + #[case] file_type: FileType, + #[case] base_url: Url, + #[case] expected_base_url: &str, + ) -> DeltaResult<()> { + let node = DynamicScan::try_new( + &sample_dynamic_scan_input_schema(), + sample_dynamic_scan_output_schema(), + file_type, + base_url, + ["c"], + column_name!("path"), + column_name!("size"), + column_name!("filemod"), + column_name!("dv"), + )?; + let proto = proto_plan::DynamicScanNode::from(&node); + assert!(proto.schema.is_some()); + assert_eq!( + proto.file_type, + proto_plan::FileType::from(file_type) as i32 + ); + assert_eq!(proto.base_url, expected_base_url); + assert_eq!(proto.file_constant_columns.len(), 1); + assert!(proto.dv_column.is_some()); + + assert_eq!( + proto.path_column.expect("path column present").path, + ["path"] + ); + assert_eq!( + proto + .file_size_column + .expect("file size column present") + .path, + ["size"] + ); + assert_eq!( + proto + .last_modified_column + .expect("last modified column present") + .path, + ["filemod"] + ); + Ok(()) + } + + #[test] + fn from_aggregate() { + let node = Aggregate { + group_by: vec![column_name!("g")], + aggs: vec![Agg::max(column_name!("a"))], + schema: sample_schema(), + }; + let proto = proto_plan::AggregateNode::from(&node); + assert_eq!(proto.group_by.len(), 1); + assert_eq!(proto.aggs.len(), 1); + assert!(proto.aggs[0].func.is_some()); + assert!(proto.schema.is_some()); + } + + #[rstest] + #[case(Agg::min(column_name!("a")), "min")] + #[case(Agg::max(column_name!("a")), "max")] + #[case(Agg::sum(column_name!("a")), "sum")] + #[case(Agg::count(column_name!("a")), "count")] + #[case(Agg::count_star(), "count_star")] + #[case(Agg::min_non_null_by(column_name!("a"), column_name!("s"), column_name!("k")), "min_non_null_by")] + #[case(Agg::max_non_null_by(column_name!("a"), column_name!("s"), column_name!("k")), "max_non_null_by")] + fn from_agg(#[case] agg: Agg, #[case] expected: &str) { + use proto_plan::agg::Func; + let proto = proto_plan::Agg::from(&agg); + let kind = match proto.func.unwrap() { + Func::Min(_) => "min", + Func::Max(_) => "max", + Func::Sum(_) => "sum", + Func::Count(_) => "count", + Func::CountStar(_) => "count_star", + Func::MinNonNullBy(_) => "min_non_null_by", + Func::MaxNonNullBy(_) => "max_non_null_by", + }; + assert_eq!(kind, expected); + } + + #[rstest] + #[case(true)] + #[case(false)] + fn from_semi_join(#[case] inverted: bool) { + let node = SemiJoin { + inverted, + probe_keys: vec![column_name!("p")], + build_keys: vec![column_name!("b")], + }; + let proto = proto_plan::SemiJoinNode::from(&node); + assert_eq!(proto.inverted, inverted); + assert_eq!(proto.probe_keys.len(), 1); + assert_eq!(proto.build_keys.len(), 1); + } + + #[rstest] + #[case(FileType::Parquet, proto_plan::FileType::Parquet)] + #[case(FileType::Json, proto_plan::FileType::Json)] + fn from_file_type(#[case] value: FileType, #[case] expected: proto_plan::FileType) { + assert_eq!(proto_plan::FileType::from(value) as i32, expected as i32); + } + + // === Expressions === + + #[rstest] + #[case(lit(1), "literal")] + #[case(col!("a"), "column")] + #[case(Expression::Predicate(Box::new(Predicate::TRUE)), "predicate")] + #[case(Expression::struct_from([lit(1)]), "struct_expr")] + #[case( + Expression::StructPatch( + ExpressionStructPatchBuilder::new().append(lit(1)).build().unwrap(), + ), + "transform" + )] + #[case(Expression::unary(UnaryExpressionOp::ToJson, lit(1)), "unary")] + #[case(Expression::binary(BinaryExpressionOp::Plus, lit(1), lit(2)), "binary")] + #[case(Expression::coalesce([lit(1), lit(2)]), "variadic")] + #[case(Expression::opaque(TestOpaqueExprOp, [lit(1)]), "opaque")] + #[case(Expression::parse_json(lit("{}"), sample_schema()), "parse_json")] + #[case( + Expression::map_to_struct(col!("m"), MapToStructOptions::default()), + "map_to_struct" + )] + #[case(Expression::element_at(col!("m"), lit("key")), "element_at")] + #[case( + Expression::cast(lit("1"), DataType::INTEGER, CastOptions::default(),), + "cast" + )] + #[case(Expression::unknown("x"), "unknown")] + fn from_expression(#[case] expr: Expression, #[case] expected: &str) { + use proto_expr::expression::Kind; + let kind = match proto_expr::Expression::from(&expr).kind.unwrap() { + Kind::Literal(_) => "literal", + Kind::Column(_) => "column", + Kind::Predicate(_) => "predicate", + Kind::StructExpr(_) => "struct_expr", + Kind::Transform(_) => "transform", + Kind::Unary(_) => "unary", + Kind::Binary(_) => "binary", + Kind::Variadic(_) => "variadic", + Kind::IfExpr(_) => "if_expr", + Kind::Opaque(_) => "opaque", + Kind::ParseJson(_) => "parse_json", + Kind::MapToStruct(_) => "map_to_struct", + Kind::Unknown(_) => "unknown", + Kind::ElementAt(_) => "element_at", + Kind::Cast(_) => "cast", + }; + assert_eq!(kind, expected); + } + + #[rstest] + #[case(Predicate::TRUE, "boolean_expression")] + #[case(Predicate::not(Predicate::TRUE), "not")] + #[case(Predicate::is_null(lit(1)), "unary")] + #[case(Predicate::gt(lit(1), lit(2)), "binary")] + #[case(Predicate::and(Predicate::TRUE, Predicate::FALSE), "junction")] + #[case(Predicate::opaque(TestOpaquePredOp, [lit(1)]), "opaque")] + #[case(Predicate::Unknown("x".to_string()), "unknown")] + fn from_predicate(#[case] pred: Predicate, #[case] expected: &str) { + use proto_expr::predicate::Kind; + let kind = match proto_expr::Predicate::from(&pred).kind.unwrap() { + Kind::BooleanExpression(_) => "boolean_expression", + Kind::Not(_) => "not", + Kind::Unary(_) => "unary", + Kind::Binary(_) => "binary", + Kind::Junction(_) => "junction", + Kind::Opaque(_) => "opaque", + Kind::Unknown(_) => "unknown", + }; + assert_eq!(kind, expected); + } + + #[test] + fn from_column_name() { + let proto = proto_expr::ColumnName::from(&column_name!("a.b.c")); + assert_eq!(proto.path, vec!["a", "b", "c"]); + } + + #[test] + fn from_struct_expression() { + let proto_expr::expression::Kind::StructExpr(plain) = + expr_kind_of(Expression::struct_from([lit(1), lit(2)])) + else { + panic!("expected a struct expression"); + }; + assert_eq!(plain.exprs.len(), 2); + assert!(plain.nullability_predicate.is_none()); + + let proto_expr::expression::Kind::StructExpr(guarded) = expr_kind_of( + Expression::struct_with_nullability_from([lit(1)], lit(true)), + ) else { + panic!("expected a struct expression"); + }; + assert_eq!(guarded.exprs.len(), 1); + assert!(guarded.nullability_predicate.is_some()); + } + + #[test] + fn from_unary_expression() { + let proto_expr::expression::Kind::Unary(unary) = + expr_kind_of(Expression::unary(UnaryExpressionOp::ToJson, lit(1))) + else { + panic!("expected a unary expression"); + }; + assert_eq!(unary.op, proto_expr::UnaryExpressionOp::ToJson as i32); + assert!(unary.expr.is_some()); + } + + #[test] + fn from_binary_expression() { + let proto_expr::expression::Kind::Binary(binary) = + expr_kind_of(Expression::binary(BinaryExpressionOp::Plus, lit(1), lit(2))) + else { + panic!("expected a binary expression"); + }; + assert_eq!(binary.op, proto_expr::BinaryExpressionOp::Plus as i32); + assert!(binary.left.is_some()); + assert!(binary.right.is_some()); + } + + #[test] + fn from_variadic_expression() { + let proto_expr::expression::Kind::Variadic(variadic) = + expr_kind_of(Expression::coalesce([lit(1), lit(2), lit(3)])) + else { + panic!("expected a variadic expression"); + }; + assert_eq!( + variadic.op, + proto_expr::VariadicExpressionOp::Coalesce as i32 + ); + assert_eq!(variadic.exprs.len(), 3); + } + + #[test] + fn from_element_at_expression() { + let proto_expr::expression::Kind::ElementAt(element_at) = + expr_kind_of(Expression::element_at(col!("map"), col!("key"))) + else { + panic!("expected an element-at expression"); + }; + assert!(element_at.map_expr.is_some()); + assert!(element_at.key_expr.is_some()); + } + + #[test] + fn from_cast_expression() { + let proto_expr::expression::Kind::Cast(cast) = expr_kind_of(Expression::cast( + lit("1"), + DataType::INTEGER, + CastOptions::default(), + )) else { + panic!("expected a cast expression"); + }; + assert!(cast.expr.is_some()); + assert_eq!(cast.target, Some((&DataType::INTEGER).into())); + assert_eq!(cast.options, None); + + let proto_expr::expression::Kind::Cast(cast) = expr_kind_of(Expression::cast( + lit("1"), + DataType::TIMESTAMP, + CastOptions::default().with_timestamp_timezone("America/Los_Angeles"), + )) else { + panic!("expected a cast expression"); + }; + assert_eq!( + cast.options + .as_ref() + .and_then(|options| options.timestamp_timezone.as_deref()), + Some("America/Los_Angeles") + ); + } + + #[test] + fn from_opaque_expression() { + let proto_expr::expression::Kind::Opaque(opaque) = + expr_kind_of(Expression::opaque(TestOpaqueExprOp, [lit(1), lit(2)])) + else { + panic!("expected an opaque expression"); + }; + assert_eq!(opaque.name, "test_opaque_expr"); + assert_eq!(opaque.exprs.len(), 2); + } + + #[test] + fn from_parse_json_expression() { + let proto_expr::expression::Kind::ParseJson(parse_json) = + expr_kind_of(Expression::parse_json(lit("{}"), sample_schema())) + else { + panic!("expected a parse_json expression"); + }; + assert!(parse_json.json_expr.is_some()); + assert!(parse_json.output_schema.is_some()); + } + + #[test] + fn from_map_to_struct_expression() { + let proto_expr::expression::Kind::MapToStruct(map_to_struct) = expr_kind_of( + Expression::map_to_struct(col!("m"), MapToStructOptions::default()), + ) else { + panic!("expected a map_to_struct expression"); + }; + assert!(map_to_struct.map_expr.is_some()); + assert_eq!(map_to_struct.options, None); + + let proto_expr::expression::Kind::MapToStruct(map_to_struct) = + expr_kind_of(Expression::map_to_struct( + col!("m"), + MapToStructOptions::default().with_timestamp_timezone("America/Los_Angeles"), + )) + else { + panic!("expected a map_to_struct expression"); + }; + assert_eq!( + map_to_struct + .options + .as_ref() + .and_then(|options| options.timestamp_timezone.as_deref()), + Some("America/Los_Angeles") + ); + } + + #[test] + fn from_unary_predicate() { + let proto_expr::predicate::Kind::Unary(unary) = pred_kind_of(Predicate::is_null(lit(1))) + else { + panic!("expected a unary predicate"); + }; + assert_eq!(unary.op, proto_expr::UnaryPredicateOp::IsNull as i32); + assert!(unary.expr.is_some()); + } + + #[test] + fn from_binary_predicate() { + let proto_expr::predicate::Kind::Binary(binary) = + pred_kind_of(Predicate::gt(lit(1), lit(2))) + else { + panic!("expected a binary predicate"); + }; + assert_eq!(binary.op, proto_expr::BinaryPredicateOp::GreaterThan as i32); + assert!(binary.left.is_some()); + assert!(binary.right.is_some()); + } + + #[test] + fn from_junction_predicate() { + let proto_expr::predicate::Kind::Junction(junction) = + pred_kind_of(Predicate::and(Predicate::TRUE, Predicate::FALSE)) + else { + panic!("expected a junction predicate"); + }; + assert_eq!(junction.op, proto_expr::JunctionPredicateOp::And as i32); + assert_eq!(junction.preds.len(), 2); + } + + #[test] + fn from_opaque_predicate() { + let proto_expr::predicate::Kind::Opaque(opaque) = + pred_kind_of(Predicate::opaque(TestOpaquePredOp, [lit(1), lit(2)])) + else { + panic!("expected an opaque predicate"); + }; + assert_eq!(opaque.name, "test_opaque_pred"); + assert_eq!(opaque.exprs.len(), 2); + } + + #[test] + fn from_struct_patch() { + let patch = ExpressionStructPatchBuilder::new() + .replace("a", lit(1)) + .insert_after("b", lit(2)) + .drop_if_exists("c") + .prepend(lit(0)) + .append(lit(3)) + .build() + .unwrap(); + + let transform = proto_expr::Transform::from(&patch); + + // `replace` drops the input field, so `is_replace` is true and it is not optional. + assert!(transform.field_transforms["a"].is_replace); + assert!(!transform.field_transforms["a"].optional); + // `insert_after` keeps the input field, so `is_replace` is false. + assert!(!transform.field_transforms["b"].is_replace); + // `drop_if_exists` drops the input field optionally. + assert!(transform.field_transforms["c"].is_replace); + assert!(transform.field_transforms["c"].optional); + // A top-level transform carries no input path. + assert!(transform.input_path.is_none()); + assert_eq!(transform.prepended_fields.len(), 1); + assert_eq!(transform.appended_fields.len(), 1); + } + + // === Expression and predicate operators === + + #[rstest] + #[case(UnaryExpressionOp::ToJson, proto_expr::UnaryExpressionOp::ToJson)] + fn from_unary_expression_op( + #[case] op: UnaryExpressionOp, + #[case] expected: proto_expr::UnaryExpressionOp, + ) { + assert_eq!( + proto_expr::UnaryExpressionOp::from(op) as i32, + expected as i32 + ); + } + + #[rstest] + #[case(BinaryExpressionOp::Plus, proto_expr::BinaryExpressionOp::Plus)] + #[case(BinaryExpressionOp::Minus, proto_expr::BinaryExpressionOp::Minus)] + #[case(BinaryExpressionOp::Multiply, proto_expr::BinaryExpressionOp::Multiply)] + #[case(BinaryExpressionOp::Divide, proto_expr::BinaryExpressionOp::Divide)] + fn from_binary_expression_op( + #[case] op: BinaryExpressionOp, + #[case] expected: proto_expr::BinaryExpressionOp, + ) { + assert_eq!( + proto_expr::BinaryExpressionOp::from(op) as i32, + expected as i32 + ); + } + + #[rstest] + #[case( + VariadicExpressionOp::Coalesce, + proto_expr::VariadicExpressionOp::Coalesce + )] + #[case(VariadicExpressionOp::Array, proto_expr::VariadicExpressionOp::Array)] + fn from_variadic_expression_op( + #[case] op: VariadicExpressionOp, + #[case] expected: proto_expr::VariadicExpressionOp, + ) { + assert_eq!( + proto_expr::VariadicExpressionOp::from(op) as i32, + expected as i32 + ); + } + + #[rstest] + #[case(UnaryPredicateOp::IsNull, proto_expr::UnaryPredicateOp::IsNull)] + fn from_unary_predicate_op( + #[case] op: UnaryPredicateOp, + #[case] expected: proto_expr::UnaryPredicateOp, + ) { + assert_eq!( + proto_expr::UnaryPredicateOp::from(op) as i32, + expected as i32 + ); + } + + #[rstest] + #[case(BinaryPredicateOp::LessThan, proto_expr::BinaryPredicateOp::LessThan)] + #[case( + BinaryPredicateOp::GreaterThan, + proto_expr::BinaryPredicateOp::GreaterThan + )] + #[case(BinaryPredicateOp::Equal, proto_expr::BinaryPredicateOp::Equal)] + #[case(BinaryPredicateOp::Distinct, proto_expr::BinaryPredicateOp::Distinct)] + #[case(BinaryPredicateOp::In, proto_expr::BinaryPredicateOp::In)] + fn from_binary_predicate_op( + #[case] op: BinaryPredicateOp, + #[case] expected: proto_expr::BinaryPredicateOp, + ) { + assert_eq!( + proto_expr::BinaryPredicateOp::from(op) as i32, + expected as i32 + ); + } + + #[rstest] + #[case(JunctionPredicateOp::And, proto_expr::JunctionPredicateOp::And)] + #[case(JunctionPredicateOp::Or, proto_expr::JunctionPredicateOp::Or)] + fn from_junction_predicate_op( + #[case] op: JunctionPredicateOp, + #[case] expected: proto_expr::JunctionPredicateOp, + ) { + assert_eq!( + proto_expr::JunctionPredicateOp::from(op) as i32, + expected as i32 + ); + } + + // === Scalars === + + #[rstest] + #[case(Scalar::Integer(7), "integer")] + #[case(Scalar::Long(7), "long")] + #[case(Scalar::Short(7), "short")] + #[case(Scalar::Byte(7), "byte")] + #[case(Scalar::Float(1.5), "float")] + #[case(Scalar::Double(1.5), "double")] + #[case(Scalar::String("hi".into()), "string")] + #[case(Scalar::Boolean(true), "boolean")] + #[case(Scalar::Timestamp(9), "timestamp")] + #[case(Scalar::TimestampNtz(9), "timestamp_ntz")] + #[case(Scalar::Date(9), "date")] + #[case(Scalar::Binary(vec![1, 2, 3]), "binary")] + #[case( + Scalar::Decimal( + DecimalData::try_new(1234i128, DecimalType::try_new(10, 2).unwrap()).unwrap(), + ), + "decimal" + )] + #[case(Scalar::Null(DataType::LONG), "null")] + #[case( + Scalar::Struct( + StructData::try_new( + vec![StructField::nullable("a", DataType::INTEGER)], + vec![Scalar::Integer(1)], + ) + .unwrap(), + ), + "struct" + )] + #[case( + Scalar::Array( + ArrayData::try_new(ArrayType::new(DataType::INTEGER, false), [1, 2, 3]).unwrap(), + ), + "array" + )] + #[case( + Scalar::Map( + MapData::try_new(MapType::new(DataType::STRING, DataType::INTEGER, false), [("k", 1)]) + .unwrap(), + ), + "map" + )] + fn from_scalar(#[case] value: Scalar, #[case] expected: &str) { + use proto_expr::scalar::Value; + let kind = match proto_expr::Scalar::from(&value).value.unwrap() { + Value::Integer(_) => "integer", + Value::Long(_) => "long", + Value::Short(_) => "short", + Value::Byte(_) => "byte", + Value::Float(_) => "float", + Value::Double(_) => "double", + Value::String(_) => "string", + Value::Boolean(_) => "boolean", + Value::Timestamp(_) => "timestamp", + Value::TimestampNtz(_) => "timestamp_ntz", + Value::IntervalYearMonth(_) => "interval_year_month", + Value::IntervalDayTime(_) => "interval_day_time", + Value::Date(_) => "date", + Value::Binary(_) => "binary", + Value::Decimal(_) => "decimal", + Value::Null(_) => "null", + Value::Struct(_) => "struct", + Value::Array(_) => "array", + Value::Map(_) => "map", + }; + assert_eq!(kind, expected); + } + + #[rstest] + #[case( + Scalar::IntervalYearMonth(30), + proto_expr::scalar::Value::IntervalYearMonth(30) + )] + #[case( + Scalar::IntervalYearMonth(i32::MIN), + proto_expr::scalar::Value::IntervalYearMonth(i32::MIN) + )] + #[case( + Scalar::IntervalDayTime(-5), + proto_expr::scalar::Value::IntervalDayTime(-5) + )] + #[case( + Scalar::IntervalDayTime(i64::MAX), + proto_expr::scalar::Value::IntervalDayTime(i64::MAX) + )] + fn from_interval_scalar_preserves_payload( + #[case] scalar: Scalar, + #[case] expected: proto_expr::scalar::Value, + ) { + assert_eq!(scalar_value_of(scalar), expected); + } + + #[test] + fn from_decimal_data() { + let decimal = DecimalData::try_new(1234i128, DecimalType::try_new(10, 2).unwrap()).unwrap(); + let proto_expr::scalar::Value::Decimal(decimal) = scalar_value_of(Scalar::Decimal(decimal)) + else { + panic!("expected a decimal scalar"); + }; + assert_eq!(decimal.bits, 1234i128.to_be_bytes().to_vec()); + let decimal_type = decimal.decimal_type.expect("decimal type present"); + assert_eq!((decimal_type.precision, decimal_type.scale), (10, 2)); + } + + #[test] + fn from_struct_data() { + let struct_data = StructData::try_new( + vec![StructField::nullable("a", DataType::INTEGER)], + vec![Scalar::Integer(1)], + ) + .unwrap(); + let proto_expr::scalar::Value::Struct(struct_data) = + scalar_value_of(Scalar::Struct(struct_data)) + else { + panic!("expected a struct scalar"); + }; + assert_eq!(struct_data.fields.len(), 1); + assert_eq!(struct_data.values.len(), 1); + assert_eq!(struct_data.fields[0].name, "a"); + } + + #[test] + fn from_array_data() { + let array_data = + ArrayData::try_new(ArrayType::new(DataType::INTEGER, false), [1, 2, 3]).unwrap(); + let proto_expr::scalar::Value::Array(array_data) = + scalar_value_of(Scalar::Array(array_data)) + else { + panic!("expected an array scalar"); + }; + assert!(array_data.array_type.is_some()); + assert_eq!(array_data.elements.len(), 3); + } + + #[test] + fn from_map_data() { + let map_data = MapData::try_new( + MapType::new(DataType::STRING, DataType::INTEGER, false), + [("k", 1)], + ) + .unwrap(); + let proto_expr::scalar::Value::Map(map_data) = scalar_value_of(Scalar::Map(map_data)) + else { + panic!("expected a map scalar"); + }; + assert!(map_data.map_type.is_some()); + assert_eq!(map_data.pairs.len(), 1); + assert!(map_data.pairs[0].key.is_some()); + assert!(map_data.pairs[0].value.is_some()); + } + + // === Schema === + + #[rstest] + #[case(DataType::INTEGER, "primitive")] + #[case(ArrayType::new(DataType::INTEGER, true).into(), "array")] + #[case( + StructType::try_new(vec![StructField::nullable("a", DataType::INTEGER)]) + .unwrap() + .into(), + "struct" + )] + #[case(MapType::new(DataType::STRING, DataType::INTEGER, true).into(), "map")] + #[case(DataType::unshredded_variant(), "variant")] + fn from_data_type(#[case] value: DataType, #[case] expected: &str) { + use proto_schema::data_type::Kind; + let kind = match proto_schema::DataType::from(&value).kind.unwrap() { + Kind::Primitive(_) => "primitive", + Kind::Array(_) => "array", + Kind::Struct(_) => "struct", + Kind::Map(_) => "map", + Kind::Variant(_) => "variant", + }; + assert_eq!(kind, expected); + } + + #[rstest] + #[case(PrimitiveType::String, proto_schema::SimplePrimitiveType::String)] + #[case(PrimitiveType::Long, proto_schema::SimplePrimitiveType::Long)] + #[case(PrimitiveType::Integer, proto_schema::SimplePrimitiveType::Integer)] + #[case(PrimitiveType::Short, proto_schema::SimplePrimitiveType::Short)] + #[case(PrimitiveType::Byte, proto_schema::SimplePrimitiveType::Byte)] + #[case(PrimitiveType::Float, proto_schema::SimplePrimitiveType::Float)] + #[case(PrimitiveType::Double, proto_schema::SimplePrimitiveType::Double)] + #[case(PrimitiveType::Boolean, proto_schema::SimplePrimitiveType::Boolean)] + #[case(PrimitiveType::Binary, proto_schema::SimplePrimitiveType::Binary)] + #[case(PrimitiveType::Date, proto_schema::SimplePrimitiveType::Date)] + #[case(PrimitiveType::Timestamp, proto_schema::SimplePrimitiveType::Timestamp)] + #[case( + PrimitiveType::TimestampNtz, + proto_schema::SimplePrimitiveType::TimestampNtz + )] + #[case(PrimitiveType::Void, proto_schema::SimplePrimitiveType::Void)] + #[case( + PrimitiveType::IntervalYearMonth, + proto_schema::SimplePrimitiveType::IntervalYearMonth + )] + #[case( + PrimitiveType::IntervalDayTime, + proto_schema::SimplePrimitiveType::IntervalDayTime + )] + fn from_primitive_type( + #[case] primitive: PrimitiveType, + #[case] expected: proto_schema::SimplePrimitiveType, + ) { + let Some(proto_schema::primitive_type::Kind::Simple(simple)) = + proto_schema::PrimitiveType::from(&primitive).kind + else { + panic!("expected a simple primitive type"); + }; + assert_eq!(simple, expected as i32); + } + + #[test] + fn from_decimal_type() { + let primitive = PrimitiveType::decimal(10, 2).unwrap(); + let Some(proto_schema::primitive_type::Kind::Decimal(decimal)) = + proto_schema::PrimitiveType::from(&primitive).kind + else { + panic!("expected a decimal primitive type"); + }; + assert_eq!((decimal.precision, decimal.scale), (10, 2)); + } + + #[rstest] + #[case(true)] + #[case(false)] + fn from_array_type(#[case] contains_null: bool) { + let proto = + proto_schema::ArrayType::from(&ArrayType::new(DataType::INTEGER, contains_null)); + assert!(proto.element_type.is_some()); + assert_eq!(proto.contains_null, contains_null); + } + + #[rstest] + #[case(true)] + #[case(false)] + fn from_map_type(#[case] value_contains_null: bool) { + let proto = proto_schema::MapType::from(&MapType::new( + DataType::STRING, + DataType::INTEGER, + value_contains_null, + )); + assert!(proto.key_type.is_some()); + assert!(proto.value_type.is_some()); + assert_eq!(proto.value_contains_null, value_contains_null); + } + + #[test] + fn from_struct_type() { + let struct_type = StructType::try_new(vec![ + StructField::nullable("a", DataType::INTEGER), + StructField::not_null("b", DataType::STRING), + ]) + .unwrap(); + let proto = proto_schema::StructType::from(&struct_type); + assert_eq!(proto.fields.len(), 2); + assert!(proto.fields[0].nullable); + assert!(!proto.fields[1].nullable); + } + + #[test] + fn from_struct_field() { + let field = StructField::nullable("a", DataType::INTEGER) + .with_metadata([("k", MetadataValue::Number(7))]); + let proto = proto_schema::StructField::from(&field); + assert_eq!(proto.name, "a"); + assert!(proto.nullable); + assert!(proto.data_type.is_some()); + assert_eq!( + proto.metadata["k"].value, + Some(proto_schema::metadata_value::Value::Number(7)) + ); + } + + #[rstest] + #[case( + MetadataValue::Number(5), + proto_schema::metadata_value::Value::Number(5) + )] + #[case( + MetadataValue::String("s".to_string()), + proto_schema::metadata_value::Value::String("s".to_string()) + )] + #[case( + MetadataValue::Boolean(true), + proto_schema::metadata_value::Value::Boolean(true) + )] + #[case( + MetadataValue::Other(serde_json::json!([1, 2])), + proto_schema::metadata_value::Value::OtherJson("[1,2]".to_string()) + )] + fn from_metadata_value( + #[case] metadata: MetadataValue, + #[case] expected: proto_schema::metadata_value::Value, + ) { + assert_eq!( + proto_schema::MetadataValue::from(&metadata).value.unwrap(), + expected + ); + } + + // === Schema decode (proto -> kernel) === + + /// Round-trips a [`DataType`] through its proto form and back, asserting it is unchanged. + fn assert_data_type_round_trips(data_type: DataType) { + let decoded = DataType::try_from(proto_schema::DataType::from(&data_type)); + assert_eq!(decoded.expect("decode succeeds"), data_type); + } + + /// Round-trips a [`StructType`] through its proto form and back, asserting it is unchanged. + fn assert_schema_round_trips(schema: StructType) { + let decoded = StructType::try_from(proto_schema::StructType::from(&schema)); + assert_eq!(decoded.expect("decode succeeds"), schema); + } + + #[rstest] + fn round_trip_primitive( + #[values( + PrimitiveType::String, + PrimitiveType::Long, + PrimitiveType::Integer, + PrimitiveType::Short, + PrimitiveType::Byte, + PrimitiveType::Float, + PrimitiveType::Double, + PrimitiveType::Boolean, + PrimitiveType::Binary, + PrimitiveType::Date, + PrimitiveType::Timestamp, + PrimitiveType::TimestampNtz, + PrimitiveType::Void, + PrimitiveType::IntervalYearMonth, + PrimitiveType::IntervalDayTime + )] + primitive: PrimitiveType, + ) { + assert_data_type_round_trips(DataType::Primitive(primitive)); + } + + #[rstest] + #[case(1, 0)] + #[case(10, 2)] + #[case(38, 0)] + #[case(38, 38)] + fn round_trip_decimal(#[case] precision: u8, #[case] scale: u8) { + assert_data_type_round_trips(DataType::Primitive( + PrimitiveType::decimal(precision, scale).unwrap(), + )); + } + + #[cfg(feature = "geo-type-in-dev")] + #[rstest] + #[case(GeometryType::try_new("OGC:CRS84").unwrap())] + #[case(GeometryType::try_new("EPSG:4326").unwrap())] + fn round_trip_geometry(#[case] geometry: GeometryType) { + assert_data_type_round_trips(DataType::Primitive(PrimitiveType::Geometry(Box::new( + geometry, + )))); + } + + #[cfg(feature = "geo-type-in-dev")] + #[rstest] + fn round_trip_geography( + #[values("OGC:CRS84", "EPSG:4326")] crs: &str, + #[values( + EdgeInterpolationAlgorithm::Spherical, + EdgeInterpolationAlgorithm::Vincenty, + EdgeInterpolationAlgorithm::Thomas, + EdgeInterpolationAlgorithm::Andoyer, + EdgeInterpolationAlgorithm::Karney + )] + algorithm: EdgeInterpolationAlgorithm, + ) { + let geography = GeographyType::try_new(crs, algorithm).unwrap(); + assert_data_type_round_trips(DataType::Primitive(PrimitiveType::Geography(Box::new( + geography, + )))); + } + + #[cfg(feature = "geo-type-in-dev")] + #[rstest] + #[case(EdgeInterpolationAlgorithm::Spherical, 1)] + #[case(EdgeInterpolationAlgorithm::Vincenty, 2)] + #[case(EdgeInterpolationAlgorithm::Thomas, 3)] + #[case(EdgeInterpolationAlgorithm::Andoyer, 4)] + #[case(EdgeInterpolationAlgorithm::Karney, 5)] + fn edge_algo_proto_discriminant_is_stable( + #[case] algorithm: EdgeInterpolationAlgorithm, + #[case] proto_tag: i32, + ) { + assert_eq!(EdgeAlgo::from(&algorithm) as i32, proto_tag); + let proto = EdgeAlgo::try_from(proto_tag).unwrap(); + assert_eq!( + EdgeInterpolationAlgorithm::try_from(proto).unwrap(), + algorithm + ); + } + + #[rstest] + #[case(DataType::from(ArrayType::new(DataType::INTEGER, true)))] + #[case(DataType::from(ArrayType::new(DataType::STRING, false)))] + #[case(DataType::from(MapType::new(DataType::STRING, DataType::LONG, true)))] + #[case(DataType::from(MapType::new(DataType::INTEGER, DataType::BOOLEAN, false)))] + #[case(DataType::from(ArrayType::new( + MapType::new(DataType::STRING, DataType::LONG, true), + true + )))] + #[case(DataType::from(StructType::try_new(vec![ + StructField::nullable("a", DataType::INTEGER), + StructField::not_null("b", DataType::STRING), + ]).unwrap()))] + fn round_trip_composite(#[case] data_type: DataType) { + assert_data_type_round_trips(data_type); + } + + #[rstest] + #[case(MetadataValue::Number(7))] + #[case(MetadataValue::String("hello".to_string()))] + #[case(MetadataValue::Boolean(true))] + #[case(MetadataValue::Other(serde_json::json!({"nested": [1, 2, 3]})))] + fn round_trip_field_metadata(#[case] value: MetadataValue) { + let field = StructField::nullable("f", DataType::INTEGER).with_metadata([("k", value)]); + let decoded = StructField::try_from(proto_schema::StructField::from(&field)); + assert_eq!(decoded.expect("decode succeeds"), field); + } + + #[test] + fn round_trip_full_schema() { + let schema = StructType::try_new(vec![ + StructField::nullable("id", DataType::LONG) + .with_metadata([("k", MetadataValue::Number(7))]), + StructField::not_null("name", DataType::STRING), + StructField::nullable("scores", ArrayType::new(DataType::INTEGER, true)), + StructField::nullable( + "attrs", + MapType::new(DataType::STRING, DataType::LONG, true), + ), + StructField::nullable( + "price", + DataType::Primitive(PrimitiveType::decimal(10, 2).unwrap()), + ), + StructField::nullable( + "nested", + StructType::try_new(vec![StructField::not_null("inner", DataType::BOOLEAN)]) + .unwrap(), + ), + ]) + .unwrap(); + assert_schema_round_trips(schema); + } + + /// The proto `VariantType` is currently opaque, so any variant (even a shredded one) + /// decodes to the canonical unshredded form rather than round-tripping its inner struct. + #[test] + fn variant_decodes_to_unshredded() { + let shredded = DataType::Variant(Box::new( + StructType::try_new(vec![ + StructField::not_null("metadata", DataType::BINARY), + StructField::not_null("value", DataType::BINARY), + StructField::nullable("typed_value", DataType::INTEGER), + ]) + .unwrap(), + )); + let decoded = DataType::try_from(proto_schema::DataType::from(&shredded)); + assert_eq!( + decoded.expect("decode succeeds"), + DataType::unshredded_variant() + ); + } + + // Builds a proto `DataType::Primitive` with the given (possibly malformed) primitive kind. + fn primitive_data_type( + kind: Option, + ) -> proto_schema::DataType { + proto_schema::DataType { + kind: Some(proto_schema::data_type::Kind::Primitive( + proto_schema::PrimitiveType { kind }, + )), + } + } + + fn simple_primitive_data_type(value: i32) -> proto_schema::DataType { + primitive_data_type(Some(proto_schema::primitive_type::Kind::Simple(value))) + } + + fn decimal_data_type(precision: u32, scale: u32) -> proto_schema::DataType { + primitive_data_type(Some(proto_schema::primitive_type::Kind::Decimal( + proto_schema::DecimalType { precision, scale }, + ))) + } + + #[rstest] + #[case::missing_data_type_kind(proto_schema::DataType { kind: None })] + #[case::missing_primitive_kind(primitive_data_type(None))] + #[case::unspecified_primitive(simple_primitive_data_type( + proto_schema::SimplePrimitiveType::Unspecified as i32 + ))] + #[case::unknown_primitive(simple_primitive_data_type(9999))] + #[case::decimal_zero_precision(decimal_data_type(0, 0))] + #[case::decimal_precision_too_large(decimal_data_type(39, 0))] + #[case::decimal_scale_exceeds_precision(decimal_data_type(5, 6))] + #[case::decimal_precision_out_of_u8_range(decimal_data_type(256, 0))] + #[case::array_missing_element_type(proto_schema::DataType { + kind: Some(proto_schema::data_type::Kind::Array(Box::new(proto_schema::ArrayType { + element_type: None, + contains_null: true, + }))), + })] + #[case::map_missing_key_type(proto_schema::DataType { + kind: Some(proto_schema::data_type::Kind::Map(Box::new(proto_schema::MapType { + key_type: None, + value_type: Some(Box::new(proto_schema::DataType::from(&DataType::STRING))), + value_contains_null: true, + }))), + })] + #[case::map_missing_value_type(proto_schema::DataType { + kind: Some(proto_schema::data_type::Kind::Map(Box::new(proto_schema::MapType { + key_type: Some(Box::new(proto_schema::DataType::from(&DataType::STRING))), + value_type: None, + value_contains_null: true, + }))), + })] + fn data_type_decode_rejects_invalid(#[case] proto: proto_schema::DataType) { + assert!(DataType::try_from(proto).is_err()); + } + + #[test] + fn struct_field_decode_rejects_missing_data_type() { + let proto = proto_schema::StructField { + name: "f".to_string(), + data_type: None, + nullable: true, + metadata: Default::default(), + }; + assert!(StructField::try_from(proto).is_err()); + } + + #[rstest] + #[case::missing_value(proto_schema::MetadataValue { value: None })] + #[case::invalid_other_json(proto_schema::MetadataValue { + value: Some(proto_schema::metadata_value::Value::OtherJson("not json".to_string())), + })] + fn metadata_value_decode_rejects_invalid(#[case] proto: proto_schema::MetadataValue) { + assert!(MetadataValue::try_from(proto).is_err()); + } +} diff --git a/kernel/src/plans/proto/mod.rs b/kernel/src/plans/proto/mod.rs new file mode 100644 index 0000000000..93b5439724 --- /dev/null +++ b/kernel/src/plans/proto/mod.rs @@ -0,0 +1,36 @@ +//! Protobuf wire format mirroring the kernel's plan / schema / expression IR. +//! +//! Each submodule wraps the prost-generated code for one `.proto` file in +//! `kernel/proto/`. Consumers can serialise a kernel-built plan and ship it +//! across a process / language boundary (Rust kernel -> JVM engine, etc.). +//! +//! - [`schema`] -- mirror of `kernel/src/schema/mod.rs` (`DataType`, `PrimitiveType`, `StructType`, +//! ...). +//! - [`expressions`] -- mirror of `kernel/src/expressions/mod.rs` and `scalars.rs` (`Expression`, +//! `Predicate`, `Scalar`, `ColumnName`, ...). +//! - [`plan`] -- mirror of `kernel/src/plans/ir/{plan,nodes}.rs` (`Plan`, `PlanNode`, `Operator`, +//! per-variant payload messages). +//! - [`operation`] -- mirror of `kernel/src/plans/ir/operation.rs` (`Operation`, `IoOperation`, and +//! the I/O payload messages). + +pub mod schema { + include!(concat!(env!("OUT_DIR"), "/delta.kernel.schema.rs")); +} + +pub mod expressions { + include!(concat!(env!("OUT_DIR"), "/delta.kernel.expressions.rs")); +} + +pub mod plan { + include!(concat!(env!("OUT_DIR"), "/delta.kernel.plan.rs")); +} + +// The top-level `Operation` message's oneof generates a nested `mod operation`, which trips +// `clippy::module_inception` against this same-named module. The name mirrors the `.proto` file +// (matching the other proto modules), so suppress the lint rather than rename. +#[allow(clippy::module_inception)] +pub mod operation { + include!(concat!(env!("OUT_DIR"), "/delta.kernel.operation.rs")); +} + +mod convert; diff --git a/kernel/src/row_tracking.rs b/kernel/src/row_tracking.rs index 5bf2a08061..1b4a2b5def 100644 --- a/kernel/src/row_tracking.rs +++ b/kernel/src/row_tracking.rs @@ -1,16 +1,20 @@ +// Allow unreachable_pub because this module is pub when test-utils is enabled, +// making the pub items reachable from integration tests. +#![allow(unreachable_pub)] + use std::sync::LazyLock; use serde::{Deserialize, Serialize}; -use crate::actions::DomainMetadata; +use crate::actions::{DomainMetadata, NUM_RECORDS}; use crate::engine_data::{GetData, RowVisitor, TypedGetData as _}; -use crate::schema::{ColumnName, ColumnNamesAndTypes, DataType}; +use crate::schema::{column_name, ColumnName, ColumnNamesAndTypes, DataType}; use crate::utils::require; use crate::{DeltaResult, Engine, Error, Snapshot}; #[derive(Debug, Deserialize, Serialize)] #[serde(rename_all = "camelCase")] -pub(crate) struct RowTrackingDomainMetadata { +pub struct RowTrackingDomainMetadata { // NB: The Delta spec does not rule out negative high water marks row_id_high_water_mark: i64, } @@ -19,12 +23,24 @@ pub(crate) struct RowTrackingDomainMetadata { pub(crate) const ROW_TRACKING_DOMAIN_NAME: &str = "delta.rowTracking"; impl RowTrackingDomainMetadata { + /// The row ID high water mark for a table with no assigned row IDs yet. The first + /// file written receives `baseRowId = MISSING_ROW_ID_HIGH_WATERMARK + 1 = 0`. + pub(crate) const MISSING_ROW_ID_HIGH_WATERMARK: i64 = -1; + pub(crate) fn new(row_id_high_water_mark: i64) -> Self { RowTrackingDomainMetadata { row_id_high_water_mark, } } + /// Creates the initial row tracking domain metadata for a newly created table. + /// + /// Sets the high water mark to -1, meaning no rows have been assigned IDs yet. + /// The first file written will receive `baseRowId = 0`. + pub(crate) fn initial() -> Self { + Self::new(Self::MISSING_ROW_ID_HIGH_WATERMARK) + } + /// Retrieves the row ID high water mark from the [`Snapshot`]'s row tracking domain metadata. /// /// This method searches through the snapshot's log segment for domain metadata actions @@ -41,7 +57,7 @@ impl RowTrackingDomainMetadata { /// This method will return an error if: /// - The domain metadata configuration cannot be read from the log segment /// - The domain metadata JSON cannot be deserialized into `RowTrackingDomainMetadata` - pub(crate) fn get_high_water_mark( + pub fn get_high_water_mark( snapshot: &Snapshot, engine: &dyn Engine, ) -> DeltaResult> { @@ -82,13 +98,12 @@ pub(crate) struct RowTrackingVisitor { } impl RowTrackingVisitor { - /// Default value for an absent high water mark - const DEFAULT_HIGH_WATER_MARK: i64 = -1; - pub(crate) fn new(row_id_high_water_mark: Option, num_batches: Option) -> Self { - // A table might not have a row ID high water mark yet, so we model the input as an Option + // A table might not have a row ID high water mark yet, so we model the input as an + // Option Self { - row_id_high_water_mark: row_id_high_water_mark.unwrap_or(Self::DEFAULT_HIGH_WATER_MARK), + row_id_high_water_mark: row_id_high_water_mark + .unwrap_or(RowTrackingDomainMetadata::MISSING_ROW_ID_HIGH_WATERMARK), base_row_id_batches: Vec::with_capacity(num_batches.unwrap_or(0)), } } @@ -98,7 +113,7 @@ impl RowVisitor for RowTrackingVisitor { fn selected_column_names_and_types(&self) -> (&'static [ColumnName], &'static [DataType]) { static NAMES_AND_TYPES: LazyLock = LazyLock::new(|| { ( - vec![ColumnName::new(["stats", "numRecords"])], + vec![column_name!("stats", NUM_RECORDS)], vec![DataType::LONG], ) .into() @@ -120,11 +135,10 @@ impl RowVisitor for RowTrackingVisitor { let mut current_hwm = self.row_id_high_water_mark; for i in 0..row_count { - let num_records: i64 = getters[0].get_opt(i, "numRecords")?.ok_or_else(|| { - Error::InternalError( - "numRecords must be present in Add actions when row tracking is enabled." - .to_string(), - ) + let num_records: i64 = getters[0].get_opt(i, NUM_RECORDS)?.ok_or_else(|| { + Error::InternalError(format!( + "{NUM_RECORDS} must be present in Add actions when row tracking is enabled." + )) })?; batch_base_row_ids.push(current_hwm + 1); current_hwm += num_records; @@ -140,7 +154,7 @@ impl RowVisitor for RowTrackingVisitor { mod tests { use super::*; use crate::engine_data::GetData; - use crate::utils::test_utils::assert_result_error_with_message; + use crate::unit_test_utils::assert_result_error_with_message; /// Mock GetData implementation for testing struct MockGetData { @@ -155,7 +169,7 @@ mod tests { impl<'a> GetData<'a> for MockGetData { fn get_long(&'a self, row_index: usize, field_name: &str) -> DeltaResult> { - if field_name == "numRecords" { + if field_name == NUM_RECORDS { Ok(self.num_records_values.get(row_index).copied().flatten()) } else { Ok(None) @@ -286,7 +300,7 @@ mod tests { let result = visitor.visit(1, &getters); assert_result_error_with_message( result, - "numRecords must be present in Add actions when row tracking is enabled", + &format!("{NUM_RECORDS} must be present in Add actions when row tracking is enabled"), ); Ok(()) @@ -297,7 +311,7 @@ mod tests { let visitor = RowTrackingVisitor::new(Some(0), None); let (names, types) = visitor.selected_column_names_and_types(); - assert_eq!(names, (vec![ColumnName::new(["stats", "numRecords"])])); + assert_eq!(names, (vec![column_name!("stats", NUM_RECORDS)])); assert_eq!(types, vec![DataType::LONG]); } diff --git a/kernel/src/scan/data_skipping.rs b/kernel/src/scan/data_skipping.rs index 22fc949f4b..8a40a3b7e8 100644 --- a/kernel/src/scan/data_skipping.rs +++ b/kernel/src/scan/data_skipping.rs @@ -1,20 +1,26 @@ use std::cmp::Ordering; use std::collections::HashSet; use std::sync::{Arc, LazyLock}; +use std::time::Instant; -use tracing::{debug, error, info}; +use tracing::{debug, error}; use crate::actions::visitors::SelectionVectorVisitor; +use crate::actions::{MAX_VALUES, MIN_VALUES, NULL_COUNT, NUM_RECORDS}; use crate::error::DeltaResult; use crate::expressions::{ - column_expr, column_name, joined_column_expr, BinaryPredicateOp, ColumnName, - Expression as Expr, ExpressionRef, JunctionPredicateOp, OpaquePredicateOpRef, - Predicate as Pred, PredicateRef, Scalar, + col, column_name, column_pred, lit, BinaryPredicateOp, CastOptions, ColumnName, + Expression as Expr, ExpressionRef, JunctionPredicateOp, MapToStructOptions, + OpaquePredicateOpRef, Predicate as Pred, PredicateRef, Scalar, }; use crate::kernel_predicates::{ DataSkippingPredicateEvaluator, KernelPredicateEvaluator, KernelPredicateEvaluatorDefaults, }; -use crate::schema::{DataType, SchemaRef, StructField, StructType}; +use crate::scan::data_skipping::stats_schema::is_skipping_eligible_datatype; +use crate::scan::log_replay::PARTITION_VALUES_PARSED_NAME; +use crate::scan::metrics::ScanMetrics; +use crate::schema::{DataType, PrimitiveType, SchemaRef, StructField, StructType}; +use crate::table_configuration::TableConfiguration; use crate::utils::require; use crate::{Engine, EngineData, Error, ExpressionEvaluator, PredicateEvaluator, RowVisitor as _}; @@ -42,37 +48,60 @@ use delta_kernel_derive::internal_api; /// predicate is dropped. #[cfg(test)] pub(crate) fn as_data_skipping_predicate(pred: &Pred) -> Option { - DataSkippingPredicateCreator::new(&Default::default()).eval(pred) + let stats_columns = all_referenced_columns(pred); + DataSkippingPredicateCreator::new(&Default::default(), &stats_columns).eval(pred) +} + +/// Permissive stats-columns set for tests that exercise rewrite mechanics, not the gate. +#[cfg(test)] +pub(crate) fn all_referenced_columns(pred: &Pred) -> HashSet { + pred.references().into_iter().cloned().collect() } #[cfg(test)] pub(crate) fn as_data_skipping_predicate_with_partitions( pred: &Pred, - partition_columns: &HashSet, + partition_columns: &HashSet, ) -> Option { - DataSkippingPredicateCreator::new(partition_columns).eval(pred) + let stats_columns = all_referenced_columns(pred); + DataSkippingPredicateCreator::new(partition_columns, &stats_columns).eval(pred) } -/// Like `as_data_skipping_predicate`, but invokes [`KernelPredicateEvaluator::eval_sql_where`] -/// instead of [`KernelPredicateEvaluator::eval`]. +/// Like [`as_data_skipping_predicate_with_partitions`] but invokes +/// [`KernelPredicateEvaluator::eval_sql_where`] instead of [`KernelPredicateEvaluator::eval`]. +#[cfg(test)] fn as_sql_data_skipping_predicate( pred: &Pred, - partition_columns: &HashSet, + partition_columns: &HashSet, ) -> Option { - DataSkippingPredicateCreator::new(partition_columns).eval_sql_where(pred) + let stats_columns = all_referenced_columns(pred); + as_sql_data_skipping_predicate_with_stats_columns(pred, partition_columns, &stats_columns) +} + +/// Like [`as_sql_data_skipping_predicate`] but only rewrites references to columns in +/// `stats_columns`; other columns return `None` from the `get_*_stat` methods and +/// junction-fold into NULL literals. +pub(crate) fn as_sql_data_skipping_predicate_with_stats_columns( + pred: &Pred, + partition_columns: &HashSet, + stats_columns: &HashSet, +) -> Option { + DataSkippingPredicateCreator::new(partition_columns, stats_columns).eval_sql_where(pred) } #[internal_api] pub(crate) struct DataSkippingFilter { /// Evaluator that extracts file-level statistics from the input batch. The caller provides /// the expression at construction time, which determines where stats come from: - /// - Scan path: `column_expr!("stats_parsed")` reads the already-parsed struct from - /// a transformed batch (where `add.*` fields are flattened to top-level columns). - /// - Table changes path: `Expression::parse_json(column_expr!("add.stats"), schema)` parses - /// JSON from a raw action batch (where stats are nested under `add.stats`). + /// - Scan path: `col!("stats_parsed")` reads the already-parsed struct from a transformed + /// batch (where `add.*` fields are flattened to top-level columns). + /// - Table changes path: `Expression::parse_json(col!("add.stats"), schema)` parses JSON from + /// a raw action batch (where stats are nested under `add.stats`). stats_evaluator: Arc, skipping_evaluator: Arc, filter_evaluator: Arc, + /// Metrics to record data skipping statistics. + metrics: Option>, } impl DataSkippingFilter { @@ -85,18 +114,28 @@ impl DataSkippingFilter { /// # Parameters /// - `engine`: Engine for creating evaluators /// - `predicate`: Optional predicate for data skipping - /// - `stats_schema`: The data stats schema (numRecords, nullCount, minValues, maxValues). - /// Pass `None` if no data stats are available. + /// - `stats_schema`: The data stats schema (numRecords, nullCount, minValues, maxValues). Pass + /// `None` if no data stats are available. /// - `stats_expr`: Expression to extract data stats from the batch, producing output matching - /// `stats_schema`. For example, `column_expr!("stats_parsed")` for pre-parsed stats, or - /// `Expression::parse_json(column_expr!("add.stats"), stats_schema)` for JSON parsing. + /// `stats_schema`. For example, `col!("stats_parsed")` for pre-parsed stats, or + /// `Expression::parse_json(col!("add.stats"), stats_schema)` for JSON parsing. /// - `partition_schema`: Schema of typed partition columns referenced by the predicate /// (physical names). Pass `None` if no partition columns are referenced. /// - `partition_expr`: Expression to extract partition values from the batch, producing output /// matching `partition_schema`. Typically a `MapToStruct` expression that converts the /// `partitionValues` string map into a typed struct. Only used when `partition_schema` is /// `Some`. + /// - `is_add_expr`: Boolean expression that is true for Add rows and false for Remove and other + /// non-file rows (e.g. `path IS NOT NULL` against a transformed batch, or `add.path IS NOT + /// NULL` against a raw action batch). Used to guard partition predicates and opaque-predicate + /// rewrites so non-Add rows are never filtered out. /// - `input_schema`: Schema of the batch that will be passed to [`apply()`](Self::apply) + /// - `stats_columns`: Physical leaf paths whose stats are in `stats_schema`. References to + /// other data columns fold to NULL (keeping the file). Must line up with `stats_schema` -- + /// otherwise the rewritten predicate references columns the unified schema doesn't have and + /// evaluator construction fails. + /// - `metrics`: Optional metrics to record data skipping statistics. + #[allow(clippy::too_many_arguments)] pub(crate) fn new( engine: &dyn Engine, predicate: Option, @@ -104,10 +143,19 @@ impl DataSkippingFilter { stats_expr: ExpressionRef, partition_schema: Option<&SchemaRef>, partition_expr: ExpressionRef, + is_add_expr: ExpressionRef, input_schema: SchemaRef, + stats_columns: &HashSet, + metrics: Option>, ) -> Option { static FILTER_PRED: LazyLock = - LazyLock::new(|| Arc::new(column_expr!("output").distinct(Expr::literal(false)))); + LazyLock::new(|| Arc::new(col!("output").distinct(lit(false)))); + static FILTER_SCHEMA: LazyLock = LazyLock::new(|| { + Arc::new(StructType::new_unchecked([StructField::nullable( + "output", + DataType::BOOLEAN, + )])) + }); let predicate = predicate?; debug!("Creating a data skipping filter for {:#?}", predicate); @@ -120,6 +168,7 @@ impl DataSkippingFilter { stats_expr, partition_schema, partition_expr, + is_add_expr, )?; let stats_evaluator = engine @@ -135,31 +184,34 @@ impl DataSkippingFilter { // Skipping happens in several steps: // // 1. The stats evaluator extracts file-level statistics from the batch (the expression - // provided by the caller determines how: reading a pre-parsed column, parsing JSON, etc.) + // provided by the caller determines how: reading a pre-parsed column, parsing JSON, + // etc.) // // 2. The predicate (skipping evaluator) produces false for any file whose stats prove we // can safely skip it. A value of true means the stats say we must keep the file, and // null means we could not determine whether the file is safe to skip, because its stats // were missing/null. // - // 3. The selection evaluator does DISTINCT(col(predicate), 'false') to produce true - // (= keep) when the predicate is true/null and false (= skip) when the predicate - // is false. + // 3. The selection evaluator does DISTINCT(col(predicate), 'false') to produce true (= + // keep) when the predicate is true/null and false (= skip) when the predicate is false. let skipping_evaluator = engine .evaluation_handler() .new_predicate_evaluator( unified_schema.clone(), - Arc::new(as_sql_data_skipping_predicate( + Arc::new(as_sql_data_skipping_predicate_with_stats_columns( &predicate, &partition_columns, + stats_columns, )?), ) .inspect_err(|e| error!("Failed to create skipping evaluator: {e}")) .ok()?; + // The filter evaluator operates on the skipping evaluator's output, which is a single + // boolean column named "output" (not the unified stats schema). let filter_evaluator = engine .evaluation_handler() - .new_predicate_evaluator(unified_schema, FILTER_PRED.clone()) + .new_predicate_evaluator(FILTER_SCHEMA.clone(), FILTER_PRED.clone()) .inspect_err(|e| error!("Failed to create filter evaluator: {e}")) .ok()?; @@ -167,9 +219,82 @@ impl DataSkippingFilter { stats_evaluator, skipping_evaluator, filter_evaluator, + metrics, }) } + /// Builds a filter over raw `{ add, remove, ... }` action batches, parsing file stats from + /// the JSON `add.stats` string. This is the shape used by log-replay paths that skip + /// *before* transforming actions (the change-data-feed path and the incremental scan), + /// unlike the scan path which reads pre-parsed `stats_parsed` from transformed batches. + /// + /// The stats schema is derived from the predicate's column references via + /// [`TableConfiguration::build_expected_stats_schemas`], matching the write side exactly; + /// references outside the table's stats columns fold to NULL (keeping the file). Partition + /// values are parsed from the raw `add.partitionValues` string map with + /// [`Expression::map_to_struct`], so predicates over partition columns prune too. + /// + /// Returns `None` (equivalent to keep-all) when the predicate is ineligible for data skipping + /// (references no stats columns, or fails evaluator construction). + /// + /// Pruning applies only to Add rows: Remove and cdc rows (null `add.path`) are never pruned, + /// guarded by the `add.path IS NOT NULL` `is_add` expression wired in below. Callers on the + /// change-data-feed path rely on this so tombstones survive a non-matching predicate. + /// + /// # Parameters + /// - `engine`: Engine for creating evaluators. + /// - `physical_predicate`: The predicate, already lowered to physical column names. + /// - `table_configuration`: Source of the stats schema, partition schema, and stats-column + /// gate. + /// - `input_schema`: Schema of the raw action batches passed to [`apply()`](Self::apply). + pub(crate) fn for_raw_action_batch( + engine: &dyn Engine, + physical_predicate: PredicateRef, + table_configuration: &TableConfiguration, + input_schema: SchemaRef, + ) -> Option { + // Predicate refs become the `requested_physical_columns` filter; refs outside + // `physical_stats_columns` fold to NULL via the gate below. Clustering columns are + // deliberately not required here (see the scan path and #2588): loading clustering + // domain metadata would require a separate scan of the log, which this filter avoids. + let predicate_refs: Vec = physical_predicate + .references() + .into_iter() + .cloned() + .collect(); + let physical_stats_columns = table_configuration.physical_stats_columns_set(None); + let physical_stats_schema = table_configuration + .build_expected_stats_schemas(None, Some(&predicate_refs)) + .ok()? + .physical; + let partition_schema = table_configuration.predicate_partition_schema(&predicate_refs); + + // Parse JSON stats from the raw action batch's `add.stats` column, parse partition values + // from the raw `add.partitionValues` string map, and identify Add rows by + // `add.path IS NOT NULL` (raw batches keep the nested layout). + let stats_expr = Arc::new(Expr::parse_json( + col!("add.stats"), + physical_stats_schema.clone(), + )); + let partition_expr = Arc::new(Expr::map_to_struct( + col!("add.partitionValues"), + MapToStructOptions::default(), + )); + let is_add_expr = Arc::new(Pred::is_not_null(col!("add.path")).into()); + Self::new( + engine, + Some(physical_predicate), + Some(&physical_stats_schema), + stats_expr, + partition_schema.as_ref(), + partition_expr, + is_add_expr, + input_schema, + &physical_stats_columns, + None, + ) + } + /// Builds the unified schema and extraction expression from separate data stats and partition /// value inputs. Returns `None` if neither stats nor partition values are available. /// @@ -186,31 +311,52 @@ impl DataSkippingFilter { stats_expr: ExpressionRef, physical_partition_schema: Option<&SchemaRef>, partition_expr: ExpressionRef, - ) -> Option<(SchemaRef, ExpressionRef, HashSet)> { - let partition_columns: HashSet = physical_partition_schema - .map(|s| s.fields().map(|f| f.name().to_string()).collect()) + is_add_expr: ExpressionRef, + ) -> Option<(SchemaRef, ExpressionRef, HashSet)> { + let partition_columns: HashSet = physical_partition_schema + .map(|s| { + s.fields() + // Intervals are ordered, but interval data skipping is not supported. + .filter(|f| { + matches!( + f.data_type(), + DataType::Primitive(primitive) + if is_skipping_eligible_datatype(primitive) + || matches!( + primitive, + PrimitiveType::Boolean | PrimitiveType::Binary + ) + ) + }) + .map(|f| ColumnName::new([f.name()])) + .collect() + }) .unwrap_or_default(); - let stats_field = |stats: &SchemaRef| { - StructField::nullable( - "stats_parsed", - DataType::Struct(Box::new(stats.as_ref().clone())), - ) - }; - let partition_field = |ps: &SchemaRef| { - StructField::nullable( - "partitionValues_parsed", - DataType::Struct(Box::new(ps.as_ref().clone())), - ) - }; + let stats_field = + |stats: &SchemaRef| StructField::nullable("stats_parsed", stats.as_ref().clone()); + let partition_field = + |ps: &SchemaRef| StructField::nullable("partitionValues_parsed", ps.as_ref().clone()); + let is_add_field = StructField::not_null("is_add", DataType::BOOLEAN); + // Always include an `is_add` boolean (extracted by the caller-provided `is_add_expr`, + // true for Add rows and false for Remove/non-file rows) so that predicates can guard + // against filtering Remove rows: partition predicates and opaque-predicate rewrites are + // wrapped with `OR(NOT is_add, ...)` (see `guard_for_removes`). let unified_schema = match (physical_stats_schema, physical_partition_schema) { (Some(stats), Some(ps)) => Arc::new(StructType::new_unchecked([ stats_field(stats), partition_field(ps), + is_add_field, + ])), + (Some(stats), None) => Arc::new(StructType::new_unchecked([ + stats_field(stats), + is_add_field, + ])), + (None, Some(ps)) => Arc::new(StructType::new_unchecked([ + partition_field(ps), + is_add_field, ])), - (Some(stats), None) => Arc::new(StructType::new_unchecked([stats_field(stats)])), - (None, Some(ps)) => Arc::new(StructType::new_unchecked([partition_field(ps)])), (None, None) => return None, }; @@ -218,9 +364,9 @@ impl DataSkippingFilter { physical_stats_schema.is_some(), physical_partition_schema.is_some(), ) { - (true, true) => Arc::new(Expr::struct_from([stats_expr, partition_expr])), - (true, false) => Arc::new(Expr::struct_from([stats_expr])), - (false, true) => Arc::new(Expr::struct_from([partition_expr])), + (true, true) => Arc::new(Expr::struct_from([stats_expr, partition_expr, is_add_expr])), + (true, false) => Arc::new(Expr::struct_from([stats_expr, is_add_expr])), + (false, true) => Arc::new(Expr::struct_from([partition_expr, is_add_expr])), (false, false) => return None, }; @@ -230,10 +376,10 @@ impl DataSkippingFilter { /// Apply the DataSkippingFilter to an EngineData batch. Returns a selection vector /// which can be applied to the batch to find rows that passed data skipping. pub(crate) fn apply(&self, batch: &dyn EngineData) -> DeltaResult> { + let start_time = Instant::now(); let batch_len = batch.len(); let file_stats = self.stats_evaluator.evaluate(batch)?; - debug_assert_eq!(file_stats.len(), batch_len); require!( file_stats.len() == batch_len, Error::internal_error(format!( @@ -244,7 +390,6 @@ impl DataSkippingFilter { ); let skipping_predicate = self.skipping_evaluator.evaluate(&*file_stats)?; - debug_assert_eq!(skipping_predicate.len(), batch_len); require!( skipping_predicate.len() == batch_len, Error::internal_error(format!( @@ -270,37 +415,47 @@ impl DataSkippingFilter { let mut visitor = SelectionVectorVisitor::default(); visitor.visit_rows_of(selection_vector.as_ref())?; - let skipped = visitor - .selection_vector - .iter() - .filter(|&&kept| !kept) - .count(); - if skipped > 0 { - info!("data skipping filtered {skipped}/{batch_len} rows from batch",); + if visitor.num_filtered > 0 { + debug!( + "data skipping filtered {}/{batch_len} rows from batch", + visitor.num_filtered + ); + } + + if let Some(metrics) = self.metrics.as_ref() { + metrics.add_predicate_filtered(visitor.num_filtered); + metrics.add_predicate_eval_time_ns(start_time.elapsed().as_nanos() as u64) } Ok(visitor.selection_vector) } } -/// Rewrites a predicate for parquet row group skipping in checkpoint/sidecar files. -/// Returns `None` if the predicate is not eligible for data skipping. +/// Rewrites `pred` for scan checkpoint and sidecar Add row-group skipping. Data references become +/// IS NULL guarded `stats_parsed.{minValues,maxValues,nullCount}.

    ` comparisons; eligible +/// partition references become exact, unguarded `partitionValues_parsed.` values. Checkpoint +/// Removes are irrelevant to scan replay, so partition predicates may prune their null add-side +/// values. A bare unsupported predicate returns `None`; unsupported junction arms become NULL +/// literals to preserve three-valued logic. /// -/// Adds IS NULL guards on each stat column reference so the parquet RowGroupFilter -/// conservatively keeps row groups containing files with missing stats (null stat values -/// are invisible to footer min/max). For example, `col_a > 100` becomes: -/// ```text -/// OR(maxValues.col_a IS NULL, maxValues.col_a > 100) -/// ``` -/// -/// Partition columns are excluded since their values live in `add.partitionValues_parsed`, -/// not `add.stats_parsed`. +/// `physical_partition_columns` may be narrowed to the predicate's references; pass an empty set +/// for unpartitioned tables. `physical_floating_partition_columns` identifies FLOAT and DOUBLE +/// partitions whose parquet min/max may omit NaNs. `physical_stats_columns` is the table-level +/// stats membership set; references outside it fold to NULL (keeping the file). pub(crate) fn as_checkpoint_skipping_predicate( pred: &Pred, - partition_columns: &[String], + physical_partition_columns: &HashSet, + physical_floating_partition_columns: &HashSet, + physical_stats_columns: &HashSet, ) -> Option { - let partition_columns: HashSet<&str> = partition_columns.iter().map(String::as_str).collect(); - NullGuardedDataSkippingPredicateCreator { partition_columns }.eval(pred) + CheckpointDataSkippingPredicateCreator { + data_skipping_columns: DataSkippingColumns { + physical_partition_columns, + physical_stats_columns, + }, + physical_floating_partition_columns, + } + .eval(pred) } /// Maps an ordering and inversion flag to the corresponding comparison predicate. @@ -333,13 +488,127 @@ fn collect_junction_preds( Some(pred) => Some(pred), None => keep_null.then(|| { keep_null = false; - Pred::null_literal() + Pred::NULL }), }) .collect(); Pred::junction(op, preds) } +/// Adjusts a comparison value before comparing against a max stat, to account for the Delta +/// protocol allowing timestamp stats to be truncated to millisecond precision (see Per-file +/// Statistics: "Timestamp columns are truncated down to milliseconds"). Truncation floors to +/// the nearest millisecond, so: `stored_max <= actual_max <= stored_max + 999us`. We subtract +/// 999us from the comparison value to avoid incorrectly pruning files whose actual max may be +/// higher than the stored (truncated) max. +/// +/// For example, if a file's actual max is `4_000_500us` (4.000500s), Spark truncates the +/// stored max stat to `4_000_000us` (4.000s). A predicate `ts > 4_000_400` would incorrectly +/// prune this file by comparing against the truncated max. By adjusting the comparison value +/// to `4_000_400 - 999 = 3_999_401`, we ensure the file is kept. +/// +/// Non-timestamp values pass through unchanged. +fn adjust_scalar_for_max_stat_truncation(val: &Scalar) -> Scalar { + match val { + Scalar::Timestamp(micros) => Scalar::Timestamp(micros.saturating_sub(999)), + Scalar::TimestampNtz(micros) => Scalar::TimestampNtz(micros.saturating_sub(999)), + other => other.clone(), + } +} + +/// The `partitionValues_parsed.` reference holding a partition column's exact value, which +/// serves as both its min and max stat. `col` is a top-level physical partition name. +fn partition_value_expr(col: &ColumnName) -> Expr { + Expr::from(column_name!(PARTITION_VALUES_PARSED_NAME).join(col)) +} + +/// Whether a rewritten stat expression references `partitionValues_parsed`. +fn is_partition_value_reference(expr: &Expr) -> bool { + matches!(expr, Expr::Column(name) + if name.path().first().is_some_and(|f| f == PARTITION_VALUES_PARSED_NAME)) +} + +/// A column carries min/max stats iff it's a primitive whose type supports min/max skipping. +/// Boolean / Binary, Array, Map, and Variant leaves carry nullCount only. Struct columns +/// have no per-struct stats; only their primitive leaves do, recursively. +/// Must match `MinMaxStatsTransform`'s acceptance rule. Otherwise the predicate creator +/// emits refs to min/max fields the stats schema doesn't contain. +fn has_min_max_stats(data_type: &DataType) -> bool { + matches!(data_type, DataType::Primitive(ptype) if is_skipping_eligible_datatype(ptype)) +} + +/// Column metadata shared by the data-skipping predicate creators. +struct DataSkippingColumns<'a> { + /// Physical names of partition columns (always single-segment: Delta partition columns are + /// top-level). Stats for these come from `partitionValues_parsed.` (exact values) instead + /// of min/max ranges; each creator applies its own partition policy. + physical_partition_columns: &'a HashSet, + /// Physical leaf paths whose stats are present in `stats_parsed` (honors + /// `delta.dataSkippingNumIndexedCols`, `delta.dataSkippingStatsColumns`, and required + /// columns). Must match the column set used to build `physical_stats_schema`; otherwise the + /// rewritten predicate references columns absent from the unified schema. + physical_stats_columns: &'a HashSet, +} + +impl DataSkippingColumns<'_> { + fn is_partition_column(&self, col: &ColumnName) -> bool { + self.physical_partition_columns.contains(col) + } + + fn is_stats_column(&self, col: &ColumnName) -> bool { + self.physical_stats_columns.contains(col) + } + + /// Min stat for `col`: the exact `partitionValues_parsed.` value for partition columns + /// (it serves as both min and max), else `stats_parsed.minValues.` (`None` when unindexed + /// or not min/max-eligible). + fn min_stat(&self, col: &ColumnName, data_type: &DataType) -> Option { + if self.is_partition_column(col) { + Some(partition_value_expr(col)) + } else { + (self.is_stats_column(col) && has_min_max_stats(data_type)) + .then(|| Expr::from(column_name!("stats_parsed", MIN_VALUES).join(col))) + } + } + + /// Max stat for `col`: the exact `partitionValues_parsed.` value for partition columns, + /// else `stats_parsed.maxValues.` (`None` when unindexed or not min/max-eligible). + fn max_stat(&self, col: &ColumnName, data_type: &DataType) -> Option { + if self.is_partition_column(col) { + Some(partition_value_expr(col)) + } else { + (self.is_stats_column(col) && has_min_max_stats(data_type)) + .then(|| Expr::from(column_name!("stats_parsed", MAX_VALUES).join(col))) + } + } + + /// The comparison value for a max-stat check. Data-column max stats may be JSON-truncated, so + /// the value is widened by [`adjust_scalar_for_max_stat_truncation`]; partition values are + /// exact and pass through unchanged. + fn max_stat_comparison_value(&self, col: &ColumnName, val: &Scalar) -> Scalar { + if self.is_partition_column(col) { + val.clone() + } else { + adjust_scalar_for_max_stat_truncation(val) + } + } + + /// Null count stat for `col`: `None` for partition columns (their values are exact, not + /// aggregated) and for data columns outside the stats set, else `stats_parsed.nullCount.`. + fn nullcount_stat(&self, col: &ColumnName) -> Option { + if self.is_partition_column(col) { + None + } else { + self.is_stats_column(col) + .then(|| Expr::from(column_name!("stats_parsed", NULL_COUNT).join(col))) + } + } + + fn rowcount_stat(&self) -> Expr { + col!("stats_parsed", NUM_RECORDS) + } +} + /// Rewrites user predicates into stats-based predicates for data skipping. /// /// For data columns, rewrites to `stats_parsed.minValues.*`/`stats_parsed.maxValues.*`/ @@ -347,19 +616,29 @@ fn collect_junction_preds( /// For partition columns, rewrites to `partitionValues_parsed.*` since the partition value is /// the exact value for every row in the file (serving as both min and max). struct DataSkippingPredicateCreator<'a> { - /// Physical names of partition columns. For these columns, stats come from - /// `partitionValues.` (exact values) instead of min/max ranges. - partition_columns: &'a HashSet, + data_skipping_columns: DataSkippingColumns<'a>, } impl<'a> DataSkippingPredicateCreator<'a> { - fn new(partition_columns: &'a HashSet) -> Self { - Self { partition_columns } + fn new( + physical_partition_columns: &'a HashSet, + physical_stats_columns: &'a HashSet, + ) -> Self { + Self { + data_skipping_columns: DataSkippingColumns { + physical_partition_columns, + physical_stats_columns, + }, + } } - fn is_partition_column(&self, col: &ColumnName) -> bool { - let path = col.path(); - path.len() == 1 && self.partition_columns.contains(path[0].as_str()) + /// Wraps a predicate with `OR(NOT is_add, pred)` to protect Remove rows from being + /// filtered. Used for partition predicates (Remove rows have null add-side partition + /// values, which would incorrectly evaluate to false) and for opaque-predicate rewrites + /// (op-computed verdicts bypass kernel's null-stats folding). The `is_add` column ensures + /// non-Add rows always pass the filter. + fn guard_for_removes(&self, pred: Pred) -> Pred { + Pred::or(Pred::not(column_pred!("is_add")), pred) } } @@ -367,45 +646,38 @@ impl DataSkippingPredicateEvaluator for DataSkippingPredicateCreator<'_> { type Output = Pred; type ColumnStat = Expr; - /// Retrieves the minimum value of a column. For partition columns, returns the exact - /// partition value (which serves as both min and max). - fn get_min_stat(&self, col: &ColumnName, _data_type: &DataType) -> Option { - if self.is_partition_column(col) { - Some(joined_column_expr!("partitionValues_parsed", col)) - } else { - Some(Expr::from(column_name!("stats_parsed.minValues").join(col))) - } + fn get_min_stat(&self, col: &ColumnName, data_type: &DataType) -> Option { + self.data_skipping_columns.min_stat(col, data_type) } - /// Retrieves the maximum value of a column. For partition columns, returns the exact - /// partition value. For data columns, excludes timestamps due to millisecond truncation. - // TODO(#1002): we currently don't support file skipping on timestamp columns' max stat since - // they are truncated to milliseconds in add.stats. fn get_max_stat(&self, col: &ColumnName, data_type: &DataType) -> Option { - if self.is_partition_column(col) { - Some(joined_column_expr!("partitionValues_parsed", col)) - } else { - match data_type { - &DataType::TIMESTAMP | &DataType::TIMESTAMP_NTZ => None, - _ => Some(Expr::from(column_name!("stats_parsed.maxValues").join(col))), - } - } + self.data_skipping_columns.max_stat(col, data_type) + } + + fn partial_cmp_max_stat( + &self, + col: &ColumnName, + val: &Scalar, + ord: Ordering, + inverted: bool, + ) -> Option { + let max = self.get_max_stat(col, &val.data_type())?; + let adjusted = self + .data_skipping_columns + .max_stat_comparison_value(col, val); + self.eval_partial_cmp(ord, max, &adjusted, inverted) } - /// Retrieves the null count of a column. Partition columns don't have nullCount stats. fn get_nullcount_stat(&self, col: &ColumnName) -> Option { - if self.is_partition_column(col) { - None - } else { - Some(Expr::from(column_name!("stats_parsed.nullCount").join(col))) - } + self.data_skipping_columns.nullcount_stat(col) } - /// Retrieves the row count statistic. fn get_rowcount_stat(&self) -> Option { - Some(column_expr!("stats_parsed.numRecords")) + Some(self.data_skipping_columns.rowcount_stat()) } + /// For partition columns, wraps the comparison with `OR(NOT is_add, comparison)` so that + /// Remove rows (which have null add-side partition values) are never filtered. fn eval_partial_cmp( &self, ord: Ordering, @@ -413,7 +685,14 @@ impl DataSkippingPredicateEvaluator for DataSkippingPredicateCreator<'_> { val: &Scalar, inverted: bool, ) -> Option { - Some(comparison_predicate(ord, col, val, inverted)) + // Detect partition columns by the prefix set in get_min_stat/get_max_stat. + let is_partition = is_partition_value_reference(&col); + let cmp = comparison_predicate(ord, col, val, inverted); + Some(if is_partition { + self.guard_for_removes(cmp) + } else { + cmp + }) } fn eval_pred_scalar(&self, val: &Scalar, inverted: bool) -> Option { @@ -424,16 +703,18 @@ impl DataSkippingPredicateEvaluator for DataSkippingPredicateCreator<'_> { KernelPredicateEvaluatorDefaults::eval_pred_scalar_is_null(val, inverted).map(Pred::literal) } - /// For partition columns, checks `partitionValues_parsed.IS [NOT] NULL` directly. + /// For partition columns, checks `partitionValues_parsed.IS [NOT] NULL` directly, + /// wrapped with `OR(NOT is_add, ...)` to protect Remove rows from being filtered. /// For data columns, uses nullCount stats. fn eval_pred_is_null(&self, col: &ColumnName, inverted: bool) -> Option { - if self.is_partition_column(col) { - let pv_expr = joined_column_expr!("partitionValues_parsed", col); - if inverted { - Some(Pred::is_not_null(pv_expr)) + if self.data_skipping_columns.is_partition_column(col) { + let pv_expr = partition_value_expr(col); + let pred = if inverted { + Pred::is_not_null(pv_expr) } else { - Some(Pred::is_null(pv_expr)) - } + Pred::is_null(pv_expr) + }; + Some(self.guard_for_removes(pred)) } else { let safe_to_skip = match inverted { true => self.get_rowcount_stat()?, // all-null @@ -454,13 +735,32 @@ impl DataSkippingPredicateEvaluator for DataSkippingPredicateCreator<'_> { .map(Pred::literal) } + // TODO(#3011): Rewrite partition CASTs over exact values through the engine evaluator. + fn eval_pred_cast( + &self, + _op: BinaryPredicateOp, + _col: &ColumnName, + _target: &DataType, + _options: &CastOptions, + _val: &Scalar, + _inverted: bool, + ) -> Option { + None + } + + /// Rewrites an opaque predicate via its `as_data_skipping_predicate`, wrapped with + /// `OR(NOT is_add, ...)`. Opaque rewrites may embed op-computed verdicts (e.g. an engine + /// callback behind FFI) that bypass kernel's null-stats folding, so kernel itself must + /// guarantee that non-Add rows (Removes and other actions, which carry null stats) are never + /// filtered -- dropping a Remove from log replay would resurrect a deleted file. fn eval_pred_opaque( &self, op: &OpaquePredicateOpRef, exprs: &[Expr], inverted: bool, ) -> Option { - op.as_data_skipping_predicate(self, exprs, inverted) + let pred = op.as_data_skipping_predicate(self, exprs, inverted)?; + Some(self.guard_for_removes(pred)) } fn finish_eval_pred_junction( @@ -473,68 +773,68 @@ impl DataSkippingPredicateEvaluator for DataSkippingPredicateCreator<'_> { } } -/// Like [`DataSkippingPredicateCreator`] but adds IS NULL guards on stat column references -/// for safe parquet row group filtering. Partition columns are excluded since their values -/// live in `add.partitionValues_parsed`, not `add.stats_parsed`. -struct NullGuardedDataSkippingPredicateCreator<'a> { - partition_columns: HashSet<&'a str>, +struct CheckpointDataSkippingPredicateCreator<'a> { + data_skipping_columns: DataSkippingColumns<'a>, + physical_floating_partition_columns: &'a HashSet, } -impl NullGuardedDataSkippingPredicateCreator<'_> { - /// Returns true if the column is a partition column (no stats in `stats_parsed`). - fn is_partition_column(&self, col: &ColumnName) -> bool { - let path = col.path(); - path.len() == 1 && self.partition_columns.contains(path[0].as_str()) +impl CheckpointDataSkippingPredicateCreator<'_> { + fn partition_min_max_may_omit_values(&self, col: &ColumnName) -> bool { + self.physical_floating_partition_columns.contains(col) } } -impl DataSkippingPredicateEvaluator for NullGuardedDataSkippingPredicateCreator<'_> { +impl DataSkippingPredicateEvaluator for CheckpointDataSkippingPredicateCreator<'_> { type Output = Pred; type ColumnStat = Expr; - // These stat methods produce unprefixed column references (e.g. `minValues.col`) because - // the checkpoint skipping path applies its own `add.stats_parsed` prefix afterward. - // Partition columns return None since their values live in `add.partitionValues_parsed`. + // Stat selection and max-stat truncation are shared through `DataSkippingColumns`. This + // creator also limits footer-ineligible columns and predicate shapes while applying + // checkpoint-specific null guards. - fn get_min_stat(&self, col: &ColumnName, _data_type: &DataType) -> Option { - if self.is_partition_column(col) { + fn get_min_stat(&self, col: &ColumnName, data_type: &DataType) -> Option { + // Parquet footer min/max exclude NaNs, so they cannot bound every floating partition value. + if self.partition_min_max_may_omit_values(col) { return None; } - Some(joined_column_expr!("minValues", col)) + self.data_skipping_columns.min_stat(col, data_type) } fn get_max_stat(&self, col: &ColumnName, data_type: &DataType) -> Option { - if self.is_partition_column(col) { + if self.partition_min_max_may_omit_values(col) { return None; } - match data_type { - &DataType::TIMESTAMP | &DataType::TIMESTAMP_NTZ => None, - _ => Some(joined_column_expr!("maxValues", col)), - } + self.data_skipping_columns.max_stat(col, data_type) } fn get_nullcount_stat(&self, col: &ColumnName) -> Option { - if self.is_partition_column(col) { - return None; - } - Some(joined_column_expr!("nullCount", col)) + self.data_skipping_columns.nullcount_stat(col) } fn get_rowcount_stat(&self) -> Option { - Some(column_expr!("numRecords")) + Some(self.data_skipping_columns.rowcount_stat()) } - /// Wraps a stat column comparison with an IS NULL guard. - /// - /// `col > 100` → `OR(maxValues.col IS NULL, maxValues.col > 100)` - /// - /// `col = 100` (calls this twice, once per stat): - /// ```text - /// AND( - /// OR(minValues.col IS NULL, minValues.col <= 100), - /// OR(maxValues.col IS NULL, maxValues.col >= 100) - /// ) - /// ``` + fn partial_cmp_max_stat( + &self, + col: &ColumnName, + val: &Scalar, + ord: Ordering, + inverted: bool, + ) -> Option { + let max = self.get_max_stat(col, &val.data_type())?; + let adjusted = self + .data_skipping_columns + .max_stat_comparison_value(col, val); + self.eval_partial_cmp(ord, max, &adjusted, inverted) + } + + /// Wraps a data-stat comparison with an IS NULL guard, so a missing stat keeps the row group. + /// `col > 100` becomes + /// `OR(stats_parsed.maxValues.col IS NULL, stats_parsed.maxValues.col > 100)`. + /// Partition comparisons reference the typed per-Add value without a null guard. The parquet + /// filter evaluates that column through row-group footer bounds; null-only groups remain + /// unknown while mixed groups may be pruned from their non-null bounds. fn eval_partial_cmp( &self, ord: Ordering, @@ -543,31 +843,59 @@ impl DataSkippingPredicateEvaluator for NullGuardedDataSkippingPredicateCreator< inverted: bool, ) -> Option { let comparison = comparison_predicate(ord, col.clone(), val, inverted); - Some(Pred::or(Pred::is_null(col), comparison)) + Some(if is_partition_value_reference(&col) { + comparison + } else { + Pred::or(Pred::is_null(col), comparison) + }) } - /// No guard needed — no stat column reference. `TRUE` → `Some(true)`. fn eval_pred_scalar(&self, val: &Scalar, inverted: bool) -> Option { KernelPredicateEvaluatorDefaults::eval_pred_scalar(val, inverted).map(Pred::literal) } - /// No guard needed — no stat column reference. `NULL IS NULL` → `Some(true)`. fn eval_pred_scalar_is_null(&self, val: &Scalar, inverted: bool) -> Option { KernelPredicateEvaluatorDefaults::eval_pred_scalar_is_null(val, inverted).map(Pred::literal) } - /// IS NULL guard on nullCount stat. - /// - /// `IS NULL` → `OR(nullCount.col IS NULL, nullCount.col != 0)`: - /// column vs literal — RowGroupFilter can evaluate via footer stats. - /// - /// `IS NOT NULL` → returns `None`. The unguarded version produces - /// `nullCount.col != numRecords`, which is column vs column. The RowGroupFilter can - /// only resolve one column at a time, so it can never prune with this predicate. + /// Rewrites casts over exact per-Add partition values. + fn eval_pred_cast( + &self, + op: BinaryPredicateOp, + col: &ColumnName, + target: &DataType, + options: &CastOptions, + val: &Scalar, + inverted: bool, + ) -> Option { + if !self.data_skipping_columns.is_partition_column(col) { + return None; + } + let ord = match op { + BinaryPredicateOp::LessThan => Ordering::Less, + BinaryPredicateOp::Equal => Ordering::Equal, + BinaryPredicateOp::GreaterThan => Ordering::Greater, + BinaryPredicateOp::Distinct | BinaryPredicateOp::In => return None, + }; + let cast = Expr::cast(partition_value_expr(col), target.clone(), options.clone()); + Some(comparison_predicate(ord, cast, val, inverted)) + } + + /// Partition NULL checks use the exact parsed value. Data `IS NULL` uses a guarded null count; + /// data `IS NOT NULL` remains unsupported because row-group filtering cannot compare it with + /// `numRecords`. // TODO(#1873): IS NOT NULL pruning requires cross-column range comparison in RowGroupFilter. // Skippable when the nullCount and numRecords ranges don't overlap (e.g. nullCount in // [0, 0] vs numRecords in [500, 2000] proves all files have non-null values). fn eval_pred_is_null(&self, col: &ColumnName, inverted: bool) -> Option { + if self.data_skipping_columns.is_partition_column(col) { + let partition_value = partition_value_expr(col); + return Some(if inverted { + Pred::is_not_null(partition_value) + } else { + Pred::is_null(partition_value) + }); + } if inverted { return None; // IS NOT NULL: column vs column, can't prune (#1873) } @@ -576,7 +904,6 @@ impl DataSkippingPredicateEvaluator for NullGuardedDataSkippingPredicateCreator< Some(Pred::or(Pred::is_null(nullcount), comparison)) } - /// No guard needed — no stat column reference. `5 < 10` → `Some(true)`. fn eval_pred_binary_scalars( &self, op: BinaryPredicateOp, @@ -588,9 +915,9 @@ impl DataSkippingPredicateEvaluator for NullGuardedDataSkippingPredicateCreator< .map(Pred::literal) } - /// Unsupported. Opaque predicates can construct stat column references directly, - /// bypassing IS NULL guards and risking false pruning. Returns `None` to conservatively - /// drop these from the skipping predicate. + /// Unsupported. Opaque predicates can construct data-stat references directly, bypassing IS + /// NULL guards and risking false pruning when a live Add has missing stats. Returns `None` to + /// conservatively drop these from the skipping predicate. fn eval_pred_opaque( &self, _op: &OpaquePredicateOpRef, @@ -600,11 +927,11 @@ impl DataSkippingPredicateEvaluator for NullGuardedDataSkippingPredicateCreator< None } - /// Combines sub-predicates with AND/OR. `col_a > 100 AND col_b < 50` → + /// Combines sub-predicates with AND/OR. `col_a > 100 AND col_b < 50` becomes /// ```text /// AND( - /// OR(maxValues.col_a IS NULL, maxValues.col_a > 100), - /// OR(minValues.col_b IS NULL, minValues.col_b < 50) + /// OR(stats_parsed.maxValues.col_a IS NULL, stats_parsed.maxValues.col_a > 100), + /// OR(stats_parsed.minValues.col_b IS NULL, stats_parsed.minValues.col_b < 50) /// ) /// ``` fn finish_eval_pred_junction( diff --git a/kernel/src/scan/data_skipping/stats_schema/column_filter.rs b/kernel/src/scan/data_skipping/stats_schema/column_filter.rs index 0c799d8fb4..2bba997a2c 100644 --- a/kernel/src/scan/data_skipping/stats_schema/column_filter.rs +++ b/kernel/src/scan/data_skipping/stats_schema/column_filter.rs @@ -3,11 +3,9 @@ //! This module contains [`StatsColumnFilter`], which determines which columns //! should have statistics collected based on table configuration. -use crate::{ - column_trie::ColumnTrie, - schema::{ColumnName, DataType, Schema, StructField}, - table_properties::DataSkippingNumIndexedCols, -}; +use crate::column_trie::ColumnTrie; +use crate::schema::{ColumnName, DataType, Schema, StructField}; +use crate::table_properties::DataSkippingNumIndexedCols; /// Configuration for statistics columns pub(crate) struct StatsConfig<'a> { @@ -15,8 +13,8 @@ pub(crate) struct StatsConfig<'a> { /// `data_skipping_num_indexed_cols`. /// See delta.dataSkippingStatsColumns in the Delta protocol for more details. pub(crate) data_skipping_stats_columns: Option<&'a [ColumnName]>, - /// Maximum number of leaf columns to include. Ignored when `data_skipping_stats_columns` is set. - /// See delta.dataSkippingNumIndexedCols in the Delta protocol for more details. + /// Maximum number of leaf columns to include. Ignored when `data_skipping_stats_columns` is + /// set. See delta.dataSkippingNumIndexedCols in the Delta protocol for more details. pub(crate) data_skipping_num_indexed_cols: Option, } @@ -139,7 +137,7 @@ impl<'col> StatsColumnFilter<'col> { continue; } // Verify the required column exists in schema before adding - if schema.walk_column_fields(col).is_ok() { + if schema.field_at(col).is_ok() { tracing::warn!( "Required column '{}' exceeds dataSkippingNumIndexedCols limit; \ adding anyway", @@ -170,16 +168,13 @@ impl<'col> StatsColumnFilter<'col> { /// Returns true if the current path should be included based on table-level filtering config. /// Required columns (e.g. clustering columns) are always included, even past the column limit. pub(crate) fn should_include_for_table(&self) -> bool { - // When using dataSkippingStatsColumns, check the trie (which includes required) - if let Some(trie) = &self.data_skipping_stats_trie { - return trie.contains_prefix_of(&self.path); - } - - // When using dataSkippingNumIndexedCols, check limit but allow required columns - if self.at_column_limit() { - self.is_required_column() - } else { - true + match &self.data_skipping_stats_trie { + // In explicit dataSkippingStatsColumns mode, include exactly columns selected by the + // trie. Required columns are already merged into the trie during + // initialization. + Some(trie) => trie.contains_prefix_of(&self.path), + // In count-based mode, include until limit; required columns can exceed the limit. + None => !self.at_column_limit() || self.is_required_column(), } } @@ -232,8 +227,9 @@ impl<'col> StatsColumnFilter<'col> { self.collect_field(child, result); } } - // Map, Array, and Variant types are not eligible for statistics collection. - DataType::Map(_) | DataType::Array(_) | DataType::Variant(_) => {} + // All non-struct types are leaf columns for stats purposes: they count against + // the column limit and are included in nullCount. Array, Map, and Variant are + // excluded from min/max by MinMaxStatsTransform. _ => { if self.should_include_for_table() { result.push(ColumnName::new(&self.path)); @@ -249,7 +245,9 @@ impl<'col> StatsColumnFilter<'col> { #[cfg(test)] mod tests { use super::*; - use crate::{schema::StructType, table_properties::TableProperties}; + use crate::expressions::column_name; + use crate::schema::StructType; + use crate::table_properties::TableProperties; fn make_props_with_num_cols(n: u64) -> TableProperties { [( @@ -361,7 +359,7 @@ mod tests { let props = make_props_with_num_cols(2); // Required column is deeply nested: user.address.city - let required_cols = vec![ColumnName::new(["user", "address", "city"])]; + let required_cols = vec![column_name!("user.address.city")]; let address_struct = StructType::new_unchecked([ StructField::nullable("street", DataType::STRING), @@ -370,7 +368,7 @@ mod tests { ]); let user_struct = StructType::new_unchecked([ StructField::nullable("name", DataType::STRING), - StructField::nullable("address", DataType::Struct(Box::new(address_struct))), + StructField::nullable("address", address_struct), ]); let other_struct = StructType::new_unchecked([ StructField::nullable("foo", DataType::STRING), @@ -380,8 +378,8 @@ mod tests { let schema = StructType::new_unchecked([ StructField::nullable("id", DataType::LONG), StructField::nullable("name", DataType::STRING), - StructField::nullable("user", DataType::Struct(Box::new(user_struct))), - StructField::nullable("other", DataType::Struct(Box::new(other_struct))), + StructField::nullable("user", user_struct), + StructField::nullable("other", other_struct), StructField::nullable("extra1", DataType::STRING), StructField::nullable("extra2", DataType::STRING), ]); @@ -392,9 +390,9 @@ mod tests { assert_eq!( columns, vec![ - ColumnName::new(["id"]), - ColumnName::new(["name"]), - ColumnName::new(["user", "address", "city"]), + column_name!("id"), + column_name!("name"), + column_name!("user.address.city"), ] ); } @@ -403,15 +401,12 @@ mod tests { fn test_required_column_not_in_schema() { // Required column that doesn't exist in schema should be silently ignored let props = make_props_with_num_cols(2); - let required_cols = vec![ColumnName::new(["nonexistent", "column"])]; + let required_cols = vec![column_name!("nonexistent.column")]; let schema = abc_schema(); let columns = collect_stats_columns(&props, Some(&required_cols), &schema); // Should only include normal columns, required column not found - assert_eq!( - columns, - vec![ColumnName::new(["a"]), ColumnName::new(["b"]),] - ); + assert_eq!(columns, vec![column_name!("a"), column_name!("b"),]); } } diff --git a/kernel/src/scan/data_skipping/stats_schema/mod.rs b/kernel/src/scan/data_skipping/stats_schema/mod.rs index d52199419d..71a5764aa6 100644 --- a/kernel/src/scan/data_skipping/stats_schema/mod.rs +++ b/kernel/src/scan/data_skipping/stats_schema/mod.rs @@ -3,18 +3,17 @@ mod column_filter; use std::borrow::Cow; -use std::sync::Arc; - -use crate::schema::{ - ArrayType, ColumnName, DataType, MapType, PrimitiveType, Schema, SchemaRef, StructField, - StructType, -}; -use crate::transforms::SchemaTransform; -use crate::{DeltaResult, Error}; use column_filter::StatsColumnFilter; pub(crate) use column_filter::StatsConfig; +use crate::actions::{MAX_VALUES, MIN_VALUES, NULL_COUNT, NUM_RECORDS, TIGHT_BOUNDS}; +use crate::schema::{ + ArrayType, ColumnName, DataType, MapType, PrimitiveType, Schema, StructField, StructType, +}; +use crate::transforms::{transform_output_type, SchemaTransform}; +use crate::DeltaResult; + /// Generates the expected schema for file statistics. /// /// The base stats schema is dependent on the current table configuration and derived via: @@ -28,10 +27,11 @@ pub(crate) use column_filter::StatsConfig; /// It tracks the count of null values for each column. All leaf fields from the base schema /// are converted to LONG type (since null counts are always integers). /// -/// Note: Map, Array, and Variant types are excluded from statistics entirely (including -/// `nullCount`) as they are not eligible for data skipping. The `nullCount` schema includes -/// primitive types that aren't eligible for min/max (e.g., Boolean, Binary) since null counts -/// are still meaningful for those types. +/// Note: Array, Map, and Variant types are included in `nullCount` (null counts are meaningful +/// for these types) but excluded from `minValues`/`maxValues` (not eligible for data skipping). +/// They count as leaf columns against the indexed column limit. The `nullCount` schema also +/// includes primitive types that aren't eligible for min/max (e.g., Boolean, Binary) since null +/// counts are still meaningful for those types. /// /// The `minValues`/`maxValues` struct fields are also nested structures mirroring the table's /// column hierarchy. They additionally filter out leaf fields with non-eligible data types @@ -48,11 +48,11 @@ pub(crate) use column_filter::StatsConfig; /// entry in `minValues`/`maxValues`. The `nullCount` entry is still present and equals /// `numRecords`. /// - String min/max values must be truncated to a prefix no longer than 32 characters. For min -/// values, simple prefix truncation is valid (the truncated value is always <= the original). -/// For max values, a tie-breaker character must be appended after truncation to ensure the -/// result is >= all actual values: ASCII DEL (0x7F) when the truncated character is ASCII, -/// or U+10FFFF otherwise. If a valid truncation point cannot be found within 64 characters, -/// the max value is omitted (returning `None`). +/// values, simple prefix truncation is valid (the truncated value is always <= the original). For +/// max values, a tie-breaker character must be appended after truncation to ensure the result is +/// greater than or equal to all actual values: ASCII DEL (0x7F) when the truncated character is +/// ASCII, or U+10FFFF otherwise. If a valid truncation point cannot be found within 64 +/// characters, the max value is omitted (returning `None`). /// - Binary min/max values are not collected (Binary is not eligible for data skipping). /// - Boolean values are not eligible for min/max statistics but do have `nullCount`. /// @@ -124,11 +124,10 @@ pub(crate) use column_filter::StatsConfig; /// /// - `data_schema`: The table's data schema (partition columns excluded). /// - `config`: Stats configuration controlling which columns are included. -/// - `required_columns`: Columns that must always be included in statistics (write path). -/// Per the Delta protocol, clustering columns must have statistics regardless of table -/// property settings. -/// - `requested_columns`: Filter output to only these columns (read path). If specified, -/// only columns that also pass the `config` filtering will be included. +/// - `required_columns`: Columns that must always be included in statistics (write path). Per the +/// Delta protocol, clustering columns must have statistics regardless of table property settings. +/// - `requested_columns`: Filter output to only these columns (read path). If specified, only +/// columns that also pass the `config` filtering will be included. #[allow(unused)] pub(crate) fn expected_stats_schema( data_schema: &Schema, @@ -137,7 +136,7 @@ pub(crate) fn expected_stats_schema( requested_columns: Option<&[ColumnName]>, ) -> DeltaResult { let mut fields = Vec::with_capacity(5); - fields.push(StructField::nullable("numRecords", DataType::LONG)); + fields.push(StructField::nullable(NUM_RECORDS, DataType::LONG)); // generate the base stats schema: // - make all fields nullable @@ -150,25 +149,24 @@ pub(crate) fn expected_stats_schema( // convert all leaf fields to data type LONG for null count let mut null_count_transform = NullCountStatsTransform; - if let Some(null_count_schema) = null_count_transform.transform_struct(&base_schema) { - fields.push(StructField::nullable( - "nullCount", - null_count_schema.into_owned(), - )); - }; + let null_count_schema = null_count_transform.transform_struct(&base_schema); + fields.push(StructField::nullable( + NULL_COUNT, + null_count_schema.into_owned(), + )); // include only min/max skipping eligible fields (data types) let mut min_max_transform = MinMaxStatsTransform; if let Some(min_max_schema) = min_max_transform.transform_struct(&base_schema) { let min_max_schema = min_max_schema.into_owned(); - fields.push(StructField::nullable("minValues", min_max_schema.clone())); - fields.push(StructField::nullable("maxValues", min_max_schema)); + fields.push(StructField::nullable(MIN_VALUES, min_max_schema.clone())); + fields.push(StructField::nullable(MAX_VALUES, min_max_schema)); } } // tightBounds indicates whether min/max statistics are accurate (true) or potentially // outdated due to deletion vectors (false) - fields.push(StructField::nullable("tightBounds", DataType::BOOLEAN)); + fields.push(StructField::nullable(TIGHT_BOUNDS, DataType::BOOLEAN)); StructType::try_new(fields) } @@ -193,45 +191,20 @@ pub(crate) fn stats_column_names( columns } -/// Creates a stats schema from a referenced schema (e.g. columns from a predicate). -/// Returns schema: `{ numRecords, nullCount, minValues, maxValues }` -/// -/// This is used to build the schema for parsing JSON stats and for reading stats_parsed -/// from checkpoints when only a subset of columns is needed (e.g. predicate-referenced columns). -pub(crate) fn build_stats_schema(referenced_schema: &StructType) -> Option { - let stats_schema = schema_with_all_fields_nullable(referenced_schema).ok()?; - - let nullcount_schema = NullCountStatsTransform - .transform_struct(&stats_schema)? - .into_owned(); - - let schema = StructType::new_unchecked([ - StructField::nullable("numRecords", DataType::LONG), - StructField::nullable("nullCount", nullcount_schema), - StructField::nullable("minValues", stats_schema.clone()), - StructField::nullable("maxValues", stats_schema), - ]); - - // Strip field metadata. The stats types are derived from the table schema, but the metadata on - // the fields should not be included in the stats fields - let schema = StripFieldMetadataTransform - .transform_struct(&schema) - .map(|s| s.into_owned()) - .unwrap_or(schema); - - Some(Arc::new(schema)) -} - -/// Strips all field metadata from a schema. +/// Strips field metadata from every field in a schema, at all levels of nesting (nested +/// sub-fields are stripped too, not just top-level fields). Field types, names, and nullability +/// are preserved; only the metadata map is cleared. /// -/// Field metadata describes the logical table column, not the stats values themselves. This -/// transform strips that metadata, and must be applied to stats schemas to avoid schema possible -/// mismatches when reading `stats_parsed` from older data since that field metadata could have +/// Used wherever a derived schema should carry no field metadata. For example, stats schemas must +/// strip it to avoid possible schema mismatches when reading `stats_parsed` from older data, since +/// that metadata (which describes the logical table column, not the stats values) could have /// changed. pub(crate) struct StripFieldMetadataTransform; impl<'a> SchemaTransform<'a> for StripFieldMetadataTransform { - fn transform_struct_field(&mut self, field: &'a StructField) -> Option> { - Some(match self.transform(&field.data_type)? { + transform_output_type!(|'a, T| Cow<'a, T>); + + fn transform_struct_field(&mut self, field: &'a StructField) -> Cow<'a, StructField> { + match self.transform(&field.data_type) { Cow::Borrowed(_) if field.metadata.is_empty() => Cow::Borrowed(field), data_type => Cow::Owned(StructField { name: field.name.clone(), @@ -239,26 +212,25 @@ impl<'a> SchemaTransform<'a> for StripFieldMetadataTransform { nullable: field.is_nullable(), metadata: Default::default(), }), - }) + } } } /// Make all fields of a schema nullable. /// Used for stats schemas where stats may not be available for all columns. -pub(crate) fn schema_with_all_fields_nullable(schema: &Schema) -> DeltaResult { - match NullableStatsTransform.transform_struct(schema) { - Some(schema) => Ok(schema.into_owned()), - None => Err(Error::internal_error("NullableStatsTransform failed")), - } +pub(crate) fn schema_with_all_fields_nullable(schema: &Schema) -> Schema { + NullableStatsTransform.transform_struct(schema).into_owned() } /// Transforms a schema to make all fields nullable. /// Used for stats schemas where stats may not be available for all columns. pub(crate) struct NullableStatsTransform; impl<'a> SchemaTransform<'a> for NullableStatsTransform { - fn transform_struct_field(&mut self, field: &'a StructField) -> Option> { - let data_type = self.transform(&field.data_type)?; - Some(make_nullable_field(field, data_type)) + transform_output_type!(|'a, T| Cow<'a, T>); + + fn transform_struct_field(&mut self, field: &'a StructField) -> Cow<'a, StructField> { + let data_type = self.transform(&field.data_type); + make_nullable_field(field, data_type) } } @@ -287,16 +259,18 @@ fn make_nullable_field<'a>( /// is preserved for all fields. pub(crate) struct NullCountStatsTransform; impl<'a> SchemaTransform<'a> for NullCountStatsTransform { - fn transform_struct_field(&mut self, field: &'a StructField) -> Option> { + transform_output_type!(|'a, T| Cow<'a, T>); + + fn transform_struct_field(&mut self, field: &'a StructField) -> Cow<'a, StructField> { // Only recurse into struct fields; convert all other types (leaf fields) to LONG match &field.data_type { DataType::Struct(_) => self.recurse_into_struct_field(field), - _ => Some(Cow::Owned(StructField { + _ => Cow::Owned(StructField { name: field.name.clone(), data_type: DataType::LONG, nullable: true, metadata: field.metadata.clone(), - })), + }), } } } @@ -304,8 +278,8 @@ impl<'a> SchemaTransform<'a> for NullCountStatsTransform { /// Transforms a table schema into a base stats schema. /// /// Base stats schema in this case refers the subsets of fields in the table schema -/// that may be considered for stats collection. Depending on the type of stats - min/max/nullcount/... - -/// additional transformations may be applied. +/// that may be considered for stats collection. Depending on the type of stats - +/// min/max/nullcount/... - additional transformations may be applied. /// /// All fields in the output are nullable. Clustering columns are always included per /// the Delta protocol. @@ -325,10 +299,24 @@ impl<'col> BaseStatsTransform<'col> { filter: StatsColumnFilter::new(config, required_columns, requested_columns), } } + + /// Checks whether a leaf column (primitive, array, map, or variant) should be included in the + /// stats schema and records it against the column limit if so. The column limit is based on + /// schema order, so we count all leaf columns that pass the table filter, but only generate + /// stats for requested columns. + fn include_leaf(&mut self) -> bool { + if !self.filter.should_include_for_table() { + return false; + } + self.filter.record_included(); + self.filter.should_include_for_requested() + } } impl<'a> SchemaTransform<'a> for BaseStatsTransform<'_> { - // Always traverse struct fields -- only primitive leaf values count against the column limit + transform_output_type!(|'a, T| Option>); + + // Always traverse struct fields. All non-struct leaf types count against the column limit. fn transform_struct_field(&mut self, field: &'a StructField) -> Option> { self.filter.enter_field(field.name()); let data_type = self.transform(&field.data_type); @@ -337,28 +325,19 @@ impl<'a> SchemaTransform<'a> for BaseStatsTransform<'_> { } fn transform_primitive(&mut self, ptype: &'a PrimitiveType) -> Option> { - if !self.filter.should_include_for_table() { - return None; - } - - // The n_columns limit is based on schema order, so we count all leaf columns that pass the - // table filter, but then we only generate stats for requested columns. - self.filter.record_included(); - self.filter - .should_include_for_requested() - .then_some(Cow::Borrowed(ptype)) + self.include_leaf().then_some(Cow::Borrowed(ptype)) } - fn transform_array(&mut self, _: &'a ArrayType) -> Option> { - None // not stats-eligible + fn transform_array(&mut self, atype: &'a ArrayType) -> Option> { + self.include_leaf().then_some(Cow::Borrowed(atype)) } - fn transform_map(&mut self, _: &'a MapType) -> Option> { - None // not stats-eligible + fn transform_map(&mut self, mtype: &'a MapType) -> Option> { + self.include_leaf().then_some(Cow::Borrowed(mtype)) } - fn transform_variant(&mut self, _: &'a StructType) -> Option> { - None // not stats-eligible + fn transform_variant(&mut self, vtype: &'a StructType) -> Option> { + self.include_leaf().then_some(Cow::Borrowed(vtype)) } } @@ -369,9 +348,10 @@ impl<'a> SchemaTransform<'a> for BaseStatsTransform<'_> { struct MinMaxStatsTransform; impl<'a> SchemaTransform<'a> for MinMaxStatsTransform { - // Array, Map, and Variant fields are filtered out by BaseStatsTransform, so these methods - // are typically not called. They're kept as a safety net in case the transform is used - // independently or the filtering logic changes. + transform_output_type!(|'a, T| Option>); + + // Array, Map, and Variant fields pass through BaseStatsTransform (for nullCount) but must + // be excluded from min/max stats. fn transform_array(&mut self, _: &'a ArrayType) -> Option> { None } @@ -392,8 +372,12 @@ impl<'a> SchemaTransform<'a> for MinMaxStatsTransform { /// This is also used to validate clustering column types, since clustering requires /// per-file statistics on clustering columns. /// -/// Note: Boolean and Binary are intentionally excluded as min/max statistics provide minimal -/// skipping benefit for low-cardinality or opaque data types. +/// Note: Boolean, Binary, and Interval are intentionally excluded as min/max statistics provide +/// minimal skipping benefit or do not have stable protocol-level ordering semantics. +/// +/// Void is also excluded: void columns are never materialized to Parquet, so min/max are not +/// meaningful. When `nullCount` stats are present for a void column, `eval_pred_is_null` can +/// use them for `IS NULL` / `IS NOT NULL` file skipping. /// /// See: pub(crate) fn is_skipping_eligible_datatype(data_type: &PrimitiveType) -> bool { @@ -415,10 +399,25 @@ pub(crate) fn is_skipping_eligible_datatype(data_type: &PrimitiveType) -> bool { #[cfg(test)] mod tests { + #[cfg(feature = "geo-type-in-dev")] + use rstest::rstest; + + use super::*; + use crate::expressions::column_name; use crate::schema::ArrayType; + #[cfg(feature = "geo-type-in-dev")] + use crate::schema::{EdgeInterpolationAlgorithm, GeographyType, GeometryType}; use crate::table_properties::TableProperties; - use super::*; + #[cfg(feature = "geo-type-in-dev")] + #[rstest] + #[case(PrimitiveType::Geometry(Box::new(GeometryType::try_new("EPSG:4326").unwrap())))] + #[case(PrimitiveType::Geography(Box::new( + GeographyType::try_new("EPSG:4326", EdgeInterpolationAlgorithm::Spherical).unwrap() + )))] + fn test_geo_types_are_not_skipping_eligible(#[case] ptype: PrimitiveType) { + assert!(!is_skipping_eligible_datatype(&ptype)); + } fn stats_config_from_table_properties(properties: &TableProperties) -> StatsConfig<'_> { StatsConfig { @@ -430,11 +429,11 @@ mod tests { /// Builds an expected stats schema from the given null count and min/max nested schemas. fn expected_stats(null_count: StructType, min_max: StructType) -> StructType { StructType::new_unchecked([ - StructField::nullable("numRecords", DataType::LONG), - StructField::nullable("nullCount", null_count), - StructField::nullable("minValues", min_max.clone()), - StructField::nullable("maxValues", min_max), - StructField::nullable("tightBounds", DataType::BOOLEAN), + StructField::nullable(NUM_RECORDS, DataType::LONG), + StructField::nullable(NULL_COUNT, null_count), + StructField::nullable(MIN_VALUES, min_max.clone()), + StructField::nullable(MAX_VALUES, min_max), + StructField::nullable(TIGHT_BOUNDS, DataType::BOOLEAN), ]) } @@ -465,7 +464,7 @@ mod tests { ]); let file_schema = StructType::new_unchecked([ StructField::not_null("id", DataType::LONG), - StructField::not_null("user", DataType::Struct(Box::new(user_struct.clone()))), + StructField::not_null("user", user_struct.clone()), ]); let stats_schema = expected_stats_schema( &file_schema, @@ -479,11 +478,9 @@ mod tests { // but make all fields nullable let expected_min_max = NullableStatsTransform .transform_struct(&file_schema) - .unwrap() .into_owned(); let null_count = NullCountStatsTransform .transform_struct(&expected_min_max) - .unwrap() .into_owned(); let expected = expected_stats(null_count, expected_min_max); @@ -503,7 +500,7 @@ mod tests { // - "score" (DOUBLE) - eligible for data skipping // Create array type for a field that's not eligible for data skipping - let array_type = DataType::Array(Box::new(ArrayType::new(DataType::STRING, false))); + let array_type = DataType::from(ArrayType::new(DataType::STRING, false)); let metadata_struct = StructType::new_unchecked([ StructField::nullable("name", DataType::STRING), StructField::nullable("tags", array_type), @@ -511,10 +508,7 @@ mod tests { ]); let file_schema = StructType::new_unchecked([ StructField::nullable("id", DataType::LONG), - StructField::nullable( - "metadata", - DataType::Struct(Box::new(metadata_struct.clone())), - ), + StructField::nullable("metadata", metadata_struct.clone()), ]); let stats_schema = expected_stats_schema( @@ -525,14 +519,15 @@ mod tests { ) .unwrap(); - // nullCount excludes array fields (tags) - only eligible primitive types + // nullCount includes array fields (tags) as leaf columns let expected_null_nested = StructType::new_unchecked([ StructField::nullable("name", DataType::LONG), + StructField::nullable("tags", DataType::LONG), StructField::nullable("score", DataType::LONG), ]); let expected_null = StructType::new_unchecked([ StructField::nullable("id", DataType::LONG), - StructField::nullable("metadata", DataType::Struct(Box::new(expected_null_nested))), + StructField::nullable("metadata", expected_null_nested), ]); let expected_nested = StructType::new_unchecked([ @@ -541,7 +536,7 @@ mod tests { ]); let expected_fields = StructType::new_unchecked([ StructField::nullable("id", DataType::LONG), - StructField::nullable("metadata", DataType::Struct(Box::new(expected_nested))), + StructField::nullable("metadata", expected_nested), ]); let expected = expected_stats(expected_null, expected_fields); @@ -563,7 +558,7 @@ mod tests { ]); let file_schema = StructType::new_unchecked([ StructField::nullable("id", DataType::LONG), - StructField::nullable("user.info", DataType::Struct(Box::new(user_struct.clone()))), + StructField::nullable("user.info", user_struct.clone()), ]); let stats_schema = expected_stats_schema( @@ -576,13 +571,10 @@ mod tests { let expected_nested = StructType::new_unchecked([StructField::nullable("name", DataType::STRING)]); - let expected_fields = StructType::new_unchecked([StructField::nullable( - "user.info", - DataType::Struct(Box::new(expected_nested)), - )]); + let expected_fields = + StructType::new_unchecked([StructField::nullable("user.info", expected_nested)]); let null_count = NullCountStatsTransform .transform_struct(&expected_fields) - .unwrap() .into_owned(); let expected = expected_stats(null_count, expected_fields); @@ -590,6 +582,55 @@ mod tests { assert_eq!(&expected, &stats_schema); } + #[test] + fn test_stats_schema_stats_columns_with_complex_types() { + // When dataSkippingStatsColumns explicitly names a complex type column, only that + // column (plus any other named columns) should appear in the stats schema. + let properties: TableProperties = [( + "delta.dataSkippingStatsColumns".to_string(), + "id,tags".to_string(), + )] + .into(); + + let file_schema = StructType::new_unchecked([ + StructField::nullable("id", DataType::LONG), + StructField::nullable("tags", ArrayType::new(DataType::STRING, false)), + StructField::nullable( + "metadata", + MapType::new(DataType::STRING, DataType::STRING, true), + ), + StructField::nullable("name", DataType::STRING), + ]); + + let stats_schema = expected_stats_schema( + &file_schema, + &stats_config_from_table_properties(&properties), + None, + None, + ) + .unwrap(); + + // nullCount: only id and tags (explicitly requested) + let expected_null_count = StructType::new_unchecked([ + StructField::nullable("id", DataType::LONG), + StructField::nullable("tags", DataType::LONG), + ]); + + // minValues/maxValues: only id (tags excluded by MinMaxStatsTransform) + let expected_min_max = + StructType::new_unchecked([StructField::nullable("id", DataType::LONG)]); + + let expected = StructType::new_unchecked([ + StructField::nullable(NUM_RECORDS, DataType::LONG), + StructField::nullable(NULL_COUNT, expected_null_count), + StructField::nullable(MIN_VALUES, expected_min_max.clone()), + StructField::nullable(MAX_VALUES, expected_min_max), + StructField::nullable(TIGHT_BOUNDS, DataType::BOOLEAN), + ]); + + assert_eq!(&expected, &stats_schema); + } + #[test] fn test_stats_schema_n_cols() { let properties: TableProperties = [( @@ -615,7 +656,6 @@ mod tests { StructType::new_unchecked([StructField::nullable("name", DataType::STRING)]); let null_count = NullCountStatsTransform .transform_struct(&expected_fields) - .unwrap() .into_owned(); let expected = expected_stats(null_count, expected_fields); @@ -631,10 +671,12 @@ mod tests { // - "id" (LONG) - eligible for both null count and min/max // - "is_active" (BOOLEAN) - eligible for null count but NOT for min/max // - "metadata" (BINARY) - eligible for null count but NOT for min/max + // - "duration" (INTERVAL_DAY_TIME) - eligible for null count but NOT for min/max let file_schema = StructType::new_unchecked([ StructField::nullable("id", DataType::LONG), StructField::nullable("is_active", DataType::BOOLEAN), StructField::nullable("metadata", DataType::BINARY), + StructField::nullable("duration", DataType::INTERVAL_DAY_TIME), ]); let stats_schema = expected_stats_schema( @@ -650,9 +692,10 @@ mod tests { StructField::nullable("id", DataType::LONG), StructField::nullable("is_active", DataType::LONG), StructField::nullable("metadata", DataType::LONG), + StructField::nullable("duration", DataType::LONG), ]); - // Expected minValues/maxValues schema: only eligible fields (no boolean, no binary) + // Expected minValues/maxValues schema: only eligible fields let expected_min_max = StructType::new_unchecked([StructField::nullable("id", DataType::LONG)]); @@ -665,7 +708,8 @@ mod tests { fn test_stats_schema_nested_different_fields_in_null_vs_minmax() { let properties: TableProperties = [("key", "value")].into(); - // Create a nested schema where some nested fields are eligible for min/max and others aren't + // Create a nested schema where some nested fields are eligible for min/max and others + // aren't let user_struct = StructType::new_unchecked([ StructField::nullable("name", DataType::STRING), // eligible for min/max StructField::nullable("is_admin", DataType::BOOLEAN), // NOT eligible for min/max @@ -675,7 +719,7 @@ mod tests { let file_schema = StructType::new_unchecked([ StructField::nullable("id", DataType::LONG), - StructField::nullable("user", DataType::Struct(Box::new(user_struct.clone()))), + StructField::nullable("user", user_struct.clone()), StructField::nullable("is_deleted", DataType::BOOLEAN), // NOT eligible for min/max ]); @@ -696,7 +740,7 @@ mod tests { ]); let expected_null_count = StructType::new_unchecked([ StructField::nullable("id", DataType::LONG), - StructField::nullable("user", DataType::Struct(Box::new(expected_null_user))), + StructField::nullable("user", expected_null_user), StructField::nullable("is_deleted", DataType::LONG), ]); @@ -707,7 +751,7 @@ mod tests { ]); let expected_min_max = StructType::new_unchecked([ StructField::nullable("id", DataType::LONG), - StructField::nullable("user", DataType::Struct(Box::new(expected_minmax_user))), + StructField::nullable("user", expected_minmax_user), ]); let expected = expected_stats(expected_null_count, expected_min_max); @@ -723,10 +767,8 @@ mod tests { let file_schema = StructType::new_unchecked([ StructField::nullable("is_active", DataType::BOOLEAN), StructField::nullable("metadata", DataType::BINARY), - StructField::nullable( - "tags", - DataType::Array(Box::new(ArrayType::new(DataType::STRING, false))), - ), + StructField::nullable("duration", DataType::INTERVAL_DAY_TIME), + StructField::nullable("tags", ArrayType::new(DataType::STRING, false)), ]); let stats_schema = expected_stats_schema( @@ -737,50 +779,79 @@ mod tests { ) .unwrap(); - // nullCount includes boolean and binary (primitives) but excludes array + // nullCount includes all selected non-struct fields let expected_null_count = StructType::new_unchecked([ StructField::nullable("is_active", DataType::LONG), StructField::nullable("metadata", DataType::LONG), + StructField::nullable("duration", DataType::LONG), + StructField::nullable("tags", DataType::LONG), ]); - // minValues/maxValues: no fields are eligible (boolean/binary excluded) + // minValues/maxValues: no fields are eligible let expected = StructType::new_unchecked([ - StructField::nullable("numRecords", DataType::LONG), - StructField::nullable("nullCount", expected_null_count), - StructField::nullable("tightBounds", DataType::BOOLEAN), + StructField::nullable(NUM_RECORDS, DataType::LONG), + StructField::nullable(NULL_COUNT, expected_null_count), + StructField::nullable(TIGHT_BOUNDS, DataType::BOOLEAN), ]); assert_eq!(&expected, &stats_schema); } + #[rstest::rstest] + #[case::num_indexed_cols("delta.dataSkippingNumIndexedCols", "1")] + #[case::stats_columns("delta.dataSkippingStatsColumns", "iv")] + fn test_interval_stats_respect_column_selection( + #[values(DataType::INTERVAL_YEAR_MONTH, DataType::INTERVAL_DAY_TIME)] interval: DataType, + #[case] property_name: &str, + #[case] property_value: &str, + ) { + let properties: TableProperties = [(property_name, property_value)].into(); + let file_schema = StructType::new_unchecked([ + StructField::nullable("iv", interval), + StructField::nullable("value", DataType::LONG), + ]); + + let stats_schema = expected_stats_schema( + &file_schema, + &stats_config_from_table_properties(&properties), + None, + None, + ) + .unwrap(); + + let expected = StructType::new_unchecked([ + StructField::nullable(NUM_RECORDS, DataType::LONG), + StructField::nullable( + NULL_COUNT, + StructType::new_unchecked([StructField::nullable("iv", DataType::LONG)]), + ), + StructField::nullable(TIGHT_BOUNDS, DataType::BOOLEAN), + ]); + assert_eq!(expected, stats_schema); + } + #[test] - fn test_stats_schema_map_array_dont_count_against_limit() { - // Test that Map and Array fields don't count against the column limit. - // With a limit of 2, if we have: array, map, col1, col2, col3 - // We should get stats for col1 and col2 (the first 2 eligible columns), - // not be limited by the array and map fields. + fn test_stats_schema_complex_types_count_against_limit() { + // Array, Map, and Variant are leaf columns that count against the column limit, + // matching Spark's truncateSchema which counts all non-struct fields. + // With a limit of 3, if we have: array, map, variant, col1, col2 + // We should get nullCount for the 3 complex types (the first 3 leaf columns), + // and col1/col2 are excluded by the limit. let properties: TableProperties = [( "delta.dataSkippingNumIndexedCols".to_string(), - "2".to_string(), + "3".to_string(), )] .into(); let file_schema = StructType::new_unchecked([ - StructField::nullable( - "tags", - DataType::Array(Box::new(ArrayType::new(DataType::STRING, false))), - ), + StructField::nullable("tags", ArrayType::new(DataType::STRING, false)), StructField::nullable( "metadata", - DataType::Map(Box::new(MapType::new( - DataType::STRING, - DataType::STRING, - true, - ))), + MapType::new(DataType::STRING, DataType::STRING, true), ), + StructField::nullable("v", DataType::unshredded_variant()), StructField::nullable("col1", DataType::LONG), StructField::nullable("col2", DataType::STRING), - StructField::nullable("col3", DataType::INTEGER), // Should be excluded by limit ]); let stats_schema = expected_stats_schema( @@ -791,21 +862,67 @@ mod tests { ) .unwrap(); - // nullCount has only eligible primitive columns (col1 and col2). - // Map/Array/Variant are excluded from all stats. + // nullCount includes array, map, and variant (the first 3 leaf columns). + // col1/col2 are excluded by the limit. let expected_null_count = StructType::new_unchecked([ - StructField::nullable("col1", DataType::LONG), - StructField::nullable("col2", DataType::LONG), + StructField::nullable("tags", DataType::LONG), + StructField::nullable("metadata", DataType::LONG), + StructField::nullable("v", DataType::LONG), ]); - // minValues/maxValues only have eligible primitive types (col1 and col2). - // Map/Array are filtered out by MinMaxStatsTransform. - let expected_min_max = StructType::new_unchecked([ - StructField::nullable("col1", DataType::LONG), - StructField::nullable("col2", DataType::STRING), + // minValues/maxValues: all 3 complex types are excluded by MinMaxStatsTransform, + // and col1/col2 are past the limit, so no min/max fields at all. + let expected = StructType::new_unchecked([ + StructField::nullable(NUM_RECORDS, DataType::LONG), + StructField::nullable(NULL_COUNT, expected_null_count), + StructField::nullable(TIGHT_BOUNDS, DataType::BOOLEAN), ]); - let expected = expected_stats(expected_null_count, expected_min_max); + assert_eq!(&expected, &stats_schema); + } + + #[test] + fn test_stats_schema_complex_type_consumes_slot_before_primitive() { + // Validates that a complex type consuming a slot causes a subsequent primitive to + // be excluded. With limit=2: id (long), tags (array), name (string), only id and + // tags get stats, name is excluded. + let properties: TableProperties = [( + "delta.dataSkippingNumIndexedCols".to_string(), + "2".to_string(), + )] + .into(); + + let file_schema = StructType::new_unchecked([ + StructField::nullable("id", DataType::LONG), + StructField::nullable("tags", ArrayType::new(DataType::STRING, false)), + StructField::nullable("name", DataType::STRING), + ]); + + let stats_schema = expected_stats_schema( + &file_schema, + &stats_config_from_table_properties(&properties), + None, + None, + ) + .unwrap(); + + // nullCount: id and tags (first 2 leaf columns). name is excluded. + let expected_null_count = StructType::new_unchecked([ + StructField::nullable("id", DataType::LONG), + StructField::nullable("tags", DataType::LONG), + ]); + + // minValues/maxValues: only id (tags excluded by MinMaxStatsTransform, name past limit) + let expected_min_max = + StructType::new_unchecked([StructField::nullable("id", DataType::LONG)]); + + let expected = StructType::new_unchecked([ + StructField::nullable(NUM_RECORDS, DataType::LONG), + StructField::nullable(NULL_COUNT, expected_null_count), + StructField::nullable(MIN_VALUES, expected_min_max.clone()), + StructField::nullable(MAX_VALUES, expected_min_max), + StructField::nullable(TIGHT_BOUNDS, DataType::BOOLEAN), + ]); assert_eq!(&expected, &stats_schema); } @@ -822,7 +939,7 @@ mod tests { ]); let file_schema = StructType::new_unchecked([ StructField::nullable("id", DataType::LONG), - StructField::nullable("user", DataType::Struct(Box::new(user_struct))), + StructField::nullable("user", user_struct), ]); let config = StatsConfig { @@ -835,9 +952,9 @@ mod tests { assert_eq!( columns, vec![ - ColumnName::new(["id"]), - ColumnName::new(["user", "name"]), - ColumnName::new(["user", "age"]), + column_name!("id"), + column_name!("user.name"), + column_name!("user.age"), ] ); } @@ -864,10 +981,7 @@ mod tests { let columns = stats_column_names(&file_schema, &config, None); // Only first 2 columns should be included - assert_eq!( - columns, - vec![ColumnName::new(["a"]), ColumnName::new(["b"]),] - ); + assert_eq!(columns, vec![column_name!("a"), column_name!("b"),]); } #[test] @@ -884,7 +998,7 @@ mod tests { ]); let file_schema = StructType::new_unchecked([ StructField::nullable("id", DataType::LONG), - StructField::nullable("user", DataType::Struct(Box::new(user_struct))), + StructField::nullable("user", user_struct), StructField::nullable("extra", DataType::STRING), ]); @@ -895,30 +1009,21 @@ mod tests { let columns = stats_column_names(&file_schema, &config, None); // Only specified columns should be included (user.name and extra excluded) - assert_eq!( - columns, - vec![ColumnName::new(["id"]), ColumnName::new(["user", "age"]),] - ); + assert_eq!(columns, vec![column_name!("id"), column_name!("user.age"),]); } #[test] - fn test_stats_column_names_skips_non_eligible_types() { + fn test_stats_column_names_includes_complex_types() { let properties: TableProperties = [("key", "value")].into(); let file_schema = StructType::new_unchecked([ StructField::nullable("id", DataType::LONG), - StructField::nullable( - "tags", - DataType::Array(Box::new(ArrayType::new(DataType::STRING, false))), - ), + StructField::nullable("tags", ArrayType::new(DataType::STRING, false)), StructField::nullable( "metadata", - DataType::Map(Box::new(MapType::new( - DataType::STRING, - DataType::STRING, - true, - ))), + MapType::new(DataType::STRING, DataType::STRING, true), ), + StructField::nullable("v", DataType::unshredded_variant()), StructField::nullable("name", DataType::STRING), ]); @@ -928,10 +1033,16 @@ mod tests { }; let columns = stats_column_names(&file_schema, &config, None); - // Array and Map types should be excluded + // Array, Map, and Variant are leaf columns and included in the stats column list assert_eq!( columns, - vec![ColumnName::new(["id"]), ColumnName::new(["name"]),] + vec![ + column_name!("id"), + column_name!("tags"), + column_name!("metadata"), + column_name!("v"), + column_name!("name"), + ] ); } @@ -953,7 +1064,7 @@ mod tests { ]); // "c" is a clustering column, should be included even though limit is 1 - let clustering_columns = vec![ColumnName::new(["c"])]; + let clustering_columns = vec![column_name!("c")]; let stats_schema = expected_stats_schema( &file_schema, &stats_config_from_table_properties(&properties), @@ -988,7 +1099,7 @@ mod tests { StructField::nullable("value", DataType::INTEGER), ]); - let columns = [ColumnName::new(["id"])]; + let columns = [column_name!("id")]; let stats_schema = expected_stats_schema( &file_schema, &stats_config_from_table_properties(&properties), @@ -1023,7 +1134,7 @@ mod tests { .unwrap(); // Should include both columns - let min_values = with_none.field("minValues").expect("should have minValues"); + let min_values = with_none.field(MIN_VALUES).expect("should have minValues"); if let DataType::Struct(inner) = min_values.data_type() { assert!(inner.field("id").is_some()); assert!(inner.field("name").is_some()); @@ -1042,7 +1153,7 @@ mod tests { ]); // "name" is outside the limit (limit is 1), and is only requested, not required - let columns = [ColumnName::new(["name"])]; + let columns = [column_name!("name")]; let stats_schema = expected_stats_schema( &file_schema, &stats_config_from_table_properties(&properties), @@ -1053,8 +1164,8 @@ mod tests { // No data columns pass both filters, so only numRecords + tightBounds let expected = StructType::new_unchecked([ - StructField::nullable("numRecords", DataType::LONG), - StructField::nullable("tightBounds", DataType::BOOLEAN), + StructField::nullable(NUM_RECORDS, DataType::LONG), + StructField::nullable(TIGHT_BOUNDS, DataType::BOOLEAN), ]); assert_eq!(&expected, &stats_schema); @@ -1070,7 +1181,7 @@ mod tests { StructField::nullable("name", DataType::STRING), ]); - let columns = [ColumnName::new(["name"])]; + let columns = [column_name!("name")]; let stats_schema = expected_stats_schema( &file_schema, &stats_config_from_table_properties(&properties), @@ -1101,7 +1212,7 @@ mod tests { StructField::nullable("value", DataType::INTEGER), ]); - let columns = [ColumnName::new(["name"])]; + let columns = [column_name!("name")]; let stats_schema = expected_stats_schema( &file_schema, &stats_config_from_table_properties(&properties), @@ -1130,7 +1241,7 @@ mod tests { StructField::nullable("value", DataType::INTEGER), ]); - let columns = [ColumnName::new(["id"]), ColumnName::new(["name"])]; + let columns = [column_name!("id"), column_name!("name")]; let stats_schema = expected_stats_schema( &file_schema, &stats_config_from_table_properties(&properties), @@ -1162,10 +1273,10 @@ mod tests { ]); let file_schema = StructType::new_unchecked([ StructField::nullable("id", DataType::LONG), - StructField::nullable("user", DataType::Struct(Box::new(user_struct))), + StructField::nullable("user", user_struct), ]); - let columns = [ColumnName::new(["user", "name"])]; + let columns = [column_name!("user.name")]; let stats_schema = expected_stats_schema( &file_schema, &stats_config_from_table_properties(&properties), @@ -1176,17 +1287,13 @@ mod tests { let expected_user_nested = StructType::new_unchecked([StructField::nullable("name", DataType::STRING)]); - let expected_nested = StructType::new_unchecked([StructField::nullable( - "user", - DataType::Struct(Box::new(expected_user_nested)), - )]); + let expected_nested = + StructType::new_unchecked([StructField::nullable("user", expected_user_nested)]); let expected_user_null = StructType::new_unchecked([StructField::nullable("name", DataType::LONG)]); - let expected_null = StructType::new_unchecked([StructField::nullable( - "user", - DataType::Struct(Box::new(expected_user_null)), - )]); + let expected_null = + StructType::new_unchecked([StructField::nullable("user", expected_user_null)]); let expected = expected_stats(expected_null, expected_nested); @@ -1230,11 +1337,11 @@ mod tests { ]); let file_schema = StructType::new_unchecked([ StructField::nullable("id", DataType::LONG), - StructField::nullable("user", DataType::Struct(Box::new(user_struct))), + StructField::nullable("user", user_struct), StructField::nullable("value", DataType::DOUBLE), ]); - let columns = [ColumnName::new(["id"]), ColumnName::new(["user", "age"])]; + let columns = [column_name!("id"), column_name!("user.age")]; let stats_schema = expected_stats_schema( &file_schema, &stats_config_from_table_properties(&properties), @@ -1247,17 +1354,14 @@ mod tests { StructType::new_unchecked([StructField::nullable("age", DataType::INTEGER)]); let expected_nested = StructType::new_unchecked([ StructField::nullable("id", DataType::LONG), - StructField::nullable( - "user", - DataType::Struct(Box::new(expected_user_nested.clone())), - ), + StructField::nullable("user", expected_user_nested.clone()), ]); let expected_user_null = StructType::new_unchecked([StructField::nullable("age", DataType::LONG)]); let expected_null = StructType::new_unchecked([ StructField::nullable("id", DataType::LONG), - StructField::nullable("user", DataType::Struct(Box::new(expected_user_null))), + StructField::nullable("user", expected_user_null), ]); let expected = expected_stats(expected_null, expected_nested); diff --git a/kernel/src/scan/data_skipping/tests.rs b/kernel/src/scan/data_skipping/tests.rs index 9542ae51d7..e2094e875a 100644 --- a/kernel/src/scan/data_skipping/tests.rs +++ b/kernel/src/scan/data_skipping/tests.rs @@ -1,9 +1,12 @@ -use super::*; +use std::collections::HashMap; -use crate::expressions::column_name; -use crate::kernel_predicates::{DefaultKernelPredicateEvaluator, UnimplementedColumnResolver}; use rstest::rstest; -use std::collections::HashMap; + +use super::*; +use crate::expressions::{col, column_expr_ref, column_name, lit}; +use crate::kernel_predicates::{ + DefaultKernelPredicateEvaluator, EmptyColumnResolver, UnimplementedColumnResolver, +}; const TRUE: Option = Some(true); const FALSE: Option = Some(false); @@ -25,7 +28,7 @@ macro_rules! expect_eq { #[test] fn test_eval_is_null() { - let col = &column_expr!("x"); + let col = &col!("x"); let predicates = [Pred::is_null(col.clone()), Pred::is_not_null(col.clone())]; let do_test = |nullcount: i64, expected: &[Option]| { @@ -59,7 +62,7 @@ fn test_eval_is_null() { #[test] fn test_eval_binary_comparisons() { - let col = &column_expr!("x"); + let col = &col!("x"); let five = &Scalar::from(5); let ten = &Scalar::from(10); let fifteen = &Scalar::from(15); @@ -167,7 +170,7 @@ fn test_eval_junction() { .iter() .map(|val| match val { Some(v) => Pred::literal(*v), - None => Pred::null_literal(), + None => Pred::NULL, }) .collect(); @@ -198,7 +201,7 @@ fn test_eval_junction() { // vs. non-NULL literals and nullcount/rowcount stats. #[test] fn test_eval_distinct() { - let col = &column_expr!("x"); + let col = &col!("x"); let five = &Scalar::from(5); let ten = &Scalar::from(10); let fifteen = &Scalar::from(15); @@ -262,12 +265,7 @@ fn test_eval_distinct() { #[test] fn test_sql_where() { - let col = &column_expr!("x"); const VAL: Expr = Expr::Literal(Scalar::Integer(10)); - const NULL: Pred = Pred::null_literal(); - const FALSE: Pred = Pred::literal(false); - const TRUE: Pred = Pred::literal(true); - const ROWCOUNT: i64 = 2; const ALL_NULL: i64 = ROWCOUNT; const SOME_NULL: i64 = 1; @@ -318,75 +316,157 @@ fn test_sql_where() { // Sanity tests -- only all-null columns should behave differently between normal and SQL WHERE. const MISSING: bool = true; const PRESENT: bool = false; - let pred = &Pred::lt(TRUE, FALSE); + let pred = &Pred::lt(Pred::TRUE, Pred::FALSE); do_test(ALL_NULL, pred, MISSING, Some(false), Some(false)); - let pred = &Pred::is_not_null(col.clone()); + let pred = &Pred::is_not_null(col!("x")); do_test(ALL_NULL, pred, PRESENT, Some(false), Some(false)); do_test(ALL_NULL, pred, MISSING, None, None); // SQL WHERE allows a present-but-all-null column to be pruned, but not a missing column. - let pred = &Pred::lt(col.clone(), VAL); + let pred = &Pred::lt(col!("x"), VAL); do_test(NO_NULL, pred, PRESENT, Some(true), Some(true)); do_test(SOME_NULL, pred, PRESENT, Some(true), Some(true)); do_test(ALL_NULL, pred, PRESENT, None, Some(false)); do_test(ALL_NULL, pred, MISSING, None, None); // Comparison inside AND works - let pred = &Pred::and(TRUE, Pred::lt(VAL, col.clone())); + let pred = &Pred::and(Pred::TRUE, Pred::lt(VAL, col!("x"))); do_test(ALL_NULL, pred, PRESENT, None, Some(false)); do_test(ALL_NULL, pred, MISSING, None, None); - // NULL inside AND allows static skipping under SQL semantics - let pred = &Pred::and(NULL, Pred::lt(col.clone(), VAL)); + // NULL literal is treated as unknown (not false) under eval_sql_where, so it does not + // force static skipping on its own. With present-but-all-null stats, the comparison arm + // still evaluates to false (null-safe check fails), so AND(unknown, false) = false. + // With missing stats, both arms are unknown, so AND(unknown, unknown) = unknown. + let pred = &Pred::and(Pred::NULL, Pred::lt(col!("x"), VAL)); do_test(ALL_NULL, pred, PRESENT, None, Some(false)); - do_test(ALL_NULL, pred, MISSING, None, Some(false)); + do_test(ALL_NULL, pred, MISSING, None, None); // Comparison inside AND inside AND works - let pred = &Pred::and(TRUE, Pred::and(TRUE, Pred::lt(col.clone(), VAL))); + let pred = &Pred::and(Pred::TRUE, Pred::and(Pred::TRUE, Pred::lt(col!("x"), VAL))); do_test(ALL_NULL, pred, PRESENT, None, Some(false)); do_test(ALL_NULL, pred, MISSING, None, None); // Comparison inside OR works - let pred = &Pred::or(FALSE, Pred::lt(col.clone(), VAL)); + let pred = &Pred::or(Pred::FALSE, Pred::lt(col!("x"), VAL)); do_test(ALL_NULL, pred, PRESENT, None, Some(false)); do_test(ALL_NULL, pred, MISSING, None, None); // Comparison inside AND inside OR works - let pred = &Pred::or(FALSE, Pred::and(TRUE, Pred::lt(col.clone(), VAL))); + let pred = &Pred::or(Pred::FALSE, Pred::and(Pred::TRUE, Pred::lt(col!("x"), VAL))); do_test(ALL_NULL, pred, PRESENT, None, Some(false)); do_test(ALL_NULL, pred, MISSING, None, None); } -// TODO(#1002): we currently don't support file skipping on timestamp columns' max stat since they -// are truncated to milliseconds in add.stats. +/// Validates that the production data-skipping path (the `eval_sql_where` rewrite used by +/// `DataSkippingFilter::new`) prunes a present-but-all-null file for every null-intolerant +/// comparison operator. The bare `eval` rewrite (test-only callers) lacks the not-all-null +/// guard and keeps the file. Each operator's predicate evaluates over stats for a file with +/// `nullCount == numRecords` (all-null) and null min/max -- the `IS NOT NULL` guard that +/// `eval_sql_where` prepends rewrites to `nullCount != numRecords`, which is FALSE here and +/// forces the comparison to skip. +#[rstest] +#[case::eq(Pred::eq(col!("x"), lit(10)))] +#[case::ne(Pred::ne(col!("x"), lit(10)))] +#[case::lt(Pred::lt(col!("x"), lit(10)))] +#[case::gt(Pred::gt(col!("x"), lit(10)))] +#[case::le(Pred::le(col!("x"), lit(10)))] +#[case::ge(Pred::ge(col!("x"), lit(10)))] +fn test_all_null_pruning_all_comparison_ops(#[case] pred: Pred) { + // All-null file: nullCount == numRecords, and min/max are NULL. + let resolver = HashMap::from_iter([ + (column_name!("stats_parsed.numRecords"), Scalar::from(2i64)), + (column_name!("stats_parsed.nullCount.x"), Scalar::from(2i64)), + ( + column_name!("stats_parsed.minValues.x"), + Scalar::Null(DataType::INTEGER), + ), + ( + column_name!("stats_parsed.maxValues.x"), + Scalar::Null(DataType::INTEGER), + ), + ]); + let filter = DefaultKernelPredicateEvaluator::from(resolver); + let stats_columns: HashSet = [column_name!("x")].into_iter().collect(); + + // Production path (eval_sql_where): the IS NOT NULL guard prunes the all-null file. + let sql_pred = as_sql_data_skipping_predicate_with_stats_columns( + &pred, + &Default::default(), + &stats_columns, + ) + .unwrap(); + expect_eq!( + filter.eval(&sql_pred), + FALSE, + "{pred:#?} became {sql_pred:#?} (production eval_sql_where -> skip)" + ); + + // Bare path (eval, test-only): no not-all-null guard, so the file is kept. + let bare_pred = as_data_skipping_predicate(&pred).unwrap(); + expect_eq!( + filter.eval(&bare_pred), + NULL, + "{pred:#?} became {bare_pred:#?} (bare eval -> keep)" + ); +} + #[test] -fn test_timestamp_skipping_disabled() { +fn test_timestamp_stats_enabled() { let empty = HashSet::new(); - let creator = DataSkippingPredicateCreator { - partition_columns: &empty, - }; + let stats_columns: HashSet = [column_name!("timestamp_col")].into_iter().collect(); + let creator = DataSkippingPredicateCreator::new(&empty, &stats_columns); let col = &column_name!("timestamp_col"); assert!( creator.get_min_stat(col, &DataType::TIMESTAMP).is_some(), - "get_min_stat should return Some: allow data skipping on timestamp minValues" + "get_min_stat should return Some for timestamp minValues" ); - assert_eq!( - creator.get_max_stat(col, &DataType::TIMESTAMP), - None, - "get_max_stat should return None: no data skipping on timestamp maxValues" + assert!( + creator.get_max_stat(col, &DataType::TIMESTAMP).is_some(), + "get_max_stat should return Some for timestamp maxValues" ); assert!( creator .get_min_stat(col, &DataType::TIMESTAMP_NTZ) .is_some(), - "get_min_stat should return Some: allow data skipping on timestamp_ntz minValues" + "get_min_stat should return Some for timestamp_ntz minValues" + ); + assert!( + creator + .get_max_stat(col, &DataType::TIMESTAMP_NTZ) + .is_some(), + "get_max_stat should return Some for timestamp_ntz maxValues" ); +} + +#[test] +fn test_adjust_scalar_for_max_stat_truncation() { + // Timestamp: subtracts 999us assert_eq!( - creator.get_max_stat(col, &DataType::TIMESTAMP_NTZ), - None, - "get_max_stat should return None: no data skipping on timestamp_ntz maxValues" + adjust_scalar_for_max_stat_truncation(&Scalar::Timestamp(1_000_000)), + Scalar::Timestamp(999_001) + ); + // TimestampNtz: subtracts 999us + assert_eq!( + adjust_scalar_for_max_stat_truncation(&Scalar::TimestampNtz(1_000_000)), + Scalar::TimestampNtz(999_001) + ); + // Non-timestamp: unchanged + assert_eq!( + adjust_scalar_for_max_stat_truncation(&Scalar::from(42i64)), + Scalar::from(42i64) + ); + // Saturating at i64::MIN + assert_eq!( + adjust_scalar_for_max_stat_truncation(&Scalar::Timestamp(i64::MIN)), + Scalar::Timestamp(i64::MIN) + ); + // Near-zero: goes negative + assert_eq!( + adjust_scalar_for_max_stat_truncation(&Scalar::Timestamp(500)), + Scalar::Timestamp(-499) ); } @@ -407,34 +487,573 @@ fn test_checkpoint_skipping_semantic( #[case] expected: Option, #[case] description: &str, ) { - let pred = Pred::gt(column_expr!("x"), Scalar::from(100)); - let skipping_pred = as_checkpoint_skipping_predicate(&pred, &[]).unwrap(); - let resolver = HashMap::from_iter([(column_name!("maxValues.x"), max_val)]); + let pred = Pred::gt(col!("x"), lit(100)); + let stats = all_referenced_columns(&pred); + let skipping_pred = + as_checkpoint_skipping_predicate(&pred, &HashSet::new(), &HashSet::new(), &stats).unwrap(); + let resolver = HashMap::from_iter([(column_name!("stats_parsed.maxValues.x"), max_val)]); let filter = DefaultKernelPredicateEvaluator::from(resolver); expect_eq!(filter.eval(&skipping_pred), expected, "{description}"); } -// Verifies that the IS NULL guard changes behavior compared to a regular data skipping predicate: -// without the guard, null stats produce NULL (unknown); with the guard, they produce TRUE (keep). +// These values model the partition footer aggregate for `part_col = "B"`. A Remove-only group has +// no min/max and is kept as unknown. When a non-matching Add with value "A" shares a group with a +// Remove, parquet ignores the Remove's null and reports "A", allowing the whole group to be pruned. +#[rstest] +#[case::remove_only(Scalar::Null(DataType::STRING), NULL)] +#[case::remove_with_non_matching_add(Scalar::from("A"), FALSE)] +fn test_checkpoint_skipping_partition_comparison_with_remove( + #[case] footer_value: Scalar, + #[case] expected: Option, +) { + let partition_columns = HashSet::from([column_name!("part_col")]); + let pred = Pred::eq(col!("part_col"), lit("B")); + let skipping_pred = as_checkpoint_skipping_predicate( + &pred, + &partition_columns, + &HashSet::new(), + &HashSet::new(), + ) + .unwrap(); + let resolver = HashMap::from_iter([( + column_name!("partitionValues_parsed.part_col"), + footer_value, + )]); + let filter = DefaultKernelPredicateEvaluator::from(resolver); + expect_eq!(filter.eval(&skipping_pred), expected, "{pred:?}"); +} + +#[rstest] +#[case::eq_lit_above(Pred::eq(col!("part_col"), lit("m")), "b", FALSE)] +#[case::eq_hit(Pred::eq(col!("part_col"), lit("b")), "b", TRUE)] +#[case::eq_lit_below(Pred::eq(col!("part_col"), lit("a")), "b", FALSE)] +#[case::neq_miss(Pred::ne(col!("part_col"), lit("b")), "b", FALSE)] +#[case::neq_hit(Pred::ne(col!("part_col"), lit("m")), "b", TRUE)] +#[case::lt_miss(Pred::lt(col!("part_col"), lit("a")), "b", FALSE)] +#[case::lt_hit(Pred::lt(col!("part_col"), lit("c")), "b", TRUE)] +#[case::le_boundary(Pred::le(col!("part_col"), lit("b")), "b", TRUE)] +#[case::gt_miss(Pred::gt(col!("part_col"), lit("z")), "b", FALSE)] +#[case::gt_hit(Pred::gt(col!("part_col"), lit("a")), "b", TRUE)] +#[case::ge_boundary(Pred::ge(col!("part_col"), lit("b")), "b", TRUE)] +fn test_checkpoint_skipping_partition_range_ops( + #[case] pred: Pred, + #[case] value: &str, + #[case] expected: Option, +) { + let partition_columns = HashSet::from([column_name!("part_col")]); + let stats = HashSet::new(); + let skipping_pred = + as_checkpoint_skipping_predicate(&pred, &partition_columns, &HashSet::new(), &stats) + .unwrap(); + let resolver = DefaultKernelPredicateEvaluator::from(HashMap::from_iter([( + column_name!("partitionValues_parsed.part_col"), + Scalar::from(value), + )])); + expect_eq!( + resolver.eval(&skipping_pred), + expected, + "value={value} {pred:?}" + ); +} + +#[rstest] +#[case::float(Scalar::Float(1.0))] +#[case::double(Scalar::Double(1.0))] +#[case::integer_literal(Scalar::from(1))] +fn test_checkpoint_skipping_floating_partition_comparison_is_disabled(#[case] value: Scalar) { + let partition_columns = HashSet::from([column_name!("part_col")]); + let pred = Pred::ne(col!("part_col"), value.clone()); + let skipping_pred = as_checkpoint_skipping_predicate( + &pred, + &partition_columns, + &partition_columns, + &HashSet::new(), + ) + .unwrap(); + let resolver = DefaultKernelPredicateEvaluator::from(HashMap::from_iter([( + column_name!("partitionValues_parsed.part_col"), + value, + )])); + + expect_eq!( + resolver.eval(&skipping_pred), + NULL, + "parquet footer min/max exclude NaN partition values" + ); +} + +#[test] +fn test_checkpoint_skipping_floating_partition_cast_rewrites_exact_value() { + let partition_columns = HashSet::from([column_name!("part_col")]); + let pred = Pred::eq( + Expr::cast(col!("part_col"), DataType::INTEGER, CastOptions::default()), + lit(42), + ); + + let skipping_pred = as_checkpoint_skipping_predicate( + &pred, + &partition_columns, + &partition_columns, + &HashSet::new(), + ); + + assert_eq!( + skipping_pred.map(|pred| pred.to_string()).as_deref(), + Some("CAST(Column(partitionValues_parsed.part_col) AS integer) = 42") + ); +} + +#[test] +fn checkpoint_partition_cast_preserves_options() { + let partition_columns = HashSet::from([column_name!("part_ts")]); + let pred = Pred::gt( + Expr::cast( + col!("part_ts"), + DataType::TIMESTAMP, + CastOptions::default().with_timestamp_timezone("America/Los_Angeles"), + ), + Scalar::Timestamp(1_705_334_400_000_000), + ); + + let skipping_pred = as_checkpoint_skipping_predicate( + &pred, + &partition_columns, + &HashSet::new(), + &HashSet::new(), + ) + .unwrap(); + + assert_eq!( + skipping_pred.to_string(), + concat!( + "CAST(Column(partitionValues_parsed.part_ts) AS timestamp USING ", + "\"America/Los_Angeles\") > 1705334400000000" + ) + ); +} + +#[rstest] +#[case::range( + { + let cast = Expr::cast( + col!("part_col"), + DataType::DATE, + CastOptions::default(), + ); + Pred::and( + Pred::ge(cast.clone(), Scalar::Date(20_641)), + Pred::lt(cast, Scalar::Date(20_644)), + ) + }, + Some(concat!( + "AND(NOT(CAST(Column(partitionValues_parsed.part_col) AS date) < 20641), ", + "CAST(Column(partitionValues_parsed.part_col) AS date) < 20644)" + )), +)] +#[case::equality( + Pred::eq( + Expr::cast( + col!("part_col"), + DataType::DATE, + CastOptions::default(), + ), + Scalar::Date(20_641), + ), + Some("CAST(Column(partitionValues_parsed.part_col) AS date) = 20641"), +)] +#[case::inverted_less( + Pred::ge( + Expr::cast( + col!("part_col"), + DataType::DATE, + CastOptions::default(), + ), + Scalar::Date(20_641), + ), + Some("NOT(CAST(Column(partitionValues_parsed.part_col) AS date) < 20641)"), +)] +#[case::inverted_greater( + Pred::le( + Expr::cast( + col!("part_col"), + DataType::DATE, + CastOptions::default(), + ), + Scalar::Date(20_641), + ), + Some("NOT(CAST(Column(partitionValues_parsed.part_col) AS date) > 20641)"), +)] +#[case::inverted_equality( + Pred::ne( + Expr::cast( + col!("part_col"), + DataType::DATE, + CastOptions::default(), + ), + Scalar::Date(20_641), + ), + Some("NOT(CAST(Column(partitionValues_parsed.part_col) AS date) = 20641)"), +)] +#[case::literal_on_left( + Pred::lt( + Scalar::Date(20_641), + Expr::cast( + col!("part_col"), + DataType::DATE, + CastOptions::default(), + ), + ), + Some("CAST(Column(partitionValues_parsed.part_col) AS date) > 20641"), +)] +#[case::distinct( + Pred::binary( + BinaryPredicateOp::Distinct, + Expr::cast( + col!("part_col"), + DataType::DATE, + CastOptions::default(), + ), + Scalar::Date(20_641), + ), + None, +)] +#[case::in_list( + Pred::binary( + BinaryPredicateOp::In, + Expr::cast( + col!("part_col"), + DataType::DATE, + CastOptions::default(), + ), + Scalar::Date(20_641), + ), + None, +)] +fn test_checkpoint_skipping_partition_date_cast_comparisons( + #[case] pred: Pred, + #[case] expected: Option<&str>, +) { + let partition_columns = test_partition_columns(); + let actual = as_checkpoint_skipping_predicate( + &pred, + &partition_columns, + &HashSet::new(), + &HashSet::new(), + ) + .map(|pred| pred.to_string()); + + assert_eq!(actual.as_deref(), expected); +} + +fn partition_date_cast() -> Expr { + Expr::cast(col!("part_col"), DataType::DATE, CastOptions::default()) +} + +#[rstest] +#[case::eq_match( + Pred::eq(partition_date_cast(), Scalar::Date(20_641)), + "2026-07-07", + TRUE +)] +#[case::eq_nomatch( + Pred::eq(partition_date_cast(), Scalar::Date(20_641)), + "2026-07-08", + FALSE +)] +#[case::lt_true( + Pred::lt(partition_date_cast(), Scalar::Date(20_641)), + "2026-07-06", + TRUE +)] +#[case::lt_false( + Pred::lt(partition_date_cast(), Scalar::Date(20_641)), + "2026-07-07", + FALSE +)] +#[case::gt_true( + Pred::gt(partition_date_cast(), Scalar::Date(20_641)), + "2026-07-08", + TRUE +)] +#[case::gt_false( + Pred::gt(partition_date_cast(), Scalar::Date(20_641)), + "2026-07-07", + FALSE +)] +#[case::ge_boundary( + Pred::ge(partition_date_cast(), Scalar::Date(20_641)), + "2026-07-07", + TRUE +)] +#[case::le_boundary( + Pred::le(partition_date_cast(), Scalar::Date(20_641)), + "2026-07-07", + TRUE +)] +#[case::ne_match( + Pred::ne(partition_date_cast(), Scalar::Date(20_641)), + "2026-07-07", + FALSE +)] +#[case::ne_nomatch( + Pred::ne(partition_date_cast(), Scalar::Date(20_641)), + "2026-07-08", + TRUE +)] +fn test_checkpoint_partition_cast_eval_discriminates_per_operator( + #[case] pred: Pred, + #[case] partition_value: &str, + #[case] expected: Option, +) { + let skipping_pred = as_checkpoint_skipping_predicate( + &pred, + &test_partition_columns(), + &HashSet::new(), + &HashSet::new(), + ) + .unwrap(); + let resolver = DefaultKernelPredicateEvaluator::from(HashMap::from_iter([( + column_name!("partitionValues_parsed.part_col"), + Scalar::from(partition_value), + )])); + + expect_eq!( + resolver.eval(&skipping_pred), + expected, + "{pred:#?} @ {partition_value}" + ); +} + +#[test] +fn test_partition_date_cast_is_checkpoint_only() { + let partition_columns = test_partition_columns(); + let partition_cast = Pred::eq( + Expr::cast(col!("part_col"), DataType::DATE, CastOptions::default()), + Scalar::Date(20_641), + ); + let data_cast = Pred::eq( + Expr::cast(col!("data_col"), DataType::DATE, CastOptions::default()), + Scalar::Date(20_641), + ); + + assert!( + as_checkpoint_skipping_predicate( + &data_cast, + &partition_columns, + &HashSet::new(), + &HashSet::from([column_name!("data_col")]), + ) + .is_none(), + "checkpoint CAST pushdown requires an exact partition value" + ); + assert!( + as_data_skipping_predicate_with_partitions(&partition_cast, &partition_columns).is_none(), + "in-memory data skipping must not evaluate partition casts" + ); +} + +#[rstest] +#[case::null_value(Scalar::Null(DataType::STRING))] +#[case::invalid_value(Scalar::from("not-a-date"))] +fn test_checkpoint_partition_cast_reference_eval_is_conservative(#[case] partition_value: Scalar) { + let partition_columns = test_partition_columns(); + let pred = Pred::eq( + Expr::cast(col!("part_col"), DataType::DATE, CastOptions::default()), + Scalar::Date(20_641), + ); + let skipping_pred = as_checkpoint_skipping_predicate( + &pred, + &partition_columns, + &HashSet::new(), + &HashSet::new(), + ) + .unwrap(); + let resolver = DefaultKernelPredicateEvaluator::from(HashMap::from_iter([( + column_name!("partitionValues_parsed.part_col"), + partition_value, + )])); + + expect_eq!( + resolver.eval(&skipping_pred), + NULL, + "null and invalid partition values must evaluate to NULL" + ); +} + +#[test] +fn partition_cast_preserves_supported_in_memory_conjunct() { + let partition_columns = test_partition_columns(); + let pred = Pred::and( + Pred::ge( + Expr::cast(col!("part_col"), DataType::DATE, CastOptions::default()), + Scalar::Date(20_641), + ), + Pred::gt(col!("data_col"), lit(100)), + ); + + let skipping_pred = + as_data_skipping_predicate_with_partitions(&pred, &partition_columns).unwrap(); + + assert_eq!( + skipping_pred.to_string(), + "AND(null, Column(stats_parsed.maxValues.data_col) > 100)" + ); + + let checkpoint_pred = as_checkpoint_skipping_predicate( + &pred, + &partition_columns, + &HashSet::new(), + &HashSet::from([column_name!("data_col")]), + ) + .unwrap() + .to_string(); + assert!( + checkpoint_pred.contains("CAST(Column(partitionValues_parsed.part_col) AS date)"), + "{checkpoint_pred}" + ); + assert!( + checkpoint_pred.contains("stats_parsed.maxValues.data_col"), + "{checkpoint_pred}" + ); +} + +// Partition null predicates read `partitionValues_parsed.part_col` directly. FALSE is the pruning +// verdict; TRUE keeps the row group. +#[rstest] +#[case::is_null_null(false, Scalar::Null(DataType::STRING), TRUE)] +#[case::is_null_value(false, Scalar::from("x"), FALSE)] +#[case::is_not_null_null(true, Scalar::Null(DataType::STRING), FALSE)] +#[case::is_not_null_value(true, Scalar::from("x"), TRUE)] +fn test_checkpoint_skipping_partition_null_predicates( + #[case] is_not_null: bool, + #[case] value: Scalar, + #[case] expected: Option, +) { + let partition_columns = HashSet::from([column_name!("part_col")]); + let pred = if is_not_null { + Pred::is_not_null(col!("part_col")) + } else { + Pred::is_null(col!("part_col")) + }; + let skipping_pred = as_checkpoint_skipping_predicate( + &pred, + &partition_columns, + &HashSet::new(), + &HashSet::new(), + ) + .unwrap(); + let resolver = DefaultKernelPredicateEvaluator::from(HashMap::from_iter([( + column_name!("partitionValues_parsed.part_col"), + value, + )])); + expect_eq!(resolver.eval(&skipping_pred), expected, "{pred:?}"); +} + +// When a checkpoint omits `partitionValues_parsed` (for example, structured checkpoint stats are +// disabled), every referenced stat is unavailable, so the predicate remains unknown and keeps the +// row group. Simulated with a resolver that has no columns. +#[test] +fn test_checkpoint_skipping_partition_missing_stats_keeps_all() { + let partition_columns = HashSet::from([column_name!("part_col")]); + let stats = HashSet::new(); + for pred in [ + Pred::eq(col!("part_col"), lit("B")), + Pred::lt(col!("part_col"), lit("B")), + Pred::eq( + Expr::cast(col!("part_col"), DataType::DATE, CastOptions::default()), + Scalar::Date(20_641), + ), + Pred::is_null(col!("part_col")), + Pred::is_not_null(col!("part_col")), + ] { + let skipping_pred = + as_checkpoint_skipping_predicate(&pred, &partition_columns, &HashSet::new(), &stats) + .unwrap(); + let filter = DefaultKernelPredicateEvaluator::from(EmptyColumnResolver); + expect_eq!( + filter.eval(&skipping_pred), + NULL, + "missing partition stats must never prune: {pred:?}" + ); + } +} + +#[rstest] +#[case::and_partition_prunes(true, "z", 500, FALSE)] +#[case::and_data_prunes(true, "b", 40, FALSE)] +#[case::and_both_keep(true, "b", 500, TRUE)] +#[case::or_both_miss(false, "z", 40, FALSE)] +#[case::or_partition_keeps(false, "b", 40, TRUE)] +#[case::or_data_keeps(false, "z", 500, TRUE)] +fn test_checkpoint_skipping_mixed_partition_and_data( + #[case] is_and: bool, + #[case] part_val: &str, + #[case] data_max: i64, + #[case] expected: Option, +) { + let partition_columns = HashSet::from([column_name!("part_col")]); + let stats: HashSet = [column_name!("data_col")].into_iter().collect(); + let part = Pred::eq(col!("part_col"), lit("b")); + let data = Pred::gt(col!("data_col"), lit(100i64)); + let pred = if is_and { + Pred::and(part, data) + } else { + Pred::or(part, data) + }; + let skipping_pred = + as_checkpoint_skipping_predicate(&pred, &partition_columns, &HashSet::new(), &stats) + .unwrap(); + let resolver = DefaultKernelPredicateEvaluator::from(HashMap::from_iter([ + ( + column_name!("partitionValues_parsed.part_col"), + Scalar::from(part_val), + ), + ( + column_name!("stats_parsed.maxValues.data_col"), + Scalar::from(data_max), + ), + ])); + expect_eq!(resolver.eval(&skipping_pred), expected, "{pred:?}"); +} + +#[test] +fn test_checkpoint_skipping_partition_timestamp_no_truncation_adjustment() { + let partition_columns = HashSet::from([column_name!("part_ts")]); + let stats = HashSet::new(); + let pred = Pred::gt(col!("part_ts"), Scalar::Timestamp(1_000_000)); + let skipping_pred = + as_checkpoint_skipping_predicate(&pred, &partition_columns, &HashSet::new(), &stats) + .unwrap(); + assert_eq!( + skipping_pred.to_string(), + "Column(partitionValues_parsed.part_ts) > 1000000", + "partition timestamp must not get the 999us data-column truncation adjustment" + ); +} + #[test] fn test_checkpoint_skipping_null_guard_vs_regular() { - let pred = Pred::gt(column_expr!("x"), Scalar::from(100)); - let resolver = - HashMap::from_iter([(column_name!("maxValues.x"), Scalar::Null(DataType::INTEGER))]); + let pred = Pred::gt(col!("x"), lit(100)); + let resolver = HashMap::from_iter([( + column_name!("stats_parsed.maxValues.x"), + Scalar::Null(DataType::INTEGER), + )]); let filter = DefaultKernelPredicateEvaluator::from(resolver); - let guarded = as_checkpoint_skipping_predicate(&pred, &[]).unwrap(); + let stats = all_referenced_columns(&pred); + let guarded = + as_checkpoint_skipping_predicate(&pred, &HashSet::new(), &HashSet::new(), &stats).unwrap(); expect_eq!( filter.eval(&guarded), TRUE, - "guarded pred with null stats → TRUE (keep)" + "guarded pred with null stats -> TRUE (keep)" ); let regular = as_data_skipping_predicate(&pred).unwrap(); expect_eq!( filter.eval(®ular), NULL, - "regular pred with null stats → NULL (unknown)" + "regular pred with null stats -> NULL (unknown)" ); } @@ -442,105 +1061,199 @@ fn test_checkpoint_skipping_null_guard_vs_regular() { // column's stats are sufficient. For `col_a > 100 AND col_b < 50`, the guarded predicate is: // // AND( -// OR(maxValues.col_a IS NULL, maxValues.col_a > 100), -// OR(minValues.col_b IS NULL, minValues.col_b < 50) +// OR(stats_parsed.maxValues.col_a IS NULL, stats_parsed.maxValues.col_a > 100), +// OR(stats_parsed.minValues.col_b IS NULL, stats_parsed.minValues.col_b < 50) // ) // // Even if col_a's stats are null, col_b's stats alone can prune the row group. #[test] fn test_checkpoint_skipping_conjunction_partial_null_stats() { let pred = Pred::and( - Pred::gt(column_expr!("col_a"), Scalar::from(100)), - Pred::lt(column_expr!("col_b"), Scalar::from(50)), + Pred::gt(col!("col_a"), lit(100)), + Pred::lt(col!("col_b"), lit(50)), ); - let skipping_pred = as_checkpoint_skipping_predicate(&pred, &[]).unwrap(); + let stats = all_referenced_columns(&pred); + let skipping_pred = + as_checkpoint_skipping_predicate(&pred, &HashSet::new(), &HashSet::new(), &stats).unwrap(); - // Both stats present and both allow pruning → skip + // Both stats present and both allow pruning -> skip let resolver = HashMap::from_iter([ - (column_name!("maxValues.col_a"), Scalar::from(50)), - (column_name!("minValues.col_b"), Scalar::from(60)), + ( + column_name!("stats_parsed.maxValues.col_a"), + Scalar::from(50), + ), + ( + column_name!("stats_parsed.minValues.col_b"), + Scalar::from(60), + ), ]); let filter = DefaultKernelPredicateEvaluator::from(resolver); expect_eq!( filter.eval(&skipping_pred), FALSE, - "both cols prunable → skip" + "both cols prunable -> skip" ); - // col_a stats null, but col_b stats alone are enough to prune → still skip + // col_a stats null, but col_b stats alone are enough to prune -> still skip let resolver = HashMap::from_iter([ ( - column_name!("maxValues.col_a"), + column_name!("stats_parsed.maxValues.col_a"), Scalar::Null(DataType::INTEGER), ), - (column_name!("minValues.col_b"), Scalar::from(60)), + ( + column_name!("stats_parsed.minValues.col_b"), + Scalar::from(60), + ), ]); let filter = DefaultKernelPredicateEvaluator::from(resolver); expect_eq!( filter.eval(&skipping_pred), FALSE, - "col_a null but col_b prunable → still skip" + "col_a null but col_b prunable -> still skip" ); - // col_a stats null and col_b doesn't allow pruning → keep + // col_a stats null and col_b doesn't allow pruning -> keep let resolver = HashMap::from_iter([ ( - column_name!("maxValues.col_a"), + column_name!("stats_parsed.maxValues.col_a"), Scalar::Null(DataType::INTEGER), ), - (column_name!("minValues.col_b"), Scalar::from(30)), + ( + column_name!("stats_parsed.minValues.col_b"), + Scalar::from(30), + ), ]); let filter = DefaultKernelPredicateEvaluator::from(resolver); expect_eq!( filter.eval(&skipping_pred), TRUE, - "col_a null and col_b not prunable → keep" + "col_a null and col_b not prunable -> keep" ); } -// TODO(#1002): we currently don't support file skipping on timestamp columns' max stat since they -// are truncated to milliseconds in add.stats. -#[test] -fn test_timestamp_predicates_dont_data_skip() { - let col = &column_expr!("ts_col"); - for timestamp in [&Scalar::Timestamp(1000000), &Scalar::TimestampNtz(1000000)] { - // LT will do minValues -> OK - let pred = Pred::lt(col.clone(), timestamp.clone()); - let skipping_pred = as_data_skipping_predicate(&pred); - assert_eq!( - skipping_pred.unwrap().to_string(), - "Column(stats_parsed.minValues.ts_col) < 1000000" - ); +// Verifies the null-guarded checkpoint skipping path also applies the 999us timestamp +// truncation adjustment to max stat comparisons. +#[rstest] +fn test_checkpoint_skipping_timestamp_adjustment( + #[values(Scalar::Timestamp(1_000_000), Scalar::TimestampNtz(1_000_000))] timestamp: Scalar, +) { + let col = &col!("ts_col"); - // GT will do maxValues -> BLOCKED - let pred = Pred::gt(col.clone(), timestamp.clone()); - let skipping_pred = as_data_skipping_predicate(&pred); - assert!( - skipping_pred.is_none(), - "Expected no data skipping for timestamp predicate: {pred:#?}, got {skipping_pred:#?}" - ); + // GT: should produce OR(maxValues.ts_col IS NULL, maxValues.ts_col > 999001) + let pred = Pred::gt(col.clone(), timestamp.clone()); + let stats = all_referenced_columns(&pred); + let skipping_pred = + as_checkpoint_skipping_predicate(&pred, &HashSet::new(), &HashSet::new(), &stats).unwrap(); + assert_eq!( + skipping_pred.to_string(), + "OR(Column(stats_parsed.maxValues.ts_col) IS NULL, \ + Column(stats_parsed.maxValues.ts_col) > 999001)" + ); - let pred = Pred::eq(col.clone(), timestamp.clone()); - let skipping_pred = as_data_skipping_predicate(&pred); - assert_eq!( - skipping_pred.unwrap().to_string(), - "AND(NOT(Column(stats_parsed.minValues.ts_col) > 1000000), null)" - ); + // EQ: max stat leg should use adjusted literal + let pred = Pred::eq(col.clone(), timestamp.clone()); + let stats = all_referenced_columns(&pred); + let skipping_pred = + as_checkpoint_skipping_predicate(&pred, &HashSet::new(), &HashSet::new(), &stats).unwrap(); + assert_eq!( + skipping_pred.to_string(), + "AND(OR(Column(stats_parsed.minValues.ts_col) IS NULL, \ + NOT(Column(stats_parsed.minValues.ts_col) > 1000000)), \ + OR(Column(stats_parsed.maxValues.ts_col) IS NULL, \ + NOT(Column(stats_parsed.maxValues.ts_col) < 999001)))" + ); +} - let pred = Pred::ne(col.clone(), timestamp.clone()); - let skipping_pred = as_data_skipping_predicate(&pred); - assert_eq!( - skipping_pred.unwrap().to_string(), - "OR(NOT(Column(stats_parsed.minValues.ts_col) = 1000000), null)" - ); - } +// Timestamp predicates use max stats with a 999us adjustment to account for millisecond +// truncation in Delta JSON stats. +#[rstest] +fn test_timestamp_predicates_use_adjusted_max_stats( + #[values(Scalar::Timestamp(1_000_000), Scalar::TimestampNtz(1_000_000))] timestamp: Scalar, +) { + let col = &col!("ts_col"); + + // LT uses minValues (no adjustment needed for min stats) + let pred = Pred::lt(col.clone(), timestamp.clone()); + assert_eq!( + as_data_skipping_predicate(&pred).unwrap().to_string(), + "Column(stats_parsed.minValues.ts_col) < 1000000" + ); + + // GT uses maxValues with adjusted literal (1000000 - 999 = 999001) + let pred = Pred::gt(col.clone(), timestamp.clone()); + assert_eq!( + as_data_skipping_predicate(&pred).unwrap().to_string(), + "Column(stats_parsed.maxValues.ts_col) > 999001" + ); + + // EQ uses both min (unadjusted) and max (adjusted) + let pred = Pred::eq(col.clone(), timestamp.clone()); + assert_eq!( + as_data_skipping_predicate(&pred).unwrap().to_string(), + "AND(NOT(Column(stats_parsed.minValues.ts_col) > 1000000), \ + NOT(Column(stats_parsed.maxValues.ts_col) < 999001))" + ); + + // NE uses both min (unadjusted) and max (adjusted) + let pred = Pred::ne(col.clone(), timestamp.clone()); + assert_eq!( + as_data_skipping_predicate(&pred).unwrap().to_string(), + "OR(NOT(Column(stats_parsed.minValues.ts_col) = 1000000), \ + NOT(Column(stats_parsed.maxValues.ts_col) = 999001))" + ); + + // GE (col >= val) uses maxValues with adjusted literal + let pred = Pred::ge(col.clone(), timestamp.clone()); + assert_eq!( + as_data_skipping_predicate(&pred).unwrap().to_string(), + "NOT(Column(stats_parsed.maxValues.ts_col) < 999001)" + ); + + // LE (col <= val) uses minValues only (no adjustment needed) + let pred = Pred::le(col.clone(), timestamp.clone()); + assert_eq!( + as_data_skipping_predicate(&pred).unwrap().to_string(), + "NOT(Column(stats_parsed.minValues.ts_col) > 1000000)" + ); +} + +// Partition timestamp columns use exact values (not truncated), so no adjustment is applied. +#[test] +fn test_partition_timestamp_column_no_adjustment() { + let partition_columns: HashSet = [column_name!("ts_part")].into(); + let pred = Pred::gt(col!("ts_part"), Scalar::Timestamp(1_000_000)); + let skipping_pred = + as_data_skipping_predicate_with_partitions(&pred, &partition_columns).unwrap(); + assert_eq!( + skipping_pred.to_string(), + "OR(NOT(Column(is_add)), Column(partitionValues_parsed.ts_part) > 1000000)" + ); +} + +#[rstest] +fn test_interval_partition_columns_are_not_pruning_columns( + #[values(DataType::INTERVAL_YEAR_MONTH, DataType::INTERVAL_DAY_TIME)] interval: DataType, +) { + let partition_schema = Arc::new(StructType::new_unchecked([StructField::nullable( + "period", interval, + )])); + let (_, _, partition_columns) = DataSkippingFilter::build_unified_schema_and_expr( + None, + column_expr_ref!("stats_parsed"), + Some(&partition_schema), + column_expr_ref!("partitionValues_parsed"), + Arc::new(Expr::literal(true)), + ) + .unwrap(); + + assert!(partition_columns.is_empty()); } // Tests for partition-aware data skipping /// Helper to build a partition columns set with a single "part_col" entry. -fn test_partition_columns() -> HashSet { - ["part_col".to_string()].into() +fn test_partition_columns() -> HashSet { + [column_name!("part_col")].into() } /// Helper to build a resolver for mixed partition + data stats evaluation. @@ -557,6 +1270,7 @@ fn mixed_resolver( column_name!("stats_parsed.maxValues.data_col"), Scalar::from(max_data), ), + (column_name!("is_add"), Scalar::from(true)), ])) } @@ -565,7 +1279,7 @@ fn test_partition_column_rewrite() { let partition_columns = test_partition_columns(); // Partition column equality rewrites to partitionValues (not minValues/maxValues) - let pred = Pred::eq(column_expr!("part_col"), Scalar::from("2025-01-01")); + let pred = Pred::eq(col!("part_col"), lit("2025-01-01")); let skipping_pred = as_data_skipping_predicate_with_partitions(&pred, &partition_columns); let pred_str = skipping_pred.as_ref().map(|p| p.to_string()); assert!( @@ -577,12 +1291,12 @@ fn test_partition_column_rewrite() { assert!( pred_str .as_ref() - .is_some_and(|s| !s.contains("minValues") && !s.contains("maxValues")), + .is_some_and(|s| !s.contains(MIN_VALUES) && !s.contains(MAX_VALUES)), "Should not contain minValues/maxValues for partition columns" ); // Data column still rewrites to stats_parsed.minValues/maxValues - let pred = Pred::gt(column_expr!("data_col"), Scalar::from(100)); + let pred = Pred::gt(col!("data_col"), lit(100)); let skipping_pred = as_data_skipping_predicate_with_partitions(&pred, &partition_columns); let pred_str = skipping_pred.as_ref().map(|p| p.to_string()); assert!( @@ -595,12 +1309,12 @@ fn test_partition_column_rewrite() { #[rstest] #[case::is_null( - Pred::is_null(column_expr!("part_col")), - "Column(partitionValues_parsed.part_col) IS NULL" + Pred::is_null(col!("part_col")), + "OR(NOT(Column(is_add)), Column(partitionValues_parsed.part_col) IS NULL)" )] #[case::is_not_null( - Pred::is_not_null(column_expr!("part_col")), - "NOT(Column(partitionValues_parsed.part_col) IS NULL)" + Pred::is_not_null(col!("part_col")), + "OR(NOT(Column(is_add)), NOT(Column(partitionValues_parsed.part_col) IS NULL))" )] fn test_partition_column_is_null(#[case] pred: Pred, #[case] expected: &str) { let partition_columns = test_partition_columns(); @@ -619,8 +1333,8 @@ fn test_mixed_partition_and_data_or_predicate() { // This should produce a valid skipping predicate (not None) because both // operands are now eligible for data skipping. let pred = Pred::or( - Pred::eq(column_expr!("part_col"), Scalar::from("X")), - Pred::gt(column_expr!("data_col"), Scalar::from(100)), + Pred::eq(col!("part_col"), lit("X")), + Pred::gt(col!("data_col"), lit(100)), ); let skipping_pred = as_data_skipping_predicate_with_partitions(&pred, &partition_columns); assert!( @@ -655,8 +1369,8 @@ fn test_mixed_partition_and_data_or_evaluation( // WHERE part_col = 'X' OR data_col > 100 let pred = Pred::or( - Pred::eq(column_expr!("part_col"), Scalar::from("X")), - Pred::gt(column_expr!("data_col"), Scalar::from(100)), + Pred::eq(col!("part_col"), lit("X")), + Pred::gt(col!("data_col"), lit(100)), ); let skipping_pred = as_data_skipping_predicate_with_partitions(&pred, &partition_columns) .expect("should exist"); @@ -683,8 +1397,8 @@ fn test_mixed_partition_and_data_and_evaluation( // WHERE part_col = 'X' AND data_col > 100 let pred = Pred::and( - Pred::eq(column_expr!("part_col"), Scalar::from("X")), - Pred::gt(column_expr!("data_col"), Scalar::from(100)), + Pred::eq(col!("part_col"), lit("X")), + Pred::gt(col!("data_col"), lit(100)), ); let skipping_pred = as_data_skipping_predicate_with_partitions(&pred, &partition_columns) .expect("should exist"); @@ -702,22 +1416,28 @@ fn test_partition_column_comparison_uses_exact_value() { let partition_columns = test_partition_columns(); // part_col > 'B' rewrites both min and max to partitionValues_parsed.part_col - let pred = Pred::gt(column_expr!("part_col"), Scalar::from("B")); + let pred = Pred::gt(col!("part_col"), lit("B")); let skipping_pred = as_data_skipping_predicate_with_partitions(&pred, &partition_columns) .expect("should exist"); // part_col='A': 'A' > 'B' is false -> skip - let resolver = DefaultKernelPredicateEvaluator::from(HashMap::from_iter([( - column_name!("partitionValues_parsed.part_col"), - Scalar::from("A"), - )])); + let resolver = DefaultKernelPredicateEvaluator::from(HashMap::from_iter([ + ( + column_name!("partitionValues_parsed.part_col"), + Scalar::from("A"), + ), + (column_name!("is_add"), Scalar::from(true)), + ])); assert_eq!(resolver.eval(&skipping_pred), FALSE); // part_col='C': 'C' > 'B' is true -> keep - let resolver = DefaultKernelPredicateEvaluator::from(HashMap::from_iter([( - column_name!("partitionValues_parsed.part_col"), - Scalar::from("C"), - )])); + let resolver = DefaultKernelPredicateEvaluator::from(HashMap::from_iter([ + ( + column_name!("partitionValues_parsed.part_col"), + Scalar::from("C"), + ), + (column_name!("is_add"), Scalar::from(true)), + ])); assert_eq!(resolver.eval(&skipping_pred), TRUE); } @@ -726,7 +1446,7 @@ fn test_partition_only_predicate() { let partition_columns = test_partition_columns(); // Partition-only: no data columns involved - let pred = Pred::eq(column_expr!("part_col"), Scalar::from("X")); + let pred = Pred::eq(col!("part_col"), lit("X")); let skipping_pred = as_data_skipping_predicate_with_partitions(&pred, &partition_columns) .expect("should exist"); let pred_str = skipping_pred.to_string(); @@ -745,7 +1465,7 @@ fn test_sql_where_partition_rewrite() { let partition_columns = test_partition_columns(); // Partition column equality: SQL WHERE should rewrite to partitionValues_parsed - let pred = Pred::eq(column_expr!("part_col"), Scalar::from("X")); + let pred = Pred::eq(col!("part_col"), lit("X")); let sql_pred = as_sql_data_skipping_predicate(&pred, &partition_columns) .expect("partition eq should produce SQL skipping pred"); let pred_str = sql_pred.to_string(); @@ -769,8 +1489,8 @@ fn test_sql_where_mixed_partition_and_data_evaluation( // WHERE part_col = 'X' AND data_col > 100 let pred = Pred::and( - Pred::eq(column_expr!("part_col"), Scalar::from("X")), - Pred::gt(column_expr!("data_col"), Scalar::from(100)), + Pred::eq(col!("part_col"), lit("X")), + Pred::gt(col!("data_col"), lit(100)), ); let sql_pred = as_sql_data_skipping_predicate(&pred, &partition_columns) .expect("mixed AND should produce SQL skipping pred"); @@ -789,6 +1509,7 @@ fn test_sql_where_mixed_partition_and_data_evaluation( column_name!("stats_parsed.maxValues.data_col"), Scalar::from(max_data), ), + (column_name!("is_add"), Scalar::from(true)), ]); let filter = DefaultKernelPredicateEvaluator::from(resolver); assert_eq!( @@ -798,6 +1519,193 @@ fn test_sql_where_mixed_partition_and_data_evaluation( ); } +// The is_add guard (OR(NOT is_add, pred)) ensures Remove rows are never pruned by +// partition predicates, regardless of whether the partition value matches. +#[rstest] +#[case::non_matching_partition("Y", false, TRUE, "non-matching partition, Remove kept via guard")] +#[case::matching_partition("X", false, TRUE, "matching partition, Remove kept via guard")] +#[case::add_non_matching("Y", true, FALSE, "non-matching partition, Add correctly pruned")] +#[case::add_matching("X", true, TRUE, "matching partition, Add correctly kept")] +fn is_add_guard_keeps_remove_rows( + #[case] part_val: &str, + #[case] is_add: bool, + #[case] expected: Option, + #[case] _scenario: &str, +) { + let partition_columns = test_partition_columns(); + let pred = Pred::eq(col!("part_col"), lit("X")); + let skipping_pred = as_data_skipping_predicate_with_partitions(&pred, &partition_columns) + .expect("should exist"); + + let resolver = DefaultKernelPredicateEvaluator::from(HashMap::from_iter([ + ( + column_name!("partitionValues_parsed.part_col"), + Scalar::from(part_val), + ), + (column_name!("is_add"), Scalar::from(is_add)), + ])); + assert_eq!( + resolver.eval(&skipping_pred), + expected, + "part_col='{part_val}' is_add={is_add}" + ); +} + +// Mixed AND with is_add=false and null stats: Remove rows have null data stats, so the data +// arm evaluates to NULL. AND(true_from_guard, NULL) = NULL, which the DISTINCT filter treats +// as "keep". This verifies Removes are not pruned even when the data arm cannot be satisfied. +#[rstest] +#[case::remove_null_stats("Y", false, "Remove: AND(guard=true, stats=NULL) = NULL -> kept")] +#[case::add_null_stats_partition_match("X", true, "Add: AND(true, NULL) = NULL -> kept")] +#[case::add_null_stats_partition_miss("Y", true, "Add: AND(false, NULL) = false -> pruned")] +fn mixed_and_with_null_stats_and_is_add_guard( + #[case] part_val: &str, + #[case] is_add: bool, + #[case] _scenario: &str, +) { + let partition_columns = test_partition_columns(); + let pred = Pred::and( + Pred::eq(col!("part_col"), lit("X")), + Pred::gt(col!("data_col"), lit(100)), + ); + let skipping_pred = as_data_skipping_predicate_with_partitions(&pred, &partition_columns) + .expect("should exist"); + + let resolver = DefaultKernelPredicateEvaluator::from(HashMap::from_iter([ + ( + column_name!("partitionValues_parsed.part_col"), + Scalar::from(part_val), + ), + ( + column_name!("stats_parsed.maxValues.data_col"), + Scalar::Null(DataType::INTEGER), + ), + (column_name!("is_add"), Scalar::from(is_add)), + ])); + let result = resolver.eval(&skipping_pred); + if !is_add { + assert_ne!(result, FALSE, "Remove rows must never be pruned"); + } +} + +// Null partition values: IS NULL / IS NOT NULL predicates on partition columns must +// correctly evaluate against null values in partitionValues_parsed. +#[rstest] +#[case::is_null_with_null_value( + Pred::is_null(col!("part_col")), + Scalar::Null(DataType::STRING), + TRUE, + "null partition value matches IS NULL" +)] +#[case::is_null_with_non_null_value( + Pred::is_null(col!("part_col")), + Scalar::from("X"), + FALSE, + "non-null partition value rejected by IS NULL" +)] +#[case::is_not_null_with_null_value( + Pred::is_not_null(col!("part_col")), + Scalar::Null(DataType::STRING), + FALSE, + "null partition value rejected by IS NOT NULL" +)] +#[case::is_not_null_with_non_null_value( + Pred::is_not_null(col!("part_col")), + Scalar::from("X"), + TRUE, + "non-null partition value matches IS NOT NULL" +)] +fn null_partition_value_evaluation( + #[case] pred: Pred, + #[case] part_val: Scalar, + #[case] expected: Option, + #[case] _scenario: &str, +) { + let partition_columns = test_partition_columns(); + let skipping_pred = as_data_skipping_predicate_with_partitions(&pred, &partition_columns) + .expect("should exist"); + + let resolver = DefaultKernelPredicateEvaluator::from(HashMap::from_iter([ + (column_name!("partitionValues_parsed.part_col"), part_val), + (column_name!("is_add"), Scalar::from(true)), + ])); + assert_eq!(resolver.eval(&skipping_pred), expected); +} + +// Multiple partition columns: predicates referencing two partition columns should both +// rewrite to partitionValues_parsed and both get is_add guards. +#[test] +fn multiple_partition_columns_rewrite_and_evaluation() { + let partition_columns: HashSet = + [column_name!("part_a"), column_name!("part_b")].into(); + + let pred = Pred::and( + Pred::eq(col!("part_a"), lit("X")), + Pred::eq(col!("part_b"), lit("Y")), + ); + let skipping_pred = as_data_skipping_predicate_with_partitions(&pred, &partition_columns) + .expect("should exist"); + let pred_str = skipping_pred.to_string(); + assert!( + pred_str.contains("partitionValues_parsed.part_a"), + "Should reference partitionValues_parsed.part_a, got {pred_str}" + ); + assert!( + pred_str.contains("partitionValues_parsed.part_b"), + "Should reference partitionValues_parsed.part_b, got {pred_str}" + ); + assert!( + !pred_str.contains("stats_parsed"), + "Should not reference stats_parsed for partition-only pred, got {pred_str}" + ); + + // Both match -> kept + let resolver = DefaultKernelPredicateEvaluator::from(HashMap::from_iter([ + ( + column_name!("partitionValues_parsed.part_a"), + Scalar::from("X"), + ), + ( + column_name!("partitionValues_parsed.part_b"), + Scalar::from("Y"), + ), + (column_name!("is_add"), Scalar::from(true)), + ])); + assert_eq!(resolver.eval(&skipping_pred), TRUE); + + // First misses -> pruned + let resolver = DefaultKernelPredicateEvaluator::from(HashMap::from_iter([ + ( + column_name!("partitionValues_parsed.part_a"), + Scalar::from("Z"), + ), + ( + column_name!("partitionValues_parsed.part_b"), + Scalar::from("Y"), + ), + (column_name!("is_add"), Scalar::from(true)), + ])); + assert_eq!(resolver.eval(&skipping_pred), FALSE); + + // Remove row: both miss but is_add=false -> kept via guard + let resolver = DefaultKernelPredicateEvaluator::from(HashMap::from_iter([ + ( + column_name!("partitionValues_parsed.part_a"), + Scalar::from("Z"), + ), + ( + column_name!("partitionValues_parsed.part_b"), + Scalar::from("W"), + ), + (column_name!("is_add"), Scalar::from(false)), + ])); + assert_ne!( + resolver.eval(&skipping_pred), + FALSE, + "Remove must not be pruned" + ); +} + // Without normalization, `AND([unknown])` would become `AND([NULL])` via // `collect_junction_preds`, which evaluates to `Some(false)` under `eval_sql_where` and // incorrectly prunes all row groups. The junction constructor normalizes `AND([unknown])` @@ -805,9 +1713,414 @@ fn test_sql_where_mixed_partition_and_data_evaluation( #[test] fn single_unsupported_pred_in_junction_disables_checkpoint_pushdown() { let pred = Pred::and_from([Pred::unknown("unsupported")]); - let skipping_pred = as_checkpoint_skipping_predicate(&pred, &[]); + let stats = all_referenced_columns(&pred); + let skipping_pred = + as_checkpoint_skipping_predicate(&pred, &HashSet::new(), &HashSet::new(), &stats); assert!( skipping_pred.is_none(), "Single unsupported predicate in a junction should disable pushdown, got: {skipping_pred:?}" ); } + +// -- Integration tests: end-to-end data skipping with real tables ------------------- +// +// Two test tables are used: +// +// `app-txn-checkpoint` (4 files, partitioned by `modified` (string)): +// - 2 files: modified="2021-02-01", value in [4, 11] +// - 2 files: modified="2021-02-02", value in [1, 3] +// - Version 0 (JSON) + version 1 (JSON + checkpoint) exercises both code paths. +// +// `parsed-stats` (6 files, non-partitioned): +// - File 1-6: id ranges [1,100]..[501,600], ts_col min values 1M..11M microseconds +// - Version 3 checkpoint + versions 4-5 JSON commits. + +use std::path::PathBuf; + +use crate::engine::sync::SyncEngine; +use crate::Snapshot; + +/// Counts files selected after data skipping for the given predicate and table. +fn count_selected(table_dir: &str, pred: PredicateRef) -> usize { + let path = std::fs::canonicalize(PathBuf::from(table_dir)).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + let engine = Arc::new(SyncEngine::new()); + let scan = Snapshot::builder_for(url) + .build(engine.as_ref()) + .unwrap() + .scan_builder() + .with_predicate(pred) + .build() + .unwrap(); + scan.scan_metadata(engine.as_ref()) + .unwrap() + .collect::, _>>() + .unwrap() + .iter() + .flat_map(|sm| sm.scan_files.selection_vector()) + .filter(|&&s| s) + .count() +} + +const PARTITIONED_TABLE: &str = "./tests/data/app-txn-checkpoint/"; +const STATS_TABLE: &str = "./tests/data/parsed-stats/"; + +// -- Partition-only predicates (app-txn-checkpoint) --------------------------- + +#[rstest] +#[case::eq_match(Pred::eq(col!("modified"), lit("2021-02-01")), 2)] +#[case::eq_no_match(Pred::eq(col!("modified"), lit("2099-01-01")), 0)] +#[case::neq(Pred::ne(col!("modified"), lit("2021-02-01")), 2)] +#[case::gt(Pred::gt(col!("modified"), lit("2021-02-01")), 2)] +#[case::lt(Pred::lt(col!("modified"), lit("2021-02-02")), 2)] +#[case::gte_all(Pred::ge(col!("modified"), lit("2021-02-01")), 4)] +#[case::lte_all(Pred::le(col!("modified"), lit("2021-02-02")), 4)] +#[case::range_anded( + Pred::and( + Pred::ge(col!("modified"), lit("2021-02-01")), + Pred::le(col!("modified"), lit("2021-02-01")), + ), + 2 +)] +fn partition_only_skipping(#[case] pred: Pred, #[case] expected: usize) { + assert_eq!(count_selected(PARTITIONED_TABLE, Arc::new(pred)), expected); +} + +// -- Data-stats-only predicates (app-txn-checkpoint) -------------------------- + +#[rstest] +#[case::gt_prunes_low(Pred::gt(col!("value"), lit(9i32)), 2)] +#[case::lt_prunes_high(Pred::lt(col!("value"), lit(4i32)), 2)] +#[case::gt_above_max(Pred::gt(col!("value"), lit(11i32)), 0)] +#[case::le_at_max(Pred::le(col!("value"), lit(11i32)), 4)] +#[case::range_anded( + Pred::and( + Pred::ge(col!("value"), lit(1i32)), + Pred::le(col!("value"), lit(3i32)), + ), + 2 +)] +fn data_stats_only_skipping(#[case] pred: Pred, #[case] expected: usize) { + assert_eq!(count_selected(PARTITIONED_TABLE, Arc::new(pred)), expected); +} + +// -- Mixed AND: both partition and data conditions must hold ------------------- + +#[rstest] +#[case::partition_match_data_match( + "2021-02-01", + 3i32, + 2, + "partition prunes 02-02; data keeps 02-01 (max=11 > 3)" +)] +#[case::partition_match_data_miss( + "2021-02-02", + 3i32, + 0, + "partition keeps 02-02 but max=3 NOT >3; partition prunes 02-01" +)] +#[case::partition_miss("2099-01-01", 0i32, 0, "no files match partition")] +fn mixed_and_skipping( + #[case] partition_val: &str, + #[case] data_threshold: i32, + #[case] expected: usize, + #[case] _scenario: &str, +) { + let pred = Arc::new(Pred::and( + col!("modified").eq(lit(partition_val)), + col!("value").gt(lit(data_threshold)), + )); + assert_eq!(count_selected(PARTITIONED_TABLE, pred), expected); +} + +// -- Mixed OR: a file survives if either leg matches -------------------------- + +#[rstest] +#[case::both_match("2021-02-02", 9i32, 4, "02-02 matches partition; 02-01 has max=11 > 9")] +#[case::partition_saves_some( + "2021-02-02", + 11i32, + 2, + "02-02 matches partition; 02-01 max=11 NOT >11 -> pruned" +)] +#[case::data_saves_some( + "2099-01-01", -1i32, 4, + "no partition match; all files have max >= 0 so value > -1 keeps all" +)] +#[case::both_miss( + "2099-01-01", + 11i32, + 0, + "no partition match; max=11 NOT >11 -> all pruned" +)] +fn mixed_or_skipping( + #[case] partition_val: &str, + #[case] data_threshold: impl Into, + #[case] expected: usize, + #[case] _scenario: &str, +) { + let pred = Arc::new(Pred::or( + col!("modified").eq(lit(partition_val)), + col!("value").gt(lit(data_threshold.into())), + )); + assert_eq!(count_selected(PARTITIONED_TABLE, pred), expected); +} + +// -- Nested AND(partition, OR(data, data)) ------------------------------------ + +#[rstest] +#[case::loose_bound(10i32, 4, "max=11 > 10 keeps 02-01; min=1 < 2 keeps 02-02")] +#[case::strict_bound(11i32, 2, "max=11 NOT >11 prunes 02-01; min=1 < 2 keeps 02-02")] +fn nested_and_or_skipping( + #[case] upper_bound: i32, + #[case] expected: usize, + #[case] _scenario: &str, +) { + let pred = Arc::new(Pred::and( + Pred::ge(col!("modified"), lit("2021-02-01")), + Pred::or( + Pred::lt(col!("value"), lit(2i32)), + Pred::gt(col!("value"), lit(upper_bound)), + ), + )); + assert_eq!(count_selected(PARTITIONED_TABLE, pred), expected); +} + +// -- Parsed stats skipping (non-partitioned table) ---------------------------- + +#[test] +fn parsed_stats_skipping() { + // id > 400 should skip files 1-4 (max id: 100, 200, 300, 400) and keep files 5-6 + let pred = Arc::new(Pred::gt(col!("id"), lit(400i64))); + assert_eq!(count_selected(STATS_TABLE, pred), 2); +} + +// -- Timestamp predicate skipping (parsed-stats table) ------------------------ +// Timestamp predicates now use max stats with a 999us adjustment for truncation. +// Table has 6 files with ts_col ranges: [1M,2M], [3M,4M], [5M,6M], [7M,8M], [9M,10M], [11M,12M] + +#[rstest] +#[case::bare_ts_gt_keeps_all( + // ts_col > 2M -> adjusted: max > 1,999,001 -> all 6 files have max >= 2M -> 6 + Pred::gt(col!("ts_col"), lit(Scalar::Timestamp(2_000_000))), + 6 +)] +#[case::bare_ts_lt_skips( + // ts_col < 3M -> min < 3M -> file 1 (min=1M) -> 1 + Pred::lt(col!("ts_col"), lit(Scalar::Timestamp(3_000_000))), + 1 +)] +#[case::and_mixed_id_and_ts( + // id > 400 keeps files 5-6; ts_col > 2M keeps all 6; AND -> 2 + Pred::and( + Pred::gt(col!("id"), lit(400i64)), + Pred::gt(col!("ts_col"), lit(Scalar::Timestamp(2_000_000))), + ), + 2 +)] +#[case::or_mixed_id_and_ts( + // id > 400 keeps 5-6; ts_col > 2M keeps 1-6; OR -> 6 + Pred::or( + Pred::gt(col!("id"), lit(400i64)), + Pred::gt(col!("ts_col"), lit(Scalar::Timestamp(2_000_000))), + ), + 6 +)] +#[case::and_two_ts_predicates( + // ts_col > 2M (adjusted max > 1,999,001 -> all) AND ts_col > 5M (adjusted max > 4,999,001 + // -> files 3-6) -> 4 + Pred::and( + Pred::gt(col!("ts_col"), lit(Scalar::Timestamp(2_000_000))), + Pred::gt(col!("ts_col"), lit(Scalar::Timestamp(5_000_000))), + ), + 4 +)] +#[case::or_two_ts_predicates( + // ts_col > 2M keeps all; ts_col > 5M keeps files 3-6; OR -> 6 + Pred::or( + Pred::gt(col!("ts_col"), lit(Scalar::Timestamp(2_000_000))), + Pred::gt(col!("ts_col"), lit(Scalar::Timestamp(5_000_000))), + ), + 6 +)] +fn timestamp_predicate_skipping(#[case] pred: Pred, #[case] expected: usize) { + assert_eq!(count_selected(STATS_TABLE, Arc::new(pred)), expected); +} + +// -- Unsupported predicate handling (parsed-stats table) ---------------------- +// Column-column comparisons are unsupported for data skipping (no literal to infer type). +// Verifies that junctions degrade gracefully when one or both legs can't be evaluated. + +#[rstest] +#[case::bare_unsupported_returns_all( + // col > col is unsupported -> None -> keep all files + Pred::gt(col!("id"), col!("salary")), + 6 +)] +#[case::and_supported_with_unsupported( + // id > 400 keeps files 5-6; id > salary is unsupported; AND -> 2 + Pred::and( + Pred::gt(col!("id"), lit(400i64)), + Pred::gt(col!("id"), col!("salary")), + ), + 2 +)] +#[case::or_supported_with_unsupported( + // id > 400 keeps 5-6; id > salary is unsupported; OR -> all 6 + Pred::or( + Pred::gt(col!("id"), lit(400i64)), + Pred::gt(col!("id"), col!("salary")), + ), + 6 +)] +#[case::and_all_unsupported( + // Both legs unsupported -> None -> keep all 6 + Pred::and( + Pred::gt(col!("id"), col!("salary")), + Pred::gt(col!("id"), col!("age")), + ), + 6 +)] +#[case::or_all_unsupported( + // Both legs unsupported -> None -> keep all 6 + Pred::or( + Pred::gt(col!("id"), col!("salary")), + Pred::gt(col!("id"), col!("age")), + ), + 6 +)] +fn unsupported_predicate_skipping(#[case] pred: Pred, #[case] expected: usize) { + assert_eq!(count_selected(STATS_TABLE, Arc::new(pred)), expected); +} + +// === Stats-columns gate: non-stat refs collapse to NULL === +// +// Covers the predicate-rewrite half of the gate. The upstream column-selection half +// (numIndexedCols cap, statsColumns, struct prefixes, etc.) is in +// `stats_schema::column_filter::tests`. + +/// Builds a stats-column set from leaf paths spelled as dotted strings. +fn stats_cols(cols: &[&str]) -> HashSet { + cols.iter().map(|s| ColumnName::new(s.split('.'))).collect() +} + +/// Rewrite shape for the production `DataSkippingPredicateCreator` SQL-WHERE path. +/// +/// Each case is a `(SQL predicate, partition columns, stats_parsed contents, expected +/// rewrite)` row. `expected` is the exact display form of the rewrite. The SQL-WHERE +/// wrapping adds a per-column not-all-null guard (`NOT(nullCount = numRecords)`) and a +/// `true` filler around each stat comparison; non-stat refs fold to a `null` literal +/// inside a `AND(null, true)` 2-arg group, evaluating to NULL ("keep the file"). +/// +/// Companion semantic test: `mixed_and_non_stat_arm_still_prunes_via_stat_arm` (below) +/// shows the rewrite from `stat_and_non_stat_folds_non_stat` actually prunes a file +/// when the stat arm's min/max bounds rule it out. +#[rstest] +#[case::non_stat_only_folds_to_trivial_null( + Pred::gt(col!("non_stat"), lit(1)), + HashSet::::new(), + stats_cols(&["stat"]), + "AND(null, true)", +)] +#[case::stat_and_non_stat_folds_non_stat( + Pred::and( + Pred::gt(col!("stat"), lit(100)), + Pred::gt(col!("non_stat"), lit(50)), + ), + HashSet::::new(), + stats_cols(&["stat"]), + "AND(AND(NOT(Column(stats_parsed.nullCount.stat) = Column(stats_parsed.numRecords)), \ + true, Column(stats_parsed.maxValues.stat) > 100), AND(null, true))", +)] +#[case::nested_stat_leaf_kept( + Pred::and( + Pred::gt(col!("user.email"), lit(100)), + Pred::gt(col!("user.password"), lit(200)), + ), + HashSet::::new(), + stats_cols(&["user.email"]), + "AND(AND(NOT(Column(stats_parsed.nullCount.user.email) = Column(stats_parsed.numRecords)), \ + true, Column(stats_parsed.maxValues.user.email) > 100), AND(null, true))", +)] +#[case::partition_stat_non_stat_three_way( + Pred::and( + Pred::gt(col!("part"), lit("p")), + Pred::and( + Pred::gt(col!("stat"), lit(100)), + Pred::gt(col!("non_stat"), lit(50)), + ), + ), + HashSet::from([column_name!("part")]), + stats_cols(&["stat"]), + "AND(AND(OR(NOT(Column(is_add)), NOT(Column(partitionValues_parsed.part) IS NULL)), \ + true, OR(NOT(Column(is_add)), Column(partitionValues_parsed.part) > 'p')), \ + AND(AND(NOT(Column(stats_parsed.nullCount.stat) = Column(stats_parsed.numRecords)), \ + true, Column(stats_parsed.maxValues.stat) > 100), AND(null, true)))", +)] +#[case::both_in_stats_set_keeps_both_arms( + Pred::and( + Pred::gt(col!("stat"), lit(100)), + Pred::gt(col!("non_stat"), lit(50)), + ), + HashSet::::new(), + stats_cols(&["stat", "non_stat"]), + "AND(AND(NOT(Column(stats_parsed.nullCount.stat) = Column(stats_parsed.numRecords)), \ + true, Column(stats_parsed.maxValues.stat) > 100), \ + AND(NOT(Column(stats_parsed.nullCount.non_stat) = Column(stats_parsed.numRecords)), \ + true, Column(stats_parsed.maxValues.non_stat) > 50))", +)] +fn stats_columns_gate_rewrite( + #[case] pred: Pred, + #[case] partition_columns: HashSet, + #[case] stats: HashSet, + #[case] expected: &str, +) { + let result = + as_sql_data_skipping_predicate_with_stats_columns(&pred, &partition_columns, &stats) + .expect("SQL-WHERE rewrite always returns Some for these cases"); + assert_eq!(result.to_string(), expected); +} + +/// Semantic companion to `stat_and_non_stat_folds_non_stat`. When `max(stat) < 100` the +/// stat arm evaluates to false, the AND short-circuits, and the file is pruned despite +/// the non-stat NULL fold. +#[test] +fn mixed_and_non_stat_arm_still_prunes_via_stat_arm() { + let pred = Pred::and( + Pred::gt(col!("stat"), lit(100)), + Pred::gt(col!("non_stat"), lit(50)), + ); + let stats = stats_cols(&["stat"]); + let result = + as_sql_data_skipping_predicate_with_stats_columns(&pred, &Default::default(), &stats) + .unwrap(); + let resolver = HashMap::from_iter([( + column_name!("stats_parsed.maxValues.stat"), + Scalar::from(50i32), + )]); + let filter = DefaultKernelPredicateEvaluator::from(resolver); + expect_eq!( + filter.eval(&result), + FALSE, + "stat arm prunes despite non-stat NULL fold" + ); +} + +/// Same scenario as `stat_and_non_stat_folds_non_stat` but through the checkpoint +/// creator, which adds an `IS NULL` guard on each stat ref for safe parquet +/// row-group filtering. The non-stat arm still folds to NULL. +#[test] +fn checkpoint_pushdown_non_stat_arm_folds_to_null_literal() { + let pred = Pred::and( + Pred::gt(col!("stat"), lit(100)), + Pred::gt(col!("non_stat"), lit(50)), + ); + let stats = stats_cols(&["stat"]); + let result = + as_checkpoint_skipping_predicate(&pred, &HashSet::new(), &HashSet::new(), &stats).unwrap(); + assert_eq!( + result.to_string(), + "AND(OR(Column(stats_parsed.maxValues.stat) IS NULL, \ + Column(stats_parsed.maxValues.stat) > 100), null)" + ); +} diff --git a/kernel/src/scan/field_classifiers.rs b/kernel/src/scan/field_classifiers.rs index f0092e4a2e..5b391e879f 100644 --- a/kernel/src/scan/field_classifiers.rs +++ b/kernel/src/scan/field_classifiers.rs @@ -13,7 +13,8 @@ use crate::table_changes::{ pub(crate) trait TransformFieldClassifier { /// Classify a field and return its transform spec. /// Returns None if the field is physical (should be read from parquet). - /// Returns Some(spec) if the field needs transformation (partition, metadata-derived, or dynamic). + /// Returns Some(spec) if the field needs transformation (partition, metadata-derived, or + /// dynamic). fn classify_field( &self, field: &StructField, diff --git a/kernel/src/scan/log_replay.rs b/kernel/src/scan/log_replay.rs index a1194cc5e9..cc29d8389d 100644 --- a/kernel/src/scan/log_replay.rs +++ b/kernel/src/scan/log_replay.rs @@ -1,17 +1,21 @@ use std::clone::Clone; -use std::collections::{HashMap, HashSet}; +use std::collections::HashSet; use std::sync::{Arc, LazyLock}; +use delta_kernel_derive::internal_api; +use serde::{Deserialize, Serialize}; + use super::data_skipping::DataSkippingFilter; +use super::metrics::ScanMetrics; use super::state_info::StateInfo; -use super::{PhysicalPredicate, ScanMetadata}; +use super::{PhysicalPredicate, ScanMetadata, COMMIT_READ_SCHEMA}; use crate::actions::deletion_vector::DeletionVectorDescriptor; -use crate::engine_data::{GetData, RowVisitor, TypedGetData as _}; +use crate::engine_data::{EngineData, GetData, RowVisitor, TypedGetData as _}; use crate::expressions::{ - column_expr, column_expr_ref, column_name, ColumnName, Expression, ExpressionRef, PredicateRef, + col, column_expr_ref, column_name, ColumnName, Expression, ExpressionRef, MapToStructOptions, + Predicate, PredicateRef, UnaryExpressionOp, }; -use crate::kernel_predicates::{DefaultKernelPredicateEvaluator, KernelPredicateEvaluator as _}; -use crate::log_replay::deduplicator::{CheckpointDeduplicator, Deduplicator}; +use crate::log_replay::deduplicator::{CheckpointDeduplicator, Deduplicator, FileActionInfo}; use crate::log_replay::{ ActionsBatch, FileActionDeduplicator, FileActionKey, LogReplayProcessor, ParallelLogReplayProcessor, @@ -19,13 +23,47 @@ use crate::log_replay::{ use crate::log_segment::CheckpointReadInfo; use crate::scan::transform_spec::{get_transform_expr, parse_partition_values, TransformSpec}; use crate::scan::Scalar; -use crate::schema::ToSchema as _; -use crate::schema::{ColumnNamesAndTypes, DataType, MapType, SchemaRef, StructField, StructType}; +use crate::schema::{ + schema_ref, ColumnNamesAndTypes, DataType, MapType, SchemaRef, SchemaStructPatchBuilder, + StructField, StructType, ToSchema as _, +}; use crate::table_features::ColumnMappingMode; -use crate::utils::require; +use crate::timestamp_timezone::TimestampTimezone; +use crate::utils::{require, FoldWithOption as _}; use crate::{DeltaResult, Engine, Error, ExpressionEvaluator}; -use delta_kernel_derive::internal_api; -use serde::{Deserialize, Serialize}; + +/// Read-time stats toggles consumed by [`ScanLogReplayProcessor`]. +#[derive(Clone, Copy, Debug, serde::Serialize, serde::Deserialize)] +pub(crate) struct ScanStatsOptions { + /// Skip reading file statistics entirely. Disables data skipping. + pub(crate) skip_stats: bool, + /// Synthesize the `stats` JSON output via `ToJson(stats_parsed)` on checkpoints + /// whose `add.stats` is null but whose `add.stats_parsed` is populated + /// (writeStatsAsJson=false, writeStatsAsStruct=true). When false, `ScanFile.stats` + /// is left null on such checkpoints; engines that consume `stats_parsed` directly + /// avoid reading JSON stats in checkpoints and the per-batch `ToJson` cost. + pub(crate) synthesize_json: bool, +} + +impl Default for ScanStatsOptions { + fn default() -> Self { + Self { + skip_stats: false, + synthesize_json: true, + } + } +} + +/// Read-time partition value toggles consumed by [`ScanLogReplayProcessor`]. +#[derive(Clone, Debug, Default, serde::Serialize, serde::Deserialize)] +pub(crate) struct ScanPartitionValuesOptions { + /// Emit the typed `partitionValues_parsed` struct column in scan metadata output, + /// independent of any predicate. + pub(crate) parsed_struct: bool, + /// Reader timezone used to interpret offset-less zoned timestamp partition strings. + #[serde(default)] + pub(crate) timestamp_timezone: Option, +} /// Internal serializable state (schemas, transform spec, column mapping, etc.) /// NOTE: This is opaque to the user - it is passed through as a blob. @@ -40,13 +78,25 @@ struct InternalScanState { /// Physical stats schema for reading/parsing stats from checkpoint files physical_stats_schema: Option, #[serde(default)] - skip_stats: bool, + stats_options: ScanStatsOptions, + #[serde(default)] + partition_values_options: ScanPartitionValuesOptions, /// Physical partition schema for checkpoint partition pruning via `partitionValues_parsed` physical_partition_schema: Option, + /// Physical leaf paths which are expected to have stats collected. Carried alongside + /// `physical_stats_schema` so the distributed `DataSkippingFilter` rebuilds the same + /// filter the sequential phase used. `#[serde(default)]` keeps older blobs readable: + /// an empty set drops every data-column reference, which means no skipping but is + /// still correct. + #[serde(default)] + physical_stats_columns: HashSet, + #[serde(default)] + is_catalog_managed: bool, + skip_row_transforms: bool, } /// Serializable processor state for distributed processing. This can be serialized using the -/// defualt serde serialization, or through custom serialization in the engine. +/// default serde serialization, or through custom serialization in the engine. /// /// This struct contains all the information needed to reconstruct a `ScanLogReplayProcessor` /// on remote compute nodes, enabling distributed log replay processing. @@ -78,22 +128,27 @@ pub struct SerializableScanState { pub(crate) checkpoint_info: CheckpointReadInfo, } -/// [`ScanLogReplayProcessor`] performs log replay (processes actions) specifically for doing a table scan. +/// [`ScanLogReplayProcessor`] performs log replay (processes actions) specifically for doing a +/// table scan. /// /// During a table scan, the processor reads batches of log actions (in reverse chronological order) /// and performs the following steps: /// -/// - Data Skipping: Applies a predicate-based filter (via [`DataSkippingFilter`]) to quickly skip -/// files that are irrelevant for the query. -/// - Partition Pruning: Uses an optional partition filter (extracted from a physical predicate) -/// to exclude actions whose partition values do not meet the required criteria. -/// - Action Deduplication: Leverages the [`FileActionDeduplicator`] to ensure that for each unique file -/// (identified by its path and deletion vector unique ID), only the latest valid Add action is processed. -/// - Transformation: Applies a built-in transformation (`add_transform`) to convert selected Add actions -/// into [`ScanMetadata`], the intermediate format passed to the engine. -/// - Row Transform Passthrough: Any user-provided row-level transformation expressions (e.g. those derived -/// from projection or filters) are preserved and passed through to the engine, which applies them as part -/// of its scan execution logic. +/// - Transformation and Data Skipping: Applies a built-in transformation (`commit_transform` or +/// `checkpoint_transform`) to parse action metadata, then applies a predicate-based filter (via +/// [`DataSkippingFilter`]). This includes both data column stats (min/max/nullCount) and +/// partition value filtering in a single columnar pass. A secondary row-level partition filter +/// catches remaining files the columnar pass cannot prune (e.g. null partition values where +/// null-safety conservatively keeps them). +/// - Action Deduplication: Leverages the [`FileActionDeduplicator`] to ensure that for each unique +/// file (identified by its path and deletion vector unique ID), only the latest valid Add action +/// is processed. +/// - Parse-error fallback: If transformation and data skipping return [`Error::ParseError`], +/// deduplicates the raw batch first, then retries transformation and data skipping on the +/// surviving actions. +/// - Row StructPatch passthrough: Any user-provided row-level transformation expressions (e.g. +/// those derived from projection or filters) are preserved and passed through to the engine, +/// which applies them as part of its scan execution logic. /// /// As an implementation of [`LogReplayProcessor`], [`ScanLogReplayProcessor`] provides the /// `process_actions_batch` method, which applies these steps to each batch of log actions and @@ -102,24 +157,33 @@ pub struct SerializableScanState { /// to be applied to the selected rows. #[allow(rustdoc::broken_intra_doc_links, rustdoc::private_intra_doc_links)] pub struct ScanLogReplayProcessor { - partition_filter: Option, data_skipping_filter: Option, - /// Transform for log batches (commit files) - uses ParseJson for stats - log_transform: Arc, - /// Transform for checkpoint batches - uses coalesce(stats_parsed, ParseJson) when available + /// StructPatch for log batches (commit files) - uses ParseJson for stats and MapToStruct + /// for partition values + commit_transform: Arc, + /// StructPatch for checkpoint batches - reads pre-parsed stats_parsed and + /// partitionValues_parsed directly when available, otherwise parses from raw columns checkpoint_transform: Arc, state_info: Arc, /// A set of (data file path, dv_unique_id) pairs that have been seen thus /// far in the log. This is used to filter out files with Remove actions as /// well as duplicate entries in the log. seen_file_keys: HashSet, - /// Skip reading file statistics. - skip_stats: bool, - /// Whether checkpoint batches have compatible `partitionValues_parsed` columns. - #[allow(unused)] - has_partition_values_parsed: bool, + /// Read-time stats options. + stats_options: ScanStatsOptions, + /// Read-time partition value options. + partition_values_options: ScanPartitionValuesOptions, /// Information about checkpoint reading for stats optimization checkpoint_info: CheckpointReadInfo, + /// Metrics related to the scan + metrics: Arc, +} + +struct RetryTransformAndDataSkipOutput { + transformed_actions: Box, + final_selection: Vec, + row_transform_exprs: Vec>, + active_add_file_sizes: Vec, } impl ScanLogReplayProcessor { @@ -127,24 +191,28 @@ impl ScanLogReplayProcessor { // `selected_column_names_and_types()` const ADD_PATH_INDEX: usize = 0; // Position of "add.path" in getters const ADD_PARTITION_VALUES_INDEX: usize = 1; // Position of "add.partitionValues" in getters - const ADD_DV_START_INDEX: usize = 2; // Start position of add deletion vector columns - const BASE_ROW_ID_INDEX: usize = 5; // Position of add.baseRowId in getters - const REMOVE_PATH_INDEX: usize = 6; // Position of "remove.path" in getters - const REMOVE_DV_START_INDEX: usize = 7; // Start position of remove deletion vector columns + const ADD_SIZE_INDEX: usize = 2; // Position of "add.size" in getters + const ADD_DV_START_INDEX: usize = 3; // Start position of add deletion vector columns + const BASE_ROW_ID_INDEX: usize = 6; // Position of add.baseRowId in getters + const REMOVE_PATH_INDEX: usize = 7; // Position of "remove.path" in getters + const REMOVE_DV_START_INDEX: usize = 8; // Start position of remove deletion vector columns /// Create a new [`ScanLogReplayProcessor`] instance pub(crate) fn new( engine: &dyn Engine, state_info: Arc, checkpoint_info: CheckpointReadInfo, - skip_stats: bool, + stats_options: ScanStatsOptions, + partition_values_options: ScanPartitionValuesOptions, ) -> DeltaResult { + let dedup_capacity = state_info.dedup_capacity_hint(); Self::new_with_seen_files( engine, state_info, checkpoint_info, - Default::default(), - skip_stats, + HashSet::with_capacity(dedup_capacity), + stats_options, + partition_values_options, ) } @@ -158,19 +226,30 @@ impl ScanLogReplayProcessor { /// - `state_info`: StateInfo containing schemas, transforms, and predicates /// - `checkpoint_info`: Information about checkpoint reading for stats optimization /// - `seen_file_keys`: Pre-computed set of file action keys that have been seen - /// - `skip_stats`: Skip reading file statistics + /// - `stats_options`: Read-time stats options (see [`ScanStatsOptions`]) + /// - `partition_values_options`: Read-time partition value options (see + /// [`ScanPartitionValuesOptions`]) pub(crate) fn new_with_seen_files( engine: &dyn Engine, state_info: Arc, checkpoint_info: CheckpointReadInfo, seen_file_keys: HashSet, - skip_stats: bool, + stats_options: ScanStatsOptions, + partition_values_options: ScanPartitionValuesOptions, ) -> DeltaResult { let CheckpointReadInfo { has_stats_parsed, has_partition_values_parsed, checkpoint_read_schema, } = checkpoint_info.clone(); + let ScanStatsOptions { + skip_stats, + synthesize_json, + } = stats_options; + let timestamp_timezone = partition_values_options.timestamp_timezone.as_deref(); + + // Create metrics first so we can pass them to DataSkippingFilter + let metrics = Arc::new(ScanMetrics::default()); // Extract the physical predicate for data skipping and partition filtering. // DataSkippingFilter expects Option<(PredicateRef, SchemaRef)>. @@ -179,59 +258,92 @@ impl ScanLogReplayProcessor { _ => None, }; - // When skip_stats is enabled, use None for stats schema to produce null stats output + // When skip_stats is enabled, disable both data column skipping and partition pruning. + // Both rely on the same DataSkippingFilter columnar pass, so they are controlled together. let stats_schema_for_transform = if skip_stats { None } else { state_info.physical_stats_schema.clone() }; - let output_schema = scan_row_schema_with_stats_parsed(stats_schema_for_transform.clone()); + // The partition schema feeds two consumers: the DataSkippingFilter (predicate + // pruning, disabled by skip_stats together with stats) and the engine-facing + // `partitionValues_parsed` output column (requested via `parsed_struct`, independent + // of skip_stats). Either consumer keeps the transform emitting the column. + let partition_schema_for_transform = + if partition_values_options.parsed_struct || !skip_stats { + state_info.physical_partition_schema.clone() + } else { + None + }; + + let output_schema = scan_row_schema_with_parsed_columns( + stats_schema_for_transform.clone(), + partition_schema_for_transform.clone(), + )?; - // Create data skipping filter that reads stats_parsed from the transformed batch. - // This avoids double JSON parsing - the transform parses JSON once, then data skipping - // reads the already-parsed stats_parsed column from the transform output. - // Disabled when skip_stats is enabled. + // Create data skipping filter that reads stats_parsed and partitionValues_parsed + // from the transformed batch. This avoids double JSON parsing -- the transform parses + // JSON once, then data skipping reads the already-parsed columns from the output. + // + // When partition columns are referenced by the predicate, the filter receives a + // partition schema + expression to extract typed partition values. Since the transform + // already produces `partitionValues_parsed`, the filter reads that column directly. let data_skipping_filter = if skip_stats { None } else { - state_info - .physical_stats_schema - .as_ref() - .and_then(|physical_stats_schema| { - DataSkippingFilter::new( - engine, - // these are all cheap arc clones - physical_predicate.as_ref().map(|(p, _)| p.clone()), - Some(physical_stats_schema), - column_expr_ref!("stats_parsed"), - None, // no partition schema yet - column_expr_ref!("partitionValues_parsed"), - output_schema.clone(), - ) - }) + DataSkippingFilter::new( + engine, + physical_predicate.as_ref().map(|(p, _)| p.clone()), + stats_schema_for_transform.as_ref(), + column_expr_ref!("stats_parsed"), + partition_schema_for_transform.as_ref(), + column_expr_ref!("partitionValues_parsed"), + // The transform flattens `add.*` to top-level columns, so `path` is non-null + // exactly for Add rows. + Arc::new(Predicate::is_not_null(col!("path")).into()), + output_schema.clone(), + &state_info.physical_stats_columns, + Some(metrics.clone()), + ) }; Ok(Self { - partition_filter: physical_predicate.as_ref().map(|(p, _)| p.clone()), data_skipping_filter, - // Log transform: always parse JSON (no stats_parsed in JSON commit files) - log_transform: engine.evaluation_handler().new_expression_evaluator( - checkpoint_read_schema.clone(), - get_add_transform_expr(stats_schema_for_transform.clone(), false, skip_stats), + // Commit transform: parse JSON for stats, MapToStruct for partition values + commit_transform: engine.evaluation_handler().new_expression_evaluator( + COMMIT_READ_SCHEMA.clone(), + get_add_transform_expr( + stats_schema_for_transform.clone(), + false, + skip_stats, + synthesize_json, + partition_schema_for_transform.clone(), + false, + timestamp_timezone, + ), output_schema.clone().into(), )?, - // Checkpoint transform: read stats_parsed directly when available, otherwise parse JSON + // Checkpoint transform: read pre-parsed columns directly when available checkpoint_transform: engine.evaluation_handler().new_expression_evaluator( checkpoint_read_schema, - get_add_transform_expr(stats_schema_for_transform, has_stats_parsed, skip_stats), + get_add_transform_expr( + stats_schema_for_transform, + has_stats_parsed, + skip_stats, + synthesize_json, + partition_schema_for_transform, + has_partition_values_parsed, + timestamp_timezone, + ), output_schema.into(), )?, seen_file_keys, - has_partition_values_parsed, state_info, - skip_stats, + stats_options, + partition_values_options, checkpoint_info, + metrics, }) } @@ -240,6 +352,14 @@ impl ScanLogReplayProcessor { &self.checkpoint_info } + pub(crate) fn get_metrics(&self) -> &ScanMetrics { + self.metrics.as_ref() + } + + pub(crate) fn is_catalog_managed(&self) -> bool { + self.state_info.is_catalog_managed + } + /// Serialize the processor state for distributed processing. /// /// Consumes the processor and returns a `SerializableScanState` containing: @@ -264,6 +384,9 @@ impl ScanLogReplayProcessor { column_mapping_mode, physical_stats_schema, physical_partition_schema, + physical_stats_columns, + is_catalog_managed, + skip_row_transforms, } = self.state_info.as_ref().clone(); // Extract predicate from PhysicalPredicate @@ -280,8 +403,12 @@ impl ScanLogReplayProcessor { predicate_schema, column_mapping_mode, physical_stats_schema, - skip_stats: self.skip_stats, + stats_options: self.stats_options, + partition_values_options: self.partition_values_options, physical_partition_schema, + physical_stats_columns, + is_catalog_managed, + skip_row_transforms, }; let internal_state_blob = serde_json::to_vec(&internal_state) .map_err(|e| Error::generic(format!("Failed to serialize internal state: {e}")))?; @@ -297,16 +424,16 @@ impl ScanLogReplayProcessor { /// Reconstruct a processor from serialized state. /// /// Creates a new processor with the provided state. All fields (partition_filter, - /// data_skipping_filter, add_transform, and seen_file_keys) are reconstructed from - /// the serialized state and engine. + /// data_skipping_filter, commit_transform, checkpoint_transform, and seen_file_keys) are + /// reconstructed from the serialized state and engine. /// /// # Parameters /// - `engine`: Engine for creating evaluators and filters - /// - `state`: The serialized state containing predicate, internal state blob, and seen file keys + /// - `state`: The serialized state containing predicate, internal state blob, and seen file + /// keys /// /// # Returns /// A new `ScanLogReplayProcessor` wrapped in an Arc. - /// #[internal_api] #[allow(unused)] pub(crate) fn from_serializable_state( @@ -338,6 +465,9 @@ impl ScanLogReplayProcessor { column_mapping_mode: internal_state.column_mapping_mode, physical_stats_schema: internal_state.physical_stats_schema, physical_partition_schema: internal_state.physical_partition_schema, + physical_stats_columns: internal_state.physical_stats_columns, + is_catalog_managed: internal_state.is_catalog_managed, + skip_row_transforms: internal_state.skip_row_transforms, }); Self::new_with_seen_files( @@ -345,128 +475,222 @@ impl ScanLogReplayProcessor { state_info, state.checkpoint_info, state.seen_file_keys, - internal_state.skip_stats, + internal_state.stats_options, + internal_state.partition_values_options, ) } + + fn transform_and_data_skip( + &self, + actions: &dyn EngineData, + is_log_batch: bool, + ) -> DeltaResult<(Box, Vec)> { + let transform = if is_log_batch { + &self.commit_transform + } else { + &self.checkpoint_transform + }; + let transformed = transform.evaluate(actions)?; + require!( + transformed.len() == actions.len(), + Error::internal_error(format!( + "transform output length {} != actions length {}", + transformed.len(), + actions.len() + )) + ); + + let selection_vector = self.build_selection_vector(transformed.as_ref())?; + require!( + selection_vector.len() == actions.len(), + Error::internal_error(format!( + "selection vector length {} != actions length {}", + selection_vector.len(), + actions.len() + )) + ); + Ok((transformed, selection_vector)) + } + + fn retry_transform_and_data_skip( + &self, + actions: Box, + is_log_batch: bool, + dedup_selection: Vec, + row_transform_exprs: Vec>, + active_add_file_sizes: Vec, + ) -> DeltaResult { + let row_transform_exprs = dedup_selection + .iter() + .enumerate() + .filter(|(_, selected)| **selected) + .map(|(row, _)| row_transform_exprs.get(row).cloned().flatten()) + .collect(); + let active_add_file_sizes = dedup_selection + .iter() + .zip(active_add_file_sizes) + .filter_map(|(selected, size)| (*selected).then_some(size)) + .collect(); + let actions = actions.apply_selection_vector(dedup_selection)?; + let (transformed_actions, final_selection) = + self.transform_and_data_skip(actions.as_ref(), is_log_batch)?; + Ok(RetryTransformAndDataSkipOutput { + transformed_actions, + final_selection, + row_transform_exprs, + active_add_file_sizes, + }) + } + + fn record_active_add_files( + &self, + selection_vector: &[bool], + active_add_file_sizes: &[u64], + ) -> DeltaResult<()> { + require!( + selection_vector.len() == active_add_file_sizes.len(), + Error::internal_error(format!( + "selection vector length {} != active Add file sizes length {}", + selection_vector.len(), + active_add_file_sizes.len() + )) + ); + for (selected, size) in selection_vector.iter().zip(active_add_file_sizes) { + if *selected { + self.metrics.record_active_add_file(*size); + } + } + Ok(()) + } } /// A visitor that deduplicates a stream of add and remove actions into a stream of valid adds. Log /// replay visits actions newest-first, so once we've seen a file action for a given (path, dvId) /// pair, we should ignore all subsequent (older) actions for that same (path, dvId) pair. If the /// first action for a given file is a remove, then that file does not show up in the result at all. -struct AddRemoveDedupVisitor { +struct AddRemoveDedupVisitor<'a, D: Deduplicator> { deduplicator: D, selection_vector: Vec, state_info: Arc, - partition_filter: Option, + timestamp_timezone: TimestampTimezone, row_transform_exprs: Vec>, + active_add_file_sizes: Vec, + metrics: &'a ScanMetrics, } -impl AddRemoveDedupVisitor { +impl<'a, D: Deduplicator> AddRemoveDedupVisitor<'a, D> { fn new( deduplicator: D, selection_vector: Vec, state_info: Arc, - partition_filter: Option, - ) -> AddRemoveDedupVisitor { - AddRemoveDedupVisitor { + metrics: &'a ScanMetrics, + timestamp_timezone: Option<&str>, + ) -> DeltaResult> { + let active_add_file_sizes = vec![0; selection_vector.len()]; + let timestamp_timezone = timestamp_timezone.map_or_else( + || Ok(TimestampTimezone::default()), + TimestampTimezone::parse, + )?; + Ok(AddRemoveDedupVisitor { deduplicator, selection_vector, state_info, - partition_filter, + timestamp_timezone, row_transform_exprs: Vec::new(), - } - } - - fn is_file_partition_pruned( - &self, - partition_values: &HashMap, - ) -> bool { - if partition_values.is_empty() { - return false; - } - let Some(partition_filter) = &self.partition_filter else { - return false; - }; - let partition_values: HashMap<_, _> = partition_values - .values() - .map(|(k, v)| (ColumnName::new([k]), v.clone())) - .collect(); - let evaluator = DefaultKernelPredicateEvaluator::from(partition_values); - evaluator.eval_sql_where(partition_filter) == Some(false) + active_add_file_sizes, + metrics, + }) } /// True if this row contains an Add action that should survive log replay. Skip it if the row /// is not an Add action, or the file has already been seen previously. - fn is_valid_add<'a>(&mut self, i: usize, getters: &[&'a dyn GetData<'a>]) -> DeltaResult { - // When processing file actions, we extract path and deletion vector information based on action type: - // - For Add actions: path is at index 0, followed by DV fields at indexes 2-4 - // - For Remove actions (in log batches only): path is at index 5, followed by DV fields at indexes 6-8 - // The file extraction logic selects the appropriate indexes based on whether we found a valid path. - // Remove getters are not included when visiting a non-log batch (checkpoint batch), so do - // not try to extract remove actions in that case. - let Some((file_key, is_add)) = self.deduplicator.extract_file_action( - i, + fn is_valid_add<'b>( + &mut self, + row: usize, + getters: &[&'b dyn GetData<'b>], + ) -> DeltaResult { + // When processing file actions, we extract path and deletion vector information based on + // action type: + // - For Add actions: path is at index 0, size at 2, then followed by DV fields at indexes + // 3-5 + // - For Remove actions (in log batches only): path is at index 7, followed by DV fields at + // indexes 8-10 + // The file extraction logic selects the appropriate indexes based on whether we found a + // valid path. Remove getters are not included when visiting a non-log batch + // (checkpoint batch), so do not try to extract remove actions in that case. + let Some(FileActionInfo { + key: file_key, + size, + is_add, + }) = self.deduplicator.extract_file_action( + row, getters, !self.deduplicator.is_log_batch(), // skip_removes. true if this is a checkpoint batch )? else { + self.metrics.incr_non_file_actions(); return Ok(false); }; - // Apply partition pruning (to adds only) before deduplication, so that we don't waste memory - // tracking pruned files. Removes don't get pruned and we'll still have to track them. - // - // WARNING: It's not safe to partition-prune removes (just like it's not safe to data skip - // removes), because they are needed to suppress earlier incompatible adds we might - // encounter if the table's schema was replaced after the most recent checkpoint. + if is_add { + self.metrics.incr_add_files_seen() + } else { + self.metrics.incr_remove_files_seen() + }; + + // Check both adds and removes (skipping already-seen), but only transform and return adds + if self.deduplicator.check_and_record_seen(file_key) || !is_add { + return Ok(false); + } + + // Parse partition values for building the per-row transform expression. + // Partition pruning is handled by DataSkippingFilter in the columnar data skipping phase, + // so we only need to parse values here for the transform. + + // Only needed for survived addFiles. let partition_values = match &self.state_info.transform_spec { - Some(transform) if is_add => { + Some(transform) if !self.state_info.skip_row_transforms => { let partition_values = getters[ScanLogReplayProcessor::ADD_PARTITION_VALUES_INDEX] - .get(i, "add.partitionValues")?; - let partition_values = parse_partition_values( + .get(row, "add.partitionValues")?; + parse_partition_values( &self.state_info.logical_schema, transform, &partition_values, self.state_info.column_mapping_mode, - )?; - if self.is_file_partition_pruned(&partition_values) { - return Ok(false); - } - partition_values + self.timestamp_timezone, + )? } _ => Default::default(), }; - // Check both adds and removes (skipping already-seen), but only transform and return adds - if self.deduplicator.check_and_record_seen(file_key) || !is_add { - return Ok(false); - } - let base_row_id: Option = - getters[ScanLogReplayProcessor::BASE_ROW_ID_INDEX].get_opt(i, "add.baseRowId")?; - let transform = self - .state_info - .transform_spec - .as_ref() - .map(|transform| { - get_transform_expr( - transform, - partition_values, - &self.state_info.physical_schema, - base_row_id, - ) - }) - .transpose()?; - if transform.is_some() { - // fill in any needed `None`s for previous rows - self.row_transform_exprs.resize_with(i, Default::default); - self.row_transform_exprs.push(transform); + if !self.state_info.skip_row_transforms { + let base_row_id: Option = + getters[ScanLogReplayProcessor::BASE_ROW_ID_INDEX].get_opt(row, "add.baseRowId")?; + let patch_expr = self + .state_info + .transform_spec + .as_ref() + .map(|transform_spec| { + get_transform_expr( + transform_spec, + partition_values, + &self.state_info.physical_schema, + base_row_id, + ) + }) + .transpose()?; + if patch_expr.is_some() { + // fill in any needed `None`s for previous rows + self.row_transform_exprs.resize_with(row, Default::default); + self.row_transform_exprs.push(patch_expr); + } } + self.active_add_file_sizes[row] = size; Ok(true) } } -impl RowVisitor for AddRemoveDedupVisitor { +impl RowVisitor for AddRemoveDedupVisitor<'_, D> { fn selected_column_names_and_types(&self) -> (&'static [ColumnName], &'static [DataType]) { // NOTE: The visitor assumes a schema with adds first and removes optionally afterward. static NAMES_AND_TYPES: LazyLock = LazyLock::new(|| { @@ -477,6 +701,7 @@ impl RowVisitor for AddRemoveDedupVisitor { let types_and_names = vec![ (STRING, column_name!("add.path")), (ss_map, column_name!("add.partitionValues")), + (LONG, column_name!("add.size")), (STRING, column_name!("add.deletionVector.storageType")), (STRING, column_name!("add.deletionVector.pathOrInlineDv")), (INTEGER, column_name!("add.deletionVector.offset")), @@ -495,13 +720,16 @@ impl RowVisitor for AddRemoveDedupVisitor { } else { // All checkpoint actions are already reconciled and Remove actions in checkpoint files // only serve as tombstones for vacuum jobs. So we only need to examine the adds here. - (&names[..6], &types[..6]) + let add_count = ScanLogReplayProcessor::REMOVE_PATH_INDEX; + (&names[..add_count], &types[..add_count]) } } fn visit<'a>(&mut self, row_count: usize, getters: &[&'a dyn GetData<'a>]) -> DeltaResult<()> { + let start = std::time::Instant::now(); + let is_log_batch = self.deduplicator.is_log_batch(); - let expected_getters = if is_log_batch { 10 } else { 6 }; + let expected_getters = if is_log_batch { 11 } else { 7 }; require!( getters.len() == expected_getters, Error::InternalError(format!( @@ -510,86 +738,126 @@ impl RowVisitor for AddRemoveDedupVisitor { )) ); - for i in 0..row_count { - if self.selection_vector[i] { - self.selection_vector[i] = self.is_valid_add(i, getters)?; + for row in 0..row_count { + if self.selection_vector[row] { + self.selection_vector[row] = self.is_valid_add(row, getters)?; } } + + self.metrics + .add_dedup_visitor_time_ns(start.elapsed().as_nanos() as u64); + Ok(()) } } pub(crate) static FILE_CONSTANT_VALUES_NAME: &str = "fileConstantValues"; +pub(crate) static PATH_NAME: &str = "path"; pub(crate) static BASE_ROW_ID_NAME: &str = "baseRowId"; pub(crate) static DEFAULT_ROW_COMMIT_VERSION_NAME: &str = "defaultRowCommitVersion"; pub(crate) static CLUSTERING_PROVIDER_NAME: &str = "clusteringProvider"; +pub(crate) static PARTITION_VALUES_NAME: &str = "partitionValues"; +pub(crate) static SIZE_NAME: &str = "size"; pub(crate) static TAGS_NAME: &str = "tags"; +pub(crate) static STATS_PARSED_NAME: &str = "stats_parsed"; +#[internal_api] +pub(crate) static PARTITION_VALUES_PARSED_NAME: &str = "partitionValues_parsed"; // NB: If you update this schema, ensure you update the comment describing it in the doc comment // for `scan_row_schema` in scan/mod.rs! You'll also need to update ScanFileVisitor as the // indexes will be off, and [`get_add_transform_expr`] below to match it. pub(crate) static SCAN_ROW_SCHEMA: LazyLock> = LazyLock::new(|| { // Note that fields projected out of a nullable struct must be nullable - let partition_values = MapType::new(DataType::STRING, DataType::STRING, true); - let file_constant_values = StructType::new_unchecked([ - StructField::nullable("partitionValues", partition_values), - StructField::nullable(BASE_ROW_ID_NAME, DataType::LONG), - StructField::nullable(DEFAULT_ROW_COMMIT_VERSION_NAME, DataType::LONG), - StructField::nullable( - "tags", - MapType::new( - DataType::STRING, - DataType::STRING, - /*valueContainsNull*/ true, - ), - ), - StructField::nullable(CLUSTERING_PROVIDER_NAME, DataType::STRING), - ]); - Arc::new(StructType::new_unchecked([ - StructField::nullable("path", DataType::STRING), - StructField::nullable("size", DataType::LONG), - StructField::nullable("modificationTime", DataType::LONG), - StructField::nullable("stats", DataType::STRING), - StructField::nullable("deletionVector", DeletionVectorDescriptor::to_schema()), - StructField::nullable(FILE_CONSTANT_VALUES_NAME, file_constant_values), - ])) + schema_ref! { + nullable PATH_NAME: STRING, + nullable SIZE_NAME: LONG, + nullable "modificationTime": LONG, + nullable "stats": STRING, + nullable "deletionVector": (DeletionVectorDescriptor::to_schema()), + nullable FILE_CONSTANT_VALUES_NAME: { + nullable PARTITION_VALUES_NAME: { STRING => nullable STRING }, + nullable BASE_ROW_ID_NAME: LONG, + nullable DEFAULT_ROW_COMMIT_VERSION_NAME: LONG, + nullable "tags": { STRING => nullable STRING }, + nullable CLUSTERING_PROVIDER_NAME: STRING, + }, + } }); -/// Build the scan row schema with optional stats_parsed column. +/// Build the scan-row schema, appending the opt-in typed `stats_parsed` / `partitionValues_parsed` +/// columns when requested. /// -/// When `stats_schema` is provided, adds a `stats_parsed` struct column with that schema. -fn scan_row_schema_with_stats_parsed(stats_schema: Option) -> SchemaRef { - match stats_schema { - Some(schema) => { - let mut fields: Vec = SCAN_ROW_SCHEMA.fields().cloned().collect(); - fields.push(StructField::nullable( - "stats_parsed", - schema.as_ref().clone(), - )); - Arc::new(StructType::new_unchecked(fields)) - } - None => SCAN_ROW_SCHEMA.clone(), +/// These typed columns are appended at the top level (siblings of `fileConstantValues`) rather than +/// nested inside it. This mirrors `stats_parsed`, keeps `fileConstantValues` a fixed shape +/// regardless of the engine's options, and keeps data-skipping paths uniform: +/// `partitionValues_parsed.` parallels `stats_parsed.minValues.`. The checkpoint source +/// is also `add.partitionValues_parsed`, a sibling of `add.stats_parsed`. +fn scan_row_schema_with_parsed_columns( + stats_schema: Option, + partition_schema: Option, +) -> DeltaResult { + let needs_extra = stats_schema.is_some() || partition_schema.is_some(); + if !needs_extra { + return Ok(SCAN_ROW_SCHEMA.clone()); } + let patch = SchemaStructPatchBuilder::new() + .fold_with(stats_schema.as_ref(), |patch, schema| { + patch.append(StructField::nullable(STATS_PARSED_NAME, schema.clone())) + }) + .fold_with(partition_schema.as_ref(), |patch, schema| { + patch.append(StructField::nullable( + PARTITION_VALUES_PARSED_NAME, + schema.clone(), + )) + }); + Ok(Arc::new(patch.build(&SCAN_ROW_SCHEMA)?)) } -/// Build the add transform expression with optional stats parsing. +/// Build the add transform expression with optional stats and partition value parsing. /// /// # Parameters /// - `physical_stats_schema`: Schema for parsing stats from JSON and for output (physical column /// names), or None if stats should not be included in output. -/// - `has_stats_parsed`: Whether checkpoint has pre-parsed stats_parsed column. +/// - `has_stats_parsed`: Whether checkpoint has pre-parsed stats_parsed column. When true and +/// `synthesize_json` is true, stats output uses `COALESCE(add.stats, ToJson(add.stats_parsed))` +/// so that `ScanFile.stats` is populated even when the checkpoint lacks JSON stats +/// (writeStatsAsJson=false). /// - `skip_stats`: When true, replaces the stats column with a null literal, avoiding reads of the -/// raw stats JSON string from checkpoint parquet files. +/// JSON stats column in checkpoint parquet files. +/// - `synthesize_json`: When false, disables the `ToJson(add.stats_parsed)` fallback regardless of +/// `has_stats_parsed`. Compatible parsed-stats checkpoints produce null JSON stats and can omit +/// the JSON stats column; JSON-only checkpoints and commits retain `add.stats` as fallback input. +/// - `partition_schema`: Schema of typed partition columns for data skipping, or None if partition +/// value parsing is not needed. +/// - `has_partition_values_parsed`: Whether the source carries a native `partitionValues_parsed` +/// column (checkpoint). When true it is read directly; otherwise the struct is reconstructed from +/// the `partitionValues` string map. +/// - `timestamp_timezone`: Reader timezone for offset-less zoned timestamp partition strings. /// -/// The transform includes `stats_parsed` only when `physical_stats_schema` is Some. +/// The transform includes `stats_parsed` only when `physical_stats_schema` is Some, +/// and `partitionValues_parsed` only when `partition_schema` is Some. /// Stats are output using physical column names. fn get_add_transform_expr( physical_stats_schema: Option, has_stats_parsed: bool, skip_stats: bool, + synthesize_json: bool, + partition_schema: Option, + has_partition_values_parsed: bool, + timestamp_timezone: Option<&str>, ) -> ExpressionRef { let stats_expr = if skip_stats { Arc::new(Expression::Literal(Scalar::Null(DataType::STRING))) + } else if has_stats_parsed && synthesize_json { + // Checkpoint may lack JSON stats when writeStatsAsJson=false. Fall back to + // serializing stats_parsed so ScanFile.stats is populated either way. + Arc::new(Expression::coalesce([ + col!("add.stats"), + Expression::unary(UnaryExpressionOp::ToJson, col!("add.stats_parsed")), + ])) + } else if has_stats_parsed { + // The compatible checkpoint projection can omit add.stats when JSON output is disabled. + Arc::new(Expression::Literal(Scalar::Null(DataType::STRING))) } else { column_expr_ref!("add.stats") }; @@ -612,22 +880,73 @@ fn get_add_transform_expr( if let Some(stats_schema) = physical_stats_schema { let stats_parsed_expr = if has_stats_parsed { // Checkpoint has stats_parsed column - read directly - column_expr!("add.stats_parsed") + col!("add.stats_parsed") } else { // No stats_parsed available (JSON log files) - parse JSON - Expression::parse_json(column_expr!("add.stats"), stats_schema) + Expression::parse_json(col!("add.stats"), stats_schema) }; fields.push(Arc::new(stats_parsed_expr)); } + // Add partitionValues_parsed when partition columns are needed for data skipping or for the + // engine-facing typed output column. + if let Some(partition_schema) = partition_schema { + let base_partition_values = + has_partition_values_parsed.then(|| column_name!("add.partitionValues_parsed")); + let pv_parsed_expr = parsed_partition_values_expr( + col!("add.partitionValues"), + base_partition_values, + &partition_schema, + timestamp_timezone, + ); + fields.push(Arc::new(pv_parsed_expr)); + } + Arc::new(Expression::struct_from(fields)) } +/// Build typed partition values, reparsing the raw map when reader-timezone interpretation is +/// required. +pub(super) fn parsed_partition_values_expr( + raw_partition_values: Expression, + base_partition_values: Option, + partition_schema: &StructType, + timestamp_timezone: Option<&str>, +) -> Expression { + let full_parse = || match timestamp_timezone { + Some(timestamp_timezone) => Expression::map_to_struct( + raw_partition_values.clone(), + MapToStructOptions::default().with_timestamp_timezone(timestamp_timezone), + ), + None => { + Expression::map_to_struct(raw_partition_values.clone(), MapToStructOptions::default()) + } + }; + + match base_partition_values { + Some(base_partition_values) + if !partition_schema.fields().any(|field| { + partition_value_requires_timezone_reparse(field.data_type(), timestamp_timezone) + }) => + { + Expression::from(base_partition_values) + } + _ => full_parse(), + } +} + +/// Whether a native checkpoint partition value must be replaced with a reader-timezone parse. +pub(super) fn partition_value_requires_timezone_reparse( + data_type: &DataType, + timestamp_timezone: Option<&str>, +) -> bool { + timestamp_timezone.is_some() && data_type == &DataType::TIMESTAMP +} + // TODO: Move this to transaction/mod.rs once `scan_metadata_from` is pub, as this is used for // deletion vector update transformations. #[allow(unused)] pub(crate) fn get_scan_metadata_transform_expr() -> ExpressionRef { - use crate::expressions::column_expr_ref; static EXPR: LazyLock = LazyLock::new(|| { Arc::new(Expression::struct_from([Arc::new( Expression::struct_from([ @@ -666,53 +985,77 @@ impl ParallelLogReplayProcessor for ScanLogReplayProcessor { Error::generic("Parallel checkpoint processor may only be applied to checkpoint files") ); - // Step 1: Apply transform FIRST (parses JSON once, outputs stats_parsed). - // This is done before data skipping so we can read the already-parsed stats. - // We use the checkpoint_transform because we checked above that we're reading a checkpoint. - let transformed = self.checkpoint_transform.evaluate(actions.as_ref())?; - debug_assert_eq!(transformed.len(), actions.len()); - require!( - transformed.len() == actions.len(), - Error::internal_error(format!( - "checkpoint transform output length {} != actions length {}", - transformed.len(), - actions.len() - )) - ); - - // Step 2: Build selection vector from TRANSFORMED batch (reads stats_parsed directly) - // This avoids double JSON parsing - the transform already parsed the stats. - let selection_vector = self.build_selection_vector(transformed.as_ref())?; - debug_assert_eq!(selection_vector.len(), actions.len()); - require!( - selection_vector.len() == actions.len(), - Error::internal_error(format!( - "selection vector length {} != actions length {}", - selection_vector.len(), - actions.len() - )) - ); + let mut should_retry_transform_and_data_skip = false; + // Step 1: Apply transform + data skipping. Do this before deduplication to reduce the size + // of the dedup map and avoid String allocations. + // This step transforms all Adds, including those superseded by Removes (dead Adds). + // The transform for a dead Add may fail because it may have a different partition column + // type. In that case, we get a ParseError and retry after deduplication. + let (pre_dedup_transform_result, pre_dedup_selection) = + match self.transform_and_data_skip(actions.as_ref(), is_log_batch) { + Ok((transformed_actions, pre_dedup_selection)) => { + (Ok(transformed_actions), pre_dedup_selection) + } + Err(err @ Error::ParseError(_, _)) => { + should_retry_transform_and_data_skip = true; + (Err(err), vec![true; actions.len()]) + } + Err(err) => return Err(err), + }; - // Step 3: Run deduplication visitor on RAW batch (needs add.path, remove.path, etc.) + // Step 2: Run deduplication visitor on RAW batch (needs add.path, remove.path, etc.) let deduplicator = CheckpointDeduplicator::try_new( &self.seen_file_keys, Self::ADD_PATH_INDEX, + Self::ADD_SIZE_INDEX, Self::ADD_DV_START_INDEX, )?; - let mut visitor = AddRemoveDedupVisitor::new( - deduplicator, - selection_vector, - self.state_info.clone(), - self.partition_filter.clone(), - ); - visitor.visit_rows_of(actions.as_ref())?; + let (dedup_selection, row_transform_exprs, active_add_file_sizes) = { + let mut visitor = AddRemoveDedupVisitor::new( + deduplicator, + pre_dedup_selection, + self.state_info.clone(), + &self.metrics, + self.partition_values_options.timestamp_timezone.as_deref(), + )?; + visitor.visit_rows_of(actions.as_ref())?; + ( + visitor.selection_vector, + visitor.row_transform_exprs, + visitor.active_add_file_sizes, + ) + }; - // Step 4: Return transformed batch with updated selection vector - ScanMetadata::try_new( - transformed, - visitor.selection_vector, - visitor.row_transform_exprs, - ) + // Step 3: Return transformed batch with updated selection vector + let RetryTransformAndDataSkipOutput { + transformed_actions, + final_selection, + row_transform_exprs, + active_add_file_sizes, + } = if should_retry_transform_and_data_skip { + // If step 1 failed with a parse error, filter out the dead Adds and retry after + // deduplication. + self.retry_transform_and_data_skip( + actions, + is_log_batch, + dedup_selection, + row_transform_exprs, + active_add_file_sizes, + )? + } else { + RetryTransformAndDataSkipOutput { + transformed_actions: pre_dedup_transform_result?, + final_selection: dedup_selection, + row_transform_exprs, + active_add_file_sizes, + } + }; + self.record_active_add_files(&final_selection, &active_add_file_sizes)?; + let scan_metadata = + ScanMetadata::try_new(transformed_actions, final_selection, row_transform_exprs)?; + self.metrics + .update_peak_hash_set_size(self.seen_file_keys.len()); + Ok(scan_metadata) } } @@ -730,62 +1073,89 @@ impl LogReplayProcessor for ScanLogReplayProcessor { is_log_batch, } = actions_batch; - // Step 1: Apply transform FIRST (parses JSON once, outputs stats_parsed) - // Use the correct transform based on batch type: - // - Log batches: use ParseJson (no stats_parsed in JSON commit files) - // - Checkpoint batches: use coalesce(stats_parsed, ParseJson) when available - let transform = if is_log_batch { - &self.log_transform - } else { - &self.checkpoint_transform - }; - let transformed = transform.evaluate(actions.as_ref())?; - debug_assert_eq!(transformed.len(), actions.len()); - require!( - transformed.len() == actions.len(), - Error::internal_error(format!( - "transform output length {} != actions length {}", - transformed.len(), - actions.len() - )) - ); - - // Step 2: Build selection vector from TRANSFORMED batch (reads stats_parsed directly) - // This avoids double JSON parsing - the transform already parsed the stats. - let selection_vector = self.build_selection_vector(transformed.as_ref())?; - debug_assert_eq!(selection_vector.len(), actions.len()); - require!( - selection_vector.len() == actions.len(), - Error::internal_error(format!( - "selection vector length {} != actions length {}", - selection_vector.len(), - actions.len() - )) - ); + let mut should_retry_transform_and_data_skip = false; + // Step 1: Apply transform + data skipping. Do this before deduplication to reduce the size + // of the dedup map and avoid String allocations. + // This step transforms all Adds, including those superseded by Removes (dead Adds). + // The transform for a dead Add may fail because it may have a different partition column + // type. In that case, we get a ParseError and retry after deduplication. + // The transform depends on the batch type: + // - Log batches: parse JSON for stats, MapToStruct for partition values + // - Checkpoint batches: read pre-parsed columns directly when available + // This avoids double JSON parsing -- the transform already parsed the stats. + // Data skipping is safe for Remove rows: their add-side columns (stats_parsed, + // partitionValues_parsed) are null. For stats, the skipping predicate wraps comparisons + // with ISNULL guards that keep rows with missing stats. For partition values, the + // predicate is wrapped with OR(NOT is_add, pred) via guard_for_removes, so non-Add + // rows always pass the partition filter regardless of null partition values. + let (pre_dedup_transform_result, pre_dedup_selection) = + match self.transform_and_data_skip(actions.as_ref(), is_log_batch) { + Ok((transformed_actions, pre_dedup_selection)) => { + (Ok(transformed_actions), pre_dedup_selection) + } + Err(err @ Error::ParseError(_, _)) => { + should_retry_transform_and_data_skip = true; + (Err(err), vec![true; actions.len()]) + } + Err(err) => return Err(err), + }; - // Step 3: Run deduplication visitor on RAW batch (needs add.path, remove.path, etc.) + // Step 2: Run deduplication visitor on RAW batch (needs add.path, remove.path, etc.) let deduplicator = FileActionDeduplicator::new( &mut self.seen_file_keys, is_log_batch, Self::ADD_PATH_INDEX, + Self::ADD_SIZE_INDEX, Self::REMOVE_PATH_INDEX, Self::ADD_DV_START_INDEX, Self::REMOVE_DV_START_INDEX, ); - let mut visitor = AddRemoveDedupVisitor::new( - deduplicator, - selection_vector, - self.state_info.clone(), - self.partition_filter.clone(), - ); - visitor.visit_rows_of(actions.as_ref())?; + let (dedup_selection, row_transform_exprs, active_add_file_sizes) = { + let mut visitor = AddRemoveDedupVisitor::new( + deduplicator, + pre_dedup_selection, + self.state_info.clone(), + &self.metrics, + self.partition_values_options.timestamp_timezone.as_deref(), + )?; + visitor.visit_rows_of(actions.as_ref())?; + ( + visitor.selection_vector, + visitor.row_transform_exprs, + visitor.active_add_file_sizes, + ) + }; - // Step 4: Return transformed batch with updated selection vector - ScanMetadata::try_new( - transformed, - visitor.selection_vector, - visitor.row_transform_exprs, - ) + // Step 3: Return transformed batch with updated selection vector + let RetryTransformAndDataSkipOutput { + transformed_actions, + final_selection, + row_transform_exprs, + active_add_file_sizes, + } = if should_retry_transform_and_data_skip { + // If step 1 failed with a parse error, filter out the dead Adds and retry after + // deduplication. + self.retry_transform_and_data_skip( + actions, + is_log_batch, + dedup_selection, + row_transform_exprs, + active_add_file_sizes, + )? + } else { + RetryTransformAndDataSkipOutput { + transformed_actions: pre_dedup_transform_result?, + final_selection: dedup_selection, + row_transform_exprs, + active_add_file_sizes, + } + }; + self.record_active_add_files(&final_selection, &active_add_file_sizes)?; + let scan_metadata = + ScanMetadata::try_new(transformed_actions, final_selection, row_transform_exprs)?; + self.metrics + .update_peak_hash_set_size(self.seen_file_keys.len()); + Ok(scan_metadata) } fn data_skipping_filter(&self) -> Option<&DataSkippingFilter> { @@ -794,13 +1164,17 @@ impl LogReplayProcessor for ScanLogReplayProcessor { } /// Given an iterator of [`ActionsBatch`]s (batches of actions read from the log) and a predicate, -/// returns an iterator of [`ScanMetadata`]s (which includes the files to be scanned as -/// [`FilteredEngineData`] and transforms that must be applied to correctly read the data). Each row -/// that is selected in the returned `engine_data` _must_ be processed to complete the scan. -/// Non-selected rows _must_ be ignored. +/// returns a tuple of: +/// 1. An iterator of [`ScanMetadata`]s (which includes the files to be scanned as +/// [`FilteredEngineData`] and transforms that must be applied to correctly read the data). +/// 2. An `Arc` containing metrics collected during log replay. +/// +/// Each row that is selected in the returned `engine_data` _must_ be processed to complete the +/// scan. Non-selected rows _must_ be ignored. /// -/// When `skip_stats` is true, file statistics are not read from checkpoint parquet files and -/// data skipping is disabled. +/// When `stats_options.skip_stats` is true, file statistics are not read from checkpoint parquet +/// files and columnar data skipping is disabled (no stats-based or partition-value-based +/// pruning), but row-level partition filtering still applies. /// /// Note: The iterator of [`ActionsBatch`]s ('action_iter' parameter) must be sorted by the order of /// the actions in the log from most recent to least recent. @@ -809,12 +1183,21 @@ pub(crate) fn scan_action_iter( action_iter: impl Iterator>, state_info: Arc, checkpoint_info: CheckpointReadInfo, - skip_stats: bool, -) -> DeltaResult>> { - Ok( - ScanLogReplayProcessor::new(engine, state_info, checkpoint_info, skip_stats)? - .process_actions_iter(action_iter), - ) + stats_options: ScanStatsOptions, + partition_values_options: ScanPartitionValuesOptions, +) -> DeltaResult<( + impl Iterator>, + Arc, +)> { + let processor = ScanLogReplayProcessor::new( + engine, + state_info, + checkpoint_info, + stats_options, + partition_values_options, + )?; + let metrics = processor.metrics.clone(); + Ok((processor.process_actions_iter(action_iter), metrics)) } #[cfg(test)] @@ -822,10 +1205,18 @@ mod tests { use std::collections::{HashMap, HashSet}; use std::sync::Arc; + use rstest::rstest; + + use super::{ + get_add_transform_expr, parsed_partition_values_expr, scan_action_iter, InternalScanState, + ScanLogReplayProcessor, ScanPartitionValuesOptions, ScanStatsOptions, + SerializableScanState, + }; use crate::actions::get_commit_schema; use crate::engine::sync::SyncEngine; use crate::expressions::{ - BinaryExpressionOp, OpaquePredicateOp, Predicate, Scalar, ScalarExpressionEvaluator, + col, column_name, lit, BinaryExpressionOp, Expression, MapToStructOptions, + OpaquePredicateOp, Predicate, Scalar, ScalarExpressionEvaluator, UnaryExpressionOp, }; use crate::kernel_predicates::{ DirectDataSkippingPredicateEvaluator, DirectPredicateEvaluator, @@ -840,20 +1231,103 @@ mod tests { use crate::scan::state_info::StateInfo; use crate::scan::test_utils::{ add_batch_for_row_id, add_batch_simple, add_batch_with_partition_col, - add_batch_with_remove, run_with_validate_callback, + add_batch_with_remove, add_batch_with_remove_and_partition, run_with_validate_callback, }; use crate::scan::PhysicalPredicate; - use crate::schema::MetadataColumnSpec; - use crate::schema::{DataType, SchemaRef, StructField, StructType}; + use crate::schema::{ + schema_ref, DataType, MetadataColumnSpec, SchemaRef, StructField, StructType, + }; use crate::table_features::ColumnMappingMode; - use crate::utils::test_utils::assert_result_error_with_message; - use crate::DeltaResult; - use crate::Expression as Expr; - use crate::ExpressionRef; + use crate::unit_test_utils::assert_result_error_with_message; + use crate::{DeltaResult, Expression as Expr, ExpressionRef}; - use super::{ - scan_action_iter, InternalScanState, ScanLogReplayProcessor, SerializableScanState, - }; + #[test] + fn partition_values_options_serde_preserves_timezone_and_defaults_when_absent() { + let options = ScanPartitionValuesOptions { + parsed_struct: true, + timestamp_timezone: Some("America/Los_Angeles".to_string()), + }; + let encoded = serde_json::to_string(&options).unwrap(); + let decoded: ScanPartitionValuesOptions = serde_json::from_str(&encoded).unwrap(); + assert!(decoded.parsed_struct); + assert_eq!( + decoded.timestamp_timezone.as_deref(), + Some("America/Los_Angeles") + ); + + let old: ScanPartitionValuesOptions = + serde_json::from_str(r#"{"parsed_struct":true}"#).unwrap(); + assert!(old.parsed_struct); + assert_eq!(old.timestamp_timezone, None); + } + + #[test] + fn parsed_partition_values_expr_parses_map_when_native_struct_is_absent() { + let raw = col!("add.partitionValues"); + let schema = + StructType::new_unchecked([StructField::nullable("physical_ts", DataType::TIMESTAMP)]); + + assert_eq!( + parsed_partition_values_expr(raw.clone(), None, &schema, None), + Expression::map_to_struct(raw.clone(), MapToStructOptions::default()) + ); + assert_eq!( + parsed_partition_values_expr(raw.clone(), None, &schema, Some("America/Los_Angeles")), + Expression::map_to_struct( + raw, + MapToStructOptions::default().with_timestamp_timezone("America/Los_Angeles"), + ) + ); + } + + #[test] + fn parsed_partition_values_expr_returns_native_struct_when_reparse_is_unnecessary() { + let raw = col!("add.partitionValues"); + let base = column_name!("add.partitionValues_parsed"); + let timestamp_schema = + StructType::new_unchecked([StructField::nullable("physical_ts", DataType::TIMESTAMP)]); + let timestamp_free_schema = StructType::new_unchecked([ + StructField::nullable("physical_ntz", DataType::TIMESTAMP_NTZ), + StructField::nullable("physical_int", DataType::INTEGER), + ]); + + assert_eq!( + parsed_partition_values_expr(raw.clone(), Some(base.clone()), ×tamp_schema, None), + Expression::from(base.clone()) + ); + assert_eq!( + parsed_partition_values_expr( + raw, + Some(base.clone()), + ×tamp_free_schema, + Some("America/Los_Angeles") + ), + Expression::from(base) + ); + } + + #[test] + fn parsed_partition_values_expr_reparses_complete_map_for_reader_timezone() { + let raw = col!("add.partitionValues"); + let base = column_name!("add.partitionValues_parsed"); + let schema = StructType::new_unchecked([ + StructField::nullable("physical_ts", DataType::TIMESTAMP), + StructField::nullable("physical_ntz", DataType::TIMESTAMP_NTZ), + StructField::nullable("physical_int", DataType::INTEGER), + ]); + + let actual = parsed_partition_values_expr( + raw.clone(), + Some(base.clone()), + &schema, + Some("America/Los_Angeles"), + ); + let expected = Expression::map_to_struct( + raw, + MapToStructOptions::default().with_timestamp_timezone("America/Los_Angeles"), + ); + assert_eq!(actual, expected); + } fn test_checkpoint_info() -> CheckpointReadInfo { CheckpointReadInfo::without_stats_parsed() @@ -950,15 +1424,19 @@ mod tests { column_mapping_mode: ColumnMappingMode::None, physical_stats_schema: None, physical_partition_schema: None, + physical_stats_columns: HashSet::new(), + is_catalog_managed: false, + skip_row_transforms: false, }); - let iter = scan_action_iter( + let (iter, _metrics) = scan_action_iter( &SyncEngine::new(), batch .into_iter() .map(|batch| Ok(ActionsBatch::new(batch as _, true))), state_info, test_checkpoint_info(), - false, + ScanStatsOptions::default(), + ScanPartitionValuesOptions::default(), ) .unwrap(); for res in iter { @@ -979,37 +1457,39 @@ mod tests { let partition_cols = vec!["date".to_string()]; let state_info = get_simple_state_info(schema, partition_cols).unwrap(); let batch = vec![add_batch_with_partition_col()]; - let iter = scan_action_iter( + let (iter, _metrics) = scan_action_iter( &SyncEngine::new(), batch .into_iter() .map(|batch| Ok(ActionsBatch::new(batch as _, true))), Arc::new(state_info), test_checkpoint_info(), - false, + ScanStatsOptions::default(), + ScanPartitionValuesOptions::default(), ) .unwrap(); - fn validate_transform(transform: Option<&ExpressionRef>, expected_date_offset: i32) { - assert!(transform.is_some()); - let Expr::Transform(transform) = transform.unwrap().as_ref() else { - panic!("Transform should always be a Transform expr"); + fn validate_patch(patch_expr: Option<&ExpressionRef>, expected_date_offset: i32) { + assert!(patch_expr.is_some()); + let Expr::StructPatch(patch) = patch_expr.unwrap().as_ref() else { + panic!("Expression should always be a StructPatch expr"); }; - // With sparse transforms, we expect only one insertion for the partition column - assert!(transform.prepended_fields.is_empty()); - let mut field_transforms = transform.field_transforms.iter(); - let (field_name, field_transform) = field_transforms.next().unwrap(); + // With sparse patches, we expect only one insertion for the partition column. + assert!(patch.prepended_fields.is_empty()); + assert!(patch.appended_fields.is_empty()); + let mut field_patches = patch.field_patches.iter(); + let (field_name, field_patch) = field_patches.next().unwrap(); assert_eq!(field_name, "value"); - assert!(!field_transform.is_replace); - let [expr] = &field_transform.exprs[..] else { + assert!(field_patch.keep_input); + let [expr] = &field_patch.insertions[..] else { panic!("Expected a single insertion"); }; let Expr::Literal(Scalar::Date(date_offset)) = expr.as_ref() else { panic!("Expected a literal date"); }; assert_eq!(*date_offset, expected_date_offset); - assert!(field_transforms.next().is_none()); + assert!(field_patches.next().is_none()); } for res in iter { @@ -1020,18 +1500,14 @@ mod tests { assert_eq!(transforms.len(), 4, "Should have 4 transforms"); assert!(transforms[0].is_none(), "transform at [0] should be None"); assert!(transforms[2].is_none(), "transform at [2] should be None"); - validate_transform(transforms[1].as_ref(), 17511); - validate_transform(transforms[3].as_ref(), 17510); + validate_patch(transforms[1].as_ref(), 17511); + validate_patch(transforms[3].as_ref(), 17510); } } #[test] - fn test_row_id_transform() { - let schema: SchemaRef = Arc::new(StructType::new_unchecked([StructField::new( - "value", - DataType::INTEGER, - true, - )])); + fn test_row_id_patch() { + let schema: SchemaRef = schema_ref! { nullable "value": INTEGER }; let state_info = get_state_info( schema.clone(), vec![], @@ -1064,14 +1540,15 @@ mod tests { ); let batch = vec![add_batch_for_row_id(get_commit_schema().clone())]; - let iter = scan_action_iter( + let (iter, _metrics) = scan_action_iter( &SyncEngine::new(), batch .into_iter() .map(|batch| Ok(ActionsBatch::new(batch as _, true))), Arc::new(state_info), test_checkpoint_info(), - false, + ScanStatsOptions::default(), + ScanPartitionValuesOptions::default(), ) .unwrap(); @@ -1079,26 +1556,26 @@ mod tests { let scan_metadata = res.unwrap(); let transforms = scan_metadata.scan_file_transforms; assert_eq!(transforms.len(), 1, "Should have 1 transform"); - if let Some(Expr::Transform(transform_expr)) = transforms[0].as_ref().map(Arc::as_ref) { - assert!(transform_expr.input_path.is_none()); - let row_id_transform = transform_expr - .field_transforms + if let Some(Expr::StructPatch(patch)) = transforms[0].as_ref().map(Arc::as_ref) { + assert!(patch.input_path.is_none()); + let row_id_patch = patch + .field_patches .get("row_id_col") - .expect("Should have row_id_col transform"); - assert!(row_id_transform.is_replace); - assert_eq!(row_id_transform.exprs.len(), 1); - let expr = &row_id_transform.exprs[0]; - let expeceted_expr = Arc::new(Expr::coalesce([ - Expr::column(["row_id_col"]), + .expect("Should have row_id_col patch"); + assert!(!row_id_patch.keep_input); + assert_eq!(row_id_patch.insertions.len(), 1); + let expr = &row_id_patch.insertions[0]; + let expected_expr = Arc::new(Expr::coalesce([ + col!("row_id_col"), Expr::binary( BinaryExpressionOp::Plus, - Expr::literal(42i64), - Expr::column(["row_indexes_for_row_id_0"]), + lit(42i64), + col!("row_indexes_for_row_id_0"), ), ])); - assert_eq!(expr, &expeceted_expr); + assert_eq!(expr, &expected_expr); } else { - panic!("Should have been a transform expression"); + panic!("Should have been a StructPatch expression"); } } } @@ -1116,7 +1593,8 @@ mod tests { &engine, Arc::new(get_simple_state_info(schema.clone(), vec![]).unwrap()), checkpoint_info.clone(), - false, + ScanStatsOptions::default(), + ScanPartitionValuesOptions::default(), ) .unwrap(); @@ -1164,10 +1642,7 @@ mod tests { StructField::new("id", DataType::INTEGER, true), StructField::new("value", DataType::STRING, true), ])); - let predicate = Arc::new(crate::expressions::Predicate::eq( - Expr::column(["id"]), - Expr::literal(10i32), - )); + let predicate = Arc::new(crate::expressions::Predicate::eq(col!("id"), lit(10i32))); let state_info = Arc::new( get_state_info( schema.clone(), @@ -1188,7 +1663,8 @@ mod tests { &engine, state_info.clone(), checkpoint_info.clone(), - false, + ScanStatsOptions::default(), + ScanPartitionValuesOptions::default(), ) .unwrap(); let deserialized = ScanLogReplayProcessor::from_serializable_state( @@ -1245,7 +1721,8 @@ mod tests { &engine, state_info.clone(), checkpoint_info.clone(), - false, + ScanStatsOptions::default(), + ScanPartitionValuesOptions::default(), ) .unwrap(); let deserialized = ScanLogReplayProcessor::from_serializable_state( @@ -1265,11 +1742,7 @@ mod tests { ColumnMappingMode::Id, ColumnMappingMode::Name, ] { - let schema: SchemaRef = Arc::new(StructType::new_unchecked([StructField::new( - "id", - DataType::INTEGER, - true, - )])); + let schema: SchemaRef = schema_ref! { nullable "id": INTEGER }; let state_info = Arc::new(StateInfo { logical_schema: schema.clone(), physical_schema: schema, @@ -1278,11 +1751,19 @@ mod tests { column_mapping_mode: mode, physical_stats_schema: None, physical_partition_schema: None, + physical_stats_columns: HashSet::new(), + is_catalog_managed: false, + skip_row_transforms: false, }); let checkpoint_info = test_checkpoint_info(); - let processor = - ScanLogReplayProcessor::new(&engine, state_info, checkpoint_info.clone(), false) - .unwrap(); + let processor = ScanLogReplayProcessor::new( + &engine, + state_info, + checkpoint_info.clone(), + ScanStatsOptions::default(), + ScanPartitionValuesOptions::default(), + ) + .unwrap(); let deserialized = ScanLogReplayProcessor::from_serializable_state( &engine, processor.into_serializable_state().unwrap(), @@ -1297,11 +1778,7 @@ mod tests { // Test edge cases: empty seen_file_keys, no predicate, no transform_spec let engine = SyncEngine::new(); let checkpoint_info = test_checkpoint_info(); - let schema: SchemaRef = Arc::new(StructType::new_unchecked([StructField::new( - "id", - DataType::INTEGER, - true, - )])); + let schema: SchemaRef = schema_ref! { nullable "id": INTEGER }; let state_info = Arc::new(StateInfo { logical_schema: schema.clone(), physical_schema: schema, @@ -1310,10 +1787,18 @@ mod tests { column_mapping_mode: ColumnMappingMode::None, physical_stats_schema: None, physical_partition_schema: None, + physical_stats_columns: HashSet::new(), + is_catalog_managed: false, + skip_row_transforms: false, }); - let processor = - ScanLogReplayProcessor::new(&engine, state_info, checkpoint_info.clone(), false) - .unwrap(); + let processor = ScanLogReplayProcessor::new( + &engine, + state_info, + checkpoint_info.clone(), + ScanStatsOptions::default(), + ScanPartitionValuesOptions::default(), + ) + .unwrap(); let serialized = processor.into_serializable_state().unwrap(); assert!(serialized.predicate.is_none()); let deserialized = @@ -1322,6 +1807,72 @@ mod tests { assert!(deserialized.state_info.transform_spec.is_none()); } + #[test] + fn test_serialization_round_trips_is_catalog_managed() { + let engine = SyncEngine::new(); + let schema: SchemaRef = schema_ref! { nullable "id": INTEGER }; + let state_info = Arc::new(StateInfo { + logical_schema: schema.clone(), + physical_schema: schema, + physical_predicate: PhysicalPredicate::None, + transform_spec: None, + column_mapping_mode: ColumnMappingMode::None, + physical_stats_schema: None, + physical_partition_schema: None, + physical_stats_columns: HashSet::new(), + is_catalog_managed: true, + skip_row_transforms: false, + }); + let processor = ScanLogReplayProcessor::new( + &engine, + state_info, + test_checkpoint_info(), + ScanStatsOptions::default(), + ScanPartitionValuesOptions::default(), + ) + .unwrap(); + let serialized = processor.into_serializable_state().unwrap(); + let deserialized = + ScanLogReplayProcessor::from_serializable_state(&engine, serialized).unwrap(); + assert!(deserialized.is_catalog_managed()); + } + + #[rstest] + fn test_serialization_round_trips_skip_row_transforms(#[values(false, true)] skip: bool) { + let engine = SyncEngine::new(); + let schema: SchemaRef = Arc::new(StructType::new_unchecked([StructField::new( + "id", + DataType::INTEGER, + true, + )])); + let state_info = Arc::new(StateInfo { + logical_schema: schema.clone(), + physical_schema: schema.clone(), + physical_predicate: PhysicalPredicate::None, + transform_spec: None, + column_mapping_mode: ColumnMappingMode::None, + physical_stats_schema: None, + physical_partition_schema: None, + physical_stats_columns: HashSet::new(), + is_catalog_managed: false, + skip_row_transforms: skip, + }); + let processor = ScanLogReplayProcessor::new( + &engine, + state_info, + test_checkpoint_info(), + ScanStatsOptions::default(), + ScanPartitionValuesOptions::default(), + ) + .unwrap(); + let deserialized = ScanLogReplayProcessor::from_serializable_state( + &engine, + processor.into_serializable_state().unwrap(), + ) + .unwrap(); + assert_eq!(deserialized.state_info.skip_row_transforms, skip); + } + #[test] fn test_serialization_invalid_json() { // Test that invalid JSON blobs are properly rejected @@ -1340,11 +1891,7 @@ mod tests { fn test_serialization_missing_predicate_schema() { // Test that missing predicate_schema when predicate exists is detected let engine = SyncEngine::new(); - let schema: SchemaRef = Arc::new(StructType::new_unchecked([StructField::new( - "id", - DataType::INTEGER, - true, - )])); + let schema: SchemaRef = schema_ref! { nullable "id": INTEGER }; let checkpoint_info = test_checkpoint_info(); let invalid_internal_state = InternalScanState { logical_schema: schema.clone(), @@ -1353,10 +1900,14 @@ mod tests { transform_spec: None, column_mapping_mode: ColumnMappingMode::None, physical_stats_schema: None, - skip_stats: false, + stats_options: ScanStatsOptions::default(), + partition_values_options: ScanPartitionValuesOptions::default(), physical_partition_schema: None, + physical_stats_columns: HashSet::new(), + is_catalog_managed: false, + skip_row_transforms: false, }; - let predicate = Arc::new(crate::expressions::Predicate::column(["id"])); + let predicate = Arc::new(crate::expressions::column_pred!("id")); let invalid_blob = serde_json::to_vec(&invalid_internal_state).unwrap(); let invalid_state = SerializableScanState { predicate: Some(predicate), // Predicate exists but schema is None @@ -1373,11 +1924,7 @@ mod tests { #[test] fn deserialize_internal_state_with_extry_fields_fails() { - let schema: SchemaRef = Arc::new(StructType::new_unchecked([StructField::new( - "id", - DataType::INTEGER, - true, - )])); + let schema: SchemaRef = schema_ref! { nullable "id": INTEGER }; let invalid_internal_state = InternalScanState { logical_schema: schema.clone(), physical_schema: schema, @@ -1385,8 +1932,12 @@ mod tests { transform_spec: None, column_mapping_mode: ColumnMappingMode::None, physical_stats_schema: None, - skip_stats: false, + stats_options: ScanStatsOptions::default(), + partition_values_options: ScanPartitionValuesOptions::default(), physical_partition_schema: None, + physical_stats_columns: HashSet::new(), + is_catalog_managed: false, + skip_row_transforms: false, }; let blob = serde_json::to_string(&invalid_internal_state).unwrap(); let mut obj: serde_json::Value = serde_json::from_str(&blob).unwrap(); @@ -1444,14 +1995,18 @@ mod tests { ])); let state_info = get_simple_state_info(schema, vec!["date".to_string()]).unwrap(); - let iter = scan_action_iter( + let (iter, _metrics) = scan_action_iter( &SyncEngine::new(), batch .into_iter() .map(|batch| Ok(ActionsBatch::new(batch as _, true))), Arc::new(state_info), test_checkpoint_info(), - true, + ScanStatsOptions { + skip_stats: true, + ..Default::default() + }, + ScanPartitionValuesOptions::default(), ) .unwrap(); @@ -1467,4 +2022,204 @@ mod tests { } assert!(found_add); } + + /// Verify that Remove actions are not pruned by data skipping. The transform reads from + /// `add.*` columns, so Remove rows produce null `stats_parsed` and `partitionValues_parsed` + /// (Remove actions have their own `remove.partitionValues` and `remove.stats`, but those + /// are not read by the transform). If a Remove were pruned, it would not be recorded in + /// `seen_file_keys`, and a subsequent Add for the same path could incorrectly survive + /// deduplication. + /// + /// Stats-based skipping is safe because null stats evaluate to NULL via ISNULL guards in + /// the predicate construction. Partition-based skipping requires the `is_add` guard + /// (`OR(NOT is_add, pred)`) because `eval_sql_where` adds IS NOT NULL guards that would + /// otherwise turn null partition values into `false`, filtering the Remove. + #[rstest] + #[case::stats_only( + Arc::new(StructType::new_unchecked([ + StructField::new("value", DataType::INTEGER, true), + ])), + vec![], + Arc::new(col!("value").gt(lit(5i32))), + false, // use batch without partition column + )] + #[case::partition_predicate( + Arc::new(StructType::new_unchecked([ + StructField::new("value", DataType::INTEGER, true), + StructField::new("date", DataType::DATE, true), + ])), + vec!["date".to_string()], + Arc::new(col!("date").eq(lit(Scalar::Date(17_510)))), + true, // use batch with partition column + )] + #[case::mixed_stats_and_partition( + Arc::new(StructType::new_unchecked([ + StructField::new("value", DataType::INTEGER, true), + StructField::new("date", DataType::DATE, true), + ])), + vec!["date".to_string()], + Arc::new(Predicate::and( + col!("value").gt(lit(5i32)), + col!("date").eq(lit(Scalar::Date(17_510))), + )), + true, // use batch with partition column + )] + fn data_skipping_does_not_prune_remove_actions( + #[case] schema: SchemaRef, + #[case] partition_columns: Vec, + #[case] predicate: Arc, + #[case] with_partition: bool, + ) { + let state_info = get_state_info( + schema, + partition_columns, + Some(predicate), + &[], + HashMap::new(), + vec![], + ) + .unwrap(); + + // Batch: [Remove c001, Add c001, Add c000, Metadata] + // The Remove must not be pruned -- it records c001 as seen, suppressing the c001 Add. + let batch = if with_partition { + vec![add_batch_with_remove_and_partition( + get_commit_schema().clone(), + )] + } else { + vec![add_batch_with_remove(get_commit_schema().clone())] + }; + let (iter, _metrics) = scan_action_iter( + &SyncEngine::new(), + batch + .into_iter() + .map(|batch| Ok(ActionsBatch::new(batch as _, true))), + Arc::new(state_info), + test_checkpoint_info(), + ScanStatsOptions::default(), + ScanPartitionValuesOptions::default(), + ) + .unwrap(); + + let mut add_paths: Vec = Vec::new(); + for res in iter { + let scan_metadata = res.unwrap(); + let paths = scan_metadata + .visit_scan_files( + Vec::new(), + |paths: &mut Vec, scan_file: ScanFile| { + paths.push(scan_file.path.to_string()); + }, + ) + .unwrap(); + add_paths.extend(paths); + } + + // Only c000 should survive: Remove suppressed c001 via deduplication + assert_eq!(add_paths.len(), 1, "Expected exactly one add to survive"); + assert!( + add_paths[0].contains("c000"), + "Expected c000 add to survive, got: {}", + add_paths[0] + ); + } + + /// Walk `expr` and count `Unary { op: ToJson, .. }` occurrences anywhere in the tree. + // Closures wrap `count_to_json` to dereference `&Arc` -> `&Expression`; + // clippy doesn't see the auto-deref so flags them as redundant. + #[allow(clippy::redundant_closure)] + fn count_to_json(expr: &Expression) -> usize { + match expr { + Expression::Unary(u) => { + let here = (u.op == UnaryExpressionOp::ToJson) as usize; + here + count_to_json(&u.expr) + } + Expression::Binary(b) => count_to_json(&b.left) + count_to_json(&b.right), + Expression::Variadic(v) => v.exprs.iter().map(|e| count_to_json(e)).sum(), + Expression::Struct(fields, nullability) => { + fields.iter().map(|e| count_to_json(e)).sum::() + + nullability.as_ref().map_or(0, |e| count_to_json(e)) + } + Expression::StructPatch(p) => { + p.prepended_fields + .iter() + .chain(p.appended_fields.iter()) + .map(|e| count_to_json(e)) + .sum::() + + p.field_patches + .values() + .flat_map(|fp| fp.insertions.iter()) + .map(|e| count_to_json(e)) + .sum::() + } + Expression::ParseJson(p) => count_to_json(&p.json_expr), + Expression::MapToStruct(m) => count_to_json(&m.map_expr), + Expression::ElementAt(e) => count_to_json(&e.map_expr) + count_to_json(&e.key_expr), + Expression::Cast(c) => count_to_json(&c.expr), + Expression::Predicate(_) + | Expression::Literal(_) + | Expression::Column(_) + | Expression::Opaque(_) + | Expression::Unknown(_) => 0, + } + } + + /// `synthesize_json=false` removes every `ToJson` node from the add transform; + /// `synthesize_json=true` leaves exactly one inside the COALESCE branch. + #[test] + fn add_transform_omits_to_json_when_synthesis_skipped() { + let stats_schema: SchemaRef = Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::LONG), + StructField::nullable("value", DataType::STRING), + ])); + let partition_schema: Option = Some(Arc::new(StructType::new_unchecked([ + StructField::nullable("date", DataType::DATE), + ]))); + + // Synthesis enabled: COALESCE branch present -> exactly one ToJson. + let with_synthesis = get_add_transform_expr( + Some(stats_schema.clone()), + true, // has_stats_parsed + false, // skip_stats + true, // synthesize_json + partition_schema.clone(), + false, // has_partition_values_parsed + None, // timestamp_timezone + ); + assert_eq!( + count_to_json(&with_synthesis), + 1, + "expected exactly one ToJson(add.stats_parsed) in the COALESCE branch when synthesis is enabled" + ); + + // Synthesis disabled: no ToJson anywhere in the transform. + let without_synthesis = get_add_transform_expr( + Some(stats_schema), + true, // has_stats_parsed + false, // skip_stats + false, // synthesize_json + partition_schema, + false, // has_partition_values_parsed + None, // timestamp_timezone + ); + assert_eq!( + count_to_json(&without_synthesis), + 0, + "expected no ToJson nodes anywhere in the transform when synthesis is skipped" + ); + assert!( + !without_synthesis + .references() + .contains(&column_name!("add.stats")), + "structured-only checkpoint transform must not reference add.stats" + ); + let Expression::Struct(fields, _) = without_synthesis.as_ref() else { + panic!("add transform must produce a struct"); + }; + assert_eq!( + fields[3].as_ref(), + &Expression::Literal(Scalar::Null(DataType::STRING)), + "structured-only checkpoint stats output must be a typed NULL" + ); + } } diff --git a/kernel/src/scan/metrics.rs b/kernel/src/scan/metrics.rs new file mode 100644 index 0000000000..6c7ffbd767 --- /dev/null +++ b/kernel/src/scan/metrics.rs @@ -0,0 +1,167 @@ +//! Metrics for scan log replay operations. + +use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; +use std::sync::Arc; +use std::time::Duration; + +use tracing::info; + +use crate::metrics::{MetricId, ScanMetadataCompleted, ScanType, TableType}; + +/// Metrics collected during scan log replay. Metrics are updated and read using relaxed ordering +/// to keep updates fast across parallel executing threads. +pub(crate) struct ScanMetrics { + /// Add files that entered deduplication. This normally excludes add files filtered by data + /// skipping. During parse-error fallback, deduplication runs first, so add files filtered by + /// retry-time data skipping are included. + num_add_files_seen: AtomicU64, + /// Add files that survived log replay (files to read). includes files that survived + /// dataskipping, partition pruning, and add/remove deduplication. + num_active_add_files: AtomicU64, + /// Number of bytes in the active add files as reported by the add action size field + active_add_files_bytes: AtomicU64, + /// Remove files seen (from delta/commit files only). + num_remove_files_seen: AtomicU64, + /// Non-file actions seen (protocol, metadata, etc.). + num_non_file_actions: AtomicU64, + /// Files filtered by predicates (data skipping + partition pruning). + num_predicate_filtered: AtomicU64, + /// Peak size of the deduplication hash set. + peak_hash_set_size: AtomicUsize, + /// Time spent in the deduplication visitor (ns). + dedup_visitor_time_ns: AtomicU64, + /// Time spent evaluating predicates (ns). This includes data skipping and partition pruning. + predicate_eval_time_ns: AtomicU64, +} + +impl Default for ScanMetrics { + fn default() -> Self { + Self { + num_add_files_seen: AtomicU64::new(0), + num_active_add_files: AtomicU64::new(0), + active_add_files_bytes: AtomicU64::new(0), + num_remove_files_seen: AtomicU64::new(0), + num_non_file_actions: AtomicU64::new(0), + num_predicate_filtered: AtomicU64::new(0), + peak_hash_set_size: AtomicUsize::new(0), + dedup_visitor_time_ns: AtomicU64::new(0), + predicate_eval_time_ns: AtomicU64::new(0), + } + } +} + +impl ScanMetrics { + pub(crate) fn incr_add_files_seen(&self) { + self.num_add_files_seen.fetch_add(1, Ordering::Relaxed); + } + + /// Record that we've seen an active add file, plus its size + pub(crate) fn record_active_add_file(&self, bytes: u64) { + self.num_active_add_files.fetch_add(1, Ordering::Relaxed); + self.active_add_files_bytes + .fetch_add(bytes, Ordering::Relaxed); + } + + pub(crate) fn incr_remove_files_seen(&self) { + self.num_remove_files_seen.fetch_add(1, Ordering::Relaxed); + } + + pub(crate) fn incr_non_file_actions(&self) { + self.num_non_file_actions.fetch_add(1, Ordering::Relaxed); + } + + pub(crate) fn add_predicate_filtered(&self, value: u64) { + self.num_predicate_filtered + .fetch_add(value, Ordering::Relaxed); + } + + pub(crate) fn update_peak_hash_set_size(&self, value: usize) { + self.peak_hash_set_size.fetch_max(value, Ordering::Relaxed); + } + + pub(crate) fn add_dedup_visitor_time_ns(&self, duration_ns: u64) { + self.dedup_visitor_time_ns + .fetch_add(duration_ns, Ordering::Relaxed); + } + + pub(crate) fn add_predicate_eval_time_ns(&self, duration_ns: u64) { + self.predicate_eval_time_ns + .fetch_add(duration_ns, Ordering::Relaxed); + } + + /// Reset counters to zero for a new phase. + /// + /// This is used between sequential and parallel phases to get fresh metrics + /// without reconstructing the entire processor. The peak hash set size is + /// preserved since it represents a high-water mark across all phases. + pub(crate) fn reset_counters(&self) { + self.num_add_files_seen.store(0, Ordering::Relaxed); + self.num_active_add_files.store(0, Ordering::Relaxed); + self.active_add_files_bytes.store(0, Ordering::Relaxed); + self.num_remove_files_seen.store(0, Ordering::Relaxed); + self.num_non_file_actions.store(0, Ordering::Relaxed); + self.num_predicate_filtered.store(0, Ordering::Relaxed); + self.dedup_visitor_time_ns.store(0, Ordering::Relaxed); + self.predicate_eval_time_ns.store(0, Ordering::Relaxed); + } + + /// Snapshot all counters into a [`ScanMetadataCompleted`] event payload. + /// + /// `scan_type` identifies whether this event was emitted by full scan metadata replay or by + /// a phase of parallel scan metadata replay. + pub(crate) fn to_event( + &self, + operation_id: MetricId, + is_catalog_managed: bool, + correlation_id: Option>, + scan_type: ScanType, + duration: Duration, + ) -> ScanMetadataCompleted { + ScanMetadataCompleted { + operation_id, + table_type: TableType::from_catalog_managed(is_catalog_managed), + correlation_id, + scan_type, + duration, + num_add_files_seen: self.num_add_files_seen.load(Ordering::Relaxed), + num_active_add_files: self.num_active_add_files.load(Ordering::Relaxed), + active_add_files_bytes: self.active_add_files_bytes.load(Ordering::Relaxed), + num_remove_files_seen: self.num_remove_files_seen.load(Ordering::Relaxed), + num_non_file_actions: self.num_non_file_actions.load(Ordering::Relaxed), + num_predicate_filtered: self.num_predicate_filtered.load(Ordering::Relaxed), + peak_hash_set_size: self.peak_hash_set_size.load(Ordering::Relaxed), + dedup_visitor_time: Duration::from_nanos( + self.dedup_visitor_time_ns.load(Ordering::Relaxed), + ), + predicate_eval_time: Duration::from_nanos( + self.predicate_eval_time_ns.load(Ordering::Relaxed), + ), + } + } + + /// Log all metrics with a message in the current tracing span context. + pub(crate) fn log(&self, message: impl AsRef) { + let add_files_seen = self.num_add_files_seen.load(Ordering::Relaxed); + let active_add_files = self.num_active_add_files.load(Ordering::Relaxed); + let active_add_files_bytes = self.active_add_files_bytes.load(Ordering::Relaxed); + let remove_files_seen = self.num_remove_files_seen.load(Ordering::Relaxed); + let non_file_actions = self.num_non_file_actions.load(Ordering::Relaxed); + let predicate_filtered = self.num_predicate_filtered.load(Ordering::Relaxed); + let peak_hash_set_size = self.peak_hash_set_size.load(Ordering::Relaxed); + let dedup_visitor_time_ns = self.dedup_visitor_time_ns.load(Ordering::Relaxed); + let predicate_eval_time_ns = self.predicate_eval_time_ns.load(Ordering::Relaxed); + info!( + add_files_seen, + active_add_files, + active_add_files_bytes, + remove_files_seen, + non_file_actions, + predicate_filtered, + peak_hash_set_size, + dedup_visitor_time_ns, + predicate_eval_time_ns, + "{}", + message.as_ref() + ); + } +} diff --git a/kernel/src/scan/mod.rs b/kernel/src/scan/mod.rs index d055969a09..f813e1317e 100644 --- a/kernel/src/scan/mod.rs +++ b/kernel/src/scan/mod.rs @@ -3,41 +3,57 @@ use std::borrow::Cow; use std::collections::{HashMap, HashSet}; use std::sync::{Arc, LazyLock}; +use std::time::Instant; use delta_kernel_derive::internal_api; use itertools::Itertools; -use tracing::debug; +use tracing::{debug, info}; use url::Url; use self::data_skipping::as_checkpoint_skipping_predicate; -use self::log_replay::get_scan_metadata_transform_expr; +use self::log_replay::{get_scan_metadata_transform_expr, scan_action_iter}; use crate::actions::deletion_vector::{ deletion_treemap_to_bools, split_vector, DeletionVectorDescriptor, }; -use crate::actions::{get_commit_schema, Add, ADD_NAME, REMOVE_NAME}; +use crate::actions::{Add, ADD_FIELD, ADD_NAME, NULL_COUNT, REMOVE_FIELD}; +use crate::cancellation::{CancellableIterator, CancellationTokenRef}; +#[cfg(feature = "declarative-plans")] +use crate::checkpoint::CheckpointShape; use crate::engine_data::FilteredEngineData; -use crate::expressions::{ColumnName, ExpressionRef, Predicate, PredicateRef, Scalar}; -use crate::kernel_predicates::{DefaultKernelPredicateEvaluator, EmptyColumnResolver}; +use crate::expressions::{column_name, ColumnName, ExpressionRef, Predicate, PredicateRef, Scalar}; +use crate::kernel_predicates::{ + DefaultKernelPredicateEvaluator, EmptyColumnResolver, KernelPredicateEvaluator as _, +}; use crate::log_replay::{ActionsBatch, HasSelectionVector}; use crate::log_segment::{ActionsWithCheckpointInfo, CheckpointReadInfo, LogSegment}; use crate::log_segment_files::LogSegmentFiles; +use crate::metrics::events::emit_scan_metadata_completed; +use crate::metrics::{MetricId, ScanType}; use crate::parallel::sequential_phase::SequentialPhase; -use crate::scan::log_replay::ScanLogReplayProcessor; -use crate::scan::log_replay::{BASE_ROW_ID_NAME, CLUSTERING_PROVIDER_NAME}; +#[cfg(feature = "declarative-plans")] +use crate::plans::ir::plan::Plan; +use crate::scan::log_replay::{ + ScanLogReplayProcessor, BASE_ROW_ID_NAME, CLUSTERING_PROVIDER_NAME, + DEFAULT_ROW_COMMIT_VERSION_NAME, +}; +use crate::scan::metrics::ScanMetrics; use crate::scan::state_info::StateInfo; use crate::schema::{ - ArrayType, DataType, MapType, PrimitiveType, Schema, SchemaRef, StructField, StructType, - ToSchema as _, + lazy_schema_ref, ArrayType, DataType, MapType, PrimitiveType, Schema, SchemaRef, StructField, + StructType, ToSchema as _, }; -use crate::table_features::{ColumnMappingMode, Operation}; -use crate::transforms::{ExpressionTransform, SchemaTransform}; +use crate::table_configuration::TableConfiguration; +use crate::table_features::{get_any_level_column_physical_name, ColumnMappingMode, Operation}; +use crate::transforms::{transform_output_type, ExpressionTransform, SchemaTransform}; +use crate::utils::{FoldWithOption as _, IteratorExt}; use crate::{DeltaResult, Engine, EngineData, Error, FileMeta, SnapshotRef, Version}; -use self::log_replay::scan_action_iter; - pub(crate) mod data_skipping; pub(crate) mod field_classifiers; pub mod log_replay; +pub(crate) mod metrics; +#[cfg(feature = "declarative-plans")] +pub(crate) mod scan_plan; pub mod state; pub(crate) mod state_info; pub(crate) mod transform_spec; @@ -48,21 +64,17 @@ pub(crate) mod test_utils; #[cfg(test)] mod tests; -// safety: we define get_commit_schema() and _know_ it contains ADD_NAME and REMOVE_NAME -#[allow(clippy::unwrap_used)] -pub(crate) static COMMIT_READ_SCHEMA: LazyLock = LazyLock::new(|| { - get_commit_schema() - .project(&[ADD_NAME, REMOVE_NAME]) - .unwrap() -}); -// safety: we define get_commit_schema() and _know_ it contains ADD_NAME and SIDECAR_NAME -#[allow(clippy::unwrap_used)] -pub(crate) static CHECKPOINT_READ_SCHEMA: LazyLock = - LazyLock::new(|| get_commit_schema().project(&[ADD_NAME]).unwrap()); - -/// Checkpoint schema WITHOUT stats for column projection pushdown. -/// When skip_stats is enabled, we use this schema to avoid reading the stats column from parquet. -pub(crate) static CHECKPOINT_READ_SCHEMA_NO_STATS: LazyLock = LazyLock::new(|| { +pub(crate) static COMMIT_READ_SCHEMA: LazyLock = lazy_schema_ref! { + (&ADD_FIELD), + (&REMOVE_FIELD), +}; +pub(crate) static CHECKPOINT_READ_SCHEMA: LazyLock = lazy_schema_ref! { + (&ADD_FIELD), +}; + +/// Initial checkpoint projection without JSON `add.stats`. +/// Discovery restores JSON stats when structured stats cannot satisfy the scan. +pub(crate) static CHECKPOINT_READ_SCHEMA_NO_JSON_STATS: LazyLock = LazyLock::new(|| { let add_schema = Add::to_schema(); let fields_no_stats: Vec<_> = add_schema .fields() @@ -81,41 +93,174 @@ pub use crate::parallel::parallel_scan_metadata::{ AfterSequentialScanMetadata, ParallelScanMetadata, ParallelState, SequentialScanMetadata, }; -/// Controls how file statistics are handled during a scan. +/// Configures structured-stats output and JSON synthesis in scan metadata. +/// Existing JSON passes through for commits and checkpoints without compatible structured stats +/// unless stats are disabled. /// -/// This enum determines whether and which statistics columns appear in scan metadata output, -/// and whether internal data skipping is enabled. -#[derive(Debug, Clone)] -pub enum StatsOutputMode { - /// Output all table stats columns in `stats_parsed`. - AllColumns, - /// Output stats for specific columns. An empty list means no stats output, but - /// predicate-based data skipping still works internally. +/// Most consumers should pick one of the named constructors: +/// - [`Self::json_only`] (default) -- JSON stats only. +/// - [`Self::all_struct`] -- all struct stats without JSON synthesis. Compatible checkpoints omit +/// JSON stats; commits and checkpoints without compatible structured stats pass existing JSON +/// through. +/// - [`Self::struct_columns`] -- selected struct stats with the same JSON behavior. +/// - [`Self::all`] -- both representations. +/// - [`Self::none`] -- neither, AND disables internal data skipping. Unlike the other four +/// constructors, this is the only one that stops kernel from reading stats from parquet at all. +#[derive(Clone, Debug)] +pub struct StatsOptions { + /// Whether to surface JSON stats on parsed-stats checkpoints (where the + /// checkpoint writes stats only as a struct, not as JSON). When true, kernel + /// re-serializes the struct stats into JSON so engines that read JSON stats + /// see a populated value; when false, JSON stats are left null on such + /// checkpoints and the engine consumes the struct stats directly. + /// + /// No effect on tables that write JSON stats directly, or on commit JSON -- + /// the existing JSON is passed through regardless. + pub(crate) synthesize_json: bool, + + /// Which struct stats columns to request in `stats_parsed`. + pub(crate) struct_stats: StructStats, +} + +/// Which struct stats columns appear in `stats_parsed` in scan metadata output. +#[derive(Clone, Debug)] +pub enum StructStats { + /// Don't emit `stats_parsed`. Kernel still reads predicate-referenced stats for + /// internal data skipping unless the caller picked [`StatsOptions::none`], which + /// disables stats reading entirely. + None, + /// Emit all indexed stats columns. + All, + /// Emit at least the specified stats columns. Predicate-referenced columns may also appear. Columns(Vec), - /// Skip reading stats entirely. Disables data skipping. - Skip, } -impl Default for StatsOutputMode { +impl Default for StatsOptions { + /// JSON only, no struct stats. fn default() -> Self { - StatsOutputMode::Columns(Vec::new()) + Self { + synthesize_json: true, + struct_stats: StructStats::None, + } + } +} + +impl StatsOptions { + /// JSON only. Equivalent to [`Default::default`]. + pub fn json_only() -> Self { + Self::default() + } + + /// All struct stats without JSON synthesis. Compatible checkpoints omit JSON stats and avoid + /// per-batch `ToJson`; commits and checkpoints without compatible structured stats pass + /// existing JSON through. + pub fn all_struct() -> Self { + Self { + synthesize_json: false, + struct_stats: StructStats::All, + } + } + + /// Struct stats for at least the specified columns without JSON synthesis. Predicate-referenced + /// columns may also appear because scan paths can retain stats used for data skipping. + pub fn struct_columns(cols: Vec) -> Self { + Self { + synthesize_json: false, + struct_stats: StructStats::Columns(cols), + } + } + + /// Both JSON and struct stats. Pays for both representations. + pub fn all() -> Self { + Self { + synthesize_json: true, + struct_stats: StructStats::All, + } + } + + /// **Disables all stats work**: no stats output, no internal data skipping (even + /// when a predicate is set). Kernel reads no stats columns from parquet at all. + /// Use when the engine handles its own pruning. + /// + /// To get internal predicate-based skipping without `stats_parsed` output, use + /// [`StatsOptions::default`] (JSON only) or set `struct_stats` to `All`/`Columns(_)`. + pub fn none() -> Self { + Self { + synthesize_json: false, + struct_stats: StructStats::None, + } + } +} + +/// Engine-facing partition value options. Pass to [`ScanBuilder::with_partition_values`] to +/// declare whether scan metadata output includes the typed `partitionValues_parsed` struct +/// alongside the raw string map (`fileConstantValues.partitionValues`), which is always present. +/// +/// When the typed struct is requested, scan metadata output gains a top-level +/// `partitionValues_parsed` struct column with one typed nullable field per partition column +/// (physical names, table partition-column order). On non-partitioned tables the column is +/// omitted. Values come directly from the checkpoint's native `partitionValues_parsed` column +/// when present, otherwise from parsing the string map. If a requested reader timezone changes +/// zoned timestamp interpretation, Kernel reparses the complete string map. +#[derive(Clone, Debug, Default)] +pub struct PartitionValuesOptions { + /// Whether to emit the typed `partitionValues_parsed` struct column. + pub(crate) parsed_struct: bool, + /// Reader timezone used to interpret offset-less zoned timestamp partition strings. + pub(crate) timestamp_timezone: Option, +} + +impl PartitionValuesOptions { + /// Raw string map only, no typed struct. Equivalent to [`Default::default`]. + pub fn string_map_only() -> Self { + Self::default() + } + + /// Emit the typed `partitionValues_parsed` struct alongside the raw string map. Lets engines + /// consume `partitionValues_parsed` directly instead of parsing the string map per row. + pub fn with_struct() -> Self { + Self { + parsed_struct: true, + timestamp_timezone: None, + } + } + + /// Interpret offset-less zoned `TIMESTAMP` partition strings in `timestamp_timezone`. + /// + /// The timezone may be an IANA name or a fixed offset. Explicit offsets in partition values + /// take precedence. Ambiguous local times use the earlier instant, and nonexistent local times + /// use the pre-transition offset. This option applies to typed scan metadata, internal + /// partition pruning, and partition-column row transforms used by [`Scan::execute`] for full + /// snapshot scans. Incremental scans continue to expose the raw partition-value map. Without + /// this option, Kernel interprets offset-less timestamps as UTC. Invalid timezones are reported + /// when scan metadata is evaluated. + pub fn with_timestamp_timezone(mut self, timestamp_timezone: impl Into) -> Self { + self.timestamp_timezone = Some(timestamp_timezone.into()); + self } } /// Builder to scan a snapshot of a table. pub struct ScanBuilder { snapshot: SnapshotRef, - schema: Option, + logical_read_schema: Option, predicate: Option, - stats_output_mode: StatsOutputMode, + stats: StatsOptions, + correlation_id: Option>, + without_row_transforms: bool, + partition_values: PartitionValuesOptions, + cancellation_token: Option, } impl std::fmt::Debug for ScanBuilder { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> { f.debug_struct("ScanBuilder") - .field("schema", &self.schema) + .field("logical_read_schema", &self.logical_read_schema) .field("predicate", &self.predicate) - .field("stats_output_mode", &self.stats_output_mode) + .field("stats", &self.stats) + .field("correlation_id", &self.correlation_id) + .field("without_row_transforms", &self.without_row_transforms) + .field("partition_values", &self.partition_values) .finish() } } @@ -125,9 +270,13 @@ impl ScanBuilder { pub fn new(snapshot: impl Into) -> Self { Self { snapshot: snapshot.into(), - schema: None, + logical_read_schema: None, predicate: None, - stats_output_mode: StatsOutputMode::default(), + stats: StatsOptions::default(), + correlation_id: None, + without_row_transforms: false, + partition_values: PartitionValuesOptions::default(), + cancellation_token: None, } } @@ -138,8 +287,8 @@ impl ScanBuilder { /// /// [`Schema`]: crate::schema::Schema /// [`Snapshot`]: crate::snapshot::Snapshot - pub fn with_schema(mut self, schema: SchemaRef) -> Self { - self.schema = Some(schema); + pub fn with_schema(mut self, logical_read_schema: SchemaRef) -> Self { + self.logical_read_schema = Some(logical_read_schema); self } @@ -148,79 +297,98 @@ impl ScanBuilder { /// /// [`Snapshot`]: crate::Snapshot pub fn with_schema_opt(self, schema_opt: Option) -> Self { - match schema_opt { - Some(schema) => self.with_schema(schema), - None => self, - } + self.fold_with(schema_opt, ScanBuilder::with_schema) } /// Optionally provide an expression to filter rows. For example, using the predicate `x < /// 4` to return a subset of the rows in the scan which satisfy the filter. If `predicate_opt` /// is `None`, this is a no-op. /// - /// NOTE: The filtering is best-effort and can produce false positives (rows that should should + /// NOTE: The filtering is best-effort and can produce false positives (rows that should /// have been filtered out but were kept). /// - /// This method can be combined with [`include_all_stats_columns`]. When both are used, the kernel - /// performs data skipping internally using the predicate AND outputs parsed statistics to the - /// engine via the `stats_parsed` column in scan metadata. + /// NOTE: Predicates referencing metadata columns the caller added to the projection via + /// [`StructType::add_metadata_column`] (row indexes, row ids, file paths) are not supported + /// and will error at build time. /// - /// [`include_all_stats_columns`]: ScanBuilder::include_all_stats_columns + /// A predicate alone enables internal data skipping; kernel does not surface stats + /// to the engine by default. Use [`with_stats`](Self::with_stats) if the engine + /// also wants stats in the scan metadata output. + /// + /// [`StructType::add_metadata_column`]: crate::schema::StructType::add_metadata_column pub fn with_predicate(mut self, predicate: impl Into>) -> Self { self.predicate = predicate.into(); self } - /// Include all parsed statistics in scan metadata. - /// - /// When enabled, the scan will include a `stats_parsed` column in the scan metadata - /// containing pre-parsed file statistics (minValues, maxValues, nullCount, numRecords) - /// that integrations can use for their own data skipping logic. + /// Configure stats output for the scan. See [`StatsOptions`]. /// - /// The statistics schema is determined by the table's configuration - /// (`delta.dataSkippingStatsColumns` or `delta.dataSkippingNumIndexedCols`). In the future, - /// a requested columns filter may limit which columns appear in the output without - /// affecting the table-level column counting. + /// Defaults to [`StatsOptions::default`] (JSON only). Engines that consume + /// `stats_parsed` directly should pass [`StatsOptions::all_struct`] so compatible + /// checkpoints omit JSON stats and skip `ToJson` synthesis. + pub fn with_stats(mut self, stats: StatsOptions) -> Self { + self.stats = stats; + self + } + + /// Attach an opaque, caller-supplied correlation id for joining this scan's metric events to + /// the caller's own request or operation id. An empty id is treated as unset. When unset, + /// behavior is unchanged. /// - /// This method can be combined with [`with_predicate`]. When both are used, the kernel - /// performs data skipping internally using the predicate AND outputs parsed statistics to the - /// engine via the `stats_parsed` column in scan metadata. + /// Note: like `operation_id`, the correlation id does not currently survive the + /// [`Scan::parallel_scan_metadata`] serialization boundary. A [`ParallelState`] reconstructed + /// from serialized bytes on a remote worker carries no correlation id (tracked in #2736). /// - /// [`with_predicate`]: ScanBuilder::with_predicate - pub fn include_all_stats_columns(mut self) -> Self { - self.stats_output_mode = StatsOutputMode::AllColumns; + /// [`ParallelState`]: crate::scan::ParallelState + pub fn with_correlation_id(mut self, correlation_id: impl Into>) -> Self { + self.correlation_id = Some(correlation_id.into()).filter(|id| !id.is_empty()); self } - /// Include specific columns in the scan metadata. + /// Declare that the engine will reconstruct logical rows itself and will not consume + /// [`ScanMetadata::scan_file_transforms`]. /// - /// When `columns` is non-empty, only those columns' statistics appear in `stats_parsed`. - /// When `columns` is empty, no stats are output (equivalent to the default behavior). + /// The kernel then skips building the per-file transform expressions and the per-row + /// partition-value parse done only to build them. The returned `scan_file_transforms` is left + /// empty (each row's transform is `None`); use [`Scan::scan_metadata`] for listing. /// - /// [`with_stats_columns`]: ScanBuilder::with_stats_columns - /// [`build`]: ScanBuilder::build - pub fn with_stats_columns(mut self, columns: Vec) -> Self { - self.stats_output_mode = StatsOutputMode::Columns(columns); + /// With this set the engine must itself apply every physical-to-logical fixup the transform + /// would normally perform: partition column injection, column-mapping renames, and generated + /// row ids. Deletion vectors are unaffected: they are delivered per file in the scan metadata + /// regardless. [`Scan::execute`] returns an error. + pub fn without_row_transforms(mut self) -> Self { + self.without_row_transforms = true; self } - /// Skip reading file statistics from checkpoint files. + /// Configure partition value output for the scan. See [`PartitionValuesOptions`]. /// - /// When enabled: - /// - Parquet checkpoint reads use column projection to skip the stats column - /// - The `stats` field in scan results will be `None` - /// - Data skipping is disabled (predicates still filter partitions, but not files) - /// - /// If called after [`include_all_stats_columns`] or [`with_stats_columns`], the last call wins. + /// Defaults to [`PartitionValuesOptions::default`] (string map only). Engines that + /// consume `partitionValues_parsed` directly should pass + /// [`PartitionValuesOptions::with_struct`] to also emit the typed struct column. + pub fn with_partition_values(mut self, partition_values: PartitionValuesOptions) -> Self { + self.partition_values = partition_values; + self + } + + /// Provide a [`CancellationToken`] so a cancelled request can stop an in-flight + /// [`scan_metadata`](Scan::scan_metadata) log replay instead of running to completion. /// - /// Use this when data skipping is handled externally (e.g., by the query engine). + /// Cancellation is cooperative: kernel polls the token at each action-batch boundary, and a + /// cancellation-aware [`Engine`] additionally races its checkpoint/commit reads against it. + /// On cancellation the scan surfaces [`Error::Cancelled`] -- either returned directly from + /// [`scan_metadata`](Scan::scan_metadata) when the token is already cancelled before replay + /// begins, or as the terminal item of its iterator -- never as a silent early `None`, so a + /// cancelled listing cannot be mistaken for a complete one. With no token the scan is not + /// cancellable. /// - /// [`include_all_stats_columns`]: ScanBuilder::include_all_stats_columns - /// [`with_stats_columns`]: ScanBuilder::with_stats_columns - pub fn with_skip_stats(mut self, skip_stats: bool) -> Self { - if skip_stats { - self.stats_output_mode = StatsOutputMode::Skip; - } + /// [`CancellationToken`]: crate::CancellationToken + /// [`Error::Cancelled`]: crate::Error::Cancelled + pub fn with_cancellation_token( + mut self, + token: impl Into>, + ) -> Self { + self.cancellation_token = token.into(); self } @@ -231,25 +399,57 @@ impl ScanBuilder { /// [`Scan`] type itself can be used to fetch the files and associated metadata required to /// perform actual data reads. pub fn build(self) -> DeltaResult { + // Predicates may reference columns outside self.logical_read_schema, so resolve against the + // full table schema + let table_schema = self.snapshot.schema(); + // Reject scans of empty-schema tables. CREATE TABLE accepts an empty schema as + // a transient state, but a scan over zero columns has no way to derive row + // counts downstream and panics in the arrow layer. Users must populate the + // schema with ALTER TABLE ADD COLUMN before scanning. + if table_schema.num_fields() == 0 { + return Err(Error::generic( + "Cannot scan Delta table with empty schema; use ALTER TABLE ADD COLUMN \ + to add at least one column before scanning", + )); + } + // if no schema is provided, use snapshot's entire schema (e.g. SELECT *) - let logical_schema = self.schema.unwrap_or_else(|| self.snapshot.schema()); + let logical_read_schema = self + .logical_read_schema + .unwrap_or_else(|| table_schema.clone()); self.snapshot .table_configuration() .ensure_operation_supported(Operation::Scan)?; - let state_info = StateInfo::try_new( - logical_schema, + let mut state_info = StateInfo::try_new( + logical_read_schema, + table_schema, self.snapshot.table_configuration(), self.predicate, - self.stats_output_mode.clone(), + &self.stats, + &self.partition_values, (), // No classifier, default is for scans )?; + // Retain the transform spec but skip building per-file expressions, which also skips the + // per-row partition-value parse done only to build them. + state_info.skip_row_transforms = self.without_row_transforms; + + let physical_stats_output_schema = build_physical_stats_output_schema( + self.snapshot.table_configuration(), + &state_info, + &self.stats, + )?; + Ok(Scan { snapshot: self.snapshot, state_info: Arc::new(state_info), - stats_output_mode: self.stats_output_mode, + stats: self.stats, + physical_stats_output_schema, + correlation_id: self.correlation_id, + partition_values: self.partition_values, + cancellation_token: self.cancellation_token, }) } } @@ -262,10 +462,10 @@ pub(crate) enum PhysicalPredicate { } impl PhysicalPredicate { - /// If we have a predicate, verify the columns it references and apply column mapping. First, get - /// the set of references; use that to filter the schema to only the columns of interest (and - /// verify that all referenced columns exist); then use the resulting logical/physical mappings - /// to rewrite the expression with physical column names. + /// If we have a predicate, verify the columns it references and apply column mapping. First, + /// get the set of references; use that to filter the schema to only the columns of interest + /// (and verify that all referenced columns exist); then use the resulting logical/physical + /// mappings to rewrite the expression with physical column names. /// /// NOTE: It is possible the predicate resolves to FALSE even ignoring column references, /// e.g. `col > 10 AND FALSE`. Such predicates can statically skip the whole query. @@ -331,7 +531,6 @@ impl PhysicalPredicate { // the predicate allows to statically skip all files. Since this is direct evaluation (not an // expression rewrite), we use a `DefaultKernelPredicateEvaluator` with an empty column resolver. fn can_statically_skip_all_files(predicate: &Predicate) -> bool { - use crate::kernel_predicates::KernelPredicateEvaluator as _; let evaluator = DefaultKernelPredicateEvaluator::from(EmptyColumnResolver); evaluator.eval_sql_where(predicate) == Some(false) } @@ -354,6 +553,8 @@ struct GetReferencedFields<'a> { column_mapping_mode: ColumnMappingMode, } impl<'a> SchemaTransform<'a> for GetReferencedFields<'a> { + transform_output_type!(|'a, T| Option>); + // Capture the path mapping for this leaf field fn transform_primitive(&mut self, ptype: &'a PrimitiveType) -> Option> { // Record the physical name mappings for all referenced leaf columns. Delta column names @@ -393,16 +594,18 @@ impl<'a> SchemaTransform<'a> for GetReferencedFields<'a> { } } -/// Prefixes all column references in a predicate with a fixed path. -/// Transforms data-skipping predicates (e.g., `minValues.x > 100`) into -/// checkpoint/sidecar-compatible predicates (e.g., `add.stats_parsed.minValues.x > 100`). +/// Prefixes all column references in a predicate with a fixed path. The checkpoint path prefixes +/// stat expressions that already carry their `stats_parsed.*` root (e.g. `stats_parsed.minValues.x +/// > 100`) with `add`, yielding `add.stats_parsed.minValues.x > 100`. struct PrefixColumns { prefix: ColumnName, } impl<'a> ExpressionTransform<'a> for PrefixColumns { - fn transform_expr_column(&mut self, name: &'a ColumnName) -> Option> { - Some(Cow::Owned(self.prefix.join(name))) + transform_output_type!(|'a, T| Cow<'a, T>); + + fn transform_expr_column(&mut self, name: &'a ColumnName) -> Cow<'a, ColumnName> { + Cow::Owned(self.prefix.join(name)) } } @@ -410,6 +613,8 @@ struct ApplyColumnMappings { column_mappings: HashMap, } impl<'a> ExpressionTransform<'a> for ApplyColumnMappings { + transform_output_type!(|'a, T| Option>); + // NOTE: We already verified all column references. But if the map probe ever did fail, the // transform would just delete any expression(s) that reference the invalid column. fn transform_expr_column(&mut self, name: &'a ColumnName) -> Option> { @@ -420,8 +625,6 @@ impl<'a> ExpressionTransform<'a> for ApplyColumnMappings { } static RESTORED_ADD_SCHEMA: LazyLock = LazyLock::new(|| { - use crate::scan::log_replay::DEFAULT_ROW_COMMIT_VERSION_NAME; - let partition_values = MapType::new(DataType::STRING, DataType::STRING, true); StructType::new_unchecked(vec![StructField::nullable( "add", @@ -458,11 +661,12 @@ pub fn get_transform_for_row( transforms.get(row).cloned().flatten() } -/// [`ScanMetadata`] contains (1) a batch of [`FilteredEngineData`] specifying data files to be scanned -/// and (2) a vector of transforms (one transform per scan file) that must be applied to the data read -/// from those files. +/// [`ScanMetadata`] contains (1) a batch of [`FilteredEngineData`] specifying data files to be +/// scanned and (2) a vector of transforms (one transform per scan file) that must be applied to the +/// data read from those files. pub struct ScanMetadata { - /// Filtered engine data with one row per file to scan (and only selected rows should be scanned) + /// Filtered engine data with one row per file to scan (and only selected rows should be + /// scanned) pub scan_files: FilteredEngineData, /// Row-level transformations to apply to data read from files. @@ -504,7 +708,55 @@ impl HasSelectionVector for ScanMetadata { pub struct Scan { snapshot: SnapshotRef, state_info: Arc, - stats_output_mode: StatsOutputMode, + stats: StatsOptions, + #[allow(dead_code)] // Only used when `declarative-plans` is enabled + physical_stats_output_schema: Option, + correlation_id: Option>, + partition_values: PartitionValuesOptions, + /// Optional cooperative cancellation token supplied via + /// [`ScanBuilder::with_cancellation_token`]. `None` means the scan is not cancellable. + cancellation_token: Option, +} + +/// Builds the physical `stats_parsed` output schema requested through `StatsOptions`. +/// +/// For example, if the caller requests `[a, b]` and the predicate references `c`, +/// `StateInfo::physical_stats_schema` contains `[a, b, c]`, while this returns `[a, b]`. +/// Returns `None` when no eligible struct stats are requested and errors when a requested column +/// cannot be resolved. +fn build_physical_stats_output_schema( + table_configuration: &TableConfiguration, + state_info: &StateInfo, + stats: &StatsOptions, +) -> DeltaResult> { + match &stats.struct_stats { + StructStats::None => Ok(None), + StructStats::All => Ok(state_info.physical_stats_schema.clone()), + StructStats::Columns(columns) if columns.is_empty() => Ok(None), + StructStats::Columns(columns) => { + let logical_schema = table_configuration.logical_schema(); + let column_mapping_mode = table_configuration.column_mapping_mode(); + let physical_columns: Vec<_> = columns + .iter() + .map(|column| { + get_any_level_column_physical_name(&logical_schema, column, column_mapping_mode) + }) + .try_collect()?; + let stats_schema = table_configuration + .build_expected_stats_schemas(None, Some(&physical_columns))? + .physical; + + Ok(stats_schema_with_data_columns(stats_schema)) + } + } +} + +/// Returns `schema` only when it contains stats for at least one data column. +/// +/// Expected stats schemas always contain `numRecords` and `tightBounds`. `nullCount` is present +/// only when at least one data column survives stats filtering. +fn stats_schema_with_data_columns(schema: SchemaRef) -> Option { + schema.field(NULL_COUNT).is_some().then_some(schema) } impl std::fmt::Debug for Scan { @@ -512,20 +764,65 @@ impl std::fmt::Debug for Scan { f.debug_struct("Scan") .field("schema", &self.state_info.logical_schema) .field("predicate", &self.state_info.physical_predicate) - .field("stats_output_mode", &self.stats_output_mode) + .field("stats", &self.stats) + .field("correlation_id", &self.correlation_id) .finish() } } impl Scan { - /// Whether stats reading is entirely skipped, disabling data skipping. + /// Whether stats reading is entirely skipped, disabling internal data skipping. fn skip_stats(&self) -> bool { - matches!(self.stats_output_mode, StatsOutputMode::Skip) + !self.stats.synthesize_json && matches!(self.stats.struct_stats, StructStats::None) + } + + fn checkpoint_read_options(&self) -> (SchemaRef, Option, Option<&StructType>) { + let skip_stats = self.skip_stats(); + // `physical_stats_schema` is the typed shape this scan can consume, not evidence that the + // checkpoint contains `stats_parsed`. Checkpoint discovery validates availability and + // restores `add.stats` before opening the reader when the structured field is incompatible. + let can_replace_json_with_structured_stats = + !self.stats.synthesize_json && self.state_info.physical_stats_schema.is_some(); + let checkpoint_schema = if skip_stats || can_replace_json_with_structured_stats { + CHECKPOINT_READ_SCHEMA_NO_JSON_STATS.clone() + } else { + CHECKPOINT_READ_SCHEMA.clone() + }; + + let meta_predicate = if skip_stats { + None + } else { + self.build_actions_meta_predicate() + }; + // Discovery uses this schema to augment the checkpoint projection, so `none()` must + // suppress it as well as the initial JSON stats field. + let physical_stats_schema = if skip_stats { + None + } else { + self.state_info.physical_stats_schema.as_deref() + }; + (checkpoint_schema, meta_predicate, physical_stats_schema) + } + + /// Build the read-options bundle passed to [`ScanLogReplayProcessor`]. + fn stats_options(&self) -> log_replay::ScanStatsOptions { + log_replay::ScanStatsOptions { + skip_stats: self.skip_stats(), + synthesize_json: self.stats.synthesize_json, + } + } + + /// Build the partition-value read options passed to [`ScanLogReplayProcessor`]. + fn partition_values_options(&self) -> log_replay::ScanPartitionValuesOptions { + log_replay::ScanPartitionValuesOptions { + parsed_struct: self.partition_values.parsed_struct, + timestamp_timezone: self.partition_values.timestamp_timezone.clone(), + } } /// The table's root URL. Any relative paths returned from `scan_data` (or in a callback from - /// [`ScanMetadata::visit_scan_files`]) must be resolved against this root to get the actual path to - /// the file. + /// [`ScanMetadata::visit_scan_files`]) must be resolved against this root to get the actual + /// path to the file. /// /// [`ScanMetadata::visit_scan_files`]: crate::scan::ScanMetadata::visit_scan_files // NOTE: this is obviously included in the snapshot, just re-exposed here for convenience. @@ -568,20 +865,26 @@ impl Scan { /// Get an iterator of [`ScanMetadata`]s that should be used to facilitate a scan. This handles /// log-replay, reconciling Add and Remove actions, and applying data skipping (if possible). + /// + /// Reports metrics: [`MetricEvent::ScanMetadataCompleted`] when the returned iterator is + /// fully exhausted. + /// + /// [`MetricEvent::ScanMetadataCompleted`]: crate::metrics::MetricEvent::ScanMetadataCompleted + /// /// Each item in the returned iterator is a struct of: /// - `Box`: Data in engine format, where each row represents a file to be /// scanned. The schema for each row can be obtained by calling [`scan_row_schema`]. /// - `Vec`: A selection vector. If a row is at index `i` and this vector is `false` at - /// index `i`, then that row should *not* be processed (i.e. it is filtered out). If the vector - /// is `true` at index `i` the row *should* be processed. If the selection vector is *shorter* - /// than the number of rows returned, missing elements are considered `true`, i.e. included in - /// the query. NB: If you are using the default engine and plan to call arrow's + /// index `i`, then that row should *not* be processed (i.e. it is filtered out). If the + /// vector is `true` at index `i` the row *should* be processed. If the selection vector is + /// *shorter* than the number of rows returned, missing elements are considered `true`, i.e. + /// included in the query. NB: If you are using the default engine and plan to call arrow's /// `filter_record_batch`, you _need_ to extend this vector to the full length of the batch or /// arrow will drop the extra rows. /// - `Vec>`: Transformation expressions that need to be applied. For each - /// row at index `i` in the above data, if an expression exists at index `i` in the `Vec`, - /// the associated expression _must_ be applied to the data read from the file specified by - /// the row. The resultant schema for this expression is guaranteed to be + /// row at index `i` in the above data, if an expression exists at index `i` in the `Vec`, the + /// associated expression _must_ be applied to the data read from the file specified by the + /// row. The resultant schema for this expression is guaranteed to be /// [`Self::logical_schema()`]. If the item at index `i` in this `Vec` is `None`, or if the /// `Vec` contains fewer than `i` elements, no expression need be applied and the data read /// from disk is already in the correct logical state. @@ -593,16 +896,17 @@ impl Scan { self.scan_metadata_inner(engine, actions_with_checkpoint_info) } - /// Get an updated iterator of [`ScanMetadata`]s based on an existing iterator of [`EngineData`]s. + /// Get an updated iterator of [`ScanMetadata`]s based on an existing iterator of + /// [`EngineData`]s. /// /// The existing iterator is assumed to contain data from a previous call to `scan_metadata`. /// Engines may decide to cache the results of `scan_metadata` to avoid additional IO operations /// required to replay the log. /// - /// As such the new scan's predicate must "contain" the previous scan's predicate. That is, the new - /// scan's predicate MUST skip all files the previous scan's predicate skipped. The new scan's - /// predicate is also allowed to skip files the previous predicate kept. For example, if the previous - /// scan predicate was + /// As such the new scan's predicate must "contain" the previous scan's predicate. That is, the + /// new scan's predicate MUST skip all files the previous scan's predicate skipped. The new + /// scan's predicate is also allowed to skip files the previous predicate kept. For example, + /// if the previous scan predicate was /// ```sql /// WHERE a < 42 AND b = 10 /// ``` @@ -711,23 +1015,18 @@ impl Scan { // For incremental reads, new_log_segment has no checkpoint but we use the // checkpoint schema returned by the function for consistency. - let (checkpoint_schema, meta_predicate) = if self.skip_stats() { - (CHECKPOINT_READ_SCHEMA_NO_STATS.clone(), None) - } else { - ( - CHECKPOINT_READ_SCHEMA.clone(), - self.build_actions_meta_predicate(), - ) - }; + let (checkpoint_schema, meta_predicate, physical_stats_schema) = + self.checkpoint_read_options(); let result = new_log_segment.read_actions_with_projected_checkpoint_actions( engine, COMMIT_READ_SCHEMA.clone(), checkpoint_schema, meta_predicate, - self.state_info - .physical_stats_schema - .as_ref() - .map(|s| s.as_ref()), + physical_stats_schema, + None, + // The incremental path relies on the batch-boundary poll in `scan_metadata_inner` + // for cancellation; it does not thread the token into the engine reads here, so a + // read already in flight is not interrupted mid-I/O. None, )?; let actions_with_checkpoint_info = ActionsWithCheckpointInfo { @@ -750,17 +1049,70 @@ impl Scan { impl Iterator>, >, ) -> DeltaResult>> { - if let PhysicalPredicate::StaticSkipAll = self.state_info.physical_predicate { - return Ok(None.into_iter().flatten()); - } - let it = scan_action_iter( - engine, - actions_with_checkpoint_info.actions, - self.state_info.clone(), - actions_with_checkpoint_info.checkpoint_info, - self.skip_stats(), + let start = Instant::now(); + let operation_id = MetricId::new(); + let is_catalog_managed = self.snapshot.table_configuration().is_catalog_managed(); + let correlation_id = self.correlation_id.clone(); + + let (iter, metrics) = match self.state_info.physical_predicate { + PhysicalPredicate::StaticSkipAll => { + info!("Predicate statically evaluated to false; skipping all files"); + (None, Arc::new(ScanMetrics::default())) + } + _ => { + // Wrap the input iterator (not the shared `process_actions_iter`) so token + // polling stays scoped to scans. + let actions = CancellableIterator::new( + actions_with_checkpoint_info.actions, + self.cancellation_token.clone(), + ); + let (it, m) = scan_action_iter( + engine, + actions, + self.state_info.clone(), + actions_with_checkpoint_info.checkpoint_info, + self.stats_options(), + self.partition_values_options(), + )?; + (Some(it), m) + } + }; + + let on_complete = move || { + let event = metrics.to_event( + operation_id, + is_catalog_managed, + correlation_id, + ScanType::Full, + start.elapsed(), + ); + info!(%event); + emit_scan_metadata_completed(&event); + }; + Ok(iter.into_iter().flatten().on_complete(on_complete)) + } + + #[cfg(feature = "declarative-plans")] + /// Builds a declarative plan that produces the scan's live `add` actions. + /// + /// `engine` supplies the plan executor used to inspect checkpoint shape. Returns `None` when + /// no Delta metadata matches this scan. + /// + /// # Errors + /// + /// Returns an error if the engine provides no [`PlanExecutor`](crate::plans::PlanExecutor), + /// or if log discovery, checkpoint inspection, or plan construction fails. + pub fn declarative_metadata_scan_plan(&self, engine: &dyn Engine) -> DeltaResult> { + // Resolve the checkpoint shape once: it selects the leaf-vs-manifest arm and reports + // whether the checkpoint carries compatible parsed stats and partition columns. + let plan_executor = engine.require_plan_executor()?; + let shape = CheckpointShape::try_new( + plan_executor.as_ref(), + &self.snapshot, + self.state_info.physical_stats_schema.as_ref(), + self.state_info.physical_partition_schema.as_ref(), )?; - Ok(Some(it).into_iter().flatten()) + self.build_metadata_scan_plan(&shape) } // Factored out to facilitate testing @@ -770,14 +1122,10 @@ impl Scan { ) -> DeltaResult< ActionsWithCheckpointInfo> + Send>, > { - let (checkpoint_schema, meta_predicate) = if self.skip_stats() { - (CHECKPOINT_READ_SCHEMA_NO_STATS.clone(), None) - } else { - ( - CHECKPOINT_READ_SCHEMA.clone(), - self.build_actions_meta_predicate(), - ) - }; + let (checkpoint_schema, meta_predicate, physical_stats_schema) = + self.checkpoint_read_options(); + // Checkpoints already represent reconciled state, so scans project only Add actions. This + // derives `add.path IS NOT NULL` and allows readers to skip non-Add row groups. self.snapshot .log_segment() .read_actions_with_projected_checkpoint_actions( @@ -785,46 +1133,73 @@ impl Scan { COMMIT_READ_SCHEMA.clone(), checkpoint_schema, meta_predicate, - self.state_info - .physical_stats_schema - .as_ref() - .map(|s| s.as_ref()), + physical_stats_schema, self.state_info .physical_partition_schema .as_ref() .map(|s| s.as_ref()), + self.cancellation_token.as_ref(), ) } /// Builds a predicate for row group skipping in checkpoint and sidecar parquet files. /// - /// The scan predicate is first transformed into a data-skipping form with IS NULL guards - /// (e.g., `x > 100` becomes `OR(maxValues.x IS NULL, maxValues.x > 100)`), then column - /// references are prefixed with `add.stats_parsed` to match the physical column layout - /// of checkpoint/sidecar files. The parquet reader's row group filter can then use - /// parquet-level statistics on these nested columns to skip entire row groups that cannot - /// contain matching files. + /// The scan predicate is rewritten into a data-skipping form scoped under the `add` action: + /// data-column references become `add.stats_parsed.{minValues,maxValues,nullCount}.` and + /// partition references become `add.partitionValues_parsed.`, so the parquet reader's row + /// group filter can use footer statistics to skip row groups that cannot contain matching + /// files. See [`as_checkpoint_skipping_predicate`] for the rewrite and its data-stat IS NULL + /// guards. /// - /// The IS NULL guards are necessary because parquet footer min/max statistics ignore null - /// values. Without them, row groups containing files with missing stats (null stat columns) - /// could be incorrectly pruned, since the footer min/max wouldn't reflect those files. + /// Returns `None` if the scan has no predicate, if neither a data-column stats schema nor a + /// partition schema is available, or if the predicate is a bare unsupported expression (e.g. + /// column-column comparison). Junctions represent unsupported arms with a NULL literal, + /// preserving three-valued logic while allowing independently decisive supported arms to + /// prune. fn build_actions_meta_predicate(&self) -> Option { let PhysicalPredicate::Some(ref predicate, _) = self.state_info.physical_predicate else { return None; }; - self.state_info.physical_stats_schema.as_ref()?; + // Skipping needs either data-column stats or partition values to rewrite against; a + // partition-only predicate has no `stats_parsed` schema, a data-only predicate on an + // unpartitioned table has no partition schema. + if self.state_info.physical_stats_schema.is_none() + && self.state_info.physical_partition_schema.is_none() + { + return None; + } - let partition_columns = self - .snapshot - .table_configuration() - .metadata() - .partition_columns(); - let skipping_pred = as_checkpoint_skipping_predicate(predicate, partition_columns)?; + // `partitionValues_parsed` is keyed by PHYSICAL partition name, and (under column mapping) + // the predicate also references physical columns, so partition detection reads the physical + // partition schema rather than the logical names in table metadata. + let mut partition_columns = HashSet::new(); + let mut floating_partition_columns = HashSet::new(); + if let Some(schema) = self.state_info.physical_partition_schema.as_ref() { + for field in schema.fields() { + if log_replay::partition_value_requires_timezone_reparse( + field.data_type(), + self.partition_values.timestamp_timezone.as_deref(), + ) { + continue; + } + let column = ColumnName::new([field.name()]); + if field.data_type() == &DataType::FLOAT || field.data_type() == &DataType::DOUBLE { + floating_partition_columns.insert(column.clone()); + } + partition_columns.insert(column); + } + } + let skipping_pred = as_checkpoint_skipping_predicate( + predicate, + &partition_columns, + &floating_partition_columns, + &self.state_info.physical_stats_columns, + )?; let mut prefixer = PrefixColumns { - prefix: ColumnName::new(["add", "stats_parsed"]), + prefix: column_name!("add"), }; - let prefixed = prefixer.transform_pred(&skipping_pred)?; + let prefixed = prefixer.transform_pred(&skipping_pred); Some(Arc::new(prefixed.into_owned())) } @@ -834,6 +1209,10 @@ impl Scan { /// checkpoint manifests sequentially. After exhausting this iterator, call `finish()` /// to determine if a distributed phase is needed. /// + /// Cancellation is not supported on this path: it errors if a token was set via + /// [`ScanBuilder::with_cancellation_token`], rather than silently running to completion. + /// Only [`scan_metadata`](Self::scan_metadata) honors the token today. + /// /// # Example /// /// ```no_run @@ -842,7 +1221,7 @@ impl Scan { /// # use delta_kernel::scan::{AfterSequentialScanMetadata, ParallelScanMetadata}; /// # use delta_kernel::Snapshot; /// # use url::Url; - /// # use delta_kernel::engine::default::DefaultEngineBuilder; + /// # use test_utils::delta_kernel_default_engine::DefaultEngineBuilder; /// # use delta_kernel::object_store::local::LocalFileSystem; /// # fn main() -> DeltaResult<()> { /// let engine = Arc::new(DefaultEngineBuilder::new(Arc::new(LocalFileSystem::new())).build()); @@ -888,11 +1267,19 @@ impl Scan { &self, engine: Arc, ) -> DeltaResult { + // Fail fast rather than silently ignore a caller-supplied token: the parallel path does + // not thread cancellation, so honoring a set token would require dropping it on the floor. + if self.cancellation_token.is_some() { + return Err(Error::unsupported( + "cancellation is not supported by parallel_scan_metadata; \ + use scan_metadata for a cancellable scan", + )); + } // For the sequential/parallel phase approach, we use a conservative checkpoint_info // since SequentialPhase reads checkpoints via CheckpointManifestReader which doesn't // currently support stats_parsed optimization. let checkpoint_read_schema = if self.skip_stats() { - CHECKPOINT_READ_SCHEMA_NO_STATS.clone() + CHECKPOINT_READ_SCHEMA_NO_JSON_STATS.clone() } else { CHECKPOINT_READ_SCHEMA.clone() }; @@ -905,23 +1292,39 @@ impl Scan { engine.as_ref(), self.state_info.clone(), checkpoint_info, - self.skip_stats(), + self.stats_options(), + self.partition_values_options(), )?; - let sequential = SequentialPhase::try_new(processor, self.snapshot.log_segment(), engine)?; + let sequential = + SequentialPhase::try_new(processor, self.snapshot.log_segment(), engine.clone())?; - Ok(SequentialScanMetadata::new(sequential)) + Ok(SequentialScanMetadata::new( + sequential, + self.correlation_id.clone(), + )) } /// Perform an "all in one" scan. This will use the provided `engine` to read and process all /// the data for the query. Each [`EngineData`] in the resultant iterator is a portion of the /// final table data. Generally connectors/engines will want to use [`Scan::scan_metadata`] so /// they can have more control over the execution of the scan. + /// + /// Returns an error if the scan was built with [`ScanBuilder::without_row_transforms`]; use + /// [`Scan::scan_metadata`] instead. // This calls [`Scan::scan_metadata`] to get an iterator of `ScanMetadata` actions for the scan, // and then uses the `engine`'s [`crate::ParquetHandler`] to read the actual table data. pub fn execute( &self, engine: Arc, ) -> DeltaResult>>> { + if self.state_info.skip_row_transforms { + return Err(Error::unsupported( + "Scan::execute is not supported when the scan was built with \ + without_row_transforms; use scan_metadata for listing and read data with your \ + own reader", + )); + } + fn scan_metadata_callback(batches: &mut Vec, file: state::ScanFile) { batches.push(file); } @@ -972,6 +1375,21 @@ impl Scan { None, )?; + let mut read_result_iter = read_result_iter.peekable(); + + // Only flag an empty iterator as a connector bug when stats are present and report + // a positive row count. When stats are absent we cannot distinguish a legitimate + // 0-row file from a buggy connector, so we conservatively allow it. + let expect_data = scan_file.stats.as_ref().is_some_and(|s| s.num_records > 0); + if expect_data && read_result_iter.peek().is_none() { + return Err(Error::internal_error(format!( + "ParquetHandler returned no data for file '{}'. This is likely a connector \ + bug -- the handler's read_parquet_files must return at least one batch for \ + each requested file that contains rows.", + scan_file.path + ))); + } + let engine = engine.clone(); // Arc clone let physical_schema_inner = physical_schema.clone(); let logical_schema_inner = logical_schema.clone(); @@ -992,10 +1410,9 @@ impl Scan { // to `rest` in a moment anyway let mut sv = selection_vector.take(); let rest = split_vector(sv.as_mut(), len, None); - let result = match sv { - Some(sv) => logical.and_then(|data| data.apply_selection_vector(sv)), - None => logical, - }; + let result = logical.fold_with(sv, |logical, sv| { + logical.and_then(|data| data.apply_selection_vector(sv)) + }); selection_vector = rest; result })) @@ -1008,7 +1425,10 @@ impl Scan { } } -/// Get the schema that scan rows (from [`Scan::scan_metadata`]) will be returned with. +/// Get the base schema that scan rows (from [`Scan::scan_metadata`]) will be returned with. +/// +/// This is the base shape; engines may add trailing `*_parsed` columns by opting in via +/// [`StatsOptions`] (`stats_parsed`) or [`PartitionValuesOptions`] (`partitionValues_parsed`). /// /// It is: /// ```ignored diff --git a/kernel/src/scan/scan_plan.rs b/kernel/src/scan/scan_plan.rs new file mode 100644 index 0000000000..6b75810d7b --- /dev/null +++ b/kernel/src/scan/scan_plan.rs @@ -0,0 +1,1050 @@ +//! Declarative metadata scan plans. +//! +//! [`Scan::build_metadata_scan_plan`] reconciles checkpoint and commit actions into live adds, +//! applying metadata pruning before newest-action-wins replay. + +use std::borrow::Cow; +use std::sync::{Arc, LazyLock}; + +use url::Url; + +use super::data_skipping::as_sql_data_skipping_predicate_with_stats_columns; +use super::state_info::StateInfo; +use super::{PhysicalPredicate, Scan}; +use crate::actions::deletion_vector::DeletionVectorDescriptor; +use crate::actions::{ + ADD_FIELD, ADD_NAME, ADD_SCHEMA, REMOVE_FIELD, SIDECAR_FIELD, SIDECAR_NAME, STATS_PARSED, +}; +use crate::checkpoint::{CheckpointShape, CheckpointType}; +use crate::expressions::{ + col, column_name, joined_column_expr, ColumnName, Expression as Expr, ExpressionRef, Predicate, +}; +use crate::plans::ir::nodes::{DynamicScan, FileType, ScanFile}; +use crate::plans::ir::plan::Plan; +use crate::scan::log_replay::{ + parsed_partition_values_expr, PARTITION_VALUES_PARSED_NAME, STATS_PARSED_NAME, +}; +use crate::schema::{ + lazy_schema_ref, schema, schema_ref, DataType, SchemaRef, SchemaStructPatchBuilder, + StructField, StructType, ToSchema as _, +}; +use crate::struct_patch::ProjectionStructPatchBuilder; +use crate::transforms::{transform_output_type, ExpressionTransform}; +use crate::utils::{CollectInto, FoldWithOption as _}; +use crate::{DeltaResult, Error, PlanBuilder}; + +// === Internal column names === + +// Both add and remove provide path + DV (storageType, pathOrInlineDv, offset) columns. We +// materialize them as one top-level `file_action_key` column that are used by the plan's +// aggregate and anti-join operators. +const FILE_ACTION_KEY: &str = "file_action_key"; +const STATS: &str = "stats"; +const PARTITION_VALUES: &str = "partitionValues"; +const PARTITION_VALUES_PARSED: &str = "partitionValues_parsed"; +// Generated partition pruning predicates reference this to retain removes. +const IS_ADD: &str = "is_add"; +const VERSION: &str = "version"; + +impl Scan { + /// Build the live-add metadata plan from checkpoint and commit actions. + /// + /// Returns `None` for an empty result or a statically false predicate. + pub(super) fn build_metadata_scan_plan( + &self, + shape: &CheckpointShape, + ) -> DeltaResult> { + let state = &self.state_info; + // A statically-unsatisfiable predicate (e.g. `x > 10 AND FALSE`) skips the whole table. + if state.physical_predicate == PhysicalPredicate::StaticSkipAll { + return Ok(None); + } + + let prune = stats_skipping_predicate(state); + let prune = prune.as_ref(); + + // The output `add` after reparsing `stats`/`partitionValues`: shared by the commit arm's + // dedup carrier and both terminal `{ add }` projections, so every arm agrees on the + // union schema. + let add_field = self.normalized_add_field()?; + let (output_expr, output_schema) = self.metadata_output_projection(&add_field)?; + + let commit_actions = self.commit_arm()?.try_fold_with(prune, |p, prune| { + // We filter so that: + // * All remove actions are kept + // * Add actions that do not match the partition pruning or stats predicate are removed. + // + // NOTE: It is important that add actions are filtered by the partition predicate + // because partition filtering may not be applied on data rows. On the other + // hand, failing to skip based on data columns is safe because the data + // predicate will also be evaluated on data rows. Thus it is crucial that we partition + // prune adds here. + // + // NOTE: It is not safe to prune remove actions using the partition filter. This is + // because a NULL result for `remove.partitionValues.partCol` may be due to + // `remove.partitionValues` being NULL, or it may be from `partCol` being + // NULL. Thus, we simply do not prune removes. + p.filter(Predicate::or(col!("add").is_null(), prune.clone())) + })?; + + let deduped_commit = commit_actions.aggregate_by([column_name!(FILE_ACTION_KEY)], |a| { + // Each group with a non-null FILE_ACTION_KEY contains the adds and removes for a given + // file; winning adds pass through unchanged while winning removes produce NULL. Non- + // file actions have NULL FILE_ACTION_KEY and map to their own NULL group. + a.max_non_null_by( + column_name!(ADD_NAME), + column_name!(FILE_ACTION_KEY), + column_name!(VERSION), + ) + })?; + + let checkpoint_adds = self + .checkpoint_arm(shape)? + .try_fold_with(prune, |p, prune| p.filter(prune.clone()))?; + + let checkpoint_live_adds = checkpoint_adds + .anti_join( + deduped_commit.clone(), + [column_name!(FILE_ACTION_KEY)], + [column_name!(FILE_ACTION_KEY)], + )? + .project(output_expr.clone(), output_schema.clone())?; + + let commit_live_adds = deduped_commit + .filter(col!("add").is_not_null())? + .project(output_expr, output_schema)?; + + PlanBuilder::union_all([commit_live_adds, checkpoint_live_adds])?.build_opt() + } + + /// Build normalized checkpoint adds. Returns an empty relation when no checkpoint exists. + /// + /// ## SQL equivalent: + // + /// SELECT STRUCT( + /// add.* EXCEPT ( + /// stats_parsed, partitionValues_parsed + /// ), + /// add.stats_parsed AS stats_parsed, + /// PARSED_PARTITION_VALUES( + /// add.partitionValues, + /// add.partitionValues_parsed, + /// physical_partitions, + /// reader_timezone + /// ) AS partitionValues_parsed + /// ) AS add, + /// version, add.path IS NOT NULL AS is_add, file_key(add) AS key + /// FROM checkpoint_actions + /// WHERE add.path IS NOT NULL + /// + /// When the checkpoint lacks native parsed stats, `FROM_JSON(add.stats, physical_stats)` + /// replaces `add.stats_parsed` above. Parsed partition values reuse the native struct when + /// possible and rebuild from the raw map when a reader timezone changes zoned timestamp + /// interpretation or no compatible native struct exists. A parsed field is omitted when its + /// schema is absent. + fn checkpoint_arm(&self, shape: &CheckpointShape) -> DeltaResult { + let log_segment = self.snapshot.log_segment(); + let physical_stats = self.state_info.physical_stats_schema.as_ref(); + let physical_partitions = self.state_info.physical_partition_schema.as_ref(); + let source_physical_stats = shape.parsed_stats_schema.as_ref(); + let source_physical_partitions = shape.parsed_partition_schema.as_ref(); + let checkpoint = log_segment.checkpoint_version_tagged_scan_files()?; + + let actions = match (&shape.checkpoint_type, checkpoint) { + (CheckpointType::Leaf, Some((FileType::Parquet, parts))) => { + let schema = + parquet_read_schema(source_physical_stats, source_physical_partitions)?; + PlanBuilder::scan_parquet(parts, &[VERSION], schema) + } + (CheckpointType::Leaf, Some((FileType::Json, parts))) => { + PlanBuilder::scan_json( + parts, + &[VERSION], + json_read_schema(/* include_remove */ false), + ) + } + (CheckpointType::Manifest, Some((file_type, parts))) => { + let schema = + parquet_read_schema(source_physical_stats, source_physical_partitions)?; + match log_segment.checkpoint_hint_version_tagged_sidecar_scan_files()? { + Some(sidecars) => PlanBuilder::scan_parquet(sidecars, &[VERSION], schema), + // Without a complete hint, load the sidecars referenced by the manifest. + None => sidecar_actions(file_type, parts, schema, &log_segment.log_root), + } + } + (CheckpointType::None, _) | (_, None) => { + PlanBuilder::values(json_read_schema(/* include_remove */ false), vec![]) + } + }?; + + actions + .filter(col!("add.path").is_not_null())? + .project_patch(|patch| { + patch + .with_parsed_add_stats(physical_stats) + .with_parsed_add_partition_values( + physical_partitions, + self.partition_values.timestamp_timezone.as_deref(), + ) + .append( + StructField::not_null(IS_ADD, DataType::BOOLEAN), + Expr::from(col!("add.path").is_not_null()), + ) + .append( + FILE_ACTION_KEY_FIELD.clone(), + file_action_key_expr(|col| joined_column_expr!("add", col)), + ) + }) + } + + /// Build the normalized commit JSON arm. + /// + /// ## SQL equivalent: + /// + /// SELECT STRUCT( + /// add.* EXCEPT (stats_parsed, partitionValues_parsed), + /// FROM_JSON(add.stats, physical_stats) AS stats_parsed, + /// MAP_TO_STRUCT(add.partitionValues, physical_partitions) AS partitionValues_parsed + /// ) AS add, + /// remove, version, add.path IS NOT NULL AS is_add, + /// file_key(COALESCE(add, remove)) AS key + /// FROM json_commits + /// WHERE add.path IS NOT NULL OR remove.path IS NOT NULL + /// + /// A parsed field is omitted when its schema is absent. + fn commit_arm(&self) -> DeltaResult { + let log_segment = self.snapshot.log_segment(); + let commit_files = log_segment.commit_cover_version_tagged_scan_files()?; + PlanBuilder::scan_json(commit_files, &[VERSION], json_read_schema(true))? + .filter(Predicate::or( + col!("add.path").is_not_null(), + col!("remove.path").is_not_null(), + ))? + .project_patch(|patch| { + // Commits never carry source-native parsed columns, so normalize from the raw + // encodings. + patch + .with_parsed_add_stats(self.state_info.physical_stats_schema.as_ref()) + .with_parsed_add_partition_values( + self.state_info.physical_partition_schema.as_ref(), + self.partition_values.timestamp_timezone.as_deref(), + ) + .append( + StructField::not_null(IS_ADD, DataType::BOOLEAN), + Expr::from(col!("add.path").is_not_null()), + ) + .append( + FILE_ACTION_KEY_FIELD.clone(), + file_action_key_expr(|col| { + Expr::coalesce([ + joined_column_expr!("add", col), + joined_column_expr!("remove", col), + ]) + }), + ) + }) + } + + fn normalized_add_field(&self) -> DeltaResult { + let physical_stats_schema = self.state_info.physical_stats_schema.as_ref(); + let physical_partition_schema = self.state_info.physical_partition_schema.as_ref(); + let patch = SchemaStructPatchBuilder::new() + .fold_with(physical_stats_schema, |patch, schema| { + patch.append(StructField::nullable(STATS_PARSED, schema.as_ref().clone())) + }) + .fold_with(physical_partition_schema, |patch, schema| { + patch.append(StructField::nullable( + PARTITION_VALUES_PARSED, + schema.as_ref().clone(), + )) + }); + Ok(StructField::nullable(ADD_NAME, patch.build(&ADD_SCHEMA)?)) + } + + /// Builds the output projection for requested stats and partition values. The base of this + /// transformation is constructed by [`Self::normalized_add_field`]. + /// + /// The output schema is: + /// ```text + /// add: struct< + /// path: string, + /// partitionValues: map, + /// size: long, + /// modificationTime: long, + /// dataChange: boolean, + /// stats: string, // when JSON stats are requested + /// tags: map, + /// deletionVector: struct<...>, + /// baseRowId: long, + /// defaultRowCommitVersion: long, + /// clusteringProvider: string, + /// stats_parsed: struct<...>, // when parsed stats are requested + /// partitionValues_parsed: struct<...>, // when parsed partition values are requested + /// > + /// ``` + /// Stats output may contain neither representation, JSON only, parsed only, or both. Parsed + /// partition values are selected independently and omitted for unpartitioned tables. Fields + /// needed only for pruning are omitted. + fn metadata_output_projection( + &self, + add_field: &StructField, + ) -> DeltaResult<(ExpressionRef, SchemaRef)> { + let input_schema = schema_ref! { (add_field.clone()) }; + let has_stats_parsed = input_schema.contains_col([ADD_NAME, STATS_PARSED_NAME]); + let projection = ProjectionStructPatchBuilder::new_nested(&input_schema, [ADD_NAME]); + + // JSON stats output. `StatsOptions` allows JSON only, parsed only, both, or neither. + let has_json_stats = input_schema.contains_col([ADD_NAME, STATS]); + let projection = match (self.stats.synthesize_json, has_json_stats) { + (true, true) | (false, false) => projection, + (true, false) => { + return Err(Error::internal_error( + "JSON stats were requested, but add.stats is missing from the metadata schema", + )); + } + (false, true) => projection.drop(STATS), + }; + + // Parsed stats output. + let projection = match (self.physical_stats_output_schema.as_ref(), has_stats_parsed) { + (Some(physical_stats), _) => projection.replace( + STATS_PARSED, + StructField::nullable(STATS_PARSED, physical_stats.as_ref().clone()), + project_nested_struct_to_schema([ADD_NAME, STATS_PARSED_NAME], physical_stats), + ), + (None, true) => projection.drop(STATS_PARSED), + (None, false) => projection, + }; + + // Parsed partition-values output. + let has_partition_values_parsed = + input_schema.contains_col([ADD_NAME, PARTITION_VALUES_PARSED_NAME]); + let physical_partitions = self + .partition_values + .parsed_struct + .then_some(self.state_info.physical_partition_schema.as_ref()) + .flatten(); + let projection = match (physical_partitions, has_partition_values_parsed) { + (Some(schema), true) => projection.replace( + PARTITION_VALUES_PARSED, + StructField::nullable(PARTITION_VALUES_PARSED, schema.as_ref().clone()), + project_nested_struct_to_schema([ADD_NAME, PARTITION_VALUES_PARSED_NAME], schema), + ), + (Some(_), false) => { + return Err(Error::internal_error( + "parsed partition values were requested, but add.partitionValues_parsed is \ + missing", + )); + } + (None, true) => projection.drop(PARTITION_VALUES_PARSED), + (None, false) => projection, + }; + + let (add_schema, add_expr) = projection.build()?; + let schema = schema_ref! { + (StructField::nullable(ADD_NAME, add_schema.as_ref().clone())) + }; + Ok((Arc::new(Expr::struct_from([add_expr])), schema)) + } +} + +/// Read actions from V2 checkpoint sidecars. +fn sidecar_actions( + file_type: FileType, + root_parts: Vec, + action_schema: SchemaRef, + log_root: &Url, +) -> DeltaResult { + const FILE_PATH: &str = "path"; + const FILE_SIZE: &str = "size"; + const FILE_MOD: &str = "filemod"; + const DV: &str = "dv"; + const SIDECAR_SIZE: &str = "sizeInBytes"; + const SIDECAR_FILE_MOD: &str = "modificationTime"; + + static SIDECAR_FILE_META_SCHEMA: LazyLock = lazy_schema_ref! { + not_null (FILE_PATH): STRING, + not_null (FILE_SIZE): LONG, + not_null (FILE_MOD): LONG, + nullable (DV): (DeletionVectorDescriptor::to_schema()), + nullable (VERSION): LONG, + }; + + static SIDECAR_READ_SCHEMA: LazyLock = lazy_schema_ref! { + (&SIDECAR_FIELD), + nullable (VERSION): LONG, + }; + + let scan = match file_type { + FileType::Json => PlanBuilder::scan_json, + FileType::Parquet => PlanBuilder::scan_parquet, + }; + let sidecar_files = scan(root_parts, &[VERSION], SIDECAR_READ_SCHEMA.clone())? + .filter(col!(SIDECAR_NAME, FILE_PATH).is_not_null())? + .project( + Expr::struct_from([ + col!(SIDECAR_NAME, FILE_PATH), + col!(SIDECAR_NAME, SIDECAR_SIZE), + col!(SIDECAR_NAME, SIDECAR_FILE_MOD), + Expr::null_literal(DeletionVectorDescriptor::to_schema().into()), + col!(VERSION), + ]), + SIDECAR_FILE_META_SCHEMA.clone(), + )?; + + let dynamic_scan = DynamicScan::try_new( + &SIDECAR_FILE_META_SCHEMA, + action_schema, + FileType::Parquet, + log_root.join("_sidecars/")?, + [VERSION], + column_name!(FILE_PATH), + column_name!(FILE_SIZE), + column_name!(FILE_MOD), + column_name!(DV), + )?; + + sidecar_files.dynamic_scan(dynamic_scan) +} + +// === Helpers === + +/// Read schema for JSON actions tagged with their log version. +/// Commits include removes; JSON checkpoint leaves do not. +fn json_read_schema(include_remove: bool) -> SchemaRef { + schema_ref! { + (&ADD_FIELD), + ..(include_remove.then_some(&REMOVE_FIELD)), + nullable (VERSION): LONG, + } +} + +/// Read schema for parquet add actions. +fn parquet_read_schema( + physical_stats: Option<&SchemaRef>, + physical_partitions: Option<&SchemaRef>, +) -> DeltaResult { + let add_patch = SchemaStructPatchBuilder::new() + .fold_with(physical_stats, |patch, schema| { + patch.append(StructField::nullable(STATS_PARSED, schema.as_ref().clone())) + }) + .fold_with(physical_partitions, |patch, schema| { + patch.append(StructField::nullable( + PARTITION_VALUES_PARSED, + schema.as_ref().clone(), + )) + }); + Ok(schema_ref! { + (StructField::nullable(ADD_NAME, add_patch.build(&ADD_SCHEMA)?)), + nullable (VERSION): LONG, + }) +} + +/// File identity used for replay. +static FILE_ACTION_KEY_FIELD: LazyLock = LazyLock::new(|| { + let schema = schema! { + nullable "path": STRING, + nullable "deletionVector": { + not_null "storageType": STRING, + not_null "pathOrInlineDv": STRING, + nullable "offset": INTEGER, + }, + }; + StructField::nullable(FILE_ACTION_KEY, schema) +}); + +/// Build a file identity from path and deletion vector. +fn file_action_key_expr(key_col_expr: impl Fn(ColumnName) -> Expr) -> Expr { + let storage_type = key_col_expr(column_name!("deletionVector.storageType")); + Expr::struct_from([ + key_col_expr(column_name!("path")), + Expr::struct_with_nullability_from( + [ + storage_type.clone(), + key_col_expr(column_name!("deletionVector.pathOrInlineDv")), + key_col_expr(column_name!("deletionVector.offset")), + ], + Expr::from_pred(storage_type.is_not_null()), + ), + ]) +} + +trait ProjectionStructPatchBuilderExt<'a> { + /// Parses add stats, preferring a compatible parsed field. + /// + /// When `physical_stats` is present, the input must contain either + /// `add.stats_parsed` or the fallback `add.stats` JSON field. + fn with_parsed_add_stats(self, physical_stats: Option<&SchemaRef>) -> Self; + + /// Parses add partition values, preferring a compatible parsed field. + fn with_parsed_add_partition_values( + self, + physical_partitions: Option<&SchemaRef>, + timestamp_timezone: Option<&str>, + ) -> Self; +} + +impl<'a> ProjectionStructPatchBuilderExt<'a> for ProjectionStructPatchBuilder<'a> { + fn with_parsed_add_stats(self, physical_stats: Option<&SchemaRef>) -> Self { + let has_stats_parsed = self + .input_schema() + .contains_col([ADD_NAME, STATS_PARSED_NAME]); + let add = [ADD_NAME]; + match physical_stats { + Some(schema) => { + let field = StructField::nullable(STATS_PARSED, schema.as_ref().clone()); + let expr = Expr::parse_json(col!("add.stats"), Arc::clone(schema)); + if has_stats_parsed { + self + } else { + self.append_at(add, field, expr) + } + } + None => self, + } + } + + fn with_parsed_add_partition_values( + self, + physical_partitions: Option<&SchemaRef>, + timestamp_timezone: Option<&str>, + ) -> Self { + let has_partition_values_parsed = self + .input_schema() + .contains_col([ADD_NAME, PARTITION_VALUES_PARSED_NAME]); + let add = [ADD_NAME]; + match physical_partitions { + Some(schema) => { + let field = StructField::nullable(PARTITION_VALUES_PARSED, schema.as_ref().clone()); + let base_partition_values = has_partition_values_parsed + .then(|| column_name!(ADD_NAME, PARTITION_VALUES_PARSED)); + let expr = parsed_partition_values_expr( + col!(ADD_NAME, PARTITION_VALUES), + base_partition_values, + schema, + timestamp_timezone, + ); + if has_partition_values_parsed { + self.replace_at(add, PARTITION_VALUES_PARSED, field, expr) + } else { + self.append_at(add, field, expr) + } + } + None => self, + } + } +} + +/// Rebuilds `root` to match a narrowed schema while preserving a null parent struct. A direct +/// column reference would retain fields not requested by the caller. +fn project_nested_struct_to_schema( + root: impl CollectInto, + schema: &StructType, +) -> Expr { + let root = root.collect_into(); + let fields = schema.fields().map(|field| { + let column = root.join(&ColumnName::new([field.name()])); + match field.data_type() { + DataType::Struct(schema) => project_nested_struct_to_schema(column, schema), + _ => Expr::from(column), + } + }); + Expr::struct_with_nullability_from( + fields, + Expr::from_pred(Expr::from(root.clone()).is_not_null()), + ) +} + +/// Build the metadata pruning predicate, or `None` when no pruning is possible. +fn stats_skipping_predicate(state: &StateInfo) -> Option { + /// Re-roots metadata columns under `add`. + struct MetadataSkippingColumnPrefixer; + + impl<'a> ExpressionTransform<'a> for MetadataSkippingColumnPrefixer { + transform_output_type!(|'a, T| Cow<'a, T>); + + fn transform_expr_column(&mut self, name: &'a ColumnName) -> Cow<'a, ColumnName> { + let path = name.path(); + let replacement_root = match path.first().map(String::as_str) { + Some(STATS_PARSED) => [ADD_NAME, STATS_PARSED], + Some(PARTITION_VALUES_PARSED) => [ADD_NAME, PARTITION_VALUES_PARSED], + _ => return Cow::Borrowed(name), + }; + Cow::Owned(ColumnName::new( + replacement_root + .into_iter() + .map(str::to_string) + .chain(path.iter().skip(1).cloned()), + )) + } + } + + let PhysicalPredicate::Some(pred, _) = &state.physical_predicate else { + return None; + }; + let partition_column_names = state + .physical_partition_schema + .iter() + .flat_map(|s| s.fields().map(|f| ColumnName::new([f.name()]))) + .collect(); + let skipping = as_sql_data_skipping_predicate_with_stats_columns( + pred, + &partition_column_names, + &state.physical_stats_columns, + )?; + // A null skipping verdict means the available metadata cannot prove the file is skippable. + let skipping = Predicate::distinct(skipping, Expr::literal(false)); + let mut prefixer = MetadataSkippingColumnPrefixer; + Some(prefixer.transform_pred(&skipping).into_owned()) +} + +#[cfg(test)] +#[path = "scan_plan/tests.rs"] +mod execution_tests; + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + + use super::*; + use crate::actions::{Metadata, Protocol}; + use crate::arrow::array::{StringArray, StructArray}; + use crate::engine::arrow_data::EngineDataArrowExt as _; + use crate::engine::sync::SyncEngine; + use crate::expressions::lit; + use crate::log_segment::LogSegment; + use crate::log_segment_files::LogSegmentFiles; + use crate::object_store::memory::InMemory; + use crate::object_store::path::Path; + use crate::object_store::ObjectStoreExt as _; + use crate::plans::ir::nodes::Operator; + use crate::plans::Operation as PlanOperation; + use crate::scan::{PartitionValuesOptions, StatsOptions}; + use crate::schema::StructType; + use crate::snapshot::Snapshot; + use crate::table_configuration::TableConfiguration; + use crate::unit_test_utils::create_log_path; + use crate::Engine as _; + + fn mock_snapshot(log_segment: LogSegment) -> DeltaResult> { + let metadata = Metadata::try_new( + None, + None, + partitioned_schema(), + vec!["p".to_string()], + 0, + HashMap::new(), + )?; + let table_configuration = TableConfiguration::try_new( + metadata, + Protocol::try_new_legacy(2, 5)?, + Url::parse("memory:///")?, + 0, + )?; + Ok(Arc::new(Snapshot::new(log_segment, table_configuration)?)) + } + + fn partitioned_schema() -> SchemaRef { + Arc::new(StructType::new_unchecked([ + StructField::nullable("x", DataType::LONG), + StructField::nullable("p", DataType::STRING), + ])) + } + + fn log_root() -> Url { + Url::parse("file:///_delta_log/").unwrap() + } + + fn log_segment(log_root: Url, commits: &[&str], checkpoint: Option<&str>) -> LogSegment { + let ascending_commit_files: Vec<_> = + commits.iter().map(|path| create_log_path(path)).collect(); + let checkpoint_parts: Vec<_> = checkpoint.into_iter().map(create_log_path).collect(); + let checkpoint_version = checkpoint_parts.first().map(|path| path.version); + let latest_commit_file = ascending_commit_files.last().cloned(); + let end_version = latest_commit_file + .as_ref() + .map(|path| path.version) + .or(checkpoint_version) + .unwrap_or_default(); + LogSegment { + end_version, + checkpoint_version, + log_root, + listed: LogSegmentFiles { + ascending_commit_files, + checkpoint_parts, + latest_commit_file, + max_published_version: Some(end_version), + ..Default::default() + }, + last_checkpoint_metadata: None, + } + } + + fn checkpoint_path(file_type: FileType) -> &'static str { + match file_type { + FileType::Json => concat!( + "file:///_delta_log/00000000000000000000.checkpoint.", + "11111111-1111-1111-1111-111111111111.json" + ), + FileType::Parquet => "file:///_delta_log/00000000000000000000.checkpoint.parquet", + } + } + + fn shape(checkpoint_type: CheckpointType, parsed_stats: Option) -> CheckpointShape { + CheckpointShape { + checkpoint_type, + parsed_stats_schema: parsed_stats, + parsed_partition_schema: None, + } + } + + fn no_checkpoint() -> CheckpointShape { + shape(CheckpointType::None, None) + } + + fn tags(plan: &Plan) -> Vec { + plan.nodes.iter().map(|node| node.op.to_string()).collect() + } + + fn add_struct(schema: &SchemaRef) -> &StructType { + let DataType::Struct(add_struct) = schema + .field(ADD_NAME) + .expect("schema should contain add") + .data_type() + else { + panic!("add should be a struct"); + }; + add_struct + } + + // One add with JSON stats and no `stats_parsed`. + fn write_parquet_checkpoint(store: &Arc, path: &str) -> DeltaResult<()> { + use crate::arrow::array::builder::{MapBuilder, MapFieldNames, StringBuilder}; + use crate::arrow::array::{ + Array, BooleanArray, Int64Array, RecordBatch, StringArray as SA, + }; + use crate::arrow::datatypes::{DataType as ADT, Field, Fields, Schema}; + use crate::parquet::arrow::arrow_writer::ArrowWriter; + + // An empty (non-null) `partitionValues` map for the single row; the canonical add schema + // requires the field, so the checkpoint file must physically carry it. The inner field + // names must match kernel's map convention (`key_value` / `key` / `value`). + let map_names = MapFieldNames { + entry: "key_value".to_string(), + key: "key".to_string(), + value: "value".to_string(), + }; + let mut map = MapBuilder::new(Some(map_names), StringBuilder::new(), StringBuilder::new()); + map.append(true).unwrap(); + let partition_values = map.finish(); + + // The reader null-fills missing *nullable* add fields, but the canonical add schema's + // non-null scalars (`path`, `size`, `modificationTime`, `dataChange`) and `partitionValues` + // must be present in the file. `stats` carries the JSON string parsed in the + // no-parsed-stats path; there is deliberately no `stats_parsed` column. + let add_fields = Fields::from(vec![ + Field::new("path", ADT::Utf8, true), + Field::new("stats", ADT::Utf8, true), + Field::new( + "partitionValues", + partition_values.data_type().clone(), + true, + ), + Field::new("size", ADT::Int64, true), + Field::new("modificationTime", ADT::Int64, true), + Field::new("dataChange", ADT::Boolean, true), + ]); + let schema = Arc::new(Schema::new(vec![ + Field::new(ADD_NAME, ADT::Struct(add_fields.clone()), true), + Field::new(VERSION, ADT::Int64, true), + ])); + let add = StructArray::new( + add_fields, + vec![ + Arc::new(SA::from(vec!["c.parquet"])), + Arc::new(SA::from(vec![ + r#"{"numRecords":1,"minValues":{"x":10},"maxValues":{"x":10}}"#, + ])), + Arc::new(partition_values), + Arc::new(Int64Array::from(vec![1i64])), + Arc::new(Int64Array::from(vec![1i64])), + Arc::new(BooleanArray::from(vec![true])), + ], + None, + ); + let batch = RecordBatch::try_new( + schema.clone(), + vec![Arc::new(add), Arc::new(Int64Array::from(vec![0i64]))], + )?; + + let mut buf = Vec::new(); + let mut writer = ArrowWriter::try_new(&mut buf, schema, None)?; + writer.write(&batch)?; + writer.close()?; + futures::executor::block_on(store.put(&Path::from(path), buf.into()))?; + Ok(()) + } + + fn struct_stats_schema() -> SchemaRef { + let segment = log_segment(log_root(), &[], None); + mock_snapshot(segment) + .unwrap() + .scan_builder() + .with_predicate(Arc::new(col!("x").gt(lit(5i64)))) + .with_stats(StatsOptions::all()) + .build() + .unwrap() + .state_info + .physical_stats_schema + .clone() + .expect("stats schema") + } + + // Commit-arm operator sequence without optional pruning. + const COMMIT_ARM_TAGS: &[&str] = &[ + "scan_json", // commits + "filter", // keep file actions + "project", // normalize + "aggregate", // newest-action-per-key + "filter", // live commit adds + "project", // extract add + ]; + + #[rstest::rstest] + #[case::leaf_parquet(shape(CheckpointType::Leaf, None), FileType::Parquet, + vec!["scan_parquet", "filter", "project", "semi_join", "project"])] + #[case::leaf_json(shape(CheckpointType::Leaf, None), FileType::Json, + vec!["scan_json", "filter", "project", "semi_join", "project"])] + #[case::manifest(shape(CheckpointType::Manifest, None), FileType::Parquet, + vec!["scan_parquet", "filter", "project", "dynamic_scan", "filter", "project", "semi_join", "project"])] + fn metadata_plan_checkpoint_arm_shape( + #[case] shape: CheckpointShape, + #[case] file_type: FileType, + #[case] checkpoint_arm_tags: Vec<&'static str>, + ) -> DeltaResult<()> { + let segment = log_segment( + log_root(), + &["file:///_delta_log/00000000000000000001.json"], + Some(checkpoint_path(file_type)), + ); + let scan = mock_snapshot(segment)?.scan_builder().build()?; + let plan = scan.build_metadata_scan_plan(&shape)?.expect("non-empty"); + + let mut expected: Vec<&str> = COMMIT_ARM_TAGS.to_vec(); + expected.extend(checkpoint_arm_tags); + expected.push("union_all"); // terminal + assert_eq!(tags(&plan), expected); + Ok(()) + } + + #[rstest::rstest] + #[case::with_parsed_stats(Some(struct_stats_schema()), true)] + #[case::without_parsed_stats(None, false)] + fn metadata_plan_manifest_sidecar_dynamic_scan_stats_columns( + #[case] parsed_stats: Option, + #[case] expect_parsed_columns: bool, + ) -> DeltaResult<()> { + let stats = StatsOptions::all(); + let partition_values = PartitionValuesOptions::with_struct(); + let segment = log_segment(log_root(), &[], Some(checkpoint_path(FileType::Parquet))); + let scan = mock_snapshot(segment)? + .scan_builder() + .with_stats(stats) + .with_partition_values(partition_values) + .build()?; + let checkpoint_shape = CheckpointShape { + checkpoint_type: CheckpointType::Manifest, + parsed_stats_schema: parsed_stats, + parsed_partition_schema: scan.state_info.physical_partition_schema.clone(), + }; + let plan = scan + .build_metadata_scan_plan(&checkpoint_shape)? + .expect("non-empty"); + + let dynamic_scan = plan + .nodes + .iter() + .find_map(|n| match &n.op { + Operator::DynamicScan(dynamic_scan) => Some(dynamic_scan), + _ => None, + }) + .expect("sidecar dynamic scan"); + assert_eq!( + add_struct(&dynamic_scan.schema) + .field(STATS_PARSED) + .is_some(), + expect_parsed_columns, + ); + assert!( + add_struct(&dynamic_scan.schema) + .field(PARTITION_VALUES_PARSED) + .is_some(), + "compatible native parsed partition values must be read from sidecars" + ); + Ok(()) + } + + #[test] + fn metadata_plan_commits_only() -> DeltaResult<()> { + let segment = log_segment( + log_root(), + &["file:///_delta_log/00000000000000000001.json"], + None, + ); + let scan = mock_snapshot(segment)?.scan_builder().build()?; + let plan = scan + .build_metadata_scan_plan(&no_checkpoint())? + .expect("non-empty"); + assert_eq!(tags(&plan), COMMIT_ARM_TAGS.to_vec()); + Ok(()) + } + + #[rstest::rstest] + #[case::leaf_parquet(shape(CheckpointType::Leaf, None), FileType::Parquet, + vec!["scan_parquet", "filter", "project", "project"])] + #[case::manifest(shape(CheckpointType::Manifest, None), FileType::Parquet, + vec!["scan_parquet", "filter", "project", "dynamic_scan", "filter", "project", "project"])] + fn metadata_plan_checkpoint_only( + #[case] shape: CheckpointShape, + #[case] file_type: FileType, + #[case] checkpoint_arm_tags: Vec<&'static str>, + ) -> DeltaResult<()> { + let segment = log_segment(log_root(), &[], Some(checkpoint_path(file_type))); + let scan = mock_snapshot(segment)?.scan_builder().build()?; + let plan = scan.build_metadata_scan_plan(&shape)?.expect("non-empty"); + assert_eq!(tags(&plan), checkpoint_arm_tags); + Ok(()) + } + + #[test] + fn metadata_plan_empty_is_none() -> DeltaResult<()> { + let segment = log_segment(log_root(), &[], None); + let scan = mock_snapshot(segment)?.scan_builder().build()?; + assert!(scan.build_metadata_scan_plan(&no_checkpoint())?.is_none()); + Ok(()) + } + + #[test] + fn metadata_plan_static_skip_all_is_none() -> DeltaResult<()> { + let segment = log_segment(log_root(), &[], None); + let scan = mock_snapshot(segment)? + .scan_builder() + .with_predicate(Arc::new(Predicate::literal(false))) + .build()?; + assert_eq!( + scan.state_info.physical_predicate, + PhysicalPredicate::StaticSkipAll + ); + assert!(scan + .build_metadata_scan_plan(&shape(CheckpointType::Leaf, None))? + .is_none()); + Ok(()) + } + + #[test] + fn metadata_plan_executes_commit_dedup_with_sync_executor() -> DeltaResult<()> { + let store = Arc::new(InMemory::new()); + futures::executor::block_on(async { + store + .put( + &Path::from("_delta_log/00000000000000000000.json"), + r#"{"add":{"path":"a.parquet","size":1,"modificationTime":1,"dataChange":true,"partitionValues":{}}} +{"add":{"path":"b.parquet","size":1,"modificationTime":1,"dataChange":true,"partitionValues":{}}} +"# + .into(), + ) + .await?; + store + .put( + &Path::from("_delta_log/00000000000000000001.json"), + r#"{"remove":{"path":"a.parquet","deletionTimestamp":2,"dataChange":true}} +"# + .into(), + ) + .await?; + DeltaResult::<()>::Ok(()) + })?; + + let segment = log_segment( + Url::parse("memory:///_delta_log/").unwrap(), + &[ + "memory:///_delta_log/00000000000000000000.json", + "memory:///_delta_log/00000000000000000001.json", + ], + None, + ); + let scan = mock_snapshot(segment)?.scan_builder().build()?; + let plan = scan + .build_metadata_scan_plan(&no_checkpoint())? + .expect("non-empty"); + + let engine = SyncEngine::new_with_store(store); + let mut batches = engine + .plan_executor() + .unwrap() + .execute_op(PlanOperation::QueryPlan(plan))? + .into_data()?; + let batch = batches + .next() + .expect("one batch")? + .try_into_record_batch()?; + assert!(batches.next().is_none()); + assert_eq!(batch.num_rows(), 1); + + let add = batch + .column_by_name(ADD_NAME) + .expect("add column") + .as_any() + .downcast_ref::() + .expect("add struct"); + let paths = add + .column_by_name("path") + .expect("add.path") + .as_any() + .downcast_ref::() + .expect("path string"); + assert_eq!(paths.value(0), "b.parquet"); + Ok(()) + } + + #[rstest::rstest] + #[case::keeps_matching_file(5, 1)] + #[case::prunes_non_matching_file(20, 0)] + fn metadata_plan_executes_leaf_without_stats_parsed( + #[case] lower_bound: i64, + #[case] expected_rows: usize, + ) -> DeltaResult<()> { + let store = Arc::new(InMemory::new()); + // A single-row parquet checkpoint carrying an `add` with a JSON `stats` string but no + // `stats_parsed` column. + write_parquet_checkpoint(&store, "_delta_log/00000000000000000000.checkpoint.parquet")?; + + let segment = log_segment( + Url::parse("memory:///_delta_log/").unwrap(), + &[], + Some("memory:///_delta_log/00000000000000000000.checkpoint.parquet"), + ); + let scan = mock_snapshot(segment)? + .scan_builder() + .with_stats(StatsOptions::all()) + .with_predicate(Arc::new(col!("x").gt(lit(lower_bound)))) + .build()?; + let plan = scan + // Leaf with no compatible parsed stats -> parse add.stats instead. + .build_metadata_scan_plan(&shape(CheckpointType::Leaf, None))? + .expect("non-empty"); + + let engine = SyncEngine::new_with_store(store); + let mut batches = engine + .plan_executor() + .unwrap() + .execute_op(PlanOperation::QueryPlan(plan))? + .into_data()?; + let actual_rows = batches.try_fold(0, |rows, batch| { + Ok::<_, crate::Error>(rows + batch?.try_into_record_batch()?.num_rows()) + })?; + assert_eq!(actual_rows, expected_rows); + Ok(()) + } +} diff --git a/kernel/src/scan/scan_plan/tests.rs b/kernel/src/scan/scan_plan/tests.rs new file mode 100644 index 0000000000..ad78c63701 --- /dev/null +++ b/kernel/src/scan/scan_plan/tests.rs @@ -0,0 +1,1239 @@ +use std::path::Path; +use std::sync::Arc; + +use ::test_utils::add_commit; +use ::test_utils::table_builder::{ + checkpoint_json_stats, checkpoint_struct_stats, no_checkpoint_stats, DataLayoutConfig, + FeatureSet, LogState, TableConfig, TestTableBuilder, +}; +use rstest::rstest; +use url::Url; + +use super::*; +use crate::arrow::array::{Array, ArrayRef, BooleanArray, StringArray, StructArray}; +use crate::arrow::compute::filter_record_batch; +use crate::arrow::datatypes::DataType as ArrowDataType; +use crate::arrow::record_batch::RecordBatch; +use crate::arrow::util::pretty::pretty_format_batches; +use crate::engine::arrow_data::EngineDataArrowExt as _; +use crate::engine::sync::SyncEngine; +use crate::engine::test_delegating::DelegatingEngine; +use crate::expressions::{col, column_name, lit, Predicate as Pred, Scalar}; +use crate::object_store::local::LocalFileSystem; +use crate::object_store::DynObjectStore; +use crate::plans::ir::nodes::Operator; +use crate::plans::Operation as PlanOperation; +use crate::scan::{PartitionValuesOptions, Scan, StatsOptions, StructStats}; +use crate::unit_test_utils::load_test_table; +use crate::{DeltaResult, Engine, PredicateRef, Snapshot}; + +// Normalizes metadata for comparison: the imperative path splits fields between the data batch +// and fileConstantValues, while the declarative path returns them in an add struct. +fn normalized_metadata_batch( + field: impl Fn(&str) -> ArrayRef, + json_stats: Option, + stats_parsed: Option, + partitions_parsed: Option, +) -> DeltaResult { + let mut columns = vec![ + ("path", field("path")), + ("size", field("size")), + ("modificationTime", field("modificationTime")), + ]; + if let Some(stats) = json_stats { + columns.push(("stats", stats)); + } + columns.extend([ + ("partitionValues", field("partitionValues")), + ("deletionVector", field("deletionVector")), + ("baseRowId", field("baseRowId")), + ("defaultRowCommitVersion", field("defaultRowCommitVersion")), + ("tags", field("tags")), + ("clusteringProvider", field("clusteringProvider")), + ]); + if let Some(stats) = stats_parsed { + columns.push(("stats_parsed", stats)); + } + if let Some(partitions) = partitions_parsed { + columns.push(("partitionValues_parsed", partitions)); + } + Ok(RecordBatch::try_from_iter(columns)?) +} + +fn imperative_metadata(scan: Scan, engine: &dyn Engine) -> DeltaResult> { + let mut batches = vec![]; + for metadata in scan.scan_metadata(engine)? { + let (data, selection) = metadata?.scan_files.into_parts(); + let batch = filter_record_batch( + &data.try_into_record_batch()?, + &BooleanArray::from(selection), + )?; + if batch.num_rows() == 0 { + continue; + } + let constants = batch + .column_by_name("fileConstantValues") + .expect("file constants") + .as_any() + .downcast_ref::() + .expect("file constants struct"); + batches.push(normalized_metadata_batch( + |name| { + batch + .column_by_name(name) + .or_else(|| constants.column_by_name(name)) + .unwrap_or_else(|| panic!("metadata field {name}")) + .clone() + }, + batch + .column_by_name(STATS) + .or_else(|| constants.column_by_name(STATS)) + .cloned(), + batch.column_by_name("stats_parsed").cloned(), + batch.column_by_name("partitionValues_parsed").cloned(), + )?); + } + Ok(batches) +} + +fn declarative_metadata(scan: &Scan, engine: &dyn Engine) -> DeltaResult> { + let Some(plan) = scan.declarative_metadata_scan_plan(engine)? else { + return Ok(vec![]); + }; + let batches = engine + .plan_executor() + .unwrap() + .execute_op(PlanOperation::QueryPlan(plan))? + .into_data()?; + + let mut projected = vec![]; + for batch in batches { + let batch = batch?.try_into_record_batch()?; + if batch.num_rows() == 0 { + continue; + } + let add = batch + .column_by_name(ADD_NAME) + .expect("add column") + .as_any() + .downcast_ref::() + .expect("add struct"); + projected.push(normalized_metadata_batch( + |name| { + add.column_by_name(name) + .unwrap_or_else(|| panic!("add.{name}")) + .clone() + }, + add.column_by_name(STATS).cloned(), + add.column_by_name(STATS_PARSED).cloned(), + add.column_by_name(PARTITION_VALUES_PARSED).cloned(), + )?); + } + Ok(projected) +} + +fn metadata_row_count(batches: &[RecordBatch]) -> usize { + batches.iter().map(RecordBatch::num_rows).sum() +} + +fn assert_metadata_eq( + actual: &[RecordBatch], + expected: &[RecordBatch], + context: &str, +) -> DeltaResult<()> { + fn sorted_pretty_lines(batches: &[RecordBatch]) -> DeltaResult> { + let formatted = pretty_format_batches(batches)?.to_string(); + let mut lines: Vec<_> = formatted.lines().map(str::to_string).collect(); + let len = lines.len(); + if len > 3 { + lines[2..len - 1].sort_unstable(); + } + Ok(lines) + } + + if let (Some(actual), Some(expected)) = (actual.first(), expected.first()) { + assert_eq!(actual.schema(), expected.schema(), "{context}"); + } + let actual = sorted_pretty_lines(actual)?; + let expected = sorted_pretty_lines(expected)?; + assert_eq!(actual, expected, "{context}"); + Ok(()) +} + +fn without_columns(batches: &[RecordBatch], excluded: &[&str]) -> DeltaResult> { + batches + .iter() + .map(|batch| { + RecordBatch::try_from_iter( + batch + .schema() + .fields() + .iter() + .zip(batch.columns()) + .filter(|(field, _)| !excluded.contains(&field.name().as_str())) + .map(|(field, column)| (field.name(), column.clone())), + ) + .map_err(Into::into) + }) + .collect() +} + +fn leaf_paths(batches: &[RecordBatch]) -> Vec { + fn append(field: &crate::arrow::datatypes::Field, prefix: &str, paths: &mut Vec) { + let path = if prefix.is_empty() { + field.name().to_string() + } else { + format!("{prefix}.{}", field.name()) + }; + if let ArrowDataType::Struct(fields) = field.data_type() { + for field in fields { + append(field, &path, paths); + } + } else { + paths.push(path); + } + } + + let mut paths = vec![]; + let schema = batches.first().expect("metadata output").schema(); + for field in schema.fields() { + append(field, "", &mut paths); + } + paths.sort_unstable(); + paths +} + +#[rstest] +#[case::v2_parquet_manifest("v2-checkpoints-parquet-with-sidecars")] +#[case::v2_json_manifest("v2-checkpoints-json-with-sidecars")] +#[case::v2_parquet_leaf("v2-checkpoints-parquet-without-sidecars")] +#[case::v2_json_leaf("v2-checkpoints-json-without-sidecars")] +#[case::v1_single_part_struct_stats("v1-single-part-struct-stats-only")] +#[case::v1_multi_part_struct_stats("v1-multi-part-struct-stats-only")] +#[case::v1_multi_part_partitioned_struct_stats("v1-multi-part-partitioned-struct-stats-only")] +fn declarative_metadata_matches_imperative_scan( + #[case] table: &str, + #[values( + None, + Some(col!("id").gt(lit(3i64))), + Some(col!("id").eq(lit(2i64))), + Some(col!("id").le(lit(0i64))), + Some(col!("id").is_not_null()) + )] + predicate: Option, +) -> DeltaResult<()> { + let (engine, snapshot, _tempdir) = crate::unit_test_utils::load_test_table(table)?; + let predicate = predicate.map(Arc::new); + + let imperative_builder = snapshot + .clone() + .scan_builder() + .with_stats(StatsOptions::all()) + .with_partition_values(PartitionValuesOptions::with_struct()); + let imperative_builder = match &predicate { + Some(predicate) => imperative_builder.with_predicate(predicate.clone()), + None => imperative_builder, + }; + let expected = imperative_metadata(imperative_builder.build()?, engine.as_ref())?; + + let declarative_builder = snapshot + .scan_builder() + .with_stats(StatsOptions::all()) + .with_partition_values(PartitionValuesOptions::with_struct()); + let declarative_builder = match predicate { + Some(predicate) => declarative_builder.with_predicate(predicate), + None => declarative_builder, + }; + let scan = declarative_builder.build()?; + let actual = declarative_metadata(&scan, engine.as_ref())?; + + if table.contains("struct-stats-only") { + assert_metadata_eq( + &without_columns(&actual, &[STATS])?, + &without_columns(&expected, &[STATS])?, + &format!("table {table}"), + ) + } else { + assert_metadata_eq(&actual, &expected, &format!("table {table}")) + } +} + +#[rstest] +#[case::json_commit(LogState::with_latest_version(1), FeatureSet::new())] +#[case::v1_checkpoint( + LogState::with_latest_version(1).with_checkpoint_at([1]), + FeatureSet::new() +)] +#[case::v2_sidecar( + LogState::with_latest_version(1) + .with_checkpoint_at([1]) + .with_sidecars_if_enabled(None), + FeatureSet::new().v2_checkpoint() +)] +#[case::column_mapping( + LogState::with_latest_version(1).with_checkpoint_at([1]), + FeatureSet::new().column_mapping("id") +)] +fn declarative_metadata_matches_imperative_with_reader_timezone( + #[case] log_state: LogState, + #[case] features: FeatureSet, +) -> DeltaResult<()> { + let table = TestTableBuilder::new() + .with_log_state(log_state) + .with_features(features) + .with_table_config(TableConfig::new().write_stats_as_struct(true)) + .with_data_layout(DataLayoutConfig::PartitionedAllTypes) + .build() + .expect("build reader-timezone parity table"); + let engine = SyncEngine::new_with_store(table.store().clone()); + let snapshot = Snapshot::builder_for(table.table_root()).build(&engine)?; + let partition_values = + || PartitionValuesOptions::with_struct().with_timestamp_timezone("America/Los_Angeles"); + let predicate: PredicateRef = Pred::gt(col!("part_ts"), Scalar::Timestamp(0)).into(); + + let expected = imperative_metadata( + snapshot + .clone() + .scan_builder() + .with_partition_values(partition_values()) + .with_predicate(predicate.clone()) + .build()?, + &engine, + )?; + let scan = snapshot + .scan_builder() + .with_partition_values(partition_values()) + .with_predicate(predicate) + .build()?; + let actual = declarative_metadata(&scan, &engine)?; + + assert_metadata_eq(&actual, &expected, "reader-timezone metadata parity") +} + +fn write_timezone_partition_table(table_path: &Path) -> Url { + std::fs::create_dir_all(table_path).unwrap(); + let url = Url::from_directory_path(table_path).unwrap(); + let table_root = url.to_string(); + let store: Arc = Arc::new(LocalFileSystem::new()); + let schema_string = serde_json::json!({ + "type": "struct", + "fields": [ + {"name": "p_ts", "type": "timestamp", "nullable": true, "metadata": {}}, + {"name": "value", "type": "integer", "nullable": true, "metadata": {}}, + ], + }) + .to_string(); + let protocol = r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#; + let metadata = serde_json::json!({ + "metaData": { + "id": "00000000-0000-0000-0000-000000000000", + "format": {"provider": "parquet", "options": {}}, + "schemaString": schema_string, + "partitionColumns": ["p_ts"], + "configuration": {"delta.checkpoint.writeStatsAsStruct": "true"}, + "createdTime": 1700000000000_i64, + }, + }) + .to_string(); + futures::executor::block_on(add_commit( + &table_root, + store.as_ref(), + 0, + format!("{protocol}\n{metadata}"), + )) + .unwrap(); + let add = serde_json::json!({ + "add": { + "path": "part.parquet", + "partitionValues": {"p_ts": "2024-01-15 12:30:45"}, + "size": 100, + "modificationTime": 1700000000000_i64, + "dataChange": true, + "stats": "{\"numRecords\":1}", + }, + }); + futures::executor::block_on(add_commit(&table_root, store.as_ref(), 1, add.to_string())) + .unwrap(); + url +} + +#[rstest] +fn reader_timezone_controls_partition_pruning_across_metadata_paths( + #[values(false, true)] native_checkpoint: bool, +) -> DeltaResult<()> { + let temp_dir = tempfile::tempdir().unwrap(); + let url = write_timezone_partition_table(temp_dir.path()); + let engine = SyncEngine::new(); + if native_checkpoint { + Snapshot::builder_for(url.clone()) + .build(&engine)? + .checkpoint(&engine, None)?; + } + let snapshot = Snapshot::builder_for(url).build(&engine)?; + let partition_values = + || PartitionValuesOptions::with_struct().with_timestamp_timezone("America/Los_Angeles"); + let predicate: PredicateRef = + Pred::gt(col!("p_ts"), Scalar::Timestamp(1_705_334_400_000_000)).into(); + + let expected = imperative_metadata( + snapshot + .clone() + .scan_builder() + .with_partition_values(partition_values()) + .with_predicate(predicate.clone()) + .build()?, + &engine, + )?; + assert_eq!( + metadata_row_count(&expected), + 1, + "imperative pruning must use the configured reader timezone" + ); + let scan = snapshot + .scan_builder() + .with_partition_values(partition_values()) + .with_predicate(predicate) + .build()?; + let actual = declarative_metadata(&scan, &engine)?; + assert_eq!( + metadata_row_count(&actual), + 1, + "declarative pruning must use the configured reader timezone" + ); + + assert_metadata_eq(&actual, &expected, "reader-timezone metadata parity") +} + +#[rstest] +#[case::parquet_manifest("v2-checkpoints-parquet-with-sidecars")] +#[case::json_manifest("v2-checkpoints-json-with-sidecars")] +fn declarative_metadata_scans_sidecars_from_checkpoint_hint( + #[case] table: &str, +) -> DeltaResult<()> { + let (engine, snapshot, _tempdir) = crate::unit_test_utils::load_test_table(table)?; + let plan = snapshot + .scan_builder() + .build()? + .declarative_metadata_scan_plan(engine.as_ref())? + .expect("metadata plan"); + + assert!(plan + .nodes + .iter() + .all(|node| !matches!(&node.op, Operator::DynamicScan(_)))); + assert!(plan.nodes.iter().any(|node| { + let Operator::ScanParquet(scan) = &node.op else { + return false; + }; + scan.files + .iter() + .any(|file| file.meta.location.path().contains("/_sidecars/")) + })); + Ok(()) +} + +#[rstest] +#[case::json_only(StatsOptions::json_only(), &[JSON_STATS_FIELDS])] +#[case::all_struct(StatsOptions::all_struct(), &[PARSED_STATS_TABLE_ALL_STATS_FIELDS])] +#[case::struct_columns( + StatsOptions::struct_columns(vec![column_name!("id")]), + &[ID_STATS_PARSED_FIELDS] +)] +#[case::empty_struct_columns(StatsOptions::struct_columns(vec![]), &[])] +#[case::all( + StatsOptions::all(), + &[PARSED_STATS_TABLE_ALL_STATS_FIELDS, JSON_STATS_FIELDS] +)] +#[case::none(StatsOptions::none(), &[])] +fn declarative_metadata_matches_imperative_across_stats_options( + #[case] stats: StatsOptions, + #[case] expected_stats_field_groups: &[&[&str]], +) -> DeltaResult<()> { + let (engine, snapshot, _tempdir) = load_test_table("parsed-stats")?; + let struct_stats = stats.struct_stats.clone(); + let no_stats = !stats.synthesize_json && matches!(&struct_stats, StructStats::None); + let expected_stats = if no_stats { + StatsOptions::json_only() + } else { + stats.clone() + }; + let predicate: PredicateRef = col!("id").gt(lit(0i64)).into(); + let expected_builder = snapshot + .clone() + .scan_builder() + .with_stats(expected_stats) + .with_partition_values(PartitionValuesOptions::with_struct()) + .with_predicate(predicate.clone()); + let expected = imperative_metadata(expected_builder.build()?, engine.as_ref())?; + let builder = snapshot + .scan_builder() + .with_stats(stats.clone()) + .with_partition_values(PartitionValuesOptions::with_struct()) + .with_predicate(predicate); + let scan = builder.build()?; + let actual = declarative_metadata(&scan, engine.as_ref())?; + let actual_fields = leaf_paths(&actual); + let imperative_fields = leaf_paths(&expected); + let unexpected_fields: Vec<_> = actual_fields + .iter() + .filter(|field| !imperative_fields.contains(field)) + .collect(); + assert!( + unexpected_fields.is_empty(), + "declarative metadata fields missing from imperative output: {unexpected_fields:?}" + ); + let actual_stats_fields: Vec<_> = actual_fields + .into_iter() + .filter(|field| field == STATS || field.starts_with("stats_parsed.")) + .collect(); + let mut expected_stats_fields: Vec<_> = expected_stats_field_groups + .iter() + .flat_map(|fields| fields.iter()) + .map(|field| { + field + .strip_prefix("add.") + .expect("expected add field") + .to_string() + }) + .collect(); + expected_stats_fields.sort_unstable(); + assert_eq!(actual_stats_fields, expected_stats_fields); + // Imperative metadata exposes source and predicate stats even when they were not requested. + // Compare only the caller-requested stats after checking the declarative schema above. + let parsed_stats_requested = match &struct_stats { + StructStats::None => false, + StructStats::Columns(columns) => !columns.is_empty(), + StructStats::All => true, + }; + if !parsed_stats_requested { + let declarative_schema = actual.first().expect("declarative metadata").schema(); + let imperative_schema = expected.first().expect("imperative metadata").schema(); + assert!(declarative_schema.field_with_name(STATS_PARSED).is_err()); + imperative_schema + .field_with_name(STATS_PARSED) + .expect("imperative predicate stats"); + } + let ignored_stats = match (stats.synthesize_json, parsed_stats_requested) { + (true, true) => { + // Both representations were requested, so the outputs are directly comparable. + &[][..] + } + (true, false) => { + // Only JSON was requested; imperative metadata also exposes predicate stats. + &[STATS_PARSED][..] + } + (false, true) => { + // Only parsed stats were requested; imperative metadata also retains source JSON. + &[STATS][..] + } + (false, false) => { + // Neither representation was requested, but imperative metadata retains source JSON + // and predicate-required parsed stats. + &[STATS, STATS_PARSED][..] + } + }; + assert_metadata_eq( + &actual, + &without_columns(&expected, ignored_stats)?, + "metadata output options", + ) +} + +const ADD_FIELDS: &[&str] = &[ + "add.path", + "add.size", + "add.modificationTime", + "add.dataChange", + "add.partitionValues", + "add.deletionVector.storageType", + "add.deletionVector.pathOrInlineDv", + "add.deletionVector.offset", + "add.deletionVector.sizeInBytes", + "add.deletionVector.cardinality", + "add.baseRowId", + "add.defaultRowCommitVersion", + "add.tags", + "add.clusteringProvider", +]; +const ALL_STATS_PARSED_FIELDS: &[&str] = &[ + "add.stats_parsed.numRecords", + "add.stats_parsed.nullCount.id", + "add.stats_parsed.nullCount.value", + "add.stats_parsed.minValues.id", + "add.stats_parsed.minValues.value", + "add.stats_parsed.maxValues.id", + "add.stats_parsed.maxValues.value", + "add.stats_parsed.tightBounds", +]; +const ID_STATS_PARSED_FIELDS: &[&str] = &[ + "add.stats_parsed.numRecords", + "add.stats_parsed.nullCount.id", + "add.stats_parsed.minValues.id", + "add.stats_parsed.maxValues.id", + "add.stats_parsed.tightBounds", +]; +const PARSED_STATS_TABLE_ALL_STATS_FIELDS: &[&str] = &[ + "add.stats_parsed.numRecords", + "add.stats_parsed.nullCount.id", + "add.stats_parsed.nullCount.name", + "add.stats_parsed.nullCount.age", + "add.stats_parsed.nullCount.salary", + "add.stats_parsed.nullCount.ts_col", + "add.stats_parsed.minValues.id", + "add.stats_parsed.minValues.name", + "add.stats_parsed.minValues.age", + "add.stats_parsed.minValues.salary", + "add.stats_parsed.minValues.ts_col", + "add.stats_parsed.maxValues.id", + "add.stats_parsed.maxValues.name", + "add.stats_parsed.maxValues.age", + "add.stats_parsed.maxValues.salary", + "add.stats_parsed.maxValues.ts_col", + "add.stats_parsed.tightBounds", +]; +const JSON_STATS_FIELDS: &[&str] = &["add.stats"]; +const PARTITION_PARSED_FIELDS: &[&str] = &["add.partitionValues_parsed.part"]; + +#[rstest] +#[should_panic(expected = "requested JSON stats must be populated")] +#[case::json_only_string_map( + StatsOptions::json_only(), + PartitionValuesOptions::string_map_only(), + &[ADD_FIELDS, JSON_STATS_FIELDS] +)] +#[should_panic(expected = "requested JSON stats must be populated")] +#[case::json_only_with_struct( + StatsOptions::json_only(), + PartitionValuesOptions::with_struct(), + &[ADD_FIELDS, JSON_STATS_FIELDS, PARTITION_PARSED_FIELDS] +)] +#[case::all_struct_string_map( + StatsOptions::all_struct(), + PartitionValuesOptions::string_map_only(), + &[ADD_FIELDS, ALL_STATS_PARSED_FIELDS] +)] +#[case::all_struct_with_struct( + StatsOptions::all_struct(), + PartitionValuesOptions::with_struct(), + &[ADD_FIELDS, ALL_STATS_PARSED_FIELDS, PARTITION_PARSED_FIELDS] +)] +#[case::struct_columns_string_map( + StatsOptions::struct_columns(vec![column_name!("id")]), + PartitionValuesOptions::string_map_only(), + &[ADD_FIELDS, ID_STATS_PARSED_FIELDS] +)] +#[case::struct_columns_with_struct( + StatsOptions::struct_columns(vec![column_name!("id")]), + PartitionValuesOptions::with_struct(), + &[ADD_FIELDS, ID_STATS_PARSED_FIELDS, PARTITION_PARSED_FIELDS] +)] +#[case::empty_struct_columns_string_map( + StatsOptions::struct_columns(vec![]), + PartitionValuesOptions::string_map_only(), + &[ADD_FIELDS] +)] +#[case::empty_struct_columns_with_struct( + StatsOptions::struct_columns(vec![]), + PartitionValuesOptions::with_struct(), + &[ADD_FIELDS, PARTITION_PARSED_FIELDS] +)] +#[should_panic(expected = "requested JSON stats must be populated")] +#[case::all_string_map( + StatsOptions::all(), + PartitionValuesOptions::string_map_only(), + &[ADD_FIELDS, ALL_STATS_PARSED_FIELDS, JSON_STATS_FIELDS] +)] +#[should_panic(expected = "requested JSON stats must be populated")] +#[case::all_with_struct( + StatsOptions::all(), + PartitionValuesOptions::with_struct(), + &[ + ADD_FIELDS, + ALL_STATS_PARSED_FIELDS, + JSON_STATS_FIELDS, + PARTITION_PARSED_FIELDS, + ] +)] +#[case::none_string_map( + StatsOptions::none(), + PartitionValuesOptions::string_map_only(), + &[ADD_FIELDS] +)] +#[case::none_with_struct( + StatsOptions::none(), + PartitionValuesOptions::with_struct(), + &[ADD_FIELDS, PARTITION_PARSED_FIELDS] +)] +fn declarative_metadata_has_exact_leaf_schema_across_output_options( + #[case] stats: StatsOptions, + #[case] partition_values: PartitionValuesOptions, + #[case] expected_field_groups: &[&[&str]], +) { + (|| -> DeltaResult<()> { + let json_requested = stats.synthesize_json; + let (engine, snapshot, _tempdir) = + load_test_table("v1-multi-part-partitioned-struct-stats-only")?; + let scan = snapshot + .scan_builder() + .with_stats(stats) + .with_partition_values(partition_values) + .build()?; + let plan = scan + .declarative_metadata_scan_plan(engine.as_ref())? + .expect("metadata plan"); + let actual = engine + .plan_executor() + .unwrap() + .execute_op(PlanOperation::QueryPlan(plan))? + .into_data()? + .map(|batch| batch?.try_into_record_batch()) + .collect::>>()?; + + if json_requested { + for batch in &actual { + let add = batch + .column_by_name(ADD_NAME) + .expect("add column") + .as_any() + .downcast_ref::() + .expect("add struct"); + let stats = add + .column_by_name(STATS) + .expect("requested JSON stats") + .as_any() + .downcast_ref::() + .expect("JSON stats"); + assert_eq!( + stats.null_count(), + 0, + "requested JSON stats must be populated" + ); + } + } + + let mut expected: Vec<_> = expected_field_groups + .iter() + .flat_map(|fields| fields.iter()) + .map(|field| field.to_string()) + .collect(); + expected.sort_unstable(); + assert_eq!(leaf_paths(&actual), expected); + Ok(()) + })() + .unwrap() +} + +#[test] +fn declarative_metadata_projects_nested_column_mapped_stats() -> DeltaResult<()> { + let (engine, snapshot, _tempdir) = load_test_table("stats-writing-all-types/delta")?; + let scan = snapshot + .scan_builder() + .with_stats(StatsOptions::struct_columns(vec![column_name!( + "nested_struct.inner_int" + )])) + .build()?; + let actual = declarative_metadata(&scan, engine.as_ref())?; + let parent = "col-481c7590-d3b8-4e9c-b40e-7b7128a972f4"; + let child = "col-7f2f94cf-7082-430c-bba7-852bc6c5215e"; + let stats_paths: Vec<_> = leaf_paths(&actual) + .into_iter() + .filter(|path| path.starts_with(STATS_PARSED)) + .collect(); + assert_eq!( + stats_paths, + [ + format!("stats_parsed.maxValues.{parent}.{child}"), + format!("stats_parsed.minValues.{parent}.{child}"), + format!("stats_parsed.nullCount.{parent}.{child}"), + "stats_parsed.numRecords".to_string(), + "stats_parsed.tightBounds".to_string(), + ] + ); + Ok(()) +} + +#[rstest] +#[case::json_commits( + LogState::with_latest_version(2), + FeatureSet::new(), + checkpoint_json_stats(), + StatsOptions::all() +)] +#[case::v1_checkpoint_json( + LogState::with_latest_version(2).with_checkpoint_at([2]), + FeatureSet::new(), + checkpoint_json_stats(), + StatsOptions::all() +)] +#[case::v1_checkpoint_struct( + LogState::with_latest_version(2).with_checkpoint_at([2]), + FeatureSet::new(), + checkpoint_struct_stats(), + StatsOptions::all_struct() +)] +#[case::v2_checkpoint_json( + LogState::with_latest_version(2) + .with_checkpoint_at([2]) + .with_sidecars_if_enabled(None), + FeatureSet::new().v2_checkpoint(), + checkpoint_json_stats(), + StatsOptions::all() +)] +#[case::v2_mixed_struct( + LogState::with_latest_version(2) + .with_checkpoint_at([1]) + .with_sidecars_if_enabled(None), + FeatureSet::new().v2_checkpoint(), + checkpoint_struct_stats(), + StatsOptions::all_struct() +)] +#[case::v2_checkpoint_without_stats( + LogState::with_latest_version(2) + .with_checkpoint_at([2]) + .with_sidecars_if_enabled(None), + FeatureSet::new().v2_checkpoint(), + no_checkpoint_stats(), + StatsOptions::all_struct() +)] +fn declarative_metadata_output_options_across_log_shapes( + #[case] log_state: LogState, + #[case] features: FeatureSet, + #[case] table_config: TableConfig, + #[case] stats: StatsOptions, +) -> DeltaResult<()> { + assert_metadata_output_options(log_state, features, table_config, stats) +} + +#[rstest] +#[case::v1( + LogState::with_latest_version(2).with_checkpoint_at([2]), + FeatureSet::new() +)] +#[case::v2( + LogState::with_latest_version(2) + .with_checkpoint_at([2]) + .with_sidecars_if_enabled(None), + FeatureSet::new().v2_checkpoint() +)] +// TODO: https://github.com/delta-io/delta-kernel-rs/issues/3040 +#[should_panic(expected = "requested JSON stats must be populated")] +fn declarative_metadata_synthesizes_json_for_struct_only_checkpoints( + #[case] log_state: LogState, + #[case] features: FeatureSet, + #[values(StatsOptions::json_only(), StatsOptions::all())] stats: StatsOptions, +) { + assert_metadata_output_options(log_state, features, checkpoint_struct_stats(), stats).unwrap(); +} + +fn assert_metadata_output_options( + log_state: LogState, + features: FeatureSet, + table_config: TableConfig, + stats: StatsOptions, +) -> DeltaResult<()> { + let json_requested = stats.synthesize_json; + let table = TestTableBuilder::new() + .with_log_state(log_state) + .with_features(features) + .with_table_config(table_config) + .with_data_layout(DataLayoutConfig::PartitionedAllTypes) + .build() + .expect("build output-options table"); + let engine = SyncEngine::new_with_store(table.store().clone()); + let snapshot = Snapshot::builder_for(table.table_root()).build(&engine)?; + let expected = imperative_metadata( + snapshot + .clone() + .scan_builder() + .with_stats(stats.clone()) + .with_partition_values(PartitionValuesOptions::with_struct()) + .build()?, + &engine, + )?; + let scan = snapshot + .scan_builder() + .with_stats(stats) + .with_partition_values(PartitionValuesOptions::with_struct()) + .build()?; + let actual = declarative_metadata(&scan, &engine)?; + + if json_requested { + for batch in &actual { + let stats = batch.column_by_name(STATS).expect("requested JSON stats"); + let stats = stats + .as_any() + .downcast_ref::() + .expect("JSON stats"); + assert_eq!( + stats.null_count(), + 0, + "requested JSON stats must be populated" + ); + } + } + + if json_requested { + assert_metadata_eq( + &actual, + &expected, + "metadata output options across log shapes", + )?; + } else { + // Imperative metadata exposes source JSON even when it was not requested. + assert_metadata_eq( + &actual, + &without_columns(&expected, &[STATS])?, + "metadata output options across log shapes", + )?; + } + Ok(()) +} + +#[rstest] +#[case::gt_three(col!("id").gt(lit(3i64)), 2)] +#[case::eq_two(col!("id").eq(lit(2i64)), 1)] +#[case::le_zero(col!("id").le(lit(0i64)), 0)] +fn declarative_metadata_data_skipping( + #[values( + "v1-multi-part-struct-stats-only", + "v2-parquet-sidecars-struct-stats-only", + "v2-json-sidecars-struct-stats-only" + )] + table: &str, + #[case] predicate: Pred, + #[case] expected_count: usize, +) -> DeltaResult<()> { + let (engine, snapshot, _tempdir) = crate::unit_test_utils::load_test_table(table)?; + let predicate = Arc::new(predicate); + let expected = imperative_metadata( + snapshot + .clone() + .scan_builder() + .with_predicate(predicate.clone()) + .with_stats(StatsOptions::all()) + .with_partition_values(PartitionValuesOptions::with_struct()) + .build()?, + engine.as_ref(), + )?; + assert_eq!(metadata_row_count(&expected), expected_count); + + let scan = snapshot + .scan_builder() + .with_predicate(predicate) + .with_stats(StatsOptions::all()) + .with_partition_values(PartitionValuesOptions::with_struct()) + .build()?; + let actual = declarative_metadata(&scan, engine.as_ref())?; + + assert_metadata_eq( + &without_columns(&actual, &[STATS])?, + &without_columns(&expected, &[STATS])?, + &format!("table {table}"), + ) +} + +#[rstest] +#[case::part_zero(col!("part").eq(lit(0i32)), 1)] +#[case::part_one(col!("part").eq(lit(1i32)), 2)] +#[case::missing_part(col!("part").eq(lit(4i32)), 0)] +fn declarative_metadata_reconstructs_partition_values_for_pruning( + #[case] predicate: Pred, + #[case] expected_count: usize, +) -> DeltaResult<()> { + let (engine, snapshot, _tempdir) = + crate::unit_test_utils::load_test_table("v1-multi-part-partitioned-struct-stats-only")?; + let predicate = Arc::new(predicate); + let expected = imperative_metadata( + snapshot + .clone() + .scan_builder() + .with_predicate(predicate.clone()) + .with_stats(StatsOptions::all()) + .with_partition_values(PartitionValuesOptions::with_struct()) + .build()?, + engine.as_ref(), + )?; + assert_eq!(metadata_row_count(&expected), expected_count); + + let scan = snapshot + .scan_builder() + .with_predicate(predicate) + .with_stats(StatsOptions::all()) + .with_partition_values(PartitionValuesOptions::with_struct()) + .build()?; + let actual = declarative_metadata(&scan, engine.as_ref())?; + + assert_metadata_eq( + &without_columns(&actual, &[STATS])?, + &without_columns(&expected, &[STATS])?, + "partition pruning", + ) +} + +#[test] +fn declarative_metadata_partition_is_null_keeps_null_partition() -> DeltaResult<()> { + let (engine, snapshot, _tempdir) = load_test_table("data-reader-timestamp_ntz")?; + let scan = snapshot + .scan_builder() + .with_predicate(Arc::new(col!("tsNtzPartition").is_null())) + .with_partition_values(PartitionValuesOptions::with_struct()) + .build()?; + let actual = declarative_metadata(&scan, engine.as_ref())?; + let formatted = pretty_format_batches(&actual)?.to_string(); + + assert_eq!(metadata_row_count(&actual), 1, "{formatted}"); + let batch = actual.first().expect("null partition metadata"); + let path = batch + .column_by_name("path") + .expect("add.path") + .as_any() + .downcast_ref::() + .expect("add.path string"); + assert_eq!( + path.value(0), + "tsNtzPartition=__HIVE_DEFAULT_PARTITION__/\ + part-00001-53fd3b3b-7773-459a-921c-bb64bf0bbd03.c000.snappy.parquet", + "{formatted}" + ); + let partitions = batch + .column_by_name(PARTITION_VALUES_PARSED) + .expect("add.partitionValues_parsed") + .as_any() + .downcast_ref::() + .expect("add.partitionValues_parsed struct"); + assert!( + partitions + .column_by_name("tsNtzPartition") + .expect("add.partitionValues_parsed.tsNtzPartition") + .is_null(0), + "{formatted}" + ); + Ok(()) +} + +#[test] +fn declarative_metadata_reconstructs_well_formed_stats_and_partitions() -> DeltaResult<()> { + let (engine, snapshot, _tempdir) = + crate::unit_test_utils::load_test_table("v1-multi-part-partitioned-struct-stats-only")?; + let scan = snapshot + .scan_builder() + .with_stats(StatsOptions::all()) + .with_partition_values(PartitionValuesOptions::with_struct()) + .build()?; + let plan = scan + .declarative_metadata_scan_plan(engine.as_ref())? + .expect("metadata plan"); + let batches = engine + .plan_executor() + .unwrap() + .execute_op(PlanOperation::QueryPlan(plan))? + .into_data()?; + + let mut projected = vec![]; + for batch in batches { + let batch = batch?.try_into_record_batch()?; + let add = batch + .column_by_name(ADD_NAME) + .expect("add column") + .as_any() + .downcast_ref::() + .expect("add struct"); + projected.push(RecordBatch::try_from_iter([ + ( + "stats", + add.column_by_name(STATS_PARSED) + .expect("add.stats_parsed") + .clone(), + ), + ( + "partitionValues", + add.column_by_name(PARTITION_VALUES_PARSED) + .expect("add.partitionValues_parsed") + .clone(), + ), + ])?); + } + + let formatted = pretty_format_batches(&projected)?.to_string(); + let mut actual_rows: Vec<_> = formatted + .lines() + .filter(|line| line.starts_with("| {numRecords:")) + .collect(); + actual_rows.sort_unstable(); + let expected_rows = [ + expected_stats_row(1, 1), + expected_stats_row(2, 2), + expected_stats_row(3, 0), + expected_stats_row(4, 1), + expected_stats_row(5, 2), + ]; + assert_eq!(actual_rows, expected_rows, "{formatted}"); + assert!(formatted.contains("| stats")); + assert!(formatted.contains("| partitionValues |")); + Ok(()) +} + +fn expected_stats_row(id: i64, partition: i32) -> String { + format!( + "| {{numRecords: 1, nullCount: {{id: 0, value: 0}}, minValues: \ + {{id: {id}, value: value_{id}}}, maxValues: {{id: {id}, value: value_{id}}}, \ + tightBounds: true}} | {{part: {partition}}} |" + ) +} + +#[test] +fn declarative_metadata_reconciles_checkpoint_with_later_commits() -> DeltaResult<()> { + let table = TestTableBuilder::new() + .with_log_state(LogState::with_latest_version(4).with_checkpoint_at([2])) + .build() + .expect("build checkpoint-plus-commits table"); + let engine = SyncEngine::new_with_store(table.store().clone()); + let snapshot = Snapshot::builder_for(table.table_root()).build(&engine)?; + + let expected = imperative_metadata( + snapshot + .clone() + .scan_builder() + .with_stats(StatsOptions::all()) + .with_partition_values(PartitionValuesOptions::with_struct()) + .build()?, + &engine, + )?; + assert_eq!(metadata_row_count(&expected), 4); + + let scan = snapshot + .scan_builder() + .with_stats(StatsOptions::all()) + .with_partition_values(PartitionValuesOptions::with_struct()) + .build()?; + let actual = declarative_metadata(&scan, &engine)?; + + assert_metadata_eq(&actual, &expected, "checkpoint with later commits") +} + +#[test] +fn declarative_metadata_pruning_keeps_remove_for_checkpoint_reconciliation() -> DeltaResult<()> { + let (engine, snapshot, _tempdir) = load_test_table("with_checkpoint_no_last_checkpoint")?; + let scan = snapshot + .scan_builder() + .with_predicate(Arc::new(col!("int").gt(lit(0i64)))) + .build()?; + let actual = declarative_metadata(&scan, engine.as_ref())?; + let formatted = pretty_format_batches(&actual)?.to_string(); + + assert_eq!(metadata_row_count(&actual), 1, "{formatted}"); + let path = actual[0] + .column_by_name("path") + .expect("add.path") + .as_any() + .downcast_ref::() + .expect("add.path string"); + assert_eq!( + path.value(0), + "part-00000-70b1dcdf-0236-4f63-a072-124cdbafd8a0-c000.snappy.parquet", + "{formatted}" + ); + Ok(()) +} + +#[rstest] +fn declarative_metadata_prunes_across_v1_log_states( + #[values( + LogState::with_latest_version(4), + LogState::with_latest_version(4).with_checkpoint_at([4]), + LogState::with_latest_version(4).with_checkpoint_at([2]) + )] + log_state: LogState, + #[values( + (col!("value").gt(lit(2500i32)), 2), + ( + col!("part_string").eq(lit("part_2000")), + 1 + ) + )] + pruning: (Pred, usize), +) -> DeltaResult<()> { + assert_declarative_metadata_matches_imperative( + log_state, + FeatureSet::new(), + pruning.0, + pruning.1, + ) +} + +#[rstest] +fn declarative_metadata_partition_prunes_v2_checkpoints( + #[values(2, 4)] checkpoint_version: u64, +) -> DeltaResult<()> { + let log_state = LogState::with_latest_version(4) + .with_checkpoint_at([checkpoint_version]) + .with_sidecars_if_enabled(None); + assert_declarative_metadata_matches_imperative( + log_state, + FeatureSet::new().v2_checkpoint(), + col!("part_string").eq(lit("part_2000")), + 1, + ) +} + +fn assert_declarative_metadata_matches_imperative( + log_state: LogState, + features: FeatureSet, + predicate: Pred, + expected_count: usize, +) -> DeltaResult<()> { + let table = TestTableBuilder::new() + .with_log_state(log_state) + .with_features(features) + .with_data_layout(DataLayoutConfig::PartitionedAllTypes) + .build() + .expect("build partitioned table"); + let engine = SyncEngine::new_with_store(table.store().clone()); + let snapshot = Snapshot::builder_for(table.table_root()).build(&engine)?; + let predicate = Arc::new(predicate); + + let expected = imperative_metadata( + snapshot + .clone() + .scan_builder() + .with_predicate(predicate.clone()) + .with_stats(StatsOptions::all()) + .with_partition_values(PartitionValuesOptions::with_struct()) + .build()?, + &engine, + )?; + assert_eq!( + metadata_row_count(&expected), + expected_count, + "{}", + table.description() + ); + let scan = snapshot + .scan_builder() + .with_predicate(predicate) + .with_stats(StatsOptions::all()) + .with_partition_values(PartitionValuesOptions::with_struct()) + .build()?; + let actual = declarative_metadata(&scan, &engine)?; + + assert_metadata_eq(&actual, &expected, table.description()) +} + +#[test] +fn test_declarative_metadata_scan_plan_no_executor_returns_unsupported() -> DeltaResult<()> { + let table = TestTableBuilder::new() + .with_log_state(LogState::with_latest_version(4).with_checkpoint_at([2])) + .build() + .expect("build checkpoint-plus-commits table"); + let sync_engine = Arc::new(SyncEngine::new_with_store(table.store().clone())); + let snapshot = Snapshot::builder_for(table.table_root()).build(sync_engine.as_ref())?; + let scan = snapshot.scan_builder().build()?; + + let no_plan_engine = DelegatingEngine::new(sync_engine).without_plan_executor(); + let err = scan + .declarative_metadata_scan_plan(&no_plan_engine) + .unwrap_err(); + + assert!(matches!(err, crate::Error::Unsupported(_))); + Ok(()) +} diff --git a/kernel/src/scan/state.rs b/kernel/src/scan/state.rs index 94d6fe05c6..e8eda6a6c3 100644 --- a/kernel/src/scan/state.rs +++ b/kernel/src/scan/state.rs @@ -3,23 +3,19 @@ use std::collections::HashMap; use std::sync::LazyLock; -use crate::actions::deletion_vector::deletion_treemap_to_bools; -use crate::scan::get_transform_for_row; -use crate::schema::Schema; -use crate::utils::require; -use crate::ExpressionRef; -use crate::{ - actions::{deletion_vector::DeletionVectorDescriptor, visitors::visit_deletion_vector_at}, - engine_data::{GetData, RowVisitor, TypedGetData as _}, - schema::{ColumnName, ColumnNamesAndTypes, DataType, SchemaRef}, - DeltaResult, Engine, EngineData, Error, -}; use roaring::RoaringTreemap; use serde::Deserialize; use tracing::warn; use super::log_replay::SCAN_ROW_SCHEMA; use super::ScanMetadata; +use crate::actions::deletion_vector::{deletion_treemap_to_bools, DeletionVectorDescriptor}; +use crate::actions::visitors::visit_deletion_vector_at; +use crate::engine_data::{FilteredRowVisitor, GetData, RowIndexIterator, TypedGetData}; +use crate::scan::get_transform_for_row; +use crate::schema::{ColumnName, ColumnNamesAndTypes, DataType, Schema, SchemaRef}; +use crate::utils::require; +use crate::{DeltaResult, Engine, EngineData, Error, ExpressionRef}; /// this struct can be used by an engine to materialize a selection vector #[derive(Default, Debug, Clone, PartialEq, Eq)] @@ -126,8 +122,8 @@ pub struct ScanFile { pub stats: Option, /// A [`DvInfo`] struct, which allows getting the selection vector for this file pub dv_info: DvInfo, - /// An optional expression that, if present, _must_ be applied to physical data to convert it to - /// the correct logical format + /// An optional expression that, if present, _must_ be applied to physical data to convert it + /// to the correct logical format pub transform: Option, /// a `HashMap` which map partition names to the value they have in this file pub partition_values: HashMap, @@ -163,28 +159,30 @@ impl ScanMetadata { pub fn visit_scan_files(&self, context: T, callback: ScanCallback) -> DeltaResult { let mut visitor = ScanFileVisitor { callback, - selection_vector: self.scan_files.selection_vector(), transforms: &self.scan_file_transforms, context, }; - visitor.visit_rows_of(self.scan_files.data())?; + visitor.visit_rows_of(&self.scan_files)?; Ok(visitor.context) } } // add some visitor magic for engines struct ScanFileVisitor<'a, T> { callback: ScanCallback, - selection_vector: &'a [bool], transforms: &'a [Option], context: T, } -impl RowVisitor for ScanFileVisitor<'_, T> { +impl FilteredRowVisitor for ScanFileVisitor<'_, T> { fn selected_column_names_and_types(&self) -> (&'static [ColumnName], &'static [DataType]) { static NAMES_AND_TYPES: LazyLock = LazyLock::new(|| SCAN_ROW_SCHEMA.leaves(None)); NAMES_AND_TYPES.as_ref() } - fn visit<'a>(&mut self, row_count: usize, getters: &[&'a dyn GetData<'a>]) -> DeltaResult<()> { + fn visit_filtered<'a>( + &mut self, + getters: &[&'a dyn GetData<'a>], + rows: RowIndexIterator<'_>, + ) -> DeltaResult<()> { require!( getters.len() == 14, Error::InternalError(format!( @@ -192,11 +190,7 @@ impl RowVisitor for ScanFileVisitor<'_, T> { getters.len() )) ); - for row_index in 0..row_count { - if !self.selection_vector[row_index] { - // skip skipped rows - continue; - } + for row_index in rows { // Since path column is required, use it to detect presence of an Add action if let Some(path) = getters[0].get_opt(row_index, "scanFile.path")? { let size = getters[1].get(row_index, "scanFile.size")?; diff --git a/kernel/src/scan/state_info.rs b/kernel/src/scan/state_info.rs index 2686b568ff..a8a356b751 100644 --- a/kernel/src/scan/state_info.rs +++ b/kernel/src/scan/state_info.rs @@ -4,18 +4,16 @@ use std::collections::HashSet; use std::sync::Arc; -use tracing::{debug, warn}; +use tracing::{debug, enabled, warn, Level}; +use crate::actions::NULL_COUNT; use crate::expressions::ColumnName; -use crate::scan::data_skipping::stats_schema::build_stats_schema; use crate::scan::field_classifiers::TransformFieldClassifier; use crate::scan::transform_spec::{FieldTransformSpec, TransformSpec}; -use crate::scan::PhysicalPredicate; -use crate::scan::StatsOutputMode; +use crate::scan::{PartitionValuesOptions, PhysicalPredicate, StatsOptions, StructStats}; use crate::schema::{DataType, MetadataColumnSpec, SchemaRef, StructType}; use crate::table_configuration::TableConfiguration; -use crate::table_features::get_any_level_column_physical_name; -use crate::table_features::ColumnMappingMode; +use crate::table_features::{get_any_level_column_physical_name, ColumnMappingMode}; use crate::{DeltaResult, Error, PredicateRef, StructField}; /// All the state needed to process a scan. @@ -34,10 +32,29 @@ pub(crate) struct StateInfo { /// Physical stats schema for reading/parsing stats from checkpoint files. /// Used to construct checkpoint read schema with stats_parsed. pub(crate) physical_stats_schema: Option, - /// Physical partition schema with native types for checkpoint partition pruning via - /// `partitionValues_parsed`. Fields use physical column names (for column mapping). - /// Only present when the table has partition columns and a predicate is provided. + /// Physical partition schema with native types for `partitionValues_parsed`. Fields use + /// physical column names (for column mapping) and are always nullable. Present when the + /// table has partition columns and either a predicate is provided (narrowed to + /// predicate-referenced columns, for partition pruning) or the engine requested the typed + /// struct in scan output (all partition columns). pub(crate) physical_partition_schema: Option, + /// Physical leaf paths which are expected to have stats collected. + /// + /// Differs from `physical_stats_schema` in that this is the per-table membership set + /// (predicate-independent). `physical_stats_schema` is the per-scan projection shape + /// (predicate-trimmed). + /// + /// Read-path mirror of `SharedWriteState.stats_columns`. + pub(crate) physical_stats_columns: HashSet, + /// Whether the table is catalog-managed, used to label scan metric events. Converted to a + /// [`TableType`](crate::metrics::TableType) at event construction. + pub(crate) is_catalog_managed: bool, + /// When set, log replay does not build per-file transform expressions: + /// `parse_partition_values` and `get_transform_expr` are skipped and every + /// `ScanMetadata::scan_file_transforms` entry is left `None`. `transform_spec` is retained + /// so the scan can still describe the transform structurally. Set by + /// [`ScanBuilder::without_row_transforms`](crate::scan::ScanBuilder::without_row_transforms). + pub(crate) skip_row_transforms: bool, } /// Validating the metadata columns also extracts information needed to properly construct the full @@ -47,13 +64,13 @@ pub(crate) struct StateInfo { struct MetadataInfo<'a> { /// What are the names of the requested metadata fields metadata_field_names: HashSet<&'a String>, - /// The name of the column that's selecting row indexes if that's been requested or None if they - /// are not requested. We remember this if it's been requested explicitly. this is so we can - /// reference this column and not re-add it as a requested column if we're _also_ requesting - /// row-ids. + /// The name of the column that's selecting row indexes if that's been requested or None if + /// they are not requested. We remember this if it's been requested explicitly. this is so + /// we can reference this column and not re-add it as a requested column if we're _also_ + /// requesting row-ids. selected_row_index_col_name: Option<&'a String>, - /// the materializedRowIdColumnName extracted from the table config if row ids are requested, or - /// None if they are not requested + /// the materializedRowIdColumnName extracted from the table config if row ids are requested, + /// or None if they are not requested materialized_row_id_column_name: Option<&'a String>, } @@ -67,7 +84,7 @@ fn validate_metadata_columns<'a>( table_configuration: &'a TableConfiguration, ) -> DeltaResult> { let mut metadata_info = MetadataInfo::default(); - let partition_columns = table_configuration.partition_columns(); + let partition_columns = table_configuration.logical_partition_columns(); for metadata_column in logical_schema.metadata_columns() { // Ensure we don't have a metadata column with same name as a partition column if partition_columns.contains(metadata_column.name()) { @@ -104,100 +121,141 @@ fn validate_metadata_columns<'a>( Ok(metadata_info) } -/// Build the physical stats schema based on `StatsOutputMode` and `PhysicalPredicate`: +/// Build data-skipping schemas based on `StructStats` and `PhysicalPredicate`. +/// +/// Returns `(physical_stats_schema, physical_partition_schema)`, where: +/// - `physical_stats_schema` contains data-column stats for `stats_parsed`. +/// - `physical_partition_schema` contains typed partition values for `partitionValues_parsed`. /// -/// - `StatsOutputMode::AllColumns` + any predicate: output all stats from expected_stats_schema. -/// Example: `(StatsOutputMode::AllColumns, _)` -> full expected stats schema. -/// - `StatsOutputMode::Columns(non-empty)` + any predicate: merge requested + predicate columns -/// for data skipping. -/// Example: `(StatsOutputMode::Columns([x, y]), _)` with predicate on `z` -> stats for {x, y, z}. -/// - `StatsOutputMode::Columns(empty)` + `PhysicalPredicate::Some`: predicate-only internal data -/// skipping. -/// Example: `(StatsOutputMode::Columns([]), PhysicalPredicate::Some(..))` -> stats from -/// predicate ref schema. -/// - `StatsOutputMode::Skip` + `PhysicalPredicate::None`: no stats at all. -/// Example: `(StatsOutputMode::Skip, PhysicalPredicate::None)` -> returns `None`. -fn build_physical_stats_schema( - stats_output_mode: &StatsOutputMode, +/// All three arms route through `TableConfiguration::build_expected_stats_schemas`: the +/// `All` arm with no `requested_physical_columns` filter, and the two scoped arms with +/// the union of requested + predicate-referenced columns. That path applies the same +/// `BaseStatsTransform` -> `MinMaxStatsTransform` pipeline writers use, so the read-side +/// stats schema's shape matches the write-side exactly. +fn build_data_skipping_schemas( + struct_stats: &StructStats, physical_predicate: &PhysicalPredicate, predicate_column_names_logical: &[ColumnName], table_configuration: &TableConfiguration, -) -> DeltaResult> { - match (stats_output_mode, physical_predicate) { - // Output all table stats columns in stats_parsed. The DataSkippingFilter - // reads stats_parsed from the transformed batch, which uses this schema. - (StatsOutputMode::AllColumns, _) => { - let expected_stats_schemas = - table_configuration.build_expected_stats_schemas(None, None)?; - Ok(Some(expected_stats_schemas.physical)) +) -> DeltaResult<(Option, Option)> { + // Narrow the table's typed partition schema to the columns the predicate references. The + // DataSkippingFilter only needs partition columns that appear in the predicate, and the + // shared helper forces every field nullable (MapToStruct can yield null for a missing key). + let predicate_partition_schema = match physical_predicate { + PhysicalPredicate::Some(pred, _ref_schema) => { + let refs: Vec = pred.references().into_iter().cloned().collect(); + table_configuration.predicate_partition_schema(&refs) } - // Non-empty requested columns -- include predicate-referenced columns - // alongside the user-requested stats columns so that the DataSkippingFilter - // has the stats it needs. Both sources are logical names that must be - // converted to physical before passing to build_expected_stats_schemas. - (StatsOutputMode::Columns(requested_columns), _) if !requested_columns.is_empty() => { - let existing: HashSet<&ColumnName> = requested_columns.iter().collect(); - let mut all_needed_logical = requested_columns.clone(); - for col in predicate_column_names_logical { - if !existing.contains(col) { - all_needed_logical.push(col.clone()); - } + _ => None, + }; + + // `DataSkippingFilter` needs stats for every column its predicate references. Refs + // without stats fold to NULL and pruning collapses to "keep every file", even when + // the caller separately requested stats for some other set of columns via + // `StructStats::Columns`. Union the two so the schema serves both. Unresolvable + // refs (e.g. a predicate typo) are dropped here. + let union_to_physical = |requested_logical: &[ColumnName]| -> Vec { + let mut union_logical: Vec = requested_logical.to_vec(); + let existing: HashSet<&ColumnName> = requested_logical.iter().collect(); + for col in predicate_column_names_logical { + if !existing.contains(col) { + union_logical.push(col.clone()); } - let logical_schema = table_configuration.logical_schema(); - let column_mapping_mode = table_configuration.column_mapping_mode(); - let all_needed_physical: Vec = all_needed_logical - .iter() - .filter_map(|col| { - // Theoretically this should always resolve -- if it doesn't, - // the column was not found in the logical schema (e.g. a - // requested stats column or predicate column that doesn't - // exist in the table schema), - // which is safe to ignore. - get_any_level_column_physical_name(&logical_schema, col, column_mapping_mode) - .inspect_err(|e| { - warn!("Failed to resolve physical name for column {col}: {e}") - }) - .ok() - }) - .collect(); - let expected_stats_schemas = table_configuration - .build_expected_stats_schemas(None, Some(&all_needed_physical))?; - Ok(Some(expected_stats_schemas.physical)) } - // Columns(empty) or Skip with a physical predicate -- build stats directly - // from the physical predicate's referenced schema for internal data skipping. - (_, PhysicalPredicate::Some(_, ref_schema)) => Ok(build_stats_schema(ref_schema)), - // No stats output and no predicate - (_, _) => Ok(None), - } + let logical_schema = table_configuration.logical_schema(); + let column_mapping_mode = table_configuration.column_mapping_mode(); + union_logical + .iter() + .filter_map(|col| { + get_any_level_column_physical_name(&logical_schema, col, column_mapping_mode) + .inspect_err(|e| warn!("Failed to resolve physical name for column {col}: {e}")) + .ok() + }) + .collect() + }; + + // A stats schema with only `numRecords` and `tightBounds` (the bookkeeping fields + // `build_expected_stats_schemas` always emits) has nothing to prune by. Return `None` + // in that case so the caller skips building a `DataSkippingFilter`. `nullCount` is the + // per-column stats wrapper, so its presence is the signal that at least one data + // column survived. The Delta protocol allows `minValues` / `maxValues` without + // `nullCount`, but `build_expected_stats_schemas` always emits `nullCount` whenever it + // emits min/max; this check relies on that implementation property. + let with_data_cols = |stats_schema: SchemaRef| -> Option { + stats_schema + .field(NULL_COUNT) + .is_some() + .then_some(stats_schema) + }; + + let stats_schema = match (struct_stats, physical_predicate) { + // Full table stats schema for stats_parsed. + (StructStats::All, _) => with_data_cols( + table_configuration + .build_expected_stats_schemas(None, None)? + .physical, + ), + // Explicit requested columns. Union in predicate refs so the stats schema covers + // both sources. + (StructStats::Columns(requested_columns), _) if !requested_columns.is_empty() => { + let requested_physical = union_to_physical(requested_columns); + with_data_cols( + table_configuration + .build_expected_stats_schemas(None, Some(&requested_physical))? + .physical, + ) + } + // No explicit requested columns, but a predicate is present. Use just the predicate + // refs so the stats schema is trimmed to what the rewritten predicate needs. + (_, PhysicalPredicate::Some(_, _)) => { + let predicate_refs_physical = union_to_physical(&[]); + with_data_cols( + table_configuration + .build_expected_stats_schemas(None, Some(&predicate_refs_physical))? + .physical, + ) + } + (_, _) => None, + }; + Ok((stats_schema, predicate_partition_schema)) } impl StateInfo { /// Create StateInfo with a custom field classifier for different scan types. /// Get the state needed to process a scan. /// - /// `logical_schema` - The logical schema of the scan output, which includes partition columns + /// `logical_read_schema` - The logical schema of the scan output + /// `table_schema` - The schema against which predicate column references are resolved. + /// Must contain every column the predicate may legitimately reference (typically the full + /// table schema, or full CDF-extended schema for CDF scans). Currently, we do not carry + /// over any metadata columns from the `logical_read_schema` to the `table_schema` (issue + /// 2633). /// `table_configuration` - The TableConfiguration for this table /// `predicate` - Optional predicate to filter data during the scan - /// `stats_output_mode` - Controls how file statistics are handled during the scan + /// `stats` - Engine-facing stats options. Drives which stats columns appear in scan + /// metadata output and whether the JSON synthesis fallback fires. + /// `partition_values` - Engine-facing partition value options. Drives whether the typed + /// `partitionValues_parsed` column appears in scan metadata output. /// `classifier` - The classifier to use for different scan types. Use `()` if not needed pub(crate) fn try_new( - logical_schema: SchemaRef, + logical_read_schema: SchemaRef, + table_schema: SchemaRef, table_configuration: &TableConfiguration, predicate: Option, - stats_output_mode: StatsOutputMode, + stats: &StatsOptions, + partition_values: &PartitionValuesOptions, classifier: C, ) -> DeltaResult { - let partition_columns = table_configuration.partition_columns(); + let partition_columns = table_configuration.logical_partition_columns(); let column_mapping_mode = table_configuration.column_mapping_mode(); - let mut read_fields = Vec::with_capacity(logical_schema.num_fields()); - let mut transform_spec = Vec::with_capacity(logical_schema.num_fields()); + let mut read_fields = Vec::with_capacity(logical_read_schema.num_fields()); + let mut transform_spec = Vec::with_capacity(logical_read_schema.num_fields()); let mut last_physical_field: Option = None; - let metadata_info = validate_metadata_columns(&logical_schema, table_configuration)?; + let metadata_info = validate_metadata_columns(&logical_read_schema, table_configuration)?; // Loop over all selected fields and build both the physical schema and transform spec - for (index, logical_field) in logical_schema.fields().enumerate() { + for (index, logical_field) in logical_read_schema.fields().enumerate() { if let Some(spec) = classifier.classify_field(logical_field, index, &last_physical_field) { @@ -223,7 +281,7 @@ impl StateInfo { // ensure we have a column name that isn't already in our schema let index_column_name = (0..) .map(|i| format!("row_indexes_for_row_id_{i}")) - .find(|name| logical_schema.field(name).is_none()) + .find(|name| logical_read_schema.field(name).is_none()) .ok_or(Error::generic( "Couldn't generate row index column name", ))?; @@ -256,8 +314,9 @@ impl StateInfo { Some(MetadataColumnSpec::RowIndex) | Some(MetadataColumnSpec::FilePath) | None => { - // note that RowIndex and FilePath are handled in the parquet reader so we just add them as - // if they're normal physical columns + // note that RowIndex and FilePath are handled in the parquet reader so we + // just add them as if they're normal physical + // columns let physical_field = logical_field.make_physical(column_mapping_mode)?; debug!("\n\n{logical_field:#?}\nAfter mapping: {physical_field:#?}\n\n"); let physical_name = physical_field.name.clone(); @@ -279,53 +338,97 @@ impl StateInfo { let physical_schema = Arc::new(StructType::try_new(read_fields)?); - // Extract column names referenced by the predicate so we can include them - // in the stats schema when stats_columns is requested. This ensures the - // DataSkippingFilter has the stats it needs for data skipping. + // Logical column names referenced by the predicate. Fed into the stats schema + // build below and into the dropped-refs observability log. let predicate_column_names: Vec = predicate .as_ref() .map(|p| p.references().into_iter().cloned().collect()) .unwrap_or_default(); + // We use table_schema here as predicate can reference columns outside projection. let physical_predicate = match predicate { - Some(pred) => PhysicalPredicate::try_new(&pred, &logical_schema, column_mapping_mode)?, + Some(pred) => PhysicalPredicate::try_new(&pred, &table_schema, column_mapping_mode)?, None => PhysicalPredicate::None, }; - // Build partition schema with physical names for checkpoint partition pruning. - // Only needed when we have a predicate and partition columns. + // Stats-eligible column set. Partition columns are excluded; they flow through + // `partitionValues_parsed` instead. + let physical_stats_columns = table_configuration.physical_stats_columns_set(None); + // Observability: predicate refs outside `physical_stats_columns` get folded to NULL + // by the gate. Surface the dropped set so an engine operator can see what got folded. + // The filter walk is bounded by predicate width but still does a physical-name + // resolution per ref, so gate it on the log level to skip the work when DEBUG is off. + if enabled!(Level::DEBUG) && matches!(physical_predicate, PhysicalPredicate::Some(_, _)) { + let dropped: Vec<&ColumnName> = predicate_column_names + .iter() + .filter(|c| { + get_any_level_column_physical_name(&table_schema, c, column_mapping_mode) + .ok() + .is_some_and(|physical| !physical_stats_columns.contains(&physical)) + }) + .collect(); + if !dropped.is_empty() { + debug!( + "Checkpoint pushdown: predicate refs to non-stats columns folded to NULL: {:?}", + dropped + ); + } + } + + // Build partition schema with physical names, used for partition pruning in data + // skipping and for the engine-facing `partitionValues_parsed` output column. Needed + // when partition columns exist and either a predicate is present or the engine + // requested the typed struct in scan output. // partition_columns stores logical names (per Delta protocol), so we zip the table's // logical and physical schemas (same field ordering, guaranteed by `make_physical`) // to match logical names and extract the corresponding physical fields without // per-field metadata lookups. - let physical_partition_schema = if !matches!( + let has_predicate = !matches!( physical_predicate, PhysicalPredicate::None | PhysicalPredicate::StaticSkipAll - ) && !partition_columns.is_empty() - { - let partition_fields: Vec = table_configuration - .logical_schema() - .fields() - .zip(table_configuration.physical_schema().fields()) - .filter(|(logical_f, _)| partition_columns.contains(logical_f.name())) - .map(|(_, physical_f)| physical_f.clone()) - .collect(); - if partition_fields.is_empty() { - None + ); + let table_partition_schema = + if (has_predicate || partition_values.parsed_struct) && !partition_columns.is_empty() { + let partition_fields: Vec = table_configuration + .logical_schema() + .fields() + .zip(table_configuration.physical_schema().fields()) + .filter(|(logical_f, _)| partition_columns.contains(logical_f.name())) + .map(|(_, physical_f)| physical_f.clone()) + .collect(); + if partition_fields.is_empty() { + None + } else { + Some(Arc::new(StructType::new_unchecked(partition_fields))) + } } else { - Some(Arc::new(StructType::new_unchecked(partition_fields))) - } - } else { - None - }; + None + }; - let physical_stats_schema = build_physical_stats_schema( - &stats_output_mode, + let (physical_stats_schema, predicate_partition_schema) = build_data_skipping_schemas( + &stats.struct_stats, &physical_predicate, &predicate_column_names, table_configuration, )?; + // When the engine requested the typed struct, emit all partition columns rather than + // the predicate-narrowed subset. The data skipping filter only references the columns + // its predicate needs, so the superset is safe for pruning too. All fields are forced + // nullable: MapToStruct can yield null even for a non-nullable column (a missing key, or + // an empty string cast to a non-string/binary type), and a non-nullable field would then + // error. + let physical_partition_schema = if partition_values.parsed_struct { + table_partition_schema.map(|tps| { + let nullable_fields = tps + .fields() + .map(|f| StructField::nullable(f.name(), f.data_type().clone())); + Arc::new(StructType::new_unchecked(nullable_fields)) + }) + } else { + predicate_partition_schema + }; + let transform_spec = if !transform_spec.is_empty() || column_mapping_mode != ColumnMappingMode::None { Some(Arc::new(transform_spec)) @@ -334,30 +437,51 @@ impl StateInfo { }; Ok(StateInfo { - logical_schema, + logical_schema: logical_read_schema, physical_schema, physical_predicate, transform_spec, column_mapping_mode, physical_stats_schema, physical_partition_schema, + physical_stats_columns, + is_catalog_managed: table_configuration.is_catalog_managed(), + skip_row_transforms: false, }) } + + /// Returns a conservative initial capacity for the dedup `HashSet` in + /// [`ScanLogReplayProcessor`]. + /// + /// The exact file count is not available at this point, so the hint is + /// derived from whether stats are enabled: stats are only computed for + /// non-trivial tables, so their presence is a reasonable proxy for table + /// size. Using 4096 vs 512 as the two tiers eliminates the first 12-14 + /// hashbrown doubling events for medium/large tables while staying cheap + /// for small ones. + pub(crate) fn dedup_capacity_hint(&self) -> usize { + if self.physical_stats_schema.is_some() { + 4096 + } else { + 512 + } + } } #[cfg(test)] pub(crate) mod tests { - use std::{collections::HashMap, sync::Arc}; + use std::collections::HashMap; + use std::sync::Arc; + use rstest::rstest; use url::Url; - use crate::actions::{Metadata, Protocol}; - use crate::expressions::{column_expr, column_name, Expression as Expr}; - use crate::schema::{ColumnMetadataKey, MetadataValue}; - use crate::table_features::{FeatureType, TableFeature}; - use crate::utils::test_utils::assert_result_error_with_message; - use super::*; + use crate::actions::{Metadata, Protocol, MAX_VALUES, MIN_VALUES}; + use crate::expressions::{col, column_name, lit, Predicate as Pred}; + use crate::schema::{schema_ref, ColumnMetadataKey, MetadataValue}; + use crate::table_features::{FeatureType, TableFeature}; + use crate::unit_test_utils::assert_result_error_with_message; // get a state info with no predicate or extra metadata pub(crate) fn get_simple_state_info( @@ -384,7 +508,7 @@ pub(crate) mod tests { features, metadata_configuration, metadata_cols, - StatsOutputMode::default(), + StatsOptions::default(), ) } @@ -395,7 +519,30 @@ pub(crate) mod tests { features: &[TableFeature], metadata_configuration: HashMap, metadata_cols: Vec<(&str, MetadataColumnSpec)>, - stats_output_mode: StatsOutputMode, + stats: StatsOptions, + ) -> DeltaResult { + get_state_info_with_options( + schema, + partition_columns, + predicate, + features, + metadata_configuration, + metadata_cols, + stats, + PartitionValuesOptions::default(), + ) + } + + #[allow(clippy::too_many_arguments)] + pub(crate) fn get_state_info_with_options( + schema: SchemaRef, + partition_columns: Vec, + predicate: Option, + features: &[TableFeature], + metadata_configuration: HashMap, + metadata_cols: Vec<(&str, MetadataColumnSpec)>, + stats: StatsOptions, + partition_values: PartitionValuesOptions, ) -> DeltaResult { let metadata = Metadata::try_new( None, @@ -439,9 +586,11 @@ pub(crate) mod tests { StateInfo::try_new( schema.clone(), + table_configuration.logical_schema(), &table_configuration, predicate, - stats_output_mode, + &stats, + &partition_values, (), ) } @@ -596,7 +745,7 @@ pub(crate) mod tests { StructField::nullable("value", DataType::LONG), ])); - let predicate = Arc::new(column_expr!("value").gt(Expr::literal(10i64))); + let predicate = Arc::new(col!("value").gt(lit(10i64))); let state_info = get_state_info( schema.clone(), @@ -655,12 +804,23 @@ pub(crate) mod tests { .collect() } + /// Builds a nullable [`StructField`] carrying column-mapping id + physical name metadata. + fn cm_field(name: &str, id: i64, physical_name: &str, ty: impl Into) -> StructField { + StructField::nullable(name, ty).with_metadata([ + ( + ColumnMetadataKey::ColumnMappingId.as_ref(), + MetadataValue::Number(id), + ), + ( + ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), + MetadataValue::String(physical_name.into()), + ), + ]) + } + #[test] fn request_row_ids() { - let schema = Arc::new(StructType::new_unchecked(vec![StructField::nullable( - "id", - DataType::STRING, - )])); + let schema = schema_ref! { nullable "id": STRING }; let state_info = get_state_info( schema.clone(), @@ -694,10 +854,8 @@ pub(crate) mod tests { #[test] fn request_row_ids_conflicting_row_index_col_name() { - let schema = Arc::new(StructType::new_unchecked(vec![StructField::nullable( - "row_indexes_for_row_id_0", // this will conflict with the first generated name for row indexes - DataType::STRING, - )])); + // "row_indexes_for_row_id_0" conflicts with the first generated name for row indexes + let schema = schema_ref! { nullable "row_indexes_for_row_id_0": STRING }; let state_info = get_state_info( schema.clone(), @@ -731,10 +889,7 @@ pub(crate) mod tests { #[test] fn request_row_ids_and_indexes() { - let schema = Arc::new(StructType::new_unchecked(vec![StructField::nullable( - "id", - DataType::STRING, - )])); + let schema = schema_ref! { nullable "id": STRING }; let state_info = get_state_info( schema.clone(), @@ -771,10 +926,7 @@ pub(crate) mod tests { #[test] fn invalid_rowtracking_config() { - let schema = Arc::new(StructType::new_unchecked(vec![StructField::nullable( - "id", - DataType::STRING, - )])); + let schema = schema_ref! { nullable "id": STRING }; // Row IDs requested but row tracking not enabled → error let res = get_state_info( @@ -804,17 +956,41 @@ pub(crate) mod tests { #[test] fn metadata_column_matches_partition_column() { - let schema = Arc::new(StructType::new_unchecked(vec![StructField::nullable( - "id", - DataType::STRING, - )])); - let res = get_state_info( - schema.clone(), - vec!["part_col".to_string()], + let table_schema = Arc::new(StructType::new_unchecked(vec![ + StructField::nullable("id", DataType::STRING), + StructField::nullable("part_col", DataType::STRING), + ])); + let metadata = Metadata::try_new( None, - &[], // no table features + None, + table_schema, + vec!["part_col".to_string()], + 10, HashMap::new(), - vec![("part_col", MetadataColumnSpec::RowId)], + ) + .unwrap(); + let table_configuration = TableConfiguration::try_new( + metadata, + Protocol::try_new_legacy(2, 5).unwrap(), + Url::parse("s3://my-table").unwrap(), + 1, + ) + .unwrap(); + + let read_schema = schema_ref! { nullable "id": STRING }; + let read_schema = Arc::new( + read_schema + .add_metadata_column("part_col", MetadataColumnSpec::RowId) + .expect("Couldn't add metadata col"), + ); + let res = StateInfo::try_new( + read_schema, + table_configuration.logical_schema(), + &table_configuration, + None, + &StatsOptions::default(), + &PartitionValuesOptions::default(), + (), ); assert_result_error_with_message( res, @@ -824,22 +1000,7 @@ pub(crate) mod tests { #[test] fn metadata_column_matches_read_field() { - let schema = Arc::new(StructType::new_unchecked(vec![StructField::nullable( - "id", - DataType::STRING, - ) - .with_metadata(HashMap::::from([ - ( - ColumnMetadataKey::ColumnMappingId.as_ref().to_string(), - 1.into(), - ), - ( - ColumnMetadataKey::ColumnMappingPhysicalName - .as_ref() - .to_string(), - "other".into(), - ), - ]))])); + let schema = schema_ref! { (cm_field("id", 1, "other", DataType::STRING)) }; let res = get_state_info( schema.clone(), vec![], @@ -861,7 +1022,7 @@ pub(crate) mod tests { StructField::nullable("value", DataType::LONG), ])); - let predicate = Arc::new(column_expr!("value").gt(Expr::literal(10i64))); + let predicate = Arc::new(col!("value").gt(lit(10i64))); let state_info = get_state_info_with_stats( schema, @@ -870,7 +1031,7 @@ pub(crate) mod tests { &[], // no table features HashMap::new(), vec![], - StatsOutputMode::AllColumns, + StatsOptions::all(), ) .unwrap(); @@ -896,7 +1057,7 @@ pub(crate) mod tests { StructField::nullable("extra", DataType::LONG), ])); - let predicate = Arc::new(column_expr!("extra").gt(Expr::literal(5i64))); + let predicate = Arc::new(col!("extra").gt(lit(5i64))); let state_info = get_state_info_with_stats( schema, @@ -905,7 +1066,10 @@ pub(crate) mod tests { &[], HashMap::new(), vec![], - StatsOutputMode::Columns(vec![column_name!("value")]), + StatsOptions { + synthesize_json: true, + struct_stats: StructStats::Columns(vec![column_name!("value")]), + }, ) .unwrap(); @@ -914,7 +1078,7 @@ pub(crate) mod tests { .expect("should have physical stats schema"); let min_values = stats_schema - .field("minValues") + .field(MIN_VALUES) .expect("should have minValues"); if let DataType::Struct(inner) = min_values.data_type() { assert!( @@ -948,7 +1112,10 @@ pub(crate) mod tests { &[], // no table features HashMap::new(), vec![], - StatsOutputMode::Columns(vec![column_name!("value")]), + StatsOptions { + synthesize_json: true, + struct_stats: StructStats::Columns(vec![column_name!("value")]), + }, ) .unwrap(); @@ -958,7 +1125,7 @@ pub(crate) mod tests { // Check that minValues/maxValues only contain 'value', not 'id' let min_values = stats_schema - .field("minValues") + .field(MIN_VALUES) .expect("should have minValues"); if let DataType::Struct(inner) = min_values.data_type() { assert!( @@ -979,46 +1146,13 @@ pub(crate) mod tests { // Verify that physical_partition_schema uses physical column names when column // mapping is enabled. The logical partition column "date" has physical name // "col-date-phys", and the schema should reflect the physical name. - let schema = Arc::new(StructType::new_unchecked(vec![ - StructField::nullable("id", DataType::STRING).with_metadata(HashMap::from([ - ( - ColumnMetadataKey::ColumnMappingId.as_ref().to_string(), - MetadataValue::Number(1), - ), - ( - ColumnMetadataKey::ColumnMappingPhysicalName - .as_ref() - .to_string(), - MetadataValue::String("col-id-phys".to_string()), - ), - ])), - StructField::nullable("date", DataType::DATE).with_metadata(HashMap::from([ - ( - ColumnMetadataKey::ColumnMappingId.as_ref().to_string(), - MetadataValue::Number(2), - ), - ( - ColumnMetadataKey::ColumnMappingPhysicalName - .as_ref() - .to_string(), - MetadataValue::String("col-date-phys".to_string()), - ), - ])), - StructField::nullable("value", DataType::LONG).with_metadata(HashMap::from([ - ( - ColumnMetadataKey::ColumnMappingId.as_ref().to_string(), - MetadataValue::Number(3), - ), - ( - ColumnMetadataKey::ColumnMappingPhysicalName - .as_ref() - .to_string(), - MetadataValue::String("col-value-phys".to_string()), - ), - ])), - ])); + let schema = schema_ref! { + (cm_field("id", 1, "col-id-phys", DataType::STRING)), + (cm_field("date", 2, "col-date-phys", DataType::DATE)), + (cm_field("value", 3, "col-value-phys", DataType::LONG)), + }; - let predicate = Arc::new(column_expr!("date").lt(Expr::literal(100i32))); + let predicate = Arc::new(col!("date").lt(lit(100i32))); let state_info = get_state_info( schema, @@ -1045,48 +1179,87 @@ pub(crate) mod tests { assert_eq!(field.data_type(), &DataType::DATE); } - #[test] - fn stats_columns_with_column_mapping_uses_physical_names() { - let field_a: StructField = serde_json::from_value(serde_json::json!({ - "name": "col_a", - "type": "long", - "nullable": true, - "metadata": { - "delta.columnMapping.id": 1, - "delta.columnMapping.physicalName": "phys_a" - } - })) - .unwrap(); - - let field_b: StructField = serde_json::from_value(serde_json::json!({ - "name": "col_b", - "type": "long", - "nullable": true, - "metadata": { - "delta.columnMapping.id": 2, - "delta.columnMapping.physicalName": "phys_b" - } - })) + /// `with_struct` builds `physical_partition_schema` from all partition columns (independent of + /// predicate and `StatsOptions::none`), and omits it entirely on non-partitioned tables. + #[rstest] + #[case::all_columns_without_predicate( + vec![ + StructField::nullable("id", DataType::LONG), + StructField::nullable("region", DataType::STRING), + StructField::nullable("date", DataType::DATE), + ], + vec!["region".to_string(), "date".to_string()], + StatsOptions::default(), + Some(vec!["region", "date"]), + )] + #[case::survives_stats_none( + vec![ + StructField::nullable("value", DataType::LONG), + StructField::nullable("date", DataType::DATE), + ], + vec!["date".to_string()], + StatsOptions::none(), + Some(vec!["date"]), + )] + #[case::omitted_for_non_partitioned_table( + vec![ + StructField::nullable("id", DataType::LONG), + StructField::nullable("value", DataType::STRING), + ], + vec![], + StatsOptions::default(), + None, + )] + fn partition_values_with_struct( + #[case] fields: Vec, + #[case] partition_columns: Vec, + #[case] stats: StatsOptions, + #[case] expected_names: Option>, + ) { + let schema = Arc::new(StructType::new_unchecked(fields)); + let state_info = get_state_info_with_options( + schema, + partition_columns, + None, + &[], + HashMap::new(), + vec![], + stats, + PartitionValuesOptions::with_struct(), + ) .unwrap(); - let field_c: StructField = serde_json::from_value(serde_json::json!({ - "name": "col_c", - "type": "long", - "nullable": true, - "metadata": { - "delta.columnMapping.id": 3, - "delta.columnMapping.physicalName": "phys_c" + match expected_names { + Some(expected) => { + let partition_schema = state_info + .physical_partition_schema + .as_ref() + .expect("with_struct should build a partition schema"); + let names: Vec<&str> = partition_schema + .fields() + .map(|f| f.name().as_str()) + .collect(); + assert_eq!(names, expected); + // MapToStruct lookups can return null, so every field must be nullable. + assert!(partition_schema.fields().all(|f| f.is_nullable())); } - })) - .unwrap(); + None => assert!(state_info.physical_partition_schema.is_none()), + } + } - let schema = Arc::new(StructType::new_unchecked(vec![field_a, field_b, field_c])); + #[test] + fn stats_columns_with_column_mapping_uses_physical_names() { + let schema = schema_ref! { + (cm_field("col_a", 1, "phys_a", DataType::LONG)), + (cm_field("col_b", 2, "phys_b", DataType::LONG)), + (cm_field("col_c", 3, "phys_c", DataType::LONG)), + }; let mut props = HashMap::new(); props.insert("delta.columnMapping.mode".to_string(), "name".to_string()); // Request col_a via stats_columns (logical), and reference col_b via predicate (logical). // Both must be translated to physical names in the output stats schema. - let predicate = Arc::new(column_expr!("col_b").gt(Expr::literal(5i64))); + let predicate = Arc::new(col!("col_b").gt(lit(5i64))); let state_info = get_state_info_with_stats( schema, @@ -1095,7 +1268,10 @@ pub(crate) mod tests { &[], props, vec![], - StatsOutputMode::Columns(vec![column_name!("col_a")]), + StatsOptions { + synthesize_json: true, + struct_stats: StructStats::Columns(vec![column_name!("col_a")]), + }, ) .unwrap(); @@ -1105,7 +1281,7 @@ pub(crate) mod tests { let present = ["phys_a", "phys_b"]; let absent = ["col_a", "col_b", "phys_c"]; - for stats_field in ["minValues", "maxValues"] { + for stats_field in [MIN_VALUES, MAX_VALUES] { let DataType::Struct(inner) = stats_schema .field(stats_field) .unwrap_or_else(|| panic!("should have {stats_field}")) @@ -1127,4 +1303,299 @@ pub(crate) mod tests { } } } + + // === physical_stats_columns trims the predicate-derived stats schema === + + /// Flat schema with `n` long columns named `c0..c{n-1}`. + fn flat_long_schema(n: usize) -> SchemaRef { + Arc::new(StructType::new_unchecked( + (0..n) + .map(|i| StructField::nullable(format!("c{i}"), DataType::LONG)) + .collect::>(), + )) + } + + /// `delta.dataSkippingNumIndexedCols=` configuration map. + fn num_indexed_cols_config(n: i32) -> HashMap { + let mut m = HashMap::new(); + m.insert( + "delta.dataSkippingNumIndexedCols".to_string(), + n.to_string(), + ); + m + } + + /// `delta.dataSkippingStatsColumns=` configuration map. + fn stats_columns_config(cols: &[&str]) -> HashMap { + let mut m = HashMap::new(); + m.insert("delta.dataSkippingStatsColumns".to_string(), cols.join(",")); + m + } + + /// Both `delta.dataSkippingStatsColumns` and `delta.dataSkippingNumIndexedCols` set + /// simultaneously. Per the Delta protocol, the explicit list takes precedence over the + /// cap; this helper exists for tests that exercise that precedence. + fn both_configs(stats_cols: &[&str], num_indexed: i32) -> HashMap { + let mut m = stats_columns_config(stats_cols); + m.insert( + "delta.dataSkippingNumIndexedCols".to_string(), + num_indexed.to_string(), + ); + m + } + + /// `delta.dataSkippingNumIndexedCols` caps the `physical_stats_columns` set to the + /// first N leaves. + #[test] + fn stats_columns_honors_num_indexed_cols() { + let schema = flat_long_schema(5); + let state_info = get_state_info( + schema, + vec![], + None, // no predicate; just check the cached set + &[], + num_indexed_cols_config(2), + vec![], + ) + .unwrap(); + let cols = HashSet::from_iter([column_name!("c0"), column_name!("c1")]); + assert_eq!(state_info.physical_stats_columns, cols); + } + + /// Predicate on a past-cap column: stats schema goes to `None` (no skipping), but + /// the physical predicate is retained so engines can still apply it per-row. + #[test] + fn predicate_on_past_cap_column_drops_stats_schema() { + let schema = flat_long_schema(5); + let predicate = Arc::new(col!("c4").gt(lit(10i64))); + let state_info = get_state_info( + schema, + vec![], + Some(predicate), + &[], + num_indexed_cols_config(2), + vec![], + ) + .unwrap(); + assert!( + state_info.physical_stats_schema.is_none(), + "Predicate on a past-cap column should produce no stats schema, got {:?}", + state_info.physical_stats_schema + ); + assert!( + matches!(state_info.physical_predicate, PhysicalPredicate::Some(_, _)), + "User predicate must be retained even when stats schema is dropped" + ); + } + + /// Indexed AND past-cap: indexed leaf survives in the stats schema, past-cap leaf + /// is dropped. + #[test] + fn predicate_on_mixed_indexed_and_past_cap_keeps_indexed_only() { + let schema = flat_long_schema(5); + let predicate = Arc::new(Pred::and( + col!("c0").gt(lit(10i64)), + col!("c4").gt(lit(10i64)), + )); + let state_info = get_state_info( + schema, + vec![], + Some(predicate), + &[], + num_indexed_cols_config(2), + vec![], + ) + .unwrap(); + let stats_schema = state_info + .physical_stats_schema + .as_ref() + .expect("should have stats schema (indexed arm survives)"); + for stats_field in [MIN_VALUES, MAX_VALUES] { + let DataType::Struct(inner) = stats_schema + .field(stats_field) + .unwrap_or_else(|| panic!("should have {stats_field}")) + .data_type() + else { + panic!("{stats_field} should be a struct"); + }; + assert!( + inner.field("c0").is_some(), + "{stats_field} should contain c0 (indexed)" + ); + assert!( + inner.field("c4").is_none(), + "{stats_field} should NOT contain c4 (past cap)" + ); + } + } + + /// `numIndexedCols=2` against `{ a, b, s: { c, d } }` keeps `a, b` and drops the + /// entire `s` struct, so a predicate on `s.c` produces no stats schema. + #[test] + fn predicate_on_nested_past_cap_leaf_drops_parent_struct() { + let schema = Arc::new(StructType::new_unchecked(vec![ + StructField::nullable("a", DataType::LONG), + StructField::nullable("b", DataType::LONG), + StructField::nullable( + "s", + StructType::new_unchecked(vec![ + StructField::nullable("c", DataType::LONG), + StructField::nullable("d", DataType::LONG), + ]), + ), + ])); + // Predicate only on the past-cap leaf -> stats schema goes empty -> None. + let predicate = Arc::new(col!("s.c").gt(lit(10i64))); + let state_info = get_state_info( + schema, + vec![], + Some(predicate), + &[], + num_indexed_cols_config(2), + vec![], + ) + .unwrap(); + assert!(state_info.physical_stats_schema.is_none()); + assert!(!state_info.physical_stats_columns.is_empty()); + assert!(!state_info + .physical_stats_columns + .contains(&column_name!("s.c"))); + } + + /// `delta.dataSkippingStatsColumns` selects exactly the listed leaves, regardless of + /// their position relative to the (default) cap. + #[rstest] + #[case::single(&["c2"], &["c2"])] + #[case::sparse_subset(&["c0", "c3"], &["c0", "c3"])] + #[case::all_listed(&["c0", "c1", "c2", "c3", "c4"], &["c0", "c1", "c2", "c3", "c4"])] + fn stats_columns_honors_explicit_stats_columns( + #[case] listed: &[&str], + #[case] expected: &[&str], + ) { + let schema = flat_long_schema(5); + let state_info = get_state_info( + schema, + vec![], + None, + &[], + stats_columns_config(listed), + vec![], + ) + .unwrap(); + let expected_cols: HashSet = + expected.iter().map(|s| ColumnName::new([*s])).collect(); + assert_eq!(state_info.physical_stats_columns, expected_cols); + } + + /// `numIndexedCols=3` against `{ a, b, s: { c, d } }` keeps `a, b, s.c` and drops + /// `s.d`. A predicate on both `s.c` and `s.d` keeps the `s` struct under + /// `minValues` / `maxValues` with `c` only. + #[test] + fn predicate_on_nested_mixed_keeps_intersection_under_parent_struct() { + let schema = Arc::new(StructType::new_unchecked(vec![ + StructField::nullable("a", DataType::LONG), + StructField::nullable("b", DataType::LONG), + StructField::nullable( + "s", + StructType::new_unchecked(vec![ + StructField::nullable("c", DataType::LONG), + StructField::nullable("d", DataType::LONG), + ]), + ), + ])); + let predicate = Arc::new(Pred::and( + col!("s.c").gt(lit(10i64)), + col!("s.d").gt(lit(10i64)), + )); + let state_info = get_state_info( + schema, + vec![], + Some(predicate), + &[], + num_indexed_cols_config(3), + vec![], + ) + .unwrap(); + let stats_schema = state_info + .physical_stats_schema + .as_ref() + .expect("indexed arm survives"); + for stats_field in [MIN_VALUES, MAX_VALUES] { + let DataType::Struct(outer) = stats_schema + .field(stats_field) + .unwrap_or_else(|| panic!("should have {stats_field}")) + .data_type() + else { + panic!("{stats_field} should be a struct"); + }; + let DataType::Struct(inner) = outer + .field("s") + .unwrap_or_else(|| panic!("{stats_field} should have s")) + .data_type() + else { + panic!("{stats_field}.s should be a struct"); + }; + assert!( + inner.field("c").is_some(), + "{stats_field}.s should keep c (indexed)" + ); + assert!( + inner.field("d").is_none(), + "{stats_field}.s should drop d (past cap)" + ); + } + } + + /// `dataSkippingStatsColumns` with a parent struct path admits every leaf under that + /// parent (the trie matches by prefix). `{ a, s: { c, d } }` with the property set to + /// `"s"` should produce `{ s.c, s.d }` (and exclude `a`, which is not in the list). + #[test] + fn stats_columns_admits_all_children_of_nested_parent_in_explicit_list() { + let schema = Arc::new(StructType::new_unchecked(vec![ + StructField::nullable("a", DataType::LONG), + StructField::nullable( + "s", + StructType::new_unchecked(vec![ + StructField::nullable("c", DataType::LONG), + StructField::nullable("d", DataType::LONG), + ]), + ), + ])); + let state_info = get_state_info( + schema, + vec![], + None, + &[], + stats_columns_config(&["s"]), + vec![], + ) + .unwrap(); + let expected = HashSet::from_iter([column_name!("s.c"), column_name!("s.d")]); + assert_eq!(state_info.physical_stats_columns, expected); + } + + /// `dataSkippingStatsColumns` ("A") takes precedence over `dataSkippingNumIndexedCols` + /// ("B") whether A wants more columns than B allows or fewer. + #[rstest] + #[case::a_broader_than_b(&["c0", "c3", "c4"], 2, &["c0", "c3", "c4"])] + #[case::a_narrower_than_b(&["c0"], 3, &["c0"])] + fn stats_columns_explicit_list_overrides_num_indexed_cols( + #[case] listed: &[&str], + #[case] num_indexed: i32, + #[case] expected: &[&str], + ) { + let schema = flat_long_schema(5); + let state_info = get_state_info( + schema, + vec![], + None, + &[], + both_configs(listed, num_indexed), + vec![], + ) + .unwrap(); + let expected_cols: HashSet = + expected.iter().map(|s| ColumnName::new([*s])).collect(); + assert_eq!(state_info.physical_stats_columns, expected_cols); + } } diff --git a/kernel/src/scan/test_utils.rs b/kernel/src/scan/test_utils.rs index 2aa50a4984..826355b512 100644 --- a/kernel/src/scan/test_utils.rs +++ b/kernel/src/scan/test_utils.rs @@ -1,27 +1,35 @@ -use crate::arrow::array::StringArray; -use crate::scan::state_info::StateInfo; -use crate::schema::StructType; -use crate::utils::test_utils::string_array_to_engine_data; -use itertools::Itertools; +use std::collections::HashSet; use std::sync::Arc; -use crate::log_replay::ActionsBatch; -use crate::log_segment::CheckpointReadInfo; -use crate::{ - actions::get_commit_schema, - engine::{ - arrow_data::ArrowEngineData, - sync::{json::SyncJsonHandler, SyncEngine}, - }, - scan::log_replay::scan_action_iter, - schema::SchemaRef, - JsonHandler, -}; - use super::state::ScanCallback; use super::PhysicalPredicate; +use crate::actions::get_commit_schema; +use crate::arrow::array::StringArray; +use crate::engine::arrow_data::ArrowEngineData; +use crate::engine::sync::json::SyncJsonHandler; +use crate::engine::sync::SyncEngine; +use crate::log_replay::ActionsBatch; +use crate::log_segment::CheckpointReadInfo; +use crate::scan::log_replay::{scan_action_iter, ScanPartitionValuesOptions, ScanStatsOptions}; +use crate::scan::state_info::StateInfo; use crate::scan::transform_spec::TransformSpec; +use crate::schema::{SchemaRef, StructType}; use crate::table_features::ColumnMappingMode; +use crate::unit_test_utils::string_array_to_engine_data; +use crate::JsonHandler; + +// Preserve schema-projected null columns because batch equality checks include them. +fn parse_batch>( + json: impl IntoIterator, + output_schema: SchemaRef, +) -> Box { + let handler = SyncJsonHandler::new(None); + let json_strings = StringArray::from_iter_values(json); + let parsed = handler + .parse_json(string_array_to_engine_data(json_strings), output_schema) + .unwrap(); + ArrowEngineData::try_from_engine_data(parsed).unwrap() +} // Generates a batch of sidecar actions with the given paths. // The schema is provided as null columns affect equality checks. @@ -40,8 +48,6 @@ pub(crate) fn sidecar_batch_with_given_paths_and_sizes( paths_and_sizes: Vec<(&str, u64)>, output_schema: SchemaRef, ) -> Box { - let handler = SyncJsonHandler {}; - let mut json_strings: Vec = paths_and_sizes .iter() .map(|(path, size)| { @@ -52,80 +58,93 @@ pub(crate) fn sidecar_batch_with_given_paths_and_sizes( .collect(); json_strings.push(r#"{"metaData":{"id":"testId","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"value\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.columnMapping.mode":"none"},"createdTime":1677811175819}}"#.to_string()); - let json_strings_array: StringArray = - json_strings.iter().map(|s| s.as_str()).collect_vec().into(); - - let parsed = handler - .parse_json( - string_array_to_engine_data(json_strings_array), - output_schema, - ) - .unwrap(); - - ArrowEngineData::try_from_engine_data(parsed).unwrap() + parse_batch(json_strings, output_schema) } // Generates a batch with an add action. // The schema is provided as null columns affect equality checks. pub(crate) fn add_batch_simple(output_schema: SchemaRef) -> Box { - let handler = SyncJsonHandler {}; - let json_strings: StringArray = vec![ - r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet","partitionValues": {"date": "2017-12-10"},"size":635,"modificationTime":1677811178336,"dataChange":true,"stats":"{\"numRecords\":10,\"minValues\":{\"value\":0},\"maxValues\":{\"value\":9},\"nullCount\":{\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1677811178336000","MIN_INSERTION_TIME":"1677811178336000","MAX_INSERTION_TIME":"1677811178336000","OPTIMIZE_TARGET_SIZE":"268435456"},"deletionVector":{"storageType":"u","pathOrInlineDv":"vBn[lx{q8@P<9BNH/isA","offset":1,"sizeInBytes":36,"cardinality":2}}}"#, - r#"{"metaData":{"id":"testId","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"value\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.columnMapping.mode":"none"},"createdTime":1677811175819}}"#, - ] - .into(); - let parsed = handler - .parse_json(string_array_to_engine_data(json_strings), output_schema) - .unwrap(); - ArrowEngineData::try_from_engine_data(parsed).unwrap() + parse_batch( + [ + r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet","partitionValues": {"date": "2017-12-10"},"size":635,"modificationTime":1677811178336,"dataChange":true,"stats":"{\"numRecords\":10,\"minValues\":{\"value\":0},\"maxValues\":{\"value\":9},\"nullCount\":{\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1677811178336000","MIN_INSERTION_TIME":"1677811178336000","MAX_INSERTION_TIME":"1677811178336000","OPTIMIZE_TARGET_SIZE":"268435456"},"deletionVector":{"storageType":"u","pathOrInlineDv":"vBn[lx{q8@P<9BNH/isA","offset":1,"sizeInBytes":36,"cardinality":2}}}"#, + r#"{"metaData":{"id":"testId","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"value\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.columnMapping.mode":"none"},"createdTime":1677811175819}}"#, + ], + output_schema, + ) } // Generates a batch with an add action. // The schema is provided as null columns affect equality checks. pub(crate) fn add_batch_for_row_id(output_schema: SchemaRef) -> Box { - let handler = SyncJsonHandler {}; - let json_strings: StringArray = vec![ - r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet","partitionValues": {"date": "2017-12-10"},"size":635,"modificationTime":1677811178336,"dataChange":true,"stats":"{\"numRecords\":10,\"minValues\":{\"value\":0},\"maxValues\":{\"value\":9},\"nullCount\":{\"value\":0},\"tightBounds\":true}","baseRowId": 42, "tags":{"INSERTION_TIME":"1677811178336000","MIN_INSERTION_TIME":"1677811178336000","MAX_INSERTION_TIME":"1677811178336000","OPTIMIZE_TARGET_SIZE":"268435456"},"deletionVector":{"storageType":"u","pathOrInlineDv":"vBn[lx{q8@P<9BNH/isA","offset":1,"sizeInBytes":36,"cardinality":2}}}"#, - r#"{"metaData":{"id":"testId","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"value\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.columnMapping.mode":"none", "delta.enableRowTracking": "true", "delta.rowTracking.materializedRowIdColumnName":"row_id_col"},"createdTime":1677811175819}}"#, - ] - .into(); - let parsed = handler - .parse_json(string_array_to_engine_data(json_strings), output_schema) - .unwrap(); - ArrowEngineData::try_from_engine_data(parsed).unwrap() + parse_batch( + [ + r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet","partitionValues": {"date": "2017-12-10"},"size":635,"modificationTime":1677811178336,"dataChange":true,"stats":"{\"numRecords\":10,\"minValues\":{\"value\":0},\"maxValues\":{\"value\":9},\"nullCount\":{\"value\":0},\"tightBounds\":true}","baseRowId": 42, "tags":{"INSERTION_TIME":"1677811178336000","MIN_INSERTION_TIME":"1677811178336000","MAX_INSERTION_TIME":"1677811178336000","OPTIMIZE_TARGET_SIZE":"268435456"},"deletionVector":{"storageType":"u","pathOrInlineDv":"vBn[lx{q8@P<9BNH/isA","offset":1,"sizeInBytes":36,"cardinality":2}}}"#, + r#"{"metaData":{"id":"testId","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"value\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.columnMapping.mode":"none", "delta.enableRowTracking": "true", "delta.rowTracking.materializedRowIdColumnName":"row_id_col"},"createdTime":1677811175819}}"#, + ], + output_schema, + ) } // An add batch with a removed file parsed with the schema provided pub(crate) fn add_batch_with_remove(output_schema: SchemaRef) -> Box { - let handler = SyncJsonHandler {}; - let json_strings: StringArray = vec![ - r#"{"remove":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c001.snappy.parquet","deletionTimestamp":1677811194426,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{},"size":635,"tags":{"INSERTION_TIME":"1677811178336000","MIN_INSERTION_TIME":"1677811178336000","MAX_INSERTION_TIME":"1677811178336000","OPTIMIZE_TARGET_SIZE":"268435456"}}}"#, - r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c001.snappy.parquet","partitionValues":{},"size":635,"modificationTime":1677811178336,"dataChange":true,"stats":"{\"numRecords\":10,\"minValues\":{\"value\":0},\"maxValues\":{\"value\":9},\"nullCount\":{\"value\":0},\"tightBounds\":false}","tags":{"INSERTION_TIME":"1677811178336000","MIN_INSERTION_TIME":"1677811178336000","MAX_INSERTION_TIME":"1677811178336000","OPTIMIZE_TARGET_SIZE":"268435456"}}}"#, - r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet","partitionValues": {"date": "2017-12-10"},"size":635,"modificationTime":1677811178336,"dataChange":true,"stats":"{\"numRecords\":10,\"minValues\":{\"value\":0},\"maxValues\":{\"value\":9},\"nullCount\":{\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1677811178336000","MIN_INSERTION_TIME":"1677811178336000","MAX_INSERTION_TIME":"1677811178336000","OPTIMIZE_TARGET_SIZE":"268435456"},"deletionVector":{"storageType":"u","pathOrInlineDv":"vBn[lx{q8@P<9BNH/isA","offset":1,"sizeInBytes":36,"cardinality":2}}}"#, - r#"{"metaData":{"id":"testId","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"value\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.columnMapping.mode":"none"},"createdTime":1677811175819}}"#, - ] - .into(); - let parsed = handler - .parse_json(string_array_to_engine_data(json_strings), output_schema) - .unwrap(); - ArrowEngineData::try_from_engine_data(parsed).unwrap() + parse_batch( + [ + r#"{"remove":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c001.snappy.parquet","deletionTimestamp":1677811194426,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{},"size":635,"tags":{"INSERTION_TIME":"1677811178336000","MIN_INSERTION_TIME":"1677811178336000","MAX_INSERTION_TIME":"1677811178336000","OPTIMIZE_TARGET_SIZE":"268435456"}}}"#, + r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c001.snappy.parquet","partitionValues":{},"size":635,"modificationTime":1677811178336,"dataChange":true,"stats":"{\"numRecords\":10,\"minValues\":{\"value\":0},\"maxValues\":{\"value\":9},\"nullCount\":{\"value\":0},\"tightBounds\":false}","tags":{"INSERTION_TIME":"1677811178336000","MIN_INSERTION_TIME":"1677811178336000","MAX_INSERTION_TIME":"1677811178336000","OPTIMIZE_TARGET_SIZE":"268435456"}}}"#, + r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet","partitionValues": {"date": "2017-12-10"},"size":635,"modificationTime":1677811178336,"dataChange":true,"stats":"{\"numRecords\":10,\"minValues\":{\"value\":0},\"maxValues\":{\"value\":9},\"nullCount\":{\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1677811178336000","MIN_INSERTION_TIME":"1677811178336000","MAX_INSERTION_TIME":"1677811178336000","OPTIMIZE_TARGET_SIZE":"268435456"},"deletionVector":{"storageType":"u","pathOrInlineDv":"vBn[lx{q8@P<9BNH/isA","offset":1,"sizeInBytes":36,"cardinality":2}}}"#, + r#"{"metaData":{"id":"testId","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"value\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.columnMapping.mode":"none"},"createdTime":1677811175819}}"#, + ], + output_schema, + ) +} + +pub(crate) fn remove_only_batch(output_schema: SchemaRef) -> Box { + parse_batch( + [ + r#"{"remove":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c001.snappy.parquet","deletionTimestamp":1677811194426,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{},"size":635}}"#, + ], + output_schema, + ) +} + +pub(crate) fn adds_only_batch(output_schema: SchemaRef) -> Box { + parse_batch( + [ + r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet","partitionValues":{},"size":635,"modificationTime":1677811178336,"dataChange":true}}"#, + r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c002.snappy.parquet","partitionValues":{},"size":635,"modificationTime":1677811178336,"dataChange":true}}"#, + ], + output_schema, + ) +} + +// A batch with a Remove action and a partition column (`date`). The Remove has +// `partitionValues: {"date": "2017-12-10"}` but the transform reads from `add.*` columns, +// so the Remove's partition values are not visible to the data skipping filter. +pub(crate) fn add_batch_with_remove_and_partition( + output_schema: SchemaRef, +) -> Box { + parse_batch( + [ + r#"{"remove":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c001.snappy.parquet","deletionTimestamp":1677811194426,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{"date":"2017-12-10"},"size":635}}"#, + r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c001.snappy.parquet","partitionValues":{"date":"2017-12-10"},"size":635,"modificationTime":1677811178336,"dataChange":true,"stats":"{\"numRecords\":10,\"minValues\":{\"value\":0},\"maxValues\":{\"value\":9},\"nullCount\":{\"value\":0}}"}}"#, + r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet","partitionValues":{"date":"2017-12-10"},"size":635,"modificationTime":1677811178336,"dataChange":true,"stats":"{\"numRecords\":10,\"minValues\":{\"value\":0},\"maxValues\":{\"value\":9},\"nullCount\":{\"value\":0}}"}}"#, + r#"{"metaData":{"id":"testId","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"value\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"date\",\"type\":\"date\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["date"],"configuration":{"delta.columnMapping.mode":"none"},"createdTime":1677811175819}}"#, + ], + output_schema, + ) } // add batch with a `date` partition col pub(crate) fn add_batch_with_partition_col() -> Box { - let handler = SyncJsonHandler {}; - let json_strings: StringArray = vec![ - r#"{"metaData":{"id":"testId","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"value\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"date\",\"type\":\"date\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["date"],"configuration":{"delta.enableDeletionVectors":"true","delta.columnMapping.mode":"none"},"createdTime":1677811175819}}"#, - r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c001.snappy.parquet","partitionValues": {"date": "2017-12-11"},"size":635,"modificationTime":1677811178336,"dataChange":true,"stats":"{\"numRecords\":10,\"minValues\":{\"value\":0},\"maxValues\":{\"value\":9},\"nullCount\":{\"value\":0},\"tightBounds\":false}","tags":{"INSERTION_TIME":"1677811178336000","MIN_INSERTION_TIME":"1677811178336000","MAX_INSERTION_TIME":"1677811178336000","OPTIMIZE_TARGET_SIZE":"268435456"}}}"#, - r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#, - r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet","partitionValues": {"date": "2017-12-10"},"size":635,"modificationTime":1677811178336,"dataChange":true,"stats":"{\"numRecords\":10,\"minValues\":{\"value\":0},\"maxValues\":{\"value\":9},\"nullCount\":{\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1677811178336000","MIN_INSERTION_TIME":"1677811178336000","MAX_INSERTION_TIME":"1677811178336000","OPTIMIZE_TARGET_SIZE":"268435456"},"deletionVector":{"storageType":"u","pathOrInlineDv":"vBn[lx{q8@P<9BNH/isA","offset":1,"sizeInBytes":36,"cardinality":2}}}"#, - ] - .into(); - let output_schema = get_commit_schema().clone(); - let parsed = handler - .parse_json(string_array_to_engine_data(json_strings), output_schema) - .unwrap(); - ArrowEngineData::try_from_engine_data(parsed).unwrap() + parse_batch( + [ + r#"{"metaData":{"id":"testId","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"value\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"date\",\"type\":\"date\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["date"],"configuration":{"delta.enableDeletionVectors":"true","delta.columnMapping.mode":"none"},"createdTime":1677811175819}}"#, + r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c001.snappy.parquet","partitionValues": {"date": "2017-12-11"},"size":635,"modificationTime":1677811178336,"dataChange":true,"stats":"{\"numRecords\":10,\"minValues\":{\"value\":0},\"maxValues\":{\"value\":9},\"nullCount\":{\"value\":0},\"tightBounds\":false}","tags":{"INSERTION_TIME":"1677811178336000","MIN_INSERTION_TIME":"1677811178336000","MAX_INSERTION_TIME":"1677811178336000","OPTIMIZE_TARGET_SIZE":"268435456"}}}"#, + r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#, + r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet","partitionValues": {"date": "2017-12-10"},"size":635,"modificationTime":1677811178336,"dataChange":true,"stats":"{\"numRecords\":10,\"minValues\":{\"value\":0},\"maxValues\":{\"value\":9},\"nullCount\":{\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1677811178336000","MIN_INSERTION_TIME":"1677811178336000","MAX_INSERTION_TIME":"1677811178336000","OPTIMIZE_TARGET_SIZE":"268435456"},"deletionVector":{"storageType":"u","pathOrInlineDv":"vBn[lx{q8@P<9BNH/isA","offset":1,"sizeInBytes":36,"cardinality":2}}}"#, + ], + get_commit_schema().clone(), + ) } /// Create a scan action iter and validate what's called back. If you pass `None` as @@ -149,16 +168,20 @@ pub(crate) fn run_with_validate_callback( column_mapping_mode: ColumnMappingMode::None, physical_stats_schema: None, physical_partition_schema: None, + physical_stats_columns: HashSet::new(), + is_catalog_managed: false, + skip_row_transforms: false, }); let checkpoint_info = CheckpointReadInfo::without_stats_parsed(); - let iter = scan_action_iter( + let (iter, _metrics) = scan_action_iter( &SyncEngine::new(), batch .into_iter() .map(|batch| Ok(ActionsBatch::new(batch as _, true))), state_info, checkpoint_info, - false, + ScanStatsOptions::default(), + ScanPartitionValuesOptions::default(), ) .unwrap(); let mut batch_count = 0; diff --git a/kernel/src/scan/tests.rs b/kernel/src/scan/tests.rs index 41f5c1f0e3..ba49ab4f36 100644 --- a/kernel/src/scan/tests.rs +++ b/kernel/src/scan/tests.rs @@ -1,60 +1,74 @@ +use std::fs; use std::path::PathBuf; -use std::sync::Arc; +use std::sync::{Arc, Mutex}; +use ::test_utils::{assert_result_error_with_message, get_column, load_test_data}; use bytes::Bytes; - use rstest::rstest; +use url::Url; +use super::*; +use crate::actions::{MAX_VALUES, MIN_VALUES, NULL_COUNT, NUM_RECORDS, STATS_PARSED}; use crate::arrow::array::{Array, BooleanArray, Int64Array, StringArray, StructArray}; use crate::arrow::compute::filter_record_batch; use crate::arrow::datatypes::{DataType as ArrowDataType, Field, Fields, Schema as ArrowSchema}; use crate::arrow::record_batch::RecordBatch; +use crate::arrow::util::display::array_value_to_string; +use crate::committer::FileSystemCommitter; use crate::engine::arrow_data::ArrowEngineData; use crate::engine::parquet_row_group_skipping::ParquetRowGroupSkipping; use crate::engine::sync::SyncEngine; +use crate::engine::test_delegating::DelegatingEngine; use crate::expressions::{ - column_expr, column_name, column_pred, ColumnName, Expression as Expr, Predicate as Pred, + col, column_name, column_pred, lit, CastOptions, Expression as Expr, Predicate as Pred, }; +use crate::object_store::memory::InMemory; +use crate::object_store::path::Path; +use crate::object_store::ObjectStoreExt as _; use crate::parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder; use crate::parquet::arrow::arrow_writer::ArrowWriter; -use crate::scan::data_skipping::as_checkpoint_skipping_predicate; +use crate::scan::data_skipping::{all_referenced_columns, as_checkpoint_skipping_predicate}; use crate::scan::state::ScanFile; -use crate::schema::{ColumnMetadataKey, DataType, StructField, StructType}; -use crate::{EngineData, Snapshot}; - -use super::*; - -/// Helper macro to extract a typed column from a RecordBatch or StructArray. -macro_rules! get_column { - ($source:expr, $name:expr, $ty:ty) => { - $source - .column_by_name($name) - .unwrap_or_else(|| panic!("should have column '{}'", $name)) - .as_any() - .downcast_ref::<$ty>() - .unwrap_or_else(|| panic!("column '{}' should be {}", $name, stringify!($ty))) - }; -} +use crate::schema::{ + self, schema_ref, ColumnMetadataKey, DataType, MetadataColumnSpec, StructField, StructType, +}; +use crate::transaction::create_table::create_table; +use crate::{ + DeltaResultIteratorStatic, Engine, EngineData, FileDataReadResultIterator, FileMeta, + ParquetFooter, ParquetHandler, PredicateRef, Snapshot, +}; fn field_names(s: &StructArray) -> Vec { s.fields().iter().map(|f| f.name().clone()).collect() } +#[test] +fn partition_values_options_carry_optional_timestamp_timezone() { + let default = PartitionValuesOptions::with_struct(); + assert!(default.parsed_struct); + assert_eq!(default.timestamp_timezone, None); + + let zoned = default.with_timestamp_timezone("America/Los_Angeles"); + assert_eq!( + zoned.timestamp_timezone.as_deref(), + Some("America/Los_Angeles") + ); +} + #[test] fn test_static_skipping() { - const NULL: Pred = Pred::null_literal(); let test_cases = [ (false, column_pred!("a")), - (true, Pred::literal(false)), - (false, Pred::literal(true)), - (true, NULL), - (true, Pred::and(column_pred!("a"), Pred::literal(false))), - (false, Pred::or(column_pred!("a"), Pred::literal(true))), - (false, Pred::or(column_pred!("a"), Pred::literal(false))), - (false, Pred::lt(column_expr!("a"), Expr::literal(10))), + (true, Pred::FALSE), + (false, Pred::TRUE), + (false, Pred::NULL), // NULL is unknown, not false -- conservative (no skip) + (true, Pred::and(column_pred!("a"), Pred::FALSE)), + (false, Pred::or(column_pred!("a"), Pred::TRUE)), + (false, Pred::or(column_pred!("a"), Pred::FALSE)), + (false, Pred::lt(col!("a"), Expr::literal(10))), (false, Pred::lt(Expr::literal(10), Expr::literal(100))), (true, Pred::gt(Expr::literal(10), Expr::literal(100))), - (true, Pred::and(NULL, column_pred!("a"))), + (false, Pred::and(Pred::NULL, column_pred!("a"))), // NULL is unknown, not false ]; for (should_skip, predicate) in test_cases { assert_eq!( @@ -104,8 +118,8 @@ fn test_physical_predicate() { // NOTE: We break several column mapping rules here because they don't matter for this // test. For example, we do not provide field ids, and not all columns have physical names. let test_cases = [ - (Pred::literal(true), Some(PhysicalPredicate::None)), - (Pred::literal(false), Some(PhysicalPredicate::StaticSkipAll)), + (Pred::TRUE, Some(PhysicalPredicate::None)), + (Pred::FALSE, Some(PhysicalPredicate::StaticSkipAll)), (column_pred!("x"), None), // no such column ( column_pred!("a"), @@ -178,9 +192,9 @@ fn test_physical_predicate() { )), ), ( - Pred::and(column_pred!("mapped.n"), Pred::literal(true)), + Pred::and(column_pred!("mapped.n"), Pred::TRUE), Some(PhysicalPredicate::Some( - Pred::and(column_pred!("phys_mapped.phys_n"), Pred::literal(true)).into(), + Pred::and(column_pred!("phys_mapped.phys_n"), Pred::TRUE).into(), StructType::new_unchecked(vec![StructField::nullable( "phys_mapped", StructType::new_unchecked(vec![StructField::nullable( @@ -200,7 +214,7 @@ fn test_physical_predicate() { )), ), ( - Pred::and(column_pred!("mapped.n"), Pred::literal(false)), + Pred::and(column_pred!("mapped.n"), Pred::FALSE), Some(PhysicalPredicate::StaticSkipAll), ), ]; @@ -226,14 +240,14 @@ fn test_physical_predicate() { StructField::nullable("Value", DataType::LONG), ]), Pred::and( - Pred::gt(column_expr!("createdat"), Expr::literal(500i64)), - Pred::lt(column_expr!("value"), Expr::literal(100i64)), + Pred::gt(col!("createdat"), lit(500i64)), + Pred::lt(col!("value"), lit(100i64)), ), ColumnMappingMode::None, PhysicalPredicate::Some( Arc::new(Pred::and( - Pred::gt(column_expr!("createdAt"), Expr::literal(500i64)), - Pred::lt(column_expr!("Value"), Expr::literal(100i64)), + Pred::gt(col!("createdAt"), lit(500i64)), + Pred::lt(col!("Value"), lit(100i64)), )), StructType::new_unchecked(vec![ StructField::nullable("createdAt", DataType::LONG), @@ -254,14 +268,14 @@ fn test_physical_predicate() { )]), ]), Pred::and( - Pred::gt(column_expr!("createdat"), Expr::literal(500i64)), - Pred::lt(column_expr!("value"), Expr::literal(100i64)), + Pred::gt(col!("createdat"), lit(500i64)), + Pred::lt(col!("value"), lit(100i64)), ), ColumnMappingMode::Name, PhysicalPredicate::Some( Arc::new(Pred::and( - Pred::gt(column_expr!("phys_created"), Expr::literal(500i64)), - Pred::lt(column_expr!("phys_value"), Expr::literal(100i64)), + Pred::gt(col!("phys_created"), lit(500i64)), + Pred::lt(col!("phys_value"), lit(100i64)), )), StructType::new_unchecked(vec![ StructField::nullable("phys_created", DataType::LONG).with_metadata([( @@ -281,14 +295,14 @@ fn test_physical_predicate() { StructField::nullable("Value", DataType::LONG), ]), Pred::and( - Pred::gt(column_expr!("value"), Expr::literal(5i64)), - Pred::lt(column_expr!("VALUE"), Expr::literal(10i64)), + Pred::gt(col!("value"), lit(5i64)), + Pred::lt(col!("VALUE"), lit(10i64)), ), ColumnMappingMode::None, PhysicalPredicate::Some( Arc::new(Pred::and( - Pred::gt(column_expr!("Value"), Expr::literal(5i64)), - Pred::lt(column_expr!("Value"), Expr::literal(10i64)), + Pred::gt(col!("Value"), lit(5i64)), + Pred::lt(col!("Value"), lit(10i64)), )), StructType::new_unchecked(vec![StructField::nullable("Value", DataType::LONG)]) .into(), @@ -336,6 +350,264 @@ fn test_physical_predicate_case_insensitive_unknown_column() { assert!(result.is_err()); } +#[test] +fn test_scan_builder_accepts_predicate_on_unprojected_data_column() { + let url = "memory:///test_table/"; + let store = Arc::new(InMemory::new()); + let engine = SyncEngine::new_with_store(store); + + let schema = Arc::new(StructType::new_unchecked([ + StructField::nullable("number", DataType::LONG), + StructField::nullable("a_float", DataType::FLOAT), + ])); + create_table(url, schema, "DefaultEngine") + .build(&engine, Box::new(FileSystemCommitter::new())) + .unwrap() + .commit(&engine) + .unwrap() + .unwrap_committed(); + + let snapshot = Snapshot::builder_for(url::Url::parse(url).unwrap()) + .build(&engine) + .unwrap(); + + let projection = snapshot.schema().project(&["a_float"]).unwrap(); + let predicate = Arc::new(col!("number").gt(lit(5_i64))); + + let scan = snapshot + .scan_builder() + .with_schema(projection) + .with_predicate(predicate) + .build() + .expect("build should accept a predicate referencing a non-projection table column"); + + assert_eq!(scan.logical_schema().fields().len(), 1); +} + +#[test] +fn test_scan_builder_rejects_predicate_on_projection_only_metadata_column() { + let url = "memory:///test_table/"; + let store = Arc::new(InMemory::new()); + let engine = SyncEngine::new_with_store(store); + + let schema = schema_ref! { nullable "id": LONG }; + create_table(url, schema, "DefaultEngine") + .build(&engine, Box::new(FileSystemCommitter::new())) + .unwrap() + .commit(&engine) + .unwrap() + .unwrap_committed(); + + let snapshot = Snapshot::builder_for(url::Url::parse(url).unwrap()) + .build(&engine) + .unwrap(); + + // `my_row_index` is computed during the scan, not stored in the table, + // so a predicate can't filter on it + let projection = Arc::new( + snapshot + .schema() + .add_metadata_column("my_row_index", MetadataColumnSpec::RowIndex) + .unwrap(), + ); + let predicate = Arc::new(col!("my_row_index").gt(lit(5_i64))); + + let err = snapshot + .scan_builder() + .with_schema(projection) + .with_predicate(predicate) + .build() + .expect_err("build should reject predicate referencing a projection-only metadata column"); + let msg = err.to_string(); + assert!( + msg.contains("Predicate references unknown column") && msg.contains("my_row_index"), + "unexpected error: {msg}" + ); +} + +/// Loads `table`'s v0 snapshot with a [`SyncEngine`], for the `without_row_transforms` tests. +fn without_transforms_snapshot(table: &str) -> (Arc, Arc) { + let path = std::fs::canonicalize(PathBuf::from(table)).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + let engine = Arc::new(SyncEngine::new()); + let snapshot = Snapshot::builder_for(url) + .at_version(0) + .build(engine.as_ref()) + .unwrap(); + (engine, snapshot) +} + +/// A partitioned table and a column-mapped table both build a transform spec. +/// `without_row_transforms` retains the spec but sets `skip_row_transforms`, so log replay +/// builds no per-file expressions while the structural transform stays describable. +#[rstest] +#[case::partitioned("./tests/data/basic_partitioned/")] +#[case::column_mapping("./tests/data/partition_cm/name/")] +fn test_without_row_transforms_retains_spec_and_sets_skip(#[case] table: &str) { + let (_engine, snapshot) = without_transforms_snapshot(table); + + let scan = snapshot.clone().scan_builder().build().unwrap(); + assert!(scan.state_info.transform_spec.is_some()); + assert!(!scan.state_info.skip_row_transforms); + + let scan = snapshot + .scan_builder() + .without_row_transforms() + .build() + .unwrap(); + assert!(scan.state_info.transform_spec.is_some()); + assert!(scan.state_info.skip_row_transforms); +} + +/// Under `without_row_transforms`, `scan_metadata` still lists files and surfaces the partition +/// values a connector needs to reconstruct rows itself, while every per-file transform is `None`. +/// Deletion vectors are covered by the sibling test. +#[test] +fn test_without_row_transforms_scan_metadata_lists_files_with_partition_values() { + let (engine, snapshot) = without_transforms_snapshot("./tests/data/basic_partitioned/"); + let scan = snapshot + .scan_builder() + .without_row_transforms() + .build() + .unwrap(); + + // (saw a file, saw non-empty partition values) + fn collect(acc: &mut (bool, bool), scan_file: ScanFile) { + acc.0 = true; + acc.1 |= !scan_file.partition_values.is_empty(); + } + let mut acc = (false, false); + for metadata in scan.scan_metadata(engine.as_ref()).unwrap() { + let metadata = metadata.unwrap(); + assert!( + metadata.scan_file_transforms.iter().all(Option::is_none), + "every per-file transform must be None under without_row_transforms" + ); + acc = metadata.visit_scan_files(acc, collect).unwrap(); + } + let (saw_file, saw_partition_values) = acc; + assert!( + saw_file, + "scan_metadata should still list files under without_row_transforms" + ); + assert!( + saw_partition_values, + "partition values must still be surfaced so the connector can inject them itself" + ); +} + +/// Column mapping also builds a per-file transform (the physical-to-logical rename). Under +/// `without_row_transforms`, `scan_metadata` still lists a column-mapped table's files with every +/// per-file transform `None`, leaving the rename to the connector. +#[test] +fn test_without_row_transforms_scan_metadata_lists_files_column_mapping() { + let table = "table-with-columnmapping-mode-name"; + let tempdir = load_test_data("tests/golden_data", table).unwrap(); + // Golden tables extract to `/delta/` (with a sibling `expected/`). + let table_path = tempdir.path().join(table).join("delta"); + let url = url::Url::from_directory_path(table_path).unwrap(); + let engine = SyncEngine::new(); + let snapshot = Snapshot::builder_for(url).build(&engine).unwrap(); + let scan = snapshot + .scan_builder() + .without_row_transforms() + .build() + .unwrap(); + + let mut saw_file = false; + for metadata in scan.scan_metadata(&engine).unwrap() { + let metadata = metadata.unwrap(); + assert!( + metadata.scan_file_transforms.iter().all(Option::is_none), + "every per-file transform must be None under without_row_transforms" + ); + saw_file = true; + } + assert!( + saw_file, + "scan_metadata should list the column-mapped table's files under without_row_transforms" + ); +} + +#[test] +fn test_without_row_transforms_rejects_execute() { + let (engine, snapshot) = without_transforms_snapshot("./tests/data/basic_partitioned/"); + let scan = snapshot + .scan_builder() + .without_row_transforms() + .build() + .unwrap(); + let err = scan + .execute(engine) + .err() + .expect("execute must error when row transforms are skipped"); + assert!( + err.to_string().contains("without_row_transforms"), + "unexpected error: {err}" + ); +} + +/// Row commit version metadata columns are unsupported by scans, so requesting one errors at +/// build time regardless of `without_row_transforms`. +#[rstest] +fn test_scan_rejects_row_commit_version(#[values(false, true)] without_row_transforms: bool) { + let (_engine, snapshot) = without_transforms_snapshot("./tests/data/basic_partitioned/"); + let schema = Arc::new( + snapshot + .schema() + .add_metadata_column("rcv", MetadataColumnSpec::RowCommitVersion) + .unwrap(), + ); + let mut builder = snapshot.scan_builder().with_schema(schema); + if without_row_transforms { + builder = builder.without_row_transforms(); + } + let err = builder + .build() + .expect_err("row commit version columns are unsupported by scans"); + assert!( + err.to_string() + .contains("Row commit versions not supported"), + "unexpected error: {err}" + ); +} + +/// Deletion vectors flow through scan metadata independently of the row transform, so they are +/// still surfaced under `without_row_transforms` while every per-file transform is `None`. +#[test] +fn test_without_row_transforms_scan_metadata_surfaces_deletion_vectors() { + // The deletion vector is added at the latest version, so load the full table (not v0). + let path = std::fs::canonicalize(PathBuf::from("./tests/data/table-with-dv-small/")).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + let engine = SyncEngine::new(); + let snapshot = Snapshot::builder_for(url).build(&engine).unwrap(); + let scan = snapshot + .scan_builder() + .without_row_transforms() + .build() + .unwrap(); + + fn dv_callback(seen: &mut bool, scan_file: ScanFile) { + *seen |= scan_file.dv_info.deletion_vector.is_some(); + } + let mut saw_dv = false; + for res in scan.scan_metadata(&engine).unwrap() { + let scan_metadata = res.unwrap(); + assert!( + scan_metadata + .scan_file_transforms + .iter() + .all(Option::is_none), + "every per-file transform must be None under without_row_transforms" + ); + saw_dv = scan_metadata.visit_scan_files(saw_dv, dv_callback).unwrap(); + } + assert!( + saw_dv, + "deletion vector info should still be surfaced under without_row_transforms" + ); +} + fn get_files_for_scan(scan: Scan, engine: &dyn Engine) -> DeltaResult> { let scan_metadata_iter = scan.scan_metadata(engine)?; fn scan_metadata_callback(paths: &mut Vec, scan_file: ScanFile) { @@ -502,12 +774,19 @@ fn test_get_partition_value() { PrimitiveType::Timestamp, Scalar::Timestamp(123456), ), + ( + // RFC 3339 with a non-UTC offset: normalized to UTC (1969-12-31T19:00:00Z) + "1970-01-01T00:00:00+05:00", + PrimitiveType::Timestamp, + Scalar::Timestamp(-18000000000), + ), ]; for (raw, data_type, expected) in &cases { let value = crate::scan::transform_spec::parse_partition_value_raw( Some(&raw.to_string()), &DataType::Primitive(data_type.clone()), + crate::timestamp_timezone::TimestampTimezone::default(), ) .unwrap(); assert_eq!(value, *expected); @@ -524,11 +803,9 @@ fn test_replay_for_scan_metadata() { let scan = snapshot.scan_builder().build().unwrap(); let result = scan.replay_for_scan_metadata(&engine).unwrap(); let data: Vec<_> = result.actions.try_collect().unwrap(); - // No predicate pushdown attempted, because at most one part of a multi-part checkpoint - // could be skipped when looking for adds/removes. - // - // NOTE: Each checkpoint part is a single-row file -- guaranteed to produce one row group. - assert_eq!(data.len(), 5); + // Metadata and protocol parts have an all-null `add.path` column and are skipped. Transaction + // parts omit that column and are retained conservatively. + assert_eq!(data.len(), 3); } #[test] @@ -547,8 +824,8 @@ fn test_data_row_group_skipping() { assert_eq!(data.len(), 1); // Ineffective predicate pushdown attempted, so the one data file should be returned. - let int_col = column_expr!("numeric.ints.int32"); - let value = Expr::literal(1000i32); + let int_col = col!("numeric.ints.int32"); + let value = lit(1000i32); let predicate = Arc::new(int_col.clone().gt(value.clone())); let scan = snapshot .clone() @@ -587,7 +864,7 @@ fn test_missing_column_row_group_skipping() { // // WARNING: https://github.com/delta-io/delta-kernel-rs/issues/434 - This // optimization is currently disabled, so the one data file is still returned. - let predicate = Arc::new(column_expr!("missing").lt(Expr::literal(1000i64))); + let predicate = Arc::new(col!("missing").lt(lit(1000i64))); let scan = snapshot .clone() .scan_builder() @@ -598,7 +875,7 @@ fn test_missing_column_row_group_skipping() { assert_eq!(data.len(), 1); // Predicate over a logically missing column fails the scan - let predicate = Arc::new(column_expr!("numeric.ints.invalid").lt(Expr::literal(1000))); + let predicate = Arc::new(col!("numeric.ints.invalid").lt(lit(1000))); snapshot .scan_builder() .with_predicate(predicate) @@ -659,7 +936,7 @@ fn assert_stats_struct_matches_json( } } -/// Test that `with_stats_columns(vec![])` outputs parsed stats in scan_metadata batches. +/// Test that [`StatsOptions::all`] outputs parsed stats in scan_metadata batches. /// Uses a table with a checkpoint that contains stats_parsed for e2e verification. #[test] fn test_scan_metadata_with_stats_columns() { @@ -672,7 +949,7 @@ fn test_scan_metadata_with_stats_columns() { let scan = snapshot .scan_builder() - .include_all_stats_columns() + .with_stats(StatsOptions::all()) .build() .unwrap(); @@ -711,10 +988,10 @@ fn test_scan_metadata_with_stats_columns() { // Extract stats_parsed struct array let stats_parsed = get_column!(filtered_batch, STATS_PARSED_COL, StructArray); - let num_records = get_column!(stats_parsed, "numRecords", Int64Array); - let min_values = get_column!(stats_parsed, "minValues", StructArray); - let max_values = get_column!(stats_parsed, "maxValues", StructArray); - let null_count = get_column!(stats_parsed, "nullCount", StructArray); + let num_records = get_column!(stats_parsed, NUM_RECORDS, Int64Array); + let min_values = get_column!(stats_parsed, MIN_VALUES, StructArray); + let max_values = get_column!(stats_parsed, MAX_VALUES, StructArray); + let null_count = get_column!(stats_parsed, NULL_COUNT, StructArray); // Extract JSON stats column let stats_json = get_column!(filtered_batch, "stats", StringArray); @@ -729,7 +1006,7 @@ fn test_scan_metadata_with_stats_columns() { serde_json::from_str(stats_json.value(i)).expect("stats JSON should be valid"); // Validate numRecords - if let Some(json_num) = json_stats.get("numRecords").and_then(|v| v.as_i64()) { + if let Some(json_num) = json_stats.get(NUM_RECORDS).and_then(|v| v.as_i64()) { assert_eq!( json_num, num_records.value(i), @@ -738,14 +1015,14 @@ fn test_scan_metadata_with_stats_columns() { } // Validate minValues, maxValues, nullCount - if let Some(obj) = json_stats.get("minValues").and_then(|v| v.as_object()) { - assert_stats_struct_matches_json(min_values, obj, i, "minValues"); + if let Some(obj) = json_stats.get(MIN_VALUES).and_then(|v| v.as_object()) { + assert_stats_struct_matches_json(min_values, obj, i, MIN_VALUES); } - if let Some(obj) = json_stats.get("maxValues").and_then(|v| v.as_object()) { - assert_stats_struct_matches_json(max_values, obj, i, "maxValues"); + if let Some(obj) = json_stats.get(MAX_VALUES).and_then(|v| v.as_object()) { + assert_stats_struct_matches_json(max_values, obj, i, MAX_VALUES); } - if let Some(obj) = json_stats.get("nullCount").and_then(|v| v.as_object()) { - assert_stats_struct_matches_json(null_count, obj, i, "nullCount"); + if let Some(obj) = json_stats.get(NULL_COUNT).and_then(|v| v.as_object()) { + assert_stats_struct_matches_json(null_count, obj, i, NULL_COUNT); } total_num_records += num_records.value(i); @@ -760,142 +1037,6 @@ fn test_scan_metadata_with_stats_columns() { ); } -/// Test that data skipping works correctly with pre-parsed stats from a checkpoint. -/// -/// The parsed-stats test table has a checkpoint at version 3 (containing stats_parsed) and -/// JSON commits at versions 4-5. This test exercises both code paths: -/// - Checkpoint batches: stats_parsed is read directly from the transformed batch -/// - JSON log batches: stats are parsed from JSON via the transform expression -/// -/// Table layout (6 files, each 100 records): -/// File 1: id [1-100], File 2: id [101-200], File 3: id [201-300] -/// File 4: id [301-400], File 5: id [401-500], File 6: id [501-600] -#[test] -fn test_data_skipping_with_parsed_stats() { - let path = std::fs::canonicalize(PathBuf::from("./tests/data/parsed-stats/")).unwrap(); - let url = url::Url::from_directory_path(path).unwrap(); - let engine = Arc::new(SyncEngine::new()); - let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); - - // Predicate: id > 400 should skip files 1-4 (max id: 100, 200, 300, 400) and keep files 5-6 - let predicate = Arc::new(Pred::gt(column_expr!("id"), Expr::literal(400i64))); - let scan = snapshot - .scan_builder() - .with_predicate(predicate) - .build() - .unwrap(); - - let scan_metadata_results: Vec<_> = scan - .scan_metadata(engine.as_ref()) - .unwrap() - .collect::, _>>() - .unwrap(); - - let mut selected_file_count = 0; - for scan_metadata in &scan_metadata_results { - let selection_vector = scan_metadata.scan_files.selection_vector(); - selected_file_count += selection_vector - .iter() - .filter(|&&selected| selected) - .count(); - } - - assert_eq!( - selected_file_count, 2, - "Data skipping with parsed stats should keep only 2 files (id [401-500] and [501-600])" - ); -} - -/// Test that `include_all_stats_columns` and `with_predicate` can be used together. -/// The scan should output stats_parsed AND perform data skipping via the predicate. -#[test] -fn test_scan_metadata_stats_columns_with_predicate() { - const STATS_PARSED_COL: &str = "stats_parsed"; - - let path = std::fs::canonicalize(PathBuf::from("./tests/data/parsed-stats/")).unwrap(); - let url = url::Url::from_directory_path(path).unwrap(); - let engine = Arc::new(SyncEngine::new()); - - let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); - - // Build scan with both a predicate and stats_columns - let predicate = Arc::new(column_expr!("id").gt(Expr::literal(0i64))); - let scan = snapshot - .scan_builder() - .with_predicate(predicate) - .include_all_stats_columns() - .build() - .expect("Should succeed when using both predicate and stats_columns"); - - // Verify the scan has a physical predicate (data skipping is active) - assert!( - scan.physical_predicate().is_some(), - "Scan should have a physical predicate for data skipping" - ); - - // Run scan_metadata and verify stats_parsed is present in the output - let scan_metadata_results: Vec<_> = scan - .scan_metadata(engine.as_ref()) - .unwrap() - .collect::, _>>() - .unwrap(); - - assert!( - !scan_metadata_results.is_empty(), - "Should have scan metadata results" - ); - - let mut file_count = 0; - for scan_metadata in scan_metadata_results { - let (underlying_data, selection_vector) = scan_metadata.scan_files.into_parts(); - let batch: RecordBatch = ArrowEngineData::try_from_engine_data(underlying_data) - .unwrap() - .into(); - let filtered_batch = - filter_record_batch(&batch, &BooleanArray::from(selection_vector)).unwrap(); - - // Verify stats_parsed column exists and is a struct type - let schema = filtered_batch.schema(); - let field = schema - .field_with_name(STATS_PARSED_COL) - .expect("Schema should contain stats_parsed column"); - assert!( - matches!(field.data_type(), ArrowDataType::Struct(_)), - "stats_parsed should be a struct type" - ); - - // Verify stats_parsed has data - let stats_parsed = get_column!(filtered_batch, STATS_PARSED_COL, StructArray); - let num_records = get_column!(stats_parsed, "numRecords", Int64Array); - for i in 0..filtered_batch.num_rows() { - if !stats_parsed.is_null(i) { - assert!(num_records.value(i) > 0, "numRecords should be positive"); - file_count += 1; - } - } - } - - assert!( - file_count > 0, - "Should have processed at least one file with stats" - ); -} - -#[test] -fn test_prefix_columns_simple() { - let mut prefixer = PrefixColumns { - prefix: ColumnName::new(["add", "stats_parsed"]), - }; - // A simple binary predicate: x > 100 - let pred = Pred::gt(column_expr!("x"), Expr::literal(100i64)); - let result = prefixer.transform_pred(&pred).unwrap().into_owned(); - - // The column reference should now be add.stats_parsed.x - let refs: Vec<_> = result.references().into_iter().collect(); - assert_eq!(refs.len(), 1); - assert_eq!(*refs[0], column_name!("add.stats_parsed.x")); -} - #[test] fn test_build_actions_meta_predicate_with_predicate() { let path = std::fs::canonicalize(PathBuf::from("./tests/data/parsed-stats/")).unwrap(); @@ -904,7 +1045,7 @@ fn test_build_actions_meta_predicate_with_predicate() { let snapshot = Snapshot::builder_for(url).build(&engine).unwrap(); // Build a scan with a predicate eligible for data skipping - let predicate = Arc::new(Pred::gt(column_expr!("id"), Expr::literal(400i64))); + let predicate = Arc::new(Pred::gt(col!("id"), lit(400i64))); let scan = snapshot .scan_builder() .with_predicate(predicate) @@ -957,7 +1098,7 @@ fn test_build_actions_meta_predicate_static_skip_all() { // A predicate that statically evaluates to false should produce StaticSkipAll, // which means build_actions_meta_predicate returns None. - let predicate = Arc::new(Pred::literal(false)); + let predicate = Arc::new(Pred::FALSE); let scan = snapshot .scan_builder() .with_predicate(predicate) @@ -970,33 +1111,258 @@ fn test_build_actions_meta_predicate_static_skip_all() { ); } -/// Helper to build a parquet file with the nested `add.stats_parsed.*` structure that -/// checkpoint files have. Returns the parquet bytes and the arrow schema. -/// -/// Each call to `write_row_group` writes one row group. Each row represents one add action's -/// stats with maxValues.id, minValues.id, nullCount.id, and numRecords. -struct CheckpointParquetBuilder { - arrow_schema: Arc, - id_fields: Fields, - stats_fields: Fields, - buffer: Vec, - writer: Option>>, +// Partition-only scans have no stats schema, so the partition schema must enable the rewrite. +#[rstest] +#[case::equality(Pred::eq( + col!("modified"), + lit("2021-02-01"), +), None)] +#[case::date_cast_range(Pred::and( + Pred::ge( + Expr::cast( + col!("modified"), + DataType::DATE, + CastOptions::default(), + ), + Scalar::Date(20_641), + ), + Pred::lt( + Expr::cast( + col!("modified"), + DataType::DATE, + CastOptions::default(), + ), + Scalar::Date(20_644), + ), +), Some("CAST(Column(add.partitionValues_parsed.modified) AS date)"))] +fn test_build_actions_meta_predicate_partition_only( + #[case] predicate: Pred, + #[case] expected_cast: Option<&str>, +) { + // `app-txn-checkpoint` is partitioned by `modified` (string), no column mapping. + let path = std::fs::canonicalize(PathBuf::from("./tests/data/app-txn-checkpoint/")).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + let engine = SyncEngine::new(); + let snapshot = Snapshot::builder_for(url).build(&engine).unwrap(); + + let scan = snapshot + .scan_builder() + .with_predicate(Arc::new(predicate)) + .build() + .unwrap(); + + let meta_pred = scan + .build_actions_meta_predicate() + .expect("partition-only predicate should produce a checkpoint meta-predicate"); + let rendered = meta_pred.to_string(); + if let Some(expected_cast) = expected_cast { + assert!(rendered.contains(expected_cast), "{rendered}"); + } + let refs: Vec = meta_pred + .references() + .into_iter() + .map(|c| c.to_string()) + .collect(); + assert_eq!( + refs, + vec!["add.partitionValues_parsed.modified".to_string()] + ); } -impl CheckpointParquetBuilder { - fn new() -> Self { - let id_fields = Fields::from(vec![Field::new("id", ArrowDataType::Int64, true)]); - let stats_fields = Fields::from(vec![ - Field::new("maxValues", ArrowDataType::Struct(id_fields.clone()), true), - Field::new("minValues", ArrowDataType::Struct(id_fields.clone()), true), - Field::new("nullCount", ArrowDataType::Struct(id_fields.clone()), true), - Field::new("numRecords", ArrowDataType::Int64, true), - ]); - let add_fields = Fields::from(vec![Field::new( - "stats_parsed", - ArrowDataType::Struct(stats_fields.clone()), - true, - )]); +#[test] +fn checkpoint_reader_timezone_omits_only_zoned_timestamp_partition_from_meta_predicate() { + let store = Arc::new(InMemory::new()); + let commit = r#"{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["timestampNtz"],"writerFeatures":["timestampNtz"]}} +{"metaData":{"id":"test","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"part_ts\",\"type\":\"timestamp\",\"nullable\":true,\"metadata\":{}},{\"name\":\"part_ts_ntz\",\"type\":\"timestamp_ntz\",\"nullable\":true,\"metadata\":{}},{\"name\":\"part_int\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["part_ts","part_ts_ntz","part_int"],"configuration":{},"createdTime":0}}"#; + futures::executor::block_on(store.put( + &Path::from("_delta_log/00000000000000000000.json"), + commit.into(), + )) + .unwrap(); + let engine = SyncEngine::new_with_store(store); + let snapshot = Snapshot::builder_for("memory:///").build(&engine).unwrap(); + let predicate = Arc::new(Pred::and( + Pred::gt(col!("part_ts"), Scalar::Timestamp(0)), + Pred::and( + Pred::gt(col!("part_ts_ntz"), Scalar::TimestampNtz(0)), + Pred::gt(col!("part_int"), lit(0)), + ), + )); + + let zoned_scan = snapshot + .clone() + .scan_builder() + .with_predicate(predicate.clone()) + .with_partition_values( + PartitionValuesOptions::with_struct().with_timestamp_timezone("America/Los_Angeles"), + ) + .build() + .unwrap(); + let zoned_refs: HashSet<_> = zoned_scan + .build_actions_meta_predicate() + .expect("safe partition leaves should retain a checkpoint predicate") + .references() + .into_iter() + .map(ToString::to_string) + .collect(); + assert_eq!( + zoned_refs, + HashSet::from([ + "add.partitionValues_parsed.part_int".to_string(), + "add.partitionValues_parsed.part_ts_ntz".to_string(), + ]) + ); + + let utc_scan = snapshot + .scan_builder() + .with_predicate(predicate) + .with_partition_values(PartitionValuesOptions::with_struct()) + .build() + .unwrap(); + let utc_refs: HashSet<_> = utc_scan + .build_actions_meta_predicate() + .expect("all UTC partition leaves should retain a checkpoint predicate") + .references() + .into_iter() + .map(ToString::to_string) + .collect(); + assert_eq!( + utc_refs, + HashSet::from([ + "add.partitionValues_parsed.part_int".to_string(), + "add.partitionValues_parsed.part_ts".to_string(), + "add.partitionValues_parsed.part_ts_ntz".to_string(), + ]) + ); +} + +// Under column mapping, `partitionValues_parsed` is keyed by the physical partition name, so the +// checkpoint meta-predicate must reference that physical name (not the logical one) in both name +// and id mapping modes. +#[rstest] +#[case::name_mode("name")] +#[case::id_mode("id")] +fn test_build_actions_meta_predicate_partition_column_mapping( + #[case] mode: &str, + #[values(false, true)] casted: bool, +) { + // `partition_cm/{name,id}` use column mapping, partitioned by `category` + // (physical name col-6dc68f07-711d-4f00-8bd6-1f5bc698e8ad in both fixtures). + let path = + std::fs::canonicalize(PathBuf::from(format!("./tests/data/partition_cm/{mode}/"))).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + let engine = SyncEngine::new(); + let snapshot = Snapshot::builder_for(url).build(&engine).unwrap(); + + let predicate = if casted { + Pred::eq( + Expr::cast(col!("category"), DataType::DATE, CastOptions::default()), + Scalar::Date(20_641), + ) + } else { + Pred::eq(col!("category"), lit("a")) + }; + let scan = snapshot + .scan_builder() + .with_predicate(Arc::new(predicate)) + .build() + .unwrap(); + + let meta_pred = scan + .build_actions_meta_predicate() + .expect("partition predicate under column mapping should produce a meta-predicate"); + let rendered = meta_pred.to_string(); + assert_eq!(rendered.contains("CAST("), casted, "{rendered}"); + let refs: Vec = meta_pred + .references() + .into_iter() + .map(|c| c.to_string()) + .collect(); + // The hyphenated physical name is backtick-quoted in the column display. + assert!( + refs.iter().any(|r| r.contains("partitionValues_parsed") + && r.contains("col-6dc68f07-711d-4f00-8bd6-1f5bc698e8ad")), + "expected a reference to the PHYSICAL partition column under partitionValues_parsed, \ + got {refs:?}" + ); +} + +struct RgSpec { + id: i64, + x_min: i64, + x_max: i64, + part: Option<&'static str>, +} + +struct CheckpointRowGroup { + id_max: Vec>, + id_min: Vec>, + id_null_counts: Vec>, + x_max: Vec>, + x_min: Vec>, + x_null_counts: Vec>, + num_records: Vec, + parts: Vec>, +} + +/// Builds checkpoint parquet with one row group per write. +struct CheckpointParquetBuilder { + arrow_schema: Arc, + stat_value_fields: Fields, + stats_fields: Fields, + partition_fields: Fields, + add_fields: Fields, + include_partition_values: bool, + buffer: Vec, + writer: Option>>, +} + +impl CheckpointParquetBuilder { + fn new() -> Self { + Self::with_partition_values(true) + } + + fn without_partition_values() -> Self { + Self::with_partition_values(false) + } + + fn with_partition_values(include_partition_values: bool) -> Self { + let stat_value_fields = Fields::from(vec![ + Field::new("id", ArrowDataType::Int64, true), + Field::new("x", ArrowDataType::Int64, true), + ]); + let stats_fields = Fields::from(vec![ + Field::new( + MAX_VALUES, + ArrowDataType::Struct(stat_value_fields.clone()), + true, + ), + Field::new( + MIN_VALUES, + ArrowDataType::Struct(stat_value_fields.clone()), + true, + ), + Field::new( + NULL_COUNT, + ArrowDataType::Struct(stat_value_fields.clone()), + true, + ), + Field::new(NUM_RECORDS, ArrowDataType::Int64, true), + ]); + let partition_fields = Fields::from(vec![Field::new("part", ArrowDataType::Utf8, true)]); + let mut add_fields = vec![Field::new( + "stats_parsed", + ArrowDataType::Struct(stats_fields.clone()), + true, + )]; + if include_partition_values { + add_fields.push(Field::new( + "partitionValues_parsed", + ArrowDataType::Struct(partition_fields.clone()), + true, + )); + } + let add_fields = Fields::from(add_fields); let arrow_schema = Arc::new(ArrowSchema::new(vec![Field::new( "add", ArrowDataType::Struct(add_fields.clone()), @@ -1004,11 +1370,13 @@ impl CheckpointParquetBuilder { )])); let buffer = Vec::new(); let writer = ArrowWriter::try_new(buffer, arrow_schema.clone(), None).unwrap(); - // ArrowWriter takes ownership of buffer, so we use a placeholder here. Self { arrow_schema, - id_fields, + stat_value_fields, stats_fields, + partition_fields, + add_fields, + include_partition_values, buffer: Vec::new(), writer: Some(writer), } @@ -1022,50 +1390,83 @@ impl CheckpointParquetBuilder { null_counts: &[Option], num_records: &[i64], ) { - let make_id_struct = |vals: &[Option]| -> StructArray { - StructArray::from(vec![( - Arc::new(Field::new("id", ArrowDataType::Int64, true)), - Arc::new(Int64Array::from(vals.to_vec())) as Arc, - )]) + let len = max_ids.len(); + self.write_values(CheckpointRowGroup { + id_max: max_ids.to_vec(), + id_min: min_ids.to_vec(), + id_null_counts: null_counts.to_vec(), + x_max: vec![None; len], + x_min: vec![None; len], + x_null_counts: vec![None; len], + num_records: num_records.to_vec(), + parts: vec![None; len], + }); + } + + fn write_rg_group(&mut self, group: &[RgSpec]) { + let ids: Vec> = group.iter().map(|spec| Some(spec.id)).collect(); + let zeros = vec![Some(0); group.len()]; + self.write_values(CheckpointRowGroup { + id_max: ids.clone(), + id_min: ids, + id_null_counts: zeros.clone(), + x_max: group.iter().map(|spec| Some(spec.x_max)).collect(), + x_min: group.iter().map(|spec| Some(spec.x_min)).collect(), + x_null_counts: zeros, + num_records: vec![1; group.len()], + parts: group.iter().map(|spec| spec.part).collect(), + }); + } + + fn write_values(&mut self, values: CheckpointRowGroup) { + let make_stat_struct = |ids: Vec>, xs: Vec>| { + StructArray::from(vec![ + ( + self.stat_value_fields[0].clone(), + Arc::new(Int64Array::from(ids)) as Arc, + ), + ( + self.stat_value_fields[1].clone(), + Arc::new(Int64Array::from(xs)) as Arc, + ), + ]) }; let stats_parsed = StructArray::from(vec![ ( - Arc::new(Field::new( - "maxValues", - ArrowDataType::Struct(self.id_fields.clone()), - true, - )), - Arc::new(make_id_struct(max_ids)) as Arc, + self.stats_fields[0].clone(), + Arc::new(make_stat_struct(values.id_max, values.x_max)) as Arc, ), ( - Arc::new(Field::new( - "minValues", - ArrowDataType::Struct(self.id_fields.clone()), - true, - )), - Arc::new(make_id_struct(min_ids)) as Arc, + self.stats_fields[1].clone(), + Arc::new(make_stat_struct(values.id_min, values.x_min)) as Arc, ), ( - Arc::new(Field::new( - "nullCount", - ArrowDataType::Struct(self.id_fields.clone()), - true, - )), - Arc::new(make_id_struct(null_counts)) as Arc, + self.stats_fields[2].clone(), + Arc::new(make_stat_struct( + values.id_null_counts, + values.x_null_counts, + )) as Arc, ), ( - Arc::new(Field::new("numRecords", ArrowDataType::Int64, true)), - Arc::new(Int64Array::from(num_records.to_vec())) as Arc, + self.stats_fields[3].clone(), + Arc::new(Int64Array::from(values.num_records)) as Arc, ), ]); - let add = StructArray::from(vec![( - Arc::new(Field::new( - "stats_parsed", - ArrowDataType::Struct(self.stats_fields.clone()), - true, - )), + let mut add_children = vec![( + self.add_fields[0].clone(), Arc::new(stats_parsed) as Arc, - )]); + )]; + if self.include_partition_values { + let partition_values = StructArray::from(vec![( + self.partition_fields[0].clone(), + Arc::new(StringArray::from(values.parts)) as Arc, + )]); + add_children.push(( + self.add_fields[1].clone(), + Arc::new(partition_values) as Arc, + )); + } + let add = StructArray::from(add_children); let batch = RecordBatch::try_new(self.arrow_schema.clone(), vec![Arc::new(add)]).unwrap(); let writer = self.writer.as_mut().unwrap(); writer.write(&batch).unwrap(); @@ -1080,18 +1481,39 @@ impl CheckpointParquetBuilder { } } -/// Builds a checkpoint skipping predicate and prefixes column references with `add.stats_parsed`. -fn build_prefixed_checkpoint_predicate(pred: &Pred) -> Option { - let skipping_pred = as_checkpoint_skipping_predicate(pred, &[])?; +fn rg(id: i64, x_min: i64, x_max: i64, part: Option<&'static str>) -> RgSpec { + RgSpec { + id, + x_min, + x_max, + part, + } +} + +fn write_checkpoint(groups: &[&[RgSpec]], include_partition_values: bool) -> Bytes { + let mut builder = if include_partition_values { + CheckpointParquetBuilder::new() + } else { + CheckpointParquetBuilder::without_partition_values() + }; + for group in groups { + builder.write_rg_group(group); + } + builder.finish() +} + +/// Builds a checkpoint meta-predicate scoped under `add`, mirroring `build_actions_meta_predicate`. +fn build_checkpoint_meta_predicate( + pred: &Pred, + partition_columns: &HashSet, + stats_columns: &HashSet, +) -> Option { + let skipping_pred = + as_checkpoint_skipping_predicate(pred, partition_columns, &HashSet::new(), stats_columns)?; let mut prefixer = PrefixColumns { - prefix: ColumnName::new(["add", "stats_parsed"]), + prefix: column_name!("add"), }; - Some( - prefixer - .transform_pred(&skipping_pred) - .unwrap() - .into_owned(), - ) + Some(prefixer.transform_pred(&skipping_pred).into_owned()) } /// Applies a meta predicate as a row group filter and returns the total rows read. @@ -1120,21 +1542,19 @@ fn apply_row_group_filter(parquet_bytes: Bytes, meta_predicate: &Pred) -> usize /// | id IS NOT NULL | no predicate (col vs col, #1873) | 6 | #[rstest] #[case::comparison( - Pred::gt(column_expr!("id"), Expr::literal(200i64)), - // Should skip RG2 and RG3, but https://github.com/apache/arrow-rs/issues/9451 - Some(6), // Some(3), + Pred::gt(col!("id"), lit(200i64)), + Some(3), "keep RG 0 (null stats) + RG 1 (max>200), skip RG 2 + RG 3 (max<200)" )] #[case::is_null( - Pred::is_null(column_expr!("id")), - // Should skip RG 1 (nullCount=0), but https://github.com/apache/arrow-rs/issues/9451 - Some(6), // Some(5), + Pred::is_null(col!("id")), + Some(5), "keep RG 0 (nullCount>0) + RG 2 (nullCount>0) + RG 3 (null nullCount), skip RG 1 (nullCount=0)" )] #[case::is_not_null( - Pred::not(Pred::is_null(column_expr!("id"))), + Pred::not(Pred::is_null(col!("id"))), None, - "IS NOT NULL produces no skipping predicate — column vs column (#1873)" + "IS NOT NULL produces no skipping predicate (column vs column, #1873)" )] fn test_checkpoint_row_group_skipping( #[case] pred: Pred, @@ -1163,7 +1583,8 @@ fn test_checkpoint_row_group_skipping( ); let parquet_bytes = builder.finish(); - let meta_predicate = build_prefixed_checkpoint_predicate(&pred); + let meta_predicate = + build_checkpoint_meta_predicate(&pred, &HashSet::new(), &all_referenced_columns(&pred)); match expected_rows { Some(expected) => { @@ -1186,58 +1607,414 @@ fn test_checkpoint_row_group_skipping( } } -#[test] -fn test_skip_stats_disables_data_skipping() { - let path = std::fs::canonicalize(PathBuf::from("./tests/data/parsed-stats/")).unwrap(); - let url = url::Url::from_directory_path(path).unwrap(); - let engine = Arc::new(SyncEngine::new()); - let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); +fn surviving_ids(parquet_bytes: Bytes, pred: &Pred) -> Vec { + let meta_predicate = build_checkpoint_meta_predicate( + pred, + &HashSet::from([column_name!("part")]), + &HashSet::from([column_name!("x")]), + ); + let mut builder = ParquetRecordBatchReaderBuilder::try_new(parquet_bytes).unwrap(); + if let Some(meta_predicate) = &meta_predicate { + builder = builder.with_row_group_filter(meta_predicate, None); + } + let mut ids: Vec = builder + .build() + .unwrap() + .map(Result::unwrap) + .flat_map(|batch| { + let add = batch + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + let stats = struct_field(add, "stats_parsed"); + let min = struct_field(stats, MIN_VALUES); + min.column_by_name("id") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap() + .iter() + .flatten() + .collect::>() + }) + .collect(); + ids.sort_unstable(); + ids +} + +fn struct_field<'a>(parent: &'a StructArray, name: &str) -> &'a StructArray { + parent + .column_by_name(name) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap() +} + +fn standard_multi_rg() -> Bytes { + write_checkpoint( + &[ + &[rg(1, 0, 10, Some("a"))], + &[rg(2, 100, 110, Some("b"))], + &[rg(3, 200, 210, Some("a"))], + &[rg(4, 900, 910, Some("c"))], + ], + true, + ) +} + +#[rstest] +#[case::stats_gt(Pred::gt(col!("x"), lit(150i64)), vec![3, 4])] +#[case::stats_le(Pred::le(col!("x"), lit(110i64)), vec![1, 2])] +#[case::stats_all_kept( + Pred::ge(col!("x"), lit(0i64)), + vec![1, 2, 3, 4] +)] +#[case::partition_eq(Pred::eq(col!("part"), lit("a")), vec![1, 3])] +#[case::partition_lt(Pred::lt(col!("part"), lit("b")), vec![1, 3])] +#[case::partition_all_pruned(Pred::eq(col!("part"), lit("z")), vec![])] +#[case::partition_cast_kept( + Pred::eq( + Expr::cast(col!("part"), DataType::DATE, CastOptions::default()), + Scalar::Date(18_628), + ), + vec![1, 2, 3, 4] +)] +#[case::partition_cast_and_stats( + Pred::and( + Pred::eq( + Expr::cast(col!("part"), DataType::DATE, CastOptions::default()), + Scalar::Date(18_628), + ), + Pred::gt(col!("x"), lit(150i64)), + ), + vec![3, 4] +)] +#[case::and_stats_and_partition( + Pred::and( + Pred::eq(col!("part"), lit("a")), + Pred::gt(col!("x"), lit(150i64)), + ), + vec![3] +)] +#[case::or_stats_or_partition( + Pred::or( + Pred::eq(col!("part"), lit("c")), + Pred::gt(col!("x"), lit(150i64)), + ), + vec![3, 4] +)] +#[case::partition_is_null(Pred::is_null(col!("part")), vec![])] +#[case::partition_is_not_null( + Pred::is_not_null(col!("part")), + vec![1, 2, 3, 4] +)] +fn test_checkpoint_reader_skips_expected_row_groups( + #[case] pred: Pred, + #[case] expected: Vec, +) { + assert_eq!( + surviving_ids(standard_multi_rg(), &pred), + expected, + "{pred:?}" + ); +} + +#[rstest] +#[case::is_null_keeps_only_null_group( + &[&[rg(1, 0, 10, Some("a"))] as &[RgSpec], &[rg(2, 100, 110, None)], &[rg(3, 200, 210, Some("c"))]], + Pred::is_null(col!("part")), + vec![2], +)] +// Footer min/max ignore the null-valued Add, so the non-matching range prunes the group. +#[case::mixed_group_with_null_partition_pruned( + &[&[rg(1, 0, 10, Some("a")), rg(2, 100, 110, None)] as &[RgSpec]], + Pred::eq(col!("part"), lit("z")), + vec![], +)] +#[case::partition_range_contains_target( + &[&[rg(1, 0, 10, Some("a")), rg(2, 100, 110, Some("c"))] as &[RgSpec]], + Pred::eq(col!("part"), lit("b")), + vec![1, 2], +)] +#[case::all_null_group_pruned_under_eq( + &[&[rg(1, 0, 10, Some("a"))] as &[RgSpec], &[rg(2, 100, 110, None), rg(3, 200, 210, None)]], + Pred::eq(col!("part"), lit("z")), + vec![], +)] +#[case::all_null_group_pruned_under_gt( + &[&[rg(1, 0, 10, Some("a"))] as &[RgSpec], &[rg(2, 100, 110, None), rg(3, 200, 210, None)]], + Pred::gt(col!("part"), lit("m")), + vec![], +)] +#[case::all_null_group_pruned_under_is_not_null( + &[&[rg(1, 0, 10, Some("a"))] as &[RgSpec], &[rg(2, 100, 110, None), rg(3, 200, 210, None)]], + Pred::is_not_null(col!("part")), + vec![1], +)] +fn test_checkpoint_reader_handles_partition_groups( + #[case] groups: &[&[RgSpec]], + #[case] pred: Pred, + #[case] expected: Vec, +) { + let parquet_bytes = write_checkpoint(groups, true); + assert_eq!(surviving_ids(parquet_bytes, &pred), expected, "{pred:?}"); +} + +/// A checkpoint may omit `partitionValues_parsed`; the absent leaf remains non-pruning. +#[rstest] +#[case::comparison(Pred::eq(col!("part"), lit("z")))] +#[case::is_not_null(Pred::is_not_null(col!("part")))] +fn test_checkpoint_reader_keeps_missing_partition_column(#[case] pred: Pred) { + let parquet_bytes = write_checkpoint(&[&[rg(1, 0, 10, Some("a"))] as &[RgSpec]], false); + assert_eq!(surviving_ids(parquet_bytes, &pred), vec![1]); +} + +#[derive(Debug)] +struct RecordedParquetRead { + files: Vec, + physical_schema: schema::SchemaRef, + predicate: Option, +} + +struct RecordingParquetHandler { + inner: Arc, + reads: Mutex>, +} + +impl RecordingParquetHandler { + fn new(inner: Arc) -> Self { + Self { + inner, + reads: Mutex::new(Vec::new()), + } + } - let predicate = Arc::new(Pred::gt(column_expr!("id"), Expr::literal(400i64))); + fn take_reads(&self) -> Vec { + std::mem::take(&mut *self.reads.lock().unwrap()) + } +} + +impl ParquetHandler for RecordingParquetHandler { + fn read_parquet_files( + &self, + files: &[FileMeta], + physical_schema: schema::SchemaRef, + predicate: Option, + ) -> DeltaResult { + self.reads.lock().unwrap().push(RecordedParquetRead { + files: files.iter().map(|file| file.location.to_string()).collect(), + physical_schema: physical_schema.clone(), + predicate: predicate.clone(), + }); + self.inner + .read_parquet_files(files, physical_schema, predicate) + } + + fn read_parquet_footer(&self, file: &FileMeta) -> DeltaResult { + self.inner.read_parquet_footer(file) + } + + fn write_parquet_file( + &self, + location: url::Url, + data: DeltaResultIteratorStatic>, + ) -> DeltaResult<()> { + self.inner.write_parquet_file(location, data) + } +} + +#[rstest] +#[case::all_struct(StatsOptions::all_struct(), false, false)] +#[case::all(StatsOptions::all(), true, false)] +#[case::none_with_predicate(StatsOptions::none(), false, true)] +fn test_checkpoint_stats_projection_matches_requested_output( + #[values( + "v1-single-part-struct-stats-only", + "v2-parquet-sidecars-struct-stats-only", + "v2-checkpoints-parquet-with-sidecars" + )] + table: &str, + #[case] stats: StatsOptions, + #[case] request_json_stats: bool, + #[case] skip_stats: bool, +) { + let extracted = load_test_data("tests/data", table).ok(); + let path = extracted + .as_ref() + .map(|dir| dir.path().join(table)) + .unwrap_or_else(|| { + fs::canonicalize(PathBuf::from(format!("./tests/data/{table}/"))).unwrap() + }); + let url = Url::from_directory_path(path).unwrap(); + let sync = Arc::new(SyncEngine::new()); + let recorder = Arc::new(RecordingParquetHandler::new(sync.parquet_handler())); + let engine = DelegatingEngine::new(sync).with_parquet_handler(recorder.clone()); + let snapshot = Snapshot::builder_for(url).build(&engine).unwrap(); + recorder.take_reads(); + + let predicate: Option = + skip_stats.then(|| Arc::new(Pred::gt(col!("id"), lit(0i64))) as PredicateRef); let scan = snapshot .scan_builder() .with_predicate(predicate) - .with_skip_stats(true) + .with_stats(stats) .build() .unwrap(); + for action in scan.replay_for_scan_metadata(&engine).unwrap().actions { + action.unwrap(); + } - let scan_metadata_results: Vec<_> = scan + let reads = recorder.take_reads(); + let compatible_structured_stats = table != "v2-checkpoints-parquet-with-sidecars"; + let expect_parsed_stats = !skip_stats && compatible_structured_stats; + let expect_json_stats = !skip_stats && (request_json_stats || !expect_parsed_stats); + let expected_file_fragment = if table.starts_with("v2-") { + "_sidecars/" + } else { + ".checkpoint." + }; + let action_reads: Vec<_> = reads + .iter() + .filter(|read| { + read.files + .iter() + .any(|file| file.contains(expected_file_fragment)) + && read.physical_schema.field("add").is_some() + }) + .collect(); + assert!(!action_reads.is_empty(), "expected checkpoint Add reads"); + for read in action_reads { + let add_field = read + .physical_schema + .field("add") + .expect("checkpoint read schema must contain add"); + let DataType::Struct(add) = add_field.data_type() else { + panic!("checkpoint add field must be a struct"); + }; + assert_eq!( + add.field("stats").is_some(), + expect_json_stats, + "JSON checkpoint stats projection must match the requested output" + ); + assert_eq!( + add.field("stats_parsed").is_some(), + expect_parsed_stats, + "structured checkpoint stats projection must match the requested output" + ); + } +} + +#[test] +fn test_all_struct_parses_json_commit_stats() { + let path = fs::canonicalize(PathBuf::from( + "./tests/data/v1-single-part-struct-stats-only/", + )) + .unwrap(); + let url = Url::from_directory_path(path).unwrap(); + let engine = Arc::new(SyncEngine::new()); + // The table's checkpoint is at version 5, so version 4 replays only JSON commits. + let snapshot = Snapshot::builder_for(url) + .at_version(4) + .build(engine.as_ref()) + .unwrap(); + let scan = snapshot + .scan_builder() + .with_stats(StatsOptions::all_struct()) + .build() + .unwrap(); + + let mut file_count = 0; + for scan_metadata in scan .scan_metadata(engine.as_ref()) .unwrap() .collect::, _>>() - .unwrap(); + .unwrap() + { + let (underlying_data, selection_vector) = scan_metadata.scan_files.into_parts(); + let batch: RecordBatch = ArrowEngineData::try_from_engine_data(underlying_data) + .unwrap() + .into(); + let filtered = filter_record_batch(&batch, &BooleanArray::from(selection_vector)).unwrap(); + let stats_parsed = get_column!(filtered, "stats_parsed", StructArray); + let num_records = get_column!(stats_parsed, NUM_RECORDS, Int64Array); - let mut selected_file_count = 0; - for scan_metadata in &scan_metadata_results { - let selection_vector = scan_metadata.scan_files.selection_vector(); - selected_file_count += selection_vector - .iter() - .filter(|&&selected| selected) - .count(); + for row in 0..filtered.num_rows() { + assert!(!stats_parsed.is_null(row)); + assert_eq!(num_records.value(row), 1); + file_count += 1; + } } + assert_eq!(file_count, 4); +} - assert_eq!(selected_file_count, 6); +#[rstest] +#[case::v1_partition( + "v1-multi-part-partitioned-struct-stats-only", + Pred::eq(col!("part"), lit(1i32)), + column_name!("add.partitionValues_parsed.part"), + ".checkpoint.", +)] +#[case::v2_sidecar( + "v2-parquet-sidecars-struct-stats-only", + Pred::gt(col!("id"), lit(2i64)), + column_name!("add.stats_parsed.maxValues.id"), + "_sidecars/", +)] +fn test_checkpoint_predicate_reaches_parquet_handler( + #[case] table: &str, + #[case] pred: Pred, + #[case] expected_ref: ColumnName, + #[case] expected_file_fragment: &str, +) { + let path = std::fs::canonicalize(PathBuf::from(format!("./tests/data/{table}/"))).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + let sync = Arc::new(SyncEngine::new()); + let recorder = Arc::new(RecordingParquetHandler::new(sync.parquet_handler())); + let engine = DelegatingEngine::new(sync).with_parquet_handler(recorder.clone()); + let snapshot = Snapshot::builder_for(url).build(&engine).unwrap(); + recorder.take_reads(); + + let scan = snapshot + .scan_builder() + .with_predicate(Arc::new(pred)) + .build() + .unwrap(); + for action in scan.replay_for_scan_metadata(&engine).unwrap().actions { + action.unwrap(); + } + + let reads = recorder.take_reads(); + assert!( + reads.iter().any(|read| { + read.files + .iter() + .any(|file| file.contains(expected_file_fragment)) + && read + .predicate + .as_ref() + .is_some_and(|predicate| predicate.references().contains(&expected_ref)) + }), + "expected {expected_ref} on a {expected_file_fragment} read, got {reads:#?}" + ); } #[test] -fn test_skip_stats_after_include_all_stats_columns_wins() { - // With StatsOutputMode enum, last call wins. Calling with_skip_stats(true) after - // include_all_stats_columns() should result in stats being skipped. +fn test_skip_stats_disables_data_skipping() { let path = std::fs::canonicalize(PathBuf::from("./tests/data/parsed-stats/")).unwrap(); let url = url::Url::from_directory_path(path).unwrap(); let engine = Arc::new(SyncEngine::new()); let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); - let predicate = Arc::new(Pred::gt(column_expr!("id"), Expr::literal(400i64))); + let predicate = Arc::new(Pred::gt(col!("id"), lit(400i64))); let scan = snapshot .scan_builder() - .include_all_stats_columns() - .with_skip_stats(true) .with_predicate(predicate) + .with_stats(StatsOptions::none()) .build() .unwrap(); - // Stats are skipped, so all files should be returned (no data skipping) let scan_metadata_results: Vec<_> = scan .scan_metadata(engine.as_ref()) .unwrap() @@ -1256,58 +2033,50 @@ fn test_skip_stats_after_include_all_stats_columns_wins() { assert_eq!(selected_file_count, 6); } +/// Calling `with_stats` twice replaces the prior value; the last call wins. #[test] -fn test_with_stats_columns_empty_no_stats_output() { - // with_stats_columns(vec![]) should produce no stats output +fn test_with_stats_last_call_wins() { let path = std::fs::canonicalize(PathBuf::from("./tests/data/parsed-stats/")).unwrap(); let url = url::Url::from_directory_path(path).unwrap(); let engine = Arc::new(SyncEngine::new()); let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); + // First call is `all()` (would emit stats_parsed); last call is `none()` (no output). let scan = snapshot .scan_builder() - .with_stats_columns(vec![]) + .with_stats(StatsOptions::all()) + .with_stats(StatsOptions::none()) .build() .unwrap(); - let scan_metadata_results: Vec<_> = scan + for scan_metadata in scan .scan_metadata(engine.as_ref()) .unwrap() .collect::, _>>() - .unwrap(); - - assert!( - !scan_metadata_results.is_empty(), - "Should have scan metadata" - ); - - for scan_metadata in scan_metadata_results { - let (underlying_data, _selection_vector) = scan_metadata.scan_files.into_parts(); + .unwrap() + { + let (underlying_data, _) = scan_metadata.scan_files.into_parts(); let batch: RecordBatch = ArrowEngineData::try_from_engine_data(underlying_data) .unwrap() .into(); - - // stats_parsed should not be present since empty Columns means no stats output assert!( batch.column_by_name("stats_parsed").is_none(), - "stats_parsed should not be present with empty stats columns" + "last call (`none`) should win: no stats_parsed in output" ); } } -/// Test that `with_stats_columns` with specific columns only returns stats for those columns. -/// Verifies that requesting `vec![col!("id")]` only includes `id` in minValues/maxValues/nullCount. #[test] -fn test_scan_metadata_with_specific_stats_columns() { +fn test_default_stats_options_no_struct_output() { + // StatsOptions::default() produces no stats_parsed output. let path = std::fs::canonicalize(PathBuf::from("./tests/data/parsed-stats/")).unwrap(); let url = url::Url::from_directory_path(path).unwrap(); let engine = Arc::new(SyncEngine::new()); let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); - // Request only "id" column stats let scan = snapshot .scan_builder() - .with_stats_columns(vec![column_name!("id")]) + .with_stats(StatsOptions::default()) .build() .unwrap(); @@ -1323,107 +2092,192 @@ fn test_scan_metadata_with_specific_stats_columns() { ); for scan_metadata in scan_metadata_results { - let (underlying_data, selection_vector) = scan_metadata.scan_files.into_parts(); + let (underlying_data, _selection_vector) = scan_metadata.scan_files.into_parts(); let batch: RecordBatch = ArrowEngineData::try_from_engine_data(underlying_data) .unwrap() .into(); - let filtered_batch = - filter_record_batch(&batch, &BooleanArray::from(selection_vector)).unwrap(); - - let stats_parsed = get_column!(filtered_batch, "stats_parsed", StructArray); - let min_values = get_column!(stats_parsed, "minValues", StructArray); - let max_values = get_column!(stats_parsed, "maxValues", StructArray); - let null_count = get_column!(stats_parsed, "nullCount", StructArray); - // Check minValues/maxValues/nullCount only have "id" - assert_eq!( - field_names(min_values), - vec!["id"], - "minValues should only contain 'id'" - ); - assert_eq!( - field_names(max_values), - vec!["id"], - "maxValues should only contain 'id'" - ); - assert_eq!( - field_names(null_count), - vec!["id"], - "nullCount should only contain 'id'" + // stats_parsed should not be present since empty Columns means no stats output + assert!( + batch.column_by_name("stats_parsed").is_none(), + "stats_parsed should not be present with empty stats columns" ); } } -/// Test that `with_stats_columns` with multiple specific columns returns stats for all of them. -#[test] -fn test_scan_metadata_with_multiple_stats_columns() { - let path = std::fs::canonicalize(PathBuf::from("./tests/data/parsed-stats/")).unwrap(); - let url = url::Url::from_directory_path(path).unwrap(); +#[rstest] +#[case::id_without_predicate( + StatsOptions::struct_columns(vec![column_name!("id")]), + &["id"], + None, + "id", + &[ + ("1", "100"), + ("101", "200"), + ("201", "300"), + ("301", "400"), + ("401", "500"), + ("501", "600"), + ], +)] +#[case::id_with_json_without_predicate( + StatsOptions { + synthesize_json: true, + struct_stats: StructStats::Columns(vec![column_name!("id")]), + }, + &["id"], + None, + "id", + &[ + ("1", "100"), + ("101", "200"), + ("201", "300"), + ("301", "400"), + ("401", "500"), + ("501", "600"), + ], +)] +#[case::id_predicate_requested( + StatsOptions::struct_columns(vec![column_name!("id")]), + &["id"], + Some(col!("id").gt(lit(400i64))), + "id", + &[("401", "500"), ("501", "600")], +)] +#[case::id_predicate_not_requested( + StatsOptions::struct_columns(vec![column_name!("name")]), + &["id", "name"], + Some(col!("id").gt(lit(400i64))), + "name", + &[("name_401", "name_500"), ("name_501", "name_600")], +)] +#[case::salary_predicate_with_multiple_requested_columns( + StatsOptions::struct_columns(vec![column_name!("id"), column_name!("name")]), + &["id", "name", "salary"], + Some(col!("salary").le(lit(70_000i64))), + "id", + &[("1", "100"), ("101", "200")], +)] +#[case::salary_requested_with_different_predicate_column( + StatsOptions::struct_columns(vec![column_name!("salary")]), + &["id", "salary"], + Some(col!("id").gt(lit(500i64))), + "salary", + &[("100100", "110000")], +)] +fn scan_metadata_struct_columns_returns_expected_stats( + #[case] stats: StatsOptions, + #[case] expected_stat_fields: &[&str], + #[case] predicate: Option, + #[case] probe_column: &str, + #[case] expected_min_max: &[(&str, &str)], +) { + let path = fs::canonicalize(PathBuf::from("./tests/data/parsed-stats/")).unwrap(); + let url = Url::from_directory_path(path).unwrap(); let engine = Arc::new(SyncEngine::new()); let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); - - // Request "id" and "name" column stats (not "age" or "salary") + let predicate = predicate.map(|predicate| Arc::new(predicate) as PredicateRef); let scan = snapshot .scan_builder() - .with_stats_columns(vec![column_name!("id"), column_name!("name")]) + .with_predicate(predicate) + .with_stats(stats) .build() .unwrap(); - let scan_metadata_results: Vec<_> = scan - .scan_metadata(engine.as_ref()) - .unwrap() - .collect::, _>>() - .unwrap(); - - assert!( - !scan_metadata_results.is_empty(), - "Should have scan metadata" - ); - - for scan_metadata in scan_metadata_results { - let (underlying_data, selection_vector) = scan_metadata.scan_files.into_parts(); + let mut actual_min_max = Vec::new(); + let mut batch_count = 0; + for scan_metadata in scan.scan_metadata(engine.as_ref()).unwrap() { + batch_count += 1; + let (underlying_data, selection_vector) = scan_metadata.unwrap().scan_files.into_parts(); let batch: RecordBatch = ArrowEngineData::try_from_engine_data(underlying_data) .unwrap() .into(); - let filtered_batch = - filter_record_batch(&batch, &BooleanArray::from(selection_vector)).unwrap(); + let stats_parsed = get_column!(batch, STATS_PARSED, StructArray); + let min_values = get_column!(stats_parsed, MIN_VALUES, StructArray); + let max_values = get_column!(stats_parsed, MAX_VALUES, StructArray); + let null_count = get_column!(stats_parsed, NULL_COUNT, StructArray); + assert_eq!(field_names(min_values), expected_stat_fields); + assert_eq!(field_names(max_values), expected_stat_fields); + assert_eq!(field_names(null_count), expected_stat_fields); + + let filtered = filter_record_batch(&batch, &BooleanArray::from(selection_vector)).unwrap(); + let stats_parsed = get_column!(filtered, STATS_PARSED, StructArray); + let num_records = get_column!(stats_parsed, NUM_RECORDS, Int64Array); + let min_values = get_column!(stats_parsed, MIN_VALUES, StructArray); + let max_values = get_column!(stats_parsed, MAX_VALUES, StructArray); + let probe_min = min_values.column_by_name(probe_column).unwrap(); + let probe_max = max_values.column_by_name(probe_column).unwrap(); + for row in 0..filtered.num_rows() { + assert!(!stats_parsed.is_null(row)); + assert_eq!(num_records.value(row), 100); + actual_min_max.push(( + array_value_to_string(probe_min.as_ref(), row).unwrap(), + array_value_to_string(probe_max.as_ref(), row).unwrap(), + )); + } + } - let stats_parsed = get_column!(filtered_batch, "stats_parsed", StructArray); - let min_values = get_column!(stats_parsed, "minValues", StructArray); - let max_values = get_column!(stats_parsed, "maxValues", StructArray); - let null_count = get_column!(stats_parsed, "nullCount", StructArray); + assert!(batch_count > 0); + actual_min_max.sort_unstable(); + let mut expected_min_max: Vec<_> = expected_min_max + .iter() + .map(|(min, max)| (min.to_string(), max.to_string())) + .collect(); + expected_min_max.sort_unstable(); + assert_eq!(actual_min_max, expected_min_max); +} - // Check minValues/maxValues/nullCount have "id" and "name" - let expected = vec!["id", "name"]; - assert_eq!( - field_names(min_values), - expected, - "minValues should contain 'id' and 'name'" - ); - assert_eq!( - field_names(max_values), - expected, - "maxValues should contain 'id' and 'name'" - ); - assert_eq!( - field_names(null_count), - expected, - "nullCount should contain 'id' and 'name'" - ); +#[test] +fn scan_metadata_struct_columns_fully_pruning_predicate_yields_no_batches() { + let path = fs::canonicalize(PathBuf::from("./tests/data/parsed-stats/")).unwrap(); + let url = Url::from_directory_path(path).unwrap(); + let engine = Arc::new(SyncEngine::new()); + let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); + let scan = snapshot + .scan_builder() + .with_predicate(Arc::new(col!("id").gt(lit(600i64)))) + .with_stats(StatsOptions::struct_columns(vec![column_name!("salary")])) + .build() + .unwrap(); - // Verify "age" and "salary" are NOT present - assert!( - min_values.column_by_name("age").is_none(), - "minValues should NOT contain 'age'" - ); - assert!( - min_values.column_by_name("salary").is_none(), - "minValues should NOT contain 'salary'" - ); + assert_eq!(scan.scan_metadata(engine.as_ref()).unwrap().count(), 0); +} + +#[rstest] +fn scan_builder_validates_predicate_and_stats_columns( + #[values( + (column_pred!("missing_predicate"), false), + (column_pred!("id"), true) + )] + predicate: (Pred, bool), + #[values( + (StatsOptions::struct_columns(vec![column_name!("missing_stats")]), false), + (StatsOptions::struct_columns(vec![column_name!("id")]), true) + )] + stats: (StatsOptions, bool), +) { + let path = std::fs::canonicalize(PathBuf::from("./tests/data/parsed-stats/")).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + let engine = SyncEngine::new(); + let snapshot = Snapshot::builder_for(url).build(&engine).unwrap(); + let (predicate, predicate_exists) = predicate; + let (stats, stats_columns_exist) = stats; + let result = snapshot + .scan_builder() + .with_predicate(Arc::new(predicate)) + .with_stats(stats) + .build(); + + match (predicate_exists, stats_columns_exist) { + (true, true) => { + result.expect("valid predicate and stats columns"); + } + (false, _) => assert_result_error_with_message(result, "missing_predicate"), + (true, false) => assert_result_error_with_message(result, "missing_stats"), } } -/// Test that `with_stats_columns` with a nonexistent column name produces empty stats for that column. +/// Test that [`StructStats::Columns`] rejects nonexistent columns. #[test] fn test_scan_metadata_with_nonexistent_stats_columns() { let path = std::fs::canonicalize(PathBuf::from("./tests/data/parsed-stats/")).unwrap(); @@ -1431,38 +2285,221 @@ fn test_scan_metadata_with_nonexistent_stats_columns() { let engine = Arc::new(SyncEngine::new()); let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); - let scan = snapshot + let result = snapshot .scan_builder() - .with_stats_columns(vec![column_name!("nonexistent_column")]) - .build() - .unwrap(); + .with_stats(StatsOptions { + synthesize_json: true, + struct_stats: StructStats::Columns(vec![column_name!("nonexistent_column")]), + }) + .build(); - let scan_metadata_results: Vec<_> = scan - .scan_metadata(engine.as_ref()) - .unwrap() - .collect::, _>>() - .unwrap(); + assert_result_error_with_message(result, "Could not resolve column 'nonexistent_column'"); +} + +/// A [`ParquetHandler`] that returns an empty iterator for every `read_parquet_files` call. +/// Used to simulate a buggy connector that drops all data for a file. +struct EmptyParquetHandler; + +impl ParquetHandler for EmptyParquetHandler { + fn read_parquet_files( + &self, + _files: &[FileMeta], + _schema: schema::SchemaRef, + _predicate: Option, + ) -> DeltaResult { + Ok(Box::new(std::iter::empty())) + } + + fn read_parquet_footer(&self, _file: &FileMeta) -> DeltaResult { + unimplemented!() + } + fn write_parquet_file( + &self, + _location: url::Url, + _data: DeltaResultIteratorStatic>, + ) -> DeltaResult<()> { + unimplemented!() + } +} + +/// When a file's Add action stats report `numRecords > 0` and the parquet handler returns an empty +/// iterator, `execute` must surface an error rather than silently producing no rows. +#[test] +fn execute_errors_when_parquet_returns_empty_for_file_with_positive_stats() { + let path = + std::fs::canonicalize(PathBuf::from("./tests/data/table-without-dv-small/")).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + let engine = Arc::new( + DelegatingEngine::new(Arc::new(SyncEngine::new())) + .with_parquet_handler(Arc::new(EmptyParquetHandler)), + ); + + let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); + let scan = snapshot.scan_builder().build().unwrap(); + + let results: Vec<_> = scan.execute(engine).unwrap().collect(); + assert_eq!(results.len(), 1, "should emit exactly one error item"); + assert!(results[0].is_err(), "the result should be an error, got Ok"); + let err = results[0].as_ref().err().unwrap().to_string(); assert!( - !scan_metadata_results.is_empty(), - "Should have scan metadata" + err.contains("ParquetHandler returned no data"), + "unexpected error message: {err}" ); +} - for scan_metadata in scan_metadata_results { - let (underlying_data, selection_vector) = scan_metadata.scan_files.into_parts(); - let batch: RecordBatch = ArrowEngineData::try_from_engine_data(underlying_data) +/// When a file's Add action has no stats, an empty iterator from the parquet handler is allowed +/// -- we conservatively treat the file as possibly legitimately empty. +#[test] +fn execute_does_not_error_when_parquet_returns_empty_and_stats_absent() { + let path = std::fs::canonicalize(PathBuf::from("./tests/data/table-with-cdf/")).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + let engine = Arc::new( + DelegatingEngine::new(Arc::new(SyncEngine::new())) + .with_parquet_handler(Arc::new(EmptyParquetHandler)), + ); + + let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); + let scan = snapshot.scan_builder().build().unwrap(); + + // All Add files in this table have no stats -- empty iterators should be silently ignored. + let results: Vec<_> = scan.execute(engine).unwrap().collect(); + assert!( + results.iter().all(|r| r.is_ok()), + "expected no errors for stats-absent files" + ); +} + +/// Tests for `ScanMetadataCompleted` event emission via the tracing-based metrics system. +mod scan_metadata_completed_tests { + use std::path::PathBuf; + use std::sync::Arc; + use std::time::Duration; + + use rstest::rstest; + + use super::ScanBuilder; + use crate::engine::sync::SyncEngine; + use crate::expressions::{col, lit, Expression as Expr, Predicate as Pred}; + use crate::metrics::MetricEvent; + use crate::unit_test_utils::{install_thread_local_metrics_reporter, CapturingReporter}; + use crate::utils::FoldWithOption as _; + use crate::Snapshot; + + fn run_scan( + table: &str, + predicate: Option>, + correlation_id: Option<&str>, + ) -> ( + Arc, + tracing::subscriber::DefaultGuard, + usize, + ) { + let path = std::fs::canonicalize(PathBuf::from(table)).unwrap(); + let url = url::Url::from_directory_path(&path).unwrap(); + let reporter = Arc::new(CapturingReporter::default()); + let engine = Arc::new(SyncEngine::new()); + let guard = install_thread_local_metrics_reporter(reporter.clone()); + let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); + let scan = snapshot + .scan_builder() + .fold_with(predicate, ScanBuilder::with_predicate) + .fold_with(correlation_id, ScanBuilder::with_correlation_id) + .build() + .unwrap(); + let results: Vec<_> = scan + .scan_metadata(engine.as_ref()) .unwrap() - .into(); - let filtered_batch = - filter_record_batch(&batch, &BooleanArray::from(selection_vector)).unwrap(); + .collect::, _>>() + .unwrap(); + (reporter, guard, results.len()) + } - let stats_parsed = get_column!(filtered_batch, "stats_parsed", StructArray); + fn get_scan_event(reporter: &CapturingReporter) -> MetricEvent { + reporter + .events() + .into_iter() + .find(|e| matches!(e, MetricEvent::ScanMetadataCompleted(_))) + .expect("expected ScanMetadataCompleted event") + } - // Should have numRecords but no minValues/maxValues/nullCount - // (or they exist but are empty structs) - assert!( - stats_parsed.column_by_name("numRecords").is_some(), - "Should still have numRecords" - ); + #[rstest] + #[case::basic_scan("./tests/data/parsed-stats/", None, 6, 6, 17236, 0, 0)] + #[case::static_skip_all( + "./tests/data/parsed-stats/", + Some(Arc::new(Pred::FALSE)), + 0, + 0, + 0, + 0, + 0 + )] + #[case::with_removes("./tests/data/table-with-cdf/", None, 1, 0, 0, 2, 0)] + #[case::with_checkpoint( + "./tests/data/with_checkpoint_no_last_checkpoint/", + None, + 2, + 1, + 1010, + 1, + 0 + )] + #[case::partition_filter( + "./tests/data/basic_partitioned/", + Some(Arc::new(Expr::eq(col!("letter"), lit("a")))), + 2, 2, 1502, 0, 4 + )] + fn test_scan_metrics( + #[case] table: &str, + #[case] predicate: Option>, + #[case] expected_add_seen: u64, + #[case] expected_active: u64, + #[case] expected_active_bytes: u64, + #[case] expected_removes: u64, + #[case] expected_filtered: u64, + ) { + let (reporter, _guard, _) = run_scan(table, predicate, None); + let MetricEvent::ScanMetadataCompleted(e) = get_scan_event(&reporter) else { + panic!("expected ScanMetadataCompleted"); + }; + assert!(e.duration > Duration::ZERO); + assert_eq!(e.num_add_files_seen, expected_add_seen); + assert_eq!(e.num_active_add_files, expected_active); + assert_eq!(e.active_add_files_bytes, expected_active_bytes); + assert_eq!(e.num_remove_files_seen, expected_removes); + assert_eq!(e.num_predicate_filtered, expected_filtered); + } + + // The parallel-scan paths (both sequential and parallel phase events) are covered by + // `parallel_scan_metadata_phases_carry_correlation_id` in `parallel::parallel_phase`. + #[rstest] + #[case::with_id(Some("scan-req-1"))] + #[case::without_id(None)] + fn scan_metadata_completed_carries_correlation_id(#[case] correlation_id: Option<&str>) { + let (reporter, _guard, _) = run_scan("./tests/data/parsed-stats/", None, correlation_id); + let MetricEvent::ScanMetadataCompleted(e) = get_scan_event(&reporter) else { + panic!("expected ScanMetadataCompleted"); + }; + assert_eq!(e.correlation_id.as_deref(), correlation_id); + } + + #[test] + fn scan_metadata_completed_not_emitted_on_early_drop() { + let path = std::fs::canonicalize(PathBuf::from("./tests/data/parsed-stats/")).unwrap(); + let url = url::Url::from_directory_path(&path).unwrap(); + let reporter = Arc::new(CapturingReporter::default()); + let engine = Arc::new(SyncEngine::new()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); + let scan = snapshot.scan_builder().build().unwrap(); + { + let mut iter = scan.scan_metadata(engine.as_ref()).unwrap(); + let _ = iter.next(); + // Drop without exhausting -- callback must not fire + } + assert!(reporter + .events() + .iter() + .all(|e| !matches!(e, MetricEvent::ScanMetadataCompleted(_)))); } } diff --git a/kernel/src/scan/transform_spec.rs b/kernel/src/scan/transform_spec.rs index 8682e1a8a7..c3ee21c28b 100644 --- a/kernel/src/scan/transform_spec.rs +++ b/kernel/src/scan/transform_spec.rs @@ -1,8 +1,8 @@ //! Transform-related types and utilities for Delta Kernel. //! -//! This module contains the types and functions needed to handle transforms -//! during scan and table changes operations, including partition value processing -//! and expression generation. +//! This module contains specs and helpers for translating physical file data into the logical scan +//! output shape, including partition value processing and expression generation. The current +//! implementation lowers those specs into `Expression::StructPatch` expressions. use std::collections::HashMap; use std::sync::Arc; @@ -10,18 +10,25 @@ use std::sync::Arc; use itertools::Itertools; use serde::{Deserialize, Serialize}; -use crate::expressions::{BinaryExpressionOp, Expression, ExpressionRef, Scalar, Transform}; +use crate::expressions::{lit, Expression, ExpressionRef, ExpressionStructPatchBuilder, Scalar}; use crate::schema::{DataType, SchemaRef, StructType}; use crate::table_features::ColumnMappingMode; +use crate::timestamp_timezone::TimestampTimezone; use crate::{DeltaResult, Error}; -/// A list of field transforms that describes a transform expression to be created at scan time. +/// A list of field transforms used to convert physical file data to logical scan output. +// TODO: Rename this physical-to-logical read fixup concept in a follow-up PR. "Transform" used to +// refer partly to the `Expression::Transform` implementation detail (now +// `Expression::StructPatch`), but also to the fact that this spec describes how to transform +// physical file data into logical scan output. Neither "transform" nor "patch" is an ideal +// description for that concept. pub(crate) type TransformSpec = Vec; -/// Describes a single field transformation to apply when converting physical data to logical schema. +/// Describes a single field transformation to apply when converting physical data to logical +/// schema. /// -/// These transformations are "sparse" - they only specify what changes, while unchanged fields -/// pass through implicitly in their original order. +/// These transformations are sparse: they only specify what changes, while unchanged fields pass +/// through implicitly in their original order. #[derive(Debug, Serialize, Deserialize, PartialEq)] pub(crate) enum FieldTransformSpec { /// Insert the given expression after the named input column (None = prepend instead) @@ -71,6 +78,7 @@ pub(crate) fn parse_partition_value( logical_schema: &SchemaRef, partition_values: &HashMap, column_mapping_mode: ColumnMappingMode, + timestamp_timezone: TimestampTimezone, ) -> DeltaResult<(usize, (String, Scalar))> { let Some(field) = logical_schema.field_at_index(field_idx) else { return Err(Error::InternalError(format!( @@ -78,7 +86,11 @@ pub(crate) fn parse_partition_value( ))); }; let name = field.physical_name(column_mapping_mode); - let partition_value = parse_partition_value_raw(partition_values.get(name), field.data_type())?; + let partition_value = parse_partition_value_raw( + partition_values.get(name), + field.data_type(), + timestamp_timezone, + )?; Ok((field_idx, (name.to_string(), partition_value))) } @@ -88,6 +100,7 @@ pub(crate) fn parse_partition_values( transform_spec: &TransformSpec, partition_values: &HashMap, column_mapping_mode: ColumnMappingMode, + timestamp_timezone: TimestampTimezone, ) -> DeltaResult> { transform_spec .iter() @@ -98,6 +111,7 @@ pub(crate) fn parse_partition_values( logical_schema, partition_values, column_mapping_mode, + timestamp_timezone, )) } FieldTransformSpec::DynamicColumn { .. } @@ -108,26 +122,26 @@ pub(crate) fn parse_partition_values( .try_collect() } -/// Build a transform expression that converts physical data to the logical schema. +/// Build an expression that converts physical data to the logical schema. /// /// An empty `transform_spec` is valid and represents the case where only column mapping is needed. -/// The resulting empty `Expression::Transform` will pass all input fields through unchanged -/// while applying the output schema for name mapping. +/// The resulting empty `Expression::StructPatch` passes all input fields through unchanged while +/// applying the output schema for name mapping. pub(crate) fn get_transform_expr( transform_spec: &TransformSpec, mut metadata_values: HashMap, physical_schema: &StructType, base_row_id: Option, ) -> DeltaResult { - let mut transform = Transform::new_top_level(); + let mut patch = ExpressionStructPatchBuilder::new(); for field_transform in transform_spec { use FieldTransformSpec::*; - transform = match field_transform { + patch = match field_transform { StaticInsert { insert_after, expr } => { - transform.with_inserted_field(insert_after.clone(), expr.clone()) + apply_insert_after(patch, insert_after, expr.clone()) } - StaticDrop { field_name } => transform.with_dropped_field(field_name.clone()), + StaticDrop { field_name } => patch.drop(field_name.clone()), GenerateRowId { field_name, row_index_field_name, @@ -137,13 +151,9 @@ pub(crate) fn get_transform_expr( })?; let expr = Arc::new(Expression::coalesce([ Expression::column([field_name]), - Expression::binary( - BinaryExpressionOp::Plus, - Expression::literal(base_row_id), - Expression::column([row_index_field_name]), - ), + lit(base_row_id) + Expression::column([row_index_field_name]), ])); - transform.with_replaced_field(field_name.clone(), expr) + patch.replace(field_name.clone(), expr) } MetadataDerivedColumn { field_index, @@ -156,7 +166,7 @@ pub(crate) fn get_transform_expr( }; let partition_value = Arc::new(partition_value.into()); - transform.with_inserted_field(insert_after.clone(), partition_value) + apply_insert_after(patch, insert_after, partition_value) } DynamicColumn { field_index, @@ -169,13 +179,11 @@ pub(crate) fn get_transform_expr( if exists_physically { // Column exists physically - reorder it via drop+insert // This ensures consistent column ordering across file types - transform = transform - .with_dropped_field(physical_name.clone()) - .with_inserted_field( - insert_after.clone(), - Arc::new(Expression::column([physical_name.clone()])), - ); - transform + apply_insert_after( + patch.drop(physical_name), + insert_after, + Arc::new(Expression::column([physical_name])), + ) } else { // Column doesn't exist physically - treat as partition column let Some((_, partition_value)) = metadata_values.remove(field_index) else { @@ -185,21 +193,47 @@ pub(crate) fn get_transform_expr( }; let partition_value = Arc::new(partition_value.into()); - transform.with_inserted_field(insert_after.clone(), partition_value) + apply_insert_after(patch, insert_after, partition_value) } } } } - Ok(Arc::new(Expression::Transform(transform))) + Ok(Arc::new(Expression::struct_patch(patch)?)) } -/// Parse a partition value from the raw string representation +// Adapter that converts the insert_after option into a method call on the patch. +fn apply_insert_after( + patch: ExpressionStructPatchBuilder, + insert_after: &Option, + expr: ExpressionRef, +) -> ExpressionStructPatchBuilder { + match insert_after { + Some(predecessor) => patch.insert_after(predecessor, expr), + None => patch.prepend(expr), + } +} + +/// Parse a partition value from the raw string representation. +/// +/// An empty string casts via [`PrimitiveType::empty_string_partition_cast`]. +/// +/// [`PrimitiveType::empty_string_partition_cast`]: crate::schema::PrimitiveType::empty_string_partition_cast pub(crate) fn parse_partition_value_raw( raw: Option<&String>, data_type: &DataType, + timestamp_timezone: TimestampTimezone, ) -> DeltaResult { match (raw, data_type.as_primitive_opt()) { + (Some(v), Some(primitive)) if v.is_empty() => Ok(primitive + .empty_string_partition_cast() + .unwrap_or_else(|| Scalar::Null(data_type.clone()))), + (Some(v), Some(primitive)) if primitive == &crate::schema::PrimitiveType::Timestamp => { + timestamp_timezone + .parse_timestamp(v) + .map(Scalar::Timestamp) + .ok_or_else(|| Error::ParseError(v.clone(), data_type.clone())) + } (Some(v), Some(primitive)) => primitive.parse_scalar(v), (Some(_), None) => Err(Error::generic(format!( "Unexpected partition column type: {data_type:?}" @@ -210,10 +244,12 @@ pub(crate) fn parse_partition_value_raw( #[cfg(test)] mod tests { + use std::collections::HashMap; + use super::*; + use crate::expressions::{col, BinaryExpressionOp}; use crate::schema::{DataType, PrimitiveType, StructField, StructType}; - use crate::utils::test_utils::assert_result_error_with_message; - use std::collections::HashMap; + use crate::unit_test_utils::assert_result_error_with_message; // Tests for parse_partition_value function #[test] @@ -224,7 +260,13 @@ mod tests { )])); let partition_values = HashMap::new(); - let result = parse_partition_value(5, &schema, &partition_values, ColumnMappingMode::None); + let result = parse_partition_value( + 5, + &schema, + &partition_values, + ColumnMappingMode::None, + TimestampTimezone::default(), + ); assert_result_error_with_message(result, "out of bounds"); } @@ -264,6 +306,7 @@ mod tests { &transform_spec, &partition_values, ColumnMappingMode::None, + TimestampTimezone::default(), ) .unwrap(); assert_eq!(result.len(), 2); @@ -290,6 +333,7 @@ mod tests { &transform_spec, &partition_values, ColumnMappingMode::None, + TimestampTimezone::default(), ) .unwrap(); assert!(result.is_empty()); @@ -298,8 +342,12 @@ mod tests { // Tests for parse_partition_value_raw function #[test] fn test_parse_partition_value_raw_string() { - let result = - parse_partition_value_raw(Some(&"test_string".to_string()), &DataType::STRING).unwrap(); + let result = parse_partition_value_raw( + Some(&"test_string".to_string()), + &DataType::STRING, + TimestampTimezone::default(), + ) + .unwrap(); assert_eq!(result, Scalar::String("test_string".to_string())); } @@ -308,6 +356,7 @@ mod tests { let result = parse_partition_value_raw( Some(&"42".to_string()), &DataType::Primitive(PrimitiveType::Integer), + TimestampTimezone::default(), ) .unwrap(); assert_eq!(result, Scalar::Integer(42)); @@ -315,15 +364,50 @@ mod tests { #[test] fn test_parse_partition_value_raw_null() { - let result = parse_partition_value_raw(None, &DataType::STRING).unwrap(); + let result = + parse_partition_value_raw(None, &DataType::STRING, TimestampTimezone::default()) + .unwrap(); assert!(result.is_null()); } + #[test] + fn test_parse_partition_value_raw_empty_string_cast_semantics() { + // An empty string casts to itself for string and to empty bytes for binary, and to null + // for every other type. A literal empty string only reaches this path from a foreign + // writer, since kernel serializes its own empty and null partition values to JSON null. + let empty = String::new(); + + let string_value = parse_partition_value_raw( + Some(&empty), + &DataType::STRING, + TimestampTimezone::default(), + ) + .unwrap(); + assert_eq!(string_value, Scalar::String(String::new())); + + let binary_value = parse_partition_value_raw( + Some(&empty), + &DataType::BINARY, + TimestampTimezone::default(), + ) + .unwrap(); + assert_eq!(binary_value, Scalar::Binary(Vec::new())); + + let int_value = parse_partition_value_raw( + Some(&empty), + &DataType::Primitive(PrimitiveType::Integer), + TimestampTimezone::default(), + ) + .unwrap(); + assert!(int_value.is_null()); + } + #[test] fn test_parse_partition_value_raw_invalid_type() { let result = parse_partition_value_raw( Some(&"value".to_string()), &DataType::struct_type_unchecked(vec![]), // Non-primitive type + TimestampTimezone::default(), ); assert_result_error_with_message(result, "Unexpected partition column type"); } @@ -333,6 +417,7 @@ mod tests { let result = parse_partition_value_raw( Some(&"not_a_number".to_string()), &DataType::Primitive(PrimitiveType::Integer), + TimestampTimezone::default(), ); assert_result_error_with_message(result, "Failed to parse value"); } @@ -384,24 +469,23 @@ mod tests { ) .unwrap(); - let Expression::Transform(transform) = result.as_ref() else { - panic!("Expected Transform expression"); + let Expression::StructPatch(patch) = result.as_ref() else { + panic!("Expected StructPatch expression"); }; // Verify StaticInsert: should insert after col1 - assert!(transform.field_transforms.contains_key("col1")); - assert!(!transform.field_transforms["col1"].is_replace); - assert_eq!(transform.field_transforms["col1"].exprs.len(), 1); - let Expression::Literal(scalar) = transform.field_transforms["col1"].exprs[0].as_ref() - else { + assert!(patch.field_patches.contains_key("col1")); + assert!(patch.field_patches["col1"].keep_input); + assert_eq!(patch.field_patches["col1"].insertions.len(), 1); + let Expression::Literal(scalar) = patch.field_patches["col1"].insertions[0].as_ref() else { panic!("Expected literal expression for insert"); }; assert_eq!(scalar, &Scalar::Integer(42)); - // Verify StaticDrop: should drop col2 (empty expressions and is_replace = true) - assert!(transform.field_transforms.contains_key("col2")); - assert!(transform.field_transforms["col2"].is_replace); - assert!(transform.field_transforms["col2"].exprs.is_empty()); + // Verify StaticDrop: should drop col2 + assert!(patch.field_patches.contains_key("col2")); + assert!(!patch.field_patches["col2"].keep_input); + assert!(patch.field_patches["col2"].insertions.is_empty()); } #[test] @@ -425,22 +509,22 @@ mod tests { &physical_schema, None, /* base_row_id */ ); - let transform_expr = result.expect("Transform expression should be created successfully"); + let patch_expr = result.expect("StructPatch expression should be created successfully"); - let Expression::Transform(transform) = transform_expr.as_ref() else { - panic!("Expected Transform expression"); + let Expression::StructPatch(patch) = patch_expr.as_ref() else { + panic!("Expected StructPatch expression"); }; // Should drop _change_type and insert it after id - assert!(transform.field_transforms.contains_key("_change_type")); - assert!(transform.field_transforms["_change_type"].is_replace); - assert!(transform.field_transforms["_change_type"].exprs.is_empty()); + assert!(patch.field_patches.contains_key("_change_type")); + assert!(!patch.field_patches["_change_type"].keep_input); + assert!(patch.field_patches["_change_type"].insertions.is_empty()); - assert!(transform.field_transforms.contains_key("id")); - assert!(!transform.field_transforms["id"].is_replace); - assert_eq!(transform.field_transforms["id"].exprs.len(), 1); + assert!(patch.field_patches.contains_key("id")); + assert!(patch.field_patches["id"].keep_input); + assert_eq!(patch.field_patches["id"].insertions.len(), 1); - let Expression::Column(column_name) = transform.field_transforms["id"].exprs[0].as_ref() + let Expression::Column(column_name) = patch.field_patches["id"].insertions[0].as_ref() else { panic!("Expected column reference"); }; @@ -473,20 +557,21 @@ mod tests { &physical_schema, None, /* base_row_id */ ); - let transform_expr = result.expect("Transform expression should be created successfully"); + let patch_expr = result.expect("StructPatch expression should be created successfully"); - let Expression::Transform(transform) = transform_expr.as_ref() else { - panic!("Expected Transform expression"); + let Expression::StructPatch(patch) = patch_expr.as_ref() else { + panic!("Expected StructPatch expression"); }; - // Should not drop _change_type (doesn't exist physically) and insert metadata value after id - assert!(!transform.field_transforms.contains_key("_change_type")); + // Should not drop _change_type (doesn't exist physically) and insert metadata value after + // id + assert!(!patch.field_patches.contains_key("_change_type")); - assert!(transform.field_transforms.contains_key("id")); - assert!(!transform.field_transforms["id"].is_replace); - assert_eq!(transform.field_transforms["id"].exprs.len(), 1); + assert!(patch.field_patches.contains_key("id")); + assert!(patch.field_patches["id"].keep_input); + assert_eq!(patch.field_patches["id"].insertions.len(), 1); - let Expression::Literal(scalar) = transform.field_transforms["id"].exprs[0].as_ref() else { + let Expression::Literal(scalar) = patch.field_patches["id"].insertions[0].as_ref() else { panic!("Expected literal"); }; assert_eq!(scalar, &Scalar::String("insert".to_string())); @@ -510,18 +595,18 @@ mod tests { &physical_schema, None, /* base_row_id */ ); - let transform_expr = result.expect("Transform expression should be created successfully"); + let patch_expr = result.expect("StructPatch expression should be created successfully"); - let Expression::Transform(transform) = transform_expr.as_ref() else { - panic!("Expected Transform expression"); + let Expression::StructPatch(patch) = patch_expr.as_ref() else { + panic!("Expected StructPatch expression"); }; // Should insert metadata value after id - assert!(transform.field_transforms.contains_key("id")); - assert!(!transform.field_transforms["id"].is_replace); - assert_eq!(transform.field_transforms["id"].exprs.len(), 1); + assert!(patch.field_patches.contains_key("id")); + assert!(patch.field_patches["id"].keep_input); + assert_eq!(patch.field_patches["id"].insertions.len(), 1); - let Expression::Literal(scalar) = transform.field_transforms["id"].exprs[0].as_ref() else { + let Expression::Literal(scalar) = patch.field_patches["id"].insertions[0].as_ref() else { panic!("Expected literal"); }; assert_eq!(scalar, &Scalar::Integer(2024)); @@ -573,30 +658,25 @@ mod tests { &physical_schema, Some(4), /* base_row_id */ ); - let transform_expr = result.expect("Transform expression should be created successfully"); + let patch_expr = result.expect("StructPatch expression should be created successfully"); - let Expression::Transform(transform) = transform_expr.as_ref() else { - panic!("Expected Transform expression"); + let Expression::StructPatch(patch) = patch_expr.as_ref() else { + panic!("Expected StructPatch expression"); }; - assert!(transform.input_path.is_none()); - let row_id_transform = transform - .field_transforms + assert!(patch.input_path.is_none()); + let row_id_patch = patch + .field_patches .get("row_id_col") - .expect("Should have row_id_col transform"); - assert!(row_id_transform.is_replace); - - let expeceted_expr = Arc::new(Expression::coalesce([ - Expression::column(["row_id_col"]), - Expression::binary( - BinaryExpressionOp::Plus, - Expression::literal(4i64), - Expression::column(["row_index_col"]), - ), + .expect("Should have row_id_col patch"); + assert!(!row_id_patch.keep_input); + + let expected_expr = Arc::new(Expression::coalesce([ + col!("row_id_col"), + Expression::binary(BinaryExpressionOp::Plus, lit(4i64), col!("row_index_col")), ])); - assert_eq!(row_id_transform.exprs.len(), 1); - let expr = &row_id_transform.exprs[0]; - assert_eq!(expr, &expeceted_expr); + let expr = &row_id_patch.insertions[0]; + assert_eq!(expr, &expected_expr); } #[test] diff --git a/kernel/src/schema/column_default.rs b/kernel/src/schema/column_default.rs new file mode 100644 index 0000000000..6b02689d8b --- /dev/null +++ b/kernel/src/schema/column_default.rs @@ -0,0 +1,426 @@ +//! A typed wrapper around Delta's `CURRENT_DEFAULT` column metadata. +//! +//! # Invariants +//! +//! The rules Kernel applies to column defaults, and where each comes from. This applies to +//! [`ColumnDefault::new`], [`validate_column_defaults_metadata`], and the IcebergCompatV3 warnings +//! in [`crate::table_features`]. Rules that follow the protocol are stated plainly; only kernel +//! divergences are called out as such. +//! +//! - `CURRENT_DEFAULT` metadata must be a string. +//! - A Variant column must default to NULL; a non-NULL variant default is rejected. +//! - A non-NULL default on an Array, Map, or Struct column is protocol-legal but kernel cannot +//! materialize it, so it is tolerated and surfaced via raw SQL (`to_scalar` returns `None`). +//! - Defaults may appear on nested struct fields, not just top-level columns, so validation +//! descends into nested fields. Kernel only *writes* top-level defaults (a kernel limitation), +//! but *loads* a snapshot of a table authored elsewhere that carries nested defaults. +//! - Parsing is best-effort: unparseable SQL (e.g. `current_timestamp()`) is not an error; the +//! connector falls back to the raw SQL. +//! - On an IcebergCompatV3 table, kernel warns when its parser cannot verify that a default is a +//! literal. + +use crate::expressions::{parse_sql, Expression, Scalar}; +use crate::schema::{DataType, StructField, StructType}; +use crate::transforms::{transform_output_type, SchemaTransform}; +use crate::{DeltaResult, Error}; + +/// A column-level default parsed from the `CURRENT_DEFAULT` metadata key of a +/// [`StructField`](crate::schema::StructField). +/// +/// Holds the raw SQL and the column's declared type. On construction the kernel parses the SQL +/// with its built-in parser and caches the result. [`to_scalar`](Self::to_scalar) returns the +/// parsed [`Scalar`], or `None` when the kernel could not parse the SQL (e.g. +/// `current_timestamp()`), in which case a connector can evaluate [`raw_sql`](Self::raw_sql) +/// itself. +/// +/// The declared type is borrowed from the logical schema, which outlives the carrier. +#[derive(Debug, Clone, PartialEq)] +pub struct ColumnDefault<'a> { + raw_sql: String, + data_type: &'a DataType, + /// The default parsed as a kernel [`Expression`], or `None` if the kernel's + /// built-in SQL parser could not parse it. + parsed_sql: Option, +} + +impl<'a> ColumnDefault<'a> { + /// Build a `ColumnDefault` from a raw SQL string and the column's declared type. + /// + /// Parses `raw_sql` with the built-in SQL parser, targeting `data_type`. A parse failure is + /// not an error: the cached form is left empty and [`to_scalar`](Self::to_scalar) returns + /// `None` so the connector can handle the raw SQL. + /// + /// # Errors + /// + /// Returns an [`Error::schema`] when `data_type` is a Variant and `raw_sql` is not `NULL` + /// (case-insensitive). A non-`NULL` default on an Array, Map, or Struct column is accepted; + /// the kernel cannot parse it, so [`to_scalar`](Self::to_scalar) returns `None`. + pub(crate) fn new(raw_sql: String, data_type: &'a DataType) -> DeltaResult { + let is_null = raw_sql.trim().eq_ignore_ascii_case("null"); + + if matches!(data_type, DataType::Variant(_)) && !is_null { + return Err(Error::schema(format!( + "a Variant column's default must be NULL, got {raw_sql:?}" + ))); + } + let parsed_sql = parse_sql(&raw_sql, data_type).ok(); + Ok(Self { + raw_sql, + data_type, + parsed_sql, + }) + } + + /// The raw SQL expression as stored in the column's metadata. + pub fn raw_sql(&self) -> &str { + &self.raw_sql + } + + /// The declared type of the column whose default this is. + pub fn data_type(&self) -> &DataType { + self.data_type + } + + /// The default as a [`Scalar`], or `None` when the kernel could not parse the SQL. + /// + /// On `None` the connector can evaluate [`raw_sql`](Self::raw_sql) with its own SQL engine. + /// + /// # Errors + /// + /// Returns an error if the parsed default is not a literal. The parser only emits literals, so + /// this is defensive. + pub fn to_scalar(&self) -> DeltaResult> { + match &self.parsed_sql { + None => Ok(None), + Some(Expression::Literal(scalar)) => Ok(Some(scalar.clone())), + Some(other) => Err(Error::generic(format!( + "kernel cannot evaluate non-literal column default expression: {other:?}" + ))), + } + } + + /// Returns `true` iff the default parsed to a literal expression. + /// + /// Returns `false` for SQL the kernel could not parse (e.g. arithmetic or function calls). Note + /// that NULL parses to a literal, so this is `true` for a NULL default regardless of the + /// column type. + pub(crate) fn is_kernel_parsable_literal(&self) -> bool { + matches!(self.parsed_sql, Some(Expression::Literal(_))) + } +} + +/// Walks a schema and returns the parsed [`ColumnDefault`] of every field carrying +/// a `CURRENT_DEFAULT`, descending through nested structs, arrays, and maps (see the nesting +/// invariant in the module docs). Each default is paired with a human-readable path. +/// +/// The path is a dot-joined string intended only for error messages. +/// +/// # Errors +/// +/// Propagates any error from +/// [`StructField::column_default`](crate::schema::StructField::column_default): a `CURRENT_DEFAULT` +/// whose value is not a string, or a non-`NULL` default on a Variant column. +pub(crate) fn try_collect_column_defaults( + schema: &StructType, +) -> DeltaResult)>> { + let mut collector = ColumnDefaultCollector { + path: Vec::new(), + defaults: Vec::new(), + }; + collector.transform_struct(schema)?; + Ok(collector.defaults) +} + +/// Recursive [`SchemaTransform`] that gathers every field's column default with its display path. +/// +/// Container element types have no metadata of their own, so the array/map hooks only push a +/// synthetic path segment before recursing toward any nested struct. +struct ColumnDefaultCollector<'a> { + path: Vec, + defaults: Vec<(String, ColumnDefault<'a>)>, +} + +impl<'a> ColumnDefaultCollector<'a> { + /// Recurse into a container element type under a synthetic path `segment` + /// (`element` for arrays, `key`/`value` for maps). + fn descend(&mut self, segment: &str, element: &'a DataType) -> DeltaResult<()> { + self.path.push(segment.to_string()); + let result = self.transform(element); + self.path.pop(); + result + } +} + +impl<'a> SchemaTransform<'a> for ColumnDefaultCollector<'a> { + transform_output_type!(|'a, T| DeltaResult<()>); + + fn transform_struct_field(&mut self, field: &'a StructField) -> DeltaResult<()> { + self.path.push(field.name().clone()); + if let Some(column_default) = field.column_default()? { + self.defaults.push((self.path.join("."), column_default)); + } + let result = self.recurse_into_struct_field(field); + self.path.pop(); + result + } + + fn transform_array_element(&mut self, etype: &'a DataType) -> DeltaResult<()> { + self.descend("element", etype) + } + + fn transform_map_key(&mut self, ktype: &'a DataType) -> DeltaResult<()> { + self.descend("key", ktype) + } + + fn transform_map_value(&mut self, vtype: &'a DataType) -> DeltaResult<()> { + self.descend("value", vtype) + } + + fn transform_variant(&mut self, _stype: &'a StructType) -> DeltaResult<()> { + Ok(()) + } +} + +/// Validates the column-default metadata on a table's logical schema and reports whether any +/// field declares a default (see [Errors](#errors)). +/// +/// Run eagerly at [`TableConfiguration`] construction so an invalid table is rejected at load. +/// Inspects nested fields as well as top-level columns; see [`try_collect_column_defaults`]. +/// +/// This does not couple defaults to the `allowColumnDefaults` feature: a `CURRENT_DEFAULT` +/// present without the feature is orphaned metadata, which is allowed (see the module docs). +/// +/// [`TableConfiguration`]: crate::table_configuration::TableConfiguration +/// +/// # Errors +/// +/// Propagates any error from [`try_collect_column_defaults`]: a `CURRENT_DEFAULT` whose value is +/// not a string, or a non-NULL default on a Variant column. +pub(crate) fn validate_column_defaults_metadata(schema: &StructType) -> DeltaResult { + Ok(!try_collect_column_defaults(schema)?.is_empty()) +} + +/// A nullable field named `name` carrying `raw_sql` as its `CURRENT_DEFAULT` metadata. +/// +/// Shared across the crate's column-default unit tests (here, `schema`, and `transaction`). +#[cfg(test)] +pub(crate) fn field_with_default( + name: &str, + data_type: impl Into, + raw_sql: &str, +) -> StructField { + use crate::schema::{ColumnMetadataKey, MetadataValue}; + StructField::nullable(name, data_type).add_metadata([( + ColumnMetadataKey::CurrentDefault.as_ref().to_string(), + MetadataValue::String(raw_sql.to_string()), + )]) +} + +/// A nullable field named `name` whose `CURRENT_DEFAULT` metadata is a non-string value (corrupt). +#[cfg(test)] +pub(crate) fn field_with_invalid_default(name: &str) -> StructField { + use crate::schema::{ColumnMetadataKey, MetadataValue}; + StructField::nullable(name, DataType::INTEGER).add_metadata([( + ColumnMetadataKey::CurrentDefault.as_ref().to_string(), + MetadataValue::Number(7), + )]) +} + +#[cfg(test)] +mod tests { + use chrono::{DateTime, NaiveDate, TimeZone, Utc}; + use rstest::rstest; + + use super::*; + use crate::expressions::col; + use crate::schema::{ArrayType, MapType, StructField}; + + fn struct_ty() -> DataType { + DataType::try_struct_type([StructField::nullable("a", DataType::INTEGER)]).unwrap() + } + + /// A struct type whose single field `inner` carries an integer default. + fn struct_with_inner_default() -> DataType { + DataType::try_struct_type([field_with_default("inner", DataType::INTEGER, "42")]).unwrap() + } + + fn date_days(year: i32, month: u32, day: u32) -> i32 { + let nd = NaiveDate::from_ymd_opt(year, month, day) + .unwrap() + .and_hms_opt(0, 0, 0) + .unwrap(); + Utc.from_utc_datetime(&nd) + .signed_duration_since(DateTime::UNIX_EPOCH) + .num_days() as i32 + } + + /// Expected outcome of [`ColumnDefault::new`] for one `(raw_sql, data_type)` pair. + #[derive(Debug)] + enum Expect { + /// `to_scalar` yields `Some(this)`. + Parsed(Scalar), + /// `to_scalar` yields `Some(Scalar::Null)` of the column's type. + ParsedNull, + /// `to_scalar` yields `None`. + Unparsable, + /// `new` fails with an error containing this substring. + NewErr(&'static str), + } + + #[rstest] + #[case::integer("42", DataType::INTEGER, Expect::Parsed(Scalar::Integer(42)))] + #[case::string("'hello'", DataType::STRING, Expect::Parsed(Scalar::String("hello".into())))] + #[case::boolean("TRUE", DataType::BOOLEAN, Expect::Parsed(Scalar::Boolean(true)))] + #[case::date( + "DATE '2024-01-01'", + DataType::DATE, + Expect::Parsed(Scalar::Date(date_days(2024, 1, 1))) + )] + #[case::null_primitive("NULL", DataType::INTEGER, Expect::ParsedNull)] + #[case::null_array( + "NULL", + DataType::from(ArrayType::new(DataType::INTEGER, true)), + Expect::ParsedNull + )] + #[case::null_map( + "NULL", + DataType::from(MapType::new(DataType::STRING, DataType::INTEGER, true)), + Expect::ParsedNull + )] + #[case::null_struct("NULL", struct_ty(), Expect::ParsedNull)] + #[case::null_variant("NULL", DataType::unshredded_variant(), Expect::ParsedNull)] + #[case::function_call("current_timestamp()", DataType::TIMESTAMP, Expect::Unparsable)] + #[case::type_mismatch("'not an int'", DataType::INTEGER, Expect::Unparsable)] + #[case::arithmetic("1 + 1", DataType::INTEGER, Expect::Unparsable)] + #[case::non_primitive_array( + "ARRAY(1)", + DataType::from(ArrayType::new(DataType::INTEGER, true)), + Expect::Unparsable + )] + #[case::non_primitive_map( + "MAP('k', 1)", + DataType::from(MapType::new(DataType::STRING, DataType::INTEGER, true)), + Expect::Unparsable + )] + #[case::non_primitive_struct("STRUCT(1)", struct_ty(), Expect::Unparsable)] + #[case::non_null_variant("1", DataType::unshredded_variant(), Expect::NewErr("Variant"))] + fn column_default_from_new( + #[case] raw_sql: &str, + #[case] data_type: DataType, + #[case] expect: Expect, + ) { + match (ColumnDefault::new(raw_sql.into(), &data_type), expect) { + (Ok(d), Expect::Parsed(scalar)) => { + assert_eq!(d.raw_sql(), raw_sql); + assert_eq!(d.data_type(), &data_type); + assert_eq!(d.to_scalar().unwrap(), Some(scalar)); + } + (Ok(d), Expect::ParsedNull) => { + assert_eq!( + d.to_scalar().unwrap(), + Some(Scalar::Null(data_type.clone())) + ); + } + (Ok(d), Expect::Unparsable) => { + assert_eq!(d.to_scalar().unwrap(), None); + // A parse failure must not drop the raw SQL; connectors fall back to it. + assert_eq!(d.raw_sql(), raw_sql); + } + (Err(e), Expect::NewErr(needle)) => { + assert!(e.to_string().contains(needle), "got: {e}"); + } + (result, expect) => { + panic!("unexpected outcome for {raw_sql:?}: {result:?} vs {expect:?}") + } + } + } + + /// Validation is independent of the `allowColumnDefaults` feature: it checks only that the + /// metadata is well-formed (string-typed, primitive-or-NULL), never whether the feature is + /// enabled. `None` expects `Ok`; `Some(needle)` expects an error containing `needle`. + #[rstest] + #[case::well_formed_default( + vec![ + field_with_default("c", DataType::INTEGER, "42"), + StructField::nullable("no_default", DataType::STRING), + ], + true, + None + )] + #[case::no_defaults(vec![StructField::nullable("c", DataType::INTEGER)], false, None)] + #[case::non_string_metadata( + vec![field_with_invalid_default("c")], + false, + Some("non-string") + )] + #[case::non_null_default_on_array_tolerated( + vec![field_with_default("arr", ArrayType::new(DataType::INTEGER, true), "ARRAY(1)")], + true, + None + )] + #[case::non_null_default_on_variant_rejected( + vec![field_with_default("v", DataType::unshredded_variant(), "1")], + false, + Some("Variant") + )] + #[case::nested_default( + vec![StructField::nullable( + "s", + DataType::try_struct_type([field_with_default("inner", DataType::INTEGER, "42")]).unwrap(), + )], + true, + None + )] + fn validate_column_defaults_cases( + #[case] fields: Vec, + #[case] expected_has_default: bool, + #[case] expected_error: Option<&str>, + ) { + let schema = StructType::try_new(fields).unwrap(); + match (validate_column_defaults_metadata(&schema), expected_error) { + (Ok(has_default), None) => assert_eq!(has_default, expected_has_default), + (Err(e), Some(needle)) => assert!(e.to_string().contains(needle), "got: {e}"), + (result, expected) => panic!("unexpected outcome: {result:?} vs {expected:?}"), + } + } + + #[rstest] + #[case::array_element( + DataType::from(ArrayType::new(struct_with_inner_default(), true)), + ["arr", "element", "inner"] + )] + #[case::map_value( + DataType::from(MapType::new(DataType::STRING, struct_with_inner_default(), true)), + ["arr", "value", "inner"] + )] + #[case::map_key( + DataType::from(MapType::new(struct_with_inner_default(), DataType::INTEGER, true)), + ["arr", "key", "inner"] + )] + fn collect_nested_container_default_path( + #[case] container: DataType, + #[case] expected_path: [&str; 3], + ) { + let schema = StructType::try_new([StructField::nullable("arr", container)]).unwrap(); + let defaults = try_collect_column_defaults(&schema).unwrap(); + let [(path, _)] = defaults.try_into().expect("exactly one default"); + assert_eq!(path, expected_path.join(".")); + } + + #[test] + fn to_scalar_errors_on_non_literal_parsed_expression() { + // The parser only emits literals, so construct a non-literal directly to reach the + // defensive error arm. + let int_ty = DataType::INTEGER; + let d = ColumnDefault { + raw_sql: "x".into(), + data_type: &int_ty, + parsed_sql: Some(col!("x")), + }; + let err = d + .to_scalar() + .expect_err("non-literal parsed expression must error") + .to_string(); + assert!(err.contains("non-literal"), "got: {err}"); + } +} diff --git a/kernel/src/schema/compare.rs b/kernel/src/schema/compare.rs index 463e32140b..1c080d3268 100644 --- a/kernel/src/schema/compare.rs +++ b/kernel/src/schema/compare.rs @@ -22,18 +22,15 @@ //! [`Schema`]: crate::schema::Schema use std::collections::{HashMap, HashSet}; -use crate::utils::require; - use super::{DataType, StructField, StructType}; +use crate::utils::require; /// The nullability flag of a schema's field. This can be compared with a read schema field's /// nullability flag using [`Nullable::can_read_as`]. -#[allow(unused)] #[derive(Clone, Copy)] pub(crate) struct Nullable(bool); /// Represents the ways a schema comparison can fail. -#[allow(unused)] #[derive(Debug, thiserror::Error)] pub(crate) enum Error { #[error("The nullability was tightened for a field")] @@ -51,22 +48,34 @@ pub(crate) enum Error { } /// A [`std::result::Result`] that has the schema comparison [`Error`] as the error variant. -#[allow(unused)] pub(crate) type SchemaComparisonResult = Result<(), Error>; /// Represents a schema compatibility check for the type. If `self` can be read as `read_type`, /// this function returns `Ok(())`. Otherwise, this function returns `Err`. -/// -/// TODO (Oussama): Remove the `allow(unused)` once this is used in CDF. -#[allow(unused)] pub(crate) trait SchemaComparison { - fn can_read_as(&self, read_type: &Self) -> SchemaComparisonResult; + fn can_read_as(&self, read_type: &Self) -> SchemaComparisonResult { + self.can_read_as_internal(read_type, true) + } + + fn can_read_as_without_type_widening(&self, read_type: &Self) -> SchemaComparisonResult { + self.can_read_as_internal(read_type, false) + } + + fn can_read_as_internal( + &self, + read_type: &Self, + allow_type_widening: bool, + ) -> SchemaComparisonResult; } impl SchemaComparison for Nullable { /// Represents a nullability comparison between two schemas' fields. Returns true if the /// read nullability is the same or wider than the nullability of self. - fn can_read_as(&self, read_nullable: &Nullable) -> SchemaComparisonResult { + fn can_read_as_internal( + &self, + read_nullable: &Nullable, + _allow_type_widening: bool, + ) -> SchemaComparisonResult { // The case to avoid is when the column is nullable, but the read schema specifies the // column as non-nullable. So we avoid the case where !read_nullable && nullable // Hence we check that !(!read_nullable && existing_nullable) @@ -82,10 +91,16 @@ impl SchemaComparison for StructField { /// 1. The read schema field mustn't be non-nullable if this [`StructField`] is nullable. /// 2. The both this field and `read_field` must have the same name. /// 3. You can read this data type as the `read_field`'s data type. - fn can_read_as(&self, read_field: &Self) -> SchemaComparisonResult { - Nullable(self.nullable).can_read_as(&Nullable(read_field.nullable))?; + fn can_read_as_internal( + &self, + read_field: &Self, + allow_type_widening: bool, + ) -> SchemaComparisonResult { + Nullable(self.nullable) + .can_read_as_internal(&Nullable(read_field.nullable), allow_type_widening)?; require!(self.name() == read_field.name(), Error::FieldNameMismatch); - self.data_type().can_read_as(read_field.data_type())?; + self.data_type() + .can_read_as_internal(read_field.data_type(), allow_type_widening)?; Ok(()) } } @@ -97,7 +112,11 @@ impl SchemaComparison for StructType { /// 3. If a field in `read_type` is not present in this struct, then it must be nullable. /// 4. Both [`StructTypes`] must be valid schemas. No two fields of a struct may share a /// name that only differs by case. - fn can_read_as(&self, read_type: &Self) -> SchemaComparisonResult { + fn can_read_as_internal( + &self, + read_type: &Self, + allow_type_widening: bool, + ) -> SchemaComparisonResult { let lowercase_field_map: HashMap = self .fields .iter() @@ -124,7 +143,9 @@ impl SchemaComparison for StructType { } for read_field in read_type.fields() { match lowercase_field_map.get(&read_field.name().to_lowercase()) { - Some(existing_field) => existing_field.can_read_as(read_field)?, + Some(existing_field) => { + existing_field.can_read_as_internal(read_field, allow_type_widening)? + } None => { // Note: Delta spark does not perform the following check. Hence it ignores // non-null fields that exist in the read schema that aren't in this schema. @@ -140,33 +161,47 @@ impl SchemaComparison for DataType { /// Returns `Ok` if this [`DataType`] can be read as `read_type`. This is the case when: /// 1. The data types are the same, OR the source type can be widened to the target type /// (see [`PrimitiveType::can_widen_to`]) - /// 2. For complex data types, the nested types must be compatible as defined by [`SchemaComparison`] + /// 2. For complex data types, the nested types must be compatible as defined by + /// [`SchemaComparison`] /// 3. For array data types, the nullability may not be tightened in the `read_type`. See /// [`Nullable::can_read_as`] /// /// [`PrimitiveType::can_widen_to`]: super::PrimitiveType::can_widen_to - fn can_read_as(&self, read_type: &Self) -> SchemaComparisonResult { + fn can_read_as_internal( + &self, + read_type: &Self, + allow_type_widening: bool, + ) -> SchemaComparisonResult { match (self, read_type) { (Self::Array(self_array), Self::Array(read_array)) => { - Nullable(self_array.contains_null()) - .can_read_as(&Nullable(read_array.contains_null()))?; + Nullable(self_array.contains_null()).can_read_as_internal( + &Nullable(read_array.contains_null()), + allow_type_widening, + )?; self_array .element_type() - .can_read_as(read_array.element_type())?; + .can_read_as_internal(read_array.element_type(), allow_type_widening)?; } (Self::Struct(self_struct), Self::Struct(read_struct)) => { - self_struct.can_read_as(read_struct)? + self_struct.can_read_as_internal(read_struct, allow_type_widening)? } (Self::Map(self_map), Self::Map(read_map)) => { - Nullable(self_map.value_contains_null()) - .can_read_as(&Nullable(read_map.value_contains_null()))?; - self_map.key_type().can_read_as(read_map.key_type())?; - self_map.value_type().can_read_as(read_map.value_type())?; + Nullable(self_map.value_contains_null()).can_read_as_internal( + &Nullable(read_map.value_contains_null()), + allow_type_widening, + )?; + self_map + .key_type() + .can_read_as_internal(read_map.key_type(), allow_type_widening)?; + self_map + .value_type() + .can_read_as_internal(read_map.value_type(), allow_type_widening)?; } // Exact match (a, b) if a == b => {} // Type widening: smaller primitive types can be read as larger ones - (Self::Primitive(a), Self::Primitive(b)) if a.can_widen_to(b) => {} + (Self::Primitive(a), Self::Primitive(b)) + if allow_type_widening && a.can_widen_to(b) => {} // Any other type change is incompatible _ => return Err(Error::TypeMismatch), }; @@ -176,88 +211,84 @@ impl SchemaComparison for DataType { #[cfg(test)] mod tests { - use crate::schema::compare::{Error, SchemaComparison}; - use crate::schema::{ArrayType, DataType, MapType, StructField, StructType}; + use rstest::rstest; + + use crate::schema::compare::{Error, SchemaComparison, SchemaComparisonResult}; + use crate::schema::{ + schema, ArrayType, DataType, MapType, PrimitiveType, StructField, StructType, + }; + + #[derive(Clone, Copy, Debug, PartialEq, Eq)] + enum ComparisonMode { + AllowTypeWidening, + ForbidTypeWidening, + } + + impl ComparisonMode { + fn can_read_as( + self, + existing: &T, + read: &T, + ) -> SchemaComparisonResult { + match self { + Self::AllowTypeWidening => existing.can_read_as(read), + Self::ForbidTypeWidening => existing.can_read_as_without_type_widening(read), + } + } + + fn allows_type_widening(self) -> bool { + self == Self::AllowTypeWidening + } + } #[test] fn can_read_is_reflexive() { - let map_key = StructType::new_unchecked([ - StructField::new("id", DataType::LONG, false), - StructField::new("name", DataType::STRING, false), - ]); - let map_value = - StructType::new_unchecked([StructField::new("age", DataType::INTEGER, true)]); - let map_type = MapType::new(map_key, map_value, true); - let array_type = ArrayType::new(DataType::TIMESTAMP, false); - let nested_struct = StructType::new_unchecked([ - StructField::new("name", DataType::STRING, false), - StructField::new("age", DataType::INTEGER, true), - ]); - let schema = StructType::new_unchecked([ - StructField::new("id", DataType::LONG, false), - StructField::new("map", map_type, false), - StructField::new("array", array_type, false), - StructField::new("nested_struct", nested_struct, false), - ]); + let schema = schema! { + not_null "id": LONG, + not_null "map": { + { not_null "id": LONG, not_null "name": STRING } + => nullable { nullable "age": INTEGER } + }, + not_null "array": [ not_null TIMESTAMP ], + not_null "nested_struct": { not_null "name": STRING, nullable "age": INTEGER }, + }; assert!(schema.can_read_as(&schema).is_ok()); } - #[test] - fn add_nullable_column_to_map_key_and_value() { - let existing_map_key = StructType::new_unchecked([ - StructField::new("id", DataType::LONG, false), - StructField::new("name", DataType::STRING, true), - ]); - let existing_map_value = - StructType::new_unchecked([StructField::new("age", DataType::INTEGER, false)]); - let existing_schema = StructType::new_unchecked([StructField::new( - "map", - MapType::new(existing_map_key, existing_map_value, false), - false, - )]); - - let read_map_key = StructType::new_unchecked([ - StructField::new("id", DataType::LONG, false), - StructField::new("name", DataType::STRING, true), - StructField::new("location", DataType::STRING, true), - ]); - let read_map_value = StructType::new_unchecked([ - StructField::new("age", DataType::INTEGER, true), - StructField::new("years_of_experience", DataType::INTEGER, true), - ]); - let read_schema = StructType::new_unchecked([StructField::new( - "map", - MapType::new(read_map_key, read_map_value, false), - false, - )]); + #[rstest] + fn add_nullable_column_to_map_key_and_value( + #[values(ComparisonMode::AllowTypeWidening, ComparisonMode::ForbidTypeWidening)] + mode: ComparisonMode, + ) { + let existing_schema = schema! { + not_null "map": { + { not_null "id": LONG, nullable "name": STRING } + => not_null { not_null "age": INTEGER } + }, + }; - assert!(existing_schema.can_read_as(&read_schema).is_ok()); + let read_schema = schema! { + not_null "map": { + { not_null "id": LONG, nullable "name": STRING, nullable "location": STRING } + => not_null { nullable "age": INTEGER, nullable "years_of_experience": INTEGER } + }, + }; + + assert!(mode.can_read_as(&existing_schema, &read_schema).is_ok()); } #[test] fn map_value_becomes_non_nullable_fails() { - let map_key = StructType::new_unchecked([ - StructField::new("id", DataType::LONG, false), - StructField::new("name", DataType::STRING, false), - ]); - let map_value = - StructType::new_unchecked([StructField::new("age", DataType::INTEGER, true)]); - let existing_schema = StructType::new_unchecked([StructField::new( - "map", - MapType::new(map_key, map_value, false), - false, - )]); - - let map_key = StructType::new_unchecked([ - StructField::new("id", DataType::LONG, false), - StructField::new("name", DataType::STRING, false), - ]); - let map_value = - StructType::new_unchecked([StructField::new("age", DataType::INTEGER, false)]); - let read_schema = StructType::new_unchecked([StructField::new( - "map", - MapType::new(map_key, map_value, false), - false, - )]); + let existing_schema = schema! { + not_null "map": { + { not_null "id": LONG, not_null "name": STRING } => not_null { nullable "age": INTEGER } + }, + }; + + let read_schema = schema! { + not_null "map": { + { not_null "id": LONG, not_null "name": STRING } => not_null { not_null "age": INTEGER } + }, + }; assert!(matches!( existing_schema.can_read_as(&read_schema), @@ -351,8 +382,11 @@ mod tests { // Read `b` as `a`. `a` is missing a column that is present in `b`. assert!(matches!(b.can_read_as(&a), Err(Error::MissingColumn))); } - #[test] - fn differ_by_non_nullable_column() { + #[rstest] + fn differ_by_non_nullable_column( + #[values(ComparisonMode::AllowTypeWidening, ComparisonMode::ForbidTypeWidening)] + mode: ComparisonMode, + ) { let a = StructType::new_unchecked([ StructField::new("id", DataType::LONG, false), StructField::new("name", DataType::STRING, false), @@ -368,12 +402,15 @@ mod tests { // Read `a` as `b`. `b` has an extra non-nullable column. assert!(matches!( - a.can_read_as(&b), + mode.can_read_as(&a, &b), Err(Error::NewNonNullableColumn) )); // Read `b` as `a`. `a` is missing a column that is present in `b`. - assert!(matches!(b.can_read_as(&a), Err(Error::MissingColumn))); + assert!(matches!( + mode.can_read_as(&b, &a), + Err(Error::MissingColumn) + )); } #[test] @@ -403,45 +440,85 @@ mod tests { )); } - #[test] - fn type_widening_integer() { - // byte -> short -> int -> long - assert!(DataType::BYTE.can_read_as(&DataType::SHORT).is_ok()); - assert!(DataType::BYTE.can_read_as(&DataType::INTEGER).is_ok()); - assert!(DataType::BYTE.can_read_as(&DataType::LONG).is_ok()); - assert!(DataType::SHORT.can_read_as(&DataType::INTEGER).is_ok()); - assert!(DataType::SHORT.can_read_as(&DataType::LONG).is_ok()); - assert!(DataType::INTEGER.can_read_as(&DataType::LONG).is_ok()); - - // Cannot narrow types - assert!(matches!( - DataType::LONG.can_read_as(&DataType::INTEGER), - Err(Error::TypeMismatch) - )); - assert!(matches!( - DataType::INTEGER.can_read_as(&DataType::SHORT), - Err(Error::TypeMismatch) - )); - assert!(matches!( - DataType::SHORT.can_read_as(&DataType::BYTE), - Err(Error::TypeMismatch) - )); + #[rstest] + #[case::byte_to_short(DataType::BYTE, DataType::SHORT)] + #[case::byte_to_integer(DataType::BYTE, DataType::INTEGER)] + #[case::byte_to_long(DataType::BYTE, DataType::LONG)] + #[case::short_to_integer(DataType::SHORT, DataType::INTEGER)] + #[case::short_to_long(DataType::SHORT, DataType::LONG)] + #[case::integer_to_long(DataType::INTEGER, DataType::LONG)] + #[case::float_to_double(DataType::FLOAT, DataType::DOUBLE)] + #[case::timestamp_to_timestamp_ntz(DataType::TIMESTAMP, DataType::TIMESTAMP_NTZ)] + #[case::timestamp_ntz_to_timestamp(DataType::TIMESTAMP_NTZ, DataType::TIMESTAMP)] + fn primitive_type_widening( + #[case] source: DataType, + #[case] target: DataType, + #[values(ComparisonMode::AllowTypeWidening, ComparisonMode::ForbidTypeWidening)] + mode: ComparisonMode, + ) { + assert_eq!( + mode.can_read_as(&source, &target).is_ok(), + mode.allows_type_widening() + ); } - #[test] - fn type_widening_float() { - // float -> double - assert!(DataType::FLOAT.can_read_as(&DataType::DOUBLE).is_ok()); - - // Cannot narrow + #[rstest] + #[case::long_to_integer(DataType::LONG, DataType::INTEGER)] + #[case::integer_to_short(DataType::INTEGER, DataType::SHORT)] + #[case::short_to_byte(DataType::SHORT, DataType::BYTE)] + #[case::double_to_float(DataType::DOUBLE, DataType::FLOAT)] + fn primitive_type_narrowing_is_rejected( + #[case] source: DataType, + #[case] target: DataType, + #[values(ComparisonMode::AllowTypeWidening, ComparisonMode::ForbidTypeWidening)] + mode: ComparisonMode, + ) { assert!(matches!( - DataType::DOUBLE.can_read_as(&DataType::FLOAT), + mode.can_read_as(&source, &target), Err(Error::TypeMismatch) )); } - #[test] - fn type_widening_in_struct() { + #[rstest] + // Physical type reinterpretation (not general widening -- can_read_as rejects these) + #[case::integer_to_date(PrimitiveType::Integer, PrimitiveType::Date, true)] + #[case::long_to_timestamp(PrimitiveType::Long, PrimitiveType::Timestamp, true)] + #[case::long_to_timestamp_ntz(PrimitiveType::Long, PrimitiveType::TimestampNtz, true)] + // Reverse (narrowing) is rejected + #[case::date_to_integer(PrimitiveType::Date, PrimitiveType::Integer, false)] + #[case::timestamp_to_long(PrimitiveType::Timestamp, PrimitiveType::Long, false)] + #[case::timestamp_ntz_to_long(PrimitiveType::TimestampNtz, PrimitiveType::Long, false)] + // Cross-type combinations are rejected + #[case::long_to_date(PrimitiveType::Long, PrimitiveType::Date, false)] + #[case::integer_to_timestamp(PrimitiveType::Integer, PrimitiveType::Timestamp, false)] + #[case::integer_to_timestamp_ntz(PrimitiveType::Integer, PrimitiveType::TimestampNtz, false)] + #[case::byte_to_date(PrimitiveType::Byte, PrimitiveType::Date, false)] + // Identity + #[case::date_identity(PrimitiveType::Date, PrimitiveType::Date, true)] + #[case::timestamp_identity(PrimitiveType::Timestamp, PrimitiveType::Timestamp, true)] + #[case::long_identity(PrimitiveType::Long, PrimitiveType::Long, true)] + // Widening (superset of can_widen_to) + #[case::byte_to_long(PrimitiveType::Byte, PrimitiveType::Long, true)] + #[case::short_to_integer(PrimitiveType::Short, PrimitiveType::Integer, true)] + #[case::float_to_double(PrimitiveType::Float, PrimitiveType::Double, true)] + #[case::timestamp_to_ntz(PrimitiveType::Timestamp, PrimitiveType::TimestampNtz, true)] + fn stats_type_compatibility( + #[case] source: PrimitiveType, + #[case] target: PrimitiveType, + #[case] expected: bool, + ) { + assert_eq!( + source.is_stats_type_compatible_with(&target), + expected, + "{source:?} -> {target:?} should be {expected}" + ); + } + + #[rstest] + fn type_widening_in_struct( + #[values(ComparisonMode::AllowTypeWidening, ComparisonMode::ForbidTypeWidening)] + mode: ComparisonMode, + ) { let source = StructType::new_unchecked([ StructField::new("id", DataType::INTEGER, false), StructField::new("value", DataType::FLOAT, true), @@ -451,29 +528,61 @@ mod tests { StructField::new("value", DataType::DOUBLE, true), ]); - // Can widen types in struct fields - assert!(source.can_read_as(&target).is_ok()); + assert_eq!( + mode.can_read_as(&source, &target).is_ok(), + mode.allows_type_widening() + ); - // Cannot narrow assert!(matches!( - target.can_read_as(&source), + mode.can_read_as(&target, &source), Err(Error::TypeMismatch) )); } - #[test] - fn incompatible_type_change() { + #[rstest] + #[case::array( + DataType::from(ArrayType::new(DataType::INTEGER, false)), + DataType::from(ArrayType::new(DataType::LONG, false)) + )] + #[case::map_key( + DataType::from(MapType::new(DataType::INTEGER, DataType::STRING, false)), + DataType::from(MapType::new(DataType::LONG, DataType::STRING, false)) + )] + #[case::map_value( + DataType::from(MapType::new(DataType::STRING, DataType::INTEGER, false)), + DataType::from(MapType::new(DataType::STRING, DataType::LONG, false)) + )] + fn type_widening_in_complex_types( + #[case] source: DataType, + #[case] target: DataType, + #[values(ComparisonMode::AllowTypeWidening, ComparisonMode::ForbidTypeWidening)] + mode: ComparisonMode, + ) { + let result = mode.can_read_as(&source, &target); + match mode { + ComparisonMode::AllowTypeWidening => assert!(result.is_ok()), + ComparisonMode::ForbidTypeWidening => { + assert!(matches!(result, Err(Error::TypeMismatch))) + } + } + } + + #[rstest] + fn incompatible_type_change( + #[values(ComparisonMode::AllowTypeWidening, ComparisonMode::ForbidTypeWidening)] + mode: ComparisonMode, + ) { // Cannot change between incompatible types assert!(matches!( - DataType::STRING.can_read_as(&DataType::INTEGER), + mode.can_read_as(&DataType::STRING, &DataType::INTEGER), Err(Error::TypeMismatch) )); assert!(matches!( - DataType::INTEGER.can_read_as(&DataType::STRING), + mode.can_read_as(&DataType::INTEGER, &DataType::STRING), Err(Error::TypeMismatch) )); assert!(matches!( - DataType::BOOLEAN.can_read_as(&DataType::INTEGER), + mode.can_read_as(&DataType::BOOLEAN, &DataType::INTEGER), Err(Error::TypeMismatch) )); } diff --git a/kernel/src/schema/derive_macro_utils.rs b/kernel/src/schema/derive_macro_utils.rs index ebf2691f96..f04ce24190 100644 --- a/kernel/src/schema/derive_macro_utils.rs +++ b/kernel/src/schema/derive_macro_utils.rs @@ -3,12 +3,27 @@ /// Not intended for use by normal code. use std::collections::{HashMap, HashSet}; -use crate::schema::{ArrayType, DataType, MapType, StructField, ToSchema}; - +use bytes::Bytes; use delta_kernel_derive::internal_api; +use crate::error::add_scalar_path_context; +use crate::expressions::{Scalar, StructData}; +use crate::schema::{ArrayType, DataType, MapType, StructField, StructType, ToSchema}; +use crate::utils::require; +use crate::{DeltaResult, Error}; + /// Converts a type to a [`DataType`]. Implemented for the primitive types and automatically derived /// for all types that implement [`ToSchema`]. +/// +/// # Warning +/// +/// If a type implementing this trait also implements `Into`, then for every value `v`, the +/// scalar `s: Scalar = v.into()` **must** satisfy: +/// - `!s.is_null()`, and +/// - `s.data_type() == Self::to_data_type()`. +/// +/// `IntoScalar` automatically marks every type with both impls, and infallible conversions like +/// `impl From> for Scalar` rely on this contract without runtime validation. #[internal_api] pub(crate) trait ToDataType { fn to_data_type() -> DataType; @@ -36,10 +51,11 @@ macro_rules! impl_to_data_type { impl_to_data_type!( (String, DataType::STRING), + (Bytes, DataType::BINARY), (i64, DataType::LONG), (i32, DataType::INTEGER), (i16, DataType::SHORT), - (char, DataType::BYTE), + (i8, DataType::BYTE), (f32, DataType::FLOAT), (f64, DataType::DOUBLE), (bool, DataType::BOOLEAN) @@ -52,6 +68,13 @@ impl ToDataType for Vec { } } +// ToDataType impl for arrays that may contain null elements +impl ToDataType for Vec> { + fn to_data_type() -> DataType { + ArrayType::new(T::to_data_type(), true).into() + } +} + // ToDataType impl for non-nullable set types impl ToDataType for HashSet { fn to_data_type() -> DataType { @@ -123,3 +146,106 @@ impl GetNullableContainerStructField for T { StructField::not_null(name, T::to_nullable_container_type()) } } + +// Optional container types produce nullable fields with nullable values. +impl GetNullableContainerStructField for Option { + fn get_nullable_container_struct_field(name: impl Into) -> StructField { + StructField::nullable(name, T::to_nullable_container_type()) + } +} + +/// Named fields consumed by the [`delta_kernel_derive::TryFromStructData`] macro. +/// +/// Field conversion errors acquire their path element as they unwind. Successful conversion does +/// not allocate or maintain path state. +#[internal_api] +pub(crate) struct StructDataFields { + expected: StructType, + fields: HashMap, +} + +impl StructDataFields { + pub(crate) fn try_new(data: StructData, expected: StructType) -> DeltaResult { + let (actual_fields, values) = data.into_parts(); + require!( + actual_fields.len() == values.len(), + Error::scalar_conversion( + format!("{} struct values", actual_fields.len()), + format!("{} struct values", values.len()), + ) + ); + require!( + actual_fields.len() == expected.num_fields(), + Error::scalar_conversion( + format!("struct with {} fields", expected.num_fields()), + format!("struct with {} fields", actual_fields.len()), + ) + ); + let mut fields = HashMap::with_capacity(actual_fields.len()); + for (field, value) in actual_fields.into_iter().zip(values) { + let name = field.name().clone(); + match fields.entry(name) { + std::collections::hash_map::Entry::Vacant(entry) => { + entry.insert((field, value)); + } + std::collections::hash_map::Entry::Occupied(entry) => { + return Err(add_scalar_path_context( + Error::scalar_conversion("one field", "duplicate fields"), + entry.key().clone(), + )); + } + } + } + Ok(Self { expected, fields }) + } + + pub(crate) fn take_field>( + &mut self, + field_name: &str, + ) -> DeltaResult { + let expected = self.expected.field(field_name).ok_or_else(|| { + Error::InternalError(format!( + "Derived schema does not contain generated field {field_name:?}" + )) + })?; + let (actual_field, value) = self.fields.remove(field_name).ok_or_else(|| { + add_scalar_path_context( + Error::scalar_conversion("present field", "missing field"), + field_name, + ) + })?; + require!( + actual_field.is_nullable() == expected.is_nullable(), + add_scalar_path_context( + Error::scalar_conversion( + if expected.is_nullable() { + "nullable field" + } else { + "non-nullable field" + }, + if actual_field.is_nullable() { + "nullable field" + } else { + "non-nullable field" + }, + ), + field_name, + ) + ); + + T::try_from(value).map_err(|error| add_scalar_path_context(error, field_name)) + } + + /// Verifies that every named field was consumed. + pub(crate) fn finish(self) -> DeltaResult<()> { + if self.fields.is_empty() { + return Ok(()); + } + let mut extra: Vec<_> = self.fields.keys().collect(); + extra.sort_unstable(); + Err(Error::scalar_conversion( + "no additional fields", + format!("fields {extra:?}"), + )) + } +} diff --git a/kernel/src/schema/diff.rs b/kernel/src/schema/diff.rs index 54f62efa96..df70ccd1ad 100644 --- a/kernel/src/schema/diff.rs +++ b/kernel/src/schema/diff.rs @@ -7,9 +7,11 @@ // This module is not used outside the diff subsystem; suppress unused-code warnings. #![allow(dead_code)] -use super::{ColumnMetadataKey, ColumnName, DataType, MetadataValue, StructField, StructType}; use std::collections::{HashMap, HashSet}; +use super::{ColumnMetadataKey, ColumnName, DataType, MetadataValue, StructField, StructType}; +use crate::expressions::joined_column_name; + /// Represents the difference between two schemas #[derive(Debug, Clone, PartialEq)] pub(crate) struct SchemaDiff { @@ -28,7 +30,7 @@ pub(crate) struct SchemaDiff { pub(crate) struct FieldChange { /// The field that was added or removed pub(crate) field: StructField, - /// The path to this field (e.g., ColumnName::new(["user", "address", "street"])) + /// The path to this field (e.g., column_name!("user.address.street")) pub(crate) path: ColumnName, } @@ -39,7 +41,7 @@ pub(crate) struct FieldUpdate { pub(crate) before: StructField, /// The field as it exists in the new schema pub(crate) after: StructField, - /// The path to this field (e.g., ColumnName::new(["user", "address", "street"])) + /// The path to this field (e.g., column_name!("user.address.street")) pub(crate) path: ColumnName, /// The types of changes that occurred (can be multiple, e.g. renamed + nullability changed) pub(crate) change_types: Vec, @@ -108,7 +110,8 @@ impl SchemaDiff { self.added_fields.len() + self.removed_fields.len() + self.updated_fields.len() } - /// Returns true if there are any breaking changes (removed fields, type changes, or tightened nullability) + /// Returns true if there are any breaking changes (removed fields, type changes, or tightened + /// nullability) pub(crate) fn has_breaking_changes(&self) -> bool { self.breaking_changes } @@ -193,21 +196,18 @@ struct FieldWithPath { /// * `after` - The after/new schema to compare against /// /// # Returns -/// A `SchemaDiff` describing all changes including nested fields, or an error if the schemas are invalid +/// A `SchemaDiff` describing all changes including nested fields, or an error if the schemas are +/// invalid fn compute_schema_diff( before: &StructType, after: &StructType, ) -> Result { // Collect all fields with their paths from both schemas - let empty_path: Vec = vec![]; + let root = ColumnName::default(); let mut before_fields = Vec::new(); - collect_all_fields_with_paths( - before, - &ColumnName::new(empty_path.clone()), - &mut before_fields, - )?; + collect_all_fields_with_paths(before, &root, &mut before_fields)?; let mut after_fields = Vec::new(); - collect_all_fields_with_paths(after, &ColumnName::new(empty_path), &mut after_fields)?; + collect_all_fields_with_paths(after, &root, &mut after_fields)?; // Build maps by field ID let before_by_id = build_field_map_by_id(&before_fields)?; @@ -457,7 +457,7 @@ fn collect_fields_from_datatype( // See: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#writer-requirements-for-icebergcompatv2 // For arrays, we use "element" as the path segment and recurse into element type - let element_path = parent_path.join(&ColumnName::new(["element"])); + let element_path = joined_column_name!(parent_path, "element"); collect_fields_from_datatype(array_type.element_type(), &element_path, out)?; } DataType::Map(map_type) => { @@ -467,10 +467,10 @@ fn collect_fields_from_datatype( // See: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#writer-requirements-for-icebergcompatv2 // For maps, we use "key" and "value" as path segments and recurse into both types - let key_path = parent_path.join(&ColumnName::new(["key"])); + let key_path = joined_column_name!(parent_path, "key"); collect_fields_from_datatype(map_type.key_type(), &key_path, out)?; - let value_path = parent_path.join(&ColumnName::new(["value"])); + let value_path = joined_column_name!(parent_path, "value"); collect_fields_from_datatype(map_type.value_type(), &value_path, out)?; } _ => { @@ -601,8 +601,10 @@ fn check_field_nullability_change( /// Checks for container nullability changes. /// /// Returns: -/// - `Some(FieldChangeType::ContainerNullabilityLoosened)` if nullability was relaxed (false -> true) -/// - `Some(FieldChangeType::ContainerNullabilityTightened)` if nullability was restricted (true -> false) +/// - `Some(FieldChangeType::ContainerNullabilityLoosened)` if nullability was relaxed (false -> +/// true) +/// - `Some(FieldChangeType::ContainerNullabilityTightened)` if nullability was restricted (true -> +/// false) /// - `None` if nullability didn't change fn check_container_nullability_change( before_nullable: bool, @@ -618,14 +620,17 @@ fn check_container_nullability_change( /// Classifies a type change between two data types. /// /// Returns: -/// - A `Vec` containing all changes detected (type changed or container nullability changed) +/// - A `Vec` containing all changes detected (type changed or container +/// nullability changed) /// - An empty vec if the types are the same container with nested changes handled elsewhere /// /// This function handles the following cases: -/// 1. **Struct containers**: Changes to nested fields are captured via field IDs, so return empty vec +/// 1. **Struct containers**: Changes to nested fields are captured via field IDs, so return empty +/// vec /// 2. **Array containers**: /// - If element types match and only nullability changed, return the specific nullability change -/// - If element types are both structs with same nullability, nested changes handled via field IDs (return empty vec) +/// - If element types are both structs with same nullability, nested changes handled via field +/// IDs (return empty vec) /// - Otherwise, it's a type change /// 3. **Map containers**: Similar logic to arrays, but for both key and value types /// 4. **Different container types or primitives**: Type change @@ -728,13 +733,15 @@ fn has_metadata_changes(before: &StructField, after: &StructField) -> bool { #[cfg(test)] mod tests { + use std::collections::HashSet; + use super::*; + use crate::expressions::column_name; use crate::schema::{ArrayType, DataType, MapType, StructField, StructType}; - use std::collections::HashSet; fn create_field_with_id( name: &str, - data_type: DataType, + data_type: impl Into, nullable: bool, id: i64, ) -> StructField { @@ -798,7 +805,7 @@ mod tests { assert_eq!(diff.added_fields.len(), 1); assert_eq!(diff.removed_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 0); - assert_eq!(diff.added_fields[0].path, ColumnName::new(["name"])); + assert_eq!(diff.added_fields[0].path, column_name!("name")); assert_eq!(diff.added_fields[0].field.name(), "name"); assert!(diff.has_breaking_changes()); // Adding non-nullable field is breaking } @@ -952,7 +959,8 @@ mod tests { #[test] fn test_multiple_with_breaking_change() { - // Test that Multiple changes are correctly identified as breaking when they contain breaking changes + // Test that Multiple changes are correctly identified as breaking when they contain + // breaking changes let before = StructType::new_unchecked([create_field_with_id( "user_name", DataType::STRING, @@ -1000,8 +1008,8 @@ mod tests { match result { Err(SchemaDiffError::DuplicateFieldId { id, path1, path2 }) => { assert_eq!(id, 1); - assert_eq!(path1, ColumnName::new(["field1"])); - assert_eq!(path2, ColumnName::new(["field2"])); + assert_eq!(path1, column_name!("field1")); + assert_eq!(path2, column_name!("field2")); } _ => panic!("Expected DuplicateFieldId error"), } @@ -1021,16 +1029,16 @@ mod tests { added_fields: vec![ FieldChange { field: top_level_field.clone(), - path: ColumnName::new(["name"]), // Top-level (depth 1) + path: column_name!("name"), // Top-level (depth 1) }, FieldChange { field: nested_field.clone(), - path: ColumnName::new(["address", "street"]), // Nested (depth 2) + path: column_name!("address.street"), // Nested (depth 2) }, ], removed_fields: vec![FieldChange { field: deeply_nested_field.clone(), - path: ColumnName::new(["user", "address", "city"]), // Deeply nested (depth 3) + path: column_name!("user.address.city"), // Deeply nested (depth 3) }], updated_fields: vec![], breaking_changes: false, @@ -1039,19 +1047,16 @@ mod tests { // Test top_level_changes - should only return depth 1 fields let (top_added, top_removed, top_updated) = diff.top_level_changes(); assert_eq!(top_added.len(), 1); - assert_eq!(top_added[0].path, ColumnName::new(["name"])); + assert_eq!(top_added[0].path, column_name!("name")); assert_eq!(top_removed.len(), 0); assert_eq!(top_updated.len(), 0); // Test nested_changes - should only return depth > 1 fields let (nested_added, nested_removed, nested_updated) = diff.nested_changes(); assert_eq!(nested_added.len(), 1); - assert_eq!(nested_added[0].path, ColumnName::new(["address", "street"])); + assert_eq!(nested_added[0].path, column_name!("address.street")); assert_eq!(nested_removed.len(), 1); - assert_eq!( - nested_removed[0].path, - ColumnName::new(["user", "address", "city"]) - ); + assert_eq!(nested_removed[0].path, column_name!("user.address.city")); assert_eq!(nested_updated.len(), 0); } @@ -1089,17 +1094,18 @@ mod tests { let diff = SchemaDiff::new(&without_user, &with_user).unwrap(); assert_eq!(diff.added_fields.len(), 1); - assert_eq!(diff.added_fields[0].path, ColumnName::new(["user"])); + assert_eq!(diff.added_fields[0].path, column_name!("user")); assert_eq!(diff.removed_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 0); - // The filtered paths would have been: user.name, user.email, user.address, user.address.street, user.address.city + // The filtered paths would have been: user.name, user.email, user.address, + // user.address.street, user.address.city assert!(diff.has_breaking_changes()); // Adding non-nullable struct field is breaking // CASE 2: Removing a parent struct - only parent should be reported, not nested fields let diff = SchemaDiff::new(&with_user, &without_user).unwrap(); assert_eq!(diff.removed_fields.len(), 1); - assert_eq!(diff.removed_fields[0].path, ColumnName::new(["user"])); + assert_eq!(diff.removed_fields[0].path, column_name!("user")); assert_eq!(diff.added_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 0); assert!(!diff.has_breaking_changes()); // Removing fields is safe @@ -1112,7 +1118,7 @@ mod tests { let before = StructType::new_unchecked([]); let after = StructType::new_unchecked([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new( + ArrayType::new( DataType::try_struct_type([create_field_with_id( "name", DataType::STRING, @@ -1121,14 +1127,14 @@ mod tests { )]) .unwrap(), false, - ))), + ), true, 1, )]); let diff = SchemaDiff::new(&before, &after).unwrap(); assert_eq!(diff.added_fields.len(), 1); - assert_eq!(diff.added_fields[0].path, ColumnName::new(["items"])); + assert_eq!(diff.added_fields[0].path, column_name!("items")); let (nested_added, nested_removed, nested_updated) = diff.nested_changes(); assert_eq!(nested_added.len(), 0); @@ -1138,7 +1144,8 @@ mod tests { #[test] fn test_container_with_nested_changes_not_reported_as_type_change() { - // Test that when a struct's nested fields change, the struct itself isn't reported as TypeChanged + // Test that when a struct's nested fields change, the struct itself isn't reported as + // TypeChanged let before = StructType::new_unchecked([create_field_with_id( "user", DataType::try_struct_type([ @@ -1166,13 +1173,10 @@ mod tests { // Should see the nested field changes but NOT a type change on the parent struct assert_eq!(diff.added_fields.len(), 1); - assert_eq!(diff.added_fields[0].path, ColumnName::new(["user", "age"])); + assert_eq!(diff.added_fields[0].path, column_name!("user.age")); assert_eq!(diff.removed_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 1); - assert_eq!( - diff.updated_fields[0].path, - ColumnName::new(["user", "full_name"]) - ); + assert_eq!(diff.updated_fields[0].path, column_name!("user.full_name")); assert_eq!( diff.updated_fields[0].change_types, vec![FieldChangeType::Renamed] @@ -1218,7 +1222,7 @@ mod tests { assert_eq!(diff.added_fields.len(), 1); assert_eq!(diff.removed_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 1); - assert_eq!(diff.updated_fields[0].path, ColumnName::new(["data"])); + assert_eq!(diff.updated_fields[0].path, column_name!("data")); assert_eq!( diff.updated_fields[0].change_types, vec![FieldChangeType::TypeChanged] @@ -1226,10 +1230,7 @@ mod tests { assert!(diff.has_breaking_changes()); // The new nested field should also be reported as added - assert_eq!( - diff.added_fields[0].path, - ColumnName::new(["data", "nested"]) - ); + assert_eq!(diff.added_fields[0].path, column_name!("data.nested")); assert!(diff.has_breaking_changes()); } @@ -1238,7 +1239,7 @@ mod tests { // Test that array containers aren't reported as changed when their struct elements change let before = StructType::new_unchecked([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new( + ArrayType::new( DataType::try_struct_type([create_field_with_id( "name", DataType::STRING, @@ -1247,20 +1248,20 @@ mod tests { )]) .unwrap(), true, - ))), + ), false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new( + ArrayType::new( DataType::try_struct_type([ create_field_with_id("title", DataType::STRING, false, 2), // Renamed ]) .unwrap(), true, - ))), + ), false, 1, )]); @@ -1273,7 +1274,7 @@ mod tests { assert_eq!(diff.updated_fields.len(), 1); assert_eq!( diff.updated_fields[0].path, - ColumnName::new(["items", "element", "title"]) + column_name!("items.element.title") ); assert_eq!( diff.updated_fields[0].change_types, @@ -1284,7 +1285,7 @@ mod tests { let array_updates: Vec<_> = diff .updated_fields .iter() - .filter(|u| u.path == ColumnName::new(["items"])) + .filter(|u| u.path == column_name!("items")) .collect(); assert_eq!(array_updates.len(), 0); assert!(!diff.has_breaking_changes()); @@ -1335,15 +1336,16 @@ mod tests { let diff = SchemaDiff::new(&before, &after).unwrap(); - // Should see: existing changed, existing_struct changed, existing_struct.old_name->new_name renamed, new_struct added + // Should see: existing changed, existing_struct changed, existing_struct.old_name->new_name + // renamed, new_struct added assert_eq!(diff.added_fields.len(), 1); - assert_eq!(diff.added_fields[0].path, ColumnName::new(["new_struct"])); + assert_eq!(diff.added_fields[0].path, column_name!("new_struct")); assert_eq!(diff.updated_fields.len(), 3); let paths = updated_paths(&diff); - assert!(paths.contains(&ColumnName::new(["existing"]))); - assert!(paths.contains(&ColumnName::new(["existing_struct"]))); - assert!(paths.contains(&ColumnName::new(["existing_struct", "new_name"]))); + assert!(paths.contains(&column_name!("existing"))); + assert!(paths.contains(&column_name!("existing_struct"))); + assert!(paths.contains(&column_name!("existing_struct.new_name"))); // Added a non-nullable struct "new_struct" assert!(diff.has_breaking_changes()); @@ -1377,7 +1379,7 @@ mod tests { assert_eq!(diff.updated_fields.len(), 1); let update = &diff.updated_fields[0]; - assert_eq!(update.path, ColumnName::new(["user", "full_name"])); + assert_eq!(update.path, column_name!("user.full_name")); assert_eq!(update.change_types, vec![FieldChangeType::Renamed]); assert!(!diff.has_breaking_changes()); // Rename is not breaking } @@ -1409,7 +1411,7 @@ mod tests { assert_eq!(diff.updated_fields.len(), 0); let added = &diff.added_fields[0]; - assert_eq!(added.path, ColumnName::new(["user", "age"])); + assert_eq!(added.path, column_name!("user.age")); assert_eq!(added.field.name(), "age"); assert!(!diff.has_breaking_changes()); // Adding nullable field is not breaking } @@ -1435,21 +1437,33 @@ mod tests { 1, )]); - let after = StructType::new_unchecked([create_field_with_id( - "level1", - DataType::try_struct_type([create_field_with_id( - "level2", - DataType::try_struct_type([ - create_field_with_id("very_deep_field", DataType::STRING, false, 3), // Renamed! - ]) - .unwrap(), - false, - 2, - )]) - .unwrap(), - false, - 1, - )]); + let after = + StructType::new_unchecked([ + create_field_with_id( + "level1", + DataType::try_struct_type( + [ + create_field_with_id( + "level2", + DataType::try_struct_type([ + create_field_with_id( + "very_deep_field", + DataType::STRING, + false, + 3, + ), // Renamed! + ]) + .unwrap(), + false, + 2, + ), + ], + ) + .unwrap(), + false, + 1, + ), + ]); let diff = SchemaDiff::new(&before, &after).unwrap(); assert_eq!(diff.added_fields.len(), 0); @@ -1457,10 +1471,7 @@ mod tests { assert_eq!(diff.updated_fields.len(), 1); let update = &diff.updated_fields[0]; - assert_eq!( - update.path, - ColumnName::new(["level1", "level2", "very_deep_field"]) - ); + assert_eq!(update.path, column_name!("level1.level2.very_deep_field")); assert_eq!(update.change_types, vec![FieldChangeType::Renamed]); } @@ -1503,15 +1514,12 @@ mod tests { assert_eq!(top_added.len(), 0); assert_eq!(top_updated.len(), 1); - assert_eq!(top_updated[0].path, ColumnName::new(["renamed_top"])); + assert_eq!(top_updated[0].path, column_name!("renamed_top")); assert_eq!(nested_added.len(), 1); - assert_eq!(nested_added[0].path, ColumnName::new(["user", "age"])); + assert_eq!(nested_added[0].path, column_name!("user.age")); assert_eq!(nested_updated.len(), 1); - assert_eq!( - nested_updated[0].path, - ColumnName::new(["user", "full_name"]) - ); + assert_eq!(nested_updated[0].path, column_name!("user.full_name")); } #[test] @@ -1556,44 +1564,44 @@ mod tests { // Check specific changes let added_paths: HashSet = diff.added_fields.iter().map(|f| f.path.clone()).collect(); - assert!(added_paths.contains(&ColumnName::new(["user", "age"]))); - assert!(added_paths.contains(&ColumnName::new(["created_at"]))); + assert!(added_paths.contains(&column_name!("user.age"))); + assert!(added_paths.contains(&column_name!("created_at"))); let removed_paths: HashSet = diff.removed_fields.iter().map(|f| f.path.clone()).collect(); - assert!(removed_paths.contains(&ColumnName::new(["user", "email"]))); + assert!(removed_paths.contains(&column_name!("user.email"))); let paths = updated_paths(&diff); - assert!(paths.contains(&ColumnName::new(["identifier"]))); - assert!(paths.contains(&ColumnName::new(["user", "full_name"]))); + assert!(paths.contains(&column_name!("identifier"))); + assert!(paths.contains(&column_name!("user.full_name"))); } #[test] fn test_array_element_struct_field_changes() { let before = StructType::new_unchecked([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new( + ArrayType::new( DataType::try_struct_type([ create_field_with_id("name", DataType::STRING, false, 2), create_field_with_id("removed_field", DataType::INTEGER, true, 3), ]) .unwrap(), true, - ))), + ), false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new( + ArrayType::new( DataType::try_struct_type([ create_field_with_id("title", DataType::STRING, false, 2), // Renamed! create_field_with_id("added_field", DataType::STRING, true, 4), // Added! ]) .unwrap(), true, - ))), + ), false, 1, )]); @@ -1607,18 +1615,18 @@ mod tests { // Check added field assert_eq!( diff.added_fields[0].path, - ColumnName::new(["items", "element", "added_field"]) + column_name!("items.element.added_field") ); // Check removed field assert_eq!( diff.removed_fields[0].path, - ColumnName::new(["items", "element", "removed_field"]) + column_name!("items.element.removed_field") ); // Check updated field (rename) let update = &diff.updated_fields[0]; - assert_eq!(update.path, ColumnName::new(["items", "element", "title"])); + assert_eq!(update.path, column_name!("items.element.title")); assert_eq!(update.change_types, vec![FieldChangeType::Renamed]); assert!(!diff.has_breaking_changes()); // Removal is safe, rename is safe @@ -1626,23 +1634,18 @@ mod tests { #[test] fn test_doubly_nested_array_type_change() { - // Test that we can detect type changes in doubly nested arrays: array> -> array> + // Test that we can detect type changes in doubly nested arrays: array> -> + // array> let before = StructType::new_unchecked([create_field_with_id( "matrix", - DataType::Array(Box::new(ArrayType::new( - DataType::Array(Box::new(ArrayType::new(DataType::INTEGER, false))), - false, - ))), + ArrayType::new(ArrayType::new(DataType::INTEGER, false), false), false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "matrix", - DataType::Array(Box::new(ArrayType::new( - DataType::Array(Box::new(ArrayType::new(DataType::DOUBLE, false))), - false, - ))), + ArrayType::new(ArrayType::new(DataType::DOUBLE, false), false), false, 1, )]); @@ -1654,7 +1657,7 @@ mod tests { assert_eq!(diff.added_fields.len(), 0); assert_eq!(diff.removed_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 1); - assert_eq!(diff.updated_fields[0].path, ColumnName::new(["matrix"])); + assert_eq!(diff.updated_fields[0].path, column_name!("matrix")); assert_eq!( diff.updated_fields[0].change_types, vec![FieldChangeType::TypeChanged] @@ -1668,14 +1671,14 @@ mod tests { // Test direct primitive element type change: array -> array let before = StructType::new_unchecked([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new(DataType::STRING, false))), + ArrayType::new(DataType::STRING, false), false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new(DataType::INTEGER, false))), + ArrayType::new(DataType::INTEGER, false), false, 1, )]); @@ -1685,7 +1688,7 @@ mod tests { assert_eq!(diff.added_fields.len(), 0); assert_eq!(diff.removed_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 1); - assert_eq!(diff.updated_fields[0].path, ColumnName::new(["items"])); + assert_eq!(diff.updated_fields[0].path, column_name!("items")); assert_eq!( diff.updated_fields[0].change_types, vec![FieldChangeType::TypeChanged] @@ -1699,20 +1702,20 @@ mod tests { // Outer array element nullability loosened (safe change) let before = StructType::new_unchecked([create_field_with_id( "matrix", - DataType::Array(Box::new(ArrayType::new( - DataType::Array(Box::new(ArrayType::new(DataType::INTEGER, false))), + ArrayType::new( + ArrayType::new(DataType::INTEGER, false), false, // Outer array elements are non-nullable - ))), + ), false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "matrix", - DataType::Array(Box::new(ArrayType::new( - DataType::Array(Box::new(ArrayType::new(DataType::INTEGER, false))), + ArrayType::new( + ArrayType::new(DataType::INTEGER, false), true, // Outer array elements now nullable - ))), + ), false, 1, )]); @@ -1722,7 +1725,7 @@ mod tests { assert_eq!(diff.added_fields.len(), 0); assert_eq!(diff.removed_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 1); - assert_eq!(diff.updated_fields[0].path, ColumnName::new(["matrix"])); + assert_eq!(diff.updated_fields[0].path, column_name!("matrix")); assert_eq!( diff.updated_fields[0].change_types, vec![FieldChangeType::ContainerNullabilityLoosened] @@ -1736,20 +1739,20 @@ mod tests { // Outer array element nullability tightened (breaking change) let before = StructType::new_unchecked([create_field_with_id( "matrix", - DataType::Array(Box::new(ArrayType::new( - DataType::Array(Box::new(ArrayType::new(DataType::INTEGER, false))), + ArrayType::new( + ArrayType::new(DataType::INTEGER, false), true, // Outer array elements are nullable - ))), + ), false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "matrix", - DataType::Array(Box::new(ArrayType::new( - DataType::Array(Box::new(ArrayType::new(DataType::INTEGER, false))), + ArrayType::new( + ArrayType::new(DataType::INTEGER, false), false, // Outer array elements now non-nullable - ))), + ), false, 1, )]); @@ -1759,7 +1762,7 @@ mod tests { assert_eq!(diff.added_fields.len(), 0); assert_eq!(diff.removed_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 1); - assert_eq!(diff.updated_fields[0].path, ColumnName::new(["matrix"])); + assert_eq!(diff.updated_fields[0].path, column_name!("matrix")); assert_eq!( diff.updated_fields[0].change_types, vec![FieldChangeType::ContainerNullabilityTightened] @@ -1773,20 +1776,20 @@ mod tests { // Inner array element nullability loosened (safe change) let before = StructType::new_unchecked([create_field_with_id( "matrix", - DataType::Array(Box::new(ArrayType::new( - DataType::Array(Box::new(ArrayType::new(DataType::INTEGER, false))), // Inner elements non-nullable + ArrayType::new( + ArrayType::new(DataType::INTEGER, false), /* Inner elements non-nullable */ false, - ))), + ), false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "matrix", - DataType::Array(Box::new(ArrayType::new( - DataType::Array(Box::new(ArrayType::new(DataType::INTEGER, true))), // Inner elements now nullable + ArrayType::new( + ArrayType::new(DataType::INTEGER, true), /* Inner elements now nullable */ false, - ))), + ), false, 1, )]); @@ -1796,7 +1799,7 @@ mod tests { assert_eq!(diff.added_fields.len(), 0); assert_eq!(diff.removed_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 1); - assert_eq!(diff.updated_fields[0].path, ColumnName::new(["matrix"])); + assert_eq!(diff.updated_fields[0].path, column_name!("matrix")); assert_eq!( diff.updated_fields[0].change_types, vec![FieldChangeType::ContainerNullabilityLoosened] @@ -1810,20 +1813,20 @@ mod tests { // Inner array element nullability tightened (breaking change) let before = StructType::new_unchecked([create_field_with_id( "matrix", - DataType::Array(Box::new(ArrayType::new( - DataType::Array(Box::new(ArrayType::new(DataType::INTEGER, true))), // Inner elements nullable + ArrayType::new( + ArrayType::new(DataType::INTEGER, true), /* Inner elements nullable */ false, - ))), + ), false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "matrix", - DataType::Array(Box::new(ArrayType::new( - DataType::Array(Box::new(ArrayType::new(DataType::INTEGER, false))), // Inner elements now non-nullable + ArrayType::new( + ArrayType::new(DataType::INTEGER, false), /* Inner elements now non-nullable */ false, - ))), + ), false, 1, )]); @@ -1833,7 +1836,7 @@ mod tests { assert_eq!(diff.added_fields.len(), 0); assert_eq!(diff.removed_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 1); - assert_eq!(diff.updated_fields[0].path, ColumnName::new(["matrix"])); + assert_eq!(diff.updated_fields[0].path, column_name!("matrix")); assert_eq!( diff.updated_fields[0].change_types, vec![FieldChangeType::ContainerNullabilityTightened] @@ -1845,14 +1848,15 @@ mod tests { fn test_array_nullability_loosened() { let before = StructType::new_unchecked([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new(DataType::STRING, false))), // Non-nullable elements + ArrayType::new(DataType::STRING, false), /* Non-nullable + * elements */ false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new(DataType::STRING, true))), // Nullable elements now + ArrayType::new(DataType::STRING, true), /* Nullable elements now */ false, 1, )]); @@ -1862,7 +1866,7 @@ mod tests { assert_eq!(diff.added_fields.len(), 0); assert_eq!(diff.removed_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 1); - assert_eq!(diff.updated_fields[0].path, ColumnName::new(["items"])); + assert_eq!(diff.updated_fields[0].path, column_name!("items")); assert_eq!( diff.updated_fields[0].change_types, vec![FieldChangeType::ContainerNullabilityLoosened] @@ -1874,14 +1878,14 @@ mod tests { fn test_array_nullability_tightened() { let before = StructType::new_unchecked([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new(DataType::STRING, true))), // Nullable elements + ArrayType::new(DataType::STRING, true), // Nullable elements false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new(DataType::STRING, false))), // Non-nullable now + ArrayType::new(DataType::STRING, false), // Non-nullable now false, 1, )]); @@ -1891,7 +1895,7 @@ mod tests { assert_eq!(diff.added_fields.len(), 0); assert_eq!(diff.removed_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 1); - assert_eq!(diff.updated_fields[0].path, ColumnName::new(["items"])); + assert_eq!(diff.updated_fields[0].path, column_name!("items")); assert_eq!( diff.updated_fields[0].change_types, vec![FieldChangeType::ContainerNullabilityTightened] @@ -1902,7 +1906,7 @@ mod tests { fn test_map_value_struct_field_changes() { let before = StructType::new_unchecked([create_field_with_id( "lookup", - DataType::Map(Box::new(MapType::new( + MapType::new( DataType::STRING, DataType::try_struct_type([ create_field_with_id("value", DataType::INTEGER, false, 2), @@ -1910,14 +1914,14 @@ mod tests { ]) .unwrap(), true, - ))), + ), false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "lookup", - DataType::Map(Box::new(MapType::new( + MapType::new( DataType::STRING, DataType::try_struct_type([ create_field_with_id("count", DataType::INTEGER, false, 2), // Renamed! @@ -1925,7 +1929,7 @@ mod tests { ]) .unwrap(), true, - ))), + ), false, 1, )]); @@ -1939,18 +1943,18 @@ mod tests { // Check added field assert_eq!( diff.added_fields[0].path, - ColumnName::new(["lookup", "value", "added_field"]) + column_name!("lookup.value.added_field") ); // Check removed field assert_eq!( diff.removed_fields[0].path, - ColumnName::new(["lookup", "value", "removed_field"]) + column_name!("lookup.value.removed_field") ); // Check updated field (rename) let update = &diff.updated_fields[0]; - assert_eq!(update.path, ColumnName::new(["lookup", "value", "count"])); + assert_eq!(update.path, column_name!("lookup.value.count")); assert_eq!(update.change_types, vec![FieldChangeType::Renamed]); assert!(!diff.has_breaking_changes()); // Removal is safe, rename is safe @@ -1962,7 +1966,7 @@ mod tests { // Struct element nullability loosened (safe change) let before = StructType::new_unchecked([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new( + ArrayType::new( DataType::try_struct_type([create_field_with_id( "name", DataType::STRING, @@ -1971,14 +1975,14 @@ mod tests { )]) .unwrap(), false, // Struct elements non-nullable - ))), + ), false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new( + ArrayType::new( DataType::try_struct_type([create_field_with_id( "name", DataType::STRING, @@ -1987,7 +1991,7 @@ mod tests { )]) .unwrap(), true, // Struct elements now nullable - ))), + ), false, 1, )]); @@ -1997,7 +2001,7 @@ mod tests { assert_eq!(diff.added_fields.len(), 0); assert_eq!(diff.removed_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 1); - assert_eq!(diff.updated_fields[0].path, ColumnName::new(["items"])); + assert_eq!(diff.updated_fields[0].path, column_name!("items")); assert_eq!( diff.updated_fields[0].change_types, vec![FieldChangeType::ContainerNullabilityLoosened] @@ -2009,22 +2013,22 @@ mod tests { fn test_map_nullability_loosened() { let before = StructType::new_unchecked([create_field_with_id( "lookup", - DataType::Map(Box::new(MapType::new( + MapType::new( DataType::STRING, DataType::INTEGER, false, // Non-nullable values - ))), + ), false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "lookup", - DataType::Map(Box::new(MapType::new( + MapType::new( DataType::STRING, DataType::INTEGER, true, // Nullable values now - ))), + ), false, 1, )]); @@ -2034,7 +2038,7 @@ mod tests { assert_eq!(diff.added_fields.len(), 0); assert_eq!(diff.removed_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 1); - assert_eq!(diff.updated_fields[0].path, ColumnName::new(["lookup"])); + assert_eq!(diff.updated_fields[0].path, column_name!("lookup")); assert_eq!( diff.updated_fields[0].change_types, vec![FieldChangeType::ContainerNullabilityLoosened] @@ -2048,7 +2052,7 @@ mod tests { // Struct element nullability tightened (breaking change) let before = StructType::new_unchecked([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new( + ArrayType::new( DataType::try_struct_type([create_field_with_id( "name", DataType::STRING, @@ -2057,14 +2061,14 @@ mod tests { )]) .unwrap(), true, // Struct elements nullable - ))), + ), false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new( + ArrayType::new( DataType::try_struct_type([create_field_with_id( "name", DataType::STRING, @@ -2073,7 +2077,7 @@ mod tests { )]) .unwrap(), false, // Struct elements now non-nullable - ))), + ), false, 1, )]); @@ -2083,7 +2087,7 @@ mod tests { assert_eq!(diff.added_fields.len(), 0); assert_eq!(diff.removed_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 1); - assert_eq!(diff.updated_fields[0].path, ColumnName::new(["items"])); + assert_eq!(diff.updated_fields[0].path, column_name!("items")); assert_eq!( diff.updated_fields[0].change_types, vec![FieldChangeType::ContainerNullabilityTightened] @@ -2095,22 +2099,22 @@ mod tests { fn test_map_nullability_tightened() { let before = StructType::new_unchecked([create_field_with_id( "lookup", - DataType::Map(Box::new(MapType::new( + MapType::new( DataType::STRING, DataType::INTEGER, true, // Nullable values - ))), + ), false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "lookup", - DataType::Map(Box::new(MapType::new( + MapType::new( DataType::STRING, DataType::INTEGER, false, // Non-nullable now - ))), + ), false, 1, )]); @@ -2120,7 +2124,7 @@ mod tests { assert_eq!(diff.added_fields.len(), 0); assert_eq!(diff.removed_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 1); - assert_eq!(diff.updated_fields[0].path, ColumnName::new(["lookup"])); + assert_eq!(diff.updated_fields[0].path, column_name!("lookup")); assert_eq!( diff.updated_fields[0].change_types, vec![FieldChangeType::ContainerNullabilityTightened] @@ -2135,14 +2139,15 @@ mod tests { // After: items: array (elements nullable, type changed) let before = StructType::new_unchecked([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new(DataType::STRING, false))), // Non-nullable elements + ArrayType::new(DataType::STRING, false), /* Non-nullable + * elements */ false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new(DataType::INTEGER, true))), // Nullable, type changed + ArrayType::new(DataType::INTEGER, true), /* Nullable, type changed */ false, 1, )]); @@ -2152,7 +2157,7 @@ mod tests { assert_eq!(diff.added_fields.len(), 0); assert_eq!(diff.removed_fields.len(), 0); assert_eq!(diff.updated_fields.len(), 1); - assert_eq!(diff.updated_fields[0].path, ColumnName::new(["items"])); + assert_eq!(diff.updated_fields[0].path, column_name!("items")); // Should have both TypeChanged and ContainerNullabilityLoosened let change_types = &diff.updated_fields[0].change_types; assert!(change_types.contains(&FieldChangeType::TypeChanged)); @@ -2165,7 +2170,7 @@ mod tests { // Test that maps with struct keys can be diffed let before = StructType::new_unchecked([create_field_with_id( "lookup", - DataType::Map(Box::new(MapType::new( + MapType::new( DataType::try_struct_type([create_field_with_id( "id", DataType::INTEGER, @@ -2175,14 +2180,14 @@ mod tests { .unwrap(), DataType::STRING, true, - ))), + ), false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "lookup", - DataType::Map(Box::new(MapType::new( + MapType::new( DataType::try_struct_type([create_field_with_id( "identifier", // Renamed key field DataType::INTEGER, @@ -2192,7 +2197,7 @@ mod tests { .unwrap(), DataType::STRING, true, - ))), + ), false, 1, )]); @@ -2205,7 +2210,7 @@ mod tests { assert_eq!(diff.updated_fields.len(), 1); assert_eq!( diff.updated_fields[0].path, - ColumnName::new(["lookup", "key", "identifier"]) + column_name!("lookup.key.identifier") ); assert_eq!( diff.updated_fields[0].change_types, @@ -2220,7 +2225,7 @@ mod tests { "data", DataType::try_struct_type([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new( + ArrayType::new( DataType::try_struct_type([create_field_with_id( "inner", DataType::try_struct_type([ @@ -2233,7 +2238,7 @@ mod tests { )]) .unwrap(), false, - ))), + ), false, 5, )]) @@ -2242,31 +2247,50 @@ mod tests { 1, )]); - let after = StructType::new_unchecked([create_field_with_id( - "data", - DataType::try_struct_type([create_field_with_id( - "items", - DataType::Array(Box::new(ArrayType::new( - DataType::try_struct_type([create_field_with_id( - "inner", - DataType::try_struct_type([ - create_field_with_id("renamed_a", DataType::INTEGER, false, 3), // Renamed! - create_field_with_id("added", DataType::LONG, true, 6), // Added! - ]) - .unwrap(), - false, - 2, - )]) - .unwrap(), - false, - ))), + let after = + StructType::new_unchecked([create_field_with_id( + "data", + DataType::try_struct_type( + [ + create_field_with_id( + "items", + ArrayType::new( + DataType::try_struct_type( + [ + create_field_with_id( + "inner", + DataType::try_struct_type([ + create_field_with_id( + "renamed_a", + DataType::INTEGER, + false, + 3, + ), // Renamed! + create_field_with_id( + "added", + DataType::LONG, + true, + 6, + ), // Added! + ]) + .unwrap(), + false, + 2, + ), + ], + ) + .unwrap(), + false, + ), + false, + 5, + ), + ], + ) + .unwrap(), false, - 5, - )]) - .unwrap(), - false, - 1, - )]); + 1, + )]); let diff = SchemaDiff::new(&before, &after).unwrap(); @@ -2275,15 +2299,15 @@ mod tests { assert_eq!(diff.updated_fields.len(), 1); assert_eq!( diff.added_fields[0].path, - ColumnName::new(["data", "items", "element", "inner", "added"]) + column_name!("data.items.element.inner.added") ); assert_eq!( diff.removed_fields[0].path, - ColumnName::new(["data", "items", "element", "inner", "removed"]) + column_name!("data.items.element.inner.removed") ); assert_eq!( diff.updated_fields[0].path, - ColumnName::new(["data", "items", "element", "inner", "renamed_a"]) + column_name!("data.items.element.inner.renamed_a") ); assert_eq!( diff.updated_fields[0].change_types, @@ -2297,11 +2321,11 @@ mod tests { // map>>> let before = StructType::new_unchecked([create_field_with_id( "lookup", - DataType::Map(Box::new(MapType::new( + MapType::new( DataType::STRING, DataType::try_struct_type([create_field_with_id( "nested", - DataType::Map(Box::new(MapType::new( + MapType::new( DataType::INTEGER, DataType::try_struct_type([create_field_with_id( "x", @@ -2311,40 +2335,50 @@ mod tests { )]) .unwrap(), false, - ))), + ), false, 2, )]) .unwrap(), false, - ))), + ), false, 1, )]); - let after = StructType::new_unchecked([create_field_with_id( - "lookup", - DataType::Map(Box::new(MapType::new( - DataType::STRING, - DataType::try_struct_type([create_field_with_id( - "nested", - DataType::Map(Box::new(MapType::new( - DataType::INTEGER, - DataType::try_struct_type([ - create_field_with_id("renamed_x", DataType::INTEGER, false, 3), // Renamed! - ]) - .unwrap(), - false, - ))), + let after = + StructType::new_unchecked([create_field_with_id( + "lookup", + MapType::new( + DataType::STRING, + DataType::try_struct_type( + [ + create_field_with_id( + "nested", + MapType::new( + DataType::INTEGER, + DataType::try_struct_type([ + create_field_with_id( + "renamed_x", + DataType::INTEGER, + false, + 3, + ), // Renamed! + ]) + .unwrap(), + false, + ), + false, + 2, + ), + ], + ) + .unwrap(), false, - 2, - )]) - .unwrap(), + ), false, - ))), - false, - 1, - )]); + 1, + )]); let diff = SchemaDiff::new(&before, &after).unwrap(); @@ -2353,7 +2387,7 @@ mod tests { assert_eq!(diff.updated_fields.len(), 1); assert_eq!( diff.updated_fields[0].path, - ColumnName::new(["lookup", "value", "nested", "value", "renamed_x"]) + column_name!("lookup.value.nested.value.renamed_x") ); assert_eq!( diff.updated_fields[0].change_types, @@ -2367,8 +2401,8 @@ mod tests { // array>> let before = StructType::new_unchecked([create_field_with_id( "matrix", - DataType::Array(Box::new(ArrayType::new( - DataType::Array(Box::new(ArrayType::new( + ArrayType::new( + ArrayType::new( DataType::try_struct_type([create_field_with_id( "x", DataType::INTEGER, @@ -2377,26 +2411,26 @@ mod tests { )]) .unwrap(), false, - ))), + ), false, - ))), + ), false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "matrix", - DataType::Array(Box::new(ArrayType::new( - DataType::Array(Box::new(ArrayType::new( + ArrayType::new( + ArrayType::new( DataType::try_struct_type([ create_field_with_id("renamed_x", DataType::INTEGER, false, 2), // Renamed! create_field_with_id("y", DataType::INTEGER, true, 3), // Added! ]) .unwrap(), false, - ))), + ), false, - ))), + ), false, 1, )]); @@ -2408,11 +2442,11 @@ mod tests { assert_eq!(diff.updated_fields.len(), 1); assert_eq!( diff.added_fields[0].path, - ColumnName::new(["matrix", "element", "element", "y"]) + column_name!("matrix.element.element.y") ); assert_eq!( diff.updated_fields[0].path, - ColumnName::new(["matrix", "element", "element", "renamed_x"]) + column_name!("matrix.element.element.renamed_x") ); assert_eq!( diff.updated_fields[0].change_types, @@ -2426,8 +2460,8 @@ mod tests { // map>, array>> let before = StructType::new_unchecked([create_field_with_id( "complex_map", - DataType::Map(Box::new(MapType::new( - DataType::Array(Box::new(ArrayType::new( + MapType::new( + ArrayType::new( DataType::try_struct_type([create_field_with_id( "key_field", DataType::INTEGER, @@ -2436,8 +2470,8 @@ mod tests { )]) .unwrap(), false, - ))), - DataType::Array(Box::new(ArrayType::new( + ), + ArrayType::new( DataType::try_struct_type([create_field_with_id( "value_field", DataType::STRING, @@ -2446,35 +2480,48 @@ mod tests { )]) .unwrap(), false, - ))), + ), false, - ))), + ), false, 1, )]); - let after = StructType::new_unchecked([create_field_with_id( - "complex_map", - DataType::Map(Box::new(MapType::new( - DataType::Array(Box::new(ArrayType::new( - DataType::try_struct_type([ - create_field_with_id("renamed_key_field", DataType::INTEGER, false, 2), // Renamed! - ]) - .unwrap(), - false, - ))), - DataType::Array(Box::new(ArrayType::new( - DataType::try_struct_type([ - create_field_with_id("renamed_value_field", DataType::STRING, false, 3), // Renamed! - ]) - .unwrap(), + let after = + StructType::new_unchecked([ + create_field_with_id( + "complex_map", + MapType::new( + ArrayType::new( + DataType::try_struct_type([ + create_field_with_id( + "renamed_key_field", + DataType::INTEGER, + false, + 2, + ), // Renamed! + ]) + .unwrap(), + false, + ), + ArrayType::new( + DataType::try_struct_type([ + create_field_with_id( + "renamed_value_field", + DataType::STRING, + false, + 3, + ), // Renamed! + ]) + .unwrap(), + false, + ), + false, + ), false, - ))), - false, - ))), - false, - 1, - )]); + 1, + ), + ]); let diff = SchemaDiff::new(&before, &after).unwrap(); @@ -2483,18 +2530,10 @@ mod tests { assert_eq!(diff.updated_fields.len(), 2); let paths = updated_paths(&diff); - assert!(paths.contains(&ColumnName::new([ - "complex_map", - "key", - "element", - "renamed_key_field" - ]))); - assert!(paths.contains(&ColumnName::new([ - "complex_map", - "value", - "element", - "renamed_value_field" - ]))); + assert!(paths.contains(&column_name!("complex_map.key.element.renamed_key_field"))); + assert!(paths.contains(&column_name!( + "complex_map.value.element.renamed_value_field" + ))); assert!(!diff.has_breaking_changes()); } @@ -2503,7 +2542,7 @@ mod tests { // map, map, struct>> let before = StructType::new_unchecked([create_field_with_id( "nested_maps", - DataType::Map(Box::new(MapType::new( + MapType::new( DataType::try_struct_type([create_field_with_id( "outer_key", DataType::INTEGER, @@ -2511,7 +2550,7 @@ mod tests { 2, )]) .unwrap(), - DataType::Map(Box::new(MapType::new( + MapType::new( DataType::try_struct_type([create_field_with_id( "inner_key", DataType::INTEGER, @@ -2525,16 +2564,16 @@ mod tests { ]) .unwrap(), false, - ))), + ), false, - ))), + ), false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "nested_maps", - DataType::Map(Box::new(MapType::new( + MapType::new( DataType::try_struct_type([create_field_with_id( "renamed_outer_key", // Renamed! DataType::INTEGER, @@ -2542,7 +2581,7 @@ mod tests { 2, )]) .unwrap(), - DataType::Map(Box::new(MapType::new( + MapType::new( DataType::try_struct_type([create_field_with_id( "renamed_inner_key", // Renamed! DataType::INTEGER, @@ -2556,9 +2595,9 @@ mod tests { ]) .unwrap(), false, - ))), + ), false, - ))), + ), false, 1, )]); @@ -2571,81 +2610,92 @@ mod tests { assert_eq!( diff.added_fields[0].path, - ColumnName::new(["nested_maps", "value", "value", "added"]) + column_name!("nested_maps.value.value.added") ); assert_eq!( diff.removed_fields[0].path, - ColumnName::new(["nested_maps", "value", "value", "removed"]) + column_name!("nested_maps.value.value.removed") ); let paths = updated_paths(&diff); - assert!(paths.contains(&ColumnName::new([ - "nested_maps", - "key", - "renamed_outer_key" - ]))); - assert!(paths.contains(&ColumnName::new([ - "nested_maps", - "value", - "key", - "renamed_inner_key" - ]))); - assert!(paths.contains(&ColumnName::new([ - "nested_maps", - "value", - "value", - "renamed_data" - ]))); + assert!(paths.contains(&column_name!("nested_maps.key.renamed_outer_key"))); + assert!(paths.contains(&column_name!("nested_maps.value.key.renamed_inner_key"))); + assert!(paths.contains(&column_name!("nested_maps.value.value.renamed_data"))); assert!(!diff.has_breaking_changes()); } #[test] fn test_deeply_nested_nullability_tightening_is_breaking() { - // array>>> -> array>>> - let before = StructType::new_unchecked([create_field_with_id( - "wrapper", - DataType::Array(Box::new(ArrayType::new( - DataType::try_struct_type([create_field_with_id( - "items", - DataType::Array(Box::new(ArrayType::new( - DataType::try_struct_type([ - create_field_with_id("value", DataType::INTEGER, true, 3), // Nullable - ]) + // array>>> -> array>>> + let before = + StructType::new_unchecked([ + create_field_with_id( + "wrapper", + ArrayType::new( + DataType::try_struct_type( + [ + create_field_with_id( + "items", + ArrayType::new( + DataType::try_struct_type([ + create_field_with_id( + "value", + DataType::INTEGER, + true, + 3, + ), // Nullable + ]) + .unwrap(), + false, + ), + false, + 2, + ), + ], + ) .unwrap(), false, - ))), + ), false, - 2, - )]) - .unwrap(), - false, - ))), - false, - 1, - )]); + 1, + ), + ]); - let after = StructType::new_unchecked([create_field_with_id( - "wrapper", - DataType::Array(Box::new(ArrayType::new( - DataType::try_struct_type([create_field_with_id( - "items", - DataType::Array(Box::new(ArrayType::new( - DataType::try_struct_type([ - create_field_with_id("value", DataType::INTEGER, false, 3), // Non-nullable now - BREAKING! - ]) + let after = + StructType::new_unchecked([ + create_field_with_id( + "wrapper", + ArrayType::new( + DataType::try_struct_type( + [ + create_field_with_id( + "items", + ArrayType::new( + DataType::try_struct_type([ + create_field_with_id( + "value", + DataType::INTEGER, + false, + 3, + ), // Non-nullable now - BREAKING! + ]) + .unwrap(), + false, + ), + false, + 2, + ), + ], + ) .unwrap(), false, - ))), + ), false, - 2, - )]) - .unwrap(), - false, - ))), - false, - 1, - )]); + 1, + ), + ]); let diff = SchemaDiff::new(&before, &after).unwrap(); @@ -2654,7 +2704,7 @@ mod tests { assert_eq!(diff.updated_fields.len(), 1); assert_eq!( diff.updated_fields[0].path, - ColumnName::new(["wrapper", "element", "items", "element", "value"]) + column_name!("wrapper.element.items.element.value") ); assert_eq!( diff.updated_fields[0].change_types, @@ -2665,13 +2715,14 @@ mod tests { #[test] fn test_deeply_nested_container_nullability_tightening_is_breaking() { - // array nullable>>> -> array not null>>> + // array nullable>>> -> array not null>>> let before = StructType::new_unchecked([create_field_with_id( "wrapper", - DataType::Array(Box::new(ArrayType::new( + ArrayType::new( DataType::try_struct_type([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new( + ArrayType::new( DataType::try_struct_type([create_field_with_id( "value", DataType::INTEGER, @@ -2680,23 +2731,23 @@ mod tests { )]) .unwrap(), true, // Array elements are nullable - ))), + ), false, 2, )]) .unwrap(), false, - ))), + ), false, 1, )]); let after = StructType::new_unchecked([create_field_with_id( "wrapper", - DataType::Array(Box::new(ArrayType::new( + ArrayType::new( DataType::try_struct_type([create_field_with_id( "items", - DataType::Array(Box::new(ArrayType::new( + ArrayType::new( DataType::try_struct_type([create_field_with_id( "value", DataType::INTEGER, @@ -2705,13 +2756,13 @@ mod tests { )]) .unwrap(), false, // Array elements now non-nullable - BREAKING! - ))), + ), false, 2, )]) .unwrap(), false, - ))), + ), false, 1, )]); @@ -2723,7 +2774,7 @@ mod tests { assert_eq!(diff.updated_fields.len(), 1); assert_eq!( diff.updated_fields[0].path, - ColumnName::new(["wrapper", "element", "items"]) + column_name!("wrapper.element.items") ); assert_eq!( diff.updated_fields[0].change_types, diff --git a/kernel/src/schema/mod.rs b/kernel/src/schema/mod.rs index be24779e66..b15eb4aa43 100644 --- a/kernel/src/schema/mod.rs +++ b/kernel/src/schema/mod.rs @@ -4,24 +4,32 @@ use std::borrow::Cow; use std::collections::{HashMap, HashSet}; use std::fmt::{Debug, Display, Formatter}; use std::iter::{DoubleEndedIterator, FusedIterator}; +use std::ops::Deref; use std::str::FromStr; use std::sync::{Arc, LazyLock}; +use delta_kernel_derive::internal_api; use indexmap::IndexMap; use itertools::Itertools; use serde::{Deserialize, Serialize}; +#[cfg(feature = "geo-type-in-dev")] +use strum::{Display as StrumDisplay, EnumString}; use tracing::warn; // re-export because many call sites that use schemas do not necessarily use expressions pub(crate) use crate::expressions::{column_name, ColumnName}; use crate::reserved_field_ids::FILE_NAME; -use crate::table_features::get_field_column_mapping_info; -use crate::table_features::ColumnMappingMode; -use crate::transforms::SchemaTransform; -use crate::utils::require; +use crate::table_features::{ + validate_and_extract_column_mapping_annotations, validate_column_mapping_id, ColumnMappingMode, + StaleAnnotationPolicy, +}; +use crate::transforms::{transform_output_type, SchemaTransform}; +use crate::utils::{require, CollectInto}; use crate::{DeltaResult, Error}; -use delta_kernel_derive::internal_api; +pub(crate) mod column_default; +pub use column_default::ColumnDefault; +pub(crate) use column_default::{try_collect_column_defaults, validate_column_defaults_metadata}; pub(crate) mod compare; #[cfg(feature = "schema-diff")] pub(crate) mod diff; @@ -30,11 +38,156 @@ pub(crate) mod diff; pub mod derive_macro_utils; #[cfg(not(feature = "internal-api"))] pub(crate) mod derive_macro_utils; +pub(crate) mod validation; pub(crate) mod variant_utils; +pub(crate) mod void_utils; + +/// Prefix of the error message the schema deserializers emit for an unsupported type. +const UNSUPPORTED_DELTA_TYPE_ERROR_PREFIX: &str = "Unsupported Delta table type"; + +/// Builds the serde custom error for a Delta type the kernel does not support. Pairs with +/// [`is_unsupported_delta_type_error`] for detecting whether a serde error is this kind. +fn unsupported_delta_type_error(name: &str) -> E { + E::custom(format!("{UNSUPPORTED_DELTA_TYPE_ERROR_PREFIX}: '{name}'")) +} + +/// Returns `true` if `error` is the "unsupported Delta type" failure produced by +/// `unsupported_delta_type_error`. +pub(crate) fn is_unsupported_delta_type_error(error: &serde_json::Error) -> bool { + error.is_data() + && error + .to_string() + .starts_with(UNSUPPORTED_DELTA_TYPE_ERROR_PREFIX) +} pub type Schema = StructType; pub type SchemaRef = Arc; +/// Sugar for `LazyLock::new(|| `[`schema_ref!`](schema_ref)` { ... })`, yielding a lazy +/// [`SchemaRef`]. +#[internal_api] +#[doc(inline)] +pub(crate) use delta_kernel_derive::lazy_schema_ref; +/// Builds a [`StructType`] from a JSON-shaped description that freely mixes literal structure +/// with interpolated runtime values, in the spirit of [`serde_json::json!`]. +/// +/// # Grammar +/// +/// ```text +/// body := (entry ',')* entry? // 0+ comma-separated entries, optional trailing comma +/// entry := nullability name ':' type // possibly nullable struct field +/// | '(' EXPR ')' // interpolate one StructField +/// | '..' '(' EXPR ')' // splice an `impl IntoIterator` +/// nullability := 'nullable' | 'not_null' +/// name := STR_LITERAL +/// | IDENT +/// | '(' EXPR ')' // interpolate an `impl Into` +/// type := '[' nullability type ']' // array with possibly-nullable elements +/// | '{' body '}' // nested struct +/// | '{' type '=>' nullability type '}' // map with possibly-nullable values +/// | '(' EXPR ')' // interpolate an `impl Into` +/// | IDENT // interpolate `DataType::` +/// ``` +/// +/// # Examples +/// +/// ``` +/// # use delta_kernel::schema::{schema, DataType, StructField, StructType}; +/// let s = schema! { +/// not_null "id": LONG, +/// nullable "name": STRING, +/// not_null "address": { +/// nullable "city": STRING, +/// nullable "zip": STRING, +/// }, +/// nullable "tags": [ not_null STRING ], // nullable array with non-null elements +/// not_null "props": { STRING => nullable STRING }, // non-nullable map with nullable values +/// }; +/// assert_eq!(s.field("id").unwrap().data_type(), &DataType::LONG); +/// ``` +/// +/// Runtime values interpolate through the expression forms: +/// +/// ``` +/// # use delta_kernel::schema::{schema, DataType, StructField}; +/// let data_type = DataType::LONG; +/// let first = StructField::not_null("y", DataType::INTEGER); +/// let rest = vec![StructField::nullable("z", DataType::STRING)]; +/// let i = 42; +/// let s = schema! { +/// not_null (format!("col_{i}")): (data_type), +/// (first), +/// ..(rest), +/// }; +/// assert_eq!(s.fields().count(), 3); +/// ``` +/// +/// Field structure is author-controlled, so this builds via [`StructType::new_unchecked`] (no +/// runtime validation). Statically-detectable duplicate field names -- repeated string +/// literals or repeated identifiers within the same struct -- are rejected at compile time. +/// Literals are compared case-insensitively, matching Delta's case-insensitive column-name +/// rule: +/// +/// ```compile_fail +/// # use delta_kernel::schema::schema; +/// const NAME: &str = "foo"; +/// let s = schema! { +/// not_null "id": LONG, +/// nullable "ID": STRING, // duplicate of "id" (case-insensitive) -- compile error +/// nullable NAME: LONG, +/// not_null NAME: STRING, // NAME used twice -- compile error +/// }; +/// ``` +/// +/// Prefer [`try_schema`] when field names are interpolated runtime values that might collide. +#[internal_api] +#[doc(inline)] +pub(crate) use delta_kernel_derive::schema; +/// Sugar for `Arc::new(`[`schema!`](schema)` { ... })`, yielding a [`SchemaRef`]. Convenient +/// for the `LazyLock` statics that pervade the action and stats schemas. +#[internal_api] +#[doc(inline)] +pub(crate) use delta_kernel_derive::schema_ref; +/// Like [`schema`], but validates field names at every level of the schema (each struct, +/// including nested ones, is built via [`StructType::try_new`] and yields +/// [`DeltaResult`]. Use when field names are runtime values that could duplicate +/// in ways the macro cannot see. +#[internal_api] +#[doc(inline)] +pub(crate) use delta_kernel_derive::try_schema; + +/// Converts field interpolation inputs in [`schema!`] and [`try_schema!`] to [`StructField`]. +#[internal_api] +pub(crate) trait ToSchemaField { + fn to_schema_field(self) -> StructField; +} + +impl ToSchemaField for StructField { + fn to_schema_field(self) -> StructField { + self + } +} + +impl ToSchemaField for &StructField { + fn to_schema_field(self) -> StructField { + self.clone() + } +} + +impl ToSchemaField for &T +where + T: Deref, +{ + fn to_schema_field(self) -> StructField { + self.deref().clone() + } +} + +/// A [`StructPatchBuilder`](crate::struct_patch::StructPatchBuilder) whose emitted items are schema +/// fields, lowered into an output [`StructType`] directly from an input schema via +/// [`build`](crate::struct_patch::StructPatchBuilder::::build). +pub type SchemaStructPatchBuilder = crate::struct_patch::StructPatchBuilder; + /// Converts a type to a [`Schema`] that represents that type. Derivable for struct types using the /// [`delta_kernel_derive::ToSchema`] derive macro. #[internal_api] @@ -100,8 +253,29 @@ impl From for MetadataValue { pub enum ColumnMetadataKey { ColumnMappingId, ColumnMappingPhysicalName, + /// Parquet field IDs for the synthesized `element` / `key` / `value` fields of an Array or + /// Map. Stored on the *nearest ancestor* StructField as a JSON object whose keys are + /// dot-paths rooted at that field's name. + /// + /// # Example: list-in-map + /// + /// For `m: map>` and the key/value/element fields having field ids + /// 100/101/102, the metadata on `m` should be: + /// + /// ```json + /// { + /// "delta.columnMapping.nested.ids": { + /// "m.key": 100, + /// "m.value": 101, + /// "m.value.element": 102 + /// } + /// } + /// ``` + ColumnMappingNestedIds, ParquetFieldId, + ParquetFieldNestedIds, GenerationExpression, + CurrentDefault, IdentityStart, IdentityStep, IdentityHighWaterMark, @@ -116,11 +290,18 @@ impl AsRef for ColumnMetadataKey { match self { Self::ColumnMappingId => "delta.columnMapping.id", Self::ColumnMappingPhysicalName => "delta.columnMapping.physicalName", + Self::ColumnMappingNestedIds => "delta.columnMapping.nested.ids", // "parquet.field.id" is not defined by the Delta protocol, but follows the convention // established by delta-spark and other Delta ecosystem implementations for storing // Parquet field IDs in StructField metadata. Self::ParquetFieldId => "parquet.field.id", + // The Delta protocol defines this key for IcebergCompatV2/V3 nested field ids. It is + // legacy and will be replaced by `delta.columnMapping.nested.ids` (which kernel + // uses everywhere). Kept here for protocol compatibility only. + // Tracking issue: + Self::ParquetFieldNestedIds => "parquet.field.nested.ids", Self::GenerationExpression => "delta.generationExpression", + Self::CurrentDefault => "CURRENT_DEFAULT", Self::IdentityAllowExplicitInsert => "delta.identity.allowExplicitInsert", Self::IdentityHighWaterMark => "delta.identity.highWaterMark", Self::IdentityStart => "delta.identity.start", @@ -210,10 +391,23 @@ pub struct StructField { pub metadata: HashMap, } +/// Parsed (and validated) pre-existing column-mapping annotations on a single field, as +/// returned by [`StructField::validate_and_extract_existing_column_mapping_annotations`]. +/// Either, both, or neither of the two fields may be present; a field with neither set has no +/// pre-populated column-mapping metadata. +#[derive(Debug, Clone, Copy)] +pub(crate) struct ExistingColumnMappingAnnotations<'a> { + /// Parsed `delta.columnMapping.id`, if present. Guaranteed non-negative. + pub id: Option, + /// Borrowed `delta.columnMapping.physicalName`, if present. Guaranteed non-empty. + pub physical_name: Option<&'a str>, +} + impl StructField { /// The name of the default row index metadata column. /// - /// Note that the dot does not indicate a nested field, it is just a separator for the metadata column name. + /// Note that the dot does not indicate a nested field, it is just a separator for the metadata + /// column name. const DEFAULT_ROW_INDEX_COLUMN_NAME: &'static str = "_metadata.row_index"; ///////////////// @@ -333,6 +527,119 @@ impl StructField { self.metadata.get(key.as_ref()) } + /// Returns this field's `delta.columnMapping.id` annotation if present and well-formed. + /// Returns `None` if the annotation is missing or carries a non-numeric value. + pub fn column_mapping_id(&self) -> Option { + match self.get_config_value(&ColumnMetadataKey::ColumnMappingId)? { + MetadataValue::Number(n) => Some(*n), + _ => None, + } + } + + /// Returns this field's column default, parsed from its `CURRENT_DEFAULT` + /// ([`ColumnMetadataKey::CurrentDefault`]) metadata, if present. + /// + /// - `Ok(None)` -- no `CURRENT_DEFAULT` metadata. + /// - `Ok(Some(_))` -- present as a [`MetadataValue::String`] and accepted by [`ColumnDefault`]. + /// - `Err(_)` -- either not a [`MetadataValue::String`] (corrupt: the protocol defines + /// `CURRENT_DEFAULT` as a SQL string, the only form the kernel writes), or rejected by + /// [`ColumnDefault`] (a non-NULL default on a Variant column, which the protocol forbids). + pub fn column_default(&self) -> DeltaResult>> { + let raw_sql = match self.get_config_value(&ColumnMetadataKey::CurrentDefault) { + None => return Ok(None), + Some(MetadataValue::String(s)) => s.clone(), + Some(other) => { + return Err(Error::schema(format!( + "Field '{}' has a non-string `{}` annotation: {other}", + self.name, + ColumnMetadataKey::CurrentDefault.as_ref(), + ))) + } + }; + ColumnDefault::new(raw_sql, &self.data_type).map(Some) + } + + /// Validates and extracts pre-existing column-mapping annotations on this field, returning + /// the parsed `id` and `physical_name` borrowed from the field's metadata. Returning the + /// parsed values lets the column-mapping assignment dispatch match on + /// `(Option, Option<&str>)` directly, which makes the dispatch total and obviates a + /// catch-all panic for malformed annotations the validator already rejects. + /// + /// Rejects: + /// - `delta.columnMapping.id` is present but not a `MetadataValue::Number`, + /// - `delta.columnMapping.id` is a `Number` but lies outside the protocol's 32-bit non-negative + /// range (negative or `> i32::MAX`); see + /// [`crate::table_features::validate_column_mapping_id`], + /// - `delta.columnMapping.physicalName` is present but not a `MetadataValue::String`, + /// - `delta.columnMapping.physicalName` is an empty `String`. + /// + /// Empty-name, negative-id, and over-`i32::MAX` id are stricter than delta-spark (which + /// accepts the first two and historically truncates the third); kernel fails fast at + /// write time so a connector that supplies bad metadata learns about it on the call that + /// produced it. Wrong-typed `id` errors take precedence over wrong-typed `physicalName` + /// errors (a connector that fixes the `id` and retries will then see the `physicalName` + /// error). + pub(crate) fn validate_and_extract_existing_column_mapping_annotations( + &self, + ) -> DeltaResult> { + let id = match self.get_config_value(&ColumnMetadataKey::ColumnMappingId) { + Some(MetadataValue::Number(n)) => { + validate_column_mapping_id(*n) + .map_err(|e| Error::schema(format!("Field '{}': {e}", self.name)))?; + Some(*n) + } + None => None, + Some(_) => { + return Err(Error::schema(format!( + "Field '{}' has a non-numeric `{}` annotation", + self.name, + ColumnMetadataKey::ColumnMappingId.as_ref(), + ))); + } + }; + let physical_name = + match self.get_config_value(&ColumnMetadataKey::ColumnMappingPhysicalName) { + Some(MetadataValue::String(s)) if s.is_empty() => { + return Err(Error::schema(format!( + "Field '{}' has an empty `{}` annotation", + self.name, + ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), + ))); + } + Some(MetadataValue::String(s)) => Some(s.as_str()), + None => None, + Some(_) => { + return Err(Error::schema(format!( + "Field '{}' has a non-string `{}` annotation", + self.name, + ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), + ))); + } + }; + Ok(ExistingColumnMappingAnnotations { id, physical_name }) + } + + /// Recursively collects every `delta.columnMapping.id` reachable from this field -- + /// the field's own ID plus any nested struct fields under Struct/Array/Map/Variant. + /// Test-only. + #[cfg(any(test, feature = "test-utils"))] + pub fn collect_column_mapping_ids(&self) -> Vec { + struct CollectIds(Vec); + impl<'a> SchemaTransform<'a> for CollectIds { + transform_output_type!(|'a, T| ()); + + fn transform_struct_field(&mut self, field: &'a StructField) { + if let Some(id) = field.column_mapping_id() { + self.0.push(id); + } + self.recurse_into_struct_field(field) + } + } + let mut visitor = CollectIds(Vec::new()); + visitor.transform_struct_field(self); + visitor.0 + } + /// Get the physical name for this field as it should be read from parquet. /// /// When `column_mapping_mode` is `None`, always returns the logical name (even if physical @@ -340,8 +647,8 @@ impl StructField { /// metadata if present, otherwise returns the logical name. /// /// NOTE: Caller affirms that the schema was already validated by - /// [`crate::table_configuration::TableConfiguration::try_new`], to ensure that annotations are - /// always and only present when column mapping mode is enabled. + /// [`crate::table_configuration::TableConfiguration::try_new`]. In `None` mode a stale + /// annotation may still be present (it is ignored, and the logical name is returned). #[internal_api] pub(crate) fn physical_name(&self, column_mapping_mode: ColumnMappingMode) -> &str { match column_mapping_mode { @@ -409,15 +716,6 @@ impl StructField { &self.metadata } - /// Convert our metadata into a HashMap. Note this copies all the data so can be - /// expensive for large metadata - pub fn metadata_with_string_values(&self) -> HashMap { - self.metadata - .iter() - .map(|(key, val)| (key.clone(), val.to_string())) - .collect() - } - /// Applies physical name and field ID mappings to this field. /// /// This function sets the field ID for the physical [`StructField`] only if the @@ -433,8 +731,11 @@ impl StructField { /// Otherwise, the field's logical name is used. /// /// Returns an error if a field has invalid or inconsistent column mapping annotations (e.g. - /// missing when column mapping is enabled, present when disabled, or wrong type), or if a - /// metadata column is encountered (metadata columns should not participate in column mapping). + /// missing or wrong-typed when column mapping is enabled), or if a metadata column is + /// encountered (metadata columns should not participate in column mapping). When column + /// mapping is disabled, a stale annotation is tolerated (resolved by logical name and dropped + /// from the physical metadata); CREATE / ALTER reject it via a separate strict validation pass + /// instead. /// /// [`read_parquet_files`]: crate::ParquetHandler::read_parquet_files #[internal_api] @@ -442,10 +743,12 @@ impl StructField { &self, column_mapping_mode: ColumnMappingMode, ) -> DeltaResult { - MakePhysical::new(column_mapping_mode).run_field(self) + MakePhysical::new(column_mapping_mode) + .transform_struct_field(self) + .map(|f| f.into_owned()) } - fn has_invariants(&self) -> bool { + pub(crate) fn has_invariants(&self) -> bool { self.metadata .contains_key(ColumnMetadataKey::Invariants.as_ref()) } @@ -456,8 +759,8 @@ impl StructField { /// NOTE: Must not be called on metadata columns, which are not subject to column mapping. /// /// NOTE: Caller affirms that `self` was already validated by - /// [`crate::table_features::get_field_column_mapping_info`], to ensure that annotations are - /// always and only present when column mapping mode is enabled. + /// [`crate::table_features::validate_and_extract_column_mapping_annotations`]. In `None` mode a + /// stale annotation may be present; this drops the column-mapping keys regardless. fn logical_to_physical_metadata( &self, column_mapping_mode: ColumnMappingMode, @@ -470,7 +773,8 @@ impl StructField { match column_mapping_mode { ColumnMappingMode::Id => { let Some(MetadataValue::Number(fid)) = field_id else { - // `get_field_column_mapping_info` should have verified that this has a field Id + // `validate_and_extract_column_mapping_annotations` should have verified that + // this has a field Id warn!("StructField with name {} is missing field id in the Id column mapping mode", self.name()); debug_assert!(false); return base_metadata; @@ -544,8 +848,8 @@ pub struct StructType { // for each field by name would be potentially quite expensive for large schemas. fields: IndexMap, /// The metadata columns in this struct - // We use a dedicated map for metadata columns to allow for fast lookup without having to iterate - // over all fields. + // We use a dedicated map for metadata columns to allow for fast lookup without having to + // iterate over all fields. metadata_columns: HashMap, } @@ -615,7 +919,8 @@ impl StructType { } } - // Delta column names are case-insensitive; reject schemas with duplicates that differ only by case. + // Delta column names are case-insensitive; reject schemas with duplicates that differ + // only by case. let key = field.name.to_lowercase(); if !seen_lowercase_names.insert(key) { return Err(Error::schema(format!( @@ -734,6 +1039,35 @@ impl StructType { self.fields.get(name.as_ref()) } + /// Retrieves the nested field named by the given column path. + /// + /// Returns an error if the path is empty, a field is not found, or an intermediate field is not + /// a struct type. + pub fn field_at<'a>(&'a self, col: &ColumnName) -> DeltaResult<&'a StructField> { + let mut field = None; + self.visit_fields_of_path(col, |f| field = Some(f))?; + field.ok_or_else(|| Error::generic("Empty path")) + } + + /// Checks whether this schema contains the field at the given column path. + pub fn contains_col(&self, col: impl CollectInto) -> bool { + let col = col.collect_into(); + self.field_at(&col).is_ok() + } + + /// Visits all fields along the given column path. + /// + /// Returns an error if the path is empty, a field is not found, or an intermediate field is not + /// a struct type. + #[internal_api] + pub(crate) fn visit_fields_of_path<'a>( + &'a self, + col: &ColumnName, + visit_field: impl FnMut(&'a StructField), + ) -> DeltaResult<()> { + self.visit_fields_of_path_by(col, |s, name| s.field(name), visit_field) + } + /// Resolves a column path through nested structs, returning references to all /// [`StructField`]s along the path. The last element is the leaf field. /// @@ -742,23 +1076,41 @@ impl StructType { /// /// Returns an error if the path is empty, a field is not found, or an intermediate /// field is not a struct type. - pub(crate) fn walk_column_fields<'a>( + #[internal_api] + pub(crate) fn fields_of_path<'a>( &'a self, col: &ColumnName, ) -> DeltaResult> { + let mut result = Vec::with_capacity(col.path().len()); + self.visit_fields_of_path(col, |f| result.push(f))?; + Ok(result) + } + + /// Visits all fields along the given column path, using a caller-provided field name resolver. + /// + /// Returns an error if the path is empty, a field is not found, or an intermediate field is not + /// a struct type. + pub(crate) fn visit_fields_of_path_by<'a, F>( + &'a self, + col: &ColumnName, + find_field: F, + mut visit_field: impl FnMut(&'a StructField), + ) -> DeltaResult<()> + where + F: for<'b> Fn(&'b StructType, &str) -> Option<&'b StructField>, + { let path = col.path(); if path.is_empty() { return Err(Error::generic("Column path cannot be empty")); } let mut current_struct = self; - let mut fields = Vec::with_capacity(path.len()); for (i, field_name) in path.iter().enumerate() { - let field = current_struct.field(field_name).ok_or_else(|| { + let field = find_field(current_struct, field_name).ok_or_else(|| { Error::generic(format!( "Could not resolve column '{col}': field '{field_name}' not found in schema" )) })?; - fields.push(field); + visit_field(field); if i < path.len() - 1 { let DataType::Struct(inner) = field.data_type() else { return Err(Error::generic(format!( @@ -769,7 +1121,7 @@ impl StructType { current_struct = inner; } } - Ok(fields) + Ok(()) } /// Gets the field with the given name and its index. @@ -798,6 +1150,56 @@ impl StructType { self.fields.into_values() } + /// Gets a mutable reference to the underlying field map. + pub(crate) fn field_map_mut(&mut self) -> &mut IndexMap { + &mut self.fields + } + + /// Walk a pre-segmented column path through this schema and return the leaf field. + /// + /// `path` is the path's individual name segments (one per nesting level), already split by + /// the caller. Lookup at each level is case-insensitive. The Delta protocol uses `.` as the + /// dotted-path separator at the API surface, but `field_at_path` itself does not split -- + /// callers typically pass [`ColumnName::path()`](crate::expressions::ColumnName::path), + /// which yields the segments directly. + /// + /// Panics if any segment is missing or an intermediate field is not a struct. Intended for + /// use in test assertions. + /// + /// # Example + /// + /// ```ignore + /// // Schema: + /// // id: INTEGER not null + /// // address: STRUCT { city: STRING not null, zip: STRING } + /// let path = vec!["address".to_string(), "city".to_string()]; + /// let city = schema.field_at_path(&path); + /// assert_eq!(city.name(), "city"); + /// assert!(!city.is_nullable()); + /// ``` + #[cfg(any(test, feature = "test-utils"))] + #[allow(clippy::panic, clippy::expect_used)] + pub fn field_at_path<'a>(&'a self, path: &[String]) -> &'a StructField { + fn find_ci<'a>( + mut fields: impl Iterator, + name: &str, + ) -> &'a StructField { + let lowered = name.to_lowercase(); + fields + .find(|f| f.name().to_lowercase() == lowered) + .unwrap_or_else(|| panic!("field '{name}' not found")) + } + let (first, rest) = path.split_first().expect("non-empty path"); + let mut field = find_ci(self.fields(), first); + for seg in rest { + let DataType::Struct(s) = field.data_type() else { + panic!("expected struct at intermediate segment '{seg}'"); + }; + field = find_ci(s.fields(), seg); + } + field + } + /// Gets all the field names in this struct type in the order they are defined. pub fn field_names(&self) -> impl ExactSizeIterator { self.fields.keys() @@ -857,7 +1259,7 @@ impl StructType { #[internal_api] pub(crate) fn leaves<'s>(&self, own_name: impl Into>) -> ColumnNamesAndTypes { let mut get_leaves = GetSchemaLeaves::new(own_name.into()); - let _ = get_leaves.transform_struct(self); + get_leaves.transform_struct(self); (get_leaves.names, get_leaves.types).into() } @@ -873,11 +1275,7 @@ impl StructType { column_mapping_mode: ColumnMappingMode, ) -> DeltaResult { let mut transformer = MakePhysical::new(column_mapping_mode); - let fields: Vec = self - .fields() - .map(|field| transformer.run_field(field)) - .try_collect()?; - Self::try_new(fields) + transformer.transform_struct(self).map(|s| s.into_owned()) } /// Validates that there are no metadata columns in the given fields. @@ -918,86 +1316,14 @@ impl StructType { Self::ensure_no_metadata_columns(&mut struct_type.fields())?; } } - // Primitive types cannot contain nested metadata columns and variant types are validated at creation + // Primitive types cannot contain nested metadata columns and variant types are + // validated at creation DataType::Primitive(_) | DataType::Variant(_) => {} }; Ok(()) } - /// Returns a StructType with `new_field` inserted after the field named `after`. - /// If `new_field` already presents in the schema, an error is returned. - /// If `after` is None, `new_field` is appended to the end. - /// If `after` is not found, an error is returned. - pub fn with_field_inserted_after( - mut self, - after: Option<&str>, - new_field: StructField, - ) -> DeltaResult { - // TODO: Upgrade to a case-insensitive duplicate check when this method is used for - // user-facing operations like ALTER TABLE ADD COLUMN. Currently only used internally - // for inserting protocol-defined fields (e.g. stats_parsed) where exact-name matching - // is sufficient. - if self.fields.contains_key(&new_field.name) { - return Err(Error::generic(format!( - "Field {} already exists", - new_field.name - ))); - } - - let insert_index = after - .map(|after| { - self.fields - .get_index_of(after) - .map(|index| index + 1) - .ok_or_else(|| Error::generic(format!("Field {after} not found"))) - }) - .unwrap_or_else(|| Ok(self.fields.len()))?; - - self.fields - .insert_before(insert_index, new_field.name.clone(), new_field); - Ok(self) - } - - /// Returns a StructType with `new_field` inserted before the field named `before`. - /// If `new_field` already presents in the schema, an error is returned. - /// If `before` is None, `new_field` is inserted at the beginning. - /// If `before` is not found, an error is returned. - pub fn with_field_inserted_before( - mut self, - before: Option<&str>, - new_field: StructField, - ) -> DeltaResult { - // TODO: Upgrade to a case-insensitive duplicate check when this method is used for - // user-facing operations like ALTER TABLE ADD COLUMN. Currently only used internally - // for inserting protocol-defined fields where exact-name matching is sufficient. - if self.fields.contains_key(&new_field.name) { - return Err(Error::generic(format!( - "Field {} already exists", - new_field.name - ))); - } - - let index_of_before = before - .map(|before| { - self.fields - .get_index_of(before) - .ok_or_else(|| Error::generic(format!("Field {before} not found"))) - }) - .unwrap_or_else(|| Ok(0))?; - - self.fields - .insert_before(index_of_before, new_field.name.clone(), new_field); - Ok(self) - } - - /// Returns a StructType with the named field removed. - /// Returns self unchanged if field doesn't exist. - pub fn with_field_removed(mut self, name: &str) -> Self { - self.fields.shift_remove(name); - self - } - /// Returns a new [`StructType`] containing only the top-level fields for which `predicate` /// returns `true`. This does not recurse into nested [`StructType`] fields. pub fn with_fields_filtered( @@ -1007,20 +1333,21 @@ impl StructType { Self::try_new(self.fields().filter(|f| predicate(f)).cloned()) } - /// Returns a StructType with the named field replaced. - /// Returns an error if field doesn't exist. - pub fn with_field_replaced( - mut self, - name: &str, - new_field: StructField, - ) -> DeltaResult { - let replace_field = self - .fields - .get_mut(name) - .ok_or_else(|| Error::generic(format!("Field {name} not found")))?; - - *replace_field = new_field; - Ok(self) + /// Returns an optional [`StructType`] containing only the top-level fields for which + /// `predicate` returns `true`. + /// + /// This is a convenience wrapper around [`StructType::with_fields_filtered`] for callers + /// that treat an empty top-level struct as "no schema". + pub fn with_fields_filtered_nonempty( + &self, + predicate: impl Fn(&StructField) -> bool, + ) -> DeltaResult> { + let filtered = self.with_fields_filtered(predicate)?; + if filtered.num_fields() == 0 { + Ok(None) + } else { + Ok(Some(filtered)) + } } } @@ -1231,16 +1558,17 @@ impl DoubleEndedIterator for StructFieldRefIter<'_> { } } -struct InvariantChecker(bool); +struct InvariantChecker; impl<'a> SchemaTransform<'a> for InvariantChecker { - fn transform_struct_field(&mut self, field: &'a StructField) -> Option> { + transform_output_type!(|'a, T| Result<(), ()>); + + fn transform_struct_field(&mut self, field: &'a StructField) -> Result<(), ()> { if field.has_invariants() { - self.0 = true; - } else if !self.0 { - let _ = self.recurse_into_struct_field(field); + Err(()) + } else { + self.recurse_into_struct_field(field) } - Some(Cow::Borrowed(field)) } } @@ -1249,9 +1577,83 @@ impl<'a> SchemaTransform<'a> for InvariantChecker { /// This traverses the entire schema to check for the presence of the `delta.invariants` /// metadata key. pub(crate) fn schema_has_invariants(schema: &Schema) -> bool { - let mut checker = InvariantChecker(false); - let _ = checker.transform_struct(schema); - checker.0 + InvariantChecker.transform_struct(schema).is_err() +} + +/// Visitor that reports whether any non-null (`nullable: false`) field exists in a schema. +/// Walks the full schema tree including nested struct, array element, and map value structs. +struct NonNullFieldChecker; + +impl<'a> SchemaTransform<'a> for NonNullFieldChecker { + transform_output_type!(|'a, T| Result<(), ()>); + + /// Skip recursion into variant internals. The `metadata` and `value` fields inside a + /// `Variant` are protocol-defined, always non-null, and not user-controlled, so they + /// must not be treated as user-declared non-null columns. + fn transform_variant(&mut self, _stype: &'a StructType) -> Result<(), ()> { + Ok(()) + } + + fn transform_struct_field(&mut self, field: &'a StructField) -> Result<(), ()> { + if !field.is_nullable() { + return Err(()); + } + + self.recurse_into_struct_field(field) + } +} + +/// Checks if any user-controlled column in the schema (including nested columns) is declared +/// non-null (`nullable: false`). +/// +/// Skips `Variant` internal struct fields, which are protocol-defined and always non-null. +pub(crate) fn schema_contains_non_null_fields(schema: &Schema) -> bool { + NonNullFieldChecker.transform_struct(schema).is_err() +} + +/// Normalizes column name field names to match the casing in the schema. +/// +/// Walks each field name through the schema's struct hierarchy, replacing user-provided +/// casing with the schema's canonical casing. If a field name isn't found +/// case-insensitively, keeps the original (subsequent validation catches it). +/// +/// For example, given schema `{ Id: int, Name: string }` and user-provided columns +/// `["id", "name"]`, returns `["Id", "Name"]` -- matching the schema's canonical casing. +/// +/// Note: Must be called before validation (`validate_partition_columns` or +/// `validate_clustering_columns`) so that case-normalized names match the schema. +pub(crate) fn normalize_column_names_to_schema_casing( + schema: &StructType, + columns: &[ColumnName], +) -> Vec { + columns + .iter() + .map(|col| { + let path = col.path(); + let mut normalized: Vec = Vec::with_capacity(path.len()); + let mut current_schema = schema; + for (i, field_name) in path.iter().enumerate() { + match current_schema + .fields() + .find(|f| f.name().eq_ignore_ascii_case(field_name)) + { + Some(f) => { + normalized.push(f.name().to_string()); + if let DataType::Struct(inner) = f.data_type() { + current_schema = inner; + } + } + None => { + // Field name not found at this level -- keep remaining path + // unchanged so validation reports the user's original input. + normalized.extend(path[i..].iter().cloned()); + break; + } + } + } + ColumnName::new(normalized.iter().map(|s| s.as_str())) + }) + .collect() } /// Helper for RowVisitor implementations @@ -1315,10 +1717,10 @@ pub struct ArrayType { } impl ArrayType { - pub fn new(element_type: DataType, contains_null: bool) -> Self { + pub fn new(element_type: impl Into, contains_null: bool) -> Self { Self { type_name: "array".into(), - element_type, + element_type: element_type.into(), contains_null, } } @@ -1377,7 +1779,8 @@ impl MapType { self.value_contains_null } - /// Create a schema assuming the map is stored as a struct with the specified key and value field names + /// Create a schema assuming the map is stored as a struct with the specified key and value + /// field names pub fn as_struct_schema(&self, key_name: String, val_name: String) -> Schema { StructType::new_unchecked([ StructField::not_null(key_name, self.key_type.clone()), @@ -1390,6 +1793,138 @@ fn default_true() -> bool { true } +/// Validates that a CRS (Coordinate Reference System) identifier is in `AUTHORITY:CODE` form, +/// e.g. `"EPSG:4326"` or `"OGC:CRS84"`: a non-empty authority and code separated by a single +/// colon, no comma, and no surrounding whitespace. Validating the value against the full set of +/// recognized CRSes is future work. +#[cfg(feature = "geo-type-in-dev")] +fn validate_crs(crs: &str) -> DeltaResult<()> { + require!( + crs == crs.trim(), + Error::invalid_geo_params(format!( + "CRS '{crs}' must not have leading or trailing whitespace" + )) + ); + require!( + !crs.contains(','), + Error::invalid_geo_params(format!("CRS '{crs}' must not contain a comma")) + ); + + let [authority, code] = crs.split(':').collect::>()[..] else { + return Err(Error::invalid_geo_params(format!( + "CRS '{crs}' must be in 'AUTHORITY:CODE' format" + ))); + }; + + require!( + !authority.is_empty(), + Error::invalid_geo_params(format!( + "CRS '{crs}' must have an authority before the colon" + )) + ); + require!( + !code.is_empty(), + Error::invalid_geo_params(format!("CRS '{crs}' must have a code after the colon")) + ); + Ok(()) +} + +/// Algorithm used to interpolate edges between two vertices of a geography path. +#[cfg(feature = "geo-type-in-dev")] +#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize, EnumString, StrumDisplay)] +#[strum(ascii_case_insensitive)] +pub enum EdgeInterpolationAlgorithm { + /// Edges are interpolated as geodesics on a sphere. + #[strum(serialize = "spherical")] + Spherical, + + /// Vincenty's formulae for geodesics on an ellipsoid. + #[strum(serialize = "vincenty")] + Vincenty, + + /// Thomas's approximation for geodesics on an ellipsoid. + #[strum(serialize = "thomas")] + Thomas, + + /// Andoyer's approximation for geodesics on an ellipsoid. + #[strum(serialize = "andoyer")] + Andoyer, + + /// Karney's algorithm for geodesics on an ellipsoid. + #[strum(serialize = "karney")] + Karney, +} + +/// A geometry column type with an associated coordinate reference system (CRS). +/// +/// Serializes as `geometry()`, e.g. `geometry(EPSG:4326)`. +#[cfg(feature = "geo-type-in-dev")] +#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub struct GeometryType { + crs: String, +} + +#[cfg(feature = "geo-type-in-dev")] +impl GeometryType { + /// Constructs a GeometryType from the given CRS, or returns an error if the CRS is + /// not in AUTHORITY:CODE form. + pub fn try_new(crs: &str) -> DeltaResult { + validate_crs(crs)?; + Ok(Self { + crs: crs.to_string(), + }) + } + + pub fn crs(&self) -> &str { + &self.crs + } +} + +#[cfg(feature = "geo-type-in-dev")] +impl Display for GeometryType { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + write!(f, "geometry({})", self.crs) + } +} + +/// Geography column type with an associated CRS and edge interpolation algorithm. +/// +/// Serializes as `geography(, )`, e.g. `geography(EPSG:4326, spherical)`. +#[cfg(feature = "geo-type-in-dev")] +#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub struct GeographyType { + crs: String, + algorithm: EdgeInterpolationAlgorithm, +} + +#[cfg(feature = "geo-type-in-dev")] +impl GeographyType { + /// Constructs a GeographyType from the given CRS and edge interpolation algorithm, or + /// returns an error if the CRS is not in AUTHORITY:CODE form. + pub fn try_new(crs: &str, algorithm: EdgeInterpolationAlgorithm) -> DeltaResult { + validate_crs(crs)?; + Ok(Self { + crs: crs.to_string(), + algorithm, + }) + } + + pub fn crs(&self) -> &str { + &self.crs + } + + pub fn algorithm(&self) -> &EdgeInterpolationAlgorithm { + &self.algorithm + } +} + +#[cfg(feature = "geo-type-in-dev")] +impl Display for GeographyType { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + write!(f, "geography({}, {})", self.crs, self.algorithm) + } +} + #[derive(Debug, Clone, Copy, Eq, PartialEq, Serialize, Deserialize)] pub struct DecimalType { precision: u8, @@ -1448,8 +1983,28 @@ pub enum PrimitiveType { Timestamp, #[serde(rename = "timestamp_ntz")] TimestampNtz, + Void, + /// Year-month interval: a signed count of months (ANSI `INTERVAL YEAR TO MONTH` and its + /// narrowed `YEAR` / `MONTH` spellings). The serde rename is the `schemaString` type-name + /// string -- spelled with spaces, unlike the single-word siblings, so the mapping is not + /// self-evident. + #[serde(rename = "interval year to month")] + IntervalYearMonth, + /// Day-time interval: a signed count of microseconds (ANSI `INTERVAL DAY TO SECOND` and + /// its narrowed `DAY` / `HOUR` / `MINUTE` / `SECOND` spellings). As with the year-month + /// variant above, the serde rename is the multi-word `schemaString` type-name string. + #[serde(rename = "interval day to second")] + IntervalDayTime, #[serde(serialize_with = "serialize_decimal", untagged)] Decimal(DecimalType), + /// Geometry column with an associated coordinate reference system (CRS). + #[cfg(feature = "geo-type-in-dev")] + #[serde(serialize_with = "serialize_geotype", untagged)] + Geometry(Box), + /// Geography column with an associated CRS and edge interpolation algorithm. + #[cfg(feature = "geo-type-in-dev")] + #[serde(serialize_with = "serialize_geotype", untagged)] + Geography(Box), } impl PrimitiveType { @@ -1457,14 +2012,21 @@ impl PrimitiveType { Ok(DecimalType::try_new(precision, scale)?.into()) } + /// Returns whether this is one of the ANSI interval primitive types. + #[internal_api] + pub(crate) fn is_interval(&self) -> bool { + matches!(self, Self::IntervalYearMonth | Self::IntervalDayTime) + } + /// Returns `true` if this primitive type can be widened to the `target` type. /// - /// Widening rules (based on Parquet reader behavior): - /// - Integer widening: byte → short → int → long - /// - Float widening: float → double - /// - /// Note: These widening rules assume the parquet reader supports reading narrower types - /// as wider types. This should be documented as a requirement in the `ParquetHandler` trait. + /// Widening rules: + /// - Integer widening: byte -> short -> int -> long (Delta protocol type widening) + /// - Float widening: float -> double (Delta protocol type widening) + /// - Timestamp interchangeability: Timestamp <-> TimestampNtz (both are i64 microseconds since + /// epoch, differing only in timezone semantics; this is a physical read accommodation, not a + /// Delta protocol type widening rule) + #[internal_api] pub(crate) fn can_widen_to(&self, target: &Self) -> bool { use PrimitiveType::*; matches!( @@ -1482,6 +2044,38 @@ impl PrimitiveType { | (TimestampNtz, Timestamp) ) } + + /// Returns `true` if `self` is a physical integer type that some checkpoint writers + /// produce when they omit Parquet logical type annotations for date or timestamp columns. + /// + /// Specifically: + /// - Integer -> Date (int32 stored without DATE annotation) + /// - Long -> Timestamp/TimestampNtz (int64 stored without TIMESTAMP annotation) + /// + /// These are **not** Delta protocol type widening rules and must not be used outside of + /// checkpoint compatibility checks. + /// + /// NOTE: The Arrow-level equivalent lives in `check_cast_compat` in + /// `engine/ensure_data_types.rs`. Changes here must be mirrored there. + pub(crate) fn is_checkpoint_cast_compatible(&self, target: &Self) -> bool { + matches!( + (self, target), + (Self::Integer, Self::Date) | (Self::Long, Self::Timestamp | Self::TimestampNtz) + ) + } + + /// Returns `true` if this primitive type is compatible with `target` for reading + /// `stats_parsed` columns from checkpoint parquet files. + /// + /// This is a superset of [`can_widen_to`]: it includes all Delta protocol type widening + /// rules plus physical Parquet encoding accommodations for checkpoint interop (see + /// [`is_checkpoint_cast_compatible`]). + /// + /// [`can_widen_to`]: PrimitiveType::can_widen_to + /// [`is_checkpoint_cast_compatible`]: PrimitiveType::is_checkpoint_cast_compatible + pub(crate) fn is_stats_type_compatible_with(&self, target: &Self) -> bool { + self == target || self.can_widen_to(target) || self.is_checkpoint_cast_compatible(target) + } } fn serialize_decimal( @@ -1491,6 +2085,14 @@ fn serialize_decimal( serializer.serialize_str(&format!("decimal({},{})", dtype.precision(), dtype.scale())) } +#[cfg(feature = "geo-type-in-dev")] +fn serialize_geotype( + value: &T, + serializer: S, +) -> Result { + serializer.serialize_str(&value.to_string()) +} + fn serialize_variant( _: &StructType, serializer: S, @@ -1498,6 +2100,60 @@ fn serialize_variant( serializer.serialize_str("variant") } +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum IntervalField { + Year, + Month, + Day, + Hour, + Minute, + Second, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct IntervalFieldRange { + pub(crate) start: IntervalField, + pub(crate) end: IntervalField, +} + +impl IntervalFieldRange { + fn primitive_type(self) -> PrimitiveType { + match self.start { + IntervalField::Year | IntervalField::Month => PrimitiveType::IntervalYearMonth, + IntervalField::Day + | IntervalField::Hour + | IntervalField::Minute + | IntervalField::Second => PrimitiveType::IntervalDayTime, + } + } +} + +pub(crate) fn parse_interval_type(s: &str) -> Option { + use IntervalField::*; + + let (start, end) = match s { + "interval year" => (Year, Year), + "interval month" => (Month, Month), + "interval year to month" => (Year, Month), + "interval day" => (Day, Day), + "interval hour" => (Hour, Hour), + "interval minute" => (Minute, Minute), + "interval second" => (Second, Second), + "interval day to hour" => (Day, Hour), + "interval day to minute" => (Day, Minute), + "interval day to second" => (Day, Second), + "interval hour to minute" => (Hour, Minute), + "interval hour to second" => (Hour, Second), + "interval minute to second" => (Minute, Second), + _ => return None, + }; + Some(IntervalFieldRange { start, end }) +} + +fn normalize_interval_type(s: &str) -> Option { + parse_interval_type(s).map(IntervalFieldRange::primitive_type) +} + // Custom Deserialize to provide clear error messages for unsupported types. // The derived impl would produce: "unknown variant `interval second`, expected one of ..." // This impl produces: "Unsupported Delta table type: 'interval second'" @@ -1521,6 +2177,11 @@ impl<'de> serde::Deserialize<'de> for PrimitiveType { "date" => Ok(PrimitiveType::Date), "timestamp" => Ok(PrimitiveType::Timestamp), "timestamp_ntz" => Ok(PrimitiveType::TimestampNtz), + "void" => Ok(PrimitiveType::Void), + // Accept canonical and narrowed interval spellings + s if s.starts_with("interval ") => { + normalize_interval_type(s).ok_or_else(|| unsupported_delta_type_error(s)) + } decimal_str if decimal_str.starts_with("decimal(") && decimal_str.ends_with(')') => { // Parse decimal type let mut parts = decimal_str[8..decimal_str.len() - 1].split(','); @@ -1548,9 +2209,37 @@ impl<'de> serde::Deserialize<'de> for PrimitiveType { .map(PrimitiveType::Decimal) .map_err(serde::de::Error::custom) } - unsupported => Err(serde::de::Error::custom(format!( - "Unsupported Delta table type: '{unsupported}'" - ))), + #[cfg(feature = "geo-type-in-dev")] + geo_str if geo_str.starts_with("geometry(") && geo_str.ends_with(')') => { + let crs = &geo_str["geometry(".len()..geo_str.len() - 1]; + GeometryType::try_new(crs.trim()) + .map(Box::new) + .map(PrimitiveType::Geometry) + .map_err(serde::de::Error::custom) + } + #[cfg(feature = "geo-type-in-dev")] + geo_str if geo_str.starts_with("geography(") && geo_str.ends_with(')') => { + let inner = &geo_str["geography(".len()..geo_str.len() - 1]; + // Kernel accepts only the canonical serialized form that every writer emits: + // geography(, ) + // TODO(#2949): reevaluate whether accepting padded input like + // geography( EPSG:4326 , vincenty ) is desired. + match inner.split_once(',') { + Some((crs, algo_str)) => { + let algorithm: EdgeInterpolationAlgorithm = + algo_str.trim().parse().map_err(serde::de::Error::custom)?; + GeographyType::try_new(crs.trim(), algorithm) + .map(Box::new) + .map(PrimitiveType::Geography) + .map_err(serde::de::Error::custom) + } + None => Err(serde::de::Error::custom(format!( + "Invalid geography type '{geo_str}': expected \ + 'geography(, )'" + ))), + } + } + unsupported => Err(unsupported_delta_type_error(unsupported)), } } } @@ -1570,9 +2259,16 @@ impl Display for PrimitiveType { PrimitiveType::Date => write!(f, "date"), PrimitiveType::Timestamp => write!(f, "timestamp"), PrimitiveType::TimestampNtz => write!(f, "timestamp_ntz"), + PrimitiveType::IntervalYearMonth => write!(f, "interval year to month"), + PrimitiveType::IntervalDayTime => write!(f, "interval day to second"), PrimitiveType::Decimal(dtype) => { write!(f, "decimal({},{})", dtype.precision(), dtype.scale()) } + PrimitiveType::Void => write!(f, "void"), + #[cfg(feature = "geo-type-in-dev")] + PrimitiveType::Geometry(t) => write!(f, "{t}"), + #[cfg(feature = "geo-type-in-dev")] + PrimitiveType::Geography(t) => write!(f, "{t}"), } } } @@ -1606,6 +2302,30 @@ impl From for DataType { PrimitiveType::from(dtype).into() } } +#[cfg(feature = "geo-type-in-dev")] +impl From for PrimitiveType { + fn from(gtype: GeometryType) -> Self { + PrimitiveType::Geometry(Box::new(gtype)) + } +} +#[cfg(feature = "geo-type-in-dev")] +impl From for DataType { + fn from(gtype: GeometryType) -> Self { + PrimitiveType::from(gtype).into() + } +} +#[cfg(feature = "geo-type-in-dev")] +impl From for PrimitiveType { + fn from(gtype: GeographyType) -> Self { + PrimitiveType::Geography(Box::new(gtype)) + } +} +#[cfg(feature = "geo-type-in-dev")] +impl From for DataType { + fn from(gtype: GeographyType) -> Self { + PrimitiveType::from(gtype).into() + } +} impl From for DataType { fn from(ptype: PrimitiveType) -> Self { DataType::Primitive(ptype) @@ -1669,15 +2389,15 @@ impl<'de> serde::Deserialize<'de> for DataType { if let Some(Value::String(type_str)) = map.get("type") { return match type_str.as_str() { "array" => ArrayType::deserialize(value) - .map(|at| DataType::Array(Box::new(at))) + .map(DataType::from) .map_err(|e| Error::custom(e.to_string())), "struct" => StructType::deserialize(value) - .map(|st| DataType::Struct(Box::new(st))) + .map(DataType::from) .map_err(|e| Error::custom(e.to_string())), "map" => MapType::deserialize(value) - .map(|mt| DataType::Map(Box::new(mt))) + .map(DataType::from) .map_err(|e| Error::custom(e.to_string())), - _ => Err(Error::custom(format!("Unknown complex type: '{type_str}'"))), + _ => Err(unsupported_delta_type_error(type_str)), }; } } @@ -1704,6 +2424,20 @@ impl DataType { pub const DATE: Self = DataType::Primitive(PrimitiveType::Date); pub const TIMESTAMP: Self = DataType::Primitive(PrimitiveType::Timestamp); pub const TIMESTAMP_NTZ: Self = DataType::Primitive(PrimitiveType::TimestampNtz); + pub const VOID: Self = DataType::Primitive(PrimitiveType::Void); + pub const INTERVAL_YEAR_MONTH: Self = DataType::Primitive(PrimitiveType::IntervalYearMonth); + pub const INTERVAL_DAY_TIME: Self = DataType::Primitive(PrimitiveType::IntervalDayTime); + + /// Compact type name for diagnostics that must not expand nested schemas. + pub(crate) fn kind_name(&self) -> String { + match self { + Self::Primitive(primitive) => primitive.to_string(), + Self::Array(_) => "array".to_string(), + Self::Struct(_) => "struct".to_string(), + Self::Map(_) => "map".to_string(), + Self::Variant(_) => "variant".to_string(), + } + } /// Create a new decimal type with the given precision and scale. pub fn decimal(precision: u8, scale: u8) -> DeltaResult { @@ -1835,114 +2569,280 @@ impl GetSchemaLeaves { } impl<'a> SchemaTransform<'a> for GetSchemaLeaves { - fn transform_struct_field(&mut self, field: &StructField) -> Option> { + transform_output_type!(|'a, T| ()); + + fn transform_struct_field(&mut self, field: &'a StructField) { self.path.push(field.name.clone()); if let DataType::Struct(_) = field.data_type { - let _ = self.recurse_into_struct_field(field); + self.recurse_into_struct_field(field); } else { self.names.push(ColumnName::new(&self.path)); self.types.push(field.data_type.clone()); } self.path.pop(); - None } } -struct MakePhysical<'a> { - column_mapping_mode: ColumnMappingMode, - path: Vec<&'a str>, - seen: HashMap, - err: Option, +/// What a [`MakePhysical`] walk does with each field. The two modes bundle the physical-rewrite +/// behavior with the matching treatment of a stale `delta.columnMapping.*` annotation left over on +/// a mapping-disabled table. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum MakePhysicalMode { + /// Rewrite each field to its physical name + metadata (the read/build path). A stale + /// annotation in `None` mode is tolerated: resolved by logical name and dropped from the + /// physical metadata. + Rewrite, + /// Validate annotations only, without rewriting (the strict write-path check). A stale + /// annotation in `None` mode is rejected. + ValidateStrict, } -impl<'a> MakePhysical<'a> { - fn new(column_mapping_mode: ColumnMappingMode) -> Self { - Self { - column_mapping_mode, - path: vec![], - seen: HashMap::new(), - err: None, + +impl MakePhysicalMode { + fn stale_annotation_policy(self) -> StaleAnnotationPolicy { + match self { + Self::Rewrite => StaleAnnotationPolicy::Ignore, + Self::ValidateStrict => StaleAnnotationPolicy::Reject, } } +} - /// Transforms a single [`StructField`] from logical to physical. Returns the physical - /// field on success, or the first error encountered during the recursive transformation. - fn run_field(&mut self, field: &'a StructField) -> DeltaResult { - let result = self.transform_struct_field(field); - match (self.err.take(), result) { - (Some(err), _) => Err(err), - // Theoretically impossible: MakePhysical only returns None when it sets an error - (None, None) => Err(Error::internal_error( - "make_physical: transform returned None without setting an error", - )), - (None, Some(field)) => Ok(field.into_owned()), +pub(crate) struct MakePhysical<'a> { + column_mapping_mode: ColumnMappingMode, + /// Logical path of current field's parent, used for error messages. + logical_path: Vec<&'a str>, + /// `delta.columnMapping.id` -> first claimer logical name. + seen_ids: HashMap, + /// Stack of sibling-`physicalName` maps. The top of the stack holds the current field's + /// siblings: key is the sibling's physical name, value is its logical name. Frames are + /// pushed in `transform_struct` (root struct included) and popped after iterating its + /// fields. Only structs introduce siblings; arrays/maps don't push frames since their + /// elements / keys / values are anonymous. + sibling_names_stack: Vec>, + /// Whether this walk rewrites fields to physical form or only validates (see + /// [`MakePhysicalMode`]). + mode: MakePhysicalMode, +} +impl<'a> MakePhysical<'a> { + fn new(column_mapping_mode: ColumnMappingMode) -> Self { + Self { + column_mapping_mode, + logical_path: vec![], + seen_ids: HashMap::new(), + sibling_names_stack: vec![], + mode: MakePhysicalMode::Rewrite, } } + /// Walks `schema` and validates its column-mapping annotations, rejecting stale annotations + /// left over on a column-mapping-disabled table. + pub(crate) fn validate_schema_column_mapping( + mode: ColumnMappingMode, + schema: &'a StructType, + ) -> DeltaResult<()> { + let mut walker = Self { + mode: MakePhysicalMode::ValidateStrict, + ..Self::new(mode) + }; + walker.transform_struct(schema).map(|_| ()) + } + fn transform_inner( &mut self, - field_name: &'a str, - transform: impl FnOnce(&mut Self) -> Option, - ) -> Option { - if self.err.is_some() { - return None; - } - self.path.push(field_name); + logical_name: &'a str, + transform: impl FnOnce(&mut Self) -> DeltaResult, + ) -> DeltaResult { + self.logical_path.push(logical_name); let result = transform(self); - self.path.pop(); + self.logical_path.pop(); result } } impl<'a> SchemaTransform<'a> for MakePhysical<'a> { - fn transform_array_element(&mut self, etype: &'a DataType) -> Option> { + transform_output_type!(|'a, T| DeltaResult>); + + fn transform_struct(&mut self, stype: &'a StructType) -> DeltaResult> { + self.sibling_names_stack.push(HashMap::new()); + let result = self.recurse_into_struct(stype); + self.sibling_names_stack.pop(); + result + } + + fn transform_array_element(&mut self, etype: &'a DataType) -> DeltaResult> { self.transform_inner("", |this| this.transform(etype)) } - fn transform_map_key(&mut self, ktype: &'a DataType) -> Option> { + fn transform_map_key(&mut self, ktype: &'a DataType) -> DeltaResult> { self.transform_inner("", |this| this.transform(ktype)) } - fn transform_map_value(&mut self, vtype: &'a DataType) -> Option> { + fn transform_map_value(&mut self, vtype: &'a DataType) -> DeltaResult> { self.transform_inner("", |this| this.transform(vtype)) } - fn transform_struct_field(&mut self, field: &'a StructField) -> Option> { - self.transform_inner(field.name(), |this| { - let (physical_name, _id) = get_field_column_mapping_info( - field, - this.column_mapping_mode, - &this.path, - Some(&mut this.seen), - ) - .map_err(|e| this.err = Some(e)) - .ok()?; + fn transform_struct_field( + &mut self, + field: &'a StructField, + ) -> DeltaResult> { + let (physical_name, _id) = validate_and_extract_column_mapping_annotations( + field, + self.column_mapping_mode, + self.mode.stale_annotation_policy(), + &self.logical_path, + Some(&mut self.seen_ids), + self.sibling_names_stack.last_mut(), + )?; - if field.is_metadata_column() { - return Some(Cow::Borrowed(field)); - } + if field.is_metadata_column() { + return Ok(Cow::Borrowed(field)); + } + self.transform_inner(field.name(), |this| { let field = this.recurse_into_struct_field(field)?; - + if this.mode == MakePhysicalMode::ValidateStrict { + return Ok(field); + } let metadata = field.logical_to_physical_metadata(this.column_mapping_mode); let name = physical_name.to_owned(); - - Some(Cow::Owned(field.with_name(name).with_metadata(metadata))) + Ok(Cow::Owned(field.with_name(name).with_metadata(metadata))) }) } - fn transform_variant(&mut self, stype: &'a StructType) -> Option> { + fn transform_variant(&mut self, stype: &'a StructType) -> DeltaResult> { // There is no column mapping metadata inside the struct fields of a variant, so // we do not recurse into the variant fields - Some(Cow::Borrowed(stype)) + Ok(Cow::Borrowed(stype)) } } #[cfg(test)] mod tests { + use rstest::rstest; + use serde_json; + + use super::*; use crate::table_features::ColumnMappingMode; - use crate::utils::test_utils::{ - assert_result_error_with_message, test_deep_nested_schema_missing_leaf_cm, + use crate::unit_test_utils::{ + assert_result_error_with_message, column_mapping_physical_name_dedup_fixtures as fixtures, + test_deep_nested_schema_missing_leaf_cm, }; - use super::*; - use rstest::rstest; - use serde_json; + #[cfg(feature = "geo-type-in-dev")] + fn geography(crs: &str, algorithm: EdgeInterpolationAlgorithm) -> PrimitiveType { + PrimitiveType::Geography(Box::new(GeographyType::try_new(crs, algorithm).unwrap())) + } + + #[cfg(feature = "geo-type-in-dev")] + fn geo_field_json(type_str: &str) -> String { + format!(r#"{{"name":"g","type":"{type_str}","nullable":true,"metadata":{{}}}}"#) + } + + #[cfg(feature = "geo-type-in-dev")] + #[rstest] + #[case( + "geometry(EPSG:4326)", + PrimitiveType::Geometry(Box::new(GeometryType::try_new("EPSG:4326").unwrap())), + "geometry(EPSG:4326)" + )] + #[case( + "geography(EPSG:4326, spherical)", + geography("EPSG:4326", EdgeInterpolationAlgorithm::Spherical), + "geography(EPSG:4326, spherical)" + )] + #[case( + "geography(EPSG:4326, vincenty)", + geography("EPSG:4326", EdgeInterpolationAlgorithm::Vincenty), + "geography(EPSG:4326, vincenty)" + )] + #[case( + "geography(EPSG:4326, thomas)", + geography("EPSG:4326", EdgeInterpolationAlgorithm::Thomas), + "geography(EPSG:4326, thomas)" + )] + #[case( + "geography(EPSG:4326, andoyer)", + geography("EPSG:4326", EdgeInterpolationAlgorithm::Andoyer), + "geography(EPSG:4326, andoyer)" + )] + #[case( + "geography(EPSG:4326, karney)", + geography("EPSG:4326", EdgeInterpolationAlgorithm::Karney), + "geography(EPSG:4326, karney)" + )] + #[case( + "geography( EPSG:4326 , karney )", + geography("EPSG:4326", EdgeInterpolationAlgorithm::Karney), + "geography(EPSG:4326, karney)" + )] + #[case( + "geography(EPSG:4326, SPHERICAL)", + geography("EPSG:4326", EdgeInterpolationAlgorithm::Spherical), + "geography(EPSG:4326, spherical)" + )] + #[case( + "geography(EPSG:4326, Vincenty)", + geography("EPSG:4326", EdgeInterpolationAlgorithm::Vincenty), + "geography(EPSG:4326, vincenty)" + )] + fn test_geo_round_trip( + #[case] type_str: &str, + #[case] expected: PrimitiveType, + #[case] canonical: &str, + ) { + let field: StructField = serde_json::from_str(&geo_field_json(type_str)).unwrap(); + assert_eq!(field.data_type, DataType::Primitive(expected)); + + let json_str = serde_json::to_string(&field).unwrap(); + assert_eq!(json_str, geo_field_json(canonical)); + } + + #[cfg(feature = "geo-type-in-dev")] + #[rstest] + #[case("geography(EPSG:4326, unknown_algo)", "Matching variant not found")] + #[case("geography(EPSG:4326,)", "Matching variant not found")] + #[case("geometry(EPSG:4326", "Unsupported Delta table type")] + #[case("geographyz", "Unsupported Delta table type")] + #[case("geometry", "Unsupported Delta table type")] + #[case("geography", "Unsupported Delta table type")] + #[case("geometry()", "must be in 'AUTHORITY:CODE' format")] + #[case("geography(, vincenty)", "must be in 'AUTHORITY:CODE' format")] + #[case("geography(EPSG:4326)", "expected 'geography(, )'")] + #[case("geography(vincenty)", "expected 'geography(, )'")] + #[case("geography(EPSG:4326, vincenty, karney)", "Matching variant not found")] + fn test_invalid_geo_format(#[case] invalid_type: &str, #[case] expected_error: &str) { + let result: Result = serde_json::from_str(&geo_field_json(invalid_type)); + let err = result.expect_err(&format!("expected '{invalid_type}' to be rejected")); + assert!( + err.to_string().contains(expected_error), + "Expected error containing '{expected_error}', got: {err}" + ); + } + + #[cfg(feature = "geo-type-in-dev")] + #[rstest] + fn test_geo_try_new_rejects_invalid_crs( + #[values( + "foo", + "authority:", + ":", + "", + ":CRS84", + " EPSG:4326", + "EPSG:4326 ", + " EPSG:4326 ", + "EPSG:4326:extra", + "a:b:c", + "EPSG:1,2" + )] + crs: &str, + ) { + let geometry_err = + GeometryType::try_new(crs).expect_err(&format!("expected '{crs}' to be rejected")); + let geography_err = GeographyType::try_new(crs, EdgeInterpolationAlgorithm::Spherical) + .expect_err(&format!("expected '{crs}' to be rejected")); + for err in [geometry_err, geography_err] { + assert!( + err.to_string().contains("CRS"), + "expected CRS error for '{crs}', got: {err}" + ); + } + } fn example_schema_metadata() -> &'static str { r#" @@ -2089,6 +2989,53 @@ mod tests { ); } + #[test] + fn test_roundtrip_void() { + let data = r#" + { + "name": "v", + "type": "void", + "nullable": true, + "metadata": {} + } + "#; + let field: StructField = serde_json::from_str(data).unwrap(); + assert_eq!(field.data_type, DataType::VOID); + + let json_str = serde_json::to_string(&field).unwrap(); + assert_eq!( + json_str, + r#"{"name":"v","type":"void","nullable":true,"metadata":{}}"# + ); + } + + #[test] + fn test_roundtrip_void_non_nullable() { + let data = r#" + { + "name": "v", + "type": "void", + "nullable": false, + "metadata": {} + } + "#; + let field: StructField = serde_json::from_str(data).unwrap(); + assert_eq!(field.data_type, DataType::VOID); + assert!(!field.nullable); + + let json_str = serde_json::to_string(&field).unwrap(); + assert_eq!( + json_str, + r#"{"name":"v","type":"void","nullable":false,"metadata":{}}"# + ); + } + + #[test] + fn test_void_display() { + assert_eq!(PrimitiveType::Void.to_string(), "void"); + assert_eq!(DataType::VOID.to_string(), "void"); + } + #[test] fn test_unshredded_variant() { let unshredded_variant_type = DataType::unshredded_variant(); @@ -2111,9 +3058,18 @@ mod tests { } #[rstest] - #[case("interval second")] - #[case("interval day")] #[case("money")] + #[case("interval fortnight")] + // invalid orderings across year-month and day-time + #[case("interval month to day")] + #[case("interval year to second")] + // invalid orderings within year-month and day-time + #[case("interval month to year")] + #[case("interval year to year")] + #[case("interval second to day")] + #[case("interval minute to minute")] + // too many fields + #[case("interval year to month to year")] fn test_unsupported_type_error_message(#[case] unsupported_type: &str) { let data = format!( r#"{{ @@ -2146,6 +3102,19 @@ mod tests { #[case("date", DataType::DATE)] #[case("timestamp", DataType::TIMESTAMP)] #[case("timestamp_ntz", DataType::TIMESTAMP_NTZ)] + #[case("interval year", DataType::INTERVAL_YEAR_MONTH)] + #[case("interval month", DataType::INTERVAL_YEAR_MONTH)] + #[case("interval year to month", DataType::INTERVAL_YEAR_MONTH)] + #[case("interval day", DataType::INTERVAL_DAY_TIME)] + #[case("interval hour", DataType::INTERVAL_DAY_TIME)] + #[case("interval minute", DataType::INTERVAL_DAY_TIME)] + #[case("interval second", DataType::INTERVAL_DAY_TIME)] + #[case("interval day to hour", DataType::INTERVAL_DAY_TIME)] + #[case("interval day to minute", DataType::INTERVAL_DAY_TIME)] + #[case("interval day to second", DataType::INTERVAL_DAY_TIME)] + #[case("interval hour to minute", DataType::INTERVAL_DAY_TIME)] + #[case("interval hour to second", DataType::INTERVAL_DAY_TIME)] + #[case("interval minute to second", DataType::INTERVAL_DAY_TIME)] fn test_primitive_type_deserialization_still_works( #[case] type_str: &str, #[case] expected_type: DataType, @@ -2208,18 +3177,18 @@ mod tests { #[rstest] #[case( r#"{"type": "array", "elementType": "integer", "containsNull": false}"#, - DataType::Array(Box::new(ArrayType::new(DataType::INTEGER, false))) + DataType::from(ArrayType::new(DataType::INTEGER, false)) )] #[case( r#"{"type": "struct", "fields": [{"name": "a", "type": "integer", "nullable": false, "metadata": {}}, {"name": "b", "type": "string", "nullable": true, "metadata": {}}]}"#, - DataType::Struct(Box::new(StructType::new_unchecked([ + DataType::from(StructType::new_unchecked([ StructField::new("a", DataType::INTEGER, false), StructField::new("b", DataType::STRING, true), - ]))) + ])) )] #[case( r#"{"type": "map", "keyType": "string", "valueType": "integer", "valueContainsNull": true}"#, - DataType::Map(Box::new(MapType::new(DataType::STRING, DataType::INTEGER, true))) + DataType::from(MapType::new(DataType::STRING, DataType::INTEGER, true)) )] #[case("\"string\"", DataType::STRING)] #[case("\"long\"", DataType::LONG)] @@ -2233,18 +3202,35 @@ mod tests { #[case("\"date\"", DataType::DATE)] #[case("\"timestamp\"", DataType::TIMESTAMP)] #[case("\"timestamp_ntz\"", DataType::TIMESTAMP_NTZ)] + #[case("\"interval year to month\"", DataType::INTERVAL_YEAR_MONTH)] + #[case("\"interval day to second\"", DataType::INTERVAL_DAY_TIME)] #[case("\"variant\"", DataType::unshredded_variant())] fn test_data_type_deserialization(#[case] type_json: &str, #[case] expected: DataType) { let data_type: DataType = serde_json::from_str(type_json).unwrap(); assert_eq!(data_type, expected); } + #[rstest] + #[case(PrimitiveType::IntervalYearMonth, "interval year to month")] + #[case(PrimitiveType::IntervalDayTime, "interval day to second")] + fn test_interval_type_name_round_trips(#[case] ptype: PrimitiveType, #[case] name: &str) { + assert_eq!(ptype.to_string(), name); + assert_eq!( + serde_json::to_string(&ptype).unwrap(), + format!("\"{name}\"") + ); + assert_eq!( + serde_json::from_str::(&format!("\"{name}\"")).unwrap(), + ptype + ); + } + #[test] fn test_make_physical_no_column_mapping() { let field = StructField::nullable( "e", ArrayType::new( - StructType::new_unchecked([StructField::not_null("d", DataType::INTEGER)]).into(), + StructType::new_unchecked([StructField::not_null("d", DataType::INTEGER)]), true, ), ); @@ -2269,10 +3255,40 @@ mod tests { } #[test] - fn test_make_physical_rejects_annotated_fields_when_column_mapping_disabled() { + fn test_make_physical_tolerates_stale_annotations_when_column_mapping_disabled() { + // A table can carry `delta.columnMapping.*` annotations after mapping was enabled and then + // disabled. They are inert while mapping is off, so `make_physical` (the read path) + // tolerates them: the field keeps its logical name and the CM keys are dropped from the + // physical metadata, leaving a schema indistinguishable from a table that never had them. let data = example_schema_metadata(); let field: StructField = serde_json::from_str(data).unwrap(); - assert!(field.make_physical(ColumnMappingMode::None).is_err()); + let physical = field.make_physical(ColumnMappingMode::None).unwrap(); + + assert_eq!(physical.name, "e"); + assert!(!physical + .metadata + .contains_key(ColumnMetadataKey::ColumnMappingId.as_ref())); + assert!(!physical + .metadata + .contains_key(ColumnMetadataKey::ColumnMappingPhysicalName.as_ref())); + // Non-column-mapping metadata is untouched. + assert!(physical.metadata.contains_key("delta.identity.start")); + + // The nested leaf `d` is likewise tolerated: logical name kept, CM keys dropped. + let DataType::Array(atype) = &physical.data_type else { + panic!("Expected an Array"); + }; + let DataType::Struct(stype) = atype.element_type() else { + panic!("Expected a Struct"); + }; + let leaf = stype.fields().next().unwrap(); + assert_eq!(leaf.name, "d"); + assert!(!leaf + .metadata + .contains_key(ColumnMetadataKey::ColumnMappingId.as_ref())); + assert!(!leaf + .metadata + .contains_key(ColumnMetadataKey::ColumnMappingPhysicalName.as_ref())); } #[test] @@ -2312,11 +3328,7 @@ mod tests { ]); let schema = StructType::new_unchecked([ cm_field("a", 1, DataType::INTEGER), - cm_field( - "b", - 2, - ArrayType::new(DataType::Struct(Box::new(inner)), true), - ), + cm_field("b", 2, ArrayType::new(inner, true)), cm_field("c", 3, DataType::STRING), ]); assert_result_error_with_message( @@ -2325,6 +3337,44 @@ mod tests { ); } + #[rstest] + #[case::accepted_same_phy_name_different_paths(fixtures::same_phy_name_different_paths(), /*expected_error_substring*/None)] + #[case::rejected_deeply_nested_repeat_physical_paths( + fixtures::deeply_nested_repeat_physical_paths(), + Some({ + let (a, b) = + fixtures::deeply_nested_collider_paths(); + format!("assigned to both '{a}' and '{b}'") + }), + )] + #[case::multiple_physical_name_collisions_reports_first( + fixtures::multiple_physical_name_collisions(), + Some("'p' assigned to both 'a' and 'b'".to_string()), + )] + fn test_make_physical_dup_physical_name( + #[case] schema: StructType, + #[case] expected_error_substring: Option, + ) { + // The same dedup rules should apply under both CM modes. + for mode in [ColumnMappingMode::Name, ColumnMappingMode::Id] { + let result = schema.make_physical(mode); + match &expected_error_substring { + None => { + result.expect("The input schema should be valid"); + } + Some(substr) => { + assert_result_error_with_message(result.as_ref().map(|_| ()), substr); + if let Err(e) = &result { + assert!( + !e.to_string().contains("'q'"), + "walker must short-circuit on first collision under {mode:?}; got: {e}" + ); + } + } + } + } + } + #[test] fn test_make_physical_column_mapping() { [ColumnMappingMode::Name, ColumnMappingMode::Id] @@ -2617,6 +3667,70 @@ mod tests { assert!(schema_has_invariants(&schema)); } + fn all_nullable_schema() -> StructType { + StructType::new_unchecked([ + StructField::nullable("a", DataType::STRING), + StructField::nullable("b", DataType::INTEGER), + ]) + } + + fn top_level_non_null_schema() -> StructType { + StructType::new_unchecked([ + StructField::not_null("id", DataType::INTEGER), + StructField::nullable("name", DataType::STRING), + ]) + } + + fn nested_non_null_schema() -> StructType { + let nested_field = StructField::nullable( + "parent", + DataType::try_struct_type([StructField::not_null("child", DataType::INTEGER)]).unwrap(), + ); + StructType::new_unchecked([StructField::nullable("a", DataType::STRING), nested_field]) + } + + fn array_non_null_schema() -> StructType { + let array_field = StructField::nullable( + "arr", + ArrayType::new( + DataType::try_struct_type([StructField::not_null("child", DataType::INTEGER)]) + .unwrap(), + true, + ), + ); + StructType::new_unchecked([array_field]) + } + + fn map_non_null_schema() -> StructType { + let map_field = StructField::nullable( + "map", + MapType::new( + DataType::STRING, + DataType::try_struct_type([StructField::not_null("child", DataType::INTEGER)]) + .unwrap(), + true, + ), + ); + StructType::new_unchecked([map_field]) + } + + fn variant_only_schema() -> StructType { + // Variant internal fields (metadata, value) are protocol-defined non-null but + // must NOT be counted as user-controlled non-null fields. + StructType::new_unchecked([StructField::nullable("v", DataType::unshredded_variant())]) + } + + #[rstest] + #[case::all_nullable(all_nullable_schema(), false)] + #[case::top_level(top_level_non_null_schema(), true)] + #[case::nested_struct(nested_non_null_schema(), true)] + #[case::array_element(array_non_null_schema(), true)] + #[case::map_value(map_non_null_schema(), true)] + #[case::variant_skipped(variant_only_schema(), false)] + fn test_schema_contains_non_null_fields(#[case] schema: StructType, #[case] expected: bool) { + assert_eq!(schema_contains_non_null_fields(&schema), expected); + } + #[test] fn test_struct_type_iterator_basic() { let fields = vec![ @@ -3173,7 +4287,7 @@ mod tests { let result = StructType::try_new([ StructField::nullable("regular_col", DataType::STRING), - StructField::nullable("nested", DataType::Struct(Box::new(nested_struct))), + StructField::nullable("nested", nested_struct), ]); assert_result_error_with_message(result, "only allowed at the top level"); @@ -3195,11 +4309,11 @@ mod tests { .collect(), metadata_columns: HashMap::new(), }; - let array_type = ArrayType::new(DataType::Struct(Box::new(nested_struct)), true); + let array_type = ArrayType::new(nested_struct, true); let result = StructType::try_new([ StructField::nullable("regular_col", DataType::STRING), - StructField::nullable("array_col", DataType::Array(Box::new(array_type))), + StructField::nullable("array_col", array_type), ]); assert_result_error_with_message(result, "only allowed at the top level"); @@ -3223,20 +4337,12 @@ mod tests { }; for map_type in [ - MapType::new( - DataType::Struct(Box::new(nested_struct.clone())), - DataType::STRING, - true, - ), - MapType::new( - DataType::STRING, - DataType::Struct(Box::new(nested_struct)), - true, - ), + MapType::new(nested_struct.clone(), DataType::STRING, true), + MapType::new(DataType::STRING, nested_struct, true), ] { let result = StructType::try_new([ StructField::nullable("regular_col", DataType::STRING), - StructField::nullable("map_col", DataType::Map(Box::new(map_type))), + StructField::nullable("map_col", map_type), ]); assert_result_error_with_message(result, "only allowed at the top level"); @@ -3407,16 +4513,12 @@ mod tests { let nested_struct = StructType::new_unchecked([ nested_field_with_metadata, StructField::new("x", DataType::DOUBLE, true), - StructField::new( - "inner_struct", - DataType::Struct(Box::new(inner_struct)), - false, - ), + StructField::new("inner_struct", inner_struct, false), ]); - let array_type = ArrayType::new(DataType::Struct(Box::new(nested_struct.clone())), true); + let array_type = ArrayType::new(nested_struct.clone(), true); let map_type = MapType::new( - DataType::Struct(Box::new(nested_struct.clone())), - DataType::Struct(Box::new(nested_struct.clone())), // kek + nested_struct.clone(), + nested_struct.clone(), // kek true, ); let fields = vec![ @@ -3424,8 +4526,8 @@ mod tests { StructField::new("y", DataType::FLOAT, false), StructField::new("z", DataType::LONG, true), StructField::new("s", nested_struct.clone(), false), - StructField::nullable("array_col", DataType::Array(Box::new(array_type))), - StructField::nullable("map_col", DataType::Map(Box::new(map_type))), + StructField::nullable("array_col", array_type), + StructField::nullable("map_col", map_type), StructField::new("a", DataType::LONG, true), ]; @@ -3522,205 +4624,104 @@ mod tests { } #[test] - fn test_with_field_inserted_empty_struct() { - let schema = StructType::try_new([]).unwrap(); - let schema = schema - .with_field_inserted_after(None, StructField::new("age", DataType::STRING, true)) - .expect("with field inserted should produce a valid schema"); - assert_eq!(schema.num_fields(), 1); - assert_eq!(schema.field_at_index(0).unwrap().name(), "age"); - } - - #[test] - fn test_with_field_inserted() { - let schema = StructType::try_new([ - StructField::new("id", DataType::INTEGER, false), - StructField::new("name", DataType::STRING, true), - ]) - .unwrap(); - let schema = schema - .with_field_inserted_after(Some("id"), StructField::new("age", DataType::STRING, true)) - .expect("with field inserted should produce a valid schema"); - assert_eq!(schema.num_fields(), 3); - assert_eq!(schema.field_at_index(0).unwrap().name(), "id"); - assert_eq!(schema.field_at_index(1).unwrap().name(), "age"); - assert_eq!(schema.field_at_index(2).unwrap().name(), "name"); - } - - #[test] - fn test_with_field_inserted_append_to_end() { - let schema = StructType::try_new([ - StructField::new("id", DataType::INTEGER, false), - StructField::new("name", DataType::STRING, true), - ]) - .unwrap(); - let schema = schema - .with_field_inserted_after(None, StructField::new("age", DataType::STRING, true)) - .expect("with field inserted should produce a valid schema"); - - assert_eq!(schema.num_fields(), 3); - assert_eq!(schema.field_at_index(0).unwrap().name(), "id"); - assert_eq!(schema.field_at_index(1).unwrap().name(), "name"); - assert_eq!(schema.field_at_index(2).unwrap().name(), "age"); - } - - #[test] - fn test_with_field_inserted_after_non_existent_field() { - let schema = - StructType::try_new([StructField::new("id", DataType::INTEGER, false)]).unwrap(); - let new_schema = schema.with_field_inserted_after( - Some("nonexistent"), - StructField::new("name", DataType::STRING, true), - ); - assert!(new_schema.is_err()); - } - - #[test] - fn test_with_field_inserted_after_duplicate_field() { - let schema = StructType::try_new([ - StructField::new("id", DataType::INTEGER, false), - StructField::new("name", DataType::STRING, true), - ]) - .unwrap(); - let new_schema = schema.with_field_inserted_after( - Some("name"), - StructField::new("id", DataType::STRING, true), - ); - assert!(new_schema.is_err()); - assert_result_error_with_message(new_schema, "Field id already exists"); - } - - #[test] - fn test_with_field_inserted_before() { - let schema = StructType::try_new([ - StructField::new("id", DataType::INTEGER, false), - StructField::new("name", DataType::STRING, true), - ]) - .unwrap(); - let schema = schema - .with_field_inserted_before( - Some("name"), - StructField::new("age", DataType::STRING, true), - ) - .expect("with field inserted before should produce a valid schema"); - assert_eq!(schema.num_fields(), 3); - assert_eq!(schema.field_at_index(0).unwrap().name(), "id"); - assert_eq!(schema.field_at_index(1).unwrap().name(), "age"); - assert_eq!(schema.field_at_index(2).unwrap().name(), "name"); - } - - #[test] - fn test_with_field_inserted_before_duplicate_field() { - let schema = StructType::try_new([ - StructField::new("id", DataType::INTEGER, false), - StructField::new("name", DataType::STRING, true), - ]) - .unwrap(); - let new_schema = schema.with_field_inserted_before( - Some("name"), - StructField::new("id", DataType::STRING, true), - ); - assert!(new_schema.is_err()); - assert_result_error_with_message(new_schema, "Field id already exists"); - } - - #[test] - fn test_with_field_inserted_before_at_beginning() { - let schema = StructType::try_new([ - StructField::new("id", DataType::INTEGER, false), - StructField::new("name", DataType::STRING, true), - ]) - .unwrap(); - let schema = schema - .with_field_inserted_before(None, StructField::new("age", DataType::STRING, true)) - .expect("with field inserted before should produce a valid schema"); - assert_eq!(schema.num_fields(), 3); - assert_eq!(schema.field_at_index(0).unwrap().name(), "age"); - assert_eq!(schema.field_at_index(1).unwrap().name(), "id"); - assert_eq!(schema.field_at_index(2).unwrap().name(), "name"); - } - - #[test] - fn test_with_field_inserted_before_non_existent_field() { - let schema = - StructType::try_new([StructField::new("id", DataType::INTEGER, false)]).unwrap(); - let new_schema = schema.with_field_inserted_before( - Some("nonexistent"), - StructField::new("name", DataType::STRING, true), + fn test_current_default_key_value() { + assert_eq!( + ColumnMetadataKey::CurrentDefault.as_ref(), + "CURRENT_DEFAULT" ); - assert!(new_schema.is_err()); } - #[test] - fn test_with_field_inserted_before_empty_struct() { - let schema = StructType::try_new([]).unwrap(); - let schema = schema - .with_field_inserted_before(None, StructField::new("age", DataType::STRING, true)) - .expect("with field inserted before on empty struct should succeed"); - assert_eq!(schema.num_fields(), 1); - assert_eq!(schema.field_at_index(0).unwrap().name(), "age"); - } + mod column_default_method { + use super::*; + use crate::schema::column_default::field_with_default; - #[test] - fn test_with_field_removed() { - let schema = - StructType::try_new([StructField::new("id", DataType::INTEGER, false)]).unwrap(); - let new_schema = schema.with_field_removed("id"); - assert_eq!(new_schema.num_fields(), 0); - } + #[test] + fn returns_none_when_no_current_default() { + let field = StructField::nullable("c", DataType::INTEGER); + assert_eq!(field.column_default().unwrap(), None); + } - #[test] - fn test_with_field_removed_non_existent_field() { - let schema = - StructType::try_new([StructField::new("id", DataType::INTEGER, false)]).unwrap(); - let new_schema = schema.with_field_removed("nonexistent"); - assert_eq!(new_schema.num_fields(), 1); - assert_eq!(new_schema.field_at_index(0).unwrap().name(), "id"); - } + #[test] + fn errors_when_current_default_is_not_a_string() { + let field = StructField::nullable("c", DataType::INTEGER).add_metadata([( + ColumnMetadataKey::CurrentDefault.as_ref().to_string(), + MetadataValue::Number(42), + )]); + let err = field + .column_default() + .expect_err("a non-string CURRENT_DEFAULT must error") + .to_string(); + assert!(err.contains("non-string"), "got: {err}"); + } - #[test] - fn test_with_field_replaced() { - let schema = - StructType::try_new([StructField::new("id", DataType::INTEGER, false)]).unwrap(); - let new_schema = schema - .with_field_replaced("id", StructField::new("name", DataType::STRING, true)) - .unwrap(); + #[rstest] + #[case::parsable_literal(DataType::INTEGER, "42", true)] + #[case::null_primitive(DataType::INTEGER, "NULL", true)] + #[case::unparsable_function_call(DataType::TIMESTAMP, "current_timestamp()", false)] + #[case::unparsable_type_mismatch(DataType::TIMESTAMP, "0.18", false)] + fn exposes_default_for_primitive( + #[case] data_type: DataType, + #[case] raw_sql: &str, + #[case] parsable: bool, + ) { + let field = field_with_default("c", data_type.clone(), raw_sql); + let column_default = field + .column_default() + .unwrap() + .expect("default must be present"); + assert_eq!(column_default.raw_sql(), raw_sql); + assert_eq!(column_default.data_type(), &data_type); + assert_eq!(column_default.to_scalar().unwrap().is_some(), parsable); + } - assert_eq!(new_schema.num_fields(), 1); - assert_eq!(new_schema.field_at_index(0).unwrap().name(), "name"); - } + #[rstest] + #[case::array(DataType::from(ArrayType::new(DataType::INTEGER, true)), "ARRAY(1)")] + #[case::map( + DataType::from(MapType::new(DataType::STRING, DataType::INTEGER, true)), + "MAP('a', 1)" + )] + fn non_null_default_on_container_is_tolerated( + #[case] data_type: DataType, + #[case] raw_sql: &str, + ) { + let field = field_with_default("c", data_type.clone(), raw_sql); + let column_default = field + .column_default() + .unwrap() + .expect("default must be present"); + // Kernel cannot parse a non-primitive default, so it surfaces via raw SQL. + assert_eq!(column_default.raw_sql(), raw_sql); + assert_eq!(column_default.data_type(), &data_type); + assert_eq!(column_default.to_scalar().unwrap(), None); + } - #[test] - fn test_with_field_replaced_non_existent_field() { - let schema = - StructType::try_new([StructField::new("id", DataType::INTEGER, false)]).unwrap(); - let new_schema = schema.with_field_replaced( - "nonexistent", - StructField::new("name", DataType::STRING, true), - ); - assert!(new_schema.is_err(), "Expected error for non-existent field"); + #[test] + fn non_null_default_on_variant_errors() { + let field = field_with_default("v", DataType::unshredded_variant(), "1"); + let err = field + .column_default() + .expect_err("a non-NULL default on a Variant column must error") + .to_string(); + assert!(err.contains("Variant"), "got: {err}"); + } } /// Schema: { a: { b: { c: double } } } — supports walks at depths 1, 2, and 3. fn walk_test_schema() -> StructType { let l3 = StructType::new_unchecked([StructField::new("c", DataType::DOUBLE, false)]); - let l2 = StructType::new_unchecked([StructField::new( - "b", - DataType::Struct(Box::new(l3)), - false, - )]); - StructType::new_unchecked([StructField::new("a", DataType::Struct(Box::new(l2)), false)]) + let l2 = StructType::new_unchecked([StructField::new("b", l3, false)]); + StructType::new_unchecked([StructField::new("a", l2, false)]) } #[rstest::rstest] - #[case::single_level(vec!["a"], vec!["a"], DataType::Struct(Box::new( - StructType::new_unchecked([StructField::new("b", DataType::Struct(Box::new( - StructType::new_unchecked([StructField::new("c", DataType::DOUBLE, false)]) - )), false)]) - )))] - #[case::nested_2(vec!["a", "b"], vec!["a", "b"], DataType::Struct(Box::new( - StructType::new_unchecked([StructField::new("c", DataType::DOUBLE, false)]) - )))] + #[case::single_level(vec!["a"], vec!["a"], DataType::from(StructType::new_unchecked([ + StructField::new("b", StructType::new_unchecked([ + StructField::new("c", DataType::DOUBLE, false) + ]), false) + ])))] + #[case::nested_2(vec!["a", "b"], vec!["a", "b"], DataType::from(StructType::new_unchecked([ + StructField::new("c", DataType::DOUBLE, false) + ])))] #[case::nested_3(vec!["a", "b", "c"], vec!["a", "b", "c"], DataType::DOUBLE)] #[test] fn test_walk_column_fields_happy( @@ -3730,7 +4731,7 @@ mod tests { ) { let schema = walk_test_schema(); let fields = schema - .walk_column_fields(&ColumnName::new(col_path.iter().copied())) + .fields_of_path(&ColumnName::new(col_path.iter().copied())) .unwrap(); assert_eq!(fields.len(), expected_names.len()); for (field, name) in fields.iter().zip(expected_names.iter()) { @@ -3747,7 +4748,46 @@ mod tests { #[test] fn test_walk_column_fields_error(#[case] col_path: Vec<&str>, #[case] expected_error: &str) { let schema = walk_test_schema(); - let result = schema.walk_column_fields(&ColumnName::new(col_path.iter().copied())); + let result = schema.fields_of_path(&ColumnName::new(col_path.iter().copied())); assert_result_error_with_message(result, expected_error); } + + #[test] + fn test_normalize_column_names_to_schema_casing() { + let inner = + StructType::new_unchecked(vec![StructField::new("City", DataType::STRING, false)]); + let schema = StructType::new_unchecked(vec![ + StructField::new("id", DataType::INTEGER, false), + StructField::new("EventDate", DataType::DATE, false), + StructField::new("Address", inner, false), + ]); + + // Mismatched casing -> normalized to schema + let cols = vec![column_name!("eventdate")]; + assert_eq!( + normalize_column_names_to_schema_casing(&schema, &cols)[0].path(), + ["EventDate"] + ); + + // Nested path -> each field name normalized + let cols = vec![column_name!("address.city")]; + assert_eq!( + normalize_column_names_to_schema_casing(&schema, &cols)[0].path(), + ["Address", "City"] + ); + + // Already matching -> unchanged + let cols = vec![column_name!("id")]; + assert_eq!( + normalize_column_names_to_schema_casing(&schema, &cols)[0].path(), + ["id"] + ); + + // Unrecognized -> keeps original + let cols = vec![column_name!("nonexistent")]; + assert_eq!( + normalize_column_names_to_schema_casing(&schema, &cols)[0].path(), + ["nonexistent"] + ); + } } diff --git a/kernel/src/schema/validation.rs b/kernel/src/schema/validation.rs new file mode 100644 index 0000000000..cf545b0679 --- /dev/null +++ b/kernel/src/schema/validation.rs @@ -0,0 +1,374 @@ +//! Schema validation utilities shared by table creation and schema evolution. +//! +//! Validates schemas per the Delta protocol specification. + +use std::collections::HashSet; + +use crate::schema::{StructField, StructType}; +use crate::table_features::ColumnMappingMode; +use crate::transforms::SchemaTransform; +use crate::{transform_output_type, DeltaResult, Error}; + +/// Characters that are invalid in Parquet column names when column mapping is disabled. +/// These characters have special meaning in Parquet schema syntax. +const INVALID_PARQUET_CHARS: &[char] = &[' ', ',', ';', '{', '}', '(', ')', '\n', '\t', '=']; + +/// Validates a schema for CREATE TABLE or ALTER TABLE. +/// +/// Performs the following checks: +/// 1. No duplicate column names (case-insensitive, including nested fields) +/// 2. Column names contain only valid characters +/// 3. Rejects fields with `delta.invariants` metadata (SQL expression invariants are not supported +/// by kernel) +pub(crate) fn validate_schema( + schema: &StructType, + column_mapping_mode: ColumnMappingMode, +) -> DeltaResult<()> { + let mut validator = SchemaValidator::new(column_mapping_mode); + // We reuse the SchemaTransform trait for its recursive traversal machinery. + // The validator never transforms the schema -- it only inspects fields and + // collects errors. The return value is intentionally discarded. + validator.transform_struct(schema); + validator.into_result() +} + +/// Schema visitor that validates field names, detects duplicates, and rejects +/// unsupported column metadata. +/// +/// Implements `SchemaTransform` to reuse the existing recursive struct/array/map traversal. +/// Collects all validation errors so the caller gets a complete list of violations in a +/// single error message. +/// +/// Note: `StructType::try_new` already catches same-level case-insensitive duplicates. +/// This validator additionally detects cross-level path duplicates and catches schemas +/// built with `new_unchecked`. +struct SchemaValidator { + cm_enabled: bool, + seen_paths: HashSet, + current_path: Vec, + errors: Vec, +} + +impl SchemaValidator { + fn new(column_mapping_mode: ColumnMappingMode) -> Self { + Self { + cm_enabled: !matches!(column_mapping_mode, ColumnMappingMode::None), + seen_paths: HashSet::new(), + current_path: Vec::new(), + errors: Vec::new(), + } + } + + fn into_result(self) -> DeltaResult<()> { + if self.errors.is_empty() { + Ok(()) + } else { + Err(Error::generic(format!( + "Schema validation failed:\n- {}", + self.errors.join("\n- ") + ))) + } + } +} + +impl<'a> SchemaTransform<'a> for SchemaValidator { + transform_output_type!(|'a, T| ()); + + fn transform_struct_field(&mut self, field: &'a StructField) { + if let Err(e) = validate_field_name(field.name(), self.cm_enabled) { + self.errors.push(e.to_string()); + } + + // Check duplicate paths. We use a null-byte separator instead of dots because + // column names can contain literal dots when column mapping is enabled. A dot + // separator would make column "a.b" indistinguishable from nested field b in + // struct a. Null bytes cannot appear in column names, so they are safe to use. + self.current_path.push(field.name().to_ascii_lowercase()); + + // Reject `delta.invariants` metadata on any field. Kernel cannot evaluate SQL + // expression invariants, so reject at create time with a clear, path-aware error. + // + // Note: unlike `NonNullFieldChecker`, this validator intentionally does NOT + // skip recursion into variant internals. Variants are not expected to carry + // `delta.invariants`; if they ever do, bubble the error up loudly instead of + // silently skipping it. + // + // When kernel gains SQL expression invariant support, remove this rejection + // and replace it with a check that delegates to the invariant evaluation + // pipeline. + if field.has_invariants() { + self.errors.push(format!( + "Column '{}' has `delta.invariants` metadata; SQL expression invariants \ + are not supported by kernel", + self.current_path.join(".") + )); + } + + let key = self.current_path.join("\0"); + if !self.seen_paths.insert(key) { + self.errors.push(format!( + "Schema contains duplicate column (case-insensitive): '{}'", + field.name() + )); + } + + self.recurse_into_struct_field(field); + self.current_path.pop(); + } +} + +/// Validates an individual field name. +/// +/// When column mapping is disabled, rejects names containing Parquet special characters. +/// When column mapping is enabled, only rejects newlines since physical names are +/// auto-generated but newlines in column names break metadata serialization regardless +/// of column mapping mode. +fn validate_field_name(name: &str, cm_enabled: bool) -> DeltaResult<()> { + if name.is_empty() { + return Err(Error::generic("Column name cannot be empty")); + } + if cm_enabled { + // Newlines break metadata serialization regardless of column mapping mode. + if name.contains('\n') { + return Err(Error::generic(format!( + "Column name '{name}' contains a newline character, which is not allowed" + ))); + } + } else if name.contains(INVALID_PARQUET_CHARS) { + let invalid: Vec = name + .chars() + .filter(|c| INVALID_PARQUET_CHARS.contains(c)) + .collect(); + return Err(Error::generic(format!( + "Column name '{name}' contains invalid character(s) {invalid:?} that are not \ + allowed in Parquet column names. \ + Enable column mapping to use special characters in column names." + ))); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + use crate::schema::{ + schema, ArrayType, ColumnMetadataKey, DataType, MapType, MetadataValue, StructField, + StructType, + }; + + // === Schema builders for test cases === + + fn simple_schema() -> StructType { + StructType::new_unchecked(vec![ + StructField::new("id", DataType::INTEGER, false), + StructField::new("name", DataType::STRING, true), + ]) + } + + fn schema_with_underscores() -> StructType { + StructType::new_unchecked(vec![ + StructField::new("col_1", DataType::INTEGER, false), + StructField::new("_private", DataType::STRING, true), + StructField::new("CamelCase123", DataType::LONG, false), + ]) + } + + fn schema_with_special_chars() -> StructType { + StructType::new_unchecked(vec![ + StructField::new("my column", DataType::INTEGER, false), + StructField::new("col;name", DataType::STRING, true), + ]) + } + + fn schema_with_dot() -> StructType { + StructType::new_unchecked(vec![ + StructField::new("a.b", DataType::INTEGER, false), + StructField::new("c", DataType::STRING, true), + ]) + } + + fn schema_different_struct_children() -> StructType { + let inner_a = + StructType::new_unchecked(vec![StructField::new("child", DataType::INTEGER, false)]); + let inner_b = + StructType::new_unchecked(vec![StructField::new("CHILD", DataType::STRING, true)]); + StructType::new_unchecked(vec![ + StructField::new("a", inner_a, false), + StructField::new("b", inner_b, false), + ]) + } + + fn schema_with_space() -> StructType { + schema! { not_null "my column": INTEGER } + } + + fn schema_with_semicolon() -> StructType { + schema! { not_null "col;name": INTEGER } + } + + fn schema_with_newline() -> StructType { + schema! { not_null "col\nname": INTEGER } + } + + fn schema_with_empty_name() -> StructType { + schema! { not_null "": INTEGER } + } + + fn schema_nested_bad_char() -> StructType { + schema! { not_null "parent": { not_null "bad column": INTEGER } } + } + + fn schema_array_bad_char() -> StructType { + schema! { not_null "arr": [ nullable { not_null "bad col": INTEGER } ] } + } + + fn schema_map_bad_char() -> StructType { + schema! { not_null "m": { STRING => nullable { not_null "bad;val": INTEGER } } } + } + + fn schema_top_level_dup() -> StructType { + let inner = + StructType::new_unchecked(vec![StructField::new("x", DataType::INTEGER, false)]); + StructType::new_unchecked(vec![ + StructField::new("a", inner, false), + StructField::new("A", DataType::STRING, true), + ]) + } + + fn schema_array_dup() -> StructType { + let inner = StructType::new_unchecked(vec![ + StructField::new("x", DataType::INTEGER, false), + StructField::new("X", DataType::STRING, true), + ]); + StructType::new_unchecked(vec![StructField::new( + "arr", + ArrayType::new(inner, true), + false, + )]) + } + + fn schema_multi_bad() -> StructType { + StructType::new_unchecked(vec![ + StructField::new("good", DataType::INTEGER, false), + StructField::new("bad column", DataType::STRING, true), + StructField::new("col;name", DataType::LONG, false), + ]) + } + + // === Helpers for building invariants metadata === + // + // These tests assert that `delta.invariants` metadata is rejected at CREATE TABLE. + // When kernel gains SQL expression invariant support (see tracking issue for + // invariant evaluation), these tests should be repurposed to feed a supported + // invariant expression through the full write path instead of being deleted + // outright. + + fn field_with_invariant(name: &str, data_type: DataType, nullable: bool) -> StructField { + let mut field = StructField::new(name, data_type, nullable); + field.metadata.insert( + ColumnMetadataKey::Invariants.as_ref().to_string(), + MetadataValue::String(r#"{"expression": {"expression": "x > 0"}}"#.to_string()), + ); + field + } + + fn schema_top_level_invariant() -> StructType { + StructType::new_unchecked(vec![ + field_with_invariant("x", DataType::INTEGER, true), + StructField::new("y", DataType::INTEGER, true), + ]) + } + + fn schema_nested_invariant() -> StructType { + let inner = + StructType::new_unchecked(vec![field_with_invariant("child", DataType::INTEGER, true)]); + StructType::new_unchecked(vec![StructField::new("parent", inner, true)]) + } + + fn schema_array_nested_invariant() -> StructType { + let inner = + StructType::new_unchecked(vec![field_with_invariant("child", DataType::INTEGER, true)]); + StructType::new_unchecked(vec![StructField::new( + "arr", + ArrayType::new(inner, true), + true, + )]) + } + + fn schema_map_nested_invariant() -> StructType { + let inner = + StructType::new_unchecked(vec![field_with_invariant("child", DataType::INTEGER, true)]); + StructType::new_unchecked(vec![StructField::new( + "map", + MapType::new(DataType::STRING, inner, true), + true, + )]) + } + + // === Valid schemas === + + #[rstest] + #[case::simple(simple_schema(), ColumnMappingMode::None)] + #[case::underscores_digits(schema_with_underscores(), ColumnMappingMode::None)] + #[case::special_chars_with_cm(schema_with_special_chars(), ColumnMappingMode::Name)] + #[case::dot_in_name_with_cm(schema_with_dot(), ColumnMappingMode::Name)] + #[case::different_struct_children(schema_different_struct_children(), ColumnMappingMode::None)] + #[case::empty_no_cm(StructType::new_unchecked(vec![]), ColumnMappingMode::None)] + #[case::empty_cm_name(StructType::new_unchecked(vec![]), ColumnMappingMode::Name)] + #[case::empty_cm_id(StructType::new_unchecked(vec![]), ColumnMappingMode::Id)] + fn valid_schema_accepted(#[case] schema: StructType, #[case] cm: ColumnMappingMode) { + assert!(validate_schema(&schema, cm).is_ok()); + } + + // === Invalid schemas === + + #[rstest] + #[case::space_without_cm(schema_with_space(), ColumnMappingMode::None, &["invalid character"])] + #[case::semicolon_without_cm(schema_with_semicolon(), ColumnMappingMode::None, &["invalid character"])] + #[case::newline_with_cm(schema_with_newline(), ColumnMappingMode::Name, &["newline"])] + #[case::empty_name(schema_with_empty_name(), ColumnMappingMode::None, &["cannot be empty"])] + #[case::nested_struct_bad_char(schema_nested_bad_char(), ColumnMappingMode::None, &["invalid character"])] + #[case::array_nested_bad_char(schema_array_bad_char(), ColumnMappingMode::None, &["invalid character"])] + #[case::map_nested_bad_char(schema_map_bad_char(), ColumnMappingMode::None, &["invalid character"])] + #[case::top_level_dup(schema_top_level_dup(), ColumnMappingMode::None, &["duplicate"])] + #[case::array_element_dup(schema_array_dup(), ColumnMappingMode::None, &["duplicate"])] + #[case::multi_error(schema_multi_bad(), ColumnMappingMode::None, &["bad column", "col;name"])] + fn invalid_schema_rejected( + #[case] schema: StructType, + #[case] cm: ColumnMappingMode, + #[case] expected_errs: &[&str], + ) { + let result = validate_schema(&schema, cm); + assert!(result.is_err()); + let err = result.unwrap_err().to_string(); + for expected in expected_errs { + assert!( + err.contains(expected), + "Expected '{expected}' in error, got: {err}" + ); + } + } + + // === delta.invariants metadata rejection === + + #[rstest] + #[case::top_level(schema_top_level_invariant(), "x")] + #[case::nested_struct(schema_nested_invariant(), "parent.child")] + #[case::array_nested(schema_array_nested_invariant(), "arr.child")] + #[case::map_nested(schema_map_nested_invariant(), "map.child")] + fn invariants_metadata_rejected(#[case] schema: StructType, #[case] expected_path: &str) { + let result = validate_schema(&schema, ColumnMappingMode::None); + let err = result.expect_err("expected delta.invariants metadata rejection"); + let msg = err.to_string(); + assert!( + msg.contains("delta.invariants"), + "Expected delta.invariants mention in error, got: {msg}" + ); + assert!( + msg.contains(expected_path), + "Expected path '{expected_path}' in error, got: {msg}" + ); + } +} diff --git a/kernel/src/schema/variant_utils.rs b/kernel/src/schema/variant_utils.rs index 03a678fbd8..d8465c0239 100644 --- a/kernel/src/schema/variant_utils.rs +++ b/kernel/src/schema/variant_utils.rs @@ -3,27 +3,24 @@ use crate::schema::{Schema, StructType}; use crate::table_configuration::TableConfiguration; use crate::table_features::TableFeature; -use crate::transforms::SchemaTransform; +use crate::transforms::{transform_output_type, SchemaTransform}; use crate::utils::require; use crate::{DeltaResult, Error}; -use std::borrow::Cow; /// Schema visitor that checks if any column in the schema uses VARIANT type -#[derive(Debug, Default)] -pub(crate) struct UsesVariant(bool); +pub(crate) struct UsesVariant; impl<'a> SchemaTransform<'a> for UsesVariant { - fn transform_variant(&mut self, _: &'a StructType) -> Option> { - self.0 = true; - None + transform_output_type!(|'a, T| Result<(), ()>); + + fn transform_variant(&mut self, _: &'a StructType) -> Result<(), ()> { + Err(()) } } /// Checks if any column in the schema (including nested columns) has VARIANT type. pub(crate) fn schema_contains_variant_type(schema: &Schema) -> bool { - let mut visitor = UsesVariant(false); - let _ = visitor.transform_struct(schema); - visitor.0 + UsesVariant.transform_struct(schema).is_err() } pub(crate) fn validate_variant_type_feature_support(tc: &TableConfiguration) -> DeltaResult<()> { @@ -46,9 +43,9 @@ pub(crate) fn validate_variant_type_feature_support(tc: &TableConfiguration) -> #[cfg(test)] mod tests { use crate::actions::Protocol; - use crate::schema::{DataType, StructField, StructType}; + use crate::schema::{schema, DataType, StructField}; use crate::table_features::TableFeature; - use crate::utils::test_utils::{ + use crate::unit_test_utils::{ assert_result_error_with_message, assert_schema_feature_validation, }; @@ -76,26 +73,18 @@ mod tests { #[test] fn test_variant_feature_validation() { - let schema_with = StructType::new_unchecked([ - StructField::new("id", DataType::INTEGER, false), - StructField::new("v", DataType::unshredded_variant(), true), - ]); - let schema_without = StructType::new_unchecked([ - StructField::new("id", DataType::INTEGER, false), - StructField::new("name", DataType::STRING, true), - ]); - let nested_schema_with = StructType::new_unchecked([ - StructField::new("id", DataType::INTEGER, false), - StructField::new( - "nested", - DataType::Struct(Box::new(StructType::new_unchecked([StructField::new( - "inner_v", - DataType::unshredded_variant(), - true, - )]))), - true, - ), - ]); + let schema_with = schema! { + not_null "id": INTEGER, + nullable "v": (DataType::unshredded_variant()), + }; + let schema_without = schema! { + not_null "id": INTEGER, + nullable "name": STRING, + }; + let nested_schema_with = schema! { + not_null "id": INTEGER, + nullable "nested": { nullable "inner_v": (DataType::unshredded_variant()) }, + }; let protocol_without = Protocol::try_new_modern(TableFeature::EMPTY_LIST, TableFeature::EMPTY_LIST).unwrap(); let err_msg = "Table contains VARIANT columns but does not have the required 'variantType' feature in reader and writer features"; diff --git a/kernel/src/schema/void_utils.rs b/kernel/src/schema/void_utils.rs new file mode 100644 index 0000000000..a9b4eb9be7 --- /dev/null +++ b/kernel/src/schema/void_utils.rs @@ -0,0 +1,700 @@ +//! Write-time validation for data types in schemas. +//! +//! The Delta protocol allows void columns in table metadata. Void columns are never written to +//! Parquet files; reads generate null values on the fly for missing void columns. However, certain +//! void placements make data writes impossible and must be rejected at write time: +//! - Void nested inside Array or Map types (not materialized by Delta, and not supported by the +//! logical-to-physical write transform, which only descends through Struct fields) +//! - Structs that contain no non-void fields (would produce an empty Parquet struct) +//! - Tables that contain no non-void columns (would produce an empty Parquet schema) + +use std::borrow::Cow; +use std::sync::Arc; + +use super::{DataType, PrimitiveType, Schema, SchemaRef, StructField, StructType}; +use crate::expressions::ExpressionStructPatchBuilder; +use crate::transforms::{transform_output_type, SchemaTransform}; +use crate::{DeltaResult, Error}; + +/// Returns true when this struct directly contains no non-void fields. The check is local -- +/// it does not recurse into nested structs, because the caller (`ValidateForWrite`) walks every +/// struct in the schema and applies this predicate at each level. Empty structs also qualify +/// (they would produce an unwriteable empty Parquet struct). +/// +/// This predicate is the validator's responsibility, not the stripper's. The stripper could +/// derive a reduced physical schema for an all-void struct, but write semantics require +/// rejecting the schema before that derivation happens. +fn has_no_non_void_fields(st: &StructType) -> bool { + st.fields().all(|f| *f.data_type() == DataType::VOID) +} + +/// Schema visitor that drops void fields at every nesting level. +/// +/// This is intentionally separate from `ValidateForWrite`: validation decides +/// whether a logical write schema is allowed, while this transform derives the +/// physical schema used for Parquet writes, including metadata-only paths. +struct StripVoidFields; + +impl<'a> SchemaTransform<'a> for StripVoidFields { + transform_output_type!(|'a, T| Option>); + + fn transform_primitive(&mut self, ptype: &'a PrimitiveType) -> Option> { + (*ptype != PrimitiveType::Void).then_some(Cow::Borrowed(ptype)) + } +} + +/// Returns `schema` with all void fields removed from structs at every nesting level. +/// In the common case where the schema contains no void fields, returns the same `Arc` +/// without copying. +/// +/// Pairs with `validate_schema_for_write`, which rejects void placements inside Array or Map +/// before the stripped physical schema is used for writing. An all-void or empty input yields +/// an empty schema; write callers are expected to run `validate_schema_for_write` before using +/// the stripped schema. +pub(crate) fn strip_void_from_schema(schema: SchemaRef) -> SchemaRef { + match StripVoidFields.transform_struct(&schema) { + Some(Cow::Owned(stripped)) => Arc::new(stripped), + Some(Cow::Borrowed(_)) => schema, + None => Arc::new(StructType::new_unchecked(Vec::::new())), + } +} + +/// Validates that a schema is suitable for writing data. This is the kernel-internal write-time +/// rejection point for unsupported data types and invalid void placements. Both JSON-deserialized +/// metadata and programmatically constructed schemas rely on this validator before writing files. +/// +/// Writes are rejected when: +/// - Void is nested inside Array or Map. Parquet's UNKNOWN logical type can in principle annotate +/// any physical type with all-null values, but Delta itself does not materialize void columns in +/// data files, and our logical-to-physical transform does not descend into Array elements or Map +/// values to drop them. +/// - A struct contains no non-void fields (would produce an empty Parquet struct) +/// - The table schema contains no non-void columns (would produce an empty Parquet schema) +pub(crate) fn validate_schema_for_write(schema: &Schema) -> DeltaResult<()> { + ValidateForWrite { + container_depth: 0, + depth: 0, + } + .transform_struct(schema) +} + +struct ValidateForWrite { + container_depth: usize, + depth: usize, +} + +impl ValidateForWrite { + fn descend_into_container(&mut self, etype: &DataType, position: &str) -> DeltaResult<()> { + if *etype == DataType::VOID { + return Err(Error::schema(format!( + "Void type is not allowed as {position}" + ))); + } + self.container_depth += 1; + let result = self.transform(etype); + self.container_depth -= 1; + result + } +} + +impl<'a> SchemaTransform<'a> for ValidateForWrite { + transform_output_type!(|'a, T| DeltaResult<()>); + + fn transform_struct(&mut self, stype: &'a StructType) -> DeltaResult<()> { + if has_no_non_void_fields(stype) { + return Err(Error::schema(if self.container_depth > 0 { + "A struct nested in Array or Map must contain at least one non-void field" + } else if self.depth == 0 { + "Table schema must contain at least one non-void column" + } else { + "Cannot write to a table with a struct that contains no non-void fields" + })); + } + self.depth += 1; + let result = self.recurse_into_struct(stype); + self.depth -= 1; + result + } + + fn transform_struct_field(&mut self, field: &'a StructField) -> DeltaResult<()> { + // Reject void inside a struct nested in Array or Map. `StripVoidFields` can drop + // the field from the physical schema, but the logical-to-physical write transform + // built by `add_void_stripping_inner` descends only through struct fields. Allowing + // this case would leave the runtime expression passing the void field through while + // the physical schema no longer expects it, putting the two out of sync at write + // time. Lifting this restriction requires extending the runtime transform to descend + // into Array elements and Map keys/values. + if self.container_depth > 0 && *field.data_type() == DataType::VOID { + return Err(Error::schema( + "Void type is not allowed inside a struct nested in Array or Map", + )); + } + self.recurse_into_struct_field(field) + } + + fn transform_array_element(&mut self, etype: &'a DataType) -> DeltaResult<()> { + self.descend_into_container(etype, "an array element type") + } + + fn transform_map_key(&mut self, etype: &'a DataType) -> DeltaResult<()> { + self.descend_into_container(etype, "a map key type") + } + + fn transform_map_value(&mut self, etype: &'a DataType) -> DeltaResult<()> { + self.descend_into_container(etype, "a map value type") + } +} + +/// Appends void-field-stripping operations to `patch`, recursing through struct fields in +/// `st` (the logical schema). Returns `patch` unchanged when `st` contains no void fields. +/// +/// Composition: `st` is read directly to locate void fields, not the partial result of applying +/// `patch`. Composing `patch` first with operations on disjoint columns (e.g. partition +/// column drops) is therefore safe, but composing with operations that already drop or replace +/// the same void-named columns is not -- the resulting patch would double-drop or conflict. +/// Don't reorder this relative to such operations. +pub(crate) fn add_void_stripping( + patch: ExpressionStructPatchBuilder, + st: &StructType, +) -> ExpressionStructPatchBuilder { + add_void_stripping_inner(patch, st, &mut Vec::new()) +} + +/// Recursive helper that records a drop for every void field in `st`, threading `path` (the field +/// names from the root struct down to `st`) so nested drops are recorded at their full path. +/// +/// The builder lowers each `drop_at` call into the appropriate nested patch and only materializes a +/// nested patch when a drop actually lands under that path, so structs with no void fields +/// contribute nothing. `path` is pushed/popped in place to avoid reallocating at each level. +/// +/// This intentionally descends only through Struct fields; write validation rejects void +/// placements inside Array or Map before this patch is used. +fn add_void_stripping_inner<'a>( + mut patch: ExpressionStructPatchBuilder, + st: &'a StructType, + path: &mut Vec<&'a str>, +) -> ExpressionStructPatchBuilder { + for field in st.fields() { + if *field.data_type() == DataType::VOID { + patch = patch.drop_at(path.iter().copied(), field.name()); + } else if let DataType::Struct(inner) = field.data_type() { + path.push(field.name()); + patch = add_void_stripping_inner(patch, inner, path); + path.pop(); + } + } + patch +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::schema::{ + ArrayType, ColumnMetadataKey, DataType, MapType, MetadataValue, StructField, StructType, + }; + + // ---- validate_schema_for_write tests ---- + + #[test] + fn test_validator_catches_void_in_map_from_json() { + let json = r#"{ + "name": "m", + "type": { + "type": "map", + "keyType": "string", + "valueType": "void", + "valueContainsNull": true + }, + "nullable": true, + "metadata": {} + }"#; + + // Deserialization succeeds — serde populates fields directly + let field: StructField = serde_json::from_str(json).unwrap(); + if let DataType::Map(map_type) = field.data_type() { + assert_eq!(*map_type.value_type(), DataType::VOID); + } else { + panic!("expected map type"); + } + + // The dedicated validator is what actually catches this + let schema = StructType::new_unchecked([field]); + assert!(validate_schema_for_write(&schema).is_err()); + } + + #[rstest::rstest] + #[case( + "void in array", + StructField::nullable("f", ArrayType::new(DataType::VOID, true)), + "array element type" + )] + #[case( + "void in map value", + StructField::nullable("f", MapType::new(DataType::STRING, DataType::VOID, true)), + "map value type" + )] + #[case( + "void in map key", + StructField::nullable("f", MapType::new(DataType::VOID, DataType::STRING, true)), + "map key type" + )] + #[case( + "void in array inside struct", + StructField::nullable( + "outer", + StructType::new_unchecked([ + StructField::nullable("inner", ArrayType::new(DataType::VOID, true)), + ]) + ), + "array element type" + )] + #[case( + "void in map inside array", + StructField::nullable( + "col", + ArrayType::new(MapType::new(DataType::STRING, DataType::VOID, true), true,), + ), + "map value type" + )] + #[case( + "void inside struct nested in array", + StructField::nullable( + "arr", + ArrayType::new( + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::VOID), + ]), + true, + ), + ), + "Void type is not allowed inside" + )] + #[case( + "void inside struct nested in map value", + StructField::nullable( + "m", + MapType::new( + DataType::STRING, + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::VOID), + ]), + true, + ), + ), + "Void type is not allowed inside" + )] + #[case( + "void inside struct nested in map key", + StructField::nullable( + "m", + MapType::new( + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::VOID), + ]), + DataType::STRING, + true, + ), + ), + "Void type is not allowed inside" + )] + #[case( + "void inside struct nested in array inside array", + StructField::nullable( + "outer", + ArrayType::new( + ArrayType::new( + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::VOID), + ]), + true, + ), + true, + ), + ), + "Void type is not allowed inside" + )] + #[case( + "void in deeply nested struct inside array", + StructField::nullable( + "arr", + ArrayType::new( + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable( + "b", + StructType::new_unchecked([ + StructField::nullable("x", DataType::INTEGER), + StructField::nullable("y", DataType::VOID), + ]), + ), + ]), + true, + ), + ), + "Void type is not allowed inside" + )] + #[case( + "void in struct inside array inside struct inside array", + StructField::nullable( + "outer", + ArrayType::new( + StructType::new_unchecked([StructField::nullable( + "inner", + ArrayType::new( + StructType::new_unchecked([StructField::nullable( + "v", + DataType::VOID, + )]), + true, + ), + )]), + true, + ), + ), + "must contain at least one non-void field" + )] + #[case( + "empty struct nested in array", + StructField::nullable( + "arr", + ArrayType::new( + StructType::new_unchecked(Vec::::new()), + true, + ), + ), + "struct nested in Array or Map must contain at least one non-void field" + )] + #[case( + "all-void struct nested in map value", + StructField::nullable( + "m", + MapType::new( + DataType::STRING, + StructType::new_unchecked([StructField::nullable( + "x", + DataType::VOID, + )]), + true, + ), + ), + "struct nested in Array or Map must contain at least one non-void field" + )] + fn test_void_in_complex_type_rejected( + #[case] desc: &str, + #[case] field: StructField, + #[case] expected_msg: &str, + ) { + let schema = StructType::new_unchecked([field]); + let result = validate_schema_for_write(&schema); + assert!( + result.unwrap_err().to_string().contains(expected_msg), + "{desc}: expected error containing '{expected_msg}'" + ); + } + + #[rstest::rstest] + #[case( + "void top level ok", + StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("void_col", DataType::VOID), + ]) + )] + #[case( + "test no void ok", + StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("name", DataType::STRING), + ]) + )] + #[case( + "void in nested struct", + StructType::new_unchecked([StructField::nullable( + "s", + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::VOID), + ]), + )]) + )] + #[case( + "array of struct without void", + StructType::new_unchecked([StructField::nullable( + "arr", + ArrayType::new( + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::STRING), + ]), + true, + ), + )]) + )] + #[case( + "map of struct without void", + StructType::new_unchecked([StructField::nullable( + "m", + MapType::new( + DataType::STRING, + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::STRING), + ]), + true, + ), + )]) + )] + fn test_valid_schema_for_complex_types(#[case] desc: &str, #[case] schema: StructType) { + validate_schema_for_write(&schema) + .unwrap_or_else(|e| panic!("{desc}: unexpected validation error: {e}")); + } + + // ---- validate_schema_for_write tests ---- + + #[rstest::rstest] + #[case( + "with void column", + StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("void_col", DataType::VOID), + ]) + )] + #[case( + "no void", + StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("name", DataType::STRING), + ]) + )] + #[case( + "struct with mixed void", + StructType::new_unchecked([StructField::nullable( + "s", + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::VOID), + ]), + )]) + )] + fn test_write_valid_schemas(#[case] desc: &str, #[case] schema: StructType) { + validate_schema_for_write(&schema) + .unwrap_or_else(|e| panic!("{desc}: unexpected validation error: {e}")); + } + + #[rstest::rstest] + #[case( + "all void table", + StructType::new_unchecked([ + StructField::nullable("a", DataType::VOID), + StructField::nullable("b", DataType::VOID), + ]), + "at least one non-void column" + )] + #[case( + "all void struct", + StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable( + "s", + StructType::new_unchecked([ + StructField::nullable("x", DataType::VOID), + StructField::nullable("y", DataType::VOID), + ]), + ), + ]), + "contains no non-void fields" + )] + #[case( + "void in array", + StructType::new_unchecked([StructField::nullable( + "arr", + ArrayType::new(DataType::VOID, true), + )]), + "array element type" + )] + #[case( + "void in map", + StructType::new_unchecked([StructField::nullable( + "m", + MapType::new( + DataType::STRING, + DataType::VOID, + true, + ), + )]), + "map value type" + )] + #[case( + "nested all void struct", + StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable( + "outer", + StructType::new_unchecked([ + StructField::nullable( + "inner", + StructType::new_unchecked([StructField::nullable("x", DataType::VOID)]), + ), + ]), + ), + ]), + "contains no non-void fields" + )] + #[case( + "empty struct at top level", + StructType::new_unchecked(Vec::::new()), + "at least one non-void column" + )] + #[case( + "nested empty struct", + StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable( + "s", + StructType::new_unchecked( + Vec::::new(), + ), + ), + ]), + "contains no non-void fields" + )] + fn test_write_rejected_schemas( + #[case] desc: &str, + #[case] schema: StructType, + #[case] expected_msg: &str, + ) { + let result = validate_schema_for_write(&schema); + assert!( + result.unwrap_err().to_string().contains(expected_msg), + "{desc}: expected error containing '{expected_msg}'" + ); + } + + // ---- strip_void_from_schema tests ---- + + #[rstest::rstest] + #[case( + "schema with no void is noop", + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::STRING), + ]), + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::STRING), + ]) + )] + #[case( + "top-level void is dropped", + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("v", DataType::VOID), + StructField::nullable("b", DataType::STRING), + ]), + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::STRING), + ]) + )] + #[case( + "nested struct with mixed void", + StructType::new_unchecked([StructField::nullable( + "s", + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::VOID), + StructField::nullable("c", DataType::STRING), + ]), + )]), + StructType::new_unchecked([StructField::nullable( + "s", + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("c", DataType::STRING), + ]), + )]) + )] + #[case( + "deeply nested void", + StructType::new_unchecked([StructField::nullable( + "outer", + StructType::new_unchecked([StructField::nullable( + "inner", + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("v", DataType::VOID), + ]), + )]), + )]), + StructType::new_unchecked([StructField::nullable( + "outer", + StructType::new_unchecked([StructField::nullable( + "inner", + StructType::new_unchecked([StructField::nullable("a", DataType::INTEGER)]), + )]), + )]) + )] + fn test_strip_void_from_schema( + #[case] desc: &str, + #[case] input: StructType, + #[case] expected: StructType, + ) { + let stripped = strip_void_from_schema(Arc::new(input)); + assert_eq!(*stripped, expected, "{desc}"); + } + + // A container that has Void as its only "interior" type collapses when the void + // primitive is filtered: ArrayType / MapType cannot be reconstructed without their + // element / key / value, so the containing field disappears. + #[rstest::rstest] + #[case::array_of_void(DataType::from(ArrayType::new(DataType::VOID, true)))] + #[case::map_with_void_value(DataType::from(MapType::new( + DataType::STRING, + DataType::VOID, + true + )))] + #[case::map_with_void_key(DataType::from(MapType::new( + DataType::VOID, + DataType::STRING, + true + )))] + fn test_strip_drops_container_with_void(#[case] field_type: DataType) { + let schema = Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("c", field_type), + ])); + let stripped = strip_void_from_schema(schema); + assert!(stripped.field("id").is_some()); + assert!(stripped.field("c").is_none()); + } + + #[test] + fn test_strip_preserves_metadata() { + let mut s_field = StructField::nullable( + "s", + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::VOID), + ]), + ); + s_field.metadata.insert( + ColumnMetadataKey::ColumnMappingPhysicalName.as_ref().into(), + MetadataValue::String("phys_s".into()), + ); + let schema = Arc::new(StructType::new_unchecked([s_field])); + let stripped = strip_void_from_schema(schema); + assert_eq!( + stripped + .field("s") + .expect("s field present after strip") + .metadata + .get(ColumnMetadataKey::ColumnMappingPhysicalName.as_ref()), + Some(&MetadataValue::String("phys_s".into())), + "metadata should be preserved after stripping" + ); + } +} diff --git a/kernel/src/snapshot.rs b/kernel/src/snapshot.rs deleted file mode 100644 index 51c69cdad7..0000000000 --- a/kernel/src/snapshot.rs +++ /dev/null @@ -1,2422 +0,0 @@ -//! In-memory representation of snapshots of tables (snapshot is a table at given point in time, it -//! has schema etc.) - -use std::collections::HashSet; -use std::sync::Arc; -use std::time::Instant; - -use delta_kernel_derive::internal_api; -use tracing::{debug, info, instrument}; -use url::Url; - -use crate::action_reconciliation::calculate_transaction_expiration_timestamp; -use crate::actions::set_transaction::SetTransactionScanner; -use crate::actions::{DomainMetadata, INTERNAL_DOMAIN_PREFIX}; -use crate::checkpoint::CheckpointWriter; -use crate::clustering::{parse_clustering_columns, CLUSTERING_DOMAIN_NAME}; -use crate::committer::{Committer, PublishMetadata}; -#[cfg(any(test, feature = "test-utils"))] -use crate::crc::Crc; -use crate::crc::{try_write_crc_file, CrcDelta, LazyCrc}; -use crate::expressions::ColumnName; -use crate::log_segment::{DomainMetadataMap, LogSegment}; -use crate::log_segment_files::LogSegmentFiles; -use crate::metrics::{MetricEvent, MetricId}; -use crate::path::ParsedLogPath; -use crate::scan::ScanBuilder; -use crate::schema::SchemaRef; -use crate::table_configuration::{InCommitTimestampEnablement, TableConfiguration}; -use crate::table_features::TableFeature; -use crate::table_properties::TableProperties; -use crate::transaction::Transaction; -use crate::utils::require; -use crate::LogCompactionWriter; -use crate::{DeltaResult, Engine, Error, Version}; - -mod builder; -pub use builder::SnapshotBuilder; - -pub type SnapshotRef = Arc; - -/// File-level statistics for a table version. -/// -/// NOTE: This is an unstable API expected to change in future releases. -#[allow(unused)] -#[internal_api] -pub(crate) type FileStats = crate::crc::FileStats; - -/// Result of attempting to write a version checksum (CRC) file. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum ChecksumWriteResult { - /// A CRC file already exists at this version. Per the Delta protocol, writers MUST NOT - /// overwrite existing version checksum files. - AlreadyExists, - /// The CRC file was successfully written to storage. - Written, -} - -// TODO expose methods for accessing the files of a table (with file pruning). -/// In-memory representation of a specific snapshot of a Delta table. While a `DeltaTable` exists -/// throughout time, `Snapshot`s represent a view of a table at a specific point in time; they -/// have a defined schema (which may change over time for any given table), specific version, and -/// frozen log segment. -pub struct Snapshot { - span: tracing::Span, - log_segment: LogSegment, - table_configuration: TableConfiguration, - lazy_crc: Arc, -} - -impl PartialEq for Snapshot { - fn eq(&self, other: &Self) -> bool { - self.log_segment == other.log_segment - && self.table_configuration == other.table_configuration - } -} - -impl Eq for Snapshot {} - -impl Drop for Snapshot { - fn drop(&mut self) { - debug!("Dropping snapshot"); - } -} - -impl std::fmt::Debug for Snapshot { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - f.debug_struct("Snapshot") - .field("path", &self.log_segment.log_root.as_str()) - .field("version", &self.version()) - .field("metadata", &self.table_configuration().metadata()) - .field("log_segment", &self.log_segment) - .finish() - } -} - -impl Snapshot { - /// Create a new [`SnapshotBuilder`] to build a new [`Snapshot`] for a given table root. If you - /// instead have an existing [`Snapshot`] you would like to do minimal work to update, consider - /// using - pub fn builder_for(table_root: impl AsRef) -> SnapshotBuilder { - SnapshotBuilder::new_for(table_root) - } - - /// Create a new [`SnapshotBuilder`] to incrementally update a [`Snapshot`] to a more recent - /// version. - /// - /// We implement a simple heuristic: - /// 1. if the new version == existing version, just return the existing snapshot - /// 2. if the new version < existing version, error: there is no optimization to do here - /// 3. list from (existing checkpoint version + 1) onward (or just existing snapshot version if - /// no checkpoint) - /// 4. a. if new checkpoint is found: just create a new snapshot from that checkpoint (and - /// commits after it) - /// b. if no new checkpoint is found: do lightweight P+M replay on the latest commits (after - /// ensuring we only retain commits > any checkpoints) - /// - /// # Parameters - /// - /// - `existing_snapshot`: reference to an existing [`Snapshot`] - /// - `engine`: Implementation of [`Engine`] apis. - /// - `version`: target version of the [`Snapshot`]. None will create a snapshot at the latest - /// version of the table. - pub fn builder_from(existing_snapshot: SnapshotRef) -> SnapshotBuilder { - SnapshotBuilder::new_from(existing_snapshot) - } - - /// Create a new [`Snapshot`] from a [`LogSegment`] and [`TableConfiguration`]. - #[internal_api] - pub(crate) fn new(log_segment: LogSegment, table_configuration: TableConfiguration) -> Self { - Self::new_with_crc( - log_segment, - table_configuration, - Arc::new(LazyCrc::new(None)), - ) - } - - /// Internal constructor that accepts an explicit [`LazyCrc`]. - pub(crate) fn new_with_crc( - log_segment: LogSegment, - table_configuration: TableConfiguration, - lazy_crc: Arc, - ) -> Self { - let span = tracing::info_span!( - parent: tracing::Span::none(), - "snap", - path = %table_configuration.table_root(), - version = table_configuration.version(), - ); - info!(parent: &span, "Created snapshot"); - Self { - span, - log_segment, - table_configuration, - lazy_crc, - } - } - - /// Implementation of snapshot creation from existing snapshot. - fn try_new_from_impl( - existing_snapshot: Arc, - log_tail: Vec, - engine: &dyn Engine, - version: impl Into>, - operation_id: MetricId, - ) -> DeltaResult> { - let old_log_segment = &existing_snapshot.log_segment; - let old_version = existing_snapshot.version(); - let new_version = version.into(); - if let Some(new_version) = new_version { - if new_version == old_version { - // Re-requesting the same version - return Ok(existing_snapshot.clone()); - } - if new_version < old_version { - // Hint is too new: error since this is effectively an incorrect optimization - return Err(Error::Generic(format!( - "Requested snapshot version {new_version} is older than snapshot hint version {old_version}" - ))); - } - } - - let log_root = old_log_segment.log_root.clone(); - let storage = engine.storage_handler(); - - // Start listing just after the previous segment's checkpoint, if any - let listing_start = old_log_segment.checkpoint_version.unwrap_or(0) + 1; - - // Check for new commits (and CRC) - let new_listed_files = LogSegmentFiles::list( - storage.as_ref(), - &log_root, - log_tail, - Some(listing_start), - new_version, - )?; - - // NB: we need to check both checkpoints and commits since we filter commits at and below - // the checkpoint version. Example: if we have a checkpoint + commit at version 1, the log - // listing above will only return the checkpoint and not the commit. - if new_listed_files.ascending_commit_files().is_empty() - && new_listed_files.checkpoint_parts().is_empty() - { - match new_version { - Some(new_version) if new_version != old_version => { - // No new commits, but we are looking for a new version - return Err(Error::Generic(format!( - "Requested snapshot version {new_version} is newer than the latest version {old_version}" - ))); - } - _ => { - // No new commits, just return the same snapshot - return Ok(existing_snapshot.clone()); - } - } - } - - // create a log segment just from existing_checkpoint.version -> new_version - // OR could be from 1 -> new_version - // Save the latest_commit before moving new_listed_files - let new_latest_commit_file = new_listed_files.latest_commit_file().clone(); - // Note: new_log_segment won't have checkpoint_schema since we're listing without a hint. - // If it has a checkpoint, we use it as-is. Otherwise, we preserve the old checkpoint_schema. - let mut new_log_segment = - LogSegment::try_new(new_listed_files, log_root.clone(), new_version, None)?; - - let new_end_version = new_log_segment.end_version; - if new_end_version < old_version { - // we should never see a new log segment with a version < the existing snapshot - // version, that would mean a commit was incorrectly deleted from the log - return Err(Error::Generic(format!( - "Unexpected state: The newest version in the log {new_end_version} is older than the old version {old_version}"))); - } - if new_end_version == old_version { - // No new commits, just return the same snapshot - return Ok(existing_snapshot.clone()); - } - - if new_log_segment.checkpoint_version.is_some() { - // we have a checkpoint in the new LogSegment, just construct a new snapshot from that - let snapshot = Self::try_new_from_log_segment_impl( - existing_snapshot.table_root().clone(), - new_log_segment, - engine, - operation_id, - ); - return Ok(Arc::new(snapshot?)); - } - - // after this point, we incrementally update the snapshot with the new log segment. - // first we remove the 'overlap' in commits, example: - // - // old logsegment checkpoint1-commit1-commit2-commit3 - // 1. new logsegment commit1-commit2-commit3 - // 2. new logsegment commit1-commit2-commit3-commit4 - // 3. new logsegment checkpoint2+commit2-commit3-commit4 - // - // retain does - // 1. new logsegment [empty] -> caught above - // 2. new logsegment [commit4] - // 3. new logsegment [checkpoint2-commit3] -> caught above - new_log_segment - .listed - .ascending_commit_files - .retain(|log_path| old_version < log_path.version); - // Deduplicate compaction files the same way: the new listing re-lists from - // checkpoint_version, so it includes compaction files already in the old segment. - // Note: This removes all _new_ compaction files that start at or before `old_version`, - // which may drop useful compaction files that span across the old/new boundary - // (e.g. a new compaction(1, 3) when old_version=2). This is conservative but safe. - new_log_segment - .listed - .ascending_compaction_files - .retain(|log_path| old_version < log_path.version); - - // we have new commits and no new checkpoint: we replay new commits for P+M and then - // create a new snapshot by combining LogSegments and building a new TableConfiguration - let lazy_crc = Arc::new(LazyCrc::new(new_log_segment.listed.latest_crc_file.clone())); - let (new_metadata, new_protocol) = - new_log_segment.read_protocol_metadata_opt(engine, &lazy_crc)?; - let table_configuration = TableConfiguration::try_new_from( - existing_snapshot.table_configuration(), - new_metadata, - new_protocol, - new_log_segment.end_version, - )?; - - // NB: we must add the new log segment to the existing snapshot's log segment - let mut ascending_commit_files = old_log_segment.listed.ascending_commit_files.clone(); - ascending_commit_files.extend(new_log_segment.listed.ascending_commit_files); - let mut ascending_compaction_files = - old_log_segment.listed.ascending_compaction_files.clone(); - ascending_compaction_files.extend(new_log_segment.listed.ascending_compaction_files); - - // Note that we _could_ go backwards if someone deletes a CRC: - // old listing: 1, 2, 2.crc, 3, 3.crc (latest is 3.crc) - // new listing: 1, 2, 2.crc, 3 (latest is 2.crc) - // and we would still pick the new listing's (older) CRC file since it ostensibly still - // exists - let latest_crc_file = new_log_segment - .listed - .latest_crc_file - .or_else(|| old_log_segment.listed.latest_crc_file.clone()); - - // Use the new latest_commit if available, otherwise use the old one - // This handles the case where the new listing returned no commits - let latest_commit_file = - new_latest_commit_file.or_else(|| old_log_segment.listed.latest_commit_file.clone()); - // we can pass in just the old checkpoint parts since by the time we reach this line, we - // know there are no checkpoints in the new log segment. - let combined_log_segment = LogSegment::try_new( - LogSegmentFiles { - ascending_commit_files, - ascending_compaction_files, - checkpoint_parts: old_log_segment.listed.checkpoint_parts.clone(), - latest_crc_file, - latest_commit_file, - max_published_version: new_log_segment - .listed - .max_published_version - .max(old_log_segment.listed.max_published_version), - }, - log_root, - new_version, - // Preserve checkpoint schema from old segment - old_log_segment.checkpoint_schema.clone(), - )?; - - let lazy_crc = Arc::new(LazyCrc::new( - combined_log_segment.listed.latest_crc_file.clone(), - )); - Ok(Arc::new(Snapshot::new_with_crc( - combined_log_segment, - table_configuration, - lazy_crc, - ))) - } - - /// Create a new [`Snapshot`] instance from an existing [`Snapshot`]. This is useful when you - /// already have a [`Snapshot`] lying around and want to do the minimal work to 'update' the - /// snapshot to a later version. - /// - /// Reports metrics: `SnapshotCompleted` or `SnapshotFailed`. - fn try_new_from( - existing_snapshot: Arc, - log_tail: Vec, - engine: &dyn Engine, - version: impl Into>, - operation_id: Option, - ) -> DeltaResult> { - let operation_id = operation_id.unwrap_or_default(); - let reporter = engine.get_metrics_reporter(); - - let start = Instant::now(); - let result = - Self::try_new_from_impl(existing_snapshot, log_tail, engine, version, operation_id); - let snapshot_duration = start.elapsed(); - - match result { - Ok(snapshot) => { - reporter.as_ref().inspect(|r| { - r.report(MetricEvent::SnapshotCompleted { - operation_id, - version: snapshot.version(), - total_duration: snapshot_duration, - }); - }); - Ok(snapshot) - } - Err(e) => { - reporter.as_ref().inspect(|r| { - r.report(MetricEvent::SnapshotFailed { - operation_id, - duration: snapshot_duration, - }); - }); - Err(e) - } - } - } - - /// Implementation of snapshot creation from log segment. - /// - /// Reports metrics: `ProtocolMetadataLoaded`. - fn try_new_from_log_segment_impl( - location: Url, - log_segment: LogSegment, - engine: &dyn Engine, - operation_id: MetricId, - ) -> DeltaResult { - let reporter = engine.get_metrics_reporter(); - - // Create lazy CRC loader for P&M optimization - let lazy_crc = Arc::new(LazyCrc::new(log_segment.listed.latest_crc_file.clone())); - - // Read protocol and metadata (may use CRC if available) - let start = Instant::now(); - let (metadata, protocol) = log_segment.read_protocol_metadata(engine, &lazy_crc)?; - let read_metadata_duration = start.elapsed(); - - reporter.as_ref().inspect(|r| { - r.report(MetricEvent::ProtocolMetadataLoaded { - operation_id, - duration: read_metadata_duration, - }); - }); - - let table_configuration = - TableConfiguration::try_new(metadata, protocol, location, log_segment.end_version)?; - - Ok(Self::new_with_crc( - log_segment, - table_configuration, - lazy_crc, - )) - } - - /// Create a new [`Snapshot`] instance. - /// - /// Reports metrics: `SnapshotCompleted` or `SnapshotFailed`. - pub(crate) fn try_new_from_log_segment( - location: Url, - log_segment: LogSegment, - engine: &dyn Engine, - operation_id: Option, - ) -> DeltaResult { - let operation_id = operation_id.unwrap_or_default(); - let reporter = engine.get_metrics_reporter(); - - let start = Instant::now(); - let result = - Self::try_new_from_log_segment_impl(location, log_segment, engine, operation_id); - let snapshot_duration = start.elapsed(); - - match result { - Ok(snapshot) => { - reporter.as_ref().inspect(|r| { - r.report(MetricEvent::SnapshotCompleted { - operation_id, - version: snapshot.table_configuration.version(), - total_duration: snapshot_duration, - }); - }); - Ok(snapshot) - } - Err(e) => { - reporter.as_ref().inspect(|r| { - r.report(MetricEvent::SnapshotFailed { - operation_id, - duration: snapshot_duration, - }); - }); - Err(e) - } - } - } - - /// Creates a new [`Snapshot`] representing the table state immediately after a commit. - /// - /// Appends the newly committed file to this snapshot's log segment and bumps the version, - /// producing a post-commit snapshot without a full log replay from storage. - /// - /// The `crc_delta` captures the CRC-relevant changes from the committed transaction - /// (file stats, domain metadata, ICT, etc.). If the pre-commit snapshot had a loaded CRC - /// at its version, the delta is applied to produce a precomputed in-memory CRC for the new - /// version -- this CRC contains all important table metadata (protocol, metadata, domain - /// metadata, set transactions, ICT) and avoids re-reading them from storage. CREATE TABLE - /// always produces a CRC at v0. If no CRC was available on the pre-commit snapshot, the - /// existing lazy CRC is carried forward unchanged. - /// - /// TODO: Handle Protocol changes in CrcDelta (when Kernel-RS supports protocol changes) - /// TODO: Handle Metadata changes in CrcDelta (when Kernel-RS supports metadata changes) - pub(crate) fn new_post_commit( - &self, - commit: ParsedLogPath, - crc_delta: CrcDelta, - ) -> DeltaResult { - require!( - commit.is_commit(), - Error::internal_error(format!( - "Cannot create post-commit Snapshot. Log file is not a commit file. \ - Path: {}, Type: {:?}.", - commit.location.location, commit.file_type - )) - ); - let read_version = self.version(); - let new_version = commit.version; - require!( - new_version == read_version.wrapping_add(1), - Error::internal_error(format!( - "Cannot create post-commit Snapshot. Log file version ({new_version}) does not \ - equal Snapshot version ({read_version}) + 1." - )) - ); - - let new_table_configuration = - TableConfiguration::new_post_commit(self.table_configuration(), new_version); - - let new_log_segment = self.log_segment.new_with_commit_appended(commit)?; - - let new_lazy_crc = self.compute_post_commit_crc(new_version, crc_delta); - - Ok(Snapshot::new_with_crc( - new_log_segment, - new_table_configuration, - new_lazy_crc, - )) - } - - /// Compute the lazy CRC for a post-commit snapshot by applying a [`CrcDelta`]. - /// - /// For CREATE TABLE, builds a fresh CRC from the `crc_delta`. For existing tables, applies - /// the `crc_delta` to the current CRC if loaded, otherwise carries forward the existing lazy CRC. - fn compute_post_commit_crc(&self, new_version: Version, crc_delta: CrcDelta) -> Arc { - let crc = if self.version() == crate::PRE_COMMIT_VERSION { - crc_delta.into_crc_for_version_zero() - } else { - self.lazy_crc - .get_if_loaded_at_version(self.version()) - .map(|base| { - let mut crc = base.as_ref().clone(); - crc.apply(crc_delta); - crc - }) - }; - - match crc { - Some(c) => Arc::new(LazyCrc::new_precomputed(c, new_version)), - None => self.lazy_crc.clone(), - } - } - - /// Creates a [`CheckpointWriter`] for generating a checkpoint from this snapshot. - /// - /// See the [`crate::checkpoint`] module documentation for more details on checkpoint types - /// and the overall checkpoint process. - pub fn create_checkpoint_writer(self: Arc) -> DeltaResult { - CheckpointWriter::try_new(self) - } - - /// Performs a complete checkpoint of this snapshot using the provided engine. - /// - /// Writes a checkpoint parquet file and the `_last_checkpoint` file. - /// - /// Note: This function uses [`crate::ParquetHandler::write_parquet_file`] and - /// [`crate::StorageHandler::head`], which may not be implemented by all engines - /// (e.g., `SyncEngine`). - /// - /// If you are using the default engine, make sure to build it with the multi-threaded executor if you want to use this method. - #[instrument(parent = &self.span, name = "snap.checkpoint", skip_all, err)] - pub fn checkpoint(self: Arc, engine: &dyn Engine) -> DeltaResult<()> { - let writer = self.create_checkpoint_writer()?; - let checkpoint_path = writer.checkpoint_path()?; - let data_iter = writer.checkpoint_data(engine)?; - let state = data_iter.state(); - let lazy_data = data_iter.map(|r| r.and_then(|f| f.apply_selection_vector())); - engine - .parquet_handler() - .write_parquet_file(checkpoint_path.clone(), Box::new(lazy_data))?; - - let file_meta = engine.storage_handler().head(&checkpoint_path)?; - - // Finalize the checkpoint (writes `_last_checkpoint` file). - writer.finalize(engine, &file_meta, &state) - } - - /// Creates a [`LogCompactionWriter`] for generating a log compaction file. - /// - /// Log compaction aggregates commit files in a version range into a single compacted file, - /// improving performance by reducing the number of files to process during log replay. - /// - /// # Parameters - /// - `start_version`: The first version to include in the compaction (inclusive) - /// - `end_version`: The last version to include in the compaction (inclusive) - /// - /// # Returns - /// A [`LogCompactionWriter`] that can be used to generate the compaction file. - pub fn log_compaction_writer( - self: Arc, - start_version: Version, - end_version: Version, - ) -> DeltaResult { - LogCompactionWriter::try_new(self, start_version, end_version) - } - - /// Log segment this snapshot uses - #[internal_api] - pub(crate) fn log_segment(&self) -> &LogSegment { - &self.log_segment - } - - pub fn table_root(&self) -> &Url { - self.table_configuration.table_root() - } - - /// Version of this `Snapshot` in the table. - pub fn version(&self) -> Version { - self.table_configuration().version() - } - - /// Table [`Schema`] at this `Snapshot`s version. - /// - /// [`Schema`]: crate::schema::Schema - pub fn schema(&self) -> SchemaRef { - self.table_configuration.logical_schema() - } - - /// Get the [`TableProperties`] for this [`Snapshot`]. - pub fn table_properties(&self) -> &TableProperties { - self.table_configuration().table_properties() - } - - /// Get the [`TableConfiguration`] for this [`Snapshot`]. - #[internal_api] - pub(crate) fn table_configuration(&self) -> &TableConfiguration { - &self.table_configuration - } - - /// Create a [`ScanBuilder`] for an `SnapshotRef`. - pub fn scan_builder(self: Arc) -> ScanBuilder { - ScanBuilder::new(self) - } - - /// Create a [`Transaction`] for this `SnapshotRef`. With the specified [`Committer`]. - /// - /// Note: For tables with clustering enabled, this performs log replay to read clustering - /// columns from domain metadata, which may have a performance cost. - pub fn transaction( - self: Arc, - committer: Box, - engine: &dyn Engine, - ) -> DeltaResult { - Transaction::try_new_existing_table(self, committer, engine) - } - - /// Fetch the latest version of the provided `application_id` for this snapshot. Filters the txn based on the SetTransactionRetentionDuration property and lastUpdated - /// - /// Note that this method performs log replay (fetches and processes metadata from storage). - // TODO: add a get_app_id_versions to fetch all at once using SetTransactionScanner::get_all - #[instrument(parent = &self.span, name = "snap.get_app_id_version", skip_all, err)] - pub fn get_app_id_version( - &self, - application_id: &str, - engine: &dyn Engine, - ) -> DeltaResult> { - let expiration_timestamp = - calculate_transaction_expiration_timestamp(self.table_properties())?; - let txn = SetTransactionScanner::get_one( - self.log_segment(), - application_id, - engine, - expiration_timestamp, - )?; - Ok(txn.map(|t| t.version)) - } - - /// Fetch the domainMetadata for a specific domain in this snapshot. This returns the latest - /// configuration for the domain, or None if the domain does not exist. - /// - /// Note that this method performs log replay (fetches and processes metadata from storage). - pub fn get_domain_metadata( - &self, - domain: &str, - engine: &dyn Engine, - ) -> DeltaResult> { - if domain.starts_with(INTERNAL_DOMAIN_PREFIX) { - return Err(Error::generic( - "User DomainMetadata are not allowed to use system-controlled 'delta.*' domain", - )); - } - - self.get_domain_metadata_internal(domain, engine) - } - - /// Get the clustering columns for this snapshot, if the table has clustering enabled. - /// - /// Returns `Ok(Some(columns))` if the ClusteredTable feature is enabled and clustering - /// columns are defined, `Ok(None)` if clustering is not enabled, or an error if the - /// clustering metadata is malformed. - /// - /// The columns are returned as physical column names, respecting the column mapping mode. - /// Note that this method performs log replay (fetches and processes metadata from storage). - #[internal_api] - pub(crate) fn get_clustering_columns_physical( - &self, - engine: &dyn Engine, - ) -> DeltaResult>> { - if !self - .table_configuration - .protocol() - .has_table_feature(&TableFeature::ClusteredTable) - { - return Ok(None); - } - match self.get_domain_metadata_internal(CLUSTERING_DOMAIN_NAME, engine)? { - Some(config) => Ok(Some(parse_clustering_columns(&config)?)), - None => Ok(None), - } - } - - /// Load domain metadata from this snapshot. If `domains` is `Some`, only load the specified - /// domains (with early termination). If `None`, load all domains. - /// - /// This is the single entry point for all domain metadata reads on a snapshot. All public - /// and internal domain metadata APIs delegate to this method. - #[internal_api] - pub(crate) fn get_domain_metadatas_internal( - &self, - engine: &dyn Engine, - domains: Option<&HashSet<&str>>, - ) -> DeltaResult { - // Fast path: serve from CRC if it tracks domain metadata at this version. - if let Some(crc) = self - .lazy_crc - .get_or_load_if_at_version(engine, self.version()) - { - if let Some(dm_map) = &crc.domain_metadata { - return Ok(match domains { - None => dm_map.clone(), - Some(filter) => dm_map - .iter() - .filter(|(k, _)| filter.contains(k.as_str())) - .map(|(k, v)| (k.clone(), v.clone())) - .collect(), - }); - } - } - // Fallback: full log replay. - self.log_segment().scan_domain_metadatas(domains, engine) - } - - /// Returns file-level statistics, or `None` if no CRC with valid stats exists at this - /// snapshot's version. - /// - /// NOTE: This is an unstable API expected to change in future releases. - #[allow(unused)] - #[internal_api] - pub(crate) fn get_file_stats(&self, engine: &dyn Engine) -> Option { - let crc = self - .lazy_crc - .get_or_load_if_at_version(engine, self.version())?; - crc.file_stats() - } - - /// Returns the CRC if one has been loaded at this snapshot's version (no I/O). - /// - /// This is a test-only helper for integration tests to inspect the CRC state. - #[cfg(any(test, feature = "test-utils"))] - pub fn get_current_crc_if_loaded_for_testing(&self) -> Option<&Crc> { - if self.lazy_crc.crc_version() != Some(self.version()) { - return None; - } - self.lazy_crc.cached.get()?.get().map(|arc| arc.as_ref()) - } - - /// Writes a version checksum (CRC) file for this snapshot. Writers should call this after - /// every commit because checksums enable faster snapshot loading and table state validation. - /// - /// Currently only supports writing from a post-commit snapshot that has pre-computed CRC - /// information in memory (i.e. the snapshot returned by - /// [`CommittedTransaction::post_commit_snapshot`]). - /// - /// Returns [`ChecksumWriteResult::AlreadyExists`] if a checksum already exists at this - /// version (safe for concurrent writers). Returns [`ChecksumWriteResult::Written`] on - /// success. - /// - /// # Errors - /// - /// - [`Error::ChecksumWriteUnsupported`] if no in-memory CRC is available at this - /// snapshot's version (e.g. a snapshot loaded from disk that has no CRC file), or if - /// the CRC's file stats are not valid. File stats can be invalid for two reasons: - /// (a) a non-incremental operation like ANALYZE STATS was encountered, which is - /// recoverable with a full state reconstruction in the future; (b) a file action had a - /// missing size (e.g. `remove.size` is null), which is permanently unrecoverable. - /// - I/O errors from the engine's storage handler if the write fails. - /// - /// [`CommittedTransaction::post_commit_snapshot`]: crate::transaction::CommittedTransaction::post_commit_snapshot - #[instrument(parent = &self.span, name = "snap.write_checksum", skip_all, err)] - pub fn write_checksum(&self, engine: &dyn Engine) -> DeltaResult { - let has_crc_on_disk = self - .log_segment - .listed - .latest_crc_file - .as_ref() - .is_some_and(|f| f.version == self.version()); - - if has_crc_on_disk { - info!( - "CRC file already exists on disk at version {}", - self.version() - ); - return Ok(ChecksumWriteResult::AlreadyExists); - } - - let crc = self - .lazy_crc - .get_if_loaded_at_version(self.version()) - .ok_or_else(|| { - Error::ChecksumWriteUnsupported( - "No in-memory CRC available at this snapshot version.".to_string(), - ) - })?; - - let crc_path = ParsedLogPath::new_crc(self.table_root(), self.version())?; - - // TODO: Would be nice to update LogSegment.listed.latest_crc_file here, but that probably - // requires changing it to a OnceLock, which is a bit of an invasive change. Perhaps, - // return an updated Snapshot instead. - - // Note: try_write_crc_file validates file stats validity before writing. - match try_write_crc_file(engine, &crc_path.location, crc) { - Ok(()) => { - info!("Wrote CRC file at {}", crc_path.location); - Ok(ChecksumWriteResult::Written) - } - Err(Error::FileAlreadyExists(_)) => { - info!( - "Another writer beat us to writing CRC file at {}", - crc_path.location - ); - Ok(ChecksumWriteResult::AlreadyExists) - } - Err(e) => Err(e), - } - } - - /// Publishes all catalog commits at this table version. Applicable only to catalog-managed - /// tables. This method is a no-op for filesystem-managed tables or if there are no catalog - /// commits to publish. - /// - /// Publishing copies ratified catalog commits to the Delta log as published Delta files, - /// reducing catalog storage requirements and enabling some table maintenance operations, - /// like checkpointing. - /// - /// # Parameters - /// - /// - `engine`: The engine to use for publishing commits - /// - /// # Errors - /// - /// Returns an error if the publish operation fails, or if there are catalog commits that need - /// publishing but the table or committer don't support publishing. - /// - /// # See Also - /// - /// - [`Committer::publish`] - #[instrument(parent = &self.span, name = "snap.publish", skip_all, err)] - pub fn publish( - self: &SnapshotRef, - engine: &dyn Engine, - committer: &dyn Committer, - ) -> DeltaResult { - let unpublished_catalog_commits = self.log_segment().get_unpublished_catalog_commits()?; - - if unpublished_catalog_commits.is_empty() { - return Ok(Arc::clone(self)); - } - - require!( - unpublished_catalog_commits - .windows(2) - .all(|commits| commits[0].version() + 1 == commits[1].version()), - Error::generic(format!( - "Expected ordered and contiguous unpublished catalog commits. \ - Got: {unpublished_catalog_commits:?}" - )) - ); - - require!( - self.table_configuration().is_catalog_managed(), - Error::generic( - "There are catalog commits that need publishing, but the table is not catalog-managed.", - ) - ); - - require!( - committer.is_catalog_committer(), - Error::generic( - "There are catalog commits that need publishing, but the committer is not a catalog committer.", - ) - ); - - let publish_metadata = - PublishMetadata::try_new(self.version(), unpublished_catalog_commits)?; - - committer.publish(engine, publish_metadata)?; - - Ok(Arc::new(Snapshot::new_with_crc( - self.log_segment().new_as_published()?, - self.table_configuration().clone(), - self.lazy_crc.clone(), - ))) - } - - /// Fetch both user-controlled and system-controlled domain metadata for a specific domain - /// in this snapshot. - /// - /// Returns the latest configuration for the domain, or `None` if the domain does not exist - /// (or was removed). Unlike [`Snapshot::get_domain_metadata`], this does not reject `delta.*` domains. - #[allow(unused)] - #[internal_api] - pub(crate) fn get_domain_metadata_internal( - &self, - domain: &str, - engine: &dyn Engine, - ) -> DeltaResult> { - let mut map = self.get_domain_metadatas_internal(engine, Some(&HashSet::from([domain])))?; - Ok(map.remove(domain).map(|dm| dm.configuration().to_owned())) - } - - /// Fetch all non-internal domain metadata for this snapshot as a `Vec`. - /// - /// Internal (`delta.*`) domains are filtered out. - #[allow(unused)] - #[internal_api] - pub(crate) fn get_all_domain_metadata( - &self, - engine: &dyn Engine, - ) -> DeltaResult> { - let all_metadata = self.get_domain_metadatas_internal(engine, None)?; - Ok(all_metadata - .into_values() - .filter(|domain| !domain.is_internal()) - .collect()) - } - - /// Get the In-Commit Timestamp (ICT) for this snapshot. - /// - /// Returns the `inCommitTimestamp` from the CommitInfo action of the commit that created this snapshot. - /// - /// # Returns - /// - `Ok(Some(timestamp))` - ICT is enabled and available for this version - /// - `Ok(None)` - ICT is not enabled - /// - `Err(...)` - ICT is enabled but cannot be read, or enablement version is invalid - #[internal_api] - #[instrument(parent = &self.span, name = "snap.get_ict", skip_all, err)] - pub(crate) fn get_in_commit_timestamp(&self, engine: &dyn Engine) -> DeltaResult> { - // Get ICT enablement info and check if we should read ICT for this version - let enablement = self - .table_configuration() - .in_commit_timestamp_enablement()?; - - // Return None if ICT is not enabled at all - if matches!(enablement, InCommitTimestampEnablement::NotEnabled) { - return Ok(None); - } - - // If ICT is enabled with an enablement version, verify the enablement version is not in the future - if let InCommitTimestampEnablement::Enabled { - enablement: Some((enablement_version, _)), - } = enablement - { - if self.version() < enablement_version { - return Err(Error::generic(format!( - "Invalid state: snapshot at version {} has ICT enablement version {} in the future", - self.version(), - enablement_version - ))); - } - } - - // Fast path: try reading ICT from CRC file (if it is at this snapshot version) - if let Some(crc) = self - .lazy_crc - .get_or_load_if_at_version(engine, self.version()) - { - match crc.in_commit_timestamp_opt { - Some(ict) => return Ok(Some(ict)), - None => { - return Err(Error::generic(format!( - "In-Commit Timestamp not found in CRC file at version {}", - self.version() - ))); - } - } - } - - // Fallback: read the ICT from latest_commit_file - match &self.log_segment.listed.latest_commit_file { - Some(commit_file_meta) => { - let ict = commit_file_meta.read_in_commit_timestamp(engine)?; - Ok(Some(ict)) - } - None => Err(Error::generic("Last commit file not found in log segment")), - } - } -} - -#[cfg(test)] -mod tests { - use super::*; - - use std::path::PathBuf; - use std::sync::Arc; - - use serde_json::json; - use test_utils::{add_commit, delta_path_for_version}; - - use crate::actions::{DomainMetadata, Protocol}; - use crate::arrow::array::StringArray; - use crate::arrow::record_batch::RecordBatch; - use crate::engine::arrow_data::ArrowEngineData; - use crate::engine::default::executor::tokio::TokioBackgroundExecutor; - use crate::engine::default::filesystem::ObjectStoreStorageHandler; - use crate::engine::default::DefaultEngineBuilder; - use crate::engine::sync::SyncEngine; - use crate::last_checkpoint_hint::LastCheckpointHint; - use crate::log_segment::LogSegment; - use crate::log_segment_files::LogSegmentFiles; - use crate::object_store::local::LocalFileSystem; - use crate::object_store::memory::InMemory; - use crate::object_store::path::Path; - use crate::object_store::ObjectStore; - use crate::parquet::arrow::ArrowWriter; - use crate::path::{LogPathFileType, ParsedLogPath}; - use crate::table_features::{ - TABLE_FEATURES_MIN_READER_VERSION, TABLE_FEATURES_MIN_WRITER_VERSION, - }; - use crate::utils::test_utils::{assert_result_error_with_message, string_array_to_engine_data}; - - /// Helper function to create a commitInfo action with optional ICT - fn create_commit_info(timestamp: i64, ict: Option) -> serde_json::Value { - let mut commit_info = json!({ - "timestamp": timestamp, - "operation": "WRITE", - }); - - if let Some(ict_value) = ict { - commit_info["inCommitTimestamp"] = json!(ict_value); - } - - json!({ - "commitInfo": commit_info - }) - } - - fn create_protocol(ict_enabled: bool, min_reader_version: Option) -> serde_json::Value { - let reader_version = min_reader_version.unwrap_or(1); - - if ict_enabled { - let mut protocol = json!({ - "protocol": { - "minReaderVersion": reader_version, - "minWriterVersion": TABLE_FEATURES_MIN_WRITER_VERSION, - "writerFeatures": ["inCommitTimestamp"] - } - }); - - // Only include readerFeatures if minReaderVersion >= table-features minimum. - if reader_version >= TABLE_FEATURES_MIN_READER_VERSION as u32 { - protocol["protocol"]["readerFeatures"] = json!([]); - } - - protocol - } else { - json!({ - "protocol": { - "minReaderVersion": reader_version, - "minWriterVersion": 2 - } - }) - } - } - - fn create_metadata( - id: Option<&str>, - schema_string: Option<&str>, - created_time: Option, - ict_config: Option<(String, String)>, - ict_enabled_but_missing_version: bool, - ) -> serde_json::Value { - let config = if ict_enabled_but_missing_version { - // Special case for testing ICT enabled but missing enablement info - json!({ - "delta.enableInCommitTimestamps": "true" - }) - } else if let Some((enablement_version, enablement_timestamp)) = ict_config { - json!({ - "delta.enableInCommitTimestamps": "true", - "delta.inCommitTimestampEnablementVersion": enablement_version, - "delta.inCommitTimestampEnablementTimestamp": enablement_timestamp - }) - } else { - json!({}) - }; - - json!({ - "metaData": { - "id": id.unwrap_or("testId"), - "format": {"provider": "parquet", "options": {}}, - "schemaString": schema_string.unwrap_or("{\"type\":\"struct\",\"fields\":[{\"name\":\"value\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}"), - "partitionColumns": [], - "configuration": config, - "createdTime": created_time.unwrap_or(1587968586154u64) - } - }) - } - - fn create_basic_commit(ict_enabled: bool, ict_config: Option<(String, String)>) -> String { - let protocol = create_protocol(ict_enabled, None); - let metadata = create_metadata(None, None, None, ict_config, false); - format!("{protocol}\n{metadata}") - } - - #[test] - fn test_snapshot_read_metadata() { - let path = - std::fs::canonicalize(PathBuf::from("./tests/data/table-with-dv-small/")).unwrap(); - let url = url::Url::from_directory_path(path).unwrap(); - - let engine = SyncEngine::new(); - let snapshot = Snapshot::builder_for(url) - .at_version(1) - .build(&engine) - .unwrap(); - - let expected = Protocol::try_new_modern(["deletionVectors"], ["deletionVectors"]).unwrap(); - assert_eq!(snapshot.table_configuration().protocol(), &expected); - - let schema_string = r#"{"type":"struct","fields":[{"name":"value","type":"integer","nullable":true,"metadata":{}}]}"#; - let expected: SchemaRef = serde_json::from_str(schema_string).unwrap(); - assert_eq!(snapshot.schema(), expected); - } - - #[test] - fn test_new_snapshot() { - let path = - std::fs::canonicalize(PathBuf::from("./tests/data/table-with-dv-small/")).unwrap(); - let url = url::Url::from_directory_path(path).unwrap(); - - let engine = SyncEngine::new(); - let snapshot = Snapshot::builder_for(url).build(&engine).unwrap(); - - let expected = Protocol::try_new_modern(["deletionVectors"], ["deletionVectors"]).unwrap(); - assert_eq!(snapshot.table_configuration().protocol(), &expected); - - let schema_string = r#"{"type":"struct","fields":[{"name":"value","type":"integer","nullable":true,"metadata":{}}]}"#; - let expected: SchemaRef = serde_json::from_str(schema_string).unwrap(); - assert_eq!(snapshot.schema(), expected); - } - - // TODO: unify this and lots of stuff in LogSegment tests and test_utils. - async fn commit( - table_root: impl AsRef, - store: &InMemory, - version: Version, - commit: Vec, - ) { - let commit_data = commit - .iter() - .map(ToString::to_string) - .collect::>() - .join("\n"); - add_commit(table_root, store, version, commit_data) - .await - .unwrap(); - } - - // interesting cases for testing Snapshot::new_from: - // 1. new version < existing version - // 2. new version == existing version - // 3. new version > existing version AND - // a. log segment hasn't changed - // b. log segment for old..=new version has a checkpoint (with new protocol/metadata) - // b. log segment for old..=new version has no checkpoint - // i. commits have (new protocol, new metadata) - // ii. commits have (new protocol, no metadata) - // iii. commits have (no protocol, new metadata) - // iv. commits have (no protocol, no metadata) - #[tokio::test] - async fn test_snapshot_new_from() -> DeltaResult<()> { - let path = - std::fs::canonicalize(PathBuf::from("./tests/data/table-with-dv-small/")).unwrap(); - let url = url::Url::from_directory_path(path).unwrap(); - - let engine = SyncEngine::new(); - let old_snapshot = Snapshot::builder_for(url.clone()) - .at_version(1) - .build(&engine) - .unwrap(); - // 1. new version < existing version: error - let snapshot_res = Snapshot::builder_from(old_snapshot.clone()) - .at_version(0) - .build(&engine); - assert!(matches!( - snapshot_res, - Err(Error::Generic(msg)) if msg == "Requested snapshot version 0 is older than snapshot hint version 1" - )); - - // 2. new version == existing version - let snapshot = Snapshot::builder_from(old_snapshot.clone()) - .at_version(1) - .build(&engine) - .unwrap(); - let expected = old_snapshot.clone(); - assert_eq!(snapshot, expected); - - // tests Snapshot::new_from by: - // 1. creating a snapshot with new API for commits 0..=2 (based on old snapshot at 0) - // 2. comparing with a snapshot created directly at version 2 - // - // the commits tested are: - // - commit 0 -> base snapshot at this version - // - commit 1 -> final snapshots at this version - // - // in each test we will modify versions 1 and 2 to test different scenarios - fn test_new_from(store: Arc) -> DeltaResult<()> { - let table_root = "memory:///"; - let engine = DefaultEngineBuilder::new(store).build(); - let base_snapshot = Snapshot::builder_for(table_root) - .at_version(0) - .build(&engine)?; - let snapshot = Snapshot::builder_from(base_snapshot.clone()) - .at_version(1) - .build(&engine)?; - let expected = Snapshot::builder_for(table_root) - .at_version(1) - .build(&engine)?; - assert_eq!(snapshot, expected); - Ok(()) - } - - // for (3) we will just engineer custom log files - let store = Arc::new(InMemory::new()); - // everything will have a starting 0 commit with commitInfo, protocol, metadata - let commit0 = vec![ - json!({ - "commitInfo": { - "timestamp": 1587968586154i64, - "operation": "WRITE", - "operationParameters": {"mode":"ErrorIfExists","partitionBy":"[]"}, - "isBlindAppend":true - } - }), - json!({ - "protocol": { - "minReaderVersion": 1, - "minWriterVersion": 2 - } - }), - json!({ - "metaData": { - "id":"5fba94ed-9794-4965-ba6e-6ee3c0d22af9", - "format": { - "provider": "parquet", - "options": {} - }, - "schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"val\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}", - "partitionColumns": [], - "configuration": {}, - "createdTime": 1587968585495i64 - } - }), - ]; - let table_root = "memory:///"; - commit(table_root, store.as_ref(), 0, commit0.clone()).await; - // 3. new version > existing version - // a. no new log segment - let engine = DefaultEngineBuilder::new(Arc::new(store.fork())).build(); - let base_snapshot = Snapshot::builder_for(table_root) - .at_version(0) - .build(&engine)?; - let snapshot = Snapshot::builder_from(base_snapshot.clone()).build(&engine)?; - let expected = Snapshot::builder_for(table_root) - .at_version(0) - .build(&engine)?; - assert_eq!(snapshot, expected); - // version exceeds latest version of the table = err - assert!(matches!( - Snapshot::builder_from(base_snapshot.clone()).at_version(1).build(&engine), - Err(Error::Generic(msg)) if msg == "Requested snapshot version 1 is newer than the latest version 0" - )); - - // b. log segment for old..=new version has a checkpoint (with new protocol/metadata) - let store_3a = store.fork(); - let mut checkpoint1 = commit0.clone(); - commit(table_root, &store_3a, 1, commit0.clone()).await; - checkpoint1[1] = json!({ - "protocol": { - "minReaderVersion": 2, - "minWriterVersion": 5 - } - }); - checkpoint1[2]["partitionColumns"] = serde_json::to_value(["some_partition_column"])?; - - let handler = engine.json_handler(); - let json_strings: StringArray = checkpoint1 - .into_iter() - .map(|json| json.to_string()) - .collect::>() - .into(); - let parsed = handler - .parse_json( - string_array_to_engine_data(json_strings), - crate::actions::get_commit_schema().clone(), - ) - .unwrap(); - let checkpoint = ArrowEngineData::try_from_engine_data(parsed).unwrap(); - let checkpoint: RecordBatch = checkpoint.into(); - - // Write the record batch to a Parquet file - let mut buffer = vec![]; - let mut writer = ArrowWriter::try_new(&mut buffer, checkpoint.schema(), None)?; - writer.write(&checkpoint)?; - writer.close()?; - - store_3a - .put( - &delta_path_for_version(1, "checkpoint.parquet"), - buffer.into(), - ) - .await - .unwrap(); - test_new_from(store_3a.into())?; - - // c. log segment for old..=new version has no checkpoint - // i. commits have (new protocol, new metadata) - let store_3c_i = Arc::new(store.fork()); - let mut commit1 = commit0.clone(); - commit1[1] = json!({ - "protocol": { - "minReaderVersion": 2, - "minWriterVersion": 5 - } - }); - commit1[2]["partitionColumns"] = serde_json::to_value(["some_partition_column"])?; - commit(table_root, store_3c_i.as_ref(), 1, commit1).await; - test_new_from(store_3c_i.clone())?; - - // new commits AND request version > end of log - let engine = DefaultEngineBuilder::new(store_3c_i).build(); - let base_snapshot = Snapshot::builder_for(table_root) - .at_version(0) - .build(&engine)?; - assert!(matches!( - Snapshot::builder_from(base_snapshot.clone()).at_version(2).build(&engine), - Err(Error::Generic(msg)) if msg == "LogSegment end version 1 not the same as the specified end version 2" - )); - - // ii. commits have (new protocol, no metadata) - let store_3c_ii = store.fork(); - let mut commit1 = commit0.clone(); - commit1[1] = json!({ - "protocol": { - "minReaderVersion": 2, - "minWriterVersion": 5 - } - }); - commit1.remove(2); // remove metadata - commit(table_root, &store_3c_ii, 1, commit1).await; - test_new_from(store_3c_ii.into())?; - - // iii. commits have (no protocol, new metadata) - let store_3c_iii = store.fork(); - let mut commit1 = commit0.clone(); - commit1[2]["partitionColumns"] = serde_json::to_value(["some_partition_column"])?; - commit1.remove(1); // remove protocol - commit(table_root, &store_3c_iii, 1, commit1).await; - test_new_from(store_3c_iii.into())?; - - // iv. commits have (no protocol, no metadata) - let store_3c_iv = store.fork(); - let commit1 = vec![commit0[0].clone()]; - commit(table_root, &store_3c_iv, 1, commit1).await; - test_new_from(store_3c_iv.into())?; - - Ok(()) - } - - // test new CRC in new log segment (old log segment has old CRC) - #[tokio::test] - async fn test_snapshot_new_from_crc() -> Result<(), Box> { - let store = Arc::new(InMemory::new()); - let table_root = "memory:///"; - let engine = DefaultEngineBuilder::new(store.clone()).build(); - let protocol = |reader_version, writer_version| { - json!({ - "protocol": { - "minReaderVersion": reader_version, - "minWriterVersion": writer_version - } - }) - }; - let metadata = json!({ - "metaData": { - "id":"5fba94ed-9794-4965-ba6e-6ee3c0d22af9", - "format": { - "provider": "parquet", - "options": {} - }, - "schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"val\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}", - "partitionColumns": [], - "configuration": {}, - "createdTime": 1587968585495i64 - } - }); - let commit0 = vec![ - json!({ - "commitInfo": { - "timestamp": 1587968586154i64, - "operation": "WRITE", - "operationParameters": {"mode":"ErrorIfExists","partitionBy":"[]"}, - "isBlindAppend":true - } - }), - protocol(1, 1), - metadata.clone(), - ]; - let commit1 = vec![ - json!({ - "commitInfo": { - "timestamp": 1587968586154i64, - "operation": "WRITE", - "operationParameters": {"mode":"ErrorIfExists","partitionBy":"[]"}, - "isBlindAppend":true - } - }), - protocol(1, 2), - ]; - - // commit 0 and 1 jsons - commit(table_root, &store, 0, commit0.clone()).await; - commit(table_root, &store, 1, commit1).await; - - // a) CRC: old one has 0.crc, no new one (expect 0.crc) - // b) CRC: old one has 0.crc, new one has 1.crc (expect 1.crc) - let crc = json!({ - "table_size_bytes": 100, - "num_files": 1, - "num_metadata": 1, - "num_protocol": 1, - "metadata": metadata, - "protocol": protocol(1, 1), - }); - - // put the old crc - let path = delta_path_for_version(0, "crc"); - store.put(&path, crc.to_string().into()).await?; - - // base snapshot is at version 0 - let base_snapshot = Snapshot::builder_for(table_root) - .at_version(0) - .build(&engine)?; - - // first test: no new crc - let snapshot = Snapshot::builder_from(base_snapshot.clone()) - .at_version(1) - .build(&engine)?; - let expected = Snapshot::builder_for(table_root) - .at_version(1) - .build(&engine)?; - assert_eq!(snapshot, expected); - assert_eq!( - snapshot - .log_segment - .listed - .latest_crc_file - .as_ref() - .unwrap() - .version, - 0 - ); - - // second test: new crc - // put the new crc - let path = delta_path_for_version(1, "crc"); - let crc = json!({ - "table_size_bytes": 100, - "num_files": 1, - "num_metadata": 1, - "num_protocol": 1, - "metadata": metadata, - "protocol": protocol(1, 2), - }); - store.put(&path, crc.to_string().into()).await?; - let snapshot = Snapshot::builder_from(base_snapshot.clone()) - .at_version(1) - .build(&engine)?; - let expected = Snapshot::builder_for(table_root) - .at_version(1) - .build(&engine)?; - assert_eq!(snapshot, expected); - assert_eq!( - snapshot - .log_segment - .listed - .latest_crc_file - .as_ref() - .unwrap() - .version, - 1 - ); - - Ok(()) - } - - #[test] - fn test_read_table_with_missing_last_checkpoint() { - // this table doesn't have a _last_checkpoint file - let path = std::fs::canonicalize(PathBuf::from( - "./tests/data/table-with-dv-small/_delta_log/", - )) - .unwrap(); - let url = url::Url::from_directory_path(path).unwrap(); - - let store = Arc::new(LocalFileSystem::new()); - let executor = Arc::new(TokioBackgroundExecutor::new()); - let storage = ObjectStoreStorageHandler::new(store, executor, None); - let cp = LastCheckpointHint::try_read(&storage, &url).unwrap(); - assert!(cp.is_none()); - } - - fn valid_last_checkpoint() -> (Vec, LastCheckpointHint) { - let checkpoint = LastCheckpointHint { - version: 1, - size: 8, - parts: None, - size_in_bytes: Some(21857), - num_of_add_files: None, - checkpoint_schema: None, - checksum: None, - tags: None, - }; - let data = checkpoint.to_json_bytes(); - (data, checkpoint) - } - - fn valid_last_checkpoint_with_tags() -> (Vec, LastCheckpointHint) { - use std::collections::HashMap; - - let (_, base_checkpoint) = valid_last_checkpoint(); - - let mut tags = HashMap::new(); - tags.insert( - "author".to_string(), - "test_read_table_with_last_checkpoint".to_string(), - ); - tags.insert("environment".to_string(), "snapshot_tests".to_string()); - tags.insert("created_by".to_string(), "delta-kernel-rs".to_string()); - - let checkpoint = LastCheckpointHint { - tags: Some(tags), - ..base_checkpoint - }; - - let data = checkpoint.to_json_bytes(); - (data, checkpoint) - } - - #[tokio::test] - async fn test_read_table_with_empty_last_checkpoint() { - // in memory file system - let store = Arc::new(InMemory::new()); - - // do a _last_checkpoint file with "{}" as content - let empty = "{}".as_bytes().to_vec(); - let invalid_path = Path::from("invalid/_last_checkpoint"); - - store - .put(&invalid_path, empty.into()) - .await - .expect("put _last_checkpoint"); - - let executor = Arc::new(TokioBackgroundExecutor::new()); - let storage = ObjectStoreStorageHandler::new(store, executor, None); - let url = Url::parse("memory:///invalid/").expect("valid url"); - let invalid = LastCheckpointHint::try_read(&storage, &url).expect("read last checkpoint"); - assert!(invalid.is_none()) - } - - #[tokio::test] - async fn test_read_table_with_last_checkpoint() { - // in memory file system - let store = Arc::new(InMemory::new()); - - // Define test cases: (path, data, expected_result) - let (data, expected) = valid_last_checkpoint(); - let (data_with_tags, expected_with_tags) = valid_last_checkpoint_with_tags(); - let test_cases = vec![ - ("valid", data, Some(expected)), - ("invalid", "invalid".as_bytes().to_vec(), None), - ("valid_with_tags", data_with_tags, Some(expected_with_tags)), - ]; - - // Write all test files to the in memory file system - for (path_prefix, data, _) in &test_cases { - let path = Path::from(format!("{path_prefix}/_last_checkpoint")); - store - .put(&path, data.clone().into()) - .await - .expect("put _last_checkpoint"); - } - - let executor = Arc::new(TokioBackgroundExecutor::new()); - let storage = ObjectStoreStorageHandler::new(store, executor, None); - - // Test reading all checkpoints from the in memory file system for cases where the data is valid, invalid and - // valid with tags. - for (path_prefix, _, expected_result) in test_cases { - let url = Url::parse(&format!("memory:///{path_prefix}/")).expect("valid url"); - let result = - LastCheckpointHint::try_read(&storage, &url).expect("read last checkpoint"); - assert_eq!(result, expected_result); - } - } - - #[test_log::test] - fn test_read_table_with_checkpoint() { - let path = std::fs::canonicalize(PathBuf::from( - "./tests/data/with_checkpoint_no_last_checkpoint/", - )) - .unwrap(); - let location = url::Url::from_directory_path(path).unwrap(); - let engine = SyncEngine::new(); - let snapshot = Snapshot::builder_for(location).build(&engine).unwrap(); - - assert_eq!(snapshot.log_segment.listed.checkpoint_parts.len(), 1); - assert_eq!( - ParsedLogPath::try_from( - snapshot.log_segment.listed.checkpoint_parts[0] - .location - .clone() - ) - .unwrap() - .unwrap() - .version, - 2, - ); - assert_eq!(snapshot.log_segment.listed.ascending_commit_files.len(), 1); - assert_eq!( - ParsedLogPath::try_from( - snapshot.log_segment.listed.ascending_commit_files[0] - .location - .clone() - ) - .unwrap() - .unwrap() - .version, - 3, - ); - } - - #[tokio::test] - async fn test_domain_metadata() -> DeltaResult<()> { - let table_root = "memory:///test_table/"; - let store = Arc::new(InMemory::new()); - let engine = DefaultEngineBuilder::new(store.clone()).build(); - - // commit0 - // - domain1: not removed - // - domain2: not removed - let commit = [ - json!({ - "protocol": { - "minReaderVersion": 1, - "minWriterVersion": 1 - } - }), - json!({ - "metaData": { - "id":"5fba94ed-9794-4965-ba6e-6ee3c0d22af9", - "format": { "provider": "parquet", "options": {} }, - "schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"val\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}", - "partitionColumns": [], - "configuration": {}, - "createdTime": 1587968585495i64 - } - }), - json!({ - "domainMetadata": { - "domain": "domain1", - "configuration": "domain1_commit0", - "removed": false - } - }), - json!({ - "domainMetadata": { - "domain": "domain2", - "configuration": "domain2_commit0", - "removed": false - } - }), - json!({ - "domainMetadata": { - "domain": "domain3", - "configuration": "domain3_commit0", - "removed": false - } - }), - ] - .map(|json| json.to_string()) - .join("\n"); - add_commit(table_root, store.clone().as_ref(), 0, commit) - .await - .unwrap(); - - // commit1 - // - domain1: removed - // - domain2: not-removed - // - internal domain - let commit = [ - json!({ - "domainMetadata": { - "domain": "domain1", - "configuration": "domain1_commit1", - "removed": true - } - }), - json!({ - "domainMetadata": { - "domain": "domain2", - "configuration": "domain2_commit1", - "removed": false - } - }), - json!({ - "domainMetadata": { - "domain": "delta.domain3", - "configuration": "domain3_commit1", - "removed": false - } - }), - ] - .map(|json| json.to_string()) - .join("\n"); - add_commit(table_root, store.as_ref(), 1, commit) - .await - .unwrap(); - - let snapshot = Snapshot::builder_for(table_root).build(&engine)?; - - // Test get_domain_metadata - - assert_eq!(snapshot.get_domain_metadata("domain1", &engine)?, None); - assert_eq!( - snapshot.get_domain_metadata("domain2", &engine)?, - Some("domain2_commit1".to_string()) - ); - assert_eq!( - snapshot.get_domain_metadata("domain3", &engine)?, - Some("domain3_commit0".to_string()) - ); - let err = snapshot - .get_domain_metadata("delta.domain3", &engine) - .unwrap_err(); - assert!(matches!(err, Error::Generic(msg) if - msg == "User DomainMetadata are not allowed to use system-controlled 'delta.*' domain")); - - // Test get_domain_metadata_internal - assert_eq!( - snapshot.get_domain_metadata_internal("delta.domain3", &engine)?, - Some("domain3_commit1".to_string()) - ); - - // Test get_all_domain_metadata - let mut metadata = snapshot.get_all_domain_metadata(&engine)?; - metadata.sort_by(|a, b| a.domain().cmp(b.domain())); - - let mut expected = vec![ - DomainMetadata::new("domain2".to_string(), "domain2_commit1".to_string()), - DomainMetadata::new("domain3".to_string(), "domain3_commit0".to_string()), - ]; - expected.sort_by(|a, b| a.domain().cmp(b.domain())); - - assert_eq!(metadata, expected); - - Ok(()) - } - - #[test] - fn test_log_compaction_writer() { - let path = - std::fs::canonicalize(PathBuf::from("./tests/data/table-with-dv-small/")).unwrap(); - let url = url::Url::from_directory_path(path).unwrap(); - - let engine = SyncEngine::new(); - let snapshot = Snapshot::builder_for(url).build(&engine).unwrap(); - - // Test creating a log compaction writer - let writer = snapshot.clone().log_compaction_writer(0, 1).unwrap(); - let path = writer.compaction_path(); - - // Verify the path format is correct - let expected_filename = "00000000000000000000.00000000000000000001.compacted.json"; - assert!(path.to_string().ends_with(expected_filename)); - - // Test invalid version range (start >= end) - let result = snapshot.clone().log_compaction_writer(2, 1); - assert_result_error_with_message(result, "Invalid version range"); - - // Test equal version range (also invalid) - let result = snapshot.log_compaction_writer(1, 1); - assert_result_error_with_message(result, "Invalid version range"); - } - - #[tokio::test] - async fn test_timestamp_with_ict_disabled() -> Result<(), Box> { - let store = Arc::new(InMemory::new()); - let table_root = "memory://test/"; - let engine = DefaultEngineBuilder::new(store.clone()).build(); - - // Create a basic commit without ICT enabled - let commit0 = create_basic_commit(false, None); - add_commit(table_root, store.as_ref(), 0, commit0).await?; - - let snapshot = Snapshot::builder_for(table_root).build(&engine)?; - - // When ICT is disabled, get_timestamp should return None - let result = snapshot.get_in_commit_timestamp(&engine)?; - assert_eq!(result, None); - - Ok(()) - } - - #[tokio::test] - async fn test_timestamp_with_ict_enablement_timeline() -> Result<(), Box> - { - let store = Arc::new(InMemory::new()); - let table_root = "memory://test/"; - let engine = DefaultEngineBuilder::new(store.clone()).build(); - - // Create initial commit without ICT - let commit0 = create_basic_commit(false, None); - add_commit(table_root, store.as_ref(), 0, commit0).await?; - - // Create commit that enables ICT (version 1 = enablement version) - let commit1 = - create_basic_commit(true, Some(("1".to_string(), "1587968586154".to_string()))); - add_commit(table_root, store.as_ref(), 1, commit1).await?; - - // Create commit with ICT enabled - let expected_timestamp = 1587968586200i64; - let commit2 = format!( - r#"{{"commitInfo":{{"timestamp":1587968586154,"inCommitTimestamp":{expected_timestamp},"operation":"WRITE"}}}}"#, - ); - add_commit(table_root, store.as_ref(), 2, commit2.to_string()).await?; - - // Read snapshot at version 0 (before ICT enablement) - let snapshot_v0 = Snapshot::builder_for(table_root) - .at_version(0) - .build(&engine)?; - // This snapshot version predates ICT enablement, so ICT is not available - let result_v0 = snapshot_v0.get_in_commit_timestamp(&engine)?; - assert_eq!(result_v0, None); - - // Read snapshot at version 2 (after ICT enabled) - let snapshot_v2 = Snapshot::builder_for(table_root) - .at_version(2) - .build(&engine)?; - // When ICT is enabled and available, timestamp() should return inCommitTimestamp - let result_v2 = snapshot_v2.get_in_commit_timestamp(&engine)?; - assert_eq!(result_v2, Some(expected_timestamp)); - - Ok(()) - } - - #[tokio::test] - async fn test_get_timestamp_enablement_version_in_future() -> DeltaResult<()> { - // Test invalid state where snapshot has enablement version in the future - should error - let table_root = "memory:///test_table/"; - let store = Arc::new(InMemory::new()); - let engine = DefaultEngineBuilder::new(store.clone()).build(); - - let commit_data = [ - json!({ - "protocol": { - "minReaderVersion": TABLE_FEATURES_MIN_READER_VERSION, - "minWriterVersion": TABLE_FEATURES_MIN_WRITER_VERSION, - "readerFeatures": [], - "writerFeatures": ["inCommitTimestamp"] - } - }), - json!({ - "metaData": { - "id": "test_id2", - "format": {"provider": "parquet", "options": {}}, - "schemaString": "{\"type\":\"struct\",\"fields\":[]}", - "partitionColumns": [], - "configuration": { - "delta.enableInCommitTimestamps": "true", - "delta.inCommitTimestampEnablementVersion": "5", // Enablement after version 1 - "delta.inCommitTimestampEnablementTimestamp": "1612345678" - }, - "createdTime": 1677811175819u64 - } - }), - ]; - commit(table_root, store.as_ref(), 0, commit_data.to_vec()).await; - - // Create commit that predates ICT enablement (no inCommitTimestamp) - let commit_predates = [create_commit_info(1234567890, None)]; - commit(table_root, store.as_ref(), 1, commit_predates.to_vec()).await; - - let snapshot_predates = Snapshot::builder_for(table_root) - .at_version(1) - .build(&engine)?; - let result_predates = snapshot_predates.get_in_commit_timestamp(&engine); - - // Version 1 with enablement at version 5 is invalid - should error - assert_result_error_with_message( - result_predates, - "Invalid state: snapshot at version 1 has ICT enablement version 5 in the future", - ); - - Ok(()) - } - - #[tokio::test] - async fn test_get_timestamp_missing_ict_when_enabled() -> DeltaResult<()> { - // Test missing ICT when it should be present - should error - let table_root = "memory:///test_table/"; - let store = Arc::new(InMemory::new()); - let engine = DefaultEngineBuilder::new(store.clone()).build(); - - let commit_data = [ - create_protocol(true, Some(TABLE_FEATURES_MIN_READER_VERSION as u32)), - create_metadata( - Some("test_id"), - Some("{\"type\":\"struct\",\"fields\":[]}"), - Some(1677811175819), - Some(("0".to_string(), "1612345678".to_string())), - false, - ), - ]; - commit(table_root, store.as_ref(), 0, commit_data.to_vec()).await; // ICT enabled from version 0 - - // Create commit without ICT despite being enabled (corrupt case) - let commit_missing_ict = [create_commit_info(1234567890, None)]; - commit(table_root, store.as_ref(), 1, commit_missing_ict.to_vec()).await; - - let snapshot_missing = Snapshot::builder_for(table_root) - .at_version(1) - .build(&engine)?; - let result = snapshot_missing.get_in_commit_timestamp(&engine); - assert_result_error_with_message(result, "In-Commit Timestamp not found"); - - Ok(()) - } - - #[tokio::test] - async fn test_get_timestamp_fails_when_commit_missing() -> DeltaResult<()> { - // When ICT is enabled but commit file is not found in log segment, - // get_in_commit_timestamp should return an error - - let url = Url::parse("memory:///")?; - let store = Arc::new(InMemory::new()); - let engine = DefaultEngineBuilder::new(store.clone()).build(); - - // Create initial commit with ICT enabled - let commit_data = [ - create_protocol(true, Some(TABLE_FEATURES_MIN_READER_VERSION as u32)), - create_metadata( - Some("test_id"), - Some("{\"type\":\"struct\",\"fields\":[]}"), - Some(1677811175819), - Some(("0".to_string(), "1612345678".to_string())), // ICT enabled from version 0 - false, - ), - ]; - commit(url.as_str(), store.as_ref(), 0, commit_data.to_vec()).await; - - // Build snapshot to get table configuration - let snapshot = Snapshot::builder_for(url.as_str()) - .at_version(0) - .build(&engine)?; - - // Create a log segment with only checkpoint and no commit file (simulating scenario - // where a checkpoint exists but the commit file has been cleaned up) - let checkpoint_parts = vec![ParsedLogPath::try_from(crate::FileMeta { - location: url.join("_delta_log/00000000000000000000.checkpoint.parquet")?, - last_modified: 0, - size: 100, - })? - .unwrap()]; - - let listed_files = LogSegmentFiles { - checkpoint_parts, - ..Default::default() - }; - - let log_segment = - LogSegment::try_new(listed_files, url.join("_delta_log/")?, Some(0), None)?; - let table_config = snapshot.table_configuration().clone(); - - // Create snapshot without commit file in log segment - let snapshot_no_commit = Snapshot::new(log_segment, table_config); - - // Should return an error when commit file is missing - let result = snapshot_no_commit.get_in_commit_timestamp(&engine); - assert_result_error_with_message(result, "Last commit file not found in log segment"); - - Ok(()) - } - - #[tokio::test] - async fn test_get_timestamp_with_checkpoint_and_commit_same_version() -> DeltaResult<()> { - // Test the scenario where both checkpoint and commit exist at the same version with ICT enabled. - let table_root = "memory:///test_table/"; - let store = Arc::new(InMemory::new()); - let engine = DefaultEngineBuilder::new(store.clone()).build(); - - // Create 00000000000000000000.json with ICT enabled - let commit0_data = [ - create_commit_info(1587968586154, None), - create_protocol(true, Some(TABLE_FEATURES_MIN_READER_VERSION as u32)), - create_metadata( - Some("5fba94ed-9794-4965-ba6e-6ee3c0d22af9"), - Some("{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"val\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}"), - Some(1587968585495), - Some(("0".to_string(), "1587968586154".to_string())), - false, - ), - ]; - commit(table_root, store.as_ref(), 0, commit0_data.to_vec()).await; - - // Create 00000000000000000001.checkpoint.parquet - let checkpoint_data = [ - create_commit_info(1587968586154, None), - create_protocol(true, Some(TABLE_FEATURES_MIN_READER_VERSION as u32)), - create_metadata( - Some("5fba94ed-9794-4965-ba6e-6ee3c0d22af9"), - Some("{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"val\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}"), - Some(1587968585495), - Some(("0".to_string(), "1587968586154".to_string())), - false, - ), - ]; - - let handler = engine.json_handler(); - let json_strings: StringArray = checkpoint_data - .into_iter() - .map(|json| json.to_string()) - .collect::>() - .into(); - let parsed = handler.parse_json( - string_array_to_engine_data(json_strings), - crate::actions::get_commit_schema().clone(), - )?; - let checkpoint = ArrowEngineData::try_from_engine_data(parsed)?; - let checkpoint: RecordBatch = checkpoint.into(); - - let mut buffer = vec![]; - let mut writer = ArrowWriter::try_new(&mut buffer, checkpoint.schema(), None)?; - writer.write(&checkpoint)?; - writer.close()?; - - let checkpoint_path = delta_path_for_version(1, "checkpoint.parquet"); - store.put(&checkpoint_path, buffer.into()).await?; - - // Create 00000000000000000001.json with ICT - let expected_ict = 1587968586200i64; - let commit1_data = [create_commit_info(1587968586200, Some(expected_ict))]; - commit(table_root, store.as_ref(), 1, commit1_data.to_vec()).await; - - // Build snapshot - LogSegment will filter out the commit file because checkpoint exists at same version - let snapshot = Snapshot::builder_for(table_root) - .at_version(1) - .build(&engine)?; - - // We should successfully read ICT by falling back to storage - let timestamp = snapshot.get_in_commit_timestamp(&engine)?; - assert_eq!(timestamp, Some(expected_ict)); - - Ok(()) - } - - #[tokio::test] - async fn test_try_new_from_empty_log_tail() -> DeltaResult<()> { - let store = Arc::new(InMemory::new()); - let table_root = "memory:///test_table/"; - let engine = DefaultEngineBuilder::new(store.clone()).build(); - - // Create initial commit - let commit0 = vec![ - json!({ - "protocol": { - "minReaderVersion": 1, - "minWriterVersion": 2 - } - }), - json!({ - "metaData": { - "id": "test-id", - "format": {"provider": "parquet", "options": {}}, - "schemaString": "{\"type\":\"struct\",\"fields\":[]}", - "partitionColumns": [], - "configuration": {}, - "createdTime": 1587968585495i64 - } - }), - ]; - commit(table_root, store.as_ref(), 0, commit0).await; - - let base_snapshot = Snapshot::builder_for(table_root) - .at_version(0) - .build(&engine)?; - - // Test with empty log tail - should return same snapshot - let result = Snapshot::try_new_from(base_snapshot.clone(), vec![], &engine, None, None)?; - assert_eq!(result, base_snapshot); - - Ok(()) - } - - #[tokio::test] - async fn test_try_new_from_latest_commit_preservation() -> DeltaResult<()> { - let store = Arc::new(InMemory::new()); - let url = Url::parse("memory:///")?; - let engine = DefaultEngineBuilder::new(store.clone()).build(); - - // Create commits 0-2 - let base_commit = vec![ - json!({"protocol": {"minReaderVersion": 1, "minWriterVersion": 2}}), - json!({ - "metaData": { - "id": "test-id", - "format": {"provider": "parquet", "options": {}}, - "schemaString": "{\"type\":\"struct\",\"fields\":[]}", - "partitionColumns": [], - "configuration": {}, - "createdTime": 1587968585495i64 - } - }), - ]; - - commit(url.as_str(), store.as_ref(), 0, base_commit.clone()).await; - commit( - url.as_str(), - store.as_ref(), - 1, - vec![json!({"commitInfo": {"timestamp": 1234}})], - ) - .await; - commit( - url.as_str(), - store.as_ref(), - 2, - vec![json!({"commitInfo": {"timestamp": 5678}})], - ) - .await; - - let base_snapshot = Snapshot::builder_for(url.as_str()) - .at_version(1) - .build(&engine)?; - - // Verify base snapshot has latest_commit_file at version 1 - assert_eq!( - base_snapshot - .log_segment - .listed - .latest_commit_file - .as_ref() - .map(|f| f.version), - Some(1) - ); - - // Create log_tail with FileMeta for version 2 - let commit_2_url = url.join("_delta_log/")?.join("00000000000000000002.json")?; - let file_meta = crate::FileMeta { - location: commit_2_url, - last_modified: 1234567890, - size: 100, - }; - let parsed_path = ParsedLogPath::try_from(file_meta)? - .ok_or_else(|| Error::Generic("Failed to parse log path".to_string()))?; - let log_tail = vec![parsed_path]; - - // Create new snapshot from base to version 2 using try_new_from directly - let new_snapshot = - Snapshot::try_new_from(base_snapshot.clone(), log_tail, &engine, Some(2), None)?; - - // Latest commit should now be version 2 - assert_eq!( - new_snapshot - .log_segment - .listed - .latest_commit_file - .as_ref() - .map(|f| f.version), - Some(2) - ); - - Ok(()) - } - - #[tokio::test] - async fn test_try_new_from_version_boundary_cases() -> DeltaResult<()> { - let store = Arc::new(InMemory::new()); - let table_root = "memory:///test_table/"; - let engine = DefaultEngineBuilder::new(store.clone()).build(); - - // Create commits - let base_commit = vec![ - json!({"protocol": {"minReaderVersion": 1, "minWriterVersion": 2}}), - json!({ - "metaData": { - "id": "test-id", - "format": {"provider": "parquet", "options": {}}, - "schemaString": "{\"type\":\"struct\",\"fields\":[]}", - "partitionColumns": [], - "configuration": {}, - "createdTime": 1587968585495i64 - } - }), - ]; - - commit(table_root, store.as_ref(), 0, base_commit).await; - commit( - table_root, - store.as_ref(), - 1, - vec![json!({"commitInfo": {"timestamp": 1234}})], - ) - .await; - - let base_snapshot = Snapshot::builder_for(table_root) - .at_version(1) - .build(&engine)?; - - // Test requesting same version - should return same snapshot - let same_version = - Snapshot::try_new_from(base_snapshot.clone(), vec![], &engine, Some(1), None)?; - assert!(Arc::ptr_eq(&same_version, &base_snapshot)); - - // Test requesting older version - should error - let older_version = - Snapshot::try_new_from(base_snapshot.clone(), vec![], &engine, Some(0), None); - assert!(matches!( - older_version, - Err(Error::Generic(msg)) if msg.contains("older than snapshot hint version") - )); - - Ok(()) - } - - #[test] - fn test_new_post_commit_simple() { - // GIVEN - let path = std::fs::canonicalize(PathBuf::from("./tests/data/basic_partitioned/")).unwrap(); - let url = url::Url::from_directory_path(path).unwrap(); - let engine = SyncEngine::new(); - let base_snapshot = Snapshot::builder_for(url.clone()).build(&engine).unwrap(); - let next_version = base_snapshot.version() + 1; - - // WHEN - let fake_new_commit = ParsedLogPath::create_parsed_published_commit(&url, next_version); - let post_commit_snapshot = base_snapshot - .new_post_commit(fake_new_commit, CrcDelta::default()) - .unwrap(); - - // THEN - assert_eq!(post_commit_snapshot.version(), next_version); - assert_eq!(post_commit_snapshot.log_segment().end_version, next_version); - } - - // Helper: create a minimal test table with commits 0-N - async fn setup_test_table_with_commits( - table_root: impl AsRef, - store: &InMemory, - num_commits: u64, - ) -> DeltaResult<()> { - // Commit 0: protocol + metadata + first file - let commit0 = vec![ - json!({"protocol": {"minReaderVersion": 1, "minWriterVersion": 2}}), - json!({ - "metaData": { - "id": "test-id", - "format": {"provider": "parquet", "options": {}}, - "schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}", - "partitionColumns": [], - "configuration": {}, - "createdTime": 1587968585495i64 - } - }), - json!({"add": {"path": "file1.parquet", "partitionValues": {}, "size": 100, "modificationTime": 1000, "dataChange": true}}), - ]; - commit(table_root.as_ref(), store, 0, commit0).await; - - // Additional commits with just add actions - for i in 1..num_commits { - let commit_i = vec![json!({ - "add": { - "path": format!("file{}.parquet", i + 1), - "partitionValues": {}, - "size": (i + 1) * 100, - "modificationTime": (i + 1) * 1000, - "dataChange": true - } - })]; - commit(table_root.as_ref(), store, i, commit_i).await; - } - Ok(()) - } - - // Helper: write a compaction file - async fn write_compaction_file(store: &InMemory, start: u64, end: u64) -> DeltaResult<()> { - let content = r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#; - store - .put( - &test_utils::compacted_log_path_for_versions(start, end, "json"), - content.into(), - ) - .await?; - Ok(()) - } - - /// The incremental snapshot path (try_new_from_impl) re-lists files from the checkpoint - /// version onwards. We must ensure that it deduplicates compaction files, since producing - /// duplicates violated the sort invariant in LogSegmentFilesBuilder::build(). - #[tokio::test] - async fn test_incremental_snapshot_with_compaction_files() -> DeltaResult<()> { - let store = Arc::new(InMemory::new()); - let table_root = "memory:///"; - let engine = DefaultEngineBuilder::new(store.clone()).build(); - - // Create commits 0-3 and compaction files (1,1) and (1,2) - setup_test_table_with_commits(table_root, &store, 3).await?; - write_compaction_file(&store, 1, 1).await?; - write_compaction_file(&store, 1, 2).await?; - - // Build snapshot at v2 (includes both compaction files) - let snapshot_v2 = Snapshot::builder_for(table_root) - .at_version(2) - .build(&engine)?; - assert_eq!( - snapshot_v2 - .log_segment - .listed - .ascending_compaction_files - .len(), - 2 - ); - - // Add commit 3 - commit( - table_root, - &store, - 3, - vec![json!({"add": {"path": "file4.parquet", "partitionValues": {}, "size": 400, "modificationTime": 4000, "dataChange": true}})], - ) - .await; - - // Build v3 incrementally - before the fix, this panicked due to duplicate compaction files - let snapshot_v3 = Snapshot::builder_from(snapshot_v2) - .at_version(3) - .build(&engine)?; - - assert_eq!(snapshot_v3.version(), 3); - assert_eq!( - snapshot_v3 - .log_segment - .listed - .ascending_compaction_files - .len(), - 2 - ); - - Ok(()) - } - - /// This test documents a limitation: When deduplicating compactions, the deduplication logic - /// only checks the start version (lo), not the hi version. So a new compaction file (1,3) - /// added after building the base snapshot at v2 gets filtered out because its start version - /// (1) <= old_version (2). - #[tokio::test] - async fn test_incremental_snapshot_with_new_compaction_files() -> DeltaResult<()> { - let store = Arc::new(InMemory::new()); - let table_root = "memory:///"; - let engine = DefaultEngineBuilder::new(store.clone()).build(); - - // Create commits 0-3 and compaction files (1,2) and (2,2) - setup_test_table_with_commits(table_root, &store, 4).await?; - write_compaction_file(&store, 1, 2).await?; - write_compaction_file(&store, 2, 2).await?; - - // Build snapshot at v2 - let snapshot_v2 = Snapshot::builder_for(table_root) - .at_version(2) - .build(&engine)?; - assert_eq!( - snapshot_v2 - .log_segment - .listed - .ascending_compaction_files - .len(), - 2 - ); - - // Add new compaction file (1,3) after building the base snapshot - write_compaction_file(&store, 1, 3).await?; - - // Build v3 incrementally - the new (1,3) file gets filtered out because - // the deduplication only looks at start version: 1 <= old_version (2) - let snapshot_v3 = Snapshot::builder_from(snapshot_v2) - .at_version(3) - .build(&engine)?; - - assert_eq!(snapshot_v3.version(), 3); - assert_eq!( - snapshot_v3 - .log_segment - .listed - .ascending_compaction_files - .len(), - 2 - ); - - // Verify we still have the original (1,2) and (2,2) files - let versions_and_his: Vec<_> = snapshot_v3 - .log_segment - .listed - .ascending_compaction_files - .iter() - .map(|p| match p.file_type { - LogPathFileType::CompactedCommit { hi } => (p.version, hi), - _ => panic!("Expected CompactedCommit"), - }) - .collect(); - assert_eq!(versions_and_his, vec![(1, 2), (2, 2)]); - - Ok(()) - } -} diff --git a/kernel/src/snapshot/builder.rs b/kernel/src/snapshot/builder.rs index 64847b32b8..d5ecd15bf0 100644 --- a/kernel/src/snapshot/builder.rs +++ b/kernel/src/snapshot/builder.rs @@ -1,13 +1,18 @@ //! Builder for creating [`Snapshot`] instances. + +use std::sync::Arc; + +use tracing::{info, instrument}; + use crate::log_path::LogPath; use crate::log_segment::LogSegment; -use crate::metrics::MetricId; +use crate::metrics::events::SNAPSHOT_COMPLETED_SPAN; +use crate::metrics::{LogSegmentLoadType, MetricId, SnapshotLoadMetricContext}; +use crate::path::LogPathFileType; use crate::snapshot::SnapshotRef; -use crate::utils::try_parse_uri; +use crate::utils::{require, try_parse_uri}; use crate::{DeltaResult, Engine, Error, Snapshot, Version}; -use tracing::{info, instrument}; - /// Builder for creating [`Snapshot`] instances. /// /// # Example @@ -36,15 +41,74 @@ pub struct SnapshotBuilder { existing_snapshot: Option, version: Option, log_tail: Vec, + max_catalog_version: Option, + incremental_replay: IncrementalReplay, + /// Kernel-minted id correlating this build's metric events with its child events. + operation_id: MetricId, + /// Opaque, caller-supplied id recorded on this build's metric events. Not interpreted by + /// kernel; set via [`with_correlation_id`](Self::with_correlation_id). + correlation_id: Option>, +} + +/// Controls whether kernel replays commits to advance a stale base CRC (the existing snapshot's +/// in-memory CRC, or an on-disk CRC) to the target snapshot version on load. A CRC already at the +/// target version is always used regardless of this setting; this only bounds the cost of +/// advancing a *stale* CRC. +/// +/// A resolved CRC gives the snapshot precomputed file statistics (file count and sizes, useful +/// for query optimization and for writers producing a post-commit CRC) along with domain metadata +/// and set transactions (useful for writers), all without extra log replay. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub enum IncrementalReplay { + /// Never advance a stale CRC; fall back to normal log replay. `UpToCommits(0)` is equivalent. + #[default] + Disabled, + /// Advance only when the CRC is within `n` commits of the target version, i.e. + /// `target_version - crc_version <= n`. + UpToCommits(u64), + /// Advance regardless of how stale the CRC is. + Unlimited, +} + +impl IncrementalReplay { + /// Whether the configured budget permits advancing a CRC at `crc_version` to `target_version`. + /// Errors if `crc_version` is ahead of `target_version`, which violates a caller invariant. + /// + /// Example: 95.crc with commits 96.json through 100.json is 5 commits, so `UpToCommits(5)` + /// advances and `UpToCommits(4)` does not; `Unlimited` always advances. + pub(crate) fn should_advance( + self, + crc_version: Version, + target_version: Version, + ) -> DeltaResult { + let distance = target_version.checked_sub(crc_version).ok_or_else(|| { + Error::internal_error(format!( + "CRC version {crc_version} is ahead of target version {target_version}" + )) + })?; + Ok(match self { + IncrementalReplay::Disabled => false, + IncrementalReplay::UpToCommits(n) => distance <= n, + IncrementalReplay::Unlimited => true, + }) + } } impl SnapshotBuilder { + // ============================================================================ + // Constructors + // ============================================================================ + pub(crate) fn new_for(table_root: impl AsRef) -> Self { Self { table_root: Some(table_root.as_ref().to_string()), existing_snapshot: None, version: None, log_tail: Vec::new(), + max_catalog_version: None, + incremental_replay: IncrementalReplay::default(), + operation_id: MetricId::new(), + correlation_id: None, } } @@ -54,9 +118,17 @@ impl SnapshotBuilder { existing_snapshot: Some(existing_snapshot), version: None, log_tail: Vec::new(), + max_catalog_version: None, + incremental_replay: IncrementalReplay::default(), + operation_id: MetricId::new(), + correlation_id: None, } } + // ============================================================================ + // Chainable configuration + // ============================================================================ + /// Set the target version of the [`Snapshot`]. When omitted, the Snapshot is created at the /// latest version of the table. pub fn at_version(mut self, version: Version) -> Self { @@ -68,71 +140,285 @@ impl SnapshotBuilder { /// systems to provide an up-to-date log tail when used to build a snapshot. /// /// Note that the log tail must be a contiguous sequence of commits from M..=N where N is the - /// latest version of the table and 0 <= M <= N. - #[cfg(feature = "catalog-managed")] + /// target version of the snapshot and 0 <= M <= N. + /// + /// See [`with_max_catalog_version`] for additional constraints when loading catalog-managed + /// tables. + /// + /// [`with_max_catalog_version`]: Self::with_max_catalog_version pub fn with_log_tail(mut self, log_tail: Vec) -> Self { self.log_tail = log_tail; self } + /// Set the maximum catalog-ratified version. When set, the snapshot will not load versions + /// beyond this limit, even if later commits exist on the filesystem. This ensures the catalog + /// remains the source of truth for catalog-managed tables. + /// + /// When no explicit time-travel version is set via [`at_version`], `max_catalog_version` is + /// used as the effective target version. When time-travelling to an explicit version, + /// `max_catalog_version` must still be set for catalog-managed tables -- the requested version + /// must not exceed it. + /// + /// # Log tail requirements + /// + /// When `max_catalog_version` is set and no time-travel version is specified, the last entry in + /// the log tail must match `max_catalog_version` exactly. When time-travelling, the last log + /// tail entry must be >= the requested version. + /// + /// [`at_version`]: Self::at_version + pub fn with_max_catalog_version(mut self, max_catalog_version: Version) -> Self { + self.max_catalog_version = Some(max_catalog_version); + self + } + + /// Bound how many commits kernel will replay to advance a stale CRC to the target version. + /// See [`IncrementalReplay`]. Defaults to [`IncrementalReplay::Disabled`]. + /// + /// Writers should set this to [`IncrementalReplay::Unlimited`] for faster writes, as should + /// readers that always want table-level file statistics for query optimization. + /// + /// Applies to both fresh and incremental builds. + pub fn with_incremental_crc_replay(mut self, mode: IncrementalReplay) -> Self { + self.incremental_replay = mode; + self + } + + /// Attach an opaque, caller-supplied correlation id for joining this build's metric events to + /// the caller's own request or operation id. An empty id is treated as unset. When unset, + /// behavior is unchanged. + pub fn with_correlation_id(mut self, correlation_id: impl Into>) -> Self { + self.correlation_id = Some(correlation_id.into()).filter(|id| !id.is_empty()); + self + } + + // ============================================================================ + // Terminal: build the Snapshot + // ============================================================================ + /// Create a new [`Snapshot`]. This returns a [`SnapshotRef`] (`Arc`), perhaps /// returning a reference to an existing snapshot if the request to build a new snapshot /// matches the version of an existing snapshot. /// + /// Reports metrics: [`MetricEvent::SnapshotBuildSuccess`] or + /// [`MetricEvent::SnapshotBuildFailure`]. + /// /// # Parameters /// /// - `engine`: Implementation of [`Engine`] apis. + /// + /// [`MetricEvent::SnapshotBuildSuccess`]: crate::metrics::MetricEvent::SnapshotBuildSuccess + /// [`MetricEvent::SnapshotBuildFailure`]: crate::metrics::MetricEvent::SnapshotBuildFailure + // `is_catalog_managed` is the requested load mode, not the confirmed protocol + // (see `IS_CATALOG_MANAGED_FIELD`). #[instrument( - name = "snap.build", + name = SNAPSHOT_COMPLETED_SPAN, skip_all, - fields(path = %self.table_path()), + fields(path = %self.table_path(), report, version = tracing::field::Empty, operation_id = %self.operation_id, is_catalog_managed = self.max_catalog_version.is_some(), correlation_id = self.correlation_id.as_deref().unwrap_or(""), load_type = self.load_type().as_ref()), err )] pub fn build(self, engine: &dyn Engine) -> DeltaResult { + // Fold the context into the message string rather than passing structured fields: this + // `info!` fires inside the `snap.build` metrics span, where any field the + // `SnapshotBuildSuccess` event doesn't recognize would trip a spurious "Invalid field" + // warning from the metrics layer. info!( - target = self.target_version_str(), - from_version = ?self.existing_snapshot.as_ref().map(|s| s.version()), - log_tail_len = self.log_tail.len(), - "building snapshot" + "building snapshot: target={}, from_version={:?}, log_tail_len={}, \ + max_catalog_version={:?}", + self.target_version_str(), + self.existing_snapshot.as_ref().map(|s| s.version()), + self.log_tail.len(), + self.max_catalog_version ); - let log_tail = self.log_tail.into_iter().map(Into::into).collect(); - let operation_id = MetricId::new(); - let reporter = engine.get_metrics_reporter(); - - if let Some(table_root) = self.table_root { - let table_url = try_parse_uri(table_root)?; - let log_segment = LogSegment::for_snapshot( - engine.storage_handler().as_ref(), - table_url.join("_delta_log/")?, - log_tail, - self.version, - reporter.as_ref(), - Some(operation_id), - )?; - - Ok(Snapshot::try_new_from_log_segment( - table_url, - log_segment, - engine, - Some(operation_id), - )? - .into()) - } else { - let existing_snapshot = self.existing_snapshot.ok_or_else(|| { - Error::internal_error( - "SnapshotBuilder should have either table_root or existing_snapshot", + let load_type = self.load_type(); + + // Destructure self so fields can be moved independently + let Self { + table_root, + existing_snapshot, + version, + log_tail, + max_catalog_version, + incremental_replay, + operation_id, + correlation_id, + } = self; + + let metric_context = SnapshotLoadMetricContext { + operation_id, + is_catalog_managed: max_catalog_version.is_some(), + correlation_id, + load_type, + }; + + let log_tail: Vec<_> = log_tail.into_iter().map(Into::into).collect(); + + // Pre-build validations for catalog-managed tables + Self::validate_catalog_managed_build_inputs(version, max_catalog_version, &log_tail)?; + + // Use time-travel version if set, otherwise fall back to max_catalog_version. Passing this + // as the version to LogSegment::for_snapshot does NOT skip the _last_checkpoint hint -- + // the hint is still used when its version <= effective_version. + let effective_version = version.or(max_catalog_version); + + // A snapshot is latest when no explicit time-travel version is requested, or when the + // requested version is exactly the max_catalog_version. + let built_as_latest = version.is_none() || version == max_catalog_version; + + let result = if let Some(table_root) = table_root { + try_parse_uri(table_root).and_then(|table_url| { + let log_segment = LogSegment::for_snapshot( + engine.storage_handler().as_ref(), + table_url.join("_delta_log/")?, + log_tail, + effective_version, + metric_context.clone(), + )?; + Snapshot::try_new_from_log_segment( + table_url, + log_segment, + engine, + metric_context, + incremental_replay, + built_as_latest, ) - })?; - - Snapshot::try_new_from( - existing_snapshot, - log_tail, - engine, - self.version, - Some(operation_id), + .map(Into::into) + }) + } else { + existing_snapshot + .ok_or_else(|| { + Error::internal_error( + "SnapshotBuilder should have either table_root or existing_snapshot", + ) + }) + .and_then(|existing_snapshot| { + Snapshot::try_new_from( + existing_snapshot, + log_tail, + engine, + effective_version, + metric_context, + incremental_replay, + built_as_latest, + ) + }) + }; + + // Post-build validations for catalog-managed tables + let result = result.and_then(|snapshot| { + Self::validate_catalog_managed_build_result(&snapshot, max_catalog_version)?; + Ok(snapshot) + }); + if let Ok(ref snapshot) = result { + tracing::Span::current().record("version", snapshot.version()); + } + result + } + + // ============================================================================ + // Helpers + // ============================================================================ + + // ===== Catalog-managed Validations ===== + + /// Pre-build validations for catalog-managed table invariants. + fn validate_catalog_managed_build_inputs( + version: Option, + max_catalog_version: Option, + log_tail: &[crate::path::ParsedLogPath], + ) -> DeltaResult<()> { + // Log tail must be sorted ascending and contiguous (no gaps or duplicates) + for pair in log_tail.windows(2) { + require!( + pair[0].version + 1 == pair[1].version, + Error::LogTailVersionsNotContiguous { + first_version: pair[0].version, + second_version: pair[1].version, + } + ); + } + + // TODO: If inline commits (or any other catalog commits) are ever supported, change this + // method to check if there are any catalog commits. + let has_catalog_commits = log_tail + .iter() + .any(|p| p.file_type == LogPathFileType::StagedCommit); + + // Staged commits require max_catalog_version + require!( + !has_catalog_commits || max_catalog_version.is_some(), + Error::MaxCatalogVersion( + "Max catalog version is required when providing staged commits in the log tail. \ + Use with_max_catalog_version()." + .to_string() ) + ); + + // Time-travel version must not exceed max_catalog_version + if let (Some(ver), Some(max_cv)) = (version, max_catalog_version) { + require!( + ver <= max_cv, + Error::MaxCatalogVersion(format!( + "Requested version {ver} exceeds max catalog version {max_cv}" + )) + ); } + + // Log tail end version validation when max_catalog_version is set + if let (Some(max_cv), Some(last)) = (max_catalog_version, log_tail.last()) { + if let Some(ver) = version { + // With time-travel: last log_tail entry must be >= requested version + require!( + last.version >= ver, + Error::MaxCatalogVersion(format!( + "Log tail version {} is less than requested version {ver} for max catalog \ + version {max_cv}", + last.version + )) + ); + } else { + // Without time-travel: last log_tail entry must == max_catalog_version + require!( + last.version == max_cv, + Error::MaxCatalogVersion(format!( + "Log tail version {} does not match max catalog version {max_cv}", + last.version + )) + ); + } + } + + Ok(()) + } + + /// Post-build validation: catalog-managed tables must have max_catalog_version, and + /// non-catalog-managed tables must not. + fn validate_catalog_managed_build_result( + snapshot: &SnapshotRef, + max_catalog_version: Option, + ) -> DeltaResult<()> { + let is_catalog_managed = snapshot.table_configuration().is_catalog_managed(); + + require!( + !is_catalog_managed || max_catalog_version.is_some(), + Error::MaxCatalogVersion( + "Max catalog version is required when loading a catalog-managed table. \ + Use with_max_catalog_version()." + .to_string() + ) + ); + if let Some(max_catalog_version) = max_catalog_version { + require!( + is_catalog_managed, + Error::MaxCatalogVersion(format!( + "Max catalog version {max_catalog_version} must not be set for a \ + non-catalog-managed table" + )) + ); + } + + Ok(()) } // ===== Instrumentation Helpers ===== @@ -148,7 +434,24 @@ impl SnapshotBuilder { .unwrap_or("unknown") } + /// A build from a table root is a fresh, full log listing; a build from an existing snapshot + /// reuses that snapshot's log root and lists only the commits above it (`table_root` is None). + fn load_type(&self) -> LogSegmentLoadType { + if self.table_root.is_some() { + LogSegmentLoadType::Full + } else { + LogSegmentLoadType::Incremental + } + } + fn target_version_str(&self) -> String { + if let Some(mcv) = self.max_catalog_version { + return match self.version { + Some(v) => format!("{v} (max_catalog_version={mcv})"), + None => format!("{mcv} (max_catalog_version)"), + }; + } + self.version .map(|v| v.to_string()) .unwrap_or_else(|| "LATEST".into()) @@ -158,93 +461,46 @@ impl SnapshotBuilder { #[cfg(test)] mod tests { use std::sync::Arc; + use std::time::Duration; - use crate::engine::default::{ - executor::tokio::TokioBackgroundExecutor, DefaultEngine, DefaultEngineBuilder, - }; - use crate::object_store::memory::InMemory; - use crate::object_store::path::Path; - use crate::object_store::{DynObjectStore, ObjectStore as _}; use itertools::Itertools; use serde_json::json; + use test_utils::{actions_to_string, add_commit, TestAction}; use super::*; + use crate::engine::sync::SyncEngine; + use crate::metrics::MetricEvent; + use crate::object_store::memory::InMemory; + use crate::object_store::path::Path; + use crate::object_store::{DynObjectStore, ObjectStoreExt as _}; + use crate::unit_test_utils::{install_thread_local_metrics_reporter, CapturingReporter}; + use crate::utils::FoldWithOption as _; - fn setup_test() -> ( - Arc>, - Arc, - String, - ) { + fn setup_test() -> (Arc, Arc, String) { let table_root = String::from("memory:///"); let store = Arc::new(InMemory::new()); - let engine = Arc::new(DefaultEngineBuilder::new(store.clone()).build()); + let engine = Arc::new(SyncEngine::new_with_store(store.clone())); (engine, store, table_root) } - // TODO (#1990): update this function to properly store the table at table_root - async fn create_table(store: &Arc, _table_root: String) -> DeltaResult<()> { - let protocol = json!({ - "minReaderVersion": 3, - "minWriterVersion": 7, - "readerFeatures": ["catalogManaged"], - "writerFeatures": ["catalogManaged"], - }); - - let metadata = json!({ - "id": "test-table-id", - "format": { - "provider": "parquet", - "options": {} - }, - "schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"val\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}", - "partitionColumns": [], - "configuration": {}, - "createdTime": 1587968585495i64 - }); - - // Create commit 0 with protocol and metadata - let commit0 = [ - json!({ - "protocol": protocol - }), - json!({ - "metaData": metadata - }), - ]; - - // Write commit 0 - let commit0_data = commit0 - .iter() - .map(ToString::to_string) - .collect_vec() - .join("\n"); - - let path = Path::from(format!("_delta_log/{:020}.json", 0).as_str()); - store.put(&path, commit0_data.into()).await?; - - // Create commit 1 with a single addFile action - let commit1 = [json!({ - "add": { - "path": "part-00000-test.parquet", - "partitionValues": {}, - "size": 1024, - "modificationTime": 1587968586000i64, - "dataChange": true, - "stats": null, - "tags": null - } - })]; - - // Write commit 1 - let commit1_data = commit1 - .iter() - .map(ToString::to_string) - .collect_vec() - .join("\n"); - - let path = Path::from(format!("_delta_log/{:020}.json", 1).as_str()); - store.put(&path, commit1_data.into()).await?; - + async fn create_table( + store: &Arc, + table_root: &str, + ) -> Result<(), Box> { + add_commit( + table_root, + store.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + add_commit( + table_root, + store.as_ref(), + 1, + actions_to_string(vec![TestAction::Add("part-00000-test.parquet".into())]), + ) + .await?; Ok(()) } @@ -252,7 +508,7 @@ mod tests { async fn test_snapshot_builder() -> Result<(), Box> { let (engine, store, table_root) = setup_test(); let engine = engine.as_ref(); - create_table(&store, table_root.clone()).await?; + create_table(&store, &table_root).await?; let snapshot = SnapshotBuilder::new_for(table_root.clone()).build(engine)?; assert_eq!(snapshot.version(), 1); @@ -282,7 +538,7 @@ mod tests { "provider": "parquet", "options": {} }, - "schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"interval_col\",\"type\":\"interval second\",\"nullable\":true,\"metadata\":{}}]}", + "schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"interval_col\",\"type\":\"interval year to second\",\"nullable\":true,\"metadata\":{}}]}", "partitionColumns": [], "configuration": {}, "createdTime": 1587968585495i64 @@ -313,10 +569,508 @@ mod tests { let err = result.unwrap_err(); let err_msg = err.to_string(); assert!( - err_msg.contains("Unsupported Delta table type: 'interval second'"), + err_msg.contains("Unsupported Delta table type: 'interval year to second'"), "Expected clear error message about unsupported type, got: {err_msg}" ); Ok(()) } + + fn measuring_reporter() -> (Arc, tracing::subscriber::DefaultGuard) { + let reporter = Arc::new(CapturingReporter::default()); + let guard = install_thread_local_metrics_reporter(reporter.clone()); + (reporter, guard) + } + + #[test_log::test(tokio::test)] + async fn snapshot_failed_emits_metric_on_error() -> Result<(), Box> { + let (engine, store, table_root) = setup_test(); + + // Unsupported schema type forces a build failure + let protocol = json!({"minReaderVersion": 1, "minWriterVersion": 2}); + let metadata = json!({ + "id": "test-table-id", + "format": {"provider": "parquet", "options": {}}, + "schemaString": r#"{"type":"struct","fields":[{"name":"id","type":"interval year to second","nullable":true,"metadata":{}}]}"#, + "partitionColumns": [], + "configuration": {}, + "createdTime": 1587968585495i64 + }); + let commit0_data = [json!({"protocol": protocol}), json!({"metaData": metadata})] + .iter() + .map(ToString::to_string) + .collect_vec() + .join("\n"); + store + .put( + &Path::from("_delta_log/00000000000000000000.json"), + commit0_data.into(), + ) + .await?; + + let (reporter, _guard) = measuring_reporter(); + let result = SnapshotBuilder::new_for(table_root).build(engine.as_ref()); + assert!(result.is_err()); + + let events = reporter.events(); + assert!( + events + .iter() + .any(|e| matches!(e, MetricEvent::SnapshotBuildFailure(_))), + "expected SnapshotBuildFailure event on build failure" + ); + assert!( + !events + .iter() + .any(|e| matches!(e, MetricEvent::SnapshotBuildSuccess(_))), + "should not emit SnapshotBuildSuccess on failure" + ); + Ok(()) + } + + #[test_log::test(tokio::test)] + async fn log_segment_load_failure_emits_metric_on_empty_log( + ) -> Result<(), Box> { + let (engine, _store, table_root) = setup_test(); + let (reporter, _guard) = measuring_reporter(); + + assert!(SnapshotBuilder::new_for(table_root) + .build(engine.as_ref()) + .is_err()); + + let events = reporter.events(); + let failure = events + .iter() + .find_map(|e| match e { + MetricEvent::LogSegmentLoadFailure(f) => Some(f), + _ => None, + }) + .expect("expected LogSegmentLoadFailure when the log has no commits"); + assert_eq!(failure.load_type, LogSegmentLoadType::Full); + Ok(()) + } + + #[test_log::test(tokio::test)] + async fn protocol_metadata_load_failure_emits_metric_when_actions_absent( + ) -> Result<(), Box> { + let (engine, store, table_root) = setup_test(); + // A commit with no protocol/metadata: the segment lists fine, then the read fails. + add_commit( + &table_root, + store.as_ref(), + 0, + actions_to_string(vec![TestAction::Add("part-00000-test.parquet".into())]), + ) + .await?; + let (reporter, _guard) = measuring_reporter(); + + assert!(SnapshotBuilder::new_for(table_root) + .build(engine.as_ref()) + .is_err()); + + let events = reporter.events(); + assert!( + events + .iter() + .any(|e| matches!(e, MetricEvent::ProtocolMetadataLoadFailure(_))), + "expected ProtocolMetadataLoadFailure when protocol/metadata are absent" + ); + assert!( + !events + .iter() + .any(|e| matches!(e, MetricEvent::ProtocolMetadataLoadSuccess(_))), + "must not emit ProtocolMetadataLoadSuccess when the load fails" + ); + Ok(()) + } + + #[test_log::test(tokio::test)] + async fn snapshot_update_from_existing_emits_metric() -> Result<(), Box> + { + let (engine, store, table_root) = setup_test(); + create_table(&store, &table_root).await?; + + // Build v0 snapshot before installing the reporter so only the update is measured + let snap_v0 = SnapshotBuilder::new_for(table_root) + .at_version(0) + .build(engine.as_ref())?; + assert_eq!(snap_v0.version(), 0); + + let (reporter, _guard) = measuring_reporter(); + + let snap_v1 = SnapshotBuilder::new_from(snap_v0).build(engine.as_ref())?; + assert_eq!(snap_v1.version(), 1); + + let events = reporter.events(); + let (version, duration) = events + .iter() + .find_map(|e| match e { + MetricEvent::SnapshotBuildSuccess(s) => Some((s.version, s.duration)), + _ => None, + }) + .expect("expected SnapshotBuildSuccess event"); + assert_eq!(version, 1, "version should match the updated snapshot"); + assert!(duration > Duration::ZERO, "duration should be non-zero"); + Ok(()) + } + + #[test_log::test(tokio::test)] + async fn snapshot_update_to_earlier_version_emits_failed_metric( + ) -> Result<(), Box> { + let (engine, store, table_root) = setup_test(); + create_table(&store, &table_root).await?; + + // Build v1 snapshot before installing the reporter + let snap_v1 = SnapshotBuilder::new_for(table_root).build(engine.as_ref())?; + assert_eq!(snap_v1.version(), 1); + + let (reporter, _guard) = measuring_reporter(); + + let result = SnapshotBuilder::new_from(snap_v1) + .at_version(0) + .build(engine.as_ref()); + assert!( + result.is_err(), + "updating to an earlier version should fail" + ); + + let events = reporter.events(); + assert!( + events + .iter() + .any(|e| matches!(e, MetricEvent::SnapshotBuildFailure(_))), + "expected SnapshotBuildFailure when version update goes backwards" + ); + assert!( + !events + .iter() + .any(|e| matches!(e, MetricEvent::SnapshotBuildSuccess(_))), + "should not emit SnapshotBuildSuccess when version update fails" + ); + Ok(()) + } + + #[test_log::test(tokio::test)] + async fn snapshot_completed_duration_exceeds_log_segment_load_duration( + ) -> Result<(), Box> { + let (engine, store, table_root) = setup_test(); + create_table(&store, &table_root).await?; + + let (reporter, _guard) = measuring_reporter(); + let _snap = SnapshotBuilder::new_for(table_root).build(engine.as_ref())?; + + let events = reporter.events(); + let snap_duration = events + .iter() + .find_map(|e| match e { + MetricEvent::SnapshotBuildSuccess(s) => Some(s.duration), + _ => None, + }) + .expect("expected SnapshotBuildSuccess event"); + let segment_duration = events + .iter() + .find_map(|e| match e { + MetricEvent::LogSegmentLoadSuccess(s) => Some(s.duration), + _ => None, + }) + .expect("expected LogSegmentLoadSuccess event"); + + assert!( + snap_duration > Duration::ZERO, + "duration should be non-zero" + ); + assert!( + snap_duration >= segment_duration, + "SnapshotBuildSuccess.duration ({snap_duration:?}) should be >= LogSegmentLoadSuccess.duration ({segment_duration:?})" + ); + Ok(()) + } + + #[rstest::rstest] + #[case::with_id(Some("req-abc-123"))] + #[case::without_id(None)] + #[test_log::test(tokio::test)] + async fn snapshot_build_and_child_events_carry_correlation_id( + #[case] correlation_id: Option<&str>, + ) -> Result<(), Box> { + let (engine, store, table_root) = setup_test(); + create_table(&store, &table_root).await?; + + let (reporter, _guard) = measuring_reporter(); + let _ = SnapshotBuilder::new_for(table_root) + .fold_with(correlation_id, SnapshotBuilder::with_correlation_id) + .build(engine.as_ref())?; + + // The build event and its snapshot-load child events must all carry the id, since they + // ride the same SnapshotLoadMetricContext. + let events = reporter.events(); + let id_of = |pick: fn(&MetricEvent) -> Option<&Option>>| { + events + .iter() + .find_map(pick) + .expect("expected event") + .as_deref() + .map(str::to_string) + }; + let build_id = id_of(|e| match e { + MetricEvent::SnapshotBuildSuccess(s) => Some(&s.correlation_id), + _ => None, + }); + let segment_id = id_of(|e| match e { + MetricEvent::LogSegmentLoadSuccess(s) => Some(&s.correlation_id), + _ => None, + }); + let metadata_id = id_of(|e| match e { + MetricEvent::ProtocolMetadataLoadSuccess(s) => Some(&s.correlation_id), + _ => None, + }); + let expected = correlation_id.map(str::to_string); + assert_eq!(build_id, expected); + assert_eq!(segment_id, expected, "log-segment child must carry the id"); + assert_eq!( + metadata_id, expected, + "protocol/metadata child must carry the id" + ); + Ok(()) + } + + mod catalog_managed_tests { + use test_utils::{ + actions_to_string, actions_to_string_catalog_managed, add_commit, add_staged_commit, + TestAction, + }; + + use super::*; + use crate::log_path::LogPath; + use crate::utils::try_parse_uri; + use crate::FileMeta; + + fn create_log_path(table_root: &str, commit_path: Path) -> LogPath { + let table_url = try_parse_uri(table_root).expect("Failed to parse table root"); + let commit_url = table_url.join(commit_path.as_ref()).unwrap(); + let file_meta = FileMeta { + location: commit_url, + last_modified: 123, + size: 100, + }; + LogPath::try_new(file_meta).expect("Failed to create LogPath") + } + + /// Creates an in-memory engine, store, and table root with an initial catalog-managed + /// commit at version 0 (protocol + metadata). + async fn setup_catalog_managed_test() -> (Arc, Arc, String) { + let (engine, store, table_root) = setup_test(); + let actions = vec![TestAction::Metadata]; + add_commit( + &table_root, + store.as_ref(), + 0, + actions_to_string_catalog_managed(actions), + ) + .await + .expect("Failed to write initial catalog-managed commit"); + (engine, store, table_root) + } + + #[test_log::test(tokio::test)] + async fn test_staged_commits_without_max_catalog_version_errors( + ) -> Result<(), Box> { + let (engine, store, table_root) = setup_catalog_managed_test().await; + let path1 = + add_staged_commit(&table_root, store.as_ref(), 1, String::from("{}")).await?; + + let log_tail = vec![create_log_path(&table_root, path1)]; + + let result = SnapshotBuilder::new_for(table_root) + .with_log_tail(log_tail) + .build(engine.as_ref()); + + assert!(matches!(result, Err(Error::MaxCatalogVersion(_)))); + + Ok(()) + } + + #[test_log::test(tokio::test)] + async fn test_version_exceeds_max_catalog_version_errors( + ) -> Result<(), Box> { + let (engine, _store, table_root) = setup_catalog_managed_test().await; + + let result = SnapshotBuilder::new_for(table_root) + .at_version(5) + .with_max_catalog_version(3) + .build(engine.as_ref()); + + assert!(matches!(result, Err(Error::MaxCatalogVersion(_)))); + + Ok(()) + } + + #[test_log::test(tokio::test)] + async fn test_log_tail_last_version_mismatch_errors( + ) -> Result<(), Box> { + let (engine, store, table_root) = setup_catalog_managed_test().await; + let actions = vec![TestAction::Add("file_1.parquet".to_string())]; + add_commit(&table_root, store.as_ref(), 1, actions_to_string(actions)).await?; + let actions = vec![TestAction::Add("file_2.parquet".to_string())]; + add_commit(&table_root, store.as_ref(), 2, actions_to_string(actions)).await?; + + let log_tail = vec![ + create_log_path(&table_root, test_utils::delta_path_for_version(1, "json")), + create_log_path(&table_root, test_utils::delta_path_for_version(2, "json")), + ]; + + // log_tail ends at v2, max_catalog_version=3, no time-travel -> error + let result = SnapshotBuilder::new_for(table_root) + .with_log_tail(log_tail) + .with_max_catalog_version(3) + .build(engine.as_ref()); + + assert!(matches!(result, Err(Error::MaxCatalogVersion(_)))); + + Ok(()) + } + + #[test_log::test(tokio::test)] + async fn test_catalog_managed_table_without_max_catalog_version_errors( + ) -> Result<(), Box> { + let (engine, _store, table_root) = setup_catalog_managed_test().await; + + let result = SnapshotBuilder::new_for(table_root).build(engine.as_ref()); + + assert!(matches!(result, Err(Error::MaxCatalogVersion(_)))); + + Ok(()) + } + + #[test_log::test(tokio::test)] + async fn test_non_catalog_managed_table_with_max_catalog_version_errors( + ) -> Result<(), Box> { + let (engine, store, table_root) = setup_test(); + + let actions = vec![TestAction::Metadata]; + add_commit(&table_root, store.as_ref(), 0, actions_to_string(actions)).await?; + + let result = SnapshotBuilder::new_for(table_root) + .with_max_catalog_version(0) + .build(engine.as_ref()); + + assert!(matches!(result, Err(Error::MaxCatalogVersion(_)))); + + Ok(()) + } + + #[test_log::test(tokio::test)] + async fn test_log_tail_last_version_less_than_time_travel_version_errors( + ) -> Result<(), Box> { + let (engine, store, table_root) = setup_catalog_managed_test().await; + let actions = vec![TestAction::Add("file_1.parquet".to_string())]; + add_commit(&table_root, store.as_ref(), 1, actions_to_string(actions)).await?; + + let log_tail = vec![create_log_path( + &table_root, + test_utils::delta_path_for_version(1, "json"), + )]; + + // Time travel to v2, but log tail only goes up to v1 + let result = SnapshotBuilder::new_for(table_root) + .at_version(2) + .with_log_tail(log_tail) + .with_max_catalog_version(3) + .build(engine.as_ref()); + + assert!(matches!(result, Err(Error::MaxCatalogVersion(_)))); + + Ok(()) + } + + #[test_log::test(tokio::test)] + async fn test_max_catalog_version_as_effective_version( + ) -> Result<(), Box> { + let (engine, store, table_root) = setup_catalog_managed_test().await; + let actions = vec![TestAction::Add("file_1.parquet".to_string())]; + add_commit(&table_root, store.as_ref(), 1, actions_to_string(actions)).await?; + let actions = vec![TestAction::Add("file_2.parquet".to_string())]; + add_commit(&table_root, store.as_ref(), 2, actions_to_string(actions)).await?; + + // max_catalog_version=1, no time-travel -> snapshot at v1 + let snapshot = SnapshotBuilder::new_for(table_root) + .with_max_catalog_version(1) + .build(engine.as_ref())?; + assert_eq!(snapshot.version(), 1); + + Ok(()) + } + + #[test_log::test(tokio::test)] + async fn test_time_travel_with_max_catalog_version( + ) -> Result<(), Box> { + let (engine, store, table_root) = setup_catalog_managed_test().await; + let actions = vec![TestAction::Add("file_1.parquet".to_string())]; + add_commit(&table_root, store.as_ref(), 1, actions_to_string(actions)).await?; + + // at_version(0) + max_catalog_version=1 -> snapshot at v0 + let snapshot = SnapshotBuilder::new_for(table_root) + .at_version(0) + .with_max_catalog_version(1) + .build(engine.as_ref())?; + assert_eq!(snapshot.version(), 0); + + Ok(()) + } + + #[test_log::test(tokio::test)] + async fn test_builder_from_catalog_managed_without_mcv_errors( + ) -> Result<(), Box> { + let (engine, store, table_root) = setup_catalog_managed_test().await; + let actions = vec![TestAction::Add("file_1.parquet".to_string())]; + add_commit(&table_root, store.as_ref(), 1, actions_to_string(actions)).await?; + + let initial = SnapshotBuilder::new_for(table_root) + .with_max_catalog_version(1) + .build(engine.as_ref())?; + + // Incremental update without mcv should fail + let result = SnapshotBuilder::new_from(initial).build(engine.as_ref()); + + assert!(matches!(result, Err(Error::MaxCatalogVersion(_)))); + + Ok(()) + } + + #[rstest::rstest] + #[case::gap(vec![1, 3], vec![1, 3], 3)] + #[case::duplicates(vec![1], vec![1, 1], 1)] + #[case::unsorted(vec![1, 2], vec![2, 1], 2)] + #[test_log::test(tokio::test)] + async fn test_non_contiguous_log_tail_errors( + #[case] commit_versions: Vec, + #[case] log_tail_versions: Vec, + #[case] mcv: u64, + ) -> Result<(), Box> { + let (engine, store, table_root) = setup_catalog_managed_test().await; + for v in &commit_versions { + let actions = vec![TestAction::Add(format!("file_{v}.parquet"))]; + add_commit(&table_root, store.as_ref(), *v, actions_to_string(actions)).await?; + } + + let log_tail: Vec<_> = log_tail_versions + .iter() + .map(|v| { + create_log_path(&table_root, test_utils::delta_path_for_version(*v, "json")) + }) + .collect(); + + let result = SnapshotBuilder::new_for(table_root) + .with_log_tail(log_tail) + .with_max_catalog_version(mcv) + .build(engine.as_ref()); + + assert!(matches!( + result, + Err(Error::LogTailVersionsNotContiguous { .. }) + )); + + Ok(()) + } + } } diff --git a/kernel/src/snapshot/incremental.rs b/kernel/src/snapshot/incremental.rs new file mode 100644 index 0000000000..e12031c8c7 --- /dev/null +++ b/kernel/src/snapshot/incremental.rs @@ -0,0 +1,2259 @@ +//! Incremental [`Snapshot`] update. +//! +//! Public API: `Snapshot::builder_from(existing).build(engine)`. The case-by-case +//! behavior lives in [`Snapshot::try_new_from`]. + +use std::sync::Arc; + +use tracing::instrument; + +use super::{IncrementalReplay, Snapshot}; +use crate::log_segment::LogSegment; +use crate::log_segment_files::LogSegmentFiles; +use crate::metrics::{ + emit_log_segment_load, emit_log_segment_load_failure, emit_protocol_metadata_load, + emit_protocol_metadata_load_failure, SnapshotLoadMetricContext, +}; +use crate::path::ParsedLogPath; +use crate::table_configuration::TableConfiguration; +use crate::{DeltaResult, Engine, Error, Version}; + +/// The assembled outcome of the listing phase of an incremental update. Listing/assembly +/// failures surface as `Err` from [`Snapshot::build_new_segment`], not a variant here. +/// +/// See [`Snapshot::try_new_from`] for the case taxonomy these outcomes map onto. +enum NewSegment { + /// No new segment to build; the caller returns the existing snapshot unchanged. + Unchanged, + /// A checkpoint ahead of the existing snapshot requires a full rebuild from it. + Rebuild(LogSegment), + /// New commits merged into the existing segment; ready for incremental P&M. + Combined(LogSegment), +} + +impl Snapshot { + // ============================================================================ + // Public entry + // ============================================================================ + + /// Incrementally update an existing [`Snapshot`] to a more recent version. Variables used + /// below: + /// + /// - `T` = effective target version: the time-travel version from + /// [`SnapshotBuilder::at_version`] when set, otherwise the `max_catalog_version` from + /// [`SnapshotBuilder::with_max_catalog_version`] for catalog-managed tables, otherwise unset + /// (defaults to latest). + /// - `S1` = existing snapshot version. + /// - `S2` = new snapshot version (= end version of the new listing). When `T` is set, the + /// listing is upper-bounded by `T`, so `S2 == T`. + /// - `C1` = existing checkpoint version (if any). + /// - `C2` = new checkpoint version (if found in the listing). Always `<= S2`, since `C2` is by + /// definition a checkpoint observed within the listing. By transitivity, `C2 <= T` when `T` + /// is set. + /// + /// The log listing is catalog-log-tail aware: any `log_tail` provided to the builder is + /// merged with filesystem listings. + /// + /// Position layout per case (`....` is the version axis; `====` marks the range read for + /// P+M replay; `listed` is the listing range; `read` is what's read for P+M): + /// + /// ```text + /// positions listed read outcome + /// Case A: C1 ........ S1 (= T) - - return existing + /// Case B: C1 .. T .. S1 - - error (T < S1) + /// Case C.1: C1 .. S1 .. T empty - error (T unreachable) + /// Case C.2: C1 .. S1 empty - return existing + /// Case D.1: C1 .. S1 .. C2 ===== S2 [C1+1, S2] [C2, S2] rebuild from C2 + /// Case D.2: C1 . C2 .. S1 ====== S2 [C1+1, S2] (S1, S2] advance base, -> F + /// Case E: C1 .. S1 (= S2) [C1+1, S1] - return existing + /// Case F: C1 .. S1 ====== S2 [C1+1, S2] (S1, S2] incremental update + /// ``` + /// + /// In the incremental cases (D.2, F), the existing snapshot's P+M at `S1` is the baseline, and + /// only commits in `(S1, S2]` are replayed for newer P+M on top of it. A base CRC newer than + /// `S1`, when present, serves as the baseline instead. + /// + /// - **A.** `T == S1`: return the existing snapshot unchanged. + /// - **B.** `T < S1`: error. The incremental path only moves forward. + /// - Otherwise (`T` unset or `T > S1`), list the log from `C1+1` (or from version 1 if there is + /// no existing checkpoint), and one of the following applies: + /// - **C.** Listing is empty: + /// - **C.1.** `T` is set: error (target is newer than anything in the log). + /// - **C.2.** `T` is unset: return the existing snapshot. + /// - **D.** Listing contains a checkpoint: + /// - **D.1.** `C2 > S1`: the new checkpoint at `C2` already captures the table state + /// through version `C2`, including changes in `(S1, C2]`, so we can use it as the new + /// base instead of replaying those commits. Build a fresh snapshot from `C2`. + /// - **D.2.** `C2 <= S1`: the existing snapshot's P+M (at `S1`) already reflects everything + /// `C2` would tell us, so we skip reading `C2` for P+M. Fall through to case F to replay + /// only commits `> S1` for P+M; the combined segment uses `C2` as its checkpoint base, + /// producing a better log segment with fewer deltas above the checkpoint (e.g. faster + /// distributed log replay). + /// - **E.** Listing contains commits but no new checkpoint, and `S2 == S1`: return the + /// existing snapshot. + /// - **F.** Listing contains new commits (no new checkpoint, or fall through from D.2): run + /// lightweight P+M replay on commits `> S1` and merge them into the existing log segment. + /// + /// Each case is marked with `// Case X` in the function body. + /// + /// [`SnapshotBuilder::at_version`]: crate::snapshot::SnapshotBuilder::at_version + /// [`SnapshotBuilder::with_max_catalog_version`]: crate::snapshot::SnapshotBuilder::with_max_catalog_version + #[instrument(err, fields(version, operation_id = %metric_context.operation_id, correlation_id = metric_context.correlation_id.as_deref().unwrap_or("")), skip(engine, target_version))] + pub(super) fn try_new_from( + existing_snapshot: Arc, + log_tail: Vec, + engine: &dyn Engine, + target_version: impl Into>, + metric_context: SnapshotLoadMetricContext, + incremental_replay: IncrementalReplay, + built_as_latest: bool, + ) -> DeltaResult> { + let existing_log_segment = &existing_snapshot.log_segment; + let existing_snapshot_version = existing_snapshot.version(); + let requested_version = target_version.into(); + if let Some(requested_version) = requested_version { + tracing::Span::current().record("version", requested_version); + // Case A: re-requesting the same version. + if requested_version == existing_snapshot_version { + return Ok(existing_snapshot.clone()); + } + // Case B: incremental path only moves forward. + if requested_version < existing_snapshot_version { + return Err(Error::Generic(format!( + "Requested snapshot version {requested_version} is older than snapshot \ + hint version {existing_snapshot_version}" + ))); + } + } else { + tracing::Span::current().record("version", existing_snapshot_version); + } + + // Assemble the new segment as one fallible unit so a load failure emits exactly once, via + // the `inspect_err` below. + let segment_load_start = std::time::Instant::now(); + let (combined_log_segment, new_end_version) = match Self::build_new_segment( + engine, + existing_log_segment, + existing_snapshot_version, + log_tail, + requested_version, + ) + .inspect_err(|_| emit_log_segment_load_failure(&metric_context))? + { + NewSegment::Unchanged => { + return Self::reuse_promoting_built_as_latest(&existing_snapshot, built_as_latest); + } + NewSegment::Rebuild(new_log_segment) => { + emit_log_segment_load( + &metric_context, + &new_log_segment, + segment_load_start.elapsed(), + ); + let snapshot = Self::try_new_from_log_segment( + existing_snapshot.table_root().clone(), + new_log_segment, + engine, + metric_context, + incremental_replay, + built_as_latest, + ); + return Ok(Arc::new(snapshot?)); + } + NewSegment::Combined(combined_log_segment) => { + emit_log_segment_load( + &metric_context, + &combined_log_segment, + segment_load_start.elapsed(), + ); + let new_end_version = combined_log_segment.end_version; + (combined_log_segment, new_end_version) + } + }; + + // Advance the latest available base (the existing snapshot's in-memory CRC, or a newer + // on-disk CRC the combined segment carries) to the new end version, subject to + // `incremental_replay`. Time from here so the CRC-advance cost lands in the P&M duration, + // matching the fresh path. + let pm_start = std::time::Instant::now(); + let base_crc = + combined_log_segment.pick_latest_base_crc(engine, existing_snapshot.base_crc()); + let crc_at_version = combined_log_segment + .try_build_crc_within_budget(engine, base_crc.as_ref(), incremental_replay) + .inspect_err(|_| emit_protocol_metadata_load_failure(&metric_context))?; + + let existing_table_config = existing_snapshot.table_configuration(); + let (new_metadata, new_protocol, source) = match &crc_at_version { + Some((crc, source)) => { + // If we were able to build a new CRC, then re-use it for TableConfiguration + // creation. + let new_metadata = (crc.metadata != *existing_table_config.metadata()) + .then(|| crc.metadata.clone()); + let new_protocol = (crc.protocol != *existing_table_config.protocol()) + .then(|| crc.protocol.clone()); + (new_metadata, new_protocol, *source) + } + None => { + // No incremental CRC to reuse: there was no base CRC, or advancing it was out of + // budget (note: we have *not* yet scanned any log files). Replay the new commits + // `(existing_snapshot_version, end]` for P&M, rooted at the base CRC when it is + // newer than the existing snapshot (else the existing snapshot is the baseline). + let newer_base = base_crc + .as_ref() + .filter(|c| c.version > existing_snapshot_version); + combined_log_segment + .segment_after_version(existing_snapshot_version) + .read_protocol_metadata_opt(engine, newer_base) + .inspect_err(|_| emit_protocol_metadata_load_failure(&metric_context))? + } + }; + emit_protocol_metadata_load(&metric_context, source, pm_start.elapsed()); + + let table_configuration = TableConfiguration::try_new_from( + existing_table_config, + new_metadata, + new_protocol, + new_end_version, + )?; + + tracing::Span::current().record("version", table_configuration.version()); + Ok(Arc::new(Snapshot::new_with_crc( + combined_log_segment, + table_configuration, + crc_at_version.map(|(crc, _)| crc).or(base_crc), + built_as_latest, + )?)) + } + + // ============================================================================ + // Helpers + // ============================================================================ + + /// List the log after the existing snapshot and assemble the new [`NewSegment`] for this + /// incremental update. Returns a non-failure [`NewSegment`] on the C/D.1/E/F cases; a + /// propagated `Err` is a genuine listing/assembly failure. + fn build_new_segment( + engine: &dyn Engine, + existing_log_segment: &LogSegment, + existing_snapshot_version: Version, + log_tail: Vec, + requested_version: Option, + ) -> DeltaResult { + let log_root = existing_log_segment.log_root.clone(); + let storage = engine.storage_handler(); + + // Start listing just after the previous segment's checkpoint, if any. + let listing_start = existing_log_segment.checkpoint_version.unwrap_or(0) + 1; + + let new_listed_files = LogSegmentFiles::list( + storage.as_ref(), + &log_root, + log_tail, + Some(listing_start), + requested_version, + )?; + + // NB: we need to check both checkpoints and commits since we filter commits at and below + // the checkpoint version. Example: if we have a checkpoint + commit at version 1, the log + // listing above will only return the checkpoint and not the commit. + if new_listed_files.ascending_commit_files().is_empty() + && new_listed_files.checkpoint_parts().is_empty() + { + return match requested_version { + // Case C.1: caller requested a specific version (necessarily > + // existing_snapshot_version since cases A and B were handled above), but + // no such commit exists in the log. + Some(requested_version) => Err(Error::Generic(format!( + "Requested snapshot version {requested_version} is not available: \ + no new commits were found after existing snapshot version \ + {existing_snapshot_version}" + ))), + // Case C.2: no new commits and no explicit target; latest is existing. + None => Ok(NewSegment::Unchanged), + }; + } + + // create a log segment just from existing_checkpoint.version -> new_version + // OR could be from 1 -> new_version + // Save the latest_commit before moving new_listed_files + let new_latest_commit_file = new_listed_files.latest_commit_file().clone(); + // Note: new_log_segment won't have last_checkpoint_metadata since we're listing without a + // hint. If the new segment has a checkpoint, we will return it as is. Otherwise, we + // will preserve last_checkpoint_metadata when merging the new log segment with the + // old one. + let mut new_log_segment = + LogSegment::try_new(new_listed_files, log_root.clone(), requested_version, None)?; + + let new_end_version = new_log_segment.end_version; + if new_end_version < existing_snapshot_version { + // we should never see a new log segment with a version < the existing snapshot + // version, that would mean a commit was incorrectly deleted from the log + return Err(Error::Generic(format!( + "Unexpected state: the newest version in the log {new_end_version} is \ + older than the existing snapshot version {existing_snapshot_version}" + ))); + } + if let Some(new_checkpoint_version) = new_log_segment.checkpoint_version { + if new_checkpoint_version > existing_snapshot_version { + // Case D.1: checkpoint ahead of existing snapshot. Commits between + // existing_snapshot_version+1 and new_checkpoint_version were filtered out + // of the listing, so a full rebuild is required. The existing snapshot's CRC + // is older than the new checkpoint and cannot apply, but a fresh on-disk CRC + // at or above the new checkpoint still advances per `incremental_replay`. + return Ok(NewSegment::Rebuild(new_log_segment)); + } + // Case D.2: checkpoint at or below existing snapshot; fall through to case F to + // replay only the new commits and advance the checkpoint base. + } + + // Case E: no new checkpoint, version did not advance; return existing. + if new_end_version == existing_snapshot_version + && new_log_segment.checkpoint_version.is_none() + { + // We must check checkpoint_version here: if a checkpoint at or below the existing + // snapshot version was discovered, we still need to fall through to advance the + // checkpoint base even though the version did not change. + return Ok(NewSegment::Unchanged); + } + + // Case F: lightweight P+M replay on new commits, merge with existing segment. + // (Also reached from Case D.2 when a checkpoint at or below the existing snapshot + // version was discovered.) + // + // The example below illustrates Case D.2 -> F. + #[rustfmt::skip] + // Example: existing segment = checkpoint@v0 + commit@v1, v2, v3 + // existing_snapshot_version = v3, listing_start = v1 + // target=v4, new checkpoint@v2 written since last build (Case D.2 -> F): + // + // listing: commit@v1, then checkpoint@v2 + commit@v3, v4 + // (checkpoint flush drops v1) + // new segment: checkpoint@v2 + commit@v3, v4 + // after dedup: drop commit@v3 (already in existing snapshot) + // -> ascending_commit_files = [commit@v4] + // commit@v3 is not lost: the existing segment contributes it below + // (existing commits above checkpoint@v2 = [commit@v3]) + // combined segment: checkpoint@v2 + commit@v3 (from existing) + commit@v4 (from new) + // checkpoint@v2 is kept in checkpoint_parts to advance the listing base. + new_log_segment + .listed + .ascending_commit_files + .retain(|log_path| existing_snapshot_version < log_path.version); + // Deduplicate compaction files the same way: the new listing re-lists from + // checkpoint_version, so it includes compaction files already in the existing segment. + // Note: This removes all _new_ compaction files that start at or before + // `existing_snapshot_version`, which may drop useful compaction files that span + // across the old/new boundary (e.g. a new compaction(1, 3) when + // existing_snapshot_version=2). This is conservative but safe. + new_log_segment + .listed + .ascending_compaction_files + .retain(|log_path| existing_snapshot_version < log_path.version); + + // The CRC file the combined segment carries on disk. + let crc_file = Self::resolve_crc_file(&new_log_segment, existing_log_segment); + + // Combine existing and new log segments. When the new listing found a checkpoint at or + // below the existing snapshot version, trim existing commits and compaction files + // already covered by it; otherwise keep all existing files. + let new_checkpoint_version = new_log_segment.checkpoint_version; + let mut ascending_commit_files = Self::files_after_checkpoint( + &existing_log_segment.listed.ascending_commit_files, + new_checkpoint_version, + ); + ascending_commit_files.extend(new_log_segment.listed.ascending_commit_files); + let mut ascending_compaction_files = Self::files_after_checkpoint( + &existing_log_segment.listed.ascending_compaction_files, + new_checkpoint_version, + ); + ascending_compaction_files.extend(new_log_segment.listed.ascending_compaction_files); + + // When the new listing found a checkpoint at or below the existing snapshot version, + // advance the base so future listings start from new_checkpoint_version+1; otherwise + // preserve the existing base. + // Note: the incremental listing never uses a _last_checkpoint hint, so + // new_log_segment.last_checkpoint_metadata is always None when a new checkpoint + // is found here. The combined segment will read the new checkpoint's parquet footer at + // scan time. When no new checkpoint was found, preserve the existing hint if available. + let (new_checkpoint_parts, new_checkpoint_hint) = if new_checkpoint_version.is_some() { + ( + new_log_segment.listed.checkpoint_parts.clone(), + new_log_segment.last_checkpoint_metadata.clone(), + ) + } else { + ( + existing_log_segment.listed.checkpoint_parts.clone(), + existing_log_segment.last_checkpoint_metadata.clone(), + ) + }; + let combined_log_segment = LogSegment::try_new( + LogSegmentFiles { + ascending_commit_files, + ascending_compaction_files, + checkpoint_parts: new_checkpoint_parts, + latest_crc_file: crc_file, + // In Case F there are new commits (new_end_version > existing_snapshot_version + // with no new checkpoint), so the new listing's `latest_commit_file` is always + // `Some(commit @ new_end_version)` and matches the combined snapshot version. + latest_commit_file: new_latest_commit_file, + max_published_version: new_log_segment + .listed + .max_published_version + .max(existing_log_segment.listed.max_published_version), + }, + log_root, + requested_version, + new_checkpoint_hint, + )?; + + Ok(NewSegment::Combined(combined_log_segment)) + } + + /// Reuse `existing`, promoting its `built_as_latest` flag to `true` if this build confirmed + /// latest. + fn reuse_promoting_built_as_latest( + existing: &Arc, + built_as_latest: bool, + ) -> DeltaResult> { + // Promote only false -> true; every other case reuses the Arc unchanged. + if existing.built_as_latest || !built_as_latest { + return Ok(existing.clone()); + } + Ok(Arc::new(Snapshot::new_with_crc( + existing.log_segment.clone(), + existing.table_configuration.clone(), + existing.base_crc().cloned(), + true, // reached only when the flag flips false -> true + )?)) + } + + /// Determine the on-disk CRC file the combined segment should carry. + /// + /// Prefers the new segment's CRC; falls back to the existing segment's CRC if and only if + /// its version is >= the new segment's checkpoint version, so the fallback never violates the + /// [`LogSegmentFiles`] invariant `crc.version >= checkpoint.version` (enforced in + /// [`LogSegment::try_new`]). + fn resolve_crc_file( + new_log_segment: &LogSegment, + existing_log_segment: &LogSegment, + ) -> Option { + let crc_satisfies_new_ckpt = |crc: &ParsedLogPath| { + new_log_segment + .checkpoint_version + .is_none_or(|ckpt_v| crc.version >= ckpt_v) + }; + let existing_crc_file = existing_log_segment + .listed + .latest_crc_file + .clone() + .filter(crc_satisfies_new_ckpt); + new_log_segment + .listed + .latest_crc_file + .clone() + .or(existing_crc_file) + } + + /// Filters `files` to only those with version above `checkpoint_version`, or clones all if + /// `checkpoint_version` is `None`. + fn files_after_checkpoint( + files: &[ParsedLogPath], + checkpoint_version: Option, + ) -> Vec { + match checkpoint_version { + Some(ckpt_v) => files + .iter() + .filter(|f| f.version > ckpt_v) + .cloned() + .collect(), + None => files.to_vec(), + } + } +} + +#[cfg(test)] +mod tests { + use std::path::PathBuf; + + use rstest::rstest; + use serde_json::json; + use test_utils::delta_path_for_version; + use test_utils::table_builder::TestTableBuilder; + use url::Url; + + use super::*; + use crate::arrow::array::StringArray; + use crate::arrow::record_batch::RecordBatch; + use crate::engine::arrow_data::ArrowEngineData; + use crate::engine::sync::SyncEngine; + use crate::metrics::{ + LogSegmentLoadSuccess, LogSegmentLoadType, MetricEvent, MetricId, + ProtocolMetadataLoadSuccess, ProtocolMetadataSource, + }; + use crate::object_store::memory::InMemory; + use crate::object_store::ObjectStoreExt as _; + use crate::parquet::arrow::ArrowWriter; + use crate::path::LogPathFileType; + use crate::snapshot::commit; + use crate::unit_test_utils::{ + install_thread_local_metrics_reporter, string_array_to_engine_data, CapturingReporter, + }; + + // ============================================================================ + // Helpers + // ============================================================================ + + // Action builders for incremental-snapshot tests. Centralized so commit setup stays + // consistent across tests (e.g. the schema matches `make_test_crc_json`). + fn protocol_action(min_reader: u32, min_writer: u32) -> serde_json::Value { + json!({"protocol": {"minReaderVersion": min_reader, "minWriterVersion": min_writer}}) + } + + fn metadata_action(configuration: serde_json::Value) -> serde_json::Value { + json!({ + "metaData": { + "id": "test-id", + "format": {"provider": "parquet", "options": {}}, + "schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}", + "partitionColumns": [], + "configuration": configuration, + "createdTime": 1587968585495i64, + } + }) + } + + fn add_action(path: &str) -> serde_json::Value { + json!({ + "add": { + "path": path, + "partitionValues": {}, + "size": 100, + "modificationTime": 1000, + "dataChange": true, + } + }) + } + + // Helper: create a minimal test table with commits 0..num_commits. + async fn setup_test_table_with_commits( + table_root: impl AsRef, + store: &InMemory, + num_commits: u64, + ) -> DeltaResult<()> { + // Commit 0: protocol + metadata + first file. + commit( + table_root.as_ref(), + store, + 0, + vec![ + protocol_action(1, 2), + metadata_action(json!({})), + add_action("file1.parquet"), + ], + ) + .await; + + // Additional commits with just add actions. + for i in 1..num_commits { + let path = format!("file{}.parquet", i + 1); + commit(table_root.as_ref(), store, i, vec![add_action(&path)]).await; + } + Ok(()) + } + + // Helper: write a compaction file + async fn write_compaction_file(store: &InMemory, start: u64, end: u64) -> DeltaResult<()> { + let content = r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#; + store + .put( + &test_utils::compacted_log_path_for_versions(start, end, "json"), + content.into(), + ) + .await?; + Ok(()) + } + + struct IncrementalSnapshotTestContext { + store: Arc, + url: Url, + engine: Arc, + } + + fn setup_incremental_snapshot_test() -> DeltaResult { + let store = Arc::new(InMemory::new()); + let url = Url::parse("memory:///")?; + let engine = Arc::new(SyncEngine::new_with_store(store.clone())); + + Ok(IncrementalSnapshotTestContext { store, url, engine }) + } + + /// Compares two Snapshots field-by-field. LogSegment fields are compared individually, + /// intentionally skipping `last_checkpoint_metadata` which is only populated on the + /// from-scratch path (via `_last_checkpoint` hint) and not on the incremental update path. + fn compare_snapshots(left: &Snapshot, right: &Snapshot) { + assert_eq!(left.table_configuration, right.table_configuration); + assert_eq!(left.log_segment.end_version, right.log_segment.end_version); + assert_eq!( + left.log_segment.checkpoint_version, + right.log_segment.checkpoint_version + ); + assert_eq!(left.log_segment.log_root, right.log_segment.log_root); + assert_eq!(left.log_segment.listed, right.log_segment.listed); + } + + // CRC JSON for the standard test table (see `setup_test_table_with_commits`). + fn make_test_crc_json(table_size_bytes: i64, num_files: i64) -> serde_json::Value { + json!({ + "tableSizeBytes": table_size_bytes, + "numFiles": num_files, + "numMetadata": 1, + "numProtocol": 1, + "protocol": {"minReaderVersion": 1, "minWriterVersion": 2}, + "metadata": { + "id": "test-id", + "format": {"provider": "parquet", "options": {}}, + "schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}", + "partitionColumns": [], + "configuration": {}, + "createdTime": 1587968585495i64 + } + }) + } + + // ============================================================================ + // Tests: try_new_from direct API + // ============================================================================ + + #[test] + fn test_try_new_from_empty_log_tail() -> DeltaResult<()> { + let table = TestTableBuilder::new().build().unwrap(); + let engine = SyncEngine::new_with_store(table.store().clone()); + + let base_snapshot = Snapshot::builder_for(table.table_root()) + .at_version(0) + .build(&engine)?; + + let result = Snapshot::try_new_from( + base_snapshot.clone(), + vec![], + &engine, + None, + SnapshotLoadMetricContext::for_test(), + IncrementalReplay::Disabled, + true, /* built_as_latest */ + )?; + assert_eq!(result, base_snapshot); + // `PartialEq` ignores `built_as_latest`, so assert it explicitly. + assert!(result.is_built_as_latest()); + + Ok(()) + } + + #[tokio::test] + async fn test_try_new_from_latest_commit_preservation() -> DeltaResult<()> { + let store = Arc::new(InMemory::new()); + let url = Url::parse("memory:///")?; + let engine = SyncEngine::new_with_store(store.clone()); + + // Create commits 0-2 + let base_commit = vec![ + json!({"protocol": {"minReaderVersion": 1, "minWriterVersion": 2}}), + json!({ + "metaData": { + "id": "test-id", + "format": {"provider": "parquet", "options": {}}, + "schemaString": "{\"type\":\"struct\",\"fields\":[]}", + "partitionColumns": [], + "configuration": {}, + "createdTime": 1587968585495i64 + } + }), + ]; + + commit(url.as_str(), store.as_ref(), 0, base_commit.clone()).await; + commit( + url.as_str(), + store.as_ref(), + 1, + vec![json!({"commitInfo": {"timestamp": 1234}})], + ) + .await; + commit( + url.as_str(), + store.as_ref(), + 2, + vec![json!({"commitInfo": {"timestamp": 5678}})], + ) + .await; + + let base_snapshot = Snapshot::builder_for(url.as_str()) + .at_version(1) + .build(&engine)?; + + // Verify base snapshot has latest_commit_file at version 1 + assert_eq!( + base_snapshot + .log_segment + .listed + .latest_commit_file + .as_ref() + .map(|f| f.version), + Some(1) + ); + + // Create log_tail with FileMeta for version 2 + let commit_2_url = url.join("_delta_log/")?.join("00000000000000000002.json")?; + let file_meta = crate::FileMeta { + location: commit_2_url, + last_modified: 1234567890, + size: 100, + }; + let parsed_path = ParsedLogPath::try_from(file_meta)? + .ok_or_else(|| Error::Generic("Failed to parse log path".to_string()))?; + let log_tail = vec![parsed_path]; + + // Create new snapshot from base to version 2 using try_new_from directly + let new_snapshot = Snapshot::try_new_from( + base_snapshot.clone(), + log_tail, + &engine, + Some(2), + SnapshotLoadMetricContext::for_test(), + IncrementalReplay::Disabled, + false, /* built_as_latest */ + )?; + + // Latest commit should now be version 2 + assert_eq!( + new_snapshot + .log_segment + .listed + .latest_commit_file + .as_ref() + .map(|f| f.version), + Some(2) + ); + + Ok(()) + } + + #[tokio::test] + async fn test_try_new_from_version_boundary_cases() -> DeltaResult<()> { + let store = Arc::new(InMemory::new()); + let table_root = "memory:///test_table/"; + let engine = SyncEngine::new_with_store(store.clone()); + + // Create commits + let base_commit = vec![ + json!({"protocol": {"minReaderVersion": 1, "minWriterVersion": 2}}), + json!({ + "metaData": { + "id": "test-id", + "format": {"provider": "parquet", "options": {}}, + "schemaString": "{\"type\":\"struct\",\"fields\":[]}", + "partitionColumns": [], + "configuration": {}, + "createdTime": 1587968585495i64 + } + }), + ]; + + commit(table_root, store.as_ref(), 0, base_commit).await; + commit( + table_root, + store.as_ref(), + 1, + vec![json!({"commitInfo": {"timestamp": 1234}})], + ) + .await; + + let base_snapshot = Snapshot::builder_for(table_root) + .at_version(1) + .build(&engine)?; + + // Test requesting same version - should return same snapshot + let same_version = Snapshot::try_new_from( + base_snapshot.clone(), + vec![], + &engine, + Some(1), + SnapshotLoadMetricContext::for_test(), + IncrementalReplay::Disabled, + false, /* built_as_latest */ + )?; + assert!(Arc::ptr_eq(&same_version, &base_snapshot)); + + // Test requesting older version - should error + let older_version = Snapshot::try_new_from( + base_snapshot.clone(), + vec![], + &engine, + Some(0), + SnapshotLoadMetricContext::for_test(), + IncrementalReplay::Disabled, + false, /* built_as_latest */ + ); + assert!(matches!( + older_version, + Err(Error::Generic(msg)) if msg.contains("older than snapshot hint version") + )); + + Ok(()) + } + + // ============================================================================ + // Tests: incremental builder paths (Cases A/B/C/D.1/D.2/E/F) + // ============================================================================ + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_incremental_snapshot_picks_up_checkpoint_written_at_current_version( + ) -> DeltaResult<()> { + let ctx = setup_incremental_snapshot_test()?; + + setup_test_table_with_commits(ctx.url.as_str(), &ctx.store, 2).await?; + + let snapshot_v1 = Snapshot::builder_for(ctx.url.as_str()) + .at_version(1) + .build(ctx.engine.as_ref())?; + assert_eq!(snapshot_v1.log_segment.checkpoint_version, None); + + snapshot_v1.clone().checkpoint(ctx.engine.as_ref(), None)?; + + let fresh = Snapshot::builder_for(ctx.url.as_str()).build(ctx.engine.as_ref())?; + assert_eq!(fresh.version(), 1); + assert_eq!(fresh.log_segment.checkpoint_version, Some(1)); + + let updated = Snapshot::builder_from(snapshot_v1).build(ctx.engine.as_ref())?; + assert_eq!(updated.version(), 1); + assert_eq!(updated.log_segment.checkpoint_version, Some(1)); + compare_snapshots(&updated, &fresh); + + Ok(()) + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_incremental_snapshot_picks_up_newer_checkpoint_below_current_version( + ) -> DeltaResult<()> { + let ctx = setup_incremental_snapshot_test()?; + + setup_test_table_with_commits(ctx.url.as_str(), &ctx.store, 4).await?; + + Snapshot::builder_for(ctx.url.as_str()) + .at_version(1) + .build(ctx.engine.as_ref())? + .checkpoint(ctx.engine.as_ref(), None)?; + + let snapshot_v3 = Snapshot::builder_for(ctx.url.as_str()) + .at_version(3) + .build(ctx.engine.as_ref())?; + assert_eq!(snapshot_v3.log_segment.checkpoint_version, Some(1)); + + Snapshot::builder_for(ctx.url.as_str()) + .at_version(2) + .build(ctx.engine.as_ref())? + .checkpoint(ctx.engine.as_ref(), None)?; + + let fresh = Snapshot::builder_for(ctx.url.as_str()).build(ctx.engine.as_ref())?; + assert_eq!(fresh.version(), 3); + assert_eq!(fresh.log_segment.checkpoint_version, Some(2)); + + let updated = Snapshot::builder_from(snapshot_v3).build(ctx.engine.as_ref())?; + assert_eq!(updated.version(), 3); + assert_eq!(updated.log_segment.checkpoint_version, Some(2)); + compare_snapshots(&updated, &fresh); + + Ok(()) + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_explicit_same_version_request_keeps_existing_snapshot_after_checkpoint_write( + ) -> DeltaResult<()> { + let ctx = setup_incremental_snapshot_test()?; + + setup_test_table_with_commits(ctx.url.as_str(), &ctx.store, 2).await?; + + let snapshot_v1 = Snapshot::builder_for(ctx.url.as_str()) + .at_version(1) + .build(ctx.engine.as_ref())?; + assert_eq!(snapshot_v1.log_segment.checkpoint_version, None); + + snapshot_v1.clone().checkpoint(ctx.engine.as_ref(), None)?; + + let refreshed = Snapshot::builder_for(ctx.url.as_str()).build(ctx.engine.as_ref())?; + assert_eq!(refreshed.log_segment.checkpoint_version, Some(1)); + + let pinned = Snapshot::builder_from(snapshot_v1.clone()) + .at_version(1) + .build(ctx.engine.as_ref())?; + assert!(Arc::ptr_eq(&pinned, &snapshot_v1)); + assert_eq!(pinned.log_segment.checkpoint_version, None); + + Ok(()) + } + + /// When a checkpoint at or below the existing snapshot version is discovered and commits + /// between that checkpoint and the existing snapshot version contain P&M changes, the + /// incremental update must preserve those P&M changes rather than regressing to the + /// checkpoint's older P&M. + /// + /// Setup: initial checkpoint@v0 so listing_start=1. A checkpoint at v1 is written (discoverable + /// since v1 >= listing_start). The P&M change at commit 2 lies between the new checkpoint (v1) + /// and existing_snapshot_version (v3); the incremental update must preserve it. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_checkpoint_at_or_below_snapshot_version_preserves_pm_from_commits_in_between( + ) -> DeltaResult<()> { + let ctx = setup_incremental_snapshot_test()?; + let table_root = ctx.url.as_str(); + + // Commit 0: initial protocol (1, 2) + metadata + first file. + commit( + table_root, + &ctx.store, + 0, + vec![ + protocol_action(1, 2), + metadata_action(json!({})), + add_action("file1.parquet"), + ], + ) + .await; + // Commit 1: data-only add. + commit(table_root, &ctx.store, 1, vec![add_action("file2.parquet")]).await; + // Commit 2: protocol upgrade to (2, 5). This is the change the incremental update + // must preserve rather than regressing to checkpoint@v1's older protocol. + commit(table_root, &ctx.store, 2, vec![protocol_action(2, 5)]).await; + // Commit 3: data-only add. + commit(table_root, &ctx.store, 3, vec![add_action("file3.parquet")]).await; + + // Write initial checkpoint at v0 so that listing_start = v0+1 = 1. + Snapshot::builder_for(table_root) + .at_version(0) + .build(ctx.engine.as_ref())? + .checkpoint(ctx.engine.as_ref(), None)?; + + // Build snapshot at v3: has checkpoint@v0, protocol (2, 5) from commit 2. + let snapshot_v3 = Snapshot::builder_for(table_root) + .at_version(3) + .build(ctx.engine.as_ref())?; + assert_eq!(snapshot_v3.log_segment.checkpoint_version, Some(0)); + assert_eq!( + snapshot_v3 + .table_configuration() + .protocol() + .min_reader_version(), + 2 + ); + assert_eq!( + snapshot_v3 + .table_configuration() + .protocol() + .min_writer_version(), + 5 + ); + + // Write a checkpoint at v1 (v1 >= listing_start=1, so it is discoverable). + // This checkpoint captures only protocol (1, 2) from commit 0. + Snapshot::builder_for(table_root) + .at_version(1) + .build(ctx.engine.as_ref())? + .checkpoint(ctx.engine.as_ref(), None)?; + + // Commit 4: add file (new state beyond existing_snapshot_version=3). + commit(table_root, &ctx.store, 4, vec![add_action("file4.parquet")]).await; + + // Incremental update from v3: listing starts from v1, discovers checkpoint@v1. + // Must preserve protocol (2, 5) from commit 2, not regress to (1, 2) from checkpoint@v1. + let updated = Snapshot::builder_from(snapshot_v3).build(ctx.engine.as_ref())?; + assert_eq!(updated.version(), 4); + assert_eq!(updated.log_segment.checkpoint_version, Some(1)); + + // Critical: protocol must still be (2, 5) from commit 2, not (1, 2) from ckpt@v1. + assert_eq!( + updated + .table_configuration() + .protocol() + .min_reader_version(), + 2 + ); + assert_eq!( + updated + .table_configuration() + .protocol() + .min_writer_version(), + 5 + ); + + // Cross-check against a fresh build. + let fresh = Snapshot::builder_for(ctx.url.as_str()).build(ctx.engine.as_ref())?; + compare_snapshots(&updated, &fresh); + + Ok(()) + } + + // interesting cases for testing Snapshot::new_from: + // 1. new version < existing version + // 2. new version == existing version + // 3. new version > existing version AND + // a. log segment hasn't changed + // b. log segment for old..=new version has a checkpoint (with new protocol/metadata) + // b. log segment for old..=new version has no checkpoint + // i. commits have (new protocol, new metadata) + // ii. commits have (new protocol, no metadata) + // iii. commits have (no protocol, new metadata) + // iv. commits have (no protocol, no metadata) + #[tokio::test] + async fn test_snapshot_new_from() -> DeltaResult<()> { + let path = + std::fs::canonicalize(PathBuf::from("./tests/data/table-with-dv-small/")).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + + let engine = SyncEngine::new(); + let old_snapshot = Snapshot::builder_for(url.clone()) + .at_version(1) + .build(&engine) + .unwrap(); + // 1. new version < existing version: error + let snapshot_res = Snapshot::builder_from(old_snapshot.clone()) + .at_version(0) + .build(&engine); + assert!(matches!( + snapshot_res, + Err(Error::Generic(msg)) if msg == "Requested snapshot version 0 is older than snapshot hint version 1" + )); + + // 2. new version == existing version + let snapshot = Snapshot::builder_from(old_snapshot.clone()) + .at_version(1) + .build(&engine) + .unwrap(); + let expected = old_snapshot.clone(); + assert_eq!(snapshot, expected); + + // tests Snapshot::new_from by: + // 1. creating a snapshot with new API for commits 0..=2 (based on old snapshot at 0) + // 2. comparing with a snapshot created directly at version 2 + // + // the commits tested are: + // - commit 0 -> base snapshot at this version + // - commit 1 -> final snapshots at this version + // + // in each test we will modify versions 1 and 2 to test different scenarios + fn test_new_from(store: Arc) -> DeltaResult<()> { + let table_root = "memory:///"; + let engine = SyncEngine::new_with_store(store); + let base_snapshot = Snapshot::builder_for(table_root) + .at_version(0) + .build(&engine)?; + let snapshot = Snapshot::builder_from(base_snapshot.clone()) + .at_version(1) + .build(&engine)?; + let expected = Snapshot::builder_for(table_root) + .at_version(1) + .build(&engine)?; + assert_eq!(snapshot, expected); + Ok(()) + } + + // for (3) we will just engineer custom log files + let store = Arc::new(InMemory::new()); + // everything will have a starting 0 commit with commitInfo, protocol, metadata + let commit0 = vec![ + json!({ + "commitInfo": { + "timestamp": 1587968586154i64, + "operation": "WRITE", + "operationParameters": {"mode":"ErrorIfExists","partitionBy":"[]"}, + "isBlindAppend":true + } + }), + json!({ + "protocol": { + "minReaderVersion": 1, + "minWriterVersion": 2 + } + }), + json!({ + "metaData": { + "id":"5fba94ed-9794-4965-ba6e-6ee3c0d22af9", + "format": { + "provider": "parquet", + "options": {} + }, + "schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"val\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}", + "partitionColumns": [], + "configuration": {}, + "createdTime": 1587968585495i64 + } + }), + ]; + let table_root = "memory:///"; + commit(table_root, store.as_ref(), 0, commit0.clone()).await; + // 3. new version > existing version + // a. no new log segment + let engine = SyncEngine::new_with_store(Arc::new(store.fork())); + let base_snapshot = Snapshot::builder_for(table_root) + .at_version(0) + .build(&engine)?; + let snapshot = Snapshot::builder_from(base_snapshot.clone()).build(&engine)?; + let expected = Snapshot::builder_for(table_root) + .at_version(0) + .build(&engine)?; + assert_eq!(snapshot, expected); + // version exceeds latest version of the table = err + assert!(matches!( + Snapshot::builder_from(base_snapshot.clone()).at_version(1).build(&engine), + Err(Error::Generic(msg)) if msg == "Requested snapshot version 1 is not available: no new commits were found after existing snapshot version 0" + )); + + // b. log segment for old..=new version has a checkpoint (with new protocol/metadata) + let store_3a = store.fork(); + let mut checkpoint1 = commit0.clone(); + commit(table_root, &store_3a, 1, commit0.clone()).await; + checkpoint1[1] = json!({ + "protocol": { + "minReaderVersion": 2, + "minWriterVersion": 5 + } + }); + checkpoint1[2]["partitionColumns"] = serde_json::to_value(["some_partition_column"])?; + + let handler = engine.json_handler(); + let json_strings: StringArray = checkpoint1 + .into_iter() + .map(|json| json.to_string()) + .collect::>() + .into(); + let parsed = handler + .parse_json( + string_array_to_engine_data(json_strings), + crate::actions::get_commit_schema().clone(), + ) + .unwrap(); + let checkpoint = ArrowEngineData::try_from_engine_data(parsed).unwrap(); + let checkpoint: RecordBatch = checkpoint.into(); + + // Write the record batch to a Parquet file + let mut buffer = vec![]; + let mut writer = ArrowWriter::try_new(&mut buffer, checkpoint.schema(), None)?; + writer.write(&checkpoint)?; + writer.close()?; + + store_3a + .put( + &delta_path_for_version(1, "checkpoint.parquet"), + buffer.into(), + ) + .await + .unwrap(); + test_new_from(store_3a.into())?; + + // c. log segment for old..=new version has no checkpoint + // i. commits have (new protocol, new metadata) + let store_3c_i = Arc::new(store.fork()); + let mut commit1 = commit0.clone(); + commit1[1] = json!({ + "protocol": { + "minReaderVersion": 2, + "minWriterVersion": 5 + } + }); + commit1[2]["partitionColumns"] = serde_json::to_value(["some_partition_column"])?; + commit(table_root, store_3c_i.as_ref(), 1, commit1).await; + test_new_from(store_3c_i.clone())?; + + // new commits AND request version > end of log + let engine = SyncEngine::new_with_store(store_3c_i); + let base_snapshot = Snapshot::builder_for(table_root) + .at_version(0) + .build(&engine)?; + assert!(matches!( + Snapshot::builder_from(base_snapshot.clone()).at_version(2).build(&engine), + Err(Error::Generic(msg)) if msg == "LogSegment end version 1 not the same as the specified end version 2" + )); + + // ii. commits have (new protocol, no metadata) + let store_3c_ii = store.fork(); + let mut commit1 = commit0.clone(); + commit1[1] = json!({ + "protocol": { + "minReaderVersion": 2, + "minWriterVersion": 5 + } + }); + commit1.remove(2); // remove metadata + commit(table_root, &store_3c_ii, 1, commit1).await; + test_new_from(store_3c_ii.into())?; + + // iii. commits have (no protocol, new metadata) + let store_3c_iii = store.fork(); + let mut commit1 = commit0.clone(); + commit1[2]["partitionColumns"] = serde_json::to_value(["some_partition_column"])?; + commit1.remove(1); // remove protocol + commit(table_root, &store_3c_iii, 1, commit1).await; + test_new_from(store_3c_iii.into())?; + + // iv. commits have (no protocol, no metadata) + let store_3c_iv = store.fork(); + let commit1 = vec![commit0[0].clone()]; + commit(table_root, &store_3c_iv, 1, commit1).await; + test_new_from(store_3c_iv.into())?; + + Ok(()) + } + + // ============================================================================ + // Tests: stale-CRC resolution on the incremental path + // ============================================================================ + + // `resolve_crc_file` when the new listing discovers a checkpoint *at or below* the existing + // snapshot version must correctly decide which CRC file the combined segment carries on disk. + // Each case below sets up commits 0-3, writes an existing CRC, builds snapshot_v3, writes + // a checkpoint, optionally writes a new CRC, then runs the incremental update and checks + // the segment's `latest_crc_file`. (The checkpoint-*ahead* path is covered separately by + // `test_incremental_snapshot_multi_hop_replay_then_rebuild_drops_stale_crc`.) + // + // The update lands at the same version (v3), so the existing snapshot's in-memory crc@v3 + // always carries forward; only the on-disk `latest_crc_file` differs per case. + // + // Cases: + // keep_existing: existing crc@v2 stays when new ckpt@v1 is below it (invariant OK) + // and no newer CRC was listed; segment carries crc@v2. + // refresh_to_newer: existing crc@v2 is superseded by a newly-listed crc@v3. + // crc_equals_ckpt: existing crc@v2 and new ckpt@v2 sit at the same version; sanity + // coverage for the invariant boundary `crc.version >= ckpt.version`. + // drop_below_ckpt: existing crc@v1 would violate the LogSegmentFiles invariant with + // new ckpt@v2 (1 < 2), so it is filtered out; segment has no CRC file. + #[rstest] + #[case::keep_existing(2, 1, None, Some(2))] + #[case::refresh_to_newer(2, 1, Some(3), Some(3))] + #[case::crc_equals_ckpt(2, 2, None, Some(2))] + #[case::drop_below_ckpt(1, 2, None, None)] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_incremental_snapshot_crc_resolution_when_checkpoint_at_or_below_snapshot( + #[case] existing_crc_v: u64, + #[case] new_ckpt_v: u64, + #[case] newly_listed_crc_v: Option, + #[case] expected_crc_file_v: Option, + ) -> DeltaResult<()> { + let ctx = setup_incremental_snapshot_test()?; + setup_test_table_with_commits(ctx.url.as_str(), &ctx.store, 4).await?; + + ctx.store + .put( + &delta_path_for_version(existing_crc_v, "crc"), + make_test_crc_json(300, 3).to_string().into(), + ) + .await?; + + let snapshot_v3 = Snapshot::builder_for(ctx.url.as_str()) + .at_version(3) + .with_incremental_crc_replay(IncrementalReplay::Unlimited) + .build(ctx.engine.as_ref())?; + // A fresh build advances the stale CRC via reverse-replay to a CRC at v3 (file stats + // Indeterminate, since these synthetic commits carry no commitInfo). The segment's + // latest CRC file still points at the on-disk CRC version. + assert_eq!(snapshot_v3.crc_at_version().map(|c| c.version), Some(3)); + assert_eq!( + snapshot_v3 + .log_segment + .listed + .latest_crc_file + .as_ref() + .map(|f| f.version), + Some(existing_crc_v) + ); + + // Write the new checkpoint, then optionally plant a newer CRC at `newly_listed_crc_v`. + Snapshot::builder_for(ctx.url.as_str()) + .at_version(new_ckpt_v) + .build(ctx.engine.as_ref())? + .checkpoint(ctx.engine.as_ref(), None)?; + if let Some(v) = newly_listed_crc_v { + ctx.store + .put( + &delta_path_for_version(v, "crc"), + make_test_crc_json(400, 4).to_string().into(), + ) + .await?; + } + + let updated = Snapshot::builder_from(snapshot_v3).build(ctx.engine.as_ref())?; + + // Observable equivalence: incremental update matches a fresh rebuild, including + // `log_segment.listed.latest_crc_file` (so a filter-bypass regression is caught by + // `compare_snapshots`, not just by the explicit CRC assertions below). + let fresh = Snapshot::builder_for(ctx.url.as_str()).build(ctx.engine.as_ref())?; + compare_snapshots(&updated, &fresh); + assert_eq!(updated.version(), 3); + assert_eq!(updated.log_segment.checkpoint_version, Some(new_ckpt_v)); + assert_eq!( + updated + .log_segment + .listed + .latest_crc_file + .as_ref() + .map(|f| f.version), + expected_crc_file_v + ); + assert_eq!(updated.crc_at_version().map(|c| c.version), Some(3)); + + Ok(()) + } + + #[rstest] + #[case::disabled(IncrementalReplay::Disabled, None)] + #[case::unlimited(IncrementalReplay::Unlimited, Some(5))] + #[case::budget_exact(IncrementalReplay::UpToCommits(2), Some(5))] + #[case::budget_more_than_enough(IncrementalReplay::UpToCommits(5), Some(5))] + #[case::budget_too_small(IncrementalReplay::UpToCommits(1), None)] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_incremental_update_advances_in_memory_crc_within_budget( + #[case] mode: IncrementalReplay, + #[case] expected_crc_v: Option, + ) -> DeltaResult<()> { + let ctx = setup_incremental_snapshot_test()?; + setup_test_table_with_commits(ctx.url.as_str(), &ctx.store, 6).await?; + ctx.store + .put( + &delta_path_for_version(1, "crc"), + make_test_crc_json(200, 2).to_string().into(), + ) + .await?; + + let snapshot_a = Snapshot::builder_for(ctx.url.as_str()) + .at_version(3) + .with_incremental_crc_replay(IncrementalReplay::Unlimited) + .build(ctx.engine.as_ref())?; + assert_eq!(snapshot_a.crc_at_version().map(|c| c.version), Some(3)); + + let updated = Snapshot::builder_from(snapshot_a) + .with_incremental_crc_replay(mode) + .build(ctx.engine.as_ref())?; + assert_eq!(updated.version(), 5); + assert_eq!(updated.crc_at_version().map(|c| c.version), expected_crc_v); + assert_eq!( + updated + .log_segment + .listed + .latest_crc_file + .as_ref() + .map(|f| f.version), + Some(1) + ); + + Ok(()) + } + + // Stronger regression for the `resolve_crc_file` invariant filter: set up a metadata change + // at the checkpoint version so that a stale inherited CRC would produce wrong + // Metadata, not just wrong bookkeeping. Without the filter, the incremental update + // would inherit the below-checkpoint CRC and return Metadata from commit 0; the fresh + // rebuild would return Metadata from commit 2. `compare_snapshots` catches that on + // `table_configuration`. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_incremental_snapshot_drops_stale_crc_preserves_correct_metadata( + ) -> DeltaResult<()> { + let ctx = setup_incremental_snapshot_test()?; + let table_root = ctx.url.as_str(); + + // commit 0: initial P&M with empty configuration. + commit( + table_root, + &ctx.store, + 0, + vec![ + protocol_action(1, 2), + metadata_action(json!({})), + add_action("f0.parquet"), + ], + ) + .await; + // commit 1: data-only add. + commit(table_root, &ctx.store, 1, vec![add_action("f1.parquet")]).await; + // commit 2: metadata change (new configuration property). A stale CRC@v1 would miss + // this. + commit( + table_root, + &ctx.store, + 2, + vec![ + metadata_action(json!({"delta.myKey": "myValue"})), + add_action("f2.parquet"), + ], + ) + .await; + // commit 3: data-only add. + commit(table_root, &ctx.store, 3, vec![add_action("f3.parquet")]).await; + + // CRC@v1 captures the BEFORE metadata (no "delta.myKey" property). This is the + // stale CRC we expect the filter to reject. + ctx.store + .put( + &delta_path_for_version(1, "crc"), + make_test_crc_json(200, 2).to_string().into(), + ) + .await?; + + let snapshot_v3 = Snapshot::builder_for(table_root) + .at_version(3) + .build(ctx.engine.as_ref())?; + // snapshot_v3 was built via log replay, which correctly picks up the v2 metadata + // change (pruned replay over commits after CRC@v1 finds the new metaData action). + assert_eq!( + snapshot_v3 + .table_properties() + .unknown_properties + .get("delta.myKey"), + Some(&"myValue".to_string()) + ); + + // Write checkpoint@v2, incorporating the new metadata. + Snapshot::builder_for(table_root) + .at_version(2) + .build(ctx.engine.as_ref())? + .checkpoint(ctx.engine.as_ref(), None)?; + + // Incremental update. With the filter: CRC@v1 dropped, log replay via checkpoint@v2 + // returns the correct new metadata. Without the filter: the inherited CRC@v1 returns + // its stale (empty) configuration, and compare_snapshots fails on table_configuration. + let updated = Snapshot::builder_from(snapshot_v3).build(ctx.engine.as_ref())?; + let fresh = Snapshot::builder_for(table_root).build(ctx.engine.as_ref())?; + compare_snapshots(&updated, &fresh); + + // Observable contract: the user-visible metadata reflects the v2 change, not the + // v0 baseline captured by the stale CRC. + assert_eq!( + updated + .table_properties() + .unknown_properties + .get("delta.myKey"), + Some(&"myValue".to_string()) + ); + + Ok(()) + } + + // Setup: commits 0-3 with a metadata change at commit 2; stale CRC@v1 captures the + // pre-change P&M; snapshot_v3 picks up the v2 metadata via log replay. A new + // checkpoint@v1 is then written (at or below snapshot_v3, so the update takes the + // P+M-replay path). The pruned replay on commits > v3 is empty, so without the skip + // it would fall back to CRC@v1 and regress to the pre-v2 configuration. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_incremental_snapshot_preserves_metadata_when_below_checkpoint_crc_is_stale( + ) -> DeltaResult<()> { + let ctx = setup_incremental_snapshot_test()?; + let table_root = ctx.url.as_str(); + + // commit 0: initial P&M with empty configuration. + commit( + table_root, + &ctx.store, + 0, + vec![ + protocol_action(1, 2), + metadata_action(json!({})), + add_action("f0.parquet"), + ], + ) + .await; + // commit 1: data-only add. + commit(table_root, &ctx.store, 1, vec![add_action("f1.parquet")]).await; + // commit 2: metadata change (new configuration property). + commit( + table_root, + &ctx.store, + 2, + vec![ + metadata_action(json!({"delta.myKey": "myValue"})), + add_action("f2.parquet"), + ], + ) + .await; + // commit 3: data-only add. + commit(table_root, &ctx.store, 3, vec![add_action("f3.parquet")]).await; + + // CRC@v1 captures pre-v2 metadata (empty configuration); "stale" relative to the + // existing snapshot's v2-derived P&M. + ctx.store + .put( + &delta_path_for_version(1, "crc"), + make_test_crc_json(200, 2).to_string().into(), + ) + .await?; + + let snapshot_v3 = Snapshot::builder_for(table_root) + .at_version(3) + .build(ctx.engine.as_ref())?; + // Log replay picks up v2's metadata change despite CRC@v1's stale snapshot. + assert_eq!( + snapshot_v3 + .table_properties() + .unknown_properties + .get("delta.myKey"), + Some(&"myValue".to_string()) + ); + + // Write checkpoint@v1 (at or below snapshot_v3, so incremental update takes the + // P+M-replay path instead of a full rebuild). + Snapshot::builder_for(table_root) + .at_version(1) + .build(ctx.engine.as_ref())? + .checkpoint(ctx.engine.as_ref(), None)?; + + // Incremental update. The pruned replay on commits > 3 is empty. With the stale- + // CRC skip, the replay returns (None, None) and `TableConfiguration::try_new_from` + // preserves existing P&M. Without the skip, the inherited CRC@v1 would load its stale + // metadata and overwrite the existing (correct) v2 configuration. + let updated = Snapshot::builder_from(snapshot_v3).build(ctx.engine.as_ref())?; + let fresh = Snapshot::builder_for(table_root).build(ctx.engine.as_ref())?; + compare_snapshots(&updated, &fresh); + + // Observable contract: v2 metadata preserved. + assert_eq!( + updated + .table_properties() + .unknown_properties + .get("delta.myKey"), + Some(&"myValue".to_string()) + ); + + Ok(()) + } + + // Multi-hop: v0 -> v5 (incremental P+M replay) -> v10 (checkpoint-ahead full rebuild). + // Verifies that a CRC carried through the v5 hop is dropped when the new checkpoint + // invalidates it, so the rebuilt v10 snapshot has no CRC. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_incremental_snapshot_multi_hop_replay_then_rebuild_drops_stale_crc( + ) -> DeltaResult<()> { + let ctx = setup_incremental_snapshot_test()?; + setup_test_table_with_commits(ctx.url.as_str(), &ctx.store, 11).await?; + + // CRC at v5 with matching P&M; it is well above any checkpoint (none exists yet), + // so every hop inherits it. + ctx.store + .put( + &delta_path_for_version(5, "crc"), + make_test_crc_json(600, 6).to_string().into(), + ) + .await?; + + // Hop 1: snapshot at v0. + let snapshot_v0 = Snapshot::builder_for(ctx.url.as_str()) + .at_version(0) + .build(ctx.engine.as_ref())?; + assert!(snapshot_v0.crc_at_version().is_none()); + + // Hop 2: incremental v0 -> v5 (new commits, no checkpoint -> P+M replay). This picks up + // CRC@v5 from the listing, which is at the snapshot version. + let snapshot_v5 = Snapshot::builder_from(snapshot_v0) + .at_version(5) + .build(ctx.engine.as_ref())?; + assert_eq!(snapshot_v5.log_segment.checkpoint_version, None); + assert_eq!(snapshot_v5.crc_at_version().map(|c| c.version), Some(5)); + + // Write checkpoint at v7. + Snapshot::builder_for(ctx.url.as_str()) + .at_version(7) + .build(ctx.engine.as_ref())? + .checkpoint(ctx.engine.as_ref(), None)?; + + // Hop 3: incremental v5 -> v10 (discovers a new checkpoint@v7 ahead of the + // existing snapshot version, triggering full rebuild). CRC@v5 is below the new + // checkpoint (5 < 7) so the filter drops it, and the rebuilt snapshot has no CRC. + let snapshot_v10 = Snapshot::builder_from(snapshot_v5).build(ctx.engine.as_ref())?; + let fresh_v10 = Snapshot::builder_for(ctx.url.as_str()).build(ctx.engine.as_ref())?; + compare_snapshots(&snapshot_v10, &fresh_v10); + assert_eq!(snapshot_v10.version(), 10); + assert_eq!(snapshot_v10.log_segment.checkpoint_version, Some(7)); + assert!(snapshot_v10.crc_at_version().is_none()); + + Ok(()) + } + + // test new CRC in new log segment (old log segment has old CRC) + #[tokio::test] + async fn test_snapshot_new_from_crc() -> Result<(), Box> { + let store = Arc::new(InMemory::new()); + let table_root = "memory:///"; + let engine = SyncEngine::new_with_store(store.clone()); + let protocol = |reader_version, writer_version| { + json!({ + "protocol": { + "minReaderVersion": reader_version, + "minWriterVersion": writer_version + } + }) + }; + let metadata = json!({ + "metaData": { + "id":"5fba94ed-9794-4965-ba6e-6ee3c0d22af9", + "format": { + "provider": "parquet", + "options": {} + }, + "schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"val\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}", + "partitionColumns": [], + "configuration": {}, + "createdTime": 1587968585495i64 + } + }); + let commit0 = vec![ + json!({ + "commitInfo": { + "timestamp": 1587968586154i64, + "operation": "WRITE", + "operationParameters": {"mode":"ErrorIfExists","partitionBy":"[]"}, + "isBlindAppend":true + } + }), + protocol(1, 1), + metadata.clone(), + ]; + let commit1 = vec![ + json!({ + "commitInfo": { + "timestamp": 1587968586154i64, + "operation": "WRITE", + "operationParameters": {"mode":"ErrorIfExists","partitionBy":"[]"}, + "isBlindAppend":true + } + }), + protocol(1, 2), + ]; + + // commit 0 and 1 jsons + commit(table_root, &store, 0, commit0.clone()).await; + commit(table_root, &store, 1, commit1).await; + + // Test CRC handling during incremental snapshot update (v0 -> v1). + // The new log listing starts at v1, so the new log segment doesn't find 0.crc. + // a) Only 0.crc exists: resolve_crc_file falls back to old segment's 0.crc. + // b) Both 0.crc and 1.crc exist: resolve_crc_file picks up 1.crc. + let crc = json!({ + "table_size_bytes": 100, + "num_files": 1, + "num_metadata": 1, + "num_protocol": 1, + "metadata": metadata, + "protocol": protocol(1, 1), + }); + + // put the old crc + let path = delta_path_for_version(0, "crc"); + store.put(&path, crc.to_string().into()).await?; + + // base snapshot is at version 0 + let base_snapshot = Snapshot::builder_for(table_root) + .at_version(0) + .build(&engine)?; + + // a) only 0.crc exists -- falls back to old segment's 0.crc + let snapshot = Snapshot::builder_from(base_snapshot.clone()) + .at_version(1) + .build(&engine)?; + assert_eq!( + snapshot + .log_segment + .listed + .latest_crc_file + .as_ref() + .unwrap() + .version, + 0 + ); + + // b) both 0.crc and 1.crc exist; resolve_crc_file picks up 1.crc + let path = delta_path_for_version(1, "crc"); + let crc = json!({ + "table_size_bytes": 100, + "num_files": 1, + "num_metadata": 1, + "num_protocol": 1, + "metadata": metadata, + "protocol": protocol(1, 2), + }); + store.put(&path, crc.to_string().into()).await?; + let snapshot = Snapshot::builder_from(base_snapshot.clone()) + .at_version(1) + .build(&engine)?; + let expected = Snapshot::builder_for(table_root) + .at_version(1) + .build(&engine)?; + assert_eq!(snapshot, expected); + assert_eq!( + snapshot + .log_segment + .listed + .latest_crc_file + .as_ref() + .unwrap() + .version, + 1 + ); + + Ok(()) + } + + // ============================================================================ + // Tests: compaction-file handling on the incremental path + // ============================================================================ + + // TODO(#2337): remove this test when log compaction is re-enabled. + #[tokio::test] + async fn test_compaction_files_ignored_on_read() -> DeltaResult<()> { + let store = Arc::new(InMemory::new()); + let table_root = "memory:///"; + let engine = SyncEngine::new_with_store(store.clone()); + + // Create commits 0-2 and write compaction files to storage + setup_test_table_with_commits(table_root, &store, 3).await?; + write_compaction_file(&store, 0, 1).await?; + write_compaction_file(&store, 0, 2).await?; + + // Compaction files exist on disk but should be skipped during listing + let snapshot = Snapshot::builder_for(table_root).build(&engine)?; + assert!( + snapshot + .log_segment + .listed + .ascending_compaction_files + .is_empty(), + "Compaction files should be ignored when log compaction is disabled" + ); + + Ok(()) + } + + // TODO(#2337): remove this test when log compaction is re-enabled. + #[tokio::test] + async fn test_incremental_snapshot_ignores_compaction_files() -> DeltaResult<()> { + let store = Arc::new(InMemory::new()); + let table_root = "memory:///"; + let engine = SyncEngine::new_with_store(store.clone()); + + // Create commits 0-2 with compaction files in storage + setup_test_table_with_commits(table_root, &store, 3).await?; + write_compaction_file(&store, 0, 1).await?; + write_compaction_file(&store, 0, 2).await?; + + // Build base snapshot at v2 + let snapshot_v2 = Snapshot::builder_for(table_root) + .at_version(2) + .build(&engine)?; + assert_eq!(snapshot_v2.version(), 2); + assert!(snapshot_v2 + .log_segment + .listed + .ascending_compaction_files + .is_empty()); + + // Add commit 3 + commit( + table_root, + &store, + 3, + vec![json!({"add": {"path": "file4.parquet", "partitionValues": {}, "size": 400, "modificationTime": 4000, "dataChange": true}})], + ) + .await; + + // Build v3 incrementally -- compaction files should still be skipped + let snapshot_v3 = Snapshot::builder_from(snapshot_v2) + .at_version(3) + .build(&engine)?; + assert_eq!(snapshot_v3.version(), 3); + assert!(snapshot_v3 + .log_segment + .listed + .ascending_compaction_files + .is_empty()); + + Ok(()) + } + + /// The incremental snapshot path (try_new_from_impl) re-lists files from the checkpoint + /// version onwards. We must ensure that it deduplicates compaction files, since producing + /// duplicates violated the sort invariant in LogSegmentFilesBuilder::build(). + #[tokio::test] + #[ignore = "log compaction disabled (#2337)"] + async fn test_incremental_snapshot_with_compaction_files() -> DeltaResult<()> { + let store = Arc::new(InMemory::new()); + let table_root = "memory:///"; + let engine = SyncEngine::new_with_store(store.clone()); + + // Create commits 0-3 and compaction files (1,1) and (1,2) + setup_test_table_with_commits(table_root, &store, 3).await?; + write_compaction_file(&store, 1, 1).await?; + write_compaction_file(&store, 1, 2).await?; + + // Build snapshot at v2 (includes both compaction files) + let snapshot_v2 = Snapshot::builder_for(table_root) + .at_version(2) + .build(&engine)?; + assert_eq!( + snapshot_v2 + .log_segment + .listed + .ascending_compaction_files + .len(), + 2 + ); + + // Add commit 3 + commit( + table_root, + &store, + 3, + vec![json!({"add": {"path": "file4.parquet", "partitionValues": {}, "size": 400, "modificationTime": 4000, "dataChange": true}})], + ) + .await; + + // Build v3 incrementally - before the fix, this panicked due to duplicate compaction files + let snapshot_v3 = Snapshot::builder_from(snapshot_v2) + .at_version(3) + .build(&engine)?; + + assert_eq!(snapshot_v3.version(), 3); + assert_eq!( + snapshot_v3 + .log_segment + .listed + .ascending_compaction_files + .len(), + 2 + ); + + Ok(()) + } + + /// This test documents a limitation: When deduplicating compactions, the deduplication logic + /// only checks the start version (lo), not the hi version. So a new compaction file (1,3) + /// added after building the base snapshot at v2 gets filtered out because its start version + /// (1) <= existing_snapshot_version (2). + #[tokio::test] + #[ignore = "log compaction disabled (#2337)"] + async fn test_incremental_snapshot_with_new_compaction_files() -> DeltaResult<()> { + let store = Arc::new(InMemory::new()); + let table_root = "memory:///"; + let engine = SyncEngine::new_with_store(store.clone()); + + // Create commits 0-3 and compaction files (1,2) and (2,2) + setup_test_table_with_commits(table_root, &store, 4).await?; + write_compaction_file(&store, 1, 2).await?; + write_compaction_file(&store, 2, 2).await?; + + // Build snapshot at v2 + let snapshot_v2 = Snapshot::builder_for(table_root) + .at_version(2) + .build(&engine)?; + assert_eq!( + snapshot_v2 + .log_segment + .listed + .ascending_compaction_files + .len(), + 2 + ); + + // Add new compaction file (1,3) after building the base snapshot + write_compaction_file(&store, 1, 3).await?; + + // Build v3 incrementally - the new (1,3) file gets filtered out because + // the deduplication only looks at start version: 1 <= existing_snapshot_version (2) + let snapshot_v3 = Snapshot::builder_from(snapshot_v2) + .at_version(3) + .build(&engine)?; + + assert_eq!(snapshot_v3.version(), 3); + assert_eq!( + snapshot_v3 + .log_segment + .listed + .ascending_compaction_files + .len(), + 2 + ); + + // Verify we still have the original (1,2) and (2,2) files + let versions_and_his: Vec<_> = snapshot_v3 + .log_segment + .listed + .ascending_compaction_files + .iter() + .map(|p| match p.file_type { + LogPathFileType::CompactedCommit { hi } => (p.version, hi), + _ => panic!("Expected CompactedCommit"), + }) + .collect(); + assert_eq!(versions_and_his, vec![(1, 2), (2, 2)]); + + Ok(()) + } + + // ============================================================================ + // ProtocolMetadataLoad source classification + // ============================================================================ + + /// Return a clone of the one event `extract` matches, asserting exactly one matches. + fn single_event( + events: &[MetricEvent], + extract: impl Fn(&MetricEvent) -> Option<&T>, + ) -> T { + let mut it = events.iter().filter_map(&extract); + let found = it.next().expect("expected an event of this type"); + assert!(it.next().is_none(), "expected exactly one matching event"); + found.clone() + } + + fn protocol_metadata_load(events: &[MetricEvent]) -> ProtocolMetadataLoadSuccess { + single_event(events, |e| match e { + MetricEvent::ProtocolMetadataLoadSuccess(s) => Some(s), + _ => None, + }) + } + + fn log_segment_load(events: &[MetricEvent]) -> LogSegmentLoadSuccess { + single_event(events, |e| match e { + MetricEvent::LogSegmentLoadSuccess(s) => Some(s), + _ => None, + }) + } + + #[rstest] + #[case::no_crc(None, IncrementalReplay::Disabled, ProtocolMetadataSource::FullReplay)] + #[case::crc_at_target( + Some(3), + IncrementalReplay::Disabled, + ProtocolMetadataSource::CrcAtTarget + )] + #[case::stale_crc_advanced( + Some(1), + IncrementalReplay::Unlimited, + ProtocolMetadataSource::CrcAdvancedByReplay + )] + #[case::stale_crc_seeds_replay( + Some(1), + IncrementalReplay::Disabled, + ProtocolMetadataSource::CrcSeededPmOnlyReplay + )] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_fresh_build_classifies_protocol_metadata_source( + #[case] planted_crc_version: Option, + #[case] mode: IncrementalReplay, + #[case] expected_source: ProtocolMetadataSource, + ) -> DeltaResult<()> { + let ctx = setup_incremental_snapshot_test()?; + setup_test_table_with_commits(ctx.url.as_str(), &ctx.store, 4).await?; + if let Some(v) = planted_crc_version { + ctx.store + .put( + &delta_path_for_version(v, "crc"), + make_test_crc_json(200, 2).to_string().into(), + ) + .await?; + } + + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let corr = "req-abc"; + let _snapshot = Snapshot::builder_for(ctx.url.as_str()) + .at_version(3) + .with_incremental_crc_replay(mode) + .with_correlation_id(corr) + .build(ctx.engine.as_ref())?; + + let events = reporter.events(); + let pm = protocol_metadata_load(&events); + assert_eq!(pm.source, expected_source); + assert_eq!(pm.load_type, LogSegmentLoadType::Full); + + let seg = log_segment_load(&events); + assert_eq!(seg.load_type, LogSegmentLoadType::Full); + // The v0-v3 table has 4 commits, no checkpoint or compaction. + assert_eq!(seg.num_commit_files, 4); + assert_eq!(seg.num_checkpoint_files, 0); + assert_eq!(seg.num_compaction_files, 0); + // No CRC decodes to None; a CRC planted at version v is `3 - v` versions behind the + // loaded version 3. + assert_eq!(seg.crc_versions_behind, planted_crc_version.map(|v| 3 - v)); + + // The operation_id and correlation_id survive the emit->decode round-trip and one + // operation_id joins both events. + assert_eq!(seg.correlation_id.as_deref(), Some(corr)); + assert_ne!(seg.operation_id, MetricId::nil()); + assert_eq!(seg.operation_id, pm.operation_id); + + // The CRC-advance replay cost is folded into the P&M duration on the advanced case. + if expected_source == ProtocolMetadataSource::CrcAdvancedByReplay { + assert!(pm.duration > std::time::Duration::ZERO); + } + Ok(()) + } + + #[rstest] + #[case::no_crc(None, IncrementalReplay::Disabled, ProtocolMetadataSource::FullReplay)] + #[case::crc_at_target( + Some(5), + IncrementalReplay::Disabled, + ProtocolMetadataSource::CrcAtTarget + )] + #[case::stale_crc_advanced( + Some(1), + IncrementalReplay::Unlimited, + ProtocolMetadataSource::CrcAdvancedByReplay + )] + // A CRC newer than the existing snapshot (v3) but below the target (v5), with advance + // disabled, seeds a pruned replay rather than being used at target or advanced. + #[case::stale_crc_seeds_replay( + Some(4), + IncrementalReplay::Disabled, + ProtocolMetadataSource::CrcSeededPmOnlyReplay + )] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_incremental_build_classifies_protocol_metadata_source( + #[case] planted_crc_version: Option, + #[case] mode: IncrementalReplay, + #[case] expected_source: ProtocolMetadataSource, + ) -> DeltaResult<()> { + let ctx = setup_incremental_snapshot_test()?; + setup_test_table_with_commits(ctx.url.as_str(), &ctx.store, 6).await?; + let base = Snapshot::builder_for(ctx.url.as_str()) + .at_version(3) + .build(ctx.engine.as_ref())?; + if let Some(v) = planted_crc_version { + ctx.store + .put( + &delta_path_for_version(v, "crc"), + make_test_crc_json(200, 2).to_string().into(), + ) + .await?; + } + + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let updated = Snapshot::builder_from(base) + .at_version(5) + .with_incremental_crc_replay(mode) + .build(ctx.engine.as_ref())?; + assert_eq!(updated.version(), 5); + + let events = reporter.events(); + let pm = protocol_metadata_load(&events); + assert_eq!(pm.source, expected_source); + assert_eq!(pm.load_type, LogSegmentLoadType::Incremental); + let seg = log_segment_load(&events); + assert_eq!(seg.load_type, LogSegmentLoadType::Incremental); + Ok(()) + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_incremental_build_no_pm_change_classifies_full_replay() -> DeltaResult<()> { + let ctx = setup_incremental_snapshot_test()?; + setup_test_table_with_commits(ctx.url.as_str(), &ctx.store, 4).await?; + + let base = Snapshot::builder_for(ctx.url.as_str()) + .at_version(1) + .build(ctx.engine.as_ref())?; + + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let updated = Snapshot::builder_from(base).build(ctx.engine.as_ref())?; + assert_eq!(updated.version(), 3); + + // No P&M change in the new commits, no newer CRC: the replay finds nothing above the + // existing snapshot, so the source is the not-CRC-rooted `FullReplay`. + let pm = protocol_metadata_load(&reporter.events()); + assert_eq!(pm.source, ProtocolMetadataSource::FullReplay); + Ok(()) + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_fresh_build_missing_protocol_metadata_emits_failure() -> DeltaResult<()> { + let ctx = setup_incremental_snapshot_test()?; + // A commit with only an add action: no protocol or metadata anywhere in the log. + commit( + ctx.url.as_str(), + &ctx.store, + 0, + vec![add_action("f0.parquet")], + ) + .await; + + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let result = Snapshot::builder_for(ctx.url.as_str()).build(ctx.engine.as_ref()); + assert!(result.is_err()); + + let events = reporter.events(); + assert_eq!( + events + .iter() + .filter(|e| matches!(e, MetricEvent::ProtocolMetadataLoadFailure(_))) + .count(), + 1, + "expected exactly one ProtocolMetadataLoadFailure" + ); + assert!( + !events + .iter() + .any(|e| matches!(e, MetricEvent::ProtocolMetadataLoadSuccess(_))), + "no success event should fire on the failure path" + ); + Ok(()) + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_incremental_build_replay_error_emits_failure() -> DeltaResult<()> { + let ctx = setup_incremental_snapshot_test()?; + setup_test_table_with_commits(ctx.url.as_str(), &ctx.store, 6).await?; + let base = Snapshot::builder_for(ctx.url.as_str()) + .at_version(3) + .build(ctx.engine.as_ref())?; + + // Corrupt a commit above the base so the incremental P&M replay of `(3, 5]` errors. + ctx.store + .put(&delta_path_for_version(4, "json"), "{ not json".into()) + .await?; + + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let result = Snapshot::builder_from(base) + .at_version(5) + .build(ctx.engine.as_ref()); + assert!(result.is_err()); + + let events = reporter.events(); + assert_eq!( + events + .iter() + .filter(|e| matches!(e, MetricEvent::ProtocolMetadataLoadFailure(_))) + .count(), + 1, + "expected exactly one ProtocolMetadataLoadFailure" + ); + assert!( + !events + .iter() + .any(|e| matches!(e, MetricEvent::ProtocolMetadataLoadSuccess(_))), + "no success event should fire on the failure path" + ); + Ok(()) + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_incremental_version_regression_emits_log_segment_load_failure() -> DeltaResult<()> + { + let ctx = setup_incremental_snapshot_test()?; + setup_test_table_with_commits(ctx.url.as_str(), &ctx.store, 6).await?; // v0..=v5 + let base = Snapshot::builder_for(ctx.url.as_str()) + .at_version(5) + .build(ctx.engine.as_ref())?; + + // Simulate commits incorrectly deleted from the log: re-listing now tops out below v5, so + // the new segment's end version is older than the base and the regression branch fires. + ctx.store.delete(&delta_path_for_version(5, "json")).await?; + ctx.store.delete(&delta_path_for_version(4, "json")).await?; + + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let result = Snapshot::builder_from(base).build(ctx.engine.as_ref()); + assert!(result.is_err()); + + let events = reporter.events(); + let failure = events + .iter() + .find_map(|e| match e { + MetricEvent::LogSegmentLoadFailure(f) => Some(f), + _ => None, + }) + .expect("expected a LogSegmentLoadFailure"); + assert_eq!(failure.load_type, LogSegmentLoadType::Incremental); + assert!( + !events + .iter() + .any(|e| matches!(e, MetricEvent::LogSegmentLoadSuccess(_))), + "no success event should fire on the failure path" + ); + Ok(()) + } + + // Case C.1: requesting a version beyond the log errors and emits a LogSegmentLoadFailure, + // matching the fresh path (which also fails a request for an unavailable version). + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_incremental_unavailable_version_emits_log_segment_load_failure() -> DeltaResult<()> + { + let ctx = setup_incremental_snapshot_test()?; + setup_test_table_with_commits(ctx.url.as_str(), &ctx.store, 4).await?; // v0..=v3 + let base = Snapshot::builder_for(ctx.url.as_str()) + .at_version(3) + .build(ctx.engine.as_ref())?; + + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + // Request a version beyond the log: the listing above v3 is empty (case C.1). + let result = Snapshot::builder_from(base) + .at_version(9) + .build(ctx.engine.as_ref()); + assert!(result.is_err()); + + let events = reporter.events(); + let failure = events + .iter() + .find_map(|e| match e { + MetricEvent::LogSegmentLoadFailure(f) => Some(f), + _ => None, + }) + .expect("expected a LogSegmentLoadFailure"); + assert_eq!(failure.load_type, LogSegmentLoadType::Incremental); + assert!( + !events + .iter() + .any(|e| matches!(e, MetricEvent::LogSegmentLoadSuccess(_))), + "no success event should fire on the failure path" + ); + Ok(()) + } + + // Case D.1 (checkpoint strictly ahead of the base) rebuilds via the fresh emitter, but the + // load was requested incrementally, so both events must still report Incremental. + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn test_incremental_rebuild_reports_incremental_load_type() -> DeltaResult<()> { + let ctx = setup_incremental_snapshot_test()?; + setup_test_table_with_commits(ctx.url.as_str(), &ctx.store, 4).await?; + + let base = Snapshot::builder_for(ctx.url.as_str()) + .at_version(1) + .build(ctx.engine.as_ref())?; + + // Checkpoint at v3, strictly ahead of the base at v1: the incremental update takes the + // Case D.1 rebuild path. + Snapshot::builder_for(ctx.url.as_str()) + .at_version(3) + .build(ctx.engine.as_ref())? + .checkpoint(ctx.engine.as_ref(), None)?; + + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let updated = Snapshot::builder_from(base).build(ctx.engine.as_ref())?; + assert_eq!(updated.version(), 3); + assert_eq!(updated.log_segment.checkpoint_version, Some(3)); + + let events = reporter.events(); + assert_eq!( + log_segment_load(&events).load_type, + LogSegmentLoadType::Incremental + ); + assert_eq!( + protocol_metadata_load(&events).load_type, + LogSegmentLoadType::Incremental + ); + Ok(()) + } +} diff --git a/kernel/src/snapshot/mod.rs b/kernel/src/snapshot/mod.rs new file mode 100644 index 0000000000..d4f5170848 --- /dev/null +++ b/kernel/src/snapshot/mod.rs @@ -0,0 +1,2575 @@ +//! In-memory representation of snapshots of tables (snapshot is a table at given point in time, it +//! has schema etc.) + +use std::collections::{HashMap, HashSet}; +use std::sync::Arc; + +use delta_kernel_derive::internal_api; +use tracing::{debug, info, instrument, warn}; +use url::Url; + +use crate::action_reconciliation::calculate_transaction_expiration_timestamp; +use crate::actions::set_transaction::SetTransactionScanner; +use crate::actions::{DomainMetadata, INTERNAL_DOMAIN_PREFIX}; +use crate::checkpoint::{ + CheckpointSpec, CheckpointWriter, V2CheckpointConfig, DEFAULT_FILE_ACTIONS_PER_SIDECAR_HINT, +}; +use crate::clustering::{parse_clustering_columns, ClusteringColumnInfo, CLUSTERING_DOMAIN_NAME}; +use crate::committer::{Committer, PublishMetadata}; +use crate::crc::{ + try_write_crc_file, Crc, CrcDelta, DomainMetadataState, FileSizeHistogram, FileStats, + SetTransactionState, +}; +use crate::expressions::ColumnName; +use crate::incremental_scan::IncrementalScanBuilder; +use crate::log_segment::{DomainMetadataMap, LogSegment}; +use crate::metrics::events::{DOMAIN_METADATA_LOADED_SPAN, SET_TRANSACTION_LOADED_SPAN}; +use crate::metrics::{ + emit_protocol_metadata_load, emit_protocol_metadata_load_failure, SnapshotLoadMetricContext, +}; +use crate::path::ParsedLogPath; +use crate::scan::ScanBuilder; +use crate::schema::SchemaRef; +use crate::table_configuration::{InCommitTimestampEnablement, TableConfiguration}; +use crate::table_features::{physical_to_logical_column_name_and_type, TableFeature}; +use crate::table_properties::TableProperties; +use crate::transaction::builder::alter_table::AlterTableTransactionBuilder; +use crate::transaction::Transaction; +use crate::utils::require; +use crate::{DeltaResult, Engine, Error, LogCompactionWriter, Version}; + +mod builder; +mod incremental; +mod snapshot_crc; +pub use builder::{IncrementalReplay, SnapshotBuilder}; +use snapshot_crc::SnapshotCrc; + +/// A shared, thread-safe reference to a [`Snapshot`]. +pub type SnapshotRef = Arc; + +/// Result of attempting to write a version checksum (CRC) file. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ChecksumWriteResult { + /// A CRC file already exists at this version. Per the Delta protocol, writers MUST NOT + /// overwrite existing version checksum files. + AlreadyExists, + /// The CRC file was successfully written to storage. + Written, +} + +/// Result of attempting to write a checkpoint file. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum CheckpointWriteResult { + /// A checkpoint already exists at this version. + AlreadyExists, + /// The checkpoint was successfully written to storage. + Written, +} + +// TODO expose methods for accessing the files of a table (with file pruning). +/// In-memory representation of a specific snapshot of a Delta table. While a `DeltaTable` exists +/// throughout time, `Snapshot`s represent a view of a table at a specific point in time; they +/// have a defined schema (which may change over time for any given table), specific version, and +/// frozen log segment. +pub struct Snapshot { + span: tracing::Span, + log_segment: LogSegment, + table_configuration: TableConfiguration, + /// The newest CRC resolved at construction, kept so later queries and CRC writes reuse it + /// instead of re-reading from disk. Queried via [`Snapshot::crc_at_version`] (authoritative, + /// zero-I/O) or [`Snapshot::base_crc`] (possibly stale). See [`SnapshotCrc`]. + crc: SnapshotCrc, + /// Best-effort "confirmed latest at build time" flag. See [`Snapshot::built_as_latest`]. + built_as_latest: bool, +} + +impl PartialEq for Snapshot { + // Content equality: `built_as_latest` is best-effort build metadata, deliberately excluded. + fn eq(&self, other: &Self) -> bool { + self.log_segment == other.log_segment + && self.table_configuration == other.table_configuration + } +} + +impl Eq for Snapshot {} + +impl Drop for Snapshot { + fn drop(&mut self) { + debug!("Dropping snapshot"); + } +} + +impl std::fmt::Debug for Snapshot { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Snapshot") + .field("path", &self.log_segment.log_root.as_str()) + .field("version", &self.version()) + .field("metadata", &self.table_configuration().metadata()) + .field("log_segment", &self.log_segment) + .finish() + } +} + +/// Build a [`Error::ChecksumWriteUnsupported`] for a resolution root that could not yield a +/// writable CRC. `reason` completes the sentence "Cannot resolve a CRC to write: ...". +fn unresolved_crc(reason: &str) -> Error { + Error::ChecksumWriteUnsupported(format!("Cannot resolve a CRC to write: {reason}")) +} + +impl Snapshot { + // ============================================================================ + // Construction entry points + // ============================================================================ + + /// Create a new [`SnapshotBuilder`] to build a new [`Snapshot`] for a given table root. If you + /// instead have an existing [`Snapshot`] you would like to do minimal work to update, consider + /// using [`Snapshot::builder_from`] instead. + pub fn builder_for(table_root: impl AsRef) -> SnapshotBuilder { + SnapshotBuilder::new_for(table_root) + } + + /// Create a new [`SnapshotBuilder`] to incrementally update an existing [`Snapshot`] to a + /// more recent version. + /// + /// See `Snapshot::try_new_from` for the case-by-case behavior. + pub fn builder_from(existing_snapshot: SnapshotRef) -> SnapshotBuilder { + SnapshotBuilder::new_from(existing_snapshot) + } + + // ============================================================================ + // Internal constructors + // ============================================================================ + + /// Create a new [`Snapshot`] from a [`LogSegment`] and [`TableConfiguration`]. + #[internal_api] + #[allow(unused)] + pub(crate) fn new( + log_segment: LogSegment, + table_configuration: TableConfiguration, + ) -> DeltaResult { + Self::new_with_crc( + log_segment, + table_configuration, + None, /* crc */ + false, /* built_as_latest */ + ) + } + + /// Internal constructor that accepts an explicit pre-resolved CRC. + /// + /// `built_as_latest` records whether the build confirmed this is the latest version + /// (best-effort). See [`Snapshot::built_as_latest`]. + pub(crate) fn new_with_crc( + log_segment: LogSegment, + table_configuration: TableConfiguration, + crc: Option>, + built_as_latest: bool, + ) -> DeltaResult { + // Will perform version validations. + let crc = SnapshotCrc::try_new( + crc, + table_configuration.version(), + log_segment.checkpoint_version, + )?; + let span = tracing::info_span!( + parent: tracing::Span::none(), + "snap", + path = %table_configuration.table_root(), + version = table_configuration.version(), + ); + info!(parent: &span, "Created snapshot"); + Ok(Self { + span, + log_segment, + table_configuration, + crc, + built_as_latest, + }) + } + + /// Create a new [`Snapshot`] from a freshly-listed [`LogSegment`]. Takes Protocol and Metadata + /// from the latest on-disk CRC, advanced to the segment's end version when `incremental_replay` + /// permits, or used to root Protocol and Metadata log replay otherwise. Falls back to full log + /// replay when no CRC is present. + #[instrument(err, fields(version, operation_id = %metric_context.operation_id, correlation_id = metric_context.correlation_id.as_deref().unwrap_or("")), skip(engine))] + fn try_new_from_log_segment( + location: Url, + log_segment: LogSegment, + engine: &dyn Engine, + metric_context: SnapshotLoadMetricContext, + incremental_replay: IncrementalReplay, + built_as_latest: bool, + ) -> DeltaResult { + let pm_start = std::time::Instant::now(); + + // Step 1: read the latest on-disk CRC and, if usable, advance it to the end version + // (or use it as-is when already there) per `incremental_replay`. + let base_crc = log_segment.read_latest_crc(engine); + let crc_at_version = log_segment + .try_build_crc_within_budget(engine, base_crc.as_ref(), incremental_replay) + .inspect_err(|_| emit_protocol_metadata_load_failure(&metric_context))?; + + // Step 2: P&M from that CRC, else log replay rooted at the base CRC, checkpoint, or + // first commit. The replay reports its own source (seeded vs full). + let (metadata, protocol, source) = match &crc_at_version { + Some((crc, source)) => (crc.metadata.clone(), crc.protocol.clone(), *source), + None => log_segment + .read_protocol_metadata(engine, base_crc.as_ref()) + .inspect_err(|_| emit_protocol_metadata_load_failure(&metric_context))?, + }; + emit_protocol_metadata_load(&metric_context, source, pm_start.elapsed()); + + let table_configuration = + TableConfiguration::try_new(metadata, protocol, location, log_segment.end_version)?; + + tracing::Span::current().record("version", table_configuration.version()); + + let crc = crc_at_version.map(|(crc, _)| crc).or(base_crc); + Self::new_with_crc(log_segment, table_configuration, crc, built_as_latest) + } + + /// Creates a new [`Snapshot`] representing the table state immediately after a commit. + /// + /// Appends the newly committed file to this snapshot's log segment and bumps the version, + /// producing a post-commit snapshot without a full log replay from storage. + /// + /// The `crc_delta` captures the CRC-relevant changes from the committed transaction + /// (file stats, domain metadata, ICT, etc.). If this snapshot had a CRC at its version, + /// the delta is applied to produce a precomputed in-memory CRC for the new version, + /// avoiding re-reading metadata from storage. A stale base is carried forward unchanged; + /// no CRC stays no CRC. CREATE TABLE handles CRC construction separately in + /// `Transaction::into_committed`. + pub(crate) fn new_post_commit( + &self, + commit: ParsedLogPath, + crc_delta: CrcDelta, + ) -> DeltaResult { + require!( + commit.is_commit(), + Error::internal_error(format!( + "Cannot create post-commit Snapshot. Log file is not a commit file. \ + Path: {}, Type: {:?}.", + commit.location.location, commit.file_type + )) + ); + let read_version = self.version(); + let new_version = commit.version; + require!( + new_version == read_version.wrapping_add(1), + Error::internal_error(format!( + "Cannot create post-commit Snapshot. Log file version ({new_version}) does not \ + equal Snapshot version ({read_version}) + 1." + )) + ); + + let new_table_configuration = TableConfiguration::new_post_commit( + self.table_configuration(), + new_version, + crc_delta.metadata.clone(), + crc_delta.protocol.clone(), + )?; + + let new_log_segment = self.log_segment.new_with_commit_appended(commit)?; + + // `crc_delta` covers what the transaction committed, so it advances an at-version CRC; + // a stale base is carried forward unadvanced. + let new_crc = match self.crc_at_version() { + Some(base) => Some(Arc::new( + base.as_ref().clone().apply(crc_delta, new_version), + )), + None => self.base_crc().cloned(), + }; + + // A successful commit means the post-commit snapshot is the latest version. + Snapshot::new_with_crc( + new_log_segment, + new_table_configuration, + new_crc, + true, /* built_as_latest */ + ) + } + + // ============================================================================ + // Field accessors and state queries + // ============================================================================ + + /// Log segment this snapshot uses + #[internal_api] + pub(crate) fn log_segment(&self) -> &LogSegment { + &self.log_segment + } + + /// The CRC iff it sits exactly at this snapshot's version. When `Some`, queries backed by the + /// CRC hit cache at zero I/O. Returns `None` when no CRC was resolved or the resolved CRC is + /// stale. + #[internal_api] + pub(crate) fn crc_at_version(&self) -> Option<&Arc> { + self.crc.at_version() + } + + /// The held CRC regardless of version, for reuse that does not require an at-version CRC + /// (e.g. checksum writes). Prefer [`Self::crc_at_version`] for authoritative queries. + fn base_crc(&self) -> Option<&Arc> { + self.crc.base() + } + + pub fn table_root(&self) -> &Url { + self.table_configuration.table_root() + } + + /// Version of this `Snapshot` in the table. + pub fn version(&self) -> Version { + self.table_configuration().version() + } + + /// Whether this snapshot was at the latest table version when it was built. + /// + /// This is best-effort: `true` when the build knows this was the newest version. That holds for + /// a build (fresh or incremental) with no time-travel version, a build (fresh or incremental) + /// at the catalog's ratified latest version, and a post-commit snapshot. It is not a + /// liveness guarantee: another writer may commit a newer version afterward, so a `true` + /// snapshot can already be stale. + /// + /// Version-preserving derivations ([`Self::checkpoint`], [`Self::write_checksum`], + /// [`Self::publish`]) do not change this flag: they carry it over from the source snapshot. + pub fn is_built_as_latest(&self) -> bool { + self.built_as_latest + } + + /// Table [`Schema`] at this `Snapshot`s version. + /// + /// [`Schema`]: crate::schema::Schema + pub fn schema(&self) -> SchemaRef { + self.table_configuration.logical_schema() + } + + /// Estimated owned heap size in bytes for this snapshot. Best-effort estimate + /// for capacity tracking, not authoritative. + /// + /// Counts only the dominant per-snapshot heap contributors, normally > 70% of the snapshot's + /// owned heap size: + /// - For every listed log path (commit, compaction, checkpoint, latest CRC, latest commit): the + /// filename / extension / Url string heap. + /// - Vec buffer capacity (`capacity * size_of::()`) for the three Vec fields on + /// `LogSegmentFiles`. + /// - The log root Url string. + /// - The raw `schemaString` JSON on table metadata. + /// + /// The Arc-shared variables (e.g. logical/physical schemas, `crc`) are not counted, + /// as they can be shared between multiple snapshots and are not owned by a single snapshot. + /// + /// Other variables' contributions to heap size are relatively small, so they are not counted + /// here. + /// + /// Runs in O(n) over listed log files. + /// + /// TODO(#2757): Optimize the estimation accuracy. + pub fn estimated_owned_heap_size_bytes(&self) -> usize { + self.log_segment.listed.estimated_heap_size_bytes() + + self.log_segment.log_root.as_str().len() + + self + .table_configuration() + .metadata() + .schema_string() + .capacity() + } + + /// Get the [`TableProperties`] for this [`Snapshot`]. + pub fn table_properties(&self) -> &TableProperties { + self.table_configuration().table_properties() + } + + /// Returns the protocol-derived table properties as a map of key-value pairs. + /// + /// This includes: + /// - `delta.minReaderVersion` and `delta.minWriterVersion` + /// - `delta.feature. = "supported"` for each reader and writer feature (when using table + /// features protocol, i.e. reader version 3 / writer version 7) + #[allow(unused)] + #[internal_api] + pub(crate) fn get_protocol_derived_properties(&self) -> HashMap { + let protocol = self.table_configuration().protocol(); + + let mut properties = HashMap::from([ + ( + "delta.minReaderVersion".into(), + protocol.min_reader_version().to_string(), + ), + ( + "delta.minWriterVersion".into(), + protocol.min_writer_version().to_string(), + ), + ]); + + let features = protocol + .reader_features() + .into_iter() + .flatten() + .chain(protocol.writer_features().into_iter().flatten()); + + for feature in features { + properties + .entry(format!("delta.feature.{}", feature.as_ref())) + .or_insert_with(|| "supported".to_string()); + } + + properties + } + + /// Get the raw metadata configuration for this table. + /// + /// This returns the `Metadata.configuration` map as stored in the Delta log, containing + /// user-defined properties, delta table properties (e.g., `delta.enableInCommitTimestamps`), + /// and application-specific properties (e.g., `io.unitycatalog.tableId`). + #[allow(unused)] + #[internal_api] + pub(crate) fn metadata_configuration(&self) -> &HashMap { + self.table_configuration().metadata().configuration() + } + + /// Get the [`TableConfiguration`] for this [`Snapshot`]. + #[internal_api] + pub(crate) fn table_configuration(&self) -> &TableConfiguration { + &self.table_configuration + } + + /// Fetch the latest version of the provided `application_id` for this snapshot. Filters the + /// txn based on the delta.setTransactionRetentionDuration property and lastUpdated. + /// + /// Serves from an at-version CRC when it has the app_id (or authoritatively lacks it); else + /// roots a tail-only scan in a stale but `Complete` CRC, skipping the checkpoint; else full log + /// replay. + /// + /// Reports metrics: `SetTransactionLoadSuccess` or `SetTransactionLoadFailure`. + // TODO: add a get_app_id_versions to fetch all at once using SetTransactionScanner::get_all + #[instrument( + parent = &self.span, + name = SET_TRANSACTION_LOADED_SPAN, + skip_all, + err, + fields(report, from_cache, found) + )] + pub fn get_app_id_version( + &self, + application_id: &str, + engine: &dyn Engine, + ) -> DeltaResult> { + fn record_metric(from_cache: bool, found: bool) { + let span = tracing::Span::current(); + span.record("from_cache", from_cache); + span.record("found", found); + } + + let expiration_timestamp = + calculate_transaction_expiration_timestamp(self.table_properties())?; + + // Fast path: serve from CRC if available at this version. + if let Some(crc) = self.crc_at_version() { + match &crc.set_transaction_state { + SetTransactionState::Complete(map) => { + // Complete is authoritative: a miss means the app_id has no transaction. + let version = map + .get(application_id) + .and_then(|txn| txn.non_expired_version(expiration_timestamp)); + record_metric(true, version.is_some()); + return Ok(version); + } + SetTransactionState::Partial(map) => { + // Hit is authoritative; miss falls through to log replay below. + if let Some(txn) = map.get(application_id) { + let version = txn.non_expired_version(expiration_timestamp); + record_metric(true, version.is_some()); + return Ok(version); + } + } + } + } + + // A stale but authoritative (`Complete`) CRC roots a tail-only scan over the commits after + // it, skipping the checkpoint. A stale `Partial` CRC (one whose file lacked the + // `setTransactions` field) has no authoritative transaction set, so it cannot root the + // scan and falls through to the full replay below. + if let Some(base) = self.base_crc() { + if let SetTransactionState::Complete(base_active) = &base.set_transaction_state { + let txn = SetTransactionScanner::get_one_rooted_in_crc( + self.log_segment(), + application_id, + base_active, + base.version, + engine, + )?; + let version = txn.and_then(|txn| txn.non_expired_version(expiration_timestamp)); + // TODO: report a distinct metric source here. A rooted tail scan is neither a + // cache hit nor a full replay, yet `from_cache = false` buckets it with + // full replay, hiding the checkpoint-skipping win. A 3-variant source enum + // (cache / crc-rooted / full-replay) would let metrics measure it. + record_metric(false, version.is_some()); + return Ok(version); + } + } + + // Fallback: full log replay. Scan for the newest txn and apply expiration to it, like the + // CRC paths above. + let txn = SetTransactionScanner::get_one(self.log_segment(), application_id, engine)?; + let version = txn.and_then(|txn| txn.non_expired_version(expiration_timestamp)); + record_metric(false, version.is_some()); + Ok(version) + } + + /// Fetch the domainMetadata for a specific domain in this snapshot. This returns the latest + /// configuration for the domain, or None if the domain does not exist. + /// + /// Note that this method performs log replay (fetches and processes metadata from storage). + pub fn get_domain_metadata( + &self, + domain: &str, + engine: &dyn Engine, + ) -> DeltaResult> { + if domain.starts_with(INTERNAL_DOMAIN_PREFIX) { + return Err(Error::generic( + "User DomainMetadata are not allowed to use system-controlled 'delta.*' domain", + )); + } + + self.get_domain_metadata_internal(domain, engine) + } + + /// Get per-clustering-column descriptors for this snapshot, if clustering is enabled. + /// + /// Returns `Ok(Some(infos))` when the ClusteredTable feature is enabled and the + /// `delta.clustering` domain has a current entry (`infos` may be empty if the entry lists + /// none), `Ok(None)` when the feature is absent or the domain has no current entry, or an + /// error if the clustering metadata is malformed. Descriptors are returned in the order the + /// columns appear in the domain. + /// + /// Each `ClusteringColumnInfo` carries the physical reference (as stored in the domain), the + /// logical reference resolved against the table schema, and the column's data type. + /// + /// Note that this method performs log replay (fetches and processes metadata from storage). + /// + /// # Errors + /// + /// Returns an error if the clustering domain metadata is malformed, or if a physical + /// column name cannot be resolved to a logical name in the schema. + #[internal_api] + pub(crate) fn get_clustering_column_infos( + &self, + engine: &dyn Engine, + ) -> DeltaResult>> { + let Some(physical_columns) = self.get_physical_clustering_columns(engine)? else { + return Ok(None); + }; + let column_mapping_mode = self.table_configuration.column_mapping_mode(); + let logical_schema = self.table_configuration.logical_schema(); + let infos = physical_columns + .into_iter() + .map(|physical_col| { + let (logical_col, data_type) = physical_to_logical_column_name_and_type( + &logical_schema, + &physical_col, + column_mapping_mode, + )?; + Ok(ClusteringColumnInfo { + physical_column: physical_col, + logical_column: logical_col, + data_type, + }) + }) + .collect::>>()?; + Ok(Some(infos)) + } + + /// Get the clustering columns for this snapshot, if the table has clustering enabled. + /// + /// Returns `Ok(Some(columns))` when the ClusteredTable feature is enabled and the + /// `delta.clustering` domain has a current entry (`columns` may be empty if the entry lists + /// none), `Ok(None)` when the feature is absent or the domain has no current entry, or an + /// error if the clustering metadata is malformed. + /// + /// The columns are returned as physical column names, respecting the column mapping mode. + /// Note that this method performs log replay (fetches and processes metadata from storage). + #[internal_api] + pub(crate) fn get_physical_clustering_columns( + &self, + engine: &dyn Engine, + ) -> DeltaResult>> { + match self.get_clustering_domain_metadata(engine)? { + Some(config) => Ok(Some(parse_clustering_columns(&config)?)), + None => Ok(None), + } + } + + /// Read the raw JSON configuration of the `delta.clustering` domain. + /// + /// Returns `Ok(None)` when the `ClusteredTable` feature is absent on the protocol, or when + /// the domain has no current entry. The JSON has the shape + /// `{"clusteringColumns":[["col1"],["addr","city"], ...]}` with physical column names. + pub(crate) fn get_clustering_domain_metadata( + &self, + engine: &dyn Engine, + ) -> DeltaResult> { + if !self + .table_configuration + .protocol() + .has_table_feature(&TableFeature::ClusteredTable) + { + return Ok(None); + } + self.get_domain_metadata_internal(CLUSTERING_DOMAIN_NAME, engine) + } + + /// Load domain metadata: if Complete in the CRC, answer from the cache; else if every + /// requested domain is in a Partial cache, also answer from the cache; else if a stale Complete + /// CRC is held, scan only the commits after it and reconcile against its map; else full log + /// replay. `domains == None` means load all. + /// + /// Reports metrics: `DomainMetadataLoadSuccess` or `DomainMetadataLoadFailure`. + #[instrument( + parent = &self.span, + name = DOMAIN_METADATA_LOADED_SPAN, + skip_all, + err, + fields(report, from_cache, num_domains_returned) + )] + #[internal_api] + pub(crate) fn get_domain_metadatas_internal( + &self, + engine: &dyn Engine, + domains: Option<&HashSet<&str>>, + ) -> DeltaResult { + fn record_metric(from_cache: bool, num_domains_returned: usize) { + let span = tracing::Span::current(); + span.record("from_cache", from_cache); + span.record("num_domains_returned", num_domains_returned as u64); + } + + // Fast path: serve from CRC if it tracks domain metadata at this version. + if let Some(crc) = self.crc_at_version() { + match &crc.domain_metadata_state { + DomainMetadataState::Complete(map) => { + let hits: DomainMetadataMap = match domains { + None => map.clone(), + // Look up each filter key. A miss means the domain does not exist + // at this version (Complete is authoritative for misses), so skip + // it and return whatever hits we found. + Some(filter) => filter + .iter() + .filter_map(|&k| map.get(k).map(|v| (k.to_string(), v.clone()))) + .collect(), + }; + record_metric(true, hits.len()); + return Ok(hits); + } + DomainMetadataState::Partial(map) => { + if let Some(filter) = domains { + // Look up each filter key. A miss means we do not know whether the + // domain exists, so abandon the cache and fall through to log + // replay. `collect::>` short-circuits on the first None. + // TODO(#2572): track tombstoned domains in `Partial` so removals + // observed during apply can return authoritative + // `None` instead of falling through. + let hits: Option = filter + .iter() + .map(|&k| map.get(k).map(|v| (k.to_string(), v.clone()))) + .collect(); + if let Some(hits) = hits { + record_metric(true, hits.len()); + return Ok(hits); + } + } + } + } + } + // A stale but authoritative (`Complete`) CRC roots a tail-only scan over the commits after + // it, skipping the checkpoint. A stale `Partial` CRC (one whose file lacked the + // `domainMetadata` field) has no authoritative active-domain set, so it cannot root the + // scan and falls through to the full replay below. + if let Some(base) = self.base_crc() { + if let DomainMetadataState::Complete(base_active) = &base.domain_metadata_state { + let rooted = self.log_segment().scan_domain_metadatas_rooted_in_crc( + base.version, + base_active, + domains, + engine, + )?; + // TODO: report a distinct metric source here. A rooted tail scan is neither a + // cache hit nor a full replay, yet `from_cache = false` buckets it with + // full replay, hiding the checkpoint-skipping win. A 3-variant source enum + // (cache / crc-rooted / full-replay) would let metrics measure it. + record_metric(false, rooted.len()); + return Ok(rooted); + } + } + + // Fallback: scan the log_segment from scratch. + // TODO: a Partial cache already covers the commits read during snapshot load. A + // miss search could skip that range and only scan the older commits, then + // union those results with the Partial cache's entries to produce the final + // answer. + let replayed = self.log_segment().scan_domain_metadatas(domains, engine)?; + record_metric(false, replayed.len()); + Ok(replayed) + } + + /// Fetch both user-controlled and system-controlled domain metadata for a specific domain + /// in this snapshot. + /// + /// Returns the latest configuration for the domain, or `None` if the domain does not exist + /// (or was removed). Unlike [`Snapshot::get_domain_metadata`], this does not reject `delta.*` + /// domains. + #[allow(unused)] + #[internal_api] + pub(crate) fn get_domain_metadata_internal( + &self, + domain: &str, + engine: &dyn Engine, + ) -> DeltaResult> { + let mut map = self.get_domain_metadatas_internal(engine, Some(&HashSet::from([domain])))?; + Ok(map.remove(domain).map(|dm| dm.configuration().to_owned())) + } + + /// Fetch all non-internal domain metadata for this snapshot as a `Vec`. + /// + /// Internal (`delta.*`) domains are filtered out. + #[allow(unused)] + #[internal_api] + pub(crate) fn get_all_domain_metadata( + &self, + engine: &dyn Engine, + ) -> DeltaResult> { + let all_metadata = self.get_domain_metadatas_internal(engine, None)?; + Ok(all_metadata + .into_values() + .filter(|domain| !domain.is_internal()) + .collect()) + } + + /// Returns file-level statistics, or `None` if this snapshot has no CRC at its version (none + /// resolved, or the resolved CRC is stale), or its CRC does not have `Complete` file stats. + /// Performs no I/O. + pub fn get_file_stats_if_present(&self) -> Option { + self.crc_at_version() + .and_then(|crc| crc.file_stats().cloned()) + } + + /// Get the In-Commit Timestamp (ICT) for this snapshot. + /// + /// Returns the `inCommitTimestamp` from the CommitInfo action of the commit that created this + /// snapshot. + /// + /// # Returns + /// - `Ok(Some(timestamp))` - ICT is enabled and available for this version + /// - `Ok(None)` - ICT is not enabled + /// - `Err(...)` - ICT is enabled but cannot be read, or enablement version is invalid + #[instrument(parent = &self.span, name = "snap.get_ict", skip_all, err)] + #[internal_api] + pub(crate) fn get_in_commit_timestamp(&self, engine: &dyn Engine) -> DeltaResult> { + // Get ICT enablement info and check if we should read ICT for this version + let enablement = self + .table_configuration() + .in_commit_timestamp_enablement()?; + + // Return None if ICT is not enabled at all + if matches!(enablement, InCommitTimestampEnablement::NotEnabled) { + return Ok(None); + } + + // If ICT is enabled with an enablement version, verify the enablement version is not in the + // future + if let InCommitTimestampEnablement::Enabled { + enablement: Some((enablement_version, _)), + } = enablement + { + if self.version() < enablement_version { + return Err(Error::generic(format!( + "Invalid state: snapshot at version {} has ICT enablement version {} in the future", + self.version(), + enablement_version + ))); + } + } + + // Fast path: serve ICT from CRC if available at this version. + if let Some(crc) = self.crc_at_version() { + match crc.in_commit_timestamp_opt { + Some(ict) => return Ok(Some(ict)), + None => { + return Err(Error::generic(format!( + "In-Commit Timestamp not found in CRC file at version {}", + self.version() + ))); + } + } + } + + // Fallback: read the ICT from latest_commit_file + match &self.log_segment.listed.latest_commit_file { + Some(commit_file_meta) => { + let ict = commit_file_meta.read_in_commit_timestamp(engine)?; + Ok(Some(ict)) + } + None => Err(Error::generic("Last commit file not found in log segment")), + } + } + + /// Get the timestamp for this snapshot's version, in milliseconds since the Unix epoch. + /// + /// When In-Commit Timestamp (ICT) are enabled, returns the In-Commit Timestamp value. + /// Otherwise, falls back to the filesystem last-modified time of the latest commit file. + /// + /// Returns an error if the commit file is missing, the ICT configuration is invalid, or the + /// ICT value cannot be read. + /// + /// See also [`get_in_commit_timestamp`] for ICT-only semantics. + /// + /// [`get_in_commit_timestamp`]: Self::get_in_commit_timestamp + #[allow(unused)] + #[instrument(parent = &self.span, name = "snap.get_ts", skip_all, err)] + pub fn get_timestamp(&self, engine: &dyn Engine) -> DeltaResult { + match self + .table_configuration() + .in_commit_timestamp_enablement()? + { + InCommitTimestampEnablement::NotEnabled => { + match &self.log_segment.listed.latest_commit_file { + Some(commit_file_meta) => { + let ts = commit_file_meta.location.last_modified; + Ok(ts) + } + None => Err(Error::generic(format!( + "Last commit file not found in log segment for version {} \ + (ICT disabled): cannot read filesystem modification timestamp", + self.version() + ))), + } + } + InCommitTimestampEnablement::Enabled { .. } => self + .get_in_commit_timestamp(engine) + .map_err(|e| { + Error::generic(format!( + "Unable to read in-commit timestamp for version {}: {e}", + self.version() + )) + })? + .ok_or_else(|| { + Error::internal_error(format!( + "Invalid state: version {}, ICT is enabled \ + but get_in_commit_timestamp returned None", + self.version() + )) + }), + } + } + + // ============================================================================ + // Downstream operation builders + // ============================================================================ + + /// Create a [`ScanBuilder`] for an `SnapshotRef`. + pub fn scan_builder(self: Arc) -> ScanBuilder { + ScanBuilder::new(self) + } + + /// Create an [`IncrementalScanBuilder`] for the range `(base_version, self.version()]`. + /// + /// Use this to advance a cached file listing from `base_version` to this snapshot's + /// version without doing a full scan. See [`IncrementalScanBuilder`] for details. + pub fn incremental_scan_builder( + self: Arc, + base_version: Version, + ) -> IncrementalScanBuilder { + IncrementalScanBuilder::new(self, base_version) + } + + /// Create a [`Transaction`] for this `SnapshotRef`. With the specified [`Committer`]. + /// + /// Note: For tables with clustering enabled, this performs log replay to read clustering + /// columns from domain metadata, which may have a performance cost. + pub fn transaction( + self: Arc, + committer: Box, + engine: &dyn Engine, + ) -> DeltaResult { + Transaction::try_new_existing_table(self, committer, engine) + } + + /// Creates a builder for altering this table's metadata. Currently supports schema change + /// operations. + /// + /// The returned builder allows chaining operations before building an + /// [`AlterTableTransaction`] that can be committed. + /// + /// [`AlterTableTransaction`]: crate::transaction::AlterTableTransaction + pub fn alter_table(self: Arc) -> AlterTableTransactionBuilder { + AlterTableTransactionBuilder::new(self) + } + + /// Creates a [`CheckpointWriter`] for generating a checkpoint from this snapshot. + /// + /// See the [`crate::checkpoint`] module documentation for more details on checkpoint types + /// and the overall checkpoint process. + pub fn create_checkpoint_writer( + self: Arc, + engine: &dyn Engine, + ) -> DeltaResult { + CheckpointWriter::try_new(self, engine) + } + + /// Creates a [`LogCompactionWriter`] for generating a log compaction file. + /// + /// Log compaction aggregates commit files in a version range into a single compacted file, + /// improving performance by reducing the number of files to process during log replay. + /// + /// # Parameters + /// - `start_version`: The first version to include in the compaction (inclusive) + /// - `end_version`: The last version to include in the compaction (inclusive) + /// + /// # Returns + /// A [`LogCompactionWriter`] that can be used to generate the compaction file. + /// + /// NOTE: This method is currently a no-op because log compaction is disabled (#2337) + pub fn log_compaction_writer( + self: Arc, + start_version: Version, + end_version: Version, + ) -> DeltaResult { + LogCompactionWriter::try_new(self, start_version, end_version) + } + + // ============================================================================ + // Mutations + // ============================================================================ + + /// Writes a version checksum (CRC) file for this snapshot. Writers should call this after + /// every commit because checksums enable faster snapshot loading and table state validation. + /// + /// The CRC is resolved best-effort from the newest available root: an in-memory CRC at this + /// version, a stale on-disk CRC advanced to this version, a checkpoint advanced to this + /// version, or the full commit history. + /// + /// Returns a tuple of [`ChecksumWriteResult`] and a [`SnapshotRef`]. On + /// [`ChecksumWriteResult::Written`], the returned snapshot has the CRC file recorded in + /// its log segment. On [`ChecksumWriteResult::AlreadyExists`], the original snapshot is + /// returned unchanged. + /// + /// # Errors + /// + /// - [`Error::ChecksumWriteUnsupported`] if no CRC can be resolved for this version, if the + /// resolved CRC's `file_stats_state` is `Indeterminate` (a non-incremental operation like + /// ANALYZE STATS, or a file action with a missing size; recoverable with a full state + /// reconstruction in the future), or if `delta.enableInCommitTimestamps` is `true` but + /// `inCommitTimestampOpt` is absent. + /// - The underlying read error if in-commit timestamps are enabled but the timestamp cannot be + /// read from the commit file. + /// - I/O errors from the engine's storage handler if the write fails. + #[instrument(parent = &self.span, name = "snap.write_checksum", skip_all, err)] + pub fn write_checksum( + self: &SnapshotRef, + engine: &dyn Engine, + ) -> DeltaResult<(ChecksumWriteResult, SnapshotRef)> { + let has_crc_on_disk = self + .log_segment + .listed + .latest_crc_file + .as_ref() + .is_some_and(|f| f.version == self.version()); + + if has_crc_on_disk { + info!( + "CRC file already exists on disk at version {}", + self.version() + ); + return Ok((ChecksumWriteResult::AlreadyExists, Arc::clone(self))); + } + + let crc = self.resolve_crc_for_write(engine)?; + + let crc_path = ParsedLogPath::new_crc(self.table_root(), self.version())?; + + match try_write_crc_file(engine, &crc_path.location, &crc) { + Ok(()) => { + info!("Wrote CRC file at {}", crc_path.location); + let new_log_segment = self.log_segment.try_new_with_crc_file(crc_path)?; + let new_snapshot = Arc::new(Snapshot::new_with_crc( + new_log_segment, + self.table_configuration().clone(), + Some(crc), + self.built_as_latest, + )?); + Ok((ChecksumWriteResult::Written, new_snapshot)) + } + Err(Error::FileAlreadyExists(_)) => { + info!( + "Another writer beat us to writing CRC file at {}", + crc_path.location + ); + Ok((ChecksumWriteResult::AlreadyExists, Arc::clone(self))) + } + Err(e) => Err(e), + } + } + + /// Resolve the CRC to write for this snapshot's version. + /// + /// Tries the newest available root, stopping at the first that applies: + /// 1. The in-memory CRC, used as-is. E.g. a snapshot at N that did incremental CRC replay from + /// a CRC at N-5, or a post-commit snapshot at N+1 built from a CRC at N (on disk or + /// computed). + /// 2. A stale on-disk CRC, advanced over the tail commits via reverse replay. + /// 3. A checkpoint, advanced over the tail commits via reverse replay. + /// 4. A full reverse replay of the commit history. + /// + /// Returns [`Error::ChecksumWriteUnsupported`] when a root is reached but cannot yield a + /// writable CRC (missing protocol or metadata, or a non-incremental tail that dooms file + /// stats). + /// + /// The `root` span field records which root resolved the CRC. + #[instrument(parent = &self.span, name = "snap.resolve_crc_for_write", skip_all, err, fields(root))] + fn resolve_crc_for_write(&self, engine: &dyn Engine) -> DeltaResult> { + let span = tracing::Span::current(); + // Case 1: an in-memory CRC at this version is ready to write as-is. + if let Some(crc) = self.crc_at_version() { + span.record("root", "in_memory"); + return Ok(crc.clone()); + } + + let end = self.version(); + let log_segment = &self.log_segment; + + // Case 2: a stale base CRC, older than this version. Advance the retained base over the + // tail commits (a held base is always at or above the checkpoint, so a tail exists). + if let Some(base) = self.base_crc() { + span.record("root", "stale_crc"); + let crc = log_segment.build_crc_from_base(engine, base)?; + return Ok(Arc::new(crc)); + } + + // Case 3: no CRC, but a checkpoint to root at. + if let Some(checkpoint_version) = log_segment.checkpoint_version { + span.record("root", "checkpoint"); + if checkpoint_version == end { + // A checkpoint carries no ICT (it has no commitInfo). Since there's no tail delta + // to carry it either, we defer to `get_in_commit_timestamp`, which reads the commit + // at the checkpoint version and returns None when ICT is disabled, or errors when + // it is enabled but unreadable. + let mut crc = log_segment + .build_crc_from_checkpoint(engine)? + .ok_or_else(|| unresolved_crc("checkpoint is missing protocol or metadata"))?; + crc.in_commit_timestamp_opt = self.get_in_commit_timestamp(engine)?; + return Ok(Arc::new(crc)); + } + // Replay the tail commits first: a non-incremental tail dooms file stats no matter + // what the checkpoint holds, so skip the larger checkpoint read in that case. + let delta = log_segment.build_crc_delta_from_base( + engine, + checkpoint_version, + Some(FileSizeHistogram::create_default()), + )?; + require!( + delta.is_incremental_safe, + unresolved_crc("commits after the checkpoint are not incremental-safe") + ); + let base = log_segment + .build_crc_from_checkpoint(engine)? + .ok_or_else(|| unresolved_crc("checkpoint is missing protocol or metadata"))?; + // The tail delta carries v_end's ICT, which `apply` sets on the result. + return Ok(Arc::new(base.apply(delta, end))); + } + + // Case 4: neither CRC nor checkpoint, so reverse-replay the full commit history. + span.record("root", "version_zero"); + let crc = log_segment + .build_crc_from_version_zero(engine)? + .ok_or_else(|| unresolved_crc("commit history is missing protocol or metadata"))?; + Ok(Arc::new(crc)) + } + + /// Performs a complete checkpoint of this snapshot using the provided engine. + /// + /// If a checkpoint already exists at this version, returns + /// [`CheckpointWriteResult::AlreadyExists`] with the original snapshot unchanged. + /// Otherwise, writes a checkpoint parquet file and the `_last_checkpoint` file and returns + /// [`CheckpointWriteResult::Written`] with an updated [`SnapshotRef`] whose log segment + /// reflects the new checkpoint. Commits and compaction files subsumed by the checkpoint are + /// dropped from the returned snapshot. + /// + /// # Parameters + /// - `engine`: Engine for data processing and I/O + /// - `spec`: Checkpoint format specification. `None` uses the default checkpoint settings + /// (auto-detecting V1/V2 from table features). For V2 checkpoints, the default is to not + /// write sidecar files. + /// + /// # Errors + /// - If `CheckpointSpec::V2` is used but the table does not support the `v2Checkpoint` feature. + /// - If `CheckpointSpec::V1` is used but the table supports `v2Checkpoint` feature. Note: the + /// Delta protocol permits writing V1 checkpoints to such tables; this is a kernel limitation. + /// - If `file_actions_per_sidecar_hint` is `Some(0)`. + /// - If the checkpoint write fails (e.g. I/O, parquet write). A `FileAlreadyExists` error is + /// not propagated; it returns [`CheckpointWriteResult::AlreadyExists`] instead. Note: this + /// also fires on the (unlikely) case of a sidecar UUID filename collision, where + /// it should ideally surface as an error. Tracked in + /// . + /// + /// Note: + /// - It is still possible that an existing checkpoint gets overwritten if that checkpoint + /// was written by a concurrent writer. + /// - This function uses [`crate::ParquetHandler::write_parquet_file`] and + /// [`crate::StorageHandler::head`], which may not be implemented by all engines. If you + /// are using the default engine, make sure to build it with the multi-threaded executor + /// if you want to use this method. + /// + /// [`CheckpointSpec`]: crate::checkpoint::CheckpointSpec + /// + /// Note: There is currently no public api for callers to determine whether a table supports V2 + /// checkpoints directly. Tracked in . + #[instrument(parent = &self.span, name = "snap.checkpoint", skip_all, err)] + pub fn checkpoint( + self: &SnapshotRef, + engine: &dyn Engine, + spec: Option<&CheckpointSpec>, + ) -> DeltaResult<(CheckpointWriteResult, SnapshotRef)> { + if self.log_segment.checkpoint_version == Some(self.log_segment.end_version) { + info!( + "Checkpoint already exists for snapshot version {}", + self.version() + ); + return Ok((CheckpointWriteResult::AlreadyExists, Arc::clone(self))); + } + info!( + "Writing checkpoint for snapshot version {} with spec {:?}", + self.version(), + spec + ); + + let v2_supported = self + .table_configuration() + .is_feature_supported(&TableFeature::V2Checkpoint); + match spec { + Some(CheckpointSpec::V2(cfg)) => { + if !v2_supported { + return Err(Error::checkpoint_write( + "CheckpointSpec::V2 requires the v2Checkpoint table feature to be supported", + )); + } + if let V2CheckpointConfig::WithSidecar { + file_actions_per_sidecar_hint: Some(0), + } = cfg + { + return Err(Error::checkpoint_write( + "file_actions_per_sidecar_hint must be greater than 0", + )); + } + } + Some(CheckpointSpec::V1) if v2_supported => { + // TODO: remove this once we support writing V1 checkpoints even if table supports + // v2Checkpoint See . + return Err(Error::unsupported( + "Kernel does not support writing V1 checkpoints when the table supports v2Checkpoint", + )); + } + _ => {} + } + + let writer = Arc::clone(self).create_checkpoint_writer(engine)?; + + let write_result = match spec { + Some(CheckpointSpec::V2(V2CheckpointConfig::WithSidecar { + file_actions_per_sidecar_hint, + })) => { + let hint = + file_actions_per_sidecar_hint.unwrap_or(DEFAULT_FILE_ACTIONS_PER_SIDECAR_HINT); + writer.write_v2_checkpoint_with_sidecars(engine, hint) + } + _ => writer.write_checkpoint_without_sidecars(engine), + }; + + let info = match write_result { + Ok(info) => info, + Err(Error::FileAlreadyExists(_)) => { + // NOTE: Per write_parquet_file's documentation, it should silently overwrite + // existing files, so we log a warning but still return the correct result. + warn!( + "ParquetHandler::write_parquet_file unexpectedly failed on \ + FileAlreadyExists for version {}", + self.version() + ); + return Ok((CheckpointWriteResult::AlreadyExists, Arc::clone(self))); + } + Err(e) => return Err(e), + }; + + writer.finalize(engine, &info.last_checkpoint_stats)?; + + let checkpoint_log_path = ParsedLogPath::try_from(info.file_meta)?.ok_or_else(|| { + Error::internal_error("Checkpoint path could not be parsed as a log path") + })?; + let new_log_segment = self + .log_segment + .try_new_with_checkpoint(checkpoint_log_path)?; + // Carry only an at-version CRC forward: the checkpoint drops the commits below it, so a + // stale base could no longer be advanced over them. + Ok(( + CheckpointWriteResult::Written, + Arc::new(Snapshot::new_with_crc( + new_log_segment, + self.table_configuration().clone(), + self.crc_at_version().cloned(), + self.built_as_latest, + )?), + )) + } + + /// Publishes all catalog commits at this table version. Applicable only to catalog-managed + /// tables. This method is a no-op for filesystem-managed tables or if there are no catalog + /// commits to publish. + /// + /// Publishing copies ratified catalog commits to the Delta log as published Delta files, + /// reducing catalog storage requirements and enabling some table maintenance operations, + /// like checkpointing. + /// + /// # Parameters + /// + /// - `engine`: The engine to use for publishing commits + /// + /// # Errors + /// + /// Returns an error if the publish operation fails, or if there are catalog commits that need + /// publishing but the table or committer don't support publishing. + /// + /// # See Also + /// + /// - [`Committer::publish`] + #[instrument(parent = &self.span, name = "snap.publish", skip_all, err)] + pub fn publish( + self: &SnapshotRef, + engine: &dyn Engine, + committer: &dyn Committer, + ) -> DeltaResult { + let unpublished_catalog_commits = self.log_segment().get_unpublished_catalog_commits()?; + + if unpublished_catalog_commits.is_empty() { + return Ok(Arc::clone(self)); + } + + require!( + unpublished_catalog_commits + .windows(2) + .all(|commits| commits[0].version() + 1 == commits[1].version()), + Error::generic(format!( + "Expected ordered and contiguous unpublished catalog commits. \ + Got: {unpublished_catalog_commits:?}" + )) + ); + + require!( + self.table_configuration().is_catalog_managed(), + Error::generic( + "There are catalog commits that need publishing, but the table is not catalog-managed.", + ) + ); + + require!( + committer.is_catalog_committer(), + Error::generic( + "There are catalog commits that need publishing, but the committer is not a catalog committer.", + ) + ); + + let publish_metadata = + PublishMetadata::try_new(self.version(), unpublished_catalog_commits)?; + + committer.publish(engine, publish_metadata)?; + + Ok(Arc::new(Snapshot::new_with_crc( + self.log_segment().new_as_published()?, + self.table_configuration().clone(), + self.base_crc().cloned(), + self.built_as_latest, + )?)) + } +} + +// TODO: unify this and lots of stuff in LogSegment tests and test_utils. +#[cfg(test)] +async fn commit( + table_root: impl AsRef, + store: &crate::object_store::memory::InMemory, + version: Version, + commit: Vec, +) { + let commit_data = commit + .iter() + .map(ToString::to_string) + .collect::>() + .join("\n"); + test_utils::add_commit(table_root, store, version, commit_data) + .await + .unwrap(); +} + +#[cfg(test)] +mod tests { + use std::path::PathBuf; + use std::sync::Arc; + + use rstest::rstest; + use serde_json::json; + use test_utils::table_builder::{ + checkpoint_json_stats, unpartitioned, FeatureSet, LogState, TestTableBuilder, VersionTarget, + }; + use test_utils::{add_commit, delta_path_for_version}; + + use super::{commit, *}; + use crate::actions::{DomainMetadata, Protocol}; + use crate::arrow::array::StringArray; + use crate::arrow::record_batch::RecordBatch; + use crate::committer::FileSystemCommitter; + use crate::engine::arrow_data::ArrowEngineData; + use crate::engine::sync::SyncEngine; + use crate::last_checkpoint_hint::LastCheckpointHint; + use crate::log_segment::LogSegment; + use crate::log_segment_files::LogSegmentFiles; + use crate::object_store::memory::InMemory; + use crate::object_store::path::Path; + use crate::object_store::ObjectStoreExt as _; + use crate::parquet::arrow::ArrowWriter; + use crate::path::ParsedLogPath; + use crate::schema::{schema_ref, DataType, StructField, StructType}; + use crate::table_features::{ + TABLE_FEATURES_MIN_READER_VERSION, TABLE_FEATURES_MIN_WRITER_VERSION, + }; + use crate::table_properties::ENABLE_IN_COMMIT_TIMESTAMPS; + use crate::transaction::create_table::create_table; + use crate::unit_test_utils::{assert_result_error_with_message, string_array_to_engine_data}; + use crate::utils::FoldWithOption as _; + + /// Helper function to create a commitInfo action with optional ICT + fn create_commit_info(timestamp: i64, ict: Option) -> serde_json::Value { + let mut commit_info = json!({ + "timestamp": timestamp, + "operation": "WRITE", + }); + + if let Some(ict_value) = ict { + commit_info["inCommitTimestamp"] = json!(ict_value); + } + + json!({ + "commitInfo": commit_info + }) + } + + fn create_protocol(ict_enabled: bool, min_reader_version: Option) -> serde_json::Value { + let reader_version = min_reader_version.unwrap_or(1); + + if ict_enabled { + let mut protocol = json!({ + "protocol": { + "minReaderVersion": reader_version, + "minWriterVersion": TABLE_FEATURES_MIN_WRITER_VERSION, + "writerFeatures": ["inCommitTimestamp"] + } + }); + + // Only include readerFeatures if minReaderVersion >= table-features minimum. + if reader_version >= TABLE_FEATURES_MIN_READER_VERSION as u32 { + protocol["protocol"]["readerFeatures"] = json!([]); + } + + protocol + } else { + json!({ + "protocol": { + "minReaderVersion": reader_version, + "minWriterVersion": 2 + } + }) + } + } + + fn create_metadata( + id: Option<&str>, + schema_string: Option<&str>, + created_time: Option, + ict_config: Option<(String, String)>, + ict_enabled_but_missing_version: bool, + ) -> serde_json::Value { + let config = if ict_enabled_but_missing_version { + // Special case for testing ICT enabled but missing enablement info + json!({ + "delta.enableInCommitTimestamps": "true" + }) + } else if let Some((enablement_version, enablement_timestamp)) = ict_config { + json!({ + "delta.enableInCommitTimestamps": "true", + "delta.inCommitTimestampEnablementVersion": enablement_version, + "delta.inCommitTimestampEnablementTimestamp": enablement_timestamp + }) + } else { + json!({}) + }; + + json!({ + "metaData": { + "id": id.unwrap_or("testId"), + "format": {"provider": "parquet", "options": {}}, + "schemaString": schema_string.unwrap_or("{\"type\":\"struct\",\"fields\":[{\"name\":\"value\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}"), + "partitionColumns": [], + "configuration": config, + "createdTime": created_time.unwrap_or(1587968586154u64) + } + }) + } + + fn create_basic_commit(ict_enabled: bool, ict_config: Option<(String, String)>) -> String { + let protocol = create_protocol(ict_enabled, None); + let metadata = create_metadata(None, None, None, ict_config, false); + format!("{protocol}\n{metadata}") + } + + fn create_snapshot_with_commit_file_absent_from_log_segment( + url: &Url, + table_cfg: TableConfiguration, + ) -> DeltaResult { + // Create a log segment with only checkpoint and no commit file (simulating scenario + // where a checkpoint exists but the commit file has been cleaned up) + let checkpoint_parts = vec![ParsedLogPath::try_from(crate::FileMeta { + location: url.join("_delta_log/00000000000000000000.checkpoint.parquet")?, + last_modified: 0, + size: 100, + })? + .unwrap()]; + + let listed_files = LogSegmentFiles { + checkpoint_parts, + ..Default::default() + }; + + let log_segment = + LogSegment::try_new(listed_files, url.join("_delta_log/")?, Some(0), None)?; + + Snapshot::new_with_crc( + log_segment, + table_cfg, + None, /* crc */ + false, /* built_as_latest */ + ) + } + + #[test] + fn test_snapshot_read_metadata() { + let path = + std::fs::canonicalize(PathBuf::from("./tests/data/table-with-dv-small/")).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + + let engine = SyncEngine::new(); + let snapshot = Snapshot::builder_for(url) + .at_version(1) + .build(&engine) + .unwrap(); + + let expected = Protocol::try_new_modern(["deletionVectors"], ["deletionVectors"]).unwrap(); + assert_eq!(snapshot.table_configuration().protocol(), &expected); + + let schema_string = r#"{"type":"struct","fields":[{"name":"value","type":"integer","nullable":true,"metadata":{}}]}"#; + let expected: SchemaRef = serde_json::from_str(schema_string).unwrap(); + assert_eq!(snapshot.schema(), expected); + } + + #[test] + fn test_new_snapshot() { + let path = + std::fs::canonicalize(PathBuf::from("./tests/data/table-with-dv-small/")).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + + let engine = SyncEngine::new(); + let snapshot = Snapshot::builder_for(url).build(&engine).unwrap(); + + let expected = Protocol::try_new_modern(["deletionVectors"], ["deletionVectors"]).unwrap(); + assert_eq!(snapshot.table_configuration().protocol(), &expected); + + let schema_string = r#"{"type":"struct","fields":[{"name":"value","type":"integer","nullable":true,"metadata":{}}]}"#; + let expected: SchemaRef = serde_json::from_str(schema_string).unwrap(); + assert_eq!(snapshot.schema(), expected); + } + + #[test] + fn test_read_table_with_missing_last_checkpoint() { + // this table doesn't have a _last_checkpoint file + let path = std::fs::canonicalize(PathBuf::from( + "./tests/data/table-with-dv-small/_delta_log/", + )) + .unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + + let engine = SyncEngine::new(); + let storage = engine.storage_handler(); + let cp = LastCheckpointHint::try_read(storage.as_ref(), &url).unwrap(); + assert!(cp.is_none()); + } + + fn valid_last_checkpoint() -> (Vec, LastCheckpointHint) { + let checkpoint = LastCheckpointHint { + v2_checkpoint: None, + version: 1, + size: 8, + parts: None, + size_in_bytes: Some(21857), + num_of_add_files: None, + checkpoint_schema: None, + checksum: None, + tags: None, + }; + let data = checkpoint.to_json_bytes(); + (data, checkpoint) + } + + fn valid_last_checkpoint_with_tags() -> (Vec, LastCheckpointHint) { + use std::collections::HashMap; + + let (_, base_checkpoint) = valid_last_checkpoint(); + + let mut tags = HashMap::new(); + tags.insert( + "author".to_string(), + "test_read_table_with_last_checkpoint".to_string(), + ); + tags.insert("environment".to_string(), "snapshot_tests".to_string()); + tags.insert("created_by".to_string(), "delta-kernel-rs".to_string()); + + let checkpoint = LastCheckpointHint { + tags: Some(tags), + ..base_checkpoint + }; + + let data = checkpoint.to_json_bytes(); + (data, checkpoint) + } + + #[tokio::test] + async fn test_read_table_with_empty_last_checkpoint() { + // in memory file system + let store = Arc::new(InMemory::new()); + + // do a _last_checkpoint file with "{}" as content + let empty = "{}".as_bytes().to_vec(); + let invalid_path = Path::from("invalid/_last_checkpoint"); + + store + .put(&invalid_path, empty.into()) + .await + .expect("put _last_checkpoint"); + + let engine = SyncEngine::new_with_store(store); + let storage = engine.storage_handler(); + let url = Url::parse("memory:///invalid/").expect("valid url"); + let invalid = + LastCheckpointHint::try_read(storage.as_ref(), &url).expect("read last checkpoint"); + assert!(invalid.is_none()) + } + + #[tokio::test] + async fn test_read_table_with_last_checkpoint() { + // in memory file system + let store = Arc::new(InMemory::new()); + + // Define test cases: (path, data, expected_result) + let (data, expected) = valid_last_checkpoint(); + let (data_with_tags, expected_with_tags) = valid_last_checkpoint_with_tags(); + let test_cases = vec![ + ("valid", data, Some(expected)), + ("invalid", "invalid".as_bytes().to_vec(), None), + ("valid_with_tags", data_with_tags, Some(expected_with_tags)), + ]; + + // Write all test files to the in memory file system + for (path_prefix, data, _) in &test_cases { + let path = Path::from(format!("{path_prefix}/_last_checkpoint")); + store + .put(&path, data.clone().into()) + .await + .expect("put _last_checkpoint"); + } + + let engine = SyncEngine::new_with_store(store); + let storage = engine.storage_handler(); + + // Test reading all checkpoints from the in memory file system for cases where the data is + // valid, invalid and valid with tags. + for (path_prefix, _, expected_result) in test_cases { + let url = Url::parse(&format!("memory:///{path_prefix}/")).expect("valid url"); + let result = + LastCheckpointHint::try_read(storage.as_ref(), &url).expect("read last checkpoint"); + assert_eq!(result, expected_result); + } + } + + #[test_log::test] + fn test_read_table_with_checkpoint() { + let path = std::fs::canonicalize(PathBuf::from( + "./tests/data/with_checkpoint_no_last_checkpoint/", + )) + .unwrap(); + let location = url::Url::from_directory_path(path).unwrap(); + let engine = SyncEngine::new(); + let snapshot = Snapshot::builder_for(location).build(&engine).unwrap(); + + assert_eq!(snapshot.log_segment.listed.checkpoint_parts.len(), 1); + assert_eq!( + ParsedLogPath::try_from( + snapshot.log_segment.listed.checkpoint_parts[0] + .location + .clone() + ) + .unwrap() + .unwrap() + .version, + 2, + ); + assert_eq!(snapshot.log_segment.listed.ascending_commit_files.len(), 1); + assert_eq!( + ParsedLogPath::try_from( + snapshot.log_segment.listed.ascending_commit_files[0] + .location + .clone() + ) + .unwrap() + .unwrap() + .version, + 3, + ); + } + + #[tokio::test] + async fn test_domain_metadata() -> DeltaResult<()> { + let table_root = "memory:///test_table/"; + let store = Arc::new(InMemory::new()); + let engine = SyncEngine::new_with_store(store.clone()); + + // commit0 + // - domain1: not removed + // - domain2: not removed + let commit = [ + json!({ + "protocol": { + "minReaderVersion": 1, + "minWriterVersion": 1 + } + }), + json!({ + "metaData": { + "id":"5fba94ed-9794-4965-ba6e-6ee3c0d22af9", + "format": { "provider": "parquet", "options": {} }, + "schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"val\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}", + "partitionColumns": [], + "configuration": {}, + "createdTime": 1587968585495i64 + } + }), + json!({ + "domainMetadata": { + "domain": "domain1", + "configuration": "domain1_commit0", + "removed": false + } + }), + json!({ + "domainMetadata": { + "domain": "domain2", + "configuration": "domain2_commit0", + "removed": false + } + }), + json!({ + "domainMetadata": { + "domain": "domain3", + "configuration": "domain3_commit0", + "removed": false + } + }), + ] + .map(|json| json.to_string()) + .join("\n"); + add_commit(table_root, store.clone().as_ref(), 0, commit) + .await + .unwrap(); + + // commit1 + // - domain1: removed + // - domain2: not-removed + // - internal domain + let commit = [ + json!({ + "domainMetadata": { + "domain": "domain1", + "configuration": "domain1_commit1", + "removed": true + } + }), + json!({ + "domainMetadata": { + "domain": "domain2", + "configuration": "domain2_commit1", + "removed": false + } + }), + json!({ + "domainMetadata": { + "domain": "delta.domain3", + "configuration": "domain3_commit1", + "removed": false + } + }), + ] + .map(|json| json.to_string()) + .join("\n"); + add_commit(table_root, store.as_ref(), 1, commit) + .await + .unwrap(); + + let snapshot = Snapshot::builder_for(table_root).build(&engine)?; + + // Test get_domain_metadata + + assert_eq!(snapshot.get_domain_metadata("domain1", &engine)?, None); + assert_eq!( + snapshot.get_domain_metadata("domain2", &engine)?, + Some("domain2_commit1".to_string()) + ); + assert_eq!( + snapshot.get_domain_metadata("domain3", &engine)?, + Some("domain3_commit0".to_string()) + ); + let err = snapshot + .get_domain_metadata("delta.domain3", &engine) + .unwrap_err(); + assert!(matches!(err, Error::Generic(msg) if + msg == "User DomainMetadata are not allowed to use system-controlled 'delta.*' domain")); + + // Test get_domain_metadata_internal + assert_eq!( + snapshot.get_domain_metadata_internal("delta.domain3", &engine)?, + Some("domain3_commit1".to_string()) + ); + + // Test get_all_domain_metadata + let mut metadata = snapshot.get_all_domain_metadata(&engine)?; + metadata.sort_by(|a, b| a.domain().cmp(b.domain())); + + let mut expected = vec![ + DomainMetadata::new("domain2".to_string(), "domain2_commit1".to_string()), + DomainMetadata::new("domain3".to_string(), "domain3_commit0".to_string()), + ]; + expected.sort_by(|a, b| a.domain().cmp(b.domain())); + + assert_eq!(metadata, expected); + + Ok(()) + } + + #[test] + #[ignore = "log compaction disabled (#2337)"] + fn test_log_compaction_writer() { + let path = + std::fs::canonicalize(PathBuf::from("./tests/data/table-with-dv-small/")).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + + let engine = SyncEngine::new(); + let snapshot = Snapshot::builder_for(url).build(&engine).unwrap(); + + // Test creating a log compaction writer + let writer = snapshot.clone().log_compaction_writer(0, 1).unwrap(); + let path = writer.compaction_path(); + + // Verify the path format is correct + let expected_filename = "00000000000000000000.00000000000000000001.compacted.json"; + assert!(path.to_string().ends_with(expected_filename)); + + // Test invalid version range (start >= end) + let result = snapshot.clone().log_compaction_writer(2, 1); + assert_result_error_with_message(result, "Invalid version range"); + + // Test equal version range (also invalid) + let result = snapshot.log_compaction_writer(1, 1); + assert_result_error_with_message(result, "Invalid version range"); + } + + // TODO(#2337): remove this test when log compaction is re-enabled. + #[test] + fn test_log_compaction_writer_unsupported() { + let path = + std::fs::canonicalize(PathBuf::from("./tests/data/table-with-dv-small/")).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + + let engine = SyncEngine::new(); + let snapshot = Snapshot::builder_for(url).build(&engine).unwrap(); + + let result = snapshot.log_compaction_writer(0, 1); + assert_result_error_with_message(result, "not currently supported"); + } + + #[tokio::test] + async fn test_timestamp_with_ict_disabled() -> Result<(), Box> { + let store = Arc::new(InMemory::new()); + let table_root = "memory://test/"; + let engine = SyncEngine::new_with_store(store.clone()); + + // Create a basic commit without ICT enabled + let commit0 = create_basic_commit(false, None); + add_commit(table_root, store.as_ref(), 0, commit0).await?; + + let snapshot = Snapshot::builder_for(table_root).build(&engine)?; + + // When ICT is disabled, get_timestamp should return None + let result = snapshot.get_in_commit_timestamp(&engine)?; + assert_eq!(result, None); + + Ok(()) + } + + #[tokio::test] + async fn test_timestamp_with_ict_enablement_timeline() -> Result<(), Box> + { + let store = Arc::new(InMemory::new()); + let table_root = "memory://test/"; + let engine = SyncEngine::new_with_store(store.clone()); + + // Create initial commit without ICT + let commit0 = create_basic_commit(false, None); + add_commit(table_root, store.as_ref(), 0, commit0).await?; + + // Create commit that enables ICT (version 1 = enablement version) + let commit1 = + create_basic_commit(true, Some(("1".to_string(), "1587968586154".to_string()))); + add_commit(table_root, store.as_ref(), 1, commit1).await?; + + // Create commit with ICT enabled + let expected_timestamp = 1587968586200i64; + let commit2 = format!( + r#"{{"commitInfo":{{"timestamp":1587968586154,"inCommitTimestamp":{expected_timestamp},"operation":"WRITE"}}}}"#, + ); + add_commit(table_root, store.as_ref(), 2, commit2.to_string()).await?; + + // Read snapshot at version 0 (before ICT enablement) + let snapshot_v0 = Snapshot::builder_for(table_root) + .at_version(0) + .build(&engine)?; + // This snapshot version predates ICT enablement, so ICT is not available + let result_v0 = snapshot_v0.get_in_commit_timestamp(&engine)?; + assert_eq!(result_v0, None); + + // Read snapshot at version 2 (after ICT enabled) + let snapshot_v2 = Snapshot::builder_for(table_root) + .at_version(2) + .build(&engine)?; + // When ICT is enabled and available, timestamp() should return inCommitTimestamp + let result_v2 = snapshot_v2.get_in_commit_timestamp(&engine)?; + assert_eq!(result_v2, Some(expected_timestamp)); + + Ok(()) + } + + #[tokio::test] + async fn test_get_timestamp_enablement_version_in_future() -> DeltaResult<()> { + // Test invalid state where snapshot has enablement version in the future - should error + let table_root = "memory:///test_table/"; + let store = Arc::new(InMemory::new()); + let engine = SyncEngine::new_with_store(store.clone()); + + let commit_data = [ + json!({ + "protocol": { + "minReaderVersion": TABLE_FEATURES_MIN_READER_VERSION, + "minWriterVersion": TABLE_FEATURES_MIN_WRITER_VERSION, + "readerFeatures": [], + "writerFeatures": ["inCommitTimestamp"] + } + }), + json!({ + "metaData": { + "id": "test_id2", + "format": {"provider": "parquet", "options": {}}, + "schemaString": "{\"type\":\"struct\",\"fields\":[]}", + "partitionColumns": [], + "configuration": { + "delta.enableInCommitTimestamps": "true", + "delta.inCommitTimestampEnablementVersion": "5", // Enablement after version 1 + "delta.inCommitTimestampEnablementTimestamp": "1612345678" + }, + "createdTime": 1677811175819u64 + } + }), + ]; + commit(table_root, store.as_ref(), 0, commit_data.to_vec()).await; + + // Create commit that predates ICT enablement (no inCommitTimestamp) + let commit_predates = [create_commit_info(1234567890, None)]; + commit(table_root, store.as_ref(), 1, commit_predates.to_vec()).await; + + let snapshot_predates = Snapshot::builder_for(table_root) + .at_version(1) + .build(&engine)?; + let result_predates = snapshot_predates.get_in_commit_timestamp(&engine); + + // Version 1 with enablement at version 5 is invalid - should error + assert_result_error_with_message( + result_predates, + "Invalid state: snapshot at version 1 has ICT enablement version 5 in the future", + ); + + Ok(()) + } + + #[tokio::test] + async fn test_get_timestamp_missing_ict_when_enabled() -> DeltaResult<()> { + // Test missing ICT when it should be present - should error + let table_root = "memory:///test_table/"; + let store = Arc::new(InMemory::new()); + let engine = SyncEngine::new_with_store(store.clone()); + + let commit_data = [ + create_protocol(true, Some(TABLE_FEATURES_MIN_READER_VERSION as u32)), + create_metadata( + Some("test_id"), + Some("{\"type\":\"struct\",\"fields\":[]}"), + Some(1677811175819), + Some(("0".to_string(), "1612345678".to_string())), + false, + ), + ]; + commit(table_root, store.as_ref(), 0, commit_data.to_vec()).await; // ICT enabled from version 0 + + // Create commit without ICT despite being enabled (corrupt case) + let commit_missing_ict = [create_commit_info(1234567890, None)]; + commit(table_root, store.as_ref(), 1, commit_missing_ict.to_vec()).await; + + let snapshot_missing = Snapshot::builder_for(table_root) + .at_version(1) + .build(&engine)?; + let result = snapshot_missing.get_in_commit_timestamp(&engine); + assert_result_error_with_message(result, "In-Commit Timestamp not found"); + + Ok(()) + } + + #[tokio::test] + async fn test_get_timestamp_fails_when_commit_missing() -> DeltaResult<()> { + // When ICT is enabled but commit file is not found in log segment, + // get_in_commit_timestamp should return an error + + let url = Url::parse("memory:///")?; + let store = Arc::new(InMemory::new()); + let engine = SyncEngine::new_with_store(store.clone()); + + // Create initial commit with ICT enabled + let commit_data = [ + create_protocol(true, Some(TABLE_FEATURES_MIN_READER_VERSION as u32)), + create_metadata( + Some("test_id"), + Some("{\"type\":\"struct\",\"fields\":[]}"), + Some(1677811175819), + Some(("0".to_string(), "1612345678".to_string())), // ICT enabled from version 0 + false, + ), + ]; + commit(url.as_str(), store.as_ref(), 0, commit_data.to_vec()).await; + + // Build snapshot to get table configuration + let snapshot = Snapshot::builder_for(url.as_str()) + .at_version(0) + .build(&engine)?; + + let snapshot_no_commit = create_snapshot_with_commit_file_absent_from_log_segment( + &url, + snapshot.table_configuration().clone(), + )?; + + // Should return an error when commit file is missing + let result = snapshot_no_commit.get_in_commit_timestamp(&engine); + assert_result_error_with_message(result, "Last commit file not found in log segment"); + + Ok(()) + } + + #[tokio::test] + async fn test_get_timestamp_with_checkpoint_and_commit_same_version() -> DeltaResult<()> { + // Test the scenario where both checkpoint and commit exist at the same version with ICT + // enabled. + let table_root = "memory:///test_table/"; + let store = Arc::new(InMemory::new()); + let engine = SyncEngine::new_with_store(store.clone()); + + // Create 00000000000000000000.json with ICT enabled + let commit0_data = [ + create_commit_info(1587968586154, None), + create_protocol(true, Some(TABLE_FEATURES_MIN_READER_VERSION as u32)), + create_metadata( + Some("5fba94ed-9794-4965-ba6e-6ee3c0d22af9"), + Some("{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"val\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}"), + Some(1587968585495), + Some(("0".to_string(), "1587968586154".to_string())), + false, + ), + ]; + commit(table_root, store.as_ref(), 0, commit0_data.to_vec()).await; + + // Create 00000000000000000001.checkpoint.parquet + let checkpoint_data = [ + create_commit_info(1587968586154, None), + create_protocol(true, Some(TABLE_FEATURES_MIN_READER_VERSION as u32)), + create_metadata( + Some("5fba94ed-9794-4965-ba6e-6ee3c0d22af9"), + Some("{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"val\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}"), + Some(1587968585495), + Some(("0".to_string(), "1587968586154".to_string())), + false, + ), + ]; + + let handler = engine.json_handler(); + let json_strings: StringArray = checkpoint_data + .into_iter() + .map(|json| json.to_string()) + .collect::>() + .into(); + let parsed = handler.parse_json( + string_array_to_engine_data(json_strings), + crate::actions::get_commit_schema().clone(), + )?; + let checkpoint = ArrowEngineData::try_from_engine_data(parsed)?; + let checkpoint: RecordBatch = checkpoint.into(); + + let mut buffer = vec![]; + let mut writer = ArrowWriter::try_new(&mut buffer, checkpoint.schema(), None)?; + writer.write(&checkpoint)?; + writer.close()?; + + let checkpoint_path = delta_path_for_version(1, "checkpoint.parquet"); + store.put(&checkpoint_path, buffer.into()).await?; + + // Create 00000000000000000001.json with ICT + let expected_ict = 1587968586200i64; + let commit1_data = [create_commit_info(1587968586200, Some(expected_ict))]; + commit(table_root, store.as_ref(), 1, commit1_data.to_vec()).await; + + // Build snapshot - LogSegment will filter out the commit file because checkpoint exists at + // same version + let snapshot = Snapshot::builder_for(table_root) + .at_version(1) + .build(&engine)?; + + // We should successfully read ICT by falling back to storage + let timestamp = snapshot.get_in_commit_timestamp(&engine)?; + assert_eq!(timestamp, Some(expected_ict)); + + Ok(()) + } + + #[rstest] + #[case::ict_disabled(false)] + #[case::ict_enabled(true)] + fn test_get_timestamp_returns_valid_timestamp(#[case] ict_enabled: bool) -> DeltaResult<()> { + let temp_dir = tempfile::tempdir().unwrap(); + let table_path = Url::from_directory_path(temp_dir.path()) + .unwrap() + .to_string(); + let engine = SyncEngine::new(); + + let schema = schema_ref! { nullable "id": INTEGER }; + + let mut create_table_builder = create_table(&table_path, schema, "Test/1.0"); + if ict_enabled { + create_table_builder = create_table_builder + .with_table_properties(vec![(ENABLE_IN_COMMIT_TIMESTAMPS, "true")]); + } + + let _ = create_table_builder + .build(&engine, Box::new(FileSystemCommitter::new()))? + .commit(&engine)?; + + let snapshot = Snapshot::builder_for(&table_path).build(&engine)?; + let ts = snapshot.get_timestamp(&engine)?; + let now_ms = chrono::Utc::now().timestamp_millis(); + let two_days_ms = 2 * 24 * 60 * 60 * 1000_i64; + assert!( + (now_ms - two_days_ms..=now_ms).contains(&ts), + "timestamp {ts} not within 2 days of now ({now_ms})" + ); + + if ict_enabled { + let ict_ts = snapshot.get_in_commit_timestamp(&engine)?.unwrap(); + assert_eq!(ts, ict_ts); + } + Ok(()) + } + + #[rstest] + #[case::ict_enabled(true)] + #[case::ict_disabled(false)] + #[tokio::test] + async fn test_get_timestamp_errors_when_commit_file_missing( + #[case] ict_enabled: bool, + ) -> DeltaResult<()> { + let url = Url::parse("memory:///")?; + let store = Arc::new(InMemory::new()); + let engine = SyncEngine::new_with_store(store.clone()); + + // TODO: refactor `ict_config` from a raw tuple to a dedicated ICTConfig struct so the + // enablement version and enablement timestamp fields are named and self-documenting. + // The ict_config tuple is (inCommitTimestampEnablementVersion, + // inCommitTimestampEnablementTimestamp): if ICT is enabled, the enablement version + // is 0 with an arbitrary enablement timestamp. + let ict_config = ict_enabled.then(|| ("0".to_string(), "1612345678".to_string())); + let reader_version = ict_enabled.then_some(TABLE_FEATURES_MIN_READER_VERSION as u32); + + let mut commit_data = vec![]; + // When ICT is enabled, commitInfo must be the first action (protocol requirement) + if ict_enabled { + commit_data.push(create_commit_info(1677811175819, Some(1677811175999))); + } + commit_data.extend([ + create_protocol(ict_enabled, reader_version), + create_metadata( + Some("test_id"), + Some("{\"type\":\"struct\",\"fields\":[]}"), + Some(1677811175819), + ict_config, + false, + ), + ]); + commit(url.as_str(), store.as_ref(), 0, commit_data).await; + + let snapshot = Snapshot::builder_for(url.as_str()) + .at_version(0) + .build(&engine)?; + + let snapshot_no_commit = create_snapshot_with_commit_file_absent_from_log_segment( + &url, + snapshot.table_configuration().clone(), + )?; + + let result = snapshot_no_commit.get_timestamp(&engine); + assert_result_error_with_message(result, "Last commit file not found in log segment"); + + Ok(()) + } + + #[tokio::test] + async fn test_get_timestamp_errors_when_ict_missing_from_commit_info() -> DeltaResult<()> { + // ICT is enabled and commit file IS present in the log segment, but the commitInfo + // action does not carry an inCommitTimestamp value (corrupt/incomplete commit). + let store = Arc::new(InMemory::new()); + let table_root = "memory:///test_table/"; + let engine = SyncEngine::new_with_store(store.clone()); + + let commit0_data = vec![ + create_commit_info(1677811175819, None), // commitInfo without inCommitTimestamp + create_protocol(true, Some(TABLE_FEATURES_MIN_READER_VERSION as u32)), + create_metadata( + Some("test_id"), + Some("{\"type\":\"struct\",\"fields\":[]}"), + Some(1677811175819), + Some(("0".to_string(), "1612345678".to_string())), /* ict enabled at version 0, and an arbitrary timestamp */ + false, + ), + ]; + commit(table_root, store.as_ref(), 0, commit0_data).await; + + let snapshot = Snapshot::builder_for(table_root).build(&engine)?; + let result = snapshot.get_timestamp(&engine); + assert_result_error_with_message(result, "In-Commit Timestamp not found in commit file"); + + Ok(()) + } + + // Verifies the test_context! macro works from kernel/src/ unit tests + // (crosses the crate type boundary via macro expansion). The kernel can't construct a + // DefaultEngine here, so we pass a SyncEngine factory. + #[test] + fn test_context_macro_works_in_unit_test() { + let (_engine, snap, _table) = test_utils::test_context!( + LogState::with_latest_version(2), + FeatureSet::empty(), + unpartitioned(), + checkpoint_json_stats(), + VersionTarget::Latest, + SyncEngine::new_with_store + ); + assert_eq!(snap.version(), 2); + } + + #[test] + fn test_new_post_commit_simple() { + // GIVEN + let path = std::fs::canonicalize(PathBuf::from("./tests/data/basic_partitioned/")).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + let engine = SyncEngine::new(); + let base_snapshot = Snapshot::builder_for(url.clone()).build(&engine).unwrap(); + let next_version = base_snapshot.version() + 1; + + // WHEN + let fake_new_commit = ParsedLogPath::create_parsed_published_commit(&url, next_version); + let post_commit_snapshot = base_snapshot + .new_post_commit(fake_new_commit, CrcDelta::default()) + .unwrap(); + + // THEN + assert_eq!(post_commit_snapshot.version(), next_version); + assert_eq!(post_commit_snapshot.log_segment().end_version, next_version); + } + + // A post-commit snapshot built from a parent holding a stale base (not an at-version CRC) + // keeps that base rather than dropping it: the single-commit delta can't advance it, but + // retaining it lets a later write reuse the parsed CRC. + #[test] + fn test_new_post_commit_from_stale_base_keeps_base() { + let path = + std::fs::canonicalize(PathBuf::from("./tests/data/table-with-dv-small/")).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + let engine = SyncEngine::new(); + + // Rebuild the v1 snapshot so it holds a stale CRC@0 (base only, not at-version). + let built = Snapshot::builder_for(url.clone()) + .at_version(1) + .build(&engine) + .unwrap(); + let stale_crc = Arc::new(Crc { + version: 0, + ..Default::default() + }); + let parent = Snapshot::new_with_crc( + built.log_segment().clone(), + built.table_configuration().clone(), + Some(stale_crc), + false, /* built_as_latest */ + ) + .unwrap(); + assert!(parent.crc_at_version().is_none()); + assert_eq!(parent.base_crc().map(|c| c.version), Some(0)); + + let commit = ParsedLogPath::create_parsed_published_commit(&url, 2); + let post = parent.new_post_commit(commit, CrcDelta::default()).unwrap(); + + assert_eq!(post.version(), 2); + assert!(post.crc_at_version().is_none()); + assert_eq!( + post.base_crc().map(|c| c.version), + Some(0), + "the stale base must be carried forward, not dropped" + ); + } + + #[test] + fn test_get_protocol_derived_properties() { + let path = + std::fs::canonicalize(PathBuf::from("./tests/data/table-with-dv-small/")).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + + let engine = SyncEngine::new(); + let snapshot = Snapshot::builder_for(url).build(&engine).unwrap(); + + let props = snapshot.get_protocol_derived_properties(); + assert_eq!( + props.get("delta.minReaderVersion").unwrap(), + &TABLE_FEATURES_MIN_READER_VERSION.to_string() + ); + assert_eq!( + props.get("delta.minWriterVersion").unwrap(), + &TABLE_FEATURES_MIN_WRITER_VERSION.to_string() + ); + assert_eq!( + props.get("delta.feature.deletionVectors").unwrap(), + "supported" + ); + } + + #[tokio::test] + async fn test_metadata_configuration() { + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///"; + let engine = SyncEngine::new_with_store(storage.clone()); + + // Create a commit with custom configuration + let actions = vec![ + json!({"commitInfo": {"timestamp": 123, "operation": "CREATE TABLE"}}), + json!({"protocol": { + "minReaderVersion": 3, + "minWriterVersion": 7, + "readerFeatures": [], + "writerFeatures": [] + }}), + json!({"metaData": { + "id": "test-id", + "format": {"provider": "parquet", "options": {}}, + "schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":false,\"metadata\":{}}]}", + "partitionColumns": [], + "configuration": { + "io.unitycatalog.tableId": "abc-123", + "myapp.setting": "value" + }, + "createdTime": 1234567890 + }}), + ]; + commit(table_root, &storage, 0, actions).await; + + let snapshot = Snapshot::builder_for(table_root).build(&engine).unwrap(); + let config = snapshot.metadata_configuration(); + assert_eq!( + config.get("io.unitycatalog.tableId"), + Some(&"abc-123".to_string()) + ); + assert_eq!(config.get("myapp.setting"), Some(&"value".to_string())); + } + + /// `physical_differs` records whether column mapping is expected to rename the column: with + /// mapping on, the domain stores a generated identifier, so only the logical name is + /// predictable and the physical one must merely differ from it. + #[rstest::rstest] + #[case::no_clustering(None, None, None, false)] + #[case::clustered_no_column_mapping(Some(vec![vec!["region"]]), None, Some(vec!["region"]), false)] + #[case::clustered_with_column_mapping( + Some(vec![vec!["region"]]), + Some("name"), + Some(vec!["region"]), + true + )] + #[case::nested_no_column_mapping( + Some(vec![vec!["address", "city"]]), + None, + Some(vec!["address", "city"]), + false + )] + #[case::nested_with_column_mapping( + Some(vec![vec!["address", "city"]]), + Some("name"), + Some(vec!["address", "city"]), + true + )] + fn test_get_clustering_column_infos( + #[case] clustering_cols: Option>>, + #[case] column_mapping_mode: Option<&str>, + #[case] expected_logical: Option>, + #[case] physical_differs: bool, + ) { + use crate::transaction::create_table::create_table; + use crate::transaction::data_layout::DataLayout; + + let storage = Arc::new(InMemory::new()); + let engine = SyncEngine::new_with_store(storage); + let schema = Arc::new( + crate::schema::StructType::try_new(vec![ + crate::schema::StructField::nullable("id", DataType::INTEGER), + crate::schema::StructField::nullable("region", DataType::STRING), + crate::schema::StructField::nullable( + "address", + crate::schema::StructType::try_new(vec![ + crate::schema::StructField::nullable("city", DataType::STRING), + crate::schema::StructField::nullable("zip", DataType::INTEGER), + ]) + .unwrap(), + ), + ]) + .unwrap(), + ); + let _ = create_table("memory:///", schema, "test") + .fold_with(clustering_cols.as_ref(), |builder, cols| { + let columns = cols + .iter() + .map(|path| ColumnName::new(path.iter().copied())) + .collect(); + builder.with_data_layout(DataLayout::Clustered { columns }) + }) + .fold_with(column_mapping_mode, |builder, mode| { + builder.with_table_properties([("delta.columnMapping.mode", mode)]) + }) + .build( + &engine, + Box::new(crate::committer::FileSystemCommitter::new()), + ) + .unwrap() + .commit(&engine) + .unwrap(); + let snapshot = Snapshot::builder_for("memory:///").build(&engine).unwrap(); + let result = snapshot.get_clustering_column_infos(&engine).unwrap(); + + match expected_logical { + None => assert_eq!(result, None), + Some(logical) => { + let infos = result.expect("clustering infos should be present"); + assert_eq!(infos.len(), 1); + let info = &infos[0]; + assert_eq!( + info.logical_column, + ColumnName::new(logical.iter().copied()) + ); + assert_eq!(info.data_type, DataType::STRING); + assert_eq!(info.physical_column.path().len(), logical.len()); + if physical_differs { + assert_ne!(info.physical_column, info.logical_column); + for part in info.physical_column.path() { + assert!( + part.starts_with("col-"), + "physical path segment '{part}' should be a column-mapping id" + ); + } + } else { + assert_eq!(info.physical_column, info.logical_column); + } + } + } + } + + // === estimated_owned_heap_size === + /// Test that the estimated_owned_heap_size is correctly considering normal commit jsons, + /// checkpoint parts, and log compaction files. + #[test] + fn estimated_owned_heap_size_on_table_with_many_log_files() { + // Baseline: 101 commits (v0..=v100), no checkpoint, no compactions. + let (_engine, baseline_snap, _table) = test_utils::test_context!( + LogState::with_latest_version(100), + FeatureSet::empty(), + unpartitioned(), + checkpoint_json_stats(), + VersionTarget::Latest, + SyncEngine::new_with_store + ); + + let baseline_heap = baseline_snap.estimated_owned_heap_size_bytes(); + let struct_size = size_of::(); + // Heap size should be at least 5 times the stack size, to account for + // the 100 commits file metadata. + assert!( + baseline_heap > 5 * struct_size, + "baseline heap {baseline_heap} should exceed 5 * sizeof(Snapshot)={}", + 5 * struct_size + ); + + // 100 extra checkpoint parts: each contributes to heap. + // Kernel doesn't yet support writing multi-part checkpoints, so we manually add them here. + let snap_extra_checkpoints = + snapshot_with_extra_files(&baseline_snap, |listed, log_root| { + for i in 1..=100u32 { + let filename = + format!("00000000000000000099.checkpoint.{i:010}.0000000100.parquet"); + let location = log_root.join(&filename).unwrap(); + let part = ParsedLogPath::try_from(crate::FileMeta { + location, + last_modified: 0, + size: 100, + }) + .unwrap() + .unwrap(); + listed.checkpoint_parts.push(part); + } + }); + let delta_checkpoints = + snap_extra_checkpoints.estimated_owned_heap_size_bytes() - baseline_heap; + assert!( + delta_checkpoints >= 15_000, + "delta_checkpoints {delta_checkpoints} should be >= 15_000 for 100 checkpoint parts" + ); + + // 100 extra log compaction files: each contributes to heap. + // Kernel disables writing log compaction files currently, so we manually add them here. + let snap_extra_compactions = + snapshot_with_extra_files(&baseline_snap, |listed, log_root| { + for i in 0..100u64 { + let start = i * 10; + let end = start + 5; + let filename = format!("{start:020}.{end:020}.compacted.json"); + let location = log_root.join(&filename).unwrap(); + let comp = ParsedLogPath::try_from(crate::FileMeta { + location, + last_modified: 0, + size: 100, + }) + .unwrap() + .unwrap(); + listed.ascending_compaction_files.push(comp); + } + }); + let delta_compactions = + snap_extra_compactions.estimated_owned_heap_size_bytes() - baseline_heap; + assert!( + delta_compactions >= 15_000, + "delta_compactions {delta_compactions} should be >= 15_000 for 100 compaction files" + ); + } + + /// Two tables that differ only in schema width: the wider schema should bump + /// estimated_owned_heap_size by approximately the schemaString delta. + #[test] + fn estimated_owned_heap_size_reflects_schema_string() { + fn snap_with_schema(schema: SchemaRef) -> SnapshotRef { + let store = Arc::new(InMemory::new()); + let engine = SyncEngine::new_with_store(store); + create_table("memory:///", schema, "test") + .build(&engine, Box::new(FileSystemCommitter::new())) + .unwrap() + .commit(&engine) + .unwrap() + .unwrap_committed(); + Snapshot::builder_for("memory:///").build(&engine).unwrap() + } + + let small_schema = Arc::new( + StructType::try_new(vec![StructField::nullable("a", DataType::INTEGER)]).unwrap(), + ); + let wide_schema = Arc::new( + StructType::try_new( + (0..50) + .map(|i| StructField::nullable(format!("field_{i:03}"), DataType::STRING)) + .collect::>(), + ) + .unwrap(), + ); + + let snap_small = snap_with_schema(small_schema); + let snap_wide = snap_with_schema(wide_schema); + + let schema_str_delta = snap_wide + .table_configuration() + .metadata() + .schema_string() + .capacity() + - snap_small + .table_configuration() + .metadata() + .schema_string() + .capacity(); + let heap_delta = snap_wide.estimated_owned_heap_size_bytes() + - snap_small.estimated_owned_heap_size_bytes(); + // Tables differ only in schemaString, so heap_delta should be approximately the + // schema_str_delta. + let ratio = heap_delta as f64 / schema_str_delta as f64; + assert!( + (0.8..=1.2).contains(&ratio), + "heap_delta {heap_delta} should be within 20% of schema_str_delta {schema_str_delta} (ratio = {ratio:.3})" + ); + } + + #[test] + fn estimated_owned_heap_size_for_version_zero() { + let table = TestTableBuilder::new().build().unwrap(); + let engine = SyncEngine::new_with_store(table.store().clone()); + let snapshot = Snapshot::builder_for(table.table_root()) + .build(&engine) + .unwrap(); + + let heap = snapshot.estimated_owned_heap_size_bytes(); + assert!(heap > 0, "heap size should be nonzero"); + assert!( + heap < 10000, + "heap size {heap} unexpectedly large for v0 snapshot" + ); + } + + fn snapshot_with_extra_files(baseline: &SnapshotRef, mutate: F) -> Snapshot + where + F: FnOnce(&mut LogSegmentFiles, &Url), + { + let mut new_log_segment = baseline.log_segment().clone(); + mutate(&mut new_log_segment.listed, &new_log_segment.log_root); + Snapshot::new(new_log_segment, baseline.table_configuration().clone()).unwrap() + } +} diff --git a/kernel/src/snapshot/snapshot_crc.rs b/kernel/src/snapshot/snapshot_crc.rs new file mode 100644 index 0000000000..a468097a6e --- /dev/null +++ b/kernel/src/snapshot/snapshot_crc.rs @@ -0,0 +1,130 @@ +//! The CRC a [`Snapshot`] holds, and the version semantics gating how it may be used. +//! +//! [`Snapshot`]: super::Snapshot + +use std::sync::Arc; + +use crate::crc::Crc; +use crate::utils::require; +use crate::{DeltaResult, Error, Version}; + +/// The newest [`Crc`] a snapshot resolved, at or before the snapshot's version. It is one CRC, +/// read from disk (or computed) at most once; keeping it lets later queries and CRC writes reuse +/// the already-parsed state instead of re-reading it. A `Some(crc)` is one of: +/// - at the snapshot version: authoritative, servable at zero I/O via [`Self::at_version`]; +/// - stale (older): kept only as a base for [`Self::base`], never served as authoritative. +/// +/// [`Snapshot`]: super::Snapshot +#[derive(Debug)] +pub(super) struct SnapshotCrc { + crc: Option>, + snapshot_version: Version, +} + +impl SnapshotCrc { + /// Wrap the CRC a snapshot at `snapshot_version` resolved. A `Some(crc)` must sit in + /// `[checkpoint_version, snapshot_version]`. + pub(super) fn try_new( + crc: Option>, + snapshot_version: Version, + checkpoint_version: Option, + ) -> DeltaResult { + if let Some(crc) = crc.as_ref() { + require!( + crc.version <= snapshot_version, + Error::internal_error(format!( + "CRC version {} is ahead of snapshot version {snapshot_version}", + crc.version + )) + ); + require!( + checkpoint_version.is_none_or(|ckpt| crc.version >= ckpt), + Error::internal_error(format!( + "CRC version {} is below checkpoint version {checkpoint_version:?}", + crc.version + )) + ); + } + Ok(Self { + crc, + snapshot_version, + }) + } + + /// The CRC iff it sits exactly at the snapshot version, so its cached values are authoritative + /// at zero I/O. Returns `None` for a stale CRC. + pub(super) fn at_version(&self) -> Option<&Arc> { + self.crc + .as_ref() + .filter(|crc| crc.version == self.snapshot_version) + } + + /// The held CRC, at-version or stale, for callers that reuse it regardless of version + /// (e.g. checksum writes). + pub(super) fn base(&self) -> Option<&Arc> { + self.crc.as_ref() + } +} + +#[cfg(test)] +mod tests { + use test_utils::assert_result_error_with_message; + + use super::*; + use crate::crc::{FileStats, FileStatsState}; + + fn crc_at(version: Version) -> Arc { + Arc::new(Crc { + version, + file_stats_state: FileStatsState::Complete(FileStats { + num_files: 1, + table_size_bytes: 100, + file_size_histogram: None, + }), + ..Default::default() + }) + } + + #[test] + fn new_rejects_crc_ahead_of_snapshot() { + assert_result_error_with_message( + SnapshotCrc::try_new(Some(crc_at(6)), 5, None), + "is ahead of snapshot version", + ); + } + + #[test] + fn new_rejects_crc_below_checkpoint() { + assert_result_error_with_message( + SnapshotCrc::try_new(Some(crc_at(2)), 5, Some(3)), + "is below checkpoint version", + ); + } + + #[test] + fn new_accepts_crc_at_checkpoint() { + let holder = SnapshotCrc::try_new(Some(crc_at(3)), 5, Some(3)).unwrap(); + assert_eq!(holder.base().map(|c| c.version), Some(3)); + } + + #[test] + fn at_version_serves_crc_matching_snapshot_version() { + let holder = SnapshotCrc::try_new(Some(crc_at(5)), 5, None).unwrap(); + assert_eq!(holder.at_version().map(|c| c.version), Some(5)); + assert_eq!(holder.base().map(|c| c.version), Some(5)); + } + + #[test] + fn stale_crc_is_base_but_not_at_version() { + let holder = SnapshotCrc::try_new(Some(crc_at(3)), 5, None).unwrap(); + assert!(holder.at_version().is_none()); + assert_eq!(holder.base().map(|c| c.version), Some(3)); + } + + #[test] + fn none_holder_serves_nothing() { + let holder = SnapshotCrc::try_new(None, 5, None).unwrap(); + assert!(holder.at_version().is_none()); + assert!(holder.base().is_none()); + } +} diff --git a/kernel/src/struct_patch.rs b/kernel/src/struct_patch.rs new file mode 100644 index 0000000000..c056308caa --- /dev/null +++ b/kernel/src/struct_patch.rs @@ -0,0 +1,1283 @@ +//! Struct patches: sparse, `O(changes)` edits to the fields of an input struct. +//! +//! A struct patch keeps, drops, replaces, or inserts fields relative to an input struct without +//! enumerating untouched fields. Patches are built with a [`StructPatchBuilder`], which validates +//! conflicting operations and lowers nested field paths into recursive patches. Three flavors share +//! the same builder surface, differing only in their item type and terminal `build` step: +//! +//! * [`ExpressionStructPatchBuilder`](crate::expressions::ExpressionStructPatchBuilder) emits +//! expressions and produces a sparse [`ExpressionStructPatch`] that is embedded in an +//! [`Expression`] and applied to data at evaluation time. +//! * [`SchemaStructPatchBuilder`](crate::schema::SchemaStructPatchBuilder) emits schema fields and +//! produces an output [`StructType`] directly from an input schema. +//! * [`ProjectionStructPatchBuilder`] pairs each output field with the expression that produces it +//! and lowers both at once to a matched ([`SchemaRef`], `[ExpressionRef`]) pair. +//! +//! # Worked example +//! +//! Every builder call names one edit relative to an input field: +//! +//! ```ignore +//! builder +//! .prepend(first) +//! .insert_after("a", x) +//! .replace("b", bb) +//! .drop("c") +//! .append(last) +//! ``` +//! +//! Applied to `{ a, b, c }`, walking the input fields in order: +//! +//! ```text +//! input field: (none) a b c (none) +//! edit: prepend keep +after replace drop append +//! emits: first a, x bb nothing last +//! +//! output: { first, a, x, bb, last } +//! ``` +//! +//! Field `a` is untouched and passes through in place, which is what makes the patch cost +//! `O(edits)` rather than `O(struct width)`. The `_at` variants (e.g. +//! [`insert_after_at`](StructPatchBuilder::insert_after_at)) apply the same edits to a nested +//! struct named by path. +//! +//! # Flattening to a dense projection +//! +//! An engine whose projection operator takes one expression per output column has to expand the +//! patch, which is the walk the example above traces. Emit +//! [`prepended_fields`](ExpressionStructPatch::prepended_fields), then visit the input struct's +//! fields in order: look each one up in +//! [`field_patches`](ExpressionStructPatch::field_patches), emit the field itself when the entry is +//! absent or sets [`keep_input`](ExpressionFieldPatch::keep_input), and emit that entry's +//! [`insertions`](ExpressionFieldPatch::insertions) after it. An entry that keeps nothing and +//! inserts nothing drops the field. Finish with +//! [`appended_fields`](ExpressionStructPatch::appended_fields). A nested patch arrives as an +//! [`Expression::StructPatch`] insertion, so the same walk recurses with that patch's +//! [`input_path`](ExpressionStructPatch::input_path) as the struct to expand against. +//! +//! [`ProjectionStructPatchBuilder`] hands back the flattened schema already, so a caller building +//! both halves can pair its dense field list with the expressions this walk produces rather than +//! deriving the field names again. + +use std::collections::{hash_map, HashMap}; +use std::sync::Arc; + +use serde::{Deserialize, Serialize}; + +use crate::expressions::{ColumnName, Expression, ExpressionRef}; +use crate::schema::{DataType, SchemaRef, StructField, StructType}; +use crate::utils::{CollectInto, FoldWithOption as _}; +use crate::{DeltaResult, Error}; + +// === Raw expression patch === + +/// A patch affecting a single input field. +/// +/// A field patch can keep or omit its input field, then insert zero or more expressions after the +/// input field's output position. +#[derive(Debug, Default, Clone, PartialEq, Serialize, Deserialize)] +pub struct ExpressionFieldPatch { + /// If true, the original input field is emitted before this patch's insertions. If false, the + /// input field is omitted and the first insertion, if present, occupies the input field's + /// output position. + pub keep_input: bool, + /// Expressions emitted after this field's output position. + pub insertions: Vec, + /// If true, this patch is silently ignored when the input field does not exist. Otherwise, a + /// missing input field produces an error. + pub optional: bool, +} + +/// A sparse expression patch over the fields of one input struct. +/// +/// `ExpressionStructPatch` achieves `O(changes)` space complexity instead of `O(schema_width)` by +/// only specifying fields that actually change (inserted, replaced, or deleted). Any input field +/// not specifically mentioned by the patch is passed through, unmodified and with the same relative +/// field ordering. This is particularly useful for wide schemas where only a few columns need to be +/// modified and/or dropped, or where a small number of columns need to be injected. +#[derive(Debug, Clone, PartialEq, Default, Serialize, Deserialize)] +pub struct ExpressionStructPatch { + /// The path to the nested input struct this patch operates on (if any). If no path is given, + /// the patch operates directly on top-level columns. + pub input_path: Option, + /// A mapping from named input fields to the patch to be performed on each field. + pub field_patches: HashMap, + /// A list of new fields to emit before processing the first input field. + pub prepended_fields: Vec, + /// A list of new fields to emit after all input fields and field-specific insertions. + pub appended_fields: Vec, +} + +impl ExpressionStructPatch { + /// True if this patch makes no changes to the selected input struct. + pub fn is_empty(&self) -> bool { + self.prepended_fields.is_empty() + && self.appended_fields.is_empty() + && self.field_patches.is_empty() + } + + /// None if this is a top-level patch. Otherwise, the path of this nested patch. + pub fn input_path(&self) -> Option<&ColumnName> { + self.input_path.as_ref() + } +} + +// === Builder === + +/// Builds a sparse struct patch from a sequence of requested patch operations. +/// +/// The builder records user intent, checks for conflicting destructive operations, and lowers +/// nested field paths into recursive struct patches. The same builder surface drives both +/// expression patching +/// ([`ExpressionStructPatchBuilder`](crate::expressions::ExpressionStructPatchBuilder)) and schema +/// patching ([`SchemaStructPatchBuilder`](crate::schema::SchemaStructPatchBuilder)); only the +/// terminal `build` step differs. +#[derive(Debug)] +pub struct StructPatchBuilder { + /// None for a top-level patch; otherwise the path of the nested struct this patch targets. + input_path: Option, + /// The patch tree assembled so far, with each builder call applied eagerly. + root: StructPatchNode, + /// The first error produced by a builder call, surfaced by `build`. Once set, later calls are + /// skipped so the original (most relevant) error is preserved. + error: DeltaResult<()>, +} + +/// The patch builder internally represents the in-progress patch specification as a tree of struct +/// and field patches, incrementally built up by validated builder calls. For example, the +/// following sequence of calls (in any order): +/// +/// ```ignore +/// let builder = StructPatchBuilder::new() +/// .prepend(item1) +/// .append(item2) +/// .drop("a") +/// .replace("b", item3) +/// .insert_after("b", item4) +/// .insert_after("b", item5) +/// .insert_after_at(["c"], "x", item6); +/// ``` +/// +/// ... produces the following tree structure: +/// +/// ```text +/// root: StructPatchNode (patches applied to the top-level struct) +/// ├── prepended_fields: [item1] +/// ├── appended_fields: [item2] +/// └── fields: +/// ├── "a" → FieldPatchNode +/// │ ├── op: Drop +/// │ └── insert_after: [] +/// │ +/// ├── "b" → FieldPatchNode +/// │ ├── op: Replace(item3) +/// │ └── insert_after: [item4, item5] +/// │ +/// └── "c" → FieldPatchNode +/// ├── op: Nested(StructPatchNode) (recurse into nested struct) +/// │ ├── prepended_fields: [] +/// │ ├── appended_fields: [] +/// │ └── fields: +/// │ └── "x" → FieldPatchNode +/// │ ├── op: Keep +/// │ └── insert_after: [item6] +/// └── insert_after: [] +/// ``` +/// +/// Conflicting operations ({Replace, Drop, Nested} x {Replace, Drop}) are detected and rejected +/// along the way, so the tree is always valid. If no conflicts were detected, the final patch is +/// produced by recursively lowering the internal tree into the output patch type, at which point it +/// is no longer possible to distinguish e.g. Drop+Insert from Replace. +#[derive(Debug)] +struct StructPatchNode { + prepended_fields: Vec, + appended_fields: Vec, + fields: HashMap>, +} + +// Do not derive `Default` -- it emits `impl Default`, even though none of the +// fields' defaults actually requires `Item: Default`. +impl Default for StructPatchNode { + fn default() -> Self { + Self { + prepended_fields: Vec::new(), + appended_fields: Vec::new(), + fields: HashMap::new(), + } + } +} + +#[derive(Debug)] +struct FieldPatchNode { + action: FieldPatchOp, + insert_after: Vec, +} + +// Do not derive `Default` -- it emits `impl Default`, even though none of the +// fields' defaults actually requires `Item: Default`. +impl Default for FieldPatchNode { + fn default() -> Self { + Self { + action: FieldPatchOp::default(), + insert_after: Vec::new(), + } + } +} + +/// Describes what happens to the input field itself; any insertions after the input are field +/// tracked separately. A node with `Keep` is either freshly-inserted into the patch tree (with the +/// true op to be applied immediately after), or is the otherwise unmodified named anchor of 1+ +/// insert-after operations. A `Nested` operation is like special `Replace`, where the replacement +/// is the computed result of applying patches to one or more of the field's children. +#[derive(Debug)] +enum FieldPatchOp { + Keep, + Drop { optional: bool }, + Replace(Item), + Nested(Box>), +} + +// Do not derive `Default` -- it emits `impl Default`, even though the default enum +// variant does not contain any `Item`. Also suppress the clippy suggestion to define a `#[default]` +// enum variant, since that's unrelated to the problematic `Item: Default` bound. +#[allow(clippy::derivable_impls)] +impl Default for FieldPatchOp { + fn default() -> Self { + FieldPatchOp::Keep + } +} + +impl FieldPatchOp { + fn is_keep(&self) -> bool { + matches!(self, FieldPatchOp::Keep) + } + + fn is_optional_drop(&self) -> bool { + matches!(self, FieldPatchOp::Drop { optional: true }) + } +} + +// Empty path passed by top-level methods to their nested-path counterparts. +const TOP_LEVEL: &[String] = &[]; + +impl StructPatchBuilder { + /// Creates a new top-level patch builder. + #[allow(clippy::new_without_default)] + pub fn new() -> Self { + Self { + input_path: None, + root: StructPatchNode::default(), + error: Ok(()), + } + } + + /// Creates a new builder that operates on fields of a nested struct identified by `path`. + pub fn new_nested(path: impl CollectInto) -> Self { + Self { + input_path: Some(path.collect_into()), + root: StructPatchNode::default(), + error: Ok(()), + } + } + + /// Records a field drop. + pub fn drop(self, field_name: impl Into) -> Self { + self.drop_at(TOP_LEVEL, field_name) + } + + /// Records a field drop in a nested struct. + pub fn drop_at( + self, + struct_path: impl CollectInto, + field_name: impl Into, + ) -> Self { + self.apply_at(struct_path, |node| node.drop(field_name, false)) + } + + /// Records an optional field drop. + pub fn drop_if_exists(self, field_name: impl Into) -> Self { + self.drop_if_exists_at(TOP_LEVEL, field_name) + } + + /// Records an optional field drop in a nested struct. + pub fn drop_if_exists_at( + self, + struct_path: impl CollectInto, + field_name: impl Into, + ) -> Self { + self.apply_at(struct_path, |node| node.drop(field_name, true)) + } + + /// Records a field replacement. + pub fn replace(self, field_name: impl Into, item: impl Into) -> Self { + self.replace_at(TOP_LEVEL, field_name, item) + } + + /// Records a field replacement in a nested struct. + pub fn replace_at( + self, + struct_path: impl CollectInto, + field_name: impl Into, + item: impl Into, + ) -> Self { + self.apply_at(struct_path, |node| { + node.set_action(field_name, FieldPatchOp::Replace(item.into())) + }) + } + + /// Records an item to emit before processing the first input field. + pub fn prepend(self, item: impl Into) -> Self { + self.prepend_at(TOP_LEVEL, item) + } + + /// Records an item to emit before processing the first input field of a nested struct. + pub fn prepend_at( + self, + struct_path: impl CollectInto, + item: impl Into, + ) -> Self { + self.apply_at(struct_path, |node| { + node.prepended_fields.push(item.into()); + Ok(()) + }) + } + + /// Records an item to insert after the named field. + pub fn insert_after(self, field_name: impl Into, item: impl Into) -> Self { + self.insert_after_at(TOP_LEVEL, field_name, item) + } + + /// Records an item to insert after the named field in a nested struct. + pub fn insert_after_at( + self, + struct_path: impl CollectInto, + field_name: impl Into, + item: impl Into, + ) -> Self { + self.apply_at(struct_path, |node| { + node.insert_after(field_name, item.into()) + }) + } + + /// Records an item to append after all input fields and field-specific insertions. + pub fn append(self, item: impl Into) -> Self { + self.append_at(TOP_LEVEL, item) + } + + /// Records an item to append after all fields of a nested struct. + pub fn append_at( + self, + struct_path: impl CollectInto, + item: impl Into, + ) -> Self { + self.apply_at(struct_path, |node| { + node.appended_fields.push(item.into()); + Ok(()) + }) + } + + // Applies `op` to the (possibly nested) struct node identified by `struct_path` (an empty path + // targets the input struct directly). Errors are deferred: the first failure is stashed in + // `self.error` and surfaced by `build`, and once set, later operations are skipped so the + // original error is preserved. + fn apply_at( + mut self, + struct_path: impl CollectInto, + op: impl FnOnce(&mut StructPatchNode) -> DeltaResult<()>, + ) -> Self { + if self.error.is_ok() { + let path = struct_path.collect_into(); + self.error = self.root.child_at_mut(&path).and_then(op); + } + self + } + + fn begin_build( + self, + input_schema: &StructType, + ) -> DeltaResult<(StructPatchNode, Option, &StructType)> { + self.error?; + let source_schema = resolve_input_schema(input_schema, self.input_path.as_ref())?; + Ok((self.root, self.input_path, source_schema)) + } +} + +impl StructPatchNode { + /// Records an item to insert immediately after the named input field. + fn insert_after( + &mut self, + field_name: impl Into, + item: impl Into, + ) -> DeltaResult<()> { + let entry = self.field_patch_mut(field_name.into(), |field_name, entry| { + if entry.action.is_optional_drop() { + return Err(Error::generic(format!( + "Field '{field_name}' cannot combine optional drop with insert-after" + ))); + } + Ok(()) + })?; + entry.insert_after.push(item.into()); + Ok(()) + } + + /// Records a drop of the named input field. `optional` tolerates an absent field at evaluation + /// time, but cannot combine with insertions after that field. + fn drop(&mut self, field_name: impl Into, optional: bool) -> DeltaResult<()> { + self.set_action(field_name, FieldPatchOp::Drop { optional }) + } + + /// Records the input field action (drop/replace/patch) for the named input field. Only one such + /// action is allowed per field. + fn set_action( + &mut self, + field_name: impl Into, + action: FieldPatchOp, + ) -> DeltaResult<()> { + let entry = self.field_patch_mut(field_name.into(), |field_name, entry| { + if !entry.action.is_keep() { + return Err(Error::generic(format!( + "Field '{field_name}' has multiple input field actions" + ))); + } + if action.is_optional_drop() && !entry.insert_after.is_empty() { + return Err(Error::generic(format!( + "Field '{field_name}' cannot combine optional drop with insert-after" + ))); + } + Ok(()) + })?; + entry.action = action; + Ok(()) + } + + fn child_at_mut(&mut self, path: &[String]) -> DeltaResult<&mut Self> { + let Some((field_name, remaining)) = path.split_first() else { + return Ok(self); + }; + // All ancestors along the path to a given field must be Nested. We can convert Keep (a + // newly-created no-op or existing insert-after) to Nested, but not Drop or Replace. + let state = self.fields.entry(field_name.to_string()).or_default(); + if state.action.is_keep() { + state.action = FieldPatchOp::Nested(Box::default()); + } + let FieldPatchOp::Nested(node) = &mut state.action else { + return Err(Error::generic(format!( + "Cannot patch nested fields under dropped/replaced field '{field_name}'" + ))); + }; + node.child_at_mut(remaining) + } + + /// Fetches mutable state for the requested field. + /// * If not yet present, create and return a new defaulted entry + /// * If already existing, invoke the validator on it and return the entry on success + fn field_patch_mut( + &mut self, + field_name: String, + validate_existing: impl FnOnce(&str, &FieldPatchNode) -> DeltaResult<()>, + ) -> DeltaResult<&mut FieldPatchNode> { + match self.fields.entry(field_name) { + hash_map::Entry::Vacant(entry) => Ok(entry.insert(FieldPatchNode::default())), + hash_map::Entry::Occupied(entry) => { + validate_existing(entry.key(), entry.get())?; + Ok(entry.into_mut()) + } + } + } +} + +/// Resolves the struct schema targeted by a top-level or nested struct patch. +fn resolve_input_schema<'a>( + input_schema: &'a StructType, + input_path: Option<&ColumnName>, +) -> DeltaResult<&'a StructType> { + let input_path = match input_path { + Some(input_path) if !input_path.path().is_empty() => input_path, + _ => return Ok(input_schema), + }; + let field = input_schema.field_at(input_path)?; + let DataType::Struct(nested_schema) = field.data_type() else { + return Err(Error::generic(format!( + "Patching failed: input path '{input_path}' references a non-struct field" + ))); + }; + Ok(nested_schema) +} + +// === Expression lowering === + +impl StructPatchBuilder { + /// Builds the final expression patch. + /// + /// # Errors + /// + /// Returns an error when builder calls request multiple drop/replace operations for the + /// same field, or when a destructive operation on one field overlapped with an operation on a + /// nested child field. + pub fn build(self) -> DeltaResult { + self.error?; + Ok(self.root.to_expr_patch(self.input_path)) + } +} + +impl TryFrom> for ExpressionStructPatch { + type Error = Error; + + fn try_from(builder: StructPatchBuilder) -> DeltaResult { + builder.build() + } +} + +trait ExpressionItem: Sized { + fn expr(&self) -> &ExpressionRef; + + fn exprs(items: &[Self]) -> impl Iterator { + items.iter().map(Self::expr).cloned() + } +} + +impl ExpressionItem for ExpressionRef { + fn expr(&self) -> &ExpressionRef { + self + } +} + +impl ExpressionItem for ProjectionItem { + fn expr(&self) -> &ExpressionRef { + &self.1 + } +} + +impl StructPatchNode { + fn to_expr_patch(&self, input_path: Option) -> ExpressionStructPatch { + let mut field_patches = HashMap::with_capacity(self.fields.len()); + for (field_name, state) in &self.fields { + let patch = state.to_expr_field_patch(input_path.as_ref(), field_name); + field_patches.insert(field_name.clone(), patch); + } + + ExpressionStructPatch { + field_patches, + prepended_fields: Item::exprs(&self.prepended_fields).collect(), + appended_fields: Item::exprs(&self.appended_fields).collect(), + input_path, + } + } +} + +impl FieldPatchNode { + fn to_expr_field_patch( + &self, + parent_input_path: Option<&ColumnName>, + field_name: &str, + ) -> ExpressionFieldPatch { + let mut field_patch = ExpressionFieldPatch::default(); + match &self.action { + FieldPatchOp::Keep => { + field_patch.keep_input = true; + } + FieldPatchOp::Drop { optional } => { + field_patch.optional = *optional; + } + FieldPatchOp::Replace(expr) => { + field_patch.insertions.push(expr.expr().clone()); + } + FieldPatchOp::Nested(node) => { + let child_input_path = join_prefix(parent_input_path, field_name); + let child_patch = node.to_expr_patch(Some(child_input_path)); + let child_patch = Arc::new(Expression::StructPatch(child_patch)); + field_patch.insertions.push(child_patch); + } + } + + let insert_after = Item::exprs(&self.insert_after); + field_patch.insertions.extend(insert_after); + field_patch + } +} + +// === Schema lowering === + +impl StructPatchBuilder { + /// Builds the output struct schema for this patch over `input_schema`. + /// + /// If this builder targets a nested path (via [`new_nested`](Self::new_nested)), the returned + /// schema is the patched schema for the nested struct at that path, not the full top-level + /// input schema. + /// + /// # Errors + /// + /// Returns an error if a builder call produced a conflicting operation, the input path cannot + /// be resolved to a struct, a required field patch references a missing input field, a nested + /// field patch targets a non-struct field, or the resulting output schema is invalid. + pub fn build(self, input_schema: &StructType) -> DeltaResult { + let (root, _input_path, source_schema) = self.begin_build(input_schema)?; + StructType::try_new(schema_walk(root, source_schema)?) + } +} + +type ProjectionItem = (StructField, ExpressionRef); + +trait SchemaPatchItem { + fn into_field(self) -> StructField; + + fn into_fields(items: Vec) -> impl Iterator + where + Self: Sized, + { + items.into_iter().map(Self::into_field) + } +} + +impl SchemaPatchItem for StructField { + fn into_field(self) -> StructField { + self + } +} + +impl SchemaPatchItem for ProjectionItem { + fn into_field(self) -> StructField { + self.0 + } +} + +fn schema_walk( + node: StructPatchNode, + input_schema: &StructType, +) -> DeltaResult> { + let mut fields = node.fields; + let mut output: Vec<_> = Item::into_fields(node.prepended_fields).collect(); + output.reserve(input_schema.num_fields() + fields.len()); + + for input_field in input_schema.fields() { + let field_name = input_field.name(); + let field_patch = fields.remove(field_name).unwrap_or_default(); + match field_patch.action { + FieldPatchOp::Drop { .. } => {} + FieldPatchOp::Keep => output.push(input_field.clone()), + FieldPatchOp::Replace(item) => output.push(item.into_field()), + FieldPatchOp::Nested(node) => { + let DataType::Struct(nested_schema) = input_field.data_type() else { + return Err(Error::generic(format!( + "Cannot patch nested fields under non-struct field '{}'", + input_field.name() + ))); + }; + let children = schema_walk(*node, nested_schema)?; + let field = StructField::new( + input_field.name(), + StructType::try_new(children)?, + input_field.nullable, + ); + output.push(field.with_metadata(input_field.metadata.clone())); + } + } + output.extend(Item::into_fields(field_patch.insert_after)); + } + + if let Some((field_name, _)) = fields + .iter() + .find(|(_, state)| !state.action.is_optional_drop()) + { + return Err(Error::generic(format!( + "Field to patch does not exist: {field_name}" + ))); + } + + output.extend(Item::into_fields(node.appended_fields)); + Ok(output) +} + +// === Projection lowering === + +/// Builds schema and expression patches together over an input schema. +/// +/// Emitted fields are paired with the expression that produces them, keeping the output schema and +/// sparse expression patch structurally aligned. Because it is bound to the input schema, +/// [`replace_expr`](Self::replace_expr) can preserve an existing [`StructField`] while replacing +/// only its expression. +#[derive(Debug)] +pub struct ProjectionStructPatchBuilder<'a> { + input_schema: &'a StructType, + inner: StructPatchBuilder, +} + +/// Generates a [`ProjectionStructPatchBuilder`] emission method body: take the wrapped builder, +/// apply a `with_*` operation whose emitted item is `field` and `expr`, and reinstall the result. +/// The macro lowers `field, expr` into the `(StructField, ExpressionRef)` item the inner builder +/// expects, so call sites never spell the tuple. Any `$arg` (e.g. a struct path and/or field name) +/// precedes the emitted item in the inner call, matching the inner builder's signatures. +macro_rules! delegate { + ($self:ident, $method:ident, $field:expr, $expr:expr $(, $arg:expr)*) => {{ + $self.inner = $self.inner.$method($($arg,)* ($field, $expr.into())); + $self + }}; +} + +impl<'a> ProjectionStructPatchBuilder<'a> { + fn existing_input_field( + &self, + struct_path: &ColumnName, + field_name: &str, + ) -> DeltaResult { + let field_path: ColumnName = [ + self.inner.input_path.clone().unwrap_or_default(), + struct_path.clone(), + ColumnName::new([field_name]), + ] + .into_iter() + .collect(); + let field = self.input_schema.field_at(&field_path)?; + Ok(field.clone()) + } + + /// Creates a new top-level projection patch builder over `input_schema`. + pub fn new(input_schema: &'a StructType) -> Self { + Self { + input_schema, + inner: StructPatchBuilder::new(), + } + } + + /// Creates a projection patch builder over `input_schema` that operates on the nested struct + /// identified by `path`. + pub fn new_nested(input_schema: &'a StructType, path: impl CollectInto) -> Self { + Self { + input_schema, + inner: StructPatchBuilder::new_nested(path), + } + } + + /// Returns the schema this projection patch is applied to. + pub fn input_schema(&self) -> &StructType { + self.input_schema + } + + // === Drops (no emitted field/expression) === + + /// Records a field drop. The dropped field is omitted from both the output schema and the + /// emitted expressions. + pub fn drop(mut self, field_name: impl Into) -> Self { + self.inner = self.inner.drop(field_name); + self + } + + /// Records a field drop in a nested struct. + pub fn drop_at( + mut self, + struct_path: impl CollectInto, + field_name: impl Into, + ) -> Self { + self.inner = self.inner.drop_at(struct_path, field_name); + self + } + + /// Records an optional field drop, tolerated when the input field is absent. + pub fn drop_if_exists(mut self, field_name: impl Into) -> Self { + self.inner = self.inner.drop_if_exists(field_name); + self + } + + /// Records an optional field drop in a nested struct. + pub fn drop_if_exists_at( + mut self, + struct_path: impl CollectInto, + field_name: impl Into, + ) -> Self { + self.inner = self.inner.drop_if_exists_at(struct_path, field_name); + self + } + + // === Field + expression emissions === + + /// Replaces the named field with `field`, produced by `expr`. + pub fn replace( + mut self, + field_name: impl Into, + field: StructField, + expr: impl Into, + ) -> Self { + delegate!(self, replace, field, expr, field_name) + } + + /// Replaces the named field in a nested struct with `field`, produced by `expr`. + pub fn replace_at( + mut self, + struct_path: impl CollectInto, + field_name: impl Into, + field: StructField, + expr: impl Into, + ) -> Self { + delegate!(self, replace_at, field, expr, struct_path, field_name) + } + + /// Replaces the named field's expression while preserving its input [`StructField`]. + /// + /// Any lookup error is recorded on the builder and returned by [`build`](Self::build). + pub fn replace_expr( + self, + field_name: impl Into, + expr: impl Into, + ) -> Self { + self.replace_expr_at(TOP_LEVEL, field_name, expr) + } + + /// Replaces a nested field's expression while preserving its input [`StructField`]. + /// + /// Any lookup error is recorded on the builder and returned by [`build`](Self::build). + pub fn replace_expr_at( + mut self, + struct_path: impl CollectInto, + field_name: impl Into, + expr: impl Into, + ) -> Self { + if self.inner.error.is_err() { + return self; + } + let struct_path = struct_path.collect_into(); + let field_name = field_name.into(); + match self.existing_input_field(&struct_path, &field_name) { + Ok(field) => self.replace_at(struct_path, field_name, field, expr), + Err(error) => { + self.inner.error = Err(error); + self + } + } + } + + /// Emits `field`, produced by `expr`, before all input fields. + pub fn prepend(mut self, field: StructField, expr: impl Into) -> Self { + delegate!(self, prepend, field, expr) + } + + /// Emits `field`, produced by `expr`, before all fields of a nested struct. + pub fn prepend_at( + mut self, + struct_path: impl CollectInto, + field: StructField, + expr: impl Into, + ) -> Self { + delegate!(self, prepend_at, field, expr, struct_path) + } + + /// Emits `field`, produced by `expr`, immediately after the named input field. + pub fn insert_after( + mut self, + field_name: impl Into, + field: StructField, + expr: impl Into, + ) -> Self { + delegate!(self, insert_after, field, expr, field_name) + } + + /// Emits `field`, produced by `expr`, after the named input field of a nested struct. + pub fn insert_after_at( + mut self, + struct_path: impl CollectInto, + field_name: impl Into, + field: StructField, + expr: impl Into, + ) -> Self { + delegate!(self, insert_after_at, field, expr, struct_path, field_name) + } + + /// Emits `field`, produced by `expr`, after all input fields and field-specific insertions. + pub fn append(mut self, field: StructField, expr: impl Into) -> Self { + delegate!(self, append, field, expr) + } + + /// Emits `field`, produced by `expr`, after all fields of a nested struct. + pub fn append_at( + mut self, + struct_path: impl CollectInto, + field: StructField, + expr: impl Into, + ) -> Self { + delegate!(self, append_at, field, expr, struct_path) + } + + // === Build === + + /// Builds a full output schema together with a matching sparse struct-patch expression. + /// Untouched input fields pass through implicitly and only inserted, replaced, dropped, or + /// nested fields are recorded. + /// + /// The returned schema is always dense (it enumerates every output field), since a schema has + /// no sparse representation; only the expression side is sparse. The two halves stay + /// structurally consistent: applying the patch to a row of the builder's input schema yields a + /// struct matching the returned schema. + /// + /// # Errors + /// + /// Returns an error if a `with_*` call produced a conflicting operation, the input path cannot + /// be resolved to a struct, a required field patch references a missing input field, a nested + /// field patch targets a non-struct field, or the resulting output schema is invalid. + pub fn build(self) -> DeltaResult<(SchemaRef, ExpressionRef)> { + let (root, input_path, source_schema) = self.inner.begin_build(self.input_schema)?; + let patch = root.to_expr_patch(input_path); + let schema = StructType::try_new(schema_walk(root, source_schema)?)?; + Ok((Arc::new(schema), Arc::new(Expression::StructPatch(patch)))) + } +} + +/// Joins an optional input prefix with a field name into a (possibly nested) input column path. +fn join_prefix(prefix: Option<&ColumnName>, name: &str) -> ColumnName { + let leaf = ColumnName::new([name]); + leaf.fold_with(prefix, |leaf, prefix| prefix.join(&leaf)) +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use rstest::rstest; + + use crate::expressions::{ + lit, Expression as Expr, ExpressionRef, ExpressionStructPatch, ExpressionStructPatchBuilder, + }; + use crate::schema::{DataType, SchemaStructPatchBuilder, StructField, StructType}; + use crate::struct_patch::ProjectionStructPatchBuilder; + use crate::unit_test_utils::assert_result_error_with_message; + + fn projection_patch(expr: &ExpressionRef) -> &ExpressionStructPatch { + let Expr::StructPatch(patch) = expr.as_ref() else { + panic!("Expected struct patch expression"); + }; + patch + } + + #[test] + fn struct_patch_builder_lowers_nested_paths_to_raw_patches() { + let patch = ExpressionStructPatchBuilder::new() + .drop_at(["add"], "gone") + .replace_at(["add"], "stub", lit("replaced")) + .insert_after_at(["add"], "x", lit(true)) + .insert_after("add", lit("after_add")) + .build() + .unwrap(); + + let add_patch = patch.field_patches.get("add").unwrap(); + assert!(!add_patch.keep_input); + assert_eq!(add_patch.insertions.len(), 2); + + let Expr::StructPatch(inner) = add_patch.insertions[0].as_ref() else { + panic!("Expected nested struct patch"); + }; + assert_eq!( + inner.input_path.as_ref().map(|p| p.to_string()).as_deref(), + Some("add") + ); + + let gone = inner.field_patches.get("gone").unwrap(); + assert!(!gone.keep_input); + assert!(gone.insertions.is_empty()); + + let stub = inner.field_patches.get("stub").unwrap(); + assert!(!stub.keep_input); + assert_eq!(stub.insertions, vec![Arc::new(lit("replaced"))]); + + let x = inner.field_patches.get("x").unwrap(); + assert!(x.keep_input); + assert_eq!(x.insertions, vec![Arc::new(lit(true))]); + + assert_eq!(add_patch.insertions[1], Arc::new(lit("after_add"))); + } + + #[test] + fn struct_patch_builder_allows_empty_root_patches() { + let patch = ExpressionStructPatchBuilder::new().build().unwrap(); + assert!(patch.is_empty()); + assert!(patch.input_path().is_none()); + + let nested_patch = ExpressionStructPatchBuilder::new_nested(["nested"]) + .build() + .unwrap(); + assert!(nested_patch.is_empty()); + assert_eq!( + nested_patch + .input_path() + .map(ToString::to_string) + .as_deref(), + Some("nested") + ); + } + + #[rstest] + #[case::drop_then_replace( + ExpressionStructPatchBuilder::new().drop("a").replace("a", lit(1)), + "multiple input field actions")] + #[case::replace_then_drop( + ExpressionStructPatchBuilder::new().replace("a", lit(1)).drop("a"), + "multiple input field actions")] + #[case::drop_with_nested_insert( + ExpressionStructPatchBuilder::new().drop("add").insert_after_at(["add"], "x", lit(true)), + "nested fields")] + #[case::nested_replace_then_drop( + ExpressionStructPatchBuilder::new().replace_at(["add"], "x", lit("one")).drop_at(["add"], "x"), + "multiple input field actions")] + fn expression_build_rejects_conflicting_field_actions( + #[case] builder: ExpressionStructPatchBuilder, + #[case] expected_msg: &str, + ) { + assert_result_error_with_message(builder.build(), expected_msg); + } + + // === Schema patch tests === + + fn field(name: impl Into) -> StructField { + StructField::nullable(name, DataType::INTEGER) + } + + fn schema(names: &[&str]) -> StructType { + StructType::new_unchecked(names.iter().map(|name| field(*name)).collect::>()) + } + + fn field_names(schema: &StructType) -> Vec { + schema + .fields() + .map(|field| field.name().to_string()) + .collect() + } + + // Top-level schema with a "nested" struct field (holding `nested_a`, `nested_b`) and a + // sibling top-level field, used to exercise nested-path patching. + fn nested_input_schema() -> StructType { + StructType::new_unchecked(vec![ + StructField::nullable("nested", schema(&["nested_a", "nested_b"])), + field("top"), + ]) + } + + // Patches that leave the input schema fully unchanged (same fields, types, and order). + #[rstest] + #[case::empty_patch(SchemaStructPatchBuilder::new())] + #[case::optional_missing_drop(SchemaStructPatchBuilder::new().drop_if_exists("missing"))] + fn schema_build_preserves_input_schema(#[case] builder: SchemaStructPatchBuilder) { + let input_schema = schema(&["a", "b"]); + assert_eq!(builder.build(&input_schema).unwrap(), input_schema); + } + + // Patches that reorder/insert/replace/drop fields, asserted by the resulting field order. + #[rstest] + #[case::empty_nested_path_targets_top_level( + schema(&["a", "b"]), + SchemaStructPatchBuilder::new_nested(Vec::::new()).insert_after("a", field("after_a")), + &["a", "after_a", "b"])] + #[case::inserts_before_and_after( + schema(&["a", "b"]), + SchemaStructPatchBuilder::new().prepend(field("prepended")).insert_after("a", field("after_a")), + &["prepended", "a", "after_a", "b"])] + #[case::appends_after_all_input_fields( + schema(&["a", "b"]), + SchemaStructPatchBuilder::new().append(field("appended_1")).append(field("appended_2")), + &["a", "b", "appended_1", "appended_2"])] + #[case::appends_to_empty_input( + StructType::new_unchecked(Vec::::new()), + SchemaStructPatchBuilder::new().append(field("only")), + &["only"])] + #[case::replaces_field_at_input_position( + schema(&["a", "b", "c"]), + SchemaStructPatchBuilder::new().replace("b", field("bb")), + &["a", "bb", "c"])] + #[case::drops_field( + schema(&["a", "b", "c"]), + SchemaStructPatchBuilder::new().drop("b"), + &["a", "c"])] + #[case::preserves_patch_ordering( + schema(&["a", "b", "c"]), + SchemaStructPatchBuilder::new() + .prepend(field("prepended")) + .insert_after("a", field("after_a")) + .replace("b", field("bb")) + .drop("c") + .append(field("appended")), + &["prepended", "a", "after_a", "bb", "appended"])] + fn schema_build_produces_expected_field_order( + #[case] input_schema: StructType, + #[case] builder: SchemaStructPatchBuilder, + #[case] expected_names: &[&str], + ) { + let output_schema = builder.build(&input_schema).unwrap(); + assert_eq!(field_names(&output_schema), expected_names); + } + + #[test] + fn schema_build_nested_path_targets_nested_struct_schema() { + let input_schema = nested_input_schema(); + let output_schema = SchemaStructPatchBuilder::new_nested(["nested"]) + .insert_after("nested_a", field("nested_inserted")) + .build(&input_schema) + .unwrap(); + + assert_eq!( + field_names(&output_schema), + ["nested_a", "nested_inserted", "nested_b"] + ); + } + + #[test] + fn schema_build_nested_field_patch_patches_in_place() { + let input_schema = nested_input_schema(); + let output_schema = SchemaStructPatchBuilder::new() + .insert_after_at(["nested"], "nested_a", field("nested_inserted")) + .build(&input_schema) + .unwrap(); + + assert_eq!(field_names(&output_schema), ["nested", "top"]); + let DataType::Struct(nested) = output_schema.field("nested").unwrap().data_type() else { + panic!("Expected nested struct field"); + }; + assert_eq!( + field_names(nested), + ["nested_a", "nested_inserted", "nested_b"] + ); + } + + #[rstest] + #[case::required_missing(SchemaStructPatchBuilder::new().drop("missing"))] + #[case::required_missing_with_optional_match( + SchemaStructPatchBuilder::new().drop_if_exists("a").drop("missing"))] + fn schema_build_required_missing_field_errors(#[case] builder: SchemaStructPatchBuilder) { + let result = builder.build(&schema(&["a", "b"])); + assert_result_error_with_message(result, "Field to patch does not exist: missing"); + } + + // === Sparse projection patch tests === + + #[test] + fn projection_sparse_build_produces_schema_and_sparse_patch() { + let input_schema = schema(&["a", "b", "c", "untouched"]); + let (output_schema, patch) = ProjectionStructPatchBuilder::new(&input_schema) + .prepend(field("prepended"), lit(0)) + .insert_after("a", field("after_a"), lit(1)) + .replace("b", field("bb"), lit(2)) + .drop("c") + .append(field("appended"), lit(3)) + .build() + .unwrap(); + let patch = projection_patch(&patch); + + assert_eq!( + field_names(&output_schema), + ["prepended", "a", "after_a", "bb", "untouched", "appended"] + ); + assert_eq!(patch.prepended_fields, vec![Arc::new(lit(0))]); + assert_eq!(patch.appended_fields, vec![Arc::new(lit(3))]); + assert!(patch.input_path().is_none()); + assert!(!patch.field_patches.contains_key("untouched")); + + let a = patch.field_patches.get("a").unwrap(); + assert!(a.keep_input); + assert_eq!(a.insertions, vec![Arc::new(lit(1))]); + + let b = patch.field_patches.get("b").unwrap(); + assert!(!b.keep_input); + assert_eq!(b.insertions, vec![Arc::new(lit(2))]); + + let c = patch.field_patches.get("c").unwrap(); + assert!(!c.keep_input); + assert!(c.insertions.is_empty()); + } + + #[test] + fn projection_sparse_build_lowers_nested_patch_to_sparse_struct_patch() { + let input_schema = nested_input_schema(); + let (output_schema, patch) = ProjectionStructPatchBuilder::new(&input_schema) + .insert_after_at(["nested"], "nested_a", field("nested_inserted"), lit(9)) + .build() + .unwrap(); + let patch = projection_patch(&patch); + + assert_eq!(field_names(&output_schema), ["nested", "top"]); + let DataType::Struct(nested_schema) = output_schema.field("nested").unwrap().data_type() + else { + panic!("Expected nested struct field"); + }; + assert_eq!( + field_names(nested_schema), + ["nested_a", "nested_inserted", "nested_b"] + ); + assert!(!patch.field_patches.contains_key("top")); + + let nested_patch = patch.field_patches.get("nested").unwrap(); + assert!(!nested_patch.keep_input); + assert_eq!(nested_patch.insertions.len(), 1); + let Expr::StructPatch(inner) = nested_patch.insertions[0].as_ref() else { + panic!("Expected nested struct patch"); + }; + assert_eq!( + inner.input_path().map(ToString::to_string).as_deref(), + Some("nested") + ); + assert!(!inner.field_patches.contains_key("nested_b")); + + let nested_a = inner.field_patches.get("nested_a").unwrap(); + assert!(nested_a.keep_input); + assert_eq!(nested_a.insertions, vec![Arc::new(lit(9))]); + } + + #[test] + fn projection_sparse_build_replace_expr_at_preserves_nested_field() { + let input_schema = nested_input_schema(); + let (output_schema, patch) = ProjectionStructPatchBuilder::new(&input_schema) + .replace_expr_at(["nested"], "nested_b", lit(9)) + .build() + .unwrap(); + let patch = projection_patch(&patch); + + assert_eq!(field_names(&output_schema), ["nested", "top"]); + let DataType::Struct(output_nested_schema) = + output_schema.field("nested").unwrap().data_type() + else { + panic!("Expected nested struct field"); + }; + let DataType::Struct(input_nested_schema) = + input_schema.field("nested").unwrap().data_type() + else { + panic!("Expected nested struct field"); + }; + assert_eq!( + output_nested_schema.field("nested_b"), + input_nested_schema.field("nested_b") + ); + + let nested_patch = patch.field_patches.get("nested").unwrap(); + let Expr::StructPatch(inner) = nested_patch.insertions[0].as_ref() else { + panic!("Expected nested struct patch"); + }; + let nested_b = inner.field_patches.get("nested_b").unwrap(); + assert!(!nested_b.keep_input); + assert_eq!(nested_b.insertions, vec![Arc::new(lit(9))]); + } + + #[test] + fn projection_sparse_build_replace_expr_uses_nested_builder_input_path() { + let input_schema = nested_input_schema(); + let (output_schema, patch) = + ProjectionStructPatchBuilder::new_nested(&input_schema, ["nested"]) + .replace_expr("nested_b", lit(9)) + .build() + .unwrap(); + let patch = projection_patch(&patch); + + assert_eq!(field_names(&output_schema), ["nested_a", "nested_b"]); + assert_eq!( + patch.input_path().map(ToString::to_string).as_deref(), + Some("nested") + ); + + let nested_b = patch.field_patches.get("nested_b").unwrap(); + assert!(!nested_b.keep_input); + assert_eq!(nested_b.insertions, vec![Arc::new(lit(9))]); + } + + #[test] + fn projection_sparse_build_required_missing_field_errors() { + let input_schema = schema(&["a", "b"]); + let result = ProjectionStructPatchBuilder::new(&input_schema) + .drop("missing") + .build(); + assert_result_error_with_message(result, "Field to patch does not exist: missing"); + } +} diff --git a/kernel/src/table_changes/log_replay.rs b/kernel/src/table_changes/log_replay.rs index c71bce24e5..187fdd8447 100644 --- a/kernel/src/table_changes/log_replay.rs +++ b/kernel/src/table_changes/log_replay.rs @@ -5,30 +5,27 @@ use std::collections::{HashMap, HashSet}; use std::slice; use std::sync::{Arc, LazyLock}; -use crate::actions::visitors::visit_deletion_vector_at; -use crate::actions::visitors::InCommitTimestampVisitor; +use itertools::Itertools; +use tracing::info; + +use crate::actions::visitors::{visit_deletion_vector_at, InCommitTimestampVisitor}; use crate::actions::{ - get_log_add_schema, Add, Cdc, Metadata, Protocol, Remove, ADD_NAME, CDC_NAME, COMMIT_INFO_NAME, - METADATA_NAME, PROTOCOL_NAME, REMOVE_NAME, + Metadata, Protocol, ADD_FIELD, CDC_FIELD, COMMIT_INFO_NAME, LOG_ADD_SCHEMA, METADATA_FIELD, + PROTOCOL_FIELD, REMOVE_FIELD, }; use crate::engine_data::{GetData, TypedGetData}; -use crate::expressions::{column_expr, column_expr_ref, column_name, ColumnName, Expression}; +use crate::expressions::{column_name, ColumnName}; use crate::path::{AsUrl, ParsedLogPath}; -use crate::scan::data_skipping::stats_schema::build_stats_schema; use crate::scan::data_skipping::DataSkippingFilter; use crate::scan::state::DvInfo; -use crate::schema::{ - ColumnNamesAndTypes, DataType, SchemaRef, StructField, StructType, ToSchema as _, -}; +use crate::schema::{schema_ref, ColumnNamesAndTypes, DataType, SchemaRef}; use crate::table_changes::scan_file::{cdf_scan_row_expression, cdf_scan_row_schema}; +use crate::table_changes::CdfMode; use crate::table_configuration::TableConfiguration; use crate::table_features::{format_features, Operation, TableFeature}; use crate::utils::require; use crate::{DeltaResult, Engine, EngineData, Error, PredicateRef, RowVisitor}; -use itertools::Itertools; -use tracing::info; - #[cfg(test)] mod tests; @@ -59,25 +56,40 @@ pub(crate) fn table_changes_action_iter( table_schema: SchemaRef, physical_predicate: Option<(PredicateRef, SchemaRef)>, ) -> DeltaResult>> { + // The data-reading (`execute`) path always uses change-data-file semantics. + table_changes_action_iter_with_mode( + engine, + start_table_configuration, + commit_files, + table_schema, + physical_predicate, + CdfMode::ChangeDataFeed, + ) +} + +/// Replays change-feed actions according to `mode`. +/// +/// [`CdfMode::ChangeDataFeed`] uses `AddCDCFile` actions because they contain changes recorded by +/// the writer. [`CdfMode::RowTracking`] ignores those actions and reconstructs changes from +/// row lineage in the data files referenced by `add` and `remove` actions. +pub(crate) fn table_changes_action_iter_with_mode( + engine: Arc, + start_table_configuration: &TableConfiguration, + commit_files: impl IntoIterator, + table_schema: SchemaRef, + physical_predicate: Option<(PredicateRef, SchemaRef)>, + mode: CdfMode, +) -> DeltaResult>> { + // Skip against the raw `{ add, remove, ... }` action batch: table_changes must resolve + // deletion vector pairs before filtering, so unlike the scan path it operates on raw + // batches with stats parsed from `add.stats` JSON. let filter = physical_predicate - .and_then(|(predicate, ref_schema)| { - let stats_schema = build_stats_schema(&ref_schema)?; - - // Parse JSON stats from the raw action batch's `add.stats` column. Unlike the scan - // path (which transforms first and reads pre-parsed stats), table_changes must - // resolve deletion vector pairs before filtering, so it operates on raw batches. - let stats_expr = Arc::new(Expression::parse_json( - column_expr!("add.stats"), - stats_schema.clone(), - )); - DataSkippingFilter::new( + .and_then(|(predicate, _ref_schema)| { + DataSkippingFilter::for_raw_action_batch( engine.as_ref(), - Some(predicate), - Some(&stats_schema), - stats_expr, - None, // no partition columns for table changes (partition_expr unused) - column_expr_ref!("partitionValues_parsed"), - get_log_add_schema().clone(), + predicate, + start_table_configuration, + LOG_ADD_SCHEMA.clone(), ) }) .map(Arc::new); @@ -91,6 +103,7 @@ pub(crate) fn table_changes_action_iter( &mut current_configuration, commit_file, &table_schema, + mode, )?; scanner.into_scan_batches(engine.clone(), filter.clone()) }) //Iterator-Result-Iterator-Result @@ -105,20 +118,22 @@ pub(crate) fn table_changes_action_iter( /// 1. Prepare phase [`LogReplayScanner::try_new`]: This iterates over every action in the commit. /// In this phase, we do the following: /// - Determine if there exist any `cdc` actions. We determine this in the first phase because -/// the selection vectors for actions are lazily constructed in phase 2. We must know ahead -/// of time whether to filter out add/remove actions. +/// the selection vectors for actions are lazily constructed in phase 2. We must know ahead of +/// time whether to filter out add/remove actions. In [`CdfMode::RowTracking`] mode `cdc` +/// actions are ignored entirely, so add/remove actions always drive the feed. /// - Constructs the remove deletion vector map from paths belonging to `remove` actions to the /// action's corresponding [`DvInfo`]. This map will be filtered to only contain paths that -/// exists in another `add` action _within the same commit_. We store the result in `remove_dvs`. -/// Deletion vector resolution affects whether a remove action is selected in the second -/// phase, so we must perform it ahead of time in phase 1. +/// exists in another `add` action _within the same commit_. We store the result in +/// `remove_dvs`. Deletion vector resolution affects whether a remove action is selected in +/// the second phase, so we must perform it ahead of time in phase 1. /// - Ensure that reading is supported on any protocol updates. -/// - Ensure that Change Data Feed is enabled for any metadata update. See [`TableProperties`] -/// - Ensure that any schema update is compatible with the provided `schema`. Currently, schema -/// compatibility is checked through schema equality. This will be expanded in the future to -/// allow limited schema evolution. +/// - Ensure that the mode's required table feature remains enabled on metadata updates. +/// - Ensure that schema updates satisfy the mode's compatibility policy. Change Data Feed mode +/// requires equality; row-tracking mode allows additive nullable columns and relaxed +/// nullability, but rejects datatype changes. +/// - Read the in-commit timestamp from `CommitInfo` when that feature is enabled. /// -/// Note: We check the protocol, change data feed enablement, and schema compatibility in phase 1 +/// Note: We check the protocol, mode-specific table feature, and schema compatibility in phase 1 /// in order to detect errors and fail early. /// /// Note: The reader feature [`ReaderFeatures::DeletionVectors`] controls whether the table is @@ -128,15 +143,10 @@ pub(crate) fn table_changes_action_iter( /// /// See https://github.com/delta-io/delta/blob/master/PROTOCOL.md#deletion-vectors /// -/// TODO: When the kernel supports in-commit timestamps, we will also have to inspect CommitInfo -/// actions to find the timestamp. These are generated when incommit timestamps is enabled. -/// This must be done in the first phase because the second phase lazily transforms engine data with -/// an extra timestamp column. Thus, the timestamp must be known ahead of time. -/// See https://github.com/delta-io/delta-kernel-rs/issues/559 -/// /// 2. Scan file generation phase [`LogReplayScanner::into_scan_batches`]: This iterates over every -/// action in the commit, and generates [`TableChangesScanMetadata`]. It does so by transforming the -/// actions using [`add_transform_expr`], and generating selection vectors with the following rules: +/// action in the commit, and generates [`TableChangesScanMetadata`]. It does so by transforming +/// the actions using [`add_transform_expr`], and generating selection vectors with the following +/// rules: /// - If a `cdc` action was found in the prepare phase, only `cdc` actions are selected /// - Otherwise, select `add` and `remove` actions. Note that only `remove` actions that do not /// share a path with an `add` action are selected. @@ -152,21 +162,14 @@ struct LogReplayScanner { remove_dvs: HashMap, // The commit file that this replay scanner will operate on. commit_file: ParsedLogPath, - // The timestamp associated with this commit. This is the file modification time - // from the commit's [`FileMeta`]. - // - // - // TODO when incommit timestamps are supported: If there is a [`CommitInfo`] with a timestamp - // generated by in-commit timestamps, that timestamp will be used instead. - // - // Note: This will be used once an expression is introduced to transform the engine data in - // [`TableChangesScanMetadata`] + // The in-commit timestamp when enabled, or the commit file modification time otherwise. timestamp: i64, } impl LogReplayScanner { - /// Constructs a LogReplayScanner, performing the Prepare phase detailed in [`LogReplayScanner`]. - /// This iterates over each action in the commit. It performs the following: + /// Constructs a LogReplayScanner, performing the Prepare phase detailed in + /// [`LogReplayScanner`]. This iterates over each action in the commit. It performs the + /// following: /// 1. Check the commits for the presence of a `cdc` action. /// 2. Construct a map from path to deletion vector of remove actions that share the same path /// as an add action. @@ -178,6 +181,7 @@ impl LogReplayScanner { table_configuration: &mut TableConfiguration, commit_file: ParsedLogPath, table_schema: &SchemaRef, + mode: CdfMode, ) -> DeltaResult { let visitor_schema = PreparePhaseVisitor::schema(); @@ -187,9 +191,9 @@ impl LogReplayScanner { // Consider a scenario with a pair of add/remove actions with the same path. The add // action has file statistics, while the remove action does not (stats is optional for // remove). In this scenario we might skip the add action, while the remove action remains. - // As a result, we would read the file path for the remove action, which is unnecessary because - // all of the rows will be filtered by the predicate. Instead, we wait until deletion - // vectors are resolved so that we can skip both actions in the pair. + // As a result, we would read the file path for the remove action, which is unnecessary + // because all of the rows will be filtered by the predicate. Instead, we wait until + // deletion vectors are resolved so that we can skip both actions in the pair. let mut action_iter = engine .json_handler() .read_json_files( @@ -217,6 +221,7 @@ impl LogReplayScanner { add_paths: &mut add_paths, remove_dvs: &mut remove_dvs, has_cdc_action: &mut has_cdc_action, + mode, }; visitor.visit_rows_of(actions.as_ref())?; @@ -227,12 +232,14 @@ impl LogReplayScanner { if let Some(ref metadata) = metadata_opt { let schema = metadata.parse_schema()?; - // Currently, schema compatibility is defined as having equal schema types. In the - // future, more permisive schema evolution will be supported. - // See: https://github.com/delta-io/delta-kernel-rs/issues/523 + // Compatibility is evaluated against the end version's logical schema. require!( - table_schema.as_ref() == &schema, - Error::change_data_feed_incompatible_schema(table_schema, &schema) + mode.schemas_compatible(&schema, table_schema.as_ref()), + Error::change_data_feed_incompatible_schema_at_version( + table_schema, + &schema, + commit_file.version + ) ); } @@ -264,19 +271,17 @@ impl LogReplayScanner { ); } - // If metadata is updated, check if Change Data Feed is enabled if has_metadata_update { require!( - table_configuration.is_feature_enabled(&TableFeature::ChangeDataFeed), - Error::change_data_feed_unsupported(commit_file.version) + table_configuration.is_feature_enabled(&mode.required_feature()), + mode.feature_disabled_error(commit_file.version) ); } - // If protocol is updated, check if Change Data Feed is supported if has_protocol_update { table_configuration .ensure_operation_supported(Operation::Cdf) - .map_err(|_| Error::change_data_feed_unsupported(commit_file.version))?; + .map_err(|e| mode.protocol_support_error(e, commit_file.version))?; } } // We resolve the remove deletion vector map after visiting the entire commit. @@ -288,7 +293,8 @@ impl LogReplayScanner { remove_dvs.retain(|rm_path, _| add_paths.contains(rm_path)); } - // If ICT is enabled, then set the timestamp to be the ICT; otherwise, default to the last_modified timestamp value + // If ICT is enabled, then set the timestamp to be the ICT; otherwise, default to the + // last_modified timestamp value let timestamp = if table_configuration.is_feature_enabled(&TableFeature::InCommitTimestamp) { let Some(in_commit_timestamp) = in_commit_timestamp_opt else { @@ -347,7 +353,7 @@ impl LogReplayScanner { .try_into() .map_err(|_| Error::generic("Failed to convert commit version to i64"))?; let evaluator = engine.evaluation_handler().new_expression_evaluator( - get_log_add_schema().clone(), + LOG_ADD_SCHEMA.clone(), Arc::new(cdf_scan_row_expression(timestamp, commit_version)), cdf_scan_row_schema().into(), )?; @@ -383,24 +389,20 @@ struct PreparePhaseVisitor<'a> { has_cdc_action: &'a mut bool, add_paths: &'a mut HashSet, remove_dvs: &'a mut HashMap, + mode: CdfMode, } impl PreparePhaseVisitor<'_> { - fn schema() -> Arc { - Arc::new(StructType::new_unchecked(vec![ - StructField::nullable(ADD_NAME, Add::to_schema()), - StructField::nullable(REMOVE_NAME, Remove::to_schema()), - StructField::nullable(CDC_NAME, Cdc::to_schema()), - StructField::nullable(METADATA_NAME, Metadata::to_schema()), - StructField::nullable(PROTOCOL_NAME, Protocol::to_schema()), - StructField::nullable( - COMMIT_INFO_NAME, - StructType::new_unchecked([StructField::new( - "inCommitTimestamp", - DataType::LONG, - true, - )]), - ), - ])) + fn schema() -> SchemaRef { + schema_ref! { + (&ADD_FIELD), + (&REMOVE_FIELD), + (&CDC_FIELD), + (&METADATA_FIELD), + (&PROTOCOL_FIELD), + nullable COMMIT_INFO_NAME: { + nullable "inCommitTimestamp": LONG, + }, + } } } @@ -452,7 +454,9 @@ impl RowVisitor for PreparePhaseVisitor<'_> { self.remove_dvs .insert(path.to_string(), DvInfo { deletion_vector }); } - } else if getters[9].get_str(i, "cdc.path")?.is_some() { + } else if getters[9].get_str(i, "cdc.path")?.is_some() + && self.mode.uses_change_data_files() + { *self.has_cdc_action = true; } } @@ -480,12 +484,12 @@ impl<'a> FileActionSelectionVisitor<'a> { remove_dvs, } } - fn schema() -> Arc { - Arc::new(StructType::new_unchecked(vec![ - StructField::nullable(CDC_NAME, Cdc::to_schema()), - StructField::nullable(ADD_NAME, Add::to_schema()), - StructField::nullable(REMOVE_NAME, Remove::to_schema()), - ])) + fn schema() -> SchemaRef { + schema_ref! { + (&CDC_FIELD), + (&ADD_FIELD), + (&REMOVE_FIELD), + } } } diff --git a/kernel/src/table_changes/log_replay/tests.rs b/kernel/src/table_changes/log_replay/tests.rs index 4b651fa8cc..b18cb38755 100644 --- a/kernel/src/table_changes/log_replay/tests.rs +++ b/kernel/src/table_changes/log_replay/tests.rs @@ -1,26 +1,32 @@ -use super::table_changes_action_iter; -use super::TableChangesScanMetadata; -use crate::actions::deletion_vector::{DeletionVectorDescriptor, DeletionVectorStorageType}; +use std::collections::HashMap; +use std::path::Path; +use std::sync::Arc; + +use itertools::Itertools; +use rstest::rstest; +use test_utils::LoggingTest; + +use super::{ + table_changes_action_iter, table_changes_action_iter_with_mode, TableChangesScanMetadata, +}; use crate::actions::{Add, Cdc, CommitInfo, Metadata, Protocol, Remove}; use crate::engine::sync::SyncEngine; -use crate::expressions::{column_expr, BinaryPredicateOp, Scalar}; +use crate::expressions::{col, lit, BinaryPredicateOp}; use crate::log_segment::LogSegment; use crate::path::ParsedLogPath; use crate::scan::state::DvInfo; use crate::scan::PhysicalPredicate; use crate::schema::{DataType, SchemaRef, StructField, StructType}; use crate::table_changes::log_replay::LogReplayScanner; +use crate::table_changes::test_utils::{ + row_tracking_metadata, row_tracking_table_config, test_deletion_vector, +}; +use crate::table_changes::CdfMode; use crate::table_configuration::TableConfiguration; use crate::table_features::{ColumnMappingMode, TableFeature}; -use crate::utils::test_utils::{assert_result_error_with_message, Action, LocalMockTable}; -use crate::Predicate; -use crate::{DeltaResult, Engine, Error, Version}; -use test_utils::LoggingTest; - -use itertools::Itertools; -use std::collections::HashMap; -use std::path::Path; -use std::sync::Arc; +use crate::table_properties::{ENABLE_ROW_TRACKING, ROW_TRACKING_SUSPENDED}; +use crate::unit_test_utils::{assert_result_error_with_message, Action, LocalMockTable}; +use crate::{DeltaResult, Engine, Error, Predicate, Version}; fn get_schema() -> SchemaRef { Arc::new(StructType::new_unchecked([ @@ -54,6 +60,31 @@ fn metadata_action(schema: SchemaRef, configuration: HashMap) -> ) } +/// Helper to create a Metadata action with row tracking enabled +fn metadata_with_row_tracking(schema: SchemaRef) -> Action { + Action::Metadata(row_tracking_metadata(schema)) +} + +/// Runs row-tracking log replay over all commits of `mock_table` against the given end schema. +fn execute_row_tracking( + engine: Arc, + mock_table: &LocalMockTable, + end_schema: SchemaRef, +) -> DeltaResult> { + let commits = get_segment(engine.as_ref(), mock_table.table_root(), 0, None)?.into_iter(); + let table_root_url = url::Url::from_directory_path(mock_table.table_root()).unwrap(); + let table_config = row_tracking_table_config(table_root_url, get_schema()); + table_changes_action_iter_with_mode( + engine, + &table_config, + commits, + end_schema, + None, + CdfMode::RowTracking, + )? + .try_collect() +} + /// Helper to create a Metadata action with CDF enabled fn metadata_with_cdf(schema: SchemaRef) -> Action { metadata_action( @@ -344,6 +375,129 @@ async fn cdf_disabled_midstream() { assert_midstream_failure(engine, &mock_table); } +#[rstest] +#[case::disabled(&[(ENABLE_ROW_TRACKING, "false")])] +#[case::suspended(&[ + (ENABLE_ROW_TRACKING, "true"), + (ROW_TRACKING_SUSPENDED, "true"), +])] +#[tokio::test] +async fn row_tracking_unavailable_midstream_fails(#[case] properties: &[(&str, &str)]) { + let engine = Arc::new(SyncEngine::new()); + let mut mock_table = LocalMockTable::new(); + + mock_table + .commit([metadata_with_row_tracking(get_schema())]) + .await; + mock_table + .commit([metadata_action( + get_schema(), + properties + .iter() + .map(|(key, value)| (key.to_string(), value.to_string())) + .collect(), + )]) + .await; + + let res = execute_row_tracking(engine, &mock_table, get_schema()).map(|_| ()); + assert!( + matches!(&res, Err(Error::RowTrackingChangeFeedUnsupported(1))), + "expected row tracking to be unavailable at version 1, got {res:?}" + ); +} + +fn nested_id_type(value_type: DataType) -> DataType { + DataType::from(StructType::new_unchecked([StructField::nullable( + "value", value_type, + )])) +} + +#[rstest] +#[case::additive(DataType::INTEGER, DataType::INTEGER, false, true, true)] +#[case::type_widening(DataType::INTEGER, DataType::LONG, false, false, false)] +#[case::nested_type_widening( + nested_id_type(DataType::INTEGER), + nested_id_type(DataType::LONG), + false, + false, + false +)] +#[case::removed_column(DataType::INTEGER, DataType::INTEGER, true, false, false)] +#[tokio::test] +async fn row_tracking_schema_compatibility( + #[case] commit_id_type: DataType, + #[case] end_id_type: DataType, + #[case] commit_has_year: bool, + #[case] end_has_year: bool, + #[case] expect_compatible: bool, +) { + let schema = |id_type: DataType, has_year: bool| { + let fields = [ + StructField::nullable("id", id_type), + StructField::nullable("value", DataType::STRING), + ] + .into_iter() + .chain(has_year.then(|| StructField::nullable("year", DataType::INTEGER))); + Arc::new(StructType::new_unchecked(fields)) + }; + let engine = Arc::new(SyncEngine::new()); + let mut mock_table = LocalMockTable::new(); + mock_table + .commit([metadata_with_row_tracking(schema( + commit_id_type, + commit_has_year, + ))]) + .await; + let res = + execute_row_tracking(engine, &mock_table, schema(end_id_type, end_has_year)).map(|_| ()); + if expect_compatible { + assert!( + res.is_ok(), + "expected compatible schema to succeed, got {res:?}" + ); + } else { + assert!( + matches!(&res, Err(Error::ChangeDataFeedIncompatibleSchema(_, _))), + "expected incompatible-schema error, got {res:?}" + ); + } +} + +#[rstest] +#[case::widen_nullability(false, true, true)] +#[case::tighten_nullability(true, false, false)] +fn row_tracking_schema_compatibility_checks_nullability( + #[case] candidate_nullable: bool, + #[case] read_nullable: bool, + #[case] expected: bool, +) { + let schema = + |nullable| StructType::new_unchecked([StructField::new("id", DataType::INTEGER, nullable)]); + assert_eq!( + CdfMode::RowTracking + .schemas_compatible(&schema(candidate_nullable), &schema(read_nullable),), + expected + ); +} + +#[rstest] +#[case::nullable(true, true)] +#[case::non_nullable(false, false)] +fn row_tracking_schema_compatibility_requires_new_columns_to_be_nullable( + #[case] new_column_nullable: bool, + #[case] expected: bool, +) { + let candidate = StructType::new_unchecked([StructField::nullable("id", DataType::INTEGER)]); + let read_schema = StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::new("new_column", DataType::STRING, new_column_nullable), + ]); + assert_eq!( + CdfMode::RowTracking.schemas_compatible(&candidate, &read_schema), + expected + ); +} + // Test that unsupported protocol features added mid-stream are rejected #[tokio::test] async fn unsupported_protocol_feature_midstream() { @@ -374,6 +528,32 @@ async fn unsupported_protocol_feature_midstream() { assert_midstream_failure(engine, &mock_table); } +#[tokio::test] +async fn row_tracking_protocol_failure_preserves_the_underlying_error() { + let engine = Arc::new(SyncEngine::new()); + let mut mock_table = LocalMockTable::new(); + let unknown = TableFeature::unknown("unsupportedFeature"); + mock_table + .commit([Action::Protocol( + Protocol::try_new_modern( + [unknown.clone()], + [ + unknown, + TableFeature::RowTracking, + TableFeature::DomainMetadata, + ], + ) + .unwrap(), + )]) + .await; + + let result = execute_row_tracking(engine, &mock_table, get_schema()).map(|_| ()); + assert!( + matches!(&result, Err(Error::Unsupported(_))), + "expected the protocol support error, got {result:?}" + ); +} + #[tokio::test] async fn incompatible_schemas_fail() { async fn assert_incompatible_schema(commit_schema: SchemaRef, cdf_schema: SchemaRef) { @@ -715,20 +895,8 @@ async fn dv() { let engine = Arc::new(SyncEngine::new()); let mut mock_table = LocalMockTable::new(); - let deletion_vector1 = DeletionVectorDescriptor { - storage_type: DeletionVectorStorageType::PersistedRelative, - path_or_inline_dv: "vBn[lx{q8@P<9BNH/isA".to_string(), - offset: Some(1), - size_in_bytes: 36, - cardinality: 2, - }; - let deletion_vector2 = DeletionVectorDescriptor { - storage_type: DeletionVectorStorageType::PersistedRelative, - path_or_inline_dv: "U5OWRz5k%CFT.Td}yCPW".to_string(), - offset: Some(1), - size_in_bytes: 38, - cardinality: 3, - }; + let deletion_vector1 = test_deletion_vector("vBn[lx{q8@P<9BNH/isA", 2); + let deletion_vector2 = test_deletion_vector("U5OWRz5k%CFT.Td}yCPW", 3); // - fake_path_1 undergoes a restore. All rows are restored, so the deletion vector is removed. // - All remaining rows of fake_path_2 are deleted mock_table @@ -783,13 +951,7 @@ async fn dv() { async fn data_skipping_filter() { let engine = Arc::new(SyncEngine::new()); let mut mock_table = LocalMockTable::new(); - let deletion_vector = Some(DeletionVectorDescriptor { - storage_type: DeletionVectorStorageType::PersistedRelative, - path_or_inline_dv: "vBn[lx{q8@P<9BNH/isA".to_string(), - offset: Some(1), - size_in_bytes: 36, - cardinality: 2, - }); + let deletion_vector = Some(test_deletion_vector("vBn[lx{q8@P<9BNH/isA", 2)); mock_table .commit([ // Remove/Add pair with max value id = 6 @@ -829,11 +991,7 @@ async fn data_skipping_filter() { .await; // Look for actions with id > 4 - let predicate = Predicate::binary( - BinaryPredicateOp::GreaterThan, - column_expr!("id"), - Scalar::from(4), - ); + let predicate = Predicate::binary(BinaryPredicateOp::GreaterThan, col!("id"), lit(4)); let logical_schema = get_schema(); let predicate = match PhysicalPredicate::try_new(&predicate, &logical_schema, ColumnMappingMode::None) { @@ -858,6 +1016,135 @@ async fn data_skipping_filter() { assert_eq!(sv, &[false, true, false, false, true]); } +// The shared `for_raw_action_batch` filter prunes on partition values on the table_changes path +// too: partition values are parsed from `add.partitionValues` via `map_to_struct`, so `part = 'x'` +// drops the Add in partition `y`. The Remove in partition `y` must survive: the `OR(NOT is_add, +// ...)` guard shields non-Add rows from the predicate, so tombstones are never dropped from the +// change feed even when their partition does not match. +#[tokio::test] +async fn data_skipping_filter_prunes_partition_values_but_keeps_removes() { + let engine = Arc::new(SyncEngine::new()); + let mut mock_table = LocalMockTable::new(); + mock_table + .commit([ + Action::Add(Add { + path: "in_x".into(), + partition_values: HashMap::from([("part".to_string(), "x".to_string())]), + data_change: true, + ..Default::default() + }), + Action::Add(Add { + path: "in_y".into(), + partition_values: HashMap::from([("part".to_string(), "y".to_string())]), + data_change: true, + ..Default::default() + }), + // A tombstone in the pruned partition `y`. If the guard broke, `part = 'x'` would + // silently drop it from the change feed. + Action::Remove(Remove { + path: "gone_y".into(), + partition_values: Some(HashMap::from([("part".to_string(), "y".to_string())])), + data_change: true, + ..Default::default() + }), + ]) + .await; + + // Partitioned schema: `part` is the partition column, `id` a data column. + let logical_schema: SchemaRef = Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("part", DataType::STRING), + ])); + let metadata = Metadata::try_new( + None, + None, + logical_schema.clone(), + vec!["part".to_string()], + 0, + HashMap::from([("delta.enableChangeDataFeed".to_string(), "true".to_string())]), + ) + .unwrap(); + let protocol = Protocol::try_new_legacy(1, 4).unwrap(); + let table_root_url = url::Url::from_directory_path(mock_table.table_root()).unwrap(); + let table_config = TableConfiguration::try_new(metadata, protocol, table_root_url, 0).unwrap(); + + let predicate = Predicate::binary(BinaryPredicateOp::Equal, col!("part"), lit("x")); + let predicate = + match PhysicalPredicate::try_new(&predicate, &logical_schema, ColumnMappingMode::None) { + Ok(PhysicalPredicate::Some(p, s)) => Some((p, s)), + other => panic!("Unexpected result: {other:?}"), + }; + let commits = get_segment(engine.as_ref(), mock_table.table_root(), 0, None) + .unwrap() + .into_iter(); + + let sv = table_changes_action_iter(engine, &table_config, commits, logical_schema, predicate) + .unwrap() + .flat_map(|scan_metadata| scan_metadata.unwrap().selection_vector) + .collect_vec(); + + // Add in `x` survives, Add in `y` is pruned, and the Remove in `y` survives via the guard. + assert_eq!(sv, &[true, false, true]); +} + +// Stats-based pruning (as opposed to partition-value pruning) with a Remove present: `id > 4` +// drops the out-of-range Add via its `add.stats`, keeps the in-range Add, and the standalone +// Remove survives regardless of the predicate because non-Add rows bypass the stats filter. +#[tokio::test] +async fn data_skipping_filter_prunes_stats_but_keeps_removes() { + let engine = Arc::new(SyncEngine::new()); + let mut mock_table = LocalMockTable::new(); + mock_table + .commit([ + // Standalone Remove (no matching Add, no DV): survives as a tombstone. + Action::Remove(Remove { + path: "gone".into(), + data_change: true, + ..Default::default() + }), + // id in [0, 2]: provably excluded by `id > 4`. + Action::Add(Add { + path: "out_of_range".into(), + stats: Some( + "{\"numRecords\":4,\"minValues\":{\"id\":0},\"maxValues\":{\"id\":2},\"nullCount\":{\"id\":0}}".into(), + ), + data_change: true, + ..Default::default() + }), + // id in [4, 6]: overlaps `id > 4`, so kept. + Action::Add(Add { + path: "in_range".into(), + stats: Some( + "{\"numRecords\":4,\"minValues\":{\"id\":4},\"maxValues\":{\"id\":6},\"nullCount\":{\"id\":0}}".into(), + ), + data_change: true, + ..Default::default() + }), + ]) + .await; + + let logical_schema = get_schema(); + let predicate = Predicate::binary(BinaryPredicateOp::GreaterThan, col!("id"), lit(4)); + let predicate = + match PhysicalPredicate::try_new(&predicate, &logical_schema, ColumnMappingMode::None) { + Ok(PhysicalPredicate::Some(p, s)) => Some((p, s)), + other => panic!("Unexpected result: {other:?}"), + }; + let commits = get_segment(engine.as_ref(), mock_table.table_root(), 0, None) + .unwrap() + .into_iter(); + + let table_root_url = url::Url::from_directory_path(mock_table.table_root()).unwrap(); + let table_config = get_default_table_config(&table_root_url); + let sv = table_changes_action_iter(engine, &table_config, commits, logical_schema, predicate) + .unwrap() + .flat_map(|scan_metadata| scan_metadata.unwrap().selection_vector) + .collect_vec(); + + // Remove survives, out-of-range Add is pruned, in-range Add survives. + assert_eq!(sv, &[true, false, true]); +} + #[tokio::test] async fn failing_protocol() { let engine = Arc::new(SyncEngine::new()); @@ -919,9 +1206,14 @@ async fn file_meta_timestamp() { let file_meta_ts = commit.location.last_modified; let table_root_url = url::Url::from_directory_path(mock_table.table_root()).unwrap(); let mut table_config = get_default_table_config(&table_root_url); - let scanner = - LogReplayScanner::try_new(engine.as_ref(), &mut table_config, commit, &get_schema()) - .unwrap(); + let scanner = LogReplayScanner::try_new( + engine.as_ref(), + &mut table_config, + commit, + &get_schema(), + CdfMode::ChangeDataFeed, + ) + .unwrap(); assert_eq!(scanner.timestamp, file_meta_ts); } @@ -994,7 +1286,8 @@ async fn print_table_info_post_phase1() { let engine = Arc::new(SyncEngine::new()); let mut mock_table = LocalMockTable::new(); - // This specific commit (with these actions) isn't necessary to test the tracing for this test, we just need to have one commit with any actions + // This specific commit (with these actions) isn't necessary to test the tracing for this test, + // we just need to have one commit with any actions mock_table .commit([ Action::Metadata( @@ -1099,20 +1392,8 @@ async fn print_table_info_post_phase1_has_dv() { let engine = Arc::new(SyncEngine::new()); let mut mock_table = LocalMockTable::new(); - let deletion_vector1 = DeletionVectorDescriptor { - storage_type: DeletionVectorStorageType::PersistedRelative, - path_or_inline_dv: "vBn[lx{q8@P<9BNH/isA".to_string(), - offset: Some(1), - size_in_bytes: 36, - cardinality: 2, - }; - let deletion_vector2 = DeletionVectorDescriptor { - storage_type: DeletionVectorStorageType::PersistedRelative, - path_or_inline_dv: "U5OWRz5k%CFT.Td}yCPW".to_string(), - offset: Some(1), - size_in_bytes: 38, - cardinality: 3, - }; + let deletion_vector1 = test_deletion_vector("vBn[lx{q8@P<9BNH/isA", 2); + let deletion_vector2 = test_deletion_vector("U5OWRz5k%CFT.Td}yCPW", 3); // - fake_path_1 undergoes a restore. All rows are restored, so the deletion vector is removed. // - All remaining rows of fake_path_2 are deleted mock_table @@ -1213,9 +1494,14 @@ async fn test_timestamp_with_ict_enabled() { let commit = commits.next().unwrap(); let table_root_url = url::Url::from_directory_path(mock_table.table_root()).unwrap(); let mut table_config = get_default_table_config(&table_root_url); - let scanner = - LogReplayScanner::try_new(engine.as_ref(), &mut table_config, commit, &get_schema()) - .unwrap(); + let scanner = LogReplayScanner::try_new( + engine.as_ref(), + &mut table_config, + commit, + &get_schema(), + CdfMode::ChangeDataFeed, + ) + .unwrap(); assert_eq!(scanner.timestamp, 2000); } @@ -1264,6 +1550,7 @@ async fn test_timestamp_with_ict_disabled() { &mut table_config, commit.clone(), &get_schema(), + CdfMode::ChangeDataFeed, ) .unwrap(); assert_ne!(scanner.timestamp, 2000); @@ -1316,8 +1603,13 @@ async fn test_timestamp_with_commit_info_not_first() { let commit = commits.next().unwrap(); let table_root_url = url::Url::from_directory_path(mock_table.table_root()).unwrap(); let mut table_config = get_default_table_config(&table_root_url); - let result = - LogReplayScanner::try_new(engine.as_ref(), &mut table_config, commit, &get_schema()); + let result = LogReplayScanner::try_new( + engine.as_ref(), + &mut table_config, + commit, + &get_schema(), + CdfMode::ChangeDataFeed, + ); // Should error because ICT is enabled but not found in the first action assert_result_error_with_message( diff --git a/kernel/src/table_changes/mod.rs b/kernel/src/table_changes/mod.rs index 68b2d12ea2..5fe40a6ffd 100644 --- a/kernel/src/table_changes/mod.rs +++ b/kernel/src/table_changes/mod.rs @@ -1,15 +1,20 @@ -//! Provides an API to read the table's change data feed between two versions. +//! Reads a table's change data feed between two versions. +//! +//! [`TableChanges::try_new`] reads change data recorded by writers and produces data through a +//! [`TableChangesScanBuilder`]. When the `internal-api` feature is enabled, +//! [`TableChanges::try_new_row_tracking_cdf_listing`] instead lists the files and row-tracking +//! metadata that a connector needs to reconstruct changes while reading the data files itself. //! //! # Example //! ```rust //! # use std::sync::Arc; -//! # use delta_kernel::engine::default::{DefaultEngine, DefaultEngineBuilder}; -//! # use delta_kernel::expressions::{column_expr, Scalar}; +//! # use test_utils::delta_kernel_default_engine::{DefaultEngine, DefaultEngineBuilder}; +//! # use delta_kernel::expressions::{col, lit}; //! # use delta_kernel::{Predicate, Snapshot, SnapshotRef, Error, Engine}; //! # use delta_kernel::table_changes::TableChanges; //! # let path = "./tests/data/table-with-cdf"; //! let url = delta_kernel::try_parse_uri(path)?; -//! # use delta_kernel::engine::default::storage::store_from_url; +//! # use test_utils::delta_kernel_default_engine::storage::store_from_url; //! # let engine = std::sync::Arc::new(DefaultEngineBuilder::new(store_from_url(&url)?).build()); //! // Get the table changes (change data feed) between version 0 and 1 //! let table_changes = TableChanges::try_new(url, engine.as_ref(), 0, Some(1))?; @@ -18,7 +23,7 @@ //! let schema = table_changes //! .schema() //! .project(&["id", "_commit_version"])?; -//! let predicate = Arc::new(Predicate::gt(column_expr!("id"), Scalar::from(10))); +//! let predicate = Arc::new(Predicate::gt(col!("id"), lit(10))); //! //! // Construct the table changes scan //! let table_changes_scan = table_changes @@ -33,23 +38,128 @@ //! ``` use std::sync::{Arc, LazyLock}; +use delta_kernel_derive::internal_api; +use log_replay::table_changes_action_iter_with_mode; use scan::TableChangesScanBuilder; +use scan_file::scan_metadata_to_scan_file; use url::Url; use crate::log_segment::LogSegment; use crate::path::AsUrl; +use crate::schema::compare::SchemaComparison as _; use crate::schema::{DataType, Schema, StructField, StructType}; use crate::snapshot::{Snapshot, SnapshotRef}; use crate::table_configuration::TableConfiguration; -use crate::table_features::Operation; -use crate::table_features::TableFeature; +use crate::table_features::{Operation, TableFeature}; +use crate::table_properties::{ + MATERIALIZED_ROW_COMMIT_VERSION_COLUMN_NAME, MATERIALIZED_ROW_ID_COLUMN_NAME, +}; +use crate::utils::require; use crate::{DeltaResult, Engine, Error, Version}; mod log_replay; +mod net_changes; mod physical_to_logical; mod resolve_dvs; pub mod scan; mod scan_file; +#[cfg(test)] +mod test_utils; + +#[internal_api] +pub(crate) use scan_file::{TableChangesFileAction, TableChangesScanFile}; + +/// Selects the history represented by [`TableChanges::scan_file_listing`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +#[internal_api] +pub(crate) enum TableChangesListingMode { + /// Preserves each commit's file actions. + /// + /// An `add` and `remove` for the same path in one commit are grouped. Actions for the same + /// path in different commits remain separate and require reconciliation by stable row ID. + AllChanges, + /// Reduces each path to its net effect between the start and end of the range. + /// + /// The remove side represents the earliest pre-image and the add side represents the latest + /// post-image. Intermediate actions are omitted, but each surviving action still requires + /// row-level reconciliation. Computing net changes buffers the full range. + NetChanges, +} + +/// Identifies the table feature that must remain enabled across the change-feed range. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum CdfMode { + /// Uses change data recorded by writers. + /// + /// This mode requires `delta.enableChangeDataFeed`. It reads `_change_data` files when a + /// commit contains them and otherwise derives changes from `add` and `remove` actions. The + /// data-reading [`scan`] and `execute` paths use these semantics. + ChangeDataFeed, + /// Uses row lineage to reconstruct changes while reading data files. + /// + /// This mode requires `delta.enableRowTracking`. It ignores `_change_data` files and returns + /// metadata from `add` and `remove` actions for [`TableChanges::scan_file_listing`]. + RowTracking, +} + +impl CdfMode { + /// The table feature that must be enabled across the entire change-feed range for this mode. + pub(crate) fn required_feature(self) -> TableFeature { + match self { + CdfMode::ChangeDataFeed => TableFeature::ChangeDataFeed, + CdfMode::RowTracking => TableFeature::RowTracking, + } + } + + /// Returns the mode-specific error when [`CdfMode::required_feature`] is disabled at `version`. + pub(crate) fn feature_disabled_error(self, version: Version) -> Error { + match self { + CdfMode::ChangeDataFeed => Error::change_data_feed_unsupported(version), + CdfMode::RowTracking => Error::row_tracking_change_feed_unsupported(version), + } + } + + /// Whether data written with `candidate` can be read using the change feed's `read_schema`. + /// + /// [`CdfMode::ChangeDataFeed`] requires exact equality. Row-tracking CDF accepts additive + /// nullable fields and wider nullability, but not datatype changes. + pub(crate) fn schemas_compatible( + self, + candidate: &StructType, + read_schema: &StructType, + ) -> bool { + match self { + CdfMode::ChangeDataFeed => candidate == read_schema, + CdfMode::RowTracking => candidate + .can_read_as_without_type_widening(read_schema) + .is_ok(), + } + } + + /// Whether `AddCDCFile` actions supersede `add` and `remove` actions. + pub(crate) fn uses_change_data_files(self) -> bool { + self == CdfMode::ChangeDataFeed + } + + /// Returns the mode-specific error for incompatible range-boundary schemas. + pub(crate) fn boundary_schema_error(self, start: &StructType, end: &StructType) -> Error { + match self { + CdfMode::ChangeDataFeed => Error::generic(format!( + "Failed to build TableChanges: Start and end version schemas are different. Found start version schema {start:?} and end version schema {end:?}", + )), + CdfMode::RowTracking => Error::change_data_feed_incompatible_schema(end, start), + } + } + + /// Maps a reader-support failure on a protocol update to the mode-specific error. + pub(crate) fn protocol_support_error(self, underlying: Error, version: Version) -> Error { + match self { + CdfMode::ChangeDataFeed => Error::change_data_feed_unsupported(version), + CdfMode::RowTracking => underlying, + } + } +} pub(crate) const CHANGE_TYPE_COL_NAME: &str = "_change_type"; pub(crate) const COMMIT_VERSION_COL_NAME: &str = "_commit_version"; @@ -64,8 +174,9 @@ static CDF_FIELDS: LazyLock<[StructField; 3]> = LazyLock::new(|| { ] }); -/// Represents a call to read the Change Data Feed (CDF) between two versions of a table. The schema of -/// `TableChanges` will be the schema of the table at the end version with three additional columns: +/// Represents a call to read the Change Data Feed (CDF) between two versions of a table. The schema +/// of `TableChanges` will be the schema of the table at the end version with three additional +/// columns: /// - `_change_type`: String representing the type of change that for that commit. This may be one /// of `delete`, `insert`, `update_preimage`, or `update_postimage`. /// - `_commit_version`: Long representing the commit the change occurred in. @@ -79,22 +190,21 @@ static CDF_FIELDS: LazyLock<[StructField; 3]> = LazyLock::new(|| { /// /// /// Three properties must hold for the entire CDF range: -/// - Reading must be supported for every commit in the range. Currently the only read feature allowed -/// is deletion vectors. This will be expanded in the future to support more delta table features. -/// Because only deletion vectors are supported, reader version 2 will not be allowed. That is -// because version 2 requires that column mapping is enabled. Reader versions 1 and 3 are allowed. -/// - Change Data Feed must be enabled for the entire range with the `delta.enableChangeDataFeed` -/// table property set to `true`. Performing change data feed on tables with column mapping is -/// currently disallowed. We check that column mapping is disabled, or the column mapping mode is `None`. -/// - The schema for each commit must be compatible with the end schema. This means that all the -/// same fields and their nullability are the same. Schema compatibility will be expanded in the -/// future to allow compatible schemas that are not the exact same. -/// See issue [#523](https://github.com/delta-io/delta-kernel-rs/issues/523) +/// - Reading must be supported for every commit in the range: every enabled reader feature must be +/// supported by the kernel. The supported read features will be expanded in the future to cover +/// more delta table features. +/// - [`TableChanges::try_new`] requires Change Data Feed to remain enabled and requires exact +/// schema equality. +/// - [`TableChanges::try_new_row_tracking_cdf_listing`] requires row tracking to remain enabled. It +/// allows additive nullable columns and relaxed nullability, but rejects datatype changes. +/// +/// Construction validates the range boundaries. Intermediate metadata and protocol updates are +/// validated when the transaction log is replayed by the scan or listing operation. /// /// # Examples /// Get `TableChanges` for versions 0 to 1 (inclusive) /// ```rust -/// # use delta_kernel::engine::default::{storage::store_from_url, DefaultEngineBuilder}; +/// # use test_utils::delta_kernel_default_engine::{storage::store_from_url, DefaultEngineBuilder}; /// # use delta_kernel::{SnapshotRef, Error}; /// # use delta_kernel::table_changes::TableChanges; /// # let path = "./tests/data/table-with-cdf"; @@ -114,13 +224,16 @@ pub struct TableChanges { start_version: Version, schema: Schema, start_table_config: TableConfiguration, + // Both feed types share range loading and log replay. The mode is fixed at construction, and + // each consumer method verifies that it supports that mode before doing work. + mode: CdfMode, } impl TableChanges { /// Creates a new [`TableChanges`] instance for the given version range. This function checks /// these properties: /// - The change data feed table feature must be enabled in both the start or end versions. - /// - Other than the deletion vector reader feature, no other reader features are enabled for the table. + /// - Every enabled reader feature must be supported by the kernel. /// - The schemas at the start and end versions are the same. /// /// Note that this does not check that change data feed is enabled for every commit in the @@ -137,6 +250,63 @@ impl TableChanges { engine: &dyn Engine, start_version: Version, end_version: Option, + ) -> DeltaResult { + Self::try_new_internal( + table_root, + engine, + start_version, + end_version, + CdfMode::ChangeDataFeed, + ) + } + + /// Creates a listing-only change feed from row-tracking metadata. + /// + /// This path requires `delta.enableRowTracking` and ignores `_change_data` files. + /// [`TableChanges::scan_file_listing`] returns the `add` and `remove` actions whose data must + /// be reconciled by row ID. + /// + /// Construction validates the range boundaries. [`TableChanges::scan_file_listing`] validates + /// intermediate metadata and protocol updates while replaying the range. Every enabled reader + /// feature must be supported by Kernel, and each schema must be readable using the end-version + /// logical schema without datatype widening. + /// + /// # Parameters + /// + /// - `table_root`: URL of the table root containing `_delta_log`. + /// - `engine`: Engine used to read the transaction log. + /// - `start_version`: First version in the change feed. + /// - `end_version`: The end version (inclusive) of the change data feed. If this is none, this + /// defaults to the newest table version. + /// + /// # Errors + /// + /// Returns an error if the range cannot be loaded or a boundary has unavailable row tracking, + /// unsupported reader features, or an incompatible schema. Errors from intermediate versions + /// are returned by [`TableChanges::scan_file_listing`]. + #[cfg_attr(not(feature = "internal-api"), allow(dead_code))] + #[internal_api] + pub(crate) fn try_new_row_tracking_cdf_listing( + table_root: Url, + engine: &dyn Engine, + start_version: Version, + end_version: Option, + ) -> DeltaResult { + Self::try_new_internal( + table_root, + engine, + start_version, + end_version, + CdfMode::RowTracking, + ) + } + + fn try_new_internal( + table_root: Url, + engine: &dyn Engine, + start_version: Version, + end_version: Option, + mode: CdfMode, ) -> DeltaResult { let log_root = table_root.join("_delta_log/")?; let log_segment = LogSegment::for_table_changes( @@ -163,32 +333,53 @@ impl TableChanges { .table_configuration() .ensure_operation_supported(Operation::Cdf)?; - // Verify CDF is enabled at the beginning and end of the interval using - // [`check_cdf_table_properties`] to fail early. This also ensures that column mapping is - // disabled. + // Verify the change feed is enabled at the beginning and end of the interval to fail early. + // The `ensure_operation_supported` calls above already validate that every enabled reader + // feature is supported by the kernel (e.g. deletion vectors and column mapping). The + // feature that must additionally be *enabled* depends on the mode: ChangeDataFeed + // for the cdc-file path, RowTracking for the row-tracking path. // // Note: We must still check each metadata and protocol action in the CDF range. - let check_table_config = |snapshot: &Snapshot| { - if snapshot - .table_configuration() - .is_feature_enabled(&TableFeature::ChangeDataFeed) - { - Ok(()) - } else { - Err(Error::change_data_feed_unsupported(snapshot.version())) - } + let check_table_config = |snapshot: &Snapshot| -> DeltaResult<()> { + require!( + snapshot + .table_configuration() + .is_feature_enabled(&mode.required_feature()), + mode.feature_disabled_error(snapshot.version()) + ); + Ok(()) }; check_table_config(&start_snapshot)?; check_table_config(&end_snapshot)?; - // Verify that the start and end schemas are compatible. We must still check schema - // compatibility for each schema update in the CDF range. - // Note: Schema compatibility check will be changed in the future to be more flexible. - // See issue [#523](https://github.com/delta-io/delta-kernel-rs/issues/523) - if start_snapshot.schema() != end_snapshot.schema() { - return Err(Error::generic(format!( - "Failed to build TableChanges: Start and end version schemas are different. Found start version schema {:?} and end version schema {:?}", start_snapshot.schema(), end_snapshot.schema(), - ))); + if mode == CdfMode::RowTracking { + let properties = end_snapshot.table_properties(); + require!( + properties.materialized_row_id_column_name.is_some(), + Error::generic(format!( + "Row tracking is enabled at version {}, but metadata property {} is missing", + end_snapshot.version(), + MATERIALIZED_ROW_ID_COLUMN_NAME + )) + ); + require!( + properties + .materialized_row_commit_version_column_name + .is_some(), + Error::generic(format!( + "Row tracking is enabled at version {}, but metadata property {} is missing", + end_snapshot.version(), + MATERIALIZED_ROW_COMMIT_VERSION_COLUMN_NAME + )) + ); + } + + // Log replay validates only commits that contain metadata, so validate the range boundary + // separately. Compatibility is evaluated against the logical schemas. + let start_schema = start_snapshot.schema(); + let end_schema = end_snapshot.schema(); + if !mode.schemas_compatible(start_schema.as_ref(), end_schema.as_ref()) { + return Err(mode.boundary_schema_error(start_schema.as_ref(), end_schema.as_ref())); } let schema = StructType::try_new( @@ -206,6 +397,7 @@ impl TableChanges { start_version, schema, start_table_config: start_snapshot.table_configuration().clone(), + mode, }) } @@ -228,6 +420,57 @@ impl TableChanges { &self.table_root } + /// Returns the physical Parquet column that stores materialized row IDs. + /// + /// # Errors + /// + /// Returns an error unless this value was created by + /// [`TableChanges::try_new_row_tracking_cdf_listing`]. + #[cfg_attr(not(feature = "internal-api"), allow(dead_code))] + #[internal_api] + pub(crate) fn materialized_row_id_column_name(&self) -> DeltaResult<&str> { + self.row_tracking_table_properties()? + .materialized_row_id_column_name + .as_deref() + .ok_or_else(|| { + Error::internal_error( + "A row-tracking TableChanges is missing its materialized row ID column name", + ) + }) + } + + /// Returns the physical Parquet column that stores materialized row commit versions. + /// + /// # Errors + /// + /// Returns an error unless this value was created by + /// [`TableChanges::try_new_row_tracking_cdf_listing`]. + #[cfg_attr(not(feature = "internal-api"), allow(dead_code))] + #[internal_api] + pub(crate) fn materialized_row_commit_version_column_name(&self) -> DeltaResult<&str> { + self.row_tracking_table_properties()? + .materialized_row_commit_version_column_name + .as_deref() + .ok_or_else(|| { + Error::internal_error( + "A row-tracking TableChanges is missing its materialized row commit version \ + column name", + ) + }) + } + + fn row_tracking_table_properties( + &self, + ) -> DeltaResult<&crate::table_properties::TableProperties> { + require!( + self.mode == CdfMode::RowTracking, + Error::unsupported( + "Row-tracking column names are only available for row-tracking change feeds" + ) + ); + Ok(self.end_snapshot.table_properties()) + } + /// Create a [`TableChangesScanBuilder`] for an `Arc`. pub fn scan_builder(self: Arc) -> TableChangesScanBuilder { TableChangesScanBuilder::new(self) @@ -237,17 +480,105 @@ impl TableChanges { pub fn into_scan_builder(self) -> TableChangesScanBuilder { TableChangesScanBuilder::new(self) } + + /// Lists the files required to reconstruct a row-tracking change feed without reading data + /// files. + /// + /// Each [`TableChangesFileAction`] contains an add side, a remove side, or both. Read each side + /// using its own deletion vector. For every selected physical row, reconstruct the row ID as + /// `coalesce(materialized_row_id, base_row_id + physical_row_index)`. Reconstruct the row + /// commit version as + /// `coalesce(materialized_row_commit_version, default_row_commit_version)`. The materialized + /// column names are returned by [`TableChanges::materialized_row_id_column_name`] and + /// [`TableChanges::materialized_row_commit_version_column_name`]. Assign physical row indexes + /// before applying deletion vectors. + /// + /// Rows present only on the add side are inserts, rows present only on the remove side are + /// deletes, and matching row IDs with different row commit versions are updates. Matching both + /// the row ID and row commit version identifies a row carried forward without change. + /// `AddCDCFile` actions are ignored. + /// + /// Both listing modes buffer the selected actions before returning. The iterator therefore + /// reports preparation errors immediately and uses memory proportional to the range's action + /// count. This method requires a [`TableChanges`] created by + /// [`TableChanges::try_new_row_tracking_cdf_listing`]. + /// + /// # Errors + /// + /// Returns an error if this value was not created by + /// [`TableChanges::try_new_row_tracking_cdf_listing`] or if the log actions cannot be replayed + /// into a valid listing. + #[cfg_attr(not(feature = "internal-api"), allow(dead_code))] + #[internal_api] + pub(crate) fn scan_file_listing( + self: Arc, + engine: Arc, + mode: TableChangesListingMode, + ) -> DeltaResult>> { + if self.mode != CdfMode::RowTracking { + return Err(Error::unsupported( + "scan_file_listing is only supported for row-tracking change feeds; construct \ + the TableChanges with TableChanges::try_new_row_tracking_cdf_listing", + )); + } + + let commits = self.log_segment.listed.ascending_commit_files.clone(); + let schema = self.end_snapshot.schema(); + let scan_metadata = table_changes_action_iter_with_mode( + engine, + &self.start_table_config, + commits, + schema, + None, + self.mode, + )?; + // Any listing error surfaces here rather than mid-iteration. + let actions = scan_metadata_to_scan_file(scan_metadata) + .map(|scan_file| TableChangesFileAction::try_from_scan_file(scan_file?)) + .collect::>>()?; + let actions = match mode { + TableChangesListingMode::AllChanges => actions, + TableChangesListingMode::NetChanges => net_changes::collapse_net_changes(actions)?, + }; + Ok(actions.into_iter().map(Ok)) + } } #[cfg(test)] mod tests { - use super::*; + use std::collections::HashMap; + use std::sync::Arc; + use itertools::{assert_equal, Itertools}; + + use super::*; + use crate::actions::deletion_vector::DeletionVectorDescriptor; + use crate::actions::{Add, Metadata, Protocol, Remove}; use crate::engine::sync::SyncEngine; - use crate::schema::{DataType, StructField}; + use crate::schema::{DataType, StructField, StructType}; + use crate::table_changes::test_utils::{ + row_tracking_properties, row_tracking_protocol, row_tracking_setup_actions, + test_deletion_vector, TEST_MATERIALIZED_ROW_COMMIT_VERSION_COLUMN_NAME, + TEST_MATERIALIZED_ROW_ID_COLUMN_NAME, + }; use crate::table_changes::CDF_FIELDS; - use crate::Error; - use itertools::assert_equal; + use crate::table_features::TableFeature; + use crate::table_properties::{ + COLUMN_MAPPING_MODE, MATERIALIZED_ROW_COMMIT_VERSION_COLUMN_NAME, + MATERIALIZED_ROW_ID_COLUMN_NAME, + }; + use crate::unit_test_utils::{ + assert_result_error_with_message, test_schema_flat_with_column_mapping, Action, + LocalMockTable, + }; + use crate::{Engine, Error}; + + fn listing_test_schema() -> Arc { + Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("value", DataType::STRING), + ])) + } #[test] fn table_changes_checks_enable_cdf_flag() { @@ -308,4 +639,426 @@ mod tests { TableChanges::try_new(url.clone(), engine.as_ref(), 0, 0.into()).unwrap(); assert_equal(expected_schema, table_changes.schema().fields().cloned()); } + + #[test] + fn scan_file_listing_rejects_cdc_file_table_changes() { + // A change-data-file `TableChanges` (built via `try_new`) must not be consumed through the + // row-tracking listing API. + let path = "./tests/data/table-with-cdf"; + let engine: Arc = Arc::new(SyncEngine::new()); + let url = delta_kernel::try_parse_uri(path).unwrap(); + let table_changes = + Arc::new(TableChanges::try_new(url, engine.as_ref(), 0, Some(1)).unwrap()); + assert_result_error_with_message( + table_changes.materialized_row_id_column_name(), + "only available for row-tracking change feeds", + ); + assert_result_error_with_message( + table_changes.materialized_row_commit_version_column_name(), + "only available for row-tracking change feeds", + ); + let res = table_changes.scan_file_listing(engine, TableChangesListingMode::AllChanges); + assert!( + matches!(res, Err(Error::Unsupported(_))), + "scan_file_listing on a cdc-file TableChanges must return an unsupported error" + ); + } + + #[rstest::rstest] + #[case::row_id(MATERIALIZED_ROW_ID_COLUMN_NAME)] + #[case::row_commit_version(MATERIALIZED_ROW_COMMIT_VERSION_COLUMN_NAME)] + #[tokio::test] + async fn try_new_row_tracking_cdf_listing_rejects_missing_materialized_column_name( + #[case] missing_property: &str, + ) { + let engine: Arc = Arc::new(SyncEngine::new()); + let mut mock_table = LocalMockTable::new(); + let mut properties = row_tracking_properties(); + properties.remove(missing_property); + let metadata = + Metadata::try_new(None, None, listing_test_schema(), vec![], 0, properties).unwrap(); + mock_table + .commit([ + Action::Protocol(row_tracking_protocol()), + Action::Metadata(metadata), + ]) + .await; + + let table_root = url::Url::from_directory_path(mock_table.table_root()).unwrap(); + assert_result_error_with_message( + TableChanges::try_new_row_tracking_cdf_listing(table_root, engine.as_ref(), 0, Some(0)), + missing_property, + ); + } + + #[test] + fn try_new_row_tracking_cdf_listing_fails_when_row_tracking_disabled() { + // table-with-cdf enables change data feed but not row tracking, so the row-tracking feed + // must be rejected at construction. + let path = "./tests/data/table-with-cdf"; + let engine = Box::new(SyncEngine::new()); + let url = delta_kernel::try_parse_uri(path).unwrap(); + let res = TableChanges::try_new_row_tracking_cdf_listing(url, engine.as_ref(), 0, Some(1)); + assert!( + matches!(&res, Err(Error::RowTrackingChangeFeedUnsupported(_))), + "expected a row-tracking-disabled error, got {res:?}" + ); + } + + #[tokio::test] + async fn try_new_row_tracking_cdf_listing_rejects_incompatible_start_schema() { + // The start-version schema (in effect for in-range commits before any metadata update) + // carries a column the end-version schema drops. No in-range commit re-declares the start + // schema, so this boundary incompatibility is caught only by the start-vs-end check, not + // the per-commit one. + let engine: Arc = Arc::new(SyncEngine::new()); + let mut mock_table = LocalMockTable::new(); + let start_schema = Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("value", DataType::STRING), + StructField::nullable("extra", DataType::INTEGER), + ])); + let end_schema = Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("value", DataType::STRING), + ])); + let rt_config = row_tracking_properties(); + + // v0: start schema + row tracking. v1: a data commit (no metadata) using the start schema. + // v2: a metadata commit that drops `extra` -- never re-declaring the start schema in range. + mock_table + .commit(row_tracking_setup_actions(start_schema)) + .await; + mock_table + .commit([Action::Add(Add { + path: "f1".into(), + data_change: true, + size: 10, + base_row_id: Some(0), + default_row_commit_version: Some(1), + ..Default::default() + })]) + .await; + mock_table + .commit([Action::Metadata( + Metadata::try_new(None, None, end_schema, vec![], 0, rt_config).unwrap(), + )]) + .await; + + let table_root = url::Url::from_directory_path(mock_table.table_root()).unwrap(); + let res = + TableChanges::try_new_row_tracking_cdf_listing(table_root, engine.as_ref(), 1, Some(2)); + assert!( + matches!(&res, Err(Error::ChangeDataFeedIncompatibleSchema(_, _))), + "expected an incompatible start schema to be rejected, got {res:?}" + ); + } + + #[tokio::test] + async fn scan_file_listing_surfaces_row_tracking_files_end_to_end() { + let engine: Arc = Arc::new(SyncEngine::new()); + let mut mock_table = LocalMockTable::new(); + + // version 0: protocol + metadata (row tracking enabled) + an insert. + let add_v0 = Add { + path: "path_0".into(), + data_change: true, + size: 100, + base_row_id: Some(0), + default_row_commit_version: Some(0), + ..Default::default() + }; + let add_v1 = Add { + path: "path_1".into(), + data_change: true, + size: 100, + base_row_id: Some(5), + default_row_commit_version: Some(1), + ..Default::default() + }; + mock_table + .commit( + row_tracking_setup_actions(listing_test_schema()) + .into_iter() + .chain([Action::Add(add_v0)]), + ) + .await; + mock_table.commit([Action::Add(add_v1)]).await; + + let table_root = url::Url::from_directory_path(mock_table.table_root()).unwrap(); + let table_changes = Arc::new( + TableChanges::try_new_row_tracking_cdf_listing(table_root, engine.as_ref(), 0, Some(1)) + .unwrap(), + ); + let listing: Vec = table_changes + .scan_file_listing(engine, TableChangesListingMode::AllChanges) + .unwrap() + .try_collect() + .unwrap(); + + // Both inserts surface (no cdc files) as single-sided add actions, each carrying its base + // row id. + assert_eq!(listing.len(), 2); + let add_side = |path: &str| { + listing + .iter() + .filter_map(|fa| fa.add.as_ref()) + .find(|a| a.path == path) + .unwrap_or_else(|| panic!("{path} add side present")) + .clone() + }; + assert!(listing.iter().all(|fa| fa.remove.is_none())); + assert_eq!(add_side("path_0").base_row_id, Some(0)); + assert_eq!(add_side("path_1").base_row_id, Some(5)); + } + + #[tokio::test] + async fn scan_file_listing_preserves_column_mapped_partition_values() { + let engine: Arc = Arc::new(SyncEngine::new()); + let mut mock_table = LocalMockTable::new(); + let schema = test_schema_flat_with_column_mapping(); + let mut properties = row_tracking_properties(); + properties.insert(COLUMN_MAPPING_MODE.to_string(), "name".to_string()); + let metadata = + Metadata::try_new(None, None, schema, vec!["name".to_string()], 0, properties).unwrap(); + let protocol = Protocol::try_new_modern( + [TableFeature::ColumnMapping], + [ + TableFeature::ColumnMapping, + TableFeature::RowTracking, + TableFeature::DomainMetadata, + ], + ) + .unwrap(); + let partition_values = HashMap::from([("phys_name".to_string(), "north".to_string())]); + mock_table + .commit([ + Action::Protocol(protocol), + Action::Metadata(metadata), + Action::Add(Add { + path: "name=north/file.parquet".into(), + partition_values: partition_values.clone(), + data_change: true, + size: 100, + base_row_id: Some(0), + default_row_commit_version: Some(0), + ..Default::default() + }), + ]) + .await; + + let table_root = url::Url::from_directory_path(mock_table.table_root()).unwrap(); + let table_changes = Arc::new( + TableChanges::try_new_row_tracking_cdf_listing(table_root, engine.as_ref(), 0, Some(0)) + .unwrap(), + ); + assert_eq!( + table_changes.materialized_row_id_column_name().unwrap(), + TEST_MATERIALIZED_ROW_ID_COLUMN_NAME + ); + assert_eq!( + table_changes + .materialized_row_commit_version_column_name() + .unwrap(), + TEST_MATERIALIZED_ROW_COMMIT_VERSION_COLUMN_NAME + ); + let listing: Vec = table_changes + .scan_file_listing(engine, TableChangesListingMode::AllChanges) + .unwrap() + .try_collect() + .unwrap(); + + assert_eq!(listing.len(), 1); + assert_eq!( + listing[0].add.as_ref().unwrap().partition_values, + partition_values + ); + } + + #[derive(Clone, Copy)] + enum DvUpdateLayout { + SameCommit, + CrossCommit, + } + + enum Expected { + Grouped { + remove_version: i64, + add_version: i64, + }, + TwoSingleSided, + } + + #[rstest::rstest] + #[case::same_commit_all_changes( + DvUpdateLayout::SameCommit, + TableChangesListingMode::AllChanges, + Expected::Grouped { remove_version: 1, add_version: 1 } + )] + #[case::same_commit_net_changes( + DvUpdateLayout::SameCommit, + TableChangesListingMode::NetChanges, + Expected::Grouped { remove_version: 1, add_version: 1 } + )] + #[case::cross_commit_all_changes( + DvUpdateLayout::CrossCommit, + TableChangesListingMode::AllChanges, + Expected::TwoSingleSided + )] + #[case::cross_commit_net_changes( + DvUpdateLayout::CrossCommit, + TableChangesListingMode::NetChanges, + Expected::Grouped { remove_version: 1, add_version: 2 } + )] + #[tokio::test] + async fn scan_file_listing_groups_dv_update( + #[case] layout: DvUpdateLayout, + #[case] mode: TableChangesListingMode, + #[case] expected: Expected, + ) { + let engine: Arc = Arc::new(SyncEngine::new()); + let mut mock_table = LocalMockTable::new(); + let dv1 = test_deletion_vector("vBn[lx{q8@P<9BNH/isA", 2); + let dv2 = test_deletion_vector("U5OWRz5k%CFT.Td}yCPW", 3); + let remove_f = |dv: &DeletionVectorDescriptor| { + Action::Remove(Remove { + path: "f".into(), + data_change: true, + deletion_vector: Some(dv.clone()), + ..Default::default() + }) + }; + let add_f = |dv: &DeletionVectorDescriptor, commit_version: i64| { + Action::Add(Add { + path: "f".into(), + data_change: true, + size: 100, + deletion_vector: Some(dv.clone()), + base_row_id: Some(0), + default_row_commit_version: Some(commit_version), + ..Default::default() + }) + }; + + mock_table + .commit( + row_tracking_setup_actions(listing_test_schema()) + .into_iter() + .chain([add_f(&dv1, 0)]), + ) + .await; + let end_version = match layout { + DvUpdateLayout::SameCommit => { + mock_table.commit([remove_f(&dv1), add_f(&dv2, 1)]).await; + 1 + } + DvUpdateLayout::CrossCommit => { + mock_table.commit([remove_f(&dv1)]).await; + mock_table.commit([add_f(&dv2, 2)]).await; + 2 + } + }; + + let table_root = url::Url::from_directory_path(mock_table.table_root()).unwrap(); + let table_changes = Arc::new( + TableChanges::try_new_row_tracking_cdf_listing( + table_root, + engine.as_ref(), + 1, + Some(end_version), + ) + .unwrap(), + ); + let listing: Vec = table_changes + .scan_file_listing(engine, mode) + .unwrap() + .try_collect() + .unwrap(); + + match expected { + Expected::Grouped { + remove_version, + add_version, + } => { + assert_eq!(listing.len(), 1, "expected one grouped action: {listing:?}"); + let update = &listing[0]; + let remove = update.remove.as_ref().expect("remove side"); + assert_eq!(remove.commit_version, remove_version); + assert_eq!(remove.deletion_vector, Some(dv1)); + let add = update.add.as_ref().expect("add side"); + assert_eq!(add.commit_version, add_version); + assert_eq!(add.deletion_vector, Some(dv2)); + } + Expected::TwoSingleSided => { + assert_eq!( + listing.len(), + 2, + "expected two ungrouped single-sided actions: {listing:?}" + ); + let all_remove = listing + .iter() + .find(|fa| fa.remove.is_some()) + .expect("a standalone remove side"); + assert!(all_remove.add.is_none(), "remove must be single-sided"); + let all_remove = all_remove.remove.as_ref().unwrap(); + assert_eq!(all_remove.commit_version, 1); + assert_eq!(all_remove.deletion_vector, Some(dv1)); + let all_add = listing + .iter() + .find(|fa| fa.add.is_some()) + .expect("a standalone add side"); + assert!(all_add.remove.is_none(), "add must be single-sided"); + let all_add = all_add.add.as_ref().unwrap(); + assert_eq!(all_add.commit_version, 2); + assert_eq!(all_add.deletion_vector, Some(dv2)); + } + } + } + + #[tokio::test] + async fn scan_builder_rejects_row_tracking_table_changes() { + let engine: Arc = Arc::new(SyncEngine::new()); + let mut mock_table = LocalMockTable::new(); + mock_table + .commit(row_tracking_setup_actions(listing_test_schema())) + .await; + + let table_root = url::Url::from_directory_path(mock_table.table_root()).unwrap(); + let table_changes = + TableChanges::try_new_row_tracking_cdf_listing(table_root, engine.as_ref(), 0, Some(0)) + .unwrap(); + let res = table_changes.into_scan_builder().build(); + assert_result_error_with_message( + res, + "A row-tracking TableChanges cannot be scanned for data", + ); + } + + #[rstest::rstest] + #[case::all_changes(TableChangesListingMode::AllChanges)] + #[case::net_changes(TableChangesListingMode::NetChanges)] + #[tokio::test] + async fn scan_file_listing_empty_range_yields_no_actions( + #[case] mode: TableChangesListingMode, + ) { + // A range whose only commit is the row-tracking setup (protocol + metadata, no data files) + // has no add/remove actions to list, so both modes must return an empty listing. + let engine: Arc = Arc::new(SyncEngine::new()); + let mut mock_table = LocalMockTable::new(); + mock_table + .commit(row_tracking_setup_actions(listing_test_schema())) + .await; + + let table_root = url::Url::from_directory_path(mock_table.table_root()).unwrap(); + let table_changes = Arc::new( + TableChanges::try_new_row_tracking_cdf_listing(table_root, engine.as_ref(), 0, Some(0)) + .unwrap(), + ); + let listing: Vec = table_changes + .scan_file_listing(engine, mode) + .unwrap() + .try_collect() + .unwrap(); + assert!(listing.is_empty(), "expected an empty listing: {listing:?}"); + } } diff --git a/kernel/src/table_changes/net_changes.rs b/kernel/src/table_changes/net_changes.rs new file mode 100644 index 0000000000..d66e96b002 --- /dev/null +++ b/kernel/src/table_changes/net_changes.rs @@ -0,0 +1,341 @@ +//! Collapses the per-commit row-tracking change-feed listing to its NET effect per file path over +//! the whole version range, for [`crate::table_changes::TableChangesListingMode::NetChanges`]. +//! +//! The per-commit listing ([`crate::table_changes::TableChangesListingMode::AllChanges`]) emits one +//! file per `add`/`remove` action in every commit. NetChanges reduces each file *path* to only its +//! boundary actions across the range: the earliest remove and the latest add. Intermediate actions +//! are dropped and never handed to the connector to scan. +//! +//! This is a scan-selection collapse, not the change-feed's correctness engine: the connector still +//! reconstructs the row-level feed (pairing an update's pre-image and post-image by row id) from +//! the files this returns. Its job is only to avoid listing files whose rows net to no change +//! across the range. + +use std::collections::HashMap; + +use crate::table_changes::scan_file::{TableChangesFileAction, TableChangesScanFile}; +use crate::{DeltaResult, Error}; + +/// One flattened side of a per-commit [`TableChangesFileAction`], tagged with which side it is so a +/// path's boundaries can be ordered. `is_add` sorts a remove before an add at the same commit. +struct Side { + is_add: bool, + file: TableChangesScanFile, +} + +/// Collapses per-commit [`TableChangesFileAction`]s to their net effect per path across the whole +/// range. Only the earliest remove-side action and the latest add-side action of a path survive, so +/// intermediate actions are dropped and never scanned. +/// +/// Each input action is flattened to its side(s) -- a same-commit deletion-vector update +/// (`{add: Some, remove: Some}`) contributes both -- keyed by file path. A file path is globally +/// unique within a table's commit range (it carries the full partition prefix and a unique write +/// UUID), so keying on the path alone never merges actions from different files or partitions. Per +/// path, the global earliest and latest sides are ordered by `(commit_version, add-side)` (a +/// remove-side precedes an add-side within a commit), and: +/// - earliest is a remove-side and latest is an add-side with differing deletion vectors -> a net +/// update: one output [`TableChangesFileAction`] carrying both boundary sides -- the pre-image +/// remove (with its DV) and the post-image add (with its DV). The connector reads each side under +/// its own DV and reconciles rows by row id; +/// - earliest is a remove-side with no later add-side -> a net delete (`{remove: Some}`); +/// - earliest and latest are both add-side -> a net insert (`{add: Some}`); +/// - added-then-removed within the range, or surviving deletion vectors identical (a carry-over +/// rewrite) -> dropped: the rows net to no change. +/// +/// Deletion-vector equality here is structural (all descriptor fields), which is conservative: a +/// pair that is logically equivalent but structurally different still emits a net update. Only +/// structurally identical boundary DVs are dropped, so a surviving path is not a guarantee its rows +/// changed; the connector's row-level reconciliation is the authoritative carry-over check. +/// +/// Each path yields at most one output action. The result is sorted by path (unique per output) for +/// a deterministic listing, since the intermediate path map has no defined iteration order. +pub(super) fn collapse_net_changes( + actions: Vec, +) -> DeltaResult> { + let mut by_path: HashMap> = HashMap::new(); + for action in actions { + if let Some(add) = action.add { + by_path.entry(add.path.clone()).or_default().push(Side { + is_add: true, + file: add, + }); + } + if let Some(remove) = action.remove { + by_path.entry(remove.path.clone()).or_default().push(Side { + is_add: false, + file: remove, + }); + } + } + + let mut out: Vec<(String, TableChangesFileAction)> = Vec::with_capacity(by_path.len()); + for (path, sides) in by_path { + let key = |s: &&Side| (s.file.commit_version, s.is_add); + let (Some(earliest), Some(latest)) = + (sides.iter().min_by_key(key), sides.iter().max_by_key(key)) + else { + return Err(Error::internal_error(format!( + "net-changes collapse produced an empty side slot for path {path}" + ))); + }; + // Only a leading remove and/or a trailing add mark a net change; clone the boundary + // file(s) that survive. + let first_remove = (!earliest.is_add).then(|| earliest.file.clone()); + let last_add = latest.is_add.then(|| latest.file.clone()); + let action = match (first_remove, last_add) { + (Some(remove), None) => TableChangesFileAction { + add: None, + remove: Some(remove), + }, + (None, Some(add)) => TableChangesFileAction { + add: Some(add), + remove: None, + }, + (Some(remove), Some(add)) if remove.deletion_vector != add.deletion_vector => { + TableChangesFileAction { + add: Some(add), + remove: Some(remove), + } + } + (Some(_), Some(_)) | (None, None) => continue, + }; + out.push((path, action)); + } + + out.sort_unstable_by(|(a, _), (b, _)| a.cmp(b)); + Ok(out.into_iter().map(|(_, action)| action).collect()) +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + + use rstest::rstest; + + use crate::actions::deletion_vector::{DeletionVectorDescriptor, DeletionVectorStorageType}; + use crate::table_changes::net_changes::collapse_net_changes; + use crate::table_changes::scan_file::{TableChangesFileAction, TableChangesScanFile}; + + /// Builds a deletion-vector descriptor identified by `id`. Two descriptors with different ids + /// compare unequal, which is how the collapse tells a real net update from a same-DV carry-over + /// rewrite. + fn test_dv(id: &str) -> DeletionVectorDescriptor { + DeletionVectorDescriptor { + storage_type: DeletionVectorStorageType::PersistedRelative, + path_or_inline_dv: id.to_string(), + offset: Some(1), + size_in_bytes: 8, + cardinality: 1, + } + } + + /// Builds one side (`TableChangesScanFile`) for the collapse tests. Only path, commit version, + /// and this side's deletion vector vary across cases; the rest are fixed. + fn side( + path: &str, + commit_version: i64, + deletion_vector: Option, + ) -> TableChangesScanFile { + TableChangesScanFile { + path: path.to_string(), + deletion_vector, + partition_values: HashMap::new(), + size: Some(10), + commit_version, + commit_timestamp: 0, + base_row_id: Some(0), + default_row_commit_version: Some(1), + } + } + + /// A single-sided insert action for `path` at `commit_version`. + fn insert( + path: &str, + commit_version: i64, + dv: Option, + ) -> TableChangesFileAction { + TableChangesFileAction { + add: Some(side(path, commit_version, dv)), + remove: None, + } + } + + /// A single-sided delete action for `path` at `commit_version`. + fn delete( + path: &str, + commit_version: i64, + dv: Option, + ) -> TableChangesFileAction { + TableChangesFileAction { + add: None, + remove: Some(side(path, commit_version, dv)), + } + } + + /// A same-commit DV update: both sides at `commit_version`, the add carrying `add_dv` and the + /// remove carrying `removed_dv`. This is the grouped shape `AllChanges` emits for an in-commit + /// DV update. + fn paired_dv_update( + path: &str, + commit_version: i64, + add_dv: Option, + removed_dv: Option, + ) -> TableChangesFileAction { + TableChangesFileAction { + add: Some(side(path, commit_version, add_dv)), + remove: Some(side(path, commit_version, removed_dv)), + } + } + + /// Summarizes an action as `(path, add commit + dv, remove commit + dv)` for assertions. + type SideSummary = Option<(i64, Option)>; + fn summarize(a: &TableChangesFileAction) -> (String, SideSummary, SideSummary) { + let s = |side: &Option| { + side.as_ref() + .map(|f| (f.commit_version, f.deletion_vector.clone())) + }; + let path = a + .add + .as_ref() + .or(a.remove.as_ref()) + .map(|f| f.path.clone()) + .expect("an action always has at least one side"); + (path, s(&a.add), s(&a.remove)) + } + + #[test] + fn net_changes_collapse_reduces_each_path_to_its_net_effect() { + let dv_a = test_dv("aaaaaaaaaaaaaaaaaaaa"); + let dv_b = test_dv("bbbbbbbbbbbbbbbbbbbb"); + let actions = vec![ + // insert.parquet: only ever added in the range -> net insert. + insert("insert.parquet", 1, None), + // delete.parquet: removed with no later add -> net delete. + delete("delete.parquet", 1, Some(dv_a.clone())), + // update.parquet: removed at v1 then re-added at v2 with a different DV -> net update, + // grouped into a remove side at v1 and an add side at v2. + delete("update.parquet", 1, Some(dv_a.clone())), + insert("update.parquet", 2, Some(dv_b.clone())), + // carryover.parquet: rewritten with the same DV on both sides -> no net change, + // dropped. + delete("carryover.parquet", 1, Some(dv_a.clone())), + insert("carryover.parquet", 2, Some(dv_a.clone())), + // churn.parquet: added then removed within the range -> no net change, dropped. + insert("churn.parquet", 1, None), + delete("churn.parquet", 2, None), + // reinsert.parquet: added, removed, then re-added within the range -> net insert at + // the last add (the (None, Some) arm reached via min/max boundaries, not a lone add). + insert("reinsert.parquet", 1, None), + delete("reinsert.parquet", 2, None), + insert("reinsert.parquet", 3, None), + ]; + + let out = collapse_net_changes(actions).unwrap(); + + // Only the three net-effect paths survive; the two no-net-change paths are dropped. Sorted + // by path, each path yields a single grouped action. + assert_eq!( + out.iter().map(summarize).collect::>(), + vec![ + // delete: single remove side. + ( + "delete.parquet".to_string(), + None, + Some((1, Some(dv_a.clone()))) + ), + // insert: single add side. + ("insert.parquet".to_string(), Some((1, None)), None), + // reinsert: single add side at the last add (v3). + ("reinsert.parquet".to_string(), Some((3, None)), None), + // update: grouped remove (v1, dv_a) + add (v2, dv_b). + ( + "update.parquet".to_string(), + Some((2, Some(dv_b))), + Some((1, Some(dv_a))) + ), + ] + ); + } + + #[rstest] + #[case::same_commit_dv_update( + vec![paired_dv_update("update.parquet", 3, Some(test_dv("b")), Some(test_dv("a")))], + vec![( + "update.parquet".to_string(), + Some((3, Some(test_dv("b")))), + Some((3, Some(test_dv("a")))), + )] + )] + #[case::multi_update_chain( + vec![ + paired_dv_update("f.parquet", 1, Some(test_dv("b")), Some(test_dv("a"))), + paired_dv_update("f.parquet", 2, Some(test_dv("c")), Some(test_dv("b"))), + ], + vec![( + "f.parquet".to_string(), + Some((2, Some(test_dv("c")))), + Some((1, Some(test_dv("a")))), + )] + )] + #[case::multi_commit_revert( + vec![ + paired_dv_update("f.parquet", 1, Some(test_dv("b")), Some(test_dv("a"))), + paired_dv_update("f.parquet", 2, Some(test_dv("a")), Some(test_dv("b"))), + ], + vec![] + )] + #[case::copy_on_write( + vec![delete("old.parquet", 1, None), insert("new.parquet", 1, None)], + vec![ + ("new.parquet".to_string(), Some((1, None)), None), + ("old.parquet".to_string(), None, Some((1, None))), + ] + )] + fn net_changes_collapse_reduces_paths_to_their_boundaries( + #[case] actions: Vec, + #[case] expected: Vec<(String, SideSummary, SideSummary)>, + ) { + let out = collapse_net_changes(actions).unwrap(); + assert_eq!(out.iter().map(summarize).collect::>(), expected); + } + + /// The row-tracking metadata (`base_row_id`, `default_row_commit_version`) on each boundary + /// side must survive the collapse unchanged -- it is what the connector uses to reconcile + /// pre-image and post-image rows by row id. Regression guard for a net update across + /// commits. + #[test] + fn net_changes_collapse_preserves_row_tracking_fields() { + let dv_a = test_dv("aaaaaaaaaaaaaaaaaaaa"); + let dv_b = test_dv("bbbbbbbbbbbbbbbbbbbb"); + // Distinct row-tracking metadata on each side so a swap or drop would be caught. The + // `side` helper's fixed defaults are overridden per side. + let actions = vec![ + TableChangesFileAction { + add: None, + remove: Some(TableChangesScanFile { + base_row_id: Some(100), + default_row_commit_version: Some(1), + ..side("f.parquet", 1, Some(dv_a)) + }), + }, + TableChangesFileAction { + add: Some(TableChangesScanFile { + base_row_id: Some(200), + default_row_commit_version: Some(2), + ..side("f.parquet", 2, Some(dv_b)) + }), + remove: None, + }, + ]; + + let out = collapse_net_changes(actions).unwrap(); + + assert_eq!(out.len(), 1, "expected one grouped net update: {out:?}"); + let remove = out[0].remove.as_ref().expect("remove side"); + let add = out[0].add.as_ref().expect("add side"); + assert_eq!(remove.base_row_id, Some(100)); + assert_eq!(remove.default_row_commit_version, Some(1)); + assert_eq!(add.base_row_id, Some(200)); + assert_eq!(add.default_row_commit_version, Some(2)); + } +} diff --git a/kernel/src/table_changes/physical_to_logical.rs b/kernel/src/table_changes/physical_to_logical.rs index 0676cc3fda..e95ac8c5d7 100644 --- a/kernel/src/table_changes/physical_to_logical.rs +++ b/kernel/src/table_changes/physical_to_logical.rs @@ -1,14 +1,14 @@ use std::collections::HashMap; +use super::scan_file::{CdfScanFile, CdfScanFileType}; +use super::{CHANGE_TYPE_COL_NAME, COMMIT_TIMESTAMP_COL_NAME, COMMIT_VERSION_COL_NAME}; use crate::expressions::Scalar; use crate::scan::state_info::StateInfo; use crate::scan::transform_spec::{get_transform_expr, parse_partition_values}; use crate::schema::{DataType, SchemaRef, StructField, StructType}; +use crate::timestamp_timezone::TimestampTimezone; use crate::{DeltaResult, Error, ExpressionRef}; -use super::scan_file::{CdfScanFile, CdfScanFileType}; -use super::{CHANGE_TYPE_COL_NAME, COMMIT_TIMESTAMP_COL_NAME, COMMIT_VERSION_COL_NAME}; - /// Gets CDF metadata columns from the logical schema and scan file. /// /// This function directly looks up CDF columns in the schema and generates their values @@ -26,7 +26,8 @@ fn get_cdf_columns( Some((idx, (name, value))) } (Some(_), CdfScanFileType::Cdc) | (None, _) => { - // Cdc files contain the `change_type_` column physically, so we do not insert a metadata-derived value + // Cdc files contain the `change_type_` column physically, so we do not insert a + // metadata-derived value None } }; @@ -63,7 +64,8 @@ pub(crate) fn scan_file_physical_schema( if scan_file.scan_type == CdfScanFileType::Cdc { let change_type = StructField::not_null(CHANGE_TYPE_COL_NAME, DataType::STRING); let fields = physical_schema.fields().cloned().chain(Some(change_type)); - // NOTE: We don't validate the fields again because CHANGE_TYPE_COL_NAME should never be used anywhere else + // NOTE: We don't validate the fields again because CHANGE_TYPE_COL_NAME should never be + // used anywhere else StructType::new_unchecked(fields).into() } else { physical_schema.clone().into() @@ -72,7 +74,7 @@ pub(crate) fn scan_file_physical_schema( // Get the transform expression for a CDF scan file // -// Returns None when no transformation is needed (identity transform), otherwise returns Some(expr). +// Returns None when no transformation is needed, otherwise returns Some(expr). // // Note: parse_partition_values returns null values for missing partition columns, // and CDF metadata columns (commit_timestamp, commit_version, change_type) are then @@ -97,7 +99,7 @@ pub(crate) fn get_cdf_transform_expr( .map(|ts| ts.as_ref()) .unwrap_or(&empty_spec); - // Return None for identity transforms to avoid unnecessary expression evaluation + // Return None for an empty transform spec to avoid unnecessary expression evaluation. if transform_spec.is_empty() { return Ok(None); } @@ -108,6 +110,7 @@ pub(crate) fn get_cdf_transform_expr( transform_spec, &scan_file.partition_values, state_info.column_mapping_mode, + TimestampTimezone::default(), )?; partition_values.extend(parsed_values); @@ -126,6 +129,9 @@ pub(crate) fn get_cdf_transform_expr( #[cfg(test)] mod tests { + use std::collections::{HashMap, HashSet}; + use std::sync::Arc; + use super::*; use crate::expressions::Expression; use crate::scan::state::DvInfo; @@ -134,8 +140,6 @@ mod tests { use crate::scan::PhysicalPredicate; use crate::schema::{DataType, StructField, StructType}; use crate::table_features::ColumnMappingMode; - use std::collections::HashMap; - use std::sync::Arc; fn create_test_logical_schema() -> SchemaRef { Arc::new(StructType::new_unchecked(vec![ @@ -169,6 +173,8 @@ mod tests { dv_info: DvInfo::default(), remove_dv: None, size: None, + base_row_id: None, + default_row_commit_version: None, } } @@ -186,6 +192,9 @@ mod tests { column_mapping_mode: ColumnMappingMode::None, physical_stats_schema: None, physical_partition_schema: None, + physical_stats_columns: HashSet::new(), + is_catalog_managed: false, + skip_row_transforms: false, } } @@ -218,24 +227,24 @@ mod tests { let expr_opt = result.unwrap(); assert!(expr_opt.is_some(), "Expected Some(expr) but got None"); let expr = expr_opt.unwrap(); - let Expression::Transform(transform) = expr.as_ref() else { - panic!("Expected Transform expression"); + let Expression::StructPatch(patch) = expr.as_ref() else { + panic!("Expected StructPatch expression"); }; - // Should have transform for "id" field with CDF metadata - assert!(transform.field_transforms.contains_key("id")); - let id_transform = &transform.field_transforms["id"]; - assert!(!id_transform.is_replace); - assert_eq!(id_transform.exprs.len(), 2); + // Should have a patch for the "id" field with CDF metadata + assert!(patch.field_patches.contains_key("id")); + let id_patch = &patch.field_patches["id"]; + assert!(id_patch.keep_input); + assert_eq!(id_patch.insertions.len(), 2); // Verify _change_type is "insert" for Add files - let Expression::Literal(change_type) = id_transform.exprs[0].as_ref() else { + let Expression::Literal(change_type) = id_patch.insertions[0].as_ref() else { panic!("Expected literal for _change_type"); }; assert_eq!(change_type, &Scalar::String("insert".to_string())); // Verify _commit_version - let Expression::Literal(version) = id_transform.exprs[1].as_ref() else { + let Expression::Literal(version) = id_patch.insertions[1].as_ref() else { panic!("Expected literal for _commit_version"); }; assert_eq!(version, &Scalar::Long(100)); @@ -264,15 +273,15 @@ mod tests { let expr_opt = result.unwrap(); assert!(expr_opt.is_some(), "Expected Some(expr) but got None"); let expr = expr_opt.unwrap(); - let Expression::Transform(transform) = expr.as_ref() else { - panic!("Expected Transform expression"); + let Expression::StructPatch(patch) = expr.as_ref() else { + panic!("Expected StructPatch expression"); }; - let name_transform = &transform.field_transforms["name"]; - assert_eq!(name_transform.exprs.len(), 1); + let name_patch = &patch.field_patches["name"]; + assert_eq!(name_patch.insertions.len(), 1); // Verify _change_type is "delete" for Remove files - let Expression::Literal(change_type) = name_transform.exprs[0].as_ref() else { + let Expression::Literal(change_type) = name_patch.insertions[0].as_ref() else { panic!("Expected literal for _change_type"); }; assert_eq!(change_type, &Scalar::String("delete".to_string())); @@ -313,14 +322,14 @@ mod tests { let expr_opt = result.unwrap(); assert!(expr_opt.is_some(), "Expected Some(expr) but got None"); let expr = expr_opt.unwrap(); - let Expression::Transform(transform) = expr.as_ref() else { - panic!("Expected Transform expression"); + let Expression::StructPatch(patch) = expr.as_ref() else { + panic!("Expected StructPatch expression"); }; // Should have age partition value - let id_transform = &transform.field_transforms["id"]; - assert_eq!(id_transform.exprs.len(), 1); - let Expression::Literal(age_value) = id_transform.exprs[0].as_ref() else { + let id_patch = &patch.field_patches["id"]; + assert_eq!(id_patch.insertions.len(), 1); + let Expression::Literal(age_value) = id_patch.insertions[0].as_ref() else { panic!("Expected literal for age"); }; assert_eq!(age_value, &Scalar::Long(30)); @@ -328,11 +337,11 @@ mod tests { // For CDC files with DynamicColumn, _change_type is handled as physical // The transform spec has DynamicColumn which becomes a Column expression for CDC files // (not a literal metadata value) - let name_transform = &transform.field_transforms["name"]; - assert_eq!(name_transform.exprs.len(), 1); + let name_patch = &patch.field_patches["name"]; + assert_eq!(name_patch.insertions.len(), 1); // Should be a Column expression, not a Literal assert!(matches!( - name_transform.exprs[0].as_ref(), + name_patch.insertions[0].as_ref(), Expression::Column(_) )); } @@ -373,7 +382,7 @@ mod tests { #[test] fn test_get_cdf_transform_expr_returns_none_for_identity() { // When there's no transform spec and no CDF metadata columns in the schema, - // the function should return None (identity transform) + // the function should return None. let scan_file = CdfScanFile { path: "test/file.parquet".to_string(), partition_values: HashMap::new(), @@ -383,6 +392,8 @@ mod tests { dv_info: DvInfo::default(), remove_dv: None, size: None, + base_row_id: None, + default_row_commit_version: None, }; // Create a simple schema without CDF metadata columns @@ -396,7 +407,7 @@ mod tests { StructField::nullable("name", DataType::STRING), ]); - // Empty transform spec - no transformations needed + // Empty transform spec - no transformation needed. let transform_spec = vec![]; let state_info = StateInfo { @@ -407,6 +418,9 @@ mod tests { column_mapping_mode: ColumnMappingMode::None, physical_stats_schema: None, physical_partition_schema: None, + physical_stats_columns: HashSet::new(), + is_catalog_managed: false, + skip_row_transforms: false, }; let result = get_cdf_transform_expr(&scan_file, &state_info, &physical_schema); @@ -415,7 +429,7 @@ mod tests { let expr_opt = result.unwrap(); assert!( expr_opt.is_none(), - "Expected None for identity transform but got Some(expr)" + "Expected None for empty transform spec but got Some(expr)" ); } } diff --git a/kernel/src/table_changes/resolve_dvs.rs b/kernel/src/table_changes/resolve_dvs.rs index c478166c38..64190e3a80 100644 --- a/kernel/src/table_changes/resolve_dvs.rs +++ b/kernel/src/table_changes/resolve_dvs.rs @@ -13,8 +13,8 @@ pub(crate) struct ResolvedCdfScanFile { pub(crate) scan_file: CdfScanFile, /// Optional vector of bools. If `selection_vector[i] = true`, then that row must be included /// in the CDF output. Otherwise the row must be filtered out. The vector may be shorter than - /// the data file. In this case, all the remaining rows are *not* selected. If `selection_vector` - /// is `None`, then all rows are selected. + /// the data file. In this case, all the remaining rows are *not* selected. If + /// `selection_vector` is `None`, then all rows are selected. pub(crate) selection_vector: Option>, } @@ -22,10 +22,11 @@ pub(crate) struct ResolvedCdfScanFile { /// types of `CdfScanFile`s: /// 1. The first case is a [`CdfScanFile`] paired with a remove deletion vector. The `scan_type` /// must be [`CdfScanFileType::Add`]. In this case, both the add and remove deletion vectors are -/// read if they exist. Then, we find the set of rows that have been added and rows that -/// have been removed. The set of removed rows (if any) will be represented by a +/// read if they exist. Then, we find the set of rows that have been added and rows that have +/// been removed. The set of removed rows (if any) will be represented by a /// [`ResolvedCdfScanFile`] with `scan_type` = [`CdfScanFileType::Remove`]. The set of added rows -/// (if any) will be represented by a [`ResolvedCdfScanFile`] with `scan_type` = [`CdfScanFileType::Add`]. +/// (if any) will be represented by a [`ResolvedCdfScanFile`] with `scan_type` = +/// [`CdfScanFileType::Add`]. /// /// Note: We allow the possibility for there to be both added rows and removed rows for a /// single add/remove pair. @@ -57,18 +58,18 @@ pub(crate) fn resolve_scan_file_dv( (add_dv, Some(rm_dv), CdfScanFileType::Add) => { let add_dv = add_dv.unwrap_or_else(Default::default); let rm_dv = rm_dv.unwrap_or_else(Default::default); - // Here we show how deletion vectors are resolved. Note that logically the `rm_dv` is the - // beginning state of the commit, and `add_dv` is the final state of the commit. In - // other words the dv went from being `rm_dv` to `add_dv`. + // Here we show how deletion vectors are resolved. Note that logically the `rm_dv` is + // the beginning state of the commit, and `add_dv` is the final state of the + // commit. In other words the dv went from being `rm_dv` to `add_dv`. // // ===== IMPORTANT ===== // Both `rm_dv` and `add_dv` are deletion treemaps. We define two types of treemaps: - // - _Deletion_ treemaps (denoted `Treemap_d`) store the indices of deleted rows. - // For instance, `Treemap_d(0, 2)` means that rows 0 and 2 are _deleted_. When - // converted to a vector of bools, it is equivalent to a deletion vector [1, 0, 1]. + // - _Deletion_ treemaps (denoted `Treemap_d`) store the indices of deleted rows. For + // instance, `Treemap_d(0, 2)` means that rows 0 and 2 are _deleted_. When converted + // to a vector of bools, it is equivalent to a deletion vector [1, 0, 1]. // - _Selection_ treemaps (denoted `Treemap_s`) store the indices of selected rows. - // `Treemap_s(0, 1)` means that rows 0 and 1 are _selected_. This is equivalent - // to the selection vector [1, 1, 0] when converted into a boolean vector. + // `Treemap_s(0, 1)` means that rows 0 and 1 are _selected_. This is equivalent to + // the selection vector [1, 1, 0] when converted into a boolean vector. // In ordinary scans, only deletion treemaps are used. However in the case of deletion // vector pairs, we generate selection treemaps. // @@ -98,11 +99,11 @@ pub(crate) fn resolve_scan_file_dv( // The selection treemap shows that row 0 was inserted // // # Deletion Selection Treemap - // The selection vector of deleted rows is calculated using `add_dv - rm_dv`. These rows went - // from unset (present) in `rm_dv` to set (deleted) in the `add_dv`. Once again, all - // other rows are either cancelled in set subtraction, or were not set in either treemap. - // Applying this to our deletion vectors: - // add_dv - rm_dv = + // The selection vector of deleted rows is calculated using `add_dv - rm_dv`. These + // rows went from unset (present) in `rm_dv` to set (deleted) in the + // `add_dv`. Once again, all other rows are either cancelled in set + // subtraction, or were not set in either treemap. Applying this to our + // deletion vectors: add_dv - rm_dv = // Treemap_d(1, 2) // - Treemap_d(0, 1) // = Treemap_s(2) @@ -153,21 +154,20 @@ pub(crate) fn resolve_scan_file_dv( #[cfg(test)] mod tests { - use std::{collections::HashMap, io::Write, path::PathBuf}; + use std::collections::HashMap; + use std::io::Write; + use std::path::PathBuf; use bytes::BufMut; use itertools::Itertools; use roaring::RoaringTreemap; - use crate::{ - actions::deletion_vector::{DeletionVectorDescriptor, DeletionVectorStorageType}, - engine::sync::SyncEngine, - scan::state::DvInfo, - table_changes::scan_file::{CdfScanFile, CdfScanFileType}, - Error, - }; - use super::resolve_scan_file_dv; + use crate::actions::deletion_vector::{DeletionVectorDescriptor, DeletionVectorStorageType}; + use crate::engine::sync::SyncEngine; + use crate::scan::state::DvInfo; + use crate::table_changes::scan_file::{CdfScanFile, CdfScanFileType}; + use crate::Error; fn treemap_to_dv_descriptor(map: RoaringTreemap) -> DeletionVectorDescriptor { let buf = Vec::new(); @@ -200,6 +200,8 @@ mod tests { commit_version: 42, commit_timestamp: 1234, size: None, + base_row_id: None, + default_row_commit_version: None, } } diff --git a/kernel/src/table_changes/scan.rs b/kernel/src/table_changes/scan.rs index 629980147d..26bd09a364 100644 --- a/kernel/src/table_changes/scan.rs +++ b/kernel/src/table_changes/scan.rs @@ -5,20 +5,19 @@ use std::sync::Arc; use itertools::Itertools; use url::Url; +use super::log_replay::{table_changes_action_iter, TableChangesScanMetadata}; +use super::physical_to_logical::{get_cdf_transform_expr, scan_file_physical_schema}; +use super::resolve_dvs::{resolve_scan_file_dv, ResolvedCdfScanFile}; +use super::scan_file::scan_metadata_to_scan_file; +use super::{CdfMode, TableChanges}; use crate::actions::deletion_vector::split_vector; use crate::scan::field_classifiers::CdfTransformFieldClassifier; use crate::scan::state_info::StateInfo; -use crate::scan::PhysicalPredicate; -use crate::scan::StatsOutputMode; +use crate::scan::{PartitionValuesOptions, PhysicalPredicate, StatsOptions}; use crate::schema::SchemaRef; +use crate::utils::FoldWithOption as _; use crate::{DeltaResult, Engine, EngineData, Error, FileMeta, PredicateRef}; -use super::log_replay::{table_changes_action_iter, TableChangesScanMetadata}; -use super::physical_to_logical::{get_cdf_transform_expr, scan_file_physical_schema}; -use super::resolve_dvs::{resolve_scan_file_dv, ResolvedCdfScanFile}; -use super::scan_file::scan_metadata_to_scan_file; -use super::TableChanges; - /// The result of building a [`TableChanges`] scan over a table. This can be used to get the change /// data feed from the table. #[derive(Debug)] @@ -31,9 +30,11 @@ pub struct TableChangesScan { /// This builder constructs a [`TableChangesScan`] that can be used to read the [`TableChanges`] /// of a table. [`TableChangesScanBuilder`] allows you to specify a schema to project the columns -/// or specify a predicate to filter rows in the Change Data Feed. Note that predicates containing Change -/// Data Feed columns `_change_type`, `_commit_version`, and `_commit_timestamp` are not currently -/// allowed. See issue [#525](https://github.com/delta-io/delta-kernel-rs/issues/525). +/// or specify a predicate to filter rows in the Change Data Feed. Predicates referencing the +/// Change Data Feed columns `_change_type`, `_commit_version`, and `_commit_timestamp` are +/// accepted but have no filtering effect, because those columns are synthesized during scan +/// execution rather than read from data files; apply such filters in connector code after the +/// scan returns. See issue [#525](https://github.com/delta-io/delta-kernel-rs/issues/525). /// /// Note: There is a lot of shared functionality between [`TableChangesScanBuilder`] and /// [`ScanBuilder`]. @@ -43,19 +44,19 @@ pub struct TableChangesScan { /// Construct a [`TableChangesScan`] from `table_changes` with a given schema and predicate /// ```rust /// # use std::sync::Arc; -/// # use delta_kernel::expressions::{column_expr, Scalar}; +/// # use delta_kernel::expressions::{col, lit}; /// # use delta_kernel::Predicate; /// # use delta_kernel::table_changes::TableChanges; /// # let path = "./tests/data/table-with-cdf"; /// # let url = delta_kernel::try_parse_uri(path).unwrap(); -/// # use delta_kernel::engine::default::{storage::store_from_url, DefaultEngineBuilder}; +/// # use test_utils::delta_kernel_default_engine::{storage::store_from_url, DefaultEngineBuilder}; /// # let engine = DefaultEngineBuilder::new(store_from_url(&url).unwrap()).build(); /// # let table_changes = TableChanges::try_new(url, &engine, 0, Some(1)).unwrap(); /// let schema = table_changes /// .schema() /// .project(&["id", "_commit_version"]) /// .unwrap(); -/// let predicate = Arc::new(Predicate::gt(column_expr!("id"), Scalar::from(10))); +/// let predicate = Arc::new(Predicate::gt(col!("id"), lit(10))); /// let scan = table_changes /// .into_scan_builder() /// .with_schema(schema) @@ -105,21 +106,33 @@ impl TableChangesScanBuilder { /// /// This does not scan the table at this point, but does do some work to ensure that the /// provided schema make sense, and to prepare some metadata that the scan will need. The - /// [`TableChangesScan`] type itself can be used to fetch the files and associated metadata required to - /// perform actual data reads. + /// [`TableChangesScan`] type itself can be used to fetch the files and associated metadata + /// required to perform actual data reads. pub fn build(self) -> DeltaResult { - // if no schema is provided, use `TableChanges`'s entire (logical) schema (e.g. SELECT *) - let logical_schema = self - .schema - .unwrap_or_else(|| self.table_changes.schema.clone().into()); + // Row-tracking CDF requires row-level reconciliation by row IDs, which this + // scanner does not perform. + if self.table_changes.mode != CdfMode::ChangeDataFeed { + return Err(Error::unsupported( + "A row-tracking TableChanges cannot be scanned for data; use \ + TableChanges::scan_file_listing instead", + )); + } + // Predicates may reference any column in the full CDF-extended schema even when + // `with_schema` narrows the output. Resolve predicate columns against the full schema + // so valid references to unprojected columns aren't rejected. + let table_schema: SchemaRef = self.table_changes.schema().clone().into(); + // If no projection is supplied, default to the full CDF-extended schema (SELECT *). + let logical_read_schema = self.schema.unwrap_or_else(|| table_schema.clone()); // Create StateInfo using CDF field classifier // CDF doesn't support stats output let state_info = StateInfo::try_new( - logical_schema, + logical_read_schema, + table_schema, self.table_changes.end_snapshot.table_configuration(), self.predicate, - StatsOutputMode::default(), + &StatsOptions::default(), + &PartitionValuesOptions::default(), CdfTransformFieldClassifier, )?; @@ -279,7 +292,7 @@ fn read_scan_file( let result = read_result_iter.map(move |batch| -> DeltaResult<_> { let batch = batch?; - // Transform the physical data into the correct logical form, or pass through unchanged + // Transform the physical data into the correct logical form, or pass through unchanged. let logical = if let Some(ref eval) = phys_to_logical_eval { eval.evaluate(batch.as_ref()) } else { @@ -313,14 +326,25 @@ fn read_scan_file( // vector becomes `[1, 0, 1, 1]`. // // # Case 3 - // These scan files are either simple adds, removes, or cdc files. This case is a noop because - // the selection vector is `None`. + // These scan files are either simple adds, removes, or cdc files. This case is a noop + // because the selection vector is `None`. let extend = Some(!is_dv_resolved_pair); let rest = split_vector(sv.as_mut(), len, extend); - let result = match sv { - Some(sv) => logical.and_then(|data| data.apply_selection_vector(sv)), - None => logical, + + // If sv is None here, but we used to have one (is_dv_resolved_pair == true) it means we've + // split the selection vector beyond the range of the original. split_vector will return + // None for rest in that case. When we try and extend that None we still get None, but in + // the CDF case we actually DO want to filter out those rows because it means nothing has + // changed. Hence, in the case that sv==None and is_dv_resolved_pair==true we need a + // full array of false. This only applies to remove/add cases where the associated + // pair may have a selection vector that crosses batch sizes. + let sv = match sv { + None if is_dv_resolved_pair => Some(vec![false; len]), + other => other, }; + let result = logical.fold_with(sv, |logical, sv| { + logical.and_then(|data| data.apply_selection_vector(sv)) + }); selection_vector = rest; result }); @@ -331,13 +355,15 @@ fn read_scan_file( mod tests { use std::sync::Arc; + use crate::committer::FileSystemCommitter; use crate::engine::sync::SyncEngine; - use crate::expressions::{column_expr, Scalar}; + use crate::expressions::{col, lit}; + use crate::object_store::memory::InMemory; use crate::scan::transform_spec::FieldTransformSpec; use crate::scan::PhysicalPredicate; use crate::schema::{DataType, StructField, StructType}; - use crate::table_changes::TableChanges; - use crate::table_changes::COMMIT_VERSION_COL_NAME; + use crate::table_changes::{TableChanges, COMMIT_VERSION_COL_NAME}; + use crate::transaction::create_table::create_table; use crate::Predicate; #[test] @@ -353,7 +379,8 @@ mod tests { // Check that StateInfo has been properly created with correct transforms // Note that this table is not partitioned. `part` is a regular field - // Schema: part (idx 0), id (idx 1), _change_type (idx 2), _commit_version (idx 3), _commit_timestamp (idx 4) + // Schema: part (idx 0), id (idx 1), _change_type (idx 2), _commit_version (idx 3), + // _commit_timestamp (idx 4) assert!(scan.state_info.transform_spec.is_some()); let transform_spec = scan.state_info.transform_spec.as_ref().unwrap(); @@ -410,7 +437,7 @@ mod tests { .schema() .project(&["id", COMMIT_VERSION_COL_NAME]) .unwrap(); - let predicate = Arc::new(Predicate::gt(column_expr!("id"), Scalar::from(10))); + let predicate = Arc::new(Predicate::gt(col!("id"), lit(10))); let scan = table_changes .into_scan_builder() .with_schema(schema.clone()) @@ -458,4 +485,70 @@ mod tests { if pred == &predicate && pred_schema.fields().len() == 1 )); } + + // Regression for issue #2468 on the CDF path + #[test] + fn cdf_predicate_on_unprojected_data_column_in_table_schema_succeeds() { + let url_str = "memory:///test_table/"; + let store = Arc::new(InMemory::new()); + let engine = SyncEngine::new_with_store(store); + + let schema = Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("part", DataType::STRING), + ])); + create_table(url_str, schema, "DefaultEngine") + .with_table_properties([("delta.enableChangeDataFeed", "true")]) + .build(&engine, Box::new(FileSystemCommitter::new())) + .unwrap() + .commit(&engine) + .unwrap() + .unwrap_committed(); + + let url = url::Url::parse(url_str).unwrap(); + // This regression validates predicate resolution at build time, which is independent + // of the CDF version span. A wider range would require a second commit with real + // change data (see LocalMockTable). + let table_changes = TableChanges::try_new(url, &engine, 0, Some(0)).unwrap(); + + // Project only "part"; predicate references unprojected "id". + let schema = table_changes.schema().project(&["part"]).unwrap(); + let predicate = Arc::new(Predicate::gt(col!("id"), lit(10))); + let scan = table_changes + .into_scan_builder() + .with_schema(schema) + .with_predicate(predicate.clone()) + .build() + .unwrap(); + + assert_eq!(scan.logical_schema().fields().len(), 1); + assert!(matches!(&scan.state_info.physical_predicate, + PhysicalPredicate::Some(pred, _) if pred == &predicate)); + } + + // See delta-io/delta-kernel-rs#525 + #[test] + fn cdf_predicate_on_unprojected_cdf_metadata_column_builds_but_is_no_op() { + let path = "./tests/data/table-with-cdf"; + let engine = Box::new(SyncEngine::new()); + let url = delta_kernel::try_parse_uri(path).unwrap(); + let table_changes = TableChanges::try_new(url, engine.as_ref(), 0, Some(1)).unwrap(); + + // Project only "id"; predicate references unprojected `_commit_version`. + let schema = table_changes.schema().project(&["id"]).unwrap(); + let predicate = Arc::new(Predicate::ge(col!("_commit_version"), lit(0_i64))); + let scan = table_changes + .into_scan_builder() + .with_schema(schema) + .with_predicate(predicate) + .build() + .unwrap(); + + // Predicate must pass build-time validation. Downstream behavior (no-op for CDF + // metadata predicates) is enforced elsewhere. + assert!(matches!( + &scan.state_info.physical_predicate, + PhysicalPredicate::Some(_, _) + )); + } } diff --git a/kernel/src/table_changes/scan_file.rs b/kernel/src/table_changes/scan_file.rs index 0bf60c7221..bbe852577d 100644 --- a/kernel/src/table_changes/scan_file.rs +++ b/kernel/src/table_changes/scan_file.rs @@ -1,15 +1,19 @@ -//! This module handles [`CdfScanFile`]s for [`TableChangesScan`]. A [`CdfScanFile`] consists of all the -//! metadata required to generate a change data feed. [`CdfScanFile`] can be constructed using -//! [`CdfScanFileVisitor`]. The visitor reads from engine data with the schema [`cdf_scan_row_schema`]. -//! You can convert engine data to this schema using the [`cdf_scan_row_expression`]. -use itertools::Itertools; +//! This module handles [`CdfScanFile`]s for [`TableChangesScan`]. A [`CdfScanFile`] consists of all +//! the metadata required to generate a change data feed. [`CdfScanFile`] can be constructed using +//! [`CdfScanFileVisitor`]. The visitor reads from engine data with the schema +//! [`cdf_scan_row_schema`]. You can convert engine data to this schema using the +//! [`cdf_scan_row_expression`]. use std::collections::HashMap; use std::sync::{Arc, LazyLock}; +use delta_kernel_derive::internal_api; +use itertools::Itertools; + use super::log_replay::TableChangesScanMetadata; +use crate::actions::deletion_vector::DeletionVectorDescriptor; use crate::actions::visitors::visit_deletion_vector_at; use crate::engine_data::{GetData, TypedGetData}; -use crate::expressions::{column_expr, Expression}; +use crate::expressions::{col, lit, Expression}; use crate::scan::state::DvInfo; use crate::schema::{ ColumnName, ColumnNamesAndTypes, DataType, MapType, SchemaRef, StructField, StructType, @@ -18,7 +22,7 @@ use crate::utils::require; use crate::{DeltaResult, Error, RowVisitor}; // The type of action associated with a [`CdfScanFile`]. -#[derive(Debug, Clone, PartialEq)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(crate) enum CdfScanFileType { Add, Remove, @@ -55,10 +59,143 @@ pub(crate) struct CdfScanFile { pub commit_timestamp: i64, /// The size of the file in bytes pub size: Option, + /// The `baseRowId` from the file action. + pub base_row_id: Option, + /// The `defaultRowCommitVersion` from the file action. + pub default_row_commit_version: Option, } pub(crate) type CdfScanCallback = fn(context: &mut T, scan_file: CdfScanFile); +/// Describes one data file and the metadata required to reconstruct row-level changes. +/// +/// Its deletion vector is the physical vector recorded in the file action. +#[derive(Debug, Clone, PartialEq)] +#[non_exhaustive] +#[internal_api] +pub(crate) struct TableChangesScanFile { + /// Path to the file, relative to the table root. + pub path: String, + + /// The file's physical deletion vector, if any. + pub deletion_vector: Option, + + /// Partition values from the file action. + /// + /// The keys are physical column names when column mapping is enabled. + pub partition_values: HashMap, + + /// The size of the file in bytes, if known. + pub size: Option, + + /// The commit version this action was performed in. + pub commit_version: i64, + + /// The commit timestamp (milliseconds since the epoch). In-commit-timestamp aware when the + /// table has in-commit timestamps enabled; otherwise the commit file's modification time. + pub commit_timestamp: i64, + + /// The base row ID assigned to the file's first physical row. + /// + /// The effective row ID is + /// `coalesce(materialized_row_id, base_row_id + physical_row_index)`, where the physical index + /// is assigned before applying the deletion vector. The physical Parquet column is returned by + /// `TableChanges::materialized_row_id_column_name`. + pub base_row_id: Option, + + /// The first commit version containing an `add` action for this path. + /// + /// This value is not necessarily the last version that updated a row. A row's commit version + /// is derived as `coalesce(materialized_row_commit_version, default_row_commit_version)`. + /// The physical Parquet column is returned by + /// `TableChanges::materialized_row_commit_version_column_name`. + pub default_row_commit_version: Option, +} + +/// Describes the add and remove file images used to reconstruct row-level changes. +/// +/// The remove side is the pre-image and the add side is the post-image. Matching rows by stable row +/// ID and comparing row commit versions distinguishes updates from rows carried forward by a +/// copy-on-write rewrite. Rows present only on the remove side are deletes; rows present only on +/// the add side are inserts. +/// +/// At least one side is present. Copy-on-write actions with different paths remain separate and +/// require reconciliation across the full listing. `AddCDCFile` actions are ignored. +#[derive(Debug, Clone, PartialEq)] +#[non_exhaustive] +#[internal_api] +pub(crate) struct TableChangesFileAction { + /// The add side, present for an insert or the post-image of an update. + pub add: Option, + /// The remove side, present for a delete or the pre-image of an update. + pub remove: Option, +} + +impl TableChangesFileAction { + /// Converts a scan file, preserving both sides of a same-commit deletion-vector update. + pub(crate) fn try_from_scan_file(scan_file: CdfScanFile) -> DeltaResult { + match scan_file.scan_type { + CdfScanFileType::Add => { + require!( + scan_file.base_row_id.is_some(), + Error::missing_data(format!( + "baseRowId for row-tracking add action at path {} in version {}", + scan_file.path, scan_file.commit_version + )) + ); + require!( + scan_file.default_row_commit_version.is_some(), + Error::missing_data(format!( + "defaultRowCommitVersion for row-tracking add action at path {} in \ + version {}", + scan_file.path, scan_file.commit_version + )) + ); + let remove = scan_file.remove_dv.as_ref().map(|dv| { + TableChangesScanFile::from_scan_file(&scan_file, dv.deletion_vector.clone()) + }); + Ok(TableChangesFileAction { + add: Some(TableChangesScanFile::from_scan_file( + &scan_file, + scan_file.dv_info.deletion_vector.clone(), + )), + remove, + }) + } + CdfScanFileType::Remove => Ok(TableChangesFileAction { + add: None, + remove: Some(TableChangesScanFile::from_scan_file( + &scan_file, + scan_file.dv_info.deletion_vector.clone(), + )), + }), + CdfScanFileType::Cdc => Err(Error::internal_error(format!( + "Row-tracking change feed listing unexpectedly produced a cdc scan file: \ + path={}, version={}", + scan_file.path, scan_file.commit_version + ))), + } + } +} + +impl TableChangesScanFile { + fn from_scan_file( + scan_file: &CdfScanFile, + deletion_vector: Option, + ) -> Self { + TableChangesScanFile { + path: scan_file.path.clone(), + deletion_vector, + partition_values: scan_file.partition_values.clone(), + size: scan_file.size, + commit_version: scan_file.commit_version, + commit_timestamp: scan_file.commit_timestamp, + base_row_id: scan_file.base_row_id, + default_row_commit_version: scan_file.default_row_commit_version, + } + } +} + /// Transforms an iterator of [`TableChangesScanMetadata`] into an iterator of /// [`CdfScanFile`] by visiting the engine data. pub(crate) fn scan_metadata_to_scan_file( @@ -78,9 +215,10 @@ pub(crate) fn scan_metadata_to_scan_file( /// scan. /// /// The arguments to the callback are: -/// * `context`: an `&mut context` argument. this can be anything that engine needs to pass through to each call -/// * `CdfScanFile`: a [`CdfScanFile`] struct that holds all the metadata required to perform Change Data -/// Feed +/// * `context`: an `&mut context` argument. this can be anything that engine needs to pass through +/// to each call +/// * `CdfScanFile`: a [`CdfScanFile`] struct that holds all the metadata required to perform Change +/// Data Feed /// /// ## Context /// A note on the `context`. This can be any value the engine wants. This function takes ownership @@ -125,10 +263,84 @@ struct CdfScanFileVisitor<'a, T> { context: T, } +struct FileSideSpec { + scan_type: CdfScanFileType, + start_index: usize, + path_field: &'static str, + partition_values_field: &'static str, + size_field: &'static str, + base_row_id_field: &'static str, + default_row_commit_version_field: &'static str, +} + +const ADD_FILE_SIDE: FileSideSpec = FileSideSpec { + scan_type: CdfScanFileType::Add, + start_index: 0, + path_field: "scanFile.add.path", + partition_values_field: "scanFile.add.fileConstantValues.partitionValues", + size_field: "scanFile.add.size", + base_row_id_field: "scanFile.add.baseRowId", + default_row_commit_version_field: "scanFile.add.defaultRowCommitVersion", +}; +const REMOVE_FILE_SIDE: FileSideSpec = FileSideSpec { + scan_type: CdfScanFileType::Remove, + start_index: 10, + path_field: "scanFile.remove.path", + partition_values_field: "scanFile.remove.fileConstantValues.partitionValues", + size_field: "scanFile.remove.size", + base_row_id_field: "scanFile.remove.baseRowId", + default_row_commit_version_field: "scanFile.remove.defaultRowCommitVersion", +}; +const CDC_PATH_INDEX: usize = 20; +const CDC_PARTITION_VALUES_INDEX: usize = 21; +const CDC_SIZE_INDEX: usize = 22; +const COMMIT_TIMESTAMP_INDEX: usize = 23; +const COMMIT_VERSION_INDEX: usize = 24; +const CDF_SCAN_FILE_GETTER_COUNT: usize = 25; + +struct FileSide { + scan_type: CdfScanFileType, + path: String, + deletion_vector: Option, + partition_values: Option>, + size: Option, + base_row_id: Option, + default_row_commit_version: Option, +} + +fn read_file_side<'a>( + row_index: usize, + getters: &[&'a dyn GetData<'a>], + spec: &FileSideSpec, +) -> DeltaResult> { + let Some(path) = getters[spec.start_index].get_opt(row_index, spec.path_field)? else { + return Ok(None); + }; + let deletion_vector = visit_deletion_vector_at( + row_index, + &getters[spec.start_index + 1..=spec.start_index + 5], + )?; + let partition_values = + getters[spec.start_index + 6].get_opt(row_index, spec.partition_values_field)?; + let size = getters[spec.start_index + 7].get_opt(row_index, spec.size_field)?; + let base_row_id = getters[spec.start_index + 8].get_opt(row_index, spec.base_row_id_field)?; + let default_row_commit_version = + getters[spec.start_index + 9].get_opt(row_index, spec.default_row_commit_version_field)?; + Ok(Some(FileSide { + scan_type: spec.scan_type, + path, + deletion_vector, + partition_values, + size, + base_row_id, + default_row_commit_version, + })) +} + impl RowVisitor for CdfScanFileVisitor<'_, T> { fn visit<'a>(&mut self, row_count: usize, getters: &[&'a dyn GetData<'a>]) -> DeltaResult<()> { require!( - getters.len() == 21, + getters.len() == CDF_SCAN_FILE_GETTER_COUNT, Error::InternalError(format!( "Wrong number of CdfScanFileVisitor getters: {}", getters.len() @@ -139,42 +351,44 @@ impl RowVisitor for CdfScanFileVisitor<'_, T> { continue; } - let (scan_type, path, deletion_vector, partition_values, size) = - if let Some(path) = getters[0].get_opt(row_index, "scanFile.add.path")? { - let scan_type = CdfScanFileType::Add; - let deletion_vector = visit_deletion_vector_at(row_index, &getters[1..=5])?; - let partition_values = getters[6] - .get_opt(row_index, "scanFile.add.fileConstantValues.partitionValues")?; - let size = getters[7].get_opt(row_index, "scanFile.add.size")?; - (scan_type, path, deletion_vector, partition_values, size) - } else if let Some(path) = getters[8].get_opt(row_index, "scanFile.remove.path")? { - let scan_type = CdfScanFileType::Remove; - let deletion_vector = visit_deletion_vector_at(row_index, &getters[9..=13])?; - let partition_values = getters[14].get_opt( - row_index, - "scanFile.remove.fileConstantValues.partitionValues", - )?; - let size = getters[15].get_opt(row_index, "scanFile.remove.size")?; - (scan_type, path, deletion_vector, partition_values, size) - } else if let Some(path) = getters[16].get_opt(row_index, "scanFile.cdc.path")? { - let scan_type = CdfScanFileType::Cdc; - let partition_values = getters[17] - .get_opt(row_index, "scanFile.cdc.fileConstantValues.partitionValues")?; - let size = getters[18].get_opt(row_index, "scanFile.cdc.size")?; - (scan_type, path, None, partition_values, size) - } else { - continue; - }; - let partition_values = partition_values.unwrap_or_else(Default::default); + let file_side = if let Some(side) = read_file_side(row_index, getters, &ADD_FILE_SIDE)? + { + side + } else if let Some(side) = read_file_side(row_index, getters, &REMOVE_FILE_SIDE)? { + side + } else if let Some(path) = + getters[CDC_PATH_INDEX].get_opt(row_index, "scanFile.cdc.path")? + { + let partition_values = getters[CDC_PARTITION_VALUES_INDEX] + .get_opt(row_index, "scanFile.cdc.fileConstantValues.partitionValues")?; + let size = getters[CDC_SIZE_INDEX].get_opt(row_index, "scanFile.cdc.size")?; + FileSide { + scan_type: CdfScanFileType::Cdc, + path, + deletion_vector: None, + partition_values, + size, + base_row_id: None, + default_row_commit_version: None, + } + } else { + continue; + }; let scan_file = CdfScanFile { - remove_dv: self.remove_dvs.get(&path).cloned(), - scan_type, - path, - dv_info: DvInfo { deletion_vector }, - partition_values, - commit_timestamp: getters[19].get(row_index, "scanFile.timestamp")?, - commit_version: getters[20].get(row_index, "scanFile.commit_version")?, - size, + remove_dv: self.remove_dvs.get(&file_side.path).cloned(), + scan_type: file_side.scan_type, + path: file_side.path, + dv_info: DvInfo { + deletion_vector: file_side.deletion_vector, + }, + partition_values: file_side.partition_values.unwrap_or_default(), + commit_timestamp: getters[COMMIT_TIMESTAMP_INDEX] + .get(row_index, "scanFile.timestamp")?, + commit_version: getters[COMMIT_VERSION_INDEX] + .get(row_index, "scanFile.commit_version")?, + size: file_side.size, + base_row_id: file_side.base_row_id, + default_row_commit_version: file_side.default_row_commit_version, }; (self.callback)(&mut self.context, scan_file) } @@ -207,12 +421,16 @@ pub(crate) fn cdf_scan_row_schema() -> SchemaRef { StructField::nullable("deletionVector", deletion_vector.clone()), StructField::nullable("fileConstantValues", file_constant_values.clone()), StructField::nullable("size", DataType::LONG), + StructField::nullable("baseRowId", DataType::LONG), + StructField::nullable("defaultRowCommitVersion", DataType::LONG), ]); let remove = StructType::new_unchecked([ StructField::nullable("path", DataType::STRING), StructField::nullable("deletionVector", deletion_vector), StructField::nullable("fileConstantValues", file_constant_values.clone()), StructField::nullable("size", DataType::LONG), + StructField::nullable("baseRowId", DataType::LONG), + StructField::nullable("defaultRowCommitVersion", DataType::LONG), ]); let cdc = StructType::new_unchecked([ StructField::nullable("path", DataType::STRING), @@ -236,24 +454,28 @@ pub(crate) fn cdf_scan_row_schema() -> SchemaRef { pub(crate) fn cdf_scan_row_expression(commit_timestamp: i64, commit_number: i64) -> Expression { Expression::struct_from([ Expression::struct_from([ - column_expr!("add.path"), - column_expr!("add.deletionVector"), - Expression::struct_from([column_expr!("add.partitionValues")]), - column_expr!("add.size"), + col!("add.path"), + col!("add.deletionVector"), + Expression::struct_from([col!("add.partitionValues")]), + col!("add.size"), + col!("add.baseRowId"), + col!("add.defaultRowCommitVersion"), ]), Expression::struct_from([ - column_expr!("remove.path"), - column_expr!("remove.deletionVector"), - Expression::struct_from([column_expr!("remove.partitionValues")]), - column_expr!("remove.size"), + col!("remove.path"), + col!("remove.deletionVector"), + Expression::struct_from([col!("remove.partitionValues")]), + col!("remove.size"), + col!("remove.baseRowId"), + col!("remove.defaultRowCommitVersion"), ]), Expression::struct_from([ - column_expr!("cdc.path"), - Expression::struct_from([column_expr!("cdc.partitionValues")]), - column_expr!("cdc.size"), + col!("cdc.path"), + Expression::struct_from([col!("cdc.partitionValues")]), + col!("cdc.size"), ]), - Expression::literal(commit_timestamp), - Expression::literal(commit_number), + lit(commit_timestamp), + lit(commit_number), ]) } @@ -263,18 +485,91 @@ mod tests { use std::sync::Arc; use itertools::Itertools; + use rstest::rstest; - use super::{scan_metadata_to_scan_file, CdfScanFile, CdfScanFileType}; + use super::{scan_metadata_to_scan_file, CdfScanFile, CdfScanFileType, TableChangesFileAction}; use crate::actions::deletion_vector::{DeletionVectorDescriptor, DeletionVectorStorageType}; - use crate::actions::{Add, Cdc, Remove}; + use crate::actions::{Add, Cdc, Metadata, Protocol, Remove}; use crate::engine::sync::SyncEngine; use crate::log_segment::LogSegment; use crate::scan::state::DvInfo; use crate::schema::{DataType, StructField, StructType}; - use crate::table_changes::log_replay::table_changes_action_iter; - use crate::utils::test_utils::{Action, LocalMockTable}; + use crate::table_changes::log_replay::{ + table_changes_action_iter, table_changes_action_iter_with_mode, + }; + use crate::table_changes::test_utils::{row_tracking_table_config, test_deletion_vector}; + use crate::table_changes::CdfMode; + use crate::table_configuration::TableConfiguration; + use crate::table_properties::{COLUMN_MAPPING_MODE, ENABLE_CHANGE_DATA_FEED}; + use crate::unit_test_utils::{assert_result_error_with_message, Action, LocalMockTable}; use crate::Engine as _; + fn row_tracking_add( + path: &str, + deletion_vector: Option, + data_change: bool, + base_row_id: i64, + default_row_commit_version: i64, + ) -> Add { + Add { + path: path.into(), + deletion_vector, + partition_values: HashMap::new(), + data_change, + size: 100, + base_row_id: Some(base_row_id), + default_row_commit_version: Some(default_row_commit_version), + ..Default::default() + } + } + + fn row_tracking_remove( + path: &str, + deletion_vector: Option, + base_row_id: Option, + default_row_commit_version: Option, + ) -> Remove { + Remove { + path: path.into(), + deletion_vector, + data_change: true, + base_row_id, + default_row_commit_version, + ..Default::default() + } + } + + fn row_tracking_scan_files( + engine: Arc, + mock_table: &LocalMockTable, + ) -> Vec { + let table_root = url::Url::from_directory_path(mock_table.table_root()).unwrap(); + let log_segment = LogSegment::for_table_changes( + engine.storage_handler().as_ref(), + table_root.join("_delta_log/").unwrap(), + 0, + None, + ) + .unwrap(); + let table_schema = Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("value", DataType::STRING), + ])); + let table_config = row_tracking_table_config(table_root, table_schema.clone()); + let scan_metadata = table_changes_action_iter_with_mode( + engine, + &table_config, + log_segment.listed.ascending_commit_files, + table_schema, + None, + CdfMode::RowTracking, + ) + .unwrap(); + scan_metadata_to_scan_file(scan_metadata) + .try_collect() + .unwrap() + } + #[tokio::test] async fn test_scan_file_visiting() { let engine = SyncEngine::new(); @@ -296,7 +591,7 @@ mod tests { size: 100i64, ..Default::default() }; - let remove_paired = Remove { + let paired_remove = Remove { path: "fake_path_1".into(), deletion_vector: None, partition_values: None, @@ -340,7 +635,7 @@ mod tests { mock_table .commit([ - Action::Remove(remove_paired.clone()), + Action::Remove(paired_remove.clone()), Action::Add(add_paired.clone()), Action::Remove(remove.clone()), ]) @@ -361,10 +656,6 @@ mod tests { ])); // Create a TableConfiguration for testing - use crate::actions::{Metadata, Protocol}; - use crate::table_configuration::TableConfiguration; - use crate::table_properties::{COLUMN_MAPPING_MODE, ENABLE_CHANGE_DATA_FEED}; - let metadata = Metadata::try_new( None, None, @@ -416,6 +707,8 @@ mod tests { commit_timestamp: timestamps[0], remove_dv: Some(expected_remove_dv), size: Some(add_paired.size), + base_row_id: None, + default_row_commit_version: None, }, CdfScanFile { scan_type: CdfScanFileType::Remove, @@ -428,6 +721,8 @@ mod tests { commit_timestamp: timestamps[0], remove_dv: None, size: remove.size, + base_row_id: None, + default_row_commit_version: None, }, CdfScanFile { scan_type: CdfScanFileType::Cdc, @@ -440,6 +735,8 @@ mod tests { commit_timestamp: timestamps[1], remove_dv: None, size: Some(cdc.size), + base_row_id: None, + default_row_commit_version: None, }, CdfScanFile { scan_type: CdfScanFileType::Remove, @@ -452,9 +749,216 @@ mod tests { commit_timestamp: timestamps[2], remove_dv: None, size: remove_no_partition.size, + base_row_id: None, + default_row_commit_version: None, }, ]; assert_eq!(scan_files, expected_scan_files); } + + #[tokio::test] + async fn test_row_tracking_listing_ignores_cdc_and_surfaces_row_tracking_fields() { + let engine = Arc::new(SyncEngine::new()); + let mut mock_table = LocalMockTable::new(); + + let post_dv = test_deletion_vector("vBn[lx{q8@P<9BNH/isA", 2); + let baseline_dv = test_deletion_vector("U5OWRz5k%CFT.Td}yCPW", 1); + let add_paired = row_tracking_add("path_1", Some(post_dv.clone()), true, 10, 0); + let paired_remove = + row_tracking_remove("path_1", Some(baseline_dv.clone()), Some(10), Some(0)); + let remove_unpaired = row_tracking_remove("path_2", None, None, None); + let cdc = Cdc { + path: "cdc_path".into(), + partition_values: HashMap::new(), + ..Default::default() + }; + let add_insert = row_tracking_add("path_4", None, true, 20, 2); + let add_no_data_change = row_tracking_add("path_6", None, false, 40, 3); + + mock_table + .commit([ + Action::Remove(paired_remove), + Action::Add(add_paired), + Action::Remove(remove_unpaired), + ]) + .await; + mock_table.commit([Action::Cdc(cdc)]).await; + mock_table.commit([Action::Add(add_insert)]).await; + mock_table.commit([Action::Add(add_no_data_change)]).await; + + let scan_files = row_tracking_scan_files(engine, &mock_table); + + // The cdc action in commit 1 is ignored: only add/remove-derived files appear. The + // metadata-only add (`data_change == false`) in commit 3 is also excluded. + assert!(scan_files + .iter() + .all(|f| f.scan_type != CdfScanFileType::Cdc)); + assert!( + !scan_files.iter().any(|f| f.path == "path_6"), + "metadata-only add (data_change=false) must be excluded" + ); + assert_eq!(scan_files.len(), 3); + + // The paired add (a DV update) carries its own (post-image) DV, the paired remove's + // (baseline) DV, and the row-tracking fields. + let paired = scan_files + .iter() + .find(|f| f.path == "path_1") + .expect("paired add present"); + assert_eq!(paired.scan_type, CdfScanFileType::Add); + assert_eq!(paired.base_row_id, Some(10)); + assert_eq!(paired.default_row_commit_version, Some(0)); + assert_eq!(paired.dv_info.deletion_vector, Some(post_dv)); + assert_eq!( + paired + .remove_dv + .as_ref() + .and_then(|dv| dv.deletion_vector.clone()), + Some(baseline_dv) + ); + + // The unpaired remove is a delete and has no paired remove DV. + let unpaired = scan_files + .iter() + .find(|f| f.path == "path_2") + .expect("unpaired remove present"); + assert_eq!(unpaired.scan_type, CdfScanFileType::Remove); + assert_eq!(unpaired.remove_dv, None); + assert_eq!(unpaired.base_row_id, None); + assert_eq!(unpaired.default_row_commit_version, None); + + // The standalone add is an insert and surfaces its base row id. + let insert = scan_files + .iter() + .find(|f| f.path == "path_4") + .expect("insert add present"); + assert_eq!(insert.scan_type, CdfScanFileType::Add); + assert_eq!(insert.base_row_id, Some(20)); + + // Converting to the public grouped type never errors (no cdc) and preserves the metadata. + // Group by path via the add/remove side that carries it. + let listing: Vec = scan_files + .into_iter() + .map(TableChangesFileAction::try_from_scan_file) + .try_collect() + .unwrap(); + let path_of = |fa: &TableChangesFileAction| { + fa.add + .as_ref() + .or(fa.remove.as_ref()) + .map(|s| s.path.clone()) + }; + + // The paired DV update (path_1) groups into both sides for the one file: the add side + // carries the post-image DV, the remove side the baseline DV, each with the row-tracking + // fields. + let paired = listing + .iter() + .find(|fa| path_of(fa).as_deref() == Some("path_1")) + .expect("paired listing present"); + let paired_add = paired.add.as_ref().expect("paired add side"); + let paired_remove = paired.remove.as_ref().expect("paired remove side"); + assert_eq!(paired_add.base_row_id, Some(10)); + assert!(paired_add.deletion_vector.is_some()); + assert!(paired_remove.deletion_vector.is_some()); + assert_eq!(paired_remove.base_row_id, Some(10)); + + // The unpaired remove (path_2) is a single-sided delete. + let unpaired_delete = listing + .iter() + .find(|fa| path_of(fa).as_deref() == Some("path_2")) + .expect("unpaired delete present"); + assert!(unpaired_delete.add.is_none()); + assert!(unpaired_delete.remove.is_some()); + + // The standalone add (path_4) is a single-sided insert. + let insert = listing + .iter() + .find(|fa| path_of(fa).as_deref() == Some("path_4")) + .expect("insert present"); + assert!(insert.remove.is_none()); + assert_eq!( + insert.add.as_ref().expect("insert add side").base_row_id, + Some(20) + ); + } + + #[rstest] + #[case::with_deletion_vector(Some(test_deletion_vector("baseline", 1)))] + #[case::without_deletion_vector(None)] + fn cdf_file_action_preserves_paired_remove_deletion_vector( + #[case] deletion_vector: Option, + ) { + let scan_file = CdfScanFile { + scan_type: CdfScanFileType::Add, + path: "path".into(), + dv_info: DvInfo { + deletion_vector: Some(test_deletion_vector("current", 2)), + }, + remove_dv: Some(DvInfo { + deletion_vector: deletion_vector.clone(), + }), + partition_values: HashMap::new(), + commit_version: 1, + commit_timestamp: 2, + size: Some(3), + base_row_id: Some(4), + default_row_commit_version: Some(5), + }; + + let action = TableChangesFileAction::try_from_scan_file(scan_file).unwrap(); + assert_eq!(action.remove.unwrap().deletion_vector, deletion_vector); + assert!(action.add.is_some()); + } + + #[rstest] + #[case::missing_base_row_id(None, Some(5), "baseRowId")] + #[case::missing_default_row_commit_version(Some(4), None, "defaultRowCommitVersion")] + fn cdf_file_action_rejects_add_missing_row_tracking_fields( + #[case] base_row_id: Option, + #[case] default_row_commit_version: Option, + #[case] expected: &str, + ) { + let scan_file = CdfScanFile { + scan_type: CdfScanFileType::Add, + path: "path".into(), + dv_info: DvInfo { + deletion_vector: None, + }, + remove_dv: None, + partition_values: HashMap::new(), + commit_version: 1, + commit_timestamp: 2, + size: Some(3), + base_row_id, + default_row_commit_version, + }; + + assert_result_error_with_message( + TableChangesFileAction::try_from_scan_file(scan_file), + expected, + ); + } + + #[test] + fn cdf_file_action_rejects_cdc_scan_file() { + // The row-tracking listing path never selects cdc files, so converting one signals an + // internal inconsistency and must error rather than silently producing a bad listing. + let cdc_scan_file = CdfScanFile { + scan_type: CdfScanFileType::Cdc, + path: "cdc".into(), + dv_info: DvInfo { + deletion_vector: None, + }, + remove_dv: None, + partition_values: HashMap::new(), + commit_version: 0, + commit_timestamp: 0, + size: None, + base_row_id: None, + default_row_commit_version: None, + }; + assert!(TableChangesFileAction::try_from_scan_file(cdc_scan_file).is_err()); + } } diff --git a/kernel/src/table_changes/test_utils.rs b/kernel/src/table_changes/test_utils.rs new file mode 100644 index 0000000000..b733a0e634 --- /dev/null +++ b/kernel/src/table_changes/test_utils.rs @@ -0,0 +1,75 @@ +//! Shared fixtures for table-change unit tests. + +use std::collections::HashMap; + +use url::Url; + +use crate::actions::deletion_vector::{DeletionVectorDescriptor, DeletionVectorStorageType}; +use crate::actions::{Metadata, Protocol}; +use crate::schema::SchemaRef; +use crate::table_configuration::TableConfiguration; +use crate::table_features::TableFeature; +use crate::table_properties::{ + ENABLE_ROW_TRACKING, MATERIALIZED_ROW_COMMIT_VERSION_COLUMN_NAME, + MATERIALIZED_ROW_ID_COLUMN_NAME, +}; +use crate::unit_test_utils::Action; + +pub(crate) const TEST_MATERIALIZED_ROW_ID_COLUMN_NAME: &str = "_row_id"; +pub(crate) const TEST_MATERIALIZED_ROW_COMMIT_VERSION_COLUMN_NAME: &str = "_row_commit_version"; + +pub(crate) fn row_tracking_properties() -> HashMap { + HashMap::from([ + (ENABLE_ROW_TRACKING.to_string(), "true".to_string()), + ( + MATERIALIZED_ROW_ID_COLUMN_NAME.to_string(), + TEST_MATERIALIZED_ROW_ID_COLUMN_NAME.to_string(), + ), + ( + MATERIALIZED_ROW_COMMIT_VERSION_COLUMN_NAME.to_string(), + TEST_MATERIALIZED_ROW_COMMIT_VERSION_COLUMN_NAME.to_string(), + ), + ]) +} + +pub(crate) fn row_tracking_metadata(schema: SchemaRef) -> Metadata { + Metadata::try_new(None, None, schema, vec![], 0, row_tracking_properties()).unwrap() +} + +pub(crate) fn row_tracking_protocol() -> Protocol { + Protocol::try_new_modern( + TableFeature::EMPTY_LIST, + [TableFeature::RowTracking, TableFeature::DomainMetadata], + ) + .unwrap() +} + +pub(crate) fn row_tracking_setup_actions(schema: SchemaRef) -> [Action; 2] { + [ + Action::Protocol(row_tracking_protocol()), + Action::Metadata(row_tracking_metadata(schema)), + ] +} + +pub(crate) fn row_tracking_table_config(table_root: Url, schema: SchemaRef) -> TableConfiguration { + TableConfiguration::try_new( + row_tracking_metadata(schema), + row_tracking_protocol(), + table_root, + 0, + ) + .unwrap() +} + +pub(crate) fn test_deletion_vector( + path: impl Into, + cardinality: i64, +) -> DeletionVectorDescriptor { + DeletionVectorDescriptor { + storage_type: DeletionVectorStorageType::PersistedRelative, + path_or_inline_dv: path.into(), + offset: Some(1), + size_in_bytes: 36, + cardinality, + } +} diff --git a/kernel/src/table_configuration.rs b/kernel/src/table_configuration.rs index 3123bf4747..2af893a991 100644 --- a/kernel/src/table_configuration.rs +++ b/kernel/src/table_configuration.rs @@ -12,6 +12,8 @@ use std::borrow::Cow; use std::collections::HashSet; use std::sync::Arc; +use delta_kernel_derive::internal_api; +use tracing::warn; use url::Url; use crate::actions::{Metadata, Protocol}; @@ -20,20 +22,24 @@ use crate::scan::data_skipping::stats_schema::{ expected_stats_schema, stats_column_names, StatsConfig, StripFieldMetadataTransform, }; pub(crate) use crate::schema::variant_utils::validate_variant_type_feature_support; -use crate::schema::{schema_has_invariants, SchemaRef, StructField, StructType}; +use crate::schema::void_utils::strip_void_from_schema; +use crate::schema::{ + schema_has_invariants, validate_column_defaults_metadata, SchemaRef, StructField, StructType, +}; +#[cfg(feature = "geo-type-in-dev")] +use crate::table_features::validate_geospatial_feature_support; use crate::table_features::{ - column_mapping_mode, get_any_level_column_physical_name, + check_reader_version_range, column_mapping_mode, extract_enabled_reader_features, + get_any_level_column_physical_name, validate_iceberg_compat_if_needed, validate_timestamp_ntz_feature_support, ColumnMappingMode, EnablementCheck, FeatureRequirement, - FeatureType, KernelSupport, Operation, TableFeature, LEGACY_READER_FEATURES, - LEGACY_WRITER_FEATURES, MAX_VALID_READER_VERSION, MAX_VALID_WRITER_VERSION, - TABLE_FEATURES_MIN_READER_VERSION, TABLE_FEATURES_MIN_WRITER_VERSION, + FeatureType, KernelSupport, Operation, TableFeature, LEGACY_WRITER_FEATURES, + MAX_VALID_WRITER_VERSION, MIN_VALID_RW_VERSION, TABLE_FEATURES_MIN_READER_VERSION, + TABLE_FEATURES_MIN_WRITER_VERSION, V3_VALIDATOR, }; use crate::table_properties::TableProperties; use crate::transforms::SchemaTransform as _; use crate::utils::require; use crate::{DeltaResult, Error, Version}; -use delta_kernel_derive::internal_api; -use tracing::warn; /// Expected schema for file statistics, using physical column names. /// @@ -64,11 +70,27 @@ pub(crate) enum InCommitTimestampEnablement { /// stats from a checkpoint written before logical metadata was added. fn strip_metadata(schema: SchemaRef) -> SchemaRef { match StripFieldMetadataTransform.transform_struct(&schema) { - Some(Cow::Owned(s)) => Arc::new(s), + Cow::Owned(s) => Arc::new(s), _ => schema, } } +fn validate_partition_columns(metadata: &Metadata, logical_schema: &StructType) -> DeltaResult<()> { + let mut seen = HashSet::new(); + for col in metadata.partition_columns() { + if !seen.insert(col) { + return Err(Error::generic(format!( + "Duplicate partition column: '{col}'" + ))); + } + require!( + logical_schema.field(col).is_some(), + Error::generic(format!("Partition column '{col}' not found in schema")) + ); + } + Ok(()) +} + /// Holds all the configuration for a table at a specific version. This includes the supported /// reader and writer features, table properties, schema, version, and table root. This can be used /// to check whether a table supports a feature or has it enabled. For example, deletion vector @@ -86,9 +108,14 @@ pub(crate) struct TableConfiguration { protocol: Protocol, /// Logical schema: field names are the user-facing (logical) column names. logical_schema: SchemaRef, - /// Physical schema: field names are the physical column names (same as logical when - /// `ColumnMappingMode::None`, otherwise derived from column mapping metadata). + /// Whether any field in the logical schema declares a column default. + has_column_with_default: bool, + /// The subset of the logical schema that remains after excluding partition columns. + logical_schema_without_partition_columns: SchemaRef, + /// Physical schema for all columns (field names respect column mapping mode). physical_schema: SchemaRef, + /// The subset of the physical schema that remains after excluding partition columns. + physical_data_schema_without_partition_columns: SchemaRef, table_properties: TableProperties, column_mapping_mode: ColumnMappingMode, table_root: Url, @@ -107,8 +134,8 @@ impl TableConfiguration { /// /// Note: In the future, we will perform stricter checks on the set of reader and writer /// features. In particular, we will check that: - /// - Non-legacy features must appear in both reader features and writer features lists. - /// If such a feature is present, the reader version and writer version must be 3, and 5 + /// - Non-legacy features must appear in both reader features and writer features lists. If + /// such a feature is present, the reader version and writer version must be 3, and 5 /// respectively. /// - Legacy reader features occur when the reader version is 3, but the writer version is /// either 5 or 6. In this case, the writer feature list must be empty. @@ -123,14 +150,67 @@ impl TableConfiguration { version: Version, ) -> DeltaResult { let logical_schema = Arc::new(metadata.parse_schema()?); + Self::try_new_inner(metadata, protocol, table_root, version, logical_schema) + } + + /// Like [`try_new`](Self::try_new), but reuses `base`'s protocol, table root, and version + /// and takes a pre-parsed `logical_schema`. + pub(crate) fn try_new_with_schema( + base: &Self, + metadata: Metadata, + logical_schema: SchemaRef, + ) -> DeltaResult { + Self::try_new_inner( + metadata, + base.protocol.clone(), + base.table_root.clone(), + base.version, + logical_schema, + ) + } + + fn try_new_inner( + metadata: Metadata, + protocol: Protocol, + table_root: Url, + version: Version, + logical_schema: SchemaRef, + ) -> DeltaResult { let table_properties = metadata.parse_table_properties(); let column_mapping_mode = column_mapping_mode(&protocol, &table_properties); let physical_schema = Arc::new(logical_schema.make_physical(column_mapping_mode)?); + let partition_columns: HashSet<&str> = metadata + .partition_columns() + .iter() + .map(|s| s.as_str()) + .collect(); + let physical_data_schema_without_partition_columns = { + let fields = logical_schema + .fields() + .zip(physical_schema.fields()) + .filter(|(logical_field, _)| { + !partition_columns.contains(logical_field.name().as_str()) + }) + .map(|(_, physical_field)| physical_field.clone()); + // Safety: subset of an already-valid schema. + Arc::new(StructType::new_unchecked(fields)) + }; + let logical_schema_without_partition_columns = { + let fields = logical_schema + .fields() + .filter(|field| !partition_columns.contains(field.name().as_str())) + .cloned(); + // Safety: subset of an already-valid schema. + Arc::new(StructType::new_unchecked(fields)) + }; - let table_config = Self { + let mut table_config = Self { logical_schema, + has_column_with_default: false, + logical_schema_without_partition_columns, physical_schema, + physical_data_schema_without_partition_columns, metadata, protocol, table_properties, @@ -139,9 +219,20 @@ impl TableConfiguration { version, }; + validate_partition_columns(&table_config.metadata, &table_config.logical_schema)?; + // Validate schema against protocol features now that we have a TC instance. validate_timestamp_ntz_feature_support(&table_config)?; validate_variant_type_feature_support(&table_config)?; + // Reject corrupt column-default metadata (a non-string `CURRENT_DEFAULT`, or a non-`NULL` + // default on a Variant column) and retain whether the validated schema declares any column + // defaults. + table_config.has_column_with_default = + validate_column_defaults_metadata(&table_config.logical_schema)?; + // Reject tables with geo-typed columns that don't declare the `geospatial` feature. + #[cfg(feature = "geo-type-in-dev")] + validate_geospatial_feature_support(&table_config)?; + validate_iceberg_compat_if_needed(&table_config, &V3_VALIDATOR)?; Ok(table_config) } @@ -174,17 +265,25 @@ impl TableConfiguration { /// Creates a new [`TableConfiguration`] representing the table configuration immediately /// after a commit. /// - /// This method takes a pre-commit table configuration and produces a post-commit - /// configuration at the committed version. This allows immediate use of the new table - /// configuration without re-reading metadata from storage. + /// This method takes the current table configuration and produces a post-commit + /// configuration at the committed version. If the commit included new Protocol or Metadata + /// actions (e.g. ALTER TABLE), those are passed in and the configuration is rebuilt with + /// full validation. Otherwise the existing configuration is cloned with only the version + /// updated. /// - /// TODO: Take in Protocol (when Kernel-RS supports protocol changes) - /// TODO: Take in Metadata (when Kernel-RS supports metadata changes) - pub(crate) fn new_post_commit(table_configuration: &Self, new_version: Version) -> Self { - Self { - version: new_version, - ..table_configuration.clone() - } + /// Returns the new [`TableConfiguration`] at `new_version`. + /// + /// # Errors + /// + /// Returns an error if the new metadata/protocol combination fails + /// [`TableConfiguration::try_new`] validation (e.g., unsupported features, invalid schema). + pub(crate) fn new_post_commit( + table_configuration: &Self, + new_version: Version, + new_metadata: Option, + new_protocol: Option, + ) -> DeltaResult { + Self::try_new_from(table_configuration, new_metadata, new_protocol, new_version) } /// Generates the expected schema for file statistics. @@ -209,8 +308,8 @@ impl TableConfiguration { /// - **`delta.dataSkippingStatsColumns`**: If set, only specified columns are included. /// - **`delta.dataSkippingNumIndexedCols`**: Otherwise, includes the first N leaf columns /// (default 32). - /// - **Required columns** (e.g. clustering columns): Per the Delta protocol, always included - /// in statistics, regardless of the above settings. + /// - **Required columns** (e.g. clustering columns): Per the Delta protocol, always included in + /// statistics, regardless of the above settings. /// - **Requested columns**: Optional output filter that limits which columns appear in the /// schema without affecting column counting. /// @@ -220,11 +319,11 @@ impl TableConfiguration { #[internal_api] pub(crate) fn build_expected_stats_schemas( &self, - required_columns_physical: Option<&[ColumnName]>, - requested_columns_physical: Option<&[ColumnName]>, + required_physical_columns: Option<&[ColumnName]>, + requested_physical_columns: Option<&[ColumnName]>, ) -> DeltaResult { let physical_data_schema = self.physical_data_schema_without_partition_columns(); - let required_physical_stats_columns = self.required_stats_columns_physical(); + let required_physical_stats_columns = self.required_physical_stats_columns(); let config = StatsConfig { data_skipping_stats_columns: required_physical_stats_columns.as_deref(), data_skipping_num_indexed_cols: self.table_properties().data_skipping_num_indexed_cols, @@ -232,8 +331,8 @@ impl TableConfiguration { let physical_stats_schema = Arc::new(expected_stats_schema( &physical_data_schema, &config, - required_columns_physical, - requested_columns_physical, + required_physical_columns, + requested_physical_columns, )?); let physical_stats_schema = strip_metadata(physical_stats_schema); @@ -243,16 +342,37 @@ impl TableConfiguration { } /// Returns the list of physical column names that should have statistics collected. - pub(crate) fn stats_column_names_physical( + /// + /// Partition columns are excluded first (their values are already in the Add action's + /// `partitionValues` field). Among the remaining columns, if `required_columns` is `Some`, + /// those columns are always included regardless of `dataSkippingNumIndexedCols` or + /// `dataSkippingStatsColumns` settings (e.g. clustering columns). + pub(crate) fn physical_stats_column_names( &self, required_columns: Option<&[ColumnName]>, ) -> Vec { - let physical_stats_columns = self.required_stats_columns_physical(); + let physical_stats_columns = self.required_physical_stats_columns(); let config = StatsConfig { data_skipping_stats_columns: physical_stats_columns.as_deref(), data_skipping_num_indexed_cols: self.table_properties().data_skipping_num_indexed_cols, }; - stats_column_names(&self.physical_schema(), &config, required_columns) + stats_column_names( + &self.physical_data_schema_without_partition_columns(), + &config, + required_columns, + ) + } + + /// Stats-column set for `DataSkippingFilter`'s predicate-rewrite gate. The gate tests + /// every column reference in the rewritten predicate against this set; every data-skipping + /// call site shares this entry point so their gate input stays in lockstep. + pub(crate) fn physical_stats_columns_set( + &self, + required_columns: Option<&[ColumnName]>, + ) -> HashSet { + self.physical_stats_column_names(required_columns) + .into_iter() + .collect() } /// Returns the physical partition schema for `partitionValues_parsed`. @@ -261,24 +381,14 @@ impl TableConfiguration { /// and field types are the actual partition column data types with their original nullability. /// Returns `None` if the table has no partition columns. pub(crate) fn build_partition_values_parsed_schema(&self) -> Option { - let partition_columns = self.metadata().partition_columns(); - if partition_columns.is_empty() { + if self.logical_partition_columns().is_empty() { return None; } - let logical_schema = self.logical_schema(); - let column_mapping_mode = self.column_mapping_mode(); - let partition_fields: Vec = partition_columns - .iter() - .filter_map(|col_name| { - let field = logical_schema.field(col_name); - if field.is_none() { - warn!("Partition column '{col_name}' not found in table schema"); - } - field - }) - .map(|field: &StructField| { + let partition_fields: Vec = self + .physical_partition_fields() + .map(|(field, physical_name)| { StructField::new( - field.physical_name(column_mapping_mode).to_owned(), + physical_name.to_owned(), field.data_type().clone(), field.is_nullable(), ) @@ -287,51 +397,55 @@ impl TableConfiguration { Some(Arc::new(StructType::new_unchecked(partition_fields))) } - /// Returns the logical schema for data columns (excludes partition columns). + /// Typed physical partition schema for `DataSkippingFilter`, narrowed to the partition + /// columns referenced by `predicate_refs` (physical leaf names) with every field forced + /// nullable. /// - /// Partition columns are excluded because statistics are only collected for data columns - /// that are physically stored in the parquet files. Partition values are stored in the - /// file path, not in the file content, so they don't have file-level statistics. - fn logical_data_schema(&self) -> SchemaRef { - let partition_columns = self.partition_columns(); - Arc::new(StructType::new_unchecked( - self.logical_schema() - .fields() - .filter(|field| !partition_columns.contains(field.name())) - .cloned(), - )) + /// Returns `None` when the table has no partition columns or the predicate references none; + /// the filter then treats partitions as unavailable and folds partition predicates to + /// keep-all. Every retained field is nullable because a `MapToStruct` over `partitionValues` + /// yields null for a missing key or the protocol's empty-string-is-null rule, which a + /// non-nullable field would reject. + pub(crate) fn predicate_partition_schema( + &self, + predicate_refs: &[ColumnName], + ) -> Option { + let referenced: HashSet<&str> = predicate_refs + .iter() + .filter_map(|c| match c.path() { + [name] => Some(name.as_str()), + _ => None, + }) + .collect(); + let nullable_fields: Vec = self + .build_partition_values_parsed_schema()? + .fields() + .filter(|f| referenced.contains(f.name().as_str())) + .map(|f| StructField::nullable(f.name(), f.data_type().clone())) + .collect(); + (!nullable_fields.is_empty()).then(|| Arc::new(StructType::new_unchecked(nullable_fields))) + } + + /// Returns the logical schema excluding partition columns. + pub(crate) fn logical_schema_without_partition_columns(&self) -> SchemaRef { + self.logical_schema_without_partition_columns.clone() } /// Returns the physical data schema excluding partition columns. - fn physical_data_schema_without_partition_columns(&self) -> SchemaRef { - let partition_columns: HashSet<&str> = self - .partition_columns() - .iter() - .map(|s| s.as_str()) - .collect(); - // Safety: subset of an already-valid schema. - let schema = StructType::new_unchecked( - self.logical_schema() - .fields() - .zip(self.physical_schema().fields()) - .filter(|(logical_field, _)| { - !partition_columns.contains(logical_field.name().as_str()) - }) - .map(|(_, physical_field)| physical_field.clone()), - ); - Arc::new(schema) + pub(crate) fn physical_data_schema_without_partition_columns(&self) -> SchemaRef { + self.physical_data_schema_without_partition_columns.clone() } /// Translates `delta.dataSkippingStatsColumns` entries to physical column names. /// /// Returns `None` if the table property is not set. Entries that cannot be resolved /// (e.g. non-existent columns) are silently skipped with a warning. - fn required_stats_columns_physical(&self) -> Option> { + fn required_physical_stats_columns(&self) -> Option> { self.table_properties() .data_skipping_stats_columns .as_ref() .map(|cols| { - let logical_schema = self.logical_data_schema(); + let logical_schema = self.logical_schema_without_partition_columns(); let mode = self.column_mapping_mode(); cols.iter() .filter_map(|col| { @@ -370,16 +484,61 @@ impl TableConfiguration { self.logical_schema.clone() } + /// Borrows this table's logical schema, tied to `&self` (no `Arc` clone). + /// + /// Use this over [`logical_schema`](Self::logical_schema) when callers need to derive + /// `&self`-bound borrows from the schema (e.g. `&DataType` of a field). + pub(crate) fn logical_schema_ref(&self) -> &SchemaRef { + &self.logical_schema + } + + /// Whether any field in the logical schema declares a column default. + /// + /// This includes nested fields and is independent of the `allowColumnDefaults` feature. + pub(crate) fn has_column_with_default(&self) -> bool { + self.has_column_with_default + } + /// The physical schema ([`SchemaRef`]) of this table at this version. /// - /// When column mapping is disabled, this is identical to [`logical_schema`](Self::logical_schema). - /// Otherwise, field names are replaced with physical column names derived from column - /// mapping metadata. + /// When column mapping is disabled, this is identical to + /// [`logical_schema`](Self::logical_schema). Otherwise, field names are replaced with + /// physical column names derived from column mapping metadata. #[internal_api] pub(crate) fn physical_schema(&self) -> SchemaRef { self.physical_schema.clone() } + /// Whether partition column values must be materialized into data files. + /// Returns true when either: + /// * The [`MaterializePartitionColumns`] writer feature is enabled, or + /// * [`IcebergCompatV3`] is enabled + /// + /// [`MaterializePartitionColumns`]: crate::table_features::TableFeature::MaterializePartitionColumns + /// [`IcebergCompatV3`]: crate::table_features::TableFeature::IcebergCompatV3 + pub(crate) fn should_materialize_partition_columns(&self) -> bool { + // TODO(#1125): add IcebergcompatV1/V2 here when they are supported. + self.is_feature_enabled(&TableFeature::MaterializePartitionColumns) + || self.is_feature_enabled(&TableFeature::IcebergCompatV3) + } + + /// The physical schema for writing data files. + /// + /// When [`should_materialize_partition_columns`] is true, returns the full physical schema + /// (partition columns are materialized in data files). Otherwise, returns the physical + /// schema with partition columns excluded. Void columns are always stripped from the + /// returned schema, since they are never written to Parquet. + /// + /// [`should_materialize_partition_columns`]: Self::should_materialize_partition_columns + pub(crate) fn physical_write_schema(&self) -> SchemaRef { + let with_partition_cols = if self.should_materialize_partition_columns() { + self.physical_schema() + } else { + self.physical_data_schema_without_partition_columns() + }; + strip_void_from_schema(with_partition_cols) + } + /// The [`TableProperties`] of this table at this version. #[internal_api] pub(crate) fn table_properties(&self) -> &TableProperties { @@ -400,12 +559,18 @@ impl TableConfiguration { self.column_mapping_mode } - /// The partition columns of this table (empty if non-partitioned) + /// The logical partition columns of this table (empty if unpartitioned). #[internal_api] - pub(crate) fn partition_columns(&self) -> &[String] { + pub(crate) fn logical_partition_columns(&self) -> &[String] { self.metadata().partition_columns() } + /// The physical partition columns of this table (empty if unpartitioned). + pub(crate) fn physical_partition_columns(&self) -> impl Iterator + '_ { + self.physical_partition_fields() + .map(|(_, physical_name)| physical_name.to_owned()) + } + /// The [`Url`] of the table this [`TableConfiguration`] belongs to #[internal_api] pub(crate) fn table_root(&self) -> &Url { @@ -418,6 +583,24 @@ impl TableConfiguration { self.version } + // TODO(#3020): Unify scan-state schema construction and write-context serialization to call + // this. + fn physical_partition_fields(&self) -> impl Iterator + '_ { + let column_mapping_mode = self.column_mapping_mode(); + self.logical_partition_columns() + .iter() + .filter_map(move |name| { + // SAFETY: Construction already validates that every partition column exists in + // the schema. Keep this iterator infallible for a simpler return type, with a + // defensive warning if the invariant is violated. + let field = self.logical_schema.field(name); + if field.is_none() { + warn!("Partition column '{name}' not found in table schema"); + } + field.map(|field| (field, field.physical_name(column_mapping_mode))) + }) + } + /// Validates that all feature requirements for a given feature are satisfied. fn validate_feature_requirements(&self, feature: &TableFeature) -> DeltaResult<()> { for req in feature.info().feature_requirements { @@ -481,7 +664,6 @@ impl TableConfiguration { check(&self.protocol, &self.table_properties, operation)?; } }; - self.validate_feature_requirements(feature) } @@ -489,24 +671,7 @@ impl TableConfiguration { /// For table features protocol (v3), returns the explicit reader_features list. /// For legacy protocol (v1-2), infers features from the version number. fn get_enabled_reader_features(&self) -> Vec { - match self.protocol.min_reader_version() { - TABLE_FEATURES_MIN_READER_VERSION => { - // Table features reader: use explicit reader_features list - self.protocol - .reader_features() - .map(|f| f.to_vec()) - .unwrap_or_default() - } - v if (1..=2).contains(&v) => { - // Legacy reader: infer features from version - LEGACY_READER_FEATURES - .iter() - .filter(|f| f.is_valid_for_legacy_reader(v)) - .cloned() - .collect() - } - _ => Vec::new(), - } + extract_enabled_reader_features(&self.protocol) } /// Returns all writer features enabled for this table based on protocol version. @@ -548,13 +713,7 @@ impl TableConfiguration { /// Internal helper for read operations (Scan, Cdf) fn ensure_read_supported(&self, operation: Operation) -> DeltaResult<()> { - // Version check: kernel supports reader versions 1..=MAX_VALID_READER_VERSION - if self.protocol.min_reader_version() > MAX_VALID_READER_VERSION { - return Err(Error::unsupported(format!( - "Unsupported minimum reader version {}", - self.protocol.min_reader_version() - ))); - } + check_reader_version_range(&self.protocol)?; // Check all enabled reader features have kernel support for feature in self.get_enabled_reader_features() { @@ -566,6 +725,15 @@ impl TableConfiguration { /// Internal helper for write operations fn ensure_write_supported(&self) -> DeltaResult<()> { + // Version check: kernel supports writer versions + // MIN_VALID_RW_VERSION..=MAX_VALID_WRITER_VERSION + require!( + self.protocol.min_writer_version() >= MIN_VALID_RW_VERSION, + Error::InvalidProtocol(format!( + "min_writer_version must be >= {MIN_VALID_RW_VERSION}, got {}", + self.protocol.min_writer_version() + )) + ); // Version check: kernel supports writer versions 1..=MAX_VALID_WRITER_VERSION if self.protocol.min_writer_version() > MAX_VALID_WRITER_VERSION { return Err(Error::unsupported(format!( @@ -629,8 +797,8 @@ impl TableConfiguration { /// Returns `true` if row tracking is suspended for this table. /// - /// Row tracking is suspended when the `delta.rowTrackingSuspended` table property is set to `true`. - /// Note that: + /// Row tracking is suspended when the `delta.rowTrackingSuspended` table property is set to + /// `true`. Note that: /// - Row tracking can be _supported_ and _suspended_ at the same time. /// - Row tracking cannot be _enabled_ while _suspended_. pub(crate) fn is_row_tracking_suspended(&self) -> bool { @@ -695,8 +863,13 @@ impl TableConfiguration { // Legacy reader: protocol reader version meets minimum requirement self.protocol.min_reader_version() >= min_reader_version } else { - // Table features reader: feature is in reader_features list + // Reader-supported if the feature is in reader_features, or it is a legacy + // ReaderWriter feature (only ColumnMapping) whose minimum reader version is + // met. The second case stays compatible with tables a past delta-spark bug + // created with ReaderWriter features in writerFeatures only, absent from + // readerFeatures. Self::has_feature(self.protocol.reader_features(), feature) + || feature.is_valid_for_legacy_reader(self.protocol.min_reader_version()) }; let writer_supported = if self.is_legacy_writer_version() { @@ -709,7 +882,7 @@ impl TableConfiguration { reader_supported && writer_supported } - FeatureType::Unknown => false, + FeatureType::Unknown => Self::has_feature(self.protocol.writer_features(), feature), } } @@ -729,6 +902,32 @@ impl TableConfiguration { EnablementCheck::EnabledIf(check_fn) => check_fn(&self.table_properties), } } + + /// Returns true when the table requires every AddFile to carry a non-null + /// `stats.numRecords`. + pub(crate) fn requires_stats_num_records(&self) -> bool { + // TODO(#1125): Add icebergCompatV2 to the list when it is supported. + self.is_feature_enabled(&TableFeature::IcebergCompatV3) + } + + /// TODO(#2538): Row-tracking is not fully supported for removeFile currently. + /// See `crate::table_features::ROW_TRACKING_INFO` for more details. + pub(crate) fn validate_feature_support_for_remove(&self) -> DeltaResult<()> { + // RowTracking is a prerequisite for IcebergCompatV3, so the IcebergCompatV3 arm is + // technically redundant. Just be conservative here to check both. + if self.should_write_row_tracking() { + return Err(Error::unsupported( + "Remove actions are not yet supported on tables with rowTracking supported \ + and not suspended", + )); + } + if self.is_feature_enabled(&TableFeature::IcebergCompatV3) { + return Err(Error::unsupported( + "Remove actions are not yet supported on tables with icebergCompatV3 enabled", + )); + } + Ok(()) + } } #[cfg(test)] @@ -737,27 +936,30 @@ mod test { use std::collections::HashMap; use std::sync::Arc; + use rstest::rstest; use url::Url; - use crate::actions::{Metadata, Protocol}; - use crate::schema::ColumnName; - use crate::schema::{DataType, SchemaRef, StructField, StructType}; - use crate::table_features::ColumnMappingMode; + use super::{InCommitTimestampEnablement, TableConfiguration}; + use crate::actions::{Metadata, Protocol, MIN_VALUES}; + use crate::schema::{ + column_name, schema_ref, ColumnName, DataType, SchemaRef, StructField, StructType, + }; use crate::table_features::{ - FeatureType, Operation, TableFeature, TABLE_FEATURES_MIN_READER_VERSION, + ColumnMappingMode, FeatureType, Operation, TableFeature, TABLE_FEATURES_MIN_READER_VERSION, TABLE_FEATURES_MIN_WRITER_VERSION, }; - use crate::table_properties::{TableProperties, COLUMN_MAPPING_MODE}; - use crate::utils::test_utils::{ + use crate::table_properties::{ + TableProperties, COLUMN_MAPPING_MODE, ENABLE_DELETION_VECTORS, ENABLE_ICEBERG_COMPAT_V1, + ENABLE_ICEBERG_COMPAT_V2, ENABLE_ICEBERG_COMPAT_V3, ENABLE_IN_COMMIT_TIMESTAMPS, + ENABLE_ROW_TRACKING, ROW_TRACKING_SUSPENDED, + }; + use crate::unit_test_utils::{ assert_result_error_with_message, test_schema_flat, test_schema_flat_with_column_mapping, test_schema_nested, test_schema_nested_with_column_mapping, test_schema_with_array, test_schema_with_array_and_column_mapping, test_schema_with_map, test_schema_with_map_and_column_mapping, }; use crate::Error; - use rstest::rstest; - - use super::{InCommitTimestampEnablement, TableConfiguration}; fn create_mock_table_config( props_to_enable: &[(&str, &str)], @@ -777,10 +979,7 @@ mod test { min_reader_version: i32, min_writer_version: i32, ) -> TableConfiguration { - let schema = Arc::new(StructType::new_unchecked([StructField::nullable( - "value", - DataType::INTEGER, - )])); + let schema = schema_ref! { nullable "value": INTEGER }; let metadata = Metadata::try_new( None, None, @@ -829,14 +1028,54 @@ mod test { TableConfiguration::try_new(metadata, protocol, table_root, 0).unwrap() } + #[test] + fn table_configuration_rejects_partition_column_missing_from_schema() { + let schema = schema_ref! { nullable "value": INTEGER }; + let metadata = Metadata::try_new( + None, + None, + schema, + vec!["missing".to_string()], + 0, + HashMap::new(), + ) + .unwrap(); + let protocol = Protocol::try_new_legacy(1, 2).unwrap(); + let table_root = Url::try_from("file:///").unwrap(); + + let result = TableConfiguration::try_new(metadata, protocol, table_root, 0); + + assert_result_error_with_message(result, "Partition column 'missing' not found in schema"); + } + + #[test] + fn table_configuration_rejects_duplicate_partition_columns() { + let schema = Arc::new(StructType::new_unchecked([ + StructField::nullable("value", DataType::INTEGER), + StructField::nullable("part", DataType::STRING), + ])); + let metadata = Metadata::try_new( + None, + None, + schema, + vec!["part".to_string(), "part".to_string()], + 0, + HashMap::new(), + ) + .unwrap(); + let protocol = Protocol::try_new_legacy(1, 2).unwrap(); + let table_root = Url::try_from("file:///").unwrap(); + + let result = TableConfiguration::try_new(metadata, protocol, table_root, 0); + + assert_result_error_with_message(result, "Duplicate partition column: 'part'"); + } + #[test] fn dv_supported_not_enabled() { use crate::table_properties::ENABLE_CHANGE_DATA_FEED; - let schema = Arc::new(StructType::new_unchecked([StructField::nullable( - "value", - DataType::INTEGER, - )])); + let schema = schema_ref! { nullable "value": INTEGER }; let metadata = Metadata::try_new( None, None, @@ -861,10 +1100,7 @@ mod test { fn dv_enabled() { use crate::table_properties::{ENABLE_CHANGE_DATA_FEED, ENABLE_DELETION_VECTORS}; - let schema = Arc::new(StructType::new_unchecked([StructField::nullable( - "value", - DataType::INTEGER, - )])); + let schema = schema_ref! { nullable "value": INTEGER }; let metadata = Metadata::try_new( None, None, @@ -888,10 +1124,33 @@ mod test { assert!(table_config.is_feature_enabled(&TableFeature::DeletionVectors)); } + #[rstest] + #[case(-1, 2, Operation::Scan)] + #[case(1, -1, Operation::Write)] + fn reject_protocol_version_below_minimum( + #[case] rv: i32, + #[case] wv: i32, + #[case] op: Operation, + ) { + let schema = schema_ref! { nullable "value": INTEGER }; + let metadata = Metadata::try_new(None, None, schema, vec![], 0, HashMap::new()).unwrap(); + let protocol = + Protocol::new_unchecked(rv, wv, TableFeature::NO_LIST, TableFeature::NO_LIST); + let table_root = Url::try_from("file:///").unwrap(); + let table_config = TableConfiguration::try_new(metadata, protocol, table_root, 0).unwrap(); + let expected = if rv < 1 { + format!("Invalid protocol action in the delta log: min_reader_version must be >= 1, got {rv}") + } else { + format!("Invalid protocol action in the delta log: min_writer_version must be >= 1, got {wv}") + }; + assert_result_error_with_message(table_config.ensure_operation_supported(op), &expected); + } + #[test] fn write_with_cdf() { - use crate::table_properties::{APPEND_ONLY, ENABLE_CHANGE_DATA_FEED}; use TableFeature::*; + + use crate::table_properties::{APPEND_ONLY, ENABLE_CHANGE_DATA_FEED}; let cases = [ ( // Writing to CDF-enabled table is supported for writes @@ -967,10 +1226,7 @@ mod test { fn ict_enabled_from_table_creation() { use crate::table_properties::ENABLE_IN_COMMIT_TIMESTAMPS; - let schema = Arc::new(StructType::new_unchecked([StructField::nullable( - "value", - DataType::INTEGER, - )])); + let schema = schema_ref! { nullable "value": INTEGER }; let metadata = Metadata::try_new( None, None, @@ -1002,10 +1258,7 @@ mod test { IN_COMMIT_TIMESTAMP_ENABLEMENT_VERSION, }; - let schema = Arc::new(StructType::new_unchecked([StructField::nullable( - "value", - DataType::INTEGER, - )])); + let schema = schema_ref! { nullable "value": INTEGER }; let metadata = Metadata::try_new( None, None, @@ -1046,10 +1299,7 @@ mod test { ENABLE_IN_COMMIT_TIMESTAMPS, IN_COMMIT_TIMESTAMP_ENABLEMENT_VERSION, }; - let schema = Arc::new(StructType::new_unchecked([StructField::nullable( - "value", - DataType::INTEGER, - )])); + let schema = schema_ref! { nullable "value": INTEGER }; let metadata = Metadata::try_new( None, None, @@ -1080,10 +1330,7 @@ mod test { } #[test] fn ict_supported_and_not_enabled() { - let schema = Arc::new(StructType::new_unchecked([StructField::nullable( - "value", - DataType::INTEGER, - )])); + let schema = schema_ref! { nullable "value": INTEGER }; let metadata = Metadata::try_new(None, None, schema, vec![], 0, HashMap::new()).unwrap(); let protocol = Protocol::try_new_modern(TableFeature::EMPTY_LIST, [TableFeature::InCommitTimestamp]) @@ -1097,10 +1344,7 @@ mod test { } #[test] fn fails_on_unsupported_feature() { - let schema = Arc::new(StructType::new_unchecked([StructField::nullable( - "value", - DataType::INTEGER, - )])); + let schema = schema_ref! { nullable "value": INTEGER }; let metadata = Metadata::try_new(None, None, schema, vec![], 0, HashMap::new()).unwrap(); let protocol = Protocol::try_new_modern(["unknown"], ["unknown"]).unwrap(); let table_root = Url::try_from("file:///").unwrap(); @@ -1113,10 +1357,7 @@ mod test { fn dv_not_supported() { use crate::table_properties::ENABLE_CHANGE_DATA_FEED; - let schema = Arc::new(StructType::new_unchecked([StructField::nullable( - "value", - DataType::INTEGER, - )])); + let schema = schema_ref! { nullable "value": INTEGER }; let metadata = Metadata::try_new( None, None, @@ -1144,10 +1385,7 @@ mod test { fn test_try_new_from() { use crate::table_properties::{ENABLE_CHANGE_DATA_FEED, ENABLE_DELETION_VECTORS}; - let schema = Arc::new(StructType::new_unchecked([StructField::nullable( - "value", - DataType::INTEGER, - )])); + let schema = schema_ref! { nullable "value": INTEGER }; let metadata = Metadata::try_new( None, None, @@ -1165,10 +1403,7 @@ mod test { let table_root = Url::try_from("file:///").unwrap(); let table_config = TableConfiguration::try_new(metadata, protocol, table_root, 0).unwrap(); - let new_schema = Arc::new(StructType::new_unchecked([StructField::nullable( - "value", - DataType::INTEGER, - )])); + let new_schema = schema_ref! { nullable "value": INTEGER }; let new_metadata = Metadata::try_new( None, None, @@ -1225,10 +1460,7 @@ mod test { #[test] fn test_timestamp_ntz_validation_integration() { // Schema with TIMESTAMP_NTZ column - let schema = Arc::new(StructType::new_unchecked([StructField::nullable( - "ts", - DataType::TIMESTAMP_NTZ, - )])); + let schema = schema_ref! { nullable "ts": TIMESTAMP_NTZ }; let metadata = Metadata::try_new(None, None, schema, vec![], 0, HashMap::new()).unwrap(); let protocol_without_timestamp_ntz_features = @@ -1263,14 +1495,44 @@ mod test { } #[test] - fn test_variant_validation_integration() { - // Schema with VARIANT column + fn test_timestamp_ntz_legacy_alias_unblocks_read_and_write() { let schema = Arc::new(StructType::new_unchecked([StructField::nullable( - "v", - DataType::unshredded_variant(), + "ts", + DataType::TIMESTAMP_NTZ, )])); let metadata = Metadata::try_new(None, None, schema, vec![], 0, HashMap::new()).unwrap(); + // Build the protocol from the legacy string alias to exercise the real read path. + let protocol = + Protocol::try_new_modern(["timestampWithoutTimezone"], ["timestampWithoutTimezone"]) + .unwrap(); + + assert_eq!( + protocol.reader_features(), + Some([TableFeature::TimestampWithoutTimezone].as_slice()) + ); + assert_eq!( + protocol.writer_features(), + Some([TableFeature::TimestampWithoutTimezone].as_slice()) + ); + + let table_root = Url::try_from("file:///").unwrap(); + let table_config = TableConfiguration::try_new(metadata, protocol, table_root, 0).unwrap(); + + table_config + .ensure_operation_supported(Operation::Scan) + .unwrap(); + table_config + .ensure_operation_supported(Operation::Write) + .unwrap(); + } + + #[test] + fn test_variant_validation_integration() { + // Schema with VARIANT column + let schema = schema_ref! { nullable "v": (DataType::unshredded_variant()) }; + let metadata = Metadata::try_new(None, None, schema, vec![], 0, HashMap::new()).unwrap(); + let protocol_without_variant_features = Protocol::try_new_modern(TableFeature::EMPTY_LIST, TableFeature::EMPTY_LIST).unwrap(); @@ -1296,31 +1558,101 @@ mod test { ); } - #[test] - fn test_is_feature_supported_returns_false_for_unknown_feature() { - let config = create_mock_table_config(&[], &[TableFeature::DeletionVectors]); - assert!(!config.is_feature_supported(&TableFeature::unknown("futureFeature"))); + #[derive(Debug, Clone, Copy)] + enum UnknownFeatureShape { + NotListed, + WriterOnly, + ReaderWriter, } - #[test] - fn test_is_feature_enabled_returns_false_for_unknown_feature() { - let config = create_mock_table_config(&[], &[TableFeature::DeletionVectors]); - assert!(!config.is_feature_enabled(&TableFeature::unknown("futureFeature"))); - } + fn create_unknown_feature_config( + shape: UnknownFeatureShape, + ) -> (TableFeature, TableConfiguration) { + const UNKNOWN: &str = "futureFeature"; + let metadata = Metadata::try_new( + None, + None, + schema_ref! { nullable "value": INTEGER }, + vec![], + 0, + HashMap::new(), + ) + .unwrap(); + let table_root = Url::try_from("file:///").unwrap(); - #[test] - fn test_is_feature_supported_writer_only() { - let feature = TableFeature::AppendOnly; + let reader_features = match shape { + UnknownFeatureShape::ReaderWriter => vec![UNKNOWN], + _ => vec![], + }; + let writer_features = match shape { + UnknownFeatureShape::NotListed => vec![], + _ => vec![UNKNOWN], + }; + let protocol = Protocol::try_new_modern(reader_features, writer_features).unwrap(); - // Test with legacy protocol writer v2 - should be supported - let config = create_mock_table_config_with_version(&[], None, 1, 2); - assert!(config.is_feature_supported(&feature)); + let tc = TableConfiguration::try_new(metadata, protocol, table_root, 0).unwrap(); + (TableFeature::unknown(UNKNOWN), tc) + } - // Test with legacy protocol writer v1 - should NOT be supported - let config = create_mock_table_config_with_version(&[], None, 1, 1); - assert!(!config.is_feature_supported(&feature)); + #[rstest] + #[case::not_listed(UnknownFeatureShape::NotListed, false)] + #[case::writer_only(UnknownFeatureShape::WriterOnly, true)] + #[case::reader_writer(UnknownFeatureShape::ReaderWriter, true)] + fn test_unknown_feature_protocol_support( + #[case] shape: UnknownFeatureShape, + #[case] expected_supported: bool, + ) { + let (unknown, config) = create_unknown_feature_config(shape); + assert_eq!(config.is_feature_supported(&unknown), expected_supported); + } - // reader=2 (legacy), writer=7 (non-legacy) - feature in list, should be supported + #[rstest] + #[case::not_listed(UnknownFeatureShape::NotListed, false)] + #[case::writer_only(UnknownFeatureShape::WriterOnly, true)] + #[case::reader_writer(UnknownFeatureShape::ReaderWriter, true)] + fn test_unknown_feature_protocol_enablement( + #[case] shape: UnknownFeatureShape, + #[case] expected_enabled: bool, + ) { + let (unknown, config) = create_unknown_feature_config(shape); + assert_eq!(config.is_feature_enabled(&unknown), expected_enabled); + } + + #[rstest] + fn test_unknown_feature_capabilities( + #[values( + UnknownFeatureShape::NotListed, + UnknownFeatureShape::WriterOnly, + UnknownFeatureShape::ReaderWriter + )] + shape: UnknownFeatureShape, + #[values(Operation::Scan, Operation::Cdf, Operation::Write)] operation: Operation, + ) { + let (_, config) = create_unknown_feature_config(shape); + let expected_ok = match shape { + UnknownFeatureShape::NotListed => true, + UnknownFeatureShape::WriterOnly => operation != Operation::Write, + UnknownFeatureShape::ReaderWriter => false, + }; + assert_eq!( + config.ensure_operation_supported(operation).is_ok(), + expected_ok + ); + } + + #[test] + fn test_is_feature_supported_writer_only() { + let feature = TableFeature::AppendOnly; + + // Test with legacy protocol writer v2 - should be supported + let config = create_mock_table_config_with_version(&[], None, 1, 2); + assert!(config.is_feature_supported(&feature)); + + // Test with legacy protocol writer v1 - should NOT be supported + let config = create_mock_table_config_with_version(&[], None, 1, 1); + assert!(!config.is_feature_supported(&feature)); + + // reader=2 (legacy), writer=7 (non-legacy) - feature in list, should be supported let config = create_mock_table_config_with_version(&[], Some(&[TableFeature::AppendOnly]), 2, 7); assert!(config.is_feature_supported(&feature)); @@ -1331,7 +1663,8 @@ mod test { create_mock_table_config_with_version(&[], Some(&[TableFeature::ChangeDataFeed]), 2, 7); assert!(!config.is_feature_supported(&feature)); - // Test with protocol reader=3, writer=7 (both non-legacy) - feature in list, should be supported + // Test with protocol reader=3, writer=7 (both non-legacy) - feature in list, should be + // supported let config = create_mock_table_config(&[], &[TableFeature::AppendOnly]); assert!(config.is_feature_supported(&feature)); @@ -1363,7 +1696,8 @@ mod test { create_mock_table_config_with_version(&[], Some(&[TableFeature::AppendOnly]), 2, 7); // ColumnMapping (ReaderWriter) should NOT be supported because: // - reader=2 (legacy) checks version: 2 >= 2 (reader_supported = true) - // - writer=7 (non-legacy) checks list: ColumnMapping not in writer_features (writer_supported = false) + // - writer=7 (non-legacy) checks list: ColumnMapping not in writer_features + // (writer_supported = false) // - Result: false (requires BOTH to be true) assert!(!config.is_feature_supported(&feature)); @@ -1376,6 +1710,43 @@ mod test { assert!(!config.is_feature_supported(&feature)); } + #[test] + fn test_is_feature_supported_orphaned_column_mapping() { + // A (3, 7) table with ColumnMapping in writerFeatures but missing from readerFeatures. + // ColumnMapping is a legacy ReaderWriter feature whose minimum reader version (2) is met by + // reader version 3, so it counts as reader-supported even though it is absent from + // readerFeatures. It is in writerFeatures, so it is writer-supported too. + let config = create_mock_table_config_with_cm( + &[], + Some(ColumnMappingMode::Name), + &TableFeature::EMPTY_LIST, + &[TableFeature::ColumnMapping], + ); + assert!(config.is_feature_supported(&TableFeature::ColumnMapping)); + + // A non-legacy ReaderWriter feature in the same writer-only position has no legacy reader + // version to fall back on, so the protocol is rejected outright rather than tolerated. + assert!(Protocol::try_new( + TABLE_FEATURES_MIN_READER_VERSION, + TABLE_FEATURES_MIN_WRITER_VERSION, + Some(TableFeature::EMPTY_LIST), + Some(vec![TableFeature::DeletionVectors]), + ) + .is_err()); + + // The conformant shape (ColumnMapping in both lists) is still reported supported: the + // legacy-version fallback does not perturb the normal reader_features membership path. + let conformant = create_mock_table_config(&[], &[TableFeature::ColumnMapping]); + assert!(conformant.is_feature_supported(&TableFeature::ColumnMapping)); + } + + #[test] + fn test_column_mapping_absent_from_both_lists_is_unsupported() { + // ColumnMapping in neither list must not be treated as supported: the writer half fails. + let config = create_mock_table_config(&[], &[TableFeature::AppendOnly]); + assert!(!config.is_feature_supported(&TableFeature::ColumnMapping)); + } + #[test] fn test_is_feature_enabled_with_property_check() { use crate::table_properties::APPEND_ONLY; @@ -1432,6 +1803,13 @@ mod test { 7, ); assert!(config.ensure_operation_supported(Operation::Scan).is_ok()); + + #[cfg(feature = "geo-type-in-dev")] + { + let config = create_mock_table_config(&[], &[TableFeature::GeospatialType]); + assert!(config.ensure_operation_supported(Operation::Scan).is_ok()); + assert!(config.ensure_operation_supported(Operation::Cdf).is_ok()); + } } #[test] @@ -1454,14 +1832,33 @@ mod test { config.ensure_operation_supported(Operation::Write), r#"Feature 'typeWidening' is not supported for writes"#, ); + + #[cfg(feature = "geo-type-in-dev")] + { + let config = create_mock_table_config(&[], &[TableFeature::GeospatialType]); + assert_result_error_with_message( + config.ensure_operation_supported(Operation::Write), + r#"Feature 'geospatial' is not supported for writes"#, + ); + } + } + + #[cfg(not(feature = "geo-type-in-dev"))] + #[rstest] + #[case::scan(Operation::Scan)] + #[case::cdf(Operation::Cdf)] + #[case::write(Operation::Write)] + fn test_geospatial_not_supported_without_cargo_feature(#[case] operation: Operation) { + let config = create_mock_table_config(&[], &[TableFeature::GeospatialType]); + assert_result_error_with_message( + config.ensure_operation_supported(operation), + "Feature 'geospatial' is not supported", + ); } #[test] fn test_illegal_writer_feature_combination() { - let schema = Arc::new(StructType::new_unchecked([StructField::nullable( - "value", - DataType::INTEGER, - )])); + let schema = schema_ref! { nullable "value": INTEGER }; let metadata = Metadata::try_new(None, None, schema, vec![], 0, HashMap::new()).unwrap(); let protocol = Protocol::try_new_modern(TableFeature::EMPTY_LIST, vec![TableFeature::RowTracking]) @@ -1476,10 +1873,7 @@ mod test { #[test] fn test_row_tracking_with_domain_metadata_requirement() { - let schema = Arc::new(StructType::new_unchecked([StructField::nullable( - "value", - DataType::INTEGER, - )])); + let schema = schema_ref! { nullable "value": INTEGER }; let metadata = Metadata::try_new(None, None, schema, vec![], 0, HashMap::new()).unwrap(); let protocol = Protocol::try_new_modern( TableFeature::EMPTY_LIST, @@ -1494,16 +1888,47 @@ mod test { ); } - #[cfg(feature = "catalog-managed")] #[test] fn test_catalog_managed_writes() { - let config = create_mock_table_config(&[], &[TableFeature::CatalogManaged]); + // CatalogManaged requires ICT to be supported and enabled + let config = create_mock_table_config( + &[(ENABLE_IN_COMMIT_TIMESTAMPS, "true")], + &[ + TableFeature::CatalogManaged, + TableFeature::InCommitTimestamp, + ], + ); assert!(config.ensure_operation_supported(Operation::Write).is_ok()); - let config = create_mock_table_config(&[], &[TableFeature::CatalogOwnedPreview]); + let config = create_mock_table_config( + &[(ENABLE_IN_COMMIT_TIMESTAMPS, "true")], + &[ + TableFeature::CatalogOwnedPreview, + TableFeature::InCommitTimestamp, + ], + ); assert!(config.ensure_operation_supported(Operation::Write).is_ok()); } + // A catalog-managed table requires inCommitTimestamp to be enabled. + #[rstest] + #[case::catalog_managed( + TableFeature::CatalogManaged, + "Feature 'catalogManaged' requires 'inCommitTimestamp' to be enabled" + )] + #[case::catalog_owned_preview( + TableFeature::CatalogOwnedPreview, + "Feature 'catalogOwned-preview' requires 'inCommitTimestamp' to be enabled" + )] + fn test_catalog_managed_requires_in_commit_timestamp( + #[case] feature: TableFeature, + #[case] expected_error: &str, + ) { + let config = create_mock_table_config(&[], &[feature]); + let result = config.ensure_operation_supported(Operation::Write); + assert_result_error_with_message(result, expected_error); + } + /// Helper to create a schema with column mapping metadata using JSON deserialization fn schema_with_column_mapping() -> SchemaRef { let field_a: StructField = serde_json::from_str( @@ -1546,6 +1971,20 @@ mod test { schema: SchemaRef, column_mapping_mode: &str, extra_props: impl IntoIterator, + ) -> TableConfiguration { + create_partitioned_table_config_with_column_mapping( + schema, + column_mapping_mode, + vec![], // partition_columns + extra_props, + ) + } + + fn create_partitioned_table_config_with_column_mapping( + schema: SchemaRef, + column_mapping_mode: &str, + partition_columns: Vec, + extra_props: impl IntoIterator, ) -> TableConfiguration { let mut props: HashMap = extra_props .into_iter() @@ -1556,7 +1995,7 @@ mod test { column_mapping_mode.to_string(), ); - let metadata = Metadata::try_new(None, None, schema, vec![], 0, props).unwrap(); + let metadata = Metadata::try_new(None, None, schema, partition_columns, 0, props).unwrap(); // Use reader version 2 which supports column mapping let protocol = Protocol::try_new_legacy(2, 5).unwrap(); @@ -1582,7 +2021,7 @@ mod test { // Verify field names are logical names let min_values = stats_schemas .physical - .field("minValues") + .field(MIN_VALUES) .unwrap() .data_type(); if let DataType::Struct(inner) = min_values { @@ -1606,7 +2045,7 @@ mod test { // Verify physical schema has physical names let physical_min_values = stats_schemas .physical - .field("minValues") + .field(MIN_VALUES) .unwrap() .data_type(); if let DataType::Struct(inner) = physical_min_values { @@ -1642,7 +2081,7 @@ mod test { // Verify physical schema has physical names let physical_min_values = stats_schemas .physical - .field("minValues") + .field(MIN_VALUES) .unwrap() .data_type(); let DataType::Struct(inner) = physical_min_values else { @@ -1682,9 +2121,11 @@ mod test { ); } - #[test] - fn test_build_expected_stats_schemas_excludes_partition_columns() { - let field_a: StructField = serde_json::from_str( + /// Schema with a data column and two partition columns, all with column mapping metadata. + /// data_col (long) -> phys_data, part_a (string) -> phys_part_a, part_b (integer) -> + /// phys_part_b + fn partitioned_schema_with_column_mapping() -> SchemaRef { + let data_col: StructField = serde_json::from_str( r#"{ "name": "data_col", "type": "long", @@ -1696,34 +2137,47 @@ mod test { }"#, ) .unwrap(); - - let field_b: StructField = serde_json::from_str( + let part_a: StructField = serde_json::from_str( r#"{ - "name": "part_col", + "name": "part_a", "type": "string", "nullable": true, "metadata": { "delta.columnMapping.id": 2, - "delta.columnMapping.physicalName": "phys_part" + "delta.columnMapping.physicalName": "phys_part_a" + } + }"#, + ) + .unwrap(); + let part_b: StructField = serde_json::from_str( + r#"{ + "name": "part_b", + "type": "integer", + "nullable": true, + "metadata": { + "delta.columnMapping.id": 3, + "delta.columnMapping.physicalName": "phys_part_b" } }"#, ) .unwrap(); + Arc::new(StructType::new_unchecked([data_col, part_a, part_b])) + } - let schema = Arc::new(StructType::new_unchecked([field_a, field_b])); - let mut props = HashMap::new(); - props.insert(COLUMN_MAPPING_MODE.to_string(), "name".to_string()); - let metadata = - Metadata::try_new(None, None, schema, vec!["part_col".to_string()], 0, props).unwrap(); - let protocol = Protocol::try_new_legacy(2, 5).unwrap(); - let table_root = Url::try_from("file:///").unwrap(); - let config = TableConfiguration::try_new(metadata, protocol, table_root, 0).unwrap(); + #[test] + fn test_build_expected_stats_schemas_excludes_partition_columns() { + let config = create_partitioned_table_config_with_column_mapping( + partitioned_schema_with_column_mapping(), + "name", + vec!["part_a".to_string(), "part_b".to_string()], + [], + ); let stats_schemas = config.build_expected_stats_schemas(None, None).unwrap(); let DataType::Struct(inner) = stats_schemas .physical - .field("minValues") + .field(MIN_VALUES) .unwrap() .data_type() else { @@ -1734,60 +2188,159 @@ mod test { "Data column should be present with physical name" ); assert!( - inner.field("phys_part").is_none(), - "Partition column should be excluded" + inner.field("phys_part_a").is_none(), + "Partition column a should be excluded" ); assert!( - inner.field("part_col").is_none(), - "Partition column logical name should also be absent" + inner.field("phys_part_b").is_none(), + "Partition column b should be excluded" ); } #[test] - fn test_stats_column_names_physical_returns_physical_names() { - // stats_column_names_physical should return physical column names + fn test_partition_columns_are_logical_under_column_mapping() { + let config = create_partitioned_table_config_with_column_mapping( + partitioned_schema_with_column_mapping(), + "name", + vec!["part_a".to_string(), "part_b".to_string()], + [], + ); + + assert_eq!(config.logical_partition_columns(), ["part_a", "part_b"]); + let partition_schema = config + .build_partition_values_parsed_schema() + .expect("partition schema should be present"); + assert!(partition_schema.field("phys_part_a").is_some()); + assert!(partition_schema.field("phys_part_b").is_some()); + assert!(partition_schema.field("part_a").is_none()); + assert!(partition_schema.field("part_b").is_none()); + } + + #[rstest] + #[case::none("none", ["part_a", "part_b"])] + #[case::name("name", ["phys_part_a", "phys_part_b"])] + #[case::id("id", ["phys_part_a", "phys_part_b"])] + fn test_physical_partition_columns( + #[case] column_mapping_mode: &str, + #[case] expected: [&str; 2], + ) { + let config = create_partitioned_table_config_with_column_mapping( + partitioned_schema_with_column_mapping(), + column_mapping_mode, + vec!["part_a".to_string(), "part_b".to_string()], + [], + ); + + assert_eq!( + config.physical_partition_columns().collect::>(), + expected + ); + } + + #[test] + fn test_physical_stats_column_names_excludes_partition_columns() { + let config = create_partitioned_table_config_with_column_mapping( + partitioned_schema_with_column_mapping(), + "name", + vec!["part_a".to_string(), "part_b".to_string()], + [], + ); + + let column_names = config.physical_stats_column_names(None); + assert_eq!(column_names, vec![column_name!("phys_data")]); + + // Also verify partition columns are excluded when passed as required columns + let required = [column_name!("phys_part_a"), column_name!("phys_part_b")]; + let column_names = config.physical_stats_column_names(Some(&required)); + assert_eq!(column_names, vec![column_name!("phys_data")]); + } + + #[test] + fn test_physical_stats_column_names_excludes_partition_columns_no_column_mapping() { + let schema = Arc::new(StructType::new_unchecked([ + StructField::nullable("data_col", DataType::LONG), + StructField::nullable("part_a", DataType::STRING), + StructField::nullable("part_b", DataType::INTEGER), + ])); + let metadata = Metadata::try_new( + None, + None, + schema, + vec!["part_a".to_string(), "part_b".to_string()], + 0, + HashMap::new(), + ) + .unwrap(); + let protocol = Protocol::try_new_legacy(1, 2).unwrap(); + let table_root = Url::try_from("file:///").unwrap(); + let config = TableConfiguration::try_new(metadata, protocol, table_root, 0).unwrap(); + + let column_names = config.physical_stats_column_names(None); + assert_eq!(column_names, vec![column_name!("data_col")]); + } + + #[test] + fn test_physical_stats_column_names_all_partition_columns_returns_empty() { + let schema = Arc::new(StructType::new_unchecked([ + StructField::nullable("part_a", DataType::STRING), + StructField::nullable("part_b", DataType::INTEGER), + ])); + let metadata = Metadata::try_new( + None, + None, + schema, + vec!["part_a".to_string(), "part_b".to_string()], + 0, + HashMap::new(), + ) + .unwrap(); + let protocol = Protocol::try_new_legacy(1, 2).unwrap(); + let table_root = Url::try_from("file:///").unwrap(); + let config = TableConfiguration::try_new(metadata, protocol, table_root, 0).unwrap(); + + let column_names = config.physical_stats_column_names(None); + assert!(column_names.is_empty()); + } + + #[test] + fn test_physical_stats_column_names_returns_physical_names() { + // physical_stats_column_names should return physical column names let schema = schema_with_column_mapping(); let config = create_table_config_with_column_mapping(schema, "name"); - let column_names = config.stats_column_names_physical(None /* required_columns */); + let column_names = config.physical_stats_column_names(None /* required_columns */); // Should return physical names, not logical names assert_eq!( column_names, - vec![ - ColumnName::new(["phys_col_a"]), - ColumnName::new(["phys_col_b"]), - ], + vec![column_name!("phys_col_a"), column_name!("phys_col_b"),], "Expected physical column names, not logical names" ); } #[test] - fn test_stats_column_names_physical_with_data_skipping_stats_columns() { + fn test_physical_stats_column_names_with_data_skipping_stats_columns() { let config = create_table_config_with_column_mapping_and_props( test_schema_nested_with_column_mapping(), "name", [("delta.dataSkippingStatsColumns", "id,info.name")], ); - let column_names = config.stats_column_names_physical(None); + let column_names = config.physical_stats_column_names(None); assert_eq!( column_names, - vec![ - ColumnName::new(["phys_id"]), - ColumnName::new(["phys_info", "phys_name"]), - ], + vec![column_name!("phys_id"), column_name!("phys_info.phys_name"),], ); } #[test] - fn test_stats_column_names_physical_skips_nonexistent_data_skipping_stats_column() { + fn test_physical_stats_column_names_skips_nonexistent_data_skipping_stats_column() { let config = create_table_config_with_column_mapping_and_props( test_schema_nested_with_column_mapping(), "name", [("delta.dataSkippingStatsColumns", "id,nonexistent")], ); - let column_names = config.stats_column_names_physical(None); - assert_eq!(column_names, vec![ColumnName::new(["phys_id"])],); + let column_names = config.physical_stats_column_names(None); + assert_eq!(column_names, vec![column_name!("phys_id")],); } #[rstest] @@ -1795,85 +2348,91 @@ mod test { #[case::flat_none( test_schema_flat(), "none", - vec![ColumnName::new(["id"]), ColumnName::new(["name"])], + vec![column_name!("id"), column_name!("name")], )] #[case::flat_name( test_schema_flat_with_column_mapping(), "name", - vec![ColumnName::new(["phys_id"]), ColumnName::new(["phys_name"])], + vec![column_name!("phys_id"), column_name!("phys_name")], )] #[case::flat_id( test_schema_flat_with_column_mapping(), "id", - vec![ColumnName::new(["phys_id"]), ColumnName::new(["phys_name"])], + vec![column_name!("phys_id"), column_name!("phys_name")], )] - // --- nested schema --- + // --- nested schema (includes map/array inside struct as leaf columns) --- #[case::nested_none( test_schema_nested(), "none", vec![ - ColumnName::new(["id"]), - ColumnName::new(["info", "name"]), - ColumnName::new(["info", "age"]), + column_name!("id"), + column_name!("info.name"), + column_name!("info.age"), + column_name!("info.tags"), + column_name!("info.scores"), ], )] #[case::nested_name( test_schema_nested_with_column_mapping(), "name", vec![ - ColumnName::new(["phys_id"]), - ColumnName::new(["phys_info", "phys_name"]), - ColumnName::new(["phys_info", "phys_age"]), + column_name!("phys_id"), + column_name!("phys_info.phys_name"), + column_name!("phys_info.phys_age"), + column_name!("phys_info.phys_tags"), + column_name!("phys_info.phys_scores"), ], )] #[case::nested_id( test_schema_nested_with_column_mapping(), "id", vec![ - ColumnName::new(["phys_id"]), - ColumnName::new(["phys_info", "phys_name"]), - ColumnName::new(["phys_info", "phys_age"]), + column_name!("phys_id"), + column_name!("phys_info.phys_name"), + column_name!("phys_info.phys_age"), + column_name!("phys_info.phys_tags"), + column_name!("phys_info.phys_scores"), ], )] - // --- schema with map (map fields excluded from stats) --- + // --- schema with map (included as leaf for nullCount stats) --- #[case::map_none( test_schema_with_map(), "none", - vec![ColumnName::new(["id"]), ColumnName::new(["name"])], + vec![column_name!("id"), column_name!("entries"), column_name!("name")], )] #[case::map_name( test_schema_with_map_and_column_mapping(), "name", - vec![ColumnName::new(["phys_id"]), ColumnName::new(["phys_name"])], + vec![column_name!("phys_id"), column_name!("phys_entries"), column_name!("phys_name")], )] #[case::map_id( test_schema_with_map_and_column_mapping(), "id", - vec![ColumnName::new(["phys_id"]), ColumnName::new(["phys_name"])], + vec![column_name!("phys_id"), column_name!("phys_entries"), column_name!("phys_name")], )] - // --- schema with array (array fields excluded from stats) --- + // --- schema with array (included as leaf for nullCount stats) --- #[case::array_none( test_schema_with_array(), "none", - vec![ColumnName::new(["id"]), ColumnName::new(["name"])], + vec![column_name!("id"), column_name!("items"), column_name!("name")], )] #[case::array_name( test_schema_with_array_and_column_mapping(), "name", - vec![ColumnName::new(["phys_id"]), ColumnName::new(["phys_name"])], + vec![column_name!("phys_id"), column_name!("phys_items"), column_name!("phys_name")], )] #[case::array_id( test_schema_with_array_and_column_mapping(), "id", - vec![ColumnName::new(["phys_id"]), ColumnName::new(["phys_name"])], + vec![column_name!("phys_id"), column_name!("phys_items"), column_name!("phys_name")], )] - fn test_stats_column_names_physical_all_schemas( + fn test_physical_stats_column_names_all_schemas( #[case] schema: SchemaRef, #[case] mode: &str, #[case] expected_physical: Vec, ) { let config = create_table_config_with_column_mapping(schema, mode); - let physical_names = config.stats_column_names_physical(None); + let physical_names = config.physical_stats_column_names(None); assert_eq!( physical_names, expected_physical, "Incorrect physical column names for mode '{mode}'" @@ -1892,4 +2451,511 @@ mod test { "ClusteredTable with DomainMetadata should be supported for writes" ); } + + // V3 supported + property set -> partition column materialized into the write schema; + // V3 supported but property unset -> partition column stripped from the write schema. + #[rstest] + #[case::v3_enabled( + &[(ENABLE_ICEBERG_COMPAT_V3, "true"), (ENABLE_ROW_TRACKING, "true")], + // pcol is included, meaning we expect the partition col to be materialized to disk. + vec!["value", "pcol"], + )] + #[case::v3_supported_but_property_unset(&[], vec!["value"])] + fn test_physical_write_schema_materializes_pv_when_iceberg_compat_v3_enabled( + #[case] extra_props: &[(&str, &str)], + #[case] expected_field_names: Vec<&str>, + ) { + // Partitioned schema: one data col + one partition col. No column mapping, so physical + // names equal logical names. + // IcebergCompatV3 requires column mapping. This test bypasses that requirement for + // convenience. + let schema = Arc::new(StructType::new_unchecked([ + StructField::nullable("value", DataType::INTEGER), + StructField::nullable("pcol", DataType::STRING), + ])); + let props: HashMap = extra_props + .iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect(); + let metadata = + Metadata::try_new(None, None, schema, vec!["pcol".to_string()], 0, props).unwrap(); + let protocol = Protocol::try_new( + TABLE_FEATURES_MIN_READER_VERSION, + TABLE_FEATURES_MIN_WRITER_VERSION, + Some(Vec::::new()), + Some(vec![ + TableFeature::IcebergCompatV3, + TableFeature::RowTracking, + TableFeature::DomainMetadata, + ]), + ) + .unwrap(); + let config = + TableConfiguration::try_new(metadata, protocol, Url::try_from("file:///").unwrap(), 0) + .unwrap(); + + let write_schema = config.physical_write_schema(); + // This is the final check: whether the partition column `pcol` is present in the + // physical schema as expected. + let field_names: Vec<&str> = write_schema.fields().map(|f| f.name.as_str()).collect(); + assert_eq!(field_names, expected_field_names); + } + + #[test] + fn test_physical_write_schema_strips_void_columns() { + // Pins the invariant that `physical_write_schema()` strips void columns from the + // returned schema (top level and nested), so callers receive a Parquet-writable + // schema without needing to apply `strip_void_from_schema` themselves. + let schema = Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("v", DataType::VOID), + StructField::nullable( + "s", + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("nested_void", DataType::VOID), + ]), + ), + ])); + let metadata = Metadata::try_new(None, None, schema, vec![], 0, HashMap::new()).unwrap(); + let protocol = Protocol::try_new( + TABLE_FEATURES_MIN_READER_VERSION, + TABLE_FEATURES_MIN_WRITER_VERSION, + Some(Vec::::new()), + Some(Vec::::new()), + ) + .unwrap(); + let config = + TableConfiguration::try_new(metadata, protocol, Url::try_from("file:///").unwrap(), 0) + .unwrap(); + + let write_schema = config.physical_write_schema(); + + // Top-level void is dropped. + let top_names: Vec<&str> = write_schema.fields().map(|f| f.name().as_str()).collect(); + assert_eq!(top_names, vec!["id", "s"]); + + // Nested void is also dropped, leaving only `a` inside `s`. + let s_field = write_schema.field("s").expect("s present after strip"); + let DataType::Struct(s_inner) = s_field.data_type() else { + panic!("s should still be a struct"); + }; + let s_names: Vec<&str> = s_inner.fields().map(|f| f.name().as_str()).collect(); + assert_eq!(s_names, vec!["a"]); + } + + #[test] + fn test_iceberg_compat_v3_write_supported() { + let config = create_mock_table_config_with_cm( + &[ + (ENABLE_ICEBERG_COMPAT_V3, "true"), + (ENABLE_ROW_TRACKING, "true"), + ], + Some(ColumnMappingMode::Name), + &[TableFeature::ColumnMapping], + &[ + TableFeature::IcebergCompatV3, + TableFeature::ColumnMapping, + TableFeature::RowTracking, + TableFeature::DomainMetadata, + ], + ); + config + .ensure_operation_supported(Operation::Write) + .expect("V3 write should be supported once kernel_support flips to Supported"); + } + + #[rstest] + #[case::unset(None, false)] + #[case::true_(Some("true"), true)] + #[case::false_(Some("false"), false)] + fn test_iceberg_compat_v3_enablement_follows_table_property( + #[case] property_value: Option<&str>, + #[case] expected_enabled: bool, + ) { + let extra = property_value + .map(|v| vec![(ENABLE_ICEBERG_COMPAT_V3, v)]) + .unwrap_or_default(); + let config = create_mock_table_config_with_cm( + &extra, + Some(ColumnMappingMode::Name), + &[TableFeature::ColumnMapping], + &[ + TableFeature::IcebergCompatV3, + TableFeature::ColumnMapping, + TableFeature::RowTracking, + TableFeature::DomainMetadata, + ], + ); + assert_eq!( + config.is_feature_enabled(&TableFeature::IcebergCompatV3), + expected_enabled, + ); + } + + #[rstest] + #[case::column_mapping_not_supported( + &[ + (ENABLE_ICEBERG_COMPAT_V3, "true"), + (ENABLE_ROW_TRACKING, "true"), + ], + None, + vec![], + vec![ + TableFeature::IcebergCompatV3, + TableFeature::RowTracking, + TableFeature::DomainMetadata, + ], + Some("requires 'columnMapping' to be enabled"), + )] + #[case::column_mapping_mode_none( + &[ + (ENABLE_ICEBERG_COMPAT_V3, "true"), + (ENABLE_ROW_TRACKING, "true"), + ], + Some(ColumnMappingMode::None), + vec![TableFeature::ColumnMapping], + vec![ + TableFeature::IcebergCompatV3, + TableFeature::ColumnMapping, + TableFeature::RowTracking, + TableFeature::DomainMetadata, + ], + // column mapping mode = none is considered as not enabled. + Some("requires 'columnMapping' to be enabled"), + )] + // RowTracking feature supported but `delta.enableRowTracking` is unset, so it's not enabled. + #[case::row_tracking_not_enabled( + &[(ENABLE_ICEBERG_COMPAT_V3, "true")], + Some(ColumnMappingMode::Name), + vec![TableFeature::ColumnMapping], + vec![ + TableFeature::IcebergCompatV3, + TableFeature::ColumnMapping, + TableFeature::RowTracking, + TableFeature::DomainMetadata, + ], + Some("requires 'rowTracking' to be enabled"), + )] + #[case::with_iceberg_compat_v1_enabled( + &[ + (ENABLE_ICEBERG_COMPAT_V3, "true"), + (ENABLE_ICEBERG_COMPAT_V1, "true"), + (ENABLE_ROW_TRACKING, "true"), + ], + Some(ColumnMappingMode::Name), + vec![TableFeature::ColumnMapping], + vec![ + TableFeature::IcebergCompatV3, + TableFeature::IcebergCompatV1, + TableFeature::ColumnMapping, + TableFeature::RowTracking, + TableFeature::DomainMetadata, + ], + Some("requires 'icebergCompatV1' to not be enabled"), + )] + #[case::with_iceberg_compat_v2_enabled( + &[ + (ENABLE_ICEBERG_COMPAT_V3, "true"), + (ENABLE_ICEBERG_COMPAT_V2, "true"), + (ENABLE_ROW_TRACKING, "true"), + ], + Some(ColumnMappingMode::Name), + vec![TableFeature::ColumnMapping], + vec![ + TableFeature::IcebergCompatV3, + TableFeature::IcebergCompatV2, + TableFeature::ColumnMapping, + TableFeature::RowTracking, + TableFeature::DomainMetadata, + ], + Some("requires 'icebergCompatV2' to not be enabled"), + )] + // Positive paths for both supported column-mapping modes (`name` and `id`). + #[case::all_satisfied_cm_name_mode( + &[ + (ENABLE_ICEBERG_COMPAT_V3, "true"), + (ENABLE_ROW_TRACKING, "true"), + ], + Some(ColumnMappingMode::Name), + vec![TableFeature::ColumnMapping], + vec![ + TableFeature::IcebergCompatV3, + TableFeature::ColumnMapping, + TableFeature::RowTracking, + TableFeature::DomainMetadata, + ], + None, + )] + #[case::all_satisfied_cm_id_mode( + &[ + (ENABLE_ICEBERG_COMPAT_V3, "true"), + (ENABLE_ROW_TRACKING, "true"), + ], + Some(ColumnMappingMode::Id), + vec![TableFeature::ColumnMapping], + vec![ + TableFeature::IcebergCompatV3, + TableFeature::ColumnMapping, + TableFeature::RowTracking, + TableFeature::DomainMetadata, + ], + None, + )] + fn test_iceberg_compat_v3_feature_requirements( + #[case] props: &[(&str, &str)], + #[case] cm_mode: Option, + #[case] reader_features: Vec, + #[case] writer_features: Vec, + #[case] expected_error_substring: Option<&str>, + ) { + let config = + create_mock_table_config_with_cm(props, cm_mode, &reader_features, &writer_features); + let result = config.validate_feature_requirements(&TableFeature::IcebergCompatV3); + match expected_error_substring { + Some(msg) => assert_result_error_with_message(result, msg), + None => assert!(result.is_ok(), "expected Ok, got {result:?}"), + } + } + + // `validate_feature_requirements` reads only `feature_requirements`, which is compiled + // regardless of the `adaptive-metadata-in-dev` gate, so these cases run in the default build. + // See the adaptiveMetadata RFC (delta-io/delta#6978) for the enablement rules being checked. + #[rstest] + #[case::all_satisfied( + all_adaptive_metadata_props(), + Some(ColumnMappingMode::Id), + all_adaptive_metadata_deps(), + None + )] + // Column mapping enabled but in `name` mode -> the `id`-mode Custom check fires. + #[case::cm_name_mode_rejected( + all_adaptive_metadata_props(), + Some(ColumnMappingMode::Name), + all_adaptive_metadata_deps(), + Some("column mapping in 'id' mode") + )] + // Column mapping feature absent entirely -> the `Enabled(ColumnMapping)` arm fires first. + #[case::column_mapping_not_supported( + all_adaptive_metadata_props(), + None, + adaptive_metadata_deps_without(TableFeature::ColumnMapping), + Some("requires 'columnMapping' to be enabled") + )] + #[case::row_tracking_not_enabled( + adaptive_metadata_props_without(ENABLE_ROW_TRACKING), + Some(ColumnMappingMode::Id), + all_adaptive_metadata_deps(), + Some("requires 'rowTracking' to be enabled") + )] + #[case::domain_metadata_not_supported( + all_adaptive_metadata_props(), + Some(ColumnMappingMode::Id), + adaptive_metadata_deps_without(TableFeature::DomainMetadata), + Some("requires 'domainMetadata' to be enabled") + )] + #[case::deletion_vectors_not_enabled( + adaptive_metadata_props_without(ENABLE_DELETION_VECTORS), + Some(ColumnMappingMode::Id), + all_adaptive_metadata_deps(), + Some("requires 'deletionVectors' to be enabled") + )] + #[case::in_commit_timestamp_not_enabled( + adaptive_metadata_props_without(ENABLE_IN_COMMIT_TIMESTAMPS), + Some(ColumnMappingMode::Id), + all_adaptive_metadata_deps(), + Some("requires 'inCommitTimestamp' to be enabled") + )] + fn test_adaptive_metadata_feature_requirements( + #[case] props: Vec<(&str, &str)>, + #[case] cm_mode: Option, + #[case] deps: Vec, + #[case] expected_error_substring: Option<&str>, + ) { + // Reader+writer features (adaptiveMetadata-preview itself, columnMapping, deletionVectors) + // must appear in both protocol lists to count as supported. + let reader_features: Vec = + std::iter::once(TableFeature::AdaptiveMetadataPreview) + .chain( + deps.iter() + .filter(|f| f.feature_type() == FeatureType::ReaderWriter) + .cloned(), + ) + .collect(); + let writer_features: Vec = + std::iter::once(TableFeature::AdaptiveMetadataPreview) + .chain(deps.iter().cloned()) + .collect(); + let config = + create_mock_table_config_with_cm(&props, cm_mode, &reader_features, &writer_features); + let result = config.validate_feature_requirements(&TableFeature::AdaptiveMetadataPreview); + match expected_error_substring { + Some(msg) => assert_result_error_with_message(result, msg), + None => assert!(result.is_ok(), "expected Ok, got {result:?}"), + } + } + + /// The table properties that enable adaptiveMetadata-preview's property-gated dependencies. + fn all_adaptive_metadata_props() -> Vec<(&'static str, &'static str)> { + vec![ + (ENABLE_ROW_TRACKING, "true"), + (ENABLE_DELETION_VECTORS, "true"), + (ENABLE_IN_COMMIT_TIMESTAMPS, "true"), + ] + } + + /// The adaptiveMetadata-preview enabling properties with `excluded` removed, to drive the + /// "dependency not enabled" requirement checks. + fn adaptive_metadata_props_without(excluded: &str) -> Vec<(&'static str, &'static str)> { + all_adaptive_metadata_props() + .into_iter() + .filter(|(k, _)| *k != excluded) + .collect() + } + + /// The full set of adaptiveMetadata-preview dependency features. + fn all_adaptive_metadata_deps() -> Vec { + vec![ + TableFeature::ColumnMapping, + TableFeature::DeletionVectors, + TableFeature::RowTracking, + TableFeature::DomainMetadata, + TableFeature::InCommitTimestamp, + ] + } + + /// The adaptiveMetadata-preview dependencies with `excluded` removed, to drive the + /// "dependency not supported" requirement checks. + fn adaptive_metadata_deps_without(excluded: TableFeature) -> Vec { + all_adaptive_metadata_deps() + .into_iter() + .filter(|f| *f != excluded) + .collect() + } + + // IcebergCompatV1/V2/V3 are pairwise mutually exclusive. + #[rstest] + #[case::v1_rejects_v2( + TableFeature::IcebergCompatV1, + TableFeature::IcebergCompatV2, + "requires 'icebergCompatV2' to not be enabled" + )] + #[case::v1_rejects_v3( + TableFeature::IcebergCompatV1, + TableFeature::IcebergCompatV3, + "requires 'icebergCompatV3' to not be enabled" + )] + #[case::v2_rejects_v1( + TableFeature::IcebergCompatV2, + TableFeature::IcebergCompatV1, + "requires 'icebergCompatV1' to not be enabled" + )] + #[case::v2_rejects_v3( + TableFeature::IcebergCompatV2, + TableFeature::IcebergCompatV3, + "requires 'icebergCompatV3' to not be enabled" + )] + #[case::v3_rejects_v1( + TableFeature::IcebergCompatV3, + TableFeature::IcebergCompatV1, + "requires 'icebergCompatV1' to not be enabled" + )] + #[case::v3_rejects_v2( + TableFeature::IcebergCompatV3, + TableFeature::IcebergCompatV2, + "requires 'icebergCompatV2' to not be enabled" + )] + fn test_iceberg_compat_mutual_exclusion( + #[case] feature_to_enable: TableFeature, + #[case] conflicting_feature: TableFeature, + #[case] expected_error_substring: &str, + ) { + // Map each IcebergCompat feature to the table property that enables it. + let conflicting_enable_property = match conflicting_feature { + TableFeature::IcebergCompatV1 => ENABLE_ICEBERG_COMPAT_V1, + TableFeature::IcebergCompatV2 => ENABLE_ICEBERG_COMPAT_V2, + TableFeature::IcebergCompatV3 => ENABLE_ICEBERG_COMPAT_V3, + ref other => panic!("unexpected feature in iceberg-compat exclusion test: {other:?}"), + }; + // V3 also requires Column mapping and RowTracking enabled; enable them unconditionally so + // V3 cases reach the mutual-exclusion check. + let config = create_mock_table_config_with_cm( + &[ + (conflicting_enable_property, "true"), + (ENABLE_ROW_TRACKING, "true"), + ], + Some(ColumnMappingMode::Name), + &[TableFeature::ColumnMapping], + &[ + feature_to_enable.clone(), + conflicting_feature, + TableFeature::ColumnMapping, + TableFeature::RowTracking, + TableFeature::DomainMetadata, + ], + ); + assert_result_error_with_message( + config.validate_feature_requirements(&feature_to_enable), + expected_error_substring, + ); + } + + /// `validate_feature_support_for_remove` must fire whenever row tracking is _supported_ + /// and not _suspended_, which is broader than _enabled_. + #[rstest] + #[case::supported_only(&[], Some("rowTracking"))] + #[case::supported_and_enabled(&[(ENABLE_ROW_TRACKING, "true")], Some("rowTracking"))] + #[case::supported_and_suspended(&[(ROW_TRACKING_SUSPENDED, "true")], None /*expected_error_substring */)] + fn test_validate_feature_support_for_remove_row_tracking( + #[case] props: &[(&str, &str)], + #[case] expected_error_substring: Option<&str>, + ) { + let config = create_mock_table_config(props, &[TableFeature::RowTracking]); + let result = config.validate_feature_support_for_remove(); + match expected_error_substring { + Some(msg) => assert_result_error_with_message(result, msg), + None => assert!(result.is_ok(), "expected Ok, got {result:?}"), + } + } + + /// Test helper: variant of `create_mock_table_config` that also takes an optional + /// column-mapping mode and requires the caller to provide reader and writer feature lists + /// explicitly. `name`/`id` modes need a column-mapping-annotated schema (otherwise + /// `TableConfiguration::try_new` rejects the metadata for missing per-field annotations); + /// this helper swaps the schema accordingly. + // TODO(#2491): Consolidate the `create_*_table_config*` helpers. + fn create_mock_table_config_with_cm( + extra_props: &[(&str, &str)], + cm_mode: Option, + reader_features: &[TableFeature], + writer_features: &[TableFeature], + ) -> TableConfiguration { + let schema: SchemaRef = match cm_mode { + Some(ColumnMappingMode::Name | ColumnMappingMode::Id) => schema_with_column_mapping(), + _ => schema_ref! { nullable "value": INTEGER }, + }; + let mut props: HashMap = extra_props + .iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect(); + if let Some(mode) = cm_mode { + let mode_str = match mode { + ColumnMappingMode::Name => "name", + ColumnMappingMode::Id => "id", + ColumnMappingMode::None => "none", + }; + props.insert(COLUMN_MAPPING_MODE.to_string(), mode_str.to_string()); + } + let metadata = Metadata::try_new(None, None, schema, vec![], 0, props).unwrap(); + + let protocol = Protocol::try_new( + TABLE_FEATURES_MIN_READER_VERSION, + TABLE_FEATURES_MIN_WRITER_VERSION, + Some(reader_features.to_vec()), + Some(writer_features.to_vec()), + ) + .unwrap(); + let table_root = Url::try_from("file:///").unwrap(); + TableConfiguration::try_new(metadata, protocol, table_root, 0).unwrap() + } } diff --git a/kernel/src/table_features/column_mapping.rs b/kernel/src/table_features/column_mapping.rs index 63d87da161..49c2987d2d 100644 --- a/kernel/src/table_features/column_mapping.rs +++ b/kernel/src/table_features/column_mapping.rs @@ -3,24 +3,24 @@ //! This module provides: //! - Read-side: Mode detection and schema validation //! - Write-side: Schema transformation for assigning IDs and physical names -use super::TableFeature; -use crate::actions::Protocol; -use crate::schema::{ - ArrayType, ColumnMetadataKey, ColumnName, DataType, MapType, MetadataValue, Schema, - StructField, StructType, -}; - -use crate::table_properties::{TableProperties, COLUMN_MAPPING_MODE}; -use crate::transforms::SchemaTransform; -use crate::{DeltaResult, Error}; - use std::borrow::Cow; use std::collections::HashMap; use serde::{Deserialize, Serialize}; use strum::EnumString; +use tracing::debug; use uuid::Uuid; +use super::TableFeature; +use crate::actions::Protocol; +use crate::schema::{ + ArrayType, ColumnMetadataKey, ColumnName, DataType, ExistingColumnMappingAnnotations, + MakePhysical, MapType, MetadataValue, Schema, StructField, StructType, +}; +use crate::table_properties::{TableProperties, COLUMN_MAPPING_MODE}; +use crate::transforms::{transform_output_type, SchemaTransform}; +use crate::{DeltaResult, Error}; + /// Modes of column mapping a table can be in #[derive(Debug, EnumString, Serialize, Deserialize, Copy, Clone, PartialEq, Eq)] #[strum(serialize_all = "camelCase")] @@ -34,6 +34,30 @@ pub enum ColumnMappingMode { Name, } +/// Maximum legal `delta.columnMapping.id` value per the Delta protocol, which restricts the id +/// to a 32-bit non-negative integer +/// (see ). Kernel +/// stores the id in `i64` end-to-end (`MetadataValue::Number` is `i64`) but enforces this bound +/// at the validation and allocation entry points so writers cannot persist out-of-range ids. +pub(crate) const MAX_COLUMN_MAPPING_ID: i64 = i32::MAX as i64; + +/// Validates that a `delta.columnMapping.id` value lies in the protocol-permitted range +/// `0..=`[`MAX_COLUMN_MAPPING_ID`]. Out-of-range ids (negative or above `i32::MAX`) are rejected +/// with a single canonical message naming the offending value and the protocol bound. Callers +/// that want to add context (e.g. the offending field name or table-property name) wrap with +/// `.map_err(|e| Error::schema(format!("Field '{name}': {e}")))?` -- matching the kernel +/// convention used by sibling validators in `schema/validation.rs`. +pub(crate) fn validate_column_mapping_id(id: i64) -> DeltaResult<()> { + if (0..=MAX_COLUMN_MAPPING_ID).contains(&id) { + return Ok(()); + } + let key = ColumnMetadataKey::ColumnMappingId.as_ref(); + Err(Error::schema(format!( + "Invalid column mapping id {id}: the Delta protocol restricts \ + `{key}` to a 32-bit non-negative integer (max {MAX_COLUMN_MAPPING_ID}).", + ))) +} + /// Determine the column mapping mode for a table based on the [`Protocol`] and [`TableProperties`] pub(crate) fn column_mapping_mode( protocol: &Protocol, @@ -52,25 +76,64 @@ pub(crate) fn column_mapping_mode( } } -/// When column mapping mode is enabled, verify that each field in the schema is annotated with a -/// physical name and field_id, and that no two fields share the same `delta.columnMapping.id` -/// value. When not enabled, verifies that no fields are annotated. +/// Validates `delta.columnMapping.id` and `delta.columnMapping.physicalName` annotations across +/// every field in `schema`. Aligns with delta-spark's validation logic. +/// +/// When `mode` is [`ColumnMappingMode::Id`] or [`ColumnMappingMode::Name`]: each field must +/// carry both annotations, no two fields may share a `delta.columnMapping.id`, and no two +/// fields may share a *full physical column path*. Two fields may share the same leaf +/// `physicalName` if they live at different physical column paths. +/// +/// When `mode` is [`ColumnMappingMode::None`]: verifies no field carries either annotation. +/// +/// Examples for physical name validation: +/// Rejected (two siblings share `delta.columnMapping.physicalName="x"`): +/// ```json +/// {"type":"struct","fields":[ +/// {"name":"a","type":"long","nullable":true, +/// "metadata":{"delta.columnMapping.id":1,"delta.columnMapping.physicalName":"x"}}, +/// {"name":"b","type":"long","nullable":true, +/// "metadata":{"delta.columnMapping.id":2,"delta.columnMapping.physicalName":"x"}} +/// ]} +/// ``` +/// +/// Accepted (same `delta.columnMapping.physicalName="x"` at different physical column paths): +/// ```json +/// {"type":"struct","fields":[ +/// {"name":"a","type":"long","nullable":true, +/// "metadata":{"delta.columnMapping.id":1,"delta.columnMapping.physicalName":"x"}}, +/// {"name":"nested","nullable":true, +/// "metadata":{"delta.columnMapping.id":2,"delta.columnMapping.physicalName":"nested"}, +/// "type":{"type":"struct","fields":[ +/// {"name":"a","type":"long","nullable":true, +/// "metadata":{"delta.columnMapping.id":3,"delta.columnMapping.physicalName":"x"}} +/// ]}} +/// ]} +/// ``` pub fn validate_schema_column_mapping(schema: &Schema, mode: ColumnMappingMode) -> DeltaResult<()> { - let mut validator = ValidateColumnMappings { - mode, - path: vec![], - seen: HashMap::new(), - err: None, - }; - let _ = validator.transform_struct(schema); - match validator.err { - Some(err) => Err(err), - None => Ok(()), - } + MakePhysical::validate_schema_column_mapping(mode, schema) +} + +/// How to treat a stale `delta.columnMapping.*` annotation found on a field while column mapping +/// is disabled ([`ColumnMappingMode::None`]). +/// +/// A table can carry residual annotations while mapping is off. They are inert -- physical names +/// resolve to logical names -- so reads tolerate them (matching delta-spark, which ignores them in +/// `NoMapping` mode). CREATE / ALTER strip the annotations a write newly introduces (so kernel +/// never *originates* a table in that shape), but leave residual ones already on the table in +/// place. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum StaleAnnotationPolicy { + /// Ignore a stale annotation and resolve the field by its logical name. + Ignore, + /// Reject a stale annotation with a schema error. + Reject, } /// Validates a field's column mapping annotations and extracts the physical name and column -/// mapping id. If `seen` is provided, also checks for duplicate column mapping IDs. +/// mapping id. If `seen_ids` is provided, also checks that this field's +/// `delta.columnMapping.id` is globally unique. If `current_field_siblings` is provided, also +/// checks that this field's `delta.columnMapping.physicalName` is unique among its siblings. /// /// Metadata columns are not subject to column mapping and must not carry column mapping /// annotations. Returns the logical field name and `None` for such fields. @@ -79,17 +142,71 @@ pub fn validate_schema_column_mapping(schema: &Schema, mode: ColumnMappingMode) /// `delta.columnMapping.physicalName` (string) and `delta.columnMapping.id` (number) annotation. /// Returns the physical name and `Some(id)`. /// -/// When disabled (`None`), neither annotation should be present. Returns the logical field name -/// and `None`. +/// When disabled (`None`), a stale annotation is handled per `stale_policy`: +/// [`StaleAnnotationPolicy::Ignore`] resolves the field by its logical name (returning the logical +/// name and `None`), while [`StaleAnnotationPolicy::Reject`] errors. Either way, in `None` mode no +/// dedup is performed (the returned "physical name" is just the logical field name and is only +/// schema-unique within its parent struct, not globally). +/// +/// # Parameters +/// +/// - `field`: The field to validate. +/// - `mode`: Column mapping mode. +/// - `stale_policy`: In `None` mode, whether to ignore or reject a stale `delta.columnMapping.*` +/// annotation. Unused when mapping is enabled. +/// - `parent_field_logical_path`: The field's parent path, used to render full paths in error +/// messages (e.g. parent `&["a", "b"]` and field `c` renders as `a.b.c`). +/// - `seen_ids`: Global map of `delta.columnMapping.id` -> first claimer logical name. `None` skips +/// the ID-dedup check. +/// - `current_field_siblings`: Map of `delta.columnMapping.physicalName` -> first claimer logical +/// name *for the current field's siblings only*. `None` skips the sibling-dedup check. +/// +/// # Errors /// -/// `path` identifies the field in error messages (e.g. `&["a", "b"]` renders as `a.b`). -pub(crate) fn get_field_column_mapping_info<'a>( +/// - The field is a metadata column carrying any CM annotation. +/// - CM is enabled but `delta.columnMapping.physicalName` is missing or non-string. +/// - CM is enabled but `delta.columnMapping.id` is missing or non-numeric. +/// - CM is disabled, `stale_policy` is [`StaleAnnotationPolicy::Reject`], and either annotation is +/// present. +/// - `seen_ids` is provided and the current field's `delta.columnMapping.id` is already in the map. +/// Example rejection (two fields share `delta.columnMapping.id=1`): +/// +/// ```json +/// {"type":"struct","fields":[ +/// {"name":"a","type":"long","nullable":true, +/// "metadata":{"delta.columnMapping.id":1,"delta.columnMapping.physicalName":"col-a"}}, +/// {"name":"b","type":"long","nullable":true, +/// "metadata":{"delta.columnMapping.id":1,"delta.columnMapping.physicalName":"col-b"}} +/// ]} +/// ``` +/// - `current_field_siblings` is provided and the current field's +/// `delta.columnMapping.physicalName` is already claimed by a sibling. Example rejection (two +/// siblings share `delta.columnMapping.physicalName="x"`): +/// +/// ```json +/// {"type":"struct","fields":[ +/// {"name":"a","type":"long","nullable":true, +/// "metadata":{"delta.columnMapping.id":1,"delta.columnMapping.physicalName":"x"}}, +/// {"name":"b","type":"long","nullable":true, +/// "metadata":{"delta.columnMapping.id":2,"delta.columnMapping.physicalName":"x"}} +/// ]} +/// ``` +pub(crate) fn validate_and_extract_column_mapping_annotations<'a>( field: &'a StructField, mode: ColumnMappingMode, - path: &[&str], - seen: Option<&mut HashMap>, + stale_policy: StaleAnnotationPolicy, + parent_field_logical_path: &[&'a str], + seen_ids: Option<&mut HashMap>, + current_field_siblings: Option<&mut HashMap<&'a str, &'a str>>, ) -> DeltaResult<(&'a str, Option)> { - let field_path = || ColumnName::new(path.iter().copied()); + let logical_field_path = || { + ColumnName::new( + parent_field_logical_path + .iter() + .copied() + .chain([field.name().as_str()]), + ) + }; let physical_name_meta = field .metadata .get(ColumnMetadataKey::ColumnMappingPhysicalName.as_ref()); @@ -114,21 +231,24 @@ pub(crate) fn get_field_column_mapping_info<'a>( (ColumnMappingMode::Name | ColumnMappingMode::Id, Some(_)) => { return Err(Error::schema(format!( "The {annotation} annotation on field '{}' must be a string", - field_path(), + logical_field_path(), ))); } (ColumnMappingMode::Name | ColumnMappingMode::Id, None) => { return Err(Error::schema(format!( "Column mapping is enabled but field '{}' lacks the {annotation} annotation", - field_path(), - ))); - } - (ColumnMappingMode::None, Some(_)) => { - return Err(Error::schema(format!( - "Column mapping is not enabled but field '{}' is annotated with {annotation}", - field_path(), + logical_field_path(), ))); } + (ColumnMappingMode::None, Some(_)) => match stale_policy { + StaleAnnotationPolicy::Ignore => field.name(), + StaleAnnotationPolicy::Reject => { + return Err(Error::schema(format!( + "Column mapping is not enabled but field '{}' is annotated with {annotation}", + logical_field_path(), + ))); + } + }, }; let annotation = ColumnMetadataKey::ColumnMappingId.as_ref(); @@ -140,81 +260,56 @@ pub(crate) fn get_field_column_mapping_info<'a>( (ColumnMappingMode::Name | ColumnMappingMode::Id, Some(_)) => { return Err(Error::schema(format!( "The {annotation} annotation on field '{}' must be a number", - field_path(), + logical_field_path(), ))); } (ColumnMappingMode::Name | ColumnMappingMode::Id, None) => { return Err(Error::schema(format!( "Column mapping is enabled but field '{}' lacks the {annotation} annotation", - field_path(), - ))); - } - (ColumnMappingMode::None, Some(_)) => { - return Err(Error::schema(format!( - "Column mapping is not enabled but field '{}' is annotated with {annotation}", - field_path(), + logical_field_path(), ))); } + (ColumnMappingMode::None, Some(_)) => match stale_policy { + StaleAnnotationPolicy::Ignore => None, + StaleAnnotationPolicy::Reject => { + return Err(Error::schema(format!( + "Column mapping is not enabled but field '{}' is annotated with {annotation}", + logical_field_path(), + ))); + } + }, }; - if let (Some(id), Some(seen)) = (id, seen) { - seen.insert(id, field.name()).map_or(Ok(()), |prev| { - Err(Error::schema(format!( - "Duplicate column mapping ID {id} assigned to both '{prev}' and '{}'", - field.name() - ))) - })?; - } - - Ok((physical_name, id)) -} - -struct ValidateColumnMappings<'a> { - mode: ColumnMappingMode, - path: Vec<&'a str>, - seen: HashMap, // column mapping id -> first field name that claimed it - err: Option, -} - -impl<'a> ValidateColumnMappings<'a> { - fn transform_inner(&mut self, field_name: &'a str, validate: impl FnOnce(&mut Self) -> R) { - if self.err.is_none() { - self.path.push(field_name); - let _ = validate(self); - self.path.pop(); + // CM-disabled mode synthesizes `physical_name = field.name()`, which only has to be unique + // within its parent struct, not globally -- so dedup is gated on CM being enabled. ID dedup + // additionally requires `Some(id)` because `id` is `None` outside CM-enabled mode. + if mode != ColumnMappingMode::None { + if let (Some(id), Some(seen_ids)) = (id, seen_ids) { + seen_ids.insert(id, field.name()).map_or(Ok(()), |prev| { + Err(Error::schema(format!( + "Duplicate column mapping ID {id} assigned to both '{prev}' and '{}'", + field.name() + ))) + })?; + } + // Dedup `physicalName` among siblings. Two distinct columns with the same full + // physical path must diverge at some ancestor struct, where two siblings share + // the same `physicalName`. + if let Some(siblings) = current_field_siblings { + siblings + .insert(physical_name.as_str(), field.name().as_str()) + .map_or(Ok(()), |prev| { + Err(Error::schema(format!( + "Duplicate `delta.columnMapping.physicalName` '{physical_name}' \ + assigned to both '{}' and '{}'", + ColumnName::new(parent_field_logical_path.iter().copied().chain([prev])), + logical_field_path(), + ))) + })?; } } -} -impl<'a> SchemaTransform<'a> for ValidateColumnMappings<'a> { - // Override array element and map key/value for better error messages - fn transform_array_element(&mut self, etype: &'a DataType) -> Option> { - self.transform_inner("", |this| this.transform(etype)); - Some(Cow::Borrowed(etype)) - } - fn transform_map_key(&mut self, ktype: &'a DataType) -> Option> { - self.transform_inner("", |this| this.transform(ktype)); - Some(Cow::Borrowed(ktype)) - } - fn transform_map_value(&mut self, vtype: &'a DataType) -> Option> { - self.transform_inner("", |this| this.transform(vtype)); - Some(Cow::Borrowed(vtype)) - } - fn transform_struct_field(&mut self, field: &'a StructField) -> Option> { - self.transform_inner(field.name(), |this| { - get_field_column_mapping_info(field, this.mode, &this.path, Some(&mut this.seen)) - .map_err(|e| this.err = Some(e)) - .ok()?; - this.recurse_into_struct_field(field) - }); - None - } - fn transform_variant(&mut self, _: &'a StructType) -> Option> { - // don't recurse into variant's fields, as they are not expected to have column mapping - // annotations - // TODO: this changes with icebergcompat right? see issue#1125 for icebergcompat. - None - } + Ok((physical_name, id)) } // ============================================================================ @@ -240,111 +335,503 @@ pub(crate) fn get_column_mapping_mode_from_properties( } } -/// Assigns column mapping metadata (id and physicalName) to all fields in a schema. +/// Assigns column mapping metadata (id and physicalName) to all fields in a schema, +/// matching delta-spark's `DeltaColumnMapping.assignColumnIdAndPhysicalName` semantics. /// /// This function recursively processes all fields in the schema, including nested structs, -/// arrays, and maps. Each field is assigned a new unique ID and physical name. +/// arrays, and maps. For each field: +/// - If both `delta.columnMapping.id` and `delta.columnMapping.physicalName` are present, they are +/// preserved verbatim and `max_id` is advanced past the preserved id. +/// - If only `id` is present, `physicalName` is filled in as `col-` and the id is preserved. +/// - If only `physicalName` is present, a new `id` is allocated as `*max_id + 1`. +/// - If neither is present, both are assigned. +/// +/// Wrong-typed annotations (`id` not a number, `physicalName` not a string), an empty +/// `physicalName`, and a negative `id` error out. The latter two are stricter than +/// delta-spark: an empty physical name fails PROTOCOL.md's "globally unique identifier" +/// requirement and would break Parquet column resolution, and a negative id is rejected so +/// `(*max_id).max(n)` cannot silently no-op below the allocator's seed. /// -/// Fields with pre-existing column mapping metadata (id or physicalName) are rejected -/// to avoid conflicts. ALTER TABLE will need different handling in the future. +/// Pre-populated `delta.columnMapping.nested.ids` and `parquet.field.nested.ids` annotations +/// are always rejected: those payloads are kernel-managed and only emitted by the nested-ids +/// assignment pass below. +/// +/// When `assign_nested_field_ids` is `true`, after the per-field flat assignment this function +/// also allocates fresh parquet field ids for the synthetic `Array.element`, +/// `Map.key`, and `Map.value` slots and stores them under +/// `delta.columnMapping.nested.ids` on the nearest ancestor `StructField`. Allocating these +/// nested ids is a hard requirement for IcebergCompatV2/3. +/// +/// Callers should seed `max_id` from [`find_max_column_id_in_schema`] so newly assigned +/// IDs do not collide with preserved ones. Duplicate preserved IDs are detected by +/// [`crate::schema::StructType::make_physical`], which runs from +/// `TableConfiguration::try_new[_with_schema]` after assignment. /// /// # Arguments /// -/// * `schema` - The schema to transform -/// * `max_id` - Tracks the highest column ID assigned. Updated in place. Should be initialized -/// to 0 for a new table. +/// * `schema` - The schema to transform. +/// * `max_id` - Tracks the highest column ID. Read and updated in place. Should be seeded from +/// [`find_max_column_id_in_schema`] (or `0` for a brand-new table with no preserved IDs anywhere) +/// so newly assigned IDs cannot collide with preserved ones. +/// * `assign_nested_field_ids` - When `true`, also allocates IDs for synthetic `Array.element` and +/// `Map.key`/`Map.value` fields, stores them in `delta.columnMapping.nested.ids`, and includes +/// them in `max_id`. Assigning IDs to these nested fields is a hard requirement for +/// IcebergCompatV2/3. /// /// # Returns /// -/// A new schema with column mapping metadata on all fields. +/// A new schema with column mapping metadata present on every field. +/// +/// # Example +/// +/// Given a top-level field `m: map, int>` with no pre-existing annotations, after +/// calling with `assign_nested_field_ids = true` the field gets (`` is the assigned +/// UUID-based physical name): +/// +/// ```json +/// { +/// "delta.columnMapping.id": 1, +/// "delta.columnMapping.physicalName": "", +/// "delta.columnMapping.nested.ids": { +/// ".key": 2, +/// ".key.element": 3, +/// ".value": 4 +/// } +/// } +/// ``` +/// +/// `max_id` ends at `4`. With `assign_nested_field_ids = false` the same field gets only the +/// flat CM metadata and `max_id` ends at `1`: +/// +/// ```json +/// { +/// "delta.columnMapping.id": 1, +/// "delta.columnMapping.physicalName": "" +/// } +/// ``` +#[delta_kernel_derive::internal_api] pub(crate) fn assign_column_mapping_metadata( schema: &StructType, max_id: &mut i64, + assign_nested_field_ids: bool, ) -> DeltaResult { + // Assign CM id + physical name to every StructField (top-level and nested). + // `delta.columnMapping.nested.ids` is assigned later separately, this is to + // align with delta-spark's behavior. let new_fields: Vec = schema .fields() - .map(|field| assign_field_column_mapping(field, max_id)) + .map(|field| try_assign_flat_column_mapping_info(field, max_id)) .collect::>>()?; - - StructType::try_new(new_fields) + let with_flat_cm_info = StructType::try_new(new_fields)?; + if !assign_nested_field_ids { + return Ok(with_flat_cm_info); + } + // Then populate `delta.columnMapping.nested.ids` per StructField. + assign_nested_cm_ids(&with_flat_cm_info, max_id) } -/// Assigns column mapping metadata to a single field, recursively processing nested types. +/// JSON object holding [`ColumnMetadataKey::ColumnMappingNestedIds`] entries for an Array/Map +/// field. +type NestedFieldIds = serde_json::Map; + +/// Advances `max_id` by one and returns the new value, surfacing `i64` overflow as an error. /// -/// Rejects fields with pre-existing column mapping metadata. Otherwise, assigns a new -/// unique ID and physical name (incrementing `max_id`). -fn assign_field_column_mapping(field: &StructField, max_id: &mut i64) -> DeltaResult { - let has_id = field - .get_config_value(&ColumnMetadataKey::ColumnMappingId) - .is_some(); - let has_physical_name = field - .get_config_value(&ColumnMetadataKey::ColumnMappingPhysicalName) - .is_some(); - - // For CREATE TABLE, reject any pre-existing column mapping metadata. - // This avoids conflicts between user-provided IDs/physical names and the ones we assign. - // ALTER TABLE (adding columns) will need different handling in the future. - // TODO: Also check for nested column IDs (`delta.columnMapping.nested.ids`) once - // Iceberg compatibility (IcebergCompatV2+) is supported. See issue #1125. - if has_id || has_physical_name { +/// `assign_column_mapping_metadata` allocates fresh column-mapping IDs as `max_id + 1`. A +/// connector that preserves an `id` near `i64::MAX` and then asks kernel to allocate a fresh +/// sibling would silently wrap to `i64::MIN` without this guard, producing a negative id that +/// would then trip the negative-id check on the next round-trip. +/// +/// In practice this should be unreachable: the Delta protocol restricts `columnMapping.id` to a +/// 32-bit non-negative integer (`i32::MAX` ~ 2.1B), so anything near `i64::MAX` (~ 9.2 +/// quintillion) is over four billion times the protocol-permitted maximum and points at a bug +/// in the connector's id allocator rather than legitimate id exhaustion. The error message +/// reflects that diagnosis. +fn next_column_mapping_id(max_id: &mut i64) -> DeltaResult { + let next = max_id.checked_add(1).ok_or_else(|| { + Error::generic(format!( + "Cannot allocate column mapping id: `max_id + 1` overflows `i64` \ + (max_id={current}). The Delta protocol caps `delta.columnMapping.id` at a \ + 32-bit non-negative integer; fix the upstream id allocator producing \ + out-of-range preserved ids.", + current = *max_id, + )) + })?; + if next > MAX_COLUMN_MAPPING_ID { return Err(Error::generic(format!( - "Field '{}' already has column mapping metadata. \ - Pre-existing column mapping metadata is not supported for CREATE TABLE.", - field.name + "Cannot allocate column mapping id {next}: exceeds the Delta protocol's 32-bit \ + non-negative maximum ({MAX_COLUMN_MAPPING_ID}). A preserved \ + `delta.columnMapping.id` is at the protocol cap; lower the largest preserved \ + id so kernel can allocate a sibling within range.", ))); } + *max_id = next; + Ok(next) +} + +/// Assigns flat column mapping metadata (id and physical name) to a single field, recursively +/// processing nested types. Implements the 3-way preserve/fill/assign behavior described on +/// [`assign_column_mapping_metadata`], matching delta-spark. The returned field always has both +/// `delta.columnMapping.id` and `delta.columnMapping.physicalName` present. +/// +/// `delta.columnMapping.nested.ids` is not assigned here; the separate nested-ids pass run by +/// [`assign_column_mapping_metadata`] handles that. +/// +/// `max_id` is advanced as follows: +/// - When a `delta.columnMapping.id` is preserved, `*max_id = max(*max_id, preserved_id)`. +/// - When a new id is allocated, `*max_id` is incremented via [`next_column_mapping_id`] (which +/// uses `checked_add`) and the new id is the post-increment value. +/// +/// # Errors +/// +/// - Field carries a pre-existing `delta.columnMapping.nested.ids` or `parquet.field.nested.ids` +/// annotation (those payloads are kernel-managed). +/// - A pre-existing `delta.columnMapping.id` annotation is wrong-typed or negative. +/// - A pre-existing `delta.columnMapping.physicalName` annotation is wrong-typed or empty. +/// - Allocating a fresh id would overflow `i64`. +pub(crate) fn try_assign_flat_column_mapping_info( + field: &StructField, + max_id: &mut i64, +) -> DeltaResult { + for key in [ + ColumnMetadataKey::ColumnMappingNestedIds, + ColumnMetadataKey::ParquetFieldNestedIds, + ] { + if field.get_config_value(&key).is_some() { + return Err(Error::generic(format!( + "Field '{}' has pre-populated `{}` metadata; this annotation is \ + kernel-managed and must not be supplied on input fields.", + field.name, + key.as_ref(), + ))); + } + } + + let ExistingColumnMappingAnnotations { id, physical_name } = + field.validate_and_extract_existing_column_mapping_annotations()?; - // Start with the existing field and assign new ID let mut new_field = field.clone(); - *max_id += 1; - new_field.metadata.insert( - ColumnMetadataKey::ColumnMappingId.as_ref().to_string(), - MetadataValue::Number(*max_id), - ); - // Assign physical name - let physical_name = format!("col-{}", Uuid::new_v4()); - new_field.metadata.insert( - ColumnMetadataKey::ColumnMappingPhysicalName - .as_ref() - .to_string(), - MetadataValue::String(physical_name), - ); + match (id, physical_name) { + // Both present: preserve verbatim and advance `max_id` past the preserved id. + (Some(preserved_id), Some(_)) => { + *max_id = (*max_id).max(preserved_id); + } + // Only `id` present: preserve id, fill in physical name. + (Some(preserved_id), None) => { + *max_id = (*max_id).max(preserved_id); + new_field.metadata.insert( + ColumnMetadataKey::ColumnMappingPhysicalName + .as_ref() + .to_string(), + MetadataValue::String(format!("col-{}", Uuid::new_v4())), + ); + } + // Only `physicalName` present: preserve name, allocate id. + (None, Some(_)) => { + let new_id = next_column_mapping_id(max_id)?; + new_field.metadata.insert( + ColumnMetadataKey::ColumnMappingId.as_ref().to_string(), + MetadataValue::Number(new_id), + ); + } + // Neither present: assign both. + (None, None) => { + let new_id = next_column_mapping_id(max_id)?; + new_field.metadata.insert( + ColumnMetadataKey::ColumnMappingId.as_ref().to_string(), + MetadataValue::Number(new_id), + ); + new_field.metadata.insert( + ColumnMetadataKey::ColumnMappingPhysicalName + .as_ref() + .to_string(), + MetadataValue::String(format!("col-{}", Uuid::new_v4())), + ); + } + } - // Recursively process nested types - new_field.data_type = process_nested_data_type(&field.data_type, max_id)?; + // Recursively process nested types (struct/array/map descend; primitive/variant pass through). + new_field.data_type = flat_cm_info_for_nested_data_type(&field.data_type, max_id)?; Ok(new_field) } -/// Process nested data types to assign column mapping metadata to any nested struct fields. -fn process_nested_data_type(data_type: &DataType, max_id: &mut i64) -> DeltaResult { +/// Process nested data types to assign flat column mapping metadata to any nested struct fields. +fn flat_cm_info_for_nested_data_type( + data_type: &DataType, + max_id: &mut i64, +) -> DeltaResult { match data_type { DataType::Struct(inner) => { - let new_inner = assign_column_mapping_metadata(inner, max_id)?; - Ok(DataType::Struct(Box::new(new_inner))) + let new_inner = assign_column_mapping_metadata( + inner, max_id, /* assign_nested_field_ids */ false, + )?; + Ok(DataType::from(new_inner)) } DataType::Array(array_type) => { - let new_element_type = process_nested_data_type(array_type.element_type(), max_id)?; - Ok(DataType::Array(Box::new(ArrayType::new( + let new_element_type = + flat_cm_info_for_nested_data_type(array_type.element_type(), max_id)?; + Ok(DataType::from(ArrayType::new( new_element_type, array_type.contains_null(), - )))) + ))) } DataType::Map(map_type) => { - let new_key_type = process_nested_data_type(map_type.key_type(), max_id)?; - let new_value_type = process_nested_data_type(map_type.value_type(), max_id)?; - Ok(DataType::Map(Box::new(MapType::new( + let new_key_type = flat_cm_info_for_nested_data_type(map_type.key_type(), max_id)?; + let new_value_type = flat_cm_info_for_nested_data_type(map_type.value_type(), max_id)?; + Ok(DataType::from(MapType::new( new_key_type, new_value_type, map_type.value_contains_null(), - )))) + ))) } // Primitive and Variant types don't contain nested struct fields - return as-is DataType::Primitive(_) | DataType::Variant(_) => Ok(data_type.clone()), } } +/// Walks `schema` (already carrying CM id + physical name on every StructField) and populates +/// `delta.columnMapping.nested.ids` on Array/Map fields. +/// +/// `delta.columnMapping.nested.ids` is a JSON object on a StructField that records the parquet +/// field ids for the synthetic Array `element` and Map `key`/`value` slots inside that field's +/// subtree. Keys are dotted paths anchored at the field's physical name; values are parquet field +/// ids. +/// +/// Example for `m: Map, int>` with physical name ``: +/// +/// ```json +/// { +/// ".key": 2, +/// ".key.element": 3, +/// ".value": 4 +/// } +/// ``` +fn assign_nested_cm_ids(schema: &StructType, max_id: &mut i64) -> DeltaResult { + fn walk( + data_type: &DataType, + max_id: &mut i64, + path: &str, + nested_ids: &mut NestedFieldIds, + ) -> DeltaResult { + match data_type { + DataType::Struct(inner) => Ok(DataType::from(assign_nested_cm_ids(inner, max_id)?)), + DataType::Array(array_type) => { + let element_path = format!("{path}.element"); + *max_id += 1; + nested_ids.insert(element_path.clone(), serde_json::Value::from(*max_id)); + let new_element = + walk(array_type.element_type(), max_id, &element_path, nested_ids)?; + Ok(DataType::from(ArrayType::new( + new_element, + array_type.contains_null(), + ))) + } + DataType::Map(map_type) => { + let key_path = format!("{path}.key"); + let value_path = format!("{path}.value"); + *max_id += 1; + nested_ids.insert(key_path.clone(), serde_json::Value::from(*max_id)); + let new_key = walk(map_type.key_type(), max_id, &key_path, nested_ids)?; + *max_id += 1; + nested_ids.insert(value_path.clone(), serde_json::Value::from(*max_id)); + let new_value = walk(map_type.value_type(), max_id, &value_path, nested_ids)?; + Ok(DataType::from(MapType::new( + new_key, + new_value, + map_type.value_contains_null(), + ))) + } + DataType::Primitive(_) | DataType::Variant(_) => Ok(data_type.clone()), + } + } + + let new_fields: Vec = schema + .fields() + .map(|field| { + let physical = expect_physical_name(field)?; + let mut nested_ids = NestedFieldIds::new(); + let new_dt = walk(&field.data_type, max_id, &physical, &mut nested_ids)?; + let mut new_field = field.clone(); + new_field.data_type = new_dt; + if !nested_ids.is_empty() { + insert_nested_field_ids_metadata(&mut new_field, nested_ids); + } + Ok(new_field) + }) + .collect::>>()?; + StructType::try_new(new_fields) +} + +// Get the physical name for a field. Error if the field is missing the physical name annotation. +fn expect_physical_name(field: &StructField) -> DeltaResult { + match field + .metadata + .get(ColumnMetadataKey::ColumnMappingPhysicalName.as_ref()) + { + Some(MetadataValue::String(s)) => Ok(s.clone()), + _ => Err(Error::internal_error(format!( + "Expect field '{}' to have a physical name annotation", + field.name, + ))), + } +} + +/// Sets the collected nested ids on `field` under the `delta.columnMapping.nested.ids` key. +fn insert_nested_field_ids_metadata(field: &mut StructField, ids: NestedFieldIds) { + field.metadata.insert( + ColumnMetadataKey::ColumnMappingNestedIds + .as_ref() + .to_string(), + MetadataValue::Other(serde_json::Value::Object(ids)), + ); +} + +/// Returns the largest column mapping id found anywhere in `schema`. This includes both +/// per-field `delta.columnMapping.id` annotations and the nested ids in +/// `delta.columnMapping.nested.ids` metadata. +#[delta_kernel_derive::internal_api] +pub(crate) fn find_max_column_id_in_schema(schema: &StructType) -> Option { + let mut visitor = MaxColumnId(None); + visitor.transform_struct(schema); + visitor.0 +} + +/// Visitor that walks a schema and records the largest column mapping id seen on any field, +/// counting both `delta.columnMapping.id` (per-field) and the integer values inside any +/// `delta.columnMapping.nested.ids` JSON map (for element/key/value of Array/Map). +struct MaxColumnId(Option); + +impl MaxColumnId { + fn observe(&mut self, n: i64) { + self.0 = Some(self.0.map_or(n, |prev| prev.max(n))); + } +} + +impl<'a> SchemaTransform<'a> for MaxColumnId { + transform_output_type!(|'a, T| ()); + + fn transform_struct_field(&mut self, field: &'a StructField) { + if let Some(n) = field.column_mapping_id() { + self.observe(n); + } + // `delta.columnMapping.nested.ids` is a JSON object set on Array/Map fields. Shape: { + // ".key": , ".value": , ".key.element": , ... } + // (string paths -> nested column-mapping ids). See the definition of + // `ColumnMetadataKey::ColumnMappingNestedIds` for more details. + if let Some(MetadataValue::Other(serde_json::Value::Object(obj))) = field + .metadata() + .get(ColumnMetadataKey::ColumnMappingNestedIds.as_ref()) + { + for v in obj.values() { + if let Some(n) = v.as_i64() { + self.observe(n); + } + } + } + // Recurse into the field's data type so we also visit nested struct/array/map members. + self.recurse_into_struct_field(field) + } +} + +/// The field-metadata keys that are meaningful only when column mapping is enabled. The detector +/// [`schema_has_column_mapping_metadata`] and the stripper [`drop_column_mapping_metadata`] both +/// key off this exact set, matching the `COLUMN_MAPPING_METADATA_KEYS` delta-spark strips and +/// detects together. +const COLUMN_MAPPING_METADATA_KEYS: &[ColumnMetadataKey] = &[ + ColumnMetadataKey::ColumnMappingId, + ColumnMetadataKey::ColumnMappingPhysicalName, + ColumnMetadataKey::ColumnMappingNestedIds, + ColumnMetadataKey::ParquetFieldId, + ColumnMetadataKey::ParquetFieldNestedIds, +]; + +/// Returns whether any field in `schema` (recursively) carries any [`COLUMN_MAPPING_METADATA_KEYS`] +/// annotation. Detection is by key presence (not value type), so a malformed annotation still +/// counts -- the detector and [`drop_column_mapping_metadata`] must agree on what "carries column +/// mapping metadata" means, else a write could strip a field the detector deemed clean (or persist +/// one it missed). +pub(crate) fn schema_has_column_mapping_metadata(schema: &StructType) -> bool { + struct HasCmMetadata(bool); + impl<'a> SchemaTransform<'a> for HasCmMetadata { + transform_output_type!(|'a, T| ()); + fn transform_struct_field(&mut self, field: &'a StructField) { + // Once a match is found the answer can't change, so skip the per-field key scan (and + // recursion) for the rest of the walk. The `SchemaTransform` trait has no early-exit + // hook, so nodes are still visited; this just short-circuits the work at each. + if self.0 { + return; + } + if COLUMN_MAPPING_METADATA_KEYS + .iter() + .any(|key| field.metadata.contains_key(key.as_ref())) + { + self.0 = true; + return; + } + self.recurse_into_struct_field(field) + } + } + let mut visitor = HasCmMetadata(false); + visitor.transform_struct(schema); + visitor.0 +} + +/// Strips the stray column-mapping annotations a write introduces into a previously-clean table, +/// returning `Some(stripped_schema)` when a strip is needed and `None` when `candidate` can be +/// persisted as-is. +/// +/// A strip is needed only when `candidate` carries column-mapping annotations that the pre-write +/// schema did not -- i.e. the write is introducing them into a clean table, so persisting +/// `candidate` verbatim would originate a self-inconsistent table. When the pre-write schema +/// already carried residual annotations they are left in place (matching delta-spark, which strips +/// only annotations a commit newly introduces); `None` is returned. +/// +/// Callers MUST gate this on `ColumnMappingMode::None`: with mapping enabled the annotations are +/// load-bearing and must never be stripped. `current_has_cm` is whether the pre-write schema +/// carried any column-mapping metadata (always `false` for CREATE, which has no prior schema); +/// passing the bool rather than the schema lets ALTER avoid cloning its pre-alter schema. +/// `candidate` is the schema the write would otherwise persist. +pub(crate) fn strip_stray_column_mapping_metadata( + current_has_cm: bool, + candidate: &StructType, +) -> Option { + (!current_has_cm && schema_has_column_mapping_metadata(candidate)).then(|| { + debug!( + "Column mapping is disabled; stripping newly-introduced `delta.columnMapping.*` \ + annotations from the schema before persisting." + ); + drop_column_mapping_metadata(candidate) + }) +} + +/// Removes every [`COLUMN_MAPPING_METADATA_KEYS`] annotation from every field in `schema` +/// (recursively), leaving all other field metadata intact. +pub(crate) fn drop_column_mapping_metadata(schema: &StructType) -> StructType { + struct DropCmMetadata; + impl<'a> SchemaTransform<'a> for DropCmMetadata { + transform_output_type!(|'a, T| Cow<'a, T>); + fn transform_struct_field(&mut self, field: &'a StructField) -> Cow<'a, StructField> { + let data_type = self.transform(&field.data_type); + let mut metadata = field.metadata.clone(); + let mut had_cm = false; + for key in COLUMN_MAPPING_METADATA_KEYS { + had_cm |= metadata.remove(key.as_ref()).is_some(); + } + match data_type { + Cow::Borrowed(_) if !had_cm => Cow::Borrowed(field), + data_type => Cow::Owned(StructField { + name: field.name.clone(), + data_type: data_type.into_owned(), + nullable: field.is_nullable(), + metadata, + }), + } + } + } + DropCmMetadata.transform_struct(schema).into_owned() +} + /// Translates a logical [`ColumnName`] to physical. It can be top level or nested. /// /// Uses `StructType::walk_column_fields` to walk the column path through nested structs, @@ -360,7 +847,7 @@ pub(crate) fn get_any_level_column_physical_name( col_name: &ColumnName, column_mapping_mode: ColumnMappingMode, ) -> DeltaResult { - let fields = schema.walk_column_fields(col_name)?; + let fields = schema.fields_of_path(col_name)?; let physical_path: Vec = fields .iter() .map(|field| -> DeltaResult { @@ -387,15 +874,47 @@ pub(crate) fn get_any_level_column_physical_name( Ok(ColumnName::new(physical_path)) } +/// Convert a physical column name to a logical column name (with the leaf field's [`DataType`]) +/// by walking the schema. +/// +/// Walks the schema once, matching each path component by `physical_name(mode)`, and returns the +/// resolved logical [`ColumnName`] together with the data type of the final (leaf) field. +pub(crate) fn physical_to_logical_column_name_and_type( + logical_schema: &StructType, + physical_col: &ColumnName, + column_mapping_mode: ColumnMappingMode, +) -> DeltaResult<(ColumnName, DataType)> { + let mut fields = vec![]; + logical_schema.visit_fields_of_path_by( + physical_col, + |s, phys_name| { + s.fields() + .find(|f| f.physical_name(column_mapping_mode) == phys_name) + }, + |field| fields.push(field), + )?; + // `visit_fields_of_path_by` rejects an empty path and otherwise pushes one field per + // component, so on success `fields` is non-empty and its last element is the leaf. + let leaf_type = fields + .last() + .ok_or_else(|| Error::generic(format!("Column path '{physical_col}' resolved no fields")))? + .data_type() + .clone(); + Ok((ColumnName::new(fields.iter().map(|f| &f.name)), leaf_type)) +} + #[cfg(test)] mod tests { - use super::*; - use crate::expressions::ColumnName; - use crate::schema::{DataType, StructType}; - use crate::utils::test_utils::make_test_tc; use std::collections::{HashMap, HashSet}; - use crate::utils::test_utils::test_deep_nested_schema_missing_leaf_cm; + use super::*; + use crate::expressions::{column_name, ColumnName}; + use crate::schema::{DataType, MetadataValue, StructField, StructType}; + use crate::unit_test_utils::{ + assert_result_error_with_message, column_mapping_physical_name_dedup_fixtures as fixtures, + make_test_tc, test_deep_nested_schema_missing_leaf_cm, + }; + use crate::utils::FoldWithOption as _; #[test] fn test_column_mapping_mode() { @@ -470,7 +989,8 @@ mod tests { assert_eq!(tc.column_mapping_mode(), ColumnMappingMode::None); } - // Creates optional schema field annotations for column mapping id and physical name, as a string. + // Creates optional schema field annotations for column mapping id and physical name, as a + // string. fn create_annotations<'a>( id: impl Into>, name: impl Into>, @@ -485,7 +1005,8 @@ mod tests { annotations.join(", ") } - // Creates a generic schema with optional field annotations for column mapping id and physical name. + // Creates a generic schema with optional field annotations for column mapping id and physical + // name. fn create_schema<'a>( inner_id: impl Into>, inner_name: impl Into>, @@ -554,6 +1075,9 @@ mod tests { }); } + // `validate_schema_column_mapping` is the strict validator: a stale annotation on a + // mapping-disabled table is rejected. The lenient path (used by reads) is covered by + // `test_validate_and_extract_tolerates_stale_annotations_when_disabled`. #[test] fn test_column_mapping_disabled() { let schema = create_schema(None, None, None, None); @@ -569,6 +1093,61 @@ mod tests { validate_schema_column_mapping(&schema, ColumnMappingMode::None).expect_err("field name"); } + /// A field carrying a stale `delta.columnMapping.*` annotation while mapping is disabled is + /// tolerated under [`StaleAnnotationPolicy::Ignore`] (the read path) -- it resolves to the + /// logical name with no id -- and rejected under [`StaleAnnotationPolicy::Reject`] (the write + /// path). + #[rstest::rstest] + #[case::physical_name_only(None, Some("col-abc"))] + #[case::id_only(Some(7), None)] + #[case::both(Some(7), Some("col-abc"))] + fn test_validate_and_extract_tolerates_stale_annotations_when_disabled( + #[case] stale_id: Option, + #[case] stale_physical_name: Option<&str>, + ) { + let mut metadata: Vec<(&str, MetadataValue)> = Vec::new(); + if let Some(id) = stale_id { + metadata.push(( + ColumnMetadataKey::ColumnMappingId.as_ref(), + MetadataValue::Number(id), + )); + } + if let Some(name) = stale_physical_name { + metadata.push(( + ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), + MetadataValue::String(name.to_string()), + )); + } + let field = StructField::not_null("stale", DataType::INTEGER).with_metadata(metadata); + + // Ignore: resolves to the logical name, no id. + let (physical_name, id) = validate_and_extract_column_mapping_annotations( + &field, + ColumnMappingMode::None, + StaleAnnotationPolicy::Ignore, + &[], + None, + None, + ) + .unwrap(); + assert_eq!(physical_name, "stale"); + assert_eq!(id, None); + + // Reject: errors, naming the field. + assert_result_error_with_message( + validate_and_extract_column_mapping_annotations( + &field, + ColumnMappingMode::None, + StaleAnnotationPolicy::Reject, + &[], + None, + None, + ) + .map(|_| ()), + "Column mapping is not enabled but field 'stale'", + ); + } + #[test] fn test_annotation_validation_reaches_struct_fields_in_map_value() { let unannotated = @@ -576,16 +1155,135 @@ mod tests { let schema = StructType::new_unchecked([make_cm_field( "b", 1, - MapType::new( - DataType::STRING, - DataType::Struct(Box::new(unannotated)), - false, - ), + MapType::new(DataType::STRING, unannotated, false), )]); validate_schema_column_mapping(&schema, ColumnMappingMode::Id) .expect_err("missing annotation on struct field inside map value"); } + /// Every key in `COLUMN_MAPPING_METADATA_KEYS` counts as column-mapping metadata, whether it + /// sits on a top-level field or a nested struct leaf. + #[rstest::rstest] + fn test_schema_has_column_mapping_metadata_detects_each_key( + #[values( + ColumnMetadataKey::ColumnMappingId, + ColumnMetadataKey::ColumnMappingPhysicalName, + ColumnMetadataKey::ColumnMappingNestedIds, + ColumnMetadataKey::ParquetFieldId, + ColumnMetadataKey::ParquetFieldNestedIds + )] + key: ColumnMetadataKey, + ) { + // Top-level field carrying only this key. + let top_level = StructType::new_unchecked([StructField::nullable("a", DataType::INTEGER) + .add_metadata([(key.as_ref(), MetadataValue::Number(1))])]); + assert!(schema_has_column_mapping_metadata(&top_level)); + + // Same key on a nested struct leaf (clean top level). + let nested = StructType::new_unchecked([StructField::nullable( + "outer", + StructType::new_unchecked([StructField::nullable("leaf", DataType::INTEGER) + .add_metadata([(key.as_ref(), MetadataValue::Number(1))])]), + )]); + assert!(schema_has_column_mapping_metadata(&nested)); + } + + #[test] + fn test_schema_has_column_mapping_metadata_edge_cases() { + // Clean schema -> false. + let clean = StructType::new_unchecked([StructField::nullable("a", DataType::INTEGER)]); + assert!(!schema_has_column_mapping_metadata(&clean)); + + // Detection is by key presence, not value type: a wrong-typed id still counts (it must, so + // the stripper -- which removes by presence -- never leaves a key the detector deemed + // absent). + let wrong_typed = + StructType::new_unchecked([StructField::nullable("a", DataType::INTEGER) + .add_metadata([( + ColumnMetadataKey::ColumnMappingId.as_ref(), + MetadataValue::String("not-a-number".to_string()), + )])]); + assert!(schema_has_column_mapping_metadata(&wrong_typed)); + } + + #[test] + fn test_drop_column_mapping_metadata() { + // Top-level annotated field also carrying an unrelated key; nested annotated leaf. + let annotated_top = StructField::nullable("a", DataType::INTEGER).add_metadata([ + ( + ColumnMetadataKey::ColumnMappingId.as_ref(), + MetadataValue::Number(1), + ), + ( + ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), + MetadataValue::String("col-a".to_string()), + ), + ("delta.identity.start", MetadataValue::Number(7)), + ]); + let schema = StructType::new_unchecked([ + annotated_top, + StructField::nullable( + "outer", + StructType::new_unchecked([make_cm_field("leaf", 2, DataType::INTEGER)]), + ), + ]); + + let stripped = drop_column_mapping_metadata(&schema); + assert!(!schema_has_column_mapping_metadata(&stripped)); + + // Unrelated metadata is preserved. + let a = stripped.field("a").unwrap(); + assert!(a.column_mapping_id().is_none()); + assert!(!a.has_physical_name_annotation()); + assert_eq!( + a.metadata().get("delta.identity.start"), + Some(&MetadataValue::Number(7)) + ); + + // Nested leaf is stripped too. + let DataType::Struct(outer) = stripped.field("outer").unwrap().data_type() else { + panic!("expected struct"); + }; + assert!(outer.field("leaf").unwrap().column_mapping_id().is_none()); + } + + #[test] + fn test_drop_column_mapping_metadata_strips_array_and_map_nested_leaves() { + // Annotated struct leaves living inside an array element and a map value. + let elem = StructType::new_unchecked([make_cm_field("in_arr", 3, DataType::INTEGER)]); + let val = StructType::new_unchecked([make_cm_field("in_map", 4, DataType::INTEGER)]); + let schema = StructType::new_unchecked([ + StructField::nullable("arr", ArrayType::new(elem, true)), + StructField::nullable("m", MapType::new(DataType::STRING, val, true)), + ]); + assert!(schema_has_column_mapping_metadata(&schema)); + + let stripped = drop_column_mapping_metadata(&schema); + assert!(!schema_has_column_mapping_metadata(&stripped)); + } + + #[test] + fn test_strip_stray_column_mapping_metadata() { + // Mode gating lives at the call sites; this helper only decides strip-vs-leave from the + // current-vs-candidate comparison. + let clean = StructType::new_unchecked([StructField::nullable("a", DataType::INTEGER)]); + let annotated = StructType::new_unchecked([make_cm_field("a", 1, DataType::INTEGER)]); + + // CREATE (no prior schema) introducing annotations -> stripped. + let stripped = strip_stray_column_mapping_metadata(false, &annotated) + .expect("newly-introduced annotations should be stripped"); + assert!(!schema_has_column_mapping_metadata(&stripped)); + + // ALTER on a clean table introducing annotations -> stripped. + assert!(strip_stray_column_mapping_metadata(false, &annotated).is_some()); + + // ALTER on an already-annotated table -> left in place (no strip). + assert!(strip_stray_column_mapping_metadata(true, &annotated).is_none()); + + // Candidate carries no annotations -> nothing to strip. + assert!(strip_stray_column_mapping_metadata(false, &clean).is_none()); + } + fn make_cm_field(name: &str, id: i64, data_type: impl Into) -> StructField { StructField::new(name, data_type, false).with_metadata([ ( @@ -611,18 +1309,14 @@ mod tests { make_cm_field("x", 5, DataType::INTEGER), make_cm_field("y", 5, DataType::INTEGER), ]); - StructType::new_unchecked([make_cm_field( - "outer", - 10, - DataType::Struct(Box::new(nested)), - )]) + StructType::new_unchecked([make_cm_field("outer", 10, nested)]) } fn cm_schema_cross_level_duplicates() -> StructType { let nested = StructType::new_unchecked([make_cm_field("inner", 1, DataType::INTEGER)]); StructType::new_unchecked([ make_cm_field("a", 1, DataType::INTEGER), - make_cm_field("b", 2, DataType::Struct(Box::new(nested))), + make_cm_field("b", 2, nested), ]) } @@ -630,11 +1324,7 @@ mod tests { let element = StructType::new_unchecked([make_cm_field("x", 1, DataType::INTEGER)]); StructType::new_unchecked([ make_cm_field("a", 1, DataType::INTEGER), - make_cm_field( - "b", - 2, - ArrayType::new(DataType::Struct(Box::new(element)), false), - ), + make_cm_field("b", 2, ArrayType::new(element, false)), ]) } @@ -642,11 +1332,7 @@ mod tests { let value = StructType::new_unchecked([make_cm_field("x", 1, DataType::INTEGER)]); StructType::new_unchecked([ make_cm_field("a", 1, DataType::INTEGER), - make_cm_field( - "b", - 2, - MapType::new(DataType::STRING, DataType::Struct(Box::new(value)), false), - ), + make_cm_field("b", 2, MapType::new(DataType::STRING, value, false)), ]) } @@ -657,7 +1343,7 @@ mod tests { #[case::across_array(cm_schema_array_duplicates())] #[case::across_map(cm_schema_map_duplicates())] fn test_duplicate_column_mapping_ids_rejected(#[case] schema: StructType) { - crate::utils::test_utils::assert_result_error_with_message( + assert_result_error_with_message( validate_schema_column_mapping(&schema, ColumnMappingMode::Id), "Duplicate column mapping ID", ); @@ -665,7 +1351,7 @@ mod tests { #[test] fn test_duplicate_column_mapping_ids_rejected_in_name_mode() { - crate::utils::test_utils::assert_result_error_with_message( + assert_result_error_with_message( validate_schema_column_mapping( &cm_schema_same_level_duplicates(), ColumnMappingMode::Name, @@ -674,6 +1360,46 @@ mod tests { ); } + #[rstest::rstest] + #[case::accepted_distinct_paths(fixtures::same_phy_name_different_paths(), None)] + #[case::rejected_deeply_nested_repeat( + fixtures::deeply_nested_repeat_physical_paths(), + Some({ + let (a, b) = + fixtures::deeply_nested_collider_paths(); + format!("assigned to both '{a}' and '{b}'") + }), + )] + #[case::multiple_violations_reports_first( + fixtures::multiple_physical_name_collisions(), + Some("'p' assigned to both 'a' and 'b'".to_string()), + )] + fn test_dup_physical_name( + #[case] schema: StructType, + #[case] expected_error_substring: Option, + ) { + // The same dedup rules should apply under both CM modes. + for mode in [ColumnMappingMode::Name, ColumnMappingMode::Id] { + let result = validate_schema_column_mapping(&schema, mode); + match &expected_error_substring { + None => { + result.expect("schema must validate"); + } + Some(substr) => { + assert_result_error_with_message(result.as_ref().map(|_| ()), substr); + // For the multiple-violations case, also confirm the deeper site was never + // reported. + if let Err(e) = &result { + assert!( + !e.to_string().contains("'q'"), + "walker must short-circuit on first collision under {mode:?}; got: {e}" + ); + } + } + } + } + } + // ========================================================================= // Tests for write-side column mapping functions // ========================================================================= @@ -701,68 +1427,387 @@ mod tests { } } - #[test] - fn test_assign_column_mapping_metadata_simple() { - let schema = StructType::new_unchecked([ - StructField::new("a", DataType::INTEGER, false), - StructField::new("b", DataType::STRING, true), - ]); + /// Schema shapes used to dimensionalize tests over field nesting depth. + #[derive(Clone, Copy, Debug)] + enum SchemaShape { + /// Field under test is a top-level sibling of an unannotated field. + Flat, + /// Field under test lives inside a 1-level nested struct, with a top-level unannotated + /// sibling. + NestedStruct, + /// Field under test lives 2 levels deep inside nested structs, with a top-level + /// unannotated sibling. + DeeplyNestedStruct, + } - let mut max_id = 0; - let result = assign_column_mapping_metadata(&schema, &mut max_id).unwrap(); + const FIELD_UNDER_TEST: &str = "field_under_test"; - // Should have assigned IDs 1 and 2 - assert_eq!(max_id, 2); - assert_eq!(result.fields().count(), 2); + /// Builds the field whose pre-existing column-mapping annotations vary by test case. + fn make_field_under_test(pre_id: Option, pre_name: Option<&str>) -> StructField { + let mut metadata: Vec<(&str, MetadataValue)> = Vec::new(); + if let Some(id) = pre_id { + metadata.push(( + ColumnMetadataKey::ColumnMappingId.as_ref(), + MetadataValue::Number(id), + )); + } + if let Some(name) = pre_name { + metadata.push(( + ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), + MetadataValue::String(name.to_string()), + )); + } + let field = StructField::not_null(FIELD_UNDER_TEST, DataType::INTEGER); + if metadata.is_empty() { + field + } else { + field.add_metadata(metadata) + } + } - // Check both fields have metadata - for (i, field) in result.fields().enumerate() { - let expected_id = (i + 1) as i64; + /// Wraps the field under test according to `shape`, alongside a top-level unannotated + /// sibling. Returns the schema and path that resolves back to the field under test. + fn build_schema_with_field_under_test( + shape: SchemaShape, + field_under_test: StructField, + ) -> (StructType, Vec) { + let path = |segments: &[&str]| segments.iter().map(|s| s.to_string()).collect(); + match shape { + SchemaShape::Flat => { + let schema = StructType::new_unchecked([ + StructField::nullable("unannotated_sibling", DataType::STRING), + field_under_test, + ]); + (schema, path(&[FIELD_UNDER_TEST])) + } + SchemaShape::NestedStruct => { + let inner = StructType::new_unchecked([field_under_test]); + let schema = StructType::new_unchecked([ + StructField::nullable("unannotated_sibling", DataType::STRING), + StructField::nullable("outer", inner), + ]); + (schema, path(&["outer", FIELD_UNDER_TEST])) + } + SchemaShape::DeeplyNestedStruct => { + let innermost = StructType::new_unchecked([field_under_test]); + let middle = + StructType::new_unchecked([StructField::nullable("middle", innermost)]); + let schema = StructType::new_unchecked([ + StructField::nullable("unannotated_sibling", DataType::STRING), + StructField::nullable("outer", middle), + ]); + (schema, path(&["outer", "middle", FIELD_UNDER_TEST])) + } + } + } + + /// Happy-path dispatch coverage for [`assign_column_mapping_metadata`]. Each `#[case]` + /// exercises one of the 4 dispatch arms (preserve-both / preserve-id-fill-name / + /// preserve-name-allocate-id / assign-both), including `id = 0` to verify that 0 is + /// accepted as a preserved id (matches delta-spark; symmetric with `maxColumnId = 0` + /// table property). The schema-shape `#[values]` axis confirms the same dispatch behavior + /// at every nesting depth. + #[rstest::rstest] + #[case::preserve_both(Some(100), Some("preserved-name"))] + #[case::preserve_id_only(Some(7), None)] + #[case::preserve_name_only(None, Some("user-supplied"))] + #[case::preserve_id_zero(Some(0), Some("zero-name"))] + #[case::neither_preserved(None, None)] + fn test_assign_column_mapping_metadata_dispatch( + #[case] pre_id: Option, + #[case] pre_name: Option<&str>, + #[values( + SchemaShape::Flat, + SchemaShape::NestedStruct, + SchemaShape::DeeplyNestedStruct + )] + shape: SchemaShape, + ) { + let field_under_test = make_field_under_test(pre_id, pre_name); + let (schema, field_under_test_path) = + build_schema_with_field_under_test(shape, field_under_test); + + let seed = find_max_column_id_in_schema(&schema).unwrap_or(0); + let mut max_id = seed; + let result = assign_column_mapping_metadata(&schema, &mut max_id, false).unwrap(); + + let result_field = result.field_at_path(&field_under_test_path); + let result_id = result_field + .column_mapping_id() + .expect("expected numeric column mapping id"); + let result_name = expect_physical_name(result_field).unwrap(); + + match pre_id { + Some(id) => assert_eq!(result_id, id, "preserved id must round-trip"), + None => assert!( + result_id > seed, + "allocated id {result_id} must exceed seed {seed}", + ), + } + match pre_name { + Some(expected) => { + assert_eq!(result_name, expected, "preserved name must round-trip") + } + None => assert!( + result_name.starts_with("col-"), + "generated name should start with `col-`, got {result_name:?}", + ), + } + } + + /// Validates delta-spark's seeding rule for [`assign_column_mapping_metadata`]. + /// + /// delta-spark's `DeltaColumnMapping.assignColumnIdAndPhysicalName` seeds: + /// maxColumnId = max(currentMaxColumnId, findMaxColumnId(schema)) + /// then assigns each new id as `maxColumnId += 1`. Consequence: every newly assigned + /// id is strictly greater than `findMaxColumnId(schema)`, and preserved ids round-trip + /// unchanged. + /// + /// Each `#[case]` supplies a different set of preserved ids to confirm the seed advances + /// correctly regardless of value sparsity (sparse positives, sequential, with `0`, + /// single-max). + #[rstest::rstest] + #[case::sparse_positives(vec![1, 5, 100])] + #[case::with_zero(vec![0, 5, 100])] + #[case::single_max(vec![100])] + #[case::sequential(vec![1, 2, 3])] + #[case::single_zero(vec![0])] + fn test_assign_column_mapping_metadata_seed_avoids_collisions(#[case] preserved_ids: Vec) { + let preserved_max = *preserved_ids.iter().max().unwrap(); + const UNANNOTATED_FIELD_NAMES: [&str; 3] = + ["unannotated_1", "unannotated_2", "unannotated_3"]; + + // Build schema: unannotated_1, [preserved_0..N], unannotated_2, unannotated_3. + let mut fields = vec![StructField::nullable( + UNANNOTATED_FIELD_NAMES[0], + DataType::STRING, + )]; + for (i, id) in preserved_ids.iter().enumerate() { + let name = format!("preserved_{i}"); + fields.push( + StructField::not_null(&name, DataType::INTEGER).add_metadata([ + ( + ColumnMetadataKey::ColumnMappingId.as_ref(), + MetadataValue::Number(*id), + ), + ( + ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), + MetadataValue::String(format!("p-{name}")), + ), + ]), + ); + } + fields.extend( + UNANNOTATED_FIELD_NAMES[1..] + .iter() + .map(|n| StructField::nullable(*n, DataType::STRING)), + ); + let schema = StructType::new_unchecked(fields); + + let mut max_id = find_max_column_id_in_schema(&schema).unwrap_or(0); + assert_eq!(max_id, preserved_max, "seed should equal the preserved max"); + + let result = assign_column_mapping_metadata(&schema, &mut max_id, false).unwrap(); + // Three unannotated fields -> +3 ids above the seed. + assert_eq!(max_id, preserved_max + UNANNOTATED_FIELD_NAMES.len() as i64); + + // Every assigned id on an unannotated field exceeds the preserved max. + for name in UNANNOTATED_FIELD_NAMES { + let field = result.field(name).unwrap(); + let id = field + .column_mapping_id() + .expect("expected numeric column mapping id"); + assert!( + id > preserved_max, + "newly assigned id {id} for '{name}' must exceed preserved max {preserved_max}", + ); + } + + // Preserved ids round-trip. + for (i, expected_id) in preserved_ids.iter().enumerate() { + let name = format!("preserved_{i}"); assert_eq!( - field.get_config_value(&ColumnMetadataKey::ColumnMappingId), - Some(&MetadataValue::Number(expected_id)) + result + .field(&name) + .unwrap() + .column_mapping_id() + .expect("expected numeric column mapping id"), + *expected_id, + "preserved id at '{name}' must round-trip", ); - assert!(field - .get_config_value(&ColumnMetadataKey::ColumnMappingPhysicalName) - .is_some()); + } + } - // Verify physical name format (col-{uuid}) - if let Some(MetadataValue::String(name)) = - field.get_config_value(&ColumnMetadataKey::ColumnMappingPhysicalName) - { - assert!( - name.starts_with("col-"), - "Physical name should start with 'col-'" - ); - } + /// Each case supplies a (possibly partial, possibly malformed) annotation pair and asserts + /// the error names both the offending key and the kind of violation. + #[rstest::rstest] + #[case::id_wrong_type_string( + Some(MetadataValue::String("not-a-number".to_string())), + Some(MetadataValue::String("p".to_string())), + "non-numeric", + ColumnMetadataKey::ColumnMappingId, + )] + #[case::name_wrong_type_number( + None, + Some(MetadataValue::Number(7)), + "non-string", + ColumnMetadataKey::ColumnMappingPhysicalName + )] + #[case::name_empty_no_id( + None, + Some(MetadataValue::String(String::new())), + "empty", + ColumnMetadataKey::ColumnMappingPhysicalName + )] + #[case::name_empty_with_id( + Some(MetadataValue::Number(5)), + Some(MetadataValue::String(String::new())), + "empty", + ColumnMetadataKey::ColumnMappingPhysicalName + )] + #[case::id_negative_with_name( + Some(MetadataValue::Number(-7)), + Some(MetadataValue::String("p".to_string())), + "Invalid column mapping id", + ColumnMetadataKey::ColumnMappingId, + )] + #[case::id_i64_min_no_name( + Some(MetadataValue::Number(i64::MIN)), + None, + "Invalid column mapping id", + ColumnMetadataKey::ColumnMappingId + )] + fn test_assign_column_mapping_metadata_rejects_invalid_annotations( + #[case] id: Option, + #[case] name: Option, + #[case] violation_kind: &str, + #[case] offending_key: ColumnMetadataKey, + #[values( + SchemaShape::Flat, + SchemaShape::NestedStruct, + SchemaShape::DeeplyNestedStruct + )] + shape: SchemaShape, + ) { + let mut metadata = Vec::new(); + if let Some(id) = id { + metadata.push((ColumnMetadataKey::ColumnMappingId.as_ref(), id)); } + if let Some(name) = name { + metadata.push((ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), name)); + } + let bad = StructField::not_null(FIELD_UNDER_TEST, DataType::INTEGER).add_metadata(metadata); + let (schema, _) = build_schema_with_field_under_test(shape, bad); + + let mut max_id = 0; + let err = assign_column_mapping_metadata(&schema, &mut max_id, false) + .unwrap_err() + .to_string(); + let key = offending_key.as_ref(); + assert!( + err.contains(violation_kind) && err.contains(key), + "Expected '{violation_kind}' and '{key}' in error, got: {err}", + ); } - #[test] - fn test_assign_column_mapping_metadata_rejects_existing_id() { - // Schema with pre-existing column mapping metadata should be rejected - let schema = StructType::new_unchecked([ - StructField::new("a", DataType::INTEGER, false).add_metadata([ - ( - ColumnMetadataKey::ColumnMappingId.as_ref(), - MetadataValue::Number(100), - ), - ( - ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), - MetadataValue::String("existing-physical".to_string()), - ), - ]), - StructField::new("b", DataType::STRING, true), - ]); + /// `next_column_mapping_id` must reject `i64` overflow rather than wrapping to a negative + /// id. Both fresh-allocation arms (`physicalName`-only and neither-preserved) flow through + /// it, so each is exercised here. Without `checked_add` a connector that preserved an id + /// near `i64::MAX` and asked kernel to allocate a sibling would wrap silently and produce + /// a negative id that would then trip the negative-id validator on the next round-trip. + #[rstest::rstest] + #[case::physical_name_only(Some("preserved-name"))] + #[case::neither_annotation(None)] + fn test_assign_column_mapping_metadata_errors_when_id_allocation_overflows( + #[case] preserved_physical_name: Option<&str>, + ) { + let field = make_field_under_test(None, preserved_physical_name); + let schema = StructType::new_unchecked([field]); + + let mut max_id = i64::MAX; + let err = assign_column_mapping_metadata(&schema, &mut max_id, false) + .unwrap_err() + .to_string(); + assert!( + err.contains("Cannot allocate column mapping id") + && err.contains("overflows `i64`") + && err.contains("32-bit non-negative integer") + && err.contains(&i64::MAX.to_string()), + "Expected overflow error citing the i64 ceiling, the protocol's 32-bit bound, \ + and the offending `max_id`; got: {err}", + ); + // `max_id` must not be advanced past `i64::MAX` on failure. + assert_eq!(max_id, i64::MAX); + } + + /// `i32::MAX` is the protocol's largest legal `delta.columnMapping.id`. A connector that + /// preserves it must round-trip; a fresh sibling allocated above it must be rejected with + /// the protocol-bound message (not the `i64` overflow message). This pins both halves of + /// the bound to one rstest so the negative space is contiguous. + #[rstest::rstest] + #[case::preserved_at_protocol_max_round_trips(i32::MAX as i64, true)] + #[case::preserved_above_protocol_max_rejected(i32::MAX as i64 + 1, false)] + fn test_assign_column_mapping_metadata_enforces_protocol_32bit_bound( + #[case] preserved_id: i64, + #[case] expect_ok: bool, + ) { + let field_under_test = make_field_under_test(Some(preserved_id), Some("preserved-name")); + let (schema, _) = build_schema_with_field_under_test(SchemaShape::Flat, field_under_test); let mut max_id = 0; - let result = assign_column_mapping_metadata(&schema, &mut max_id); + let result = assign_column_mapping_metadata(&schema, &mut max_id, false); + if expect_ok { + let schema = result.expect("preserved id at protocol max must round-trip"); + assert_eq!( + schema + .field(FIELD_UNDER_TEST) + .unwrap() + .column_mapping_id() + .expect("expected numeric column mapping id"), + i32::MAX as i64, + ); + assert_eq!(max_id, i32::MAX as i64); + } else { + let err = result.unwrap_err().to_string(); + assert!( + err.contains("Invalid column mapping id") + && err.contains(&MAX_COLUMN_MAPPING_ID.to_string()), + "Expected canonical out-of-range error citing the 32-bit max, got: {err}", + ); + } + } - assert!(result.is_err()); - let err_msg = result.unwrap_err().to_string(); + /// Allocating a sibling next to a preserved id at the protocol cap must error with the + /// allocator-flavored bound message (distinct from the `i64::MAX` overflow message; the + /// `checked_add` succeeds, then the bound check trips). + #[test] + fn test_assign_column_mapping_metadata_errors_when_allocation_would_exceed_protocol_max() { + // Field 1 preserves id = i32::MAX; field 2 has only physicalName so allocation fires. + let preserved = StructField::not_null("preserved", DataType::INTEGER).add_metadata([ + ( + ColumnMetadataKey::ColumnMappingId.as_ref(), + MetadataValue::Number(i32::MAX as i64), + ), + ( + ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), + MetadataValue::String("preserved-name".to_string()), + ), + ]); + let needs_allocation = StructField::not_null("needs_alloc", DataType::INTEGER) + .add_metadata([( + ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), + MetadataValue::String("user-supplied".to_string()), + )]); + let schema = StructType::new_unchecked([preserved, needs_allocation]); + + let mut max_id = find_max_column_id_in_schema(&schema).unwrap_or(0); + let err = assign_column_mapping_metadata(&schema, &mut max_id, false) + .unwrap_err() + .to_string(); assert!( - err_msg.contains("already has column mapping metadata"), - "Expected error about existing column mapping metadata, got: {err_msg}" + err.contains("Cannot allocate column mapping id") + && err.contains("exceeds the Delta protocol's 32-bit non-negative maximum"), + "Expected allocator-flavored protocol-bound error, got: {err}", ); } @@ -775,11 +1820,11 @@ mod tests { let schema = StructType::new_unchecked([ StructField::new("a", DataType::INTEGER, false), - StructField::new("nested", DataType::Struct(Box::new(inner)), true), + StructField::new("nested", inner, true), ]); let mut max_id = 0; - let result = assign_column_mapping_metadata(&schema, &mut max_id).unwrap(); + let result = assign_column_mapping_metadata(&schema, &mut max_id, false).unwrap(); // Should have assigned IDs to all 4 fields assert_eq!(max_id, 4); @@ -819,32 +1864,23 @@ mod tests { seen_ids: &mut HashSet, seen_physical_names: &mut HashSet, ) { - let id = field - .get_config_value(&ColumnMetadataKey::ColumnMappingId) - .unwrap_or_else(|| panic!("Field '{}' should have column mapping ID", field.name)); - let MetadataValue::Number(id_val) = id else { + let id_val = field.column_mapping_id().unwrap_or_else(|| { panic!( - "Field '{}' column mapping ID should be a number", + "Field '{}' should have a numeric column mapping ID", field.name - ); - }; + ) + }); assert!( - seen_ids.insert(*id_val), - "Duplicate column mapping ID {} on field '{}'", - id_val, + seen_ids.insert(id_val), + "Duplicate column mapping ID {id_val} on field '{}'", field.name ); - let physical = field - .get_config_value(&ColumnMetadataKey::ColumnMappingPhysicalName) - .unwrap_or_else(|| panic!("Field '{}' should have physical name", field.name)); - let MetadataValue::String(physical_name) = physical else { - panic!("Field '{}' physical name should be a string", field.name); - }; + let physical_name = expect_physical_name(field) + .unwrap_or_else(|e| panic!("Field '{}' should have physical name: {e}", field.name)); assert!( seen_physical_names.insert(physical_name.clone()), - "Duplicate physical name '{}' on field '{}'", - physical_name, + "Duplicate physical name '{physical_name}' on field '{}'", field.name ); } @@ -867,20 +1903,12 @@ mod tests { let value_struct = StructType::new_unchecked([StructField::new("v", DataType::INTEGER, false)]); - let map_type = MapType::new( - DataType::Struct(Box::new(key_struct)), - DataType::Struct(Box::new(value_struct)), - true, - ); + let map_type = MapType::new(key_struct, value_struct, true); - let schema = StructType::new_unchecked([StructField::new( - "my_map", - DataType::Map(Box::new(map_type)), - true, - )]); + let schema = StructType::new_unchecked([StructField::new("my_map", map_type, true)]); let mut max_id = 0; - let result = assign_column_mapping_metadata(&schema, &mut max_id).unwrap(); + let result = assign_column_mapping_metadata(&schema, &mut max_id, false).unwrap(); // Should assign IDs to: my_map (1), k (2), v (3) assert_eq!(max_id, 3); @@ -917,16 +1945,12 @@ mod tests { let elem_struct = StructType::new_unchecked([StructField::new("elem", DataType::INTEGER, false)]); - let array_type = ArrayType::new(DataType::Struct(Box::new(elem_struct)), true); + let array_type = ArrayType::new(elem_struct, true); - let schema = StructType::new_unchecked([StructField::new( - "my_array", - DataType::Array(Box::new(array_type)), - true, - )]); + let schema = StructType::new_unchecked([StructField::new("my_array", array_type, true)]); let mut max_id = 0; - let result = assign_column_mapping_metadata(&schema, &mut max_id).unwrap(); + let result = assign_column_mapping_metadata(&schema, &mut max_id, false).unwrap(); // Should assign IDs to: my_array (1), elem (2) assert_eq!(max_id, 2); @@ -958,17 +1982,14 @@ mod tests { let deep_struct = StructType::new_unchecked([StructField::new("deep", DataType::INTEGER, false)]); - let inner_array = ArrayType::new(DataType::Struct(Box::new(deep_struct)), true); - let outer_array = ArrayType::new(DataType::Array(Box::new(inner_array)), true); + let inner_array = ArrayType::new(deep_struct, true); + let outer_array = ArrayType::new(inner_array, true); - let schema = StructType::new_unchecked([StructField::new( - "nested_arrays", - DataType::Array(Box::new(outer_array)), - true, - )]); + let schema = + StructType::new_unchecked([StructField::new("nested_arrays", outer_array, true)]); let mut max_id = 0; - let result = assign_column_mapping_metadata(&schema, &mut max_id).unwrap(); + let result = assign_column_mapping_metadata(&schema, &mut max_id, false).unwrap(); // Should assign IDs to: nested_arrays (1), deep (2) assert_eq!(max_id, 2); @@ -1005,25 +2026,17 @@ mod tests { let value_struct = StructType::new_unchecked([StructField::new("v", DataType::INTEGER, false)]); - let key_array = ArrayType::new(DataType::Struct(Box::new(key_struct)), true); - let value_array = ArrayType::new(DataType::Struct(Box::new(value_struct)), true); + let key_array = ArrayType::new(key_struct, true); + let value_array = ArrayType::new(value_struct, true); - let inner_map = MapType::new( - DataType::Array(Box::new(key_array)), - DataType::Array(Box::new(value_array)), - true, - ); + let inner_map = MapType::new(key_array, value_array, true); - let outer_array = ArrayType::new(DataType::Map(Box::new(inner_map)), true); + let outer_array = ArrayType::new(inner_map, true); - let schema = StructType::new_unchecked([StructField::new( - "cursed", - DataType::Array(Box::new(outer_array)), - true, - )]); + let schema = StructType::new_unchecked([StructField::new("cursed", outer_array, true)]); let mut max_id = 0; - let result = assign_column_mapping_metadata(&schema, &mut max_id).unwrap(); + let result = assign_column_mapping_metadata(&schema, &mut max_id, false).unwrap(); // Should assign IDs to: cursed (1), k (2), v (3) assert_eq!(max_id, 3); @@ -1078,26 +2091,22 @@ mod tests { ), ])]); - let schema = StructType::new_unchecked([StructField::new( - "a", - DataType::Struct(Box::new(inner)), - true, - ) - .add_metadata([ - ( - ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), - MetadataValue::String("col-outer-a".to_string()), - ), - ( - ColumnMetadataKey::ColumnMappingId.as_ref(), - MetadataValue::Number(1), - ), - ])]); + let schema = + StructType::new_unchecked([StructField::new("a", inner, true).add_metadata([ + ( + ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), + MetadataValue::String("col-outer-a".to_string()), + ), + ( + ColumnMetadataKey::ColumnMappingId.as_ref(), + MetadataValue::Number(1), + ), + ])]); // Top-level column let result = get_any_level_column_physical_name( &schema, - &ColumnName::new(["a"]), + &column_name!("a"), ColumnMappingMode::Name, ) .unwrap(); @@ -1107,7 +2116,7 @@ mod tests { // Nested column let result = get_any_level_column_physical_name( &schema, - &ColumnName::new(["a", "y"]), + &column_name!("a.y"), ColumnMappingMode::Name, ) .unwrap(); @@ -1117,11 +2126,11 @@ mod tests { // No mapping mode returns logical names (annotations are ignored) let result = get_any_level_column_physical_name( &schema, - &ColumnName::new(["a", "y"]), + &column_name!("a.y"), ColumnMappingMode::None, ) .unwrap(); - assert_eq!(result, ColumnName::new(["a", "y"])); + assert_eq!(result, column_name!("a.y")); assert_eq!(result.path().len(), 2); } @@ -1132,7 +2141,7 @@ mod tests { // Non-existent top-level column let result = get_any_level_column_physical_name( &schema, - &ColumnName::new(["nonexistent"]), + &column_name!("nonexistent"), ColumnMappingMode::None, ); assert!(result.is_err()); @@ -1140,7 +2149,7 @@ mod tests { // Nested path on a non-struct field let result = get_any_level_column_physical_name( &schema, - &ColumnName::new(["a", "b"]), + &column_name!("a.b"), ColumnMappingMode::None, ); assert!(result.is_err()); @@ -1173,25 +2182,21 @@ mod tests { } let inner = StructType::new_unchecked([inner_field]); - let schema = StructType::new_unchecked([StructField::new( - "a", - DataType::Struct(Box::new(inner)), - true, - ) - .add_metadata([ - ( - ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), - MetadataValue::String("col-outer-a".to_string()), - ), - ( - ColumnMetadataKey::ColumnMappingId.as_ref(), - MetadataValue::Number(1), - ), - ])]); + let schema = + StructType::new_unchecked([StructField::new("a", inner, true).add_metadata([ + ( + ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), + MetadataValue::String("col-outer-a".to_string()), + ), + ( + ColumnMetadataKey::ColumnMappingId.as_ref(), + MetadataValue::Number(1), + ), + ])]); let err = get_any_level_column_physical_name( &schema, - &ColumnName::new(["a", "y"]), + &column_name!("a.y"), ColumnMappingMode::Name, ) .unwrap_err() @@ -1202,6 +2207,33 @@ mod tests { ); } + #[rstest::rstest] + #[case::string_value(Some(MetadataValue::String("col-x".into())), Some("col-x"))] + #[case::missing(None /* annotation */, None /* expected(None means error) */)] + #[case::wrong_type_number(Some(MetadataValue::Number(7)), None /* expected(None means error) */)] + #[case::wrong_type_boolean(Some(MetadataValue::Boolean(true)), None /* expected(None means error) */)] + #[case::wrong_type_other( + Some(MetadataValue::Other(serde_json::Value::from("col-x"))), + None /* expected(None means error) */, + )] + fn test_expect_physical_name( + #[case] annotation: Option, + #[case] expected: Option<&str>, + ) { + let field = + StructField::new("a", DataType::INTEGER, true).fold_with(annotation, |field, value| { + field.add_metadata([(ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), value)]) + }); + let result = expect_physical_name(&field); + match expected { + Some(expected) => assert_eq!(result.unwrap(), expected), + None => assert_result_error_with_message( + result, + "Expect field 'a' to have a physical name annotation", + ), + } + } + #[test] fn validate_schema_column_mapping_error_includes_full_path() { let schema = test_deep_nested_schema_missing_leaf_cm(); @@ -1213,4 +2245,217 @@ mod tests { "Expected full nested path in error, got: {err}" ); } + + #[test] + fn physical_to_logical_no_mapping() { + let schema = StructType::new_unchecked(vec![ + StructField::new("id", DataType::INTEGER, false), + StructField::new("name", DataType::STRING, true), + ]); + let physical_col = column_name!("id"); + let (logical, data_type) = physical_to_logical_column_name_and_type( + &schema, + &physical_col, + ColumnMappingMode::None, + ) + .unwrap(); + assert_eq!(logical, column_name!("id")); + assert_eq!(data_type, DataType::INTEGER); + } + + #[test] + fn physical_to_logical_with_name_mapping() { + let field = StructField::new("user_id", DataType::INTEGER, false).with_metadata([( + "delta.columnMapping.physicalName".to_string(), + MetadataValue::String("col-abc-123".to_string()), + )]); + let schema = StructType::new_unchecked(vec![field]); + + let physical_col = ColumnName::new(["col-abc-123"]); + let (logical, data_type) = physical_to_logical_column_name_and_type( + &schema, + &physical_col, + ColumnMappingMode::Name, + ) + .unwrap(); + assert_eq!(logical, column_name!("user_id")); + assert_eq!(data_type, DataType::INTEGER); + } + + #[test] + fn physical_to_logical_not_found() { + let schema = + StructType::new_unchecked(vec![StructField::new("id", DataType::INTEGER, false)]); + let physical_col = column_name!("nonexistent"); + let result = physical_to_logical_column_name_and_type( + &schema, + &physical_col, + ColumnMappingMode::None, + ); + assert!(result.is_err()); + assert!(result + .unwrap_err() + .to_string() + .contains("not found in schema")); + } + + #[test] + fn physical_to_logical_nested_struct_with_mapping() { + let inner_field = StructField::new("city", DataType::STRING, true).with_metadata([( + "delta.columnMapping.physicalName".to_string(), + MetadataValue::String("col-inner-456".to_string()), + )]); + let inner_struct = StructType::new_unchecked(vec![inner_field]); + let outer_field = StructField::new("address", inner_struct, true).with_metadata([( + "delta.columnMapping.physicalName".to_string(), + MetadataValue::String("col-outer-123".to_string()), + )]); + let schema = StructType::new_unchecked(vec![outer_field]); + + let physical_col = ColumnName::new(["col-outer-123", "col-inner-456"]); + let (logical, data_type) = physical_to_logical_column_name_and_type( + &schema, + &physical_col, + ColumnMappingMode::Name, + ) + .unwrap(); + assert_eq!(logical, column_name!("address.city")); + assert_eq!(data_type, DataType::STRING); + } + + #[test] + fn physical_to_logical_non_struct_intermediate_errors() { + let schema = + StructType::new_unchecked(vec![StructField::new("id", DataType::INTEGER, false)]); + let physical_col = column_name!("id.nested"); + let result = physical_to_logical_column_name_and_type( + &schema, + &physical_col, + ColumnMappingMode::None, + ); + assert!(result.is_err()); + assert!(result + .unwrap_err() + .to_string() + .contains("is not a struct type")); + } + + // === find_max_column_id_in_schema tests === + + fn field_with_id(name: &str, ty: impl Into, id: i64) -> StructField { + let mut f = StructField::nullable(name, ty); + f.metadata.insert( + ColumnMetadataKey::ColumnMappingId.as_ref().to_string(), + MetadataValue::Number(id), + ); + f + } + + #[test] + fn find_max_column_id_empty_schema_is_none() { + let schema = + StructType::try_new(vec![StructField::nullable("a", DataType::STRING)]).unwrap(); + assert_eq!(find_max_column_id_in_schema(&schema), None); + } + + #[test] + fn find_max_column_id_top_level_only() { + let schema = StructType::try_new(vec![ + field_with_id("a", DataType::STRING, 1), + field_with_id("b", DataType::INTEGER, 3), + field_with_id("c", DataType::STRING, 2), + ]) + .unwrap(); + assert_eq!(find_max_column_id_in_schema(&schema), Some(3)); + } + + #[test] + fn find_max_column_id_nested_struct() { + let inner = DataType::from( + StructType::try_new(vec![ + field_with_id("x", DataType::STRING, 7), + field_with_id("y", DataType::STRING, 5), + ]) + .unwrap(), + ); + let schema = StructType::try_new(vec![ + field_with_id("outer", inner, 2), + field_with_id("sibling", DataType::STRING, 3), + ]) + .unwrap(); + assert_eq!(find_max_column_id_in_schema(&schema), Some(7)); + } + + #[test] + fn find_max_column_id_array_and_map_recurse_into_element_types() { + let array_elem_struct = DataType::from(ArrayType::new( + StructType::try_new(vec![field_with_id("deep", DataType::STRING, 42)]).unwrap(), + true, + )); + let map_ty = DataType::from(MapType::new( + DataType::STRING, + StructType::try_new(vec![field_with_id("inside", DataType::STRING, 9)]).unwrap(), + false, + )); + let schema = StructType::try_new(vec![ + field_with_id("arr", array_elem_struct, 1), + field_with_id("m", map_ty, 2), + ]) + .unwrap(); + assert_eq!(find_max_column_id_in_schema(&schema), Some(42)); + } + + #[test] + fn find_max_column_id_map_with_struct_key_recurses() { + let key_struct = DataType::from( + StructType::try_new(vec![field_with_id("key_id", DataType::INTEGER, 17)]).unwrap(), + ); + let value_struct = DataType::from( + StructType::try_new(vec![field_with_id("val_id", DataType::INTEGER, 11)]).unwrap(), + ); + let map_ty = DataType::from(MapType::new(key_struct, value_struct, false)); + let schema = StructType::try_new(vec![field_with_id("m", map_ty, 1)]).unwrap(); + // Max should come from the key struct's `key_id = 17`, beating value's 11 and + // top-level's 1. + assert_eq!(find_max_column_id_in_schema(&schema), Some(17)); + } + + /// Every inner variant field carries a cm.id so the assertion proves all three are walked, + /// not just one. The `shred=23` is the max and beats `metadata=10`, `value=20`, and + /// top-level `v=4`. + #[test] + fn find_max_column_id_variant_recurses_into_inner_fields() { + let variant = DataType::Variant(Box::new( + StructType::try_new(vec![ + field_with_id("metadata", DataType::BINARY, 10), + field_with_id("value", DataType::BINARY, 20), + field_with_id("shred", DataType::STRING, 23), + ]) + .unwrap(), + )); + let schema = StructType::try_new(vec![field_with_id("v", variant, 4)]).unwrap(); + assert_eq!(find_max_column_id_in_schema(&schema), Some(23)); + } + + /// Nested-ids JSON entries (assigned to element/key/value of Array/Map) are real column-mapping + /// ids and must beat the per-field `delta.columnMapping.id` when larger. + #[test] + fn find_max_column_id_picks_up_nested_ids_metadata() { + let mut field = field_with_id( + "m", + MapType::new(DataType::INTEGER, DataType::INTEGER, false), + 1, + ); + field.metadata.insert( + ColumnMetadataKey::ColumnMappingNestedIds + .as_ref() + .to_string(), + MetadataValue::Other(serde_json::json!({ + "m.key": 2, + "m.value": 99, + })), + ); + let schema = StructType::try_new(vec![field]).unwrap(); + assert_eq!(find_max_column_id_in_schema(&schema), Some(99)); + } } diff --git a/kernel/src/table_features/geospatial.rs b/kernel/src/table_features/geospatial.rs new file mode 100644 index 0000000000..a709ae20ba --- /dev/null +++ b/kernel/src/table_features/geospatial.rs @@ -0,0 +1,118 @@ +//! Validation logic for the `geospatial` table feature. + +use super::TableFeature; +use crate::schema::{PrimitiveType, Schema}; +use crate::table_configuration::TableConfiguration; +use crate::transforms::{transform_output_type, SchemaTransform}; +use crate::utils::require; +use crate::{DeltaResult, Error}; + +/// Returns `true` if the schema contains at least one geometry or geography column, +/// including nested structs, arrays, and maps. +fn schema_contains_geospatial(schema: &Schema) -> bool { + UsesGeo.transform_struct(schema).is_err() +} + +struct UsesGeo; + +impl<'a> SchemaTransform<'a> for UsesGeo { + transform_output_type!(|'a, T| Result<(), ()>); + + fn transform_primitive(&mut self, ptype: &'a PrimitiveType) -> Result<(), ()> { + match ptype { + PrimitiveType::Geometry(_) | PrimitiveType::Geography(_) => Err(()), + _ => Ok(()), + } + } +} + +/// Validates that if a table schema contains geometry or geography columns, the table must have +/// the `geospatial` feature in both reader and writer features. +pub(crate) fn validate_geospatial_feature_support(tc: &TableConfiguration) -> DeltaResult<()> { + if schema_contains_geospatial(&tc.logical_schema()) { + require!( + tc.protocol().has_table_feature(&TableFeature::GeospatialType), + Error::unsupported( + "Table contains geometry or geography columns but does not have the required 'geospatial' feature in reader and writer features" + ) + ); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + use crate::actions::Protocol; + use crate::schema::{ + ArrayType, DataType, EdgeInterpolationAlgorithm, MapType, StructField, StructType, + }; + use crate::unit_test_utils::{assert_schema_feature_validation, geography_type, geometry_type}; + + fn schema_with_field(dt: DataType) -> StructType { + StructType::new_unchecked([ + StructField::new("id", DataType::INTEGER, false), + StructField::new("g", dt, true), + ]) + } + + #[rstest] + #[case::top_level_geometry(schema_with_field(geometry_type("EPSG:4326")), true)] + #[case::nested_struct( + schema_with_field(DataType::Struct(Box::new(StructType::new_unchecked([ + StructField::new("inner_geo", geography_type("EPSG:4326", EdgeInterpolationAlgorithm::Spherical), true), + ])))), + true + )] + #[case::in_array(schema_with_field(ArrayType::new(geometry_type("EPSG:4326"), true).into()), true)] + #[case::in_map( + schema_with_field(MapType::new(DataType::STRING, geography_type("EPSG:4326", EdgeInterpolationAlgorithm::Spherical), true).into()), + true + )] + #[case::no_geo(schema_with_field(DataType::STRING), false)] + fn test_schema_contains_geospatial(#[case] schema: StructType, #[case] expected: bool) { + assert_eq!(schema_contains_geospatial(&schema), expected); + } + + #[test] + fn test_geospatial_feature_validation() { + let schema_with_geotype = StructType::new_unchecked([ + StructField::new("id", DataType::INTEGER, false), + StructField::new("geom", geometry_type("EPSG:4326"), true), + ]); + let schema_without_geotype = StructType::new_unchecked([ + StructField::new("id", DataType::INTEGER, false), + StructField::new("name", DataType::STRING, true), + ]); + let nested_schema_with_geotype = StructType::new_unchecked([ + StructField::new("id", DataType::INTEGER, false), + StructField::new( + "nested", + DataType::Struct(Box::new(StructType::new_unchecked([StructField::new( + "inner_geo", + geography_type("EPSG:4326", EdgeInterpolationAlgorithm::Spherical), + true, + )]))), + true, + ), + ]); + let protocol_with_geotype = Protocol::try_new_modern( + [TableFeature::GeospatialType], + [TableFeature::GeospatialType], + ) + .unwrap(); + let protocol_without_geotype = + Protocol::try_new_modern(TableFeature::EMPTY_LIST, TableFeature::EMPTY_LIST).unwrap(); + + assert_schema_feature_validation( + &schema_with_geotype, + &schema_without_geotype, + &protocol_with_geotype, + &protocol_without_geotype, + &[&nested_schema_with_geotype], + "Table contains geometry or geography columns but does not have the required 'geospatial' feature in reader and writer features", + ); + } +} diff --git a/kernel/src/table_features/iceberg_compat/mod.rs b/kernel/src/table_features/iceberg_compat/mod.rs new file mode 100644 index 0000000000..1693a0e404 --- /dev/null +++ b/kernel/src/table_features/iceberg_compat/mod.rs @@ -0,0 +1,356 @@ +//! IcebergCompat invariant checks shared across versions. +//! +//! Each `delta.enableIcebergCompatV{N}` version owns a submodule (currently only +//! [`v3`]), and exposes a single +//! `pub(crate) const V{N}_VALIDATOR: IcebergCompatValidator`. Callers feed that +//! constant to [`validate_iceberg_compat_if_needed`]. + +pub(crate) mod v3; + +use crate::schema::{ColumnMetadataKey, DataType, StructField}; +use crate::table_configuration::TableConfiguration; +use crate::table_features::TableFeature; +use crate::transforms::{transform_output_type, SchemaTransform}; +use crate::{DeltaResult, Error}; + +pub(crate) enum IcebergCompatVersion { + // TODO: Add V1, V2 when kernel supports them. + V3, +} + +impl IcebergCompatVersion { + pub(super) fn as_table_feature(&self) -> TableFeature { + match self { + Self::V3 => TableFeature::IcebergCompatV3, + } + } +} + +/// A single invariant that must hold when an icebergCompat version is enabled. +pub(crate) type IcebergCompatCheck = fn(&TableConfiguration) -> DeltaResult<()>; + +/// Pairs an [`IcebergCompatVersion`] with the ordered list of invariants that +/// run when that version is enabled. +pub(crate) struct IcebergCompatValidator { + pub(super) version: IcebergCompatVersion, + pub(super) checks: &'static [IcebergCompatCheck], +} + +pub(crate) fn validate_iceberg_compat_if_needed( + tc: &TableConfiguration, + validator: &IcebergCompatValidator, +) -> DeltaResult<()> { + if !tc.is_feature_enabled(&validator.version.as_table_feature()) { + return Ok(()); + } + for check in validator.checks { + check(tc)?; + } + Ok(()) +} + +/// Walks `tc.logical_schema()` and errors on the first field whose data type +/// fails `is_supported`. Reports the offending column path + type in the error. +pub(super) fn check_only_supported_types( + tc: &TableConfiguration, + is_supported: fn(&DataType) -> bool, + feature_label: &str, +) -> DeltaResult<()> { + let mut v = TypeAllowListVisitor { + is_supported, + offender: None, + path: vec![], + }; + v.transform_struct(&tc.logical_schema()); + let Some(offender) = v.offender else { + return Ok(()); + }; + Err(Error::generic(format!( + "{feature_label} does not support type at column: {offender}", + ))) +} + +struct TypeAllowListVisitor { + is_supported: fn(&DataType) -> bool, + /// The first unsupported field encountered, rendered as ` ()`, or `None` + /// if every field is supported. + /// + /// For example, given schema `struct>>` (and assuming + /// `timestamp` is unsupported), the offender is `events.element.ts (timestamp)`. + offender: Option, + path: Vec, +} + +impl TypeAllowListVisitor { + /// If `dt` isn't allowed, record it as the offender. No-op if an offender is already set. + fn check(&mut self, dt: &DataType) { + if self.offender.is_none() && !(self.is_supported)(dt) { + self.offender = Some(format!("{} ({})", self.path.join("."), dt)); + } + } + + /// Descend into a nested element type (array element, map key, or map value). + fn visit_nested(&mut self, seg: &str, ele_type: &DataType) { + if self.offender.is_some() { + return; + } + self.path.push(seg.to_string()); + self.check(ele_type); + self.transform(ele_type); + self.path.pop(); + } +} + +impl<'a> SchemaTransform<'a> for TypeAllowListVisitor { + transform_output_type!(|'a, T| ()); + + fn transform_struct_field(&mut self, f: &'a StructField) { + if self.offender.is_some() { + return; + } + self.path.push(f.name().to_string()); + self.check(f.data_type()); + self.recurse_into_struct_field(f); + self.path.pop(); + } + + fn transform_array_element(&mut self, etype: &'a DataType) { + self.visit_nested("element", etype); + } + + fn transform_map_key(&mut self, etype: &'a DataType) { + self.visit_nested("key", etype); + } + + fn transform_map_value(&mut self, etype: &'a DataType) { + self.visit_nested("value", etype); + } +} + +/// Rejects fields carrying the legacy `parquet.field.nested.ids` metadata. +/// +/// See . +pub(super) fn check_no_legacy_nested_ids(tc: &TableConfiguration) -> DeltaResult<()> { + let mut v = LegacyNestedIdsVisitor { + path: vec![], + offender: None, + }; + v.transform_struct(&tc.logical_schema()); + let Some(offender) = v.offender else { + return Ok(()); + }; + Err(Error::generic(format!( + "field `{offender}` carries deprecated `{}` metadata; use `{}` instead. \ + See https://github.com/delta-io/delta/issues/6688", + ColumnMetadataKey::ParquetFieldNestedIds.as_ref(), + ColumnMetadataKey::ColumnMappingNestedIds.as_ref(), + ))) +} + +struct LegacyNestedIdsVisitor { + path: Vec, + offender: Option, +} + +impl<'a> SchemaTransform<'a> for LegacyNestedIdsVisitor { + transform_output_type!(|'a, T| ()); + + fn transform_struct_field(&mut self, f: &'a StructField) { + if self.offender.is_some() { + return; + } + self.path.push(f.name().to_string()); + if f.metadata() + .contains_key(ColumnMetadataKey::ParquetFieldNestedIds.as_ref()) + { + self.offender = Some(self.path.join(".")); + return; + } + self.recurse_into_struct_field(f); + self.path.pop(); + } +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + use crate::schema::{schema, ArrayType, MetadataValue, PrimitiveType, StructType}; + + // === LegacyNestedIdsVisitor: parquet.field.nested.ids detection === + + fn field_with_metadata(name: &str, dtype: impl Into, key: &str) -> StructField { + let mut f = StructField::nullable(name, dtype); + f.metadata.insert( + key.to_string(), + MetadataValue::Other(serde_json::json!({ "x.element": 1 })), + ); + f + } + + // `array` fields carrying nested-id metadata under the new vs. legacy key. + fn nested_ids_field(name: &str) -> StructField { + let ty = ArrayType::new(DataType::INTEGER, true); + field_with_metadata(name, ty, ColumnMetadataKey::ColumnMappingNestedIds.as_ref()) + } + fn legacy_ids_field(name: &str) -> StructField { + let ty = ArrayType::new(DataType::INTEGER, true); + field_with_metadata(name, ty, ColumnMetadataKey::ParquetFieldNestedIds.as_ref()) + } + + fn simple_schema() -> StructType { + schema! { + not_null "id": INTEGER, + nullable "name": STRING, + } + } + + fn schema_with_good_nested_ids() -> StructType { + schema! { (nested_ids_field("x")) } + } + + fn schema_with_legacy_at(name: &str) -> StructType { + schema! { (legacy_ids_field(name)) } + } + + fn schema_struct_with_legacy_at_inner() -> StructType { + schema! { nullable "parent": { (legacy_ids_field("inner")) } } + } + + fn schema_array_struct_with_legacy_at_inner() -> StructType { + schema! { nullable "arr": [ nullable { (legacy_ids_field("inner")) } ] } + } + + fn schema_map_value_struct_with_legacy_at_inner() -> StructType { + schema! { nullable "m": { STRING => nullable { (legacy_ids_field("inner")) } } } + } + + fn schema_two_legacy_fields() -> StructType { + schema! { (legacy_ids_field("a")), (legacy_ids_field("b")) } + } + + #[rstest] + #[case::clean_schema(simple_schema(), None)] + #[case::column_mapping_nested_id_key_only(schema_with_good_nested_ids(), None)] + #[case::top_level_legacy(schema_with_legacy_at("top"), Some("top".to_string()))] + #[case::nested_struct_legacy(schema_struct_with_legacy_at_inner(), Some("parent.inner".to_string()))] + #[case::array_struct_legacy(schema_array_struct_with_legacy_at_inner(), Some("arr.inner".to_string()))] + #[case::map_value_struct_legacy(schema_map_value_struct_with_legacy_at_inner(), Some("m.inner".to_string()))] + #[case::first_offender_wins(schema_two_legacy_fields(), Some("a".to_string()))] + fn legacy_nested_ids_visitor_finds_first_offender( + #[case] schema: StructType, + #[case] expected: Option, + ) { + let mut v = LegacyNestedIdsVisitor { + path: vec![], + offender: None, + }; + v.transform_struct(&schema); + assert_eq!(v.offender, expected); + } + + // === TypeAllowlistVisitor tests === + + /// Narrow allowlist: Integer, String, and Struct. + fn narrow_allowlist(dt: &DataType) -> bool { + matches!( + dt, + DataType::Primitive(PrimitiveType::Integer | PrimitiveType::String) + | DataType::Struct(_) + ) + } + + /// Wide allowlist: Integer, String, Struct, Array, Map. + fn wide_allowlist(dt: &DataType) -> bool { + matches!( + dt, + DataType::Primitive(PrimitiveType::Integer | PrimitiveType::String) + | DataType::Struct(_) + | DataType::Array(_) + | DataType::Map(_) + ) + } + + fn schema_struct_with_float_inner() -> StructType { + schema! { nullable "s": { nullable "f": FLOAT } } + } + + fn schema_array_of_float() -> StructType { + schema! { nullable "arr": [ nullable FLOAT ] } + } + + fn schema_map_key_float() -> StructType { + schema! { nullable "m": { FLOAT => nullable STRING } } + } + + fn schema_map_value_float() -> StructType { + schema! { nullable "m": { STRING => nullable FLOAT } } + } + + fn schema_float_long() -> StructType { + schema! { + nullable "a": LONG, + nullable "b": FLOAT, + } + } + + /// `a: array>>` followed by a sibling `b: array`. + fn schema_deep_nested_float() -> StructType { + schema! { + nullable "a": [ nullable { STRING => nullable [ nullable FLOAT ] } ], + nullable "b": [ nullable FLOAT ], + } + } + + #[rstest] + #[case::accept_int_string(narrow_allowlist, simple_schema(), None)] + #[case::reject_top_level_array( + narrow_allowlist, + schema_with_legacy_at("top"), + Some("top (array)".to_string()), + )] + #[case::reject_nested_primitive_in_struct( + narrow_allowlist, + schema_struct_with_float_inner(), + Some("s.f (float)".to_string()), + )] + #[case::reject_primitive_inside_array( + wide_allowlist, + schema_array_of_float(), + Some("arr.element (float)".to_string()), + )] + #[case::reject_primitive_inside_map_key( + wide_allowlist, + schema_map_key_float(), + Some("m.key (float)".to_string()), + )] + #[case::reject_primitive_inside_map_value( + wide_allowlist, + schema_map_value_float(), + Some("m.value (float)".to_string()), + )] + #[case::first_offender_wins( + wide_allowlist, + schema_float_long(), + Some("a (long)".to_string()), + )] + #[case::reject_deep_then_short_circuits_sibling( + wide_allowlist, + schema_deep_nested_float(), + Some("a.element.value.element (float)".to_string()), + )] + fn type_allowlist_visitor( + #[case] is_supported: fn(&DataType) -> bool, + #[case] schema: StructType, + #[case] expected: Option, + ) { + let mut v = TypeAllowListVisitor { + is_supported, + offender: None, + path: vec![], + }; + v.transform_struct(&schema); + assert_eq!(v.offender, expected); + } +} diff --git a/kernel/src/table_features/iceberg_compat/v3.rs b/kernel/src/table_features/iceberg_compat/v3.rs new file mode 100644 index 0000000000..9899bf1ef7 --- /dev/null +++ b/kernel/src/table_features/iceberg_compat/v3.rs @@ -0,0 +1,256 @@ +//! IcebergCompatV3 checks. + +use tracing::warn; + +use super::{ + check_no_legacy_nested_ids, check_only_supported_types, IcebergCompatCheck, + IcebergCompatValidator, IcebergCompatVersion, +}; +use crate::schema::PrimitiveType::*; +use crate::schema::{try_collect_column_defaults, DataType}; +use crate::table_configuration::TableConfiguration; +use crate::DeltaResult; + +/// V3 invariants paired with the version constant. Fed to +/// [`super::validate_iceberg_compat_if_needed`]. +pub(crate) const V3_VALIDATOR: IcebergCompatValidator = IcebergCompatValidator { + version: IcebergCompatVersion::V3, + checks: V3_CHECKS, +}; + +const V3_CHECKS: &[IcebergCompatCheck] = &[check_v3_supported_types, check_no_legacy_nested_ids]; + +fn is_v3_supported_type(dt: &DataType) -> bool { + matches!( + dt, + DataType::Primitive( + Byte | Short + | Integer + | Long + | Float + | Double + | Boolean + | Binary + | String + | Date + | Timestamp + | TimestampNtz + | Decimal(_) + ) | DataType::Array(_) + | DataType::Map(_) + | DataType::Struct(_) + | DataType::Variant(_) + ) +} + +fn check_v3_supported_types(tc: &TableConfiguration) -> DeltaResult<()> { + check_only_supported_types( + tc, + is_v3_supported_type, + IcebergCompatVersion::V3.as_table_feature().as_ref(), + ) +} + +/// Validates IcebergCompatV3 column defaults and logs warnings kernel cannot verify. +/// +/// The IcebergCompatV3 spec requires column defaults to be literals. Kernel warns when its parser +/// cannot verify that requirement. This warning can be a false positive when the expression is a +/// literal that kernel's parser cannot parse. +/// +/// This condition remains a warning because the table has already passed metadata validation and +/// rejecting a DML transaction could block valid tables based on kernel parser limitations. The +/// check provides defense in depth without treating an interoperability risk as definite +/// corruption. +/// +/// # Errors +/// +/// Propagates malformed column-default metadata errors from [`try_collect_column_defaults`]. +pub(crate) fn iceberg_compat_v3_column_defaults_validation( + table_configuration: &TableConfiguration, +) -> DeltaResult<()> { + for (path, column_default) in + try_collect_column_defaults(table_configuration.logical_schema_ref())? + { + if !column_default.is_kernel_parsable_literal() { + warn!( + "kernel could not verify that the icebergCompatV3 column default for '{path}' is a \ + literal, got: {}", + column_default.raw_sql() + ); + } + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::schema::{schema, ArrayType, MapType}; + + #[test] + fn is_v3_supported_type_accepted_datatypes() { + let primitives = [ + DataType::STRING, + DataType::LONG, + DataType::INTEGER, + DataType::SHORT, + DataType::BYTE, + DataType::FLOAT, + DataType::DOUBLE, + DataType::BOOLEAN, + DataType::BINARY, + DataType::DATE, + DataType::TIMESTAMP, + DataType::TIMESTAMP_NTZ, + DataType::decimal(10, 2).unwrap(), + ]; + for dt in primitives { + assert!( + is_v3_supported_type(&dt), + "primitive {dt} should be V3-supported" + ); + } + let nested = [ + DataType::from(ArrayType::new(DataType::INTEGER, true)), + DataType::from(MapType::new(DataType::STRING, DataType::INTEGER, true)), + DataType::from(schema! { nullable "x": INTEGER }), + DataType::unshredded_variant(), + ]; + for dt in nested { + assert!( + is_v3_supported_type(&dt), + "nested {dt} should be V3-supported" + ); + } + } + + #[test] + fn is_v3_supported_type_rejects_void() { + // Void is excluded from the V3 allowlist (by omission) to match delta-spark, which + // cannot consume an icebergCompatV3 table containing a void column. + assert!(!is_v3_supported_type(&DataType::VOID)); + } +} + +#[cfg(test)] +mod column_default_tests { + use std::collections::HashMap; + + use rstest::rstest; + use test_utils::LoggingTest; + use url::Url; + + use super::iceberg_compat_v3_column_defaults_validation; + use crate::actions::{Metadata, Protocol}; + use crate::schema::ColumnMetadataKey::CurrentDefault; + use crate::schema::{ArrayType, DataType, MetadataValue, StructField, StructType}; + use crate::table_configuration::TableConfiguration; + use crate::table_features::TableFeature; + + /// Builds a `TableConfiguration` carrying `schema` with `allowColumnDefaults` enabled, so + /// the IcebergCompatV3 column-default validation can be driven directly. The config does not + /// enable IcebergCompatV3 itself (whose required dependencies are heavy to assemble here); the + /// validation is invoked directly instead, and the end-to-end V3 path is covered by the + /// integration tests. + fn table_config_with_schema(schema: StructType) -> TableConfiguration { + let metadata = Metadata::try_new( + None, + None, + std::sync::Arc::new(schema), + vec![], + 0, + HashMap::new(), + ) + .unwrap(); + let protocol = Protocol::try_new_modern( + TableFeature::EMPTY_LIST, + [TableFeature::AllowColumnDefaults], + ) + .unwrap(); + TableConfiguration::try_new(metadata, protocol, Url::parse("file:///t/").unwrap(), 0) + .unwrap() + } + + fn field_with_default( + name: &str, + data_type: impl Into, + default_sql: &str, + ) -> StructField { + StructField::nullable(name, data_type).add_metadata([( + CurrentDefault.as_ref().to_string(), + MetadataValue::String(default_sql.to_string()), + )]) + } + + #[rstest] + #[case::primitive_literal( + StructType::try_new([field_with_default("a", DataType::INTEGER, "42")]).unwrap(), + "a", + None + )] + #[case::primitive_null( + StructType::try_new([field_with_default("a", DataType::INTEGER, "NULL")]).unwrap(), + "a", + None + )] + #[case::non_literal_primitive( + StructType::try_new([field_with_default( + "a", + DataType::TIMESTAMP, + "current_timestamp()" + )]).unwrap(), + "a", + Some("could not verify") + )] + #[case::null_on_non_primitive( + StructType::try_new([field_with_default( + "a", + ArrayType::new(DataType::INTEGER, true), + "NULL" + )]).unwrap(), + "a", + None + )] + #[case::non_null_on_non_primitive( + StructType::try_new([field_with_default( + "a", + ArrayType::new(DataType::INTEGER, true), + "ARRAY(1)" + )]).unwrap(), + "a", + Some("could not verify") + )] + #[case::nested_non_literal( + StructType::try_new([StructField::nullable( + "s", + DataType::try_struct_type([field_with_default( + "inner", + DataType::TIMESTAMP, + "current_timestamp()" + )]).unwrap(), + )]).unwrap(), + "s.inner", + Some("could not verify") + )] + fn v3_column_default_validation( + #[case] schema: StructType, + #[case] expected_path: &str, + #[case] expected_warning: Option<&str>, + ) { + let table_configuration = table_config_with_schema(schema); + let logging = LoggingTest::new(); + iceberg_compat_v3_column_defaults_validation(&table_configuration).unwrap(); + let logs = logging.logs(); + + match expected_warning { + None => assert!( + !logs.contains("icebergCompatV3 column default"), + "logs: {logs}" + ), + Some(needle) => { + assert!(logs.contains(expected_path), "logs: {logs}"); + assert!(logs.contains(needle), "logs: {logs}"); + } + } + } +} diff --git a/kernel/src/table_features/mod.rs b/kernel/src/table_features/mod.rs index cbf9d7072f..3d90853153 100644 --- a/kernel/src/table_features/mod.rs +++ b/kernel/src/table_features/mod.rs @@ -1,30 +1,46 @@ +#[internal_api] +pub(crate) use column_mapping::get_any_level_column_physical_name; +#[deprecated = "Enable internal-api and use TableConfiguration instead"] +pub use column_mapping::validate_schema_column_mapping; +pub use column_mapping::ColumnMappingMode; +#[internal_api] +pub(crate) use column_mapping::{assign_column_mapping_metadata, find_max_column_id_in_schema}; +pub(crate) use column_mapping::{ + column_mapping_mode, get_column_mapping_mode_from_properties, + physical_to_logical_column_name_and_type, schema_has_column_mapping_metadata, + strip_stray_column_mapping_metadata, try_assign_flat_column_mapping_info, + validate_and_extract_column_mapping_annotations, validate_column_mapping_id, + StaleAnnotationPolicy, +}; +use delta_kernel_derive::internal_api; +#[cfg(feature = "geo-type-in-dev")] +pub(crate) use geospatial::validate_geospatial_feature_support; +pub(crate) use iceberg_compat::v3::{iceberg_compat_v3_column_defaults_validation, V3_VALIDATOR}; +pub(crate) use iceberg_compat::validate_iceberg_compat_if_needed; use itertools::Itertools; use serde::{Deserialize, Serialize}; use strum::{AsRefStr, Display as StrumDisplay, EnumCount, EnumIter, EnumString}; +pub(crate) use timestamp_ntz::{ + schema_contains_timestamp_ntz, validate_timestamp_ntz_feature_support, +}; use crate::actions::Protocol; use crate::expressions::Scalar; use crate::schema::derive_macro_utils::ToDataType; use crate::schema::DataType; use crate::table_properties::TableProperties; +use crate::utils::require; use crate::{DeltaResult, Error}; -use delta_kernel_derive::internal_api; -#[internal_api] -pub(crate) use column_mapping::get_any_level_column_physical_name; -#[deprecated = "Enable internal-api and use TableConfiguration instead"] -pub use column_mapping::validate_schema_column_mapping; -pub use column_mapping::ColumnMappingMode; -pub(crate) use column_mapping::{ - assign_column_mapping_metadata, column_mapping_mode, get_column_mapping_mode_from_properties, - get_field_column_mapping_info, -}; -pub(crate) use timestamp_ntz::{ - schema_contains_timestamp_ntz, validate_timestamp_ntz_feature_support, -}; mod column_mapping; +#[cfg(feature = "geo-type-in-dev")] +mod geospatial; +mod iceberg_compat; mod timestamp_ntz; +/// Minimum reader/writer protocol version that the kernel can handle. +pub const MIN_VALID_RW_VERSION: i32 = 1; + /// Maximum reader protocol version that the kernel can handle. pub const MAX_VALID_READER_VERSION: i32 = 3; @@ -48,7 +64,8 @@ pub const SET_TABLE_FEATURE_SUPPORTED_PREFIX: &str = "delta.feature."; /// Example: `"delta.feature.deletionVectors" -> "supported"` pub const SET_TABLE_FEATURE_SUPPORTED_VALUE: &str = "supported"; -/// Table features represent protocol capabilities required to correctly read or write a given table. +/// Table features represent protocol capabilities required to correctly read or write a given +/// table. /// - Readers must implement all features required for correct table reads. /// - Writers must implement all features required for correct table writes. /// @@ -57,7 +74,7 @@ pub const SET_TABLE_FEATURE_SUPPORTED_VALUE: &str = "supported"; /// - **WriterOnly** (applies only to writers). /// There are no ReaderOnly features. See `TableFeature::feature_type` for the category of each. /// -/// The kernel currently supports all reader features except `V2Checkpoint`. +/// The kernel currently supports all reader features. #[derive( Serialize, Deserialize, @@ -71,11 +88,7 @@ pub const SET_TABLE_FEATURE_SUPPORTED_VALUE: &str = "supported"; EnumCount, Hash, )] -#[strum( - serialize_all = "camelCase", - parse_err_fn = xxx__not_needed__default_variant_means_parsing_is_infallible__xxx, - parse_err_ty = Infallible // ignored, sadly: https://github.com/Peternator7/strum/issues/430 -)] +#[strum(serialize_all = "camelCase")] #[serde(rename_all = "camelCase")] #[internal_api] #[derive(EnumIter)] @@ -103,10 +116,12 @@ pub(crate) enum TableFeature { RowTracking, /// domain specific metadata DomainMetadata, - /// Iceberg compatibility support + /// Iceberg V1 compatibility support IcebergCompatV1, - /// Iceberg compatibility support + /// Iceberg V2 compatibility support IcebergCompatV2, + /// Iceberg V3 compatibility support + IcebergCompatV3, /// The Clustered Table feature facilitates the physical clustering of rows /// that share similar values on a predefined set of clustering columns. #[strum(serialize = "clustering")] @@ -114,6 +129,8 @@ pub(crate) enum TableFeature { ClusteredTable, /// Materialize partition columns in parquet data files. MaterializePartitionColumns, + /// Column Default Values. + AllowColumnDefaults, /////////////////////////// // ReaderWriter features // @@ -128,9 +145,14 @@ pub(crate) enum TableFeature { ColumnMapping, /// Deletion vectors for merge, update, delete DeletionVectors, - /// timestamps without timezone support - #[strum(serialize = "timestampNtz")] - #[serde(rename = "timestampNtz")] + /// Timestamps without timezone support. The canonical protocol feature name is `timestampNtz`. + /// + /// `timestampWithoutTimezone` is not a Delta protocol feature name, but some existing tables + /// carry it in their reader/writer feature arrays. Kernel accepts it on read for compatibility + /// with those tables and always writes the canonical `timestampNtz`. See + /// . + #[strum(to_string = "timestampNtz", serialize = "timestampWithoutTimezone")] + #[serde(rename = "timestampNtz", alias = "timestampWithoutTimezone")] TimestampWithoutTimezone, // Allow columns to change type TypeWidening, @@ -139,17 +161,28 @@ pub(crate) enum TableFeature { TypeWideningPreview, /// version 2 of checkpointing V2Checkpoint, - /// vacuumProtocolCheck ReaderWriter feature ensures consistent application of reader and writer - /// protocol checks during VACUUM operations + /// vacuumProtocolCheck ReaderWriter feature ensures consistent application of reader and + /// writer protocol checks during VACUUM operations VacuumProtocolCheck, /// This feature enables support for the variant data type, which stores semi-structured data. VariantType, #[strum(serialize = "variantType-preview")] #[serde(rename = "variantType-preview")] VariantTypePreview, + VariantShredding, #[strum(serialize = "variantShredding-preview")] #[serde(rename = "variantShredding-preview")] VariantShreddingPreview, + /// Iceberg V4 adaptive metadata tree as the table's native content metadata format. + /// + /// TODO(#2866): gated by the `adaptive-metadata-in-dev` cargo feature until fully supported. + #[strum(serialize = "adaptiveMetadata-preview")] + #[serde(rename = "adaptiveMetadata-preview")] + AdaptiveMetadataPreview, + /// Geospatial type support (geometry and geography columns) + #[strum(serialize = "geospatial")] + #[serde(rename = "geospatial")] + GeospatialType, #[serde(untagged)] #[strum(default)] @@ -160,8 +193,8 @@ pub(crate) enum TableFeature { /// Only ColumnMapping qualifies with min_reader_version = 2. pub(crate) static LEGACY_READER_FEATURES: [TableFeature; 1] = [TableFeature::ColumnMapping]; -/// Writer and ReaderWriter features that can be supported by legacy writers (min_writer_version < 7). -/// These are features with min_writer_version in range [1, 6]. +/// Writer and ReaderWriter features that can be supported by legacy writers (min_writer_version < +/// 7). These are features with min_writer_version in range [1, 6]. pub(crate) static LEGACY_WRITER_FEATURES: [TableFeature; 7] = [ // Writer-only features (min_writer < 7) TableFeature::AppendOnly, // min_writer = 2 @@ -190,7 +223,8 @@ pub(crate) enum FeatureType { pub(crate) enum EnablementCheck { /// Feature is enabled if it's supported (appears in protocol feature lists) AlwaysIfSupported, - /// Feature is enabled if supported AND the provided function returns true when checking table properties + /// Feature is enabled if supported AND the provided function returns true when checking table + /// properties EnabledIf(fn(&TableProperties) -> bool), } @@ -229,7 +263,7 @@ pub(crate) enum FeatureRequirement { NotSupported(TableFeature), /// Feature must NOT be enabled (may be supported but property must not activate it) NotEnabled(TableFeature), - /// Custom validation logic + /// Custom validation logic run against the protocol and table properties. Custom(fn(&Protocol, &TableProperties) -> DeltaResult<()>), } @@ -245,7 +279,8 @@ impl MinReaderWriterVersion { } } -/// Rich metadata about a table feature including version requirements, dependencies, and support status +/// Rich metadata about a table feature including version requirements, dependencies, and support +/// status pub(crate) struct FeatureInfo { /// The type of feature (WriterOnly, ReaderWriter, or Unknown) pub feature_type: FeatureType, @@ -276,9 +311,10 @@ static APPEND_ONLY_INFO: FeatureInfo = FeatureInfo { enablement_check: EnablementCheck::EnabledIf(|props| props.append_only == Some(true)), }; -// Although kernel marks invariants as "Supported", invariants must NOT actually be present in the table schema. -// Kernel will fail to write to any table that actually uses invariants (see check in TableConfiguration::ensure_write_supported). -// This is to allow legacy tables with the Invariants feature enabled but not in use. +// Although kernel marks invariants as "Supported", invariants must NOT actually be present in the +// table schema. Kernel will fail to write to any table that actually uses invariants (see check in +// TableConfiguration::ensure_write_supported). This is to allow legacy tables with the Invariants +// feature enabled but not in use. static INVARIANTS_INFO: FeatureInfo = FeatureInfo { feature_type: FeatureType::WriterOnly, min_legacy_version: Some(MinReaderWriterVersion::new(1, 2)), @@ -333,6 +369,16 @@ static IN_COMMIT_TIMESTAMP_INFO: FeatureInfo = FeatureInfo { }), }; +// TODO(#2538): Currently we reject `Transaction::commit` when it contains staged remove-file +// actions on RowTracking-supported (and not-suspended) tables because +// 1. kernel does not yet materialize stable row IDs / commit versions on write, which blocks COW +// rewrites, +// 2. kernel does not yet validate if remove actions correctly reserved row IDs / commit versions. +// Unblock after both 1 and 2 are supported. +// +// TODO: When kernel writes the materialized `row_id` / `row_commit_version` columns, they must +// use the reserved parquet field IDs defined by the protocol on IcebergCompatV3 tables, not +// auto-assigned IDs. static ROW_TRACKING_INFO: FeatureInfo = FeatureInfo { feature_type: FeatureType::WriterOnly, min_legacy_version: None, @@ -352,28 +398,20 @@ static DOMAIN_METADATA_INFO: FeatureInfo = FeatureInfo { }; // TODO(#1125): IcebergCompatV1 requires schema type validation to block Map, Array, and Void types. -// This validation is not yet implemented. The feature is marked as NotSupported for writes until proper validation is added. +// This validation is not yet implemented. The feature is marked as NotSupported for writes until +// proper validation is added. +// // See Delta Spark: IcebergCompat.scala CheckNoListMapNullType (lines 422-433) -// See Java Kernel: IcebergWriterCompatMetadataValidatorAndUpdater.java UNSUPPORTED_TYPES_CHECK -// See https://github.com/delta-io/delta/blob/master/PROTOCOL.md#writer-requirements-for-icebergcompatv1 for more requirements to support +// See Java Kernel: IcebergWriterCompatMetadataValidatorAndUpdater.java +// UNSUPPORTED_TYPES_CHECK See https://github.com/delta-io/delta/blob/master/PROTOCOL.md#writer-requirements-for-icebergcompatv1 for more requirements to support static ICEBERG_COMPAT_V1_INFO: FeatureInfo = FeatureInfo { feature_type: FeatureType::WriterOnly, min_legacy_version: None, feature_requirements: &[ FeatureRequirement::Enabled(TableFeature::ColumnMapping), - FeatureRequirement::Custom(|_protocol, properties| { - let mode = properties.column_mapping_mode; - if !matches!( - mode, - Some(ColumnMappingMode::Name) | Some(ColumnMappingMode::Id) - ) { - return Err(Error::generic( - "IcebergCompatV1 requires Column Mapping in 'name' or 'id' mode", - )); - } - Ok(()) - }), FeatureRequirement::NotSupported(TableFeature::DeletionVectors), + FeatureRequirement::NotEnabled(TableFeature::IcebergCompatV2), + FeatureRequirement::NotEnabled(TableFeature::IcebergCompatV3), ], kernel_support: KernelSupport::NotSupported, enablement_check: EnablementCheck::EnabledIf(|props| { @@ -383,7 +421,9 @@ static ICEBERG_COMPAT_V1_INFO: FeatureInfo = FeatureInfo { // TODO(#1125): IcebergCompatV2 requires schema type validation. Unlike V1, V2 allows Map and Array // types but needs validation against an allowlist of supported types. -// This validation is not yet implemented. The feature is marked as NotSupported for writes until proper validation is added. +// This validation is not yet implemented. The feature is marked as NotSupported for writes until +// proper validation is added. + // See Delta Spark: IcebergCompat.scala CheckTypeInV2AllowList (lines 450-459) // See Java Kernel: IcebergCompatMetadataValidatorAndUpdater.java V2_SUPPORTED_TYPES // See https://github.com/delta-io/delta/blob/master/PROTOCOL.md#writer-requirements-for-icebergcompatv2 for more requirements to support. @@ -392,20 +432,9 @@ static ICEBERG_COMPAT_V2_INFO: FeatureInfo = FeatureInfo { min_legacy_version: None, feature_requirements: &[ FeatureRequirement::Enabled(TableFeature::ColumnMapping), - FeatureRequirement::Custom(|_protocol, properties| { - let mode = properties.column_mapping_mode; - if !matches!( - mode, - Some(ColumnMappingMode::Name) | Some(ColumnMappingMode::Id) - ) { - return Err(Error::generic( - "IcebergCompatV2 requires Column Mapping in 'name' or 'id' mode", - )); - } - Ok(()) - }), FeatureRequirement::NotEnabled(TableFeature::IcebergCompatV1), FeatureRequirement::NotEnabled(TableFeature::DeletionVectors), + FeatureRequirement::NotEnabled(TableFeature::IcebergCompatV3), ], kernel_support: KernelSupport::NotSupported, enablement_check: EnablementCheck::EnabledIf(|props| { @@ -413,6 +442,45 @@ static ICEBERG_COMPAT_V2_INFO: FeatureInfo = FeatureInfo { }), }; +/// IcebergCompatV3 ensures tables can be converted to Apache Iceberg V3. +/// +/// Spec: +/// +/// TODO: Implement the write-side requirements for IcebergCompatV3. +/// TODO: Support ALTER TABLE on tables with IcebergCompatV3 enabled. +/// +/// Attention in the future: +/// - Geo types: when supported, they must not be usable as partition columns on IcebergCompatV3 +/// tables. +/// - REPLACE TABLE: when supported, partition columns must not change across the replace. +/// - Timestamp parquet encoding: when kernel can write INT96 or INT64, IcebergCompatV3 tables must +/// always use INT64; INT96 is forbidden. +/// - ALTER TABLE SET/UNSET TBLPROPERTIES: when supported, reject any property change that would +/// disable IcebergCompatV3 on an existing table. +/// - Void type: when delta-spark supports VOID type on icebergCompatV3 tables, add it to V3's type +/// allowlist (`is_v3_supported_type` in `iceberg_compat::v3`). +/// +/// Tracking issue: +static ICEBERG_COMPAT_V3_INFO: FeatureInfo = FeatureInfo { + feature_type: FeatureType::WriterOnly, + min_legacy_version: None, + feature_requirements: &[ + FeatureRequirement::Enabled(TableFeature::ColumnMapping), + FeatureRequirement::Enabled(TableFeature::RowTracking), + // Unlike V1/V2, V3 intentionally permits DeletionVectors per the RFC. No + // `NotEnabled(DeletionVectors)` requirement is needed. + // + // V1/V2 may remain in `writerFeatures` (supported) as long as they are not active, + // hence `NotEnabled` rather than `NotSupported`. + FeatureRequirement::NotEnabled(TableFeature::IcebergCompatV1), + FeatureRequirement::NotEnabled(TableFeature::IcebergCompatV2), + ], + kernel_support: KernelSupport::Supported, + enablement_check: EnablementCheck::EnabledIf(|props| { + props.enable_iceberg_compat_v3 == Some(true) + }), +}; + static CLUSTERED_TABLE_INFO: FeatureInfo = FeatureInfo { feature_type: FeatureType::WriterOnly, min_legacy_version: None, @@ -429,35 +497,37 @@ static MATERIALIZE_PARTITION_COLUMNS_INFO: FeatureInfo = FeatureInfo { enablement_check: EnablementCheck::AlwaysIfSupported, }; +static ALLOW_COLUMN_DEFAULTS_INFO: FeatureInfo = FeatureInfo { + feature_type: FeatureType::WriterOnly, + min_legacy_version: None, + feature_requirements: &[], + kernel_support: KernelSupport::Supported, + enablement_check: EnablementCheck::AlwaysIfSupported, +}; + static CATALOG_MANAGED_INFO: FeatureInfo = FeatureInfo { feature_type: FeatureType::ReaderWriter, min_legacy_version: None, - feature_requirements: &[], - #[cfg(feature = "catalog-managed")] + feature_requirements: &[FeatureRequirement::Enabled(TableFeature::InCommitTimestamp)], kernel_support: KernelSupport::Custom(|_, _, op| match op { Operation::Scan | Operation::Write => Ok(()), Operation::Cdf => Err(Error::unsupported( "Feature 'catalogManaged' is not supported for CDF", )), }), - #[cfg(not(feature = "catalog-managed"))] - kernel_support: KernelSupport::NotSupported, enablement_check: EnablementCheck::AlwaysIfSupported, }; static CATALOG_OWNED_PREVIEW_INFO: FeatureInfo = FeatureInfo { feature_type: FeatureType::ReaderWriter, min_legacy_version: None, - feature_requirements: &[], - #[cfg(feature = "catalog-managed")] + feature_requirements: &[FeatureRequirement::Enabled(TableFeature::InCommitTimestamp)], kernel_support: KernelSupport::Custom(|_, _, op| match op { Operation::Scan | Operation::Write => Ok(()), Operation::Cdf => Err(Error::unsupported( "Feature 'catalogOwned-preview' is not supported for CDF", )), }), - #[cfg(not(feature = "catalog-managed"))] - kernel_support: KernelSupport::NotSupported, enablement_check: EnablementCheck::AlwaysIfSupported, }; @@ -476,8 +546,9 @@ static DELETION_VECTORS_INFO: FeatureInfo = FeatureInfo { feature_type: FeatureType::ReaderWriter, min_legacy_version: None, feature_requirements: &[], - // We support writing to tables with DeletionVectors enabled, but we never write DV files - // ourselves (no DML). The kernel only performs append operations. + // The kernel can read DV-bearing tables and install connector-authored DV descriptors via + // `Transaction::update_deletion_vectors`, including through the FFI + // `transaction_update_deletion_vectors` path. kernel_support: KernelSupport::Supported, enablement_check: EnablementCheck::EnabledIf(|props| { props.enable_deletion_vectors == Some(true) @@ -492,6 +563,9 @@ static TIMESTAMP_WITHOUT_TIMEZONE_INFO: FeatureInfo = FeatureInfo { enablement_check: EnablementCheck::AlwaysIfSupported, }; +/// TODO: When type widening is supported on writes, restrict the allowed +/// widenings on IcebergCompatV3 tables to the subset permitted by the Iceberg v3 +/// schema-evolution rules. Ref: static TYPE_WIDENING_INFO: FeatureInfo = FeatureInfo { feature_type: FeatureType::ReaderWriter, min_legacy_version: None, @@ -505,6 +579,9 @@ static TYPE_WIDENING_INFO: FeatureInfo = FeatureInfo { enablement_check: EnablementCheck::EnabledIf(|props| props.enable_type_widening == Some(true)), }; +/// TODO: When type widening is supported on writes, restrict the allowed +/// widenings on IcebergCompatV3 tables to the subset permitted by the Iceberg +/// schema-evolution rules. Ref: static TYPE_WIDENING_PREVIEW_INFO: FeatureInfo = FeatureInfo { feature_type: FeatureType::ReaderWriter, min_legacy_version: None, @@ -550,6 +627,14 @@ static VARIANT_TYPE_PREVIEW_INFO: FeatureInfo = FeatureInfo { enablement_check: EnablementCheck::AlwaysIfSupported, }; +static VARIANT_SHREDDING_INFO: FeatureInfo = FeatureInfo { + feature_type: FeatureType::ReaderWriter, + min_legacy_version: None, + feature_requirements: &[], + kernel_support: KernelSupport::Supported, + enablement_check: EnablementCheck::AlwaysIfSupported, +}; + static VARIANT_SHREDDING_PREVIEW_INFO: FeatureInfo = FeatureInfo { feature_type: FeatureType::ReaderWriter, min_legacy_version: None, @@ -558,6 +643,52 @@ static VARIANT_SHREDDING_PREVIEW_INFO: FeatureInfo = FeatureInfo { enablement_check: EnablementCheck::AlwaysIfSupported, }; +// Dependencies per the adaptiveMetadata RFC (delta-io/delta#6978) "Table Feature Enablement" +// section. Enforcement is covered by `test_adaptive_metadata_feature_requirements`. +// TODO(#2866): drop the `adaptive-metadata-in-dev` gate once adaptiveMetadata is fully supported. +static ADAPTIVE_METADATA_PREVIEW_INFO: FeatureInfo = FeatureInfo { + feature_type: FeatureType::ReaderWriter, + min_legacy_version: None, + feature_requirements: &[ + FeatureRequirement::Enabled(TableFeature::ColumnMapping), + FeatureRequirement::Custom(|_protocol, properties| { + require!( + properties.column_mapping_mode == Some(ColumnMappingMode::Id), + Error::invalid_protocol( + "Feature 'adaptiveMetadata-preview' requires column mapping in 'id' mode" + ) + ); + Ok(()) + }), + FeatureRequirement::Enabled(TableFeature::RowTracking), + FeatureRequirement::Enabled(TableFeature::DomainMetadata), + FeatureRequirement::Enabled(TableFeature::DeletionVectors), + FeatureRequirement::Enabled(TableFeature::InCommitTimestamp), + ], + #[cfg(feature = "adaptive-metadata-in-dev")] + kernel_support: KernelSupport::Supported, + #[cfg(not(feature = "adaptive-metadata-in-dev"))] + kernel_support: KernelSupport::NotSupported, + enablement_check: EnablementCheck::AlwaysIfSupported, +}; + +// TODO(#2949): drop the `geo-type-in-dev` gate once full geospatial support ships. +static GEOSPATIAL_TYPE_INFO: FeatureInfo = FeatureInfo { + feature_type: FeatureType::ReaderWriter, + min_legacy_version: None, + feature_requirements: &[], + #[cfg(feature = "geo-type-in-dev")] + kernel_support: KernelSupport::Custom(|_, _, op| match op { + Operation::Scan | Operation::Cdf => Ok(()), + Operation::Write => Err(Error::unsupported( + "Feature 'geospatial' is not supported for writes", + )), + }), + #[cfg(not(feature = "geo-type-in-dev"))] + kernel_support: KernelSupport::NotSupported, + enablement_check: EnablementCheck::AlwaysIfSupported, +}; + /// By definition, kernel cannot know how to handle unknown features and must assume they're always /// enabled if supported in protocol. However, the read path ignores all writer-only features, /// including unknown ones. Unknown features are never inferred from legacy protocol versions. @@ -588,7 +719,10 @@ impl TableFeature { | TableFeature::VacuumProtocolCheck | TableFeature::VariantType | TableFeature::VariantTypePreview - | TableFeature::VariantShreddingPreview => FeatureType::ReaderWriter, + | TableFeature::VariantShredding + | TableFeature::VariantShreddingPreview + | TableFeature::AdaptiveMetadataPreview + | TableFeature::GeospatialType => FeatureType::ReaderWriter, TableFeature::AppendOnly | TableFeature::DomainMetadata | TableFeature::Invariants @@ -600,8 +734,10 @@ impl TableFeature { | TableFeature::InCommitTimestamp | TableFeature::IcebergCompatV1 | TableFeature::IcebergCompatV2 + | TableFeature::IcebergCompatV3 | TableFeature::ClusteredTable | TableFeature::MaterializePartitionColumns => FeatureType::WriterOnly, + TableFeature::AllowColumnDefaults => FeatureType::WriterOnly, TableFeature::Unknown(_) => FeatureType::Unknown, } } @@ -634,8 +770,10 @@ impl TableFeature { TableFeature::DomainMetadata => &DOMAIN_METADATA_INFO, TableFeature::IcebergCompatV1 => &ICEBERG_COMPAT_V1_INFO, TableFeature::IcebergCompatV2 => &ICEBERG_COMPAT_V2_INFO, + TableFeature::IcebergCompatV3 => &ICEBERG_COMPAT_V3_INFO, TableFeature::ClusteredTable => &CLUSTERED_TABLE_INFO, TableFeature::MaterializePartitionColumns => &MATERIALIZE_PARTITION_COLUMNS_INFO, + TableFeature::AllowColumnDefaults => &ALLOW_COLUMN_DEFAULTS_INFO, // ReaderWriter features TableFeature::CatalogManaged => &CATALOG_MANAGED_INFO, @@ -649,7 +787,10 @@ impl TableFeature { TableFeature::VacuumProtocolCheck => &VACUUM_PROTOCOL_CHECK_INFO, TableFeature::VariantType => &VARIANT_TYPE_INFO, TableFeature::VariantTypePreview => &VARIANT_TYPE_PREVIEW_INFO, + TableFeature::VariantShredding => &VARIANT_SHREDDING_INFO, TableFeature::VariantShreddingPreview => &VARIANT_SHREDDING_PREVIEW_INFO, + TableFeature::AdaptiveMetadataPreview => &ADAPTIVE_METADATA_PREVIEW_INFO, + TableFeature::GeospatialType => &GEOSPATIAL_TYPE_INFO, // Unknown features: not supported by kernel, no legacy version inference. TableFeature::Unknown(_) => &UNKNOWN_FEATURE_INFO, @@ -676,49 +817,137 @@ impl TableFeature { } } -/// Like `Into`, but avoids collisions between strum's derived `EnumString` and the -/// blanket impl `TryFrom<&str>` that `From<&str> for TableFeature` would trigger. -/// -/// Parsing is infallible: the `Unknown` default variant catches any unrecognized feature name. If -/// https://github.com/Peternator7/strum/pull/432 merges, use impl From for TableFeature instead. -pub(crate) trait IntoTableFeature { - fn into_table_feature(self) -> TableFeature; +impl From for TableFeature { + fn from(value: String) -> Self { + value.as_str().into() + } } -impl IntoTableFeature for TableFeature { - fn into_table_feature(self) -> TableFeature { - self +impl From<&TableFeature> for TableFeature { + fn from(value: &TableFeature) -> Self { + value.clone() } } -impl IntoTableFeature for &TableFeature { - fn into_table_feature(self) -> TableFeature { - self.clone() +/// Formats a slice of table features using Delta's standard serialization (camelCase). +pub(crate) fn format_features(features: &[TableFeature]) -> String { + let feature_strings: Vec<&str> = features.iter().map(|f| f.as_ref()).collect_vec(); + format!("[{}]", feature_strings.join(", ")) +} + +/// Extract the reader features enabled for `protocol`. For `min_reader_version == 3` returns the +/// explicit `reader_features` list; for `1..=2` returns the legacy-inferred features. +pub(crate) fn extract_enabled_reader_features(protocol: &Protocol) -> Vec { + match protocol.min_reader_version() { + TABLE_FEATURES_MIN_READER_VERSION => protocol + .reader_features() + .map(|f| f.to_vec()) + .unwrap_or_default(), + v if (1..=2).contains(&v) => LEGACY_READER_FEATURES + .iter() + .filter(|f| f.is_valid_for_legacy_reader(v)) + .cloned() + .collect(), + _ => Vec::new(), } } -/// Parsing is infallible thanks to `TableFeature::Unknown` default variant -impl IntoTableFeature for &str { - fn into_table_feature(self) -> TableFeature { - #[allow(clippy::unwrap_used)] // infallible, see strum parse_err_fn - self.parse().unwrap() +/// Add `feature` to the appropriate feature list(s) for its type, skipping duplicates. +pub(crate) fn add_feature_to_lists( + feature: TableFeature, + reader_features: &mut Vec, + writer_features: &mut Vec, +) { + match feature.feature_type() { + FeatureType::ReaderWriter => { + if !reader_features.contains(&feature) { + reader_features.push(feature.clone()); + } + if !writer_features.contains(&feature) { + writer_features.push(feature); + } + } + FeatureType::WriterOnly | FeatureType::Unknown => { + if !writer_features.contains(&feature) { + writer_features.push(feature); + } + } } } -impl IntoTableFeature for String { - fn into_table_feature(self) -> TableFeature { - self.as_str().into_table_feature() +/// Enable each `allowed_table_features` entry whose [`EnablementCheck::EnabledIf`] check is +/// satisfied by `table_properties`, appending it to `reader_features`/`writer_features` +/// (deduplicated). Features with [`EnablementCheck::AlwaysIfSupported`] are skipped since they need +/// no property-driven enablement. `RowTracking` additionally pulls in its `DomainMetadata` +/// dependency. +pub(crate) fn auto_enable_property_driven_features( + allowed_table_features: &[TableFeature], + table_properties: &TableProperties, + reader_features: &mut Vec, + writer_features: &mut Vec, +) { + for table_feature in allowed_table_features { + if let EnablementCheck::EnabledIf(check) = table_feature.info().enablement_check { + if check(table_properties) { + add_feature_to_lists(table_feature.clone(), reader_features, writer_features); + if *table_feature == TableFeature::RowTracking { + add_feature_to_lists( + TableFeature::DomainMetadata, + reader_features, + writer_features, + ); + } + } + } } } -/// Formats a slice of table features using Delta's standard serialization (camelCase). -pub(crate) fn format_features(features: &[TableFeature]) -> String { - let feature_strings: Vec<&str> = features.iter().map(|f| f.as_ref()).collect_vec(); - format!("[{}]", feature_strings.join(", ")) +/// Enforce that `protocol.min_reader_version()` lies within +/// [`MIN_VALID_RW_VERSION`]..=[`MAX_VALID_READER_VERSION`]. Below the minimum yields +/// [`Error::InvalidProtocol`]; above the maximum yields [`Error::Unsupported`]. +pub(crate) fn check_reader_version_range(protocol: &Protocol) -> DeltaResult<()> { + require!( + protocol.min_reader_version() >= MIN_VALID_RW_VERSION, + Error::InvalidProtocol(format!( + "min_reader_version must be >= {MIN_VALID_RW_VERSION}, got {}", + protocol.min_reader_version() + )) + ); + if protocol.min_reader_version() > MAX_VALID_READER_VERSION { + return Err(Error::unsupported(format!( + "Unsupported minimum reader version {}", + protocol.min_reader_version() + ))); + } + Ok(()) +} + +/// Protocol-level check that the kernel can read tables governed by `protocol`. +/// +/// Unlike `TableConfiguration::ensure_operation_supported`, this does not require a +/// `Metadata` action or any table properties. +pub(crate) fn ensure_table_can_be_read(protocol: &Protocol) -> DeltaResult<()> { + check_reader_version_range(protocol)?; + + for feature in extract_enabled_reader_features(protocol) { + match feature.info().kernel_support { + KernelSupport::Supported => {} + KernelSupport::NotSupported => { + return Err(Error::unsupported(format!( + "Feature '{feature}' is not supported by kernel", + ))); + } + KernelSupport::Custom(_) => {} + } + } + + Ok(()) } #[cfg(test)] mod tests { + use rstest::rstest; + use super::*; #[test] @@ -755,6 +984,126 @@ mod tests { assert_eq!(&typed_writer, mixed_writer); } + /// Expected outcome of `ensure_table_can_be_read` for a given protocol: either readable, + /// or an error of a specific variant. + enum ExpectRead { + Ok, + InvalidProtocol, + Unsupported, + } + + #[rstest] + #[case::reader_version_below_minimum( + Protocol::new_unchecked(0, 1, None, None), + ExpectRead::InvalidProtocol + )] + #[case::reader_version_above_maximum( + Protocol::new_unchecked(99, 1, None, None), + ExpectRead::Unsupported + )] + #[case::legacy_reader_v1(Protocol::try_new_legacy(1, 1).unwrap(), ExpectRead::Ok)] + #[case::legacy_reader_v2(Protocol::try_new_legacy(2, 5).unwrap(), ExpectRead::Ok)] + #[case::v3_empty_reader_features( + Protocol::new_unchecked(3, 7, Some(vec![]), Some(vec![])), + ExpectRead::Ok + )] + #[case::supported_explicit_feature( + Protocol::try_new_modern( + [TableFeature::DeletionVectors], + [TableFeature::DeletionVectors], + ) + .unwrap(), + ExpectRead::Ok + )] + #[case::unknown_reader_feature( + Protocol::try_new_modern( + [TableFeature::unknown("notARealFeature")], + [TableFeature::unknown("notARealFeature")], + ) + .unwrap(), + ExpectRead::Unsupported + )] + #[case::custom_support_feature( + Protocol::try_new_modern( + [TableFeature::CatalogManaged], + [TableFeature::CatalogManaged], + ) + .unwrap(), + ExpectRead::Ok + )] + // adaptiveMetadata-preview is gated by the `adaptive-metadata-in-dev` cargo feature: readable + // only when the flag is on, otherwise rejected as unsupported. + #[cfg_attr( + feature = "adaptive-metadata-in-dev", + case::adaptive_metadata_supported( + Protocol::try_new_modern( + [TableFeature::AdaptiveMetadataPreview], + [TableFeature::AdaptiveMetadataPreview], + ) + .unwrap(), + ExpectRead::Ok + ) + )] + #[cfg_attr( + not(feature = "adaptive-metadata-in-dev"), + case::adaptive_metadata_gated_off( + Protocol::try_new_modern( + [TableFeature::AdaptiveMetadataPreview], + [TableFeature::AdaptiveMetadataPreview], + ) + .unwrap(), + ExpectRead::Unsupported + ) + )] + fn validate_protocol_for_read(#[case] protocol: Protocol, #[case] expected: ExpectRead) { + let result = ensure_table_can_be_read(&protocol); + match expected { + ExpectRead::Ok => result.expect("protocol must be readable"), + ExpectRead::InvalidProtocol => assert!( + matches!(result, Err(Error::InvalidProtocol(_))), + "expected InvalidProtocol, got: {result:?}" + ), + ExpectRead::Unsupported => assert!( + matches!(result, Err(Error::Unsupported(_))), + "expected Unsupported, got: {result:?}" + ), + } + } + + #[test] + fn test_timestamp_ntz_legacy_alias() { + assert_eq!( + TableFeature::from("timestampNtz"), + TableFeature::TimestampWithoutTimezone + ); + assert_eq!( + TableFeature::from("timestampWithoutTimezone"), + TableFeature::TimestampWithoutTimezone + ); + + assert_eq!( + serde_json::from_str::("\"timestampNtz\"").unwrap(), + TableFeature::TimestampWithoutTimezone + ); + assert_eq!( + serde_json::from_str::("\"timestampWithoutTimezone\"").unwrap(), + TableFeature::TimestampWithoutTimezone + ); + + assert_eq!( + TableFeature::TimestampWithoutTimezone.to_string(), + "timestampNtz" + ); + assert_eq!( + TableFeature::TimestampWithoutTimezone.as_ref(), + "timestampNtz" + ); + assert_eq!( + serde_json::to_string(&TableFeature::TimestampWithoutTimezone).unwrap(), + "\"timestampNtz\"" + ); + } + #[test] fn test_roundtrip_table_features() { use strum::IntoEnumIterator as _; @@ -772,6 +1121,7 @@ mod tests { TableFeature::DomainMetadata => "domainMetadata", TableFeature::IcebergCompatV1 => "icebergCompatV1", TableFeature::IcebergCompatV2 => "icebergCompatV2", + TableFeature::IcebergCompatV3 => "icebergCompatV3", TableFeature::ClusteredTable => "clustering", TableFeature::MaterializePartitionColumns => "materializePartitionColumns", TableFeature::CatalogManaged => "catalogManaged", @@ -785,13 +1135,17 @@ mod tests { TableFeature::VacuumProtocolCheck => "vacuumProtocolCheck", TableFeature::VariantType => "variantType", TableFeature::VariantTypePreview => "variantType-preview", + TableFeature::VariantShredding => "variantShredding", TableFeature::VariantShreddingPreview => "variantShredding-preview", + TableFeature::AdaptiveMetadataPreview => "adaptiveMetadata-preview", + TableFeature::AllowColumnDefaults => "allowColumnDefaults", + TableFeature::GeospatialType => "geospatial", TableFeature::Unknown(_) => continue, // tested in test_unknown_features }; // strum assert_eq!(feature.to_string(), expected); - assert_eq!(feature, expected.into_table_feature()); + assert_eq!(feature, TableFeature::from(expected)); // json let serialized = serde_json::to_string(&feature).unwrap(); diff --git a/kernel/src/table_features/timestamp_ntz.rs b/kernel/src/table_features/timestamp_ntz.rs index 0230bbe2af..b85ca084ce 100644 --- a/kernel/src/table_features/timestamp_ntz.rs +++ b/kernel/src/table_features/timestamp_ntz.rs @@ -3,12 +3,10 @@ use super::TableFeature; use crate::schema::{PrimitiveType, Schema}; use crate::table_configuration::TableConfiguration; -use crate::transforms::SchemaTransform; +use crate::transforms::{transform_output_type, SchemaTransform}; use crate::utils::require; use crate::{DeltaResult, Error}; -use std::borrow::Cow; - /// Validates that if a table schema contains TIMESTAMP_NTZ columns, the table must have the /// TimestampWithoutTimezone feature in both reader and writer features. pub(crate) fn validate_timestamp_ntz_feature_support(tc: &TableConfiguration) -> DeltaResult<()> { @@ -27,19 +25,19 @@ pub(crate) fn validate_timestamp_ntz_feature_support(tc: &TableConfiguration) -> /// Checks if any column in the schema (including nested structs, arrays, maps) uses /// the TIMESTAMP_NTZ primitive type. pub(crate) fn schema_contains_timestamp_ntz(schema: &Schema) -> bool { - let mut uses_timestamp_ntz = UsesTimestampNtz(false); - let _ = uses_timestamp_ntz.transform_struct(schema); - uses_timestamp_ntz.0 + UsesTimestampNtz.transform_struct(schema).is_err() } -struct UsesTimestampNtz(bool); +struct UsesTimestampNtz; impl<'a> SchemaTransform<'a> for UsesTimestampNtz { - fn transform_primitive(&mut self, ptype: &'a PrimitiveType) -> Option> { - if *ptype == PrimitiveType::TimestampNtz { - self.0 = true; + transform_output_type!(|'a, T| Result<(), ()>); + + fn transform_primitive(&mut self, ptype: &'a PrimitiveType) -> Result<(), ()> { + match ptype { + PrimitiveType::TimestampNtz => Err(()), + _ => Ok(()), } - None } } @@ -48,7 +46,7 @@ mod tests { use crate::actions::Protocol; use crate::schema::{DataType, PrimitiveType, StructField, StructType}; use crate::table_features::TableFeature; - use crate::utils::test_utils::assert_schema_feature_validation; + use crate::unit_test_utils::assert_schema_feature_validation; #[test] fn test_timestamp_ntz_feature_validation() { @@ -64,11 +62,11 @@ mod tests { StructField::new("id", DataType::INTEGER, false), StructField::new( "nested", - DataType::Struct(Box::new(StructType::new_unchecked([StructField::new( + StructType::new_unchecked([StructField::new( "inner_ts", DataType::Primitive(PrimitiveType::TimestampNtz), true, - )]))), + )]), true, ), ]); diff --git a/kernel/src/table_properties/deserialize.rs b/kernel/src/table_properties/deserialize.rs index bd4f38a03d..129c38fe83 100644 --- a/kernel/src/table_properties/deserialize.rs +++ b/kernel/src/table_properties/deserialize.rs @@ -5,13 +5,13 @@ use std::num::NonZero; use std::time::Duration; +use tracing::warn; + use super::*; use crate::expressions::ColumnName; use crate::table_features::ColumnMappingMode; use crate::utils::require; -use tracing::warn; - const SECONDS_PER_MINUTE: u64 = 60; const SECONDS_PER_HOUR: u64 = 60 * SECONDS_PER_MINUTE; const SECONDS_PER_DAY: u64 = 24 * SECONDS_PER_HOUR; @@ -54,6 +54,9 @@ fn try_parse(props: &mut TableProperties, k: &str, v: &str) -> Option<()> { props.checkpoint_write_stats_as_struct = Some(parse_bool(v)?) } COLUMN_MAPPING_MODE => props.column_mapping_mode = ColumnMappingMode::try_from(v).ok(), + COLUMN_MAPPING_MAX_COLUMN_ID => { + props.column_mapping_max_column_id = Some(parse_non_negative(v)?) + } DATA_SKIPPING_NUM_INDEXED_COLS => { props.data_skipping_num_indexed_cols = DataSkippingNumIndexedCols::try_from(v).ok() } @@ -68,6 +71,7 @@ fn try_parse(props: &mut TableProperties, k: &str, v: &str) -> Option<()> { ENABLE_TYPE_WIDENING => props.enable_type_widening = Some(parse_bool(v)?), ENABLE_ICEBERG_COMPAT_V1 => props.enable_iceberg_compat_v1 = Some(parse_bool(v)?), ENABLE_ICEBERG_COMPAT_V2 => props.enable_iceberg_compat_v2 = Some(parse_bool(v)?), + ENABLE_ICEBERG_COMPAT_V3 => props.enable_iceberg_compat_v3 = Some(parse_bool(v)?), ISOLATION_LEVEL => props.isolation_level = IsolationLevel::try_from(v).ok(), LOG_RETENTION_DURATION => props.log_retention_duration = Some(parse_interval(v)?), ENABLE_EXPIRED_LOG_CLEANUP => props.enable_expired_log_cleanup = Some(parse_bool(v)?), @@ -87,6 +91,12 @@ fn try_parse(props: &mut TableProperties, k: &str, v: &str) -> Option<()> { props.materialized_row_commit_version_column_name = Some(v.to_string()) } ROW_TRACKING_SUSPENDED => props.row_tracking_suspended = Some(parse_bool(v)?), + PARQUET_FORMAT_VERSION => props.parquet_format_version = Some(v.to_string()), + PARQUET_COMPRESSION_CODEC => { + // Preserve unrecognized codecs in `unknown_properties` so connectors can fall back + // themselves. + props.parquet_compression_codec = Some(ParquetCompressionCodec::try_from(v).ok()?) + } ENABLE_IN_COMMIT_TIMESTAMPS => props.enable_in_commit_timestamps = Some(parse_bool(v)?), IN_COMMIT_TIMESTAMP_ENABLEMENT_VERSION => { props.in_commit_timestamp_enablement_version = Some(parse_non_negative(v)?) @@ -106,8 +116,8 @@ pub(crate) fn parse_positive_int(s: &str) -> Option> { NonZero::new(parse_non_negative(s)?) } -/// Deserialize a string representing a non-negative integer into an `Option`. Returns `Some` if -/// successfully parses, and `None` otherwise. +/// Deserialize a string representing a non-negative integer into an `Option`. Returns `Some` +/// if successfully parses, and `None` otherwise. pub(crate) fn parse_non_negative(s: &str) -> Option where i64: TryInto, @@ -217,6 +227,8 @@ fn parse_interval_impl(value: &str) -> Result { #[cfg(test)] mod tests { + use ParquetCompressionCodec::*; + use super::*; #[test] @@ -371,4 +383,79 @@ mod tests { "Interval 'interval -25 hours' cannot be negative".to_string() ); } + + #[test] + fn test_parse_compression_codec_accepts_all_variants_case_insensitively() { + for (input, expected) in [ + ("uncompressed", Uncompressed), + ("UNCOMPRESSED", Uncompressed), + ("UnCoMpReSsEd", Uncompressed), + ("none", Uncompressed), + ("NONE", Uncompressed), + ("nOnE", Uncompressed), + ("snappy", Snappy), + ("SNAPPY", Snappy), + ("sNaPpY", Snappy), + ("gzip", Gzip), + ("GZIP", Gzip), + ("gZiP", Gzip), + ("lz4", Lz4), + ("LZ4", Lz4), + ("lZ4", Lz4), + ("lz4_raw", Lz4Raw), + ("LZ4_RAW", Lz4Raw), + ("Lz4_Raw", Lz4Raw), + ("zstd", Zstd), + ("ZSTD", Zstd), + ("zStD", Zstd), + ] { + let props = TableProperties::from([(PARQUET_COMPRESSION_CODEC, input)]); + assert_eq!(props.parquet_compression_codec, Some(expected), "{input}"); + assert!(props.unknown_properties.is_empty(), "{input}"); + } + } + + #[test] + fn test_parse_compression_codec_unknown_falls_through_to_unknown_properties() { + let props = TableProperties::from([(PARQUET_COMPRESSION_CODEC, "brotli")]); + assert_eq!(props.parquet_compression_codec, None); + assert_eq!( + props.unknown_properties.get(PARQUET_COMPRESSION_CODEC), + Some(&"brotli".to_string()) + ); + } + + #[test] + fn test_compression_codec_round_trips_to_canonical_protocol_string() { + // Each variant must Display-format to its Delta-protocol canonical name (lowercased, + // snake_case). The `Uncompressed` variant has `none` as a parser-only alias; its + // canonical output is `uncompressed`. + for (variant, expected) in [ + (Zstd, "zstd"), + (Uncompressed, "uncompressed"), + (Snappy, "snappy"), + (Gzip, "gzip"), + (Lz4, "lz4"), + (Lz4Raw, "lz4_raw"), + ] { + assert_eq!(variant.to_string(), expected); + assert_eq!(<&'static str>::from(variant), expected); + // And the canonical string parses back to the same variant. + assert_eq!( + ParquetCompressionCodec::try_from(expected).unwrap(), + variant + ); + } + } + + #[test] + fn test_compression_codec_or_default_falls_back_to_zstd_when_absent() { + // Absent: applies the spec-recommended Zstd fallback. + let props = TableProperties::default(); + assert_eq!(props.compression_codec_or_default(), Zstd); + + // Present: returns the parsed value as-is. + let props = TableProperties::from([(PARQUET_COMPRESSION_CODEC, "snappy")]); + assert_eq!(props.compression_codec_or_default(), Snappy); + } } diff --git a/kernel/src/table_properties.rs b/kernel/src/table_properties/mod.rs similarity index 77% rename from kernel/src/table_properties.rs rename to kernel/src/table_properties/mod.rs index 947ea5e7d7..21d2740953 100644 --- a/kernel/src/table_properties.rs +++ b/kernel/src/table_properties/mod.rs @@ -14,12 +14,12 @@ use std::collections::HashMap; use std::num::NonZero; use std::time::Duration; +use strum::{Display, EnumString, IntoStaticStr}; + use crate::expressions::ColumnName; use crate::table_features::ColumnMappingMode; use crate::{Error, Version}; -use strum::EnumString; - mod deserialize; pub use deserialize::ParseIntervalError; @@ -43,6 +43,7 @@ pub(crate) const ENABLE_DELETION_VECTORS: &str = "delta.enableDeletionVectors"; pub(crate) const ENABLE_TYPE_WIDENING: &str = "delta.enableTypeWidening"; pub(crate) const ENABLE_ICEBERG_COMPAT_V1: &str = "delta.enableIcebergCompatV1"; pub(crate) const ENABLE_ICEBERG_COMPAT_V2: &str = "delta.enableIcebergCompatV2"; +pub(crate) const ENABLE_ICEBERG_COMPAT_V3: &str = "delta.enableIcebergCompatV3"; pub(crate) const ISOLATION_LEVEL: &str = "delta.isolationLevel"; pub(crate) const LOG_RETENTION_DURATION: &str = "delta.logRetentionDuration"; pub(crate) const ENABLE_EXPIRED_LOG_CLEANUP: &str = "delta.enableExpiredLogCleanup"; @@ -58,6 +59,8 @@ pub(crate) const MATERIALIZED_ROW_ID_COLUMN_NAME: &str = pub(crate) const MATERIALIZED_ROW_COMMIT_VERSION_COLUMN_NAME: &str = "delta.rowTracking.materializedRowCommitVersionColumnName"; pub(crate) const ROW_TRACKING_SUSPENDED: &str = "delta.rowTrackingSuspended"; +pub(crate) const PARQUET_FORMAT_VERSION: &str = "delta.parquet.format.version"; +pub(crate) const PARQUET_COMPRESSION_CODEC: &str = "delta.parquet.compression.codec"; pub(crate) const ENABLE_IN_COMMIT_TIMESTAMPS: &str = "delta.enableInCommitTimestamps"; pub(crate) const IN_COMMIT_TIMESTAMP_ENABLEMENT_VERSION: &str = "delta.inCommitTimestampEnablementVersion"; @@ -87,7 +90,8 @@ pub struct TableProperties { /// E.g. if checkpoint interval = 10, then a checkpoint should be written every 10 commits. pub checkpoint_interval: Option>, - /// true for Delta Lake to write file statistics in checkpoints in JSON format for the stats column. + /// true for Delta Lake to write file statistics in checkpoints in JSON format for the stats + /// column. pub checkpoint_write_stats_as_json: Option, /// true for Delta Lake to write file statistics to checkpoints in struct format for the @@ -98,12 +102,17 @@ pub struct TableProperties { /// Parquet columns that use different names. pub column_mapping_mode: Option, + /// The largest column-mapping ID assigned in the table schema. ALTER TABLE operations + /// that add new column-mapped columns must allocate IDs strictly greater than this value + /// and bump the property accordingly. + pub column_mapping_max_column_id: Option, + /// The number of columns for Delta Lake to collect statistics about for data skipping. /// A value of -1 means to collect statistics for all columns. Updating this property does /// not automatically collect statistics again; instead, it redefines the statistics schema /// of the Delta table. Specifically, it changes the behavior of future statistics collection - /// (such as during appends and optimizations) as well as data skipping (such as ignoring column - /// statistics beyond this number, even when such statistics exist). + /// (such as during appends and optimizations) as well as data skipping (such as ignoring + /// column statistics beyond this number, even when such statistics exist). pub data_skipping_num_indexed_cols: Option, /// A comma-separated list of column names on which Delta Lake collects statistics to enhance @@ -112,8 +121,8 @@ pub struct TableProperties { pub data_skipping_stats_columns: Option>, /// The shortest duration for Delta Lake to keep logically deleted data files before deleting - /// them physically. This is to prevent failures in stale readers after compactions or partition - /// overwrites. + /// them physically. This is to prevent failures in stale readers after compactions or + /// partition overwrites. /// /// This value should be large enough to ensure that: /// @@ -142,6 +151,10 @@ pub struct TableProperties { /// ensures compatibility with Apache Iceberg V2 table format. pub enable_iceberg_compat_v2: Option, + /// Whether Iceberg compatibility V3 is enabled for this table. When enabled, Delta Lake + /// ensures compatibility with Apache Iceberg V3 table format. + pub enable_iceberg_compat_v3: Option, + /// The degree to which a transaction must be isolated from modifications made by concurrent /// transactions. /// @@ -160,14 +173,15 @@ pub struct TableProperties { /// Whether to clean up expired checkpoints/commits in the delta log. pub enable_expired_log_cleanup: Option, - /// true for Delta to generate a random prefix for a file path instead of partition information. + /// true for Delta to generate a random prefix for a file path instead of partition + /// information. /// /// For example, this may improve Amazon S3 performance when Delta Lake needs to send very high /// volumes of Amazon S3 calls to better partition across S3 servers. pub randomize_file_prefixes: Option, - /// When delta.randomizeFilePrefixes is set to true, the number of characters that Delta - /// generates for random prefixes. + /// The number of characters to use for random file path prefixes. Used when + /// `delta.randomizeFilePrefixes` is true or when column mapping is enabled. pub random_prefix_length: Option>, /// The shortest duration within which new snapshots will retain transaction identifiers (for @@ -202,6 +216,17 @@ pub struct TableProperties { /// The name of the internal column that contains the materialized row commit version. pub materialized_row_commit_version_column_name: Option, + /// The Parquet format version used when writing data files. Valid values are `"1.0.0"` + /// (DataPageV1) and any `"2.x.x"` such as `"2.12.0"` (DataPageV2). Writers SHOULD default + /// to `"1.0.0"` when absent. Connectors read this to configure their Parquet writers. + pub parquet_format_version: Option, + + /// Compression codec to use when writing new Parquet data and checkpoint files. Connectors + /// SHOULD honor this property when configuring their Parquet writer. Use + /// [`TableProperties::compression_codec_or_default`] to apply the protocol-recommended + /// fallback ([`ParquetCompressionCodec::Zstd`]) when this field is `None`. + pub parquet_compression_codec: Option, + /// Whether to enable [In-Commit Timestamps]. The in-commit timestamps writer feature strongly /// associates a monotonically increasing timestamp with each commit by storing it in the /// commit's metadata. @@ -212,8 +237,8 @@ pub struct TableProperties { /// The version of the table at which in-commit timestamps were enabled. pub in_commit_timestamp_enablement_version: Option, - /// The timestamp of the table at which in-commit timestamps were enabled. This must be the same - /// as the inCommitTimestamp of the commit when this feature was enabled. + /// The timestamp of the table at which in-commit timestamps were enabled. This must be the + /// same as the inCommitTimestamp of the commit when this feature was enabled. pub in_commit_timestamp_enablement_timestamp: Option, /// any unrecognized properties are passed through and ignored by the parser @@ -232,6 +257,33 @@ impl TableProperties { pub fn should_write_stats_as_struct(&self) -> bool { self.checkpoint_write_stats_as_struct.unwrap_or(false) } + + /// Returns whether to emit a random alphanumeric prefix in file paths regardless of column + /// mapping mode. Default: `false`. + pub fn should_randomize_file_prefixes(&self) -> bool { + self.randomize_file_prefixes.unwrap_or(false) + } + + /// Returns the number of characters to use for random file path prefixes. + /// Default: `2`. + pub fn random_prefix_length(&self) -> NonZero { + const DEFAULT: NonZero = NonZero::::new(2).unwrap(); + self.random_prefix_length + .and_then(|n| usize::try_from(n.get()).ok()) + .and_then(NonZero::new) + .unwrap_or(DEFAULT) + } + + /// Returns the Parquet compression codec for new data and checkpoint files, applying the + /// Delta protocol's recommended fallback ([`ParquetCompressionCodec::Zstd`]) when the + /// table property is absent. + /// + /// Use the returned value's `Display` impl (`codec.to_string()`) or `Into<&'static str>` + /// (`codec.into()`) to get the canonical Delta-protocol string for a Parquet writer. + pub fn compression_codec_or_default(&self) -> ParquetCompressionCodec { + self.parquet_compression_codec + .unwrap_or(ParquetCompressionCodec::Zstd) + } } /// Default number of leaf columns to collect statistics on when `dataSkippingNumIndexedCols` @@ -287,9 +339,9 @@ pub enum IsolationLevel { WriteSerializable, /// SnapshotIsolation is a guarantee that all reads made in a transaction will see a consistent - /// snapshot of the database (in practice it reads the last committed values that existed at the - /// time it started), and the transaction itself will successfully commit only if no updates - /// it has made conflict with any concurrent updates made since that snapshot. + /// snapshot of the database (in practice it reads the last committed values that existed at + /// the time it started), and the transaction itself will successfully commit only if no + /// updates it has made conflict with any concurrent updates made since that snapshot. SnapshotIsolation, } @@ -304,12 +356,39 @@ pub enum CheckpointPolicy { V2, } +/// Compression codec for newly written Parquet data files, controlled by the +/// `delta.parquet.compression.codec` table property. +/// +/// Per the Delta protocol, parsing is case-insensitive, and `none` is accepted as an alias for +/// `uncompressed`. When the property is absent, writers SHOULD default to [`Self::Zstd`]. +/// +/// See [Table Properties] in the Delta protocol. +/// +/// [Table Properties]: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#table-properties +#[derive(Debug, Display, EnumString, IntoStaticStr, Copy, Clone, PartialEq, Eq)] +#[strum(serialize_all = "snake_case", ascii_case_insensitive)] +pub enum ParquetCompressionCodec { + /// `zstd`. Recommended fallback per the Delta protocol when the property is absent. + Zstd, + /// `uncompressed` (alias: `none`). No compression. + #[strum(serialize = "uncompressed", serialize = "none")] + Uncompressed, + /// `snappy`. + Snappy, + /// `gzip`. + Gzip, + /// `lz4`. Deprecated by the Delta protocol (Hadoop framing); prefer [`Self::Lz4Raw`]. + Lz4, + /// `lz4_raw`. LZ4 block format. + Lz4Raw, +} + #[cfg(test)] mod tests { - use super::*; + use std::collections::HashMap; + use super::*; use crate::expressions::column_name; - use std::collections::HashMap; #[test] fn test_property_key_constants() { @@ -345,6 +424,7 @@ mod tests { assert_eq!(ENABLE_TYPE_WIDENING, "delta.enableTypeWidening"); assert_eq!(ENABLE_ICEBERG_COMPAT_V1, "delta.enableIcebergCompatV1"); assert_eq!(ENABLE_ICEBERG_COMPAT_V2, "delta.enableIcebergCompatV2"); + assert_eq!(ENABLE_ICEBERG_COMPAT_V3, "delta.enableIcebergCompatV3"); assert_eq!(ISOLATION_LEVEL, "delta.isolationLevel"); assert_eq!(LOG_RETENTION_DURATION, "delta.logRetentionDuration"); assert_eq!(ENABLE_EXPIRED_LOG_CLEANUP, "delta.enableExpiredLogCleanup"); @@ -370,6 +450,8 @@ mod tests { "delta.rowTracking.materializedRowCommitVersionColumnName" ); assert_eq!(ROW_TRACKING_SUSPENDED, "delta.rowTrackingSuspended"); + assert_eq!(PARQUET_FORMAT_VERSION, "delta.parquet.format.version"); + assert_eq!(PARQUET_COMPRESSION_CODEC, "delta.parquet.compression.codec"); assert_eq!( ENABLE_IN_COMMIT_TIMESTAMPS, "delta.enableInCommitTimestamps" @@ -421,6 +503,19 @@ mod tests { assert_eq!(table_properties.enable_iceberg_compat_v2, Some(false)); } + #[test] + fn test_parse_iceberg_compat_v3() { + let properties = + HashMap::from([(ENABLE_ICEBERG_COMPAT_V3.to_string(), "true".to_string())]); + let table_properties = TableProperties::from(properties.iter()); + assert_eq!(table_properties.enable_iceberg_compat_v3, Some(true)); + + let properties = + HashMap::from([(ENABLE_ICEBERG_COMPAT_V3.to_string(), "false".to_string())]); + let table_properties = TableProperties::from(properties.iter()); + assert_eq!(table_properties.enable_iceberg_compat_v3, Some(false)); + } + #[test] fn known_key_unknown_val() { let properties = HashMap::from([(APPEND_ONLY.to_string(), "wack".to_string())]); @@ -433,6 +528,34 @@ mod tests { assert_eq!(table_properties, expected); } + #[test] + fn column_mapping_max_column_id_invalid_falls_through_to_unknown() { + // Non-integer and negative values are invalid; following the established pattern, + // they fall through to `unknown_properties` rather than failing the whole parse. + for invalid in ["not_a_number", "-5", "1.5"] { + let properties = HashMap::from([( + COLUMN_MAPPING_MAX_COLUMN_ID.to_string(), + invalid.to_string(), + )]); + let parsed = TableProperties::from(properties.iter()); + assert_eq!( + parsed.column_mapping_max_column_id, None, + "input: {invalid}" + ); + assert!(parsed + .unknown_properties + .contains_key(COLUMN_MAPPING_MAX_COLUMN_ID)); + } + } + + #[test] + fn column_mapping_max_column_id_zero_is_valid() { + let properties = + HashMap::from([(COLUMN_MAPPING_MAX_COLUMN_ID.to_string(), "0".to_string())]); + let parsed = TableProperties::from(properties.iter()); + assert_eq!(parsed.column_mapping_max_column_id, Some(0)); + } + #[test] fn allow_unknown_keys() { let properties = [("unknown_properties".to_string(), "two words".to_string())]; @@ -462,6 +585,7 @@ mod tests { (CHECKPOINT_WRITE_STATS_AS_JSON, "true"), (CHECKPOINT_WRITE_STATS_AS_STRUCT, "true"), (COLUMN_MAPPING_MODE, "id"), + (COLUMN_MAPPING_MAX_COLUMN_ID, "42"), (DATA_SKIPPING_NUM_INDEXED_COLS, "-1"), (DATA_SKIPPING_STATS_COLUMNS, "col1,col2"), (DELETED_FILE_RETENTION_DURATION, "interval 1 second"), @@ -470,6 +594,7 @@ mod tests { (ENABLE_TYPE_WIDENING, "true"), (ENABLE_ICEBERG_COMPAT_V1, "true"), (ENABLE_ICEBERG_COMPAT_V2, "true"), + (ENABLE_ICEBERG_COMPAT_V3, "true"), (ISOLATION_LEVEL, "snapshotIsolation"), (LOG_RETENTION_DURATION, "interval 2 seconds"), (ENABLE_EXPIRED_LOG_CLEANUP, "true"), @@ -489,6 +614,8 @@ mod tests { (ENABLE_IN_COMMIT_TIMESTAMPS, "true"), (IN_COMMIT_TIMESTAMP_ENABLEMENT_VERSION, "15"), (IN_COMMIT_TIMESTAMP_ENABLEMENT_TIMESTAMP, "1612345678"), + (PARQUET_FORMAT_VERSION, "2.12.0"), + (PARQUET_COMPRESSION_CODEC, "zstd"), ]; let actual = TableProperties::from(properties.into_iter()); let expected = TableProperties { @@ -499,6 +626,7 @@ mod tests { checkpoint_write_stats_as_json: Some(true), checkpoint_write_stats_as_struct: Some(true), column_mapping_mode: Some(ColumnMappingMode::Id), + column_mapping_max_column_id: Some(42), data_skipping_num_indexed_cols: Some(DataSkippingNumIndexedCols::AllColumns), data_skipping_stats_columns: Some(vec![column_name!("col1"), column_name!("col2")]), deleted_file_retention_duration: Some(Duration::new(1, 0)), @@ -507,6 +635,7 @@ mod tests { enable_type_widening: Some(true), enable_iceberg_compat_v1: Some(true), enable_iceberg_compat_v2: Some(true), + enable_iceberg_compat_v3: Some(true), isolation_level: Some(IsolationLevel::SnapshotIsolation), log_retention_duration: Some(Duration::new(2, 0)), enable_expired_log_cleanup: Some(true), @@ -524,6 +653,8 @@ mod tests { row_tracking_suspended: Some(false), enable_in_commit_timestamps: Some(true), in_commit_timestamp_enablement_version: Some(15), + parquet_format_version: Some("2.12.0".to_string()), + parquet_compression_codec: Some(ParquetCompressionCodec::Zstd), in_commit_timestamp_enablement_timestamp: Some(1_612_345_678), unknown_properties: HashMap::new(), }; diff --git a/kernel/src/timestamp_timezone.rs b/kernel/src/timestamp_timezone.rs new file mode 100644 index 0000000000..08e84391c6 --- /dev/null +++ b/kernel/src/timestamp_timezone.rs @@ -0,0 +1,199 @@ +//! Reader-timezone handling for offset-less timestamp strings. + +use chrono::{ + DateTime, FixedOffset, MappedLocalTime, NaiveDate, NaiveDateTime, NaiveTime, Offset, TimeDelta, + TimeZone, +}; +use chrono_tz::Tz; + +use crate::{DeltaResult, Error}; + +/// A validated IANA timezone or fixed UTC offset used to interpret local timestamps. +#[derive(Clone, Copy, Debug)] +pub(crate) enum TimestampTimezone { + Iana(Tz), + Fixed(FixedOffset), +} + +impl Default for TimestampTimezone { + fn default() -> Self { + Self::Iana(chrono_tz::UTC) + } +} + +impl TimestampTimezone { + /// Parses an IANA timezone name or a fixed offset. + /// + /// # Errors + /// + /// Returns an error when `value` is neither a known IANA timezone nor a supported fixed + /// offset. + pub(crate) fn parse(value: &str) -> DeltaResult { + if value.starts_with('+') || value.starts_with('-') { + return parse_fixed_offset(value) + .map(Self::Fixed) + .ok_or_else(|| Error::generic(format!("Invalid timestamp timezone: {value}"))); + } + value + .parse::() + .map(Self::Iana) + .map_err(|_| Error::generic(format!("Invalid timestamp timezone: {value}"))) + } + + /// Parses a timestamp string, honoring an explicit offset in `raw` when present. + /// + /// Returns `None` when `raw` is not a valid timestamp or cannot be represented. + pub(crate) fn parse_timestamp(self, raw: &str) -> Option { + if let Ok(timestamp) = DateTime::parse_from_str(raw, "%+") { + return Some(timestamp.timestamp_micros()); + } + let local_datetime = NaiveDateTime::parse_from_str(raw, "%Y-%m-%d %H:%M:%S%.f") + .or_else(|_| NaiveDateTime::parse_from_str(raw, "%Y-%m-%dT%H:%M:%S%.f")) + .or_else(|_| { + NaiveDate::parse_from_str(raw, "%Y-%m-%d").map(|date| date.and_time(NaiveTime::MIN)) + }) + .ok()?; + match self { + Self::Iana(timezone) => resolve_local_timestamp(local_datetime, timezone), + Self::Fixed(timezone) => resolve_local_timestamp(local_datetime, timezone), + } + } +} + +fn parse_fixed_offset(value: &str) -> Option { + let (sign, value) = match value.as_bytes().first()? { + b'+' => (1, &value[1..]), + b'-' => (-1, &value[1..]), + _ => return None, + }; + let mut parts = value.split(':'); + let parse_component = |part: &str| { + (!part.is_empty() && part.bytes().all(|byte| byte.is_ascii_digit())) + .then(|| part.parse::().ok()) + .flatten() + }; + let hours = parse_component(parts.next()?)?; + let minutes = parts.next().map_or(Some(0), parse_component)?; + let seconds = parts.next().map_or(Some(0), parse_component)?; + if parts.next().is_some() + || value.is_empty() + || hours > 18 + || minutes > 59 + || seconds > 59 + || (hours == 18 && (minutes != 0 || seconds != 0)) + { + return None; + } + FixedOffset::east_opt(sign * (hours * 3_600 + minutes * 60 + seconds)) +} + +fn resolve_local_timestamp( + local_datetime: NaiveDateTime, + timezone: T, +) -> Option { + match timezone.from_local_datetime(&local_datetime) { + MappedLocalTime::Single(timestamp) => Some(timestamp.timestamp_micros()), + MappedLocalTime::Ambiguous(earlier, _) => Some(earlier.timestamp_micros()), + MappedLocalTime::None => { + // A forward transition can skip more than one hour, so walk backward until reaching + // the pre-transition side of the gap. + let offset = (1..=48).find_map(|hours| { + let before_transition = + local_datetime.checked_sub_signed(TimeDelta::hours(hours))?; + match timezone.from_local_datetime(&before_transition) { + MappedLocalTime::Single(timestamp) => Some(timestamp.offset().fix()), + MappedLocalTime::Ambiguous(earlier, _) => Some(earlier.offset().fix()), + MappedLocalTime::None => None, + } + })?; + local_datetime + .checked_sub_signed(TimeDelta::seconds(i64::from(offset.local_minus_utc()))) + .map(|utc_datetime| utc_datetime.and_utc().timestamp_micros()) + } + } +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + + fn expected_timestamp_micros(value: &str) -> i64 { + DateTime::parse_from_rfc3339(value) + .unwrap() + .timestamp_micros() + } + + #[rstest] + #[case::utc("UTC", "2024-01-15 12:30:45.123456", "2024-01-15T12:30:45.123456Z")] + #[case::iana_winter( + "America/Los_Angeles", + "2024-01-15 12:30:45.123456", + "2024-01-15T20:30:45.123456Z" + )] + #[case::iana_summer( + "America/Los_Angeles", + "2024-06-15 08:00:00.500500", + "2024-06-15T15:00:00.500500Z" + )] + #[case::fixed_offset( + "+12:45:30", + "2024-01-15 12:30:45.123456", + "2024-01-14T23:45:15.123456Z" + )] + #[case::explicit_offset_wins( + "America/Los_Angeles", + "2024-01-15 12:30:45+02:00", + "2024-01-15T10:30:45Z" + )] + #[case::iso_t_separator("UTC", "2024-01-15T12:30:45", "2024-01-15T12:30:45Z")] + #[case::date_only("UTC", "2024-01-15", "2024-01-15T00:00:00Z")] + #[case::dst_overlap("America/Los_Angeles", "2024-11-03 01:30:00", "2024-11-03T08:30:00Z")] + #[case::dst_gap("America/Los_Angeles", "2024-03-10 02:30:00", "2024-03-10T10:30:00Z")] + #[case::skipped_day("Pacific/Apia", "2011-12-30 12:00:00", "2011-12-30T22:00:00Z")] + fn parses_timestamp_in_reader_timezone( + #[case] timezone: &str, + #[case] raw: &str, + #[case] expected: &str, + ) { + assert_eq!( + TimestampTimezone::parse(timezone) + .unwrap() + .parse_timestamp(raw), + Some(expected_timestamp_micros(expected)) + ); + } + + #[rstest] + #[case::zero("+0")] + #[case::hours("+5")] + #[case::hours_minutes("-05:30")] + #[case::hours_minutes_seconds("+12:45:30")] + #[case::positive_limit("+18:00:00")] + #[case::negative_limit("-18")] + fn accepts_supported_fixed_offsets(#[case] timezone: &str) { + assert!(TimestampTimezone::parse(timezone).is_ok()); + } + + #[rstest] + #[case::empty("")] + #[case::bare_positive_sign("+")] + #[case::over_max_second("+18:00:01")] + #[case::over_max_hour("+19")] + #[case::minutes_out_of_range("+00:60")] + #[case::seconds_out_of_range("+00:00:60")] + #[case::extra_component("+05:30:00:00")] + #[case::unknown_name("Not/AZone")] + fn rejects_invalid_timezones(#[case] timezone: &str) { + assert!(TimestampTimezone::parse(timezone).is_err()); + } + + #[test] + fn rejects_invalid_timestamp() { + assert_eq!( + TimestampTimezone::default().parse_timestamp("not a timestamp"), + None + ); + } +} diff --git a/kernel/src/transaction/alter_table.rs b/kernel/src/transaction/alter_table.rs new file mode 100644 index 0000000000..0ebf56005d --- /dev/null +++ b/kernel/src/transaction/alter_table.rs @@ -0,0 +1,82 @@ +//! Alter table transaction types and constructor. +//! +//! This module defines the [`AlterTableTransaction`] type alias and the +//! [`try_new_alter_table`](AlterTableTransaction::try_new_alter_table) constructor. +//! The builder logic lives in [`builder::alter_table`](super::builder::alter_table). + +#![allow(unreachable_pub)] + +use std::marker::PhantomData; +use std::sync::Arc; + +use crate::committer::Committer; +use crate::metrics::MetricId; +use crate::snapshot::SnapshotRef; +use crate::table_configuration::TableConfiguration; +use crate::transaction::{AlterTable, Transaction}; +use crate::utils::current_time_ms; +use crate::DeltaResult; + +/// A type alias for alter-table transactions. +/// +/// This provides a restricted API surface that only exposes operations valid during ALTER +/// commands. Data file operations are not available at compile time because `AlterTable` +/// does not implement [`SupportsDataFiles`](super::SupportsDataFiles). +pub type AlterTableTransaction = Transaction; + +impl AlterTableTransaction { + /// Create a new transaction for altering a table's schema. Produces a metadata-only commit + /// that emits an updated Metadata action with the evolved schema. + /// + /// The `effective_table_config` is the evolved table configuration (new schema, same + /// protocol). It must be fully validated before calling this constructor (e.g. schema + /// operations applied, protocol feature checks passed). The `read_snapshot` provides the + /// pre-commit table state (version, previous protocol/metadata, ICT timestamps) used for + /// commit versioning and post-commit snapshots. + /// + /// This is typically called via `AlterTableTransactionBuilder::build()` rather than directly. + pub(crate) fn try_new_alter_table( + read_snapshot: SnapshotRef, + effective_table_config: TableConfiguration, + committer: Box, + correlation_id: Option>, + ) -> DeltaResult { + let span = tracing::info_span!( + "txn", + path = %read_snapshot.table_root(), + read_version = read_snapshot.version(), + operation = "ALTER TABLE", + ); + + Ok(Transaction { + span, + operation_id: MetricId::new(), + correlation_id, + read_snapshot_opt: Some(read_snapshot), + effective_table_config, + should_emit_protocol: false, + should_emit_metadata: true, + committer, + operation: Some("ALTER TABLE".to_string()), + engine_info: None, + add_files_metadata: vec![], + remove_files_metadata: vec![], + set_transactions: vec![], + commit_timestamp: current_time_ms()?, + user_domain_metadata_additions: vec![], + system_domain_metadata_additions: vec![], + user_domain_removals: vec![], + data_change: false, + column_defaults_acknowledged: false, + engine_commit_info: None, + // TODO(#2446): match delta-spark's per-op isBlindAppend policy + // (ADD/DROP/DROP NOT NULL -> true, SET NOT NULL -> false). Hardcoded false for + // now: safe, but misses the true-case optimization delta-spark applies. + is_blind_append: false, + dv_matched_files: vec![], + num_dv_updates: 0, + physical_clustering_columns: None, + _state: PhantomData, + }) + } +} diff --git a/kernel/src/transaction/builder/alter_table.rs b/kernel/src/transaction/builder/alter_table.rs new file mode 100644 index 0000000000..6803248d47 --- /dev/null +++ b/kernel/src/transaction/builder/alter_table.rs @@ -0,0 +1,241 @@ +//! Builder for ALTER TABLE (schema evolution) transactions. +//! +//! This module contains [`AlterTableTransactionBuilder`], which uses a type-state pattern to +//! enforce valid operation chaining at compile time. +//! +//! # Type States +//! +//! - [`Ready`]: Initial state. Operations are available, but `build()` is not (at least one +//! operation is required). +//! - [`Modifying`]: After any chainable schema operation. More ops can be chained, and `build()` is +//! available. See [`AlterTableTransactionBuilder`] for ops. +//! +//! # Transitions +//! +//! Each `impl` block below is gated by a state bound and documents which operations that +//! state enables. Chainable schema operations live on `impl` and transition +//! the builder to a chainable state; `build()` lives on states that are buildable. +//! +//! ```ignore +//! // Allowed: at least one op queued before build(). +//! snapshot.alter_table().add_column(field).build(engine, committer)?; +//! +//! // Not allowed: build() is not defined on Ready (no ops queued). +//! snapshot.alter_table().build(engine, committer)?; // compile error +//! ``` + +use std::marker::PhantomData; +use std::sync::Arc; + +use crate::committer::Committer; +use crate::expressions::ColumnName; +use crate::schema::StructField; +use crate::snapshot::SnapshotRef; +use crate::table_configuration::TableConfiguration; +use crate::table_features::{ + schema_has_column_mapping_metadata, strip_stray_column_mapping_metadata, ColumnMappingMode, + Operation, TableFeature, +}; +use crate::table_properties::COLUMN_MAPPING_MAX_COLUMN_ID; +use crate::transaction::alter_table::AlterTableTransaction; +use crate::transaction::schema_evolution::{ + apply_schema_operations, SchemaEvolutionResult, SchemaOperation, +}; +use crate::utils::FoldWithOption as _; +use crate::{DeltaResult, Engine, Error}; + +/// Initial state: `build()` is not yet available (at least one operation is required). +/// See [`Chainable`] for the operations available on this state. +pub struct Ready; + +/// State after at least one operation has been added. `build()` is available. +/// See [`Chainable`] for the operations available on this state. +pub struct Modifying; + +/// Marker trait for builder states that accept chainable schema operations. Grouping states +/// under one bound lets each op (like `add_column`) live on a single `impl` +/// block -- chainable states share the body rather than duplicating it per state. +/// +/// Sealed: external types cannot implement this, keeping the set of chainable states closed. +pub trait Chainable: sealed::Sealed {} +impl Chainable for Ready {} +impl Chainable for Modifying {} + +mod sealed { + pub trait Sealed {} + impl Sealed for super::Ready {} + impl Sealed for super::Modifying {} +} + +/// Builder for constructing an [`AlterTableTransaction`] with schema evolution operations. +/// +/// Uses a type-state pattern (`S`) to enforce at compile time: +/// - At least one schema operation must be queued before `build()` is callable. +/// - Only operations valid for the current state can be chained. This will disallow incompatible +/// chaining. +pub struct AlterTableTransactionBuilder { + snapshot: SnapshotRef, + operations: Vec, + correlation_id: Option>, + // PhantomData marker for builder state (Ready or Modifying). + // Zero-sized; only affects which methods are available at compile time. + _state: PhantomData, +} + +impl AlterTableTransactionBuilder { + // Reconstructs the builder with a different PhantomData marker, changing which methods + // are available at compile time (e.g. Ready -> Modifying enables `build()`). All real + // fields are moved as-is; only the zero-sized type state changes. + // + // `T` (distinct from the struct's `S`) lets the caller pick the target state: + // `self.transition::()` returns `AlterTableTransactionBuilder`. + fn transition(self) -> AlterTableTransactionBuilder { + AlterTableTransactionBuilder { + snapshot: self.snapshot, + operations: self.operations, + correlation_id: self.correlation_id, + _state: PhantomData, + } + } + + /// Attach an opaque, caller-supplied correlation id for joining the alter-table commit's metric + /// events to the caller's own request or operation id. An empty id is treated as unset. + pub fn with_correlation_id(mut self, correlation_id: impl Into>) -> Self { + self.correlation_id = Some(correlation_id.into()).filter(|id| !id.is_empty()); + self + } +} + +impl AlterTableTransactionBuilder { + /// Create a new builder from a snapshot. + pub(crate) fn new(snapshot: SnapshotRef) -> Self { + AlterTableTransactionBuilder { + snapshot, + operations: Vec::new(), + correlation_id: None, + _state: PhantomData, + } + } +} + +impl AlterTableTransactionBuilder { + /// Add a new top-level column to the table schema. + /// + /// The field must not already exist in the schema (case-insensitive). The field must be + /// nullable because existing data files do not contain this column and will read NULL for it. + /// `field` and any of its nested fields must not carry `delta.columnMapping.id` or + /// `delta.columnMapping.physicalName` annotations. + /// + /// These constraints are validated during [`build()`](AlterTableTransactionBuilder::build). + pub fn add_column(mut self, field: StructField) -> AlterTableTransactionBuilder { + self.operations.push(SchemaOperation::AddColumn { field }); + self.transition() + } + + /// Change a column's nullability from NOT NULL to nullable. If the column is already + /// nullable, the op is a no-op but still generates a commit. + /// + /// Note: this matches Spark's behavior. + pub fn set_nullable(mut self, column: ColumnName) -> AlterTableTransactionBuilder { + self.operations + .push(SchemaOperation::SetNullable { column }); + self.transition() + } +} + +impl AlterTableTransactionBuilder { + /// Validate and apply schema operations, then build the [`AlterTableTransaction`]. + /// + /// This method: + /// 1. Validates the table supports writes + /// 2. Applies each operation sequentially against the evolving schema + /// 3. Constructs new Metadata action with evolved schema + /// 4. Builds the evolved table configuration + /// 5. Creates the transaction + /// + /// # Errors + /// + /// - The table enables `icebergCompatV3` or `allowColumnDefaults`, which ALTER TABLE does not + /// yet support + /// - Any individual operation fails validation (see per-method errors above) + /// - Table does not support writes (unsupported features) + /// - The evolved schema requires protocol features not enabled on the table (e.g. adding a + /// `timestampNtz` column without the `timestampNtz` feature) + pub fn build( + self, + _engine: &dyn Engine, + committer: Box, + ) -> DeltaResult { + let table_config = self.snapshot.table_configuration(); + // We don't support ALTER TABLE on tables with icebergCompatV3 enabled yet. See + // [`crate::table_features::ICEBERG_COMPAT_V3_INFO`] for the tracking issue. + if table_config.is_feature_enabled(&TableFeature::IcebergCompatV3) { + return Err(Error::unsupported( + "ALTER TABLE is not yet supported on tables with icebergCompatV3 enabled", + )); + } + // TODO(#2630): Support ALTER TABLE on tables with column defaults. + if table_config.is_feature_enabled(&TableFeature::AllowColumnDefaults) { + return Err(Error::unsupported( + "ALTER TABLE is not yet supported on tables with allowColumnDefaults enabled", + )); + } + // Rejects writes to tables kernel can't safely commit to: writer version out of + // kernel's supported range, unsupported writer features, or schemas with SQL-expression + // invariants. Runs on the pre-alter snapshot; future ALTER variants that change the + // protocol must also re-check this on the evolved `TableConfiguration`. + table_config.ensure_operation_supported(Operation::Write)?; + + let schema = Arc::unwrap_or_clone(table_config.logical_schema()); + let column_mapping_mode = table_config.column_mapping_mode(); + let current_max_column_id = table_config.table_properties().column_mapping_max_column_id; + // Whether the pre-alter schema already carried column-mapping metadata -- the only fact the + // strip below needs from it. Captured as a bool (not a clone) before + // `apply_schema_operations` consumes `schema` by value. Short-circuits outside + // `None` mode, where no strip fires. + let current_has_cm = column_mapping_mode == ColumnMappingMode::None + && schema_has_column_mapping_metadata(&schema); + let SchemaEvolutionResult { + schema: evolved_schema, + new_max_column_id, + } = apply_schema_operations( + schema, + self.operations, + column_mapping_mode, + current_max_column_id, + )?; + + // Only in `None` mode: if this ALTER introduced column-mapping annotations into a table + // that was clean before it, strip them; residual annotations already present on the + // table are left in place (see `strip_stray_column_mapping_metadata`). + let evolved_schema = if column_mapping_mode == ColumnMappingMode::None { + strip_stray_column_mapping_metadata(current_has_cm, &evolved_schema) + .map_or(evolved_schema, Arc::new) + } else { + evolved_schema + }; + + let evolved_metadata = table_config + .metadata() + .clone() + .with_schema(evolved_schema.clone())? + .fold_with(new_max_column_id, |evolved_metadata, id| { + evolved_metadata + .with_configuration_entry(COLUMN_MAPPING_MAX_COLUMN_ID, id.to_string()) + }); + + // Validates the evolved metadata against the protocol. + let evolved_table_config = TableConfiguration::try_new_with_schema( + table_config, + evolved_metadata, + evolved_schema, + )?; + + AlterTableTransaction::try_new_alter_table( + self.snapshot, + evolved_table_config, + committer, + self.correlation_id, + ) + } +} diff --git a/kernel/src/transaction/builder/create_table.rs b/kernel/src/transaction/builder/create_table.rs index 68b20762ca..28cec45b3e 100644 --- a/kernel/src/transaction/builder/create_table.rs +++ b/kernel/src/transaction/builder/create_table.rs @@ -6,36 +6,47 @@ //! Use [`create_table()`](super::super::create_table::create_table) as the entry point rather //! than constructing the builder directly. -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; use std::sync::Arc; use itertools::Itertools; use url::Url; +use uuid::Uuid; use crate::actions::{DomainMetadata, Metadata, Protocol}; use crate::clustering::{create_clustering_domain_metadata, validate_clustering_columns}; use crate::committer::Committer; use crate::expressions::ColumnName; -use crate::log_segment::LogSegment; +use crate::schema::validation::validate_schema; use crate::schema::variant_utils::schema_contains_variant_type; -use crate::schema::SchemaRef; -use crate::snapshot::Snapshot; +use crate::schema::{ + normalize_column_names_to_schema_casing, schema_contains_non_null_fields, DataType, SchemaRef, + StructType, +}; use crate::table_configuration::TableConfiguration; use crate::table_features::{ - assign_column_mapping_metadata, get_any_level_column_physical_name, - get_column_mapping_mode_from_properties, schema_contains_timestamp_ntz, ColumnMappingMode, - FeatureType, TableFeature, SET_TABLE_FEATURE_SUPPORTED_PREFIX, - SET_TABLE_FEATURE_SUPPORTED_VALUE, + add_feature_to_lists, assign_column_mapping_metadata, auto_enable_property_driven_features, + find_max_column_id_in_schema, get_any_level_column_physical_name, + get_column_mapping_mode_from_properties, schema_contains_timestamp_ntz, + strip_stray_column_mapping_metadata, ColumnMappingMode, TableFeature, + SET_TABLE_FEATURE_SUPPORTED_PREFIX, SET_TABLE_FEATURE_SUPPORTED_VALUE, }; use crate::table_properties::{ - COLUMN_MAPPING_MAX_COLUMN_ID, COLUMN_MAPPING_MODE, DELTA_PROPERTY_PREFIX, - ENABLE_IN_COMMIT_TIMESTAMPS, + TableProperties, APPEND_ONLY, CHECKPOINT_INTERVAL, CHECKPOINT_WRITE_STATS_AS_JSON, + CHECKPOINT_WRITE_STATS_AS_STRUCT, COLUMN_MAPPING_MAX_COLUMN_ID, COLUMN_MAPPING_MODE, + DATA_SKIPPING_NUM_INDEXED_COLS, DATA_SKIPPING_STATS_COLUMNS, DELETED_FILE_RETENTION_DURATION, + DELTA_PROPERTY_PREFIX, ENABLE_CHANGE_DATA_FEED, ENABLE_DELETION_VECTORS, + ENABLE_EXPIRED_LOG_CLEANUP, ENABLE_ICEBERG_COMPAT_V1, ENABLE_ICEBERG_COMPAT_V2, + ENABLE_ICEBERG_COMPAT_V3, ENABLE_IN_COMMIT_TIMESTAMPS, ENABLE_ROW_TRACKING, + ENABLE_TYPE_WIDENING, LOG_RETENTION_DURATION, MATERIALIZED_ROW_COMMIT_VERSION_COLUMN_NAME, + MATERIALIZED_ROW_ID_COLUMN_NAME, PARQUET_FORMAT_VERSION, ROW_TRACKING_SUSPENDED, + SET_TRANSACTION_RETENTION_DURATION, }; use crate::transaction::create_table::CreateTableTransaction; use crate::transaction::data_layout::DataLayout; use crate::transaction::Transaction; use crate::utils::{current_time_ms, try_parse_uri}; -use crate::{DeltaResult, Engine, Error, StorageHandler, PRE_COMMIT_VERSION}; +use crate::{DeltaResult, Engine, Error, StorageHandler}; /// Table features allowed to be enabled via `delta.feature.*=supported` during CREATE TABLE. /// @@ -50,26 +61,69 @@ const ALLOWED_DELTA_FEATURES: &[TableFeature] = &[ TableFeature::InCommitTimestamp, // VacuumProtocolCheck ensures consistent protocol checks during VACUUM TableFeature::VacuumProtocolCheck, + // CatalogManaged enables catalog-managed table support + TableFeature::CatalogManaged, // Note: Clustering is NOT included here. Users should not enable clustering via // `delta.feature.clustering = supported`. Instead, clustering is enabled by // specifying clustering columns via `with_data_layout()`. - // As features are supported, add them here: - // TableFeature::DeletionVectors, + TableFeature::DeletionVectors, TableFeature::V2Checkpoint, + // Simple protocol-only features: enabling these only updates the protocol action. + // They can also be auto-enabled via their enablement properties (e.g. delta.appendOnly=true) + // through `maybe_auto_enable_property_driven_features`. + TableFeature::AppendOnly, + TableFeature::ChangeDataFeed, + TableFeature::TypeWidening, + TableFeature::RowTracking, + // Invariants is auto-enabled by `maybe_enable_invariants` when the schema has non-null + // fields. Allowing explicit `delta.feature.invariants=supported` lets users pre-enable + // the feature on an all-nullable table so a later ALTER TABLE ADD COLUMN NOT NULL does + // not need a protocol upgrade. + TableFeature::Invariants, + // MaterializePartitionColumns keeps partition columns in the data files instead of + // dropping them on write. There is no `delta.*` enablement property; the only opt-in at + // create time is the explicit feature signal + // `delta.feature.materializePartitionColumns=supported`. + TableFeature::MaterializePartitionColumns, + // IcebergCompatV3 is a writer-only feature that gates Iceberg V3 conversion compatibility. + // Dependent features (ColumnMapping, RowTracking, DomainMetadata) are auto-added during + // create table. + TableFeature::IcebergCompatV3, ]; -/// Delta properties allowed to be set during CREATE TABLE. -/// -/// This list will expand as more features are supported. -/// The allow list will be deprecated once auto feature enablement is implemented -/// like the Java Kernel. +/// The single allow-list of `delta.*` properties accepted during CREATE TABLE. Any `delta.*` +/// key not present here is rejected. const ALLOWED_DELTA_PROPERTIES: &[&str] = &[ // ColumnMapping mode property: triggers column mapping transform COLUMN_MAPPING_MODE, // InCommitTimestamp enablement property: triggers ICT auto-enablement ENABLE_IN_COMMIT_TIMESTAMPS, - // As features are supported, add them here: - // "delta.enableDeletionVectors", + // Checkpoint stats format properties + CHECKPOINT_WRITE_STATS_AS_JSON, + CHECKPOINT_WRITE_STATS_AS_STRUCT, + // Data skipping stats configuration. Neither property enables a feature; both control + // the per-file stats schema the writer collects. + DATA_SKIPPING_NUM_INDEXED_COLS, + DATA_SKIPPING_STATS_COLUMNS, + // Property-driven feature enablement properties + ENABLE_DELETION_VECTORS, + ENABLE_CHANGE_DATA_FEED, + ENABLE_TYPE_WIDENING, + APPEND_ONLY, + ENABLE_ROW_TRACKING, + // Set transaction retention duration: controls expiration of txn identifiers + SET_TRANSACTION_RETENTION_DURATION, + // Parquet format version: controls the Parquet writer version for data files + PARQUET_FORMAT_VERSION, + // IcebergCompatV3 enablement: triggers auto-enablement of ColumnMapping, + // RowTracking, DomainMetadata. + ENABLE_ICEBERG_COMPAT_V3, + // Log/checkpoint maintenance configs: stored verbatim, consumed by log cleanup / checkpoint + // scheduling. + LOG_RETENTION_DURATION, + DELETED_FILE_RETENTION_DURATION, + ENABLE_EXPIRED_LOG_CLEANUP, + CHECKPOINT_INTERVAL, ]; /// Ensures that no Delta table exists at the given path. @@ -134,39 +188,19 @@ struct ValidatedTableProperties { writer_features: Vec, } -/// Adds a feature to the appropriate reader/writer feature lists based on its type. -/// -/// - ReaderWriter features are added to both reader and writer lists -/// - Writer and Unknown features are added only to the writer list -/// -/// This function is idempotent - it won't add duplicate features. -fn add_feature_to_lists( - feature: TableFeature, - reader_features: &mut Vec, - writer_features: &mut Vec, -) { - match feature.feature_type() { - FeatureType::ReaderWriter => { - if !reader_features.contains(&feature) { - reader_features.push(feature.clone()); - } - if !writer_features.contains(&feature) { - writer_features.push(feature); - } - } - FeatureType::WriterOnly | FeatureType::Unknown => { - if !writer_features.contains(&feature) { - writer_features.push(feature); - } - } +impl ValidatedTableProperties { + /// Returns `true` iff `properties[key] == "true"`. + fn is_property_true(&self, key: &str) -> bool { + self.properties.get(key).map(String::as_str) == Some("true") } } -/// Configures clustering support for table creation (used by unit tests). +/// Test-only helper for clustering support during table creation. /// -/// Validates clustering columns, adds required features (DomainMetadata, ClusteredTable), -/// and creates the domain metadata action. -fn apply_clustering_for_table_create( +/// Validates clustering columns, adds the `DomainMetadata` and `ClusteredTable` features +/// directly, and creates the domain metadata action. +#[cfg(test)] +fn validate_clustering_and_make_domain_metadata( logical_schema: &SchemaRef, logical_columns: &[ColumnName], reader_features: &mut Vec, @@ -189,48 +223,110 @@ fn apply_clustering_for_table_create( Ok(create_clustering_domain_metadata(logical_columns)) } -/// Conditionally enables clustering for table creation based on the data layout. +/// Result of applying data layout configuration during table creation. /// -/// If clustering is specified in the data layout, this function: -/// 1. Validates clustering columns against the schema (using logical names) -/// 2. Resolves logical to physical column names (if column mapping is enabled) -/// 3. Adds DomainMetadata and ClusteredTable features to the protocol -/// 4. Creates the clustering domain metadata with physical column names +/// Contains all outputs needed by `build()` from the data layout processing step. +#[derive(Debug, Default)] +struct DataLayoutResult { + /// Domain metadata actions (clustering stores `delta.clustering` domain metadata). + system_domain_metadata: Vec, + /// Clustering columns for stats schema (physical names, `None` if not clustered). + clustering_columns: Option>, + /// Partition columns (logical names, `None` if not partitioned). + partition_columns: Option>, +} + +/// Validates partition columns against the table schema. /// -/// # Arguments +/// Similar to [`validate_clustering_columns`] (duplicate check, schema lookup), but with +/// stricter constraints: partition columns must be top-level and primitive-typed, while +/// clustering columns may be nested and accept all stats-eligible types. /// -/// * `data_layout` - The data layout (may specify clustering columns) -/// * `effective_schema` - The schema to validate against -/// * `column_mapping_mode` - The column mapping mode (determines name resolution) -/// * `validated` - The validated table properties (features will be added) +/// Partition columns must be: +/// 1. Top-level columns (nested paths are not supported) +/// 2. Present in the schema +/// 3. Not duplicated +/// 4. Of a supported primitive type (Struct, Array, and Map are rejected because the Delta protocol +/// does not define their partition-value serialization) +/// 5. A strict subset of the schema columns (at least one non-partition column required) +fn validate_partition_columns( + schema: &StructType, + partition_columns: &[ColumnName], +) -> DeltaResult<()> { + if partition_columns.is_empty() { + return Err(Error::generic("Partitioning requires at least one column")); + } + if partition_columns.len() >= schema.fields().len() { + return Err(Error::generic( + "Table must have at least one non-partition column", + )); + } + + let mut seen = HashSet::new(); + for col in partition_columns { + let path = col.path(); + if path.len() != 1 { + return Err(Error::generic(format!( + "Partition column '{}' must be a top-level column (nested paths are not supported)", + col + ))); + } + + if !seen.insert(col) { + return Err(Error::generic(format!( + "Duplicate partition column: '{col}'" + ))); + } + + // Safety: path.len() == 1 is enforced by the top-level check above + let col_name = &path[0]; + let field = schema.field(col_name).ok_or_else(|| { + Error::generic(format!("Partition column '{col}' not found in schema")) + })?; + + let DataType::Primitive(_) = field.data_type() else { + return Err(Error::generic(format!( + "Partition column '{col}' has non-primitive type '{}'. \ + Partition columns must have primitive types.", + field.data_type() + ))); + }; + } + Ok(()) +} + +/// Applies data layout configuration for table creation. /// -/// # Returns +/// Handles all [`DataLayout`] variants: /// -/// A tuple of (domain_metadata_list, clustering_columns_for_stats). -/// The clustering columns returned are physical names. -fn maybe_enable_clustering( +/// - **None**: Returns defaults (no domain metadata, no clustering/partition columns). +/// - **Clustered**: Validates clustering columns, resolves to physical names, adds the +/// `DomainMetadata` and `ClusteredTable` features, creates clustering domain metadata. +/// - **Partitioned**: Validates partition columns and stores logical names. No domain metadata or +/// special features are needed (partitioning is a core Delta feature). +fn apply_data_layout( data_layout: &DataLayout, effective_schema: &SchemaRef, column_mapping_mode: ColumnMappingMode, validated: &mut ValidatedTableProperties, -) -> DeltaResult<(Vec, Option>)> { +) -> DeltaResult { match data_layout { + DataLayout::None => Ok(DataLayoutResult::default()), + DataLayout::Clustered { columns } => { - // Validate using logical names against the schema - // (Schema field names are always logical, even with column mapping) - validate_clustering_columns(effective_schema, columns)?; - - // Resolve logical to physical column names for domain metadata. - // When column mapping is enabled, clustering stores physical names. - // Supports both top-level and nested columns. - let physical_columns: Vec = columns + // Normalize clustering column names to match schema casing. This allows users + // to specify clustering columns case-insensitively (e.g. schema has columns + // "A", "B", "C" and user clusters by "c", "a"). + let normalized = normalize_column_names_to_schema_casing(effective_schema, columns); + validate_clustering_columns(effective_schema, &normalized)?; + + let physical_columns: Vec = normalized .iter() .map(|c| { get_any_level_column_physical_name(effective_schema, c, column_mapping_mode) }) .try_collect()?; - // Add required features add_feature_to_lists( TableFeature::DomainMetadata, &mut validated.reader_features, @@ -242,12 +338,25 @@ fn maybe_enable_clustering( &mut validated.writer_features, ); - // Create domain metadata with physical names let dm = create_clustering_domain_metadata(&physical_columns); - Ok((vec![dm], Some(physical_columns))) + Ok(DataLayoutResult { + system_domain_metadata: vec![dm], + clustering_columns: Some(physical_columns), + partition_columns: None, + }) + } + + DataLayout::Partitioned { columns } => { + let normalized = normalize_column_names_to_schema_casing(effective_schema, columns); + validate_partition_columns(effective_schema, &normalized)?; + + Ok(DataLayoutResult { + system_domain_metadata: vec![], + clustering_columns: None, + partition_columns: Some(normalized), + }) } - DataLayout::None => Ok((vec![], None)), } } @@ -275,20 +384,189 @@ fn maybe_enable_timestamp_ntz(schema: &SchemaRef, validated: &mut ValidatedTable } } -/// Conditionally adds the `inCommitTimestamp` feature to the protocol when -/// `delta.enableInCommitTimestamps=true` is set in the table properties. -fn maybe_enable_in_commit_timestamps(validated: &mut ValidatedTableProperties) { - let enabled = validated +/// Conditionally adds the `invariants` writer feature to the protocol when the schema contains +/// any non-null column (`nullable: false`) anywhere in the tree. +/// +/// Delta-Spark treats `nullable: false` as an implicit column invariant and requires the +/// `invariants` writer feature to be listed in the protocol's `writerFeatures` to read/write +/// such tables. Auto-enabling ensures kernel-created tables with non-null columns are +/// compatible with Spark readers/writers. +/// +/// Explicit `delta.invariants` metadata annotations are rejected by +/// `validate_schema`, so this only flips on the feature for nullability-driven +/// invariants. Kernel does not itself enforce the null mask at write time -- it relies on +/// the engine's `ParquetHandler` to do so. Kernel's default `ParquetHandler` uses +/// `arrow-rs`, whose `RecordBatch::try_new` rejects null values in fields marked +/// `nullable: false`. Other engine implementations must provide an equivalent guarantee +/// in their write path. +fn maybe_enable_invariants(schema: &SchemaRef, validated: &mut ValidatedTableProperties) { + if schema_contains_non_null_fields(schema) { + add_feature_to_lists( + TableFeature::Invariants, + &mut validated.reader_features, + &mut validated.writer_features, + ); + } +} + +/// Auto-enables allowed property-driven features from the table properties (see +/// [`auto_enable_property_driven_features`]). +fn maybe_auto_enable_property_driven_features(validated: &mut ValidatedTableProperties) { + let table_properties = TableProperties::from(validated.properties.iter()); + auto_enable_property_driven_features( + ALLOWED_DELTA_FEATURES, + &table_properties, + &mut validated.reader_features, + &mut validated.writer_features, + ); +} + +/// Sets materialized column name properties when row tracking is enabled. +/// +/// Writes `delta.rowTracking.materializedRowIdColumnName` and +/// `delta.rowTracking.materializedRowCommitVersionColumnName` into the table +/// properties using UUID-based column names (`_row-id-col-{uuid}` and +/// `_row-commit-version-col-{uuid}`). These names record which physical columns +/// store materialized row IDs and commit versions. +/// +/// Only fires when `delta.enableRowTracking=true` is set. Feature-signal-only tables +/// (`delta.feature.rowTracking=supported` without the enablement property) do not get +/// these properties because the materialized columns are part of the "enabled" state, not +/// the "supported" state. +fn maybe_set_materialized_row_tracking_column_name_properties( + validated: &mut ValidatedTableProperties, +) { + if validated .properties - .get(ENABLE_IN_COMMIT_TIMESTAMPS) - .is_some_and(|v| v == "true"); - if enabled { + .get(ENABLE_ROW_TRACKING) + .is_none_or(|v| v != "true") + { + return; + } + validated.properties.insert( + MATERIALIZED_ROW_ID_COLUMN_NAME.to_string(), + format!("_row-id-col-{}", Uuid::new_v4()), + ); + validated.properties.insert( + MATERIALIZED_ROW_COMMIT_VERSION_COLUMN_NAME.to_string(), + format!("_row-commit-version-col-{}", Uuid::new_v4()), + ); +} + +/// Ensures that `inCommitTimestamp` is enabled when `catalogManaged` is present. Adds the ICT +/// feature to the protocol and sets the enablement property if not already present. +fn maybe_enable_ict_for_catalog_managed( + validated: &mut ValidatedTableProperties, +) -> DeltaResult<()> { + let has_catalog_managed = validated + .writer_features + .contains(&TableFeature::CatalogManaged); + if has_catalog_managed { + if validated + .properties + .get(ENABLE_IN_COMMIT_TIMESTAMPS) + .is_some_and(|v| v != "true") + { + return Err(Error::generic(format!( + "Catalog-managed tables require '{ENABLE_IN_COMMIT_TIMESTAMPS}=true', \ + but it was explicitly set to '{}'", + validated.properties[ENABLE_IN_COMMIT_TIMESTAMPS] + ))); + } add_feature_to_lists( TableFeature::InCommitTimestamp, &mut validated.reader_features, &mut validated.writer_features, ); + validated + .properties + .entry(ENABLE_IN_COMMIT_TIMESTAMPS.to_string()) + .or_insert_with(|| "true".to_string()); } + Ok(()) +} + +/// Witness that all property-flipping passes which must run before column mapping is applied +/// have completed. +#[must_use] +#[derive(Debug)] +struct PreColumnMappingResolved; + +/// When `delta.enableIcebergCompatV3=true` is set, auto-enables V3's required dependencies in +/// `validated.properties` (defaulting them when absent, rejecting conflicting values). +/// +/// Specifically: +/// * Set `delta.columnMapping.mode` to `name` when absent, reject if it's `none`. +/// * Set `delta.enableRowTracking` to `true` when absent, reject if it's `false`. +/// * Reject if `delta.rowTrackingSuspended` is `true`. +/// * Reject if `delta.enableIcebergCompatV1` or `delta.enableIcebergCompatV2` is `true`. +/// +/// Returns a [`PreColumnMappingResolved`] witness that +/// [`maybe_apply_column_mapping_for_table_create`] requires, ensuring this pass runs first. +fn maybe_enable_iceberg_compat_v3_dependencies( + validated: &mut ValidatedTableProperties, +) -> DeltaResult { + if !validated.is_property_true(ENABLE_ICEBERG_COMPAT_V3) { + return Ok(PreColumnMappingResolved); + } + + // Column mapping: require `name` or `id`; default to `name`. + match validated + .properties + .get(COLUMN_MAPPING_MODE) + .map(String::as_str) + { + None => { + validated + .properties + .insert(COLUMN_MAPPING_MODE.to_string(), "name".to_string()); + } + Some("name") | Some("id") => {} + Some(other) => { + return Err(Error::generic(format!( + "IcebergCompatV3 requires '{COLUMN_MAPPING_MODE}' to be 'name' or 'id', got \ + '{other}'" + ))); + } + } + + // Row tracking must not be suspended (suspension cannot coexist with row tracking actively + // enabled, which V3 requires). + if validated.is_property_true(ROW_TRACKING_SUSPENDED) { + return Err(Error::generic(format!( + "IcebergCompatV3 cannot be enabled while '{ROW_TRACKING_SUSPENDED}' is 'true'" + ))); + } + + // Row tracking enablement: require `true`; default to `true`. + match validated + .properties + .get(ENABLE_ROW_TRACKING) + .map(String::as_str) + { + None => { + validated + .properties + .insert(ENABLE_ROW_TRACKING.to_string(), "true".to_string()); + } + Some("true") => {} + Some(other) => { + return Err(Error::generic(format!( + "IcebergCompatV3 requires '{ENABLE_ROW_TRACKING}' to be 'true', got '{other}'" + ))); + } + } + + // V1/V2 must not be active. + for key in [ENABLE_ICEBERG_COMPAT_V1, ENABLE_ICEBERG_COMPAT_V2] { + if validated.is_property_true(key) { + return Err(Error::generic(format!( + "IcebergCompatV3 cannot be enabled together with '{key}'" + ))); + } + } + + Ok(PreColumnMappingResolved) } /// Conditionally applies column mapping for table creation based on the mode in properties. @@ -312,6 +590,7 @@ fn maybe_enable_in_commit_timestamps(validated: &mut ValidatedTableProperties) { fn maybe_apply_column_mapping_for_table_create( schema: &SchemaRef, validated: &mut ValidatedTableProperties, + _pre_cm: PreColumnMappingResolved, ) -> DeltaResult<(SchemaRef, ColumnMappingMode)> { let column_mapping_mode = get_column_mapping_mode_from_properties(&validated.properties)?; @@ -324,9 +603,18 @@ fn maybe_apply_column_mapping_for_table_create( &mut validated.writer_features, ); - // Transform schema: assign IDs and physical names to all fields - let mut max_id = 0i64; - let transformed_schema = assign_column_mapping_metadata(schema, &mut max_id)?; + // Transform schema: preserve any pre-populated `delta.columnMapping.id` / + // `delta.columnMapping.physicalName` annotations on input fields, fill in any + // missing piece of that pair, and assign fresh metadata to bare fields (matches + // delta-spark's `DeltaColumnMapping.assignColumnIdAndPhysicalName`). Seed + // `max_id` from the schema's existing max so newly assigned IDs cannot collide + // with preserved ones. When IcebergCompatV3 is enabled, also allocate nested ids + // for `Array.element` and `Map.key`/`Map.value` and store them under + // `delta.columnMapping.nested.ids` on the nearest ancestor `StructField`. + let assign_nested_field_ids = validated.is_property_true(ENABLE_ICEBERG_COMPAT_V3); + let mut max_id = find_max_column_id_in_schema(schema).unwrap_or(0); + let transformed_schema = + assign_column_mapping_metadata(schema, &mut max_id, assign_nested_field_ids)?; // Add maxColumnId to properties validated @@ -350,6 +638,12 @@ fn maybe_apply_column_mapping_for_table_create( /// 4. Extracts reader/writer features from validated feature signals /// /// Non-delta properties (user/application properties) are always allowed. +/// +/// Note: This function does not auto-set enablement properties. A feature signal like +/// `delta.feature.deletionVectors=supported` adds the feature to the protocol but does +/// not insert `delta.enableDeletionVectors=true` into the properties. Property-driven +/// auto-enablement is handled separately by [`maybe_auto_enable_property_driven_features`] +/// called after validation. fn validate_extract_table_features_and_properties( properties: HashMap, ) -> DeltaResult { @@ -389,10 +683,19 @@ fn validate_extract_table_features_and_properties( } // Add to appropriate feature lists based on feature type + let needs_domain_metadata = feature == TableFeature::RowTracking; add_feature_to_lists(feature, &mut reader_features, &mut writer_features); + // RowTracking requires DomainMetadata as a dependency + if needs_domain_metadata { + add_feature_to_lists( + TableFeature::DomainMetadata, + &mut reader_features, + &mut writer_features, + ); + } } - // Validate remaining delta.* properties against allow list + // Validate remaining delta.* properties against the allow list for key in properties.keys() { if key.starts_with(DELTA_PROPERTY_PREFIX) && !ALLOWED_DELTA_PROPERTIES.contains(&key.as_str()) @@ -422,6 +725,7 @@ pub struct CreateTableTransactionBuilder { engine_info: String, table_properties: HashMap, data_layout: DataLayout, + correlation_id: Option>, } impl CreateTableTransactionBuilder { @@ -436,6 +740,7 @@ impl CreateTableTransactionBuilder { engine_info: engine_info.into(), table_properties: HashMap::new(), data_layout: DataLayout::None, + correlation_id: None, } } @@ -443,7 +748,8 @@ impl CreateTableTransactionBuilder { /// /// Custom application properties (those not starting with `delta.`) are always allowed. /// Delta properties (`delta.*`) are validated against an allow list during [`build()`]. - /// Feature flags (`delta.feature.*`) are not supported during CREATE TABLE. + /// Feature flags (`delta.feature.*=supported`) are supported for the subset of features + /// listed in `ALLOWED_DELTA_FEATURES`. /// /// This method can be called multiple times. If a property key already exists from a /// previous call, the new value will overwrite the old one. @@ -459,7 +765,7 @@ impl CreateTableTransactionBuilder { /// # use delta_kernel::schema::{StructType, DataType, StructField}; /// # use std::sync::Arc; /// # fn example() -> delta_kernel::DeltaResult<()> { - /// # let schema = Arc::new(StructType::try_new(vec![StructField::new("id", DataType::INTEGER, false)])?); + /// # let schema = Arc::new(StructType::try_new(vec![StructField::new("id", DataType::INTEGER, true)])?); /// let builder = create_table("/path/to/table", schema, "MyApp/1.0") /// .with_table_properties([ /// ("myapp.version", "1.0"), @@ -487,6 +793,13 @@ impl CreateTableTransactionBuilder { /// /// - [`DataLayout::None`]: No special organization (default) /// - [`DataLayout::Clustered`]: Data files are optimized for queries on clustering columns + /// - [`DataLayout::Partitioned`]: Data files are organized into directories by partition column + /// values + /// + /// Partitioning and clustering are mutually exclusive. + /// + /// Calling this method multiple times replaces the previous layout. Only the last + /// `with_data_layout()` call takes effect. /// /// # Example /// @@ -497,11 +810,16 @@ impl CreateTableTransactionBuilder { /// # use std::sync::Arc; /// # fn example() -> delta_kernel::DeltaResult<()> { /// # let schema = Arc::new(StructType::try_new(vec![ - /// # StructField::new("id", DataType::INTEGER, false), - /// # StructField::new("date", DataType::STRING, false), + /// # StructField::new("id", DataType::INTEGER, true), + /// # StructField::new("date", DataType::STRING, true), /// # ])?); - /// let builder = create_table("/path/to/table", schema, "MyApp/1.0") + /// // Clustered layout: + /// let builder = create_table("/path/to/table", schema.clone(), "MyApp/1.0") /// .with_data_layout(DataLayout::clustered(["id"])); + /// + /// // Partitioned layout: + /// let builder = create_table("/path/to/table", schema, "MyApp/1.0") + /// .with_data_layout(DataLayout::partitioned(["date"])); /// # Ok(()) /// # } /// ``` @@ -510,6 +828,13 @@ impl CreateTableTransactionBuilder { self } + /// Attach an opaque, caller-supplied correlation id for joining the create-table commit's + /// metric events to the caller's own request or operation id. An empty id is treated as unset. + pub fn with_correlation_id(mut self, correlation_id: impl Into>) -> Self { + self.correlation_id = Some(correlation_id.into()).filter(|id| !id.is_empty()); + self + } + /// Builds a [`CreateTableTransaction`] that can be committed to create the table. /// /// The returned [`CreateTableTransaction`] only exposes operations that are valid for @@ -519,9 +844,16 @@ impl CreateTableTransactionBuilder { /// This method performs validation: /// - Checks that the table path is valid /// - Verifies the table doesn't already exist - /// - Validates the schema is non-empty + /// - Rejects schemas with `delta.invariants` metadata annotations (unsupported by kernel) + /// - Validates the data layout is valid /// - Validates table properties against the allow list /// + /// Non-null columns (`nullable: false`) are allowed. The `invariants` writer feature is + /// auto-added to the protocol when the schema has any non-null column. + /// + /// Empty schemas are accepted. The resulting table cannot be read or blind-appended to + /// until columns are added via `ALTER TABLE ADD COLUMN`. + /// /// # Arguments /// /// * `engine` - The engine instance to use for validation @@ -532,7 +864,8 @@ impl CreateTableTransactionBuilder { /// Returns an error if: /// - The table path is invalid /// - A table already exists at the given path - /// - The schema is empty + /// - The schema has `delta.invariants` metadata on any column + /// - The data layout is invalid /// - Unsupported delta properties or feature flags are specified pub fn build( self, @@ -542,10 +875,6 @@ impl CreateTableTransactionBuilder { // Validate path let table_url = try_parse_uri(&self.path)?; - // Validate schema is non-empty - if self.schema.fields().len() == 0 { - return Err(Error::generic("Schema cannot be empty")); - } // Check if table already exists by looking for _delta_log directory let delta_log_url = table_url.join("_delta_log/")?; let storage = engine.storage_handler(); @@ -557,26 +886,52 @@ impl CreateTableTransactionBuilder { // - Returns reader/writer features to add to protocol let mut validated = validate_extract_table_features_and_properties(self.table_properties)?; - // Apply column mapping if mode is name or id (must happen BEFORE clustering) - let (effective_schema, column_mapping_mode) = - maybe_apply_column_mapping_for_table_create(&self.schema, &mut validated)?; + // When IcebergCompatV3 is enabled, fill in / validate required dependencies before + // column mapping is applied so the CM mode is in place. The returned witness is + // required by `maybe_apply_column_mapping_for_table_create` below. + let pre_cm = maybe_enable_iceberg_compat_v3_dependencies(&mut validated)?; + + // Apply column mapping if mode is name or id (must happen BEFORE data layout) + let (mut effective_schema, column_mapping_mode) = + maybe_apply_column_mapping_for_table_create(&self.schema, &mut validated, pre_cm)?; + + // Validate schema (column names, duplicates, no `delta.invariants` metadata). + // Empty schemas are intentionally allowed. + validate_schema(&effective_schema, column_mapping_mode)?; + + // Strip CM metadata in `None` mode: a new table has no prior schema (passed as `None`), so + // any annotation the caller supplied is newly introduced (see + // `strip_stray_column_mapping_metadata`). + if column_mapping_mode == ColumnMappingMode::None { + // A new table has no prior schema, so nothing was carried in (`current_has_cm = + // false`). + if let Some(stripped) = strip_stray_column_mapping_metadata(false, &effective_schema) { + effective_schema = Arc::new(stripped); + } + } - // Handle clustering (validates, resolves to physical names, adds features) - let (system_domain_metadata, clustering_columns) = maybe_enable_clustering( + // Validate data layout and resolve column names (physical for clustering, logical + // for partitioning). Adds required table features for clustering. + let data_layout_result = apply_data_layout( &self.data_layout, &effective_schema, column_mapping_mode, &mut validated, )?; - // Auto-enable variantType feature if schema contains Variant columns + // Schema-driven auto-enablement: detect types or annotations that require a feature maybe_enable_variant_type(&effective_schema, &mut validated); - - // Auto-enable timestampNtz feature if schema contains TimestampNTZ columns maybe_enable_timestamp_ntz(&effective_schema, &mut validated); + maybe_enable_invariants(&effective_schema, &mut validated); - // Auto-enable inCommitTimestamp feature if property is set - maybe_enable_in_commit_timestamps(&mut validated); + // Property-driven auto-enablement: check enablement properties + maybe_auto_enable_property_driven_features(&mut validated); + + // Auto-enable inCommitTimestamp for catalogManaged tables + maybe_enable_ict_for_catalog_managed(&mut validated)?; + + // Set materialized row tracking column names when row tracking is enabled. + maybe_set_materialized_row_tracking_column_name_properties(&mut validated); // Create Protocol action with table features support let protocol = @@ -584,46 +939,60 @@ impl CreateTableTransactionBuilder { // Create Metadata action with filtered properties (feature signals removed) // Use effective_schema which includes column mapping annotations if enabled + // Partition columns are validated to be top-level, so each ColumnName has + // exactly one path component. Extract it with remove(0). + let partition_columns: Vec = data_layout_result + .partition_columns + .map(|cols| cols.into_iter().map(|c| c.into_inner().remove(0)).collect()) + .unwrap_or_default(); let metadata = Metadata::try_new( None, // name None, // description effective_schema.clone(), - Vec::new(), // partition_columns - added with data layout support + partition_columns, current_time_ms()?, validated.properties, )?; - // Create pre-commit snapshot from protocol/metadata - let log_root = table_url.join("_delta_log/")?; - let log_segment = LogSegment::for_pre_commit(log_root); - let table_configuration = - TableConfiguration::try_new(metadata, protocol, table_url, PRE_COMMIT_VERSION)?; + // Build TableConfiguration directly for the new table + let table_configuration = TableConfiguration::try_new(metadata, protocol, table_url, 0)?; - // Create Transaction with pre-commit snapshot + // Create Transaction with the effective table configuration Transaction::try_new_create_table( - Arc::new(Snapshot::new(log_segment, table_configuration)), + table_configuration, self.engine_info, committer, - system_domain_metadata, - clustering_columns, + data_layout_result.system_domain_metadata, + data_layout_result.clustering_columns, + self.correlation_id, ) } } #[cfg(test)] mod tests { - use super::*; - use crate::schema::{DataType, StructField, StructType}; - - use crate::utils::test_utils::assert_result_error_with_message; use std::sync::Arc; + use rstest::rstest; + + use super::*; + use crate::expressions::{column_name, ColumnName}; + use crate::scan::data_skipping::stats_schema::StripFieldMetadataTransform; + use crate::schema::{ + schema_ref, ColumnMetadataKey, DataType, MetadataValue, StructField, StructType, + }; + use crate::table_features::FeatureType; + use crate::table_properties::{ + COLUMN_MAPPING_MAX_COLUMN_ID, ENABLE_ICEBERG_COMPAT_V1, ENABLE_ICEBERG_COMPAT_V3, + PARQUET_FORMAT_VERSION, + }; + use crate::transforms::SchemaTransform; + use crate::unit_test_utils::{ + assert_result_error_with_message, build_complex_nested_kernel_schema, + }; + fn test_schema() -> SchemaRef { - Arc::new(StructType::new_unchecked(vec![StructField::new( - "id", - DataType::INTEGER, - false, - )])) + schema_ref! { not_null "id": INTEGER } } #[test] @@ -707,44 +1076,89 @@ mod tests { validated.properties.get("custom.setting"), Some(&"value".to_string()) ); + } - // Feature signal for domainMetadata IS allowed (it's in ALLOWED_DELTA_FEATURES) - let properties = HashMap::from([( - "delta.feature.domainMetadata".to_string(), - "supported".to_string(), - )]); - let result = validate_extract_table_features_and_properties(properties); - assert!(result.is_ok()); - let validated = result.unwrap(); - // Feature signals are removed from properties (not stored in metadata) - assert!(validated.properties.is_empty()); - // DomainMetadata is a writer-only feature + #[test] + fn test_parquet_format_version_accepted() { + let properties = + HashMap::from([(PARQUET_FORMAT_VERSION.to_string(), "2.12.0".to_string())]); + let validated = validate_extract_table_features_and_properties(properties).unwrap(); + assert_eq!( + validated.properties.get(PARQUET_FORMAT_VERSION), + Some(&"2.12.0".to_string()), + ); assert!(validated.reader_features.is_empty()); - assert!(validated - .writer_features - .contains(&TableFeature::DomainMetadata)); + assert!(validated.writer_features.is_empty()); + } + + #[rstest::rstest] + #[case::num_indexed_cols_cap(DATA_SKIPPING_NUM_INDEXED_COLS, "16")] + #[case::num_indexed_cols_all(DATA_SKIPPING_NUM_INDEXED_COLS, "-1")] + #[case::stats_columns_multi(DATA_SKIPPING_STATS_COLUMNS, "a,b,c")] + #[case::stats_columns_single(DATA_SKIPPING_STATS_COLUMNS, "a")] + fn test_data_skipping_properties_accepted(#[case] key: &str, #[case] value: &str) { + let properties = HashMap::from([(key.to_string(), value.to_string())]); + let validated = validate_extract_table_features_and_properties(properties).unwrap(); + assert_eq!(validated.properties.get(key), Some(&value.to_string())); + assert!(validated.reader_features.is_empty()); + assert!(validated.writer_features.is_empty()); } #[test] - fn test_validate_unsupported_properties() { - use crate::table_properties::{APPEND_ONLY, ENABLE_CHANGE_DATA_FEED}; + fn test_metadata_maintenance_properties_accepted() { + // Log/checkpoint maintenance configs are metadata-only: accepted at CREATE, stored + // verbatim, and enable no table feature. + let properties = HashMap::from([ + ( + LOG_RETENTION_DURATION.to_string(), + "interval 30 days".to_string(), + ), + ( + DELETED_FILE_RETENTION_DURATION.to_string(), + "interval 7 days".to_string(), + ), + (ENABLE_EXPIRED_LOG_CLEANUP.to_string(), "true".to_string()), + (CHECKPOINT_INTERVAL.to_string(), "10".to_string()), + ]); + let validated = validate_extract_table_features_and_properties(properties).unwrap(); + assert_eq!( + validated.properties.get(LOG_RETENTION_DURATION), + Some(&"interval 30 days".to_string()), + ); + assert_eq!( + validated.properties.get(DELETED_FILE_RETENTION_DURATION), + Some(&"interval 7 days".to_string()), + ); + assert_eq!( + validated.properties.get(ENABLE_EXPIRED_LOG_CLEANUP), + Some(&"true".to_string()), + ); + assert_eq!( + validated.properties.get(CHECKPOINT_INTERVAL), + Some(&"10".to_string()), + ); + assert!(validated.reader_features.is_empty()); + assert!(validated.writer_features.is_empty()); + } + #[test] + fn test_validate_unsupported_properties() { // Delta properties not on allow list are rejected let mut properties = HashMap::new(); - properties.insert(ENABLE_CHANGE_DATA_FEED.to_string(), "true".to_string()); + properties.insert(ENABLE_ICEBERG_COMPAT_V1.to_string(), "true".to_string()); assert_result_error_with_message( validate_extract_table_features_and_properties(properties), - "Setting delta property 'delta.enableChangeDataFeed' is not supported", + "Setting delta property 'delta.enableIcebergCompatV1' is not supported", ); // Feature signals for features not in ALLOWED_DELTA_FEATURES are rejected let properties = HashMap::from([( - "delta.feature.deletionVectors".to_string(), + "delta.feature.identityColumns".to_string(), "supported".to_string(), )]); assert_result_error_with_message( validate_extract_table_features_and_properties(properties), - "Enabling feature 'deletionVectors' via 'delta.feature.deletionVectors' is not supported", + "Enabling feature 'identityColumns' via 'delta.feature.identityColumns' is not supported", ); // Clustering feature signal is rejected - users must use with_clustering_columns() instead @@ -760,17 +1174,17 @@ mod tests { // Mixed properties with unsupported delta property are rejected let mut properties = HashMap::new(); properties.insert("myapp.version".to_string(), "1.0".to_string()); - properties.insert(APPEND_ONLY.to_string(), "true".to_string()); + properties.insert(ENABLE_ICEBERG_COMPAT_V1.to_string(), "true".to_string()); assert_result_error_with_message( validate_extract_table_features_and_properties(properties), - "Setting delta property 'delta.appendOnly' is not supported", + "Setting delta property 'delta.enableIcebergCompatV1' is not supported", ); } #[test] fn test_clustering_support_valid() { use crate::clustering::CLUSTERING_DOMAIN_NAME; - use crate::expressions::ColumnName; + use crate::expressions::column_name; let schema = Arc::new(StructType::new_unchecked(vec![ StructField::new("id", DataType::INTEGER, false), @@ -780,9 +1194,9 @@ mod tests { let mut reader_features = vec![]; let mut writer_features = vec![]; - let dm = apply_clustering_for_table_create( + let dm = validate_clustering_and_make_domain_metadata( &schema, - &[ColumnName::new(["id"])], + &[column_name!("id")], &mut reader_features, &mut writer_features, ) @@ -798,7 +1212,7 @@ mod tests { #[test] fn test_clustering_support_multiple_columns() { - use crate::expressions::ColumnName; + use crate::expressions::column_name; let schema = Arc::new(StructType::new_unchecked(vec![ StructField::new("id", DataType::INTEGER, false), @@ -809,9 +1223,9 @@ mod tests { let mut reader_features = vec![]; let mut writer_features = vec![]; - let dm = apply_clustering_for_table_create( + let dm = validate_clustering_and_make_domain_metadata( &schema, - &[ColumnName::new(["id"]), ColumnName::new(["date"])], + &[column_name!("id"), column_name!("date")], &mut reader_features, &mut writer_features, ) @@ -827,20 +1241,16 @@ mod tests { #[test] fn test_clustering_column_not_in_schema() { - use crate::expressions::ColumnName; + use crate::expressions::column_name; - let schema = Arc::new(StructType::new_unchecked(vec![StructField::new( - "id", - DataType::INTEGER, - false, - )])); + let schema = schema_ref! { not_null "id": INTEGER }; let mut reader_features = vec![]; let mut writer_features = vec![]; - let result = apply_clustering_for_table_create( + let result = validate_clustering_and_make_domain_metadata( &schema, - &[ColumnName::new(["nonexistent"])], + &[column_name!("nonexistent")], &mut reader_features, &mut writer_features, ); @@ -855,7 +1265,7 @@ mod tests { #[test] fn test_clustering_nested_column_accepted() { use crate::clustering::CLUSTERING_DOMAIN_NAME; - use crate::expressions::ColumnName; + use crate::expressions::column_name; let address_struct = StructType::new_unchecked(vec![ StructField::new("city", DataType::STRING, true), @@ -863,14 +1273,14 @@ mod tests { ]); let schema = Arc::new(StructType::new_unchecked(vec![ StructField::new("id", DataType::INTEGER, false), - StructField::new("address", DataType::Struct(Box::new(address_struct)), true), + StructField::new("address", address_struct, true), ])); let mut reader_features = vec![]; let mut writer_features = vec![]; - let nested_col = ColumnName::new(["address", "city"]); - let dm = apply_clustering_for_table_create( + let nested_col = column_name!("address.city"); + let dm = validate_clustering_and_make_domain_metadata( &schema, &[nested_col], &mut reader_features, @@ -882,55 +1292,45 @@ mod tests { assert!(writer_features.contains(&TableFeature::ClusteredTable)); } - #[test] - fn test_with_data_layout() { + #[rstest::rstest] + #[case::clustered(DataLayout::clustered(["id"]), true, false)] + #[case::partitioned(DataLayout::partitioned(["id"]), false, true)] + #[case::none(DataLayout::default(), false, false)] + fn test_with_data_layout( + #[case] layout: DataLayout, + #[case] expect_clustered: bool, + #[case] expect_partitioned: bool, + ) { let schema = test_schema(); let builder = CreateTableTransactionBuilder::new("/path/to/table", schema, "TestApp/1.0") - .with_data_layout(DataLayout::clustered(["id"])); + .with_data_layout(layout); - assert!(builder.data_layout.is_clustered()); + assert_eq!(builder.data_layout.is_clustered(), expect_clustered); + assert_eq!(builder.data_layout.is_partitioned(), expect_partitioned); } #[rstest::rstest] #[case::variant_top_level( - Arc::new(StructType::new_unchecked(vec![ - StructField::new("id", DataType::INTEGER, false), - StructField::new("v", DataType::unshredded_variant(), true), - ])), + schema_ref! { not_null "id": INTEGER, nullable "v": (DataType::unshredded_variant()) }, &[TableFeature::VariantType], )] #[case::variant_nested( - Arc::new(StructType::new_unchecked(vec![ - StructField::new("id", DataType::INTEGER, false), - StructField::new( - "nested", - DataType::Struct(Box::new(StructType::new_unchecked(vec![ - StructField::new("inner_v", DataType::unshredded_variant(), true), - ]))), - true, - ), - ])), + schema_ref! { + not_null "id": INTEGER, + nullable "nested": { nullable "inner_v": (DataType::unshredded_variant()) }, + }, &[TableFeature::VariantType], )] #[case::ntz_top_level( - Arc::new(StructType::new_unchecked(vec![ - StructField::new("id", DataType::INTEGER, false), - StructField::new("ts", DataType::TIMESTAMP_NTZ, true), - ])), + schema_ref! { not_null "id": INTEGER, nullable "ts": TIMESTAMP_NTZ }, &[TableFeature::TimestampWithoutTimezone], )] #[case::ntz_nested( - Arc::new(StructType::new_unchecked(vec![ - StructField::new("id", DataType::INTEGER, false), - StructField::new( - "nested", - DataType::Struct(Box::new(StructType::new_unchecked(vec![ - StructField::new("inner_ts", DataType::TIMESTAMP_NTZ, true), - ]))), - true, - ), - ])), + schema_ref! { + not_null "id": INTEGER, + nullable "nested": { nullable "inner_ts": TIMESTAMP_NTZ }, + }, &[TableFeature::TimestampWithoutTimezone], )] #[case::both_variant_and_ntz( @@ -982,6 +1382,45 @@ mod tests { ); } + #[rstest::rstest] + #[case::all_nullable( + schema_ref! { nullable "id": INTEGER, nullable "name": STRING }, + false, + )] + #[case::top_level_non_null( + schema_ref! { not_null "id": INTEGER, nullable "name": STRING }, + true, + )] + #[case::nested_non_null( + schema_ref! { nullable "parent": { not_null "child": INTEGER } }, + true, + )] + #[case::variant_only( + schema_ref! { nullable "id": INTEGER, nullable "v": (DataType::unshredded_variant()) }, + false, + )] + fn test_maybe_enable_invariants(#[case] schema: SchemaRef, #[case] expect_invariants: bool) { + let mut validated = ValidatedTableProperties { + properties: HashMap::new(), + reader_features: vec![], + writer_features: vec![], + }; + + maybe_enable_invariants(&schema, &mut validated); + + assert_eq!( + validated + .writer_features + .contains(&TableFeature::Invariants), + expect_invariants, + "Expected Invariants feature presence = {expect_invariants}" + ); + assert!( + validated.reader_features.is_empty(), + "Invariants is writer-only; reader_features should be empty" + ); + } + #[rstest::rstest] #[case::property_true(&[("delta.enableInCommitTimestamps", "true")], true, true)] #[case::property_false(&[("delta.enableInCommitTimestamps", "false")], false, true)] @@ -998,7 +1437,7 @@ mod tests { .collect(); let mut validated = validate_extract_table_features_and_properties(properties).unwrap(); - maybe_enable_in_commit_timestamps(&mut validated); + maybe_auto_enable_property_driven_features(&mut validated); assert_eq!( validated @@ -1018,28 +1457,658 @@ mod tests { ); } + #[rstest::rstest] + #[case::vacuum_protocol_check(TableFeature::VacuumProtocolCheck, "vacuumProtocolCheck")] + #[case::domain_metadata(TableFeature::DomainMetadata, "domainMetadata")] + #[case::column_mapping(TableFeature::ColumnMapping, "columnMapping")] + #[case::in_commit_timestamp(TableFeature::InCommitTimestamp, "inCommitTimestamp")] + #[case::deletion_vectors(TableFeature::DeletionVectors, "deletionVectors")] + #[case::v2_checkpoint(TableFeature::V2Checkpoint, "v2Checkpoint")] + #[case::append_only(TableFeature::AppendOnly, "appendOnly")] + #[case::change_data_feed(TableFeature::ChangeDataFeed, "changeDataFeed")] + #[case::type_widening(TableFeature::TypeWidening, "typeWidening")] + #[case::catalog_managed(TableFeature::CatalogManaged, "catalogManaged")] + #[case::invariants(TableFeature::Invariants, "invariants")] + fn test_feature_signal_accepted(#[case] feature: TableFeature, #[case] feature_name: &str) { + let key = format!("delta.feature.{feature_name}"); + let properties = HashMap::from([(key, "supported".to_string())]); + let validated = validate_extract_table_features_and_properties(properties).unwrap(); + + assert!( + validated.properties.is_empty(), + "Feature signal should be removed from properties" + ); + assert!( + validated.writer_features.contains(&feature), + "{feature:?} should be in writer_features" + ); + match feature.feature_type() { + FeatureType::ReaderWriter => assert!( + validated.reader_features.contains(&feature), + "{feature:?} is ReaderWriter but missing from reader_features" + ), + _ => assert!( + validated.reader_features.is_empty(), + "{feature:?} is WriterOnly but reader_features is not empty" + ), + } + } + + fn multi_column_schema() -> SchemaRef { + Arc::new(StructType::new_unchecked(vec![ + StructField::new("id", DataType::INTEGER, false), + StructField::new("name", DataType::STRING, true), + StructField::new("date", DataType::DATE, true), + ])) + } + + struct DataLayoutExpectation { + layout: DataLayout, + has_domain_metadata: bool, + has_clustering_columns: bool, + expected_partition_columns: Option>, + expected_writer_features: Vec, + } + + #[rstest::rstest] + #[case::none(DataLayoutExpectation { + layout: DataLayout::default(), + has_domain_metadata: false, + has_clustering_columns: false, + expected_partition_columns: None, + expected_writer_features: vec![], + })] + #[case::clustered(DataLayoutExpectation { + layout: DataLayout::clustered(["id"]), + has_domain_metadata: true, + has_clustering_columns: true, + expected_partition_columns: None, + expected_writer_features: vec![TableFeature::DomainMetadata, TableFeature::ClusteredTable], + })] + #[case::partitioned_single(DataLayoutExpectation { + layout: DataLayout::partitioned(["date"]), + has_domain_metadata: false, + has_clustering_columns: false, + expected_partition_columns: Some(vec![column_name!("date")]), + expected_writer_features: vec![], + })] + #[case::partitioned_multiple(DataLayoutExpectation { + layout: DataLayout::partitioned(["id", "date"]), + has_domain_metadata: false, + has_clustering_columns: false, + expected_partition_columns: Some(vec![column_name!("id"), column_name!("date")]), + expected_writer_features: vec![], + })] + fn test_apply_data_layout(#[case] expectation: DataLayoutExpectation) { + let schema = multi_column_schema(); + let mut validated = ValidatedTableProperties { + properties: HashMap::new(), + reader_features: vec![], + writer_features: vec![], + }; + + let result = apply_data_layout( + &expectation.layout, + &schema, + ColumnMappingMode::None, + &mut validated, + ) + .unwrap(); + + assert_eq!( + !result.system_domain_metadata.is_empty(), + expectation.has_domain_metadata + ); + assert_eq!( + result.clustering_columns.is_some(), + expectation.has_clustering_columns + ); + assert_eq!( + result.partition_columns, + expectation.expected_partition_columns + ); + + for feature in &expectation.expected_writer_features { + assert!( + validated.writer_features.contains(feature), + "Expected {feature:?} in writer_features" + ); + } + } + + #[rstest::rstest] + #[case::clustered_invalid_col(DataLayout::clustered(["nonexistent"]), "not found in schema")] + #[case::partitioned_invalid_col(DataLayout::partitioned(["nonexistent"]), "not found in schema")] + #[case::partitioned_duplicate(DataLayout::partitioned(["id", "id"]), "Duplicate partition column")] + #[case::partitioned_empty(DataLayout::Partitioned { columns: vec![] }, "at least one column")] + #[case::partitioned_all_columns(DataLayout::partitioned(["id", "name", "date"]), "at least one non-partition column")] + fn test_apply_data_layout_validation_errors( + #[case] layout: DataLayout, + #[case] expected_error: &str, + ) { + let schema = multi_column_schema(); + let mut validated = ValidatedTableProperties { + properties: HashMap::new(), + reader_features: vec![], + writer_features: vec![], + }; + + let result = apply_data_layout(&layout, &schema, ColumnMappingMode::None, &mut validated); + assert!(result.is_err()); + assert!( + result.unwrap_err().to_string().contains(expected_error), + "Expected error containing '{expected_error}'" + ); + } + #[test] - fn test_vacuum_protocol_check_feature_signal() { + fn test_validate_partition_columns_nested_rejected() { + let address_struct = + StructType::new_unchecked(vec![StructField::new("city", DataType::STRING, true)]); + let schema = StructType::new_unchecked(vec![ + StructField::new("id", DataType::INTEGER, false), + StructField::new("address", address_struct, true), + ]); + + let columns = vec![column_name!("address.city")]; + let result = validate_partition_columns(&schema, &columns); + assert!(result.is_err()); + assert!(result + .unwrap_err() + .to_string() + .contains("must be a top-level column")); + } + + #[rstest::rstest] + #[case::struct_type( + "struct_col", + DataType::from(StructType::new_unchecked(vec![StructField::new("inner", DataType::STRING, false)])), + )] + #[case::array_type( + "array_col", + DataType::from(crate::schema::ArrayType::new(DataType::INTEGER, false)) + )] + #[case::map_type( + "map_col", + DataType::from(crate::schema::MapType::new(DataType::STRING, DataType::INTEGER, false)) + )] + fn test_validate_partition_columns_complex_types_rejected( + #[case] col_name: &str, + #[case] data_type: DataType, + ) { + let schema = StructType::new_unchecked(vec![ + StructField::new("id", DataType::INTEGER, false), + StructField::new(col_name, data_type, false), + ]); + let columns = vec![ColumnName::new([col_name])]; + let result = validate_partition_columns(&schema, &columns); + assert!(result.is_err()); + assert!(result + .unwrap_err() + .to_string() + .contains("non-primitive type")); + } + + #[rstest::rstest] + fn test_validate_partition_columns_nested_interval_types_rejected( + #[values(DataType::INTERVAL_YEAR_MONTH, DataType::INTERVAL_DAY_TIME)] data_type: DataType, + ) { + let schema = StructType::new_unchecked([ + StructField::not_null("id", DataType::INTEGER), + StructField::not_null( + "nested", + StructType::new_unchecked([StructField::not_null("col", data_type)]), + ), + ]); + + let error = validate_partition_columns(&schema, &[column_name!("nested.col")]) + .expect_err("nested partition columns must be rejected") + .to_string(); + assert!(error.contains("must be a top-level column")); + } + + #[rstest::rstest] + #[case::integer(DataType::INTEGER)] + #[case::string(DataType::STRING)] + #[case::date(DataType::DATE)] + #[case::timestamp(DataType::TIMESTAMP)] + #[case::boolean(DataType::BOOLEAN)] + #[case::long(DataType::LONG)] + #[case::interval_year_month(DataType::INTERVAL_YEAR_MONTH)] + #[case::interval_day_time(DataType::INTERVAL_DAY_TIME)] + fn test_validate_partition_columns_primitive_types_accepted(#[case] data_type: DataType) { + let schema = StructType::new_unchecked(vec![ + StructField::new("id", DataType::INTEGER, false), + StructField::new("col", data_type, false), + ]); + let columns = vec![column_name!("col")]; + assert!(validate_partition_columns(&schema, &columns).is_ok()); + } + + #[test] + fn test_catalog_managed_auto_enables_ict() { let properties = HashMap::from([( - "delta.feature.vacuumProtocolCheck".to_string(), + "delta.feature.catalogManaged".to_string(), "supported".to_string(), )]); - let validated = validate_extract_table_features_and_properties(properties).unwrap(); + let mut validated = validate_extract_table_features_and_properties(properties).unwrap(); + maybe_auto_enable_property_driven_features(&mut validated); + maybe_enable_ict_for_catalog_managed(&mut validated).unwrap(); + assert!( - validated.properties.is_empty(), - "Feature signal should be removed from properties" + validated + .writer_features + .contains(&TableFeature::InCommitTimestamp), + "ICT should be auto-added to writer_features" ); + assert_eq!( + validated.properties.get(ENABLE_IN_COMMIT_TIMESTAMPS), + Some(&"true".to_string()), + "delta.enableInCommitTimestamps should be set to true" + ); + } + + #[test] + fn test_catalog_managed_with_ict_true_succeeds() { + let properties = HashMap::from([ + ( + "delta.feature.catalogManaged".to_string(), + "supported".to_string(), + ), + ( + "delta.enableInCommitTimestamps".to_string(), + "true".to_string(), + ), + ]); + let mut validated = validate_extract_table_features_and_properties(properties).unwrap(); + maybe_auto_enable_property_driven_features(&mut validated); + maybe_enable_ict_for_catalog_managed(&mut validated).unwrap(); + + assert!(validated + .writer_features + .contains(&TableFeature::InCommitTimestamp)); + assert_eq!( + validated.properties.get(ENABLE_IN_COMMIT_TIMESTAMPS), + Some(&"true".to_string()), + ); + } + + /// Verifies that both activation paths add `RowTracking` and `DomainMetadata` to + /// `writer_features`. For the feature-signal path, `delta.enableRowTracking` must NOT + /// be present in the properties (signal grants support, not enablement). + #[rstest::rstest] + #[case::enablement_property( + HashMap::from([(ENABLE_ROW_TRACKING.to_string(), "true".to_string())]), + true, // enablement property is set + )] + #[case::feature_signal( + HashMap::from([("delta.feature.rowTracking".to_string(), "supported".to_string())]), + false, // enablement property is NOT set + )] + fn test_row_tracking_activation_adds_required_features( + #[case] properties: HashMap, + #[case] expect_enablement_property: bool, + ) { + let mut validated = validate_extract_table_features_and_properties(properties).unwrap(); + maybe_auto_enable_property_driven_features(&mut validated); + assert!( validated .writer_features - .contains(&TableFeature::VacuumProtocolCheck), - "VacuumProtocolCheck should be in writer_features" + .contains(&TableFeature::RowTracking), + "Expected RowTracking in writer_features" ); assert!( validated - .reader_features - .contains(&TableFeature::VacuumProtocolCheck), - "VacuumProtocolCheck should be in reader_features (ReaderWriter feature)" + .writer_features + .contains(&TableFeature::DomainMetadata), + "Expected DomainMetadata in writer_features" + ); + assert_eq!( + validated.properties.contains_key(ENABLE_ROW_TRACKING), + expect_enablement_property, + "delta.enableRowTracking presence mismatch" + ); + } + + #[test] + fn test_catalog_managed_with_ict_false_fails() { + let properties = HashMap::from([ + ( + "delta.feature.catalogManaged".to_string(), + "supported".to_string(), + ), + ( + "delta.enableInCommitTimestamps".to_string(), + "false".to_string(), + ), + ]); + let mut validated = validate_extract_table_features_and_properties(properties).unwrap(); + maybe_auto_enable_property_driven_features(&mut validated); + let err = maybe_enable_ict_for_catalog_managed(&mut validated).unwrap_err(); + assert!( + err.to_string().contains("enableInCommitTimestamps"), + "expected ICT conflict error, got: {err}" ); } + + /// Builds the icebergCompatV3 create-table test schema: + /// + /// ```json + /// { + /// "type": "struct", + /// "fields": [ + /// { + /// "name": "top", + /// "type": { + /// "type": "map", + /// "keyType": {"type": "array", "elementType": "integer"}, + /// "valueType": {"type": "struct", "fields": [{ + /// "name": "inner", + /// "type": {"type": "map", "keyType": "integer", + /// "valueType": {"type": "array", "elementType": "integer"}} + /// }]} + /// } + /// }, + /// {"name": "region", "type": "string"} + /// ] + /// } + /// ``` + fn build_iceberg_compat_v3_test_schema() -> SchemaRef { + let with_metadata = + build_complex_nested_kernel_schema(ColumnMetadataKey::ColumnMappingNestedIds.as_ref()); + let complex = StripFieldMetadataTransform + .transform_struct(&with_metadata) + .into_owned(); + let mut fields: Vec = complex.fields().cloned().collect(); + fields.push(StructField::nullable("region", DataType::STRING)); + Arc::new(StructType::try_new(fields).unwrap()) + } + + /// V3 create-table flow with the same schema for minimum and maximum feature sets. + /// Validates final features, CM id assignments, and nested-id JSON contents. + #[rstest] + #[case::v3_only( + /* extra_props */ &[], + /* expected_features */ &[ + TableFeature::IcebergCompatV3, + TableFeature::ColumnMapping, + TableFeature::RowTracking, + TableFeature::DomainMetadata, + ], + )] + #[case::v3_with_cross_features( + /* extra_props */ &[ + (ENABLE_DELETION_VECTORS, "true"), + (ENABLE_IN_COMMIT_TIMESTAMPS, "true"), + (ENABLE_TYPE_WIDENING, "true"), + (APPEND_ONLY, "true"), + (ENABLE_CHANGE_DATA_FEED, "true"), + ], + /* expected_features */ &[ + TableFeature::IcebergCompatV3, + TableFeature::ColumnMapping, + TableFeature::RowTracking, + TableFeature::DomainMetadata, + TableFeature::DeletionVectors, + TableFeature::InCommitTimestamp, + TableFeature::TypeWidening, + TableFeature::AppendOnly, + TableFeature::ChangeDataFeed, + ], + )] + fn test_create_table_iceberg_compat_v3( + #[case] extra_props: &[(&str, &str)], + #[case] expected_features: &[TableFeature], + ) { + let schema = build_iceberg_compat_v3_test_schema(); + let mut props: HashMap = + HashMap::from([(ENABLE_ICEBERG_COMPAT_V3.to_string(), "true".to_string())]); + for (k, v) in extra_props { + props.insert(k.to_string(), v.to_string()); + } + let mut validated = validate_extract_table_features_and_properties(props).unwrap(); + + // === V3 dependency defaults === + let pre_cm = maybe_enable_iceberg_compat_v3_dependencies(&mut validated).unwrap(); + assert_eq!( + validated + .properties + .get(COLUMN_MAPPING_MODE) + .map(String::as_str), + Some("name"), + ); + assert_eq!( + validated + .properties + .get(ENABLE_ROW_TRACKING) + .map(String::as_str), + Some("true"), + ); + + // === Column mapping + nested-id assignment === + let (effective_schema, mode) = + maybe_apply_column_mapping_for_table_create(&schema, &mut validated, pre_cm).unwrap(); + assert_eq!(mode, ColumnMappingMode::Name); + + // Spark-aligned two-pass numbering: + // Pass 1 (CM ids, DFS over StructFields): top=1, inner=2, region=3. + // Pass 2 (nested ids, per StructField, starting from max=3): + // top: {key:4, key.element:5, value:6} + // inner: {key:7, value:8, value.element:9} + let top = effective_schema.field("top").expect("missing top"); + let top_physical = expect_field_id_and_physical_name(top, 1); + assert_eq!( + extract_nested_ids(top), + serde_json::json!({ + format!("{top_physical}.key"): 4, + format!("{top_physical}.key.element"): 5, + format!("{top_physical}.value"): 6, + }), + "top's nested-ids JSON", + ); + + let DataType::Map(top_map) = top.data_type() else { + panic!("top must be a map"); + }; + let DataType::Struct(value_struct) = top_map.value_type() else { + panic!("top's value must be a struct"); + }; + let inner = value_struct.fields().next().expect("missing inner"); + let inner_physical = expect_field_id_and_physical_name(inner, 2); + assert_eq!( + extract_nested_ids(inner), + serde_json::json!({ + format!("{inner_physical}.key"): 7, + format!("{inner_physical}.value"): 8, + format!("{inner_physical}.value.element"): 9, + }), + "inner's nested-ids JSON", + ); + + let region = effective_schema.field("region").expect("missing region"); + let _ = expect_field_id_and_physical_name(region, 3); + assert!(!region + .metadata + .contains_key(ColumnMetadataKey::ColumnMappingNestedIds.as_ref())); + + // maxColumnId tracks the largest assigned CM id, including nested ids. + assert_eq!( + validated + .properties + .get(COLUMN_MAPPING_MAX_COLUMN_ID) + .map(String::as_str), + Some("9"), + ); + + // === Property-driven feature enablement + Invariants === + maybe_enable_invariants(&effective_schema, &mut validated); + maybe_auto_enable_property_driven_features(&mut validated); + + for f in expected_features { + assert!( + validated.writer_features.contains(f) || validated.reader_features.contains(f), + "expected {f:?} in protocol features (writer={:?}, reader={:?})", + validated.writer_features, + validated.reader_features, + ); + } + // V3 implies partition materialization without adding the MaterializePartitionColumns flag. + assert!( + !validated + .writer_features + .contains(&TableFeature::MaterializePartitionColumns), + "MaterializePartitionColumns flag should not be auto-added by V3", + ); + } + + /// Property combinations that violate V3's dependency requirements must be rejected by + /// `maybe_enable_iceberg_compat_v3_dependencies`. Some of these properties (e.g. + /// `delta.rowTrackingSuspended`, `delta.enableIcebergCompatV1`) aren't in the CREATE-TABLE + /// allow-list, so we construct `ValidatedTableProperties` directly to get around the + /// allow-list. + #[rstest] + #[case::cm_mode_none( + &[(COLUMN_MAPPING_MODE, "none")], + "delta.columnMapping.mode", + )] + #[case::row_tracking_false( + &[(ENABLE_ROW_TRACKING, "false")], + "delta.enableRowTracking", + )] + #[case::row_tracking_suspended( + &[("delta.rowTrackingSuspended", "true")], + "delta.rowTrackingSuspended", + )] + #[case::v1_concurrent( + &[(ENABLE_ICEBERG_COMPAT_V1, "true")], + "delta.enableIcebergCompatV1", + )] + #[case::v2_concurrent( + &[("delta.enableIcebergCompatV2", "true")], + "delta.enableIcebergCompatV2", + )] + fn test_v3_dependencies_rejects_invalid_combinations( + #[case] extra_props: &[(&str, &str)], + #[case] expected_substring: &str, + ) { + let mut properties: HashMap = + HashMap::from([(ENABLE_ICEBERG_COMPAT_V3.to_string(), "true".to_string())]); + for (k, v) in extra_props { + properties.insert(k.to_string(), v.to_string()); + } + let mut validated = ValidatedTableProperties { + properties, + reader_features: Vec::new(), + writer_features: Vec::new(), + }; + let err = maybe_enable_iceberg_compat_v3_dependencies(&mut validated).unwrap_err(); + assert!( + err.to_string().contains(expected_substring), + "expected error mentioning '{expected_substring}', got: {err}", + ); + } + + /// `assign_column_mapping_metadata` rejects schemas where any field has pre-populated + /// nested-ids metadata under either the canonical or the legacy key. + #[rstest] + #[case::canonical_key(ColumnMetadataKey::ColumnMappingNestedIds.as_ref())] + #[case::legacy_key(ColumnMetadataKey::ParquetFieldNestedIds.as_ref())] + fn test_create_table_v3_rejects_pre_existing_nested_ids(#[case] nested_ids_meta_key: &str) { + // The fixture's kernel schema already carries the chosen nested-ids key on the + // `top` and `inner` StructFields, so feeding it through the V3 create path must + // surface the pre-population error. + let schema = Arc::new(build_complex_nested_kernel_schema(nested_ids_meta_key)); + let mut validated = validate_extract_table_features_and_properties(HashMap::from([( + ENABLE_ICEBERG_COMPAT_V3.to_string(), + "true".to_string(), + )])) + .unwrap(); + let pre_cm = maybe_enable_iceberg_compat_v3_dependencies(&mut validated).unwrap(); + let err = maybe_apply_column_mapping_for_table_create(&schema, &mut validated, pre_cm) + .unwrap_err(); + assert!( + err.to_string().contains("has pre-populated"), + "expected pre-populated metadata error, got: {err}", + ); + } + + /// Listing IcebergCompatV3 in writerFeatures (i.e. "supported") without setting + /// `delta.enableIcebergCompatV3=true` does NOT activate V3 behavior: column mapping is + /// applied per the explicit property, but no nested ids are set on Array/Map fields. + #[test] + fn test_create_table_v3_supported_but_not_enabled_skips_nested_ids() { + let schema = build_iceberg_compat_v3_test_schema(); + let mut validated = validate_extract_table_features_and_properties(HashMap::from([ + ( + "delta.feature.icebergCompatV3".to_string(), + "supported".to_string(), + ), + (COLUMN_MAPPING_MODE.to_string(), "name".to_string()), + ])) + .unwrap(); + assert!( + validated + .writer_features + .contains(&TableFeature::IcebergCompatV3), + "V3 must be in writerFeatures (supported) for this test to be meaningful", + ); + let pre_cm = maybe_enable_iceberg_compat_v3_dependencies(&mut validated).unwrap(); + let (effective_schema, mode) = + maybe_apply_column_mapping_for_table_create(&schema, &mut validated, pre_cm).unwrap(); + assert_eq!(mode, ColumnMappingMode::Name); + + // Top-level Map field gets CM id + physicalName but no nested-ids metadata under + // either the canonical or the legacy key. + let top = effective_schema.field("top").expect("missing field top"); + assert!( + !top.metadata() + .contains_key(ColumnMetadataKey::ColumnMappingNestedIds.as_ref()), + "top should not carry delta.columnMapping.nested.ids when V3 is not enabled", + ); + assert!( + !top.metadata() + .contains_key(ColumnMetadataKey::ParquetFieldNestedIds.as_ref()), + "top should not carry parquet.field.nested.ids when V3 is not enabled", + ); + } + + /// Asserts the field has CM id == `expected_id` and a `col-` prefixed physical name; returns + /// the physical name for further use. + fn expect_field_id_and_physical_name(field: &StructField, expected_id: i64) -> String { + assert_eq!( + field + .metadata + .get(ColumnMetadataKey::ColumnMappingId.as_ref()), + Some(&MetadataValue::Number(expected_id)), + "field '{}' CM id mismatch", + field.name(), + ); + let MetadataValue::String(physical_name) = field + .metadata + .get(ColumnMetadataKey::ColumnMappingPhysicalName.as_ref()) + .unwrap_or_else(|| panic!("field '{}' missing physicalName", field.name())) + else { + panic!("field '{}' physicalName must be a String", field.name()); + }; + assert!( + physical_name.starts_with("col-"), + "field '{}' physicalName '{physical_name}' must start with 'col-'", + field.name(), + ); + physical_name.clone() + } + + /// Extracts the `delta.columnMapping.nested.ids` JSON object from a field, panicking if + /// it's missing or wrongly typed. + fn extract_nested_ids(field: &StructField) -> serde_json::Value { + let MetadataValue::Other(value) = field + .metadata + .get(ColumnMetadataKey::ColumnMappingNestedIds.as_ref()) + .unwrap_or_else(|| panic!("field '{}' missing nested-ids", field.name())) + else { + panic!("field '{}' nested-ids must be a JSON object", field.name()); + }; + value.clone() + } } diff --git a/kernel/src/transaction/builder/mod.rs b/kernel/src/transaction/builder/mod.rs index c6785cdd8a..87c516298d 100644 --- a/kernel/src/transaction/builder/mod.rs +++ b/kernel/src/transaction/builder/mod.rs @@ -5,4 +5,5 @@ // and for tests. Also allow dead_code since these are used by integration tests. #![allow(unreachable_pub, dead_code)] +pub mod alter_table; pub mod create_table; diff --git a/kernel/src/transaction/commit_info.rs b/kernel/src/transaction/commit_info.rs index 6f95ccf03d..4d592e2b0b 100644 --- a/kernel/src/transaction/commit_info.rs +++ b/kernel/src/transaction/commit_info.rs @@ -1,11 +1,11 @@ use std::sync::Arc; -use crate::actions::{get_log_commit_info_schema, CommitInfo, COMMIT_INFO_NAME}; -use crate::expressions::{MapData, Scalar, Transform}; -use crate::schema::{MapType, StructField, StructType, ToSchema}; -use crate::{DataType, Engine, EngineData, Error, Expression, ExpressionRef, IntoEngineData}; - use super::Transaction; +use crate::actions::{CommitInfo, COMMIT_INFO_NAME, LOG_COMMIT_INFO_SCHEMA}; +use crate::expressions::{MapData, Scalar}; +use crate::schema::{schema_ref, MapType, ToSchema}; +use crate::struct_patch::ProjectionStructPatchBuilder; +use crate::{DataType, Engine, EngineData, Error, Expression, ExpressionRef, IntoEngineData}; /// Builds a list of `(field_name, literal_expression)` pairs covering every [`CommitInfo`] /// field. Field names match the camelCase schema names produced by the `ToSchema` derive macro. @@ -37,7 +37,7 @@ fn commit_info_literal_exprs( map.into_iter() .map(|(k, v)| (Scalar::String(k), Scalar::String(v))), )?), - None => Scalar::Null(DataType::Map(Box::new(op_params_map_type))), + None => Scalar::null(op_params_map_type), }, )), ), @@ -73,55 +73,40 @@ impl Transaction { Some((engine_commit_info, engine_commit_info_schema)) => { let kernel_schema = CommitInfo::to_schema(); - // Step 1: Build output schema - all engine fields first, then any kernel-only - // fields that are not already present in the engine schema appended at the end. - let output_fields: Vec<_> = engine_commit_info_schema - .fields() - .map(|field| kernel_schema.field(field.name()).unwrap_or(field)) - .cloned() - .chain( - kernel_schema - .fields() - .filter(|field| !engine_commit_info_schema.contains(field.name())) - .cloned(), - ) - .collect(); - - let output_schema = StructType::new_unchecked(output_fields); - - // Step 2: Build literal expressions for each CommitInfo field. + // Step 1: Build literal expressions for each CommitInfo field. let literal_exprs = commit_info_literal_exprs(kernel_commit_info)?; - // Step 3: Build Transform. Replacements must be registered before insertions so - // that for the last engine field (which may itself be replaced), exprs is ordered - // as [replace_expr, insert_exprs...]. The evaluator emits exprs in declaration - // order, so the replace value must come first. - let last_engine_field = engine_commit_info_schema.field_names().last().cloned(); - let mut transform = Transform::new_top_level(); - - // First pass: replace fields that already exist in the engine schema. + // Step 2: Build the output schema and expression patch together. Engine fields + // pass through first, overlapping kernel fields are replaced in place, and + // kernel-only fields are appended after the last engine field. + let mut patch = ProjectionStructPatchBuilder::new(engine_commit_info_schema); for (field_name, expr_ref) in &literal_exprs { + let field = kernel_schema.field(*field_name).ok_or_else(|| { + Error::internal_error(format!( + "CommitInfo schema is missing field '{field_name}'" + )) + })?; if engine_commit_info_schema.contains(*field_name) { - transform = transform.with_replaced_field(*field_name, expr_ref.clone()); + patch = patch.replace(*field_name, field.clone(), expr_ref.clone()); } } - // Second pass: append kernel-only fields after the last engine field. for (field_name, expr_ref) in &literal_exprs { + let field = kernel_schema.field(*field_name).ok_or_else(|| { + Error::internal_error(format!( + "CommitInfo schema is missing field '{field_name}'" + )) + })?; if !engine_commit_info_schema.contains(*field_name) { - transform = transform - .with_inserted_field(last_engine_field.as_deref(), expr_ref.clone()); + patch = patch.append(field.clone(), expr_ref.clone()); } } + let (output_schema, patch) = patch.build()?; - // Step 4: Wrap the transform in a struct expression so the output matches the + // Step 3: Wrap the patch in a struct expression so the output matches the // Delta log action format `{ "commitInfo": { merged fields... } }`, consistent - // with the None branch which uses `get_log_commit_info_schema()`. - let wrapped_expr = - Expression::struct_from([Arc::new(Expression::transform(transform))]); - let wrapped_schema = Arc::new(StructType::new_unchecked([StructField::nullable( - COMMIT_INFO_NAME, - output_schema, - )])); + // with the None branch which uses `LOG_COMMIT_INFO_SCHEMA`. + let wrapped_expr = Expression::struct_from([patch]); + let wrapped_schema = schema_ref! { nullable (COMMIT_INFO_NAME): (output_schema) }; let evaluator = engine.evaluation_handler().new_expression_evaluator( engine_commit_info_schema.clone(), Arc::new(wrapped_expr), @@ -129,9 +114,7 @@ impl Transaction { )?; evaluator.evaluate(engine_commit_info.as_ref()) } - None => { - kernel_commit_info.into_engine_data(get_log_commit_info_schema().clone(), engine) - } + None => kernel_commit_info.into_engine_data(LOG_COMMIT_INFO_SCHEMA.clone(), engine), } } } @@ -154,7 +137,8 @@ mod tests { use crate::engine::arrow_data::ArrowEngineData; use crate::schema::{Schema, SchemaRef, StructField, StructType, ToSchema}; use crate::transaction::Transaction; - use crate::utils::test_utils::load_test_table; + use crate::unit_test_utils::load_test_table; + use crate::utils::FoldWithOption as _; use crate::{DeltaResult, Engine, EngineData}; // ── build_commit_info tests ──────────────────────────────────────────────── @@ -248,17 +232,17 @@ mod tests { engine_commit_info: Option<(Box, SchemaRef)>, ) -> DeltaResult<(Arc, Transaction)> { let (engine, snapshot, _tempdir) = load_test_table("table-without-dv-small")?; - let mut txn = snapshot + let txn = snapshot .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_operation("WRITE".to_string()); - if let Some((engine_commit_info_data, engine_commit_info_schema)) = engine_commit_info { - txn = txn.with_commit_info(engine_commit_info_data, engine_commit_info_schema); - } + .with_operation("WRITE".to_string()) + .fold_with(engine_commit_info, |txn, (data, schema)| { + txn.with_commit_info(data, schema) + }); Ok((engine, txn)) } /// no engine_commit_info -- output is the kernel CommitInfo wrapped in a "commitInfo" - /// outer struct, matching the Delta log action format produced by `get_log_commit_info_schema`. + /// outer struct, matching the Delta log action format produced by `LOG_COMMIT_INFO_SCHEMA`. #[test] fn test_build_commit_info_none_branch() -> DeltaResult<()> { let (engine, txn) = make_txn(None)?; @@ -379,7 +363,8 @@ mod tests { } /// engine schema has partial overlap -- overlapping fields are replaced, engine-only - /// fields pass through, and remaining CommitInfo fields are appended after the last engine field. + /// fields pass through, and remaining CommitInfo fields are appended after the last engine + /// field. #[test] fn test_build_commit_info_partial_overlap() -> DeltaResult<()> { let (data, schema) = make_engine_commit_info( diff --git a/kernel/src/transaction/create_table.rs b/kernel/src/transaction/create_table.rs index c4844196b9..7dcb8933ef 100644 --- a/kernel/src/transaction/create_table.rs +++ b/kernel/src/transaction/create_table.rs @@ -15,7 +15,7 @@ //! # fn example(engine: &dyn Engine) -> delta_kernel::DeltaResult<()> { //! //! let schema = Arc::new(StructType::try_new(vec![ -//! StructField::new("id", DataType::INTEGER, false), +//! StructField::new("id", DataType::INTEGER, true), //! ])?); //! //! let result = create_table("/path/to/table", schema, "MyApp/1.0") @@ -32,19 +32,20 @@ #![allow(unreachable_pub, dead_code)] use std::marker::PhantomData; +use std::sync::Arc; +// Re-export the builder so callers can still access it from this module path. +pub use super::builder::create_table::CreateTableTransactionBuilder; use crate::actions::DomainMetadata; use crate::committer::Committer; use crate::expressions::ColumnName; +use crate::metrics::MetricId; use crate::schema::SchemaRef; -use crate::snapshot::SnapshotRef; +use crate::table_configuration::TableConfiguration; use crate::transaction::{CreateTable, Transaction}; use crate::utils::current_time_ms; use crate::DeltaResult; -// Re-export the builder so callers can still access it from this module path. -pub use super::builder::create_table::CreateTableTransactionBuilder; - /// A type alias for create-table transactions. /// /// This provides a restricted API surface that only exposes operations valid during table @@ -53,8 +54,8 @@ pub use super::builder::create_table::CreateTableTransactionBuilder; /// /// # Operations NOT available on create-table transactions /// -/// - **`with_domain_metadata_removed()`** — Cannot remove domain metadata from a table -/// that doesn't exist yet. +/// - **`with_domain_metadata_removed()`** — Cannot remove domain metadata from a table that doesn't +/// exist yet. /// - **`remove_files()`** — Cannot remove files from a table that has no files. /// - **`with_blind_append()`** — Blind append semantics don't apply to table creation. /// - **`update_deletion_vectors()`** — Deletion vectors require an existing table. @@ -72,7 +73,7 @@ pub use super::builder::create_table::CreateTableTransactionBuilder; /// # fn example(engine: &dyn Engine) -> delta_kernel::DeltaResult<()> { /// /// let schema = Arc::new(StructType::try_new(vec![ -/// StructField::new("id", DataType::INTEGER, false), +/// StructField::new("id", DataType::INTEGER, true), /// ])?); /// /// let result = create_table("/path/to/table", schema, "MyApp/1.0") @@ -101,12 +102,12 @@ pub type CreateTableTransaction = Transaction; /// use delta_kernel::transaction::create_table::create_table; /// use delta_kernel::schema::{DataType, StructField, StructType}; /// use delta_kernel::committer::FileSystemCommitter; -/// use delta_kernel::engine::default::DefaultEngineBuilder; -/// use delta_kernel::engine::default::storage::store_from_url; +/// use test_utils::delta_kernel_default_engine::DefaultEngineBuilder; +/// use test_utils::delta_kernel_default_engine::storage::store_from_url; /// /// # fn main() -> delta_kernel::DeltaResult<()> { /// let schema = Arc::new(StructType::new_unchecked(vec![ -/// StructField::new("id", DataType::INTEGER, false), +/// StructField::new("id", DataType::INTEGER, true), /// StructField::new("name", DataType::STRING, true), /// ])); /// @@ -133,31 +134,31 @@ impl CreateTableTransaction { /// Create a new transaction for creating a new table. This is used when the table doesn't /// exist yet and we need to create it with Protocol and Metadata actions. /// - /// The `pre_commit_snapshot` is a synthetic snapshot created from the protocol and metadata - /// that will be committed. It uses `PRE_COMMIT_VERSION` as a sentinel to indicate no - /// version exists yet on disk. + /// The `effective_table_config` is the table configuration that will be committed (protocol, + /// metadata, schema). /// /// This is typically called via `CreateTableTransactionBuilder::build()` rather than directly. pub(crate) fn try_new_create_table( - pre_commit_snapshot: SnapshotRef, + effective_table_config: TableConfiguration, engine_info: String, committer: Box, system_domain_metadata: Vec, clustering_columns: Option>, + correlation_id: Option>, ) -> DeltaResult { - // TODO(sanuj) Today transactions expect a read snapshot to be passed in and we pass - // in the pre_commit_snapshot for CREATE. To support other operations such as ALTERs - // there might be cleaner alternatives which can clearly disambiguate b/w a snapshot - // the was read vs the effective snapshot we will use for the commit. let span = tracing::info_span!( "txn", - path = %pre_commit_snapshot.table_root(), + path = %effective_table_config.table_root(), operation = "CREATE", ); - Ok(Transaction { span, - read_snapshot: pre_commit_snapshot, + operation_id: MetricId::new(), + correlation_id, + read_snapshot_opt: None, + effective_table_config, + should_emit_protocol: true, + should_emit_metadata: true, committer, operation: Some("CREATE TABLE".to_string()), engine_info: Some(engine_info), @@ -169,10 +170,12 @@ impl CreateTableTransaction { system_domain_metadata_additions: system_domain_metadata, user_domain_removals: vec![], data_change: true, + column_defaults_acknowledged: false, engine_commit_info: None, is_blind_append: false, dv_matched_files: vec![], - clustering_columns_physical: clustering_columns, + num_dv_updates: 0, + physical_clustering_columns: clustering_columns, _state: PhantomData, }) } diff --git a/kernel/src/transaction/data_layout.rs b/kernel/src/transaction/data_layout.rs index 859e28332a..67a13320d9 100644 --- a/kernel/src/transaction/data_layout.rs +++ b/kernel/src/transaction/data_layout.rs @@ -5,6 +5,7 @@ //! //! - **None**: No special organization (default) //! - **Clustered**: Data files optimized for queries on clustering columns +//! - **Partitioned**: Data files organized into directories by partition column values // Allow unreachable_pub because this module is pub when internal-api is enabled // but pub(crate) otherwise. The items need to be pub for the public API. @@ -19,8 +20,9 @@ use crate::expressions::ColumnName; /// /// - [`DataLayout::None`]: No special organization (default) /// - [`DataLayout::Clustered`]: Data files optimized for queries on clustering columns +/// - [`DataLayout::Partitioned`]: Data files organized into directories by partition column values /// -/// TODO(#1795): Add `Partitioned` variant for partition column support. +/// Partitioning and clustering are mutually exclusive -- only one variant can be active at a time. #[derive(Debug, Clone, Default)] pub enum DataLayout { /// No special data organization (default). @@ -34,6 +36,14 @@ pub enum DataLayout { /// Columns to cluster by (in order). columns: Vec, }, + + /// Data files organized into directories by partition column values. + /// Only top-level columns are supported. Partition column values are stored + /// in the directory path rather than in the data files themselves. + Partitioned { + /// Columns to partition by (in order). + columns: Vec, + }, } impl DataLayout { @@ -58,8 +68,9 @@ impl DataLayout { /// Nested columns (construct the variant directly): /// /// ```ignore + /// use delta_kernel::expressions::column_name; /// let layout = DataLayout::Clustered { - /// columns: vec![ColumnName::new(["user", "address", "city"])], + /// columns: vec![column_name!("user.address.city")], /// }; /// ``` pub fn clustered(columns: I) -> Self @@ -75,31 +86,83 @@ impl DataLayout { DataLayout::Clustered { columns } } + /// Create a partitioned layout with the given top-level column names. + /// + /// Each string is treated as a single top-level column name. Partition columns + /// must be top-level columns in the schema (nested columns are not supported). + /// + /// This method constructs the layout without validation. Full validation + /// (duplicates, schema compatibility, data types) is performed during + /// `CreateTableTransactionBuilder::build()` via `validate_partition_columns()`. + /// + /// # Example + /// + /// ```ignore + /// let layout = DataLayout::partitioned(["year", "month"]); + /// ``` + pub fn partitioned(columns: I) -> Self + where + I: IntoIterator, + S: AsRef, + { + let columns: Vec = columns + .into_iter() + .map(|s| ColumnName::new([s.as_ref()])) + .collect(); + + DataLayout::Partitioned { columns } + } + /// Returns true if this layout specifies clustering. + #[cfg(test)] pub fn is_clustered(&self) -> bool { matches!(self, DataLayout::Clustered { .. }) } + + /// Returns true if this layout specifies partitioning. + #[cfg(test)] + pub fn is_partitioned(&self) -> bool { + matches!(self, DataLayout::Partitioned { .. }) + } } #[cfg(test)] mod tests { use super::*; + #[rstest::rstest] + #[case::clustered(DataLayout::clustered(["id"]), true, false)] + #[case::partitioned(DataLayout::partitioned(["id"]), false, true)] + #[case::none(DataLayout::default(), false, false)] + fn test_data_layout_predicates( + #[case] layout: DataLayout, + #[case] expect_clustered: bool, + #[case] expect_partitioned: bool, + ) { + assert_eq!(layout.is_clustered(), expect_clustered); + assert_eq!(layout.is_partitioned(), expect_partitioned); + } + #[test] fn test_clustered_layout_construction() { let layout = DataLayout::clustered(["col1", "col2"]); - assert!(layout.is_clustered()); if let DataLayout::Clustered { columns } = layout { assert_eq!(columns.len(), 2); + } else { + panic!("Expected Clustered variant"); } } #[test] - fn test_default_layout() { - let layout = DataLayout::default(); - assert!(!layout.is_clustered()); + fn test_partitioned_layout_construction() { + let layout = DataLayout::partitioned(["year", "month"]); + if let DataLayout::Partitioned { columns } = layout { + assert_eq!(columns.len(), 2); + } else { + panic!("Expected Partitioned variant"); + } } - // Note: Validation tests (empty, too many, duplicates) are in clustering.rs - // since validate_clustering_columns() performs all validation at build time. + // Note: Validation tests (duplicates, schema compatibility, data types) are in + // clustering.rs and builder/create_table.rs since validation is performed at build time. } diff --git a/kernel/src/transaction/domain_metadata.rs b/kernel/src/transaction/domain_metadata.rs index 3d89cf269b..df23ce8508 100644 --- a/kernel/src/transaction/domain_metadata.rs +++ b/kernel/src/transaction/domain_metadata.rs @@ -1,13 +1,12 @@ use std::collections::HashSet; -use crate::actions::{get_log_domain_metadata_schema, DomainMetadata, INTERNAL_DOMAIN_PREFIX}; +use super::{EngineDataResultIterator, Transaction}; +use crate::actions::{DomainMetadata, INTERNAL_DOMAIN_PREFIX, LOG_DOMAIN_METADATA_SCHEMA}; use crate::error::Error; use crate::row_tracking::{RowTrackingDomainMetadata, ROW_TRACKING_DOMAIN_NAME}; use crate::table_features::TableFeature; use crate::{DeltaResult, Engine, IntoEngineData}; -use super::{EngineDataResultIterator, Transaction}; - impl Transaction { /// Validate domain metadata operations for both create-table and existing-table transactions. /// @@ -29,8 +28,7 @@ impl Transaction { } if !self - .read_snapshot - .table_configuration() + .effective_table_config .is_feature_supported(&TableFeature::DomainMetadata) { return Err(Error::unsupported( @@ -109,12 +107,13 @@ impl Transaction { Ok(()) } - /// Validate that a system domain corresponds to a known feature and that the feature is supported. + /// Validate that a system domain corresponds to a known feature and that the feature is + /// supported. /// /// This prevents arbitrary `delta.*` domains from being added during table creation. /// Each known system domain must have its corresponding feature enabled in the protocol. fn validate_system_domain_feature(&self, domain: &str) -> DeltaResult<()> { - let table_config = self.read_snapshot.table_configuration(); + let table_config = &self.effective_table_config; // Map domain to its required feature let required_feature = match domain { @@ -162,7 +161,7 @@ impl Transaction { .map(String::as_str) .collect(); let existing_domains = self - .read_snapshot + .read_snapshot()? .get_domain_metadatas_internal(engine, Some(&domains))?; // Create removal tombstones with pre-image configurations @@ -182,13 +181,13 @@ impl Transaction { /// /// Returns a tuple of `(action_iter, domain_metadata_vec)`. /// - The action iterator contains EngineData to be written to the commit file (`00N.json`). - /// - The `Vec` is used to construct a [`CrcDelta`](crate::crc::CrcDelta), - /// which feeds the post-commit snapshot with the domain metadata written in this transaction - /// and powers CRC file writes. + /// - The `Vec` is used to construct a [`CrcDelta`](crate::crc::CrcDelta), which + /// feeds the post-commit snapshot with the domain metadata written in this transaction and + /// powers CRC file writes. /// - /// This function may perform an expensive log replay operation if there are any domain removals. - /// The log replay is required to fetch the previous configuration value for the domain to preserve - /// in removal tombstones as mandated by the Delta spec. + /// This function may perform an expensive log replay operation if there are any domain + /// removals. The log replay is required to fetch the previous configuration value for the + /// domain to preserve in removal tombstones as mandated by the Delta spec. pub(super) fn generate_domain_metadata_actions<'a>( &'a self, engine: &'a dyn Engine, @@ -199,14 +198,7 @@ impl Transaction { // Validate domain operations (includes feature validation) self.validate_domain_metadata_operations()?; - // TODO(sanuj) Create-table must not have row tracking or removals - // Defensive. Needs to be updated when row tracking support is added. if is_create { - if row_tracking_high_watermark.is_some() { - return Err(Error::internal_error( - "CREATE TABLE cannot have row tracking domain metadata", - )); - } // user_domain_removals already validated above, but be explicit debug_assert!(self.user_domain_removals.is_empty()); } @@ -214,7 +206,7 @@ impl Transaction { // Generate removal actions (empty for create-table due to validation above) let removal_actions = self.generate_user_domain_removal_actions(engine)?; - // Generate row tracking domain action (None for create-table) + // Generate row tracking domain action. let row_tracking_domain_action = row_tracking_high_watermark .map(DomainMetadata::try_from) .transpose()? @@ -230,7 +222,7 @@ impl Transaction { .chain(removal_actions) .collect(); - let schema = get_log_domain_metadata_schema().clone(); + let schema = LOG_DOMAIN_METADATA_SCHEMA.clone(); let dm_actions_iter: Vec<_> = dm_actions_vec .iter() diff --git a/kernel/src/transaction/mod.rs b/kernel/src/transaction/mod.rs index b8b9958712..3786b8535b 100644 --- a/kernel/src/transaction/mod.rs +++ b/kernel/src/transaction/mod.rs @@ -1,40 +1,57 @@ -use std::collections::HashSet; +use std::collections::{HashMap, HashSet}; use std::iter; use std::marker::PhantomData; use std::ops::Deref; use std::sync::{Arc, LazyLock}; +use std::time::{Duration, Instant}; -use tracing::{info, instrument, warn}; -use url::Url; +use delta_kernel_derive::internal_api; +use tracing::instrument; -use crate::actions::deletion_vector::DeletionVectorPath; use crate::actions::{ - as_log_add_schema, get_commit_schema, get_log_remove_schema, get_log_txn_schema, CommitInfo, - DomainMetadata, SetTransaction, METADATA_NAME, PROTOCOL_NAME, + as_log_add_schema, CommitInfo, DomainMetadata, Metadata, Protocol, SetTransaction, + LOG_METADATA_SCHEMA, LOG_PROTOCOL_SCHEMA, LOG_REMOVE_SCHEMA, LOG_TXN_SCHEMA, MAX_VALUES, + MIN_VALUES, NULL_COUNT, NUM_RECORDS, TIGHT_BOUNDS, +}; +use crate::committer::{ + CommitMetadata, CommitProtocolMetadata, CommitResponse, CommitType, Committer, }; -use crate::committer::{CommitMetadata, CommitResponse, Committer}; -use crate::crc::{CrcDelta, FileStatsDelta}; +use crate::crc::{is_incremental_safe_operation, CrcDelta, FileStatsDelta}; use crate::engine_data::FilteredEngineData; use crate::error::Error; -use crate::expressions::ColumnName; -use crate::expressions::{ArrayData, Transform, UnaryExpressionOp::ToJson}; +use crate::expressions::UnaryExpressionOp::ToJson; +use crate::expressions::{ + col, column_name, lit, ArrayData, ColumnName, ExpressionStructPatch, + ExpressionStructPatchBuilder, Scalar, +}; +use crate::log_replay::HasSelectionVector; +use crate::log_segment::LogSegment; +use crate::metrics::events::TRANSACTION_COMMIT_SPAN; +use crate::metrics::{CommitFailureReason, MetricId}; +use crate::partition::serialization::serialize_partition_value; +use crate::partition::validation::validate_partition_values; use crate::path::{LogRoot, ParsedLogPath}; use crate::row_tracking::{RowTrackingDomainMetadata, RowTrackingVisitor}; use crate::scan::data_skipping::stats_schema::schema_with_all_fields_nullable; use crate::scan::log_replay::{ - BASE_ROW_ID_NAME, DEFAULT_ROW_COMMIT_VERSION_NAME, FILE_CONSTANT_VALUES_NAME, TAGS_NAME, + BASE_ROW_ID_NAME, DEFAULT_ROW_COMMIT_VERSION_NAME, FILE_CONSTANT_VALUES_NAME, + PARTITION_VALUES_NAME, PARTITION_VALUES_PARSED_NAME, SIZE_NAME, STATS_PARSED_NAME, TAGS_NAME, }; use crate::scan::scan_row_schema; -use crate::schema::{ArrayType, MapType, SchemaRef, StructField, StructType, StructTypeBuilder}; -use crate::snapshot::SnapshotRef; -use crate::table_features::{ColumnMappingMode, TableFeature}; +use crate::schema::void_utils::{add_void_stripping, validate_schema_for_write}; +use crate::schema::{ + lazy_schema_ref, ArrayType, ColumnDefault, SchemaRef, SchemaStructPatchBuilder, StructField, + StructType, +}; +use crate::snapshot::{Snapshot, SnapshotRef}; +use crate::struct_patch::ProjectionStructPatchBuilder; +use crate::table_configuration::TableConfiguration; +use crate::table_features::TableFeature; use crate::utils::require; -use crate::FileMeta; use crate::{ - DataType, DeltaResult, Engine, EngineData, Expression, ExpressionRef, IntoEngineData, - RowVisitor, Version, PRE_COMMIT_VERSION, + version_as_i64, DataType, DeltaResult, Engine, EngineData, Expression, FileMeta, + IntoEngineData, Predicate, RowVisitor, Version, }; -use delta_kernel_derive::internal_api; #[cfg(feature = "internal-api")] pub mod builder; @@ -51,33 +68,40 @@ pub mod data_layout; #[cfg(not(feature = "internal-api"))] pub(crate) mod data_layout; +pub(crate) mod alter_table; +pub use alter_table::AlterTableTransaction; mod commit_info; mod domain_metadata; +pub(crate) mod schema_evolution; +#[cfg(feature = "internal-api")] +pub mod stats_verifier; +#[cfg(not(feature = "internal-api"))] mod stats_verifier; mod update; -use stats_verifier::StatsVerifier; +mod write_context; +mod write_validation; + +use stats_verifier::StatsColumnVerifier; +use write_context::SharedWriteState; +pub use write_context::WriteContext; /// Type alias for an iterator of [`EngineData`] results. pub(crate) type EngineDataResultIterator<'a> = Box>> + Send + 'a>; -/// The static instance referenced by [`add_files_schema`] that doesn't contain the dataChange column. -pub(crate) static MANDATORY_ADD_FILE_SCHEMA: LazyLock = LazyLock::new(|| { - Arc::new(StructType::new_unchecked(vec![ - StructField::not_null("path", DataType::STRING), - StructField::not_null( - "partitionValues", - MapType::new(DataType::STRING, DataType::STRING, true), - ), - StructField::not_null("size", DataType::LONG), - StructField::not_null("modificationTime", DataType::LONG), - ])) -}); +/// The static instance referenced by [`add_files_schema`] that doesn't contain the dataChange +/// column. +pub(crate) static MANDATORY_ADD_FILE_SCHEMA: LazyLock = lazy_schema_ref! { + not_null "path": STRING, + not_null "partitionValues": { STRING => nullable STRING }, + not_null "size": LONG, + not_null "modificationTime": LONG, +}; /// Returns a reference to the mandatory fields in an add action. /// /// Note this does not include "dataChange" which is a required field but -/// but should be set on the transactoin level. Getting the full schema +/// should be set on the transaction level. Getting the full schema /// can be done with [`Transaction::add_files_schema`]. pub(crate) fn mandatory_add_file_schema() -> &'static SchemaRef { &MANDATORY_ADD_FILE_SCHEMA @@ -86,7 +110,7 @@ pub(crate) fn mandatory_add_file_schema() -> &'static SchemaRef { /// The base schema for add file metadata, referenced by [`Transaction::add_files_schema`]. /// /// The `stats` field represents the minimum structure. The actual stats written by -/// [`DefaultEngine::write_parquet`] include additional fields computed from the data: +/// `DefaultEngine::write_parquet` include additional fields computed from the data: /// - `nullCount`: nested struct mirroring the data schema (all fields LONG) /// - `minValues`: nested struct with min/max eligible column types /// - `maxValues`: nested struct with min/max eligible column types @@ -94,65 +118,34 @@ pub(crate) fn mandatory_add_file_schema() -> &'static SchemaRef { /// The nested structures within nullCount/minValues/maxValues depend on the table's data schema /// and which columns have statistics enabled. Use [`Transaction::stats_schema`] to get the /// expected stats schema for a specific table. -/// -/// [`DefaultEngine::write_parquet`]: crate::engine::default::DefaultEngine::write_parquet -pub(crate) static BASE_ADD_FILES_SCHEMA: LazyLock = LazyLock::new(|| { - let stats = StructField::nullable( - "stats", - DataType::struct_type_unchecked(vec![ - StructField::nullable("numRecords", DataType::LONG), - // nullCount, minValues, maxValues are dynamic based on data schema. - // Empty struct placeholders indicate these fields exist but their inner - // structure depends on the table schema and stats column configuration. - StructField::nullable("nullCount", DataType::struct_type_unchecked(vec![])), - StructField::nullable("minValues", DataType::struct_type_unchecked(vec![])), - StructField::nullable("maxValues", DataType::struct_type_unchecked(vec![])), - StructField::nullable("tightBounds", DataType::BOOLEAN), - ]), - ); - - StructTypeBuilder::from_schema(mandatory_add_file_schema()) - .add_field(stats) - .build_arc_unchecked() -}); +pub(crate) static BASE_ADD_FILES_SCHEMA: LazyLock = lazy_schema_ref! { + ..(mandatory_add_file_schema().fields().cloned()), + nullable "stats": { + nullable NUM_RECORDS: LONG, + // nullCount, minValues, maxValues are dynamic based on data schema. Empty struct + // placeholders indicate these fields exist but their inner structure depends on the + // table schema and stats column configuration. + nullable NULL_COUNT: {}, + nullable MIN_VALUES: {}, + nullable MAX_VALUES: {}, + nullable TIGHT_BOUNDS: BOOLEAN, + }, +}; static DATA_CHANGE_COLUMN: LazyLock = LazyLock::new(|| StructField::not_null("dataChange", DataType::BOOLEAN)); -/// The static instance referenced by [`add_files_schema`] that contains the dataChange column. -static ADD_FILES_SCHEMA_WITH_DATA_CHANGE: LazyLock = LazyLock::new(|| { - let mut fields = BASE_ADD_FILES_SCHEMA.fields().collect::>(); - let len = fields.len(); - let insert_position = fields - .iter() - .position(|f| f.name() == "modificationTime") - .unwrap_or(len); - fields.insert(insert_position + 1, &DATA_CHANGE_COLUMN); - Arc::new(StructType::new_unchecked(fields.into_iter().cloned())) -}); - -/// Extend a schema with a statistics column and return a new SchemaRef. -/// -/// The stats column is of type string as required by the spec. -/// -/// Note that this method is only useful to extend an Add action schema. -fn with_stats_col(schema: &SchemaRef) -> SchemaRef { - StructTypeBuilder::from_schema(schema) - .add_field(StructField::nullable("stats", DataType::STRING)) - .build_arc_unchecked() -} - /// Extend a schema with row tracking columns and return a new SchemaRef. /// /// Note that this method is only useful to extend an Add action schema. -fn with_row_tracking_cols(schema: &SchemaRef) -> SchemaRef { - StructTypeBuilder::from_schema(schema) - .add_field(StructField::nullable("baseRowId", DataType::LONG)) - .add_field(StructField::nullable( +fn with_row_tracking_cols(schema: &SchemaRef) -> DeltaResult { + let patch = SchemaStructPatchBuilder::new() + .append(StructField::nullable("baseRowId", DataType::LONG)) + .append(StructField::nullable( "defaultRowCommitVersion", DataType::LONG, - )) - .build_arc_unchecked() + )); + Ok(Arc::new(patch.build(schema)?)) } /// Marker type for transactions on existing tables. @@ -169,6 +162,22 @@ pub struct ExistingTable; #[derive(Debug)] pub struct CreateTable; +/// Marker type for alter-table (schema evolution) transactions. +/// +/// Transactions in this state perform metadata-only commits. Data file operations are not +/// available at compile time because `AlterTable` does not implement [`SupportsDataFiles`]. +#[derive(Debug)] +pub struct AlterTable; + +/// Marker trait for transaction states that support data file operations. +/// +/// Only transaction types that implement this trait can access methods for adding, removing, or +/// updating data files. This prevents compile-time misuse by states like `AlterTable` that +/// only perform metadata-only commits. +pub trait SupportsDataFiles {} +impl SupportsDataFiles for ExistingTable {} +impl SupportsDataFiles for CreateTable {} + /// A transaction represents an in-progress write to a table. After creating a transaction, changes /// to the table may be staged via the transaction methods before calling `commit` to commit the /// changes to the table. @@ -190,9 +199,22 @@ pub struct CreateTable; /// ``` pub struct Transaction { span: tracing::Span, - // The snapshot this transaction is based on. For create-table transactions, - // this is a pre-commit snapshot with PRE_COMMIT_VERSION. - read_snapshot: SnapshotRef, + // Correlates all metric events emitted by this transaction. + operation_id: MetricId, + // Opaque, caller-supplied id recorded on this transaction's commit metric events alongside + // `operation_id`. Set via `with_correlation_id`; not interpreted by kernel. + correlation_id: Option>, + // The snapshot this transaction is based on. None for CREATE TABLE (no pre-existing table). + // Use `read_snapshot()` to access; it returns an error if None. + read_snapshot_opt: Option, + // The table configuration that this commit will produce. For writes that don't change the + // config, this is cloned from the read snapshot; when the config changes (e.g. schema + // evolution), it is constructed separately with the new schema/protocol. + effective_table_config: TableConfiguration, + // Whether to emit a Protocol action. True for CREATE TABLE and ALTER TABLE, false otherwise. + should_emit_protocol: bool, + // Whether to emit a Metadata action. True for CREATE TABLE and ALTER TABLE, false otherwise. + should_emit_metadata: bool, committer: Box, operation: Option, engine_info: Option, @@ -202,8 +224,8 @@ pub struct Transaction { // NB: hashmap would require either duplicating the appid or splitting SetTransaction // key/payload. HashSet requires Borrow<&str> with matching Eq, Ord, and Hash. Plus, // HashSet::insert drops the to-be-inserted value without returning the existing one, which - // would make error messaging unnecessarily difficult. Thus, we keep Vec here and deduplicate in - // the commit method. + // would make error messaging unnecessarily difficult. Thus, we keep Vec here and deduplicate + // in the commit method. set_transactions: Vec, // commit-wide timestamp (in milliseconds since epoch) - used in ICT, `txn` action, etc. to // keep all timestamps within the same commit consistent. @@ -220,15 +242,21 @@ pub struct Transaction { user_domain_removals: Vec, // Whether this transaction contains any logical data changes. data_change: bool, + // TODO(#2499): Replace this state when Conntector responsibilities encode column-default + // handling. Whether the connector acknowledged responsibility for applying column + // defaults. + column_defaults_acknowledged: bool, // Whether this transaction should be marked as a blind append. is_blind_append: bool, // Files matched by update_deletion_vectors() with new DV descriptors appended. These are used // to generate remove/add action pairs during commit, ensuring file statistics are preserved. dv_matched_files: Vec, + // Count of files whose deletion vector was updated. + num_dv_updates: usize, // Clustering columns from domain metadata. Only populated if the ClusteredTable feature is // enabled. Used for determining which columns require statistics collection. Expected to be // physical column names. - clustering_columns_physical: Option>, + physical_clustering_columns: Option>, // PhantomData marker for transaction state (ExistingTable or CreateTable). // Zero-sized; only affects the type system. _state: PhantomData, @@ -236,10 +264,9 @@ pub struct Transaction { impl std::fmt::Debug for Transaction { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - let version_info = if self.is_create_table() { - "create_table".to_string() - } else { - format!("{}", self.read_snapshot.version()) + let version_info = match &self.read_snapshot_opt { + Some(snap) => format!("{}", snap.version()), + None => "create_table".to_string(), }; f.write_str(&format!( "Transaction {{ read_snapshot version: {}, engine_info: {} }}", @@ -249,6 +276,52 @@ impl std::fmt::Debug for Transaction { } } +/// Builds the projection for converting add file metadata into commit-ready Add actions. +fn build_add_action_projection( + input_schema: &StructType, + data_change: bool, +) -> DeltaResult<(SchemaRef, Expression)> { + let (output_schema, patch) = ProjectionStructPatchBuilder::new(input_schema) + .insert_after( + "modificationTime", + DATA_CHANGE_COLUMN.clone(), + lit(data_change), + ) + .replace( + "stats", + StructField::nullable("stats", DataType::STRING), + Expression::unary(ToJson, col!("stats")), + ) + .build()?; + let patch = Expression::struct_from([patch]); + Ok((output_schema, patch)) +} + +/// Transforms add file metadata into commit-ready add actions by converting stats to JSON and +/// setting the `dataChange` field. +fn build_add_actions<'a, I, T>( + engine: &dyn Engine, + add_files_metadata: I, + input_schema: SchemaRef, + data_change: bool, +) -> DeltaResult>> + 'a> +where + I: Iterator> + Send + 'a, + T: Deref + Send + 'a, +{ + let evaluation_handler = engine.evaluation_handler(); + let (output_schema, adds_expr) = build_add_action_projection(&input_schema, data_change)?; + let adds_expr = Arc::new(adds_expr); + Ok(add_files_metadata.map(move |add_files_batch| { + let adds_evaluator = evaluation_handler.new_expression_evaluator( + input_schema.clone(), + adds_expr.clone(), + as_log_add_schema(output_schema.clone()).into(), + )?; + adds_evaluator.evaluate(add_files_batch?.deref()) + })) +} + // ============================================================================= // Shared methods available on ALL transaction types // ============================================================================= @@ -260,19 +333,37 @@ impl Transaction { /// - Err(Error) indicates a non-retryable error (e.g. logic/validation error). #[instrument( parent = &self.span, - name = "txn.commit", + name = TRANSACTION_COMMIT_SPAN, skip_all, fields( + report, + operation_id = %self.operation_id, + is_catalog_managed = self.effective_table_config.is_catalog_managed(), + correlation_id = self.correlation_id.as_deref().unwrap_or(""), commit_version = self.get_commit_version(), + num_add_files, + num_remove_files, + num_dv_updates, + add_files_bytes, + remove_files_bytes, + is_blind_append, + data_change, + operation, + prepare_duration_ns, + committer_duration_ns, + failure_reason, ), err )] pub fn commit(self, engine: &dyn Engine) -> DeltaResult> { - info!( - num_add_files = self.add_files_metadata.len(), - num_remove_files = self.remove_files_metadata.len(), - num_dv_updates = self.dv_matched_files.len(), - ); + let commit_start = Instant::now(); + + // Some table features don't yet support removeFiles. Reject here. + if !self.remove_files_metadata.is_empty() { + self.effective_table_config + .validate_feature_support_for_remove()?; + } + // Step 1: Check for duplicate app_ids and generate set transactions (`txn`) // Note: The commit info must always be the first action in the commit but we generate it in // step 2 to fail early on duplicate transaction appIds @@ -291,6 +382,14 @@ impl Transaction { } self.validate_blind_append_semantics()?; + self.validate_append_only_semantics()?; + self.ensure_schema_non_empty_for_data_writes()?; + + // Validate that the schema supports data writes when files are being added. Reads and + // metadata-only commits are always allowed. + if !self.add_files_metadata.is_empty() { + validate_schema_for_write(&self.effective_table_config.logical_schema())?; + } // CDF check only applies to existing tables (not create table) // If there are add and remove files with data change in the same transaction, we block it. @@ -303,8 +402,7 @@ impl Transaction { && self.data_change { let cdf_enabled = self - .read_snapshot - .table_configuration() + .effective_table_config .table_properties() .enable_change_data_feed .unwrap_or(false); @@ -318,15 +416,30 @@ impl Transaction { ); } - // Validate clustering column stats if ClusteredTable feature is enabled + // Validate protocol-required add-file statistics. + // Note: Stats validation cannot use `StagedDataValidator` because its columns and types + // are determined at runtime, whereas `RowVisitor::selected_column_names_and_types` must + // return a static projection. Consequently, stats validation makes a separate pass for + // each stats column. self.validate_add_files_stats(&self.add_files_metadata)?; + // Validate required fields for addFile. + write_validation::StagedDataValidator::staged_add_file( + self.effective_table_config.physical_partition_columns(), + ) + .validate(&self.add_files_metadata)?; + + write_validation::StagedDataValidator::staged_dv_matched_file( + self.effective_table_config.physical_partition_columns(), + )? + .validate_filtered(&self.dv_matched_files)?; + // Step 1: Generate SetTransaction actions let set_transaction_actions = self .set_transactions .clone() .into_iter() - .map(|txn| txn.into_engine_data(get_log_txn_schema().clone(), engine)); + .map(|txn| txn.into_engine_data(LOG_TXN_SCHEMA.clone(), engine)); // Step 2: Construct commit info with ICT if enabled let in_commit_timestamp = self.get_in_commit_timestamp(engine)?; @@ -339,24 +452,26 @@ impl Transaction { ); let commit_info_action = self.generate_commit_info(engine, kernel_commit_info); - // Step 3: Generate Protocol and Metadata actions for create-table - let (protocol_action, metadata_action) = if self.is_create_table() { - let table_config = self.read_snapshot.table_configuration(); - let protocol = table_config.protocol().clone(); - let metadata = table_config.metadata().clone(); - - let protocol_schema = get_commit_schema().project(&[PROTOCOL_NAME])?; - let metadata_schema = get_commit_schema().project(&[METADATA_NAME])?; - - let protocol_data = protocol.into_engine_data(protocol_schema, engine)?; - let metadata_data = metadata.into_engine_data(metadata_schema, engine)?; - - (Some(protocol_data), Some(metadata_data)) + // Step 3: Generate Protocol and Metadata actions based on emit flags + let (protocol_action, protocol) = if self.should_emit_protocol { + let protocol = self.effective_table_config.protocol().clone(); + let schema = LOG_PROTOCOL_SCHEMA.clone(); + let action = protocol.clone().into_engine_data(schema, engine)?; + (Some(action), Some(protocol)) + } else { + (None, None) + }; + let (metadata_action, metadata) = if self.should_emit_metadata { + let metadata = self.effective_table_config.metadata().clone(); + let schema = LOG_METADATA_SCHEMA.clone(); + let action = metadata.clone().into_engine_data(schema, engine)?; + (Some(action), Some(metadata)) } else { (None, None) }; - // Step 4: Generate add actions and get data for domain metadata actions (e.g. row tracking high watermark) + // Step 4: Generate add actions and get data for domain metadata actions (e.g. row tracking + // high watermark) let commit_version = self.get_commit_version(); let (add_actions, row_tracking_domain_metadata) = self.generate_adds(engine, commit_version)?; @@ -373,8 +488,9 @@ impl Transaction { self.generate_remove_actions(engine, self.remove_files_metadata.iter(), &[])?; // Build the action chain - // For create-table: CommitInfo -> Protocol -> Metadata -> adds -> txns -> domain_metadata -> removes - // For existing table: CommitInfo -> adds -> txns -> domain_metadata -> removes + // For create-table: CommitInfo -> Protocol -> Metadata -> adds -> txns -> domain_metadata + // -> removes For existing table: CommitInfo -> adds -> txns -> domain_metadata -> + // removes let actions = iter::once(commit_info_action) .chain(protocol_action.map(Ok)) .chain(metadata_action.map(Ok)) @@ -388,51 +504,89 @@ impl Transaction { .chain(dv_update_actions); // Step 7: Commit via the committer - // Block FileSystemCommitter for catalog-managed tables (including create-table with catalog features) - #[cfg(feature = "catalog-managed")] - if !self.committer.is_catalog_committer() - && self - .read_snapshot - .table_configuration() - .is_catalog_managed() - { - return Err(Error::generic( - "A catalog committer must be used to commit to catalog-managed tables. Please \ - provide a catalog committer via Snapshot::transaction().", - )); - } - let log_root = LogRoot::new(self.read_snapshot.table_root().clone())?; - let commit_metadata = CommitMetadata::new( - log_root, + let commit_metadata = self.create_commit_metadata( commit_version, - self.commit_timestamp, - self.read_snapshot - .log_segment() - .listed - .max_published_version, - ); - match self - .committer - .commit(engine, Box::new(filtered_actions), commit_metadata) - { + in_commit_timestamp, + protocol, + metadata, + dm_changes.clone(), + )?; + let prepare_duration = commit_start.elapsed(); + let committer_start = Instant::now(); + let commit_response = + self.committer + .commit(engine, Box::new(filtered_actions), commit_metadata); + let committer_duration = committer_start.elapsed(); + match commit_response { Ok(CommitResponse::Committed { file_meta }) => { - let crc_delta = self.build_crc_delta(in_commit_timestamp, dm_changes)?; + // TODO(#2717): the commit already succeeded atomically; the post-commit `?` + // below must not fail the txn (and must not mislabel the metric). + let bin_boundaries = self + .read_snapshot_opt + .as_ref() + .and_then(|snap| snap.get_file_stats_if_present()) + .and_then(|s| s.file_size_histogram) + .map(|h| h.sorted_bin_boundaries); + let file_stats = FileStatsDelta::try_compute_for_txn( + &self.add_files_metadata, + &self.remove_files_metadata, + bin_boundaries.as_deref(), + )?; + self.record_commit_success_metrics( + &file_stats, + prepare_duration, + committer_duration, + ); + let crc_delta = + self.build_crc_delta(file_stats, in_commit_timestamp, dm_changes)?; Ok(CommitResult::CommittedTransaction( self.into_committed(file_meta, crc_delta)?, )) } - Ok(CommitResponse::Conflict { version }) => Ok(CommitResult::ConflictedTransaction( - self.into_conflicted(version), - )), + Ok(CommitResponse::Conflict { version }) => { + // Flips the metric event from success -> failure. + tracing::Span::current() + .record("failure_reason", CommitFailureReason::Conflict.as_ref()); + Ok(CommitResult::ConflictedTransaction( + self.into_conflicted(version), + )) + } // TODO: we may want to be more or less selective about what is retryable (this is tied // to the idea of "what kind of Errors should write_json_file return?") Err(e @ Error::IOError(_)) => { + // Flips the metric event from success -> failure. + tracing::Span::current() + .record("failure_reason", CommitFailureReason::RetryableIo.as_ref()); Ok(CommitResult::RetryableTransaction(self.into_retryable(e))) } Err(e) => Err(e), } } + fn record_commit_success_metrics( + &self, + file_stats: &FileStatsDelta, + prepare_duration: Duration, + committer_duration: Duration, + ) { + let span = tracing::Span::current(); + span.record("num_add_files", file_stats.gross_add_files); + span.record("num_remove_files", file_stats.gross_remove_files); + span.record("num_dv_updates", self.num_dv_updates as u64); + span.record("add_files_bytes", file_stats.gross_add_bytes); + span.record("remove_files_bytes", file_stats.gross_remove_bytes); + span.record("is_blind_append", self.is_blind_append); + span.record("data_change", self.data_change); + if let Some(operation) = self.operation.as_deref() { + span.record("operation", operation); + } + span.record("prepare_duration_ns", prepare_duration.as_nanos() as u64); + span.record( + "committer_duration_ns", + committer_duration.as_nanos() as u64, + ); + } + /// Set the data change flag. /// /// True indicates this commit is a "data changing" commit. False indicates table data was @@ -440,8 +594,9 @@ impl Transaction { /// /// Data change might be set to false in the following scenarios: /// 1. Operations that only change metadata (e.g. backfilling statistics) - /// 2. Operations that make no logical changes to the contents of the table (i.e. rows are only moved - /// from old files to new ones. OPTIMIZE commands is one example of this type of optimizaton). + /// 2. Operations that make no logical changes to the contents of the table (i.e. rows are only + /// moved from old files to new ones. OPTIMIZE commands is one example of this type of + /// optimizaton). pub fn with_data_change(mut self, data_change: bool) -> Self { self.data_change = data_change; self @@ -461,9 +616,18 @@ impl Transaction { self } - /// Set the content of the commitInfo action for this transaction. Note that kernel will _always_ write a commitInfo, - /// this function simply allows engines to add their own data into that action if they wish. - /// Note that the following fields in `engine_commit_info` will be overridden by kernel if they are set (meaning you should not set them): + /// Attach an opaque, caller-supplied correlation id for joining this transaction's commit + /// metric events to the caller's own request or operation id. An empty id is treated as unset. + /// When unset, behavior is unchanged. + pub fn with_correlation_id(mut self, correlation_id: impl Into>) -> Self { + self.correlation_id = Some(correlation_id.into()).filter(|id| !id.is_empty()); + self + } + + /// Set the content of the commitInfo action for this transaction. Note that kernel will + /// _always_ write a commitInfo, this function simply allows engines to add their own data + /// into that action if they wish. Note that the following fields in `engine_commit_info` + /// will be overridden by kernel if they are set (meaning you should not set them): /// - timestamp /// - inCommitTimestamp /// - operation @@ -504,6 +668,91 @@ impl Transaction { self } + /// Determines the commit type based on whether this is a create-table operation and whether + /// the table is catalog-managed. + fn determine_commit_type( + is_create: bool, + table_config: &crate::table_configuration::TableConfiguration, + ) -> CommitType { + let is_catalog_managed = table_config.is_catalog_managed(); + + // TODO: Handle UpgradeToCatalogManaged and DowngradeToPathBased when ALTER TABLE + // SET TBLPROPERTIES is supported. + match (is_create, is_catalog_managed) { + (true, true) => CommitType::CatalogManagedCreate, + (true, false) => CommitType::PathBasedCreate, + (false, true) => CommitType::CatalogManagedWrite, + (false, false) => CommitType::PathBasedWrite, + } + } + + /// Validates that the committer type matches the commit type. A catalog committer must be + /// used for catalog-managed operations, and a non-catalog committer for path-based operations. + fn validate_commit_type( + is_catalog_committer: bool, + commit_type: &CommitType, + ) -> DeltaResult<()> { + match ( + is_catalog_committer, + commit_type.requires_catalog_committer(), + ) { + (true, true) | (false, false) => Ok(()), + (false, true) => Err(Error::generic( + "This table is catalog-managed and requires a catalog committer. \ + Please provide a catalog committer via Snapshot::transaction().", + )), + (true, false) => Err(Error::generic( + "This table is path-based and cannot be committed to with a catalog committer.", + )), + } + } + + /// Builds the [`CommitMetadata`] for this transaction. Determines the commit type, + /// validates the committer, and assembles the protocol/metadata state. + fn create_commit_metadata( + &self, + commit_version: Version, + in_commit_timestamp: Option, + new_protocol: Option, + new_metadata: Option, + domain_metadata_changes: Vec, + ) -> DeltaResult { + let log_root = LogRoot::new(self.effective_table_config.table_root().clone())?; + let is_create = self.is_create_table(); + let commit_type = Self::determine_commit_type(is_create, &self.effective_table_config); + Self::validate_commit_type(self.committer.is_catalog_committer(), &commit_type)?; + // For create-table: previous P&M is None (no prior table), new P&M is set. + // For existing table with metadata change: previous P&M is from snapshot, new P&M + // is from effective config. + // For existing table without metadata change: previous P&M is from snapshot, new is None. + let (read_protocol, read_metadata, max_published_version) = if is_create { + (None, None, None) + } else { + let snap = self.read_snapshot()?; + let read_config = snap.table_configuration(); + ( + Some(read_config.protocol().clone()), + Some(read_config.metadata().clone()), + snap.log_segment().listed.max_published_version, + ) + }; + let protocol_metadata = CommitProtocolMetadata::try_new( + read_protocol, + read_metadata, + new_protocol, + new_metadata, + )?; + Ok(CommitMetadata::new( + log_root, + commit_version, + commit_type, + in_commit_timestamp.unwrap_or(self.commit_timestamp), + max_published_version, + protocol_metadata, + domain_metadata_changes, + )) + } + /// Validate that the transaction is eligible to be marked as a blind append. /// /// Note: Domain metadata additions/removals are allowed; blind append only constrains @@ -539,16 +788,98 @@ impl Transaction { Ok(()) } + // Reject data-file removals / DV updates on appendOnly tables when `data_change` is true. + fn validate_append_only_semantics(&self) -> DeltaResult<()> { + if !self.data_change + || !self + .effective_table_config + .is_feature_enabled(&TableFeature::AppendOnly) + { + return Ok(()); + } + + let removes_data = self + .remove_files_metadata + .iter() + .chain(&self.dv_matched_files) + .any(HasSelectionVector::has_selected_rows); + require!( + !removes_data, + Error::invalid_transaction_state( + "Append-only tables cannot remove files or update deletion vectors when data_change is true", + ) + ); + Ok(()) + } + + /// Reject data file writes (add/remove/DV) against an empty-schema table. + /// CREATE TABLE and metadata-only commits are exempt. + fn ensure_schema_non_empty_for_data_writes(&self) -> DeltaResult<()> { + if self.is_create_table() { + return Ok(()); + } + if self.has_data_file_actions() { + self.ensure_schema_non_empty_for_write_context()?; + } + Ok(()) + } + + /// Reject `WriteContext` handouts on empty-schema tables, so engines fail + /// before staging any parquet. CREATE TABLE is exempt. + fn ensure_schema_non_empty_for_write_context(&self) -> DeltaResult<()> { + if self.is_create_table() { + return Ok(()); + } + if self.effective_table_config.logical_schema().num_fields() == 0 { + return Err(Error::generic( + "Cannot write data files to a Delta table with empty schema; \ + use `snapshot.alter_table().add_column(...)` to add at least one \ + column before writing data", + )); + } + Ok(()) + } + + /// Rejects write-context creation when a table declares column defaults and the connector has + /// not acknowledged handling them. + fn ensure_column_defaults_acknowledged(&self) -> DeltaResult<()> { + require!( + self.column_defaults_acknowledged + || !self + .effective_table_config + .is_feature_enabled(&TableFeature::AllowColumnDefaults) + || !self.effective_table_config.has_column_with_default(), + Error::invalid_transaction_state( + "Writing data to a table with column defaults requires calling \ + Transaction::ack_column_defaults() first", + ) + ); + Ok(()) + } + /// Returns true if this is a create-table transaction. - /// A create-table transaction has operation "CREATE TABLE" and a pre-commit snapshot - /// with PRE_COMMIT_VERSION. + /// A create-table transaction has no read snapshot (no pre-existing table). fn is_create_table(&self) -> bool { - let is_create = self.operation.as_deref() == Some("CREATE TABLE"); debug_assert!( - !is_create || self.read_snapshot.version() == PRE_COMMIT_VERSION, - "CREATE TABLE transaction must have PRE_COMMIT_VERSION snapshot" + self.operation.as_deref() != Some("CREATE TABLE") || self.read_snapshot_opt.is_none(), + "CREATE TABLE operation should not have a read snapshot" ); - is_create + self.read_snapshot_opt.is_none() + } + + /// True iff this transaction stages any data-file action (add, remove, or DV update). + fn has_data_file_actions(&self) -> bool { + !self.add_files_metadata.is_empty() + || !self.remove_files_metadata.is_empty() + || !self.dv_matched_files.is_empty() + } + + // Returns the read snapshot. Returns an error if this is a create-table transaction. + // To get the `Option` directly, use the `read_snapshot_opt` field. + fn read_snapshot(&self) -> DeltaResult<&Snapshot> { + self.read_snapshot_opt.as_deref().ok_or_else(|| { + Error::internal_error("read_snapshot() called on create-table transaction") + }) } /// Computes the in-commit timestamp for this transaction if ICT is enabled. @@ -557,8 +888,7 @@ impl Transaction { /// property must also be `true` (`is_feature_enabled`). fn get_in_commit_timestamp(&self, engine: &dyn Engine) -> DeltaResult> { let has_ict = self - .read_snapshot - .table_configuration() + .effective_table_config .is_feature_enabled(&TableFeature::InCommitTimestamp); if !has_ict { @@ -575,29 +905,31 @@ impl Transaction { // - The time at which the writer attempted the commit // - One millisecond later than the previous commit's inCommitTimestamp Ok(self - .read_snapshot + .read_snapshot()? .get_in_commit_timestamp(engine)? .map(|prev_ict| self.commit_timestamp.max(prev_ict + 1))) } /// Returns the commit version for this transaction. /// For existing table transactions, this is snapshot.version() + 1. - /// For create-table transactions (PRE_COMMIT_VERSION + 1 wraps to 0), this is 0. + /// For create-table transactions, this is 0. fn get_commit_version(&self) -> Version { - // PRE_COMMIT_VERSION (u64::MAX) + 1 wraps to 0, which is the correct first version - self.read_snapshot.version().wrapping_add(1) + match &self.read_snapshot_opt { + Some(snap) => snap.version() + 1, + None => 0, + } } - /// The schema that the [`Engine`]'s [`ParquetHandler`] is expected to use when reporting information about - /// a Parquet write operation back to Kernel. + /// The schema that the [`Engine`]'s [`ParquetHandler`] is expected to use when reporting + /// information about a Parquet write operation back to Kernel. /// - /// Concretely, it is the expected schema for [`EngineData`] passed to [`add_files`], as it is the base - /// for constructing an add_file. Each row represents metadata about a - /// file to be added to the table. Kernel takes this information and extends it to the full add_file - /// action schema, adding internal fields (e.g., baseRowID) as necessary. + /// Concretely, it is the expected schema for [`EngineData`] passed to [`add_files`], as it is + /// the base for constructing an add_file. Each row represents metadata about a + /// file to be added to the table. Kernel takes this information and extends it to the full + /// add_file action schema, adding internal fields (e.g., baseRowID) as necessary. /// /// The `stats` field contains file-level statistics. The schema returned here shows the base - /// structure; the actual stats written by [`DefaultEngine::write_parquet`] include dynamically + /// structure; the actual stats written by `DefaultEngine::write_parquet` include dynamically /// computed fields (numRecords, nullCount, minValues, maxValues, tightBounds) based on the /// data schema and table configuration. See [`stats_schema`] for the table-specific expected /// stats schema. @@ -607,11 +939,27 @@ impl Transaction { /// /// [`add_files`]: crate::transaction::Transaction::add_files /// [`ParquetHandler`]: crate::ParquetHandler - /// [`DefaultEngine::write_parquet`]: crate::engine::default::DefaultEngine::write_parquet /// [`stats_schema`]: Transaction::stats_schema pub fn add_files_schema(&self) -> &'static SchemaRef { &BASE_ADD_FILES_SCHEMA } +} + +// ============================================================================= +// Data file methods -- only available on transaction types that support data files +// ============================================================================= +impl Transaction { + // TODO(#2499): Remove this API when Engine responsibilities encode column-default handling. + /// Acknowledges that the connector applies column defaults before writing data files. + /// + /// Call this before requesting a write context for a table that enables the + /// `allowColumnDefaults` feature and declares at least one column default. The connector must + /// materialize every omitted column's default itself; this method records that responsibility + /// but does not apply any defaults. Without this acknowledgement, write-context creation fails + /// with an error. + pub fn ack_column_defaults(&mut self) { + self.column_defaults_acknowledged = true; + } /// Returns the expected schema for file statistics. /// @@ -638,16 +986,16 @@ impl Transaction { /// settings. #[allow(unused)] pub fn stats_schema(&self) -> DeltaResult { - let tc = self.read_snapshot.table_configuration(); - let stats_schemas = - tc.build_expected_stats_schemas(self.clustering_columns_physical.as_deref(), None)?; + let stats_schemas = self + .effective_table_config + .build_expected_stats_schemas(self.physical_clustering_columns.as_deref(), None)?; Ok(stats_schemas.physical) } /// Returns the list of column names that should have statistics collected. /// /// This returns leaf column paths as [`ColumnName`] objects. Each `ColumnName` - /// stores path components separately (e.g., `ColumnName::new(["nested", "field"])`). + /// stores path components separately (e.g., `column_name!("nested.field")`). /// See [`ColumnName`'s `Display` implementation][ColumnName#impl-Display-for-ColumnName] /// for details on string formatting and escaping. /// @@ -657,88 +1005,245 @@ impl Transaction { /// regardless of `dataSkippingStatsColumns` or `dataSkippingNumIndexedCols` settings. #[allow(unused)] pub fn stats_columns(&self) -> Vec { - self.read_snapshot - .table_configuration() - .stats_column_names_physical(self.clustering_columns_physical.as_deref()) - } - - // Generate the logical-to-physical transform expression which must be evaluated on every data - // chunk before writing. At the moment, this is a transaction-wide expression. - fn generate_logical_to_physical(&self) -> Expression { - let partition_cols = self - .read_snapshot - .table_configuration() - .partition_columns() - .to_vec(); - // Check if materializePartitionColumns feature is enabled - let materialize_partition_columns = self - .read_snapshot - .table_configuration() - .is_feature_enabled(&TableFeature::MaterializePartitionColumns); - // Build a Transform expression that drops partition columns from the input - // (unless materializePartitionColumns is enabled). - let mut transform = Transform::new_top_level(); - if !materialize_partition_columns { - for col in &partition_cols { - transform = transform.with_dropped_field_if_exists(col); + self.effective_table_config + .physical_stats_column_names(self.physical_clustering_columns.as_deref()) + } + + // Generate the logical-to-physical expression which must be evaluated on every data chunk + // before writing. + fn generate_logical_to_physical( + &self, + partition_values: Option<&HashMap>, + ) -> DeltaResult { + let logical_schema = self.effective_table_config.logical_schema(); + let mut patch = ExpressionStructPatchBuilder::new(); + if self + .effective_table_config + .should_materialize_partition_columns() + { + let partition_cols: HashSet<&str> = self + .effective_table_config + .logical_partition_columns() + .iter() + .map(String::as_str) + .collect(); + // Insert each partition column after the nearest preceding surviving field + // (non-partition and non-void), in the order they appear in the logical schema. + // This keeps the post-transform data aligned with the physical schema. + let mut predecessor: Option<&str> = None; + for field in logical_schema.fields() { + let name = field.name().as_str(); + if partition_cols.contains(name) { + let value = partition_values.and_then(|m| m.get(name)).ok_or_else(|| { + Error::internal_error(format!( + "partition column '{name}' missing while building logical-to-physical \ + expression" + )) + })?; + let literal = lit(value.clone()); + patch = match predecessor { + Some(predecessor) => patch.insert_after(predecessor, literal), + None => patch.prepend(literal), + }; + } else if *field.data_type() != DataType::VOID { + predecessor = Some(name); + } } } - Expression::transform(transform) - } - - /// Get the write context for this transaction. At the moment, this is constant for the whole - /// transaction. - // Note: after we introduce metadata updates (modify table schema, etc.), we need to make sure - // that engines cannot call this method after a metadata change, since the write context could - // have invalid metadata. - // Note: Callers that use get_write_context may be writing data to the table and they might - // have invalid metadata. - pub fn get_write_context(&self) -> WriteContext { - let target_dir = self.read_snapshot.table_root(); - let snapshot_schema = self.read_snapshot.schema(); - let logical_to_physical = self.generate_logical_to_physical(); - let column_mapping_mode = self - .read_snapshot - .table_configuration() - .column_mapping_mode(); - - // Compute physical schema: exclude partition columns since they're stored in the path - // (unless materializePartitionColumns is enabled), and apply column mapping to transform - // logical field names to physical names. - let partition_columns: Vec = self - .read_snapshot - .table_configuration() - .partition_columns() - .to_vec(); - let materialize_partition_columns = self - .read_snapshot - .table_configuration() - .is_feature_enabled(&TableFeature::MaterializePartitionColumns); - let physical_fields = snapshot_schema - .fields() - .filter(|f| { - materialize_partition_columns || !partition_columns.contains(&f.name().to_string()) - }) - .map(|f| { - // NOTE: This should never fail, as schema was already validated during TableConfiguration construction. - f.make_physical(column_mapping_mode).unwrap_or_else(|e| { - warn!("make_physical failed: {e}"); - f.clone() - }) - }); - let physical_schema = Arc::new(StructType::new_unchecked(physical_fields)); - - // Get stats columns from table configuration - let stats_columns = self.stats_columns(); - - WriteContext::new( - target_dir.clone(), - snapshot_schema, - physical_schema, - Arc::new(logical_to_physical), - column_mapping_mode, - stats_columns, - ) + let patch = add_void_stripping(patch, &logical_schema); + Expression::struct_patch(patch) + } + + /// Returns the logical partition column names for this table. + pub fn logical_partition_columns(&self) -> &[String] { + self.effective_table_config.logical_partition_columns() + } + + // TODO(#2630): Expose nested column defaults through the transaction API. + /// Returns the column default for every top-level column in this table's logical schema that + /// declares one, keyed by logical column name. + /// + /// Connectors use this to discover which columns have defaults, then call + /// [`ColumnDefault::to_scalar`] on each (or fall back to [`ColumnDefault::raw_sql`] when the + /// kernel cannot parse the default) to materialize the column before writing. After handling + /// every omitted column, call [`ack_column_defaults`](Self::ack_column_defaults) before + /// requesting a write context. + /// + /// Keys are `String` rather than [`ColumnName`] because the kernel currently surfaces defaults + /// only for top-level columns, consistent with partition columns. This is a kernel limitation, + /// not a protocol one. + /// + /// Malformed defaults (a non-string `CURRENT_DEFAULT`, or a non-`NULL` default on a Variant + /// column) are rejected eagerly at snapshot load (when constructing the + /// [`TableConfiguration`]), so by the time this runs the metadata is already validated. + /// Orphaned metadata (a `CURRENT_DEFAULT` without the `allowColumnDefaults` writer feature) + /// is tolerated at table load but is not surfaced through this method. + /// + /// # Errors + /// + /// Propagates any error from [`StructField::column_default`]. + pub fn top_level_column_defaults(&self) -> DeltaResult>> { + if !self + .effective_table_config + .is_feature_enabled(&TableFeature::AllowColumnDefaults) + { + tracing::info!( + "allowColumnDefaults is not enabled; the schema may contain orphaned column-default metadata" + ); + return Ok(HashMap::new()); + } + let mut defaults = HashMap::new(); + for field in self.effective_table_config.logical_schema_ref().fields() { + if let Some(column_default) = field.column_default()? { + defaults.insert(field.name().clone(), column_default); + } + } + Ok(defaults) + } + + /// Validates that the table's logical schema supports data writes. + /// + /// Called at the top of [`partitioned_write_context`](Self::partitioned_write_context) and + /// [`unpartitioned_write_context`](Self::unpartitioned_write_context), before any Parquet is + /// written, so connectors fail fast when the schema contains unsupported data types or void + /// placements that cannot produce valid files. + /// The commit-time check in [`commit`](Self::commit) remains as defense-in-depth for callers + /// that reach [`add_files`](Self::add_files) without going through a write context. + fn validate_for_data_write(&self) -> DeltaResult<()> { + validate_schema_for_write(&self.effective_table_config.logical_schema()) + } + + /// Builds the [`SharedWriteState`] for a write context. + fn shared_write_state(&self) -> DeltaResult> { + let table_config = &self.effective_table_config; + let props = table_config.table_properties(); + Ok(Arc::new(SharedWriteState { + table_root: table_config.table_root().clone(), + logical_schema: table_config.logical_schema_without_partition_columns(), + physical_schema: table_config.physical_write_schema(), + column_mapping_mode: table_config.column_mapping_mode(), + stats_columns: self.stats_columns(), + logical_partition_columns: table_config.logical_partition_columns().to_vec(), + randomize_file_prefixes: props.should_randomize_file_prefixes(), + random_prefix_length: props.random_prefix_length(), + })) + } + + /// Creates a write context for writing data to a specific partition. + /// + /// Performs the following validations and transformations: + /// + /// - **Key completeness**: ensures all partition columns are present and no extra keys exist. + /// For example, if the table has partition columns `["year", "region"]` and you pass + /// `{"year": Scalar::Integer(2024)}`, this returns an error for missing "region". + /// + /// - **Case normalization**: matches keys case-insensitively against the schema and normalizes + /// to schema case. For example, passing `"YEAR"` for a column named `"year"` is accepted and + /// normalized. + /// + /// - **Type checking**: rejects non-primitive partition column types (struct, array, map) and + /// validates that each non-null `Scalar`'s type matches the partition column's schema type. + /// For example, passing `Scalar::String("2024")` for an `INTEGER` column returns an error. + /// Null-equivalent scalars (null scalars, empty strings, and empty binary) all of which + /// collapse to JSON null in `partitionValues`) skip the value type check, but they are only + /// legal when the partition column is nullable; passing any of these for a `nullable: false` + /// partition column returns an error. + /// + /// - **Value serialization**: serializes each `Scalar` to a protocol-compliant string per the + /// Delta protocol's "Partition Value Serialization" rules. `Scalar::Null(...)` becomes `None` + /// in `add.partitionValues` (JSON null). `Scalar::String("")` also becomes `None` (empty + /// string equals null for all types). `Scalar::Date(19723)` becomes `Some("2024-01-01")`. + /// + /// - **Key translation**: translates logical column names to physical names using the table's + /// column mapping mode. For example, under `ColumnMappingMode::Name`, logical `"year"` might + /// become physical `"col-abc-123"` in the `partitionValues` map. + /// + /// - **Partition column materialization**: the returned [`WriteContext`]'s + /// [`logical_to_physical`] expression injects partition columns when the table requires + /// materializing partition columns (e.g. `materializePartitionColumns` or `icebergCompatV3`). + /// The input data fed to that expression must not contain partition columns. + /// + /// The returned [`WriteContext`] also provides a [`write_dir`] that returns the correct + /// target directory (Hive-style paths when column mapping is off, random prefix when on). + /// + /// Returns an error if the table is not partitioned (use + /// [`unpartitioned_write_context`](Self::unpartitioned_write_context) instead), or if the + /// table enables `allowColumnDefaults`, declares at least one column default, and + /// [`ack_column_defaults`](Self::ack_column_defaults) has not been called. + /// + /// [`write_dir`]: WriteContext::write_dir + /// [`logical_to_physical`]: WriteContext::logical_to_physical + pub fn partitioned_write_context( + &self, + partition_values: HashMap, + ) -> DeltaResult { + self.ensure_schema_non_empty_for_write_context()?; + self.ensure_column_defaults_acknowledged()?; + self.validate_for_data_write()?; + let shared = self.shared_write_state()?; + require!( + !shared.logical_partition_columns.is_empty(), + Error::generic("table is not partitioned; use unpartitioned_write_context() instead") + ); + // Validate keys (completeness, case normalization) and value types, then return + // the map re-keyed to schema case. + let full_logical_schema = self.effective_table_config.logical_schema(); + let normalized = validate_partition_values( + &shared.logical_partition_columns, + &full_logical_schema, + partition_values, + )?; + + // Serialize values and translate keys from logical to physical names. + let mut serialized = HashMap::with_capacity(normalized.len()); + for logical_name in &shared.logical_partition_columns { + let scalar = normalized.get(logical_name).ok_or_else(|| { + Error::internal_error(format!( + "partition column '{logical_name}' missing after validation" + )) + })?; + let value = serialize_partition_value(scalar)?; + let physical_name = full_logical_schema + .field(logical_name) + .ok_or_else(|| { + Error::internal_error(format!( + "partition column '{logical_name}' not found in schema after validation" + )) + })? + .physical_name(shared.column_mapping_mode) + .to_string(); + serialized.insert(physical_name, value); + } + let logical_to_physical = Arc::new(self.generate_logical_to_physical(Some(&normalized))?); + + Ok(WriteContext { + shared, + logical_to_physical, + physical_partition_values: serialized, + }) + } + + /// Creates a write context for writing data to an unpartitioned table. + /// + /// Returns an error if the table has partition columns (use + /// [`partitioned_write_context`](Self::partitioned_write_context) instead), or if the table + /// enables `allowColumnDefaults`, declares at least one column default, and + /// [`ack_column_defaults`](Self::ack_column_defaults) has not been called. + pub fn unpartitioned_write_context(&self) -> DeltaResult { + self.ensure_schema_non_empty_for_write_context()?; + self.ensure_column_defaults_acknowledged()?; + self.validate_for_data_write()?; + let shared = self.shared_write_state()?; + require!( + shared.logical_partition_columns.is_empty(), + Error::generic("table is partitioned; use partitioned_write_context() instead") + ); + let logical_to_physical = Arc::new(self.generate_logical_to_physical(None)?); + Ok(WriteContext { + shared, + logical_to_physical, + physical_partition_values: HashMap::new(), + }) } /// Add files to include in this transaction. This API generally enables the engine to @@ -749,26 +1254,43 @@ impl Transaction { pub fn add_files(&mut self, add_metadata: Box) { self.add_files_metadata.push(add_metadata); } +} - /// Validate that add files have required statistics for clustering columns. +// ============================================================================= +// Internal methods available on ALL transaction types (used by commit path) +// ============================================================================= +impl Transaction { + /// Validate that add files carry the per-file statistics required by the table's protocol. + /// + /// Currently checks two protocol requirements: + /// - `stats.numRecords` must be present when [`requires_stats_num_records`] returns true. + /// - Per-file min/max/nullCount must be present for clustering columns when the + /// `ClusteredTable` feature is enabled. + /// + /// Other stat columns (e.g. the conventional "first 32 columns") are not validated here + /// because they are not protocol-required. /// - /// Per the Delta protocol, writers MUST collect per-file statistics for clustering columns - /// when the `ClusteredTable` feature is enabled. Other stat columns (e.g. the conventional - /// "first 32 columns") are not validated here because they are not protocol-required. + /// Only add files are validated(remove files do not carry statistics). /// - /// Only add files are validated — remove files do not carry statistics. + /// [`requires_stats_num_records`]: crate::table_configuration::TableConfiguration::requires_stats_num_records fn validate_add_files_stats(&self, add_files: &[Box]) -> DeltaResult<()> { if add_files.is_empty() { return Ok(()); } - if let Some(ref clustering_cols) = self.clustering_columns_physical { + if self.effective_table_config.requires_stats_num_records() { + // TODO: Likely it's better to merge this with the clustering column validation below, + // benchmark it and see if it's faster. If so, refactor this to do both validations in + // one pass. + stats_verifier::verify_num_records_present(add_files)?; + } + if let Some(ref clustering_cols) = self.physical_clustering_columns { if !clustering_cols.is_empty() { - let physical_schema = self.read_snapshot.table_configuration().physical_schema(); + let physical_schema = self.effective_table_config.physical_schema(); let columns_with_types: Vec<(ColumnName, DataType)> = clustering_cols .iter() .map(|col| { let data_type = physical_schema - .walk_column_fields(col)? + .fields_of_path(col)? .last() .map(|field| field.data_type().clone()) .ok_or_else(|| { @@ -779,14 +1301,14 @@ impl Transaction { Ok((col.clone(), data_type)) }) .collect::>()?; - let verifier = StatsVerifier::new(columns_with_types); + let verifier = StatsColumnVerifier::new(columns_with_types); verifier.verify(add_files)?; } } Ok(()) } - /// Generate add actions, handling row tracking internally if needed + /// Generates add actions and row tracking domain metadata for a commit. #[instrument(name = "txn.gen_adds", skip_all, err)] fn generate_adds<'a>( &'a self, @@ -796,128 +1318,106 @@ impl Transaction { EngineDataResultIterator<'a>, Option, )> { - fn build_add_actions<'a, I, T>( - engine: &dyn Engine, - add_files_metadata: I, - input_schema: SchemaRef, - output_schema: SchemaRef, - data_change: bool, - ) -> impl Iterator>> + 'a - where - I: Iterator> + Send + 'a, - T: Deref + Send + 'a, - { - let evaluation_handler = engine.evaluation_handler(); - - add_files_metadata.map(move |add_files_batch| { - // Convert stats to a JSON string and nest the add action in a top-level struct - let transform = Expression::transform( - Transform::new_top_level() - .with_inserted_field( - Some("modificationTime"), - Expression::literal(data_change).into(), - ) - .with_replaced_field( - "stats", - Expression::unary(ToJson, Expression::column(["stats"])).into(), - ), - ); - let adds_expr = Expression::struct_from([transform]); - let adds_evaluator = evaluation_handler.new_expression_evaluator( - input_schema.clone(), - Arc::new(adds_expr), - as_log_add_schema(output_schema.clone()).into(), - )?; - adds_evaluator.evaluate(add_files_batch?.deref()) - }) - } + // Note: this does not require delta.enableRowTracking=true. "supported" is sufficient + // for writers to assign row IDs. + let row_tracking_supported = self.effective_table_config.should_write_row_tracking(); if self.add_files_metadata.is_empty() { - return Ok((Box::new(iter::empty()), None)); - } - - let commit_version = i64::try_from(commit_version) - .map_err(|_| Error::generic("Commit version too large to fit in i64"))?; - - let needs_row_tracking = self - .read_snapshot - .table_configuration() - .should_write_row_tracking(); - - // Row tracking is not yet supported for create-table with data - if needs_row_tracking && self.is_create_table() { - return Err(Error::unsupported( - "Row tracking is not yet supported for create table with data", - )); + // No files to add. For an empty CREATE TABLE with row tracking, emit the initial + // high water mark domain metadata (rowIdHighWaterMark = -1) so subsequent writes + // have a valid starting point. For all other empty commits (metadata-only, etc.), + // nothing row-tracking-related needs to be written. + let row_tracking_dm = (row_tracking_supported && self.is_create_table()) + .then(RowTrackingDomainMetadata::initial); + return Ok((Box::new(iter::empty()), row_tracking_dm)); } - if needs_row_tracking { - // Read the current rowIdHighWaterMark from the snapshot's row tracking domain metadata - let row_id_high_water_mark = - RowTrackingDomainMetadata::get_high_water_mark(&self.read_snapshot, engine)?; - - // Create a row tracking visitor and visit all files to collect row tracking information - let mut row_tracking_visitor = RowTrackingVisitor::new( - row_id_high_water_mark, - Some(self.add_files_metadata.len()), - ); - - // We visit all files with the row visitor before creating the add action iterator - // because we need to know the final row ID high water mark to create the domain metadata action - for add_files_batch in &self.add_files_metadata { - row_tracking_visitor.visit_rows_of(add_files_batch.deref())?; - } - - // Deconstruct the row tracking visitor to avoid borrowing issues - let RowTrackingVisitor { - base_row_id_batches, - row_id_high_water_mark, - } = row_tracking_visitor; - - // Create extended add files with row tracking columns - let extended_add_files = self.add_files_metadata.iter().zip(base_row_id_batches).map( - move |(add_files_batch, base_row_ids)| { - let commit_versions = vec![commit_version; base_row_ids.len()]; - let base_row_ids_array = - ArrayData::try_new(ArrayType::new(DataType::LONG, true), base_row_ids)?; - let commit_versions_array = - ArrayData::try_new(ArrayType::new(DataType::LONG, true), commit_versions)?; - - add_files_batch.append_columns( - with_row_tracking_cols(&Arc::new(StructType::new_unchecked(vec![]))), - vec![base_row_ids_array, commit_versions_array], - ) - }, - ); - - // Generate add actions including row tracking metadata - let add_actions = build_add_actions( - engine, - extended_add_files, - with_row_tracking_cols(self.add_files_schema()), - with_row_tracking_cols(&with_stats_col(&ADD_FILES_SCHEMA_WITH_DATA_CHANGE.clone())), - self.data_change, - ); + let commit_version = version_as_i64(commit_version)?; - // Generate a row tracking domain metadata based on the final high water mark - let row_tracking_domain_metadata: RowTrackingDomainMetadata = - RowTrackingDomainMetadata::new(row_id_high_water_mark); - - Ok((Box::new(add_actions), Some(row_tracking_domain_metadata))) + if row_tracking_supported { + self.generate_adds_with_row_tracking(engine, commit_version) } else { - // Simple case without row tracking let add_actions = build_add_actions( engine, self.add_files_metadata.iter().map(|a| Ok(a.deref())), self.add_files_schema().clone(), - with_stats_col(&ADD_FILES_SCHEMA_WITH_DATA_CHANGE.clone()), self.data_change, - ); - + )?; Ok((Box::new(add_actions), None)) } } + /// Generates add actions with row tracking columns and the row ID high water mark + /// domain metadata. + /// + /// Visits all add file batches once to read `numRecords` per file, assigning a unique + /// non-overlapping `baseRowId` range to each file and computing the final high water mark + /// for the domain metadata action. The initial high water mark is read from the snapshot + /// for existing tables, or defaults to -1 for create-table (no prior log to read from). + fn generate_adds_with_row_tracking<'a>( + &'a self, + engine: &dyn Engine, + commit_version: i64, + ) -> DeltaResult<( + EngineDataResultIterator<'a>, + Option, + )> { + let row_id_high_water_mark = if self.is_create_table() { + None + } else { + RowTrackingDomainMetadata::get_high_water_mark(self.read_snapshot()?, engine)? + }; + + // Create a row tracking visitor and visit all files to collect row tracking information + let mut row_tracking_visitor = + RowTrackingVisitor::new(row_id_high_water_mark, Some(self.add_files_metadata.len())); + + // We visit all files with the row visitor before creating the add action iterator because + // we need to know the final row ID high water mark to create the domain metadata action. + for add_files_batch in &self.add_files_metadata { + row_tracking_visitor.visit_rows_of(add_files_batch.deref())?; + } + + // Destructure the visitor to move base_row_id_batches into the add-files iterator + // while also extracting the final high water mark for the domain metadata action. + let RowTrackingVisitor { + base_row_id_batches, + row_id_high_water_mark, + } = row_tracking_visitor; + + // Create extended add files with row tracking columns + let extended_add_files = self.add_files_metadata.iter().zip(base_row_id_batches).map( + move |(add_files_batch, base_row_ids)| { + let commit_versions = vec![commit_version; base_row_ids.len()]; + let base_row_ids_array = + ArrayData::try_new(ArrayType::new(DataType::LONG, true), base_row_ids)?; + let commit_versions_array = + ArrayData::try_new(ArrayType::new(DataType::LONG, true), commit_versions)?; + + let row_tracking_schema = + with_row_tracking_cols(&Arc::new(StructType::new_unchecked(vec![])))?; + add_files_batch.append_columns( + row_tracking_schema, + vec![base_row_ids_array, commit_versions_array], + ) + }, + ); + + // Generate add actions including row tracking metadata + let add_actions = build_add_actions( + engine, + extended_add_files, + with_row_tracking_cols(self.add_files_schema())?, + self.data_change, + )?; + + // Generate a row tracking domain metadata based on the final high water mark + let row_tracking_domain_metadata: RowTrackingDomainMetadata = + RowTrackingDomainMetadata::new(row_id_high_water_mark); + + Ok((Box::new(add_actions), Some(row_tracking_domain_metadata))) + } + fn into_committed( self, file_meta: FileMeta, @@ -927,47 +1427,92 @@ impl Transaction { let commit_version = parsed_commit.version; - let post_commit_stats = PostCommitStats { - commits_since_checkpoint: self.read_snapshot.log_segment().commits_since_checkpoint() - + 1, - commits_since_log_compaction: self - .read_snapshot - .log_segment() - .commits_since_log_compaction_or_checkpoint() - + 1, + let (post_commit_stats, post_commit_snapshot) = match &self.read_snapshot_opt { + Some(snap) => { + // Existing table path: use the read snapshot to compute post-commit state. + let stats = PostCommitStats { + commits_since_checkpoint: snap.log_segment().commits_since_checkpoint() + 1, + commits_since_log_compaction: snap + .log_segment() + .commits_since_log_compaction_or_checkpoint() + + 1, + }; + let snapshot = snap.new_post_commit(parsed_commit, crc_delta)?; + (stats, Arc::new(snapshot)) + } + None => { + // CREATE TABLE path: build a fresh Snapshot at version 0. + let log_root = self + .effective_table_config + .table_root() + .join("_delta_log/")?; + let log_segment = LogSegment::new_for_version_zero(log_root, parsed_commit)?; + let crc = crc_delta.into_complete_crc(0).ok_or_else(|| { + Error::internal_error("CREATE TABLE CRC delta is missing protocol or metadata") + })?; + let stats = PostCommitStats { + commits_since_checkpoint: 1, + commits_since_log_compaction: 1, + }; + let snapshot = Snapshot::new_with_crc( + log_segment, + self.effective_table_config, + Some(Arc::new(crc)), + true, /* built_as_latest */ + )?; + (stats, Arc::new(snapshot)) + } }; Ok(CommittedTransaction { commit_version, post_commit_stats, - post_commit_snapshot: Some(Arc::new( - self.read_snapshot - .new_post_commit(parsed_commit, crc_delta)?, - )), + post_commit_snapshot: Some(post_commit_snapshot), }) } - /// Build a [`CrcDelta`] from the transaction's staged file metadata and commit state. + /// Build a [`CrcDelta`] from the transaction's commit state and a precomputed + /// [`FileStatsDelta`]. fn build_crc_delta( &self, + file_stats: FileStatsDelta, in_commit_timestamp: Option, dm_changes: Vec, ) -> DeltaResult { - let file_stats = FileStatsDelta::try_compute_for_txn( - &self.add_files_metadata, - &self.remove_files_metadata, - )?; - let is_create = self.is_create_table(); + // TODO: drop these conversions by migrating the upstream chain + // (`CommitMetadata.domain_metadata_changes`, `Transaction.set_transactions`) + // to `HashMap`, lifting protocol-mandated uniqueness from runtime + // checks into the type system. + let domain_metadata = dm_changes + .into_iter() + .map(|dm| (dm.domain().to_string(), dm)) + .collect(); + let set_transactions = self + .set_transactions + .iter() + .map(|txn| (txn.app_id.clone(), txn.clone())) + .collect(); + // Although `remove.size` is optional per the Delta protocol, the kernel write path + // enforces presence: `try_compute_for_txn` above errors with `MissingData` if any + // add or remove row lacks `size` (see `FileStatsVisitor::visit` in + // `kernel/src/crc/file_stats.rs`). So at this point every size is known to be + // present, and only operation classification can flip `is_incremental_safe`. + let is_incremental_safe = self + .operation + .as_deref() + .is_some_and(is_incremental_safe_operation); Ok(CrcDelta { file_stats, - protocol: is_create - .then(|| self.read_snapshot.table_configuration().protocol().clone()), - metadata: is_create - .then(|| self.read_snapshot.table_configuration().metadata().clone()), - domain_metadata_changes: dm_changes, + protocol: self + .should_emit_protocol + .then(|| self.effective_table_config.protocol().clone()), + metadata: self + .should_emit_metadata + .then(|| self.effective_table_config.metadata().clone()), + domain_metadata, + set_transactions, in_commit_timestamp, - operation: self.operation.clone(), - has_missing_file_size: false, // writes always have sizes + is_incremental_safe, }) } @@ -995,9 +1540,9 @@ impl Transaction { /// /// - `engine`: The engine used for expression evaluation /// - `remove_files_metadata`: Iterator over scan file metadata to transform into Remove actions - /// - `columns_to_drop`: Column names to drop from the scan metadata before transformation. - /// This is used to remove temporary columns like the intermediate deletion vector column - /// added during DV updates. + /// - `columns_to_drop`: Column names to drop from the scan metadata before transformation. This + /// is used to remove temporary columns like the intermediate deletion vector column added + /// during DV updates. /// /// # Returns /// @@ -1021,58 +1566,41 @@ impl Transaction { } let input_schema = scan_row_schema(); - let target_schema = schema_with_all_fields_nullable(get_log_remove_schema())?; + let target_schema = schema_with_all_fields_nullable(&LOG_REMOVE_SCHEMA); let evaluation_handler = engine.evaluation_handler(); - // Create the transform expression once, since it only contains literals and column references - let mut transform = Transform::new_top_level() - // deletionTimestamp - .with_inserted_field( - Some("path"), - Expression::literal(self.commit_timestamp).into(), - ) - // dataChange - .with_inserted_field(Some("path"), Expression::literal(self.data_change).into()) - .with_inserted_field( - // extended_file_metadata - Some("path"), - Expression::literal(true).into(), - ) - .with_inserted_field( - Some("path"), - Expression::column([FILE_CONSTANT_VALUES_NAME, "partitionValues"]).into(), - ) - // tags - .with_inserted_field( - Some("stats"), - Expression::column([FILE_CONSTANT_VALUES_NAME, TAGS_NAME]).into(), - ) - .with_inserted_field( - Some("deletionVector"), - Expression::column([FILE_CONSTANT_VALUES_NAME, BASE_ROW_ID_NAME]).into(), - ) - .with_inserted_field( - Some("deletionVector"), - Expression::column([FILE_CONSTANT_VALUES_NAME, DEFAULT_ROW_COMMIT_VERSION_NAME]) - .into(), + let make_eval = |coalesce_stats_with_parsed: bool| -> DeltaResult<_> { + let patch = build_remove_struct_patch( + self.commit_timestamp, + self.data_change, + columns_to_drop, + coalesce_stats_with_parsed, + )?; + let expr = Arc::new(Expression::struct_from([Expression::struct_patch(patch)?])); + evaluation_handler.new_expression_evaluator( + input_schema.clone(), + expr, + target_schema.clone().into(), ) - .with_dropped_field(FILE_CONSTANT_VALUES_NAME) - .with_dropped_field("modificationTime"); - - // Drop any additional columns specified in columns_to_drop - for column_to_drop in columns_to_drop { - transform = transform.with_dropped_field(*column_to_drop); - } + }; - let expr = Arc::new(Expression::struct_from([Expression::transform(transform)])); - let file_action_eval = Arc::new(evaluation_handler.new_expression_evaluator( - input_schema.clone(), - expr.clone(), - target_schema.into(), - )?); + // Build two evaluators: one for the common case where scan files do not include a + // stats_parsed column, and one for predicate-based scans that include stats_parsed. + // The stats_parsed evaluator coalesces stats with ToJson(stats_parsed) to handle the + // case where stats is null (e.g., on V2 checkpoints with writeStatsAsJson=false) and + // then drops the stats_parsed column. + let base_eval = Arc::new(make_eval(false)?); + let stats_parsed_eval = Arc::new(make_eval(true)?); + let stats_parsed_col = column_name!(STATS_PARSED_NAME); Ok(remove_files_metadata.map(move |file_metadata_batch| { - let updated_engine_data = file_action_eval.evaluate(file_metadata_batch.data())?; + let data = file_metadata_batch.data(); + let evaluator = if data.has_field(&stats_parsed_col) { + &stats_parsed_eval + } else { + &base_eval + }; + let updated_engine_data = evaluator.evaluate(data)?; FilteredEngineData::try_new( updated_engine_data, file_metadata_batch.selection_vector().to_vec(), @@ -1081,90 +1609,72 @@ impl Transaction { } } -/// WriteContext is data derived from a [`Transaction`] that can be provided to writers in order to -/// write table data. +/// Builds the struct patch for converting scan row metadata into a Remove action. /// -/// [`Transaction`]: struct.Transaction.html -pub struct WriteContext { - target_dir: Url, - logical_schema: SchemaRef, - physical_schema: SchemaRef, - logical_to_physical: ExpressionRef, - column_mapping_mode: ColumnMappingMode, - /// Column names that should have statistics collected during writes. - stats_columns: Vec, -} - -impl WriteContext { - fn new( - target_dir: Url, - logical_schema: SchemaRef, - physical_schema: SchemaRef, - logical_to_physical: ExpressionRef, - column_mapping_mode: ColumnMappingMode, - stats_columns: Vec, - ) -> Self { - WriteContext { - target_dir, - logical_schema, - physical_schema, - logical_to_physical, - column_mapping_mode, - stats_columns, - } - } - - pub fn target_dir(&self) -> &Url { - &self.target_dir - } - - pub fn logical_schema(&self) -> &SchemaRef { - &self.logical_schema - } - - pub fn physical_schema(&self) -> &SchemaRef { - &self.physical_schema - } +/// Handles two "parsed" columns that predicate-based scans add to scan metadata: +/// +/// - `stats_parsed`: when `coalesce_stats_with_parsed` is true, the `stats` field is replaced with +/// `COALESCE(stats, TO_JSON(stats_parsed))` and `stats_parsed` is dropped. The coalesce handles +/// cases where `stats` is null (e.g., V2 checkpoints with `writeStatsAsJson=false`) by +/// reconstructing the JSON from the parsed representation. +/// - `partitionValues_parsed`: dropped if present. Unlike stats, no reconstruction is needed: the +/// Remove action's `partitionValues` is sourced from `fileConstantValues.partitionValues`, which +/// scans always populate from `add.partitionValues`. +fn build_remove_struct_patch( + commit_timestamp: i64, + data_change: bool, + columns_to_drop: &[&str], + coalesce_stats_with_parsed: bool, +) -> DeltaResult { + // Note: The Delta protocol requires `partitionValues`, `size`, and `tags` when + // `extendedFileMetadata` is true. We require only `partitionValues` and `size` to match Spark. + let extended_file_metadata = Predicate::and_from([ + col!(SIZE_NAME).is_not_null(), + col!(FILE_CONSTANT_VALUES_NAME, PARTITION_VALUES_NAME).is_not_null(), + ]); + let mut patch = ExpressionStructPatchBuilder::new() + // deletionTimestamp + .insert_after("path", lit(commit_timestamp)) + // dataChange + .insert_after("path", lit(data_change)) + // extended_file_metadata + .insert_after("path", Expression::from(extended_file_metadata)) + .insert_after( + "path", + col!(FILE_CONSTANT_VALUES_NAME, PARTITION_VALUES_NAME), + ); - pub fn logical_to_physical(&self) -> ExpressionRef { - self.logical_to_physical.clone() + if coalesce_stats_with_parsed { + // Replace stats with COALESCE(stats, TO_JSON(stats_parsed)) and drop stats_parsed. + let coalesce_stats = Expression::coalesce([ + col!("stats"), + Expression::unary(ToJson, col!(STATS_PARSED_NAME)), + ]); + patch = patch + .replace("stats", coalesce_stats) + .drop(STATS_PARSED_NAME); } - /// The [`ColumnMappingMode`] for this table. - pub fn column_mapping_mode(&self) -> ColumnMappingMode { - self.column_mapping_mode - } + patch = patch + .insert_after("stats", col!(FILE_CONSTANT_VALUES_NAME, TAGS_NAME)) + .insert_after( + "deletionVector", + col!(FILE_CONSTANT_VALUES_NAME, BASE_ROW_ID_NAME), + ) + .insert_after( + "deletionVector", + col!(FILE_CONSTANT_VALUES_NAME, DEFAULT_ROW_COMMIT_VERSION_NAME), + ) + .drop(FILE_CONSTANT_VALUES_NAME) + .drop("modificationTime") + // Added to scan output when the predicate touches a partition column. + .drop_if_exists(PARTITION_VALUES_PARSED_NAME); - /// Returns the column names that should have statistics collected during writes. - /// - /// Based on table configuration (dataSkippingNumIndexedCols, dataSkippingStatsColumns). - pub fn stats_columns(&self) -> &[ColumnName] { - &self.stats_columns + for column_to_drop in columns_to_drop { + patch = patch.drop(*column_to_drop); } - /// Generate a new unique absolute URL for a deletion vector file. - /// - /// This method generates a unique file name in the table directory. - /// Each call to this method returns a new unique path. - /// - /// # Arguments - /// - /// * `random_prefix` - A random prefix to use for the deletion vector file name. - /// Making this non-empty can help distributed load on object storage when writing/reading - /// to avoid throttling. Typically a random string fo 2-4 characters is sufficient - /// for this purpose. - /// - /// - /// # Examples - /// - /// ```rust,ignore - /// let write_context = transaction.get_write_context(); - /// let dv_path = write_context.new_deletion_vector_path(String::from(rand_string())); - /// // dv_url might be: s3://bucket/table/deletion_vector_d2c639aa-8816-431a-aaf6-d3fe2512ff61.bin - /// ``` - pub fn new_deletion_vector_path(&self, random_prefix: String) -> DeletionVectorPath { - DeletionVectorPath::new(self.target_dir.clone(), random_prefix) - } + patch.build() } /// Kernel exposes information about the state of the table that engines might want to use to @@ -1174,9 +1684,9 @@ pub struct PostCommitStats { /// The number of commits since this table has been checkpointed. Note that commit 0 is /// considered a checkpoint for the purposes of this computation. pub commits_since_checkpoint: u64, - /// The number of commits since the log has been compacted on this table. Note that a checkpoint - /// is considered a compaction for the purposes of this computation. Thus this is really the - /// number of commits since a compaction OR a checkpoint. + /// The number of commits since the log has been compacted on this table. Note that a + /// checkpoint is considered a compaction for the purposes of this computation. Thus this + /// is really the number of commits since a compaction OR a checkpoint. pub commits_since_log_compaction: u64, } @@ -1185,8 +1695,8 @@ pub struct PostCommitStats { /// error occurred, the result is Err(Error). /// /// The commit result can be one of the following: -/// - [CommittedTransaction]: the transaction was successfully committed. [PostCommitStats] and -/// in the future a post-commit snapshot can be obtained from the committed transaction. +/// - [CommittedTransaction]: the transaction was successfully committed. [PostCommitStats] and in +/// the future a post-commit snapshot can be obtained from the committed transaction. /// - [ConflictedTransaction]: the transaction conflicted with an existing version. This transcation /// must be rebased before retrying. (currently no rebase APIs exist, caller must create new txn) /// - [RetryableTransaction]: an IO (retryable) error occurred during the commit. This transaction @@ -1224,10 +1734,23 @@ impl CommitResult { other => panic!("Expected CommittedTransaction, got: {other:?}"), } } + + /// Unwraps the post-commit snapshot of the [`CommittedTransaction`], panicking if the + /// commit was not successful or the post-commit snapshot is missing. + /// TODO(#2494): Refactor existing tests to use this. + #[cfg(any(test, feature = "test-utils"))] + #[allow(clippy::panic, clippy::expect_used)] + pub fn unwrap_post_commit_snapshot(self) -> SnapshotRef { + self.unwrap_committed() + .post_commit_snapshot() + .expect("expected post-commit snapshot") + .clone() + } } /// This is the result of a successfully committed [Transaction]. One can retrieve the -/// [post_commit_stats], [commit version], and optionally the [post-commit snapshot] from this struct. +/// [post_commit_stats], [commit version], and optionally the [post-commit snapshot] from this +/// struct. /// /// [post_commit_stats]: Self::post_commit_stats /// [commit version]: Self::commit_version @@ -1295,41 +1818,53 @@ pub struct RetryableTransaction { #[cfg(test)] mod tests { use std::collections::HashMap; + use std::path::PathBuf; + use std::sync::Mutex; + + use ::test_utils::get_column; + use rstest::rstest; + use url::Url; use super::*; use crate::actions::deletion_vector::DeletionVectorDescriptor; use crate::actions::CommitInfo; + use crate::arrow::array::builder::{MapBuilder, MapFieldNames, StringBuilder}; use crate::arrow::array::{ - ArrayRef, Int32Array, Int64Array, ListArray, MapArray, StringArray, StructArray, + new_null_array, ArrayRef, Float64Array, Int32Array, Int64Array, NullArray, StringArray, + StructArray, }; - use crate::arrow::buffer::OffsetBuffer; use crate::arrow::datatypes::{ - DataType as ArrowDataType, Field as ArrowField, Fields, Schema as ArrowSchema, + DataType as ArrowDataType, Field as ArrowField, Schema as ArrowSchema, }; use crate::arrow::record_batch::RecordBatch; use crate::committer::{FileSystemCommitter, PublishMetadata}; - use crate::engine::arrow_conversion::TryIntoArrow; + use crate::engine::arrow_conversion::{TryFromArrow, TryIntoArrow}; use crate::engine::arrow_data::ArrowEngineData; use crate::engine::arrow_expression::ArrowEvaluationHandler; use crate::engine::sync::SyncEngine; - use crate::object_store::local::LocalFileSystem; - use crate::schema::MapType; + use crate::expressions::{MapData, Scalar, StructData}; + use crate::metrics::{MetricEvent, TableType, TransactionCommitFailure}; + use crate::object_store::memory::InMemory; + use crate::object_store::path::Path; + use crate::object_store::ObjectStoreExt as _; + use crate::scan::log_replay::PATH_NAME; + use crate::schema::{schema_ref, MapType}; use crate::table_features::ColumnMappingMode; + use crate::table_properties::APPEND_ONLY; use crate::transaction::create_table::create_table; - use crate::utils::test_utils::{ + use crate::transaction::data_layout::DataLayout; + use crate::unit_test_utils::{ + copy_test_table, create_valid_add_file_batch, install_thread_local_metrics_reporter, load_test_table, string_array_to_engine_data, test_schema_flat, test_schema_nested, - test_schema_with_array, test_schema_with_map, + test_schema_with_array, test_schema_with_map, CapturingReporter, }; - use crate::EvaluationHandler; - use crate::Snapshot; - use rstest::rstest; - use std::path::PathBuf; + use crate::{DeltaResultIterator, EvaluationHandler, Snapshot}; impl Transaction { /// Set clustering columns for testing purposes without needing a table /// with the ClusteredTable feature enabled. fn with_clustering_columns_for_test(mut self, columns: Vec) -> Self { - self.clustering_columns_physical = Some(columns); + self.physical_clustering_columns = Some(columns); self } } @@ -1341,7 +1876,7 @@ mod tests { fn commit( &self, _engine: &dyn Engine, - _actions: Box> + Send + '_>, + _actions: DeltaResultIterator<'_, FilteredEngineData>, _commit_metadata: CommitMetadata, ) -> DeltaResult { Err(Error::IOError(std::io::Error::other("simulated IO error"))) @@ -1358,6 +1893,56 @@ mod tests { } } + /// A mock committer that always returns a non-retryable (non-IO) error, used to test the + /// terminal error path. + struct GenericErrorCommitter; + + impl Committer for GenericErrorCommitter { + fn commit( + &self, + _engine: &dyn Engine, + _actions: DeltaResultIterator<'_, FilteredEngineData>, + _commit_metadata: CommitMetadata, + ) -> DeltaResult { + Err(Error::generic("simulated commit error")) + } + fn is_catalog_committer(&self) -> bool { + false + } + fn publish( + &self, + _engine: &dyn Engine, + _publish_metadata: PublishMetadata, + ) -> DeltaResult<()> { + Ok(()) + } + } + + /// A mock catalog committer, used to test catalog committer validation. + struct MockCatalogCommitter; + + impl Committer for MockCatalogCommitter { + fn commit( + &self, + _engine: &dyn Engine, + _actions: DeltaResultIterator<'_, FilteredEngineData>, + _commit_metadata: CommitMetadata, + ) -> DeltaResult { + // This won't be reached in tests — the validation error fires before commit. + Ok(CommitResponse::Conflict { version: 0 }) + } + fn is_catalog_committer(&self) -> bool { + true + } + fn publish( + &self, + _engine: &dyn Engine, + _publish_metadata: PublishMetadata, + ) -> DeltaResult<()> { + Ok(()) + } + } + /// Sets up a snapshot for a table with deletion vector support at version 1 fn setup_dv_enabled_table() -> (SyncEngine, Arc) { let engine = SyncEngine::new(); @@ -1380,7 +1965,45 @@ mod tests { (engine, snapshot) } - /// Creates a test deletion vector descriptor with default values (the DV might not exist on disk) + fn setup_dv_supported_but_disabled_table() -> DeltaResult<(Arc, Arc)> { + let storage = Arc::new(InMemory::new()); + let table_root = url::Url::parse("memory:///").unwrap(); + let engine = Arc::new(SyncEngine::new_with_store(storage.clone())); + let schema_json = serde_json::json!({ + "type": "struct", + "fields": [{ + "name": "id", + "type": "integer", + "nullable": true, + "metadata": {} + }] + }); + let actions = [ + r#"{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors"]}}"#.to_string(), + serde_json::json!({ + "metaData": { + "id": "test-id", + "format": {"provider": "parquet", "options": {}}, + "schemaString": schema_json.to_string(), + "partitionColumns": [], + "configuration": {}, + "createdTime": 1234567890 + } + }) + .to_string(), + ] + .join("\n"); + + let commit_path = Path::from("_delta_log/00000000000000000000.json"); + let rt = tokio::runtime::Runtime::new().unwrap(); + rt.block_on(storage.put(&commit_path, actions.into()))?; + let engine: Arc = engine; + let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; + Ok((engine, snapshot)) + } + + /// Creates a test deletion vector descriptor with default values (the DV might not exist on + /// disk) fn create_test_dv_descriptor(path_suffix: &str) -> DeletionVectorDescriptor { use crate::actions::deletion_vector::{ DeletionVectorDescriptor, DeletionVectorStorageType, @@ -1431,11 +2054,11 @@ mod tests { StructField::nullable( "stats", DataType::struct_type_unchecked(vec![ - StructField::nullable("numRecords", DataType::LONG), - StructField::nullable("nullCount", DataType::struct_type_unchecked(vec![])), - StructField::nullable("minValues", DataType::struct_type_unchecked(vec![])), - StructField::nullable("maxValues", DataType::struct_type_unchecked(vec![])), - StructField::nullable("tightBounds", DataType::BOOLEAN), + StructField::nullable(NUM_RECORDS, DataType::LONG), + StructField::nullable(NULL_COUNT, DataType::struct_type_unchecked(vec![])), + StructField::nullable(MIN_VALUES, DataType::struct_type_unchecked(vec![])), + StructField::nullable(MAX_VALUES, DataType::struct_type_unchecked(vec![])), + StructField::nullable(TIGHT_BOUNDS, DataType::BOOLEAN), ]), ), ]); @@ -1443,6 +2066,71 @@ mod tests { Ok(()) } + #[rstest] + #[case::base(false)] + #[case::row_tracking(true)] + fn test_add_action_projection_schema(#[case] row_tracking: bool) -> DeltaResult<()> { + let input_schema = if row_tracking { + with_row_tracking_cols(&BASE_ADD_FILES_SCHEMA)? + } else { + BASE_ADD_FILES_SCHEMA.clone() + }; + let (schema, _) = build_add_action_projection(input_schema.as_ref(), true)?; + let field_names: Vec<_> = schema.fields().map(|f| f.name().as_str()).collect(); + let expected_field_names = if row_tracking { + vec![ + "path", + "partitionValues", + "size", + "modificationTime", + "dataChange", + "stats", + "baseRowId", + "defaultRowCommitVersion", + ] + } else { + vec![ + "path", + "partitionValues", + "size", + "modificationTime", + "dataChange", + "stats", + ] + }; + assert_eq!(field_names, expected_field_names); + assert_eq!(schema.field("dataChange"), Some(&*DATA_CHANGE_COLUMN)); + assert_eq!( + schema.field("stats").unwrap().data_type(), + &DataType::STRING + ); + Ok(()) + } + + #[test] + fn test_remove_action_projection_sets_extended_metadata() -> DeltaResult<()> { + let patch = build_remove_struct_patch( + 0, /* commit_timestamp */ + true, /* data_change */ + &[], /* columns_to_drop */ + false, /* coalesce_stats_with_parsed */ + )?; + let path_patch = patch + .field_patches + .get("path") + .expect("path should have inserted fields"); + let extended_file_metadata = path_patch + .insertions + .get(2) + .expect("extendedFileMetadata should follow deletionTimestamp and dataChange"); + let expected = Expression::from_pred(Predicate::and_from([ + col!(SIZE_NAME).is_not_null(), + col!(FILE_CONSTANT_VALUES_NAME, PARTITION_VALUES_NAME).is_not_null(), + ])); + assert_eq!(extended_file_metadata.as_ref(), &expected); + Ok(()) + } + #[test] fn test_new_deletion_vector_path() -> Result<(), Box> { let engine = SyncEngine::new(); @@ -1456,7 +2144,7 @@ mod tests { let txn = snapshot .transaction(Box::new(FileSystemCommitter::new()), &engine)? .with_engine_info("default engine"); - let write_context = txn.get_write_context(); + let write_context = txn.unpartitioned_write_context().unwrap(); // Test with empty prefix let dv_path1 = write_context.new_deletion_vector_path(String::from("")); @@ -1479,7 +2167,182 @@ mod tests { } #[test] - fn test_physical_schema_excludes_partition_columns() -> Result<(), Box> { + fn write_context_reflects_updated_effective_table_config( + ) -> Result<(), Box> { + let (engine, snapshot) = setup_non_dv_table(); + let mut txn = snapshot + .clone() + .transaction(Box::new(FileSystemCommitter::new()), &engine)? + .with_engine_info("default engine"); + + // Regression coverage for stale SharedWriteState caching: keep the first context alive + // while the transaction's effective table config changes. + let initial_write_context = txn.unpartitioned_write_context()?; + assert!(!initial_write_context + .logical_schema() + .contains("fresh_column")); + + let mut evolved_fields: Vec = txn + .effective_table_config + .logical_schema() + .fields() + .cloned() + .collect(); + evolved_fields.push(StructField::nullable("fresh_column", DataType::INTEGER)); + let evolved_schema = Arc::new(StructType::new_unchecked(evolved_fields)); + let evolved_metadata = txn + .effective_table_config + .metadata() + .clone() + .with_schema(evolved_schema.clone())?; + txn.effective_table_config = TableConfiguration::try_new_with_schema( + &txn.effective_table_config, + evolved_metadata, + evolved_schema, + )?; + + let updated_write_context = txn.unpartitioned_write_context()?; + assert!(updated_write_context + .logical_schema() + .contains("fresh_column")); + assert!(updated_write_context + .physical_schema() + .contains("fresh_column")); + assert!(!initial_write_context + .logical_schema() + .contains("fresh_column")); + + Ok(()) + } + + mod column_defaults { + use super::*; + use crate::schema::column_default::{field_with_default, field_with_invalid_default}; + + // NB: `test_utils::schema_with_column_defaults` cannot be used here. In `--lib` unit tests + // the crate under test and the `delta_kernel` that `test_utils` links are two distinct + // crate instances, so kernel schema types don't unify across the `test_utils` boundary. + // The `field_with_*` helpers live in-crate (`schema::column_default`) for the same reason. + + /// Builds a transaction whose effective logical schema is `schema`, with the + /// `allowColumnDefaults` writer feature enabled so any declared defaults are honored. + fn txn_with_schema(schema: StructType) -> Transaction { + txn_with_schema_and_writer_features(schema, [TableFeature::AllowColumnDefaults]) + } + + /// Like [`txn_with_schema`] but with an explicit writer-feature list, so a test can + /// exercise a table that does *not* enable `allowColumnDefaults`. Panics if the table + /// configuration fails to construct; use [`try_table_config`] to assert that error. + fn txn_with_schema_and_writer_features( + schema: StructType, + writer_features: impl IntoIterator, + ) -> Transaction { + let (engine, snapshot) = setup_non_dv_table(); + let mut txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), &engine) + .unwrap(); + txn.effective_table_config = try_table_config(&txn, schema, writer_features).unwrap(); + txn + } + + /// Builds the [`TableConfiguration`] a transaction would carry for `schema` and + /// `writer_features`, swapping a synthetic schema/protocol onto a real snapshot's config so + /// the validation `try_new` runs at construction can be exercised without `create_table`. + /// Returns the construction result so a test can assert eager-validation errors. + fn try_table_config( + base: &Transaction, + schema: StructType, + writer_features: impl IntoIterator, + ) -> DeltaResult { + let metadata = base + .effective_table_config + .metadata() + .clone() + .with_schema(Arc::new(schema)) + .unwrap(); + let protocol = + Protocol::try_new_modern(TableFeature::EMPTY_LIST, writer_features).unwrap(); + let version = base.effective_table_config.version(); + TableConfiguration::try_new_from( + &base.effective_table_config, + Some(metadata), + Some(protocol), + version, + ) + } + + /// A transaction over a real (non-DV) table, to use as the base config in + /// [`try_table_config`]. + fn base_txn() -> Transaction { + let (engine, snapshot) = setup_non_dv_table(); + snapshot + .transaction(Box::new(FileSystemCommitter::new()), &engine) + .unwrap() + } + + #[test] + fn collects_present_defaults_and_skips_columns_without_one() { + let schema = StructType::try_new(vec![ + field_with_default("parsable", DataType::INTEGER, "42"), + field_with_default("unparsable", DataType::TIMESTAMP, "current_timestamp()"), + StructField::nullable("no_default", DataType::STRING), + ]) + .unwrap(); + let txn = txn_with_schema(schema); + + let defaults = txn.top_level_column_defaults().unwrap(); + assert_eq!( + defaults.len(), + 2, + "only columns with a default are returned" + ); + assert!(!defaults.contains_key("no_default")); + + let parsable = &defaults["parsable"]; + assert_eq!(parsable.raw_sql(), "42"); + assert!(parsable.to_scalar().unwrap().is_some()); + + let unparsable = &defaults["unparsable"]; + assert_eq!(unparsable.raw_sql(), "current_timestamp()"); + assert!(unparsable.to_scalar().unwrap().is_none()); + } + + #[test] + fn returns_empty_map_when_no_column_has_a_default() { + let schema = StructType::try_new(vec![ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::STRING), + ]) + .unwrap(); + let txn = txn_with_schema(schema); + assert!(txn.top_level_column_defaults().unwrap().is_empty()); + } + + #[test] + fn load_rejects_malformed_default() { + let schema = StructType::try_new(vec![field_with_invalid_default("c")]).unwrap(); + + let err = try_table_config(&base_txn(), schema, [TableFeature::AllowColumnDefaults]) + .expect_err("non-string CURRENT_DEFAULT must error at load") + .to_string(); + assert!(err.contains("non-string"), "got: {err}"); + } + + #[test] + fn load_tolerates_default_present_but_feature_not_enabled() { + let schema = + StructType::try_new(vec![field_with_default("c", DataType::INTEGER, "42")]) + .unwrap(); + + // Orphaned column-default metadata (no `allowColumnDefaults` feature) is tolerated. + let txn = txn_with_schema_and_writer_features(schema, []); + assert!(txn.top_level_column_defaults().unwrap().is_empty()); + } + } + + #[test] + fn test_write_context_schemas_exclude_partition_columns( + ) -> Result<(), Box> { let engine = SyncEngine::new(); let path = std::fs::canonicalize(PathBuf::from("./tests/data/basic_partitioned/")).unwrap(); let url = url::Url::from_directory_path(path).unwrap(); @@ -1488,17 +2351,18 @@ mod tests { .transaction(Box::new(FileSystemCommitter::new()), &engine)? .with_engine_info("default engine"); - let write_context = txn.get_write_context(); + let write_context = txn.partitioned_write_context(HashMap::from([( + "letter".to_string(), + Scalar::String("a".into()), + )]))?; let logical_schema = write_context.logical_schema(); let physical_schema = write_context.physical_schema(); - // Logical schema should include the partition column + // Both schemas exclude partition columns. assert!( - logical_schema.contains("letter"), - "Logical schema should contain partition column 'letter'" + !logical_schema.contains("letter"), + "Logical schema should not contain partition column 'letter'" ); - - // Physical schema should exclude the partition column assert!( !physical_schema.contains("letter"), "Physical schema should not contain partition column 'letter' (stored in path)" @@ -1518,9 +2382,11 @@ mod tests { Ok(()) } - /// Helper: loads a test table snapshot and returns both the snapshot and its write context. - fn snapshot_and_write_context( + /// Loads a snapshot from `table_path` and builds a partitioned write context for the given + /// partition values. The table must be partitioned. + fn snapshot_and_partitioned_write_context( table_path: &str, + partition_values: HashMap, ) -> Result<(Arc, WriteContext), Box> { let engine = SyncEngine::new(); let path = std::fs::canonicalize(PathBuf::from(table_path)).unwrap(); @@ -1529,7 +2395,8 @@ mod tests { let txn = snapshot .clone() .transaction(Box::new(FileSystemCommitter::new()), &engine)?; - Ok((snapshot, txn.get_write_context())) + let wc = txn.partitioned_write_context(partition_values)?; + Ok((snapshot, wc)) } /// Helper: evaluates the logical-to-physical transform on the given batch and returns the @@ -1538,13 +2405,13 @@ mod tests { wc: &WriteContext, batch: RecordBatch, ) -> Result> { - let logical_schema = wc.logical_schema(); + let input_schema = StructType::try_from_arrow(batch.schema())?; let physical_schema = wc.physical_schema(); let l2p = wc.logical_to_physical(); let handler = ArrowEvaluationHandler; let evaluator = handler.new_expression_evaluator( - logical_schema.clone(), + input_schema.into(), l2p, physical_schema.clone().into(), )?; @@ -1554,47 +2421,132 @@ mod tests { Ok(result.record_batch().clone()) } - #[test] - fn test_materialize_partition_columns_in_write_context( + #[rstest] + #[case::not_materialized("./tests/data/basic_partitioned/", false)] + #[case::materialized("./tests/data/partitioned_with_materialize_feature/", true)] + fn test_partition_columns_materialized_in_logical_to_physical( + #[case] table_path: &str, + #[case] materialized: bool, ) -> Result<(), Box> { - // Without materializePartitionColumns, partition column should be dropped - let (snap_without, wc_without) = - snapshot_and_write_context("./tests/data/basic_partitioned/")?; - let partition_cols = snap_without.table_configuration().partition_columns(); - assert_eq!(partition_cols.len(), 1); - assert_eq!(partition_cols[0], "letter"); - assert!( - !snap_without + let (snapshot, wc) = snapshot_and_partitioned_write_context( + table_path, + HashMap::from([("letter".to_string(), Scalar::String("a".into()))]), + )?; + assert_eq!( + snapshot .table_configuration() .protocol() .has_table_feature(&TableFeature::MaterializePartitionColumns), - "basic_partitioned should not have materializePartitionColumns feature" - ); - let expr_str = format!("{}", wc_without.logical_to_physical()); - assert!( - expr_str.contains("drop letter"), - "Partition column 'letter' should be dropped. Expression: {expr_str}" + materialized ); - // With materializePartitionColumns, no columns should be dropped (identity transform) - let (snap_with, wc_with) = - snapshot_and_write_context("./tests/data/partitioned_with_materialize_feature/")?; - let partition_cols = snap_with.table_configuration().partition_columns(); - assert_eq!(partition_cols.len(), 1); - assert_eq!(partition_cols[0], "letter"); - assert!( - snap_with - .table_configuration() - .protocol() - .has_table_feature(&TableFeature::MaterializePartitionColumns), - "partitioned_with_materialize_feature should have materializePartitionColumns feature" - ); - let expr_str = format!("{}", wc_with.logical_to_physical()); - assert!( - !expr_str.contains("drop"), - "No columns should be dropped with materializePartitionColumns. Expression: {expr_str}" - ); + // The input data must exclude the partition column "letter". + let input_schema = Arc::new(ArrowSchema::new(vec![ + ArrowField::new("number", ArrowDataType::Int64, true), + ArrowField::new("a_float", ArrowDataType::Float64, true), + ])); + let batch = RecordBatch::try_new( + input_schema, + vec![ + Arc::new(Int64Array::from(vec![42])) as ArrayRef, + Arc::new(Float64Array::from(vec![1.5])), + ], + )?; + let rb = eval_logical_to_physical(&wc, batch)?; + let rb_schema = rb.schema(); + let names: Vec<&str> = rb_schema + .fields() + .iter() + .map(|f| f.name().as_str()) + .collect(); + if materialized { + assert_eq!(names, vec!["letter", "number", "a_float"]); + assert_eq!(get_column!(rb, "letter", StringArray).value(0), "a"); + } else { + assert_eq!(names, vec!["number", "a_float"]); + } + Ok(()) + } + + #[rstest] + #[case::cm_none(ColumnMappingMode::None)] + #[case::cm_name(ColumnMappingMode::Name)] + #[case::cm_id(ColumnMappingMode::Id)] + fn test_materialized_partition_column_insert( + #[case] cm_mode: ColumnMappingMode, + ) -> Result<(), Box> { + let cm = match cm_mode { + ColumnMappingMode::None => "none", + ColumnMappingMode::Name => "name", + ColumnMappingMode::Id => "id", + }; + let engine: Arc = + Arc::new(SyncEngine::new_with_store(Arc::new(InMemory::new()))); + // Logical order: [p1, p2, d1, v(void), p3, p4, d2]; partition cols = p1, p2, p3, p4. + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("p1", DataType::STRING), + StructField::nullable("p2", DataType::INTEGER), + StructField::nullable("d1", DataType::INTEGER), + StructField::nullable("v", DataType::VOID), + StructField::nullable("p3", DataType::STRING), + StructField::nullable("p4", DataType::INTEGER), + StructField::nullable("d2", DataType::INTEGER), + ])?); + let txn = create_table("memory:///t", schema, "DefaultEngine") + .with_data_layout(DataLayout::partitioned(["p1", "p2", "p3", "p4"])) + .with_table_properties([ + ("delta.feature.materializePartitionColumns", "supported"), + ("delta.columnMapping.mode", cm), + ]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))?; + + let wc = txn.partitioned_write_context(HashMap::from([ + ("p1".to_string(), Scalar::String("aa".into())), + ("p2".to_string(), Scalar::Integer(7)), + ("p3".to_string(), Scalar::String("cc".into())), + ("p4".to_string(), Scalar::Integer(9)), + ]))?; + + // Input excludes partition columns but keeps the void column, in logical schema + // order: [d1, v, d2]. + let input_schema = Arc::new(ArrowSchema::new(vec![ + ArrowField::new("d1", ArrowDataType::Int32, true), + ArrowField::new("v", ArrowDataType::Null, true), + ArrowField::new("d2", ArrowDataType::Int32, true), + ])); + let batch = RecordBatch::try_new( + input_schema, + vec![ + Arc::new(Int32Array::from(vec![10])) as ArrayRef, + Arc::new(NullArray::new(1)), + Arc::new(Int32Array::from(vec![20])), + ], + )?; + let rb = eval_logical_to_physical(&wc, batch)?; + + // With void stripped and partition literals inserted, the output names/order must match + // the physical schema exactly. + let rb_schema = rb.schema(); + let names: Vec<&str> = rb_schema + .fields() + .iter() + .map(|f| f.name().as_str()) + .collect(); + let physical_schema = wc.physical_schema(); + let expected_names: Vec<&str> = physical_schema + .fields() + .map(|f| f.name().as_str()) + .collect(); + assert_eq!(names, expected_names); + + // Verify the transformed data. + assert_eq!(get_column!(rb, names[0], StringArray).value(0), "aa"); // p1 (prepended) + assert_eq!(get_column!(rb, names[1], Int32Array).value(0), 7); // p2 (prepended) + assert_eq!(get_column!(rb, names[2], Int32Array).value(0), 10); // d1 + assert_eq!(get_column!(rb, names[3], StringArray).value(0), "cc"); // p3 (after d1, void skipped) + assert_eq!(get_column!(rb, names[4], Int32Array).value(0), 9); // p4 (after d1) + assert_eq!(get_column!(rb, names[5], Int32Array).value(0), 20); // d2 Ok(()) } @@ -1602,16 +2554,10 @@ mod tests { #[test] fn test_physical_schema_includes_partition_columns_when_materialized( ) -> Result<(), Box> { - let engine = SyncEngine::new(); - let path = std::fs::canonicalize(PathBuf::from( + let (_snapshot, write_context) = snapshot_and_partitioned_write_context( "./tests/data/partitioned_with_materialize_feature/", - )) - .unwrap(); - let url = url::Url::from_directory_path(path).unwrap(); - let snapshot = Snapshot::builder_for(url).at_version(1).build(&engine)?; - - let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), &engine)?; - let write_context = txn.get_write_context(); + HashMap::from([("letter".to_string(), Scalar::String("a".into()))]), + )?; let physical_schema = write_context.physical_schema(); assert!( @@ -1625,6 +2571,41 @@ mod tests { Ok(()) } + /// Using the wrong write context method for the table's partitioning returns an error. + #[rstest] + #[case::partitioned_on_unpartitioned( + "./tests/data/table-without-dv-small/", + true, + "not partitioned" + )] + #[case::unpartitioned_on_partitioned( + "./tests/data/basic_partitioned/", + false, + "table is partitioned" + )] + fn test_wrong_write_context_method_returns_error( + #[case] table_path: &str, + #[case] call_partitioned: bool, + #[case] expected_msg: &str, + ) -> Result<(), Box> { + let engine = SyncEngine::new(); + let path = std::fs::canonicalize(PathBuf::from(table_path)).unwrap(); + let url = url::Url::from_directory_path(path).unwrap(); + let snapshot = Snapshot::builder_for(url).build(&engine)?; + let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), &engine)?; + let result = if call_partitioned { + txn.partitioned_write_context(HashMap::from([("x".to_string(), Scalar::Integer(1))])) + } else { + txn.unpartitioned_write_context() + }; + let err = result.unwrap_err().to_string(); + assert!( + err.contains(expected_msg), + "expected '{expected_msg}' in error, got: {err}" + ); + Ok(()) + } + /// Tests that update_deletion_vectors validates table protocol requirements. /// Validates that attempting DV updates on unsupported tables returns protocol error. #[test] @@ -1645,6 +2626,27 @@ mod tests { Ok(()) } + #[test] + fn test_update_deletion_vectors_requires_enablement_property( + ) -> Result<(), Box> { + let (engine, snapshot) = setup_dv_supported_but_disabled_table()?; + let mut txn = create_dv_transaction(snapshot, engine.as_ref())?; + + let err = txn + .update_deletion_vectors(HashMap::new(), std::iter::empty()) + .expect_err("DV updates should require delta.enableDeletionVectors=true"); + + assert!( + matches!(err, Error::Unsupported(_)), + "unexpected error: {err}" + ); + assert!( + err.to_string().contains("delta.enableDeletionVectors"), + "error should mention the enablement property, got: {err}" + ); + Ok(()) + } + /// Tests that update_deletion_vectors validates DV descriptors match scan files. /// Validates detection of mismatch between provided DV descriptors and actual files. #[test] @@ -1669,6 +2671,99 @@ mod tests { Ok(()) } + /// Tests that a mismatch after scanning some files does not leave staged DV updates behind. + #[test] + fn test_update_deletion_vectors_mismatch_does_not_mutate_transaction( + ) -> Result<(), Box> { + let (engine, snapshot) = setup_dv_enabled_table(); + let mut txn = create_dv_transaction(snapshot.clone(), &engine)?; + let scan = snapshot.scan_builder().build()?; + let scan_metadata = scan + .scan_metadata(&engine)? + .collect::>>()?; + + let mut paths = Vec::new(); + for metadata in &scan_metadata { + paths = + metadata.visit_scan_files(paths, |paths, scan_file| paths.push(scan_file.path))?; + } + let existing_path = paths + .into_iter() + .next() + .ok_or_else(|| Error::generic("expected at least one scan file"))?; + + let mut dv_map = HashMap::new(); + dv_map.insert(existing_path, create_test_dv_descriptor("matched")); + dv_map.insert( + "non_existent_file.parquet".to_string(), + create_test_dv_descriptor("missing"), + ); + + let result = txn.update_deletion_vectors( + dv_map, + scan_metadata + .into_iter() + .map(|metadata| Ok(metadata.scan_files)), + ); + + assert!( + result.is_err(), + "Should fail when only some DV descriptors match scan files" + ); + assert!( + txn.dv_matched_files.is_empty(), + "Failed DV update should not leave staged file updates" + ); + Ok(()) + } + + #[test] + fn test_update_deletion_vectors_iter_error_does_not_mutate_transaction( + ) -> Result<(), Box> { + let (engine, snapshot) = setup_dv_enabled_table(); + let mut txn = create_dv_transaction(snapshot.clone(), &engine)?; + txn.dv_matched_files + .push(FilteredEngineData::with_all_rows_selected( + string_array_to_engine_data(StringArray::from(vec!["sentinel"])), + )); + let staged_len_before = txn.dv_matched_files.len(); + let scan = snapshot.scan_builder().build()?; + let scan_metadata = scan + .scan_metadata(&engine)? + .collect::>>()?; + + let mut paths = Vec::new(); + for metadata in &scan_metadata { + paths = + metadata.visit_scan_files(paths, |paths, scan_file| paths.push(scan_file.path))?; + } + let existing_path = paths + .into_iter() + .next() + .ok_or_else(|| Error::generic("expected at least one scan file"))?; + + let mut dv_map = HashMap::new(); + dv_map.insert(existing_path, create_test_dv_descriptor("matched")); + + let result = txn.update_deletion_vectors( + dv_map, + scan_metadata + .into_iter() + .map(|metadata| Ok(metadata.scan_files)) + .chain(std::iter::once(Err(Error::generic( + "simulated scan metadata failure", + )))), + ); + + assert!(result.is_err(), "iterator error should propagate"); + assert_eq!( + txn.dv_matched_files.len(), + staged_len_before, + "Failed DV update should not stage additional file updates" + ); + Ok(()) + } + /// Tests that update_deletion_vectors handles empty DV updates correctly as a no-op. /// This edge case occurs when a DELETE operation matches no rows. #[test] @@ -1690,14 +2785,100 @@ mod tests { // ============================================================================ // validate_blind_append tests // ============================================================================ - fn add_dummy_file(txn: &mut Transaction) { - let data = string_array_to_engine_data(StringArray::from(vec!["dummy"])); - txn.add_files(data); + fn add_dummy_file(txn: &mut Transaction) { + let batch = create_valid_add_file_batch(false /* all_nullable */); + txn.add_files(Box::new(ArrowEngineData::new(batch))); + } + + #[derive(Clone, Copy, Debug)] + enum DataRemoval { + RemoveFile, + DeletionVectorUpdate, } - fn create_existing_table_txn( - ) -> DeltaResult<(Arc, Transaction, Option)> { - let (engine, snapshot, tempdir) = load_test_table("table-without-dv-small")?; + fn set_append_only(txn: &mut Transaction, enabled: bool) -> DeltaResult<()> { + let metadata = txn + .effective_table_config + .metadata() + .clone() + .with_configuration_entry(APPEND_ONLY, enabled.to_string()); + txn.effective_table_config = TableConfiguration::try_new_from( + &txn.effective_table_config, + Some(metadata), + None, + txn.effective_table_config.version(), + )?; + Ok(()) + } + + fn make_scan_files(selection_vector: &[bool]) -> FilteredEngineData { + let schema: ArrowSchema = scan_row_schema().as_ref().try_into_arrow().unwrap(); + let row_count = selection_vector.len(); + let columns = schema + .fields() + .iter() + .map(|field| { + if field.name() == PATH_NAME { + Arc::new(StringArray::from_iter_values( + (0..row_count).map(|index| format!("file-{index}.parquet")), + )) as ArrayRef + } else if field.name() == SIZE_NAME { + Arc::new(Int64Array::from(vec![1; row_count])) + } else if field.name() == FILE_CONSTANT_VALUES_NAME { + let ArrowDataType::Struct(fields) = field.data_type() else { + panic!("fileConstantValues should be a struct"); + }; + let child_arrays = fields + .iter() + .map(|field| { + if field.name() == PARTITION_VALUES_NAME { + let names = MapFieldNames { + entry: "key_value".to_string(), + key: "key".to_string(), + value: "value".to_string(), + }; + let mut builder = MapBuilder::new( + Some(names), + StringBuilder::new(), + StringBuilder::new(), + ); + for _ in 0..row_count { + builder.append(true).unwrap(); + } + Arc::new(builder.finish()) as ArrayRef + } else { + new_null_array(field.data_type(), row_count) + } + }) + .collect(); + Arc::new(StructArray::new(fields.clone(), child_arrays, None)) + } else { + new_null_array(field.data_type(), row_count) + } + }) + .collect(); + let batch = RecordBatch::try_new(Arc::new(schema), columns).unwrap(); + FilteredEngineData::try_new( + Box::new(ArrowEngineData::new(batch)), + selection_vector.to_vec(), + ) + .unwrap() + } + + fn stage_data_removal(txn: &mut Transaction, removal: DataRemoval, selection_vector: &[bool]) { + let data = make_scan_files(selection_vector); + match removal { + DataRemoval::RemoveFile => txn.remove_files(data), + DataRemoval::DeletionVectorUpdate => txn.dv_matched_files.push(data), + } + } + + /// Build a transaction on a writable copy of the `table-without-dv-small` fixture. + fn create_existing_table_txn() -> DeltaResult<(Arc, Transaction, tempfile::TempDir)> + { + let (url, tempdir) = copy_test_table("table-without-dv-small")?; + let engine: Arc = Arc::new(SyncEngine::new()); + let snapshot = Snapshot::builder_for(url).build(engine.as_ref())?; let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())?; Ok((engine, txn, tempdir)) } @@ -1711,6 +2892,66 @@ mod tests { Ok(()) } + #[rstest] + #[case::append_only_disabled( + false, /* append_only */ + true, /* data_change */ + &[true, true, true], /* selection_vector */ + false, /* expected_error */ + )] + #[case::no_data_change( + true, /* append_only */ + false, /* data_change */ + &[true, true, true], /* selection_vector */ + false, /* expected_error */ + )] + #[case::all_unselected( + true, /* append_only */ + true, /* data_change */ + &[false, false, false], /* selection_vector */ + false, /* expected_error */ + )] + #[case::partial_selected( + true, /* append_only */ + true, /* data_change */ + &[false, true, false], /* selection_vector */ + true, /* expected_error */ + )] + #[case::all_selected( + true, /* append_only */ + true, /* data_change */ + &[true, true, true], /* selection_vector */ + true, /* expected_error */ + )] + fn append_only_rejects_data_removal_when_data_change( + #[values(DataRemoval::RemoveFile, DataRemoval::DeletionVectorUpdate)] removal: DataRemoval, + #[values(0, 1)] batch_index: usize, + #[case] append_only: bool, + #[case] data_change: bool, + #[case] selection_vector: &[bool], + #[case] expected_error: bool, + ) -> DeltaResult<()> { + let (_engine, mut txn, _tempdir) = create_existing_table_txn()?; + set_append_only(&mut txn, append_only)?; + txn.set_data_change(data_change); + for index in 0..2 { + let selection_vector = if index == batch_index { + selection_vector + } else { + &[false, false, false] + }; + stage_data_removal(&mut txn, removal, selection_vector); + } + + let result = txn.validate_append_only_semantics(); + if expected_error { + assert!(matches!(result, Err(Error::InvalidTransactionState(_)))); + } else { + result?; + } + Ok(()) + } + #[test] fn test_validate_blind_append_requires_adds() -> DeltaResult<()> { let (_engine, mut txn, _tempdir) = create_existing_table_txn()?; @@ -1762,12 +3003,8 @@ mod tests { #[test] fn test_validate_blind_append_rejects_create_table() -> DeltaResult<()> { let tempdir = tempfile::tempdir()?; - let schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "id", - DataType::INTEGER, - )])?); - let store = Arc::new(LocalFileSystem::new()); - let engine = Arc::new(crate::engine::default::DefaultEngineBuilder::new(store).build()); + let schema = schema_ref! { nullable "id": INTEGER }; + let engine = Arc::new(crate::engine::sync::SyncEngine::new()); let mut txn = create_table( tempdir.path().to_str().expect("valid temp path"), schema, @@ -1830,7 +3067,7 @@ mod tests { // Note: Additional test coverage for partial file matching (where some files in a scan // have DV updates but others don't) is provided by the end-to-end integration test - // kernel/tests/dv.rs and kernel/tests/write.rs, which exercises + // kernel/tests/features/dv.rs and kernel/tests/write/remove_dv.rs, which exercise // the full deletion vector write workflow including the DvMatchVisitor logic. #[test] @@ -1889,9 +3126,9 @@ mod tests { #[case] schema: SchemaRef, #[case] mode: ColumnMappingMode, ) -> DeltaResult<()> { - let (_engine, txn) = crate::utils::test_utils::setup_column_mapping_txn(schema, mode)?; - let write_context = txn.get_write_context(); - crate::utils::test_utils::validate_physical_schema_column_mapping( + let (_engine, txn) = crate::unit_test_utils::setup_column_mapping_txn(schema, mode)?; + let write_context = txn.unpartitioned_write_context().unwrap(); + crate::unit_test_utils::validate_physical_schema_column_mapping( write_context.logical_schema(), write_context.physical_schema(), mode, @@ -1899,84 +3136,46 @@ mod tests { Ok(()) } - /// Builds a RecordBatch with logical field names matching [`test_schema_nested`]. - fn build_test_record_batch() -> DeltaResult { - let arrow_schema: ArrowSchema = test_schema_nested().as_ref().try_into_arrow()?; - - let id_arr: ArrayRef = Arc::new(Int64Array::from(vec![1i64, 2])); - - // info struct fields - let name_arr: ArrayRef = Arc::new(StringArray::from(vec!["alice", "bob"])); - let age_arr: ArrayRef = Arc::new(Int32Array::from(vec![30, 25])); - - // info.tags: Map - let keys = StringArray::from(vec!["k1", "k2"]); - let vals = StringArray::from(vec!["v1", "v2"]); - let entries_field = ArrowField::new( - "key_value", - ArrowDataType::Struct( - vec![ - ArrowField::new("key", ArrowDataType::Utf8, false), - ArrowField::new("value", ArrowDataType::Utf8, true), - ] - .into(), - ), - false, - ); - let entries = StructArray::try_new( - vec![ - ArrowField::new("key", ArrowDataType::Utf8, false), - ArrowField::new("value", ArrowDataType::Utf8, true), - ] - .into(), - vec![Arc::new(keys), Arc::new(vals)], - None, - )?; - let map_offsets = crate::arrow::buffer::OffsetBuffer::new(vec![0i32, 1, 2].into()); - let tags_arr: ArrayRef = Arc::new(MapArray::new( - Arc::new(entries_field), - map_offsets, - entries, - None, - false, - )); - - // info.scores: Array - let score_values = Int32Array::from(vec![10, 20, 30]); - let offsets = crate::arrow::buffer::OffsetBuffer::new(vec![0i32, 2, 3].into()); - let scores_arr: ArrayRef = Arc::new(ListArray::try_new( - Arc::new(ArrowField::new("element", ArrowDataType::Int32, true)), - offsets, - Arc::new(score_values), - None, - )?); - - // info struct + /// Builds two-row [`EngineData`] with logical field names matching [`test_schema_nested`]. + fn build_test_record_batch() -> DeltaResult> { + let schema = test_schema_nested(); + let tag_type = MapType::new(DataType::STRING, DataType::STRING, true); + let score_type = ArrayType::new(DataType::INTEGER, true); let info_fields = vec![ - ArrowField::new("name", ArrowDataType::Utf8, true), - ArrowField::new("age", ArrowDataType::Int32, true), - ArrowField::new("tags", tags_arr.data_type().clone(), true), - ArrowField::new("scores", scores_arr.data_type().clone(), true), + StructField::nullable("name", DataType::STRING), + StructField::nullable("age", DataType::INTEGER), + StructField::nullable("tags", tag_type.clone()), + StructField::nullable("scores", score_type.clone()), ]; - let info_arr: ArrayRef = Arc::new(StructArray::try_new( - info_fields.into(), - vec![name_arr, age_arr, tags_arr, scores_arr], - None, + let info1 = Scalar::Struct(StructData::try_new( + info_fields.clone(), + vec![ + "alice".into(), + 30i32.into(), + Scalar::Map(MapData::try_new(tag_type.clone(), [("k1", "v1")])?), + Scalar::Array(ArrayData::try_new(score_type.clone(), [10i32, 20i32])?), + ], )?); - - Ok(RecordBatch::try_new( - Arc::new(arrow_schema), - vec![id_arr, info_arr], - )?) + let info2 = Scalar::Struct(StructData::try_new( + info_fields, + vec![ + "bob".into(), + 25i32.into(), + Scalar::Map(MapData::try_new(tag_type, [("k2", "v2")])?), + Scalar::Array(ArrayData::try_new(score_type, [30i32])?), + ], + )?); + ArrowEvaluationHandler.create_many(schema, &[&[1i64.into(), info1], &[2i64.into(), info2]]) } - /// Validates that [`WriteContext::logical_to_physical`] correctly renames fields at all nesting levels. - /// Builds a RecordBatch with logical names, evaluates the transform, and checks that the - /// output uses physical names from the physical schema — including nested struct children. + /// Validates that [`WriteContext::logical_to_physical`] correctly renames fields at all nesting + /// levels. Builds a RecordBatch with logical names, evaluates the transform, and checks + /// that the output uses physical names from the physical schema — including nested struct + /// children. fn validate_logical_to_physical_transform(mode: ColumnMappingMode) -> DeltaResult<()> { let schema = test_schema_nested(); - let (_engine, txn) = crate::utils::test_utils::setup_column_mapping_txn(schema, mode)?; - let write_context = txn.get_write_context(); + let (_engine, txn) = crate::unit_test_utils::setup_column_mapping_txn(schema, mode)?; + let write_context = txn.unpartitioned_write_context().unwrap(); let logical_schema = write_context.logical_schema(); let physical_schema = write_context.physical_schema(); let logical_to_physical_expression = write_context.logical_to_physical(); @@ -1988,7 +3187,7 @@ mod tests { ); } - let batch = build_test_record_batch()?; + let data = build_test_record_batch()?; // Evaluate the logical_to_physical expression let input_schema: SchemaRef = logical_schema.clone(); @@ -1998,7 +3197,7 @@ mod tests { logical_to_physical_expression.clone(), physical_schema.clone().into(), )?; - let result = evaluator.evaluate(&ArrowEngineData::new(batch))?; + let result = evaluator.evaluate(data.as_ref())?; let result = ArrowEngineData::try_from_engine_data(result)?; let result_batch = result.record_batch(); @@ -2025,32 +3224,6 @@ mod tests { validate_logical_to_physical_transform(mode) } - #[rstest] - #[case::dropped("./tests/data/basic_partitioned/", 2, &[])] - #[case::kept("./tests/data/partitioned_with_materialize_feature/", 3, &["letter"])] - fn test_partition_column_in_eval_output( - #[case] table_path: &str, - #[case] expected_cols: usize, - #[case] expected_partition_cols: &[&str], - ) -> Result<(), Box> { - use crate::arrow::array::Float64Array; - let (_snap, wc) = snapshot_and_write_context(table_path)?; - let batch = RecordBatch::try_new( - Arc::new(wc.logical_schema().as_ref().try_into_arrow()?), - vec![ - Arc::new(StringArray::from(vec!["x"])) as ArrayRef, - Arc::new(Int64Array::from(vec![42])), - Arc::new(Float64Array::from(vec![1.5])), - ], - )?; - let rb = eval_logical_to_physical(&wc, batch)?; - assert_eq!(rb.num_columns(), expected_cols); - for col in expected_partition_cols { - assert!(rb.schema().fields().iter().any(|f| f.name() == *col)); - } - Ok(()) - } - // ========================================================================= // Stats validation tests for clustering columns // ========================================================================= @@ -2067,157 +3240,86 @@ mod tests { /// Creates test add file metadata with configurable stats for the "value" column. fn create_test_add_files(paths: Vec<&str>, stats: Vec) -> Box { - let path_array = StringArray::from(paths.to_vec()); - let size_array = Int64Array::from(vec![1024i64; paths.len()]); - let mod_time_array = Int64Array::from(vec![1000000i64; paths.len()]); - - // Create stats struct with full structure for "value" column (matches test table schema) - let value_field = Arc::new(ArrowField::new("value", ArrowDataType::Int64, true)); - - let num_records: Vec> = stats - .iter() - .map(|s| match s { - TestFileStats::None => Option::None, - _ => Some(100), - }) - .collect(); - let null_count_values: Vec> = stats - .iter() - .map(|s| match s { - TestFileStats::None => Option::None, - TestFileStats::Present => Some(0), - TestFileStats::AllNull => Some(100), - }) - .collect(); - let min_values: Vec> = stats - .iter() - .map(|s| match s { - TestFileStats::Present => Some(1), - _ => Option::None, - }) - .collect(); - let max_values: Vec> = stats - .iter() - .map(|s| match s { - TestFileStats::Present => Some(100), - _ => Option::None, - }) - .collect(); - - let num_records_array = Int64Array::from(num_records); - let null_count_array = Int64Array::from(null_count_values); - let null_count_struct = StructArray::new( - Fields::from(vec![value_field.clone()]), - vec![Arc::new(null_count_array) as ArrayRef], - None, - ); - let min_values_array = Int64Array::from(min_values); - let min_values_struct = StructArray::new( - Fields::from(vec![value_field.clone()]), - vec![Arc::new(min_values_array) as ArrayRef], - None, - ); - let max_values_array = Int64Array::from(max_values); - let max_values_struct = StructArray::new( - Fields::from(vec![value_field]), - vec![Arc::new(max_values_array) as ArrayRef], - None, - ); - - // Build stats struct fields - let value_struct_type = ArrowDataType::Struct(Fields::from(vec![ArrowField::new( - "value", - ArrowDataType::Int64, - true, - )])); - let stats_fields = Fields::from(vec![ - ArrowField::new("numRecords", ArrowDataType::Int64, true), - ArrowField::new("nullCount", value_struct_type.clone(), true), - ArrowField::new("minValues", value_struct_type.clone(), true), - ArrowField::new("maxValues", value_struct_type, true), + let value_fields = vec![StructField::nullable("value", DataType::LONG)]; + let value_struct_type = DataType::struct_type_unchecked(value_fields.clone()); + let stats_type = DataType::struct_type_unchecked(vec![ + StructField::nullable(NUM_RECORDS, DataType::LONG), + StructField::nullable(NULL_COUNT, value_struct_type.clone()), + StructField::nullable(MIN_VALUES, value_struct_type.clone()), + StructField::nullable(MAX_VALUES, value_struct_type.clone()), ]); - - // Create validity bitmap - stats struct is null when stats are absent - let stats_validity: Vec = stats - .iter() - .map(|s| !matches!(s, TestFileStats::None)) - .collect(); - let stats_struct = StructArray::new( - stats_fields.clone(), - vec![ - Arc::new(num_records_array) as ArrayRef, - Arc::new(null_count_struct) as ArrayRef, - Arc::new(min_values_struct) as ArrayRef, - Arc::new(max_values_struct) as ArrayRef, - ], - Some(stats_validity.into()), - ); - - // Create empty partition values map - let entries_field = Arc::new(ArrowField::new( - "key_value", - ArrowDataType::Struct( - vec![ - Arc::new(ArrowField::new("key", ArrowDataType::Utf8, false)), - Arc::new(ArrowField::new("value", ArrowDataType::Utf8, true)), - ] - .into(), - ), - false, - )); - let empty_keys = StringArray::from(Vec::<&str>::new()); - let empty_values = StringArray::from(Vec::>::new()); - let empty_entries = StructArray::from(vec![ - ( - Arc::new(ArrowField::new("key", ArrowDataType::Utf8, false)), - Arc::new(empty_keys) as ArrayRef, - ), - ( - Arc::new(ArrowField::new("value", ArrowDataType::Utf8, true)), - Arc::new(empty_values) as ArrayRef, - ), - ]); - let offsets = OffsetBuffer::from_lengths(vec![0; paths.len()]); - let partition_values = MapArray::new(entries_field, offsets, empty_entries, None, false); - - let schema = Arc::new(ArrowSchema::new(vec![ - ArrowField::new("path", ArrowDataType::Utf8, false), - ArrowField::new( + let stats_fields = vec![ + StructField::nullable(NUM_RECORDS, DataType::LONG), + StructField::nullable(NULL_COUNT, value_struct_type.clone()), + StructField::nullable(MIN_VALUES, value_struct_type.clone()), + StructField::nullable(MAX_VALUES, value_struct_type), + ]; + let schema = Arc::new(StructType::new_unchecked(vec![ + StructField::not_null("path", DataType::STRING), + StructField::not_null( "partitionValues", - ArrowDataType::Map( - Arc::new(ArrowField::new( - "key_value", - ArrowDataType::Struct( - vec![ - Arc::new(ArrowField::new("key", ArrowDataType::Utf8, false)), - Arc::new(ArrowField::new("value", ArrowDataType::Utf8, true)), - ] - .into(), - ), - false, - )), - false, - ), - false, + MapType::new(DataType::STRING, DataType::STRING, true), ), - ArrowField::new("size", ArrowDataType::Int64, false), - ArrowField::new("modificationTime", ArrowDataType::Int64, false), - ArrowField::new("stats", ArrowDataType::Struct(stats_fields), true), + StructField::not_null("size", DataType::LONG), + StructField::not_null("modificationTime", DataType::LONG), + StructField::nullable("stats", stats_type.clone()), ])); - let batch = RecordBatch::try_new( - schema, - vec![ - Arc::new(path_array), - Arc::new(partition_values), - Arc::new(size_array), - Arc::new(mod_time_array), - Arc::new(stats_struct), - ], - ) - .unwrap(); + let empty_map = Scalar::Map( + MapData::try_new( + MapType::new(DataType::STRING, DataType::STRING, true), + Vec::<(&str, &str)>::new(), + ) + .unwrap(), + ); - Box::new(ArrowEngineData::new(batch)) + let rows: Vec> = paths + .iter() + .zip(stats.iter()) + .map(|(path, stat)| { + let stats_scalar = match stat { + TestFileStats::None => Scalar::Null(stats_type.clone()), + TestFileStats::Present | TestFileStats::AllNull => { + let value_struct = |v: Option| { + let scalar = v.map_or(Scalar::Null(DataType::LONG), |n| n.into()); + Scalar::Struct( + StructData::try_new(value_fields.clone(), vec![scalar]).unwrap(), + ) + }; + let (null_count, min, max) = match stat { + TestFileStats::Present => ( + value_struct(Some(0)), + value_struct(Some(1)), + value_struct(Some(100)), + ), + _ => ( + value_struct(Some(100)), + value_struct(None), + value_struct(None), + ), + }; + Scalar::Struct( + StructData::try_new( + stats_fields.clone(), + vec![100i64.into(), null_count, min, max], + ) + .unwrap(), + ) + } + }; + vec![ + (*path).into(), + empty_map.clone(), + 1024i64.into(), + 1000000i64.into(), + stats_scalar, + ] + }) + .collect(); + let row_refs: Vec<&[Scalar]> = rows.iter().map(|r| r.as_slice()).collect(); + ArrowEvaluationHandler + .create_many(schema, &row_refs) + .unwrap() } #[test] @@ -2227,7 +3329,7 @@ mod tests { .transaction(Box::new(FileSystemCommitter::new()), &engine) .unwrap() .with_operation("WRITE".to_string()) - .with_clustering_columns_for_test(vec![ColumnName::new(["value"])]); + .with_clustering_columns_for_test(vec![column_name!("value")]); let add_files = create_test_add_files(vec!["file1.parquet"], vec![TestFileStats::AllNull]); @@ -2247,7 +3349,7 @@ mod tests { .unwrap() .with_operation("WRITE".to_string()) // Enable clustering columns for this test - .with_clustering_columns_for_test(vec![ColumnName::new(["value"])]); + .with_clustering_columns_for_test(vec![column_name!("value")]); // Add files WITHOUT stats let add_files = create_test_add_files(vec!["file1.parquet"], vec![TestFileStats::None]); @@ -2275,7 +3377,7 @@ mod tests { .unwrap() .with_operation("WRITE".to_string()) // Enable clustering columns for this test - .with_clustering_columns_for_test(vec![ColumnName::new(["value"])]); + .with_clustering_columns_for_test(vec![column_name!("value")]); // Add files WITH stats let add_files = create_test_add_files(vec!["file1.parquet"], vec![TestFileStats::Present]); @@ -2309,4 +3411,216 @@ mod tests { "Stats validation should be skipped without clustering, got: {result:?}" ); } + + #[test] + fn disallow_catalog_committer_for_non_catalog_managed_table() { + let storage = Arc::new(InMemory::new()); + let table_root = url::Url::parse("memory:///").unwrap(); + let engine = crate::engine::sync::SyncEngine::new_with_store(storage.clone()); + + // Create a non-catalog-managed table (no catalogManaged feature) + let actions = [ + r#"{"commitInfo":{"timestamp":12345678900,"inCommitTimestamp":12345678900}}"#, + r#"{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":[],"writerFeatures":["inCommitTimestamp"]}}"#, + r#"{"metaData":{"id":"test-id","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[]}","partitionColumns":[],"configuration":{"delta.enableInCommitTimestamps":"true"},"createdTime":1234567890}}"#, + ].join("\n"); + + let commit_path = Path::from("_delta_log/00000000000000000000.json"); + let rt = tokio::runtime::Runtime::new().unwrap(); + rt.block_on(storage.put(&commit_path, actions.into())) + .unwrap(); + + let snapshot = Snapshot::builder_for(table_root).build(&engine).unwrap(); + + // Try to commit with a catalog committer to a non-catalog-managed table + let committer = Box::new(MockCatalogCommitter); + let err = snapshot + .transaction(committer, &engine) + .unwrap() + .commit(&engine) + .unwrap_err(); + assert!(matches!( + err, + crate::Error::Generic(e) if e.contains("This table is path-based and cannot be committed to with a catalog committer") + )); + } + + #[test] + fn disallow_catalog_committer_for_non_catalog_managed_create_table() { + let storage = Arc::new(InMemory::new()); + let engine = crate::engine::sync::SyncEngine::new_with_store(storage); + + // Create a non-catalog-managed table using a catalog committer + let schema = Arc::new(crate::schema::StructType::new_unchecked(vec![ + crate::schema::StructField::new("id", crate::schema::DataType::INTEGER, true), + ])); + let committer = Box::new(MockCatalogCommitter); + let err = create_table("memory:///", schema, "test-engine") + .build(&engine, committer) + .unwrap() + .commit(&engine) + .unwrap_err(); + assert!(matches!( + err, + crate::Error::Generic(e) if e.contains("This table is path-based and cannot be committed to with a catalog committer") + )); + } + + struct CapturingCommitter { + captured: Arc>>, + } + + impl CapturingCommitter { + fn new() -> (Self, Arc>>) { + let captured = Arc::new(Mutex::new(None)); + ( + Self { + captured: captured.clone(), + }, + captured, + ) + } + } + + impl Committer for CapturingCommitter { + fn commit( + &self, + _engine: &dyn Engine, + _actions: DeltaResultIterator<'_, FilteredEngineData>, + commit_metadata: CommitMetadata, + ) -> DeltaResult { + *self.captured.lock().unwrap() = Some(commit_metadata.in_commit_timestamp()); + Ok(CommitResponse::Conflict { + version: commit_metadata.version(), + }) + } + fn is_catalog_committer(&self) -> bool { + false + } + fn publish( + &self, + _engine: &dyn Engine, + _publish_metadata: PublishMetadata, + ) -> DeltaResult<()> { + Ok(()) + } + } + + #[test] + fn test_commit_metadata_receives_ict_not_wall_time() -> DeltaResult<()> { + // Set up a table with ICT enabled and a very high previous ICT so that the + // monotonicity rule (max(wall_time, prev_ict + 1)) produces a value strictly + // greater than the current wall time. This lets us verify the computed ICT is + // passed to CommitMetadata (not the wall-clock timestamp). + let tempdir = tempfile::tempdir().unwrap(); + let log_dir = tempdir.path().join("_delta_log"); + std::fs::create_dir_all(&log_dir).unwrap(); + + let future_ict: i64 = 9_999_999_999_999; // far-future timestamp in ms + let commit_info = serde_json::json!({ + "commitInfo": { + "timestamp": 1000, + "operation": "WRITE", + "inCommitTimestamp": future_ict + } + }); + let protocol = serde_json::json!({ + "protocol": { + "minReaderVersion": 3, + "minWriterVersion": 7, + "readerFeatures": [], + "writerFeatures": ["inCommitTimestamp"] + } + }); + let schema_json = serde_json::json!({ + "type": "struct", + "fields": [{ + "name": "id", + "type": "integer", + "nullable": true, + "metadata": {} + }] + }); + let metadata = serde_json::json!({ + "metaData": { + "id": "test-id", + "format": {"provider": "parquet", "options": {}}, + "schemaString": schema_json.to_string(), + "partitionColumns": [], + "configuration": { + "delta.enableInCommitTimestamps": "true" + } + } + }); + let commit0 = format!("{commit_info}\n{protocol}\n{metadata}\n"); + std::fs::write(log_dir.join("00000000000000000000.json"), commit0).unwrap(); + + let table_url = Url::from_directory_path(tempdir.path()).unwrap(); + let engine = SyncEngine::new(); + let snapshot = Snapshot::builder_for(table_url).build(&engine)?; + + let prev_ict = snapshot.get_in_commit_timestamp(&engine)?; + assert_eq!(prev_ict, Some(future_ict)); + + let (committer, captured_ts) = CapturingCommitter::new(); + let mut txn = snapshot.transaction(Box::new(committer), &engine)?; + add_dummy_file(&mut txn); + + let result = txn.commit(&engine)?; + assert!( + matches!(result, CommitResult::ConflictedTransaction(_)), + "Expected ConflictedTransaction from capturing committer" + ); + + // The ICT in CommitMetadata must be prev_ict + 1 (monotonicity), NOT the wall time. + let captured = captured_ts + .lock() + .unwrap() + .expect("should have captured a timestamp"); + assert_eq!( + captured, + future_ict + 1, + "CommitMetadata.in_commit_timestamp should be the computed ICT (prev_ict + 1), \ + not the wall-clock time" + ); + Ok(()) + } + + // ===== Commit failure-metric tests ===== + + fn commit_failure_event(reporter: &CapturingReporter) -> Option { + reporter.events().into_iter().find_map(|event| match event { + MetricEvent::TransactionCommitFailure(f) => Some(f), + _ => None, + }) + } + + #[test] + fn test_commit_io_error_emits_retryable_io_failure_metric() -> DeltaResult<()> { + let (engine, snapshot, _tempdir) = load_test_table("table-without-dv-small")?; + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + let mut txn = snapshot.transaction(Box::new(IoErrorCommitter), engine.as_ref())?; + add_dummy_file(&mut txn); + let result = txn.commit(engine.as_ref())?; + assert!(matches!(result, CommitResult::RetryableTransaction(_))); + let failure = commit_failure_event(&reporter).expect("commit failure event"); + assert_eq!(failure.reason, CommitFailureReason::RetryableIo); + assert_eq!(failure.table_type, TableType::PathBased); + Ok(()) + } + + #[test] + fn test_commit_terminal_error_emits_error_failure_metric() -> DeltaResult<()> { + let (engine, snapshot, _tempdir) = load_test_table("table-without-dv-small")?; + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + let mut txn = snapshot.transaction(Box::new(GenericErrorCommitter), engine.as_ref())?; + add_dummy_file(&mut txn); + assert!(txn.commit(engine.as_ref()).is_err()); + let failure = commit_failure_event(&reporter).expect("commit failure event"); + assert_eq!(failure.reason, CommitFailureReason::Error); + assert_eq!(failure.table_type, TableType::PathBased); + Ok(()) + } } diff --git a/kernel/src/transaction/schema_evolution.rs b/kernel/src/transaction/schema_evolution.rs new file mode 100644 index 0000000000..835e7137d9 --- /dev/null +++ b/kernel/src/transaction/schema_evolution.rs @@ -0,0 +1,784 @@ +//! Schema evolution operations for ALTER TABLE. +//! +//! This module defines the [`SchemaOperation`] enum and the [`apply_schema_operations`] function +//! that validates and applies schema changes to produce an evolved schema. + +use std::cmp::Ordering; + +use indexmap::IndexMap; + +use crate::error::Error; +use crate::expressions::ColumnName; +use crate::schema::validation::validate_schema; +use crate::schema::{DataType, SchemaRef, StructField, StructType}; +use crate::table_features::{ + find_max_column_id_in_schema, try_assign_flat_column_mapping_info, validate_column_mapping_id, + ColumnMappingMode, +}; +use crate::DeltaResult; + +/// A schema evolution operation to be applied during ALTER TABLE. +/// +/// Operations are validated and applied in order during +/// [`apply_schema_operations`]. Each operation sees the schema state after all prior operations +/// have been applied. +#[derive(Debug, Clone)] +pub(crate) enum SchemaOperation { + /// Add a top-level column. + AddColumn { field: StructField }, + + /// Change a column's nullability from NOT NULL to nullable. + SetNullable { column: ColumnName }, +} + +// Helper to modify a nested column. For each component in `path`, locates the matching field +// (case-insensitive), then descends into the next nested struct. At the leaf, calls `modifier` +// to mutate the field in place. +// +// `modifier` is expected to mutate the field's nullability, metadata, or `data_type` -- but +// not its name. Renames need additional handling (IndexMap re-keying + sibling-conflict check) +// that downstream PRs will introduce alongside the rename caller. +// +// Returns an error if a field in the path does not exist or an intermediate field is not a struct. +// +// Example: +// fields = [ id: int not null, address: struct { city: string not null, zip: string } ] +// path = ["address", "city"] +// modifier = |f| { f.nullable = true; Ok(()) } +// yields: +// [ id: int not null, address: struct { city: string, zip: string } ] +fn modify_field_at_path( + fields: &mut IndexMap, + path: &[String], + modifier: &dyn Fn(&mut StructField) -> DeltaResult<()>, +) -> DeltaResult<()> { + let (first, rest) = path + .split_first() + .ok_or_else(|| Error::generic("empty column path"))?; + + // Delta column names are case-insensitive. + let lowered = first.to_lowercase(); + let idx = fields + .iter() + .position(|(_, f)| f.name().to_lowercase() == lowered) + .ok_or_else(|| Error::generic(format!("field '{first}' does not exist")))?; + + if !rest.is_empty() { + let (_, field) = fields + .get_index_mut(idx) + .ok_or_else(|| Error::internal_error("idx from position() invalid"))?; + let DataType::Struct(inner) = &mut field.data_type else { + return Err(Error::generic(format!( + "intermediate field '{first}' is not a struct" + ))); + }; + return modify_field_at_path(inner.field_map_mut(), rest, modifier); + } + + // === Leaf handling === + let (_, field) = fields + .get_index_mut(idx) + .ok_or_else(|| Error::internal_error("idx from position() invalid"))?; + modifier(field) +} + +/// The result of applying schema operations. +#[derive(Debug)] +pub(crate) struct SchemaEvolutionResult { + /// The evolved schema after all operations are applied. + pub schema: SchemaRef, + /// If `Some(id)`, `delta.columnMapping.maxColumnId` must be updated to this new value. + /// `None` means the property should remain unchanged. + pub new_max_column_id: Option, +} + +/// Applies a sequence of schema operations to the given schema, returning a +/// [`SchemaEvolutionResult`] (see the struct's docs for details). +/// +/// Operations are applied sequentially: each one validates against and modifies the schema +/// produced by all preceding operations, not the original input schema. +/// +/// # Errors +/// +/// Returns an error if any operation fails validation. The error message identifies which +/// operation failed and why. +pub(crate) fn apply_schema_operations( + mut schema: StructType, + operations: Vec, + column_mapping_mode: ColumnMappingMode, + current_max_column_id: Option, +) -> DeltaResult { + let cm_enabled = column_mapping_mode != ColumnMappingMode::None; + + // Reject a persisted seed that already violates the protocol's 32-bit non-negative bound + // before it propagates into the allocator. A negative or above-i32::MAX seed almost + // certainly indicates a non-conforming writer; bail out rather than silently letting the + // allocator either trip the per-field validator on every preserved sibling or produce + // out-of-range fresh ids. + if let Some(seed) = current_max_column_id { + validate_column_mapping_id(seed).map_err(|e| { + Error::invalid_protocol(format!( + "Table property `delta.columnMapping.maxColumnId`: {e}" + )) + })?; + } + + // When column mapping is enabled and the property is set, defensively take the max with + // the schema's actual max in case a non-conforming writer left the property stale. + let mut max_id = if cm_enabled { + current_max_column_id.map(|cfg| cfg.max(find_max_column_id_in_schema(&schema).unwrap_or(0))) + } else { + current_max_column_id + }; + + for op in operations { + match op { + // Protocol feature checks for the field's data type (e.g. `timestampNtz`) happen + // later when the caller builds a new TableConfiguration from the evolved schema -- + // the alter is rejected if the table doesn't already have the required feature + // enabled. This matches Spark, which also rejects with + // `DELTA_FEATURES_REQUIRE_MANUAL_ENABLEMENT` and requires the user to enable the + // feature explicitly before adding such a column. + SchemaOperation::AddColumn { field } => { + if field.is_metadata_column() { + return Err(Error::schema(format!( + "Cannot add column '{}': metadata columns are not allowed in \ + a table schema", + field.name() + ))); + } + if !matches!(field.data_type, DataType::Primitive(_)) { + StructType::ensure_no_metadata_columns_in_field(&field)?; + } + // Case-insensitive sibling-conflict check (O(N) per AddColumn). + let lowered = field.name().to_lowercase(); + if schema.fields().any(|f| f.name().to_lowercase() == lowered) { + return Err(Error::schema(format!( + "Cannot add column '{}': a column with that name already exists", + field.name() + ))); + } + // Validate field is nullable (Delta protocol requires added columns to be + // nullable so existing data files can return NULL for the new column) + // NOTE: non-nullable columns depend on invariants feature + if !field.is_nullable() { + return Err(Error::schema(format!( + "Cannot add non-nullable column '{}'. Added columns must be nullable \ + because existing data files do not contain this column.", + field.name() + ))); + } + let field = if cm_enabled { + let id = max_id.as_mut().ok_or_else(|| { + Error::invalid_protocol( + "Column mapping is enabled but delta.columnMapping.maxColumnId \ + is not set in table properties", + ) + })?; + // ALTER TABLE doesn't support icebergCompatV3 yet, so the new field never + // gets `delta.columnMapping.nested.ids` here. Tracking issue: + // + try_assign_flat_column_mapping_info(&field, id)? + } else { + // Stray CM metadata on a mapping-disabled table is handled by the AlterTable + // builder, which strips annotations this ALTER newly introduces from the + // evolved schema. + field + }; + schema.field_map_mut().insert(field.name().clone(), field); + } + SchemaOperation::SetNullable { column } => { + modify_field_at_path(schema.field_map_mut(), column.path(), &|f| { + f.nullable = true; + Ok(()) + }) + .map_err(|e| { + Error::generic(format!("Cannot set nullable on column '{column}': {e}")) + })?; + } + } + } + + validate_schema(&schema, column_mapping_mode)?; + + // `max_id` is only ever incremented by `try_assign_flat_column_mapping_info`. If it grew, + // the new value must be persisted; if it went backwards, that's a bug. + let new_max_column_id = match max_id.cmp(¤t_max_column_id) { + Ordering::Greater => max_id, + Ordering::Equal => None, + Ordering::Less => { + return Err(Error::internal_error( + "max column ID went backwards during schema evolution", + )) + } + }; + Ok(SchemaEvolutionResult { + schema: schema.into(), + new_max_column_id, + }) +} + +#[cfg(test)] +mod tests { + use std::collections::HashSet; + + use rstest::rstest; + + use super::*; + use crate::expressions::{column_name, ColumnName}; + use crate::schema::{ + schema, ArrayType, ColumnMetadataKey, DataType, MapType, MetadataColumnSpec, MetadataValue, + StructField, StructType, + }; + + fn simple_schema() -> StructType { + StructType::try_new(vec![ + StructField::not_null("id", DataType::INTEGER), + StructField::nullable("name", DataType::STRING), + ]) + .unwrap() + } + + fn add_col(name: &str, nullable: bool) -> SchemaOperation { + let field = if nullable { + StructField::nullable(name, DataType::STRING) + } else { + StructField::not_null(name, DataType::STRING) + }; + SchemaOperation::AddColumn { field } + } + + // Builds a struct column whose nested leaf field has the given name. Used to prove that + // `validate_schema` (not just the top-level dup check or `StructType::try_new`) is + // reached from `apply_schema_operations`. + fn add_struct_with_nested_leaf(name: &str, leaf_name: &str) -> SchemaOperation { + let inner = + StructType::try_new(vec![StructField::nullable(leaf_name, DataType::STRING)]).unwrap(); + SchemaOperation::AddColumn { + field: StructField::nullable(name, inner), + } + } + + fn nested_schema() -> StructType { + StructType::try_new(vec![ + StructField::not_null("id", DataType::INTEGER), + StructField::nullable( + "address", + StructType::try_new(vec![ + StructField::not_null("city", DataType::STRING), + StructField::nullable("zip", DataType::STRING), + ]) + .unwrap(), + ), + ]) + .unwrap() + } + + // === modify_field_at_path tests === + + // Convert a StructType into the IndexMap shape that + // `modify_field_at_path` operates on. + fn into_field_map(schema: StructType) -> IndexMap { + schema + .into_fields() + .map(|f| (f.name().clone(), f)) + .collect() + } + + fn set_nullable_modifier(f: &mut StructField) -> DeltaResult<()> { + f.nullable = true; + Ok(()) + } + + fn modify_field_at_path_test_helper( + schema: StructType, + path: &[String], + ) -> DeltaResult> { + let mut fields = into_field_map(schema); + modify_field_at_path(&mut fields, path, &set_nullable_modifier)?; + Ok(fields) + } + + #[test] + fn modify_top_level_field_sets_nullable() { + let path = vec!["id".to_string()]; + let result = modify_field_at_path_test_helper(simple_schema(), &path).unwrap(); + let id = result.values().find(|f| f.name() == "id").unwrap(); + assert!(id.is_nullable()); + } + + #[test] + fn modify_nested_field_modifies_only_leaf() { + let path = vec!["address".to_string(), "city".to_string()]; + let result = modify_field_at_path_test_helper(nested_schema(), &path).unwrap(); + let addr = result.values().find(|f| f.name() == "address").unwrap(); + match addr.data_type() { + DataType::Struct(s) => assert!(s.field("city").unwrap().is_nullable()), + other => panic!("Expected Struct, got: {other:?}"), + } + } + + /// Modifying one nested leaf (`address.city`) must not touch any other field. + /// Guards against the recursive rebuild accidentally replacing siblings when it reconstructs + /// the enclosing struct. + #[test] + fn modify_nested_leaf_preserves_other_fields() { + let path = vec!["address".to_string(), "city".to_string()]; + let result = modify_field_at_path_test_helper(nested_schema(), &path).unwrap(); + let id = result.values().find(|f| f.name() == "id").unwrap(); + assert!(!id.is_nullable()); + let addr = result.values().find(|f| f.name() == "address").unwrap(); + match addr.data_type() { + DataType::Struct(s) => assert!(s.field("zip").unwrap().is_nullable()), + other => panic!("Expected Struct, got: {other:?}"), + } + } + + #[test] + fn modify_nonexistent_field_fails() { + let path = vec!["nope".to_string()]; + let err = modify_field_at_path_test_helper(simple_schema(), &path).unwrap_err(); + assert!(err.to_string().contains("does not exist")); + } + + /// A path that descends into a non-struct intermediate field (here: `name.inner`, where + /// `name` is a STRING, not a struct) must error rather than silently succeed or panic. + #[test] + fn modify_through_non_struct_fails() { + let path = vec!["name".to_string(), "inner".to_string()]; + let err = modify_field_at_path_test_helper(simple_schema(), &path).unwrap_err(); + assert!(err.to_string().contains("not a struct")); + } + + #[test] + fn modify_case_insensitive_lookup_finds_field() { + let path = vec!["ID".to_string()]; + let result = modify_field_at_path_test_helper(simple_schema(), &path).unwrap(); + let id = result.values().find(|f| f.name() == "id").unwrap(); + assert!(id.is_nullable()); + } + + // === apply_schema_operations tests === + + #[rstest] + #[case::dup_exact(vec![add_col("name", true)], "already exists")] + #[case::dup_case_insensitive(vec![add_col("Name", true)], "already exists")] + #[case::dup_within_batch( + vec![add_col("email", true), add_col("email", true)], + "already exists" + )] + #[case::non_nullable(vec![add_col("age", false)], "non-nullable")] + #[case::invalid_parquet_char(vec![add_col("foo,bar", true)], "invalid character")] + #[case::nested_invalid_parquet_char( + vec![add_struct_with_nested_leaf("addr", "bad,leaf")], + "invalid character" + )] + #[case::metadata_column( + vec![SchemaOperation::AddColumn { + field: StructField::create_metadata_column("row_idx", MetadataColumnSpec::RowIndex), + }], + "metadata columns are not allowed" + )] + fn apply_schema_operations_rejects( + #[case] ops: Vec, + #[case] error_contains: &str, + ) { + let err = apply_schema_operations(simple_schema(), ops, ColumnMappingMode::None, None) + .unwrap_err(); + assert!(err.to_string().contains(error_contains)); + } + + #[rstest] + #[case::single(vec![add_col("email", true)], &["id", "name", "email"])] + #[case::multiple( + vec![add_col("email", true), add_col("age", true)], + &["id", "name", "email", "age"] + )] + fn apply_schema_operations_succeeds( + #[case] ops: Vec, + #[case] expected_names: &[&str], + ) { + let result = + apply_schema_operations(simple_schema(), ops, ColumnMappingMode::None, None).unwrap(); + let actual: Vec<&str> = result.schema.fields().map(|f| f.name().as_str()).collect(); + assert_eq!(&actual, expected_names); + } + + // === apply_schema_operations: SetNullable tests === + + fn deeply_nested_required_schema() -> StructType { + schema! { + not_null "id": INTEGER, + nullable "address": { + nullable "location": { + not_null "zipcode": STRING, + }, + }, + } + } + + #[rstest] + #[case::on_required_field(simple_schema(), column_name!("id"))] + #[case::already_nullable_is_noop(simple_schema(), column_name!("name"))] + #[case::case_insensitive(simple_schema(), column_name!("ID"))] + #[case::nested_field(nested_schema(), column_name!("address.city"))] + #[case::deeply_nested_field(deeply_nested_required_schema(), column_name!("address.location.zipcode"))] + fn set_nullable_succeeds(#[case] schema: StructType, #[case] column: ColumnName) { + let ops = vec![SchemaOperation::SetNullable { + column: column.clone(), + }]; + let result = apply_schema_operations(schema, ops, ColumnMappingMode::None, None).unwrap(); + assert!(result.schema.field_at_path(column.path()).is_nullable()); + } + + #[rstest] + #[case::nonexistent_column(column_name!("nonexistent"), "does not exist")] + #[case::through_non_struct(column_name!("name.inner"), "not a struct")] + #[case::empty_path(ColumnName::new(Vec::::new()), "empty column path")] + fn set_nullable_fails(#[case] column: ColumnName, #[case] error_contains: &str) { + let ops = vec![SchemaOperation::SetNullable { column }]; + let err = apply_schema_operations(simple_schema(), ops, ColumnMappingMode::None, None) + .unwrap_err(); + assert!( + err.to_string().contains(error_contains), + "expected error to contain '{error_contains}', got: {err}" + ); + } + + /// Setting a struct itself nullable must not mutate inner fields. Kept separate from the + /// `set_nullable_succeeds` rstest because it asserts on inner-field preservation. + #[test] + fn set_nullable_on_struct_itself_preserves_inner_fields() { + let schema = StructType::try_new(vec![StructField::not_null( + "address", + StructType::try_new(vec![StructField::not_null("city", DataType::STRING)]).unwrap(), + )]) + .unwrap(); + let ops = vec![SchemaOperation::SetNullable { + column: column_name!("address"), + }]; + let result = apply_schema_operations(schema, ops, ColumnMappingMode::None, None).unwrap(); + let addr = result.schema.field("address").unwrap(); + assert!(addr.is_nullable(), "struct itself must be nullable"); + match addr.data_type() { + DataType::Struct(s) => assert!( + !s.field("city").unwrap().is_nullable(), + "inner field must remain NOT NULL" + ), + other => panic!("Expected Struct, got: {other:?}"), + } + } + + #[test] + fn chain_add_and_set_nullable_applies_both() { + let ops = vec![ + SchemaOperation::AddColumn { + field: StructField::nullable("email", DataType::STRING), + }, + SchemaOperation::SetNullable { + column: column_name!("id"), + }, + ]; + let result = + apply_schema_operations(simple_schema(), ops, ColumnMappingMode::None, None).unwrap(); + assert_eq!(result.schema.fields().count(), 3); + assert!(result.schema.field("email").is_some()); + assert!(result.schema.field("id").unwrap().is_nullable()); + } + + #[test] + fn set_nullable_nested_preserves_top_level_order() { + // SetNullable on a nested field within a middle top-level field must not reorder + // the top-level IndexMap. + let schema = StructType::try_new(vec![ + StructField::not_null("alpha", DataType::INTEGER), + StructField::nullable( + "beta", + StructType::try_new(vec![StructField::not_null("nested", DataType::STRING)]) + .unwrap(), + ), + StructField::not_null("gamma", DataType::STRING), + ]) + .unwrap(); + let ops = vec![SchemaOperation::SetNullable { + column: column_name!("beta.nested"), + }]; + let result = apply_schema_operations(schema, ops, ColumnMappingMode::None, None).unwrap(); + let names: Vec<&String> = result.schema.fields().map(|f| f.name()).collect(); + assert_eq!(names, vec!["alpha", "beta", "gamma"]); + } + + // === Column mapping tests === + + fn get_cm_id(field: &StructField) -> i64 { + field + .column_mapping_id() + .expect("field should have column mapping ID") + } + + fn get_physical_name(field: &StructField) -> String { + match field + .get_config_value(&ColumnMetadataKey::ColumnMappingPhysicalName) + .expect("field should have physical name") + { + MetadataValue::String(s) => s.clone(), + other => panic!("expected String, got {other:?}"), + } + } + + #[rstest] + #[case::name_mode(ColumnMappingMode::Name, 2, 3)] + #[case::id_mode(ColumnMappingMode::Id, 5, 6)] + fn add_column_with_column_mapping_assigns_id_and_physical_name( + #[case] mode: ColumnMappingMode, + #[case] current_max: i64, + #[case] expected_id: i64, + ) { + let ops = vec![SchemaOperation::AddColumn { + field: StructField::nullable("email", DataType::STRING), + }]; + let result = + apply_schema_operations(simple_schema(), ops, mode, Some(current_max)).unwrap(); + let email_field = result.schema.field("email").unwrap(); + + assert_eq!(get_cm_id(email_field), expected_id); + assert!(get_physical_name(email_field).starts_with("col-")); + assert_eq!(result.new_max_column_id, Some(expected_id)); + } + + #[test] + fn add_column_without_max_column_id_fails_when_mapping_enabled() { + let ops = vec![SchemaOperation::AddColumn { + field: StructField::nullable("email", DataType::STRING), + }]; + let err = apply_schema_operations(simple_schema(), ops, ColumnMappingMode::Name, None) + .unwrap_err(); + assert!(matches!(err, Error::InvalidProtocol(_))); + assert!(err.to_string().contains("maxColumnId")); + } + + /// Multiple columns added in a single ALTER on a CM table: each must get a distinct ID + /// (strictly monotone), a distinct physical name, and `new_max_column_id` must advance by + /// exactly the number of added columns. + #[test] + fn add_multiple_columns_with_column_mapping_assigns_unique_ids() { + let ops = vec![ + SchemaOperation::AddColumn { + field: StructField::nullable("a", DataType::STRING), + }, + SchemaOperation::AddColumn { + field: StructField::nullable("b", DataType::STRING), + }, + SchemaOperation::AddColumn { + field: StructField::nullable("c", DataType::STRING), + }, + ]; + let result = + apply_schema_operations(simple_schema(), ops, ColumnMappingMode::Name, Some(10)) + .unwrap(); + + let id_a = get_cm_id(result.schema.field("a").unwrap()); + let id_b = get_cm_id(result.schema.field("b").unwrap()); + let id_c = get_cm_id(result.schema.field("c").unwrap()); + assert_eq!(id_a, 11); + assert_eq!(id_b, 12); + assert_eq!(id_c, 13); + + let name_a = get_physical_name(result.schema.field("a").unwrap()); + let name_b = get_physical_name(result.schema.field("b").unwrap()); + let name_c = get_physical_name(result.schema.field("c").unwrap()); + assert_ne!(name_a, name_b); + assert_ne!(name_b, name_c); + assert_ne!(name_a, name_c); + + assert_eq!(result.new_max_column_id, Some(13)); + } + + fn struct_of_two_primitives() -> DataType { + DataType::from( + StructType::try_new(vec![ + StructField::nullable("a", DataType::STRING), + StructField::nullable("b", DataType::STRING), + ]) + .unwrap(), + ) + } + + /// Adding a complex column on a CM table: every inner struct field reachable through + /// Struct/Array/Map recursion must receive a distinct ID greater than the previous max, + /// and `new_max_column_id` must advance to the largest assigned ID. + #[rstest] + #[case::nested_struct(struct_of_two_primitives(), 3)] + #[case::array_of_primitive(DataType::from(ArrayType::new(DataType::STRING, true)), 1)] + #[case::map_of_primitives( + DataType::from(MapType::new(DataType::STRING, DataType::INTEGER, true)), + 1 + )] + #[case::array_of_struct(DataType::from(ArrayType::new(struct_of_two_primitives(), true)), 3)] + #[case::map_value_is_struct( + DataType::from(MapType::new(DataType::STRING, struct_of_two_primitives(), true,)), + 3 + )] + #[case::map_key_is_struct( + DataType::from(MapType::new(struct_of_two_primitives(), DataType::INTEGER, true,)), + 3 + )] + fn add_complex_column_with_column_mapping_assigns_ids_to_all_inner_fields( + #[case] data_type: DataType, + #[case] expected_id_count: usize, + ) { + let ops = vec![SchemaOperation::AddColumn { + field: StructField::nullable("col", data_type), + }]; + let result = + apply_schema_operations(simple_schema(), ops, ColumnMappingMode::Name, Some(10)) + .unwrap(); + + let added = result.schema.field("col").unwrap(); + let ids = added.collect_column_mapping_ids(); + let unique: HashSet<_> = ids.iter().copied().collect(); + + assert_eq!(ids.len(), expected_id_count, "expected ID count mismatch"); + assert_eq!(unique.len(), ids.len(), "all assigned IDs must be distinct"); + assert!( + ids.iter().all(|&id| id > 10), + "all assigned IDs must exceed previous max" + ); + assert_eq!( + result.new_max_column_id, + ids.iter().max().copied(), + "new_max_column_id must equal the largest assigned ID", + ); + } + + fn field_with_id_only(name: &str, ty: DataType, id: i64) -> StructField { + let mut f = StructField::nullable(name, ty); + f.metadata.insert( + ColumnMetadataKey::ColumnMappingId.as_ref().to_string(), + MetadataValue::Number(id), + ); + f + } + + /// CM-enabled: a connector may pre-populate `delta.columnMapping.id` on the new column, + /// even at nested depth. The id is preserved and the missing `physicalName` is filled in, + /// matching delta-spark's `assignColumnIdAndPhysicalName`. + #[rstest] + #[case::top_level(field_with_id_only("tainted", DataType::STRING, 99))] + #[case::nested_in_struct(StructField::nullable( + "outer", + StructType::try_new(vec![field_with_id_only("inner", DataType::STRING, 99)]).unwrap(), + ))] + fn add_column_with_preexisting_cm_metadata_is_preserved_under_cm(#[case] field: StructField) { + let ops = vec![SchemaOperation::AddColumn { field }]; + let result = apply_schema_operations( + simple_schema(), + ops, + ColumnMappingMode::Name, + Some(2), // current_max_column_id + ) + .unwrap(); + + // The preserved id=99 must surface as the schema's max, even though the persisted + // maxColumnId was only 2 going in. + assert_eq!(find_max_column_id_in_schema(&result.schema), Some(99)); + assert_eq!(result.new_max_column_id, Some(99)); + } + + /// CM-enabled, only `physicalName` provided: id is allocated as `max_id + 1` and the + /// physical name is preserved. + #[test] + fn add_column_with_only_physical_name_allocates_id() { + let mut field = StructField::nullable("named", DataType::STRING); + field.metadata.insert( + ColumnMetadataKey::ColumnMappingPhysicalName + .as_ref() + .to_string(), + MetadataValue::String("user-supplied-name".to_string()), + ); + let ops = vec![SchemaOperation::AddColumn { field }]; + let result = + apply_schema_operations(simple_schema(), ops, ColumnMappingMode::Name, Some(7)) + .unwrap(); + let added = result.schema.field("named").unwrap(); + assert_eq!(get_cm_id(added), 8); + assert_eq!( + added.get_config_value(&ColumnMetadataKey::ColumnMappingPhysicalName), + Some(&MetadataValue::String("user-supplied-name".to_string())) + ); + assert_eq!(result.new_max_column_id, Some(8)); + } + + /// A persisted `delta.columnMapping.maxColumnId` above the protocol's 32-bit cap is + /// rejected before evolution starts, so the allocator never sees an out-of-range seed. + /// `i64::MAX` and `i32::MAX + 1` both cross the bound; the negative case ensures the + /// `0..=MAX_COLUMN_MAPPING_ID` range is closed on the low end too. + #[rstest] + #[case::above_protocol_max(i32::MAX as i64 + 1)] + #[case::i64_max(i64::MAX)] + #[case::negative(-1)] + fn alter_with_out_of_range_persisted_max_column_id_is_rejected(#[case] seed: i64) { + let ops = vec![SchemaOperation::AddColumn { + field: StructField::nullable("anything", DataType::INTEGER), + }]; + let err = + apply_schema_operations(simple_schema(), ops, ColumnMappingMode::Name, Some(seed)) + .unwrap_err() + .to_string(); + assert!( + err.contains("Invalid column mapping id") + && err.contains("Table property `delta.columnMapping.maxColumnId`") + && err.contains(&seed.to_string()), + "expected canonical out-of-range rejection naming the seed location and value, \ + got: {err}", + ); + } + + /// CM-enabled, wrong-typed `delta.columnMapping.id` annotation: error. + #[test] + fn add_column_with_wrong_typed_id_is_rejected() { + let mut field = StructField::nullable("bad", DataType::STRING); + field.metadata.insert( + ColumnMetadataKey::ColumnMappingId.as_ref().to_string(), + MetadataValue::String("not-a-number".to_string()), + ); + let ops = vec![SchemaOperation::AddColumn { field }]; + let err = apply_schema_operations( + simple_schema(), + ops, + ColumnMappingMode::Name, + Some(2), // current_max_column_id + ) + .unwrap_err() + .to_string(); + assert!( + err.contains("non-numeric") && err.contains("delta.columnMapping.id"), + "error should name the wrong-typed id annotation, got: {err}" + ); + } + + /// If the persisted `maxColumnId` is stale (smaller than the actual max ID present in + /// the schema), the defensive seed rebases on the schema's max so a newly added column + /// cannot collide with an existing field's ID. Matches delta-spark's `findMaxColumnId`. + #[test] + fn stale_max_column_id_is_self_healed_by_schema_walk() { + let mut existing = StructField::nullable("existing", DataType::STRING); + existing.metadata.insert( + ColumnMetadataKey::ColumnMappingId.as_ref().to_string(), + MetadataValue::Number(42), + ); + let schema = StructType::try_new(vec![existing]).unwrap(); + let ops = vec![SchemaOperation::AddColumn { + field: StructField::nullable("new", DataType::STRING), + }]; + // Persisted maxColumnId is stale at 5, but the schema actually contains id=42. + let result = + apply_schema_operations(schema, ops, ColumnMappingMode::Name, Some(5)).unwrap(); + let new_id = get_cm_id(result.schema.field("new").unwrap()); + assert_eq!( + new_id, 43, + "new id must follow schema max (42), not stale property (5)" + ); + assert_eq!(result.new_max_column_id, Some(43)); + } +} diff --git a/kernel/src/transaction/stats_verifier.rs b/kernel/src/transaction/stats_verifier.rs index 46bf8a37c8..8774ff1b40 100644 --- a/kernel/src/transaction/stats_verifier.rs +++ b/kernel/src/transaction/stats_verifier.rs @@ -7,6 +7,7 @@ use std::sync::LazyLock; +use crate::actions::{MAX_VALUES, MIN_VALUES, NULL_COUNT, NUM_RECORDS}; use crate::engine_data::{GetData, RowVisitor, TypedGetData as _}; use crate::error::Error; use crate::expressions::{column_name, ColumnName}; @@ -19,13 +20,13 @@ use crate::DeltaResult; /// For each required column, validates that `nullCount` is present (non-null) and that /// `minValues` and `maxValues` are present unless the column is all-null /// (`nullCount == numRecords`). -pub(crate) struct StatsVerifier { +pub struct StatsColumnVerifier { required_columns: Vec<(ColumnName, DataType)>, } -impl StatsVerifier { +impl StatsColumnVerifier { /// Create a new verifier that checks statistics for the given required columns and types. - pub(crate) fn new(required_columns: Vec<(ColumnName, DataType)>) -> Self { + pub fn new(required_columns: Vec<(ColumnName, DataType)>) -> Self { Self { required_columns } } @@ -33,7 +34,7 @@ impl StatsVerifier { /// /// For each required column, extracts all three stat columns (nullCount, minValues, /// maxValues) in a single `visit_rows` call per batch. - pub(crate) fn verify(&self, add_files: &[Box]) -> DeltaResult<()> { + pub fn verify(&self, add_files: &[Box]) -> DeltaResult<()> { if self.required_columns.is_empty() { return Ok(()); } @@ -54,11 +55,11 @@ impl StatsVerifier { data_type: &DataType, ) -> DeltaResult<()> { let column_names = vec![ - ColumnName::new(["path"]), - ColumnName::new(["stats", "numRecords"]), - build_stat_path(column, "nullCount"), - build_stat_path(column, "minValues"), - build_stat_path(column, "maxValues"), + column_name!("path"), + column_name!("stats", NUM_RECORDS), + build_stat_path(column, NULL_COUNT), + build_stat_path(column, MIN_VALUES), + build_stat_path(column, MAX_VALUES), ]; let types = column_types_for(data_type)?; @@ -67,7 +68,7 @@ impl StatsVerifier { let mut missing_max: Vec = Vec::new(); for batch in add_files { - let mut visitor = ColumnStatsVisitor { + let mut visitor = ColumnStatsValidator { data_type, types, missing_null_count: &mut missing_null_count, @@ -133,6 +134,8 @@ macro_rules! define_column_types { } define_column_types!(COL_TYPES_BOOL, DataType::BOOLEAN); +define_column_types!(COL_TYPES_BYTE, DataType::BYTE); +define_column_types!(COL_TYPES_SHORT, DataType::SHORT); define_column_types!(COL_TYPES_INT, DataType::INTEGER); define_column_types!(COL_TYPES_LONG, DataType::LONG); define_column_types!(COL_TYPES_STRING, DataType::STRING); @@ -161,10 +164,19 @@ static COL_TYPES_DECIMAL: LazyLock = LazyLock::new(|| { (names, types).into() }); +/// [`ColumnNamesAndTypes`] for [`NumRecordsValidator`]. +static NUM_RECORDS_TYPES: LazyLock = LazyLock::new(|| { + let names = vec![column_name!("path"), column_name!("stats.numRecords")]; + let types = vec![DataType::STRING, DataType::LONG]; + (names, types).into() +}); + /// Select the predefined static type array for a given column data type. fn column_types_for(dt: &DataType) -> DeltaResult<&'static ColumnNamesAndTypes> { match dt { &DataType::BOOLEAN => Ok(&COL_TYPES_BOOL), + &DataType::BYTE => Ok(&COL_TYPES_BYTE), + &DataType::SHORT => Ok(&COL_TYPES_SHORT), &DataType::INTEGER => Ok(&COL_TYPES_INT), &DataType::LONG => Ok(&COL_TYPES_LONG), &DataType::STRING => Ok(&COL_TYPES_STRING), @@ -175,7 +187,18 @@ fn column_types_for(dt: &DataType) -> DeltaResult<&'static ColumnNamesAndTypes> &DataType::TIMESTAMP => Ok(&COL_TYPES_TIMESTAMP), &DataType::TIMESTAMP_NTZ => Ok(&COL_TYPES_TIMESTAMP_NTZ), DataType::Primitive(PrimitiveType::Decimal(_)) => Ok(&COL_TYPES_DECIMAL), - _ => Err(Error::internal_error(format!( + &DataType::INTERVAL_YEAR_MONTH | &DataType::INTERVAL_DAY_TIME => Err(Error::unsupported( + format!("Interval types are not supported for stats validation: {dt}"), + )), + #[cfg(feature = "geo-type-in-dev")] + DataType::Primitive(PrimitiveType::Geometry(_) | PrimitiveType::Geography(_)) => Err( + Error::unsupported(format!("Unsupported data type for stats validation: {dt}")), + ), + &DataType::VOID + | DataType::Struct(_) + | DataType::Array(_) + | DataType::Map(_) + | DataType::Variant(_) => Err(Error::internal_error(format!( "Unsupported data type for stats validation: {dt}" ))), } @@ -190,6 +213,8 @@ fn is_stat_present<'b>( let field_name = "stat"; match data_type { &DataType::BOOLEAN => Ok(getter.get_bool(row_idx, field_name)?.is_some()), + &DataType::BYTE => Ok(getter.get_byte(row_idx, field_name)?.is_some()), + &DataType::SHORT => Ok(getter.get_short(row_idx, field_name)?.is_some()), &DataType::INTEGER => Ok(getter.get_int(row_idx, field_name)?.is_some()), &DataType::LONG => Ok(getter.get_long(row_idx, field_name)?.is_some()), &DataType::FLOAT => Ok(getter.get_float(row_idx, field_name)?.is_some()), @@ -203,7 +228,20 @@ fn is_stat_present<'b>( DataType::Primitive(PrimitiveType::Decimal(_)) => { Ok(getter.get_decimal(row_idx, field_name)?.is_some()) } - _ => Err(Error::internal_error(format!( + &DataType::INTERVAL_YEAR_MONTH | &DataType::INTERVAL_DAY_TIME => Err(Error::unsupported( + format!("Interval types are not supported for stats presence check: {data_type}"), + )), + #[cfg(feature = "geo-type-in-dev")] + DataType::Primitive(PrimitiveType::Geometry(_) | PrimitiveType::Geography(_)) => { + Err(Error::unsupported(format!( + "Unsupported data type for stats presence check: {data_type}" + ))) + } + &DataType::VOID + | DataType::Struct(_) + | DataType::Array(_) + | DataType::Map(_) + | DataType::Variant(_) => Err(Error::internal_error(format!( "Unsupported data type for stats presence check: {data_type}" ))), } @@ -211,7 +249,7 @@ fn is_stat_present<'b>( /// Visitor that checks nullCount, minValues, and maxValues for a single column in one pass. /// Expects 5 getters: [path, numRecords, nullCount, minValues, maxValues]. -struct ColumnStatsVisitor<'a> { +struct ColumnStatsValidator<'a> { data_type: &'a DataType, types: &'static ColumnNamesAndTypes, missing_null_count: &'a mut Vec, @@ -219,7 +257,7 @@ struct ColumnStatsVisitor<'a> { missing_max: &'a mut Vec, } -impl RowVisitor for ColumnStatsVisitor<'_> { +impl RowVisitor for ColumnStatsValidator<'_> { fn selected_column_names_and_types(&self) -> (&'static [ColumnName], &'static [DataType]) { self.types.as_ref() } @@ -235,8 +273,8 @@ impl RowVisitor for ColumnStatsVisitor<'_> { for row_idx in 0..row_count { let path: String = getters[0].get(row_idx, "path")?; - let num_records = getters[1].get_long(row_idx, "numRecords")?; - let null_count = getters[2].get_long(row_idx, "nullCount")?; + let num_records = getters[1].get_long(row_idx, NUM_RECORDS)?; + let null_count = getters[2].get_long(row_idx, NULL_COUNT)?; // When all rows are null (or the file is empty), minValues/maxValues are // expected to be null since there are no non-null values to aggregate. @@ -257,434 +295,73 @@ impl RowVisitor for ColumnStatsVisitor<'_> { } } -#[cfg(test)] -mod tests { - use super::*; - - use std::sync::Arc; - - use rstest::rstest; - - use crate::arrow::array::{ - Array, ArrayRef, Int64Array, LargeStringArray, RecordBatch, StringArray, StringViewArray, - StructArray, - }; - use crate::arrow::datatypes::{ - DataType as ArrowDataType, Field as ArrowField, Fields, Schema as ArrowSchema, - }; - use crate::engine::arrow_data::ArrowEngineData; - use crate::engine::default::stats::collect_stats; - use crate::expressions::column_name; - use crate::EngineData; - - /// Creates test add file data with stats.numRecords, stats.nullCount.col, - /// stats.minValues.col, and stats.maxValues.col — all of type LONG. - fn create_add_file_batch( - paths: Vec<&str>, - num_records: Vec>, - null_counts: Vec>, - min_values: Vec>, - max_values: Vec>, - ) -> Box { - assert_eq!(paths.len(), num_records.len()); - assert_eq!(paths.len(), null_counts.len()); - assert_eq!(paths.len(), min_values.len()); - assert_eq!(paths.len(), max_values.len()); - - let path_array = StringArray::from(paths.to_vec()); - let col_field = Arc::new(ArrowField::new("col", ArrowDataType::Int64, true)); - - let num_records_array = Int64Array::from(num_records); - let null_count_struct = StructArray::new( - Fields::from(vec![col_field.clone()]), - vec![Arc::new(Int64Array::from(null_counts)) as ArrayRef], - None, - ); - let min_values_struct = StructArray::new( - Fields::from(vec![col_field.clone()]), - vec![Arc::new(Int64Array::from(min_values)) as ArrayRef], - None, - ); - let max_values_struct = StructArray::new( - Fields::from(vec![col_field]), - vec![Arc::new(Int64Array::from(max_values)) as ArrayRef], - None, - ); - - let inner_struct_type = |name: &str| { - ArrowField::new( - name, - ArrowDataType::Struct(Fields::from(vec![ArrowField::new( - "col", - ArrowDataType::Int64, - true, - )])), - true, - ) +/// Verify that every `add` action has `stats.numRecords` populated. Short-circuits on the first +/// violation and returns an error containing the `add.path`. +pub fn verify_num_records_present(add_files: &[Box]) -> DeltaResult<()> { + let column_names = vec![column_name!("path"), column_name!("stats", NUM_RECORDS)]; + let mut first_missing: Option = None; + for batch in add_files { + let mut visitor = NumRecordsValidator { + first_missing: &mut first_missing, }; - - let stats_fields = Fields::from(vec![ - ArrowField::new("numRecords", ArrowDataType::Int64, true), - inner_struct_type("nullCount"), - inner_struct_type("minValues"), - inner_struct_type("maxValues"), - ]); - let stats_struct = StructArray::new( - stats_fields.clone(), - vec![ - Arc::new(num_records_array) as ArrayRef, - Arc::new(null_count_struct) as ArrayRef, - Arc::new(min_values_struct) as ArrayRef, - Arc::new(max_values_struct) as ArrayRef, - ], - None, - ); - - let schema = Arc::new(ArrowSchema::new(vec![ - ArrowField::new("path", ArrowDataType::Utf8, false), - ArrowField::new("stats", ArrowDataType::Struct(stats_fields), true), - ])); - - let batch = - RecordBatch::try_new(schema, vec![Arc::new(path_array), Arc::new(stats_struct)]) - .unwrap(); - - Box::new(ArrowEngineData::new(batch)) - } - - #[test] - fn test_verifier_with_empty_add_files() { - let columns = vec![(ColumnName::new(["col"]), DataType::LONG)]; - let verifier = StatsVerifier::new(columns); - let result = verifier.verify(&[]); - assert!(result.is_ok()); - } - - #[test] - fn test_verify_valid_stats() { - let batch = create_add_file_batch( - vec!["file1.parquet", "file2.parquet"], - vec![Some(100), Some(100)], - vec![Some(0), Some(5)], - vec![Some(1), Some(10)], - vec![Some(100), Some(50)], - ); - - let columns = vec![(ColumnName::new(["col"]), DataType::LONG)]; - let verifier = StatsVerifier::new(columns); - let result = verifier.verify(&[batch]); - assert!(result.is_ok()); - } - - #[test] - fn test_verify_missing_stat_category() { - let cases = [ - ("nullCount", vec![None], vec![Some(1)], vec![Some(100)]), - ("minValues", vec![Some(0)], vec![None], vec![Some(100)]), - ("maxValues", vec![Some(0)], vec![Some(1)], vec![None]), - ]; - for (category, null_counts, min_values, max_values) in cases { - let batch = create_add_file_batch( - vec!["file1.parquet"], - vec![Some(100)], - null_counts, - min_values, - max_values, - ); - let verifier = StatsVerifier::new(vec![(ColumnName::new(["col"]), DataType::LONG)]); - let err_msg = verifier.verify(&[batch]).unwrap_err().to_string(); - assert!(err_msg.contains("file1.parquet"), "case: {category}"); - assert!(err_msg.contains(category), "case: {category}"); + batch.visit_rows(&column_names, &mut visitor)?; + if first_missing.is_some() { + break; } } - - #[test] - fn test_verify_multiple_batches() { - let batch1 = create_add_file_batch( - vec!["good_file.parquet"], - vec![Some(100)], - vec![Some(0)], - vec![Some(1)], - vec![Some(100)], - ); - let batch2 = create_add_file_batch( - vec!["bad_file.parquet"], - vec![Some(100)], - vec![None], - vec![None], - vec![None], - ); - - let columns = vec![(ColumnName::new(["col"]), DataType::LONG)]; - let verifier = StatsVerifier::new(columns); - let result = verifier.verify(&[batch1, batch2]); - - assert!(result.is_err()); - let err_msg = result.unwrap_err().to_string(); - assert!(err_msg.contains("bad_file.parquet")); - assert!(!err_msg.contains("good_file.parquet")); - } - - #[test] - fn test_verify_no_required_columns() { - let batch = create_add_file_batch( - vec!["file1.parquet"], - vec![Some(100)], - vec![None], - vec![None], - vec![None], - ); - - let verifier = StatsVerifier::new(vec![]); - let result = verifier.verify(&[batch]); - assert!(result.is_ok()); - } - - #[test] - fn test_verify_all_null_column_allows_null_min_max() { - // nullCount == numRecords means all rows are null, so null min/max is valid - let batch = create_add_file_batch( - vec!["file1.parquet"], - vec![Some(100)], - vec![Some(100)], - vec![None], - vec![None], - ); - - let columns = vec![(ColumnName::new(["col"]), DataType::LONG)]; - let verifier = StatsVerifier::new(columns); - assert!(verifier.verify(&[batch]).is_ok()); + if let Some(path) = first_missing { + return Err(Error::stats_validation(format!( + "'stats.numRecords' is required for this table (see \ + `TableConfiguration::requires_stats_num_records`), but is missing for file '{path}'", + ))); } + Ok(()) +} - #[test] - fn test_verify_partial_null_column_requires_min_max() { - // nullCount < numRecords means not all rows are null, so min/max must be present - let batch = create_add_file_batch( - vec!["file1.parquet"], - vec![Some(100)], - vec![Some(50)], - vec![None], - vec![None], - ); - - let columns = vec![(ColumnName::new(["col"]), DataType::LONG)]; - let verifier = StatsVerifier::new(columns); - let result = verifier.verify(&[batch]); - assert!(matches!(result, Err(Error::StatsValidation(_)))); - let err = result.unwrap_err().to_string(); - assert!(err.contains("minValues")); - } - - /// Creates test data with two columns (col_a, col_b) in stats, both LONG. - #[allow(clippy::too_many_arguments)] - fn create_two_column_batch( - paths: Vec<&str>, - num_records: Vec>, - col_a_nullcount: Vec>, - col_a_min: Vec>, - col_a_max: Vec>, - col_b_nullcount: Vec>, - col_b_min: Vec>, - col_b_max: Vec>, - ) -> Box { - let path_array = StringArray::from(paths.to_vec()); - let col_a_field = Arc::new(ArrowField::new("col_a", ArrowDataType::Int64, true)); - let col_b_field = Arc::new(ArrowField::new("col_b", ArrowDataType::Int64, true)); - let both_fields = Fields::from(vec![col_a_field, col_b_field]); - - let make_struct = |a: Vec>, b: Vec>| { - StructArray::new( - both_fields.clone(), - vec![ - Arc::new(Int64Array::from(a)) as ArrayRef, - Arc::new(Int64Array::from(b)) as ArrayRef, - ], - None, - ) - }; - - let num_records_array = Int64Array::from(num_records); - let null_count_struct = make_struct(col_a_nullcount, col_b_nullcount); - let min_values_struct = make_struct(col_a_min, col_b_min); - let max_values_struct = make_struct(col_a_max, col_b_max); - - let inner_type = ArrowDataType::Struct(Fields::from(vec![ - ArrowField::new("col_a", ArrowDataType::Int64, true), - ArrowField::new("col_b", ArrowDataType::Int64, true), - ])); - let stats_fields = Fields::from(vec![ - ArrowField::new("numRecords", ArrowDataType::Int64, true), - ArrowField::new("nullCount", inner_type.clone(), true), - ArrowField::new("minValues", inner_type.clone(), true), - ArrowField::new("maxValues", inner_type, true), - ]); - let stats_struct = StructArray::new( - stats_fields.clone(), - vec![ - Arc::new(num_records_array) as ArrayRef, - Arc::new(null_count_struct) as ArrayRef, - Arc::new(min_values_struct) as ArrayRef, - Arc::new(max_values_struct) as ArrayRef, - ], - None, - ); +/// Visitor validates that every `add` action has `stats.numRecords` populated. +/// Stops at the first row whose `numRecords` is null and records that row's `add.path`. +struct NumRecordsValidator<'a> { + first_missing: &'a mut Option, +} - let schema = Arc::new(ArrowSchema::new(vec![ - ArrowField::new("path", ArrowDataType::Utf8, false), - ArrowField::new("stats", ArrowDataType::Struct(stats_fields), true), - ])); - let batch = - RecordBatch::try_new(schema, vec![Arc::new(path_array), Arc::new(stats_struct)]) - .unwrap(); - Box::new(ArrowEngineData::new(batch)) +impl RowVisitor for NumRecordsValidator<'_> { + fn selected_column_names_and_types(&self) -> (&'static [ColumnName], &'static [DataType]) { + NUM_RECORDS_TYPES.as_ref() } - #[test] - fn test_verify_multiple_columns() { - // Both columns have valid stats - let batch = create_two_column_batch( - vec!["file1.parquet"], - vec![Some(100)], - vec![Some(0)], - vec![Some(1)], - vec![Some(10)], - vec![Some(0)], - vec![Some(2)], - vec![Some(20)], - ); - let columns = vec![ - (ColumnName::new(["col_a"]), DataType::LONG), - (ColumnName::new(["col_b"]), DataType::LONG), - ]; - assert!(StatsVerifier::new(columns).verify(&[batch]).is_ok()); - - // col_a valid, col_b missing minValues - let batch = create_two_column_batch( - vec!["file1.parquet"], - vec![Some(100)], - vec![Some(0)], - vec![Some(1)], - vec![Some(10)], - vec![Some(0)], - vec![None], - vec![Some(20)], + fn visit<'b>(&mut self, row_count: usize, getters: &[&'b dyn GetData<'b>]) -> DeltaResult<()> { + require!( + getters.len() == 2, + Error::internal_error(format!( + "Expected 2 getters for numRecords validation, got {}", + getters.len() + )) ); - let columns = vec![ - (ColumnName::new(["col_a"]), DataType::LONG), - (ColumnName::new(["col_b"]), DataType::LONG), - ]; - let err_msg = StatsVerifier::new(columns) - .verify(&[batch]) - .unwrap_err() - .to_string(); - assert!(err_msg.contains("col_b")); - assert!(err_msg.contains("minValues")); - assert!(!err_msg.contains("col_a")); - } - - /// Verifies that stats collected from non-standard Arrow string representations - /// (LargeUtf8/LargeStringArray, Utf8View/StringViewArray) can be validated by - /// StatsVerifier, which expects Delta's logical STRING type. Engines may use any of - /// these representations, and the stats pipeline must handle them without type errors. - #[rstest] - #[case::large_utf8(Arc::new(LargeStringArray::from(vec!["Austin", "Boston", "Chicago"])) as ArrayRef)] - #[case::utf8_view(Arc::new(StringViewArray::from(vec!["Austin", "Boston", "Chicago"])) as ArrayRef)] - fn test_verify_string_stats(#[case] values: ArrayRef) { - let schema = Arc::new(ArrowSchema::new(vec![ArrowField::new( - "city", - values.data_type().clone(), - false, - )])); - let batch = RecordBatch::try_new(schema, vec![values]).unwrap(); - - let stats = collect_stats(&batch, &[column_name!("city")]).unwrap(); - - let path_array = StringArray::from(vec!["file1.parquet"]); - let add_file_schema = Arc::new(ArrowSchema::new(vec![ - ArrowField::new("path", ArrowDataType::Utf8, false), - ArrowField::new("stats", stats.data_type().clone(), true), - ])); - let add_file_batch = RecordBatch::try_new( - add_file_schema, - vec![ - Arc::new(path_array) as ArrayRef, - Arc::new(stats) as ArrayRef, - ], - ) - .unwrap(); - - let engine_data: Box = Box::new(ArrowEngineData::new(add_file_batch)); - - let verifier = StatsVerifier::new(vec![(ColumnName::new(["city"]), DataType::STRING)]); - verifier.verify(&[engine_data]).unwrap(); + for row_idx in 0..row_count { + if getters[1].get_long(row_idx, NUM_RECORDS)?.is_none() { + *self.first_missing = Some(getters[0].get(row_idx, "path")?); + return Ok(()); + } + } + Ok(()) } +} - /// Round-trip test: collect_stats produces stats that pass verification for non-null data. - #[test] - fn test_collected_stats_pass_verification() { - let values: ArrayRef = Arc::new(Int64Array::from(vec![Some(1), Some(2), Some(3)])); - let schema = Arc::new(ArrowSchema::new(vec![ArrowField::new( - "col", - values.data_type().clone(), - true, - )])); - let batch = RecordBatch::try_new(schema, vec![values]).unwrap(); - - let stats = collect_stats(&batch, &[column_name!("col")]).unwrap(); - - let path_array = StringArray::from(vec!["file1.parquet"]); - let add_file_schema = Arc::new(ArrowSchema::new(vec![ - ArrowField::new("path", ArrowDataType::Utf8, false), - ArrowField::new("stats", stats.data_type().clone(), true), - ])); - let add_file_batch = RecordBatch::try_new( - add_file_schema, - vec![ - Arc::new(path_array) as ArrayRef, - Arc::new(stats) as ArrayRef, - ], - ) - .unwrap(); - - let engine_data: Box = Box::new(ArrowEngineData::new(add_file_batch)); +#[cfg(all(test, feature = "geo-type-in-dev"))] +mod tests { + use rstest::rstest; - let verifier = StatsVerifier::new(vec![(ColumnName::new(["col"]), DataType::LONG)]); - verifier.verify(&[engine_data]).unwrap(); - } + use super::*; + use crate::schema::{EdgeInterpolationAlgorithm, GeographyType, GeometryType}; - /// Verify collect_stats produces correct stats shape for all-null and empty batches. - /// These cases omit the column from minValues/maxValues entirely. #[rstest] - #[case::all_null_values(Arc::new(Int64Array::from(vec![None::, None, None])) as ArrayRef)] - #[case::empty_batch(Arc::new(Int64Array::from(Vec::>::new())) as ArrayRef)] - fn test_collected_stats_shape_for_all_null_and_empty(#[case] values: ArrayRef) { - let num_rows = values.len(); - let schema = Arc::new(ArrowSchema::new(vec![ArrowField::new( - "col", - values.data_type().clone(), - true, - )])); - let batch = RecordBatch::try_new(schema, vec![values]).unwrap(); - - let stats = collect_stats(&batch, &[column_name!("col")]).unwrap(); - - // numRecords should match row count - let num_records = stats - .column_by_name("numRecords") - .unwrap() - .as_any() - .downcast_ref::() - .unwrap(); - assert_eq!(num_records.value(0), num_rows as i64); - - // All-null/empty columns are omitted from minValues/maxValues - assert!( - stats.column_by_name("minValues").is_none(), - "minValues should be absent when all stats columns are all-null" - ); - assert!( - stats.column_by_name("maxValues").is_none(), - "maxValues should be absent when all stats columns are all-null" - ); + #[case(DataType::Primitive(PrimitiveType::Geometry(Box::new( + GeometryType::try_new("EPSG:4326").unwrap() + ))))] + #[case(DataType::Primitive(PrimitiveType::Geography(Box::new( + GeographyType::try_new("EPSG:4326", EdgeInterpolationAlgorithm::Spherical).unwrap() + ))))] + fn test_geo_types_unsupported_for_stats(#[case] dt: DataType) { + assert!(column_types_for(&dt).is_err()); } } diff --git a/kernel/src/transaction/update.rs b/kernel/src/transaction/update.rs index ca63bb1315..1464f1142e 100644 --- a/kernel/src/transaction/update.rs +++ b/kernel/src/transaction/update.rs @@ -13,26 +13,31 @@ use std::collections::HashMap; use std::marker::PhantomData; use std::sync::{Arc, LazyLock}; +use delta_kernel_derive::internal_api; use tracing::instrument; +use super::Transaction; use crate::actions::deletion_vector::DeletionVectorDescriptor; -use crate::actions::get_log_add_schema; +use crate::actions::{LOG_ADD_SCHEMA, NUM_RECORDS, TIGHT_BOUNDS}; use crate::committer::Committer; -use crate::engine_data::FilteredEngineData; -use crate::engine_data::{GetData, TypedGetData}; +use crate::engine_data::{ + FilteredEngineData, FilteredRowVisitor, GetData, RowIndexIterator, TypedGetData, +}; use crate::error::Error; -use crate::expressions::{column_name, ArrayData, ColumnName, Scalar, StructData, Transform}; +use crate::expressions::{ + col, column_name, lit, ArrayData, ColumnName, ExpressionStructPatchBuilder, Scalar, StructData, +}; +use crate::metrics::MetricId; use crate::scan::data_skipping::stats_schema::schema_with_all_fields_nullable; use crate::scan::log_replay::get_scan_metadata_transform_expr; use crate::scan::{restored_add_schema, scan_row_schema}; use crate::schema::{ArrayType, SchemaRef, StructField, StructType, ToSchema}; use crate::snapshot::SnapshotRef; -use crate::table_features::{Operation, TableFeature}; +use crate::table_features::{ + iceberg_compat_v3_column_defaults_validation, Operation, TableFeature, +}; use crate::utils::current_time_ms; -use crate::{DataType, DeltaResult, Engine, Expression, RowVisitor}; -use delta_kernel_derive::internal_api; - -use super::Transaction; +use crate::{DataType, DeltaResult, Engine, Expression}; // ============================================================================= // Update table transactions only @@ -61,7 +66,7 @@ impl Transaction { .ensure_operation_supported(Operation::Write)?; // Read clustering columns from snapshot (returns None if clustering not enabled) - let clustering_columns = read_snapshot.get_clustering_columns_physical(engine)?; + let clustering_columns = read_snapshot.get_physical_clustering_columns(engine)?; let commit_timestamp = current_time_ms()?; @@ -71,9 +76,22 @@ impl Transaction { read_version = read_snapshot.version(), ); + let effective_table_config = read_snapshot.table_configuration().clone(); + + // Surface IcebergCompatV3 interoperability risks without rejecting tables based on + // kernel parser limitations. + if effective_table_config.is_feature_enabled(&TableFeature::IcebergCompatV3) { + iceberg_compat_v3_column_defaults_validation(&effective_table_config)?; + } + Ok(Transaction { span, - read_snapshot, + operation_id: MetricId::new(), + correlation_id: None, + read_snapshot_opt: Some(read_snapshot), + effective_table_config, + should_emit_protocol: false, + should_emit_metadata: false, committer, operation: None, engine_info: None, @@ -85,10 +103,12 @@ impl Transaction { system_domain_metadata_additions: vec![], user_domain_removals: vec![], data_change: true, + column_defaults_acknowledged: false, engine_commit_info: None, is_blind_append: false, dv_matched_files: vec![], - clustering_columns_physical: clustering_columns, + num_dv_updates: 0, + physical_clustering_columns: clustering_columns, _state: PhantomData, }) } @@ -119,8 +139,8 @@ impl Transaction { /// If the domain does not exist in the Delta log, this is a no-op. /// Note that each domain can only appear once per transaction. That is, multiple operations /// on the same domain are disallowed in a single transaction, as well as setting and removing - /// the same domain in a single transaction. If a duplicate domain is included, the `commit` will - /// fail (that is, we don't eagerly check domain validity here). + /// the same domain in a single transaction. If a duplicate domain is included, the `commit` + /// will fail (that is, we don't eagerly check domain validity here). /// Removing metadata for multiple distinct domains is allowed. pub fn with_domain_metadata_removed(mut self, domain: String) -> Self { self.user_domain_removals.push(domain); @@ -133,7 +153,8 @@ impl Transaction { /// /// The expected schema for `remove_metadata` is given by [`scan_row_schema`]. It is expected /// this will be the result of passing [`FilteredEngineData`] returned from a scan - /// with the selection vector modified to select rows for removal (selected rows in the selection vector are the ones to be removed). + /// with the selection vector modified to select rows for removal (selected rows in the + /// selection vector are the ones to be removed). /// /// # Example /// @@ -141,11 +162,8 @@ impl Transaction { /// # use std::sync::Arc; /// # use delta_kernel::Engine; /// # use delta_kernel::snapshot::Snapshot; - /// # #[cfg(feature = "catalog-managed")] /// # use delta_kernel::committer::FileSystemCommitter; /// # fn example(engine: Arc, table_url: url::Url) -> delta_kernel::DeltaResult<()> { - /// # #[cfg(feature = "catalog-managed")] - /// # { /// // Create a snapshot and transaction /// let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; /// let mut txn = snapshot.clone().transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())?; @@ -164,7 +182,6 @@ impl Transaction { /// /// // Commit the transaction /// txn.commit(engine.as_ref())?; - /// # } /// # Ok(()) /// # } /// ``` @@ -179,7 +196,7 @@ impl Transaction { /// Helper function to convert scan metadata iterator to filtered engine data iterator. /// /// This adapter extracts the `scan_files` field from each [`crate::scan::ScanMetadata`] item, - /// making it easy to pass scan results directly to [`Self::update_deletion_vectors`]. + /// making it easy to pass scan results directly to `update_deletion_vectors`. /// /// # Example /// @@ -199,24 +216,37 @@ impl Transaction { /// Update deletion vectors for files in the table. /// - /// This method can be called multiple times to update deletion vectors for different sets of files. + /// This method can be called multiple times to update deletion vectors for different sets of + /// files. /// /// This method takes a map of file paths to new deletion vector descriptors and an iterator /// of scan file data. It joins the two together internally and will generate appropriate /// remove/add actions on commit to update the deletion vectors. /// + /// Required AddFile fields on matched rows are validated at commit. Staging can therefore + /// succeed for metadata that commit later rejects, including an empty path, negative size, + /// or incorrect physical partition keys. + /// + /// On commit, each matched file's add action carries `stats.tightBounds: false`. + /// /// # Arguments /// - /// * `new_dv_descriptors` - A map from data file path (as provided in scan operations) to - /// the new deletion vector descriptor for that file. - /// * `existing_data_files` - An iterator over FilteredEngineData from scan metadata. The - /// selected elements of each FilteredEngineData must be a superset of the paths that key - /// `new_dv_descriptors`. + /// * `new_dv_descriptors` - A map from data file path (as provided in scan operations) to the + /// new deletion vector descriptor for that file. + /// * `existing_data_files` - An iterator over FilteredEngineData from scan metadata using + /// [`scan_row_schema`]. Selected rows must preserve the scan-file values and cover every path + /// in `new_dv_descriptors`. Files with deletion vectors must have an accurate + /// `stats.numRecords` value. /// /// # Errors /// /// Returns an error if: + /// - The transaction targets table creation instead of an existing table + /// - The table does not have deletion vectors enabled via protocol support and the + /// `delta.enableDeletionVectors=true` table property /// - A file path in `new_dv_descriptors` is not found in `existing_data_files` + /// - A matched file's scan metadata is missing or has invalid `stats.numRecords` + /// - A matched file's `stats` is not valid JSON or is not a JSON object /// /// # Examples /// @@ -256,54 +286,42 @@ impl Transaction { "Deletion vector operations require an existing table", )); } - if !self - .read_snapshot - .table_configuration() - .is_feature_supported(&TableFeature::DeletionVectors) - { - return Err(Error::unsupported( - "Deletion vector operations require reader version 3, writer version 7, \ - and the 'deletionVectors' feature in both reader and writer features", - )); - } + self.ensure_deletion_vectors_enabled()?; let mut matched_dv_files = 0; + let mut matched_files = Vec::new(); let mut visitor = DvMatchVisitor::new(&new_dv_descriptors); // Process each batch of scan file metadata to prepare for DV updates: // 1. Visit rows to match file paths against the DV descriptor map - // 2. Append new DV descriptors as a temporary column to matched files + // 2. Append new DV descriptors and stats as a temporary column to matched files // 3. Update selection vector to only keep files that need DV updates // 4. Cache the result in dv_matched_files for generating remove/add actions during commit for scan_file_result in existing_data_files { let scan_file = scan_file_result?; - visitor.new_dv_entries.clear(); + visitor.new_entries.clear(); visitor.matched_file_indexes.clear(); - let (data, mut selection_vector) = scan_file.into_parts(); - visitor.visit_rows_of(data.as_ref())?; + visitor.visit_rows_of(&scan_file)?; + let (data, _) = scan_file.into_parts(); // Update selection vector to keep only files that matched DV descriptors. // This ensures we only generate remove/add actions for files being updated. - let mut current_matched_index = 0; - for (i, selected) in selection_vector.iter_mut().enumerate() { - if current_matched_index < visitor.matched_file_indexes.len() { - if visitor.matched_file_indexes[current_matched_index] != i { - *selected = false; - } else { - current_matched_index += 1; - matched_dv_files += if *selected { 1 } else { 0 }; - } - } else { - // Deselect any files after the last matched file - *selected = false; - } - } - - let new_columns = vec![ArrayData::try_new( - struct_deletion_vector_schema().clone(), - visitor.new_dv_entries.clone(), - )?]; - self.dv_matched_files.push(FilteredEngineData::try_new( + let selection_vector = + selection_vector_for_matches(data.len(), &visitor.matched_file_indexes); + matched_dv_files += visitor.matched_file_indexes.len(); + + // Append two temporary columns to the scan data: the new DV descriptor and the + // rewritten stats (with `tightBounds: false`). + let (dv_entries, stats_entries): (Vec, Vec) = visitor + .new_entries + .drain(..) + .map(|entry| (entry.deletion_vector, entry.stats)) + .unzip(); + let new_columns = vec![ + ArrayData::try_new(struct_deletion_vector_schema().clone(), dv_entries)?, + ArrayData::try_new(ArrayType::new(DataType::STRING, true), stats_entries)?, + ]; + matched_files.push(FilteredEngineData::try_new( data.append_columns(new_dv_column_schema().clone(), new_columns)?, selection_vector, )?); @@ -317,29 +335,66 @@ impl Transaction { ))); } + self.dv_matched_files.extend(matched_files); + self.num_dv_updates += matched_dv_files; + Ok(()) + } + + /// Verify the table has deletion vectors *enabled* (feature supported in both reader and + /// writer features AND the `delta.enableDeletionVectors` table property set to `true`). + fn ensure_deletion_vectors_enabled(&self) -> DeltaResult<()> { + if !self + .effective_table_config + .is_feature_enabled(&TableFeature::DeletionVectors) + { + return Err(Error::unsupported( + "Deletion vector writes require reader version 3, writer version 7, the \ + 'deletionVectors' feature in both reader and writer features, and the \ + `delta.enableDeletionVectors` table property set to `true`", + )); + } Ok(()) } } +fn selection_vector_for_matches(num_rows: usize, matched_file_indexes: &[usize]) -> Vec { + let mut next_match = 0; + (0..num_rows) + .map(|row_index| { + if matched_file_indexes.get(next_match) == Some(&row_index) { + next_match += 1; + true + } else { + false + } + }) + .collect() +} + // ============================================================================= // Deletion vector schemas and commit helpers // ============================================================================= /// Column name for temporary column used during deletion vector updates. -/// This column holds new DV descriptors appended to scan file metadata before transforming to final add actions. +/// This column holds new DV descriptors appended to scan file metadata before transforming to final +/// add actions. static NEW_DELETION_VECTOR_NAME: &str = "newDeletionVector"; +/// Column name for the temporary column holding the rewritten `stats` string for a DV update. +static NEW_STATS_NAME: &str = "newStats"; + /// Schema for scan row data with an additional column for new deletion vector descriptors. -/// This is an intermediate schema used during deletion vector updates before transforming to final add actions. +/// This is an intermediate schema used during deletion vector updates before transforming to final +/// add actions. static INTERMEDIATE_DV_SCHEMA: LazyLock = LazyLock::new(|| { Arc::new(StructType::new_unchecked( - scan_row_schema() - .fields() - .cloned() - .chain([StructField::nullable( + scan_row_schema().fields().cloned().chain([ + StructField::nullable( NEW_DELETION_VECTOR_NAME.to_string(), DeletionVectorDescriptor::to_schema(), - )]), + ), + StructField::nullable(NEW_STATS_NAME.to_string(), DataType::STRING), + ]), )) }); @@ -350,14 +405,8 @@ fn intermediate_dv_schema() -> &'static SchemaRef { /// Schema for scan row data with nullable statistics fields. /// Used when generating remove actions to ensure statistics can be null if missing. -// Safety: The panic here is acceptable because scan_row_schema() is a known valid schema. -// If transformation fails, it indicates a programmer error in schema construction that should be caught during development. -#[allow(clippy::panic)] -static NULLABLE_SCAN_ROWS_SCHEMA: LazyLock = LazyLock::new(|| { - schema_with_all_fields_nullable(scan_row_schema().as_ref()) - .unwrap_or_else(|_| panic!("Failed to transform scan_row_schema")) - .into() -}); +static NULLABLE_SCAN_ROWS_SCHEMA: LazyLock = + LazyLock::new(|| schema_with_all_fields_nullable(scan_row_schema().as_ref()).into()); /// Returns the nullable scan row schema. fn nullable_scan_rows_schema() -> &'static SchemaRef { @@ -366,30 +415,18 @@ fn nullable_scan_rows_schema() -> &'static SchemaRef { /// Schema for restored add actions with nullable statistics fields. /// Used when transforming scan data back to add actions with potentially missing statistics. -// Safety: The panic here is acceptable because restored_add_schema() is a known valid schema. -// If transformation fails, it indicates a programmer error in schema construction that should be caught during development. -#[allow(clippy::panic)] -static NULLABLE_RESTORED_ADD_SCHEMA: LazyLock = LazyLock::new(|| { - schema_with_all_fields_nullable(restored_add_schema()) - .unwrap_or_else(|_| panic!("Failed to transform restored_add_schema")) - .into() -}); +static NULLABLE_RESTORED_ADD_SCHEMA: LazyLock = + LazyLock::new(|| schema_with_all_fields_nullable(restored_add_schema()).into()); /// Returns the nullable restored add action schema. fn nullable_restored_add_schema() -> &'static SchemaRef { &NULLABLE_RESTORED_ADD_SCHEMA } -/// Schema for add actions that is nullable for use in transforms as as a workaround to avoid issues with null values in required fields -/// that aren't selected. -// Safety: The panic here is acceptable because add_log_schema is a known valid schema. -// If transformation fails, it indicates a programmer error in schema construction that should be caught during development. -#[allow(clippy::panic)] -static NULLABLE_ADD_LOG_SCHEMA: LazyLock = LazyLock::new(|| { - schema_with_all_fields_nullable(get_log_add_schema()) - .unwrap_or_else(|_| panic!("Failed to transform nullable_restored_add_schema")) - .into() -}); +/// Schema for add actions that is nullable for use in transforms as as a workaround to avoid issues +/// with null values in required fields that aren't selected. +static NULLABLE_ADD_LOG_SCHEMA: LazyLock = + LazyLock::new(|| schema_with_all_fields_nullable(&LOG_ADD_SCHEMA).into()); /// Returns the schema for nullable restored add actions with dataChange field. /// This schema extends the nullable restored add schema with a dataChange boolean field @@ -402,7 +439,7 @@ fn nullable_add_log_schema() -> &'static SchemaRef { /// Used when appending DV columns to scan file data. #[cfg_attr(not(feature = "internal-api"), allow(dead_code))] static STRUCT_DELETION_VECTOR_SCHEMA: LazyLock = - LazyLock::new(|| ArrayType::new(DeletionVectorDescriptor::to_schema().into(), true)); + LazyLock::new(|| ArrayType::new(DeletionVectorDescriptor::to_schema(), true)); /// Returns the schema for an array of deletion vector descriptors. #[cfg_attr(not(feature = "internal-api"), allow(dead_code))] @@ -414,10 +451,13 @@ fn struct_deletion_vector_schema() -> &'static ArrayType { /// This temporary column is dropped during transformation to final add actions. #[cfg_attr(not(feature = "internal-api"), allow(dead_code))] static NEW_DV_COLUMN_SCHEMA: LazyLock = LazyLock::new(|| { - Arc::new(StructType::new_unchecked(vec![StructField::nullable( - NEW_DELETION_VECTOR_NAME, - DeletionVectorDescriptor::to_schema(), - )])) + Arc::new(StructType::new_unchecked(vec![ + StructField::nullable( + NEW_DELETION_VECTOR_NAME, + DeletionVectorDescriptor::to_schema(), + ), + StructField::nullable(NEW_STATS_NAME, DataType::STRING), + ])) }); /// Returns the schema for the intermediate column holding new DV descriptors. @@ -429,12 +469,12 @@ fn new_dv_column_schema() -> &'static SchemaRef { // These methods are generic over the transaction state `S` because they are called from the // shared `commit()` path in `mod.rs` (`impl Transaction`). DV updates can only be // populated on `ExistingTableTransaction`, so the `is_create_table()` guard below is -// defence-in-depth against future misuse. +// defense-in-depth against future misuse. impl Transaction { /// Generate remove/add action pairs for files with DV updates. /// - /// This method processes the cached matched files, generating the necessary Remove and Add actions. - /// For each file: + /// This method processes the cached matched files, generating the necessary Remove and Add + /// actions. For each file: /// 1. A Remove action is generated for the old file /// 2. An Add action is generated with the new DV descriptor pub(super) fn generate_dv_update_actions<'a>( @@ -448,7 +488,8 @@ impl Transaction { )); } - static COLUMNS_TO_DROP: &[&str] = &[NEW_DELETION_VECTOR_NAME]; + // The rewritten stats are for the add action only, so they are dropped here. + static COLUMNS_TO_DROP: &[&str] = &[NEW_DELETION_VECTOR_NAME, NEW_STATS_NAME]; let remove_actions = self.generate_remove_actions(engine, self.dv_matched_files.iter(), COLUMNS_TO_DROP)?; let add_actions = self.generate_adds_for_dv_update(engine, self.dv_matched_files.iter())?; @@ -465,21 +506,22 @@ impl Transaction { file_metadata_batch: impl Iterator + Send + 'a, ) -> DeltaResult> + Send + 'a> { let evaluation_handler = engine.evaluation_handler(); - // Transform to replace the deletionVector field with the new DV from NEW_DELETION_VECTOR_NAME, - // then drop the NEW_DELETION_VECTOR_NAME column. The engine data has this temporary column - // appended by update_deletion_vectors(), but it is not expected by the transforms used in - // generate_remove_actions() which expect only the scan row schema fields. - let with_new_dv_transform = Expression::transform( - Transform::new_top_level() - .with_replaced_field( - "deletionVector", - Expression::column([NEW_DELETION_VECTOR_NAME]).into(), - ) - .with_dropped_field(NEW_DELETION_VECTOR_NAME), - ); + // Struct patch to replace the deletionVector field with the new DV/stats from + // NEW_DELETION_VECTOR_NAME/NEW_STATS_NAME, then drop the + // NEW_DELETION_VECTOR_NAME/NEW_STATS_NAME columns. The engine data has this + // temporary column appended by update_deletion_vectors(), but it is not expected by + // the transforms used in generate_remove_actions() which expect only the scan row + // schema fields. + let with_new_dv_expr = Expression::struct_patch( + ExpressionStructPatchBuilder::new() + .replace("deletionVector", col!(NEW_DELETION_VECTOR_NAME)) + .replace("stats", col!(NEW_STATS_NAME)) + .drop(NEW_DELETION_VECTOR_NAME) + .drop(NEW_STATS_NAME), + )?; let with_new_dv_eval = evaluation_handler.new_expression_evaluator( intermediate_dv_schema().clone(), - Arc::new(with_new_dv_transform), + Arc::new(with_new_dv_expr), nullable_scan_rows_schema().clone().into(), )?; let restored_add_eval = evaluation_handler.new_expression_evaluator( @@ -487,16 +529,14 @@ impl Transaction { get_scan_metadata_transform_expr(), nullable_restored_add_schema().clone().into(), )?; - let with_data_change_transform = - Arc::new(Expression::struct_from([Expression::transform( - Transform::new_nested(["add"]).with_inserted_field( - Some("modificationTime"), - Expression::literal(self.data_change).into(), - ), - )])); + let with_data_change_patch = Expression::struct_patch( + ExpressionStructPatchBuilder::new_nested(["add"]) + .insert_after("modificationTime", lit(self.data_change)), + )?; + let with_data_change_expr = Arc::new(Expression::struct_from([with_data_change_patch])); let with_data_change_eval = evaluation_handler.new_expression_evaluator( nullable_restored_add_schema().clone(), - with_data_change_transform, + with_data_change_expr, nullable_add_log_schema().clone().into(), )?; Ok(file_metadata_batch.map( @@ -521,14 +561,35 @@ impl Transaction { // DvMatchVisitor: matches file paths from scan data against new DV descriptors // ============================================================================= +/// One visited row's new deletion vector descriptor paired with its rewritten stats. +struct DvUpdateEntry { + deletion_vector: Scalar, + stats: Scalar, +} + +impl DvUpdateEntry { + /// The (null DV, null stats) entry for rows that are not getting a DV update. + fn null() -> Self { + static NULL_DV: LazyLock = + LazyLock::new(|| Scalar::Null(DataType::from(DeletionVectorDescriptor::to_schema()))); + static NULL_STATS: LazyLock = LazyLock::new(|| Scalar::Null(DataType::STRING)); + Self { + deletion_vector: NULL_DV.clone(), + stats: NULL_STATS.clone(), + } + } +} + /// Visitor that matches file paths from scan data against new deletion vector descriptors. /// Used by update_deletion_vectors() to attach new DV descriptors to scan file metadata. #[cfg_attr(not(feature = "internal-api"), allow(dead_code))] struct DvMatchVisitor<'a> { /// Map from file path to the new deletion vector descriptor for that file dv_updates: &'a HashMap, - /// Accumulated DV descriptors (or nulls) for each visited row, in visit order - new_dv_entries: Vec, + /// Accumulated (DV descriptor, rewritten stats) pairs for each visited row, in visit order. + /// Matched rows carry the new DV and their original stats with `tightBounds` forced to + /// `false`. + new_entries: Vec, /// Indexes of rows that matched a file path in dv_update. These must be in /// ascending order matched_file_indexes: Vec, @@ -537,51 +598,104 @@ struct DvMatchVisitor<'a> { impl<'a> DvMatchVisitor<'a> { #[cfg_attr(not(feature = "internal-api"), allow(dead_code))] const PATH_INDEX: usize = 0; + #[cfg_attr(not(feature = "internal-api"), allow(dead_code))] + const STATS_INDEX: usize = 1; /// Creates a new DvMatchVisitor that will match file paths against the provided DV updates map. #[cfg_attr(not(feature = "internal-api"), allow(dead_code))] fn new(dv_updates: &'a HashMap) -> Self { Self { dv_updates, - new_dv_entries: Vec::new(), + new_entries: Vec::new(), matched_file_indexes: Vec::new(), } } } -/// A `RowVisitor` that matches file paths against the provided DV updates map. -impl RowVisitor for DvMatchVisitor<'_> { +/// A `FilteredRowVisitor` that matches file paths against the provided DV updates map. +impl FilteredRowVisitor for DvMatchVisitor<'_> { fn selected_column_names_and_types(&self) -> (&'static [ColumnName], &'static [DataType]) { static NAMES_AND_TYPES: LazyLock<(Vec, Vec)> = LazyLock::new(|| { - let names = vec![column_name!("path")]; - let types = vec![DataType::STRING]; + let names = vec![column_name!("path"), column_name!("stats")]; + let types = vec![DataType::STRING, DataType::STRING]; (names, types) }); (&NAMES_AND_TYPES.0, &NAMES_AND_TYPES.1) } - /// For each path checks if it is in the hash-map and if it is, extract DV - /// details that can be appended back to the EngineData. Also track matched - /// rows so the selected rows can be updated to only contain matches. - fn visit<'a>(&mut self, row_count: usize, getters: &[&'a dyn GetData<'a>]) -> DeltaResult<()> { - for i in 0..row_count { - // Use get_opt since path is nullable in the schema - let path_opt: Option = getters[Self::PATH_INDEX].get_opt(i, "path")?; - - // Skip rows with null paths (these are rows that were deselected by the selection vector, - // but still appear in the EngineData since visitors operate on the full EngineData) + /// For each selected row checks if the path is in the hash-map and if so, extracts DV + /// details that can be appended back to the EngineData. Also tracks matched row indexes + /// so the selection vector can be updated to contain only DV-matched files. + fn visit_filtered<'a>( + &mut self, + getters: &[&'a dyn GetData<'a>], + rows: RowIndexIterator<'_>, + ) -> DeltaResult<()> { + static DV_SCHEMA_FIELDS: LazyLock> = LazyLock::new(|| { + DeletionVectorDescriptor::to_schema() + .into_fields() + .collect() + }); + let num_rows = rows.num_rows(); + self.new_entries.reserve(num_rows); + for row_index in rows { + // Fill in null entries for any deselected rows before this one. + self.new_entries.resize_with(row_index, DvUpdateEntry::null); + let path_opt: Option = getters[Self::PATH_INDEX].get_opt(row_index, "path")?; let Some(path) = path_opt else { - self.new_dv_entries.push(Scalar::Null(DataType::from( - DeletionVectorDescriptor::to_schema(), - ))); + // Null path means a non-add action row (remove, metadata, etc.) + self.new_entries.push(DvUpdateEntry::null()); continue; }; - if let Some(dv_result) = self.dv_updates.get(&path) { - self.new_dv_entries.push(Scalar::Struct(StructData::try_new( - DeletionVectorDescriptor::to_schema() - .into_fields() - .collect(), + // DVs require an accurate numRecords stat per the Delta protocol. + let stats: Option = + getters[Self::STATS_INDEX].get_opt(row_index, "stats")?; + let stats = stats.ok_or_else(|| { + Error::generic(format!( + "update_deletion_vectors: file {path} has no stats; \ + deletion vectors require an accurate {NUM_RECORDS}" + )) + })?; + let mut parsed: serde_json::Value = serde_json::from_str(&stats).map_err(|e| { + Error::generic(format!( + "update_deletion_vectors: stats for {path} is not valid JSON: {e}" + )) + })?; + let stats_obj = parsed.as_object_mut().ok_or_else(|| { + Error::generic(format!( + "update_deletion_vectors: stats for {path} is not a JSON object" + )) + })?; + if stats_obj + .get(NUM_RECORDS) + .and_then(serde_json::Value::as_u64) + .is_none() + { + return Err(Error::generic(format!( + "update_deletion_vectors: stats for {path} is missing {NUM_RECORDS} \ + or it is not a non-negative integer" + ))); + } + + // Widen tightBounds to false (unless already false) instead of recomputing the + // stats. + let already_wide = stats_obj + .get(TIGHT_BOUNDS) + .and_then(serde_json::Value::as_bool) + == Some(false); + let new_stats = if already_wide { + stats + } else { + stats_obj.insert(TIGHT_BOUNDS.to_string(), serde_json::Value::Bool(false)); + serde_json::to_string(&parsed).map_err(|e| { + Error::generic(format!( + "update_deletion_vectors: failed to re-serialize stats for {path}: {e}" + )) + })? + }; + let deletion_vector = Scalar::Struct(StructData::try_new( + DV_SCHEMA_FIELDS.clone(), vec![ Scalar::from(dv_result.storage_type.to_string()), Scalar::from(dv_result.path_or_inline_dv.clone()), @@ -589,14 +703,114 @@ impl RowVisitor for DvMatchVisitor<'_> { Scalar::from(dv_result.size_in_bytes), Scalar::from(dv_result.cardinality), ], - )?)); - self.matched_file_indexes.push(i); + )?); + self.new_entries.push(DvUpdateEntry { + deletion_vector, + stats: Scalar::from(new_stats), + }); + self.matched_file_indexes.push(row_index); } else { - self.new_dv_entries.push(Scalar::Null(DataType::from( - DeletionVectorDescriptor::to_schema(), - ))); + self.new_entries.push(DvUpdateEntry::null()); } } + // Pad with trailing null entries for any deselected rows at the end. + self.new_entries.resize_with(num_rows, DvUpdateEntry::null); Ok(()) } } + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + use crate::actions::deletion_vector::DeletionVectorStorageType; + use crate::arrow::array::{ArrayRef, StringArray}; + use crate::arrow::datatypes::{DataType as ArrowDataType, Field as ArrowField, Schema}; + use crate::arrow::record_batch::RecordBatch; + use crate::engine::arrow_data::ArrowEngineData; + + const TEST_PATH: &str = "data/file.parquet"; + + /// Build a single-row scan-metadata-like batch. `path` and `stats` are addressed by name + /// (matching `DvMatchVisitor::selected_column_names_and_types`), so the surrounding + /// `extra_columns` and their positions deliberately exercise name-based getter binding. + fn make_scan_metadata_row( + path: &str, + stats: Option<&str>, + extra_columns: &[&str], + ) -> FilteredEngineData { + let mut fields = vec![ + ArrowField::new("path", ArrowDataType::Utf8, true), + ArrowField::new("stats", ArrowDataType::Utf8, true), + ]; + let mut columns: Vec = vec![ + Arc::new(StringArray::from(vec![Some(path)])), + Arc::new(StringArray::from(vec![stats])), + ]; + for name in extra_columns { + fields.push(ArrowField::new(*name, ArrowDataType::Utf8, true)); + columns.push(Arc::new(StringArray::from(vec![Some("ignored")]))); + } + let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns).unwrap(); + FilteredEngineData::with_all_rows_selected(Box::new(ArrowEngineData::new(batch))) + } + + fn dv_updates_for(path: &str) -> HashMap { + let descriptor = DeletionVectorDescriptor::try_new( + DeletionVectorStorageType::Inline, + "AAAAAAAA".to_string(), + None, + 8, + 0, + ) + .unwrap(); + HashMap::from([(path.to_string(), descriptor)]) + } + + #[rstest] + #[case::null_stats(None, &[], Some(NUM_RECORDS))] + #[case::stats_missing_num_records(Some("{}"), &[], Some(NUM_RECORDS))] + #[case::stats_not_json_object(Some("5"), &[], Some("not a JSON object"))] + #[case::stats_invalid_json(Some("{not json"), &[], Some("not valid JSON"))] + #[case::stats_with_num_records(Some(r#"{"numRecords":10}"#), &[], None)] + #[case::extra_columns_dont_shift_indexes( + Some(r#"{"numRecords":10}"#), + &["size", "modificationTime"], + None, + )] + fn dv_match_visitor_validates_matched_row_stats( + #[case] stats: Option<&str>, + #[case] extra_columns: &[&str], + #[case] expected_err: Option<&str>, + ) { + let dv_updates = dv_updates_for(TEST_PATH); + let mut visitor = DvMatchVisitor::new(&dv_updates); + let data = make_scan_metadata_row(TEST_PATH, stats, extra_columns); + + let result = visitor.visit_rows_of(&data); + match expected_err { + Some(sub_str) => { + let msg = result.expect_err("expected error").to_string(); + assert!(msg.contains(sub_str), "message was: {msg}"); + assert!(msg.contains(TEST_PATH), "message was: {msg}"); + } + None => { + result.expect("validation should pass"); + assert_eq!(visitor.matched_file_indexes, vec![0]); + } + } + } + + #[test] + fn dv_match_visitor_skips_validation_for_unmatched_rows() { + let dv_updates = dv_updates_for("other/file.parquet"); + let mut visitor = DvMatchVisitor::new(&dv_updates); + let data = make_scan_metadata_row(TEST_PATH, None, &[]); + + visitor + .visit_rows_of(&data) + .expect("unmatched rows must not trigger stats validation"); + assert!(visitor.matched_file_indexes.is_empty()); + } +} diff --git a/kernel/src/transaction/write_context.rs b/kernel/src/transaction/write_context.rs new file mode 100644 index 0000000000..242a56bdc1 --- /dev/null +++ b/kernel/src/transaction/write_context.rs @@ -0,0 +1,623 @@ +use std::collections::HashMap; +use std::num::NonZero; +use std::sync::Arc; + +use rand::Rng; +use url::Url; + +use crate::actions::deletion_vector::DeletionVectorPath; +use crate::expressions::{ColumnName, ExpressionRef}; +use crate::partition::hive::{build_partition_path, uri_encode_path}; +use crate::schema::SchemaRef; +use crate::table_features::ColumnMappingMode; +use crate::{DeltaResult, Error}; + +/// Table-wide write state shared across all [`WriteContext`] instances created by a +/// [`Transaction`]. Holds the target directory, schemas, column mapping mode, stats columns, +/// logical partition column names, and randomized-prefix configuration. +/// +/// [`Transaction`]: super::Transaction +#[derive(Debug)] +pub(super) struct SharedWriteState { + pub(super) table_root: Url, + /// Logical schema of the data to write: the table schema minus partition columns. + pub(super) logical_schema: SchemaRef, + pub(super) physical_schema: SchemaRef, + pub(super) column_mapping_mode: ColumnMappingMode, + pub(super) stats_columns: Vec, + /// Logical partition column names in metadata-defined order. + pub(super) logical_partition_columns: Vec, + /// Resolved value of the `delta.randomizeFilePrefixes` table property. When true, + /// [`WriteContext::write_dir`] emits a random alphanumeric prefix regardless of column + /// mapping mode. + pub(super) randomize_file_prefixes: bool, + /// Resolved value of the `delta.randomPrefixLength` table property. Drives the length + /// of the random prefix in [`WriteContext::write_dir`] for both the column mapping and + /// `randomizeFilePrefixes` paths. + pub(super) random_prefix_length: NonZero, +} + +/// A write context for a specific partition or an unpartitioned table. Created by +/// [`Transaction::partitioned_write_context`] or [`Transaction::unpartitioned_write_context`]. +/// +/// Note: clustered tables are unpartitioned and use `unpartitioned_write_context`. +/// +/// Contains both table-wide state (shared cheaply via `Arc`) and per-partition state +/// (serialized partition values with physical column names as keys). How you use a +/// `WriteContext` depends on your engine: +/// +/// - **`DefaultEngine` consumers**: pass this to `DefaultEngine::write_parquet`, which handles +/// everything (transform, write, partition metadata). +/// - **Arrow-based custom engines**: write parquet yourself, then call `build_add_file_metadata` +/// with the resulting `DataFileMetadata` and this `WriteContext` to produce the Add action +/// `EngineData` for [`Transaction::add_files`]. +/// - **Fully custom (non-Arrow) engines**: use [`physical_partition_values`] to build the +/// `partitionValues` map in Add actions directly. +/// +/// [`Transaction::partitioned_write_context`]: super::Transaction::partitioned_write_context +/// [`Transaction::unpartitioned_write_context`]: super::Transaction::unpartitioned_write_context +/// [`Transaction::add_files`]: super::Transaction::add_files +/// [`physical_partition_values`]: WriteContext::physical_partition_values +#[derive(Debug)] +pub struct WriteContext { + pub(super) shared: Arc, + /// Transforms logical data to physical data for writing. The logical data must not contain + /// any partition columns. The expression injects the partition columns when needed. + pub(super) logical_to_physical: ExpressionRef, + /// Physical column name -> serialized value (`None` = null partition value). + /// Empty for unpartitioned tables. Ordering for hive-style paths comes from + /// `shared.logical_partition_columns`, not from this map. + pub(super) physical_partition_values: HashMap>, +} + +impl WriteContext { + /// Returns the table root URL. + pub fn table_root_dir(&self) -> &Url { + &self.shared.table_root + } + + /// Returns the recommended directory URL for writing Parquet data files. Connectors + /// should write files as `/.parquet`. Not strictly required (data files + /// can live anywhere under the table root), but produces the conventional layout. + /// + /// # The returned URL is URI-encoded + /// + /// For CM=none partitioned tables, the Hive-escaped partition prefix is double-encoded + /// in the URL (e.g. `%3A` appears as `%253A`). Concrete examples for a single STRING + /// partition column `p`: + /// + /// ```text + /// partition value | encoded path prefix | URI-decoded (filesystem path) + /// -----------------+----------------------------+-------------------------------- + /// "abc" | p=abc/ | p=abc/ + /// "a%c" | p=a%2525c/ | p=a%25c/ + /// "a " | p=a%20/ | p=a / + /// ``` + /// + /// On Windows, the Hive layer additionally escapes space, so `"a "` produces encoded + /// path prefix `p=a%2520/` with filesystem directory `p=a%20/`. + /// + /// The same URL drives two outputs and custom engines must handle each correctly: + /// + /// 1. **Filesystem write path** — URI-decode once to get the on-disk directory name. Custom + /// engines MUST decode before feeding `url.path()` to an OS-filesystem API. Use + /// `object_store::path::Path::from_url_path` or an equivalent decoder. Feeding `url.path()` + /// directly to the filesystem produces directories literally named `p=a%253Ab/` and breaks + /// interop with every other Delta writer. + /// + /// 2. **`add.path` in the Delta log** — keep the URL URI-encoded. After writing the parquet + /// file, pass the full (still-encoded) file URL — this URL plus the generated filename — to + /// [`WriteContext::resolve_file_path`] to produce `add.path`. `make_relative` preserves the + /// URI-encoded form, which is what the Delta protocol requires. Arrow-based engines can use + /// `build_add_file_metadata` which handles this step. + /// + /// `DefaultEngine::write_parquet` handles both steps automatically via `object_store` + /// and `build_add_file_metadata`. + /// + /// # Layout + /// + /// A random alphanumeric prefix is emitted whenever column mapping is on or the + /// `delta.randomizeFilePrefixes` table property is true. The prefix length is + /// controlled by `delta.randomPrefixLength`. When a random prefix is used on a + /// partitioned table, Hive-style path components are suppressed; the partition + /// values are still recorded in `add.partitionValues`. + /// + /// ```text + /// | randomize=false | randomize=true + /// ---------------------------|-------------------------------------|-------------------------------- + /// CM=None, unpartitioned | /.parquet | //.parquet + /// CM=None, partitioned | /col=val/.../.pq | //.parquet + /// CM=Id/Name, any | //.parquet| //.parquet + /// ``` + /// + /// Each call generates a fresh prefix. The alphanumeric charset is RFC 3986 + /// unreserved, so the prefix is URI-safe at any length. + // TODO(#2436): revisit this API shape. Returning a `Url` forces callers to URI-decode + // before filesystem writes and keep it encoded for `add.path`, which is unintuitive. + pub fn write_dir(&self) -> Url { + let mut url = self.shared.table_root.clone(); + // A random prefix is used when column mapping is on (to avoid leaking physical + // UUID column names into paths) or when `delta.randomizeFilePrefixes` is set (to + // avoid S3 hotspots). When a random prefix is used, the Hive-style partition path + // is suppressed; partition values are recorded in `add.partitionValues` instead. + let should_prefix = self.shared.column_mapping_mode != ColumnMappingMode::None + || self.shared.randomize_file_prefixes; + if should_prefix { + // The alphanumeric charset is RFC 3986 unreserved, so the prefix is URI-safe + // as-is and needs no further encoding. + let prefix = random_alphanumeric_prefix(self.shared.random_prefix_length); + url.set_path(&format!("{}{}/", url.path(), prefix)); + } else if !self.shared.logical_partition_columns.is_empty() { + // CM=None, no randomization: emit a Hive-style partition path. URI-encode on + // top of Hive-escaping because the fn-level contract (see doc above) requires + // callers to URI-decode once before using the URL as a filesystem path. That + // decode recovers the Hive-escaped form, which is the on-disk layout + // Delta-Spark and kernel-java produce. + let hive_escaped = self.hive_partition_path_suffix(); + let uri_encoded = uri_encode_path(&hive_escaped); + url.set_path(&format!("{}{}", url.path(), uri_encoded)); + } + url + } + + /// Returns the schema which connectors' logical data should conform to. + pub fn logical_schema(&self) -> &SchemaRef { + &self.shared.logical_schema + } + + /// Returns the physical schema (partition columns removed if applicable, column mapping + /// applied). Partition columns are kept when `materializePartitionColumns` is enabled. + pub fn physical_schema(&self) -> &SchemaRef { + &self.shared.physical_schema + } + + /// Returns the expression that transforms logical data to physical data for writing. + pub fn logical_to_physical(&self) -> ExpressionRef { + self.logical_to_physical.clone() + } + + /// The [`ColumnMappingMode`] for this table. + pub fn column_mapping_mode(&self) -> ColumnMappingMode { + self.shared.column_mapping_mode + } + + /// Returns the column names that should have statistics collected during writes. + /// + /// Based on table configuration (dataSkippingNumIndexedCols, dataSkippingStatsColumns). + pub fn stats_columns(&self) -> &[ColumnName] { + &self.shared.stats_columns + } + + /// Returns the serialized partition values for this write context. Keys are physical + /// column names; values are protocol-serialized strings (`None` = null). + /// + /// For unpartitioned tables, this is empty. + pub fn physical_partition_values(&self) -> &HashMap> { + &self.physical_partition_values + } + + /// Builds the Hive-style partition path suffix (e.g., `year=2024/region=US/`). + /// Only called when column mapping is OFF. + fn hive_partition_path_suffix(&self) -> String { + debug_assert!( + self.shared.column_mapping_mode == ColumnMappingMode::None, + "Hive-style paths should only be used when column mapping is OFF" + ); + let columns: Vec<(&str, Option<&str>)> = self + .shared + .logical_partition_columns + .iter() + .map(|logical_name| { + // CM is None, so physical == logical. Use the logical name as both the + // directory name (e.g. "year" in "year=2024/") and the key into + // physical_partition_values. + let value = self + .physical_partition_values + .get(logical_name.as_str()) + .and_then(|v| v.as_deref()); + (logical_name.as_str(), value) + }) + .collect(); + build_partition_path(&columns) + } + + /// Computes the relative `add.path` value for the Delta log from a file's absolute URL. + /// + /// Custom engines that write parquet files themselves (bypassing + /// `DefaultEngine::write_parquet`) should call this after writing each file to produce + /// the path for their Add action metadata. + /// + /// # Examples + /// + /// Given a table root of `s3://bucket/table/`: + /// - `s3://bucket/table/abc.parquet` -> `"abc.parquet"` + /// - `s3://bucket/table/year=2024/abc.parquet` -> `"year=2024/abc.parquet"` + /// + /// Returns an error if the file is not under the table root. + pub fn resolve_file_path(&self, file_location: &Url) -> DeltaResult { + let relative = self + .shared + .table_root + .make_relative(file_location) + .ok_or_else(|| { + Error::internal_error(format!( + "file '{}' is not under table root '{}'", + file_location, self.shared.table_root + )) + })?; + if relative.starts_with("..") { + return Err(Error::internal_error(format!( + "file '{}' is not under table root '{}'", + file_location, self.shared.table_root + ))); + } + Ok(relative) + } + + /// Generate a new unique absolute URL for a deletion vector file. + /// + /// This method generates a unique file name in the table directory. + /// Each call to this method returns a new unique path. + /// + /// # Arguments + /// + /// * `random_prefix` - A random prefix to use for the deletion vector file name. Making this + /// non-empty can help distributed load on object storage when writing/reading to avoid + /// throttling. Typically a random string of 2-4 characters is sufficient for this purpose. + /// + /// # Examples + /// + /// ```rust,ignore + /// let write_context = transaction.unpartitioned_write_context()?; + /// let dv_path = write_context.new_deletion_vector_path(String::from(rand_string())); + /// ``` + // TODO(#2357): generate the random prefix internally based on table properties + // (delta.randomizeFilePrefixes / delta.randomPrefixLength) instead of requiring the + // caller to pass it. Connectors that need custom paths can use table_root_dir() directly. + pub fn new_deletion_vector_path(&self, random_prefix: String) -> DeletionVectorPath { + DeletionVectorPath::new(self.shared.table_root.clone(), random_prefix) + } +} + +/// Generates a random alphanumeric prefix of the given length for data file directory paths. +/// Used to avoid S3 hotspots and to keep physical UUID column names out of paths when column +/// mapping is enabled. +fn random_alphanumeric_prefix(len: NonZero) -> String { + const CHARSET: &[u8] = b"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"; + let mut rng = rand::rng(); + (0..len.get()) + .map(|_| CHARSET[rng.random_range(0..CHARSET.len())] as char) + .collect() +} + +#[cfg(test)] +mod tests { + use std::collections::HashSet; + use std::sync::Arc; + + use rstest::rstest; + + use super::*; + use crate::expressions::Expression; + use crate::schema::schema_ref; + + fn make_write_context( + cm_mode: ColumnMappingMode, + partition_columns: Vec, + partition_values: HashMap>, + randomize_file_prefixes: bool, + random_prefix_length: usize, + ) -> WriteContext { + let schema = schema_ref! { nullable "value": INTEGER }; + let shared = Arc::new(SharedWriteState { + table_root: Url::parse("s3://bucket/table/").unwrap(), + logical_schema: schema.clone(), + physical_schema: schema.clone(), + column_mapping_mode: cm_mode, + stats_columns: vec![], + logical_partition_columns: partition_columns, + randomize_file_prefixes, + random_prefix_length: NonZero::new(random_prefix_length) + .expect("test prefix length must be > 0"), + }); + WriteContext { + shared, + logical_to_physical: Arc::new(Expression::literal(true)), + physical_partition_values: partition_values, + } + } + + /// Tests the cross product of ColumnMappingMode x partitioned/unpartitioned. + #[rstest] + fn test_write_dir_structure( + #[values( + ColumnMappingMode::None, + ColumnMappingMode::Name, + ColumnMappingMode::Id + )] + cm_mode: ColumnMappingMode, + #[values(true, false)] is_partitioned: bool, + ) { + let (cols, pvs) = if is_partitioned { + ( + vec!["year".into(), "month".into()], + HashMap::from([ + ("year".into(), Some("2024".into())), + ("month".into(), Some("03".into())), + ]), + ) + } else { + (vec![], HashMap::new()) + }; + let wc = make_write_context(cm_mode, cols, pvs, false, 2); + let path = wc.write_dir().path().to_string(); + + match cm_mode { + ColumnMappingMode::None if !is_partitioned => { + assert_eq!( + path, "/table/", + "CM off, unpartitioned: should be table root" + ); + } + ColumnMappingMode::None => { + assert_eq!( + path, "/table/year=2024/month=03/", + "CM off, partitioned: full Hive-style path" + ); + } + ColumnMappingMode::Name | ColumnMappingMode::Id => { + assert!( + !path.contains("year="), + "CM on: should NOT contain Hive-style dirs, got: {path}" + ); + // Path should be /table/<2-char-alphanumeric>/ regardless of partitioning. + let prefix_dir = path + .strip_prefix("/table/") + .unwrap() + .strip_suffix('/') + .unwrap(); + assert_eq!( + prefix_dir.len(), + 2, + "expected 2-char prefix, got: {prefix_dir}" + ); + assert!( + prefix_dir.chars().all(|c| c.is_ascii_alphanumeric()), + "prefix should be alphanumeric, got: {prefix_dir}" + ); + } + } + } + + #[test] + fn test_write_dir_cm_on_generates_different_prefixes_per_call() { + let wc = make_write_context(ColumnMappingMode::Name, vec![], HashMap::new(), false, 2); + let dirs: Vec = (0..20).map(|_| wc.write_dir().path().to_string()).collect(); + let unique: HashSet<_> = dirs.iter().collect(); + assert!( + unique.len() > 1, + "20 calls should produce at least 2 distinct prefixes" + ); + } + + #[test] + fn test_write_dir_cm_off_partitioned_null_value_uses_hive_default() { + let wc = make_write_context( + ColumnMappingMode::None, + vec!["region".into()], + HashMap::from([("region".into(), None)]), + false, + 2, + ); + let path = wc.write_dir().path().to_string(); + assert!( + path.contains("__HIVE_DEFAULT_PARTITION__"), + "null partition value should use HIVE_DEFAULT_PARTITION, got: {path}" + ); + } + + /// CM=None + partitioned: `write_dir` applies URI encoding on top of Hive escaping, so + /// a Hive `%XX` sequence in the suffix comes out as `%25XX` in the URL path. Guards + /// against a refactor that drops the URI layer; such a regression would silently leave + /// the previous Hive-only layout, which every other Delta writer misreads. + #[rstest] + #[case::colon("p", "2025-03-31T15:30:00Z", "/table/p=2025-03-31T15%253A30%253A00Z/")] + #[case::slash("region", "US/East", "/table/region=US%252FEast/")] + #[case::percent_literal("col", "100%", "/table/col=100%2525/")] + fn test_write_dir_cm_off_partitioned_double_encodes_hive_output( + #[case] col: &str, + #[case] value: &str, + #[case] expected_path: &str, + ) { + let wc = make_write_context( + ColumnMappingMode::None, + vec![col.into()], + HashMap::from([(col.into(), Some(value.into()))]), + false, + 2, + ); + assert_eq!(wc.write_dir().path(), expected_path); + } + + /// CM=Id/Name: `write_dir` emits the random prefix verbatim into the URL path. The + /// prefix charset is guarded by `test_random_alphanumeric_prefix_format`; this test + /// just checks that the CM-on arm doesn't introduce a `%` or mangle the prefix. + #[rstest] + #[case::name_mode(ColumnMappingMode::Name)] + #[case::id_mode(ColumnMappingMode::Id)] + fn test_write_dir_cm_on_prefix_is_uri_safe(#[case] cm_mode: ColumnMappingMode) { + let wc = make_write_context(cm_mode, vec!["p".into()], HashMap::new(), false, 2); + let path = wc.write_dir().path().to_string(); + assert!( + !path.contains('%'), + "CM-on path must not contain '%': {path}" + ); + let prefix = path + .strip_prefix("/table/") + .unwrap() + .strip_suffix('/') + .unwrap(); + assert!( + prefix.chars().all(|c| c.is_ascii_alphanumeric()), + "prefix should be URI-safe: {prefix:?}" + ); + } + + #[rstest] + #[case(1)] + #[case(2)] + #[case(8)] + #[case(32)] + fn test_random_alphanumeric_prefix_format(#[case] len: usize) { + let nz_len = NonZero::new(len).unwrap(); + for _ in 0..100 { + let prefix = random_alphanumeric_prefix(nz_len); + assert_eq!(prefix.len(), len, "prefix should be exactly {len} chars"); + assert!( + prefix.chars().all(|c| c.is_ascii_alphanumeric()), + "prefix should be alphanumeric, got: {prefix}" + ); + } + } + + /// When CM is off and `randomizeFilePrefixes=true`, the random prefix replaces the + /// Hive-style path on disk for partitioned tables. Combined with the existing CM-on + /// behavior, this covers the full matrix described in the `write_dir` doc comment. + #[rstest] + fn test_write_dir_with_randomize_property( + #[values( + ColumnMappingMode::None, + ColumnMappingMode::Name, + ColumnMappingMode::Id + )] + cm_mode: ColumnMappingMode, + #[values(true, false)] randomize: bool, + #[values(true, false)] is_partitioned: bool, + ) { + let (cols, pvs) = if is_partitioned { + ( + vec!["year".into()], + HashMap::from([("year".into(), Some("2024".into()))]), + ) + } else { + (vec![], HashMap::new()) + }; + let wc = make_write_context(cm_mode, cols, pvs, randomize, 2); + let path = wc.write_dir().path().to_string(); + + let should_prefix = cm_mode != ColumnMappingMode::None || randomize; + if should_prefix { + assert!( + !path.contains("year="), + "random prefix should suppress Hive dirs, got: {path}" + ); + let prefix = path + .strip_prefix("/table/") + .unwrap() + .strip_suffix('/') + .unwrap(); + assert_eq!(prefix.len(), 2, "expected 2-char prefix, got: {prefix}"); + assert!( + prefix.chars().all(|c| c.is_ascii_alphanumeric()), + "prefix should be alphanumeric, got: {prefix}" + ); + } else if is_partitioned { + assert_eq!(path, "/table/year=2024/"); + } else { + assert_eq!(path, "/table/"); + } + } + + /// `randomPrefixLength` controls the prefix length for both the column-mapping path + /// and the `randomizeFilePrefixes` path. + #[rstest] + #[case::cm_on_default(ColumnMappingMode::Name, false, 2)] + #[case::cm_on_short(ColumnMappingMode::Name, false, 1)] + #[case::cm_on_long(ColumnMappingMode::Name, false, 16)] + #[case::randomize_short(ColumnMappingMode::None, true, 1)] + #[case::randomize_default(ColumnMappingMode::None, true, 2)] + #[case::randomize_long(ColumnMappingMode::None, true, 16)] + fn test_write_dir_random_prefix_length_property( + #[case] cm_mode: ColumnMappingMode, + #[case] randomize: bool, + #[case] prefix_len: usize, + ) { + let wc = make_write_context(cm_mode, vec![], HashMap::new(), randomize, prefix_len); + let path = wc.write_dir().path().to_string(); + let prefix = path + .strip_prefix("/table/") + .unwrap() + .strip_suffix('/') + .unwrap(); + assert_eq!( + prefix.len(), + prefix_len, + "expected {prefix_len}-char prefix, got: {prefix:?}" + ); + assert!( + prefix.chars().all(|c| c.is_ascii_alphanumeric()), + "prefix should be alphanumeric, got: {prefix}" + ); + } + + /// When CM is off and a partitioned table has `randomizeFilePrefixes=true`, the random + /// prefix replaces the Hive-style path on disk. Partition values are still recorded in + /// `add.partitionValues`; this test only checks the on-disk layout. + #[test] + fn test_write_dir_cm_off_randomize_suppresses_hive() { + let wc = make_write_context( + ColumnMappingMode::None, + vec!["year".into(), "month".into()], + HashMap::from([ + ("year".into(), Some("2024".into())), + ("month".into(), Some("03".into())), + ]), + true, + 2, + ); + let path = wc.write_dir().path().to_string(); + assert!( + !path.contains('='), + "randomize=true should suppress Hive dirs, got: {path}" + ); + let prefix = path + .strip_prefix("/table/") + .unwrap() + .strip_suffix('/') + .unwrap(); + assert_eq!(prefix.len(), 2); + } + + // === resolve_file_path tests === + + #[rstest] + #[case::relative_bare_file("s3://bucket/table/abc.parquet", Ok("abc.parquet"))] + #[case::relative_with_subdirectory( + "s3://bucket/table/year=2024/abc.parquet", + Ok("year=2024/abc.parquet") + )] + #[case::uri_encoded_partition( + "s3://bucket/table/p=a%253Ab/uuid.parquet", + Ok("p=a%253Ab/uuid.parquet") + )] + #[case::double_percent_partition( + "s3://bucket/table/p=100%252525/uuid.parquet", + Ok("p=100%252525/uuid.parquet") + )] + #[case::multi_partition_encoded( + "s3://bucket/table/year=2025/region=US%252FEast/uuid.parquet", + Ok("year=2025/region=US%252FEast/uuid.parquet") + )] + #[case::error_different_scheme("gs://other-bucket/table/abc.parquet", Err(()))] + #[case::error_different_host("s3://other-bucket/table/abc.parquet", Err(()))] + #[case::error_outside_table_root("s3://bucket/other/abc.parquet", Err(()))] + #[test] + fn test_resolve_file_path(#[case] file_url: &str, #[case] expected: Result<&str, ()>) { + let wc = make_write_context(ColumnMappingMode::None, vec![], HashMap::new(), false, 2); + let file = Url::parse(file_url).unwrap(); + match expected { + Ok(exp) => assert_eq!(wc.resolve_file_path(&file).unwrap(), exp), + Err(()) => assert!(wc.resolve_file_path(&file).is_err()), + } + } +} diff --git a/kernel/src/transaction/write_validation/addfile.rs b/kernel/src/transaction/write_validation/addfile.rs new file mode 100644 index 0000000000..e5961d9b26 --- /dev/null +++ b/kernel/src/transaction/write_validation/addfile.rs @@ -0,0 +1,303 @@ +//! Add-file validations. + +use std::collections::HashSet; +use std::sync::LazyLock; + +use super::utils::{validate_partition_keys, validate_required_field_exist}; +use super::{StagedDataValidator, Validation}; +use crate::engine_data::{GetData, TypedGetData as _}; +use crate::schema::ColumnNamesAndTypes; +use crate::transaction::mandatory_add_file_schema; +use crate::{DeltaResult, Error}; + +/// Column indices, matching the order in [`MANDATORY_ADD_FILE_COLUMNS`]. +const PATH: usize = 0; +const PARTITION_VALUES: usize = 1; +const SIZE: usize = 2; +const MODIFICATION_TIME: usize = 3; + +static MANDATORY_ADD_FILE_COLUMNS: LazyLock = + LazyLock::new(|| mandatory_add_file_schema().leaves(None)); + +impl StagedDataValidator { + /// Creates a validator that validates every staged add-file row. + pub(crate) fn staged_add_file( + physical_partition_columns: impl IntoIterator, + ) -> Self { + StagedDataValidator::new( + &MANDATORY_ADD_FILE_COLUMNS, + vec![Box::new(AddFileRequiredFields { + physical_partition_columns: physical_partition_columns.into_iter().collect(), + })], + ) + } +} + +/// Validates required-field existence and that each row's `partitionValues` keys match the table's +/// physical partition columns. +/// +/// Required fields: `path`, `partitionValues`, `size`, `modificationTime`, and `dataChange`. +/// Optional fields: `stats`, `tags`, `deletionVector`, `baseRowId`, +/// `defaultRowCommitVersion`, and `clusteringProvider`. +/// +/// NOTE: Currently, Kernel doesn't require connectors to set dataChange for staged addFile. +/// TODO(2869): Add intent-based validation for dataChange. +pub(crate) struct AddFileRequiredFields { + physical_partition_columns: HashSet, +} + +impl Validation for AddFileRequiredFields { + fn validate_row<'a>(&mut self, row: usize, getters: &[&'a dyn GetData<'a>]) -> DeltaResult<()> { + let path: &str = getters[PATH] + .get_opt(row, "path")? + .ok_or_else(|| Error::missing_data("AddFile is missing required field 'path'"))?; + if path.is_empty() { + return Err(Error::generic("AddFile path must not be empty")); + } + + let partition_values = validate_required_field_exist( + getters[PARTITION_VALUES].get_map(row, "partitionValues")?, + path, + "partitionValues", + )?; + validate_partition_keys(path, partition_values, &self.physical_partition_columns)?; + let size = validate_required_field_exist::( + getters[SIZE].get_opt(row, "size")?, + path, + "size", + )?; + if size < 0 { + return Err(Error::generic(format!( + "AddFile for '{path}' has negative size {size}; size must be non-negative" + ))); + } + validate_required_field_exist::( + getters[MODIFICATION_TIME].get_opt(row, "modificationTime")?, + path, + "modificationTime", + )?; + + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use rstest::rstest; + + use super::*; + use crate::arrow::array::{Int64Array, StringArray}; + use crate::arrow::record_batch::RecordBatch; + use crate::engine::arrow_data::ArrowEngineData; + use crate::expressions::column_name; + use crate::unit_test_utils::{ + add_files_with_partition_values, assert_result_error_with_message, nullable_add_file, + nullable_add_files, replace_column, set_field_as_null, + }; + use crate::EngineData; + + fn add_file_validator(physical_partition_columns: &[&str]) -> StagedDataValidator { + StagedDataValidator::staged_add_file( + physical_partition_columns.iter().map(|s| s.to_string()), + ) + } + + fn as_engine_data(batch: RecordBatch) -> [Box; 1] { + [Box::new(ArrowEngineData::new(batch)) as Box] + } + + /// Guards the invariant that the column-index consts match the leaf order of + /// `MANDATORY_ADD_FILE_COLUMNS`. + #[test] + fn column_indices_match_schema_order() { + let (names, _) = MANDATORY_ADD_FILE_COLUMNS.as_ref(); + assert_eq!(names[PATH], column_name!("path")); + assert_eq!(names[PARTITION_VALUES], column_name!("partitionValues")); + assert_eq!(names[SIZE], column_name!("size")); + assert_eq!(names[MODIFICATION_TIME], column_name!("modificationTime")); + assert_eq!( + names.len(), + [PATH, PARTITION_VALUES, SIZE, MODIFICATION_TIME].len() + ); + } + + #[rstest] + #[case::path("path")] + #[case::partition_values("partitionValues")] + #[case::size("size")] + #[case::modification_time("modificationTime")] + fn required_field_missing_at_any_batch_and_row_rejected( + #[case] field: &str, + #[values(0, 1, 2)] invalid_batch: usize, + #[values(0, 1, 2)] invalid_row: usize, + ) { + const BATCH_COUNT: usize = 3; + const ROW_COUNT: usize = 3; + + let adds: Vec> = (0..BATCH_COUNT) + .map(|batch_index| { + let batch = nullable_add_files(ROW_COUNT); + let batch = if batch_index == invalid_batch { + set_field_as_null(&batch, field, invalid_row) + } else { + batch + }; + Box::new(ArrowEngineData::new(batch)) as Box + }) + .collect(); + assert_result_error_with_message( + add_file_validator(&[] /* physical_partition_columns */).validate(&adds), + &format!("missing required field '{field}'"), + ); + } + + #[rstest] + #[case::valid( + "dummy" /* path */, + 1 /* size */, + 1 /* modification_time */, + None /* expected_error */, + )] + #[case::empty_path( + "" /* path */, + 1 /* size */, + 1 /* modification_time */, + Some("path must not be empty") /* expected_error */, + )] + #[case::negative_size( + "dummy" /* path */, + -1 /* size */, + 1 /* modification_time */, + Some("size must be non-negative") /* expected_error */, + )] + #[case::zero_size( + "dummy" /* path */, + 0 /* size */, + 1 /* modification_time */, + None /* expected_error */, + )] + #[case::negative_modification_time( + "dummy" /* path */, + 1 /* size */, + -1 /* modification_time */, + None /* expected_error */, + )] + fn add_file_values_accepted_or_rejected( + #[case] path: &str, + #[case] size: i64, + #[case] modification_time: i64, + #[case] expected_error: Option<&str>, + ) { + let batch = replace_column( + &nullable_add_file(), + "path", + Arc::new(StringArray::from(vec![path])), + ); + let batch = replace_column(&batch, "size", Arc::new(Int64Array::from(vec![size]))); + let batch = replace_column( + &batch, + "modificationTime", + Arc::new(Int64Array::from(vec![modification_time])), + ); + let adds = [Box::new(ArrowEngineData::new(batch)) as Box]; + let result = add_file_validator(&[] /* physical_partition_columns */).validate(&adds); + + if let Some(expected_error) = expected_error { + assert_result_error_with_message(result, expected_error); + } else { + result.unwrap(); + } + } + + #[test] + fn partition_values_exact_match_ok() { + let batch = add_files_with_partition_values(&[&[("p1", Some("a")), ("p2", Some("b"))]]); + add_file_validator(&["p1", "p2"] /* physical_partition_columns */) + .validate(&as_engine_data(batch)) + .unwrap(); + } + + #[test] + fn partition_value_null_still_counts_as_present() { + let batch = add_files_with_partition_values(&[&[("p1", Some("a")), ("p2", None)]]); + add_file_validator(&["p1", "p2"] /* physical_partition_columns */) + .validate(&as_engine_data(batch)) + .unwrap(); + } + + #[rstest] + #[case::missing_second_partition_column( + &[("p1", Some("a")), ("p2", Some("b")), ("p3", Some("c"))], + &[("p1", Some("a")), ("p3", Some("c"))], + &["p1", "p2", "p3"], + "partitionValues keys" + )] + #[case::missing_third_partition_column( + &[("p1", Some("a")), ("p2", Some("b")), ("p3", Some("c"))], + &[("p1", Some("a")), ("p2", Some("b"))], + &["p1", "p2", "p3"], + "partitionValues keys" + )] + #[case::extra_partition_columns( + &[("p1", Some("a")), ("p2", Some("b"))], + &[("p1", Some("a")), ("p2", Some("b")), ("p3", Some("c"))], + &["p1", "p2"], + "partitionValues keys" + )] + #[case::wrong_partition_column_name( + &[("p1", Some("a")), ("p2", Some("b"))], + &[("p1", Some("a")), ("wrong", Some("b"))], + &["p1", "p2"], + "partitionValues keys" + )] + #[case::duplicate_partition_column( + &[("p1", Some("a")), ("p2", Some("b"))], + &[("p1", Some("a")), ("p1", Some("b")), ("p2", Some("c"))], + &["p1", "p2"], + "duplicate partition column names" + )] + fn partition_column_mismatch_rejected( + #[case] valid_partition_values: &[(&str, Option<&str>)], + #[case] invalid_partition_values: &[(&str, Option<&str>)], + #[case] physical_partition_columns: &[&str], + #[case] expected_error: &str, + #[values(0, 1, 2)] invalid_batch: usize, + #[values(0, 1, 2)] invalid_row: usize, + ) { + const BATCH_COUNT: usize = 3; + const ROW_COUNT: usize = 3; + + let adds: Vec> = (0..BATCH_COUNT) + .map(|batch_index| { + let mut partition_values = [valid_partition_values; ROW_COUNT]; + if batch_index == invalid_batch { + partition_values[invalid_row] = invalid_partition_values; + } + let batch = add_files_with_partition_values(&partition_values); + Box::new(ArrowEngineData::new(batch)) as Box + }) + .collect(); + let error = add_file_validator(physical_partition_columns) + .validate(&adds) + .expect_err("invalid partition values should be rejected"); + let Error::InvalidPartitionValues(message) = error else { + panic!("expected InvalidPartitionValues, got {error:?}"); + }; + assert!( + message.contains(expected_error), + "expected error message to contain {expected_error:?}, got {message:?}" + ); + } + + #[test] + fn unpartitioned_table_rejects_partition_values() { + let batch = add_files_with_partition_values(&[&[("stray", Some("x"))]]); + assert_result_error_with_message( + add_file_validator(&[] /* physical_partition_columns */) + .validate(&as_engine_data(batch)), + "partitionValues keys", + ); + } +} diff --git a/kernel/src/transaction/write_validation/dv.rs b/kernel/src/transaction/write_validation/dv.rs new file mode 100644 index 0000000000..3f15768511 --- /dev/null +++ b/kernel/src/transaction/write_validation/dv.rs @@ -0,0 +1,260 @@ +//! Deletion-vector update validations. + +use std::collections::HashSet; +use std::sync::LazyLock; + +use super::utils::{validate_partition_keys, validate_required_field_exist}; +use super::{StagedDataValidator, Validation}; +use crate::engine_data::{GetData, TypedGetData as _}; +use crate::expressions::column_name; +use crate::scan::log_replay::{ + FILE_CONSTANT_VALUES_NAME, PARTITION_VALUES_NAME, PATH_NAME, SIZE_NAME, +}; +use crate::scan::scan_row_schema; +use crate::schema::ColumnNamesAndTypes; +use crate::utils::require; +use crate::{DeltaResult, Error}; + +const PATH: usize = 0; +const SIZE: usize = 1; +const MODIFICATION_TIME: usize = 2; +const PARTITION_VALUES: usize = 3; +const MODIFICATION_TIME_NAME: &str = "modificationTime"; + +static DV_MATCHED_FILE_COLUMNS: LazyLock> = LazyLock::new(|| { + let names = vec![ + column_name!(PATH_NAME), + column_name!(SIZE_NAME), + column_name!(MODIFICATION_TIME_NAME), + column_name!(FILE_CONSTANT_VALUES_NAME, PARTITION_VALUES_NAME), + ]; + // Derive types from the canonical scan schema so this projection stays compatible with scan + // metadata if those field definitions change. + let types = names + .iter() + .map(|name| { + scan_row_schema() + .field_at(name) + .map(|field| field.data_type().clone()) + }) + .collect::>>()?; + Ok((names, types).into()) +}); + +struct DvMatchedFileRequiredFields { + physical_partition_columns: HashSet, +} + +impl Validation for DvMatchedFileRequiredFields { + fn validate_row<'a>(&mut self, row: usize, getters: &[&'a dyn GetData<'a>]) -> DeltaResult<()> { + let path: &str = getters[PATH] + .get_opt(row, PATH_NAME)? + .ok_or_else(|| Error::missing_data("AddFile is missing required field 'path'"))?; + require!( + !path.is_empty(), + Error::generic("AddFile path must not be empty") + ); + + let partition_values = validate_required_field_exist( + getters[PARTITION_VALUES].get_map(row, PARTITION_VALUES_NAME)?, + path, + PARTITION_VALUES_NAME, + )?; + validate_partition_keys(path, partition_values, &self.physical_partition_columns)?; + + let size = validate_required_field_exist::( + getters[SIZE].get_opt(row, SIZE_NAME)?, + path, + SIZE_NAME, + )?; + require!( + size >= 0, + Error::generic(format!( + "AddFile for '{path}' has negative size {size}; size must be non-negative" + )) + ); + validate_required_field_exist::( + getters[MODIFICATION_TIME].get_opt(row, MODIFICATION_TIME_NAME)?, + path, + MODIFICATION_TIME_NAME, + )?; + Ok(()) + } +} + +impl StagedDataValidator { + /// Creates a validator for selected rows staged for deletion-vector updates. + /// + /// Errors if the required columns are absent from the scan-row schema. + pub(crate) fn staged_dv_matched_file( + physical_partition_columns: impl IntoIterator, + ) -> DeltaResult { + let columns = DV_MATCHED_FILE_COLUMNS.as_ref().map_err(|error| { + Error::internal_error(format!( + "DV validation columns must exist in the scan-row schema: {error}" + )) + })?; + Ok(StagedDataValidator::new( + columns, + vec![Box::new(DvMatchedFileRequiredFields { + physical_partition_columns: physical_partition_columns.into_iter().collect(), + })], + )) + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use rstest::rstest; + + use super::*; + use crate::arrow::array::{new_null_array, ArrayRef, Int64Array, StructArray}; + use crate::arrow::datatypes::{DataType as ArrowDataType, Schema as ArrowSchema}; + use crate::arrow::record_batch::RecordBatch; + use crate::engine::arrow_conversion::TryIntoArrow; + use crate::engine::arrow_data::ArrowEngineData; + use crate::engine_data::FilteredEngineData; + use crate::expressions::column_name; + use crate::unit_test_utils::{ + add_files_with_partition_values, assert_result_error_with_message, nullable_add_files, + replace_column, set_field_as_null, + }; + + fn make_staged_dv_from_addfile( + batch: RecordBatch, + selection_vector: Vec, + ) -> FilteredEngineData { + let column = |name| { + batch + .column( + batch + .schema() + .index_of(name) + .expect("field in add-file schema"), + ) + .clone() + }; + let schema: ArrowSchema = scan_row_schema() + .as_ref() + .try_into_arrow() + .expect("scan-row schema should convert to Arrow"); + let columns = schema + .fields() + .iter() + .map(|field| match field.name().as_str() { + "path" | "size" | "modificationTime" => column(field.name()), + "fileConstantValues" => { + let ArrowDataType::Struct(fields) = field.data_type() else { + panic!("fileConstantValues should be a struct"); + }; + let values = fields + .iter() + .map(|field| match field.name().as_str() { + "partitionValues" => column(field.name()), + _ => new_null_array(field.data_type(), batch.num_rows()), + }) + .collect(); + Arc::new(StructArray::new(fields.clone(), values, None)) as ArrayRef + } + _ => new_null_array(field.data_type(), batch.num_rows()), + }) + .collect(); + let batch = RecordBatch::try_new(Arc::new(schema), columns) + .expect("staged DV schema and columns should form a valid batch"); + FilteredEngineData::try_new(Box::new(ArrowEngineData::new(batch)), selection_vector) + .expect("selection vector length should match staged DV row count") + } + + #[test] + fn column_indices_match_schema_order() { + let columns = DV_MATCHED_FILE_COLUMNS + .as_ref() + .expect("DV validation columns should exist in the scan-row schema"); + let (names, _) = columns.as_ref(); + assert_eq!(names[PATH], column_name!(PATH_NAME)); + assert_eq!(names[SIZE], column_name!(SIZE_NAME)); + assert_eq!( + names[MODIFICATION_TIME], + column_name!(MODIFICATION_TIME_NAME) + ); + assert_eq!( + names[PARTITION_VALUES], + column_name!(FILE_CONSTANT_VALUES_NAME, PARTITION_VALUES_NAME) + ); + } + + #[rstest] + #[case::zero_size("size", 0)] + #[case::negative_modification_time("modificationTime", -1)] + fn valid_boundary_value_is_accepted(#[case] field: &str, #[case] value: i64) { + let batch = replace_column( + &nullable_add_files(1 /* row_count */), + field, + Arc::new(Int64Array::from(vec![value])), + ); + let batches = [make_staged_dv_from_addfile(batch, vec![true])]; + StagedDataValidator::staged_dv_matched_file(std::iter::empty()) + .expect("DV validator should use the scan-row schema") + .validate_filtered(&batches) + .expect("protocol-valid boundary value should be accepted"); + } + + #[rstest] + #[case::path("path")] + #[case::partition_values("partitionValues")] + #[case::size("size")] + #[case::modification_time("modificationTime")] + fn missing_required_field_rejected( + #[case] field: &str, + #[values(0, 1, 2)] invalid_batch: usize, + ) { + const BATCH_COUNT: usize = 3; + + let batches: Vec<_> = (0..BATCH_COUNT) + .map(|batch_index| { + let batch = nullable_add_files(2 /* row_count */); + let batch = if batch_index == invalid_batch { + set_field_as_null(&batch, field, 1 /* row */) + } else { + batch + }; + make_staged_dv_from_addfile(batch, vec![true, true]) + }) + .collect(); + assert_result_error_with_message( + StagedDataValidator::staged_dv_matched_file(std::iter::empty()) + .expect("DV validator should use the scan-row schema") + .validate_filtered(&batches), + field, + ); + } + + #[rstest] + #[case::selected(&[true, true], Some("partitionValues keys"))] + #[case::implicitly_selected(&[false], Some("partitionValues keys"))] + #[case::unselected(&[true, false], None)] + fn partition_column_mismatch_validates_selected_rows( + #[case] selection_vector: &[bool], + #[case] expected_error: Option<&str>, + ) { + let batch = add_files_with_partition_values(&[ + &[("p1", Some("a")), ("p2", Some("b"))], + &[("p1", Some("a"))], + ]); + let batches = [make_staged_dv_from_addfile( + batch, + selection_vector.to_vec(), + )]; + let result = + StagedDataValidator::staged_dv_matched_file(["p1".to_string(), "p2".to_string()]) + .expect("DV validator should use the scan-row schema") + .validate_filtered(&batches); + if let Some(expected_error) = expected_error { + assert_result_error_with_message(result, expected_error); + } else { + result.expect("unselected invalid row should be ignored"); + } + } +} diff --git a/kernel/src/transaction/write_validation/mod.rs b/kernel/src/transaction/write_validation/mod.rs new file mode 100644 index 0000000000..21e577b615 --- /dev/null +++ b/kernel/src/transaction/write_validation/mod.rs @@ -0,0 +1,98 @@ +//! Pre-commit validation of data staged on a [`Transaction`]. +//! +//! [`Transaction`]: super::Transaction + +// TODO(#2869): Add the remaining write-side validations: +// - Required fields for `txn.remove_files_metadata` +// - No duplicate (path, DvId) in `txn.add_files_metadata`, `txn.remove_files_metadata`, +// `txn.dv_matched_files` + +mod addfile; +mod dv; +mod utils; + +use crate::engine_data::{ + FilteredEngineData, FilteredRowVisitor, GetData, RowIndexIterator, RowVisitor, +}; +use crate::expressions::ColumnName; +use crate::schema::{ColumnNamesAndTypes, DataType}; +use crate::{DeltaResult, EngineData}; + +/// A single row-level validation. +pub(crate) trait Validation { + fn validate_row<'a>(&mut self, row: usize, getters: &[&'a dyn GetData<'a>]) -> DeltaResult<()>; +} + +/// Runs validations over batches that share one staged-data schema. +/// +/// Each instance uses one column projection and applies its configured validations to every staged +/// row. Every [`Validation`] sees the full getter list and reads the columns it needs. +pub(crate) struct StagedDataValidator { + columns_and_types: &'static ColumnNamesAndTypes, + validations: Vec>, +} + +impl StagedDataValidator { + pub(crate) fn new( + columns_and_types: &'static ColumnNamesAndTypes, + validations: Vec>, + ) -> Self { + Self { + columns_and_types, + validations, + } + } + + /// Run every validation against each batch. Returns the first validation error encountered. + pub(crate) fn validate(mut self, batches: &[Box]) -> DeltaResult<()> { + for batch in batches { + RowVisitor::visit_rows_of(&mut self, batch.as_ref())?; + } + Ok(()) + } + + /// Runs every validation against each selected staged-data row. + pub(crate) fn validate_filtered(mut self, batches: &[FilteredEngineData]) -> DeltaResult<()> { + for batch in batches { + FilteredRowVisitor::visit_rows_of(&mut self, batch)?; + } + Ok(()) + } + + fn validate_rows<'a>( + &mut self, + rows: impl Iterator, + getters: &[&'a dyn GetData<'a>], + ) -> DeltaResult<()> { + for row in rows { + for validation in &mut self.validations { + validation.validate_row(row, getters)?; + } + } + Ok(()) + } +} + +impl RowVisitor for StagedDataValidator { + fn selected_column_names_and_types(&self) -> (&'static [ColumnName], &'static [DataType]) { + self.columns_and_types.as_ref() + } + + fn visit<'a>(&mut self, row_count: usize, getters: &[&'a dyn GetData<'a>]) -> DeltaResult<()> { + self.validate_rows(0..row_count, getters) + } +} + +impl FilteredRowVisitor for StagedDataValidator { + fn selected_column_names_and_types(&self) -> (&'static [ColumnName], &'static [DataType]) { + self.columns_and_types.as_ref() + } + + fn visit_filtered<'a>( + &mut self, + getters: &[&'a dyn GetData<'a>], + rows: RowIndexIterator<'_>, + ) -> DeltaResult<()> { + self.validate_rows(rows, getters) + } +} diff --git a/kernel/src/transaction/write_validation/utils.rs b/kernel/src/transaction/write_validation/utils.rs new file mode 100644 index 0000000000..343b272cd3 --- /dev/null +++ b/kernel/src/transaction/write_validation/utils.rs @@ -0,0 +1,46 @@ +use std::collections::HashSet; + +use crate::engine_data::MapItem; +use crate::utils::require; +use crate::{DeltaResult, Error}; + +pub(super) fn validate_required_field_exist( + value: Option, + path: &str, + field: &str, +) -> DeltaResult { + value.ok_or_else(|| { + Error::missing_data(format!( + "AddFile for '{path}' is missing required field '{field}'" + )) + }) +} + +pub(super) fn validate_partition_keys( + path: &str, + actual_partition_values: MapItem<'_>, + expected_physical_partition_columns: &HashSet, +) -> DeltaResult<()> { + let actual_keys_vec: Vec<&str> = actual_partition_values.keys().collect(); + let actual_keys_set: HashSet<&str> = actual_keys_vec.iter().copied().collect(); + let keys_match = actual_keys_set.len() == expected_physical_partition_columns.len() + && actual_keys_set + .iter() + .all(|key| expected_physical_partition_columns.contains(*key)); + + require!( + actual_keys_vec.len() == actual_keys_set.len(), + Error::invalid_partition_values(format!( + "AddFile for '{path}' has duplicate partition column names in partitionValues: \ + {actual_keys_vec:?}" + )) + ); + require!( + keys_match, + Error::invalid_partition_values(format!( + "AddFile for '{path}' has partitionValues keys {actual_keys_vec:?}, but the table's \ + physical partition columns are {expected_physical_partition_columns:?}" + )) + ); + Ok(()) +} diff --git a/kernel/src/transforms/carrier.rs b/kernel/src/transforms/carrier.rs new file mode 100644 index 0000000000..add1f3822d --- /dev/null +++ b/kernel/src/transforms/carrier.rs @@ -0,0 +1,193 @@ +use std::borrow::{Cow, ToOwned}; +use std::convert::Infallible; +use std::fmt::Debug; + +/// Carrier abstraction for transform outputs. +/// +/// A carrier can represent one of two outcomes for a transformed node: +/// - present value (`Cow<'a, T>`) +/// - residual outcome (`Self::Residual`) +/// +/// Depending on carrier choice, a residual encodes filtered-out and/or error cases. A non-filtering +/// infallible carrier has `Residual = Infallible`. +/// +/// Different carriers encode those states differently: +/// - `()` and `Result<(), E>` are carriers for read-only visitor transforms that return no value +/// - `Cow<'a, T>` and `Result, E>` are non-filtering carriers +/// - `Option>` and `Result>, E>` are filtering carriers +pub trait Carrier<'a, T: ToOwned + ?Sized>: Sized { + type Residual; + + /// For filtering carriers, Some carrier containing a residual value equivalent to + /// `Option::None`. None for non-filtering carriers. + const NONE: Option; + + /// Wrap a present transformed node. + fn from_inner(inner: Cow<'a, T>) -> Self; + /// Wrap a residual outcome for this carrier (e.g. None or Err). + fn from_residual(residual: Self::Residual) -> Self; + /// Extracts a present node or returns a residual. + /// + /// Filtering and/or fallible carriers encode "filtered out" and "error" in the residual side. + fn into_inner(self) -> Result, Self::Residual>; + /// Extracts an optional node or returns a residual. + /// + /// This bridges filtering and non-filtering carriers into a common shape: + /// - filtering carriers may return `Ok(None)` to indicate "filtered out" + /// - non-filtering carriers always return `Ok(Some(_))` + fn into_inner_opt(self) -> Result>, Self::Residual>; +} + +/// Emulates `?` for the result of calling [`Carrier::into_inner_opt`]: If the result is Err, +/// immediately return the residual. Otherwise, unpack the inner value. +macro_rules! carrier_into_inner_opt { + ($expr:expr) => { + match $expr.into_inner_opt() { + Ok(inner) => inner, + Err(residual) => return Carrier::from_residual(residual), + } + }; +} +pub(crate) use carrier_into_inner_opt; + +/// Attempts to early-return, as if by `None?`, by calling [`Carrier::try_none`]: If the result is +/// Ok(none), immediately return it. Otherwise, do nothing. +macro_rules! carrier_try_none { + () => { + if let Some(none) = Carrier::NONE { + return none; + } + }; +} +pub(crate) use carrier_try_none; + +/// Carrier for infallible non-filtering transforms that can replace but not drop nodes. +impl<'a, T: ToOwned + ?Sized> Carrier<'a, T> for Cow<'a, T> { + type Residual = Infallible; + + const NONE: Option = None; + + fn from_inner(inner: Cow<'a, T>) -> Self { + inner + } + fn from_residual(residual: Infallible) -> Self { + match residual {} + } + fn into_inner(self) -> Result, Infallible> { + Ok(self) + } + fn into_inner_opt(self) -> Result>, Infallible> { + Ok(Some(self)) + } +} + +/// Carrier for infallible filtering transforms that can drop nodes by returning None. +impl<'a, T: ToOwned + ?Sized> Carrier<'a, T> for Option> { + /// The only valid residual value is `()`, which indicates a filtered-out node. + type Residual = (); + + const NONE: Option = Some(None); + + fn from_inner(inner: Cow<'a, T>) -> Self { + Some(inner) + } + fn from_residual(_residual: ()) -> Self { + None + } + fn into_inner(self) -> Result, ()> { + self.ok_or(()) + } + fn into_inner_opt(self) -> Result>, ()> { + Ok(self) + } +} + +/// Carrier for infallible visitors, which are modeled as always-filter transforms. It silently +/// discards any inner value assigned to it, and is hard-wired to always produce None (drop node). +impl<'a, T: ToOwned + ?Sized> Carrier<'a, T> for () { + /// The only valid residual value is `()`, which indicates a filtered-out node. + type Residual = (); + + const NONE: Option = Some(()); + + fn from_inner(_inner: Cow<'a, T>) -> Self {} + fn from_residual(_residual: ()) -> Self {} + fn into_inner(self) -> Result, ()> { + Err(()) + } + fn into_inner_opt(self) -> Result>, ()> { + Ok(None) + } +} + +/// Carrier for fallible non-filtering transforms, which immediately abort on Err. +impl<'a, T: ToOwned + ?Sized, E: Debug> Carrier<'a, T> for Result, E> { + type Residual = E; + + const NONE: Option = None; + + fn from_inner(inner: Cow<'a, T>) -> Self { + Ok(inner) + } + fn from_residual(residual: E) -> Self { + Err(residual) + } + fn into_inner(self) -> Result, E> { + self + } + fn into_inner_opt(self) -> Result>, E> { + self.map(Some) + } +} + +/// Carrier for fallible filtering transforms, which immediately abort on Err and can also drop +/// nodes by returning None. +impl<'a, T: ToOwned + ?Sized, E: Debug> Carrier<'a, T> for Result>, E> { + /// Residual None means drop, Some error means abort + type Residual = Option; + + const NONE: Option = Some(Ok(None)); + + fn from_inner(inner: Cow<'a, T>) -> Self { + Ok(Some(inner)) + } + fn from_residual(residual: Option) -> Self { + residual.map_or(Ok(None), Err) + } + fn into_inner(self) -> Result, Option> { + match self { + Ok(Some(inner)) => Ok(inner), + Ok(None) => Err(None), + Err(err) => Err(Some(err)), + } + } + fn into_inner_opt(self) -> Result>, Option> { + self.map_err(Some) + } +} + +/// Carrier for fallible visitors, which are modeled as fallible always-filter transforms. It +/// silently discards any inner value assigned to it, and is hard-wired to always produce None +/// unless an error aborts it. +impl<'a, T: ToOwned + ?Sized, E: Debug> Carrier<'a, T> for Result<(), E> { + /// Residual None means drop, Some error means abort + type Residual = Option; + + const NONE: Option = Some(Ok(())); + + fn from_inner(_inner: Cow<'a, T>) -> Self { + Ok(()) + } + fn from_residual(residual: Option) -> Self { + residual.map_or(Ok(()), Err) + } + fn into_inner(self) -> Result, Option> { + Err(self.err()) + } + fn into_inner_opt(self) -> Result>, Option> { + match self { + Ok(()) => Ok(None), + Err(err) => Err(Some(err)), + } + } +} diff --git a/kernel/src/transforms/expression.rs b/kernel/src/transforms/expression.rs index e87579d646..13473cb157 100644 --- a/kernel/src/transforms/expression.rs +++ b/kernel/src/transforms/expression.rs @@ -2,393 +2,461 @@ use std::borrow::{Cow, ToOwned}; use std::sync::Arc; use crate::expressions::{ - BinaryExpression, BinaryPredicate, ColumnName, Expression, ExpressionRef, JunctionPredicate, - MapToStructExpression, OpaqueExpression, OpaquePredicate, ParseJsonExpression, Predicate, - Scalar, Transform, UnaryExpression, UnaryPredicate, VariadicExpression, + BinaryExpression, BinaryPredicate, CastExpression, ColumnName, ElementAtExpression, Expression, + ExpressionRef, ExpressionStructPatch, JunctionPredicate, MapToStructExpression, + OpaqueExpression, OpaquePredicate, ParseJsonExpression, Predicate, Scalar, UnaryExpression, + UnaryPredicate, VariadicExpression, }; -use crate::transforms::{map_owned_children_or_else, CowExt as _}; +use crate::transforms::{ + map_owned_children_or_else, map_owned_or_else, map_owned_pair_or_else, transform_output_type, + Carrier, +}; +use crate::{DeltaResult, Error}; -/// Generic framework for recursive bottom-up transforms of expressions and -/// predicates. Transformations return `Option` with the following semantics: +/// Generic framework for recursive bottom-up transforms of expressions and predicates. /// -/// * `Some(Cow::Owned)` -- The input was transformed and the parent should be updated with it. -/// * `Some(Cow::Borrowed)` -- The input was not transformed. -/// * `None` -- The input was filtered out and the parent should be updated to not reference it. +/// The transform entry point is generally [`Self::transform_expr`] or [`Self::transform_pred`] (for +/// expressions or predicates, respectively), but callers can also directly invoke the transform +/// for a specific expression/predicate variant (e.g. [`Self::transform_expr_column`] for +/// [`ColumnName`] or [`Self::transform_pred_unary`] for [`UnaryPredicate`]). /// -/// The transform can start from the generic [`Self::transform_expr`] or [`Self::transform_pred`], -/// or directly from a specific expression/predicate variant (e.g. [`Self::transform_expr_column`] -/// for [`ColumnName`], [`Self::transform_pred_unary`] for [`UnaryPredicate`]). +/// The provided `transform_xxx` methods all default to no-op (usually by invoking the corresponding +/// recursive helper method), and implementations should selectively override specific +/// `transform_xxx` methods as needed for the task at hand. /// -/// The provided `transform_xxx` methods all default to no-op (returning their input as -/// `Some(Cow::Borrowed)`), and implementations should selectively override specific `transform_xxx` -/// methods as needed for the task at hand. +/// # Recursive helper methods /// /// The provided `recurse_into_xxx` methods encapsulate the boilerplate work of recursing into the -/// children of each expression or predicate variant. Implementations can call these as needed but -/// will generally not need to override them. +/// child expression of each expression type. Except as specifically noted otherwise, these +/// recursive helpers all behave uniformly, based on the number of children the parent has: +/// +/// * Leaf (no children) - Leaf `transform_xxx` methods simply return their argument unchanged, and +/// no corresponding `recurse_into_xxx` method is provided. +/// +/// * Unary (single child) - If the child was filtered out, filter out the parent. If the child +/// changed, build a new parent around it. Otherwise, return the parent unchanged. +/// +/// * Binary (two children) - If either child was filtered out, filter out the parent. If at least +/// one child changed, build a new parent around them. Otherwise, return the parent unchanged. +/// +/// * Variadic (0+ children) - If no children remain (all filtered out), filter out the parent. +/// Otherwise, if at least one child changed or was filtered out, build a new parent around the +/// children. Otherwise, return the parent unchanged. +/// +/// Implementations can call these as needed but will generally not need to override them. +/// +/// # Transform carrier selection +/// +/// Implementations choose an output [`Carrier`] instance based on the operation to be +/// performed. That carrier determines the return type of each transform method. +/// +/// For example, a simple read-only visitor would use `()` as a carrier, while a validity checker +/// could use `DeltaResult<()>` instead. A mutating transform uses `Cow<_>`, returning `Cow::Owned` +/// for changed/replaced nodes, and a filtering transform uses `Option>`, where `None` +/// indicates the node should be dropped rather than replaced. `DeltaResult>` and +/// `Result>, E>` round out the set as fallible mutating and fitering transforms that +/// short circuit immediately upon `Err`. pub trait ExpressionTransform<'a> { - /// Called for each literal encountered during the expression traversal. - fn transform_expr_literal(&mut self, value: &'a Scalar) -> Option> { - Some(Cow::Borrowed(value)) - } - - /// Called for each column reference encountered during the expression traversal. - fn transform_expr_column(&mut self, name: &'a ColumnName) -> Option> { - Some(Cow::Borrowed(name)) - } - - /// Called for the expression list of each [`Expression::Struct`] encountered during the - /// traversal. Implementations can call [`Self::recurse_into_expr_struct`] if they wish to - /// recursively transform the child expressions. + /// [`Carrier`] output type for transformed nodes. + /// + /// Implementations can use [`crate::transforms::transform_output_type`] to define `Output` + /// and `Residual` together. + type Output: Carrier<'a, T, Residual = Self::Residual>; + /// Residual type propagated by this transform's output [`Carrier`]. + /// + /// Implementations can use [`crate::transforms::transform_output_type`] to define `Output` + /// and `Residual` together. Or, define it manually like this: + /// ```rust,no_run + /// # use std::borrow::Cow; + /// # use delta_kernel::transforms::{Carrier, ExpressionTransform}; + /// # struct X; + /// # impl<'a> ExpressionTransform<'a> for X { + /// # type Output = Cow<'a, T>; + /// type Residual = as Carrier<'a, ()>>::Residual; + /// # } + /// ``` + /// (required because associated type defaults are not stable rust yet) + type Residual; + + /// Called for each literal encountered during the traversal (leaf). + fn transform_expr_literal(&mut self, value: &'a Scalar) -> Self::Output { + Carrier::from_inner(Cow::Borrowed(value)) + } + + /// Called for each column reference encountered during the traversal (leaf). + fn transform_expr_column(&mut self, name: &'a ColumnName) -> Self::Output { + Carrier::from_inner(Cow::Borrowed(name)) + } + + /// Called for the expression list of each struct expression encountered during the + /// traversal. The provided implementation just forwards to [`Self::recurse_into_expr_struct`]. fn transform_expr_struct( &mut self, fields: &'a [ExpressionRef], - ) -> Option> { + ) -> Self::Output<[ExpressionRef]> { self.recurse_into_expr_struct(fields) } - /// Called for each [`OpaqueExpression`] encountered during the traversal. Implementations can - /// call [`Self::recurse_into_expr_opaque`] if they wish to recursively transform the children. + /// Called for each opaque expression encountered during the traversal. The provided + /// implementation just forwards to [`Self::recurse_into_expr_opaque`]. fn transform_expr_opaque( &mut self, expr: &'a OpaqueExpression, - ) -> Option> { + ) -> Self::Output { self.recurse_into_expr_opaque(expr) } - /// Called for each [`Expression::Unknown`] encountered during the traversal. - fn transform_expr_unknown(&mut self, name: &'a String) -> Option> { - Some(Cow::Borrowed(name)) + /// Called for each unknown expression encountered during the traversal (leaf). + fn transform_expr_unknown(&mut self, name: &'a String) -> Self::Output { + Carrier::from_inner(Cow::Borrowed(name)) } - /// Called for each [`Transform`] encountered during the traversal. By default, it is a no-op - /// that simply returns its argument and does _NOT_ recurse into its children. - fn transform_expr_transform(&mut self, transform: &'a Transform) -> Option> { - Some(Cow::Borrowed(transform)) + /// Called for each struct patch expression encountered during the traversal (leaf). + /// + /// The provided implementation does _NOT_ recurse into its children. + fn transform_expr_struct_patch( + &mut self, + patch: &'a ExpressionStructPatch, + ) -> Self::Output { + Carrier::from_inner(Cow::Borrowed(patch)) } - /// Called for each [`ParseJsonExpression`] encountered during the traversal. Implementations - /// can call [`Self::recurse_into_expr_parse_json`] if they wish to recursively transform the - /// child expression. + /// Called for each parse-json expression encountered during the traversal. The provided + /// implementation just forwards to [`Self::recurse_into_expr_parse_json`]. fn transform_expr_parse_json( &mut self, expr: &'a ParseJsonExpression, - ) -> Option> { + ) -> Self::Output { self.recurse_into_expr_parse_json(expr) } - /// Called for each [`MapToStructExpression`] encountered during the traversal. Implementations - /// can call [`Self::recurse_into_expr_map_to_struct`] if they wish to recursively transform - /// the child expression. + /// Called for each map-to-struct expression encountered during the traversal. The provided + /// implementation just forwards to [`Self::recurse_into_expr_map_to_struct`]. fn transform_expr_map_to_struct( &mut self, expr: &'a MapToStructExpression, - ) -> Option> { + ) -> Self::Output { self.recurse_into_expr_map_to_struct(expr) } - /// Called for the child predicate of each [`Expression::Predicate`] encountered during the - /// traversal. Implementations can call [`Self::recurse_into_expr_pred`] if they wish to - /// recursively transform the child predicate. - fn transform_expr_pred(&mut self, pred: &'a Predicate) -> Option> { + /// Called for each element-at expression encountered during the traversal. + fn transform_expr_element_at( + &mut self, + expr: &'a ElementAtExpression, + ) -> Self::Output { + self.recurse_into_expr_element_at(expr) + } + + /// Called for each cast expression encountered during the traversal. The provided + /// implementation just forwards to [`Self::recurse_into_expr_cast`]. + fn transform_expr_cast(&mut self, expr: &'a CastExpression) -> Self::Output { + self.recurse_into_expr_cast(expr) + } + + /// Called for the child of each predicate expression encountered during the + /// traversal. The provided implementation just forwards to [`Self::recurse_into_expr_pred`]. + fn transform_expr_pred(&mut self, pred: &'a Predicate) -> Self::Output { self.recurse_into_expr_pred(pred) } - /// Called for the child predicate of each [`Predicate::Not`] encountered during the - /// traversal. Implementations can call [`Self::recurse_into_pred_not`] if they wish to - /// recursively transform the child expression. - fn transform_pred_not(&mut self, pred: &'a Predicate) -> Option> { + /// Called for the child of each NOT predicate encountered during the + /// traversal. The provided implementation just forwards to [`Self::recurse_into_pred_not`]. + fn transform_pred_not(&mut self, pred: &'a Predicate) -> Self::Output { self.recurse_into_pred_not(pred) } - /// Called for each [`UnaryExpression`] encountered during the traversal. Implementations can - /// call [`Self::recurse_into_expr_unary`] if they wish to recursively transform the child. - fn transform_expr_unary( - &mut self, - expr: &'a UnaryExpression, - ) -> Option> { + /// Called for each unary expression encountered during the traversal. The provided + /// implementation just forwards to [`Self::recurse_into_expr_unary`]. + fn transform_expr_unary(&mut self, expr: &'a UnaryExpression) -> Self::Output { self.recurse_into_expr_unary(expr) } - /// Called for each [`UnaryPredicate`] encountered during the traversal. Implementations can - /// call [`Self::recurse_into_pred_unary`] if they wish to recursively transform the child. - fn transform_pred_unary( - &mut self, - pred: &'a UnaryPredicate, - ) -> Option> { + /// Called for each unary predicate encountered during the traversal. The provided + /// implementation just forwards to [`Self::recurse_into_pred_unary`]. + fn transform_pred_unary(&mut self, pred: &'a UnaryPredicate) -> Self::Output { self.recurse_into_pred_unary(pred) } - /// Called for each [`BinaryExpression`] encountered during the traversal. Implementations can - /// call [`Self::recurse_into_expr_binary`] if they wish to recursively transform the children. + /// Called for each binary expression encountered during the traversal. The provided + /// implementation just forwards to [`Self::recurse_into_expr_binary`]. fn transform_expr_binary( &mut self, expr: &'a BinaryExpression, - ) -> Option> { + ) -> Self::Output { self.recurse_into_expr_binary(expr) } - /// Called for each [`BinaryPredicate`] encountered during the traversal. Implementations can - /// call [`Self::recurse_into_pred_binary`] if they wish to recursively transform the children. + /// Called for each binary predicate encountered during the traversal. The provided + /// implementation just forwards to [`Self::recurse_into_pred_binary`]. fn transform_pred_binary( &mut self, pred: &'a BinaryPredicate, - ) -> Option> { + ) -> Self::Output { self.recurse_into_pred_binary(pred) } - /// Called for each [`VariadicExpression`] encountered during the traversal. Implementations can - /// call [`Self::recurse_into_expr_variadic`] if they wish to recursively transform the children. + /// Called for each variadic expression encountered during the traversal. The provided + /// implementation just forwards to [`Self::recurse_into_expr_variadic`]. fn transform_expr_variadic( &mut self, expr: &'a VariadicExpression, - ) -> Option> { + ) -> Self::Output { self.recurse_into_expr_variadic(expr) } - /// Called for each [`JunctionPredicate`] encountered during the traversal. Implementations can - /// call [`Self::recurse_into_pred_junction`] if they wish to recursively transform the children. + /// Called for each junction predicate encountered during the traversal. The provided + /// implementation just forwards to [`Self::recurse_into_pred_junction`]. fn transform_pred_junction( &mut self, pred: &'a JunctionPredicate, - ) -> Option> { + ) -> Self::Output { self.recurse_into_pred_junction(pred) } - /// Called for each [`OpaquePredicate`] encountered during the traversal. Implementations can - /// call [`Self::recurse_into_pred_opaque`] if they wish to recursively transform the children. + /// Called for each opaque predicate encountered during the traversal. The provided + /// implementation just forwards to [`Self::recurse_into_pred_opaque`]. fn transform_pred_opaque( &mut self, pred: &'a OpaquePredicate, - ) -> Option> { + ) -> Self::Output { self.recurse_into_pred_opaque(pred) } - /// Called for each [`Predicate::Unknown`] encountered during the traversal. - fn transform_pred_unknown(&mut self, name: &'a String) -> Option> { - Some(Cow::Borrowed(name)) + /// Called for each unknown predicate encountered during the traversal (leaf). + fn transform_pred_unknown(&mut self, name: &'a String) -> Self::Output { + Carrier::from_inner(Cow::Borrowed(name)) } /// General entry point for transforming an expression. This method will dispatch to the /// specific transform for each expression variant. Also invoked internally in order to recurse - /// on the child(ren) of non-leaf variants. - fn transform_expr(&mut self, expr: &'a Expression) -> Option> { - let expr = match expr { - Expression::Literal(s) => self - .transform_expr_literal(s)? - .map_owned_or_else(expr, Expression::Literal), - Expression::Column(c) => self - .transform_expr_column(c)? - .map_owned_or_else(expr, Expression::Column), - Expression::Predicate(p) => self - .transform_expr_pred(p)? - .map_owned_or_else(expr, Expression::from), - Expression::Struct(s, nullability) => self - .transform_expr_struct(s)? - .map_owned_or_else(expr, |exprs| Expression::Struct(exprs, nullability.clone())), - Expression::Transform(t) => self - .transform_expr_transform(t)? - .map_owned_or_else(expr, Expression::Transform), - Expression::Unary(u) => self - .transform_expr_unary(u)? - .map_owned_or_else(expr, Expression::Unary), - Expression::Binary(b) => self - .transform_expr_binary(b)? - .map_owned_or_else(expr, Expression::Binary), - Expression::Variadic(v) => self - .transform_expr_variadic(v)? - .map_owned_or_else(expr, Expression::Variadic), - Expression::Opaque(o) => self - .transform_expr_opaque(o)? - .map_owned_or_else(expr, Expression::Opaque), - Expression::ParseJson(p) => self - .transform_expr_parse_json(p)? - .map_owned_or_else(expr, Expression::ParseJson), - Expression::MapToStruct(m) => self - .transform_expr_map_to_struct(m)? - .map_owned_or_else(expr, Expression::MapToStruct), - Expression::Unknown(u) => self - .transform_expr_unknown(u)? - .map_owned_or_else(expr, Expression::Unknown), - }; - Some(expr) + /// on the child(ren) of non-leaf expressions. + fn transform_expr(&mut self, expr: &'a Expression) -> Self::Output { + match expr { + Expression::Literal(s) => { + let child = self.transform_expr_literal(s); + map_owned_or_else(expr, child, Expression::Literal) + } + Expression::Column(c) => { + let child = self.transform_expr_column(c); + map_owned_or_else(expr, child, Expression::Column) + } + Expression::Predicate(p) => { + let child = self.transform_expr_pred(p); + map_owned_or_else(expr, child, Expression::from) + } + Expression::Struct(s, nullability) => { + let map_owned = |exprs| Expression::Struct(exprs, nullability.clone()); + map_owned_or_else(expr, self.transform_expr_struct(s), map_owned) + } + Expression::StructPatch(t) => { + let child = self.transform_expr_struct_patch(t); + map_owned_or_else(expr, child, Expression::StructPatch) + } + Expression::Unary(u) => { + let child = self.transform_expr_unary(u); + map_owned_or_else(expr, child, Expression::Unary) + } + Expression::Binary(b) => { + let child = self.transform_expr_binary(b); + map_owned_or_else(expr, child, Expression::Binary) + } + Expression::Variadic(v) => { + let child = self.transform_expr_variadic(v); + map_owned_or_else(expr, child, Expression::Variadic) + } + Expression::Opaque(o) => { + let child = self.transform_expr_opaque(o); + map_owned_or_else(expr, child, Expression::Opaque) + } + Expression::ParseJson(p) => { + let child = self.transform_expr_parse_json(p); + map_owned_or_else(expr, child, Expression::ParseJson) + } + Expression::MapToStruct(m) => { + let child = self.transform_expr_map_to_struct(m); + map_owned_or_else(expr, child, Expression::MapToStruct) + } + Expression::ElementAt(e) => { + let child = self.transform_expr_element_at(e); + map_owned_or_else(expr, child, Expression::ElementAt) + } + Expression::Cast(c) => { + let child = self.transform_expr_cast(c); + map_owned_or_else(expr, child, Expression::Cast) + } + Expression::Unknown(u) => { + let child = self.transform_expr_unknown(u); + map_owned_or_else(expr, child, Expression::Unknown) + } + } } /// General entry point for transforming a predicate. This method will dispatch to the specific /// transform for each predicate variant. Also invoked internally in order to recurse on the /// child(ren) of non-leaf variants. - fn transform_pred(&mut self, pred: &'a Predicate) -> Option> { - let pred = match pred { - Predicate::BooleanExpression(e) => self - .transform_expr(e)? - .map_owned_or_else(pred, Predicate::BooleanExpression), - Predicate::Not(p) => self.transform_pred_not(p)?.map_owned_or_else(pred, |p| p), - Predicate::Unary(u) => self - .transform_pred_unary(u)? - .map_owned_or_else(pred, Predicate::Unary), - Predicate::Binary(b) => self - .transform_pred_binary(b)? - .map_owned_or_else(pred, Predicate::Binary), + fn transform_pred(&mut self, pred: &'a Predicate) -> Self::Output { + match pred { + Predicate::BooleanExpression(e) => { + let child = self.transform_expr(e); + map_owned_or_else(pred, child, Predicate::BooleanExpression) + } + Predicate::Not(p) => { + let child = self.transform_pred_not(p); + map_owned_or_else(pred, child, |p| p) + } + Predicate::Unary(u) => { + let child = self.transform_pred_unary(u); + map_owned_or_else(pred, child, Predicate::Unary) + } + Predicate::Binary(b) => { + let child = self.transform_pred_binary(b); + map_owned_or_else(pred, child, Predicate::Binary) + } // Route through the constructor to normalize in case the transform removed children. // When `transform_pred` returns `None` for a child, it is filtered out, which may // reduce the junction to one or zero elements. The constructor normalizes these. - Predicate::Junction(j) => self - .transform_pred_junction(j)? - .map_owned_or_else(pred, |j| Predicate::junction(j.op, j.preds)), - Predicate::Opaque(o) => self - .transform_pred_opaque(o)? - .map_owned_or_else(pred, Predicate::Opaque), - Predicate::Unknown(u) => self - .transform_pred_unknown(u)? - .map_owned_or_else(pred, Predicate::Unknown), - }; - Some(pred) + Predicate::Junction(j) => { + let child = self.transform_pred_junction(j); + map_owned_or_else(pred, child, |j| Predicate::junction(j.op, j.preds)) + } + Predicate::Opaque(o) => { + let child = self.transform_pred_opaque(o); + map_owned_or_else(pred, child, Predicate::Opaque) + } + Predicate::Unknown(u) => { + let child = self.transform_pred_unknown(u); + map_owned_or_else(pred, child, Predicate::Unknown) + } + } } - /// Recursively transforms a struct's child expressions. Returns `None` if all children were - /// removed, `Some(Cow::Owned)` if at least one child was changed or removed, and - /// `Some(Cow::Borrowed)` otherwise. + /// Recursively transforms a struct's child expressions (variadic). fn recurse_into_expr_struct( &mut self, fields: &'a [ExpressionRef], - ) -> Option> { - let transformed_children = fields.iter().map(|f| { - let transformed = self.transform_expr(f)?; - Some(transformed.map_owned_or_else(f, Arc::new)) + ) -> Self::Output<[ExpressionRef]> { + let children = fields.iter().map(|f| -> Self::Output { + map_owned_or_else(f, self.transform_expr(f), Arc::new) }); - map_owned_children_or_else(fields, transformed_children, |fields| fields) + map_owned_children_or_else(fields, children, |fields| fields) } - /// Recursively transforms the child expression of a [`ParseJsonExpression`]. The schema is - /// not transformed. Returns `None` if the child was removed, `Some(Cow::Owned)` if the child - /// was changed, and `Some(Cow::Borrowed)` otherwise. + /// Recursively transforms the child expression of a parse-json expression (unary). fn recurse_into_expr_parse_json( &mut self, expr: &'a ParseJsonExpression, - ) -> Option> { - let nested = self.transform_expr(&expr.json_expr)?; - Some(nested.map_owned_or_else(expr, |json_expr| { - ParseJsonExpression::new(json_expr, expr.output_schema.clone()) - })) + ) -> Self::Output { + let f = |json_expr| ParseJsonExpression::new(json_expr, expr.output_schema.clone()); + map_owned_or_else(expr, self.transform_expr(&expr.json_expr), f) } - /// Recursively transforms the child expression of a [`MapToStructExpression`]. Returns `None` - /// if the child was removed, `Some(Cow::Owned)` if the child was changed, and - /// `Some(Cow::Borrowed)` otherwise. + /// Recursively transforms the child expression of a map-to-struct expression (unary). fn recurse_into_expr_map_to_struct( &mut self, expr: &'a MapToStructExpression, - ) -> Option> { - let nested = self.transform_expr(&expr.map_expr)?; - Some(nested.map_owned_or_else(expr, MapToStructExpression::new)) + ) -> Self::Output { + let nested = self.transform_expr(&expr.map_expr); + let rebuild = |map_expr| MapToStructExpression::new(map_expr, expr.options.clone()); + map_owned_or_else(expr, nested, rebuild) + } + + /// Recursively transforms both children of an element-at expression. + fn recurse_into_expr_element_at( + &mut self, + expr: &'a ElementAtExpression, + ) -> Self::Output { + let map_expr = self.transform_expr(&expr.map_expr); + let key_expr = self.transform_expr(&expr.key_expr); + let rebuild = |(map_expr, key_expr)| ElementAtExpression::new(map_expr, key_expr); + map_owned_pair_or_else(expr, map_expr, key_expr, rebuild) } - /// Recursively transforms the children of an [`OpaqueExpression`]. Returns `None` if all - /// children were removed, `Some(Cow::Owned)` if at least one child was changed or removed, and - /// `Some(Cow::Borrowed)` otherwise. + /// Recursively transforms the child expression of a cast expression (unary). + fn recurse_into_expr_cast(&mut self, expr: &'a CastExpression) -> Self::Output { + let rebuild = |child| CastExpression::new(child, expr.target.clone(), expr.options.clone()); + map_owned_or_else(expr, self.transform_expr(&expr.expr), rebuild) + } + + /// Recursively transforms the children of an opaque expression (variadic). fn recurse_into_expr_opaque( &mut self, o: &'a OpaqueExpression, - ) -> Option> { + ) -> Self::Output { let transformed_children = o.exprs.iter().map(|e| self.transform_expr(e)); let map_owned = |exprs| OpaqueExpression::new(o.op.clone(), exprs); map_owned_children_or_else(o, transformed_children, map_owned) } - /// Recursively transforms the child of an [`Expression::Predicate`]. Returns `None` if all - /// children were removed, `Some(Cow::Owned)` if at least one child was changed or removed, and - /// `Some(Cow::Borrowed)` otherwise. - fn recurse_into_expr_pred(&mut self, pred: &'a Predicate) -> Option> { + /// Recursively transforms the child of a predicate expression (unary). + fn recurse_into_expr_pred(&mut self, pred: &'a Predicate) -> Self::Output { self.transform_pred(pred) } - /// Recursively transforms the child of a [`Predicate::Not`] expression. Returns `None` if the - /// child was removed, `Some(Cow::Owned)` if the child was changed, and `Some(Cow::Borrowed)` - /// otherwise. - fn recurse_into_pred_not(&mut self, p: &'a Predicate) -> Option> { - Some(self.transform_pred(p)?.map_owned_or_else(p, Predicate::not)) + /// Recursively transforms the child of a not predicate expression (unary). + fn recurse_into_pred_not(&mut self, p: &'a Predicate) -> Self::Output { + map_owned_or_else(p, self.transform_pred(p), Predicate::not) } - /// Recursively transforms a unary predicate's child. Returns `None` if the child was removed, - /// `Some(Cow::Owned)` if the child was changed, and `Some(Cow::Borrowed)` otherwise. - fn recurse_into_pred_unary( - &mut self, - u: &'a UnaryPredicate, - ) -> Option> { - let nested_result = self.transform_expr(&u.expr)?; - Some(nested_result.map_owned_or_else(u, |expr| UnaryPredicate::new(u.op, expr))) + /// Recursively transforms a unary predicate's child (unary). + fn recurse_into_pred_unary(&mut self, u: &'a UnaryPredicate) -> Self::Output { + let nested = self.transform_expr(&u.expr); + map_owned_or_else(u, nested, |expr| UnaryPredicate::new(u.op, expr)) } - /// Recursively transforms a binary predicate's children. Returns `None` if at least one child - /// was removed, `Some(Cow::Owned)` if at least one child changed, and `Some(Cow::Borrowed)` - /// otherwise. + /// Recursively transforms a binary predicate's children (binary). fn recurse_into_pred_binary( &mut self, b: &'a BinaryPredicate, - ) -> Option> { - let left = self.transform_expr(&b.left)?; - let right = self.transform_expr(&b.right)?; + ) -> Self::Output { + let left = self.transform_expr(&b.left); + let right = self.transform_expr(&b.right); let f = |(left, right)| BinaryPredicate::new(b.op, left, right); - Some((left, right).map_owned_or_else(b, f)) + map_owned_pair_or_else(b, left, right, f) } - /// Recursively transforms a unary expression's child. Returns `None` if the child was removed, - /// `Some(Cow::Owned)` if the child was changed, and `Some(Cow::Borrowed)` otherwise. - fn recurse_into_expr_unary( - &mut self, - u: &'a UnaryExpression, - ) -> Option> { - let nested_result = self.transform_expr(&u.expr)?; - Some(nested_result.map_owned_or_else(u, |expr| UnaryExpression::new(u.op, expr))) + /// Recursively transforms a unary expression's child (unary). + fn recurse_into_expr_unary(&mut self, u: &'a UnaryExpression) -> Self::Output { + let nested = self.transform_expr(&u.expr); + map_owned_or_else(u, nested, |expr| UnaryExpression::new(u.op, expr)) } - /// Recursively transforms a binary expression's children. Returns `None` if at least one child - /// was removed, `Some(Cow::Owned)` if at least one child changed, and `Some(Cow::Borrowed)` - /// otherwise. + /// Recursively transforms a binary expression's children (binary). fn recurse_into_expr_binary( &mut self, b: &'a BinaryExpression, - ) -> Option> { - let left = self.transform_expr(&b.left)?; - let right = self.transform_expr(&b.right)?; + ) -> Self::Output { + let left = self.transform_expr(&b.left); + let right = self.transform_expr(&b.right); let f = |(left, right)| BinaryExpression::new(b.op, left, right); - Some((left, right).map_owned_or_else(b, f)) + map_owned_pair_or_else(b, left, right, f) } - /// Recursively transforms a variadic expression's children. Returns `None` if all children were - /// removed, `Some(Cow::Owned)` if at least one child was changed or removed, and - /// `Some(Cow::Borrowed)` otherwise. + /// Recursively transforms a variadic expression's children (variadic). fn recurse_into_expr_variadic( &mut self, v: &'a VariadicExpression, - ) -> Option> { - let transformed_children = v.exprs.iter().map(|e| self.transform_expr(e)); - let map_owned = |exprs| VariadicExpression::new(v.op, exprs); - map_owned_children_or_else(v, transformed_children, map_owned) + ) -> Self::Output { + let children = v.exprs.iter().map(|e| self.transform_expr(e)); + map_owned_children_or_else(v, children, |exprs| VariadicExpression::new(v.op, exprs)) } - /// Recursively transforms a junction predicate's children. Returns `None` if all children were - /// removed, `Some(Cow::Owned)` if at least one child was changed or removed, and - /// `Some(Cow::Borrowed)` otherwise. + /// Recursively transforms a junction predicate's children (variadic). fn recurse_into_pred_junction( &mut self, j: &'a JunctionPredicate, - ) -> Option> { - let transformed_children = j.preds.iter().map(|p| self.transform_pred(p)); - let map_owned = |preds| JunctionPredicate::new(j.op, preds); - map_owned_children_or_else(j, transformed_children, map_owned) + ) -> Self::Output { + let children = j.preds.iter().map(|p| self.transform_pred(p)); + map_owned_children_or_else(j, children, |preds| JunctionPredicate::new(j.op, preds)) } - /// Recursively transforms the children of an [`OpaquePredicate`]. Returns `None` if all - /// children were removed, `Some(Cow::Owned)` if at least one child was changed or removed, and - /// `Some(Cow::Borrowed)` otherwise. + /// Recursively transforms an opaque predicate's children (variadic). fn recurse_into_pred_opaque( &mut self, o: &'a OpaquePredicate, - ) -> Option> { - let transformed_children = o.exprs.iter().map(|e| self.transform_expr(e)); + ) -> Self::Output { + let children = o.exprs.iter().map(|e| self.transform_expr(e)); let map_owned = |exprs| OpaquePredicate::new(o.op.clone(), exprs); - map_owned_children_or_else(o, transformed_children, map_owned) + map_owned_children_or_else(o, children, map_owned) } } @@ -420,14 +488,14 @@ impl ExpressionDepthChecker { // Exposed for testing fn check_expr_with_call_count(expr: &Expression, depth_limit: usize) -> (usize, usize) { let mut checker = Self::new(depth_limit); - checker.transform_expr(expr); + let _ = checker.transform_expr(expr); (checker.max_depth_seen, checker.call_count) } // Exposed for testing fn check_pred_with_call_count(pred: &Predicate, depth_limit: usize) -> (usize, usize) { let mut checker = Self::new(depth_limit); - checker.transform_pred(pred); + let _ = checker.transform_pred(pred); (checker.max_depth_seen, checker.call_count) } @@ -443,110 +511,95 @@ impl ExpressionDepthChecker { // Triggers the requested recursion only doing so would not exceed the depth limit. fn depth_limited<'a, T: std::fmt::Debug + ToOwned + ?Sized>( &mut self, - recurse: impl FnOnce(&mut Self, &'a T) -> Option>, + recurse: impl FnOnce(&mut Self, &'a T) -> DeltaResult<()>, arg: &'a T, - ) -> Option> { + ) -> DeltaResult<()> { self.call_count += 1; - if self.max_depth_seen < self.current_depth { + if self.current_depth > self.max_depth_seen { self.max_depth_seen = self.current_depth; - if self.depth_limit < self.current_depth { - tracing::warn!( + if self.current_depth > self.depth_limit { + return Err(Error::schema(format!( "Max expression depth {} exceeded by {arg:?}", self.depth_limit - ); + ))); } } - if self.max_depth_seen <= self.depth_limit { - self.current_depth += 1; - let _ = recurse(self, arg); - self.current_depth -= 1; - } - None + self.current_depth += 1; + let result = recurse(self, arg); + self.current_depth -= 1; + result } } impl<'a> ExpressionTransform<'a> for ExpressionDepthChecker { - fn transform_expr_struct( - &mut self, - fields: &'a [ExpressionRef], - ) -> Option> { + transform_output_type!(|'a, T| DeltaResult<()>); + + fn transform_expr_cast(&mut self, expr: &'a CastExpression) -> DeltaResult<()> { + self.depth_limited(Self::recurse_into_expr_cast, expr) + } + + fn transform_expr_struct(&mut self, fields: &'a [ExpressionRef]) -> DeltaResult<()> { self.depth_limited(Self::recurse_into_expr_struct, fields) } - fn transform_expr_pred(&mut self, pred: &'a Predicate) -> Option> { + fn transform_expr_pred(&mut self, pred: &'a Predicate) -> DeltaResult<()> { self.depth_limited(Self::recurse_into_expr_pred, pred) } - fn transform_pred_not(&mut self, pred: &'a Predicate) -> Option> { + fn transform_pred_not(&mut self, pred: &'a Predicate) -> DeltaResult<()> { self.depth_limited(Self::recurse_into_pred_not, pred) } - fn transform_pred_unary( - &mut self, - pred: &'a UnaryPredicate, - ) -> Option> { + fn transform_pred_unary(&mut self, pred: &'a UnaryPredicate) -> DeltaResult<()> { self.depth_limited(Self::recurse_into_pred_unary, pred) } - fn transform_expr_binary( - &mut self, - expr: &'a BinaryExpression, - ) -> Option> { + fn transform_expr_binary(&mut self, expr: &'a BinaryExpression) -> DeltaResult<()> { self.depth_limited(Self::recurse_into_expr_binary, expr) } - fn transform_pred_binary( - &mut self, - pred: &'a BinaryPredicate, - ) -> Option> { + fn transform_pred_binary(&mut self, pred: &'a BinaryPredicate) -> DeltaResult<()> { self.depth_limited(Self::recurse_into_pred_binary, pred) } - fn transform_pred_junction( - &mut self, - pred: &'a JunctionPredicate, - ) -> Option> { + fn transform_pred_junction(&mut self, pred: &'a JunctionPredicate) -> DeltaResult<()> { self.depth_limited(Self::recurse_into_pred_junction, pred) } - fn transform_pred_opaque( - &mut self, - pred: &'a OpaquePredicate, - ) -> Option> { + fn transform_pred_opaque(&mut self, pred: &'a OpaquePredicate) -> DeltaResult<()> { self.depth_limited(Self::recurse_into_pred_opaque, pred) } - fn transform_expr_opaque( - &mut self, - expr: &'a OpaqueExpression, - ) -> Option> { + fn transform_expr_opaque(&mut self, expr: &'a OpaqueExpression) -> DeltaResult<()> { self.depth_limited(Self::recurse_into_expr_opaque, expr) } - fn transform_expr_map_to_struct( - &mut self, - expr: &'a MapToStructExpression, - ) -> Option> { + fn transform_expr_map_to_struct(&mut self, expr: &'a MapToStructExpression) -> DeltaResult<()> { self.depth_limited(Self::recurse_into_expr_map_to_struct, expr) } + + fn transform_expr_element_at(&mut self, expr: &'a ElementAtExpression) -> DeltaResult<()> { + self.depth_limited(Self::recurse_into_expr_element_at, expr) + } } #[cfg(test)] mod tests { + use std::borrow::Cow; + use std::sync::Arc; + use super::*; use crate::expressions::VariadicExpressionOp::Coalesce; use crate::expressions::{ - column_expr, column_pred, Expression, Expression as Expr, OpaqueExpressionOp, - OpaquePredicateOp, ParseJsonExpression, Predicate as Pred, Scalar, - ScalarExpressionEvaluator, VariadicExpression, + col, column_name, column_pred, lit, CastOptions, Expression, Expression as Expr, + MapToStructOptions, OpaqueExpressionOp, OpaquePredicateOp, ParseJsonExpression, + Predicate as Pred, Scalar, ScalarExpressionEvaluator, VariadicExpression, }; use crate::kernel_predicates::{ DirectDataSkippingPredicateEvaluator, DirectPredicateEvaluator, IndirectDataSkippingPredicateEvaluator, }; use crate::schema::{DataType, StructField, StructType}; - use crate::DeltaResult; - use std::sync::Arc; #[derive(Debug, PartialEq)] struct OpaqueTestOp(String); @@ -599,15 +652,19 @@ mod tests { } struct NoopTransform; - impl ExpressionTransform<'_> for NoopTransform {} + impl<'a> ExpressionTransform<'a> for NoopTransform { + transform_output_type!(|'a, T| Cow<'a, T>); + } struct ColumnReplacer; impl<'a> ExpressionTransform<'a> for ColumnReplacer { - fn transform_expr_column(&mut self, name: &'a ColumnName) -> Option> { + transform_output_type!(|'a, T| Cow<'a, T>); + + fn transform_expr_column(&mut self, name: &'a ColumnName) -> Cow<'a, ColumnName> { if name.len() == 1 && name[0] == "old_col" { - Some(Cow::Owned(ColumnName::new(["new_col"]))) + Cow::Owned(column_name!("new_col")) } else { - Some(Cow::Borrowed(name)) + Cow::Borrowed(name) } } } @@ -615,16 +672,13 @@ mod tests { #[test] fn test_transform_expr_variadic_noop() { // Test default no-op behavior - should return Cow::Borrowed - let variadic_expr = VariadicExpression::new( - Coalesce, - vec![Expr::literal(1), column_expr!("x"), Expr::literal("test")], - ); + let variadic_expr = VariadicExpression::new(Coalesce, vec![lit(1), col!("x"), lit("test")]); let mut transform = NoopTransform; let result = transform.transform_expr_variadic(&variadic_expr); - assert!(matches!(result, Some(Cow::Borrowed(_)))); - if let Some(Cow::Borrowed(result_expr)) = result { + assert!(matches!(result, Cow::Borrowed(_))); + if let Cow::Borrowed(result_expr) = result { assert_eq!(result_expr, &variadic_expr); } } @@ -637,9 +691,8 @@ mod tests { let mut transform = NoopTransform; let result = transform.transform_expr_variadic(&variadic_expr); - // Empty children list with no-op transform returns None because new_children.is_empty() - // This is the behavior of recurse_into_children when starting with empty slice - assert!(result.is_none()); + // Empty variadic lists remain present for non-filtering carriers. + assert!(matches!(result, Cow::Borrowed(_))); } #[test] @@ -647,19 +700,13 @@ mod tests { // Test transformation of child expressions - should return Cow::Owned let variadic_expr = VariadicExpression::new( Coalesce, - vec![ - Expr::literal(1), - column_expr!("old_col"), - column_expr!("unchanged_col"), - Expr::literal("test"), - ], + vec![lit(1), col!("old_col"), col!("unchanged_col"), lit("test")], ); - let mut transform = ColumnReplacer; - let result = transform.transform_expr_variadic(&variadic_expr); + let result = ColumnReplacer.transform_expr_variadic(&variadic_expr); - assert!(matches!(result, Some(Cow::Owned(_)))); - if let Some(Cow::Owned(result_expr)) = result { + assert!(matches!(result, Cow::Owned(_))); + if let Cow::Owned(result_expr) = result { assert_eq!(result_expr.op, Coalesce); assert_eq!(result_expr.exprs.len(), 4); @@ -688,20 +735,15 @@ mod tests { // Test removal of child expressions - should return Cow::Owned with fewer children struct LiteralRemover; impl<'a> ExpressionTransform<'a> for LiteralRemover { + transform_output_type!(|'a, T| Option>); + fn transform_expr_literal(&mut self, _value: &'a Scalar) -> Option> { None // Remove all literals } } - let variadic_expr = VariadicExpression::new( - Coalesce, - vec![ - Expr::literal(1), - column_expr!("x"), - Expr::literal("test"), - column_expr!("y"), - ], - ); + let variadic_expr = + VariadicExpression::new(Coalesce, vec![lit(1), col!("x"), lit("test"), col!("y")]); let mut transform = LiteralRemover; let result = transform.transform_expr_variadic(&variadic_expr); @@ -732,6 +774,8 @@ mod tests { // Test edge case where all children are removed - should return None struct RemoveAll; impl<'a> ExpressionTransform<'a> for RemoveAll { + transform_output_type!(|'a, T| Option>); + fn transform_expr_literal(&mut self, _value: &'a Scalar) -> Option> { None } @@ -743,10 +787,7 @@ mod tests { } } - let variadic_expr = VariadicExpression::new( - Coalesce, - vec![Expr::literal(1), column_expr!("x"), Expr::literal("test")], - ); + let variadic_expr = VariadicExpression::new(Coalesce, vec![lit(1), col!("x"), lit("test")]); let mut transform = RemoveAll; let result = transform.transform_expr_variadic(&variadic_expr); @@ -759,11 +800,13 @@ mod tests { // Test mixed scenario: some children transformed, some removed, some unchanged struct MixedTransform; impl<'a> ExpressionTransform<'a> for MixedTransform { + transform_output_type!(|'a, T| Option>); + fn transform_expr_literal(&mut self, value: &'a Scalar) -> Option> { match value { Scalar::Integer(1) => None, // Remove literal 1 Scalar::String(s) if s == "remove" => None, // Remove "remove" string - Scalar::Integer(n) => Some(Cow::Owned(Scalar::Integer(n * 2))), // Double other integers + Scalar::Integer(n) => Some(Cow::Owned(Scalar::Integer(n * 2))), /* Double other integers */ _ => Some(Cow::Borrowed(value)), // Keep others unchanged } } @@ -772,7 +815,7 @@ mod tests { name: &'a ColumnName, ) -> Option> { if name.len() == 1 && name[0] == "transform_me" { - Some(Cow::Owned(ColumnName::new(["transformed"]))) + Some(Cow::Owned(column_name!("transformed"))) } else { Some(Cow::Borrowed(name)) } @@ -782,12 +825,12 @@ mod tests { let variadic_expr = VariadicExpression::new( Coalesce, vec![ - Expr::literal(1), // Will be removed - column_expr!("unchanged"), // Will stay unchanged - Expr::literal(5), // Will be transformed to 10 - Expr::literal("remove"), // Will be removed - column_expr!("transform_me"), // Will be transformed - Expr::literal("keep"), // Will stay unchanged + lit(1), // Will be removed + col!("unchanged"), // Will stay unchanged + lit(5), // Will be transformed to 10 + lit("remove"), // Will be removed + col!("transform_me"), // Will be transformed + lit("keep"), // Will stay unchanged ], ); @@ -830,14 +873,13 @@ mod tests { #[test] fn test_transform_expr_parse_json_noop() { // Test default no-op behavior - should return Cow::Borrowed - let parse_json_expr = - ParseJsonExpression::new(column_expr!("json_col"), test_output_schema()); + let parse_json_expr = ParseJsonExpression::new(col!("json_col"), test_output_schema()); let mut transform = NoopTransform; let result = transform.transform_expr_parse_json(&parse_json_expr); - assert!(matches!(result, Some(Cow::Borrowed(_)))); - if let Some(Cow::Borrowed(result_expr)) = result { + assert!(matches!(result, Cow::Borrowed(_))); + if let Cow::Borrowed(result_expr) = result { assert_eq!(result_expr, &parse_json_expr); } } @@ -845,14 +887,12 @@ mod tests { #[test] fn test_transform_expr_parse_json_child_transformation() { // Test transformation of child expression - should return Cow::Owned - let parse_json_expr = - ParseJsonExpression::new(column_expr!("old_col"), test_output_schema()); + let parse_json_expr = ParseJsonExpression::new(col!("old_col"), test_output_schema()); - let mut transform = ColumnReplacer; - let result = transform.transform_expr_parse_json(&parse_json_expr); + let result = ColumnReplacer.transform_expr_parse_json(&parse_json_expr); - assert!(matches!(result, Some(Cow::Owned(_)))); - if let Some(Cow::Owned(result_expr)) = result { + assert!(matches!(result, Cow::Owned(_))); + if let Cow::Owned(result_expr) = result { // Check that the column was replaced if let Expr::Column(col) = result_expr.json_expr.as_ref() { assert_eq!(col.len(), 1); @@ -865,17 +905,49 @@ mod tests { } } + #[test] + fn test_map_to_struct_options_survive_child_transform() { + let expr = Expr::map_to_struct( + col!("old_col"), + MapToStructOptions::default().with_timestamp_timezone("Europe/Berlin"), + ); + let Expr::MapToStruct(transformed) = ColumnReplacer.transform_expr(&expr).into_owned() + else { + panic!("expected map-to-struct expression"); + }; + assert_eq!(transformed.map_expr.as_ref(), &col!("new_col")); + assert_eq!( + transformed.options.timestamp_timezone(), + Some("Europe/Berlin") + ); + } + + #[test] + fn test_cast_options_survive_child_transform() { + let expr = Expr::cast( + col!("old_col"), + DataType::TIMESTAMP, + CastOptions::default().with_timestamp_timezone("Europe/Berlin"), + ); + let Expr::Cast(transformed) = ColumnReplacer.transform_expr(&expr).into_owned() else { + panic!("expected cast expression"); + }; + assert_eq!(transformed.expr.as_ref(), &col!("new_col")); + assert_eq!( + transformed.options.timestamp_timezone(), + Some("Europe/Berlin") + ); + } + #[test] fn test_transform_expr_parse_json_child_unchanged() { // Test when child column doesn't match replacement criteria - should return Cow::Borrowed - let parse_json_expr = - ParseJsonExpression::new(column_expr!("unchanged_col"), test_output_schema()); + let parse_json_expr = ParseJsonExpression::new(col!("unchanged_col"), test_output_schema()); - let mut transform = ColumnReplacer; - let result = transform.transform_expr_parse_json(&parse_json_expr); + let result = ColumnReplacer.transform_expr_parse_json(&parse_json_expr); - // Since "unchanged_col" doesn't match "old_col", nothing changes - assert!(matches!(result, Some(Cow::Borrowed(_)))); + // Since "json_col" doesn't match "old_col", nothing changes + assert!(matches!(result, Cow::Borrowed(_))); } #[test] @@ -883,6 +955,8 @@ mod tests { // Test removal of child expression - should return None struct ColumnRemover; impl<'a> ExpressionTransform<'a> for ColumnRemover { + transform_output_type!(|'a, T| Option>); + fn transform_expr_column( &mut self, _name: &'a ColumnName, @@ -891,8 +965,7 @@ mod tests { } } - let parse_json_expr = - ParseJsonExpression::new(column_expr!("json_col"), test_output_schema()); + let parse_json_expr = ParseJsonExpression::new(col!("json_col"), test_output_schema()); let mut transform = ColumnRemover; let result = transform.transform_expr_parse_json(&parse_json_expr); @@ -906,24 +979,26 @@ mod tests { // Test with a more complex nested child expression struct LiteralDoubler; impl<'a> ExpressionTransform<'a> for LiteralDoubler { - fn transform_expr_literal(&mut self, value: &'a Scalar) -> Option> { + transform_output_type!(|'a, T| Cow<'a, T>); + + fn transform_expr_literal(&mut self, value: &'a Scalar) -> Cow<'a, Scalar> { if let Scalar::Integer(n) = value { - Some(Cow::Owned(Scalar::Integer(n * 2))) + Cow::Owned(Scalar::Integer(n * 2)) } else { - Some(Cow::Borrowed(value)) + Cow::Borrowed(value) } } } // ParseJson with a binary expression as child: column + 5 - let child_expr = column_expr!("x") + Expr::literal(5); + let child_expr = col!("x") + lit(5); let parse_json_expr = ParseJsonExpression::new(child_expr, test_output_schema()); let mut transform = LiteralDoubler; let result = transform.transform_expr_parse_json(&parse_json_expr); - assert!(matches!(result, Some(Cow::Owned(_)))); - if let Some(Cow::Owned(result_expr)) = result { + assert!(matches!(result, Cow::Owned(_))); + if let Cow::Owned(result_expr) = result { // The literal 5 should have been doubled to 10 if let Expr::Binary(binary) = result_expr.json_expr.as_ref() { if let Expr::Literal(Scalar::Integer(n)) = &*binary.right { @@ -943,40 +1018,31 @@ mod tests { Pred::and_from([ Pred::opaque( OpaqueTestOp("opaque".to_string()), - vec![ - Expr::literal(10) + column_expr!("x"), - Expr::unknown("unknown") - column_expr!("b"), - ], + vec![lit(10) + col!("x"), Expr::unknown("unknown") - col!("b")], ), - Pred::literal(true), - Pred::not(Pred::literal(true)), + Pred::TRUE, + Pred::not(Pred::TRUE), ]), Pred::and_from([ - Pred::is_null(column_expr!("b")), - Pred::gt(Expr::literal(10), column_expr!("x")), + Pred::is_null(col!("b")), + Pred::gt(lit(10), col!("x")), Pred::or( Pred::gt( - Expr::literal(5) - + Expr::opaque( - OpaqueTestOp("inscrutable".to_string()), - vec![Expr::literal(10)], - ), - Expr::literal(20), + lit(5) + + Expr::opaque(OpaqueTestOp("inscrutable".to_string()), vec![lit(10)]), + lit(20), ), column_pred!("y"), ), Pred::unknown("mystery"), ]), - Pred::eq( - Expr::literal(42), - Expr::struct_from([Expr::literal(10), column_expr!("b")]), - ), + Pred::eq(lit(42), Expr::struct_from([lit(10), col!("b")])), ]); // Verify the default/no-op transform, since we have this nice complex expression handy. assert!(matches!( NoopTransform.transform_pred(&pred), - Some(std::borrow::Cow::Borrowed(_)) + std::borrow::Cow::Borrowed(_) )); // Similar to ExpressionDepthChecker::check_pred, but also returns call count @@ -991,7 +1057,7 @@ mod tests { // * NOT // * AND // * EQ - assert_eq!(check_with_call_count(1), (2, 6)); + assert_eq!(check_with_call_count(1), (2, 3)); // OR // * AND @@ -1001,7 +1067,7 @@ mod tests { // * NOT // * AND // * EQ - assert_eq!(check_with_call_count(2), (3, 8)); + assert_eq!(check_with_call_count(2), (3, 4)); // OR // * AND @@ -1016,7 +1082,7 @@ mod tests { // * GT // * PLUS >LIMIT< // * EQ - assert_eq!(check_with_call_count(3), (4, 13)); + assert_eq!(check_with_call_count(3), (4, 12)); // OR // * AND @@ -1032,7 +1098,7 @@ mod tests { // * PLUS // * OPAQUE >LIMIT< // * EQ - assert_eq!(check_with_call_count(4), (5, 14)); + assert_eq!(check_with_call_count(4), (5, 13)); // Depth limit not hit (full traversal required) // @@ -1078,27 +1144,81 @@ mod tests { // * OPAQUE > LIMIT 5 < // * EQ // * STRUCT - assert_eq!(check_with_call_count(1), (2, 5)); - assert_eq!(check_with_call_count(2), (3, 7)); - assert_eq!(check_with_call_count(3), (4, 9)); - assert_eq!(check_with_call_count(4), (5, 14)); - assert_eq!(check_with_call_count(5), (6, 15)); + assert_eq!(check_with_call_count(1), (2, 3)); + assert_eq!(check_with_call_count(2), (3, 4)); + assert_eq!(check_with_call_count(3), (4, 5)); + assert_eq!(check_with_call_count(4), (5, 13)); + assert_eq!(check_with_call_count(5), (6, 14)); assert_eq!(check_with_call_count(6), (6, 16)); assert_eq!(check_with_call_count(7), (6, 16)); } + #[test] + fn test_depth_checker_counts_cast_expressions() { + let expr = Expr::cast( + Expr::cast(col!("x"), DataType::INTEGER, CastOptions::default()), + DataType::LONG, + CastOptions::default(), + ); + + assert_eq!( + ExpressionDepthChecker::check_expr_with_call_count(&expr, 0), + (1, 2) + ); + } + + #[test] + fn test_element_at_references_and_transforms_both_children() { + let expr = Expr::element_at(col!("old_col"), col!("partitionKey")); + let references = expr.references(); + assert_eq!(references.len(), 2); + assert!(references.contains(&ColumnName::new(["old_col"]))); + assert!(references.contains(&ColumnName::new(["partitionKey"]))); + + let transformed = ColumnReplacer.transform_expr(&expr).into_owned(); + assert_eq!( + transformed, + Expr::element_at(col!("new_col"), col!("partitionKey")) + ); + + struct ColumnRemover; + impl<'a> ExpressionTransform<'a> for ColumnRemover { + transform_output_type!(|'a, T| Option>); + + fn transform_expr_column( + &mut self, + name: &'a ColumnName, + ) -> Option> { + (name[0] != "partitionKey").then_some(Cow::Borrowed(name)) + } + } + assert!(ColumnRemover.transform_expr(&expr).is_none()); + } + + #[test] + fn test_depth_checker_counts_element_at_expressions() { + let expr = Expr::element_at(col!("partitionValues"), col!("partitionKey")); + + assert_eq!( + ExpressionDepthChecker::check_expr_with_call_count(&expr, 0), + (0, 1) + ); + } + #[test] fn transform_junction_to_single_child_unwraps() { // A transform that removes one child from AND(a, b) should produce the surviving // predicate directly, not a degenerate single-element junction AND(a). struct LiteralRemover; impl<'a> ExpressionTransform<'a> for LiteralRemover { + transform_output_type!(|'a, T| Option>); + fn transform_expr_literal(&mut self, _value: &'a Scalar) -> Option> { None } } - let pred = Pred::and(column_pred!("x"), Pred::literal(true)); + let pred = Pred::and(column_pred!("x"), Pred::TRUE); let mut transform = LiteralRemover; let result = transform.transform_pred(&pred); let result = result.map(Cow::into_owned); @@ -1112,6 +1232,8 @@ mod tests { // rather than producing an empty junction or identity literal. struct ColumnRemover; impl<'a> ExpressionTransform<'a> for ColumnRemover { + transform_output_type!(|'a, T| Option>); + fn transform_expr_column( &mut self, _name: &'a ColumnName, diff --git a/kernel/src/transforms/mod.rs b/kernel/src/transforms/mod.rs index b9cae91a62..8927eacaeb 100644 --- a/kernel/src/transforms/mod.rs +++ b/kernel/src/transforms/mod.rs @@ -1,75 +1,160 @@ +//! Shared transform infrastructure. use std::borrow::{Cow, ToOwned}; +mod carrier; mod expression; mod schema; -pub use self::expression::{ExpressionDepthChecker, ExpressionTransform}; -pub use self::schema::{SchemaDepthChecker, SchemaTransform}; - -// Extension trait for Cow<'_, T> -pub(crate) trait CowExt { - /// The owned type that corresponds to Self - type Owned; - - /// Propagate the results of nested transforms. If the nested transform made no change (borrowed - /// `self`), then return a borrowed result `s` as well. Otherwise, invoke the provided mapping - /// function `f` to convert the owned nested result into an owned result. - fn map_owned_or_else(self, s: &S, f: impl FnOnce(Self::Owned) -> S) -> Cow<'_, S>; -} -// Basic implementation for a single Cow value -impl CowExt for Cow<'_, T> { - type Owned = T::Owned; - - fn map_owned_or_else(self, s: &S, f: impl FnOnce(T::Owned) -> S) -> Cow<'_, S> { - match self { - Cow::Owned(v) => Cow::Owned(f(v)), - Cow::Borrowed(_) => Cow::Borrowed(s), - } - } -} +pub use carrier::Carrier; +pub(crate) use carrier::{carrier_into_inner_opt, carrier_try_none}; -// Additional implementation for a pair of Cow values -impl<'a, T: ToOwned + ?Sized> CowExt<(Cow<'a, T>, Cow<'a, T>)> for (Cow<'a, T>, Cow<'a, T>) { - type Owned = (T::Owned, T::Owned); +pub use self::expression::{ExpressionDepthChecker, ExpressionTransform}; +pub use self::schema::{SchemaDepthChecker, SchemaTransform}; - fn map_owned_or_else(self, s: &S, f: impl FnOnce(Self::Owned) -> S) -> Cow<'_, S> { - match self { - (Cow::Borrowed(_), Cow::Borrowed(_)) => Cow::Borrowed(s), - (left, right) => Cow::Owned(f((left.into_owned(), right.into_owned()))), - } - } +/// Defines a transform's `Output` and `Residual` associated types. +/// +/// Example: fallible schema visitor +/// ```rust,no_run +/// # use delta_kernel::transform_output_type; +/// # use delta_kernel::schema::StructField; +/// # use delta_kernel::transforms::SchemaTransform; +/// # use delta_kernel::DeltaResult; +/// struct Validate; +/// +/// impl<'a> SchemaTransform<'a> for Validate { +/// transform_output_type!(|'a, T| DeltaResult<()>); +/// +/// fn transform_struct_field(&mut self, _field: &'a StructField) -> DeltaResult<()> { +/// todo!() +/// } +/// } +/// ``` +/// +/// Example: infallible filtering expression transform +/// ```rust,no_run +/// # use std::borrow::Cow; +/// # use delta_kernel::transform_output_type; +/// # use delta_kernel::expressions::ColumnName; +/// # use delta_kernel::transforms::ExpressionTransform; +/// struct KeepSomeColumns; +/// +/// impl<'a> ExpressionTransform<'a> for KeepSomeColumns { +/// transform_output_type!(|'a, T| Option>); +/// +/// fn transform_expr_column(&mut self, _name: &'a ColumnName) -> Option> { +/// todo!() +/// } +/// } +/// ``` +#[macro_export] +macro_rules! transform_output_type { + (|$lt:lifetime, $T:ident| $out:ty) => { + type Output<$T: ::std::borrow::ToOwned + ?Sized + $lt> = $out; + type Residual = as $crate::transforms::Carrier<$lt, ()>>::Residual; + }; } +#[doc(inline)] +pub use transform_output_type; /// Rebuilds a parent from transformed children only when needed. /// -/// Child transforms may filter nodes by returning `None`. If all children are filtered out, this -/// returns `None`. If all original children survive as borrowed values, this returns a borrowed -/// parent. Otherwise, it rebuilds and returns an owned parent. -pub(crate) fn map_owned_children_or_else<'a, Parent, Child>( +/// This helper consumes transformed child carriers. When a filtering carrier is used and all +/// children were filtered out, filter out the parent as well. If all children were unchanged, it +/// returns a carrier for the original parent. Otherwise returns a carrier for a new owned parent, +/// produced by the provided `map_owned`. +pub(crate) fn map_owned_children_or_else<'a, Parent, Child, ChildCarrier, ParentCarrier, R>( parent: &'a Parent, - children: impl ExactSizeIterator>>, - map_owned: impl FnOnce(Vec) -> Parent::Owned, -) -> Option> + children: impl ExactSizeIterator, + map_owned: impl FnOnce(Vec<::Owned>) -> ::Owned, +) -> ParentCarrier where Parent: ToOwned + ?Sized, Child: ToOwned + ?Sized + 'a, + ChildCarrier: Carrier<'a, Child, Residual = R>, + ParentCarrier: Carrier<'a, Parent, Residual = R>, { let num_children = children.len(); let mut num_borrowed = 0; + + // NOTE: A transform in visitor mode uses the ZST () as Carrier (= vec allocates no memory), let mut new_children = Vec::with_capacity(num_children); - for child in children.flatten() { - if let Cow::Borrowed(_) = child { - num_borrowed += 1; + for child in children { + if let Some(transformed) = carrier_into_inner_opt!(child) { + if let Cow::Borrowed(_) = transformed { + num_borrowed += 1; + } + new_children.push(transformed); } - new_children.push(child); } + // If all children were filtered out, try to return the "None" carrier. If not supported, then + // the parent must have already been empty and it's safe to continue (return borrowed parent). if new_children.is_empty() { - None - } else if num_borrowed < num_children { + carrier_try_none!(); + } + + if num_borrowed < num_children { let owned = new_children.into_iter().map(Cow::into_owned).collect(); - Some(Cow::Owned(map_owned(owned))) + Carrier::from_inner(Cow::Owned(map_owned(owned))) } else { - Some(Cow::Borrowed(parent)) + Carrier::from_inner(Cow::Borrowed(parent)) } } + +/// Rebuilds a two-child parent from transformed children only when needed. +/// +/// If either child is filtered out (`None`), filter out the parent by returning `None`. If both +/// children survive as borrowed values, this returns a borrowed parent. Otherwise, it uses the +/// provided `map_owned` function to rebuild and return an owned parent. +pub(crate) fn map_owned_pair_or_else<'a, Parent, Child, ChildCarrier, ParentCarrier, R>( + parent: &'a Parent, + left: ChildCarrier, + right: ChildCarrier, + map_owned: impl FnOnce((Child::Owned, Child::Owned)) -> Parent, +) -> ParentCarrier +where + Parent: Clone, + Child: ToOwned + ?Sized + 'a, + ChildCarrier: Carrier<'a, Child, Residual = R>, + ParentCarrier: Carrier<'a, Parent, Residual = R>, +{ + let left = carrier_into_inner_opt!(left); + let right = carrier_into_inner_opt!(right); + let (Some(left), Some(right)) = (left, right) else { + // SAFETY: Only a filtering carrier could produce None => try_none must succeed. + carrier_try_none!(); + unreachable!(); + }; + Carrier::from_inner(match (left, right) { + (Cow::Borrowed(_), Cow::Borrowed(_)) => Cow::Borrowed(parent), + (left, right) => Cow::Owned(map_owned((left.into_owned(), right.into_owned()))), + }) +} + +/// Rebuilds a single-child parent from a transformed child only when needed. +/// +/// If the child is filtered out (`None`), filter out the parent by returning `None`. If the child +/// survives as a borrowed value, this returns a borrowed parent. Otherwise, it uses the provided +/// `map_owned` function to rebuild and return an owned parent. +pub(crate) fn map_owned_or_else<'a, Parent, Child, ChildCarrier, ParentCarrier, R>( + parent: &'a Parent, + child: ChildCarrier, + map_owned: impl FnOnce(Child::Owned) -> Parent, +) -> ParentCarrier +where + Parent: Clone, + Child: ToOwned + ?Sized + 'a, + ChildCarrier: Carrier<'a, Child, Residual = R>, + ParentCarrier: Carrier<'a, Parent, Residual = R>, +{ + let child = carrier_into_inner_opt!(child); + let Some(child) = child else { + // SAFETY: Only a filtering carrier could produce None => try_none must succeed. + carrier_try_none!(); + unreachable!(); + }; + Carrier::from_inner(match child { + Cow::Owned(v) => Cow::Owned(map_owned(v)), + Cow::Borrowed(_) => Cow::Borrowed(parent), + }) +} diff --git a/kernel/src/transforms/schema.rs b/kernel/src/transforms/schema.rs index 8abb77ad9b..9dead2c138 100644 --- a/kernel/src/transforms/schema.rs +++ b/kernel/src/transforms/schema.rs @@ -1,151 +1,195 @@ -use std::borrow::Cow; +use std::borrow::{Cow, ToOwned}; use crate::schema::{ArrayType, DataType, MapType, PrimitiveType, StructField, StructType}; -use crate::transforms::{map_owned_children_or_else, CowExt as _}; +use crate::transforms::{ + map_owned_children_or_else, map_owned_or_else, map_owned_pair_or_else, transform_output_type, + Carrier, +}; -/// Generic framework for describing recursive bottom-up schema transforms. Transformations return -/// `Option` with the following semantics: -/// * `Some(Cow::Owned)` -- The schema element was transformed and should propagate to its parent. -/// * `Some(Cow::Borrowed)` -- The schema element was not transformed. -/// * `None` -- The schema element was filtered out and the parent should no longer reference it. +/// Generic framework for describing recursive bottom-up schema transforms. /// /// The transform can start from whatever schema element is available /// (e.g. [`Self::transform_struct`] to start with [`StructType`]), or it can start from the generic /// [`Self::transform`]. /// -/// The provided `transform_xxx` methods all default to no-op, and implementations should -/// selectively override specific `transform_xxx` methods as needed for the task at hand. +/// The provided `transform_xxx` methods all default to no-op (usually by invoking the corresponding +/// recursive helper method), and implementations should selectively override specific +/// `transform_xxx` methods as needed for the task at hand. +/// +/// # Recursive helper methods /// /// The provided `recurse_into_xxx` methods encapsulate the boilerplate work of recursing into the -/// child schema elements of each schema element. Implementations can call these as needed but will -/// generally not need to override them. +/// child schema elements of each schema element. Except as specifically noted otherwise, these +/// recursive helpers all behave uniformly, based on the number of children the schema element has: +/// +/// * Leaf (no children) - Leaf `transform_xxx` methods simply return their argument unchanged, and +/// no corresponding `recurse_into_xxx` method is provided. +/// +/// * Unary (single child) - If the child was filtered out, filter out the parent. If the child +/// changed, build a new parent around it. Otherwise, return the parent unchanged. +/// +/// * Binary (two children) - If either child was filtered out, filter out the parent. If at least +/// one child changed, build a new parent around them. Otherwise, return the parent unchanged. +/// +/// * Variadic (0+ children) - If no children remain (all filtered out), filter out the parent. +/// Otherwise, if at least one child changed or was filtered out, build a new parent around the +/// children. Otherwise, return the parent unchanged. +/// +/// Implementations can call these as needed, but will generally not need to override them. +/// +/// # Transform carrier selection +/// +/// Implementations choose an output [`Carrier`] instance based on the operation to be +/// performed. That carrier determines the return type of each transform method. +/// +/// For example, a simple read-only visitor would use `()` as a carrier, while a validity checker +/// could use `DeltaResult<()>` instead. A mutating transform uses `Cow<_>`, returning `Cow::Owned` +/// for changed/replaced nodes, and a filtering transform uses `Option>`, where `None` +/// indicates the node should be dropped rather than replaced. `DeltaResult>` and +/// `Result>, E>` round out the set as fallible mutating and fitering transforms that +/// short circuit immediately upon `Err`. pub trait SchemaTransform<'a> { - /// Called for each primitive encountered during the schema traversal. - fn transform_primitive(&mut self, ptype: &'a PrimitiveType) -> Option> { - Some(Cow::Borrowed(ptype)) + /// [`Carrier`] output type for transformed nodes. + /// + /// Implementations can use [`crate::transforms::transform_output_type`] to define `Output` + /// and `Residual` together. + type Output: Carrier<'a, T, Residual = Self::Residual>; + + /// Residual type propagated by this transform's output [`Carrier`]. This type is statically + /// derivable, but still required due to limitations in rust type system expressiveness. + /// + /// Implementations can use [`crate::transforms::transform_output_type`] to define `Output` + /// and `Residual` together. Or, define it manually like this: + /// ```rust,no_run + /// # use std::borrow::Cow; + /// # use delta_kernel::transforms::SchemaTransform; + /// # use delta_kernel::transforms::Carrier; + /// # struct X; + /// # impl<'a> SchemaTransform<'a> for X { + /// # type Output = Cow<'a, T>; + /// type Residual = as Carrier<'a, ()>>::Residual; + /// # } + /// ``` + type Residual; + + /// Called for each primitive encountered during the traversal (leaf). + fn transform_primitive(&mut self, ptype: &'a PrimitiveType) -> Self::Output { + Carrier::from_inner(Cow::Borrowed(ptype)) } - /// Called for each struct encountered during the schema traversal. Implementations can call - /// [`Self::recurse_into_struct`] if they wish to recursively transform the struct's fields. - fn transform_struct(&mut self, stype: &'a StructType) -> Option> { + /// Called for each struct encountered during the traversal. The provided implementation just + /// forwards to [`Self::recurse_into_struct`]. + fn transform_struct(&mut self, stype: &'a StructType) -> Self::Output { self.recurse_into_struct(stype) } - /// Called for each struct field encountered during the schema traversal. Implementations can - /// call [`Self::recurse_into_struct_field`] if they wish to recursively transform the field's - /// data type. - fn transform_struct_field(&mut self, field: &'a StructField) -> Option> { + /// Called for each struct field encountered during the traversal. The provided implementation + /// forwards to [`Self::recurse_into_struct_field`]. + fn transform_struct_field(&mut self, field: &'a StructField) -> Self::Output { self.recurse_into_struct_field(field) } - /// Called for each array encountered during the schema traversal. Implementations can call - /// [`Self::recurse_into_array`] if they wish to recursively transform the array's element type. - fn transform_array(&mut self, atype: &'a ArrayType) -> Option> { + /// Called for each array encountered during the traversal. The provided implementation just + /// forwards to [`Self::recurse_into_array`]. + fn transform_array(&mut self, atype: &'a ArrayType) -> Self::Output { self.recurse_into_array(atype) } - /// Called for each array element encountered during the schema traversal. Implementations can - /// call [`Self::transform`] if they wish to recursively transform the array element type. - fn transform_array_element(&mut self, etype: &'a DataType) -> Option> { + /// Called for each array element type encountered during the traversal. The provided + /// implementation forwards to [`Self::transform`]. + fn transform_array_element(&mut self, etype: &'a DataType) -> Self::Output { self.transform(etype) } - /// Called for each map encountered during the schema traversal. Implementations can call - /// [`Self::recurse_into_map`] if they wish to recursively transform the map's key and/or value - /// types. - fn transform_map(&mut self, mtype: &'a MapType) -> Option> { + /// Called for each map encountered during the traversal. The provided implementation just + /// forwards to [`Self::recurse_into_map`]. + fn transform_map(&mut self, mtype: &'a MapType) -> Self::Output { self.recurse_into_map(mtype) } - /// Called for each map key encountered during the schema traversal. Implementations can call - /// [`Self::transform`] if they wish to recursively transform the map key type. - fn transform_map_key(&mut self, etype: &'a DataType) -> Option> { + /// Called for each map key encountered during the traversal. The provided implementation + /// forwards to [`Self::transform`]. + fn transform_map_key(&mut self, etype: &'a DataType) -> Self::Output { self.transform(etype) } - /// Called for each map value encountered during the schema traversal. Implementations can call - /// [`Self::transform`] if they wish to recursively transform the map value type. - fn transform_map_value(&mut self, etype: &'a DataType) -> Option> { + /// Called for each map value encountered during the traversal. The provided implementation + /// forwards to [`Self::transform`]. + fn transform_map_value(&mut self, etype: &'a DataType) -> Self::Output { self.transform(etype) } - /// Called for each variant value encountered. By default, recurses into the fields of the - /// variant struct type. - fn transform_variant(&mut self, stype: &'a StructType) -> Option> { + /// Called for each variant value encountered. The provided implementation just + /// forwards to [`Self::recurse_into_struct`]. + fn transform_variant(&mut self, stype: &'a StructType) -> Self::Output { self.recurse_into_struct(stype) } /// General entry point for a recursive traversal over any data type. Also invoked internally to /// dispatch on nested data types encountered during the traversal. - fn transform(&mut self, data_type: &'a DataType) -> Option> { - use DataType::*; - let result = match data_type { - Primitive(ptype) => self - .transform_primitive(ptype)? - .map_owned_or_else(data_type, DataType::from), - Array(atype) => self - .transform_array(atype)? - .map_owned_or_else(data_type, DataType::from), - Struct(stype) => self - .transform_struct(stype)? - .map_owned_or_else(data_type, DataType::from), - Map(mtype) => self - .transform_map(mtype)? - .map_owned_or_else(data_type, DataType::from), - Variant(stype) => self - .transform_variant(stype)? - .map_owned_or_else(data_type, |s| DataType::Variant(Box::new(s))), - }; - Some(result) + fn transform(&mut self, data_type: &'a DataType) -> Self::Output { + match data_type { + DataType::Primitive(ptype) => { + let child = self.transform_primitive(ptype); + map_owned_or_else(data_type, child, DataType::from) + } + DataType::Array(atype) => { + let child = self.transform_array(atype); + map_owned_or_else(data_type, child, DataType::from) + } + DataType::Struct(stype) => { + let child = self.transform_struct(stype); + map_owned_or_else(data_type, child, DataType::from) + } + DataType::Map(mtype) => { + let child = self.transform_map(mtype); + map_owned_or_else(data_type, child, DataType::from) + } + DataType::Variant(stype) => { + let child = self.transform_variant(stype); + map_owned_or_else(data_type, child, |s| DataType::Variant(Box::new(s))) + } + } } - /// Recursively transforms a struct field's data type. If the data type changes, update the - /// field to reference it. Otherwise, no-op. - fn recurse_into_struct_field( - &mut self, - field: &'a StructField, - ) -> Option> { - let result = self.transform(&field.data_type)?; - let f = |new_data_type| StructField { + /// Recursively transforms a struct field's data type (unary). + fn recurse_into_struct_field(&mut self, field: &'a StructField) -> Self::Output { + let child = self.transform(&field.data_type); + map_owned_or_else(field, child, |new_data_type| StructField { name: field.name.clone(), data_type: new_data_type, nullable: field.nullable, metadata: field.metadata.clone(), - }; - Some(result.map_owned_or_else(field, f)) + }) } - /// Recursively transforms a struct's fields. If one or more fields were changed or removed, - /// update the struct to reference all surviving fields. Otherwise, no-op. - fn recurse_into_struct(&mut self, stype: &'a StructType) -> Option> { - let transformed_children = stype.fields().map(|f| self.transform_struct_field(f)); - map_owned_children_or_else(stype, transformed_children, StructType::new_unchecked) + /// Recursively transforms a struct's fields (variadic). + fn recurse_into_struct(&mut self, stype: &'a StructType) -> Self::Output { + let children = stype.fields().map(|f| self.transform_struct_field(f)); + map_owned_children_or_else(stype, children, StructType::new_unchecked) } - /// Recursively transforms an array's element type. If the element type changes, update the - /// array to reference it. Otherwise, no-op. - fn recurse_into_array(&mut self, atype: &'a ArrayType) -> Option> { - let result = self.transform_array_element(&atype.element_type)?; - let f = |element_type| ArrayType { + /// Recursively transforms an array's element type (unary). + fn recurse_into_array(&mut self, atype: &'a ArrayType) -> Self::Output { + let child = self.transform_array_element(&atype.element_type); + map_owned_or_else(atype, child, |element_type| ArrayType { type_name: atype.type_name.clone(), element_type, contains_null: atype.contains_null, - }; - Some(result.map_owned_or_else(atype, f)) + }) } - /// Recursively transforms a map's key and value types. If either one changes, update the map to - /// reference them. If either one is removed, remove the map as well. Otherwise, no-op. - fn recurse_into_map(&mut self, mtype: &'a MapType) -> Option> { - let key_type = self.transform_map_key(&mtype.key_type)?; - let value_type = self.transform_map_value(&mtype.value_type)?; + /// Recursively transforms a map's key and value types (binary). + fn recurse_into_map(&mut self, mtype: &'a MapType) -> Self::Output { + let key_type = self.transform_map_key(&mtype.key_type); + let value_type = self.transform_map_value(&mtype.value_type); let f = |(key_type, value_type)| MapType { type_name: mtype.type_name.clone(), key_type, value_type, value_contains_null: mtype.value_contains_null, }; - Some((key_type, value_type).map_owned_or_else(mtype, f)) + map_owned_pair_or_else(mtype, key_type, value_type, f) } } @@ -174,42 +218,42 @@ impl SchemaDepthChecker { current_depth: 0, call_count: 0, }; - checker.transform(data_type); + let _ = checker.transform(data_type); (checker.max_depth_seen, checker.call_count) } - // Triggers the requested recursion only doing so would not exceed the depth limit. - fn depth_limited<'a, T: Clone + std::fmt::Debug>( + // Triggers the requested recursion only if doing so would not exceed the depth limit. + // If the next step would exceed the limit, short-circuit the traversal. + fn depth_limited<'a, T: ToOwned + std::fmt::Debug + ?Sized>( &mut self, - recurse: impl FnOnce(&mut Self, &'a T) -> Option>, + recurse: impl FnOnce(&mut Self, &'a T) -> Result<(), ()>, arg: &'a T, - ) -> Option> { + ) -> Result<(), ()> { self.call_count += 1; - if self.max_depth_seen < self.current_depth { - self.max_depth_seen = self.current_depth; - if self.depth_limit < self.current_depth { - tracing::warn!("Max schema depth {} exceeded by {arg:?}", self.depth_limit); - } - } - if self.max_depth_seen <= self.depth_limit { - self.current_depth += 1; - let _ = recurse(self, arg); - self.current_depth -= 1; + self.max_depth_seen = self.max_depth_seen.max(self.current_depth); + if self.current_depth > self.depth_limit { + tracing::warn!("Max schema depth {} exceeded by {arg:?}", self.depth_limit); + return Err(()); } - None + self.current_depth += 1; + let result = recurse(self, arg); + self.current_depth -= 1; + result } } impl<'a> SchemaTransform<'a> for SchemaDepthChecker { - fn transform_struct(&mut self, stype: &'a StructType) -> Option> { + transform_output_type!(|'a, T| Result<(), ()>); + + fn transform_struct(&mut self, stype: &'a StructType) -> Result<(), ()> { self.depth_limited(Self::recurse_into_struct, stype) } - fn transform_struct_field(&mut self, field: &'a StructField) -> Option> { + fn transform_struct_field(&mut self, field: &'a StructField) -> Result<(), ()> { self.depth_limited(Self::recurse_into_struct_field, field) } - fn transform_array(&mut self, atype: &'a ArrayType) -> Option> { + fn transform_array(&mut self, atype: &'a ArrayType) -> Result<(), ()> { self.depth_limited(Self::recurse_into_array, atype) } - fn transform_map(&mut self, mtype: &'a MapType) -> Option> { + fn transform_map(&mut self, mtype: &'a MapType) -> Result<(), ()> { self.depth_limited(Self::recurse_into_map, mtype) } } @@ -291,19 +335,16 @@ mod tests { let check_with_call_count = |depth_limit| SchemaDepthChecker::check_with_call_count(&schema, depth_limit); - // Hit depth limit at "a" but still have to look at "b" "c" "d" - assert_eq!(check_with_call_count(1), (2, 5)); - assert_eq!(check_with_call_count(2), (3, 6)); + // Short-circuit when the first over-limit node is encountered. + assert_eq!(check_with_call_count(1), (2, 3)); + assert_eq!(check_with_call_count(2), (3, 4)); - // Hit depth limit at "w" but still have to look at "x" "y" "z" - assert_eq!(check_with_call_count(3), (4, 10)); - assert_eq!(check_with_call_count(4), (5, 11)); + assert_eq!(check_with_call_count(3), (4, 5)); + assert_eq!(check_with_call_count(4), (5, 7)); - // Depth limit hit at "n" but still have to look at "m" - assert_eq!(check_with_call_count(5), (6, 15)); + assert_eq!(check_with_call_count(5), (6, 12)); - // Depth limit not hit until "u" - assert_eq!(check_with_call_count(6), (7, 28)); + assert_eq!(check_with_call_count(6), (7, 24)); // Depth limit not hit (full traversal required) assert_eq!(check_with_call_count(7), (7, 32)); diff --git a/kernel/src/unit_test_utils.rs b/kernel/src/unit_test_utils.rs new file mode 100644 index 0000000000..a955367235 --- /dev/null +++ b/kernel/src/unit_test_utils.rs @@ -0,0 +1,1051 @@ +use std::path::{Path, PathBuf}; +use std::sync::{Arc, Mutex}; + +use itertools::Itertools; +use serde::Serialize; +use tempfile::TempDir; +use test_utils::{ + copy_directory, delta_path_for_version, load_test_data, modify_add_file_partition_keys, + replace_array_row, AddFilePartitionKeyModify, +}; +use tracing::subscriber::DefaultGuard; +use tracing_subscriber::util::SubscriberInitExt as _; +use url::Url; + +use crate::actions::{get_all_actions_schema, Add, Cdc, CommitInfo, Metadata, Protocol, Remove}; +use crate::arrow::array::{ + new_empty_array, new_null_array, ArrayRef, Int64Array, MapArray, RecordBatch, StringArray, + StructArray, +}; +use crate::arrow::buffer::{OffsetBuffer, ScalarBuffer}; +use crate::arrow::compute::concat_batches; +use crate::arrow::datatypes::{DataType, Field, Schema as ArrowSchema}; +use crate::committer::FileSystemCommitter; +use crate::engine::arrow_conversion::{parquet_field_id_metadata, TryIntoArrow as _}; +use crate::engine::arrow_data::ArrowEngineData; +use crate::engine::sync::SyncEngine; +use crate::metrics::{MetricEvent, MetricsReporter, WithMetricsReporterLayer as _}; +use crate::object_store::local::LocalFileSystem; +use crate::object_store::memory::InMemory; +use crate::object_store::ObjectStoreExt as _; +use crate::parquet::arrow::PARQUET_FIELD_ID_META_KEY; +use crate::path::ParsedLogPath; +use crate::table_features::ColumnMappingMode; +use crate::transaction::create_table::create_table; +use crate::transaction::{CreateTable, Transaction, BASE_ADD_FILES_SCHEMA}; +use crate::{DeltaResult, Engine, EngineData, Error, FileMeta, Snapshot, SnapshotRef}; + +/// Parses `path` (a full URL string) into a [`ParsedLogPath`] with zero size, for building +/// synthetic log-file listings in tests. +pub(crate) fn create_log_path(path: &str) -> ParsedLogPath { + create_log_path_with_size(path, 0) +} + +/// [`create_log_path`] with an explicit file size. +pub(crate) fn create_log_path_with_size(path: &str, size: u64) -> ParsedLogPath { + ParsedLogPath::try_from(FileMeta { + location: Url::parse(path).expect("Invalid file URL"), + last_modified: 0, + size, + }) + .unwrap() + .unwrap() +} + +/// A metrics reporter that captures all events for test assertions. +#[derive(Debug, Default)] +pub(crate) struct CapturingReporter { + events: Mutex>, +} + +impl MetricsReporter for CapturingReporter { + fn report(&self, event: MetricEvent) { + self.events.lock().unwrap().push(event); + } +} + +impl CapturingReporter { + /// Returns a copy of all captured events. + pub(crate) fn events(&self) -> Vec { + self.events.lock().unwrap().clone() + } +} + +/// Kernel-internal twin of [`test_utils::install_thread_local_metrics_reporter`]. +/// +/// Internal tests need their own helper because the trait identity of `MetricsReporter` +/// differs across the test_utils <-> kernel path-dep boundary. Both helpers wrap +/// [`test_utils::ensure_metrics_compatible_global_subscriber`] + a thread-local +/// `set_default` and serve the same purpose: install a metrics-collecting subscriber +/// in a way that is robust against tracing callsite-cache poisoning. +pub(crate) fn install_thread_local_metrics_reporter( + reporter: Arc, +) -> DefaultGuard { + test_utils::ensure_metrics_compatible_global_subscriber(); + tracing_subscriber::registry() + .with_metrics_reporter_layer(reporter) + .set_default() +} + +#[derive(Serialize)] +pub(crate) enum Action { + #[serde(rename = "add")] + Add(Add), + #[serde(rename = "remove")] + Remove(Remove), + #[serde(rename = "cdc")] + Cdc(Cdc), + #[serde(rename = "metaData")] + Metadata(Metadata), + #[serde(rename = "protocol")] + Protocol(Protocol), + #[allow(unused)] + #[serde(rename = "commitInfo")] + CommitInfo(CommitInfo), +} + +use crate::schema::{ + schema, schema_ref, ArrayType, ColumnMetadataKey, DataType as KernelDataType, MapType, + MetadataValue, SchemaRef, StructField, StructType, +}; +#[cfg(feature = "geo-type-in-dev")] +use crate::schema::{EdgeInterpolationAlgorithm, GeographyType, GeometryType, PrimitiveType}; + +/// A mock table that writes commits to a local temporary delta log. This can be used to +/// construct a delta log used for testing. +pub(crate) struct LocalMockTable { + commit_num: u64, + store: Arc, + dir: TempDir, +} + +impl LocalMockTable { + pub(crate) fn new() -> Self { + let dir = tempfile::tempdir().unwrap(); + let store = Arc::new(LocalFileSystem::new_with_prefix(dir.path()).unwrap()); + Self { + commit_num: 0, + store, + dir, + } + } + /// Writes all `actions` to a new commit in the log + pub(crate) async fn commit(&mut self, actions: impl IntoIterator) { + let data = actions + .into_iter() + .map(|action| serde_json::to_string(&action).unwrap()) + .join("\n"); + + let path = delta_path_for_version(self.commit_num, "json"); + self.commit_num += 1; + + self.store + .put(&path, data.into()) + .await + .expect("put log file in store"); + } + + /// Get the path to the root of the table. + pub(crate) fn table_root(&self) -> &Path { + self.dir.path() + } +} + +/// Try to convert an `EngineData` into a `RecordBatch`. Panics if not using `ArrowEngineData` +/// from the default module +fn into_record_batch(engine_data: Box) -> RecordBatch { + ArrowEngineData::try_from_engine_data(engine_data) + .unwrap() + .into() +} + +/// Checks that two `EngineData` objects are equal by converting them to `RecordBatch` and +/// comparing +pub(crate) fn assert_batch_matches(actual: Box, expected: Box) { + assert_eq!(into_record_batch(actual), into_record_batch(expected)); +} + +/// Helper for building valid add-file batches. +/// +/// Tests can generate add-file metadata without writing a Parquet file using this helper. All +/// required fields are populated. When `all_nullable` is true, every field in the batch +/// schema is nullable. +pub(crate) fn create_valid_add_file_batch(all_nullable: bool) -> RecordBatch { + let schema = if all_nullable { + let fields = BASE_ADD_FILES_SCHEMA + .fields() + .map(|f| StructField::nullable(f.name().clone(), f.data_type().clone())); + StructType::try_new(fields).expect("nullable add-file schema") + } else { + BASE_ADD_FILES_SCHEMA.as_ref().clone() + }; + let arrow_schema: ArrowSchema = (&schema).try_into_arrow().expect("arrow schema"); + let columns: Vec = arrow_schema + .fields() + .iter() + .map(|field| match field.name().as_str() { + "path" => Arc::new(StringArray::from(vec!["dummy"])) as ArrayRef, + "size" | "modificationTime" => Arc::new(Int64Array::from(vec![1i64])) as ArrayRef, + "partitionValues" => { + let DataType::Map(entries_field, sorted) = field.data_type() else { + panic!("partitionValues must be a map type"); + }; + let entries = new_empty_array(entries_field.data_type()) + .as_any() + .downcast_ref::() + .expect("map entries struct") + .clone(); + // One row, empty partition map. + let offsets = OffsetBuffer::new(ScalarBuffer::from(vec![0i32, 0])); + Arc::new( + MapArray::try_new(entries_field.clone(), offsets, entries, None, *sorted) + .expect("empty map array"), + ) + } + // Non-mandatory fields (e.g. `stats`) are left null. + _ => new_null_array(field.data_type(), 1), + }) + .collect(); + RecordBatch::try_new(Arc::new(arrow_schema), columns).expect("valid add-file batch") +} + +/// Builds one valid add-file row with a fully nullable schema. +pub(crate) fn nullable_add_file() -> RecordBatch { + create_valid_add_file_batch(true /* all_nullable */) +} + +/// Builds `row_count` valid add-file rows with a fully nullable schema. +pub(crate) fn nullable_add_files(row_count: usize) -> RecordBatch { + let batch = nullable_add_file(); + concat_batches(&batch.schema(), &vec![batch; row_count]) + .expect("failed to concatenate rows into a multi-row add-file batch") +} + +pub(crate) fn replace_column(batch: &RecordBatch, field: &str, column: ArrayRef) -> RecordBatch { + let schema = batch.schema(); + let index = schema.index_of(field).expect("field in schema"); + let mut columns = batch.columns().to_vec(); + columns[index] = column; + RecordBatch::try_new(schema, columns).expect("failed to rebuild batch after replacing a column") +} + +pub(crate) fn set_field_as_null(batch: &RecordBatch, field: &str, row: usize) -> RecordBatch { + let schema = batch.schema(); + let index = schema.index_of(field).expect("field in schema"); + let mut columns = batch.columns().to_vec(); + let null = new_null_array(schema.field(index).data_type(), 1); + columns[index] = replace_array_row(&columns[index], null, row); + RecordBatch::try_new(schema, columns) + .expect("failed to rebuild batch after replacing a field value with null") +} + +/// Returns nullable add-file rows with `partitionValues` replaced by `partition_values`. +pub(crate) fn add_files_with_partition_values( + partition_values: &[&[(&str, Option<&str>)]], +) -> RecordBatch { + let batches: Vec<_> = partition_values + .iter() + .map(|entries| { + let modifications: Vec<_> = entries + .iter() + .map(|(key, value)| AddFilePartitionKeyModify::Insert { key, value: *value }) + .collect(); + modify_add_file_partition_keys(nullable_add_file(), &modifications) + }) + .collect(); + concat_batches(&batches[0].schema(), &batches) + .expect("failed to concatenate rows with partition values") +} + +pub(crate) fn string_array_to_engine_data(string_array: StringArray) -> Box { + let string_field = Arc::new(Field::new("a", DataType::Utf8, true)); + let schema = Arc::new(ArrowSchema::new(vec![string_field])); + let batch = RecordBatch::try_new(schema, vec![Arc::new(string_array)]) + .expect("Can't convert to record batch"); + Box::new(ArrowEngineData::new(batch)) +} + +pub(crate) fn parse_json_batch(json_strings: StringArray) -> Box { + let engine = SyncEngine::new(); + let json_handler = engine.json_handler(); + let output_schema = get_all_actions_schema().clone(); + json_handler + .parse_json(string_array_to_engine_data(json_strings), output_schema) + .unwrap() +} + +pub(crate) fn action_batch() -> Box { + let json_strings: StringArray = vec![ + r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet","partitionValues":{},"size":635,"modificationTime":1677811178336,"dataChange":true,"stats":"{\"numRecords\":10,\"minValues\":{\"value\":0},\"maxValues\":{\"value\":9},\"nullCount\":{\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1677811178336000","MIN_INSERTION_TIME":"1677811178336000","MAX_INSERTION_TIME":"1677811178336000","OPTIMIZE_TARGET_SIZE":"268435456"}}}"#, + r#"{"remove":{"path":"part-00003-f525f459-34f9-46f5-82d6-d42121d883fd.c000.snappy.parquet","deletionTimestamp":1670892998135,"dataChange":true,"partitionValues":{"c1":"4","c2":"c"},"size":452}}"#, + r#"{"commitInfo":{"timestamp":1677811178585,"operation":"WRITE","operationParameters":{"mode":"ErrorIfExists","partitionBy":"[]"},"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"10","numOutputBytes":"635"},"engineInfo":"Databricks-Runtime/","txnId":"a6a94671-55ef-450e-9546-b8465b9147de"}}"#, + r#"{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors"]}}"#, + r#"{"metaData":{"id":"testId","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"value\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.columnMapping.mode":"none", "delta.enableChangeDataFeed":"true"},"createdTime":1677811175819}}"#, + r#"{"cdc":{"path":"_change_data/age=21/cdc-00000-93f7fceb-281a-446a-b221-07b88132d203.c000.snappy.parquet","partitionValues":{"age":"21"},"size":1033,"dataChange":false}}"#, + r#"{"sidecar":{"path":"016ae953-37a9-438e-8683-9a9a4a79a395.parquet","sizeInBytes":9268,"modificationTime":1714496113961,"tags":{"tag_foo":"tag_bar"}}}"#, + r#"{"txn":{"appId":"myApp","version": 3}}"#, + r#"{"checkpointMetadata":{"version":2, "tags":{"tag_foo":"tag_bar"}}}"#, + ] + .into(); + parse_json_batch(json_strings) +} + +// TODO: allow tests to pass in context (issue#1133) +#[track_caller] +pub(crate) fn assert_result_error_with_message(res: Result, message: &str) { + match res { + Ok(_) => panic!("Expected error with message {message}, but got Ok result"), + Err(error) => { + let error_str = error.to_string(); + assert!( + error_str.contains(message), + "Error message does not contain the expected message.\nExpected message:\t{message}\nActual message:\t\t{error_str}" + ); + } + } +} + +/// Asserts the 2x2 matrix of (schema_has_feature, protocol_supports_feature) outcomes +/// for schema-level feature validators. The expected pattern is: +/// - schema + protocol => Ok +/// - no schema + no protocol => Ok +/// - no schema + protocol => Ok +/// - schema + no protocol => Err (orphaned schema presence) +/// +/// Additional error schemas (e.g. nested) are also tested against `protocol_without`. +#[track_caller] +pub(crate) fn assert_schema_feature_validation( + schema_with: &StructType, + schema_without: &StructType, + protocol_with: &Protocol, + protocol_without: &Protocol, + extra_err_schemas: &[&StructType], + err_msg: &str, +) { + make_test_tc(schema_with.clone(), protocol_with.clone(), []) + .expect("feature present + supported"); + make_test_tc(schema_without.clone(), protocol_without.clone(), []) + .expect("feature absent + unsupported"); + make_test_tc(schema_without.clone(), protocol_with.clone(), []) + .expect("feature absent + supported"); + assert_result_error_with_message( + make_test_tc(schema_with.clone(), protocol_without.clone(), []), + err_msg, + ); + for schema in extra_err_schemas { + assert_result_error_with_message( + make_test_tc((*schema).clone(), protocol_without.clone(), []), + err_msg, + ); + } +} + +/// Creates a [`TableConfiguration`] from a schema, protocol, and table properties. +/// Useful for testing validators that need a TC. +pub(crate) fn make_test_tc( + schema: StructType, + protocol: Protocol, + props: impl IntoIterator, +) -> crate::DeltaResult { + let schema = std::sync::Arc::new(schema); + let metadata = + Metadata::try_new(None, None, schema, vec![], 0, props.into_iter().collect()).unwrap(); + let table_root = Url::try_from("file:///").unwrap(); + crate::table_configuration::TableConfiguration::try_new(metadata, protocol, table_root, 0) +} + +// ==================== Test schema helpers ==================== +// +// Reusable test schemas +// Each variant exists with and without column mapping metadata. + +/// Builds a nullable [`StructField`] carrying column-mapping id + physical name metadata. +fn cm_field( + name: &str, + id: i64, + physical_name: &str, + ty: impl Into, +) -> StructField { + StructField::nullable(name, ty).with_metadata([ + ( + ColumnMetadataKey::ColumnMappingId.as_ref(), + MetadataValue::Number(id), + ), + ( + ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), + MetadataValue::String(physical_name.into()), + ), + ]) +} + +/// Shared fixture for nested field-id propagation tests. +pub(crate) struct NestedFieldIdFixture { + pub(crate) kernel_schema: StructType, + pub(crate) input_arrow_data: StructArray, + pub(crate) expected_arrow_schema: ArrowSchema, +} + +/// Recursively collect `(field_name, metadata_value)` pairs for the given metadata key +/// across all (nested) Arrow fields in `schema`. +pub(crate) fn collect_arrow_field_metadata( + schema: &ArrowSchema, + metadata_key: &str, +) -> Vec<(String, String)> { + fn collect_from_fields( + fields: &[Arc], + metadata_key: &str, + out: &mut Vec<(String, String)>, + ) { + for field in fields { + collect_from_field(field, metadata_key, out); + } + } + + fn collect_from_field(field: &Field, metadata_key: &str, out: &mut Vec<(String, String)>) { + if let Some(value) = field.metadata().get(metadata_key) { + out.push((field.name().clone(), value.clone())); + } + + match field.data_type() { + DataType::Struct(fields) => collect_from_fields(fields, metadata_key, out), + DataType::List(entry) | DataType::Map(entry, _) => { + collect_from_field(entry, metadata_key, out) + } + _ => {} + } + } + + let mut out = Vec::new(); + collect_from_fields(schema.fields(), metadata_key, &mut out); + out +} + +/// Build the kernel schema for `array_in_map: map>` with caller-provided +/// top-level field metadata. +pub(crate) fn array_in_map_kernel_schema( + metadata: impl IntoIterator, +) -> StructType { + let array_in_map = StructField::nullable( + "array_in_map", + MapType::new( + KernelDataType::INTEGER, + ArrayType::new(KernelDataType::INTEGER, true), + true, + ), + ) + .with_metadata(metadata); + StructType::try_new(vec![array_in_map]).unwrap() +} + +/// Build an [`array_in_map_kernel_schema`] with `parquet.field.id` on the top-level field +/// and a nested-ids JSON map (key/value/element) under `nested_ids_meta_key`. +pub(crate) fn array_in_map_with_field_ids(nested_ids_meta_key: &str) -> StructType { + let nested_ids = MetadataValue::Other(test_utils::nested_ids_json(&[ + ("array_in_map.key", 100), + ("array_in_map.value", 101), + ("array_in_map.value.element", 102), + ])); + array_in_map_kernel_schema([ + ( + ColumnMetadataKey::ParquetFieldId.as_ref().to_string(), + MetadataValue::from(1i64), + ), + (nested_ids_meta_key.to_string(), nested_ids), + ]) +} + +/// Build an empty Arrow `StructArray` matching [`array_in_map_kernel_schema`] with no field-id +/// metadata. +pub(crate) fn array_in_map_arrow_data_without_field_ids() -> StructArray { + let kernel_schema = array_in_map_kernel_schema(std::iter::empty::<(String, MetadataValue)>()); + let arrow_schema: ArrowSchema = (&kernel_schema).try_into_arrow().unwrap(); + let batch = RecordBatch::new_empty(Arc::new(arrow_schema)); + StructArray::try_new( + batch.schema().fields.clone(), + batch.columns().to_vec(), + None, + ) + .unwrap() +} + +/// Build a [`NestedFieldIdFixture`] covering Array+Map nested-id propagation through a +/// Struct boundary. +/// +/// ## 1. Kernel schema +/// +/// Two [`StructField`]s `top` and `inner` carry `parquet.field.id` (rewritten to +/// `PARQUET:field_id` in the Arrow output) plus a `` JSON map rooted +/// at each field's name. Each carries an array inside a map. +/// +/// ```json +/// { +/// "type": "struct", +/// "fields": [{ +/// "name": "top", +/// "type": { +/// "type": "map", +/// "keyType": {"type": "array", "elementType": "integer"}, +/// "valueType": {"type": "struct", "fields": [{ +/// "name": "inner", +/// "type": {"type": "map", "keyType": "integer", +/// "valueType": {"type": "array", "elementType": "integer"}}, +/// "metadata": { +/// "parquet.field.id": 2, +/// "": { +/// "inner.key": 200, "inner.value": 201, "inner.value.element": 202 +/// } +/// } +/// }]} +/// }, +/// "metadata": { +/// "parquet.field.id": 1, +/// "": { +/// "top.key": 100, "top.key.element": 101, "top.value": 102 +/// } +/// } +/// }] +/// } +/// ``` +/// +/// ## 2. Input Arrow schema +/// +/// Same shape as kernel schema with no metadata anywhere, except a *stale* +/// `PARQUET:field_id=999` on the synthesized `top.key.element` field. +/// +/// ## 3. Expected output Arrow schema +/// +/// What `try_into_arrow(kernel schema)` and +/// `apply_schema(input arrow schema, kernel schema)` should both produce: +/// - `top` and `inner` carry `PARQUET:field_id` (rewritten from `parquet.field.id`). +/// - Synthesized list/map `key`/`value`/`element` fields each carry `PARQUET:field_id` pulled from +/// the corresponding nested-ids JSON entry. `top.key.element` has `101` (kernel's), not the stale +/// `999` from the input. +/// +/// ```json +/// { +/// "fields": [{ +/// "name": "top", "type": "map", +/// "metadata": {"PARQUET:field_id": "1"}, +/// "entries": { "name": "key_value", "type": "struct", "fields": [ +/// { +/// "name": "key", "type": "list", +/// "metadata": {"PARQUET:field_id": "100"}, +/// "element": { +/// "name": "element", "type": "int32", +/// "metadata": {"PARQUET:field_id": "101"} +/// } +/// }, +/// { +/// "name": "value", "type": "struct", +/// "metadata": {"PARQUET:field_id": "102"}, +/// "fields": [{ +/// "name": "inner", "type": "map", +/// "metadata": {"PARQUET:field_id": "2"}, +/// "entries": { "name": "key_value", "type": "struct", "fields": [ +/// { +/// "name": "key", "type": "int32", +/// "metadata": {"PARQUET:field_id": "200"} +/// }, +/// { +/// "name": "value", "type": "list", +/// "metadata": {"PARQUET:field_id": "201"}, +/// "element": { +/// "name": "element", "type": "int32", +/// "metadata": {"PARQUET:field_id": "202"} +/// } +/// } +/// ]} +/// }] +/// } +/// ]} +/// }] +/// } +/// ``` +pub(crate) fn complex_nested_with_field_ids(nested_ids_meta_key: &str) -> NestedFieldIdFixture { + NestedFieldIdFixture { + kernel_schema: build_complex_nested_kernel_schema(nested_ids_meta_key), + input_arrow_data: build_arrow_input_with_stale_element_id(), + expected_arrow_schema: expected_complex_nested_arrow_schema(), + } +} + +/// Build the input Arrow data for [`complex_nested_with_field_ids`] by +/// striping the metadata from [`build_complex_nested_kernel_schema`], and +/// add one stale `PARQUET:field_id` to the `top.key.element` field. +fn build_arrow_input_with_stale_element_id() -> StructArray { + // Get the no-meta Arrow shape from the kernel schema. + let plain_inner = StructField::nullable("inner", complex_nested_inner_map_type()); + let plain_top = StructField::nullable( + "top", + complex_nested_outer_map_type(StructType::try_new(vec![plain_inner]).unwrap()), + ); + let plain_kernel_schema = StructType::try_new(vec![plain_top]).unwrap(); + let plain_arrow_schema: ArrowSchema = (&plain_kernel_schema).try_into_arrow().unwrap(); + + // Add stale `PARQUET:field_id` to the `top.key.element` field. + let top = plain_arrow_schema.field(0); + let DataType::Map(entries, sorted) = top.data_type() else { + unreachable!("top is a Map by construction"); + }; + let DataType::Struct(entries_fields) = entries.data_type() else { + unreachable!("map entries is a Struct by construction"); + }; + let outer_key = &entries_fields[0]; + let DataType::List(outer_element) = outer_key.data_type() else { + unreachable!("outer map key is a List by construction"); + }; + let stale_element = outer_element + .as_ref() + .clone() + .with_metadata([(PARQUET_FIELD_ID_META_KEY.to_string(), "999".to_string())].into()); + let new_outer_key = Field::new( + outer_key.name(), + DataType::List(Arc::new(stale_element)), + outer_key.is_nullable(), + ); + let new_entries = Field::new( + entries.name(), + DataType::Struct(vec![new_outer_key, entries_fields[1].as_ref().clone()].into()), + entries.is_nullable(), + ); + let new_top = Field::new( + top.name(), + DataType::Map(Arc::new(new_entries), *sorted), + top.is_nullable(), + ); + let arrow_input_schema = ArrowSchema::new(vec![new_top]); + let batch = RecordBatch::new_empty(Arc::new(arrow_input_schema)); + StructArray::try_new( + batch.schema().fields.clone(), + batch.columns().to_vec(), + None, + ) + .unwrap() +} + +fn complex_nested_inner_map_type() -> KernelDataType { + KernelDataType::from(MapType::new( + KernelDataType::INTEGER, + ArrayType::new(KernelDataType::INTEGER, true), + true, + )) +} + +fn complex_nested_outer_map_type(struct_value: StructType) -> KernelDataType { + KernelDataType::from(MapType::new( + ArrayType::new(KernelDataType::INTEGER, true), + struct_value, + true, + )) +} + +/// Build the kernel schema described by [`complex_nested_with_field_ids`]. +pub(crate) fn build_complex_nested_kernel_schema(nested_ids_meta_key: &str) -> StructType { + let top_nested_ids = test_utils::nested_ids_json(&[ + ("top.key", 100), + ("top.key.element", 101), + ("top.value", 102), + ]); + let inner_nested_ids = test_utils::nested_ids_json(&[ + ("inner.key", 200), + ("inner.value", 201), + ("inner.value.element", 202), + ]); + // Each `StructField` carries `parquet.field.id` plus the nested-ids JSON. + let inner_field = StructField::nullable("inner", complex_nested_inner_map_type()) + .with_metadata([ + ( + ColumnMetadataKey::ParquetFieldId.as_ref().to_string(), + MetadataValue::from(2i64), + ), + ( + nested_ids_meta_key.to_string(), + MetadataValue::Other(inner_nested_ids), + ), + ]); + let top_field = StructField::nullable( + "top", + complex_nested_outer_map_type(StructType::try_new(vec![inner_field]).unwrap()), + ) + .with_metadata([ + ( + ColumnMetadataKey::ParquetFieldId.as_ref().to_string(), + MetadataValue::from(1i64), + ), + ( + nested_ids_meta_key.to_string(), + MetadataValue::Other(top_nested_ids), + ), + ]); + StructType::try_new(vec![top_field]).unwrap() +} + +/// Build the expected output Arrow schema for [`complex_nested_with_field_ids`]. +fn expected_complex_nested_arrow_schema() -> ArrowSchema { + // top.value.inner.value.element: int (PARQUET:field_id=202). + let inner_list_element = Field::new("element", DataType::Int32, true) + .with_metadata(parquet_field_id_metadata(Some(202))); + // top.value.inner.value: list (PARQUET:field_id=201). + let inner_value = Field::new("value", DataType::List(Arc::new(inner_list_element)), true) + .with_metadata(parquet_field_id_metadata(Some(201))); + // top.value.inner.key: int (PARQUET:field_id=200). + let inner_key = Field::new("key", DataType::Int32, false) + .with_metadata(parquet_field_id_metadata(Some(200))); + // top.value.inner.key_value: synthesized map-entries struct (no field id). + let inner_entries = Field::new( + "key_value", + DataType::Struct(vec![inner_key, inner_value].into()), + false, + ); + // top.value.inner: map> (PARQUET:field_id=2). + let inner_field = Field::new("inner", DataType::Map(Arc::new(inner_entries), false), true) + .with_metadata(parquet_field_id_metadata(Some(2))); + // top.value: struct (PARQUET:field_id=102). + let struct_value_field = Field::new("value", DataType::Struct(vec![inner_field].into()), true) + .with_metadata(parquet_field_id_metadata(Some(102))); + // top.key.element: int (PARQUET:field_id=101). + let outer_key_element = Field::new("element", DataType::Int32, true) + .with_metadata(parquet_field_id_metadata(Some(101))); + // top.key: list (PARQUET:field_id=100). + let outer_key = Field::new("key", DataType::List(Arc::new(outer_key_element)), false) + .with_metadata(parquet_field_id_metadata(Some(100))); + // top.key_value: synthesized map-entries struct (no field id). + let outer_entries = Field::new( + "key_value", + DataType::Struct(vec![outer_key, struct_value_field].into()), + false, + ); + // top: map, struct<...>> (PARQUET:field_id=1). + let top_field = Field::new("top", DataType::Map(Arc::new(outer_entries), false), true) + .with_metadata(parquet_field_id_metadata(Some(1))); + ArrowSchema::new(vec![top_field]) +} + +/// Flat schema: `[id: long, name: string]` +pub(crate) fn test_schema_flat() -> SchemaRef { + schema_ref! { + nullable "id": LONG, + nullable "name": STRING, + } +} + +/// Flat schema with column mapping metadata. +pub(crate) fn test_schema_flat_with_column_mapping() -> SchemaRef { + schema_ref! { + (cm_field("id", 1, "phys_id", KernelDataType::LONG)), + (cm_field("name", 2, "phys_name", KernelDataType::STRING)), + } +} + +/// Nested struct schema with array and map inside the struct +pub(crate) fn test_schema_nested() -> SchemaRef { + schema_ref! { + nullable "id": LONG, + nullable "info": { + nullable "name": STRING, + nullable "age": INTEGER, + nullable "tags": { STRING => nullable STRING }, + nullable "scores": [ nullable INTEGER ], + }, + } +} + +/// Nested struct schema with column mapping metadata. +pub(crate) fn test_schema_nested_with_column_mapping() -> SchemaRef { + schema_ref! { + (cm_field("id", 1, "phys_id", KernelDataType::LONG)), + (cm_field("info", 2, "phys_info", schema! { + (cm_field("name", 3, "phys_name", KernelDataType::STRING)), + (cm_field("age", 4, "phys_age", KernelDataType::INTEGER)), + (cm_field("tags", 5, "phys_tags", + MapType::new(KernelDataType::STRING, KernelDataType::STRING, true))), + (cm_field("scores", 6, "phys_scores", + ArrayType::new(KernelDataType::INTEGER, true))), + })), + } +} + +/// Schema with a map +pub(crate) fn test_schema_with_map() -> SchemaRef { + schema_ref! { + nullable "id": LONG, + nullable "entries": { STRING => nullable { + nullable "key": STRING, + nullable "value": INTEGER, + } }, + nullable "name": STRING, + } +} + +/// Schema with a map and column mapping metadata. +pub(crate) fn test_schema_with_map_and_column_mapping() -> SchemaRef { + let value_struct = schema! { + (cm_field("key", 4, "phys_key", KernelDataType::STRING)), + (cm_field("value", 5, "phys_value", KernelDataType::INTEGER)), + }; + schema_ref! { + (cm_field("id", 1, "phys_id", KernelDataType::LONG)), + (cm_field("entries", 2, "phys_entries", + MapType::new(KernelDataType::STRING, value_struct, true))), + (cm_field("name", 3, "phys_name", KernelDataType::STRING)), + } +} + +/// Schema with an array +pub(crate) fn test_schema_with_array() -> SchemaRef { + schema_ref! { + nullable "id": LONG, + nullable "items": [ nullable { + nullable "label": STRING, + nullable "count": INTEGER, + } ], + nullable "name": STRING, + } +} + +/// Schema with an array and column mapping metadata. +pub(crate) fn test_schema_with_array_and_column_mapping() -> SchemaRef { + let item_struct = schema! { + (cm_field("label", 4, "phys_label", KernelDataType::STRING)), + (cm_field("count", 5, "phys_count", KernelDataType::INTEGER)), + }; + schema_ref! { + (cm_field("id", 1, "phys_id", KernelDataType::LONG)), + (cm_field("items", 2, "phys_items", ArrayType::new(item_struct, true))), + (cm_field("name", 3, "phys_name", KernelDataType::STRING)), + } +} + +/// Deeply nested schema: struct -> array -> struct -> map(value) -> struct. +/// +/// The leaf struct field is intentionally **not** annotated with column mapping metadata, +/// so this schema can be used to test error paths when column mapping is enabled. +pub(crate) fn test_deep_nested_schema_missing_leaf_cm() -> StructType { + let map_type = MapType::new( + KernelDataType::STRING, + schema! { not_null "leaf": INTEGER }, + true, + ); + let array_type = ArrayType::new( + schema! { (cm_field("mid_field", 2, "phys_mid_field", map_type)) }, + true, + ); + schema! { + (cm_field("top", 1, "phys_top", array_type)), + } +} + +/// Build a create-table transaction with the given schema and column mapping mode. +/// Returns the engine and uncommitted transaction. +pub(crate) fn setup_column_mapping_txn( + schema: SchemaRef, + mode: ColumnMappingMode, +) -> DeltaResult<(Arc, Transaction)> { + let mode_str = match mode { + ColumnMappingMode::Name => "name", + ColumnMappingMode::Id => "id", + ColumnMappingMode::None => "none", + }; + let store = Arc::new(InMemory::new()); + let engine: Arc = Arc::new(SyncEngine::new_with_store(store)); + + let txn = create_table("memory:///test_table", schema, "DefaultEngine") + .with_table_properties([("delta.columnMapping.mode", mode_str)]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))?; + Ok((engine, txn)) +} + +/// Validate that a physical schema matches the logical schema's column mapping metadata. +/// For Name/Id modes, checks physicalName, columnMapping.id, and parquet.field.id on +/// each field. For None mode, only checks field names match. +pub(crate) fn validate_physical_schema_column_mapping( + logical_schema: &StructType, + physical_schema: &StructType, + mode: ColumnMappingMode, +) { + assert_eq!( + physical_schema.fields().count(), + logical_schema.fields().count() + ); + + // Collect expected (physical_name, field_id) from logical schema + let expected: Vec<_> = logical_schema + .fields() + .map(|f| { + let physical_name = + match f.get_config_value(&ColumnMetadataKey::ColumnMappingPhysicalName) { + Some(MetadataValue::String(name)) => name.clone(), + _ if mode == ColumnMappingMode::None => f.name().to_string(), + _ => panic!("Logical field '{}' missing physicalName metadata", f.name()), + }; + let field_id = match f.get_config_value(&ColumnMetadataKey::ColumnMappingId) { + Some(MetadataValue::Number(id)) => *id, + _ if mode == ColumnMappingMode::None => -1, + _ => panic!( + "Logical field '{}' missing columnMapping.id metadata", + f.name() + ), + }; + (physical_name, field_id) + }) + .collect(); + + // Validate each physical field against expected values + for (physical_field, (expected_name, expected_id)) in + physical_schema.fields().zip(expected.iter()) + { + assert_eq!( + physical_field.name(), + expected_name, + "Physical field name mismatch" + ); + + if mode == ColumnMappingMode::None { + continue; + } + + assert_eq!( + physical_field.get_config_value(&ColumnMetadataKey::ColumnMappingPhysicalName), + Some(&MetadataValue::String(expected_name.clone())), + "columnMapping.physicalName mismatch for '{}'", + physical_field.name() + ); + + assert_eq!( + physical_field.get_config_value(&ColumnMetadataKey::ColumnMappingId), + Some(&MetadataValue::Number(*expected_id)), + "columnMapping.id mismatch for '{}'", + physical_field.name() + ); + + assert_eq!( + physical_field.get_config_value(&ColumnMetadataKey::ParquetFieldId), + Some(&MetadataValue::Number(*expected_id)), + "parquet.field.id mismatch for '{}'", + physical_field.name() + ); + } +} + +fn resolve_test_table_path(table_name: &str) -> DeltaResult<(PathBuf, Option)> { + match load_test_data("tests/data", table_name) { + Ok(test_dir) => { + let test_path = test_dir.path().join(table_name); + Ok((test_path, Some(test_dir))) + } + Err(_) => { + let path = PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("tests/data") + .join(table_name); + let path = std::fs::canonicalize(path) + .map_err(|e| Error::generic(format!("Failed to canonicalize path: {e}")))?; + Ok((path, None)) + } + } +} + +/// Copies a test-table fixture into a writable temporary directory. +pub(crate) fn copy_test_table(table_name: &str) -> DeltaResult<(Url, TempDir)> { + let (source, _source_tempdir) = resolve_test_table_path(table_name)?; + let tempdir = tempfile::tempdir()?; + let table_path = tempdir.path().join(table_name); + copy_directory(&source, &table_path) + .map_err(|e| Error::generic(format!("Failed to copy test table: {e}")))?; + let url = Url::from_directory_path(&table_path) + .map_err(|_| Error::generic("Failed to create URL from path"))?; + Ok((url, tempdir)) +} + +/// Load a test table from tests/data directory. +/// Tries compressed (tar.zst) first, falls back to extracted. +/// Returns (engine, snapshot, optional tempdir). The TempDir must be kept alive +/// for the duration of the test to prevent premature cleanup of extracted files. +pub(crate) fn load_test_table( + table_name: &str, +) -> DeltaResult<(Arc, SnapshotRef, Option)> { + let (path, tempdir) = resolve_test_table_path(table_name)?; + + let url = Url::from_directory_path(&path) + .map_err(|_| Error::generic("Failed to create URL from path"))?; + + let engine = Arc::new(SyncEngine::new()); + let snapshot = Snapshot::builder_for(url).build(engine.as_ref())?; + Ok((engine, snapshot, tempdir)) +} + +pub(crate) mod column_mapping_physical_name_dedup_fixtures { + use crate::schema::{ + schema, ArrayType, ColumnMetadataKey, DataType, MapType, MetadataValue, StructField, + StructType, + }; + + /// Two fields with the same physical name at different physical paths should be accepted. + pub(crate) fn same_phy_name_different_paths() -> StructType { + let nested = StructType::new_unchecked([cm_field("id", 3, "id", DataType::INTEGER)]); + StructType::new_unchecked([ + cm_field("id", 1, "id", DataType::INTEGER), + cm_field("nested", 2, "nested", nested), + ]) + } + + /// Two nested fields with same physical path should be rejected. + pub(crate) fn deeply_nested_repeat_physical_paths() -> StructType { + let inner = StructType::new_unchecked([ + cm_field("a", 2, "x", DataType::INTEGER), + cm_field("b", 3, "x", DataType::INTEGER), + ]); + let arr_of_struct = ArrayType::new(inner, true); + let map_to_arr = MapType::new(DataType::STRING, arr_of_struct, true); + StructType::new_unchecked([cm_field("outer", 1, "outer", map_to_arr)]) + } + + /// Full logical paths of the two colliding fields in + /// [`deeply_nested_repeat_physical_paths`], in the order the walker visits them. + pub(crate) fn deeply_nested_collider_paths() -> (&'static str, &'static str) { + ( + "outer.``.``.a", + "outer.``.``.b", + ) + } + + /// Two collision sites in the same schema: + /// - **shallower** (visited first by DFS): top-level siblings `a` and `b`, both have physical + /// name "p". + /// - **deeper** (never reached): inside `nested` struct, siblings `x` and `y`, both have + /// physical name "q". + /// + /// Dedup must error at the shallower site and never report the deeper one. + pub(crate) fn multiple_physical_name_collisions() -> StructType { + schema! { + (cm_field("a", 1, "p", schema! { (cm_field("aa", 6, "aa", DataType::INTEGER)) })), + (cm_field("b", 2, "p", schema! { (cm_field("bb", 7, "bb", DataType::INTEGER)) })), + (cm_field("nested", 3, "nested", schema! { + (cm_field("x", 4, "q", DataType::INTEGER)), + (cm_field("y", 5, "q", DataType::INTEGER)), + })), + } + } + + fn cm_field(name: &str, id: i64, phys: &str, ty: impl Into) -> StructField { + StructField::new(name, ty, true).with_metadata([ + ( + ColumnMetadataKey::ColumnMappingId.as_ref(), + MetadataValue::Number(id), + ), + ( + ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), + MetadataValue::String(phys.to_string()), + ), + ]) + } +} + +#[cfg(feature = "geo-type-in-dev")] +pub(crate) fn geometry_type(crs: &str) -> KernelDataType { + PrimitiveType::Geometry(Box::new(GeometryType::try_new(crs).unwrap())).into() +} + +#[cfg(feature = "geo-type-in-dev")] +pub(crate) fn geography_type(crs: &str, algorithm: EdgeInterpolationAlgorithm) -> KernelDataType { + PrimitiveType::Geography(Box::new(GeographyType::try_new(crs, algorithm).unwrap())).into() +} diff --git a/kernel/src/utils.rs b/kernel/src/utils.rs index fb8c9d76bb..558c57e9f4 100644 --- a/kernel/src/utils.rs +++ b/kernel/src/utils.rs @@ -4,10 +4,10 @@ use std::ops::Deref; use std::path::PathBuf; use std::time::{Duration, SystemTime, UNIX_EPOCH}; +use delta_kernel_derive::internal_api; use url::Url; use crate::{DeltaResult, Error}; -use delta_kernel_derive::internal_api; /// convenient way to return an error if a condition isn't true macro_rules! require { @@ -20,6 +20,41 @@ macro_rules! require { pub(crate) use require; +/// Dual of the `FromIterator` trait, similar to how `Into` is the dual of `From`. It is +/// automatically implemented for any iterable whose items collect into `T`, and can drastically +/// simplify type bounds. For example, `CollectInto` allows to write this: +/// +/// ``` +/// # use delta_kernel::CollectInto; +/// # struct Foo; +/// fn foo(arg: impl CollectInto) -> Foo { +/// arg.collect_into() +/// } +/// ``` +/// +/// instead of the much more verbose: +/// +/// ``` +/// # struct Foo; +/// fn foo(arg: impl IntoIterator) -> Foo +/// where +/// Foo: FromIterator, +/// { +/// Foo::from_iter(arg) +/// } +/// ``` +pub trait CollectInto: IntoIterator + Sized { + /// Collects this iterable into a `T` + fn collect_into(self) -> T; +} + +// blanket impl +impl> CollectInto for I { + fn collect_into(self) -> T { + T::from_iter(self) + } +} + /// Try to parse string uri into a URL for a table path. This will do it's best to handle things /// like `/local/paths`, and even `../relative/paths`. #[allow(unused)] @@ -109,657 +144,92 @@ pub(crate) fn current_time_ms() -> DeltaResult { .map_err(|_| Error::generic("Current timestamp exceeds i64 millisecond range")) } -#[cfg(test)] -pub(crate) mod test_utils { - use std::path::PathBuf; - use std::{path::Path, sync::Arc}; - - use itertools::Itertools; - use serde::Serialize; - use tempfile::TempDir; - use test_utils::{delta_path_for_version, load_test_data}; - use url::Url; - - use crate::actions::{ - get_all_actions_schema, Add, Cdc, CommitInfo, Metadata, Protocol, Remove, - }; - use crate::arrow::array::{RecordBatch, StringArray}; - use crate::arrow::datatypes::{DataType, Field, Schema as ArrowSchema}; - use crate::committer::FileSystemCommitter; - use crate::engine::arrow_data::ArrowEngineData; - use crate::engine::default::DefaultEngineBuilder; - use crate::engine::sync::SyncEngine; - use crate::object_store::local::LocalFileSystem; - use crate::object_store::memory::InMemory; - use crate::object_store::ObjectStore; - use crate::table_features::ColumnMappingMode; - use crate::transaction::create_table::create_table; - use crate::transaction::{CreateTable, Transaction}; - use crate::{DeltaResult, EngineData, Error, SnapshotRef}; - use crate::{Engine, Snapshot}; - - #[derive(Serialize)] - pub(crate) enum Action { - #[serde(rename = "add")] - Add(Add), - #[serde(rename = "remove")] - Remove(Remove), - #[serde(rename = "cdc")] - Cdc(Cdc), - #[serde(rename = "metaData")] - Metadata(Metadata), - #[serde(rename = "protocol")] - Protocol(Protocol), - #[allow(unused)] - #[serde(rename = "commitInfo")] - CommitInfo(CommitInfo), - } - - use crate::schema::{ - ArrayType, ColumnMetadataKey, DataType as KernelDataType, MapType, MetadataValue, - PrimitiveType, SchemaRef, StructField, StructType, - }; - - /// A mock table that writes commits to a local temporary delta log. This can be used to - /// construct a delta log used for testing. - pub(crate) struct LocalMockTable { - commit_num: u64, - store: Arc, - dir: TempDir, - } - - impl LocalMockTable { - pub(crate) fn new() -> Self { - let dir = tempfile::tempdir().unwrap(); - let store = Arc::new(LocalFileSystem::new_with_prefix(dir.path()).unwrap()); - Self { - commit_num: 0, - store, - dir, - } - } - /// Writes all `actions` to a new commit in the log - pub(crate) async fn commit(&mut self, actions: impl IntoIterator) { - let data = actions - .into_iter() - .map(|action| serde_json::to_string(&action).unwrap()) - .join("\n"); - - let path = delta_path_for_version(self.commit_num, "json"); - self.commit_num += 1; - - self.store - .put(&path, data.into()) - .await - .expect("put log file in store"); - } - - /// Get the path to the root of the table. - pub(crate) fn table_root(&self) -> &Path { - self.dir.path() +/// Extension trait for folding zero or one value from an [`Option`] into a base value. +#[internal_api] +pub(crate) trait FoldWithOption: Sized { + /// Applies an optional fold operation `f` to `self` if `opt` is [`Some`]; otherwise returns + /// `self` unchanged. + /// + /// Similar to `opt.iter().fold(self, |acc, value| f(acc, value))`, but accepting `FnOnce` + /// instead of requiring `FnMut`, and with the base value as receiver instead of the option. + fn fold_with(self, opt: Option, f: impl FnOnce(Self, U) -> Self) -> Self { + match opt { + Some(value) => f(self, value), + None => self, } } - /// Try to convert an `EngineData` into a `RecordBatch`. Panics if not using `ArrowEngineData` from - /// the default module - fn into_record_batch(engine_data: Box) -> RecordBatch { - ArrowEngineData::try_from_engine_data(engine_data) - .unwrap() - .into() - } - - /// Checks that two `EngineData` objects are equal by converting them to `RecordBatch` and comparing - pub(crate) fn assert_batch_matches(actual: Box, expected: Box) { - assert_eq!(into_record_batch(actual), into_record_batch(expected)); - } - - pub(crate) fn string_array_to_engine_data(string_array: StringArray) -> Box { - let string_field = Arc::new(Field::new("a", DataType::Utf8, true)); - let schema = Arc::new(ArrowSchema::new(vec![string_field])); - let batch = RecordBatch::try_new(schema, vec![Arc::new(string_array)]) - .expect("Can't convert to record batch"); - Box::new(ArrowEngineData::new(batch)) - } - - pub(crate) fn parse_json_batch(json_strings: StringArray) -> Box { - let engine = SyncEngine::new(); - let json_handler = engine.json_handler(); - let output_schema = get_all_actions_schema().clone(); - json_handler - .parse_json(string_array_to_engine_data(json_strings), output_schema) - .unwrap() - } - - pub(crate) fn action_batch() -> Box { - let json_strings: StringArray = vec![ - r#"{"add":{"path":"part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet","partitionValues":{},"size":635,"modificationTime":1677811178336,"dataChange":true,"stats":"{\"numRecords\":10,\"minValues\":{\"value\":0},\"maxValues\":{\"value\":9},\"nullCount\":{\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1677811178336000","MIN_INSERTION_TIME":"1677811178336000","MAX_INSERTION_TIME":"1677811178336000","OPTIMIZE_TARGET_SIZE":"268435456"}}}"#, - r#"{"remove":{"path":"part-00003-f525f459-34f9-46f5-82d6-d42121d883fd.c000.snappy.parquet","deletionTimestamp":1670892998135,"dataChange":true,"partitionValues":{"c1":"4","c2":"c"},"size":452}}"#, - r#"{"commitInfo":{"timestamp":1677811178585,"operation":"WRITE","operationParameters":{"mode":"ErrorIfExists","partitionBy":"[]"},"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"10","numOutputBytes":"635"},"engineInfo":"Databricks-Runtime/","txnId":"a6a94671-55ef-450e-9546-b8465b9147de"}}"#, - r#"{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors"]}}"#, - r#"{"metaData":{"id":"testId","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"value\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.columnMapping.mode":"none", "delta.enableChangeDataFeed":"true"},"createdTime":1677811175819}}"#, - r#"{"cdc":{"path":"_change_data/age=21/cdc-00000-93f7fceb-281a-446a-b221-07b88132d203.c000.snappy.parquet","partitionValues":{"age":"21"},"size":1033,"dataChange":false}}"#, - r#"{"sidecar":{"path":"016ae953-37a9-438e-8683-9a9a4a79a395.parquet","sizeInBytes":9268,"modificationTime":1714496113961,"tags":{"tag_foo":"tag_bar"}}}"#, - r#"{"txn":{"appId":"myApp","version": 3}}"#, - r#"{"checkpointMetadata":{"version":2, "tags":{"tag_foo":"tag_bar"}}}"#, - ] - .into(); - parse_json_batch(json_strings) - } - - // TODO: allow tests to pass in context (issue#1133) - #[track_caller] - pub(crate) fn assert_result_error_with_message( - res: Result, - message: &str, - ) { - match res { - Ok(_) => panic!("Expected error with message {message}, but got Ok result"), - Err(error) => { - let error_str = error.to_string(); - assert!( - error_str.contains(message), - "Error message does not contain the expected message.\nExpected message:\t{message}\nActual message:\t\t{error_str}" - ); - } + /// Fallible [`fold_with`](Self::fold_with): applies `Result`-returning `f` to `self` if `opt` + /// is [`Some`], otherwise returns `self` unchanged (wrapped in `Ok`). + fn try_fold_with( + self, + opt: Option, + f: impl FnOnce(Self, U) -> Result, + ) -> Result { + match opt { + Some(value) => f(self, value), + None => Ok(self), } } +} - /// Asserts the 2x2 matrix of (schema_has_feature, protocol_supports_feature) outcomes - /// for schema-level feature validators. The expected pattern is: - /// - schema + protocol => Ok - /// - no schema + no protocol => Ok - /// - no schema + protocol => Ok - /// - schema + no protocol => Err (orphaned schema presence) +// Blanket impl -- every type can fold_with an Option. +impl FoldWithOption for T {} + +/// Extension trait for adding completion callbacks to iterators. +pub(crate) trait IteratorExt: Iterator + Sized { + /// Wraps this iterator to call a closure when fully exhausted. /// - /// Additional error schemas (e.g. nested) are also tested against `protocol_without`. - #[track_caller] - pub(crate) fn assert_schema_feature_validation( - schema_with: &StructType, - schema_without: &StructType, - protocol_with: &Protocol, - protocol_without: &Protocol, - extra_err_schemas: &[&StructType], - err_msg: &str, - ) { - make_test_tc(schema_with.clone(), protocol_with.clone(), []) - .expect("feature present + supported"); - make_test_tc(schema_without.clone(), protocol_without.clone(), []) - .expect("feature absent + unsupported"); - make_test_tc(schema_without.clone(), protocol_with.clone(), []) - .expect("feature absent + supported"); - assert_result_error_with_message( - make_test_tc(schema_with.clone(), protocol_without.clone(), []), - err_msg, - ); - for schema in extra_err_schemas { - assert_result_error_with_message( - make_test_tc((*schema).clone(), protocol_without.clone(), []), - err_msg, - ); + /// The closure is called only when `next()` returns `None`. If the iterator + /// is dropped before exhaustion, a warning is logged but the closure is not called. + fn on_complete(self, f: F) -> OnComplete { + OnComplete { + inner: self, + on_complete: Some(f), } } +} - /// Creates a [`TableConfiguration`] from a schema, protocol, and table properties. - /// Useful for testing validators that need a TC. - pub(crate) fn make_test_tc( - schema: StructType, - protocol: Protocol, - props: impl IntoIterator, - ) -> crate::DeltaResult { - let schema = std::sync::Arc::new(schema); - let metadata = - Metadata::try_new(None, None, schema, vec![], 0, props.into_iter().collect()).unwrap(); - let table_root = Url::try_from("file:///").unwrap(); - crate::table_configuration::TableConfiguration::try_new(metadata, protocol, table_root, 0) - } +impl IteratorExt for I {} - /// Helper to get a field from a StructType by name, panicking if not found. - pub(crate) fn get_schema_field(struct_type: &StructType, name: &str) -> StructField { - struct_type - .fields() - .find(|f| f.name() == name) - .unwrap_or_else(|| panic!("Field '{name}' not found")) - .clone() - } +/// Iterator adaptor that executes a closure when fully exhausted. +pub(crate) struct OnComplete { + inner: I, + on_complete: Option, +} - /// Validates that a schema has the expected checkpoint structure with top-level action fields - /// and proper nested types for add, metaData, and protocol actions. - pub(crate) fn validate_checkpoint_schema(schema: &SchemaRef) { - // Verify top-level action fields exist and are structs - let top_level_fields = ["txn", "add", "remove", "metaData", "protocol"]; - for field_name in top_level_fields { - let field = get_schema_field(schema, field_name); - assert!( - matches!(field.data_type(), KernelDataType::Struct(_)), - "Field '{field_name}' should be a struct type" +impl Drop for OnComplete { + fn drop(&mut self) { + if self.on_complete.is_some() { + tracing::debug!( + "OnComplete iterator dropped before exhaustion; completion callback not called" ); } - - // Verify 'add' struct has expected fields with correct types - let add_field = get_schema_field(schema, "add"); - let add_struct = match add_field.data_type() { - KernelDataType::Struct(s) => s, - _ => panic!("'add' should be a struct"), - }; - assert_eq!( - get_schema_field(add_struct, "path").data_type(), - &KernelDataType::Primitive(PrimitiveType::String) - ); - assert_eq!( - get_schema_field(add_struct, "size").data_type(), - &KernelDataType::Primitive(PrimitiveType::Long) - ); - assert!( - matches!( - get_schema_field(add_struct, "partitionValues").data_type(), - KernelDataType::Map(_) - ), - "'partitionValues' should be a map type" - ); - - // Verify 'metaData' struct has nested 'format' struct - let metadata_field = get_schema_field(schema, "metaData"); - let metadata_struct = match metadata_field.data_type() { - KernelDataType::Struct(s) => s, - _ => panic!("'metaData' should be a struct"), - }; - let format_field = get_schema_field(metadata_struct, "format"); - let format_struct = match format_field.data_type() { - KernelDataType::Struct(s) => s, - _ => panic!("'format' should be a struct"), - }; - assert_eq!( - get_schema_field(format_struct, "provider").data_type(), - &KernelDataType::Primitive(PrimitiveType::String) - ); - - // Verify 'protocol' struct has version fields - let protocol_field = get_schema_field(schema, "protocol"); - let protocol_struct = match protocol_field.data_type() { - KernelDataType::Struct(s) => s, - _ => panic!("'protocol' should be a struct"), - }; - assert_eq!( - get_schema_field(protocol_struct, "minReaderVersion").data_type(), - &KernelDataType::Primitive(PrimitiveType::Integer) - ); - assert_eq!( - get_schema_field(protocol_struct, "minWriterVersion").data_type(), - &KernelDataType::Primitive(PrimitiveType::Integer) - ); - } - - // ==================== Test schema helpers ==================== - // - // Reusable test schemas - // Each variant exists with and without column mapping metadata. - - /// Helper to add column mapping metadata to a [`StructField`]. - fn with_column_mapping(field: StructField, id: i64, physical_name: &str) -> StructField { - field.with_metadata([ - ( - ColumnMetadataKey::ColumnMappingId.as_ref(), - MetadataValue::Number(id), - ), - ( - ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), - MetadataValue::String(physical_name.into()), - ), - ]) - } - - /// Flat schema: `[id: long, name: string]` - pub(crate) fn test_schema_flat() -> SchemaRef { - Arc::new(StructType::new_unchecked([ - StructField::new("id", KernelDataType::LONG, false), - StructField::nullable("name", KernelDataType::STRING), - ])) - } - - /// Flat schema with column mapping metadata. - pub(crate) fn test_schema_flat_with_column_mapping() -> SchemaRef { - Arc::new(StructType::new_unchecked([ - with_column_mapping( - StructField::new("id", KernelDataType::LONG, false), - 1, - "phys_id", - ), - with_column_mapping( - StructField::nullable("name", KernelDataType::STRING), - 2, - "phys_name", - ), - ])) - } - - /// Nested struct schema with array and map inside the struct - pub(crate) fn test_schema_nested() -> SchemaRef { - Arc::new(StructType::new_unchecked([ - StructField::new("id", KernelDataType::LONG, false), - StructField::nullable( - "info", - StructType::new_unchecked([ - StructField::nullable("name", KernelDataType::STRING), - StructField::nullable("age", KernelDataType::INTEGER), - StructField::nullable( - "tags", - MapType::new(KernelDataType::STRING, KernelDataType::STRING, true), - ), - StructField::nullable("scores", ArrayType::new(KernelDataType::INTEGER, true)), - ]), - ), - ])) - } - - /// Nested struct schema with column mapping metadata. - pub(crate) fn test_schema_nested_with_column_mapping() -> SchemaRef { - Arc::new(StructType::new_unchecked([ - with_column_mapping( - StructField::new("id", KernelDataType::LONG, false), - 1, - "phys_id", - ), - with_column_mapping( - StructField::nullable( - "info", - StructType::new_unchecked([ - with_column_mapping( - StructField::nullable("name", KernelDataType::STRING), - 3, - "phys_name", - ), - with_column_mapping( - StructField::nullable("age", KernelDataType::INTEGER), - 4, - "phys_age", - ), - with_column_mapping( - StructField::nullable( - "tags", - MapType::new(KernelDataType::STRING, KernelDataType::STRING, true), - ), - 5, - "phys_tags", - ), - with_column_mapping( - StructField::nullable( - "scores", - ArrayType::new(KernelDataType::INTEGER, true), - ), - 6, - "phys_scores", - ), - ]), - ), - 2, - "phys_info", - ), - ])) - } - - /// Schema with a map - pub(crate) fn test_schema_with_map() -> SchemaRef { - let value_struct = StructType::new_unchecked([ - StructField::nullable("key", KernelDataType::STRING), - StructField::nullable("value", KernelDataType::INTEGER), - ]); - Arc::new(StructType::new_unchecked([ - StructField::new("id", KernelDataType::LONG, false), - StructField::nullable( - "entries", - MapType::new( - KernelDataType::STRING, - KernelDataType::Struct(Box::new(value_struct)), - true, - ), - ), - StructField::nullable("name", KernelDataType::STRING), - ])) - } - - /// Schema with a map and column mapping metadata. - pub(crate) fn test_schema_with_map_and_column_mapping() -> SchemaRef { - let value_struct = StructType::new_unchecked([ - with_column_mapping( - StructField::nullable("key", KernelDataType::STRING), - 4, - "phys_key", - ), - with_column_mapping( - StructField::nullable("value", KernelDataType::INTEGER), - 5, - "phys_value", - ), - ]); - Arc::new(StructType::new_unchecked([ - with_column_mapping( - StructField::new("id", KernelDataType::LONG, false), - 1, - "phys_id", - ), - with_column_mapping( - StructField::nullable( - "entries", - MapType::new( - KernelDataType::STRING, - KernelDataType::Struct(Box::new(value_struct)), - true, - ), - ), - 2, - "phys_entries", - ), - with_column_mapping( - StructField::nullable("name", KernelDataType::STRING), - 3, - "phys_name", - ), - ])) - } - - /// Schema with an array - pub(crate) fn test_schema_with_array() -> SchemaRef { - let item_struct = StructType::new_unchecked([ - StructField::nullable("label", KernelDataType::STRING), - StructField::nullable("count", KernelDataType::INTEGER), - ]); - Arc::new(StructType::new_unchecked([ - StructField::new("id", KernelDataType::LONG, false), - StructField::nullable( - "items", - ArrayType::new(KernelDataType::Struct(Box::new(item_struct)), true), - ), - StructField::nullable("name", KernelDataType::STRING), - ])) - } - - /// Schema with an array and column mapping metadata. - pub(crate) fn test_schema_with_array_and_column_mapping() -> SchemaRef { - let item_struct = StructType::new_unchecked([ - with_column_mapping( - StructField::nullable("label", KernelDataType::STRING), - 4, - "phys_label", - ), - with_column_mapping( - StructField::nullable("count", KernelDataType::INTEGER), - 5, - "phys_count", - ), - ]); - Arc::new(StructType::new_unchecked([ - with_column_mapping( - StructField::new("id", KernelDataType::LONG, false), - 1, - "phys_id", - ), - with_column_mapping( - StructField::nullable( - "items", - ArrayType::new(KernelDataType::Struct(Box::new(item_struct)), true), - ), - 2, - "phys_items", - ), - with_column_mapping( - StructField::nullable("name", KernelDataType::STRING), - 3, - "phys_name", - ), - ])) } +} - /// Deeply nested schema: struct -> array -> struct -> map(value) -> struct. - /// - /// The leaf struct field is intentionally **not** annotated with column mapping metadata, - /// so this schema can be used to test error paths when column mapping is enabled. - pub(crate) fn test_deep_nested_schema_missing_leaf_cm() -> StructType { - let leaf_struct = - StructType::new_unchecked([StructField::new("leaf", KernelDataType::INTEGER, false)]); - let map_type = MapType::new( - KernelDataType::STRING, - KernelDataType::Struct(Box::new(leaf_struct)), - true, - ); - let mid_struct = StructType::new_unchecked([with_column_mapping( - StructField::nullable("mid_field", map_type), - 2, - "phys_mid_field", - )]); - let array_type = ArrayType::new(KernelDataType::Struct(Box::new(mid_struct)), true); - StructType::new_unchecked([with_column_mapping( - StructField::nullable("top", array_type), - 1, - "phys_top", - )]) - } +impl Iterator for OnComplete +where + I: Iterator, + F: FnOnce(), +{ + type Item = I::Item; - /// Build a create-table transaction with the given schema and column mapping mode. - /// Returns the engine and uncommitted transaction. - pub(crate) fn setup_column_mapping_txn( - schema: SchemaRef, - mode: ColumnMappingMode, - ) -> DeltaResult<(Arc, Transaction)> { - let mode_str = match mode { - ColumnMappingMode::Name => "name", - ColumnMappingMode::Id => "id", - ColumnMappingMode::None => "none", - }; - let store = Arc::new(InMemory::new()); - let engine: Arc = Arc::new(DefaultEngineBuilder::new(store).build()); - - let txn = create_table("memory:///test_table", schema, "DefaultEngine") - .with_table_properties([("delta.columnMapping.mode", mode_str)]) - .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))?; - Ok((engine, txn)) + fn size_hint(&self) -> (usize, Option) { + self.inner.size_hint() } - /// Validate that a physical schema matches the logical schema's column mapping metadata. - /// For Name/Id modes, checks physicalName, columnMapping.id, and parquet.field.id on - /// each field. For None mode, only checks field names match. - pub(crate) fn validate_physical_schema_column_mapping( - logical_schema: &StructType, - physical_schema: &StructType, - mode: ColumnMappingMode, - ) { - assert_eq!( - physical_schema.fields().count(), - logical_schema.fields().count() - ); - - // Collect expected (physical_name, field_id) from logical schema - let expected: Vec<_> = logical_schema - .fields() - .map(|f| { - let physical_name = - match f.get_config_value(&ColumnMetadataKey::ColumnMappingPhysicalName) { - Some(MetadataValue::String(name)) => name.clone(), - _ if mode == ColumnMappingMode::None => f.name().to_string(), - _ => panic!("Logical field '{}' missing physicalName metadata", f.name()), - }; - let field_id = match f.get_config_value(&ColumnMetadataKey::ColumnMappingId) { - Some(MetadataValue::Number(id)) => *id, - _ if mode == ColumnMappingMode::None => -1, - _ => panic!( - "Logical field '{}' missing columnMapping.id metadata", - f.name() - ), - }; - (physical_name, field_id) - }) - .collect(); - - // Validate each physical field against expected values - for (physical_field, (expected_name, expected_id)) in - physical_schema.fields().zip(expected.iter()) - { - assert_eq!( - physical_field.name(), - expected_name, - "Physical field name mismatch" - ); - - if mode == ColumnMappingMode::None { - continue; + fn next(&mut self) -> Option { + match self.inner.next() { + Some(item) => Some(item), + None => { + if let Some(f) = self.on_complete.take() { + f(); + } + None } - - assert_eq!( - physical_field.get_config_value(&ColumnMetadataKey::ColumnMappingPhysicalName), - Some(&MetadataValue::String(expected_name.clone())), - "columnMapping.physicalName mismatch for '{}'", - physical_field.name() - ); - - assert_eq!( - physical_field.get_config_value(&ColumnMetadataKey::ColumnMappingId), - Some(&MetadataValue::Number(*expected_id)), - "columnMapping.id mismatch for '{}'", - physical_field.name() - ); - - assert_eq!( - physical_field.get_config_value(&ColumnMetadataKey::ParquetFieldId), - Some(&MetadataValue::Number(*expected_id)), - "parquet.field.id mismatch for '{}'", - physical_field.name() - ); } } - - /// Load a test table from tests/data directory. - /// Tries compressed (tar.zst) first, falls back to extracted. - /// Returns (engine, snapshot, optional tempdir). The TempDir must be kept alive - /// for the duration of the test to prevent premature cleanup of extracted files. - pub(crate) fn load_test_table( - table_name: &str, - ) -> DeltaResult<(Arc, SnapshotRef, Option)> { - // Try loading compressed table first, fall back to extracted - let (path, tempdir) = match load_test_data("tests/data", table_name) { - Ok(test_dir) => { - let test_path = test_dir.path().join(table_name); - (test_path, Some(test_dir)) - } - Err(_) => { - // Fall back to already-extracted table - let manifest_dir = env!("CARGO_MANIFEST_DIR"); - let mut path = PathBuf::from(manifest_dir); - path.push("tests/data"); - path.push(table_name); - let path = std::fs::canonicalize(path) - .map_err(|e| Error::Generic(format!("Failed to canonicalize path: {e}")))?; - (path, None) - } - }; - - // Create engine and snapshot from the resolved path - let url = Url::from_directory_path(&path) - .map_err(|_| Error::Generic("Failed to create URL from path".to_string()))?; - - let store = Arc::new(LocalFileSystem::new()); - let engine = Arc::new(DefaultEngineBuilder::new(store).build()); - let snapshot = Snapshot::builder_for(url).build(engine.as_ref())?; - Ok((engine, snapshot, tempdir)) - } } #[cfg(test)] @@ -817,4 +287,54 @@ mod tests { "s3://foo/__unitystorage/catalogs/cid/tables/tid/" ); } + + mod on_complete_tests { + use std::sync::atomic::{AtomicBool, AtomicU32, Ordering}; + use std::sync::Arc; + + use super::*; + + #[test] + fn test_calls_on_exhaustion() { + let called = Arc::new(AtomicBool::new(false)); + let called_clone = called.clone(); + let mut iter = vec![1, 2].into_iter().on_complete(move || { + called_clone.store(true, Ordering::SeqCst); + }); + assert_eq!(iter.next(), Some(1)); + assert!(!called.load(Ordering::SeqCst)); + assert_eq!(iter.next(), Some(2)); + assert_eq!(iter.next(), None); + assert!(called.load(Ordering::SeqCst)); + } + + #[test] + fn test_does_not_call_on_early_drop() { + let called = Arc::new(AtomicBool::new(false)); + let called_clone = called.clone(); + { + let mut iter = vec![1, 2].into_iter().on_complete(move || { + called_clone.store(true, Ordering::SeqCst); + }); + assert_eq!(iter.next(), Some(1)); + // Drop without exhausting - callback should NOT be called + } + assert!(!called.load(Ordering::SeqCst)); + } + + #[test] + fn test_calls_only_once() { + let count = Arc::new(AtomicU32::new(0)); + let count_clone = count.clone(); + { + let mut iter = vec![1].into_iter().on_complete(move || { + count_clone.fetch_add(1, Ordering::SeqCst); + }); + assert_eq!(iter.next(), Some(1)); + assert_eq!(iter.next(), None); // triggers callback + assert_eq!(iter.next(), None); // should not trigger again + } // drop should not trigger again + assert_eq!(count.load(Ordering::SeqCst), 1); + } + } } diff --git a/kernel/tests/README.md b/kernel/tests/README.md index eda2a93629..e77e7bb7e2 100644 --- a/kernel/tests/README.md +++ b/kernel/tests/README.md @@ -9,8 +9,8 @@ Test tables organized by feature area. Tables live in two locations: | Table | Location | Schema | Protocol (R/W) | Features | Description | Tests | |-------|----------|--------|----------|----------|-------------|-------| -| `table-with-dv-small` | data/ | `value: int` | v3/v7 | r:`deletionVectors` w:`deletionVectors` | 10 rows, 2 soft-deleted by DV, 8 visible. Most heavily referenced test table. | `dv.rs::test_table_scan(with_dv)`, `write.rs::test_remove_files_adds_expected_entries`, `write.rs::test_update_deletion_vectors_adds_expected_entries`, `read.rs::with_predicate_and_removes`, `path.rs::test_to_uri/test_child/test_child_escapes`, `snapshot.rs::test_snapshot_read_metadata/test_new_snapshot/test_snapshot_new_from/test_read_table_with_missing_last_checkpoint/test_log_compaction_writer`, `deletion_vector.rs` tests, `transaction/mod.rs::setup_dv_enabled_table/test_add_files_schema/test_new_deletion_vector_path`, `default/parquet.rs` read test, `default/json.rs` read test, `log_compaction/tests.rs::create_mock_snapshot`, `resolve_dvs.rs` tests | -| `table-without-dv-small` | data/ | `value: long` | v1/v2 | | 10 rows, all visible. Companion to table-with-dv-small. | `dv.rs::test_table_scan(without_dv)`, `transaction/mod.rs::setup_non_dv_table/create_existing_table_txn/test_commit_io_error_returns_retryable_transaction`, `sequential_phase.rs::test_sequential_v2_with_commits_only/test_sequential_finish_before_exhaustion_error`, `parallel_phase.rs` tests, `scan/tests.rs::test_scan_metadata_paths/test_scan_metadata/test_scan_metadata_from_same_version` | +| `table-with-dv-small` | data/ | `value: int` | v3/v7 | r:`deletionVectors` w:`deletionVectors` | 10 rows, 2 soft-deleted by DV, 8 visible. Most heavily referenced test table. | `features::dv::test_table_scan(with_dv)`, `write::remove_dv::test_remove_files_adds_expected_entries`, `write::remove_dv::test_update_deletion_vectors_adds_expected_entries`, `read.rs::with_predicate_and_removes`, `path.rs::test_to_uri/test_child/test_child_escapes`, `snapshot.rs::test_snapshot_read_metadata/test_new_snapshot/test_snapshot_new_from/test_read_table_with_missing_last_checkpoint/test_log_compaction_writer`, `deletion_vector.rs` tests, `transaction/mod.rs::setup_dv_enabled_table/test_add_files_schema/test_new_deletion_vector_path`, `default/parquet.rs` read test, `default/json.rs` read test, `log_compaction/tests.rs::create_mock_snapshot`, `resolve_dvs.rs` tests | +| `table-without-dv-small` | data/ | `value: long` | v1/v2 | | 10 rows, all visible. Companion to table-with-dv-small. | `features::dv::test_table_scan(without_dv)`, `transaction/mod.rs::setup_non_dv_table/create_existing_table_txn/test_commit_io_error_returns_retryable_transaction`, `sequential_phase.rs::test_sequential_v2_with_commits_only/test_sequential_finish_before_exhaustion_error`, `parallel_phase.rs` tests, `scan/tests.rs::test_scan_metadata_paths/test_scan_metadata/test_scan_metadata_from_same_version` | | `with-short-dv` | data/ | `id: long, value: string, timestamp: timestamp, rand: double` | v3/v7 | r:`deletionVectors` w:`deletionVectors` | 2 files x 5 rows. First file has inline DV (`storageType="u"`) deleting 3 rows. | `read.rs::short_dv` | | `dv-partitioned-with-checkpoint` | golden_data/ | `value: int, part: int` partitioned by `part` | v3/v7 | r:`deletionVectors` w:`deletionVectors` | DVs on a partitioned table with a checkpoint | `golden_tables.rs::golden_test!` | | `dv-with-columnmapping` | golden_data/ | `value: int` | v3/v7 | r:`deletionVectors,columnMapping` w:`deletionVectors,columnMapping`, `columnMapping.mode=name` | DVs combined with column mapping | `golden_tables.rs::golden_test!` | @@ -21,29 +21,29 @@ Test tables organized by feature area. Tables live in two locations: | Table | Location | Schema | Protocol (R/W) | Features | Description | Tests | |-------|----------|--------|----------|----------|-------------|-------| | `table-with-cdf` | data/ | `part: int, id: int` | v1/v4 | `enableChangeDataFeed=true` | Fake file paths for CDF validation logic | `table_changes/mod.rs::test_table_changes_errors/test_cdf_start_end_version/test_cdf_no_end_version`, `table_changes/scan.rs::test_table_changes_scan_errors/test_create_cdf_scan_batches_errors` | -| `cdf-table` | data/cdf-table.tar.zst | `id: int, name: string, birthday: date` partitioned by `birthday` | v1/v4 | `enableChangeDataFeed=true` | Base CDF table | `cdf.rs::basic_cdf` | -| `cdf-table-simple` | data/cdf-table-simple.tar.zst | `id: long` | v1/v7 | w:`changeDataFeed,appendOnly,invariants`, `enableChangeDataFeed=true` | Version range queries and validation | `cdf.rs::simple_cdf_version_ranges/invalid_range_end_before_start/invalid_range_start_after_last_version_of_table` | -| `cdf-table-non-partitioned` | data/cdf-table-non-partitioned.tar.zst | `id: int, name: string, birthday: date, long_field: long, boolean_field: boolean, double_field: double, smallint_field: short` | v1/v4 | `enableChangeDataFeed=true` | Non-partitioned CDF table | `cdf.rs::cdf_non_partitioned` | -| `cdf-table-partitioned` | data/cdf-table-partitioned.tar.zst | `id: long, text: string, part: long` partitioned by `part` | v1/v7 | w:`changeDataFeed,appendOnly,invariants`, `enableChangeDataFeed=true` | Partitioned CDF table | `cdf.rs::partition_table` | -| `cdf-table-delete-unconditional` | data/cdf-table-delete-unconditional.tar.zst | `id: long` | v1/v7 | w:`changeDataFeed,appendOnly,invariants`, `enableChangeDataFeed=true` | Unconditional DELETE | `cdf.rs::unconditional_delete` | -| `cdf-table-delete-conditional-all-rows` | data/cdf-table-delete-conditional-all-rows.tar.zst | `id: long` | v1/v7 | w:`changeDataFeed,appendOnly,invariants`, `enableChangeDataFeed=true` | Conditional DELETE removing all rows | `cdf.rs::conditional_delete_all_rows` | -| `cdf-table-delete-conditional-two-rows` | data/cdf-table-delete-conditional-two-rows.tar.zst | `id: long` | v1/v7 | w:`changeDataFeed,appendOnly,invariants`, `enableChangeDataFeed=true` | Conditional DELETE removing two rows | `cdf.rs::conditional_delete_two_rows` | -| `cdf-table-update-ops` | data/cdf-table-update-ops.tar.zst | `id: long` | v1/v7 | w:`changeDataFeed,appendOnly,invariants`, `enableChangeDataFeed=true` | UPDATE operations | `cdf.rs::update_operations` | -| `cdf-table-data-change` | data/cdf-table-data-change.tar.zst | `id: long` | v1/v7 | w:`changeDataFeed,appendOnly,invariants`, `enableChangeDataFeed=true` | Operations with `dataChange=false` | `cdf.rs::false_data_change_is_ignored` | -| `cdf-table-with-dv` | data/cdf-table-with-dv.tar.zst | `value: int` | v3/v7 | r:`deletionVectors` w:`deletionVectors,changeDataFeed`, `enableChangeDataFeed=true` | CDF with deletion vectors | `cdf.rs::cdf_with_deletion_vector` | -| `cdf-table-with-cdc-and-dvs` | data/cdf-table-with-cdc-and-dvs.tar.zst | `id: int, comment: string` | v3/v7 | r:`deletionVectors` w:`deletionVectors,changeDataFeed`, `enableChangeDataFeed=true` | CDF with both CDC files and DVs | `cdf.rs::cdf_with_cdc_and_dvs` | -| `cdf-table-backtick-column-names` | data/cdf-table-backtick-column-names.tar.zst | `` `id.num`: int, `id.num\`s`: int, struct_col: struct{field: int, `field.one`: int} `` | v1/v7 | w:`changeDataFeed,appendOnly,invariants`, `enableChangeDataFeed=true` | Backtick-escaped column names | `cdf.rs::backtick_column_names` | -| `cdf-column-mapping-name-mode` | data/cdf-column-mapping-name-mode.tar.zst | `id: long, name: string, value: double` | v2/v5 | Column mapping (name) | CDF + column mapping name mode | `cdf.rs::cdf_with_column_mapping_name_mode` | -| `cdf-column-mapping-name-mode-3-7` | data/cdf-column-mapping-name-mode-3-7.tar.zst | `id: long, name: string, value: double` | v3/v7 | r:`columnMapping,deletionVectors` w:`columnMapping,deletionVectors` | CDF + column mapping name mode (v3/v7) | `cdf.rs::cdf_with_column_mapping_name_mode` | -| `cdf-column-mapping-id-mode` | data/cdf-column-mapping-id-mode.tar.zst | `id: long, name: string, value: double` | v2/v5 | Column mapping (id) | CDF + column mapping id mode | `cdf.rs::cdf_with_column_mapping_id_mode` | +| `cdf-table` | data/cdf-table.tar.zst | `id: int, name: string, birthday: date` partitioned by `birthday` | v1/v4 | `enableChangeDataFeed=true` | Base CDF table | `features::cdf::basic_cdf` | +| `cdf-table-simple` | data/cdf-table-simple.tar.zst | `id: long` | v1/v7 | w:`changeDataFeed,appendOnly,invariants`, `enableChangeDataFeed=true` | Version range queries and validation | `features::cdf::simple_cdf_version_ranges/invalid_range_end_before_start/invalid_range_start_after_last_version_of_table` | +| `cdf-table-non-partitioned` | data/cdf-table-non-partitioned.tar.zst | `id: int, name: string, birthday: date, long_field: long, boolean_field: boolean, double_field: double, smallint_field: short` | v1/v4 | `enableChangeDataFeed=true` | Non-partitioned CDF table | `features::cdf::cdf_non_partitioned` | +| `cdf-table-partitioned` | data/cdf-table-partitioned.tar.zst | `id: long, text: string, part: long` partitioned by `part` | v1/v7 | w:`changeDataFeed,appendOnly,invariants`, `enableChangeDataFeed=true` | Partitioned CDF table | `features::cdf::partition_table` | +| `cdf-table-delete-unconditional` | data/cdf-table-delete-unconditional.tar.zst | `id: long` | v1/v7 | w:`changeDataFeed,appendOnly,invariants`, `enableChangeDataFeed=true` | Unconditional DELETE | `features::cdf::unconditional_delete` | +| `cdf-table-delete-conditional-all-rows` | data/cdf-table-delete-conditional-all-rows.tar.zst | `id: long` | v1/v7 | w:`changeDataFeed,appendOnly,invariants`, `enableChangeDataFeed=true` | Conditional DELETE removing all rows | `features::cdf::conditional_delete_all_rows` | +| `cdf-table-delete-conditional-two-rows` | data/cdf-table-delete-conditional-two-rows.tar.zst | `id: long` | v1/v7 | w:`changeDataFeed,appendOnly,invariants`, `enableChangeDataFeed=true` | Conditional DELETE removing two rows | `features::cdf::conditional_delete_two_rows` | +| `cdf-table-update-ops` | data/cdf-table-update-ops.tar.zst | `id: long` | v1/v7 | w:`changeDataFeed,appendOnly,invariants`, `enableChangeDataFeed=true` | UPDATE operations | `features::cdf::update_operations` | +| `cdf-table-data-change` | data/cdf-table-data-change.tar.zst | `id: long` | v1/v7 | w:`changeDataFeed,appendOnly,invariants`, `enableChangeDataFeed=true` | Operations with `dataChange=false` | `features::cdf::false_data_change_is_ignored` | +| `cdf-table-with-dv` | data/cdf-table-with-dv.tar.zst | `value: int` | v3/v7 | r:`deletionVectors` w:`deletionVectors,changeDataFeed`, `enableChangeDataFeed=true` | CDF with deletion vectors | `features::cdf::cdf_with_deletion_vector` | +| `cdf-table-with-cdc-and-dvs` | data/cdf-table-with-cdc-and-dvs.tar.zst | `id: int, comment: string` | v3/v7 | r:`deletionVectors` w:`deletionVectors,changeDataFeed`, `enableChangeDataFeed=true` | CDF with both CDC files and DVs | `features::cdf::cdf_with_cdc_and_dvs` | +| `cdf-table-backtick-column-names` | data/cdf-table-backtick-column-names.tar.zst | `` `id.num`: int, `id.num\`s`: int, struct_col: struct{field: int, `field.one`: int} `` | v1/v7 | w:`changeDataFeed,appendOnly,invariants`, `enableChangeDataFeed=true` | Backtick-escaped column names | `features::cdf::backtick_column_names` | +| `cdf-column-mapping-name-mode` | data/cdf-column-mapping-name-mode.tar.zst | `id: long, name: string, value: double` | v2/v5 | Column mapping (name) | CDF + column mapping name mode | `features::cdf::cdf_with_column_mapping_name_mode` | +| `cdf-column-mapping-name-mode-3-7` | data/cdf-column-mapping-name-mode-3-7.tar.zst | `id: long, name: string, value: double` | v3/v7 | r:`columnMapping,deletionVectors` w:`columnMapping,deletionVectors` | CDF + column mapping name mode (v3/v7) | `features::cdf::cdf_with_column_mapping_name_mode` | +| `cdf-column-mapping-id-mode` | data/cdf-column-mapping-id-mode.tar.zst | `id: long, name: string, value: double` | v2/v5 | Column mapping (id) | CDF + column mapping id mode | `features::cdf::cdf_with_column_mapping_id_mode` | ## Column Mapping | Table | Location | Schema | Protocol (R/W) | Features | Description | Tests | |-------|----------|--------|----------|----------|-------------|-------| -| `partition_cm/none` | data/ | `value: int, category: string` partitioned by `category` | v1/v1 | `columnMapping.mode=none` | Partitioned write with CM disabled | `write.rs::test_column_mapping_partitioned_write(cm_none)` | -| `partition_cm/id` | data/ | `value: int, category: string` partitioned by `category` | v3/v7 | r:`columnMapping` w:`columnMapping`, `columnMapping.mode=id` | Partitioned write with CM id mode | `write.rs::test_column_mapping_partitioned_write(cm_id)` | -| `partition_cm/name` | data/ | `value: int, category: string` partitioned by `category` | v3/v7 | r:`columnMapping` w:`columnMapping`, `columnMapping.mode=name` | Partitioned write with CM name mode | `write.rs::test_column_mapping_partitioned_write(cm_name)` | +| `partition_cm/none` | data/ | `value: int, category: string` partitioned by `category` | v1/v1 | `columnMapping.mode=none` | Partitioned write with CM disabled | `write::column_mapping::test_column_mapping_partitioned_write(cm_none)` | +| `partition_cm/id` | data/ | `value: int, category: string` partitioned by `category` | v3/v7 | r:`columnMapping` w:`columnMapping`, `columnMapping.mode=id` | Partitioned write with CM id mode | `write::column_mapping::test_column_mapping_partitioned_write(cm_id)` | +| `partition_cm/name` | data/ | `value: int, category: string` partitioned by `category` | v3/v7 | r:`columnMapping` w:`columnMapping`, `columnMapping.mode=name` | Partitioned write with CM name mode | `write::column_mapping::test_column_mapping_partitioned_write(cm_name)` | | `table-with-columnmapping-mode-name` | golden_data/ | `ByteType: byte, ShortType: short, IntegerType: int, LongType: long, FloatType: float, DoubleType: double, decimal: decimal(10,2), BooleanType: boolean, StringType: string, BinaryType: binary, DateType: date, TimestampType: timestamp, nested_struct: struct{aa: string, ac: struct{aca: int}}, array_of_prims: array, array_of_arrays: array>, array_of_structs: array, map_of_prims: map, map_of_rows: map, map_of_arrays: map>` | v2/v5 | `columnMapping.mode=name` | Column mapping name mode | `golden_tables.rs::golden_test!` | | `table-with-columnmapping-mode-id` | golden_data/ | `ByteType: byte, ShortType: short, IntegerType: int, LongType: long, FloatType: float, DoubleType: double, decimal: decimal(10,2), BooleanType: boolean, StringType: string, BinaryType: binary, DateType: date, TimestampType: timestamp, nested_struct: struct{aa: string, ac: struct{aca: int}}, array_of_prims: array, array_of_arrays: array>, array_of_structs: array, map_of_prims: map, map_of_rows: map, map_of_arrays: map>` | v2/v5 | `columnMapping.mode=id` | Column mapping id mode | `golden_tables.rs::golden_test!` | @@ -52,7 +52,7 @@ Test tables organized by feature area. Tables live in two locations: | Table | Location | Schema | Protocol (R/W) | Features | Description | Tests | |-------|----------|--------|----------|----------|-------------|-------| | `with_checkpoint_no_last_checkpoint` | data/ | `letter: string, int: long, date: date` | v1/v2 | `checkpointInterval=2` | Checkpoint at v2 but missing `_last_checkpoint` hint file | `snapshot.rs::test_read_table_with_checkpoint`, `scan/tests.rs::test_scan_with_checkpoint`, `sequential_phase.rs::test_sequential_checkpoint_no_commits`, `checkpoint_manifest.rs` tests, `sync/parquet.rs` test, `default/parquet.rs` test | -| `external-table-different-nullability` | data/ | `i: int` | v1/v2 | `checkpointInterval=2` | Parquet files have different nullability than Delta schema; includes checkpoint | `write.rs::test_checkpoint_non_kernel_written_table` | +| `external-table-different-nullability` | data/ | `i: int` | v1/v2 | `checkpointInterval=2` | Parquet files have different nullability than Delta schema; includes checkpoint | `write::clustered::test_checkpoint_non_kernel_written_table` | | `checkpoint` | golden_data/ | `intCol: int` | v1/v2 | | Basic checkpoint read | `golden_tables.rs::golden_test!(checkpoint_test)` | | `corrupted-last-checkpoint-kernel` | golden_data/ | `id: long` | v1/v2 | | Corrupted `_last_checkpoint` file | `golden_tables.rs::golden_test!` | | `multi-part-checkpoint` | golden_data/ | `id: long` | v1/v2 | `checkpointInterval=1` | Multi-part checkpoint files | `golden_tables.rs::golden_test!` | @@ -66,14 +66,14 @@ All `data/` V2 checkpoint tables share: schema=`id: long`, protocol v3/v7, r:`v2 | Table | Location | Format | Sidecars | `_last_checkpoint` | Tests | |-------|----------|--------|----------|--------------------|-------| -| `v2-checkpoints-json-with-sidecars` | data/v2-checkpoints-json-with-sidecars.tar.zst | JSON | Yes | No | `v2_checkpoints.rs::v2_checkpoints_json_with_sidecars`, `checkpoint_manifest.rs` test, `sequential_phase.rs::test_sequential_v2_with_sidecars`, `parallel_phase.rs` tests | -| `v2-checkpoints-json-without-sidecars` | data/v2-checkpoints-json-without-sidecars.tar.zst | JSON | No | No | `v2_checkpoints.rs::v2_checkpoints_json_without_sidecars`, `sequential_phase.rs::test_sequential_checkpoint_without_sidecars` | -| `v2-checkpoints-json-with-last-checkpoint` | data/v2-checkpoints-json-with-last-checkpoint.tar.zst | JSON | Yes | Yes | `v2_checkpoints.rs::v2_checkpoints_json_with_last_checkpoint` | -| `v2-checkpoints-parquet-with-sidecars` | data/v2-checkpoints-parquet-with-sidecars.tar.zst | Parquet | Yes | No | `v2_checkpoints.rs::v2_checkpoints_parquet_with_sidecars`, `checkpoint_manifest.rs` test, `sequential_phase.rs::test_sequential_parquet_checkpoint_with_sidecars`, `parallel_phase.rs` test | -| `v2-checkpoints-parquet-without-sidecars` | data/v2-checkpoints-parquet-without-sidecars.tar.zst | Parquet | No | No | `v2_checkpoints.rs::v2_checkpoints_parquet_without_sidecars` | -| `v2-checkpoints-parquet-with-last-checkpoint` | data/v2-checkpoints-parquet-with-last-checkpoint.tar.zst | Parquet | Yes | Yes | `v2_checkpoints.rs::v2_checkpoints_parquet_with_last_checkpoint` | -| `v2-classic-checkpoint-json` | data/v2-classic-checkpoint-json.tar.zst | JSON | Classic compat | No | `v2_checkpoints.rs::v2_classic_checkpoint_json` | -| `v2-classic-checkpoint-parquet` | data/v2-classic-checkpoint-parquet.tar.zst | Parquet | Classic compat | No | `v2_checkpoints.rs::v2_classic_checkpoint_parquet` | +| `v2-checkpoints-json-with-sidecars` | data/v2-checkpoints-json-with-sidecars.tar.zst | JSON | Yes | No | `log::v2_checkpoints::v2_checkpoints_json_with_sidecars`, `checkpoint_manifest.rs` test, `sequential_phase.rs::test_sequential_v2_with_sidecars`, `parallel_phase.rs` tests | +| `v2-checkpoints-json-without-sidecars` | data/v2-checkpoints-json-without-sidecars.tar.zst | JSON | No | No | `log::v2_checkpoints::v2_checkpoints_json_without_sidecars`, `sequential_phase.rs::test_sequential_checkpoint_without_sidecars` | +| `v2-checkpoints-json-with-last-checkpoint` | data/v2-checkpoints-json-with-last-checkpoint.tar.zst | JSON | Yes | Yes | `log::v2_checkpoints::v2_checkpoints_json_with_last_checkpoint` | +| `v2-checkpoints-parquet-with-sidecars` | data/v2-checkpoints-parquet-with-sidecars.tar.zst | Parquet | Yes | No | `log::v2_checkpoints::v2_checkpoints_parquet_with_sidecars`, `checkpoint_manifest.rs` test, `sequential_phase.rs::test_sequential_parquet_checkpoint_with_sidecars`, `parallel_phase.rs` test | +| `v2-checkpoints-parquet-without-sidecars` | data/v2-checkpoints-parquet-without-sidecars.tar.zst | Parquet | No | No | `log::v2_checkpoints::v2_checkpoints_parquet_without_sidecars` | +| `v2-checkpoints-parquet-with-last-checkpoint` | data/v2-checkpoints-parquet-with-last-checkpoint.tar.zst | Parquet | Yes | Yes | `log::v2_checkpoints::v2_checkpoints_parquet_with_last_checkpoint` | +| `v2-classic-checkpoint-json` | data/v2-classic-checkpoint-json.tar.zst | JSON | Classic compat | No | `log::v2_checkpoints::v2_classic_checkpoint_json` | +| `v2-classic-checkpoint-parquet` | data/v2-classic-checkpoint-parquet.tar.zst | Parquet | Classic compat | No | `log::v2_checkpoints::v2_classic_checkpoint_parquet` | The golden V2 checkpoint tables have a different protocol (v1/v2, no table features): @@ -86,8 +86,15 @@ The golden V2 checkpoint tables have a different protocol (v1/v2, no table featu | Table | Location | Schema | Protocol (R/W) | Features | Description | Tests | |-------|----------|--------|----------|----------|-------------|-------| -| `parsed-stats` | data/ | `id: long, name: string, age: long, salary: long` | v1/v2 | `checkpointInterval=3` | 100 rows. Checkpoint at v3 with `stats_parsed` columns. | `scan/tests.rs::test_scan_metadata_with_stats_columns/test_data_skipping_with_parsed_stats/test_scan_metadata_stats_columns_with_predicate/test_prefix_columns_simple/test_build_actions_meta_predicate_with_predicate/test_build_actions_meta_predicate_no_predicate/test_build_actions_meta_predicate_static_skip_all/test_skip_stats_disables_data_skipping/test_skip_stats_and_include_stats_columns_errors` | +| `parsed-stats` | data/ | `id: long, name: string, age: long, salary: long, ts_col: timestamp` | v1/v2 | `checkpointInterval=3` | 100 rows per commit (v0-v5). Checkpoint at v3 with `stats_parsed` columns including `ts_col` for testing timestamp stat truncation adjustment and data skipping predicates. | `scan/tests.rs::test_scan_metadata_with_stats_columns/scan_metadata_struct_columns_returns_expected_stats/scan_metadata_struct_columns_fully_pruning_predicate_yields_no_batches/test_build_actions_meta_predicate_with_predicate/test_build_actions_meta_predicate_no_predicate/test_build_actions_meta_predicate_static_skip_all/timestamp_predicate_skipping/unsupported_predicate_skipping/test_skip_stats_disables_data_skipping/test_with_stats_last_call_wins/test_default_stats_options_no_struct_output/scan_builder_validates_predicate_and_stats_columns/test_scan_metadata_with_nonexistent_stats_columns` | +| `timestamp-truncation-stats` | data/ | `id: long, ts_col: timestamp` | v1/v2 | | Three Spark-written files with sub-millisecond timestamps. File 2 has max truncated from 4.000500s to 4.000s in JSON stats. Verifies max stat pruning with truncation adjustment. | `read.rs::timestamp_max_stat_pruning_with_real_table` | | `parquet_row_group_skipping` | data/ | `bool, chrono{date32,timestamp,timestamp_ntz}, numeric{decimals{...},floats{...},ints{...}}, varlen{binary,utf8}` | v3/v7 | r:`timestampNtz` w:`timestampNtz`, `dataSkippingNumIndexedCols=0` | Deep nested structs, all types. Delta stats disabled; tests parquet-level row group skipping. | `read.rs::predicate_references_invalid_missing_column`, `scan/tests.rs::test_replay_for_scan_metadata/test_data_row_group_skipping/test_missing_column_row_group_skipping`, `set_transaction.rs::test_replay_for_app_ids`, `parquet_row_group_skipping/tests.rs`, `protocol_metadata_replay.rs` | +| `v1-single-part-struct-stats-only` | data/ | `id: long, value: string` | v3/v7 | r:`deletionVectors` w:`deletionVectors,appendOnly,invariants`, `writeStatsAsStruct=true`, `writeStatsAsJson=false` | 5 rows across 5 commits, single-part V1 checkpoint at v5 with `stats_parsed` only (no JSON stats). | `read.rs::stats_parsed_skipping::v1_single_part`, `scan/tests.rs::test_checkpoint_stats_projection_matches_requested_output/test_all_struct_parses_json_commit_stats` | +| `v1-multi-part-struct-stats-only` | data/ | `id: long, value: string` | v3/v7 | r:`deletionVectors` w:`deletionVectors,appendOnly,invariants`, `writeStatsAsStruct=true`, `writeStatsAsJson=false` | 5 rows across 5 commits, 3-part V1 checkpoint at v5 with `stats_parsed` only (no JSON stats). | `read.rs::stats_parsed_skipping::v1_multi_part` | +| `v1-multi-part-partitioned-struct-stats-only` | data/ | `id: long, value: string, part: int` partitioned by `part` | v3/v7 | r:`deletionVectors` w:`deletionVectors,appendOnly,invariants`, `writeStatsAsStruct=true`, `writeStatsAsJson=false` | 5 rows across 5 commits (part=i%3), 3-part V1 checkpoint at v5 with `partitionValues_parsed`. | `read.rs::partition_values_parsed_skipping` | +| `v2-parquet-sidecars-struct-stats-only` | data/ | `id: long, value: string` | v3/v7 | r:`deletionVectors,v2Checkpoint` w:`deletionVectors,v2Checkpoint,appendOnly,invariants`, `writeStatsAsStruct=true`, `writeStatsAsJson=false`, `checkpointPolicy=v2` | 5 rows across 5 commits, UUID-named V2 parquet checkpoint at v5 with sidecars containing `stats_parsed`. | `read.rs::stats_parsed_skipping::v2_parquet_sidecars`, `scan/tests.rs::test_checkpoint_stats_projection_matches_requested_output` | +| `v2-json-sidecars-struct-stats-only` | data/ | `id: long, value: string` | v3/v7 | r:`deletionVectors,v2Checkpoint` w:`deletionVectors,v2Checkpoint,appendOnly,invariants`, `writeStatsAsStruct=true`, `writeStatsAsJson=false`, `checkpointPolicy=v2` | 5 rows across 5 commits, UUID-named V2 JSON checkpoint at v5 with sidecars containing `stats_parsed`. | `read.rs::stats_parsed_skipping::v2_json_sidecars` | +| `v2-classic-parquet-struct-stats-only` | data/ | `id: long, value: string` | v3/v7 | r:`deletionVectors` w:`deletionVectors,appendOnly,invariants`, `writeStatsAsStruct=true`, `writeStatsAsJson=false`, `checkpointPolicy=classic` | 5 rows across 5 commits, classic-named V2 parquet checkpoint at v5 with `stats_parsed`. | `read.rs::stats_parsed_skipping::v2_classic_parquet` | | `mixed-nulls` | data/ | `id: long, part: long, value: string, n: string` partitioned by `part` | v1/v2 | | part=0: `n` all null. part=1: no nulls. part=2: mixed nulls. | `read.rs::mixed_null/mixed_not_null` | | `stats-writing-all-types` | data/stats-writing-all-types/delta/ | 16 columns: all primitives + array, map, nested struct | v3/v7 | r:`timestampNtz,columnMapping` w:`timestampNtz,columnMapping`, `columnMapping.mode=name` | Verifies stats collection matches Spark output | `default/stats.rs::test_collect_stats_matches_spark` | | `data-skipping-basic-stats-all-types` | golden_data/ | `as_int: int, as_long: long, as_byte: byte, as_short: short, as_float: float, as_double: double, as_string: string, as_date: date, as_timestamp: timestamp, as_big_decimal: decimal(1,0)` | v1/v2 | | Basic stats for all types | `golden_tables.rs::golden_test!` | @@ -145,8 +152,8 @@ The golden V2 checkpoint tables have a different protocol (v1/v2, no table featu | Table | Location | Schema | Protocol (R/W) | Features | Description | Tests | |-------|----------|--------|----------|----------|-------------|-------| -| `crc-full` | data/ | `id: long` | v3/v7 | r:`deletionVectors` w:`domainMetadata,deletionVectors,rowTracking` | Full CRC with `allFiles` (10 files), 2 `setTransactions`, `domainMetadata`, `fileSizeHistogram` | `crc/reader.rs::test_read_crc`, `crc/lazy.rs::test_lazy_crc` | -| `crc-malformed` | data/ | N/A | N/A | | CRC file contains only `"malformed"` | `crc/lazy.rs::test_lazy_crc_malformed_file`, `crc/reader.rs` error test | +| `crc-full` | data/ | `id: long` | v3/v7 | r:`deletionVectors` w:`domainMetadata,deletionVectors,rowTracking` | Full CRC with `allFiles` (10 files), 2 `setTransactions`, `domainMetadata`, `fileSizeHistogram` | `crc/reader.rs::test_read_crc_file`, `metrics/snapshot_load.rs::snapshot_with_crc_at_target_version_skips_json_replay` | +| `crc-malformed` | data/ | N/A | N/A | | CRC file contains only `"malformed"` | `crc/reader.rs::test_read_malformed_crc_file_emits_metric_then_fails` | ## Partitioning & Write Path diff --git a/kernel/tests/commit_range.rs b/kernel/tests/commit_range.rs new file mode 100644 index 0000000000..613c37eaa5 --- /dev/null +++ b/kernel/tests/commit_range.rs @@ -0,0 +1,84 @@ +//! Integration tests exercising `delta_kernel::commit_range` via its public API. + +use std::error::Error; +use std::path::PathBuf; +use std::sync::Arc; + +use delta_kernel::arrow::array::RecordBatch; +use delta_kernel::commit_range::{CommitAction, CommitOrdering, CommitRange, DeltaAction}; +use delta_kernel::engine::arrow_data::EngineDataArrowExt as _; +use delta_kernel::{DeltaResult, Engine, Snapshot, Version}; +use test_utils::create_default_engine; +use url::Url; + +fn setup_test(rel_path: &str) -> Result<(Url, Arc), Box> { + let abs = std::fs::canonicalize(PathBuf::from(rel_path))?; + let url = + Url::from_directory_path(&abs).map_err(|()| format!("could not build URL from {abs:?}"))?; + let engine = create_default_engine(&url)?; + Ok((url, engine)) +} + +/// Count rows with a non-null entry in each requested action column for a single commit. +/// `actions` must match what was passed to `CommitRange::commits` so the field indices align +/// with the emitted top-level columns. +fn count_action_rows( + commit: &CommitAction, + actions: &[DeltaAction], + engine: &dyn Engine, +) -> DeltaResult> { + let mut counts = vec![0usize; actions.len()]; + for batch_res in commit.get_actions(engine)? { + let rb: RecordBatch = batch_res?.try_into_record_batch()?; + for (idx, count) in counts.iter_mut().enumerate() { + let col = rb.column(idx); + *count += col.len() - col.null_count(); + } + } + Ok(counts) +} + +#[rstest::rstest] +#[case::ascending_default(None, 0, vec![(0, vec![1, 0]), (1, vec![1, 1])])] +#[case::descending( + Some(CommitOrdering::DescendingOrder), + 1, + vec![(1, vec![1, 1]), (0, vec![1, 0])], +)] +fn reads_all_commits_in_requested_order( + #[case] ordering: Option, + #[case] snapshot_version: Version, + #[case] expected_per_commit: Vec<(Version, Vec)>, +) -> Result<(), Box> { + let (url, engine) = setup_test("./tests/data/table-with-dv-small")?; + + let anchor_snapshot = Snapshot::builder_for(url.as_str()) + .at_version(snapshot_version) + .build(engine.as_ref())?; + + let mut builder = CommitRange::builder_for(url.as_str(), 0).with_end_version(1); + if let Some(order) = ordering { + builder = builder.with_ordering(order); + } + let range = builder.build(engine.as_ref())?; + + assert_eq!(range.start_version(), 0); + assert_eq!(range.end_version(), 1); + + let actions = [DeltaAction::Add, DeltaAction::Remove]; + let commits = range + .commits(engine.clone(), Some(anchor_snapshot), &actions)? + .collect::>>()?; + assert_eq!(commits.len(), expected_per_commit.len(), "commit count"); + + for (commit, (expected_version, expected_counts)) in + commits.into_iter().zip(expected_per_commit) + { + let version = commit.version(); + assert_eq!(version, expected_version, "version order"); + let counts = count_action_rows(&commit, &actions, engine.as_ref())?; + assert_eq!(counts, expected_counts, "v={version} action counts"); + } + + Ok(()) +} diff --git a/kernel/tests/crc.rs b/kernel/tests/crc.rs deleted file mode 100644 index 1b4e9713bb..0000000000 --- a/kernel/tests/crc.rs +++ /dev/null @@ -1,634 +0,0 @@ -//! Integration tests for CRC (version checksum) file-based APIs on Snapshot. - -use std::path::PathBuf; -use std::sync::Arc; - -use delta_kernel::arrow::array::{ArrayRef, Int32Array}; -use delta_kernel::committer::FileSystemCommitter; -use delta_kernel::crc::{Crc, FileStatsValidity}; -use delta_kernel::engine::default::DefaultEngineBuilder; -use delta_kernel::object_store::local::LocalFileSystem; -use delta_kernel::schema::{DataType, StructField, StructType}; -use delta_kernel::snapshot::{ChecksumWriteResult, FileStats, Snapshot}; -use delta_kernel::transaction::create_table::create_table; -use delta_kernel::transaction::data_layout::DataLayout; -use delta_kernel::{DeltaResult, Engine}; -use rstest::rstest; -use test_utils::{insert_data, test_table_setup}; - -// ============================================================================ -// File stats from CRC on disk -// ============================================================================ - -#[tokio::test] -async fn test_get_file_stats_from_crc() -> DeltaResult<()> { - let path = std::fs::canonicalize(PathBuf::from("./tests/data/crc-full/")).unwrap(); - let table_root = url::Url::from_directory_path(path).unwrap(); - - let store = Arc::new(LocalFileSystem::new()); - let engine = DefaultEngineBuilder::new(store).build(); - - let snapshot = Snapshot::builder_for(table_root).build(&engine)?; - assert_eq!(snapshot.version(), 0); - - let file_stats = snapshot.get_file_stats(&engine); - let expected = FileStats { - table_size_bytes: 5259, - num_files: 10, - }; - assert_eq!(file_stats, Some(expected)); - - Ok(()) -} - -#[tokio::test] -async fn test_get_file_stats_no_crc() -> DeltaResult<()> { - let (_temp_dir, table_path, engine) = test_table_setup()?; - - let schema = Arc::new(StructType::try_new(vec![ - StructField::new("id", DataType::INTEGER, false), - StructField::new("value", DataType::STRING, true), - ])?); - - let _ = create_table(&table_path, schema, "Test/1.0") - .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? - .commit(engine.as_ref())?; - - let table_url = delta_kernel::try_parse_uri(&table_path)?; - let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; - assert_eq!(snapshot.version(), 0); - - let file_stats = snapshot.get_file_stats(engine.as_ref()); - assert_eq!(file_stats, None); - - Ok(()) -} - -#[tokio::test] -async fn test_get_file_stats_crc_not_at_snapshot_version() -> DeltaResult<()> { - use test_utils::copy_directory; - - // ===== GIVEN ===== - let (_temp_dir, table_path, engine) = test_table_setup()?; - - // Copy crc-full table (has CRC at version 0) into the temp dir - let source_path = std::fs::canonicalize(PathBuf::from("./tests/data/crc-full/")).unwrap(); - copy_directory(&source_path, _temp_dir.path()).unwrap(); - - // Verify the table starts at version 0 with valid CRC stats - let snapshot = Snapshot::builder_for(table_path.clone()).build(engine.as_ref())?; - assert_eq!(snapshot.version(), 0); - assert!(snapshot.get_file_stats(engine.as_ref()).is_some()); - - // ===== WHEN ===== - // Empty commit to advance to version 1 (no new CRC file written) - let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())?; - let _ = txn.commit(engine.as_ref())?; - - // ===== THEN ===== - // Load a fresh snapshot at version 1 - let snapshot = Snapshot::builder_for(table_path).build(engine.as_ref())?; - assert_eq!(snapshot.version(), 1); - - // No CRC at version 1, so file stats should be None - let file_stats = snapshot.get_file_stats(engine.as_ref()); - assert_eq!(file_stats, None); - - Ok(()) -} - -// ============================================================================ -// CRC test visibility: get_current_crc_if_loaded_for_testing -// ============================================================================ - -#[tokio::test] -async fn test_get_current_crc_if_loaded_returns_loaded_crc() -> DeltaResult<()> { - // ===== GIVEN ===== - let path = std::fs::canonicalize(PathBuf::from("./tests/data/crc-full/")).unwrap(); - let table_root = url::Url::from_directory_path(path).unwrap(); - - let store = Arc::new(LocalFileSystem::new()); - let engine = DefaultEngineBuilder::new(store).build(); - - let snapshot = Snapshot::builder_for(table_root).build(&engine)?; - assert_eq!(snapshot.version(), 0); - - // ===== WHEN ===== - let crc = snapshot.get_current_crc_if_loaded_for_testing().unwrap(); - - // ===== THEN ===== - let file_stats = crc.file_stats().unwrap(); - assert_eq!(file_stats.table_size_bytes, 5259); - assert_eq!(file_stats.num_files, 10); - assert_eq!(crc.num_metadata, 1); - assert_eq!(crc.num_protocol, 1); - - // Protocol and metadata should match the snapshot's table configuration - assert_eq!(crc.protocol, *snapshot.table_configuration().protocol()); - assert_eq!(crc.metadata, *snapshot.table_configuration().metadata()); - - // Domain metadata - let dms = crc.domain_metadata.as_ref().unwrap(); - assert_eq!(dms.len(), 3); - assert!(dms.contains_key("delta.clustering")); - assert!(dms.contains_key("delta.rowTracking")); - assert!(dms.contains_key("myApp.metadata")); - - Ok(()) -} - -#[tokio::test] -async fn test_get_current_crc_if_loaded_returns_none_when_no_crc() -> DeltaResult<()> { - let (_temp_dir, table_path, engine) = test_table_setup()?; - - let schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "id", - DataType::INTEGER, - )])?); - - let _ = create_table(&table_path, schema, "Test/1.0") - .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? - .commit(engine.as_ref())?; - - let table_url = delta_kernel::try_parse_uri(&table_path)?; - let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; - assert_eq!(snapshot.version(), 0); - - // No CRC file exists, so get_current_crc_if_loaded_for_testing should return None - assert!(snapshot.get_current_crc_if_loaded_for_testing().is_none()); - - Ok(()) -} - -// ============================================================================ -// Post-commit CRC existence: does a CRC exist on the post-commit snapshot? -// ============================================================================ - -fn create_table_and_commit( - table_path: &str, - engine: &dyn delta_kernel::Engine, -) -> DeltaResult { - let schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "id", - DataType::INTEGER, - )])?); - let txn = create_table(table_path, schema, "test_engine") - .with_data_layout(DataLayout::clustered(["id"])) - .build(engine, Box::new(FileSystemCommitter::new()))? - .with_domain_metadata("zip".to_string(), "zap0".to_string()); - - Ok(txn.commit(engine)?.unwrap_committed()) -} - -#[tokio::test] -async fn test_create_table_produces_post_commit_crc() -> DeltaResult<()> { - // ===== GIVEN / WHEN: Create the table ===== - let (_temp_dir, table_path, engine) = test_table_setup()?; - let committed = create_table_and_commit(&table_path, engine.as_ref())?; - - // ===== THEN: should have CRC at v0 ===== - assert_eq!(committed.commit_version(), 0); - let snapshot = committed.post_commit_snapshot().unwrap(); - let crc = snapshot.get_current_crc_if_loaded_for_testing().unwrap(); - - let file_stats = crc.file_stats().unwrap(); - assert_eq!(file_stats.num_files, 0); - assert_eq!(file_stats.table_size_bytes, 0); - assert_eq!(crc.num_metadata, 1); - assert_eq!(crc.num_protocol, 1); - assert_eq!(crc.protocol, *snapshot.table_configuration().protocol()); - assert_eq!(crc.metadata, *snapshot.table_configuration().metadata()); - let dms = crc.domain_metadata.as_ref().unwrap(); - assert_eq!(dms["zip"].configuration(), "zap0"); - - Ok(()) -} - -#[rstest] -#[case::with_in_memory_crc(true)] -#[case::without_crc(false)] -#[tokio::test] -async fn test_post_commit_crc_chains_only_if_read_snapshot_has_crc( - #[case] use_post_commit_snapshot: bool, -) -> DeltaResult<()> { - let (_temp_dir, table_path, engine) = test_table_setup()?; - let create_committed = create_table_and_commit(&table_path, engine.as_ref())?; - - let read_snapshot = if use_post_commit_snapshot { - // Post-commit snapshot has in-memory CRC from the previous commit. - create_committed.post_commit_snapshot().unwrap().clone() - } else { - // Fresh-from-disk snapshot has no CRC (no .crc file on disk). - Snapshot::builder_for(table_path).build(engine.as_ref())? - }; - assert_eq!( - read_snapshot - .get_current_crc_if_loaded_for_testing() - .is_some(), - use_post_commit_snapshot - ); - - let committed = read_snapshot - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_operation("WRITE".to_string()) - .with_domain_metadata("zip".to_string(), "zap1".to_string()) - .commit(engine.as_ref())? - .unwrap_committed(); - - // The new post-commit snapshot should only have a CRC if the read snapshot had one. - assert_eq!(committed.commit_version(), 1); - assert_eq!( - committed - .post_commit_snapshot() - .unwrap() - .get_current_crc_if_loaded_for_testing() - .is_some(), - use_post_commit_snapshot - ); - - Ok(()) -} - -// ================================================================================ -// Post-commit CRC correctness: are the CRC fields accurate after write and reload? -// ================================================================================ - -/// Writes the in-memory CRC to disk, reloads a fresh snapshot, and asserts that the -/// round-tripped CRC matches the in-memory one. Returns the loaded CRC for further assertions. -fn write_and_verify_crc( - snapshot: &Snapshot, - table_path: &str, - engine: &dyn delta_kernel::Engine, -) -> Crc { - let crc_in_memory = snapshot.get_current_crc_if_loaded_for_testing().unwrap(); - snapshot.write_checksum(engine).unwrap(); - - let snapshot_fresh = Snapshot::builder_for(table_path).build(engine).unwrap(); - let crc_from_disk = snapshot_fresh - .get_current_crc_if_loaded_for_testing() - .unwrap(); - assert_eq!(crc_in_memory, crc_from_disk); - crc_from_disk.clone() -} - -#[tokio::test] -async fn test_post_commit_crc_tracks_file_stats_across_inserts() -> DeltaResult<()> { - let (_temp_dir, table_path, engine) = test_table_setup()?; - - // ===== GIVEN: Create the table ===== - let committed = create_table_and_commit(&table_path, engine.as_ref())?; - let snapshot_v0 = committed.post_commit_snapshot().unwrap().clone(); - - // ===== WHEN: Insert values 1..=10 ===== - let col1: ArrayRef = Arc::new(Int32Array::from((1..=10).collect::>())); - let committed = insert_data(snapshot_v0, &engine, vec![col1]) - .await? - .unwrap_committed(); - - // ===== THEN: should have CRC at v1 with right file stats ===== - assert_eq!(committed.commit_version(), 1); - let snapshot_v1 = committed.post_commit_snapshot().unwrap(); - let crc_v1 = write_and_verify_crc(snapshot_v1, &table_path, engine.as_ref()); - let stats_v1 = crc_v1.file_stats().unwrap(); - assert_eq!(stats_v1.num_files, 1); // <--- 1 file added - assert!(stats_v1.table_size_bytes > 0); // <--- size is non-zero - - // ===== WHEN: Insert values 11..=20 ===== - let col2: ArrayRef = Arc::new(Int32Array::from((11..=20).collect::>())); - let committed = insert_data(snapshot_v1.clone(), &engine, vec![col2]) - .await? - .unwrap_committed(); - - // ===== THEN: should have CRC at v2 with right file stats ===== - assert_eq!(committed.commit_version(), 2); - let snapshot_v2 = committed.post_commit_snapshot().unwrap(); - let crc_v2 = write_and_verify_crc(snapshot_v2, &table_path, engine.as_ref()); - let stats_v2 = crc_v2.file_stats().unwrap(); - assert_eq!(stats_v2.num_files, 2); // <--- 2 files added - assert!(stats_v2.table_size_bytes > stats_v1.table_size_bytes); // <--- size is greater than after first insert - - // ===== WHEN: Remove all files ===== - let scan = snapshot_v2.clone().scan_builder().build()?; - let mut txn = snapshot_v2 - .clone() - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_operation("DELETE".to_string()) - .with_data_change(true); - for sm in scan.scan_metadata(engine.as_ref())? { - txn.remove_files(sm?.scan_files); - } - let committed = txn.commit(engine.as_ref())?.unwrap_committed(); - - // ===== THEN: should have CRC at v3 with right file stats ===== - assert_eq!(committed.commit_version(), 3); - let snapshot_v3 = committed.post_commit_snapshot().unwrap(); - let crc_v3 = write_and_verify_crc(snapshot_v3, &table_path, engine.as_ref()); - let stats_v3 = crc_v3.file_stats().unwrap(); - assert_eq!(stats_v3.num_files, 0); // <--- 0 net file in the table - assert_eq!(stats_v3.table_size_bytes, 0); // <--- size is 0 - - Ok(()) -} - -#[tokio::test] -async fn test_post_commit_crc_tracks_domain_metadata_changes() -> DeltaResult<()> { - let (_temp_dir, table_path, engine) = test_table_setup()?; - - // ===== WHEN: CREATE TABLE with zip -> zap0 ===== - let committed = create_table_and_commit(&table_path, engine.as_ref())?; - let snapshot_v0 = committed.post_commit_snapshot().unwrap(); - - // ===== THEN: should have CRC at v0 with zip -> zap0 ===== - let crc_v0 = write_and_verify_crc(snapshot_v0, &table_path, engine.as_ref()); - let dms = crc_v0.domain_metadata.as_ref().unwrap(); - assert_eq!(dms["zip"].configuration(), "zap0"); - - // ===== WHEN: update zip -> zap1, add foo -> bar ===== - let txn = snapshot_v0 - .clone() - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_operation("WRITE".to_string()) - .with_domain_metadata("zip".to_string(), "zap1".to_string()) // <-- set to zap1 - .with_domain_metadata("foo".to_string(), "bar".to_string()); // <-- add foo - let committed = txn.commit(engine.as_ref())?.unwrap_committed(); - - // ===== THEN: should have CRC at v1 with zip -> zap1, foo -> bar ===== - let snapshot_v1 = committed.post_commit_snapshot().unwrap(); - let crc_v1 = write_and_verify_crc(snapshot_v1, &table_path, engine.as_ref()); - let dms = crc_v1.domain_metadata.as_ref().unwrap(); - assert_eq!(dms["zip"].configuration(), "zap1"); // <-- must be zap1 - assert_eq!(dms["foo"].configuration(), "bar"); // <-- must be bar - - // ===== WHEN: remove zip, keep foo ===== - let txn = snapshot_v1 - .clone() - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_operation("WRITE".to_string()) - .with_domain_metadata_removed("zip".to_string()); // <-- remove zip - let committed = txn.commit(engine.as_ref())?.unwrap_committed(); - - // ===== THEN: should have CRC at v2 with zip gone, foo still there ===== - let snapshot_v2 = committed.post_commit_snapshot().unwrap(); - let crc_v2 = write_and_verify_crc(snapshot_v2, &table_path, engine.as_ref()); - let dms = crc_v2.domain_metadata.as_ref().unwrap(); - assert!(!dms.contains_key("zip")); // <-- must be gone - assert_eq!(dms["foo"].configuration(), "bar"); // <-- must still be bar - - Ok(()) -} - -#[tokio::test] -async fn test_post_commit_crc_non_incremental_op_makes_file_stats_indeterminate() -> DeltaResult<()> -{ - let (_temp_dir, table_path, engine) = test_table_setup()?; - - // ===== GIVEN: Create table (v0) and insert data (v1) ===== - let committed = create_table_and_commit(&table_path, engine.as_ref())?; - let snapshot_v0 = committed.post_commit_snapshot().unwrap().clone(); - - let col: ArrayRef = Arc::new(Int32Array::from(vec![1, 2, 3])); - let committed = insert_data(snapshot_v0, &engine, vec![col]) - .await? - .unwrap_committed(); - let snapshot_v1 = committed.post_commit_snapshot().unwrap(); - - // ===== WHEN: Commit a non-incremental operation (ANALYZE STATS) ===== - let committed = snapshot_v1 - .clone() - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_operation("ANALYZE STATS".to_string()) - .commit(engine.as_ref())? - .unwrap_committed(); - - // ===== THEN: CRC at v2 has indeterminate file stats ===== - assert_eq!(committed.commit_version(), 2); - let snapshot_v2 = committed.post_commit_snapshot().unwrap(); - let crc_v2 = snapshot_v2.get_current_crc_if_loaded_for_testing().unwrap(); - assert_eq!(crc_v2.file_stats_validity, FileStatsValidity::Indeterminate); - - Ok(()) -} - -// ============================================================================ -// Write checksum to disk -// ============================================================================ - -#[tokio::test] -async fn test_write_checksum_success_simple() -> DeltaResult<()> { - let (_temp_dir, table_path, engine) = test_table_setup()?; - let committed = create_table_and_commit(&table_path, engine.as_ref())?; - let snapshot = committed.post_commit_snapshot().unwrap(); - - let result = snapshot.write_checksum(engine.as_ref())?; - assert_eq!(result, ChecksumWriteResult::Written); - - // Verify the CRC file is readable by loading a fresh snapshot from disk - let fresh_snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; - assert!(fresh_snapshot - .get_current_crc_if_loaded_for_testing() - .is_some()); - - Ok(()) -} - -#[rstest] -#[case::same_snapshot(false)] -#[case::fresh_snapshot(true)] -#[tokio::test] -async fn test_write_checksum_double_write_returns_already_exists( - #[case] reload_snapshot: bool, -) -> DeltaResult<()> { - let (_temp_dir, table_path, engine) = test_table_setup()?; - let committed = create_table_and_commit(&table_path, engine.as_ref())?; - let snapshot = committed.post_commit_snapshot().unwrap(); - - let first = snapshot.write_checksum(engine.as_ref())?; - assert_eq!(first, ChecksumWriteResult::Written); - - let second = if reload_snapshot { - let fresh = Snapshot::builder_for(&table_path).build(engine.as_ref())?; - fresh.write_checksum(engine.as_ref())? - } else { - snapshot.write_checksum(engine.as_ref())? - }; - assert_eq!(second, ChecksumWriteResult::AlreadyExists); - - Ok(()) -} - -#[tokio::test] -async fn test_write_checksum_with_no_in_memory_crc_returns_error() -> DeltaResult<()> { - let (_temp_dir, table_path, engine) = test_table_setup()?; - let _ = create_table_and_commit(&table_path, engine.as_ref())?; - - // Load from disk -- no CRC file on disk, so no in-memory CRC - let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; - - let result = snapshot.write_checksum(engine.as_ref()); - assert!(result.is_err()); - - Ok(()) -} - -#[tokio::test] -async fn test_in_memory_crc_chains_across_multiple_commits_then_writes() -> DeltaResult<()> { - let (_temp_dir, table_path, engine) = test_table_setup()?; - let committed = create_table_and_commit(&table_path, engine.as_ref())?; - let mut snapshot = committed.post_commit_snapshot().unwrap().clone(); - assert!(snapshot.get_current_crc_if_loaded_for_testing().is_some()); - - // Chain several commits without writing CRC to disk - for i in 0..5 { - let col: ArrayRef = Arc::new(Int32Array::from(vec![i])); - let committed = insert_data(snapshot, &engine, vec![col]) - .await? - .unwrap_committed(); - snapshot = committed.post_commit_snapshot().unwrap().clone(); - assert!( - snapshot.get_current_crc_if_loaded_for_testing().is_some(), - "in-memory CRC lost at commit {}", - committed.commit_version() - ); - } - - // Only now write the CRC -- should have accumulated all 5 inserts - assert_eq!(snapshot.version(), 5); - let crc = write_and_verify_crc(&snapshot, &table_path, engine.as_ref()); - let crc_stats = crc.file_stats().unwrap(); - assert_eq!(crc_stats.num_files, 5); - assert!(crc_stats.table_size_bytes > 0); - - Ok(()) -} - -// CRC should always write domainMetadata as an empty list (not omit the field) when there are -// no domain metadata actions, regardless of whether the feature is supported. -#[rstest] -#[case::dm_feature_supported(true)] -#[case::dm_feature_not_supported(false)] -#[tokio::test] -async fn test_write_checksum_with_no_dms_writes_empty_list( - #[case] dm_supported: bool, -) -> DeltaResult<()> { - use std::collections::HashMap; - - let (_temp_dir, table_path, engine) = test_table_setup()?; - - let schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "id", - DataType::INTEGER, - )])?); - - let mut builder = create_table(&table_path, schema, "test_engine"); - if dm_supported { - let properties = HashMap::from([( - "delta.feature.domainMetadata".to_string(), - "supported".to_string(), - )]); - builder = builder.with_table_properties(properties); - } - let committed = builder - .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? - .commit(engine.as_ref())? - .unwrap_committed(); - - let snapshot = committed.post_commit_snapshot().unwrap(); - assert!(snapshot - .get_all_domain_metadata(engine.as_ref())? - .is_empty()); - let crc = write_and_verify_crc(snapshot, &table_path, engine.as_ref()); - assert_eq!(crc.domain_metadata, Some(Default::default())); - - Ok(()) -} - -// ============================================================================ -// Domain metadata CRC fast path -// ============================================================================ - -/// Engine that panics if any handler is accessed. -struct FailingEngine; - -impl Engine for FailingEngine { - fn evaluation_handler(&self) -> Arc { - unimplemented!() - } - fn storage_handler(&self) -> Arc { - unimplemented!() - } - fn json_handler(&self) -> Arc { - unimplemented!() - } - fn parquet_handler(&self) -> Arc { - unimplemented!() - } -} - -#[tokio::test] -async fn test_get_domain_metadata_with_crc_skips_log_replay() -> DeltaResult<()> { - let (_temp_dir, table_path, engine) = test_table_setup()?; - - // v0: CREATE TABLE with zip -> zap0 (and clustering DM from create_table_and_commit) - let committed = create_table_and_commit(&table_path, engine.as_ref())?; - let snapshot_v0 = committed.post_commit_snapshot().unwrap(); - - // v1: update zip -> zap1, add foo -> bar - let committed = snapshot_v0 - .clone() - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_operation("WRITE".to_string()) - .with_domain_metadata("zip".to_string(), "zap1".to_string()) - .with_domain_metadata("foo".to_string(), "bar".to_string()) - .commit(engine.as_ref())? - .unwrap_committed(); - - // Asserts domain metadata on any snapshot, regardless of how it was loaded. - let assert_domain_metadata = |snapshot: &Snapshot, engine: &dyn delta_kernel::Engine| { - assert_eq!( - snapshot.get_domain_metadata("zip", engine).unwrap(), - Some("zap1".to_string()) - ); - assert_eq!( - snapshot.get_domain_metadata("foo", engine).unwrap(), - Some("bar".to_string()) - ); - assert!(snapshot - .get_domain_metadata_internal("delta.clustering", engine) - .unwrap() - .is_some()); - assert_eq!( - snapshot - .get_domain_metadatas_internal(engine, None) - .unwrap() - .len(), - 3 - ); - }; - - // Case 1: Post-commit snapshot with in-memory CRC => DM loaded from CRC (fast path). - // Use NoJsonReadsEngine to prove no log replay occurs. - let post_commit_snapshot = committed.post_commit_snapshot().unwrap(); - assert!(post_commit_snapshot - .get_current_crc_if_loaded_for_testing() - .is_some()); - assert_domain_metadata(post_commit_snapshot, &FailingEngine); - - // Case 2: Fresh snapshot loaded from disk, no CRC file => DM loaded via log replay (slow path) - let fresh_snapshot_no_crc = Snapshot::builder_for(&table_path).build(engine.as_ref())?; - assert!(fresh_snapshot_no_crc - .get_current_crc_if_loaded_for_testing() - .is_none()); - assert_domain_metadata(&fresh_snapshot_no_crc, engine.as_ref()); - - // Case 3: Write CRC to disk, then reload fresh snapshot => DM loaded from CRC (fast path) - // Use NoJsonReadsEngine to prove no log replay occurs. - post_commit_snapshot.write_checksum(engine.as_ref())?; - - let fresh_snapshot_with_crc = Snapshot::builder_for(&table_path).build(engine.as_ref())?; - assert!(fresh_snapshot_with_crc - .get_current_crc_if_loaded_for_testing() - .is_some()); - assert_domain_metadata(&fresh_snapshot_with_crc, &FailingEngine); - - Ok(()) -} diff --git a/kernel/tests/create_table/main.rs b/kernel/tests/create_table/main.rs deleted file mode 100644 index 4d4dfb8aff..0000000000 --- a/kernel/tests/create_table/main.rs +++ /dev/null @@ -1,367 +0,0 @@ -//! Integration tests for the CreateTable API - -mod clustering; -mod column_mapping; -mod ctas; -mod ict; -mod timestamp_ntz; -mod variant; - -use std::sync::Arc; - -use delta_kernel::committer::FileSystemCommitter; -use delta_kernel::schema::{DataType, StructField, StructType}; -use delta_kernel::snapshot::Snapshot; -use delta_kernel::table_features::{ - TableFeature, TABLE_FEATURES_MIN_READER_VERSION, TABLE_FEATURES_MIN_WRITER_VERSION, -}; -use delta_kernel::table_properties::TableProperties; -use delta_kernel::transaction::create_table::{create_table, CreateTableTransaction}; -use delta_kernel::DeltaResult; -use serde_json::Value; -use test_utils::{assert_result_error_with_message, test_table_setup}; - -/// Helper to create a simple two-column schema for tests. -/// Shared with sub-modules. -pub(crate) fn simple_schema() -> DeltaResult> { - Ok(Arc::new(StructType::try_new(vec![ - StructField::new("id", DataType::INTEGER, false), - StructField::new("value", DataType::STRING, true), - ])?)) -} - -#[tokio::test] -async fn test_create_simple_table() -> DeltaResult<()> { - let (_temp_dir, table_path, engine) = test_table_setup()?; - - // Create schema for an events table - let schema = Arc::new(StructType::try_new(vec![ - StructField::new("event_id", DataType::LONG, false), - StructField::new("user_id", DataType::LONG, false), - StructField::new("event_type", DataType::STRING, false), - StructField::new("timestamp", DataType::TIMESTAMP, false), - StructField::new("properties", DataType::STRING, true), - ])?); - - // Create table using new API - let _ = create_table(&table_path, schema.clone(), "DeltaKernel-RS/0.17.0") - .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? - .commit(engine.as_ref())?; - - // Verify table was created - let table_url = delta_kernel::try_parse_uri(&table_path)?; - let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; - - assert_eq!(snapshot.version(), 0); - assert_eq!(snapshot.schema().fields().len(), 5); - - // Verify protocol versions via snapshot - let protocol = snapshot.table_configuration().protocol(); - assert_eq!( - protocol.min_reader_version(), - TABLE_FEATURES_MIN_READER_VERSION - ); - assert_eq!( - protocol.min_writer_version(), - TABLE_FEATURES_MIN_WRITER_VERSION - ); - // Verify no reader/writer features are set (empty for table features mode) - assert!(protocol.reader_features().is_some_and(|f| f.is_empty())); - assert!(protocol.writer_features().is_some_and(|f| f.is_empty())); - - // Verify no table properties are set via public API - assert_eq!(snapshot.table_properties(), &TableProperties::default()); - - // Verify schema field names - let field_names: Vec<_> = snapshot - .schema() - .fields() - .map(|f| f.name().to_string()) - .collect(); - assert!(field_names.contains(&"event_id".to_string())); - assert!(field_names.contains(&"user_id".to_string())); - assert!(field_names.contains(&"event_type".to_string())); - assert!(field_names.contains(&"timestamp".to_string())); - assert!(field_names.contains(&"properties".to_string())); - - Ok(()) -} - -#[tokio::test] -async fn test_create_table_with_user_domain_metadata() -> DeltaResult<()> { - let (_temp_dir, table_path, engine) = test_table_setup()?; - - let schema = simple_schema()?; - - // Create table with domainMetadata feature enabled - let txn = create_table(&table_path, schema, "Test/1.0") - .with_table_properties([("delta.feature.domainMetadata", "supported")]) - .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))?; - - // Add user domain metadata during table creation - let domain = "app.settings"; - let config = r#"{"version": 1, "enabled": true}"#; - - let _ = txn - .with_domain_metadata(domain.to_string(), config.to_string()) - .commit(engine.as_ref())?; - - // Load snapshot and verify domain metadata was persisted - let table_url = delta_kernel::try_parse_uri(&table_path)?; - let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; - - // Verify domainMetadata feature is enabled in protocol - assert!( - snapshot - .table_configuration() - .is_feature_supported(&TableFeature::DomainMetadata), - "DomainMetadata feature should be enabled" - ); - - // Verify domain metadata string was persisted correctly - let retrieved_config = snapshot.get_domain_metadata(domain, engine.as_ref())?; - assert_eq!( - retrieved_config, - Some(config.to_string()), - "Domain metadata should be persisted and retrievable" - ); - - // Parse and verify the JSON contents - let parsed: Value = serde_json::from_str(retrieved_config.as_ref().unwrap())?; - assert_eq!(parsed["version"], 1); - assert_eq!(parsed["enabled"], true); - - // Verify non-existent domain returns None - let missing = snapshot.get_domain_metadata("nonexistent.domain", engine.as_ref())?; - assert!(missing.is_none(), "Non-existent domain should return None"); - - Ok(()) -} - -#[tokio::test] -async fn test_create_table_already_exists() -> DeltaResult<()> { - let (_temp_dir, table_path, engine) = test_table_setup()?; - - // Create schema for a user profiles table - let schema = Arc::new(StructType::try_new(vec![ - StructField::new("user_id", DataType::LONG, false), - StructField::new("username", DataType::STRING, false), - StructField::new("email", DataType::STRING, false), - StructField::new("created_at", DataType::TIMESTAMP, false), - StructField::new("is_active", DataType::BOOLEAN, false), - ])?); - - // Create table first time - let _ = create_table(&table_path, schema.clone(), "UserManagementService/1.2.0") - .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? - .commit(engine.as_ref())?; - - // Try to create again - should fail at build time (table already exists) - let result = create_table(&table_path, schema.clone(), "UserManagementService/1.2.0") - .build(engine.as_ref(), Box::new(FileSystemCommitter::new())); - - assert_result_error_with_message(result, "already exists"); - - Ok(()) -} - -#[tokio::test] -async fn test_create_table_empty_schema_not_supported() -> DeltaResult<()> { - let (_temp_dir, table_path, engine) = test_table_setup()?; - - // Create empty schema - let schema = Arc::new(StructType::try_new(vec![])?); - - // Try to create table with empty schema - should fail at build time - let result = create_table(&table_path, schema, "InvalidApp/0.1.0") - .build(engine.as_ref(), Box::new(FileSystemCommitter::new())); - - assert_result_error_with_message(result, "cannot be empty"); - - Ok(()) -} - -#[tokio::test] -async fn test_create_table_log_actions() -> DeltaResult<()> { - let (_temp_dir, table_path, engine) = test_table_setup()?; - - // Create schema - let schema = Arc::new(StructType::try_new(vec![ - StructField::new("user_id", DataType::LONG, false), - StructField::new("action", DataType::STRING, false), - ])?); - - let engine_info = "AuditService/2.1.0"; - - // Create table - let _ = create_table(&table_path, schema, engine_info) - .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? - .commit(engine.as_ref())?; - - // Read the actual Delta log file - let log_file_path = format!("{table_path}/_delta_log/00000000000000000000.json"); - let log_contents = std::fs::read_to_string(&log_file_path).expect("Failed to read log file"); - - // Parse each line (each line is a separate JSON action) - let actions: Vec = log_contents - .lines() - .map(|line| serde_json::from_str(line).expect("Failed to parse JSON")) - .collect(); - - // Verify we have exactly 3 actions: CommitInfo, Protocol, Metadata - // CommitInfo is first to comply with ICT (In-Commit Timestamps) protocol requirements - assert_eq!( - actions.len(), - 3, - "Expected 3 actions (commitInfo, protocol, metaData), found {}", - actions.len() - ); - - // Verify CommitInfo action (first for ICT compliance) - let commit_info_action = &actions[0]; - assert!( - commit_info_action.get("commitInfo").is_some(), - "First action should be commitInfo" - ); - let commit_info = commit_info_action.get("commitInfo").unwrap(); - assert!( - commit_info.get("timestamp").is_some(), - "CommitInfo should have timestamp" - ); - assert!( - commit_info.get("engineInfo").is_some(), - "CommitInfo should have engineInfo" - ); - assert!( - commit_info.get("operation").is_some(), - "CommitInfo should have operation" - ); - assert_eq!( - commit_info["operation"], "CREATE TABLE", - "Operation should be CREATE TABLE" - ); - - // Verify Protocol action - let protocol_action = &actions[1]; - assert!( - protocol_action.get("protocol").is_some(), - "Second action should be protocol" - ); - let protocol = protocol_action.get("protocol").unwrap(); - assert_eq!( - protocol["minReaderVersion"], - TABLE_FEATURES_MIN_READER_VERSION - ); - assert_eq!( - protocol["minWriterVersion"], - TABLE_FEATURES_MIN_WRITER_VERSION - ); - - // Verify Metadata action - let metadata_action = &actions[2]; - assert!( - metadata_action.get("metaData").is_some(), - "Third action should be metaData" - ); - let metadata = metadata_action.get("metaData").unwrap(); - assert!(metadata.get("id").is_some(), "Metadata should have id"); - assert!( - metadata.get("schemaString").is_some(), - "Metadata should have schemaString" - ); - assert!( - metadata.get("createdTime").is_some(), - "Metadata should have createdTime" - ); - - // Additional CommitInfo verification (commit_info was already extracted from actions[0] above) - assert_eq!( - commit_info["engineInfo"], engine_info, - "CommitInfo should contain the engine info we provided" - ); - - assert!( - commit_info.get("txnId").is_some(), - "CommitInfo should have txnId" - ); - - // Verify kernelVersion is present - let kernel_version = commit_info.get("kernelVersion"); - assert!( - kernel_version.is_some(), - "CommitInfo should have kernelVersion" - ); - assert!( - kernel_version.unwrap().as_str().unwrap().starts_with("v"), - "Kernel version should start with 'v'" - ); - - Ok(()) -} - -/// Helper to create a `CreateTableTransaction` for tests. -fn create_test_create_table_txn() -> DeltaResult<( - Arc, - CreateTableTransaction, - tempfile::TempDir, -)> { - let (tempdir, table_path, engine) = test_table_setup()?; - let schema = Arc::new( - StructType::try_new(vec![ - StructField::nullable("id", DataType::INTEGER), - StructField::nullable("name", DataType::STRING), - ]) - .expect("valid schema"), - ); - let txn = create_table(&table_path, schema, "test_engine") - .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))?; - Ok((engine, txn, tempdir)) -} - -#[tokio::test] -async fn test_create_table_txn_debug() -> DeltaResult<()> { - let (_engine, txn, _tempdir) = create_test_create_table_txn()?; - let debug_str = format!("{txn:?}"); - assert!( - debug_str.contains("Transaction") && debug_str.contains("create_table"), - "Debug output should contain Transaction info: {debug_str}" - ); - Ok(()) -} - -#[test] -fn test_create_table_with_vacuum_protocol_check() -> DeltaResult<()> { - let (_temp_dir, table_path, engine) = test_table_setup()?; - - let _ = create_table(&table_path, simple_schema()?, "Test/1.0") - .with_table_properties([("delta.feature.vacuumProtocolCheck", "supported")]) - .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? - .commit(engine.as_ref())?; - - let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; - let table_config = snapshot.table_configuration(); - - assert!( - table_config.is_feature_supported(&TableFeature::VacuumProtocolCheck), - "vacuumProtocolCheck should be supported" - ); - assert!( - table_config.is_feature_enabled(&TableFeature::VacuumProtocolCheck), - "vacuumProtocolCheck should be enabled (AlwaysIfSupported)" - ); - let protocol = table_config.protocol(); - assert!( - protocol - .writer_features() - .is_some_and(|f| f.contains(&TableFeature::VacuumProtocolCheck)), - "vacuumProtocolCheck should be in writer features" - ); - assert!( - protocol - .reader_features() - .is_some_and(|f| f.contains(&TableFeature::VacuumProtocolCheck)), - "vacuumProtocolCheck should be in reader features" - ); - - Ok(()) -} diff --git a/kernel/tests/data/parsed-stats/_delta_log/00000000000000000000.json b/kernel/tests/data/parsed-stats/_delta_log/00000000000000000000.json index 57a5d1011f..b640c269d2 100644 --- a/kernel/tests/data/parsed-stats/_delta_log/00000000000000000000.json +++ b/kernel/tests/data/parsed-stats/_delta_log/00000000000000000000.json @@ -1,4 +1,4 @@ {"protocol":{"minReaderVersion":1,"minWriterVersion":2}} -{"metaData":{"id":"07b3a271-f0d6-4c3b-a300-3170847ceafe","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"age\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"salary\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.checkpointInterval":"3"},"createdTime":1765928274753}} -{"add":{"path":"part-00000-06d85a38-b141-479b-a315-4157335e9a11-c000.snappy.parquet","partitionValues":{},"size":3103,"modificationTime":1765928276060,"dataChange":true,"stats":"{\"numRecords\":100,\"minValues\":{\"id\":1,\"name\":\"name_1\",\"age\":20,\"salary\":50100},\"maxValues\":{\"id\":100,\"name\":\"name_99\",\"age\":69,\"salary\":60000},\"nullCount\":{\"id\":0,\"name\":0,\"age\":0,\"salary\":0}}"}} +{"metaData":{"id":"07b3a271-f0d6-4c3b-a300-3170847ceafe","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"age\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"salary\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"ts_col\",\"type\":\"timestamp\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.checkpointInterval":"3"},"createdTime":1765928274753}} +{"add":{"path":"part-00000-06d85a38-b141-479b-a315-4157335e9a11-c000.snappy.parquet","partitionValues":{},"size":3103,"modificationTime":1765928276060,"dataChange":true,"stats":"{\"numRecords\":100,\"minValues\":{\"id\":1,\"name\":\"name_1\",\"age\":20,\"salary\":50100,\"ts_col\":\"1970-01-01T00:00:01.000Z\"},\"maxValues\":{\"id\":100,\"name\":\"name_99\",\"age\":69,\"salary\":60000,\"ts_col\":\"1970-01-01T00:00:02.000Z\"},\"nullCount\":{\"id\":0,\"name\":0,\"age\":0,\"salary\":0,\"ts_col\":0}}"}} {"commitInfo":{"timestamp":1765928276979,"operation":"WRITE","operationParameters":{"mode":"Overwrite","partitionBy":"[]"},"isolationLevel":"Serializable","isBlindAppend":false,"operationMetrics":{"numFiles":"1","numOutputRows":"100","numOutputBytes":"3103"},"engineInfo":"Apache-Spark/3.2.4 Delta-Lake/2.1.0","txnId":"e3e4034b-7a93-4d2b-9de4-3fda01735de4"}} diff --git a/kernel/tests/data/parsed-stats/_delta_log/00000000000000000001.json b/kernel/tests/data/parsed-stats/_delta_log/00000000000000000001.json index 5f2597c27f..602885ea22 100644 --- a/kernel/tests/data/parsed-stats/_delta_log/00000000000000000001.json +++ b/kernel/tests/data/parsed-stats/_delta_log/00000000000000000001.json @@ -1,2 +1,2 @@ -{"add":{"path":"part-00000-065eae2b-b4ea-4708-bb30-0888f35cabdd-c000.snappy.parquet","partitionValues":{},"size":2826,"modificationTime":1765928279084,"dataChange":true,"stats":"{\"numRecords\":100,\"minValues\":{\"id\":101,\"name\":\"name_101\",\"age\":20,\"salary\":60100},\"maxValues\":{\"id\":200,\"name\":\"name_200\",\"age\":69,\"salary\":70000},\"nullCount\":{\"id\":0,\"name\":0,\"age\":0,\"salary\":0}}"}} +{"add":{"path":"part-00000-065eae2b-b4ea-4708-bb30-0888f35cabdd-c000.snappy.parquet","partitionValues":{},"size":2826,"modificationTime":1765928279084,"dataChange":true,"stats":"{\"numRecords\":100,\"minValues\":{\"id\":101,\"name\":\"name_101\",\"age\":20,\"salary\":60100,\"ts_col\":\"1970-01-01T00:00:03.000Z\"},\"maxValues\":{\"id\":200,\"name\":\"name_200\",\"age\":69,\"salary\":70000,\"ts_col\":\"1970-01-01T00:00:04.000Z\"},\"nullCount\":{\"id\":0,\"name\":0,\"age\":0,\"salary\":0,\"ts_col\":0}}"}} {"commitInfo":{"timestamp":1765928279091,"operation":"WRITE","operationParameters":{"mode":"Append","partitionBy":"[]"},"readVersion":0,"isolationLevel":"Serializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"100","numOutputBytes":"2826"},"engineInfo":"Apache-Spark/3.2.4 Delta-Lake/2.1.0","txnId":"6b5f28e4-a9cb-4c53-8b29-99e13d48a280"}} diff --git a/kernel/tests/data/parsed-stats/_delta_log/00000000000000000002.json b/kernel/tests/data/parsed-stats/_delta_log/00000000000000000002.json index 2829387d25..25620d532c 100644 --- a/kernel/tests/data/parsed-stats/_delta_log/00000000000000000002.json +++ b/kernel/tests/data/parsed-stats/_delta_log/00000000000000000002.json @@ -1,2 +1,2 @@ -{"add":{"path":"part-00000-a4c1def5-742e-4248-8c58-fc9f4018e43d-c000.snappy.parquet","partitionValues":{},"size":2828,"modificationTime":1765928279752,"dataChange":true,"stats":"{\"numRecords\":100,\"minValues\":{\"id\":201,\"name\":\"name_201\",\"age\":20,\"salary\":70100},\"maxValues\":{\"id\":300,\"name\":\"name_300\",\"age\":69,\"salary\":80000},\"nullCount\":{\"id\":0,\"name\":0,\"age\":0,\"salary\":0}}"}} +{"add":{"path":"part-00000-a4c1def5-742e-4248-8c58-fc9f4018e43d-c000.snappy.parquet","partitionValues":{},"size":2828,"modificationTime":1765928279752,"dataChange":true,"stats":"{\"numRecords\":100,\"minValues\":{\"id\":201,\"name\":\"name_201\",\"age\":20,\"salary\":70100,\"ts_col\":\"1970-01-01T00:00:05.000Z\"},\"maxValues\":{\"id\":300,\"name\":\"name_300\",\"age\":69,\"salary\":80000,\"ts_col\":\"1970-01-01T00:00:06.000Z\"},\"nullCount\":{\"id\":0,\"name\":0,\"age\":0,\"salary\":0,\"ts_col\":0}}"}} {"commitInfo":{"timestamp":1765928279759,"operation":"WRITE","operationParameters":{"mode":"Append","partitionBy":"[]"},"readVersion":1,"isolationLevel":"Serializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"100","numOutputBytes":"2828"},"engineInfo":"Apache-Spark/3.2.4 Delta-Lake/2.1.0","txnId":"23695a3f-264a-427d-a3f1-8e0eda4016f5"}} diff --git a/kernel/tests/data/parsed-stats/_delta_log/00000000000000000003.checkpoint.parquet b/kernel/tests/data/parsed-stats/_delta_log/00000000000000000003.checkpoint.parquet index 96e4924ab3..bde063d257 100644 Binary files a/kernel/tests/data/parsed-stats/_delta_log/00000000000000000003.checkpoint.parquet and b/kernel/tests/data/parsed-stats/_delta_log/00000000000000000003.checkpoint.parquet differ diff --git a/kernel/tests/data/parsed-stats/_delta_log/00000000000000000003.json b/kernel/tests/data/parsed-stats/_delta_log/00000000000000000003.json index 8fe41abeea..7b284aaf83 100644 --- a/kernel/tests/data/parsed-stats/_delta_log/00000000000000000003.json +++ b/kernel/tests/data/parsed-stats/_delta_log/00000000000000000003.json @@ -1,2 +1,2 @@ -{"add":{"path":"part-00000-2d9663e0-37c0-425e-98df-2e7141f9b5fb-c000.snappy.parquet","partitionValues":{},"size":2827,"modificationTime":1765928280364,"dataChange":true,"stats":"{\"numRecords\":100,\"minValues\":{\"id\":301,\"name\":\"name_301\",\"age\":20,\"salary\":80100},\"maxValues\":{\"id\":400,\"name\":\"name_400\",\"age\":69,\"salary\":90000},\"nullCount\":{\"id\":0,\"name\":0,\"age\":0,\"salary\":0}}"}} +{"add":{"path":"part-00000-2d9663e0-37c0-425e-98df-2e7141f9b5fb-c000.snappy.parquet","partitionValues":{},"size":2827,"modificationTime":1765928280364,"dataChange":true,"stats":"{\"numRecords\":100,\"minValues\":{\"id\":301,\"name\":\"name_301\",\"age\":20,\"salary\":80100,\"ts_col\":\"1970-01-01T00:00:07.000Z\"},\"maxValues\":{\"id\":400,\"name\":\"name_400\",\"age\":69,\"salary\":90000,\"ts_col\":\"1970-01-01T00:00:08.000Z\"},\"nullCount\":{\"id\":0,\"name\":0,\"age\":0,\"salary\":0,\"ts_col\":0}}"}} {"commitInfo":{"timestamp":1765928280369,"operation":"WRITE","operationParameters":{"mode":"Append","partitionBy":"[]"},"readVersion":2,"isolationLevel":"Serializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"100","numOutputBytes":"2827"},"engineInfo":"Apache-Spark/3.2.4 Delta-Lake/2.1.0","txnId":"88155a70-c671-4065-9a07-6e1eb3bbad68"}} diff --git a/kernel/tests/data/parsed-stats/_delta_log/00000000000000000004.json b/kernel/tests/data/parsed-stats/_delta_log/00000000000000000004.json index 7d87c00607..8073aaecf6 100644 --- a/kernel/tests/data/parsed-stats/_delta_log/00000000000000000004.json +++ b/kernel/tests/data/parsed-stats/_delta_log/00000000000000000004.json @@ -1,2 +1,2 @@ -{"add":{"path":"part-00000-40525115-50e1-4475-aae1-c8edc59274e6-c000.snappy.parquet","partitionValues":{},"size":2824,"modificationTime":1765928281356,"dataChange":true,"stats":"{\"numRecords\":100,\"minValues\":{\"id\":401,\"name\":\"name_401\",\"age\":20,\"salary\":90100},\"maxValues\":{\"id\":500,\"name\":\"name_500\",\"age\":69,\"salary\":100000},\"nullCount\":{\"id\":0,\"name\":0,\"age\":0,\"salary\":0}}"}} +{"add":{"path":"part-00000-40525115-50e1-4475-aae1-c8edc59274e6-c000.snappy.parquet","partitionValues":{},"size":2824,"modificationTime":1765928281356,"dataChange":true,"stats":"{\"numRecords\":100,\"minValues\":{\"id\":401,\"name\":\"name_401\",\"age\":20,\"salary\":90100,\"ts_col\":\"1970-01-01T00:00:09.000Z\"},\"maxValues\":{\"id\":500,\"name\":\"name_500\",\"age\":69,\"salary\":100000,\"ts_col\":\"1970-01-01T00:00:10.000Z\"},\"nullCount\":{\"id\":0,\"name\":0,\"age\":0,\"salary\":0,\"ts_col\":0}}"}} {"commitInfo":{"timestamp":1765928281364,"operation":"WRITE","operationParameters":{"mode":"Append","partitionBy":"[]"},"readVersion":3,"isolationLevel":"Serializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"100","numOutputBytes":"2824"},"engineInfo":"Apache-Spark/3.2.4 Delta-Lake/2.1.0","txnId":"53ed470b-13c7-437f-829d-9a6704a285a0"}} diff --git a/kernel/tests/data/parsed-stats/_delta_log/00000000000000000005.json b/kernel/tests/data/parsed-stats/_delta_log/00000000000000000005.json index 97cec9fa13..5a7fc3cd51 100644 --- a/kernel/tests/data/parsed-stats/_delta_log/00000000000000000005.json +++ b/kernel/tests/data/parsed-stats/_delta_log/00000000000000000005.json @@ -1,2 +1,2 @@ -{"add":{"path":"part-00000-c0cbdedc-d11b-4e4c-b6f2-5f40c55ef515-c000.snappy.parquet","partitionValues":{},"size":2828,"modificationTime":1765928282040,"dataChange":true,"stats":"{\"numRecords\":100,\"minValues\":{\"id\":501,\"name\":\"name_501\",\"age\":20,\"salary\":100100},\"maxValues\":{\"id\":600,\"name\":\"name_600\",\"age\":69,\"salary\":110000},\"nullCount\":{\"id\":0,\"name\":0,\"age\":0,\"salary\":0}}"}} +{"add":{"path":"part-00000-c0cbdedc-d11b-4e4c-b6f2-5f40c55ef515-c000.snappy.parquet","partitionValues":{},"size":2828,"modificationTime":1765928282040,"dataChange":true,"stats":"{\"numRecords\":100,\"minValues\":{\"id\":501,\"name\":\"name_501\",\"age\":20,\"salary\":100100,\"ts_col\":\"1970-01-01T00:00:11.000Z\"},\"maxValues\":{\"id\":600,\"name\":\"name_600\",\"age\":69,\"salary\":110000,\"ts_col\":\"1970-01-01T00:00:12.000Z\"},\"nullCount\":{\"id\":0,\"name\":0,\"age\":0,\"salary\":0,\"ts_col\":0}}"}} {"commitInfo":{"timestamp":1765928282047,"operation":"WRITE","operationParameters":{"mode":"Append","partitionBy":"[]"},"readVersion":4,"isolationLevel":"Serializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"100","numOutputBytes":"2828"},"engineInfo":"Apache-Spark/3.2.4 Delta-Lake/2.1.0","txnId":"e0681515-134d-427b-b265-da87d06ec8c3"}} diff --git a/kernel/tests/data/parsed-stats/_delta_log/_last_checkpoint b/kernel/tests/data/parsed-stats/_delta_log/_last_checkpoint index 136ec5b697..96b858e325 100644 --- a/kernel/tests/data/parsed-stats/_delta_log/_last_checkpoint +++ b/kernel/tests/data/parsed-stats/_delta_log/_last_checkpoint @@ -1 +1 @@ -{"version":3,"size":6,"sizeInBytes":12170,"numOfAddFiles":4,"checkpointSchema":{"type":"struct","fields":[{"name":"txn","type":{"type":"struct","fields":[{"name":"appId","type":"string","nullable":true,"metadata":{}},{"name":"version","type":"long","nullable":true,"metadata":{}},{"name":"lastUpdated","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"add","type":{"type":"struct","fields":[{"name":"path","type":"string","nullable":true,"metadata":{}},{"name":"partitionValues","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"size","type":"long","nullable":true,"metadata":{}},{"name":"modificationTime","type":"long","nullable":true,"metadata":{}},{"name":"dataChange","type":"boolean","nullable":true,"metadata":{}},{"name":"tags","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"stats","type":"string","nullable":true,"metadata":{}},{"name":"stats_parsed","type":{"type":"struct","fields":[{"name":"numRecords","type":"long","nullable":true,"metadata":{}},{"name":"minValues","type":{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"age","type":"long","nullable":true,"metadata":{}},{"name":"salary","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"maxValues","type":{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"age","type":"long","nullable":true,"metadata":{}},{"name":"salary","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"nullCount","type":{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"name","type":"long","nullable":true,"metadata":{}},{"name":"age","type":"long","nullable":true,"metadata":{}},{"name":"salary","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"remove","type":{"type":"struct","fields":[{"name":"path","type":"string","nullable":true,"metadata":{}},{"name":"deletionTimestamp","type":"long","nullable":true,"metadata":{}},{"name":"dataChange","type":"boolean","nullable":true,"metadata":{}},{"name":"extendedFileMetadata","type":"boolean","nullable":true,"metadata":{}},{"name":"partitionValues","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"size","type":"long","nullable":true,"metadata":{}},{"name":"tags","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"metaData","type":{"type":"struct","fields":[{"name":"id","type":"string","nullable":true,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"description","type":"string","nullable":true,"metadata":{}},{"name":"format","type":{"type":"struct","fields":[{"name":"provider","type":"string","nullable":true,"metadata":{}},{"name":"options","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"schemaString","type":"string","nullable":true,"metadata":{}},{"name":"partitionColumns","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"configuration","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"createdTime","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"protocol","type":{"type":"struct","fields":[{"name":"minReaderVersion","type":"integer","nullable":true,"metadata":{}},{"name":"minWriterVersion","type":"integer","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}}]}} +{"version":3,"size":6,"sizeInBytes":22972,"numOfAddFiles":4} diff --git a/kernel/tests/data/timestamp-truncation-stats/_delta_log/00000000000000000000.json b/kernel/tests/data/timestamp-truncation-stats/_delta_log/00000000000000000000.json new file mode 100644 index 0000000000..0ffa5fc26d --- /dev/null +++ b/kernel/tests/data/timestamp-truncation-stats/_delta_log/00000000000000000000.json @@ -0,0 +1,4 @@ +{"protocol":{"minReaderVersion":1,"minWriterVersion":2}} +{"metaData":{"id":"95c116f6-c389-410c-bf6d-269fa789ec46","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"ts_col\",\"type\":\"timestamp\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.checkpointInterval":"100"},"createdTime":1774497261439}} +{"add":{"path":"part-00000-a234b260-4c3a-46cc-a9f2-67b5dc46e82b-c000.snappy.parquet","partitionValues":{},"size":725,"modificationTime":1774497263192,"dataChange":true,"stats":"{\"numRecords\":2,\"minValues\":{\"id\":1,\"ts_col\":\"1970-01-01T00:00:01.000Z\"},\"maxValues\":{\"id\":2,\"ts_col\":\"1970-01-01T00:00:02.000Z\"},\"nullCount\":{\"id\":0,\"ts_col\":0}}"}} +{"commitInfo":{"timestamp":1774497264199,"operation":"WRITE","operationParameters":{"mode":"Overwrite","partitionBy":"[]"},"isolationLevel":"Serializable","isBlindAppend":false,"operationMetrics":{"numFiles":"1","numOutputRows":"2","numOutputBytes":"725"},"engineInfo":"Apache-Spark/3.2.4 Delta-Lake/2.0.2","txnId":"a39707c6-e539-441b-8ae8-5016810f5010"}} diff --git a/kernel/tests/data/timestamp-truncation-stats/_delta_log/00000000000000000001.json b/kernel/tests/data/timestamp-truncation-stats/_delta_log/00000000000000000001.json new file mode 100644 index 0000000000..8c00dccefe --- /dev/null +++ b/kernel/tests/data/timestamp-truncation-stats/_delta_log/00000000000000000001.json @@ -0,0 +1,2 @@ +{"add":{"path":"part-00000-8be8dbf9-6bbb-481c-9ff4-31146eb021e8-c000.snappy.parquet","partitionValues":{},"size":725,"modificationTime":1774497267308,"dataChange":true,"stats":"{\"numRecords\":2,\"minValues\":{\"id\":3,\"ts_col\":\"1970-01-01T00:00:03.000Z\"},\"maxValues\":{\"id\":4,\"ts_col\":\"1970-01-01T00:00:04.000Z\"},\"nullCount\":{\"id\":0,\"ts_col\":0}}"}} +{"commitInfo":{"timestamp":1774497267317,"operation":"WRITE","operationParameters":{"mode":"Append","partitionBy":"[]"},"readVersion":0,"isolationLevel":"Serializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"2","numOutputBytes":"725"},"engineInfo":"Apache-Spark/3.2.4 Delta-Lake/2.0.2","txnId":"2e5fdfd7-f775-4c8e-b8a8-add9b64d01f7"}} diff --git a/kernel/tests/data/timestamp-truncation-stats/_delta_log/00000000000000000002.json b/kernel/tests/data/timestamp-truncation-stats/_delta_log/00000000000000000002.json new file mode 100644 index 0000000000..d980895f98 --- /dev/null +++ b/kernel/tests/data/timestamp-truncation-stats/_delta_log/00000000000000000002.json @@ -0,0 +1,2 @@ +{"add":{"path":"part-00000-f05ea4e8-94b0-4215-802d-d5050c89d584-c000.snappy.parquet","partitionValues":{},"size":725,"modificationTime":1774497269072,"dataChange":true,"stats":"{\"numRecords\":2,\"minValues\":{\"id\":5,\"ts_col\":\"1970-01-01T00:00:07.000Z\"},\"maxValues\":{\"id\":6,\"ts_col\":\"1970-01-01T00:00:08.000Z\"},\"nullCount\":{\"id\":0,\"ts_col\":0}}"}} +{"commitInfo":{"timestamp":1774497269078,"operation":"WRITE","operationParameters":{"mode":"Append","partitionBy":"[]"},"readVersion":1,"isolationLevel":"Serializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"2","numOutputBytes":"725"},"engineInfo":"Apache-Spark/3.2.4 Delta-Lake/2.0.2","txnId":"9866f6f1-12d3-4de3-82c8-3f6532b7de4f"}} diff --git a/kernel/tests/data/timestamp-truncation-stats/part-00000-8be8dbf9-6bbb-481c-9ff4-31146eb021e8-c000.snappy.parquet b/kernel/tests/data/timestamp-truncation-stats/part-00000-8be8dbf9-6bbb-481c-9ff4-31146eb021e8-c000.snappy.parquet new file mode 100644 index 0000000000..25c95ef74c Binary files /dev/null and b/kernel/tests/data/timestamp-truncation-stats/part-00000-8be8dbf9-6bbb-481c-9ff4-31146eb021e8-c000.snappy.parquet differ diff --git a/kernel/tests/data/timestamp-truncation-stats/part-00000-a234b260-4c3a-46cc-a9f2-67b5dc46e82b-c000.snappy.parquet b/kernel/tests/data/timestamp-truncation-stats/part-00000-a234b260-4c3a-46cc-a9f2-67b5dc46e82b-c000.snappy.parquet new file mode 100644 index 0000000000..ddf330df1d Binary files /dev/null and b/kernel/tests/data/timestamp-truncation-stats/part-00000-a234b260-4c3a-46cc-a9f2-67b5dc46e82b-c000.snappy.parquet differ diff --git a/kernel/tests/data/timestamp-truncation-stats/part-00000-f05ea4e8-94b0-4215-802d-d5050c89d584-c000.snappy.parquet b/kernel/tests/data/timestamp-truncation-stats/part-00000-f05ea4e8-94b0-4215-802d-d5050c89d584-c000.snappy.parquet new file mode 100644 index 0000000000..2fce1f6007 Binary files /dev/null and b/kernel/tests/data/timestamp-truncation-stats/part-00000-f05ea4e8-94b0-4215-802d-d5050c89d584-c000.snappy.parquet differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000000.crc.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000000.crc.crc new file mode 100644 index 0000000000..cefde913ff Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000000.crc.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000000.json.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000000.json.crc new file mode 100644 index 0000000000..213193f6c9 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000000.json.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000001.crc.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000001.crc.crc new file mode 100644 index 0000000000..4c579b15a7 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000001.crc.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000001.json.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000001.json.crc new file mode 100644 index 0000000000..318cf0aa26 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000001.json.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000002.crc.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000002.crc.crc new file mode 100644 index 0000000000..b45996f754 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000002.crc.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000002.json.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000002.json.crc new file mode 100644 index 0000000000..e686df603f Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000002.json.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000003.crc.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000003.crc.crc new file mode 100644 index 0000000000..1eadd7aebc Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000003.crc.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000003.json.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000003.json.crc new file mode 100644 index 0000000000..6a039f5615 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000003.json.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000004.crc.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000004.crc.crc new file mode 100644 index 0000000000..bf404d7de9 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000004.crc.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000004.json.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000004.json.crc new file mode 100644 index 0000000000..329ec52c07 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000004.json.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000001.0000000003.parquet.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000001.0000000003.parquet.crc new file mode 100644 index 0000000000..e0f48813c0 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000001.0000000003.parquet.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000002.0000000003.parquet.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000002.0000000003.parquet.crc new file mode 100644 index 0000000000..69c03ece05 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000002.0000000003.parquet.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000003.0000000003.parquet.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000003.0000000003.parquet.crc new file mode 100644 index 0000000000..cf5cef05c2 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000003.0000000003.parquet.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000005.crc.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000005.crc.crc new file mode 100644 index 0000000000..8dfb00a8e9 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000005.crc.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000005.json.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000005.json.crc new file mode 100644 index 0000000000..cc2cf61b53 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/.00000000000000000005.json.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/._last_checkpoint.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/._last_checkpoint.crc new file mode 100644 index 0000000000..a0c1389482 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/._last_checkpoint.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000000.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000000.crc new file mode 100644 index 0000000000..b90fe9f475 --- /dev/null +++ b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000000.crc @@ -0,0 +1 @@ +{"txnId":"16de0a5e-f2ce-4e96-8ad2-eb3a4a232324","tableSizeBytes":0,"numFiles":0,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"4a880820-e52f-437c-be8a-0551b00a16ee","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"part\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["part"],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774306407453},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[0,0,0,0,0,0,0,0,0,0]},"allFiles":[]} diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000000.json b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000000.json new file mode 100644 index 0000000000..98b3b69604 --- /dev/null +++ b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000000.json @@ -0,0 +1,3 @@ +{"commitInfo":{"timestamp":1774306407573,"operation":"CREATE TABLE","operationParameters":{"partitionBy":"[\"part\"]","clusterBy":"[]","description":null,"isManaged":"false","properties":"{\"delta.checkpoint.writeStatsAsStruct\":\"true\",\"delta.checkpoint.writeStatsAsJson\":\"false\",\"delta.checkpointInterval\":\"10\",\"delta.enableDeletionVectors\":\"true\"}","statsOnLoad":false},"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{},"tags":{"restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"16de0a5e-f2ce-4e96-8ad2-eb3a4a232324"}} +{"metaData":{"id":"4a880820-e52f-437c-be8a-0551b00a16ee","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"part\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["part"],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774306407453}} +{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]}} diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000001.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000001.crc new file mode 100644 index 0000000000..db07b623a9 --- /dev/null +++ b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000001.crc @@ -0,0 +1 @@ +{"txnId":"c55316e0-b463-49e0-b814-4e390332655d","tableSizeBytes":986,"numFiles":1,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"4a880820-e52f-437c-be8a-0551b00a16ee","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"part\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["part"],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774306407453},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[986,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[1,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"part=1/test%25file%25prefix-part-00000-ad2e4232-aae1-41ea-b46c-569d88253339.c000.snappy.parquet","partitionValues":{"part":"1"},"size":986,"modificationTime":1774306408636,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306408636000","MIN_INSERTION_TIME":"1774306408636000","MAX_INSERTION_TIME":"1774306408636000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000001.json b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000001.json new file mode 100644 index 0000000000..b034b2f071 --- /dev/null +++ b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000001.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774306408772,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":0,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"986"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"c55316e0-b463-49e0-b814-4e390332655d"}} +{"add":{"path":"part=1/test%25file%25prefix-part-00000-ad2e4232-aae1-41ea-b46c-569d88253339.c000.snappy.parquet","partitionValues":{"part":"1"},"size":986,"modificationTime":1774306408636,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306408636000","MIN_INSERTION_TIME":"1774306408636000","MAX_INSERTION_TIME":"1774306408636000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000002.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000002.crc new file mode 100644 index 0000000000..6dd901ca5b --- /dev/null +++ b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000002.crc @@ -0,0 +1 @@ +{"txnId":"98e9f38c-758c-425f-9455-14255391ed04","tableSizeBytes":1972,"numFiles":2,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"4a880820-e52f-437c-be8a-0551b00a16ee","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"part\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["part"],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774306407453},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[1972,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[2,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"part=2/test%25file%25prefix-part-00000-b101d239-17b8-4e26-9b34-5a43db693182.c000.snappy.parquet","partitionValues":{"part":"2"},"size":986,"modificationTime":1774306409768,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306409768000","MIN_INSERTION_TIME":"1774306409768000","MAX_INSERTION_TIME":"1774306409768000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"part=1/test%25file%25prefix-part-00000-ad2e4232-aae1-41ea-b46c-569d88253339.c000.snappy.parquet","partitionValues":{"part":"1"},"size":986,"modificationTime":1774306408636,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306408636000","MIN_INSERTION_TIME":"1774306408636000","MAX_INSERTION_TIME":"1774306408636000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000002.json b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000002.json new file mode 100644 index 0000000000..87051861e2 --- /dev/null +++ b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000002.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774306409894,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":1,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"986"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"98e9f38c-758c-425f-9455-14255391ed04"}} +{"add":{"path":"part=2/test%25file%25prefix-part-00000-b101d239-17b8-4e26-9b34-5a43db693182.c000.snappy.parquet","partitionValues":{"part":"2"},"size":986,"modificationTime":1774306409768,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306409768000","MIN_INSERTION_TIME":"1774306409768000","MAX_INSERTION_TIME":"1774306409768000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000003.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000003.crc new file mode 100644 index 0000000000..f8be8748c5 --- /dev/null +++ b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000003.crc @@ -0,0 +1 @@ +{"txnId":"00106389-a5e9-43b2-8175-f8fce2a9755c","tableSizeBytes":2957,"numFiles":3,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"4a880820-e52f-437c-be8a-0551b00a16ee","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"part\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["part"],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774306407453},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[2957,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[3,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"part=0/test%25file%25prefix-part-00000-dea8716a-0901-4b81-a9c9-fa6d76ea5ea9.c000.snappy.parquet","partitionValues":{"part":"0"},"size":985,"modificationTime":1774306410880,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306410880000","MIN_INSERTION_TIME":"1774306410880000","MAX_INSERTION_TIME":"1774306410880000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"part=1/test%25file%25prefix-part-00000-ad2e4232-aae1-41ea-b46c-569d88253339.c000.snappy.parquet","partitionValues":{"part":"1"},"size":986,"modificationTime":1774306408636,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306408636000","MIN_INSERTION_TIME":"1774306408636000","MAX_INSERTION_TIME":"1774306408636000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"part=2/test%25file%25prefix-part-00000-b101d239-17b8-4e26-9b34-5a43db693182.c000.snappy.parquet","partitionValues":{"part":"2"},"size":986,"modificationTime":1774306409768,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306409768000","MIN_INSERTION_TIME":"1774306409768000","MAX_INSERTION_TIME":"1774306409768000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000003.json b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000003.json new file mode 100644 index 0000000000..89b828542a --- /dev/null +++ b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000003.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774306411023,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":2,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"985"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"00106389-a5e9-43b2-8175-f8fce2a9755c"}} +{"add":{"path":"part=0/test%25file%25prefix-part-00000-dea8716a-0901-4b81-a9c9-fa6d76ea5ea9.c000.snappy.parquet","partitionValues":{"part":"0"},"size":985,"modificationTime":1774306410880,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306410880000","MIN_INSERTION_TIME":"1774306410880000","MAX_INSERTION_TIME":"1774306410880000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000004.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000004.crc new file mode 100644 index 0000000000..58b9846451 --- /dev/null +++ b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000004.crc @@ -0,0 +1 @@ +{"txnId":"4a3b18dd-2601-4c94-bd4e-98fc73495586","tableSizeBytes":3943,"numFiles":4,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"4a880820-e52f-437c-be8a-0551b00a16ee","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"part\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["part"],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774306407453},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[3943,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[4,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"part=1/test%25file%25prefix-part-00000-a7a1cef9-8cb8-4adb-b7ba-adc2d3c33f39.c000.snappy.parquet","partitionValues":{"part":"1"},"size":986,"modificationTime":1774306411956,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306411956000","MIN_INSERTION_TIME":"1774306411956000","MAX_INSERTION_TIME":"1774306411956000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"part=0/test%25file%25prefix-part-00000-dea8716a-0901-4b81-a9c9-fa6d76ea5ea9.c000.snappy.parquet","partitionValues":{"part":"0"},"size":985,"modificationTime":1774306410880,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306410880000","MIN_INSERTION_TIME":"1774306410880000","MAX_INSERTION_TIME":"1774306410880000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"part=2/test%25file%25prefix-part-00000-b101d239-17b8-4e26-9b34-5a43db693182.c000.snappy.parquet","partitionValues":{"part":"2"},"size":986,"modificationTime":1774306409768,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306409768000","MIN_INSERTION_TIME":"1774306409768000","MAX_INSERTION_TIME":"1774306409768000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"part=1/test%25file%25prefix-part-00000-ad2e4232-aae1-41ea-b46c-569d88253339.c000.snappy.parquet","partitionValues":{"part":"1"},"size":986,"modificationTime":1774306408636,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306408636000","MIN_INSERTION_TIME":"1774306408636000","MAX_INSERTION_TIME":"1774306408636000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000004.json b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000004.json new file mode 100644 index 0000000000..fd29f6f348 --- /dev/null +++ b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000004.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774306412085,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":3,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"986"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"4a3b18dd-2601-4c94-bd4e-98fc73495586"}} +{"add":{"path":"part=1/test%25file%25prefix-part-00000-a7a1cef9-8cb8-4adb-b7ba-adc2d3c33f39.c000.snappy.parquet","partitionValues":{"part":"1"},"size":986,"modificationTime":1774306411956,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306411956000","MIN_INSERTION_TIME":"1774306411956000","MAX_INSERTION_TIME":"1774306411956000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000001.0000000003.parquet b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000001.0000000003.parquet new file mode 100644 index 0000000000..8307d145f0 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000001.0000000003.parquet differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000002.0000000003.parquet b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000002.0000000003.parquet new file mode 100644 index 0000000000..8c83a24598 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000002.0000000003.parquet differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000003.0000000003.parquet b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000003.0000000003.parquet new file mode 100644 index 0000000000..8385c9b02a Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000003.0000000003.parquet differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000005.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000005.crc new file mode 100644 index 0000000000..9eb8d5de94 --- /dev/null +++ b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000005.crc @@ -0,0 +1 @@ +{"txnId":"d7bea328-c983-476b-b099-e17d3c1f2fc6","tableSizeBytes":4929,"numFiles":5,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"4a880820-e52f-437c-be8a-0551b00a16ee","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"part\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["part"],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774306407453},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[4929,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[5,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"part=1/test%25file%25prefix-part-00000-a7a1cef9-8cb8-4adb-b7ba-adc2d3c33f39.c000.snappy.parquet","partitionValues":{"part":"1"},"size":986,"modificationTime":1774306411956,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306411956000","MIN_INSERTION_TIME":"1774306411956000","MAX_INSERTION_TIME":"1774306411956000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"part=0/test%25file%25prefix-part-00000-dea8716a-0901-4b81-a9c9-fa6d76ea5ea9.c000.snappy.parquet","partitionValues":{"part":"0"},"size":985,"modificationTime":1774306410880,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306410880000","MIN_INSERTION_TIME":"1774306410880000","MAX_INSERTION_TIME":"1774306410880000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"part=1/test%25file%25prefix-part-00000-ad2e4232-aae1-41ea-b46c-569d88253339.c000.snappy.parquet","partitionValues":{"part":"1"},"size":986,"modificationTime":1774306408636,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306408636000","MIN_INSERTION_TIME":"1774306408636000","MAX_INSERTION_TIME":"1774306408636000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"part=2/test%25file%25prefix-part-00000-b101d239-17b8-4e26-9b34-5a43db693182.c000.snappy.parquet","partitionValues":{"part":"2"},"size":986,"modificationTime":1774306409768,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306409768000","MIN_INSERTION_TIME":"1774306409768000","MAX_INSERTION_TIME":"1774306409768000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"part=2/test%25file%25prefix-part-00000-03c40ea1-51b8-4636-9900-fb0885b47b87.c000.snappy.parquet","partitionValues":{"part":"2"},"size":986,"modificationTime":1774306413016,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":5,\"value\":\"value_5\"},\"maxValues\":{\"id\":5,\"value\":\"value_5\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306413016000","MIN_INSERTION_TIME":"1774306413016000","MAX_INSERTION_TIME":"1774306413016000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000005.json b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000005.json new file mode 100644 index 0000000000..f02e870b8a --- /dev/null +++ b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/00000000000000000005.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774306413146,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":4,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"986"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"d7bea328-c983-476b-b099-e17d3c1f2fc6"}} +{"add":{"path":"part=2/test%25file%25prefix-part-00000-03c40ea1-51b8-4636-9900-fb0885b47b87.c000.snappy.parquet","partitionValues":{"part":"2"},"size":986,"modificationTime":1774306413016,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":5,\"value\":\"value_5\"},\"maxValues\":{\"id\":5,\"value\":\"value_5\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306413016000","MIN_INSERTION_TIME":"1774306413016000","MAX_INSERTION_TIME":"1774306413016000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/_last_checkpoint b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/_last_checkpoint new file mode 100644 index 0000000000..28c4f37780 --- /dev/null +++ b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/_delta_log/_last_checkpoint @@ -0,0 +1 @@ +{"version":5,"size":7,"parts":3,"sizeInBytes":58606,"numOfAddFiles":5,"checkpointSchema":{"type":"struct","fields":[{"name":"txn","type":{"type":"struct","fields":[{"name":"appId","type":"string","nullable":true,"metadata":{}},{"name":"version","type":"long","nullable":true,"metadata":{}},{"name":"lastUpdated","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"add","type":{"type":"struct","fields":[{"name":"path","type":"string","nullable":true,"metadata":{}},{"name":"partitionValues","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"size","type":"long","nullable":true,"metadata":{}},{"name":"modificationTime","type":"long","nullable":true,"metadata":{}},{"name":"dataChange","type":"boolean","nullable":true,"metadata":{}},{"name":"tags","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"deletionVector","type":{"type":"struct","fields":[{"name":"storageType","type":"string","nullable":true,"metadata":{}},{"name":"pathOrInlineDv","type":"string","nullable":true,"metadata":{}},{"name":"offset","type":"integer","nullable":true,"metadata":{}},{"name":"sizeInBytes","type":"integer","nullable":true,"metadata":{}},{"name":"cardinality","type":"long","nullable":true,"metadata":{}},{"name":"maxRowIndex","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"baseRowId","type":"long","nullable":true,"metadata":{}},{"name":"defaultRowCommitVersion","type":"long","nullable":true,"metadata":{}},{"name":"clusteringProvider","type":"string","nullable":true,"metadata":{}},{"name":"partitionValues_parsed","type":{"type":"struct","fields":[{"name":"part","type":"integer","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"stats_parsed","type":{"type":"struct","fields":[{"name":"numRecords","type":"long","nullable":true,"metadata":{}},{"name":"minValues","type":{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"value","type":"string","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"maxValues","type":{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"value","type":"string","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"nullCount","type":{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"value","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"tightBounds","type":"boolean","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"remove","type":{"type":"struct","fields":[{"name":"path","type":"string","nullable":true,"metadata":{}},{"name":"deletionTimestamp","type":"long","nullable":true,"metadata":{}},{"name":"dataChange","type":"boolean","nullable":true,"metadata":{}},{"name":"extendedFileMetadata","type":"boolean","nullable":true,"metadata":{}},{"name":"partitionValues","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"size","type":"long","nullable":true,"metadata":{}},{"name":"deletionVector","type":{"type":"struct","fields":[{"name":"storageType","type":"string","nullable":true,"metadata":{}},{"name":"pathOrInlineDv","type":"string","nullable":true,"metadata":{}},{"name":"offset","type":"integer","nullable":true,"metadata":{}},{"name":"sizeInBytes","type":"integer","nullable":true,"metadata":{}},{"name":"cardinality","type":"long","nullable":true,"metadata":{}},{"name":"maxRowIndex","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"baseRowId","type":"long","nullable":true,"metadata":{}},{"name":"defaultRowCommitVersion","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"metaData","type":{"type":"struct","fields":[{"name":"id","type":"string","nullable":true,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"description","type":"string","nullable":true,"metadata":{}},{"name":"format","type":{"type":"struct","fields":[{"name":"provider","type":"string","nullable":true,"metadata":{}},{"name":"options","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"schemaString","type":"string","nullable":true,"metadata":{}},{"name":"partitionColumns","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"configuration","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"createdTime","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"protocol","type":{"type":"struct","fields":[{"name":"minReaderVersion","type":"integer","nullable":true,"metadata":{}},{"name":"minWriterVersion","type":"integer","nullable":true,"metadata":{}},{"name":"readerFeatures","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"writerFeatures","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"domainMetadata","type":{"type":"struct","fields":[{"name":"domain","type":"string","nullable":true,"metadata":{}},{"name":"configuration","type":"string","nullable":true,"metadata":{}},{"name":"removed","type":"boolean","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}}]},"checksum":"4f28f78d7d36c441541a7d760a1d853c"} diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=0/.test%file%prefix-part-00000-dea8716a-0901-4b81-a9c9-fa6d76ea5ea9.c000.snappy.parquet.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=0/.test%file%prefix-part-00000-dea8716a-0901-4b81-a9c9-fa6d76ea5ea9.c000.snappy.parquet.crc new file mode 100644 index 0000000000..48ac303d76 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=0/.test%file%prefix-part-00000-dea8716a-0901-4b81-a9c9-fa6d76ea5ea9.c000.snappy.parquet.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=0/test%file%prefix-part-00000-dea8716a-0901-4b81-a9c9-fa6d76ea5ea9.c000.snappy.parquet b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=0/test%file%prefix-part-00000-dea8716a-0901-4b81-a9c9-fa6d76ea5ea9.c000.snappy.parquet new file mode 100644 index 0000000000..c657bb11f0 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=0/test%file%prefix-part-00000-dea8716a-0901-4b81-a9c9-fa6d76ea5ea9.c000.snappy.parquet differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=1/.test%file%prefix-part-00000-a7a1cef9-8cb8-4adb-b7ba-adc2d3c33f39.c000.snappy.parquet.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=1/.test%file%prefix-part-00000-a7a1cef9-8cb8-4adb-b7ba-adc2d3c33f39.c000.snappy.parquet.crc new file mode 100644 index 0000000000..51ecfe0daa Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=1/.test%file%prefix-part-00000-a7a1cef9-8cb8-4adb-b7ba-adc2d3c33f39.c000.snappy.parquet.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=1/.test%file%prefix-part-00000-ad2e4232-aae1-41ea-b46c-569d88253339.c000.snappy.parquet.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=1/.test%file%prefix-part-00000-ad2e4232-aae1-41ea-b46c-569d88253339.c000.snappy.parquet.crc new file mode 100644 index 0000000000..32805edc53 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=1/.test%file%prefix-part-00000-ad2e4232-aae1-41ea-b46c-569d88253339.c000.snappy.parquet.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=1/test%file%prefix-part-00000-a7a1cef9-8cb8-4adb-b7ba-adc2d3c33f39.c000.snappy.parquet b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=1/test%file%prefix-part-00000-a7a1cef9-8cb8-4adb-b7ba-adc2d3c33f39.c000.snappy.parquet new file mode 100644 index 0000000000..9dbcc4d0c5 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=1/test%file%prefix-part-00000-a7a1cef9-8cb8-4adb-b7ba-adc2d3c33f39.c000.snappy.parquet differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=1/test%file%prefix-part-00000-ad2e4232-aae1-41ea-b46c-569d88253339.c000.snappy.parquet b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=1/test%file%prefix-part-00000-ad2e4232-aae1-41ea-b46c-569d88253339.c000.snappy.parquet new file mode 100644 index 0000000000..072ed609f0 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=1/test%file%prefix-part-00000-ad2e4232-aae1-41ea-b46c-569d88253339.c000.snappy.parquet differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=2/.test%file%prefix-part-00000-03c40ea1-51b8-4636-9900-fb0885b47b87.c000.snappy.parquet.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=2/.test%file%prefix-part-00000-03c40ea1-51b8-4636-9900-fb0885b47b87.c000.snappy.parquet.crc new file mode 100644 index 0000000000..ee945fa527 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=2/.test%file%prefix-part-00000-03c40ea1-51b8-4636-9900-fb0885b47b87.c000.snappy.parquet.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=2/.test%file%prefix-part-00000-b101d239-17b8-4e26-9b34-5a43db693182.c000.snappy.parquet.crc b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=2/.test%file%prefix-part-00000-b101d239-17b8-4e26-9b34-5a43db693182.c000.snappy.parquet.crc new file mode 100644 index 0000000000..681e4c1106 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=2/.test%file%prefix-part-00000-b101d239-17b8-4e26-9b34-5a43db693182.c000.snappy.parquet.crc differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=2/test%file%prefix-part-00000-03c40ea1-51b8-4636-9900-fb0885b47b87.c000.snappy.parquet b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=2/test%file%prefix-part-00000-03c40ea1-51b8-4636-9900-fb0885b47b87.c000.snappy.parquet new file mode 100644 index 0000000000..a10f42d61e Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=2/test%file%prefix-part-00000-03c40ea1-51b8-4636-9900-fb0885b47b87.c000.snappy.parquet differ diff --git a/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=2/test%file%prefix-part-00000-b101d239-17b8-4e26-9b34-5a43db693182.c000.snappy.parquet b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=2/test%file%prefix-part-00000-b101d239-17b8-4e26-9b34-5a43db693182.c000.snappy.parquet new file mode 100644 index 0000000000..93f8e18a4a Binary files /dev/null and b/kernel/tests/data/v1-multi-part-partitioned-struct-stats-only/part=2/test%file%prefix-part-00000-b101d239-17b8-4e26-9b34-5a43db693182.c000.snappy.parquet differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/.test%file%prefix-part-00000-2699f745-4b33-4eb9-b3cf-04f6af08307f-c000.snappy.parquet.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/.test%file%prefix-part-00000-2699f745-4b33-4eb9-b3cf-04f6af08307f-c000.snappy.parquet.crc new file mode 100644 index 0000000000..561aed6528 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/.test%file%prefix-part-00000-2699f745-4b33-4eb9-b3cf-04f6af08307f-c000.snappy.parquet.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/.test%file%prefix-part-00000-323f4e76-58ff-48ce-bf0d-14d179e9bf0c-c000.snappy.parquet.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/.test%file%prefix-part-00000-323f4e76-58ff-48ce-bf0d-14d179e9bf0c-c000.snappy.parquet.crc new file mode 100644 index 0000000000..d3733258b8 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/.test%file%prefix-part-00000-323f4e76-58ff-48ce-bf0d-14d179e9bf0c-c000.snappy.parquet.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/.test%file%prefix-part-00000-743ccd8e-15b0-49f2-b0e2-aa0efbf148ae-c000.snappy.parquet.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/.test%file%prefix-part-00000-743ccd8e-15b0-49f2-b0e2-aa0efbf148ae-c000.snappy.parquet.crc new file mode 100644 index 0000000000..045a2dd4da Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/.test%file%prefix-part-00000-743ccd8e-15b0-49f2-b0e2-aa0efbf148ae-c000.snappy.parquet.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/.test%file%prefix-part-00000-f98612d6-6213-41f1-a006-a11beb0bb544-c000.snappy.parquet.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/.test%file%prefix-part-00000-f98612d6-6213-41f1-a006-a11beb0bb544-c000.snappy.parquet.crc new file mode 100644 index 0000000000..798f2039d9 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/.test%file%prefix-part-00000-f98612d6-6213-41f1-a006-a11beb0bb544-c000.snappy.parquet.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/.test%file%prefix-part-00000-ff529603-203f-4a68-9ab1-d495e5c1c409-c000.snappy.parquet.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/.test%file%prefix-part-00000-ff529603-203f-4a68-9ab1-d495e5c1c409-c000.snappy.parquet.crc new file mode 100644 index 0000000000..e5247f2373 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/.test%file%prefix-part-00000-ff529603-203f-4a68-9ab1-d495e5c1c409-c000.snappy.parquet.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000000.crc.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000000.crc.crc new file mode 100644 index 0000000000..d5bffa5f88 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000000.crc.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000000.json.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000000.json.crc new file mode 100644 index 0000000000..1372cecb26 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000000.json.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000001.crc.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000001.crc.crc new file mode 100644 index 0000000000..1dacc17a20 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000001.crc.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000001.json.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000001.json.crc new file mode 100644 index 0000000000..546507c441 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000001.json.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000002.crc.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000002.crc.crc new file mode 100644 index 0000000000..dd8eafd28e Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000002.crc.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000002.json.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000002.json.crc new file mode 100644 index 0000000000..e52117daa6 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000002.json.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000003.crc.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000003.crc.crc new file mode 100644 index 0000000000..90e39ba969 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000003.crc.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000003.json.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000003.json.crc new file mode 100644 index 0000000000..0c4083e121 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000003.json.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000004.crc.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000004.crc.crc new file mode 100644 index 0000000000..71b8edf204 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000004.crc.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000004.json.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000004.json.crc new file mode 100644 index 0000000000..a98d46fb6b Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000004.json.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000001.0000000003.parquet.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000001.0000000003.parquet.crc new file mode 100644 index 0000000000..d6464fc417 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000001.0000000003.parquet.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000002.0000000003.parquet.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000002.0000000003.parquet.crc new file mode 100644 index 0000000000..9090eecb2a Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000002.0000000003.parquet.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000003.0000000003.parquet.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000003.0000000003.parquet.crc new file mode 100644 index 0000000000..a66073d640 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000005.checkpoint.0000000003.0000000003.parquet.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000005.crc.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000005.crc.crc new file mode 100644 index 0000000000..6acac728ce Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000005.crc.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000005.json.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000005.json.crc new file mode 100644 index 0000000000..22eff0cb48 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/.00000000000000000005.json.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/._last_checkpoint.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/._last_checkpoint.crc new file mode 100644 index 0000000000..4227fc1c5b Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/._last_checkpoint.crc differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000000.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000000.crc new file mode 100644 index 0000000000..44ea44ce1f --- /dev/null +++ b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000000.crc @@ -0,0 +1 @@ +{"txnId":"cd694959-725d-44cc-b36e-fdda171ae1ad","tableSizeBytes":0,"numFiles":0,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"7fb4ebca-e84f-4dcb-9ec9-ab4032c10032","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774306387714},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[0,0,0,0,0,0,0,0,0,0]},"allFiles":[]} diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000000.json b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000000.json new file mode 100644 index 0000000000..cd9461a304 --- /dev/null +++ b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000000.json @@ -0,0 +1,3 @@ +{"commitInfo":{"timestamp":1774306390194,"operation":"CREATE TABLE","operationParameters":{"partitionBy":"[]","clusterBy":"[]","description":null,"isManaged":"false","properties":"{\"delta.checkpoint.writeStatsAsStruct\":\"true\",\"delta.checkpoint.writeStatsAsJson\":\"false\",\"delta.checkpointInterval\":\"10\",\"delta.enableDeletionVectors\":\"true\"}","statsOnLoad":false},"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{},"tags":{"restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"cd694959-725d-44cc-b36e-fdda171ae1ad"}} +{"metaData":{"id":"7fb4ebca-e84f-4dcb-9ec9-ab4032c10032","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774306387714}} +{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]}} diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000001.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000001.crc new file mode 100644 index 0000000000..c4e41802e9 --- /dev/null +++ b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000001.crc @@ -0,0 +1 @@ +{"txnId":"27faf0dc-2bbc-474c-b6a5-974606d30811","tableSizeBytes":761,"numFiles":1,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"7fb4ebca-e84f-4dcb-9ec9-ab4032c10032","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774306387714},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[761,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[1,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-f98612d6-6213-41f1-a006-a11beb0bb544-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774306398915,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306398915000","MIN_INSERTION_TIME":"1774306398915000","MAX_INSERTION_TIME":"1774306398915000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000001.json b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000001.json new file mode 100644 index 0000000000..911a5d76e2 --- /dev/null +++ b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000001.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774306399541,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":0,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"27faf0dc-2bbc-474c-b6a5-974606d30811"}} +{"add":{"path":"test%25file%25prefix-part-00000-f98612d6-6213-41f1-a006-a11beb0bb544-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774306398915,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306398915000","MIN_INSERTION_TIME":"1774306398915000","MAX_INSERTION_TIME":"1774306398915000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000002.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000002.crc new file mode 100644 index 0000000000..6847ff21dd --- /dev/null +++ b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000002.crc @@ -0,0 +1 @@ +{"txnId":"6fe44e3e-6c3b-4e59-8dea-5f24d12bf46f","tableSizeBytes":1522,"numFiles":2,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"7fb4ebca-e84f-4dcb-9ec9-ab4032c10032","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774306387714},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[1522,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[2,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-2699f745-4b33-4eb9-b3cf-04f6af08307f-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774306401007,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306401007000","MIN_INSERTION_TIME":"1774306401007000","MAX_INSERTION_TIME":"1774306401007000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-f98612d6-6213-41f1-a006-a11beb0bb544-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774306398915,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306398915000","MIN_INSERTION_TIME":"1774306398915000","MAX_INSERTION_TIME":"1774306398915000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000002.json b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000002.json new file mode 100644 index 0000000000..5aceff3f54 --- /dev/null +++ b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000002.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774306401184,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":1,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"6fe44e3e-6c3b-4e59-8dea-5f24d12bf46f"}} +{"add":{"path":"test%25file%25prefix-part-00000-2699f745-4b33-4eb9-b3cf-04f6af08307f-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774306401007,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306401007000","MIN_INSERTION_TIME":"1774306401007000","MAX_INSERTION_TIME":"1774306401007000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000003.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000003.crc new file mode 100644 index 0000000000..5d7944d64f --- /dev/null +++ b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000003.crc @@ -0,0 +1 @@ +{"txnId":"ede546c5-e345-461e-ad4e-9d0428f23df1","tableSizeBytes":2282,"numFiles":3,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"7fb4ebca-e84f-4dcb-9ec9-ab4032c10032","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774306387714},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[2282,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[3,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-f98612d6-6213-41f1-a006-a11beb0bb544-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774306398915,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306398915000","MIN_INSERTION_TIME":"1774306398915000","MAX_INSERTION_TIME":"1774306398915000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-2699f745-4b33-4eb9-b3cf-04f6af08307f-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774306401007,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306401007000","MIN_INSERTION_TIME":"1774306401007000","MAX_INSERTION_TIME":"1774306401007000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-ff529603-203f-4a68-9ab1-d495e5c1c409-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774306402499,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306402499000","MIN_INSERTION_TIME":"1774306402499000","MAX_INSERTION_TIME":"1774306402499000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000003.json b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000003.json new file mode 100644 index 0000000000..96a77babd1 --- /dev/null +++ b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000003.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774306402691,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":2,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"760"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"ede546c5-e345-461e-ad4e-9d0428f23df1"}} +{"add":{"path":"test%25file%25prefix-part-00000-ff529603-203f-4a68-9ab1-d495e5c1c409-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774306402499,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306402499000","MIN_INSERTION_TIME":"1774306402499000","MAX_INSERTION_TIME":"1774306402499000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000004.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000004.crc new file mode 100644 index 0000000000..2836d3ba84 --- /dev/null +++ b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000004.crc @@ -0,0 +1 @@ +{"txnId":"12423c00-bb1e-493f-b23a-618d1efaaf2e","tableSizeBytes":3043,"numFiles":4,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"7fb4ebca-e84f-4dcb-9ec9-ab4032c10032","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774306387714},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[3043,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[4,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-2699f745-4b33-4eb9-b3cf-04f6af08307f-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774306401007,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306401007000","MIN_INSERTION_TIME":"1774306401007000","MAX_INSERTION_TIME":"1774306401007000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-f98612d6-6213-41f1-a006-a11beb0bb544-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774306398915,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306398915000","MIN_INSERTION_TIME":"1774306398915000","MAX_INSERTION_TIME":"1774306398915000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-323f4e76-58ff-48ce-bf0d-14d179e9bf0c-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774306403960,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306403960000","MIN_INSERTION_TIME":"1774306403960000","MAX_INSERTION_TIME":"1774306403960000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-ff529603-203f-4a68-9ab1-d495e5c1c409-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774306402499,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306402499000","MIN_INSERTION_TIME":"1774306402499000","MAX_INSERTION_TIME":"1774306402499000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000004.json b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000004.json new file mode 100644 index 0000000000..e18a6aae9e --- /dev/null +++ b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000004.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774306404163,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":3,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"12423c00-bb1e-493f-b23a-618d1efaaf2e"}} +{"add":{"path":"test%25file%25prefix-part-00000-323f4e76-58ff-48ce-bf0d-14d179e9bf0c-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774306403960,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306403960000","MIN_INSERTION_TIME":"1774306403960000","MAX_INSERTION_TIME":"1774306403960000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000001.0000000003.parquet b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000001.0000000003.parquet new file mode 100644 index 0000000000..aea9e46f4a Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000001.0000000003.parquet differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000002.0000000003.parquet b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000002.0000000003.parquet new file mode 100644 index 0000000000..0f5095f324 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000002.0000000003.parquet differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000003.0000000003.parquet b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000003.0000000003.parquet new file mode 100644 index 0000000000..152486d07d Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000005.checkpoint.0000000003.0000000003.parquet differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000005.crc b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000005.crc new file mode 100644 index 0000000000..3b6f6c25de --- /dev/null +++ b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000005.crc @@ -0,0 +1 @@ +{"txnId":"77b1e654-b6b8-45b0-8df7-4db0e1aff223","tableSizeBytes":3804,"numFiles":5,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"7fb4ebca-e84f-4dcb-9ec9-ab4032c10032","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774306387714},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[3804,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[5,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-f98612d6-6213-41f1-a006-a11beb0bb544-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774306398915,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306398915000","MIN_INSERTION_TIME":"1774306398915000","MAX_INSERTION_TIME":"1774306398915000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-2699f745-4b33-4eb9-b3cf-04f6af08307f-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774306401007,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306401007000","MIN_INSERTION_TIME":"1774306401007000","MAX_INSERTION_TIME":"1774306401007000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-323f4e76-58ff-48ce-bf0d-14d179e9bf0c-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774306403960,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306403960000","MIN_INSERTION_TIME":"1774306403960000","MAX_INSERTION_TIME":"1774306403960000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-743ccd8e-15b0-49f2-b0e2-aa0efbf148ae-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774306405288,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":5,\"value\":\"value_5\"},\"maxValues\":{\"id\":5,\"value\":\"value_5\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306405288000","MIN_INSERTION_TIME":"1774306405288000","MAX_INSERTION_TIME":"1774306405288000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-ff529603-203f-4a68-9ab1-d495e5c1c409-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774306402499,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306402499000","MIN_INSERTION_TIME":"1774306402499000","MAX_INSERTION_TIME":"1774306402499000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000005.json b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000005.json new file mode 100644 index 0000000000..7d54ec8b10 --- /dev/null +++ b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/00000000000000000005.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774306405430,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":4,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"77b1e654-b6b8-45b0-8df7-4db0e1aff223"}} +{"add":{"path":"test%25file%25prefix-part-00000-743ccd8e-15b0-49f2-b0e2-aa0efbf148ae-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774306405288,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":5,\"value\":\"value_5\"},\"maxValues\":{\"id\":5,\"value\":\"value_5\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774306405288000","MIN_INSERTION_TIME":"1774306405288000","MAX_INSERTION_TIME":"1774306405288000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/_last_checkpoint b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/_last_checkpoint new file mode 100644 index 0000000000..cc2eabf8ed --- /dev/null +++ b/kernel/tests/data/v1-multi-part-struct-stats-only/_delta_log/_last_checkpoint @@ -0,0 +1 @@ +{"version":5,"size":7,"parts":3,"sizeInBytes":46005,"numOfAddFiles":5,"checkpointSchema":{"type":"struct","fields":[{"name":"txn","type":{"type":"struct","fields":[{"name":"appId","type":"string","nullable":true,"metadata":{}},{"name":"version","type":"long","nullable":true,"metadata":{}},{"name":"lastUpdated","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"add","type":{"type":"struct","fields":[{"name":"path","type":"string","nullable":true,"metadata":{}},{"name":"partitionValues","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"size","type":"long","nullable":true,"metadata":{}},{"name":"modificationTime","type":"long","nullable":true,"metadata":{}},{"name":"dataChange","type":"boolean","nullable":true,"metadata":{}},{"name":"tags","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"deletionVector","type":{"type":"struct","fields":[{"name":"storageType","type":"string","nullable":true,"metadata":{}},{"name":"pathOrInlineDv","type":"string","nullable":true,"metadata":{}},{"name":"offset","type":"integer","nullable":true,"metadata":{}},{"name":"sizeInBytes","type":"integer","nullable":true,"metadata":{}},{"name":"cardinality","type":"long","nullable":true,"metadata":{}},{"name":"maxRowIndex","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"baseRowId","type":"long","nullable":true,"metadata":{}},{"name":"defaultRowCommitVersion","type":"long","nullable":true,"metadata":{}},{"name":"clusteringProvider","type":"string","nullable":true,"metadata":{}},{"name":"stats_parsed","type":{"type":"struct","fields":[{"name":"numRecords","type":"long","nullable":true,"metadata":{}},{"name":"minValues","type":{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"value","type":"string","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"maxValues","type":{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"value","type":"string","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"nullCount","type":{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"value","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"tightBounds","type":"boolean","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"remove","type":{"type":"struct","fields":[{"name":"path","type":"string","nullable":true,"metadata":{}},{"name":"deletionTimestamp","type":"long","nullable":true,"metadata":{}},{"name":"dataChange","type":"boolean","nullable":true,"metadata":{}},{"name":"extendedFileMetadata","type":"boolean","nullable":true,"metadata":{}},{"name":"partitionValues","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"size","type":"long","nullable":true,"metadata":{}},{"name":"deletionVector","type":{"type":"struct","fields":[{"name":"storageType","type":"string","nullable":true,"metadata":{}},{"name":"pathOrInlineDv","type":"string","nullable":true,"metadata":{}},{"name":"offset","type":"integer","nullable":true,"metadata":{}},{"name":"sizeInBytes","type":"integer","nullable":true,"metadata":{}},{"name":"cardinality","type":"long","nullable":true,"metadata":{}},{"name":"maxRowIndex","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"baseRowId","type":"long","nullable":true,"metadata":{}},{"name":"defaultRowCommitVersion","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"metaData","type":{"type":"struct","fields":[{"name":"id","type":"string","nullable":true,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"description","type":"string","nullable":true,"metadata":{}},{"name":"format","type":{"type":"struct","fields":[{"name":"provider","type":"string","nullable":true,"metadata":{}},{"name":"options","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"schemaString","type":"string","nullable":true,"metadata":{}},{"name":"partitionColumns","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"configuration","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"createdTime","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"protocol","type":{"type":"struct","fields":[{"name":"minReaderVersion","type":"integer","nullable":true,"metadata":{}},{"name":"minWriterVersion","type":"integer","nullable":true,"metadata":{}},{"name":"readerFeatures","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"writerFeatures","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"domainMetadata","type":{"type":"struct","fields":[{"name":"domain","type":"string","nullable":true,"metadata":{}},{"name":"configuration","type":"string","nullable":true,"metadata":{}},{"name":"removed","type":"boolean","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}}]},"checksum":"fd3a25e1faa0025de64503c5038a18c6"} diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/test%file%prefix-part-00000-2699f745-4b33-4eb9-b3cf-04f6af08307f-c000.snappy.parquet b/kernel/tests/data/v1-multi-part-struct-stats-only/test%file%prefix-part-00000-2699f745-4b33-4eb9-b3cf-04f6af08307f-c000.snappy.parquet new file mode 100644 index 0000000000..7f098ecab2 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/test%file%prefix-part-00000-2699f745-4b33-4eb9-b3cf-04f6af08307f-c000.snappy.parquet differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/test%file%prefix-part-00000-323f4e76-58ff-48ce-bf0d-14d179e9bf0c-c000.snappy.parquet b/kernel/tests/data/v1-multi-part-struct-stats-only/test%file%prefix-part-00000-323f4e76-58ff-48ce-bf0d-14d179e9bf0c-c000.snappy.parquet new file mode 100644 index 0000000000..55de20cfae Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/test%file%prefix-part-00000-323f4e76-58ff-48ce-bf0d-14d179e9bf0c-c000.snappy.parquet differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/test%file%prefix-part-00000-743ccd8e-15b0-49f2-b0e2-aa0efbf148ae-c000.snappy.parquet b/kernel/tests/data/v1-multi-part-struct-stats-only/test%file%prefix-part-00000-743ccd8e-15b0-49f2-b0e2-aa0efbf148ae-c000.snappy.parquet new file mode 100644 index 0000000000..9d1435acf4 Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/test%file%prefix-part-00000-743ccd8e-15b0-49f2-b0e2-aa0efbf148ae-c000.snappy.parquet differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/test%file%prefix-part-00000-f98612d6-6213-41f1-a006-a11beb0bb544-c000.snappy.parquet b/kernel/tests/data/v1-multi-part-struct-stats-only/test%file%prefix-part-00000-f98612d6-6213-41f1-a006-a11beb0bb544-c000.snappy.parquet new file mode 100644 index 0000000000..60eb11bf0d Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/test%file%prefix-part-00000-f98612d6-6213-41f1-a006-a11beb0bb544-c000.snappy.parquet differ diff --git a/kernel/tests/data/v1-multi-part-struct-stats-only/test%file%prefix-part-00000-ff529603-203f-4a68-9ab1-d495e5c1c409-c000.snappy.parquet b/kernel/tests/data/v1-multi-part-struct-stats-only/test%file%prefix-part-00000-ff529603-203f-4a68-9ab1-d495e5c1c409-c000.snappy.parquet new file mode 100644 index 0000000000..d39e06afab Binary files /dev/null and b/kernel/tests/data/v1-multi-part-struct-stats-only/test%file%prefix-part-00000-ff529603-203f-4a68-9ab1-d495e5c1c409-c000.snappy.parquet differ diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000000.crc b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000000.crc new file mode 100644 index 0000000000..b2e2a1425f --- /dev/null +++ b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000000.crc @@ -0,0 +1 @@ +{"txnId":"c9205b56-4d26-4f2b-ad0a-fe0ccf1778ec","tableSizeBytes":0,"numFiles":0,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"3afc4c9a-c57b-4c20-8a53-f00ec003c9b4","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774386277024},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[0,0,0,0,0,0,0,0,0,0]},"allFiles":[]} diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000000.json b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000000.json new file mode 100644 index 0000000000..39fe291082 --- /dev/null +++ b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000000.json @@ -0,0 +1,3 @@ +{"commitInfo":{"timestamp":1774386279464,"operation":"CREATE TABLE","operationParameters":{"partitionBy":"[]","clusterBy":"[]","description":null,"isManaged":"false","properties":"{\"delta.checkpoint.writeStatsAsStruct\":\"true\",\"delta.checkpoint.writeStatsAsJson\":\"false\",\"delta.checkpointInterval\":\"10\",\"delta.enableDeletionVectors\":\"true\"}","statsOnLoad":false},"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{},"tags":{"restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"c9205b56-4d26-4f2b-ad0a-fe0ccf1778ec"}} +{"metaData":{"id":"3afc4c9a-c57b-4c20-8a53-f00ec003c9b4","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774386277024}} +{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]}} diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000001.crc b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000001.crc new file mode 100644 index 0000000000..fb754878ec --- /dev/null +++ b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000001.crc @@ -0,0 +1 @@ +{"txnId":"811e51aa-84dc-4dc9-8033-52f571aea1a8","tableSizeBytes":761,"numFiles":1,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"3afc4c9a-c57b-4c20-8a53-f00ec003c9b4","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774386277024},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[761,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[1,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-09ac7369-06f6-48dd-99ac-5a97ff6b8cb8-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774386288245,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386288245000","MIN_INSERTION_TIME":"1774386288245000","MAX_INSERTION_TIME":"1774386288245000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000001.json b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000001.json new file mode 100644 index 0000000000..0564672530 --- /dev/null +++ b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000001.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774386288935,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":0,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"811e51aa-84dc-4dc9-8033-52f571aea1a8"}} +{"add":{"path":"test%25file%25prefix-part-00000-09ac7369-06f6-48dd-99ac-5a97ff6b8cb8-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774386288245,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386288245000","MIN_INSERTION_TIME":"1774386288245000","MAX_INSERTION_TIME":"1774386288245000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000002.crc b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000002.crc new file mode 100644 index 0000000000..ed8e4bc8e8 --- /dev/null +++ b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000002.crc @@ -0,0 +1 @@ +{"txnId":"a4f2cfb3-0ccb-42a8-becd-13ab502f10cb","tableSizeBytes":1522,"numFiles":2,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"3afc4c9a-c57b-4c20-8a53-f00ec003c9b4","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774386277024},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[1522,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[2,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-ca7dba04-ea61-4e38-882e-ff6ee82f0d46-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774386290289,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386290289000","MIN_INSERTION_TIME":"1774386290289000","MAX_INSERTION_TIME":"1774386290289000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-09ac7369-06f6-48dd-99ac-5a97ff6b8cb8-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774386288245,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386288245000","MIN_INSERTION_TIME":"1774386288245000","MAX_INSERTION_TIME":"1774386288245000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000002.json b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000002.json new file mode 100644 index 0000000000..6a590666cc --- /dev/null +++ b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000002.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774386290471,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":1,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"a4f2cfb3-0ccb-42a8-becd-13ab502f10cb"}} +{"add":{"path":"test%25file%25prefix-part-00000-ca7dba04-ea61-4e38-882e-ff6ee82f0d46-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774386290289,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386290289000","MIN_INSERTION_TIME":"1774386290289000","MAX_INSERTION_TIME":"1774386290289000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000003.crc b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000003.crc new file mode 100644 index 0000000000..9fc76c9865 --- /dev/null +++ b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000003.crc @@ -0,0 +1 @@ +{"txnId":"e73d383b-18ba-4a2e-ab90-f3ce723a8f13","tableSizeBytes":2282,"numFiles":3,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"3afc4c9a-c57b-4c20-8a53-f00ec003c9b4","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774386277024},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[2282,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[3,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-54d78e6d-3c0d-402d-97cf-b6b46ab54533-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774386291681,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386291681000","MIN_INSERTION_TIME":"1774386291681000","MAX_INSERTION_TIME":"1774386291681000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-ca7dba04-ea61-4e38-882e-ff6ee82f0d46-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774386290289,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386290289000","MIN_INSERTION_TIME":"1774386290289000","MAX_INSERTION_TIME":"1774386290289000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-09ac7369-06f6-48dd-99ac-5a97ff6b8cb8-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774386288245,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386288245000","MIN_INSERTION_TIME":"1774386288245000","MAX_INSERTION_TIME":"1774386288245000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000003.json b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000003.json new file mode 100644 index 0000000000..ef3d7cd653 --- /dev/null +++ b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000003.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774386291840,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":2,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"760"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"e73d383b-18ba-4a2e-ab90-f3ce723a8f13"}} +{"add":{"path":"test%25file%25prefix-part-00000-54d78e6d-3c0d-402d-97cf-b6b46ab54533-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774386291681,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386291681000","MIN_INSERTION_TIME":"1774386291681000","MAX_INSERTION_TIME":"1774386291681000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000004.crc b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000004.crc new file mode 100644 index 0000000000..9eac110a6c --- /dev/null +++ b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000004.crc @@ -0,0 +1 @@ +{"txnId":"fbcfa79d-2daa-49db-be93-9eeda474dfe5","tableSizeBytes":3043,"numFiles":4,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"3afc4c9a-c57b-4c20-8a53-f00ec003c9b4","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774386277024},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[3043,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[4,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-1b98908c-7430-4b7c-af27-ac1452047ed1-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774386292913,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386292913000","MIN_INSERTION_TIME":"1774386292913000","MAX_INSERTION_TIME":"1774386292913000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-54d78e6d-3c0d-402d-97cf-b6b46ab54533-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774386291681,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386291681000","MIN_INSERTION_TIME":"1774386291681000","MAX_INSERTION_TIME":"1774386291681000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-ca7dba04-ea61-4e38-882e-ff6ee82f0d46-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774386290289,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386290289000","MIN_INSERTION_TIME":"1774386290289000","MAX_INSERTION_TIME":"1774386290289000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-09ac7369-06f6-48dd-99ac-5a97ff6b8cb8-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774386288245,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386288245000","MIN_INSERTION_TIME":"1774386288245000","MAX_INSERTION_TIME":"1774386288245000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000004.json b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000004.json new file mode 100644 index 0000000000..f3585bf53b --- /dev/null +++ b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000004.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774386293054,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":3,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"fbcfa79d-2daa-49db-be93-9eeda474dfe5"}} +{"add":{"path":"test%25file%25prefix-part-00000-1b98908c-7430-4b7c-af27-ac1452047ed1-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774386292913,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386292913000","MIN_INSERTION_TIME":"1774386292913000","MAX_INSERTION_TIME":"1774386292913000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000005.checkpoint.parquet b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000005.checkpoint.parquet new file mode 100644 index 0000000000..a0acb448b6 Binary files /dev/null and b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000005.checkpoint.parquet differ diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000005.crc b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000005.crc new file mode 100644 index 0000000000..2e7523842a --- /dev/null +++ b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000005.crc @@ -0,0 +1 @@ +{"txnId":"9cd89dcd-2e5d-4200-ae26-df7d8d5a4bf1","tableSizeBytes":3804,"numFiles":5,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"3afc4c9a-c57b-4c20-8a53-f00ec003c9b4","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.checkpoint.writeStatsAsStruct":"true","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10","delta.enableDeletionVectors":"true"},"createdTime":1774386277024},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[3804,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[5,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-1b98908c-7430-4b7c-af27-ac1452047ed1-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774386292913,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386292913000","MIN_INSERTION_TIME":"1774386292913000","MAX_INSERTION_TIME":"1774386292913000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-f993c7d8-518a-4056-a77c-7f0d06a6e07e-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774386294073,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":5,\"value\":\"value_5\"},\"maxValues\":{\"id\":5,\"value\":\"value_5\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386294073000","MIN_INSERTION_TIME":"1774386294073000","MAX_INSERTION_TIME":"1774386294073000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-54d78e6d-3c0d-402d-97cf-b6b46ab54533-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774386291681,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386291681000","MIN_INSERTION_TIME":"1774386291681000","MAX_INSERTION_TIME":"1774386291681000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-ca7dba04-ea61-4e38-882e-ff6ee82f0d46-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774386290289,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386290289000","MIN_INSERTION_TIME":"1774386290289000","MAX_INSERTION_TIME":"1774386290289000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-09ac7369-06f6-48dd-99ac-5a97ff6b8cb8-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774386288245,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386288245000","MIN_INSERTION_TIME":"1774386288245000","MAX_INSERTION_TIME":"1774386288245000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000005.json b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000005.json new file mode 100644 index 0000000000..5ced5e56fd --- /dev/null +++ b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/00000000000000000005.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774386294218,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":4,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"9cd89dcd-2e5d-4200-ae26-df7d8d5a4bf1"}} +{"add":{"path":"test%25file%25prefix-part-00000-f993c7d8-518a-4056-a77c-7f0d06a6e07e-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774386294073,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":5,\"value\":\"value_5\"},\"maxValues\":{\"id\":5,\"value\":\"value_5\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774386294073000","MIN_INSERTION_TIME":"1774386294073000","MAX_INSERTION_TIME":"1774386294073000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/_last_checkpoint b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/_last_checkpoint new file mode 100644 index 0000000000..18694d44ff --- /dev/null +++ b/kernel/tests/data/v1-single-part-struct-stats-only/_delta_log/_last_checkpoint @@ -0,0 +1 @@ +{"version":5,"size":7,"sizeInBytes":20415,"numOfAddFiles":5,"checkpointSchema":{"type":"struct","fields":[{"name":"txn","type":{"type":"struct","fields":[{"name":"appId","type":"string","nullable":true,"metadata":{}},{"name":"version","type":"long","nullable":true,"metadata":{}},{"name":"lastUpdated","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"add","type":{"type":"struct","fields":[{"name":"path","type":"string","nullable":true,"metadata":{}},{"name":"partitionValues","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"size","type":"long","nullable":true,"metadata":{}},{"name":"modificationTime","type":"long","nullable":true,"metadata":{}},{"name":"dataChange","type":"boolean","nullable":true,"metadata":{}},{"name":"tags","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"deletionVector","type":{"type":"struct","fields":[{"name":"storageType","type":"string","nullable":true,"metadata":{}},{"name":"pathOrInlineDv","type":"string","nullable":true,"metadata":{}},{"name":"offset","type":"integer","nullable":true,"metadata":{}},{"name":"sizeInBytes","type":"integer","nullable":true,"metadata":{}},{"name":"cardinality","type":"long","nullable":true,"metadata":{}},{"name":"maxRowIndex","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"baseRowId","type":"long","nullable":true,"metadata":{}},{"name":"defaultRowCommitVersion","type":"long","nullable":true,"metadata":{}},{"name":"clusteringProvider","type":"string","nullable":true,"metadata":{}},{"name":"stats_parsed","type":{"type":"struct","fields":[{"name":"numRecords","type":"long","nullable":true,"metadata":{}},{"name":"minValues","type":{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"value","type":"string","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"maxValues","type":{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"value","type":"string","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"nullCount","type":{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"value","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"tightBounds","type":"boolean","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"remove","type":{"type":"struct","fields":[{"name":"path","type":"string","nullable":true,"metadata":{}},{"name":"deletionTimestamp","type":"long","nullable":true,"metadata":{}},{"name":"dataChange","type":"boolean","nullable":true,"metadata":{}},{"name":"extendedFileMetadata","type":"boolean","nullable":true,"metadata":{}},{"name":"partitionValues","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"size","type":"long","nullable":true,"metadata":{}},{"name":"deletionVector","type":{"type":"struct","fields":[{"name":"storageType","type":"string","nullable":true,"metadata":{}},{"name":"pathOrInlineDv","type":"string","nullable":true,"metadata":{}},{"name":"offset","type":"integer","nullable":true,"metadata":{}},{"name":"sizeInBytes","type":"integer","nullable":true,"metadata":{}},{"name":"cardinality","type":"long","nullable":true,"metadata":{}},{"name":"maxRowIndex","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"baseRowId","type":"long","nullable":true,"metadata":{}},{"name":"defaultRowCommitVersion","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"metaData","type":{"type":"struct","fields":[{"name":"id","type":"string","nullable":true,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"description","type":"string","nullable":true,"metadata":{}},{"name":"format","type":{"type":"struct","fields":[{"name":"provider","type":"string","nullable":true,"metadata":{}},{"name":"options","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"schemaString","type":"string","nullable":true,"metadata":{}},{"name":"partitionColumns","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"configuration","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"createdTime","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"protocol","type":{"type":"struct","fields":[{"name":"minReaderVersion","type":"integer","nullable":true,"metadata":{}},{"name":"minWriterVersion","type":"integer","nullable":true,"metadata":{}},{"name":"readerFeatures","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"writerFeatures","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"domainMetadata","type":{"type":"struct","fields":[{"name":"domain","type":"string","nullable":true,"metadata":{}},{"name":"configuration","type":"string","nullable":true,"metadata":{}},{"name":"removed","type":"boolean","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}}]},"checkpointFiles":[{"path":"00000000000000000005.checkpoint.parquet","length":20415,"isDir":false,"modificationTime":1774386295689}],"checksum":"b03649834c60f41ff3ac3438cb49e29e"} diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/test%file%prefix-part-00000-09ac7369-06f6-48dd-99ac-5a97ff6b8cb8-c000.snappy.parquet b/kernel/tests/data/v1-single-part-struct-stats-only/test%file%prefix-part-00000-09ac7369-06f6-48dd-99ac-5a97ff6b8cb8-c000.snappy.parquet new file mode 100644 index 0000000000..e23848797a Binary files /dev/null and b/kernel/tests/data/v1-single-part-struct-stats-only/test%file%prefix-part-00000-09ac7369-06f6-48dd-99ac-5a97ff6b8cb8-c000.snappy.parquet differ diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/test%file%prefix-part-00000-1b98908c-7430-4b7c-af27-ac1452047ed1-c000.snappy.parquet b/kernel/tests/data/v1-single-part-struct-stats-only/test%file%prefix-part-00000-1b98908c-7430-4b7c-af27-ac1452047ed1-c000.snappy.parquet new file mode 100644 index 0000000000..bfc9dcfa44 Binary files /dev/null and b/kernel/tests/data/v1-single-part-struct-stats-only/test%file%prefix-part-00000-1b98908c-7430-4b7c-af27-ac1452047ed1-c000.snappy.parquet differ diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/test%file%prefix-part-00000-54d78e6d-3c0d-402d-97cf-b6b46ab54533-c000.snappy.parquet b/kernel/tests/data/v1-single-part-struct-stats-only/test%file%prefix-part-00000-54d78e6d-3c0d-402d-97cf-b6b46ab54533-c000.snappy.parquet new file mode 100644 index 0000000000..9713b2d963 Binary files /dev/null and b/kernel/tests/data/v1-single-part-struct-stats-only/test%file%prefix-part-00000-54d78e6d-3c0d-402d-97cf-b6b46ab54533-c000.snappy.parquet differ diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/test%file%prefix-part-00000-ca7dba04-ea61-4e38-882e-ff6ee82f0d46-c000.snappy.parquet b/kernel/tests/data/v1-single-part-struct-stats-only/test%file%prefix-part-00000-ca7dba04-ea61-4e38-882e-ff6ee82f0d46-c000.snappy.parquet new file mode 100644 index 0000000000..b6aabb8552 Binary files /dev/null and b/kernel/tests/data/v1-single-part-struct-stats-only/test%file%prefix-part-00000-ca7dba04-ea61-4e38-882e-ff6ee82f0d46-c000.snappy.parquet differ diff --git a/kernel/tests/data/v1-single-part-struct-stats-only/test%file%prefix-part-00000-f993c7d8-518a-4056-a77c-7f0d06a6e07e-c000.snappy.parquet b/kernel/tests/data/v1-single-part-struct-stats-only/test%file%prefix-part-00000-f993c7d8-518a-4056-a77c-7f0d06a6e07e-c000.snappy.parquet new file mode 100644 index 0000000000..1f8980ad29 Binary files /dev/null and b/kernel/tests/data/v1-single-part-struct-stats-only/test%file%prefix-part-00000-f993c7d8-518a-4056-a77c-7f0d06a6e07e-c000.snappy.parquet differ diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000000.crc b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000000.crc new file mode 100644 index 0000000000..f75eb23364 --- /dev/null +++ b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000000.crc @@ -0,0 +1 @@ +{"txnId":"97c7c862-efa8-499f-ad8b-a2cc7de3b384","tableSizeBytes":0,"numFiles":0,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"23b172f9-95c3-43d0-8c0c-423bb644526a","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"classic","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385170349},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[0,0,0,0,0,0,0,0,0,0]},"allFiles":[]} diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000000.json b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000000.json new file mode 100644 index 0000000000..e0428732ee --- /dev/null +++ b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000000.json @@ -0,0 +1,3 @@ +{"commitInfo":{"timestamp":1774385170444,"operation":"CREATE TABLE","operationParameters":{"partitionBy":"[]","clusterBy":"[]","description":null,"isManaged":"false","properties":"{\"delta.enableDeletionVectors\":\"true\",\"delta.checkpoint.writeStatsAsStruct\":\"true\",\"delta.checkpointPolicy\":\"classic\",\"delta.checkpoint.writeStatsAsJson\":\"false\",\"delta.checkpointInterval\":\"10\"}","statsOnLoad":false},"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{},"tags":{"restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"97c7c862-efa8-499f-ad8b-a2cc7de3b384"}} +{"metaData":{"id":"23b172f9-95c3-43d0-8c0c-423bb644526a","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"classic","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385170349}} +{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]}} diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000001.crc b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000001.crc new file mode 100644 index 0000000000..68dc8f5e73 --- /dev/null +++ b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000001.crc @@ -0,0 +1 @@ +{"txnId":"0eb5d6cd-b9d6-44d6-b958-b559d825d310","tableSizeBytes":761,"numFiles":1,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"23b172f9-95c3-43d0-8c0c-423bb644526a","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"classic","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385170349},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[761,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[1,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-981d116a-6a3c-4c09-b604-3695d0db5dc8-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385171192,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385171192000","MIN_INSERTION_TIME":"1774385171192000","MAX_INSERTION_TIME":"1774385171192000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000001.json b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000001.json new file mode 100644 index 0000000000..36488f6b0e --- /dev/null +++ b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000001.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774385171301,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":0,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"0eb5d6cd-b9d6-44d6-b958-b559d825d310"}} +{"add":{"path":"test%25file%25prefix-part-00000-981d116a-6a3c-4c09-b604-3695d0db5dc8-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385171192,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385171192000","MIN_INSERTION_TIME":"1774385171192000","MAX_INSERTION_TIME":"1774385171192000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000002.crc b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000002.crc new file mode 100644 index 0000000000..4a929422b3 --- /dev/null +++ b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000002.crc @@ -0,0 +1 @@ +{"txnId":"68c75a15-27e0-4e2f-904e-41fb4cc8607e","tableSizeBytes":1522,"numFiles":2,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"23b172f9-95c3-43d0-8c0c-423bb644526a","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"classic","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385170349},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[1522,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[2,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-981d116a-6a3c-4c09-b604-3695d0db5dc8-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385171192,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385171192000","MIN_INSERTION_TIME":"1774385171192000","MAX_INSERTION_TIME":"1774385171192000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-e2cba303-0dc3-4091-87cb-ce11d0447bda-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385172140,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385172140000","MIN_INSERTION_TIME":"1774385172140000","MAX_INSERTION_TIME":"1774385172140000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000002.json b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000002.json new file mode 100644 index 0000000000..852f871b7b --- /dev/null +++ b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000002.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774385172250,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":1,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"68c75a15-27e0-4e2f-904e-41fb4cc8607e"}} +{"add":{"path":"test%25file%25prefix-part-00000-e2cba303-0dc3-4091-87cb-ce11d0447bda-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385172140,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385172140000","MIN_INSERTION_TIME":"1774385172140000","MAX_INSERTION_TIME":"1774385172140000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000003.crc b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000003.crc new file mode 100644 index 0000000000..3df2b4f89d --- /dev/null +++ b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000003.crc @@ -0,0 +1 @@ +{"txnId":"fe119ce8-b73f-43d7-9739-aa410867a8e3","tableSizeBytes":2282,"numFiles":3,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"23b172f9-95c3-43d0-8c0c-423bb644526a","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"classic","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385170349},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[2282,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[3,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-bb4858c3-3c9a-4325-ab3f-a0e07e6a94d4-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774385173080,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385173080000","MIN_INSERTION_TIME":"1774385173080000","MAX_INSERTION_TIME":"1774385173080000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-981d116a-6a3c-4c09-b604-3695d0db5dc8-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385171192,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385171192000","MIN_INSERTION_TIME":"1774385171192000","MAX_INSERTION_TIME":"1774385171192000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-e2cba303-0dc3-4091-87cb-ce11d0447bda-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385172140,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385172140000","MIN_INSERTION_TIME":"1774385172140000","MAX_INSERTION_TIME":"1774385172140000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000003.json b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000003.json new file mode 100644 index 0000000000..386c947989 --- /dev/null +++ b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000003.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774385173188,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":2,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"760"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"fe119ce8-b73f-43d7-9739-aa410867a8e3"}} +{"add":{"path":"test%25file%25prefix-part-00000-bb4858c3-3c9a-4325-ab3f-a0e07e6a94d4-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774385173080,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385173080000","MIN_INSERTION_TIME":"1774385173080000","MAX_INSERTION_TIME":"1774385173080000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000004.crc b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000004.crc new file mode 100644 index 0000000000..981d7df1f6 --- /dev/null +++ b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000004.crc @@ -0,0 +1 @@ +{"txnId":"2b6eec86-6211-42fc-8574-7e3687c1c84c","tableSizeBytes":3043,"numFiles":4,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"23b172f9-95c3-43d0-8c0c-423bb644526a","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"classic","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385170349},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[3043,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[4,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-1f06e739-2467-49d8-bec1-dffa9c6ca027-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385174004,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385174004000","MIN_INSERTION_TIME":"1774385174004000","MAX_INSERTION_TIME":"1774385174004000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-bb4858c3-3c9a-4325-ab3f-a0e07e6a94d4-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774385173080,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385173080000","MIN_INSERTION_TIME":"1774385173080000","MAX_INSERTION_TIME":"1774385173080000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-981d116a-6a3c-4c09-b604-3695d0db5dc8-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385171192,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385171192000","MIN_INSERTION_TIME":"1774385171192000","MAX_INSERTION_TIME":"1774385171192000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-e2cba303-0dc3-4091-87cb-ce11d0447bda-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385172140,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385172140000","MIN_INSERTION_TIME":"1774385172140000","MAX_INSERTION_TIME":"1774385172140000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000004.json b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000004.json new file mode 100644 index 0000000000..3cb9a69add --- /dev/null +++ b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000004.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774385174111,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":3,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"2b6eec86-6211-42fc-8574-7e3687c1c84c"}} +{"add":{"path":"test%25file%25prefix-part-00000-1f06e739-2467-49d8-bec1-dffa9c6ca027-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385174004,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385174004000","MIN_INSERTION_TIME":"1774385174004000","MAX_INSERTION_TIME":"1774385174004000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000005.checkpoint.parquet b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000005.checkpoint.parquet new file mode 100644 index 0000000000..7d3be34c06 Binary files /dev/null and b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000005.checkpoint.parquet differ diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000005.crc b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000005.crc new file mode 100644 index 0000000000..57202f88ca --- /dev/null +++ b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000005.crc @@ -0,0 +1 @@ +{"txnId":"4e570e60-4733-42cf-82ed-285f8fd4f44d","tableSizeBytes":3804,"numFiles":5,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"23b172f9-95c3-43d0-8c0c-423bb644526a","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"classic","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385170349},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures":["deletionVectors","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[3804,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[5,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-1f06e739-2467-49d8-bec1-dffa9c6ca027-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385174004,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385174004000","MIN_INSERTION_TIME":"1774385174004000","MAX_INSERTION_TIME":"1774385174004000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-bb4858c3-3c9a-4325-ab3f-a0e07e6a94d4-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774385173080,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385173080000","MIN_INSERTION_TIME":"1774385173080000","MAX_INSERTION_TIME":"1774385173080000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-981d116a-6a3c-4c09-b604-3695d0db5dc8-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385171192,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385171192000","MIN_INSERTION_TIME":"1774385171192000","MAX_INSERTION_TIME":"1774385171192000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-f91cd7b7-b47f-4f76-b052-971c4a313235-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385174928,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":5,\"value\":\"value_5\"},\"maxValues\":{\"id\":5,\"value\":\"value_5\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385174928000","MIN_INSERTION_TIME":"1774385174928000","MAX_INSERTION_TIME":"1774385174928000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-e2cba303-0dc3-4091-87cb-ce11d0447bda-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385172140,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385172140000","MIN_INSERTION_TIME":"1774385172140000","MAX_INSERTION_TIME":"1774385172140000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000005.json b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000005.json new file mode 100644 index 0000000000..b1da73d912 --- /dev/null +++ b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/00000000000000000005.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774385175036,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":4,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"4e570e60-4733-42cf-82ed-285f8fd4f44d"}} +{"add":{"path":"test%25file%25prefix-part-00000-f91cd7b7-b47f-4f76-b052-971c4a313235-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385174928,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":5,\"value\":\"value_5\"},\"maxValues\":{\"id\":5,\"value\":\"value_5\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385174928000","MIN_INSERTION_TIME":"1774385174928000","MAX_INSERTION_TIME":"1774385174928000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/_last_checkpoint b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/_last_checkpoint new file mode 100644 index 0000000000..7bc3ed76f2 --- /dev/null +++ b/kernel/tests/data/v2-classic-parquet-struct-stats-only/_delta_log/_last_checkpoint @@ -0,0 +1 @@ +{"version":5,"size":7,"sizeInBytes":20434,"numOfAddFiles":5,"checkpointSchema":{"type":"struct","fields":[{"name":"txn","type":{"type":"struct","fields":[{"name":"appId","type":"string","nullable":true,"metadata":{}},{"name":"version","type":"long","nullable":true,"metadata":{}},{"name":"lastUpdated","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"add","type":{"type":"struct","fields":[{"name":"path","type":"string","nullable":true,"metadata":{}},{"name":"partitionValues","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"size","type":"long","nullable":true,"metadata":{}},{"name":"modificationTime","type":"long","nullable":true,"metadata":{}},{"name":"dataChange","type":"boolean","nullable":true,"metadata":{}},{"name":"tags","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"deletionVector","type":{"type":"struct","fields":[{"name":"storageType","type":"string","nullable":true,"metadata":{}},{"name":"pathOrInlineDv","type":"string","nullable":true,"metadata":{}},{"name":"offset","type":"integer","nullable":true,"metadata":{}},{"name":"sizeInBytes","type":"integer","nullable":true,"metadata":{}},{"name":"cardinality","type":"long","nullable":true,"metadata":{}},{"name":"maxRowIndex","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"baseRowId","type":"long","nullable":true,"metadata":{}},{"name":"defaultRowCommitVersion","type":"long","nullable":true,"metadata":{}},{"name":"clusteringProvider","type":"string","nullable":true,"metadata":{}},{"name":"stats_parsed","type":{"type":"struct","fields":[{"name":"numRecords","type":"long","nullable":true,"metadata":{}},{"name":"minValues","type":{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"value","type":"string","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"maxValues","type":{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"value","type":"string","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"nullCount","type":{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"value","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"tightBounds","type":"boolean","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"remove","type":{"type":"struct","fields":[{"name":"path","type":"string","nullable":true,"metadata":{}},{"name":"deletionTimestamp","type":"long","nullable":true,"metadata":{}},{"name":"dataChange","type":"boolean","nullable":true,"metadata":{}},{"name":"extendedFileMetadata","type":"boolean","nullable":true,"metadata":{}},{"name":"partitionValues","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"size","type":"long","nullable":true,"metadata":{}},{"name":"deletionVector","type":{"type":"struct","fields":[{"name":"storageType","type":"string","nullable":true,"metadata":{}},{"name":"pathOrInlineDv","type":"string","nullable":true,"metadata":{}},{"name":"offset","type":"integer","nullable":true,"metadata":{}},{"name":"sizeInBytes","type":"integer","nullable":true,"metadata":{}},{"name":"cardinality","type":"long","nullable":true,"metadata":{}},{"name":"maxRowIndex","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"baseRowId","type":"long","nullable":true,"metadata":{}},{"name":"defaultRowCommitVersion","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"metaData","type":{"type":"struct","fields":[{"name":"id","type":"string","nullable":true,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"description","type":"string","nullable":true,"metadata":{}},{"name":"format","type":{"type":"struct","fields":[{"name":"provider","type":"string","nullable":true,"metadata":{}},{"name":"options","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"schemaString","type":"string","nullable":true,"metadata":{}},{"name":"partitionColumns","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"configuration","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"createdTime","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"protocol","type":{"type":"struct","fields":[{"name":"minReaderVersion","type":"integer","nullable":true,"metadata":{}},{"name":"minWriterVersion","type":"integer","nullable":true,"metadata":{}},{"name":"readerFeatures","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"writerFeatures","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"domainMetadata","type":{"type":"struct","fields":[{"name":"domain","type":"string","nullable":true,"metadata":{}},{"name":"configuration","type":"string","nullable":true,"metadata":{}},{"name":"removed","type":"boolean","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}}]},"checkpointFiles":[{"path":"00000000000000000005.checkpoint.parquet","length":20434,"isDir":false,"modificationTime":1774385175900}],"checksum":"c4d2f11ca6ccd2a86dfc0c2ea42087a7"} diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/test%file%prefix-part-00000-1f06e739-2467-49d8-bec1-dffa9c6ca027-c000.snappy.parquet b/kernel/tests/data/v2-classic-parquet-struct-stats-only/test%file%prefix-part-00000-1f06e739-2467-49d8-bec1-dffa9c6ca027-c000.snappy.parquet new file mode 100644 index 0000000000..bfc9dcfa44 Binary files /dev/null and b/kernel/tests/data/v2-classic-parquet-struct-stats-only/test%file%prefix-part-00000-1f06e739-2467-49d8-bec1-dffa9c6ca027-c000.snappy.parquet differ diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/test%file%prefix-part-00000-981d116a-6a3c-4c09-b604-3695d0db5dc8-c000.snappy.parquet b/kernel/tests/data/v2-classic-parquet-struct-stats-only/test%file%prefix-part-00000-981d116a-6a3c-4c09-b604-3695d0db5dc8-c000.snappy.parquet new file mode 100644 index 0000000000..e23848797a Binary files /dev/null and b/kernel/tests/data/v2-classic-parquet-struct-stats-only/test%file%prefix-part-00000-981d116a-6a3c-4c09-b604-3695d0db5dc8-c000.snappy.parquet differ diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/test%file%prefix-part-00000-bb4858c3-3c9a-4325-ab3f-a0e07e6a94d4-c000.snappy.parquet b/kernel/tests/data/v2-classic-parquet-struct-stats-only/test%file%prefix-part-00000-bb4858c3-3c9a-4325-ab3f-a0e07e6a94d4-c000.snappy.parquet new file mode 100644 index 0000000000..9713b2d963 Binary files /dev/null and b/kernel/tests/data/v2-classic-parquet-struct-stats-only/test%file%prefix-part-00000-bb4858c3-3c9a-4325-ab3f-a0e07e6a94d4-c000.snappy.parquet differ diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/test%file%prefix-part-00000-e2cba303-0dc3-4091-87cb-ce11d0447bda-c000.snappy.parquet b/kernel/tests/data/v2-classic-parquet-struct-stats-only/test%file%prefix-part-00000-e2cba303-0dc3-4091-87cb-ce11d0447bda-c000.snappy.parquet new file mode 100644 index 0000000000..b6aabb8552 Binary files /dev/null and b/kernel/tests/data/v2-classic-parquet-struct-stats-only/test%file%prefix-part-00000-e2cba303-0dc3-4091-87cb-ce11d0447bda-c000.snappy.parquet differ diff --git a/kernel/tests/data/v2-classic-parquet-struct-stats-only/test%file%prefix-part-00000-f91cd7b7-b47f-4f76-b052-971c4a313235-c000.snappy.parquet b/kernel/tests/data/v2-classic-parquet-struct-stats-only/test%file%prefix-part-00000-f91cd7b7-b47f-4f76-b052-971c4a313235-c000.snappy.parquet new file mode 100644 index 0000000000..1f8980ad29 Binary files /dev/null and b/kernel/tests/data/v2-classic-parquet-struct-stats-only/test%file%prefix-part-00000-f91cd7b7-b47f-4f76-b052-971c4a313235-c000.snappy.parquet differ diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000000.crc b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000000.crc new file mode 100644 index 0000000000..4996568a8f --- /dev/null +++ b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000000.crc @@ -0,0 +1 @@ +{"txnId":"d2627053-64d8-40cc-ade5-bc8eae64b0f6","tableSizeBytes":0,"numFiles":0,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"aab52b7c-387a-4e1b-b804-4140a460bfd6","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"v2","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385164156},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors","v2Checkpoint"],"writerFeatures":["deletionVectors","v2Checkpoint","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[0,0,0,0,0,0,0,0,0,0]},"allFiles":[]} diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000000.json b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000000.json new file mode 100644 index 0000000000..689bbb5494 --- /dev/null +++ b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000000.json @@ -0,0 +1,3 @@ +{"commitInfo":{"timestamp":1774385164257,"operation":"CREATE TABLE","operationParameters":{"partitionBy":"[]","clusterBy":"[]","description":null,"isManaged":"false","properties":"{\"delta.enableDeletionVectors\":\"true\",\"delta.checkpoint.writeStatsAsStruct\":\"true\",\"delta.checkpointPolicy\":\"v2\",\"delta.checkpoint.writeStatsAsJson\":\"false\",\"delta.checkpointInterval\":\"10\"}","statsOnLoad":false},"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{},"tags":{"restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"d2627053-64d8-40cc-ade5-bc8eae64b0f6"}} +{"metaData":{"id":"aab52b7c-387a-4e1b-b804-4140a460bfd6","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"v2","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385164156}} +{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors","v2Checkpoint"],"writerFeatures":["deletionVectors","v2Checkpoint","appendOnly","invariants"]}} diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000001.crc b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000001.crc new file mode 100644 index 0000000000..999f0687e9 --- /dev/null +++ b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000001.crc @@ -0,0 +1 @@ +{"txnId":"f184b35c-f967-4c4f-91f9-b78e70bbdd20","tableSizeBytes":761,"numFiles":1,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"aab52b7c-387a-4e1b-b804-4140a460bfd6","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"v2","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385164156},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors","v2Checkpoint"],"writerFeatures":["deletionVectors","v2Checkpoint","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[761,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[1,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-b93d3d1f-a7ec-42f2-ade5-27837538abcd-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385165060,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385165060000","MIN_INSERTION_TIME":"1774385165060000","MAX_INSERTION_TIME":"1774385165060000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000001.json b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000001.json new file mode 100644 index 0000000000..0219121b96 --- /dev/null +++ b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000001.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774385165182,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":0,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"f184b35c-f967-4c4f-91f9-b78e70bbdd20"}} +{"add":{"path":"test%25file%25prefix-part-00000-b93d3d1f-a7ec-42f2-ade5-27837538abcd-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385165060,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385165060000","MIN_INSERTION_TIME":"1774385165060000","MAX_INSERTION_TIME":"1774385165060000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000002.crc b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000002.crc new file mode 100644 index 0000000000..78e9478dd3 --- /dev/null +++ b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000002.crc @@ -0,0 +1 @@ +{"txnId":"f91439b4-5a79-4fc9-8550-e03818be0ef7","tableSizeBytes":1522,"numFiles":2,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"aab52b7c-387a-4e1b-b804-4140a460bfd6","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"v2","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385164156},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors","v2Checkpoint"],"writerFeatures":["deletionVectors","v2Checkpoint","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[1522,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[2,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-4c7c7cd8-3a62-4e66-b3b9-e89f44612e30-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385166068,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385166068000","MIN_INSERTION_TIME":"1774385166068000","MAX_INSERTION_TIME":"1774385166068000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-b93d3d1f-a7ec-42f2-ade5-27837538abcd-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385165060,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385165060000","MIN_INSERTION_TIME":"1774385165060000","MAX_INSERTION_TIME":"1774385165060000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000002.json b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000002.json new file mode 100644 index 0000000000..17c0541a81 --- /dev/null +++ b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000002.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774385166186,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":1,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"f91439b4-5a79-4fc9-8550-e03818be0ef7"}} +{"add":{"path":"test%25file%25prefix-part-00000-4c7c7cd8-3a62-4e66-b3b9-e89f44612e30-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385166068,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385166068000","MIN_INSERTION_TIME":"1774385166068000","MAX_INSERTION_TIME":"1774385166068000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000003.crc b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000003.crc new file mode 100644 index 0000000000..982f3e145d --- /dev/null +++ b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000003.crc @@ -0,0 +1 @@ +{"txnId":"e6971bdb-020d-4964-8d37-eff3c426b4bf","tableSizeBytes":2282,"numFiles":3,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"aab52b7c-387a-4e1b-b804-4140a460bfd6","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"v2","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385164156},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors","v2Checkpoint"],"writerFeatures":["deletionVectors","v2Checkpoint","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[2282,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[3,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-4c7c7cd8-3a62-4e66-b3b9-e89f44612e30-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385166068,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385166068000","MIN_INSERTION_TIME":"1774385166068000","MAX_INSERTION_TIME":"1774385166068000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-b93d3d1f-a7ec-42f2-ade5-27837538abcd-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385165060,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385165060000","MIN_INSERTION_TIME":"1774385165060000","MAX_INSERTION_TIME":"1774385165060000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-68f487f2-d7f8-48d8-8b34-4a567dc046f4-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774385167032,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385167032000","MIN_INSERTION_TIME":"1774385167032000","MAX_INSERTION_TIME":"1774385167032000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000003.json b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000003.json new file mode 100644 index 0000000000..3c86b8921a --- /dev/null +++ b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000003.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774385167154,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":2,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"760"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"e6971bdb-020d-4964-8d37-eff3c426b4bf"}} +{"add":{"path":"test%25file%25prefix-part-00000-68f487f2-d7f8-48d8-8b34-4a567dc046f4-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774385167032,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385167032000","MIN_INSERTION_TIME":"1774385167032000","MAX_INSERTION_TIME":"1774385167032000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000004.crc b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000004.crc new file mode 100644 index 0000000000..ac5da9aa8f --- /dev/null +++ b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000004.crc @@ -0,0 +1 @@ +{"txnId":"82b91af6-eee8-4837-ab86-6b5becab5e6f","tableSizeBytes":3043,"numFiles":4,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"aab52b7c-387a-4e1b-b804-4140a460bfd6","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"v2","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385164156},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors","v2Checkpoint"],"writerFeatures":["deletionVectors","v2Checkpoint","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[3043,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[4,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-4c7c7cd8-3a62-4e66-b3b9-e89f44612e30-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385166068,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385166068000","MIN_INSERTION_TIME":"1774385166068000","MAX_INSERTION_TIME":"1774385166068000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-c89fe644-eade-4db6-b00f-6c5e1f8343ea-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385168156,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385168156000","MIN_INSERTION_TIME":"1774385168156000","MAX_INSERTION_TIME":"1774385168156000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-b93d3d1f-a7ec-42f2-ade5-27837538abcd-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385165060,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385165060000","MIN_INSERTION_TIME":"1774385165060000","MAX_INSERTION_TIME":"1774385165060000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-68f487f2-d7f8-48d8-8b34-4a567dc046f4-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774385167032,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385167032000","MIN_INSERTION_TIME":"1774385167032000","MAX_INSERTION_TIME":"1774385167032000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000004.json b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000004.json new file mode 100644 index 0000000000..803f4232cf --- /dev/null +++ b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000004.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774385168305,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":3,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"82b91af6-eee8-4837-ab86-6b5becab5e6f"}} +{"add":{"path":"test%25file%25prefix-part-00000-c89fe644-eade-4db6-b00f-6c5e1f8343ea-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385168156,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385168156000","MIN_INSERTION_TIME":"1774385168156000","MAX_INSERTION_TIME":"1774385168156000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000005.checkpoint.99cabe18-f541-4a52-b8fb-3f488d113032.json b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000005.checkpoint.99cabe18-f541-4a52-b8fb-3f488d113032.json new file mode 100644 index 0000000000..6b3fdc11dd --- /dev/null +++ b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000005.checkpoint.99cabe18-f541-4a52-b8fb-3f488d113032.json @@ -0,0 +1,4 @@ +{"checkpointMetadata":{"version":5,"tags":{"sidecarNumActions":"5","sidecarSizeInBytes":"12599","numOfAddFiles":"5","sidecarFileSchema":"{\"type\":\"struct\",\"fields\":[{\"name\":\"add\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"path\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"partitionValues\",\"type\":{\"type\":\"map\",\"keyType\":\"string\",\"valueType\":\"string\",\"valueContainsNull\":true},\"nullable\":true,\"metadata\":{}},{\"name\":\"size\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"modificationTime\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"dataChange\",\"type\":\"boolean\",\"nullable\":true,\"metadata\":{}},{\"name\":\"tags\",\"type\":{\"type\":\"map\",\"keyType\":\"string\",\"valueType\":\"string\",\"valueContainsNull\":true},\"nullable\":true,\"metadata\":{}},{\"name\":\"deletionVector\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"storageType\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"pathOrInlineDv\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"offset\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"sizeInBytes\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"cardinality\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"maxRowIndex\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"baseRowId\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"defaultRowCommitVersion\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"clusteringProvider\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"stats_parsed\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"numRecords\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"minValues\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"maxValues\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"nullCount\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"tightBounds\",\"type\":\"boolean\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"remove\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"path\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"deletionTimestamp\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"dataChange\",\"type\":\"boolean\",\"nullable\":true,\"metadata\":{}},{\"name\":\"extendedFileMetadata\",\"type\":\"boolean\",\"nullable\":true,\"metadata\":{}},{\"name\":\"partitionValues\",\"type\":{\"type\":\"map\",\"keyType\":\"string\",\"valueType\":\"string\",\"valueContainsNull\":true},\"nullable\":true,\"metadata\":{}},{\"name\":\"size\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"deletionVector\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"storageType\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"pathOrInlineDv\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"offset\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"sizeInBytes\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"cardinality\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"maxRowIndex\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"baseRowId\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"defaultRowCommitVersion\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}}]}"}}} +{"sidecar":{"path":"00000000000000000005.checkpoint.0000000001.0000000001.aec23d5c-e86d-4012-adcc-d4f08ad67230.parquet","sizeInBytes":12599,"modificationTime":1774385170196}} +{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors","v2Checkpoint"],"writerFeatures":["deletionVectors","v2Checkpoint","appendOnly","invariants"]}} +{"metaData":{"id":"aab52b7c-387a-4e1b-b804-4140a460bfd6","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"v2","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385164156}} diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000005.crc b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000005.crc new file mode 100644 index 0000000000..c8d7cd40f5 --- /dev/null +++ b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000005.crc @@ -0,0 +1 @@ +{"txnId":"0be3928d-1ddf-4a2b-aca5-d0ddc96fcf42","tableSizeBytes":3804,"numFiles":5,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"aab52b7c-387a-4e1b-b804-4140a460bfd6","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"v2","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385164156},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors","v2Checkpoint"],"writerFeatures":["deletionVectors","v2Checkpoint","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[3804,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[5,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-4c7c7cd8-3a62-4e66-b3b9-e89f44612e30-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385166068,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385166068000","MIN_INSERTION_TIME":"1774385166068000","MAX_INSERTION_TIME":"1774385166068000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-c89fe644-eade-4db6-b00f-6c5e1f8343ea-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385168156,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385168156000","MIN_INSERTION_TIME":"1774385168156000","MAX_INSERTION_TIME":"1774385168156000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-b93d3d1f-a7ec-42f2-ade5-27837538abcd-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385165060,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385165060000","MIN_INSERTION_TIME":"1774385165060000","MAX_INSERTION_TIME":"1774385165060000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-e4fcdaa7-dff0-4c20-b263-fc29f1cba12c-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385169176,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":5,\"value\":\"value_5\"},\"maxValues\":{\"id\":5,\"value\":\"value_5\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385169176000","MIN_INSERTION_TIME":"1774385169176000","MAX_INSERTION_TIME":"1774385169176000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-68f487f2-d7f8-48d8-8b34-4a567dc046f4-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774385167032,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385167032000","MIN_INSERTION_TIME":"1774385167032000","MAX_INSERTION_TIME":"1774385167032000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000005.json b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000005.json new file mode 100644 index 0000000000..ce26721926 --- /dev/null +++ b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/00000000000000000005.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774385169289,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":4,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"0be3928d-1ddf-4a2b-aca5-d0ddc96fcf42"}} +{"add":{"path":"test%25file%25prefix-part-00000-e4fcdaa7-dff0-4c20-b263-fc29f1cba12c-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385169176,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":5,\"value\":\"value_5\"},\"maxValues\":{\"id\":5,\"value\":\"value_5\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385169176000","MIN_INSERTION_TIME":"1774385169176000","MAX_INSERTION_TIME":"1774385169176000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/_last_checkpoint b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/_last_checkpoint new file mode 100644 index 0000000000..000935edfb --- /dev/null +++ b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/_last_checkpoint @@ -0,0 +1 @@ +{"version":5,"size":9,"sizeInBytes":17710,"numOfAddFiles":5,"v2Checkpoint":{"path":"00000000000000000005.checkpoint.99cabe18-f541-4a52-b8fb-3f488d113032.json","sizeInBytes":5111,"modificationTime":1774385170232,"nonFileActions":[{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors","v2Checkpoint"],"writerFeatures":["deletionVectors","v2Checkpoint","appendOnly","invariants"]}},{"metaData":{"id":"aab52b7c-387a-4e1b-b804-4140a460bfd6","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"v2","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385164156}},{"checkpointMetadata":{"version":5,"tags":{"sidecarNumActions":"5","sidecarSizeInBytes":"12599","numOfAddFiles":"5","sidecarFileSchema":"{\"type\":\"struct\",\"fields\":[{\"name\":\"add\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"path\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"partitionValues\",\"type\":{\"type\":\"map\",\"keyType\":\"string\",\"valueType\":\"string\",\"valueContainsNull\":true},\"nullable\":true,\"metadata\":{}},{\"name\":\"size\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"modificationTime\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"dataChange\",\"type\":\"boolean\",\"nullable\":true,\"metadata\":{}},{\"name\":\"tags\",\"type\":{\"type\":\"map\",\"keyType\":\"string\",\"valueType\":\"string\",\"valueContainsNull\":true},\"nullable\":true,\"metadata\":{}},{\"name\":\"deletionVector\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"storageType\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"pathOrInlineDv\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"offset\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"sizeInBytes\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"cardinality\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"maxRowIndex\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"baseRowId\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"defaultRowCommitVersion\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"clusteringProvider\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"stats_parsed\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"numRecords\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"minValues\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"maxValues\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"nullCount\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"tightBounds\",\"type\":\"boolean\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"remove\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"path\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"deletionTimestamp\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"dataChange\",\"type\":\"boolean\",\"nullable\":true,\"metadata\":{}},{\"name\":\"extendedFileMetadata\",\"type\":\"boolean\",\"nullable\":true,\"metadata\":{}},{\"name\":\"partitionValues\",\"type\":{\"type\":\"map\",\"keyType\":\"string\",\"valueType\":\"string\",\"valueContainsNull\":true},\"nullable\":true,\"metadata\":{}},{\"name\":\"size\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"deletionVector\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"storageType\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"pathOrInlineDv\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"offset\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"sizeInBytes\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"cardinality\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"maxRowIndex\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"baseRowId\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"defaultRowCommitVersion\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}}]}"}}}],"sidecarFiles":[{"path":"00000000000000000005.checkpoint.0000000001.0000000001.aec23d5c-e86d-4012-adcc-d4f08ad67230.parquet","sizeInBytes":12599,"modificationTime":1774385170196}]},"checksum":"eb36d5d7ee6a2d503cbfd76b2f116b2e"} diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/_sidecars/00000000000000000005.checkpoint.0000000001.0000000001.aec23d5c-e86d-4012-adcc-d4f08ad67230.parquet b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/_sidecars/00000000000000000005.checkpoint.0000000001.0000000001.aec23d5c-e86d-4012-adcc-d4f08ad67230.parquet new file mode 100644 index 0000000000..8507e8d4a7 Binary files /dev/null and b/kernel/tests/data/v2-json-sidecars-struct-stats-only/_delta_log/_sidecars/00000000000000000005.checkpoint.0000000001.0000000001.aec23d5c-e86d-4012-adcc-d4f08ad67230.parquet differ diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/test%file%prefix-part-00000-4c7c7cd8-3a62-4e66-b3b9-e89f44612e30-c000.snappy.parquet b/kernel/tests/data/v2-json-sidecars-struct-stats-only/test%file%prefix-part-00000-4c7c7cd8-3a62-4e66-b3b9-e89f44612e30-c000.snappy.parquet new file mode 100644 index 0000000000..b6aabb8552 Binary files /dev/null and b/kernel/tests/data/v2-json-sidecars-struct-stats-only/test%file%prefix-part-00000-4c7c7cd8-3a62-4e66-b3b9-e89f44612e30-c000.snappy.parquet differ diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/test%file%prefix-part-00000-68f487f2-d7f8-48d8-8b34-4a567dc046f4-c000.snappy.parquet b/kernel/tests/data/v2-json-sidecars-struct-stats-only/test%file%prefix-part-00000-68f487f2-d7f8-48d8-8b34-4a567dc046f4-c000.snappy.parquet new file mode 100644 index 0000000000..9713b2d963 Binary files /dev/null and b/kernel/tests/data/v2-json-sidecars-struct-stats-only/test%file%prefix-part-00000-68f487f2-d7f8-48d8-8b34-4a567dc046f4-c000.snappy.parquet differ diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/test%file%prefix-part-00000-b93d3d1f-a7ec-42f2-ade5-27837538abcd-c000.snappy.parquet b/kernel/tests/data/v2-json-sidecars-struct-stats-only/test%file%prefix-part-00000-b93d3d1f-a7ec-42f2-ade5-27837538abcd-c000.snappy.parquet new file mode 100644 index 0000000000..e23848797a Binary files /dev/null and b/kernel/tests/data/v2-json-sidecars-struct-stats-only/test%file%prefix-part-00000-b93d3d1f-a7ec-42f2-ade5-27837538abcd-c000.snappy.parquet differ diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/test%file%prefix-part-00000-c89fe644-eade-4db6-b00f-6c5e1f8343ea-c000.snappy.parquet b/kernel/tests/data/v2-json-sidecars-struct-stats-only/test%file%prefix-part-00000-c89fe644-eade-4db6-b00f-6c5e1f8343ea-c000.snappy.parquet new file mode 100644 index 0000000000..bfc9dcfa44 Binary files /dev/null and b/kernel/tests/data/v2-json-sidecars-struct-stats-only/test%file%prefix-part-00000-c89fe644-eade-4db6-b00f-6c5e1f8343ea-c000.snappy.parquet differ diff --git a/kernel/tests/data/v2-json-sidecars-struct-stats-only/test%file%prefix-part-00000-e4fcdaa7-dff0-4c20-b263-fc29f1cba12c-c000.snappy.parquet b/kernel/tests/data/v2-json-sidecars-struct-stats-only/test%file%prefix-part-00000-e4fcdaa7-dff0-4c20-b263-fc29f1cba12c-c000.snappy.parquet new file mode 100644 index 0000000000..1f8980ad29 Binary files /dev/null and b/kernel/tests/data/v2-json-sidecars-struct-stats-only/test%file%prefix-part-00000-e4fcdaa7-dff0-4c20-b263-fc29f1cba12c-c000.snappy.parquet differ diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000000.crc b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000000.crc new file mode 100644 index 0000000000..ad431e43f1 --- /dev/null +++ b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000000.crc @@ -0,0 +1 @@ +{"txnId":"d029b8be-4b6c-40be-8c98-051174082a0f","tableSizeBytes":0,"numFiles":0,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"69f7cd6b-63b4-4e31-85b1-450ac0a9f70e","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"v2","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385157314},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors","v2Checkpoint"],"writerFeatures":["deletionVectors","v2Checkpoint","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[0,0,0,0,0,0,0,0,0,0]},"allFiles":[]} diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000000.json b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000000.json new file mode 100644 index 0000000000..d48e73bd6e --- /dev/null +++ b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000000.json @@ -0,0 +1,3 @@ +{"commitInfo":{"timestamp":1774385157417,"operation":"CREATE TABLE","operationParameters":{"partitionBy":"[]","clusterBy":"[]","description":null,"isManaged":"false","properties":"{\"delta.enableDeletionVectors\":\"true\",\"delta.checkpoint.writeStatsAsStruct\":\"true\",\"delta.checkpointPolicy\":\"v2\",\"delta.checkpoint.writeStatsAsJson\":\"false\",\"delta.checkpointInterval\":\"10\"}","statsOnLoad":false},"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{},"tags":{"restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"d029b8be-4b6c-40be-8c98-051174082a0f"}} +{"metaData":{"id":"69f7cd6b-63b4-4e31-85b1-450ac0a9f70e","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"v2","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385157314}} +{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors","v2Checkpoint"],"writerFeatures":["deletionVectors","v2Checkpoint","appendOnly","invariants"]}} diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000001.crc b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000001.crc new file mode 100644 index 0000000000..52bd9eeeb8 --- /dev/null +++ b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000001.crc @@ -0,0 +1 @@ +{"txnId":"508beb32-0b1f-4e43-aea3-95dd221a0ece","tableSizeBytes":761,"numFiles":1,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"69f7cd6b-63b4-4e31-85b1-450ac0a9f70e","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"v2","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385157314},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors","v2Checkpoint"],"writerFeatures":["deletionVectors","v2Checkpoint","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[761,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[1,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-ba9dae5e-b947-4940-961f-6cf6a3c2fbfe-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385158244,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385158244000","MIN_INSERTION_TIME":"1774385158244000","MAX_INSERTION_TIME":"1774385158244000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000001.json b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000001.json new file mode 100644 index 0000000000..3ac90fab5f --- /dev/null +++ b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000001.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774385158387,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":0,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"508beb32-0b1f-4e43-aea3-95dd221a0ece"}} +{"add":{"path":"test%25file%25prefix-part-00000-ba9dae5e-b947-4940-961f-6cf6a3c2fbfe-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385158244,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385158244000","MIN_INSERTION_TIME":"1774385158244000","MAX_INSERTION_TIME":"1774385158244000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000002.crc b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000002.crc new file mode 100644 index 0000000000..442510ddbb --- /dev/null +++ b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000002.crc @@ -0,0 +1 @@ +{"txnId":"4d25d6cc-be1d-47bf-b39c-035cdbbf3cec","tableSizeBytes":1522,"numFiles":2,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"69f7cd6b-63b4-4e31-85b1-450ac0a9f70e","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"v2","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385157314},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors","v2Checkpoint"],"writerFeatures":["deletionVectors","v2Checkpoint","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[1522,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[2,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-ba9dae5e-b947-4940-961f-6cf6a3c2fbfe-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385158244,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385158244000","MIN_INSERTION_TIME":"1774385158244000","MAX_INSERTION_TIME":"1774385158244000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-2ce1aba3-6914-48d4-b2b3-9cc25d61b211-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385159272,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385159272000","MIN_INSERTION_TIME":"1774385159272000","MAX_INSERTION_TIME":"1774385159272000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000002.json b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000002.json new file mode 100644 index 0000000000..325e641bae --- /dev/null +++ b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000002.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774385159401,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":1,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"4d25d6cc-be1d-47bf-b39c-035cdbbf3cec"}} +{"add":{"path":"test%25file%25prefix-part-00000-2ce1aba3-6914-48d4-b2b3-9cc25d61b211-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385159272,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385159272000","MIN_INSERTION_TIME":"1774385159272000","MAX_INSERTION_TIME":"1774385159272000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000003.crc b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000003.crc new file mode 100644 index 0000000000..5fa5dbc61e --- /dev/null +++ b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000003.crc @@ -0,0 +1 @@ +{"txnId":"767c1612-d46b-4d50-b16d-7aa2d0a55e8b","tableSizeBytes":2282,"numFiles":3,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"69f7cd6b-63b4-4e31-85b1-450ac0a9f70e","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"v2","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385157314},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors","v2Checkpoint"],"writerFeatures":["deletionVectors","v2Checkpoint","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[2282,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[3,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-ba9dae5e-b947-4940-961f-6cf6a3c2fbfe-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385158244,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385158244000","MIN_INSERTION_TIME":"1774385158244000","MAX_INSERTION_TIME":"1774385158244000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-7268d56a-7420-4691-8062-5b0c3dbdd803-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774385160264,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385160264000","MIN_INSERTION_TIME":"1774385160264000","MAX_INSERTION_TIME":"1774385160264000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-2ce1aba3-6914-48d4-b2b3-9cc25d61b211-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385159272,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385159272000","MIN_INSERTION_TIME":"1774385159272000","MAX_INSERTION_TIME":"1774385159272000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000003.json b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000003.json new file mode 100644 index 0000000000..03557b82ac --- /dev/null +++ b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000003.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774385160385,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":2,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"760"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"767c1612-d46b-4d50-b16d-7aa2d0a55e8b"}} +{"add":{"path":"test%25file%25prefix-part-00000-7268d56a-7420-4691-8062-5b0c3dbdd803-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774385160264,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385160264000","MIN_INSERTION_TIME":"1774385160264000","MAX_INSERTION_TIME":"1774385160264000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000004.crc b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000004.crc new file mode 100644 index 0000000000..8d072b8526 --- /dev/null +++ b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000004.crc @@ -0,0 +1 @@ +{"txnId":"23458c90-e99f-4f42-bcaa-3a00fab3a35d","tableSizeBytes":3043,"numFiles":4,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"69f7cd6b-63b4-4e31-85b1-450ac0a9f70e","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"v2","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385157314},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors","v2Checkpoint"],"writerFeatures":["deletionVectors","v2Checkpoint","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[3043,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[4,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-ea8ba2ac-45e6-497c-99e5-ad5b5e78920e-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385161268,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385161268000","MIN_INSERTION_TIME":"1774385161268000","MAX_INSERTION_TIME":"1774385161268000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-ba9dae5e-b947-4940-961f-6cf6a3c2fbfe-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385158244,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385158244000","MIN_INSERTION_TIME":"1774385158244000","MAX_INSERTION_TIME":"1774385158244000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-7268d56a-7420-4691-8062-5b0c3dbdd803-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774385160264,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385160264000","MIN_INSERTION_TIME":"1774385160264000","MAX_INSERTION_TIME":"1774385160264000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-2ce1aba3-6914-48d4-b2b3-9cc25d61b211-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385159272,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385159272000","MIN_INSERTION_TIME":"1774385159272000","MAX_INSERTION_TIME":"1774385159272000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000004.json b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000004.json new file mode 100644 index 0000000000..3e4a42d40a --- /dev/null +++ b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000004.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774385161395,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":3,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"23458c90-e99f-4f42-bcaa-3a00fab3a35d"}} +{"add":{"path":"test%25file%25prefix-part-00000-ea8ba2ac-45e6-497c-99e5-ad5b5e78920e-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385161268,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385161268000","MIN_INSERTION_TIME":"1774385161268000","MAX_INSERTION_TIME":"1774385161268000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000005.checkpoint.f376503f-80c5-44c4-a353-a741181e8197.parquet b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000005.checkpoint.f376503f-80c5-44c4-a353-a741181e8197.parquet new file mode 100644 index 0000000000..ccef7737ad Binary files /dev/null and b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000005.checkpoint.f376503f-80c5-44c4-a353-a741181e8197.parquet differ diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000005.crc b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000005.crc new file mode 100644 index 0000000000..e6df7f936b --- /dev/null +++ b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000005.crc @@ -0,0 +1 @@ +{"txnId":"b61e8d13-f21c-44a5-b3e2-73e5b2cf4067","tableSizeBytes":3804,"numFiles":5,"numDeletedRecordsOpt":0,"numDeletionVectorsOpt":0,"numMetadata":1,"numProtocol":1,"setTransactions":[],"domainMetadata":[],"metadata":{"id":"69f7cd6b-63b4-4e31-85b1-450ac0a9f70e","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"v2","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385157314},"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors","v2Checkpoint"],"writerFeatures":["deletionVectors","v2Checkpoint","appendOnly","invariants"]},"histogramOpt":{"sortedBinBoundaries":[0,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,12582912,16777216,20971520,25165824,29360128,33554432,37748736,41943040,50331648,58720256,67108864,75497472,83886080,92274688,100663296,109051904,117440512,125829120,130023424,134217728,138412032,142606336,146800640,150994944,167772160,184549376,201326592,218103808,234881024,251658240,268435456,285212672,301989888,318767104,335544320,352321536,369098752,385875968,402653184,419430400,436207616,452984832,469762048,486539264,503316480,520093696,536870912,553648128,570425344,587202560,603979776,671088640,738197504,805306368,872415232,939524096,1006632960,1073741824,1140850688,1207959552,1275068416,1342177280,1409286144,1476395008,1610612736,1744830464,1879048192,2013265920,2147483648,2415919104,2684354560,2952790016,3221225472,3489660928,3758096384,4026531840,4294967296,8589934592,17179869184,34359738368,68719476736,137438953472,274877906944],"fileCounts":[5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"totalBytes":[3804,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"deletedRecordCountsHistogramOpt":{"deletedRecordCounts":[5,0,0,0,0,0,0,0,0,0]},"allFiles":[{"path":"test%25file%25prefix-part-00000-ea8ba2ac-45e6-497c-99e5-ad5b5e78920e-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385161268,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":4,\"value\":\"value_4\"},\"maxValues\":{\"id\":4,\"value\":\"value_4\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385161268000","MIN_INSERTION_TIME":"1774385161268000","MAX_INSERTION_TIME":"1774385161268000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-ba9dae5e-b947-4940-961f-6cf6a3c2fbfe-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385158244,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"value\":\"value_1\"},\"maxValues\":{\"id\":1,\"value\":\"value_1\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385158244000","MIN_INSERTION_TIME":"1774385158244000","MAX_INSERTION_TIME":"1774385158244000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-8a3d46e0-9431-4cbd-b97d-6b62a2be1dc8-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385162256,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":5,\"value\":\"value_5\"},\"maxValues\":{\"id\":5,\"value\":\"value_5\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385162256000","MIN_INSERTION_TIME":"1774385162256000","MAX_INSERTION_TIME":"1774385162256000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-7268d56a-7420-4691-8062-5b0c3dbdd803-c000.snappy.parquet","partitionValues":{},"size":760,"modificationTime":1774385160264,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3,\"value\":\"value_3\"},\"maxValues\":{\"id\":3,\"value\":\"value_3\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385160264000","MIN_INSERTION_TIME":"1774385160264000","MAX_INSERTION_TIME":"1774385160264000","OPTIMIZE_TARGET_SIZE":"268435456"}},{"path":"test%25file%25prefix-part-00000-2ce1aba3-6914-48d4-b2b3-9cc25d61b211-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385159272,"dataChange":false,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2,\"value\":\"value_2\"},\"maxValues\":{\"id\":2,\"value\":\"value_2\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385159272000","MIN_INSERTION_TIME":"1774385159272000","MAX_INSERTION_TIME":"1774385159272000","OPTIMIZE_TARGET_SIZE":"268435456"}}]} diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000005.json b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000005.json new file mode 100644 index 0000000000..abc47ddb65 --- /dev/null +++ b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/00000000000000000005.json @@ -0,0 +1,2 @@ +{"commitInfo":{"timestamp":1774385162370,"operation":"WRITE","operationParameters":{"mode":"Append","statsOnLoad":false,"partitionBy":"[]"},"readVersion":4,"isolationLevel":"WriteSerializable","isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputRows":"1","numOutputBytes":"761"},"tags":{"noRowsCopied":"true","restoresDeletedRows":"false"},"engineInfo":"Databricks-Runtime/","txnId":"b61e8d13-f21c-44a5-b3e2-73e5b2cf4067"}} +{"add":{"path":"test%25file%25prefix-part-00000-8a3d46e0-9431-4cbd-b97d-6b62a2be1dc8-c000.snappy.parquet","partitionValues":{},"size":761,"modificationTime":1774385162256,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":5,\"value\":\"value_5\"},\"maxValues\":{\"id\":5,\"value\":\"value_5\"},\"nullCount\":{\"id\":0,\"value\":0},\"tightBounds\":true}","tags":{"INSERTION_TIME":"1774385162256000","MIN_INSERTION_TIME":"1774385162256000","MAX_INSERTION_TIME":"1774385162256000","OPTIMIZE_TARGET_SIZE":"268435456"}}} diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/_last_checkpoint b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/_last_checkpoint new file mode 100644 index 0000000000..03ff778cfb --- /dev/null +++ b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/_last_checkpoint @@ -0,0 +1 @@ +{"version":5,"size":9,"sizeInBytes":37325,"numOfAddFiles":5,"checkpointSchema":{"type":"struct","fields":[{"name":"txn","type":{"type":"struct","fields":[{"name":"appId","type":"string","nullable":true,"metadata":{}},{"name":"version","type":"long","nullable":true,"metadata":{}},{"name":"lastUpdated","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"add","type":{"type":"struct","fields":[{"name":"path","type":"string","nullable":true,"metadata":{}},{"name":"partitionValues","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"size","type":"long","nullable":true,"metadata":{}},{"name":"modificationTime","type":"long","nullable":true,"metadata":{}},{"name":"dataChange","type":"boolean","nullable":true,"metadata":{}},{"name":"stats","type":"string","nullable":true,"metadata":{}},{"name":"tags","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"deletionVector","type":{"type":"struct","fields":[{"name":"storageType","type":"string","nullable":true,"metadata":{}},{"name":"pathOrInlineDv","type":"string","nullable":true,"metadata":{}},{"name":"offset","type":"integer","nullable":true,"metadata":{}},{"name":"sizeInBytes","type":"integer","nullable":true,"metadata":{}},{"name":"cardinality","type":"long","nullable":true,"metadata":{}},{"name":"maxRowIndex","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"baseRowId","type":"long","nullable":true,"metadata":{}},{"name":"defaultRowCommitVersion","type":"long","nullable":true,"metadata":{}},{"name":"clusteringProvider","type":"string","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"remove","type":{"type":"struct","fields":[{"name":"path","type":"string","nullable":true,"metadata":{}},{"name":"deletionTimestamp","type":"long","nullable":true,"metadata":{}},{"name":"dataChange","type":"boolean","nullable":true,"metadata":{}},{"name":"extendedFileMetadata","type":"boolean","nullable":true,"metadata":{}},{"name":"partitionValues","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"size","type":"long","nullable":true,"metadata":{}},{"name":"tags","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"deletionVector","type":{"type":"struct","fields":[{"name":"storageType","type":"string","nullable":true,"metadata":{}},{"name":"pathOrInlineDv","type":"string","nullable":true,"metadata":{}},{"name":"offset","type":"integer","nullable":true,"metadata":{}},{"name":"sizeInBytes","type":"integer","nullable":true,"metadata":{}},{"name":"cardinality","type":"long","nullable":true,"metadata":{}},{"name":"maxRowIndex","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"baseRowId","type":"long","nullable":true,"metadata":{}},{"name":"defaultRowCommitVersion","type":"long","nullable":true,"metadata":{}},{"name":"stats","type":"string","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"metaData","type":{"type":"struct","fields":[{"name":"id","type":"string","nullable":true,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"description","type":"string","nullable":true,"metadata":{}},{"name":"format","type":{"type":"struct","fields":[{"name":"provider","type":"string","nullable":true,"metadata":{}},{"name":"options","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"schemaString","type":"string","nullable":true,"metadata":{}},{"name":"partitionColumns","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"configuration","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}},{"name":"createdTime","type":"long","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"protocol","type":{"type":"struct","fields":[{"name":"minReaderVersion","type":"integer","nullable":true,"metadata":{}},{"name":"minWriterVersion","type":"integer","nullable":true,"metadata":{}},{"name":"readerFeatures","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"writerFeatures","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"domainMetadata","type":{"type":"struct","fields":[{"name":"domain","type":"string","nullable":true,"metadata":{}},{"name":"configuration","type":"string","nullable":true,"metadata":{}},{"name":"removed","type":"boolean","nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"checkpointMetadata","type":{"type":"struct","fields":[{"name":"version","type":"long","nullable":true,"metadata":{}},{"name":"tags","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"sidecar","type":{"type":"struct","fields":[{"name":"path","type":"string","nullable":true,"metadata":{}},{"name":"sizeInBytes","type":"long","nullable":true,"metadata":{}},{"name":"modificationTime","type":"long","nullable":true,"metadata":{}},{"name":"tags","type":{"type":"map","keyType":"string","valueType":"string","valueContainsNull":true},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}}]},"v2Checkpoint":{"path":"00000000000000000005.checkpoint.f376503f-80c5-44c4-a353-a741181e8197.parquet","sizeInBytes":24737,"modificationTime":1774385163992,"nonFileActions":[{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors","v2Checkpoint"],"writerFeatures":["deletionVectors","v2Checkpoint","appendOnly","invariants"]}},{"metaData":{"id":"69f7cd6b-63b4-4e31-85b1-450ac0a9f70e","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableDeletionVectors":"true","delta.checkpoint.writeStatsAsStruct":"true","delta.checkpointPolicy":"v2","delta.checkpoint.writeStatsAsJson":"false","delta.checkpointInterval":"10"},"createdTime":1774385157314}},{"checkpointMetadata":{"version":5,"tags":{"sidecarNumActions":"5","sidecarSizeInBytes":"12588","numOfAddFiles":"5","sidecarFileSchema":"{\"type\":\"struct\",\"fields\":[{\"name\":\"add\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"path\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"partitionValues\",\"type\":{\"type\":\"map\",\"keyType\":\"string\",\"valueType\":\"string\",\"valueContainsNull\":true},\"nullable\":true,\"metadata\":{}},{\"name\":\"size\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"modificationTime\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"dataChange\",\"type\":\"boolean\",\"nullable\":true,\"metadata\":{}},{\"name\":\"tags\",\"type\":{\"type\":\"map\",\"keyType\":\"string\",\"valueType\":\"string\",\"valueContainsNull\":true},\"nullable\":true,\"metadata\":{}},{\"name\":\"deletionVector\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"storageType\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"pathOrInlineDv\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"offset\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"sizeInBytes\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"cardinality\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"maxRowIndex\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"baseRowId\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"defaultRowCommitVersion\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"clusteringProvider\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"stats_parsed\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"numRecords\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"minValues\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"maxValues\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"nullCount\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"tightBounds\",\"type\":\"boolean\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"remove\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"path\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"deletionTimestamp\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"dataChange\",\"type\":\"boolean\",\"nullable\":true,\"metadata\":{}},{\"name\":\"extendedFileMetadata\",\"type\":\"boolean\",\"nullable\":true,\"metadata\":{}},{\"name\":\"partitionValues\",\"type\":{\"type\":\"map\",\"keyType\":\"string\",\"valueType\":\"string\",\"valueContainsNull\":true},\"nullable\":true,\"metadata\":{}},{\"name\":\"size\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"deletionVector\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"storageType\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"pathOrInlineDv\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"offset\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"sizeInBytes\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"cardinality\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"maxRowIndex\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"baseRowId\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"defaultRowCommitVersion\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}}]}"}}}],"sidecarFiles":[{"path":"00000000000000000005.checkpoint.0000000001.0000000001.014cb627-30e0-46dd-a539-09d3247e9b7d.parquet","sizeInBytes":12588,"modificationTime":1774385163552}]},"checksum":"616f829d2bc1e5b6569609a6155cbf04"} diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/_sidecars/00000000000000000005.checkpoint.0000000001.0000000001.014cb627-30e0-46dd-a539-09d3247e9b7d.parquet b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/_sidecars/00000000000000000005.checkpoint.0000000001.0000000001.014cb627-30e0-46dd-a539-09d3247e9b7d.parquet new file mode 100644 index 0000000000..2630b09108 Binary files /dev/null and b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/_delta_log/_sidecars/00000000000000000005.checkpoint.0000000001.0000000001.014cb627-30e0-46dd-a539-09d3247e9b7d.parquet differ diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/test%file%prefix-part-00000-2ce1aba3-6914-48d4-b2b3-9cc25d61b211-c000.snappy.parquet b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/test%file%prefix-part-00000-2ce1aba3-6914-48d4-b2b3-9cc25d61b211-c000.snappy.parquet new file mode 100644 index 0000000000..b6aabb8552 Binary files /dev/null and b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/test%file%prefix-part-00000-2ce1aba3-6914-48d4-b2b3-9cc25d61b211-c000.snappy.parquet differ diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/test%file%prefix-part-00000-7268d56a-7420-4691-8062-5b0c3dbdd803-c000.snappy.parquet b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/test%file%prefix-part-00000-7268d56a-7420-4691-8062-5b0c3dbdd803-c000.snappy.parquet new file mode 100644 index 0000000000..9713b2d963 Binary files /dev/null and b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/test%file%prefix-part-00000-7268d56a-7420-4691-8062-5b0c3dbdd803-c000.snappy.parquet differ diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/test%file%prefix-part-00000-8a3d46e0-9431-4cbd-b97d-6b62a2be1dc8-c000.snappy.parquet b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/test%file%prefix-part-00000-8a3d46e0-9431-4cbd-b97d-6b62a2be1dc8-c000.snappy.parquet new file mode 100644 index 0000000000..1f8980ad29 Binary files /dev/null and b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/test%file%prefix-part-00000-8a3d46e0-9431-4cbd-b97d-6b62a2be1dc8-c000.snappy.parquet differ diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/test%file%prefix-part-00000-ba9dae5e-b947-4940-961f-6cf6a3c2fbfe-c000.snappy.parquet b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/test%file%prefix-part-00000-ba9dae5e-b947-4940-961f-6cf6a3c2fbfe-c000.snappy.parquet new file mode 100644 index 0000000000..e23848797a Binary files /dev/null and b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/test%file%prefix-part-00000-ba9dae5e-b947-4940-961f-6cf6a3c2fbfe-c000.snappy.parquet differ diff --git a/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/test%file%prefix-part-00000-ea8ba2ac-45e6-497c-99e5-ad5b5e78920e-c000.snappy.parquet b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/test%file%prefix-part-00000-ea8ba2ac-45e6-497c-99e5-ad5b5e78920e-c000.snappy.parquet new file mode 100644 index 0000000000..bfc9dcfa44 Binary files /dev/null and b/kernel/tests/data/v2-parquet-sidecars-struct-stats-only/test%file%prefix-part-00000-ea8ba2ac-45e6-497c-99e5-ad5b5e78920e-c000.snappy.parquet differ diff --git a/kernel/tests/data/void-column/_delta_log/.00000000000000000000.crc.crc b/kernel/tests/data/void-column/_delta_log/.00000000000000000000.crc.crc new file mode 100644 index 0000000000..d6cc2ccac6 Binary files /dev/null and b/kernel/tests/data/void-column/_delta_log/.00000000000000000000.crc.crc differ diff --git a/kernel/tests/data/void-column/_delta_log/.00000000000000000000.json.crc b/kernel/tests/data/void-column/_delta_log/.00000000000000000000.json.crc new file mode 100644 index 0000000000..50d047480a Binary files /dev/null and b/kernel/tests/data/void-column/_delta_log/.00000000000000000000.json.crc differ diff --git a/kernel/tests/data/void-column/_delta_log/00000000000000000000.json b/kernel/tests/data/void-column/_delta_log/00000000000000000000.json new file mode 100644 index 0000000000..52f057cd4b --- /dev/null +++ b/kernel/tests/data/void-column/_delta_log/00000000000000000000.json @@ -0,0 +1,6 @@ +{"commitInfo":{"timestamp":1771623961050,"operation":"WRITE","operationParameters":{"mode":"Overwrite","partitionBy":"[]"},"isolationLevel":"Serializable","isBlindAppend":false,"operationMetrics":{"numFiles":"4","numRemovedFiles":"0","numRemovedBytes":"0","numOutputRows":"3","numOutputBytes":"1736"},"engineInfo":"Apache-Spark/4.1.1 Delta-Lake/4.1.0","txnId":"5f17b8b2-6e4c-4873-813d-61aade2eca87"}} +{"metaData":{"id":"7c6cbb63-8b75-4c79-b75f-fa3d72ca5c7c","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"void_col\",\"type\":\"void\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1771623958969}} +{"protocol":{"minReaderVersion":1,"minWriterVersion":2}} +{"add":{"path":"part-00010-f5c1ad06-423d-4713-9407-39b8be13ec24-c000.snappy.parquet","partitionValues":{},"size":480,"modificationTime":1771623960477,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":1},\"maxValues\":{\"id\":1},\"nullCount\":{\"id\":0,\"void_col\":1}}"}} +{"add":{"path":"part-00021-cfd890b1-faad-4e47-b913-dfd49366d746-c000.snappy.parquet","partitionValues":{},"size":480,"modificationTime":1771623960477,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":2},\"maxValues\":{\"id\":2},\"nullCount\":{\"id\":0,\"void_col\":1}}"}} +{"add":{"path":"part-00031-4c9bec96-ae06-4ea8-808b-4e09b44d088b-c000.snappy.parquet","partitionValues":{},"size":480,"modificationTime":1771623960477,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"id\":3},\"maxValues\":{\"id\":3},\"nullCount\":{\"id\":0,\"void_col\":1}}"}} diff --git a/kernel/tests/data/void-column/part-00010-f5c1ad06-423d-4713-9407-39b8be13ec24-c000.snappy.parquet b/kernel/tests/data/void-column/part-00010-f5c1ad06-423d-4713-9407-39b8be13ec24-c000.snappy.parquet new file mode 100644 index 0000000000..e70d9ffec1 Binary files /dev/null and b/kernel/tests/data/void-column/part-00010-f5c1ad06-423d-4713-9407-39b8be13ec24-c000.snappy.parquet differ diff --git a/kernel/tests/data/void-column/part-00021-cfd890b1-faad-4e47-b913-dfd49366d746-c000.snappy.parquet b/kernel/tests/data/void-column/part-00021-cfd890b1-faad-4e47-b913-dfd49366d746-c000.snappy.parquet new file mode 100644 index 0000000000..42c0f0d410 Binary files /dev/null and b/kernel/tests/data/void-column/part-00021-cfd890b1-faad-4e47-b913-dfd49366d746-c000.snappy.parquet differ diff --git a/kernel/tests/data/void-column/part-00031-4c9bec96-ae06-4ea8-808b-4e09b44d088b-c000.snappy.parquet b/kernel/tests/data/void-column/part-00031-4c9bec96-ae06-4ea8-808b-4e09b44d088b-c000.snappy.parquet new file mode 100644 index 0000000000..33f98d11cf Binary files /dev/null and b/kernel/tests/data/void-column/part-00031-4c9bec96-ae06-4ea8-808b-4e09b44d088b-c000.snappy.parquet differ diff --git a/kernel/tests/incremental_scan.rs b/kernel/tests/incremental_scan.rs new file mode 100644 index 0000000000..a4428eb5db --- /dev/null +++ b/kernel/tests/incremental_scan.rs @@ -0,0 +1,1993 @@ +//! Integration tests for [`IncrementalScanBuilder`]. +//! +//! These cover the dedup edge cases that were known to bite consumers doing their own +//! cross-commit deduplication: cross-commit cancellation, chained add/remove/add, log +//! compaction interactions, catalog-managed staged commits, and vacuum races. + +use std::collections::{HashMap, HashSet}; +use std::num::NonZero; +use std::sync::Arc; + +use delta_kernel::arrow::array::{Int32Array, RecordBatch}; +use delta_kernel::arrow::datatypes::Schema as ArrowSchema; +use delta_kernel::engine::arrow_conversion::TryIntoArrow as _; +use delta_kernel::expressions::{col, lit, Predicate as Pred, PredicateRef}; +use delta_kernel::incremental_scan::{ + IncrementalListing, IncrementalListingAgainstBase, IncrementalScanStream, + IncrementalScanSummary, +}; +use delta_kernel::log_replay::FileActionKey; +use delta_kernel::object_store::memory::InMemory; +use delta_kernel::object_store::path::Path as ObjectStorePath; +use delta_kernel::object_store::ObjectStoreExt; +use delta_kernel::scan::state::ScanFile; +use delta_kernel::schema::{DataType, StructField, StructType}; +use delta_kernel::{ + Engine, EvaluationHandler, JsonHandler, ParquetHandler, Snapshot, SnapshotRef, StorageHandler, +}; +use rstest::rstest; +use test_utils::delta_kernel_default_engine::executor::tokio::{ + TokioBackgroundExecutor, TokioMultiThreadExecutor, +}; +use test_utils::delta_kernel_default_engine::json::DefaultJsonHandler; +use test_utils::delta_kernel_default_engine::{DefaultEngine, DefaultEngineBuilder}; +use test_utils::{ + actions_to_string, actions_to_string_catalog_managed, actions_to_string_partitioned, + add_commit, add_staged_commit, assert_result_error_with_message, + compacted_log_path_for_versions, create_log_path, create_table_and_load_snapshot, + delta_path_for_version, test_table_setup_mt, write_batch_to_table, TestAction, +}; +use url::Url; + +fn setup_test() -> ( + Arc, + Arc>, + Url, +) { + let storage = Arc::new(InMemory::new()); + let table_root = Url::parse("memory:///").unwrap(); + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + (storage, engine, table_root) +} + +fn key(path: &str) -> FileActionKey { + FileActionKey::new(path, None) +} + +fn key_with_dv(path: &str, dv_unique_id: &str) -> FileActionKey { + FileActionKey::new(path, Some(dv_unique_id.to_string())) +} + +fn live_add_count(listing: &IncrementalListing) -> usize { + listing + .add_files + .iter() + .map(|f| f.selection_vector().iter().filter(|s| **s).count()) + .sum() +} + +fn unwrap_listing(result: Option) -> IncrementalListing { + result + .expect("expected Some(stream), got None (commits unavailable)") + .into_listing() + .expect("into_listing succeeded") +} + +// Cancellation: walking newest-first with `(path, dv_unique_id)` first-seen-wins, +// every Add whose path matches a later Remove gets cancelled. Cases vary the chain +// depth: a single add-then-remove pair (no chain) versus a 3-commit compaction +// chain where every intermediate add is cancelled by the next commit's remove. +#[rstest] +#[case::single_pair( + vec![ + vec![TestAction::Add("X".to_string())], + vec![TestAction::Remove("X".to_string())], + ], + 0, + vec!["X"], +)] +#[case::compaction_chain( + vec![ + vec![TestAction::Add("A".to_string())], + vec![TestAction::Add("B".to_string()), TestAction::Remove("A".to_string())], + vec![TestAction::Add("C".to_string()), TestAction::Remove("B".to_string())], + ], + 1, + vec!["A", "B"], +)] +#[tokio::test] +async fn within_range_cancellation( + #[case] commits: Vec>, + #[case] expected_live: usize, + #[case] expected_removes: Vec<&'static str>, +) -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + let target_version = commits.len() as u64; + for (idx, body) in commits.into_iter().enumerate() { + add_commit( + table_root, + storage.as_ref(), + (idx + 1) as u64, + actions_to_string(body), + ) + .await?; + } + + let target = Snapshot::builder_for(table_url) + .at_version(target_version) + .build(engine.as_ref())?; + let listing = unwrap_listing(target.incremental_scan_builder(0).build(engine.as_ref())?); + + assert_eq!(live_add_count(&listing), expected_live); + let expected: HashSet = expected_removes.iter().map(|p| key(p)).collect(); + assert_eq!(listing.summary.removes, expected); + + Ok(()) +} + +// Catalog-managed: commits in the range live as staged JSONs under `_staged_commits/` +// (not in the published log) and are surfaced to the snapshot via `with_log_tail`. +// `IncrementalScanBuilder` must walk the snapshot's commit list -- not re-list storage -- +// or those commits are silently invisible and the diff is wrong. +#[tokio::test] +async fn picks_up_staged_commits_from_log_tail() -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + // v0: published, catalog-managed metadata. v1, v2: staged-only. + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string_catalog_managed(vec![TestAction::Metadata]), + ) + .await?; + let staged1 = add_staged_commit( + table_root, + storage.as_ref(), + 1, + actions_to_string(vec![TestAction::Add("staged-1.parquet".to_string())]), + ) + .await?; + let staged2 = add_staged_commit( + table_root, + storage.as_ref(), + 2, + actions_to_string(vec![TestAction::Add("staged-2.parquet".to_string())]), + ) + .await?; + + let log_tail = vec![ + create_log_path(&table_url, staged1), + create_log_path(&table_url, staged2), + ]; + let target = Snapshot::builder_for(table_url) + .with_log_tail(log_tail) + .with_max_catalog_version(2) + .build(engine.as_ref())?; + assert_eq!(target.version(), 2); + + let listing = unwrap_listing(target.incremental_scan_builder(0).build(engine.as_ref())?); + + // Both staged Adds must survive -- they're only reachable via log_tail. + assert_eq!( + live_add_count(&listing), + 2, + "expected 2 live Adds from staged commits in log_tail" + ); + assert_eq!(listing.summary.target_version, 2); + + Ok(()) +} + +// A commit in the range that contains no Adds (removes-only or metadata-only) should +// produce no entries in `add_files` and surface only the Remove paths (if any) in +// `removes`. +#[rstest] +#[case::removes_only( + vec![ + TestAction::Remove("gone-a.parquet".to_string()), + TestAction::Remove("gone-b.parquet".to_string()), + ], + vec!["gone-a.parquet", "gone-b.parquet"], +)] +#[case::metadata_only(vec![TestAction::Metadata], vec![])] +#[tokio::test] +async fn commit_with_no_live_adds( + #[case] v1_actions: Vec, + #[case] expected_remove_paths: Vec<&'static str>, +) -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 1, + actions_to_string(v1_actions), + ) + .await?; + + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(engine.as_ref())?; + let listing = unwrap_listing(target.incremental_scan_builder(0).build(engine.as_ref())?); + + assert_eq!(live_add_count(&listing), 0); + assert!(listing.add_files.is_empty(), "no Adds means no add batches"); + let expected_removes: HashSet = + expected_remove_paths.iter().map(|p| key(p)).collect(); + assert_eq!(listing.summary.removes, expected_removes); + + Ok(()) +} + +// Locks the iterator contract: `next()` returns `None` once every in-range commit has been +// processed, even when no commit produced a live Add. Polling past exhaustion stays +// `None` and never panics. +#[tokio::test] +async fn next_returns_none_after_exhaustion() -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 1, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(engine.as_ref())?; + let mut stream = target + .incremental_scan_builder(0) + .build(engine.as_ref())? + .expect("expected Some(stream)"); + + for item in stream.by_ref() { + item?; + } + assert!( + stream.next().is_none(), + "polling past exhaustion stays None" + ); + assert!(stream.next().is_none(), "still None on repeat poll"); + + Ok(()) +} + +// === DV-aware dedup === +// The kernel's dedup key is `(path, dv_unique_id)`, where `dv_unique_id` is built from +// `(storageType, pathOrInlineDv, offset)`. These tests construct raw commit JSON +// (TestAction doesn't emit DV-bearing Adds) to exercise the DV cases. + +const ACTION_METADATA: &str = "{\"metaData\":{\"id\":\"test-id\",\"format\":{\"provider\":\"parquet\",\"options\":{}},\"schemaString\":\"{\\\"type\\\":\\\"struct\\\",\\\"fields\\\":[]}\",\"partitionColumns\":[],\"configuration\":{},\"createdTime\":1700000000000}}\n{\"protocol\":{\"minReaderVersion\":3,\"minWriterVersion\":7,\"readerFeatures\":[\"deletionVectors\"],\"writerFeatures\":[\"deletionVectors\"]}}"; + +fn add_with_dv(path: &str, storage_type: &str, path_or_inline: &str, offset: i32) -> String { + format!( + "{{\"add\":{{\"path\":\"{path}\",\"partitionValues\":{{}},\"size\":100,\"modificationTime\":1700000000000,\"dataChange\":true,\"stats\":null,\"tags\":null,\"deletionVector\":{{\"storageType\":\"{storage_type}\",\"pathOrInlineDv\":\"{path_or_inline}\",\"offset\":{offset},\"sizeInBytes\":10,\"cardinality\":1}},\"baseRowId\":null,\"defaultRowCommitVersion\":null,\"clusteringProvider\":null}}}}" + ) +} + +fn add_no_dv(path: &str) -> String { + format!( + "{{\"add\":{{\"path\":\"{path}\",\"partitionValues\":{{}},\"size\":100,\"modificationTime\":1700000000000,\"dataChange\":true,\"stats\":null,\"tags\":null,\"deletionVector\":null,\"baseRowId\":null,\"defaultRowCommitVersion\":null,\"clusteringProvider\":null}}}}" + ) +} + +// Same path with DV-bearing Adds across commits. Dedup keys on `(path, dv_unique_id)`: +// different DV ids produce distinct keys (both survive), same DV ids collapse to one +// live Add (newest-wins). +#[rstest] +#[case::different_dvs( + ("u", "abc", 1), + ("u", "xyz", 2), + 2, + vec![("X.parquet", "uabc@1"), ("X.parquet", "uxyz@2")], +)] +#[case::same_dvs( + ("u", "abc", 1), + ("u", "abc", 1), + 1, + vec![("X.parquet", "uabc@1")], +)] +#[tokio::test] +async fn same_path_across_commits_dedups_by_dv( + #[case] v1_dv: (&'static str, &'static str, i32), + #[case] v2_dv: (&'static str, &'static str, i32), + #[case] expected_live: usize, + #[case] expected_keys: Vec<(&'static str, &'static str)>, +) -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit(table_root, storage.as_ref(), 0, ACTION_METADATA.to_string()).await?; + add_commit( + table_root, + storage.as_ref(), + 1, + add_with_dv("X.parquet", v1_dv.0, v1_dv.1, v1_dv.2), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 2, + add_with_dv("X.parquet", v2_dv.0, v2_dv.1, v2_dv.2), + ) + .await?; + + let target = Snapshot::builder_for(table_url) + .at_version(2) + .build(engine.as_ref())?; + let listing = unwrap_listing(target.incremental_scan_builder(0).build(engine.as_ref())?); + + assert_eq!(live_add_count(&listing), expected_live); + assert!(listing.summary.removes.is_empty()); + let expected_adds: HashSet = expected_keys + .iter() + .map(|(p, dv)| key_with_dv(p, dv)) + .collect(); + assert_eq!(listing.summary.live_adds, expected_adds); + + Ok(()) +} + +// DV-update pattern: v1 introduces the file with no DV; v2 emits Remove(X, no_dv) + +// Add(X, dv1). The Remove has key `(X, NULL)` and cancels the v1 Add. The new Add has +// key `(X, dv1)` -- distinct -- and stays live. The consumer sees one live Add +// with the new DV and one Remove for the old (no-DV) version. +#[tokio::test] +async fn dv_update_remove_no_dv_add_with_dv() -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit(table_root, storage.as_ref(), 0, ACTION_METADATA.to_string()).await?; + // v1: Add(X) without a DV. + add_commit(table_root, storage.as_ref(), 1, add_no_dv("X.parquet")).await?; + // v2: Remove(X, no_dv) + Add(X, dv=u/abc/1). + let remove_no_dv = "{\"remove\":{\"path\":\"X.parquet\",\"deletionTimestamp\":1700000000000,\"dataChange\":true,\"deletionVector\":null}}".to_string(); + let v2_actions = format!( + "{}\n{}", + remove_no_dv, + add_with_dv("X.parquet", "u", "abc", 1), + ); + add_commit(table_root, storage.as_ref(), 2, v2_actions).await?; + + let target = Snapshot::builder_for(table_url) + .at_version(2) + .build(engine.as_ref())?; + + let listing = unwrap_listing(target.incremental_scan_builder(0).build(engine.as_ref())?); + + // The new DV-bearing Add (key `(X, dv1)`) survives. The v1 no-DV Add (key `(X, NULL)`) + // is cancelled by the v2 no-DV Remove (same key). + assert_eq!( + live_add_count(&listing), + 1, + "the new DV-bearing Add survives; the old no-DV Add is cancelled" + ); + assert!( + listing.summary.removes.contains(&key("X.parquet")), + "the no-DV Remove survives in `removes` (key `(X, NULL)`)" + ); + + Ok(()) +} + +// A commit file the snapshot listed is removed before the stream reads it (e.g. a +// vacuum races between snapshot construction and the incremental scan). The build() +// call only does metadata-only checks and succeeds, but the stream surfaces +// `FileNotFound` while reading the missing commit. Consumers fall back to a full scan +// on any stream error. +#[tokio::test] +async fn missing_commit_file_surfaces_error_during_iteration( +) -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 1, + actions_to_string(vec![TestAction::Add("a.parquet".to_string())]), + ) + .await?; + + let target = Snapshot::builder_for(table_url.clone()) + .at_version(1) + .build(engine.as_ref())?; + + let v1_path: ObjectStorePath = delta_path_for_version(1, "json"); + storage.delete(&v1_path).await?; + + let mut stream = target + .incremental_scan_builder(0) + .build(engine.as_ref())? + .expect("expected Some(stream)"); + + let item = stream + .next() + .expect("stream should yield an error item before exhausting"); + let err = match item { + Ok(_) => panic!("expected FileNotFound from the stream, got an Ok batch"), + Err(e) => e, + }; + assert!( + matches!(err, delta_kernel::Error::FileNotFound(_)), + "expected FileNotFound, got {err:?}" + ); + + let finish_err = stream + .into_summary() + .expect_err("into_summary should error on a previously-errored stream"); + assert!( + finish_err.to_string().contains("previously errored"), + "unexpected into_summary error: {finish_err}" + ); + + Ok(()) +} + +// Caller errors: a base_version that is not strictly less than the target snapshot's version +// must surface as `Err`, not as a `CommitsUnavailable` listing. The latter is reserved for +// log-retention / vacuum-race scenarios; equal or inverted ranges are programmer mistakes. +// Target snapshot is at v1; the parameter is the (invalid) base_version. +#[rstest] +#[case::base_equals_target(1)] +#[case::base_above_target(2)] +#[case::base_at_max(u64::MAX)] +#[tokio::test] +async fn rejects_invalid_base_version( + #[case] bad_base: u64, +) -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 1, + actions_to_string(vec![TestAction::Add("a.parquet".to_string())]), + ) + .await?; + + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(engine.as_ref())?; + + let err = target + .incremental_scan_builder(bad_base) + .build(engine.as_ref()) + .expect_err("expected Err for invalid base_version"); + assert!( + err.to_string().contains("must be less than"), + "unexpected error: {err}" + ); + + Ok(()) +} + +// A log compaction file alongside the JSON commits in the range must not affect the +// result. Kernel's incremental path reads commit JSONs directly (the snapshot's log +// segment exposes commits and compactions on separate fields, and this builder iterates +// only `ascending_commit_files`). The compaction is ignored. We assert the listing matches +// what we'd get without the compaction file present at all. +#[tokio::test] +async fn compaction_file_in_range_is_ignored() -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + // v0 Metadata; v1 add A,B; v2 remove A; v3 add C. + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 1, + actions_to_string(vec![ + TestAction::Add("A".to_string()), + TestAction::Add("B".to_string()), + ]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 2, + actions_to_string(vec![TestAction::Remove("A".to_string())]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 3, + actions_to_string(vec![TestAction::Add("C".to_string())]), + ) + .await?; + + // Drop a compaction file covering 1-3 next to the commit JSONs. Body is irrelevant + // because the incremental scan never opens it. + let compaction_path: ObjectStorePath = compacted_log_path_for_versions(1, 3, "json"); + storage + .put(&compaction_path, bytes::Bytes::new().into()) + .await?; + + let target = Snapshot::builder_for(table_url) + .at_version(3) + .build(engine.as_ref())?; + + let listing = unwrap_listing(target.incremental_scan_builder(0).build(engine.as_ref())?); + + assert_eq!( + live_add_count(&listing), + 2, + "B (live) and C (live) survive; A is cancelled by the v2 Remove" + ); + assert_eq!( + listing.summary.removes, + HashSet::from([key("A")]), + "Remove(A) survives" + ); + + Ok(()) +} + +// `into_summary` returns the live Add and Remove file-key sets directly (without going +// through the iterator first). Connectors that don't need per-batch streaming can drain +// via `into_summary` and apply their own logic over the keys. +#[tokio::test] +async fn into_summary_returns_live_keys() -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 1, + actions_to_string(vec![ + TestAction::Add("A".to_string()), + TestAction::Add("B".to_string()), + ]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 2, + actions_to_string(vec![ + TestAction::Remove("A".to_string()), + TestAction::Add("C".to_string()), + ]), + ) + .await?; + + let target = Snapshot::builder_for(table_url) + .at_version(2) + .build(engine.as_ref())?; + + let stream = target + .incremental_scan_builder(0) + .build(engine.as_ref())? + .expect("expected Some(stream)"); + + let footer: IncrementalScanSummary = stream.into_summary()?; + assert_eq!(footer.base_version, 0); + assert_eq!(footer.target_version, 2); + assert_eq!(footer.live_adds, HashSet::from([key("B"), key("C")]),); + assert_eq!(footer.removes, HashSet::from([key("A")])); + + Ok(()) +} + +// The iterator yields one batch per source commit that produced live Adds, in +// descending commit-version order. Commits whose Adds were all cancelled by later +// Removes do not produce an item, but they still update dedup state for older commits. +#[tokio::test] +async fn streaming_yields_batches_newest_first_skipping_cancelled_commits( +) -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + // v0 Metadata; v1 add(A); v2 add(B); v3 remove(A) remove(B); v4 add(C). + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 1, + actions_to_string(vec![TestAction::Add("A".to_string())]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 2, + actions_to_string(vec![TestAction::Add("B".to_string())]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 3, + actions_to_string(vec![ + TestAction::Remove("A".to_string()), + TestAction::Remove("B".to_string()), + ]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 4, + actions_to_string(vec![TestAction::Add("C".to_string())]), + ) + .await?; + + let target = Snapshot::builder_for(table_url) + .at_version(4) + .build(engine.as_ref())?; + + let mut stream = target + .incremental_scan_builder(0) + .build(engine.as_ref())? + .expect("expected Some(stream)"); + + let mut batches: Vec = Vec::new(); + for item in stream.by_ref() { + batches.push(item?); + } + + // v4 produced the live add(C); v3 had only Removes (no add item); + // v2 and v1 had their adds cancelled by v3 (no add items). So exactly one batch. + assert_eq!( + batches.len(), + 1, + "expected one yielded batch (v4 only); v3 has no Adds, v1/v2 cancelled" + ); + let live: usize = batches[0].selection_vector().iter().filter(|s| **s).count(); + assert_eq!(live, 1, "the single yielded batch contains add(C)"); + + let footer = stream.into_summary()?; + assert_eq!(footer.removes, HashSet::from([key("A"), key("B")]),); + + Ok(()) +} + +// A single batch where some Adds survive dedup and some are cancelled. Exercises the +// non-trivial selection-vector path in `process_batch` (mixed booleans, not all-true +// or all-false). +#[tokio::test] +async fn mixed_intra_batch_selection_vector() -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + // v0 Metadata; v1 add(A); v2 add(B), add(C), add(D); v3 remove(B), remove(D). + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 1, + actions_to_string(vec![TestAction::Add("A".to_string())]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 2, + actions_to_string(vec![ + TestAction::Add("B".to_string()), + TestAction::Add("C".to_string()), + TestAction::Add("D".to_string()), + ]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 3, + actions_to_string(vec![ + TestAction::Remove("B".to_string()), + TestAction::Remove("D".to_string()), + ]), + ) + .await?; + + let target = Snapshot::builder_for(table_url) + .at_version(3) + .build(engine.as_ref())?; + + let listing = unwrap_listing(target.incremental_scan_builder(0).build(engine.as_ref())?); + + // Two yielded batches: v2 (with mixed selection [false,true,false]) and v1 (with [true]). + assert_eq!(listing.add_files.len(), 2); + assert_eq!( + live_add_count(&listing), + 2, + "C and A survive; B and D cancelled" + ); + assert_eq!(listing.summary.removes, HashSet::from([key("B"), key("D")]),); + + // Find the batch whose selection has both true and false entries (the v2 batch). + let mixed_batch = listing + .add_files + .iter() + .find(|b| { + let sv = b.selection_vector(); + sv.iter().any(|s| *s) && sv.iter().any(|s| !*s) + }) + .expect("expected at least one batch with mixed selection"); + assert_eq!(mixed_batch.selection_vector().len(), 3); + + Ok(()) +} + +// === Batch-splitting invariance === +// +// Engine wrapper that overrides only the JSON handler so we can force `read_json_files` +// to chunk a single commit file across many `ActionsBatch` yields. Used by the +// `single_commit_split_across_batches_dedups_correctly` test to verify the kernel does +// not carry per-batch dedup state. A naive consumer that assumes "one yield = one commit" +// would mis-cancel a same-commit `add(P, dv=new) + remove(P, dv=old)` pair when the two +// rows land in different batches. Our dedup is keyed on `(path, dv_unique_id)` against a +// global `seen_file_keys`, so output must be identical regardless of how the JSON reader +// chunks rows. +struct CustomBatchSizeEngine { + inner: Arc>, + json: Arc>, +} + +impl CustomBatchSizeEngine { + fn new(storage: Arc, batch_size: usize) -> Self { + let task_executor = Arc::new(TokioBackgroundExecutor::new()); + let json = Arc::new( + DefaultJsonHandler::new(storage.clone(), task_executor.clone()) + .with_batch_size(NonZero::new(batch_size).expect("batch_size is non-zero")), + ); + let inner = Arc::new( + DefaultEngineBuilder::new(storage) + .with_task_executor(task_executor) + .build(), + ); + Self { inner, json } + } +} + +impl Engine for CustomBatchSizeEngine { + fn evaluation_handler(&self) -> Arc { + self.inner.evaluation_handler() + } + fn storage_handler(&self) -> Arc { + self.inner.storage_handler() + } + fn json_handler(&self) -> Arc { + self.json.clone() + } + fn parquet_handler(&self) -> Arc { + self.inner.parquet_handler() + } +} + +// Regression test for batch-splitting invariance. A single commit contains a mix that +// stresses path-only state machines: plain Adds, two DV-replacement pairs (same path, Add +// with new DV plus Remove with old/null DV), and a standalone Remove. Forcing the JSON +// reader to split this commit into many `ActionsBatch` yields (batch_size = 1, 2, 3) +// must produce the exact same live Add and Remove path sets as the unsplit case +// (batch_size large enough to hold every row). If our dedup state ever becomes +// per-batch instead of global, this test fails. +#[rstest] +#[case::row_per_batch(1)] +#[case::two_per_batch(2)] +#[case::three_per_batch(3)] +#[case::all_in_one_batch(1000)] +#[tokio::test] +async fn single_commit_split_across_batches_dedups_correctly( + #[case] batch_size: usize, +) -> Result<(), Box> { + let (storage, _, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit(table_root, storage.as_ref(), 0, ACTION_METADATA.to_string()).await?; + + // v1 has 10 rows. The DV-replacement pairs are written `Remove(P, dv=old)` BEFORE + // `Add(P, dv=new)`. This is the row order that breaks a path-only state machine: it + // sees the Remove first, records P as removed, then cancels the matching Add when it + // arrives in a later batch, losing both rows. Kernel's `(path, dv_unique_id)` keying + // keeps `(P, null)` and `(P, dv=new)` distinct, so Add and Remove are independent + // regardless of batch boundary. + let remove_no_dv = |path: &str| -> String { + format!( + "{{\"remove\":{{\"path\":\"{path}\",\"deletionTimestamp\":1700000000000,\ + \"dataChange\":true,\"deletionVector\":null}}}}" + ) + }; + let v1_actions = [ + add_no_dv("a.parquet"), + add_no_dv("b.parquet"), + // DV-replacement pair #1: Remove BEFORE matching Add. This is the bug-triggering + // order for path-only consumers. + remove_no_dv("d.parquet"), + add_with_dv("d.parquet", "u", "abc", 1), + add_no_dv("c.parquet"), + // DV-replacement pair #2: Remove BEFORE matching Add. + remove_no_dv("f.parquet"), + add_with_dv("f.parquet", "u", "xyz", 2), + add_no_dv("e.parquet"), + // Standalone Remove for a path not added in this range. + remove_no_dv("stale.parquet"), + add_no_dv("g.parquet"), + ] + .join("\n"); + add_commit(table_root, storage.as_ref(), 1, v1_actions).await?; + + let engine = CustomBatchSizeEngine::new(storage, batch_size); + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(&engine)?; + let listing = unwrap_listing(target.incremental_scan_builder(0).build(&engine)?); + + // 7 live Adds: a, b, c, e, g (plain) plus d and f (the new DV-bearing copies). + // The no-DV Removes for d and f have key `(path, NULL)`, distinct from the new + // DV-bearing Adds, so they survive too. + let expected_adds: HashSet = [ + key("a.parquet"), + key("b.parquet"), + key("c.parquet"), + key_with_dv("d.parquet", "uabc@1"), + key("e.parquet"), + key_with_dv("f.parquet", "uxyz@2"), + key("g.parquet"), + ] + .into_iter() + .collect(); + let expected_removes: HashSet = + [key("d.parquet"), key("f.parquet"), key("stale.parquet")] + .into_iter() + .collect(); + + assert_eq!( + listing.summary.live_adds, expected_adds, + "live_adds must not depend on batch_size (got batch_size={batch_size})" + ); + assert_eq!( + listing.summary.removes, expected_removes, + "removes must not depend on batch_size (got batch_size={batch_size})" + ); + assert_eq!( + live_add_count(&listing), + 7, + "selection-vector counts must not depend on batch_size (got batch_size={batch_size})" + ); + + Ok(()) +} + +// === Classification === +// +// `into_summary_against_base_iter(base_keys)` / `into_listing_against_base_iter(base_keys)` +// intersect the consumer's base file keys (`(path, dv_unique_id)`) against the live +// Adds to surface metadata-only re-adds in `duplicate_adds`. The Add row itself stays in +// the streamed Adds; the key is also surfaced separately so the consumer can mask the +// stale base entry. + +fn unwrap_classified_listing<'a>( + result: Option, + base_keys: impl IntoIterator, +) -> IncrementalListingAgainstBase { + result + .expect("expected Some(stream), got None (commits unavailable)") + .into_listing_against_base_iter(base_keys) + .expect("into_listing_against_base_iter succeeded") +} + +fn classified_add_count(listing: &IncrementalListingAgainstBase) -> usize { + listing + .add_files + .iter() + .map(|f| f.selection_vector().iter().filter(|s| **s).count()) + .sum() +} + +#[rstest] +#[case::all_re_adds( + vec!["re-added.parquet"], + vec!["re-added.parquet"], + 1, + vec!["re-added.parquet"], +)] +#[case::mixed_new_and_re_added( + vec!["brand-new.parquet", "re-added.parquet"], + vec!["re-added.parquet"], + 2, + vec!["re-added.parquet"], +)] +#[tokio::test] +async fn classifies_metadata_only_re_adds( + #[case] range_adds: Vec<&'static str>, + #[case] base_paths: Vec<&'static str>, + #[case] expected_live: usize, + #[case] expected_duplicates: Vec<&'static str>, +) -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + let actions: Vec = range_adds + .iter() + .map(|p| TestAction::Add(p.to_string())) + .collect(); + add_commit(table_root, storage.as_ref(), 1, actions_to_string(actions)).await?; + + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(engine.as_ref())?; + let base_keys: Vec = base_paths.iter().map(|p| key(p)).collect(); + let listing = unwrap_classified_listing( + target.incremental_scan_builder(0).build(engine.as_ref())?, + &base_keys, + ); + + assert_eq!(classified_add_count(&listing), expected_live); + let expected: HashSet = expected_duplicates.iter().map(|p| key(p)).collect(); + assert_eq!(listing.summary.duplicate_adds, expected); + assert!(listing.summary.removes.is_empty()); + + Ok(()) +} + +// The pull-then-finalize flow: drive the iterator manually with `next()`, then call +// `into_summary_against_base_iter(base_keys)`. This is the documented streaming consumer +// pattern. +#[tokio::test] +async fn into_summary_after_manual_streaming_classifies_duplicates( +) -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 1, + actions_to_string(vec![ + TestAction::Add("brand-new.parquet".to_string()), + TestAction::Add("re-added.parquet".to_string()), + ]), + ) + .await?; + + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(engine.as_ref())?; + + let mut stream = target + .incremental_scan_builder(0) + .build(engine.as_ref())? + .expect("expected Some(stream)"); + + let mut yielded = 0; + for item in stream.by_ref() { + let _batch = item?; + yielded += 1; + } + assert_eq!(yielded, 1, "v1 produced one Add batch"); + + let base_keys = [key("re-added.parquet"), key("not-in-range.parquet")]; + let summary = stream.into_summary_against_base_iter(&base_keys)?; + assert_eq!( + summary.duplicate_adds, + HashSet::from([key("re-added.parquet")]), + "non-matching base key is filtered out" + ); + assert!(summary.removes.is_empty()); + + Ok(()) +} + +// Predicate variants (`*_with`) take a closure and produce the same `duplicate_adds` +// as the iterator variants. Asserts that for a fixed (range, base) pair both forms +// agree on the classified output. Demonstrates HashMap-backed base via closure. +#[tokio::test] +async fn predicate_variants_match_iterator_variants() -> Result<(), Box> { + use std::collections::HashMap; + + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 1, + actions_to_string(vec![ + TestAction::Add("brand-new.parquet".to_string()), + TestAction::Add("re-added.parquet".to_string()), + ]), + ) + .await?; + + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(engine.as_ref())?; + + // Base lives in a HashMap (key -> metadata). Closure does the contains-check. + let base_index: HashMap = [ + (key("re-added.parquet"), 100), + (key("not-in-range.parquet"), 200), + ] + .into(); + + let stream = target + .clone() + .incremental_scan_builder(0) + .build(engine.as_ref())? + .expect("expected Some(stream)"); + let summary_with = stream.into_summary_against_base_closure(|k| base_index.contains_key(k))?; + assert_eq!( + summary_with.duplicate_adds, + HashSet::from([key("re-added.parquet")]), + ); + assert!(summary_with.removes.is_empty()); + + let stream = target + .incremental_scan_builder(0) + .build(engine.as_ref())? + .expect("expected Some(stream)"); + let listing_with = stream.into_listing_against_base_closure(|k| base_index.contains_key(k))?; + assert_eq!( + listing_with.summary.duplicate_adds, + summary_with.duplicate_adds + ); + assert_eq!(listing_with.summary.removes, summary_with.removes); + assert_eq!(classified_add_count(&listing_with), 2); + + Ok(()) +} + +// Empty base: `duplicate_adds` is empty (nothing in the range overlaps the empty base), +// but `removes` still surfaces every range remove. Exercises both against-base forms. +#[tokio::test] +async fn empty_base_keys_produces_no_duplicates() -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 1, + actions_to_string(vec![ + TestAction::Add("a.parquet".to_string()), + TestAction::Remove("gone.parquet".to_string()), + ]), + ) + .await?; + + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(engine.as_ref())?; + + // By-ref iterator with empty input. + let stream = target + .clone() + .incremental_scan_builder(0) + .build(engine.as_ref())? + .expect("expected Some(stream)"); + let empty: [FileActionKey; 0] = []; + let summary_iter = stream.into_summary_against_base_iter(&empty)?; + assert!(summary_iter.duplicate_adds.is_empty()); + assert_eq!(summary_iter.removes, HashSet::from([key("gone.parquet")])); + + // Predicate closure that always returns false. + let stream = target + .incremental_scan_builder(0) + .build(engine.as_ref())? + .expect("expected Some(stream)"); + let summary_with = stream.into_summary_against_base_closure(|_| false)?; + assert!(summary_with.duplicate_adds.is_empty()); + assert_eq!(summary_with.removes, HashSet::from([key("gone.parquet")])); + + Ok(()) +} + +// === Predicate pushdown === +// +// `with_predicate` prunes streamed live Adds against file stats using the same +// `DataSkippingFilter` the read path uses. These tests use raw commit JSON so each Add can +// carry a distinct `id` min/max range. The metadata (`test_utils::METADATA`) declares an +// `id: integer` column, so `id`-predicates are eligible for data skipping. Skipping is +// conservative: an Add is dropped only when its stats prove no row matches; Adds with no +// stats and all Removes are always kept. + +// Build a raw Add action carrying `id` stats over the closed range `[id_min, id_max]`. +fn add_with_id_stats(path: &str, id_min: i32, id_max: i32) -> String { + let stats = format!( + "{{\\\"numRecords\\\":10,\\\"nullCount\\\":{{\\\"id\\\":0}},\ + \\\"minValues\\\":{{\\\"id\\\":{id_min}}},\\\"maxValues\\\":{{\\\"id\\\":{id_max}}}}}" + ); + format!( + "{{\"add\":{{\"path\":\"{path}\",\"partitionValues\":{{}},\"size\":100,\ + \"modificationTime\":1700000000000,\"dataChange\":true,\"stats\":\"{stats}\"}}}}" + ) +} + +// Build a raw Add action with no stats (`stats` omitted). Data skipping must always keep it. +fn add_without_stats(path: &str) -> String { + format!( + "{{\"add\":{{\"path\":\"{path}\",\"partitionValues\":{{}},\"size\":100,\ + \"modificationTime\":1700000000000,\"dataChange\":true}}}}" + ) +} + +// Build a raw Remove action. +fn remove_action(path: &str) -> String { + format!( + "{{\"remove\":{{\"path\":\"{path}\",\"deletionTimestamp\":1700000000000,\ + \"dataChange\":true}}}}" + ) +} + +// Build a raw Add action with a partition value for the `val` partition column and no stats. +fn add_partitioned(path: &str, val: &str) -> String { + format!( + "{{\"add\":{{\"path\":\"{path}\",\"partitionValues\":{{\"val\":\"{val}\"}},\"size\":100,\ + \"modificationTime\":1700000000000,\"dataChange\":true}}}}" + ) +} + +// Build a raw Add carrying both a `val` partition value and `id` stats over `[id_min, id_max]`, +// for predicates that touch both a partition and a data column. +fn add_partitioned_with_id_stats(path: &str, val: &str, id_min: i32, id_max: i32) -> String { + let stats = format!( + "{{\\\"numRecords\\\":10,\\\"nullCount\\\":{{\\\"id\\\":0}},\ + \\\"minValues\\\":{{\\\"id\\\":{id_min}}},\\\"maxValues\\\":{{\\\"id\\\":{id_max}}}}}" + ); + format!( + "{{\"add\":{{\"path\":\"{path}\",\"partitionValues\":{{\"val\":\"{val}\"}},\"size\":100,\ + \"modificationTime\":1700000000000,\"dataChange\":true,\"stats\":\"{stats}\"}}}}" + ) +} + +// Column-mapping (name mode) metadata whose `id` column maps to the physical name `col-id`. +const COLUMN_MAPPING_METADATA: &str = "{\"metaData\":{\"id\":\"cm-test-id\",\"format\":{\"provider\":\"parquet\",\"options\":{}},\"schemaString\":\"{\\\"type\\\":\\\"struct\\\",\\\"fields\\\":[{\\\"name\\\":\\\"id\\\",\\\"type\\\":\\\"integer\\\",\\\"nullable\\\":true,\\\"metadata\\\":{\\\"delta.columnMapping.id\\\":1,\\\"delta.columnMapping.physicalName\\\":\\\"col-id\\\"}}]}\",\"partitionColumns\":[],\"configuration\":{\"delta.columnMapping.mode\":\"name\",\"delta.columnMapping.maxColumnId\":\"1\"},\"createdTime\":1700000000000}}\n{\"protocol\":{\"minReaderVersion\":3,\"minWriterVersion\":7,\"readerFeatures\":[\"columnMapping\"],\"writerFeatures\":[\"columnMapping\"]}}"; + +// Build a raw Add whose stats are keyed by the physical column name `col-id` (column mapping). +fn add_with_physical_id_stats(path: &str, id_min: i32, id_max: i32) -> String { + let stats = format!( + "{{\\\"numRecords\\\":10,\\\"nullCount\\\":{{\\\"col-id\\\":0}},\ + \\\"minValues\\\":{{\\\"col-id\\\":{id_min}}},\\\"maxValues\\\":{{\\\"col-id\\\":{id_max}}}}}" + ); + format!( + "{{\"add\":{{\"path\":\"{path}\",\"partitionValues\":{{}},\"size\":100,\ + \"modificationTime\":1700000000000,\"dataChange\":true,\"stats\":\"{stats}\"}}}}" + ) +} + +// Metadata declaring an `id: integer` column AND the deletionVectors feature, so a DV-bearing +// Add carrying `id` stats parses and an `id`-predicate is skipping-eligible. +const DV_AND_ID_METADATA: &str = "{\"metaData\":{\"id\":\"dv-id-test\",\"format\":{\"provider\":\"parquet\",\"options\":{}},\"schemaString\":\"{\\\"type\\\":\\\"struct\\\",\\\"fields\\\":[{\\\"name\\\":\\\"id\\\",\\\"type\\\":\\\"integer\\\",\\\"nullable\\\":true,\\\"metadata\\\":{}}]}\",\"partitionColumns\":[],\"configuration\":{},\"createdTime\":1700000000000}}\n{\"protocol\":{\"minReaderVersion\":3,\"minWriterVersion\":7,\"readerFeatures\":[\"deletionVectors\"],\"writerFeatures\":[\"deletionVectors\"]}}"; + +// Build a raw DV-bearing Add carrying `id` stats over `[id_min, id_max]`. +fn add_with_dv_and_id_stats( + path: &str, + storage_type: &str, + path_or_inline: &str, + offset: i32, + id_min: i32, + id_max: i32, +) -> String { + let stats = format!( + "{{\\\"numRecords\\\":10,\\\"nullCount\\\":{{\\\"id\\\":0}},\ + \\\"minValues\\\":{{\\\"id\\\":{id_min}}},\\\"maxValues\\\":{{\\\"id\\\":{id_max}}}}}" + ); + format!( + "{{\"add\":{{\"path\":\"{path}\",\"partitionValues\":{{}},\"size\":100,\ + \"modificationTime\":1700000000000,\"dataChange\":true,\"stats\":\"{stats}\",\ + \"deletionVector\":{{\"storageType\":\"{storage_type}\",\"pathOrInlineDv\":\"{path_or_inline}\",\ + \"offset\":{offset},\"sizeInBytes\":10,\"cardinality\":1}}}}}}" + ) +} + +// Pushdown drops added files whose stats prove they cannot match and keeps the rest; the +// `None` predicate keeps every live Add unchanged. Three files span disjoint `id` ranges over +// a single commit. Each case asserts the surviving key set and that the emitted selection +// vector agrees with `live_adds` (no summary/stream drift). +#[rstest] +#[case::pushdown( + Some(Arc::new(col!("id").gt(lit(25i32))) as PredicateRef), + vec!["high.parquet"], +)] +#[case::no_predicate(None, vec!["low.parquet", "mid.parquet", "high.parquet"])] +#[tokio::test] +async fn pushdown_keeps_only_matching_added_files( + #[case] predicate: Option, + #[case] expected_live: Vec<&str>, +) -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + // low: id in [0, 9]; mid: id in [10, 19]; high: id in [20, 30]. `id > 25` can only match + // high; low and mid are provably excluded. No predicate keeps all three. + let v1 = [ + add_with_id_stats("low.parquet", 0, 9), + add_with_id_stats("mid.parquet", 10, 19), + add_with_id_stats("high.parquet", 20, 30), + ] + .join("\n"); + add_commit(table_root, storage.as_ref(), 1, v1).await?; + + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(engine.as_ref())?; + let listing = unwrap_listing( + target + .incremental_scan_builder(0) + .with_predicate(predicate) + .build(engine.as_ref())?, + ); + + let expected: HashSet = expected_live.iter().map(|p| key(p)).collect(); + assert_eq!(listing.summary.live_adds, expected); + // The streamed selection vector must agree with the summary's live_adds count. + assert_eq!(live_add_count(&listing), expected.len()); + assert!(listing.summary.removes.is_empty()); + + Ok(()) +} + +// A file with no stats is always kept, even under a predicate whose range would exclude it if +// stats were present. Conservative skipping never drops a stats-less file. +#[tokio::test] +async fn added_file_without_stats_is_always_kept() -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + // no_stats has no stats; in_range has id in [20, 30]. Both must survive `id > 25`: + // no_stats because it can't be disproven, in_range because [20, 30] overlaps. + let v1 = [ + add_without_stats("no_stats.parquet"), + add_with_id_stats("in_range.parquet", 20, 30), + add_with_id_stats("out_of_range.parquet", 0, 5), + ] + .join("\n"); + add_commit(table_root, storage.as_ref(), 1, v1).await?; + + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(engine.as_ref())?; + + let predicate: PredicateRef = Arc::new(col!("id").gt(lit(25i32))); + let listing = unwrap_listing( + target + .incremental_scan_builder(0) + .with_predicate(predicate) + .build(engine.as_ref())?, + ); + + assert_eq!( + listing.summary.live_adds, + HashSet::from([key("no_stats.parquet"), key("in_range.parquet")]), + "the stats-less file is kept; only the provably-excluded file is dropped" + ); + + Ok(()) +} + +// Removes are never filtered by the predicate. A removed file whose stat range the predicate +// excludes must still appear in `removes` in full -- the consumer needs every removal to evict +// stale cached entries. +#[tokio::test] +async fn removes_are_never_filtered_by_predicate() -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + // v1 adds gone.parquet (id in [0, 5], which `id > 25` would exclude). + add_commit( + table_root, + storage.as_ref(), + 1, + add_with_id_stats("gone.parquet", 0, 5), + ) + .await?; + // v2 removes gone.parquet and adds kept.parquet (id in [20, 30]). + let v2 = [ + remove_action("gone.parquet"), + add_with_id_stats("kept.parquet", 20, 30), + ] + .join("\n"); + add_commit(table_root, storage.as_ref(), 2, v2).await?; + + let target = Snapshot::builder_for(table_url) + .at_version(2) + .build(engine.as_ref())?; + + let predicate: PredicateRef = Arc::new(col!("id").gt(lit(25i32))); + let listing = unwrap_listing( + target + .incremental_scan_builder(0) + .with_predicate(predicate) + .build(engine.as_ref())?, + ); + + // The Remove is reported in full even though the predicate excludes its stat range. + assert_eq!( + listing.summary.removes, + HashSet::from([key("gone.parquet")]), + "removes must never be filtered by the predicate" + ); + // Only kept.parquet ([20, 30]) survives among the live Adds. + assert_eq!( + listing.summary.live_adds, + HashSet::from([key("kept.parquet")]) + ); + + Ok(()) +} + +// A predicate that statically excludes every file (`id > 25 AND FALSE`) yields an empty live-Add +// stream, but Removes are still reported. A tautology (`id > 25 OR TRUE`) is useless for +// skipping and behaves like no predicate. +#[rstest] +#[case::static_skip_all(Pred::and(col!("id").gt(lit(25i32)), Pred::FALSE), 0)] +#[case::tautology(Pred::or(col!("id").gt(lit(25i32)), Pred::TRUE), 2)] +#[tokio::test] +async fn static_skip_all_and_tautology( + #[case] predicate: Pred, + #[case] expected_live: usize, +) -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + // v1 adds two files and removes a third, so removes is non-empty regardless of the predicate. + let v1 = [ + add_with_id_stats("a.parquet", 0, 9), + add_with_id_stats("b.parquet", 20, 30), + remove_action("removed.parquet"), + ] + .join("\n"); + add_commit(table_root, storage.as_ref(), 1, v1).await?; + + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(engine.as_ref())?; + let listing = unwrap_listing( + target + .incremental_scan_builder(0) + .with_predicate(Arc::new(predicate)) + .build(engine.as_ref())?, + ); + + assert_eq!(live_add_count(&listing), expected_live); + // Removes are reported regardless of the Add-side skipping strategy. + assert_eq!( + listing.summary.removes, + HashSet::from([key("removed.parquet")]), + "removes are reported even when all Adds are statically skipped" + ); + + Ok(()) +} + +// Consumer classification (`into_summary_against_base_*`) composes with the predicate: base +// intersection runs over the predicate-pruned live Adds, so a base key whose file the +// predicate dropped is not reported as a duplicate. +#[tokio::test] +async fn against_base_classification_composes_with_predicate( +) -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + // matches.parquet (id in [20, 30]) survives `id > 25`; dropped.parquet (id in [0, 5]) does not. + let v1 = [ + add_with_id_stats("matches.parquet", 20, 30), + add_with_id_stats("dropped.parquet", 0, 5), + ] + .join("\n"); + add_commit(table_root, storage.as_ref(), 1, v1).await?; + + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(engine.as_ref())?; + + // Both files are in the consumer's base. Only the predicate-surviving one should classify + // as a duplicate; the dropped file was pruned before base intersection. + let base_keys = [key("matches.parquet"), key("dropped.parquet")]; + let predicate: PredicateRef = Arc::new(col!("id").gt(lit(25i32))); + let stream = target + .incremental_scan_builder(0) + .with_predicate(predicate) + .build(engine.as_ref())? + .expect("expected Some(stream)"); + let summary = stream.into_summary_against_base_iter(&base_keys)?; + + assert_eq!( + summary.duplicate_adds, + HashSet::from([key("matches.parquet")]), + "the pruned file is not classified as a duplicate" + ); + assert!(summary.removes.is_empty()); + + Ok(()) +} + +// Newest-wins holds when the newest Add of a key is pruned but an older Add of the same key +// (with a matching range) exists earlier in the range. The pruned newest Add still records the +// key as `seen`, so the older duplicate must not leak into `live_adds`. The file's live (newest) +// metadata does not match the predicate, so a cold scan at the target would exclude it too -- +// net zero live Adds. Guards the "record seen regardless of the mask" invariant. +#[tokio::test] +async fn pruned_newest_add_suppresses_older_matching_duplicate( +) -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + // v1: dupe.parquet with a range that WOULD match `id > 25`. + add_commit( + table_root, + storage.as_ref(), + 1, + add_with_id_stats("dupe.parquet", 20, 30), + ) + .await?; + // v2: dupe.parquet re-added with a range the predicate provably excludes ([0, 5]). + add_commit( + table_root, + storage.as_ref(), + 2, + add_with_id_stats("dupe.parquet", 0, 5), + ) + .await?; + + let target = Snapshot::builder_for(table_url) + .at_version(2) + .build(engine.as_ref())?; + + let predicate: PredicateRef = Arc::new(col!("id").gt(lit(25i32))); + let listing = unwrap_listing( + target + .incremental_scan_builder(0) + .with_predicate(predicate) + .build(engine.as_ref())?, + ); + + assert!( + listing.summary.live_adds.is_empty(), + "the pruned newest Add records `seen`, so the older matching duplicate must not leak" + ); + assert_eq!(live_add_count(&listing), 0); + + Ok(()) +} + +// A predicate referencing a column absent from the table schema fails at `build` (fail-fast), +// not lazily mid-stream, matching the documented `# Errors` contract. +#[tokio::test] +async fn unknown_column_predicate_fails_at_build() -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 1, + add_with_id_stats("a.parquet", 0, 9), + ) + .await?; + + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(engine.as_ref())?; + + let predicate: PredicateRef = Arc::new(col!("nonexistent").gt(lit(1i32))); + let result = target + .incremental_scan_builder(0) + .with_predicate(predicate) + .build(engine.as_ref()); + + // Fail-fast at build, surfacing the unresolved column. The concrete error is + // `Error::MissingColumn`, but a captured backtrace wraps it in `Error::Backtraced`, so match + // on the message rather than the variant. + assert_result_error_with_message(result, "unknown column: nonexistent"); + + Ok(()) +} + +// A predicate over a partition column prunes added files by their partition value: values are +// parsed from the raw `add.partitionValues` map via `map_to_struct`, so `val = 'x'` drops the +// file in partition `y`. +#[tokio::test] +async fn partition_column_predicate_prunes_added_files() -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + // METADATA_WITH_PARTITION_COLS partitions by `val`. + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string_partitioned(vec![TestAction::Metadata]), + ) + .await?; + let v1 = [ + add_partitioned("a.parquet", "x"), + add_partitioned("b.parquet", "y"), + ] + .join("\n"); + add_commit(table_root, storage.as_ref(), 1, v1).await?; + + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(engine.as_ref())?; + + let predicate: PredicateRef = Arc::new(col!("val").eq(lit("x"))); + let listing = unwrap_listing( + target + .incremental_scan_builder(0) + .with_predicate(predicate) + .build(engine.as_ref())?, + ); + + assert_eq!( + listing.summary.live_adds, + HashSet::from([key("a.parquet")]), + "only the file in partition `val = 'x'` survives" + ); + + Ok(()) +} + +// A mixed predicate `id > 25 AND val = 'x'` prunes on both axes: the data conjunct skips on +// `add.stats`, the partition conjunct skips on `add.partitionValues`. Only the file satisfying +// both survives; a file failing either is dropped. +#[tokio::test] +async fn mixed_data_and_partition_predicate_prunes_on_both( +) -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string_partitioned(vec![TestAction::Metadata]), + ) + .await?; + // match: id in [20,30] and val=x. wrong_val: in-range id but val=y. wrong_id: val=x but + // id in [0,5]. Only `match` satisfies `id > 25 AND val = 'x'`. + let v1 = [ + add_partitioned_with_id_stats("match.parquet", "x", 20, 30), + add_partitioned_with_id_stats("wrong_val.parquet", "y", 20, 30), + add_partitioned_with_id_stats("wrong_id.parquet", "x", 0, 5), + ] + .join("\n"); + add_commit(table_root, storage.as_ref(), 1, v1).await?; + + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(engine.as_ref())?; + + let predicate: PredicateRef = Arc::new(Pred::and( + col!("id").gt(lit(25i32)), + col!("val").eq(lit("x")), + )); + let listing = unwrap_listing( + target + .incremental_scan_builder(0) + .with_predicate(predicate) + .build(engine.as_ref())?, + ); + + assert_eq!( + listing.summary.live_adds, + HashSet::from([key("match.parquet")]), + "only the file matching both the data and partition conjuncts survives" + ); + + Ok(()) +} + +// Under column mapping, `with_predicate` lowers the logical predicate column to its physical +// name before reading stats keyed by that physical name. A logical `id > 25` skips against the +// physical-name stats, so only the in-range file survives -- proving the physical-vs-logical +// lowering on this path. +#[tokio::test] +async fn pushdown_resolves_column_mapping_physical_names() -> Result<(), Box> +{ + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + COLUMN_MAPPING_METADATA.to_string(), + ) + .await?; + // Stats are keyed by the physical name `col-id`, while the predicate references logical `id`. + let v1 = [ + add_with_physical_id_stats("low.parquet", 0, 5), + add_with_physical_id_stats("high.parquet", 20, 30), + ] + .join("\n"); + add_commit(table_root, storage.as_ref(), 1, v1).await?; + + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(engine.as_ref())?; + + let predicate: PredicateRef = Arc::new(col!("id").gt(lit(25i32))); + let listing = unwrap_listing( + target + .incremental_scan_builder(0) + .with_predicate(predicate) + .build(engine.as_ref())?, + ); + + assert_eq!( + listing.summary.live_adds, + HashSet::from([key("high.parquet")]), + "predicate on logical `id` resolves to physical `col-id` stats and drops the low file" + ); + + Ok(()) +} + +// A DV-bearing Add is pruned by its stats like any other, and the surviving live Add is keyed by +// its full `(path, dv_unique_id)`, not path alone. Two DV-bearing Adds with disjoint `id` ranges: +// `id > 25` drops the out-of-range one and keeps the in-range one under its DV key. +#[tokio::test] +async fn pushdown_prunes_dv_bearing_adds_keyed_by_dv() -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + DV_AND_ID_METADATA.to_string(), + ) + .await?; + let v1 = [ + add_with_dv_and_id_stats("keep.parquet", "u", "abc", 1, 20, 30), + add_with_dv_and_id_stats("prune.parquet", "u", "xyz", 2, 0, 5), + ] + .join("\n"); + add_commit(table_root, storage.as_ref(), 1, v1).await?; + + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(engine.as_ref())?; + + let predicate: PredicateRef = Arc::new(col!("id").gt(lit(25i32))); + let listing = unwrap_listing( + target + .incremental_scan_builder(0) + .with_predicate(predicate) + .build(engine.as_ref())?, + ); + + assert_eq!( + listing.summary.live_adds, + HashSet::from([key_with_dv("keep.parquet", "uabc@1")]), + "the in-range DV Add survives keyed by its full (path, dv_unique_id)" + ); + + Ok(()) +} + +// A predicate that references no columns and is not statically false (`lit(true)`) resolves to +// `PhysicalPredicate::None`, which folds to keep-all. Distinct from the tautology case (which +// references `id` and keeps all via the filter), this exercises the `None -> KeepAll` arm. +#[tokio::test] +async fn column_free_predicate_keeps_all_added_files() -> Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![TestAction::Metadata]), + ) + .await?; + let v1 = [ + add_with_id_stats("a.parquet", 0, 9), + add_with_id_stats("b.parquet", 20, 30), + ] + .join("\n"); + add_commit(table_root, storage.as_ref(), 1, v1).await?; + + let target = Snapshot::builder_for(table_url) + .at_version(1) + .build(engine.as_ref())?; + + let predicate: PredicateRef = Arc::new(Pred::TRUE); + let listing = unwrap_listing( + target + .incremental_scan_builder(0) + .with_predicate(predicate) + .build(engine.as_ref())?, + ); + + assert_eq!( + listing.summary.live_adds, + HashSet::from([key("a.parquet"), key("b.parquet")]), + "a column-free predicate is ineligible for skipping and keeps every live Add" + ); + + Ok(()) +} + +// === Round-trip against a full scan === +// +// The correctness promise of `with_predicate` is that the incremental live-Add set, reconciled +// against the base, reproduces what a full `Scan` at the target with the same predicate keeps. +// The two paths parse stats differently: a full scan reads pre-parsed `stats_parsed` (from a +// checkpoint), while the incremental path parses `add.stats` JSON via `for_raw_action_batch`. +// +// The incremental scan reads only raw commit JSON and never touches checkpoints, so it cannot +// serve a range predating a checkpoint (the target snapshot's log segment drops pre-checkpoint +// commit JSON). The checkpoint therefore sits at the base version: the full scan at the base +// prunes via the checkpoint's `stats_parsed`, the incremental delta over `(base, target]` prunes +// via JSON, and `full_scan(base) union live_adds` must equal `full_scan(target)`. + +type MtEngine = DefaultEngine; + +// Single-column `id: integer` table. +fn id_schema() -> Arc { + Arc::new(StructType::new_unchecked([StructField::nullable( + "id", + DataType::INTEGER, + )])) +} + +// One `RecordBatch` holding a single `id` value, so each written file gets a tight [id, id] range. +fn id_batch(schema: &Arc, id: i32) -> RecordBatch { + let arrow_schema: ArrowSchema = schema.as_ref().try_into_arrow().unwrap(); + RecordBatch::try_new( + Arc::new(arrow_schema), + vec![Arc::new(Int32Array::from(vec![id]))], + ) + .unwrap() +} + +// Surviving file paths from a full scan at `snapshot` under `predicate`. A snapshot loaded after a +// checkpoint prunes using the checkpoint's `stats_parsed`. +fn full_scan_surviving_paths( + snapshot: SnapshotRef, + engine: &MtEngine, + predicate: PredicateRef, +) -> Result, Box> { + fn insert_path(paths: &mut HashSet, scan_file: ScanFile) { + paths.insert(scan_file.path); + } + + let scan = snapshot.scan_builder().with_predicate(predicate).build()?; + let mut paths: HashSet = HashSet::new(); + for sm in scan.scan_metadata(engine)? { + paths = sm?.visit_scan_files(paths, insert_path)?; + } + Ok(paths) +} + +// Reconciling a base full scan with the incremental delta reproduces a full scan at the target, +// even though the base prunes via the checkpoint's `stats_parsed` while the incremental delta +// prunes via `add.stats` JSON. Guards the stats-parsing divergence the two paths could exhibit. +#[tokio::test(flavor = "multi_thread")] +async fn predicate_pushdown_reconciled_with_base_matches_full_scan_across_checkpoint( +) -> Result<(), Box> { + let (_tmp_dir, table_path, engine) = test_table_setup_mt()?; + let table_url = Url::from_directory_path(&table_path).map_err(|_| "invalid table path")?; + let schema = id_schema(); + let mut snapshot = + create_table_and_load_snapshot(&table_path, schema.clone(), engine.as_ref(), &[])?; + + // Files added before the checkpoint: id 5, 15, 25. The checkpoint captures their stats. + for id in [5, 15, 25] { + snapshot = write_batch_to_table( + &snapshot, + engine.as_ref(), + id_batch(&schema, id), + HashMap::new(), + ) + .await?; + } + let (_, snapshot) = snapshot.checkpoint(engine.as_ref(), None)?; + let base_version = snapshot.version(); + + // Files added after the checkpoint: id 35, 45. These live only in raw commit JSON. + let mut snapshot = snapshot; + for id in [35, 45] { + snapshot = write_batch_to_table( + &snapshot, + engine.as_ref(), + id_batch(&schema, id), + HashMap::new(), + ) + .await?; + } + let target_version = snapshot.version(); + + let predicate: PredicateRef = Arc::new(col!("id").gt(lit(20i32))); + + // Base full scan (reads the checkpoint's `stats_parsed`) and target full scan, both loaded + // fresh off disk so they replay through the checkpoint. + let base = Snapshot::builder_for(table_url.clone()) + .at_version(base_version) + .build(engine.as_ref())?; + let base_paths = full_scan_surviving_paths(base, engine.as_ref(), predicate.clone())?; + + let target = Snapshot::builder_for(table_url) + .at_version(target_version) + .build(engine.as_ref())?; + let target_paths = full_scan_surviving_paths(target, engine.as_ref(), predicate.clone())?; + + // Incremental delta over `(base, target]` (parses `add.stats` JSON). + let listing = unwrap_listing( + snapshot + .incremental_scan_builder(base_version) + .with_predicate(predicate) + .build(engine.as_ref())?, + ); + let live_adds: HashSet = listing + .summary + .live_adds + .iter() + .map(|k| k.path().to_string()) + .collect(); + + // Sanity: the checkpoint kept a pre-checkpoint match (id 25) that the incremental delta does + // not re-report, so the reconciliation is doing real work. + assert!( + !base_paths.is_empty() && !live_adds.is_empty(), + "both the base scan and the incremental delta should contribute matches" + ); + assert!( + listing.summary.removes.is_empty(), + "no removes in this range" + ); + + // No in-range removes, so reconciliation is a union of the base and the incremental live Adds. + let reconciled: HashSet = base_paths.union(&live_adds).cloned().collect(); + assert_eq!( + reconciled, target_paths, + "base full scan plus incremental live_adds must equal the full scan at the target" + ); + + Ok(()) +} diff --git a/kernel/tests/common/mod.rs b/kernel/tests/integration/common/mod.rs similarity index 95% rename from kernel/tests/common/mod.rs rename to kernel/tests/integration/common/mod.rs index a49d460f41..ed6781a2e0 100644 --- a/kernel/tests/common/mod.rs +++ b/kernel/tests/integration/common/mod.rs @@ -1,3 +1,6 @@ +pub mod read_utils; +pub mod write_utils; + #[macro_export] macro_rules! sort_lines { ($lines: expr) => {{ diff --git a/kernel/tests/integration/common/read_utils.rs b/kernel/tests/integration/common/read_utils.rs new file mode 100644 index 0000000000..92719d9a4a --- /dev/null +++ b/kernel/tests/integration/common/read_utils.rs @@ -0,0 +1,17 @@ +//! Shared read helpers for integration tests. + +use std::fs::File; + +use delta_kernel::arrow::array::RecordBatch; +use delta_kernel::arrow::compute::concat_batches; +use delta_kernel::parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder; + +pub fn read_parquet_file(path: &std::path::Path) -> RecordBatch { + let file = File::open(path).expect("failed to open parquet file"); + let reader = ParquetRecordBatchReaderBuilder::try_new(file) + .expect("failed to create parquet reader") + .build() + .expect("failed to build parquet reader"); + let batches: Vec = reader.map(|b| b.unwrap()).collect(); + concat_batches(&batches[0].schema(), &batches).expect("failed to concat batches") +} diff --git a/kernel/tests/integration/common/write_utils.rs b/kernel/tests/integration/common/write_utils.rs new file mode 100644 index 0000000000..3074fb6767 --- /dev/null +++ b/kernel/tests/integration/common/write_utils.rs @@ -0,0 +1,505 @@ +//! Shared helpers for integration tests that exercise Delta write paths. +//! +//! Each integration test file compiles as its own test binary and uses only a subset of these +//! helpers; the blanket `#![allow(dead_code)]` suppresses the per-binary dead-code warnings that +//! the unused subset would otherwise generate. + +#![allow(dead_code)] + +use std::collections::HashMap; +use std::sync::Arc; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; + +use delta_kernel::actions::deletion_vector::{DeletionVectorDescriptor, DeletionVectorStorageType}; +use delta_kernel::actions::deletion_vector_writer::{ + KernelDeletionVector, StreamingDeletionVectorWriter, +}; +use delta_kernel::actions::{MAX_VALUES, MIN_VALUES}; +use delta_kernel::arrow::array::{Int32Array, StructArray}; +use delta_kernel::arrow::record_batch::RecordBatch; +use delta_kernel::engine::arrow_conversion::TryIntoArrow as _; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::engine_data::FilteredEngineData; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::{DynObjectStore, ObjectStore, ObjectStoreExt}; +use delta_kernel::parquet::file::reader::{FileReader, SerializedFileReader}; +use delta_kernel::parquet::schema::types::Type as ParquetType; +use delta_kernel::path::ParsedLogPath; +use delta_kernel::schema::{DataType, SchemaRef, StructField, StructType}; +use delta_kernel::table_features::ColumnMappingMode; +use delta_kernel::transaction::{CommitResult, Transaction, WriteContext}; +use delta_kernel::{DeltaResult, Engine, Snapshot, Version}; +use serde_json::json; +use test_utils::delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor; +use test_utils::delta_kernel_default_engine::DefaultEngine; +use test_utils::{ + begin_transaction, create_add_files_metadata, create_table, engine_store_setup, + into_record_batch, load_and_begin_transaction, modify_add_file_partition_keys, + AddFilePartitionKeyModify, +}; +use url::Url; +use uuid::Uuid; + +/// Deterministic placeholder for test commit JSON comparisons. +pub const ZERO_UUID: &str = "00000000-0000-0000-0000-000000000000"; + +/// Single-column nullable `id: int` schema. +pub fn get_simple_schema() -> SchemaRef { + Arc::new(StructType::try_new(vec![StructField::new("id", DataType::INTEGER, true)]).unwrap()) +} + +/// Builds a `RecordBatch` matching [`get_simple_schema`] from a vector of `id` values. +pub fn simple_id_batch(schema: &SchemaRef, values: Vec) -> RecordBatch { + RecordBatch::try_new( + Arc::new(schema.as_ref().try_into_arrow().unwrap()), + vec![Arc::new(Int32Array::from(values))], + ) + .unwrap() +} + +/// Finds the single checkpoint parquet file in `_delta_log` for the given version. +/// +/// Recognizes V1 single-part, V2 classic-named, and V2 UUID-named variants. Does NOT support +/// V1 multi-part checkpoints. +pub fn load_existing_single_file_checkpoint_path( + table_path: &str, + version: u64, +) -> std::path::PathBuf { + let log_dir = std::path::Path::new(table_path).join("_delta_log"); + for entry in std::fs::read_dir(&log_dir).expect("failed to read _delta_log") { + let entry = entry.unwrap(); + let url = Url::from_file_path(entry.path()).expect("entry path to url"); + let Some(parsed) = ParsedLogPath::try_from(url).expect("parse log path") else { + continue; + }; + if parsed.is_checkpoint() && parsed.version == version { + return entry.path(); + } + } + panic!("checkpoint parquet file not found for version {version} in {log_dir:?}"); +} + +/// Open a parquet file and return its root schema. +pub fn read_parquet_root_schema(parquet_file: &std::path::Path) -> ParquetType { + let file = std::fs::File::open(parquet_file).unwrap(); + let reader = SerializedFileReader::new(file).unwrap(); + reader + .metadata() + .file_metadata() + .schema_descr() + .root_schema() + .clone() +} + +/// Returns the native parquet `field_id` for a field at the given physical path in a parquet file, +/// or `None` if the field has no `field_id` set. +/// +/// Panics if the file cannot be read or the physical path doesn't exist in the parquet schema. +pub fn get_parquet_field_id( + parquet_file: &std::path::Path, + physical_path: &[String], +) -> Option { + let root = read_parquet_root_schema(parquet_file); + let mut current = &root; + for name in physical_path { + current = current + .get_fields() + .iter() + .find(|f| f.name() == name) + .unwrap_or_else(|| panic!("parquet schema missing field '{name}'")); + } + + let info = current.get_basic_info(); + info.has_id().then(|| info.id()) +} + +/// Recursively walks a parquet schema tree and returns a map from each node's absolute +/// dot-separated path to its `field_id`. Top-level fields appear under their bare name. +pub fn collect_all_parquet_field_ids(parquet_file: &std::path::Path) -> HashMap { + fn walk(t: &ParquetType, path: &str, out: &mut HashMap) { + let info = t.get_basic_info(); + if info.has_id() { + out.insert(path.to_string(), info.id() as i64); + } + if let ParquetType::GroupType { fields, .. } = t { + for f in fields { + walk(f, &format!("{path}.{}", f.name()), out); + } + } + } + + let mut ids = HashMap::new(); + if let ParquetType::GroupType { fields, .. } = &read_parquet_root_schema(parquet_file) { + for f in fields { + walk(f, f.name(), &mut ids); + } + } + ids +} + +/// Validate that `commitInfo["txnId"]` is a valid UUID. +pub fn validate_txn_id(commit_info: &serde_json::Value) { + let txn_id = commit_info["txnId"] + .as_str() + .expect("txnId should be present in commitInfo"); + Uuid::parse_str(txn_id).expect("txnId should be valid UUID format"); +} + +/// Validate that `commitInfo["timestamp"]` is within the last two days. +pub fn validate_timestamp(commit_info: &serde_json::Value) { + let timestamp = commit_info["timestamp"] + .as_i64() + .expect("timestamp should be present in commitInfo"); + let current_ts: i64 = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_millis() + .try_into() + .unwrap(); + let two_days_ms = Duration::from_secs(2 * 24 * 60 * 60).as_millis() as i64; + assert!( + (timestamp <= current_ts && timestamp > current_ts - two_days_ms), + "commit timestamp should be at most 2 days behind current system time: got {timestamp}, now {current_ts}" + ); +} + +/// Check that the timestamps in commit_info and add actions are within 10s of SystemTime::now(). +pub fn check_action_timestamps<'a>( + parsed_commits: impl Iterator, +) -> Result<(), Box> { + let now: i64 = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_millis() + .try_into() + .unwrap(); + + parsed_commits.for_each(|commit| { + if let Some(commit_info_ts) = &commit.pointer("/commitInfo/timestamp") { + assert!((now - commit_info_ts.as_i64().unwrap()).abs() < 10_000); + } + if let Some(add_ts) = &commit.pointer("/add/modificationTime") { + assert!((now - add_ts.as_i64().unwrap()).abs() < 10_000); + } + }); + + Ok(()) +} + +/// List all the files at `path`, assert that exactly two are parquet files with matching sizes, +/// and return that size. +pub async fn get_and_check_all_parquet_sizes(store: Arc, path: &str) -> u64 { + use futures::stream::StreamExt; + let files: Vec<_> = store.list(Some(&Path::from(path))).collect().await; + let parquet_files = files + .into_iter() + .filter(|f| match f { + Ok(f) => f.location.extension() == Some("parquet"), + Err(_) => false, + }) + .collect::>(); + assert_eq!(parquet_files.len(), 2); + let size = parquet_files.first().unwrap().as_ref().unwrap().size; + assert!(parquet_files + .iter() + .all(|f| f.as_ref().unwrap().size == size)); + size +} + +/// Write two small parquet files to the table and verify commit-level post-commit stats. +pub async fn write_data_and_check_result_and_stats( + table_url: Url, + schema: SchemaRef, + engine: Arc>, + expected_since_commit: u64, +) -> Result<(), Box> { + let mut txn = test_utils::load_and_begin_transaction(table_url.clone(), engine.as_ref())? + .with_data_change(true); + + // create two new arrow record batches to append + let append_data = [[1, 2, 3], [4, 5, 6]].map(|data| -> DeltaResult<_> { + let data = RecordBatch::try_new( + Arc::new(schema.as_ref().try_into_arrow()?), + vec![Arc::new(Int32Array::from(data.to_vec()))], + )?; + Ok(Box::new(ArrowEngineData::new(data))) + }); + + // write data out by spawning async tasks to simulate executors + let write_context = Arc::new(txn.unpartitioned_write_context().unwrap()); + let tasks = append_data.into_iter().map(|data| { + // arc clones + let engine = engine.clone(); + let write_context = write_context.clone(); + tokio::task::spawn(async move { + engine + .write_parquet(data.as_ref().unwrap(), write_context.as_ref()) + .await + }) + }); + + let add_files_metadata = futures::future::join_all(tasks).await.into_iter().flatten(); + for meta in add_files_metadata { + txn.add_files(meta?); + } + + // commit! + match txn.commit(engine.as_ref())? { + CommitResult::CommittedTransaction(committed) => { + assert_eq!(committed.commit_version(), expected_since_commit as Version); + assert_eq!( + committed.post_commit_stats().commits_since_checkpoint, + expected_since_commit + ); + assert_eq!( + committed.post_commit_stats().commits_since_log_compaction, + expected_since_commit + ); + } + _ => panic!("Commit should have succeeded"), + }; + + Ok(()) +} + +/// A simple schema with a single nullable INTEGER column named `number`. +pub fn get_simple_int_schema() -> Arc { + Arc::new(StructType::try_new(vec![StructField::nullable("number", DataType::INTEGER)]).unwrap()) +} + +/// Write a metadata-update commit that sets a table property on the existing table. +/// Returns a fresh snapshot reflecting the new commit. +/// Used in tests as a hack to set table properties when create table doesn't support the property. +pub fn set_table_properties( + table_path: &str, + table_url: &Url, + engine: &dyn Engine, + current_version: Version, + properties: &[(&str, &str)], +) -> Result, Box> { + let v0_path = std::path::Path::new(table_path).join("_delta_log/00000000000000000000.json"); + let mut meta: serde_json::Value = std::fs::read_to_string(&v0_path)? + .lines() + .find_map(|line| { + serde_json::from_str::(line) + .ok() + .filter(|v| v.get("metaData").is_some()) + }) + .expect("version 0 should contain a metaData action"); + + for &(key, value) in properties { + meta["metaData"]["configuration"][key] = json!(value); + } + + let new_commit = std::path::Path::new(table_path) + .join(format!("_delta_log/{:020}.json", current_version + 1)); + std::fs::write(&new_commit, serde_json::to_string(&meta)?)?; + Ok(Snapshot::builder_for(table_url.clone()).build(engine)?) +} + +/// Assert that the snapshot's column mapping mode matches the given `cm_mode` string, +/// and return the resolved mode. +pub fn assert_column_mapping_mode(snapshot: &Snapshot, cm_mode: &str) -> ColumnMappingMode { + let expected = match cm_mode { + "none" => ColumnMappingMode::None, + "name" => ColumnMappingMode::Name, + "id" => ColumnMappingMode::Id, + _ => panic!("unexpected cm_mode: {cm_mode}"), + }; + let actual = snapshot + .table_properties() + .column_mapping_mode + .expect("column mapping mode should be set"); + assert_eq!(actual, expected); + actual +} + +/// Resolve a nested column inside a [`StructArray`] by walking the given field-name path, +/// and downcast the leaf to the requested array type. +pub fn resolve_struct_field<'a, T: 'static>(root: &'a StructArray, path: &[String]) -> &'a T { + assert!(!path.is_empty(), "path must be non-empty"); + let mut current: &StructArray = root; + for (i, name) in path.iter().enumerate() { + let col = current + .column_by_name(name) + .unwrap_or_else(|| panic!("missing field: {name}")); + if i == path.len() - 1 { + return col + .as_any() + .downcast_ref::() + .expect("leaf array type mismatch"); + } + current = col + .as_any() + .downcast_ref::() + .unwrap_or_else(|| panic!("expected StructArray at field: {name}")); + } + unreachable!() +} + +/// Navigate into a nested JSON value by following a sequence of object keys. +/// E.g. `resolve_json_path(stats, &["address", "street"])` returns `stats["address"]["street"]`. +pub fn resolve_json_path<'a>( + root: &'a serde_json::Value, + path: &[String], +) -> &'a serde_json::Value { + path.iter().fold(root, |v, key| &v[key]) +} + +/// Assert that `stats["minValues"]` and `stats["maxValues"]` at the given physical path equal the +/// expected values. +pub fn assert_min_max_stats( + stats: &serde_json::Value, + physical_path: &[String], + expected_min: impl Into, + expected_max: impl Into, +) { + assert_eq!( + *resolve_json_path(&stats[MIN_VALUES], physical_path), + expected_min.into() + ); + assert_eq!( + *resolve_json_path(&stats[MAX_VALUES], physical_path), + expected_max.into() + ); +} + +/// Creates a table with deletion vector support and writes files with `partition_values`. +/// An empty `partition_values` creates an unpartitioned table. +pub async fn create_dv_table_with_files( + table_name: &str, + schema: Arc, + partition_values: &[(&str, Option<&str>)], + file_paths: &[&str], +) -> Result< + ( + Arc, + Arc, + Url, + Vec, + ), + Box, +> { + let (store, engine, table_url) = engine_store_setup(table_name, None); + let engine = Arc::new(engine); + let partition_columns: Vec<_> = partition_values.iter().map(|(key, _)| *key).collect(); + + // Create table with DV support (protocol 3/7 with deletionVectors feature) + create_table( + store.clone(), + table_url.clone(), + schema.clone(), + &partition_columns, + true, // use_37_protocol + vec!["deletionVectors"], + vec!["deletionVectors"], + ) + .await?; + + // Write files + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let mut txn = begin_transaction(snapshot.clone(), engine.as_ref())? + .with_engine_info("test engine") + .with_operation("WRITE".to_string()) + .with_data_change(true); + + let add_files_schema = txn.add_files_schema(); + + // Build metadata for all files at once + let files: Vec<(&str, i64, i64, Option)> = file_paths + .iter() + .enumerate() + .map(|(i, &path)| { + ( + path, + 1024 + i as i64 * 100, // size + 1000000 + i as i64, // mod_time + Some(3), // num_records + ) + }) + .collect(); + let metadata = create_add_files_metadata(add_files_schema, files)?; + let metadata = if partition_values.is_empty() { + metadata + } else { + let modifications: Vec<_> = partition_values + .iter() + .map(|(key, value)| AddFilePartitionKeyModify::Insert { key, value: *value }) + .collect(); + Box::new(ArrowEngineData::new(modify_add_file_partition_keys( + into_record_batch(metadata), + &modifications, + ))) + }; + txn.add_files(metadata); + + let _ = txn.commit(engine.as_ref())?; + + let paths: Vec = file_paths.iter().map(|&s| s.to_string()).collect(); + Ok((store, engine, table_url, paths)) +} + +/// Build a `path -> descriptor` map assigning each file a distinct synthetic DV descriptor. +pub fn sequential_dv_descriptors( + file_paths: &[String], +) -> HashMap { + file_paths + .iter() + .enumerate() + .map(|(idx, path)| { + ( + path.clone(), + DeletionVectorDescriptor { + storage_type: DeletionVectorStorageType::PersistedRelative, + path_or_inline_dv: format!("dv_file_{idx}.bin"), + offset: Some(idx as i32 * 10), + size_in_bytes: 40 + idx as i32, + cardinality: idx as i64 + 1, + }, + ) + }) + .collect() +} + +/// Extracts scan files from a snapshot for use in deletion vector updates. +pub fn get_scan_files( + snapshot: Arc, + engine: &dyn delta_kernel::Engine, +) -> DeltaResult> { + let scan = snapshot.scan_builder().build()?; + let all_scan_metadata: Vec<_> = scan.scan_metadata(engine)?.collect::, _>>()?; + + Ok(all_scan_metadata + .into_iter() + .map(|sm| sm.scan_files) + .collect()) +} + +/// Serialize a deletion vector, write it to the object store, and return its descriptor. +pub async fn write_deletion_vector_to_store( + store: &Arc, + write_context: &WriteContext, + dv: KernelDeletionVector, + prefix: &str, +) -> Result> { + let dv_path = write_context.new_deletion_vector_path(String::from(prefix)); + let dv_object_path = Path::parse(dv_path.absolute_path()?.path())?; + + let mut dv_buffer = Vec::new(); + let mut dv_writer = StreamingDeletionVectorWriter::new(&mut dv_buffer); + let dv_write_result = dv_writer.write_deletion_vector(dv)?; + dv_writer.finalize()?; + + ObjectStoreExt::put(store, &dv_object_path, dv_buffer.into()).await?; + + Ok(dv_write_result.to_descriptor(&dv_path)) +} + +/// Begin a transaction set up for deletion vector updates. +pub fn create_dv_update_transaction( + table_url: &Url, + engine: &dyn Engine, +) -> Result> { + Ok(load_and_begin_transaction(table_url.clone(), engine)? + .with_engine_info("test engine") + .with_operation("DELETE".to_string())) +} diff --git a/kernel/tests/create_table/clustering.rs b/kernel/tests/integration/create_table/clustering.rs similarity index 86% rename from kernel/tests/create_table/clustering.rs rename to kernel/tests/integration/create_table/clustering.rs index b6ac41e8ad..85aaa322b0 100644 --- a/kernel/tests/create_table/clustering.rs +++ b/kernel/tests/integration/create_table/clustering.rs @@ -3,7 +3,7 @@ use std::sync::Arc; use delta_kernel::committer::FileSystemCommitter; -use delta_kernel::expressions::ColumnName; +use delta_kernel::expressions::{column_name, ColumnName}; use delta_kernel::schema::{DataType, StructField, StructType}; use delta_kernel::snapshot::Snapshot; use delta_kernel::table_features::TableFeature; @@ -24,45 +24,40 @@ fn clustering_test_schema() -> DeltaResult> { StructField::new("zip", DataType::STRING, true), ])?; let l4 = StructType::try_new(vec![StructField::new("value", DataType::DOUBLE, true)])?; - let l3 = StructType::try_new(vec![StructField::new( - "l4", - DataType::Struct(Box::new(l4)), - true, - )])?; - let l2 = StructType::try_new(vec![StructField::new( - "l3", - DataType::Struct(Box::new(l3)), - true, - )])?; - let l1 = StructType::try_new(vec![StructField::new( - "l2", - DataType::Struct(Box::new(l2)), - true, - )])?; + let l3 = StructType::try_new(vec![StructField::new("l4", l4, true)])?; + let l2 = StructType::try_new(vec![StructField::new("l3", l3, true)])?; + let l1 = StructType::try_new(vec![StructField::new("l2", l2, true)])?; Ok(Arc::new(StructType::try_new(vec![ - StructField::new("id", DataType::INTEGER, false), + StructField::new("id", DataType::INTEGER, true), StructField::new("name", DataType::STRING, true), - StructField::new("address", DataType::Struct(Box::new(address)), true), - StructField::new("l1", DataType::Struct(Box::new(l1)), true), + StructField::new("address", address, true), + StructField::new("l1", l1, true), ])?)) } #[rstest] #[case::top_level(vec![vec!["id"]])] #[case::nested_2(vec![vec!["address", "city"]])] +#[case::mismatched_casing(vec![vec!["ID"]])] +#[case::nested_mismatched_casing(vec![vec!["ADDRESS", "CITY"]])] #[case::mixed(vec![vec!["id"], vec!["name"], vec!["address", "city"], vec!["address", "zip"], vec!["l1", "l2", "l3", "l4", "value"]])] #[tokio::test] async fn test_create_clustered_table(#[case] col_paths: Vec>) -> DeltaResult<()> { let (_temp_dir, table_path, engine) = test_table_setup()?; let schema = clustering_test_schema()?; - let expected_cols: Vec = col_paths + let input_cols: Vec = col_paths .iter() .map(|p| ColumnName::new(p.iter().copied())) .collect(); + // Schema fields are all lowercase; normalization converts user casing to schema casing + let expected_cols: Vec = col_paths + .iter() + .map(|p| ColumnName::new(p.iter().map(|s| s.to_lowercase()).collect::>())) + .collect(); let txn = create_table(&table_path, schema, "Test/1.0") .with_data_layout(DataLayout::Clustered { - columns: expected_cols.clone(), + columns: input_cols, }) .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))?; @@ -79,7 +74,7 @@ async fn test_create_clustered_table(#[case] col_paths: Vec>) -> Delta let table_url = delta_kernel::try_parse_uri(&table_path)?; let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; - let clustering_columns = snapshot.get_clustering_columns_physical(engine.as_ref())?; + let clustering_columns = snapshot.get_physical_clustering_columns(engine.as_ref())?; assert_eq!(clustering_columns, Some(expected_cols)); let table_configuration = snapshot.table_configuration(); @@ -95,7 +90,8 @@ async fn test_create_clustered_table(#[case] col_paths: Vec>) -> Delta Ok(()) } -/// Test that combining explicit feature signals with auto-enabled features doesn't create duplicates. +/// Test that combining explicit feature signals with auto-enabled features doesn't create +/// duplicates. /// /// This tests the edge case where a user provides `delta.feature.domainMetadata=supported` /// AND uses `DataLayout::Clustered`. Both would try to add DomainMetadata, but we should @@ -133,8 +129,8 @@ async fn test_clustering_with_explicit_feature_signal_no_duplicates() -> DeltaRe ); // Verify clustering columns via snapshot read path - let clustering_columns = snapshot.get_clustering_columns_physical(engine.as_ref())?; - assert_eq!(clustering_columns, Some(vec![ColumnName::new(["id"])])); + let clustering_columns = snapshot.get_physical_clustering_columns(engine.as_ref())?; + assert_eq!(clustering_columns, Some(vec![column_name!("id")])); Ok(()) } diff --git a/kernel/tests/create_table/column_mapping.rs b/kernel/tests/integration/create_table/column_mapping.rs similarity index 55% rename from kernel/tests/create_table/column_mapping.rs rename to kernel/tests/integration/create_table/column_mapping.rs index 0e8eb76d04..f29cf8c3e3 100644 --- a/kernel/tests/create_table/column_mapping.rs +++ b/kernel/tests/integration/create_table/column_mapping.rs @@ -9,18 +9,22 @@ use std::sync::Arc; use delta_kernel::committer::FileSystemCommitter; use delta_kernel::schema::{ - ArrayType, ColumnMetadataKey, DataType, MapType, StructField, StructType, + schema_ref, ArrayType, ColumnMetadataKey, DataType, MapType, MetadataValue, StructField, + StructType, }; use delta_kernel::snapshot::Snapshot; use delta_kernel::table_features::{ColumnMappingMode, TableFeature}; use delta_kernel::transaction::create_table::create_table; use delta_kernel::transaction::data_layout::DataLayout; use delta_kernel::DeltaResult; -use test_utils::{create_table_and_load_snapshot, test_table_setup}; +use test_utils::{ + column_mapping_fixtures as fixtures, create_table_and_load_snapshot, test_table_setup, +}; use super::simple_schema; -/// Helper to strip column mapping metadata (IDs and physical names) from all StructFields recursively. +/// Helper to strip column mapping metadata (IDs and physical names) from all StructFields +/// recursively. pub(super) fn strip_column_mapping_metadata(schema: &StructType) -> StructType { let cm_id = ColumnMetadataKey::ColumnMappingId.as_ref(); let cm_name = ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(); @@ -38,7 +42,7 @@ pub(super) fn strip_column_mapping_metadata(schema: &StructType) -> StructType { match dt { DataType::Struct(s) => { let fields: Vec<_> = s.fields().map(|f| strip_field(f, cm_id, cm_name)).collect(); - DataType::Struct(Box::new(StructType::new_unchecked(fields))) + DataType::from(StructType::new_unchecked(fields)) } DataType::Array(a) => DataType::from(ArrayType::new( strip_data_type(a.element_type(), cm_id, cm_name), @@ -63,7 +67,9 @@ pub(super) fn strip_column_mapping_metadata(schema: &StructType) -> StructType { /// Assert column mapping configuration on a snapshot. /// /// For `Name` / `Id`: feature supported & enabled, mode matches, `maxColumnId` equals -/// the recursive field count. +/// the maximum `delta.columnMapping.id` reachable in the schema. Callers can pre-populate +/// CM annotations on the input schema, in which case the persisted `maxColumnId` reflects +/// any preserved IDs (matching Spark's `assignColumnIdAndPhysicalName`). /// /// For `None`: mode is `None`, no `maxColumnId`, and no column mapping metadata (IDs or /// physical names) on any field. Note: whether `ColumnMapping` appears in the protocol @@ -89,7 +95,12 @@ pub(super) fn assert_column_mapping_config(snapshot: &Snapshot, expected_mode: C "ColumnMapping feature should be enabled" ); - let expected_max_id = snapshot.schema().total_struct_fields(); + let expected_max_id = snapshot + .schema() + .fields() + .flat_map(|f| f.collect_column_mapping_ids()) + .max() + .expect("CM-enabled table must have at least one column mapping id"); let max_id_str = expected_max_id.to_string(); let config = table_config.metadata().configuration(); assert_eq!( @@ -97,7 +108,7 @@ pub(super) fn assert_column_mapping_config(snapshot: &Snapshot, expected_mode: C .get("delta.columnMapping.maxColumnId") .map(|s| s.as_str()), Some(max_id_str.as_str()), - "maxColumnId should equal the total number of struct fields ({expected_max_id})" + "maxColumnId should equal the largest column mapping id in the schema ({expected_max_id})" ); } ColumnMappingMode::None => { @@ -111,9 +122,7 @@ pub(super) fn assert_column_mapping_config(snapshot: &Snapshot, expected_mode: C // No column mapping metadata on any field for field in snapshot.schema().fields() { assert!( - field - .get_config_value(&ColumnMetadataKey::ColumnMappingId) - .is_none(), + field.column_mapping_id().is_none(), "Field '{}' should not have a column mapping ID when mode is None", field.name() ); @@ -151,7 +160,7 @@ fn test_create_table_with_column_mapping_name_mode() -> DeltaResult<()> { let id_field = read_schema.field("id").expect("id field should exist"); assert_eq!(id_field.data_type(), &DataType::INTEGER); - assert!(!id_field.is_nullable()); + assert!(id_field.is_nullable()); let value_field = read_schema .field("value") @@ -166,11 +175,7 @@ fn test_create_table_with_column_mapping_name_mode() -> DeltaResult<()> { fn test_create_table_with_column_mapping_id_mode() -> DeltaResult<()> { let (_temp_dir, table_path, engine) = test_table_setup()?; - let schema = Arc::new(StructType::try_new(vec![StructField::new( - "id", - DataType::INTEGER, - false, - )])?); + let schema = schema_ref! { nullable "id": INTEGER }; // Create table and load snapshot (validates column mapping on read) let snapshot = create_table_and_load_snapshot( @@ -187,7 +192,7 @@ fn test_create_table_with_column_mapping_id_mode() -> DeltaResult<()> { assert_eq!(read_schema.fields().count(), 1); let id_field = read_schema.field("id").expect("id field should exist"); assert_eq!(id_field.data_type(), &DataType::INTEGER); - assert!(!id_field.is_nullable()); + assert!(id_field.is_nullable()); Ok(()) } @@ -258,7 +263,7 @@ fn test_column_mapping_invalid_mode_rejected() { let (_temp_dir, table_path, engine) = test_table_setup().unwrap(); let schema = Arc::new( - StructType::try_new(vec![StructField::new("id", DataType::INTEGER, false)]).unwrap(), + StructType::try_new(vec![StructField::new("id", DataType::INTEGER, true)]).unwrap(), ); // Try to create table with invalid column mapping mode @@ -273,6 +278,75 @@ fn test_column_mapping_invalid_mode_rejected() { .contains("Invalid column mapping mode")); } +/// CREATE TABLE with column mapping disabled strips every column-mapping annotation the input +/// schema carries: a new table has no prior schema, so the annotations are newly introduced and +/// dropped (matching delta-spark's `dropColumnMappingMetadata`), leaving a clean persisted schema +/// rather than a self-inconsistent table. Parametrized over each detected key. +#[rstest::rstest] +fn test_create_table_strips_stale_column_mapping_when_disabled( + #[values( + ColumnMetadataKey::ColumnMappingId, + ColumnMetadataKey::ColumnMappingPhysicalName, + ColumnMetadataKey::ColumnMappingNestedIds, + ColumnMetadataKey::ParquetFieldId, + ColumnMetadataKey::ParquetFieldNestedIds + )] + key: ColumnMetadataKey, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("value", DataType::INTEGER) + .add_metadata([(key.as_ref(), MetadataValue::Number(2))]), + ])?); + + // No column mapping mode set -> mode resolves to None -> the stray annotation is stripped. + let snapshot = create_table_and_load_snapshot(&table_path, schema, engine.as_ref(), &[])?; + + // Mode is None and the persisted schema carries no residual column-mapping metadata. + assert_column_mapping_config(&snapshot, ColumnMappingMode::None); + let value = snapshot.schema().field("value").unwrap().clone(); + assert!( + value.get_config_value(&key).is_none(), + "stray {} should be stripped in None mode", + key.as_ref() + ); + Ok(()) +} + +/// The strip is `None`-mode-only. With column mapping enabled (`id` / `name`) a field's +/// pre-populated annotations are preserved (delta-spark's `assignColumnIdAndPhysicalName` keeps +/// existing ids); only `None` mode drops them. +#[rstest::rstest] +#[case::none(ColumnMappingMode::None, &[], false)] +#[case::id(ColumnMappingMode::Id, &[("delta.columnMapping.mode", "id")], true)] +#[case::name(ColumnMappingMode::Name, &[("delta.columnMapping.mode", "name")], true)] +fn test_create_table_column_mapping_strip_is_none_mode_only( + #[case] expected_mode: ColumnMappingMode, + #[case] properties: &[(&str, &str)], + #[case] annotation_kept: bool, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + fixtures::cm_field("value", 2, "col-2f8a", DataType::INTEGER), + ])?); + + let snapshot = + create_table_and_load_snapshot(&table_path, schema, engine.as_ref(), properties)?; + + assert_column_mapping_config(&snapshot, expected_mode); + let value = snapshot.schema().field("value").unwrap().clone(); + assert_eq!( + value.column_mapping_id().is_some(), + annotation_kept, + "column mapping id under {expected_mode:?} mode" + ); + Ok(()) +} + /// Test cases for clustering columns with column mapping enabled. /// Each case specifies: (logical_column_names, description) #[rstest::rstest] @@ -307,7 +381,7 @@ fn test_create_clustered_table_with_column_mapping( assert!(table_config.is_feature_supported(&TableFeature::DomainMetadata)); // Verify clustering domain metadata exists and uses physical column names - let clustering_columns = snapshot.get_clustering_columns_physical(engine.as_ref())?; + let clustering_columns = snapshot.get_physical_clustering_columns(engine.as_ref())?; let columns = clustering_columns.expect("Clustering columns should be present"); assert_eq!( columns.len(), @@ -342,8 +416,8 @@ fn test_column_mapping_nested_schema() -> DeltaResult<()> { ])?; let schema = Arc::new(StructType::try_new(vec![ - StructField::new("id", DataType::INTEGER, false), - StructField::new("address", DataType::Struct(Box::new(address_type)), true), + StructField::new("id", DataType::INTEGER, true), + StructField::new("address", address_type, true), ])?); // Create table and load snapshot (validates column mapping for nested schema on read) @@ -364,7 +438,7 @@ fn test_column_mapping_nested_schema() -> DeltaResult<()> { // Verify top-level fields let id_field = read_schema.field("id").expect("id field should exist"); assert_eq!(id_field.data_type(), &DataType::INTEGER); - assert!(!id_field.is_nullable()); + assert!(id_field.is_nullable()); let address_field = read_schema .field("address") @@ -403,32 +477,14 @@ fn test_column_mapping_schema_with_maps_and_arrays() -> DeltaResult<()> { // metadata: struct< // labels: map> // > - let labels_type = MapType::new( - DataType::STRING, - ArrayType::new(DataType::INTEGER, true), - true, - ); - - let metadata_type = StructType::try_new(vec![StructField::new( - "labels", - DataType::from(labels_type), - true, - )])?; - - let schema = Arc::new(StructType::try_new(vec![ - StructField::new("id", DataType::INTEGER, false), - StructField::new( - "tags", - DataType::from(MapType::new(DataType::STRING, DataType::STRING, true)), - true, - ), - StructField::new( - "scores", - DataType::from(ArrayType::new(DataType::INTEGER, true)), - true, - ), - StructField::new("metadata", DataType::Struct(Box::new(metadata_type)), true), - ])?); + let schema = schema_ref! { + nullable "id": INTEGER, + nullable "tags": { STRING => nullable STRING }, + nullable "scores": [nullable INTEGER], + nullable "metadata": { + nullable "labels": { STRING => nullable [nullable INTEGER] }, + }, + }; // Create table with column mapping and read back the snapshot. // The snapshot read exercises validate_schema_column_mapping, which verifies @@ -454,33 +510,16 @@ fn test_column_mapping_schema_with_maps_and_arrays() -> DeltaResult<()> { /// Builds a schema that supports clustering at depths 1, 2, and 5: /// { id: int, name: string, address: { city: string, zip: string }, /// l1: { l2: { l3: { l4: { value: double } } } } } -fn clustering_cm_test_schema() -> DeltaResult> { - let address = StructType::try_new(vec![ - StructField::new("city", DataType::STRING, true), - StructField::new("zip", DataType::STRING, true), - ])?; - let l4 = StructType::try_new(vec![StructField::new("value", DataType::DOUBLE, true)])?; - let l3 = StructType::try_new(vec![StructField::new( - "l4", - DataType::Struct(Box::new(l4)), - true, - )])?; - let l2 = StructType::try_new(vec![StructField::new( - "l3", - DataType::Struct(Box::new(l3)), - true, - )])?; - let l1 = StructType::try_new(vec![StructField::new( - "l2", - DataType::Struct(Box::new(l2)), - true, - )])?; - Ok(Arc::new(StructType::try_new(vec![ - StructField::new("id", DataType::INTEGER, false), - StructField::new("name", DataType::STRING, true), - StructField::new("address", DataType::Struct(Box::new(address)), true), - StructField::new("l1", DataType::Struct(Box::new(l1)), true), - ])?)) +fn clustering_cm_test_schema() -> Arc { + schema_ref! { + nullable "id": INTEGER, + nullable "name": STRING, + nullable "address": { + nullable "city": STRING, + nullable "zip": STRING, + }, + nullable "l1": { nullable "l2": { nullable "l3": { nullable "l4": { nullable "value": DOUBLE } } } }, + } } #[rstest::rstest] @@ -501,7 +540,7 @@ fn test_create_clustered_table_nested_with_column_mapping( use delta_kernel::expressions::ColumnName; let (_temp_dir, table_path, engine) = test_table_setup()?; - let schema = clustering_cm_test_schema()?; + let schema = clustering_cm_test_schema(); let expected_cols: Vec = col_paths .iter() .map(|p| ColumnName::new(p.iter().copied())) @@ -535,7 +574,7 @@ fn test_create_clustered_table_nested_with_column_mapping( }; assert_column_mapping_config(&snapshot, expected_cm_mode); - let clustering_columns = snapshot.get_clustering_columns_physical(engine.as_ref())?; + let clustering_columns = snapshot.get_physical_clustering_columns(engine.as_ref())?; let columns = clustering_columns.expect("Clustering columns should be present"); assert_eq!(columns.len(), expected_cols.len()); @@ -559,3 +598,256 @@ fn test_create_clustered_table_nested_with_column_mapping( Ok(()) } + +#[rstest::rstest] +#[case::single_column(&["id"])] +#[case::multiple_columns(&["id", "date"])] +fn test_partitioned_table_stores_logical_column_names_with_column_mapping( + #[case] partition_cols: &[&str], +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let schema = super::partition_test_schema()?; + + let _ = create_table(&table_path, schema, "Test/1.0") + .with_table_properties([("delta.columnMapping.mode", "name")]) + .with_data_layout(DataLayout::partitioned(partition_cols.iter().copied())) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let table_url = delta_kernel::try_parse_uri(&table_path)?; + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + + assert_column_mapping_config(&snapshot, ColumnMappingMode::Name); + + let log_file_path = format!("{table_path}/_delta_log/00000000000000000000.json"); + let log_contents = std::fs::read_to_string(&log_file_path).expect("Failed to read log file"); + let actions: Vec = log_contents + .lines() + .map(|line| serde_json::from_str(line).expect("Failed to parse JSON")) + .collect(); + + let metadata_action = actions + .iter() + .find(|a| a.get("metaData").is_some()) + .expect("Should have metaData action"); + let metadata = metadata_action.get("metaData").unwrap(); + let stored_partition_columns: Vec = metadata["partitionColumns"] + .as_array() + .expect("partitionColumns should be an array") + .iter() + .map(|v| v.as_str().unwrap().to_string()) + .collect(); + + assert_eq!(stored_partition_columns.len(), partition_cols.len()); + + for (i, stored_name) in stored_partition_columns.iter().enumerate() { + let logical_name = partition_cols[i]; + assert_eq!( + stored_name, logical_name, + "partition column {i} should be logical name '{logical_name}', got '{stored_name}'" + ); + } + + let clustering = snapshot.get_physical_clustering_columns(engine.as_ref())?; + assert!( + clustering.is_none(), + "Partitioned table should not have clustering columns" + ); + + Ok(()) +} + +/// Create table with pre-existing `delta.columnMapping.physicalName` annotations, +/// verify the physical name duplicate validation logic. +/// +/// - **accept**: same leaf physical name under two different parent structs. +/// - **reject**: two siblings inside `struct -> map -> array -> struct` share same physical name. +#[rstest::rstest] +#[case::accept(fixtures::same_leaf_phy_name_under_different_parents(), None)] +#[case::reject( + fixtures::nested_field_with_same_phy_path(), + Some("Duplicate `delta.columnMapping.physicalName`") +)] +fn test_create_table_dup_physical_name( + #[case] schema: StructType, + #[case] expected_error_substring: Option<&str>, + #[values("name", "id")] cm_mode: &str, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let result = create_table(&table_path, Arc::new(schema), "Test/1.0") + .with_table_properties([("delta.columnMapping.mode", cm_mode)]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new())); + + match expected_error_substring { + None => { + let _commit = result?.commit(engine.as_ref())?; + } + Some(substr) => { + let msg = result + .expect_err("dup physicalName must be rejected at create-table build") + .to_string(); + assert!( + msg.contains(substr) && msg.contains(".a'") && msg.contains(".b'"), + "expected path-aware dedup error naming colliding leaves under {cm_mode}, got: {msg}" + ); + } + } + Ok(()) +} + +// ============================================================================ +// CREATE TABLE accepts pre-populated column mapping metadata (Spark parity). +// See https://github.com/delta-io/delta-kernel-rs/issues/2377. +// ============================================================================ + +/// Connector-supplied CM metadata survives create -> reload: a complete annotation is kept +/// verbatim, a lone `id` keeps its value and gets a generated `col-` physical name, and +/// a lone `physicalName` is kept while an id is allocated. `assert_column_mapping_config` also +/// pins `maxColumnId` to the largest reachable id. +#[rstest::rstest] +#[case::complete( + fixtures::cm_field("f", 5, "phys-f", DataType::INTEGER), + Some(5), + Some("phys-f") +)] +#[case::id_only(fixtures::cm_field_id_only("f", 7, DataType::INTEGER), Some(7), None)] +#[case::physical_name_only( + fixtures::cm_field_physical_name_only("f", "user-supplied", DataType::INTEGER), + None, + Some("user-supplied") +)] +fn test_create_table_preserves_or_fills_cm_metadata( + #[case] field: StructField, + #[case] expected_id: Option, + #[case] expected_physical: Option<&str>, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let schema = Arc::new(StructType::try_new(vec![field])?); + + let snapshot = create_table_and_load_snapshot( + &table_path, + schema, + engine.as_ref(), + &[("delta.columnMapping.mode", "name")], + )?; + + assert_column_mapping_config(&snapshot, ColumnMappingMode::Name); + let read_schema = snapshot.schema(); + let f = read_schema.field("f").unwrap(); + + match expected_id { + Some(id) => assert_eq!(f.column_mapping_id(), Some(id)), + // No id supplied: kernel allocates one. + None => assert!(f.column_mapping_id().expect("id must be allocated") > 0), + } + match expected_physical { + Some(phys) => assert_eq!( + f.get_config_value(&ColumnMetadataKey::ColumnMappingPhysicalName), + Some(&MetadataValue::String(phys.to_string())) + ), + // No physical name supplied: kernel fills `col-`. + None => match f.get_config_value(&ColumnMetadataKey::ColumnMappingPhysicalName) { + Some(MetadataValue::String(s)) => assert!(s.starts_with("col-")), + other => panic!("expected col-, got {other:?}"), + }, + } + + Ok(()) +} + +/// Sparse preserved ids (1, 5, 100) seed assignment: bare fields get ids strictly above the +/// preserved max, in field order, and `maxColumnId` (checked by `assert_column_mapping_config`) +/// lands on the largest assigned id. +#[test] +fn test_create_table_sparse_preserved_ids_seed_assignment() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let schema = Arc::new(StructType::try_new(vec![ + StructField::new("bare1", DataType::STRING, true), + fixtures::cm_field("a", 1, "phys-a", DataType::INTEGER), + StructField::new("bare2", DataType::STRING, true), + fixtures::cm_field("b", 100, "phys-b", DataType::INTEGER), + fixtures::cm_field("c", 5, "phys-c", DataType::INTEGER), + StructField::new("bare3", DataType::STRING, true), + ])?); + + let snapshot = create_table_and_load_snapshot( + &table_path, + schema, + engine.as_ref(), + &[("delta.columnMapping.mode", "name")], + )?; + + assert_column_mapping_config(&snapshot, ColumnMappingMode::Name); + let read_schema = snapshot.schema(); + + // Preserved ids untouched. + assert_eq!(read_schema.field("a").unwrap().column_mapping_id(), Some(1)); + assert_eq!( + read_schema.field("b").unwrap().column_mapping_id(), + Some(100) + ); + assert_eq!(read_schema.field("c").unwrap().column_mapping_id(), Some(5)); + + // Bare fields seed above the preserved max (100), in field order. + for (name, expected_id) in [("bare1", 101), ("bare2", 102), ("bare3", 103)] { + assert_eq!( + read_schema.field(name).unwrap().column_mapping_id(), + Some(expected_id), + "bare field {name} should get id {expected_id}" + ); + } + + Ok(()) +} + +/// Preserved CM metadata under a nested struct is kept verbatim; bare fields elsewhere are +/// assigned ids strictly above the nested preserved max, in field order. +#[test] +fn test_create_table_preserves_preexisting_metadata_in_nested_types() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // Nested struct with one fully-annotated leaf preserved at id=42. + let nested_struct = StructType::try_new(vec![fixtures::cm_field( + "leaf", + 42, + "phys-leaf", + DataType::INTEGER, + )])?; + + let schema = Arc::new(StructType::try_new(vec![ + StructField::new("bare", DataType::STRING, true), + StructField::new("outer", DataType::Struct(Box::new(nested_struct)), true), + ])?); + + let snapshot = create_table_and_load_snapshot( + &table_path, + schema, + engine.as_ref(), + &[("delta.columnMapping.mode", "name")], + )?; + + assert_column_mapping_config(&snapshot, ColumnMappingMode::Name); + let read_schema = snapshot.schema(); + + // Preserved leaf kept verbatim. + let outer = read_schema.field("outer").unwrap(); + let DataType::Struct(inner) = outer.data_type() else { + panic!("outer must remain a struct"); + }; + let leaf = inner.field("leaf").unwrap(); + assert_eq!(leaf.column_mapping_id(), Some(42)); + assert_eq!( + leaf.get_config_value(&ColumnMetadataKey::ColumnMappingPhysicalName), + Some(&MetadataValue::String("phys-leaf".to_string())) + ); + + // bare and outer seed above the nested preserved max (42), in field order. + assert_eq!( + read_schema.field("bare").unwrap().column_mapping_id(), + Some(43) + ); + assert_eq!(outer.column_mapping_id(), Some(44)); + + Ok(()) +} diff --git a/kernel/tests/create_table/ctas.rs b/kernel/tests/integration/create_table/ctas.rs similarity index 94% rename from kernel/tests/create_table/ctas.rs rename to kernel/tests/integration/create_table/ctas.rs index 44f99f5366..c52bfec3cd 100644 --- a/kernel/tests/create_table/ctas.rs +++ b/kernel/tests/integration/create_table/ctas.rs @@ -7,15 +7,14 @@ use std::collections::HashMap; use std::sync::Arc; +use delta_kernel::actions::MIN_VALUES; use delta_kernel::arrow::array::{Array, Int64Array, StringArray, StructArray}; use delta_kernel::committer::FileSystemCommitter; use delta_kernel::engine::arrow_data::ArrowEngineData; -use delta_kernel::engine::default::executor::tokio::TokioMultiThreadExecutor; -use delta_kernel::engine::default::DefaultEngineBuilder; -use delta_kernel::expressions::ColumnName; +use delta_kernel::expressions::{column_name, ColumnName}; use delta_kernel::object_store::local::LocalFileSystem; use delta_kernel::object_store::path::Path; -use delta_kernel::object_store::DynObjectStore; +use delta_kernel::object_store::{DynObjectStore, ObjectStoreExt as _}; use delta_kernel::snapshot::Snapshot; use delta_kernel::table_features::{ get_any_level_column_physical_name, ColumnMappingMode, TableFeature, @@ -24,12 +23,13 @@ use delta_kernel::transaction::create_table::create_table; use delta_kernel::transaction::data_layout::DataLayout; use delta_kernel::transaction::CommitResult; use delta_kernel::{Engine, FileMeta}; -use url::Url; - +use test_utils::delta_kernel_default_engine::executor::tokio::TokioMultiThreadExecutor; +use test_utils::delta_kernel_default_engine::DefaultEngineBuilder; use test_utils::{ assert_schema_has_field, nested_batches, nested_schema, read_add_infos, test_table_setup, write_batch_to_table, }; +use url::Url; const VERIFIED_PATHS: &[&[&str]] = &[&["row_number"], &["address", "street"]]; @@ -69,10 +69,10 @@ fn verify_column_names_in_stats( let stats = add_actions .iter() .filter_map(|a| a.stats.as_ref()) - .find(|s| s.get("minValues").is_some()); + .find(|s| s.get(MIN_VALUES).is_some()); if let Some(stats) = stats { - let min_values = &stats["minValues"]; + let min_values = &stats[MIN_VALUES]; for logical_path in VERIFIED_PATHS { let col = ColumnName::new(logical_path.iter().copied()); let expected = @@ -102,7 +102,7 @@ fn verify_column_names_in_clustering_metadata( ) -> Result<(), Box> { let schema = snapshot.schema(); let clustering_columns = snapshot - .get_clustering_columns_physical(engine)? + .get_physical_clustering_columns(engine)? .expect("Clustering columns should be present"); assert_eq!( @@ -111,7 +111,7 @@ fn verify_column_names_in_clustering_metadata( "Expected exactly one clustering column" ); let stored_path = clustering_columns[0].path(); - let col = ColumnName::new(["row_number"]); + let col = column_name!("row_number"); let expected = get_any_level_column_physical_name(schema.as_ref(), &col, cm_mode)?.into_inner(); assert_eq!( @@ -189,8 +189,8 @@ fn cm_mode_property(mode: ColumnMappingMode) -> Option<&'static str> { /// 3. Scan all data from the source table /// 4. Create target table and write scanned data in a single CTAS transaction /// 5. Verify target version, feature flags, and column naming consistency -/// 6. Verify data integrity: scan target and check row count, row_number -/// values, and nested address.street values all match the source +/// 6. Verify data integrity: scan target and check row count, row_number values, and nested +/// address.street values all match the source async fn run_ctas_test( src_cm: ColumnMappingMode, src_clustered: bool, @@ -260,13 +260,9 @@ async fn run_ctas_test( } let mut tgt_txn = tgt_builder.build(engine.as_ref(), Box::new(FileSystemCommitter::new()))?; - let write_context = Arc::new(tgt_txn.get_write_context()); + let write_context = Arc::new(tgt_txn.unpartitioned_write_context()?); let add_meta = engine - .write_parquet( - &ArrowEngineData::new(source_data), - write_context.as_ref(), - HashMap::new(), - ) + .write_parquet(&ArrowEngineData::new(source_data), write_context.as_ref()) .await?; tgt_txn.add_files(add_meta); @@ -304,8 +300,8 @@ async fn run_ctas_test( ) .await?; - // 6. Verify data integrity: scan target and check row count, row_number - // values, and nested address.street values all match the source + // 6. Verify data integrity: scan target and check row count, row_number values, and nested + // address.street values all match the source let tgt_snapshot_for_scan = Snapshot::builder_for(tgt_url.clone()).build(engine.as_ref())?; let tgt_scan = tgt_snapshot_for_scan.scan_builder().build()?; let tgt_batches = test_utils::read_scan(&tgt_scan, engine.clone())?; diff --git a/kernel/tests/integration/create_table/iceberg_compat.rs b/kernel/tests/integration/create_table/iceberg_compat.rs new file mode 100644 index 0000000000..761dd36a90 --- /dev/null +++ b/kernel/tests/integration/create_table/iceberg_compat.rs @@ -0,0 +1,172 @@ +//! IcebergCompatV3 integration tests for the CreateTable API. + +use std::sync::Arc; + +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::schema::{ + schema_ref, ArrayType, ColumnMetadataKey, DataType, MapType, StructField, StructType, +}; +use delta_kernel::snapshot::Snapshot; +use delta_kernel::table_features::{ColumnMappingMode, TableFeature}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::DeltaResult; +use rstest::rstest; +use test_utils::test_table_setup; + +/// V3 create-table negative paths: enabling V3 alongside an incompatible property must fail at +/// `.build(...)` with a clear error. +/// +/// `cm_mode_none` and `row_tracking_disabled` are blocked by V3's dependency check +/// (`maybe_enable_iceberg_compat_v3_dependencies`); the others are blocked earlier because +/// the property is not in `ALLOWED_DELTA_PROPERTIES` for CREATE TABLE. Keep them here +/// so that in the future when we support these properties for create table, we will remember +/// to update this test. +#[rstest] +#[case::cm_mode_none( + &[("delta.columnMapping.mode", "none")], + "to be 'name' or 'id', got 'none'", +)] +#[case::row_tracking_disabled( + &[("delta.enableRowTracking", "false")], + "to be 'true', got 'false'", +)] +#[case::iceberg_compat_v1_active( + &[("delta.enableIcebergCompatV1", "true")], + "Setting delta property 'delta.enableIcebergCompatV1' is not supported", +)] +#[case::iceberg_compat_v2_active( + &[("delta.enableIcebergCompatV2", "true")], + "Setting delta property 'delta.enableIcebergCompatV2' is not supported", +)] +fn v3_create_table_rejects_incompatible_props( + #[case] extra_props: &[(&str, &str)], + #[case] err_substring: &str, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let mut props: Vec<(&str, &str)> = vec![("delta.enableIcebergCompatV3", "true")]; + props.extend_from_slice(extra_props); + + let err = create_table(&table_path, super::simple_schema()?, "Test/1.0") + .with_table_properties(props) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new())) + .unwrap_err() + .to_string(); + assert!( + err.contains(err_substring), + "expected error containing {err_substring:?}, got: {err}", + ); + Ok(()) +} + +/// Void columns are legal in table metadata, but icebergCompatV3 omits void from its type +/// allowlist (delta-spark cannot consume a void column), so enabling V3 alongside a void column +/// must fail at `.build(...)` regardless of where the void sits. `create_table` itself does not +/// reject void placements, so the V3 allowlist (#2587) is the sole rejection point here. +#[rstest] +#[case::top_level(StructField::nullable("maybe", DataType::VOID))] +#[case::in_struct(StructField::nullable( + "s", + StructType::new_unchecked([StructField::nullable("x", DataType::VOID)]), +))] +#[case::in_array(StructField::nullable("arr", ArrayType::new(DataType::VOID, true),))] +#[case::in_map_value(StructField::nullable( + "m", + MapType::new(DataType::STRING, DataType::VOID, true), +))] +fn v3_create_table_rejects_void_column(#[case] void_field: StructField) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::LONG), + void_field, + ])?); + + let err = create_table(&table_path, schema, "Test/1.0") + .with_table_properties([("delta.enableIcebergCompatV3", "true")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new())) + .unwrap_err() + .to_string(); + assert!( + err.contains("does not support type at column") && err.contains("(void)"), + "expected V3 allowlist rejection of the void column, got: {err}", + ); + Ok(()) +} + +/// IcebergV3 has no interval type, so icebergCompatV3 omits intervals from +/// its type allowlist. Enabling V3 alongside an interval column must fail at `.build(...)`. +#[rstest] +fn v3_create_table_rejects_interval_column( + #[values(DataType::INTERVAL_YEAR_MONTH, DataType::INTERVAL_DAY_TIME)] interval: DataType, + #[values(false, true)] nested: bool, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let interval_field = if nested { + StructField::nullable( + "nested", + StructType::new_unchecked([StructField::nullable("iv", interval)]), + ) + } else { + StructField::nullable("iv", interval) + }; + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::LONG), + interval_field, + ])?); + + let err = create_table(&table_path, schema, "Test/1.0") + .with_table_properties([("delta.enableIcebergCompatV3", "true")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new())) + .unwrap_err() + .to_string(); + assert!( + err.contains("does not support type at column"), + "expected V3 allowlist rejection of the interval column, got: {err}", + ); + Ok(()) +} + +/// Listing IcebergCompatV3 in writerFeatures (i.e. "supported") without setting +/// `delta.enableIcebergCompatV3=true` must not activate V3: column mapping stays off and +/// no nested-id metadata is set on the Map field. +#[test] +fn v3_supported_but_not_enabled_skips_cm_and_nested_ids() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let schema = schema_ref! { + nullable "data": { INTEGER => nullable [nullable INTEGER] }, + }; + + let _ = create_table(&table_path, schema, "Test/1.0") + .with_table_properties([("delta.feature.icebergCompatV3", "supported")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + + // 1. V3 is in writerFeatures (supported). + let writer_features = snapshot + .table_configuration() + .protocol() + .writer_features() + .expect("writerFeatures present"); + assert!( + writer_features.contains(&TableFeature::IcebergCompatV3), + "expected icebergCompatV3 in writerFeatures, got: {writer_features:?}", + ); + + // 2. CM is not enabled (no auto-enable from V3 being merely supported). + assert_eq!( + snapshot.table_configuration().column_mapping_mode(), + ColumnMappingMode::None, + ); + + // 3. No nested-id metadata set on the Map field. + let loaded_schema = snapshot.schema(); + let data_field = loaded_schema.field("data").expect("data field present"); + assert!( + !data_field + .metadata + .contains_key(ColumnMetadataKey::ColumnMappingNestedIds.as_ref()), + "unexpected delta.columnMapping.nested.ids on data: {:?}", + data_field.metadata, + ); + Ok(()) +} diff --git a/kernel/tests/create_table/ict.rs b/kernel/tests/integration/create_table/ict.rs similarity index 100% rename from kernel/tests/create_table/ict.rs rename to kernel/tests/integration/create_table/ict.rs diff --git a/kernel/tests/integration/create_table/interval.rs b/kernel/tests/integration/create_table/interval.rs new file mode 100644 index 0000000000..0439941bc0 --- /dev/null +++ b/kernel/tests/integration/create_table/interval.rs @@ -0,0 +1,114 @@ +//! Interval-type integration tests for the CreateTable API. + +use std::sync::Arc; + +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::expressions::column_name; +use delta_kernel::schema::{DataType, StructField, StructType}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::transaction::data_layout::DataLayout; +use delta_kernel::DeltaResult; +use test_utils::test_table_setup; + +#[rstest::rstest] +fn test_create_table_rejects_interval_clustering( + #[values(DataType::INTERVAL_YEAR_MONTH, DataType::INTERVAL_DAY_TIME)] interval: DataType, + #[values(false, true)] nested: bool, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let (interval_field, clustering_column) = if nested { + ( + StructField::nullable( + "nested", + StructType::new_unchecked([StructField::nullable("iv", interval)]), + ), + column_name!("nested.iv"), + ) + } else { + (StructField::nullable("iv", interval), column_name!("iv")) + }; + let schema = Arc::new(StructType::try_new(vec![ + StructField::not_null("id", DataType::INTEGER), + interval_field, + ])?); + + let result = create_table(&table_path, schema, "Test/1.0") + .with_data_layout(DataLayout::Clustered { + columns: vec![clustering_column], + }) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new())); + test_utils::assert_result_error_with_message(result, "unsupported type"); + Ok(()) +} + +mod supported { + use delta_kernel::schema::SchemaRef; + use delta_kernel::snapshot::Snapshot; + use delta_kernel::table_features::ColumnMappingMode; + use rstest::rstest; + use test_utils::cm_properties; + + use super::super::column_mapping::{ + assert_column_mapping_config, strip_column_mapping_metadata, + }; + use super::*; + + /// Top-level schema carrying the given interval `DataType`. + fn top_level_interval_schema(interval: DataType) -> SchemaRef { + Arc::new(StructType::new_unchecked([ + StructField::new("id", DataType::INTEGER, false), + StructField::new("iv", interval, true), + ])) + } + + /// Schema with the given interval `DataType` nested inside a struct. + fn nested_interval_schema(interval: DataType) -> SchemaRef { + Arc::new(StructType::new_unchecked([ + StructField::new("id", DataType::INTEGER, false), + StructField::new( + "nested", + StructType::new_unchecked([StructField::new("inner_iv", interval, true)]), + true, + ), + ])) + } + + /// Creating a table with interval columns preserves its schema across column mapping modes. + #[rstest] + fn test_create_table_with_interval_round_trips_schema( + #[values(DataType::INTERVAL_YEAR_MONTH, DataType::INTERVAL_DAY_TIME)] interval: DataType, + #[values(top_level_interval_schema, nested_interval_schema)] make_schema: fn( + DataType, + ) + -> SchemaRef, + #[values("none", "name", "id")] cm_mode: &str, + ) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let schema = make_schema(interval); + + let _ = create_table(&table_path, schema.clone(), "Test/1.0") + .with_table_properties(cm_properties(cm_mode)) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let table_url = delta_kernel::try_parse_uri(&table_path)?; + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + + let expected_cm_mode = match cm_mode { + "none" => ColumnMappingMode::None, + "name" => ColumnMappingMode::Name, + "id" => ColumnMappingMode::Id, + _ => unreachable!(), + }; + assert_column_mapping_config(&snapshot, expected_cm_mode); + + let read_schema = snapshot.schema(); + let stripped_schema = strip_column_mapping_metadata(&read_schema); + assert_eq!( + &stripped_schema, + schema.as_ref(), + "logical schema should round-trip through create table" + ); + Ok(()) + } +} diff --git a/kernel/tests/integration/create_table/mod.rs b/kernel/tests/integration/create_table/mod.rs new file mode 100644 index 0000000000..28f27fe79b --- /dev/null +++ b/kernel/tests/integration/create_table/mod.rs @@ -0,0 +1,719 @@ +//! Integration tests for the CreateTable API + +mod clustering; +mod column_mapping; +mod ctas; +mod iceberg_compat; +mod ict; +mod interval; +mod partitioned; +mod row_tracking; +mod timestamp_ntz; +mod variant; + +use std::sync::Arc; + +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::schema::{ColumnMetadataKey, DataType, MetadataValue, StructField, StructType}; +use delta_kernel::snapshot::Snapshot; +use delta_kernel::table_features::{ + TableFeature, TABLE_FEATURES_MIN_READER_VERSION, TABLE_FEATURES_MIN_WRITER_VERSION, +}; +use delta_kernel::table_properties::TableProperties; +use delta_kernel::transaction::create_table::{create_table, CreateTableTransaction}; +use delta_kernel::transaction::data_layout::DataLayout; +use delta_kernel::DeltaResult; +use rstest::rstest; +use serde_json::Value; +use test_utils::{assert_result_error_with_message, test_table_setup, test_table_setup_mt}; + +/// Helper to create a simple two-column schema for tests. +/// Shared with sub-modules. +pub(crate) fn simple_schema() -> DeltaResult> { + Ok(Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("value", DataType::STRING), + ])?)) +} + +/// Helper to create a three-column schema for partition tests (id, date, value). +/// Shared with sub-modules. +pub(crate) fn partition_test_schema() -> DeltaResult> { + Ok(Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("date", DataType::DATE), + StructField::nullable("value", DataType::STRING), + ])?)) +} + +#[tokio::test] +async fn test_create_simple_table() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // Create schema for an events table + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("event_id", DataType::LONG), + StructField::nullable("user_id", DataType::LONG), + StructField::nullable("event_type", DataType::STRING), + StructField::nullable("timestamp", DataType::TIMESTAMP), + StructField::nullable("properties", DataType::STRING), + ])?); + + // Create table using new API + let _ = create_table(&table_path, schema.clone(), "DeltaKernel-RS/0.17.0") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + // Verify table was created + let table_url = delta_kernel::try_parse_uri(&table_path)?; + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + + assert_eq!(snapshot.version(), 0); + assert_eq!(snapshot.schema().fields().len(), 5); + + // Verify protocol versions via snapshot + let protocol = snapshot.table_configuration().protocol(); + assert_eq!( + protocol.min_reader_version(), + TABLE_FEATURES_MIN_READER_VERSION + ); + assert_eq!( + protocol.min_writer_version(), + TABLE_FEATURES_MIN_WRITER_VERSION + ); + // Verify no reader/writer features are set (empty for table features mode) + assert!(protocol.reader_features().is_some_and(|f| f.is_empty())); + assert!(protocol.writer_features().is_some_and(|f| f.is_empty())); + + // Verify no table properties are set via public API + assert_eq!(snapshot.table_properties(), &TableProperties::default()); + + // Verify schema field names + let field_names: Vec<_> = snapshot + .schema() + .fields() + .map(|f| f.name().to_string()) + .collect(); + assert!(field_names.contains(&"event_id".to_string())); + assert!(field_names.contains(&"user_id".to_string())); + assert!(field_names.contains(&"event_type".to_string())); + assert!(field_names.contains(&"timestamp".to_string())); + assert!(field_names.contains(&"properties".to_string())); + + Ok(()) +} + +#[tokio::test] +async fn test_create_table_with_user_domain_metadata() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let schema = simple_schema()?; + + // Create table with domainMetadata feature enabled + let txn = create_table(&table_path, schema, "Test/1.0") + .with_table_properties([("delta.feature.domainMetadata", "supported")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))?; + + // Add user domain metadata during table creation + let domain = "app.settings"; + let config = r#"{"version": 1, "enabled": true}"#; + + let _ = txn + .with_domain_metadata(domain.to_string(), config.to_string()) + .commit(engine.as_ref())?; + + // Load snapshot and verify domain metadata was persisted + let table_url = delta_kernel::try_parse_uri(&table_path)?; + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + + // Verify domainMetadata feature is enabled in protocol + assert!( + snapshot + .table_configuration() + .is_feature_supported(&TableFeature::DomainMetadata), + "DomainMetadata feature should be enabled" + ); + + // Verify domain metadata string was persisted correctly + let retrieved_config = snapshot.get_domain_metadata(domain, engine.as_ref())?; + assert_eq!( + retrieved_config, + Some(config.to_string()), + "Domain metadata should be persisted and retrievable" + ); + + // Parse and verify the JSON contents + let parsed: Value = serde_json::from_str(retrieved_config.as_ref().unwrap())?; + assert_eq!(parsed["version"], 1); + assert_eq!(parsed["enabled"], true); + + // Verify non-existent domain returns None + let missing = snapshot.get_domain_metadata("nonexistent.domain", engine.as_ref())?; + assert!(missing.is_none(), "Non-existent domain should return None"); + + Ok(()) +} + +#[tokio::test] +async fn test_create_table_already_exists() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // Create schema for a user profiles table + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("user_id", DataType::LONG), + StructField::nullable("username", DataType::STRING), + StructField::nullable("email", DataType::STRING), + StructField::nullable("created_at", DataType::TIMESTAMP), + StructField::nullable("is_active", DataType::BOOLEAN), + ])?); + + // Create table first time + let _ = create_table(&table_path, schema.clone(), "UserManagementService/1.2.0") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + // Try to create again - should fail at build time (table already exists) + let result = create_table(&table_path, schema.clone(), "UserManagementService/1.2.0") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new())); + + assert_result_error_with_message(result, "already exists"); + + Ok(()) +} + +#[tokio::test] +async fn test_create_table_empty_schema_succeeds() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // CREATE TABLE with no columns is a valid Delta operation; users may add columns + // later via ALTER TABLE ADD COLUMN. + let schema = Arc::new(StructType::try_new(vec![])?); + + create_table(&table_path, schema, "EmptySchemaApp/0.1.0") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + + let table_url = delta_kernel::try_parse_uri(&table_path)?; + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + assert_eq!(snapshot.version(), 0); + assert_eq!(snapshot.schema().num_fields(), 0); + + Ok(()) +} + +/// Checkpoints serialize the table schema into their parquet payload, so a zero-column +/// schema is a corner the writer can plausibly mishandle. This test creates an empty +/// table, checkpoints immediately, then reloads from scratch to confirm the schema +/// survives the round-trip across all column-mapping modes. +#[rstest] +#[case::no_cm(None)] +#[case::cm_name(Some("name"))] +#[case::cm_id(Some("id"))] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_create_table_empty_schema_checkpoint_round_trip( + #[case] cm_mode: Option<&str>, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + + let schema = Arc::new(StructType::try_new(vec![])?); + let mut builder = create_table(&table_path, schema, "EmptySchemaApp/0.1.0"); + if let Some(mode) = cm_mode { + builder = builder.with_table_properties([("delta.columnMapping.mode", mode)]); + } + builder + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + + let table_url = delta_kernel::try_parse_uri(&table_path)?; + let v0 = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let (_, _ckpt_snapshot) = v0.checkpoint(engine.as_ref(), None)?; + + let reloaded = Snapshot::builder_for(table_url).build(engine.as_ref())?; + assert_eq!(reloaded.version(), 0); + assert_eq!(reloaded.schema().num_fields(), 0); + let max_id = reloaded + .table_configuration() + .metadata() + .configuration() + .get("delta.columnMapping.maxColumnId") + .and_then(|v| v.parse::().ok()); + assert_eq!(max_id, cm_mode.map(|_| 0)); + + Ok(()) +} + +#[rstest] +#[case::partitioned_empty(DataLayout::partitioned::<_, &str>([]), "at least one column")] +#[case::clustered_missing_column(DataLayout::clustered(["foo"]), "not found in schema")] +#[tokio::test] +async fn test_create_table_empty_schema_layout_errors( + #[case] layout: DataLayout, + #[case] expected_err: &str, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let schema = Arc::new(StructType::try_new(vec![])?); + let result = create_table(&table_path, schema, "EmptySchemaApp/0.1.0") + .with_data_layout(layout) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new())); + + assert_result_error_with_message(result, expected_err); + + Ok(()) +} + +fn top_level_non_null_schema() -> Arc { + Arc::new( + StructType::try_new(vec![ + StructField::not_null("id", DataType::INTEGER), + StructField::nullable("value", DataType::STRING), + ]) + .expect("non-null top-level schema should be valid"), + ) +} + +fn nested_non_null_schema() -> Arc { + let nested = StructType::try_new(vec![StructField::not_null("child", DataType::INTEGER)]) + .expect("nested non-null schema should be valid"); + Arc::new( + StructType::try_new(vec![StructField::nullable("nested", nested)]) + .expect("top-level nested schema should be valid"), + ) +} + +/// CREATE TABLE with non-null columns succeeds and auto-enables the `invariants` +/// writer feature in the protocol. Delta-Spark treats non-null schema columns as +/// implicit invariants and requires the feature to be declared. +#[rstest] +#[case::top_level_non_null(top_level_non_null_schema())] +#[case::nested_non_null(nested_non_null_schema())] +fn test_create_table_with_non_null_columns_auto_enables_invariants( + #[case] schema: Arc, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let _ = create_table(&table_path, schema, "Test/1.0") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let protocol = snapshot.table_configuration().protocol(); + assert!( + protocol + .writer_features() + .is_some_and(|f| f.contains(&TableFeature::Invariants)), + "Invariants feature should be auto-added to writer features for non-null schema" + ); + // Invariants is writer-only; reader features should not contain it. + assert!( + !protocol + .reader_features() + .is_some_and(|f| f.contains(&TableFeature::Invariants)), + "Invariants should not appear in reader features" + ); + + Ok(()) +} + +/// CREATE TABLE with `delta.feature.invariants=supported` + a non-null schema succeeds +/// and lands the feature in writerFeatures exactly once. Non-null columns auto-enable +/// the feature and the explicit signal also tries to enable it; both paths firing +/// together must not duplicate the entry. This also verifies users can pre-enable the +/// feature so a later ALTER TABLE ADD COLUMN NOT NULL does not need a protocol upgrade. +#[tokio::test] +async fn test_create_table_with_invariants_feature_signal_allowed() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + // Non-null schema auto-enables Invariants; the feature signal also tries to add it. + // Both code paths firing together must produce exactly one Invariants entry. + let schema = top_level_non_null_schema(); + + let _ = create_table(&table_path, schema, "Test/1.0") + .with_table_properties([("delta.feature.invariants", "supported")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let protocol = snapshot.table_configuration().protocol(); + let writer_features = protocol + .writer_features() + .expect("writer features should be present"); + let invariants_count = writer_features + .iter() + .filter(|f| **f == TableFeature::Invariants) + .count(); + assert_eq!( + invariants_count, 1, + "Invariants should appear exactly once in writerFeatures; got {invariants_count}" + ); + assert!( + !protocol + .reader_features() + .is_some_and(|f| f.contains(&TableFeature::Invariants)), + "Invariants should not appear in readerFeatures (writer-only feature)" + ); + + Ok(()) +} + +/// CREATE TABLE rejects any schema with `delta.invariants` metadata annotations +/// because kernel cannot evaluate SQL expression invariants. +#[tokio::test] +async fn test_create_table_rejects_delta_invariants_metadata() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let mut field = StructField::nullable("x", DataType::INTEGER); + field.metadata.insert( + ColumnMetadataKey::Invariants.as_ref().to_string(), + MetadataValue::String(r#"{"expression": {"expression": "x > 0"}}"#.to_string()), + ); + let schema = Arc::new(StructType::try_new(vec![field])?); + + let result = create_table(&table_path, schema, "Test/1.0") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new())); + + assert_result_error_with_message(result, "delta.invariants"); + + Ok(()) +} + +#[tokio::test] +async fn test_create_table_log_actions() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // Create schema + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("user_id", DataType::LONG), + StructField::nullable("action", DataType::STRING), + ])?); + + let engine_info = "AuditService/2.1.0"; + + // Create table + let _ = create_table(&table_path, schema, engine_info) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + // Read the actual Delta log file + let log_file_path = format!("{table_path}/_delta_log/00000000000000000000.json"); + let log_contents = std::fs::read_to_string(&log_file_path).expect("Failed to read log file"); + + // Parse each line (each line is a separate JSON action) + let actions: Vec = log_contents + .lines() + .map(|line| serde_json::from_str(line).expect("Failed to parse JSON")) + .collect(); + + // Verify we have exactly 3 actions: CommitInfo, Protocol, Metadata + // CommitInfo is first to comply with ICT (In-Commit Timestamps) protocol requirements + assert_eq!( + actions.len(), + 3, + "Expected 3 actions (commitInfo, protocol, metaData), found {}", + actions.len() + ); + + // Verify CommitInfo action (first for ICT compliance) + let commit_info_action = &actions[0]; + assert!( + commit_info_action.get("commitInfo").is_some(), + "First action should be commitInfo" + ); + let commit_info = commit_info_action.get("commitInfo").unwrap(); + assert!( + commit_info.get("timestamp").is_some(), + "CommitInfo should have timestamp" + ); + assert!( + commit_info.get("engineInfo").is_some(), + "CommitInfo should have engineInfo" + ); + assert!( + commit_info.get("operation").is_some(), + "CommitInfo should have operation" + ); + assert_eq!( + commit_info["operation"], "CREATE TABLE", + "Operation should be CREATE TABLE" + ); + + // Verify Protocol action + let protocol_action = &actions[1]; + assert!( + protocol_action.get("protocol").is_some(), + "Second action should be protocol" + ); + let protocol = protocol_action.get("protocol").unwrap(); + assert_eq!( + protocol["minReaderVersion"], + TABLE_FEATURES_MIN_READER_VERSION + ); + assert_eq!( + protocol["minWriterVersion"], + TABLE_FEATURES_MIN_WRITER_VERSION + ); + + // Verify Metadata action + let metadata_action = &actions[2]; + assert!( + metadata_action.get("metaData").is_some(), + "Third action should be metaData" + ); + let metadata = metadata_action.get("metaData").unwrap(); + assert!(metadata.get("id").is_some(), "Metadata should have id"); + assert!( + metadata.get("schemaString").is_some(), + "Metadata should have schemaString" + ); + assert!( + metadata.get("createdTime").is_some(), + "Metadata should have createdTime" + ); + + // Additional CommitInfo verification (commit_info was already extracted from actions[0] above) + assert_eq!( + commit_info["engineInfo"], engine_info, + "CommitInfo should contain the engine info we provided" + ); + + assert!( + commit_info.get("txnId").is_some(), + "CommitInfo should have txnId" + ); + + // Verify kernelVersion is present + let kernel_version = commit_info.get("kernelVersion"); + assert!( + kernel_version.is_some(), + "CommitInfo should have kernelVersion" + ); + assert!( + kernel_version.unwrap().as_str().unwrap().starts_with("v"), + "Kernel version should start with 'v'" + ); + + Ok(()) +} + +/// Helper to create a `CreateTableTransaction` for tests. +fn create_test_create_table_txn() -> DeltaResult<( + Arc, + CreateTableTransaction, + tempfile::TempDir, +)> { + let (tempdir, table_path, engine) = test_table_setup()?; + let schema = Arc::new( + StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("name", DataType::STRING), + ]) + .expect("valid schema"), + ); + let txn = create_table(&table_path, schema, "test_engine") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))?; + Ok((engine, txn, tempdir)) +} + +#[tokio::test] +async fn test_create_table_txn_debug() -> DeltaResult<()> { + let (_engine, txn, _tempdir) = create_test_create_table_txn()?; + let debug_str = format!("{txn:?}"); + assert!( + debug_str.contains("Transaction") && debug_str.contains("create_table"), + "Debug output should contain Transaction info: {debug_str}" + ); + Ok(()) +} + +#[rstest] +// ReaderWriter features (AlwaysIfSupported) +#[case("vacuumProtocolCheck", TableFeature::VacuumProtocolCheck, true, true)] +#[case("v2Checkpoint", TableFeature::V2Checkpoint, true, true)] +// ReaderWriter features (EnabledIf -- feature signal alone does not enable) +#[case("deletionVectors", TableFeature::DeletionVectors, true, false)] +#[case("typeWidening", TableFeature::TypeWidening, true, false)] +// WriterOnly features (EnabledIf -- feature signal alone does not enable) +#[case("appendOnly", TableFeature::AppendOnly, false, false)] +#[case("changeDataFeed", TableFeature::ChangeDataFeed, false, false)] +#[case("rowTracking", TableFeature::RowTracking, false, false)] +// WriterOnly features (AlwaysIfSupported) +#[case( + "materializePartitionColumns", + TableFeature::MaterializePartitionColumns, + false, + true +)] +fn test_create_table_with_feature_signal( + #[case] feature_name: &str, + #[case] feature: TableFeature, + #[case] is_reader_writer: bool, + #[case] enabled_when_supported: bool, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let property_key = format!("delta.feature.{feature_name}"); + let _ = create_table(&table_path, simple_schema()?, "Test/1.0") + .with_table_properties([(property_key.as_str(), "supported")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let table_config = snapshot.table_configuration(); + + assert!( + table_config.is_feature_supported(&feature), + "{feature_name} should be supported" + ); + assert_eq!( + table_config.is_feature_enabled(&feature), + enabled_when_supported, + "{feature_name}: is_feature_enabled should be {enabled_when_supported}" + ); + let protocol = table_config.protocol(); + assert!( + protocol + .writer_features() + .is_some_and(|f| f.contains(&feature)), + "{feature_name} should be in writer features" + ); + if is_reader_writer { + assert!( + protocol + .reader_features() + .is_some_and(|f| f.contains(&feature)), + "{feature_name} should be in reader features" + ); + } + + Ok(()) +} + +#[rstest] +fn test_create_table_with_checkpoint_stats_properties( + #[values(true, false)] write_stats_as_json: bool, + #[values(true, false)] write_stats_as_struct: bool, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let json_val = write_stats_as_json.to_string(); + let struct_val = write_stats_as_struct.to_string(); + + let _ = create_table(&table_path, simple_schema()?, "Test/1.0") + .with_table_properties([ + ("delta.checkpoint.writeStatsAsJson", json_val.as_str()), + ("delta.checkpoint.writeStatsAsStruct", struct_val.as_str()), + ]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let tp = snapshot.table_properties(); + assert_eq!(tp.checkpoint_write_stats_as_json, Some(write_stats_as_json)); + assert_eq!( + tp.checkpoint_write_stats_as_struct, + Some(write_stats_as_struct) + ); + + Ok(()) +} + +#[rstest] +// ReaderWriter features +#[case("delta.enableDeletionVectors", TableFeature::DeletionVectors, true)] +#[case("delta.enableTypeWidening", TableFeature::TypeWidening, true)] +// WriterOnly features +#[case("delta.enableChangeDataFeed", TableFeature::ChangeDataFeed, false)] +#[case("delta.appendOnly", TableFeature::AppendOnly, false)] +#[case("delta.enableRowTracking", TableFeature::RowTracking, false)] +fn test_create_table_with_enablement_property( + #[case] property: &str, + #[case] feature: TableFeature, + #[case] is_reader_writer: bool, + #[values(true, false)] expect_enabled: bool, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let value = expect_enabled.to_string(); + + let _ = create_table(&table_path, simple_schema()?, "Test/1.0") + .with_table_properties([(property, value.as_str())]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let table_config = snapshot.table_configuration(); + + assert_eq!( + table_config.is_feature_supported(&feature), + expect_enabled, + "{property}={value}: feature supported should be {expect_enabled}" + ); + assert_eq!( + table_config.is_feature_enabled(&feature), + expect_enabled, + "{property}={value}: feature enabled should be {expect_enabled}" + ); + let protocol = table_config.protocol(); + assert_eq!( + protocol + .writer_features() + .is_some_and(|f| f.contains(&feature)), + expect_enabled, + "{property}={value}: in writer features should be {expect_enabled}" + ); + if is_reader_writer { + assert_eq!( + protocol + .reader_features() + .is_some_and(|f| f.contains(&feature)), + expect_enabled, + "{property}={value}: in reader features should be {expect_enabled}" + ); + } + + Ok(()) +} + +#[rstest] +#[case::without_cm(false)] +#[case::with_cm(true)] +fn test_create_table_special_char_column_name(#[case] cm_enabled: bool) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("valid_col", DataType::INTEGER), + StructField::nullable("bad column", DataType::STRING), + ])?); + + let mut builder = create_table(&table_path, schema, "Test/1.0"); + if cm_enabled { + builder = builder.with_table_properties([("delta.columnMapping.mode", "name")]); + } + let result = builder.build(engine.as_ref(), Box::new(FileSystemCommitter::new())); + + if cm_enabled { + let txn = result?; + let _ = txn.commit(engine.as_ref())?; + + let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!(snapshot.version(), 0); + let field_names: Vec<_> = snapshot + .schema() + .fields() + .map(|f| f.name().clone()) + .collect(); + assert!( + field_names.contains(&"bad column".to_string()), + "Schema should contain field 'bad column', got: {field_names:?}" + ); + } else { + assert!(result.is_err()); + let err = result.unwrap_err().to_string(); + assert!( + err.contains("invalid character"), + "Expected invalid character error, got: {err}" + ); + } + + Ok(()) +} diff --git a/kernel/tests/integration/create_table/partitioned.rs b/kernel/tests/integration/create_table/partitioned.rs new file mode 100644 index 0000000000..15551961c0 --- /dev/null +++ b/kernel/tests/integration/create_table/partitioned.rs @@ -0,0 +1,79 @@ +//! Partition integration tests for the CreateTable API. +//! +//! TODO(#2201): Add end-to-end tests for insert + scan + checkpoint on partitioned tables. + +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::snapshot::Snapshot; +use delta_kernel::table_features::TableFeature; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::transaction::data_layout::DataLayout; +use delta_kernel::DeltaResult; +use rstest::rstest; +use test_utils::test_table_setup; + +use super::{partition_test_schema, simple_schema}; + +#[rstest] +#[case::exact_casing("date")] +#[case::mismatched_casing("DATE")] +fn test_create_table_partitioned_basic(#[case] partition_col: &str) -> DeltaResult<()> { + let schema = partition_test_schema()?; + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let _ = create_table(&table_path, schema, "Test/1.0") + .with_data_layout(DataLayout::partitioned([partition_col])) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!(snapshot.version(), 0); + + // Partition column should be stored matching the casing in the schema + let partition_cols = snapshot.table_configuration().logical_partition_columns(); + assert_eq!(partition_cols, &["date"]); + + let clustering = snapshot.get_physical_clustering_columns(engine.as_ref())?; + assert!( + clustering.is_none(), + "Partitioned table should not have clustering columns" + ); + + Ok(()) +} + +/// CREATE TABLE accepts `delta.feature.materializePartitionColumns=supported` regardless of +/// whether the table is partitioned. The feature is harmless on a non-partitioned table +/// (nothing to materialize) and Delta-Spark also does not error in that case, so kernel +/// matches that behavior. +#[rstest] +#[case::partitioned(true)] +#[case::non_partitioned(false)] +fn test_create_table_with_materialize_partition_columns_partitioned_and_not( + #[case] partitioned: bool, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let schema = if partitioned { + partition_test_schema()? + } else { + simple_schema()? + }; + + let mut builder = create_table(&table_path, schema, "Test/1.0") + .with_table_properties([("delta.feature.materializePartitionColumns", "supported")]); + if partitioned { + builder = builder.with_data_layout(DataLayout::partitioned(["date"])); + } + let _ = builder + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert!( + snapshot + .table_configuration() + .is_feature_enabled(&TableFeature::MaterializePartitionColumns), + "MaterializePartitionColumns should be enabled (partitioned={partitioned})" + ); + + Ok(()) +} diff --git a/kernel/tests/integration/create_table/row_tracking.rs b/kernel/tests/integration/create_table/row_tracking.rs new file mode 100644 index 0000000000..3dd99cff8f --- /dev/null +++ b/kernel/tests/integration/create_table/row_tracking.rs @@ -0,0 +1,457 @@ +//! Row tracking integration tests for the CreateTable API. +//! +//! Tests that creating a table with `delta.enableRowTracking=true` automatically adds the +//! `rowTracking` and `domainMetadata` features to the protocol and writes the initial row +//! tracking domain metadata with `rowIdHighWaterMark = -1`. + +use std::sync::Arc; + +use delta_kernel::arrow::array::{Int32Array, StringArray}; +use delta_kernel::arrow::record_batch::RecordBatch; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::engine::arrow_conversion::TryIntoArrow as _; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::row_tracking::RowTrackingDomainMetadata; +use delta_kernel::snapshot::Snapshot; +use delta_kernel::table_features::{ + TableFeature, TABLE_FEATURES_MIN_READER_VERSION, TABLE_FEATURES_MIN_WRITER_VERSION, +}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::transaction::data_layout::DataLayout; +use delta_kernel::DeltaResult; +use rstest::rstest; +use test_utils::{ + get_materialized_row_tracking_column_names, get_row_tracking_add_actions, insert_data, + read_actions_from_commit, test_table_setup, +}; +use url::Url; + +/// Asserts protocol features are correct for row tracking on a snapshot. +fn assert_row_tracking_protocol(snapshot: &Snapshot) { + let protocol = snapshot.table_configuration().protocol(); + + assert!(protocol.min_reader_version() >= TABLE_FEATURES_MIN_READER_VERSION); + assert!(protocol.min_writer_version() >= TABLE_FEATURES_MIN_WRITER_VERSION); + + // RowTracking is writer-only + assert!( + protocol + .writer_features() + .is_some_and(|f| f.contains(&TableFeature::RowTracking)), + "rowTracking should be in writer features" + ); + assert!( + !protocol + .reader_features() + .is_some_and(|f| f.contains(&TableFeature::RowTracking)), + "rowTracking should NOT be in reader features" + ); + + // DomainMetadata dependency is writer-only + assert!( + protocol + .writer_features() + .is_some_and(|f| f.contains(&TableFeature::DomainMetadata)), + "domainMetadata should be in writer features" + ); +} + +/// Verifies row tracking across both activation paths (enablement property and feature signal) +/// and both table shapes (empty CREATE TABLE and CTAS with a single file of 5 rows). +/// +/// Key behavioral differences by case: +/// - Activation path: `delta.enableRowTracking=true` sets materialized column name properties; +/// feature-signal-only does not. +/// - Table shape: empty tables get `rowIdHighWaterMark = -1` with no add actions; CTAS assigns +/// `baseRowId = 0` and `defaultRowCommitVersion = 0` and sets `rowIdHighWaterMark = 4`. +#[rstest] +#[tokio::test] +async fn test_create_table_with_row_tracking( + #[values( + ("delta.enableRowTracking", "true"), + ("delta.feature.rowTracking", "supported") + )] + activation: (&str, &str), + #[values(false, true)] with_data: bool, +) -> DeltaResult<()> { + let (key, value) = activation; + let expect_property_enabled = key == "delta.enableRowTracking"; + + let (_temp_dir, table_path, engine) = test_table_setup()?; + let schema = super::simple_schema()?; + + let mut txn = create_table(&table_path, schema.clone(), "Test/1.0") + .with_table_properties([(key, value)]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))?; + + if with_data { + // Write one parquet file with 5 rows + let arrow_schema = Arc::new(schema.as_ref().try_into_arrow()?); + let batch = RecordBatch::try_new( + arrow_schema, + vec![ + Arc::new(Int32Array::from(vec![1, 2, 3, 4, 5])), + Arc::new(StringArray::from(vec!["a", "b", "c", "d", "e"])), + ], + ) + .map_err(|e| delta_kernel::Error::generic(e.to_string()))?; + + let write_context = Arc::new(txn.unpartitioned_write_context()?); + let add_files = engine + .write_parquet(&ArrowEngineData::new(batch), write_context.as_ref()) + .await?; + txn.add_files(add_files); + } + + let committed = txn.commit(engine.as_ref())?.unwrap_committed(); + let snapshot = committed + .post_commit_snapshot() + .expect("should have snapshot"); + assert_row_tracking_protocol(snapshot); + + // Verify domain metadata persists to disk for empty tables. CTAS is not checked here + // because the high water mark is written via add-file processing, not the initial + // domain metadata path. + if !with_data { + let disk_snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_row_tracking_protocol(&disk_snapshot); + } + + let table_url = Url::from_directory_path(&table_path).expect("valid path"); + + // Verify row tracking high water mark via snapshot API + let disk_snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let expected_high_water_mark: i64 = if with_data { 4 } else { -1 }; + assert_eq!( + RowTrackingDomainMetadata::get_high_water_mark(&disk_snapshot, engine.as_ref())?, + Some(expected_high_water_mark), + ); + + // Verify add actions + let add_actions = get_row_tracking_add_actions(&table_url, 0).expect("failed to read commit"); + if with_data { + assert_eq!(add_actions.len(), 1, "Expected one add action"); + assert_eq!(add_actions[0].base_row_id, Some(0), "baseRowId should be 0"); + assert_eq!( + add_actions[0].default_row_commit_version, + Some(0), + "defaultRowCommitVersion should be 0" + ); + } else { + assert!( + add_actions.is_empty(), + "Expected no add actions for empty create" + ); + } + + // Verify materialized column name properties. + // Only set when delta.enableRowTracking=true; feature-signal-only tables do not get them. + let col_names = + get_materialized_row_tracking_column_names(&table_url, 0).expect("failed to read commit"); + + if expect_property_enabled { + let row_id_col = col_names + .row_id_column_name + .as_deref() + .expect("materializedRowIdColumnName should be set"); + assert!(row_id_col.starts_with("_row-id-col-"), "got {row_id_col}"); + + let commit_version_col = col_names + .row_commit_version_column_name + .as_deref() + .expect("materializedRowCommitVersionColumnName should be set"); + assert!( + commit_version_col.starts_with("_row-commit-version-col-"), + "got {commit_version_col}" + ); + } else { + assert!( + col_names.row_id_column_name.is_none(), + "materializedRowIdColumnName should NOT be set for feature-signal-only tables" + ); + assert!( + col_names.row_commit_version_column_name.is_none(), + "materializedRowCommitVersionColumnName should NOT be set for feature-signal-only tables" + ); + + // Also verify the enablement property itself is absent + let metadata_actions = + read_actions_from_commit(&table_url, 0, "metaData").expect("failed to read commit"); + assert!( + metadata_actions[0] + .get("configuration") + .and_then(|c| c.get("delta.enableRowTracking")) + .is_none(), + "delta.enableRowTracking should NOT be set for feature-signal-only tables" + ); + } + + Ok(()) +} + +/// Verifies that CTAS with multiple files assigns non-overlapping baseRowId ranges and +/// computes the correct cumulative high water mark. +#[tokio::test] +async fn test_create_table_with_multiple_files_and_row_tracking() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let schema = super::simple_schema()?; + let mut txn = create_table(&table_path, schema.clone(), "Test/1.0") + .with_table_properties([("delta.enableRowTracking", "true")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))?; + + let arrow_schema: Arc = + Arc::new(schema.as_ref().try_into_arrow()?); + + // Write two separate parquet files: 3 rows and 5 rows + let batch1 = RecordBatch::try_new( + arrow_schema.clone(), + vec![ + Arc::new(Int32Array::from(vec![1, 2, 3])), + Arc::new(StringArray::from(vec!["a", "b", "c"])), + ], + ) + .map_err(|e| delta_kernel::Error::generic(e.to_string()))?; + + let batch2 = RecordBatch::try_new( + arrow_schema, + vec![ + Arc::new(Int32Array::from(vec![4, 5, 6, 7, 8])), + Arc::new(StringArray::from(vec!["d", "e", "f", "g", "h"])), + ], + ) + .map_err(|e| delta_kernel::Error::generic(e.to_string()))?; + + let write_context = Arc::new(txn.unpartitioned_write_context()?); + let adds1 = engine + .write_parquet(&ArrowEngineData::new(batch1), write_context.as_ref()) + .await?; + let adds2 = engine + .write_parquet(&ArrowEngineData::new(batch2), write_context.as_ref()) + .await?; + + txn.add_files(adds1); + txn.add_files(adds2); + + let committed = txn.commit(engine.as_ref())?.unwrap_committed(); + assert_eq!(committed.commit_version(), 0); + + let table_url = Url::from_directory_path(&table_path).expect("valid path"); + + // Verify add actions (already sorted by baseRowId) + let add_actions = get_row_tracking_add_actions(&table_url, 0).expect("failed to read commit"); + assert_eq!(add_actions.len(), 2, "Expected two add actions"); + + // First file (3 rows): baseRowId = 0 + assert_eq!(add_actions[0].base_row_id, Some(0)); + assert_eq!(add_actions[0].default_row_commit_version, Some(0)); + + // Second file (5 rows): baseRowId = 3 (first file had 3 rows) + assert_eq!(add_actions[1].base_row_id, Some(3)); + assert_eq!(add_actions[1].default_row_commit_version, Some(0)); + + // HWM should be 7 (IDs 0-2 from file 1, IDs 3-7 from file 2) + let disk_snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!( + RowTrackingDomainMetadata::get_high_water_mark(&disk_snapshot, engine.as_ref())?, + Some(7), + "HWM should be 7 for 8 total rows (3 + 5) starting from -1" + ); + + Ok(()) +} + +/// Verifies that row tracking and clustering can be enabled together. Both features require +/// DomainMetadata, which should appear exactly once in the protocol. Both domain metadata +/// entries (delta.rowTracking and delta.clustering) should be present in the commit. +#[test] +fn test_create_table_with_row_tracking_and_clustering() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let committed = create_table(&table_path, super::simple_schema()?, "Test/1.0") + .with_table_properties([("delta.enableRowTracking", "true")]) + .with_data_layout(DataLayout::clustered(["id"])) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + + let snapshot = committed + .post_commit_snapshot() + .expect("should have snapshot"); + let table_url = Url::from_directory_path(&table_path).expect("valid path"); + let protocol = snapshot.table_configuration().protocol(); + let writer_features = protocol + .writer_features() + .expect("should have writer features"); + + assert!(writer_features.contains(&TableFeature::RowTracking)); + assert!(writer_features.contains(&TableFeature::ClusteredTable)); + // DomainMetadata should appear exactly once despite two features requiring it + let dm_count = writer_features + .iter() + .filter(|f| **f == TableFeature::DomainMetadata) + .count(); + assert_eq!(dm_count, 1, "DomainMetadata should not be duplicated"); + + // Verify both domain metadata entries are present in the commit + let dm_actions = + read_actions_from_commit(&table_url, 0, "domainMetadata").expect("failed to read commit"); + assert!( + dm_actions + .iter() + .any(|dm| dm["domain"] == "delta.rowTracking"), + "row tracking domain metadata should be present" + ); + assert!( + dm_actions + .iter() + .any(|dm| dm["domain"] == "delta.clustering"), + "clustering domain metadata should be present" + ); + + Ok(()) +} + +/// Verifies that row tracking and clustering work together when the table has data (CTAS). +/// Both features generate domain metadata and the add files need row tracking columns. +/// Verifies that both domain metadata entries survive when add files are also written. +#[tokio::test] +async fn test_create_table_with_row_tracking_and_clustering_and_data() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let schema = super::simple_schema()?; + let mut txn = create_table(&table_path, schema.clone(), "Test/1.0") + .with_table_properties([("delta.enableRowTracking", "true")]) + .with_data_layout(DataLayout::clustered(["id"])) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))?; + + let arrow_schema = Arc::new(schema.as_ref().try_into_arrow()?); + let batch = RecordBatch::try_new( + arrow_schema, + vec![ + Arc::new(Int32Array::from(vec![1, 2, 3, 4, 5])), + Arc::new(StringArray::from(vec!["a", "b", "c", "d", "e"])), + ], + ) + .map_err(|e| delta_kernel::Error::generic(e.to_string()))?; + + let write_context = Arc::new(txn.unpartitioned_write_context()?); + let add_files = engine + .write_parquet(&ArrowEngineData::new(batch), write_context.as_ref()) + .await?; + txn.add_files(add_files); + + let committed = txn.commit(engine.as_ref())?.unwrap_committed(); + let snapshot = committed + .post_commit_snapshot() + .expect("should have snapshot"); + let table_url = Url::from_directory_path(&table_path).expect("valid path"); + + // Both features must be present in the protocol exactly once + let writer_features = snapshot + .table_configuration() + .protocol() + .writer_features() + .expect("should have writer features"); + assert!(writer_features.contains(&TableFeature::RowTracking)); + assert!(writer_features.contains(&TableFeature::ClusteredTable)); + assert_eq!( + writer_features + .iter() + .filter(|f| **f == TableFeature::DomainMetadata) + .count(), + 1, + "DomainMetadata should appear exactly once" + ); + + // Both domain metadata entries must survive alongside the add actions + let dm_actions = + read_actions_from_commit(&table_url, 0, "domainMetadata").expect("failed to read commit"); + assert!( + dm_actions + .iter() + .any(|dm| dm["domain"] == "delta.rowTracking"), + "row tracking domain metadata should be present" + ); + assert!( + dm_actions + .iter() + .any(|dm| dm["domain"] == "delta.clustering"), + "clustering domain metadata should be present" + ); + + // Add action must have correct row tracking fields + let add_actions = get_row_tracking_add_actions(&table_url, 0).expect("failed to read commit"); + assert_eq!(add_actions.len(), 1, "Expected one add action"); + assert_eq!(add_actions[0].base_row_id, Some(0)); + assert_eq!(add_actions[0].default_row_commit_version, Some(0)); + + // High water mark should reflect the 5 written rows + let disk_snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!( + RowTrackingDomainMetadata::get_high_water_mark(&disk_snapshot, engine.as_ref())?, + Some(4), + "5 rows -> high water mark = 4" + ); + + Ok(()) +} + +/// Verifies that a table created with the feature signal only +/// (`delta.feature.rowTracking=supported`, no `delta.enableRowTracking=true`) correctly handles a +/// subsequent data append. The initial create writes `rowIdHighWaterMark = -1`; the append must +/// read that and assign `baseRowId = 0` to the first file. +#[tokio::test] +async fn test_feature_signal_create_then_append_assigns_correct_base_row_id() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // Create empty table with feature signal only (no enablement property) + let _ = create_table(&table_path, super::simple_schema()?, "Test/1.0") + .with_table_properties([("delta.feature.rowTracking", "supported")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let table_url = Url::from_directory_path(&table_path).expect("valid path"); + + // Initial commit must have written rowIdHighWaterMark = -1 + let v0_snapshot = Snapshot::builder_for(&table_path) + .at_version(0) + .build(engine.as_ref())?; + assert_eq!( + RowTrackingDomainMetadata::get_high_water_mark(&v0_snapshot, engine.as_ref())?, + Some(-1), + "Initial high water mark should be -1" + ); + + // Append 3 rows to the table + let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let _ = insert_data( + snapshot, + &engine, + vec![ + Arc::new(Int32Array::from(vec![1, 2, 3])), + Arc::new(StringArray::from(vec!["a", "b", "c"])), + ], + ) + .await?; + + // The append must read the HWM from the create commit and assign baseRowId = 0 + let add_actions = get_row_tracking_add_actions(&table_url, 1).expect("failed to read commit"); + assert_eq!(add_actions.len(), 1, "Expected one add action"); + assert_eq!( + add_actions[0].base_row_id, + Some(0), + "First file after create should start at baseRowId 0 (high water mark was -1)" + ); + assert_eq!(add_actions[0].default_row_commit_version, Some(1)); + + // High water mark after append: 3 rows starting from 0 -> high water mark = 2 + let v1_snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!( + RowTrackingDomainMetadata::get_high_water_mark(&v1_snapshot, engine.as_ref())?, + Some(2), + "3 rows starting from 0 -> high water mark = 2" + ); + + Ok(()) +} diff --git a/kernel/tests/create_table/timestamp_ntz.rs b/kernel/tests/integration/create_table/timestamp_ntz.rs similarity index 97% rename from kernel/tests/create_table/timestamp_ntz.rs rename to kernel/tests/integration/create_table/timestamp_ntz.rs index 9e97e50ac1..f67bc6999a 100644 --- a/kernel/tests/create_table/timestamp_ntz.rs +++ b/kernel/tests/integration/create_table/timestamp_ntz.rs @@ -88,7 +88,7 @@ fn test_create_table_no_timestamp_ntz_no_feature() -> DeltaResult<()> { let (_temp_dir, table_path, engine) = test_table_setup()?; let schema = Arc::new(StructType::try_new(vec![ - StructField::new("id", DataType::INTEGER, false), + StructField::new("id", DataType::INTEGER, true), StructField::new("name", DataType::STRING, true), ])?); @@ -114,7 +114,7 @@ fn test_create_table_timestamp_ntz_and_variant() -> DeltaResult<()> { let (_temp_dir, table_path, engine) = test_table_setup()?; let schema = Arc::new(StructType::new_unchecked(vec![ - StructField::new("id", DataType::INTEGER, false), + StructField::new("id", DataType::INTEGER, true), StructField::new("ts", DataType::TIMESTAMP_NTZ, true), StructField::new("v", DataType::unshredded_variant(), true), ])); diff --git a/kernel/tests/create_table/variant.rs b/kernel/tests/integration/create_table/variant.rs similarity index 98% rename from kernel/tests/create_table/variant.rs rename to kernel/tests/integration/create_table/variant.rs index f6bf9e408f..b25d99556d 100644 --- a/kernel/tests/create_table/variant.rs +++ b/kernel/tests/integration/create_table/variant.rs @@ -95,7 +95,7 @@ fn test_create_table_no_variant_no_feature() -> DeltaResult<()> { let (_temp_dir, table_path, engine) = test_table_setup()?; let schema = Arc::new(StructType::try_new(vec![ - StructField::new("id", DataType::INTEGER, false), + StructField::new("id", DataType::INTEGER, true), StructField::new("name", DataType::STRING, true), ])?); diff --git a/kernel/tests/integration/cross_product/mod.rs b/kernel/tests/integration/cross_product/mod.rs new file mode 100644 index 0000000000..c45121f0fa --- /dev/null +++ b/kernel/tests/integration/cross_product/mod.rs @@ -0,0 +1,106 @@ +use std::sync::Arc; + +use delta_kernel::schema::MetadataColumnSpec; +use delta_kernel::snapshot::ChecksumWriteResult; +use delta_kernel::{DeltaResult, Engine, Snapshot}; +use rstest::rstest; +use rstest_reuse::apply; +use test_utils::delta_kernel_default_engine::DefaultEngineBuilder; +use test_utils::table_builder::{ + all_features_cm_id, all_features_cm_name, checkpoint_at_end, checkpoint_at_end_crc_at_end, + checkpoint_at_end_no_hint, checkpoint_at_end_no_hint_post_cleanup, + checkpoint_at_end_post_cleanup, checkpoint_mid, checkpoint_mid_crc_above_mid_post_cleanup, + checkpoint_mid_crc_at_end_post_cleanup, checkpoint_mid_crc_at_mid_post_cleanup, + checkpoint_mid_no_hint, checkpoint_mid_no_hint_post_cleanup, checkpoint_mid_post_cleanup, + clustered, commits_only, crc_at_end, crc_at_mid, no_checkpoint_stats, no_features, + num_indexed_cols_all, num_indexed_cols_narrow, num_indexed_cols_zero, partitioned, + stats_columns_empty, stats_columns_reordered, test_table, two_checkpoints_stale_hint, + two_checkpoints_stale_hint_post_cleanup, unpartitioned, version_at_mid, + version_at_timestamp_max, version_incremental_from_mid_to_latest, + version_incremental_from_mid_to_pre_latest, version_latest, with_json_stats, with_struct_stats, + DataLayoutConfig, FeatureSet, LogState, TableConfig, VersionTarget, +}; +use test_utils::{assert_row_ids_unique, build_snapshot, default_sweep, read_scan}; + +/// `TestTableBuilder`'s default is one file per data commit with this many rows, so a +/// snapshot at version `v` has exactly `v * ROWS_PER_COMMIT` total rows. File count +/// per commit isn't asserted because partitioned layouts may emit multiple files. +const ROWS_PER_COMMIT: usize = 10; + +/// `default_sweep` is the canonical `{LogState x FeatureSet x (DataLayoutConfig, +/// TableConfig) x VersionTarget}` cross-product defined in `test_utils`. Data layout and +/// stats config share one bundled axis (see `layout_config_values`) rather than being +/// crossed, to keep the case count within budget. Each combination expands into its own +/// test runner case. Invoking `test_table` here also exercises the write path that produces +/// each table state. +#[apply(default_sweep)] +fn test_cross_product_read_write( + log_state: LogState, + feature_set: FeatureSet, + layout_config: (DataLayoutConfig, TableConfig), + version_target: VersionTarget, +) -> DeltaResult<()> { + let (data_layout, table_config) = layout_config; + let table = test_table(log_state.clone(), feature_set, data_layout, table_config); + let engine: Arc = + Arc::new(DefaultEngineBuilder::new(table.store().clone()).build()); + let snap = build_snapshot!(version_target, table.table_root(), engine.as_ref()); + + let expected_version = match &version_target { + VersionTarget::Latest | VersionTarget::IncrementalToLatest { .. } => { + log_state.latest_version() + } + // `i64::MAX` is the only timestamp in the sweep; it always resolves to latest. + VersionTarget::AtTimestamp(ts) if *ts == i64::MAX => log_state.latest_version(), + VersionTarget::AtVersion(v) => *v, + VersionTarget::IncrementalFrom { to, .. } => *to, + VersionTarget::AtTimestamp(ts) => { + panic!("sweep only uses AtTimestamp(i64::MAX), got {ts}") + } + }; + assert_eq!(snap.version(), expected_version); + + let scan = snap.clone().scan_builder().build()?; + let batches = read_scan(&scan, engine.clone())?; + let rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(rows, expected_version as usize * ROWS_PER_COMMIT); + + if snap.table_properties().enable_row_tracking == Some(true) { + let scan_schema = Arc::new( + snap.schema() + .add_metadata_column("row_id", MetadataColumnSpec::RowId)?, + ); + let scan = snap + .clone() + .scan_builder() + .with_schema(scan_schema) + .build()?; + let batches = read_scan(&scan, engine.clone())?; + assert_row_ids_unique(&batches); + } + + write_and_assert_checksum(&snap, &log_state, expected_version, engine.as_ref())?; + + Ok(()) +} + +/// Write a CRC for `snap` and assert the exact outcome. A CRC can be written for virtually any +/// table state; the only cases that block it are a missing remove `size` or a non-incremental +/// operation (e.g. ANALYZE STATS) in the replayed commits, which force `Indeterminate` file +/// stats. The sweep exercises neither, so the write succeeds unless a CRC already exists at this +/// version, which is exactly known from the sweep. +fn write_and_assert_checksum( + snap: &delta_kernel::snapshot::SnapshotRef, + log_state: &LogState, + version: u64, + engine: &dyn Engine, +) -> DeltaResult<()> { + let expected_result = if log_state.crcs_at().contains(&version) { + ChecksumWriteResult::AlreadyExists + } else { + ChecksumWriteResult::Written + }; + let (result, _) = snap.write_checksum(engine)?; + assert_eq!(result, expected_result); + Ok(()) +} diff --git a/kernel/tests/integration/data_skipping.rs b/kernel/tests/integration/data_skipping.rs new file mode 100644 index 0000000000..f3001e6853 --- /dev/null +++ b/kernel/tests/integration/data_skipping.rs @@ -0,0 +1,1354 @@ +//! Integration tests for past-cap data-skipping behavior. +//! +//! Covers a table with `delta.dataSkippingNumIndexedCols=N` plus a predicate referencing a +//! past-cap column. The rewritten predicate must drop refs to columns missing from the +//! unified stats schema (and from the checkpoint parquet projection); otherwise scan setup +//! fails with "Schema error: No such field". The tests exercise three code paths that all +//! share that rewrite: +//! +//! - `Scan::scan_metadata` (in-memory `DataSkippingFilter`) +//! - `Scan::parallel_scan_metadata` (worker rebuilds `DataSkippingFilter` from `InternalScanState`) +//! - `Scan::build_actions_meta_predicate` (`as_checkpoint_skipping_predicate` pushed down to the +//! checkpoint parquet row-group filter) + +use std::collections::HashMap; +use std::sync::Arc; + +use delta_kernel::arrow::array::{Int64Array, RecordBatch}; +use delta_kernel::arrow::datatypes::Schema as ArrowSchema; +use delta_kernel::checkpoint::{CheckpointSpec, V2CheckpointConfig}; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::engine::arrow_conversion::TryIntoArrow as _; +use delta_kernel::expressions::{ + col, lit, CastOptions, Expression as Expr, Predicate as Pred, PredicateRef, Scalar, +}; +use delta_kernel::metrics::{MetricEvent, ScanType}; +use delta_kernel::object_store::local::LocalFileSystem; +use delta_kernel::scan::{AfterSequentialScanMetadata, ParallelScanMetadata}; +use delta_kernel::schema::{DataType, SchemaRef, StructField, StructType}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::transaction::data_layout::DataLayout; +use delta_kernel::{Error, Snapshot, SnapshotRef}; +use rstest::rstest; +use serde::Serialize; +use serde_json::{json, Value}; +use test_utils::delta_kernel_default_engine::executor::tokio::TokioMultiThreadExecutor; +use test_utils::delta_kernel_default_engine::DefaultEngine; +use test_utils::{ + add_commit, create_table_and_load_snapshot, install_thread_local_metrics_reporter, + test_table_setup_mt, write_batch_to_table, CapturingReporter, +}; +use url::Url; + +use crate::common::write_utils::set_table_properties; + +type TestEngine = DefaultEngine; + +/// Flat schema with `c0..c4`. `delta.dataSkippingNumIndexedCols=2` keeps stats on `c0` and +/// `c1`; `c2..c4` are past-cap. +fn capped_schema() -> SchemaRef { + Arc::new( + StructType::try_new((0..5).map(|i| StructField::nullable(format!("c{i}"), DataType::LONG))) + .unwrap(), + ) +} + +/// Builds one `RecordBatch` of `rows` rows. `c0` ranges over `c0_start..c0_start+rows`, with +/// `c1 = c0 + 100` so `c1` ranges share the same width but at higher values. `c2..c4` are +/// filled with constant `c0_start` so they have no useful min/max even if they were indexed. +fn long_batch(schema: &SchemaRef, c0_start: i64, rows: i64) -> RecordBatch { + let arrow_schema: ArrowSchema = schema.as_ref().try_into_arrow().unwrap(); + let c0 = Int64Array::from_iter_values(c0_start..c0_start + rows); + let c1 = Int64Array::from_iter_values(c0_start + 100..c0_start + 100 + rows); + let cn = Int64Array::from_iter_values(std::iter::repeat_n(c0_start, rows as usize)); + RecordBatch::try_new( + Arc::new(arrow_schema), + vec![ + Arc::new(c0), + Arc::new(c1), + Arc::new(cn.clone()), + Arc::new(cn.clone()), + Arc::new(cn), + ], + ) + .unwrap() +} + +/// Creates a fresh table with `dataSkippingNumIndexedCols=2`, writes three add-files with +/// disjoint `c0`/`c1` ranges, then checkpoints. Returns `(temp_dir, table_path, engine)`. +/// The `temp_dir` keeps the on-disk table alive for the duration of the test. +async fn build_capped_table_with_checkpoint( +) -> Result<(tempfile::TempDir, String, Arc), Box> { + let schema = capped_schema(); + let (tmp_dir, table_path, engine) = test_table_setup_mt()?; + let table_url = Url::from_directory_path(&table_path) + .map_err(|_| "table_path should be a valid file URL")?; + // `create_table_and_load_snapshot` rejects `delta.dataSkippingNumIndexedCols` at CREATE + // TABLE time, so backfill it via a v0-rewrite metadata commit. + let snapshot = + create_table_and_load_snapshot(&table_path, schema.clone(), engine.as_ref(), &[])?; + let mut snapshot = set_table_properties( + &table_path, + &table_url, + engine.as_ref(), + snapshot.version(), + &[("delta.dataSkippingNumIndexedCols", "2")], + )?; + + // file A: c0 in [10, 19], c1 in [110, 119] + // file B: c0 in [50, 59], c1 in [150, 159] + // file C: c0 in [100, 109], c1 in [200, 209] + for c0_start in [10, 50, 100] { + snapshot = write_batch_to_table( + &snapshot, + engine.as_ref(), + long_batch(&schema, c0_start, 10), + HashMap::new(), + ) + .await?; + } + + snapshot.checkpoint(engine.as_ref(), None)?; + Ok((tmp_dir, table_path, engine)) +} + +/// Returns the paths of files surviving data skipping. Exercises the sequential +/// `Scan::scan_metadata` path when `use_parallel` is `false`, or the two-phase +/// `Scan::parallel_scan_metadata` + `ParallelScanMetadata` path when `true`. The parallel +/// branch dispatches one file per `ParallelScanMetadata` to maximize coverage of the worker +/// rebuild path. +fn selected_paths( + snapshot: SnapshotRef, + engine: Arc, + predicate: PredicateRef, + use_parallel: bool, +) -> Result, Box> { + let scan = snapshot.scan_builder().with_predicate(predicate).build()?; + + fn push_path(paths: &mut Vec, scan_file: delta_kernel::scan::state::ScanFile) { + paths.push(scan_file.path); + } + + let mut paths: Vec = Vec::new(); + + if use_parallel { + let mut sequential = scan.parallel_scan_metadata(engine.clone())?; + for sm in sequential.by_ref() { + paths = sm?.visit_scan_files(paths, push_path)?; + } + if let AfterSequentialScanMetadata::Parallel { state, files } = sequential.finish()? { + let state = Arc::from(state); + for file in files { + let parallel = + ParallelScanMetadata::try_new(engine.clone(), Arc::clone(&state), vec![file])?; + for sm in parallel { + paths = sm?.visit_scan_files(paths, push_path)?; + } + } + state.log_metrics(); + } + } else { + for sm in scan.scan_metadata(engine.as_ref())? { + paths = sm?.visit_scan_files(paths, push_path)?; + } + } + + Ok(paths) +} + +/// Loads a fresh snapshot off the same on-disk table and returns the surviving file count +/// for `predicate` along both the sequential and parallel scan-metadata paths. +fn surviving_files( + table_path: &str, + engine: Arc, + predicate: PredicateRef, + use_parallel: bool, +) -> Result> { + Ok(surviving_paths(table_path, engine, predicate, use_parallel)?.len()) +} + +/// Like [`surviving_files`] but returns the surviving file paths, sorted so a test can make an +/// order-independent exact-set assertion (`scan_metadata`, and especially the parallel path, does +/// not guarantee a stable file emission order). +/// +/// Test-only: this never runs in the production scan path. The sort is trivial here because the +/// test tables yield only a handful of files, but it is O(n log n) in the surviving-file count and +/// so would not belong on a hot path over a large result set. +fn surviving_paths( + table_path: &str, + engine: Arc, + predicate: PredicateRef, + use_parallel: bool, +) -> Result, Box> { + let url = delta_kernel::try_parse_uri(table_path)?; + let snapshot = Snapshot::builder_for(url).build(engine.as_ref())?; + let mut paths = selected_paths(snapshot, engine, predicate, use_parallel)?; + paths.sort(); + Ok(paths) +} + +// === Predicate matrix === +// +// Table layout (3 files, `dataSkippingNumIndexedCols=2`): +// file A: c0 in [10, 19], c1 in [110, 119] +// file B: c0 in [50, 59], c1 in [150, 159] +// file C: c0 in [100, 109], c1 in [200, 209] +// c2/c3/c4: past-cap, no stats. + +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn all_in_cap_prunes( + #[values(false, true)] use_parallel: bool, +) -> Result<(), Box> { + let (_tmp_dir, table_path, engine) = build_capped_table_with_checkpoint().await?; + let pred = Arc::new(Pred::gt(col!("c0"), lit(60i64))); + assert_eq!(surviving_files(&table_path, engine, pred, use_parallel)?, 1); + Ok(()) +} + +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn all_in_cap_keeps_all( + #[values(false, true)] use_parallel: bool, +) -> Result<(), Box> { + let (_tmp_dir, table_path, engine) = build_capped_table_with_checkpoint().await?; + let pred = Arc::new(Pred::gt(col!("c0"), lit(0i64))); + assert_eq!(surviving_files(&table_path, engine, pred, use_parallel)?, 3); + Ok(()) +} + +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn all_past_cap_keeps_all( + #[values(false, true)] use_parallel: bool, +) -> Result<(), Box> { + let (_tmp_dir, table_path, engine) = build_capped_table_with_checkpoint().await?; + let pred = Arc::new(Pred::gt(col!("c4"), lit(1_000_000i64))); + assert_eq!(surviving_files(&table_path, engine, pred, use_parallel)?, 3); + Ok(()) +} + +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn mixed_and_in_cap_prunes( + #[values(false, true)] use_parallel: bool, +) -> Result<(), Box> { + let (_tmp_dir, table_path, engine) = build_capped_table_with_checkpoint().await?; + // c0 > 60 prunes files A and B; c4 > 50 is past-cap and folds to NULL. + // AND(false, NULL) = false keeps the prune; AND(true, NULL) = NULL keeps file C. + let pred = Arc::new(Pred::and( + Pred::gt(col!("c0"), lit(60i64)), + Pred::gt(col!("c4"), lit(50i64)), + )); + assert_eq!(surviving_files(&table_path, engine, pred, use_parallel)?, 1); + Ok(()) +} + +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn mixed_or_keeps_all( + #[values(false, true)] use_parallel: bool, +) -> Result<(), Box> { + let (_tmp_dir, table_path, engine) = build_capped_table_with_checkpoint().await?; + // c0 > 1000 would prune all 3 by max; c4 > 50 is past-cap and folds to NULL. + // OR(false, NULL) = NULL keeps every file. + let pred = Arc::new(Pred::or( + Pred::gt(col!("c0"), lit(1000i64)), + Pred::gt(col!("c4"), lit(50i64)), + )); + assert_eq!(surviving_files(&table_path, engine, pred, use_parallel)?, 3); + Ok(()) +} + +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn boundary_c1_prunes_all( + #[values(false, true)] use_parallel: bool, +) -> Result<(), Box> { + let (_tmp_dir, table_path, engine) = build_capped_table_with_checkpoint().await?; + // c1 max across files is 209 < 250 so the in-cap arm rules out everything. + // c2 > 50 is past-cap and folds to NULL; AND(false, NULL) = false everywhere. + let pred = Arc::new(Pred::and( + Pred::gt(col!("c1"), lit(250i64)), + Pred::gt(col!("c2"), lit(50i64)), + )); + assert_eq!(surviving_files(&table_path, engine, pred, use_parallel)?, 0); + Ok(()) +} + +// === Per-cell NULL on malformed stats === +// +// A Delta producer can emit an Add whose `stats` JSON contains an extended-year ISO 8601 +// timestamp like `+48690-07-02T22:50:38.211Z`. Arrow's `TimestampParser` rejects such +// strings. With per-cell NULL only the bad cell becomes NULL, so other files in the same +// batch keep their stats and the predicate prunes them correctly. + +fn timestamp_stats_schema() -> SchemaRef { + Arc::new( + StructType::try_new(vec![ + StructField::nullable("EventTime", DataType::TIMESTAMP), + StructField::nullable("UserId", DataType::LONG), + ]) + .unwrap(), + ) +} + +/// Creates a `timestamp_stats_schema` table and returns the pieces needed to inject raw commits: +/// `(tmp_dir, table_path, engine, table_url, store)`. +#[allow(clippy::type_complexity)] +fn timestamp_stats_table_setup( + properties: &[(&str, &str)], +) -> Result< + ( + tempfile::TempDir, + String, + Arc>, + Url, + Arc, + ), + Box, +> { + let (tmp_dir, table_path, engine) = test_table_setup_mt()?; + create_table_and_load_snapshot( + &table_path, + timestamp_stats_schema(), + engine.as_ref(), + properties, + )?; + let store: Arc = Arc::new(LocalFileSystem::new()); + let table_url = Url::from_directory_path(&table_path) + .map_err(|_| "table_path should be a valid file URL")?; + Ok((tmp_dir, table_path, engine, table_url, store)) +} + +/// An `EventTime` predicate against a microsecond bound, e.g. `timestamp_pred(Pred::le, micros)`. +fn timestamp_pred(op: fn(Expr, Expr) -> Pred, micros: i64) -> PredicateRef { + Arc::new(op(col!("EventTime"), lit(Scalar::Timestamp(micros)))) +} + +/// Builds a stringified Delta `stats` JSON given EventTime/UserId min/max bounds. +fn stats_json( + num_records: i64, + event_time_min: &str, + event_time_max: &str, + user_id_min: i64, + user_id_max: i64, +) -> String { + format!( + r#"{{"numRecords":{num_records},"minValues":{{"EventTime":"{event_time_min}","UserId":{user_id_min}}},"maxValues":{{"EventTime":"{event_time_max}","UserId":{user_id_max}}},"nullCount":{{"EventTime":0,"UserId":0}},"tightBounds":true}}"# + ) +} + +/// Builds a Delta commit body containing a `commitInfo` plus one Add per `(path, stats)`. +fn commit_with_adds(version: u64, adds: &[(&str, String)]) -> String { + let mut lines = vec![format!( + r#"{{"commitInfo":{{"timestamp":1700000000000,"operation":"WRITE","version":{version}}}}}"# + )]; + for (path, stats) in adds { + let stats_escaped = serde_json::Value::String(stats.clone()).to_string(); + lines.push(format!( + r#"{{"add":{{"path":"{path}","size":1024,"modificationTime":1700000000000,"dataChange":true,"partitionValues":{{}},"stats":{stats_escaped}}}}}"# + )); + } + lines.join("\n") +} + +/// Builds a Delta commit body containing a `commitInfo` plus a single `remove` action. +fn commit_with_remove(version: u64, path: &str, deletion_timestamp: i64) -> String { + let commit_info = format!( + r#"{{"commitInfo":{{"timestamp":{deletion_timestamp},"operation":"DELETE","version":{version}}}}}"# + ); + let remove = format!( + r#"{{"remove":{{"path":"{path}","deletionTimestamp":{deletion_timestamp},"dataChange":true,"extendedFileMetadata":false}}}}"# + ); + format!("{commit_info}\n{remove}") +} + +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn extended_year_timestamp_stats_dont_collapse_skipping( + #[values(false, true)] use_parallel: bool, +) -> Result<(), Box> { + let (_tmp_dir, table_path, engine, table_url, store) = timestamp_stats_table_setup(&[])?; + let table_url_string = table_url.to_string(); + + // Co-locate the malformed `file_B` with a valid `file_D` (Sep 2024, out of predicate + // range) in the same commit. If the parse error all-nulled the batch, file_D would + // survive too; with per-cell NULL its stats are preserved and the predicate prunes it. + add_commit( + &table_url_string, + store.as_ref(), + 1, + commit_with_adds( + 1, + &[( + "file_A.parquet", + stats_json( + 10, + "2024-01-15T00:00:00.000Z", + "2024-01-31T00:00:00.000Z", + 1, + 100, + ), + )], + ), + ) + .await?; + add_commit( + &table_url_string, + store.as_ref(), + 2, + commit_with_adds( + 2, + &[ + ( + "file_B.parquet", + stats_json( + 20, + "+48690-07-02T22:50:38.211Z", + "+48690-07-02T22:50:38.211Z", + 5, + 200, + ), + ), + ( + "file_D.parquet", + stats_json( + 40, + "2024-09-01T00:00:00.000Z", + "2024-09-30T00:00:00.000Z", + 7, + 70, + ), + ), + ], + ), + ) + .await?; + add_commit( + &table_url_string, + store.as_ref(), + 3, + commit_with_adds( + 3, + &[( + "file_C.parquet", + stats_json( + 30, + "2025-01-01T00:00:00.000Z", + "2025-12-31T00:00:00.000Z", + 10, + 300, + ), + )], + ), + ) + .await?; + + // Predicate: EventTime < 2024-06-01T00:00:00Z AND UserId > 0. + // 2024-06-01T00:00:00 UTC = 1_717_200_000 seconds since epoch = 1_717_200_000_000_000 us. + let june_first_2024_us: i64 = 1_717_200_000_000_000; + let predicate = Arc::new(Pred::and( + Pred::lt( + col!("EventTime"), + lit(Scalar::Timestamp(june_first_2024_us)), + ), + Pred::gt(col!("UserId"), lit(0i64)), + )); + + // Expected survivors: + // file_A: EventTime min Jan 15 < Jun 1, UserId max 100 > 0, kept. + // file_B: EventTime min/max NULL from safe-cast, conservatively kept; UserId max 200 > 0. + // file_C: EventTime min Jan 2025 > Jun 2024, pruned. + // file_D: EventTime min Sep 2024 > Jun 2024, pruned. Would survive if file_B's bad + // stats had collapsed the whole v2 batch to NULL stats. + assert_eq!( + surviving_files(&table_path, engine, predicate, use_parallel)?, + 2 + ); + Ok(()) +} + +/// Round-trip the safe-cast through checkpoint materialization + post-checkpoint JSON + +/// Remove honoring. +/// +/// Sequence: +/// v0: CREATE TABLE with `delta.checkpoint.writeStatsAsStruct = true`. +/// v1: One commit containing file_A (valid Jan 2024, would-survive), +/// file_B (malformed extended-year EventTime), and +/// file_E (valid Jan-Dec 2025, would-be-pruned by the predicate). +/// Co-locating them forces the checkpoint-write `parse_json` call to see all three +/// in one batch, which is where the per-cell NULL property matters. +/// checkpoint at v1: `build_stats_parsed_expr` evaluates +/// `COALESCE(stats_parsed, ParseJson(stats))`; safe-cast preserves file_A and file_E +/// stats and NULLs only file_B's `EventTime` min/max. +/// v2: file_C added via JSON (post-checkpoint), exercises +/// `scan/log_replay.rs` `parse_json` on a normal JSON commit. +/// v3: Remove for file_C, exercises Remove reconciliation against a post-checkpoint Add. +/// +/// Predicate: `EventTime < 2024-06-01 AND UserId > 0`. +/// +/// Expected survivors (driven by both sources): +/// file_A: kept via checkpoint `stats_parsed` (Jan 2024 satisfies predicate). +/// file_B: kept via checkpoint `stats_parsed` (NULL EventTime, conservative). +/// file_E: pruned via checkpoint `stats_parsed` (Jan 2025+). Would survive if file_B's +/// bad stats had blanked the whole v1 batch during checkpoint write. +/// file_C: would-survive predicate via JSON `parse_json`, but the v3 Remove suppresses +/// it during log replay. +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn extended_year_timestamp_round_trip_via_checkpoint_and_remove( + #[values(false, true)] use_parallel: bool, +) -> Result<(), Box> { + // writeStatsAsStruct makes the checkpoint materialize stats_parsed, the column where the + // safe-cast result lands. + let (_tmp_dir, table_path, engine, table_url, store) = + timestamp_stats_table_setup(&[("delta.checkpoint.writeStatsAsStruct", "true")])?; + let table_url_string = table_url.to_string(); + + // v1: file_A + file_B + file_E in one commit. The shared batch is what makes the + // per-cell NULL property observable during checkpoint write. + add_commit( + &table_url_string, + store.as_ref(), + 1, + commit_with_adds( + 1, + &[ + ( + "file_A.parquet", + stats_json( + 10, + "2024-01-15T00:00:00.000Z", + "2024-01-31T00:00:00.000Z", + 1, + 100, + ), + ), + ( + "file_B.parquet", + stats_json( + 20, + "+48690-07-02T22:50:38.211Z", + "+48690-07-02T22:50:38.211Z", + 5, + 200, + ), + ), + ( + "file_E.parquet", + stats_json( + 50, + "2025-01-01T00:00:00.000Z", + "2025-12-31T00:00:00.000Z", + 10, + 300, + ), + ), + ], + ), + ) + .await?; + + // Checkpoint at v1: build_stats_parsed_expr COALESCE evaluates parse_json on the v1 + // batch and materializes stats_parsed with file_B's EventTime safe-cast to NULL. + let snapshot_v1 = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + snapshot_v1.checkpoint(engine.as_ref(), None)?; + + // v2: file_C via JSON commit, post-checkpoint. Reads still see this as JSON stats and + // run parse_json at scan time. + add_commit( + &table_url_string, + store.as_ref(), + 2, + commit_with_adds( + 2, + &[( + "file_C.parquet", + stats_json( + 15, + "2024-02-01T00:00:00.000Z", + "2024-02-29T00:00:00.000Z", + 50, + 80, + ), + )], + ), + ) + .await?; + + // v3: Remove file_C. Log replay must reconcile the v2 Add with this Remove so that + // file_C does not appear in the scan output, regardless of predicate. + add_commit( + &table_url_string, + store.as_ref(), + 3, + commit_with_remove(3, "file_C.parquet", 1_700_000_001_000), + ) + .await?; + + let june_first_2024_us: i64 = 1_717_200_000_000_000; + let predicate = Arc::new(Pred::and( + Pred::lt( + col!("EventTime"), + lit(Scalar::Timestamp(june_first_2024_us)), + ), + Pred::gt(col!("UserId"), lit(0i64)), + )); + + assert_eq!( + surviving_files(&table_path, engine, predicate, use_parallel)?, + 2 + ); + Ok(()) +} + +/// Millisecond-truncated timestamp stats must not over-prune files whose real values live in the +/// dropped sub-millisecond tail. `adjust_scalar_for_max_stat_truncation` is what makes it safe. +/// +/// The file holds timestamps in [.298677, .307735], so its truncated stats are [.298, .307]. Every +/// predicate below selects a real row in the file, so pruning it would drop committed data. +/// +/// The lower-bound cases are the ones that exercise `adjust_scalar_for_max_stat_truncation`: a +/// `>=` bound between the stored (truncated) max `.307000` and the real max `.307735` compares +/// against the max stat, and without the 999us widening the file is wrongly pruned. +#[rstest] +// Upper bound inside the truncated range. +#[case::upper_bound_within_range(timestamp_pred(Pred::le, 1_783_007_755_299_000), 1)] +// Upper bound in the tail below the real max but above the truncated min. +#[case::upper_bound_in_truncated_tail(timestamp_pred(Pred::le, 1_783_007_755_298_900), 1)] +// Upper bound below the file's truncated min: nothing in the file can match, prune is correct. +#[case::upper_bound_below_min(timestamp_pred(Pred::le, 1_783_007_754_000_000), 0)] +// Lower bound inside the sub-millisecond tail dropped from the max stat. The file really does +// hold `.307735`, so it must survive despite the stat claiming `.307000`. +#[case::lower_bound_in_truncated_tail(timestamp_pred(Pred::ge, 1_783_007_755_307_500), 1)] +// Lower bound at the real max: still inside the tail, still must survive. +#[case::lower_bound_at_real_max(timestamp_pred(Pred::ge, 1_783_007_755_307_735), 1)] +// Lower bound far past the tail (beyond stored max + 999us): pruning is correct here. +#[case::lower_bound_past_tail(timestamp_pred(Pred::ge, 1_783_007_755_400_000), 0)] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn millisecond_truncated_timestamp_stats_dont_overprune( + #[case] predicate: PredicateRef, + #[case] expected_survivors: usize, + #[values(false, true)] use_parallel: bool, +) -> Result<(), Box> { + let (_tmp_dir, table_path, engine, table_url, store) = timestamp_stats_table_setup(&[])?; + + // Real values span [.298677, .307735]; a protocol-conforming writer floors the stats to + // [.298, .307]. + add_commit( + table_url.as_str(), + store.as_ref(), + 1, + commit_with_adds( + 1, + &[( + "file_streaming.parquet", + stats_json( + 10, + "2026-07-02T15:55:55.298Z", + "2026-07-02T15:55:55.307Z", + 1, + 100, + ), + )], + ), + ) + .await?; + + assert_eq!( + surviving_files(&table_path, engine, predicate, use_parallel)?, + expected_survivors, + ); + Ok(()) +} + +/// Replace table may change partition and data column types. A scan after replacement should +/// remain compatible with the old addFiles. +#[rstest] +#[case::partition( + Arc::new(Pred::eq(col!("part"), lit(1i64))), + &["new-1.parquet"], + 1, + 101 +)] +#[case::stats( + Arc::new(Pred::eq(col!("value"), lit(20i64))), + &["new-2.parquet"], + 1, + 102 +)] +#[case::partition_and_stats( + Arc::new(Pred::and( + Pred::eq(col!("part"), lit(3i64)), + Pred::eq(col!("value"), lit(30i64)), + )), + &["new-3.parquet"], + 1, + 103 +)] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn scan_with_replace_table_schema_change( + #[case] predicate: PredicateRef, + #[case] expected_paths: &[&str], + #[case] expected_active_files: u64, + #[case] expected_active_bytes: u64, + #[values(false, true)] use_parallel: bool, + #[values(false, true)] checkpoint_old_add: bool, +) -> Result<(), Box> { + let (_tmp_dir, table_path, engine) = test_table_setup_mt()?; + + // Commit 1 creates the table with a string partition column. + let old_schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("part", DataType::STRING), + StructField::nullable("value", DataType::STRING), + ])?); + create_table(&table_path, old_schema, "Test/1.0") + .with_data_layout(DataLayout::partitioned(["part"])) + .with_table_properties([("delta.feature.v2Checkpoint", "supported")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + + let store: Arc = Arc::new(LocalFileSystem::new()); + let table_url = Url::from_directory_path(&table_path) + .map_err(|_| "table_path should be a valid file URL")?; + + // Commit 2 writes addFiles using `old_schema`. + let old_paths = ["old-1.parquet", "old-2.parquet", "old-3.parquet"]; + let old_adds = [ + json!({ + "commitInfo": { + "timestamp": 1700000000000i64, + "operation": "WRITE", + "version": 1, + } + }), + add_file_action( + old_paths[0], + 201, /* size */ + "not-an-integer-partition-1", /* partition_value */ + "not-an-integer-data-value-1", /* data_value */ + )?, + add_file_action( + old_paths[1], + 202, /* size */ + "not-an-integer-partition-2", /* partition_value */ + "not-an-integer-data-value-2", /* data_value */ + )?, + add_file_action( + old_paths[2], + 203, /* size */ + "not-an-integer-partition-3", /* partition_value */ + "not-an-integer-data-value-3", /* data_value */ + )?, + ]; + add_commit( + table_url.as_str(), + store.as_ref(), + 1, + old_adds.map(|action| action.to_string()).join("\n"), + ) + .await?; + + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + if checkpoint_old_add { + let checkpoint_spec = CheckpointSpec::V2(V2CheckpointConfig::WithSidecar { + file_actions_per_sidecar_hint: Some(1), + }); + snapshot.checkpoint(engine.as_ref(), Some(&checkpoint_spec))?; + } + + // Commit 3 replaces the table metadata and removes the addFiles. + let replacement_schema = StructType::try_new(vec![ + StructField::nullable("part", DataType::LONG), + StructField::nullable("value", DataType::LONG), + ])?; + let replacement = [ + json!({ + "commitInfo": { + "timestamp": 1700000001000i64, + "operation": "CREATE OR REPLACE TABLE", + "version": 2, + } + }), + json!({ + "metaData": { + "id": "replacement-table", + "format": { "provider": "parquet", "options": {} }, + "schemaString": serde_json::to_string(&replacement_schema)?, + "partitionColumns": ["part"], + "configuration": {}, + "createdTime": 1700000001000i64, + } + }), + remove_file_action(old_paths[0])?, + remove_file_action(old_paths[1])?, + remove_file_action(old_paths[2])?, + ]; + add_commit( + table_url.as_str(), + store.as_ref(), + 2, + replacement.map(|action| action.to_string()).join("\n"), + ) + .await?; + + // Commit 4 writes addFiles with distinct partition values and statistics. + let new_adds = [ + json!({ + "commitInfo": { + "timestamp": 1700000002000i64, + "operation": "WRITE", + "version": 3, + } + }), + add_file_action( + "new-1.parquet", + 101, /* size */ + "1", /* partition_value */ + 10, /* data_value */ + )?, + add_file_action( + "new-2.parquet", + 102, /* size */ + "2", /* partition_value */ + 20, /* data_value */ + )?, + add_file_action( + "new-3.parquet", + 103, /* size */ + "3", /* partition_value */ + 30, /* data_value */ + )?, + ]; + add_commit( + table_url.as_str(), + store.as_ref(), + 3, + new_adds.map(|action| action.to_string()).join("\n"), + ) + .await?; + + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + assert_eq!( + surviving_paths(&table_path, engine.clone(), predicate.clone(), use_parallel)?, + expected_paths + .iter() + .map(|path| (*path).to_string()) + .collect::>() + ); + let scan_events = reporter + .events() + .into_iter() + .filter_map(|event| match event { + MetricEvent::ScanMetadataCompleted(event) => Some(event), + _ => None, + }) + .collect::>(); + if use_parallel && checkpoint_old_add { + assert!( + scan_events + .iter() + .any(|event| event.scan_type == ScanType::ParallelPhase), + "the V2 sidecar should be processed by the parallel phase" + ); + } + assert_eq!( + scan_events + .iter() + .map(|e| e.num_add_files_seen) + .sum::(), + 4 + ); + assert_eq!( + scan_events + .iter() + .map(|e| e.num_active_add_files) + .sum::(), + expected_active_files + ); + assert_eq!( + scan_events + .iter() + .map(|e| e.active_add_files_bytes) + .sum::(), + expected_active_bytes + ); + assert_eq!( + scan_events + .iter() + .map(|e| e.num_remove_files_seen) + .sum::(), + 3 + ); + assert_eq!( + scan_events + .iter() + .map(|e| e.num_predicate_filtered) + .sum::(), + 2 + ); + // Negative test: a live addFile with incompatible schema should fail the scan. + let active_incompatible_add = [ + json!({ + "commitInfo": { + "timestamp": 1700000003000i64, + "operation": "WRITE", + "version": 4, + } + }), + add_file_action( + "active-incompatible.parquet", + 301, /* size */ + "not-an-integer-active-partition", /* partition_value */ + "not-an-integer-active-value", /* data_value */ + )?, + ]; + add_commit( + table_url.as_str(), + store.as_ref(), + 4, + active_incompatible_add + .map(|action| action.to_string()) + .join("\n"), + ) + .await?; + + if use_parallel { + // Put the active incompatible Add in a sidecar so the parallel processor rejects it. + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let checkpoint_spec = CheckpointSpec::V2(V2CheckpointConfig::WithSidecar { + file_actions_per_sidecar_hint: Some(1), + }); + snapshot.checkpoint(engine.as_ref(), Some(&checkpoint_spec))?; + } + let error = surviving_paths(&table_path, engine, predicate, use_parallel) + .expect_err("an active incompatible add file should fail the scan"); + assert!(matches!( + error.downcast_ref::(), + Some(Error::ParseError(_, _)) + )); + Ok(()) +} + +// === RFC 3339 offset partition values (#2733) === +// +// A foreign writer can emit a timestamp partition value with a non-UTC RFC 3339 offset. The +// offset must be honored and the value normalized to UTC, e.g. `2024-06-15T14:30:00+05:00` +// denotes 09:30 UTC, not 14:30 UTC. + +/// Builds a Delta commit body containing a `commitInfo` plus one stats-less Add per +/// `(path, ts_partition_value)`. +fn commit_with_ts_partitioned_adds(version: u64, adds: &[(&str, &str)]) -> String { + let mut lines = vec![format!( + r#"{{"commitInfo":{{"timestamp":1700000000000,"operation":"WRITE","version":{version}}}}}"# + )]; + for (path, ts) in adds { + lines.push(format!( + r#"{{"add":{{"path":"{path}","size":1024,"modificationTime":1700000000000,"dataChange":true,"partitionValues":{{"ts":"{ts}"}}}}}}"# + )); + } + lines.join("\n") +} + +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn partition_pruning_honors_rfc3339_offset_partition_values( + #[values(false, true)] use_parallel: bool, +) -> Result<(), Box> { + let (_tmp_dir, table_path, engine) = test_table_setup_mt()?; + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("ts", DataType::TIMESTAMP), + StructField::nullable("v", DataType::LONG), + ])?); + create_table(&table_path, schema, "Test/1.0") + .with_data_layout(DataLayout::partitioned(["ts"])) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + + // v1: adds in the style of an offset-emitting foreign writer. Pruning never opens data + // files, so fake paths are fine. The two partition values denote *different* instants: + // file_A is 2024-06-15T09:30:00Z once its +05:00 offset is honored, file_B is 14:30:00Z. + let store: Arc = Arc::new(LocalFileSystem::new()); + let table_url = Url::from_directory_path(&table_path) + .map_err(|_| "table_path should be a valid file URL")?; + add_commit( + table_url.as_str(), + store.as_ref(), + 1, + commit_with_ts_partitioned_adds( + 1, + &[ + ("file_A.parquet", "2024-06-15T14:30:00+05:00"), + ("file_B.parquet", "2024-06-15T14:30:00Z"), + ], + ), + ) + .await?; + + let nine_thirty_utc_us: i64 = 1_718_443_800_000_000; // 2024-06-15T09:30:00Z + let fourteen_thirty_utc_us: i64 = 1_718_461_800_000_000; // 2024-06-15T14:30:00Z + + // ts == 09:30Z must keep only file_A (its +05:00 value normalized to 09:30 UTC). + let predicate = Arc::new(Pred::eq( + col!("ts"), + lit(Scalar::Timestamp(nine_thirty_utc_us)), + )); + assert_eq!( + surviving_files(&table_path, engine.clone(), predicate, use_parallel)?, + 1 + ); + + // ts == 14:30Z must keep only file_B. + let predicate = Arc::new(Pred::eq( + col!("ts"), + lit(Scalar::Timestamp(fourteen_thirty_utc_us)), + )); + assert_eq!( + surviving_files(&table_path, engine, predicate, use_parallel)?, + 1 + ); + Ok(()) +} + +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn checkpoint_pruning_preserves_partition_cast_timezone( + #[values(false, true)] use_parallel: bool, +) -> Result<(), Box> { + let (_tmp_dir, table_path, engine) = test_table_setup_mt()?; + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("ts", DataType::STRING), + StructField::nullable("v", DataType::LONG), + ])?); + create_table(&table_path, schema, "Test/1.0") + .with_data_layout(DataLayout::partitioned(["ts"])) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + + let store: Arc = Arc::new(LocalFileSystem::new()); + let table_url = Url::from_directory_path(&table_path) + .map_err(|_| "table_path should be a valid file URL")?; + add_commit( + table_url.as_str(), + store.as_ref(), + 1, + commit_with_ts_partitioned_adds(1, &[("file.parquet", "2024-01-15 12:30:45")]), + ) + .await?; + Snapshot::builder_for(table_url) + .build(engine.as_ref())? + .checkpoint(engine.as_ref(), None)?; + + let predicate = Arc::new(Pred::gt( + Expr::cast( + col!("ts"), + DataType::TIMESTAMP, + CastOptions::default().with_timestamp_timezone("America/Los_Angeles"), + ), + Scalar::Timestamp(1_705_334_400_000_000), + )); + assert_eq!( + surviving_files(&table_path, engine, predicate, use_parallel)?, + 1 + ); + Ok(()) +} + +#[rstest] +#[case::year_month( + DataType::INTERVAL_YEAR_MONTH, + Scalar::IntervalYearMonth(12), + Scalar::IntervalYearMonth(12), + Scalar::IntervalYearMonth(24) +)] +#[case::day_time( + DataType::INTERVAL_DAY_TIME, + Scalar::IntervalDayTime(3_600_000_000), + Scalar::IntervalDayTime(3_600_000_000), + Scalar::IntervalDayTime(7_200_000_000) +)] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn interval_partition_values_do_not_prune_files( + #[case] interval: DataType, + #[case] predicate_value: Scalar, + #[case] first_value: Scalar, + #[case] second_value: Scalar, + #[values(false, true)] use_parallel: bool, +) -> Result<(), Box> { + let (_tmp_dir, table_path, engine) = test_table_setup_mt()?; + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("period", interval), + StructField::nullable("v", DataType::LONG), + ])?); + let mut snapshot = create_table(&table_path, schema, "Test/1.0") + .with_data_layout(DataLayout::partitioned(["period"])) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + let data_schema = StructType::try_new([StructField::nullable("v", DataType::LONG)])?; + let batch = RecordBatch::try_new( + Arc::new((&data_schema).try_into_arrow()?), + vec![Arc::new(Int64Array::from(vec![1]))], + )?; + snapshot = write_batch_to_table( + &snapshot, + engine.as_ref(), + batch.clone(), + HashMap::from([("period".to_string(), first_value)]), + ) + .await?; + let _snapshot = write_batch_to_table( + &snapshot, + engine.as_ref(), + batch, + HashMap::from([("period".to_string(), second_value)]), + ) + .await?; + + let predicate = Arc::new(Pred::eq(col!("period"), lit(predicate_value))); + assert_eq!( + surviving_files(&table_path, engine, predicate, use_parallel)?, + 2 + ); + Ok(()) +} + +// === All-null file pruning across log-replay sources === +// +// A file whose queried column is entirely NULL can never satisfy a null-intolerant comparison +// (`=`, `!=`, `<`, `>`, `<=`, `>=`), so data skipping should prune it. `DataSkippingFilter` +// builds its skipping predicate via `eval_sql_where`, which rewrites the `col IS NOT NULL` +// guard to `nullCount != numRecords` -- FALSE for an all-null file, forcing the prune. That +// single guarded predicate is applied to every transformed action batch, so pruning must hold +// no matter how the all-null file's Add is sourced or how its stats are stored: a JSON commit +// (`log_transform` parses `add.stats`), a checkpoint with native `stats_parsed` +// (`writeStatsAsStruct=true`), or a checkpoint with JSON stats (the protocol default, parsed +// from `add.stats` at read time). One exception is captured at the assertion: the parallel scan +// path does not yet read `stats_parsed`, so it cannot skip an all-null file sourced from a +// struct-only checkpoint. +#[derive(Clone, Copy, Debug)] +enum AllNullSource { + /// All-null file added via a JSON commit, with no checkpoint (the pure delta-log path). + CommitOnly, + /// All-null file materialized into the checkpoint as a native `stats_parsed` struct + /// (`delta.checkpoint.writeStatsAsStruct=true`). + CheckpointStructStats, + /// All-null file materialized into the checkpoint with JSON stats (the protocol default; + /// kernel parses `add.stats` at read time). + CheckpointJsonStats, + /// One all-null file in the checkpoint and another in a post-checkpoint JSON commit. + Both, +} + +/// Builds a stringified Delta `stats` JSON for the single `value: long` column. +/// `null_count == num_records` with `bounds == None` mimics an all-null file (Delta writers +/// omit min/max for an all-null column); pass `Some((min, max))` for a normal file. +fn value_stats_json(num_records: i64, null_count: i64, bounds: Option<(i64, i64)>) -> String { + let min_max = match bounds { + Some((min, max)) => { + format!(r#""minValues":{{"value":{min}}},"maxValues":{{"value":{max}}},"#) + } + None => r#""minValues":{},"maxValues":{},"#.to_string(), + }; + format!( + r#"{{"numRecords":{num_records},{min_max}"nullCount":{{"value":{null_count}}},"tightBounds":true}}"# + ) +} + +#[rstest] +// Two representative operators: `=` exercises the direct null-intolerant arm of +// `eval_sql_where` and `!=` the NOT-wrapped arm. The not-all-null guard is operator-agnostic, and +// the unit test `test_all_null_pruning_all_comparison_ops` covers all six operators at the rewrite +// level, so the source/parallel matrix here does not repeat every operator. +#[case::eq(Pred::eq(col!("value"), lit(5i64)))] +#[case::ne(Pred::ne(col!("value"), lit(5i64)))] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn all_null_files_pruned_regardless_of_source( + #[case] predicate: Pred, + #[values( + AllNullSource::CommitOnly, + AllNullSource::CheckpointStructStats, + AllNullSource::CheckpointJsonStats, + AllNullSource::Both + )] + source: AllNullSource, + #[values(false, true)] use_parallel: bool, +) -> Result<(), Box> { + let (_tmp_dir, table_path, engine) = test_table_setup_mt()?; + let schema = Arc::new(StructType::try_new(vec![StructField::nullable( + "value", + DataType::LONG, + )])?); + // For the struct-stats cases, also disable `writeStatsAsJson` so the checkpoint carries ONLY + // `stats_parsed`. That forces the scan to read the pre-parsed struct (no JSON fallback), + // genuinely validating that path. The JSON-stats case keeps the defaults + // (`writeStatsAsJson=true`, `writeStatsAsStruct=false`), so its checkpoint carries only JSON + // stats. Either way the all-null file's `nullCount` survives into the checkpoint. + let table_properties: &[(&str, &str)] = match source { + AllNullSource::CheckpointStructStats | AllNullSource::Both => &[ + ("delta.checkpoint.writeStatsAsStruct", "true"), + ("delta.checkpoint.writeStatsAsJson", "false"), + ], + AllNullSource::CommitOnly | AllNullSource::CheckpointJsonStats => &[], + }; + let _ = create_table_and_load_snapshot(&table_path, schema, engine.as_ref(), table_properties)?; + let store: Arc = Arc::new(LocalFileSystem::new()); + let table_url = Url::from_directory_path(&table_path) + .map_err(|_| "table_path should be a valid file URL")?; + let table_url_string = table_url.to_string(); + + let use_checkpoint = matches!( + source, + AllNullSource::CheckpointStructStats + | AllNullSource::CheckpointJsonStats + | AllNullSource::Both + ); + let add_post_checkpoint_commit = + matches!(source, AllNullSource::CommitOnly | AllNullSource::Both); + + // v1: a matching file (`keep.parquet`, value in [1, 10]) is always present. For + // checkpoint-sourced scenarios, co-locate an all-null file so it lands in the checkpoint. + let mut v1_adds = vec![("keep.parquet", value_stats_json(10, 0, Some((1, 10))))]; + if use_checkpoint { + v1_adds.push(("allnull_ckpt.parquet", value_stats_json(10, 10, None))); + } + add_commit( + &table_url_string, + store.as_ref(), + 1, + commit_with_adds(1, &v1_adds), + ) + .await?; + + // Checkpoint so the all-null file (and the kept file) are sourced from the checkpoint parquet. + if use_checkpoint { + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + snapshot.checkpoint(engine.as_ref(), None)?; + } + + // v2: post-checkpoint JSON commit adding an all-null file (the delta-log source). + if add_post_checkpoint_commit { + add_commit( + &table_url_string, + store.as_ref(), + 2, + commit_with_adds( + 2, + &[("allnull_commit.parquet", value_stats_json(10, 10, None))], + ), + ) + .await?; + } + + // `5` can never match an all-null file, but it sits inside the kept file's `[1, 10]` range, so + // `keep.parquet` always survives; every all-null file should otherwise be pruned. + let survivors = surviving_paths(&table_path, engine, Arc::new(predicate), use_parallel)?; + let mut expected = vec!["keep.parquet".to_string()]; + // TODO(#2832): remove this branch once the parallel scan path reads `stats_parsed`. + // `Scan::parallel_scan_metadata` sets `has_stats_parsed = false` and re-parses JSON + // `add.stats`, so over a struct-only checkpoint (JSON stats disabled) it has no stats to read + // and conservatively keeps the checkpoint-sourced all-null file -- a missed optimization, + // never wrong data; the sequential path reads `stats_parsed` and prunes it. When the parallel + // checkpoint reader gains `stats_parsed` support, `allnull_ckpt.parquet` will be pruned here + // too. The commit-sourced all-null file in `Both` carries JSON stats, so it is pruned on both + // paths. + let struct_only_checkpoint = matches!( + source, + AllNullSource::CheckpointStructStats | AllNullSource::Both + ); + if use_parallel && struct_only_checkpoint { + expected.push("allnull_ckpt.parquet".to_string()); + } + expected.sort(); + assert_eq!( + survivors, expected, + "unexpected survivors for source {source:?} (use_parallel={use_parallel})" + ); + Ok(()) +} + +#[derive(Serialize)] +struct AddActionFixture { + add: AddFileFixture, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct AddFileFixture { + path: String, + size: i64, + modification_time: i64, + data_change: bool, + partition_values: HashMap, + stats: String, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct StatsFixture { + num_records: i64, + min_values: ValueFixture, + max_values: ValueFixture, + null_count: ValueFixture, +} + +#[derive(Clone, Serialize)] +struct ValueFixture { + value: T, +} + +fn add_file_action( + path: &str, + size: i64, + partition_value: &str, + data_value: T, +) -> serde_json::Result { + let stats = serde_json::to_string(&StatsFixture { + num_records: 1, + min_values: ValueFixture { + value: data_value.clone(), + }, + max_values: ValueFixture { value: data_value }, + null_count: ValueFixture { value: 0 }, + })?; + serde_json::to_value(AddActionFixture { + add: AddFileFixture { + path: path.to_string(), + size, + modification_time: 1700000000000, + data_change: true, + partition_values: HashMap::from([("part".to_string(), partition_value.to_string())]), + stats, + }, + }) +} + +#[derive(Serialize)] +struct RemoveActionFixture { + remove: RemoveFileFixture, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct RemoveFileFixture { + path: String, + deletion_timestamp: i64, + data_change: bool, + extended_file_metadata: bool, +} + +fn remove_file_action(path: &str) -> serde_json::Result { + serde_json::to_value(RemoveActionFixture { + remove: RemoveFileFixture { + path: path.to_string(), + deletion_timestamp: 1700000001000, + data_change: true, + extended_file_metadata: false, + }, + }) +} diff --git a/kernel/tests/integration/features/alter_table.rs b/kernel/tests/integration/features/alter_table.rs new file mode 100644 index 0000000000..b3590b1078 --- /dev/null +++ b/kernel/tests/integration/features/alter_table.rs @@ -0,0 +1,1290 @@ +//! Integration tests for ALTER TABLE schema evolution. + +use std::collections::{HashMap, HashSet}; +use std::sync::Arc; + +use delta_kernel::arrow::array::{Array, Int32Array, StringArray}; +use delta_kernel::arrow::record_batch::RecordBatch; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::engine::arrow_conversion::TryIntoArrow as _; +use delta_kernel::expressions::{column_name, ColumnName, Scalar}; +use delta_kernel::schema::{ + ArrayType, ColumnMetadataKey, DataType, MapType, MetadataValue, SchemaRef, StructField, + StructType, +}; +use delta_kernel::snapshot::Snapshot; +use delta_kernel::table_features::ColumnMappingMode; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::transaction::data_layout::DataLayout; +use delta_kernel::DeltaResult; +use rstest::rstest; +use test_utils::{ + add_commit, column_mapping_fixtures as fixtures, create_table as create_test_table, + create_table_and_load_snapshot, engine_store_setup, test_table_setup, test_table_setup_mt, + write_batch_to_table, +}; + +fn simple_schema() -> SchemaRef { + Arc::new( + StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("name", DataType::STRING), + ]) + .unwrap(), + ) +} + +fn committer() -> Box { + Box::new(FileSystemCommitter::new()) +} + +/// Reads `delta.columnMapping.maxColumnId` from the snapshot's metadata. Returns +/// `None` when the property is absent (e.g. non-CM tables) so callers can compare it +/// directly against `cm_mode.map(...)`-style expectations. CM-only callers should +/// `.expect(...)` at the use site to surface the protocol violation explicitly. +fn max_column_id(snap: &Snapshot) -> Option { + snap.table_configuration() + .metadata() + .configuration() + .get("delta.columnMapping.maxColumnId") + .and_then(|v| v.parse().ok()) +} + +// ============================================================================ +// Add column tests +// ============================================================================ + +/// End-to-end lifecycle: write, ALTER to add columns, scan, write populated rows, scan again. +/// Each column is added in its own alter commit with a checkpoint after, exercising +/// "do some ops -> checkpoint -> do more ops -> checkpoint". Under CM, also verifies fresh +/// ids/physical names and that `maxColumnId` advanced. +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn add_columns_lifecycle( + #[values(None, Some("name"), Some("id"))] cm_mode: Option<&str>, + #[values(1, 3)] num_columns: usize, +) -> Result<(), Box> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let properties: Vec<(&str, &str)> = cm_mode + .map(|m| vec![("delta.columnMapping.mode", m)]) + .unwrap_or_default(); + let snapshot = + create_table_and_load_snapshot(&table_path, simple_schema(), engine.as_ref(), &properties)?; + let original_max_id = + cm_mode.map(|_| max_column_id(&snapshot).expect("CM table must have maxColumnId")); + + // Write two rows with only the original columns populated. + let batch = RecordBatch::try_new( + Arc::new(simple_schema().as_ref().try_into_arrow().unwrap()), + vec![ + Arc::new(Int32Array::from(vec![1, 2])), + Arc::new(StringArray::from(vec!["a", "b"])), + ], + ) + .unwrap(); + let snapshot = write_batch_to_table(&snapshot, engine.as_ref(), batch, HashMap::new()).await?; + + let new_col_names: Vec = (0..num_columns).map(|i| format!("col_{i}")).collect(); + + // One alter+checkpoint cycle per column. + let mut current = snapshot; + for name in &new_col_names { + let committed = current + .alter_table() + .add_column(StructField::nullable(name, DataType::STRING)) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_committed(); + let post = committed + .post_commit_snapshot() + .expect("post-commit snapshot"); + let (_, ckpt) = post.clone().checkpoint(engine.as_ref(), None)?; + current = ckpt; + } + + // Reload from storage to verify persistence. v0 = create, v1 = write, then `num_columns` + // alter commits. + let alter_end_version = 1 + num_columns as u64; + let reloaded = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!(reloaded.version(), alter_end_version); + let schema = reloaded.schema(); + assert_eq!(schema.fields().count(), 2 + num_columns); + for name in &new_col_names { + let field = schema.field(name).expect("added field should exist"); + assert_eq!(field.data_type(), &DataType::STRING); + assert!(field.is_nullable()); + } + + // When CM is enabled: each new column must have a fresh id/physical name, and the + // table's maxColumnId must have advanced past the original value. When CM is disabled: + // the property must remain absent. + if let Some(orig) = original_max_id { + for name in &new_col_names { + let field = schema.field(name).unwrap(); + let cm_id = field.column_mapping_id().expect("CM id should be assigned"); + assert!( + cm_id > orig, + "new column '{name}' id {cm_id} must exceed original max {orig}" + ); + match field + .get_config_value(&ColumnMetadataKey::ColumnMappingPhysicalName) + .expect("physical name should be assigned") + { + MetadataValue::String(s) => assert!(s.starts_with("col-")), + other => panic!("expected String, got {other:?}"), + } + } + assert!(max_column_id(&reloaded).expect("CM table must have maxColumnId") > orig); + } else { + assert!(reloaded + .table_configuration() + .metadata() + .configuration() + .get("delta.columnMapping.maxColumnId") + .is_none()); + } + + // Scan back -- old rows should have NULL for every new column. + let evolved_arrow_schema: delta_kernel::arrow::datatypes::SchemaRef = + Arc::new(reloaded.schema().as_ref().try_into_arrow().unwrap()); + let scan = reloaded.scan_builder().build()?; + let batches = test_utils::read_scan(&scan, engine.clone())?; + assert!(!batches.is_empty()); + let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total_rows, 2); + assert_eq!(batches[0].num_columns(), 2 + num_columns); + for name in &new_col_names { + let col = batches[0].column_by_name(name).expect("new column"); + assert_eq!(col.null_count(), col.len()); + } + + // Write two more rows with all columns populated. + let reloaded = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let mut new_arrays: Vec> = vec![ + Arc::new(Int32Array::from(vec![3, 4])), + Arc::new(StringArray::from(vec!["c", "d"])), + ]; + for _ in 0..num_columns { + new_arrays.push(Arc::new(StringArray::from(vec!["new_c", "new_d"]))); + } + let batch2 = RecordBatch::try_new(evolved_arrow_schema, new_arrays).unwrap(); + let reloaded = Arc::new(reloaded); + let _ = write_batch_to_table(&reloaded, engine.as_ref(), batch2, HashMap::new()).await?; + + // Scan back -- 4 rows total, each new column has 2 NULLs (old rows) and 2 values (new rows). + let final_snap = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!(final_snap.version(), alter_end_version + 1); + let scan = final_snap.scan_builder().build()?; + let batches = test_utils::read_scan(&scan, engine.clone())?; + let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total_rows, 4); + for name in &new_col_names { + let null_count: usize = batches + .iter() + .map(|b| b.column_by_name(name).expect("new column").null_count()) + .sum(); + assert_eq!(null_count, 2, "column {name} should have 2 NULLs"); + } + + Ok(()) +} + +/// Adding columns of complex types (struct, array, map) -- with and without column mapping. +/// Verifies the data type round-trips and, under CM, that every reachable struct field +/// receives a distinct fresh ID and `maxColumnId` advances accordingly. `expected_id_count` +/// is the number of CM IDs the column should receive (1 for the parent + however many inner +/// struct fields the recursion reaches). +#[rstest] +#[case::struct_column( + StructField::nullable( + "address", + StructType::try_new(vec![ + StructField::nullable("city", DataType::STRING), + StructField::nullable("zip", DataType::STRING), + ]).unwrap(), + ), + 3, +)] +#[case::array_of_primitive( + StructField::nullable("tags", ArrayType::new(DataType::STRING, true)), + 1 +)] +#[case::map_of_primitives( + StructField::nullable("labels", MapType::new(DataType::STRING, DataType::INTEGER, true)), + 1 +)] +#[case::array_of_struct( + StructField::nullable( + "items", + ArrayType::new( + StructType::try_new(vec![ + StructField::nullable("a", DataType::STRING), + StructField::nullable("b", DataType::INTEGER), + ]) + .unwrap(), + true, + ), + ), + 3, +)] +#[case::map_value_is_struct( + StructField::nullable( + "by_id", + MapType::new( + DataType::STRING, + StructType::try_new(vec![ + StructField::nullable("a", DataType::STRING), + StructField::nullable("b", DataType::INTEGER), + ]) + .unwrap(), + true, + ), + ), + 3, +)] +#[case::map_key_is_struct( + StructField::nullable( + "lookup", + MapType::new( + StructType::try_new(vec![ + StructField::nullable("a", DataType::STRING), + StructField::nullable("b", DataType::INTEGER), + ]) + .unwrap(), + DataType::INTEGER, + true, + ), + ), + 3, +)] +#[tokio::test] +async fn add_complex_type_column( + #[case] field: StructField, + #[case] expected_id_count: usize, + #[values(None, Some("name"), Some("id"))] cm_mode: Option<&str>, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let properties: Vec<(&str, &str)> = cm_mode + .map(|m| vec![("delta.columnMapping.mode", m)]) + .unwrap_or_default(); + let snapshot = + create_table_and_load_snapshot(&table_path, simple_schema(), engine.as_ref(), &properties)?; + let original_max_id = + cm_mode.map(|_| max_column_id(&snapshot).expect("CM table must have maxColumnId")); + + let field_name = field.name().to_string(); + let expected_type = field.data_type().clone(); + + snapshot + .alter_table() + .add_column(field) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_committed(); + + let reloaded = Snapshot::builder_for(table_path).build(engine.as_ref())?; + let schema = reloaded.schema(); + let added = schema.field(&field_name).expect("added field should exist"); + + if let Some(orig_max) = original_max_id { + // Under CM, inner struct fields carry CM metadata that `expected_type` doesn't; + // strict DataType equality won't hold. The ID-count check below implicitly verifies + // that the type structure round-tripped correctly. + let ids = added.collect_column_mapping_ids(); + let unique: HashSet<_> = ids.iter().copied().collect(); + assert_eq!(ids.len(), expected_id_count, "expected ID count mismatch"); + assert_eq!(unique.len(), ids.len(), "all assigned IDs must be distinct"); + assert!( + ids.iter().all(|&id| id > orig_max), + "all assigned IDs must exceed original max" + ); + assert_eq!( + max_column_id(&reloaded).expect("CM table must have maxColumnId"), + ids.iter().copied().max().unwrap(), + "table maxColumnId must equal the largest assigned ID", + ); + } else { + assert_eq!(added.data_type(), &expected_type); + } + Ok(()) +} + +#[rstest] +#[case::duplicate_column(&[], StructField::nullable("name", DataType::STRING), "already exists")] +#[case::duplicate_column_case_insensitive( + &[], + StructField::nullable("NAME", DataType::STRING), + "already exists" +)] +#[case::timestamp_ntz_without_feature( + &[], + StructField::nullable("ts", DataType::TIMESTAMP_NTZ), + "timestampNtz" +)] +#[case::non_nullable(&[], StructField::not_null("age", DataType::INTEGER), "non-nullable")] +#[tokio::test] +async fn add_column_failures( + #[case] properties: &[(&str, &str)], + #[case] field: StructField, + #[case] error_contains: &str, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let snapshot = + create_table_and_load_snapshot(&table_path, simple_schema(), engine.as_ref(), properties)?; + + let err = snapshot + .alter_table() + .add_column(field) + .build(engine.as_ref(), committer()); + assert!(err.is_err()); + assert!(err.unwrap_err().to_string().contains(error_contains)); + + Ok(()) +} + +/// Back-to-back alters with a checkpoint in between, then a write against the evolved schema. +/// Exercises: create (v0) → alter add A (v1) → checkpoint at v1 → alter add B (v2) → write +/// row with values in both new columns (v3) → reload. The reload must rebuild the snapshot +/// from the checkpoint + alter commits + data commit and return the written values. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn back_to_back_alters_with_checkpoint() -> Result<(), Box> { + // Checkpoint writing requires the multi-threaded engine (like `maintenance_ops.rs`). + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + + // v0: create. + let snapshot = + create_table_and_load_snapshot(&table_path, simple_schema(), engine.as_ref(), &[])?; + + // v1: add column "a". + let v1 = snapshot + .alter_table() + .add_column(StructField::nullable("a", DataType::STRING)) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_committed(); + let v1_snap = v1 + .post_commit_snapshot() + .expect("post-commit snapshot at v1"); + + // Checkpoint at v1. + let (_, v1_ckpt) = v1_snap.clone().checkpoint(engine.as_ref(), None)?; + + // v2: add column "b" on top of the checkpointed snapshot. + let v2 = v1_ckpt + .alter_table() + .add_column(StructField::nullable("b", DataType::INTEGER)) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_committed(); + let v2_snap = v2 + .post_commit_snapshot() + .expect("post-commit snapshot at v2"); + + // v3: write one row populating both new columns. + let evolved_arrow_schema: delta_kernel::arrow::datatypes::SchemaRef = + Arc::new(v2_snap.schema().as_ref().try_into_arrow().unwrap()); + let batch = RecordBatch::try_new( + evolved_arrow_schema, + vec![ + Arc::new(Int32Array::from(vec![1])), + Arc::new(StringArray::from(vec!["alice"])), + Arc::new(StringArray::from(vec!["val_a"])), + Arc::new(delta_kernel::arrow::array::Int32Array::from(vec![100])), + ], + ) + .unwrap(); + write_batch_to_table(v2_snap, engine.as_ref(), batch, HashMap::new()).await?; + + // Reload from scratch: kernel must rebuild from checkpoint + alter commits + data commit. + let reloaded = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!(reloaded.version(), 3); + let schema = reloaded.schema(); + assert!( + schema.field("a").is_some(), + "column added at v1 must survive checkpoint" + ); + assert!( + schema.field("b").is_some(), + "column added at v2 must be present" + ); + + // Scan the row back and verify values in both newly-added columns. + let scan = reloaded.scan_builder().build()?; + let batches = test_utils::read_scan(&scan, engine.clone())?; + let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total_rows, 1); + let a_col = batches[0] + .column_by_name("a") + .expect("column a") + .as_any() + .downcast_ref::() + .expect("a is string"); + assert_eq!(a_col.value(0), "val_a"); + let b_col = batches[0] + .column_by_name("b") + .expect("column b") + .as_any() + .downcast_ref::() + .expect("b is int"); + assert_eq!(b_col.value(0), 100); + + Ok(()) +} + +/// Empty-schema tables are valid intermediate state, so they need to behave normally +/// once a column is added. This test runs the full lifecycle (create empty, ALTER ADD +/// COLUMN, write rows, scan them back, time-travel to v0) across all column-mapping +/// modes so the CM bookkeeping (maxColumnId, column-mapping id, physical name) is +/// exercised on a schema that started empty rather than being created with columns +/// up front. +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn empty_create_then_add_column( + #[values(None, Some("name"), Some("id"))] cm_mode: Option<&str>, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let properties: Vec<(&str, &str)> = cm_mode + .map(|m| vec![("delta.columnMapping.mode", m)]) + .unwrap_or_default(); + + let empty_schema = Arc::new(StructType::try_new(vec![])?); + let v0 = + create_table_and_load_snapshot(&table_path, empty_schema, engine.as_ref(), &properties)?; + assert_eq!(v0.version(), 0); + assert_eq!(v0.schema().num_fields(), 0); + assert_eq!(max_column_id(&v0), cm_mode.map(|_| 0)); + + // Scans and blind appends against the empty-schema snapshot are blocked with + // friendly errors that point users at ALTER TABLE ADD COLUMN. + let scan_err = v0 + .clone() + .scan_builder() + .build() + .expect_err("scan_builder().build() must reject empty-schema snapshots"); + assert!( + scan_err.to_string().contains("empty schema") + && scan_err.to_string().contains("ALTER TABLE ADD COLUMN"), + "scan error must point at ALTER TABLE ADD COLUMN, got: {scan_err}" + ); + let write_err = v0 + .clone() + .transaction(committer(), engine.as_ref())? + .with_engine_info("EmptySchemaApp/0.1.0") + .unpartitioned_write_context() + .expect_err("unpartitioned_write_context() must reject empty-schema snapshots"); + assert!( + write_err.to_string().contains("empty schema") + && write_err.to_string().contains("alter_table"), + "write_context error must point at alter_table, got: {write_err}" + ); + + v0.alter_table() + .add_column(StructField::nullable("id", DataType::INTEGER)) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_committed(); + + let v1 = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!(v1.version(), 1); + let schema = v1.schema(); + assert_eq!(schema.num_fields(), 1); + let id_field = schema.field("id").expect("added field should exist"); + assert_eq!(id_field.data_type(), &DataType::INTEGER); + assert!(id_field.is_nullable()); + assert_eq!(max_column_id(&v1), cm_mode.map(|_| 1)); + assert_eq!(id_field.column_mapping_id(), cm_mode.map(|_| 1i64)); + assert_eq!( + id_field + .get_config_value(&ColumnMetadataKey::ColumnMappingPhysicalName) + .is_some(), + cm_mode.is_some(), + "physical name presence must track column-mapping enablement", + ); + + // Write rows through the freshly-added column and confirm the read path round-trips + // them. Under CM this also exercises the physical-name plumbing for a column whose + // schema started empty. + let arrow_schema = v1.schema().as_ref().try_into_arrow().unwrap(); + let batch = RecordBatch::try_new( + Arc::new(arrow_schema), + vec![Arc::new(Int32Array::from(vec![Some(1), None, Some(99)]))], + ) + .unwrap(); + let v2 = write_batch_to_table(&v1, engine.as_ref(), batch, HashMap::new()) + .await + .map_err(|e| delta_kernel::Error::generic(format!("write_batch_to_table failed: {e}")))?; + assert_eq!(v2.version(), 2); + + let scan = v2.scan_builder().build()?; + let batches = test_utils::read_scan(&scan, engine.clone())?; + let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!( + total_rows, 3, + "post-ALTER write should yield 3 rows on read" + ); + + // The empty-schema snapshot must still be loadable via time-travel after later + // commits introduce a column and data files; pins the contract that the v0 history + // survives schema evolution. + let v0_after = Snapshot::builder_for(&table_path) + .at_version(0) + .build(engine.as_ref())?; + assert_eq!(v0_after.version(), 0); + assert_eq!(v0_after.schema().num_fields(), 0); + + Ok(()) +} + +// ============================================================================ +// SET NULLABLE tests +// ============================================================================ + +/// Cross-product: 3 schema/column cases x 3 CM modes (off, name, id). +#[rstest] +#[case::already_nullable(simple_schema(), column_name!("name"))] +#[case::required_top_level( + Arc::new(StructType::try_new(vec![ + StructField::not_null("id", DataType::INTEGER), + StructField::nullable("name", DataType::STRING), + ]).unwrap()), + column_name!("id") +)] +#[case::required_nested( + Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable( + "address", + StructType::try_new(vec![ + StructField::not_null("city", DataType::STRING), + StructField::nullable("zip", DataType::STRING), + ]).unwrap(), + ), + ]).unwrap()), + column_name!("address.city") +)] +#[tokio::test] +async fn set_nullable_succeeds( + #[case] schema: SchemaRef, + #[case] column: ColumnName, + #[values(None, Some("name"), Some("id"))] cm_mode: Option<&str>, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let properties: Vec<(&str, &str)> = cm_mode + .map(|m| vec![("delta.columnMapping.mode", m)]) + .unwrap_or_default(); + let snapshot = + create_table_and_load_snapshot(&table_path, schema, engine.as_ref(), &properties)?; + // Snapshot the field before the alter so we can prove set_nullable changes only the + // nullable bit -- preserving name, data type, and ALL metadata (including column-mapping + // id and physical name when CM is enabled). + let before = snapshot.schema().field_at_path(column.path()).clone(); + + snapshot + .alter_table() + .set_nullable(column.clone()) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_committed(); + + let reloaded = Snapshot::builder_for(table_path).build(engine.as_ref())?; + let reloaded_schema = reloaded.schema(); + let after = reloaded_schema.field_at_path(column.path()); + assert!(after.is_nullable()); + assert_eq!(after.name(), before.name()); + assert_eq!(after.data_type(), before.data_type()); + assert_eq!( + after.metadata(), + before.metadata(), + "field metadata (incl. column mapping id/physical name) must be preserved" + ); + Ok(()) +} + +/// End-to-end: create a table with a non-null layout column (partition or clustering), +/// write a row, flip the layout column to nullable, checkpoint, reload from scratch, scan. +/// Cross-product: layout kind (partitioned, clustered) x column-mapping mode (off, name, id). +#[rstest] +#[case::partition("date", DataLayout::partitioned(["date"]), "2026-01-01")] +#[case::clustered("region", DataLayout::clustered(["region"]), "us")] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn set_nullable_on_layout_column_with_checkpoint( + #[case] col_name: &str, + #[case] layout: DataLayout, + #[case] col_value: &str, + #[values(None, Some("name"), Some("id"))] cm_mode: Option<&str>, +) -> Result<(), Box> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + // Partition values live in the directory path; clustering values live in the row batch. + let is_partitioned = matches!(layout, DataLayout::Partitioned { .. }); + + // v0: create the table with the layout column as non-null. + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::not_null(col_name, DataType::STRING), + ])?); + let properties: Vec<(&str, &str)> = cm_mode + .map(|m| vec![("delta.columnMapping.mode", m)]) + .unwrap_or_default(); + create_table(&table_path, schema.clone(), "Test/1.0") + .with_data_layout(layout) + .with_table_properties(properties) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_committed(); + let v0 = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert!(!v0.schema().field(col_name).unwrap().is_nullable()); + + // v1: write a single row. + let v0_arc = Arc::new(v0); + let v1 = if is_partitioned { + // Partition cols are excluded from the row batch and passed via partition_values. + let nonpartition_arrow_schema: delta_kernel::arrow::datatypes::SchemaRef = + Arc::new(delta_kernel::arrow::datatypes::Schema::new(vec![ + delta_kernel::arrow::datatypes::Field::new( + "id", + delta_kernel::arrow::datatypes::DataType::Int32, + true, + ), + ])); + let batch = RecordBatch::try_new( + nonpartition_arrow_schema, + vec![Arc::new(Int32Array::from(vec![1]))], + )?; + let mut partition_values = HashMap::new(); + partition_values.insert(col_name.to_string(), Scalar::String(col_value.to_string())); + write_batch_to_table(&v0_arc, engine.as_ref(), batch, partition_values).await? + } else { + // Clustering cols are regular columns; partition_values is empty. + let arrow_schema: delta_kernel::arrow::datatypes::SchemaRef = + Arc::new(schema.as_ref().try_into_arrow().unwrap()); + let batch = RecordBatch::try_new( + arrow_schema, + vec![ + Arc::new(Int32Array::from(vec![1])), + Arc::new(StringArray::from(vec![col_value])), + ], + )?; + write_batch_to_table(&v0_arc, engine.as_ref(), batch, HashMap::new()).await? + }; + + // v2: ALTER TABLE -- set the layout column nullable. + let v2 = v1 + .alter_table() + .set_nullable(ColumnName::new([col_name])) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_committed(); + let v2_snap = v2 + .post_commit_snapshot() + .expect("post-commit snapshot at v2"); + assert!(v2_snap.schema().field(col_name).unwrap().is_nullable()); + + // Checkpoint at v2 so reload exercises the checkpoint path. + v2_snap.clone().checkpoint(engine.as_ref(), None)?; + + // Reload from scratch and verify the schema and row survive. + let reloaded = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!(reloaded.version(), 2); + assert!(reloaded.schema().field(col_name).unwrap().is_nullable()); + + let scan = reloaded.scan_builder().build()?; + let batches = test_utils::read_scan(&scan, engine.clone())?; + let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total_rows, 1); + let col = batches[0] + .column_by_name(col_name) + .expect("layout column") + .as_any() + .downcast_ref::() + .expect("layout column is string"); + assert_eq!(col.value(0), col_value); + Ok(()) +} + +#[tokio::test] +async fn set_nullable_nonexistent_column_fails() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let snapshot = + create_table_and_load_snapshot(&table_path, simple_schema(), engine.as_ref(), &[])?; + + let err = snapshot + .alter_table() + .set_nullable(column_name!("nonexistent")) + .build(engine.as_ref(), committer()); + assert!(err.is_err()); + assert!(err.unwrap_err().to_string().contains("does not exist")); + + Ok(()) +} + +// ============================================================================ +// CHAIN tests +// ============================================================================ + +/// Alternating chain: ADD COLUMN, SET NULLABLE, ADD COLUMN, SET NULLABLE. Verifies that +/// chaining mixed ops applies them in order and produces the expected final schema. Each +/// SET NULLABLE flips a still-NOT-NULL column from the original schema. Under CM, also +/// verifies existing fields' column mapping IDs are preserved by set_nullable while +/// add_column receives a new CM ID and bumps maxColumnId. +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn chain_add_column_and_set_nullable( + #[values(None, Some("name"), Some("id"))] cm_mode: Option<&str>, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let schema = Arc::new(StructType::try_new(vec![ + StructField::not_null("id", DataType::INTEGER), + StructField::not_null("name", DataType::STRING), + ])?); + let properties: Vec<(&str, &str)> = cm_mode + .map(|m| vec![("delta.columnMapping.mode", m)]) + .unwrap_or_default(); + let snapshot = + create_table_and_load_snapshot(&table_path, schema, engine.as_ref(), &properties)?; + + let original_id_cm_id = cm_mode.map(|_| { + snapshot + .schema() + .field("id") + .unwrap() + .column_mapping_id() + .expect("existing field should already have a column mapping ID") + }); + let original_max_id = + cm_mode.map(|_| max_column_id(&snapshot).expect("CM table must have maxColumnId")); + + // Two alter+checkpoint cycles: (add email + nullable id), (add age + nullable name). + let v1 = snapshot + .alter_table() + .add_column(StructField::nullable("email", DataType::STRING)) + .set_nullable(column_name!("id")) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_committed(); + let v1_snap = v1 + .post_commit_snapshot() + .expect("post-commit snapshot at v1"); + let (_, v1_ckpt) = v1_snap.clone().checkpoint(engine.as_ref(), None)?; + let v2 = v1_ckpt + .alter_table() + .add_column(StructField::nullable("age", DataType::INTEGER)) + .set_nullable(column_name!("name")) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_committed(); + let v2_snap = v2 + .post_commit_snapshot() + .expect("post-commit snapshot at v2"); + v2_snap.clone().checkpoint(engine.as_ref(), None)?; + + let reloaded = Snapshot::builder_for(table_path).build(engine.as_ref())?; + let schema = reloaded.schema(); + assert_eq!(schema.fields().count(), 4); + for name in ["id", "name", "email", "age"] { + let field = schema.field(name).expect("field should be present"); + assert!(field.is_nullable(), "field '{name}' should be nullable"); + } + + if let (Some(orig_id), Some(orig_max)) = (original_id_cm_id, original_max_id) { + for added in ["email", "age"] { + assert!( + schema.field(added).unwrap().column_mapping_id().is_some(), + "added field '{added}' should have a column mapping ID" + ); + } + let id_after = schema + .field("id") + .unwrap() + .column_mapping_id() + .expect("existing id column mapping"); + assert_eq!(id_after, orig_id, "existing id CM id must not change"); + assert!( + max_column_id(&reloaded).expect("CM table must have maxColumnId") > orig_max, + "chained add_column must bump maxColumnId" + ); + } + + Ok(()) +} + +fn field_with_stray_key(name: &str, key: &ColumnMetadataKey, ty: DataType) -> StructField { + let mut f = StructField::nullable(name, ty); + f.metadata + .insert(key.as_ref().to_string(), MetadataValue::Number(99)); + f +} + +/// On a clean non-CM table, an ALTER that adds a column carrying stray CM metadata has that +/// metadata stripped (the commit introduces it into a previously-clean table), rather than +/// rejected -- matching delta-spark. Parametrized over each detected key and over placement +/// (top-level vs nested in a struct). +#[rstest] +#[case::top_level(false)] +#[case::nested_in_struct(true)] +#[tokio::test] +async fn add_column_with_stray_cm_metadata_on_non_cm_table_is_stripped( + #[case] nested: bool, + #[values( + ColumnMetadataKey::ColumnMappingId, + ColumnMetadataKey::ColumnMappingPhysicalName, + ColumnMetadataKey::ColumnMappingNestedIds, + ColumnMetadataKey::ParquetFieldId, + ColumnMetadataKey::ParquetFieldNestedIds + )] + key: ColumnMetadataKey, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let snapshot = + create_table_and_load_snapshot(&table_path, simple_schema(), engine.as_ref(), &[])?; + + let (field, stripped_path): (StructField, Vec) = if nested { + let outer = StructField::nullable( + "outer", + StructType::try_new(vec![field_with_stray_key("inner", &key, DataType::STRING)])?, + ); + (outer, vec!["outer".to_string(), "inner".to_string()]) + } else { + ( + field_with_stray_key("tainted", &key, DataType::STRING), + vec!["tainted".to_string()], + ) + }; + + snapshot + .alter_table() + .add_column(field) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_committed(); + + // Reload from disk so we assert on the persisted schemaString, not the in-memory config. + let reloaded = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let reloaded_schema = reloaded.schema(); + let leaf = reloaded_schema.field_at_path(&stripped_path); + assert!( + leaf.get_config_value(&key).is_none(), + "stray {} at {stripped_path:?} should be stripped", + key.as_ref() + ); + Ok(()) +} + +/// The ALTER strip is `None`-mode-only. Adding a column with pre-populated column-mapping metadata +/// preserves it when mapping is enabled (`id` / `name`, delta-spark's +/// `assignColumnIdAndPhysicalName` keeps existing ids) and strips it only in `None` mode. +#[rstest] +#[case::none(ColumnMappingMode::None, &[], false)] +#[case::id(ColumnMappingMode::Id, &[("delta.columnMapping.mode", "id")], true)] +#[case::name(ColumnMappingMode::Name, &[("delta.columnMapping.mode", "name")], true)] +#[tokio::test] +async fn add_column_strip_is_none_mode_only( + #[case] expected_mode: ColumnMappingMode, + #[case] properties: &[(&str, &str)], + #[case] annotation_kept: bool, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let snapshot = + create_table_and_load_snapshot(&table_path, simple_schema(), engine.as_ref(), properties)?; + assert_eq!( + snapshot.table_configuration().column_mapping_mode(), + expected_mode + ); + + // A well-formed id+physicalName pair so enabled modes have valid metadata to preserve. + let field = fixtures::cm_field("added", 99, "phys-added", DataType::STRING); + snapshot + .alter_table() + .add_column(field) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_committed(); + + let reloaded = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let added = reloaded.schema().field("added").unwrap().clone(); + assert_eq!( + added.column_mapping_id().is_some(), + annotation_kept, + "column mapping id under {expected_mode:?} mode" + ); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn alter_blocked_when_iceberg_compat_v3_enabled() -> Result<(), Box> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let snapshot = create_table_and_load_snapshot( + &table_path, + simple_schema(), + engine.as_ref(), + &[("delta.enableIcebergCompatV3", "true")], + )?; + + let msg = snapshot + .alter_table() + .add_column(StructField::nullable("new_col", DataType::STRING)) + .build(engine.as_ref(), committer()) + .unwrap_err() + .to_string(); + assert!( + msg.contains("ALTER TABLE is not yet supported on tables with icebergCompatV3 enabled"), + "unexpected error: {msg}", + ); + + Ok(()) +} + +#[tokio::test] +async fn add_column_with_orphan_default_metadata_succeeds() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let snapshot = + create_table_and_load_snapshot(&table_path, simple_schema(), engine.as_ref(), &[])?; + let field = StructField::nullable("new_col", DataType::INTEGER).add_metadata([( + ColumnMetadataKey::CurrentDefault.as_ref().to_string(), + MetadataValue::String("42".to_string()), + )]); + + snapshot + .alter_table() + .add_column(field) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_committed(); + + let reloaded = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let reloaded_schema = reloaded.schema(); + let default = reloaded_schema + .field("new_col") + .expect("new_col must exist after ALTER") + .column_default()? + .expect("CURRENT_DEFAULT metadata must survive ALTER"); + assert_eq!(default.raw_sql(), "42"); + + let txn = reloaded.transaction(committer(), engine.as_ref())?; + assert!( + txn.top_level_column_defaults()?.is_empty(), + "default metadata must remain inert without allowColumnDefaults", + ); + + Ok(()) +} + +#[tokio::test] +async fn alter_blocked_when_allow_column_defaults_enabled() -> Result<(), Box> +{ + let (store, engine, table_url) = engine_store_setup("alter_column_defaults", None); + let table_url = create_test_table( + store, + table_url, + simple_schema(), + &[], + true, + vec![], + vec!["allowColumnDefaults"], + ) + .await?; + let snapshot = Snapshot::builder_for(table_url).build(&engine)?; + + let msg = snapshot + .alter_table() + .add_column(StructField::nullable("new_col", DataType::STRING)) + .build(&engine, committer()) + .unwrap_err() + .to_string(); + assert!( + msg.contains("ALTER TABLE is not yet supported on tables with allowColumnDefaults enabled"), + "unexpected error: {msg}", + ); + + Ok(()) +} + +// ============================================================================ +// ALTER TABLE ADD COLUMN preserves / fills pre-populated column mapping metadata +// (delta-spark parity per `DeltaColumnMapping.assignColumnIdAndPhysicalName`). +// See https://github.com/delta-io/delta-kernel-rs/issues/2377. +// ============================================================================ + +fn cm_id_for_field(field: &StructField) -> i64 { + field + .column_mapping_id() + .expect("field must have a column mapping id") +} + +fn physical_name_for_field(field: &StructField) -> &str { + match field.get_config_value(&ColumnMetadataKey::ColumnMappingPhysicalName) { + Some(MetadataValue::String(s)) => s.as_str(), + other => panic!("expected physicalName string, got {other:?}"), + } +} + +/// ADD COLUMN with both `delta.columnMapping.id` and `delta.columnMapping.physicalName` +/// pre-populated: the connector-supplied metadata is preserved verbatim. `maxColumnId` +/// advances to the supplied id when it exceeds the existing max. +#[rstest] +#[tokio::test] +async fn add_column_preserves_complete_cm_metadata( + #[values("name", "id")] cm_mode: &str, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let snapshot = create_table_and_load_snapshot( + &table_path, + simple_schema(), + engine.as_ref(), + &[("delta.columnMapping.mode", cm_mode)], + )?; + let original_max = max_column_id(&snapshot).expect("CM table must have maxColumnId"); + + // Supplied id is well above the table's max so we can verify maxColumnId follows it. + let supplied_id = original_max + 100; + let field = fixtures::cm_field( + "preserved", + supplied_id, + "user-supplied-physical", + DataType::STRING, + ); + + snapshot + .alter_table() + .add_column(field) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_committed(); + + let reloaded = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let schema = reloaded.schema(); + let added = schema.field("preserved").unwrap(); + assert_eq!(cm_id_for_field(added), supplied_id); + assert_eq!(physical_name_for_field(added), "user-supplied-physical"); + assert_eq!( + max_column_id(&reloaded).expect("CM table must have maxColumnId"), + supplied_id + ); + Ok(()) +} + +/// ADD COLUMN with only `delta.columnMapping.physicalName` supplied: kernel allocates +/// `id = old maxColumnId + 1`, preserves the user-provided physical name, and bumps +/// `maxColumnId` to the new id. +#[rstest] +#[tokio::test] +async fn add_column_with_only_physical_name_allocates_id( + #[values("name", "id")] cm_mode: &str, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let snapshot = create_table_and_load_snapshot( + &table_path, + simple_schema(), + engine.as_ref(), + &[("delta.columnMapping.mode", cm_mode)], + )?; + let original_max = max_column_id(&snapshot).expect("CM table must have maxColumnId"); + + let field = + fixtures::cm_field_physical_name_only("named_only", "phys-named-only", DataType::STRING); + + snapshot + .alter_table() + .add_column(field) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_committed(); + + let reloaded = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let schema = reloaded.schema(); + let added = schema.field("named_only").unwrap(); + assert_eq!(cm_id_for_field(added), original_max + 1); + assert_eq!(physical_name_for_field(added), "phys-named-only"); + assert_eq!( + max_column_id(&reloaded).expect("CM table must have maxColumnId"), + original_max + 1 + ); + Ok(()) +} + +/// ADD COLUMN with only `delta.columnMapping.id` supplied: id is preserved, missing +/// `physicalName` is filled with `col-`. +#[rstest] +#[tokio::test] +async fn add_column_with_only_id_fills_physical_name( + #[values("name", "id")] cm_mode: &str, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let snapshot = create_table_and_load_snapshot( + &table_path, + simple_schema(), + engine.as_ref(), + &[("delta.columnMapping.mode", cm_mode)], + )?; + let original_max = max_column_id(&snapshot).expect("CM table must have maxColumnId"); + let supplied_id = original_max + 7; + + let field = fixtures::cm_field_id_only("id_only", supplied_id, DataType::STRING); + + snapshot + .alter_table() + .add_column(field) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_committed(); + + let reloaded = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let schema = reloaded.schema(); + let added = schema.field("id_only").unwrap(); + assert_eq!(cm_id_for_field(added), supplied_id); + assert!( + physical_name_for_field(added).starts_with("col-"), + "physical name should be filled with col-, got {}", + physical_name_for_field(added) + ); + assert_eq!( + max_column_id(&reloaded).expect("CM table must have maxColumnId"), + supplied_id + ); + Ok(()) +} + +/// ADD COLUMN where the supplied `id` is *less than* the existing `maxColumnId` but does +/// not collide with any existing field's id: succeeds, with the supplied id preserved +/// verbatim and `maxColumnId` unchanged. Matches delta-spark; diverges from the Java Kernel +/// proposal in https://github.com/delta-io/delta/pull/4520, which would reject this. +#[tokio::test] +async fn add_column_with_id_below_max_column_id_succeeds() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // Pre-populate the table with sparse ids (1, 100) using the create-table preserve path. + let schema = Arc::new(StructType::try_new(vec![ + fixtures::cm_field("a", 1, "phys-a", DataType::INTEGER), + fixtures::cm_field("b", 100, "phys-b", DataType::STRING), + ])?); + let snapshot = create_table_and_load_snapshot( + &table_path, + schema, + engine.as_ref(), + &[("delta.columnMapping.mode", "name")], + )?; + assert_eq!( + max_column_id(&snapshot).expect("CM table must have maxColumnId"), + 100 + ); + + // Now add a new column with id=50, which is well below maxColumnId=100 and not used. + let field = fixtures::cm_field("inserted_below_max", 50, "phys-inserted", DataType::STRING); + + snapshot + .alter_table() + .add_column(field) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_committed(); + + let reloaded = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let schema = reloaded.schema(); + let added = schema.field("inserted_below_max").unwrap(); + assert_eq!(cm_id_for_field(added), 50); + assert_eq!(physical_name_for_field(added), "phys-inserted"); + // maxColumnId stays at 100 because the supplied id (50) didn't exceed it. + assert_eq!( + max_column_id(&reloaded).expect("CM table must have maxColumnId"), + 100 + ); + Ok(()) +} + +/// ADD COLUMN where the supplied `id` collides with an existing field's id: fails. The +/// duplicate-id check happens when the alter builder constructs the new +/// `TableConfiguration` via `make_physical`. +#[tokio::test] +async fn add_column_with_id_colliding_existing_field_is_rejected() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let snapshot = create_table_and_load_snapshot( + &table_path, + simple_schema(), + engine.as_ref(), + &[("delta.columnMapping.mode", "name")], + )?; + + // Pick an id that already exists in the simple_schema (1, 2 typically). + let existing_id = snapshot + .schema() + .field("id") + .unwrap() + .column_mapping_id() + .expect("simple_schema 'id' must have a CM id under name mode"); + + let field = fixtures::cm_field("colliding", existing_id, "phys-colliding", DataType::STRING); + + let err = snapshot + .alter_table() + .add_column(field) + .build(engine.as_ref(), committer()) + .unwrap_err() + .to_string(); + assert!( + err.contains("Duplicate column mapping ID") && err.contains(&existing_id.to_string()), + "expected duplicate-id error naming id {existing_id}, got: {err}" + ); + Ok(()) +} + +/// A mapping-disabled table that already carries residual `delta.columnMapping.*` annotations is +/// left untouched by an ALTER: the pre-existing annotation on `value` survives, and whatever the +/// added column carries is persisted verbatim -- kernel strips only annotations a commit newly +/// introduces into a *clean* table, so an already-dirty table is never rewritten (matching +/// delta-spark). The clean-table strip is covered by +/// `add_column_with_stray_cm_metadata_on_non_cm_table_is_stripped`. +/// +/// Cases: adding a clean column (stays clean) and adding a stray-annotated column (annotation +/// left in place, not stripped). +#[rstest] +#[case::clean_column(StructField::nullable("added", DataType::STRING), None)] +#[case::introduced_stray( + field_with_stray_key("added", &ColumnMetadataKey::ColumnMappingId, DataType::STRING), + Some(99) +)] +#[tokio::test] +async fn add_column_on_stale_table_leaves_schema_untouched( + #[case] added_field: StructField, + #[case] expected_added_cm_id: Option, +) -> DeltaResult<()> { + let (store, engine, table_url) = engine_store_setup("alter_stale_cm", None); + + // `value` carries a stale id; protocol omits columnMapping and no mode is set (resolves to + // None) -- residual annotations already on the table. + let stale_schema = StructType::try_new([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("value", DataType::INTEGER) + .add_metadata([("delta.columnMapping.id", MetadataValue::Number(2))]), + ])?; + let escaped = serde_json::to_string(&serde_json::to_string(&stale_schema)?).unwrap(); + // v0 written directly to bypass create_table validation (which strips stale annotations). + let v0 = format!( + r#"{{"protocol":{{"minReaderVersion":1,"minWriterVersion":2}}}} +{{"metaData":{{"id":"alter-stale-cm","format":{{"provider":"parquet","options":{{}}}},"schemaString":{escaped},"partitionColumns":[],"configuration":{{}},"createdTime":1700000000000}}}} +"# + ); + add_commit(table_url.as_str(), store.as_ref(), 0, v0) + .await + .unwrap(); + + let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; + snapshot + .alter_table() + .add_column(added_field) + .build(&engine, committer())? + .commit(&engine)? + .unwrap_committed(); + + // Reload from disk so we assert on the persisted schemaString, not the in-memory config. + let reloaded = Snapshot::builder_for(table_url).build(&engine)?; + let schema = reloaded.schema(); + + // The pre-existing annotation on the untouched `value` field survives verbatim. + assert_eq!(schema.field("value").unwrap().column_mapping_id(), Some(2)); + // The added column is persisted verbatim -- clean stays clean, stray annotation is left in + // place (not stripped, because the table was already dirty). + assert_eq!( + schema.field("added").unwrap().column_mapping_id(), + expected_added_cm_id + ); + Ok(()) +} diff --git a/kernel/tests/cdf.rs b/kernel/tests/integration/features/cdf.rs similarity index 87% rename from kernel/tests/cdf.rs rename to kernel/tests/integration/features/cdf.rs index 8f6d306ac9..32c43d219e 100644 --- a/kernel/tests/cdf.rs +++ b/kernel/tests/integration/features/cdf.rs @@ -1,28 +1,42 @@ use std::error; +use std::sync::Arc; use delta_kernel::arrow::array::RecordBatch; use delta_kernel::arrow::datatypes::Schema as ArrowSchema; -use delta_kernel::engine::arrow_data::EngineDataArrowExt as _; -use itertools::Itertools; - +use delta_kernel::arrow::util::pretty::pretty_format_batches; use delta_kernel::engine::arrow_conversion::TryFromKernel as _; +use delta_kernel::engine::arrow_data::EngineDataArrowExt as _; +use delta_kernel::expressions::{col, lit, Predicate as Pred}; +use delta_kernel::schema::{DataType, StructField, StructType}; use delta_kernel::table_changes::TableChanges; use delta_kernel::{DeltaResult, Error, PredicateRef, Version}; - -mod common; - -use test_utils::load_test_data; +use itertools::Itertools; +use test_utils::{ + add_commit, create_default_engine, create_default_engine_with_batch, create_table, + engine_store_setup, load_test_data, +}; +use url::Url; fn read_cdf_for_table( test_name: impl AsRef, start_version: Version, end_version: impl Into>, predicate: impl Into>, +) -> DeltaResult> { + read_cdf_for_table_with_batch_size(test_name, start_version, end_version, predicate, None) +} + +fn read_cdf_for_table_with_batch_size( + test_name: impl AsRef, + start_version: Version, + end_version: impl Into>, + predicate: impl Into>, + batch_size: Option, ) -> DeltaResult> { let test_dir = load_test_data("tests/data", test_name.as_ref()).unwrap(); let test_path = test_dir.path().join(test_name.as_ref()); let test_path = delta_kernel::try_parse_uri(test_path.to_str().expect("table path to string"))?; - let engine = test_utils::create_default_engine(&test_path)?; + let engine = create_default_engine_with_batch(&test_path, batch_size)?; let table_changes = TableChanges::try_new( test_path, engine.as_ref(), @@ -51,7 +65,7 @@ fn read_cdf_for_table( .map(|data| -> DeltaResult<_> { let record_batch = data?.try_into_record_batch()?; // Verify that the arrow record batches match the expected schema - assert!(record_batch.schema().as_ref() == &scan_schema_as_arrow); + assert_eq!(record_batch.schema().as_ref(), &scan_schema_as_arrow); Ok(record_batch) }) .try_collect()?; @@ -104,6 +118,31 @@ fn cdf_with_deletion_vector() -> Result<(), Box> { Ok(()) } +#[test] +fn cdf_with_deletion_vector_is_batch_size_invariant() -> Result<(), Box> { + let expected = read_cdf_for_table("cdf-table-with-dv", 0, None, None)?; + for batch_size in [1usize, 2, 3, 4, 7] { + let actual = read_cdf_for_table_with_batch_size( + "cdf-table-with-dv", + 0, + None, + None, + Some(batch_size), + )?; + let expected_rows: usize = expected.iter().map(RecordBatch::num_rows).sum(); + let actual_rows: usize = actual.iter().map(RecordBatch::num_rows).sum(); + assert_eq!( + actual_rows, expected_rows, + "row count changed at batch_size={batch_size}: got {actual_rows}, want {expected_rows}" + ); + let formatted = pretty_format_batches(&expected)?.to_string(); + let mut expected_lines: Vec<&str> = formatted.trim().lines().collect(); + sort_lines!(expected_lines); + assert_batches_sorted_eq!(expected_lines, &actual); + } + Ok(()) +} + #[test] fn basic_cdf() -> Result<(), Box> { let batches = read_cdf_for_table("cdf-table", 0, None, None)?; @@ -584,6 +623,70 @@ fn cdf_with_column_mapping_name_mode() -> Result<(), Box> { Ok(()) } +/// CDF over a commit whose Add carries an extended-year ISO 8601 timestamp in `stats`. +/// The bad `EventTime` cell becomes a per-cell NULL while sibling `UserId` stats survive; +/// the `UserId > 1000` predicate then prunes the file inside the `DataSkippingFilter` in +/// `table_changes` log-replay, so `execute()` never tries to read the phantom parquet +/// file and returns zero rows. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn cdf_per_cell_null_on_malformed_stats() -> Result<(), Box> { + let schema = Arc::new( + StructType::try_new(vec![ + StructField::nullable("EventTime", DataType::TIMESTAMP), + StructField::nullable("UserId", DataType::LONG), + ]) + .unwrap(), + ); + + let tmp_dir = tempfile::tempdir()?; + let tmp_url = Url::from_directory_path(tmp_dir.path()).unwrap(); + let (store, _engine, table_url) = engine_store_setup("cdf_per_cell_null", Some(&tmp_url)); + + create_table( + store.clone(), + table_url.clone(), + schema.clone(), + &[], + true, + vec![], + vec!["changeDataFeed"], + ) + .await?; + + let stats = r#"{"numRecords":20,"minValues":{"EventTime":"+48690-07-02T22:50:38.211Z","UserId":5},"maxValues":{"EventTime":"+48690-07-02T22:50:38.211Z","UserId":200},"nullCount":{"EventTime":0,"UserId":0},"tightBounds":true}"#; + let stats_escaped = serde_json::Value::String(stats.to_string()).to_string(); + let commit_info = + r#"{"commitInfo":{"timestamp":1700000000000,"operation":"WRITE","version":1}}"#; + let add = format!( + r#"{{"add":{{"path":"phantom_file.parquet","size":1024,"modificationTime":1700000000000,"dataChange":true,"partitionValues":{{}},"stats":{stats_escaped}}}}}"# + ); + let commit_body = format!("{commit_info}\n{add}"); + add_commit(table_url.as_str(), store.as_ref(), 1, commit_body).await?; + + let engine = create_default_engine(&table_url)?; + let table_changes = TableChanges::try_new(table_url.clone(), engine.as_ref(), 1, Some(1))?; + + let predicate: PredicateRef = Arc::new(Pred::gt(col!("UserId"), lit(1000i64))); + let scan = table_changes + .into_scan_builder() + .with_predicate(predicate) + .build()?; + + // execute() must succeed and yield zero rows. The phantom file would crash an actual + // parquet read; the test passing means the per-cell NULL kept UserId stats valid and + // the predicate pruned the file before reading. + let batches: Vec = scan + .execute(engine)? + .map(|data| -> DeltaResult<_> { data?.try_into_record_batch() }) + .try_collect()?; + let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!( + total_rows, 0, + "predicate should have pruned the phantom file" + ); + Ok(()) +} + #[test] fn cdf_with_column_mapping_id_mode() -> Result<(), Box> { // NOTE: these tables only have CDF enabled in version 1+, so we start reading from 1. This is diff --git a/kernel/tests/clustering_e2e.rs b/kernel/tests/integration/features/clustering_e2e.rs similarity index 55% rename from kernel/tests/clustering_e2e.rs rename to kernel/tests/integration/features/clustering_e2e.rs index fbecfa9636..fefb7303b1 100644 --- a/kernel/tests/clustering_e2e.rs +++ b/kernel/tests/integration/features/clustering_e2e.rs @@ -7,15 +7,16 @@ use std::collections::HashMap; use std::sync::Arc; +use delta_kernel::actions::{MAX_VALUES, MIN_VALUES, NULL_COUNT, NUM_RECORDS}; +use delta_kernel::arrow::array::{ArrayRef, Int32Array}; use delta_kernel::committer::FileSystemCommitter; -use delta_kernel::expressions::ColumnName; +use delta_kernel::expressions::column_name; use delta_kernel::schema::{DataType, StructField, StructType}; use delta_kernel::snapshot::Snapshot; use delta_kernel::transaction::create_table::create_table; use delta_kernel::transaction::data_layout::DataLayout; use delta_kernel::transaction::CommitResult; use rstest::rstest; - use test_utils::{ generate_batch, read_add_infos, read_scan, test_table_setup_mt, write_batch_to_table, IntoArray, }; @@ -34,13 +35,13 @@ async fn test_clustered_table_write_and_checkpoint( let (_temp_dir, table_path, engine) = test_table_setup_mt()?; let schema = Arc::new( StructType::try_new(vec![ - StructField::new("id", DataType::INTEGER, false), + StructField::new("id", DataType::INTEGER, true), StructField::new("name", DataType::STRING, true), StructField::new("city", DataType::STRING, true), ]) .unwrap(), ); - let expected_clustering = vec![ColumnName::new(["id"]), ColumnName::new(["city"])]; + let expected_clustering = vec![column_name!("id"), column_name!("city")]; // Create table clustered on "id" and "city" let create_result = create_table(&table_path, schema, "Test/1.0") @@ -66,18 +67,21 @@ async fn test_clustered_table_write_and_checkpoint( // First write: 3 rows let batch = generate_batch(vec![ - ("id", vec![1, 2, 3].into_array()), - ("name", vec!["alice", "bob", "charlie"].into_array()), - ("city", vec!["seattle", "portland", "seattle"].into_array()), + ("id", vec![1, 2, 3].into_arrow_array()), + ("name", vec!["alice", "bob", "charlie"].into_arrow_array()), + ( + "city", + vec!["seattle", "portland", "seattle"].into_arrow_array(), + ), ])?; let snapshot = write_batch_to_table(&snapshot, engine.as_ref(), batch, HashMap::new()).await?; assert_eq!(snapshot.version(), 1); // Second write: 2 more rows let batch = generate_batch(vec![ - ("id", vec![4, 5].into_array()), - ("name", vec!["dave", "eve"].into_array()), - ("city", vec!["austin", "portland"].into_array()), + ("id", vec![4, 5].into_arrow_array()), + ("name", vec!["dave", "eve"].into_arrow_array()), + ("city", vec!["austin", "portland"].into_arrow_array()), ])?; let snapshot = write_batch_to_table(&snapshot, engine.as_ref(), batch, HashMap::new()).await?; assert_eq!(snapshot.version(), 2); @@ -90,11 +94,11 @@ async fn test_clustered_table_write_and_checkpoint( for col in &expected_clustering { let col_name = col.to_string(); assert!( - stats["minValues"].get(&col_name).is_some(), + stats[MIN_VALUES].get(&col_name).is_some(), "Stats should include minValues for clustering column '{col_name}'" ); assert!( - stats["maxValues"].get(&col_name).is_some(), + stats[MAX_VALUES].get(&col_name).is_some(), "Stats should include maxValues for clustering column '{col_name}'" ); } @@ -107,7 +111,7 @@ async fn test_clustered_table_write_and_checkpoint( assert_eq!(total_rows, 5); // Checkpoint - snapshot.checkpoint(engine.as_ref())?; + snapshot.checkpoint(engine.as_ref(), None)?; // Load fresh snapshot from checkpoint and verify everything survived let table_url = delta_kernel::try_parse_uri(&table_path)?; @@ -126,11 +130,11 @@ async fn test_clustered_table_write_and_checkpoint( for col in &expected_clustering { let col_name = col.to_string(); assert!( - stats["minValues"].get(&col_name).is_some(), + stats[MIN_VALUES].get(&col_name).is_some(), "Stats should include minValues for clustering column '{col_name}' after checkpoint" ); assert!( - stats["maxValues"].get(&col_name).is_some(), + stats[MAX_VALUES].get(&col_name).is_some(), "Stats should include maxValues for clustering column '{col_name}' after checkpoint" ); } @@ -138,3 +142,73 @@ async fn test_clustered_table_write_and_checkpoint( Ok(()) } + +/// Regression test: writing a batch where a clustering column has ALL null values should succeed. +/// +/// `collect_stats` emits null-valued min/max entries for all-null columns, allowing +/// `StatsColumnVerifier` to find the field and confirm `nullCount == numRecords`. The JSON +/// serializer omits null fields on disk, matching Spark's `ignoreNullFields` behavior. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_clustered_table_write_all_null_clustering_column() { + let (_temp_dir, table_path, engine) = test_table_setup_mt().unwrap(); + let schema = Arc::new( + StructType::try_new(vec![ + StructField::new("category", DataType::STRING, true), + StructField::new("region_id", DataType::INTEGER, true), + ]) + .unwrap(), + ); + + // Create table clustered on "category" and "region_id" + let create_result = create_table(&table_path, schema, "Test/1.0") + .with_data_layout(DataLayout::Clustered { + columns: vec![column_name!("category"), column_name!("region_id")], + }) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new())) + .unwrap() + .commit(engine.as_ref()) + .unwrap(); + + let snapshot = create_result + .unwrap_committed() + .post_commit_snapshot() + .expect("post-commit snapshot should exist") + .clone(); + + // Write a batch where region_id is ALL nulls. + // This should succeed -- all-null clustering columns are valid. + let all_null_region: ArrayRef = Arc::new(Int32Array::from(vec![None, None, None])); + let batch = generate_batch(vec![ + ("category", vec!["a", "b", "c"].into_arrow_array()), + ("region_id", all_null_region), + ]) + .unwrap(); + + let snapshot = write_batch_to_table(&snapshot, engine.as_ref(), batch, HashMap::new()) + .await + .unwrap(); + assert_eq!(snapshot.version(), 1); + + // Verify data is readable + let scan = snapshot.clone().scan_builder().build().unwrap(); + let batches = read_scan(&scan, engine.clone()).unwrap(); + let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total_rows, 3); + + // Verify stats: region_id should have nullCount=3. The JSON serializer omits null + // fields (matching Spark's ignoreNullFields), so minValues/maxValues should not + // contain region_id in the serialized JSON stats. + let add_infos = read_add_infos(&snapshot, engine.as_ref()).unwrap(); + assert_eq!(add_infos.len(), 1); + let stats = add_infos[0].stats.as_ref().expect("should have stats"); + assert_eq!(stats[NUM_RECORDS], 3); + assert_eq!(stats[NULL_COUNT]["region_id"], 3); + assert!( + stats[MIN_VALUES].get("region_id").is_none(), + "JSON minValues should omit region_id when all values are null" + ); + assert!( + stats[MAX_VALUES].get("region_id").is_none(), + "JSON maxValues should omit region_id when all values are null" + ); +} diff --git a/kernel/tests/integration/features/column_mapping.rs b/kernel/tests/integration/features/column_mapping.rs new file mode 100644 index 0000000000..4ed255bd45 --- /dev/null +++ b/kernel/tests/integration/features/column_mapping.rs @@ -0,0 +1,306 @@ +//! Integration tests for reading column-mapping tables. + +use std::collections::HashMap; +use std::sync::Arc; + +use delta_kernel::arrow::array::{Int32Array, RecordBatch, StringArray}; +use delta_kernel::arrow::datatypes::{ + DataType as ArrowDataType, Field as ArrowField, Schema as ArrowSchema, +}; +use delta_kernel::object_store::path::Path as ObjectStorePath; +use delta_kernel::object_store::{DynObjectStore, ObjectStoreExt as _}; +use delta_kernel::schema::{DataType, StructType}; +use delta_kernel::table_features::TableFeature; +use delta_kernel::Snapshot; +use rstest::rstest; +use serde_json::json; +use test_utils::column_mapping_fixtures::cm_field; +use test_utils::{add_commit, engine_store_setup, read_scan, record_batch_to_bytes}; + +/// Logical (display) names, physical names, and column-mapping ids. Physical names differ from the +/// logical ones so a physical-name read is distinguishable from a logical-name read. +const ID_LOGICAL: &str = "id"; +const ID_PHYSICAL: &str = "col-9f1c2b7e-0000-4000-8000-000000000000"; +const ID_FIELD_ID: i64 = 1; +const PAYLOAD_LOGICAL: &str = "payload_renamed"; +const PAYLOAD_PHYSICAL: &str = "col-9f1c2b7e-0000-4000-8000-000000000001"; +const PAYLOAD_FIELD_ID: i64 = 2; + +/// Builds the `metaData` action for a column-mapping table in the given mode. `cm_field` attaches +/// the per-field `delta.columnMapping.{id,physicalName}` metadata; the physical names differ from +/// the logical ones so the two read paths stay distinguishable. +fn metadata_action(mode: &str) -> serde_json::Value { + let schema = StructType::new_unchecked([ + cm_field(ID_LOGICAL, ID_FIELD_ID, ID_PHYSICAL, DataType::INTEGER), + cm_field( + PAYLOAD_LOGICAL, + PAYLOAD_FIELD_ID, + PAYLOAD_PHYSICAL, + DataType::STRING, + ), + ]); + + json!({ + "metaData": { + "id": "cm-reader-features-test-table", + "format": { "provider": "parquet", "options": {} }, + "schemaString": serde_json::to_string(&schema).unwrap(), + "partitionColumns": [], + "configuration": { + "delta.columnMapping.mode": mode, + "delta.columnMapping.maxColumnId": "2" + }, + "createdTime": 1_700_000_000_000i64 + } + }) +} + +/// `protocol` action for reader version 3 / writer version 7 with the supplied feature lists. +fn protocol_action(reader_features: &[&str], writer_features: &[&str]) -> serde_json::Value { + json!({ + "protocol": { + "minReaderVersion": 3, + "minWriterVersion": 7, + "readerFeatures": reader_features, + "writerFeatures": writer_features, + } + }) +} + +/// Commits version 0 (protocol + metadata) for a `name`-mode column-mapping table. +async fn setup_table( + store: &DynObjectStore, + table_root: &str, + reader_features: &[&str], + writer_features: &[&str], +) -> Result<(), Box> { + let commit = format!( + "{}\n{}\n", + protocol_action(reader_features, writer_features), + metadata_action("name") + ); + add_commit(table_root, store, 0, commit).await?; + Ok(()) +} + +fn field_names(snapshot: &Snapshot) -> Vec { + snapshot + .schema() + .fields() + .map(|f| f.name().to_string()) + .collect() +} + +#[rstest] +#[case::both_lists(&["columnMapping"], &["columnMapping"], true)] +#[case::orphaned_writer_only(&[], &["columnMapping"], true)] +#[case::absent_from_both(&[], &["appendOnly"], false)] +#[tokio::test] +async fn column_mapping_support_matrix( + #[case] reader_features: &[&str], + #[case] writer_features: &[&str], + #[case] expected_supported: bool, +) -> Result<(), Box> { + let scenario = format!( + "cm_support_{}_{}", + reader_features.len(), + writer_features.len() + ); + let (store, engine, table_url) = engine_store_setup(&scenario, None); + setup_table( + store.as_ref(), + table_url.as_str(), + reader_features, + writer_features, + ) + .await?; + + let snapshot = Snapshot::builder_for(table_url).build(&engine)?; + let table_config = snapshot.table_configuration(); + assert_eq!( + table_config.is_feature_supported(&TableFeature::ColumnMapping), + expected_supported, + ); + // Enablement tracks support here: the mode property is set, so a supported feature is enabled + // and an unsupported one is not. + assert_eq!( + table_config.is_feature_enabled(&TableFeature::ColumnMapping), + expected_supported, + ); + // The logical schema always reports display names, whatever the reader-list shape. + assert_eq!(field_names(&snapshot), vec![ID_LOGICAL, PAYLOAD_LOGICAL]); + Ok(()) +} + +/// Once an orphaned column-mapping table loads, a scan must resolve columns the same way +/// delta-spark does: by physical name in `name` mode, by parquet field id in `id` mode. Both modes +/// carry the full mapping metadata; only the data-file column identity differs, which is what this +/// asserts by rebuilding the parquet file per mode. Reading by logical name would miss every column +/// and return nulls, so recovering the values proves the resolution ran. +#[rstest] +#[case::name_mode("name")] +#[case::id_mode("id")] +#[tokio::test] +async fn orphaned_column_mapping_resolves_data_columns( + #[case] mode: &str, +) -> Result<(), Box> { + let scenario = format!("cm_orphan_read_{mode}"); + let (store, engine, table_url) = engine_store_setup(&scenario, None); + + // Orphaned protocol: columnMapping writer-side only. + let commit = format!( + "{}\n{}\n", + protocol_action(&[], &["columnMapping"]), + metadata_action(mode) + ); + add_commit(table_url.as_str(), store.as_ref(), 0, commit).await?; + + // The data file's columns are addressed the way each mode expects: `name` mode matches by the + // physical column name; `id` mode matches by the parquet `PARQUET:field_id` metadata (the + // physical names are intentionally omitted so a name-based read could not accidentally pass). + let (id_field, payload_field) = match mode { + "name" => ( + ArrowField::new(ID_PHYSICAL, ArrowDataType::Int32, true), + ArrowField::new(PAYLOAD_PHYSICAL, ArrowDataType::Utf8, true), + ), + "id" => ( + ArrowField::new("c0", ArrowDataType::Int32, true).with_metadata(HashMap::from([( + "PARQUET:field_id".to_string(), + ID_FIELD_ID.to_string(), + )])), + ArrowField::new("c1", ArrowDataType::Utf8, true).with_metadata(HashMap::from([( + "PARQUET:field_id".to_string(), + PAYLOAD_FIELD_ID.to_string(), + )])), + ), + other => panic!("unexpected mode {other}"), + }; + let batch = RecordBatch::try_new( + Arc::new(ArrowSchema::new(vec![id_field, payload_field])), + vec![ + Arc::new(Int32Array::from(vec![1, 2])), + Arc::new(StringArray::from(vec!["alpha", "beta"])), + ], + )?; + let parquet_bytes = record_batch_to_bytes(&batch); + let parquet_len = parquet_bytes.len(); + let data_path = "data_file.parquet"; + let data_url = table_url.join(data_path)?; + store + .put( + &ObjectStorePath::from_url_path(data_url.path())?, + parquet_bytes.into(), + ) + .await?; + + let add = json!({ + "add": { + "path": data_path, + "partitionValues": {}, + "size": parquet_len, + "modificationTime": 1, + "dataChange": true, + } + }); + add_commit(table_url.as_str(), store.as_ref(), 1, add.to_string()).await?; + + let snapshot = Snapshot::builder_for(table_url).build(&engine)?; + let scan = snapshot.scan_builder().build()?; + let batches = read_scan(&scan, Arc::new(engine))?; + + let batch = &batches[0]; + assert_eq!(batch.num_rows(), 2); + // Scan output is addressed by logical names ... + assert_eq!(batch.schema().field(0).name(), ID_LOGICAL); + assert_eq!(batch.schema().field(1).name(), PAYLOAD_LOGICAL); + // ... and carries the real values for every column, so the mode-specific column resolution + // fired. Assert `id` too: a broken physical mapping for it would otherwise surface as nulls + // with the test still green. + let id = batch + .column(0) + .as_any() + .downcast_ref::() + .expect("id column is an int32 array"); + assert_eq!(id.value(0), 1); + assert_eq!(id.value(1), 2); + let payload = batch + .column(1) + .as_any() + .downcast_ref::() + .expect("payload column is a string array"); + assert_eq!(payload.value(0), "alpha"); + assert_eq!(payload.value(1), "beta"); + Ok(()) +} + +/// Commits the given feature lists, loads the snapshot, and returns the load error. +async fn load_error( + scenario: &str, + reader_features: &[&str], + writer_features: &[&str], +) -> delta_kernel::Error { + let (store, engine, table_url) = engine_store_setup(scenario, None); + setup_table( + store.as_ref(), + table_url.as_str(), + reader_features, + writer_features, + ) + .await + .unwrap(); + Snapshot::builder_for(table_url) + .build(&engine) + .expect_err(&format!("{scenario}: expected snapshot load to fail")) +} + +/// Shapes that must still be rejected at load. The orphaned-feature exemption is scoped to legacy +/// ReaderWriter features present writer-side, so it must not admit any of these. +#[rstest] +// A non-legacy ReaderWriter feature (deletionVectors) missing from readerFeatures. +#[case::non_legacy_writer_only(&[], &["columnMapping", "deletionVectors"])] +// A ReaderWriter feature listed only in readerFeatures (the spec forbids this outright). +#[case::reader_only(&["columnMapping"], &["appendOnly"])] +#[tokio::test] +async fn invalid_protocol_shapes_are_rejected( + #[case] reader_features: &[&str], + #[case] writer_features: &[&str], +) { + let scenario = format!( + "cm_reject_{}_{}", + reader_features.len(), + writer_features.len() + ); + let err = load_error(&scenario, reader_features, writer_features).await; + assert!( + matches!(err, delta_kernel::Error::InvalidProtocol(_)), + "{err}" + ); +} + +/// Reader version 3 with `readerFeatures` absent entirely (not merely empty) is rejected before the +/// feature-consistency checks run: the protocol requires the field to be present at reader v3. The +/// orphaned-feature exemption deliberately does not extend to this shape. +#[tokio::test] +async fn reader_v3_without_reader_features_is_rejected() { + let (store, engine, table_url) = engine_store_setup("cm_absent_reader_list", None); + // readerFeatures is omitted from the protocol action entirely. + let protocol = json!({ + "protocol": { + "minReaderVersion": 3, + "minWriterVersion": 7, + "writerFeatures": ["columnMapping"], + } + }); + let commit = format!("{}\n{}\n", protocol, metadata_action("name")); + add_commit(table_url.as_str(), store.as_ref(), 0, commit) + .await + .unwrap(); + + let err = Snapshot::builder_for(table_url) + .build(&engine) + .expect_err("expected snapshot load to fail"); + assert!( + matches!(err, delta_kernel::Error::InvalidProtocol(_)), + "{err}" + ); +} diff --git a/kernel/tests/dv.rs b/kernel/tests/integration/features/dv.rs similarity index 55% rename from kernel/tests/dv.rs rename to kernel/tests/integration/features/dv.rs index 35a3980f1d..bd82615759 100644 --- a/kernel/tests/dv.rs +++ b/kernel/tests/integration/features/dv.rs @@ -5,21 +5,28 @@ use std::ops::Add; use std::path::PathBuf; use std::sync::Arc; -use delta_kernel::actions::deletion_vector_writer::{ - KernelDeletionVector, StreamingDeletionVectorWriter, -}; -use delta_kernel::committer::FileSystemCommitter; -use delta_kernel::engine_data::FilteredEngineData; -use delta_kernel::schema::{DataType, StructField, StructType}; +use delta_kernel::actions::deletion_vector_writer::KernelDeletionVector; +use delta_kernel::actions::{NUM_RECORDS, TIGHT_BOUNDS}; +use delta_kernel::arrow::array::{BooleanArray, Int64Array, StructArray}; +use delta_kernel::arrow::record_batch::RecordBatch; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::object_store::ObjectStoreExt as _; +use delta_kernel::scan::StatsOptions; +use delta_kernel::schema::{schema_ref, DataType, StructField, StructType}; use delta_kernel::transaction::CommitResult; use delta_kernel::{DeltaResult, EngineData, Snapshot}; +use itertools::Itertools; use tempfile::tempdir; use test_utils::{ - create_add_files_metadata, create_table, engine_store_setup, generate_batch, into_record_batch, - record_batch_to_bytes, IntoArray, + add_commit, create_add_files_metadata, create_default_engine_mt_executor, create_table, + engine_store_setup, generate_batch, into_record_batch, load_and_begin_transaction, + read_actions_from_commit, record_batch_to_bytes, IntoArray, }; -use itertools::Itertools; +use crate::common::write_utils::{ + create_dv_update_transaction, get_scan_files, resolve_struct_field, + write_deletion_vector_to_store, +}; /// Helper to write a parquet file with the given data to the table. /// Returns the file path (relative to table root) that was written. @@ -71,68 +78,6 @@ fn test_table_scan( Ok(()) } -/// Helper to extract scan files from a snapshot -fn get_scan_files( - snapshot: Arc, - engine: &dyn delta_kernel::Engine, -) -> DeltaResult> { - let scan = snapshot.scan_builder().build()?; - let all_scan_metadata: Vec<_> = scan.scan_metadata(engine)?.collect::, _>>()?; - - Ok(all_scan_metadata - .into_iter() - .map(|sm| sm.scan_files) - .collect()) -} - -/// Helper to get a write context for creating deletion vector paths. -fn get_write_context( - table_url: &url::Url, - engine: &dyn delta_kernel::Engine, -) -> Result> { - let snapshot = Snapshot::builder_for(table_url.clone()).build(engine)?; - let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), engine)?; - Ok(txn.get_write_context()) -} - -/// Helper to write a deletion vector to object store and return its descriptor. -async fn write_deletion_vector_to_store( - store: &Arc, - write_context: &delta_kernel::transaction::WriteContext, - dv: KernelDeletionVector, - prefix: &str, -) -> Result< - delta_kernel::actions::deletion_vector::DeletionVectorDescriptor, - Box, -> { - use delta_kernel::object_store::path::Path as ObjectStorePath; - - let dv_path = write_context.new_deletion_vector_path(String::from(prefix)); - let dv_absolute_path = dv_path.absolute_path()?; - let dv_object_path = ObjectStorePath::parse(dv_absolute_path.path())?; - - let mut dv_buffer = Vec::new(); - let mut dv_writer = StreamingDeletionVectorWriter::new(&mut dv_buffer); - let dv_write_result = dv_writer.write_deletion_vector(dv)?; - dv_writer.finalize()?; - - store.put(&dv_object_path, dv_buffer.into()).await?; - - Ok(dv_write_result.to_descriptor(&dv_path)) -} - -/// Helper to create a transaction for deletion vector updates. -fn create_dv_update_transaction( - table_url: &url::Url, - engine: &dyn delta_kernel::Engine, -) -> Result> { - let snapshot = Snapshot::builder_for(table_url.clone()).build(engine)?; - Ok(snapshot - .transaction(Box::new(FileSystemCommitter::new()), engine)? - .with_engine_info("test engine") - .with_operation("DELETE".to_string())) -} - /// Helper to verify that scan results match expected ids and values (after sorting). /// Extracts int32 id column and string value column from batches and compares with expected. fn verify_sorted_scan_results( @@ -223,21 +168,21 @@ async fn test_write_deletion_vectors_end_to_end() -> Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box, +) -> Result<(), Box> { + // Nested schema: a `point` struct with two leaf columns. + let schema = schema_ref! { + nullable "point": { nullable "x": INTEGER, nullable "y": INTEGER }, + }; + + let temp_dir = tempdir()?; + let base_url = url::Url::from_directory_path(temp_dir.path()).unwrap(); + let (store, engine, table_url) = engine_store_setup("test_table", Some(&base_url)); + let engine = Arc::new(engine); + + // Hand-write v0 as create table doesn't support `delta.checkpoint.writeStatsAsStruct`. + std::fs::create_dir_all(table_url.to_file_path().unwrap())?; + let protocol = serde_json::json!({ + "protocol": { + "minReaderVersion": 3, + "minWriterVersion": 7, + "readerFeatures": ["deletionVectors"], + "writerFeatures": ["deletionVectors"], + } + }); + let metadata = serde_json::json!({ + "metaData": { + "id": "test_id", + "format": {"provider": "parquet", "options": {}}, + "schemaString": serde_json::to_string(&schema)?, + "partitionColumns": [], + "configuration": { + "delta.enableDeletionVectors": "true", + "delta.checkpoint.writeStatsAsStruct": "true", + }, + "createdTime": 1677811175819u64, + } + }); + let v0 = format!( + "{}\n{}\n", + serde_json::to_string(&protocol)?, + serde_json::to_string(&metadata)? + ); + add_commit(table_url.as_str(), store.as_ref(), 0, v0).await?; + + // Commit an Add at v1 with hand-crafted nested stats and the parameterized `tightBounds`. + // Kernel inserts always set tightBounds: true, so we handcraft the commit here. + let data_file_path = "data_file_1.parquet"; + let mut stats = serde_json::json!({ + "numRecords": 10, + "nullCount": {"point": {"x": 0, "y": 2}}, + "minValues": {"point": {"x": 1, "y": 5}}, + "maxValues": {"point": {"x": 100, "y": 50}}, + }); + if let Some(tb) = initial_tight_bounds { + stats["tightBounds"] = serde_json::Value::Bool(tb); + } + let stats_str = serde_json::to_string(&stats)?; + let add = serde_json::json!({ + "add": { + "path": data_file_path, + "partitionValues": {}, + "size": 1000, + "modificationTime": 1, + "dataChange": true, + "stats": stats_str, + } + }); + add_commit( + table_url.as_str(), + store.as_ref(), + 1, + serde_json::to_string(&add)?, + ) + .await?; + + // Apply a DV to the file and commit. + let mut dv = KernelDeletionVector::new(); + dv.add_deleted_row_indexes([3u64]); + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let mut txn = create_dv_update_transaction(&table_url, engine.as_ref())?; + let write_context = txn.unpartitioned_write_context()?; + let dv_descriptor = write_deletion_vector_to_store(&store, &write_context, dv, "").await?; + + let scan_files = get_scan_files(snapshot, engine.as_ref())?; + let mut dv_map = HashMap::new(); + dv_map.insert(data_file_path.to_string(), dv_descriptor); + txn.update_deletion_vectors(dv_map, scan_files.into_iter().map(Ok))?; + txn.commit(engine.as_ref())?.unwrap_committed(); + + // The new AddFile must report tightBounds: false while preserving every other stats field. + let v2_adds = read_actions_from_commit(&table_url, 2, "add")?; + assert_eq!(v2_adds.len(), 1, "expected exactly one add at v2"); + let add_stats: serde_json::Value = serde_json::from_str( + v2_adds[0]["stats"] + .as_str() + .expect("add stats should be a JSON string"), + )?; + assert_eq!( + add_stats["tightBounds"].as_bool(), + Some(false), + "DV-update add must report tightBounds: false regardless of the original value" + ); + assert_eq!(add_stats["numRecords"], stats["numRecords"]); + assert_eq!( + add_stats["nullCount"], stats["nullCount"], + "nested nullCount must survive verbatim" + ); + assert_eq!( + add_stats["minValues"], stats["minValues"], + "nested minValues must survive verbatim" + ); + assert_eq!( + add_stats["maxValues"], stats["maxValues"], + "nested maxValues must survive verbatim" + ); + + // The paired RemoveFile must preserve the original stats unchanged. + let v2_removes = read_actions_from_commit(&table_url, 2, "remove")?; + assert_eq!(v2_removes.len(), 1, "expected exactly one remove at v2"); + let remove_stats: serde_json::Value = serde_json::from_str( + v2_removes[0]["stats"] + .as_str() + .expect("remove stats should be a JSON string"), + )?; + assert_eq!( + remove_stats["tightBounds"].as_bool(), + initial_tight_bounds, + "remove action must preserve the original tightBounds unchanged" + ); + + // Checkpoint and validate the tightBounds of stats_parsed. + let snapshot_v2 = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let mt_engine = create_default_engine_mt_executor(&table_url)?; + snapshot_v2.checkpoint(mt_engine.as_ref(), None)?; + + let ckpt_snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let scan = ckpt_snapshot + .scan_builder() + .with_stats(StatsOptions::all()) + .build()?; + let mut checked_stats_parsed = false; + for scan_metadata in scan.scan_metadata(engine.as_ref())? { + let (data, selection_vector) = scan_metadata?.scan_files.into_parts(); + let batch: RecordBatch = ArrowEngineData::try_from_engine_data(data)?.into(); + let batch_struct = StructArray::from(batch.clone()); + let tight_bounds: &BooleanArray = + resolve_struct_field(&batch_struct, &["stats_parsed".into(), TIGHT_BOUNDS.into()]); + let num_records: &Int64Array = + resolve_struct_field(&batch_struct, &["stats_parsed".into(), NUM_RECORDS.into()]); + for (i, &selected) in selection_vector.iter().enumerate().take(batch.num_rows()) { + if !selected { + continue; + } + assert!( + !tight_bounds.value(i), + "checkpoint stats_parsed.tightBounds must be false after a DV update" + ); + assert_eq!( + num_records.value(i), + 10, + "checkpoint stats_parsed.numRecords must be preserved" + ); + checked_stats_parsed = true; + } + } + assert!( + checked_stats_parsed, + "expected at least one selected file with stats_parsed in the checkpoint" + ); + + Ok(()) +} diff --git a/kernel/tests/integration/features/iceberg_compat.rs b/kernel/tests/integration/features/iceberg_compat.rs new file mode 100644 index 0000000000..3438b0bc6e --- /dev/null +++ b/kernel/tests/integration/features/iceberg_compat.rs @@ -0,0 +1,985 @@ +//! Integration tests for icebergCompat + +use std::collections::HashMap; +use std::sync::Arc; + +use delta_kernel::arrow::array::{ + ArrayRef, BinaryArray, BooleanArray, Date32Array, Decimal128Array, Float32Array, Float64Array, + Int16Array, Int32Array, Int64Array, Int8Array, MapArray, RecordBatch, StringArray, StructArray, + TimestampMicrosecondArray, +}; +use delta_kernel::arrow::buffer::{NullBuffer, OffsetBuffer}; +use delta_kernel::arrow::datatypes::{ + DataType as ArrowDataType, Field as ArrowField, Schema as ArrowSchema, +}; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::engine::arrow_conversion::{TryFromKernel, TryIntoArrow as _}; +use delta_kernel::expressions::{ColumnName, Scalar}; +use delta_kernel::object_store::local::LocalFileSystem; +use delta_kernel::object_store::memory::InMemory; +use delta_kernel::object_store::DynObjectStore; +use delta_kernel::parquet::basic::Type as ParquetPhysicalType; +use delta_kernel::parquet::schema::types::Type as ParquetType; +use delta_kernel::schema::{ + ArrayType, ColumnMetadataKey, DataType, MapType, MetadataValue, StructField, StructType, +}; +use delta_kernel::snapshot::Snapshot; +use delta_kernel::table_features::{get_any_level_column_physical_name, ColumnMappingMode}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::transaction::data_layout::DataLayout; +use delta_kernel::transforms::{transform_output_type, SchemaTransform}; +use test_utils::delta_kernel_default_engine::executor::tokio::{ + TokioBackgroundExecutor, TokioMultiThreadExecutor, +}; +use test_utils::delta_kernel_default_engine::{DefaultEngine, DefaultEngineBuilder}; +use test_utils::{ + add_commit, create_add_files_metadata, into_record_batch, read_add_infos, test_table_setup_mt, + write_batch_to_table, +}; +use url::Url; + +use crate::common::write_utils::{collect_all_parquet_field_ids, read_parquet_root_schema}; + +const TABLE_ROOT: &str = "memory:///"; + +/// Hand-crafts a V0 create commit on an icebergCompatV3 table whose `data` field carries the +/// to-be-deprecated `parquet.field.nested.ids` metadata. Loading the snapshot must fail. +#[tokio::test] +async fn snapshot_blocked_when_v3_schema_has_legacy_nested_ids() { + let (storage, engine) = make_default_engine_and_store(); + // Create table doesn't support the legacy nested ids metadata, so we hand-craft a V0 commit. + let nested_ids_legacy = serde_json::json!({ + "data.key": 100, + "data.value": 101, + "data.value.element": 102, + }); + let schema = StructType::try_new(vec![StructField::nullable( + "data", + MapType::new( + DataType::INTEGER, + ArrayType::new(DataType::INTEGER, true), + true, + ), + ) + .with_metadata([ + ( + ColumnMetadataKey::ColumnMappingId.as_ref(), + MetadataValue::from(1), + ), + ( + ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), + MetadataValue::from("col-1"), + ), + ( + ColumnMetadataKey::ParquetFieldNestedIds.as_ref(), + MetadataValue::Other(nested_ids_legacy), + ), + ])]) + .unwrap(); + let schema_string = serde_json::to_string(&schema).unwrap(); + + let commit = [ + serde_json::json!({ + "commitInfo": { + "timestamp": 1587968586154_i64, + "operation": "CREATE TABLE", + "operationParameters": {}, + "isBlindAppend": true, + } + }), + serde_json::json!({ + "protocol": { + "minReaderVersion": 3, + "minWriterVersion": 7, + "readerFeatures": ["columnMapping"], + "writerFeatures": [ + "icebergCompatV3", + "columnMapping", + "rowTracking", + "domainMetadata", + ], + } + }), + serde_json::json!({ + "metaData": { + "id": "deadbeef-1234-5678-abcd-000000000001", + "format": { "provider": "parquet", "options": {} }, + "schemaString": schema_string, + "partitionColumns": [], + "configuration": { + "delta.enableIcebergCompatV3": "true", + "delta.columnMapping.mode": "name", + "delta.enableRowTracking": "true", + "delta.rowTracking.materializedRowIdColumnName": "_row_id", + "delta.rowTracking.materializedRowCommitVersionColumnName": + "_row_commit_version", + "delta.columnMapping.maxColumnId": "1", + }, + "createdTime": 1234567890000_i64, + } + }), + ] + .into_iter() + .map(|action| serde_json::to_string(&action).unwrap()) + .collect::>() + .join("\n"); + add_commit(TABLE_ROOT, storage.as_ref(), 0, commit) + .await + .unwrap(); + + let err = Snapshot::builder_for(TABLE_ROOT) + .build(engine.as_ref()) + .unwrap_err() + .to_string(); + assert!( + err.contains("parquet.field.nested.ids") + && err.contains("delta.columnMapping.nested.ids") + && err.contains("data"), + "expected error mentioning the legacy key, replacement key, and field path, got: {err}", + ); +} + +#[rstest::rstest] +#[case::missing_num_records(None/* num_records */, Err("'stats.numRecords' is required"))] +#[case::with_num_records(Some(3), Ok(1))] +#[tokio::test] +async fn v3_commit_validates_num_records( + #[case] num_records: Option, + #[case] expected: Result, +) { + let (_, engine) = make_default_engine_and_store(); + + let _ = create_table(TABLE_ROOT, simple_schema(), "Test/1.0") + .with_table_properties([("delta.enableIcebergCompatV3", "true")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new())) + .unwrap() + .commit(engine.as_ref()) + .unwrap(); + let snapshot = Snapshot::builder_for(TABLE_ROOT) + .build(engine.as_ref()) + .unwrap(); + + let mut txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref()) + .unwrap() + .with_engine_info("Test/1.0") + .with_data_change(true); + let add_files = create_add_files_metadata( + txn.add_files_schema(), + vec![("part-fake.parquet", 1024, 1_000_000, num_records)], + ) + .unwrap(); + txn.add_files(add_files); + + match expected { + Ok(expected_version) => { + let committed = txn.commit(engine.as_ref()).unwrap().unwrap_committed(); + assert_eq!(committed.commit_version(), expected_version); + } + Err(needle) => { + let err = txn.commit(engine.as_ref()).unwrap_err().to_string(); + assert!( + err.contains(needle) && err.contains("part-fake.parquet"), + "expected error containing {needle:?} and 'part-fake.parquet', got: {err}", + ); + } + } +} + +/// V3 partitioned end-to-end: create + 8 commits with a checkpoint in the middle, then validate +/// protocol features, parquet field IDs (top-level + nested), partition materialization, timestamp +/// physical type, and exact scan content. +/// +/// IcebergCompatV3 supports all delta types, so we use a schema with all delta types here. +/// Two test cases: +/// - `min`: minimum feature set. Enables only `delta.enableIcebergCompatV3=true` and relies on V3's +/// auto-enablement of `columnMapping=name` + `rowTracking=true`. +/// - `max`: maximum feature set we are able to enable through create table, with exceptions for: +/// `materializePartitionColumns` (omitted so the partition-materialization check below proves V3 +/// implies it), `typeWidening` (kernel rejects writes against tables declaring it), and +/// `catalogManaged` (requires a catalog committer). +#[rstest::rstest] +#[case::min( + /* extra_props */ &[], + /* enable_features */ &[], + /* expected_features */ &[V3_BASELINE_FEATURES], +)] +#[case::max( + // Intentionally set `delta.columnMapping.mode=id`, since the `min` case already covers the + // V3-default `name` mode. + /* extra_props */ &[("delta.columnMapping.mode", "id")], + // Some features are enabled via schema content (e.g. TS_NTZ) so not in this list. + /* enable_features */ &[ + "deletionVectors", "inCommitTimestamp", "changeDataFeed", "appendOnly", + "v2Checkpoint", "vacuumProtocolCheck", "invariants", + ], + /* expected_features */ &[READER_WRITER_FEATURES, WRITER_FEATURES], +)] +#[tokio::test(flavor = "multi_thread")] +async fn v3_e2e_partitioned_writes_with_field_ids( + #[case] extra_props: &[(&str, &str)], + #[case] enable_features: &[&str], + #[case] expected_features: &[&[&str]], +) { + // === Setup === + let (_tmp_dir, table_path, _) = test_table_setup_mt().unwrap(); + let table_url = Url::from_directory_path(&table_path).unwrap(); + let store: Arc = Arc::new(LocalFileSystem::new()); + let engine = Arc::new( + DefaultEngineBuilder::new(store.clone()) + .with_task_executor(Arc::new(TokioMultiThreadExecutor::new( + tokio::runtime::Handle::current(), + ))) + .build(), + ); + + // === Create partitioned V3 table === + let schema = nested_schema_with_all_delta_types(); + // Features without an enable property are added via `delta.feature.X=supported`. + let feature_enabled_by_supported: Vec<(String, String)> = enable_features + .iter() + .filter(|f| enable_property_for(f).is_none()) + .map(|f| (format!("delta.feature.{f}"), "supported".to_string())) + .collect(); + let mut props: Vec<(&str, &str)> = vec![("delta.enableIcebergCompatV3", "true")]; + props.extend(extra_props.iter().copied()); + props.extend( + enable_features + .iter() + .filter_map(|f| enable_property_for(f).map(|p| (p, "true"))), + ); + props.extend( + feature_enabled_by_supported + .iter() + .map(|(k, v)| (k.as_str(), v.as_str())), + ); + let _ = create_table(&table_path, schema.clone(), "Test/1.0") + .with_table_properties(props) + .with_data_layout(DataLayout::partitioned(["region"])) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new())) + .unwrap() + .commit(engine.as_ref()) + .unwrap(); + + // === 4 commits (2 outer iters x 2 partitions), checkpoint, 4 more commits === + for commit_idx in 0..2_i32 { + write_partitioned_data(&table_url, engine.clone(), commit_idx).await; + } + Snapshot::builder_for(table_url.clone()) + .build(engine.as_ref()) + .unwrap() + .checkpoint(engine.as_ref(), None) + .unwrap(); + for commit_idx in 2..4_i32 { + write_partitioned_data(&table_url, engine.clone(), commit_idx).await; + } + + let final_snap = Snapshot::builder_for(table_url.clone()) + .build(engine.as_ref()) + .unwrap(); + + // === Verify protocol features and column mapping mode === + let expected: Vec<&str> = expected_features + .iter() + .flat_map(|list| list.iter().copied()) + .collect(); + verify_protocol(&final_snap, &expected); + let cm_mode = final_snap + .table_properties() + .column_mapping_mode + .expect("V3 must enable column mapping"); + let expected_cm_mode = extra_props + .iter() + .find(|(k, _)| *k == "delta.columnMapping.mode") + .map(|(_, v)| match *v { + "id" => ColumnMappingMode::Id, + "name" => ColumnMappingMode::Name, + other => panic!("unexpected column mapping mode {other:?}"), + }) + .unwrap_or(ColumnMappingMode::Name); + assert_eq!(cm_mode, expected_cm_mode); + + // === Per-data-file validations === + let add_actions = read_add_infos(&final_snap, engine.as_ref()).unwrap(); + assert_eq!(add_actions.len(), 8, "expected 8 add files",); + let logical_schema = final_snap.schema(); + for add in &add_actions { + let parquet_url = table_url.join(&add.path).unwrap(); + let local_path = parquet_url.to_file_path().unwrap(); + let ids = collect_all_parquet_field_ids(&local_path); + let parquet_root_schema = read_parquet_root_schema(&local_path); + + // Top-level + nested column-mapping IDs all appear as expected. + // Expected count for `nested_schema_with_all_delta_types`: + // 16 top-level fields + // + 3 nested.ids on `complex` (`complex.{key, key.element, value}`) + // + 2 struct fields inside `complex.value` (`inner_map`, `n`) + // + 3 nested.ids on `inner_map` (`inner_map.{key, key.element, value}`) + // = 24 + verify_column_mapping_ids_in_parquet( + logical_schema.as_ref(), + cm_mode, + &ids, + &parquet_url, + 24, /* expected_num_ids */ + ); + + // Partition column is physically materialized (V3 invariant). + let region_physical = + get_top_level_physical_name(logical_schema.as_ref(), "region", cm_mode); + assert!( + find_top_level_field_in_parquet(&parquet_root_schema, ®ion_physical).is_some(), + "partition column `region` (physical {region_physical}) not materialized in {parquet_url}", + ); + + // timestamp / timestamp_ntz are INT64 in parquet. + for col in ["timestamp_col", "timestamp_ntz_col"] { + let physical = get_top_level_physical_name(logical_schema.as_ref(), col, cm_mode); + assert_eq!( + find_top_level_physical_type_in_parquet(&parquet_root_schema, &physical), + ParquetPhysicalType::INT64, + "{col} must be INT64 in parquet", + ); + } + } + + // === Exact scan contents === + verify_scan_contents(final_snap.clone(), engine.clone()); +} + +// === Helpers === + +fn make_default_engine_and_store() -> (Arc, Arc>) { + let storage = Arc::new(InMemory::new()); + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + (storage, engine) +} + +fn simple_schema() -> Arc { + Arc::new( + StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("name", DataType::STRING), + ]) + .unwrap(), + ) +} + +/// Schema covering every Delta types(primitive, struct, map, array, variant) +/// Including a deeply nested map, struct, int>, n: int>> column. +/// `region` is the partition column. +fn nested_schema_with_all_delta_types() -> Arc { + Arc::new( + StructType::try_new(vec![ + StructField::nullable("region", DataType::STRING), + StructField::nullable("int_col", DataType::INTEGER), + StructField::nullable("bool_col", DataType::BOOLEAN), + StructField::nullable("byte_col", DataType::BYTE), + StructField::nullable("short_col", DataType::SHORT), + StructField::nullable("long_col", DataType::LONG), + StructField::nullable("float_col", DataType::FLOAT), + StructField::nullable("double_col", DataType::DOUBLE), + StructField::nullable("decimal_col", DataType::decimal(10, 2).unwrap()), + StructField::nullable("string_col", DataType::STRING), + StructField::nullable("binary_col", DataType::BINARY), + StructField::nullable("date_col", DataType::DATE), + StructField::nullable("timestamp_col", DataType::TIMESTAMP), + StructField::nullable("timestamp_ntz_col", DataType::TIMESTAMP_NTZ), + StructField::nullable("variant_col", DataType::unshredded_variant()), + StructField::nullable("complex", complex_nested_data_type()), + ]) + .unwrap(), + ) +} + +/// `map, struct, int>, n: int>>`. A deeply nested +/// type used to validate field-id propagation through every level of map/list/struct. +fn complex_nested_data_type() -> DataType { + let inner_struct = StructType::try_new(vec![ + StructField::nullable( + "inner_map", + MapType::new( + ArrayType::new(DataType::INTEGER, true), + DataType::INTEGER, + true, + ), + ), + StructField::nullable("n", DataType::INTEGER), + ]) + .unwrap(); + DataType::from(MapType::new( + ArrayType::new(DataType::INTEGER, true), + inner_struct, + true, + )) +} + +const ROWS_PER_PARTITION: i32 = 3; +const PARTITION_REGIONS: &[&str] = &["a", "b"]; + +/// Build a [`RecordBatch`] with [`ROWS_PER_PARTITION`] rows following the schema of +/// [`nested_schema_with_all_delta_types`], minus the `region` partition column. `random_seed` is +/// used directly to produce different values so different calls produce distinguishable rows. +fn build_data_batch(random_seed: i32) -> RecordBatch { + let rows = ROWS_PER_PARTITION as usize; + + // Data schema excludes the `region` partition column. + let data_schema = StructType::try_new( + nested_schema_with_all_delta_types() + .fields() + .filter(|f| f.name() != "region") + .cloned(), + ) + .unwrap(); + let arrow_schema: ArrowSchema = (&data_schema).try_into_arrow().unwrap(); + + let int_col: ArrayRef = Arc::new(Int32Array::from( + (0..ROWS_PER_PARTITION) + .map(|i| random_seed + i) + .collect::>(), + )); + let bool_col: ArrayRef = Arc::new(BooleanArray::from( + (0..rows).map(|i| i % 2 == 0).collect::>(), + )); + let byte_col: ArrayRef = Arc::new(Int8Array::from( + (0..ROWS_PER_PARTITION) + .map(|i| (random_seed + i) as i8) + .collect::>(), + )); + let short_col: ArrayRef = Arc::new(Int16Array::from( + (0..ROWS_PER_PARTITION) + .map(|i| (random_seed + i) as i16) + .collect::>(), + )); + let long_col: ArrayRef = Arc::new(Int64Array::from( + (0..ROWS_PER_PARTITION) + .map(|i| (random_seed + i) as i64) + .collect::>(), + )); + let float_col: ArrayRef = Arc::new(Float32Array::from( + (0..ROWS_PER_PARTITION) + .map(|i| (random_seed + i) as f32 + 0.5) + .collect::>(), + )); + let double_col: ArrayRef = Arc::new(Float64Array::from( + (0..ROWS_PER_PARTITION) + .map(|i| (random_seed + i) as f64 + 0.25) + .collect::>(), + )); + // Decimal(10, 2): values stored as scaled i128. + let decimal_col: ArrayRef = Arc::new( + Decimal128Array::from( + (0..ROWS_PER_PARTITION) + .map(|i| (random_seed + i) as i128 * 100) + .collect::>(), + ) + .with_precision_and_scale(10, 2) + .unwrap(), + ); + let string_col: ArrayRef = Arc::new(StringArray::from( + (0..rows) + .map(|i| format!("s-{random_seed}-{i}")) + .collect::>(), + )); + let binary_col: ArrayRef = Arc::new(BinaryArray::from_iter_values( + (0..rows).map(|i| vec![(random_seed as u8).wrapping_add(i as u8)]), + )); + // Days since 1970-01-01. + let date_col: ArrayRef = Arc::new(Date32Array::from( + (0..ROWS_PER_PARTITION) + .map(|i| 19000 + random_seed + i) + .collect::>(), + )); + // Microseconds since epoch: TIMESTAMP carries UTC ("+00:00"), TIMESTAMP_NTZ has no zone. + let timestamp_col: ArrayRef = Arc::new( + TimestampMicrosecondArray::from( + (0..ROWS_PER_PARTITION) + .map(|i| 1_700_000_000_000_000_i64 + (random_seed + i) as i64) + .collect::>(), + ) + .with_timezone("UTC"), + ); + let timestamp_ntz_col: ArrayRef = Arc::new(TimestampMicrosecondArray::from( + (0..ROWS_PER_PARTITION) + .map(|i| 1_700_000_000_000_000_i64 + (random_seed + i) as i64 + 1) + .collect::>(), + )); + let variant_col: ArrayRef = build_all_null_variant_array(rows); + let complex: ArrayRef = build_all_null_complex_array(rows); + + RecordBatch::try_new( + Arc::new(arrow_schema), + vec![ + int_col, + bool_col, + byte_col, + short_col, + long_col, + float_col, + double_col, + decimal_col, + string_col, + binary_col, + date_col, + timestamp_col, + timestamp_ntz_col, + variant_col, + complex, + ], + ) + .unwrap() +} + +/// Builds an all-null unshredded variant column with [`rows`] rows. +fn build_all_null_variant_array(rows: usize) -> ArrayRef { + let metadata_bytes: Vec<&[u8]> = (0..rows).map(|_| &[0x01, 0x00, 0x00][..]).collect(); + let value_bytes: Vec<&[u8]> = (0..rows).map(|_| &[0x0C, 0x01][..]).collect(); + let null_bitmap = NullBuffer::from(vec![false; rows]); + let metadata_arr: ArrayRef = Arc::new(BinaryArray::from(metadata_bytes)); + let value_arr: ArrayRef = Arc::new(BinaryArray::from(value_bytes)); + let arrow_variant: ArrowDataType = + ArrowDataType::try_from_kernel(&DataType::unshredded_variant()).unwrap(); + let fields = match arrow_variant { + ArrowDataType::Struct(fields) => fields, + _ => panic!("variant arrow type must be struct"), + }; + Arc::new( + StructArray::try_new(fields, vec![metadata_arr, value_arr], Some(null_bitmap)).unwrap(), + ) +} + +/// Builds an all-null `complex` MapArray of length `rows` matching [`complex_nested_data_type`]. +/// All map entries are null; the underlying entries struct is empty. +fn build_all_null_complex_array(rows: usize) -> ArrayRef { + let complex_arrow_type: ArrowDataType = (&complex_nested_data_type()).try_into_arrow().unwrap(); + let (entries_field, _sorted) = match complex_arrow_type { + ArrowDataType::Map(entries, sorted) => (entries, sorted), + other => panic!("complex must be a Map type, got {other:?}"), + }; + let entries_struct_fields = match entries_field.data_type() { + ArrowDataType::Struct(fields) => fields.clone(), + other => panic!("map entries must be struct, got {other:?}"), + }; + // Empty entries struct. + let empty_entries = StructArray::new( + entries_struct_fields, + entries_field_arrays_empty(entries_field.as_ref()), + None, + ); + let offsets = OffsetBuffer::from_lengths(vec![0_usize; rows]); + let nulls = NullBuffer::from(vec![false; rows]); + Arc::new(MapArray::new( + entries_field, + offsets, + empty_entries, + Some(nulls), + false, + )) +} + +fn entries_field_arrays_empty(entries_field: &ArrowField) -> Vec { + match entries_field.data_type() { + ArrowDataType::Struct(fields) => fields + .iter() + .map(|f| delta_kernel::arrow::array::new_empty_array(f.data_type())) + .collect(), + other => panic!("expected struct entries, got {other:?}"), + } +} + +/// Write one commit per partition (regions "a" and "b"). Stats and +/// `numRecords` are auto-collected by the default engine, so the numRecords invariant is +/// satisfied. Assume the table schema is [`nested_schema_with_all_delta_types`]. +async fn write_partitioned_data( + table_url: &Url, + engine: Arc>, + random_seed: i32, +) { + let mut snapshot = Snapshot::builder_for(table_url.clone()) + .build(engine.as_ref()) + .unwrap(); + // Defensive sanity check: confirm the table schema matches what `build_data_batch` + // produces. We compare top-level field names rather than full structs for simplicity. + let actual_schema = snapshot.schema(); + let actual_fields: Vec<&str> = actual_schema.fields().map(|f| f.name().as_str()).collect(); + let expected_schema = nested_schema_with_all_delta_types(); + let expected_fields: Vec<&str> = expected_schema + .fields() + .map(|f| f.name().as_str()) + .collect(); + assert_eq!( + actual_fields, expected_fields, + "table schema does not match `nested_schema_with_all_delta_types`", + ); + + for region in PARTITION_REGIONS { + let batch = build_data_batch(random_seed); + let partition_values = + HashMap::from([("region".to_string(), Scalar::String(region.to_string()))]); + snapshot = write_batch_to_table(&snapshot, engine.as_ref(), batch, partition_values) + .await + .unwrap(); + } +} + +/// Features that V3 auto-enables (plus TS_NTZ and variant types which are auto-enabled by our test +/// schema). +const V3_BASELINE_FEATURES: &[&str] = &[ + "icebergCompatV3", + "columnMapping", + "rowTracking", + "domainMetadata", + "timestampNtz", + "variantType", +]; + +/// ReaderWriter features used in this test file. +const READER_WRITER_FEATURES: &[&str] = &[ + "columnMapping", + "deletionVectors", + "timestampNtz", + "v2Checkpoint", + "vacuumProtocolCheck", + "variantType", +]; + +/// Writer-only features used in this test file. +const WRITER_FEATURES: &[&str] = &[ + "appendOnly", + "changeDataFeed", + "domainMetadata", + "icebergCompatV3", + "inCommitTimestamp", + "invariants", + "rowTracking", +]; + +/// Maps each enablement-driven feature to its `delta.enable*` (or `delta.appendOnly`) +/// property name. +const FEATURE_ENABLE_PROPERTY: &[(&str, &str)] = &[ + ("appendOnly", "delta.appendOnly"), + ("changeDataFeed", "delta.enableChangeDataFeed"), + ("deletionVectors", "delta.enableDeletionVectors"), + ("icebergCompatV3", "delta.enableIcebergCompatV3"), + ("inCommitTimestamp", "delta.enableInCommitTimestamps"), + ("rowTracking", "delta.enableRowTracking"), +]; + +/// Returns the `delta.enable*` property name for `feature` if one exists, or `None` if the +/// feature must be added via the `delta.feature.X=supported` signal instead. +fn enable_property_for(feature: &str) -> Option<&'static str> { + FEATURE_ENABLE_PROPERTY + .iter() + .find(|(name, _)| *name == feature) + .map(|(_, prop)| *prop) +} + +/// Assert each name in `expected_features` appears in `writerFeatures`, plus that any +/// reader+writer feature also appears in `readerFeatures`. +fn verify_protocol(snapshot: &Snapshot, expected_features: &[&str]) { + let protocol = snapshot.table_configuration().protocol(); + let writer_features: Vec = protocol + .writer_features() + .expect("V3 table must publish writerFeatures") + .iter() + .map(|f| f.as_ref().to_string()) + .collect(); + let reader_features: Vec = protocol + .reader_features() + .map(|fs| fs.iter().map(|f| f.as_ref().to_string()).collect()) + .unwrap_or_default(); + for f in expected_features { + assert!( + writer_features.iter().any(|w| w == f), + "writerFeatures missing {f}; got {writer_features:?}", + ); + let is_reader_writer = READER_WRITER_FEATURES.contains(f); + let is_writer_only = WRITER_FEATURES.contains(f); + assert!( + is_reader_writer || is_writer_only, + "feature '{f}' is not classified in READER_WRITER_FEATURES or WRITER_FEATURES; \ + update one of those lists", + ); + if is_reader_writer { + assert!( + reader_features.iter().any(|r| r == f), + "readerFeatures missing {f} (reader+writer feature); got {reader_features:?}", + ); + } else { + assert!( + !reader_features.iter().any(|r| r == f), + "readerFeatures unexpectedly contains writer-only feature {f}; \ + got {reader_features:?}", + ); + } + } +} + +/// Look up the physical name of a top-level logical column. Panics if not found. +fn get_top_level_physical_name( + schema: &StructType, + logical: &str, + mode: ColumnMappingMode, +) -> String { + let physical = get_any_level_column_physical_name(schema, &ColumnName::new([logical]), mode) + .unwrap() + .into_inner(); + physical + .into_iter() + .next() + .unwrap_or_else(|| panic!("no physical path for column {logical}")) +} + +/// Assert that every column-mapping ID lands at the right `parquet_schema_path` in the parquet +/// schema. Walks the kernel schema, builds an expected `parquet_schema_path -> id` map (from +/// both `delta.columnMapping.id` and `delta.columnMapping.nested.ids`), then for each entry +/// asserts the parquet file has the same id at that path. +/// +/// A `parquet_schema_path` is the dot-joined path of a field from the parquet schema root. +/// +/// Example. A kernel schema with one column `m: map, int>` carrying: +/// ```text +/// delta.columnMapping.id = 1 +/// delta.columnMapping.physicalName = "col-X" +/// delta.columnMapping.nested.ids = { +/// "col-X.key": 2, +/// "col-X.key.element": 3, +/// "col-X.value": 4, +/// } +/// ``` +/// produces the expected `parquet_schema_path -> id` map: +/// ```text +/// { +/// "col-X": 1, +/// "col-X.key_value.key": 2, +/// "col-X.key_value.key.list.element": 3, +/// "col-X.key_value.value": 4, +/// } +/// ``` +fn verify_column_mapping_ids_in_parquet( + logical_schema: &StructType, + mode: ColumnMappingMode, + parquet_ids: &HashMap, + parquet_url: &Url, + expected_num_ids: usize, +) { + let mut visitor = ExpectedFieldIdMap { + mode, + parquet_schema_path_stack: Vec::new(), + expected: HashMap::new(), + }; + visitor.transform_struct(logical_schema); + assert_eq!( + parquet_ids.len(), + expected_num_ids, + "expected {expected_num_ids} field IDs in {parquet_url}, got {}: {parquet_ids:?}", + parquet_ids.len(), + ); + for (parquet_schema_path, expected_id) in &visitor.expected { + let actual_id = parquet_ids + .get(parquet_schema_path) + .copied() + .unwrap_or_else(|| { + panic!( + "expected field_id {expected_id} at parquet_schema_path \ + '{parquet_schema_path}' in {parquet_url}; actual ids: {parquet_ids:?}", + ) + }); + assert_eq!( + actual_id, *expected_id, + "parquet field_id mismatch at parquet_schema_path \ + '{parquet_schema_path}' in {parquet_url}: expected {expected_id}, got {actual_id}", + ); + } +} + +/// Schema visitor that builds `parquet_schema_path -> id` for +/// [`verify_column_mapping_ids_in_parquet`]. +struct ExpectedFieldIdMap { + mode: ColumnMappingMode, + parquet_schema_path_stack: Vec, + expected: HashMap, +} + +impl ExpectedFieldIdMap { + fn current_parquet_schema_path(&self) -> String { + self.parquet_schema_path_stack.join(".") + } +} + +impl<'a> SchemaTransform<'a> for ExpectedFieldIdMap { + transform_output_type!(|'a, T| ()); + + fn transform_struct_field(&mut self, field: &'a StructField) { + self.parquet_schema_path_stack + .push(field.physical_name(self.mode).to_string()); + let field_parquet_schema_path = self.current_parquet_schema_path(); + if let Some(id) = field.column_mapping_id() { + self.expected.insert(field_parquet_schema_path.clone(), id); + } + if let Some(MetadataValue::Other(json)) = + field.get_config_value(&ColumnMetadataKey::ColumnMappingNestedIds) + { + if let Some(obj) = json.as_object() { + // Each key is a relative path rooted at the field's physical name, e.g. + // `.key`, `.key.element`. We translate it to its corresponding + // `parquet_schema_path`. + for (physical_nested_path, value) in obj { + if let Some(id) = value.as_i64() { + self.expected.insert( + translate_nested_path(&field_parquet_schema_path, physical_nested_path), + id, + ); + } + } + } + } + self.recurse_into_struct_field(field); + self.parquet_schema_path_stack.pop(); + } + + fn transform_map(&mut self, mtype: &'a MapType) { + self.parquet_schema_path_stack.push("key_value".to_string()); + self.parquet_schema_path_stack.push("key".to_string()); + self.transform_map_key(&mtype.key_type); + self.parquet_schema_path_stack.pop(); + self.parquet_schema_path_stack.push("value".to_string()); + self.transform_map_value(&mtype.value_type); + self.parquet_schema_path_stack.pop(); + self.parquet_schema_path_stack.pop(); + } + + fn transform_array(&mut self, atype: &'a ArrayType) { + self.parquet_schema_path_stack.push("list".to_string()); + self.parquet_schema_path_stack.push("element".to_string()); + self.transform_array_element(&atype.element_type); + self.parquet_schema_path_stack.pop(); + self.parquet_schema_path_stack.pop(); + } + + // Variant has its own internal struct (metadata/value) but the kernel parquet writer + // does not carry column-mapping IDs into it; skip recursion to avoid recording paths + // that don't correspond to parquet field IDs. + fn transform_variant(&mut self, _stype: &'a StructType) {} +} + +/// Translate a `delta.columnMapping.nested.ids` key (e.g. `.key.element`) to its +/// `parquet_schema_path` for map/array element/key/value fields. +fn translate_nested_path(field_parquet_schema_path: &str, physical_nested_path: &str) -> String { + let segs: Vec<&str> = physical_nested_path.split('.').collect(); + let mut out = field_parquet_schema_path.to_string(); + for seg in &segs[1..] { + match *seg { + "key" => out.push_str(".key_value.key"), + "value" => out.push_str(".key_value.value"), + "element" => out.push_str(".list.element"), + other => { + panic!("unexpected nested-id path segment '{other}' in '{physical_nested_path}'") + } + } + } + out +} + +/// Find a top-level field by physical name in a parquet root schema, or `None` if absent. +fn find_top_level_field_in_parquet<'a>( + root: &'a ParquetType, + name: &str, +) -> Option<&'a ParquetType> { + root.get_fields() + .iter() + .find(|f| f.name() == name) + .map(|f| f.as_ref()) +} + +/// Get the parquet physical type of a top-level field (looked up by physical name). Panics if +/// the field is missing or is a group (i.e. not a primitive leaf). +fn find_top_level_physical_type_in_parquet(root: &ParquetType, name: &str) -> ParquetPhysicalType { + let field = find_top_level_field_in_parquet(root, name) + .unwrap_or_else(|| panic!("top-level field {name} not found in parquet schema")); + match field { + ParquetType::PrimitiveType { physical_type, .. } => *physical_type, + ParquetType::GroupType { .. } => { + panic!("top-level field {name} is a group, expected primitive leaf") + } + } +} + +/// Run a full scan and assert exact `(region, int_col, bool_col)` content for every row, plus +/// that `variant_col` and `complex` are entirely null. `int_col` derives from the writer's +/// `random_seed`, so the expected list is reproduced by replaying the same seeds (0..4) and +/// regions; sorting both sides by `(region, int_col)` makes the comparison order-independent. +fn verify_scan_contents( + snapshot: Arc, + engine: Arc>, +) { + let scan = snapshot.scan_builder().build().unwrap(); + let mut rows: Vec<(String, i32, bool)> = Vec::new(); + let mut variant_null_count = 0_usize; + let mut complex_null_count = 0_usize; + let mut total_rows = 0_usize; + + for res in scan.execute(engine).unwrap() { + let batch = into_record_batch(res.unwrap()); + + let region_arr = batch + .column_by_name("region") + .and_then(|c| c.as_any().downcast_ref::()) + .expect("region column missing or wrong type"); + let int_arr = batch + .column_by_name("int_col") + .and_then(|c| c.as_any().downcast_ref::()) + .expect("int_col column missing or wrong type"); + let bool_arr = batch + .column_by_name("bool_col") + .and_then(|c| c.as_any().downcast_ref::()) + .expect("bool_col column missing or wrong type"); + let variant_arr = batch + .column_by_name("variant_col") + .expect("variant_col column missing"); + let complex_arr = batch + .column_by_name("complex") + .expect("complex column missing"); + + for i in 0..batch.num_rows() { + rows.push(( + region_arr.value(i).to_string(), + int_arr.value(i), + bool_arr.value(i), + )); + if variant_arr.is_null(i) { + variant_null_count += 1; + } + if complex_arr.is_null(i) { + complex_null_count += 1; + } + total_rows += 1; + } + } + + // Sort by (region, int_col): same `random_seed` produces equal `int_col` across regions, + // so `(region, int_col)` is the unique row key. + rows.sort(); + + let mut expected: Vec<(String, i32, bool)> = Vec::new(); + for random_seed in 0..4_i32 { + for region in PARTITION_REGIONS { + for i in 0..ROWS_PER_PARTITION { + expected.push((region.to_string(), random_seed + i, i % 2 == 0)); + } + } + } + expected.sort(); + + let total_expected = expected.len(); + assert_eq!(rows, expected, "scan contents mismatch"); + assert_eq!(total_rows, total_expected); + assert_eq!( + variant_null_count, total_expected, + "all variant_col values must be null", + ); + assert_eq!( + complex_null_count, total_expected, + "all complex (map) values must be null", + ); +} diff --git a/kernel/tests/integration/features/interval.rs b/kernel/tests/integration/features/interval.rs new file mode 100644 index 0000000000..25875ac9d9 --- /dev/null +++ b/kernel/tests/integration/features/interval.rs @@ -0,0 +1,26 @@ +//! Reader-side behavior for ANSI interval columns. + +use std::sync::Arc; + +use delta_kernel::schema::{DataType, StructField, StructType}; +use delta_kernel::Snapshot; +use test_utils::{create_table, engine_store_setup}; + +#[rstest::rstest] +#[case::year_month("interval_read_ym", DataType::INTERVAL_YEAR_MONTH)] +#[case::day_time("interval_read_dt", DataType::INTERVAL_DAY_TIME)] +#[tokio::test] +async fn test_build_scan_over_interval_table( + #[case] name: &str, + #[case] interval: DataType, +) -> Result<(), Box> { + let schema = Arc::new(StructType::try_new(vec![StructField::nullable( + "iv", interval, + )])?); + let (store, engine, table_location) = engine_store_setup(name, None); + let table_url = create_table(store, table_location, schema, &[], true, vec![], vec![]).await?; + + let snapshot = Snapshot::builder_for(table_url).build(&engine)?; + snapshot.scan_builder().build()?; + Ok(()) +} diff --git a/kernel/tests/integration/features/maintenance_ops.rs b/kernel/tests/integration/features/maintenance_ops.rs new file mode 100644 index 0000000000..02759aa728 --- /dev/null +++ b/kernel/tests/integration/features/maintenance_ops.rs @@ -0,0 +1,107 @@ +//! Integration tests for table maintenance operations (checkpoint, checksum). + +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::schema::schema_ref; +use delta_kernel::snapshot::{CheckpointWriteResult, ChecksumWriteResult}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::{DeltaResult, Snapshot}; +use rstest::rstest; +use test_utils::test_table_setup_mt; + +#[rstest] +#[case::v1_checkpoint(false)] +#[case::v2_checkpoint(true)] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_checkpoint_and_checksum_return_updated_snapshots( + #[case] v2_checkpoint: bool, +) -> DeltaResult<()> { + // ===== GIVEN ===== + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let schema = schema_ref! { nullable "id": INTEGER }; + let mut builder = create_table(&table_path, schema, "test_engine"); + if v2_checkpoint { + builder = builder.with_table_properties([("delta.feature.v2Checkpoint", "supported")]); + } + let committed = builder + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + let snapshot = committed.post_commit_snapshot().unwrap(); + + // Precondition: no checkpoint, no CRC + let seg = snapshot.log_segment(); + assert!(seg.listed.checkpoint_parts.is_empty()); + assert!(seg.checkpoint_version.is_none()); + assert!(seg.listed.latest_crc_file.is_none()); + + // ===== WHEN: we checkpoint ===== + let (_, snapshot_w_ckpt) = snapshot.checkpoint(engine.as_ref(), None)?; + let seg = snapshot_w_ckpt.log_segment(); + + // ===== THEN ===== + // Checkpoint version and parts are set + assert_eq!(seg.checkpoint_version, Some(snapshot.version())); + assert_eq!(seg.listed.checkpoint_parts.len(), 1); + assert_eq!(seg.listed.checkpoint_parts[0].version, snapshot.version()); + + // Commits and compactions subsumed by the checkpoint are cleared + assert!(seg.listed.ascending_commit_files.is_empty()); + assert!(seg.listed.ascending_compaction_files.is_empty()); + + // ===== WHEN: we write checksum ===== + let (crc_result, snapshot_w_both) = snapshot_w_ckpt.write_checksum(engine.as_ref())?; + let seg = snapshot_w_both.log_segment(); + + // ===== THEN ===== + // CRC file is recorded at the correct version + assert_eq!(crc_result, ChecksumWriteResult::Written); + let crc_file = seg + .listed + .latest_crc_file + .as_ref() + .expect("snapshot should have latest_crc_file set"); + assert_eq!(crc_file.version, snapshot.version()); + + // The checkpoint is still present after the CRC write + assert_eq!(seg.checkpoint_version, Some(snapshot.version())); + + Ok(()) +} + +#[rstest] +#[case::v1_checkpoint(false)] +#[case::v2_checkpoint(true)] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_checkpoint_already_exists(#[case] v2_checkpoint: bool) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let schema = schema_ref! { nullable "id": INTEGER }; + let mut builder = create_table(&table_path, schema, "test_engine"); + if v2_checkpoint { + builder = builder.with_table_properties([("delta.feature.v2Checkpoint", "supported")]); + } + let committed = builder + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + let snapshot = committed.post_commit_snapshot().unwrap(); + + // First checkpoint writes successfully + let (result, snapshot_w_ckpt) = snapshot.checkpoint(engine.as_ref(), None)?; + assert_eq!(result, CheckpointWriteResult::Written); + + // Calling checkpoint again on the returned snapshot detects the existing checkpoint + let (result, unchanged) = snapshot_w_ckpt.checkpoint(engine.as_ref(), None)?; + assert_eq!(result, CheckpointWriteResult::AlreadyExists); + assert_eq!(unchanged.version(), snapshot_w_ckpt.version()); + + // A fresh snapshot loaded from storage also returns AlreadyExists + let fresh = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!( + fresh.log_segment().checkpoint_version, + Some(snapshot.version()) + ); + let (result, _) = fresh.checkpoint(engine.as_ref(), None)?; + assert_eq!(result, CheckpointWriteResult::AlreadyExists); + + Ok(()) +} diff --git a/kernel/tests/integration/features/mod.rs b/kernel/tests/integration/features/mod.rs new file mode 100644 index 0000000000..83fb757eaf --- /dev/null +++ b/kernel/tests/integration/features/mod.rs @@ -0,0 +1,11 @@ +//! Reader-side protocol-feature integration tests. + +mod alter_table; +mod cdf; +mod clustering_e2e; +mod column_mapping; +mod dv; +mod iceberg_compat; +mod interval; +mod maintenance_ops; +mod row_tracking; diff --git a/kernel/tests/row_tracking.rs b/kernel/tests/integration/features/row_tracking.rs similarity index 52% rename from kernel/tests/row_tracking.rs rename to kernel/tests/integration/features/row_tracking.rs index c5d58f291c..537df9d904 100644 --- a/kernel/tests/row_tracking.rs +++ b/kernel/tests/integration/features/row_tracking.rs @@ -1,25 +1,34 @@ use std::collections::HashMap; use std::sync::Arc; -use itertools::Itertools; -use serde_json::{Deserializer, Value}; -use tempfile::{tempdir, TempDir}; -use url::Url; - -use delta_kernel::arrow::array::{Array, Int32Array, Int64Array, StringArray}; -use delta_kernel::arrow::datatypes::Schema as ArrowSchema; +use delta_kernel::actions::deletion_vector_writer::KernelDeletionVector; +use delta_kernel::arrow::array::{Array, AsArray, Int32Array, Int64Array, StringArray}; +use delta_kernel::arrow::datatypes::{Int32Type, Int64Type, Schema as ArrowSchema}; use delta_kernel::arrow::record_batch::RecordBatch; -use delta_kernel::committer::FileSystemCommitter; use delta_kernel::engine::arrow_conversion::TryIntoArrow; use delta_kernel::engine::arrow_data::ArrowEngineData; -use delta_kernel::engine::default::executor::tokio::TokioBackgroundExecutor; -use delta_kernel::engine::default::DefaultEngine; -use delta_kernel::object_store::{path::Path, DynObjectStore, ObjectStore as _}; -use delta_kernel::schema::{DataType, SchemaRef, StructField, StructType}; +use delta_kernel::engine::to_json_bytes; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::{DynObjectStore, ObjectStoreExt}; +use delta_kernel::schema::{DataType, MetadataColumnSpec, SchemaRef, StructField, StructType}; use delta_kernel::transaction::CommitResult; use delta_kernel::{DeltaResult, Error, Snapshot}; +use itertools::Itertools; +use rstest::rstest; +use serde_json::{Deserializer, Value}; +use tempfile::{tempdir, TempDir}; +use test_utils::delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor; +use test_utils::delta_kernel_default_engine::DefaultEngine; +use test_utils::{ + begin_transaction, collect_row_ids, create_default_engine_mt_executor, create_table, + engine_store_setup, load_and_begin_transaction, read_actions_from_commit, read_add_infos, + read_scan, test_read, +}; +use url::Url; -use test_utils::{create_table, engine_store_setup, read_scan, test_read}; +use crate::common::write_utils::{ + create_dv_update_transaction, get_scan_files, write_deletion_vector_to_store, +}; /// Helper function to create a simple table with row tracking enabled. async fn create_row_tracking_table( @@ -30,20 +39,39 @@ async fn create_row_tracking_table( Url, Arc>, Arc, +)> { + create_row_tracking_table_with_features(tmp_dir, table_name, schema, &[], &[]).await +} + +/// Helper function to create a row-tracking table with additional features +async fn create_row_tracking_table_with_features( + tmp_dir: &TempDir, + table_name: &str, + schema: SchemaRef, + extra_reader_writer_features: &[&str], + extra_writer_features: &[&str], +) -> DeltaResult<( + Url, + Arc>, + Arc, )> { let tmp_test_dir_url = Url::from_directory_path(tmp_dir.path()) .map_err(|_| Error::generic("Failed to convert directory path to URL"))?; let (store, engine, table_location) = engine_store_setup(table_name, Some(&tmp_test_dir_url)); - // Create table with row tracking feature enabled + let reader_features = extra_reader_writer_features.to_vec(); + let mut writer_features = vec!["domainMetadata", "rowTracking"]; + writer_features.extend_from_slice(extra_reader_writer_features); + writer_features.extend_from_slice(extra_writer_features); + let table_url = create_table( store.clone(), table_location, schema, - &[], // no partition columns - true, // use 37 protocol - vec![], // no reader features - vec!["domainMetadata", "rowTracking"], + &[], // no partition columns + true, // use 37 protocol + reader_features, + writer_features, ) .await .map_err(|e| Error::generic(format!("Failed to create table: {e}")))?; @@ -57,22 +85,15 @@ async fn write_data_to_table( engine: Arc>, data: Vec, ) -> DeltaResult { - let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; - let committer = Box::new(FileSystemCommitter::new()); - let mut txn = snapshot - .transaction(committer, engine.as_ref())? - .with_data_change(true); + let mut txn = + load_and_begin_transaction(table_url.clone(), engine.as_ref())?.with_data_change(true); // Write data out by spawning async tasks to simulate executors - let write_context = Arc::new(txn.get_write_context()); + let write_context = Arc::new(txn.unpartitioned_write_context()?); let tasks = data.into_iter().map(|data| { let engine = engine.clone(); let write_context = write_context.clone(); - tokio::task::spawn(async move { - engine - .write_parquet(&data, write_context.as_ref(), HashMap::new()) - .await - }) + tokio::task::spawn(async move { engine.write_parquet(&data, write_context.as_ref()).await }) }); let add_files_metadata = futures::future::join_all(tasks).await.into_iter().flatten(); @@ -86,6 +107,47 @@ async fn write_data_to_table( txn.commit(engine.as_ref()) } +/// Helper function to create a row-tracking table with a single `number: INTEGER` column. +async fn setup_number_table( + tmp_dir: &TempDir, + name: &str, +) -> DeltaResult<( + SchemaRef, + Url, + Arc>, + Arc, +)> { + setup_number_table_with_features(tmp_dir, name, &[], &[]).await +} + +/// Helper function to create a row-tracking table with a single `number: INTEGER` column and +/// additional features enabled. +async fn setup_number_table_with_features( + tmp_dir: &TempDir, + name: &str, + extra_reader_writer_features: &[&str], + extra_writer_features: &[&str], +) -> DeltaResult<( + SchemaRef, + Url, + Arc>, + Arc, +)> { + let schema = Arc::new(StructType::try_new(vec![StructField::nullable( + "number", + DataType::INTEGER, + )])?); + let (table_url, engine, store) = create_row_tracking_table_with_features( + tmp_dir, + name, + schema.clone(), + extra_reader_writer_features, + extra_writer_features, + ) + .await?; + Ok((schema, table_url, engine, store)) +} + /// Helper function to create an Arc from an i32 vector. fn int32_array(data: Vec) -> Arc { Arc::new(Int32Array::from(data)) @@ -205,13 +267,8 @@ async fn test_row_tracking_append() -> DeltaResult<()> { // Setup let _ = tracing_subscriber::fmt::try_init(); let tmp_test_dir = tempdir()?; - let schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "number", - DataType::INTEGER, - )])?); - - let (table_url, engine, store) = - create_row_tracking_table(&tmp_test_dir, "test_append", schema.clone()).await?; + let (schema, table_url, engine, store) = + setup_number_table(&tmp_test_dir, "test_append").await?; // Create two new arrow record batches to append let data = generate_data( @@ -253,13 +310,8 @@ async fn test_row_tracking_single_record_batches() -> DeltaResult<()> { // Setup let _ = tracing_subscriber::fmt::try_init(); let tmp_test_dir = tempdir()?; - let schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "number", - DataType::INTEGER, - )])?); - - let (table_url, engine, store) = - create_row_tracking_table(&tmp_test_dir, "test_single_records", schema.clone()).await?; + let (schema, table_url, engine, store) = + setup_number_table(&tmp_test_dir, "test_single_records").await?; // Write individual records in separate batches let data = generate_data( @@ -292,13 +344,8 @@ async fn test_row_tracking_large_batch() -> DeltaResult<()> { // Setup let _ = tracing_subscriber::fmt::try_init(); let tmp_test_dir = tempdir()?; - let schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "number", - DataType::INTEGER, - )])?); - - let (table_url, engine, store) = - create_row_tracking_table(&tmp_test_dir, "test_large_batch", schema.clone()).await?; + let (schema, table_url, engine, store) = + setup_number_table(&tmp_test_dir, "test_large_batch").await?; // Write a large batch with 1000 records let large_batch: Vec = (1..=1000).collect(); @@ -335,14 +382,8 @@ async fn test_row_tracking_consecutive_transactions() -> DeltaResult<()> { // Setup let _ = tracing_subscriber::fmt::try_init(); let tmp_test_dir = tempdir()?; - let schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "number", - DataType::INTEGER, - )])?); - - let (table_url, engine, store) = - create_row_tracking_table(&tmp_test_dir, "test_consecutive_commits", schema.clone()) - .await?; + let (schema, table_url, engine, store) = + setup_number_table(&tmp_test_dir, "test_consecutive_commits").await?; // First transaction: write two batches with 3 records each let data_1 = generate_data( @@ -367,7 +408,8 @@ async fn test_row_tracking_consecutive_transactions() -> DeltaResult<()> { .await?; // Second transaction: write one batch with 2 records - // This should read the existing row tracking domain metadata and assign base row IDs starting from 6 + // This should read the existing row tracking domain metadata and assign base row IDs starting + // from 6 let data_2 = generate_data(schema.clone(), [vec![int32_array(vec![7, 8])]])?; assert!(write_data_to_table(&table_url, engine.clone(), data_2) .await? @@ -493,13 +535,8 @@ async fn test_row_tracking_with_regular_and_empty_adds() -> DeltaResult<()> { // Setup let _ = tracing_subscriber::fmt::try_init(); let tmp_test_dir = tempdir()?; - let schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "number", - DataType::INTEGER, - )])?); - - let (table_url, engine, store) = - create_row_tracking_table(&tmp_test_dir, "test_append", schema.clone()).await?; + let (schema, table_url, engine, store) = + setup_number_table(&tmp_test_dir, "test_append").await?; // Create two regular and one empty arrow record batches to append let data = generate_data( @@ -542,13 +579,8 @@ async fn test_row_tracking_with_empty_adds() -> DeltaResult<()> { // Setup let _ = tracing_subscriber::fmt::try_init(); let tmp_test_dir = tempdir()?; - let schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "number", - DataType::INTEGER, - )])?); - - let (table_url, engine, store) = - create_row_tracking_table(&tmp_test_dir, "test_append", schema.clone()).await?; + let (schema, table_url, engine, store) = + setup_number_table(&tmp_test_dir, "test_append").await?; // Create two new _empty_ arrow record batches to append let data = generate_data( @@ -563,10 +595,10 @@ async fn test_row_tracking_with_empty_adds() -> DeltaResult<()> { .is_committed()); // Verify the commit was written correctly - // NB: The expected high water mark is a bit unintuitive here, as we are appending empty batches. - // Appending empty batches means that we assign the same base row ID multiple times and that the - // high water mark is lower than the last assigned base row ID (because that base row ID has no - // actual row attached to it). + // NB: The expected high water mark is a bit unintuitive here, as we are appending empty + // batches. Appending empty batches means that we assign the same base row ID multiple times + // and that the high water mark is lower than the last assigned base row ID (because that + // base row ID has no actual row attached to it). verify_row_tracking_in_commit( &store, &table_url, @@ -591,16 +623,9 @@ async fn test_row_tracking_without_adds() -> DeltaResult<()> { // Setup let _ = tracing_subscriber::fmt::try_init(); let tmp_test_dir = tempdir()?; - let schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "number", - DataType::INTEGER, - )])?); - - let (table_url, engine, store) = - create_row_tracking_table(&tmp_test_dir, "test_consecutive_commits", schema.clone()) - .await?; - let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; - let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())?; + let (_schema, table_url, engine, store) = + setup_number_table(&tmp_test_dir, "test_consecutive_commits").await?; + let txn = load_and_begin_transaction(table_url.clone(), engine.as_ref())?; // Commit without adding any add files assert!(txn.commit(engine.as_ref())?.is_committed()); @@ -614,7 +639,8 @@ async fn test_row_tracking_without_adds() -> DeltaResult<()> { .try_collect()?; // Verify that there only is a commit info action - // NOTE: We specifically test that we don't write domain metadata for commits without actual data + // NOTE: We specifically test that we don't write domain metadata for commits without actual + // data assert_eq!(parsed_actions.len(), 1, "Expected only one action"); assert!(parsed_actions[0].get("commitInfo").is_some()); @@ -626,14 +652,8 @@ async fn test_row_tracking_parallel_transactions_conflict() -> DeltaResult<()> { // Setup let _ = tracing_subscriber::fmt::try_init(); let tmp_test_dir = tempdir()?; - let schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "number", - DataType::INTEGER, - )])?); - - let (table_url, engine, store) = - create_row_tracking_table(&tmp_test_dir, "test_parallel_row_tracking", schema.clone()) - .await?; + let (schema, table_url, engine, store) = + setup_number_table(&tmp_test_dir, "test_parallel_row_tracking").await?; let engine1 = engine.clone(); let engine2 = engine; @@ -643,12 +663,10 @@ async fn test_row_tracking_parallel_transactions_conflict() -> DeltaResult<()> { let snapshot2 = Snapshot::builder_for(table_url.clone()).build(engine2.as_ref())?; // Create two transactions from the same snapshot (simulating parallel transactions) - let mut txn1 = snapshot1 - .transaction(Box::new(FileSystemCommitter::new()), engine1.as_ref())? + let mut txn1 = begin_transaction(snapshot1, engine1.as_ref())? .with_engine_info("transaction 1") .with_data_change(true); - let mut txn2 = snapshot2 - .transaction(Box::new(FileSystemCommitter::new()), engine2.as_ref())? + let mut txn2 = begin_transaction(snapshot2, engine2.as_ref())? .with_engine_info("transaction 2") .with_data_change(true); @@ -663,23 +681,15 @@ async fn test_row_tracking_parallel_transactions_conflict() -> DeltaResult<()> { )?; // Write data for both transactions - let write_context1 = Arc::new(txn1.get_write_context()); - let write_context2 = Arc::new(txn2.get_write_context()); + let write_context1 = Arc::new(txn1.unpartitioned_write_context()?); + let write_context2 = Arc::new(txn2.unpartitioned_write_context()?); let metadata1 = engine1 - .write_parquet( - &ArrowEngineData::new(data1), - write_context1.as_ref(), - HashMap::new(), - ) + .write_parquet(&ArrowEngineData::new(data1), write_context1.as_ref()) .await?; let metadata2 = engine2 - .write_parquet( - &ArrowEngineData::new(data2), - write_context2.as_ref(), - HashMap::new(), - ) + .write_parquet(&ArrowEngineData::new(data2), write_context2.as_ref()) .await?; txn1.add_files(metadata1); @@ -847,3 +857,440 @@ async fn test_no_row_tracking_fields_without_feature() -> DeltaResult<()> { Ok(()) } + +/// Build a scan with `MetadataColumnSpec::RowId` appended to the snapshot schema and execute it. +fn read_row_id_scan( + snapshot: Arc, + engine: Arc, +) -> DeltaResult> { + let scan_schema = Arc::new( + snapshot + .schema() + .add_metadata_column("row_id", MetadataColumnSpec::RowId)?, + ); + let scan = snapshot.scan_builder().with_schema(scan_schema).build()?; + read_scan(&scan, engine) +} + +/// Basic read: write one file with 3 rows, verify row IDs are sequential starting from 0. +#[tokio::test] +async fn test_read_row_ids_basic() -> DeltaResult<()> { + let _ = tracing_subscriber::fmt::try_init(); + let tmp_dir = tempdir()?; + let (schema, table_url, engine, _store) = + setup_number_table(&tmp_dir, "test_read_row_ids_basic").await?; + + let data = generate_data(schema.clone(), [vec![int32_array(vec![10, 20, 30])]])?; + assert!(write_data_to_table(&table_url, engine.clone(), data) + .await? + .is_committed()); + + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let batches = read_row_id_scan(snapshot, engine)?; + + let mut row_ids = collect_row_ids(&batches); + row_ids.sort_unstable(); + assert_eq!(row_ids, vec![0, 1, 2], "Row IDs must be sequential from 0"); + + Ok(()) +} + +/// Collect `(number, row_id)` pairs from a row-id scan, keyed by the `number` data column. +fn collect_number_to_row_id(batches: &[RecordBatch]) -> HashMap { + let mut map = HashMap::new(); + for batch in batches { + let numbers = batch + .column_by_name("number") + .expect("number column not found") + .as_primitive::(); + let row_ids = batch + .column_by_name("row_id") + .expect("row_id column not found") + .as_primitive::(); + for i in 0..batch.num_rows() { + map.insert(numbers.value(i), row_ids.value(i)); + } + } + map +} + +/// Deletion vector: must not renumber the surviving rows' stable row IDs and not change any row +/// tracking metadata. +#[rstest] +#[case::middle(&[4, 5, 6])] +#[case::first(&[0, 1, 2])] +#[case::last(&[7, 8, 9])] +#[case::first_and_last(&[0, 1, 8, 9])] +#[case::first_middle_last(&[0, 4, 5, 9])] +#[tokio::test] +async fn test_read_row_ids_stable_across_deletion_vector_update( + #[case] deleted_indexes: &[u64], +) -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + let tmp_dir = tempdir()?; + let (schema, table_url, engine, store) = setup_number_table_with_features( + &tmp_dir, + "test_read_row_ids_stable_across_dv", + &["deletionVectors"], + &[], + ) + .await?; + + // Write a single file with 10 rows: values 100..=109 at physical indexes 0..=9 + let data = generate_data( + schema.clone(), + [vec![int32_array((100..110).collect::>())]], + )?; + write_data_to_table(&table_url, engine.clone(), data) + .await? + .unwrap_committed(); + + // Snapshot the (value -> row_id) mapping before any deletion. value v sits at physical index + // (v - 100), and with baseRowId 0 that is also its row ID. + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let before = collect_number_to_row_id(&read_row_id_scan(snapshot.clone(), engine.clone())?); + assert_eq!( + before, + (100..110) + .map(|v| (v, (v - 100) as i64)) + .collect::>(), + "row IDs must equal each row's physical index before deletion" + ); + + // The original Add's row-tracking fields, to confirm they survive the DV update unchanged. + let v1_adds = read_actions_from_commit(&table_url, 1, "add")?; + let original_add = &v1_adds[0]; + let original_base_row_id = original_add["baseRowId"].as_i64(); + let original_default_row_commit_version = original_add["defaultRowCommitVersion"].as_i64(); + assert_eq!(original_base_row_id, Some(0)); + assert_eq!(original_default_row_commit_version, Some(1)); + + // Apply a deletion vector that removes the parameterized physical indexes (value v sits at + // physical index v - 100). + let mut dv = KernelDeletionVector::new(); + dv.add_deleted_row_indexes(deleted_indexes.iter().copied()); + let mut txn = create_dv_update_transaction(&table_url, engine.as_ref())?; + let write_context = txn.unpartitioned_write_context()?; + let dv_descriptor = write_deletion_vector_to_store(&store, &write_context, dv, "").await?; + + let file_path = read_add_infos(snapshot.as_ref(), engine.as_ref())?[0] + .path + .clone(); + txn.update_deletion_vectors( + HashMap::from([(file_path, dv_descriptor)]), + get_scan_files(snapshot.clone(), engine.as_ref())? + .into_iter() + .map(Ok), + )?; + txn.commit(engine.as_ref())?.unwrap_committed(); + + // Every survivor keeps the exact row ID it had before. + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let after = collect_number_to_row_id(&read_row_id_scan(snapshot, engine.clone())?); + + let expected_survivors: HashMap = (100..110) + .filter(|v| !deleted_indexes.contains(&((v - 100) as u64))) + .map(|v| (v, (v - 100) as i64)) + .collect(); + assert_eq!( + after, expected_survivors, + "surviving rows must keep their original row IDs, not be renumbered" + ); + + // The DV update must preserve the original row-tracking fields on the rewritten Add. + let v2_adds = read_actions_from_commit(&table_url, 2, "add")?; + let updated_add = &v2_adds[0]; + assert_eq!(updated_add["baseRowId"].as_i64(), original_base_row_id); + assert_eq!( + updated_add["defaultRowCommitVersion"].as_i64(), + original_default_row_commit_version, + ); + + Ok(()) +} + +/// Multiple files in one commit: each file's row IDs start at its baseRowId with no overlap. +#[tokio::test] +async fn test_read_row_ids_multiple_files_one_commit() -> DeltaResult<()> { + let _ = tracing_subscriber::fmt::try_init(); + let tmp_dir = tempdir()?; + let (schema, table_url, engine, _store) = + setup_number_table(&tmp_dir, "test_read_row_ids_multiple_files").await?; + + // Two files: 3 rows (baseRowId=0) and 4 rows (baseRowId=3). + let data = generate_data( + schema.clone(), + [ + vec![int32_array(vec![1, 2, 3])], + vec![int32_array(vec![4, 5, 6, 7])], + ], + )?; + assert!(write_data_to_table(&table_url, engine.clone(), data) + .await? + .is_committed()); + + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let batches = read_row_id_scan(snapshot, engine)?; + + let mut all_ids = collect_row_ids(&batches); + all_ids.sort_unstable(); + + // 7 rows total, IDs must be 0..=6 with no duplicates. + assert_eq!( + all_ids, + (0i64..7).collect::>(), + "Row IDs must be non-overlapping across files" + ); + + // Each batch's IDs must form a contiguous block starting at its baseRowId. + for batch in &batches { + let ids: Vec = batch + .column_by_name("row_id") + .expect("row_id column not found") + .as_primitive::() + .values() + .to_vec(); + let min = *ids.iter().min().unwrap(); + let expected = (min..min + ids.len() as i64).collect::>(); + assert_eq!(ids, expected, "IDs within a file must be contiguous"); + } + + Ok(()) +} + +/// Multiple commits: row IDs are globally unique and monotonically increasing across commits. +#[tokio::test] +async fn test_read_row_ids_multiple_commits() -> DeltaResult<()> { + let _ = tracing_subscriber::fmt::try_init(); + let tmp_dir = tempdir()?; + let (schema, table_url, engine, _store) = + setup_number_table(&tmp_dir, "test_read_row_ids_multiple_commits").await?; + + // Commit 1: 3 rows -> IDs 0, 1, 2. + let data1 = generate_data(schema.clone(), [vec![int32_array(vec![1, 2, 3])]])?; + assert!(write_data_to_table(&table_url, engine.clone(), data1) + .await? + .is_committed()); + + // Commit 2: 2 rows -> IDs 3, 4. + let data2 = generate_data(schema.clone(), [vec![int32_array(vec![4, 5])]])?; + assert!(write_data_to_table(&table_url, engine.clone(), data2) + .await? + .is_committed()); + + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let batches = read_row_id_scan(snapshot, engine)?; + + let mut all_ids = collect_row_ids(&batches); + all_ids.sort_unstable(); + + // 5 rows total, IDs must be 0..=4 with no duplicates or gaps. + assert_eq!( + all_ids, + vec![0, 1, 2, 3, 4], + "Row IDs must be globally unique and monotonically increasing across commits" + ); + + Ok(()) +} + +/// After checkpoint: row IDs survive a checkpoint and new writes continue from the high watermark. +/// +/// Uses a multi-threaded runtime and `TokioMultiThreadExecutor` for checkpoint because +/// `checkpoint()` consumes a lazy iterator inside a `block_on()` future where each item read +/// triggers another `block_on()`. With `TokioBackgroundExecutor` this causes nested blocking on +/// the same background thread (deadlock). `TokioMultiThreadExecutor` uses `block_in_place()` +/// instead, which requires a multi-threaded runtime to delegate work to other workers. +/// Writes use the standard `TokioBackgroundExecutor` engine, matching all other tests in this file. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_read_row_ids_after_checkpoint() -> DeltaResult<()> { + let _ = tracing_subscriber::fmt::try_init(); + let tmp_dir = tempdir()?; + let (schema, table_url, engine, _store) = + setup_number_table(&tmp_dir, "test_read_row_ids_after_checkpoint").await?; + + // Write 3 rows -> IDs 0, 1, 2. + let data = generate_data(schema.clone(), [vec![int32_array(vec![1, 2, 3])]])?; + assert!(write_data_to_table(&table_url, engine.clone(), data) + .await? + .is_committed()); + + // Checkpoint at version 1. Uses TokioMultiThreadExecutor to avoid the nested block_on + // deadlock that occurs with TokioBackgroundExecutor during checkpoint's lazy I/O. + let mt_engine = create_default_engine_mt_executor(&table_url)?; + let snapshot = Snapshot::builder_for(table_url.clone()).build(mt_engine.as_ref())?; + snapshot.checkpoint(mt_engine.as_ref(), None)?; + + // Fresh snapshot loaded from the checkpoint must return the same row IDs. + let fresh_snapshot = Snapshot::builder_for(table_url.clone()).build(mt_engine.as_ref())?; + let batches = read_row_id_scan(fresh_snapshot, mt_engine.clone())?; + + let mut ids_after_ckpt = collect_row_ids(&batches); + ids_after_ckpt.sort_unstable(); + assert_eq!( + ids_after_ckpt, + vec![0, 1, 2], + "Row IDs must be unchanged after loading from checkpoint" + ); + + // Write 2 more rows -> must continue from watermark, no resets or duplicates. + let data2 = generate_data(schema.clone(), [vec![int32_array(vec![4, 5])]])?; + assert!(write_data_to_table(&table_url, engine.clone(), data2) + .await? + .is_committed()); + + let snapshot2 = Snapshot::builder_for(table_url.clone()).build(mt_engine.as_ref())?; + let batches2 = read_row_id_scan(snapshot2, mt_engine)?; + + let mut all_ids = collect_row_ids(&batches2); + all_ids.sort_unstable(); + assert_eq!( + all_ids, + vec![0, 1, 2, 3, 4], + "Row IDs must continue from the high watermark after checkpoint with no resets or duplicates" + ); + + Ok(()) +} + +/// Row IDs coexist with row index: both columns are correct when requested together. +/// +/// Row index is file-local (resets to 0 per file); row ID is global (baseRowId + row_index). +#[tokio::test] +async fn test_read_row_ids_coexist_with_row_index() -> DeltaResult<()> { + let _ = tracing_subscriber::fmt::try_init(); + let tmp_dir = tempdir()?; + let (schema, table_url, engine, _store) = + setup_number_table(&tmp_dir, "test_read_row_ids_coexist_with_row_index").await?; + + // Two files: 3 rows (baseRowId=0) and 2 rows (baseRowId=3). + let data = generate_data( + schema.clone(), + [ + vec![int32_array(vec![1, 2, 3])], + vec![int32_array(vec![4, 5])], + ], + )?; + assert!(write_data_to_table(&table_url, engine.clone(), data) + .await? + .is_committed()); + + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let scan_schema = Arc::new( + snapshot + .schema() + .add_metadata_column("row_id", MetadataColumnSpec::RowId)? + .add_metadata_column("row_index", MetadataColumnSpec::RowIndex)?, + ); + let scan = snapshot.scan_builder().with_schema(scan_schema).build()?; + let batches = read_scan(&scan, engine)?; + + for batch in &batches { + assert_eq!( + batch.num_columns(), + 3, + "Expected number | row_id | row_index" + ); + + let row_ids: Vec = batch + .column_by_name("row_id") + .expect("row_id column not found") + .as_primitive::() + .values() + .to_vec(); + let row_indexes: Vec = batch + .column_by_name("row_index") + .expect("row_index column not found") + .as_primitive::() + .values() + .to_vec(); + + let n = batch.num_rows() as i64; + + // Row index is file-local: always 0..n. + assert_eq!( + row_indexes, + (0..n).collect::>(), + "Row index must reset to 0 for each file" + ); + + // Row ID = baseRowId + row_index. Since row_index starts at 0, the minimum row ID in + // this batch is the baseRowId, and IDs within the batch must be contiguous. + let base = *row_ids.iter().min().unwrap(); + assert_eq!( + row_ids, + (base..base + n).collect::>(), + "Row IDs within a file must equal baseRowId + row_index" + ); + } + + // All row IDs across both files must cover 0..=4 with no duplicates. + let mut all_ids = collect_row_ids(&batches); + all_ids.sort_unstable(); + assert_eq!( + all_ids, + vec![0, 1, 2, 3, 4], + "Row IDs must be globally unique when coexisting with row index" + ); + + Ok(()) +} + +/// After log compaction: row IDs are preserved in the compacted log and scan correctly. +/// +/// Writes data across two commits, creates a log compaction file covering all commits +/// (versions 0..=2), loads a fresh snapshot, and verifies that row IDs still read back +/// correctly through the scan path. This ensures that `baseRowId` and row-tracking domain +/// metadata survive compaction without being dropped or corrupted. +#[tokio::test] +#[ignore = "log compaction is not yet supported, tracked in #2337"] +async fn test_read_row_ids_after_log_compaction() -> DeltaResult<()> { + let _ = tracing_subscriber::fmt::try_init(); + let tmp_dir = tempdir()?; + let (schema, table_url, engine, store) = + setup_number_table(&tmp_dir, "test_read_row_ids_after_log_compaction").await?; + + // Commit 1: 3 rows -> IDs 0, 1, 2. + let data1 = generate_data(schema.clone(), [vec![int32_array(vec![1, 2, 3])]])?; + assert!(write_data_to_table(&table_url, engine.clone(), data1) + .await? + .is_committed()); + + // Commit 2: 2 rows -> IDs 3, 4. + let data2 = generate_data(schema.clone(), [vec![int32_array(vec![4, 5])]])?; + assert!(write_data_to_table(&table_url, engine.clone(), data2) + .await? + .is_committed()); + + // Create a log compaction file spanning all commits so far (versions 0..=2). + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let mut writer = snapshot.log_compaction_writer(0, 2)?; + let compaction_path = writer.compaction_path().clone(); + let batches = writer + .compaction_data(engine.as_ref())? + .collect::>>()?; + + let json_bytes = to_json_bytes(batches.into_iter().map(Ok))?; + store + .put( + &Path::from_url_path(compaction_path.path())?, + json_bytes.into(), + ) + .await + .map_err(|e| Error::generic(e.to_string()))?; + + // Load a fresh snapshot -- it should read Protocol and Metadata and file list from the + // compaction file. + let fresh_snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let scan_batches = read_row_id_scan(fresh_snapshot, engine)?; + + let mut all_ids = collect_row_ids(&scan_batches); + all_ids.sort_unstable(); + assert_eq!( + all_ids, + vec![0, 1, 2, 3, 4], + "Row IDs must be preserved and correct after log compaction" + ); + + Ok(()) +} diff --git a/kernel/tests/golden_tables.rs b/kernel/tests/integration/golden_tables.rs similarity index 97% rename from kernel/tests/golden_tables.rs rename to kernel/tests/integration/golden_tables.rs index 017b21c578..0761e168a1 100644 --- a/kernel/tests/golden_tables.rs +++ b/kernel/tests/integration/golden_tables.rs @@ -7,29 +7,25 @@ use std::path::{Path, PathBuf}; use std::sync::Arc; use delta_kernel::arrow::array::{Array, AsArray, StructArray}; -use delta_kernel::arrow::compute::{concat_batches, take}; -use delta_kernel::arrow::compute::{lexsort_to_indices, SortColumn}; +use delta_kernel::arrow::compute::{concat_batches, lexsort_to_indices, take, SortColumn}; use delta_kernel::arrow::datatypes::{DataType, FieldRef, Schema}; use delta_kernel::arrow::record_batch::RecordBatch; +use delta_kernel::engine::arrow_conversion::TryFromKernel as _; use delta_kernel::engine::arrow_data::EngineDataArrowExt; +use delta_kernel::object_store::local::LocalFileSystem; +use delta_kernel::object_store::ObjectStore; use delta_kernel::parquet::arrow::async_reader::{ ParquetObjectReader, ParquetRecordBatchStreamBuilder, }; - -use delta_kernel::engine::arrow_conversion::TryFromKernel as _; -use delta_kernel::engine::default::executor::tokio::TokioBackgroundExecutor; -use delta_kernel::engine::default::DefaultEngine; -use delta_kernel::object_store::{local::LocalFileSystem, ObjectStore}; use delta_kernel::{DeltaResult, Snapshot}; - -use futures::{stream::TryStreamExt, StreamExt}; +use futures::stream::TryStreamExt; +use futures::StreamExt; use itertools::Itertools; use paste::paste; -use url::Url; - -mod common; - +use test_utils::delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor; +use test_utils::delta_kernel_default_engine::DefaultEngine; use test_utils::load_test_data; +use url::Url; // NB adapted from DAT: read all parquet files in the directory and concatenate them async fn read_expected(path: &Path) -> DeltaResult { @@ -421,7 +417,8 @@ skip_test!("canonicalized-paths-special-b": "BUG: path canonicalization"); // assert(foundFiles.map(_.getPath.split('/').last).toSet == Set("foo", "bar")) // // We added two add files with the same path `foo`. The first should have been removed. // // The second should remain, and should have a hard-coded modification time of 1700000000000L -// assert(foundFiles.find(_.getPath.endsWith("foo")).exists(_.getModificationTime == 1700000000000L)) +// assert(foundFiles.find(_.getPath.endsWith("foo")).exists(_.getModificationTime == +// 1700000000000L)) skip_test!("delete-re-add-same-file-different-transactions": "test not yet implemented"); // data file doesn't exist, get the relative path to compare diff --git a/kernel/tests/hdfs.rs b/kernel/tests/integration/hdfs.rs similarity index 99% rename from kernel/tests/hdfs.rs rename to kernel/tests/integration/hdfs.rs index 4f991b5ad8..c50974e58d 100644 --- a/kernel/tests/hdfs.rs +++ b/kernel/tests/integration/hdfs.rs @@ -7,12 +7,13 @@ // cargo test --features integration-test --test hdfs #![cfg(all(feature = "integration-test", not(target_os = "windows")))] -use delta_kernel::Snapshot; -use hdfs_native::{Client, WriteOptions}; -use hdfs_native_object_store::minidfs::MiniDfs; use std::collections::HashSet; use std::fs; use std::path::Path; + +use delta_kernel::Snapshot; +use hdfs_native::{Client, WriteOptions}; +use hdfs_native_object_store::minidfs::MiniDfs; extern crate walkdir; use walkdir::WalkDir; diff --git a/kernel/tests/checkpoint_transform.rs b/kernel/tests/integration/log/checkpoint_transform.rs similarity index 80% rename from kernel/tests/checkpoint_transform.rs rename to kernel/tests/integration/log/checkpoint_transform.rs index a1bd0a24b8..ac53d8b178 100644 --- a/kernel/tests/checkpoint_transform.rs +++ b/kernel/tests/integration/log/checkpoint_transform.rs @@ -15,16 +15,15 @@ use delta_kernel::arrow::compute::{concat_batches, sort_to_indices, take}; use delta_kernel::arrow::datatypes::{ DataType as ArrowDataType, Field, Int64Type, Schema as ArrowSchema, TimestampMicrosecondType, }; -use delta_kernel::engine::default::executor::tokio::TokioMultiThreadExecutor; -use delta_kernel::engine::default::DefaultEngineBuilder; +use delta_kernel::expressions::{col, lit, Scalar}; use delta_kernel::object_store::memory::InMemory; use delta_kernel::object_store::path::Path; -use delta_kernel::object_store::ObjectStore; +use delta_kernel::object_store::ObjectStoreExt as _; use delta_kernel::parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder; -use delta_kernel::DeltaResult; -use delta_kernel::Snapshot; - +use delta_kernel::{DeltaResult, Snapshot}; use serde_json::json; +use test_utils::delta_kernel_default_engine::executor::tokio::TokioMultiThreadExecutor; +use test_utils::delta_kernel_default_engine::DefaultEngineBuilder; use test_utils::{insert_data, read_scan, write_batch_to_table}; use url::Url; @@ -132,7 +131,7 @@ async fn test_checkpoint_stats_config_with_real_data( // Checkpoint 1 with (json1, struct1) settings let snapshot = Snapshot::builder_for(table_root.clone()).build(engine.as_ref())?; - snapshot.checkpoint(engine.as_ref())?; + snapshot.checkpoint(engine.as_ref(), None)?; // Version 2: write more data let snapshot = Snapshot::builder_for(table_root.clone()).build(engine.as_ref())?; @@ -157,7 +156,7 @@ async fn test_checkpoint_stats_config_with_real_data( // Checkpoint 2 with (json2, struct2) settings (reads from checkpoint 1 + commits 2-3) let snapshot = Snapshot::builder_for(table_root.clone()).build(engine.as_ref())?; - snapshot.checkpoint(engine.as_ref())?; + snapshot.checkpoint(engine.as_ref(), None)?; // Read all data back and verify correctness let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; @@ -263,15 +262,18 @@ async fn test_checkpoint_partitioned_with_real_data( engine.as_ref(), batch, HashMap::from([ - ("created_at".to_string(), "2024-01-15 10:30:00".to_string()), - ("tag".to_string(), "hello".to_string()), + ( + "created_at".to_string(), + Scalar::Timestamp(1_705_314_600_000_000), + ), + ("tag".to_string(), Scalar::Binary(b"hello".to_vec())), ]), ) .await?; // Checkpoint 1 with (json1, struct1) settings let snapshot = Snapshot::builder_for(table_root.clone()).build(engine.as_ref())?; - snapshot.checkpoint(engine.as_ref())?; + snapshot.checkpoint(engine.as_ref(), None)?; // Version 2: write data for partition created_at=2025-03-01 09:15:30.123456, tag=world let snapshot = Snapshot::builder_for(table_root.clone()).build(engine.as_ref())?; @@ -292,9 +294,9 @@ async fn test_checkpoint_partitioned_with_real_data( HashMap::from([ ( "created_at".to_string(), - "2025-03-01 09:15:30.123456".to_string(), + Scalar::Timestamp(1_740_820_530_123_456), ), - ("tag".to_string(), "world".to_string()), + ("tag".to_string(), Scalar::Binary(b"world".to_vec())), ]), ) .await?; @@ -315,7 +317,7 @@ async fn test_checkpoint_partitioned_with_real_data( // Checkpoint 2 with (json2, struct2) settings (reads from checkpoint 1 + commits 2-3) let snapshot = Snapshot::builder_for(table_root.clone()).build(engine.as_ref())?; - snapshot.checkpoint(engine.as_ref())?; + snapshot.checkpoint(engine.as_ref(), None)?; // Verify partitionValues_parsed content directly in the checkpoint if struct2 { @@ -472,13 +474,13 @@ async fn test_checkpoint_partition_values_parsed_with_column_mapping( &snapshot, engine.as_ref(), batch, - HashMap::from([("category".to_string(), "books".to_string())]), + HashMap::from([("category".to_string(), Scalar::String("books".into()))]), ) .await?; // Create checkpoint with writeStatsAsStruct=true let snapshot = Snapshot::builder_for(table_root.clone()).build(engine.as_ref())?; - snapshot.checkpoint(engine.as_ref())?; + snapshot.checkpoint(engine.as_ref(), None)?; // Read checkpoint and verify partitionValues_parsed uses physical name let checkpoint_path = Path::from("_delta_log/00000000000000000001.checkpoint.parquet"); @@ -529,3 +531,96 @@ async fn test_checkpoint_partition_values_parsed_with_column_mapping( Ok(()) } + +const WIDE_SCHEMA: &str = r#"{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"age","type":"long","nullable":true,"metadata":{}}]}"#; + +/// Regression test for https://github.com/delta-io/delta-kernel-rs/issues/2165 +/// +/// When a schema-evolved table has a checkpoint with stats_parsed covering only the +/// pre-evolution columns, scanning with a predicate on a newly added column should +/// not panic. The missing stats column should be treated as unknown (no data skipping). +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_scan_schema_evolved_table_with_checkpoint_predicate_on_new_column( +) -> Result<(), Box> { + let (store, table_root) = new_in_memory_store(); + let executor = Arc::new(TokioMultiThreadExecutor::new( + tokio::runtime::Handle::current(), + )); + let engine = Arc::new( + DefaultEngineBuilder::new(store.clone()) + .with_task_executor(executor) + .build(), + ); + + // Version 0: protocol + metadata with schema [id, name], stats as struct enabled + write_commit( + &store, + &build_commit(NON_PARTITIONED_SCHEMA, &[], true, true, true), + 0, + ) + .await?; + + // Version 1: write data with [id, name] + let snapshot = Snapshot::builder_for(table_root.clone()).build(engine.as_ref())?; + let result = insert_data( + snapshot, + &engine, + vec![ + Arc::new(Int64Array::from(vec![1, 2, 3])) as ArrayRef, + Arc::new(StringArray::from(vec!["Alice", "Bob", "Charlie"])), + ], + ) + .await?; + assert!(result.is_committed()); + + // Checkpoint at V1: stats_parsed covers only [id, name] + let snapshot = Snapshot::builder_for(table_root.clone()).build(engine.as_ref())?; + snapshot.checkpoint(engine.as_ref(), None)?; + + // Version 2: schema evolution - add `age` column via new metadata action + write_commit( + &store, + &build_commit(WIDE_SCHEMA, &[], true, true, false), + 2, + ) + .await?; + + // Version 3: write data with [id, name, age] + let snapshot = Snapshot::builder_for(table_root.clone()).build(engine.as_ref())?; + let result = insert_data( + snapshot, + &engine, + vec![ + Arc::new(Int64Array::from(vec![4, 5, 6])) as ArrayRef, + Arc::new(StringArray::from(vec!["Diana", "Eve", "Frank"])), + Arc::new(Int64Array::from(vec![25, 35, 45])), + ], + ) + .await?; + assert!(result.is_committed()); + + // Scan with predicate on `id` (present in checkpoint stats_parsed) should work + let snapshot = Snapshot::builder_for(table_root.clone()).build(engine.as_ref())?; + let predicate = Arc::new(col!("id").gt(lit(2i64))); + let scan = snapshot.scan_builder().with_predicate(predicate).build()?; + let batches = read_scan(&scan, engine.clone())?; + let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert!(total_rows > 0, "should return rows matching id > 2"); + + // Scan with predicate on `age` (missing from checkpoint stats_parsed) should NOT panic. + // The kernel should handle the missing stats column gracefully. + let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; + let predicate = Arc::new(col!("age").gt(lit(30i64))); + let scan = snapshot.scan_builder().with_predicate(predicate).build()?; + let batches = read_scan(&scan, engine.clone())?; + let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + // The predicate is used for data skipping only (not row-level filtering). + // All 6 rows should be returned since data skipping cannot skip any files: + // V1 data has no stats for `age`, and V3 data's age range includes values > 30. + assert_eq!( + total_rows, 6, + "all rows returned when data skipping cannot filter" + ); + + Ok(()) +} diff --git a/kernel/tests/integration/log/crc.rs b/kernel/tests/integration/log/crc.rs new file mode 100644 index 0000000000..0d543ca85e --- /dev/null +++ b/kernel/tests/integration/log/crc.rs @@ -0,0 +1,2672 @@ +//! Integration tests for CRC (version checksum) file-based APIs on Snapshot. + +use std::collections::{HashMap, HashSet}; +use std::path::PathBuf; +use std::sync::Arc; + +use delta_kernel::arrow::array::{ArrayRef, Int32Array, RecordBatch, StringArray}; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::crc::{Crc, DomainMetadataState, SetTransactionState}; +use delta_kernel::engine::arrow_conversion::TryFromKernel; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::object_store::local::LocalFileSystem; +use delta_kernel::path::ParsedLogPath; +use delta_kernel::schema::{schema_ref, DataType, SchemaRef, StructField, StructType}; +use delta_kernel::snapshot::{ChecksumWriteResult, IncrementalReplay, Snapshot, SnapshotRef}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::transaction::data_layout::DataLayout; +use delta_kernel::transaction::Transaction; +use delta_kernel::{ + DeltaResult, DeltaResultIteratorStatic, Engine, EngineData, EvaluationHandler, + FileDataReadResultIterator, FileMeta, FileStats, JsonHandler, ParquetFooter, ParquetHandler, + PredicateRef, StorageHandler, Version, +}; +use rstest::rstest; +use test_utils::delta_kernel_default_engine::executor::TaskExecutor; +use test_utils::delta_kernel_default_engine::{DefaultEngine, DefaultEngineBuilder}; +use test_utils::{ + add_commit, begin_transaction, copy_directory, insert_data, test_table_setup, + test_table_setup_mt, +}; +use url::Url; + +// ============================================================================ +// File stats from CRC on disk +// ============================================================================ + +#[tokio::test] +async fn test_get_file_stats_from_crc() -> DeltaResult<()> { + let path = std::fs::canonicalize(PathBuf::from("./tests/data/crc-full/")).unwrap(); + let table_root = url::Url::from_directory_path(path).unwrap(); + + let store = Arc::new(LocalFileSystem::new()); + let engine = DefaultEngineBuilder::new(store).build(); + + let snapshot = Snapshot::builder_for(table_root).build(&engine)?; + assert_eq!(snapshot.version(), 0); + + let file_stats = snapshot.get_file_stats_if_present().unwrap(); + assert_eq!(file_stats.num_files(), 10); + assert_eq!(file_stats.table_size_bytes(), 5259); + assert!(file_stats.file_size_histogram().is_some()); + + Ok(()) +} + +#[tokio::test] +async fn test_get_file_stats_no_crc() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let schema = Arc::new(StructType::try_new(vec![ + StructField::new("id", DataType::INTEGER, true), + StructField::new("value", DataType::STRING, true), + ])?); + + let _ = create_table(&table_path, schema, "Test/1.0") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let table_url = delta_kernel::try_parse_uri(&table_path)?; + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + assert_eq!(snapshot.version(), 0); + + let file_stats = snapshot.get_file_stats_if_present(); + assert_eq!(file_stats, None); + + Ok(()) +} + +#[tokio::test] +async fn test_get_file_stats_stale_crc_advances_via_safe_commit_serves_stats() -> DeltaResult<()> { + // ===== GIVEN ===== + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // Copy crc-full table (has CRC at version 0) into the temp dir + let source_path = std::fs::canonicalize(PathBuf::from("./tests/data/crc-full/")).unwrap(); + copy_directory(&source_path, _temp_dir.path()).unwrap(); + + // Verify the table starts at version 0 with valid CRC stats + let snapshot = Snapshot::builder_for(table_path.clone()).build(engine.as_ref())?; + assert_eq!(snapshot.version(), 0); + assert!(snapshot.get_file_stats_if_present().is_some()); + + // ===== WHEN ===== + // Safe (WRITE) commit with no file actions advances to version 1 (no new CRC written). + begin_transaction(snapshot, engine.as_ref())? + .with_operation("WRITE".to_string()) + .commit(engine.as_ref())? + .unwrap_committed(); + + // ===== THEN ===== + // The fresh v1 build advances the stale v0 CRC; the safe commit added no files, so file + // stats stay Complete and are served at v1 unchanged. + let snapshot = Snapshot::builder_for(table_path) + .with_incremental_crc_replay(IncrementalReplay::Unlimited) + .build(engine.as_ref())?; + assert_eq!(snapshot.version(), 1); + assert_eq!(snapshot.crc_at_version().unwrap().version, 1); + let stats = snapshot.get_file_stats_if_present().unwrap(); + assert_eq!(stats.num_files(), 10); + assert_eq!(stats.table_size_bytes(), 5259); + + Ok(()) +} + +// Tests incremental CRC replay when building a new snapshot from a base snapshot. +// Base snapshot has a CRC at v3. +#[rstest] +// no checkpoint, no newer CRC -> advance the base snapshot's crc@v3. +#[case::in_memory_base(None, None, Some(5))] +// a newer crc@v4 lands on disk and beats the base snapshot's crc@v3 -> advance crc@v4. +#[case::newer_disk_crc(None, Some(4), Some(5))] +// a checkpoint at v2 (below the base snapshot version) appears; the base snapshot's crc@v3 still +// applies (a CRC may sit above its checkpoint), so advance it. +#[case::checkpoint_before_base_snap_reuses_crc(Some(2), None, Some(5))] +// a checkpoint at v4 (above the base snapshot version) forces a rebuild; the base snapshot's +// crc@v3 is below it and dropped, so the on-disk crc@v4 advances instead. +#[case::checkpoint_after_base_snap_with_crc(Some(4), Some(4), Some(5))] +// a checkpoint at v4 (above the base snapshot version) forces a rebuild with no CRC at or above it +// -> no stats. +#[case::checkpoint_after_base_snap_no_crc(Some(4), None, None)] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_incremental_update_advances_crc_with_real_file_stats( + #[case] checkpoint_version: Option, + #[case] crc_version: Option, + #[case] expected_num_files: Option, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + + // ===== GIVEN: v0 (0 files, crc on disk), then five WRITE inserts (one file each) to v5 ===== + let committed = create_table_and_commit(&table_path, engine.as_ref())?; + let mut snapshot = committed.post_commit_snapshot().unwrap().clone(); + snapshot.write_checksum(engine.as_ref())?; + for i in 1..=5i32 { + snapshot = insert_data(snapshot, &engine, vec![Arc::new(Int32Array::from(vec![i]))]) + .await? + .unwrap_committed() + .post_commit_snapshot() + .unwrap() + .clone(); + } + + // ===== AND: load the base snapshot from disk at v3 (will have in-memory CRC) ===== + let base_snapshot = Snapshot::builder_for(&table_path) + .at_version(3) + .with_incremental_crc_replay(IncrementalReplay::Unlimited) + .build(engine.as_ref())?; + assert_eq!(base_snapshot.crc_at_version().unwrap().version, 3); + + // ===== AND: conditionally write a CRC and/or checkpoint ===== + if let Some(v) = crc_version { + Snapshot::builder_for(&table_path) + .at_version(v) + .with_incremental_crc_replay(IncrementalReplay::Unlimited) + .build(engine.as_ref())? + .write_checksum(engine.as_ref())?; + } + if let Some(v) = checkpoint_version { + Snapshot::builder_for(&table_path) + .at_version(v) + .build(engine.as_ref())? + .checkpoint(engine.as_ref(), None)?; + } + + // ===== WHEN: incrementally update the base snapshot to the latest version ===== + let updated_snapshot = Snapshot::builder_from(base_snapshot) + .with_incremental_crc_replay(IncrementalReplay::Unlimited) + .build(engine.as_ref())?; + + // ===== THEN: an in-memory CRC is loaded exactly when expected; when loaded it sits at the + // snapshot version and reflects all five files ===== + assert_eq!(updated_snapshot.version(), 5); + match expected_num_files { + Some(num_files) => { + let crc = updated_snapshot + .crc_at_version() + .expect("expected an in-memory CRC"); + assert_eq!(crc.version, updated_snapshot.version()); + let stats = updated_snapshot.get_file_stats_if_present().unwrap(); + assert_eq!(stats.num_files(), num_files); + } + None => { + assert!(updated_snapshot.crc_at_version().is_none()); + assert!(updated_snapshot.get_file_stats_if_present().is_none()); + } + } + + Ok(()) +} + +// An unreadable CRC at the snapshot version must not break loading: the snapshot falls back +// to log replay for P&M and exposes no CRC. +#[tokio::test] +async fn test_snapshot_loads_when_crc_at_version_is_corrupt() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let schema = schema_ref! { nullable "id": INTEGER }; + let _ = create_table(&table_path, schema, "Test/1.0") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + // Plant a garbage CRC file at the table version. + let crc_path = _temp_dir.path().join("_delta_log/00000000000000000000.crc"); + std::fs::write(&crc_path, b"not valid crc json").unwrap(); + + let table_url = delta_kernel::try_parse_uri(&table_path)?; + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + assert_eq!(snapshot.version(), 0); + assert!(snapshot.crc_at_version().is_none()); + assert!(snapshot.get_file_stats_if_present().is_none()); + + Ok(()) +} + +// ============================================================================ +// CRC test visibility: Snapshot::crc +// ============================================================================ + +#[tokio::test] +async fn test_crc_returns_resolved_crc_at_snapshot_version() -> DeltaResult<()> { + // ===== GIVEN ===== + let path = std::fs::canonicalize(PathBuf::from("./tests/data/crc-full/")).unwrap(); + let table_root = url::Url::from_directory_path(path).unwrap(); + + let store = Arc::new(LocalFileSystem::new()); + let engine = DefaultEngineBuilder::new(store).build(); + + let snapshot = Snapshot::builder_for(table_root).build(&engine)?; + assert_eq!(snapshot.version(), 0); + + // ===== WHEN ===== + let crc = snapshot.crc_at_version().unwrap(); + + // ===== THEN ===== + let file_stats = crc.file_stats().unwrap(); + assert_eq!(file_stats.table_size_bytes(), 5259); + assert_eq!(file_stats.num_files(), 10); + + // Protocol and metadata should match the snapshot's table configuration + assert_eq!(crc.protocol, *snapshot.table_configuration().protocol()); + assert_eq!(crc.metadata, *snapshot.table_configuration().metadata()); + + // Domain metadata + let dms = crc.domain_metadata_state.expect_complete(); + assert_eq!(dms.len(), 3); + assert!(dms.contains_key("delta.clustering")); + assert!(dms.contains_key("delta.rowTracking")); + assert!(dms.contains_key("myApp.metadata")); + + Ok(()) +} + +#[tokio::test] +async fn test_crc_returns_none_when_no_crc() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let schema = schema_ref! { nullable "id": INTEGER }; + + let _ = create_table(&table_path, schema, "Test/1.0") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let table_url = delta_kernel::try_parse_uri(&table_path)?; + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + assert_eq!(snapshot.version(), 0); + + assert!(snapshot.crc_at_version().is_none()); + + Ok(()) +} + +// ============================================================================ +// Post-commit CRC existence: does a CRC exist on the post-commit snapshot? +// ============================================================================ + +fn create_table_and_commit( + table_path: &str, + engine: &dyn delta_kernel::Engine, +) -> DeltaResult { + let schema = schema_ref! { nullable "id": INTEGER }; + let txn = create_table(table_path, schema, "test_engine") + .with_data_layout(DataLayout::clustered(["id"])) + .build(engine, Box::new(FileSystemCommitter::new()))? + .with_domain_metadata("zip".to_string(), "zap0".to_string()); + + Ok(txn.commit(engine)?.unwrap_committed()) +} + +#[tokio::test] +async fn test_create_table_produces_post_commit_crc() -> DeltaResult<()> { + // ===== GIVEN / WHEN: Create the table ===== + let (_temp_dir, table_path, engine) = test_table_setup()?; + let committed = create_table_and_commit(&table_path, engine.as_ref())?; + + // ===== THEN: should have CRC at v0 ===== + assert_eq!(committed.commit_version(), 0); + let snapshot = committed.post_commit_snapshot().unwrap(); + let crc = snapshot.crc_at_version().unwrap(); + + let file_stats = crc.file_stats().unwrap(); + assert_eq!(file_stats.num_files(), 0); + assert_eq!(file_stats.table_size_bytes(), 0); + assert_eq!(crc.protocol, *snapshot.table_configuration().protocol()); + assert_eq!(crc.metadata, *snapshot.table_configuration().metadata()); + let dms = crc.domain_metadata_state.expect_complete(); + assert_eq!(dms["zip"].configuration(), "zap0"); + + Ok(()) +} + +#[rstest] +#[case::with_in_memory_crc(true)] +#[case::without_crc(false)] +#[tokio::test] +async fn test_post_commit_crc_chains_only_if_read_snapshot_has_crc( + #[case] use_post_commit_snapshot: bool, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let create_committed = create_table_and_commit(&table_path, engine.as_ref())?; + + let read_snapshot = if use_post_commit_snapshot { + // Post-commit snapshot has in-memory CRC from the previous commit. + create_committed.post_commit_snapshot().unwrap().clone() + } else { + // Fresh-from-disk snapshot has no CRC (no .crc file on disk). + Snapshot::builder_for(table_path).build(engine.as_ref())? + }; + assert_eq!( + read_snapshot.crc_at_version().is_some(), + use_post_commit_snapshot + ); + + let committed = begin_transaction(read_snapshot, engine.as_ref())? + .with_operation("WRITE".to_string()) + .with_domain_metadata("zip".to_string(), "zap1".to_string()) + .commit(engine.as_ref())? + .unwrap_committed(); + + // The new post-commit snapshot should only have a CRC if the read snapshot had one. + assert_eq!(committed.commit_version(), 1); + assert_eq!( + committed + .post_commit_snapshot() + .unwrap() + .crc_at_version() + .is_some(), + use_post_commit_snapshot + ); + + Ok(()) +} + +// ================================================================================ +// Post-commit CRC correctness: are the CRC fields accurate after write and reload? +// ================================================================================ + +/// Writes the in-memory CRC to disk, reloads a fresh snapshot, and asserts that the +/// round-tripped CRC matches the in-memory one. Returns the loaded CRC for further assertions. +fn write_and_verify_crc( + snapshot: &SnapshotRef, + table_path: &str, + engine: &dyn delta_kernel::Engine, +) -> Crc { + let crc_in_memory = snapshot.crc_at_version().unwrap(); + snapshot.write_checksum(engine).unwrap(); + + let snapshot_fresh = Snapshot::builder_for(table_path).build(engine).unwrap(); + let crc_from_disk = snapshot_fresh.crc_at_version().unwrap(); + assert_eq!(crc_in_memory, crc_from_disk); + crc_from_disk.as_ref().clone() +} + +#[tokio::test] +async fn test_post_commit_crc_tracks_file_stats_across_inserts() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // ===== GIVEN: Create the table ===== + let committed = create_table_and_commit(&table_path, engine.as_ref())?; + let snapshot_v0 = committed.post_commit_snapshot().unwrap().clone(); + + // ===== WHEN: Insert values 1..=10 ===== + let col1: ArrayRef = Arc::new(Int32Array::from((1..=10).collect::>())); + let committed = insert_data(snapshot_v0, &engine, vec![col1]) + .await? + .unwrap_committed(); + + // ===== THEN: should have CRC at v1 with right file stats ===== + assert_eq!(committed.commit_version(), 1); + let snapshot_v1 = committed.post_commit_snapshot().unwrap(); + let crc_v1 = write_and_verify_crc(snapshot_v1, &table_path, engine.as_ref()); + let stats_v1 = crc_v1.file_stats().unwrap(); + assert_eq!(stats_v1.num_files(), 1); // <--- 1 file added + assert!(stats_v1.table_size_bytes() > 0); // <--- size is non-zero + + // ===== WHEN: Insert values 11..=20 ===== + let col2: ArrayRef = Arc::new(Int32Array::from((11..=20).collect::>())); + let committed = insert_data(snapshot_v1.clone(), &engine, vec![col2]) + .await? + .unwrap_committed(); + + // ===== THEN: should have CRC at v2 with right file stats ===== + assert_eq!(committed.commit_version(), 2); + let snapshot_v2 = committed.post_commit_snapshot().unwrap(); + let crc_v2 = write_and_verify_crc(snapshot_v2, &table_path, engine.as_ref()); + let stats_v2 = crc_v2.file_stats().unwrap(); + assert_eq!(stats_v2.num_files(), 2); // <--- 2 files added + assert!(stats_v2.table_size_bytes() > stats_v1.table_size_bytes()); // <--- size is greater than after first insert + + // ===== WHEN: Remove all files ===== + let scan = snapshot_v2.clone().scan_builder().build()?; + let mut txn = begin_transaction(snapshot_v2.clone(), engine.as_ref())? + .with_operation("DELETE".to_string()) + .with_data_change(true); + for sm in scan.scan_metadata(engine.as_ref())? { + txn.remove_files(sm?.scan_files); + } + let committed = txn.commit(engine.as_ref())?.unwrap_committed(); + + // ===== THEN: should have CRC at v3 with right file stats ===== + assert_eq!(committed.commit_version(), 3); + let snapshot_v3 = committed.post_commit_snapshot().unwrap(); + let crc_v3 = write_and_verify_crc(snapshot_v3, &table_path, engine.as_ref()); + let stats_v3 = crc_v3.file_stats().unwrap(); + assert_eq!(stats_v3.num_files(), 0); // <--- 0 net file in the table + assert_eq!(stats_v3.table_size_bytes(), 0); // <--- size is 0 + + Ok(()) +} + +#[tokio::test] +async fn test_post_commit_crc_tracks_domain_metadata_changes() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // ===== WHEN: CREATE TABLE with zip -> zap0 ===== + let committed = create_table_and_commit(&table_path, engine.as_ref())?; + let snapshot_v0 = committed.post_commit_snapshot().unwrap(); + + // ===== THEN: should have CRC at v0 with zip -> zap0 ===== + let crc_v0 = write_and_verify_crc(snapshot_v0, &table_path, engine.as_ref()); + let dms = crc_v0.domain_metadata_state.expect_complete(); + assert_eq!(dms["zip"].configuration(), "zap0"); + + // ===== WHEN: update zip -> zap1, add foo -> bar ===== + let txn = begin_transaction(snapshot_v0.clone(), engine.as_ref())? + .with_operation("WRITE".to_string()) + .with_domain_metadata("zip".to_string(), "zap1".to_string()) // <-- set to zap1 + .with_domain_metadata("foo".to_string(), "bar".to_string()); // <-- add foo + let committed = txn.commit(engine.as_ref())?.unwrap_committed(); + + // ===== THEN: should have CRC at v1 with zip -> zap1, foo -> bar ===== + let snapshot_v1 = committed.post_commit_snapshot().unwrap(); + let crc_v1 = write_and_verify_crc(snapshot_v1, &table_path, engine.as_ref()); + let dms = crc_v1.domain_metadata_state.expect_complete(); + assert_eq!(dms["zip"].configuration(), "zap1"); // <-- must be zap1 + assert_eq!(dms["foo"].configuration(), "bar"); // <-- must be bar + + // ===== WHEN: remove zip, keep foo ===== + let txn = begin_transaction(snapshot_v1.clone(), engine.as_ref())? + .with_operation("WRITE".to_string()) + .with_domain_metadata_removed("zip".to_string()); // <-- remove zip + let committed = txn.commit(engine.as_ref())?.unwrap_committed(); + + // ===== THEN: should have CRC at v2 with zip gone, foo still there ===== + let snapshot_v2 = committed.post_commit_snapshot().unwrap(); + let crc_v2 = write_and_verify_crc(snapshot_v2, &table_path, engine.as_ref()); + let dms = crc_v2.domain_metadata_state.expect_complete(); + assert!(!dms.contains_key("zip")); // <-- must be gone + assert_eq!(dms["foo"].configuration(), "bar"); // <-- must still be bar + + Ok(()) +} + +#[tokio::test] +async fn test_post_commit_crc_non_incremental_op_makes_file_stats_indeterminate() -> DeltaResult<()> +{ + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // ===== GIVEN: Create table (v0) and insert data (v1) ===== + let committed = create_table_and_commit(&table_path, engine.as_ref())?; + let snapshot_v0 = committed.post_commit_snapshot().unwrap().clone(); + + let col: ArrayRef = Arc::new(Int32Array::from(vec![1, 2, 3])); + let committed = insert_data(snapshot_v0, &engine, vec![col]) + .await? + .unwrap_committed(); + let snapshot_v1 = committed.post_commit_snapshot().unwrap(); + + // ===== WHEN: Commit a non-incremental operation (ANALYZE STATS) ===== + let committed = begin_transaction(snapshot_v1.clone(), engine.as_ref())? + .with_operation("ANALYZE STATS".to_string()) + .commit(engine.as_ref())? + .unwrap_committed(); + + // ===== THEN: CRC at v2 has indeterminate file stats ===== + assert_eq!(committed.commit_version(), 2); + let snapshot_v2 = committed.post_commit_snapshot().unwrap(); + let crc_v2 = snapshot_v2.crc_at_version().unwrap(); + assert!(crc_v2.file_stats_state().is_indeterminate()); + + Ok(()) +} + +// ============================================================================ +// Write checksum to disk +// ============================================================================ + +#[tokio::test] +async fn test_write_checksum_success_simple() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let committed = create_table_and_commit(&table_path, engine.as_ref())?; + let snapshot = committed.post_commit_snapshot().unwrap(); + + let (result, _updated) = snapshot.write_checksum(engine.as_ref())?; + assert_eq!(result, ChecksumWriteResult::Written); + + // Verify the CRC file is readable by loading a fresh snapshot from disk + let fresh_snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert!(fresh_snapshot.crc_at_version().is_some()); + + Ok(()) +} + +#[rstest] +#[case::same_snapshot(false)] +#[case::fresh_snapshot(true)] +#[tokio::test] +async fn test_write_checksum_double_write_returns_already_exists( + #[case] reload_snapshot: bool, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let committed = create_table_and_commit(&table_path, engine.as_ref())?; + let snapshot = committed.post_commit_snapshot().unwrap(); + + let (first, updated) = snapshot.write_checksum(engine.as_ref())?; + assert_eq!(first, ChecksumWriteResult::Written); + + let second = if reload_snapshot { + let fresh = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let (result, _) = fresh.write_checksum(engine.as_ref())?; + result + } else { + let (result, _) = updated.write_checksum(engine.as_ref())?; + result + }; + assert_eq!(second, ChecksumWriteResult::AlreadyExists); + + Ok(()) +} + +/// The root that `resolve_crc_for_write` resolves the CRC from. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum WriteRoot { + /// No checkpoint and no CRC. + VersionZero, + /// A checkpoint at the snapshot's version with no tail commits. + CheckpointNoTail, + /// A checkpoint below the snapshot's version with tail commits. + CheckpointWithTail, + /// A stale on-disk CRC, loaded with [`IncrementalReplay::Unlimited`]. + StaleCrcIncrementalBuild, + /// A stale on-disk CRC, loaded with [`IncrementalReplay::Disabled`]. + StaleCrcNonIncrementalBuild, +} + +/// For each resolution root: build a table, load a snapshot, write the CRC, and validate its +/// contents (file stats, active domains, set transactions). +#[rstest] +#[tokio::test(flavor = "multi_thread")] +async fn test_write_checksum_resolves_correct_crc_from_each_root( + #[values( + WriteRoot::VersionZero, + WriteRoot::CheckpointNoTail, + WriteRoot::CheckpointWithTail, + WriteRoot::StaleCrcIncrementalBuild, + WriteRoot::StaleCrcNonIncrementalBuild + )] + root: WriteRoot, + #[values(false, true)] ict_enabled: bool, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + + // === Create the table with domain metadata (and optionally ICT) enabled === + let schema = schema_ref! { nullable "id": INTEGER }; + let mut builder = create_table(&table_path, schema, "test_engine") + .with_table_properties([("delta.feature.domainMetadata", "supported")]); + if ict_enabled { + builder = builder.with_table_properties([("delta.enableInCommitTimestamps", "true")]); + } + let mut snap = builder + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + + const CHECKPOINT_OR_CRC_VERSION: i64 = 4; + let (checkpoint_or_crc_version, latest): (Option, i64) = match root { + WriteRoot::VersionZero => (None, CHECKPOINT_OR_CRC_VERSION), + WriteRoot::CheckpointNoTail => (Some(CHECKPOINT_OR_CRC_VERSION), CHECKPOINT_OR_CRC_VERSION), + WriteRoot::CheckpointWithTail + | WriteRoot::StaleCrcIncrementalBuild + | WriteRoot::StaleCrcNonIncrementalBuild => ( + Some(CHECKPOINT_OR_CRC_VERSION), + CHECKPOINT_OR_CRC_VERSION + 2, + ), + }; + let removed_domain = "d1"; + + // === Commit loop: accumulate domain metadata and set transactions === + // Each commit v adds one file, sets domain "d{v}"->"cfg{v}" and set-txn "app{v}"->v. At v=3 we + // also remove "d1", so the final CRC must reflect the removal. + for v in 1..=latest { + let arrow_schema = TryFromKernel::try_from_kernel(snap.schema().as_ref())?; + let batch = RecordBatch::try_new( + Arc::new(arrow_schema), + vec![Arc::new(Int32Array::from(vec![v as i32]))], + ) + .map_err(|e| delta_kernel::Error::generic(e.to_string()))?; + let mut txn = snap + .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? + .with_operation("WRITE".to_string()) + .with_data_change(true) + .with_domain_metadata(format!("d{v}"), format!("cfg{v}")) + .with_transaction_id(format!("app{v}"), v); + if v == 3 { + txn = txn.with_domain_metadata_removed(removed_domain.to_string()); + } + let write_context = txn.unpartitioned_write_context()?; + let adds = engine + .write_parquet(&ArrowEngineData::new(batch), &write_context) + .await?; + txn.add_files(adds); + snap = txn.commit(engine.as_ref())?.unwrap_post_commit_snapshot(); + + if checkpoint_or_crc_version == Some(v) { + match root { + WriteRoot::CheckpointNoTail | WriteRoot::CheckpointWithTail => { + snap = snap.checkpoint(engine.as_ref(), None)?.1; + } + WriteRoot::StaleCrcIncrementalBuild | WriteRoot::StaleCrcNonIncrementalBuild => { + snap.write_checksum(engine.as_ref())?; + } + WriteRoot::VersionZero => unreachable!("VersionZero has no seed version"), + } + } + } + + // === Load from disk, write the checksum, reload, assert the persisted CRC === + let incremental_build = root == WriteRoot::StaleCrcIncrementalBuild; + let replay = if incremental_build { + IncrementalReplay::Unlimited + } else { + IncrementalReplay::Disabled + }; + let fresh = Snapshot::builder_for(&table_path) + .with_incremental_crc_replay(replay) + .build(engine.as_ref())?; + assert_eq!(fresh.crc_at_version().is_some(), incremental_build); + + // Confirm the load actually reached the intended resolution root, so a mis-resolution that + // still produced correct contents cannot pass silently. + match root { + WriteRoot::CheckpointNoTail | WriteRoot::CheckpointWithTail => assert_eq!( + fresh.log_segment().checkpoint_version, + Some(CHECKPOINT_OR_CRC_VERSION as u64) + ), + WriteRoot::VersionZero => assert!(fresh.log_segment().checkpoint_version.is_none()), + WriteRoot::StaleCrcIncrementalBuild | WriteRoot::StaleCrcNonIncrementalBuild => { + assert!(crc_file_path(&table_path, CHECKPOINT_OR_CRC_VERSION as u64).exists()) + } + } + + assert_eq!( + fresh.write_checksum(engine.as_ref())?.0, + ChecksumWriteResult::Written + ); + + // Reload from disk so the assertions below run against the persisted CRC. + let reloaded = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let crc = reloaded.crc_at_version().unwrap(); + + assert_eq!(crc.version as i64, latest); + assert_eq!(crc.in_commit_timestamp_opt.is_some(), ict_enabled); + + // File stats cover every live file (one per commit), checked against disk ground truth. + let disk = parquet_file_sizes_on_disk(&table_path); + let stats = crc.file_stats().unwrap(); + assert_eq!(stats.num_files() as usize, disk.len()); + assert_eq!(stats.table_size_bytes(), disk.iter().sum::()); + + // Domain metadata: every set domain is present with its config, except the removed one. + let dms = crc.domain_metadata_state.expect_complete(); + assert!(!dms.contains_key(removed_domain)); + for v in 1..=latest { + let domain = format!("d{v}"); + if domain == removed_domain { + continue; + } + assert_eq!(dms[&domain].configuration(), format!("cfg{v}")); + } + + // Set transactions: every committed app id is present. + let txns = crc.set_transaction_state.expect_complete(); + assert_eq!(txns.len() as i64, latest); + for v in 1..=latest { + assert!(txns.contains_key(&format!("app{v}"))); + } + + Ok(()) +} + +/// A `Disabled`-mode load over a stale on-disk CRC retains it as a base: `base_crc()` is +/// `Some(stale)` while `crc_at_version()` is `None`. Asserting only `crc_at_version().is_none()` +/// would pass even if retention regressed to a full discard, so this drives the retained base +/// through the observable write path. +#[tokio::test(flavor = "multi_thread")] +async fn test_disabled_load_retains_stale_crc_as_base() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + // crc-full has a CRC at version 0. + let source_path = std::fs::canonicalize(PathBuf::from("./tests/data/crc-full/")).unwrap(); + copy_directory(&source_path, _temp_dir.path()).unwrap(); + + // Safe WRITE commit advances to v1 without writing a new CRC; the newest on-disk CRC stays v0. + let snap0 = Snapshot::builder_for(table_path.clone()).build(engine.as_ref())?; + begin_transaction(snap0, engine.as_ref())? + .with_operation("WRITE".to_string()) + .commit(engine.as_ref())? + .unwrap_committed(); + + // Default Disabled load at v1: the stale CRC@0 is not advanced, but it is retained as a base. + let snap1 = Snapshot::builder_for(table_path).build(engine.as_ref())?; + assert_eq!(snap1.version(), 1); + assert!( + snap1.crc_at_version().is_none(), + "a stale CRC must not be served as at-version" + ); + assert_eq!( + snap1.get_file_stats_if_present(), + None, + "file stats come from an at-version CRC only" + ); + // `write_checksum` observably advances the retained base over v1. + assert_eq!( + snap1.write_checksum(engine.as_ref())?.0, + ChecksumWriteResult::Written + ); + let reloaded = Snapshot::builder_for(snap1.table_root().clone()).build(engine.as_ref())?; + assert_eq!(reloaded.crc_at_version().map(|c| c.version), Some(1)); + + Ok(()) +} + +/// Regression: a stale base carried through `checkpoint()` must not break a later `write_checksum`. +/// The checkpoint drops the commits below it, so the stale base can no longer be advanced over +/// them; resolution must fall back to the checkpoint root instead of erroring on a missing commit. +#[tokio::test(flavor = "multi_thread")] +async fn test_write_checksum_after_checkpoint_with_stale_base_resolves_from_checkpoint( +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let source_path = std::fs::canonicalize(PathBuf::from("./tests/data/crc-full/")).unwrap(); + copy_directory(&source_path, _temp_dir.path()).unwrap(); + + // v0 has CRC@0; a safe WRITE commit advances to v1 with no new CRC. + let snap0 = Snapshot::builder_for(table_path.clone()).build(engine.as_ref())?; + begin_transaction(snap0, engine.as_ref())? + .with_operation("WRITE".to_string()) + .commit(engine.as_ref())? + .unwrap_committed(); + + // Disabled load retains stale CRC@0 as base; checkpoint at v1 carries state forward and drops + // the v1 commit. + let snap1 = Snapshot::builder_for(table_path).build(engine.as_ref())?; + assert!(snap1.crc_at_version().is_none()); + let (_, checkpointed) = snap1.checkpoint(engine.as_ref(), None)?; + assert_eq!(checkpointed.log_segment().checkpoint_version, Some(1)); + + // Resolution must not trip the below-checkpoint base on the emptied commit tail. + assert_eq!( + checkpointed.write_checksum(engine.as_ref())?.0, + ChecksumWriteResult::Written + ); + let reloaded = + Snapshot::builder_for(checkpointed.table_root().clone()).build(engine.as_ref())?; + assert_eq!(reloaded.crc_at_version().map(|c| c.version), Some(1)); + + Ok(()) +} + +/// Builds a table with commits 0..=3 (one file each) and a stale CRC written at v1, then returns a +/// base snapshot at v3 that still holds the un-advanced in-memory CRC@1 (loaded `Disabled` so the +/// base is NOT advanced to v3), plus a checkpoint written at v2. An incremental update from this +/// base discovers checkpoint@2 and trims the combined segment's commits above v2, leaving the +/// retained base@1 BELOW the checkpoint. +async fn setup_incremental_below_checkpoint_base( + engine: &Arc>, + table_path: &str, +) -> DeltaResult { + let schema = schema_ref! { nullable "id": INTEGER }; + let mut snap = create_table(table_path, schema, "test_engine") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + for v in 1..=3i32 { + let arrow_schema = TryFromKernel::try_from_kernel(snap.schema().as_ref())?; + let batch = RecordBatch::try_new( + Arc::new(arrow_schema), + vec![Arc::new(Int32Array::from(vec![v]))], + ) + .map_err(|e| delta_kernel::Error::generic(e.to_string()))?; + let mut txn = snap + .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? + .with_operation("WRITE".to_string()) + .with_data_change(true); + let write_context = txn.unpartitioned_write_context()?; + let adds = engine + .write_parquet(&ArrowEngineData::new(batch), &write_context) + .await?; + txn.add_files(adds); + snap = txn.commit(engine.as_ref())?.unwrap_post_commit_snapshot(); + if v == 1 { + snap.write_checksum(engine.as_ref())?; + } + } + + // Disabled load at v3 retains the un-advanced CRC@1 (Unlimited would advance it to v3 and + // defeat the below-checkpoint scenario). + let base = Snapshot::builder_for(table_path) + .at_version(3) + .build(engine.as_ref())?; + + // Checkpoint at v2, above the base's CRC@1 but below the base version. + Snapshot::builder_for(table_path) + .at_version(2) + .build(engine.as_ref())? + .checkpoint(engine.as_ref(), None)?; + + Ok(base) +} + +/// Regression (incremental load-advance path): an incremental update under `Unlimited` must not +/// try to advance a retained base that sits below a newly-listed checkpoint. `pick_latest_base_crc` +/// would otherwise feed CRC@1 into `build_crc_from_base` on a segment whose checkpoint@2 dropped +/// commit 2, erroring on the missing commit. The below-checkpoint base must be skipped, so the +/// build falls back to log replay. +#[tokio::test(flavor = "multi_thread")] +async fn test_incremental_unlimited_skips_below_checkpoint_base() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let base = setup_incremental_below_checkpoint_base(&engine, &table_path).await?; + + let updated = Snapshot::builder_from(base) + .with_incremental_crc_replay(IncrementalReplay::Unlimited) + .build(engine.as_ref())?; + assert_eq!(updated.version(), 3); + assert_eq!(updated.log_segment().checkpoint_version, Some(2)); + + Ok(()) +} + +/// Regression (incremental write path): after an incremental update retains a base below a +/// newly-listed checkpoint, `write_checksum` must skip that below-checkpoint base and resolve from +/// the checkpoint root rather than erroring on the commits the checkpoint subsumed. +#[tokio::test(flavor = "multi_thread")] +async fn test_write_checksum_incremental_stale_base_below_new_checkpoint_resolves( +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let base = setup_incremental_below_checkpoint_base(&engine, &table_path).await?; + + // Disabled update keeps the below-checkpoint base@1 un-advanced on the combined segment. + let updated = Snapshot::builder_from(base).build(engine.as_ref())?; + assert_eq!(updated.log_segment().checkpoint_version, Some(2)); + + // Case 2 must skip the below-checkpoint base and resolve from the checkpoint root. + assert_eq!( + updated.write_checksum(engine.as_ref())?.0, + ChecksumWriteResult::Written + ); + + // The checkpoint-root fallback must count every live file, not just the post-checkpoint tail. + // The fixture writes one file per commit (v1..=3), checked against disk ground truth. + let reloaded = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let stats = reloaded.get_file_stats_if_present().unwrap(); + let disk = parquet_file_sizes_on_disk(&table_path); + assert_eq!(stats.num_files() as usize, disk.len()); + assert_eq!(stats.table_size_bytes(), disk.iter().sum::()); + + Ok(()) +} + +/// ICT enabled, checkpoint at the current version, but v_end's commit file is unreadable: the +/// ICT read error propagates instead of being laundered into a generic "CRC unresolved" error. +#[tokio::test(flavor = "multi_thread")] +async fn test_write_checksum_from_checkpoint_ict_enabled_but_commit_unreadable_propagates_read_error( +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + + let schema = schema_ref! { nullable "id": INTEGER }; + let snap = create_table(&table_path, schema, "test_engine") + .with_table_properties([ + ("delta.feature.inCommitTimestamp", "supported"), + ("delta.enableInCommitTimestamps", "true"), + ]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + let snap = insert_data( + snap, + &engine, + vec![Arc::new(Int32Array::from(vec![1, 2, 3]))], + ) + .await? + .unwrap_post_commit_snapshot(); + let (_, snap) = snap.checkpoint(engine.as_ref(), None)?; + let checkpoint_version = snap.version(); + + // Corrupt v_end's commit file so its ICT can't be read (the snapshot still loads from the + // checkpoint at the same version). + let commit = _temp_dir + .path() + .join(format!("_delta_log/{checkpoint_version:020}.json")); + std::fs::write(&commit, b"}}} not valid commit json").unwrap(); + + let fresh = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert!(fresh.crc_at_version().is_none()); + // The failure is the propagated ICT read error, not a laundered `ChecksumWriteUnsupported`. + assert!(matches!( + fresh.write_checksum(engine.as_ref()), + Err(e) if !matches!(e, delta_kernel::Error::ChecksumWriteUnsupported(_)) + )); + + Ok(()) +} + +#[tokio::test(flavor = "multi_thread")] +async fn test_write_checksum_no_crc_with_non_incremental_tail_returns_unsupported( +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + + let schema = schema_ref! { nullable "id": INTEGER }; + let snap = create_table(&table_path, schema, "test_engine") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + let snap = insert_data( + snap, + &engine, + vec![Arc::new(Int32Array::from(vec![1, 2, 3]))], + ) + .await? + .unwrap_post_commit_snapshot(); + let (_, snap) = snap.checkpoint(engine.as_ref(), None)?; + // Non-incremental operation in the tail dooms file stats regardless of the checkpoint. + begin_transaction(snap, engine.as_ref())? + .with_operation("ANALYZE STATS".to_string()) + .commit(engine.as_ref())? + .unwrap_committed(); + + let fresh = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert!(fresh.crc_at_version().is_none()); + assert!(matches!( + fresh.write_checksum(engine.as_ref()), + Err(delta_kernel::Error::ChecksumWriteUnsupported(_)) + )); + + Ok(()) +} + +#[tokio::test] +async fn test_in_memory_crc_chains_across_multiple_commits_then_writes() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let committed = create_table_and_commit(&table_path, engine.as_ref())?; + let mut snapshot = committed.post_commit_snapshot().unwrap().clone(); + assert!(snapshot.crc_at_version().is_some()); + + // Chain several commits without writing CRC to disk + for i in 0..5 { + let col: ArrayRef = Arc::new(Int32Array::from(vec![i])); + let committed = insert_data(snapshot, &engine, vec![col]) + .await? + .unwrap_committed(); + snapshot = committed.post_commit_snapshot().unwrap().clone(); + assert!( + snapshot.crc_at_version().is_some(), + "in-memory CRC lost at commit {}", + committed.commit_version() + ); + } + + // Only now write the CRC -- should have accumulated all 5 inserts + assert_eq!(snapshot.version(), 5); + let crc = write_and_verify_crc(&snapshot, &table_path, engine.as_ref()); + let crc_stats = crc.file_stats().unwrap(); + assert_eq!(crc_stats.num_files(), 5); + assert!(crc_stats.table_size_bytes() > 0); + + // Verify histogram totals match disk ground truth + let disk_sizes = parquet_file_sizes_on_disk(&table_path); + assert_eq!(disk_sizes.len(), 5); + assert_histogram_totals(crc_stats, 5, disk_sizes.iter().sum()); + + Ok(()) +} + +// When an incremental snapshot update picks up a CRC file at the new version from the new log +// segment, that CRC is resolved and stored on the resulting snapshot. +#[tokio::test] +async fn test_incremental_snapshot_preserves_loaded_crc() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // Create table at v0 and write its CRC to disk + let committed_v0 = create_table_and_commit(&table_path, engine.as_ref())?; + let snapshot_v0 = committed_v0.post_commit_snapshot().unwrap(); + snapshot_v0.write_checksum(engine.as_ref())?; + + // Insert data at v1 and write its CRC to disk + let col: ArrayRef = Arc::new(Int32Array::from(vec![1, 2, 3])); + let committed_v1 = insert_data(snapshot_v0.clone(), &engine, vec![col]) + .await? + .unwrap_committed(); + committed_v1 + .post_commit_snapshot() + .unwrap() + .write_checksum(engine.as_ref())?; + + // Load a fresh snapshot at v0 (from disk, not post-commit) + let fresh_v0 = Snapshot::builder_for(&table_path) + .at_version(0) + .build(engine.as_ref())?; + assert_eq!(fresh_v0.version(), 0); + + // Incrementally update from v0 -> v1 + let incremental_v1 = Snapshot::builder_from(fresh_v0).build(engine.as_ref())?; + assert_eq!(incremental_v1.version(), 1); + + // The CRC at v1 should be loaded from the incremental update (not discarded) + assert_eq!(incremental_v1.crc_at_version().map(|c| c.version), Some(1)); + assert!( + incremental_v1.crc_at_version().is_some(), + "CRC should be loaded at v1 after incremental snapshot update" + ); + + // Committing from this snapshot produces a post-commit CRC by applying the delta to the + // chained CRC. + let col: ArrayRef = Arc::new(Int32Array::from(vec![4, 5, 6])); + let committed_v2 = insert_data(incremental_v1, &engine, vec![col]) + .await? + .unwrap_committed(); + assert_eq!(committed_v2.commit_version(), 2); + let snapshot_v2 = committed_v2.post_commit_snapshot().unwrap(); + assert!( + snapshot_v2.crc_at_version().is_some(), + "Post-commit CRC should chain from incremental snapshot's CRC" + ); + + Ok(()) +} + +// Incremental update where only the old segment has a CRC file (no new CRC written). +// The old segment's CRC file is preserved on the combined segment, but with the default +// (disabled) replay budget the v0 CRC is not advanced to v1, so the snapshot carries no CRC. +#[tokio::test] +async fn test_incremental_snapshot_old_crc_no_new_crc() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // Create table at v0 and write CRC to disk + let committed_v0 = create_table_and_commit(&table_path, engine.as_ref())?; + committed_v0 + .post_commit_snapshot() + .unwrap() + .write_checksum(engine.as_ref())?; + + // Insert data at v1 -- do NOT write CRC for v1 + let col: ArrayRef = Arc::new(Int32Array::from(vec![1, 2, 3])); + let committed_v1 = insert_data( + committed_v0.post_commit_snapshot().unwrap().clone(), + &engine, + vec![col], + ) + .await? + .unwrap_committed(); + assert_eq!(committed_v1.commit_version(), 1); + + // Load a fresh snapshot at v0 (this loads 0.crc during P&M reading) + let fresh_v0 = Snapshot::builder_for(&table_path) + .at_version(0) + .build(engine.as_ref())?; + assert!( + fresh_v0.crc_at_version().is_some(), + "Fresh v0 snapshot should have CRC loaded from 0.crc" + ); + + // Incrementally update from v0 -> v1. The new listing (starting at v1) doesn't find + // any CRC file, so the combined segment keeps the old segment's 0.crc. + let incremental_v1 = Snapshot::builder_from(fresh_v0).build(engine.as_ref())?; + assert_eq!(incremental_v1.version(), 1); + + // builder_from defaults to IncrementalReplay::Disabled, so the v0 CRC is not advanced to + // v1 and the snapshot carries no CRC. (With Unlimited it would advance to v1.) + assert!( + incremental_v1.crc_at_version().is_none(), + "CRC at v0 should not be stored on the v1 snapshot (replay disabled)" + ); + + Ok(()) +} + +// CRC should always write domainMetadata as an empty list (not omit the field) when there are +// no domain metadata actions, regardless of whether the feature is supported. +#[rstest] +#[case::dm_feature_supported(true)] +#[case::dm_feature_not_supported(false)] +#[tokio::test] +async fn test_write_checksum_with_no_dms_writes_empty_list( + #[case] dm_supported: bool, +) -> DeltaResult<()> { + use std::collections::HashMap; + + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let schema = schema_ref! { nullable "id": INTEGER }; + + let mut builder = create_table(&table_path, schema, "test_engine"); + if dm_supported { + let properties = HashMap::from([( + "delta.feature.domainMetadata".to_string(), + "supported".to_string(), + )]); + builder = builder.with_table_properties(properties); + } + let committed = builder + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + + let snapshot = committed.post_commit_snapshot().unwrap(); + assert!(snapshot + .get_all_domain_metadata(engine.as_ref())? + .is_empty()); + let crc = write_and_verify_crc(snapshot, &table_path, engine.as_ref()); + // CREATE TABLE without any DM actions produces an authoritative empty map. + assert_eq!( + crc.domain_metadata_state, + DomainMetadataState::Complete(HashMap::new()) + ); + + Ok(()) +} + +// ============================================================================ +// Domain metadata CRC fast path +// ============================================================================ + +/// Engine that panics if any handler is accessed. +struct FailingEngine; + +impl Engine for FailingEngine { + fn evaluation_handler(&self) -> Arc { + unimplemented!() + } + fn storage_handler(&self) -> Arc { + unimplemented!() + } + fn json_handler(&self) -> Arc { + unimplemented!() + } + fn parquet_handler(&self) -> Arc { + unimplemented!() + } +} + +#[tokio::test] +async fn test_get_domain_metadata_with_crc_skips_log_replay() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // v0: CREATE TABLE with zip -> zap0 (and clustering DM from create_table_and_commit) + let committed = create_table_and_commit(&table_path, engine.as_ref())?; + let snapshot_v0 = committed.post_commit_snapshot().unwrap(); + + // v1: update zip -> zap1, add foo -> bar + let committed = begin_transaction(snapshot_v0.clone(), engine.as_ref())? + .with_operation("WRITE".to_string()) + .with_domain_metadata("zip".to_string(), "zap1".to_string()) + .with_domain_metadata("foo".to_string(), "bar".to_string()) + .commit(engine.as_ref())? + .unwrap_committed(); + + // Asserts domain metadata on any snapshot, regardless of how it was loaded. + let assert_domain_metadata = |snapshot: &Snapshot, engine: &dyn delta_kernel::Engine| { + assert_eq!( + snapshot.get_domain_metadata("zip", engine).unwrap(), + Some("zap1".to_string()) + ); + assert_eq!( + snapshot.get_domain_metadata("foo", engine).unwrap(), + Some("bar".to_string()) + ); + // Miss on a Complete cache: served as authoritative None without log replay. + assert_eq!( + snapshot.get_domain_metadata("nonexistent", engine).unwrap(), + None + ); + assert!(snapshot + .get_domain_metadata_internal("delta.clustering", engine) + .unwrap() + .is_some()); + assert_eq!( + snapshot + .get_domain_metadatas_internal(engine, None) + .unwrap() + .len(), + 3 + ); + }; + + // Case 1: Post-commit snapshot with in-memory CRC => DM loaded from CRC (fast path). + // Use NoJsonReadsEngine to prove no log replay occurs. + let post_commit_snapshot = committed.post_commit_snapshot().unwrap(); + assert!(post_commit_snapshot.crc_at_version().is_some()); + assert_domain_metadata(post_commit_snapshot, &FailingEngine); + + // Case 2: Fresh snapshot loaded from disk, no CRC file => DM loaded via log replay (slow path) + let fresh_snapshot_no_crc = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert!(fresh_snapshot_no_crc.crc_at_version().is_none()); + assert_domain_metadata(&fresh_snapshot_no_crc, engine.as_ref()); + + // Case 3: Write CRC to disk, then reload fresh snapshot => DM loaded from CRC (fast path) + // Use NoJsonReadsEngine to prove no log replay occurs. + let _ = post_commit_snapshot.write_checksum(engine.as_ref())?; + + let fresh_snapshot_with_crc = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert!(fresh_snapshot_with_crc.crc_at_version().is_some()); + assert_domain_metadata(&fresh_snapshot_with_crc, &FailingEngine); + + Ok(()) +} + +fn crc_file_path(table_path: &str, version: Version) -> PathBuf { + let url = Url::from_directory_path(table_path).unwrap(); + ParsedLogPath::new_crc(&url, version) + .unwrap() + .location + .to_file_path() + .unwrap() +} + +fn read_crc_json(table_path: &str, version: Version) -> serde_json::Value { + let bytes = std::fs::read(crc_file_path(table_path, version)).unwrap(); + serde_json::from_slice(&bytes).unwrap() +} + +/// Rewrites the on-disk CRC at `version` to drop the named top-level field, leaving every +/// other field intact. +fn strip_field_from_crc(table_path: &str, version: Version, field: &str) { + let mut value = read_crc_json(table_path, version); + value.as_object_mut().unwrap().remove(field); + let path = crc_file_path(table_path, version); + std::fs::write(&path, serde_json::to_vec(&value).unwrap()).unwrap(); +} + +#[tokio::test] +async fn test_partial_dm_serves_hits_and_falls_through_for_misses() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // v0: CREATE TABLE with zip -> zap0 (post-commit writes CRC with full DM). + let committed = create_table_and_commit(&table_path, engine.as_ref())?; + let snapshot_v0_orig = committed.post_commit_snapshot().unwrap(); + snapshot_v0_orig.write_checksum(engine.as_ref())?; + + // Strip DM from v0 CRC, then reload: base snapshot has `Partial(empty)` DM rather than + // the post-commit `Complete(...)` state. + strip_field_from_crc(&table_path, 0, "domainMetadata"); + let snapshot_v0 = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!( + snapshot_v0.crc_at_version().unwrap().domain_metadata_state, + DomainMetadataState::Partial(HashMap::new()) + ); + + // v1: post-commit chain accumulates DM into `Partial(map)`. + let committed = begin_transaction(snapshot_v0.clone(), engine.as_ref())? + .with_operation("WRITE".to_string()) + .with_domain_metadata("foo".to_string(), "bar".to_string()) + .commit(engine.as_ref())? + .unwrap_committed(); + let snapshot_v1 = committed.post_commit_snapshot().unwrap(); + + let crc_v1 = snapshot_v1.crc_at_version().unwrap(); + let map = crc_v1.domain_metadata_state.expect_partial(); + assert!(map.contains_key("foo")); + + // Hit: "foo" is in the Partial cache, so served without log replay. + assert_eq!( + snapshot_v1.get_domain_metadata("foo", &FailingEngine)?, + Some("bar".to_string()) + ); + + // Miss: "zip" is not in this Partial cache; FailingEngine panics, real engine finds it. + assert!(std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + snapshot_v1.get_domain_metadata("zip", &FailingEngine).ok() + })) + .is_err()); + assert_eq!( + snapshot_v1.get_domain_metadata("zip", engine.as_ref())?, + Some("zap0".to_string()) + ); + + // Miss on a nonexistent domain falls through and returns None. + assert_eq!( + snapshot_v1.get_domain_metadata("nonexistent", engine.as_ref())?, + None + ); + + // Multi-key filter with a mixed hit ("foo") and miss ("zip"): the first miss + // short-circuits the cache lookup and the full result comes from log replay. + let filter = HashSet::from(["foo", "zip"]); + let map = snapshot_v1.get_domain_metadatas_internal(engine.as_ref(), Some(&filter))?; + assert_eq!(map.len(), 2); + assert_eq!(map["foo"].configuration(), "bar"); + assert_eq!(map["zip"].configuration(), "zap0"); + + // "All" queries against Partial always fall through. The replay-derived set must + // include BOTH entries. + let mut all = snapshot_v1.get_all_domain_metadata(engine.as_ref())?; + all.sort_by(|a, b| a.domain().cmp(b.domain())); + assert_eq!(all.len(), 2); + assert_eq!(all[0].domain(), "foo"); + assert_eq!(all[1].domain(), "zip"); + + Ok(()) +} + +// ============================================================================ +// Set transaction CRC tracking +// ============================================================================ + +/// Comprehensive test for set transaction CRC tracking: verifies that set transactions are +/// correctly tracked in the CRC across commits, round-trip through write/reload, and that +/// the CRC fast path (no log replay) works for set transaction queries. +#[tokio::test] +async fn test_set_transaction_crc_tracking_and_fast_path() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // -- v0: CREATE TABLE (no set transactions) -- + let committed = create_table_and_commit(&table_path, engine.as_ref())?; + let snapshot_v0 = committed.post_commit_snapshot().unwrap(); + + // Post-commit CRC has empty Complete set_transaction_state (not Partial). + let crc_v0 = write_and_verify_crc(snapshot_v0, &table_path, engine.as_ref()); + assert_eq!( + crc_v0.set_transaction_state, + SetTransactionState::Complete(HashMap::new()) + ); + + // Fresh snapshot with CRC on disk serves queries via fast path (no log replay) + let fresh_v0 = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert!(fresh_v0.crc_at_version().is_some()); + assert_eq!( + fresh_v0 + .get_app_id_version("my-app", &FailingEngine) + .unwrap(), + None + ); + + // -- v1: commit with my-app=1 -- + let committed = begin_transaction(snapshot_v0.clone(), engine.as_ref())? + .with_operation("WRITE".to_string()) + .with_transaction_id("my-app".to_string(), 1) + .commit(engine.as_ref())? + .unwrap_committed(); + let snapshot_v1 = committed.post_commit_snapshot().unwrap(); + + // Post-commit CRC tracks my-app=1, queryable via fast path + assert_eq!( + snapshot_v1 + .get_app_id_version("my-app", &FailingEngine) + .unwrap(), + Some(1) + ); + assert_eq!( + snapshot_v1 + .get_app_id_version("nonexistent", &FailingEngine) + .unwrap(), + None + ); + + // Write CRC to disk, reload, verify round-trip and fast path + let crc_v1 = write_and_verify_crc(snapshot_v1, &table_path, engine.as_ref()); + let txns_v1 = crc_v1.set_transaction_state.expect_complete(); + assert_eq!(txns_v1.len(), 1); + assert!(txns_v1.contains_key("my-app")); + + let fresh_v1 = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert!(fresh_v1.crc_at_version().is_some()); + assert_eq!( + fresh_v1 + .get_app_id_version("my-app", &FailingEngine) + .unwrap(), + Some(1) + ); + assert_eq!( + fresh_v1 + .get_app_id_version("nonexistent", &FailingEngine) + .unwrap(), + None + ); + + // -- v2: commit with my-app=2 (upsert) + other-app=1 (new) -- + let committed = begin_transaction(snapshot_v1.clone(), engine.as_ref())? + .with_operation("WRITE".to_string()) + .with_transaction_id("my-app".to_string(), 2) + .with_transaction_id("other-app".to_string(), 1) + .commit(engine.as_ref())? + .unwrap_committed(); + let snapshot_v2 = committed.post_commit_snapshot().unwrap(); + + // Post-commit CRC tracks updated versions, queryable via fast path + assert_eq!( + snapshot_v2 + .get_app_id_version("my-app", &FailingEngine) + .unwrap(), + Some(2) + ); + assert_eq!( + snapshot_v2 + .get_app_id_version("other-app", &FailingEngine) + .unwrap(), + Some(1) + ); + + // Write CRC to disk, reload, verify round-trip and fast path + let crc_v2 = write_and_verify_crc(snapshot_v2, &table_path, engine.as_ref()); + let txns_v2 = crc_v2.set_transaction_state.expect_complete(); + assert_eq!(txns_v2.len(), 2); + + let fresh_v2 = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert!(fresh_v2.crc_at_version().is_some()); + assert_eq!( + fresh_v2 + .get_app_id_version("my-app", &FailingEngine) + .unwrap(), + Some(2) + ); + assert_eq!( + fresh_v2 + .get_app_id_version("other-app", &FailingEngine) + .unwrap(), + Some(1) + ); + + Ok(()) +} + +#[tokio::test] +async fn test_partial_set_txn_serves_hits_and_falls_through_for_misses() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // v0: CREATE TABLE. v1: commit with v1-app=1, then write CRC to disk. + let committed = create_table_and_commit(&table_path, engine.as_ref())?; + let snapshot_v0 = committed.post_commit_snapshot().unwrap(); + let committed = begin_transaction(snapshot_v0.clone(), engine.as_ref())? + .with_operation("WRITE".to_string()) + .with_transaction_id("v1-app".to_string(), 1) + .commit(engine.as_ref())? + .unwrap_committed(); + let snapshot_v1 = committed.post_commit_snapshot().unwrap(); + snapshot_v1.write_checksum(engine.as_ref())?; + + // Strip setTransactions from the v1 CRC so it reloads as Partial(empty). + strip_field_from_crc(&table_path, 1, "setTransactions"); + + let snapshot_v1_reloaded = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!( + snapshot_v1_reloaded + .crc_at_version() + .unwrap() + .set_transaction_state, + SetTransactionState::Partial(HashMap::new()) + ); + + // v2: commit with my-app=1; post-commit CRC accumulates into Partial. + let committed = begin_transaction(snapshot_v1_reloaded.clone(), engine.as_ref())? + .with_operation("WRITE".to_string()) + .with_transaction_id("my-app".to_string(), 1) + .commit(engine.as_ref())? + .unwrap_committed(); + let snapshot_v2 = committed.post_commit_snapshot().unwrap(); + + let map = snapshot_v2 + .crc_at_version() + .unwrap() + .set_transaction_state + .expect_partial(); + assert!(map.contains_key("my-app")); + assert!(!map.contains_key("v1-app")); + + // Hit: "my-app" is in the Partial cache -- served without log replay. + assert_eq!( + snapshot_v2.get_app_id_version("my-app", &FailingEngine)?, + Some(1) + ); + + // Miss: "v1-app" is NOT in the Partial cache; FailingEngine panics, real engine finds it. + assert!(std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + snapshot_v2 + .get_app_id_version("v1-app", &FailingEngine) + .ok() + })) + .is_err()); + assert_eq!( + snapshot_v2.get_app_id_version("v1-app", engine.as_ref())?, + Some(1) + ); + + // Miss: completely absent app_id falls through and returns None. + assert_eq!( + snapshot_v2.get_app_id_version("nonexistent", engine.as_ref())?, + None + ); + + // Writing the v2 CRC must drop the Partial setTransactions map on serialize: the on-disk + // file has a `null` setTransactions field even though the in-memory map has entries. + snapshot_v2.write_checksum(engine.as_ref())?; + assert!(read_crc_json(&table_path, 2)["setTransactions"].is_null()); + + Ok(()) +} + +// ============================================================================ +// Set transaction CRC expiration +// ============================================================================ + +/// Tests the CRC fast path for set transaction expiration filtering. Since `lastUpdated` is set +/// to now, "interval 0 seconds" yields `expiration_timestamp = now`, so `last_updated <= now` +/// holds and the txn expires. A large retention or no retention should keep the txn visible. +#[rstest] +#[case::zero_retention_expires(Some("interval 0 seconds"), None)] +#[case::large_retention_not_expired(Some("interval 365 days"), Some(1))] +#[case::no_retention_no_filtering(None, Some(1))] +#[tokio::test] +async fn test_set_txn_expiration_via_crc_fast_path( + #[case] retention: Option<&str>, + #[case] expected: Option, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let schema = schema_ref! { nullable "id": INTEGER }; + + // v0: create the table with optional retention property + let mut builder = create_table(&table_path, schema, "test_engine"); + if let Some(r) = retention { + builder = builder.with_table_properties([("delta.setTransactionRetentionDuration", r)]); + } + let committed = builder + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + + // v1: commit a set transaction for "my-app" (lastUpdated = now) + let snapshot_v0 = committed.post_commit_snapshot().unwrap().clone(); + let committed = begin_transaction(snapshot_v0, engine.as_ref())? + .with_operation("WRITE".to_string()) + .with_transaction_id("my-app".to_string(), 1) + .commit(engine.as_ref())? + .unwrap_committed(); + + // Write CRC at v1 so the fast path is used on reload + let snapshot_v1 = committed.post_commit_snapshot().unwrap(); + snapshot_v1.write_checksum(engine.as_ref())?; + + let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!(snapshot.version(), 1); + + // Verify CRC was loaded from disk + assert!(snapshot.crc_at_version().is_some()); + + // FailingEngine proves the CRC fast path is used (no log replay) + assert_eq!( + snapshot + .get_app_id_version("my-app", &FailingEngine) + .unwrap(), + expected + ); + + Ok(()) +} + +/// Mirrors `test_set_txn_expiration_via_crc_fast_path` for the `Partial` branch: a cached +/// transaction whose `lastUpdated` is older than retention must return `None` via the fast path, +/// without falling through to log replay. +#[tokio::test] +async fn test_partial_set_txn_expired_hit_returns_none_via_fast_path() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let schema = schema_ref! { nullable "id": INTEGER }; + + // v0: create the table with zero-second retention so any past lastUpdated expires. + let committed = create_table(&table_path, schema, "test_engine") + .with_table_properties([( + "delta.setTransactionRetentionDuration", + "interval 0 seconds", + )]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + + // v1: commit my-app=1, write CRC at v1. + let snapshot_v0 = committed.post_commit_snapshot().unwrap().clone(); + let committed = begin_transaction(snapshot_v0, engine.as_ref())? + .with_operation("WRITE".to_string()) + .with_transaction_id("my-app".to_string(), 1) + .commit(engine.as_ref())? + .unwrap_committed(); + committed + .post_commit_snapshot() + .unwrap() + .write_checksum(engine.as_ref())?; + + // Strip setTransactions from the v1 CRC so it reloads as Partial(empty). + strip_field_from_crc(&table_path, 1, "setTransactions"); + let snapshot_v1_reloaded = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + + // v2: commit my-app=2 from the Partial base; post-commit CRC is Partial(map) containing my-app. + let committed = begin_transaction(snapshot_v1_reloaded, engine.as_ref())? + .with_operation("WRITE".to_string()) + .with_transaction_id("my-app".to_string(), 2) + .commit(engine.as_ref())? + .unwrap_committed(); + let snapshot_v2 = committed.post_commit_snapshot().unwrap(); + + let map = snapshot_v2 + .crc_at_version() + .unwrap() + .set_transaction_state + .expect_partial(); + assert!(map.contains_key("my-app")); + + // FailingEngine proves the Partial fast path is used; the expiration filter drops the txn. + assert_eq!( + snapshot_v2.get_app_id_version("my-app", &FailingEngine)?, + None + ); + + Ok(()) +} + +/// Verifies that a set transaction with null `last_updated` never expires, even with the most +/// aggressive retention ("interval 0 seconds"). +#[tokio::test] +async fn test_set_txn_null_last_updated_never_expires_via_log_replay() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // v0: create table with aggressive retention + let schema = schema_ref! { nullable "id": INTEGER }; + create_table(&table_path, schema, "test_engine") + .with_table_properties([( + "delta.setTransactionRetentionDuration", + "interval 0 seconds", + )]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + + // v1: raw commit with txn action that omits lastUpdated + let store = Arc::new(LocalFileSystem::new()); + add_commit( + &table_path, + store.as_ref(), + 1, + r#"{"txn":{"appId":"null-app","version":42}}"#.to_string(), + ) + .await + .unwrap(); + + // Reload fresh snapshot at v1 -- no CRC covers v1, so log replay is used + let fresh = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!(fresh.version(), 1); + + // Despite aggressive retention, null last_updated means the txn never expires + assert_eq!( + fresh.get_app_id_version("null-app", engine.as_ref())?, + Some(42) + ); + + Ok(()) +} + +// The newest txn by log order for an app_id wins, then expiration is applied to it: an expired +// newest yields None, it does NOT fall back to an older non-expired txn. Uses non-monotonic +// lastUpdated (v1 far-future, v2 tiny) so the newest-by-log-order txn (v2) is the expired one. +#[tokio::test] +async fn test_set_txn_expired_newest_returns_none_not_older_via_log_replay() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let schema = schema_ref! { nullable "id": INTEGER }; + create_table(&table_path, schema, "test_engine") + .with_table_properties([("delta.setTransactionRetentionDuration", "interval 365 days")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + + let store = Arc::new(LocalFileSystem::new()); + add_commit( + &table_path, + store.as_ref(), + 1, + r#"{"txn":{"appId":"app","version":1,"lastUpdated":99999999999999}}"#.to_string(), + ) + .await + .unwrap(); + add_commit( + &table_path, + store.as_ref(), + 2, + r#"{"txn":{"appId":"app","version":2,"lastUpdated":1000}}"#.to_string(), + ) + .await + .unwrap(); + + let fresh = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!(fresh.version(), 2); + assert!(fresh.crc_at_version().is_none()); + assert_eq!(fresh.get_app_id_version("app", engine.as_ref())?, None); + + Ok(()) +} + +// ============================================================================ +// File Histogram Tracking Across Commits +// ============================================================================ + +/// Returns paths of all `.parquet` files in the table root directory. +/// +/// NOTE: Uses non-recursive `read_dir`, so this only finds parquet files +/// directly in the table root. Partitioned tables store parquet files in +/// subdirectories and would require a recursive walk. +fn parquet_paths_on_disk(table_path: &str) -> Vec { + let url = delta_kernel::try_parse_uri(table_path).unwrap(); + let dir = url.to_file_path().unwrap(); + std::fs::read_dir(dir) + .unwrap() + .filter_map(|entry| { + let path = entry.unwrap().path(); + if path.extension().is_some_and(|ext| ext == "parquet") { + Some(path) + } else { + None + } + }) + .collect() +} + +/// Returns sorted sizes of all `.parquet` files in the table directory as +/// independent ground truth (not derived from the CRC computation). +/// +/// NOTE: After a Delta remove, call `delete_parquet_files_on_disk` first +/// so that the disk state reflects only logically-active files (Delta +/// removes are logical, not physical). +fn parquet_file_sizes_on_disk(table_path: &str) -> Vec { + let mut sizes: Vec = parquet_paths_on_disk(table_path) + .iter() + .map(|p| std::fs::metadata(p).unwrap().len() as i64) + .collect(); + sizes.sort(); + sizes +} + +/// Deletes all `.parquet` files in the table directory. Called after Delta +/// remove actions to keep `parquet_file_sizes_on_disk` accurate. +fn delete_parquet_files_on_disk(table_path: &str) { + for path in parquet_paths_on_disk(table_path) { + std::fs::remove_file(path).unwrap(); + } +} + +/// Asserts that the histogram totals (summed across all bins) match the expected values, +/// and that the histogram file count sum equals [`FileStats::num_files`]. +fn assert_histogram_totals( + file_stats: &FileStats, + expected_file_count: i64, + expected_total_bytes: i64, +) { + let hist = file_stats + .file_size_histogram() + .expect("histogram should be present"); + let count_sum: i64 = hist.file_counts().iter().sum(); + assert_eq!(count_sum, expected_file_count); + assert_eq!(count_sum, file_stats.num_files()); + assert_eq!(hist.total_bytes().iter().sum::(), expected_total_bytes); +} + +/// The first non-zero default histogram bin boundary (8KB). +const FIRST_BIN_BOUNDARY: i64 = 8192; + +/// Approximate bytes per row for `(int32, 100-char padded string)` parquet data. +const APPROX_BYTES_PER_ROW: i64 = 104; + +/// Row count guaranteed to produce a parquet file exceeding [`FIRST_BIN_BOUNDARY`]. +/// Uses 2x the boundary divided by per-row size as a generous margin. +const LARGE_FILE_ROW_COUNT: i32 = (FIRST_BIN_BOUNDARY * 2 / APPROX_BYTES_PER_ROW) as i32; + +/// Verifies that the in-memory CRC histogram correctly tracks file adds and removes across +/// multiple bins, cross-checked against actual file sizes on disk at each step. The CRC is +/// maintained in memory via `post_commit_snapshot` and written to disk at each step for +/// verification. +/// +/// - v0: empty table -> histogram all zeros +/// - v1: insert small file (< 8KB, bin 0) -> 1 file in bin 0 +/// - v2: insert large file (>= 8KB, bin 1+) -> files span two bins +/// - v3: remove all files, delete parquet from disk -> histogram returns to all zeros +#[tokio::test] +async fn test_file_histogram_tracks_adds_and_removes_across_bins() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("data", DataType::STRING), + ])?); + + // ===== v0: empty table ===== + let committed = create_table(&table_path, schema, "test_engine") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + let snapshot = committed.post_commit_snapshot().unwrap(); + let crc_v0 = write_and_verify_crc(snapshot, &table_path, engine.as_ref()); + assert_histogram_totals(crc_v0.file_stats().unwrap(), 0, 0); + + // ===== v1: insert small file (< 8KB -> bin 0) ===== + let ids: ArrayRef = Arc::new(Int32Array::from(vec![1, 2, 3])); + let data: ArrayRef = Arc::new(StringArray::from(vec!["a", "b", "c"])); + let committed = insert_data(snapshot.clone(), &engine, vec![ids, data]) + .await? + .unwrap_committed(); + let snapshot = committed.post_commit_snapshot().unwrap(); + let disk_sizes = parquet_file_sizes_on_disk(&table_path); + assert_eq!(disk_sizes.len(), 1); + let crc_v1 = write_and_verify_crc(snapshot, &table_path, engine.as_ref()); + let stats_v1 = crc_v1.file_stats().unwrap(); + assert_histogram_totals(stats_v1, 1, disk_sizes.iter().sum()); + + // Verify boundary metadata via public getters + let hist = stats_v1.file_size_histogram().unwrap(); + assert_eq!(hist.sorted_bin_boundaries()[0], 0); + assert_eq!(hist.sorted_bin_boundaries().len(), 95); + + // ===== v2: insert large file (>= 8KB -> bin 1+) ===== + let n = LARGE_FILE_ROW_COUNT; + let ids: ArrayRef = Arc::new(Int32Array::from((0..n).collect::>())); + let strings: Vec = (0..n).map(|i| format!("{i:0>100}")).collect(); + let data: ArrayRef = Arc::new(StringArray::from(strings)); + let committed = insert_data(snapshot.clone(), &engine, vec![ids, data]) + .await? + .unwrap_committed(); + let snapshot = committed.post_commit_snapshot().unwrap(); + + // Cross-check: files land in different bins + let disk_sizes = parquet_file_sizes_on_disk(&table_path); + assert_eq!(disk_sizes.len(), 2); + let small_size = disk_sizes[0]; + let large_size = disk_sizes[1]; + assert!( + small_size < FIRST_BIN_BOUNDARY, + "expected small file < 8KB, got {small_size}" + ); + assert!( + large_size >= FIRST_BIN_BOUNDARY, + "expected large file >= 8KB, got {large_size}" + ); + + let crc_v2 = write_and_verify_crc(snapshot, &table_path, engine.as_ref()); + let stats_v2 = crc_v2.file_stats().unwrap(); + let hist = stats_v2.file_size_histogram().unwrap(); + assert_eq!(hist.file_counts()[0], 1, "bin 0 should have the small file"); + assert_eq!(hist.total_bytes()[0], small_size); + + // Find the exact bin for the large file based on its actual size + let boundaries = hist.sorted_bin_boundaries(); + let large_bin = boundaries + .windows(2) + .enumerate() + .find(|(_, w)| large_size >= w[0] && large_size < w[1]) + .map(|(i, _)| i) + .unwrap_or(boundaries.len() - 1); + assert_eq!( + hist.file_counts()[large_bin], + 1, + "large file ({large_size} bytes) should be in bin {large_bin}" + ); + assert_eq!(hist.total_bytes()[large_bin], large_size); + + // ===== v3: remove all files ===== + let scan = snapshot.clone().scan_builder().build()?; + let mut txn = begin_transaction(snapshot.clone(), engine.as_ref())? + .with_operation("DELETE".to_string()) + .with_data_change(true); + for sm in scan.scan_metadata(engine.as_ref())? { + txn.remove_files(sm?.scan_files); + } + let committed = txn.commit(engine.as_ref())?.unwrap_committed(); + let snapshot = committed.post_commit_snapshot().unwrap(); + + // Delete physical parquet files so disk ground truth reflects the empty table + delete_parquet_files_on_disk(&table_path); + assert!(parquet_file_sizes_on_disk(&table_path).is_empty()); + + let crc_v3 = write_and_verify_crc(snapshot, &table_path, engine.as_ref()); + assert_histogram_totals(crc_v3.file_stats().unwrap(), 0, 0); + + Ok(()) +} + +/// Verifies the disk round-trip path: write the in-memory CRC to disk at v1, load a fresh +/// snapshot from disk (which deserializes the v1 CRC from JSON), then insert at v2. The v2 +/// post-commit CRC is computed by applying the v2 delta to the deserialized v1 CRC, testing +/// that the in-memory chain works with a disk-loaded base. +#[tokio::test] +async fn test_file_histogram_survives_disk_round_trip_then_delta_merge() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let committed = create_table_and_commit(&table_path, engine.as_ref())?; + let snapshot_v0 = committed.post_commit_snapshot().unwrap(); + + // v1: insert data and write CRC to disk + let col1: ArrayRef = Arc::new(Int32Array::from((1..=10).collect::>())); + let committed = insert_data(snapshot_v0.clone(), &engine, vec![col1]) + .await? + .unwrap_committed(); + let snapshot_v1 = committed.post_commit_snapshot().unwrap(); + let v1_bytes = snapshot_v1 + .crc_at_version() + .unwrap() + .file_stats() + .unwrap() + .table_size_bytes(); + snapshot_v1.write_checksum(engine.as_ref())?; + + // Load a FRESH snapshot from disk at v1 (CRC deserialized from JSON, not in-memory) + let fresh_v1 = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!(fresh_v1.version(), 1); + assert!(fresh_v1.crc_at_version().is_some()); + + // v2: insert using the fresh (disk-loaded) snapshot -- the post-commit CRC at v2 + // is computed by applying the v2 delta to the deserialized v1 CRC + let col2: ArrayRef = Arc::new(Int32Array::from((11..=20).collect::>())); + let committed = insert_data(fresh_v1, &engine, vec![col2]) + .await? + .unwrap_committed(); + assert_eq!(committed.commit_version(), 2); + + // Verify the merged histogram: 2 files, bytes match actual parquet files on disk + let snapshot_v2 = committed.post_commit_snapshot().unwrap(); + let crc_v2 = write_and_verify_crc(snapshot_v2, &table_path, engine.as_ref()); + let disk_sizes = parquet_file_sizes_on_disk(&table_path); + assert_eq!(disk_sizes.len(), 2); + assert!(disk_sizes.iter().sum::() > v1_bytes); + assert_histogram_totals(crc_v2.file_stats().unwrap(), 2, disk_sizes.iter().sum()); + + Ok(()) +} + +/// Rewrites the histogram in an on-disk CRC file to use custom 2-bin +/// boundaries `[0, 100]`. Since any parquet file is > 100 bytes (metadata +/// alone exceeds this), all files land +/// deterministically in bin 1 regardless of compression. The file_counts and total_bytes +/// arrays are rebuilt to match the new boundaries using the provided file sizes. +fn rewrite_crc_with_custom_bins(table_path: &str, version: u64, file_sizes: &[i64]) { + let url = delta_kernel::try_parse_uri(table_path).unwrap(); + let crc_file = url + .to_file_path() + .unwrap() + .join(format!("_delta_log/{version:020}.crc")); + + let json_str = std::fs::read_to_string(&crc_file).unwrap(); + let mut crc: serde_json::Value = serde_json::from_str(&json_str).unwrap(); + + // All files are > 100 bytes, so bin 0 is empty and bin 1 holds everything + let total_bytes: i64 = file_sizes.iter().sum(); + let num_files = file_sizes.len() as i64; + crc["fileSizeHistogram"] = serde_json::json!({ + "sortedBinBoundaries": [0, 100], + "fileCounts": [0, num_files], + "totalBytes": [0, total_bytes], + }); + + std::fs::write(&crc_file, serde_json::to_string(&crc).unwrap()).unwrap(); +} + +/// Cross-product test: (custom bins | default bins) x (incremental | non-incremental). +/// +/// For incremental operations, the in-memory CRC histogram should survive with correct +/// values and boundary type. For non-incremental operations, the histogram is dropped to +/// None and file stats become Indeterminate, regardless of boundary type. +/// +/// Custom bins are injected by rewriting the on-disk CRC to use 2-bin boundaries `[0, 100]` +/// before loading a fresh snapshot. The fresh snapshot deserializes the custom-bin CRC, and +/// the next commit's in-memory delta inherits those boundaries. +#[rstest] +#[tokio::test] +async fn test_file_histogram_with_bin_type_and_operation_type( + #[values(true, false)] use_custom_bins: bool, + #[values(true, false)] incremental: bool, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // ===== GIVEN: table with 1 file at v1 and CRC on disk ===== + let committed = create_table_and_commit(&table_path, engine.as_ref())?; + let snapshot_v0 = committed.post_commit_snapshot().unwrap(); + let col: ArrayRef = Arc::new(Int32Array::from(vec![1, 2, 3])); + let committed = insert_data(snapshot_v0.clone(), &engine, vec![col]) + .await? + .unwrap_committed(); + let snapshot_v1 = committed.post_commit_snapshot().unwrap(); + snapshot_v1.write_checksum(engine.as_ref())?; + + let v1_disk_sizes = parquet_file_sizes_on_disk(&table_path); + assert_eq!(v1_disk_sizes.len(), 1); + + if use_custom_bins { + rewrite_crc_with_custom_bins(&table_path, 1, &v1_disk_sizes); + } + + // Load fresh snapshot from disk (reads the possibly-modified CRC) + let fresh_v1 = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!(fresh_v1.version(), 1); + assert!(fresh_v1.crc_at_version().is_some()); + + // ===== WHEN: perform the operation ===== + let snapshot_v2 = if incremental { + let col: ArrayRef = Arc::new(Int32Array::from(vec![4, 5, 6])); + let committed = insert_data(fresh_v1, &engine, vec![col]) + .await? + .unwrap_committed(); + assert_eq!(committed.commit_version(), 2); + committed.post_commit_snapshot().unwrap().clone() + } else { + let committed = begin_transaction(fresh_v1, engine.as_ref())? + .with_operation("ANALYZE STATS".to_string()) + .commit(engine.as_ref())? + .unwrap_committed(); + assert_eq!(committed.commit_version(), 2); + committed.post_commit_snapshot().unwrap().clone() + }; + + // ===== THEN: verify histogram state ===== + let crc_v2 = snapshot_v2.crc_at_version().unwrap(); + + if !incremental { + // Non-incremental operations drop the histogram regardless of bin type + assert!(crc_v2.file_stats_state().is_indeterminate()); + assert!(crc_v2.file_stats().is_none()); + return Ok(()); + } + + // Incremental: histogram should be present and correct + let stats_v2 = crc_v2.file_stats().unwrap(); + let hist = stats_v2 + .file_size_histogram() + .expect("incremental op should preserve histogram"); + let counts = hist.file_counts(); + let bytes = hist.total_bytes(); + let v2_disk_sizes = parquet_file_sizes_on_disk(&table_path); + assert_eq!(v2_disk_sizes.len(), 2); + let total_disk_bytes: i64 = v2_disk_sizes.iter().sum(); + + if use_custom_bins { + // Custom [0, 100] boundaries: 2 bins, all files in bin 1 (all > 100 bytes) + assert_eq!(counts.len(), 2, "custom bins should have exactly 2 bins"); + assert_eq!(counts[0], 0, "no files should be in bin 0 ([0, 100))"); + assert_eq!(counts[1], 2, "both files should be in bin 1 ([100, inf))"); + assert_eq!(bytes[0], 0); + assert_eq!(bytes[1], total_disk_bytes); + } else { + // Default 95-bin boundaries: all small test files land in bin 0 ([0, 8KB)) + assert_eq!(counts.len(), 95, "default bins should have 95 bins"); + assert_histogram_totals(stats_v2, 2, total_disk_bytes); + } + + Ok(()) +} + +// ============================================================================ +// Stale-CRC advance on fresh build (reverse-replay) +// ============================================================================ + +// Commit one data file plus DomainMetadata "domain"->"value_{v}" and SetTxn "app"->{v} where `v` is +// the commit version. +async fn commit_with_dm_and_txn( + snapshot: SnapshotRef, + engine: &Arc>, + v: i64, +) -> DeltaResult { + commit_data(snapshot, engine, v, |txn| { + txn.with_domain_metadata("domain".to_string(), format!("value_{v}")) + .with_transaction_id("app".to_string(), v) + }) + .await +} + +/// Commit one data file at version `v`, letting `customize` attach the version-specific actions +/// (domain metadata, set transactions, removals) to the WRITE transaction. +async fn commit_data( + snapshot: SnapshotRef, + engine: &Arc>, + v: i64, + customize: impl FnOnce(Transaction) -> Transaction, +) -> DeltaResult { + let arrow_schema = TryFromKernel::try_from_kernel(snapshot.schema().as_ref())?; + let batch = RecordBatch::try_new( + Arc::new(arrow_schema), + vec![Arc::new(Int32Array::from(vec![v as i32]))], + ) + .map_err(|e| delta_kernel::Error::generic(e.to_string()))?; + let txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? + .with_operation("WRITE".to_string()) + .with_data_change(true); + let mut txn = customize(txn); + let write_context = txn.unpartitioned_write_context()?; + let adds = engine + .write_parquet(&ArrowEngineData::new(batch), &write_context) + .await?; + txn.add_files(adds); + Ok(txn.commit(engine.as_ref())?.unwrap_post_commit_snapshot()) +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum CrcStaleness { + MidSegment, + AtCheckpoint, + Absent, +} + +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_stale_crc_fresh_build_advance_matrix( + #[values( + CrcStaleness::MidSegment, + CrcStaleness::AtCheckpoint, + CrcStaleness::Absent + )] + crc_staleness: CrcStaleness, + #[values(false, true)] crc_missing_opt_fields: bool, +) -> DeltaResult<()> { + const CHECKPOINT_VERSION: i64 = 10; + const LATEST_VERSION: i64 = 20; + let crc_version = match crc_staleness { + CrcStaleness::MidSegment => Some((CHECKPOINT_VERSION + LATEST_VERSION) / 2), + CrcStaleness::AtCheckpoint => Some(CHECKPOINT_VERSION), + CrcStaleness::Absent => None, + }; + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + + // === Step 1: Create table with clustering, rowTracking, ICT === + let schema = schema_ref! { nullable "id": INTEGER }; + let mut snap = create_table(&table_path, schema, "test_engine") + .with_data_layout(DataLayout::clustered(["id"])) + .with_table_properties([ + ("delta.enableRowTracking", "true"), + ("delta.enableInCommitTimestamps", "true"), + ]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .with_domain_metadata("domain_at_create".to_string(), "value_0".to_string()) + .with_transaction_id("app_at_create".to_string(), 0) + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + + // === Step 2: Commits up to CHECKPOINT_VERSION, followed by a checkpoint. === + for v in 1..=CHECKPOINT_VERSION { + snap = commit_with_dm_and_txn(snap, &engine, v).await?; + } + snap = snap.checkpoint(engine.as_ref(), None)?.1; + + // === Step 3: Commit up to LATEST_VERSION, writing the CRC at its target version. === + // + // Each commit will write: + // - DomainMetadata: "domain"->"value_{v}" + // - SetTxn: "app"->{v} + if crc_version == Some(CHECKPOINT_VERSION) { + snap.write_checksum(engine.as_ref())?; + } + for v in (CHECKPOINT_VERSION + 1)..=LATEST_VERSION { + snap = commit_with_dm_and_txn(snap, &engine, v).await?; + if crc_version == Some(v) { + snap.write_checksum(engine.as_ref())?; + } + } + + // If test setup required missing CRC fields, then remove all optional ones. + if let Some(v) = crc_version.filter(|_| crc_missing_opt_fields) { + for field in ["fileSizeHistogram", "domainMetadata", "setTransactions"] { + strip_field_from_crc(&table_path, v as u64, field); + } + } + + // === Step 4: A fresh Snapshot at the latest version. === + let fresh = Snapshot::builder_for(&table_path) + .with_incremental_crc_replay(IncrementalReplay::Unlimited) + .build(engine.as_ref())?; + assert_eq!(fresh.version() as i64, LATEST_VERSION); + + let expect_crc_present = crc_staleness != CrcStaleness::Absent; + + // Define our engines that will be used below. + let real_engine_iff_crc_missing: &dyn Engine = if expect_crc_present { + &FailingEngine + } else { + engine.as_ref() + }; + let real_engine_iff_crc_missing_or_crc_missing_opt_fields: &dyn Engine = + if expect_crc_present && !crc_missing_opt_fields { + &FailingEngine + } else { + engine.as_ref() + }; + + // === Check: CRC presence === + assert_eq!( + fresh.crc_at_version().map(|c| c.version as i64), + expect_crc_present.then_some(LATEST_VERSION) + ); + + // === Check: file stats === + let stats = fresh.get_file_stats_if_present(); + assert_eq!(stats.is_some(), expect_crc_present); + if let Some(stats) = stats { + let disk = parquet_file_sizes_on_disk(&table_path); + assert_eq!(stats.num_files() as usize, disk.len()); + assert_eq!(stats.table_size_bytes(), disk.iter().sum::()); + assert_eq!( + stats.file_size_histogram().is_some(), + !crc_missing_opt_fields + ); + } + + // === Check: ICT === + // - If no CRC, then we must re-read the latest commit -> need real engine. + // - Else, we did CRC replay and cached the result -> use fake engine. + assert!(fresh + .get_in_commit_timestamp(real_engine_iff_crc_missing)? + .is_some()); + + // For both domain metadata and set transaction checks below: + // - If no CRC, then we must read non-zero commits -> need real engine. + // - Else, there is a CRC: + // - If we want a value set *before* the CRC was written (e.g. in create), then we need a real + // engine only if the CRC is missing optional fields. + // - If we want a value set *after* the CRC was written (e.g. in an insert), then we can use a + // fake engine. + + // === Check: domain metadata written *before* the CRC === + for domain in ["domain_at_create", "delta.clustering"] { + assert!(fresh + .get_domain_metadata_internal( + domain, + real_engine_iff_crc_missing_or_crc_missing_opt_fields + )? + .is_some()); + } + + // === Check: domain metadata written *after* the CRC === + for domain in ["domain", "delta.rowTracking"] { + assert!(fresh + .get_domain_metadata_internal(domain, real_engine_iff_crc_missing)? + .is_some()); + } + + // === Check: set transactions written *before* the CRC === + assert_eq!( + fresh.get_app_id_version( + "app_at_create", + real_engine_iff_crc_missing_or_crc_missing_opt_fields + )?, + Some(0) + ); + + // === Check: set transactions written *after* the CRC === + assert_eq!( + fresh.get_app_id_version("app", real_engine_iff_crc_missing)?, + Some(LATEST_VERSION) + ); + + Ok(()) +} + +#[tokio::test] +async fn test_stale_crc_fresh_build_non_incremental_op_trips_indeterminate() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // ===== GIVEN: a CRC at v0 (made stale by an insert at v1) ===== + let snap = create_table_and_commit(&table_path, engine.as_ref())? + .post_commit_snapshot() + .unwrap() + .clone(); + snap.write_checksum(engine.as_ref())?; + let snap = insert_data( + snap, + &engine, + vec![Arc::new(Int32Array::from(vec![1, 2, 3]))], + ) + .await? + .unwrap_post_commit_snapshot(); + + // ===== WHEN: a non-incremental operation (ANALYZE STATS) commits at v2 ===== + begin_transaction(snap, engine.as_ref())? + .with_operation("ANALYZE STATS".to_string()) + .commit(engine.as_ref())? + .unwrap_committed(); + + // ===== THEN: advancing the stale CRC trips file stats to Indeterminate, so they are not + // served and write_checksum is rejected ===== + let fresh = Snapshot::builder_for(&table_path) + .with_incremental_crc_replay(IncrementalReplay::Unlimited) + .build(engine.as_ref())?; + assert_eq!(fresh.version(), 2); + assert!(fresh + .crc_at_version() + .unwrap() + .file_stats_state() + .is_indeterminate()); + assert_eq!(fresh.get_file_stats_if_present(), None); + assert!(matches!( + fresh.write_checksum(engine.as_ref()), + Err(delta_kernel::Error::ChecksumWriteUnsupported(_)) + )); + + Ok(()) +} + +#[tokio::test] +async fn test_stale_crc_fresh_build_fails_load_when_advance_commit_is_corrupt() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + let snap = create_table_and_commit(&table_path, engine.as_ref())? + .post_commit_snapshot() + .unwrap() + .clone(); + snap.write_checksum(engine.as_ref())?; + insert_data( + snap, + &engine, + vec![Arc::new(Int32Array::from(vec![1, 2, 3]))], + ) + .await? + .unwrap_committed(); + + let commit_v1 = _temp_dir + .path() + .join("_delta_log/00000000000000000001.json"); + std::fs::write(&commit_v1, b"}}} not valid commit json").unwrap(); + + assert!(Snapshot::builder_for(&table_path) + .with_incremental_crc_replay(IncrementalReplay::Unlimited) + .build(engine.as_ref()) + .is_err()); + + Ok(()) +} + +// ============================================================================ +// Domain metadata rooted in a stale (but authoritative) CRC +// ============================================================================ + +/// Delegates every handler to `inner`, but panics on `read_parquet_files`. A domain-metadata query +/// reads only JSON commits, so the only parquet a scan would touch is the V1 checkpoint; the panic +/// proves the rooted path skips it. That the pruned tail segment itself excludes the checkpoint and +/// every commit at/below the base CRC is covered by `test_segment_crc_filtering`. +struct NoParquetReadsEngine { + inner: Arc, +} + +struct NoParquetReadsHandler { + inner: Arc, +} + +impl ParquetHandler for NoParquetReadsHandler { + fn read_parquet_files( + &self, + _files: &[FileMeta], + _physical_schema: SchemaRef, + _predicate: Option, + ) -> DeltaResult { + panic!("read_parquet_files called: the checkpoint must not be read on the rooted path"); + } + + fn write_parquet_file( + &self, + location: Url, + data: DeltaResultIteratorStatic>, + ) -> DeltaResult<()> { + self.inner.write_parquet_file(location, data) + } + + fn read_parquet_footer(&self, file: &FileMeta) -> DeltaResult { + self.inner.read_parquet_footer(file) + } +} + +impl Engine for NoParquetReadsEngine { + fn evaluation_handler(&self) -> Arc { + self.inner.evaluation_handler() + } + fn storage_handler(&self) -> Arc { + self.inner.storage_handler() + } + fn json_handler(&self) -> Arc { + self.inner.json_handler() + } + fn parquet_handler(&self) -> Arc { + Arc::new(NoParquetReadsHandler { + inner: self.inner.parquet_handler(), + }) + } +} + +/// Builds checkpoint@10, commits to v20, and a CRC at v15. Loaded with the default +/// `Disabled` replay so the v15 CRC is retained as a stale base (`crc_at_version()` is `None`). +/// Seeds four domains covering every reconcile arm: +/// - `dom_before` set at v5 and never touched again (base stands), +/// - `dom_updated` set at v6 then re-set at v16 (tail config overrides the base), +/// - `dom_after` set at v18 (new in the tail), +/// - `dom_removed` set at v8 then tombstoned at v17 (tail removal shadows the base). +async fn setup_stale_crc_dm_table( + engine: &Arc>, + table_path: &str, +) -> DeltaResult<()> { + let schema = schema_ref! { nullable "id": INTEGER }; + let mut snap = create_table(table_path, schema, "test_engine") + .with_table_properties([("delta.feature.domainMetadata", "supported")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + + for v in 1..=20i64 { + snap = commit_data(snap, engine, v, |txn| match v { + 5 => txn.with_domain_metadata("dom_before".to_string(), "cfg_before".to_string()), + 6 => txn.with_domain_metadata("dom_updated".to_string(), "cfg_updated".to_string()), + 8 => txn.with_domain_metadata("dom_removed".to_string(), "cfg_removed".to_string()), + 16 => { + txn.with_domain_metadata("dom_updated".to_string(), "cfg_updated_v16".to_string()) + } + 17 => txn.with_domain_metadata_removed("dom_removed".to_string()), + 18 => txn.with_domain_metadata("dom_after".to_string(), "cfg_after".to_string()), + _ => txn, + }) + .await?; + + if v == 10 { + snap = snap.checkpoint(engine.as_ref(), None)?.1; + } + if v == 15 { + snap.write_checksum(engine.as_ref())?; + } + } + Ok(()) +} + +/// The active-domain set at v20 the rooted scan must resolve, regardless of query shape. +fn expected_active() -> HashMap { + [ + ("dom_before", "cfg_before"), + ("dom_updated", "cfg_updated_v16"), + ("dom_after", "cfg_after"), + ] + .iter() + .map(|(d, c)| (d.to_string(), c.to_string())) + .collect() +} + +/// A domain-metadata query shape and its expected result against [`setup_stale_crc_dm_table`]. +enum Query { + /// `get_domain_metadata(domain)` for one domain. + One(&'static str, Option<&'static str>), + /// `get_domain_metadatas_internal` with a multi-domain filter (the real caller's shape). + Filter(&'static [&'static str]), + /// `get_all_domain_metadata()`. + All, +} + +// Every query shape resolves against a stale Complete CRC by scanning only the commit tail. The +// NoParquetReadsEngine panics if the v10 checkpoint is read, proving the checkpoint is skipped. +#[rstest] +#[case::base_stands(Query::One("dom_before", Some("cfg_before")))] +#[case::tail_overrides_base(Query::One("dom_updated", Some("cfg_updated_v16")))] +#[case::tail_added(Query::One("dom_after", Some("cfg_after")))] +#[case::tail_removed(Query::One("dom_removed", None))] +#[case::never_existed(Query::One("dom_missing", None))] +#[case::multi_domain_filter(Query::Filter(&[ + "dom_before", "dom_updated", "dom_after", "dom_removed", "dom_missing", +]))] +#[case::unfiltered(Query::All)] +#[tokio::test(flavor = "multi_thread")] +async fn test_dm_query_rooted_in_stale_complete_crc(#[case] query: Query) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + setup_stale_crc_dm_table(&engine, &table_path).await?; + + let fresh = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!(fresh.version(), 20); + assert!( + fresh.crc_at_version().is_none(), + "the stale CRC must not sit at the snapshot version" + ); + + let engine = NoParquetReadsEngine { + inner: engine.clone(), + }; + match query { + Query::One(domain, expected) => assert_eq!( + fresh.get_domain_metadata(domain, &engine)?, + expected.map(str::to_string) + ), + Query::Filter(keys) => { + let filter = keys.iter().copied().collect(); + let got: HashMap = fresh + .get_domain_metadatas_internal(&engine, Some(&filter))? + .into_iter() + .map(|(domain, dm)| (domain, dm.configuration().to_string())) + .collect(); + assert_eq!(got, expected_active()); + } + Query::All => { + let all: HashMap = fresh + .get_all_domain_metadata(&engine)? + .into_iter() + .map(|dm| (dm.domain().to_string(), dm.configuration().to_string())) + .collect(); + assert_eq!(all, expected_active()); + } + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread")] +async fn test_dm_query_stale_partial_crc_falls_through_to_full_scan() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + setup_stale_crc_dm_table(&engine, &table_path).await?; + + // Strip domainMetadata from the v15 CRC so it reloads as Partial: not authoritative, so the + // query must not take the rooted path. + strip_field_from_crc(&table_path, 15, "domainMetadata"); + + let fresh = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!(fresh.version(), 20); + assert!(fresh.crc_at_version().is_none()); + + // Results are still correct via the full scan (with the real engine). + assert_eq!( + fresh.get_domain_metadata("dom_before", engine.as_ref())?, + Some("cfg_before".to_string()) + ); + assert_eq!( + fresh.get_domain_metadata("dom_removed", engine.as_ref())?, + None + ); + + // A full scan reads the v10 checkpoint, so NoParquetReadsEngine panics: the Partial base did + // NOT take the rooted (checkpoint-skipping) path. + let no_parquet = NoParquetReadsEngine { + inner: engine.clone(), + }; + assert!(std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + fresh.get_domain_metadata("dom_before", &no_parquet).ok() + })) + .is_err()); + + Ok(()) +} + +// ============================================================================ +// Set transactions rooted in a stale (but authoritative) CRC +// ============================================================================ + +/// Builds checkpoint@10, commits to v20, and a CRC at v15. Loaded with the default `Disabled` +/// replay so the v15 CRC is retained as a stale base (`crc_at_version()` is `None`). Seeds three +/// app_ids covering every reconcile arm: +/// - `app_before` set at v5 and never touched again (base stands), +/// - `app_updated` set at v6 then re-set to a higher version at v16 (tail supersedes the base), +/// - `app_after` set at v18 (new in the tail). +async fn setup_stale_crc_txn_table( + engine: &Arc>, + table_path: &str, +) -> DeltaResult<()> { + let schema = schema_ref! { nullable "id": INTEGER }; + let mut snap = create_table(table_path, schema, "test_engine") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + + for v in 1..=20i64 { + snap = commit_data(snap, engine, v, |txn| match v { + 5 => txn.with_transaction_id("app_before".to_string(), 5), + 6 => txn.with_transaction_id("app_updated".to_string(), 6), + 16 => txn.with_transaction_id("app_updated".to_string(), 16), + 18 => txn.with_transaction_id("app_after".to_string(), 18), + _ => txn, + }) + .await?; + + if v == 10 { + snap = snap.checkpoint(engine.as_ref(), None)?.1; + } + if v == 15 { + snap.write_checksum(engine.as_ref())?; + } + } + Ok(()) +} + +// Every app_id resolves against a stale Complete CRC by scanning only the commit tail. The +// NoParquetReadsEngine panics if the v10 checkpoint is read, proving the checkpoint is skipped. +#[rstest] +#[case::base_stands("app_before", Some(5))] +#[case::tail_supersedes_base("app_updated", Some(16))] +#[case::tail_added("app_after", Some(18))] +#[case::never_existed("app_missing", None)] +#[tokio::test(flavor = "multi_thread")] +async fn test_txn_query_rooted_in_stale_complete_crc( + #[case] app_id: &str, + #[case] expected: Option, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + setup_stale_crc_txn_table(&engine, &table_path).await?; + + let fresh = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!(fresh.version(), 20); + assert!( + fresh.crc_at_version().is_none(), + "the stale CRC must not sit at the snapshot version" + ); + + let engine = NoParquetReadsEngine { + inner: engine.clone(), + }; + assert_eq!(fresh.get_app_id_version(app_id, &engine)?, expected); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread")] +async fn test_txn_query_stale_partial_crc_falls_through_to_full_scan() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + setup_stale_crc_txn_table(&engine, &table_path).await?; + + // Strip setTransactions from the v15 CRC so it reloads as Partial: not authoritative, so the + // query must not take the rooted path. + strip_field_from_crc(&table_path, 15, "setTransactions"); + + let fresh = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + assert_eq!(fresh.version(), 20); + assert!(fresh.crc_at_version().is_none()); + + // Results are still correct via the full scan (with the real engine). + assert_eq!( + fresh.get_app_id_version("app_before", engine.as_ref())?, + Some(5) + ); + assert_eq!( + fresh.get_app_id_version("app_updated", engine.as_ref())?, + Some(16) + ); + + // A full scan reads the v10 checkpoint, so NoParquetReadsEngine panics: the Partial base did + // NOT take the rooted (checkpoint-skipping) path. + let no_parquet = NoParquetReadsEngine { + inner: engine.clone(), + }; + assert!(std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + fresh.get_app_id_version("app_before", &no_parquet).ok() + })) + .is_err()); + + Ok(()) +} diff --git a/kernel/tests/integration/log/empty_log_files.rs b/kernel/tests/integration/log/empty_log_files.rs new file mode 100644 index 0000000000..6491b993f5 --- /dev/null +++ b/kernel/tests/integration/log/empty_log_files.rs @@ -0,0 +1,197 @@ +//! Integration tests for snapshot loading with empty (0-byte) log files. +//! +//! These tests verify that kernel handles corrupt log files gracefully: +//! - 0-byte compacted files are skipped, falling back to individual commits +//! - 0-byte commit files produce a clear error message +//! - 0-byte checkpoint files are skipped, falling back to an older checkpoint +//! - 0-byte CRC files are skipped (CRC is optional) + +use std::sync::Arc; + +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::ObjectStoreExt as _; +use delta_kernel::schema::{DataType, StructField, StructType}; +use delta_kernel::Snapshot; +use test_utils::{add_commit, compacted_log_path_for_versions, create_table, engine_store_setup}; + +fn simple_schema() -> Arc { + Arc::new( + StructType::try_new(vec![StructField::nullable("id", DataType::INTEGER)]) + .expect("valid schema"), + ) +} + +fn commit_with_add(file_name: &str, num_records: i64) -> String { + format!( + r#"{{"commitInfo":{{"timestamp":1587968586000,"operation":"WRITE","operationParameters":{{"mode":"Append"}},"isBlindAppend":true}}}} +{{"add":{{"path":"{file_name}","partitionValues":{{}},"size":1024,"modificationTime":1587968586000,"dataChange":true,"stats":"{{\"numRecords\":{num_records},\"nullCount\":{{\"id\":0}},\"minValues\":{{\"id\":1}},\"maxValues\":{{\"id\":10}}}}"}}}} +"# + ) +} + +#[tokio::test] +async fn snapshot_loads_with_zero_byte_compaction() -> Result<(), Box> { + let (store, engine, table_location) = engine_store_setup("test_zero_byte_compaction", None); + + let table_url = create_table( + store.clone(), + table_location, + simple_schema(), + &[], + false, + vec![], + vec![], + ) + .await?; + + add_commit( + table_url.as_str(), + store.as_ref(), + 1, + commit_with_add("part-00000-file1.parquet", 10), + ) + .await?; + + add_commit( + table_url.as_str(), + store.as_ref(), + 2, + commit_with_add("part-00001-file2.parquet", 20), + ) + .await?; + + // Write a 0-byte compaction covering v0-v2 + let compaction_path = compacted_log_path_for_versions(0, 2, "json"); + let table_compaction = Path::from(format!( + "test_zero_byte_compaction/{}", + compaction_path.as_ref() + )); + store + .put(&table_compaction, bytes::Bytes::new().into()) + .await?; + + // Snapshot should load successfully by falling back to individual commits + let snapshot = Snapshot::builder_for(table_url).build(&engine)?; + assert_eq!(snapshot.version(), 2); + + Ok(()) +} + +#[tokio::test] +async fn snapshot_loads_with_zero_byte_commit() -> Result<(), Box> { + let (store, engine, table_location) = engine_store_setup("test_zero_byte_commit", None); + + let table_url = create_table( + store.clone(), + table_location, + simple_schema(), + &[], + false, + vec![], + vec![], + ) + .await?; + + add_commit( + table_url.as_str(), + store.as_ref(), + 1, + commit_with_add("part-00000-file1.parquet", 10), + ) + .await?; + + // Overwrite commit v1 with 0 bytes -- the file stays in the listing + // (commits are not skipped). The JSON handler reads it as an empty + // commit (no actions). + let empty_commit_path = + Path::from("test_zero_byte_commit/_delta_log/00000000000000000001.json"); + store + .put(&empty_commit_path, bytes::Bytes::new().into()) + .await?; + + let snapshot = Snapshot::builder_for(table_url).build(&engine)?; + assert_eq!(snapshot.version(), 1); + + Ok(()) +} + +#[tokio::test] +async fn snapshot_loads_with_zero_byte_checkpoint() -> Result<(), Box> { + let (store, engine, table_location) = engine_store_setup("test_zero_byte_checkpoint", None); + + let table_url = create_table( + store.clone(), + table_location, + simple_schema(), + &[], + false, + vec![], + vec![], + ) + .await?; + + add_commit( + table_url.as_str(), + store.as_ref(), + 1, + commit_with_add("part-00000-file1.parquet", 10), + ) + .await?; + + add_commit( + table_url.as_str(), + store.as_ref(), + 2, + commit_with_add("part-00001-file2.parquet", 20), + ) + .await?; + + // Write a 0-byte checkpoint file at v2 + let empty_checkpoint_path = + Path::from("test_zero_byte_checkpoint/_delta_log/00000000000000000002.checkpoint.parquet"); + store + .put(&empty_checkpoint_path, bytes::Bytes::new().into()) + .await?; + + // Snapshot should load by using commits from v0 (the empty checkpoint is skipped) + let snapshot = Snapshot::builder_for(table_url).build(&engine)?; + assert_eq!(snapshot.version(), 2); + + Ok(()) +} + +#[tokio::test] +async fn snapshot_loads_with_zero_byte_crc() -> Result<(), Box> { + let (store, engine, table_location) = engine_store_setup("test_zero_byte_crc", None); + + let table_url = create_table( + store.clone(), + table_location, + simple_schema(), + &[], + false, + vec![], + vec![], + ) + .await?; + + add_commit( + table_url.as_str(), + store.as_ref(), + 1, + commit_with_add("part-00000-file1.parquet", 10), + ) + .await?; + + // Write a 0-byte CRC file at v1 + let empty_crc_path = Path::from("test_zero_byte_crc/_delta_log/00000000000000000001.crc"); + store + .put(&empty_crc_path, bytes::Bytes::new().into()) + .await?; + + // Snapshot should load normally (CRC is optional) + let snapshot = Snapshot::builder_for(table_url).build(&engine)?; + assert_eq!(snapshot.version(), 1); + + Ok(()) +} diff --git a/kernel/tests/integration/log/history_manager.rs b/kernel/tests/integration/log/history_manager.rs new file mode 100644 index 0000000000..193e1202e5 --- /dev/null +++ b/kernel/tests/integration/log/history_manager.rs @@ -0,0 +1,160 @@ +//! Integration tests for the public `history_manager` API. + +use std::fs::OpenOptions; +use std::ops::RangeInclusive; +use std::time::{Duration, SystemTime}; + +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::history_manager::{get_earliest_commit, latest_version_as_of, HistoryCommitType}; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::{ObjectStore, ObjectStoreExt as _}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::{DeltaResult, Snapshot, Version}; +use rstest::rstest; +use test_utils::delta_kernel_default_engine::DefaultEngineBuilder; +use test_utils::table_builder::{FeatureSet, LogState, TestTableBuilder, VersionTarget}; +use test_utils::{build_snapshot, test_table_setup}; +use url::Url; + +use crate::common::write_utils::get_simple_int_schema; + +async fn remove_commits(store: &dyn ObjectStore, versions: RangeInclusive) { + for version in versions { + let path = Path::from(format!("_delta_log/{version:020}.json")); + store.delete(&path).await.unwrap(); + } +} + +#[rstest] +#[case::recreatable_v0_present(None, None, HistoryCommitType::Recreatable, 0)] +#[case::published_v0_present(None, None, HistoryCommitType::Published, 0)] +#[case::recreatable_checkpoint_anchors(Some(0..=4), None, HistoryCommitType::Recreatable, 5)] +#[case::published_lowest_surviving_json(Some(0..=5), None, HistoryCommitType::Published, 6)] +#[case::recreatable_commit_at_checkpoint_survives(Some(0..=4), Some(9), HistoryCommitType::Recreatable, 5)] +#[case::published_commit_at_checkpoint_survives(Some(0..=4), Some(9), HistoryCommitType::Published, 5)] +#[tokio::test(flavor = "multi_thread")] +async fn test_get_earliest_commit( + #[case] cleanup: Option>, + #[case] earliest_ratified_commit_version: Option, + #[case] commit_type: HistoryCommitType, + #[case] expected_version: Version, +) -> Result<(), Box> { + let table = TestTableBuilder::new() + .with_log_state(LogState::with_latest_version(8).with_checkpoint_at([5])) + .build()?; + let store = table.store().clone(); + let engine = DefaultEngineBuilder::new(store.clone()).build(); + let log_root = Url::parse(table.table_root())?.join("_delta_log/")?; + + if let Some(versions) = cleanup { + remove_commits(store.as_ref(), versions).await; + } + + let earliest = get_earliest_commit( + &engine, + &log_root, + earliest_ratified_commit_version, + commit_type, + )?; + assert_eq!(earliest, expected_version); + Ok(()) +} + +/// End-to-end check that [`VersionTarget::AtTimestamp`] resolves through `build_snapshot!` +/// to the version `latest_version_as_of` selects. Written on the local filesystem with +/// explicit, distinct per-commit modification times, which the in-memory sweep can't provide +/// (see `version_at_timestamp_max`). Resolution correctness across layouts is covered by the +/// `history_manager` unit tests. +#[test] +fn test_at_timestamp_resolves_to_intermediate_version() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + + // v0: CreateTable + let schema = get_simple_int_schema(); + let mut snap = create_table(&table_path, schema, "AtTimestampTest/1.0") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + + // v1..=4: noop commits (each writes a metaData-free, add-free commit JSON). + for _ in 1..=4 { + snap = test_utils::begin_transaction(snap.clone(), engine.as_ref())? + .with_engine_info("AtTimestampTest") + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + } + + // Set each commit's mtime to a distinct, monotonic value (in ms). + let table_url = delta_kernel::try_parse_uri(&table_path)?; + let log_dir = table_url.to_file_path().unwrap().join("_delta_log"); + for v in 0..=4u64 { + let file_path = log_dir.join(format!("{v:020}.json")); + let file = OpenOptions::new().write(true).open(&file_path).unwrap(); + let time = SystemTime::UNIX_EPOCH + Duration::from_millis((v + 1) * 1000); + file.set_modified(time).unwrap(); + } + + // 2500ms lands strictly between v1 (2000) and v2 (3000), so it resolves to v1. + let target = VersionTarget::AtTimestamp(2500); + let snap_at_ts = build_snapshot!(target, &table_path, engine.as_ref()); + assert_eq!(snap_at_ts.version(), 1); + + // i64::MAX resolves to latest. + let target_max = VersionTarget::AtTimestamp(i64::MAX); + let snap_max = build_snapshot!(target_max, &table_path, engine.as_ref()); + assert_eq!(snap_max.version(), 4); + + Ok(()) +} + +/// Targets a specific commit's timestamp end-to-end through the builder. With ICT enabled the +/// kernel writes a distinct, monotonic timestamp per commit, so each version's timestamp can be +/// read back (via a snapshot capped at that version) and time-traveled to via +/// [`VersionTarget::AtTimestamp`]. Querying a version's exact timestamp resolves to that version; +/// querying one millisecond earlier resolves to its predecessor. Deterministic without setting +/// commit timestamps explicitly, since ICT values are derived from the table rather than hardcoded. +#[test] +fn test_at_timestamp_targets_specific_commit_via_ict() -> DeltaResult<()> { + let table = TestTableBuilder::new() + .with_features(FeatureSet::new().ict()) + .with_log_state(LogState::with_latest_version(4)) + .build()?; + let engine = table.engine(); + let table_root = table.table_root(); + + for v in 0..=4u64 { + let capped = Snapshot::builder_for(table_root) + .at_version(v) + .build(&engine)?; + let commit = + latest_version_as_of(&capped, &engine, i64::MAX, HistoryCommitType::Recreatable)?; + assert_eq!(commit.version, v); + + let at_exact = build_snapshot!( + VersionTarget::AtTimestamp(commit.timestamp), + table_root, + &engine + ); + assert_eq!( + at_exact.version(), + v, + "v{v}'s exact timestamp must resolve to v{v}" + ); + + if v > 0 { + let at_prev = build_snapshot!( + VersionTarget::AtTimestamp(commit.timestamp - 1), + table_root, + &engine + ); + assert_eq!( + at_prev.version(), + v - 1, + "one ms before v{v}'s timestamp must resolve to v{}", + v - 1 + ); + } + } + + Ok(()) +} diff --git a/kernel/tests/log_compaction.rs b/kernel/tests/integration/log/log_compaction.rs similarity index 97% rename from kernel/tests/log_compaction.rs rename to kernel/tests/integration/log/log_compaction.rs index 2a2eaf1cd5..6fcde0ed8a 100644 --- a/kernel/tests/log_compaction.rs +++ b/kernel/tests/integration/log/log_compaction.rs @@ -1,13 +1,11 @@ -use std::sync::Arc; use std::time::{SystemTime, UNIX_EPOCH}; use delta_kernel::engine::to_json_bytes; use delta_kernel::object_store::path::Path; -use delta_kernel::object_store::ObjectStore; -use delta_kernel::schema::{DataType, StructField, StructType}; +use delta_kernel::object_store::ObjectStoreExt as _; +use delta_kernel::schema::schema_ref; use delta_kernel::Snapshot; use test_utils::{create_table, engine_store_setup}; - use url::Url; /// Convert a URL to a `delta_kernel::object_store::Path` @@ -22,14 +20,12 @@ fn url_to_object_store_path(url: &Url) -> Result Result<(), Box> { let _ = tracing_subscriber::fmt::try_init(); // Create a simple table schema: one int column named 'id' - let schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "id", - DataType::INTEGER, - )])?); + let schema = schema_ref! { nullable "id": INTEGER }; // Setup engine and storage - this creates a proper temporary table let (store, engine, table_location) = engine_store_setup("test_compaction_table", None); @@ -195,13 +191,11 @@ async fn action_reconciliation_round_trip() -> Result<(), Box Result<(), Box> { let _ = tracing_subscriber::fmt::try_init(); - let schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "id", - DataType::INTEGER, - )])?); + let schema = schema_ref! { nullable "id": INTEGER }; let (store, engine, table_location) = engine_store_setup("test_expired_tombstone_table", None); diff --git a/kernel/tests/log_tail.rs b/kernel/tests/integration/log/log_tail.rs similarity index 68% rename from kernel/tests/log_tail.rs rename to kernel/tests/integration/log/log_tail.rs index 773fce5792..62f9a47021 100644 --- a/kernel/tests/log_tail.rs +++ b/kernel/tests/integration/log/log_tail.rs @@ -1,31 +1,15 @@ use std::sync::Arc; -use url::Url; - -use delta_kernel::engine::default::executor::tokio::TokioBackgroundExecutor; -use delta_kernel::engine::default::{DefaultEngine, DefaultEngineBuilder}; +use delta_kernel::history_manager::{first_version_after, latest_version_as_of, HistoryCommitType}; use delta_kernel::object_store::memory::InMemory; -use delta_kernel::object_store::path::Path; -use delta_kernel::{FileMeta, LogPath, Snapshot}; - +use delta_kernel::Snapshot; +use test_utils::delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor; +use test_utils::delta_kernel_default_engine::{DefaultEngine, DefaultEngineBuilder}; use test_utils::{ - actions_to_string, add_commit, add_staged_commit, delta_path_for_version, TestAction, + actions_to_string, actions_to_string_catalog_managed, add_commit, add_staged_commit, + create_log_path, delta_path_for_version, TestAction, }; - -/// Helper function to create a LogPath for a commit at the given version -fn create_log_path(table_root: &Url, commit_path: Path) -> LogPath { - let file_meta = create_file_meta(table_root, commit_path); - LogPath::try_new(file_meta).expect("Failed to create LogPath") -} - -fn create_file_meta(table_root: &Url, commit_path: Path) -> FileMeta { - let commit_url = table_root.join(commit_path.as_ref()).unwrap(); - FileMeta { - location: commit_url, - last_modified: 123, - size: 100, // arbitrary size - } -} +use url::Url; fn setup_test() -> ( Arc, @@ -44,12 +28,18 @@ async fn basic_snapshot_with_log_tail_staged_commits() -> Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box> { + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + let ict_v0: i64 = 1587968586154; + let ict_v1: i64 = ict_v0 + 100; + let ict_v2: i64 = ict_v0 + 200; + let staged_ict_body = |ict: i64| { + format!( + r#"{{"commitInfo":{{"timestamp":{ict},"inCommitTimestamp":{ict},"operation":"WRITE","isBlindAppend":true}}}}"# + ) + }; + + // publish v0; v1, v2 are ratified staged commits. Also include the corresponding ICT to each + // version. + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string_catalog_managed(vec![TestAction::Metadata]), + ) + .await?; + let path1 = add_staged_commit(table_root, storage.as_ref(), 1, staged_ict_body(ict_v1)).await?; + let path2 = add_staged_commit(table_root, storage.as_ref(), 2, staged_ict_body(ict_v2)).await?; + + let log_tail = vec![ + create_log_path(&table_url, path1), + create_log_path(&table_url, path2), + ]; + let snapshot = Snapshot::builder_for(table_root) + .with_log_tail(log_tail) + .with_max_catalog_version(2) + .build(engine.as_ref())?; + assert_eq!(snapshot.version(), 2); + + // latest_version_as_of rounds down; ICT between v1 and v2 rounds to v1. + let e = engine.as_ref(); + let ct = HistoryCommitType::Published; + assert_eq!(latest_version_as_of(&snapshot, e, ict_v1, ct)?.version, 1); + assert_eq!( + latest_version_as_of(&snapshot, e, ict_v1 + 50, ct)?.version, + 1 + ); + assert_eq!(latest_version_as_of(&snapshot, e, ict_v2, ct)?.version, 2); + // timestamps before v0 is out of range + assert!(latest_version_as_of(&snapshot, e, ict_v0 - 1, ct).is_err()); + + // first_version_after rounds up; ICT between v0 and v1 rounds to v1. + assert_eq!( + first_version_after(&snapshot, e, ict_v0 + 1, ct)?.version, + 1 + ); + assert_eq!(first_version_after(&snapshot, e, ict_v2, ct)?.version, 2); + + Ok(()) +} + #[tokio::test] async fn basic_snapshot_with_log_tail() -> Result<(), Box> { let (storage, engine, table_url) = setup_test(); @@ -236,9 +293,15 @@ async fn incremental_snapshot_with_log_tail() -> Result<(), Box Result<(), Box Result<(), Box Result<(), Box Result<(), Box> +{ + let (storage, engine, table_url) = setup_test(); + let table_root = table_url.as_str(); + + // commits 0, 1, 2 in storage (catalog-managed) + let actions = vec![TestAction::Metadata]; + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string_catalog_managed(actions), + ) + .await?; + let actions = vec![TestAction::Add("file_1.parquet".to_string())]; + add_commit(table_root, storage.as_ref(), 1, actions_to_string(actions)).await?; + let actions = vec![TestAction::Add("file_2.parquet".to_string())]; + add_commit(table_root, storage.as_ref(), 2, actions_to_string(actions)).await?; + + // Simulate a request to the catalog which reports max_catalog_version = 2 + let mcv = 2; + + // Build initial snapshot at version 1, catalog knows about version 2 + let initial_snapshot = Snapshot::builder_for(table_root) + .at_version(1) + .with_max_catalog_version(mcv) + .build(engine.as_ref())?; + assert_eq!(initial_snapshot.version(), 1); + + // Catalog reported v2 as the max ratified version. A moment later, v3 was + // ratified and published to the log -- but the client is unaware of v3. + let actions = vec![TestAction::Add("file_3.parquet".to_string())]; + add_commit(table_root, storage.as_ref(), 3, actions_to_string(actions)).await?; + + // Incremental update: catalog reported v2 as max, log_tail includes v2 + let log_tail = vec![create_log_path( + &table_url, + delta_path_for_version(mcv, "json"), + )]; + let new_snapshot = Snapshot::builder_from(initial_snapshot) + .with_log_tail(log_tail) + .with_max_catalog_version(mcv) + .build(engine.as_ref())?; + + // Snapshot respects the catalog's reported max version (v2) + assert_eq!(new_snapshot.version(), 2); + + Ok(()) +} + #[tokio::test] async fn log_tail_exceeds_requested_version() -> Result<(), Box> { let (storage, engine, table_url) = setup_test(); diff --git a/kernel/tests/integration/log/mod.rs b/kernel/tests/integration/log/mod.rs new file mode 100644 index 0000000000..8cfc5757e5 --- /dev/null +++ b/kernel/tests/integration/log/mod.rs @@ -0,0 +1,10 @@ +//! Log-replay and checkpoint-structure integration tests. + +mod checkpoint_transform; +mod crc; +mod empty_log_files; +mod history_manager; +mod log_compaction; +mod log_tail; +mod snapshot_build; +mod v2_checkpoints; diff --git a/kernel/tests/integration/log/snapshot_build.rs b/kernel/tests/integration/log/snapshot_build.rs new file mode 100644 index 0000000000..2fc65c461d --- /dev/null +++ b/kernel/tests/integration/log/snapshot_build.rs @@ -0,0 +1,246 @@ +//! Integration tests for [`Snapshot`] build semantics. + +use std::sync::Arc; + +use delta_kernel::arrow::array::{ArrayRef, Int32Array}; +use delta_kernel::committer::{Committer, FileSystemCommitter}; +use delta_kernel::schema::{schema_ref, DataType, StructField, StructType}; +use delta_kernel::snapshot::{ + CheckpointWriteResult, ChecksumWriteResult, IncrementalReplay, SnapshotBuilder, +}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::{DeltaResult, Error, Snapshot, Version}; +use rstest::rstest; +use serde_json::json; +use test_utils::delta_kernel_default_engine::executor::TaskExecutor; +use test_utils::delta_kernel_default_engine::DefaultEngine; +use test_utils::{ + add_commit, assert_result_error_with_message, engine_store_setup, insert_data_with, + test_table_setup_mt, TestCatalogCommitter, +}; + +/// The newest version of the table built by [`setup_multi_version_table`]. +const LATEST_VERSION: Version = 3; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum TableKind { + FileSystem, + CatalogManaged, +} + +impl TableKind { + fn committer(self) -> Box { + match self { + TableKind::FileSystem => Box::new(FileSystemCommitter::new()), + TableKind::CatalogManaged => Box::new(TestCatalogCommitter), + } + } +} + +fn maybe_attach_max_catalog_version( + builder: SnapshotBuilder, + max_catalog_version: Version, + kind: TableKind, +) -> SnapshotBuilder { + match kind { + TableKind::FileSystem => builder, + TableKind::CatalogManaged => builder.with_max_catalog_version(max_catalog_version), + } +} + +async fn append_row( + snapshot: Arc, + engine: &Arc>, + kind: TableKind, + value: i32, +) -> DeltaResult> { + let column: ArrayRef = Arc::new(Int32Array::from(vec![value])); + Ok(insert_data_with( + snapshot, + engine, + vec![column], + kind.committer(), + "WRITE", + true, /* data_change */ + false, /* is_blind_append */ + ) + .await? + .unwrap_post_commit_snapshot()) +} + +/// Builds a table of the given kind at versions 0..=3 (create-table + three appends) and persists +/// a version-0 CRC so a later build with [`IncrementalReplay::Unlimited`] can resolve an in-memory +/// CRC and exercise `write_checksum`. The latest version is 3. +async fn setup_multi_version_table( + engine: &Arc>, + table_path: &str, + kind: TableKind, +) -> DeltaResult<()> { + let schema = schema_ref! { nullable "id": INTEGER }; + let builder = create_table(table_path, schema, "test_engine"); + let builder = match kind { + TableKind::FileSystem => builder, + TableKind::CatalogManaged => builder.with_table_properties([ + ("delta.feature.catalogManaged", "supported"), + ("io.unitycatalog.tableId", "snapshot-build-test"), + ]), + }; + let create_snapshot = builder + .build(engine.as_ref(), kind.committer())? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + + // The create-table snapshot is built as latest (version 0 is necessarily the latest). + assert!(create_snapshot.is_built_as_latest()); + create_snapshot.write_checksum(engine.as_ref())?; + + let mut snap = maybe_attach_max_catalog_version(Snapshot::builder_for(table_path), 0, kind) + .build(engine.as_ref())?; + for value in 1..=LATEST_VERSION as i32 { + snap = append_row(snap, engine, kind, value).await?; + } + Ok(()) +} + +#[tokio::test] +async fn deeply_nested_schema_snapshot_load_returns_schema_error( +) -> Result<(), Box> { + let deeply_nested_schema = (0..42).fold( + StructType::new_unchecked([StructField::nullable("leaf", DataType::INTEGER)]), + |schema, depth| { + StructType::new_unchecked([StructField::nullable(format!("level_{depth}"), schema)]) + }, + ); + let (store, engine, table_url) = engine_store_setup("deeply_nested_schema", None); + let commit = [ + json!({"protocol": {"minReaderVersion": 1, "minWriterVersion": 2}}), + json!({ + "metaData": { + "id": "deeply-nested-schema", + "format": {"provider": "parquet", "options": {}}, + "schemaString": serde_json::to_string(&deeply_nested_schema)?, + "partitionColumns": [], + "configuration": {}, + "createdTime": 0 + } + }), + ] + .map(|action| action.to_string()) + .join("\n"); + add_commit(table_url.as_str(), store.as_ref(), 0, commit).await?; + + let result = Snapshot::builder_for(table_url).build(&engine); + assert_result_error_with_message( + result.as_ref(), + concat!( + "Schema error: Table schema is too deeply nested: decoding ", + "metaData.schemaString exceeded serde_json's ", + "recursion limit: recursion limit exceeded" + ), + ); + let error = match result.unwrap_err() { + Error::Backtraced { source, .. } => *source, + error => error, + }; + assert!(matches!(error, Error::Schema(_))); + Ok(()) +} + +/// The version-preserving derivations `checkpoint`, `write_checksum`, and `publish` inherit the +/// source snapshot's `built_as_latest`, while a post-commit advance is always latest. +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn built_as_latest_is_inherited_by_derived_snapshots( + #[values(true, false)] base_snap_time_travel_to_latest: bool, + #[values(TableKind::FileSystem, TableKind::CatalogManaged)] kind: TableKind, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + setup_multi_version_table(&engine, &table_path, kind).await?; + + let mut builder = + maybe_attach_max_catalog_version(Snapshot::builder_for(&table_path), LATEST_VERSION, kind) + .with_incremental_crc_replay(IncrementalReplay::Unlimited); + if base_snap_time_travel_to_latest { + builder = builder.at_version(LATEST_VERSION); + } + let base = builder.build(engine.as_ref())?; + assert_eq!(base.version(), LATEST_VERSION); + // A build with no time-travel version is latest; time-travel to the catalog's latest ratified + // version is also considered built as latest. + let base_is_latest = !base_snap_time_travel_to_latest || kind == TableKind::CatalogManaged; + assert_eq!(base.is_built_as_latest(), base_is_latest); + + // checkpoint and write_checksum inherit the base's flag. + let (ckpt_result, after_checkpoint) = base.checkpoint(engine.as_ref(), None)?; + assert_eq!(ckpt_result, CheckpointWriteResult::Written); + assert_eq!(after_checkpoint.is_built_as_latest(), base_is_latest); + + let (crc_result, after_checksum) = after_checkpoint.write_checksum(engine.as_ref())?; + assert_eq!(crc_result, ChecksumWriteResult::Written); + assert_eq!(after_checksum.is_built_as_latest(), base_is_latest); + + // publish() (catalog-managed only) also inherits the base's flag. + if kind == TableKind::CatalogManaged { + let published = base.publish(engine.as_ref(), &TestCatalogCommitter)?; + assert_eq!(published.is_built_as_latest(), base_is_latest); + } + + // A post-commit snapshot is latest. + let post_commit = append_row(base, &engine, kind, 4).await?; + assert_eq!(post_commit.version(), 4); + assert!(post_commit.is_built_as_latest()); + + Ok(()) +} + +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn built_as_latest_on_fresh_and_incremental_build( + #[values(None, Some(1), Some(3))] time_travel_version: Option, + #[values(TableKind::FileSystem, TableKind::CatalogManaged)] kind: TableKind, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + setup_multi_version_table(&engine, &table_path, kind).await?; + + let pins_catalog_latest = + kind == TableKind::CatalogManaged && time_travel_version == Some(LATEST_VERSION); + + let mut builder = + maybe_attach_max_catalog_version(Snapshot::builder_for(&table_path), LATEST_VERSION, kind); + if let Some(v) = time_travel_version { + builder = builder.at_version(v); + } + let base = builder.build(engine.as_ref())?; + assert_eq!( + base.version(), + time_travel_version.unwrap_or(LATEST_VERSION) + ); + // Time travel to the catalog's latest version is also considered built as latest. + let base_is_latest = time_travel_version.is_none() || pins_catalog_latest; + assert_eq!(base.is_built_as_latest(), base_is_latest); + + // Incremental build without an explicit version is latest. + let refreshed = maybe_attach_max_catalog_version( + Snapshot::builder_from(base.clone()), + LATEST_VERSION, + kind, + ) + .build(engine.as_ref())?; + assert_eq!(refreshed.version(), LATEST_VERSION); + assert!(refreshed.is_built_as_latest()); + + // Incremental build time-travelling to LATEST_VERSION. `pinned` is built as latest iff either: + // - the base was already built as latest(and we are traveling to the same version), or + // - the time-travel version is the catalog's latest ratified version. + let pinned = + maybe_attach_max_catalog_version(Snapshot::builder_from(base), LATEST_VERSION, kind) + .at_version(LATEST_VERSION) + .build(engine.as_ref())?; + assert_eq!(pinned.version(), LATEST_VERSION); + assert_eq!( + pinned.is_built_as_latest(), + base_is_latest || kind == TableKind::CatalogManaged + ); + + Ok(()) +} diff --git a/kernel/tests/integration/log/v1_checkpoints.rs b/kernel/tests/integration/log/v1_checkpoints.rs new file mode 100644 index 0000000000..3c463e21b7 --- /dev/null +++ b/kernel/tests/integration/log/v1_checkpoints.rs @@ -0,0 +1,50 @@ +//! Integration tests for V1 checkpoints written via `Snapshot::checkpoint`. + +use std::collections::HashMap; + +use delta_kernel::arrow::array::RecordBatchReader; +use delta_kernel::checkpoint::CheckpointSpec; +use delta_kernel::parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder; +use test_utils::{create_table_and_load_snapshot, test_table_setup_mt, write_batch_to_table}; + +use crate::common::write_utils::{ + get_simple_schema, load_existing_single_file_checkpoint_path, simple_id_batch, +}; + +/// On a V1 table (no `v2Checkpoint` feature), passing either `None` or +/// `Some(CheckpointSpec::V1)` to `snapshot.checkpoint()` produces a V1 checkpoint: the main +/// parquet schema must not contain a `checkpointMetadata` column. +#[rstest::rstest] +#[case::none(None)] +#[case::v1(Some(CheckpointSpec::V1))] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_snapshot_checkpoint_on_v1_table( + #[case] spec: Option, +) -> Result<(), Box> { + let schema = get_simple_schema(); + let (_tmp_dir, table_path, engine) = test_table_setup_mt()?; + let mut snapshot = + create_table_and_load_snapshot(&table_path, schema.clone(), engine.as_ref(), &[])?; + + snapshot = write_batch_to_table( + &snapshot, + engine.as_ref(), + simple_id_batch(&schema, vec![1, 2]), + HashMap::new(), + ) + .await?; + + let version = snapshot.version(); + snapshot.checkpoint(engine.as_ref(), spec.as_ref())?; + + let ckpt_path = load_existing_single_file_checkpoint_path(&table_path, version); + let file = std::fs::File::open(&ckpt_path)?; + let reader = ParquetRecordBatchReaderBuilder::try_new(file)?.build()?; + let schema = reader.schema(); + assert!( + schema.field_with_name("checkpointMetadata").is_err(), + "V1 checkpoint must not contain `checkpointMetadata` column, found schema: {schema:?}" + ); + + Ok(()) +} diff --git a/kernel/tests/integration/log/v2_checkpoints.rs b/kernel/tests/integration/log/v2_checkpoints.rs new file mode 100644 index 0000000000..edc4633da0 --- /dev/null +++ b/kernel/tests/integration/log/v2_checkpoints.rs @@ -0,0 +1,1750 @@ +use std::collections::HashMap; +use std::sync::Arc; + +use delta_kernel::actions::deletion_vector::{DeletionVectorDescriptor, DeletionVectorStorageType}; +use delta_kernel::actions::{MAX_VALUES, MIN_VALUES, NUM_RECORDS}; +use delta_kernel::arrow::array::{ + Array, ArrayRef, AsArray, Int32Array, Int64Array, RecordBatch, RecordBatchReader, StringArray, + StructArray, +}; +use delta_kernel::arrow::datatypes::{ + DataType as ArrowDataType, Field as ArrowField, Schema as ArrowSchema, +}; +use delta_kernel::checkpoint::{CheckpointSpec, V2CheckpointConfig}; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::engine::arrow_conversion::TryFromKernel; +use delta_kernel::expressions::Scalar; +use delta_kernel::parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder; +use delta_kernel::schema::{schema_ref, DataType, StructField, StructType}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::transaction::data_layout::DataLayout; +use delta_kernel::transaction::CommitResult; +use delta_kernel::{DeltaResult, Engine, Snapshot}; +use itertools::Itertools; +use test_utils::delta_kernel_default_engine::executor::TaskExecutor; +use test_utils::{ + begin_transaction, create_add_files_metadata, create_table_and_load_snapshot, insert_data, + load_test_data, read_add_infos, read_scan, test_table_setup_mt, write_batch_to_table, +}; + +use crate::common::read_utils::read_parquet_file; +use crate::common::write_utils::{ + get_simple_schema, load_existing_single_file_checkpoint_path, resolve_struct_field, + simple_id_batch, +}; + +fn read_v2_checkpoint_table(test_name: impl AsRef) -> DeltaResult> { + let test_dir = load_test_data("tests/data", test_name.as_ref()).unwrap(); + let test_path = test_dir.path().join(test_name.as_ref()); + + let url = + delta_kernel::try_parse_uri(test_path.to_str().expect("table path to string")).unwrap(); + let engine = test_utils::create_default_engine(&url)?; + let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); + let scan = snapshot.scan_builder().build()?; + let batches = read_scan(&scan, engine)?; + + Ok(batches) +} + +fn test_v2_checkpoint_with_table( + table_name: &str, + mut expected_table: Vec, +) -> DeltaResult<()> { + let batches = read_v2_checkpoint_table(table_name)?; + + sort_lines!(expected_table); + assert_batches_sorted_eq!(expected_table, &batches); + Ok(()) +} + +/// Helper function to convert string slice vectors to String vectors +fn to_string_vec(string_slice_vec: Vec<&str>) -> Vec { + string_slice_vec + .into_iter() + .map(|s| s.to_string()) + .collect() +} + +fn generate_sidecar_expected_data() -> Vec { + let header = vec![ + "+-----+".to_string(), + "| id |".to_string(), + "+-----+".to_string(), + ]; + + // Generate rows for different ranges + let generate_rows = |count: usize| -> Vec { + (0..count) + .map(|id| format!("| {: Vec { + to_string_vec(vec![ + "+----+", + "| id |", + "+----+", + "| 0 |", + "| 1 |", + "| 2 |", + "| 3 |", + "| 4 |", + "| 5 |", + "| 6 |", + "| 7 |", + "| 8 |", + "| 9 |", + "+----+", + ]) +} + +// Rustfmt is disabled to maintain the readability of the expected table +#[rustfmt::skip] +fn get_classic_checkpoint_table() -> Vec { + to_string_vec(vec![ + "+----+", + "| id |", + "+----+", + "| 0 |", + "| 1 |", + "| 2 |", + "| 3 |", + "| 4 |", + "| 5 |", + "| 6 |", + "| 7 |", + "| 8 |", + "| 9 |", + "| 10 |", + "| 11 |", + "| 12 |", + "| 13 |", + "| 14 |", + "| 15 |", + "| 16 |", + "| 17 |", + "| 18 |", + "| 19 |", + "+----+", + ]) +} + +// Rustfmt is disabled to maintain the readability of the expected table +#[rustfmt::skip] +fn get_without_sidecars_table() -> Vec { + to_string_vec(vec![ + "+------+", + "| id |", + "+------+", + "| 0 |", + "| 1 |", + "| 2 |", + "| 3 |", + "| 4 |", + "| 5 |", + "| 6 |", + "| 7 |", + "| 8 |", + "| 9 |", + "| 2718 |", + "+------+", + ]) +} + +/// The test cases below are derived from delta-spark's `CheckpointSuite`. +/// +/// These tests are converted from delta-spark using the following process: +/// 1. Specific test cases of interest in `delta-spark` were modified to persist their generated +/// tables +/// 2. These tables were compressed into `.tar.zst` archives and copied to delta-kernel-rs +/// 3. Each test loads a stored table, scans it, and asserts that the returned table state matches +/// the expected state derived from the corresponding table insertions in `delta-spark` +/// +/// The following is the ported list of `delta-spark` tests -> `delta-kernel-rs` tests: +/// +/// - `multipart v2 checkpoint` -> `v2_checkpoints_json_with_sidecars` +/// - `multipart v2 checkpoint` -> `v2_checkpoints_parquet_with_sidecars` +/// - `All actions in V2 manifest` -> `v2_checkpoints_json_without_sidecars` +/// - `All actions in V2 manifest` -> `v2_checkpoints_parquet_without_sidecars` +/// - `V2 Checkpoint compat file equivalency to normal V2 Checkpoint` -> +/// `v2_classic_checkpoint_json` +/// - `V2 Checkpoint compat file equivalency to normal V2 Checkpoint` -> +/// `v2_classic_checkpoint_parquet` +/// - `last checkpoint contains correct schema for v1/v2 Checkpoints` -> +/// `v2_checkpoints_json_with_last_checkpoint` +/// - `last checkpoint contains correct schema for v1/v2 Checkpoints` -> +/// `v2_checkpoints_parquet_with_last_checkpoint` +#[test] +fn v2_checkpoints_json_with_sidecars() -> DeltaResult<()> { + test_v2_checkpoint_with_table( + "v2-checkpoints-json-with-sidecars", + generate_sidecar_expected_data(), + ) +} + +#[test] +fn v2_checkpoints_parquet_with_sidecars() -> DeltaResult<()> { + test_v2_checkpoint_with_table( + "v2-checkpoints-parquet-with-sidecars", + generate_sidecar_expected_data(), + ) +} + +#[test] +fn v2_checkpoints_json_without_sidecars() -> DeltaResult<()> { + test_v2_checkpoint_with_table( + "v2-checkpoints-json-without-sidecars", + get_without_sidecars_table(), + ) +} + +#[test] +fn v2_checkpoints_parquet_without_sidecars() -> DeltaResult<()> { + test_v2_checkpoint_with_table( + "v2-checkpoints-parquet-without-sidecars", + get_without_sidecars_table(), + ) +} + +#[test] +fn v2_classic_checkpoint_json() -> DeltaResult<()> { + test_v2_checkpoint_with_table("v2-classic-checkpoint-json", get_classic_checkpoint_table()) +} + +#[test] +fn v2_classic_checkpoint_parquet() -> DeltaResult<()> { + test_v2_checkpoint_with_table( + "v2-classic-checkpoint-parquet", + get_classic_checkpoint_table(), + ) +} + +#[test] +fn v2_checkpoints_json_with_last_checkpoint() -> DeltaResult<()> { + test_v2_checkpoint_with_table( + "v2-checkpoints-json-with-last-checkpoint", + get_simple_id_table(), + ) +} + +#[test] +fn v2_checkpoints_parquet_with_last_checkpoint() -> DeltaResult<()> { + test_v2_checkpoint_with_table( + "v2-checkpoints-parquet-with-last-checkpoint", + get_simple_id_table(), + ) +} + +/// Tests that writing a V2 checkpoint to parquet succeeds. +/// +/// V2 checkpoints include a checkpointMetadata batch in addition to the regular action +/// batches. All batches in a parquet file must share the same schema. This test verifies +/// that `snapshot.checkpoint()` can write a V2 checkpoint without schema mismatch errors. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_v2_checkpoint_parquet_write() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + + let schema = schema_ref! { nullable "value": INTEGER }; + let _ = create_table(&table_path, schema.clone(), "Test/1.0") + .with_table_properties([("delta.feature.v2Checkpoint", "supported")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + // Commit an add action via the transaction API so the checkpoint has action batches + let snapshot0 = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let result = insert_data( + snapshot0, + &engine, + vec![Arc::new(Int32Array::from(vec![1]))], + ) + .await?; + + let CommitResult::CommittedTransaction(committed) = result else { + panic!("Expected CommittedTransaction"); + }; + + let snapshot = committed + .post_commit_snapshot() + .expect("expected post-commit snapshot"); + + // This writes to parquet — will fail if the checkpointMetadata batch has a different + // schema than the action batches. + snapshot.checkpoint(engine.as_ref(), None)?; + + // Verify the checkpoint was written and is used by a fresh snapshot + let snapshot2 = Snapshot::builder_for(table_url).build(engine.as_ref())?; + assert_eq!(snapshot2.version(), 1); + let log_segment = snapshot2.log_segment(); + assert!( + !log_segment.listed.checkpoint_parts.is_empty(), + "expected snapshot to use the written checkpoint, but checkpoint_parts is empty" + ); + assert_eq!( + log_segment.checkpoint_version, + Some(1), + "expected checkpoint at version 1" + ); + assert!( + log_segment.listed.ascending_commit_files.is_empty(), + "expected no commit files after checkpoint, but found: {:?}", + log_segment.listed.ascending_commit_files + ); + + // Verify reading data from the checkpointed snapshot returns the expected rows + let scan = snapshot2.scan_builder().build()?; + let batches = read_scan(&scan, engine.clone() as Arc)?; + assert_batches_sorted_eq!( + vec![ + "+-------+", + "| value |", + "+-------+", + "| 1 |", + "+-------+", + ], + &batches + ); + + Ok(()) +} + +/// E2E test for V2 sidecar checkpoint write and read. Builds a V2 table with mixed types of +/// delta actions, writes a sidecar checkpoint, then adds more commits on top. Verifies +/// `_last_checkpoint`, the checkpoint parquet contents, and performs a scan to verify the data +/// is correct. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_v2_checkpoint_with_sidecars() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + + // === Step 1: Create V2 table with adds, removes, domain metadata, and set-transactions === + let snapshot = v2_table_with_domain_metadata_and_txn(&table_path, &table_url, &engine).await?; + let version = snapshot.version() as i64; + + // === Step 2: Write V2 checkpoint with sidecars (hint=2 splits actions across sidecars) === + let checkpoint_spec = CheckpointSpec::V2(V2CheckpointConfig::WithSidecar { + file_actions_per_sidecar_hint: Some(2), + }); + snapshot.checkpoint(engine.as_ref(), Some(&checkpoint_spec))?; + + // === Step 3: Add post-checkpoint commits (insert + domain metadata update) === + let info_field = Arc::new(ArrowField::new("name", ArrowDataType::Utf8, true)); + let info_array: ArrayRef = Arc::new(StructArray::from(vec![( + info_field, + Arc::new(StringArray::from(vec![Some("quinn")])) as ArrayRef, + )])); + let post_ckpt_snapshot = insert_data( + Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?, + &engine, + vec![Arc::new(Int32Array::from(vec![17])) as ArrayRef, info_array], + ) + .await? + .unwrap_post_commit_snapshot(); + + let post_ckpt_snapshot = begin_transaction(post_ckpt_snapshot, engine.as_ref())? + .with_domain_metadata("app.settings".to_string(), r#"{"version":3}"#.to_string()) + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + + // === Step 4: Validate `_last_checkpoint` (version, size, sizeInBytes, numOfAddFiles) === + let last_ckpt = read_last_checkpoint(&table_path); + let ckpt_file = load_existing_single_file_checkpoint_path(&table_path, version as _); + let ckpt_file_size = std::fs::metadata(&ckpt_file).unwrap().len() as i64; + let sidecars_dir_for_size = std::path::Path::new(&table_path).join("_delta_log/_sidecars"); + let sidecars_total_size: i64 = list_sidecar_parquet_files(&sidecars_dir_for_size) + .iter() + .map(|e| e.metadata().unwrap().len() as i64) + .sum(); + + assert_eq!(last_ckpt["version"], version); + // size = 1 protocol + 1 metadata + 8 adds + 8 removes + 3 domain metadata + // + 2 set-transactions (app1 reconciled to the latest) + 1 checkpointMetadata + // + 5 sidecar references = 29 + assert_eq!(last_ckpt["size"].as_i64().unwrap(), 29); + assert_eq!( + last_ckpt["sizeInBytes"].as_i64().unwrap(), + ckpt_file_size + sidecars_total_size + ); + assert_eq!(last_ckpt["numOfAddFiles"], 8); + + // === Step 5: Read raw checkpoint parquet and validate fields === + // - Main checkpoint has sidecar, protocol, metadata, checkpointMetadata, domainMetadata, txn + // - Main checkpoint has NO add/remove actions (they live in sidecars) + // - 5 sidecar references match the actual sidecar files on disk + // - Sidecar files contain only add/remove actions + // - Per-sidecar distribution: 1 sidecar with 8 removes, 4 sidecars with 2 adds each + // - checkpointMetadata version matches checkpoint version + // - 3 reconciled domain metadata actions + // - 2 reconciled txn actions (`app1@3`, `app2@5`) + let ckpt_batch = read_parquet_file(&ckpt_file); + let ckpt_schema = ckpt_batch.schema(); + + // Main checkpoint must contain non-file action columns plus a `sidecar` column. + for field_name in [ + "protocol", + "metaData", + "checkpointMetadata", + "domainMetadata", + "sidecar", + "txn", + ] { + assert!( + ckpt_schema.field_with_name(field_name).is_ok(), + "checkpoint should have {field_name}" + ); + } + + // Validate sidecar actions in the main checkpoint + let sidecar_col = get_struct_column_from_record_batch(&ckpt_batch, "sidecar"); + let sidecar_rows = valid_row_indices(sidecar_col, ckpt_batch.num_rows()); + // 1 sidecar for the bulk-remove batch (8 removes) + 4 sidecars for adds (2 each) + assert_eq!( + sidecar_rows.len(), + 5, + "checkpoint should contain 5 sidecar action rows" + ); + + assert_sidecar_actions_match_disk(sidecar_col, &sidecar_rows, &table_path, engine.as_ref()); + + // Main checkpoint must NOT contain add/remove actions (they live in sidecars) + for action in ["add", "remove"] { + let col = get_struct_column_from_record_batch(&ckpt_batch, action); + let rows = valid_row_indices(col, ckpt_batch.num_rows()); + assert!( + rows.is_empty(), + "main checkpoint should not contain {action} actions when sidecars are used, \ + but found {} rows", + rows.len() + ); + } + + // Validate sidecar files contain ONLY add/remove file actions + let sidecars_dir = std::path::Path::new(&table_path).join("_delta_log/_sidecars"); + let per_sidecar = assert_sidecars_contain_only_file_actions(&sidecars_dir); + // 1 sidecar has 8 removes (bulk delete can't be split), 4 sidecars have 2 adds each + assert_eq!( + per_sidecar, + vec![(0, 8), (2, 0), (2, 0), (2, 0), (2, 0)], + "per-sidecar (adds, removes) distribution" + ); + + // Validate checkpointMetadata action + let ckpt_meta_col = get_struct_column_from_record_batch(&ckpt_batch, "checkpointMetadata"); + let ckpt_meta_rows = valid_row_indices(ckpt_meta_col, ckpt_batch.num_rows()); + assert_eq!( + ckpt_meta_rows.len(), + 1, + "should have exactly one checkpointMetadata action" + ); + let version_col = ckpt_meta_col + .column_by_name("version") + .expect("checkpointMetadata should have version field"); + assert_eq!( + version_col + .as_primitive::() + .value(ckpt_meta_rows[0]), + version, + "checkpointMetadata version should match checkpoint version" + ); + + // Validate domainMetadata actions: exactly 3 (app.settings, app.feature_flags, app.analytics) + let dm_col = get_struct_column_from_record_batch(&ckpt_batch, "domainMetadata"); + let dm_rows = valid_row_indices(dm_col, ckpt_batch.num_rows()); + assert_eq!( + dm_rows.len(), + 3, + "checkpoint should contain exactly 3 domainMetadata actions" + ); + + // Validate txn actions: exactly 2 after reconciliation (app1@3 and app2@5). + let txn_col = get_struct_column_from_record_batch(&ckpt_batch, "txn"); + let txn_rows = valid_row_indices(txn_col, ckpt_batch.num_rows()); + assert_eq!( + txn_rows.len(), + 2, + "checkpoint should contain exactly 2 txn actions after reconciliation" + ); + let txn_app_id_col = txn_col + .column_by_name("appId") + .expect("txn should have appId field") + .as_string::(); + let txn_version_col = txn_col + .column_by_name("version") + .expect("txn should have version field") + .as_primitive::(); + let mut txn_pairs: Vec<(String, i64)> = txn_rows + .iter() + .map(|&r| { + ( + txn_app_id_col.value(r).to_string(), + txn_version_col.value(r), + ) + }) + .collect(); + txn_pairs.sort(); + assert_eq!( + txn_pairs, + vec![("app1".to_string(), 3), ("app2".to_string(), 5)], + "txn actions should reflect reconciliation (app1 kept at latest version)" + ); + + // === Step 6: Load fresh snapshot from checkpoint + post-checkpoint commits === + let fresh_snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + assert_eq!(fresh_snapshot.version(), post_ckpt_snapshot.version()); + // Assert that we are reading from the checkpoint + post-checkpoint commits + let log_segment = fresh_snapshot.log_segment(); + assert!( + !log_segment.listed.checkpoint_parts.is_empty(), + "fresh snapshot should load from checkpoint" + ); + assert_eq!(log_segment.checkpoint_version, Some(version as u64)); + assert_eq!( + log_segment.listed.ascending_commit_files.len(), + 2, + "expected 2 commit files after checkpoint (insert + domain metadata update)" + ); + + // === Step 7: Verify domain metadata (including post-checkpoint update) === + assert_eq!( + fresh_snapshot.get_domain_metadata("app.settings", engine.as_ref())?, + Some(r#"{"version":3}"#.to_string()), + "app.settings domain metadata should reflect the post-checkpoint update" + ); + assert_eq!( + fresh_snapshot.get_domain_metadata("app.feature_flags", engine.as_ref())?, + Some(r#"{"dark_mode":true}"#.to_string()), + "app.feature_flags domain metadata should be preserved across checkpoint" + ); + assert_eq!( + fresh_snapshot.get_domain_metadata("app.analytics", engine.as_ref())?, + Some(r#"{"tracking":false}"#.to_string()), + "app.analytics domain metadata should be preserved across checkpoint" + ); + + // === Step 8: Scan and verify data correctness === + let scan = fresh_snapshot.scan_builder().build()?; + let batches = read_scan(&scan, engine.clone() as Arc)?; + // Note: The data is sorted in lexicographical order. So id 9 appears at last. + assert_batches_sorted_eq!( + vec![ + "+----+---------------+", + "| id | info |", + "+----+---------------+", + "| 10 | {name: judy} |", + "| 11 | {name: karl} |", + "| 12 | {name: lena} |", + "| 13 | {name: mike} |", + "| 14 | {name: nina} |", + "| 15 | {name: omar} |", + "| 16 | {name: pat} |", + "| 17 | {name: quinn} |", + "| 9 | {name: ivan} |", + "+----+---------------+", + ], + &batches + ); + + Ok(()) +} + +/// E2E test: create a partitioned table with stats, write several commits, checkpoint, +/// then read the raw checkpoint parquet to verify `stats_parsed`, `stats`, and +/// `partitionValues_parsed` fields are correctly populated. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_v2_checkpoint_partition_values_parsed_and_stats( +) -> Result<(), Box> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + + let snapshot2 = create_partitioned_stats_table(&table_path, &table_url, &engine).await?; + + // Write V2 checkpoint with sidecars at version 2 + let checkpoint_spec = CheckpointSpec::V2(V2CheckpointConfig::WithSidecar { + file_actions_per_sidecar_hint: Some(1), + }); + snapshot2.checkpoint(engine.as_ref(), Some(&checkpoint_spec))?; + + // === Validate sidecar structure === + let sidecars_dir = std::path::Path::new(&table_path).join("_delta_log/_sidecars"); + let per_sidecar = assert_sidecars_contain_only_file_actions(&sidecars_dir); + // 2 adds with hint=1 -> 2 sidecars with 1 add each + assert_eq!( + per_sidecar, + vec![(1, 0), (1, 0)], + "per-sidecar (adds, removes) distribution" + ); + + // === Read sidecar files and validate stats_parsed, stats JSON, and partitionValues_parsed === + let sidecar_files = list_sidecar_parquet_files(&sidecars_dir); + + let mut all_record_counts = Vec::new(); + let mut all_part_values = Vec::new(); + let mut all_min_ids = Vec::new(); + let mut all_max_ids = Vec::new(); + let mut all_stats_json = Vec::new(); + + for sidecar_entry in &sidecar_files { + let sidecar_batch = read_parquet_file(&sidecar_entry.path()); + let add_col = get_struct_column_from_record_batch(&sidecar_batch, "add"); + let add_rows = valid_row_indices(add_col, sidecar_batch.num_rows()); + if add_rows.is_empty() { + continue; + } + + // `add.stats` is a raw JSON string. + let stats_col = add_col + .column_by_name("stats") + .expect("add should have stats"); + for &row in &add_rows { + assert!(stats_col.is_valid(row), "add.stats should be non-null"); + all_stats_json.push(stats_col.as_string::().value(row).to_string()); + } + + let stats_parsed = get_struct_column_from_struct_array(add_col, "stats_parsed"); + + let num_records_col = stats_parsed + .column_by_name(NUM_RECORDS) + .expect("stats_parsed should have numRecords"); + for &row in &add_rows { + all_record_counts.push( + num_records_col + .as_primitive::() + .value(row), + ); + } + + let min_values = get_struct_column_from_struct_array(stats_parsed, MIN_VALUES); + let min_id_col = min_values + .column_by_name("id") + .expect("minValues should have id"); + for &row in &add_rows { + all_min_ids.push( + min_id_col + .as_primitive::() + .value(row), + ); + } + + let max_values = get_struct_column_from_struct_array(stats_parsed, MAX_VALUES); + let max_id_col = max_values + .column_by_name("id") + .expect("maxValues should have id"); + for &row in &add_rows { + all_max_ids.push( + max_id_col + .as_primitive::() + .value(row), + ); + } + + let pv_parsed = get_struct_column_from_struct_array(add_col, "partitionValues_parsed"); + let part_key_col = pv_parsed + .column_by_name("part_key") + .expect("partitionValues_parsed should have part_key field"); + for &row in &add_rows { + all_part_values.push(part_key_col.as_string::().value(row).to_string()); + } + } + + all_record_counts.sort(); + assert_eq!( + all_record_counts, + vec![2, 3], + "stats_parsed.numRecords should be [2, 3] (one per partition)" + ); + + all_min_ids.sort(); + assert_eq!( + all_min_ids, + vec![1, 3], + "stats_parsed.minValues.id should be [1, 3] (min id per partition)" + ); + + all_max_ids.sort(); + assert_eq!( + all_max_ids, + vec![2, 5], + "stats_parsed.maxValues.id should be [2, 5] (max id per partition)" + ); + + all_part_values.sort(); + assert_eq!( + all_part_values, + vec!["a", "b"], + "partitionValues_parsed.part_key should be ['a', 'b']" + ); + + // Validate stats JSON. + let mut parsed_stats: Vec = all_stats_json + .iter() + .map(|s| serde_json::from_str(s).expect("add.stats should be valid JSON")) + .collect(); + parsed_stats.sort_by_key(|v| v[NUM_RECORDS].as_i64().unwrap()); + assert_eq!( + parsed_stats, + vec![ + serde_json::json!({ + "numRecords": 2, + "minValues": { "id": 1, "name": "alice" }, + "maxValues": { "id": 2, "name": "bob" }, + "nullCount": { "id": 0, "name": 0 }, + "tightBounds": true, + }), + serde_json::json!({ + "numRecords": 3, + "minValues": { "id": 3, "name": "charlie" }, + "maxValues": { "id": 5, "name": "eve" }, + "nullCount": { "id": 0, "name": 0 }, + "tightBounds": true, + }), + ], + "add.stats JSON should match the expected stats" + ); + + // === Verify scan reads all data correctly after sidecar checkpoint === + let snapshot3 = Snapshot::builder_for(table_url).build(engine.as_ref())?; + let scan = snapshot3.scan_builder().build()?; + let batches = read_scan(&scan, engine.clone() as Arc)?; + assert_batches_sorted_eq!( + vec![ + "+----+---------+----------+", + "| id | name | part_key |", + "+----+---------+----------+", + "| 1 | alice | a |", + "| 2 | bob | a |", + "| 3 | charlie | b |", + "| 4 | dave | b |", + "| 5 | eve | b |", + "+----+---------+----------+", + ], + &batches + ); + + Ok(()) +} + +/// Parameterized negative tests for writing v2 checkpoint with sidecars. +#[rstest::rstest] +#[case::v2_spec_requires_v2checkpoint_feature( + false, + CheckpointSpec::V2(V2CheckpointConfig::NoSidecar), + "v2Checkpoint" +)] +#[case::v1_rejected_on_v2_table(true, CheckpointSpec::V1, "V1")] +#[case::sidecar_hint_zero_rejected( + true, + CheckpointSpec::V2(V2CheckpointConfig::WithSidecar { + file_actions_per_sidecar_hint: Some(0), + }), + "greater than 0" +)] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_checkpoint_spec_rejected( + #[case] enable_v2checkpoint: bool, + #[case] spec: CheckpointSpec, + #[case] err_substring: &str, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + + let schema = schema_ref! { nullable "value": INTEGER }; + + let mut builder = create_table(&table_path, schema, "Test/1.0"); + if enable_v2checkpoint { + builder = builder.with_table_properties([("delta.feature.v2Checkpoint", "supported")]); + } + let _ = builder + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + + let result = snapshot.checkpoint(engine.as_ref(), Some(&spec)); + let err_msg = result.expect_err("spec should be rejected").to_string(); + assert!( + err_msg.contains(err_substring), + "error should mention {err_substring:?}, got: {err_msg}" + ); + + Ok(()) +} + +/// V2 checkpoint with sidecar on a table that has no file actions: the loop writes zero +/// sidecar files, the main checkpoint contains no `sidecar` action rows, and `_last_checkpoint` +/// reports `numOfAddFiles = 0` and `sizeInBytes` equal to the main checkpoint file size. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_v2_sidecar_checkpoint_with_no_file_actions() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + + let schema = schema_ref! { nullable "value": INTEGER }; + + // v2 table, no data commits -> only protocol + metadata at version 0. + let _ = create_table(&table_path, schema, "Test/1.0") + .with_table_properties([("delta.feature.v2Checkpoint", "supported")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + let version = snapshot.version(); + + let spec = CheckpointSpec::V2(V2CheckpointConfig::WithSidecar { + file_actions_per_sidecar_hint: Some(2), + }); + snapshot.checkpoint(engine.as_ref(), Some(&spec))?; + + // No sidecar files on disk: when there are no file actions, our writer should not even + // create the `_sidecars` directory. + let sidecars_dir = std::path::Path::new(&table_path).join("_delta_log/_sidecars"); + assert!( + !sidecars_dir.exists(), + "_sidecars directory should not exist when there are no file actions, found: {}", + sidecars_dir.display() + ); + + // Main checkpoint contains no `sidecar` action rows. + let ckpt_file = load_existing_single_file_checkpoint_path(&table_path, version); + let ckpt_batch = read_parquet_file(&ckpt_file); + let sidecar_col = get_struct_column_from_record_batch(&ckpt_batch, "sidecar"); + let sidecar_rows = valid_row_indices(sidecar_col, ckpt_batch.num_rows()); + assert!( + sidecar_rows.is_empty(), + "main checkpoint should contain no sidecar action rows, found {}", + sidecar_rows.len() + ); + + // `_last_checkpoint` reflects zero adds and a single-file size. + let last_ckpt = read_last_checkpoint(&table_path); + assert_eq!(last_ckpt["numOfAddFiles"], 0); + let ckpt_file_size = std::fs::metadata(&ckpt_file).unwrap().len() as i64; + assert_eq!( + last_ckpt["sizeInBytes"].as_i64().unwrap(), + ckpt_file_size, + "sizeInBytes should equal main checkpoint size when there are no sidecars" + ); + + Ok(()) +} + +/// Reads the `_last_checkpoint` JSON file from the table's `_delta_log` directory. +fn read_last_checkpoint(table_path: &str) -> serde_json::Value { + let path = std::path::Path::new(table_path).join("_delta_log/_last_checkpoint"); + let content = std::fs::read_to_string(&path).expect("failed to read _last_checkpoint"); + serde_json::from_str(&content).expect("failed to parse _last_checkpoint JSON") +} + +/// Creates a v2Checkpoint + domainMetadata table with a nested schema (`id: int`, +/// `info: struct { name: string }`). Inserts 8 files across several commits, removes all of +/// them, adds domain metadata (with a update for same domain), commits `txn` set-transactions +/// (with a reconciliation case for `app1`), then inserts 8 fresh files. Returns the final +/// snapshot. The table should have 8 live adds, 8 remove tombstones, 3 domain metadata +/// entries, and 2 reconciled `txn` actions (`app1@3`, `app2@5`). +/// +/// Exercises every action in the V2 checkpoint schema except `sidecar`/`checkpointMetadata`, +/// which are produced by the checkpoint writer itself. +async fn v2_table_with_domain_metadata_and_txn( + table_path: &str, + table_url: &url::Url, + engine: &Arc>, +) -> DeltaResult> { + fn make_info_array(names: &[&str]) -> ArrayRef { + let name_array: ArrayRef = Arc::new(StringArray::from( + names.iter().map(|s| Some(*s)).collect::>(), + )); + let field = Arc::new(ArrowField::new("name", ArrowDataType::Utf8, true)); + Arc::new(StructArray::from(vec![(field, name_array)])) + } + + fn make_columns(id: i32, name: &str) -> Vec { + vec![ + Arc::new(Int32Array::from(vec![id])) as ArrayRef, + make_info_array(&[name]), + ] + } + + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable( + "info", + DataType::try_struct_type([StructField::nullable("name", DataType::STRING)])?, + ), + ])?); + + let _ = create_table(table_path, schema.clone(), "Test/1.0") + .with_table_properties([ + ("delta.feature.v2Checkpoint", "supported"), + ("delta.feature.domainMetadata", "supported"), + ]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + // Insert 8 files (one per commit) -> ids 1..=8 + let mut snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let names = [ + "alice", "bob", "carol", "dave", "eve", "frank", "grace", "heidi", + ]; + for (i, name) in names.iter().enumerate() { + snapshot = insert_data(snapshot, engine, make_columns(i as i32 + 1, name)) + .await? + .unwrap_post_commit_snapshot(); + } + + // Domain metadata commit (no data) -- exercises the empty-file-batch skip path in the + // sidecar splitter. Sets two domains initially. + snapshot = begin_transaction(snapshot, engine.as_ref())? + .with_domain_metadata("app.settings".to_string(), r#"{"version":1}"#.to_string()) + .with_domain_metadata( + "app.feature_flags".to_string(), + r#"{"dark_mode":true}"#.to_string(), + ) + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + + // Another domain metadata commit -- updates "app.settings" to verify reconciliation + // picks the latest value, and adds a new domain. + snapshot = begin_transaction(snapshot, engine.as_ref())? + .with_domain_metadata( + "app.analytics".to_string(), + r#"{"tracking":false}"#.to_string(), + ) + .with_domain_metadata("app.settings".to_string(), r#"{"version":2}"#.to_string()) + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + + // SetTransaction commits -- exercise `txn` actions in checkpoint. Two distinct app_ids + // plus a second update to `app1` to verify reconciliation picks the latest version. + for (app_id, version) in [("app1", 1i64), ("app2", 5), ("app1", 3)] { + snapshot = begin_transaction(snapshot, engine.as_ref())? + .with_transaction_id(app_id.to_string(), version) + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + } + + // Remove all 8 files -> 8 remove tombstones + let scan = snapshot.clone().scan_builder().build()?; + let mut txn = begin_transaction(snapshot, engine.as_ref())? + .with_operation("DELETE".to_string()) + .with_data_change(true); + for sm in scan.scan_metadata(engine.as_ref())? { + txn.remove_files(sm?.scan_files); + } + snapshot = txn.commit(engine.as_ref())?.unwrap_post_commit_snapshot(); + + // Insert 8 fresh files (one per commit) -> ids 9..=16 + let names = [ + "ivan", "judy", "karl", "lena", "mike", "nina", "omar", "pat", + ]; + for (i, name) in names.iter().enumerate() { + snapshot = insert_data(snapshot, engine, make_columns(i as i32 + 9, name)) + .await? + .unwrap_post_commit_snapshot(); + } + + Ok(snapshot) +} + +/// Extracts a named struct column from a `RecordBatch`, panicking if missing or wrong type. +fn get_struct_column_from_record_batch<'a>(batch: &'a RecordBatch, name: &str) -> &'a StructArray { + batch + .column_by_name(name) + .unwrap_or_else(|| panic!("batch should have column '{name}'")) + .as_any() + .downcast_ref::() + .unwrap_or_else(|| panic!("column '{name}' should be a StructArray")) +} + +/// Returns indices of non-null (valid) rows for a column in a batch. +fn valid_row_indices(col: &dyn Array, num_rows: usize) -> Vec { + (0..num_rows).filter(|&i| col.is_valid(i)).collect() +} + +/// Extracts a named struct sub-column from a `StructArray`, panicking if missing or wrong type. +fn get_struct_column_from_struct_array<'a>(parent: &'a StructArray, name: &str) -> &'a StructArray { + parent + .column_by_name(name) + .unwrap_or_else(|| panic!("struct should have field '{name}'")) + .as_any() + .downcast_ref::() + .unwrap_or_else(|| panic!("field '{name}' should be a StructArray")) +} + +/// Lists all sidecar parquet files in the `_sidecars` directory. +fn list_sidecar_parquet_files(sidecars_dir: &std::path::Path) -> Vec { + std::fs::read_dir(sidecars_dir) + .expect("failed to list sidecars dir") + .filter_map(|e| e.ok()) + .filter(|e| e.file_name().to_string_lossy().ends_with(".parquet")) + .collect() +} + +/// Validates that sidecar action rows in the main checkpoint have correct path, sizeInBytes, +/// and modificationTime matching actual files on disk. +fn assert_sidecar_actions_match_disk( + sidecar_col: &StructArray, + sidecar_rows: &[usize], + table_path: &str, + engine: &dyn Engine, +) { + let sidecar_path_col = sidecar_col + .column_by_name("path") + .expect("sidecar should have path field"); + let sidecar_size_col = sidecar_col + .column_by_name("sizeInBytes") + .expect("sidecar should have sizeInBytes field"); + let sidecar_mtime_col = sidecar_col + .column_by_name("modificationTime") + .expect("sidecar should have modificationTime field"); + let sidecar_tags_col = sidecar_col + .column_by_name("tags") + .expect("sidecar should have tags field"); + let sidecars_base = delta_kernel::try_parse_uri(table_path) + .unwrap() + .join("_delta_log/_sidecars/") + .unwrap(); + + for &row in sidecar_rows { + let path = sidecar_path_col.as_string::().value(row); + assert!( + path.ends_with(".parquet"), + "sidecar path should be a parquet file, got: {path}" + ); + + let sidecar_url = sidecars_base.join(path).unwrap(); + let file_meta = engine + .storage_handler() + .head(&sidecar_url) + .unwrap_or_else(|e| panic!("sidecar file {path} should exist: {e}")); + + let recorded_size = sidecar_size_col + .as_primitive::() + .value(row); + assert_eq!( + recorded_size, file_meta.size as i64, + "sidecar sizeInBytes should match actual file size for {path}" + ); + + let recorded_mtime = sidecar_mtime_col + .as_primitive::() + .value(row); + assert_eq!( + recorded_mtime, file_meta.last_modified, + "sidecar modificationTime should match actual file mtime for {path}" + ); + + assert!( + sidecar_tags_col.is_null(row), + "sidecar tags should be null for {path}" + ); + } +} + +/// Reads all sidecar parquet files from the `_sidecars` directory. Asserts each sidecar +/// contains only add/remove file actions (no non-file actions). Returns a sorted list of +/// `(add_count, remove_count)` per sidecar. +fn assert_sidecars_contain_only_file_actions( + sidecars_dir: &std::path::Path, +) -> Vec<(usize, usize)> { + let sidecar_files = list_sidecar_parquet_files(sidecars_dir); + assert!( + !sidecar_files.is_empty(), + "should have at least one sidecar parquet file in _delta_log/_sidecars/" + ); + + let mut per_sidecar_counts: Vec<(usize, usize)> = Vec::new(); + for sidecar_entry in &sidecar_files { + let sidecar_batch = read_parquet_file(&sidecar_entry.path()); + let sidecar_schema = sidecar_batch.schema(); + + // Sidecar must have add and/or remove columns + assert!( + sidecar_schema.field_with_name("add").is_ok() + || sidecar_schema.field_with_name("remove").is_ok(), + "sidecar file should contain add and/or remove columns" + ); + + // Sidecar must NOT contain non-file actions + for forbidden in &[ + "protocol", + "metaData", + "checkpointMetadata", + "domainMetadata", + "txn", + "sidecar", + ] { + if let Ok(field) = sidecar_schema.field_with_name(forbidden) { + let col = sidecar_batch.column_by_name(forbidden).unwrap(); + let non_null = valid_row_indices(col, sidecar_batch.num_rows()).len(); + assert_eq!( + non_null, 0, + "sidecar should not contain non-null {forbidden} actions, \ + but found {non_null} (field type: {field:?})" + ); + } + } + + let adds = sidecar_batch + .column_by_name("add") + .map(|c| valid_row_indices(c, sidecar_batch.num_rows()).len()) + .unwrap_or(0); + let removes = sidecar_batch + .column_by_name("remove") + .map(|c| valid_row_indices(c, sidecar_batch.num_rows()).len()) + .unwrap_or(0); + per_sidecar_counts.push((adds, removes)); + } + per_sidecar_counts.sort(); + per_sidecar_counts +} + +/// Creates a partitioned table with stats (`id: long`, `name: string`, partition `part_key`), +/// writes two commits (partition "a" with 2 rows, partition "b" with 3 rows), and returns +/// the snapshot after both commits. +async fn create_partitioned_stats_table( + table_path: &str, + table_url: &url::Url, + engine: &Arc>, +) -> Result, Box> { + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::LONG), + StructField::nullable("name", DataType::STRING), + StructField::nullable("part_key", DataType::STRING), + ])?); + + let _ = create_table(table_path, schema.clone(), "Test/1.0") + .with_table_properties([ + ("delta.feature.v2Checkpoint", "supported"), + ("delta.checkpoint.writeStatsAsStruct", "true"), + ]) + .with_data_layout(DataLayout::partitioned(["part_key"])) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let data_schema = StructType::try_new(vec![ + StructField::nullable("id", DataType::LONG), + StructField::nullable("name", DataType::STRING), + ])?; + let arrow_schema = Arc::new(ArrowSchema::try_from_kernel(&data_schema)?); + + let batch1 = RecordBatch::try_new( + arrow_schema.clone(), + vec![ + Arc::new(Int64Array::from(vec![1, 2])) as ArrayRef, + Arc::new(StringArray::from(vec![Some("alice"), Some("bob")])) as ArrayRef, + ], + ) + .unwrap(); + let snapshot0 = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let snapshot1 = write_batch_to_table( + &snapshot0, + engine.as_ref(), + batch1, + HashMap::from([("part_key".to_string(), Scalar::from("a"))]), + ) + .await?; + + let batch2 = RecordBatch::try_new( + arrow_schema, + vec![ + Arc::new(Int64Array::from(vec![3, 4, 5])) as ArrayRef, + Arc::new(StringArray::from(vec![ + Some("charlie"), + Some("dave"), + Some("eve"), + ])) as ArrayRef, + ], + ) + .unwrap(); + let snapshot2 = write_batch_to_table( + &snapshot1, + engine.as_ref(), + batch2, + HashMap::from([("part_key".to_string(), Scalar::from("b"))]), + ) + .await?; + + Ok(snapshot2) +} + +/// On a V2 table (table with `v2Checkpoint` feature), passing either `None` or +/// `Some(CheckpointSpec::V2(V2CheckpointConfig::NoSidecar))` to `snapshot.checkpoint()` produces +/// a V2 classic checkpoint with no sidecars. +#[rstest::rstest] +#[case::none(None)] +#[case::v2_no_sidecar(Some(CheckpointSpec::V2(V2CheckpointConfig::NoSidecar)))] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_snapshot_checkpoint_default_on_v2_table( + #[case] spec: Option, +) -> Result<(), Box> { + let schema = get_simple_schema(); + let (_tmp_dir, table_path, engine) = test_table_setup_mt()?; + let mut snapshot = create_table_and_load_snapshot( + &table_path, + schema.clone(), + engine.as_ref(), + &[("delta.feature.v2Checkpoint", "supported")], + )?; + + snapshot = write_batch_to_table( + &snapshot, + engine.as_ref(), + simple_id_batch(&schema, vec![1, 2]), + HashMap::new(), + ) + .await?; + + let version = snapshot.version(); + snapshot.checkpoint(engine.as_ref(), spec.as_ref())?; + + let sidecars_dir = std::path::Path::new(&table_path).join("_delta_log/_sidecars"); + assert!( + !sidecars_dir.exists(), + "_sidecars directory should not exist for a no-sidecar V2 checkpoint, found: {}", + sidecars_dir.display() + ); + + // V2 checkpoint must contain a `checkpointMetadata` column. + let ckpt_path = load_existing_single_file_checkpoint_path(&table_path, version); + let file = std::fs::File::open(&ckpt_path)?; + let reader = ParquetRecordBatchReaderBuilder::try_new(file)?.build()?; + let schema = reader.schema(); + assert!( + schema.field_with_name("checkpointMetadata").is_ok(), + "V2 checkpoint must contain `checkpointMetadata` column, found schema: {schema:?}" + ); + + Ok(()) +} + +// ============================================================================ +// Cross-feature V2 sidecar checkpoint tests +// ============================================================================ + +/// V2 sidecar checkpoint works end-to-end on a table that combines `v2Checkpoint` with +/// one other feature/layout. After the checkpoint, a fresh snapshot loads from the +/// checkpoint with no trailing commits and the scan returns the full 6-row dataset. +#[rstest::rstest] +#[case::clustered(vec![CrossFeature::Clustered])] +#[case::column_mapping_and_partitioned(vec![CrossFeature::ColumnMapping, CrossFeature::Partitioned])] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_v2_sidecar_checkpoint_cross_feature( + #[case] features: Vec, +) -> Result<(), Box> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + + let snapshot = build_v2_table_with_feature(&table_path, &table_url, &engine, &features).await?; + let version = snapshot.version(); + + // 3 add files with hint=2 -> 2 sidecars: one with 2 adds, one with 1 add. + let spec = CheckpointSpec::V2(V2CheckpointConfig::WithSidecar { + file_actions_per_sidecar_hint: Some(2), + }); + snapshot.checkpoint(engine.as_ref(), Some(&spec))?; + + let sidecars_dir = std::path::Path::new(&table_path).join("_delta_log/_sidecars"); + let per_sidecar = assert_sidecars_contain_only_file_actions(&sidecars_dir); + assert_eq!( + per_sidecar, + vec![(1, 0), (2, 0)], + "per-sidecar (adds, removes) distribution does not match expected" + ); + + // Fresh snapshot loads from the checkpoint with no trailing commits. + let fresh = Snapshot::builder_for(table_url).build(engine.as_ref())?; + assert_eq!(fresh.version(), version); + let log_segment = fresh.log_segment(); + assert!( + !log_segment.listed.checkpoint_parts.is_empty(), + "fresh snapshot should load from the checkpoint" + ); + assert_eq!(log_segment.checkpoint_version, Some(version)); + assert!( + log_segment.listed.ascending_commit_files.is_empty(), + "no commit files should remain after the checkpoint" + ); + + let scan = fresh.scan_builder().build()?; + let batches = read_scan(&scan, engine.clone() as Arc)?; + assert_batches_sorted_eq!( + vec![ + "+----+-------+--------+", + "| id | value | region |", + "+----+-------+--------+", + "| 1 | a | east |", + "| 2 | b | east |", + "| 3 | c | west |", + "| 4 | d | west |", + "| 5 | e | north |", + "| 6 | f | north |", + "+----+-------+--------+", + ], + &batches + ); + + Ok(()) +} + +/// Two consecutive V2 sidecar checkpoints with writes in between. After the second +/// checkpoint, `_last_checkpoint` advances to the latest version, the fresh snapshot +/// loads from that checkpoint with no trailing commits, and the scan returns the union +/// of both write phases. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_v2_sidecar_consecutive_checkpoints() -> Result<(), Box> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + + let schema = get_simple_schema(); + let snapshot = create_table_and_load_snapshot( + &table_path, + schema.clone(), + engine.as_ref(), + &[("delta.feature.v2Checkpoint", "supported")], + )?; + + let spec = CheckpointSpec::V2(V2CheckpointConfig::WithSidecar { + file_actions_per_sidecar_hint: Some(1), + }); + + // === Phase 1: write [1, 2] and [3, 4], then checkpoint. === + let snapshot = write_batch_to_table( + &snapshot, + engine.as_ref(), + simple_id_batch(&schema, vec![1, 2]), + HashMap::new(), + ) + .await?; + let snapshot = write_batch_to_table( + &snapshot, + engine.as_ref(), + simple_id_batch(&schema, vec![3, 4]), + HashMap::new(), + ) + .await?; + let v_first = snapshot.version(); + snapshot.checkpoint(engine.as_ref(), Some(&spec))?; + assert_eq!( + read_last_checkpoint(&table_path)["version"].as_u64(), + Some(v_first), + "_last_checkpoint should point to the first checkpoint" + ); + + // === Phase 2: write [5, 6] and [7, 8] from a fresh snapshot, then checkpoint again. === + let after_first_ckpt = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let snapshot = write_batch_to_table( + &after_first_ckpt, + engine.as_ref(), + simple_id_batch(&schema, vec![5, 6]), + HashMap::new(), + ) + .await?; + let snapshot = write_batch_to_table( + &snapshot, + engine.as_ref(), + simple_id_batch(&schema, vec![7, 8]), + HashMap::new(), + ) + .await?; + let v_second = snapshot.version(); + snapshot.checkpoint(engine.as_ref(), Some(&spec))?; + + // === Validate _last_checkpoint advanced and reflects all 4 add files. === + let last = read_last_checkpoint(&table_path); + assert_eq!(last["version"].as_u64(), Some(v_second)); + assert_eq!(last["numOfAddFiles"].as_i64(), Some(4)); + + // === Fresh snapshot loads from the second checkpoint with no trailing commits. === + let fresh = Snapshot::builder_for(table_url).build(engine.as_ref())?; + assert_eq!(fresh.version(), v_second); + let log_segment = fresh.log_segment(); + assert_eq!(log_segment.checkpoint_version, Some(v_second)); + assert!( + log_segment.listed.ascending_commit_files.is_empty(), + "no commit files should remain after the second checkpoint" + ); + + // === Scan returns all 8 ids written across both phases. === + let scan = fresh.scan_builder().build()?; + let batches = read_scan(&scan, engine.clone() as Arc)?; + #[rustfmt::skip] + let expected = vec![ + "+----+", + "| id |", + "+----+", + "| 1 |", + "| 2 |", + "| 3 |", + "| 4 |", + "| 5 |", + "| 6 |", + "| 7 |", + "| 8 |", + "+----+", + ]; + assert_batches_sorted_eq!(expected, &batches); + + Ok(()) +} + +/// On a table with `deletionVectors` + `rowTracking` enabled, insert data, apply DV and write a +/// V2 sidecar checkpoint. Scan back and make sure the DV and row-tracking columns are present. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_v2_sidecar_preserves_dv_and_row_tracking_on_add( +) -> Result<(), Box> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + let schema = get_simple_schema(); + + // === Step 1: Create v2 + DV + row-tracking table. === + let _ = create_table(&table_path, schema.clone(), "Test/1.0") + .with_table_properties([ + ("delta.feature.v2Checkpoint", "supported"), + ("delta.feature.deletionVectors", "supported"), + ("delta.enableDeletionVectors", "true"), + ("delta.enableRowTracking", "true"), + ]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + // === Step 2: Append one batch. === + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let snapshot = write_batch_to_table( + &snapshot, + engine.as_ref(), + simple_id_batch(&schema, vec![1, 2, 3]), + HashMap::new(), + ) + .await?; + + // === Step 3: Apply a DV to the just-written file. === + let path = read_add_infos(snapshot.as_ref(), engine.as_ref())?[0] + .path + .clone(); + let dv = DeletionVectorDescriptor { + storage_type: DeletionVectorStorageType::PersistedRelative, + path_or_inline_dv: "abcdefghijklmnopqrst".to_string(), + offset: Some(7), + size_in_bytes: 42, + cardinality: 1, + }; + let scan_files: Vec<_> = snapshot + .clone() + .scan_builder() + .build()? + .scan_metadata(engine.as_ref())? + .map_ok(|sm| sm.scan_files) + .try_collect()?; + let mut txn = begin_transaction(snapshot, engine.as_ref())?.with_data_change(true); + txn.update_deletion_vectors( + HashMap::from([(path, dv.clone())]), + scan_files.into_iter().map(Ok), + )?; + let snapshot = txn.commit(engine.as_ref())?.unwrap_post_commit_snapshot(); + + // === Step 4: Write a V2 sidecar checkpoint. === + snapshot.checkpoint( + engine.as_ref(), + Some(&CheckpointSpec::V2(V2CheckpointConfig::WithSidecar { + file_actions_per_sidecar_hint: None, + })), + )?; + + // === Step 5: Read the sidecar parquet directly and assert protocol fields on the + // live add: `add.deletionVector.*`, `add.baseRowId`, `add.defaultRowCommitVersion`. === + let sidecars = + list_sidecar_parquet_files(&std::path::Path::new(&table_path).join("_delta_log/_sidecars")); + assert_eq!(sidecars.len(), 1, "default hint should produce one sidecar"); + let batch = read_parquet_file(&sidecars[0].path()); + let add = get_struct_column_from_record_batch(&batch, "add"); + let live = valid_row_indices(add, batch.num_rows()); + assert_eq!(live.len(), 1, "expected one live add in sidecar"); + let row = live[0]; + + // Closures that read a leaf value at the given field path under `add`. + let path_to_vec = |path: &[&str]| path.iter().map(|s| s.to_string()).collect::>(); + let read_str = |path| resolve_struct_field::(add, &path_to_vec(path)).value(row); + let read_i32 = |path| resolve_struct_field::(add, &path_to_vec(path)).value(row); + let read_i64 = |path| resolve_struct_field::(add, &path_to_vec(path)).value(row); + assert_eq!(read_str(&["deletionVector", "storageType"]), "u"); + assert_eq!( + read_str(&["deletionVector", "pathOrInlineDv"]), + dv.path_or_inline_dv + ); + assert_eq!(read_i32(&["deletionVector", "offset"]), 7); + assert_eq!(read_i32(&["deletionVector", "sizeInBytes"]), 42); + assert_eq!(read_i64(&["deletionVector", "cardinality"]), 1); + // `update_deletion_vectors` preserves the original `baseRowId` (0 -- first file) and + // `defaultRowCommitVersion` (1 -- version that first wrote the file). + assert_eq!(read_i64(&["baseRowId"]), 0); + assert_eq!(read_i64(&["defaultRowCommitVersion"]), 1); + + Ok(()) +} + +/// Verify the default `file_actions_per_sidecar_hint` (50_000): staging 60_000 add actions +/// across 60 batches of 1000 (so the splitter cuts cleanly at the 50k boundary) must +/// produce exactly 2 sidecar files. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_v2_sidecar_default_hint_splits_at_50k() -> Result<(), Box> { + const PER_COMMIT: usize = 1_000; + const COMMITS: usize = 60; + + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + + // === Step 1: Create v2Checkpoint table. === + let _ = create_table(&table_path, get_simple_schema(), "Test/1.0") + .with_table_properties([("delta.feature.v2Checkpoint", "supported")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + // === Step 2: Run 60 commits of 1_000 synthetic adds each (60_000 total). === + let mut snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + for c in 0..COMMITS { + let mut txn = begin_transaction(snapshot, engine.as_ref())?.with_data_change(true); + let add_files_schema = txn.add_files_schema().clone(); + let paths: Vec = (0..PER_COMMIT) + .map(|i| format!("part-{c:03}-{i:04}.parquet")) + .collect(); + // Each tuple is (path, size_bytes, modification_time, num_records). + let files: Vec<(&str, i64, i64, Option)> = paths + .iter() + .map(|p| (p.as_str(), 100, 0, Some(1))) + .collect(); + txn.add_files(create_add_files_metadata(&add_files_schema, files)?); + snapshot = txn.commit(engine.as_ref())?.unwrap_post_commit_snapshot(); + } + + // === Step 3: Sidecar checkpoint with default hint (None -> 50_000). === + snapshot.checkpoint( + engine.as_ref(), + Some(&CheckpointSpec::V2(V2CheckpointConfig::WithSidecar { + file_actions_per_sidecar_hint: None, + })), + )?; + + // === Step 4: 50 commits' worth (50k adds) fills the first sidecar at the 50k hint; + // the remaining 10 commits (10k adds) go into the second sidecar. === + let sidecars_dir = std::path::Path::new(&table_path).join("_delta_log/_sidecars"); + let per_sidecar = assert_sidecars_contain_only_file_actions(&sidecars_dir); + assert_eq!( + per_sidecar, + vec![(10_000, 0), (50_000, 0)], + "per-sidecar (adds, removes) distribution" + ); + assert_eq!( + read_last_checkpoint(&table_path)["numOfAddFiles"].as_i64(), + Some((PER_COMMIT * COMMITS) as i64), + ); + + Ok(()) +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum CrossFeature { + ColumnMapping, + Partitioned, + Clustered, +} + +/// Schema shared across all `CrossFeature` variants: `id: int, value: string, +/// region: string`. The `region` column is used as the partition column in the +/// `Partitioned` variant; `id` is the clustering column in the `Clustered` variant. +fn cross_feature_schema() -> Arc { + Arc::new( + StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("value", DataType::STRING), + StructField::nullable("region", DataType::STRING), + ]) + .unwrap(), + ) +} + +/// Creates a V2 table for the given `features` and writes 3 commits totaling 6 rows +/// across 3 distinct `region` values, returning the post-write snapshot. All variants +/// share [`cross_feature_schema`] so the post-checkpoint scan can be asserted with the +/// same expected output. +async fn build_v2_table_with_feature( + table_path: &str, + table_url: &url::Url, + engine: &Arc>, + features: &[CrossFeature], +) -> Result, Box> { + let schema = cross_feature_schema(); + + let mut props: Vec<(&str, &str)> = vec![("delta.feature.v2Checkpoint", "supported")]; + let mut layout: Option = None; + for feature in features { + match feature { + CrossFeature::ColumnMapping => { + // As we are verifying sidecar works with column mapping, either name or id mode + // is acceptable here. + props.push(("delta.columnMapping.mode", "name")); + } + CrossFeature::Partitioned => { + assert!( + layout.is_none(), + "Partitioned and Clustered are mutually exclusive" + ); + layout = Some(DataLayout::partitioned(["region"])); + } + CrossFeature::Clustered => { + assert!( + layout.is_none(), + "Partitioned and Clustered are mutually exclusive" + ); + layout = Some(DataLayout::clustered(["id"])); + } + } + } + + let mut builder = + create_table(table_path, schema.clone(), "Test/1.0").with_table_properties(props); + if let Some(layout) = layout { + builder = builder.with_data_layout(layout); + } + let _ = builder + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + // For partitioned tables, the data batches must NOT include the partition column -- + // the partition value is supplied separately in `partition_values`. + let partitioned = features.contains(&CrossFeature::Partitioned); + let data_arrow_schema = if partitioned { + Arc::new(ArrowSchema::try_from_kernel(&StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("value", DataType::STRING), + ])?)?) + } else { + Arc::new(ArrowSchema::try_from_kernel(schema.as_ref())?) + }; + + let writes: [(&str, Vec, Vec<&str>); 3] = [ + ("east", vec![1, 2], vec!["a", "b"]), + ("west", vec![3, 4], vec!["c", "d"]), + ("north", vec![5, 6], vec!["e", "f"]), + ]; + + let mut snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + for (region, ids, values) in writes { + let id_arr: ArrayRef = Arc::new(Int32Array::from(ids.clone())); + let val_arr: ArrayRef = Arc::new(StringArray::from(values)); + let columns: Vec = if partitioned { + vec![id_arr, val_arr] + } else { + let region_arr: ArrayRef = Arc::new(StringArray::from(vec![region; ids.len()])); + vec![id_arr, val_arr, region_arr] + }; + let batch = RecordBatch::try_new(data_arrow_schema.clone(), columns)?; + let pv = if partitioned { + HashMap::from([("region".to_string(), Scalar::from(region))]) + } else { + HashMap::new() + }; + snapshot = write_batch_to_table(&snapshot, engine.as_ref(), batch, pv).await?; + } + + Ok(snapshot) +} + +/// A version holding two complete checkpoints must load from the uuid-named one, since it outranks +/// the classic-named one. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn snapshot_selects_uuid_checkpoint_over_classic_at_one_version() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + + let schema = schema_ref! { nullable "value": INTEGER }; + let _ = create_table(&table_path, schema, "Test/1.0") + .with_table_properties([("delta.feature.v2Checkpoint", "supported")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let snapshot0 = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let snapshot = insert_data( + snapshot0, + &engine, + vec![Arc::new(Int32Array::from(vec![1, 2, 3]))], + ) + .await? + .unwrap_post_commit_snapshot(); + let version = snapshot.version(); + + // Writes `.checkpoint.parquet` plus a `_last_checkpoint` describing it. + snapshot.checkpoint( + engine.as_ref(), + Some(&CheckpointSpec::V2(V2CheckpointConfig::NoSidecar)), + )?; + + // As of v0.27.0 kernel writes only classic names, which this test depends on. + let classic_path = load_existing_single_file_checkpoint_path(&table_path, version); + assert_eq!( + classic_path.file_name().unwrap().to_str().unwrap(), + format!("{version:020}.checkpoint.parquet"), + ); + + // Add a second complete checkpoint at this version, uuid-named so it outranks the classic one. + // Kernel's write path cannot produce one, so copy the V2 (classic-named) checkpoint it wrote to + // a V2 (uuid-named) path: same contents, and both namings are spec-valid under + // `v2Checkpoint`. + let uuid_name = format!("{version:020}.checkpoint.{}.parquet", uuid::Uuid::new_v4()); + let uuid_path = std::path::Path::new(&table_path) + .join("_delta_log") + .join(&uuid_name); + std::fs::copy(&classic_path, &uuid_path).expect("copy classic checkpoint to a uuid name"); + + // No path and no `parts` implies the classic checkpoint. Kernel doesn't write the + // `v2Checkpoint` field yet (TODO #1052), so a hint can never name a uuid checkpoint. + let last_checkpoint = read_last_checkpoint(&table_path); + assert_eq!(last_checkpoint["version"].as_u64(), Some(version)); + assert!(last_checkpoint.get("parts").is_none()); + assert!(last_checkpoint.get("v2Checkpoint").is_none()); + + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + assert_eq!(snapshot.version(), version); + + let log_segment = snapshot.log_segment(); + let selected: Vec<&str> = log_segment + .listed + .checkpoint_parts + .iter() + .map(|p| p.filename.as_str()) + .collect(); + assert_eq!( + selected, + vec![uuid_name.as_str()], + "expected the uuid-named checkpoint to outrank the classic one" + ); + assert_eq!(log_segment.checkpoint_version, Some(version)); + + // The hint implies the classic checkpoint, not the selected uuid one, so its fields get + // dropped. + assert!(log_segment.checkpoint_hint().is_none()); + + // Replay returns the rows written above, so the hand-placed copy is a readable checkpoint. + let scan = snapshot.scan_builder().build()?; + let rows: usize = read_scan(&scan, engine.clone() as Arc)? + .iter() + .map(|b| b.num_rows()) + .sum(); + assert_eq!(rows, 3); + + Ok(()) +} diff --git a/kernel/tests/integration/main.rs b/kernel/tests/integration/main.rs new file mode 100644 index 0000000000..18d4b3d153 --- /dev/null +++ b/kernel/tests/integration/main.rs @@ -0,0 +1,21 @@ +//! Single integration test crate for delta_kernel. +//! +//! All integration tests are linked into one binary so they share a single +//! compile + link cycle. Each topic lives in its own submodule below and is +//! organized by file/directory. + +#[macro_use] +mod common; + +mod create_table; +mod cross_product; +mod data_skipping; +mod features; +mod golden_tables; +mod hdfs; +mod log; +mod metrics; +mod parsed_partition_values; +mod parsed_stats; +mod read; +mod write; diff --git a/kernel/tests/integration/metrics/commit.rs b/kernel/tests/integration/metrics/commit.rs new file mode 100644 index 0000000000..c3338da4f0 --- /dev/null +++ b/kernel/tests/integration/metrics/commit.rs @@ -0,0 +1,380 @@ +//! Metrics tests for `Transaction::commit`. +//! +//! Each test builds a table, attaches a metrics reporter, runs a commit, and asserts the +//! commit metrics a real connector would observe. + +use std::sync::{Arc, Mutex}; + +use delta_kernel::arrow::array::Int32Array; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::metrics::{MetricEvent, MetricsReporter, TableType, TransactionCommitSuccess}; +use delta_kernel::object_store::local::LocalFileSystem; +use delta_kernel::schema::{DataType, StructField, StructType}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::transaction::CommitResult; +use delta_kernel::{DeltaResult, Snapshot}; +use rstest::rstest; +use test_utils::delta_kernel_default_engine::DefaultEngineBuilder; +use test_utils::{ + assert_result_error_with_message, begin_transaction, create_add_files_metadata, insert_data, + insert_data_with, install_thread_local_metrics_reporter, test_table_setup_mt, +}; +use url::Url; + +use super::table_type::{create_simple_table, make_committer}; +use super::{measuring_engine, simple_schema}; +use crate::common::write_utils::{ + create_dv_table_with_files, get_scan_files, sequential_dv_descriptors, +}; + +/// Reporter that keeps the last `TransactionCommitSuccess` for field-level assertions. +#[derive(Debug, Default)] +struct LastCommitSuccess(Mutex>); + +impl MetricsReporter for LastCommitSuccess { + fn report(&self, event: MetricEvent) { + if let MetricEvent::TransactionCommitSuccess(success) = event { + *self.0.lock().unwrap() = Some(success); + } + } +} + +impl LastCommitSuccess { + /// Returns the last captured commit-success event, panicking if none was recorded. + fn take_success(&self) -> TransactionCommitSuccess { + self.0 + .lock() + .unwrap() + .clone() + .expect("commit success event") + } +} + +fn setup_empty_table() -> DeltaResult<(tempfile::TempDir, Url)> { + let (temp_dir, table_path, setup_engine) = test_table_setup_mt()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + create_table(&table_path, simple_schema(), "Test/1.0") + .build(setup_engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(setup_engine.as_ref())? + .unwrap_committed(); + Ok((temp_dir, table_url)) +} + +#[rstest] +#[case::write_append("WRITE", true, false)] +#[case::blind_append("WRITE", true, true)] +#[case::optimize_no_data_change("OPTIMIZE", false, false)] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn commit_append_emits_success_metrics( + #[case] operation: &str, + #[case] data_change: bool, + #[case] is_blind_append: bool, +) -> DeltaResult<()> { + let (_temp_dir, table_url) = setup_empty_table()?; + let reporter = Arc::new(LastCommitSuccess::default()); + let engine = Arc::new(DefaultEngineBuilder::new(Arc::new(LocalFileSystem::new())).build()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + let snap = Snapshot::builder_for(table_url).build(engine.as_ref())?; + + insert_data_with( + snap, + &engine, + vec![Arc::new(Int32Array::from(vec![1, 2, 3]))], + Box::new(FileSystemCommitter::new()), + operation, + data_change, + is_blind_append, + ) + .await? + .unwrap_committed(); + + let success = reporter.take_success(); + assert_eq!(success.table_type, TableType::PathBased); + assert_eq!(success.commit_version, 1); + assert_eq!(success.num_add_files, 1); + assert!(success.add_files_bytes > 0); + assert_eq!(success.num_remove_files, 0); + assert_eq!(success.remove_files_bytes, 0); + assert_eq!(success.operation.as_deref(), Some(operation)); + assert_eq!(success.data_change, data_change); + assert_eq!(success.is_blind_append, is_blind_append); + assert!(success.total_duration >= success.prepare_duration + success.committer_duration); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn commit_reports_added_file_count_not_batch_count() -> DeltaResult<()> { + // num_add_files counts added FILES, not add_files() batches: two batches of two files each + // must report 4. A regression to add_files_metadata.len() (a batch count) would report 2. + let (_temp_dir, table_url) = setup_empty_table()?; + let reporter = Arc::new(LastCommitSuccess::default()); + let engine = Arc::new(DefaultEngineBuilder::new(Arc::new(LocalFileSystem::new())).build()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + let snap = Snapshot::builder_for(table_url).build(engine.as_ref())?; + + let mut txn = begin_transaction(snap, engine.as_ref())? + .with_operation("WRITE".to_string()) + .with_data_change(true); + let add_files_schema = txn.add_files_schema(); + // Two separate add_files() calls -> two batches, four files total. + let batches = vec![ + vec![ + ("f0.parquet", 1024, 1_000_000, Some(3)), + ("f1.parquet", 1124, 1_000_001, Some(3)), + ], + vec![ + ("f2.parquet", 1224, 1_000_002, Some(3)), + ("f3.parquet", 1324, 1_000_003, Some(3)), + ], + ]; + for batch in batches { + let metadata = create_add_files_metadata(add_files_schema, batch) + .map_err(|e| delta_kernel::Error::generic(e.to_string()))?; + txn.add_files(metadata); + } + txn.commit(engine.as_ref())?.unwrap_committed(); + + let success = reporter.take_success(); + assert_eq!(success.num_add_files, 4); + Ok(()) +} + +/// Sets the correlation id on the `Transaction` returned by `build()` and checks it reaches the +/// commit metric event. The two tests below instead set it on the builder. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn commit_success_carries_correlation_id() -> DeltaResult<()> { + let (_temp_dir, table_path, setup_engine) = test_table_setup_mt()?; + let reporter = Arc::new(LastCommitSuccess::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + create_table(&table_path, simple_schema(), "Test/1.0") + .build(setup_engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .with_correlation_id("commit-req-1") + .commit(setup_engine.as_ref())? + .unwrap_committed(); + + let success = reporter.take_success(); + assert_eq!(success.commit_version, 0); + assert_eq!(success.correlation_id.as_deref(), Some("commit-req-1")); + Ok(()) +} + +/// A correlation id set on the create-table *builder* (rather than on the `Transaction` it +/// produces) reaches the commit metric event, and an empty id is treated as unset. This is the +/// builder-level setter added in issue #2833. +#[rstest] +#[case::with_id(Some("create-req-1"), Some("create-req-1"))] +#[case::without_id(None, None)] +#[case::empty_id_is_unset(Some(""), None)] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn create_table_builder_carries_correlation_id( + #[case] correlation_id: Option<&str>, + #[case] expected: Option<&str>, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let reporter = Arc::new(LastCommitSuccess::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let mut builder = create_table(&table_path, simple_schema(), "Test/1.0"); + if let Some(id) = correlation_id { + builder = builder.with_correlation_id(id); + } + builder + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + + let success = reporter.take_success(); + assert_eq!(success.commit_version, 0); + assert_eq!(success.correlation_id.as_deref(), expected); + Ok(()) +} + +/// A correlation id set on the alter-table *builder* reaches the commit metric event. The id is +/// set before any schema operation (in the `Ready` state), so this also verifies it survives the +/// builder's `Ready -> Modifying` transition. An empty id is treated as unset. Builder-level +/// setter added in issue #2833. +#[rstest] +#[case::with_id(Some("alter-req-1"), Some("alter-req-1"))] +#[case::without_id(None, None)] +#[case::empty_id_is_unset(Some(""), None)] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn alter_table_builder_carries_correlation_id( + #[case] correlation_id: Option<&str>, + #[case] expected: Option<&str>, +) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + create_table(&table_path, simple_schema(), "Test/1.0") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + + // Install the reporter after the create commit so the captured event is the alter commit. + let reporter = Arc::new(LastCommitSuccess::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let table_url = delta_kernel::try_parse_uri(&table_path)?; + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + // Set the id in the `Ready` state (before `add_column`) to exercise the carry-through. + let mut builder = snapshot.alter_table(); + if let Some(id) = correlation_id { + builder = builder.with_correlation_id(id); + } + builder + .add_column(StructField::nullable("extra", DataType::STRING)) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + + let success = reporter.take_success(); + assert_eq!(success.commit_version, 1); + assert_eq!(success.correlation_id.as_deref(), expected); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn commit_conflict_emits_conflict_metric() -> DeltaResult<()> { + // GIVEN a table at v0 (00.json) and a snapshot pinned to it. + let (_temp_dir, table_url) = setup_empty_table()?; + let (engine, reporter, _guard) = measuring_engine(Arc::new(LocalFileSystem::new())); + let engine = Arc::new(engine); + let snap = Snapshot::builder_for(table_url).build(engine.as_ref())?; + + reporter.reset(); + // WHEN a first append commits against that snapshot, advancing the table to v1 (01.json). + insert_data( + snap.clone(), + &engine, + vec![Arc::new(Int32Array::from(vec![1]))], + ) + .await? + .unwrap_committed(); + // AND a second append reuses the SAME v0 snapshot, so it also targets v1 (already written). + let result = insert_data(snap, &engine, vec![Arc::new(Int32Array::from(vec![2]))]).await?; + + // THEN the second commit conflicts and emits exactly one conflict metric. + assert!(matches!(result, CommitResult::ConflictedTransaction(_))); + assert_eq!(reporter.transaction_commits.get(), 1); + assert_eq!(reporter.commit_conflicts.get(), 1); + assert_eq!(reporter.commit_errors.get(), 0); + Ok(()) +} + +#[rstest] +#[case::path_based(false)] +#[case::catalog_managed(true)] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn commit_success_carries_table_type(#[case] catalog_managed: bool) -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + create_simple_table(engine.as_ref(), &table_path, catalog_managed)?; + + let reporter = Arc::new(LastCommitSuccess::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let builder = Snapshot::builder_for(table_url); + let builder = if catalog_managed { + builder.with_max_catalog_version(0) + } else { + builder + }; + let snapshot = builder.build(engine.as_ref())?; + insert_data_with( + snapshot, + &engine, + vec![Arc::new(Int32Array::from(vec![1]))], + make_committer(catalog_managed), + "WRITE", + /* data_change */ true, + /* is_blind_append */ false, + ) + .await? + .unwrap_committed(); + + let success = reporter.take_success(); + assert_eq!( + success.table_type, + TableType::from_catalog_managed(catalog_managed) + ); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn commit_dv_update_reports_updated_file_count_not_batch_count( +) -> Result<(), Box> { + // Three files get a new deletion vector in a single update_deletion_vectors call. + // num_dv_updates must be the count of updated FILES (3), not the number of scan-metadata + // batches they arrive in -- a single batch carrying all three would otherwise report 1. + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("value", DataType::STRING), + ])?); + let file_names = &["file0.parquet", "file1.parquet", "file2.parquet"]; + let (_store, engine, table_url, file_paths) = + create_dv_table_with_files("dv_metrics_table", schema, &[], file_names).await?; + + let reporter = Arc::new(LastCommitSuccess::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + let mut txn = begin_transaction(snapshot.clone(), engine.as_ref())? + .with_operation("UPDATE".to_string()) + .with_data_change(true); + + let mut paths_with_unmatched = file_paths.clone(); + paths_with_unmatched.push("missing.parquet".to_string()); + let mut scan_files = get_scan_files(snapshot.clone(), engine.as_ref())?; + assert_result_error_with_message( + txn.update_deletion_vectors( + sequential_dv_descriptors(&paths_with_unmatched), + scan_files.drain(..).map(Ok), + ), + "Number of matched DV files does not match number of new DV descriptors: 3 != 4", + ); + + // A failed call with an unmatched path must not contribute its three matched files. + let mut scan_files = get_scan_files(snapshot, engine.as_ref())?; + let dv_map = sequential_dv_descriptors(&file_paths); + txn.update_deletion_vectors(dv_map, scan_files.drain(..).map(Ok))?; + txn.commit(engine.as_ref())?.unwrap_committed(); + + let success = reporter.take_success(); + assert_eq!(success.num_dv_updates, 3); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn commit_dv_update_accumulates_file_count_across_calls( +) -> Result<(), Box> { + // num_dv_updates must accumulate (`+=`) across multiple update_deletion_vectors calls on one + // transaction: two calls updating one file each must report 2, not the last call's 1. + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("value", DataType::STRING), + ])?); + let file_names = &["file0.parquet", "file1.parquet"]; + let (_store, engine, table_url, file_paths) = + create_dv_table_with_files("dv_metrics_multi_call_table", schema, &[], file_names).await?; + + let reporter = Arc::new(LastCommitSuccess::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + let mut txn = begin_transaction(snapshot.clone(), engine.as_ref())? + .with_operation("UPDATE".to_string()) + .with_data_change(true); + + // Each call re-derives scan files from the same snapshot and updates exactly one file, so + // matched_dv_files is 1 per call and the accumulator must reach 2. + let all_descriptors = sequential_dv_descriptors(&file_paths); + for path in &file_paths { + let mut scan_files = get_scan_files(snapshot.clone(), engine.as_ref())?; + let dv_map = std::iter::once((path.clone(), all_descriptors[path].clone())).collect(); + txn.update_deletion_vectors(dv_map, scan_files.drain(..).map(Ok))?; + } + txn.commit(engine.as_ref())?.unwrap_committed(); + + let success = reporter.take_success(); + assert_eq!(success.num_dv_updates, 2); + Ok(()) +} diff --git a/kernel/tests/integration/metrics/mod.rs b/kernel/tests/integration/metrics/mod.rs new file mode 100644 index 0000000000..c4da569550 --- /dev/null +++ b/kernel/tests/integration/metrics/mod.rs @@ -0,0 +1,128 @@ +//! Integration tests that verify [`CountingReporter`] metrics across different table +//! states and API call patterns. +//! +//! All tests use only public kernel APIs (`create_table`, `Transaction`, `write_parquet`, +//! `log_compaction_writer`, `scan.execute()`) to ensure that the metrics surface is +//! verified through the same code paths a real engine connector would exercise. +//! +//! Tests are organized by concern: +//! - [`snapshot_load`]: snapshot-loading scenarios (delta-only, checkpoint, compaction, CRC, and +//! on-demand API calls like `get_domain_metadata`) +//! - [`scan`]: scan execution scenarios (`scan.execute()` parquet data-file reads) +//! - [`commit`]: `Transaction::commit()` success / conflict metrics +//! +//! Where possible, tests use [`TestTableBuilder`] for table setup. Tests that need +//! checkpoint, CRC, or log compaction features still use manual helpers until those +//! [`LogState`] variants land. + +use std::sync::Arc; + +use delta_kernel::arrow::array::Int32Array; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::schema::{DataType, StructField, StructType}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::{DeltaResult, Snapshot}; +use test_utils::delta_kernel_default_engine::executor::tokio::TokioMultiThreadExecutor; +use test_utils::delta_kernel_default_engine::{DefaultEngine, DefaultEngineBuilder}; +use test_utils::table_builder::{LogState, TestTableBuilder}; +use test_utils::{ + insert_data, install_thread_local_metrics_reporter, test_table_setup_mt, CountingReporter, +}; +use url::Url; + +mod commit; +mod parallel_scan; +mod scan; +mod snapshot_load; +mod table_type; + +/// Build a `DefaultEngine` + `CountingReporter` backed by `store`, for use in the +/// *measurement* phase (building a snapshot and asserting metric counters). +/// +/// Returns the engine, the reporter, and a tracing `DefaultGuard` that must be kept alive +/// for the duration of the test. When the guard is dropped the subscriber is uninstalled. +fn measuring_engine( + store: Arc, +) -> ( + DefaultEngine< + test_utils::delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor, + >, + Arc, + tracing::subscriber::DefaultGuard, +) { + let reporter = Arc::new(CountingReporter::default()); + let engine = DefaultEngineBuilder::new(store).build(); + let guard = install_thread_local_metrics_reporter(reporter.clone()); + (engine, reporter, guard) +} + +/// Build a minimal single-column INTEGER schema for test tables. +fn simple_schema() -> Arc { + Arc::new( + StructType::try_new(vec![StructField::nullable("id", DataType::INTEGER)]) + .expect("valid schema"), + ) +} + +/// Insert `count` rows (starting from `start_val`) into an existing table, using +/// `post_commit_snapshot` to chain snapshots instead of rebuilding from scratch. +async fn insert_rows( + table_url: &Url, + engine: &Arc< + DefaultEngine< + test_utils::delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor, + >, + >, + start_val: i32, + count: i32, +) -> DeltaResult<()> { + let mut snap = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + for val in start_val..(start_val + count) { + let committed = insert_data(snap, engine, vec![Arc::new(Int32Array::from(vec![val]))]) + .await? + .unwrap_committed(); + snap = committed + .post_commit_snapshot() + .expect("post-commit snapshot") + .clone(); + } + Ok(()) +} + +// TODO: migrate to TestTableBuilder when checkpoint LogState variants land (#2284) +/// Create a table at v0, insert one row, and write a v1 parquet checkpoint. +/// +/// Returns `(table_url, setup_engine, _temp_dir)` where `_temp_dir` must be kept +/// alive for the duration of the test to prevent early cleanup. Callers can add +/// further commits via `insert_data` or directly build the measuring snapshot. +/// +/// Uses `TokioMultiThreadExecutor` because `snapshot.checkpoint()` issues nested +/// `block_on` calls; `TokioBackgroundExecutor` deadlocks in that pattern. +async fn setup_table_with_v1_checkpoint() -> DeltaResult<( + Url, + Arc>, + tempfile::TempDir, +)> { + let (temp_dir, table_path, setup_engine) = test_table_setup_mt()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + + let _ = create_table(&table_path, simple_schema(), "Test/1.0") + .build(setup_engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(setup_engine.as_ref())?; + + let snap0 = Snapshot::builder_for(table_url.clone()).build(setup_engine.as_ref())?; + let committed = insert_data( + snap0, + &setup_engine, + vec![Arc::new(Int32Array::from(vec![1]))], + ) + .await? + .unwrap_committed(); + committed + .post_commit_snapshot() + .expect("post-commit snapshot") + .clone() + .checkpoint(setup_engine.as_ref(), None)?; + + Ok((table_url, setup_engine, temp_dir)) +} diff --git a/kernel/tests/integration/metrics/parallel_scan.rs b/kernel/tests/integration/metrics/parallel_scan.rs new file mode 100644 index 0000000000..40540cd829 --- /dev/null +++ b/kernel/tests/integration/metrics/parallel_scan.rs @@ -0,0 +1,79 @@ +//! Verifies that the parallel scan-metadata path (`parallel_scan_metadata`) emits +//! `ScanMetadataCompleted` events carrying the correct [`TableType`], exercising the live +//! `StateInfo` -> phase-event wiring (a different source than the full-scan path). + +use std::sync::Arc; + +use delta_kernel::arrow::array::Int32Array; +use delta_kernel::metrics::TableType; +use delta_kernel::scan::{AfterSequentialScanMetadata, ParallelScanMetadata}; +use delta_kernel::{DeltaResult, Engine, Snapshot}; +use rstest::rstest; +use test_utils::{ + insert_data_with, install_thread_local_metrics_reporter, test_table_setup_mt, CapturingReporter, +}; + +use super::table_type::{assert_table_types, create_simple_table, make_committer}; + +#[rstest] +#[case::path_based(false)] +#[case::catalog_managed(true)] +#[tokio::test(flavor = "multi_thread")] +async fn test_parallel_scan_metadata_events_carry_table_type( + #[case] catalog_managed: bool, +) -> DeltaResult<()> { + let (_tmp, table_path, engine) = test_table_setup_mt()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + let build_snapshot = |version: u64| -> DeltaResult> { + let builder = Snapshot::builder_for(table_url.clone()); + let builder = if catalog_managed { + builder.with_max_catalog_version(version) + } else { + builder + }; + builder.build(engine.as_ref()) + }; + + // Given: a table with one appended row. + create_simple_table(engine.as_ref(), &table_path, catalog_managed)?; + insert_data_with( + build_snapshot(0)?, + &engine, + vec![Arc::new(Int32Array::from(vec![1]))], + make_committer(catalog_managed), + "WRITE", + /* data_change */ true, + /* is_blind_append */ false, + ) + .await? + .unwrap_committed(); + + // When: the table is scanned via the parallel (sequential + optional distributed) path. + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + let scan = build_snapshot(1)?.scan_builder().build()?; + let engine_dyn: Arc = engine.clone(); + let mut sequential = scan.parallel_scan_metadata(engine_dyn.clone())?; + for result in sequential.by_ref() { + result?; + } + if let AfterSequentialScanMetadata::Parallel { state, files } = sequential.finish()? { + let state = Arc::new(*state); + for file in files { + let parallel = + ParallelScanMetadata::try_new(engine_dyn.clone(), state.clone(), vec![file])?; + for result in parallel { + result?; + } + } + state.log_metrics(); + } + + // Then: the emitted scan-metadata event(s) carry the expected table_type. + assert_table_types( + &reporter.events(), + &["ScanMetadataCompleted"], + TableType::from_catalog_managed(catalog_managed), + ); + Ok(()) +} diff --git a/kernel/tests/integration/metrics/scan.rs b/kernel/tests/integration/metrics/scan.rs new file mode 100644 index 0000000000..8344435ee0 --- /dev/null +++ b/kernel/tests/integration/metrics/scan.rs @@ -0,0 +1,60 @@ +//! Scan execution metrics tests. +//! +//! Covers how `scan.execute()` contributes to I/O metrics: parquet data-file reads +//! through `DefaultParquetHandler::read_parquet_files` and the JSON log replay that +//! `scan.execute()` performs internally to collect Add/Remove scan metadata. + +use std::sync::Arc; + +use delta_kernel::{DeltaResult, Engine, Snapshot}; + +use super::{measuring_engine, LogState, TestTableBuilder}; + +// ============================================================================ +// scan.execute() contributes parquet data-file reads +// ============================================================================ + +/// `scan.execute()` reads the actual parquet data files written during inserts. +/// These go through `DefaultParquetHandler::read_parquet_files` and appear in +/// `parquet_read_calls`, separately from any checkpoint reads. Resetting the +/// reporter after snapshot construction isolates the scan I/O. +/// +/// Note: `scan.execute()` also does its own log replay (to collect Add/Remove +/// actions for scan metadata), so `json_read_calls` is non-zero even after the +/// reporter reset. +#[test] +fn scan_execute_contributes_parquet_data_file_reads() -> DeltaResult<()> { + let table = TestTableBuilder::new() + .with_log_state(LogState::with_latest_version(2)) + .with_data(1, 1) + .build()?; + + let (engine, reporter, _guard) = measuring_engine(table.store().clone()); + let snap = Snapshot::builder_for(table.table_root()).build(&engine)?; + + // Reset after snapshot build to isolate scan I/O + reporter.reset(); + + let engine: Arc = Arc::new(engine); + let mut batches_seen = 0usize; + for result in snap.scan_builder().build()?.execute(engine)? { + result?; + batches_seen += 1; + } + assert_eq!( + batches_seen, 2, + "scan should return one batch per data file" + ); + + // scan calls read_parquet_files once per data file (not batched), so 2 calls for 2 files + assert_eq!(reporter.parquet_read_calls.get(), 2); + assert_eq!(reporter.parquet_files_read.get(), 2); + // On Windows (NTFS), listing a recently written file can return size=0 because the OS + // has not yet flushed size metadata to the directory entry. + #[cfg(not(windows))] + assert!(reporter.parquet_bytes_read.get() > 0); + // scan.execute() does its own log replay for Add/Remove scan metadata + assert_eq!(reporter.json_read_calls.get(), 1); + + Ok(()) +} diff --git a/kernel/tests/integration/metrics/snapshot_load.rs b/kernel/tests/integration/metrics/snapshot_load.rs new file mode 100644 index 0000000000..39ded9038c --- /dev/null +++ b/kernel/tests/integration/metrics/snapshot_load.rs @@ -0,0 +1,548 @@ +//! Metrics tests for snapshot loading and on-demand snapshot APIs. +//! +//! Each test builds a table in a known state, attaches a `CountingReporter`, and asserts the +//! exact metric counters a real connector would observe. + +use std::collections::HashMap; +use std::path::PathBuf; +use std::sync::Arc; + +use delta_kernel::arrow::array::Int32Array; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::engine::to_json_bytes; +use delta_kernel::object_store::local::LocalFileSystem; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::ObjectStoreExt as _; +use delta_kernel::snapshot::IncrementalReplay; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::transaction::data_layout::DataLayout; +use delta_kernel::{DeltaResult, Snapshot}; +use rstest::rstest; +use test_utils::delta_kernel_default_engine::DefaultEngineBuilder; +use test_utils::{insert_data, test_table_setup, test_table_setup_mt}; +use url::Url; + +use super::{ + insert_rows, measuring_engine, setup_table_with_v1_checkpoint, simple_schema, LogState, + TestTableBuilder, +}; + +// ============================================================================ +// Scenario 1: delta-only (2 commits, no checkpoint, no compaction) +// ============================================================================ + +/// A snapshot built from two JSON commits -- no checkpoint, no CRC, no compaction -- +/// reports exactly the commit file count and triggers one JSON read call covering all +/// commit files. +#[test] +fn delta_only_snapshot_emits_expected_metrics() -> DeltaResult<()> { + let table = TestTableBuilder::new() + .with_log_state(LogState::with_latest_version(1)) + .with_data(1, 1) + .build()?; + + let (engine, reporter, _guard) = measuring_engine(table.store().clone()); + let _snap = Snapshot::builder_for(table.table_root()).build(&engine)?; + + assert_eq!(reporter.snapshot_completions.get(), 1); + assert_eq!(reporter.log_segment_loads.get(), 1); + assert_eq!(reporter.commit_files.get(), 2); + assert_eq!(reporter.checkpoint_files.get(), 0); + assert_eq!(reporter.compaction_files.get(), 0); + + // Both commit files read in a single JSON call + assert_eq!(reporter.json_read_calls.get(), 1); + assert_eq!(reporter.json_files_read.get(), 2); + assert_eq!(reporter.parquet_read_calls.get(), 0); + + // Exactly one listing covering exactly the 2 commit files + assert_eq!(reporter.list_calls.get(), 1); + assert_eq!(reporter.list_files_seen.get(), 2); + + Ok(()) +} + +// ============================================================================ +// Scenario 2: v1 parquet checkpoint + one tail commit +// ============================================================================ +// TODO: migrate to TestTableBuilder when checkpoint LogState variants land (#2284) + +/// After a v1 parquet checkpoint is written at version 1 and a further commit is added, +/// a fresh snapshot sees one checkpoint file, one tail commit, and performs a single +/// parquet read (checkpoint) plus a single JSON read (tail commit). +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn snapshot_with_v1_checkpoint_and_tail_commit_emits_expected_metrics() -> DeltaResult<()> { + let (table_url, setup_engine, _temp_dir) = setup_table_with_v1_checkpoint().await?; + + // commit 2: insert another row after the checkpoint + let snap2 = Snapshot::builder_for(table_url.clone()).build(setup_engine.as_ref())?; + let _ = insert_data( + snap2, + &setup_engine, + vec![Arc::new(Int32Array::from(vec![2]))], + ) + .await? + .unwrap_committed(); + + let (measure_engine, reporter, _guard) = measuring_engine(Arc::new(LocalFileSystem::new())); + let _snap = Snapshot::builder_for(table_url).build(&measure_engine)?; + + assert_eq!(reporter.snapshot_completions.get(), 1); + assert_eq!(reporter.log_segment_loads.get(), 1); + assert_eq!(reporter.commit_files.get(), 1); // only tail commit (v2) + assert_eq!(reporter.checkpoint_files.get(), 1); + assert_eq!(reporter.compaction_files.get(), 0); + + // One JSON read for the tail commit; one Parquet read for the checkpoint + assert_eq!(reporter.json_read_calls.get(), 1); + assert_eq!(reporter.json_files_read.get(), 1); + assert_eq!(reporter.parquet_read_calls.get(), 1); + assert_eq!(reporter.parquet_files_read.get(), 1); + // On Windows (NTFS), listing a file written moments earlier can return size=0 because + // the OS has not yet flushed size metadata to the directory entry. bytes_read is sourced + // from these FileMeta::size values, so it can be 0 even when real I/O occurred. + #[cfg(not(windows))] + assert!(reporter.json_bytes_read.get() > 0); + #[cfg(not(windows))] + assert!(reporter.parquet_bytes_read.get() > 0); + + Ok(()) +} + +// ============================================================================ +// Scenario 3: v1 parquet checkpoint at latest version (no tail commits) +// ============================================================================ +// TODO: migrate to TestTableBuilder when checkpoint LogState variants land (#2284) + +/// When the latest version has a checkpoint and no subsequent commits exist, the snapshot +/// has zero commit files and the JSON handler is called with an empty file list. +/// CommitReader always invokes read_json_files even for an empty commit cover. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn snapshot_at_checkpoint_tip_emits_expected_metrics() -> DeltaResult<()> { + let (table_url, _setup_engine, _temp_dir) = setup_table_with_v1_checkpoint().await?; + + let (measure_engine, reporter, _guard) = measuring_engine(Arc::new(LocalFileSystem::new())); + let _snap = Snapshot::builder_for(table_url).build(&measure_engine)?; + + assert_eq!(reporter.snapshot_completions.get(), 1); + assert_eq!(reporter.log_segment_loads.get(), 1); + assert_eq!(reporter.commit_files.get(), 0); + assert_eq!(reporter.checkpoint_files.get(), 1); + assert_eq!(reporter.compaction_files.get(), 0); + + // JSON handler is called with zero files; Parquet reads the checkpoint + assert_eq!(reporter.json_read_calls.get(), 1); + assert_eq!(reporter.json_files_read.get(), 0); + assert_eq!(reporter.parquet_read_calls.get(), 1); + assert_eq!(reporter.parquet_files_read.get(), 1); + + Ok(()) +} + +// ============================================================================ +// Scenario 4: log compaction covering early commits + one tail commit +// ============================================================================ + +// TODO: migrate to TestTableBuilder when compaction LogState variants land (#2284) + +/// When early commits are covered by a compacted log file, the snapshot reports both +/// the individual commit count and the compaction count. The JSON handler reads the +/// compaction file and the tail commit in a single call (the minimal cover). +// TODO(#2337): re-enable when log compaction is re-enabled +#[ignore = "log compaction is temporarily disabled (#2337)"] +#[tokio::test] +async fn snapshot_with_log_compaction_emits_expected_metrics() -> DeltaResult<()> { + let table = TestTableBuilder::new() + .with_log_state(LogState::with_latest_version(2)) + .with_schema(simple_schema()) + .with_data(1, 1) + .build()?; + let store = table.store().clone(); + let table_url = Url::parse(table.table_root()).unwrap(); + let setup_engine = Arc::new(DefaultEngineBuilder::new(store.clone()).build()); + + // Write a compacted log file covering versions 0-2 using the public API + let snap2 = Snapshot::builder_for(table.table_root()).build(setup_engine.as_ref())?; + let mut writer = snap2.log_compaction_writer(0, 2)?; + let compaction_url = writer.compaction_path().clone(); + let batches: Vec<_> = writer + .compaction_data(setup_engine.as_ref())? + .collect::>>()?; + let json_bytes = to_json_bytes(batches.into_iter().map(Ok))?; + let compaction_path = Path::from_url_path(compaction_url.path()) + .map_err(|e| delta_kernel::Error::generic(e.to_string()))?; + store + .put(&compaction_path, json_bytes.into()) + .await + .map_err(|e| delta_kernel::Error::generic(e.to_string()))?; + + // commit 3: tail commit after the compaction + insert_rows(&table_url, &setup_engine, 3, 1).await?; + + let (engine, reporter, _guard) = measuring_engine(store); + let _snap = Snapshot::builder_for(table.table_root()).build(&engine)?; + + assert_eq!(reporter.snapshot_completions.get(), 1); + assert_eq!(reporter.log_segment_loads.get(), 1); + // ascending_commit_files contains all 4 individual .json files (0, 1, 2, 3) + assert_eq!(reporter.commit_files.get(), 4); + assert_eq!(reporter.checkpoint_files.get(), 0); + assert_eq!(reporter.compaction_files.get(), 1); + + // find_commit_cover selects [3.json, 0.2.compacted.json] -- 2 JSON files, one read call + assert_eq!(reporter.json_read_calls.get(), 1); + assert_eq!(reporter.json_files_read.get(), 2); + assert_eq!(reporter.parquet_read_calls.get(), 0); + + Ok(()) +} + +// ============================================================================ +// Scenario 5: CRC fast-path bypasses JSON replay +// ============================================================================ +// TODO: migrate to TestTableBuilder when CRC LogState variants land (#2284) + +/// When a CRC file exists at the target snapshot version, Protocol+Metadata are loaded +/// directly from it, skipping all JSON log replay. The JSON handler is never called. +#[tokio::test] +async fn snapshot_with_crc_at_target_version_skips_json_replay() -> DeltaResult<()> { + // The crc-full golden table has commit 0 + a CRC file at version 0. + let path = std::fs::canonicalize(PathBuf::from("./tests/data/crc-full/")) + .map_err(|e| delta_kernel::Error::generic(e.to_string()))?; + let table_root = + Url::from_directory_path(path).map_err(|_| delta_kernel::Error::generic("invalid path"))?; + + let (engine, reporter, _guard) = measuring_engine(Arc::new(LocalFileSystem::new())); + let _snap = Snapshot::builder_for(table_root).build(&engine)?; + + assert_eq!(reporter.snapshot_completions.get(), 1); + assert_eq!(reporter.log_segment_loads.get(), 1); + assert_eq!(reporter.commit_files.get(), 1); + assert_eq!(reporter.checkpoint_files.get(), 0); + + // CRC at target version -- P+M loaded from CRC, no JSON or Parquet reads needed + assert_eq!(reporter.json_read_calls.get(), 0); + assert_eq!(reporter.parquet_read_calls.get(), 0); + // Two storage reads: (1) _last_checkpoint hint attempt (file absent for this table), + // (2) the CRC file itself. Both go through StorageHandler::read_files. + assert_eq!(reporter.storage_read_calls.get(), 2); + // Listing sees both the commit file and the CRC file + assert_eq!(reporter.list_calls.get(), 1); + assert_eq!(reporter.list_files_seen.get(), 2); + + Ok(()) +} + +// ============================================================================ +// Scenario 6: CRC older than the latest version +// ============================================================================ +// TODO: migrate to TestTableBuilder when CRC LogState variants land (#2284) + +/// A CRC older than the snapshot version: kernel roots Protocol & Metadata at the CRC and reads +/// only the commits above it (plus the CRC), never the first commit. Both modes: +/// - `Disabled` (default): the tail is replayed for P&M, falling back to the CRC's P&M; the stale +/// CRC is not advanced, so the snapshot stores no CRC. +/// - `Unlimited`: the same tail is reverse-replayed to advance the CRC to the target version, which +/// the snapshot stores. +/// +/// Reading only the tail (not the first commit) is the regression guard: a full-log replay under +/// `Disabled` would read v0 too. +/// +/// `write_checksum` needs stats from a post-commit snapshot, so this uses `create_table` -> +/// `post_commit_snapshot` -> `write_checksum` to write the CRC at v0. +#[rstest] +#[case::disabled(IncrementalReplay::Disabled, None)] +#[case::unlimited(IncrementalReplay::Unlimited, Some(2))] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn crc_at_prior_version_roots_replay_at_crc_for_both_modes( + #[case] mode: IncrementalReplay, + #[case] expected_crc_version: Option, +) -> DeltaResult<()> { + let (_temp_dir, table_path, setup_engine) = test_table_setup_mt()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + + // commit 0: create table, then write its CRC (write_checksum needs the post-commit CRC). + let create_committed = create_table(&table_path, simple_schema(), "Test/1.0") + .build(setup_engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(setup_engine.as_ref())? + .unwrap_committed(); + create_committed + .post_commit_snapshot() + .expect("post-commit snapshot") + .clone() + .write_checksum(setup_engine.as_ref())?; + + // commits 1 and 2: pure Add actions, no Protocol/Metadata changes + let mut snap = create_committed + .post_commit_snapshot() + .expect("post-commit snapshot") + .clone(); + for val in [1i32, 2] { + let c = insert_data( + snap, + &setup_engine, + vec![Arc::new(Int32Array::from(vec![val]))], + ) + .await? + .unwrap_committed(); + snap = c + .post_commit_snapshot() + .expect("post-commit snapshot") + .clone(); + } + + // Measurement: build the snapshot at latest (v2); the CRC is at v0, two versions behind. + let (measure_engine, reporter, _guard) = measuring_engine(Arc::new(LocalFileSystem::new())); + let snap = Snapshot::builder_for(table_url) + .with_incremental_crc_replay(mode) + .build(&measure_engine)?; + assert_eq!( + snap.crc_at_version().map(|c| c.version), + expected_crc_version + ); + + assert_eq!(reporter.snapshot_completions.get(), 1); + assert_eq!(reporter.log_segment_loads.get(), 1); + assert_eq!(reporter.commit_files.get(), 3); // v0, v1, v2 + + // Only the tail commits v1 and v2 (after the CRC at v0) are read, in one JSON call. + assert_eq!(reporter.json_read_calls.get(), 1); + assert_eq!(reporter.json_files_read.get(), 2); + // The base CRC at v0 is read from storage. + assert!(reporter.storage_read_calls.get() >= 1); + + Ok(()) +} + +// ============================================================================ +// Scenario 7: checkpoint behind latest version (with multiple tail commits) +// ============================================================================ +// TODO: migrate to TestTableBuilder when checkpoint LogState variants land (#2284) + +/// A checkpoint that is multiple versions behind the latest forces a longer tail replay. +/// Checkpoint at v1 plus 3 additional commits (v2, v3, v4) verifies that all tail commits +/// are read in a single JSON call and both byte counters are non-zero. The specific counts +/// here reflect this table's setup: checkpoint at v1, three tail commits. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn checkpoint_with_multiple_tail_commits_emits_expected_metrics() -> DeltaResult<()> { + let (table_url, setup_engine, _temp_dir) = setup_table_with_v1_checkpoint().await?; + + // commits 2, 3, 4: insert more data after the checkpoint + let mut snap = Snapshot::builder_for(table_url.clone()).build(setup_engine.as_ref())?; + for val in [2i32, 3, 4] { + let c = insert_data( + snap, + &setup_engine, + vec![Arc::new(Int32Array::from(vec![val]))], + ) + .await? + .unwrap_committed(); + snap = c + .post_commit_snapshot() + .expect("post-commit snapshot") + .clone(); + } + + let (measure_engine, reporter, _guard) = measuring_engine(Arc::new(LocalFileSystem::new())); + let _snap = Snapshot::builder_for(table_url).build(&measure_engine)?; + + assert_eq!(reporter.snapshot_completions.get(), 1); + assert_eq!(reporter.log_segment_loads.get(), 1); + // Checkpoint at v1 -- listing starts from v2; tail is v2, v3, v4 + assert_eq!(reporter.commit_files.get(), 3); + assert_eq!(reporter.checkpoint_files.get(), 1); + // All 3 tail commits read in a single JSON call; checkpoint in a single Parquet call + assert_eq!(reporter.json_read_calls.get(), 1); + assert_eq!(reporter.json_files_read.get(), 3); + assert_eq!(reporter.parquet_read_calls.get(), 1); + // See Scenario 2 comment: Windows NTFS may report stale size=0 for recently written files. + #[cfg(not(windows))] + assert!(reporter.json_bytes_read.get() > 0); + #[cfg(not(windows))] + assert!(reporter.parquet_bytes_read.get() > 0); + + Ok(()) +} + +// ============================================================================ +// Scenario 8: on-demand domain metadata query incurs additional log replay +// ============================================================================ + +/// `snapshot.get_domain_metadata()` always performs a full log replay when no CRC is +/// present at the target version. Calling it after a snapshot is already built generates +/// a second round of JSON reads, demonstrating that on-demand metadata queries carry +/// their own I/O cost. +/// +/// The specific count (`json_read_calls = 1`) reflects this test's table: 2 commits, no +/// checkpoint, no CRC. Tables with different log structures will produce different counts. +#[test] +fn get_domain_metadata_when_no_latest_crc_incurs_additional_log_replay() -> DeltaResult<()> { + let table = TestTableBuilder::new() + .with_log_state(LogState::with_latest_version(1)) + .with_data(1, 1) + .build()?; + + let (engine, reporter, _guard) = measuring_engine(table.store().clone()); + let snap = Snapshot::builder_for(table.table_root()).build(&engine)?; + + // Snapshot build reads both commit files in one JSON call + assert_eq!(reporter.json_read_calls.get(), 1); + + // Reset so the domain metadata query cost is isolated + reporter.reset(); + let _ = snap.get_domain_metadata("myapp.config", &engine)?; + + assert_eq!( + reporter.json_read_calls.get(), + 1, + "get_domain_metadata replays the log, incurring one additional JSON read call" + ); + assert!(reporter.json_files_read.get() > 0); + + Ok(()) +} + +// ============================================================================ +// Scenario 9: domain metadata and set-transaction loads +// ============================================================================ + +// Creates a table with clustering and rowTracking enabled (two system domain metadatas), plus a +// user domain metadata and a SetTransaction. +async fn setup_table_with_dms_and_set_txns( + write_crc: bool, +) -> DeltaResult<(Url, tempfile::TempDir)> { + let (temp_dir, table_path, engine) = test_table_setup()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + let committer = || Box::new(FileSystemCommitter::new()); + + let properties = HashMap::from([("delta.enableRowTracking".to_string(), "true".to_string())]); + let snap_v0 = create_table(&table_path, simple_schema(), "Test/1.0") + .with_table_properties(properties) + .with_data_layout(DataLayout::clustered(["id"])) + .build(engine.as_ref(), committer())? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + + let snap_v1 = snap_v0 + .transaction(committer(), engine.as_ref())? + .with_operation("WRITE".to_string()) + .with_domain_metadata("myapp.config".to_string(), "v1".to_string()) + .with_transaction_id("my-app".to_string(), 1) + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + + if write_crc { + snap_v1.write_checksum(engine.as_ref())?; + } + Ok((table_url, temp_dir)) +} + +#[rstest] +#[case::log_replay(false)] +#[case::crc_cache(true)] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn write_transaction_loads_domain_metadata_internally( + #[case] with_crc: bool, +) -> DeltaResult<()> { + let (table_url, _temp_dir) = setup_table_with_dms_and_set_txns(with_crc).await?; + + let (engine, reporter, _guard) = measuring_engine(Arc::new(LocalFileSystem::new())); + let engine = Arc::new(engine); + let snap = Snapshot::builder_for(table_url).build(engine.as_ref())?; + + // Isolate the transaction's internal loads from the snapshot build. + reporter.reset(); + insert_data(snap, &engine, vec![Arc::new(Int32Array::from(vec![1]))]) + .await? + .unwrap_post_commit_snapshot(); + + // Clustering domain (creating the transaction) + row-tracking domain (committing the append). + assert_eq!(reporter.domain_metadata_loads.get(), 2); + assert_eq!( + reporter.domain_metadata_cache_hits.get(), + 2 * with_crc as u64 + ); + assert_eq!(reporter.domain_metadata_domains_returned.get(), 2); + + Ok(()) +} + +#[rstest] +#[case::log_replay(false)] +#[case::crc_cache(true)] +#[tokio::test] +async fn get_app_id_version_load_emits_loaded_metric(#[case] with_crc: bool) -> DeltaResult<()> { + let (table_url, _temp_dir) = setup_table_with_dms_and_set_txns(with_crc).await?; + + let (engine, reporter, _guard) = measuring_engine(Arc::new(LocalFileSystem::new())); + let snap = Snapshot::builder_for(table_url).build(&engine)?; + + // Hit on an app id with a committed transaction. + reporter.reset(); + assert_eq!(snap.get_app_id_version("my-app", &engine)?, Some(1)); + assert_eq!(reporter.set_transaction_loads.get(), 1); + assert_eq!(reporter.set_transaction_cache_hits.get(), with_crc as u64); + assert_eq!(reporter.set_transaction_found.get(), 1); + + // Miss on an unknown app id. A Complete CRC answers the miss from cache; without a CRC it + // falls back to log replay. + reporter.reset(); + assert_eq!(snap.get_app_id_version("no-such-app", &engine)?, None); + assert_eq!(reporter.set_transaction_loads.get(), 1); + assert_eq!(reporter.set_transaction_cache_hits.get(), with_crc as u64); + assert_eq!(reporter.set_transaction_found.get(), 0); + + Ok(()) +} + +#[rstest] +#[case::log_replay(false)] +#[case::crc_cache(true)] +#[tokio::test] +async fn get_domain_metadata_load_emits_loaded_metric(#[case] with_crc: bool) -> DeltaResult<()> { + let (table_url, _temp_dir) = setup_table_with_dms_and_set_txns(with_crc).await?; + + let (engine, reporter, _guard) = measuring_engine(Arc::new(LocalFileSystem::new())); + let snap = Snapshot::builder_for(table_url).build(&engine)?; + + // Hit on the user domain. + reporter.reset(); + assert!(snap.get_domain_metadata("myapp.config", &engine)?.is_some()); + assert_eq!(reporter.domain_metadata_loads.get(), 1); + assert_eq!(reporter.domain_metadata_cache_hits.get(), with_crc as u64); + assert_eq!(reporter.domain_metadata_domains_returned.get(), 1); + + // Miss on a nonexistent domain. A Complete CRC answers the miss from cache; without a CRC it + // falls back to log replay. + reporter.reset(); + assert!(snap + .get_domain_metadata("does.not.exist", &engine)? + .is_none()); + assert_eq!(reporter.domain_metadata_loads.get(), 1); + assert_eq!(reporter.domain_metadata_cache_hits.get(), with_crc as u64); + assert_eq!(reporter.domain_metadata_domains_returned.get(), 0); + + Ok(()) +} + +#[tokio::test] +async fn failed_loads_emit_failure_metric() -> DeltaResult<()> { + let (table_url, _temp_dir) = setup_table_with_dms_and_set_txns(false).await?; + + let (engine, reporter, _guard) = measuring_engine(Arc::new(LocalFileSystem::new())); + let snap = Snapshot::builder_for(table_url.clone()).build(&engine)?; + + // Remove a commit so the log has a gap and the replay-based load fails. + let log_dir = table_url.to_file_path().unwrap().join("_delta_log"); + std::fs::remove_file(log_dir.join("00000000000000000001.json")).unwrap(); + + reporter.reset(); + assert!(snap.get_domain_metadata("myapp.config", &engine).is_err()); + assert!(snap.get_app_id_version("my-app", &engine).is_err()); + assert_eq!(reporter.domain_metadata_loads.get(), 0); + assert_eq!(reporter.set_transaction_loads.get(), 0); + assert_eq!(reporter.domain_metadata_load_failures.get(), 1); + assert_eq!(reporter.set_transaction_load_failures.get(), 1); + + Ok(()) +} diff --git a/kernel/tests/integration/metrics/table_type.rs b/kernel/tests/integration/metrics/table_type.rs new file mode 100644 index 0000000000..1979dc1c3b --- /dev/null +++ b/kernel/tests/integration/metrics/table_type.rs @@ -0,0 +1,233 @@ +//! Verifies that snapshot-load and scan metric events carry the correct [`TableType`] for both +//! path-based and catalog-managed tables, exercised end-to-end through public APIs +//! (`create_table`, `Transaction`, `Scan`). + +use std::collections::HashSet; +use std::sync::Arc; + +use delta_kernel::arrow::array::Int32Array; +use delta_kernel::committer::{Committer, FileSystemCommitter}; +use delta_kernel::metrics::{MetricEvent, TableType}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::{DeltaResult, Engine, Snapshot}; +use rstest::rstest; +use test_utils::{ + add_commit, engine_store_setup, insert_data_with, install_thread_local_metrics_reporter, + test_table_setup_mt, CapturingReporter, TestCatalogCommitter, +}; + +use super::simple_schema; + +#[rstest] +#[case::path_based(false)] +#[case::catalog_managed(true)] +#[tokio::test(flavor = "multi_thread")] +async fn test_metric_events_carry_table_type(#[case] catalog_managed: bool) -> DeltaResult<()> { + let (_tmp, table_path, engine) = test_table_setup_mt()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + + let build_snapshot = |max_catalog_version: u64| -> DeltaResult> { + let builder = Snapshot::builder_for(table_url.clone()); + let builder = if catalog_managed { + builder.with_max_catalog_version(max_catalog_version) + } else { + builder + }; + builder.build(engine.as_ref()) + }; + + // Given: a table (path-based or catalog-managed) with one appended row. + create_simple_table(engine.as_ref(), &table_path, catalog_managed)?; + insert_data_with( + build_snapshot(0)?, + &engine, + vec![Arc::new(Int32Array::from(vec![1]))], + make_committer(catalog_managed), + "WRITE", + /* data_change */ true, + /* is_blind_append */ false, + ) + .await? + .unwrap_committed(); + + // When: a fresh snapshot is built and scanned with a metrics reporter installed. + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + let snapshot = build_snapshot(1)?; + let scan = snapshot.scan_builder().build()?; + for batch in scan.scan_metadata(engine.as_ref())? { + batch?; + } + + // Then: every snapshot-load and scan event carries the expected table_type. + let expected = TableType::from_catalog_managed(catalog_managed); + assert_table_types( + &reporter.events(), + &[ + "LogSegmentLoadSuccess", + "ProtocolMetadataLoadSuccess", + "SnapshotBuildSuccess", + "ScanMetadataCompleted", + ], + expected, + ); + Ok(()) +} + +#[rstest] +#[case::path_based(false)] +#[case::catalog_managed(true)] +#[tokio::test(flavor = "multi_thread")] +async fn test_snapshot_failure_events_carry_table_type(#[case] catalog_managed: bool) { + let (_tmp, table_path, engine) = test_table_setup_mt().unwrap(); + let table_url = delta_kernel::try_parse_uri(&table_path).unwrap(); + + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let builder = Snapshot::builder_for(table_url); + let builder = if catalog_managed { + builder.with_max_catalog_version(0) + } else { + builder + }; + assert!(builder.build(engine.as_ref()).is_err()); + + let expected = TableType::from_catalog_managed(catalog_managed); + assert_table_types( + &reporter.events(), + &["LogSegmentLoadFailure", "SnapshotBuildFailure"], + expected, + ); +} + +#[rstest] +#[case::path_based(false)] +#[case::catalog_managed(true)] +#[tokio::test] +async fn test_protocol_metadata_load_failure_carries_table_type(#[case] catalog_managed: bool) { + let (store, engine, table_url) = engine_store_setup("pm_failure", None); + // An Add-only commit (no Protocol/Metadata) makes the protocol+metadata replay fail. + let add_only = r#"{"add":{"path":"f.parquet","partitionValues":{},"size":1,"modificationTime":1,"dataChange":true}}"#; + add_commit(table_url.as_str(), store.as_ref(), 0, add_only.to_string()) + .await + .unwrap(); + + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let builder = Snapshot::builder_for(table_url); + let builder = if catalog_managed { + builder.with_max_catalog_version(0) + } else { + builder + }; + assert!(builder.build(&engine).is_err()); + + let expected = TableType::from_catalog_managed(catalog_managed); + assert_table_types( + &reporter.events(), + &["ProtocolMetadataLoadFailure", "SnapshotBuildFailure"], + expected, + ); +} + +/// Validation fails after the protocol read succeeds, so events carry the requested mode (the +/// documented best guess), not the protocol truth. +#[rstest] +#[case::catalog_version_on_path_based(false, true, TableType::CatalogManaged)] +#[case::no_catalog_version_on_catalog_managed(true, false, TableType::PathBased)] +#[tokio::test(flavor = "multi_thread")] +async fn test_build_validation_mismatch_events_carry_requested_table_type( + #[case] create_catalog_managed: bool, + #[case] request_catalog_version: bool, + #[case] expected: TableType, +) -> DeltaResult<()> { + let (_tmp, table_path, engine) = test_table_setup_mt()?; + let table_url = delta_kernel::try_parse_uri(&table_path)?; + + create_simple_table(engine.as_ref(), &table_path, create_catalog_managed)?; + + let reporter = Arc::new(CapturingReporter::default()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + + let builder = Snapshot::builder_for(table_url); + let builder = if request_catalog_version { + builder.with_max_catalog_version(0) + } else { + builder + }; + assert!(builder.build(engine.as_ref()).is_err()); + + assert_table_types( + &reporter.events(), + &["ProtocolMetadataLoadSuccess", "SnapshotBuildFailure"], + expected, + ); + Ok(()) +} + +pub(super) fn make_committer(catalog_managed: bool) -> Box { + if catalog_managed { + Box::new(TestCatalogCommitter) + } else { + Box::new(FileSystemCommitter::new()) + } +} + +pub(super) fn create_simple_table( + engine: &dyn Engine, + table_path: &str, + catalog_managed: bool, +) -> DeltaResult<()> { + let builder = create_table(table_path, simple_schema(), "Test/1.0"); + let builder = if catalog_managed { + builder.with_table_properties([ + ("delta.feature.catalogManaged", "supported"), + ("delta.feature.vacuumProtocolCheck", "supported"), + ("io.unitycatalog.tableId", "metrics-table-type-test"), + ]) + } else { + builder + }; + builder + .build(engine, make_committer(catalog_managed))? + .commit(engine)? + .unwrap_committed(); + Ok(()) +} + +fn table_types(events: &[MetricEvent]) -> Vec<(&'static str, TableType)> { + events + .iter() + .filter_map(|e| match e { + MetricEvent::LogSegmentLoadSuccess(e) => Some(("LogSegmentLoadSuccess", e.table_type)), + MetricEvent::LogSegmentLoadFailure(e) => Some(("LogSegmentLoadFailure", e.table_type)), + MetricEvent::ProtocolMetadataLoadSuccess(e) => { + Some(("ProtocolMetadataLoadSuccess", e.table_type)) + } + MetricEvent::ProtocolMetadataLoadFailure(e) => { + Some(("ProtocolMetadataLoadFailure", e.table_type)) + } + MetricEvent::SnapshotBuildSuccess(e) => Some(("SnapshotBuildSuccess", e.table_type)), + MetricEvent::SnapshotBuildFailure(e) => Some(("SnapshotBuildFailure", e.table_type)), + MetricEvent::ScanMetadataCompleted(e) => Some(("ScanMetadataCompleted", e.table_type)), + _ => None, + }) + .collect() +} + +#[track_caller] +pub(super) fn assert_table_types(events: &[MetricEvent], required: &[&str], expected: TableType) { + let observed = table_types(events); + let kinds: HashSet<&str> = observed.iter().map(|(kind, _)| *kind).collect(); + for required in required { + assert!( + kinds.contains(required), + "missing {required} event; saw {kinds:?}" + ); + } + for (kind, table_type) in observed { + assert_eq!(table_type, expected, "{kind} carried the wrong table_type"); + } +} diff --git a/kernel/tests/integration/metrics_main.rs b/kernel/tests/integration/metrics_main.rs new file mode 100644 index 0000000000..48e2c3c984 --- /dev/null +++ b/kernel/tests/integration/metrics_main.rs @@ -0,0 +1,66 @@ +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::Arc; +use std::time::Duration; + +use delta_kernel::metrics::{MetricEvent, MetricsReporter, WithMetricsReporterLayer as _}; +use delta_kernel::Snapshot; +use test_utils::delta_kernel_default_engine::DefaultEngineBuilder; +use test_utils::table_builder::{LogState, TestTableBuilder}; +use tracing_subscriber::layer::SubscriberExt as _; + +#[derive(Debug, Default)] +struct NoopReporter; +impl MetricsReporter for NoopReporter { + fn report(&self, _: MetricEvent) {} +} + +fn main() { + let table = TestTableBuilder::new() + .with_log_state(LogState::with_latest_version(1)) + .with_data(1, 1) + .build() + .expect("build test table"); + let table_url = delta_kernel::try_parse_uri(table.table_root()).expect("parse table url"); + let engine = DefaultEngineBuilder::new(table.store().clone()).build(); + + let subscriber = tracing_subscriber::registry() + .with(tracing_subscriber::fmt::layer().with_ansi(false)) + .with_metrics_reporter_layer(Arc::new(NoopReporter)); + tracing::dispatcher::set_global_default(tracing::Dispatch::new(subscriber)).unwrap(); + + let done = Arc::new(AtomicBool::new(false)); + let watch = Arc::clone(&done); + std::thread::spawn(move || { + std::thread::sleep(Duration::from_secs(5)); + if !watch.load(Ordering::SeqCst) { + eprintln!("WATCHDOG: deadlock - aborting"); + std::process::abort(); + } + }); + + let _snap = Snapshot::builder_for(table_url).build(&engine); + + { + // trigger the invalid uuid warn + let _ = tracing::info_span!( + "snap.build", + operation_id = "not-a-uuid", + version = 0u64, + report = tracing::field::Empty, + ); + } + { + // trigger invalid field warn + let s = tracing::info_span!( + "snap.build", + operation_id = %uuid::Uuid::new_v4(), + version = 0u64, + report = tracing::field::Empty, + invalid_field = tracing::field::Empty, + ); + s.record("invalid_field", 42u64); + } + + done.store(true, Ordering::SeqCst); + println!("PASS"); +} diff --git a/kernel/tests/integration/parsed_partition_values.rs b/kernel/tests/integration/parsed_partition_values.rs new file mode 100644 index 0000000000..02e1afabe2 --- /dev/null +++ b/kernel/tests/integration/parsed_partition_values.rs @@ -0,0 +1,670 @@ +//! Integration tests for parsed partition-value output (`partitionValues_parsed`). + +use std::fs::File; +use std::sync::Arc; + +use delta_kernel::arrow::array::{ + Array, ArrayRef, BinaryArray, BooleanArray, Int32Array, RecordBatch, StringArray, StructArray, + TimestampMicrosecondArray, +}; +use delta_kernel::arrow::compute::filter_record_batch; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::expressions::{col, lit, ColumnName, Predicate}; +use delta_kernel::object_store::local::LocalFileSystem; +use delta_kernel::object_store::DynObjectStore; +use delta_kernel::parquet::arrow::ArrowWriter; +use delta_kernel::scan::state::ScanFile; +use delta_kernel::scan::PartitionValuesOptions; +use delta_kernel::table_features::{get_any_level_column_physical_name, ColumnMappingMode}; +use delta_kernel::Snapshot; +use rstest::rstest; +use test_utils::delta_kernel_default_engine::DefaultEngineBuilder; +use test_utils::table_builder::{partitioned, version_latest, FeatureSet, LogState, TableConfig}; +use test_utils::{ + add_commit, create_default_engine_mt_executor, get_column, + install_thread_local_metrics_reporter, test_context, CountingReporter, +}; +use url::Url; + +/// Requesting the typed struct via `with_struct()` on a partitioned table must emit a +/// `partitionValues_parsed` column with one field per partition column, keyed by physical name. +/// +/// Run across every column mapping mode and both log-segment shapes: +/// - `native_checkpoint = false`: no checkpoint, so the struct is synthesized from the +/// `partitionValues` string map via `MAP_TO_STRUCT` on a JSON commit. +/// - `native_checkpoint = true`: a checkpoint with `writeStatsAsStruct=true`, so the read takes the +/// checkpoint's native `partitionValues_parsed` column directly. For the non-null values this +/// table writes, that column matches what `MAP_TO_STRUCT` would reconstruct, so both sources +/// yield the same struct. +/// +/// The struct keys on physical names, so a logical-vs-physical mismatch under `Id`/`Name` mapping +/// would surface every partition value as null. The builder writes well-defined (non-null) +/// partition values, so the non-null assertion guards physical-name matching in all combinations. +#[rstest] +fn scan_metadata_emits_partition_values_parsed_across_column_mapping( + #[values( + ColumnMappingMode::None, + ColumnMappingMode::Id, + ColumnMappingMode::Name + )] + cm_mode: ColumnMappingMode, + #[values(false, true)] native_checkpoint: bool, +) { + let cm_str = match cm_mode { + ColumnMappingMode::None => "none", + ColumnMappingMode::Id => "id", + ColumnMappingMode::Name => "name", + }; + // A native `partitionValues_parsed` checkpoint column is only written when + // `writeStatsAsStruct=true`; otherwise the struct is synthesized from the string map on read. + let log_state = if native_checkpoint { + LogState::with_latest_version(1).with_checkpoint_at([1]) + } else { + LogState::with_latest_version(1) + }; + let table_config = if native_checkpoint { + TableConfig::new().write_stats_as_struct(true) + } else { + TableConfig::new() + }; + let (engine, snapshot, _table) = test_context!( + log_state, + FeatureSet::empty().column_mapping(cm_str), + partitioned(), + table_config, + version_latest(), + ); + + let schema = snapshot.schema(); + let scan = snapshot + .scan_builder() + .with_partition_values(PartitionValuesOptions::with_struct()) + .build() + .unwrap(); + + // Resolve the physical name kernel uses for a partition column under the active mapping mode. + let physical_name = |logical: &str| -> String { + get_any_level_column_physical_name(schema.as_ref(), &ColumnName::new([logical]), cm_mode) + .unwrap() + .into_inner() + .into_iter() + .next() + .unwrap() + }; + + let scan_metadata_results: Vec<_> = scan + .scan_metadata(&engine) + .unwrap() + .collect::, _>>() + .unwrap(); + + assert!( + !scan_metadata_results.is_empty(), + "Should have scan metadata" + ); + + let mut file_count = 0; + for scan_metadata in scan_metadata_results { + let (underlying_data, selection_vector) = scan_metadata.scan_files.into_parts(); + let batch: RecordBatch = ArrowEngineData::try_from_engine_data(underlying_data) + .unwrap() + .into(); + let filtered_batch = + filter_record_batch(&batch, &BooleanArray::from(selection_vector)).unwrap(); + if filtered_batch.num_rows() == 0 { + continue; + } + + let pv_parsed = get_column!(filtered_batch, "partitionValues_parsed", StructArray); + + // `partitioned()` partitions by all 13 primitive types in `partitioned_schema()`. + assert_eq!( + pv_parsed.num_columns(), + 13, + "expected one parsed field per partition column (cm={cm_str}, native_checkpoint={native_checkpoint})" + ); + + // Every partition value the builder writes is well-defined, so each field must be + // non-null. A logical-vs-physical name mismatch under column mapping would surface nulls. + for field in pv_parsed.columns() { + assert_eq!( + field.null_count(), + 0, + "partition value unexpectedly null (cm={cm_str}, native_checkpoint={native_checkpoint})" + ); + } + + // The struct must be keyed by physical name. Under Id/Name mapping the physical name + // differs from the logical name, so a logical-keyed struct would fail this lookup. + for logical in ["part_int", "part_string"] { + let phys = physical_name(logical); + assert!( + pv_parsed.column_by_name(&phys).is_some(), + "partitionValues_parsed should key {logical} by physical name {phys} \ + (cm={cm_str}, native_checkpoint={native_checkpoint})" + ); + } + + file_count += filtered_batch.num_rows(); + } + + assert_eq!(file_count, 1, "Should have processed exactly one file"); +} + +// === Foreign-writer literal empty-string partition values === +// +// The kernel never persists a literal "" partition value (it serializes its own empty and null +// partition values to JSON null on write), so these tests stand in a raw-JSON foreign writer that +// did, then assert the kernel reconstructs `partitionValues_parsed` with the empty-string cast: "" +// stays "" for string, becomes empty bytes for binary, and becomes null for every other type. + +/// Writes a foreign-writer table under `table_path`: protocol + metadata declaring string, binary, +/// and integer partition columns (with `writeStatsAsStruct` enabled so a checkpoint writes its own +/// `partitionValues_parsed` column), followed by one `add` commit holding `add_actions`. +/// Returns the table URL. +async fn write_foreign_partition_table( + table_path: &std::path::Path, + add_actions: &[String], +) -> Url { + std::fs::create_dir_all(table_path).unwrap(); + let url = Url::from_directory_path(table_path).unwrap(); + let table_root = url.to_string(); + let store: Arc = Arc::new(LocalFileSystem::new()); + + let schema_string = serde_json::json!({ + "type": "struct", + "fields": [ + {"name": "p_str", "type": "string", "nullable": true, "metadata": {}}, + {"name": "p_bin", "type": "binary", "nullable": true, "metadata": {}}, + {"name": "p_int", "type": "integer", "nullable": true, "metadata": {}}, + {"name": "value", "type": "integer", "nullable": true, "metadata": {}}, + ], + }) + .to_string(); + let protocol = r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#; + let metadata = serde_json::json!({ + "metaData": { + "id": "00000000-0000-0000-0000-000000000000", + "format": {"provider": "parquet", "options": {}}, + "schemaString": schema_string, + "partitionColumns": ["p_str", "p_bin", "p_int"], + "configuration": {"delta.checkpoint.writeStatsAsStruct": "true"}, + "createdTime": 1700000000000_i64, + }, + }) + .to_string(); + + add_commit( + &table_root, + store.as_ref(), + 0, + format!("{protocol}\n{metadata}"), + ) + .await + .unwrap(); + add_commit(&table_root, store.as_ref(), 1, add_actions.join("\n")) + .await + .unwrap(); + url +} + +/// Builds an `add` action whose `partitionValues` map holds the given raw strings. +fn add_action(path: &str, p_str: &str, p_bin: &str, p_int: &str) -> String { + serde_json::json!({ + "add": { + "path": path, + "partitionValues": {"p_str": p_str, "p_bin": p_bin, "p_int": p_int}, + "size": 100, + "modificationTime": 1700000000000_i64, + "dataChange": true, + "stats": "{\"numRecords\":1}", + }, + }) + .to_string() +} + +/// A foreign writer can persist a literal "" in the `partitionValues` map. On read, kernel +/// reconstructs `partitionValues_parsed` with the empty-string cast: "" stays "" for string, +/// becomes empty bytes for binary, and becomes null for every other type. +/// +/// The result is identical whether the value is reconstructed from the `partitionValues` map (JSON +/// commit) or read from a kernel-written checkpoint's native `partitionValues_parsed` column: the +/// checkpoint reconstructs that column with the same cast, so a checkpoint never changes the value +/// a scan surfaces. The `native_checkpoint` axis exercises both sources. +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn parsed_partition_values_read_foreign_empty_string( + #[values(false, true)] native_checkpoint: bool, +) { + let temp_dir = tempfile::tempdir().unwrap(); + let table_path = temp_dir.path().join("foreign-empty-string"); + let url = write_foreign_partition_table( + &table_path, + &[add_action( + "p_str=/p_bin=/p_int=/part-0.parquet", + "", + "", + "", + )], + ) + .await; + let engine = create_default_engine_mt_executor(&url).unwrap(); + + if native_checkpoint { + let snapshot = Snapshot::builder_for(url.clone()) + .build(engine.as_ref()) + .unwrap(); + snapshot.checkpoint(engine.as_ref(), None).unwrap(); + } + + // Confirm the scan reads from the intended source: the checkpoint axis must actually place a + // checkpoint in the snapshot's log segment (and the non-checkpoint axis must not), otherwise a + // silently-skipped checkpoint would re-test the JSON-commit path twice. + let reporter = Arc::new(CountingReporter::new()); + let _guard = install_thread_local_metrics_reporter(reporter.clone()); + let snapshot = Snapshot::builder_for(url.clone()) + .build(engine.as_ref()) + .unwrap(); + assert_eq!( + reporter.checkpoint_files.get(), + u64::from(native_checkpoint), + "log segment checkpoint parts must match native_checkpoint={native_checkpoint}" + ); + let scan = snapshot + .scan_builder() + .with_partition_values(PartitionValuesOptions::with_struct()) + .build() + .unwrap(); + + let mut asserted_rows = 0; + for scan_metadata in scan.scan_metadata(engine.as_ref()).unwrap() { + let (data, selection) = scan_metadata.unwrap().scan_files.into_parts(); + let batch: RecordBatch = ArrowEngineData::try_from_engine_data(data).unwrap().into(); + let batch = filter_record_batch(&batch, &BooleanArray::from(selection)).unwrap(); + let pv = get_column!(batch, "partitionValues_parsed", StructArray); + + let p_str = pv.column_by_name("p_str").unwrap(); + let p_str = p_str.as_any().downcast_ref::().unwrap(); + let p_bin = pv.column_by_name("p_bin").unwrap(); + let p_bin = p_bin.as_any().downcast_ref::().unwrap(); + let p_int = pv.column_by_name("p_int").unwrap(); + let p_int = p_int.as_any().downcast_ref::().unwrap(); + + // Identical whether read from the map (JSON commit) or the checkpoint's native column. + assert!(!p_str.is_null(0), "string \"\" reconstructs as \"\""); + assert_eq!(p_str.value(0), ""); + assert!(!p_bin.is_null(0), "binary \"\" reconstructs as empty bytes"); + assert_eq!(p_bin.value(0), b""); + assert!(p_int.is_null(0), "non-string \"\" must be null"); + + asserted_rows += batch.num_rows(); + } + assert_eq!( + asserted_rows, 1, + "expected exactly one file (native_checkpoint={native_checkpoint})" + ); +} + +fn collect_path(paths: &mut Vec, scan_file: ScanFile) { + paths.push(scan_file.path); +} + +/// A file whose partition value is a foreign literal "" is a real empty value, not null, so +/// partition skipping treats it accordingly: +/// - `p_str = ''` keeps it and `p_str = 'other'` prunes it. +/// - `p_str IS NULL` prunes it and `p_str IS NOT NULL` keeps it (the value is "", not null). +/// - the same holds for the binary column (`p_bin`), whose "" reconstructs as empty bytes. +/// +/// The `native_checkpoint` axis exercises that pruning is identical before and after a kernel +/// checkpoint: the checkpoint reconstructs the same "" into its native `partitionValues_parsed` +/// column, so skipping keeps and prunes the same files a scan of the JSON commit would. +#[rstest] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn empty_string_partition_pruning(#[values(false, true)] native_checkpoint: bool) { + let temp_dir = tempfile::tempdir().unwrap(); + let table_path = temp_dir.path().join("empty-string-pruning"); + let url = write_foreign_partition_table( + &table_path, + &[ + add_action("p_str=/empty.parquet", "", "", ""), + add_action("p_str=other/other.parquet", "other", "other", "7"), + ], + ) + .await; + let engine = create_default_engine_mt_executor(&url).unwrap(); + + if native_checkpoint { + let snapshot = Snapshot::builder_for(url.clone()) + .build(engine.as_ref()) + .unwrap(); + snapshot.checkpoint(engine.as_ref(), None).unwrap(); + } + + let surviving = |predicate: Predicate| -> Vec { + let snapshot = Snapshot::builder_for(url.clone()) + .build(engine.as_ref()) + .unwrap(); + let scan = snapshot + .scan_builder() + .with_predicate(Arc::new(predicate)) + .build() + .unwrap(); + let mut paths = Vec::new(); + for scan_metadata in scan.scan_metadata(engine.as_ref()).unwrap() { + paths = scan_metadata + .unwrap() + .visit_scan_files(paths, collect_path) + .unwrap(); + } + paths.sort(); + paths + }; + + let empty = "p_str=/empty.parquet".to_string(); + let other = "p_str=other/other.parquet".to_string(); + let both = vec![empty.clone(), other.clone()]; + + // The empty-string value is a real "", so equality and null predicates treat it as such. + assert_eq!( + surviving(Predicate::eq(col!("p_str"), lit(""))), + vec![empty.clone()], + "empty-string file must be kept under p_str = ''" + ); + assert_eq!( + surviving(Predicate::eq(col!("p_str"), lit("other"))), + vec![other.clone()], + "empty-string file must be pruned under p_str = 'other'" + ); + // Both partition values are non-null ("" and "other"), so IS NULL prunes both and IS NOT NULL + // keeps both. + assert!( + surviving(Predicate::is_null(col!("p_str"))).is_empty(), + "no file has a null p_str, so IS NULL prunes both (the empty file's value is \"\", not null)" + ); + assert_eq!( + surviving(Predicate::is_not_null(col!("p_str"))), + both, + "both files must be kept under p_str IS NOT NULL" + ); + + // The binary column reconstructs "" as empty bytes, pruned the same way. + assert_eq!( + surviving(Predicate::eq(col!("p_bin"), lit(b"other".as_slice()))), + vec![other.clone()], + "empty-bytes file must be pruned under p_bin = X'6f74686572'" + ); +} + +async fn write_timezone_partition_table(table_path: &std::path::Path, timestamp: &str) -> Url { + std::fs::create_dir_all(table_path).unwrap(); + let values: ArrayRef = Arc::new(Int32Array::from(vec![42])); + let batch = RecordBatch::try_from_iter([("value", values)]).unwrap(); + let mut writer = ArrowWriter::try_new( + File::create(table_path.join("part.parquet")).unwrap(), + batch.schema(), + None, + ) + .unwrap(); + writer.write(&batch).unwrap(); + writer.close().unwrap(); + let file_size = std::fs::metadata(table_path.join("part.parquet")) + .unwrap() + .len(); + let url = Url::from_directory_path(table_path).unwrap(); + let table_root = url.to_string(); + let store: Arc = Arc::new(LocalFileSystem::new()); + let schema_string = serde_json::json!({ + "type": "struct", + "fields": [ + {"name": "p_ts", "type": "timestamp", "nullable": true, "metadata": {}}, + {"name": "p_ntz", "type": "timestamp_ntz", "nullable": true, "metadata": {}}, + {"name": "p_int", "type": "integer", "nullable": true, "metadata": {}}, + {"name": "value", "type": "integer", "nullable": true, "metadata": {}}, + ], + }) + .to_string(); + let protocol = r#"{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["timestampNtz"],"writerFeatures":["timestampNtz"]}}"#; + let metadata = serde_json::json!({ + "metaData": { + "id": "00000000-0000-0000-0000-000000000000", + "format": {"provider": "parquet", "options": {}}, + "schemaString": schema_string, + "partitionColumns": ["p_ts", "p_ntz", "p_int"], + "configuration": {"delta.checkpoint.writeStatsAsStruct": "true"}, + "createdTime": 1700000000000_i64, + }, + }) + .to_string(); + add_commit( + &table_root, + store.as_ref(), + 0, + format!("{protocol}\n{metadata}"), + ) + .await + .unwrap(); + let add = serde_json::json!({ + "add": { + "path": "part.parquet", + "partitionValues": { + "p_ts": timestamp, + "p_ntz": "2024-01-15 12:30:45.123456", + "p_int": "7", + }, + "size": file_size, + "modificationTime": 1700000000000_i64, + "dataChange": true, + "stats": "{\"numRecords\":1}", + }, + }); + add_commit(&table_root, store.as_ref(), 1, add.to_string()) + .await + .unwrap(); + url +} + +#[rstest] +#[case::utc(None, "2024-01-15 12:30:45.123456", "2024-01-15T12:30:45.123456Z")] +#[case::los_angeles_winter( + Some("America/Los_Angeles"), + "2024-01-15 12:30:45.123456", + "2024-01-15T20:30:45.123456Z" +)] +#[case::los_angeles_summer( + Some("America/Los_Angeles"), + "2024-06-15 08:00:00.500500", + "2024-06-15T15:00:00.500500Z" +)] +#[case::fixed_offset_seconds( + Some("+12:45:30"), + "2024-01-15 12:30:45.123456", + "2024-01-14T23:45:15.123456Z" +)] +#[case::explicit_offset_wins( + Some("America/Los_Angeles"), + "2024-01-15 12:30:45+02:00", + "2024-01-15T10:30:45Z" +)] +#[case::dst_overlap_uses_earlier_instant( + Some("America/Los_Angeles"), + "2024-11-03 01:30:00", + "2024-11-03T08:30:00Z" +)] +#[case::dst_gap_uses_pre_transition_offset( + Some("America/Los_Angeles"), + "2024-03-10 02:30:00", + "2024-03-10T10:30:00Z" +)] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn timezone_aware_parsed_partition_values_across_json_and_native_checkpoint( + #[case] timestamp_timezone: Option<&str>, + #[case] raw_timestamp: &str, + #[case] expected_timestamp: &str, + #[values(false, true)] native_checkpoint: bool, +) { + let temp_dir = tempfile::tempdir().unwrap(); + let url = write_timezone_partition_table(temp_dir.path(), raw_timestamp).await; + let engine = create_default_engine_mt_executor(&url).unwrap(); + if native_checkpoint { + let snapshot = Snapshot::builder_for(url.clone()) + .build(engine.as_ref()) + .unwrap(); + snapshot.checkpoint(engine.as_ref(), None).unwrap(); + } + + let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); + let partition_values = timestamp_timezone + .map_or_else(PartitionValuesOptions::with_struct, |timezone| { + PartitionValuesOptions::with_struct().with_timestamp_timezone(timezone) + }); + let scan = snapshot + .scan_builder() + .with_partition_values(partition_values) + .build() + .unwrap(); + let mut rows = 0; + for metadata in scan.scan_metadata(engine.as_ref()).unwrap() { + let (data, selection) = metadata.unwrap().scan_files.into_parts(); + let batch: RecordBatch = ArrowEngineData::try_from_engine_data(data).unwrap().into(); + let batch = filter_record_batch(&batch, &BooleanArray::from(selection)).unwrap(); + if batch.num_rows() == 0 { + continue; + } + let parsed = get_column!(batch, "partitionValues_parsed", StructArray); + let timestamp = parsed + .column_by_name("p_ts") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + let timestamp_ntz = parsed + .column_by_name("p_ntz") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + let integer = parsed + .column_by_name("p_int") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!( + timestamp.value(0), + chrono::DateTime::parse_from_rfc3339(expected_timestamp) + .unwrap() + .timestamp_micros() + ); + assert_eq!( + timestamp_ntz.value(0), + chrono::DateTime::parse_from_rfc3339("2024-01-15T12:30:45.123456Z") + .unwrap() + .timestamp_micros(), + "reader timezone must not affect TIMESTAMP_NTZ" + ); + assert_eq!(integer.value(0), 7); + rows += batch.num_rows(); + } + assert_eq!(rows, 1); +} + +#[rstest] +#[case::reader_timezone( + "2024-01-15 12:30:45.123456", + "America/Los_Angeles", + "2024-01-15T20:30:45.123456Z" +)] +#[case::explicit_offset_wins( + "2024-01-15 12:30:45+02:00", + "America/Los_Angeles", + "2024-01-15T10:30:45Z" +)] +#[case::dst_overlap_uses_earlier_instant( + "2024-11-03 01:30:00", + "America/Los_Angeles", + "2024-11-03T08:30:00Z" +)] +#[case::dst_gap_uses_pre_transition_offset( + "2024-03-10 02:30:00", + "America/Los_Angeles", + "2024-03-10T10:30:00Z" +)] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn scan_execute_partition_row_transform_respects_reader_timezone( + #[case] raw_timestamp: &str, + #[case] timestamp_timezone: &str, + #[case] expected_timestamp: &str, +) { + let temp_dir = tempfile::tempdir().unwrap(); + let url = write_timezone_partition_table(temp_dir.path(), raw_timestamp).await; + let engine = create_default_engine_mt_executor(&url).unwrap(); + let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); + let scan = snapshot + .scan_builder() + .with_partition_values( + PartitionValuesOptions::string_map_only().with_timestamp_timezone(timestamp_timezone), + ) + .build() + .unwrap(); + + let batches: Vec = scan + .execute(engine) + .unwrap() + .map(|result| { + ArrowEngineData::try_from_engine_data(result.unwrap()) + .unwrap() + .into() + }) + .collect(); + assert_eq!(batches.len(), 1); + let batch = &batches[0]; + assert_eq!(batch.num_rows(), 1); + let timestamp = get_column!(batch, "p_ts", TimestampMicrosecondArray); + let timestamp_ntz = get_column!(batch, "p_ntz", TimestampMicrosecondArray); + let integer = get_column!(batch, "p_int", Int32Array); + assert_eq!( + timestamp.value(0), + chrono::DateTime::parse_from_rfc3339(expected_timestamp) + .unwrap() + .timestamp_micros() + ); + assert_eq!( + timestamp_ntz.value(0), + chrono::DateTime::parse_from_rfc3339("2024-01-15T12:30:45.123456Z") + .unwrap() + .timestamp_micros(), + "reader timezone must not affect TIMESTAMP_NTZ" + ); + assert_eq!(integer.value(0), 7); +} + +#[rstest] +#[case::invalid_zone("2024-01-15 12:30:45", "Not/AZone", "Not/AZone")] +#[case::invalid_timestamp("not a timestamp", "America/Los_Angeles", "not a timestamp")] +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn timezone_aware_parsed_partition_values_report_invalid_input( + #[case] raw_timestamp: &str, + #[case] timestamp_timezone: &str, + #[case] expected_error: &str, +) { + let temp_dir = tempfile::tempdir().unwrap(); + let url = write_timezone_partition_table(temp_dir.path(), raw_timestamp).await; + let engine = create_default_engine_mt_executor(&url).unwrap(); + let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); + let scan = snapshot + .scan_builder() + .with_partition_values( + PartitionValuesOptions::with_struct().with_timestamp_timezone(timestamp_timezone), + ) + .build() + .unwrap(); + let result = scan + .scan_metadata(engine.as_ref()) + .unwrap() + .next() + .expect("one metadata result"); + let Err(error) = result else { + panic!("invalid timestamp input must fail map parsing"); + }; + assert!(error.to_string().contains(expected_error), "{error}"); +} diff --git a/kernel/tests/integration/parsed_stats.rs b/kernel/tests/integration/parsed_stats.rs new file mode 100644 index 0000000000..941414f05f --- /dev/null +++ b/kernel/tests/integration/parsed_stats.rs @@ -0,0 +1,277 @@ +//! Integration tests for parsed-stats output. + +use delta_kernel::actions::{MAX_VALUES, MIN_VALUES, NULL_COUNT, NUM_RECORDS, STATS_PARSED}; +use delta_kernel::arrow::array::{ + Array, BooleanArray, Decimal128Array, Float32Array, Float64Array, Int16Array, Int32Array, + Int64Array, Int8Array, RecordBatch, StringArray, StructArray, +}; +use delta_kernel::arrow::compute::filter_record_batch; +use delta_kernel::arrow::datatypes::DataType as ArrowDataType; +use delta_kernel::arrow::util::display::{ArrayFormatter, FormatOptions}; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::scan::StatsOptions; +use delta_kernel::table_features::ColumnMappingMode; +use delta_kernel::Snapshot; +use rstest::rstest; +use test_utils::delta_kernel_default_engine::DefaultEngineBuilder; +use test_utils::table_builder::{unpartitioned, version_latest, FeatureSet, LogState, TableConfig}; +use test_utils::{get_column, test_context}; + +/// Validate that JSON stats object values match the corresponding parsed struct array. +/// +/// Panics on missing fields to surface regressions where the parsed-stats schema drops a column. +// TODO: cover interval columns once the default engine supports writing/stats for them. +fn assert_stats_struct_matches_json( + struct_array: &StructArray, + json_object: &serde_json::Map, + row_idx: usize, + field_path: &str, +) { + for (col_name, json_val) in json_object { + let path = format!("{field_path}.{col_name}"); + let col = struct_array + .column_by_name(col_name) + .unwrap_or_else(|| panic!("{path}: present in JSON but missing from parsed struct")); + if col.is_null(row_idx) { + assert!( + json_val.is_null(), + "{path}: parsed is null but JSON is {json_val:?} at row {row_idx}" + ); + continue; + } + match json_val { + serde_json::Value::Number(n) => { + if let Some(arr) = col.as_any().downcast_ref::() { + assert_eq!( + n.as_i64().unwrap(), + i64::from(arr.value(row_idx)), + "{path} mismatch at row {row_idx}" + ); + } else if let Some(arr) = col.as_any().downcast_ref::() { + assert_eq!( + n.as_i64().unwrap(), + i64::from(arr.value(row_idx)), + "{path} mismatch at row {row_idx}" + ); + } else if let Some(arr) = col.as_any().downcast_ref::() { + assert_eq!( + n.as_i64().unwrap(), + i64::from(arr.value(row_idx)), + "{path} mismatch at row {row_idx}" + ); + } else if let Some(arr) = col.as_any().downcast_ref::() { + assert_eq!( + n.as_i64().unwrap(), + arr.value(row_idx), + "{path} mismatch at row {row_idx}" + ); + } else if let Some(arr) = col.as_any().downcast_ref::() { + assert_eq!( + n.as_f64().unwrap(), + f64::from(arr.value(row_idx)), + "{path} mismatch at row {row_idx}" + ); + } else if let Some(arr) = col.as_any().downcast_ref::() { + assert_eq!( + n.as_f64().unwrap(), + arr.value(row_idx), + "{path} mismatch at row {row_idx}" + ); + } else if let Some(arr) = col.as_any().downcast_ref::() { + let ArrowDataType::Decimal128(_, scale) = arr.data_type() else { + unreachable!("Decimal128Array always has a Decimal128 data type") + }; + assert_eq!( + n.as_f64().unwrap(), + arr.value(row_idx) as f64 / 10f64.powi(i32::from(*scale)), + "{path} mismatch at row {row_idx}" + ); + } else { + panic!("{path}: expected numeric array, got {:?}", col.data_type()); + } + } + serde_json::Value::String(s) => { + let format_options = FormatOptions::default() + .with_display_error(true) + .with_timestamp_format(Some("%Y-%m-%dT%H:%M:%S%.3f")) + .with_timestamp_tz_format(Some("%Y-%m-%dT%H:%M:%S%.3fZ")); + let formatter = ArrayFormatter::try_new(col.as_ref(), &format_options) + .unwrap_or_else(|e| panic!("{path}: cannot build formatter: {e}")); + let actual = formatter.value(row_idx).to_string(); + assert_eq!(&actual, s, "{path} mismatch at row {row_idx}"); + } + serde_json::Value::Object(sub_obj) => { + let sub_struct = col + .as_any() + .downcast_ref::() + .unwrap_or_else(|| { + panic!("{path}: expected StructArray, got {:?}", col.data_type()) + }); + assert_stats_struct_matches_json(sub_struct, sub_obj, row_idx, &path); + } + serde_json::Value::Null => { + assert!( + col.is_null(row_idx), + "{path}: JSON is null but parsed is non-null at row {row_idx}" + ); + } + other => panic!("{path}: unsupported JSON variant {other:?} at row {row_idx}"), + } + } +} + +/// Builds a table with `delta.checkpoint.writeStatsAsStruct=true` and a nested schema, +/// then verifies the parsed-stats struct column matches the JSON `stats` string. +#[rstest] +fn scan_metadata_with_stats_columns_kernel_written( + #[values( + ColumnMappingMode::None, + ColumnMappingMode::Id, + ColumnMappingMode::Name + )] + cm_mode: ColumnMappingMode, +) { + let cm_str = match cm_mode { + ColumnMappingMode::None => "none", + ColumnMappingMode::Id => "id", + ColumnMappingMode::Name => "name", + }; + let (engine, snapshot, _table) = test_context!( + LogState::with_latest_version(1).with_checkpoint_at([1]), + FeatureSet::empty().column_mapping(cm_str), + unpartitioned(), + TableConfig::new().write_stats_as_struct(true), + version_latest(), + ); + + let scan = snapshot + .scan_builder() + .with_stats(StatsOptions::all()) + .build() + .unwrap(); + + let scan_metadata_results: Vec<_> = scan + .scan_metadata(&engine) + .unwrap() + .collect::, _>>() + .unwrap(); + + assert!( + !scan_metadata_results.is_empty(), + "Should have scan metadata" + ); + + let mut total_num_records: i64 = 0; + let mut file_count = 0; + + for scan_metadata in scan_metadata_results { + let (underlying_data, selection_vector) = scan_metadata.scan_files.into_parts(); + let batch: RecordBatch = ArrowEngineData::try_from_engine_data(underlying_data) + .unwrap() + .into(); + let filtered_batch = + filter_record_batch(&batch, &BooleanArray::from(selection_vector)).unwrap(); + + let stats_parsed = get_column!(filtered_batch, STATS_PARSED, StructArray); + let num_records = get_column!(stats_parsed, NUM_RECORDS, Int64Array); + let min_values = get_column!(stats_parsed, MIN_VALUES, StructArray); + let max_values = get_column!(stats_parsed, MAX_VALUES, StructArray); + let null_count = get_column!(stats_parsed, NULL_COUNT, StructArray); + let stats_json = get_column!(filtered_batch, "stats", StringArray); + + for i in 0..stats_json.len() { + if stats_parsed.is_null(i) || stats_json.is_null(i) { + continue; + } + + let json_stats: serde_json::Value = + serde_json::from_str(stats_json.value(i)).expect("stats JSON should be valid"); + + let json_num = json_stats + .get(NUM_RECORDS) + .and_then(|v| v.as_i64()) + .expect("stats JSON must contain numRecords"); + assert_eq!( + json_num, + num_records.value(i), + "numRecords mismatch at row {i}" + ); + + let min_obj = json_stats + .get(MIN_VALUES) + .and_then(|v| v.as_object()) + .expect("stats JSON must contain minValues object"); + assert_stats_struct_matches_json(min_values, min_obj, i, MIN_VALUES); + + let max_obj = json_stats + .get(MAX_VALUES) + .and_then(|v| v.as_object()) + .expect("stats JSON must contain maxValues object"); + assert_stats_struct_matches_json(max_values, max_obj, i, MAX_VALUES); + + let null_obj = json_stats + .get(NULL_COUNT) + .and_then(|v| v.as_object()) + .expect("stats JSON must contain nullCount object"); + assert_stats_struct_matches_json(null_count, null_obj, i, NULL_COUNT); + + total_num_records += num_records.value(i); + file_count += 1; + } + } + + // The builder writes one data commit (v=1; v=0 is create-table with no data) of one + // file with the default 10 rows. + assert_eq!(file_count, 1, "Should have processed exactly one file"); + assert_eq!(total_num_records, 10, "Should have exactly 10 numRecords"); +} + +#[test] +fn json_stats_truncate_timestamps_to_milliseconds() { + let (engine, snapshot, _table) = test_context!( + LogState::with_latest_version(1).with_checkpoint_at([1]), + FeatureSet::empty(), + unpartitioned(), + TableConfig::new().write_stats_as_struct(true), + version_latest(), + ); + + let scan = snapshot + .scan_builder() + .with_stats(StatsOptions::all()) + .build() + .unwrap(); + + let mut checked = 0; + for scan_metadata in scan.scan_metadata(&engine).unwrap() { + let (underlying_data, selection_vector) = scan_metadata.unwrap().scan_files.into_parts(); + let batch: RecordBatch = ArrowEngineData::try_from_engine_data(underlying_data) + .unwrap() + .into(); + let filtered_batch = + filter_record_batch(&batch, &BooleanArray::from(selection_vector)).unwrap(); + let stats_json = get_column!(filtered_batch, "stats", StringArray); + + for i in 0..filtered_batch.num_rows() { + let json: serde_json::Value = serde_json::from_str(stats_json.value(i)).unwrap(); + for bound in [MIN_VALUES, MAX_VALUES] { + for (col, rendered) in json[bound].as_object().unwrap() { + let Some(ts) = rendered.as_str().filter(|s| s.contains('T')) else { + continue; // not a timestamp column + }; + let fraction = ts + .trim_end_matches('Z') + .rsplit_once('.') + .unwrap_or_else(|| panic!("{bound}.{col} = {ts:?} has no fraction")) + .1; + assert_eq!(fraction.len(), 3, "{bound}.{col} = {ts:?}"); + // The builder's values end in .298677, which floors to .298 rather than + // rounding to .299. + assert!(ts.ends_with(".298Z") || ts.ends_with(".298"), "{ts:?}"); + checked += 1; + } + } + } + } + assert!(checked > 0, "no timestamp stats were checked"); +} diff --git a/kernel/tests/integration/read/mod.rs b/kernel/tests/integration/read/mod.rs new file mode 100644 index 0000000000..d93cf06462 --- /dev/null +++ b/kernel/tests/integration/read/mod.rs @@ -0,0 +1,3153 @@ +// Scan-metadata cancellation is a read-path concern, so its coverage lives alongside the other +// read tests. +mod scan_cancellation; + +use std::path::PathBuf; +use std::sync::Arc; +use std::vec; + +use delta_kernel::actions::deletion_vector::split_vector; +use delta_kernel::arrow::array::{ArrayRef, AsArray as _, RecordBatch, TimestampMicrosecondArray}; +use delta_kernel::arrow::compute::{concat_batches, filter_record_batch}; +use delta_kernel::arrow::datatypes::{ + DataType as ArrowDataType, Field as ArrowField, Int64Type, Schema as ArrowSchema, TimeUnit, +}; +use delta_kernel::engine::arrow_conversion::TryFromKernel as _; +use delta_kernel::engine::arrow_data::EngineDataArrowExt as _; +use delta_kernel::expressions::{ + col, column_pred, lit, Expression as Expr, ExpressionRef, Predicate as Pred, PredicateRef, + Scalar, +}; +use delta_kernel::log_segment::LogSegment; +use delta_kernel::object_store::memory::InMemory; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::ObjectStoreExt as _; +use delta_kernel::parquet::file::properties::{EnabledStatistics, WriterProperties}; +use delta_kernel::path::ParsedLogPath; +use delta_kernel::scan::state::{transform_to_logical, ScanFile}; +use delta_kernel::scan::{Scan, StatsOptions}; +use delta_kernel::schema::{DataType, MetadataColumnSpec, Schema, StructField, StructType}; +use delta_kernel::{Engine, FileMeta, Snapshot}; +use itertools::Itertools; +use test_utils::delta_kernel_default_engine::DefaultEngineBuilder; +use test_utils::{ + actions_to_string, add_commit, generate_batch, generate_simple_batch, into_record_batch, + load_test_data, read_scan, record_batch_to_bytes, record_batch_to_bytes_with_props, IntoArray, + TestAction, METADATA, +}; +use url::Url; + +const PARQUET_FILE1: &str = "part-00000-a72b1fb3-f2df-41fe-a8f0-e65b746382dd-c000.snappy.parquet"; +const PARQUET_FILE2: &str = "part-00001-c506e79a-0bf8-4e2b-a42b-9731b2e490ae-c000.snappy.parquet"; +const PARQUET_FILE3: &str = "part-00002-c506e79a-0bf8-4e2b-a42b-9731b2e490ff-c000.snappy.parquet"; + +/// Convert all top-level fields in a RecordBatch to nullable, matching Delta table schema +/// conventions where the table metadata declares columns as nullable. +fn make_top_level_fields_nullable(batch: &RecordBatch) -> RecordBatch { + let schema = Arc::new(ArrowSchema::new( + batch + .schema() + .fields() + .iter() + .map(|f| ArrowField::new(f.name(), f.data_type().clone(), true)) + .collect::>(), + )); + RecordBatch::try_new(schema, batch.columns().to_vec()).unwrap() +} + +#[tokio::test] +async fn single_commit_two_add_files() -> Result<(), Box> { + let batch = generate_simple_batch()?; + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///"; + let parquet_bytes = record_batch_to_bytes(&batch); + let file_size = parquet_bytes.len() as u64; + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![ + TestAction::Metadata, + TestAction::AddWithSize(PARQUET_FILE1.to_string(), file_size), + TestAction::AddWithSize(PARQUET_FILE2.to_string(), file_size), + ]), + ) + .await?; + storage + .put( + &Path::from(PARQUET_FILE1), + record_batch_to_bytes(&batch).into(), + ) + .await?; + storage + .put( + &Path::from(PARQUET_FILE2), + record_batch_to_bytes(&batch).into(), + ) + .await?; + + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + + let expected = make_top_level_fields_nullable(&batch); + let expected_data = vec![expected.clone(), expected]; + + let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; + let scan = snapshot.scan_builder().build()?; + + let mut files = 0; + let stream = scan.execute(engine)?.zip(expected_data); + + for (data, expected) in stream { + files += 1; + assert_eq!(into_record_batch(data?), expected); + } + assert_eq!(2, files, "Expected to have scanned two files"); + Ok(()) +} + +#[tokio::test] +async fn two_commits() -> Result<(), Box> { + let batch = generate_simple_batch()?; + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///"; + let parquet_bytes = record_batch_to_bytes(&batch); + let file_size = parquet_bytes.len() as u64; + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![ + TestAction::Metadata, + TestAction::AddWithSize(PARQUET_FILE1.to_string(), file_size), + ]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 1, + actions_to_string(vec![TestAction::AddWithSize( + PARQUET_FILE2.to_string(), + file_size, + )]), + ) + .await?; + storage + .put( + &Path::from(PARQUET_FILE1), + record_batch_to_bytes(&batch).into(), + ) + .await?; + storage + .put( + &Path::from(PARQUET_FILE2), + record_batch_to_bytes(&batch).into(), + ) + .await?; + + let engine = DefaultEngineBuilder::new(storage.clone()).build(); + + let expected = make_top_level_fields_nullable(&batch); + let expected_data = vec![expected.clone(), expected]; + + let snapshot = Snapshot::builder_for(table_root).build(&engine)?; + let scan = snapshot.scan_builder().build()?; + + let mut files = 0; + let stream = scan.execute(Arc::new(engine))?.zip(expected_data); + + for (data, expected) in stream { + files += 1; + assert_eq!(into_record_batch(data?), expected); + } + assert_eq!(2, files, "Expected to have scanned two files"); + + Ok(()) +} + +#[tokio::test] +async fn remove_action() -> Result<(), Box> { + let batch = generate_simple_batch()?; + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///"; + let parquet_bytes = record_batch_to_bytes(&batch); + let file_size = parquet_bytes.len() as u64; + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![ + TestAction::Metadata, + TestAction::AddWithSize(PARQUET_FILE1.to_string(), file_size), + ]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 1, + actions_to_string(vec![TestAction::AddWithSize( + PARQUET_FILE2.to_string(), + file_size, + )]), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 2, + actions_to_string(vec![TestAction::RemoveWithSize( + PARQUET_FILE2.to_string(), + file_size, + )]), + ) + .await?; + storage + .put( + &Path::from(PARQUET_FILE1), + record_batch_to_bytes(&batch).into(), + ) + .await?; + + let engine = DefaultEngineBuilder::new(storage.clone()).build(); + + let expected = make_top_level_fields_nullable(&batch); + let expected_data = vec![expected]; + + let snapshot = Snapshot::builder_for(table_root).build(&engine)?; + let scan = snapshot.scan_builder().build()?; + + let stream = scan.execute(Arc::new(engine))?.zip(expected_data); + + let mut files = 0; + for (data, expected) in stream { + files += 1; + assert_eq!(into_record_batch(data?), expected); + } + assert_eq!(1, files, "Expected to have scanned one file"); + Ok(()) +} + +#[tokio::test] +async fn stats() -> Result<(), Box> { + fn generate_commit2(actions: Vec) -> String { + actions + .into_iter() + .map(|test_action| match test_action { + TestAction::Add(path) => format!(r#"{{"{action}":{{"path":"{path}","partitionValues":{{}},"size":262,"modificationTime":1587968586000,"dataChange":true, "stats":"{{\"numRecords\":2,\"nullCount\":{{\"id\":0}},\"minValues\":{{\"id\": 5}},\"maxValues\":{{\"id\":7}}}}"}}}}"#, action = "add", path = path), + TestAction::Remove(path) => format!(r#"{{"{action}":{{"path":"{path}","partitionValues":{{}},"size":262,"modificationTime":1587968586000,"dataChange":true}}}}"#, action = "remove", path = path), + TestAction::Metadata => METADATA.into(), + TestAction::AddWithSize(path, size) => format!(r#"{{"{action}":{{"path":"{path}","partitionValues":{{}},"size":{size},"modificationTime":1587968586000,"dataChange":true, "stats":"{{\"numRecords\":2,\"nullCount\":{{\"id\":0}},\"minValues\":{{\"id\": 5}},\"maxValues\":{{\"id\":7}}}}"}}}}"#, action = "add", path = path), + TestAction::RemoveWithSize(path, size) => format!(r#"{{"{action}":{{"path":"{path}","partitionValues":{{}},"size":{size},"modificationTime":1587968586000,"dataChange":true}}}}"#, action = "remove", path = path), + }) + .fold(String::new(), |a, b| a + &b + "\n") + } + + let batch1 = make_top_level_fields_nullable(&generate_simple_batch()?); + let batch2 = make_top_level_fields_nullable(&generate_batch(vec![ + ("id", vec![5, 7].into_arrow_array()), + ("val", vec!["e", "g"].into_arrow_array()), + ])?); + let file_size1 = record_batch_to_bytes(&batch1).len() as u64; + let file_size2 = record_batch_to_bytes(&batch2).len() as u64; + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///"; + // valid commit with min/max (0, 2) + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![ + TestAction::Metadata, + TestAction::AddWithSize(PARQUET_FILE1.to_string(), file_size1), + ]), + ) + .await?; + // storage.add_commit(1, &format!("{}\n", + // r#"{{"add":{{"path":"doesnotexist","partitionValues":{{}},"size":262,"modificationTime": + // 1587968586000,"dataChange":true, + // "stats":"{{\"numRecords\":2,\"nullCount\":{{\"id\":0}},\"minValues\":{{\"id\": + // 0}},\"maxValues\":{{\"id\":2}}}}"}}}}"#)); + add_commit( + table_root, + storage.as_ref(), + 1, + generate_commit2(vec![TestAction::AddWithSize( + PARQUET_FILE2.to_string(), + file_size2, + )]), + ) + .await?; + + storage + .put( + &Path::from(PARQUET_FILE1), + record_batch_to_bytes(&batch1).into(), + ) + .await?; + + storage + .put( + &Path::from(PARQUET_FILE2), + record_batch_to_bytes(&batch2).into(), + ) + .await?; + + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; + + // The first file has id between 1 and 3; the second has id between 5 and 7. For each operator, + // we validate the boundary values where we expect the set of matched files to change. + // + // NOTE: For cases that match both batch1 and batch2, we list batch2 first because log replay + // returns most recently added files first. + #[allow(clippy::type_complexity)] // otherwise it's even more complex because no `_` + let test_cases: Vec<(fn(Expr, Expr) -> _, _, _)> = vec![ + (Pred::eq, 0i32, vec![]), + (Pred::eq, 1, vec![&batch1]), + (Pred::eq, 3, vec![&batch1]), + (Pred::eq, 4, vec![]), + (Pred::eq, 5, vec![&batch2]), + (Pred::eq, 7, vec![&batch2]), + (Pred::eq, 8, vec![]), + (Pred::lt, 1, vec![]), + (Pred::lt, 2, vec![&batch1]), + (Pred::lt, 5, vec![&batch1]), + (Pred::lt, 6, vec![&batch2, &batch1]), + (Pred::le, 0, vec![]), + (Pred::le, 1, vec![&batch1]), + (Pred::le, 4, vec![&batch1]), + (Pred::le, 5, vec![&batch2, &batch1]), + (Pred::gt, 2, vec![&batch2, &batch1]), + (Pred::gt, 3, vec![&batch2]), + (Pred::gt, 6, vec![&batch2]), + (Pred::gt, 7, vec![]), + (Pred::ge, 3, vec![&batch2, &batch1]), + (Pred::ge, 4, vec![&batch2]), + (Pred::ge, 7, vec![&batch2]), + (Pred::ge, 8, vec![]), + (Pred::ne, 0, vec![&batch2, &batch1]), + (Pred::ne, 1, vec![&batch2, &batch1]), + (Pred::ne, 3, vec![&batch2, &batch1]), + (Pred::ne, 4, vec![&batch2, &batch1]), + (Pred::ne, 5, vec![&batch2, &batch1]), + (Pred::ne, 7, vec![&batch2, &batch1]), + (Pred::ne, 8, vec![&batch2, &batch1]), + ]; + for (pred_fn, value, expected_batches) in test_cases { + let predicate = pred_fn(col!("id"), lit(value)); + let scan = snapshot + .clone() + .scan_builder() + .with_predicate(Arc::new(predicate.clone())) + .build()?; + + let expected_files = expected_batches.len(); + let mut files_scanned = 0; + let stream = scan.execute(engine.clone())?.zip(expected_batches); + + for (batch, expected) in stream { + files_scanned += 1; + assert_eq!(into_record_batch(batch?), expected.clone()); + } + assert_eq!(expected_files, files_scanned, "{predicate:?}"); + } + Ok(()) +} + +fn read_with_execute( + engine: Arc, + scan: &Scan, + expected: &[String], +) -> Result<(), Box> { + let result_schema = Arc::new(ArrowSchema::try_from_kernel( + scan.logical_schema().as_ref(), + )?); + let batches = read_scan(scan, engine)?; + + if expected.is_empty() { + assert_eq!(batches.len(), 0); + } else { + let batch = concat_batches(&result_schema, &batches)?; + assert_batches_sorted_eq!(expected, &[batch]); + } + Ok(()) +} + +fn scan_metadata_callback(batches: &mut Vec, scan_file: ScanFile) { + batches.push(scan_file); +} + +fn read_with_scan_metadata( + location: &Url, + engine: &dyn Engine, + scan: &Scan, + expected: &[String], +) -> Result<(), Box> { + let result_schema = Arc::new(ArrowSchema::try_from_kernel( + scan.logical_schema().as_ref(), + )?); + let scan_metadata = scan.scan_metadata(engine)?; + let mut scan_files = vec![]; + for res in scan_metadata { + let scan_metadata = res?; + scan_files = scan_metadata.visit_scan_files(scan_files, scan_metadata_callback)?; + } + + let mut batches = vec![]; + for scan_file in scan_files.into_iter() { + let file_path = location.join(&scan_file.path)?; + let mut selection_vector = scan_file + .dv_info + .get_selection_vector(engine, location) + .unwrap(); + let meta = FileMeta { + last_modified: 0, + size: scan_file.size.try_into().unwrap(), + location: file_path, + }; + let read_results = engine + .parquet_handler() + .read_parquet_files( + &[meta], + scan.physical_schema().clone(), + scan.physical_predicate().clone(), + ) + .unwrap(); + + for read_result in read_results { + let read_result = read_result.unwrap(); + let len = read_result.len(); + // to transform the physical data into the correct logical form + let logical = transform_to_logical( + engine, + read_result, + scan.physical_schema(), + scan.logical_schema(), + scan_file.transform.clone(), + ) + .unwrap(); + let record_batch = logical.try_into_record_batch()?; + let rest = split_vector(selection_vector.as_mut(), len, Some(true)); + let batch = if let Some(mask) = selection_vector.clone() { + // apply the selection vector + filter_record_batch(&record_batch, &mask.into()).unwrap() + } else { + record_batch + }; + selection_vector = rest; + batches.push(batch); + } + } + + if expected.is_empty() { + assert_eq!(batches.len(), 0); + } else { + let batch = concat_batches(&result_schema, &batches)?; + assert_batches_sorted_eq!(expected, &[batch]); + } + Ok(()) +} + +fn read_table_data( + path: &str, + select_cols: Option<&[&str]>, + predicate: Option, + mut expected: Vec, +) -> Result<(), Box> { + let path = std::fs::canonicalize(PathBuf::from(path))?; + let predicate = predicate.map(Arc::new); + let url = url::Url::from_directory_path(path).unwrap(); + let engine = test_utils::create_default_engine(&url)?; + + let snapshot = Snapshot::builder_for(url.clone()).build(engine.as_ref())?; + + let read_schema = select_cols.map(|select_cols| { + let table_schema = snapshot.schema(); + let selected_fields = select_cols + .iter() + .map(|col| table_schema.field(col).cloned().unwrap()); + Arc::new(Schema::new_unchecked(selected_fields)) + }); + println!("Read {url:?} with schema {read_schema:#?} and predicate {predicate:#?}"); + let scan = snapshot + .scan_builder() + .with_schema_opt(read_schema) + .with_predicate(predicate.clone()) + .build()?; + + sort_lines!(expected); + read_with_scan_metadata(&url, engine.as_ref(), &scan, &expected)?; + read_with_execute(engine, &scan, &expected)?; + Ok(()) +} + +// util to take a Vec<&str> and call read_table_data with Vec +fn read_table_data_str( + path: &str, + select_cols: Option<&[&str]>, + predicate: Option, + expected: Vec<&str>, +) -> Result<(), Box> { + read_table_data( + path, + select_cols, + predicate, + expected.into_iter().map(String::from).collect(), + ) +} + +#[test] +fn data() -> Result<(), Box> { + let expected = vec![ + "+--------+--------+---------+", + "| letter | number | a_float |", + "+--------+--------+---------+", + "| | 6 | 6.6 |", + "| a | 1 | 1.1 |", + "| a | 4 | 4.4 |", + "| b | 2 | 2.2 |", + "| c | 3 | 3.3 |", + "| e | 5 | 5.5 |", + "+--------+--------+---------+", + ]; + read_table_data_str("./tests/data/basic_partitioned", None, None, expected)?; + + Ok(()) +} + +#[test] +fn column_ordering() -> Result<(), Box> { + let expected = vec![ + "+---------+--------+--------+", + "| a_float | letter | number |", + "+---------+--------+--------+", + "| 6.6 | | 6 |", + "| 4.4 | a | 4 |", + "| 5.5 | e | 5 |", + "| 1.1 | a | 1 |", + "| 2.2 | b | 2 |", + "| 3.3 | c | 3 |", + "+---------+--------+--------+", + ]; + read_table_data_str( + "./tests/data/basic_partitioned", + Some(&["a_float", "letter", "number"]), + None, + expected, + )?; + + Ok(()) +} + +#[test] +fn column_ordering_and_projection() -> Result<(), Box> { + let expected = vec![ + "+---------+--------+", + "| a_float | number |", + "+---------+--------+", + "| 6.6 | 6 |", + "| 4.4 | 4 |", + "| 5.5 | 5 |", + "| 1.1 | 1 |", + "| 2.2 | 2 |", + "| 3.3 | 3 |", + "+---------+--------+", + ]; + read_table_data_str( + "./tests/data/basic_partitioned", + Some(&["a_float", "number"]), + None, + expected, + )?; + + Ok(()) +} + +// get the basic_partitioned table for a set of expected numbers +fn table_for_numbers(nums: Vec) -> Vec { + let mut res: Vec = vec![ + "+---------+--------+", + "| a_float | number |", + "+---------+--------+", + ] + .into_iter() + .map(String::from) + .collect(); + for num in nums.iter() { + res.push(format!("| {num}.{num} | {num} |")); + } + res.push("+---------+--------+".to_string()); + res +} + +// get the basic_partitioned table for a set of expected letters +fn table_for_letters(letters: &[char]) -> Vec { + let mut res: Vec = vec![ + "+--------+--------+", + "| letter | number |", + "+--------+--------+", + ] + .into_iter() + .map(String::from) + .collect(); + let rows = vec![(1, 'a'), (2, 'b'), (3, 'c'), (4, 'a'), (5, 'e')]; + for (num, letter) in rows { + if letters.contains(&letter) { + res.push(format!("| {letter} | {num} |")); + } + } + res.push("+--------+--------+".to_string()); + res +} + +#[rstest::rstest] +#[case::less_than( + col!("number").lt(lit(4i64)), + table_for_numbers(vec![1, 2, 3]) +)] +#[case::less_than_or_equal( + col!("number").le(lit(4i64)), + table_for_numbers(vec![1, 2, 3, 4]) +)] +#[case::greater_than( + col!("number").gt(lit(4i64)), + table_for_numbers(vec![5, 6]) +)] +#[case::greater_than_or_equal( + col!("number").ge(lit(4i64)), + table_for_numbers(vec![4, 5, 6]) +)] +#[case::equal( + col!("number").eq(lit(4i64)), + table_for_numbers(vec![4]) +)] +#[case::not_equal( + col!("number").ne(lit(4i64)), + table_for_numbers(vec![1, 2, 3, 5, 6]) +)] +fn predicate_on_number( + #[case] pred: Pred, + #[case] expected: Vec, +) -> Result<(), Box> { + read_table_data( + "./tests/data/basic_partitioned", + Some(&["a_float", "number"]), + Some(pred), + expected, + )?; + Ok(()) +} + +#[rstest::rstest] +#[case::is_null( + col!("letter").is_null(), + vec![ + "+--------+--------+", + "| letter | number |", + "+--------+--------+", + "| | 6 |", + "+--------+--------+", + ] + .into_iter() + .map(String::from) + .collect() +)] +#[case::is_not_null( + col!("letter").is_not_null(), + table_for_letters(&['a', 'b', 'c', 'e']) +)] +#[case::less_than( + col!("letter").lt(lit("c")), + table_for_letters(&['a', 'b']) +)] +#[case::less_than_or_equal( + col!("letter").le(lit("c")), + table_for_letters(&['a', 'b', 'c']) +)] +#[case::greater_than( + col!("letter").gt(lit("c")), + table_for_letters(&['e']) +)] +#[case::greater_than_or_equal( + col!("letter").ge(lit("c")), + table_for_letters(&['c', 'e']) +)] +#[case::equal( + col!("letter").eq(lit("c")), + table_for_letters(&['c']) +)] +#[case::not_equal( + col!("letter").ne(lit("c")), + table_for_letters(&['a', 'b', 'e']) +)] +fn predicate_on_letter( + #[case] pred: Pred, + #[case] expected: Vec, +) -> Result<(), Box> { + // Test basic column pruning. Note that the actual predicate machinery is already well-tested, + // so we're just testing wiring here. + read_table_data( + "./tests/data/basic_partitioned", + Some(&["letter", "number"]), + Some(pred), + expected, + )?; + Ok(()) +} + +#[rstest::rstest] +#[case::or_with_pruning( + Pred::or( + col!("letter").gt(lit("a")), + col!("number").gt(lit(3i64)), + ), + // Unified data skipping evaluates partition + data predicates in a single pass. + // File a/1 (letter='a', max(number)=1): OR('a'>'a', 1>3) = FALSE -> pruned + vec![ + "+--------+--------+", + "| letter | number |", + "+--------+--------+", + "| | 6 |", + "| a | 4 |", + "| b | 2 |", + "| c | 3 |", + "| e | 5 |", + "+--------+--------+", + ] + .into_iter() + .map(String::from) + .collect() +)] +#[case::and_with_pruning( + Pred::and( + col!("letter").gt(lit("a")), // numbers 2, 3, 5 + col!("number").gt(lit(3i64)), // letters a, e + ), + table_for_letters(&['e']) +)] +#[case::and_with_nested_or( + Pred::and( + col!("letter").gt(lit("a")), // numbers 2, 3, 5 + Pred::or( + col!("letter").eq(lit("c")), + col!("number").eq(lit(3i64)), + ), + ), + // Unified data skipping evaluates the full expression: + // b/2: AND(TRUE, OR(FALSE, FALSE)) = FALSE -> pruned + // c/3: AND(TRUE, OR(TRUE, TRUE)) = TRUE -> kept + // e/5: AND(TRUE, OR(FALSE, FALSE)) = FALSE -> pruned + table_for_letters(&['c']) +)] +fn predicate_on_letter_and_number( + #[case] pred: Pred, + #[case] expected: Vec, +) -> Result<(), Box> { + // Unified data skipping evaluates partition + data predicates together in a single + // columnar pass, enabling pruning for mixed predicates including OR expressions. + read_table_data( + "./tests/data/basic_partitioned", + Some(&["letter", "number"]), + Some(pred), + expected, + )?; + Ok(()) +} + +// Regression test for issue #2468 +#[rstest::rstest] +#[case::predicate_on_unprojected_data_column_in_table_schema_succeeds( + // `number` is a data column not present in the projected schema (`a_float`). + col!("number").gt(lit(4i64)), + vec![ + "+---------+", + "| a_float |", + "+---------+", + "| 5.5 |", + "| 6.6 |", + "+---------+", + ] + .into_iter() + .map(String::from) + .collect() +)] +#[case::predicate_on_partition_column_in_table_schema_succeeds( + col!("letter").eq(lit("a")), + vec![ + "+---------+", + "| a_float |", + "+---------+", + "| 1.1 |", + "| 4.4 |", + "+---------+", + ] + .into_iter() + .map(String::from) + .collect() +)] +#[case::predicate_on_mixed_projected_and_unprojected_columns_succeeds( + // a_float is projected, number is unprojected + Pred::and( + col!("a_float").gt(lit(4.0)), + col!("number").lt(lit(6i64)), + ), + vec![ + "+---------+", + "| a_float |", + "+---------+", + "| 4.4 |", + "| 5.5 |", + "+---------+", + ] + .into_iter() + .map(String::from) + .collect() +)] +fn predicate_on_unprojected_column( + #[case] pred: Pred, + #[case] expected: Vec, +) -> Result<(), Box> { + read_table_data( + "./tests/data/basic_partitioned", + Some(&["a_float"]), + Some(pred), + expected, + )?; + Ok(()) +} + +/// Test partition pruning on a table with a checkpoint containing `partitionValues_parsed`. +/// This exercises the checkpoint code path where typed partition values are read directly +/// from the parquet column rather than parsed from the string map via `MapToStruct`. +/// +/// Table: app-txn-checkpoint (checkpoint at v1, partition column: `modified` (string)) +/// - 2 files with modified=2021-02-01 (value 4-11, 8 rows each) +/// - 2 files with modified=2021-02-02 (value 1-3, 3 rows each) +#[rstest::rstest] +#[case::partition_only_prunes_one_partition( + // Partition-only predicate: modified = '2021-02-02' should prune 2021-02-01 files + col!("modified").eq(lit("2021-02-02")), + vec![ + "+----+------------+-------+", + "| id | modified | value |", + "+----+------------+-------+", + "| A | 2021-02-02 | 1 |", + "| A | 2021-02-02 | 1 |", + "| A | 2021-02-02 | 3 |", + "| A | 2021-02-02 | 3 |", + "| B | 2021-02-02 | 2 |", + "| B | 2021-02-02 | 2 |", + "+----+------------+-------+", + ] + .into_iter() + .map(String::from) + .collect() +)] +#[case::partition_prunes_other_partition( + // modified = '2021-02-01' should prune 2021-02-02 files, keeping all 2021-02-01 rows + col!("modified").eq(lit("2021-02-01")), + vec![ + "+----+------------+-------+", + "| id | modified | value |", + "+----+------------+-------+", + "| A | 2021-02-01 | 10 |", + "| A | 2021-02-01 | 10 |", + "| A | 2021-02-01 | 11 |", + "| A | 2021-02-01 | 11 |", + "| A | 2021-02-01 | 5 |", + "| A | 2021-02-01 | 5 |", + "| A | 2021-02-01 | 6 |", + "| A | 2021-02-01 | 6 |", + "| A | 2021-02-01 | 7 |", + "| A | 2021-02-01 | 7 |", + "| B | 2021-02-01 | 4 |", + "| B | 2021-02-01 | 4 |", + "| B | 2021-02-01 | 8 |", + "| B | 2021-02-01 | 8 |", + "| B | 2021-02-01 | 9 |", + "| B | 2021-02-01 | 9 |", + "+----+------------+-------+", + ] + .into_iter() + .map(String::from) + .collect() +)] +fn partition_pruning_with_checkpoint_parsed_values( + #[case] pred: Pred, + #[case] expected: Vec, +) -> Result<(), Box> { + read_table_data( + "./tests/data/app-txn-checkpoint", + Some(&["id", "modified", "value"]), + Some(pred), + expected, + )?; + Ok(()) +} + +/// Test mixed predicates (partition + data stats) on a checkpoint with both +/// `partitionValues_parsed` and `stats_parsed`. This exercises the unified columnar data skipping +/// pass that evaluates both partition values and data column statistics together. +/// +/// Table: app-txn-checkpoint (checkpoint at v1, partition column: `modified` (string)) +/// - 2 files: modified=2021-02-02 -- 3 rows each, value in [1, 3] +/// - 2 files: modified=2021-02-01 -- 8 rows each, value in [4, 11] +#[rstest::rstest] +#[case::and_keeps_partition_matched_files( + // Data skipping keeps 2021-02-01 files (partition matches, max(value)=11 > 9) and + // prunes 2021-02-02 files (partition mismatch). All rows from kept files are returned + // since kernel does not apply row-level predicate filtering. + Pred::and( + col!("modified").eq(lit("2021-02-01")), + col!("value").gt(lit(9i32)), + ), + vec![ + "+----+------------+-------+", + "| id | modified | value |", + "+----+------------+-------+", + "| A | 2021-02-01 | 10 |", + "| A | 2021-02-01 | 10 |", + "| A | 2021-02-01 | 11 |", + "| A | 2021-02-01 | 11 |", + "| A | 2021-02-01 | 5 |", + "| A | 2021-02-01 | 5 |", + "| A | 2021-02-01 | 6 |", + "| A | 2021-02-01 | 6 |", + "| A | 2021-02-01 | 7 |", + "| A | 2021-02-01 | 7 |", + "| B | 2021-02-01 | 4 |", + "| B | 2021-02-01 | 4 |", + "| B | 2021-02-01 | 8 |", + "| B | 2021-02-01 | 8 |", + "| B | 2021-02-01 | 9 |", + "| B | 2021-02-01 | 9 |", + "+----+------------+-------+", + ] + .into_iter() + .map(String::from) + .collect() +)] +#[case::and_prunes_all_files( + // 2021-02-02: partition matches but data stats fail (max value=3, NOT > 3). + // 2021-02-01: partition mismatch. All 4 files pruned. + Pred::and( + col!("modified").eq(lit("2021-02-02")), + col!("value").gt(lit(3i32)), + ), + vec![] +)] +#[case::or_prunes_by_both_partition_and_stats( + // 2021-02-01 pruned: partition mismatch AND max(value)=11 NOT > 11. + // 2021-02-02 kept by partition match. Only 2021-02-02 rows returned. + Pred::or( + col!("modified").eq(lit("2021-02-02")), + col!("value").gt(lit(11i32)), + ), + vec![ + "+----+------------+-------+", + "| id | modified | value |", + "+----+------------+-------+", + "| A | 2021-02-02 | 1 |", + "| A | 2021-02-02 | 1 |", + "| A | 2021-02-02 | 3 |", + "| A | 2021-02-02 | 3 |", + "| B | 2021-02-02 | 2 |", + "| B | 2021-02-02 | 2 |", + "+----+------------+-------+", + ] + .into_iter() + .map(String::from) + .collect() +)] +#[case::or_keeps_all_files( + // 2021-02-02 kept by partition match, 2021-02-01 kept by data stats (max=11 > 9). + // All rows from all 4 files are returned. + Pred::or( + col!("modified").eq(lit("2021-02-02")), + col!("value").gt(lit(9i32)), + ), + vec![ + "+----+------------+-------+", + "| id | modified | value |", + "+----+------------+-------+", + "| A | 2021-02-01 | 10 |", + "| A | 2021-02-01 | 10 |", + "| A | 2021-02-01 | 11 |", + "| A | 2021-02-01 | 11 |", + "| A | 2021-02-01 | 5 |", + "| A | 2021-02-01 | 5 |", + "| A | 2021-02-01 | 6 |", + "| A | 2021-02-01 | 6 |", + "| A | 2021-02-01 | 7 |", + "| A | 2021-02-01 | 7 |", + "| A | 2021-02-02 | 1 |", + "| A | 2021-02-02 | 1 |", + "| A | 2021-02-02 | 3 |", + "| A | 2021-02-02 | 3 |", + "| B | 2021-02-01 | 4 |", + "| B | 2021-02-01 | 4 |", + "| B | 2021-02-01 | 8 |", + "| B | 2021-02-01 | 8 |", + "| B | 2021-02-01 | 9 |", + "| B | 2021-02-01 | 9 |", + "| B | 2021-02-02 | 2 |", + "| B | 2021-02-02 | 2 |", + "+----+------------+-------+", + ] + .into_iter() + .map(String::from) + .collect() +)] +fn mixed_predicate_with_checkpoint_parsed_columns( + #[case] pred: Pred, + #[case] expected: Vec, +) -> Result<(), Box> { + // Exercises the unified data skipping path that reads both `partitionValues_parsed` and + // `stats_parsed` from the checkpoint parquet file in a single columnar pass. + read_table_data( + "./tests/data/app-txn-checkpoint", + Some(&["id", "modified", "value"]), + Some(pred), + expected, + )?; + Ok(()) +} + +/// Test partition pruning on a table with column mapping (name mode). The logical partition +/// column "category" has physical name "phys_category". With column mapping, `partitionValues` +/// in the log uses physical column names, and the partition schema + predicate must also use +/// physical names for `MapToStruct` extraction and data skipping to work correctly. +#[rstest::rstest] +#[case::partition_only( + // Partition-only predicate: category = 'A' prunes the category=B file + Arc::new(Pred::eq(col!("category"), lit("A"))), + None, + 1 +)] +#[case::mixed_partition_and_data( + // Mixed predicate: category = 'A' OR val > 'z'. Category=A kept by partition match. + // Category=B: partition mismatch, but max(val)='z' NOT > 'z', so data skipping prunes it. + Arc::new(Pred::or( + Pred::eq(col!("category"), lit("A")), + Pred::gt(col!("val"), lit("z")), + )), + None, + 1 +)] +#[case::predicate_on_unprojected_data_column( + // Project only "category"; predicate references unprojected "val" (logical) whose + // physical name is "phys_val". Kernel must resolve to phys_val via the full table + // schema and prune both files: max(phys_val)='z' is NOT > 'z'. + Arc::new(Pred::gt(col!("val"), lit("z"))), + Some(vec!["category"]), + 0 +)] +#[case::predicate_on_unprojected_partition_column( + // Project only "val"; predicate references unprojected partition column "category" + // (logical, physical name "phys_category"). Partition pruning must still kick in + // using the physical partition name, keeping only the category=A file. + Arc::new(Pred::eq(col!("category"), lit("A"))), + Some(vec!["val"]), + 1 +)] +#[tokio::test] +async fn test_partition_pruning_with_column_mapping( + #[case] predicate: Arc, + #[case] select_cols: Option>, + #[case] expected_files: usize, +) -> Result<(), Box> { + let batch = generate_batch(vec![("phys_val", vec!["x", "y", "z"].into_arrow_array())])?; + + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///"; + + // Column mapping name mode: logical "category" -> physical "phys_category", + // logical "val" -> physical "phys_val" + let schema_str = r#"{"type":"struct","fields":[{"name":"category","type":"string","nullable":true,"metadata":{"delta.columnMapping.id":1,"delta.columnMapping.physicalName":"phys_category"}},{"name":"val","type":"string","nullable":true,"metadata":{"delta.columnMapping.id":2,"delta.columnMapping.physicalName":"phys_val"}}]}"#; + + let actions = [ + r#"{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["columnMapping"],"writerFeatures":["columnMapping"]}}"#.to_string(), + r#"{"commitInfo":{"timestamp":1587968586154,"operation":"WRITE","isBlindAppend":true}}"#.to_string(), + format!( + r#"{{"metaData":{{"id":"test-cm","format":{{"provider":"parquet","options":{{}}}},"schemaString":"{schema}","partitionColumns":["category"],"configuration":{{"delta.columnMapping.mode":"name","delta.columnMapping.maxColumnId":"2"}},"createdTime":1587968585495}}}}"#, + schema = schema_str.replace('"', r#"\""#), + ), + // partitionValues uses physical column name when column mapping is enabled + format!( + r#"{{"add":{{"path":"phys_category=A/{PARQUET_FILE1}","partitionValues":{{"phys_category":"A"}},"size":0,"modificationTime":1587968586000,"dataChange":true,"stats":"{{\"numRecords\":3,\"nullCount\":{{\"phys_val\":0}},\"minValues\":{{\"phys_val\":\"x\"}},\"maxValues\":{{\"phys_val\":\"z\"}}}}" }}}}"# + ), + format!( + r#"{{"add":{{"path":"phys_category=B/{PARQUET_FILE2}","partitionValues":{{"phys_category":"B"}},"size":0,"modificationTime":1587968586000,"dataChange":true,"stats":"{{\"numRecords\":3,\"nullCount\":{{\"phys_val\":0}},\"minValues\":{{\"phys_val\":\"x\"}},\"maxValues\":{{\"phys_val\":\"z\"}}}}" }}}}"# + ), + ]; + + add_commit(table_root, storage.as_ref(), 0, actions.iter().join("\n")).await?; + storage + .put( + &Path::from("phys_category=A").join(PARQUET_FILE1), + record_batch_to_bytes(&batch).into(), + ) + .await?; + storage + .put( + &Path::from("phys_category=B").join(PARQUET_FILE2), + record_batch_to_bytes(&batch).into(), + ) + .await?; + + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; + + // Predicates use logical column names -- kernel must map to physical names. The + // optional projection narrows the output schema; when set, the predicate may still + // reference columns outside it + let projection = select_cols + .as_ref() + .map(|cols| snapshot.schema().project(cols)) + .transpose()?; + let scan = snapshot + .scan_builder() + .with_schema_opt(projection) + .with_predicate(predicate) + .build()?; + + let stream = scan.execute(engine)?; + let mut files_scanned = 0; + // "category" is in the output iff there's no projection (SELECT *) or the projection + // explicitly includes it. + let expect_category = select_cols + .as_ref() + .is_none_or(|cols| cols.contains(&"category")); + for engine_data in stream { + let result_batch = into_record_batch(engine_data?); + // The "category" partition column should be present exactly when projected, and + // should be 'A' for every surviving row. + match result_batch.schema().index_of("category") { + Ok(category_idx) => { + assert!( + expect_category, + "category present but not expected in projection" + ); + let category_col = result_batch.column(category_idx).as_string::(); + for i in 0..result_batch.num_rows() { + assert_eq!(category_col.value(i), "A"); + } + } + Err(_) => assert!( + !expect_category, + "category missing but expected in projection" + ), + } + files_scanned += 1; + } + assert_eq!( + expected_files, files_scanned, + "Expected partition pruning to return {expected_files} file(s)" + ); + + Ok(()) +} + +#[rstest::rstest] +#[case::not_less_than( + Pred::not(col!("number").lt(lit(4i64))), + table_for_numbers(vec![4, 5, 6]) +)] +#[case::not_less_than_or_equal( + Pred::not(col!("number").le(lit(4i64))), + table_for_numbers(vec![5, 6]) +)] +#[case::not_greater_than( + Pred::not(col!("number").gt(lit(4i64))), + table_for_numbers(vec![1, 2, 3, 4]) +)] +#[case::not_greater_than_or_equal( + Pred::not(col!("number").ge(lit(4i64))), + table_for_numbers(vec![1, 2, 3]) +)] +#[case::not_equal( + Pred::not(col!("number").eq(lit(4i64))), + table_for_numbers(vec![1, 2, 3, 5, 6]) +)] +#[case::not_not_equal( + Pred::not(col!("number").ne(lit(4i64))), + table_for_numbers(vec![4]) +)] +fn predicate_on_number_not( + #[case] pred: Pred, + #[case] expected: Vec, +) -> Result<(), Box> { + read_table_data( + "./tests/data/basic_partitioned", + Some(&["a_float", "number"]), + Some(pred), + expected, + )?; + Ok(()) +} + +#[test] +fn predicate_on_number_with_not_null() -> Result<(), Box> { + let expected = vec![ + "+---------+--------+", + "| a_float | number |", + "+---------+--------+", + "| 1.1 | 1 |", + "| 2.2 | 2 |", + "+---------+--------+", + ]; + read_table_data_str( + "./tests/data/basic_partitioned", + Some(&["a_float", "number"]), + Some(Pred::and( + col!("number").is_not_null(), + col!("number").lt(lit(3i64)), + )), + expected, + )?; + Ok(()) +} + +#[test] +fn predicate_null() -> Result<(), Box> { + let expected = vec![]; // number is never null + read_table_data_str( + "./tests/data/basic_partitioned", + Some(&["a_float", "number"]), + Some(col!("number").is_null()), + expected, + )?; + Ok(()) +} + +#[test] +fn mixed_null() -> Result<(), Box> { + let expected = vec![ + "+------+--------------+", + "| part | n |", + "+------+--------------+", + "| 0 | |", + "| 0 | |", + "| 0 | |", + "| 0 | |", + "| 0 | |", + "| 2 | |", + "| 2 | non-null-mix |", + "| 2 | |", + "| 2 | non-null-mix |", + "| 2 | |", + "+------+--------------+", + ]; + read_table_data_str( + "./tests/data/mixed-nulls", + Some(&["part", "n"]), + Some(col!("n").is_null()), + expected, + )?; + Ok(()) +} + +#[test] +fn mixed_not_null() -> Result<(), Box> { + let expected = vec![ + "+------+--------------+", + "| part | n |", + "+------+--------------+", + "| 1 | non-null |", + "| 1 | non-null |", + "| 1 | non-null |", + "| 1 | non-null |", + "| 1 | non-null |", + "| 2 | |", + "| 2 | |", + "| 2 | |", + "| 2 | non-null-mix |", + "| 2 | non-null-mix |", + "+------+--------------+", + ]; + read_table_data_str( + "./tests/data/mixed-nulls", + Some(&["part", "n"]), + Some(col!("n").is_not_null()), + expected, + )?; + Ok(()) +} + +#[rstest::rstest] +#[case::and_both_conditions( + Pred::and( + col!("number").gt(lit(4i64)), + col!("a_float").gt(lit(5.5)), + ), + table_for_numbers(vec![6]) +)] +#[case::and_with_negation( + Pred::and( + col!("number").gt(lit(4i64)), + Pred::not(col!("a_float").gt(lit(5.5))), + ), + table_for_numbers(vec![5]) +)] +#[case::or_either_condition( + Pred::or( + col!("number").gt(lit(4i64)), + col!("a_float").gt(lit(5.5)), + ), + table_for_numbers(vec![5, 6]) +)] +#[case::or_with_negation( + Pred::or( + col!("number").gt(lit(4i64)), + Pred::not(col!("a_float").gt(lit(5.5))), + ), + table_for_numbers(vec![1, 2, 3, 4, 5, 6]) +)] +fn and_or_predicates( + #[case] pred: Pred, + #[case] expected: Vec, +) -> Result<(), Box> { + read_table_data( + "./tests/data/basic_partitioned", + Some(&["a_float", "number"]), + Some(pred), + expected, + )?; + Ok(()) +} + +#[rstest::rstest] +#[case::not_and_both_conditions( + Pred::not(Pred::and( + col!("number").gt(lit(4i64)), + col!("a_float").gt(lit(5.5)), + )), + table_for_numbers(vec![1, 2, 3, 4, 5]) +)] +#[case::not_and_with_negation( + Pred::not(Pred::and( + col!("number").gt(lit(4i64)), + Pred::not(col!("a_float").gt(lit(5.5))), + )), + table_for_numbers(vec![1, 2, 3, 4, 6]) +)] +#[case::not_or_either_condition( + Pred::not(Pred::or( + col!("number").gt(lit(4i64)), + col!("a_float").gt(lit(5.5)), + )), + table_for_numbers(vec![1, 2, 3, 4]) +)] +#[case::not_or_with_negation( + Pred::not(Pred::or( + col!("number").gt(lit(4i64)), + Pred::not(col!("a_float").gt(lit(5.5))), + )), + vec![] +)] +fn not_and_or_predicates( + #[case] pred: Pred, + #[case] expected: Vec, +) -> Result<(), Box> { + read_table_data( + "./tests/data/basic_partitioned", + Some(&["a_float", "number"]), + Some(pred), + expected, + )?; + Ok(()) +} + +#[rstest::rstest] +#[case::literal_false(Pred::FALSE, table_for_numbers(vec![]))] +#[case::and_with_literal_false( + Pred::and(column_pred!("number"), Pred::FALSE), + table_for_numbers(vec![]) +)] +#[case::literal_true( + Pred::TRUE, + table_for_numbers(vec![1, 2, 3, 4, 5, 6]) +)] +#[case::from_literal_expr( + Pred::from_expr(lit(3i64)), + table_for_numbers(vec![1, 2, 3, 4, 5, 6]) +)] +#[case::distinct_value( + col!("number").distinct(lit(3i64)), + table_for_numbers(vec![1, 2, 4, 5, 6]) +)] +#[case::distinct_null( + col!("number").distinct(Expr::null_literal(DataType::LONG)), + table_for_numbers(vec![1, 2, 3, 4, 5, 6]) +)] +#[case::not_distinct_value( + Pred::not(col!("number").distinct(lit(3i64))), + table_for_numbers(vec![3]) +)] +#[case::not_distinct_null( + Pred::not(col!("number").distinct(Expr::null_literal(DataType::LONG))), + table_for_numbers(vec![]) +)] +#[case::gt_empty_struct( + col!("number").gt(Expr::struct_from(Vec::::new())), + table_for_numbers(vec![1, 2, 3, 4, 5, 6]) +)] +#[case::not_gt_empty_struct( + Pred::not(col!("number").gt(Expr::struct_from(Vec::::new()))), + table_for_numbers(vec![1, 2, 3, 4, 5, 6]) +)] +fn invalid_skips_none_predicates( + #[case] pred: Pred, + #[case] expected: Vec, +) -> Result<(), Box> { + read_table_data( + "./tests/data/basic_partitioned", + Some(&["a_float", "number"]), + Some(pred), + expected, + )?; + Ok(()) +} + +#[test] +fn with_predicate_and_removes() -> Result<(), Box> { + let expected = vec![ + "+-------+", + "| value |", + "+-------+", + "| 1 |", + "| 2 |", + "| 3 |", + "| 4 |", + "| 5 |", + "| 6 |", + "| 7 |", + "| 8 |", + "+-------+", + ]; + read_table_data_str( + "./tests/data/table-with-dv-small/", + None, + Some(Pred::gt(col!("value"), lit(3))), + expected, + )?; + Ok(()) +} + +#[tokio::test] +async fn predicate_on_non_nullable_partition_column() -> Result<(), Box> { + // Test for https://github.com/delta-io/delta-kernel-rs/issues/698 + let batch = generate_batch(vec![("val", vec!["a", "b", "c"].into_arrow_array())])?; + + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///"; + let actions = [ + r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#.to_string(), + r#"{"commitInfo":{"timestamp":1587968586154,"operation":"WRITE","operationParameters":{"mode":"ErrorIfExists","partitionBy":"[\"id\"]"},"isBlindAppend":true}}"#.to_string(), + r#"{"metaData":{"id":"5fba94ed-9794-4965-ba6e-6ee3c0d22af9","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":false,\"metadata\":{}},{\"name\":\"val\",\"type\":\"string\",\"nullable\":false,\"metadata\":{}}]}","partitionColumns":["id"],"configuration":{},"createdTime":1587968585495}}"#.to_string(), + format!(r#"{{"add":{{"path":"id=1/{PARQUET_FILE1}","partitionValues":{{"id":"1"}},"size":0,"modificationTime":1587968586000,"dataChange":true, "stats":"{{\"numRecords\":3,\"nullCount\":{{\"val\":0}},\"minValues\":{{\"val\":\"a\"}},\"maxValues\":{{\"val\":\"c\"}}}}"}}}}"#), + format!(r#"{{"add":{{"path":"id=2/{PARQUET_FILE2}","partitionValues":{{"id":"2"}},"size":0,"modificationTime":1587968586000,"dataChange":true, "stats":"{{\"numRecords\":3,\"nullCount\":{{\"val\":0}},\"minValues\":{{\"val\":\"a\"}},\"maxValues\":{{\"val\":\"c\"}}}}"}}}}"#), + ]; + + add_commit(table_root, storage.as_ref(), 0, actions.iter().join("\n")).await?; + storage + .put( + &Path::from("id=1").join(PARQUET_FILE1), + record_batch_to_bytes(&batch).into(), + ) + .await?; + storage + .put( + &Path::from("id=2").join(PARQUET_FILE2), + record_batch_to_bytes(&batch).into(), + ) + .await?; + + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; + + let predicate = Pred::eq(col!("id"), lit(2)); + let scan = snapshot + .scan_builder() + .with_predicate(Arc::new(predicate)) + .build()?; + + let stream = scan.execute(engine)?; + + let mut files_scanned = 0; + for engine_data in stream { + let mut result_batch = into_record_batch(engine_data?); + let _ = result_batch.remove_column(result_batch.schema().index_of("id")?); + assert_eq!(&batch, &result_batch); + files_scanned += 1; + } + assert_eq!(1, files_scanned); + Ok(()) +} + +#[tokio::test] +async fn predicate_on_non_nullable_column_missing_stats() -> Result<(), Box> +{ + let batch_1 = generate_batch(vec![("val", vec!["a", "b", "c"].into_arrow_array())])?; + let batch_2 = generate_batch(vec![("val", vec!["d", "e", "f"].into_arrow_array())])?; + + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///"; + let actions = [ + r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#.to_string(), + r#"{"commitInfo":{"timestamp":1587968586154,"operation":"WRITE","operationParameters":{"mode":"ErrorIfExists","partitionBy":"[]"},"isBlindAppend":true}}"#.to_string(), + r#"{"metaData":{"id":"5fba94ed-9794-4965-ba6e-6ee3c0d22af9","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"val\",\"type\":\"string\",\"nullable\":false,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1587968585495}}"#.to_string(), + // Add one file with stats, one file without + format!(r#"{{"add":{{"path":"{PARQUET_FILE1}","partitionValues":{{}},"size":0,"modificationTime":1587968586000,"dataChange":true, "stats":"{{\"numRecords\":3,\"nullCount\":{{\"val\":0}},\"minValues\":{{\"val\":\"a\"}},\"maxValues\":{{\"val\":\"c\"}}}}"}}}}"#), + format!(r#"{{"add":{{"path":"{PARQUET_FILE2}","partitionValues":{{}},"size":0,"modificationTime":1587968586000,"dataChange":true, "stats":"{{\"numRecords\":3,\"nullCount\":{{}},\"minValues\":{{}},\"maxValues\":{{}}}}"}}}}"#), + ]; + + // Disable writing Parquet statistics so these cannot be used for pruning row groups + let writer_props = WriterProperties::builder() + .set_statistics_enabled(EnabledStatistics::None) + .build(); + + add_commit(table_root, storage.as_ref(), 0, actions.iter().join("\n")).await?; + storage + .put( + &Path::from(PARQUET_FILE1), + record_batch_to_bytes_with_props(&batch_1, writer_props.clone()).into(), + ) + .await?; + storage + .put( + &Path::from(PARQUET_FILE2), + record_batch_to_bytes_with_props(&batch_2, writer_props).into(), + ) + .await?; + + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; + + let predicate = Pred::eq(col!("val"), lit("g")); + let scan = snapshot + .scan_builder() + .with_predicate(Arc::new(predicate)) + .build()?; + + let stream = scan.execute(engine)?; + + let mut files_scanned = 0; + for engine_data in stream { + let result_batch = into_record_batch(engine_data?); + assert_eq!(&batch_2, &result_batch); + files_scanned += 1; + } + // One file is scanned as stats are missing so we don't know the predicate isn't satisfied + assert_eq!(1, files_scanned); + + Ok(()) +} + +#[test] +fn short_dv() -> Result<(), Box> { + let expected = vec![ + "+----+-------+--------------------------+---------------------+", + "| id | value | timestamp | rand |", + "+----+-------+--------------------------+---------------------+", + "| 3 | 3 | 2023-05-31T18:58:33.633Z | 0.7918174793484931 |", + "| 4 | 4 | 2023-05-31T18:58:33.633Z | 0.9281049271981882 |", + "| 5 | 5 | 2023-05-31T18:58:33.633Z | 0.27796520310701633 |", + "| 6 | 6 | 2023-05-31T18:58:33.633Z | 0.15263801464228832 |", + "| 7 | 7 | 2023-05-31T18:58:33.633Z | 0.1981143710215575 |", + "| 8 | 8 | 2023-05-31T18:58:33.633Z | 0.3069439236599195 |", + "| 9 | 9 | 2023-05-31T18:58:33.633Z | 0.5175919190815845 |", + "+----+-------+--------------------------+---------------------+", + ]; + read_table_data_str("./tests/data/with-short-dv/", None, None, expected)?; + Ok(()) +} + +#[test] +fn basic_decimal() -> Result<(), Box> { + let expected = vec![ + "+----------------+---------+--------------+------------------------+", + "| part | col1 | col2 | col3 |", + "+----------------+---------+--------------+------------------------+", + "| -2342342.23423 | -999.99 | -99999.99999 | -9999999999.9999999999 |", + "| 0.00004 | 0.00 | 0.00000 | 0.0000000000 |", + "| 234.00000 | 1.00 | 2.00000 | 3.0000000000 |", + "| 2342222.23454 | 111.11 | 22222.22222 | 3333333333.3333333333 |", + "+----------------+---------+--------------+------------------------+", + ]; + read_table_data_str("./tests/data/basic-decimal-table/", None, None, expected)?; + Ok(()) +} + +#[test] +fn timestamp_ntz() -> Result<(), Box> { + let expected = vec![ + "+----+----------------------------+----------------------------+", + "| id | tsNtz | tsNtzPartition |", + "+----+----------------------------+----------------------------+", + "| 0 | 2021-11-18T02:30:00.123456 | 2021-11-18T02:30:00.123456 |", + "| 1 | 2013-07-05T17:01:00.123456 | 2021-11-18T02:30:00.123456 |", + "| 2 | | 2021-11-18T02:30:00.123456 |", + "| 3 | 2021-11-18T02:30:00.123456 | 2013-07-05T17:01:00.123456 |", + "| 4 | 2013-07-05T17:01:00.123456 | 2013-07-05T17:01:00.123456 |", + "| 5 | | 2013-07-05T17:01:00.123456 |", + "| 6 | 2021-11-18T02:30:00.123456 | |", + "| 7 | 2013-07-05T17:01:00.123456 | |", + "| 8 | | |", + "+----+----------------------------+----------------------------+", + ]; + read_table_data_str( + "./tests/data/data-reader-timestamp_ntz/", + None, + None, + expected, + )?; + Ok(()) +} + +#[test] +fn type_widening_basic() -> Result<(), Box> { + let expected = vec![ + "+---------------------+---------------------+--------------------+----------------+----------------+----------------+----------------------------+", + "| byte_long | int_long | float_double | byte_double | short_double | int_double | date_timestamp_ntz |", + "+---------------------+---------------------+--------------------+----------------+----------------+----------------+----------------------------+", + "| 1 | 2 | 3.4000000953674316 | 5.0 | 6.0 | 7.0 | 2024-09-09T00:00:00 |", + "| 9223372036854775807 | 9223372036854775807 | 1.234567890123 | 1.234567890123 | 1.234567890123 | 1.234567890123 | 2024-09-09T12:34:56.123456 |", + "+---------------------+---------------------+--------------------+----------------+----------------+----------------+----------------------------+", + ]; + let select_cols: Option<&[&str]> = Some(&[ + "byte_long", + "int_long", + "float_double", + "byte_double", + "short_double", + "int_double", + "date_timestamp_ntz", + ]); + + read_table_data_str("./tests/data/type-widening/", select_cols, None, expected) +} + +#[test] +fn type_widening_decimal() -> Result<(), Box> { + let expected = vec![ + "+----------------------------+-------------------------------+--------------+---------------+--------------+----------------------+", + "| decimal_decimal_same_scale | decimal_decimal_greater_scale | byte_decimal | short_decimal | int_decimal | long_decimal |", + "+----------------------------+-------------------------------+--------------+---------------+--------------+----------------------+", + "| 123.45 | 67.89000 | 1.0 | 2.0 | 3.0 | 4.0 |", + "| 12345678901234.56 | 12345678901.23456 | 123.4 | 12345.6 | 1234567890.1 | 123456789012345678.9 |", + "+----------------------------+-------------------------------+--------------+---------------+--------------+----------------------+", + ]; + let select_cols: Option<&[&str]> = Some(&[ + "decimal_decimal_same_scale", + "decimal_decimal_greater_scale", + "byte_decimal", + "short_decimal", + "int_decimal", + "long_decimal", + ]); + read_table_data_str("./tests/data/type-widening/", select_cols, None, expected) +} + +// Verify that predicates over invalid/missing columns do not cause skipping. +#[test] +fn predicate_references_invalid_missing_column() -> Result<(), Box> { + // Attempted skipping over a logically valid but physically missing column. We should be able to + // skip the data file because the missing column is inferred to be all-null. + // + // WARNING: https://github.com/delta-io/delta-kernel-rs/issues/434 -- currently disabled. + // + //let expected = vec![ + // "+--------+", + // "| chrono |", + // "+--------+", + // "+--------+", + //]; + let columns = &["chrono", "missing"]; + let expected = vec![ + "+-------------------------------------------------------------------------------------------+---------+", + "| chrono | missing |", + "+-------------------------------------------------------------------------------------------+---------+", + "| {date32: 1971-01-01, timestamp: 1970-02-01T08:00:00Z, timestamp_ntz: 1970-01-02T00:00:00} | |", + "| {date32: 1971-01-02, timestamp: 1970-02-01T09:00:00Z, timestamp_ntz: 1970-01-02T00:01:00} | |", + "| {date32: 1971-01-03, timestamp: 1970-02-01T10:00:00Z, timestamp_ntz: 1970-01-02T00:02:00} | |", + "| {date32: 1971-01-04, timestamp: 1970-02-01T11:00:00Z, timestamp_ntz: 1970-01-02T00:03:00} | |", + "| {date32: 1971-01-05, timestamp: 1970-02-01T12:00:00Z, timestamp_ntz: 1970-01-02T00:04:00} | |", + "+-------------------------------------------------------------------------------------------+---------+", + ]; + let predicate = col!("missing").lt(lit(10i64)); + read_table_data_str( + "./tests/data/parquet_row_group_skipping/", + Some(columns), + Some(predicate), + expected, + )?; + + // Attempted skipping over an invalid (logically missing) column. Ideally this should throw a + // query error, but at a minimum it should not cause incorrect data skipping. + let expected = vec![ + "+-------------------------------------------------------------------------------------------+", + "| chrono |", + "+-------------------------------------------------------------------------------------------+", + "| {date32: 1971-01-01, timestamp: 1970-02-01T08:00:00Z, timestamp_ntz: 1970-01-02T00:00:00} |", + "| {date32: 1971-01-02, timestamp: 1970-02-01T09:00:00Z, timestamp_ntz: 1970-01-02T00:01:00} |", + "| {date32: 1971-01-03, timestamp: 1970-02-01T10:00:00Z, timestamp_ntz: 1970-01-02T00:02:00} |", + "| {date32: 1971-01-04, timestamp: 1970-02-01T11:00:00Z, timestamp_ntz: 1970-01-02T00:03:00} |", + "| {date32: 1971-01-05, timestamp: 1970-02-01T12:00:00Z, timestamp_ntz: 1970-01-02T00:04:00} |", + "+-------------------------------------------------------------------------------------------+", + ]; + let predicate = col!("invalid").lt(lit(10)); + read_table_data_str( + "./tests/data/parquet_row_group_skipping/", + Some(columns), + Some(predicate), + expected, + ) + .expect_err("unknown column"); + Ok(()) +} + +// Note: This test is disabled for windows because it creates a directory with name +// `time=1971-07-22T03:06:40.000000Z`. This is disallowed in windows due to having a `:` in +// the name. +#[cfg(not(windows))] +#[test] +fn timestamp_partitioned_table() -> Result<(), Box> { + let expected = vec![ + "+----+-----+---+----------------------+", + "| id | x | s | time |", + "+----+-----+---+----------------------+", + "| 1 | 0.5 | | 1971-07-22T03:06:40Z |", + "+----+-----+---+----------------------+", + ]; + let test_name = "timestamp-partitioned-table"; + let test_dir = load_test_data("./tests/data", test_name).unwrap(); + let test_path = test_dir.path().join(test_name); + read_table_data_str(test_path.to_str().unwrap(), None, None, expected) +} + +#[test] +fn compacted_log_files_table() -> Result<(), Box> { + let expected = vec![ + "+----+--------------------+", + "| id | comment |", + "+----+--------------------+", + "| 0 | new |", + "| 1 | after-large-delete |", + "| 2 | |", + "| 10 | merge1-insert |", + "| 12 | merge2-insert |", + "+----+--------------------+", + ]; + let test_name = "compacted-log-files-table"; + let test_dir = load_test_data("./tests/data", test_name).unwrap(); + let test_path = test_dir.path().join(test_name); + read_table_data_str(test_path.to_str().unwrap(), None, None, expected) +} + +#[test] +fn unshredded_variant_table() -> Result<(), Box> { + let expected = include!("../../data/unshredded-variant.expected.in"); + let test_name = "unshredded-variant"; + let test_dir = load_test_data("./tests/data", test_name).unwrap(); + let test_path = test_dir.path().join(test_name); + read_table_data_str(test_path.to_str().unwrap(), None, None, expected) +} + +#[tokio::test] +async fn test_row_index_metadata_column() -> Result<(), Box> { + // Setup up an in-memory table with different numbers of rows in each file + let batch1 = generate_batch(vec![ + ("id", vec![1i32, 2, 3, 4, 5].into_arrow_array()), + ("value", vec!["a", "b", "c", "d", "e"].into_arrow_array()), + ])?; + let batch2 = generate_batch(vec![ + ("id", vec![10i32, 20, 30].into_arrow_array()), + ("value", vec!["x", "y", "z"].into_arrow_array()), + ])?; + let batch3 = generate_batch(vec![ + ("id", vec![100i32, 200, 300, 400].into_arrow_array()), + ("value", vec!["p", "q", "r", "s"].into_arrow_array()), + ])?; + + let file_size1 = record_batch_to_bytes(&batch1).len() as u64; + let file_size2 = record_batch_to_bytes(&batch2).len() as u64; + let file_size3 = record_batch_to_bytes(&batch3).len() as u64; + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///"; + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![ + TestAction::Metadata, + TestAction::AddWithSize(PARQUET_FILE1.to_string(), file_size1), + TestAction::AddWithSize(PARQUET_FILE2.to_string(), file_size2), + TestAction::AddWithSize(PARQUET_FILE3.to_string(), file_size3), + ]), + ) + .await?; + + for (parquet_file, batch) in [ + (PARQUET_FILE1, &batch1), + (PARQUET_FILE2, &batch2), + (PARQUET_FILE3, &batch3), + ] { + storage + .put( + &Path::from(parquet_file), + record_batch_to_bytes(batch).into(), + ) + .await?; + } + + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + + // Create a schema that includes a row index metadata column + let schema = Arc::new(StructType::try_new([ + StructField::nullable("id", DataType::INTEGER), + StructField::create_metadata_column("row_index", MetadataColumnSpec::RowIndex), + StructField::nullable("value", DataType::STRING), + ])?); + + let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; + let scan = snapshot.scan_builder().with_schema(schema).build()?; + + let mut file_count = 0; + let expected_row_counts = [5, 3, 4]; + let stream = scan.execute(engine.clone())?; + + for data in stream { + let batch = into_record_batch(data?); + file_count += 1; + + // Verify the schema structure + assert_eq!(batch.num_columns(), 3, "Expected 3 columns in the batch"); + assert_eq!( + batch.schema().field(0).name(), + "id", + "First column should be 'id'" + ); + assert_eq!( + batch.schema().field(1).name(), + "row_index", + "Second column should be 'row_index'" + ); + assert_eq!( + batch.schema().field(2).name(), + "value", + "Third column should be 'value'" + ); + + // Each file should have row indexes starting from 0 (file-local indexing) + let row_index_array = batch.column(1).as_primitive::(); + let expected_values: Vec = (0..batch.num_rows() as i64).collect(); + assert_eq!( + row_index_array.values().to_vec(), + expected_values, + "Row index values incorrect for file {} (expected {} rows)", + file_count, + expected_row_counts[file_count - 1] + ); + } + + assert_eq!(file_count, 3, "Expected to scan 3 files"); + Ok(()) +} + +#[tokio::test] +async fn test_file_path_metadata_column() -> Result<(), Box> { + use delta_kernel::arrow::array::{Array, StringArray}; + + // Set up an in-memory table with multiple data files + let batch1 = generate_batch(vec![ + ("id", vec![1i32, 2, 3].into_arrow_array()), + ("value", vec!["a", "b", "c"].into_arrow_array()), + ])?; + let batch2 = generate_batch(vec![ + ("id", vec![10i32, 20].into_arrow_array()), + ("value", vec!["x", "y"].into_arrow_array()), + ])?; + + let file_size1 = record_batch_to_bytes(&batch1).len() as u64; + let file_size2 = record_batch_to_bytes(&batch2).len() as u64; + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///"; + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![ + TestAction::Metadata, + TestAction::AddWithSize(PARQUET_FILE1.to_string(), file_size1), + TestAction::AddWithSize(PARQUET_FILE2.to_string(), file_size2), + ]), + ) + .await?; + + for (parquet_file, batch) in [(PARQUET_FILE1, &batch1), (PARQUET_FILE2, &batch2)] { + storage + .put( + &Path::from(parquet_file), + record_batch_to_bytes(batch).into(), + ) + .await?; + } + + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + + // Create a schema that includes the file path metadata column + let schema = Arc::new(StructType::try_new([ + StructField::nullable("id", DataType::INTEGER), + StructField::create_metadata_column("_file", MetadataColumnSpec::FilePath), + StructField::nullable("value", DataType::STRING), + ])?); + + let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; + let scan = snapshot.scan_builder().with_schema(schema).build()?; + + let mut file_count = 0; + let expected_files = [PARQUET_FILE1, PARQUET_FILE2]; + let expected_row_counts = [3, 2]; + let stream = scan.execute(engine.clone())?; + + for data in stream { + let batch = into_record_batch(data?); + + // Verify the schema structure + assert_eq!(batch.num_columns(), 3, "Expected 3 columns in the batch"); + assert_eq!( + batch.schema().field(0).name(), + "id", + "First column should be 'id'" + ); + assert_eq!( + batch.schema().field(1).name(), + "_file", + "Second column should be '_file'" + ); + assert_eq!( + batch.schema().field(2).name(), + "value", + "Third column should be 'value'" + ); + + // Verify the file path column contains the expected file name + let file_path_array = batch.column(1); + let expected_file_name = expected_files[file_count]; + let expected_path = format!("{table_root}{expected_file_name}"); + + // The file path array should be a plain StringArray with the path repeated for each row. + let string_array = file_path_array + .as_any() + .downcast_ref::() + .expect("File path column should be a StringArray"); + + assert_eq!( + string_array.len(), + expected_row_counts[file_count], + "File {} should have {} rows", + expected_file_name, + expected_row_counts[file_count] + ); + assert!( + string_array + .iter() + .all(|v| v == Some(expected_path.as_str())), + "All rows should contain file path '{expected_path}'" + ); + + file_count += 1; + } + + assert_eq!(file_count, 2, "Expected to scan 2 files"); + Ok(()) +} + +#[tokio::test] +async fn test_unsupported_metadata_columns() -> Result<(), Box> { + // Prepare an in-memory table with some data + let batch = generate_simple_batch()?; + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///"; + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![ + TestAction::Metadata, + TestAction::Add(PARQUET_FILE1.to_string()), + ]), + ) + .await?; + storage + .put( + &Path::from(PARQUET_FILE1), + record_batch_to_bytes(&batch).into(), + ) + .await?; + + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + + // Test that unsupported metadata columns fail with appropriate errors + let test_cases = [ + ( + "row_id", + MetadataColumnSpec::RowId, + "Row ids are not enabled on this table", + ), + ( + "row_commit_version", + MetadataColumnSpec::RowCommitVersion, + "Row commit versions not supported", + ), + ]; + + for (column_name, metadata_spec, error_text) in test_cases { + let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; + let schema = Arc::new(StructType::try_new([ + StructField::nullable("id", DataType::INTEGER), + StructField::create_metadata_column(column_name, metadata_spec), + ])?); + + let scan_err = snapshot + .scan_builder() + .with_schema(schema) + .build() + .unwrap_err(); + let error_msg = scan_err.to_string(); + assert!( + error_msg.contains(error_text), + "Expected {error_msg} to contain {error_text}" + ); + } + + Ok(()) +} + +#[tokio::test] +async fn test_invalid_files_are_skipped() -> Result<(), Box> { + let batch = generate_simple_batch()?; + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///"; + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![ + TestAction::Metadata, + TestAction::Add(PARQUET_FILE1.to_string()), + TestAction::Add(PARQUET_FILE2.to_string()), + ]), + ) + .await?; + storage + .put( + &Path::from(PARQUET_FILE1), + record_batch_to_bytes(&batch).into(), + ) + .await?; + storage + .put( + &Path::from(PARQUET_FILE2), + record_batch_to_bytes(&batch).into(), + ) + .await?; + + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + + let invalid_files = [ + "_delta_log/0.zip", + "_delta_log/_copy_into_log/0.zip", + "_delta_log/_ignore_me/00000000000000000000.json", + "_delta_log/_and_me/00000000000000000000.checkpoint.parquet", + "_delta_log/02184.json", + "_delta_log/0x000000000000000000.checkpoint.parquet", + "00000000000000000000.json", + "_delta_log/_staged_commits/_staged_commits/00000000000000000000.3a0d65cd-4056-49b8-937b-95f9e3ee90e5.json", + "_delta_log/my_random_dir/_staged_commits/00000000000000000000.3a0d65cd-4056-49b8-937b-95f9e3ee90e5.json", + "_delta_log/my_random_dir/_delta_log/_staged_commits/00000000000000000000.3a0d65cd-4056-49b8-937b-95f9e3ee90e5.json", + "_delta_log/_delta_log/00000000000000000000.json", + "_delta_log/_delta_log/00000000000000000000.checkpoint.parquet", + "_delta_log/something/_delta_log/00000000000000000000.crc", + "_delta_log/something/_delta_log/00000000000000000000.json", + "_delta_log/something/_delta_log/00000000000000000000.checkpoint.parquet", + ]; + + fn get_file_path_for_test(path: &ParsedLogPath) -> &str { + &path.location.location.as_str()[10..] + } + + fn ensure_segment_does_not_contain(invalid_files: &[&str], segment: &LogSegment) { + assert!( + !segment.listed.ascending_commit_files.iter().any(|p| { + let test_path = get_file_path_for_test(p); + invalid_files.contains(&test_path) + }), + "ascending_commit_files contained invalid file" + ); + assert!( + !segment.listed.ascending_compaction_files.iter().any(|p| { + let test_path = get_file_path_for_test(p); + invalid_files.contains(&test_path) + }), + "ascending_compaction_files contained invalid file" + ); + assert!( + !segment.listed.checkpoint_parts.iter().any(|p| { + let test_path = get_file_path_for_test(p); + invalid_files.contains(&test_path) + }), + "checkpoint_parts contained invalid file" + ); + if let Some(ref crc) = segment.listed.latest_crc_file { + assert!( + !invalid_files.contains(&get_file_path_for_test(crc)), + "Latest crc contained invalid file" + ); + } + if let Some(ref latest_commit) = segment.listed.latest_commit_file { + assert!( + !invalid_files.contains(&get_file_path_for_test(latest_commit)), + "Latest commit contained invalid file" + ); + } + } + + for invalid_file in invalid_files.iter() { + let invalid_path = Path::from(*invalid_file); + storage.put(&invalid_path, vec![1u8].into()).await?; + let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; + ensure_segment_does_not_contain(&invalid_files, snapshot.log_segment()); + storage.delete(&invalid_path).await?; + } + + // final test with _all_ the files we should ignore + for invalid_file in invalid_files.iter() { + let invalid_path = Path::from(*invalid_file); + storage.put(&invalid_path, vec![1u8].into()).await?; + } + let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; + ensure_segment_does_not_contain(&invalid_files, snapshot.log_segment()); + + Ok(()) +} + +// Verifies data skipping works via stats_parsed across all checkpoint types. +// All tables have writeStatsAsStruct=true, writeStatsAsJson=false (struct stats only), +// schema (id: long, value: string), 5 rows (id 1-5), checkpoint at v5. +// Predicate id > 3 skips files where max(id) <= 3, returning only rows 4 and 5. +#[rstest::rstest] +#[test] +fn checkpoint_stats_skipping( + #[values( + "v1-single-part", + "v1-multi-part", + "v2-parquet-sidecars", + "v2-json-sidecars", + "v2-classic-parquet" + )] + checkpoint_type: &str, +) -> Result<(), Box> { + let table_path = format!("./tests/data/{checkpoint_type}-struct-stats-only/"); + let expected = vec![ + "+----+---------+", + "| id | value |", + "+----+---------+", + "| 4 | value_4 |", + "| 5 | value_5 |", + "+----+---------+", + ]; + let predicate = col!("id").gt(lit(3i64)); + read_table_data_str(&table_path, None, Some(predicate), expected)?; + Ok(()) +} + +// Verifies ScanFile.stats handling for parsed-stats checkpoints. Tables have +// writeStatsAsStruct=true, writeStatsAsJson=false (no JSON stats in checkpoint), +// schema (id: long, value: string), 5 files with 1 row each, checkpoint at v5. +// Cross-product covers all five checkpoint variants against four stats option +// shapes: ScanFile.stats should be populated via the COALESCE/ToJson fallback +// when both `json=true` and `struct_stats=All` are set; otherwise null on these +// struct-stats-only checkpoints. +#[rstest::rstest] +#[case::default_json_only(StatsOptions::default(), false)] +#[case::all_both(StatsOptions::all(), true)] +#[case::all_struct_only(StatsOptions::all_struct(), false)] +#[case::none(StatsOptions::none(), false)] +fn struct_stats_surfaced_in_scan_file( + #[values( + "v1-single-part", + "v1-multi-part", + "v2-parquet-sidecars", + "v2-json-sidecars", + "v2-classic-parquet" + )] + checkpoint_type: &str, + #[case] stats: StatsOptions, + #[case] expect_json_stats: bool, +) -> Result<(), Box> { + let table_path = format!("./tests/data/{checkpoint_type}-struct-stats-only/"); + let path = std::fs::canonicalize(PathBuf::from(table_path))?; + let url = url::Url::from_directory_path(path).unwrap(); + let engine = test_utils::create_default_engine(&url)?; + + let snapshot = Snapshot::builder_for(url).build(engine.as_ref())?; + let scan = snapshot.scan_builder().with_stats(stats).build()?; + + let mut scan_files = vec![]; + for res in scan.scan_metadata(engine.as_ref())? { + let scan_metadata = res?; + scan_files = scan_metadata.visit_scan_files(scan_files, scan_metadata_callback)?; + } + + assert_eq!(scan_files.len(), 5, "expected 5 files in checkpoint"); + for scan_file in &scan_files { + if expect_json_stats { + assert!( + scan_file.stats.is_some(), + "ScanFile.stats should be populated, path: {}", + scan_file.path + ); + assert_eq!( + scan_file.stats.as_ref().unwrap().num_records, + 1, + "each file has exactly 1 row, path: {}", + scan_file.path + ); + } else { + assert!( + scan_file.stats.is_none(), + "ScanFile.stats should be None, path: {}", + scan_file.path + ); + } + } + Ok(()) +} + +// On a JSON-only table (no parsed-stats checkpoint), `has_stats_parsed` is +// false and the COALESCE branch is never selected, so `StatsOptions::all_struct` +// vs `StatsOptions::default` is a no-op for the `stats` JSON column: `add.stats` +// is read directly from the commit JSON either way. +#[rstest::rstest] +fn struct_stats_only_no_op_on_json_only_table( + #[values(StatsOptions::default(), StatsOptions::all_struct())] stats: StatsOptions, +) -> Result<(), Box> { + let path = std::fs::canonicalize(PathBuf::from("./tests/data/basic_partitioned/"))?; + let url = url::Url::from_directory_path(path).unwrap(); + let engine = test_utils::create_default_engine(&url)?; + + let snapshot = Snapshot::builder_for(url).build(engine.as_ref())?; + let scan = snapshot.scan_builder().with_stats(stats).build()?; + + let mut scan_files = vec![]; + for res in scan.scan_metadata(engine.as_ref())? { + let scan_metadata = res?; + scan_files = scan_metadata.visit_scan_files(scan_files, scan_metadata_callback)?; + } + + assert!(!scan_files.is_empty()); + for scan_file in &scan_files { + assert!( + scan_file.stats.is_some(), + "JSON commits populate stats from add.stats" + ); + } + Ok(()) +} + +// Confirms `stats_parsed` is still consumed by data skipping when only struct +// stats are requested (no JSON synthesis). Predicate `id > 3` should skip files +// where `max(id) <= 3`, returning rows 4 and 5 only. If `all_struct` accidentally +// disabled the stats_parsed read path, all 5 rows would come back. +#[rstest::rstest] +fn struct_stats_only_preserves_data_skipping( + #[values( + "v1-single-part", + "v1-multi-part", + "v2-parquet-sidecars", + "v2-json-sidecars", + "v2-classic-parquet" + )] + checkpoint_type: &str, +) -> Result<(), Box> { + let table_path = format!("./tests/data/{checkpoint_type}-struct-stats-only/"); + let path = std::fs::canonicalize(PathBuf::from(table_path))?; + let url = url::Url::from_directory_path(path).unwrap(); + let engine = test_utils::create_default_engine(&url)?; + let snapshot = Snapshot::builder_for(url).build(engine.as_ref())?; + + let predicate: PredicateRef = Arc::new(col!("id").gt(lit(3i64))); + let scan = snapshot + .scan_builder() + .with_stats(StatsOptions::all_struct()) + .with_predicate(predicate) + .build()?; + + let mut scan_files = vec![]; + for res in scan.scan_metadata(engine.as_ref())? { + let scan_metadata = res?; + scan_files = scan_metadata.visit_scan_files(scan_files, scan_metadata_callback)?; + } + + assert_eq!( + scan_files.len(), + 2, + "data skipping via stats_parsed should leave only 2 files (id=4, id=5)" + ); + for scan_file in &scan_files { + assert!( + scan_file.stats.is_none(), + "ScanFile.stats must remain null when synthesis is skipped, path: {}", + scan_file.path + ); + } + Ok(()) +} + +// Multi-part V1 checkpoint with partitionValues_parsed on a partitioned table. +// Schema: (id: long, value: string, part: int) partitioned by part. +// Each commit inserts one row with part = i % 3 (parts 0, 1, 2). +#[test] +fn partition_values_parsed_skipping() -> Result<(), Box> { + // Predicate part = 0 should skip partitions 1 and 2, returning rows with part=0. + // i % 3 == 0: i=3 -> (3, "value_3", 0) + let expected = vec![ + "+----+---------+------+", + "| id | value | part |", + "+----+---------+------+", + "| 3 | value_3 | 0 |", + "+----+---------+------+", + ]; + let predicate = col!("part").eq(lit(0i32)); + read_table_data_str( + "./tests/data/v1-multi-part-partitioned-struct-stats-only/", + None, + Some(predicate), + expected, + )?; + Ok(()) +} + +// In-memory test with crafted truncated JSON stats. Three files: +// file 1: ts_col [1s, 2s] -- max at ms boundary +// file 2: ts_col [3s, 4.000500s] -- JSON max truncated to 4.000s +// file 3: ts_col [7s, 8s] -- max at ms boundary +// +// Predicate `ts_col > 4_000_400us`: +// file 1: max=2s << adjusted predicate (3_999_401) -> pruned (skipping works) +// file 2: truncated max=4s > adjusted predicate (3_999_401) -> kept (truncation safe) +// file 3: max=8s >> adjusted predicate -> kept +#[tokio::test] +async fn timestamp_max_stat_truncation_does_not_over_prune( +) -> Result<(), Box> { + let ts_metadata = "{\ + \"id\":\"test-ts-table\",\ + \"format\":{\"provider\":\"parquet\",\"options\":{}},\ + \"schemaString\":\"{\\\"type\\\":\\\"struct\\\",\\\"fields\\\":[\ + {\\\"name\\\":\\\"ts_col\\\",\\\"type\\\":\\\"timestamp\\\",\\\"nullable\\\":true,\\\"metadata\\\":{}}\ + ]}\",\ + \"partitionColumns\":[],\ + \"configuration\":{},\ + \"createdTime\":1700000000000\ + }"; + + let ts_schema = Arc::new(ArrowSchema::new(vec![ArrowField::new( + "ts_col", + delta_kernel::arrow::datatypes::DataType::Timestamp( + TimeUnit::Microsecond, + Some("UTC".into()), + ), + true, + )])); + + // file1: max at ms boundary, will be pruned + let file1_stats = r#"{"numRecords":2,"nullCount":{"ts_col":0},"minValues":{"ts_col":"1970-01-01T00:00:01.000Z"},"maxValues":{"ts_col":"1970-01-01T00:00:02.000Z"}}"#; + // file2: max truncated from 4.000500s to 4.000s -- truncation adjustment must keep this + let file2_stats = r#"{"numRecords":2,"nullCount":{"ts_col":0},"minValues":{"ts_col":"1970-01-01T00:00:03.000Z"},"maxValues":{"ts_col":"1970-01-01T00:00:04.000Z"}}"#; + // file3: max clearly above predicate + let file3_stats = r#"{"numRecords":2,"nullCount":{"ts_col":0},"minValues":{"ts_col":"1970-01-01T00:00:07.000Z"},"maxValues":{"ts_col":"1970-01-01T00:00:08.000Z"}}"#; + + let batch1 = RecordBatch::try_new( + ts_schema.clone(), + vec![Arc::new( + TimestampMicrosecondArray::from(vec![1_000_000i64, 2_000_000]).with_timezone("UTC"), + )], + )?; + let batch2 = RecordBatch::try_new( + ts_schema.clone(), + vec![Arc::new( + TimestampMicrosecondArray::from(vec![3_000_000i64, 4_000_500]).with_timezone("UTC"), + )], + )?; + let batch3 = RecordBatch::try_new( + ts_schema, + vec![Arc::new( + TimestampMicrosecondArray::from(vec![7_000_000i64, 8_000_000]).with_timezone("UTC"), + )], + )?; + + let file1_bytes = record_batch_to_bytes(&batch1); + let file2_bytes = record_batch_to_bytes(&batch2); + let file3_bytes = record_batch_to_bytes(&batch3); + + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///"; + + let make_add = |name: &str, size: usize, stats: &str| -> String { + format!( + "{{\"add\":{{\"path\":\"{name}\",\"partitionValues\":{{}},\"size\":{size},\ + \"modificationTime\":1700000000000,\"dataChange\":true,\ + \"stats\":\"{stats_escaped}\"}}}}", + stats_escaped = stats.replace('"', "\\\""), + ) + }; + + let commit0 = format!( + "{{\"protocol\":{{\"minReaderVersion\":1,\"minWriterVersion\":2}}}}\n\ + {{\"metaData\":{ts_metadata}}}\n\ + {}", + make_add("file1.parquet", file1_bytes.len(), file1_stats) + ); + add_commit(table_root, storage.as_ref(), 0, commit0).await?; + add_commit( + table_root, + storage.as_ref(), + 1, + make_add("file2.parquet", file2_bytes.len(), file2_stats), + ) + .await?; + add_commit( + table_root, + storage.as_ref(), + 2, + make_add("file3.parquet", file3_bytes.len(), file3_stats), + ) + .await?; + + storage + .put(&Path::from("file1.parquet"), file1_bytes.into()) + .await?; + storage + .put(&Path::from("file2.parquet"), file2_bytes.into()) + .await?; + storage + .put(&Path::from("file3.parquet"), file3_bytes.into()) + .await?; + + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; + + let row_count = |predicate_us: i64| -> Result> { + let predicate = Arc::new(Pred::gt( + col!("ts_col"), + lit(Scalar::Timestamp(predicate_us)), + )); + let scan = snapshot + .clone() + .scan_builder() + .with_predicate(predicate) + .build()?; + let batches = read_scan(&scan, engine.clone())?; + Ok(batches.iter().map(|b| b.num_rows()).sum()) + }; + + // Mid-ms value (4.000400s): adjusted to 3_999_401 + // file1 max=2s < 3_999_401 -> pruned; file2+3 kept (4 rows) + assert_eq!(row_count(4_000_400)?, 4, "mid-ms: file2+file3 kept"); + + // Exact ms boundary (4.000000s = truncated max of file2): adjusted to 3_999_001 + // file1 max=2s < 3_999_001 -> pruned; file2 max=4s > 3_999_001 -> kept (4 rows) + assert_eq!( + row_count(4_000_000)?, + 4, + "exact ms boundary: file2+file3 kept" + ); + + // 1us above ms boundary (4.000001s): adjusted to 3_999_002 + // file1 pruned; file2 max=4s > 3_999_002 -> kept (4 rows) + assert_eq!(row_count(4_000_001)?, 4, "1us above ms: file2+file3 kept"); + + // 998us above ms boundary (4.000998s): adjusted to 3_999_999 + // file2 max=4s > 3_999_999 -> kept (just not prunable) + assert_eq!( + row_count(4_000_998)?, + 4, + "just not prunable: file2+file3 kept" + ); + + // 999us above ms boundary (4.000999s): adjusted to 4_000_000 + // file2 max=4s == 4_000_000 -> NOT strictly greater -> pruned (just prunable) + assert_eq!(row_count(4_000_999)?, 2, "just prunable: only file3 kept"); + + // Next ms boundary (4.001000s): adjusted to 4_000_001 + // file2 max=4s < 4_000_001 -> pruned (2 rows) + assert_eq!( + row_count(4_001_000)?, + 2, + "next ms boundary: only file3 kept" + ); + + Ok(()) +} + +// Regression test for https://github.com/delta-io/delta-kernel-rs/issues/739 +// Void columns should be present in scan results as all-null columns. +#[tokio::test] +async fn read_table_with_void_column() -> Result<(), Box> { + // Parquet batch has only the non-void column (parquet cannot represent void) + let batch = generate_batch(vec![("id", vec![1, 2, 3].into_arrow_array())])?; + + let storage = Arc::new(InMemory::new()); + let actions = [ + r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#.to_string(), + r#"{"commitInfo":{"timestamp":1587968586154,"operation":"WRITE","operationParameters":{"mode":"ErrorIfExists","partitionBy":"[]"},"isBlindAppend":true}}"#.to_string(), + r#"{"metaData":{"id":"test-void","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"void_col\",\"type\":\"void\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1587968585495}}"#.to_string(), + format!(r#"{{"add":{{"path":"{PARQUET_FILE1}","partitionValues":{{}},"size":0,"modificationTime":1587968586000,"dataChange":true}}}}"#), + ]; + + add_commit("memory:///", storage.as_ref(), 0, actions.iter().join("\n")).await?; + storage + .put( + &Path::from(PARQUET_FILE1), + record_batch_to_bytes(&batch).into(), + ) + .await?; + + let location = Url::parse("memory:///")?; + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + let snapshot = Snapshot::builder_for(location).build(engine.as_ref())?; + + // The table schema has both "id" and "void_col" + assert!(snapshot.schema().field("void_col").is_some()); + assert!(snapshot.schema().field("id").is_some()); + + let scan = snapshot.scan_builder().build()?; + + // The scan's logical schema should contain the void column (returned as null) + let logical_schema = scan.logical_schema(); + assert!(logical_schema.field("void_col").is_some()); + assert!(logical_schema.field("id").is_some()); + assert_eq!(logical_schema.fields().count(), 2); + + // Execute the scan and verify void column appears as all-null + let batches = read_scan(&scan, engine)?; + assert_eq!(batches.len(), 1); + assert_eq!(batches[0].num_columns(), 2); + assert_eq!(batches[0].schema().field(0).name(), "id"); + assert_eq!(batches[0].schema().field(1).name(), "void_col"); + assert_eq!(batches[0].num_rows(), 3); + // void column should be Arrow Null type + assert_eq!( + *batches[0].schema().field(1).data_type(), + ArrowDataType::Null + ); + + Ok(()) +} + +// End-to-end tests using a Spark-written Delta table with real truncated JSON stats. +// Table has three files: +// file 1: id=[1,2], ts_col=[1s, 2s] -- max at ms boundary +// file 2: id=[3,4], ts_col=[3s, 4.000500s] -- max truncated to 4.000s in JSON stats +// file 3: id=[5,6], ts_col=[7s, 8s] -- max at ms boundary +// +// Predicate value 4.000400s sits between the truncated max (4.000s) and actual max +// (4.000500s) of file 2, exercising the truncation adjustment. + +// GT: file1 pruned (max=2s < adjusted 3.999401s), file2+3 kept +#[test] +fn timestamp_truncation_real_table_gt() -> Result<(), Box> { + read_table_data_str( + "./tests/data/timestamp-truncation-stats", + None, + Some(Pred::gt(col!("ts_col"), lit(Scalar::Timestamp(4_000_400)))), + vec![ + "+----+-----------------------------+", + "| id | ts_col |", + "+----+-----------------------------+", + "| 3 | 1970-01-01T00:00:03Z |", + "| 4 | 1970-01-01T00:00:04.000500Z |", + "| 5 | 1970-01-01T00:00:07Z |", + "| 6 | 1970-01-01T00:00:08Z |", + "+----+-----------------------------+", + ], + ) +} + +// Verify that an explicit projection including a void column returns it as null. +#[tokio::test] +async fn explicit_projection_with_void_column_returns_nulls( +) -> Result<(), Box> { + let batch = generate_batch(vec![("id", vec![1, 2, 3].into_arrow_array())])?; + + let storage = Arc::new(InMemory::new()); + let actions = [ + r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#.to_string(), + r#"{"commitInfo":{"timestamp":1587968586154,"operation":"WRITE","operationParameters":{"mode":"ErrorIfExists","partitionBy":"[]"},"isBlindAppend":true}}"#.to_string(), + r#"{"metaData":{"id":"test-void","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"void_col\",\"type\":\"void\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1587968585495}}"#.to_string(), + format!(r#"{{"add":{{"path":"{PARQUET_FILE1}","partitionValues":{{}},"size":0,"modificationTime":1587968586000,"dataChange":true}}}}"#), + ]; + + add_commit("memory:///", storage.as_ref(), 0, actions.iter().join("\n")).await?; + storage + .put( + &Path::from(PARQUET_FILE1), + record_batch_to_bytes(&batch).into(), + ) + .await?; + + let location = Url::parse("memory:///")?; + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + let snapshot = Snapshot::builder_for(location).build(engine.as_ref())?; + + // Explicitly request both columns, including the void one + let schema = snapshot.schema(); + let scan = snapshot.scan_builder().with_schema(schema).build()?; + + // The void column should be present in the logical schema (returned as null) + let logical_schema = scan.logical_schema(); + assert!(logical_schema.field("void_col").is_some()); + assert!(logical_schema.field("id").is_some()); + assert_eq!(logical_schema.fields().count(), 2); + + // Execute and verify void column appears as all-null + let batches = read_scan(&scan, engine)?; + assert_eq!(batches.len(), 1); + assert_eq!(batches[0].num_columns(), 2); + assert_eq!(batches[0].schema().field(0).name(), "id"); + assert_eq!(batches[0].schema().field(1).name(), "void_col"); + assert_eq!( + *batches[0].schema().field(1).data_type(), + ArrowDataType::Null + ); + + Ok(()) +} + +// Verify that void fields inside nested structs are preserved in the scan schema and data. +// Delta schema: {id: int, info: struct} +// Parquet schema: {id: int, info: struct} (void field missing from Parquet) +// Result: void field appears as all-null in the returned data. +#[tokio::test] +async fn read_table_with_void_in_nested_struct() -> Result<(), Box> { + // Parquet batch: {id: int, info: struct} (without the void field) + let name_array: ArrayRef = Arc::new(delta_kernel::arrow::array::StringArray::from(vec![ + "alice", "bob", + ])); + let inner_fields = vec![delta_kernel::arrow::datatypes::Field::new( + "name", + ArrowDataType::Utf8, + true, + )]; + let struct_array = + delta_kernel::arrow::array::StructArray::new(inner_fields.into(), vec![name_array], None); + let batch = RecordBatch::try_from_iter(vec![ + ( + "id", + Arc::new(delta_kernel::arrow::array::Int32Array::from(vec![1, 2])) as ArrayRef, + ), + ("info", Arc::new(struct_array) as ArrayRef), + ])?; + + let storage = Arc::new(InMemory::new()); + let actions = [ + r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#.to_string(), + r#"{"commitInfo":{"timestamp":1587968586154,"operation":"WRITE","operationParameters":{"mode":"ErrorIfExists","partitionBy":"[]"},"isBlindAppend":true}}"#.to_string(), + r#"{"metaData":{"id":"test-void-nested","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"info\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"v\",\"type\":\"void\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1587968585495}}"#.to_string(), + format!(r#"{{"add":{{"path":"{PARQUET_FILE1}","partitionValues":{{}},"size":0,"modificationTime":1587968586000,"dataChange":true}}}}"#), + ]; + + add_commit("memory:///", storage.as_ref(), 0, actions.iter().join("\n")).await?; + storage + .put( + &Path::from(PARQUET_FILE1), + record_batch_to_bytes(&batch).into(), + ) + .await?; + + let location = Url::parse("memory:///")?; + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + let snapshot = Snapshot::builder_for(location).build(engine.as_ref())?; + + // Table schema has the void field inside nested struct + let table_schema = snapshot.schema(); + let info_field = table_schema.field("info").expect("info should exist"); + if let DataType::Struct(inner) = info_field.data_type() { + assert!( + inner.field("v").is_some(), + "table schema should have void field 'v'" + ); + } + + let scan = snapshot.scan_builder().build()?; + + // Scan logical schema should preserve the void field in the nested struct + let logical_schema = scan.logical_schema(); + let info_field = logical_schema + .field("info") + .expect("info should exist in scan"); + if let DataType::Struct(inner) = info_field.data_type() { + assert!( + inner.field("v").is_some(), + "void field 'v' should be preserved" + ); + assert!( + inner.field("name").is_some(), + "non-void field 'name' should remain" + ); + assert_eq!(inner.fields().count(), 2); + } else { + panic!("info should be a struct type"); + } + + // Execute the scan and verify void field appears as null in the struct + let batches = read_scan(&scan, engine)?; + assert_eq!(batches.len(), 1); + assert_eq!(batches[0].num_rows(), 2); + + // The info struct should have 2 fields: name (string) and v (null) + let info_col = batches[0].column_by_name("info").expect("info column"); + let info_struct = info_col + .as_any() + .downcast_ref::() + .expect("info should be struct"); + assert_eq!(info_struct.num_columns(), 2); + assert_eq!(info_struct.column_by_name("name").unwrap().len(), 2); + let v_col = info_struct + .column_by_name("v") + .expect("v field should exist"); + assert_eq!(*v_col.data_type(), ArrowDataType::Null); + assert_eq!(v_col.len(), 2); + + Ok(()) +} + +// Integration test using a real Delta table created by Spark with a void column. +// Verifies that the kernel returns void column as all-null on reads. +#[test] +fn read_spark_table_with_void_column() -> Result<(), Box> { + read_table_data_str( + "./tests/data/void-column", + None, // SELECT * -- void column should appear as null + None, + vec![ + "+----+----------+", + "| id | void_col |", + "+----+----------+", + "| 1 | |", + "| 2 | |", + "| 3 | |", + "+----+----------+", + ], + ) +} + +// GE: file1 pruned (max=2s < adjusted 3.999401s), file2+3 kept +#[test] +fn timestamp_truncation_real_table_ge() -> Result<(), Box> { + read_table_data_str( + "./tests/data/timestamp-truncation-stats", + None, + Some(Pred::ge(col!("ts_col"), lit(Scalar::Timestamp(4_000_400)))), + vec![ + "+----+-----------------------------+", + "| id | ts_col |", + "+----+-----------------------------+", + "| 3 | 1970-01-01T00:00:03Z |", + "| 4 | 1970-01-01T00:00:04.000500Z |", + "| 5 | 1970-01-01T00:00:07Z |", + "| 6 | 1970-01-01T00:00:08Z |", + "+----+-----------------------------+", + ], + ) +} + +// LT: file3 pruned (min=7s > 4.000400s), file1+2 kept +#[test] +fn timestamp_truncation_real_table_lt() -> Result<(), Box> { + read_table_data_str( + "./tests/data/timestamp-truncation-stats", + None, + Some(Pred::lt(col!("ts_col"), lit(Scalar::Timestamp(4_000_400)))), + vec![ + "+----+-----------------------------+", + "| id | ts_col |", + "+----+-----------------------------+", + "| 1 | 1970-01-01T00:00:01Z |", + "| 2 | 1970-01-01T00:00:02Z |", + "| 3 | 1970-01-01T00:00:03Z |", + "| 4 | 1970-01-01T00:00:04.000500Z |", + "+----+-----------------------------+", + ], + ) +} + +// LE: file3 pruned (min=7s > 4.000400s), file1+2 kept +#[test] +fn timestamp_truncation_real_table_le() -> Result<(), Box> { + read_table_data_str( + "./tests/data/timestamp-truncation-stats", + None, + Some(Pred::le(col!("ts_col"), lit(Scalar::Timestamp(4_000_400)))), + vec![ + "+----+-----------------------------+", + "| id | ts_col |", + "+----+-----------------------------+", + "| 1 | 1970-01-01T00:00:01Z |", + "| 2 | 1970-01-01T00:00:02Z |", + "| 3 | 1970-01-01T00:00:03Z |", + "| 4 | 1970-01-01T00:00:04.000500Z |", + "+----+-----------------------------+", + ], + ) +} + +// EQ: file1 pruned (max=2s < adjusted 3.999401s), file3 pruned (min=7s > 4.000400s). +// Only file2 kept (ids 3,4). +#[test] +fn timestamp_truncation_real_table_eq() -> Result<(), Box> { + read_table_data_str( + "./tests/data/timestamp-truncation-stats", + None, + Some(Pred::eq(col!("ts_col"), lit(Scalar::Timestamp(4_000_400)))), + vec![ + "+----+-----------------------------+", + "| id | ts_col |", + "+----+-----------------------------+", + "| 3 | 1970-01-01T00:00:03Z |", + "| 4 | 1970-01-01T00:00:04.000500Z |", + "+----+-----------------------------+", + ], + ) +} + +async fn scan_with_void_schema( + schema_string: &str, + table_id: &str, + num_rows: i32, +) -> Result, Box> { + let batch = generate_batch(vec![( + "id", + (1..=num_rows).collect::>().into_arrow_array(), + )])?; + + let storage = Arc::new(InMemory::new()); + let actions = [ + r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#.to_string(), + format!( + r#"{{"metaData":{{"id":"{table_id}","format":{{"provider":"parquet","options":{{}}}},"schemaString":"{schema_string}","partitionColumns":[],"configuration":{{}},"createdTime":1587968585495}}}}"# + ), + format!( + r#"{{"add":{{"path":"{PARQUET_FILE1}","partitionValues":{{}},"size":0,"modificationTime":1587968586000,"dataChange":true}}}}"# + ), + ]; + + add_commit("memory:///", storage.as_ref(), 0, actions.iter().join("\n")).await?; + storage + .put( + &Path::from(PARQUET_FILE1), + record_batch_to_bytes(&batch).into(), + ) + .await?; + + let location = Url::parse("memory:///")?; + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + let snapshot = Snapshot::builder_for(location).build(engine.as_ref())?; + let scan = snapshot.scan_builder().build()?; + Ok(read_scan(&scan, engine)?) +} + +// Verify that a table with void nested inside an Array can be read at runtime. +// Write-time validation rejects void-in-array, but reads and metadata ops always work. +// The arr column is absent from the Parquet schema; kernel synthesizes a null column at read +// time (the entire column is NULL, not an array of null elements). +#[tokio::test] +async fn read_void_in_array_type_ok() -> Result<(), Box> { + let schema_string = r#"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"arr\",\"type\":{\"type\":\"array\",\"elementType\":\"void\",\"containsNull\":true},\"nullable\":true,\"metadata\":{}}]}"#; + let batches = scan_with_void_schema(schema_string, "test-void-array", 2).await?; + assert_eq!(batches.len(), 1); + assert_eq!(batches[0].num_rows(), 2); + + let arr_col = batches[0] + .column_by_name("arr") + .expect("arr column must be in RecordBatch"); + assert_eq!( + arr_col.null_count(), + arr_col.len(), + "entire column should be null" + ); + match arr_col.data_type() { + ArrowDataType::List(elem) => assert_eq!(*elem.data_type(), ArrowDataType::Null), + other => panic!("expected List, got {other:?}"), + } + + Ok(()) +} + +// Verify that a table with void nested inside a Map value can be read at runtime. +// Write-time validation rejects void-in-map, but reads and metadata ops always work. +// The m column is absent from the Parquet schema; kernel synthesizes a null column at read +// time (the entire column is NULL, not a map with null entries). +#[tokio::test] +async fn read_void_in_map_type_ok() -> Result<(), Box> { + let schema_string = r#"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"m\",\"type\":{\"type\":\"map\",\"keyType\":\"string\",\"valueType\":\"void\",\"valueContainsNull\":true},\"nullable\":true,\"metadata\":{}}]}"#; + let batches = scan_with_void_schema(schema_string, "test-void-map", 2).await?; + assert_eq!(batches.len(), 1); + assert_eq!(batches[0].num_rows(), 2); + + let m_col = batches[0] + .column_by_name("m") + .expect("m column must be in RecordBatch"); + assert_eq!( + m_col.null_count(), + m_col.len(), + "entire column should be null" + ); + match m_col.data_type() { + ArrowDataType::Map(entries, _) => { + let entries = entries.data_type(); + match entries { + ArrowDataType::Struct(fields) => { + assert_eq!(*fields[0].data_type(), ArrowDataType::Utf8); + assert_eq!(*fields[1].data_type(), ArrowDataType::Null); + } + other => panic!("expected Struct inside Map, got {other:?}"), + } + } + other => panic!("expected Map, got {other:?}"), + } + + Ok(()) +} + +// Verify runtime read of a struct where all fields are void. +// Delta schema: {id: int, s: struct} +// Parquet has only {id} — the entire struct is missing and must materialize as null. +#[tokio::test] +async fn read_all_void_struct() -> Result<(), Box> { + let schema_string = r#"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"s\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"x\",\"type\":\"void\",\"nullable\":true,\"metadata\":{}},{\"name\":\"y\",\"type\":\"void\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}}]}"#; + let batches = scan_with_void_schema(schema_string, "test-all-void-struct", 3).await?; + assert_eq!(batches.len(), 1); + assert_eq!(batches[0].num_rows(), 3); + + let s_col = batches[0].column_by_name("s").expect("s column"); + let s_struct = s_col + .as_any() + .downcast_ref::() + .expect("s should be struct"); + assert_eq!(s_struct.num_columns(), 2); + + let x_col = s_struct.column_by_name("x").expect("x field"); + let y_col = s_struct.column_by_name("y").expect("y field"); + assert_eq!(*x_col.data_type(), ArrowDataType::Null); + assert_eq!(*y_col.data_type(), ArrowDataType::Null); + assert_eq!(x_col.len(), 3); + assert_eq!(y_col.len(), 3); + + Ok(()) +} + +// Verify that a table where ALL field are void can still be read (returns all-null rows). +// Reads always succeed; only writes fail for all-void tables. +#[tokio::test] +async fn read_all_void_table() -> Result<(), Box> { + // Empty Parquet file (no columns) — the row count comes from the Parquet metadata + let batch = RecordBatch::new_empty(Arc::new(ArrowSchema::empty())); + + let storage = Arc::new(InMemory::new()); + let actions = [ + r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#.to_string(), + r#"{"commitInfo":{"timestamp":1587968586154,"operation":"WRITE","operationParameters":{"mode":"ErrorIfExists","partitionBy":"[]"},"isBlindAppend":true}}"#.to_string(), + r#"{"metaData":{"id":"test-all-void","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"a\",\"type\":\"void\",\"nullable\":true,\"metadata\":{}},{\"name\":\"b\",\"type\":\"void\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1587968585495}}"#.to_string(), + format!(r#"{{"add":{{"path":"{PARQUET_FILE1}","partitionValues":{{}},"size":0,"modificationTime":1587968586000,"dataChange":true}}}}"#), + ]; + + add_commit("memory:///", storage.as_ref(), 0, actions.iter().join("\n")).await?; + storage + .put( + &Path::from(PARQUET_FILE1), + record_batch_to_bytes(&batch).into(), + ) + .await?; + + let location = Url::parse("memory:///")?; + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + let snapshot = Snapshot::builder_for(location).build(engine.as_ref())?; + + // Schema should contain both void columns + assert_eq!(snapshot.schema().fields().count(), 2); + assert!(snapshot.schema().field("a").is_some()); + assert!(snapshot.schema().field("b").is_some()); + + let scan = snapshot.scan_builder().build()?; + let logical_schema = scan.logical_schema(); + assert_eq!(logical_schema.fields().count(), 2); + + // Execute read_scan — runtime must not crash even with all-void schema. + // Parquet file has 0 columns/0 rows → empty row group → reader produces no batches. + // The key verification is that read_scan() doesn't error. + let batches = read_scan(&scan, engine)?; + let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total_rows, 0); + + Ok(()) +} + +// Verify that a void column used as a partition column works on reads. +// The partition value is missing (null), so parse_partition_value_raw returns Scalar::Null(VOID). +#[tokio::test] +async fn read_table_with_void_partition_column() -> Result<(), Box> { + // Parquet file has only the non-partition, non-void column + let batch = generate_batch(vec![("id", vec![1, 2].into_arrow_array())])?; + + let storage = Arc::new(InMemory::new()); + let actions = [ + r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#.to_string(), + r#"{"commitInfo":{"timestamp":1587968586154,"operation":"WRITE","operationParameters":{"mode":"ErrorIfExists","partitionBy":"[\"void_part\"]"},"isBlindAppend":true}}"#.to_string(), + r#"{"metaData":{"id":"test-void-part","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"void_part\",\"type\":\"void\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["void_part"],"configuration":{},"createdTime":1587968585495}}"#.to_string(), + format!(r#"{{"add":{{"path":"{PARQUET_FILE1}","partitionValues":{{}},"size":0,"modificationTime":1587968586000,"dataChange":true}}}}"#), + ]; + + add_commit("memory:///", storage.as_ref(), 0, actions.iter().join("\n")).await?; + storage + .put( + &Path::from(PARQUET_FILE1), + record_batch_to_bytes(&batch).into(), + ) + .await?; + + let location = Url::parse("memory:///")?; + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + let snapshot = Snapshot::builder_for(location).build(engine.as_ref())?; + + let scan = snapshot.scan_builder().build()?; + let logical_schema = scan.logical_schema(); + assert!(logical_schema.field("void_part").is_some()); + assert!(logical_schema.field("id").is_some()); + + // Execute and verify both columns appear + let batches = read_scan(&scan, engine)?; + assert_eq!(batches.len(), 1); + assert_eq!(batches[0].num_rows(), 2); + // void partition column should appear as Null type + let void_col = batches[0] + .column_by_name("void_part") + .expect("void partition column"); + assert_eq!(*void_col.data_type(), ArrowDataType::Null); + assert_eq!(void_col.len(), 2); + + Ok(()) +} + +// Verify that predicate pushdown on a void column works correctly. +// A predicate like `void_col IS NULL` is always true for void columns, so no rows are skipped. +#[tokio::test] +async fn read_with_predicate_on_void_column() -> Result<(), Box> { + let batch = generate_batch(vec![("id", vec![1, 2, 3].into_arrow_array())])?; + + let storage = Arc::new(InMemory::new()); + let actions = [ + r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#.to_string(), + r#"{"commitInfo":{"timestamp":1587968586154,"operation":"WRITE","operationParameters":{"mode":"ErrorIfExists","partitionBy":"[]"},"isBlindAppend":true}}"#.to_string(), + r#"{"metaData":{"id":"test-void-pred","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"void_col\",\"type\":\"void\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1587968585495}}"#.to_string(), + format!(r#"{{"add":{{"path":"{PARQUET_FILE1}","partitionValues":{{}},"size":0,"modificationTime":1587968586000,"dataChange":true}}}}"#), + ]; + + add_commit("memory:///", storage.as_ref(), 0, actions.iter().join("\n")).await?; + storage + .put( + &Path::from(PARQUET_FILE1), + record_batch_to_bytes(&batch).into(), + ) + .await?; + + let location = Url::parse("memory:///")?; + let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); + let snapshot = Snapshot::builder_for(location).build(engine.as_ref())?; + + // Predicate: void_col IS NULL — always true for void, should return all rows + let predicate = Arc::new(col!("void_col").is_null()); + let scan = snapshot + .clone() + .scan_builder() + .with_predicate(predicate) + .build()?; + + let batches = read_scan(&scan, engine.clone())?; + let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total_rows, 3, "IS NULL on void should return all rows"); + + // Predicate: void_col IS NOT NULL — always false for void. The Add action above has no + // `stats` string, so kernel has nothing to skip on. All rows are returned. + // Skipping driven by `nullCount` is exercised by `void_predicate_skips_via_null_count`. + let predicate_not_null = Arc::new(col!("void_col").is_not_null()); + let scan_not_null = snapshot + .scan_builder() + .with_predicate(predicate_not_null) + .build()?; + let batches_not_null = read_scan(&scan_not_null, engine)?; + let total_rows_not_null: usize = batches_not_null.iter().map(|b| b.num_rows()).sum(); + assert_eq!( + total_rows_not_null, 3, + "IS NOT NULL on void: no row-level filtering, all rows returned" + ); + + Ok(()) +} + +// File-level skipping via the `nullCount` Delta stat for void columns. The Spark fixture's 3 +// Add actions each carry stats with `nullCount.void_col == numRecords == 1`, so kernel can prune +// every file for `IS NOT NULL`. Contrast with `read_with_predicate_on_void_column`, whose Add +// action has no `stats` and consequently returns all 3 rows for the same predicate -- confirming +// that pruning (not row-level filtering) is what produces the empty result here. +#[rstest::rstest] +#[case::is_null( + col!("void_col").is_null(), + vec![ + "+----+----------+", + "| id | void_col |", + "+----+----------+", + "| 1 | |", + "| 2 | |", + "| 3 | |", + "+----+----------+", + ] +)] +#[case::is_not_null(col!("void_col").is_not_null(), vec![])] +fn void_predicate_skips_via_null_count( + #[case] predicate: Pred, + #[case] expected: Vec<&str>, +) -> Result<(), Box> { + read_table_data_str("./tests/data/void-column", None, Some(predicate), expected) +} + +// Direct evidence that void-column predicates drive file-level pruning: count the surviving +// ScanFiles after `scan_metadata` rather than relying on the empty-batches contrast. With the +// Spark fixture's `nullCount.void_col == numRecords` stats, `IS NOT NULL` prunes all 3 files +// and `IS NULL` keeps all 3. +#[rstest::rstest] +#[case::is_null_keeps_all(col!("void_col").is_null(), 3)] +#[case::is_not_null_prunes_all(col!("void_col").is_not_null(), 0)] +fn void_predicate_pruning_scan_file_count( + #[case] predicate: Pred, + #[case] expected_files: usize, +) -> Result<(), Box> { + let path = std::fs::canonicalize(PathBuf::from("./tests/data/void-column"))?; + let url = Url::from_directory_path(path).unwrap(); + let engine = test_utils::create_default_engine(&url)?; + let snapshot = Snapshot::builder_for(url).build(engine.as_ref())?; + let scan = snapshot + .scan_builder() + .with_predicate(Arc::new(predicate)) + .build()?; + + let mut scan_files: Vec = vec![]; + for res in scan.scan_metadata(engine.as_ref())? { + scan_files = res?.visit_scan_files(scan_files, scan_metadata_callback)?; + } + assert_eq!(scan_files.len(), expected_files); + Ok(()) +} diff --git a/kernel/tests/integration/read/scan_cancellation.rs b/kernel/tests/integration/read/scan_cancellation.rs new file mode 100644 index 0000000000..4bcf69e9ae --- /dev/null +++ b/kernel/tests/integration/read/scan_cancellation.rs @@ -0,0 +1,214 @@ +//! Integration coverage for `ScanBuilder::with_cancellation_token`: a cancelled scan must +//! surface `Error::Cancelled` through the real Default Engine and can never be mistaken for a +//! complete listing. + +use std::sync::Arc; + +use delta_kernel::object_store::memory::InMemory; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::ObjectStoreExt as _; +use delta_kernel::scan::StatsOptions; +use delta_kernel::{CancellationTokenRef, Error, Snapshot}; +use rstest::rstest; +use test_utils::delta_kernel_default_engine::DefaultEngineBuilder; +use test_utils::{ + actions_to_string, add_commit, generate_simple_batch, load_test_data, record_batch_to_bytes, + TestAction, TestCancellationToken, +}; + +const PARQUET_FILE1: &str = "part-00000-a72b1fb3-f2df-41fe-a8f0-e65b746382dd-c000.snappy.parquet"; + +/// Builds a two-commit JSON-log table (no checkpoint) in memory and returns `(storage, root)`. +async fn json_only_table() -> Result<(Arc, &'static str), Box> { + let batch = generate_simple_batch()?; + let storage = Arc::new(InMemory::new()); + let table_root = "memory:///"; + let file_size = record_batch_to_bytes(&batch).len() as u64; + add_commit( + table_root, + storage.as_ref(), + 0, + actions_to_string(vec![ + TestAction::Metadata, + TestAction::AddWithSize(PARQUET_FILE1.to_string(), file_size), + ]), + ) + .await?; + storage + .put( + &Path::from(PARQUET_FILE1), + record_batch_to_bytes(&batch).into(), + ) + .await?; + Ok((storage, table_root)) +} + +// A scan whose builder was given an already-cancelled token yields exactly one +// `Error::Cancelled` and then ends -- it never produces a (partial) complete-looking listing. +// Parametrized over stats mode because a predicate/stats scan takes a different replay path +// (checkpoint parquet reads + stats parsing) than the JSON-only default, and both must honor the +// token: `None` is the plain default (no `with_stats` call), `Some` opts into struct stats. +#[rstest] +#[case::json_default(None)] +#[case::with_stats(Some(StatsOptions::all_struct()))] +#[tokio::test] +async fn precancelled_scan_yields_cancelled( + #[case] stats: Option, +) -> Result<(), Box> { + let (storage, table_root) = json_only_table().await?; + let engine = DefaultEngineBuilder::new(storage).build(); + let snapshot = Snapshot::builder_for(table_root).build(&engine)?; + + let token: CancellationTokenRef = Arc::new(TestCancellationToken::cancelled()); + let mut builder = snapshot.scan_builder().with_cancellation_token(token); + if let Some(stats) = stats { + builder = builder.with_stats(stats); + } + let scan = builder.build()?; + + // Cancellation surfaces as `Error::Cancelled`, never as a complete listing. It may arrive + // either from the eager setup reads that `scan_metadata` performs (returning `Err` directly) + // or as the iterator's terminal item -- assert whichever, and that no successful batch and no + // silent `None`-only stream is ever produced. + assert_cancelled(scan.scan_metadata(&engine)); + Ok(()) +} + +/// Asserts that a scan_metadata result represents cancellation: either the call itself returned +/// `Err(Cancelled)`, or the iterator yields `Err(Cancelled)` before any `Ok` and then fuses. +fn assert_cancelled< + I: Iterator>, +>( + result: delta_kernel::DeltaResult, +) { + match result { + Err(Error::Cancelled) => {} + Err(other) => panic!("expected Cancelled, got {other:?}"), + Ok(mut iter) => { + assert!( + matches!(iter.next(), Some(Err(Error::Cancelled))), + "cancelled scan must yield Err(Cancelled), never an Ok batch or bare None" + ); + assert!( + iter.next().is_none(), + "iterator must fuse after cancellation" + ); + } + } +} + +// Control: the same scan with no cancellation token (the default) completes normally, proving +// the cancellation path is opt-in and does not otherwise change behavior. +#[tokio::test] +async fn uncancelled_json_scan_completes() -> Result<(), Box> { + let (storage, table_root) = json_only_table().await?; + let engine = DefaultEngineBuilder::new(storage).build(); + let snapshot = Snapshot::builder_for(table_root).build(&engine)?; + + let scan = snapshot.clone().scan_builder().build()?; + let count = scan.scan_metadata(&engine)?.filter(|r| r.is_ok()).count(); + assert!(count > 0, "uncancelled scan should yield scan metadata"); + + // An uncancelled token behaves identically. + let token: CancellationTokenRef = Arc::new(TestCancellationToken::default()); + let scan = snapshot + .scan_builder() + .with_cancellation_token(token) + .build()?; + for res in scan.scan_metadata(&engine)? { + assert!(res.is_ok(), "uncancelled token must not inject errors"); + } + Ok(()) +} + +// Cancelling a LIVE token after iteration has started surfaces exactly ONE terminal +// `Err(Cancelled)` and then fuses -- exercising both cancellation layers (kernel batch-boundary +// poll + the engine yielding its own cancelled error), the composition the pre-cancelled tests +// can't reach. Guards against the double-emit the two layers would otherwise produce. +#[tokio::test(flavor = "multi_thread")] +async fn mid_stream_cancellation_yields_exactly_one_error() -> Result<(), Box> +{ + let (storage, table_root) = json_only_table().await?; + let engine = DefaultEngineBuilder::new(storage).build(); + let snapshot = Snapshot::builder_for(table_root).build(&engine)?; + + let token = Arc::new(TestCancellationToken::default()); + let scan = snapshot + .scan_builder() + .with_cancellation_token(token.clone() as CancellationTokenRef) + .build()?; + + let mut iter = scan.scan_metadata(&engine)?; + assert!( + matches!(iter.next(), Some(Ok(_))), + "expected an Ok batch before cancellation" + ); + + token.cancel(); + + assert!(matches!(iter.next(), Some(Err(Error::Cancelled)))); + assert!( + iter.next().is_none(), + "iterator must fuse after the single error" + ); + Ok(()) +} + +// A pre-cancelled scan over a table WITH a checkpoint surfaces `Err(Cancelled)` -- reaching the +// checkpoint/sidecar/footer `check_cancelled` guards that the JSON-only fixtures never enter. +// `packed` distinguishes a `.tar.zst` fixture (unpacked to a temp dir) from a plain directory. +#[rstest] +#[case::v1_checkpoint("with_checkpoint_no_last_checkpoint", false)] +#[case::v2_parquet_sidecars("v2-checkpoints-parquet-with-sidecars", true)] +#[tokio::test] +async fn precancelled_scan_over_checkpoint_yields_cancelled( + #[case] test_name: &str, + #[case] packed: bool, +) -> Result<(), Box> { + // `_tempdir` holds the unpacked fixture (packed case) for the test's lifetime. + let (table_path, _tempdir) = if packed { + let dir = load_test_data("./tests/data", test_name)?; + let path = dir.path().join(test_name); + (path, Some(dir)) + } else { + ( + std::path::PathBuf::from(format!("./tests/data/{test_name}")), + None, + ) + }; + let url = url::Url::from_directory_path(std::fs::canonicalize(table_path)?).unwrap(); + let engine = test_utils::create_default_engine(&url)?; + + let snapshot = Snapshot::builder_for(url).build(engine.as_ref())?; + let token: CancellationTokenRef = Arc::new(TestCancellationToken::cancelled()); + let scan = snapshot + .scan_builder() + .with_cancellation_token(token) + .build()?; + + assert_cancelled(scan.scan_metadata(engine.as_ref())); + Ok(()) +} + +// `parallel_scan_metadata` does not support cancellation; setting a token makes it error rather +// than silently run to completion. +#[tokio::test] +async fn parallel_scan_metadata_errors_when_token_set() -> Result<(), Box> { + let (storage, table_root) = json_only_table().await?; + let engine: Arc = + Arc::new(DefaultEngineBuilder::new(storage).build()); + let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; + + let token: CancellationTokenRef = Arc::new(TestCancellationToken::default()); + let scan = snapshot + .scan_builder() + .with_cancellation_token(token) + .build()?; + + let result = scan.parallel_scan_metadata(engine); + assert!( + matches!(result, Err(Error::Unsupported(_))), + "parallel_scan_metadata must reject a cancellation token" + ); + Ok(()) +} diff --git a/kernel/tests/integration/write/append.rs b/kernel/tests/integration/write/append.rs new file mode 100644 index 0000000000..2add502dd0 --- /dev/null +++ b/kernel/tests/integration/write/append.rs @@ -0,0 +1,539 @@ +//! Integration tests for basic append write paths (single writes, repeated writes, +//! partitioned appends, and schema-mismatch rejection). + +use std::collections::HashMap; +use std::sync::Arc; + +use delta_kernel::arrow::array::{new_null_array, Int32Array, StringArray}; +use delta_kernel::arrow::datatypes::{Field as ArrowField, Schema as ArrowSchema}; +use delta_kernel::arrow::error::ArrowError; +use delta_kernel::arrow::record_batch::RecordBatch; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::engine::arrow_conversion::TryIntoArrow as _; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::expressions::Scalar; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::ObjectStoreExt as _; +use delta_kernel::schema::{schema_ref, DataType, StructField, StructType}; +use delta_kernel::table_features::ColumnMappingMode; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::transaction::data_layout::DataLayout; +use delta_kernel::transaction::Transaction; +use delta_kernel::{DeltaResult, Error as KernelError, Snapshot}; +use itertools::Itertools; +use rstest::rstest; +use serde_json::{json, Deserializer}; +use test_utils::{ + assert_result_error_with_message, into_record_batch, load_and_begin_transaction, + modify_add_file_partition_keys, set_json_value, setup_test_tables, test_read, + AddFilePartitionKeyModify, +}; + +use crate::common::write_utils::{ + check_action_timestamps, get_and_check_all_parquet_sizes, get_simple_int_schema, + validate_txn_id, write_data_and_check_result_and_stats, ZERO_UUID, +}; + +#[tokio::test] +async fn test_append() -> Result<(), Box> { + // setup tracing + let _ = tracing_subscriber::fmt::try_init(); + // create a simple table: one int column named 'number' + let schema = get_simple_int_schema(); + + for (table_url, engine, store, table_name) in + setup_test_tables(schema.clone(), &[], None, "test_table").await? + { + // write data out by spawning async tasks to simulate executors + let engine = Arc::new(engine); + write_data_and_check_result_and_stats(table_url.clone(), schema.clone(), engine.clone(), 1) + .await?; + + let commit1 = store + .get(&Path::from(format!( + "/{table_name}/_delta_log/00000000000000000001.json" + ))) + .await?; + + let mut parsed_commits: Vec<_> = Deserializer::from_slice(&commit1.bytes().await?) + .into_iter::() + .try_collect()?; + + let size = + get_and_check_all_parquet_sizes(store.clone(), format!("/{table_name}/").as_str()) + .await; + // check that the timestamps in commit_info and add actions are within 10s of + // SystemTime::now() before we clear them for comparison + check_action_timestamps(parsed_commits.iter())?; + // check that the txn_id is valid before we clear it for comparison + validate_txn_id(&parsed_commits[0]["commitInfo"]); + + // set timestamps to 0, paths and txn_id to known string values for comparison + // (otherwise timestamps are non-deterministic, paths and txn_id are random UUIDs) + set_json_value(&mut parsed_commits[0], "commitInfo.timestamp", json!(0))?; + set_json_value(&mut parsed_commits[0], "commitInfo.txnId", json!(ZERO_UUID))?; + set_json_value(&mut parsed_commits[1], "add.modificationTime", json!(0))?; + set_json_value(&mut parsed_commits[1], "add.path", json!("first.parquet"))?; + set_json_value(&mut parsed_commits[2], "add.modificationTime", json!(0))?; + set_json_value(&mut parsed_commits[2], "add.path", json!("second.parquet"))?; + + let expected_commit = vec![ + json!({ + "commitInfo": { + "timestamp": 0, + "operation": "UNKNOWN", + "kernelVersion": format!("v{}", env!("CARGO_PKG_VERSION")), + "operationParameters": {}, + "txnId": ZERO_UUID + } + }), + json!({ + "add": { + "path": "first.parquet", + "partitionValues": {}, + "size": size, + "modificationTime": 0, + "dataChange": true, + "stats": "{\"numRecords\":3,\"nullCount\":{\"number\":0},\"minValues\":{\"number\":1},\"maxValues\":{\"number\":3},\"tightBounds\":true}" + } + }), + json!({ + "add": { + "path": "second.parquet", + "partitionValues": {}, + "size": size, + "modificationTime": 0, + "dataChange": true, + "stats": "{\"numRecords\":3,\"nullCount\":{\"number\":0},\"minValues\":{\"number\":4},\"maxValues\":{\"number\":6},\"tightBounds\":true}" + } + }), + ]; + + assert_eq!(parsed_commits, expected_commit); + + test_read( + &ArrowEngineData::new(RecordBatch::try_new( + Arc::new(schema.as_ref().try_into_arrow()?), + vec![Arc::new(Int32Array::from(vec![1, 2, 3, 4, 5, 6]))], + )?), + &table_url, + engine, + )?; + } + Ok(()) +} + +#[tokio::test] +async fn test_no_add_actions() -> Result<(), Box> { + // setup tracing + let _ = tracing_subscriber::fmt::try_init(); + // create a simple table: one int column named 'number' + let schema = get_simple_int_schema(); + + for (table_url, engine, store, table_name) in + setup_test_tables(schema.clone(), &[], None, "test_table").await? + { + let txn = load_and_begin_transaction(table_url.clone(), &engine)? + .with_engine_info("default engine"); + + // Commit without adding any add files + assert!(txn.commit(&engine)?.is_committed()); + + let commit1 = store + .get(&Path::from(format!( + "/{table_name}/_delta_log/00000000000000000001.json" + ))) + .await?; + + let parsed_actions: Vec<_> = Deserializer::from_slice(&commit1.bytes().await?) + .into_iter::() + .try_collect()?; + + // Verify that there only is a commit info action + assert_eq!(parsed_actions.len(), 1, "Expected only one action"); + assert!(parsed_actions[0].get("commitInfo").is_some()); + } + Ok(()) +} + +#[tokio::test] +async fn test_append_twice() -> Result<(), Box> { + // setup tracing + let _ = tracing_subscriber::fmt::try_init(); + // create a simple table: one int column named 'number' + let schema = get_simple_int_schema(); + + for (table_url, engine, _, _) in + setup_test_tables(schema.clone(), &[], None, "test_table").await? + { + let engine = Arc::new(engine); + write_data_and_check_result_and_stats(table_url.clone(), schema.clone(), engine.clone(), 1) + .await?; + write_data_and_check_result_and_stats(table_url.clone(), schema.clone(), engine.clone(), 2) + .await?; + + test_read( + &ArrowEngineData::new(RecordBatch::try_new( + Arc::new(schema.as_ref().try_into_arrow()?), + vec![Arc::new(Int32Array::from(vec![ + 1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6, + ]))], + )?), + &table_url, + engine, + )?; + } + Ok(()) +} + +#[tokio::test] +async fn test_append_partitioned() -> Result<(), Box> { + // setup tracing + let _ = tracing_subscriber::fmt::try_init(); + + let partition_col = "partition"; + + // create a simple partitioned table: one int column named 'number', partitioned by string + // column named 'partition' + let table_schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("number", DataType::INTEGER), + StructField::nullable("partition", DataType::STRING), + ])?); + let data_schema = schema_ref! { nullable "number": INTEGER }; + + for (table_url, engine, store, table_name) in + setup_test_tables(table_schema.clone(), &[partition_col], None, "test_table").await? + { + let mut txn = load_and_begin_transaction(table_url.clone(), &engine)? + .with_engine_info("default engine") + .with_data_change(false); + + // create two new arrow record batches to append + let append_data = [[1, 2, 3], [4, 5, 6]].map(|data| -> DeltaResult<_> { + let data = RecordBatch::try_new( + Arc::new(data_schema.as_ref().try_into_arrow()?), + vec![Arc::new(Int32Array::from(data.to_vec()))], + )?; + Ok(Box::new(ArrowEngineData::new(data))) + }); + let partition_vals = vec!["a", "b"]; + + // write data out by spawning async tasks to simulate executors + let engine = Arc::new(engine); + let tasks = append_data + .into_iter() + .zip(partition_vals) + .map(|(data, partition_val)| { + let write_context = Arc::new( + txn.partitioned_write_context(HashMap::from([( + partition_col.to_string(), + Scalar::String(partition_val.into()), + )])) + .unwrap(), + ); + // arc clones + let engine = engine.clone(); + tokio::task::spawn(async move { + engine + .write_parquet(data.as_ref().unwrap(), write_context.as_ref()) + .await + }) + }); + + let add_files_metadata = futures::future::join_all(tasks).await.into_iter().flatten(); + for meta in add_files_metadata { + txn.add_files(meta?); + } + + // commit! + assert!(txn.commit(engine.as_ref())?.is_committed()); + + let commit1 = store + .get(&Path::from(format!( + "/{table_name}/_delta_log/00000000000000000001.json" + ))) + .await?; + + let mut parsed_commits: Vec<_> = Deserializer::from_slice(&commit1.bytes().await?) + .into_iter::() + .try_collect()?; + + let size = + get_and_check_all_parquet_sizes(store.clone(), format!("/{table_name}/").as_str()) + .await; + // check that the timestamps in commit_info and add actions are within 10s of + // SystemTime::now() before we clear them for comparison + check_action_timestamps(parsed_commits.iter())?; + // check that the txn_id is valid before we clear it for comparison + validate_txn_id(&parsed_commits[0]["commitInfo"]); + + // set timestamps to 0, paths and txn_id to known string values for comparison + // (otherwise timestamps are non-deterministic, paths and txn_id are random UUIDs) + set_json_value(&mut parsed_commits[0], "commitInfo.timestamp", json!(0))?; + set_json_value(&mut parsed_commits[0], "commitInfo.txnId", json!(ZERO_UUID))?; + set_json_value(&mut parsed_commits[1], "add.modificationTime", json!(0))?; + set_json_value(&mut parsed_commits[1], "add.path", json!("first.parquet"))?; + set_json_value(&mut parsed_commits[2], "add.modificationTime", json!(0))?; + set_json_value(&mut parsed_commits[2], "add.path", json!("second.parquet"))?; + + let expected_commit = vec![ + json!({ + "commitInfo": { + "timestamp": 0, + "operation": "UNKNOWN", + "kernelVersion": format!("v{}", env!("CARGO_PKG_VERSION")), + "operationParameters": {}, + "engineInfo": "default engine", + "txnId": ZERO_UUID + } + }), + json!({ + "add": { + "path": "first.parquet", + "partitionValues": { + "partition": "a" + }, + "size": size, + "modificationTime": 0, + "dataChange": false, + "stats": "{\"numRecords\":3,\"nullCount\":{\"number\":0},\"minValues\":{\"number\":1},\"maxValues\":{\"number\":3},\"tightBounds\":true}" + } + }), + json!({ + "add": { + "path": "second.parquet", + "partitionValues": { + "partition": "b" + }, + "size": size, + "modificationTime": 0, + "dataChange": false, + "stats": "{\"numRecords\":3,\"nullCount\":{\"number\":0},\"minValues\":{\"number\":4},\"maxValues\":{\"number\":6},\"tightBounds\":true}" + } + }), + ]; + + assert_eq!(parsed_commits, expected_commit); + + test_read( + &ArrowEngineData::new(RecordBatch::try_new( + Arc::new(table_schema.as_ref().try_into_arrow()?), + vec![ + Arc::new(Int32Array::from(vec![1, 2, 3, 4, 5, 6])), + Arc::new(StringArray::from(vec!["a", "a", "a", "b", "b", "b"])), + ], + )?), + &table_url, + engine, + )?; + } + Ok(()) +} + +#[tokio::test] +async fn test_append_invalid_schema() -> Result<(), Box> { + // setup tracing + let _ = tracing_subscriber::fmt::try_init(); + // create a simple table: one int column named 'number' + let table_schema = schema_ref! { nullable "number": INTEGER }; + // incompatible data schema: one string column named 'string' + let data_schema = schema_ref! { nullable "string": STRING }; + + for (table_url, engine, _store, _table_name) in + setup_test_tables(table_schema, &[], None, "test_table").await? + { + let txn = load_and_begin_transaction(table_url.clone(), &engine)? + .with_engine_info("default engine"); + + // create two new arrow record batches to append + let append_data = [["a", "b"], ["c", "d"]].map(|data| -> DeltaResult<_> { + let data = RecordBatch::try_new( + Arc::new(data_schema.as_ref().try_into_arrow()?), + vec![Arc::new(StringArray::from(data.to_vec()))], + )?; + Ok(Box::new(ArrowEngineData::new(data))) + }); + + // write data out by spawning async tasks to simulate executors + let engine = Arc::new(engine); + let write_context = Arc::new(txn.unpartitioned_write_context().unwrap()); + let tasks = append_data.into_iter().map(|data| { + // arc clones + let engine = engine.clone(); + let write_context = write_context.clone(); + tokio::task::spawn(async move { + engine + .write_parquet(data.as_ref().unwrap(), write_context.as_ref()) + .await + }) + }); + + let mut add_files_metadata = futures::future::join_all(tasks).await.into_iter().flatten(); + assert!(add_files_metadata.all(|res| match res { + Err(KernelError::Arrow(ArrowError::InvalidArgumentError(_))) => true, + Err(KernelError::Backtraced { source, .. }) + if matches!( + &*source, + KernelError::Arrow(ArrowError::InvalidArgumentError(_)) + ) => + true, + _ => false, + })); + } + Ok(()) +} + +#[tokio::test] +async fn commit_rejects_add_missing_required_field() -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + let schema = get_simple_int_schema(); + + for field in ["path", "partitionValues", "size", "modificationTime"] { + let (table_url, engine, _store, _table_name) = + setup_test_tables(schema.clone(), &[], None, "required_field_table") + .await? + .into_iter() + .next() + .expect("at least one test table"); + let engine = Arc::new(engine); + let mut txn = + load_and_begin_transaction(table_url, engine.as_ref())?.with_data_change(true); + + let data = ArrowEngineData::new(RecordBatch::try_new( + Arc::new(schema.as_ref().try_into_arrow()?), + vec![Arc::new(Int32Array::from(vec![1, 2, 3]))], + )?); + let write_context = Arc::new(txn.unpartitioned_write_context()?); + + // Corrupt the addFile at the second batch. + let valid_meta = engine.write_parquet(&data, write_context.as_ref()).await?; + txn.add_files(valid_meta); + let to_be_corrupted_meta = engine.write_parquet(&data, write_context.as_ref()).await?; + + let batch = into_record_batch(to_be_corrupted_meta); + let index = batch.schema().index_of(field)?; + + // The add-metadata schema declares these fields non-nullable, so rebuild the schema with + // the target field made nullable before inserting a null column. + let mut fields: Vec = batch + .schema() + .fields() + .iter() + .map(|f| f.as_ref().clone()) + .collect(); + fields[index] = fields[index].clone().with_nullable(true); + let nullable_schema = Arc::new(ArrowSchema::new(fields)); + let mut columns = batch.columns().to_vec(); + columns[index] = new_null_array(nullable_schema.field(index).data_type(), batch.num_rows()); + let corrupted = RecordBatch::try_new(nullable_schema, columns)?; + txn.add_files(Box::new(ArrowEngineData::new(corrupted))); + + let err = txn + .commit(engine.as_ref()) + .expect_err(&format!( + "commit should reject an add missing required field '{field}'" + )) + .to_string(); + assert!( + err.contains(&format!("missing required field '{field}'")), + "field {field}: unexpected error {err:?}" + ); + } + Ok(()) +} + +#[rstest] +#[case::missing(None, &[AddFilePartitionKeyModify::Drop { key: "p2" }])] +#[case::extra(None, &[AddFilePartitionKeyModify::Insert { + key: "p3", + value: Some("extra"), +}])] +#[case::incorrect_name(None, &[ + AddFilePartitionKeyModify::Drop { key: "p2" }, + AddFilePartitionKeyModify::Insert { + key: "partition_2", + value: Some("6"), + }, +])] +#[case::logical_partition_name_when_cm_name_mode( + Some("name"), + &[ + AddFilePartitionKeyModify::Drop { key: "p2" }, + AddFilePartitionKeyModify::Insert { key: "p2", value: Some("6") }, + ], +)] +#[case::logical_partition_name_when_cm_id_mode( + Some("id"), + &[ + AddFilePartitionKeyModify::Drop { key: "p2" }, + AddFilePartitionKeyModify::Insert { key: "p2", value: Some("6") }, + ], +)] +#[tokio::test(flavor = "multi_thread")] +async fn commit_rejects_add_with_invalid_partition_keys( + #[case] column_mapping_mode: Option<&str>, + #[case] modifications: &[AddFilePartitionKeyModify<'_>], +) -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + let table_schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("d", DataType::INTEGER), + StructField::nullable("p1", DataType::STRING), + StructField::nullable("p2", DataType::INTEGER), + ])?); + let (_tmp_dir, table_path, engine) = test_utils::test_table_setup_mt()?; + let mut builder = create_table(&table_path, table_schema, "test/1.0") + .with_data_layout(DataLayout::partitioned(["p1", "p2"])); + if let Some(mode) = column_mapping_mode { + builder = builder.with_table_properties([("delta.columnMapping.mode", mode)]); + } + builder + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + + let data_schema: Arc = Arc::new( + (&StructType::try_new(vec![StructField::nullable("d", DataType::INTEGER)])?) + .try_into_arrow()?, + ); + let make_add = |txn: &Transaction, p1: &str, p2: i32| { + let wc = txn.partitioned_write_context(HashMap::from([ + ("p1".to_string(), Scalar::String(p1.into())), + ("p2".to_string(), Scalar::Integer(p2)), + ]))?; + let data = RecordBatch::try_new( + data_schema.clone(), + vec![Arc::new(Int32Array::from(vec![1]))], + )?; + futures::executor::block_on(engine.write_parquet(&ArrowEngineData::new(data), &wc)) + }; + + let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let mode = snapshot + .table_properties() + .column_mapping_mode + .unwrap_or(ColumnMappingMode::None); + let logical_schema = snapshot.schema(); + // Translate the `modifications` to the physical partition column names. + let modifications: Vec<_> = modifications + .iter() + .map(|modification| match *modification { + AddFilePartitionKeyModify::Drop { key } => { + let key = logical_schema + .field(key) + .map(|field| field.physical_name(mode)) + .unwrap_or(key); + AddFilePartitionKeyModify::Drop { key } + } + insertion => insertion, + }) + .collect(); + let mut txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? + .with_data_change(true); + let add = make_add(&txn, "b", 6)?; + let corrupted = modify_add_file_partition_keys(into_record_batch(add), &modifications); + txn.add_files(Box::new(ArrowEngineData::new(corrupted))); + assert_result_error_with_message(txn.commit(engine.as_ref()), "partitionValues keys"); + Ok(()) +} diff --git a/kernel/tests/integration/write/cdf.rs b/kernel/tests/integration/write/cdf.rs new file mode 100644 index 0000000000..b56b4369de --- /dev/null +++ b/kernel/tests/integration/write/cdf.rs @@ -0,0 +1,221 @@ +//! Integration tests for change-data-feed aware write paths. + +use std::sync::Arc; + +use delta_kernel::arrow::array::Int32Array; +use delta_kernel::arrow::record_batch::RecordBatch; +use delta_kernel::engine::arrow_conversion::TryIntoArrow as _; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::engine_data::FilteredEngineData; +use delta_kernel::schema::SchemaRef; +use delta_kernel::transaction::CommitResult; +use delta_kernel::{Snapshot, Version}; +use tempfile::{tempdir, TempDir}; +use test_utils::delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor; +use test_utils::delta_kernel_default_engine::DefaultEngine; +use test_utils::{ + assert_result_error_with_message, begin_transaction, create_table, engine_store_setup, + load_and_begin_transaction, +}; +use url::Url; + +use crate::common::write_utils::get_simple_int_schema; + +// Helper function to create a table with CDF enabled +async fn create_cdf_table( + table_name: &str, + schema: SchemaRef, +) -> Result<(Url, Arc>, TempDir), Box> +{ + let tmp_dir = tempdir()?; + let tmp_test_dir_url = Url::from_directory_path(tmp_dir.path()).unwrap(); + + let (store, engine, table_location) = engine_store_setup(table_name, Some(&tmp_test_dir_url)); + + let table_url = create_table( + store.clone(), + table_location, + schema.clone(), + &[], + true, // use protocol 3.7 + vec![], + vec!["changeDataFeed"], + ) + .await?; + + Ok((table_url, Arc::new(engine), tmp_dir)) +} + +// Helper function to write data to a table +async fn write_data_to_table( + table_url: &Url, + engine: &Arc>, + schema: SchemaRef, + values: Vec, +) -> Result> { + let mut txn = + load_and_begin_transaction(table_url.clone(), engine.as_ref())?.with_engine_info("test"); + + add_files_to_transaction(&mut txn, engine, schema, values).await?; + + let result = txn.commit(engine.as_ref())?; + match result { + CommitResult::CommittedTransaction(committed) => Ok(committed.commit_version()), + _ => panic!("Transaction should be committed"), + } +} + +// Helper function to add files to an existing transaction +async fn add_files_to_transaction( + txn: &mut delta_kernel::transaction::Transaction, + engine: &Arc>, + schema: SchemaRef, + values: Vec, +) -> Result<(), Box> { + let data = RecordBatch::try_new( + Arc::new(schema.as_ref().try_into_arrow()?), + vec![Arc::new(Int32Array::from(values))], + )?; + + let write_context = Arc::new(txn.unpartitioned_write_context().unwrap()); + let add_files_metadata = engine + .write_parquet(&ArrowEngineData::new(data), write_context.as_ref()) + .await?; + txn.add_files(add_files_metadata); + Ok(()) +} + +#[tokio::test] +async fn test_cdf_write_all_adds_succeeds() -> Result<(), Box> { + // This test verifies that add-only transactions work with CDF enabled + let _ = tracing_subscriber::fmt::try_init(); + + let schema = get_simple_int_schema(); + + let (table_url, engine, _tmp_dir) = + create_cdf_table("test_cdf_all_adds", schema.clone()).await?; + + // Add files - this should succeed + let version = write_data_to_table(&table_url, &engine, schema, vec![1, 2, 3]).await?; + assert_eq!(version, 1); + + Ok(()) +} + +#[tokio::test] +async fn test_cdf_write_all_removes_succeeds() -> Result<(), Box> { + // This test verifies that remove-only transactions work with CDF enabled + let _ = tracing_subscriber::fmt::try_init(); + + let schema = get_simple_int_schema(); + + let (table_url, engine, _tmp_dir) = + create_cdf_table("test_cdf_all_removes", schema.clone()).await?; + + // First, add some data + write_data_to_table(&table_url, &engine, schema, vec![1, 2, 3]).await?; + + // Now remove the files + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let mut txn = begin_transaction(snapshot.clone(), engine.as_ref())? + .with_engine_info("cdf remove test") + .with_data_change(true); + + let scan = snapshot.scan_builder().build()?; + let scan_metadata = scan.scan_metadata(engine.as_ref())?.next().unwrap()?; + let (data, selection_vector) = scan_metadata.scan_files.into_parts(); + txn.remove_files(FilteredEngineData::try_new(data, selection_vector)?); + + // This should succeed - remove-only transactions are allowed with CDF + let result = txn.commit(engine.as_ref())?; + match result { + CommitResult::CommittedTransaction(committed) => { + assert_eq!(committed.commit_version(), 2); + } + _ => panic!("Transaction should be committed"), + } + + Ok(()) +} + +#[tokio::test] +async fn test_cdf_write_mixed_no_data_change_succeeds() -> Result<(), Box> { + // This test verifies that mixed add+remove transactions work when dataChange=false. + // It's allowed because the transaction does not contain any logical data changes. + // This can happen when a table is being optimized/compacted. + let _ = tracing_subscriber::fmt::try_init(); + + let schema = get_simple_int_schema(); + + let (table_url, engine, _tmp_dir) = + create_cdf_table("test_cdf_mixed_no_data_change", schema.clone()).await?; + + // First, add some data + write_data_to_table(&table_url, &engine, schema.clone(), vec![1, 2, 3]).await?; + + // Now create a transaction with both add AND remove files, but dataChange=false + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let mut txn = begin_transaction(snapshot.clone(), engine.as_ref())? + .with_engine_info("cdf mixed test") + .with_data_change(false); // dataChange=false is key here + + // Add new files + add_files_to_transaction(&mut txn, &engine, schema, vec![4, 5, 6]).await?; + + // Also remove existing files + let scan = snapshot.scan_builder().build()?; + let scan_metadata = scan.scan_metadata(engine.as_ref())?.next().unwrap()?; + let (data, selection_vector) = scan_metadata.scan_files.into_parts(); + txn.remove_files(FilteredEngineData::try_new(data, selection_vector)?); + + // This should succeed - mixed operations are allowed when dataChange=false + let result = txn.commit(engine.as_ref())?; + match result { + CommitResult::CommittedTransaction(committed) => { + assert_eq!(committed.commit_version(), 2); + } + _ => panic!("Transaction should be committed"), + } + + Ok(()) +} + +#[tokio::test] +async fn test_cdf_write_mixed_with_data_change_fails() -> Result<(), Box> { + // This test verifies that mixed add+remove transactions fail with helpful error when + // dataChange=true + let _ = tracing_subscriber::fmt::try_init(); + + let schema = get_simple_int_schema(); + + let (table_url, engine, _tmp_dir) = + create_cdf_table("test_cdf_mixed_with_data_change", schema.clone()).await?; + + // First, add some data + write_data_to_table(&table_url, &engine, schema.clone(), vec![1, 2, 3]).await?; + + // Now create a transaction with both add AND remove files with dataChange=true + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let mut txn = begin_transaction(snapshot.clone(), engine.as_ref())? + .with_engine_info("cdf mixed fail test") + .with_data_change(true); // dataChange=true - this should fail + + // Add new files + add_files_to_transaction(&mut txn, &engine, schema, vec![4, 5, 6]).await?; + + // Also remove existing files + let scan = snapshot.scan_builder().build()?; + let scan_metadata = scan.scan_metadata(engine.as_ref())?.next().unwrap()?; + let (data, selection_vector) = scan_metadata.scan_files.into_parts(); + txn.remove_files(FilteredEngineData::try_new(data, selection_vector)?); + + // This should fail with our new error message + assert_result_error_with_message( + txn.commit(engine.as_ref()), + "Cannot add and remove data in the same transaction when Change Data Feed is enabled (delta.enableChangeDataFeed = true). \ + This would require writing CDC files for DML operations, which is not yet supported. \ + Consider using separate transactions: one to add files, another to remove files." + ); + + Ok(()) +} diff --git a/kernel/tests/integration/write/clustered.rs b/kernel/tests/integration/write/clustered.rs new file mode 100644 index 0000000000..8b3ce09ead --- /dev/null +++ b/kernel/tests/integration/write/clustered.rs @@ -0,0 +1,346 @@ +//! Integration tests for checkpointing non-kernel tables and clustered-table write/stats flows. + +use std::collections::HashMap; +use std::sync::Arc; + +use delta_kernel::actions::{MAX_VALUES, MIN_VALUES}; +use delta_kernel::arrow::array::{Array, Int64Array, StringArray, StructArray}; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::expressions::{column_name, ColumnName}; +use delta_kernel::object_store::local::LocalFileSystem; +use delta_kernel::object_store::DynObjectStore; +use delta_kernel::schema::StructType; +use delta_kernel::table_features::{get_any_level_column_physical_name, ColumnMappingMode}; +use delta_kernel::transaction::create_table::create_table as create_table_txn; +use delta_kernel::Snapshot; +use itertools::Itertools; +use tempfile::TempDir; +use test_utils::delta_kernel_default_engine::executor::tokio::TokioMultiThreadExecutor; +use test_utils::delta_kernel_default_engine::DefaultEngine; +use test_utils::{ + create_default_engine_mt_executor, nested_batches, nested_schema, read_actions_from_commit, + test_table_setup, write_batch_to_table, +}; +use url::Url; + +use crate::common::write_utils::{ + assert_column_mapping_mode, assert_min_max_stats, resolve_json_path, resolve_struct_field, + set_table_properties, +}; + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_checkpoint_non_kernel_written_table() { + // Table written by a non-kernel-integrated connector: 7 rows with columns (i, j, k), where + // parquet field nullabilities differ from the Delta schema. DefaultEngine reads it, coerces + // nullabilities to match the Delta schema, creates a checkpoint, and verifies the data is + // unchanged. + let source_path = std::path::Path::new("./tests/data/external-table-different-nullability"); + let temp_dir = tempfile::tempdir().unwrap(); + let table_path = temp_dir.path().join("test-checkpoint-table"); + test_utils::copy_directory(source_path, &table_path).unwrap(); + + let url = Url::from_directory_path(&table_path).unwrap(); + let store: Arc = Arc::new(LocalFileSystem::new()); + let executor = Arc::new( + test_utils::delta_kernel_default_engine::executor::tokio::TokioMultiThreadExecutor::new( + tokio::runtime::Handle::current(), + ), + ); + let engine: Arc> = Arc::new( + test_utils::delta_kernel_default_engine::DefaultEngineBuilder::new(store) + .with_task_executor(executor) + .build(), + ); + + // Read data before checkpoint + let snapshot = Snapshot::builder_for(url.clone()) + .build(engine.as_ref()) + .unwrap(); + let scan_before = Arc::clone(&snapshot).scan_builder().build().unwrap(); + let batches_before = test_utils::read_scan(&scan_before, engine.clone()).unwrap(); + + // Create checkpoint via snapshot.checkpoint() + snapshot.checkpoint(engine.as_ref(), None).unwrap(); + + // Read data after checkpoint + let snapshot_after = Snapshot::builder_for(url.clone()) + .build(engine.as_ref()) + .unwrap(); + let scan_after = snapshot_after.scan_builder().build().unwrap(); + let batches_after = test_utils::read_scan(&scan_after, engine.clone()).unwrap(); + + // Verify data unchanged + let formatted_before = + delta_kernel::arrow::util::pretty::pretty_format_batches(&batches_before) + .unwrap() + .to_string(); + let formatted_after = delta_kernel::arrow::util::pretty::pretty_format_batches(&batches_after) + .unwrap() + .to_string(); + assert_eq!( + formatted_before, formatted_after, + "Row data changed after checkpoint creation!" + ); + + // Verify checkpoint file exists + let delta_log_path = table_path.join("_delta_log"); + let has_checkpoint = std::fs::read_dir(&delta_log_path) + .unwrap() + .filter_map(|e| e.ok()) + .any(|e| { + e.file_name() + .to_str() + .is_some_and(|n| n.contains(".checkpoint.parquet")) + }); + assert!(has_checkpoint, "Expected at least one checkpoint file"); +} + +struct ClusteredTableSetup { + _tmp_dir: TempDir, + table_path: String, + table_url: Url, + engine: Arc>, + snapshot: Arc, +} + +/// Creates a clustered table with column mapping and sets table properties. +fn setup_clustered_table( + cm_mode: &str, + schema: Arc, + clustering_cols: Vec, + table_properties: &[(&str, &str)], +) -> Result> { + use delta_kernel::transaction::data_layout::DataLayout; + + let (_tmp_dir, table_path, _) = test_table_setup()?; + let table_url = Url::from_directory_path(&table_path).unwrap(); + let engine = create_default_engine_mt_executor(&table_url)?; + + let _ = create_table_txn(table_url.as_str(), schema, "Test/1.0") + .with_table_properties([("delta.columnMapping.mode", cm_mode)]) + .with_data_layout(DataLayout::Clustered { + columns: clustering_cols, + }) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let snapshot = set_table_properties( + &table_path, + &table_url, + engine.as_ref(), + 0, + table_properties, + )?; + + Ok(ClusteredTableSetup { + _tmp_dir, + table_path, + table_url, + engine, + snapshot, + }) +} + +/// E2E test: create a clustered table with column mapping, write data, and verify that +/// add.stats in the commit log contains min/max statistics for the clustering columns +/// (including a nested column). +#[rstest::rstest] +#[case::cm_none("none")] +#[case::cm_name("name")] +#[case::cm_id("id")] +#[tokio::test(flavor = "multi_thread")] +async fn test_clustered_table_write_has_stats( + #[case] cm_mode: &str, +) -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + let clustering_cols = vec![column_name!("row_number"), column_name!("address.street")]; + let setup = setup_clustered_table( + cm_mode, + nested_schema()?, + clustering_cols.clone(), + &[("delta.dataSkippingNumIndexedCols", "0")], + )?; + let engine = &setup.engine; + let mut snapshot = setup.snapshot; + for batch in nested_batches()? { + snapshot = write_batch_to_table(&snapshot, engine.as_ref(), batch, HashMap::new()).await?; + } + + let cm = assert_column_mapping_mode(&snapshot, cm_mode); + let physical_paths: Vec> = clustering_cols + .iter() + .map(|c| { + get_any_level_column_physical_name(snapshot.schema().as_ref(), c, cm) + .unwrap() + .into_inner() + }) + .collect(); + if cm != ColumnMappingMode::None { + let logical_paths: Vec> = vec![vec!["row_number"], vec!["address", "street"]]; + for (phys, logical) in physical_paths.iter().zip(&logical_paths) { + assert_ne!( + phys.iter().map(String::as_str).collect_vec(), + *logical, + "physical path should differ from logical when cm={cm:?}" + ); + } + } + + // Resolve a non-clustering column to verify it's excluded from stats + let non_clustering_physical = + get_any_level_column_physical_name(snapshot.schema().as_ref(), &column_name!("name"), cm)? + .into_inner(); + + // Verify stats for each write commit (v2 and v3, since v1 is the property update). + // Batch 1 (v2): row_number 1..3, address.street "st1".."st3" + // Batch 2 (v3): row_number 4..6, address.street "st4".."st6" + let expected: [(i64, i64, &str, &str); 2] = [(1, 3, "st1", "st3"), (4, 6, "st4", "st6")]; + for (version, (min_rn, max_rn, min_st, max_st)) in expected.iter().enumerate() { + let version = (version + 2) as u64; + let add_actions = read_actions_from_commit(&setup.table_url, version, "add")?; + assert!( + !add_actions.is_empty(), + "v{version}: should have add actions" + ); + + for add in &add_actions { + let stats: serde_json::Value = serde_json::from_str( + add.get("stats") + .and_then(|s| s.as_str()) + .expect("add action should have stats"), + )?; + // Clustering columns should have stats despite numIndexedCols=0 + assert_min_max_stats(&stats, &physical_paths[0], *min_rn, *max_rn); + assert_min_max_stats(&stats, &physical_paths[1], *min_st, *max_st); + + // Non-clustering column "name" should NOT have stats + let non_cluster_min = resolve_json_path(&stats[MIN_VALUES], &non_clustering_physical); + assert!( + non_cluster_min.is_null(), + "v{version}: non-clustering column 'name' should not have stats" + ); + } + } + + Ok(()) +} + +/// E2E test: create a clustered table with column mapping, enable writeStatsAsStruct, +/// write data, checkpoint, and verify stats_parsed. +#[rstest::rstest] +#[case::cm_none("none")] +#[case::cm_name("name")] +#[case::cm_id("id")] +#[tokio::test(flavor = "multi_thread")] +async fn test_clustered_table_write_has_stats_parsed( + #[case] cm_mode: &str, +) -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + let clustering_cols = vec![column_name!("row_number"), column_name!("address.street")]; + let setup = setup_clustered_table( + cm_mode, + nested_schema()?, + clustering_cols.clone(), + &[ + ("delta.checkpoint.writeStatsAsStruct", "true"), + ("delta.dataSkippingNumIndexedCols", "0"), + ], + )?; + let engine = &setup.engine; + let mut snapshot = setup.snapshot; + for batch in nested_batches()? { + snapshot = write_batch_to_table(&snapshot, engine.as_ref(), batch, HashMap::new()).await?; + } + + let cm = assert_column_mapping_mode(&snapshot, cm_mode); + let physical_paths: Vec> = clustering_cols + .iter() + .map(|c| { + get_any_level_column_physical_name(snapshot.schema().as_ref(), c, cm) + .unwrap() + .into_inner() + }) + .collect(); + if cm != ColumnMappingMode::None { + let logical_paths: Vec> = vec![vec!["row_number"], vec!["address", "street"]]; + for (phys, logical) in physical_paths.iter().zip(&logical_paths) { + assert_ne!( + phys.iter().map(String::as_str).collect_vec(), + *logical, + "physical path should differ from logical when cm={cm:?}" + ); + } + } + let non_clustering_physical = + get_any_level_column_physical_name(snapshot.schema().as_ref(), &column_name!("name"), cm)? + .into_inner(); + + snapshot.checkpoint(engine.as_ref(), None)?; + + // Read checkpoint parquet directly to verify stats_parsed contains only clustering columns. + // `ScanBuilder::with_stats(StatsOptions::all())` doesn't support stats_parsed when + // dataSkippingNumIndexedCols=0. Read directly from the checkpoint parquet file instead. + use delta_kernel::parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder; + let delta_log = std::path::Path::new(&setup.table_path).join("_delta_log"); + let ckpt_path = std::fs::read_dir(&delta_log)? + .filter_map(|e| e.ok()) + .find(|e| { + e.file_name() + .to_str() + .is_some_and(|n| n.contains(".checkpoint.parquet")) + }) + .expect("checkpoint parquet should exist") + .path(); + let file = std::fs::File::open(&ckpt_path)?; + let reader = ParquetRecordBatchReaderBuilder::try_new(file)?.build()?; + + let min_path = |field: &[String]| -> Vec { [&[MIN_VALUES.into()], field].concat() }; + let max_path = |field: &[String]| -> Vec { [&[MAX_VALUES.into()], field].concat() }; + + let mut stats_rows: Vec<(i64, i64, String, String)> = Vec::new(); + for batch in reader { + let batch = batch?; + let batch_struct = StructArray::from(batch); + let add: &StructArray = resolve_struct_field(&batch_struct, &["add".into()]); + let stats_parsed: &StructArray = resolve_struct_field(add, &["stats_parsed".into()]); + + // Non-clustering column should not appear in stats_parsed + let min_values: &StructArray = resolve_struct_field(stats_parsed, &[MIN_VALUES.into()]); + assert!( + min_values + .column_by_name(&non_clustering_physical[0]) + .is_none(), + "non-clustering column '{}' should not have stats_parsed", + non_clustering_physical[0] + ); + + let min_row_num: &Int64Array = + resolve_struct_field(stats_parsed, &min_path(&physical_paths[0])); + let max_row_num: &Int64Array = + resolve_struct_field(stats_parsed, &max_path(&physical_paths[0])); + let min_st: &StringArray = + resolve_struct_field(stats_parsed, &min_path(&physical_paths[1])); + let max_st: &StringArray = + resolve_struct_field(stats_parsed, &max_path(&physical_paths[1])); + + for i in 0..stats_parsed.len() { + if !stats_parsed.is_null(i) { + stats_rows.push(( + min_row_num.value(i), + max_row_num.value(i), + min_st.value(i).to_string(), + max_st.value(i).to_string(), + )); + } + } + } + + stats_rows.sort_by_key(|r| r.0); + assert_eq!(stats_rows.len(), 2, "should have stats_parsed for 2 files"); + assert_eq!(stats_rows[0], (1, 3, "st1".to_string(), "st3".to_string())); + assert_eq!(stats_rows[1], (4, 6, "st4".to_string(), "st6".to_string())); + + Ok(()) +} diff --git a/kernel/tests/integration/write/column_defaults.rs b/kernel/tests/integration/write/column_defaults.rs new file mode 100644 index 0000000000..44df6badd7 --- /dev/null +++ b/kernel/tests/integration/write/column_defaults.rs @@ -0,0 +1,977 @@ +//! Integration tests for the `allowColumnDefaults` writer feature. + +use std::collections::HashMap; +use std::fs; +use std::path::Path; +use std::sync::Arc; + +use delta_kernel::actions::{MAX_VALUES, MIN_VALUES, NULL_COUNT}; +use delta_kernel::arrow::array::{ArrayRef, Int32Array, Int64Array, StringArray}; +use delta_kernel::arrow::record_batch::RecordBatch; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::engine::arrow_conversion::{TryFromArrow as _, TryIntoArrow as _}; +use delta_kernel::engine::arrow_data::{ArrowEngineData, EngineDataArrowExt as _}; +use delta_kernel::expressions::{ColumnName, Scalar}; +use delta_kernel::parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder; +use delta_kernel::schema::{ + schema_ref, ArrayType, ColumnMetadataKey, DataType, MetadataValue, SchemaRef, StructField, + StructType, +}; +use delta_kernel::table_features::{ + get_any_level_column_physical_name, ColumnMappingMode, TableFeature, +}; +use delta_kernel::transaction::create_table::create_table as kernel_create_table; +use delta_kernel::transaction::data_layout::DataLayout; +use delta_kernel::{DeltaResult, Engine, Snapshot}; +use rstest::rstest; +use test_utils::delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor; +use test_utils::delta_kernel_default_engine::DefaultEngine; +use test_utils::{ + copy_directory, create_default_engine, create_table, engine_store_setup, get_column, + insert_data, read_actions_from_commit, read_scan, resolve_field, schema_with_column_defaults, + test_read, test_table_setup, test_table_setup_mt, write_batch_to_table, LoggingTest, +}; +use url::Url; + +use crate::common::write_utils::load_existing_single_file_checkpoint_path; + +async fn setup_unpartitioned_table( + name: &str, + schema: SchemaRef, + writer_features: Vec<&str>, +) -> Result<(DefaultEngine, Url), Box> { + let (store, engine, table_location) = engine_store_setup(name, None); + let table_url = create_table( + store, + table_location, + schema, + &[], + true, + vec![], + writer_features, + ) + .await?; + Ok((engine, table_url)) +} + +fn add_column_defaults_feature_commit( + table_path: &Path, + version: u64, + schema: Option<&StructType>, +) -> Result<(), Box> { + let initial_commit = + fs::read_to_string(table_path.join("_delta_log/00000000000000000000.json"))?; + let mut actions = initial_commit + .lines() + .map(serde_json::from_str::) + .collect::, _>>()?; + let protocol = { + let protocol = actions + .iter_mut() + .find(|action| action.get("protocol").is_some()) + .expect("initial commit must contain a protocol action"); + let writer_features = protocol["protocol"]["writerFeatures"] + .as_array_mut() + .expect("writer v7 protocol must contain writerFeatures"); + writer_features.push(serde_json::json!("allowColumnDefaults")); + protocol.clone() + }; + + let metadata = { + let metadata = actions + .iter_mut() + .find(|action| action.get("metaData").is_some()) + .expect("initial commit must contain a metadata action"); + if let Some(schema) = schema { + metadata["metaData"]["schemaString"] = + serde_json::json!(serde_json::to_string(schema)?); + } + metadata.clone() + }; + + let commit = format!( + "{}\n{}\n", + serde_json::to_string(&protocol)?, + serde_json::to_string(&metadata)?, + ); + fs::write( + table_path.join(format!("_delta_log/{version:020}.json")), + commit, + )?; + Ok(()) +} + +fn assert_top_level_default( + snapshot: &Arc, + engine: &dyn Engine, + column: &str, + expected: Scalar, +) -> DeltaResult<()> { + let txn = snapshot + .clone() + .transaction(Box::new(FileSystemCommitter::new()), engine)?; + assert_eq!( + txn.top_level_column_defaults()?[column].to_scalar()?, + Some(expected) + ); + Ok(()) +} + +fn assert_no_column_default_metadata(schema: &StructType, path: &[&str]) { + let display = path.join("."); + let field = resolve_field(schema, path) + .unwrap_or_else(|error| panic!("checkpoint schema must contain '{display}': {error}")); + assert!( + field + .get_config_value(&ColumnMetadataKey::CurrentDefault) + .is_none(), + "checkpoint-derived field '{display}' must not carry CURRENT_DEFAULT metadata", + ); +} + +fn assert_checkpoint_parsed_columns_have_no_column_defaults( + checkpoint_schema: &StructType, + data_column: &str, + partition_column: &str, +) { + assert_no_column_default_metadata( + checkpoint_schema, + &["add", "partitionValues_parsed", partition_column], + ); + for stats_field in [MIN_VALUES, MAX_VALUES, NULL_COUNT] { + assert_no_column_default_metadata( + checkpoint_schema, + &["add", "stats_parsed", stats_field, data_column], + ); + } +} + +// TODO(#2630): Allow create table to support column defaults +#[test] +fn test_create_table_rejects_col_defaults() -> DeltaResult<()> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let schema = schema_ref! { nullable "id": LONG }; + + let err = kernel_create_table(&table_path, schema, "Test/1.0") + .with_table_properties([("delta.feature.allowColumnDefaults", "supported")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new())) + .expect_err("kernel create_table must reject allowColumnDefaults") + .to_string(); + assert!( + err.contains("allowColumnDefaults"), + "error must name the unsupported feature; got: {err}", + ); + Ok(()) +} + +#[test] +fn test_schema_with_column_defaults_errors_on_unknown_column() { + let schema = StructType::try_new(vec![StructField::nullable("c", DataType::INTEGER)]).unwrap(); + + let err = schema_with_column_defaults(&schema, HashMap::from([("does_not_exist", "1")])) + .expect_err("unknown column must produce an error") + .to_string(); + assert!( + err.contains("does_not_exist"), + "error should name the unknown column; got: {err}", + ); +} + +#[test] +fn test_schema_with_column_defaults_reports_all_unknown_columns() { + let schema = StructType::try_new(vec![StructField::nullable("c", DataType::INTEGER)]).unwrap(); + + let err = + schema_with_column_defaults(&schema, HashMap::from([("ghost1", "1"), ("ghost2", "2")])) + .expect_err("all unknown columns must produce an error") + .to_string(); + assert!(err.contains("ghost1"), "error must name ghost1; got: {err}"); + assert!(err.contains("ghost2"), "error must name ghost2; got: {err}"); +} + +#[test] +fn test_schema_with_column_defaults_overwrites_existing_default() { + let base = StructType::try_new(vec![StructField::nullable("c", DataType::INTEGER)]).unwrap(); + let with_first = schema_with_column_defaults(&base, HashMap::from([("c", "1")])).unwrap(); + let with_second = + schema_with_column_defaults(&with_first, HashMap::from([("c", "99")])).unwrap(); + + let field = with_second.field("c").expect("c field must exist"); + assert_eq!( + field.get_config_value(&ColumnMetadataKey::CurrentDefault), + Some(&MetadataValue::String("99".to_string())), + "second call must overwrite the first CURRENT_DEFAULT value", + ); +} + +#[tokio::test] +async fn test_blind_append_to_column_defaults_table_is_supported( +) -> Result<(), Box> { + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::LONG), + StructField::nullable("name", DataType::STRING), + ])?); + + let (store, engine, table_location) = engine_store_setup("test_table_col_defaults", None); + // Use the JSON helper instead of the kernel `create_table` builder because the + // latter does not whitelist this feature in `ALLOWED_DELTA_FEATURES`. + let table_url = create_table( + store.clone(), + table_location, + schema.clone(), + &[], /* partition_columns */ + true, /* use_37_protocol */ + vec![], /* reader_features */ + vec!["allowColumnDefaults"], /* writer_features */ + ) + .await?; + let engine = Arc::new(engine); + + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let writer_features = snapshot + .table_configuration() + .protocol() + .writer_features() + .expect("writer_features must be present on a writer v7 table"); + assert!( + writer_features.contains(&TableFeature::AllowColumnDefaults), + "writer_features must include AllowColumnDefaults; got {writer_features:?}", + ); + + // Blind-append a small batch. No column has a `CURRENT_DEFAULT` yet, so this is a plain + // append; we only assert the feature does not block the write path. + let columns: Vec = vec![ + Arc::new(Int64Array::from(vec![1, 2, 3])), + Arc::new(StringArray::from(vec!["a", "b", "c"])), + ]; + assert!(insert_data(snapshot, &engine, columns.clone()) + .await? + .is_committed()); + + // Round-trip read. + let data = RecordBatch::try_new(Arc::new(schema.as_ref().try_into_arrow()?), columns)?; + test_read(&ArrowEngineData::new(data), &table_url, engine)?; + + Ok(()) +} + +#[rstest] +#[case::unpartitioned("unpartitioned", &[])] +#[case::partitioned("partitioned", &["p"])] +#[tokio::test] +async fn write_context_acknowledgement_depends_on_column_defaults( + #[case] label: &str, + #[case] partition_columns: &[&str], + #[values(false, true)] has_default: bool, +) -> Result<(), Box> { + let base = StructType::try_new(vec![ + StructField::nullable("c", DataType::INTEGER), + StructField::nullable("p", DataType::INTEGER), + ])?; + let schema = if has_default { + schema_with_column_defaults(&base, HashMap::from([("c", "42")]))? + } else { + Arc::new(base) + }; + let table_name = format!("write_context_ack_{label}_{has_default}"); + let (store, engine, table_location) = engine_store_setup(&table_name, None); + let table_url = create_table( + store, + table_location, + schema, + partition_columns, + true, + vec![], + vec!["allowColumnDefaults"], + ) + .await?; + + let snapshot = Snapshot::builder_for(table_url).build(&engine)?; + let mut txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), &engine)?; + + let defaults = txn.top_level_column_defaults()?; + if has_default { + assert_eq!(defaults["c"].to_scalar()?, Some(Scalar::Integer(42))); + } else { + assert!(defaults.is_empty()); + } + drop(defaults); + + let partition_values = HashMap::from([("p".to_string(), Scalar::Integer(7))]); + if has_default { + let error = if partition_columns.is_empty() { + txn.unpartitioned_write_context() + } else { + txn.partitioned_write_context(partition_values.clone()) + } + .expect_err("inspecting defaults must not implicitly acknowledge them"); + assert!(matches!( + &error, + delta_kernel::Error::InvalidTransactionState(_) + )); + assert!(error.to_string().contains("ack_column_defaults")); + + txn.ack_column_defaults(); + } + if partition_columns.is_empty() { + txn.unpartitioned_write_context()?; + } else { + txn.partitioned_write_context(partition_values)?; + } + + Ok(()) +} + +/// Creates a single-column table, materializes its default through the connector-facing API, +/// and asserts the metadata and written value round-trip. +async fn assert_materialized_column_default_round_trips( + data_type: DataType, + default_sql: &str, +) -> Result<(), Box> { + let extra_features: &[&str] = if data_type == DataType::TIMESTAMP_NTZ { + &["timestampNtz"] + } else { + &[] + }; + let base = StructType::try_new(vec![StructField::nullable("c", data_type.clone())])?; + let schema = schema_with_column_defaults(&base, HashMap::from([("c", default_sql)]))?; + + let writer_features = [&["allowColumnDefaults"], extra_features].concat(); + let (store, engine, table_location) = engine_store_setup("test_create_with_default", None); + let table_url = create_table( + store, + table_location, + schema, + &[], /* partition_columns */ + true, /* use_37_protocol */ + extra_features.to_vec(), /* reader_features */ + writer_features, + ) + .await?; + + let engine = Arc::new(engine); + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let loaded_schema = snapshot.schema(); + let field = loaded_schema + .field("c") + .expect("c field must exist in loaded schema"); + + assert_eq!(field.data_type(), &data_type); + let column_default = field + .column_default() + .expect("column_default must not error for a valid primitive default") + .expect("CURRENT_DEFAULT metadata must be present in the loaded schema"); + assert_eq!( + column_default.raw_sql(), + default_sql, + "CURRENT_DEFAULT raw SQL must round-trip verbatim", + ); + assert_eq!(column_default.data_type(), &data_type); + + let scalar = { + let txn = snapshot + .clone() + .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())?; + let defaults = txn.top_level_column_defaults()?; + defaults["c"] + .to_scalar()? + .expect("a primitive literal default must materialize") + }; + let values = scalar.to_array(1)?; + insert_data(snapshot, &engine, vec![values.clone()]) + .await? + .unwrap_committed(); + + let expected = RecordBatch::try_new( + Arc::new(loaded_schema.as_ref().try_into_arrow()?), + vec![values], + )?; + test_read(&ArrowEngineData::new(expected), &table_url, engine)?; + + Ok(()) +} + +#[rstest] +#[case::integer(DataType::INTEGER, "42")] +#[case::long(DataType::LONG, "9876543210")] +#[case::short(DataType::SHORT, "7")] +#[case::byte(DataType::BYTE, "1")] +#[case::float(DataType::FLOAT, "1.5")] +#[case::double(DataType::DOUBLE, "3.14")] +#[case::string(DataType::STRING, "'hello'")] +#[case::boolean(DataType::BOOLEAN, "TRUE")] +#[case::binary(DataType::BINARY, "X'deadbeef'")] +#[case::date(DataType::DATE, "DATE '2024-01-01'")] +#[case::timestamp(DataType::TIMESTAMP, "TIMESTAMP '2024-01-01T12:00:00Z'")] +#[case::timestamp_ntz(DataType::TIMESTAMP_NTZ, "TIMESTAMP_NTZ '2024-01-01 12:00:00'")] +#[case::decimal(DataType::decimal(10, 2).unwrap(), "1.23")] +#[tokio::test] +async fn test_materialized_primitive_column_default_round_trips( + #[case] data_type: DataType, + #[case] default_sql: &str, +) -> Result<(), Box> { + assert_materialized_column_default_round_trips(data_type, default_sql).await +} + +#[rstest] +#[case::all_data_columns(&[])] +#[case::default_on_partition_column(&["b"])] +#[tokio::test] +async fn test_transaction_top_level_column_defaults_excludes_nested_defaults( + #[case] partition_columns: &[&str], +) -> Result<(), Box> { + let nested_default = StructField::nullable("inner", DataType::INTEGER).add_metadata([( + ColumnMetadataKey::CurrentDefault.as_ref().to_string(), + MetadataValue::String("7".to_string()), + )]); + + // `a`: no default, `b`: kernel-parsable default, `c`: non-kernel-parsable default, + // `s.inner`: nested default that the top-level API must not return. + let base = StructType::try_new(vec![ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::INTEGER), + StructField::nullable("c", DataType::TIMESTAMP), + StructField::nullable("s", DataType::try_struct_type([nested_default])?), + ])?; + let schema = schema_with_column_defaults( + &base, + HashMap::from([("b", "1337"), ("c", "current_timestamp()")]), + )?; + + let (store, engine, table_location) = engine_store_setup("test_txn_column_defaults", None); + let table_url = create_table( + store, + table_location, + schema, + partition_columns, + true, /* use_37_protocol */ + vec![], /* reader_features */ + vec!["allowColumnDefaults"], /* writer_features */ + ) + .await?; + + let snapshot = Snapshot::builder_for(table_url).build(&engine)?; + let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), &engine)?; + + let defaults = txn.top_level_column_defaults()?; + assert_eq!(defaults.len(), 2, "only b and c declare a default"); + assert!(!defaults.contains_key("a"), "a has no default"); + assert!( + !defaults.contains_key("s") && !defaults.contains_key("s.inner"), + "nested defaults must not be returned by the top-level API" + ); + + let b = &defaults["b"]; + assert_eq!(b.raw_sql(), "1337"); + assert_eq!(b.to_scalar()?, Some(Scalar::Integer(1337))); + + let c = &defaults["c"]; + assert_eq!(c.raw_sql(), "current_timestamp()"); + assert_eq!( + c.to_scalar()?, + None, + "a non-kernel-parsable default is not parsed by the kernel", + ); + + Ok(()) +} + +/// On an `icebergCompatV3` table, a default kernel cannot verify as a literal produces a +/// warning rather than an error, so the snapshot loads and a DML transaction constructs. +#[rstest] +#[case::non_literal( + "non_literal", + DataType::TIMESTAMP, + "current_timestamp()", + "could not verify" +)] +#[case::unparsable_non_primitive( + "non_primitive", + DataType::from(ArrayType::new(DataType::INTEGER, true)), + "ARRAY(1)", + "could not verify" +)] +#[tokio::test] +async fn test_load_and_write_tolerate_v3_unverifiable_default( + #[case] label: &str, + #[case] field_type: DataType, + #[case] default_sql: &str, + #[case] warning_text: &str, +) -> Result<(), Box> { + let base = StructType::try_new(vec![StructField::nullable("c", field_type)])?; + let schema = schema_with_column_defaults(&base, HashMap::from([("c", default_sql)]))?; + + let (engine, table_url) = setup_unpartitioned_table( + &format!("test_v3_tolerates_{label}"), + schema, + vec!["allowColumnDefaults", "icebergCompatV3"], + ) + .await?; + + // Read: the snapshot loads despite the unverifiable default. + let snapshot = Snapshot::builder_for(table_url).build(&engine)?; + + let logging = LoggingTest::new(); + snapshot.transaction(Box::new(FileSystemCommitter::new()), &engine)?; + assert!( + logging.logs().contains(warning_text), + "logs: {}", + logging.logs() + ); + + Ok(()) +} + +/// A non-string `CURRENT_DEFAULT` value is corrupt and rejected at snapshot load. Built by +/// hand because `schema_with_column_defaults` only writes string values. +#[tokio::test] +async fn test_load_rejects_non_string_column_default() -> Result<(), Box> { + let field = StructField::nullable("c", DataType::INTEGER).add_metadata([( + ColumnMetadataKey::CurrentDefault.as_ref().to_string(), + MetadataValue::Number(7), + )]); + let schema = Arc::new(StructType::try_new(vec![field])?); + + let (engine, table_url) = setup_unpartitioned_table( + "test_load_rejects_non_string_default", + schema, + vec!["allowColumnDefaults"], + ) + .await?; + + let err = Snapshot::builder_for(table_url) + .build(&engine) + .expect_err("a non-string CURRENT_DEFAULT must be rejected at load") + .to_string(); + assert!(err.contains("non-string"), "got: {err}"); + + Ok(()) +} + +/// Orphaned column-default metadata (a `CURRENT_DEFAULT` without the `allowColumnDefaults` +/// feature) is tolerated: the snapshot loads and a write context builds without error. +#[tokio::test] +async fn test_load_and_write_allow_orphan_default() -> Result<(), Box> { + let base = StructType::try_new(vec![StructField::nullable("c", DataType::INTEGER)])?; + let schema = schema_with_column_defaults(&base, HashMap::from([("c", "42")]))?; + + let (engine, table_url) = + setup_unpartitioned_table("test_load_orphan_default", schema, vec![]).await?; + + // Read: snapshot loads despite the orphaned metadata. + let snapshot = Snapshot::builder_for(table_url).build(&engine)?; + + // Write: a write context builds without error. + let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), &engine)?; + assert!( + txn.top_level_column_defaults()?.is_empty(), + "orphaned defaults must not be surfaced without allowColumnDefaults", + ); + txn.unpartitioned_write_context()?; + + Ok(()) +} + +#[rstest] +#[case::null_default("null", "NULL", None)] +#[case::non_null_default("non_null", "'value'", Some("must be NULL"))] +#[tokio::test] +async fn test_variant_column_default_validation_at_snapshot_load( + #[case] label: &str, + #[case] default_sql: &str, + #[case] expected_error: Option<&str>, +) -> Result<(), Box> { + let variant_type = DataType::unshredded_variant(); + let base = StructType::try_new(vec![StructField::nullable("v", variant_type)])?; + let schema = schema_with_column_defaults(&base, HashMap::from([("v", default_sql)]))?; + + let (store, engine, table_location) = + engine_store_setup(&format!("test_variant_default_{label}"), None); + let table_url = create_table( + store, + table_location, + schema, + &[], + true, + vec!["variantType"], + vec!["variantType", "allowColumnDefaults"], + ) + .await?; + + match expected_error { + Some(expected_error) => { + let error = Snapshot::builder_for(table_url) + .build(&engine) + .expect_err("a non-NULL Variant default must be rejected at snapshot load") + .to_string(); + assert!(error.contains(expected_error), "got: {error}"); + } + None => { + let snapshot = Snapshot::builder_for(table_url).build(&engine)?; + let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), &engine)?; + let defaults = txn.top_level_column_defaults()?; + let column_default = &defaults["v"]; + assert_eq!(column_default.raw_sql(), default_sql); + assert!( + column_default + .to_scalar()? + .is_some_and(|value| value.is_null()), + "a Variant NULL default must surface as a null scalar", + ); + } + } + + Ok(()) +} + +/// Defaults kernel cannot materialize surface via raw SQL so the connector can evaluate them. +#[rstest] +#[case::type_mismatch("type_mismatch", DataType::INTEGER, "'not an int'")] +#[case::non_primitive( + "non_primitive", + DataType::from(ArrayType::new(DataType::INTEGER, true)), + "ARRAY(1)" +)] +#[tokio::test] +async fn test_load_tolerates_unmaterializable_default( + #[case] label: &str, + #[case] data_type: DataType, + #[case] default_sql: &str, +) -> Result<(), Box> { + let base = StructType::try_new(vec![StructField::nullable("c", data_type)])?; + let schema = schema_with_column_defaults(&base, HashMap::from([("c", default_sql)]))?; + + let (engine, table_url) = setup_unpartitioned_table( + &format!("test_load_{label}_default"), + schema, + vec!["allowColumnDefaults"], + ) + .await?; + + let snapshot = Snapshot::builder_for(table_url).build(&engine)?; + let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), &engine)?; + let defaults = txn.top_level_column_defaults()?; + + let c = &defaults["c"]; + assert_eq!(c.raw_sql(), default_sql); + assert_eq!( + c.to_scalar()?, + None, + "kernel must leave an unmaterializable default to the connector" + ); + + Ok(()) +} + +#[test] +fn test_column_default_composes_with_deletion_vectors() -> Result<(), Box> { + let source_path = fs::canonicalize("./tests/data/table-with-dv-small/")?; + let temp_dir = tempfile::tempdir()?; + let table_path = temp_dir.path().join("table-with-dv-and-column-default"); + copy_directory(&source_path, &table_path)?; + + let base = StructType::try_new(vec![StructField::nullable("value", DataType::INTEGER)])?; + let schema = schema_with_column_defaults(&base, HashMap::from([("value", "42")]))?; + add_column_defaults_feature_commit(&table_path, 2, Some(schema.as_ref()))?; + + let table_url = Url::from_directory_path(&table_path).expect("table path must be a URL"); + let engine = create_default_engine(&table_url)?; + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + assert_top_level_default(&snapshot, engine.as_ref(), "value", Scalar::Integer(42))?; + + let scan = snapshot.scan_builder().build()?; + let batches = read_scan(&scan, engine)?; + let mut values = Vec::new(); + for batch in batches { + let value_array = get_column!(batch, "value", Int32Array); + values.extend((0..value_array.len()).map(|index| value_array.value(index))); + } + values.sort_unstable(); + assert_eq!(values, (1..9).collect::>()); + + Ok(()) +} + +#[tokio::test(flavor = "multi_thread")] +async fn test_defaulted_clustering_column_round_trips_with_stats( +) -> Result<(), Box> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let base = StructType::try_new(vec![ + StructField::nullable("id", DataType::LONG), + StructField::nullable("c", DataType::INTEGER), + ])?; + let schema = schema_with_column_defaults(&base, HashMap::from([("c", "42")]))?; + + kernel_create_table(&table_path, schema.clone(), "Test/1.0") + .with_data_layout(DataLayout::clustered(["c"])) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + add_column_defaults_feature_commit(Path::new(&table_path), 1, None)?; + + let table_url = Url::from_directory_path(&table_path).expect("table path must be a URL"); + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + assert_top_level_default(&snapshot, engine.as_ref(), "c", Scalar::Integer(42))?; + + let columns: Vec = vec![ + Arc::new(Int64Array::from(vec![1, 2, 3])), + Arc::new(Int32Array::from(vec![42, 42, 42])), + ]; + let batch = RecordBatch::try_new(Arc::new(schema.as_ref().try_into_arrow()?), columns.clone())?; + write_batch_to_table(&snapshot, engine.as_ref(), batch, HashMap::new()).await?; + + let add_actions = read_actions_from_commit(&table_url, 2, "add")?; + let stats: serde_json::Value = serde_json::from_str( + add_actions[0]["stats"] + .as_str() + .expect("clustered write must include stats"), + )?; + assert_eq!(stats["minValues"]["c"], 42); + assert_eq!(stats["maxValues"]["c"], 42); + + let expected = RecordBatch::try_new(Arc::new(schema.as_ref().try_into_arrow()?), columns)?; + test_read(&ArrowEngineData::new(expected), &table_url, engine)?; + + Ok(()) +} + +#[rstest] +#[case::cm_none("none")] +#[case::cm_name("name")] +#[case::cm_id("id")] +#[tokio::test(flavor = "multi_thread")] +async fn test_column_default_round_trips_with_column_mapping_and_checkpoint( + #[case] column_mapping_mode: &str, +) -> Result<(), Box> { + let base = StructType::try_new(vec![ + StructField::nullable("id", DataType::LONG), + StructField::nullable("c", DataType::INTEGER), + StructField::nullable("p", DataType::INTEGER), + ])?; + let schema = schema_with_column_defaults(&base, HashMap::from([("c", "42"), ("p", "7")]))?; + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let mut table_properties = vec![("delta.checkpoint.writeStatsAsStruct", "true")]; + if column_mapping_mode != "none" { + table_properties.push(("delta.columnMapping.mode", column_mapping_mode)); + } + kernel_create_table(&table_path, schema.clone(), "Test/1.0") + .with_data_layout(DataLayout::partitioned(["p"])) + .with_table_properties(table_properties) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + add_column_defaults_feature_commit(Path::new(&table_path), 1, None)?; + + let table_url = Url::from_directory_path(&table_path).expect("table path must be a URL"); + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + assert_top_level_default(&snapshot, engine.as_ref(), "c", Scalar::Integer(42))?; + assert_top_level_default(&snapshot, engine.as_ref(), "p", Scalar::Integer(7))?; + let column_mapping_mode = column_mapping_mode.parse::()?; + let physical_name = |logical: &str| -> String { + get_any_level_column_physical_name( + snapshot.schema().as_ref(), + &ColumnName::new([logical]), + column_mapping_mode, + ) + .expect("top-level column must have a physical name") + .into_inner() + .into_iter() + .next() + .expect("top-level physical column path must not be empty") + }; + let data_column = physical_name("c"); + let partition_column = physical_name("p"); + + let data_schema = StructType::try_new(vec![ + StructField::nullable("id", DataType::LONG), + StructField::nullable("c", DataType::INTEGER), + ])?; + let data_columns: Vec = vec![ + Arc::new(Int64Array::from(vec![1, 2, 3])), + Arc::new(Int32Array::from(vec![42, 42, 42])), + ]; + let batch = RecordBatch::try_new( + Arc::new((&data_schema).try_into_arrow()?), + data_columns.clone(), + )?; + let post_commit_snapshot = write_batch_to_table( + &snapshot, + engine.as_ref(), + batch, + HashMap::from([("p".to_string(), Scalar::Integer(7))]), + ) + .await?; + + // Check the engine-facing schema because the Parquet writer does not preserve arbitrary + // Arrow field metadata. + let checkpoint_writer = post_commit_snapshot + .clone() + .create_checkpoint_writer(engine.as_ref())?; + let mut checkpoint_data = checkpoint_writer.checkpoint_data(engine.as_ref())?; + let checkpoint_batch = checkpoint_data + .next() + .expect("checkpoint data must contain at least one batch")? + .apply_selection_vector()? + .try_into_record_batch()?; + for batch in checkpoint_data { + batch?; + } + let checkpoint_data_schema = StructType::try_from_arrow(checkpoint_batch.schema())?; + assert_checkpoint_parsed_columns_have_no_column_defaults( + &checkpoint_data_schema, + &data_column, + &partition_column, + ); + + let checkpoint_version = post_commit_snapshot.version(); + post_commit_snapshot.checkpoint(engine.as_ref(), None)?; + let checkpoint_path = + load_existing_single_file_checkpoint_path(&table_path, checkpoint_version); + let checkpoint_reader = + ParquetRecordBatchReaderBuilder::try_new(fs::File::open(checkpoint_path)?)?; + let checkpoint_file_schema = StructType::try_from_arrow(checkpoint_reader.schema().clone())?; + assert_checkpoint_parsed_columns_have_no_column_defaults( + &checkpoint_file_schema, + &data_column, + &partition_column, + ); + + let reloaded_snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let reloaded_table_schema = reloaded_snapshot.schema(); + let default = reloaded_table_schema + .field("c") + .expect("c field must exist after checkpoint reload") + .column_default()? + .expect("c default must survive checkpoint reload"); + assert_eq!(default.to_scalar()?, Some(Scalar::Integer(42))); + + let expected = RecordBatch::try_new( + Arc::new(schema.as_ref().try_into_arrow()?), + vec![ + data_columns[0].clone(), + data_columns[1].clone(), + Arc::new(Int32Array::from(vec![7, 7, 7])), + ], + )?; + test_read(&ArrowEngineData::new(expected), &table_url, engine)?; + + Ok(()) +} + +#[tokio::test] +async fn test_partition_column_default_round_trips_on_read( +) -> Result<(), Box> { + let base = StructType::try_new(vec![ + StructField::nullable("id", DataType::LONG), + StructField::nullable("p", DataType::INTEGER), + ])?; + let schema = schema_with_column_defaults(&base, HashMap::from([("p", "42")]))?; + + let (store, engine, table_location) = + engine_store_setup("test_partition_column_default_round_trip", None); + let table_url = create_table( + store, + table_location, + schema.clone(), + &["p"], + true, + vec![], + vec!["allowColumnDefaults"], + ) + .await?; + let engine = Arc::new(engine); + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + + let data_schema = StructType::try_new(vec![StructField::nullable("id", DataType::LONG)])?; + let data = RecordBatch::try_new( + Arc::new((&data_schema).try_into_arrow()?), + vec![Arc::new(Int64Array::from(vec![1, 2, 3]))], + )?; + write_batch_to_table( + &snapshot, + engine.as_ref(), + data, + HashMap::from([("p".to_string(), Scalar::Integer(42))]), + ) + .await?; + + let expected = RecordBatch::try_new( + Arc::new(schema.as_ref().try_into_arrow()?), + vec![ + Arc::new(Int64Array::from(vec![1, 2, 3])), + Arc::new(Int32Array::from(vec![42, 42, 42])), + ], + )?; + test_read(&ArrowEngineData::new(expected), &table_url, engine)?; + + Ok(()) +} + +/// End-to-end: a literal column default composes with `icebergCompatV3`. The default survives +/// the column-mapping transform (so it is still discoverable by its logical name), and a write +/// that materializes the default round-trips on read. +#[tokio::test(flavor = "multi_thread")] +async fn test_column_default_with_iceberg_compat_v3_e2e() -> Result<(), Box> +{ + let base = StructType::try_new(vec![ + StructField::nullable("id", DataType::LONG), + StructField::nullable("c", DataType::INTEGER), + ])?; + let schema = schema_with_column_defaults(&base, HashMap::from([("c", "42")]))?; + + let (store, engine, table_location) = engine_store_setup("test_v3_col_default_e2e", None); + // The helper auto-enables V3's dependencies (columnMapping, rowTracking, domainMetadata) + // and assigns the per-field column-mapping metadata a valid V3 table requires. + let table_url = create_table( + store, + table_location, + schema.clone(), + &[], /* partition_columns */ + true, /* use_37_protocol */ + vec![], /* reader_features */ + vec!["allowColumnDefaults", "icebergCompatV3"], + ) + .await?; + let engine = Arc::new(engine); + + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let writer_features = snapshot + .table_configuration() + .protocol() + .writer_features() + .expect("writer_features must be present on a writer v7 table"); + for feature in [ + TableFeature::IcebergCompatV3, + TableFeature::RowTracking, + TableFeature::ColumnMapping, + TableFeature::AllowColumnDefaults, + ] { + assert!( + writer_features.contains(&feature), + "writer_features must include {feature:?}; got {writer_features:?}", + ); + } + + // The default is still keyed by the logical name `c` and parses to its literal. + let txn = snapshot + .clone() + .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())?; + let defaults = txn.top_level_column_defaults()?; + assert_eq!(defaults["c"].to_scalar()?, Some(Scalar::Integer(42))); + drop(defaults); + drop(txn); + + // The connector materializes the default (42) into the batch, then writes. + let columns: Vec = vec![ + Arc::new(Int64Array::from(vec![1, 2, 3])), + Arc::new(Int32Array::from(vec![42, 42, 42])), + ]; + assert!(insert_data(snapshot, &engine, columns.clone()) + .await? + .is_committed()); + + let data = RecordBatch::try_new(Arc::new(schema.as_ref().try_into_arrow()?), columns)?; + test_read(&ArrowEngineData::new(data), &table_url, engine)?; + + Ok(()) +} diff --git a/kernel/tests/integration/write/column_mapping.rs b/kernel/tests/integration/write/column_mapping.rs new file mode 100644 index 0000000000..ab82d1ac73 --- /dev/null +++ b/kernel/tests/integration/write/column_mapping.rs @@ -0,0 +1,555 @@ +//! Integration tests for column-mapping-aware write paths (stats, partition values, removes). + +use std::collections::HashMap; +use std::sync::Arc; + +use delta_kernel::actions::{MAX_VALUES, MIN_VALUES}; +use delta_kernel::arrow::array::{ + Array, ArrayRef, Int32Array, Int64Array, RecordBatch, StringArray, StructArray, +}; +use delta_kernel::arrow::datatypes::{DataType as ArrowDataType, Schema as ArrowSchema}; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::engine::arrow_conversion::TryIntoArrow as _; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::expressions::{column_name, ColumnName, Scalar}; +use delta_kernel::object_store::local::LocalFileSystem; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::{DynObjectStore, ObjectStoreExt as _}; +use delta_kernel::scan::StatsOptions; +use delta_kernel::schema::{schema_ref, MetadataValue, StructType}; +use delta_kernel::table_features::{get_any_level_column_physical_name, ColumnMappingMode}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::{Engine, FileMeta, Snapshot}; +use test_utils::delta_kernel_default_engine::executor::tokio::TokioMultiThreadExecutor; +use test_utils::delta_kernel_default_engine::DefaultEngineBuilder; +use test_utils::{ + add_commit, assert_partition_values, assert_schema_has_field, + column_mapping_fixtures as fixtures, copy_directory, create_table_and_load_snapshot, + engine_store_setup, nested_batches, nested_schema, read_actions_from_commit, read_add_infos, + read_scan, remove_all_and_get_remove_actions, resolve_field, test_table_setup, + write_batch_to_table, +}; +use url::Url; + +use crate::common::write_utils::{ + assert_min_max_stats, get_parquet_field_id, resolve_struct_field, set_table_properties, +}; + +/// 1. Creates a table with the given column mapping mode +/// 2. Writes two batches of data +/// 3. Checkpoints and verifies add.stats uses physical column names in the checkpoint +/// 4. Reads a parquet footer to verify physical names/IDs +/// 5. Reads data back to verify correctness +/// 6. Removes files and verifies remove.stats matches the original add.stats +#[rstest::rstest] +#[case::cm_none(ColumnMappingMode::None)] +#[case::cm_id(ColumnMappingMode::Id)] +#[case::cm_name(ColumnMappingMode::Name)] +#[tokio::test(flavor = "multi_thread")] +async fn test_column_mapping_write( + #[case] cm_mode: ColumnMappingMode, +) -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + let schema = nested_schema()?; + + let (_tmp_dir, table_path, _) = test_table_setup()?; + let table_url = Url::from_directory_path(&table_path).unwrap(); + let store: Arc = Arc::new(LocalFileSystem::new()); + let engine = Arc::new( + DefaultEngineBuilder::new(store.clone()) + .with_task_executor(Arc::new(TokioMultiThreadExecutor::new( + tokio::runtime::Handle::current(), + ))) + .build(), + ); + + // Step 1: Create table + let mode_str = match cm_mode { + ColumnMappingMode::None => "none", + ColumnMappingMode::Id => "id", + ColumnMappingMode::Name => "name", + }; + let mut latest_snapshot = create_table_and_load_snapshot( + &table_path, + schema.clone(), + engine.as_ref(), + &[("delta.columnMapping.mode", mode_str)], + )?; + + // Get physical field paths for stats verification (top-level and nested) + let cm = latest_snapshot + .table_properties() + .column_mapping_mode + .unwrap_or(ColumnMappingMode::None); + let row_number_physical = get_any_level_column_physical_name( + latest_snapshot.schema().as_ref(), + &column_name!("row_number"), + cm, + )? + .into_inner(); + let street_physical = get_any_level_column_physical_name( + latest_snapshot.schema().as_ref(), + &column_name!("address.street"), + cm, + )? + .into_inner(); + + // Step 2: Write two batches + for data in nested_batches()? { + latest_snapshot = + write_batch_to_table(&latest_snapshot, engine.as_ref(), data, HashMap::new()).await?; + } + + // Enable writeStatsAsStruct so the checkpoint contains native stats_parsed. + // CREATE TABLE doesn't allow this property yet, so we write a metadata-update commit directly. + latest_snapshot = set_table_properties( + &table_path, + &table_url, + engine.as_ref(), + latest_snapshot.version(), + &[("delta.checkpoint.writeStatsAsStruct", "true")], + )?; + + // Step 3: Checkpoint and verify add.stats uses correct column names + let snapshot_for_checkpoint = latest_snapshot.clone(); + snapshot_for_checkpoint.checkpoint(engine.as_ref(), None)?; + let ckpt_snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let add_actions = read_add_infos(&ckpt_snapshot, engine.as_ref())?; + let mut all_stats: Vec<_> = add_actions + .iter() + .filter_map(|a| a.stats.as_ref()) + .filter(|s| s.get(MIN_VALUES).is_some()) + .collect(); + assert_eq!(all_stats.len(), 2, "should have stats for 2 files"); + all_stats.sort_by_key(|s| s[MIN_VALUES][&row_number_physical[0]].as_i64().unwrap()); + + // Batch 1: row_number 1..3, address.street "st1".."st3" + assert_min_max_stats(all_stats[0], &row_number_physical, 1, 3); + assert_min_max_stats(all_stats[0], &street_physical, "st1", "st3"); + + // Batch 2: row_number 4..6, address.street "st4".."st6" + assert_min_max_stats(all_stats[1], &row_number_physical, 4, 6); + assert_min_max_stats(all_stats[1], &street_physical, "st4", "st6"); + + // Step 3b: Verify stats_parsed in scan metadata uses correct physical column names + { + let scan = ckpt_snapshot + .scan_builder() + .with_stats(StatsOptions::all()) + .build()?; + let scan_metadata_results: Vec<_> = scan + .scan_metadata(engine.as_ref())? + .collect::, _>>()?; + + let mut stats_rows: Vec<(i64, i64, String, String)> = Vec::new(); + for sm in scan_metadata_results { + let (data, sel) = sm.scan_files.into_parts(); + let batch: RecordBatch = ArrowEngineData::try_from_engine_data(data)?.into(); + + let batch_struct = StructArray::from(batch.clone()); + let stats_parsed: &StructArray = + resolve_struct_field(&batch_struct, &["stats_parsed".into()]); + + let min_path = |field: &[String]| -> Vec { + [&["stats_parsed".into(), MIN_VALUES.into()], field].concat() + }; + let max_path = |field: &[String]| -> Vec { + [&["stats_parsed".into(), MAX_VALUES.into()], field].concat() + }; + let min_row_num: &Int64Array = + resolve_struct_field(&batch_struct, &min_path(&row_number_physical)); + let max_row_num: &Int64Array = + resolve_struct_field(&batch_struct, &max_path(&row_number_physical)); + let min_st: &StringArray = + resolve_struct_field(&batch_struct, &min_path(&street_physical)); + let max_st: &StringArray = + resolve_struct_field(&batch_struct, &max_path(&street_physical)); + + for (i, &selected) in sel.iter().enumerate().take(batch.num_rows()) { + if selected && !stats_parsed.is_null(i) { + stats_rows.push(( + min_row_num.value(i), + max_row_num.value(i), + min_st.value(i).to_string(), + max_st.value(i).to_string(), + )); + } + } + } + + stats_rows.sort_by_key(|r| r.0); + assert_eq!(stats_rows.len(), 2, "should have stats_parsed for 2 files"); + assert_eq!(stats_rows[0], (1, 3, "st1".to_string(), "st3".to_string())); + assert_eq!(stats_rows[1], (4, 6, "st4".to_string(), "st6".to_string())); + } + + // Step 4: Read parquet footer to verify physical names and native field_id + { + let parquet_path = &add_actions + .first() + .expect("should have at least one add file") + .path; + let parquet_url = table_url.join(parquet_path)?; + let local_path = parquet_url.to_file_path().unwrap(); + + let obj_meta = store + .head(&Path::from_url_path(parquet_url.path())?) + .await?; + let file_meta = FileMeta::new( + parquet_url, + 0, /* last_modified */ + obj_meta.size as u64, + ); + let footer = engine.parquet_handler().read_parquet_footer(&file_meta)?; + let footer_schema = footer.schema; + + let logical_schema = latest_snapshot.schema(); + for logical_path in [&["row_number"][..], &["address", "street"]] { + let col = ColumnName::new(logical_path.iter().copied()); + let physical = + get_any_level_column_physical_name(logical_schema.as_ref(), &col, cm)?.into_inner(); + assert_schema_has_field(&footer_schema, &physical); + + let field_id = get_parquet_field_id(&local_path, &physical); + let logical_field = resolve_field(logical_schema.as_ref(), logical_path).unwrap(); + match cm_mode { + ColumnMappingMode::Id | ColumnMappingMode::Name => { + let expected_id = logical_field + .column_mapping_id() + .expect("expected ColumnMappingId number") + as i32; + assert_eq!( + field_id, + Some(expected_id), + "parquet field_id mismatch for {logical_path:?}" + ); + } + ColumnMappingMode::None => { + assert_eq!( + field_id, None, + "parquet field_id should not be set in None column mapping mode" + ); + } + } + } + } + + // Step 5: Read data back to verify correctness + { + let post_ckpt_snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let scan = post_ckpt_snapshot.scan_builder().build()?; + let batches: Vec = scan + .execute(engine.clone())? + .map(|r| { + let data = r.unwrap(); + let arrow = ArrowEngineData::try_from_engine_data(data).unwrap(); + arrow.record_batch().clone() + }) + .collect(); + + let result_schema = batches[0].schema(); + let combined = delta_kernel::arrow::compute::concat_batches(&result_schema, &batches)?; + assert_eq!( + combined.num_rows(), + 6, + "Should have 6 rows from two written batches" + ); + + // Verify logical column names and data values + let combined_struct = StructArray::from(combined); + + // Top-level: row_number should contain [1..=6] + let row_numbers: &Int64Array = + resolve_struct_field(&combined_struct, &["row_number".into()]); + let mut vals: Vec = (0..row_numbers.len()) + .map(|i| row_numbers.value(i)) + .collect(); + vals.sort(); + assert_eq!(vals, vec![1, 2, 3, 4, 5, 6]); + + // Nested: address.street should contain ["st1"..="st6"] + let streets: &StringArray = + resolve_struct_field(&combined_struct, &["address".into(), "street".into()]); + let mut street_vals: Vec<&str> = (0..streets.len()).map(|i| streets.value(i)).collect(); + street_vals.sort(); + assert_eq!(street_vals, vec!["st1", "st2", "st3", "st4", "st5", "st6"]); + } + + // Step 6: Remove files and verify remove.stats matches original add.stats + { + let original_add_stats: Vec = + add_actions.iter().filter_map(|a| a.stats.clone()).collect(); + assert!( + !original_add_stats.is_empty(), + "should have at least one add with stats" + ); + + let remove_actions = + remove_all_and_get_remove_actions(&latest_snapshot, &table_url, engine.as_ref())?; + assert!( + !remove_actions.is_empty(), + "Expected at least one remove action" + ); + + let remove_stats: Vec = remove_actions + .iter() + .filter_map(|r| { + r["stats"] + .as_str() + .map(|s| serde_json::from_str(s).unwrap()) + }) + .collect(); + assert_eq!( + remove_stats, original_add_stats, + "remove.stats should match original add.stats" + ); + } + + Ok(()) +} + +/// Verifies that partitioned writes use physical column names in add.partitionValues. +#[rstest::rstest] +#[case::cm_none("./tests/data/partition_cm/none")] +#[case::cm_id("./tests/data/partition_cm/id")] +#[case::cm_name("./tests/data/partition_cm/name")] +#[tokio::test(flavor = "multi_thread")] +async fn test_column_mapping_partitioned_write( + #[case] table_dir: &str, +) -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + // Copy test data to a temp dir so we can write to it + let tmp_dir = tempfile::tempdir()?; + copy_directory(std::path::Path::new(table_dir), tmp_dir.path())?; + let table_url = Url::from_directory_path(tmp_dir.path()).unwrap(); + let store: Arc = Arc::new(LocalFileSystem::new()); + let engine = Arc::new( + DefaultEngineBuilder::new(store.clone()) + .with_task_executor(Arc::new(TokioMultiThreadExecutor::new( + tokio::runtime::Handle::current(), + ))) + .build(), + ); + + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let cm = snapshot + .table_properties() + .column_mapping_mode + .unwrap_or(ColumnMappingMode::None); + let physical_name = get_any_level_column_physical_name( + snapshot.schema().as_ref(), + &column_name!("category"), + cm, + )? + .into_inner() + .remove(0); + + // Verify physical name for column mapping mode + if table_dir.ends_with("none") { + assert_eq!(physical_name, "category"); + } else { + assert_ne!( + physical_name, "category", + "physical name should differ from logical name under column mapping" + ); + } + + // Write data with partition value + let data_schema = schema_ref! { nullable "value": INTEGER }; + let batch = RecordBatch::try_new( + Arc::new(data_schema.as_ref().try_into_arrow()?), + vec![Arc::new(Int32Array::from(vec![1, 2]))], + )?; + let partition_values = HashMap::from([("category".to_string(), Scalar::String("A".into()))]); + write_batch_to_table(&snapshot, engine.as_ref(), batch, partition_values).await?; + + // Read commit log and verify add.partitionValues key uses physical name + let add_actions = read_actions_from_commit(&table_url, 1, "add")?; + assert!(!add_actions.is_empty(), "no add action found in commit log"); + for add in &add_actions { + assert_partition_values(add, &physical_name, "A"); + } + + // Remove the written file and verify remove action preserves physical names + let post_write_snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let remove_actions = + remove_all_and_get_remove_actions(&post_write_snapshot, &table_url, engine.as_ref())?; + assert!( + !remove_actions.is_empty(), + "no remove action found in commit log" + ); + for remove in &remove_actions { + assert_partition_values(remove, &physical_name, "A"); + } + + Ok(()) +} + +// Two fields with same physical name at different physical paths is valid. Write +// should succeed. +#[rstest::rstest] +#[tokio::test(flavor = "multi_thread")] +async fn test_same_phy_name_different_path( + #[values("name", "id")] cm_mode: &str, +) -> Result<(), Box> { + let (_tmp_dir, table_path, _) = test_table_setup()?; + let table_url = Url::from_directory_path(&table_path).unwrap(); + let store: Arc = Arc::new(LocalFileSystem::new()); + let engine = Arc::new( + DefaultEngineBuilder::new(store.clone()) + .with_task_executor(Arc::new(TokioMultiThreadExecutor::new( + tokio::runtime::Handle::current(), + ))) + .build(), + ); + + let logical_schema = Arc::new(fixtures::same_leaf_phy_name_under_different_parents()); + let snapshot = create_table(table_url.as_str(), logical_schema.clone(), "Test/1.0") + .with_table_properties([("delta.columnMapping.mode", cm_mode)]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + + // Build a RecordBatch matching the logical schema. + let arrow_schema: ArrowSchema = logical_schema.as_ref().try_into_arrow()?; + let inner_fields_of = + |outer: &str| match arrow_schema.field_with_name(outer).unwrap().data_type() { + ArrowDataType::Struct(fields) => fields.clone(), + _ => panic!("expected struct for {outer}"), + }; + let outer1_array = StructArray::new( + inner_fields_of("outer1"), + vec![Arc::new(Int32Array::from(vec![10, 20])) as ArrayRef], + None, + ); + let outer2_array = StructArray::new( + inner_fields_of("outer2"), + vec![Arc::new(Int32Array::from(vec![100, 200])) as ArrayRef], + None, + ); + let batch = RecordBatch::try_new( + Arc::new(arrow_schema), + vec![Arc::new(outer1_array), Arc::new(outer2_array)], + )?; + + let snapshot = write_batch_to_table(&snapshot, engine.as_ref(), batch, HashMap::new()).await?; + + // Scan back and verify the round-tripped rows. + let scan = snapshot.scan_builder().build()?; + let batches = read_scan(&scan, engine.clone())?; + let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total_rows, 2, "expected 2 rows back"); + let combined = StructArray::from(batches[0].clone()); + let outer1_a: &Int32Array = resolve_struct_field(&combined, &["outer1".into(), "a".into()]); + let outer2_a: &Int32Array = resolve_struct_field(&combined, &["outer2".into(), "a".into()]); + assert_eq!(outer1_a.values(), &[10, 20]); + assert_eq!(outer2_a.values(), &[100, 200]); + Ok(()) +} + +/// A schema with two fields sharing same physical path should be rejected. +#[rstest::rstest] +#[tokio::test] +async fn test_duplicated_phy_path_rejected( + #[values("name", "id")] cm_mode: &str, +) -> Result<(), Box> { + let (store, engine, table_url) = engine_store_setup("dup_phys_path", None); + let schema = fixtures::nested_field_with_same_phy_path(); + let schema_json = serde_json::to_string(&schema)?; + let escaped = serde_json::to_string(&schema_json)?; + // Create a v0 commit in a hack way to bypass create_table validation. + let v0 = format!( + r#"{{"protocol":{{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["columnMapping"],"writerFeatures":["columnMapping"]}}}} +{{"metaData":{{"id":"test-id","format":{{"provider":"parquet","options":{{}}}},"schemaString":{escaped},"partitionColumns":[],"configuration":{{"delta.columnMapping.mode":"{cm_mode}","delta.columnMapping.maxColumnId":"4"}},"createdTime":1700000000000}}}} +"# + ); + add_commit(table_url.as_str(), store.as_ref(), 0, v0).await?; + + let msg = Snapshot::builder_for(table_url) + .build(&engine) + .expect_err("dup physicalName must be rejected at snapshot load") + .to_string(); + assert!( + msg.contains("Duplicate `delta.columnMapping.physicalName`") + && msg.contains(".a'") + && msg.contains(".b'"), + "expected path-aware dedup error naming colliding leaves, got: {msg}" + ); + Ok(()) +} + +/// A table left with residual `delta.columnMapping.*` annotations while mapping is off must still +/// be readable and appendable: the annotations are inert and columns resolve by logical name. +/// Kernel matches delta-spark, which ignores residual annotations in `NoMapping` mode. +#[tokio::test(flavor = "multi_thread")] +async fn test_read_and_append_tolerates_stale_column_mapping_when_disabled( +) -> Result<(), Box> { + let tmp_dir = tempfile::tempdir()?; + let table_url = Url::from_directory_path(tmp_dir.path()).unwrap(); + let store: Arc = Arc::new(LocalFileSystem::new()); + let engine = Arc::new( + DefaultEngineBuilder::new(store.clone()) + .with_task_executor(Arc::new(TokioMultiThreadExecutor::new( + tokio::runtime::Handle::current(), + ))) + .build(), + ); + + // A `value` column carrying a stale physicalName + id, with no columnMapping feature in the + // protocol and mode absent (so it resolves to None). + let stale_schema = schema_ref! { + nullable "id": INTEGER, + nullable "value": INTEGER + }; + let stale_schema = StructType::try_new([ + stale_schema.field("id").unwrap().clone(), + stale_schema.field("value").unwrap().clone().add_metadata([ + ("delta.columnMapping.id", MetadataValue::Number(2)), + ( + "delta.columnMapping.physicalName", + MetadataValue::String("col-2f8a".to_string()), + ), + ]), + ])?; + let escaped = serde_json::to_string(&serde_json::to_string(&stale_schema)?)?; + // Create a v0 commit directly to bypass create_table validation (which would reject the + // stale annotations on the write path). + let v0 = format!( + r#"{{"protocol":{{"minReaderVersion":1,"minWriterVersion":2}}}} +{{"metaData":{{"id":"stale-cm-id","format":{{"provider":"parquet","options":{{}}}},"schemaString":{escaped},"partitionColumns":[],"configuration":{{}},"createdTime":1700000000000}}}} +"# + ); + add_commit(table_url.as_str(), store.as_ref(), 0, v0).await?; + + // Snapshot load succeeds and reports column mapping disabled. + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + assert_eq!(snapshot.table_properties().column_mapping_mode, None); + // The `value` column resolves by its logical name, not the stale physical name. + let physical_name = get_any_level_column_physical_name( + snapshot.schema().as_ref(), + &column_name!("value"), + ColumnMappingMode::None, + )? + .into_inner() + .remove(0); + assert_eq!(physical_name, "value"); + + // Write data through the tolerated snapshot and read it back to confirm the full path works. + let batch = RecordBatch::try_new( + Arc::new(snapshot.schema().as_ref().try_into_arrow()?), + vec![ + Arc::new(Int32Array::from(vec![1, 2])), + Arc::new(Int32Array::from(vec![10, 20])), + ], + )?; + let post_write = + write_batch_to_table(&snapshot, engine.as_ref(), batch, HashMap::new()).await?; + + let scan = post_write.scan_builder().build()?; + let batches = read_scan(&scan, engine.clone())?; + let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total_rows, 2, "expected the two written rows back"); + Ok(()) +} diff --git a/kernel/tests/integration/write/commit_info.rs b/kernel/tests/integration/write/commit_info.rs new file mode 100644 index 0000000000..c8bfefcacf --- /dev/null +++ b/kernel/tests/integration/write/commit_info.rs @@ -0,0 +1,194 @@ +//! Integration tests that exercise CommitInfo generation for kernel-authored commits. + +use std::sync::Arc; + +use delta_kernel::arrow::array::{ArrayRef, RecordBatch, StringArray}; +use delta_kernel::arrow::datatypes::{DataType as ArrowDataType, Field, Schema as ArrowSchema}; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::ObjectStoreExt as _; +use delta_kernel::schema::{DataType, StructField, StructType}; +use itertools::Itertools; +use serde_json::{json, Deserializer}; +use test_utils::{load_and_begin_transaction, set_json_value, setup_test_tables}; + +use crate::common::write_utils::{ + get_simple_int_schema, validate_timestamp, validate_txn_id, ZERO_UUID, +}; + +#[tokio::test] +async fn test_commit_info() -> Result<(), Box> { + // setup tracing + let _ = tracing_subscriber::fmt::try_init(); + + // create a simple table: one int column named 'number' + let schema = get_simple_int_schema(); + + for (table_url, engine, store, table_name) in + setup_test_tables(schema, &[], None, "test_table").await? + { + // create a transaction + let txn = load_and_begin_transaction(table_url.clone(), &engine)? + .with_engine_info("default engine"); + + // commit! + let _ = txn.commit(&engine)?; + + let commit1 = store + .get(&Path::from(format!( + "/{table_name}/_delta_log/00000000000000000001.json" + ))) + .await?; + + let mut parsed_commit: serde_json::Value = serde_json::from_slice(&commit1.bytes().await?)?; + + validate_txn_id(&parsed_commit["commitInfo"]); + + set_json_value(&mut parsed_commit, "commitInfo.timestamp", json!(0))?; + set_json_value(&mut parsed_commit, "commitInfo.txnId", json!(ZERO_UUID))?; + + let expected_commit = json!({ + "commitInfo": { + "timestamp": 0, + "operation": "UNKNOWN", + "kernelVersion": format!("v{}", env!("CARGO_PKG_VERSION")), + "operationParameters": {}, + "engineInfo": "default engine", + "txnId": ZERO_UUID, + } + }); + + assert_eq!(parsed_commit, expected_commit); + } + Ok(()) +} + +#[tokio::test] +async fn test_commit_info_action() -> Result<(), Box> { + // setup tracing + let _ = tracing_subscriber::fmt::try_init(); + // create a simple table: one int column named 'number' + let schema = get_simple_int_schema(); + + for (table_url, engine, store, table_name) in + setup_test_tables(schema.clone(), &[], None, "test_table").await? + { + let txn = load_and_begin_transaction(table_url.clone(), &engine)? + .with_engine_info("default engine"); + + let _ = txn.commit(&engine)?; + + let commit = store + .get(&Path::from(format!( + "/{table_name}/_delta_log/00000000000000000001.json" + ))) + .await?; + + let mut parsed_commits: Vec<_> = Deserializer::from_slice(&commit.bytes().await?) + .into_iter::() + .try_collect()?; + + validate_txn_id(&parsed_commits[0]["commitInfo"]); + + // set timestamps to 0, paths and txn_id to known string values for comparison + // (otherwise timestamps are non-deterministic, paths and txn_id are random UUIDs) + set_json_value(&mut parsed_commits[0], "commitInfo.timestamp", json!(0))?; + set_json_value(&mut parsed_commits[0], "commitInfo.txnId", json!(ZERO_UUID))?; + + let expected_commit = vec![json!({ + "commitInfo": { + "timestamp": 0, + "operation": "UNKNOWN", + "kernelVersion": format!("v{}", env!("CARGO_PKG_VERSION")), + "operationParameters": {}, + "engineInfo": "default engine", + "txnId": ZERO_UUID + } + })]; + + assert_eq!(parsed_commits, expected_commit); + } + Ok(()) +} + +/// Verifies that when `engine_commit_info` is provided (the `Some` branch of `build_commit_info`): +/// - The written JSON is correctly wrapped in a top-level `"commitInfo"` key. +/// - Engine-only fields (not in `CommitInfo::to_schema()`) pass through to the log unchanged. +/// - Fields that overlap with kernel-managed CommitInfo fields are overridden by kernel values, +/// - All kernel-managed fields (`timestamp`, `kernelVersion`, `txnId`, `operationParameters`) are +/// present with correct values. +#[tokio::test] +async fn test_commit_info_with_engine_commit_info() -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + let schema = get_simple_int_schema(); + + for (table_url, engine, store, table_name) in + setup_test_tables(schema, &[], None, "test_table").await? + { + // Build engine_commit_info with: + // - "myApp" : engine-only field, must pass through unchanged. + // - "myVersion": engine-only field, must pass through unchanged. + // - "operation": overlapping with CommitInfo; kernel must override with "WRITE". + let arrow_schema = Arc::new(ArrowSchema::new(vec![ + Field::new("myApp", ArrowDataType::Utf8, false), + Field::new("myVersion", ArrowDataType::Utf8, false), + Field::new("operation", ArrowDataType::Utf8, false), + ])); + let batch = RecordBatch::try_new( + arrow_schema, + vec![ + Arc::new(StringArray::from(vec!["spark"])) as ArrayRef, + Arc::new(StringArray::from(vec!["3.5.0"])) as ArrayRef, + Arc::new(StringArray::from(vec!["STALE_OP"])) as ArrayRef, + ], + )?; + let engine_schema = Arc::new(StructType::new_unchecked(vec![ + StructField::not_null("myApp", DataType::STRING), + StructField::not_null("myVersion", DataType::STRING), + StructField::nullable("operation", DataType::STRING), + ])); + + let txn = load_and_begin_transaction(table_url.clone(), &engine)? + .with_operation("WRITE".to_string()) + .with_commit_info(Box::new(ArrowEngineData::new(batch)), engine_schema); + + let _ = txn.commit(&engine)?; + + let commit = store + .get(&Path::from(format!( + "/{table_name}/_delta_log/00000000000000000001.json" + ))) + .await?; + + let mut parsed_commits: Vec<_> = Deserializer::from_slice(&commit.bytes().await?) + .into_iter::() + .try_collect()?; + + validate_txn_id(&parsed_commits[0]["commitInfo"]); + validate_timestamp(&parsed_commits[0]["commitInfo"]); + + // Zero out non-deterministic fields for stable comparison. + set_json_value(&mut parsed_commits[0], "commitInfo.timestamp", json!(0))?; + set_json_value(&mut parsed_commits[0], "commitInfo.txnId", json!(ZERO_UUID))?; + + // Null-valued CommitInfo fields (inCommitTimestamp, isBlindAppend, engineInfo) are + // omitted from the JSON -- consistent with how the Delta log serializes optional fields. + let expected_commits = vec![json!({ + "commitInfo": { + // Engine-only fields pass through unchanged. + "myApp": "spark", + "myVersion": "3.5.0", + // Kernel overrides the engine's stale "STALE_OP" with the real operation. + "operation": "WRITE", + // Remaining kernel-managed non-null fields are appended. + "operationParameters": {}, + "kernelVersion": format!("v{}", env!("CARGO_PKG_VERSION")), + "txnId": ZERO_UUID, + "timestamp": 0, + } + })]; + + assert_eq!(parsed_commits, expected_commits); + } + Ok(()) +} diff --git a/kernel/tests/integration/write/domain_metadata.rs b/kernel/tests/integration/write/domain_metadata.rs new file mode 100644 index 0000000000..bf462ca2d0 --- /dev/null +++ b/kernel/tests/integration/write/domain_metadata.rs @@ -0,0 +1,384 @@ +//! Integration tests for domain metadata set/remove flows. + +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::ObjectStoreExt as _; +use delta_kernel::Snapshot; +use itertools::Itertools; +use serde_json::Deserializer; +use test_utils::{ + assert_result_error_with_message, begin_transaction, create_table, engine_store_setup, + load_and_begin_transaction, +}; + +use crate::common::write_utils::get_simple_int_schema; + +#[tokio::test] +async fn test_set_domain_metadata_basic() -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + let schema = get_simple_int_schema(); + + let table_name = "test_domain_metadata_basic"; + + let (store, engine, table_location) = engine_store_setup(table_name, None); + let table_url = create_table( + store.clone(), + table_location, + schema.clone(), + &[], + true, + vec![], + vec!["domainMetadata"], + ) + .await?; + + let txn = load_and_begin_transaction(table_url.clone(), &engine)?; + + // write context does not conflict with domain metadata + let _write_context = txn.unpartitioned_write_context().unwrap(); + + // set multiple domain metadata + let domain1 = "app.config"; + let config1 = r#"{"version": 1}"#; + let domain2 = "spark.settings"; + let config2 = r#"{"cores": 4}"#; + + assert!(txn + .with_domain_metadata(domain1.to_string(), config1.to_string()) + .with_domain_metadata(domain2.to_string(), config2.to_string()) + .commit(&engine)? + .is_committed()); + + let commit_data = store + .get(&Path::from(format!( + "/{table_name}/_delta_log/00000000000000000001.json" + ))) + .await? + .bytes() + .await?; + + let actions: Vec = Deserializer::from_slice(&commit_data) + .into_iter() + .try_collect()?; + + let domain_actions: Vec<_> = actions + .iter() + .filter(|v| v.get("domainMetadata").is_some()) + .collect(); + + for action in &domain_actions { + let domain = action["domainMetadata"]["domain"].as_str().unwrap(); + let config = action["domainMetadata"]["configuration"].as_str().unwrap(); + assert!(!action["domainMetadata"]["removed"].as_bool().unwrap()); + + match domain { + d if d == domain1 => assert_eq!(config, config1), + d if d == domain2 => assert_eq!(config, config2), + _ => panic!("Unexpected domain: {domain}"), + } + } + + let final_snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; + let domain1_config = final_snapshot.get_domain_metadata(domain1, &engine)?; + assert_eq!(domain1_config, Some(config1.to_string())); + let domain2_config = final_snapshot.get_domain_metadata(domain2, &engine)?; + assert_eq!(domain2_config, Some(config2.to_string())); + Ok(()) +} + +#[tokio::test] +async fn test_set_domain_metadata_errors() -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + let schema = get_simple_int_schema(); + + let table_name = "test_domain_metadata_errors"; + let (store, engine, table_location) = engine_store_setup(table_name, None); + let table_url = create_table( + store.clone(), + table_location, + schema.clone(), + &[], + true, + vec![], + vec!["domainMetadata"], + ) + .await?; + + let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; + + // System domain rejection + let txn = begin_transaction(snapshot.clone(), &engine)?; + let res = txn + .with_domain_metadata("delta.system".to_string(), "config".to_string()) + .commit(&engine); + assert_result_error_with_message( + res, + "Cannot modify domains that start with 'delta.' as those are system controlled", + ); + + // Duplicate domain rejection + let txn2 = begin_transaction(snapshot.clone(), &engine)?; + let res = txn2 + .with_domain_metadata("app.config".to_string(), "v1".to_string()) + .with_domain_metadata("app.config".to_string(), "v2".to_string()) + .commit(&engine); + assert_result_error_with_message( + res, + "Metadata for domain app.config already specified in this transaction", + ); + + Ok(()) +} + +#[tokio::test] +async fn test_set_domain_metadata_unsupported_writer_feature( +) -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + let schema = get_simple_int_schema(); + + let table_name = "test_domain_metadata_unsupported"; + + // Create table WITHOUT domain metadata writer feature support + let (store, engine, table_location) = engine_store_setup(table_name, None); + let table_url = create_table( + store.clone(), + table_location, + schema.clone(), + &[], + true, + vec![], + vec![], + ) + .await?; + + let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; + let res = begin_transaction(snapshot, &engine)? + .with_domain_metadata("app.config".to_string(), "test_config".to_string()) + .commit(&engine); + + assert_result_error_with_message(res, "Domain metadata operations require writer version 7 and the 'domainMetadata' writer feature"); + + Ok(()) +} + +#[tokio::test] +async fn test_remove_domain_metadata_unsupported_writer_feature( +) -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + let schema = get_simple_int_schema(); + + let table_name = "test_remove_domain_metadata_unsupported"; + + // Create table WITHOUT domain metadata writer feature support + let (store, engine, table_location) = engine_store_setup(table_name, None); + let table_url = create_table( + store.clone(), + table_location, + schema.clone(), + &[], + true, + vec![], + vec![], + ) + .await?; + + let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; + let res = begin_transaction(snapshot, &engine)? + .with_domain_metadata_removed("app.config".to_string()) + .commit(&engine); + + assert_result_error_with_message(res, "Domain metadata operations require writer version 7 and the 'domainMetadata' writer feature"); + + Ok(()) +} + +#[tokio::test] +async fn test_remove_domain_metadata_non_existent_domain() -> Result<(), Box> +{ + let _ = tracing_subscriber::fmt::try_init(); + + let schema = get_simple_int_schema(); + + let table_name = "test_domain_metadata_unsupported"; + + let (store, engine, table_location) = engine_store_setup(table_name, None); + let table_url = create_table( + store.clone(), + table_location, + schema.clone(), + &[], + true, + vec![], + vec!["domainMetadata"], + ) + .await?; + + let txn = load_and_begin_transaction(table_url.clone(), &engine)?; + + let domain = "app.deprecated"; + + // removing domain metadata that doesn't exist should NOT write a tombstone + let _ = txn + .with_domain_metadata_removed(domain.to_string()) + .commit(&engine)?; + + let commit_data = store + .get(&Path::from(format!( + "/{table_name}/_delta_log/00000000000000000001.json" + ))) + .await? + .bytes() + .await?; + let actions: Vec = Deserializer::from_slice(&commit_data) + .into_iter() + .try_collect()?; + + let domain_action = actions.iter().find(|v| v.get("domainMetadata").is_some()); + assert!( + domain_action.is_none(), + "No tombstone should be written for non-existent domain" + ); + + let final_snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; + let config = final_snapshot.get_domain_metadata(domain, &engine)?; + assert_eq!(config, None); + + Ok(()) +} + +#[tokio::test] +async fn test_domain_metadata_set_remove_conflicts() -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + let schema = get_simple_int_schema(); + + let table_name = "test_domain_metadata_unsupported"; + + let (store, engine, table_location) = engine_store_setup(table_name, None); + let table_url = create_table( + store.clone(), + table_location, + schema.clone(), + &[], + true, + vec![], + vec!["domainMetadata"], + ) + .await?; + + let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; + + // set then remove same domain + let txn = begin_transaction(snapshot.clone(), &engine)?; + let err = txn + .with_domain_metadata("app.config".to_string(), "v1".to_string()) + .with_domain_metadata_removed("app.config".to_string()) + .commit(&engine) + .unwrap_err(); + assert!(err + .to_string() + .contains("already specified in this transaction")); + + // remove then set same domain + let txn2 = begin_transaction(snapshot.clone(), &engine)?; + let err = txn2 + .with_domain_metadata_removed("test.domain".to_string()) + .with_domain_metadata("test.domain".to_string(), "v1".to_string()) + .commit(&engine) + .unwrap_err(); + assert!(err + .to_string() + .contains("already specified in this transaction")); + + // remove same domain twice + let txn3 = begin_transaction(snapshot.clone(), &engine)?; + let err = txn3 + .with_domain_metadata_removed("another.domain".to_string()) + .with_domain_metadata_removed("another.domain".to_string()) + .commit(&engine) + .unwrap_err(); + assert!(err + .to_string() + .contains("already specified in this transaction")); + + // remove system domain + let txn4 = begin_transaction(snapshot.clone(), &engine)?; + let err = txn4 + .with_domain_metadata_removed("delta.system".to_string()) + .commit(&engine) + .unwrap_err(); + assert!(err + .to_string() + .contains("Cannot modify domains that start with 'delta.' as those are system controlled")); + + Ok(()) +} + +#[tokio::test] +async fn test_domain_metadata_set_then_remove() -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + let schema = get_simple_int_schema(); + + let table_name = "test_domain_metadata_unsupported"; + + let (store, engine, table_location) = engine_store_setup(table_name, None); + let table_url = create_table( + store.clone(), + table_location, + schema.clone(), + &[], + true, + vec![], + vec!["domainMetadata"], + ) + .await?; + + let domain = "app.config"; + let configuration = r#"{"version": 1}"#; + + // txn 1: set domain metadata + let txn = load_and_begin_transaction(table_url.clone(), &engine)?; + let _ = txn + .with_domain_metadata(domain.to_string(), configuration.to_string()) + .commit(&engine)?; + + // txn 2: remove the same domain metadata + let txn = load_and_begin_transaction(table_url.clone(), &engine)?; + let _ = txn + .with_domain_metadata_removed(domain.to_string()) + .commit(&engine)?; + + // verify removal commit preserves the previous configuration + let commit_data = store + .get(&Path::from(format!( + "/{table_name}/_delta_log/00000000000000000002.json" + ))) + .await? + .bytes() + .await?; + let actions: Vec = Deserializer::from_slice(&commit_data) + .into_iter() + .try_collect()?; + + let domain_action = actions + .iter() + .find(|v| v.get("domainMetadata").is_some()) + .unwrap(); + assert_eq!(domain_action["domainMetadata"]["domain"], domain); + assert_eq!( + domain_action["domainMetadata"]["configuration"], + configuration + ); + assert_eq!(domain_action["domainMetadata"]["removed"], true); + + // verify reads see the metadata removal + let final_snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; + let domain_config = final_snapshot.get_domain_metadata(domain, &engine)?; + assert_eq!(domain_config, None); + + Ok(()) +} diff --git a/kernel/tests/integration/write/ict.rs b/kernel/tests/integration/write/ict.rs new file mode 100644 index 0000000000..37937b4e43 --- /dev/null +++ b/kernel/tests/integration/write/ict.rs @@ -0,0 +1,193 @@ +//! Integration tests for the in-commit timestamp (ICT) write path. + +use std::sync::Arc; + +use delta_kernel::arrow::array::Int32Array; +use delta_kernel::arrow::record_batch::RecordBatch; +use delta_kernel::engine::arrow_conversion::TryIntoArrow as _; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::{DynObjectStore, ObjectStoreExt as _}; +use delta_kernel::schema::SchemaRef; +use delta_kernel::transaction::CommitResult; +use delta_kernel::Snapshot; +use tempfile::TempDir; +use test_utils::delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor; +use test_utils::delta_kernel_default_engine::DefaultEngine; +use test_utils::{engine_store_setup, load_and_begin_transaction}; +use url::Url; + +use crate::common::write_utils::get_simple_int_schema; + +async fn get_ict_at_version( + store: Arc, + table_url: &Url, + version: u64, +) -> Result> { + let commit_path = table_url.join(&format!("_delta_log/{version:020}.json"))?; + let commit = store.get(&Path::from_url_path(commit_path.path())?).await?; + let commit_content = String::from_utf8(commit.bytes().await?.to_vec())?; + + // Parse each line of the commit log (NDJSON format) + // CommitInfo MUST be the first action when ICT is enabled + let lines: Vec<_> = commit_content + .lines() + .filter(|line| !line.trim().is_empty()) + .collect(); + assert!( + !lines.is_empty(), + "Commit log at version {version} should not be empty" + ); + + // First line should contain commitInfo with inCommitTimestamp + let first_action: serde_json::Value = serde_json::from_str(lines[0])?; + let commit_info = first_action + .get("commitInfo") + .expect("First action must be commitInfo when ICT is enabled"); + let ict = commit_info + .get("inCommitTimestamp") + .expect("commitInfo must have inCommitTimestamp when ICT is enabled") + .as_i64() + .unwrap(); + Ok(ict) +} + +/// Helper function to generate a simple data file and add it to the transaction +/// This simplifies repetitive data generation in tests +async fn generate_and_add_data_file( + txn: &mut delta_kernel::transaction::Transaction, + engine: &DefaultEngine, + schema: SchemaRef, + values: Vec, +) -> Result<(), Box> { + let data = RecordBatch::try_new( + Arc::new(schema.as_ref().try_into_arrow()?), + vec![Arc::new(Int32Array::from(values))], + )?; + + let write_context = Arc::new(txn.unpartitioned_write_context().unwrap()); + let file_meta = engine + .write_parquet(&ArrowEngineData::new(data), write_context.as_ref()) + .await?; + txn.add_files(file_meta); + Ok(()) +} + +#[tokio::test] +async fn test_ict_commit_e2e() -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + // create a simple table: one int column named 'number' with ICT enabled + let schema = get_simple_int_schema(); + + let tmp_dir = TempDir::new()?; + let tmp_test_dir_url = Url::from_file_path(&tmp_dir).unwrap(); + + let (store, engine, table_location) = + engine_store_setup("test_ict_first_commit", Some(&tmp_test_dir_url)); + + // Create table with ICT enabled (writer version 7) + let table_url = test_utils::create_table( + store.clone(), + table_location, + schema.clone(), + &[], // no partition columns + true, // use protocol 3.7 + vec![], // no reader features + vec!["inCommitTimestamp"], // Enable ICT! Note: table feature is also set. + ) + .await?; + + // FIRST COMMIT: This exercises version() == 0 branch and generates ICT + { + let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; + assert_eq!( + snapshot.version(), + 0, + "Initial snapshot should be version 0" + ); + } + + let mut txn = + load_and_begin_transaction(table_url.clone(), &engine)?.with_engine_info("ict test"); + + // Add some data + generate_and_add_data_file(&mut txn, &engine, schema.clone(), vec![1, 2, 3]).await?; + + // First commit + let commit_result = txn.commit(&engine)?; + match commit_result { + CommitResult::CommittedTransaction(committed) => { + assert_eq!( + committed.commit_version(), + 1, + "First commit should result in version 1" + ); + } + CommitResult::ConflictedTransaction(conflicted) => { + panic!( + "First commit should not conflict, got conflict at version {}", + conflicted.conflict_version() + ); + } + CommitResult::RetryableTransaction(_) => { + panic!("First commit should not be retryable error"); + } + } + + // VERIFY: Check that the commit log contains inCommitTimestamp + let first_ict = get_ict_at_version(store.clone(), &table_url, 1).await?; + + assert!( + first_ict > 1612345678, + "First commit ICT ({first_ict}) should be greater than enablement timestamp (1612345678)" + ); + + // Second commit + { + let snapshot2 = Snapshot::builder_for(table_url.clone()).build(&engine)?; + assert_eq!( + snapshot2.version(), + 1, + "Second snapshot should be version 1" + ); + } + + let mut txn2 = + load_and_begin_transaction(table_url.clone(), &engine)?.with_engine_info("ict test 2"); + + // Add more data + generate_and_add_data_file(&mut txn2, &engine, schema, vec![4, 5, 6]).await?; + + // Second commit + let commit_result2 = txn2.commit(&engine)?; + match commit_result2 { + CommitResult::CommittedTransaction(committed) => { + assert_eq!( + committed.commit_version(), + 2, + "Second commit should result in version 2" + ); + } + CommitResult::ConflictedTransaction(conflicted) => { + panic!( + "Second commit should not conflict, got conflict at version {}", + conflicted.conflict_version() + ); + } + CommitResult::RetryableTransaction(_) => { + panic!("Second commit should not be retryable error"); + } + } + + // VERIFY: Check that second commit has proper monotonic ICT + let second_ict = get_ict_at_version(store, &table_url, 2).await?; + + // Verify monotonic property: second_ict > first_ict + assert!( + second_ict > first_ict, + "Second ICT ({second_ict}) should be greater than first ICT ({first_ict})" + ); + + Ok(()) +} diff --git a/kernel/tests/integration/write/interval.rs b/kernel/tests/integration/write/interval.rs new file mode 100644 index 0000000000..0cc79c6c6e --- /dev/null +++ b/kernel/tests/integration/write/interval.rs @@ -0,0 +1,253 @@ +//! Integration tests for writing ANSI interval columns. + +use std::sync::Arc; + +use delta_kernel::schema::{DataType, StructField, StructType}; +use test_utils::load_and_begin_transaction; + +mod supported { + use std::collections::HashMap; + + use delta_kernel::actions::{MAX_VALUES, MIN_VALUES, NULL_COUNT, STATS_PARSED}; + use delta_kernel::arrow::array::{ + Array, ArrayRef, Int32Array, Int64Array, StringArray, StructArray, + }; + use delta_kernel::arrow::datatypes::{DataType as ArrowDataType, Schema as ArrowSchema}; + use delta_kernel::arrow::record_batch::RecordBatch; + use delta_kernel::committer::FileSystemCommitter; + use delta_kernel::engine::arrow_conversion::TryIntoArrow as _; + use delta_kernel::engine::arrow_data::ArrowEngineData; + use delta_kernel::transaction::create_table::create_table as create_table_transaction; + use rstest::rstest; + use test_utils::{ + get_column, read_actions_from_commit, test_read, test_table_setup, test_table_setup_mt, + write_batch_to_table, + }; + + use super::*; + use crate::common::read_utils::read_parquet_file; + use crate::common::write_utils::load_existing_single_file_checkpoint_path; + + /// Unpartitioned blind-append round-trip for both interval families, covering null / zero / + /// negative values in a nested schema. Also verifies that ordinary columns retain their full + /// statistics while the interval column gets only `nullCount`. + #[rstest] + #[case::year_month( + DataType::INTERVAL_YEAR_MONTH, + Arc::new(Int32Array::from(vec![Some(0), Some(30), None, Some(-18)])) as ArrayRef, + )] + #[case::day_time( + DataType::INTERVAL_DAY_TIME, + Arc::new(Int64Array::from(vec![Some(0), Some(131_445_000_000), None, Some(-5)])) as ArrayRef, + )] + #[tokio::test(flavor = "multi_thread")] + async fn test_append_interval_roundtrip( + #[case] interval: DataType, + #[case] column: ArrayRef, + ) -> Result<(), Box> { + let schema = Arc::new(StructType::try_new(vec![ + StructField::not_null("id", DataType::LONG), + StructField::nullable( + "nested", + StructType::new_unchecked([ + StructField::nullable("iv", interval), + StructField::nullable("label", DataType::STRING), + ]), + ), + ])?); + + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let snapshot = create_table_transaction(&table_path, schema.clone(), "Test/1.0") + .with_table_properties([("delta.checkpoint.writeStatsAsStruct", "true")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + let table_url = snapshot.table_root().clone(); + + let mut txn = load_and_begin_transaction(table_url.clone(), engine.as_ref())? + .with_engine_info("default engine"); + let arrow_schema: ArrowSchema = schema.as_ref().try_into_arrow()?; + let arrow_schema = Arc::new(arrow_schema); + let nested_fields = match arrow_schema.field_with_name("nested").unwrap().data_type() { + ArrowDataType::Struct(fields) => fields.clone(), + data_type => panic!("expected nested struct, got {data_type:?}"), + }; + let nested = StructArray::new( + nested_fields, + vec![ + column, + Arc::new(StringArray::from(vec!["a", "b", "c", "d"])) as ArrayRef, + ], + None, + ); + let data = RecordBatch::try_new( + arrow_schema, + vec![ + Arc::new(Int64Array::from(vec![1, 2, 3, 4])) as ArrayRef, + Arc::new(nested) as ArrayRef, + ], + )?; + + let write_context = Arc::new(txn.unpartitioned_write_context().unwrap()); + let add_files_metadata = engine + .write_parquet(&ArrowEngineData::new(data.clone()), write_context.as_ref()) + .await?; + txn.add_files(add_files_metadata); + let snapshot = txn.commit(engine.as_ref())?.unwrap_post_commit_snapshot(); + + let add_actions = read_actions_from_commit(&table_url, 1, "add")?; + let add = &add_actions[0]; + let stats = add + .get("stats") + .and_then(|s| s.as_str()) + .expect("add action must carry stats"); + let stats: serde_json::Value = serde_json::from_str(stats)?; + assert_eq!( + stats.pointer("/nullCount/id"), + Some(&serde_json::json!(0)), + "id must have nullCount; got: {stats}", + ); + assert_eq!( + stats.pointer("/nullCount/nested/iv"), + Some(&serde_json::json!(1)), + "interval column must have nullCount; got: {stats}", + ); + assert_eq!( + stats.pointer("/nullCount/nested/label"), + Some(&serde_json::json!(0)), + "label must have nullCount; got: {stats}", + ); + for category in ["minValues", "maxValues"] { + let nested_stats = stats + .get(category) + .and_then(|v| v.get("nested")) + .and_then(|v| v.as_object()) + .expect("ordinary nested columns must have min/max stats"); + assert!( + !nested_stats.contains_key("iv"), + "interval column must not have {category} stats; got: {stats}", + ); + assert!( + nested_stats.contains_key("label"), + "label must have {category} stats; got: {stats}", + ); + } + assert_eq!(stats.pointer("/minValues/id"), Some(&serde_json::json!(1))); + assert_eq!(stats.pointer("/maxValues/id"), Some(&serde_json::json!(4))); + assert_eq!( + stats.pointer("/minValues/nested/label"), + Some(&serde_json::json!("a")) + ); + assert_eq!( + stats.pointer("/maxValues/nested/label"), + Some(&serde_json::json!("d")) + ); + + snapshot.checkpoint(engine.as_ref(), None)?; + let checkpoint_path = + load_existing_single_file_checkpoint_path(&table_path, snapshot.version()); + let checkpoint_batch = read_parquet_file(&checkpoint_path); + let checkpoint_add = get_column!(checkpoint_batch, "add", StructArray); + let add_row = (0..checkpoint_batch.num_rows()) + .find(|&row| checkpoint_add.is_valid(row)) + .expect("checkpoint must contain an add action"); + let stats_parsed = get_column!(checkpoint_add, STATS_PARSED, StructArray); + assert!(stats_parsed.is_valid(add_row)); + + let null_count = get_column!(stats_parsed, NULL_COUNT, StructArray); + let nested_null_count = get_column!(null_count, "nested", StructArray); + assert_eq!(get_column!(null_count, "id", Int64Array).value(add_row), 0); + assert_eq!( + get_column!(nested_null_count, "iv", Int64Array).value(add_row), + 1 + ); + assert_eq!( + get_column!(nested_null_count, "label", Int64Array).value(add_row), + 0 + ); + + for (category, expected_id, expected_label) in [(MIN_VALUES, 1, "a"), (MAX_VALUES, 4, "d")] + { + let values = get_column!(stats_parsed, category, StructArray); + let nested_values = get_column!(values, "nested", StructArray); + assert_eq!( + get_column!(values, "id", Int64Array).value(add_row), + expected_id + ); + assert!(nested_values.column_by_name("iv").is_none()); + assert_eq!( + get_column!(nested_values, "label", StringArray).value(add_row), + expected_label + ); + } + + test_read(&ArrowEngineData::new(data), &table_url, engine)?; + Ok(()) + } + + /// Interval columns omitted by the configured stats selection produce no statistics. + #[rstest] + #[case::stats_columns("delta.dataSkippingStatsColumns", "value")] + #[case::num_indexed_cols("delta.dataSkippingNumIndexedCols", "1")] + #[tokio::test] + async fn test_interval_stats_excluded_by_column_selection( + #[values(DataType::INTERVAL_YEAR_MONTH, DataType::INTERVAL_DAY_TIME)] interval: DataType, + #[case] property_name: &str, + #[case] property_value: &str, + ) -> Result<(), Box> { + let schema = Arc::new(StructType::try_new(vec![ + StructField::not_null("value", DataType::LONG), + StructField::nullable("iv", interval.clone()), + ])?); + let interval_column: ArrayRef = if interval == DataType::INTERVAL_YEAR_MONTH { + Arc::new(Int32Array::from(vec![Some(12), None, Some(-6)])) + } else { + Arc::new(Int64Array::from(vec![Some(86_400_000_000), None, Some(-1)])) + }; + + let (_temp_dir, table_path, engine) = test_table_setup()?; + let snapshot = create_table_transaction(&table_path, schema.clone(), "Test/1.0") + .with_table_properties([(property_name, property_value)]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + + let arrow_schema: ArrowSchema = schema.as_ref().try_into_arrow()?; + let batch = RecordBatch::try_new( + Arc::new(arrow_schema), + vec![ + Arc::new(Int64Array::from(vec![1, 2, 3])) as ArrayRef, + interval_column, + ], + )?; + let _ = write_batch_to_table(&snapshot, engine.as_ref(), batch, HashMap::new()).await?; + + let add_actions = read_actions_from_commit(snapshot.table_root(), 1, "add")?; + let stats: serde_json::Value = serde_json::from_str( + add_actions[0] + .get("stats") + .and_then(|value| value.as_str()) + .expect("add action should have stats"), + )?; + + assert_eq!( + stats.pointer("/nullCount/value"), + Some(&serde_json::json!(0)) + ); + assert_eq!( + stats.pointer("/minValues/value"), + Some(&serde_json::json!(1)) + ); + assert_eq!( + stats.pointer("/maxValues/value"), + Some(&serde_json::json!(3)) + ); + for category in ["nullCount", "minValues", "maxValues"] { + assert!( + stats.get(category).and_then(|value| value.get("iv")).is_none(), + "interval column excluded by {property_name} must not have {category}; got: {stats}", + ); + } + Ok(()) + } +} diff --git a/kernel/tests/integration/write/mod.rs b/kernel/tests/integration/write/mod.rs new file mode 100644 index 0000000000..ede123f6a3 --- /dev/null +++ b/kernel/tests/integration/write/mod.rs @@ -0,0 +1,20 @@ +//! Write-path integration tests. + +mod append; +mod cdf; +mod clustered; +mod column_defaults; +mod column_mapping; +mod commit_info; +mod domain_metadata; +mod ict; +mod interval; +mod nested_field_ids; +mod partitioned; +mod post_commit; +mod relative_paths; +mod remove_dv; +mod row_tracking; +mod stats; +mod txn; +mod types; diff --git a/kernel/tests/integration/write/nested_field_ids.rs b/kernel/tests/integration/write/nested_field_ids.rs new file mode 100644 index 0000000000..1685130237 --- /dev/null +++ b/kernel/tests/integration/write/nested_field_ids.rs @@ -0,0 +1,127 @@ +//! End-to-end nested parquet field id propagation tests for `DefaultParquetHandler`. +use std::collections::HashMap; +use std::sync::Arc; + +use delta_kernel::arrow::array::RecordBatch; +use delta_kernel::arrow::datatypes::Schema as ArrowSchema; +use delta_kernel::engine::arrow_conversion::TryIntoArrow as _; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::object_store::local::LocalFileSystem; +use delta_kernel::schema::{ + ArrayType, ColumnMetadataKey, DataType, MapType, MetadataValue, StructField, StructType, +}; +use delta_kernel::{EngineData, ParquetHandler}; +use test_utils::delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor; +use test_utils::delta_kernel_default_engine::parquet::DefaultParquetHandler; +use test_utils::nested_ids_json; +use url::Url; + +use crate::common::write_utils::collect_all_parquet_field_ids; + +#[tokio::test] +async fn test_nested_field_ids_round_trip() { + let nested_ids_meta_key = ColumnMetadataKey::ColumnMappingNestedIds.as_ref(); + let kernel_schema = build_kernel_schema(nested_ids_meta_key); + let arrow_schema: ArrowSchema = (&kernel_schema).try_into_arrow().unwrap(); + + // Empty record batch is enough; the parquet writer still emits the schema (with field ids) + // in the file footer regardless of row count. + let record_batch = RecordBatch::new_empty(Arc::new(arrow_schema)); + let local_path = write_via_default_engine(record_batch); + let id_by_path = collect_all_parquet_field_ids(&local_path); + + let expected: HashMap = [ + // array_in_map: map> + ("array_in_map", 1), + ("array_in_map.key_value.key", 100), + ("array_in_map.key_value.value", 101), + ("array_in_map.key_value.value.list.element", 102), + // map_in_list: list> + ("map_in_list", 2), + ("map_in_list.list.element", 200), + ("map_in_list.list.element.key_value.key", 201), + ("map_in_list.list.element.key_value.value", 202), + ] + .into_iter() + .map(|(k, v)| (k.to_string(), v)) + .collect(); + assert_eq!( + id_by_path, expected, + "parquet field ids by path don't match expected; got {id_by_path:?}", + ); +} + +/// Build a kernel schema with two doubly nested top-level fields. +/// +/// Layout: +/// - `array_in_map: map>` +/// - top-level parquet.field.id = 1 +/// - nested ids: `array_in_map.key` -> 100, `.value` -> 101, `.value.element` -> 102 +/// - `map_in_list: list>` +/// - top-level parquet.field.id = 2 +/// - nested ids: `map_in_list.element` -> 200, `.element.key` -> 201, `.element.value` -> 202 +fn build_kernel_schema(nested_ids_meta_key: &str) -> StructType { + let with_field_ids = |name: &str, dtype, top_id: i64, nested: &[(&str, i64)]| { + StructField::nullable(name, dtype).with_metadata([ + ( + ColumnMetadataKey::ParquetFieldId.as_ref(), + MetadataValue::from(top_id), + ), + ( + nested_ids_meta_key, + MetadataValue::Other(nested_ids_json(nested)), + ), + ]) + }; + + let array_in_map = with_field_ids( + "array_in_map", + DataType::from(MapType::new( + DataType::INTEGER, + ArrayType::new(DataType::INTEGER, true), + true, + )), + 1, + &[ + ("array_in_map.key", 100), + ("array_in_map.value", 101), + ("array_in_map.value.element", 102), + ], + ); + + let map_in_list = with_field_ids( + "map_in_list", + DataType::from(ArrayType::new( + MapType::new(DataType::INTEGER, DataType::INTEGER, true), + true, + )), + 2, + &[ + ("map_in_list.element", 200), + ("map_in_list.element.key", 201), + ("map_in_list.element.value", 202), + ], + ); + + StructType::try_new(vec![array_in_map, map_in_list]).unwrap() +} + +/// Write `record_batch` via the `ParquetHandler::write_parquet_file` trait method to a +/// temp directory and return the resulting on-disk path. +fn write_via_default_engine(record_batch: RecordBatch) -> std::path::PathBuf { + let temp_dir = tempfile::tempdir().unwrap(); + // Keep the tempdir so the file outlives this function; the OS cleans it up on test exit. + let dir_path = temp_dir.keep(); + let file_path = dir_path.join("data.parquet"); + let location = Url::from_file_path(&file_path).unwrap(); + + let store = Arc::new(LocalFileSystem::new()); + let handler = DefaultParquetHandler::new(store, Arc::new(TokioBackgroundExecutor::new())); + let data: delta_kernel::DeltaResultIteratorStatic> = + Box::new(std::iter::once(Ok( + Box::new(ArrowEngineData::new(record_batch)) as Box, + ))); + + ParquetHandler::write_parquet_file(&handler, location, data).unwrap(); + file_path +} diff --git a/kernel/tests/integration/write/partitioned.rs b/kernel/tests/integration/write/partitioned.rs new file mode 100644 index 0000000000..7f6d002b41 --- /dev/null +++ b/kernel/tests/integration/write/partitioned.rs @@ -0,0 +1,1439 @@ +//! Integration tests for writing to partitioned Delta tables. + +use std::collections::HashMap; +use std::sync::Arc; + +use chrono::{NaiveDate, NaiveDateTime, TimeZone, Utc}; +use delta_kernel::actions::{MAX_VALUES, MIN_VALUES, NULL_COUNT}; +use delta_kernel::arrow::array::{ + Array, ArrayRef, BinaryArray, BooleanArray, Date32Array, Decimal128Array, Float32Array, + Float64Array, Int16Array, Int32Array, Int64Array, Int8Array, RecordBatch, StringArray, + TimestampMicrosecondArray, +}; +use delta_kernel::arrow::datatypes::Schema as ArrowSchema; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::engine::arrow_conversion::TryIntoArrow as _; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::expressions::Scalar; +use delta_kernel::schema::{DataType, StructField, StructType}; +use delta_kernel::table_features::ColumnMappingMode; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::transaction::data_layout::DataLayout; +use delta_kernel::Snapshot; +use rstest::rstest; +use test_utils::{ + begin_transaction, get_column, read_scan, test_table_setup_mt, write_batch_to_table, +}; +use url::Url; + +use crate::common::read_utils::read_parquet_file; + +// ============================================================================== +// Tests +// ============================================================================== + +/// Writes one row with a normal everyday value for every partition column type, reads it +/// back, checkpoints, reloads from checkpoint, and verifies the values survive both paths. +#[rstest] +#[case::cm_none(ColumnMappingMode::None)] +#[case::cm_name(ColumnMappingMode::Name)] +#[case::cm_id(ColumnMappingMode::Id)] +#[tokio::test(flavor = "multi_thread")] +async fn test_write_partitioned_normal_values_roundtrip( + #[case] cm_mode: ColumnMappingMode, + #[values(true, false)] write_partition_values_parsed: bool, +) -> Result<(), Box> { + // ===== Step 1: Create table and write one row with normal partition values. ===== + let (_tmp_dir, table_path, snapshot, engine) = setup_and_write( + all_types_schema(), + PARTITION_COLS, + cm_mode, + write_partition_values_parsed, + normal_arrow_columns(), + normal_partition_values()?, + ) + .await?; + assert_eq!( + snapshot + .table_configuration() + .logical_partition_columns() + .len(), + 13 + ); + + // ===== Step 2: Validate add.path structure in the commit log JSON. ===== + let (add, rel_path) = read_single_add(&table_path, 1)?; + match cm_mode { + ColumnMappingMode::None => { + // Every value except the timestamps is unreserved ASCII and passes through + // both encoding layers unchanged. The two timestamps differ on `:`: + // `:` -> Hive `%3A` -> URI `%253A` (all platforms) + // TIMESTAMP_NTZ additionally contains a space, which diverges by platform: + // ` ` on non-Windows: Hive passes it through -> URI `%20` + // ` ` on Windows: Hive escapes to `%20` -> URI `%2520` + // TIMESTAMP uses ISO-Z format (no space), so it's platform-identical. + let ntz_segment = if cfg!(target_os = "windows") { + "p_timestamp_ntz=2025-03-31%252015%253A30%253A00.123456/" + } else { + "p_timestamp_ntz=2025-03-31%2015%253A30%253A00.123456/" + }; + let expected_prefix = format!( + "p_string=hello/p_int=42/p_long=9876543210/p_short=7/\ + p_byte=3/p_float=1.25/p_double=99.99/p_boolean=true/p_date=2025-03-31/\ + p_timestamp=2025-03-31T15%253A30%253A00.123456Z/p_decimal=123.45/\ + p_binary=Hello/{ntz_segment}" + ); + assert!( + rel_path.starts_with(&expected_prefix), + "CM off: relative path mismatch.\n \ + expected: {expected_prefix}.parquet\n got: {rel_path}" + ); + assert!(rel_path.ends_with(".parquet")); + } + ColumnMappingMode::Name | ColumnMappingMode::Id => { + assert_cm_path(&rel_path); + } + } + + // ===== Step 3: Validate add.partitionValues content. ===== + let pv = add["partitionValues"].as_object().unwrap(); + match cm_mode { + ColumnMappingMode::None => { + // Keys are logical column names, values are serialized strings. + for (key, val) in EXPECTED_NORMAL_PVS { + assert_eq!( + pv.get(*key).and_then(|v| v.as_str()), + Some(*val), + "partitionValues[{key}] mismatch" + ); + } + } + ColumnMappingMode::Name | ColumnMappingMode::Id => { + // Keys are physical names. Look up via schema and verify each value. + let logical_schema = snapshot.schema(); + for (logical_key, expected_val) in EXPECTED_NORMAL_PVS { + let field = logical_schema.field(logical_key).unwrap(); + let physical_key = field.physical_name(cm_mode); + assert_eq!( + pv.get(physical_key).and_then(|v| v.as_str()), + Some(*expected_val), + "partitionValues[{physical_key}] (logical: {logical_key}) mismatch" + ); + } + } + } + + // ===== Step 4: Scan and verify values survive checkpoint + reload. ===== + // This is the only step affected by `write_partition_values_parsed`: the + // post-checkpoint scan reads back through `partitionValues_parsed`. + verify_and_checkpoint(&snapshot, engine, assert_normal_values)?; + + Ok(()) +} + +/// Writes an interval partition column, asserts the partitionValues entry is the Spark ANSI +/// interval literal (CM=None), and verifies the value round-trips through a scan. +#[rstest] +#[case::year_month( + DataType::INTERVAL_YEAR_MONTH, + Scalar::IntervalYearMonth(30), + "INTERVAL '2-6' YEAR TO MONTH" +)] +#[case::day_time( + DataType::INTERVAL_DAY_TIME, + Scalar::IntervalDayTime(131_445_000_000), + "INTERVAL '1 12:30:45' DAY TO SECOND" +)] +#[tokio::test(flavor = "multi_thread")] +async fn test_write_partitioned_interval_roundtrip( + #[case] interval: DataType, + #[case] partition_value: Scalar, + #[case] expected_partition_value: &str, + #[values( + ColumnMappingMode::None, + ColumnMappingMode::Name, + ColumnMappingMode::Id + )] + cm_mode: ColumnMappingMode, +) -> Result<(), Box> { + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("value", DataType::INTEGER), + StructField::nullable("period", interval), + ])?); + let (_tmp_dir, table_path, engine) = test_table_setup_mt()?; + let snapshot = + create_interval_partitioned_table(&table_path, engine.as_ref(), schema, cm_mode, false)?; + let data_schema = StructType::try_new([StructField::nullable("value", DataType::INTEGER)])?; + let batch = RecordBatch::try_new( + Arc::new((&data_schema).try_into_arrow()?), + vec![Arc::new(Int32Array::from(vec![7]))], + )?; + let snapshot = write_batch_to_table( + &snapshot, + engine.as_ref(), + batch, + HashMap::from([("period".to_string(), partition_value.clone())]), + ) + .await?; + + // CM=None: the raw partitionValues entry is the Spark ANSI interval literal. + if cm_mode == ColumnMappingMode::None { + let (add, _rel) = read_single_add(&table_path, 1)?; + assert_eq!( + add["partitionValues"]["period"].as_str(), + Some(expected_partition_value), + "interval partition value should serialize to the ANSI literal" + ); + } + + // The partition value round-trips: scan materializes `period` back as its physical integer. + let sorted = read_sorted(&snapshot, engine as Arc)?; + assert_eq!( + get_column!(&sorted, "value", Int32Array).value(0), + 7, + "value column" + ); + assert_interval_value(&sorted, "period", &partition_value); + + Ok(()) +} + +/// Materialized interval partition columns are written into the Parquet file as physical integer +/// values and still round-trip through scan output as logical partition columns. +#[rstest] +#[case::year_month( + DataType::INTERVAL_YEAR_MONTH, + Scalar::IntervalYearMonth(30), + "INTERVAL '2-6' YEAR TO MONTH" +)] +#[case::day_time( + DataType::INTERVAL_DAY_TIME, + Scalar::IntervalDayTime(131_445_000_000), + "INTERVAL '1 12:30:45' DAY TO SECOND" +)] +#[tokio::test(flavor = "multi_thread")] +async fn test_materialized_partitioned_interval_roundtrip( + #[case] interval: DataType, + #[case] partition_value: Scalar, + #[case] expected_partition_value: &str, + #[values( + ColumnMappingMode::None, + ColumnMappingMode::Name, + ColumnMappingMode::Id + )] + cm_mode: ColumnMappingMode, +) -> Result<(), Box> { + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("value", DataType::INTEGER), + StructField::nullable("period", interval), + ])?); + + let (_tmp_dir, table_path, engine) = test_table_setup_mt()?; + let snapshot = + create_interval_partitioned_table(&table_path, engine.as_ref(), schema, cm_mode, true)?; + + let data_schema = StructType::try_new(vec![StructField::nullable("value", DataType::INTEGER)])?; + let batch = RecordBatch::try_new( + Arc::new((&data_schema).try_into_arrow()?), + vec![Arc::new(Int32Array::from(vec![7]))], + )?; + let snapshot = write_batch_to_table( + &snapshot, + engine.as_ref(), + batch, + HashMap::from([("period".to_string(), partition_value.clone())]), + ) + .await?; + + let logical_schema = snapshot.schema(); + let value_physical = logical_schema + .field("value") + .unwrap() + .physical_name(cm_mode); + let period_physical = logical_schema + .field("period") + .unwrap() + .physical_name(cm_mode); + let (add, rel_path) = read_single_add(&table_path, 1)?; + assert_eq!( + add["partitionValues"][period_physical].as_str(), + Some(expected_partition_value), + "interval partition value should serialize to the ANSI literal" + ); + + let parquet_path = Url::from_directory_path(&table_path) + .unwrap() + .join(&rel_path)? + .to_file_path() + .unwrap(); + let file_batch = read_parquet_file(&parquet_path); + assert_eq!( + get_column!(file_batch, value_physical, Int32Array).value(0), + 7 + ); + assert_interval_value(&file_batch, period_physical, &partition_value); + + let stats: serde_json::Value = serde_json::from_str(add["stats"].as_str().unwrap()).unwrap(); + assert!( + stats[MIN_VALUES].get(value_physical).is_some(), + "data column 'value' should have minValues" + ); + assert!( + stats[MIN_VALUES].get(period_physical).is_none(), + "materialized partition column should not have minValues" + ); + assert!( + stats[MAX_VALUES].get(period_physical).is_none(), + "materialized partition column should not have maxValues" + ); + assert!( + stats[NULL_COUNT].get(period_physical).is_none(), + "materialized partition column should not have nullCount" + ); + + let sorted = read_sorted(&snapshot, engine.clone() as Arc)?; + assert_eq!(get_column!(sorted, "value", Int32Array).value(0), 7); + assert_interval_value(&sorted, "period", &partition_value); + + Ok(()) +} + +/// Writes one row with all null partition values, reads it back, checkpoints, reloads +/// from checkpoint, and verifies every partition column is null in both reads. +#[rstest] +#[case::cm_none(ColumnMappingMode::None)] +#[case::cm_name(ColumnMappingMode::Name)] +#[case::cm_id(ColumnMappingMode::Id)] +#[tokio::test(flavor = "multi_thread")] +async fn test_write_partitioned_null_values_roundtrip( + #[case] cm_mode: ColumnMappingMode, + #[values(true, false)] write_partition_values_parsed: bool, +) -> Result<(), Box> { + // ===== Step 1: Create table and write one row with all-null partition values. ===== + let (_tmp_dir, table_path, snapshot, engine) = setup_and_write( + all_types_schema(), + PARTITION_COLS, + cm_mode, + write_partition_values_parsed, + null_arrow_columns(), + null_partition_values()?, + ) + .await?; + + // ===== Step 2: Validate add.path structure in the commit log JSON. ===== + let (add, rel_path) = read_single_add(&table_path, 1)?; + match cm_mode { + ColumnMappingMode::None => { + // Every partition column should use HIVE_DEFAULT_PARTITION in the path. + let expected_prefix = hive_prefix(PARTITION_COLS, "__HIVE_DEFAULT_PARTITION__"); + assert!( + rel_path.starts_with(&expected_prefix), + "CM off null: relative path mismatch.\n \ + expected: {expected_prefix}.parquet\n got: {rel_path}" + ); + } + ColumnMappingMode::Name | ColumnMappingMode::Id => { + assert_cm_path(&rel_path); + } + } + + // ===== Step 3: Validate add.partitionValues content (all values should be JSON null). ===== + let pv = add["partitionValues"].as_object().unwrap(); + assert_eq!(pv.len(), PARTITION_COLS.len()); + for val in pv.values() { + assert!( + val.is_null(), + "all partition values should be null, got: {val}" + ); + } + + // ===== Step 4: Scan and verify all-null values survive checkpoint + reload. ===== + // This is the only step affected by `write_partition_values_parsed`: the + // post-checkpoint scan reads back through `partitionValues_parsed`. + verify_and_checkpoint(&snapshot, engine, assert_all_partition_columns_null)?; + + Ok(()) +} + +// On Windows the Hive escape set additionally includes space, `<`, `>`, `|`. They get +// Hive-escaped to `%XX` first, and the URI layer double-encodes the leading `%`. Non-Windows +// passes them through Hive, so the URI layer applies only single encoding. +macro_rules! platform_path { + ($win:literal, $non_win:literal) => { + if cfg!(target_os = "windows") { + $win + } else { + $non_win + } + }; +} + +/// Asserts `add.path` correctly encodes every non-control character in +/// `HADOOP_URI_PATH_ENCODE_SET`, a sample of ASCII controls, plus the Hive-only chars that +/// exercise the Hive -> URI double-encoding path. Partition value is always a three-char +/// string `ab`, so `add.path` must start with `p=ab/` (the encoded partition +/// directory plus its trailing separator) and end in `.parquet`. The `partitionValues` map +/// should carry the raw value unchanged. +#[rstest] +// === Chars in both Hive and URI sets: Hive `%XX` -> URI `%25XX` (double-encoded) === +#[case::percent("a%b", "p=a%2525b/")] +#[case::quote("a\"b", "p=a%2522b/")] +#[case::hash("a#b", "p=a%2523b/")] +#[case::question("a?b", "p=a%253Fb/")] +#[case::backslash("a\\b", "p=a%255Cb/")] +#[case::caret("a^b", "p=a%255Eb/")] +#[case::left_brace("a{b", "p=a%257Bb/")] +#[case::left_bracket("a[b", "p=a%255Bb/")] +#[case::right_bracket("a]b", "p=a%255Db/")] +// === Hive set only (URI just double-encodes the `%` Hive produced) === +#[case::colon("a:b", "p=a%253Ab/")] +#[case::slash("a/b", "p=a%252Fb/")] +#[case::equals("a=b", "p=a%253Db/")] +#[case::apostrophe("a'b", "p=a%2527b/")] +#[case::asterisk("a*b", "p=a%252Ab/")] +#[case::slash_percent("Serbia/srb%", "p=Serbia%252Fsrb%2525/")] +// === URI set only (Hive passthrough; URI single-encodes) === +#[case::backtick("a`b", "p=a%60b/")] +#[case::right_brace("a}b", "p=a%7Db/")] +// === Platform-divergent: Hive set on Windows only === +#[case::space("a b", platform_path!("p=a%2520b/", "p=a%20b/"))] +#[case::less_than("ab", platform_path!("p=a%253Eb/", "p=a%3Eb/"))] +#[case::pipe("a|b", platform_path!("p=a%257Cb/", "p=a%7Cb/"))] +#[case::multi_space("a b", platform_path!("p=a%2520%2520%2520b/", "p=a%20%20%20b/"))] +// === A few ASCII control characters (sample: NUL, TAB, LF, DEL) === +#[case::null_byte("a\0b", "p=a%2500b/")] +#[case::tab("a\tb", "p=a%2509b/")] +#[case::newline("a\nb", "p=a%250Ab/")] +#[case::del("a\x7Fb", "p=a%257Fb/")] +#[tokio::test(flavor = "multi_thread")] +async fn test_write_partitioned_path_encodes_special_chars( + #[case] value: &str, + #[case] expected_path_prefix: &str, + #[values( + ColumnMappingMode::None, + ColumnMappingMode::Name, + ColumnMappingMode::Id + )] + cm_mode: ColumnMappingMode, +) -> Result<(), Box> { + // ===== Step 1: Create a single-STRING-partition table and write one row. ===== + let schema = Arc::new( + StructType::try_new(vec![ + StructField::nullable("value", DataType::INTEGER), + StructField::nullable("p", DataType::STRING), + ]) + .unwrap(), + ); + let (_tmp_dir, table_path, snapshot, engine) = setup_and_write( + schema, + &["p"], + cm_mode, + true, // write_partition_values_parsed + vec![ + Arc::new(Int32Array::from(vec![1])), + Arc::new(StringArray::from(vec![value])) as ArrayRef, + ], + HashMap::from([("p".to_string(), Scalar::String(value.into()))]), + ) + .await?; + + // ===== Step 2: Read the single add action from the commit log JSON. ===== + let (add, rel_path) = read_single_add(&table_path, 1)?; + + // ===== Step 3: Validate add.partitionValues carries the raw value unchanged. ===== + let logical_schema = snapshot.schema(); + let physical_key = logical_schema.field("p").unwrap().physical_name(cm_mode); + let pv = add["partitionValues"].as_object().unwrap(); + assert_eq!( + pv.get(physical_key).and_then(|v| v.as_str()), + Some(value), + "partitionValues[{physical_key}] mismatch" + ); + + // ===== Step 4: Validate add.path has the expected URI-encoded layout. ===== + match cm_mode { + ColumnMappingMode::None => { + assert!( + rel_path.starts_with(expected_path_prefix), + "CM off: expected path to start with {expected_path_prefix:?}, got {rel_path:?}" + ); + assert!(rel_path.ends_with(".parquet")); + } + ColumnMappingMode::Name | ColumnMappingMode::Id => { + assert_cm_path(&rel_path); + } + } + + // ===== Step 5: Scan and verify the row round-trips across checkpoint + reload. ===== + verify_and_checkpoint(&snapshot, engine, |sorted| { + assert_eq!(sorted.num_rows(), 1); + assert_eq!( + sorted + .column(0) + .as_any() + .downcast_ref::() + .unwrap() + .value(0), + 1, + "value column mismatch" + ); + assert_eq!( + sorted + .column(1) + .as_any() + .downcast_ref::() + .unwrap() + .value(0), + value, + "partition column `p` mismatch" + ); + })?; + + Ok(()) +} + +// TODO: Test extreme low/high values +// e.g. i32::MIN/MAX, f64::MAX, f64::MIN_POSITIVE, f64::INFINITY, f64::NEG_INFINITY, +// f64::NAN, Date(-719_162) (year 0001), Date(2_932_896) (year 9999) + +// TODO(#2423): Test non-ASCII UTF-8 strings once `add.path` encoding matches Delta-Spark. +// Kernel currently percent-encodes non-ASCII UTF-8 bytes (e.g. München -> +// M%C3%BCnchen) while Delta-Spark leaves them raw in `add.path`. +// e.g. "M\u{00FC}nchen", "日本語", "🎵🎶" + +// TODO: Test binary valid UTF-8 +// e.g. Scalar::Binary(b"HELLO".to_vec()), Scalar::Binary(vec![0x2F, 0x3D, 0x25]) + +// TODO: Test binary with non-UTF-8 +// e.g. Scalar::Binary(vec![0xDE, 0xAD, 0xBE, 0xEF]), Scalar::Binary(vec![0x00, 0xFF]) + +// ============================================================================== +// Schema and partition column definitions +// ============================================================================== + +fn all_types_schema() -> Arc { + Arc::new( + StructType::try_new(vec![ + StructField::nullable("value", DataType::INTEGER), + StructField::nullable("p_string", DataType::STRING), + StructField::nullable("p_int", DataType::INTEGER), + StructField::nullable("p_long", DataType::LONG), + StructField::nullable("p_short", DataType::SHORT), + StructField::nullable("p_byte", DataType::BYTE), + StructField::nullable("p_float", DataType::FLOAT), + StructField::nullable("p_double", DataType::DOUBLE), + StructField::nullable("p_boolean", DataType::BOOLEAN), + StructField::nullable("p_date", DataType::DATE), + StructField::nullable("p_timestamp", DataType::TIMESTAMP), + StructField::nullable("p_decimal", DataType::decimal(10, 2).unwrap()), + StructField::nullable("p_binary", DataType::BINARY), + StructField::nullable("p_timestamp_ntz", DataType::TIMESTAMP_NTZ), + ]) + .unwrap(), + ) +} + +const PARTITION_COLS: &[&str] = &[ + "p_string", + "p_int", + "p_long", + "p_short", + "p_byte", + "p_float", + "p_double", + "p_boolean", + "p_date", + "p_timestamp", + "p_decimal", + "p_binary", + "p_timestamp_ntz", +]; + +// ============================================================================== +// Normal-values test data +// ============================================================================== + +/// Arrow columns for one row with normal everyday values for all 13 partition types. +fn normal_arrow_columns() -> Vec { + let ts = ts_to_micros("2025-03-31 15:30:00.123456"); + vec![ + Arc::new(Int32Array::from(vec![1])), + Arc::new(StringArray::from(vec!["hello"])), + Arc::new(Int32Array::from(vec![42])), + Arc::new(Int64Array::from(vec![9_876_543_210i64])), + Arc::new(Int16Array::from(vec![7i16])), + Arc::new(Int8Array::from(vec![3i8])), + Arc::new(Float32Array::from(vec![1.25f32])), + Arc::new(Float64Array::from(vec![99.99f64])), + Arc::new(BooleanArray::from(vec![true])), + Arc::new(Date32Array::from(vec![date_to_days("2025-03-31")])), + ts_array(ts), + decimal_array(12345, 10, 2), + Arc::new(BinaryArray::from_vec(vec![b"Hello"])), + ts_ntz_array(ts), + ] +} + +/// Typed partition values matching `normal_arrow_columns`. +fn normal_partition_values() -> Result, Box> { + let ts = ts_to_micros("2025-03-31 15:30:00.123456"); + Ok(HashMap::from([ + ("p_string".into(), Scalar::String("hello".into())), + ("p_int".into(), Scalar::Integer(42)), + ("p_long".into(), Scalar::Long(9_876_543_210)), + ("p_short".into(), Scalar::Short(7)), + ("p_byte".into(), Scalar::Byte(3)), + ("p_float".into(), Scalar::Float(1.25)), + ("p_double".into(), Scalar::Double(99.99)), + ("p_boolean".into(), Scalar::Boolean(true)), + ("p_date".into(), Scalar::Date(date_to_days("2025-03-31"))), + ("p_timestamp".into(), Scalar::Timestamp(ts)), + ("p_decimal".into(), Scalar::decimal(12345, 10, 2)?), + ("p_binary".into(), Scalar::Binary(b"Hello".to_vec())), + ("p_timestamp_ntz".into(), Scalar::TimestampNtz(ts)), + ])) +} + +/// Expected serialized partition values (logical key -> serialized string) for normal data. +const EXPECTED_NORMAL_PVS: &[(&str, &str)] = &[ + ("p_string", "hello"), + ("p_int", "42"), + ("p_long", "9876543210"), + ("p_short", "7"), + ("p_byte", "3"), + ("p_float", "1.25"), + ("p_double", "99.99"), + ("p_boolean", "true"), + ("p_date", "2025-03-31"), + ("p_timestamp", "2025-03-31T15:30:00.123456Z"), + ("p_decimal", "123.45"), + ("p_binary", "Hello"), + ("p_timestamp_ntz", "2025-03-31 15:30:00.123456"), +]; + +// ============================================================================== +// Null-values test data +// ============================================================================== + +/// Arrow columns for one row with all null partition values. +fn null_arrow_columns() -> Vec { + vec![ + Arc::new(Int32Array::from(vec![1])), + Arc::new(StringArray::from(vec![None::<&str>])), + Arc::new(Int32Array::from(vec![None::])), + Arc::new(Int64Array::from(vec![None::])), + Arc::new(Int16Array::from(vec![None::])), + Arc::new(Int8Array::from(vec![None::])), + Arc::new(Float32Array::from(vec![None::])), + Arc::new(Float64Array::from(vec![None::])), + Arc::new(BooleanArray::from(vec![None::])), + Arc::new(Date32Array::from(vec![None::])), + Arc::new(TimestampMicrosecondArray::from(vec![None::]).with_timezone("UTC")), + Arc::new( + Decimal128Array::from(vec![None::]) + .with_precision_and_scale(10, 2) + .unwrap(), + ), + Arc::new(BinaryArray::from(vec![None::<&[u8]>])), + Arc::new(TimestampMicrosecondArray::from(vec![None::])), + ] +} + +/// Typed null partition values for all 13 partition columns. +fn null_partition_values() -> Result, Box> { + Ok(HashMap::from([ + ("p_string".into(), Scalar::Null(DataType::STRING)), + ("p_int".into(), Scalar::Null(DataType::INTEGER)), + ("p_long".into(), Scalar::Null(DataType::LONG)), + ("p_short".into(), Scalar::Null(DataType::SHORT)), + ("p_byte".into(), Scalar::Null(DataType::BYTE)), + ("p_float".into(), Scalar::Null(DataType::FLOAT)), + ("p_double".into(), Scalar::Null(DataType::DOUBLE)), + ("p_boolean".into(), Scalar::Null(DataType::BOOLEAN)), + ("p_date".into(), Scalar::Null(DataType::DATE)), + ("p_timestamp".into(), Scalar::Null(DataType::TIMESTAMP)), + ("p_decimal".into(), Scalar::Null(DataType::decimal(10, 2)?)), + ("p_binary".into(), Scalar::Null(DataType::BINARY)), + ( + "p_timestamp_ntz".into(), + Scalar::Null(DataType::TIMESTAMP_NTZ), + ), + ])) +} + +// ============================================================================== +// Assertions +// ============================================================================== + +/// Downcast column `$idx` to `$arr_ty` and assert row 0's value equals `$expected`. +macro_rules! assert_col { + ($batch:expr, $idx:expr, $arr_ty:ty, $expected:expr) => { + assert_eq!( + $batch + .column($idx) + .as_any() + .downcast_ref::<$arr_ty>() + .unwrap() + .value(0), + $expected, + "column {} ({}) value mismatch", + $idx, + $batch.schema().field($idx).name() + ); + }; +} + +fn assert_interval_value(batch: &RecordBatch, column_name: &str, expected: &Scalar) { + match expected { + Scalar::IntervalYearMonth(months) => { + assert_eq!( + get_column!(batch, column_name, Int32Array).value(0), + *months, + "interval year-month column {column_name}" + ); + } + Scalar::IntervalDayTime(micros) => { + assert_eq!( + get_column!(batch, column_name, Int64Array).value(0), + *micros, + "interval day-time column {column_name}" + ); + } + _ => panic!("expected interval scalar, got {expected:?}"), + } +} + +/// Asserts the normal-values row reads back correctly (1 row, all 13 partition columns). +fn assert_normal_values(sorted: &RecordBatch) { + let ts = ts_to_micros("2025-03-31 15:30:00.123456"); + assert_eq!(sorted.num_rows(), 1); + assert_col!(sorted, 0, Int32Array, 1); // value + assert_col!(sorted, 1, StringArray, "hello"); // p_string + assert_col!(sorted, 2, Int32Array, 42); // p_int + assert_col!(sorted, 3, Int64Array, 9_876_543_210i64); // p_long + assert_col!(sorted, 4, Int16Array, 7i16); // p_short + assert_col!(sorted, 5, Int8Array, 3i8); // p_byte + assert_col!(sorted, 6, Float32Array, 1.25f32); // p_float + assert_col!(sorted, 7, Float64Array, 99.99f64); // p_double + assert_col!(sorted, 8, BooleanArray, true); // p_boolean + assert_col!(sorted, 9, Date32Array, date_to_days("2025-03-31")); // p_date + assert_col!(sorted, 10, TimestampMicrosecondArray, ts); // p_timestamp + assert_col!(sorted, 11, Decimal128Array, 12345); // p_decimal + assert_eq!( + // p_binary (slice comparison) + sorted + .column(12) + .as_any() + .downcast_ref::() + .unwrap() + .value(0), + b"Hello" + ); + assert_col!(sorted, 13, TimestampMicrosecondArray, ts); // p_timestamp_ntz +} + +/// Asserts all partition columns (indices 1-13) are null for the single row. +fn assert_all_partition_columns_null(sorted: &RecordBatch) { + assert_eq!(sorted.num_rows(), 1); + for col_idx in 1..=13 { + assert!( + sorted.column(col_idx).is_null(0), + "partition column at index {col_idx} ({}) should be null", + sorted.schema().field(col_idx).name() + ); + } +} + +/// Asserts a CM=name/id relative path has the shape `<2char>/.parquet`. +fn assert_cm_path(rel_path: &str) { + let segments: Vec<&str> = rel_path.split('/').collect(); + assert_eq!( + segments.len(), + 2, + "CM on: path should be /, got: {rel_path}" + ); + assert_eq!(segments[0].len(), 2, "prefix should be 2 chars"); + assert!(segments[0].chars().all(|c| c.is_ascii_alphanumeric())); + assert!(segments[1].ends_with(".parquet")); +} + +// ============================================================================== +// Table setup and utility helpers +// ============================================================================== + +fn cm_mode_str(mode: ColumnMappingMode) -> &'static str { + match mode { + ColumnMappingMode::None => "none", + ColumnMappingMode::Id => "id", + ColumnMappingMode::Name => "name", + } +} + +fn create_interval_partitioned_table( + table_path: &str, + engine: &dyn delta_kernel::Engine, + schema: Arc, + cm_mode: ColumnMappingMode, + materialize_partition_columns: bool, +) -> Result, Box> { + let mut properties = vec![]; + if materialize_partition_columns { + properties.push(("delta.feature.materializePartitionColumns", "supported")); + } + if cm_mode != ColumnMappingMode::None { + properties.push(("delta.columnMapping.mode", cm_mode_str(cm_mode))); + } + let snapshot = create_table(table_path, schema, "test/1.0") + .with_data_layout(DataLayout::partitioned(["period"])) + .with_table_properties(properties) + .build(engine, Box::new(FileSystemCommitter::new()))? + .commit(engine)? + .unwrap_post_commit_snapshot(); + Ok(snapshot) +} + +fn create_partitioned_table( + table_path: &str, + engine: &dyn delta_kernel::Engine, + schema: Arc, + partition_cols: &[&str], + cm_mode: ColumnMappingMode, + write_partition_values_parsed: bool, +) -> Result, Box> { + let mut builder = create_table(table_path, schema, "test/1.0") + .with_data_layout(DataLayout::partitioned(partition_cols)) + .with_table_properties([( + // `delta.checkpoint.writeStatsAsStruct` is what turns on `partitionValues_parsed`, + // which (per its name) lives only in the checkpoint; commit JSON is unaffected. + "delta.checkpoint.writeStatsAsStruct", + write_partition_values_parsed.to_string(), + )]); + if cm_mode != ColumnMappingMode::None { + builder = + builder.with_table_properties([("delta.columnMapping.mode", cm_mode_str(cm_mode))]); + } + let _ = builder + .build(engine, Box::new(FileSystemCommitter::new()))? + .commit(engine)?; + Ok(Snapshot::builder_for(table_path).build(engine)?) +} + +fn read_sorted( + snapshot: &Arc, + engine: Arc, +) -> Result> { + let scan = snapshot.clone().scan_builder().build()?; + let batches = read_scan(&scan, engine)?; + assert!(!batches.is_empty(), "expected at least one batch"); + let merged = delta_kernel::arrow::compute::concat_batches(&batches[0].schema(), &batches)?; + let sort_indices = delta_kernel::arrow::compute::sort_to_indices(merged.column(0), None, None)?; + let sorted_columns: Vec = merged + .columns() + .iter() + .map(|col| delta_kernel::arrow::compute::take(col.as_ref(), &sort_indices, None).unwrap()) + .collect(); + Ok(RecordBatch::try_new(merged.schema(), sorted_columns)?) +} + +fn ts_to_micros(s: &str) -> i64 { + let ndt = NaiveDateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S%.f").unwrap(); + Utc.from_utc_datetime(&ndt) + .signed_duration_since(chrono::DateTime::UNIX_EPOCH) + .num_microseconds() + .unwrap() +} + +fn date_to_days(s: &str) -> i32 { + let date = NaiveDate::parse_from_str(s, "%Y-%m-%d").unwrap(); + let dt = Utc.from_utc_datetime(&date.and_hms_opt(0, 0, 0).unwrap()); + dt.signed_duration_since(chrono::DateTime::UNIX_EPOCH) + .num_days() as i32 +} + +fn ts_array(micros: i64) -> ArrayRef { + Arc::new(TimestampMicrosecondArray::from(vec![micros]).with_timezone("UTC")) +} + +fn ts_ntz_array(micros: i64) -> ArrayRef { + Arc::new(TimestampMicrosecondArray::from(vec![micros])) +} + +fn decimal_array(value: i128, precision: u8, scale: i8) -> ArrayRef { + Arc::new( + Decimal128Array::from(vec![value]) + .with_precision_and_scale(precision, scale) + .unwrap(), + ) +} + +/// Creates a partitioned table, writes one batch, and returns the post-commit snapshot. +/// Callers can subsequently use [`read_single_add`] with the returned `table_path` to +/// inspect the commit log. +async fn setup_and_write( + schema: Arc, + partition_cols: &[&str], + cm_mode: ColumnMappingMode, + write_partition_values_parsed: bool, + arrow_columns: Vec, + partition_values: HashMap, +) -> Result< + ( + tempfile::TempDir, + String, + Arc, + Arc, + ), + Box, +> { + let (tmp_dir, table_path, engine) = test_table_setup_mt()?; + // Data fields must not contain partition columns. + let (data_fields, data_columns): (Vec<_>, Vec) = schema + .fields() + .cloned() + .zip(arrow_columns) + .filter(|(f, _)| !partition_cols.contains(&f.name().as_str())) + .unzip(); + let kernel_data_schema = StructType::try_new(data_fields)?; + let arrow_data_schema: Arc = Arc::new((&kernel_data_schema).try_into_arrow()?); + let snapshot = create_partitioned_table( + &table_path, + engine.as_ref(), + schema, + partition_cols, + cm_mode, + write_partition_values_parsed, + )?; + + let batch = RecordBatch::try_new(arrow_data_schema, data_columns)?; + let snapshot = + write_batch_to_table(&snapshot, engine.as_ref(), batch, partition_values).await?; + + Ok(( + tmp_dir, + table_path, + snapshot, + engine as Arc, + )) +} + +/// Reads the snapshot, runs `assert_fn` on the sorted scan result, then checkpoints, +/// reloads a fresh snapshot from disk, and runs `assert_fn` again on the reloaded scan. +fn verify_and_checkpoint( + snapshot: &Arc, + engine: Arc, + assert_fn: impl Fn(&RecordBatch), +) -> Result<(), Box> { + let sorted = read_sorted(snapshot, engine.clone())?; + assert_fn(&sorted); + + snapshot.checkpoint(engine.as_ref(), None)?; + let reloaded = Snapshot::builder_for(snapshot.table_root()).build(engine.as_ref())?; + let sorted = read_sorted(&reloaded, engine)?; + assert_fn(&sorted); + Ok(()) +} + +// ============================================================================== +// JSON commit log helpers +// ============================================================================== + +/// Builds an unescaped Hive-style path prefix like `col1=val/col2=val/`. +/// Only correct when `value` contains no Hive-special characters. +fn hive_prefix(cols: &[&str], value: &str) -> String { + cols.iter() + .map(|c| format!("{c}={value}")) + .collect::>() + .join("/") + + "/" +} + +/// Reads the commit JSON at the given version and returns all add actions. +fn read_add_actions_json( + table_path: &str, + version: u64, +) -> Result, Box> { + let commit_path = format!("{table_path}/_delta_log/{version:020}.json"); + let content = std::fs::read_to_string(commit_path)?; + let parsed: Vec = serde_json::Deserializer::from_str(&content) + .into_iter::() + .collect::, _>>()?; + Ok(parsed + .into_iter() + .filter_map(|v| v.get("add").cloned()) + .collect()) +} + +/// Reads the single add action from the commit at `version` and returns the action +/// together with its path. Asserts the path is relative (no `scheme://` prefix). +fn read_single_add( + table_path: &str, + version: u64, +) -> Result<(serde_json::Value, String), Box> { + let adds = read_add_actions_json(table_path, version)?; + assert_eq!(adds.len(), 1, "should have exactly one add action"); + let add = adds.into_iter().next().unwrap(); + let rel_path = add["path"].as_str().unwrap().to_string(); + assert!( + !rel_path.contains("://"), + "should produce relative paths, got: {rel_path}" + ); + Ok((add, rel_path)) +} + +// ============================================================================== +// materializePartitionColumns tests +// ============================================================================== + +/// Materialized partition columns are physically present in the parquet file, but their +/// stats must be omitted from the Add action because the value is already in +/// `partitionValues`. Verifies that contract end-to-end through the kernel `create_table` +/// builder. +#[tokio::test(flavor = "multi_thread")] +async fn test_materialized_partition_columns_excluded_from_stats( +) -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + let partition_col = "partition"; + let table_schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("number", DataType::INTEGER), + StructField::nullable(partition_col, DataType::STRING), + ])?); + + let (_tmp_dir, table_path, engine) = test_table_setup_mt()?; + let _ = create_table(&table_path, table_schema.clone(), "test/1.0") + .with_data_layout(DataLayout::partitioned([partition_col])) + .with_table_properties([("delta.feature.materializePartitionColumns", "supported")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let mut txn = test_utils::load_and_begin_transaction(&table_path, engine.as_ref())? + .with_engine_info("default engine"); + + // Data batch must not contain the partition column. + let data_schema = Arc::new(StructType::try_new(vec![StructField::nullable( + "number", + DataType::INTEGER, + )])?); + let arrow_schema = Arc::new(data_schema.as_ref().try_into_arrow()?); + let batch = RecordBatch::try_new( + arrow_schema, + vec![Arc::new(Int32Array::from(vec![1, 2, 3]))], + )?; + let data = Box::new(ArrowEngineData::new(batch)); + + let write_context = txn.partitioned_write_context(HashMap::from([( + partition_col.to_string(), + Scalar::String("a".into()), + )]))?; + let result = engine.write_parquet(&data, &write_context).await?; + txn.add_files(result); + assert!(txn.commit(engine.as_ref())?.is_committed()); + + let (add, _) = read_single_add(&table_path, 1)?; + let stats: serde_json::Value = serde_json::from_str(add["stats"].as_str().unwrap()).unwrap(); + + // Stats should contain the data column but NOT the partition column. + assert!( + stats[MIN_VALUES].get("number").is_some(), + "data column 'number' should have minValues" + ); + assert!( + stats[MAX_VALUES].get("number").is_some(), + "data column 'number' should have maxValues" + ); + assert!( + stats[MIN_VALUES].get(partition_col).is_none(), + "partition column should not have minValues even when materialized" + ); + assert!( + stats[MAX_VALUES].get(partition_col).is_none(), + "partition column should not have maxValues even when materialized" + ); + assert!( + stats[NULL_COUNT].get(partition_col).is_none(), + "partition column should not have nullCount even when materialized" + ); + + Ok(()) +} + +/// End-to-end happy path for `materializePartitionColumns`: Writes two batches +/// and read & verify both the raw parquet and the scanned result. +#[rstest] +#[case::cm_none(ColumnMappingMode::None)] +#[case::cm_name(ColumnMappingMode::Name)] +#[case::cm_id(ColumnMappingMode::Id)] +#[tokio::test(flavor = "multi_thread")] +async fn test_materialize_partition_columns_e2e( + #[case] cm_mode: ColumnMappingMode, +) -> Result<(), Box> { + let cm = cm_mode_str(cm_mode); + // Partition columns p1, p2 sit in the middle of the data columns. + let table_schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("d1", DataType::INTEGER), + StructField::nullable("p1", DataType::STRING), + StructField::nullable("p2", DataType::INTEGER), + StructField::nullable("d2", DataType::INTEGER), + ])?); + + let (_tmp_dir, table_path, engine) = test_table_setup_mt()?; + let _ = create_table(&table_path, table_schema, "test/1.0") + .with_data_layout(DataLayout::partitioned(["p1", "p2"])) + .with_table_properties([ + ("delta.feature.materializePartitionColumns", "supported"), + ("delta.columnMapping.mode", cm), + ]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + + // Data schema excludes partition columns. + let kernel_data_schema = StructType::try_new(vec![ + StructField::nullable("d1", DataType::INTEGER), + StructField::nullable("d2", DataType::INTEGER), + ])?; + let arrow_data_schema: Arc = Arc::new((&kernel_data_schema).try_into_arrow()?); + let make_batch = |d1: Vec, d2: Vec| { + RecordBatch::try_new( + arrow_data_schema.clone(), + vec![ + Arc::new(Int32Array::from(d1)) as ArrayRef, + Arc::new(Int32Array::from(d2)), + ], + ) + .unwrap() + }; + + let partition_values = |p1: &str, p2: i32| { + HashMap::from([ + ("p1".to_string(), Scalar::String(p1.into())), + ("p2".to_string(), Scalar::Integer(p2)), + ]) + }; + + // A single commit writing two distinct partitions. + let mut txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? + .with_engine_info("default engine") + .with_data_change(true); + for (d1, d2, p1, p2) in [ + (vec![1, 2, 3], vec![10, 20, 30], "x", 5), + (vec![4, 5], vec![40, 50], "y", 6), + ] { + let wc = txn.partitioned_write_context(partition_values(p1, p2))?; + let add = engine + .write_parquet(&ArrowEngineData::new(make_batch(d1, d2)), &wc) + .await?; + txn.add_files(add); + } + let snapshot = txn.commit(engine.as_ref())?.unwrap_post_commit_snapshot(); + + // ===== Verify the materialized partition columns from parquet ===== + let logical_schema = snapshot.schema(); + let p1_phys = logical_schema.field("p1").unwrap().physical_name(cm_mode); + let p2_phys = logical_schema.field("p2").unwrap().physical_name(cm_mode); + let adds = read_add_actions_json(&table_path, 1)?; + assert_eq!(adds.len(), 2, "one commit should write two partition files"); + let mut found: Vec<(String, i32, usize)> = Vec::new(); + for add in &adds { + let rel_path = add["path"].as_str().unwrap(); + let parquet_path = Url::from_directory_path(&table_path) + .unwrap() + .join(rel_path)? + .to_file_path() + .unwrap(); + let file_batch = read_parquet_file(&parquet_path); + + let pv = add["partitionValues"].as_object().unwrap(); + let p1_from_delta_log = pv.get(p1_phys).and_then(|v| v.as_str()).unwrap(); + let p2_from_delta_log: i32 = pv.get(p2_phys).and_then(|v| v.as_str()).unwrap().parse()?; + + let p1 = get_column!(file_batch, p1_phys, StringArray); + let p2 = get_column!(file_batch, p2_phys, Int32Array); + assert!( + p1.iter().all(|v| v == Some(p1_from_delta_log)), + "materialized p1 should equal declared '{p1_from_delta_log}' in every row, got {p1:?}" + ); + assert!( + p2.iter().all(|v| v == Some(p2_from_delta_log)), + "materialized p2 should equal declared {p2_from_delta_log} in every row, got {p2:?}" + ); + found.push(( + p1_from_delta_log.to_string(), + p2_from_delta_log, + file_batch.num_rows(), + )); + } + found.sort(); + assert_eq!( + found, + vec![("x".to_string(), 5, 3), ("y".to_string(), 6, 2)] + ); + + // ===== Scan round-trip across both partitions. ===== + let sorted = read_sorted(&snapshot, engine.clone() as Arc)?; + let int_col = |name: &str| get_column!(sorted, name, Int32Array).values().to_vec(); + let p1_scan: Vec> = get_column!(sorted, "p1", StringArray).iter().collect(); + assert_eq!(int_col("d1"), vec![1, 2, 3, 4, 5]); + assert_eq!(int_col("d2"), vec![10, 20, 30, 40, 50]); + assert_eq!( + p1_scan, + vec![Some("x"), Some("x"), Some("x"), Some("y"), Some("y")] + ); + assert_eq!(int_col("p2"), vec![5, 5, 5, 6, 6]); + + Ok(()) +} + +#[tokio::test(flavor = "multi_thread")] +async fn test_materialize_all_primitive_partition_types() -> Result<(), Box> +{ + let (_tmp_dir, table_path, engine) = test_table_setup_mt()?; + let _ = create_table(&table_path, all_types_schema(), "test/1.0") + .with_data_layout(DataLayout::partitioned(PARTITION_COLS.iter().copied())) + .with_table_properties([("delta.feature.materializePartitionColumns", "supported")]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + + // Data schema excludes partition columns. + let data_schema = StructType::try_new(vec![StructField::nullable("value", DataType::INTEGER)])?; + let batch = RecordBatch::try_new( + Arc::new((&data_schema).try_into_arrow()?), + vec![normal_arrow_columns()[0].clone()], + )?; + let snapshot = write_batch_to_table( + &snapshot, + engine.as_ref(), + batch, + normal_partition_values()?, + ) + .await?; + + // Read raw parquet and verify the materialized partition columns. + let (_add, rel_path) = read_single_add(&table_path, 1)?; + let parquet_path = Url::from_directory_path(&table_path) + .unwrap() + .join(&rel_path)? + .to_file_path() + .unwrap(); + assert_normal_values(&read_parquet_file(&parquet_path)); + + // Validate the scan result for partition columns. + let sorted = read_sorted(&snapshot, engine.clone() as Arc)?; + assert_normal_values(&sorted); + + Ok(()) +} + +/// Including a partition column in the input data violates the write contract and errors, whether +/// or not the table materializes partition columns. +#[rstest] +#[tokio::test(flavor = "multi_thread")] +async fn test_input_data_with_partition_column_errors( + #[values(true, false)] materialized: bool, + #[values( + ColumnMappingMode::None, + ColumnMappingMode::Name, + ColumnMappingMode::Id + )] + cm_mode: ColumnMappingMode, +) -> Result<(), Box> { + let cm = cm_mode_str(cm_mode); + let partition_col = "partition"; + let table_schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("number", DataType::INTEGER), + StructField::nullable(partition_col, DataType::STRING), + ])?); + + let (_tmp_dir, table_path, engine) = test_table_setup_mt()?; + let mut properties = vec![("delta.columnMapping.mode", cm)]; + if materialized { + properties.push(("delta.feature.materializePartitionColumns", "supported")); + } + let _ = create_table(&table_path, table_schema.clone(), "test/1.0") + .with_data_layout(DataLayout::partitioned([partition_col])) + .with_table_properties(properties) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let txn = test_utils::load_and_begin_transaction(&table_path, engine.as_ref())? + .with_engine_info("default engine"); + + // Contract violation: the batch includes the `partition` column. + let arrow_schema = Arc::new(table_schema.as_ref().try_into_arrow()?); + let batch = RecordBatch::try_new( + arrow_schema, + vec![ + Arc::new(Int32Array::from(vec![1, 2, 3])), + Arc::new(StringArray::from(vec!["a", "a", "a"])), + ], + )?; + let data = Box::new(ArrowEngineData::new(batch)); + + let write_context = txn.partitioned_write_context(HashMap::from([( + partition_col.to_string(), + Scalar::String("a".into()), + )]))?; + let err = engine + .write_parquet(&data, &write_context) + .await + .err() + .expect("writing data that includes the partition column must fail") + .to_string(); + // The two cases fail at different layers. When materialized, the non-empty transform injects + // the partition literal and overflows the output schema. When not, the transform is empty, + // error comes when applying the physical schema to the transformed data. + let needle = if materialized { + "Too few fields in output schema" + } else { + "Passed struct had 2 columns, but transformed column has 1" + }; + assert!( + err.contains(needle), + "expected error containing {needle:?} (materialized={materialized}), got: {err}" + ); + + Ok(()) +} + +// ============================================================================== +// NOT NULL partition column tests +// ============================================================================== +// +// See [#2465](https://github.com/delta-io/delta-kernel-rs/issues/2465). These tests pin the +// kernel contract that mirrors Delta-Spark's `DELTA_NOT_NULL_CONSTRAINT_VIOLATED` (SQLSTATE +// 23502): a NULL written into a `NOT NULL` partition column must be rejected on the default +// engine. + +/// Validates the e2e NOT NULL contract on partition values: a null-equivalent value into a +/// `nullable: false` partition column is rejected before serialization, and ordinary +/// non-null values pass through unchanged. +/// +/// Three value cases, cross-multiplied against column-mapping modes and materialization: +/// +/// - `Scalar::Null(STRING)` -- explicit null, must be rejected. +/// - `Scalar::String("a")` -- ordinary non-null value, must be accepted. +/// - `Scalar::String("")` -- empty string, which the Delta protocol treats as JSON null in +/// `partitionValues`. It must be rejected for the same reason as `Scalar::Null` -- otherwise it +/// would slip past the nullability check and land a null value in a NOT NULL column. +#[rstest] +#[tokio::test(flavor = "multi_thread")] +async fn test_partition_null_validation( + #[values(false, true)] materialized: bool, + #[values( + ColumnMappingMode::None, + ColumnMappingMode::Name, + ColumnMappingMode::Id + )] + cm_mode: ColumnMappingMode, + #[values( + (Scalar::Null(DataType::STRING), Some("not nullable")), + (Scalar::String("a".into()), None), + (Scalar::String(String::new()), Some("not nullable")), + )] + case: (Scalar, Option<&'static str>), +) -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + let (value, expected_err) = case; + + // Schema: a nullable data column + a NOT NULL string partition column. + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("value", DataType::INTEGER), + StructField::not_null("p", DataType::STRING), + ])?); + + let (_tmp_dir, table_path, engine) = test_table_setup_mt()?; + let mut properties = vec![("delta.columnMapping.mode", cm_mode_str(cm_mode))]; + if materialized { + properties.push(("delta.feature.materializePartitionColumns", "supported")); + } + let _ = create_table(&table_path, schema, "test/1.0") + .with_data_layout(DataLayout::partitioned(["p"])) + .with_table_properties(properties) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + + let result = begin_transaction(snapshot, engine.as_ref())? + .with_engine_info("default engine") + .partitioned_write_context(HashMap::from([("p".to_string(), value)])); + + match expected_err { + Some(needle) => { + let err = result + .err() + .ok_or( + "expected partitioned_write_context to error for a null-equivalent value into NOT NULL partition", + )? + .to_string(); + assert!(err.contains(needle), "{err}"); + assert!(err.contains("'p'"), "{err}"); + } + None => { + result?; + } + } + + Ok(()) +} + +/// Verifies mixed-nullability partition schemas enforce each partition column's own contract: +/// null-equivalent values are accepted for nullable partition columns and rejected for NOT NULL +/// partition columns. +#[rstest] +#[tokio::test(flavor = "multi_thread")] +async fn test_partition_null_validation_mixed_nullability( + #[values( + ColumnMappingMode::None, + ColumnMappingMode::Name, + ColumnMappingMode::Id + )] + cm_mode: ColumnMappingMode, +) -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("value", DataType::INTEGER), + StructField::not_null("p_required", DataType::STRING), + StructField::nullable("p_optional", DataType::STRING), + ])?); + + let (_tmp_dir, table_path, engine) = test_table_setup_mt()?; + let snapshot = create_partitioned_table( + &table_path, + engine.as_ref(), + schema, + &["p_required", "p_optional"], + cm_mode, + false, // write_partition_values_parsed; unused, no checkpoint in this test + )?; + + begin_transaction(snapshot.clone(), engine.as_ref())? + .with_engine_info("default engine") + .partitioned_write_context(HashMap::from([ + ("p_required".to_string(), Scalar::String("a".into())), + ("p_optional".to_string(), Scalar::Null(DataType::STRING)), + ]))?; + + begin_transaction(snapshot.clone(), engine.as_ref())? + .with_engine_info("default engine") + .partitioned_write_context(HashMap::from([ + ("p_required".to_string(), Scalar::String("a".into())), + ("p_optional".to_string(), Scalar::String(String::new())), + ]))?; + + let err = begin_transaction(snapshot, engine.as_ref())? + .with_engine_info("default engine") + .partitioned_write_context(HashMap::from([ + ("p_required".to_string(), Scalar::Null(DataType::STRING)), + ("p_optional".to_string(), Scalar::String("b".into())), + ])) + .unwrap_err() + .to_string(); + assert!(err.contains("not nullable"), "{err}"); + assert!(err.contains("'p_required'"), "{err}"); + + Ok(()) +} diff --git a/kernel/tests/integration/write/post_commit.rs b/kernel/tests/integration/write/post_commit.rs new file mode 100644 index 0000000000..abd42501e7 --- /dev/null +++ b/kernel/tests/integration/write/post_commit.rs @@ -0,0 +1,147 @@ +//! Integration tests for post-commit snapshot propagation and write-parquet partition-context +//! error paths. + +use std::collections::HashMap; +use std::sync::Arc; + +use delta_kernel::arrow::array::{Int32Array, RecordBatch}; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::engine::arrow_conversion::TryIntoArrow as _; +use delta_kernel::expressions::Scalar; +use delta_kernel::schema::{schema_ref, DataType, StructField, StructType}; +use delta_kernel::transaction::create_table::create_table as create_table_txn; +use delta_kernel::transaction::CommitResult; +use delta_kernel::{DeltaResult, Snapshot}; +use tempfile::tempdir; +use test_utils::{ + begin_transaction, create_default_engine, setup_test_tables, write_batch_to_table, +}; +use url::Url; + +use crate::common::write_utils::get_simple_int_schema; + +#[tokio::test] +async fn test_post_commit_snapshot_create_then_insert() -> DeltaResult<()> { + let _ = tracing_subscriber::fmt::try_init(); + + let temp_dir = tempdir().unwrap(); + let table_url = Url::from_directory_path(temp_dir.path()).unwrap(); + let engine = create_default_engine(&table_url)?; + let schema = get_simple_int_schema(); + + // Create table and verify post_commit_snapshot + let create_result = create_table_txn(table_url.as_str(), schema, env!("CARGO_PKG_VERSION")) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let mut current_snapshot = match create_result { + CommitResult::CommittedTransaction(committed) => { + assert_eq!(committed.commit_version(), 0); + // CREATE TABLE is the first commit: 1 commit since last checkpoint/compaction + assert_eq!(committed.post_commit_stats().commits_since_checkpoint, 1); + assert_eq!( + committed.post_commit_stats().commits_since_log_compaction, + 1 + ); + let post_snapshot = committed + .post_commit_snapshot() + .expect("should have post_commit_snapshot"); + assert_eq!(post_snapshot.version(), 0); + post_snapshot.clone() + } + _ => panic!("Create should succeed"), + }; + + // Do 10 inserts and verify post_commit_snapshot for each + for i in 1..11 { + let base_version = current_snapshot.version(); + + let txn = + begin_transaction(current_snapshot.clone(), engine.as_ref())?.with_engine_info("test"); + + match txn.commit(engine.as_ref())? { + CommitResult::CommittedTransaction(committed) => { + let post_snapshot = committed + .post_commit_snapshot() + .expect("should have post_commit_snapshot"); + + assert_eq!(post_snapshot.version(), base_version + 1); + assert_eq!(post_snapshot.version(), committed.commit_version()); + assert_eq!(post_snapshot.schema(), current_snapshot.schema()); + assert_eq!(post_snapshot.table_root(), current_snapshot.table_root()); + + current_snapshot = post_snapshot.clone(); + } + _ => panic!("Commit {i} should succeed"), + } + } + + Ok(()) +} + +#[tokio::test] +async fn test_write_parquet_succeed_with_logical_partition_names( +) -> Result<(), Box> { + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("letter", DataType::STRING), + ])?); + + for (table_url, engine, _store, _table_name) in setup_test_tables( + schema.clone(), + &["letter"], + None, + "test_partition_translate", + ) + .await? + { + let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; + + // Create data with only the non-partition column + let data_schema = schema_ref! { nullable "id": INTEGER }; + let batch = RecordBatch::try_new( + Arc::new(data_schema.as_ref().try_into_arrow()?), + vec![Arc::new(Int32Array::from(vec![1, 2]))], + )?; + + // Pass partition values with logical name -- should succeed + let result = write_batch_to_table( + &snapshot, + &engine, + batch, + HashMap::from([("letter".to_string(), Scalar::String("a".into()))]), + ) + .await; + assert!( + result.is_ok(), + "write_parquet should succeed with valid logical partition name" + ); + } + Ok(()) +} + +#[tokio::test] +async fn test_write_parquet_rejects_partitioned_write_context_on_unpartitioned_table( +) -> Result<(), Box> { + let schema = get_simple_int_schema(); + + for (table_url, engine, _store, _table_name) in + setup_test_tables(schema.clone(), &[], None, "test_partition_reject").await? + { + let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; + let txn = begin_transaction(snapshot.clone(), &engine)?.with_engine_info("test"); + + let result = txn.partitioned_write_context(HashMap::from([( + "nonexistent".to_string(), + Scalar::String("val".into()), + )])); + let err = + result.expect_err("should fail with partitioned_write_context on unpartitioned table"); + let err_msg = err.to_string(); + assert!( + err_msg.contains("table is not partitioned"), + "Error should indicate table is not partitioned, got: {err_msg}" + ); + } + Ok(()) +} diff --git a/kernel/tests/integration/write/relative_paths.rs b/kernel/tests/integration/write/relative_paths.rs new file mode 100644 index 0000000000..93edda6ed3 --- /dev/null +++ b/kernel/tests/integration/write/relative_paths.rs @@ -0,0 +1,165 @@ +//! Integration tests that verify kernel-written files are referenced by relative path. + +use std::collections::HashMap; +use std::sync::Arc; + +use delta_kernel::arrow::record_batch::RecordBatch; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::transaction::create_table::create_table as create_table_txn; +use delta_kernel::Snapshot; +use test_utils::delta_kernel_default_engine::executor::tokio::TokioBackgroundExecutor; +use test_utils::delta_kernel_default_engine::DefaultEngine; +use test_utils::{ + begin_transaction, create_table_and_load_snapshot, read_add_infos, test_table_setup, + write_batch_to_table, +}; +use url::Url; + +use crate::common::write_utils::{get_simple_schema, simple_id_batch}; + +/// Helper to write a batch and return the post-commit snapshot. +async fn write_batch_to_table_simple( + snapshot: &Arc, + engine: &DefaultEngine, + data: RecordBatch, +) -> Result, Box> { + let mut txn = begin_transaction(snapshot.clone(), engine)?.with_engine_info("test"); + let write_context = txn.unpartitioned_write_context()?; + let add_meta = engine + .write_parquet(&ArrowEngineData::new(data), &write_context) + .await?; + txn.add_files(add_meta); + let committed = txn.commit(engine)?.unwrap_committed(); + Ok(committed.post_commit_snapshot().unwrap().clone()) +} + +#[tokio::test] +async fn test_write_uses_relative_paths_and_readback() -> Result<(), Box> { + let schema = get_simple_schema(); + let (_tmp_dir, table_path, engine) = test_table_setup()?; + let snapshot = + create_table_and_load_snapshot(&table_path, schema.clone(), engine.as_ref(), &[])?; + + let snapshot = write_batch_to_table_simple( + &snapshot, + engine.as_ref(), + simple_id_batch(&schema, vec![1, 2, 3]), + ) + .await?; + + // Verify paths in the log are relative (no scheme like "s3://") + let add_infos = read_add_infos(&snapshot, engine.as_ref())?; + assert_eq!(add_infos.len(), 1); + let path = &add_infos[0].path; + assert!( + !path.contains("://"), + "should produce relative paths, got: {path}" + ); + + // Verify data is readable via scan + let scan = snapshot.scan_builder().build()?; + let batches = test_utils::read_scan(&scan, engine)?; + assert_eq!(batches.len(), 1); + assert_eq!(batches[0].num_rows(), 3); + + Ok(()) +} + +#[tokio::test] +async fn test_multiple_files_in_commit_all_use_relative_paths( +) -> Result<(), Box> { + let schema = get_simple_schema(); + let (_tmp_dir, table_path, engine) = test_table_setup()?; + let snapshot = + create_table_and_load_snapshot(&table_path, schema.clone(), engine.as_ref(), &[])?; + + let mut txn = begin_transaction(snapshot.clone(), engine.as_ref())?.with_engine_info("test"); + let write_context = txn.unpartitioned_write_context().unwrap(); + for values in [vec![1, 2], vec![3, 4]] { + let add_meta = engine + .write_parquet( + &ArrowEngineData::new(simple_id_batch(&schema, values)), + &write_context, + ) + .await?; + txn.add_files(add_meta); + } + let committed = txn.commit(engine.as_ref())?.unwrap_committed(); + let snapshot = committed.post_commit_snapshot().unwrap().clone(); + + let add_infos = read_add_infos(&snapshot, engine.as_ref())?; + assert_eq!(add_infos.len(), 2); + for info in &add_infos { + assert!( + !info.path.contains("://"), + "Expected relative path, got: {}", + info.path + ); + } + + Ok(()) +} + +#[tokio::test] +async fn test_multiple_commits_with_relative_paths_all_readable( +) -> Result<(), Box> { + let schema = get_simple_schema(); + let (_tmp_dir, table_path, engine) = test_table_setup()?; + let mut snapshot = + create_table_and_load_snapshot(&table_path, schema.clone(), engine.as_ref(), &[])?; + + for values in [vec![1, 2], vec![3, 4], vec![5, 6]] { + snapshot = write_batch_to_table( + &snapshot, + engine.as_ref(), + simple_id_batch(&schema, values), + HashMap::new(), + ) + .await?; + } + + let scan = snapshot.scan_builder().build()?; + let batches = test_utils::read_scan(&scan, engine)?; + let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total_rows, 6); + + Ok(()) +} + +#[tokio::test] +async fn test_create_table_with_data_uses_relative_paths() -> Result<(), Box> +{ + let schema = get_simple_schema(); + let (_tmp_dir, table_path, engine) = test_table_setup()?; + let table_url = Url::from_directory_path(&table_path).unwrap(); + + let mut txn = create_table_txn(table_url.as_str(), schema.clone(), "test/1.0") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))?; + let write_context = txn.unpartitioned_write_context()?; + let add_meta = engine + .write_parquet( + &ArrowEngineData::new(simple_id_batch(&schema, vec![10, 20])), + &write_context, + ) + .await?; + txn.add_files(add_meta); + let committed = txn.commit(engine.as_ref())?.unwrap_committed(); + let snapshot = committed.post_commit_snapshot().unwrap().clone(); + + let add_infos = read_add_infos(&snapshot, engine.as_ref())?; + assert_eq!(add_infos.len(), 1); + let path = &add_infos[0].path; + assert!( + !path.contains("://"), + "should produce relative paths, got: {path}" + ); + + // Verify data is readable + let scan = snapshot.scan_builder().build()?; + let batches = test_utils::read_scan(&scan, engine)?; + assert_eq!(batches.len(), 1); + assert_eq!(batches[0].num_rows(), 2); + + Ok(()) +} diff --git a/kernel/tests/integration/write/remove_dv.rs b/kernel/tests/integration/write/remove_dv.rs new file mode 100644 index 0000000000..21582ec452 --- /dev/null +++ b/kernel/tests/integration/write/remove_dv.rs @@ -0,0 +1,1858 @@ +//! Integration tests for remove-file and deletion-vector-update write paths. + +use std::collections::HashMap; +use std::sync::Arc; + +use delta_kernel::actions::deletion_vector::{DeletionVectorDescriptor, DeletionVectorStorageType}; +use delta_kernel::actions::{NUM_RECORDS, TIGHT_BOUNDS}; +use delta_kernel::arrow::array::builder::{MapBuilder, MapFieldNames, StringBuilder}; +use delta_kernel::arrow::array::{ + new_null_array, Array, ArrayRef, AsArray, Int32Array, Int64Array, RecordBatch, StringArray, + StructArray, +}; +use delta_kernel::arrow::compute::concat_batches; +use delta_kernel::arrow::datatypes::{ + DataType as ArrowDataType, Field as ArrowField, Schema as ArrowSchema, +}; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::engine::arrow_conversion::TryIntoArrow as _; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::engine_data::FilteredEngineData; +use delta_kernel::expressions::{col, lit, ExpressionStructPatchBuilder, MapData, Scalar}; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::ObjectStoreExt as _; +use delta_kernel::scan::{scan_row_schema, StatsOptions}; +use delta_kernel::schema::{schema_ref, DataType, MapType, StructField, StructType}; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::transaction::CommitResult; +use delta_kernel::{DeltaResult, Engine, Error, Expression as Expr, Predicate as Pred, Snapshot}; +use itertools::Itertools; +use serde_json::Deserializer; +use tempfile::tempdir; +use test_utils::{ + assert_result_error_with_message, begin_transaction, copy_directory, create_default_engine, + create_default_engine_mt_executor, insert_data, into_record_batch, load_and_begin_transaction, + read_actions_from_commit, replace_array_row, setup_test_tables, test_table_setup, +}; +use url::Url; + +use crate::common::write_utils::{ + create_dv_table_with_files, get_scan_files, get_simple_int_schema, sequential_dv_descriptors, + set_table_properties, write_data_and_check_result_and_stats, +}; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum AppendOnlyWrite { + Remove, + DeletionVectorUpdate, +} + +#[rstest::rstest] +#[case::no_data_change_all_selected( + false, /* data_change */ + &[true, true, true], /* selection_vector */ + None, /* expected_error */ +)] +#[case::data_change_all_selected( + true, /* data_change */ + &[true, true, true], /* selection_vector */ + Some("Append-only tables cannot remove files"), /* expected_error */ +)] +#[case::no_data_change_partially_selected( + false, /* data_change */ + &[true, false, true], /* selection_vector */ + None, /* expected_error */ +)] +#[case::data_change_partially_selected( + true, /* data_change */ + &[true, false, true], /* selection_vector */ + Some("Append-only tables cannot remove files"), /* expected_error */ +)] +#[case::data_change_none_selected( + true, /* data_change */ + &[false, false, false], /* selection_vector */ + None, /* expected_error */ +)] +#[tokio::test] +async fn append_only_enforces_data_change_for_file_actions( + #[values(AppendOnlyWrite::Remove, AppendOnlyWrite::DeletionVectorUpdate)] + operation: AppendOnlyWrite, + #[values(0, 1)] batch_index: usize, + #[case] data_change: bool, + #[case] selection_vector: &[bool], + #[case] expected_error: Option<&str>, +) -> Result<(), Box> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let table_url = Url::from_directory_path(&table_path).unwrap(); + let schema = schema_ref! { nullable "number": INTEGER }; + + let snapshot = create_table(&table_path, schema, "Test/1.0") + .with_table_properties([ + ("delta.appendOnly", "true"), + ("delta.enableDeletionVectors", "true"), + ]) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + let mut txn = begin_transaction(snapshot, engine.as_ref())?.with_data_change(true); + let write_context = txn.unpartitioned_write_context()?; + let arrow_schema: Arc = + Arc::new(write_context.physical_schema().as_ref().try_into_arrow()?); + for value in [1, 2, 3] { + let data = ArrowEngineData::new(RecordBatch::try_new( + arrow_schema.clone(), + vec![Arc::new(Int32Array::from(vec![value]))], + )?); + txn.add_files(engine.write_parquet(&data, &write_context).await?); + } + let snapshot = txn.commit(engine.as_ref())?.unwrap_post_commit_snapshot(); + + let staged_batches = (0..2) + .map(|index| { + let scan_files = selected_scan_file_batch(snapshot.clone(), engine.as_ref())?; + let (data, _) = scan_files.into_parts(); + assert_eq!(data.len(), selection_vector.len()); + let selection_vector = if index == batch_index { + selection_vector.to_vec() + } else { + vec![false; data.len()] + }; + FilteredEngineData::try_new(data, selection_vector) + }) + .collect::>>()?; + let mut txn = begin_transaction(snapshot, engine.as_ref())? + .with_operation("DELETE".to_string()) + .with_data_change(data_change); + // NOTE: The data_change=false cases do not preserve removed records in replacement AddFiles. + // This is not a valid rearrangement, we construct such commit only for testing. + let commit_result = match operation { + AppendOnlyWrite::Remove => { + for scan_files in staged_batches { + txn.remove_files(scan_files); + } + txn.commit(engine.as_ref()) + } + AppendOnlyWrite::DeletionVectorUpdate => { + let add_actions = read_actions_from_commit(&table_url, 1, "add")?; + let dv_map = add_actions + .iter() + .zip(selection_vector) + .enumerate() + .filter(|(_, (_, selected))| **selected) + .map(|(index, (add, _))| { + let path = add["path"] + .as_str() + .expect("add path should be present") + .to_string(); + let dv = DeletionVectorDescriptor { + storage_type: DeletionVectorStorageType::PersistedRelative, + path_or_inline_dv: format!("dv-{index}.bin"), + offset: Some(0), + size_in_bytes: 1, + cardinality: 1, + }; + (path, dv) + }) + .collect(); + txn.update_deletion_vectors(dv_map, staged_batches.into_iter().map(Ok))?; + txn.commit(engine.as_ref()) + } + }; + + if let Some(expected_error) = expected_error { + assert_result_error_with_message(commit_result, expected_error); + } else { + commit_result?.unwrap_committed(); + } + + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + assert_eq!( + snapshot.version(), + if expected_error.is_some() { 1 } else { 2 } + ); + let scan = snapshot.scan_builder().build()?; + let mut active_files = 0; + for scan_metadata in scan.scan_metadata(engine.as_ref())? { + let scan_files = scan_metadata?.scan_files; + active_files += scan_files + .selection_vector() + .iter() + .filter(|selected| **selected) + .count() + + scan_files.data().len() + - scan_files.selection_vector().len(); + } + let selected_files = selection_vector + .iter() + .filter(|selected| **selected) + .count(); + let expected_active_files = match operation { + AppendOnlyWrite::Remove if expected_error.is_none() => 3 - selected_files, + AppendOnlyWrite::Remove | AppendOnlyWrite::DeletionVectorUpdate => 3, + }; + assert_eq!(active_files, expected_active_files); + + Ok(()) +} + +fn selected_scan_file_batch( + snapshot: Arc, + engine: &dyn Engine, +) -> DeltaResult { + for scan_files in get_scan_files(snapshot, engine)? { + let data = scan_files.apply_selection_vector()?; + if !data.is_empty() { + return Ok(FilteredEngineData::with_all_rows_selected(data)); + } + } + Err(Error::generic("expected at least one scan file")) +} + +#[tokio::test] +async fn test_remove_files_adds_expected_entries() -> Result<(), Box> { + // This test verifies that Remove actions generated from scan metadata contain all expected + // fields from the Remove struct (defined in kernel/src/actions/mod.rs). + // + // This test uses the table-with-dv-small dataset which contains files with tags and deletion + // vectors. + // + // Not populated in the dataset are (covered by row_tracking tests): + // baseRowId (optional i64) + // defaultRowCommitVersion (optional i64) + use std::path::PathBuf; + + let _ = tracing_subscriber::fmt::try_init(); + + let tmp_dir = tempdir()?; + let tmp_table_path = tmp_dir.path().join("table-with-dv-small"); + let source_path = std::fs::canonicalize(PathBuf::from("./tests/data/table-with-dv-small/"))?; + copy_directory(&source_path, &tmp_table_path)?; + + let table_url = url::Url::from_directory_path(&tmp_table_path).unwrap(); + let engine = create_default_engine(&table_url)?; + + let snapshot = Snapshot::builder_for(table_url.clone()) + .at_version(1) + .build(engine.as_ref())?; + + let mut txn = begin_transaction(snapshot.clone(), engine.as_ref())? + .with_engine_info("test engine") + .with_data_change(true); + + let scan = snapshot.scan_builder().build()?; + let scan_metadata = scan.scan_metadata(engine.as_ref())?.next().unwrap()?; + + let (data, selection_vector) = scan_metadata.scan_files.into_parts(); + let remove_metadata = FilteredEngineData::try_new(data, selection_vector)?; + + txn.remove_files(remove_metadata); + + let result = txn.commit(engine.as_ref())?; + + match result { + CommitResult::CommittedTransaction(committed) => { + let commit_version = committed.commit_version(); + + // Read the commit log directly to verify remove actions + let commit_path = tmp_table_path.join(format!("_delta_log/{commit_version:020}.json")); + let commit_content = std::fs::read_to_string(commit_path)?; + + let parsed_commits: Vec<_> = Deserializer::from_str(&commit_content) + .into_iter::() + .try_collect()?; + + // Verify we have at least commitInfo and remove actions + assert!( + parsed_commits.len() >= 2, + "Expected at least 2 actions (commitInfo + remove)" + ); + + // Extract the commitInfo timestamp to validate against deletionTimestamp + let commit_info_action = parsed_commits + .iter() + .find(|action| action.get("commitInfo").is_some()) + .expect("Missing commitInfo action"); + let commit_info = &commit_info_action["commitInfo"]; + let commit_timestamp = commit_info["timestamp"] + .as_i64() + .expect("Missing timestamp in commitInfo"); + + // Verify remove actions + let remove_actions: Vec<_> = parsed_commits + .iter() + .filter(|action| action.get("remove").is_some()) + .collect(); + + assert!( + !remove_actions.is_empty(), + "Expected at least one remove action" + ); + + assert_eq!(remove_actions.len(), 1); + let remove_action = remove_actions[0]; + let remove = &remove_action["remove"]; + + // path (required) + assert!(remove.get("path").is_some(), "Missing path field"); + let path = remove["path"].as_str().expect("path should be a string"); + assert_eq!( + path, + "part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet" + ); + + // dataChange (required) + assert_eq!(remove["dataChange"].as_bool(), Some(true)); + + // deletionTimestamp (optional) - should match commit timestamp + let deletion_timestamp = remove["deletionTimestamp"] + .as_i64() + .expect("Missing deletionTimestamp"); + assert_eq!( + deletion_timestamp, commit_timestamp, + "deletionTimestamp should match commit timestamp" + ); + + // extendedFileMetadata (optional) + assert_eq!(remove["extendedFileMetadata"].as_bool(), Some(true)); + + // partitionValues (optional) + let partition_vals = remove["partitionValues"] + .as_object() + .expect("Missing partitionValues"); + assert_eq!(partition_vals.len(), 0); + + // size (optional) + let size = remove["size"].as_i64().expect("Missing size"); + assert_eq!(size, 635); + + // stats (optional) + let stats = remove["stats"].as_str().expect("Missing stats"); + let stats_json: serde_json::Value = serde_json::from_str(stats)?; + assert_eq!(stats_json[NUM_RECORDS], 10); + + // tags (optional) + let tags = remove["tags"].as_object().expect("Missing tags"); + assert_eq!( + tags.get("INSERTION_TIME").and_then(|v| v.as_str()), + Some("1677811178336000") + ); + assert_eq!( + tags.get("MIN_INSERTION_TIME").and_then(|v| v.as_str()), + Some("1677811178336000") + ); + assert_eq!( + tags.get("MAX_INSERTION_TIME").and_then(|v| v.as_str()), + Some("1677811178336000") + ); + assert_eq!( + tags.get("OPTIMIZE_TARGET_SIZE").and_then(|v| v.as_str()), + Some("268435456") + ); + + // deletionVector (optional) + let dv = remove["deletionVector"] + .as_object() + .expect("Missing deletionVector"); + assert_eq!(dv.get("storageType").and_then(|v| v.as_str()), Some("u")); + assert_eq!( + dv.get("pathOrInlineDv").and_then(|v| v.as_str()), + Some("vBn[lx{q8@P<9BNH/isA") + ); + assert_eq!(dv.get("offset").and_then(|v| v.as_i64()), Some(1)); + assert_eq!(dv.get("sizeInBytes").and_then(|v| v.as_i64()), Some(36)); + assert_eq!(dv.get("cardinality").and_then(|v| v.as_i64()), Some(2)); + + // Row tracking fields should be absent as the feature is was not enabled on writing + // row_tracking tests cover having these populated. + assert!(remove.get("baseRowId").is_none()); + assert!(remove.get("defaultRowCommitVersion").is_none()); + } + _ => panic!("Transaction should be committed"), + } + + Ok(()) +} + +/// Verifies `extendedFileMetadata` is true exactly when `size` and `partitionValues` are present; +/// `tags` does not affect it. +#[rstest::rstest] +#[case::all_present(&[], true)] +#[case::missing_size(&[ExtendedMetadataField::Size], false)] +#[case::missing_partition_values(&[ExtendedMetadataField::PartitionValues], false)] +#[case::missing_tags(&[ExtendedMetadataField::Tags], true)] +#[case::only_size(&[ + ExtendedMetadataField::PartitionValues, + ExtendedMetadataField::Tags, +], false)] +#[case::only_partition_values(&[ExtendedMetadataField::Size, ExtendedMetadataField::Tags], false)] +#[case::only_tags(&[ + ExtendedMetadataField::Size, + ExtendedMetadataField::PartitionValues, +], false)] +#[case::none_present(&ExtendedMetadataField::ALL, false)] +#[tokio::test] +async fn test_remove_scanned_file_sets_extended_metadata( + #[case] missing_fields: &[ExtendedMetadataField], + #[case] expected_extended_file_metadata: bool, +) -> Result<(), Box> { + let (_temp_dir, table_path, engine) = test_table_setup()?; + let table_url = Url::from_directory_path(&table_path).unwrap(); + let schema = schema_ref! { nullable "number": INTEGER }; + + let snapshot = create_table(&table_path, schema, "Test/1.0") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + let snapshot = insert_data( + snapshot, + &engine, + vec![Arc::new(Int32Array::from(vec![1, 2, 3]))], + ) + .await? + .unwrap_post_commit_snapshot(); + + let scan = snapshot.clone().scan_builder().build()?; + let mut txn = begin_transaction(snapshot, engine.as_ref())?.with_data_change(true); + for scan_metadata in scan.scan_metadata(engine.as_ref())? { + txn.remove_files(with_missing_extended_metadata_fields( + engine.as_ref(), + scan_metadata?.scan_files, + missing_fields, + )?); + } + let commit_result = txn.commit(engine.as_ref()); + if missing_fields.contains(&ExtendedMetadataField::Size) { + // TODO(#2717): The commit is materialized before post-commit validation returns this error, + // so the committed Remove action remains available for validation below. + assert_result_error_with_message(commit_result, "Data missing for field size"); + } else { + commit_result?.unwrap_committed(); + } + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + + let remove_actions = read_actions_from_commit(&table_url, 2, "remove")?; + assert_eq!(remove_actions.len(), 1); + let remove = &remove_actions[0]; + assert_eq!( + remove["extendedFileMetadata"], + expected_extended_file_metadata + ); + for field in ExtendedMetadataField::ALL { + let present = remove + .get(field.name()) + .is_some_and(|value| !value.is_null()); + assert_eq!(present, !missing_fields.contains(&field)); + } + + let scan = snapshot.scan_builder().build()?; + let mut surviving_files = 0; + for scan_metadata in scan.scan_metadata(engine.as_ref())? { + surviving_files += scan_metadata? + .scan_files + .selection_vector() + .iter() + .filter(|selected| **selected) + .count(); + } + assert_eq!(surviving_files, 0); + + Ok(()) +} + +#[tokio::test] +async fn test_update_deletion_vectors_adds_expected_entries( +) -> Result<(), Box> { + // This test verifies that deletion vector updates write proper Remove and Add actions + // to the transaction log. + // + // NOTE: Additional unit tests for update_deletion_vectors exist in + // kernel/src/transaction/mod.rs + // + // The test validates: + // 1. Transaction setup for DV updates + // 2. Scanning and extracting scan files with DV data + // 3. Creating new DV descriptors for the files + // 4. Calling update_deletion_vectors to update the DVs + // 5. Committing and verifying the generated actions + // + // Expected commit log structure: + // - commitInfo: Contains metadata about the transaction + // - remove: Contains OLD deletion vector data and original file metadata + // - add: Contains NEW deletion vector data and updated file metadata + // + // The test ensures: + // - Remove action has the OLD DV descriptor with all 5 fields + // - Add action has the NEW DV descriptor with all 5 fields + // - All file metadata is preserved (size, stats, tags, partitionValues) + // - dataChange is properly set to true + // - deletionTimestamp matches commit timestamp + use std::path::PathBuf; + + let _ = tracing_subscriber::fmt::try_init(); + + let tmp_dir = tempdir()?; + let tmp_table_path = tmp_dir.path().join("table-with-dv-small"); + let source_path = std::fs::canonicalize(PathBuf::from("./tests/data/table-with-dv-small/"))?; + copy_directory(&source_path, &tmp_table_path)?; + + let table_url = url::Url::from_directory_path(&tmp_table_path).unwrap(); + let engine = create_default_engine(&table_url)?; + + let snapshot = Snapshot::builder_for(table_url.clone()) + .at_version(1) + .build(engine.as_ref())?; + + // Create transaction with DV update mode enabled + let mut txn = begin_transaction(snapshot.clone(), engine.as_ref())? + .with_engine_info("test engine") + .with_operation("UPDATE".to_string()) + .with_data_change(true); + + // Build scan and collect all scan metadata + let scan = snapshot.clone().scan_builder().build()?; + let all_scan_metadata: Vec<_> = scan + .scan_metadata(engine.as_ref())? + .collect::, _>>()?; + + // Extract scan files for DV update + let scan_files: Vec<_> = all_scan_metadata + .into_iter() + .map(|sm| sm.scan_files) + .collect(); + + // Create new DV descriptors for the files + let file_path = "part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet"; + let mut dv_map = HashMap::new(); + + // Create a NEW deletion vector descriptor (different from the original) + let new_dv = DeletionVectorDescriptor { + storage_type: DeletionVectorStorageType::PersistedRelative, + path_or_inline_dv: "cd^-aqEH.-t@S}K{vb[*k^".to_string(), + offset: Some(10), + size_in_bytes: 40, + cardinality: 3, + }; + dv_map.insert(file_path.to_string(), new_dv); + + // Call update_deletion_vectors to exercise the API + txn.update_deletion_vectors(dv_map, scan_files.into_iter().map(Ok))?; + + // Commit the transaction + let result = txn.commit(engine.as_ref())?; + + match result { + CommitResult::CommittedTransaction(committed) => { + let commit_version = committed.commit_version(); + + // Read the original version 1 log to get original file metadata + let original_log_path = tmp_table_path.join("_delta_log/00000000000000000001.json"); + let original_log_content = std::fs::read_to_string(original_log_path)?; + let original_commits: Vec<_> = Deserializer::from_str(&original_log_content) + .into_iter::() + .try_collect()?; + + let file_path = "part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet"; + + // Extract original file metadata from version 1 + let original_add = original_commits + .iter() + .find(|action| { + action + .get("add") + .and_then(|add| add.get("path").and_then(|p| p.as_str())) + == Some(file_path) + }) + .expect("Missing original add action in version 1") + .get("add") + .expect("Should have add field"); + + let original_size = original_add["size"] + .as_i64() + .expect("Original add action should have size"); + let original_partition_values = original_add["partitionValues"] + .as_object() + .expect("Original add action should have partitionValues"); + let original_tags = original_add.get("tags"); + let original_stats = original_add.get("stats"); + + // Read the commit log directly + let commit_path = tmp_table_path.join(format!("_delta_log/{commit_version:020}.json")); + let commit_content = std::fs::read_to_string(commit_path)?; + + let parsed_commits: Vec<_> = Deserializer::from_str(&commit_content) + .into_iter::() + .try_collect()?; + + // Should have commitInfo, remove, and add actions + assert!( + parsed_commits.len() >= 3, + "Expected at least 3 actions (commitInfo + remove + add), got {}", + parsed_commits.len() + ); + + // Extract commitInfo timestamp + let commit_info_action = parsed_commits + .iter() + .find(|action| action.get("commitInfo").is_some()) + .expect("Missing commitInfo action"); + let commit_info = &commit_info_action["commitInfo"]; + let commit_timestamp = commit_info["timestamp"] + .as_i64() + .expect("Missing timestamp in commitInfo"); + + // Verify remove action contains OLD DV information + let remove_actions: Vec<_> = parsed_commits + .iter() + .filter(|action| action.get("remove").is_some()) + .collect(); + + assert_eq!( + remove_actions.len(), + 1, + "Expected exactly one remove action" + ); + + let remove_action = remove_actions[0]; + let remove = &remove_action["remove"]; + + assert_eq!( + remove["path"].as_str(), + Some(file_path), + "Remove path should match" + ); + assert_eq!(remove["dataChange"].as_bool(), Some(true)); + assert_eq!( + remove["deletionTimestamp"].as_i64(), + Some(commit_timestamp), + "deletionTimestamp should match commit timestamp" + ); + + // Verify OLD deletion vector in remove action + let old_dv = remove["deletionVector"] + .as_object() + .expect("Remove action should have deletionVector"); + assert_eq!( + old_dv.get("storageType").and_then(|v| v.as_str()), + Some("u"), + "Old DV storage type should be 'u'" + ); + assert_eq!( + old_dv.get("pathOrInlineDv").and_then(|v| v.as_str()), + Some("vBn[lx{q8@P<9BNH/isA"), + "Old DV path should match original" + ); + assert_eq!( + old_dv.get("offset").and_then(|v| v.as_i64()), + Some(1), + "Old DV offset should be 1" + ); + assert_eq!( + old_dv.get("sizeInBytes").and_then(|v| v.as_i64()), + Some(36), + "Old DV size should be 36" + ); + assert_eq!( + old_dv.get("cardinality").and_then(|v| v.as_i64()), + Some(2), + "Old DV cardinality should be 2" + ); + + // Verify file metadata is preserved in remove action + let remove_size = remove["size"] + .as_i64() + .expect("Remove action should have size"); + let remove_partition_values = remove["partitionValues"] + .as_object() + .expect("Remove action should have partitionValues"); + let remove_tags = remove.get("tags"); + let remove_stats = remove.get("stats"); + + // Verify add action contains NEW DV information + let add_actions: Vec<_> = parsed_commits + .iter() + .filter(|action| action.get("add").is_some()) + .collect(); + + assert_eq!(add_actions.len(), 1, "Expected exactly one add action"); + + let add_action = add_actions[0]; + let add = &add_action["add"]; + + assert_eq!( + add["path"].as_str(), + Some(file_path), + "Add path should match" + ); + assert_eq!(add["dataChange"].as_bool(), Some(true)); + + // Verify NEW deletion vector in add action + let new_dv = add["deletionVector"] + .as_object() + .expect("Add action should have deletionVector"); + assert_eq!( + new_dv.get("storageType").and_then(|v| v.as_str()), + Some("u"), + "New DV storage type should be 'u'" + ); + assert_eq!( + new_dv.get("pathOrInlineDv").and_then(|v| v.as_str()), + Some("cd^-aqEH.-t@S}K{vb[*k^"), + "New DV path should match updated value" + ); + assert_eq!( + new_dv.get("offset").and_then(|v| v.as_i64()), + Some(10), + "New DV offset should be 10" + ); + assert_eq!( + new_dv.get("sizeInBytes").and_then(|v| v.as_i64()), + Some(40), + "New DV size should be 40" + ); + assert_eq!( + new_dv.get("cardinality").and_then(|v| v.as_i64()), + Some(3), + "New DV cardinality should be 3" + ); + + // Verify file metadata is preserved in add action + let add_size = add["size"].as_i64().expect("Add action should have size"); + let add_partition_values = add["partitionValues"] + .as_object() + .expect("Add action should have partitionValues"); + let add_tags = add.get("tags"); + let add_stats = add.get("stats"); + + // Ensure metadata is consistent between remove and add actions + assert_eq!( + remove_size, add_size, + "File size should be preserved between remove and add" + ); + assert_eq!( + remove_partition_values, add_partition_values, + "Partition values should be preserved between remove and add" + ); + assert_eq!( + remove_tags, add_tags, + "Tags should be preserved between remove and add" + ); + assert_eq!( + remove_stats, add_stats, + "Stats should be preserved between remove and add" + ); + + // Ensure metadata matches the original file metadata from version 1 + assert_eq!( + remove_size, original_size, + "Remove action size should match original file size" + ); + assert_eq!( + add_size, original_size, + "Add action size should match original file size" + ); + assert_eq!( + remove_partition_values, original_partition_values, + "Remove action partition values should match original" + ); + assert_eq!( + add_partition_values, original_partition_values, + "Add action partition values should match original" + ); + assert_eq!( + remove_tags, original_tags, + "Remove action tags should match original" + ); + assert_eq!( + add_tags, original_tags, + "Add action tags should match original" + ); + assert_eq!( + remove_stats, original_stats, + "Remove action stats should match original" + ); + assert_eq!( + add_stats, original_stats, + "Add action stats should match original" + ); + } + _ => panic!("Transaction should be committed"), + } + + Ok(()) +} + +#[derive(Clone)] +struct ScanFileModification { + field_name: &'static str, + value: ArrayRef, + row_id: usize, +} + +#[rstest::rstest] +#[case::missing_path( + ScanFileModification { + field_name: "path", + value: new_null_array(&ArrowDataType::Utf8, 1), + row_id: 0, + }, + "Number of matched DV files does not match number of new DV descriptors" +)] +#[case::empty_path( + ScanFileModification { + field_name: "path", + value: string_array(""), + row_id: 0, + }, + "path must not be empty" +)] +#[case::missing_partition_values( + ScanFileModification { + field_name: "partitionValues", + value: null_partition_values(), + row_id: 1, + }, + "missing required field 'partitionValues'" +)] +#[case::extra_partition_key( + ScanFileModification { + field_name: "partitionValues", + value: string_map_array(&[("stray", Some("value"))]), + row_id: 2, + }, + "partitionValues keys" +)] +#[case::duplicate_partition_key( + ScanFileModification { + field_name: "partitionValues", + value: string_map_array(&[ + ("stray", Some("first")), + ("stray", Some("second")), + ]), + row_id: 0, + }, + "duplicate partition column names" +)] +#[case::missing_size( + ScanFileModification { + field_name: "size", + value: new_null_array(&ArrowDataType::Int64, 1), + row_id: 1, + }, + "missing required field 'size'" +)] +#[case::negative_size( + ScanFileModification { + field_name: "size", + value: int64_array(-1), + row_id: 2, + }, + "size must be non-negative" +)] +#[case::missing_modification_time( + ScanFileModification { + field_name: "modificationTime", + value: new_null_array(&ArrowDataType::Int64, 1), + row_id: 0, + }, + "missing required field 'modificationTime'" +)] +#[tokio::test] +async fn test_update_deletion_vectors_rejects_corrupted_scan_files( + #[case] modification: ScanFileModification, + #[case] expected_error: &str, + #[values(0, 1, 2)] invalid_batch_index: usize, +) -> Result<(), Box> { + const BATCH_COUNT: usize = 3; + + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("part", DataType::STRING), + ])?); + let (_store, engine, table_url, file_paths) = create_dv_table_with_files( + "test_table", + schema, + &[("part", Some("value"))], + &["file0.parquet", "file1.parquet", "file2.parquet"], + ) + .await?; + + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + let scan_file = get_scan_files(snapshot.clone(), engine.as_ref())? + .into_iter() + .next() + .expect("table should contain one scan-file batch"); + let scan_files = + make_scan_file_batches(scan_file, &modification, invalid_batch_index, BATCH_COUNT); + let mut txn = begin_transaction(snapshot, engine.as_ref())?.with_data_change(true); + let mut descriptors = sequential_dv_descriptors(&file_paths); + if modification.field_name == "path" { + if modification.value.is_null(0) { + assert_result_error_with_message( + txn.update_deletion_vectors(descriptors, scan_files.into_iter().map(Ok)), + expected_error, + ); + return Ok(()); + } + let path = modification.value.as_string::().value(0); + let descriptor = descriptors + .remove(&file_paths[0]) + .expect("descriptor for the original path modified by this test"); + descriptors.insert(path.to_string(), descriptor); + } + txn.update_deletion_vectors(descriptors, scan_files.into_iter().map(Ok))?; + + assert_result_error_with_message(txn.commit(engine.as_ref()), expected_error); + Ok(()) +} + +fn make_scan_file_batches( + scan_file: FilteredEngineData, + modification: &ScanFileModification, + invalid_batch_index: usize, + batch_count: usize, +) -> Vec { + let (data, selection_vector) = scan_file.into_parts(); + let batch = into_record_batch(data); + (0..batch_count) + .map(|batch_index| { + let selection_vector = if batch_index == invalid_batch_index { + selection_vector.clone() + } else { + vec![false; batch.num_rows()] + }; + let scan_file = FilteredEngineData::try_new( + Box::new(ArrowEngineData::new(batch.clone())), + selection_vector, + ) + .expect("selection vector length should match scan-file row count"); + if batch_index == invalid_batch_index { + modify_scan_file(scan_file, modification) + } else { + scan_file + } + }) + .collect() +} + +fn modify_scan_file( + scan_file: FilteredEngineData, + modification: &ScanFileModification, +) -> FilteredEngineData { + let (data, selection_vector) = scan_file.into_parts(); + let batch = into_record_batch(data); + let schema = batch.schema(); + let mut columns = batch.columns().to_vec(); + let row_index = (0..batch.num_rows()) + .filter(|&row_index| selection_vector.get(row_index).copied().unwrap_or(true)) + .nth(modification.row_id) + .expect("modified selected row must exist in scan-file batch"); + + if modification.field_name == "partitionValues" { + let constants_index = schema + .index_of("fileConstantValues") + .expect("fileConstantValues field in scan data"); + let constants = columns[constants_index].as_struct(); + let partition_values_index = constants + .fields() + .iter() + .position(|field| field.name() == "partitionValues") + .expect("partitionValues field in fileConstantValues"); + let partition_values = constants.column(partition_values_index); + let mut constant_columns = constants.columns().to_vec(); + constant_columns[partition_values_index] = + replace_array_row(partition_values, modification.value.clone(), row_index); + columns[constants_index] = Arc::new(StructArray::new( + constants.fields().clone(), + constant_columns, + constants.nulls().cloned(), + )); + } else { + let field_index = schema + .index_of(modification.field_name) + .expect("modified field in scan data"); + columns[field_index] = + replace_array_row(&columns[field_index], modification.value.clone(), row_index); + } + + let batch = RecordBatch::try_new(schema, columns) + .expect("modified scan-file schema and columns should form a valid batch"); + FilteredEngineData::try_new(Box::new(ArrowEngineData::new(batch)), selection_vector) + .expect("selection vector length should match modified scan-file row count") +} + +fn string_array(value: &str) -> ArrayRef { + Arc::new(StringArray::from(vec![value])) +} + +fn int64_array(value: i64) -> ArrayRef { + Arc::new(Int64Array::from(vec![value])) +} + +fn null_partition_values() -> ArrayRef { + let partition_values = string_map_array(&[]); + new_null_array(partition_values.data_type(), 1) +} + +fn string_map_array(values: &[(&str, Option<&str>)]) -> ArrayRef { + let mut builder = MapBuilder::new( + Some(MapFieldNames { + entry: "key_value".to_string(), + key: "key".to_string(), + value: "value".to_string(), + }), + StringBuilder::new(), + StringBuilder::new(), + ) + .with_keys_field(ArrowField::new("key", ArrowDataType::Utf8, false)) + .with_values_field(ArrowField::new("value", ArrowDataType::Utf8, true)); + for (key, value) in values { + builder.keys().append_value(key); + match value { + Some(value) => builder.values().append_value(value), + None => builder.values().append_null(), + } + } + builder + .append(true) + .expect("partition-values row should be valid"); + Arc::new(builder.finish()) +} + +#[rstest::rstest] +#[case::unpartitioned(&[])] +#[case::partitioned(&[("value", Some("partition"))])] +#[tokio::test] +async fn test_update_deletion_vectors_multiple_files( + #[case] partition_values: &[(&str, Option<&str>)], +) -> Result<(), Box> { + // This test verifies that update_deletion_vectors can update multiple files + // in a single call, creating proper Remove and Add actions for each file. + let _ = tracing_subscriber::fmt::try_init(); + + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("value", DataType::STRING), + ])?); + + // Setup: Create table with 3 files + let file_names = &["file0.parquet", "file1.parquet", "file2.parquet"]; + let (store, engine, table_url, file_paths) = + create_dv_table_with_files("test_table", schema, partition_values, file_names).await?; + + // Create DV update transaction + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let mut txn = begin_transaction(snapshot.clone(), engine.as_ref())? + .with_engine_info("test engine") + .with_operation("UPDATE".to_string()) + .with_data_change(true); + + let mut scan_files = get_scan_files(snapshot.clone(), engine.as_ref())?; + + // Update deletion vectors for all 3 files in a single call + let dv_map = sequential_dv_descriptors(&file_paths); + + txn.update_deletion_vectors(dv_map, scan_files.drain(..).map(Ok))?; + + // Commit the transaction + let result = txn.commit(engine.as_ref())?; + + match result { + CommitResult::CommittedTransaction(committed) => { + let commit_version = committed.commit_version(); + + // Read the commit log directly from object store + let final_commit_path = + table_url.join(&format!("_delta_log/{commit_version:020}.json"))?; + let commit_content = store + .get(&Path::from_url_path(final_commit_path.path())?) + .await? + .bytes() + .await?; + + let parsed_commits: Vec<_> = Deserializer::from_slice(&commit_content) + .into_iter::() + .try_collect()?; + + // Extract all remove and add actions + let remove_actions: Vec<_> = parsed_commits + .iter() + .filter(|action| action.get("remove").is_some()) + .collect(); + + let add_actions: Vec<_> = parsed_commits + .iter() + .filter(|action| action.get("add").is_some()) + .collect(); + + // Should have 3 remove and 3 add actions + assert_eq!( + remove_actions.len(), + 3, + "Expected 3 remove actions for 3 files" + ); + assert_eq!(add_actions.len(), 3, "Expected 3 add actions for 3 files"); + + // Verify each file has a DV in both remove and add + for (idx, file_path) in file_paths.iter().enumerate() { + // Find the remove action for this file + let remove_action = remove_actions + .iter() + .find(|action| action["remove"]["path"].as_str() == Some(file_path.as_str())) + .unwrap_or_else(|| panic!("Should find remove action for {file_path}")); + + // Find the add action for this file + let add_action = add_actions + .iter() + .find(|action| action["add"]["path"].as_str() == Some(file_path.as_str())) + .unwrap_or_else(|| panic!("Should find add action for {file_path}")); + + // Verify remove action does NOT have a DV (since these were newly written files) + assert!( + remove_action["remove"]["deletionVector"].is_null(), + "Remove action for newly written file should not have a DV" + ); + + // Verify add action has the NEW DV + let add_dv = add_action["add"]["deletionVector"] + .as_object() + .expect("Add action should have deletionVector"); + + let expected_path = format!("dv_file_{idx}.bin"); + assert_eq!( + add_dv.get("pathOrInlineDv").and_then(|v| v.as_str()), + Some(expected_path.as_str()), + "DV path should match for file {file_path}" + ); + assert_eq!( + add_dv.get("offset").and_then(|v| v.as_i64()), + Some(idx as i64 * 10), + "DV offset should match for file {file_path}" + ); + assert_eq!( + add_dv.get("sizeInBytes").and_then(|v| v.as_i64()), + Some(40 + idx as i64), + "DV size should match for file {file_path}" + ); + assert_eq!( + add_dv.get("cardinality").and_then(|v| v.as_i64()), + Some(idx as i64 + 1), + "DV cardinality should match for file {file_path}" + ); + } + } + _ => panic!("Transaction should be committed"), + } + + Ok(()) +} + +#[rstest::rstest] +#[case::target_in_explicit_prefix(&[true, true], &[0], false)] +#[case::target_in_implicit_tail(&[true, true], &[3], false)] +#[case::multiple_targets(&[true, true], &[0, 3], false)] +#[case::empty_selection_vector(&[], &[3], false)] +#[case::unselected_rows_are_ignored(&[true, false, true, false], &[2], false)] +#[case::unselected_target_is_rejected(&[true, false, true, false], &[2, 3], true)] +#[case::no_updates_short_selection_vector(&[true, true], &[], false)] +#[case::no_updates_empty_selection_vector(&[], &[], false)] +#[tokio::test] +async fn test_update_deletion_vectors_respects_selection_vector( + #[case] selection_vector: &[bool], + #[case] target_indexes: &[usize], + #[case] expect_mismatch: bool, +) -> Result<(), Box> { + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("value", DataType::STRING), + ])?); + + let file_names = &[ + "file0.parquet", + "file1.parquet", + "file2.parquet", + "file3.parquet", + ]; + let (store, engine, table_url, file_paths) = + create_dv_table_with_files("test_table", schema, &[], file_names).await?; + + // Attach DV to all files first, if later DV updates incorrectly remove the existing DV, + // the test will fail. + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let mut setup_txn = begin_transaction(snapshot.clone(), engine.as_ref())?; + let initial_dvs = sequential_dv_descriptors(&file_paths); + let initial_dv_ids: HashMap<_, _> = initial_dvs + .iter() + .map(|(path, dv)| (path.clone(), dv.unique_id())) + .collect(); + setup_txn.update_deletion_vectors( + initial_dvs, + get_scan_files(snapshot, engine.as_ref())? + .into_iter() + .map(Ok), + )?; + setup_txn.commit(engine.as_ref())?.unwrap_committed(); + + let targeted: Vec = target_indexes + .iter() + .map(|&index| file_paths[index].clone()) + .collect(); + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let mut txn = begin_transaction(snapshot.clone(), engine.as_ref())? + .with_engine_info("test engine") + .with_operation("UPDATE".to_string()) + .with_data_change(true); + + let dv_map: HashMap = targeted + .iter() + .map(|path| { + ( + path.clone(), + DeletionVectorDescriptor { + storage_type: DeletionVectorStorageType::PersistedRelative, + path_or_inline_dv: format!("dv_{path}.bin"), + offset: Some(1), + size_in_bytes: 40, + cardinality: 1, + }, + ) + }) + .collect(); + let updated_dv_ids: HashMap<_, _> = dv_map + .iter() + .map(|(path, dv)| (path.clone(), dv.unique_id())) + .collect(); + + let batches = get_scan_files(snapshot, engine.as_ref())? + .into_iter() + .map(|scan_files| scan_files.apply_selection_vector().map(into_record_batch)) + .collect::, _>>()?; + let batch = concat_batches(&batches[0].schema(), &batches)?; + let scan_files = FilteredEngineData::try_new( + Box::new(ArrowEngineData::new(batch)), + selection_vector.to_vec(), + )?; + + let update_result = txn.update_deletion_vectors(dv_map, std::iter::once(Ok(scan_files))); + if expect_mismatch { + assert_result_error_with_message( + update_result, + "Number of matched DV files does not match number of new DV descriptors", + ); + } else { + update_result?; + } + let committed = txn.commit(engine.as_ref())?.unwrap_committed(); + let version = committed.commit_version(); + + // Read the commit directly from the (in-memory) store. + let commit_path = table_url.join(&format!("_delta_log/{version:020}.json"))?; + let commit_content = store + .get(&Path::from_url_path(commit_path.path())?) + .await? + .bytes() + .await?; + let actions: Vec = Deserializer::from_slice(&commit_content) + .into_iter() + .try_collect()?; + let adds: Vec<&serde_json::Value> = actions.iter().filter_map(|a| a.get("add")).collect(); + let removes: Vec<&serde_json::Value> = actions.iter().filter_map(|a| a.get("remove")).collect(); + + let expected_count = if expect_mismatch { + 0 + } else { + target_indexes.len() + }; + assert_eq!(adds.len(), expected_count, "unexpected re-added files"); + assert_eq!(removes.len(), expected_count, "unexpected removed files"); + + let mut added_paths: Vec<&str> = adds.iter().map(|a| a["path"].as_str().unwrap()).collect(); + added_paths.sort(); + let mut removed_paths: Vec<&str> = removes + .iter() + .map(|a| a["path"].as_str().unwrap()) + .collect(); + removed_paths.sort(); + let mut expected: Vec<&str> = if expect_mismatch { + Vec::new() + } else { + targeted.iter().map(String::as_str).collect() + }; + expected.sort(); + assert_eq!( + added_paths, expected, + "re-added paths must be exactly the targeted files" + ); + assert_eq!( + removed_paths, expected, + "removed paths must be exactly the targeted files" + ); + + let dv_id = |action: &serde_json::Value| { + let dv = &action["deletionVector"]; + format!( + "{}{}@{}", + dv["storageType"].as_str().unwrap(), + dv["pathOrInlineDv"].as_str().unwrap(), + dv["offset"].as_i64().unwrap() + ) + }; + for add in &adds { + let path = add["path"].as_str().unwrap(); + assert_eq!(dv_id(add), updated_dv_ids[path]); + } + for remove in &removes { + let path = remove["path"].as_str().unwrap(); + assert_eq!(dv_id(remove), initial_dv_ids[path]); + } + + // Each new add carries its DV and widened tightBounds, with numRecords preserved. + for add in &adds { + assert!( + add["deletionVector"].is_object(), + "new add must carry a deletion vector" + ); + let stats: serde_json::Value = serde_json::from_str(add["stats"].as_str().unwrap())?; + assert_eq!( + stats[TIGHT_BOUNDS].as_bool(), + Some(false), + "DV-updated add must widen tightBounds" + ); + assert_eq!( + stats[NUM_RECORDS].as_i64(), + Some(3), + "numRecords must be preserved" + ); + } + + Ok(()) +} + +#[tokio::test] +async fn test_remove_files_verify_files_excluded_from_scan( +) -> Result<(), Box> { + // Adds and then removes files and then verifies they don't appear in the scan. + + // setup tracing + let _ = tracing_subscriber::fmt::try_init(); + + // create a simple table: one int column named 'number' + let schema = get_simple_int_schema(); + + for (table_url, engine, _store, _table_name) in + setup_test_tables(schema.clone(), &[], None, "test_table").await? + { + // First, add some files to the table + let engine = Arc::new(engine); + write_data_and_check_result_and_stats(table_url.clone(), schema.clone(), engine.clone(), 1) + .await?; + + // Get initial file count + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let scan = snapshot.clone().scan_builder().build()?; + let scan_metadata = scan.scan_metadata(engine.as_ref())?.next().unwrap()?; + let (_, selection_vector) = scan_metadata.scan_files.into_parts(); + let initial_file_count = selection_vector.iter().filter(|&x| *x).count(); + + assert!(initial_file_count > 0); + + // Now create a transaction to remove files + let mut txn = begin_transaction(snapshot.clone(), engine.as_ref())?; + + // Create a new scan to get file metadata for removal + let scan2 = snapshot.scan_builder().build()?; + let scan_metadata2 = scan2.scan_metadata(engine.as_ref())?.next().unwrap()?; + + // Create FilteredEngineData for removal (select all rows for removal) + let file_remove_count = (scan_metadata2.scan_files.data().len() + - scan_metadata2.scan_files.selection_vector().len()) + + scan_metadata2 + .scan_files + .selection_vector() + .iter() + .filter(|&x| *x) + .count(); + assert!(file_remove_count > 0); + + // Add remove files to transaction + txn.remove_files(scan_metadata2.scan_files); + + // Commit the transaction + let result = txn.commit(engine.as_ref()); + + match result? { + CommitResult::CommittedTransaction(committed) => { + assert_eq!(committed.commit_version(), 2); + + let new_snapshot = Snapshot::builder_for(table_url.clone()) + .at_version(2) + .build(engine.as_ref())?; + + let new_scan = new_snapshot.scan_builder().build()?; + let mut new_file_count = 0; + for new_metadata in new_scan.scan_metadata(engine.as_ref())? { + new_file_count += new_metadata?.scan_files.data().len(); + } + + // All files were removed, so new_file_count should be zero + assert_eq!(new_file_count, 0); + } + _ => panic!("Transaction did not succeeed."), + } + } + Ok(()) +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum ExtendedMetadataField { + Size, + PartitionValues, + Tags, +} + +impl ExtendedMetadataField { + const ALL: [Self; 3] = [Self::Size, Self::PartitionValues, Self::Tags]; + + fn name(self) -> &'static str { + match self { + Self::Size => "size", + Self::PartitionValues => "partitionValues", + Self::Tags => "tags", + } + } +} + +fn with_missing_extended_metadata_fields( + engine: &dyn Engine, + scan_files: FilteredEngineData, + missing_fields: &[ExtendedMetadataField], +) -> Result> { + let (data, selection_vector) = scan_files.into_parts(); + let map_type = MapType::new(DataType::STRING, DataType::STRING, true); + let tags = if missing_fields.contains(&ExtendedMetadataField::Tags) { + Scalar::Null(DataType::from(map_type.clone())) + } else { + Scalar::Map(MapData::try_new(map_type.clone(), [("key", "value")])?) + }; + let mut patch = + ExpressionStructPatchBuilder::new().replace_at(["fileConstantValues"], "tags", lit(tags)); + for field in missing_fields { + patch = match field { + ExtendedMetadataField::Size => patch.replace("size", lit(Scalar::Null(DataType::LONG))), + ExtendedMetadataField::PartitionValues => patch.replace_at( + ["fileConstantValues"], + field.name(), + lit(Scalar::Null(DataType::from(map_type.clone()))), + ), + ExtendedMetadataField::Tags => patch, + }; + } + let schema = scan_row_schema(); + let evaluator = engine.evaluation_handler().new_expression_evaluator( + schema.clone(), + Arc::new(Expr::struct_patch(patch)?), + schema.into(), + )?; + let data = evaluator.evaluate(data.as_ref())?; + Ok(FilteredEngineData::try_new(data, selection_vector)?) +} + +#[tokio::test] +async fn test_remove_files_with_modified_selection_vector() -> Result<(), Box> +{ + // This test verifies that we can selectively remove files by: + // 1. Calling remove_files multiple times with different subsets + // 2. Modifying the selection vector to choose which files to remove + + let _ = tracing_subscriber::fmt::try_init(); + + let schema = get_simple_int_schema(); + + for (table_url, engine, _store, _table_name) in + setup_test_tables(schema.clone(), &[], None, "test_table").await? + { + let engine = Arc::new(engine); + + // Write data multiple times to create multiple files + for i in 1..=5 { + write_data_and_check_result_and_stats( + table_url.clone(), + schema.clone(), + engine.clone(), + i, + ) + .await?; + } + + // Get initial file count + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let scan = snapshot.clone().scan_builder().build()?; + + let mut initial_file_count = 0; + for metadata in scan.scan_metadata(engine.as_ref())? { + let metadata = metadata?; + initial_file_count += metadata + .scan_files + .selection_vector() + .iter() + .filter(|&x| *x) + .count(); + } + + assert!( + initial_file_count >= 3, + "Need at least 3 files for this test, got {initial_file_count}" + ); + + // Create a transaction to remove files in two batches + let mut txn = begin_transaction(snapshot.clone(), engine.as_ref())? + .with_engine_info("selective remove test") + .with_operation("DELETE".to_string()) + .with_data_change(true); + + // First batch: Remove only the first file + let scan2 = snapshot.clone().scan_builder().build()?; + let scan_metadata2 = scan2.scan_metadata(engine.as_ref())?.next().unwrap()?; + let (data, mut selection_vector) = scan_metadata2.scan_files.into_parts(); + + // Select only the first file for removal + let mut first_batch_removed = 0; + for selected in selection_vector.iter_mut() { + if *selected && first_batch_removed < 1 { + // Keep selected for removal + first_batch_removed += 1; + } else { + // Don't remove + *selected = false; + } + } + + assert_eq!( + first_batch_removed, 1, + "Should remove exactly 1 file in first batch" + ); + txn.remove_files(FilteredEngineData::try_new(data, selection_vector)?); + + // Second batch: Remove only the last file + let scan3 = snapshot.clone().scan_builder().build()?; + let scan_metadata3 = scan3.scan_metadata(engine.as_ref())?.next().unwrap()?; + let (data2, mut selection_vector2) = scan_metadata3.scan_files.into_parts(); + + // Find the last selected file and keep only that one selected + let mut last_selected_idx = None; + for (i, &selected) in selection_vector2.iter().enumerate() { + if selected { + last_selected_idx = Some(i); + } + } + + // Deselect all except the last one + for (i, selected) in selection_vector2.iter_mut().enumerate() { + if Some(i) != last_selected_idx { + *selected = false; + } + } + + let second_batch_removed = selection_vector2.iter().filter(|&x| *x).count(); + assert_eq!( + second_batch_removed, 1, + "Should remove exactly 1 file in second batch" + ); + txn.remove_files(FilteredEngineData::try_new(data2, selection_vector2)?); + + // Commit the transaction + let result = txn.commit(engine.as_ref())?; + + match result { + CommitResult::CommittedTransaction(committed) => { + assert_eq!(committed.commit_version(), 6); + + // Verify that exactly 2 files were removed (1 from each batch) + let new_snapshot = Snapshot::builder_for(table_url.clone()) + .at_version(6) + .build(engine.as_ref())?; + + let new_scan = new_snapshot.scan_builder().build()?; + let mut new_file_count = 0; + for new_metadata in new_scan.scan_metadata(engine.as_ref())? { + let metadata = new_metadata?; + new_file_count += metadata + .scan_files + .selection_vector() + .iter() + .filter(|&x| *x) + .count(); + } + + // Verify we removed exactly 2 files (1 + 1) + let total_removed = first_batch_removed + second_batch_removed; + assert_eq!(total_removed, 2); + assert_eq!(new_file_count, initial_file_count - total_removed); + assert!(new_file_count > 0, "At least one file should remain"); + } + _ => panic!("Transaction did not succeed"), + } + } + Ok(()) +} + +/// Regression test for https://github.com/delta-io/delta-kernel-rs/issues/2040 +/// +/// When `scan_metadata()` is called with a predicate, the scan row schema includes a +/// `stats_parsed` column (7th column). Passing that scan metadata to `remove_files()` then +/// `commit()` previously failed with "Too few fields in output schema" because the transform +/// evaluator exhausted the output schema when it encountered the extra column. +/// +/// Both predicate and non-predicate scans are tested because `remove_files` should behave +/// identically regardless. Cases also vary the checkpoint format: +/// - `use_struct_stats_checkpoint=false`: `stats` is non-null (raw JSON from the Add action). The +/// remove action's `stats` comes from passthrough. +/// - `use_struct_stats_checkpoint=true`: a checkpoint is written with `writeStatsAsJson=false, +/// writeStatsAsStruct=true`, so the checkpoint stores `stats_parsed` but omits the raw `stats` +/// JSON string. Scan rows from that checkpoint have `stats=null` but `stats_parsed` non-null. The +/// remove action's `stats` is produced via `coalesce(null, to_json(stats_parsed))`, which +/// exercises the coalesce path in the fix. +#[rstest::rstest] +#[case(false, false)] +#[case(false, true)] +#[case(true, false)] +#[case(true, true)] +// Multi-thread runtime required because the struct-stats checkpoint case calls +// `Snapshot::checkpoint`, which makes nested `block_on` calls that deadlock +// a single-threaded executor. `TokioMultiThreadExecutor` uses `block_in_place` +// which avoids the deadlock. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_remove_files_after_predicate_scan_includes_stats_parsed( + #[case] use_struct_stats_checkpoint: bool, + #[case] use_predicate: bool, +) -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + let schema = get_simple_int_schema(); + + // Use a local directory so we can inspect the commit log for stats content. + let tmp_dir = tempdir()?; + let tmp_url = Url::from_directory_path(tmp_dir.path()).unwrap(); + + for (table_url, engine, _store, _table_name) in + setup_test_tables(schema.clone(), &[], Some(&tmp_url), "test_table").await? + { + let engine = Arc::new(engine); + + // Write data (two parquet files with numbers [1,2,3] and [4,5,6]). + write_data_and_check_result_and_stats(table_url.clone(), schema.clone(), engine.clone(), 1) + .await?; + + // When use_struct_stats_checkpoint=true, update table properties so the checkpoint + // omits the stats JSON string (writeStatsAsJson=false) but stores stats as a struct + // (writeStatsAsStruct=true). After checkpointing, scan rows from the checkpoint have + // stats=null and stats_parsed=non-null, exercising the coalesce path in the fix. + let snapshot = if use_struct_stats_checkpoint { + let table_path = table_url.to_file_path().unwrap(); + let snapshot_v2 = set_table_properties( + table_path.to_str().unwrap(), + &table_url, + engine.as_ref(), + 1, + &[ + ("delta.checkpoint.writeStatsAsJson", "false"), + ("delta.checkpoint.writeStatsAsStruct", "true"), + ], + )?; + // `Snapshot::checkpoint` makes nested `block_on` calls internally (it reads the + // log segment lazily while writing). This requires `TokioMultiThreadExecutor`, + // which uses `block_in_place` to avoid deadlocking a single-thread runtime. + let mt_engine = create_default_engine_mt_executor(&table_url)?; + snapshot_v2.checkpoint(mt_engine.as_ref(), None)?; + Snapshot::builder_for(table_url.clone()).build(engine.as_ref())? + } else { + Snapshot::builder_for(table_url.clone()).build(engine.as_ref())? + }; + + // commit_version = 2 (no checkpoint) or 3 (properties commit + checkpoint bump) + let expected_commit_version = if use_struct_stats_checkpoint { 3 } else { 2 }; + + // Always request all stats columns so stats_parsed is present in scan metadata + // regardless of whether a predicate is used. This ensures remove_files can always + // reconstruct stats (including the coalesce path when writeStatsAsJson=false). + let mut scan_builder = snapshot + .clone() + .scan_builder() + .with_stats(StatsOptions::all()); + if use_predicate { + scan_builder = + scan_builder.with_predicate(Arc::new(Pred::gt(col!("number"), lit(0_i32)))); + } + let scan = scan_builder.build()?; + + let mut txn = begin_transaction(snapshot, engine.as_ref())?.with_data_change(true); + + // Pass scan metadata (which contains stats_parsed) directly to remove_files. + // This previously failed with "Too few fields in output schema". + for scan_metadata in scan.scan_metadata(engine.as_ref())? { + txn.remove_files(scan_metadata?.scan_files); + } + + let committed = txn.commit(engine.as_ref())?.unwrap_committed(); + assert_eq!(committed.commit_version(), expected_commit_version); + + let remove_actions = + read_actions_from_commit(&table_url, expected_commit_version, "remove")?; + assert!( + !remove_actions.is_empty(), + "expected remove actions in commit" + ); + + // stats must be populated in every remove action: stats_parsed is always present + // (via `StatsOptions::all()`), so the coalesce path handles even checkpoints + // that omit the raw JSON stats string (writeStatsAsJson=false). + for remove in &remove_actions { + let stats_str = remove["stats"] + .as_str() + .expect("stats field should be a non-null JSON string"); + let stats: serde_json::Value = serde_json::from_str(stats_str)?; + assert!( + stats[NUM_RECORDS].as_i64().unwrap_or(0) > 0, + "stats.numRecords should be populated, got: {stats}" + ); + } + } + Ok(()) +} + +/// Remove files via scan metadata on a partitioned table. Covers three predicate +/// shapes against the same table so the remove-transform correctly handles the +/// parsed scan columns in every combination: +/// - no predicate: no `partitionValues_parsed`. +/// - data-column predicate: no `partitionValues_parsed` (negative case; the fix must not affect +/// scans whose predicate misses the partition columns). +/// - partition predicate: `partitionValues_parsed` present. +/// +/// Every case sets `.with_stats(StatsOptions::all())`, which forces `stats_parsed` +/// into the scan output regardless of the predicate shape, so the partition- +/// predicate case exercises both parsed-column drop paths together while the +/// other two exercise only the `stats_parsed` drop path. The coalesce +/// *reconstruction* of `stats` from `stats_parsed` is not exercised here +/// because `stats` is non-null; the sibling +/// `test_remove_files_after_predicate_scan_includes_stats_parsed` covers that. +/// +/// `expected_partitions` is the multiset of `country` values expected across +/// the generated Remove actions. Its length gives the expected Remove count, +/// and its contents pin the correct partition was chosen (catches regressions +/// where the wrong partition is removed). +#[rstest::rstest] +#[case::no_predicate(None, &["usa", "japan"])] +#[case::data_predicate( + Some(Pred::gt(col!("id"), lit(0_i32))), + &["usa", "japan"] +)] +#[case::partition_predicate( + Some(Pred::eq(col!("country"), lit("usa".to_string()))), + &["usa"] +)] +#[tokio::test] +async fn test_remove_files_partitioned_with_parsed_columns( + #[case] predicate: Option, + #[case] expected_partitions: &[&str], +) -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + let partition_col = "country"; + let table_schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("country", DataType::STRING), + ])?); + let data_schema = schema_ref! { nullable "id": INTEGER }; + + // Local directory backing: `read_actions_from_commit` reads commit JSON off disk + // and does not support the default in-memory store's `memory://` URL. + let tmp_dir = tempdir()?; + let tmp_url = Url::from_directory_path(tmp_dir.path()).unwrap(); + + for (table_url, engine, _store, _table_name) in setup_test_tables( + table_schema.clone(), + &[partition_col], + Some(&tmp_url), + "test_table", + ) + .await? + { + let engine = Arc::new(engine); + + // Write two partitions: country="usa" and country="japan". + let mut txn = + load_and_begin_transaction(table_url.clone(), engine.as_ref())?.with_data_change(true); + let append_data = [[1, 2, 3], [10, 20, 30]].map(|data| -> delta_kernel::DeltaResult<_> { + let data = RecordBatch::try_new( + Arc::new(data_schema.as_ref().try_into_arrow()?), + vec![Arc::new(Int32Array::from(data.to_vec()))], + )?; + Ok(Box::new(ArrowEngineData::new(data))) + }); + for (data, partition_val) in append_data.into_iter().zip(["usa", "japan"]) { + let ctx = Arc::new(txn.partitioned_write_context(HashMap::from([( + partition_col.to_string(), + Scalar::String(partition_val.into()), + )]))?); + let add_meta = engine.write_parquet(data?.as_ref(), ctx.as_ref()).await?; + txn.add_files(add_meta); + } + txn.commit(engine.as_ref())?.unwrap_committed(); + + let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let mut scan_builder = snapshot + .clone() + .scan_builder() + .with_stats(StatsOptions::all()); + if let Some(pred) = predicate.clone() { + scan_builder = scan_builder.with_predicate(Arc::new(pred)); + } + let scan = scan_builder.build()?; + + let mut txn = begin_transaction(snapshot, engine.as_ref())?.with_data_change(true); + for scan_metadata in scan.scan_metadata(engine.as_ref())? { + txn.remove_files(scan_metadata?.scan_files); + } + let committed = txn.commit(engine.as_ref())?.unwrap_committed(); + assert_eq!(committed.commit_version(), 2); + + let remove_actions = read_actions_from_commit(&table_url, 2, "remove")?; + assert_eq!( + remove_actions.len(), + expected_partitions.len(), + "unexpected remove count; got {}: {remove_actions:?}", + remove_actions.len() + ); + + let mut actual_partitions: Vec = remove_actions + .iter() + .filter_map(|r| { + r["partitionValues"][partition_col] + .as_str() + .map(String::from) + }) + .collect(); + actual_partitions.sort(); + let mut expected_sorted: Vec = + expected_partitions.iter().map(|s| s.to_string()).collect(); + expected_sorted.sort(); + assert_eq!( + actual_partitions, expected_sorted, + "partitionValues mismatch across removes; got: {remove_actions:?}" + ); + + // stats_parsed is present on every scan row, so the stats-with-parsed + // evaluator is selected for every case; it must still yield a populated + // stats JSON on every remove action. + for remove in &remove_actions { + let stats_str = remove["stats"] + .as_str() + .expect("stats field should be a non-null JSON string"); + let stats: serde_json::Value = serde_json::from_str(stats_str)?; + assert!( + stats[NUM_RECORDS].as_i64().unwrap_or(0) > 0, + "stats.numRecords should be populated, got: {stats}" + ); + } + } + Ok(()) +} diff --git a/kernel/tests/integration/write/row_tracking.rs b/kernel/tests/integration/write/row_tracking.rs new file mode 100644 index 0000000000..8e0c3041ce --- /dev/null +++ b/kernel/tests/integration/write/row_tracking.rs @@ -0,0 +1,104 @@ +use std::sync::Arc; + +use delta_kernel::arrow::array::Int32Array; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::schema::schema_ref; +use delta_kernel::transaction::create_table::create_table as kernel_create_table; +use delta_kernel::Snapshot; +use test_utils::test_table_setup_mt; +use url::Url; + +use crate::common::write_utils::set_table_properties; + +/// `Transaction::commit` is rejected when it contains staged removeFiles and row +/// tracking is _supported_ and not _suspended_, which is broader than _enabled_. +#[rstest::rstest] +#[case::enabled( + &[("delta.enableRowTracking", "true")], + false, /* suspend_after_create */ + true, /* expect_err */ +)] +#[case::supported_only( + &[("delta.feature.rowTracking", "supported")], + false, /* suspend_after_create */ + true, /* expect_err */ +)] +#[case::supported_and_suspended( + &[("delta.feature.rowTracking", "supported")], + true, /* suspend_after_create */ + false, /* expect_err */ +)] +#[case::iceberg_compat_v3( + // V3 auto-enables row tracking, so the gate fires. + &[("delta.enableIcebergCompatV3", "true")], + false, /* suspend_after_create */ + true, /* expect_err */ +)] +#[tokio::test(flavor = "multi_thread")] +async fn test_row_tracking_remove_gate( + #[case] create_properties: &[(&str, &str)], + #[case] suspend_after_create: bool, + #[case] expect_err: bool, +) -> Result<(), Box> { + let (_temp_dir, table_path, engine) = test_table_setup_mt()?; + let schema = schema_ref! { nullable "number": INTEGER }; + let table_url = Url::from_directory_path(&table_path).unwrap(); + + // v0: create table with the requested initial properties. + kernel_create_table(table_path.as_str(), schema.clone(), "Test/1.0") + .with_table_properties(create_properties.iter().copied()) + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_committed(); + + // Optional v1: inject a metadata-only commit that sets `delta.rowTrackingSuspended=true`. + // kernel's create_table rejects this property at create time, so we set it via + // the integration test hack here. + let initial_snapshot = if suspend_after_create { + set_table_properties( + &table_path, + &table_url, + engine.as_ref(), + 0, /* current_version */ + &[("delta.rowTrackingSuspended", "true")], + )? + } else { + Snapshot::builder_for(&table_path).build(engine.as_ref())? + }; + + // Insert a file. + test_utils::insert_data( + initial_snapshot, + &engine, + vec![Arc::new(Int32Array::from(vec![1, 2, 3]))], + ) + .await? + .unwrap_committed(); + + // Remove the inserted file. + let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + let scan = snapshot.clone().scan_builder().build()?; + let scan_files = scan + .scan_metadata(engine.as_ref())? + .next() + .unwrap()? + .scan_files; + let mut txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? + .with_data_change(true); + txn.remove_files(scan_files); + + if expect_err { + let err = txn + .commit(engine.as_ref()) + .expect_err("commit must fail when rowTracking is supported and not suspended"); + let msg = err.to_string(); + assert!( + msg.contains("Remove actions are not yet supported") && msg.contains("rowTracking"), + "expected remove-block error mentioning rowTracking, got: {msg}", + ); + } else { + txn.commit(engine.as_ref())?.unwrap_committed(); + } + Ok(()) +} diff --git a/kernel/tests/integration/write/stats.rs b/kernel/tests/integration/write/stats.rs new file mode 100644 index 0000000000..d1997918fd --- /dev/null +++ b/kernel/tests/integration/write/stats.rs @@ -0,0 +1,413 @@ +//! Integration tests verifying stats collection for complex-typed columns. + +use std::collections::HashMap; +use std::sync::Arc; + +use delta_kernel::actions::{MAX_VALUES, MIN_VALUES, NULL_COUNT, NUM_RECORDS}; +use delta_kernel::arrow::array::{ + ArrayRef, BinaryArray, Int64Array, ListArray, MapArray, RecordBatch, StringArray, StructArray, +}; +use delta_kernel::arrow::buffer::{NullBuffer, OffsetBuffer}; +use delta_kernel::arrow::datatypes::{DataType as ArrowDataType, Schema as ArrowSchema}; +use delta_kernel::engine::arrow_conversion::TryIntoArrow as _; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::expressions::{col, lit, ColumnName}; +use delta_kernel::schema::{ArrayType, DataType, MapType, StructField, StructType}; +use delta_kernel::table_features::{get_any_level_column_physical_name, ColumnMappingMode}; +use delta_kernel::{Predicate as Pred, Snapshot}; +use test_utils::{ + create_table_and_load_snapshot, read_actions_from_commit, test_table_setup, + test_table_setup_mt, write_batch_to_table, +}; +use url::Url; + +use crate::common::write_utils::set_table_properties; + +/// Builds a RecordBatch with schema (id: long, tags: array, props: map, +/// v: variant). Each row gets one entry in tags, one entry in props, and a simple integer variant. +/// If `nulls` is provided, it sets the null buffer for tags, props, and variant columns. +fn complex_type_batch(schema: &StructType, ids: &[i64], nulls: Option<&[bool]>) -> RecordBatch { + let arrow_schema: ArrowSchema = schema.try_into_arrow().unwrap(); + let arrow_schema = Arc::new(arrow_schema); + let n = ids.len(); + + let id_array = Int64Array::from(ids.to_vec()); + + // tags: one string element per row + let tag_values = StringArray::from((0..n).map(|i| format!("t{i}")).collect::>()); + let tag_offsets = OffsetBuffer::new((0..=n).map(|i| i as i32).collect::>().into()); + let list_field = match arrow_schema.field_with_name("tags").unwrap().data_type() { + ArrowDataType::List(f) => f.clone(), + other => panic!("expected List, got {other:?}"), + }; + let tag_array = ListArray::new( + list_field, + tag_offsets, + Arc::new(tag_values), + nulls.map(NullBuffer::from), + ); + + // props: one {"k": id} entry per row + let map_keys = StringArray::from(vec!["k"; n]); + let map_vals = Int64Array::from(ids.to_vec()); + let (map_entries_field, map_sorted) = + match arrow_schema.field_with_name("props").unwrap().data_type() { + ArrowDataType::Map(f, sorted) => (f.clone(), *sorted), + other => panic!("expected Map, got {other:?}"), + }; + let entries_fields = match map_entries_field.data_type() { + ArrowDataType::Struct(f) => f.clone(), + other => panic!("expected Struct, got {other:?}"), + }; + let entries = StructArray::new( + entries_fields, + vec![ + Arc::new(map_keys) as ArrayRef, + Arc::new(map_vals) as ArrayRef, + ], + None, + ); + let map_offsets = OffsetBuffer::new((0..=n).map(|i| i as i32).collect::>().into()); + let map_array = MapArray::new( + map_entries_field, + map_offsets, + entries, + nulls.map(NullBuffer::from), + map_sorted, + ); + + // v: simple integer variant per row (metadata=[0x01,0x00,0x00], value=[0x0C, low_byte]) + let variant_meta = BinaryArray::from( + ids.iter() + .map(|_| Some(&[0x01u8, 0x00, 0x00][..])) + .collect::>(), + ); + let variant_val_data: Vec<[u8; 2]> = ids.iter().map(|&id| [0x0Cu8, id as u8]).collect(); + let variant_val = BinaryArray::from( + variant_val_data + .iter() + .map(|v| Some(&v[..])) + .collect::>(), + ); + let variant_fields = match arrow_schema.field_with_name("v").unwrap().data_type() { + ArrowDataType::Struct(fields) => fields.clone(), + other => panic!("expected Struct, got {other:?}"), + }; + let variant_array = StructArray::new( + variant_fields, + vec![ + Arc::new(variant_meta) as ArrayRef, + Arc::new(variant_val) as ArrayRef, + ], + nulls.map(NullBuffer::from), + ); + + RecordBatch::try_new( + arrow_schema, + vec![ + Arc::new(id_array) as ArrayRef, + Arc::new(tag_array) as ArrayRef, + Arc::new(map_array) as ArrayRef, + Arc::new(variant_array) as ArrayRef, + ], + ) + .unwrap() +} + +/// Verifies that writing a table with array, map, and variant columns produces nullCount +/// statistics for those columns while excluding them from minValues/maxValues. Then writes a +/// second file and scans with a predicate to verify data skipping works e2e. Parameterized +/// over column mapping mode and whether a checkpoint is taken before the scan. +#[rstest::rstest] +#[tokio::test(flavor = "multi_thread")] +async fn test_write_stats_for_complex_type_columns( + #[values( + ColumnMappingMode::None, + ColumnMappingMode::Id, + ColumnMappingMode::Name + )] + cm_mode: ColumnMappingMode, + #[values(false, true)] use_checkpoint: bool, +) -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::LONG), + StructField::nullable("tags", ArrayType::new(DataType::STRING, true)), + StructField::nullable( + "props", + MapType::new(DataType::STRING, DataType::LONG, true), + ), + StructField::nullable("v", DataType::unshredded_variant()), + ])?); + + let mode_str = match cm_mode { + ColumnMappingMode::None => "none", + ColumnMappingMode::Id => "id", + ColumnMappingMode::Name => "name", + }; + let (_tmp_dir, table_path, engine) = test_table_setup_mt()?; + let snapshot = create_table_and_load_snapshot( + &table_path, + schema.clone(), + engine.as_ref(), + &[("delta.columnMapping.mode", mode_str)], + )?; + let table_url = Url::from_directory_path(&table_path).unwrap(); + + // Resolve physical column names for stats JSON verification + let cm = snapshot + .table_properties() + .column_mapping_mode + .unwrap_or(ColumnMappingMode::None); + let physical_name = |logical: &str| -> String { + get_any_level_column_physical_name( + snapshot.schema().as_ref(), + &ColumnName::new([logical]), + cm, + ) + .unwrap() + .into_inner() + .into_iter() + .next() + .unwrap() + }; + let id_phys = physical_name("id"); + let tags_phys = physical_name("tags"); + let props_phys = physical_name("props"); + let v_phys = physical_name("v"); + + // Batch 1: ids [1,2,3] with one null per complex column + let batch1 = complex_type_batch(&schema, &[1, 2, 3], Some(&[true, false, true])); + let _snapshot = + write_batch_to_table(&snapshot, engine.as_ref(), batch1, HashMap::new()).await?; + + // Read the commit and verify stats use correct (physical) column names + let add_actions = read_actions_from_commit(&table_url, 1, "add")?; + assert_eq!(add_actions.len(), 1); + + let stats: serde_json::Value = serde_json::from_str( + add_actions[0] + .get("stats") + .and_then(|s| s.as_str()) + .expect("add action should have stats"), + )?; + + assert_eq!(stats[NUM_RECORDS], 3); + + // nullCount should be present for all columns including array, map, and variant + assert_eq!(stats[NULL_COUNT][&id_phys], 0); + assert_eq!(stats[NULL_COUNT][&tags_phys], 1); + assert_eq!(stats[NULL_COUNT][&props_phys], 1); + assert_eq!(stats[NULL_COUNT][&v_phys], 1); + + // minValues/maxValues should have id but NOT complex types + assert!(stats[MIN_VALUES][&id_phys].is_number()); + assert!(stats[MAX_VALUES][&id_phys].is_number()); + for col in [&tags_phys, &props_phys, &v_phys] { + assert!( + stats[MIN_VALUES].get(col).is_none(), + "minValues should not contain {col}" + ); + assert!( + stats[MAX_VALUES].get(col).is_none(), + "maxValues should not contain {col}" + ); + } + + // Batch 2: ids [10,11,12] with no nulls, written to a separate file + let batch2 = complex_type_batch(&schema, &[10, 11, 12], None); + let snapshot2 = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; + let snapshot2 = + write_batch_to_table(&snapshot2, engine.as_ref(), batch2, HashMap::new()).await?; + + // Optionally checkpoint to verify stats survive the checkpoint round-trip + let scan_snapshot = if use_checkpoint { + snapshot2.checkpoint(engine.as_ref(), None)?; + Snapshot::builder_for(table_url.clone()).build(engine.as_ref())? + } else { + snapshot2 + }; + + // Scan with predicate id > 5. File 1 has id in [1,3], file 2 has id in [10,12]. + // Data skipping should skip file 1 entirely and return only file 2's rows. + let scan = scan_snapshot + .scan_builder() + .with_predicate(Arc::new(Pred::gt(col!("id"), lit(5_i64)))) + .build()?; + let batches: Vec = scan + .execute(engine.clone())? + .map(|r| { + let data = r.unwrap(); + ArrowEngineData::try_from_engine_data(data) + .unwrap() + .record_batch() + .clone() + }) + .collect(); + + let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!( + total_rows, 3, + "predicate id > 5 should return only the second file's 3 rows" + ); + + let result_schema = batches[0].schema(); + let combined = delta_kernel::arrow::compute::concat_batches(&result_schema, &batches)?; + let ids: Vec = combined + .column_by_name("id") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap() + .values() + .iter() + .copied() + .collect(); + assert_eq!(ids, vec![10, 11, 12]); + + Ok(()) +} + +/// Verifies that complex types in a nested schema count against `dataSkippingNumIndexedCols`. +/// Schema: (id: long, data: struct, props: map>). +/// With numIndexedCols=3, the first 3 leaf columns (id, data.name, data.tags) get stats. The +/// 4th leaf (data.props) is excluded because the limit is reached. +#[tokio::test] +async fn test_write_stats_nested_complex_types_respect_column_limit( +) -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + let schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("id", DataType::LONG), + StructField::nullable( + "data", + DataType::try_struct_type(vec![ + StructField::nullable("name", DataType::STRING), + StructField::nullable("tags", ArrayType::new(DataType::STRING, true)), + StructField::nullable( + "props", + MapType::new(DataType::STRING, DataType::LONG, true), + ), + ])?, + ), + ])?); + + let (_tmp_dir, table_path, engine) = test_table_setup()?; + let table_url = Url::from_directory_path(&table_path).unwrap(); + let snapshot = + create_table_and_load_snapshot(&table_path, schema.clone(), engine.as_ref(), &[])?; + let snapshot = set_table_properties( + &table_path, + &table_url, + engine.as_ref(), + snapshot.version(), + &[("delta.dataSkippingNumIndexedCols", "3")], + )?; + + // Build a batch with 3 rows + let arrow_schema: ArrowSchema = schema.as_ref().try_into_arrow()?; + let arrow_schema = Arc::new(arrow_schema); + + let id_array = Int64Array::from(vec![1, 2, 3]); + let name_array = StringArray::from(vec!["a", "b", "c"]); + + // tags: [["x"], null, ["y"]] + let data_field = arrow_schema.field_with_name("data").unwrap(); + let data_struct_fields = match data_field.data_type() { + ArrowDataType::Struct(f) => f, + other => panic!("expected Struct, got {other:?}"), + }; + let tags_field = &data_struct_fields[1]; + let list_field = match tags_field.data_type() { + ArrowDataType::List(f) => f.clone(), + other => panic!("expected List, got {other:?}"), + }; + let tag_values = StringArray::from(vec!["x", "y"]); + let tag_offsets = OffsetBuffer::new(vec![0, 1, 1, 2].into()); + let tag_array = ListArray::new( + list_field, + tag_offsets, + Arc::new(tag_values), + Some(NullBuffer::from_iter([true, false, true])), + ); + + // props: [{"k": 1}, {"k": 2}, null] + let props_field = &data_struct_fields[2]; + let (map_entries_field, map_sorted) = match props_field.data_type() { + ArrowDataType::Map(f, sorted) => (f.clone(), *sorted), + other => panic!("expected Map, got {other:?}"), + }; + let entries_fields = match map_entries_field.data_type() { + ArrowDataType::Struct(f) => f.clone(), + other => panic!("expected Struct, got {other:?}"), + }; + let map_keys = StringArray::from(vec!["k", "k"]); + let map_vals = Int64Array::from(vec![1i64, 2]); + let entries = StructArray::new( + entries_fields, + vec![ + Arc::new(map_keys) as ArrayRef, + Arc::new(map_vals) as ArrayRef, + ], + None, + ); + let map_offsets = OffsetBuffer::new(vec![0, 1, 2, 2].into()); + let map_array = MapArray::new( + map_entries_field, + map_offsets, + entries, + Some(NullBuffer::from_iter([true, true, false])), + map_sorted, + ); + + // Assemble the data struct + let data_array = StructArray::new( + data_struct_fields.clone(), + vec![ + Arc::new(name_array) as ArrayRef, + Arc::new(tag_array) as ArrayRef, + Arc::new(map_array) as ArrayRef, + ], + None, + ); + + let batch = RecordBatch::try_new( + arrow_schema, + vec![ + Arc::new(id_array) as ArrayRef, + Arc::new(data_array) as ArrayRef, + ], + )?; + + let _snapshot = write_batch_to_table(&snapshot, engine.as_ref(), batch, HashMap::new()).await?; + + // Version 0=create, 1=set properties, 2=data write + let add_actions = read_actions_from_commit(&table_url, 2, "add")?; + let stats: serde_json::Value = serde_json::from_str( + add_actions[0] + .get("stats") + .and_then(|s| s.as_str()) + .expect("add action should have stats"), + )?; + + assert_eq!(stats[NUM_RECORDS], 3); + + // First 3 leaves: id, data.name, data.tags all get nullCount + assert_eq!(stats[NULL_COUNT]["id"], 0); + assert_eq!(stats[NULL_COUNT]["data"]["name"], 0); + assert_eq!(stats[NULL_COUNT]["data"]["tags"], 1); + + // 4th leaf data.props is excluded by the column limit + assert!( + stats[NULL_COUNT]["data"].get("props").is_none(), + "props should be excluded by numIndexedCols=3" + ); + + // id and data.name get min/max; data.tags does not (complex type) + assert!(stats[MIN_VALUES]["id"].is_number()); + assert!(stats[MIN_VALUES]["data"]["name"].is_string()); + assert!(stats[MIN_VALUES]["data"].get("tags").is_none()); + + Ok(()) +} diff --git a/kernel/tests/integration/write/txn.rs b/kernel/tests/integration/write/txn.rs new file mode 100644 index 0000000000..0338612a8b --- /dev/null +++ b/kernel/tests/integration/write/txn.rs @@ -0,0 +1,130 @@ +//! Integration tests for transaction-identifier write paths. + +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::ObjectStoreExt as _; +use delta_kernel::{Error as KernelError, Snapshot}; +use itertools::Itertools; +use serde_json::{json, Deserializer}; +use test_utils::{load_and_begin_transaction, set_json_value, setup_test_tables}; + +use crate::common::write_utils::{get_simple_int_schema, validate_txn_id, ZERO_UUID}; + +#[tokio::test] +async fn test_write_txn_actions() -> Result<(), Box> { + // setup tracing + let _ = tracing_subscriber::fmt::try_init(); + + // create a simple table: one int column named 'number' + let schema = get_simple_int_schema(); + + for (table_url, engine, store, table_name) in + setup_test_tables(schema, &[], None, "test_table").await? + { + // can't have duplicate app_id in same transaction + assert!(matches!( + load_and_begin_transaction(table_url.clone(), &engine)? + .with_transaction_id("app_id1".to_string(), 0) + .with_transaction_id("app_id1".to_string(), 1) + .commit(&engine), + Err(KernelError::Generic(msg)) if msg == "app_id app_id1 already exists in transaction" + )); + + let txn = load_and_begin_transaction(table_url.clone(), &engine)? + .with_engine_info("default engine") + .with_transaction_id("app_id1".to_string(), 1) + .with_transaction_id("app_id2".to_string(), 2); + + // commit! + assert!(txn.commit(&engine)?.is_committed()); + + let snapshot = Snapshot::builder_for(table_url.clone()) + .at_version(1) + .build(&engine)?; + assert_eq!(snapshot.get_app_id_version("app_id1", &engine)?, Some(1)); + assert_eq!(snapshot.get_app_id_version("app_id2", &engine)?, Some(2)); + assert_eq!(snapshot.get_app_id_version("app_id3", &engine)?, None); + + let commit1 = store + .get(&Path::from(format!( + "/{table_name}/_delta_log/00000000000000000001.json" + ))) + .await?; + + let mut parsed_commits: Vec<_> = Deserializer::from_slice(&commit1.bytes().await?) + .into_iter::() + .try_collect()?; + + set_json_value(&mut parsed_commits[0], "commitInfo.timestamp", json!(0)).unwrap(); + + let time_ms: i64 = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_millis() + .try_into() + .unwrap(); + + // check that last_updated times are identical + let last_updated1 = parsed_commits[1] + .get("txn") + .unwrap() + .get("lastUpdated") + .unwrap(); + let last_updated2 = parsed_commits[2] + .get("txn") + .unwrap() + .get("lastUpdated") + .unwrap(); + assert_eq!(last_updated1, last_updated2); + + let last_updated = parsed_commits[1] + .get_mut("txn") + .unwrap() + .get_mut("lastUpdated") + .unwrap(); + // sanity check that last_updated time is within 10s of now + assert!((last_updated.as_i64().unwrap() - time_ms).abs() < 10_000); + *last_updated = serde_json::Value::Number(1.into()); + + let last_updated = parsed_commits[2] + .get_mut("txn") + .unwrap() + .get_mut("lastUpdated") + .unwrap(); + // sanity check that last_updated time is within 10s of now + assert!((last_updated.as_i64().unwrap() - time_ms).abs() < 10_000); + *last_updated = serde_json::Value::Number(2.into()); + + validate_txn_id(&parsed_commits[0]["commitInfo"]); + + set_json_value(&mut parsed_commits[0], "commitInfo.txnId", json!(ZERO_UUID))?; + + let expected_commit = vec![ + json!({ + "commitInfo": { + "timestamp": 0, + "operation": "UNKNOWN", + "kernelVersion": format!("v{}", env!("CARGO_PKG_VERSION")), + "operationParameters": {}, + "engineInfo": "default engine", + "txnId": ZERO_UUID + } + }), + json!({ + "txn": { + "appId": "app_id1", + "version": 1, + "lastUpdated": 1 + } + }), + json!({ + "txn": { + "appId": "app_id2", + "version": 2, + "lastUpdated": 2 + } + }), + ]; + + assert_eq!(parsed_commits, expected_commit); + } + Ok(()) +} diff --git a/kernel/tests/integration/write/types.rs b/kernel/tests/integration/write/types.rs new file mode 100644 index 0000000000..e30b79b143 --- /dev/null +++ b/kernel/tests/integration/write/types.rs @@ -0,0 +1,1010 @@ +//! Integration tests for type-specific writes (timestampNtz, variant, shredded variant). + +use std::collections::HashMap; +use std::sync::Arc; + +use delta_kernel::arrow::array::{ + new_null_array, ArrayRef, BinaryArray, Int32Array, StructArray, TimestampMicrosecondArray, +}; +use delta_kernel::arrow::buffer::NullBuffer; +use delta_kernel::arrow::datatypes::{DataType as ArrowDataType, Field}; +use delta_kernel::arrow::record_batch::RecordBatch; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::engine::arrow_conversion::{TryFromKernel, TryIntoArrow as _}; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::expressions::Scalar; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::ObjectStoreExt as _; +use delta_kernel::schema::{ArrayType, DataType, MapType, SchemaRef, StructField, StructType}; +use delta_kernel::transaction::create_table::create_table as kernel_create_table; +use delta_kernel::{Error as KernelError, Snapshot}; +use itertools::Itertools; +use rstest::rstest; +use serde_json::Deserializer; +use tempfile::tempdir; +use test_utils::{ + begin_transaction, create_add_files_metadata, create_table, engine_store_setup, test_read, + test_table_setup, +}; +use url::Url; + +#[tokio::test] +async fn test_append_timestamp_ntz() -> Result<(), Box> { + // setup tracing + let _ = tracing_subscriber::fmt::try_init(); + + // create a table with TIMESTAMP_NTZ column + let schema = Arc::new(StructType::try_new(vec![StructField::nullable( + "ts_ntz", + DataType::TIMESTAMP_NTZ, + )])?); + + let (store, engine, table_location) = engine_store_setup("test_table_timestamp_ntz", None); + let table_url = create_table( + store.clone(), + table_location, + schema.clone(), + &[], + true, + vec!["timestampNtz"], + vec!["timestampNtz"], + ) + .await?; + + let mut txn = test_utils::load_and_begin_transaction(table_url.clone(), &engine)? + .with_engine_info("default engine"); + + // Create Arrow data with TIMESTAMP_NTZ values including edge cases + // These are microseconds since Unix epoch + let timestamp_values = vec![ + 0i64, // Unix epoch (1970-01-01T00:00:00.000000) + 1634567890123456i64, // 2021-10-18T12:31:30.123456 + 1634567950654321i64, // 2021-10-18T12:32:30.654321 + 1672531200000000i64, // 2023-01-01T00:00:00.000000 + 253402300799999999i64, // 9999-12-31T23:59:59.999999 (near max valid timestamp) + -62135596800000000i64, // 0001-01-01T00:00:00.000000 (near min valid timestamp) + ]; + + let data = RecordBatch::try_new( + Arc::new(schema.as_ref().try_into_arrow()?), + vec![Arc::new(TimestampMicrosecondArray::from(timestamp_values))], + )?; + + // Write data + let engine = Arc::new(engine); + let write_context = Arc::new(txn.unpartitioned_write_context().unwrap()); + + let add_files_metadata = engine + .write_parquet(&ArrowEngineData::new(data.clone()), write_context.as_ref()) + .await?; + + txn.add_files(add_files_metadata); + + // Commit the transaction + assert!(txn.commit(engine.as_ref())?.is_committed()); + + // Verify the commit was written correctly + let commit1 = store + .get(&Path::from( + "/test_table_timestamp_ntz/_delta_log/00000000000000000001.json", + )) + .await?; + + let parsed_commits: Vec<_> = Deserializer::from_slice(&commit1.bytes().await?) + .into_iter::() + .try_collect()?; + + // Check that we have the expected number of commits (commitInfo + add) + assert_eq!(parsed_commits.len(), 2); + + // Check that the add action exists + assert!(parsed_commits[1].get("add").is_some()); + // Ensure default of data change is true. + assert!(parsed_commits[1] + .get("add") + .unwrap() + .get("dataChange") + .unwrap() + .as_bool() + .unwrap()); + + let stats: serde_json::Value = + serde_json::from_str(parsed_commits[1]["add"]["stats"].as_str().unwrap())?; + assert_eq!(stats["minValues"]["ts_ntz"], "0001-01-01T00:00:00.000"); + assert_eq!(stats["maxValues"]["ts_ntz"], "9999-12-31T23:59:59.999"); + + // Verify the data can be read back correctly + test_read(&ArrowEngineData::new(data), &table_url, engine)?; + + Ok(()) +} + +#[tokio::test] +async fn test_append_timestamp_stats_are_millisecond_truncated( +) -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + let schema = Arc::new(StructType::try_new(vec![StructField::nullable( + "ts", + DataType::TIMESTAMP, + )])?); + + let (store, engine, table_location) = engine_store_setup("test_table_timestamp_stats", None); + let table_url = create_table( + store.clone(), + table_location, + schema.clone(), + &[], + true, + vec![], + vec![], + ) + .await?; + + let mut txn = test_utils::load_and_begin_transaction(table_url.clone(), &engine)? + .with_engine_info("default engine"); + + // Spans [.298677, .307735]; a conforming writer floors the stats to [.298, .307]. + let timestamp_values = vec![1_783_007_755_298_677i64, 1_783_007_755_307_735i64]; + let data = RecordBatch::try_new( + Arc::new(schema.as_ref().try_into_arrow()?), + vec![Arc::new( + TimestampMicrosecondArray::from(timestamp_values).with_timezone("UTC"), + )], + )?; + + let engine = Arc::new(engine); + let write_context = Arc::new(txn.unpartitioned_write_context().unwrap()); + let add_files_metadata = engine + .write_parquet(&ArrowEngineData::new(data.clone()), write_context.as_ref()) + .await?; + txn.add_files(add_files_metadata); + assert!(txn.commit(engine.as_ref())?.is_committed()); + + let commit1 = store + .get(&Path::from( + "/test_table_timestamp_stats/_delta_log/00000000000000000001.json", + )) + .await?; + let parsed_commits: Vec<_> = Deserializer::from_slice(&commit1.bytes().await?) + .into_iter::() + .try_collect()?; + + let stats: serde_json::Value = + serde_json::from_str(parsed_commits[1]["add"]["stats"].as_str().unwrap())?; + assert_eq!(stats["minValues"]["ts"], "2026-07-02T15:55:55.298Z"); + assert_eq!(stats["maxValues"]["ts"], "2026-07-02T15:55:55.307Z"); + + // Kernel must be able to parse the stats it just wrote. + test_read(&ArrowEngineData::new(data), &table_url, engine)?; + + Ok(()) +} + +#[rstest] +#[case::shredding_preview(vec!["variantType", "variantShredding-preview"])] +#[case::shredding(vec!["variantType", "variantShredding"])] +#[tokio::test] +async fn test_append_variant( + #[case] features: Vec<&str>, +) -> Result<(), Box> { + // setup tracing + let _ = tracing_subscriber::fmt::try_init(); + fn unshredded_variant_schema_flipped() -> DataType { + DataType::variant_type([ + StructField::not_null("value", DataType::BINARY), + StructField::not_null("metadata", DataType::BINARY), + ]) + .unwrap() + } + fn variant_arrow_type_flipped() -> ArrowDataType { + let metadata_field = Field::new("metadata", ArrowDataType::Binary, false); + let value_field = Field::new("value", ArrowDataType::Binary, false); + let fields = vec![value_field, metadata_field]; + ArrowDataType::Struct(fields.into()) + } + + // create a table with VARIANT column + let table_schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("v", DataType::unshredded_variant()), + StructField::nullable("i", DataType::INTEGER), + StructField::nullable( + "nested", + // We flip the value and metadata fields in the actual parquet file for the test + StructType::try_new(vec![StructField::nullable( + "nested_v", + unshredded_variant_schema_flipped(), + )])?, + ), + ])?); + + let write_schema = table_schema.clone(); + + let tmp_test_dir = tempdir()?; + let tmp_test_dir_url = Url::from_directory_path(tmp_test_dir.path()).unwrap(); + + let (store, engine, table_location) = + engine_store_setup("test_table_variant", Some(&tmp_test_dir_url)); + + // We can add shredding features as well as we are allowed to write unshredded variants + // into shredded tables and shredded reads are explicitly blocked in the default + // engine's parquet reader. + let table_url = create_table( + store.clone(), + table_location, + table_schema.clone(), + &[], + true, + features.clone(), + features, + ) + .await?; + + let mut txn = + test_utils::load_and_begin_transaction(table_url.clone(), &engine)?.with_data_change(true); + + // First value corresponds to the variant value "1". Third value corresponds to the variant + // representing the JSON Object {"a":2}. + let metadata_v = vec![ + Some(&[0x01, 0x00, 0x00][..]), + None, + Some(&[0x01, 0x01, 0x00, 0x01, 0x61][..]), + ]; + let value_v = vec![ + Some(&[0x0C, 0x01][..]), + None, + Some(&[0x02, 0x01, 0x00, 0x00, 0x01, 0x02][..]), + ]; + + let metadata_v_array = Arc::new(BinaryArray::from(metadata_v)) as ArrayRef; + let value_v_array = Arc::new(BinaryArray::from(value_v)) as ArrayRef; + + // First value corresponds to the variant value "2". Third value corresponds to the variant + // representing the JSON Object {"b":3}. + let metadata_nested_v = vec![ + Some(&[0x01, 0x00, 0x00][..]), + None, + Some(&[0x01, 0x01, 0x00, 0x01, 0x62][..]), + ]; + let value_nested_v = vec![ + Some(&[0x0C, 0x02][..]), + None, + Some(&[0x02, 0x01, 0x00, 0x00, 0x01, 0x03][..]), + ]; + + let value_nested_v_array = Arc::new(BinaryArray::from(value_nested_v)) as ArrayRef; + let metadata_nested_v_array = Arc::new(BinaryArray::from(metadata_nested_v)) as ArrayRef; + + let variant_arrow = ArrowDataType::try_from_kernel(&DataType::unshredded_variant()).unwrap(); + let variant_arrow_flipped = variant_arrow_type_flipped(); + + let i_values = vec![31, 32, 33]; + + let fields = match variant_arrow { + ArrowDataType::Struct(fields) => Ok(fields), + _ => Err(KernelError::Generic( + "Variant arrow data type is not struct.".to_string(), + )), + }?; + let fields_flipped = match variant_arrow_flipped { + ArrowDataType::Struct(fields) => Ok(fields), + _ => Err(KernelError::Generic( + "Variant arrow data type is not struct.".to_string(), + )), + }?; + + let null_bitmap = NullBuffer::from_iter([true, false, true]); + + let variant_v_array = StructArray::try_new( + fields.clone(), + vec![metadata_v_array, value_v_array], + Some(null_bitmap.clone()), + )?; + + let variant_nested_v_array = Arc::new(StructArray::try_new( + fields_flipped.clone(), + vec![ + value_nested_v_array.clone(), + metadata_nested_v_array.clone(), + ], + Some(null_bitmap.clone()), + )?); + + let data = RecordBatch::try_new( + Arc::new(write_schema.as_ref().try_into_arrow()?), + vec![ + // v variant + Arc::new(variant_v_array.clone()), + // i int + Arc::new(Int32Array::from(i_values.clone())), + // nested struct + Arc::new(StructArray::try_new( + vec![Field::new("nested_v", variant_arrow_type_flipped(), true)].into(), + vec![variant_nested_v_array.clone()], + None, + )?), + ], + ) + .unwrap(); + + // Write data + let engine = Arc::new(engine); + let write_context = Arc::new(txn.unpartitioned_write_context().unwrap()); + + let add_files_metadata = (*engine) + .default_parquet_handler() + .write_parquet_file(Box::new(ArrowEngineData::new(data.clone())), &write_context) + .await?; + + txn.add_files(add_files_metadata); + + // Commit the transaction + assert!(txn.commit(engine.as_ref())?.is_committed()); + + // Verify the commit was written correctly + let commit1_url = tmp_test_dir_url + .join("test_table_variant/_delta_log/00000000000000000001.json") + .unwrap(); + let commit1 = store + .get(&Path::from_url_path(commit1_url.path()).unwrap()) + .await?; + + let parsed_commits: Vec<_> = Deserializer::from_slice(&commit1.bytes().await?) + .into_iter::() + .try_collect()?; + + // Check that we have the expected number of commits (commitInfo + add) + assert_eq!(parsed_commits.len(), 2); + + // Check that the add action exists + assert!(parsed_commits[1].get("add").is_some()); + + // The scanned data will match the logical schema, not the physical one + let expected_schema = Arc::new(StructType::try_new(vec![ + StructField::nullable("v", DataType::unshredded_variant()), + StructField::nullable("i", DataType::INTEGER), + StructField::nullable( + "nested", + StructType::try_new(vec![StructField::nullable( + "nested_v", + DataType::unshredded_variant(), + )]) + .unwrap(), + ), + ])?); + + // During the read, the flipped fields should be reordered into metadata, value. + let variant_nested_v_array_expected = Arc::new(StructArray::try_new( + fields, + vec![metadata_nested_v_array, value_nested_v_array], + Some(null_bitmap), + )?); + let variant_arrow_type: ArrowDataType = + ArrowDataType::try_from_kernel(&DataType::unshredded_variant()).unwrap(); + let expected_data = RecordBatch::try_new( + Arc::new(expected_schema.as_ref().try_into_arrow()?), + vec![ + // v variant + Arc::new(variant_v_array), + // i int + Arc::new(Int32Array::from(i_values)), + // nested struct + Arc::new(StructArray::try_new( + vec![Field::new("nested_v", variant_arrow_type, true)].into(), + vec![variant_nested_v_array_expected], + None, + )?), + ], + ) + .unwrap(); + + test_read(&ArrowEngineData::new(expected_data), &table_url, engine)?; + + Ok(()) +} + +#[tokio::test] +async fn test_shredded_variant_read_rejection() -> Result<(), Box> { + // Ensure that shredded variants are rejected by the default engine's parquet reader + + // setup tracing + let _ = tracing_subscriber::fmt::try_init(); + let table_schema = Arc::new(StructType::try_new(vec![StructField::nullable( + "v", + DataType::unshredded_variant(), + )])?); + + // The table will be attempted to be written in this form but be read into + // STRUCT. The read should fail because the default engine + // currently does not support shredded reads. + let shredded_write_schema = Arc::new(StructType::try_new(vec![StructField::nullable( + "v", + DataType::try_struct_type([ + StructField::new("metadata", DataType::BINARY, true), + StructField::new("value", DataType::BINARY, true), + StructField::new("typed_value", DataType::INTEGER, true), + ])?, + )])?); + + let tmp_test_dir = tempdir()?; + let tmp_test_dir_url = Url::from_directory_path(tmp_test_dir.path()).unwrap(); + + let (store, engine, table_location) = + engine_store_setup("test_table_variant_2", Some(&tmp_test_dir_url)); + let table_url = create_table( + store.clone(), + table_location, + table_schema.clone(), + &[], + true, + vec!["variantType", "variantShredding-preview"], + vec!["variantType", "variantShredding-preview"], + ) + .await?; + + let mut txn = + test_utils::load_and_begin_transaction(table_url.clone(), &engine)?.with_data_change(true); + + // First value corresponds to the variant value "1". Third value corresponds to the variant + // representing the JSON Object {"a":2}. + let metadata_v = vec![ + Some(&[0x01, 0x00, 0x00][..]), + Some(&[0x01, 0x01, 0x00, 0x01, 0x61][..]), + ]; + let value_v = vec![ + Some(&[0x0C, 0x01][..]), + Some(&[0x02, 0x01, 0x00, 0x00, 0x01, 0x02][..]), + ]; + let typed_value_v = vec![Some(21), Some(3)]; + + let metadata_v_array = Arc::new(BinaryArray::from(metadata_v)) as ArrayRef; + let value_v_array = Arc::new(BinaryArray::from(value_v)) as ArrayRef; + let typed_value_v_array = Arc::new(Int32Array::from(typed_value_v)) as ArrayRef; + + let variant_arrow = ArrowDataType::Struct( + vec![ + Field::new("metadata", ArrowDataType::Binary, true), + Field::new("value", ArrowDataType::Binary, true), + Field::new("typed_value", ArrowDataType::Int32, true), + ] + .into(), + ); + + let fields = match variant_arrow { + ArrowDataType::Struct(fields) => Ok(fields), + _ => Err(KernelError::Generic( + "Variant arrow data type is not struct.".to_string(), + )), + }?; + + let variant_v_array = StructArray::try_new( + fields.clone(), + vec![metadata_v_array, value_v_array, typed_value_v_array], + None, + )?; + + let data = RecordBatch::try_new( + Arc::new(shredded_write_schema.as_ref().try_into_arrow()?), + vec![ + // v variant + Arc::new(variant_v_array.clone()), + ], + ) + .unwrap(); + + let engine = Arc::new(engine); + let write_context = Arc::new(txn.unpartitioned_write_context().unwrap()); + + let add_files_metadata = (*engine) + .default_parquet_handler() + .write_parquet_file(Box::new(ArrowEngineData::new(data.clone())), &write_context) + .await?; + + txn.add_files(add_files_metadata); + + // Commit the transaction + assert!(txn.commit(engine.as_ref())?.is_committed()); + + // Verify the commit was written correctly + let commit1_url = tmp_test_dir_url + .join("test_table_variant_2/_delta_log/00000000000000000001.json") + .unwrap(); + let commit1 = store + .get(&Path::from_url_path(commit1_url.path()).unwrap()) + .await?; + + let parsed_commits: Vec<_> = Deserializer::from_slice(&commit1.bytes().await?) + .into_iter::() + .try_collect()?; + + // Check that we have the expected number of commits (commitInfo + add) + assert_eq!(parsed_commits.len(), 2); + + // Check that the add action exists + assert!(parsed_commits[1].get("add").is_some()); + + let res = test_read(&ArrowEngineData::new(data), &table_url, engine); + assert!(matches!(res, + Err(e) if e.to_string().contains("The default engine does not support shredded reads"))); + + Ok(()) +} + +// ============================================================================= +// NOT NULL data column tests (default engine) +// ============================================================================= +// +// These tests validate data-column nullability on the default engine. Kernel preserves +// `nullable: false` in the connector-facing schema and relies on the engine to reject +// nulls before writing. + +/// Verifies the default-engine NOT NULL contract for non-partition columns +/// across a representative set of primitive types. The contract: +/// +/// 1. `nullable: false` propagates from kernel's logical schema through the connector-facing +/// physical schema and into the Arrow schema engines build batches against. +/// 2. Constructing the Arrow batch an engine would hand to the kernel write API with a null in the +/// NOT NULL column fails at batch construction, before the engine is invoked. +/// +/// See [#2465](https://github.com/delta-io/delta-kernel-rs/issues/2465). +#[rstest] +#[case::integer(DataType::INTEGER)] +#[case::long(DataType::LONG)] +#[case::string(DataType::STRING)] +#[case::binary(DataType::BINARY)] +#[case::boolean(DataType::BOOLEAN)] +#[case::timestamp(DataType::TIMESTAMP)] +#[case::decimal(DataType::decimal(10, 2).unwrap())] +#[tokio::test] +async fn test_not_null_data_column_rejects_null_in_batch( + #[case] data_type: DataType, +) -> Result<(), Box> { + let _ = tracing_subscriber::fmt::try_init(); + + // Create a table with a NOT NULL column. + let schema = Arc::new(StructType::try_new(vec![StructField::not_null( + "c", + data_type.clone(), + )])?); + let (_tmp_dir, table_path, engine) = test_table_setup()?; + let _ = kernel_create_table(&table_path, schema.clone(), "test/1.0") + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())?; + + let snapshot = Snapshot::builder_for(&table_path).build(engine.as_ref())?; + // The non-null schema auto-enables the `invariants` writer feature. + assert!( + snapshot + .table_configuration() + .protocol() + .writer_features() + .is_some_and(|f| f.contains(&delta_kernel::table_features::TableFeature::Invariants)), + "non-null schema must auto-enable the `invariants` writer feature", + ); + + let write_context = begin_transaction(snapshot, engine.as_ref())? + .with_engine_info("default engine") + .unpartitioned_write_context()?; + + // Use the connector-facing physical schema (not the logical one); the logical schema + // would hide bugs in the logical->physical mapping that engines hit in production. + let physical = write_context.physical_schema(); + let physical_field = physical + .field("c") + .expect("physical schema must contain column 'c'"); + assert!( + !physical_field.nullable, + "physical schema must preserve `nullable: false` for column 'c'", + ); + let arrow_schema: delta_kernel::arrow::datatypes::Schema = + physical.as_ref().try_into_arrow()?; + assert!( + !arrow_schema.field(0).is_nullable(), + "Arrow conversion of physical schema must preserve `nullable: false`", + ); + + let arrow_type = arrow_schema.field(0).data_type().clone(); + let result = RecordBatch::try_new(Arc::new(arrow_schema), vec![new_null_array(&arrow_type, 1)]); + assert!( + result.is_err(), + "batch construction should reject null in NOT NULL column ({data_type:?}); got: {result:?}", + ); + + Ok(()) +} + +// ---- Void type write-time validation tests ---- +// +// The rstest cases below use `StructType::new_unchecked` for convenience; `StructType::try_new` +// would also accept them, since it validates structural properties (field-name uniqueness, +// metadata-column rules) and does not reject void placements. The validator +// (`validate_schema_for_write`) is the kernel-internal write-time rejection point for void in +// `Array`/`Map` and all-void structs, and it also protects schemas loaded from existing table +// metadata (JSON-deserialized from the log) and any `new_unchecked` paths. + +/// Helper to create a table with a given schema and attempt a commit with dummy add_files. +/// Returns the commit error (panics if commit succeeds). +async fn try_write_with_void_schema(schema: SchemaRef) -> KernelError { + let (store, engine, table_location) = engine_store_setup("void_write_test", None); + let table_url = create_table(store, table_location, schema, &[], false, vec![], vec![]) + .await + .expect("table creation should succeed"); + let engine = Arc::new(engine); + let snapshot = Snapshot::builder_for(table_url) + .build(engine.as_ref()) + .expect("snapshot should build"); + let mut txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref()) + .expect("transaction should create"); + + // Add dummy file metadata to trigger write validation + let add_schema = txn.add_files_schema().clone(); + let metadata = + create_add_files_metadata(&add_schema, vec![("file.parquet", 100, 1000, Some(1))]) + .expect("metadata creation should succeed"); + txn.add_files(metadata); + txn.commit(engine.as_ref()) + .expect_err("commit should fail for invalid void schema") +} + +#[rstest] +#[case::void_array_element( + Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable( + "arr", + ArrayType::new(DataType::VOID, true), + ), + ])), + "array element type" +)] +#[case::void_map_value( + Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable( + "m", + MapType::new(DataType::STRING, DataType::VOID, true), + ), + ])), + "map value type" +)] +#[case::void_map_key( + Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable( + "m", + MapType::new(DataType::VOID, DataType::STRING, true), + ), + ])), + "map key type" +)] +#[case::void_in_struct_in_array( + Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable( + "arr", + ArrayType::new( + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::VOID), + ]), + true, + ), + ), + ])), + "Void type is not allowed inside" +)] +#[case::void_in_struct_in_map_value( + Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable( + "m", + MapType::new( + DataType::STRING, + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::VOID), + ]), + true, + ), + ), + ])), + "Void type is not allowed inside" +)] +#[case::all_void_table( + Arc::new(StructType::new_unchecked([ + StructField::nullable("a", DataType::VOID), + StructField::nullable("b", DataType::VOID), + ])), + "at least one non-void column" +)] +#[case::all_void_struct( + Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable( + "s", + StructType::new_unchecked([ + StructField::nullable("x", DataType::VOID), + StructField::nullable("y", DataType::VOID), + ]), + ), + ])), + "contains no non-void fields" +)] +// A zero-field top-level schema is an empty schema, rejected by the empty-schema +// write gate (it has no columns at all, void or otherwise) rather than by void validation. +#[case::empty_struct_top_level( + Arc::new(StructType::new_unchecked(Vec::::new())), + "empty schema" +)] +#[case::nested_empty_struct( + Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable( + "s", + StructType::new_unchecked(Vec::::new()), + ), + ])), + "contains no non-void fields" +)] +#[case::empty_struct_in_array( + Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable( + "arr", + ArrayType::new(StructType::new_unchecked(Vec::::new()), true), + ), + ])), + "struct nested in Array or Map must contain at least one non-void field" +)] +#[tokio::test] +async fn write_rejects_invalid_void_placement( + #[case] schema: SchemaRef, + #[case] expected_msg: &str, +) { + let err = try_write_with_void_schema(schema).await; + assert!( + err.to_string().contains(expected_msg), + "Expected error containing '{expected_msg}', got: {err}" + ); +} + +// Validation must trigger when the connector requests a write context, before any Parquet is +// written. Both public entry points (partitioned and unpartitioned) are covered, and the +// `all_void_table` case is included because it is the shape where `strip_void_from_schema` +// would otherwise silently produce an empty physical schema -- so the fail-fast guard is most +// load-bearing there. The commit-time check (exercised by `write_rejects_invalid_void_placement`) +// remains as defense-in-depth. +#[rstest] +#[case::unpartitioned_void_array( + "void_fail_fast_unpartitioned_array", + vec![], + HashMap::new(), + Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("arr", ArrayType::new(DataType::VOID, true)), + ])), + "array element type", +)] +#[case::partitioned_void_array( + "void_fail_fast_partitioned_array", + vec!["id"], + HashMap::from([("id".to_string(), Scalar::Integer(1))]), + Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("arr", ArrayType::new(DataType::VOID, true)), + ])), + "array element type", +)] +#[case::unpartitioned_all_void( + "void_fail_fast_unpartitioned_all_void", + vec![], + HashMap::new(), + Arc::new(StructType::new_unchecked([ + StructField::nullable("a", DataType::VOID), + StructField::nullable("b", DataType::VOID), + ])), + "at least one non-void column", +)] +#[tokio::test] +async fn write_context_creation_fails_fast_on_invalid_void_schema( + #[case] test_name: &str, + #[case] partition_columns: Vec<&str>, + #[case] partition_values: HashMap, + #[case] schema: SchemaRef, + #[case] expected_msg: &str, +) { + let (store, engine, table_location) = engine_store_setup(test_name, None); + let table_url = create_table( + store, + table_location, + schema, + &partition_columns, + false, + vec![], + vec![], + ) + .await + .expect("table creation should succeed"); + let engine = Arc::new(engine); + let snapshot = Snapshot::builder_for(table_url) + .build(engine.as_ref()) + .expect("snapshot should build"); + let txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref()) + .expect("transaction should create"); + + let err = if partition_columns.is_empty() { + txn.unpartitioned_write_context() + .expect_err("unpartitioned_write_context should reject invalid void schema") + } else { + txn.partitioned_write_context(partition_values) + .expect_err("partitioned_write_context should reject invalid void schema") + }; + assert!( + err.to_string().contains(expected_msg), + "expected error containing '{expected_msg}', got: {err}" + ); +} + +#[tokio::test] +async fn write_context_excludes_void_from_physical_schema() -> Result<(), Box> +{ + let schema = Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("v", DataType::VOID), + StructField::nullable("name", DataType::STRING), + ])); + let (store, engine, table_location) = engine_store_setup("void_physical_test", None); + let table_url = create_table(store, table_location, schema, &[], false, vec![], vec![]).await?; + let engine = Arc::new(engine); + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + + // Logical schema should contain void column + { + let logical = snapshot.schema(); + assert_eq!(logical.fields().count(), 3); + assert!(logical.field("v").is_some()); + } + + let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())?; + + let wc = txn.unpartitioned_write_context()?; + let physical = wc.physical_schema(); + + // Physical schema should NOT contain void column + assert_eq!(physical.fields().count(), 2); + assert!(physical.field("id").is_some()); + assert!(physical.field("name").is_some()); + assert!(physical.field("v").is_none()); + + Ok(()) +} + +// Metadata-only operations should always succeed, even for schemas that are +// invalid for data writes (void-in-array, void-in-map, all-void structs). +#[tokio::test] +async fn metadata_only_commit_with_void_in_array_succeeds() -> Result<(), Box> +{ + let schema = Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable("arr", ArrayType::new(DataType::VOID, true)), + ])); + let (store, engine, table_location) = engine_store_setup("void_metadata_test", None); + let table_url = create_table(store, table_location, schema, &[], false, vec![], vec![]).await?; + let engine = Arc::new(engine); + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())?; + + // Commit with NO add_files — this is a metadata-only operation and should succeed + let result = txn.commit(engine.as_ref()); + assert!( + result.is_ok(), + "Metadata-only commit on void-in-array schema should succeed, got: {:?}", + result.err() + ); + + Ok(()) +} + +// Verify that nested void fields inside structs are excluded from the physical schema. +// Schema: {id: int, s: struct} +// Physical: {id: int, s: struct} +#[tokio::test] +async fn write_context_excludes_nested_void_from_physical_schema( +) -> Result<(), Box> { + let schema = Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable( + "s", + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::VOID), + ]), + ), + ])); + let (store, engine, table_location) = engine_store_setup("void_nested_physical_test", None); + let table_url = create_table(store, table_location, schema, &[], false, vec![], vec![]).await?; + let engine = Arc::new(engine); + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + + // Logical schema should contain the void field inside the struct + { + let logical = snapshot.schema(); + let s_field = logical.field("s").expect("s should exist"); + if let DataType::Struct(inner) = s_field.data_type() { + assert!( + inner.field("b").is_some(), + "logical should have void field b" + ); + assert_eq!(inner.fields().count(), 2); + } + } + + let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())?; + let wc = txn.unpartitioned_write_context()?; + let physical = wc.physical_schema(); + + // Physical schema struct should NOT contain the void field + let s_field = physical.field("s").expect("s should exist in physical"); + if let DataType::Struct(inner) = s_field.data_type() { + assert_eq!( + inner.fields().count(), + 1, + "physical struct should have 1 field (void dropped)" + ); + assert!(inner.field("a").is_some(), "non-void field should remain"); + assert!(inner.field("b").is_none(), "void field should be dropped"); + } else { + panic!("s should be a struct type"); + } + + Ok(()) +} + +// Verify that the logical_to_physical transform drops nested void fields from structs. +// The transform expression should contain a nested transform that drops the void sub-field, +// matching the physical schema which has void stripped recursively. +#[tokio::test] +async fn write_transform_drops_nested_void_fields() -> Result<(), Box> { + let schema = Arc::new(StructType::new_unchecked([ + StructField::nullable("id", DataType::INTEGER), + StructField::nullable( + "s", + StructType::new_unchecked([ + StructField::nullable("a", DataType::INTEGER), + StructField::nullable("b", DataType::VOID), + ]), + ), + ])); + let (store, engine, table_location) = engine_store_setup("void_nested_transform_test", None); + let table_url = create_table(store, table_location, schema, &[], false, vec![], vec![]).await?; + let engine = Arc::new(engine); + let snapshot = Snapshot::builder_for(table_url).build(engine.as_ref())?; + + let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())?; + let wc = txn.unpartitioned_write_context()?; + + // The transform expression should mention dropping "b" inside the struct + let l2p = wc.logical_to_physical(); + let expr_str = format!("{l2p}"); + assert!( + expr_str.contains("drop b"), + "Transform should drop nested void field 'b'. Expression: {expr_str}" + ); + + // Physical schema should also not contain void + let physical = wc.physical_schema(); + let s_field = physical.field("s").expect("s should exist in physical"); + if let DataType::Struct(inner) = s_field.data_type() { + assert_eq!(inner.fields().count(), 1); + assert!( + inner.field("b").is_none(), + "void field b should be stripped" + ); + } else { + panic!("s should be struct"); + } + + Ok(()) +} diff --git a/kernel/tests/read.rs b/kernel/tests/read.rs deleted file mode 100644 index e2d2624fb3..0000000000 --- a/kernel/tests/read.rs +++ /dev/null @@ -1,1727 +0,0 @@ -use std::path::PathBuf; -use std::sync::Arc; - -use delta_kernel::actions::deletion_vector::split_vector; -use delta_kernel::arrow::array::{AsArray as _, RecordBatch}; -use delta_kernel::arrow::compute::{concat_batches, filter_record_batch}; -use delta_kernel::arrow::datatypes::{Field as ArrowField, Int64Type, Schema as ArrowSchema}; -use delta_kernel::engine::arrow_conversion::TryFromKernel as _; -use delta_kernel::engine::arrow_data::EngineDataArrowExt as _; -use delta_kernel::engine::default::DefaultEngineBuilder; -use delta_kernel::expressions::{ - column_expr, column_pred, Expression as Expr, ExpressionRef, Predicate as Pred, -}; -use delta_kernel::log_segment::LogSegment; -use delta_kernel::object_store::{memory::InMemory, path::Path, ObjectStore}; -use delta_kernel::parquet::file::properties::{EnabledStatistics, WriterProperties}; -use delta_kernel::path::ParsedLogPath; -use delta_kernel::scan::state::{transform_to_logical, ScanFile}; -use delta_kernel::scan::Scan; -use delta_kernel::schema::{DataType, MetadataColumnSpec, Schema, StructField, StructType}; -use delta_kernel::{Engine, FileMeta, Snapshot}; - -use itertools::Itertools; -use test_utils::{ - actions_to_string, add_commit, generate_batch, generate_simple_batch, into_record_batch, - load_test_data, read_scan, record_batch_to_bytes, record_batch_to_bytes_with_props, IntoArray, - TestAction, METADATA, -}; -use url::Url; - -mod common; - -const PARQUET_FILE1: &str = "part-00000-a72b1fb3-f2df-41fe-a8f0-e65b746382dd-c000.snappy.parquet"; -const PARQUET_FILE2: &str = "part-00001-c506e79a-0bf8-4e2b-a42b-9731b2e490ae-c000.snappy.parquet"; -const PARQUET_FILE3: &str = "part-00002-c506e79a-0bf8-4e2b-a42b-9731b2e490ff-c000.snappy.parquet"; - -/// Convert all top-level fields in a RecordBatch to nullable, matching Delta table schema -/// conventions where the table metadata declares columns as nullable. -fn make_top_level_fields_nullable(batch: &RecordBatch) -> RecordBatch { - let schema = Arc::new(ArrowSchema::new( - batch - .schema() - .fields() - .iter() - .map(|f| ArrowField::new(f.name(), f.data_type().clone(), true)) - .collect::>(), - )); - RecordBatch::try_new(schema, batch.columns().to_vec()).unwrap() -} - -#[tokio::test] -async fn single_commit_two_add_files() -> Result<(), Box> { - let batch = generate_simple_batch()?; - let storage = Arc::new(InMemory::new()); - let table_root = "memory:///"; - let parquet_bytes = record_batch_to_bytes(&batch); - let file_size = parquet_bytes.len() as u64; - add_commit( - table_root, - storage.as_ref(), - 0, - actions_to_string(vec![ - TestAction::Metadata, - TestAction::AddWithSize(PARQUET_FILE1.to_string(), file_size), - TestAction::AddWithSize(PARQUET_FILE2.to_string(), file_size), - ]), - ) - .await?; - storage - .put( - &Path::from(PARQUET_FILE1), - record_batch_to_bytes(&batch).into(), - ) - .await?; - storage - .put( - &Path::from(PARQUET_FILE2), - record_batch_to_bytes(&batch).into(), - ) - .await?; - - let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); - - let expected = make_top_level_fields_nullable(&batch); - let expected_data = vec![expected.clone(), expected]; - - let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; - let scan = snapshot.scan_builder().build()?; - - let mut files = 0; - let stream = scan.execute(engine)?.zip(expected_data); - - for (data, expected) in stream { - files += 1; - assert_eq!(into_record_batch(data?), expected); - } - assert_eq!(2, files, "Expected to have scanned two files"); - Ok(()) -} - -#[tokio::test] -async fn two_commits() -> Result<(), Box> { - let batch = generate_simple_batch()?; - let storage = Arc::new(InMemory::new()); - let table_root = "memory:///"; - let parquet_bytes = record_batch_to_bytes(&batch); - let file_size = parquet_bytes.len() as u64; - add_commit( - table_root, - storage.as_ref(), - 0, - actions_to_string(vec![ - TestAction::Metadata, - TestAction::AddWithSize(PARQUET_FILE1.to_string(), file_size), - ]), - ) - .await?; - add_commit( - table_root, - storage.as_ref(), - 1, - actions_to_string(vec![TestAction::AddWithSize( - PARQUET_FILE2.to_string(), - file_size, - )]), - ) - .await?; - storage - .put( - &Path::from(PARQUET_FILE1), - record_batch_to_bytes(&batch).into(), - ) - .await?; - storage - .put( - &Path::from(PARQUET_FILE2), - record_batch_to_bytes(&batch).into(), - ) - .await?; - - let engine = DefaultEngineBuilder::new(storage.clone()).build(); - - let expected = make_top_level_fields_nullable(&batch); - let expected_data = vec![expected.clone(), expected]; - - let snapshot = Snapshot::builder_for(table_root).build(&engine)?; - let scan = snapshot.scan_builder().build()?; - - let mut files = 0; - let stream = scan.execute(Arc::new(engine))?.zip(expected_data); - - for (data, expected) in stream { - files += 1; - assert_eq!(into_record_batch(data?), expected); - } - assert_eq!(2, files, "Expected to have scanned two files"); - - Ok(()) -} - -#[tokio::test] -async fn remove_action() -> Result<(), Box> { - let batch = generate_simple_batch()?; - let storage = Arc::new(InMemory::new()); - let table_root = "memory:///"; - let parquet_bytes = record_batch_to_bytes(&batch); - let file_size = parquet_bytes.len() as u64; - add_commit( - table_root, - storage.as_ref(), - 0, - actions_to_string(vec![ - TestAction::Metadata, - TestAction::AddWithSize(PARQUET_FILE1.to_string(), file_size), - ]), - ) - .await?; - add_commit( - table_root, - storage.as_ref(), - 1, - actions_to_string(vec![TestAction::AddWithSize( - PARQUET_FILE2.to_string(), - file_size, - )]), - ) - .await?; - add_commit( - table_root, - storage.as_ref(), - 2, - actions_to_string(vec![TestAction::RemoveWithSize( - PARQUET_FILE2.to_string(), - file_size, - )]), - ) - .await?; - storage - .put( - &Path::from(PARQUET_FILE1), - record_batch_to_bytes(&batch).into(), - ) - .await?; - - let engine = DefaultEngineBuilder::new(storage.clone()).build(); - - let expected = make_top_level_fields_nullable(&batch); - let expected_data = vec![expected]; - - let snapshot = Snapshot::builder_for(table_root).build(&engine)?; - let scan = snapshot.scan_builder().build()?; - - let stream = scan.execute(Arc::new(engine))?.zip(expected_data); - - let mut files = 0; - for (data, expected) in stream { - files += 1; - assert_eq!(into_record_batch(data?), expected); - } - assert_eq!(1, files, "Expected to have scanned one file"); - Ok(()) -} - -#[tokio::test] -async fn stats() -> Result<(), Box> { - fn generate_commit2(actions: Vec) -> String { - actions - .into_iter() - .map(|test_action| match test_action { - TestAction::Add(path) => format!(r#"{{"{action}":{{"path":"{path}","partitionValues":{{}},"size":262,"modificationTime":1587968586000,"dataChange":true, "stats":"{{\"numRecords\":2,\"nullCount\":{{\"id\":0}},\"minValues\":{{\"id\": 5}},\"maxValues\":{{\"id\":7}}}}"}}}}"#, action = "add", path = path), - TestAction::Remove(path) => format!(r#"{{"{action}":{{"path":"{path}","partitionValues":{{}},"size":262,"modificationTime":1587968586000,"dataChange":true}}}}"#, action = "remove", path = path), - TestAction::Metadata => METADATA.into(), - TestAction::AddWithSize(path, size) => format!(r#"{{"{action}":{{"path":"{path}","partitionValues":{{}},"size":{size},"modificationTime":1587968586000,"dataChange":true, "stats":"{{\"numRecords\":2,\"nullCount\":{{\"id\":0}},\"minValues\":{{\"id\": 5}},\"maxValues\":{{\"id\":7}}}}"}}}}"#, action = "add", path = path), - TestAction::RemoveWithSize(path, size) => format!(r#"{{"{action}":{{"path":"{path}","partitionValues":{{}},"size":{size},"modificationTime":1587968586000,"dataChange":true}}}}"#, action = "remove", path = path), - }) - .fold(String::new(), |a, b| a + &b + "\n") - } - - let batch1 = make_top_level_fields_nullable(&generate_simple_batch()?); - let batch2 = make_top_level_fields_nullable(&generate_batch(vec![ - ("id", vec![5, 7].into_array()), - ("val", vec!["e", "g"].into_array()), - ])?); - let file_size1 = record_batch_to_bytes(&batch1).len() as u64; - let file_size2 = record_batch_to_bytes(&batch2).len() as u64; - let storage = Arc::new(InMemory::new()); - let table_root = "memory:///"; - // valid commit with min/max (0, 2) - add_commit( - table_root, - storage.as_ref(), - 0, - actions_to_string(vec![ - TestAction::Metadata, - TestAction::AddWithSize(PARQUET_FILE1.to_string(), file_size1), - ]), - ) - .await?; - // storage.add_commit(1, &format!("{}\n", r#"{{"add":{{"path":"doesnotexist","partitionValues":{{}},"size":262,"modificationTime":1587968586000,"dataChange":true, "stats":"{{\"numRecords\":2,\"nullCount\":{{\"id\":0}},\"minValues\":{{\"id\": 0}},\"maxValues\":{{\"id\":2}}}}"}}}}"#)); - add_commit( - table_root, - storage.as_ref(), - 1, - generate_commit2(vec![TestAction::AddWithSize( - PARQUET_FILE2.to_string(), - file_size2, - )]), - ) - .await?; - - storage - .put( - &Path::from(PARQUET_FILE1), - record_batch_to_bytes(&batch1).into(), - ) - .await?; - - storage - .put( - &Path::from(PARQUET_FILE2), - record_batch_to_bytes(&batch2).into(), - ) - .await?; - - let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); - let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; - - // The first file has id between 1 and 3; the second has id between 5 and 7. For each operator, - // we validate the boundary values where we expect the set of matched files to change. - // - // NOTE: For cases that match both batch1 and batch2, we list batch2 first because log replay - // returns most recently added files first. - #[allow(clippy::type_complexity)] // otherwise it's even more complex because no `_` - let test_cases: Vec<(fn(Expr, Expr) -> _, _, _)> = vec![ - (Pred::eq, 0i32, vec![]), - (Pred::eq, 1, vec![&batch1]), - (Pred::eq, 3, vec![&batch1]), - (Pred::eq, 4, vec![]), - (Pred::eq, 5, vec![&batch2]), - (Pred::eq, 7, vec![&batch2]), - (Pred::eq, 8, vec![]), - (Pred::lt, 1, vec![]), - (Pred::lt, 2, vec![&batch1]), - (Pred::lt, 5, vec![&batch1]), - (Pred::lt, 6, vec![&batch2, &batch1]), - (Pred::le, 0, vec![]), - (Pred::le, 1, vec![&batch1]), - (Pred::le, 4, vec![&batch1]), - (Pred::le, 5, vec![&batch2, &batch1]), - (Pred::gt, 2, vec![&batch2, &batch1]), - (Pred::gt, 3, vec![&batch2]), - (Pred::gt, 6, vec![&batch2]), - (Pred::gt, 7, vec![]), - (Pred::ge, 3, vec![&batch2, &batch1]), - (Pred::ge, 4, vec![&batch2]), - (Pred::ge, 7, vec![&batch2]), - (Pred::ge, 8, vec![]), - (Pred::ne, 0, vec![&batch2, &batch1]), - (Pred::ne, 1, vec![&batch2, &batch1]), - (Pred::ne, 3, vec![&batch2, &batch1]), - (Pred::ne, 4, vec![&batch2, &batch1]), - (Pred::ne, 5, vec![&batch2, &batch1]), - (Pred::ne, 7, vec![&batch2, &batch1]), - (Pred::ne, 8, vec![&batch2, &batch1]), - ]; - for (pred_fn, value, expected_batches) in test_cases { - let predicate = pred_fn(column_expr!("id"), Expr::literal(value)); - let scan = snapshot - .clone() - .scan_builder() - .with_predicate(Arc::new(predicate.clone())) - .build()?; - - let expected_files = expected_batches.len(); - let mut files_scanned = 0; - let stream = scan.execute(engine.clone())?.zip(expected_batches); - - for (batch, expected) in stream { - files_scanned += 1; - assert_eq!(into_record_batch(batch?), expected.clone()); - } - assert_eq!(expected_files, files_scanned, "{predicate:?}"); - } - Ok(()) -} - -fn read_with_execute( - engine: Arc, - scan: &Scan, - expected: &[String], -) -> Result<(), Box> { - let result_schema = Arc::new(ArrowSchema::try_from_kernel( - scan.logical_schema().as_ref(), - )?); - let batches = read_scan(scan, engine)?; - - if expected.is_empty() { - assert_eq!(batches.len(), 0); - } else { - let batch = concat_batches(&result_schema, &batches)?; - assert_batches_sorted_eq!(expected, &[batch]); - } - Ok(()) -} - -fn scan_metadata_callback(batches: &mut Vec, scan_file: ScanFile) { - batches.push(scan_file); -} - -fn read_with_scan_metadata( - location: &Url, - engine: &dyn Engine, - scan: &Scan, - expected: &[String], -) -> Result<(), Box> { - let result_schema = Arc::new(ArrowSchema::try_from_kernel( - scan.logical_schema().as_ref(), - )?); - let scan_metadata = scan.scan_metadata(engine)?; - let mut scan_files = vec![]; - for res in scan_metadata { - let scan_metadata = res?; - scan_files = scan_metadata.visit_scan_files(scan_files, scan_metadata_callback)?; - } - - let mut batches = vec![]; - for scan_file in scan_files.into_iter() { - let file_path = location.join(&scan_file.path)?; - let mut selection_vector = scan_file - .dv_info - .get_selection_vector(engine, location) - .unwrap(); - let meta = FileMeta { - last_modified: 0, - size: scan_file.size.try_into().unwrap(), - location: file_path, - }; - let read_results = engine - .parquet_handler() - .read_parquet_files( - &[meta], - scan.physical_schema().clone(), - scan.physical_predicate().clone(), - ) - .unwrap(); - - for read_result in read_results { - let read_result = read_result.unwrap(); - let len = read_result.len(); - // to transform the physical data into the correct logical form - let logical = transform_to_logical( - engine, - read_result, - scan.physical_schema(), - scan.logical_schema(), - scan_file.transform.clone(), - ) - .unwrap(); - let record_batch = logical.try_into_record_batch()?; - let rest = split_vector(selection_vector.as_mut(), len, Some(true)); - let batch = if let Some(mask) = selection_vector.clone() { - // apply the selection vector - filter_record_batch(&record_batch, &mask.into()).unwrap() - } else { - record_batch - }; - selection_vector = rest; - batches.push(batch); - } - } - - if expected.is_empty() { - assert_eq!(batches.len(), 0); - } else { - let batch = concat_batches(&result_schema, &batches)?; - assert_batches_sorted_eq!(expected, &[batch]); - } - Ok(()) -} - -fn read_table_data( - path: &str, - select_cols: Option<&[&str]>, - predicate: Option, - mut expected: Vec, -) -> Result<(), Box> { - let path = std::fs::canonicalize(PathBuf::from(path))?; - let predicate = predicate.map(Arc::new); - let url = url::Url::from_directory_path(path).unwrap(); - let engine = test_utils::create_default_engine(&url)?; - - let snapshot = Snapshot::builder_for(url.clone()).build(engine.as_ref())?; - - let read_schema = select_cols.map(|select_cols| { - let table_schema = snapshot.schema(); - let selected_fields = select_cols - .iter() - .map(|col| table_schema.field(col).cloned().unwrap()); - Arc::new(Schema::new_unchecked(selected_fields)) - }); - println!("Read {url:?} with schema {read_schema:#?} and predicate {predicate:#?}"); - let scan = snapshot - .scan_builder() - .with_schema_opt(read_schema) - .with_predicate(predicate.clone()) - .build()?; - - sort_lines!(expected); - read_with_scan_metadata(&url, engine.as_ref(), &scan, &expected)?; - read_with_execute(engine, &scan, &expected)?; - Ok(()) -} - -// util to take a Vec<&str> and call read_table_data with Vec -fn read_table_data_str( - path: &str, - select_cols: Option<&[&str]>, - predicate: Option, - expected: Vec<&str>, -) -> Result<(), Box> { - read_table_data( - path, - select_cols, - predicate, - expected.into_iter().map(String::from).collect(), - ) -} - -#[test] -fn data() -> Result<(), Box> { - let expected = vec![ - "+--------+--------+---------+", - "| letter | number | a_float |", - "+--------+--------+---------+", - "| | 6 | 6.6 |", - "| a | 1 | 1.1 |", - "| a | 4 | 4.4 |", - "| b | 2 | 2.2 |", - "| c | 3 | 3.3 |", - "| e | 5 | 5.5 |", - "+--------+--------+---------+", - ]; - read_table_data_str("./tests/data/basic_partitioned", None, None, expected)?; - - Ok(()) -} - -#[test] -fn column_ordering() -> Result<(), Box> { - let expected = vec![ - "+---------+--------+--------+", - "| a_float | letter | number |", - "+---------+--------+--------+", - "| 6.6 | | 6 |", - "| 4.4 | a | 4 |", - "| 5.5 | e | 5 |", - "| 1.1 | a | 1 |", - "| 2.2 | b | 2 |", - "| 3.3 | c | 3 |", - "+---------+--------+--------+", - ]; - read_table_data_str( - "./tests/data/basic_partitioned", - Some(&["a_float", "letter", "number"]), - None, - expected, - )?; - - Ok(()) -} - -#[test] -fn column_ordering_and_projection() -> Result<(), Box> { - let expected = vec![ - "+---------+--------+", - "| a_float | number |", - "+---------+--------+", - "| 6.6 | 6 |", - "| 4.4 | 4 |", - "| 5.5 | 5 |", - "| 1.1 | 1 |", - "| 2.2 | 2 |", - "| 3.3 | 3 |", - "+---------+--------+", - ]; - read_table_data_str( - "./tests/data/basic_partitioned", - Some(&["a_float", "number"]), - None, - expected, - )?; - - Ok(()) -} - -// get the basic_partitioned table for a set of expected numbers -fn table_for_numbers(nums: Vec) -> Vec { - let mut res: Vec = vec![ - "+---------+--------+", - "| a_float | number |", - "+---------+--------+", - ] - .into_iter() - .map(String::from) - .collect(); - for num in nums.iter() { - res.push(format!("| {num}.{num} | {num} |")); - } - res.push("+---------+--------+".to_string()); - res -} - -// get the basic_partitioned table for a set of expected letters -fn table_for_letters(letters: &[char]) -> Vec { - let mut res: Vec = vec![ - "+--------+--------+", - "| letter | number |", - "+--------+--------+", - ] - .into_iter() - .map(String::from) - .collect(); - let rows = vec![(1, 'a'), (2, 'b'), (3, 'c'), (4, 'a'), (5, 'e')]; - for (num, letter) in rows { - if letters.contains(&letter) { - res.push(format!("| {letter} | {num} |")); - } - } - res.push("+--------+--------+".to_string()); - res -} - -#[rstest::rstest] -#[case::less_than( - column_expr!("number").lt(Expr::literal(4i64)), - table_for_numbers(vec![1, 2, 3]) -)] -#[case::less_than_or_equal( - column_expr!("number").le(Expr::literal(4i64)), - table_for_numbers(vec![1, 2, 3, 4]) -)] -#[case::greater_than( - column_expr!("number").gt(Expr::literal(4i64)), - table_for_numbers(vec![5, 6]) -)] -#[case::greater_than_or_equal( - column_expr!("number").ge(Expr::literal(4i64)), - table_for_numbers(vec![4, 5, 6]) -)] -#[case::equal( - column_expr!("number").eq(Expr::literal(4i64)), - table_for_numbers(vec![4]) -)] -#[case::not_equal( - column_expr!("number").ne(Expr::literal(4i64)), - table_for_numbers(vec![1, 2, 3, 5, 6]) -)] -fn predicate_on_number( - #[case] pred: Pred, - #[case] expected: Vec, -) -> Result<(), Box> { - read_table_data( - "./tests/data/basic_partitioned", - Some(&["a_float", "number"]), - Some(pred), - expected, - )?; - Ok(()) -} - -#[rstest::rstest] -#[case::is_null( - column_expr!("letter").is_null(), - vec![ - "+--------+--------+", - "| letter | number |", - "+--------+--------+", - "| | 6 |", - "+--------+--------+", - ] - .into_iter() - .map(String::from) - .collect() -)] -#[case::is_not_null( - column_expr!("letter").is_not_null(), - table_for_letters(&['a', 'b', 'c', 'e']) -)] -#[case::less_than( - column_expr!("letter").lt(Expr::literal("c")), - table_for_letters(&['a', 'b']) -)] -#[case::less_than_or_equal( - column_expr!("letter").le(Expr::literal("c")), - table_for_letters(&['a', 'b', 'c']) -)] -#[case::greater_than( - column_expr!("letter").gt(Expr::literal("c")), - table_for_letters(&['e']) -)] -#[case::greater_than_or_equal( - column_expr!("letter").ge(Expr::literal("c")), - table_for_letters(&['c', 'e']) -)] -#[case::equal( - column_expr!("letter").eq(Expr::literal("c")), - table_for_letters(&['c']) -)] -#[case::not_equal( - column_expr!("letter").ne(Expr::literal("c")), - table_for_letters(&['a', 'b', 'e']) -)] -fn predicate_on_letter( - #[case] pred: Pred, - #[case] expected: Vec, -) -> Result<(), Box> { - // Test basic column pruning. Note that the actual predicate machinery is already well-tested, - // so we're just testing wiring here. - read_table_data( - "./tests/data/basic_partitioned", - Some(&["letter", "number"]), - Some(pred), - expected, - )?; - Ok(()) -} - -#[rstest::rstest] -#[case::or_no_pruning( - Pred::or( - // No pruning power - column_expr!("letter").gt(Expr::literal("a")), - column_expr!("number").gt(Expr::literal(3i64)), - ), - vec![ - "+--------+--------+", - "| letter | number |", - "+--------+--------+", - "| | 6 |", - "| a | 1 |", - "| a | 4 |", - "| b | 2 |", - "| c | 3 |", - "| e | 5 |", - "+--------+--------+", - ] - .into_iter() - .map(String::from) - .collect() -)] -#[case::and_with_pruning( - Pred::and( - column_expr!("letter").gt(Expr::literal("a")), // numbers 2, 3, 5 - column_expr!("number").gt(Expr::literal(3i64)), // letters a, e - ), - table_for_letters(&['e']) -)] -#[case::and_with_nested_or( - Pred::and( - column_expr!("letter").gt(Expr::literal("a")), // numbers 2, 3, 5 - Pred::or( - // No pruning power - column_expr!("letter").eq(Expr::literal("c")), - column_expr!("number").eq(Expr::literal(3i64)), - ), - ), - table_for_letters(&['b', 'c', 'e']) -)] -fn predicate_on_letter_and_number( - #[case] pred: Pred, - #[case] expected: Vec, -) -> Result<(), Box> { - // Partition skipping and file skipping are currently implemented separately. Mixing them in an - // AND clause will evaulate each separately, but mixing them in an OR clause disables both. - read_table_data( - "./tests/data/basic_partitioned", - Some(&["letter", "number"]), - Some(pred), - expected, - )?; - Ok(()) -} - -#[rstest::rstest] -#[case::not_less_than( - Pred::not(column_expr!("number").lt(Expr::literal(4i64))), - table_for_numbers(vec![4, 5, 6]) -)] -#[case::not_less_than_or_equal( - Pred::not(column_expr!("number").le(Expr::literal(4i64))), - table_for_numbers(vec![5, 6]) -)] -#[case::not_greater_than( - Pred::not(column_expr!("number").gt(Expr::literal(4i64))), - table_for_numbers(vec![1, 2, 3, 4]) -)] -#[case::not_greater_than_or_equal( - Pred::not(column_expr!("number").ge(Expr::literal(4i64))), - table_for_numbers(vec![1, 2, 3]) -)] -#[case::not_equal( - Pred::not(column_expr!("number").eq(Expr::literal(4i64))), - table_for_numbers(vec![1, 2, 3, 5, 6]) -)] -#[case::not_not_equal( - Pred::not(column_expr!("number").ne(Expr::literal(4i64))), - table_for_numbers(vec![4]) -)] -fn predicate_on_number_not( - #[case] pred: Pred, - #[case] expected: Vec, -) -> Result<(), Box> { - read_table_data( - "./tests/data/basic_partitioned", - Some(&["a_float", "number"]), - Some(pred), - expected, - )?; - Ok(()) -} - -#[test] -fn predicate_on_number_with_not_null() -> Result<(), Box> { - let expected = vec![ - "+---------+--------+", - "| a_float | number |", - "+---------+--------+", - "| 1.1 | 1 |", - "| 2.2 | 2 |", - "+---------+--------+", - ]; - read_table_data_str( - "./tests/data/basic_partitioned", - Some(&["a_float", "number"]), - Some(Pred::and( - column_expr!("number").is_not_null(), - column_expr!("number").lt(Expr::literal(3i64)), - )), - expected, - )?; - Ok(()) -} - -#[test] -fn predicate_null() -> Result<(), Box> { - let expected = vec![]; // number is never null - read_table_data_str( - "./tests/data/basic_partitioned", - Some(&["a_float", "number"]), - Some(column_expr!("number").is_null()), - expected, - )?; - Ok(()) -} - -#[test] -fn mixed_null() -> Result<(), Box> { - let expected = vec![ - "+------+--------------+", - "| part | n |", - "+------+--------------+", - "| 0 | |", - "| 0 | |", - "| 0 | |", - "| 0 | |", - "| 0 | |", - "| 2 | |", - "| 2 | non-null-mix |", - "| 2 | |", - "| 2 | non-null-mix |", - "| 2 | |", - "+------+--------------+", - ]; - read_table_data_str( - "./tests/data/mixed-nulls", - Some(&["part", "n"]), - Some(column_expr!("n").is_null()), - expected, - )?; - Ok(()) -} - -#[test] -fn mixed_not_null() -> Result<(), Box> { - let expected = vec![ - "+------+--------------+", - "| part | n |", - "+------+--------------+", - "| 1 | non-null |", - "| 1 | non-null |", - "| 1 | non-null |", - "| 1 | non-null |", - "| 1 | non-null |", - "| 2 | |", - "| 2 | |", - "| 2 | |", - "| 2 | non-null-mix |", - "| 2 | non-null-mix |", - "+------+--------------+", - ]; - read_table_data_str( - "./tests/data/mixed-nulls", - Some(&["part", "n"]), - Some(column_expr!("n").is_not_null()), - expected, - )?; - Ok(()) -} - -#[rstest::rstest] -#[case::and_both_conditions( - Pred::and( - column_expr!("number").gt(Expr::literal(4i64)), - column_expr!("a_float").gt(Expr::literal(5.5)), - ), - table_for_numbers(vec![6]) -)] -#[case::and_with_negation( - Pred::and( - column_expr!("number").gt(Expr::literal(4i64)), - Pred::not(column_expr!("a_float").gt(Expr::literal(5.5))), - ), - table_for_numbers(vec![5]) -)] -#[case::or_either_condition( - Pred::or( - column_expr!("number").gt(Expr::literal(4i64)), - column_expr!("a_float").gt(Expr::literal(5.5)), - ), - table_for_numbers(vec![5, 6]) -)] -#[case::or_with_negation( - Pred::or( - column_expr!("number").gt(Expr::literal(4i64)), - Pred::not(column_expr!("a_float").gt(Expr::literal(5.5))), - ), - table_for_numbers(vec![1, 2, 3, 4, 5, 6]) -)] -fn and_or_predicates( - #[case] pred: Pred, - #[case] expected: Vec, -) -> Result<(), Box> { - read_table_data( - "./tests/data/basic_partitioned", - Some(&["a_float", "number"]), - Some(pred), - expected, - )?; - Ok(()) -} - -#[rstest::rstest] -#[case::not_and_both_conditions( - Pred::not(Pred::and( - column_expr!("number").gt(Expr::literal(4i64)), - column_expr!("a_float").gt(Expr::literal(5.5)), - )), - table_for_numbers(vec![1, 2, 3, 4, 5]) -)] -#[case::not_and_with_negation( - Pred::not(Pred::and( - column_expr!("number").gt(Expr::literal(4i64)), - Pred::not(column_expr!("a_float").gt(Expr::literal(5.5))), - )), - table_for_numbers(vec![1, 2, 3, 4, 6]) -)] -#[case::not_or_either_condition( - Pred::not(Pred::or( - column_expr!("number").gt(Expr::literal(4i64)), - column_expr!("a_float").gt(Expr::literal(5.5)), - )), - table_for_numbers(vec![1, 2, 3, 4]) -)] -#[case::not_or_with_negation( - Pred::not(Pred::or( - column_expr!("number").gt(Expr::literal(4i64)), - Pred::not(column_expr!("a_float").gt(Expr::literal(5.5))), - )), - vec![] -)] -fn not_and_or_predicates( - #[case] pred: Pred, - #[case] expected: Vec, -) -> Result<(), Box> { - read_table_data( - "./tests/data/basic_partitioned", - Some(&["a_float", "number"]), - Some(pred), - expected, - )?; - Ok(()) -} - -#[rstest::rstest] -#[case::literal_false(Pred::literal(false), table_for_numbers(vec![]))] -#[case::and_with_literal_false( - Pred::and(column_pred!("number"), Pred::literal(false)), - table_for_numbers(vec![]) -)] -#[case::literal_true( - Pred::literal(true), - table_for_numbers(vec![1, 2, 3, 4, 5, 6]) -)] -#[case::from_literal_expr( - Pred::from_expr(Expr::literal(3i64)), - table_for_numbers(vec![1, 2, 3, 4, 5, 6]) -)] -#[case::distinct_value( - column_expr!("number").distinct(Expr::literal(3i64)), - table_for_numbers(vec![1, 2, 4, 5, 6]) -)] -#[case::distinct_null( - column_expr!("number").distinct(Expr::null_literal(DataType::LONG)), - table_for_numbers(vec![1, 2, 3, 4, 5, 6]) -)] -#[case::not_distinct_value( - Pred::not(column_expr!("number").distinct(Expr::literal(3i64))), - table_for_numbers(vec![3]) -)] -#[case::not_distinct_null( - Pred::not(column_expr!("number").distinct(Expr::null_literal(DataType::LONG))), - table_for_numbers(vec![]) -)] -#[case::gt_empty_struct( - column_expr!("number").gt(Expr::struct_from(Vec::::new())), - table_for_numbers(vec![1, 2, 3, 4, 5, 6]) -)] -#[case::not_gt_empty_struct( - Pred::not(column_expr!("number").gt(Expr::struct_from(Vec::::new()))), - table_for_numbers(vec![1, 2, 3, 4, 5, 6]) -)] -fn invalid_skips_none_predicates( - #[case] pred: Pred, - #[case] expected: Vec, -) -> Result<(), Box> { - read_table_data( - "./tests/data/basic_partitioned", - Some(&["a_float", "number"]), - Some(pred), - expected, - )?; - Ok(()) -} - -#[test] -fn with_predicate_and_removes() -> Result<(), Box> { - let expected = vec![ - "+-------+", - "| value |", - "+-------+", - "| 1 |", - "| 2 |", - "| 3 |", - "| 4 |", - "| 5 |", - "| 6 |", - "| 7 |", - "| 8 |", - "+-------+", - ]; - read_table_data_str( - "./tests/data/table-with-dv-small/", - None, - Some(Pred::gt(column_expr!("value"), Expr::literal(3))), - expected, - )?; - Ok(()) -} - -#[tokio::test] -async fn predicate_on_non_nullable_partition_column() -> Result<(), Box> { - // Test for https://github.com/delta-io/delta-kernel-rs/issues/698 - let batch = generate_batch(vec![("val", vec!["a", "b", "c"].into_array())])?; - - let storage = Arc::new(InMemory::new()); - let table_root = "memory:///"; - let actions = [ - r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#.to_string(), - r#"{"commitInfo":{"timestamp":1587968586154,"operation":"WRITE","operationParameters":{"mode":"ErrorIfExists","partitionBy":"[\"id\"]"},"isBlindAppend":true}}"#.to_string(), - r#"{"metaData":{"id":"5fba94ed-9794-4965-ba6e-6ee3c0d22af9","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":false,\"metadata\":{}},{\"name\":\"val\",\"type\":\"string\",\"nullable\":false,\"metadata\":{}}]}","partitionColumns":["id"],"configuration":{},"createdTime":1587968585495}}"#.to_string(), - format!(r#"{{"add":{{"path":"id=1/{PARQUET_FILE1}","partitionValues":{{"id":"1"}},"size":0,"modificationTime":1587968586000,"dataChange":true, "stats":"{{\"numRecords\":3,\"nullCount\":{{\"val\":0}},\"minValues\":{{\"val\":\"a\"}},\"maxValues\":{{\"val\":\"c\"}}}}"}}}}"#), - format!(r#"{{"add":{{"path":"id=2/{PARQUET_FILE2}","partitionValues":{{"id":"2"}},"size":0,"modificationTime":1587968586000,"dataChange":true, "stats":"{{\"numRecords\":3,\"nullCount\":{{\"val\":0}},\"minValues\":{{\"val\":\"a\"}},\"maxValues\":{{\"val\":\"c\"}}}}"}}}}"#), - ]; - - add_commit(table_root, storage.as_ref(), 0, actions.iter().join("\n")).await?; - storage - .put( - &Path::from("id=1").child(PARQUET_FILE1), - record_batch_to_bytes(&batch).into(), - ) - .await?; - storage - .put( - &Path::from("id=2").child(PARQUET_FILE2), - record_batch_to_bytes(&batch).into(), - ) - .await?; - - let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); - let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; - - let predicate = Pred::eq(column_expr!("id"), Expr::literal(2)); - let scan = snapshot - .scan_builder() - .with_predicate(Arc::new(predicate)) - .build()?; - - let stream = scan.execute(engine)?; - - let mut files_scanned = 0; - for engine_data in stream { - let mut result_batch = into_record_batch(engine_data?); - let _ = result_batch.remove_column(result_batch.schema().index_of("id")?); - assert_eq!(&batch, &result_batch); - files_scanned += 1; - } - assert_eq!(1, files_scanned); - Ok(()) -} - -#[tokio::test] -async fn predicate_on_non_nullable_column_missing_stats() -> Result<(), Box> -{ - let batch_1 = generate_batch(vec![("val", vec!["a", "b", "c"].into_array())])?; - let batch_2 = generate_batch(vec![("val", vec!["d", "e", "f"].into_array())])?; - - let storage = Arc::new(InMemory::new()); - let table_root = "memory:///"; - let actions = [ - r#"{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}"#.to_string(), - r#"{"commitInfo":{"timestamp":1587968586154,"operation":"WRITE","operationParameters":{"mode":"ErrorIfExists","partitionBy":"[]"},"isBlindAppend":true}}"#.to_string(), - r#"{"metaData":{"id":"5fba94ed-9794-4965-ba6e-6ee3c0d22af9","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"val\",\"type\":\"string\",\"nullable\":false,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1587968585495}}"#.to_string(), - // Add one file with stats, one file without - format!(r#"{{"add":{{"path":"{PARQUET_FILE1}","partitionValues":{{}},"size":0,"modificationTime":1587968586000,"dataChange":true, "stats":"{{\"numRecords\":3,\"nullCount\":{{\"val\":0}},\"minValues\":{{\"val\":\"a\"}},\"maxValues\":{{\"val\":\"c\"}}}}"}}}}"#), - format!(r#"{{"add":{{"path":"{PARQUET_FILE2}","partitionValues":{{}},"size":0,"modificationTime":1587968586000,"dataChange":true, "stats":"{{\"numRecords\":3,\"nullCount\":{{}},\"minValues\":{{}},\"maxValues\":{{}}}}"}}}}"#), - ]; - - // Disable writing Parquet statistics so these cannot be used for pruning row groups - let writer_props = WriterProperties::builder() - .set_statistics_enabled(EnabledStatistics::None) - .build(); - - add_commit(table_root, storage.as_ref(), 0, actions.iter().join("\n")).await?; - storage - .put( - &Path::from(PARQUET_FILE1), - record_batch_to_bytes_with_props(&batch_1, writer_props.clone()).into(), - ) - .await?; - storage - .put( - &Path::from(PARQUET_FILE2), - record_batch_to_bytes_with_props(&batch_2, writer_props).into(), - ) - .await?; - - let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); - let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; - - let predicate = Pred::eq(column_expr!("val"), Expr::literal("g")); - let scan = snapshot - .scan_builder() - .with_predicate(Arc::new(predicate)) - .build()?; - - let stream = scan.execute(engine)?; - - let mut files_scanned = 0; - for engine_data in stream { - let result_batch = into_record_batch(engine_data?); - assert_eq!(&batch_2, &result_batch); - files_scanned += 1; - } - // One file is scanned as stats are missing so we don't know the predicate isn't satisfied - assert_eq!(1, files_scanned); - - Ok(()) -} - -#[test] -fn short_dv() -> Result<(), Box> { - let expected = vec![ - "+----+-------+--------------------------+---------------------+", - "| id | value | timestamp | rand |", - "+----+-------+--------------------------+---------------------+", - "| 3 | 3 | 2023-05-31T18:58:33.633Z | 0.7918174793484931 |", - "| 4 | 4 | 2023-05-31T18:58:33.633Z | 0.9281049271981882 |", - "| 5 | 5 | 2023-05-31T18:58:33.633Z | 0.27796520310701633 |", - "| 6 | 6 | 2023-05-31T18:58:33.633Z | 0.15263801464228832 |", - "| 7 | 7 | 2023-05-31T18:58:33.633Z | 0.1981143710215575 |", - "| 8 | 8 | 2023-05-31T18:58:33.633Z | 0.3069439236599195 |", - "| 9 | 9 | 2023-05-31T18:58:33.633Z | 0.5175919190815845 |", - "+----+-------+--------------------------+---------------------+", - ]; - read_table_data_str("./tests/data/with-short-dv/", None, None, expected)?; - Ok(()) -} - -#[test] -fn basic_decimal() -> Result<(), Box> { - let expected = vec![ - "+----------------+---------+--------------+------------------------+", - "| part | col1 | col2 | col3 |", - "+----------------+---------+--------------+------------------------+", - "| -2342342.23423 | -999.99 | -99999.99999 | -9999999999.9999999999 |", - "| 0.00004 | 0.00 | 0.00000 | 0.0000000000 |", - "| 234.00000 | 1.00 | 2.00000 | 3.0000000000 |", - "| 2342222.23454 | 111.11 | 22222.22222 | 3333333333.3333333333 |", - "+----------------+---------+--------------+------------------------+", - ]; - read_table_data_str("./tests/data/basic-decimal-table/", None, None, expected)?; - Ok(()) -} - -#[test] -fn timestamp_ntz() -> Result<(), Box> { - let expected = vec![ - "+----+----------------------------+----------------------------+", - "| id | tsNtz | tsNtzPartition |", - "+----+----------------------------+----------------------------+", - "| 0 | 2021-11-18T02:30:00.123456 | 2021-11-18T02:30:00.123456 |", - "| 1 | 2013-07-05T17:01:00.123456 | 2021-11-18T02:30:00.123456 |", - "| 2 | | 2021-11-18T02:30:00.123456 |", - "| 3 | 2021-11-18T02:30:00.123456 | 2013-07-05T17:01:00.123456 |", - "| 4 | 2013-07-05T17:01:00.123456 | 2013-07-05T17:01:00.123456 |", - "| 5 | | 2013-07-05T17:01:00.123456 |", - "| 6 | 2021-11-18T02:30:00.123456 | |", - "| 7 | 2013-07-05T17:01:00.123456 | |", - "| 8 | | |", - "+----+----------------------------+----------------------------+", - ]; - read_table_data_str( - "./tests/data/data-reader-timestamp_ntz/", - None, - None, - expected, - )?; - Ok(()) -} - -#[test] -fn type_widening_basic() -> Result<(), Box> { - let expected = vec![ - "+---------------------+---------------------+--------------------+----------------+----------------+----------------+----------------------------+", - "| byte_long | int_long | float_double | byte_double | short_double | int_double | date_timestamp_ntz |", - "+---------------------+---------------------+--------------------+----------------+----------------+----------------+----------------------------+", - "| 1 | 2 | 3.4000000953674316 | 5.0 | 6.0 | 7.0 | 2024-09-09T00:00:00 |", - "| 9223372036854775807 | 9223372036854775807 | 1.234567890123 | 1.234567890123 | 1.234567890123 | 1.234567890123 | 2024-09-09T12:34:56.123456 |", - "+---------------------+---------------------+--------------------+----------------+----------------+----------------+----------------------------+", - ]; - let select_cols: Option<&[&str]> = Some(&[ - "byte_long", - "int_long", - "float_double", - "byte_double", - "short_double", - "int_double", - "date_timestamp_ntz", - ]); - - read_table_data_str("./tests/data/type-widening/", select_cols, None, expected) -} - -#[test] -fn type_widening_decimal() -> Result<(), Box> { - let expected = vec![ - "+----------------------------+-------------------------------+--------------+---------------+--------------+----------------------+", - "| decimal_decimal_same_scale | decimal_decimal_greater_scale | byte_decimal | short_decimal | int_decimal | long_decimal |", - "+----------------------------+-------------------------------+--------------+---------------+--------------+----------------------+", - "| 123.45 | 67.89000 | 1.0 | 2.0 | 3.0 | 4.0 |", - "| 12345678901234.56 | 12345678901.23456 | 123.4 | 12345.6 | 1234567890.1 | 123456789012345678.9 |", - "+----------------------------+-------------------------------+--------------+---------------+--------------+----------------------+", - ]; - let select_cols: Option<&[&str]> = Some(&[ - "decimal_decimal_same_scale", - "decimal_decimal_greater_scale", - "byte_decimal", - "short_decimal", - "int_decimal", - "long_decimal", - ]); - read_table_data_str("./tests/data/type-widening/", select_cols, None, expected) -} - -// Verify that predicates over invalid/missing columns do not cause skipping. -#[test] -fn predicate_references_invalid_missing_column() -> Result<(), Box> { - // Attempted skipping over a logically valid but physically missing column. We should be able to - // skip the data file because the missing column is inferred to be all-null. - // - // WARNING: https://github.com/delta-io/delta-kernel-rs/issues/434 -- currently disabled. - // - //let expected = vec![ - // "+--------+", - // "| chrono |", - // "+--------+", - // "+--------+", - //]; - let columns = &["chrono", "missing"]; - let expected = vec![ - "+-------------------------------------------------------------------------------------------+---------+", - "| chrono | missing |", - "+-------------------------------------------------------------------------------------------+---------+", - "| {date32: 1971-01-01, timestamp: 1970-02-01T08:00:00Z, timestamp_ntz: 1970-01-02T00:00:00} | |", - "| {date32: 1971-01-02, timestamp: 1970-02-01T09:00:00Z, timestamp_ntz: 1970-01-02T00:01:00} | |", - "| {date32: 1971-01-03, timestamp: 1970-02-01T10:00:00Z, timestamp_ntz: 1970-01-02T00:02:00} | |", - "| {date32: 1971-01-04, timestamp: 1970-02-01T11:00:00Z, timestamp_ntz: 1970-01-02T00:03:00} | |", - "| {date32: 1971-01-05, timestamp: 1970-02-01T12:00:00Z, timestamp_ntz: 1970-01-02T00:04:00} | |", - "+-------------------------------------------------------------------------------------------+---------+", - ]; - let predicate = column_expr!("missing").lt(Expr::literal(10i64)); - read_table_data_str( - "./tests/data/parquet_row_group_skipping/", - Some(columns), - Some(predicate), - expected, - )?; - - // Attempted skipping over an invalid (logically missing) column. Ideally this should throw a - // query error, but at a minimum it should not cause incorrect data skipping. - let expected = vec![ - "+-------------------------------------------------------------------------------------------+", - "| chrono |", - "+-------------------------------------------------------------------------------------------+", - "| {date32: 1971-01-01, timestamp: 1970-02-01T08:00:00Z, timestamp_ntz: 1970-01-02T00:00:00} |", - "| {date32: 1971-01-02, timestamp: 1970-02-01T09:00:00Z, timestamp_ntz: 1970-01-02T00:01:00} |", - "| {date32: 1971-01-03, timestamp: 1970-02-01T10:00:00Z, timestamp_ntz: 1970-01-02T00:02:00} |", - "| {date32: 1971-01-04, timestamp: 1970-02-01T11:00:00Z, timestamp_ntz: 1970-01-02T00:03:00} |", - "| {date32: 1971-01-05, timestamp: 1970-02-01T12:00:00Z, timestamp_ntz: 1970-01-02T00:04:00} |", - "+-------------------------------------------------------------------------------------------+", - ]; - let predicate = column_expr!("invalid").lt(Expr::literal(10)); - read_table_data_str( - "./tests/data/parquet_row_group_skipping/", - Some(columns), - Some(predicate), - expected, - ) - .expect_err("unknown column"); - Ok(()) -} - -// Note: This test is disabled for windows because it creates a directory with name -// `time=1971-07-22T03:06:40.000000Z`. This is disallowed in windows due to having a `:` in -// the name. -#[cfg(not(windows))] -#[test] -fn timestamp_partitioned_table() -> Result<(), Box> { - let expected = vec![ - "+----+-----+---+----------------------+", - "| id | x | s | time |", - "+----+-----+---+----------------------+", - "| 1 | 0.5 | | 1971-07-22T03:06:40Z |", - "+----+-----+---+----------------------+", - ]; - let test_name = "timestamp-partitioned-table"; - let test_dir = load_test_data("./tests/data", test_name).unwrap(); - let test_path = test_dir.path().join(test_name); - read_table_data_str(test_path.to_str().unwrap(), None, None, expected) -} - -#[test] -fn compacted_log_files_table() -> Result<(), Box> { - let expected = vec![ - "+----+--------------------+", - "| id | comment |", - "+----+--------------------+", - "| 0 | new |", - "| 1 | after-large-delete |", - "| 2 | |", - "| 10 | merge1-insert |", - "| 12 | merge2-insert |", - "+----+--------------------+", - ]; - let test_name = "compacted-log-files-table"; - let test_dir = load_test_data("./tests/data", test_name).unwrap(); - let test_path = test_dir.path().join(test_name); - read_table_data_str(test_path.to_str().unwrap(), None, None, expected) -} - -#[test] -fn unshredded_variant_table() -> Result<(), Box> { - let expected = include!("data/unshredded-variant.expected.in"); - let test_name = "unshredded-variant"; - let test_dir = load_test_data("./tests/data", test_name).unwrap(); - let test_path = test_dir.path().join(test_name); - read_table_data_str(test_path.to_str().unwrap(), None, None, expected) -} - -#[tokio::test] -async fn test_row_index_metadata_column() -> Result<(), Box> { - // Setup up an in-memory table with different numbers of rows in each file - let batch1 = generate_batch(vec![ - ("id", vec![1i32, 2, 3, 4, 5].into_array()), - ("value", vec!["a", "b", "c", "d", "e"].into_array()), - ])?; - let batch2 = generate_batch(vec![ - ("id", vec![10i32, 20, 30].into_array()), - ("value", vec!["x", "y", "z"].into_array()), - ])?; - let batch3 = generate_batch(vec![ - ("id", vec![100i32, 200, 300, 400].into_array()), - ("value", vec!["p", "q", "r", "s"].into_array()), - ])?; - - let file_size1 = record_batch_to_bytes(&batch1).len() as u64; - let file_size2 = record_batch_to_bytes(&batch2).len() as u64; - let file_size3 = record_batch_to_bytes(&batch3).len() as u64; - let storage = Arc::new(InMemory::new()); - let table_root = "memory:///"; - add_commit( - table_root, - storage.as_ref(), - 0, - actions_to_string(vec![ - TestAction::Metadata, - TestAction::AddWithSize(PARQUET_FILE1.to_string(), file_size1), - TestAction::AddWithSize(PARQUET_FILE2.to_string(), file_size2), - TestAction::AddWithSize(PARQUET_FILE3.to_string(), file_size3), - ]), - ) - .await?; - - for (parquet_file, batch) in [ - (PARQUET_FILE1, &batch1), - (PARQUET_FILE2, &batch2), - (PARQUET_FILE3, &batch3), - ] { - storage - .put( - &Path::from(parquet_file), - record_batch_to_bytes(batch).into(), - ) - .await?; - } - - let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); - - // Create a schema that includes a row index metadata column - let schema = Arc::new(StructType::try_new([ - StructField::nullable("id", DataType::INTEGER), - StructField::create_metadata_column("row_index", MetadataColumnSpec::RowIndex), - StructField::nullable("value", DataType::STRING), - ])?); - - let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; - let scan = snapshot.scan_builder().with_schema(schema).build()?; - - let mut file_count = 0; - let expected_row_counts = [5, 3, 4]; - let stream = scan.execute(engine.clone())?; - - for data in stream { - let batch = into_record_batch(data?); - file_count += 1; - - // Verify the schema structure - assert_eq!(batch.num_columns(), 3, "Expected 3 columns in the batch"); - assert_eq!( - batch.schema().field(0).name(), - "id", - "First column should be 'id'" - ); - assert_eq!( - batch.schema().field(1).name(), - "row_index", - "Second column should be 'row_index'" - ); - assert_eq!( - batch.schema().field(2).name(), - "value", - "Third column should be 'value'" - ); - - // Each file should have row indexes starting from 0 (file-local indexing) - let row_index_array = batch.column(1).as_primitive::(); - let expected_values: Vec = (0..batch.num_rows() as i64).collect(); - assert_eq!( - row_index_array.values().to_vec(), - expected_values, - "Row index values incorrect for file {} (expected {} rows)", - file_count, - expected_row_counts[file_count - 1] - ); - } - - assert_eq!(file_count, 3, "Expected to scan 3 files"); - Ok(()) -} - -#[tokio::test] -async fn test_file_path_metadata_column() -> Result<(), Box> { - use delta_kernel::arrow::array::{Array, StringArray}; - - // Set up an in-memory table with multiple data files - let batch1 = generate_batch(vec![ - ("id", vec![1i32, 2, 3].into_array()), - ("value", vec!["a", "b", "c"].into_array()), - ])?; - let batch2 = generate_batch(vec![ - ("id", vec![10i32, 20].into_array()), - ("value", vec!["x", "y"].into_array()), - ])?; - - let file_size1 = record_batch_to_bytes(&batch1).len() as u64; - let file_size2 = record_batch_to_bytes(&batch2).len() as u64; - let storage = Arc::new(InMemory::new()); - let table_root = "memory:///"; - add_commit( - table_root, - storage.as_ref(), - 0, - actions_to_string(vec![ - TestAction::Metadata, - TestAction::AddWithSize(PARQUET_FILE1.to_string(), file_size1), - TestAction::AddWithSize(PARQUET_FILE2.to_string(), file_size2), - ]), - ) - .await?; - - for (parquet_file, batch) in [(PARQUET_FILE1, &batch1), (PARQUET_FILE2, &batch2)] { - storage - .put( - &Path::from(parquet_file), - record_batch_to_bytes(batch).into(), - ) - .await?; - } - - let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); - - // Create a schema that includes the file path metadata column - let schema = Arc::new(StructType::try_new([ - StructField::nullable("id", DataType::INTEGER), - StructField::create_metadata_column("_file", MetadataColumnSpec::FilePath), - StructField::nullable("value", DataType::STRING), - ])?); - - let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; - let scan = snapshot.scan_builder().with_schema(schema).build()?; - - let mut file_count = 0; - let expected_files = [PARQUET_FILE1, PARQUET_FILE2]; - let expected_row_counts = [3, 2]; - let stream = scan.execute(engine.clone())?; - - for data in stream { - let batch = into_record_batch(data?); - - // Verify the schema structure - assert_eq!(batch.num_columns(), 3, "Expected 3 columns in the batch"); - assert_eq!( - batch.schema().field(0).name(), - "id", - "First column should be 'id'" - ); - assert_eq!( - batch.schema().field(1).name(), - "_file", - "Second column should be '_file'" - ); - assert_eq!( - batch.schema().field(2).name(), - "value", - "Third column should be 'value'" - ); - - // Verify the file path column contains the expected file name - let file_path_array = batch.column(1); - let expected_file_name = expected_files[file_count]; - let expected_path = format!("{table_root}{expected_file_name}"); - - // The file path array should be a plain StringArray with the path repeated for each row. - let string_array = file_path_array - .as_any() - .downcast_ref::() - .expect("File path column should be a StringArray"); - - assert_eq!( - string_array.len(), - expected_row_counts[file_count], - "File {} should have {} rows", - expected_file_name, - expected_row_counts[file_count] - ); - assert!( - string_array - .iter() - .all(|v| v == Some(expected_path.as_str())), - "All rows should contain file path '{expected_path}'" - ); - - file_count += 1; - } - - assert_eq!(file_count, 2, "Expected to scan 2 files"); - Ok(()) -} - -#[tokio::test] -async fn test_unsupported_metadata_columns() -> Result<(), Box> { - // Prepare an in-memory table with some data - let batch = generate_simple_batch()?; - let storage = Arc::new(InMemory::new()); - let table_root = "memory:///"; - add_commit( - table_root, - storage.as_ref(), - 0, - actions_to_string(vec![ - TestAction::Metadata, - TestAction::Add(PARQUET_FILE1.to_string()), - ]), - ) - .await?; - storage - .put( - &Path::from(PARQUET_FILE1), - record_batch_to_bytes(&batch).into(), - ) - .await?; - - let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); - - // Test that unsupported metadata columns fail with appropriate errors - let test_cases = [ - ( - "row_id", - MetadataColumnSpec::RowId, - "Row ids are not enabled on this table", - ), - ( - "row_commit_version", - MetadataColumnSpec::RowCommitVersion, - "Row commit versions not supported", - ), - ]; - - for (column_name, metadata_spec, error_text) in test_cases { - let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; - let schema = Arc::new(StructType::try_new([ - StructField::nullable("id", DataType::INTEGER), - StructField::create_metadata_column(column_name, metadata_spec), - ])?); - - let scan_err = snapshot - .scan_builder() - .with_schema(schema) - .build() - .unwrap_err(); - let error_msg = scan_err.to_string(); - assert!( - error_msg.contains(error_text), - "Expected {error_msg} to contain {error_text}" - ); - } - - Ok(()) -} - -#[tokio::test] -async fn test_invalid_files_are_skipped() -> Result<(), Box> { - let batch = generate_simple_batch()?; - let storage = Arc::new(InMemory::new()); - let table_root = "memory:///"; - add_commit( - table_root, - storage.as_ref(), - 0, - actions_to_string(vec![ - TestAction::Metadata, - TestAction::Add(PARQUET_FILE1.to_string()), - TestAction::Add(PARQUET_FILE2.to_string()), - ]), - ) - .await?; - storage - .put( - &Path::from(PARQUET_FILE1), - record_batch_to_bytes(&batch).into(), - ) - .await?; - storage - .put( - &Path::from(PARQUET_FILE2), - record_batch_to_bytes(&batch).into(), - ) - .await?; - - let engine = Arc::new(DefaultEngineBuilder::new(storage.clone()).build()); - - let invalid_files = [ - "_delta_log/0.zip", - "_delta_log/_copy_into_log/0.zip", - "_delta_log/_ignore_me/00000000000000000000.json", - "_delta_log/_and_me/00000000000000000000.checkpoint.parquet", - "_delta_log/02184.json", - "_delta_log/0x000000000000000000.checkpoint.parquet", - "00000000000000000000.json", - "_delta_log/_staged_commits/_staged_commits/00000000000000000000.3a0d65cd-4056-49b8-937b-95f9e3ee90e5.json", - "_delta_log/my_random_dir/_staged_commits/00000000000000000000.3a0d65cd-4056-49b8-937b-95f9e3ee90e5.json", - "_delta_log/my_random_dir/_delta_log/_staged_commits/00000000000000000000.3a0d65cd-4056-49b8-937b-95f9e3ee90e5.json", - "_delta_log/_delta_log/00000000000000000000.json", - "_delta_log/_delta_log/00000000000000000000.checkpoint.parquet", - "_delta_log/something/_delta_log/00000000000000000000.crc", - "_delta_log/something/_delta_log/00000000000000000000.json", - "_delta_log/something/_delta_log/00000000000000000000.checkpoint.parquet", - ]; - - fn get_file_path_for_test(path: &ParsedLogPath) -> &str { - &path.location.location.as_str()[10..] - } - - fn ensure_segment_does_not_contain(invalid_files: &[&str], segment: &LogSegment) { - assert!( - !segment.listed.ascending_commit_files.iter().any(|p| { - let test_path = get_file_path_for_test(p); - invalid_files.contains(&test_path) - }), - "ascending_commit_files contained invalid file" - ); - assert!( - !segment.listed.ascending_compaction_files.iter().any(|p| { - let test_path = get_file_path_for_test(p); - invalid_files.contains(&test_path) - }), - "ascending_compaction_files contained invalid file" - ); - assert!( - !segment.listed.checkpoint_parts.iter().any(|p| { - let test_path = get_file_path_for_test(p); - invalid_files.contains(&test_path) - }), - "checkpoint_parts contained invalid file" - ); - if let Some(ref crc) = segment.listed.latest_crc_file { - assert!( - !invalid_files.contains(&get_file_path_for_test(crc)), - "Latest crc contained invalid file" - ); - } - if let Some(ref latest_commit) = segment.listed.latest_commit_file { - assert!( - !invalid_files.contains(&get_file_path_for_test(latest_commit)), - "Latest commit contained invalid file" - ); - } - } - - for invalid_file in invalid_files.iter() { - let invalid_path = Path::from(*invalid_file); - storage.put(&invalid_path, vec![1u8].into()).await?; - let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; - ensure_segment_does_not_contain(&invalid_files, snapshot.log_segment()); - storage.delete(&invalid_path).await?; - } - - // final test with _all_ the files we should ignore - for invalid_file in invalid_files.iter() { - let invalid_path = Path::from(*invalid_file); - storage.put(&invalid_path, vec![1u8].into()).await?; - } - let snapshot = Snapshot::builder_for(table_root).build(engine.as_ref())?; - ensure_segment_does_not_contain(&invalid_files, snapshot.log_segment()); - - Ok(()) -} diff --git a/kernel/tests/scripts/coverage-compare.py b/kernel/tests/scripts/coverage-compare.py new file mode 100755 index 0000000000..6f2fbf7a37 --- /dev/null +++ b/kernel/tests/scripts/coverage-compare.py @@ -0,0 +1,373 @@ +#!/usr/bin/env python3 +""" +Compare test coverage between the current branch and a base branch (default: main). + +Accepts one or more test filters. Both branches are built once (the slow step), +then each filter is run as a separate test invocation (fast) so you pay the +compilation cost only once regardless of how many filters you provide. + +Usage: + python3 scripts/coverage-compare.py [filter2 ...] + + Each filter is passed to `cargo nextest run` as a test name substring/regex + (same syntax as `cargo test `). If no filters are passed, all tests are run. + +Environment: + COVERAGE_BASE_BRANCH -- branch to compare against (default: main) + +Prerequisites: + cargo install cargo-llvm-cov + rustup component add llvm-tools-preview + +Exit codes: + 0 -- no differences, or only newly-gained coverage (across all filters) + 1 -- coverage was lost for at least one filter + 2 -- script error (missing tool, git failure, etc.) +""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +from pathlib import Path + + +# === Coverage collection === + +_CARGO_COMMON = [ + "cargo", "llvm-cov", "nextest", + "--workspace", "--all-features", + # Benchmarks unpack large tarballs into the source tree during their + # build script; exclude them to avoid filling the worktree directory. + "--exclude", "delta_kernel_benchmarks", + # Without --no-clean, cargo-llvm-cov wipes instrumented build artefacts + # before every run, forcing a full recompile each time. + "--no-clean", +] + + +def make_env(target_dir: str) -> dict[str, str]: + """Build an environment that redirects all cargo/rustc output to target_dir. + + rustc writes temporary files (e.g. symbols.o) to TMPDIR even when + CARGO_TARGET_DIR is set, so both are redirected to keep everything off /tmp. + """ + env = os.environ.copy() + env["CARGO_TARGET_DIR"] = target_dir + tmp_dir = os.path.join(target_dir, "tmp") + os.makedirs(tmp_dir, exist_ok=True) + env["TMPDIR"] = tmp_dir + return env + + +def clean_coverage_data(target_dir: str) -> None: + """Delete all profraw and profdata files before a test run. + + --no-clean tells cargo-llvm-cov to skip its own cleanup so the compiled + binaries are preserved between filter runs. However, it also skips + removing coverage data files, so we do it ourselves to ensure each filter + gets a fresh, uncontaminated report. + """ + for ext in ("*.profraw", "*.profdata"): + for p in Path(target_dir).rglob(ext): + p.unlink() + + +def run_one_filter( + work_dir: str, + output_file: str, + filter_str: str, + target_dir: str, +) -> bool: + """Run the tests matching filter_str and write a coverage JSON to output_file. + + The instrumented binaries must already exist (call build_only first). + Stale profraw files are removed before running so counts are not inflated + by prior runs. + + Returns True on success, False if no tests matched the filter. + """ + clean_coverage_data(target_dir) + cmd = _CARGO_COMMON + ["--json", f"--output-path={output_file}", "--", filter_str] + result = subprocess.run(cmd, cwd=work_dir, env=make_env(target_dir)) + if result.returncode == 0: + return True + # nextest exits 4 when no tests match the filter. + if result.returncode == 4: + print(f" (no matching tests in {work_dir})", flush=True) + return False + print( + f"cargo llvm-cov failed with exit code {result.returncode}.", + file=sys.stderr, + ) + sys.exit(2) + + +# === Coverage loading === + +# Each segment in the llvm-cov JSON is a 6-element list: +# [line, col, count, has_count, is_region_entry, is_gap_region] +_SEG_LINE = 0 +_SEG_COL = 1 +_SEG_COUNT = 2 +_SEG_HAS_COUNT = 3 +_SEG_IS_GAP = 5 + +CoverageMap = dict[str, dict[tuple[int, int], int]] + + +def load_coverage( + json_file: str, + strip_prefix: str, + skip_files: frozenset[str] = frozenset(), + skip_prefixes: frozenset[str] = frozenset(), +) -> CoverageMap: + """Parse an llvm-cov JSON export into a map of relative path -> segment counts. + + Only segments with has_count=True and is_gap_region=False are included. + Files outside strip_prefix (e.g. stdlib, .cargo/registry) are skipped. + Files whose relative path is in skip_files are also skipped; pass the set + of files changed between branches to avoid false positives from line-number + drift in modified files. + Files whose relative path starts with any entry in skip_prefixes are also + skipped; used to exclude proc-macro crates whose coverage is captured at + compile time rather than test-run time. + """ + with open(json_file) as f: + data = json.load(f) + + prefix = strip_prefix.rstrip("/") + "/" + coverage: CoverageMap = {} + + for entry in data.get("data", []): + for file_data in entry.get("files", []): + filename: str = file_data["filename"] + if not filename.startswith(prefix): + continue + rel = filename[len(prefix):] + if rel in skip_files: + continue + if any(rel.startswith(p) for p in skip_prefixes): + continue + segments: dict[tuple[int, int], int] = {} + for seg in file_data.get("segments", []): + if seg[_SEG_HAS_COUNT] and not seg[_SEG_IS_GAP]: + segments[(seg[_SEG_LINE], seg[_SEG_COL])] = seg[_SEG_COUNT] + if segments: + coverage[rel] = segments + + return coverage + + +# === Comparison === + +FilterResult = tuple[list[tuple[str, tuple[int, int], int, int]], # lost + list[tuple[str, tuple[int, int], int, int]]] # gained + + +def compare(base: CoverageMap, current: CoverageMap) -> FilterResult: + """Return (lost, gained) segment lists. + + Only LOST (>0 -> 0) and GAINED (0 -> >0) segments are returned. Pure count + changes in already-covered segments are ignored: execution counts are + influenced by non-deterministic background work (thread pool startup, binary + initialisation across nextest workers) and are not a reliable signal. + """ + lost: list[tuple[str, tuple[int, int], int, int]] = [] + gained: list[tuple[str, tuple[int, int], int, int]] = [] + + for rel_path in sorted(set(base) | set(current)): + base_segs = base.get(rel_path, {}) + curr_segs = current.get(rel_path, {}) + for seg in sorted(set(base_segs) | set(curr_segs)): + b = base_segs.get(seg, 0) + c = curr_segs.get(seg, 0) + if b > 0 and c == 0: + lost.append((rel_path, seg, b, c)) + elif b == 0 and c > 0: + gained.append((rel_path, seg, b, c)) + + return lost, gained + + +# === Reporting === + +def print_filter_result(filter_str: str, lost: list, gained: list) -> None: + print(f"\n{'='*70}") + print(f"Filter: {filter_str!r}") + print(f"{'='*70}") + + if not lost and not gained: + print(" No coverage differences.") + return + + # Group by file for readability. + def print_group(entries: list, label: str) -> None: + by_file: dict[str, list] = {} + for path, seg, b, c in entries: + by_file.setdefault(path, []).append((seg, b, c)) + for path in sorted(by_file): + print(f"\n {label} in {path}:") + for (line, col), b, c in sorted(by_file[path]): + print(f" line {line}:{col:<3} base={b:>8} current={c:>8}") + + if lost: + print_group(lost, "LOST") + if gained: + print_group(gained, "GAINED") + + print() + print(f" Lost: {len(lost)} Gained: {len(gained)}") + + +def print_summary(results: dict[str, FilterResult]) -> None: + print(f"\n{'='*70}") + print("Summary") + print(f"{'='*70}") + + any_lost = False + for filter_str, (lost, gained) in results.items(): + status = "OK" if not lost else "LOST" + print(f" [{status:4}] lost={len(lost):3} gained={len(gained):3} {filter_str!r}") + if lost: + any_lost = True + + print() + if any_lost: + print("FAIL: coverage was lost for one or more filters.") + else: + print("PASS: no coverage lost.") + + +# === Entry point === + +def check_prerequisites() -> None: + if subprocess.run(["cargo", "llvm-cov", "--version"], capture_output=True).returncode != 0: + print( + "cargo-llvm-cov is not installed. Install it with:\n" + " cargo install cargo-llvm-cov\n" + " rustup component add llvm-tools-preview", + file=sys.stderr, + ) + sys.exit(2) + + +def git_repo_root() -> str: + return subprocess.check_output( + ["git", "rev-parse", "--show-toplevel"], text=True + ).strip() + + +# Proc-macro crates execute at compile time, not at test-run time. Their +# coverage is an artefact of the build phase and must not be compared against +# test-run coverage. +_PROC_MACRO_PREFIXES: frozenset[str] = frozenset([ + "derive-macros/src/", + "ffi-proc-macros/src/", +]) + + +def main() -> int: + filters = sys.argv[1:] + if not filters: + filters = [""] + + base_branch = os.environ.get("COVERAGE_BASE_BRANCH", "main") + + check_prerequisites() + repo_root = git_repo_root() + + branch_label = subprocess.check_output( + ["git", "rev-parse", "--abbrev-ref", "HEAD"], text=True + ).strip() + + # Files changed between the current branch and base. Comparing coverage at + # line granularity is unreliable for these files because inserting or + # removing lines shifts all subsequent line numbers, causing the same + # (file, line) pair to refer to different code in each branch. + changed_files = frozenset(subprocess.check_output( + ["git", "diff", f"{base_branch}...HEAD", "--name-only"], + cwd=repo_root, text=True, + ).split()) + + + # All artefacts live under target/ to avoid filling /tmp. The worktree also + # lives here because the benchmarks build script unpacks large tarballs into + # the source tree. + cov_dir = os.path.join(repo_root, "target", "cov-compare") + os.makedirs(cov_dir, exist_ok=True) + + worktree_path = os.path.join(cov_dir, "worktree") + current_target = os.path.join(cov_dir, "build-current") + base_target = os.path.join(cov_dir, "build-base") + + # Remove stale worktree from a previous interrupted run, if any. + if os.path.exists(worktree_path): + subprocess.run( + ["git", "worktree", "remove", "--force", worktree_path], + cwd=repo_root, capture_output=True, + ) + + print(f"Setting up worktree for '{base_branch}' ...", flush=True) + subprocess.run( + ["git", "worktree", "add", "--quiet", worktree_path, base_branch], + cwd=repo_root, check=True, + ) + + try: + # Run all filters on the current branch first, then all on the base + # branch. Each branch is compiled on the first filter invocation; + # --no-clean keeps the instrumented artefacts so all subsequent + # filters on the same branch skip recompilation. + + print(f"\nRunning {len(filters)} filter(s) on '{branch_label}' ...", flush=True) + current_ok_map: dict[str, bool] = {} + for i, filter_str in enumerate(filters, 1): + print(f" [{i}/{len(filters)}] {filter_str!r}", flush=True) + current_json = os.path.join(cov_dir, f"current-{i}.json") + current_ok_map[filter_str] = run_one_filter( + repo_root, current_json, filter_str, current_target + ) + + print(f"\nRunning {len(filters)} filter(s) on '{base_branch}' ...", flush=True) + base_ok_map: dict[str, bool] = {} + for i, filter_str in enumerate(filters, 1): + print(f" [{i}/{len(filters)}] {filter_str!r}", flush=True) + base_json = os.path.join(cov_dir, f"base-{i}.json") + base_ok_map[filter_str] = run_one_filter( + worktree_path, base_json, filter_str, base_target + ) + + results: dict[str, FilterResult] = {} + for i, filter_str in enumerate(filters, 1): + current_ok = current_ok_map[filter_str] + base_ok = base_ok_map[filter_str] + if not current_ok and not base_ok: + results[filter_str] = ([], []) + continue + current_json = os.path.join(cov_dir, f"current-{i}.json") + base_json = os.path.join(cov_dir, f"base-{i}.json") + base_cov = load_coverage(base_json, worktree_path, changed_files, _PROC_MACRO_PREFIXES) if base_ok else {} + current_cov = load_coverage(current_json, repo_root, changed_files, _PROC_MACRO_PREFIXES) if current_ok else {} + results[filter_str] = compare(base_cov, current_cov) + + finally: + subprocess.run( + ["git", "worktree", "remove", "--force", worktree_path], + cwd=repo_root, capture_output=True, + ) + + # === Report === + for filter_str, (lost, gained) in results.items(): + print_filter_result(filter_str, lost, gained) + + print_summary(results) + + any_lost = any(lost for lost, _ in results.values()) + return 1 if any_lost else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/kernel/tests/v2_checkpoints.rs b/kernel/tests/v2_checkpoints.rs deleted file mode 100644 index 82e51bcbf5..0000000000 --- a/kernel/tests/v2_checkpoints.rs +++ /dev/null @@ -1,303 +0,0 @@ -use std::sync::Arc; - -use delta_kernel::arrow::array::{Int32Array, RecordBatch}; -use delta_kernel::committer::FileSystemCommitter; -use delta_kernel::schema::{DataType, StructField, StructType}; -use delta_kernel::transaction::create_table::create_table; -use delta_kernel::transaction::CommitResult; -use delta_kernel::{DeltaResult, Snapshot}; - -mod common; - -use itertools::Itertools; -use test_utils::{insert_data, load_test_data, read_scan, test_table_setup_mt}; - -fn read_v2_checkpoint_table(test_name: impl AsRef) -> DeltaResult> { - let test_dir = load_test_data("tests/data", test_name.as_ref()).unwrap(); - let test_path = test_dir.path().join(test_name.as_ref()); - - let url = - delta_kernel::try_parse_uri(test_path.to_str().expect("table path to string")).unwrap(); - let engine = test_utils::create_default_engine(&url)?; - let snapshot = Snapshot::builder_for(url).build(engine.as_ref()).unwrap(); - let scan = snapshot.scan_builder().build()?; - let batches = read_scan(&scan, engine)?; - - Ok(batches) -} - -fn test_v2_checkpoint_with_table( - table_name: &str, - mut expected_table: Vec, -) -> DeltaResult<()> { - let batches = read_v2_checkpoint_table(table_name)?; - - sort_lines!(expected_table); - assert_batches_sorted_eq!(expected_table, &batches); - Ok(()) -} - -/// Helper function to convert string slice vectors to String vectors -fn to_string_vec(string_slice_vec: Vec<&str>) -> Vec { - string_slice_vec - .into_iter() - .map(|s| s.to_string()) - .collect() -} - -fn generate_sidecar_expected_data() -> Vec { - let header = vec![ - "+-----+".to_string(), - "| id |".to_string(), - "+-----+".to_string(), - ]; - - // Generate rows for different ranges - let generate_rows = |count: usize| -> Vec { - (0..count) - .map(|id| format!("| {: Vec { - to_string_vec(vec![ - "+----+", - "| id |", - "+----+", - "| 0 |", - "| 1 |", - "| 2 |", - "| 3 |", - "| 4 |", - "| 5 |", - "| 6 |", - "| 7 |", - "| 8 |", - "| 9 |", - "+----+", - ]) -} - -// Rustfmt is disabled to maintain the readability of the expected table -#[rustfmt::skip] -fn get_classic_checkpoint_table() -> Vec { - to_string_vec(vec![ - "+----+", - "| id |", - "+----+", - "| 0 |", - "| 1 |", - "| 2 |", - "| 3 |", - "| 4 |", - "| 5 |", - "| 6 |", - "| 7 |", - "| 8 |", - "| 9 |", - "| 10 |", - "| 11 |", - "| 12 |", - "| 13 |", - "| 14 |", - "| 15 |", - "| 16 |", - "| 17 |", - "| 18 |", - "| 19 |", - "+----+", - ]) -} - -// Rustfmt is disabled to maintain the readability of the expected table -#[rustfmt::skip] -fn get_without_sidecars_table() -> Vec { - to_string_vec(vec![ - "+------+", - "| id |", - "+------+", - "| 0 |", - "| 1 |", - "| 2 |", - "| 3 |", - "| 4 |", - "| 5 |", - "| 6 |", - "| 7 |", - "| 8 |", - "| 9 |", - "| 2718 |", - "+------+", - ]) -} - -/// The test cases below are derived from delta-spark's `CheckpointSuite`. -/// -/// These tests are converted from delta-spark using the following process: -/// 1. Specific test cases of interest in `delta-spark` were modified to persist their generated tables -/// 2. These tables were compressed into `.tar.zst` archives and copied to delta-kernel-rs -/// 3. Each test loads a stored table, scans it, and asserts that the returned table state -/// matches the expected state derived from the corresponding table insertions in `delta-spark` -/// -/// The following is the ported list of `delta-spark` tests -> `delta-kernel-rs` tests: -/// -/// - `multipart v2 checkpoint` -> `v2_checkpoints_json_with_sidecars` -/// - `multipart v2 checkpoint` -> `v2_checkpoints_parquet_with_sidecars` -/// - `All actions in V2 manifest` -> `v2_checkpoints_json_without_sidecars` -/// - `All actions in V2 manifest` -> `v2_checkpoints_parquet_without_sidecars` -/// - `V2 Checkpoint compat file equivalency to normal V2 Checkpoint` -> `v2_classic_checkpoint_json` -/// - `V2 Checkpoint compat file equivalency to normal V2 Checkpoint` -> `v2_classic_checkpoint_parquet` -/// - `last checkpoint contains correct schema for v1/v2 Checkpoints` -> `v2_checkpoints_json_with_last_checkpoint` -/// - `last checkpoint contains correct schema for v1/v2 Checkpoints` -> `v2_checkpoints_parquet_with_last_checkpoint` -#[test] -fn v2_checkpoints_json_with_sidecars() -> DeltaResult<()> { - test_v2_checkpoint_with_table( - "v2-checkpoints-json-with-sidecars", - generate_sidecar_expected_data(), - ) -} - -#[test] -fn v2_checkpoints_parquet_with_sidecars() -> DeltaResult<()> { - test_v2_checkpoint_with_table( - "v2-checkpoints-parquet-with-sidecars", - generate_sidecar_expected_data(), - ) -} - -#[test] -fn v2_checkpoints_json_without_sidecars() -> DeltaResult<()> { - test_v2_checkpoint_with_table( - "v2-checkpoints-json-without-sidecars", - get_without_sidecars_table(), - ) -} - -#[test] -fn v2_checkpoints_parquet_without_sidecars() -> DeltaResult<()> { - test_v2_checkpoint_with_table( - "v2-checkpoints-parquet-without-sidecars", - get_without_sidecars_table(), - ) -} - -#[test] -fn v2_classic_checkpoint_json() -> DeltaResult<()> { - test_v2_checkpoint_with_table("v2-classic-checkpoint-json", get_classic_checkpoint_table()) -} - -#[test] -fn v2_classic_checkpoint_parquet() -> DeltaResult<()> { - test_v2_checkpoint_with_table( - "v2-classic-checkpoint-parquet", - get_classic_checkpoint_table(), - ) -} - -#[test] -fn v2_checkpoints_json_with_last_checkpoint() -> DeltaResult<()> { - test_v2_checkpoint_with_table( - "v2-checkpoints-json-with-last-checkpoint", - get_simple_id_table(), - ) -} - -#[test] -fn v2_checkpoints_parquet_with_last_checkpoint() -> DeltaResult<()> { - test_v2_checkpoint_with_table( - "v2-checkpoints-parquet-with-last-checkpoint", - get_simple_id_table(), - ) -} - -/// Tests that writing a V2 checkpoint to parquet succeeds. -/// -/// V2 checkpoints include a checkpointMetadata batch in addition to the regular action -/// batches. All batches in a parquet file must share the same schema. This test verifies -/// that `snapshot.checkpoint()` can write a V2 checkpoint without schema mismatch errors. -#[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn test_v2_checkpoint_parquet_write() -> DeltaResult<()> { - let (_temp_dir, table_path, engine) = test_table_setup_mt()?; - let table_url = delta_kernel::try_parse_uri(&table_path)?; - - let schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "value", - DataType::INTEGER, - )])?); - let _ = create_table(&table_path, schema.clone(), "Test/1.0") - .with_table_properties([("delta.feature.v2Checkpoint", "supported")]) - .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? - .commit(engine.as_ref())?; - - // Commit an add action via the transaction API so the checkpoint has action batches - let snapshot0 = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; - let result = insert_data( - snapshot0, - &engine, - vec![Arc::new(Int32Array::from(vec![1]))], - ) - .await?; - - let CommitResult::CommittedTransaction(committed) = result else { - panic!("Expected CommittedTransaction"); - }; - - let snapshot = committed - .post_commit_snapshot() - .expect("expected post-commit snapshot"); - - // This writes to parquet — will fail if the checkpointMetadata batch has a different - // schema than the action batches. - snapshot.clone().checkpoint(engine.as_ref())?; - - // Verify the checkpoint was written and is used by a fresh snapshot - let snapshot2 = Snapshot::builder_for(table_url).build(engine.as_ref())?; - assert_eq!(snapshot2.version(), 1); - let log_segment = snapshot2.log_segment(); - assert!( - !log_segment.listed.checkpoint_parts.is_empty(), - "expected snapshot to use the written checkpoint, but checkpoint_parts is empty" - ); - assert_eq!( - log_segment.checkpoint_version, - Some(1), - "expected checkpoint at version 1" - ); - assert!( - log_segment.listed.ascending_commit_files.is_empty(), - "expected no commit files after checkpoint, but found: {:?}", - log_segment.listed.ascending_commit_files - ); - - // Verify reading data from the checkpointed snapshot returns the expected rows - let scan = snapshot2.scan_builder().build()?; - let batches = read_scan(&scan, engine.clone() as Arc)?; - assert_batches_sorted_eq!( - vec![ - "+-------+", - "| value |", - "+-------+", - "| 1 |", - "+-------+", - ], - &batches - ); - - Ok(()) -} diff --git a/kernel/tests/write.rs b/kernel/tests/write.rs deleted file mode 100644 index fe898c7608..0000000000 --- a/kernel/tests/write.rs +++ /dev/null @@ -1,3953 +0,0 @@ -use std::collections::HashMap; -use std::sync::Arc; -use std::time::{Duration, SystemTime, UNIX_EPOCH}; - -use delta_kernel::committer::FileSystemCommitter; -use delta_kernel::Error as KernelError; -use delta_kernel::{DeltaResult, Engine, Snapshot, Version}; -use url::Url; -use uuid::Uuid; - -use delta_kernel::actions::deletion_vector::{DeletionVectorDescriptor, DeletionVectorStorageType}; -use delta_kernel::arrow::array::{Array, ArrayRef, BinaryArray, Int64Array, StructArray}; -use delta_kernel::arrow::array::{Int32Array, StringArray, TimestampMicrosecondArray}; -use delta_kernel::arrow::buffer::NullBuffer; -use delta_kernel::arrow::datatypes::{DataType as ArrowDataType, Field}; -use delta_kernel::arrow::error::ArrowError; -use delta_kernel::arrow::record_batch::RecordBatch; - -use delta_kernel::engine::arrow_conversion::{TryFromKernel, TryIntoArrow as _}; -use delta_kernel::engine::arrow_data::ArrowEngineData; -use delta_kernel::engine::default::executor::tokio::{ - TokioBackgroundExecutor, TokioMultiThreadExecutor, -}; -use delta_kernel::engine::default::parquet::DefaultParquetHandler; -use delta_kernel::engine::default::DefaultEngine; -use delta_kernel::engine::default::DefaultEngineBuilder; -use delta_kernel::engine_data::FilteredEngineData; -use delta_kernel::object_store::local::LocalFileSystem; -use delta_kernel::object_store::path::Path; -use delta_kernel::object_store::{DynObjectStore, ObjectStore as _}; -use delta_kernel::transaction::create_table::create_table as create_table_txn; -use delta_kernel::transaction::CommitResult; -use tempfile::TempDir; - -use test_utils::set_json_value; - -use itertools::Itertools; -use serde_json::json; -use serde_json::Deserializer; -use tempfile::tempdir; - -use delta_kernel::expressions::ColumnName; -use delta_kernel::parquet::file::reader::{FileReader, SerializedFileReader}; -use delta_kernel::schema::{ - ColumnMetadataKey, DataType, MetadataValue, SchemaRef, StructField, StructType, -}; -use delta_kernel::table_features::{get_any_level_column_physical_name, ColumnMappingMode}; -use delta_kernel::FileMeta; - -use test_utils::create_default_engine_mt_executor; -use test_utils::{ - assert_partition_values, assert_result_error_with_message, assert_schema_has_field, - copy_directory, create_add_files_metadata, create_default_engine, create_table, - create_table_and_load_snapshot, engine_store_setup, nested_batches, nested_schema, - read_actions_from_commit, read_add_infos, remove_all_and_get_remove_actions, resolve_field, - setup_test_tables, test_read, test_table_setup, write_batch_to_table, -}; - -mod common; - -/// Returns the native parquet `field_id` for a field at the given physical path in a parquet file, -/// or `None` if the field has no `field_id` set. -/// -/// Panics if the file cannot be read or the physical path doesn't exist in the parquet schema. -fn get_parquet_field_id(parquet_file: &std::path::Path, physical_path: &[String]) -> Option { - let file = std::fs::File::open(parquet_file).unwrap(); - let reader = SerializedFileReader::new(file).unwrap(); - let root = reader - .metadata() - .file_metadata() - .schema_descr() - .root_schema() - .clone(); - - let mut current = &root; - for name in physical_path { - current = current - .get_fields() - .iter() - .find(|f| f.name() == name) - .unwrap_or_else(|| panic!("parquet schema missing field '{name}'")); - } - - let info = current.get_basic_info(); - info.has_id().then(|| info.id()) -} - -fn validate_txn_id(commit_info: &serde_json::Value) { - let txn_id = commit_info["txnId"] - .as_str() - .expect("txnId should be present in commitInfo"); - Uuid::parse_str(txn_id).expect("txnId should be valid UUID format"); -} - -fn validate_timestamp(commit_info: &serde_json::Value) { - let timestamp = commit_info["timestamp"] - .as_i64() - .expect("timestamp should be present in commitInfo"); - let current_ts: i64 = SystemTime::now() - .duration_since(UNIX_EPOCH) - .unwrap() - .as_millis() - .try_into() - .unwrap(); - let two_days_ms = Duration::from_secs(2 * 24 * 60 * 60).as_millis() as i64; - assert!( - (timestamp <= current_ts && timestamp > current_ts - two_days_ms), - "commit timestamp should be at most 2 days behind current system time: got {timestamp}, now {current_ts}" - ); -} - -const ZERO_UUID: &str = "00000000-0000-0000-0000-000000000000"; - -/// Creates a table with deletion vector support and writes the specified files -async fn create_dv_table_with_files( - table_name: &str, - schema: Arc, - file_paths: &[&str], -) -> Result< - ( - Arc, - Arc, - Url, - Vec, - ), - Box, -> { - let (store, engine, table_url) = engine_store_setup(table_name, None); - let engine = Arc::new(engine); - - // Create table with DV support (protocol 3/7 with deletionVectors feature) - create_table( - store.clone(), - table_url.clone(), - schema.clone(), - &[], - true, // use_37_protocol - vec!["deletionVectors"], - vec!["deletionVectors"], - ) - .await?; - - // Write files - let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; - let mut txn = snapshot - .clone() - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_engine_info("test engine") - .with_operation("WRITE".to_string()) - .with_data_change(true); - - let add_files_schema = txn.add_files_schema(); - - // Build metadata for all files at once - let files: Vec<(&str, i64, i64, i64)> = file_paths - .iter() - .enumerate() - .map(|(i, &path)| { - ( - path, - 1024 + i as i64 * 100, // size - 1000000 + i as i64, // mod_time - 3, // num_records - ) - }) - .collect(); - let metadata = create_add_files_metadata(add_files_schema, files)?; - txn.add_files(metadata); - - let _ = txn.commit(engine.as_ref())?; - - let paths: Vec = file_paths.iter().map(|&s| s.to_string()).collect(); - Ok((store, engine, table_url, paths)) -} - -/// Extracts scan files from a snapshot for use in deletion vector updates -fn get_scan_files( - snapshot: Arc, - engine: &dyn delta_kernel::Engine, -) -> DeltaResult> { - let scan = snapshot.scan_builder().build()?; - let all_scan_metadata: Vec<_> = scan.scan_metadata(engine)?.collect::, _>>()?; - - Ok(all_scan_metadata - .into_iter() - .map(|sm| sm.scan_files) - .collect()) -} - -fn get_simple_int_schema() -> Arc { - Arc::new(StructType::try_new(vec![StructField::nullable("number", DataType::INTEGER)]).unwrap()) -} - -/// Write a metadata-update commit that sets a table property on the existing table. -/// Returns a fresh snapshot reflecting the new commit. -/// Used in tests as a hack to set table properties when create table doesn't support the property. -fn set_table_properties( - table_path: &str, - table_url: &Url, - engine: &dyn Engine, - current_version: Version, - properties: &[(&str, &str)], -) -> Result, Box> { - let v0_path = std::path::Path::new(table_path).join("_delta_log/00000000000000000000.json"); - let mut meta: serde_json::Value = std::fs::read_to_string(&v0_path)? - .lines() - .find_map(|line| { - serde_json::from_str::(line) - .ok() - .filter(|v| v.get("metaData").is_some()) - }) - .expect("version 0 should contain a metaData action"); - - for &(key, value) in properties { - meta["metaData"]["configuration"][key] = json!(value); - } - - let new_commit = std::path::Path::new(table_path) - .join(format!("_delta_log/{:020}.json", current_version + 1)); - std::fs::write(&new_commit, serde_json::to_string(&meta)?)?; - Ok(Snapshot::builder_for(table_url.clone()).build(engine)?) -} - -/// Assert that the snapshot's column mapping mode matches the given `cm_mode` string, -/// and return the resolved mode. -fn assert_column_mapping_mode(snapshot: &Snapshot, cm_mode: &str) -> ColumnMappingMode { - let expected = match cm_mode { - "none" => ColumnMappingMode::None, - "name" => ColumnMappingMode::Name, - "id" => ColumnMappingMode::Id, - _ => panic!("unexpected cm_mode: {cm_mode}"), - }; - let actual = snapshot - .table_properties() - .column_mapping_mode - .expect("column mapping mode should be set"); - assert_eq!(actual, expected); - actual -} - -/// Resolve a nested column inside a [`StructArray`] by walking the given field-name path, -/// and downcast the leaf to the requested array type. -fn resolve_struct_field<'a, T: 'static>(root: &'a StructArray, path: &[String]) -> &'a T { - assert!(!path.is_empty(), "path must be non-empty"); - let mut current: &StructArray = root; - for (i, name) in path.iter().enumerate() { - let col = current - .column_by_name(name) - .unwrap_or_else(|| panic!("missing field: {name}")); - if i == path.len() - 1 { - return col - .as_any() - .downcast_ref::() - .expect("leaf array type mismatch"); - } - current = col - .as_any() - .downcast_ref::() - .unwrap_or_else(|| panic!("expected StructArray at field: {name}")); - } - unreachable!() -} - -/// Navigate into a nested JSON value by following a sequence of object keys. -/// E.g. `resolve_json_path(stats, &["address", "street"])` returns `stats["address"]["street"]`. -fn resolve_json_path<'a>(root: &'a serde_json::Value, path: &[String]) -> &'a serde_json::Value { - path.iter().fold(root, |v, key| &v[key]) -} - -/// Assert that `stats["minValues"]` and `stats["maxValues"]` at the given physical path equal the -/// expected values. -fn assert_min_max_stats( - stats: &serde_json::Value, - physical_path: &[String], - expected_min: impl Into, - expected_max: impl Into, -) { - assert_eq!( - *resolve_json_path(&stats["minValues"], physical_path), - expected_min.into() - ); - assert_eq!( - *resolve_json_path(&stats["maxValues"], physical_path), - expected_max.into() - ); -} - -#[tokio::test] -async fn test_commit_info() -> Result<(), Box> { - // setup tracing - let _ = tracing_subscriber::fmt::try_init(); - - // create a simple table: one int column named 'number' - let schema = get_simple_int_schema(); - - for (table_url, engine, store, table_name) in - setup_test_tables(schema, &[], None, "test_table").await? - { - // create a transaction - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let committer = Box::new(FileSystemCommitter::new()); - let txn = snapshot - .transaction(committer, &engine)? - .with_engine_info("default engine"); - - // commit! - let _ = txn.commit(&engine)?; - - let commit1 = store - .get(&Path::from(format!( - "/{table_name}/_delta_log/00000000000000000001.json" - ))) - .await?; - - let mut parsed_commit: serde_json::Value = serde_json::from_slice(&commit1.bytes().await?)?; - - validate_txn_id(&parsed_commit["commitInfo"]); - - set_json_value(&mut parsed_commit, "commitInfo.timestamp", json!(0))?; - set_json_value(&mut parsed_commit, "commitInfo.txnId", json!(ZERO_UUID))?; - - let expected_commit = json!({ - "commitInfo": { - "timestamp": 0, - "operation": "UNKNOWN", - "kernelVersion": format!("v{}", env!("CARGO_PKG_VERSION")), - "operationParameters": {}, - "engineInfo": "default engine", - "txnId": ZERO_UUID, - } - }); - - assert_eq!(parsed_commit, expected_commit); - } - Ok(()) -} - -// check that the timestamps in commit_info and add actions are within 10s of SystemTime::now() -fn check_action_timestamps<'a>( - parsed_commits: impl Iterator, -) -> Result<(), Box> { - let now: i64 = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH)? - .as_millis() - .try_into() - .unwrap(); - - parsed_commits.for_each(|commit| { - if let Some(commit_info_ts) = &commit.pointer("/commitInfo/timestamp") { - assert!((now - commit_info_ts.as_i64().unwrap()).abs() < 10_000); - } - if let Some(add_ts) = &commit.pointer("/add/modificationTime") { - assert!((now - add_ts.as_i64().unwrap()).abs() < 10_000); - } - }); - - Ok(()) -} - -// list all the files at `path` and check that all parquet files have the same size, and return -// that size -async fn get_and_check_all_parquet_sizes(store: Arc, path: &str) -> u64 { - use futures::stream::StreamExt; - let files: Vec<_> = store.list(Some(&Path::from(path))).collect().await; - let parquet_files = files - .into_iter() - .filter(|f| match f { - Ok(f) => f.location.extension() == Some("parquet"), - Err(_) => false, - }) - .collect::>(); - assert_eq!(parquet_files.len(), 2); - let size = parquet_files.first().unwrap().as_ref().unwrap().size; - assert!(parquet_files - .iter() - .all(|f| f.as_ref().unwrap().size == size)); - size -} - -async fn write_data_and_check_result_and_stats( - table_url: Url, - schema: SchemaRef, - engine: Arc>, - expected_since_commit: u64, -) -> Result<(), Box> { - let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; - let committer = Box::new(FileSystemCommitter::new()); - let mut txn = snapshot - .transaction(committer, engine.as_ref())? - .with_data_change(true); - - // create two new arrow record batches to append - let append_data = [[1, 2, 3], [4, 5, 6]].map(|data| -> DeltaResult<_> { - let data = RecordBatch::try_new( - Arc::new(schema.as_ref().try_into_arrow()?), - vec![Arc::new(Int32Array::from(data.to_vec()))], - )?; - Ok(Box::new(ArrowEngineData::new(data))) - }); - - // write data out by spawning async tasks to simulate executors - let write_context = Arc::new(txn.get_write_context()); - let tasks = append_data.into_iter().map(|data| { - // arc clones - let engine = engine.clone(); - let write_context = write_context.clone(); - tokio::task::spawn(async move { - engine - .write_parquet( - data.as_ref().unwrap(), - write_context.as_ref(), - HashMap::new(), - ) - .await - }) - }); - - let add_files_metadata = futures::future::join_all(tasks).await.into_iter().flatten(); - for meta in add_files_metadata { - txn.add_files(meta?); - } - - // commit! - match txn.commit(engine.as_ref())? { - CommitResult::CommittedTransaction(committed) => { - assert_eq!(committed.commit_version(), expected_since_commit as Version); - assert_eq!( - committed.post_commit_stats().commits_since_checkpoint, - expected_since_commit - ); - assert_eq!( - committed.post_commit_stats().commits_since_log_compaction, - expected_since_commit - ); - } - _ => panic!("Commit should have succeeded"), - }; - - Ok(()) -} - -#[tokio::test] -async fn test_commit_info_action() -> Result<(), Box> { - // setup tracing - let _ = tracing_subscriber::fmt::try_init(); - // create a simple table: one int column named 'number' - let schema = get_simple_int_schema(); - - for (table_url, engine, store, table_name) in - setup_test_tables(schema.clone(), &[], None, "test_table").await? - { - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let txn = snapshot - .transaction(Box::new(FileSystemCommitter::new()), &engine)? - .with_engine_info("default engine"); - - let _ = txn.commit(&engine)?; - - let commit = store - .get(&Path::from(format!( - "/{table_name}/_delta_log/00000000000000000001.json" - ))) - .await?; - - let mut parsed_commits: Vec<_> = Deserializer::from_slice(&commit.bytes().await?) - .into_iter::() - .try_collect()?; - - validate_txn_id(&parsed_commits[0]["commitInfo"]); - - // set timestamps to 0, paths and txn_id to known string values for comparison - // (otherwise timestamps are non-deterministic, paths and txn_id are random UUIDs) - set_json_value(&mut parsed_commits[0], "commitInfo.timestamp", json!(0))?; - set_json_value(&mut parsed_commits[0], "commitInfo.txnId", json!(ZERO_UUID))?; - - let expected_commit = vec![json!({ - "commitInfo": { - "timestamp": 0, - "operation": "UNKNOWN", - "kernelVersion": format!("v{}", env!("CARGO_PKG_VERSION")), - "operationParameters": {}, - "engineInfo": "default engine", - "txnId": ZERO_UUID - } - })]; - - assert_eq!(parsed_commits, expected_commit); - } - Ok(()) -} - -/// Verifies that when `engine_commit_info` is provided (the `Some` branch of `build_commit_info`): -/// - The written JSON is correctly wrapped in a top-level `"commitInfo"` key. -/// - Engine-only fields (not in `CommitInfo::to_schema()`) pass through to the log unchanged. -/// - Fields that overlap with kernel-managed CommitInfo fields are overridden by kernel values, -/// - All kernel-managed fields (`timestamp`, `kernelVersion`, `txnId`, `operationParameters`) -/// are present with correct values. -#[tokio::test] -async fn test_commit_info_with_engine_commit_info() -> Result<(), Box> { - let _ = tracing_subscriber::fmt::try_init(); - let schema = get_simple_int_schema(); - - for (table_url, engine, store, table_name) in - setup_test_tables(schema, &[], None, "test_table").await? - { - // Build engine_commit_info with: - // - "myApp" : engine-only field, must pass through unchanged. - // - "myVersion": engine-only field, must pass through unchanged. - // - "operation": overlapping with CommitInfo; kernel must override with "WRITE". - let arrow_schema = Arc::new(delta_kernel::arrow::datatypes::Schema::new(vec![ - Field::new("myApp", ArrowDataType::Utf8, false), - Field::new("myVersion", ArrowDataType::Utf8, false), - Field::new("operation", ArrowDataType::Utf8, false), - ])); - let batch = RecordBatch::try_new( - arrow_schema, - vec![ - Arc::new(StringArray::from(vec!["spark"])) as ArrayRef, - Arc::new(StringArray::from(vec!["3.5.0"])) as ArrayRef, - Arc::new(StringArray::from(vec!["STALE_OP"])) as ArrayRef, - ], - )?; - let engine_schema = Arc::new(StructType::new_unchecked(vec![ - StructField::not_null("myApp", DataType::STRING), - StructField::not_null("myVersion", DataType::STRING), - StructField::nullable("operation", DataType::STRING), - ])); - - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let txn = snapshot - .transaction(Box::new(FileSystemCommitter::new()), &engine)? - .with_operation("WRITE".to_string()) - .with_commit_info(Box::new(ArrowEngineData::new(batch)), engine_schema); - - let _ = txn.commit(&engine)?; - - let commit = store - .get(&Path::from(format!( - "/{table_name}/_delta_log/00000000000000000001.json" - ))) - .await?; - - let mut parsed_commits: Vec<_> = Deserializer::from_slice(&commit.bytes().await?) - .into_iter::() - .try_collect()?; - - validate_txn_id(&parsed_commits[0]["commitInfo"]); - validate_timestamp(&parsed_commits[0]["commitInfo"]); - - // Zero out non-deterministic fields for stable comparison. - set_json_value(&mut parsed_commits[0], "commitInfo.timestamp", json!(0))?; - set_json_value(&mut parsed_commits[0], "commitInfo.txnId", json!(ZERO_UUID))?; - - // Null-valued CommitInfo fields (inCommitTimestamp, isBlindAppend, engineInfo) are - // omitted from the JSON — consistent with how the Delta log serializes optional fields. - let expected_commits = vec![json!({ - "commitInfo": { - // Engine-only fields pass through unchanged. - "myApp": "spark", - "myVersion": "3.5.0", - // Kernel overrides the engine's stale "STALE_OP" with the real operation. - "operation": "WRITE", - // Remaining kernel-managed non-null fields are appended. - "operationParameters": {}, - "kernelVersion": format!("v{}", env!("CARGO_PKG_VERSION")), - "txnId": ZERO_UUID, - "timestamp": 0, - } - })]; - - assert_eq!(parsed_commits, expected_commits); - } - Ok(()) -} - -#[tokio::test] -async fn test_append() -> Result<(), Box> { - // setup tracing - let _ = tracing_subscriber::fmt::try_init(); - // create a simple table: one int column named 'number' - let schema = get_simple_int_schema(); - - for (table_url, engine, store, table_name) in - setup_test_tables(schema.clone(), &[], None, "test_table").await? - { - // write data out by spawning async tasks to simulate executors - let engine = Arc::new(engine); - write_data_and_check_result_and_stats(table_url.clone(), schema.clone(), engine.clone(), 1) - .await?; - - let commit1 = store - .get(&Path::from(format!( - "/{table_name}/_delta_log/00000000000000000001.json" - ))) - .await?; - - let mut parsed_commits: Vec<_> = Deserializer::from_slice(&commit1.bytes().await?) - .into_iter::() - .try_collect()?; - - let size = - get_and_check_all_parquet_sizes(store.clone(), format!("/{table_name}/").as_str()) - .await; - // check that the timestamps in commit_info and add actions are within 10s of SystemTime::now() - // before we clear them for comparison - check_action_timestamps(parsed_commits.iter())?; - // check that the txn_id is valid before we clear it for comparison - validate_txn_id(&parsed_commits[0]["commitInfo"]); - - // set timestamps to 0, paths and txn_id to known string values for comparison - // (otherwise timestamps are non-deterministic, paths and txn_id are random UUIDs) - set_json_value(&mut parsed_commits[0], "commitInfo.timestamp", json!(0))?; - set_json_value(&mut parsed_commits[0], "commitInfo.txnId", json!(ZERO_UUID))?; - set_json_value(&mut parsed_commits[1], "add.modificationTime", json!(0))?; - set_json_value(&mut parsed_commits[1], "add.path", json!("first.parquet"))?; - set_json_value(&mut parsed_commits[2], "add.modificationTime", json!(0))?; - set_json_value(&mut parsed_commits[2], "add.path", json!("second.parquet"))?; - - let expected_commit = vec![ - json!({ - "commitInfo": { - "timestamp": 0, - "operation": "UNKNOWN", - "kernelVersion": format!("v{}", env!("CARGO_PKG_VERSION")), - "operationParameters": {}, - "txnId": ZERO_UUID - } - }), - json!({ - "add": { - "path": "first.parquet", - "partitionValues": {}, - "size": size, - "modificationTime": 0, - "dataChange": true, - "stats": "{\"numRecords\":3,\"nullCount\":{\"number\":0},\"minValues\":{\"number\":1},\"maxValues\":{\"number\":3},\"tightBounds\":true}" - } - }), - json!({ - "add": { - "path": "second.parquet", - "partitionValues": {}, - "size": size, - "modificationTime": 0, - "dataChange": true, - "stats": "{\"numRecords\":3,\"nullCount\":{\"number\":0},\"minValues\":{\"number\":4},\"maxValues\":{\"number\":6},\"tightBounds\":true}" - } - }), - ]; - - assert_eq!(parsed_commits, expected_commit); - - test_read( - &ArrowEngineData::new(RecordBatch::try_new( - Arc::new(schema.as_ref().try_into_arrow()?), - vec![Arc::new(Int32Array::from(vec![1, 2, 3, 4, 5, 6]))], - )?), - &table_url, - engine, - )?; - } - Ok(()) -} - -#[tokio::test] -async fn test_no_add_actions() -> Result<(), Box> { - // setup tracing - let _ = tracing_subscriber::fmt::try_init(); - // create a simple table: one int column named 'number' - let schema = get_simple_int_schema(); - - for (table_url, engine, store, table_name) in - setup_test_tables(schema.clone(), &[], None, "test_table").await? - { - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let txn = snapshot - .transaction(Box::new(FileSystemCommitter::new()), &engine)? - .with_engine_info("default engine"); - - // Commit without adding any add files - assert!(txn.commit(&engine)?.is_committed()); - - let commit1 = store - .get(&Path::from(format!( - "/{table_name}/_delta_log/00000000000000000001.json" - ))) - .await?; - - let parsed_actions: Vec<_> = Deserializer::from_slice(&commit1.bytes().await?) - .into_iter::() - .try_collect()?; - - // Verify that there only is a commit info action - assert_eq!(parsed_actions.len(), 1, "Expected only one action"); - assert!(parsed_actions[0].get("commitInfo").is_some()); - } - Ok(()) -} - -#[tokio::test] -async fn test_append_twice() -> Result<(), Box> { - // setup tracing - let _ = tracing_subscriber::fmt::try_init(); - // create a simple table: one int column named 'number' - let schema = get_simple_int_schema(); - - for (table_url, engine, _, _) in - setup_test_tables(schema.clone(), &[], None, "test_table").await? - { - let engine = Arc::new(engine); - write_data_and_check_result_and_stats(table_url.clone(), schema.clone(), engine.clone(), 1) - .await?; - write_data_and_check_result_and_stats(table_url.clone(), schema.clone(), engine.clone(), 2) - .await?; - - test_read( - &ArrowEngineData::new(RecordBatch::try_new( - Arc::new(schema.as_ref().try_into_arrow()?), - vec![Arc::new(Int32Array::from(vec![ - 1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6, - ]))], - )?), - &table_url, - engine, - )?; - } - Ok(()) -} - -#[tokio::test] -async fn test_append_partitioned() -> Result<(), Box> { - // setup tracing - let _ = tracing_subscriber::fmt::try_init(); - - let partition_col = "partition"; - - // create a simple partitioned table: one int column named 'number', partitioned by string - // column named 'partition' - let table_schema = Arc::new(StructType::try_new(vec![ - StructField::nullable("number", DataType::INTEGER), - StructField::nullable("partition", DataType::STRING), - ])?); - let data_schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "number", - DataType::INTEGER, - )])?); - - for (table_url, engine, store, table_name) in - setup_test_tables(table_schema.clone(), &[partition_col], None, "test_table").await? - { - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let mut txn = snapshot - .transaction(Box::new(FileSystemCommitter::new()), &engine)? - .with_engine_info("default engine") - .with_data_change(false); - - // create two new arrow record batches to append - let append_data = [[1, 2, 3], [4, 5, 6]].map(|data| -> DeltaResult<_> { - let data = RecordBatch::try_new( - Arc::new(data_schema.as_ref().try_into_arrow()?), - vec![Arc::new(Int32Array::from(data.to_vec()))], - )?; - Ok(Box::new(ArrowEngineData::new(data))) - }); - let partition_vals = vec!["a", "b"]; - - // write data out by spawning async tasks to simulate executors - let engine = Arc::new(engine); - let write_context = Arc::new(txn.get_write_context()); - let tasks = append_data - .into_iter() - .zip(partition_vals) - .map(|(data, partition_val)| { - // arc clones - let engine = engine.clone(); - let write_context = write_context.clone(); - tokio::task::spawn(async move { - engine - .write_parquet( - data.as_ref().unwrap(), - write_context.as_ref(), - HashMap::from([(partition_col.to_string(), partition_val.to_string())]), - ) - .await - }) - }); - - let add_files_metadata = futures::future::join_all(tasks).await.into_iter().flatten(); - for meta in add_files_metadata { - txn.add_files(meta?); - } - - // commit! - assert!(txn.commit(engine.as_ref())?.is_committed()); - - let commit1 = store - .get(&Path::from(format!( - "/{table_name}/_delta_log/00000000000000000001.json" - ))) - .await?; - - let mut parsed_commits: Vec<_> = Deserializer::from_slice(&commit1.bytes().await?) - .into_iter::() - .try_collect()?; - - let size = - get_and_check_all_parquet_sizes(store.clone(), format!("/{table_name}/").as_str()) - .await; - // check that the timestamps in commit_info and add actions are within 10s of SystemTime::now() - // before we clear them for comparison - check_action_timestamps(parsed_commits.iter())?; - // check that the txn_id is valid before we clear it for comparison - validate_txn_id(&parsed_commits[0]["commitInfo"]); - - // set timestamps to 0, paths and txn_id to known string values for comparison - // (otherwise timestamps are non-deterministic, paths and txn_id are random UUIDs) - set_json_value(&mut parsed_commits[0], "commitInfo.timestamp", json!(0))?; - set_json_value(&mut parsed_commits[0], "commitInfo.txnId", json!(ZERO_UUID))?; - set_json_value(&mut parsed_commits[1], "add.modificationTime", json!(0))?; - set_json_value(&mut parsed_commits[1], "add.path", json!("first.parquet"))?; - set_json_value(&mut parsed_commits[2], "add.modificationTime", json!(0))?; - set_json_value(&mut parsed_commits[2], "add.path", json!("second.parquet"))?; - - let expected_commit = vec![ - json!({ - "commitInfo": { - "timestamp": 0, - "operation": "UNKNOWN", - "kernelVersion": format!("v{}", env!("CARGO_PKG_VERSION")), - "operationParameters": {}, - "engineInfo": "default engine", - "txnId": ZERO_UUID - } - }), - json!({ - "add": { - "path": "first.parquet", - "partitionValues": { - "partition": "a" - }, - "size": size, - "modificationTime": 0, - "dataChange": false, - "stats": "{\"numRecords\":3,\"nullCount\":{\"number\":0},\"minValues\":{\"number\":1},\"maxValues\":{\"number\":3},\"tightBounds\":true}" - } - }), - json!({ - "add": { - "path": "second.parquet", - "partitionValues": { - "partition": "b" - }, - "size": size, - "modificationTime": 0, - "dataChange": false, - "stats": "{\"numRecords\":3,\"nullCount\":{\"number\":0},\"minValues\":{\"number\":4},\"maxValues\":{\"number\":6},\"tightBounds\":true}" - } - }), - ]; - - assert_eq!(parsed_commits, expected_commit); - - test_read( - &ArrowEngineData::new(RecordBatch::try_new( - Arc::new(table_schema.as_ref().try_into_arrow()?), - vec![ - Arc::new(Int32Array::from(vec![1, 2, 3, 4, 5, 6])), - Arc::new(StringArray::from(vec!["a", "a", "a", "b", "b", "b"])), - ], - )?), - &table_url, - engine, - )?; - } - Ok(()) -} - -#[tokio::test] -async fn test_append_invalid_schema() -> Result<(), Box> { - // setup tracing - let _ = tracing_subscriber::fmt::try_init(); - // create a simple table: one int column named 'number' - let table_schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "number", - DataType::INTEGER, - )])?); - // incompatible data schema: one string column named 'string' - let data_schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "string", - DataType::STRING, - )])?); - - for (table_url, engine, _store, _table_name) in - setup_test_tables(table_schema, &[], None, "test_table").await? - { - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let txn = snapshot - .transaction(Box::new(FileSystemCommitter::new()), &engine)? - .with_engine_info("default engine"); - - // create two new arrow record batches to append - let append_data = [["a", "b"], ["c", "d"]].map(|data| -> DeltaResult<_> { - let data = RecordBatch::try_new( - Arc::new(data_schema.as_ref().try_into_arrow()?), - vec![Arc::new(StringArray::from(data.to_vec()))], - )?; - Ok(Box::new(ArrowEngineData::new(data))) - }); - - // write data out by spawning async tasks to simulate executors - let engine = Arc::new(engine); - let write_context = Arc::new(txn.get_write_context()); - let tasks = append_data.into_iter().map(|data| { - // arc clones - let engine = engine.clone(); - let write_context = write_context.clone(); - tokio::task::spawn(async move { - engine - .write_parquet( - data.as_ref().unwrap(), - write_context.as_ref(), - HashMap::new(), - ) - .await - }) - }); - - let mut add_files_metadata = futures::future::join_all(tasks).await.into_iter().flatten(); - assert!(add_files_metadata.all(|res| match res { - Err(KernelError::Arrow(ArrowError::InvalidArgumentError(_))) => true, - Err(KernelError::Backtraced { source, .. }) - if matches!( - &*source, - KernelError::Arrow(ArrowError::InvalidArgumentError(_)) - ) => - true, - _ => false, - })); - } - Ok(()) -} - -#[tokio::test] -async fn test_write_txn_actions() -> Result<(), Box> { - // setup tracing - let _ = tracing_subscriber::fmt::try_init(); - - // create a simple table: one int column named 'number' - let schema = get_simple_int_schema(); - - for (table_url, engine, store, table_name) in - setup_test_tables(schema, &[], None, "test_table").await? - { - // can't have duplicate app_id in same transaction - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - assert!(matches!( - snapshot - .transaction(Box::new(FileSystemCommitter::new()), &engine)? - .with_transaction_id("app_id1".to_string(), 0) - .with_transaction_id("app_id1".to_string(), 1) - .commit(&engine), - Err(KernelError::Generic(msg)) if msg == "app_id app_id1 already exists in transaction" - )); - - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let txn = snapshot - .transaction(Box::new(FileSystemCommitter::new()), &engine)? - .with_engine_info("default engine") - .with_transaction_id("app_id1".to_string(), 1) - .with_transaction_id("app_id2".to_string(), 2); - - // commit! - assert!(txn.commit(&engine)?.is_committed()); - - let snapshot = Snapshot::builder_for(table_url.clone()) - .at_version(1) - .build(&engine)?; - assert_eq!(snapshot.get_app_id_version("app_id1", &engine)?, Some(1)); - assert_eq!(snapshot.get_app_id_version("app_id2", &engine)?, Some(2)); - assert_eq!(snapshot.get_app_id_version("app_id3", &engine)?, None); - - let commit1 = store - .get(&Path::from(format!( - "/{table_name}/_delta_log/00000000000000000001.json" - ))) - .await?; - - let mut parsed_commits: Vec<_> = Deserializer::from_slice(&commit1.bytes().await?) - .into_iter::() - .try_collect()?; - - set_json_value(&mut parsed_commits[0], "commitInfo.timestamp", json!(0)).unwrap(); - - let time_ms: i64 = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH)? - .as_millis() - .try_into() - .unwrap(); - - // check that last_updated times are identical - let last_updated1 = parsed_commits[1] - .get("txn") - .unwrap() - .get("lastUpdated") - .unwrap(); - let last_updated2 = parsed_commits[2] - .get("txn") - .unwrap() - .get("lastUpdated") - .unwrap(); - assert_eq!(last_updated1, last_updated2); - - let last_updated = parsed_commits[1] - .get_mut("txn") - .unwrap() - .get_mut("lastUpdated") - .unwrap(); - // sanity check that last_updated time is within 10s of now - assert!((last_updated.as_i64().unwrap() - time_ms).abs() < 10_000); - *last_updated = serde_json::Value::Number(1.into()); - - let last_updated = parsed_commits[2] - .get_mut("txn") - .unwrap() - .get_mut("lastUpdated") - .unwrap(); - // sanity check that last_updated time is within 10s of now - assert!((last_updated.as_i64().unwrap() - time_ms).abs() < 10_000); - *last_updated = serde_json::Value::Number(2.into()); - - validate_txn_id(&parsed_commits[0]["commitInfo"]); - - set_json_value(&mut parsed_commits[0], "commitInfo.txnId", json!(ZERO_UUID))?; - - let expected_commit = vec![ - json!({ - "commitInfo": { - "timestamp": 0, - "operation": "UNKNOWN", - "kernelVersion": format!("v{}", env!("CARGO_PKG_VERSION")), - "operationParameters": {}, - "engineInfo": "default engine", - "txnId": ZERO_UUID - } - }), - json!({ - "txn": { - "appId": "app_id1", - "version": 1, - "lastUpdated": 1 - } - }), - json!({ - "txn": { - "appId": "app_id2", - "version": 2, - "lastUpdated": 2 - } - }), - ]; - - assert_eq!(parsed_commits, expected_commit); - } - Ok(()) -} - -#[tokio::test] -async fn test_append_timestamp_ntz() -> Result<(), Box> { - // setup tracing - let _ = tracing_subscriber::fmt::try_init(); - - // create a table with TIMESTAMP_NTZ column - let schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "ts_ntz", - DataType::TIMESTAMP_NTZ, - )])?); - - let (store, engine, table_location) = engine_store_setup("test_table_timestamp_ntz", None); - let table_url = create_table( - store.clone(), - table_location, - schema.clone(), - &[], - true, - vec!["timestampNtz"], - vec!["timestampNtz"], - ) - .await?; - - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let mut txn = snapshot - .transaction(Box::new(FileSystemCommitter::new()), &engine)? - .with_engine_info("default engine"); - - // Create Arrow data with TIMESTAMP_NTZ values including edge cases - // These are microseconds since Unix epoch - let timestamp_values = vec![ - 0i64, // Unix epoch (1970-01-01T00:00:00.000000) - 1634567890123456i64, // 2021-10-18T12:31:30.123456 - 1634567950654321i64, // 2021-10-18T12:32:30.654321 - 1672531200000000i64, // 2023-01-01T00:00:00.000000 - 253402300799999999i64, // 9999-12-31T23:59:59.999999 (near max valid timestamp) - -62135596800000000i64, // 0001-01-01T00:00:00.000000 (near min valid timestamp) - ]; - - let data = RecordBatch::try_new( - Arc::new(schema.as_ref().try_into_arrow()?), - vec![Arc::new(TimestampMicrosecondArray::from(timestamp_values))], - )?; - - // Write data - let engine = Arc::new(engine); - let write_context = Arc::new(txn.get_write_context()); - - let add_files_metadata = engine - .write_parquet( - &ArrowEngineData::new(data.clone()), - write_context.as_ref(), - HashMap::new(), - ) - .await?; - - txn.add_files(add_files_metadata); - - // Commit the transaction - assert!(txn.commit(engine.as_ref())?.is_committed()); - - // Verify the commit was written correctly - let commit1 = store - .get(&Path::from( - "/test_table_timestamp_ntz/_delta_log/00000000000000000001.json", - )) - .await?; - - let parsed_commits: Vec<_> = Deserializer::from_slice(&commit1.bytes().await?) - .into_iter::() - .try_collect()?; - - // Check that we have the expected number of commits (commitInfo + add) - assert_eq!(parsed_commits.len(), 2); - - // Check that the add action exists - assert!(parsed_commits[1].get("add").is_some()); - // Ensure default of data change is true. - assert!(parsed_commits[1] - .get("add") - .unwrap() - .get("dataChange") - .unwrap() - .as_bool() - .unwrap()); - - // Verify the data can be read back correctly - test_read(&ArrowEngineData::new(data), &table_url, engine)?; - - Ok(()) -} - -#[tokio::test] -async fn test_append_variant() -> Result<(), Box> { - // setup tracing - let _ = tracing_subscriber::fmt::try_init(); - fn unshredded_variant_schema_flipped() -> DataType { - DataType::variant_type([ - StructField::not_null("value", DataType::BINARY), - StructField::not_null("metadata", DataType::BINARY), - ]) - .unwrap() - } - fn variant_arrow_type_flipped() -> ArrowDataType { - let metadata_field = Field::new("metadata", ArrowDataType::Binary, false); - let value_field = Field::new("value", ArrowDataType::Binary, false); - let fields = vec![value_field, metadata_field]; - ArrowDataType::Struct(fields.into()) - } - - // create a table with VARIANT column - let table_schema = Arc::new(StructType::try_new(vec![ - StructField::nullable("v", DataType::unshredded_variant()), - StructField::nullable("i", DataType::INTEGER), - StructField::nullable( - "nested", - // We flip the value and metadata fields in the actual parquet file for the test - StructType::try_new(vec![StructField::nullable( - "nested_v", - unshredded_variant_schema_flipped(), - )])?, - ), - ])?); - - let write_schema = table_schema.clone(); - - let tmp_test_dir = tempdir()?; - let tmp_test_dir_url = Url::from_directory_path(tmp_test_dir.path()).unwrap(); - - let (store, engine, table_location) = - engine_store_setup("test_table_variant", Some(&tmp_test_dir_url)); - - // We can add shredding features as well as we are allowed to write unshredded variants - // into shredded tables and shredded reads are explicitly blocked in the default - // engine's parquet reader. - let table_url = create_table( - store.clone(), - table_location, - table_schema.clone(), - &[], - true, - vec!["variantType", "variantShredding-preview"], - vec!["variantType", "variantShredding-preview"], - ) - .await?; - - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let mut txn = snapshot - .transaction(Box::new(FileSystemCommitter::new()), &engine)? - .with_data_change(true); - - // First value corresponds to the variant value "1". Third value corresponds to the variant - // representing the JSON Object {"a":2}. - let metadata_v = vec![ - Some(&[0x01, 0x00, 0x00][..]), - None, - Some(&[0x01, 0x01, 0x00, 0x01, 0x61][..]), - ]; - let value_v = vec![ - Some(&[0x0C, 0x01][..]), - None, - Some(&[0x02, 0x01, 0x00, 0x00, 0x01, 0x02][..]), - ]; - - let metadata_v_array = Arc::new(BinaryArray::from(metadata_v)) as ArrayRef; - let value_v_array = Arc::new(BinaryArray::from(value_v)) as ArrayRef; - - // First value corresponds to the variant value "2". Third value corresponds to the variant - // representing the JSON Object {"b":3}. - let metadata_nested_v = vec![ - Some(&[0x01, 0x00, 0x00][..]), - None, - Some(&[0x01, 0x01, 0x00, 0x01, 0x62][..]), - ]; - let value_nested_v = vec![ - Some(&[0x0C, 0x02][..]), - None, - Some(&[0x02, 0x01, 0x00, 0x00, 0x01, 0x03][..]), - ]; - - let value_nested_v_array = Arc::new(BinaryArray::from(value_nested_v)) as ArrayRef; - let metadata_nested_v_array = Arc::new(BinaryArray::from(metadata_nested_v)) as ArrayRef; - - let variant_arrow = ArrowDataType::try_from_kernel(&DataType::unshredded_variant()).unwrap(); - let variant_arrow_flipped = variant_arrow_type_flipped(); - - let i_values = vec![31, 32, 33]; - - let fields = match variant_arrow { - ArrowDataType::Struct(fields) => Ok(fields), - _ => Err(KernelError::Generic( - "Variant arrow data type is not struct.".to_string(), - )), - }?; - let fields_flipped = match variant_arrow_flipped { - ArrowDataType::Struct(fields) => Ok(fields), - _ => Err(KernelError::Generic( - "Variant arrow data type is not struct.".to_string(), - )), - }?; - - let null_bitmap = NullBuffer::from_iter([true, false, true]); - - let variant_v_array = StructArray::try_new( - fields.clone(), - vec![metadata_v_array, value_v_array], - Some(null_bitmap.clone()), - )?; - - let variant_nested_v_array = Arc::new(StructArray::try_new( - fields_flipped.clone(), - vec![ - value_nested_v_array.clone(), - metadata_nested_v_array.clone(), - ], - Some(null_bitmap.clone()), - )?); - - let data = RecordBatch::try_new( - Arc::new(write_schema.as_ref().try_into_arrow()?), - vec![ - // v variant - Arc::new(variant_v_array.clone()), - // i int - Arc::new(Int32Array::from(i_values.clone())), - // nested struct - Arc::new(StructArray::try_new( - vec![Field::new("nested_v", variant_arrow_type_flipped(), true)].into(), - vec![variant_nested_v_array.clone()], - None, - )?), - ], - ) - .unwrap(); - - // Write data - let engine = Arc::new(engine); - let write_context = Arc::new(txn.get_write_context()); - - let add_files_metadata = (*engine) - .parquet_handler() - .as_any() - .downcast_ref::>() - .unwrap() - .write_parquet_file( - write_context.target_dir(), - Box::new(ArrowEngineData::new(data.clone())), - HashMap::new(), - Some(write_context.stats_columns()), - ) - .await?; - - txn.add_files(add_files_metadata); - - // Commit the transaction - assert!(txn.commit(engine.as_ref())?.is_committed()); - - // Verify the commit was written correctly - let commit1_url = tmp_test_dir_url - .join("test_table_variant/_delta_log/00000000000000000001.json") - .unwrap(); - let commit1 = store - .get(&Path::from_url_path(commit1_url.path()).unwrap()) - .await?; - - let parsed_commits: Vec<_> = Deserializer::from_slice(&commit1.bytes().await?) - .into_iter::() - .try_collect()?; - - // Check that we have the expected number of commits (commitInfo + add) - assert_eq!(parsed_commits.len(), 2); - - // Check that the add action exists - assert!(parsed_commits[1].get("add").is_some()); - - // The scanned data will match the logical schema, not the physical one - let expected_schema = Arc::new(StructType::try_new(vec![ - StructField::nullable("v", DataType::unshredded_variant()), - StructField::nullable("i", DataType::INTEGER), - StructField::nullable( - "nested", - StructType::try_new(vec![StructField::nullable( - "nested_v", - DataType::unshredded_variant(), - )]) - .unwrap(), - ), - ])?); - - // During the read, the flipped fields should be reordered into metadata, value. - let variant_nested_v_array_expected = Arc::new(StructArray::try_new( - fields, - vec![metadata_nested_v_array, value_nested_v_array], - Some(null_bitmap), - )?); - let variant_arrow_type: ArrowDataType = - ArrowDataType::try_from_kernel(&DataType::unshredded_variant()).unwrap(); - let expected_data = RecordBatch::try_new( - Arc::new(expected_schema.as_ref().try_into_arrow()?), - vec![ - // v variant - Arc::new(variant_v_array), - // i int - Arc::new(Int32Array::from(i_values)), - // nested struct - Arc::new(StructArray::try_new( - vec![Field::new("nested_v", variant_arrow_type, true)].into(), - vec![variant_nested_v_array_expected], - None, - )?), - ], - ) - .unwrap(); - - test_read(&ArrowEngineData::new(expected_data), &table_url, engine)?; - - Ok(()) -} - -#[tokio::test] -async fn test_shredded_variant_read_rejection() -> Result<(), Box> { - // Ensure that shredded variants are rejected by the default engine's parquet reader - - // setup tracing - let _ = tracing_subscriber::fmt::try_init(); - let table_schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "v", - DataType::unshredded_variant(), - )])?); - - // The table will be attempted to be written in this form but be read into - // STRUCT. The read should fail because the default engine - // currently does not support shredded reads. - let shredded_write_schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "v", - DataType::try_struct_type([ - StructField::new("metadata", DataType::BINARY, true), - StructField::new("value", DataType::BINARY, true), - StructField::new("typed_value", DataType::INTEGER, true), - ])?, - )])?); - - let tmp_test_dir = tempdir()?; - let tmp_test_dir_url = Url::from_directory_path(tmp_test_dir.path()).unwrap(); - - let (store, engine, table_location) = - engine_store_setup("test_table_variant_2", Some(&tmp_test_dir_url)); - let table_url = create_table( - store.clone(), - table_location, - table_schema.clone(), - &[], - true, - vec!["variantType", "variantShredding-preview"], - vec!["variantType", "variantShredding-preview"], - ) - .await?; - - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let mut txn = snapshot - .transaction(Box::new(FileSystemCommitter::new()), &engine)? - .with_data_change(true); - - // First value corresponds to the variant value "1". Third value corresponds to the variant - // representing the JSON Object {"a":2}. - let metadata_v = vec![ - Some(&[0x01, 0x00, 0x00][..]), - Some(&[0x01, 0x01, 0x00, 0x01, 0x61][..]), - ]; - let value_v = vec![ - Some(&[0x0C, 0x01][..]), - Some(&[0x02, 0x01, 0x00, 0x00, 0x01, 0x02][..]), - ]; - let typed_value_v = vec![Some(21), Some(3)]; - - let metadata_v_array = Arc::new(BinaryArray::from(metadata_v)) as ArrayRef; - let value_v_array = Arc::new(BinaryArray::from(value_v)) as ArrayRef; - let typed_value_v_array = Arc::new(Int32Array::from(typed_value_v)) as ArrayRef; - - let variant_arrow = ArrowDataType::Struct( - vec![ - Field::new("metadata", ArrowDataType::Binary, true), - Field::new("value", ArrowDataType::Binary, true), - Field::new("typed_value", ArrowDataType::Int32, true), - ] - .into(), - ); - - let fields = match variant_arrow { - ArrowDataType::Struct(fields) => Ok(fields), - _ => Err(KernelError::Generic( - "Variant arrow data type is not struct.".to_string(), - )), - }?; - - let variant_v_array = StructArray::try_new( - fields.clone(), - vec![metadata_v_array, value_v_array, typed_value_v_array], - None, - )?; - - let data = RecordBatch::try_new( - Arc::new(shredded_write_schema.as_ref().try_into_arrow()?), - vec![ - // v variant - Arc::new(variant_v_array.clone()), - ], - ) - .unwrap(); - - let engine = Arc::new(engine); - let write_context = Arc::new(txn.get_write_context()); - - let add_files_metadata = (*engine) - .parquet_handler() - .as_any() - .downcast_ref::>() - .unwrap() - .write_parquet_file( - write_context.target_dir(), - Box::new(ArrowEngineData::new(data.clone())), - HashMap::new(), - Some(write_context.stats_columns()), - ) - .await?; - - txn.add_files(add_files_metadata); - - // Commit the transaction - assert!(txn.commit(engine.as_ref())?.is_committed()); - - // Verify the commit was written correctly - let commit1_url = tmp_test_dir_url - .join("test_table_variant_2/_delta_log/00000000000000000001.json") - .unwrap(); - let commit1 = store - .get(&Path::from_url_path(commit1_url.path()).unwrap()) - .await?; - - let parsed_commits: Vec<_> = Deserializer::from_slice(&commit1.bytes().await?) - .into_iter::() - .try_collect()?; - - // Check that we have the expected number of commits (commitInfo + add) - assert_eq!(parsed_commits.len(), 2); - - // Check that the add action exists - assert!(parsed_commits[1].get("add").is_some()); - - let res = test_read(&ArrowEngineData::new(data), &table_url, engine); - assert!(matches!(res, - Err(e) if e.to_string().contains("The default engine does not support shredded reads"))); - - Ok(()) -} - -#[tokio::test] -async fn test_set_domain_metadata_basic() -> Result<(), Box> { - let _ = tracing_subscriber::fmt::try_init(); - - let schema = get_simple_int_schema(); - - let table_name = "test_domain_metadata_basic"; - - let (store, engine, table_location) = engine_store_setup(table_name, None); - let table_url = create_table( - store.clone(), - table_location, - schema.clone(), - &[], - true, - vec![], - vec!["domainMetadata"], - ) - .await?; - - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - - let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), &engine)?; - - // write context does not conflict with domain metadata - let _write_context = txn.get_write_context(); - - // set multiple domain metadata - let domain1 = "app.config"; - let config1 = r#"{"version": 1}"#; - let domain2 = "spark.settings"; - let config2 = r#"{"cores": 4}"#; - - assert!(txn - .with_domain_metadata(domain1.to_string(), config1.to_string()) - .with_domain_metadata(domain2.to_string(), config2.to_string()) - .commit(&engine)? - .is_committed()); - - let commit_data = store - .get(&Path::from(format!( - "/{table_name}/_delta_log/00000000000000000001.json" - ))) - .await? - .bytes() - .await?; - - let actions: Vec = Deserializer::from_slice(&commit_data) - .into_iter() - .try_collect()?; - - let domain_actions: Vec<_> = actions - .iter() - .filter(|v| v.get("domainMetadata").is_some()) - .collect(); - - for action in &domain_actions { - let domain = action["domainMetadata"]["domain"].as_str().unwrap(); - let config = action["domainMetadata"]["configuration"].as_str().unwrap(); - assert!(!action["domainMetadata"]["removed"].as_bool().unwrap()); - - match domain { - d if d == domain1 => assert_eq!(config, config1), - d if d == domain2 => assert_eq!(config, config2), - _ => panic!("Unexpected domain: {domain}"), - } - } - - let final_snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let domain1_config = final_snapshot.get_domain_metadata(domain1, &engine)?; - assert_eq!(domain1_config, Some(config1.to_string())); - let domain2_config = final_snapshot.get_domain_metadata(domain2, &engine)?; - assert_eq!(domain2_config, Some(config2.to_string())); - Ok(()) -} - -#[tokio::test] -async fn test_set_domain_metadata_errors() -> Result<(), Box> { - let _ = tracing_subscriber::fmt::try_init(); - - let schema = get_simple_int_schema(); - - let table_name = "test_domain_metadata_errors"; - let (store, engine, table_location) = engine_store_setup(table_name, None); - let table_url = create_table( - store.clone(), - table_location, - schema.clone(), - &[], - true, - vec![], - vec!["domainMetadata"], - ) - .await?; - - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - - // System domain rejection - let txn = snapshot - .clone() - .transaction(Box::new(FileSystemCommitter::new()), &engine)?; - let res = txn - .with_domain_metadata("delta.system".to_string(), "config".to_string()) - .commit(&engine); - assert_result_error_with_message( - res, - "Cannot modify domains that start with 'delta.' as those are system controlled", - ); - - // Duplicate domain rejection - let txn2 = snapshot - .clone() - .transaction(Box::new(FileSystemCommitter::new()), &engine)?; - let res = txn2 - .with_domain_metadata("app.config".to_string(), "v1".to_string()) - .with_domain_metadata("app.config".to_string(), "v2".to_string()) - .commit(&engine); - assert_result_error_with_message( - res, - "Metadata for domain app.config already specified in this transaction", - ); - - Ok(()) -} - -#[tokio::test] -async fn test_set_domain_metadata_unsupported_writer_feature( -) -> Result<(), Box> { - let _ = tracing_subscriber::fmt::try_init(); - - let schema = get_simple_int_schema(); - - let table_name = "test_domain_metadata_unsupported"; - - // Create table WITHOUT domain metadata writer feature support - let (store, engine, table_location) = engine_store_setup(table_name, None); - let table_url = create_table( - store.clone(), - table_location, - schema.clone(), - &[], - true, - vec![], - vec![], - ) - .await?; - - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let res = snapshot - .transaction(Box::new(FileSystemCommitter::new()), &engine)? - .with_domain_metadata("app.config".to_string(), "test_config".to_string()) - .commit(&engine); - - assert_result_error_with_message(res, "Domain metadata operations require writer version 7 and the 'domainMetadata' writer feature"); - - Ok(()) -} - -#[tokio::test] -async fn test_remove_domain_metadata_unsupported_writer_feature( -) -> Result<(), Box> { - let _ = tracing_subscriber::fmt::try_init(); - - let schema = get_simple_int_schema(); - - let table_name = "test_remove_domain_metadata_unsupported"; - - // Create table WITHOUT domain metadata writer feature support - let (store, engine, table_location) = engine_store_setup(table_name, None); - let table_url = create_table( - store.clone(), - table_location, - schema.clone(), - &[], - true, - vec![], - vec![], - ) - .await?; - - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let res = snapshot - .transaction(Box::new(FileSystemCommitter::new()), &engine)? - .with_domain_metadata_removed("app.config".to_string()) - .commit(&engine); - - assert_result_error_with_message(res, "Domain metadata operations require writer version 7 and the 'domainMetadata' writer feature"); - - Ok(()) -} - -#[tokio::test] -async fn test_remove_domain_metadata_non_existent_domain() -> Result<(), Box> -{ - let _ = tracing_subscriber::fmt::try_init(); - - let schema = get_simple_int_schema(); - - let table_name = "test_domain_metadata_unsupported"; - - let (store, engine, table_location) = engine_store_setup(table_name, None); - let table_url = create_table( - store.clone(), - table_location, - schema.clone(), - &[], - true, - vec![], - vec!["domainMetadata"], - ) - .await?; - - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), &engine)?; - - let domain = "app.deprecated"; - - // removing domain metadata that doesn't exist should NOT write a tombstone - let _ = txn - .with_domain_metadata_removed(domain.to_string()) - .commit(&engine)?; - - let commit_data = store - .get(&Path::from(format!( - "/{table_name}/_delta_log/00000000000000000001.json" - ))) - .await? - .bytes() - .await?; - let actions: Vec = Deserializer::from_slice(&commit_data) - .into_iter() - .try_collect()?; - - let domain_action = actions.iter().find(|v| v.get("domainMetadata").is_some()); - assert!( - domain_action.is_none(), - "No tombstone should be written for non-existent domain" - ); - - let final_snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let config = final_snapshot.get_domain_metadata(domain, &engine)?; - assert_eq!(config, None); - - Ok(()) -} - -#[tokio::test] -async fn test_domain_metadata_set_remove_conflicts() -> Result<(), Box> { - let _ = tracing_subscriber::fmt::try_init(); - - let schema = get_simple_int_schema(); - - let table_name = "test_domain_metadata_unsupported"; - - let (store, engine, table_location) = engine_store_setup(table_name, None); - let table_url = create_table( - store.clone(), - table_location, - schema.clone(), - &[], - true, - vec![], - vec!["domainMetadata"], - ) - .await?; - - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - - // set then remove same domain - let txn = snapshot - .clone() - .transaction(Box::new(FileSystemCommitter::new()), &engine)?; - let err = txn - .with_domain_metadata("app.config".to_string(), "v1".to_string()) - .with_domain_metadata_removed("app.config".to_string()) - .commit(&engine) - .unwrap_err(); - assert!(err - .to_string() - .contains("already specified in this transaction")); - - // remove then set same domain - let txn2 = snapshot - .clone() - .transaction(Box::new(FileSystemCommitter::new()), &engine)?; - let err = txn2 - .with_domain_metadata_removed("test.domain".to_string()) - .with_domain_metadata("test.domain".to_string(), "v1".to_string()) - .commit(&engine) - .unwrap_err(); - assert!(err - .to_string() - .contains("already specified in this transaction")); - - // remove same domain twice - let txn3 = snapshot - .clone() - .transaction(Box::new(FileSystemCommitter::new()), &engine)?; - let err = txn3 - .with_domain_metadata_removed("another.domain".to_string()) - .with_domain_metadata_removed("another.domain".to_string()) - .commit(&engine) - .unwrap_err(); - assert!(err - .to_string() - .contains("already specified in this transaction")); - - // remove system domain - let txn4 = snapshot - .clone() - .transaction(Box::new(FileSystemCommitter::new()), &engine)?; - let err = txn4 - .with_domain_metadata_removed("delta.system".to_string()) - .commit(&engine) - .unwrap_err(); - assert!(err - .to_string() - .contains("Cannot modify domains that start with 'delta.' as those are system controlled")); - - Ok(()) -} - -#[tokio::test] -async fn test_domain_metadata_set_then_remove() -> Result<(), Box> { - let _ = tracing_subscriber::fmt::try_init(); - - let schema = get_simple_int_schema(); - - let table_name = "test_domain_metadata_unsupported"; - - let (store, engine, table_location) = engine_store_setup(table_name, None); - let table_url = create_table( - store.clone(), - table_location, - schema.clone(), - &[], - true, - vec![], - vec!["domainMetadata"], - ) - .await?; - - let domain = "app.config"; - let configuration = r#"{"version": 1}"#; - - // txn 1: set domain metadata - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), &engine)?; - let _ = txn - .with_domain_metadata(domain.to_string(), configuration.to_string()) - .commit(&engine)?; - - // txn 2: remove the same domain metadata - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let txn = snapshot.transaction(Box::new(FileSystemCommitter::new()), &engine)?; - let _ = txn - .with_domain_metadata_removed(domain.to_string()) - .commit(&engine)?; - - // verify removal commit preserves the previous configuration - let commit_data = store - .get(&Path::from(format!( - "/{table_name}/_delta_log/00000000000000000002.json" - ))) - .await? - .bytes() - .await?; - let actions: Vec = Deserializer::from_slice(&commit_data) - .into_iter() - .try_collect()?; - - let domain_action = actions - .iter() - .find(|v| v.get("domainMetadata").is_some()) - .unwrap(); - assert_eq!(domain_action["domainMetadata"]["domain"], domain); - assert_eq!( - domain_action["domainMetadata"]["configuration"], - configuration - ); - assert_eq!(domain_action["domainMetadata"]["removed"], true); - - // verify reads see the metadata removal - let final_snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let domain_config = final_snapshot.get_domain_metadata(domain, &engine)?; - assert_eq!(domain_config, None); - - Ok(()) -} - -async fn get_ict_at_version( - store: Arc, - table_url: &Url, - version: u64, -) -> Result> { - let commit_path = table_url.join(&format!("_delta_log/{version:020}.json"))?; - let commit = store.get(&Path::from_url_path(commit_path.path())?).await?; - let commit_content = String::from_utf8(commit.bytes().await?.to_vec())?; - - // Parse each line of the commit log (NDJSON format) - // CommitInfo MUST be the first action when ICT is enabled - let lines: Vec<_> = commit_content - .lines() - .filter(|line| !line.trim().is_empty()) - .collect(); - assert!( - !lines.is_empty(), - "Commit log at version {version} should not be empty" - ); - - // First line should contain commitInfo with inCommitTimestamp - let first_action: serde_json::Value = serde_json::from_str(lines[0])?; - let commit_info = first_action - .get("commitInfo") - .expect("First action must be commitInfo when ICT is enabled"); - let ict = commit_info - .get("inCommitTimestamp") - .expect("commitInfo must have inCommitTimestamp when ICT is enabled") - .as_i64() - .unwrap(); - Ok(ict) -} - -/// Helper function to generate a simple data file and add it to the transaction -/// This simplifies repetitive data generation in tests -async fn generate_and_add_data_file( - txn: &mut delta_kernel::transaction::Transaction, - engine: &DefaultEngine, - schema: SchemaRef, - values: Vec, -) -> Result<(), Box> { - let data = RecordBatch::try_new( - Arc::new(schema.as_ref().try_into_arrow()?), - vec![Arc::new(Int32Array::from(values))], - )?; - - let write_context = Arc::new(txn.get_write_context()); - let file_meta = engine - .write_parquet( - &ArrowEngineData::new(data), - write_context.as_ref(), - HashMap::new(), - ) - .await?; - txn.add_files(file_meta); - Ok(()) -} - -#[tokio::test] -async fn test_ict_commit_e2e() -> Result<(), Box> { - let _ = tracing_subscriber::fmt::try_init(); - - // create a simple table: one int column named 'number' with ICT enabled - let schema = get_simple_int_schema(); - - let tmp_dir = TempDir::new()?; - let tmp_test_dir_url = Url::from_file_path(&tmp_dir).unwrap(); - - let (store, engine, table_location) = - engine_store_setup("test_ict_first_commit", Some(&tmp_test_dir_url)); - - // Create table with ICT enabled (writer version 7) - let table_url = test_utils::create_table( - store.clone(), - table_location, - schema.clone(), - &[], // no partition columns - true, // use protocol 3.7 - vec![], // no reader features - vec!["inCommitTimestamp"], // Enable ICT! Note: table feature is also set. - ) - .await?; - - // FIRST COMMIT: This exercises version() == 0 branch and generates ICT - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - assert_eq!( - snapshot.version(), - 0, - "Initial snapshot should be version 0" - ); - - let mut txn = snapshot - .transaction(Box::new(FileSystemCommitter::new()), &engine)? - .with_engine_info("ict test"); - - // Add some data - generate_and_add_data_file(&mut txn, &engine, schema.clone(), vec![1, 2, 3]).await?; - - // First commit - let commit_result = txn.commit(&engine)?; - match commit_result { - CommitResult::CommittedTransaction(committed) => { - assert_eq!( - committed.commit_version(), - 1, - "First commit should result in version 1" - ); - } - CommitResult::ConflictedTransaction(conflicted) => { - panic!( - "First commit should not conflict, got conflict at version {}", - conflicted.conflict_version() - ); - } - CommitResult::RetryableTransaction(_) => { - panic!("First commit should not be retryable error"); - } - } - - // VERIFY: Check that the commit log contains inCommitTimestamp - let first_ict = get_ict_at_version(store.clone(), &table_url, 1).await?; - - assert!( - first_ict > 1612345678, - "First commit ICT ({first_ict}) should be greater than enablement timestamp (1612345678)" - ); - - // Second commit - let snapshot2 = Snapshot::builder_for(table_url.clone()).build(&engine)?; - assert_eq!( - snapshot2.version(), - 1, - "Second snapshot should be version 1" - ); - - let mut txn2 = snapshot2 - .transaction(Box::new(FileSystemCommitter::new()), &engine)? - .with_engine_info("ict test 2"); - - // Add more data - generate_and_add_data_file(&mut txn2, &engine, schema, vec![4, 5, 6]).await?; - - // Second commit - let commit_result2 = txn2.commit(&engine)?; - match commit_result2 { - CommitResult::CommittedTransaction(committed) => { - assert_eq!( - committed.commit_version(), - 2, - "Second commit should result in version 2" - ); - } - CommitResult::ConflictedTransaction(conflicted) => { - panic!( - "Second commit should not conflict, got conflict at version {}", - conflicted.conflict_version() - ); - } - CommitResult::RetryableTransaction(_) => { - panic!("Second commit should not be retryable error"); - } - } - - // VERIFY: Check that second commit has proper monotonic ICT - let second_ict = get_ict_at_version(store, &table_url, 2).await?; - - // Verify monotonic property: second_ict > first_ict - assert!( - second_ict > first_ict, - "Second ICT ({second_ict}) should be greater than first ICT ({first_ict})" - ); - - Ok(()) -} - -#[tokio::test] -async fn test_remove_files_adds_expected_entries() -> Result<(), Box> { - // This test verifies that Remove actions generated from scan metadata contain all expected fields - // from the Remove struct (defined in kernel/src/actions/mod.rs). - // - // This test uses the table-with-dv-small dataset which contains files with tags and deletion vectors. - // - // Not populated in the dataset are (covered by row_tracking tests): - // baseRowId (optional i64) - // defaultRowCommitVersion (optional i64) - use std::path::PathBuf; - - let _ = tracing_subscriber::fmt::try_init(); - - let tmp_dir = tempdir()?; - let tmp_table_path = tmp_dir.path().join("table-with-dv-small"); - let source_path = std::fs::canonicalize(PathBuf::from("./tests/data/table-with-dv-small/"))?; - copy_directory(&source_path, &tmp_table_path)?; - - let table_url = url::Url::from_directory_path(&tmp_table_path).unwrap(); - let engine = create_default_engine(&table_url)?; - - let snapshot = Snapshot::builder_for(table_url.clone()) - .at_version(1) - .build(engine.as_ref())?; - - let mut txn = snapshot - .clone() - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_engine_info("test engine") - .with_data_change(true); - - let scan = snapshot.scan_builder().build()?; - let scan_metadata = scan.scan_metadata(engine.as_ref())?.next().unwrap()?; - - let (data, selection_vector) = scan_metadata.scan_files.into_parts(); - let remove_metadata = FilteredEngineData::try_new(data, selection_vector)?; - - txn.remove_files(remove_metadata); - - let result = txn.commit(engine.as_ref())?; - - match result { - CommitResult::CommittedTransaction(committed) => { - let commit_version = committed.commit_version(); - - // Read the commit log directly to verify remove actions - let commit_path = tmp_table_path.join(format!("_delta_log/{commit_version:020}.json")); - let commit_content = std::fs::read_to_string(commit_path)?; - - let parsed_commits: Vec<_> = Deserializer::from_str(&commit_content) - .into_iter::() - .try_collect()?; - - // Verify we have at least commitInfo and remove actions - assert!( - parsed_commits.len() >= 2, - "Expected at least 2 actions (commitInfo + remove)" - ); - - // Extract the commitInfo timestamp to validate against deletionTimestamp - let commit_info_action = parsed_commits - .iter() - .find(|action| action.get("commitInfo").is_some()) - .expect("Missing commitInfo action"); - let commit_info = &commit_info_action["commitInfo"]; - let commit_timestamp = commit_info["timestamp"] - .as_i64() - .expect("Missing timestamp in commitInfo"); - - // Verify remove actions - let remove_actions: Vec<_> = parsed_commits - .iter() - .filter(|action| action.get("remove").is_some()) - .collect(); - - assert!( - !remove_actions.is_empty(), - "Expected at least one remove action" - ); - - assert_eq!(remove_actions.len(), 1); - let remove_action = remove_actions[0]; - let remove = &remove_action["remove"]; - - // path (required) - assert!(remove.get("path").is_some(), "Missing path field"); - let path = remove["path"].as_str().expect("path should be a string"); - assert_eq!( - path, - "part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet" - ); - - // dataChange (required) - assert_eq!(remove["dataChange"].as_bool(), Some(true)); - - // deletionTimestamp (optional) - should match commit timestamp - let deletion_timestamp = remove["deletionTimestamp"] - .as_i64() - .expect("Missing deletionTimestamp"); - assert_eq!( - deletion_timestamp, commit_timestamp, - "deletionTimestamp should match commit timestamp" - ); - - // extendedFileMetadata (optional) - assert_eq!(remove["extendedFileMetadata"].as_bool(), Some(true)); - - // partitionValues (optional) - let partition_vals = remove["partitionValues"] - .as_object() - .expect("Missing partitionValues"); - assert_eq!(partition_vals.len(), 0); - - // size (optional) - let size = remove["size"].as_i64().expect("Missing size"); - assert_eq!(size, 635); - - // stats (optional) - let stats = remove["stats"].as_str().expect("Missing stats"); - let stats_json: serde_json::Value = serde_json::from_str(stats)?; - assert_eq!(stats_json["numRecords"], 10); - - // tags (optional) - let tags = remove["tags"].as_object().expect("Missing tags"); - assert_eq!( - tags.get("INSERTION_TIME").and_then(|v| v.as_str()), - Some("1677811178336000") - ); - assert_eq!( - tags.get("MIN_INSERTION_TIME").and_then(|v| v.as_str()), - Some("1677811178336000") - ); - assert_eq!( - tags.get("MAX_INSERTION_TIME").and_then(|v| v.as_str()), - Some("1677811178336000") - ); - assert_eq!( - tags.get("OPTIMIZE_TARGET_SIZE").and_then(|v| v.as_str()), - Some("268435456") - ); - - // deletionVector (optional) - let dv = remove["deletionVector"] - .as_object() - .expect("Missing deletionVector"); - assert_eq!(dv.get("storageType").and_then(|v| v.as_str()), Some("u")); - assert_eq!( - dv.get("pathOrInlineDv").and_then(|v| v.as_str()), - Some("vBn[lx{q8@P<9BNH/isA") - ); - assert_eq!(dv.get("offset").and_then(|v| v.as_i64()), Some(1)); - assert_eq!(dv.get("sizeInBytes").and_then(|v| v.as_i64()), Some(36)); - assert_eq!(dv.get("cardinality").and_then(|v| v.as_i64()), Some(2)); - - // Row tracking fields should be absent as the feature is was not enabled on writing - // row_tracking tests cover having these populated. - assert!(remove.get("baseRowId").is_none()); - assert!(remove.get("defaultRowCommitVersion").is_none()); - } - _ => panic!("Transaction should be committed"), - } - - Ok(()) -} - -#[tokio::test] -async fn test_update_deletion_vectors_adds_expected_entries( -) -> Result<(), Box> { - // This test verifies that deletion vector updates write proper Remove and Add actions - // to the transaction log. - // - // NOTE: Additional unit tests for update_deletion_vectors exist in kernel/src/transaction/mod.rs - // - // The test validates: - // 1. Transaction setup for DV updates - // 2. Scanning and extracting scan files with DV data - // 3. Creating new DV descriptors for the files - // 4. Calling update_deletion_vectors to update the DVs - // 5. Committing and verifying the generated actions - // - // Expected commit log structure: - // - commitInfo: Contains metadata about the transaction - // - remove: Contains OLD deletion vector data and original file metadata - // - add: Contains NEW deletion vector data and updated file metadata - // - // The test ensures: - // - Remove action has the OLD DV descriptor with all 5 fields - // - Add action has the NEW DV descriptor with all 5 fields - // - All file metadata is preserved (size, stats, tags, partitionValues) - // - dataChange is properly set to true - // - deletionTimestamp matches commit timestamp - use std::path::PathBuf; - - let _ = tracing_subscriber::fmt::try_init(); - - let tmp_dir = tempdir()?; - let tmp_table_path = tmp_dir.path().join("table-with-dv-small"); - let source_path = std::fs::canonicalize(PathBuf::from("./tests/data/table-with-dv-small/"))?; - copy_directory(&source_path, &tmp_table_path)?; - - let table_url = url::Url::from_directory_path(&tmp_table_path).unwrap(); - let engine = create_default_engine(&table_url)?; - - let snapshot = Snapshot::builder_for(table_url.clone()) - .at_version(1) - .build(engine.as_ref())?; - - // Create transaction with DV update mode enabled - let mut txn = snapshot - .clone() - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_engine_info("test engine") - .with_operation("UPDATE".to_string()) - .with_data_change(true); - - // Build scan and collect all scan metadata - let scan = snapshot.clone().scan_builder().build()?; - let all_scan_metadata: Vec<_> = scan - .scan_metadata(engine.as_ref())? - .collect::, _>>()?; - - // Extract scan files for DV update - let scan_files: Vec<_> = all_scan_metadata - .into_iter() - .map(|sm| sm.scan_files) - .collect(); - - // Create new DV descriptors for the files - let file_path = "part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet"; - let mut dv_map = HashMap::new(); - - // Create a NEW deletion vector descriptor (different from the original) - let new_dv = DeletionVectorDescriptor { - storage_type: DeletionVectorStorageType::PersistedRelative, - path_or_inline_dv: "cd^-aqEH.-t@S}K{vb[*k^".to_string(), - offset: Some(10), - size_in_bytes: 40, - cardinality: 3, - }; - dv_map.insert(file_path.to_string(), new_dv); - - // Call update_deletion_vectors to exercise the API - txn.update_deletion_vectors(dv_map, scan_files.into_iter().map(Ok))?; - - // Commit the transaction - let result = txn.commit(engine.as_ref())?; - - match result { - CommitResult::CommittedTransaction(committed) => { - let commit_version = committed.commit_version(); - - // Read the original version 1 log to get original file metadata - let original_log_path = tmp_table_path.join("_delta_log/00000000000000000001.json"); - let original_log_content = std::fs::read_to_string(original_log_path)?; - let original_commits: Vec<_> = Deserializer::from_str(&original_log_content) - .into_iter::() - .try_collect()?; - - let file_path = "part-00000-fae5310a-a37d-4e51-827b-c3d5516560ca-c000.snappy.parquet"; - - // Extract original file metadata from version 1 - let original_add = original_commits - .iter() - .find(|action| { - action - .get("add") - .and_then(|add| add.get("path").and_then(|p| p.as_str())) - == Some(file_path) - }) - .expect("Missing original add action in version 1") - .get("add") - .expect("Should have add field"); - - let original_size = original_add["size"] - .as_i64() - .expect("Original add action should have size"); - let original_partition_values = original_add["partitionValues"] - .as_object() - .expect("Original add action should have partitionValues"); - let original_tags = original_add.get("tags"); - let original_stats = original_add.get("stats"); - - // Read the commit log directly - let commit_path = tmp_table_path.join(format!("_delta_log/{commit_version:020}.json")); - let commit_content = std::fs::read_to_string(commit_path)?; - - let parsed_commits: Vec<_> = Deserializer::from_str(&commit_content) - .into_iter::() - .try_collect()?; - - // Should have commitInfo, remove, and add actions - assert!( - parsed_commits.len() >= 3, - "Expected at least 3 actions (commitInfo + remove + add), got {}", - parsed_commits.len() - ); - - // Extract commitInfo timestamp - let commit_info_action = parsed_commits - .iter() - .find(|action| action.get("commitInfo").is_some()) - .expect("Missing commitInfo action"); - let commit_info = &commit_info_action["commitInfo"]; - let commit_timestamp = commit_info["timestamp"] - .as_i64() - .expect("Missing timestamp in commitInfo"); - - // Verify remove action contains OLD DV information - let remove_actions: Vec<_> = parsed_commits - .iter() - .filter(|action| action.get("remove").is_some()) - .collect(); - - assert_eq!( - remove_actions.len(), - 1, - "Expected exactly one remove action" - ); - - let remove_action = remove_actions[0]; - let remove = &remove_action["remove"]; - - assert_eq!( - remove["path"].as_str(), - Some(file_path), - "Remove path should match" - ); - assert_eq!(remove["dataChange"].as_bool(), Some(true)); - assert_eq!( - remove["deletionTimestamp"].as_i64(), - Some(commit_timestamp), - "deletionTimestamp should match commit timestamp" - ); - - // Verify OLD deletion vector in remove action - let old_dv = remove["deletionVector"] - .as_object() - .expect("Remove action should have deletionVector"); - assert_eq!( - old_dv.get("storageType").and_then(|v| v.as_str()), - Some("u"), - "Old DV storage type should be 'u'" - ); - assert_eq!( - old_dv.get("pathOrInlineDv").and_then(|v| v.as_str()), - Some("vBn[lx{q8@P<9BNH/isA"), - "Old DV path should match original" - ); - assert_eq!( - old_dv.get("offset").and_then(|v| v.as_i64()), - Some(1), - "Old DV offset should be 1" - ); - assert_eq!( - old_dv.get("sizeInBytes").and_then(|v| v.as_i64()), - Some(36), - "Old DV size should be 36" - ); - assert_eq!( - old_dv.get("cardinality").and_then(|v| v.as_i64()), - Some(2), - "Old DV cardinality should be 2" - ); - - // Verify file metadata is preserved in remove action - let remove_size = remove["size"] - .as_i64() - .expect("Remove action should have size"); - let remove_partition_values = remove["partitionValues"] - .as_object() - .expect("Remove action should have partitionValues"); - let remove_tags = remove.get("tags"); - let remove_stats = remove.get("stats"); - - // Verify add action contains NEW DV information - let add_actions: Vec<_> = parsed_commits - .iter() - .filter(|action| action.get("add").is_some()) - .collect(); - - assert_eq!(add_actions.len(), 1, "Expected exactly one add action"); - - let add_action = add_actions[0]; - let add = &add_action["add"]; - - assert_eq!( - add["path"].as_str(), - Some(file_path), - "Add path should match" - ); - assert_eq!(add["dataChange"].as_bool(), Some(true)); - - // Verify NEW deletion vector in add action - let new_dv = add["deletionVector"] - .as_object() - .expect("Add action should have deletionVector"); - assert_eq!( - new_dv.get("storageType").and_then(|v| v.as_str()), - Some("u"), - "New DV storage type should be 'u'" - ); - assert_eq!( - new_dv.get("pathOrInlineDv").and_then(|v| v.as_str()), - Some("cd^-aqEH.-t@S}K{vb[*k^"), - "New DV path should match updated value" - ); - assert_eq!( - new_dv.get("offset").and_then(|v| v.as_i64()), - Some(10), - "New DV offset should be 10" - ); - assert_eq!( - new_dv.get("sizeInBytes").and_then(|v| v.as_i64()), - Some(40), - "New DV size should be 40" - ); - assert_eq!( - new_dv.get("cardinality").and_then(|v| v.as_i64()), - Some(3), - "New DV cardinality should be 3" - ); - - // Verify file metadata is preserved in add action - let add_size = add["size"].as_i64().expect("Add action should have size"); - let add_partition_values = add["partitionValues"] - .as_object() - .expect("Add action should have partitionValues"); - let add_tags = add.get("tags"); - let add_stats = add.get("stats"); - - // Ensure metadata is consistent between remove and add actions - assert_eq!( - remove_size, add_size, - "File size should be preserved between remove and add" - ); - assert_eq!( - remove_partition_values, add_partition_values, - "Partition values should be preserved between remove and add" - ); - assert_eq!( - remove_tags, add_tags, - "Tags should be preserved between remove and add" - ); - assert_eq!( - remove_stats, add_stats, - "Stats should be preserved between remove and add" - ); - - // Ensure metadata matches the original file metadata from version 1 - assert_eq!( - remove_size, original_size, - "Remove action size should match original file size" - ); - assert_eq!( - add_size, original_size, - "Add action size should match original file size" - ); - assert_eq!( - remove_partition_values, original_partition_values, - "Remove action partition values should match original" - ); - assert_eq!( - add_partition_values, original_partition_values, - "Add action partition values should match original" - ); - assert_eq!( - remove_tags, original_tags, - "Remove action tags should match original" - ); - assert_eq!( - add_tags, original_tags, - "Add action tags should match original" - ); - assert_eq!( - remove_stats, original_stats, - "Remove action stats should match original" - ); - assert_eq!( - add_stats, original_stats, - "Add action stats should match original" - ); - } - _ => panic!("Transaction should be committed"), - } - - Ok(()) -} - -#[tokio::test] -async fn test_update_deletion_vectors_multiple_files() -> Result<(), Box> { - // This test verifies that update_deletion_vectors can update multiple files - // in a single call, creating proper Remove and Add actions for each file. - let _ = tracing_subscriber::fmt::try_init(); - - let schema = Arc::new(StructType::try_new(vec![ - StructField::nullable("id", DataType::INTEGER), - StructField::nullable("value", DataType::STRING), - ])?); - - // Setup: Create table with 3 files - let file_names = &["file0.parquet", "file1.parquet", "file2.parquet"]; - let (store, engine, table_url, file_paths) = - create_dv_table_with_files("test_table", schema, file_names).await?; - - // Create DV update transaction - let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; - let mut txn = snapshot - .clone() - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_engine_info("test engine") - .with_operation("UPDATE".to_string()) - .with_data_change(true); - - let mut scan_files = get_scan_files(snapshot.clone(), engine.as_ref())?; - - // Update deletion vectors for all 3 files in a single call - let mut dv_map = HashMap::new(); - for (idx, file_path) in file_paths.iter().enumerate() { - let descriptor = DeletionVectorDescriptor { - storage_type: DeletionVectorStorageType::PersistedRelative, - path_or_inline_dv: format!("dv_file_{idx}.bin"), - offset: Some(idx as i32 * 10), - size_in_bytes: 40 + idx as i32, - cardinality: idx as i64 + 1, - }; - dv_map.insert(file_path.to_string(), descriptor); - } - - txn.update_deletion_vectors(dv_map, scan_files.drain(..).map(Ok))?; - - // Commit the transaction - let result = txn.commit(engine.as_ref())?; - - match result { - CommitResult::CommittedTransaction(committed) => { - let commit_version = committed.commit_version(); - - // Read the commit log directly from object store - let final_commit_path = - table_url.join(&format!("_delta_log/{commit_version:020}.json"))?; - let commit_content = store - .get(&Path::from_url_path(final_commit_path.path())?) - .await? - .bytes() - .await?; - - let parsed_commits: Vec<_> = Deserializer::from_slice(&commit_content) - .into_iter::() - .try_collect()?; - - // Extract all remove and add actions - let remove_actions: Vec<_> = parsed_commits - .iter() - .filter(|action| action.get("remove").is_some()) - .collect(); - - let add_actions: Vec<_> = parsed_commits - .iter() - .filter(|action| action.get("add").is_some()) - .collect(); - - // Should have 3 remove and 3 add actions - assert_eq!( - remove_actions.len(), - 3, - "Expected 3 remove actions for 3 files" - ); - assert_eq!(add_actions.len(), 3, "Expected 3 add actions for 3 files"); - - // Verify each file has a DV in both remove and add - for (idx, file_path) in file_paths.iter().enumerate() { - // Find the remove action for this file - let remove_action = remove_actions - .iter() - .find(|action| action["remove"]["path"].as_str() == Some(file_path.as_str())) - .unwrap_or_else(|| panic!("Should find remove action for {file_path}")); - - // Find the add action for this file - let add_action = add_actions - .iter() - .find(|action| action["add"]["path"].as_str() == Some(file_path.as_str())) - .unwrap_or_else(|| panic!("Should find add action for {file_path}")); - - // Verify remove action does NOT have a DV (since these were newly written files) - assert!( - remove_action["remove"]["deletionVector"].is_null(), - "Remove action for newly written file should not have a DV" - ); - - // Verify add action has the NEW DV - let add_dv = add_action["add"]["deletionVector"] - .as_object() - .expect("Add action should have deletionVector"); - - let expected_path = format!("dv_file_{idx}.bin"); - assert_eq!( - add_dv.get("pathOrInlineDv").and_then(|v| v.as_str()), - Some(expected_path.as_str()), - "DV path should match for file {file_path}" - ); - assert_eq!( - add_dv.get("offset").and_then(|v| v.as_i64()), - Some(idx as i64 * 10), - "DV offset should match for file {file_path}" - ); - assert_eq!( - add_dv.get("sizeInBytes").and_then(|v| v.as_i64()), - Some(40 + idx as i64), - "DV size should match for file {file_path}" - ); - assert_eq!( - add_dv.get("cardinality").and_then(|v| v.as_i64()), - Some(idx as i64 + 1), - "DV cardinality should match for file {file_path}" - ); - } - } - _ => panic!("Transaction should be committed"), - } - - Ok(()) -} - -#[tokio::test] -async fn test_remove_files_verify_files_excluded_from_scan( -) -> Result<(), Box> { - // Adds and then removes files and then verifies they don't appear in the scan. - - // setup tracing - let _ = tracing_subscriber::fmt::try_init(); - - // create a simple table: one int column named 'number' - let schema = get_simple_int_schema(); - - for (table_url, engine, _store, _table_name) in - setup_test_tables(schema.clone(), &[], None, "test_table").await? - { - // First, add some files to the table - let engine = Arc::new(engine); - write_data_and_check_result_and_stats(table_url.clone(), schema.clone(), engine.clone(), 1) - .await?; - - // Get initial file count - let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; - let scan = snapshot.clone().scan_builder().build()?; - let scan_metadata = scan.scan_metadata(engine.as_ref())?.next().unwrap()?; - let (_, selection_vector) = scan_metadata.scan_files.into_parts(); - let initial_file_count = selection_vector.iter().filter(|&x| *x).count(); - - assert!(initial_file_count > 0); - - // Now create a transaction to remove files - let mut txn = snapshot - .clone() - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())?; - - // Create a new scan to get file metadata for removal - let scan2 = snapshot.scan_builder().build()?; - let scan_metadata2 = scan2.scan_metadata(engine.as_ref())?.next().unwrap()?; - - // Create FilteredEngineData for removal (select all rows for removal) - let file_remove_count = (scan_metadata2.scan_files.data().len() - - scan_metadata2.scan_files.selection_vector().len()) - + scan_metadata2 - .scan_files - .selection_vector() - .iter() - .filter(|&x| *x) - .count(); - assert!(file_remove_count > 0); - - // Add remove files to transaction - txn.remove_files(scan_metadata2.scan_files); - - // Commit the transaction - let result = txn.commit(engine.as_ref()); - - match result? { - CommitResult::CommittedTransaction(committed) => { - assert_eq!(committed.commit_version(), 2); - - let new_snapshot = Snapshot::builder_for(table_url.clone()) - .at_version(2) - .build(engine.as_ref())?; - - let new_scan = new_snapshot.scan_builder().build()?; - let mut new_file_count = 0; - for new_metadata in new_scan.scan_metadata(engine.as_ref())? { - new_file_count += new_metadata?.scan_files.data().len(); - } - - // All files were removed, so new_file_count should be zero - assert_eq!(new_file_count, 0); - } - _ => panic!("Transaction did not succeeed."), - } - } - Ok(()) -} - -#[tokio::test] -async fn test_remove_files_with_modified_selection_vector() -> Result<(), Box> -{ - // This test verifies that we can selectively remove files by: - // 1. Calling remove_files multiple times with different subsets - // 2. Modifying the selection vector to choose which files to remove - - let _ = tracing_subscriber::fmt::try_init(); - - let schema = get_simple_int_schema(); - - for (table_url, engine, _store, _table_name) in - setup_test_tables(schema.clone(), &[], None, "test_table").await? - { - let engine = Arc::new(engine); - - // Write data multiple times to create multiple files - for i in 1..=5 { - write_data_and_check_result_and_stats( - table_url.clone(), - schema.clone(), - engine.clone(), - i, - ) - .await?; - } - - // Get initial file count - let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; - let scan = snapshot.clone().scan_builder().build()?; - - let mut initial_file_count = 0; - for metadata in scan.scan_metadata(engine.as_ref())? { - let metadata = metadata?; - initial_file_count += metadata - .scan_files - .selection_vector() - .iter() - .filter(|&x| *x) - .count(); - } - - assert!( - initial_file_count >= 3, - "Need at least 3 files for this test, got {initial_file_count}" - ); - - // Create a transaction to remove files in two batches - let mut txn = snapshot - .clone() - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_engine_info("selective remove test") - .with_operation("DELETE".to_string()) - .with_data_change(true); - - // First batch: Remove only the first file - let scan2 = snapshot.clone().scan_builder().build()?; - let scan_metadata2 = scan2.scan_metadata(engine.as_ref())?.next().unwrap()?; - let (data, mut selection_vector) = scan_metadata2.scan_files.into_parts(); - - // Select only the first file for removal - let mut first_batch_removed = 0; - for selected in selection_vector.iter_mut() { - if *selected && first_batch_removed < 1 { - // Keep selected for removal - first_batch_removed += 1; - } else { - // Don't remove - *selected = false; - } - } - - assert_eq!( - first_batch_removed, 1, - "Should remove exactly 1 file in first batch" - ); - txn.remove_files(FilteredEngineData::try_new(data, selection_vector)?); - - // Second batch: Remove only the last file - let scan3 = snapshot.clone().scan_builder().build()?; - let scan_metadata3 = scan3.scan_metadata(engine.as_ref())?.next().unwrap()?; - let (data2, mut selection_vector2) = scan_metadata3.scan_files.into_parts(); - - // Find the last selected file and keep only that one selected - let mut last_selected_idx = None; - for (i, &selected) in selection_vector2.iter().enumerate() { - if selected { - last_selected_idx = Some(i); - } - } - - // Deselect all except the last one - for (i, selected) in selection_vector2.iter_mut().enumerate() { - if Some(i) != last_selected_idx { - *selected = false; - } - } - - let second_batch_removed = selection_vector2.iter().filter(|&x| *x).count(); - assert_eq!( - second_batch_removed, 1, - "Should remove exactly 1 file in second batch" - ); - txn.remove_files(FilteredEngineData::try_new(data2, selection_vector2)?); - - // Commit the transaction - let result = txn.commit(engine.as_ref())?; - - match result { - CommitResult::CommittedTransaction(committed) => { - assert_eq!(committed.commit_version(), 6); - - // Verify that exactly 2 files were removed (1 from each batch) - let new_snapshot = Snapshot::builder_for(table_url.clone()) - .at_version(6) - .build(engine.as_ref())?; - - let new_scan = new_snapshot.scan_builder().build()?; - let mut new_file_count = 0; - for new_metadata in new_scan.scan_metadata(engine.as_ref())? { - let metadata = new_metadata?; - new_file_count += metadata - .scan_files - .selection_vector() - .iter() - .filter(|&x| *x) - .count(); - } - - // Verify we removed exactly 2 files (1 + 1) - let total_removed = first_batch_removed + second_batch_removed; - assert_eq!(total_removed, 2); - assert_eq!(new_file_count, initial_file_count - total_removed); - assert!(new_file_count > 0, "At least one file should remain"); - } - _ => panic!("Transaction did not succeed"), - } - } - Ok(()) -} - -// Helper function to create a table with CDF enabled -async fn create_cdf_table( - table_name: &str, - schema: SchemaRef, -) -> Result<(Url, Arc>, TempDir), Box> -{ - let tmp_dir = tempdir()?; - let tmp_test_dir_url = Url::from_directory_path(tmp_dir.path()).unwrap(); - - let (store, engine, table_location) = engine_store_setup(table_name, Some(&tmp_test_dir_url)); - - let table_url = create_table( - store.clone(), - table_location, - schema.clone(), - &[], - true, // use protocol 3.7 - vec![], - vec!["changeDataFeed"], - ) - .await?; - - Ok((table_url, Arc::new(engine), tmp_dir)) -} - -// Helper function to write data to a table -async fn write_data_to_table( - table_url: &Url, - engine: &Arc>, - schema: SchemaRef, - values: Vec, -) -> Result> { - let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; - let mut txn = snapshot - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_engine_info("test"); - - add_files_to_transaction(&mut txn, engine, schema, values).await?; - - let result = txn.commit(engine.as_ref())?; - match result { - CommitResult::CommittedTransaction(committed) => Ok(committed.commit_version()), - _ => panic!("Transaction should be committed"), - } -} - -// Helper function to add files to an existing transaction -async fn add_files_to_transaction( - txn: &mut delta_kernel::transaction::Transaction, - engine: &Arc>, - schema: SchemaRef, - values: Vec, -) -> Result<(), Box> { - let data = RecordBatch::try_new( - Arc::new(schema.as_ref().try_into_arrow()?), - vec![Arc::new(Int32Array::from(values))], - )?; - - let write_context = Arc::new(txn.get_write_context()); - let add_files_metadata = engine - .write_parquet( - &ArrowEngineData::new(data), - write_context.as_ref(), - HashMap::new(), - ) - .await?; - txn.add_files(add_files_metadata); - Ok(()) -} - -#[tokio::test] -async fn test_cdf_write_all_adds_succeeds() -> Result<(), Box> { - // This test verifies that add-only transactions work with CDF enabled - let _ = tracing_subscriber::fmt::try_init(); - - let schema = get_simple_int_schema(); - - let (table_url, engine, _tmp_dir) = - create_cdf_table("test_cdf_all_adds", schema.clone()).await?; - - // Add files - this should succeed - let version = write_data_to_table(&table_url, &engine, schema, vec![1, 2, 3]).await?; - assert_eq!(version, 1); - - Ok(()) -} - -#[tokio::test] -async fn test_cdf_write_all_removes_succeeds() -> Result<(), Box> { - // This test verifies that remove-only transactions work with CDF enabled - let _ = tracing_subscriber::fmt::try_init(); - - let schema = get_simple_int_schema(); - - let (table_url, engine, _tmp_dir) = - create_cdf_table("test_cdf_all_removes", schema.clone()).await?; - - // First, add some data - write_data_to_table(&table_url, &engine, schema, vec![1, 2, 3]).await?; - - // Now remove the files - let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; - let mut txn = snapshot - .clone() - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_engine_info("cdf remove test") - .with_data_change(true); - - let scan = snapshot.scan_builder().build()?; - let scan_metadata = scan.scan_metadata(engine.as_ref())?.next().unwrap()?; - let (data, selection_vector) = scan_metadata.scan_files.into_parts(); - txn.remove_files(FilteredEngineData::try_new(data, selection_vector)?); - - // This should succeed - remove-only transactions are allowed with CDF - let result = txn.commit(engine.as_ref())?; - match result { - CommitResult::CommittedTransaction(committed) => { - assert_eq!(committed.commit_version(), 2); - } - _ => panic!("Transaction should be committed"), - } - - Ok(()) -} - -#[tokio::test] -async fn test_cdf_write_mixed_no_data_change_succeeds() -> Result<(), Box> { - // This test verifies that mixed add+remove transactions work when dataChange=false. - // It's allowed because the transaction does not contain any logical data changes. - // This can happen when a table is being optimized/compacted. - let _ = tracing_subscriber::fmt::try_init(); - - let schema = get_simple_int_schema(); - - let (table_url, engine, _tmp_dir) = - create_cdf_table("test_cdf_mixed_no_data_change", schema.clone()).await?; - - // First, add some data - write_data_to_table(&table_url, &engine, schema.clone(), vec![1, 2, 3]).await?; - - // Now create a transaction with both add AND remove files, but dataChange=false - let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; - let mut txn = snapshot - .clone() - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_engine_info("cdf mixed test") - .with_data_change(false); // dataChange=false is key here - - // Add new files - add_files_to_transaction(&mut txn, &engine, schema, vec![4, 5, 6]).await?; - - // Also remove existing files - let scan = snapshot.scan_builder().build()?; - let scan_metadata = scan.scan_metadata(engine.as_ref())?.next().unwrap()?; - let (data, selection_vector) = scan_metadata.scan_files.into_parts(); - txn.remove_files(FilteredEngineData::try_new(data, selection_vector)?); - - // This should succeed - mixed operations are allowed when dataChange=false - let result = txn.commit(engine.as_ref())?; - match result { - CommitResult::CommittedTransaction(committed) => { - assert_eq!(committed.commit_version(), 2); - } - _ => panic!("Transaction should be committed"), - } - - Ok(()) -} - -#[tokio::test] -async fn test_cdf_write_mixed_with_data_change_fails() -> Result<(), Box> { - // This test verifies that mixed add+remove transactions fail with helpful error when dataChange=true - let _ = tracing_subscriber::fmt::try_init(); - - let schema = get_simple_int_schema(); - - let (table_url, engine, _tmp_dir) = - create_cdf_table("test_cdf_mixed_with_data_change", schema.clone()).await?; - - // First, add some data - write_data_to_table(&table_url, &engine, schema.clone(), vec![1, 2, 3]).await?; - - // Now create a transaction with both add AND remove files with dataChange=true - let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; - let mut txn = snapshot - .clone() - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_engine_info("cdf mixed fail test") - .with_data_change(true); // dataChange=true - this should fail - - // Add new files - add_files_to_transaction(&mut txn, &engine, schema, vec![4, 5, 6]).await?; - - // Also remove existing files - let scan = snapshot.scan_builder().build()?; - let scan_metadata = scan.scan_metadata(engine.as_ref())?.next().unwrap()?; - let (data, selection_vector) = scan_metadata.scan_files.into_parts(); - txn.remove_files(FilteredEngineData::try_new(data, selection_vector)?); - - // This should fail with our new error message - assert_result_error_with_message( - txn.commit(engine.as_ref()), - "Cannot add and remove data in the same transaction when Change Data Feed is enabled (delta.enableChangeDataFeed = true). \ - This would require writing CDC files for DML operations, which is not yet supported. \ - Consider using separate transactions: one to add files, another to remove files." - ); - - Ok(()) -} - -#[tokio::test] -async fn test_post_commit_snapshot_create_then_insert() -> DeltaResult<()> { - let _ = tracing_subscriber::fmt::try_init(); - - let temp_dir = tempdir().unwrap(); - let table_url = Url::from_directory_path(temp_dir.path()).unwrap(); - let engine = create_default_engine(&table_url)?; - let schema = get_simple_int_schema(); - - // Create table and verify post_commit_snapshot - let create_result = create_table_txn(table_url.as_str(), schema, env!("CARGO_PKG_VERSION")) - .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? - .commit(engine.as_ref())?; - - let mut current_snapshot = match create_result { - CommitResult::CommittedTransaction(committed) => { - assert_eq!(committed.commit_version(), 0); - let post_snapshot = committed - .post_commit_snapshot() - .expect("should have post_commit_snapshot"); - assert_eq!(post_snapshot.version(), 0); - post_snapshot.clone() - } - _ => panic!("Create should succeed"), - }; - - // Do 10 inserts and verify post_commit_snapshot for each - for i in 1..11 { - let base_version = current_snapshot.version(); - - let txn = current_snapshot - .clone() - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_engine_info("test"); - - match txn.commit(engine.as_ref())? { - CommitResult::CommittedTransaction(committed) => { - let post_snapshot = committed - .post_commit_snapshot() - .expect("should have post_commit_snapshot"); - - assert_eq!(post_snapshot.version(), base_version + 1); - assert_eq!(post_snapshot.version(), committed.commit_version()); - assert_eq!(post_snapshot.schema(), current_snapshot.schema()); - assert_eq!(post_snapshot.table_root(), current_snapshot.table_root()); - - current_snapshot = post_snapshot.clone(); - } - _ => panic!("Commit {i} should succeed"), - } - } - - Ok(()) -} - -#[tokio::test] -async fn test_write_parquet_succeed_with_logical_partition_names( -) -> Result<(), Box> { - let schema = Arc::new(StructType::try_new(vec![ - StructField::nullable("id", DataType::INTEGER), - StructField::nullable("letter", DataType::STRING), - ])?); - - for (table_url, engine, _store, _table_name) in setup_test_tables( - schema.clone(), - &["letter"], - None, - "test_partition_translate", - ) - .await? - { - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - - // Create data with only the non-partition column - let data_schema = Arc::new( - StructType::try_new(vec![StructField::nullable("id", DataType::INTEGER)]).unwrap(), - ); - let batch = RecordBatch::try_new( - Arc::new(data_schema.as_ref().try_into_arrow()?), - vec![Arc::new(Int32Array::from(vec![1, 2]))], - )?; - - // Pass partition values with logical name — should succeed - let result = write_batch_to_table( - &snapshot, - &engine, - batch, - HashMap::from([("letter".to_string(), "a".to_string())]), - ) - .await; - assert!( - result.is_ok(), - "write_parquet should succeed with valid logical partition name" - ); - } - Ok(()) -} - -#[tokio::test] -async fn test_write_parquet_rejects_unknown_partition_column( -) -> Result<(), Box> { - let schema = get_simple_int_schema(); - - for (table_url, engine, _store, _table_name) in - setup_test_tables(schema.clone(), &[], None, "test_partition_reject").await? - { - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - - let batch = RecordBatch::try_new( - Arc::new(schema.as_ref().try_into_arrow()?), - vec![Arc::new(Int32Array::from(vec![1, 2]))], - )?; - - let result = write_batch_to_table( - &snapshot, - &engine, - batch, - HashMap::from([("nonexistent".to_string(), "val".to_string())]), - ) - .await; - let err = result.expect_err("write_parquet should fail with unknown partition column"); - let err_msg = err.to_string(); - assert!( - err_msg.contains("Partition column 'nonexistent' not found in table schema"), - "Error should mention the unknown column name, got: {err_msg}" - ); - } - Ok(()) -} - -/// 1. Creates a table with the given column mapping mode -/// 2. Writes two batches of data -/// 3. Checkpoints and verifies add.stats uses physical column names in the checkpoint -/// 4. Reads a parquet footer to verify physical names/IDs -/// 5. Reads data back to verify correctness -/// 6. Removes files and verifies remove.stats matches the original add.stats -#[rstest::rstest] -#[case::cm_none(ColumnMappingMode::None)] -#[case::cm_id(ColumnMappingMode::Id)] -#[case::cm_name(ColumnMappingMode::Name)] -#[tokio::test(flavor = "multi_thread")] -async fn test_column_mapping_write( - #[case] cm_mode: ColumnMappingMode, -) -> Result<(), Box> { - let _ = tracing_subscriber::fmt::try_init(); - - let schema = nested_schema()?; - - let (_tmp_dir, table_path, _) = test_table_setup()?; - let table_url = Url::from_directory_path(&table_path).unwrap(); - let store: Arc = Arc::new(LocalFileSystem::new()); - let engine = Arc::new( - DefaultEngineBuilder::new(store.clone()) - .with_task_executor(Arc::new(TokioMultiThreadExecutor::new( - tokio::runtime::Handle::current(), - ))) - .build(), - ); - - // Step 1: Create table - let mode_str = match cm_mode { - ColumnMappingMode::None => "none", - ColumnMappingMode::Id => "id", - ColumnMappingMode::Name => "name", - }; - let mut latest_snapshot = create_table_and_load_snapshot( - &table_path, - schema.clone(), - engine.as_ref(), - &[("delta.columnMapping.mode", mode_str)], - )?; - - // Get physical field paths for stats verification (top-level and nested) - let cm = latest_snapshot - .table_properties() - .column_mapping_mode - .unwrap_or(ColumnMappingMode::None); - let row_number_physical = get_any_level_column_physical_name( - latest_snapshot.schema().as_ref(), - &ColumnName::new(["row_number"]), - cm, - )? - .into_inner(); - let street_physical = get_any_level_column_physical_name( - latest_snapshot.schema().as_ref(), - &ColumnName::new(["address", "street"]), - cm, - )? - .into_inner(); - - // Step 2: Write two batches - for data in nested_batches()? { - latest_snapshot = - write_batch_to_table(&latest_snapshot, engine.as_ref(), data, HashMap::new()).await?; - } - - // Enable writeStatsAsStruct so the checkpoint contains native stats_parsed. - // CREATE TABLE doesn't allow this property yet, so we write a metadata-update commit directly. - latest_snapshot = set_table_properties( - &table_path, - &table_url, - engine.as_ref(), - latest_snapshot.version(), - &[("delta.checkpoint.writeStatsAsStruct", "true")], - )?; - - // Step 3: Checkpoint and verify add.stats uses correct column names - let snapshot_for_checkpoint = latest_snapshot.clone(); - snapshot_for_checkpoint.checkpoint(engine.as_ref())?; - let ckpt_snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; - let add_actions = read_add_infos(&ckpt_snapshot, engine.as_ref())?; - let mut all_stats: Vec<_> = add_actions - .iter() - .filter_map(|a| a.stats.as_ref()) - .filter(|s| s.get("minValues").is_some()) - .collect(); - assert_eq!(all_stats.len(), 2, "should have stats for 2 files"); - all_stats.sort_by_key(|s| s["minValues"][&row_number_physical[0]].as_i64().unwrap()); - - // Batch 1: row_number 1..3, address.street "st1".."st3" - assert_min_max_stats(all_stats[0], &row_number_physical, 1, 3); - assert_min_max_stats(all_stats[0], &street_physical, "st1", "st3"); - - // Batch 2: row_number 4..6, address.street "st4".."st6" - assert_min_max_stats(all_stats[1], &row_number_physical, 4, 6); - assert_min_max_stats(all_stats[1], &street_physical, "st4", "st6"); - - // Step 3b: Verify stats_parsed in scan metadata uses correct physical column names - { - let scan = ckpt_snapshot - .scan_builder() - .include_all_stats_columns() - .build()?; - let scan_metadata_results: Vec<_> = scan - .scan_metadata(engine.as_ref())? - .collect::, _>>()?; - - let mut stats_rows: Vec<(i64, i64, String, String)> = Vec::new(); - for sm in scan_metadata_results { - let (data, sel) = sm.scan_files.into_parts(); - let batch: RecordBatch = ArrowEngineData::try_from_engine_data(data)?.into(); - - let batch_struct = StructArray::from(batch.clone()); - let stats_parsed: &StructArray = - resolve_struct_field(&batch_struct, &["stats_parsed".into()]); - - let min_path = |field: &[String]| -> Vec { - [&["stats_parsed".into(), "minValues".into()], field].concat() - }; - let max_path = |field: &[String]| -> Vec { - [&["stats_parsed".into(), "maxValues".into()], field].concat() - }; - let min_row_num: &Int64Array = - resolve_struct_field(&batch_struct, &min_path(&row_number_physical)); - let max_row_num: &Int64Array = - resolve_struct_field(&batch_struct, &max_path(&row_number_physical)); - let min_st: &StringArray = - resolve_struct_field(&batch_struct, &min_path(&street_physical)); - let max_st: &StringArray = - resolve_struct_field(&batch_struct, &max_path(&street_physical)); - - for (i, &selected) in sel.iter().enumerate().take(batch.num_rows()) { - if selected && !stats_parsed.is_null(i) { - stats_rows.push(( - min_row_num.value(i), - max_row_num.value(i), - min_st.value(i).to_string(), - max_st.value(i).to_string(), - )); - } - } - } - - stats_rows.sort_by_key(|r| r.0); - assert_eq!(stats_rows.len(), 2, "should have stats_parsed for 2 files"); - assert_eq!(stats_rows[0], (1, 3, "st1".to_string(), "st3".to_string())); - assert_eq!(stats_rows[1], (4, 6, "st4".to_string(), "st6".to_string())); - } - - // Step 4: Read parquet footer to verify physical names and native field_id - { - let parquet_path = &add_actions - .first() - .expect("should have at least one add file") - .path; - let parquet_url = table_url.join(parquet_path)?; - let local_path = parquet_url.to_file_path().unwrap(); - - let obj_meta = store - .head(&Path::from_url_path(parquet_url.path())?) - .await?; - let file_meta = FileMeta::new( - parquet_url, - 0, /* last_modified */ - obj_meta.size as u64, - ); - let footer = engine.parquet_handler().read_parquet_footer(&file_meta)?; - let footer_schema = footer.schema; - - let logical_schema = latest_snapshot.schema(); - for logical_path in [&["row_number"][..], &["address", "street"]] { - let col = ColumnName::new(logical_path.iter().copied()); - let physical = - get_any_level_column_physical_name(logical_schema.as_ref(), &col, cm)?.into_inner(); - assert_schema_has_field(&footer_schema, &physical); - - let field_id = get_parquet_field_id(&local_path, &physical); - let logical_field = resolve_field(logical_schema.as_ref(), logical_path).unwrap(); - match cm_mode { - ColumnMappingMode::Id | ColumnMappingMode::Name => { - let expected_id = - match logical_field.get_config_value(&ColumnMetadataKey::ColumnMappingId) { - Some(MetadataValue::Number(n)) => *n as i32, - other => panic!("expected ColumnMappingId number, got {other:?}"), - }; - assert_eq!( - field_id, - Some(expected_id), - "parquet field_id mismatch for {logical_path:?}" - ); - } - ColumnMappingMode::None => { - assert_eq!( - field_id, None, - "parquet field_id should not be set in None column mapping mode" - ); - } - } - } - } - - // Step 5: Read data back to verify correctness - { - let post_ckpt_snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; - let scan = post_ckpt_snapshot.scan_builder().build()?; - let batches: Vec = scan - .execute(engine.clone())? - .map(|r| { - let data = r.unwrap(); - let arrow = ArrowEngineData::try_from_engine_data(data).unwrap(); - arrow.record_batch().clone() - }) - .collect(); - - let result_schema = batches[0].schema(); - let combined = delta_kernel::arrow::compute::concat_batches(&result_schema, &batches)?; - assert_eq!( - combined.num_rows(), - 6, - "Should have 6 rows from two written batches" - ); - - // Verify logical column names and data values - let combined_struct = StructArray::from(combined); - - // Top-level: row_number should contain [1..=6] - let row_numbers: &Int64Array = - resolve_struct_field(&combined_struct, &["row_number".into()]); - let mut vals: Vec = (0..row_numbers.len()) - .map(|i| row_numbers.value(i)) - .collect(); - vals.sort(); - assert_eq!(vals, vec![1, 2, 3, 4, 5, 6]); - - // Nested: address.street should contain ["st1"..="st6"] - let streets: &StringArray = - resolve_struct_field(&combined_struct, &["address".into(), "street".into()]); - let mut street_vals: Vec<&str> = (0..streets.len()).map(|i| streets.value(i)).collect(); - street_vals.sort(); - assert_eq!(street_vals, vec!["st1", "st2", "st3", "st4", "st5", "st6"]); - } - - // Step 6: Remove files and verify remove.stats matches original add.stats - { - let original_add_stats: Vec = - add_actions.iter().filter_map(|a| a.stats.clone()).collect(); - assert!( - !original_add_stats.is_empty(), - "should have at least one add with stats" - ); - - let remove_actions = - remove_all_and_get_remove_actions(&latest_snapshot, &table_url, engine.as_ref())?; - assert!( - !remove_actions.is_empty(), - "Expected at least one remove action" - ); - - let remove_stats: Vec = remove_actions - .iter() - .filter_map(|r| { - r["stats"] - .as_str() - .map(|s| serde_json::from_str(s).unwrap()) - }) - .collect(); - assert_eq!( - remove_stats, original_add_stats, - "remove.stats should match original add.stats" - ); - } - - Ok(()) -} - -/// Verifies that partitioned writes use physical column names in add.partitionValues. -#[rstest::rstest] -#[case::cm_none("./tests/data/partition_cm/none")] -#[case::cm_id("./tests/data/partition_cm/id")] -#[case::cm_name("./tests/data/partition_cm/name")] -#[tokio::test(flavor = "multi_thread")] -async fn test_column_mapping_partitioned_write( - #[case] table_dir: &str, -) -> Result<(), Box> { - let _ = tracing_subscriber::fmt::try_init(); - - // Copy test data to a temp dir so we can write to it - let tmp_dir = tempdir()?; - copy_directory(std::path::Path::new(table_dir), tmp_dir.path())?; - let table_url = Url::from_directory_path(tmp_dir.path()).unwrap(); - let store: Arc = Arc::new(LocalFileSystem::new()); - let engine = Arc::new( - DefaultEngineBuilder::new(store.clone()) - .with_task_executor(Arc::new(TokioMultiThreadExecutor::new( - tokio::runtime::Handle::current(), - ))) - .build(), - ); - - let snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; - let cm = snapshot - .table_properties() - .column_mapping_mode - .unwrap_or(ColumnMappingMode::None); - let physical_name = get_any_level_column_physical_name( - snapshot.schema().as_ref(), - &ColumnName::new(["category"]), - cm, - )? - .into_inner() - .remove(0); - - // Verify physical name for column mapping mode - if table_dir.ends_with("none") { - assert_eq!(physical_name, "category"); - } else { - assert_ne!( - physical_name, "category", - "physical name should differ from logical name under column mapping" - ); - } - - // Write data with partition value - let data_schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "value", - DataType::INTEGER, - )])?); - let batch = RecordBatch::try_new( - Arc::new(data_schema.as_ref().try_into_arrow()?), - vec![Arc::new(Int32Array::from(vec![1, 2]))], - )?; - let partition_values = HashMap::from([("category".to_string(), "A".to_string())]); - write_batch_to_table(&snapshot, engine.as_ref(), batch, partition_values).await?; - - // Read commit log and verify add.partitionValues key uses physical name - let add_actions = read_actions_from_commit(&table_url, 1, "add")?; - assert!(!add_actions.is_empty(), "no add action found in commit log"); - for add in &add_actions { - assert_partition_values(add, &physical_name, "A"); - } - - // Remove the written file and verify remove action preserves physical names - let post_write_snapshot = Snapshot::builder_for(table_url.clone()).build(engine.as_ref())?; - let remove_actions = - remove_all_and_get_remove_actions(&post_write_snapshot, &table_url, engine.as_ref())?; - assert!( - !remove_actions.is_empty(), - "no remove action found in commit log" - ); - for remove in &remove_actions { - assert_partition_values(remove, &physical_name, "A"); - } - - Ok(()) -} - -#[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn test_checkpoint_non_kernel_written_table() { - // Table written by a non-kernel-integrated connector: 7 rows with columns (i, j, k), where - // parquet field nullabilities differ from the Delta schema. DefaultEngine reads it, coerces - // nullabilities to match the Delta schema, creates a checkpoint, and verifies the data is - // unchanged. - let source_path = std::path::Path::new("./tests/data/external-table-different-nullability"); - let temp_dir = tempfile::tempdir().unwrap(); - let table_path = temp_dir.path().join("test-checkpoint-table"); - test_utils::copy_directory(source_path, &table_path).unwrap(); - - let url = Url::from_directory_path(&table_path).unwrap(); - let store: Arc = Arc::new(LocalFileSystem::new()); - let executor = Arc::new( - delta_kernel::engine::default::executor::tokio::TokioMultiThreadExecutor::new( - tokio::runtime::Handle::current(), - ), - ); - let engine: Arc> = Arc::new( - delta_kernel::engine::default::DefaultEngineBuilder::new(store) - .with_task_executor(executor) - .build(), - ); - - // Read data before checkpoint - let snapshot = Snapshot::builder_for(url.clone()) - .build(engine.as_ref()) - .unwrap(); - let scan_before = Arc::clone(&snapshot).scan_builder().build().unwrap(); - let batches_before = test_utils::read_scan(&scan_before, engine.clone()).unwrap(); - - // Create checkpoint via snapshot.checkpoint() - Arc::clone(&snapshot).checkpoint(engine.as_ref()).unwrap(); - - // Read data after checkpoint - let snapshot_after = Snapshot::builder_for(url.clone()) - .build(engine.as_ref()) - .unwrap(); - let scan_after = snapshot_after.scan_builder().build().unwrap(); - let batches_after = test_utils::read_scan(&scan_after, engine.clone()).unwrap(); - - // Verify data unchanged - let formatted_before = - delta_kernel::arrow::util::pretty::pretty_format_batches(&batches_before) - .unwrap() - .to_string(); - let formatted_after = delta_kernel::arrow::util::pretty::pretty_format_batches(&batches_after) - .unwrap() - .to_string(); - assert_eq!( - formatted_before, formatted_after, - "Row data changed after checkpoint creation!" - ); - - // Verify checkpoint file exists - let delta_log_path = table_path.join("_delta_log"); - let has_checkpoint = std::fs::read_dir(&delta_log_path) - .unwrap() - .filter_map(|e| e.ok()) - .any(|e| { - e.file_name() - .to_str() - .is_some_and(|n| n.contains(".checkpoint.parquet")) - }); - assert!(has_checkpoint, "Expected at least one checkpoint file"); -} - -struct ClusteredTableSetup { - _tmp_dir: TempDir, - table_path: String, - table_url: Url, - engine: Arc>, - snapshot: Arc, -} - -/// Creates a clustered table with column mapping and sets table properties. -fn setup_clustered_table( - cm_mode: &str, - schema: Arc, - clustering_cols: Vec, - table_properties: &[(&str, &str)], -) -> Result> { - use delta_kernel::transaction::data_layout::DataLayout; - - let (_tmp_dir, table_path, _) = test_table_setup()?; - let table_url = Url::from_directory_path(&table_path).unwrap(); - let engine = create_default_engine_mt_executor(&table_url)?; - - let _ = create_table_txn(table_url.as_str(), schema, "Test/1.0") - .with_table_properties([("delta.columnMapping.mode", cm_mode)]) - .with_data_layout(DataLayout::Clustered { - columns: clustering_cols, - }) - .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? - .commit(engine.as_ref())?; - - let snapshot = set_table_properties( - &table_path, - &table_url, - engine.as_ref(), - 0, - table_properties, - )?; - - Ok(ClusteredTableSetup { - _tmp_dir, - table_path, - table_url, - engine, - snapshot, - }) -} - -/// E2E test: create a clustered table with column mapping, write data, and verify that -/// add.stats in the commit log contains min/max statistics for the clustering columns -/// (including a nested column). -#[rstest::rstest] -#[case::cm_none("none")] -#[case::cm_name("name")] -#[case::cm_id("id")] -#[tokio::test(flavor = "multi_thread")] -async fn test_clustered_table_write_has_stats( - #[case] cm_mode: &str, -) -> Result<(), Box> { - let _ = tracing_subscriber::fmt::try_init(); - - let clustering_cols = vec![ - ColumnName::new(["row_number"]), - ColumnName::new(["address", "street"]), - ]; - let setup = setup_clustered_table( - cm_mode, - nested_schema()?, - clustering_cols.clone(), - &[("delta.dataSkippingNumIndexedCols", "0")], - )?; - let engine = &setup.engine; - let mut snapshot = setup.snapshot; - for batch in nested_batches()? { - snapshot = write_batch_to_table(&snapshot, engine.as_ref(), batch, HashMap::new()).await?; - } - - let cm = assert_column_mapping_mode(&snapshot, cm_mode); - let physical_paths: Vec> = clustering_cols - .iter() - .map(|c| { - get_any_level_column_physical_name(snapshot.schema().as_ref(), c, cm) - .unwrap() - .into_inner() - }) - .collect(); - if cm != ColumnMappingMode::None { - let logical_paths: Vec> = vec![vec!["row_number"], vec!["address", "street"]]; - for (phys, logical) in physical_paths.iter().zip(&logical_paths) { - assert_ne!( - phys.iter().map(String::as_str).collect_vec(), - *logical, - "physical path should differ from logical when cm={cm:?}" - ); - } - } - - // Resolve a non-clustering column to verify it's excluded from stats - let non_clustering_physical = get_any_level_column_physical_name( - snapshot.schema().as_ref(), - &ColumnName::new(["name"]), - cm, - )? - .into_inner(); - - // Verify stats for each write commit (v2 and v3, since v1 is the property update). - // Batch 1 (v2): row_number 1..3, address.street "st1".."st3" - // Batch 2 (v3): row_number 4..6, address.street "st4".."st6" - let expected: [(i64, i64, &str, &str); 2] = [(1, 3, "st1", "st3"), (4, 6, "st4", "st6")]; - for (version, (min_rn, max_rn, min_st, max_st)) in expected.iter().enumerate() { - let version = (version + 2) as u64; - let add_actions = read_actions_from_commit(&setup.table_url, version, "add")?; - assert!( - !add_actions.is_empty(), - "v{version}: should have add actions" - ); - - for add in &add_actions { - let stats: serde_json::Value = serde_json::from_str( - add.get("stats") - .and_then(|s| s.as_str()) - .expect("add action should have stats"), - )?; - // Clustering columns should have stats despite numIndexedCols=0 - assert_min_max_stats(&stats, &physical_paths[0], *min_rn, *max_rn); - assert_min_max_stats(&stats, &physical_paths[1], *min_st, *max_st); - - // Non-clustering column "name" should NOT have stats - let non_cluster_min = resolve_json_path(&stats["minValues"], &non_clustering_physical); - assert!( - non_cluster_min.is_null(), - "v{version}: non-clustering column 'name' should not have stats" - ); - } - } - - Ok(()) -} - -/// E2E test: create a clustered table with column mapping, enable writeStatsAsStruct, -/// write data, checkpoint, and verify stats_parsed. -#[rstest::rstest] -#[case::cm_none("none")] -#[case::cm_name("name")] -#[case::cm_id("id")] -#[tokio::test(flavor = "multi_thread")] -async fn test_clustered_table_write_has_stats_parsed( - #[case] cm_mode: &str, -) -> Result<(), Box> { - let _ = tracing_subscriber::fmt::try_init(); - - let clustering_cols = vec![ - ColumnName::new(["row_number"]), - ColumnName::new(["address", "street"]), - ]; - let setup = setup_clustered_table( - cm_mode, - nested_schema()?, - clustering_cols.clone(), - &[ - ("delta.checkpoint.writeStatsAsStruct", "true"), - ("delta.dataSkippingNumIndexedCols", "0"), - ], - )?; - let engine = &setup.engine; - let mut snapshot = setup.snapshot; - for batch in nested_batches()? { - snapshot = write_batch_to_table(&snapshot, engine.as_ref(), batch, HashMap::new()).await?; - } - - let cm = assert_column_mapping_mode(&snapshot, cm_mode); - let physical_paths: Vec> = clustering_cols - .iter() - .map(|c| { - get_any_level_column_physical_name(snapshot.schema().as_ref(), c, cm) - .unwrap() - .into_inner() - }) - .collect(); - if cm != ColumnMappingMode::None { - let logical_paths: Vec> = vec![vec!["row_number"], vec!["address", "street"]]; - for (phys, logical) in physical_paths.iter().zip(&logical_paths) { - assert_ne!( - phys.iter().map(String::as_str).collect_vec(), - *logical, - "physical path should differ from logical when cm={cm:?}" - ); - } - } - let non_clustering_physical = get_any_level_column_physical_name( - snapshot.schema().as_ref(), - &ColumnName::new(["name"]), - cm, - )? - .into_inner(); - - snapshot.checkpoint(engine.as_ref())?; - - // Read checkpoint parquet directly to verify stats_parsed contains only clustering columns. - // ScanBuilder::include_all_stats_columns() doesn't support stats_parsed when - // dataSkippingNumIndexedCols=0. Read directly from the checkpoint parquet file instead. - use delta_kernel::parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder; - let delta_log = std::path::Path::new(&setup.table_path).join("_delta_log"); - let ckpt_path = std::fs::read_dir(&delta_log)? - .filter_map(|e| e.ok()) - .find(|e| { - e.file_name() - .to_str() - .is_some_and(|n| n.contains(".checkpoint.parquet")) - }) - .expect("checkpoint parquet should exist") - .path(); - let file = std::fs::File::open(&ckpt_path)?; - let reader = ParquetRecordBatchReaderBuilder::try_new(file)?.build()?; - - let min_path = |field: &[String]| -> Vec { [&["minValues".into()], field].concat() }; - let max_path = |field: &[String]| -> Vec { [&["maxValues".into()], field].concat() }; - - let mut stats_rows: Vec<(i64, i64, String, String)> = Vec::new(); - for batch in reader { - let batch = batch?; - let batch_struct = StructArray::from(batch); - let add: &StructArray = resolve_struct_field(&batch_struct, &["add".into()]); - let stats_parsed: &StructArray = resolve_struct_field(add, &["stats_parsed".into()]); - - // Non-clustering column should not appear in stats_parsed - let min_values: &StructArray = resolve_struct_field(stats_parsed, &["minValues".into()]); - assert!( - min_values - .column_by_name(&non_clustering_physical[0]) - .is_none(), - "non-clustering column '{}' should not have stats_parsed", - non_clustering_physical[0] - ); - - let min_row_num: &Int64Array = - resolve_struct_field(stats_parsed, &min_path(&physical_paths[0])); - let max_row_num: &Int64Array = - resolve_struct_field(stats_parsed, &max_path(&physical_paths[0])); - let min_st: &StringArray = - resolve_struct_field(stats_parsed, &min_path(&physical_paths[1])); - let max_st: &StringArray = - resolve_struct_field(stats_parsed, &max_path(&physical_paths[1])); - - for i in 0..stats_parsed.len() { - if !stats_parsed.is_null(i) { - stats_rows.push(( - min_row_num.value(i), - max_row_num.value(i), - min_st.value(i).to_string(), - max_st.value(i).to_string(), - )); - } - } - } - - stats_rows.sort_by_key(|r| r.0); - assert_eq!(stats_rows.len(), 2, "should have stats_parsed for 2 files"); - assert_eq!(stats_rows[0], (1, 3, "st1".to_string(), "st3".to_string())); - assert_eq!(stats_rows[1], (4, 6, "st4".to_string(), "st6".to_string())); - - Ok(()) -} diff --git a/kernel/tests/write_row_tracking.rs b/kernel/tests/write_row_tracking.rs deleted file mode 100644 index 5a4a0015ae..0000000000 --- a/kernel/tests/write_row_tracking.rs +++ /dev/null @@ -1,193 +0,0 @@ -use std::collections::HashMap; -use std::sync::Arc; - -use delta_kernel::committer::FileSystemCommitter; -use delta_kernel::Snapshot; -use url::Url; - -use delta_kernel::arrow::array::Int32Array; -use delta_kernel::arrow::record_batch::RecordBatch; - -use delta_kernel::engine::arrow_conversion::TryIntoArrow as _; -use delta_kernel::engine::arrow_data::ArrowEngineData; -use delta_kernel::engine_data::FilteredEngineData; -use delta_kernel::object_store::path::Path; -use delta_kernel::object_store::ObjectStore; -use delta_kernel::transaction::CommitResult; - -use itertools::Itertools; -use serde_json::Deserializer; -use tempfile::tempdir; - -use delta_kernel::schema::{DataType, StructField, StructType}; - -use test_utils::{create_table, engine_store_setup}; - -/// Test that verifies baseRowId and defaultRowCommitVersion are correctly populated -/// when row tracking is enabled on the table when a remove action is generated for a -/// a file that had row tracking enabled. -/// -/// This test creates a table with row tracking enabled, writes data to it, and then -/// removes the data. It then verifies the remove action row ID fields. Propogating the -/// values is required by the delta protocol [1]. -/// -/// This complements the existing test `test_remove_files_adds_expected_entries` which -/// verifies that baseRowId and defaultRowCommitVersion are absent when row tracking is NOT enabled. -/// -/// [1]: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#writer-requirements-for-row-tracking -#[tokio::test] -async fn test_row_tracking_fields_in_add_and_remove_actions( -) -> Result<(), Box> { - let _ = tracing_subscriber::fmt::try_init(); - - let schema = Arc::new(StructType::try_new(vec![StructField::nullable( - "number", - DataType::INTEGER, - )])?); - - let tmp_dir = tempdir()?; - let tmp_test_dir_url = Url::from_directory_path(tmp_dir.path()).unwrap(); - - let (store, engine, table_location) = - engine_store_setup("test_row_tracking", Some(&tmp_test_dir_url)); - - let table_url = create_table( - store.clone(), - table_location, - schema.clone(), - &[], - true, - vec![], - vec!["rowTracking", "domainMetadata"], - ) - .await?; - - // ===== FIRST COMMIT: Add files with row tracking ===== - let snapshot = Snapshot::builder_for(table_url.clone()).build(&engine)?; - let mut txn = snapshot - .transaction(Box::new(FileSystemCommitter::new()), &engine)? - .with_engine_info("row tracking test") - .with_data_change(true); - - let data = RecordBatch::try_new( - Arc::new(schema.as_ref().try_into_arrow()?), - vec![Arc::new(Int32Array::from(vec![1, 2, 3, 4, 5]))], - )?; - - let engine_arc = Arc::new(engine); - let write_context = Arc::new(txn.get_write_context()); - let add_files_metadata = engine_arc - .write_parquet( - &ArrowEngineData::new(data), - write_context.as_ref(), - HashMap::new(), - ) - .await?; - - txn.add_files(add_files_metadata); - - let result = txn.commit(engine_arc.as_ref())?; - match result { - CommitResult::CommittedTransaction(committed) => { - assert_eq!(committed.commit_version(), 1); - } - _ => panic!("First commit should be committed"), - } - - // ===== VERIFY: Check add action contains row tracking fields ===== - let commit1_url = tmp_test_dir_url - .join("test_row_tracking/_delta_log/00000000000000000001.json") - .unwrap(); - let commit1 = store - .get(&Path::from_url_path(commit1_url.path()).unwrap()) - .await?; - - let parsed_commits: Vec<_> = Deserializer::from_slice(&commit1.bytes().await?) - .into_iter::() - .try_collect()?; - - // Find the add action - let add_actions: Vec<_> = parsed_commits - .iter() - .filter(|action| action.get("add").is_some()) - .collect(); - - assert_eq!(add_actions.len(), 1, "Expected exactly one add action"); - - let add = &add_actions[0]["add"]; - - // Verify baseRowId is present and has expected value - assert!( - add.get("baseRowId").is_some(), - "baseRowId MUST be present when row tracking is enabled" - ); - let base_row_id = add["baseRowId"] - .as_i64() - .expect("baseRowId should be an i64"); - // For the first file in a table with row tracking, baseRowId should start at 0 - // (high water mark starts at -1, so first baseRowId is -1 + 1 = 0) - assert_eq!(base_row_id, 0, "First file should have baseRowId 0"); - - let default_row_commit_version = add["defaultRowCommitVersion"] - .as_i64() - .expect("Missing defaultRowCommitVersion"); - assert_eq!(default_row_commit_version, 1); - - // ===== SECOND COMMIT: Remove the file ===== - let snapshot2 = Snapshot::builder_for(table_url.clone()).build(engine_arc.as_ref())?; - let mut txn2 = snapshot2 - .clone() - .transaction(Box::new(FileSystemCommitter::new()), engine_arc.as_ref())? - .with_engine_info("row tracking remove test") - .with_data_change(true); - - let scan = snapshot2.scan_builder().build()?; - let scan_metadata = scan.scan_metadata(engine_arc.as_ref())?.next().unwrap()?; - - let (data, selection_vector) = scan_metadata.scan_files.into_parts(); - let remove_metadata = FilteredEngineData::try_new(data, selection_vector)?; - - txn2.remove_files(remove_metadata); - - let result2 = txn2.commit(engine_arc.as_ref())?; - match result2 { - CommitResult::CommittedTransaction(committed) => { - assert_eq!(committed.commit_version(), 2); - } - _ => panic!("Second commit should be committed"), - } - - // ===== VERIFY: Check remove action contains row tracking fields ===== - let commit2_url = tmp_test_dir_url - .join("test_row_tracking/_delta_log/00000000000000000002.json") - .unwrap(); - let commit2 = store - .get(&Path::from_url_path(commit2_url.path()).unwrap()) - .await?; - - let parsed_commits2: Vec<_> = Deserializer::from_slice(&commit2.bytes().await?) - .into_iter::() - .try_collect()?; - - let remove_actions: Vec<_> = parsed_commits2 - .iter() - .filter(|action| action.get("remove").is_some()) - .collect(); - - assert_eq!(remove_actions.len(), 1); - - let remove = &remove_actions[0]["remove"]; - - let remove_base_row_id = remove["baseRowId"].as_i64().expect("Missing baseRowId"); - assert_eq!(remove_base_row_id, base_row_id); - - let remove_default_row_commit_version = remove["defaultRowCommitVersion"] - .as_i64() - .expect("Missing defaultRowCommitVersion"); - assert_eq!( - remove_default_row_commit_version, - default_row_commit_version - ); - - Ok(()) -} diff --git a/mem-test/Cargo.toml b/mem-test/Cargo.toml index 777589614f..b0cb352464 100644 --- a/mem-test/Cargo.toml +++ b/mem-test/Cargo.toml @@ -13,8 +13,15 @@ version.workspace = true [package.metadata.release] release = false +[features] +default = ["arrow"] +arrow = ["arrow-59"] +arrow-59 = ["delta_kernel/arrow-59"] +arrow-58 = ["delta_kernel/arrow-58"] + [dependencies] -delta_kernel = { path = "../kernel", features = ["arrow", "default-engine-rustls"] } +delta_kernel = { path = "../kernel", features = ["default-engine-base"] } +delta_kernel_default_engine = { path = "../default-engine", features = ["rustls"] } dhat = "0.3" rayon = "1.10" serde_json = "1" diff --git a/mem-test/tests/dhat_large_table_data.rs b/mem-test/tests/dhat_large_table_data.rs index e748c85a6e..d7567ec652 100644 --- a/mem-test/tests/dhat_large_table_data.rs +++ b/mem-test/tests/dhat_large_table_data.rs @@ -10,12 +10,11 @@ use std::sync::Arc; use delta_kernel::arrow::array::{ArrayRef, Int64Array, StringArray}; use delta_kernel::arrow::record_batch::RecordBatch; use delta_kernel::engine::arrow_data::EngineDataArrowExt as _; -use delta_kernel::engine::default::DefaultEngineBuilder; use delta_kernel::object_store::local::LocalFileSystem; use delta_kernel::parquet::arrow::ArrowWriter; use delta_kernel::parquet::file::properties::WriterProperties; use delta_kernel::Snapshot; - +use delta_kernel_default_engine::DefaultEngineBuilder; use serde_json::json; use tempfile::tempdir; use url::Url; diff --git a/mem-test/tests/dhat_large_table_log.rs b/mem-test/tests/dhat_large_table_log.rs index b6a515e401..75a740ccb0 100644 --- a/mem-test/tests/dhat_large_table_log.rs +++ b/mem-test/tests/dhat_large_table_log.rs @@ -7,9 +7,9 @@ use std::io::Write; use std::path::Path; use std::sync::Arc; -use delta_kernel::engine::default::DefaultEngineBuilder; use delta_kernel::object_store::local::LocalFileSystem; use delta_kernel::Snapshot; +use delta_kernel_default_engine::DefaultEngineBuilder; use serde_json::json; use tempfile::tempdir; use tracing::info; diff --git a/release.sh b/release.sh index b16605c9d1..0cfe8d5d03 100755 --- a/release.sh +++ b/release.sh @@ -7,8 +7,8 @@ ################################################################################################### # This is a script to automate a large portion of the release process for the crates we publish to -# crates.io. Currently only `delta_kernel` (in the kernel/ dir) and `delta_kernel_derive` (in the -# derive-macros/ dir) are released. +# crates.io. Currently `delta_kernel` (in the kernel/ dir), `delta_kernel_derive` (in the +# derive-macros/ dir), and `delta_kernel_default_engine` (in the default-engine/ dir) are released. # Exit on error, undefined variables, and pipe failures set -euo pipefail @@ -122,8 +122,10 @@ handle_release_branch() { # Handle main branch workflow (publish and tag) handle_main_branch() { # could potentially just use full 'cargo release' command here + # publish order matters: each crate depends on the previous at the same workspace version publish "delta_kernel_derive" publish "delta_kernel" + publish "delta_kernel_default_engine" # hack: just redo getting the version local version diff --git a/rustfmt.toml b/rustfmt.toml new file mode 100644 index 0000000000..ad544eeb36 --- /dev/null +++ b/rustfmt.toml @@ -0,0 +1,4 @@ +imports_granularity = "Module" +group_imports = "StdExternalCrate" +wrap_comments = true +comment_width = 100 diff --git a/test-utils/Cargo.toml b/test-utils/Cargo.toml index 1b1e3b8899..0cab159c10 100644 --- a/test-utils/Cargo.toml +++ b/test-utils/Cargo.toml @@ -12,8 +12,20 @@ version.workspace = true [package.metadata.release] release = false +[features] +default = ["arrow"] +arrow = ["arrow-59"] +arrow-59 = ["delta_kernel/arrow-59", "delta_kernel_default_engine/arrow-59"] +arrow-58 = ["delta_kernel/arrow-58", "delta_kernel_default_engine/arrow-58"] + [dependencies] -delta_kernel = { path = "../kernel", features = [ "default-engine-rustls", "arrow" ] } +# NOTE: We MUST build with SOME arrow version (or compilation would fail). So we pull in `arrow-58`, +# which ensures `cargo build` succeeds. But we ALSO default to `arrow`, which activates `arrow-59` +# and overrides `arrow-58`. +delta_kernel = { path = "../kernel", features = [ "arrow-58", "internal-api", "prettyprint", "test-utils" ] } +# Re-exported so kernel's integration tests (which cannot dev-dep default-engine directly due to +# the cycle through this crate) can reach it via `test_utils::delta_kernel_default_engine::*`. +delta_kernel_default_engine = { path = "../default-engine", default-features = false, features = ["arrow-58", "rustls"] } itertools = "0.14.0" serde_json = "1.0.142" tar = "0.4" @@ -28,3 +40,15 @@ tracing-subscriber = { version = "0.3", default-features = false, features = [ "env-filter", "fmt", ] } + +[dev-dependencies] +rstest = "0.23" +rstest_reuse = "0.7" + +# rstest/rstest_reuse are also needed as regular deps so that rstest_reuse templates +# defined in table_builder.rs (non-test code) can be exported to other crates' tests. +[dependencies.rstest] +version = "0.23" + +[dependencies.rstest_reuse] +version = "0.7" diff --git a/test-utils/src/column_mapping_fixtures.rs b/test-utils/src/column_mapping_fixtures.rs new file mode 100644 index 0000000000..e5439f7325 --- /dev/null +++ b/test-utils/src/column_mapping_fixtures.rs @@ -0,0 +1,54 @@ +//! Schema fixtures for column mappings. +use delta_kernel::schema::{ + ArrayType, ColumnMetadataKey, DataType, MapType, MetadataValue, StructField, StructType, +}; + +pub fn same_leaf_phy_name_under_different_parents() -> StructType { + let parent1_children = StructType::new_unchecked([cm_field("a", 2, "x", DataType::INTEGER)]); + let parent2_children = StructType::new_unchecked([cm_field("a", 4, "x", DataType::INTEGER)]); + StructType::new_unchecked([ + cm_field("outer1", 1, "outer1", parent1_children), + cm_field("outer2", 3, "outer2", parent2_children), + ]) +} + +pub fn nested_field_with_same_phy_path() -> StructType { + let innermost = StructType::new_unchecked([ + cm_field("a", 3, "x", DataType::INTEGER), + cm_field("b", 4, "x", DataType::INTEGER), + ]); + let arr_of_struct = ArrayType::new(innermost, true); + let map_to_arr = MapType::new(DataType::STRING, arr_of_struct, true); + let inner_struct = StructType::new_unchecked([cm_field("inner", 2, "inner", map_to_arr)]); + StructType::new_unchecked([cm_field("outer", 1, "outer", inner_struct)]) +} + +/// `StructField` carrying both `delta.columnMapping.id` and `delta.columnMapping.physicalName`. +pub fn cm_field(name: &str, id: i64, phys: &str, ty: impl Into) -> StructField { + StructField::new(name, ty, true).with_metadata([ + ( + ColumnMetadataKey::ColumnMappingId.as_ref(), + MetadataValue::Number(id), + ), + ( + ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), + MetadataValue::String(phys.to_string()), + ), + ]) +} + +/// `StructField` carrying only `delta.columnMapping.id` (physical name left to be filled). +pub fn cm_field_id_only(name: &str, id: i64, ty: impl Into) -> StructField { + StructField::new(name, ty, true).with_metadata([( + ColumnMetadataKey::ColumnMappingId.as_ref(), + MetadataValue::Number(id), + )]) +} + +/// `StructField` carrying only `delta.columnMapping.physicalName` (id left to be allocated). +pub fn cm_field_physical_name_only(name: &str, phys: &str, ty: impl Into) -> StructField { + StructField::new(name, ty, true).with_metadata([( + ColumnMetadataKey::ColumnMappingPhysicalName.as_ref(), + MetadataValue::String(phys.to_string()), + )]) +} diff --git a/test-utils/src/counting_reporter.rs b/test-utils/src/counting_reporter.rs new file mode 100644 index 0000000000..d5cdf4677f --- /dev/null +++ b/test-utils/src/counting_reporter.rs @@ -0,0 +1,722 @@ +//! A [`MetricsReporter`] implementation that accumulates operation counts via atomic counters. +//! +//! Useful in tests to assert exact IO costs and in benchmarks to print per-call IO profiles. +//! Attach it to a `DefaultEngine` via `DefaultEngineBuilder::with_metrics_reporter`, then +//! inspect the counters or call [`CountingReporter::print_summary`]. + +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::{Arc, Mutex, OnceLock}; + +use delta_kernel::metrics::{ + CommitFailureReason, MetricEvent, MetricsReporter, WithMetricsReporterLayer as _, +}; +use tracing::subscriber::DefaultGuard; +use tracing_subscriber::util::SubscriberInitExt as _; + +/// Install `reporter` as a thread-local metrics-collecting subscriber, with the safety net +/// required to avoid tracing callsite-cache poisoning. Returns a guard that uninstalls the +/// subscriber when dropped. +/// +/// This is the recommended way to wire a [`MetricsReporter`] into a test. Use it instead +/// of hand-rolling [`ensure_metrics_compatible_global_subscriber`] + +/// [`tracing_subscriber::registry()`] + [`set_default`] -- forgetting the first call +/// silently produces flaky tests where counters intermittently read zero (see the +/// [`ensure_metrics_compatible_global_subscriber`] doc for the underlying mechanism). +/// +/// # Example +/// +/// ```ignore +/// use std::sync::Arc; +/// use test_utils::{install_thread_local_metrics_reporter, CountingReporter}; +/// +/// let reporter = Arc::new(CountingReporter::new()); +/// let _guard = install_thread_local_metrics_reporter(reporter.clone()); +/// // ... run kernel operations; `reporter`'s counters now reflect their I/O ... +/// ``` +/// +/// [`set_default`]: tracing_subscriber::util::SubscriberInitExt::set_default +pub fn install_thread_local_metrics_reporter(reporter: Arc) -> DefaultGuard { + ensure_metrics_compatible_global_subscriber(); + tracing_subscriber::registry() + .with_metrics_reporter_layer(reporter) + .set_default() +} + +/// Ensure a process-global tracing subscriber is installed exactly once for the lifetime +/// of the test binary. +/// +/// Most tests should call [`install_thread_local_metrics_reporter`] instead -- it bundles +/// this call with the thread-local subscriber install. Reach for this directly only when +/// you need to layer additional `tracing_subscriber` layers (e.g. `fmt`) alongside the +/// metrics layer. +/// +/// # Why +/// +/// `tracing` caches per-callsite `Interest` *process-globally* on first registration. The +/// kernel emits metrics from `Drop` impls that fire `tracing::span!` on whichever thread +/// happens to drop the iterator -- often a tokio worker thread owned by the +/// `DefaultEngine`'s background runtime, which has no thread-local subscriber. When such +/// a thread is the first to hit a metric callsite, it consults `NoSubscriber` (the +/// fallback), gets `Interest::never`, and that verdict is cached process-globally. Every +/// later emission of that callsite -- including emissions from threads that *do* have the +/// test's subscriber installed -- is then a no-op. Counters that depend on those metrics +/// sit at zero and assertions fail. +/// +/// `set_default` (thread-local) does *not* invalidate the callsite cache; only +/// `set_global_default` does. This helper installs a bare `Registry` as the global +/// default once per process. It does nothing on its own (no metrics layer attached to +/// it), but it ensures every thread sees a real subscriber whenever it first hits a +/// callsite, which keeps the cached interest in the `always`/`sometimes` regime. +/// Per-test `set_default(...)` then routes events to the test's `CountingReporter` as +/// usual; metrics emissions on threads without a thread-local override (tokio workers, +/// libtest scaffolding) silently fall through to the bare `Registry`, which is correct +/// because those threads have no test counter to update. +pub fn ensure_metrics_compatible_global_subscriber() { + static INIT: OnceLock<()> = OnceLock::new(); + INIT.get_or_init(|| { + // Install a bare Registry. `try_init` calls `set_global_default`, which can only + // succeed once per process and triggers an interest-cache rebuild internally on + // success. If a global default is already installed (e.g. by a test runner via + // `RUST_LOG`), respect it and rebuild the cache ourselves so any callsites that + // were registered before this call are re-evaluated against the current dispatcher. + if tracing_subscriber::registry().try_init().is_err() { + tracing::callsite::rebuild_interest_cache(); + } + }); +} + +/// An atomic `u64` counter using [`Ordering::Relaxed`] throughout. +/// +/// Relaxed ordering is sufficient here: metrics are reported after the operations they +/// describe have completed, so there are no inter-counter ordering dependencies. +#[derive(Debug, Default)] +pub struct RelaxedCounter(AtomicU64); + +impl RelaxedCounter { + /// Increment the counter by one. + pub fn inc(&self) { + self.0.fetch_add(1, Ordering::Relaxed); + } + + /// Add `n` to the counter. + pub fn add(&self, n: u64) { + self.0.fetch_add(n, Ordering::Relaxed); + } + + /// Read the current value. + pub fn get(&self) -> u64 { + self.0.load(Ordering::Relaxed) + } + + /// Reset the counter to zero. + pub fn reset(&self) { + self.0.store(0, Ordering::Relaxed); + } +} + +/// Accumulates storage and operation metrics via the [`MetricsReporter`] interface. +/// +/// # Note: update [`reset`] and the `MetricsReporter` impl when adding fields. +/// +/// [`reset`]: Self::reset +#[derive(Debug, Default)] +pub struct CountingReporter { + // Storage-layer IO counters (StorageHandler::list_from / read_files / copy_atomic) + /// Number of `list_from` calls (one per [`MetricEvent::StorageListCompleted`]). + pub list_calls: RelaxedCounter, + /// Total files returned across all list calls. + pub list_files_seen: RelaxedCounter, + /// Number of `StorageHandler::read_files` calls (one per + /// [`MetricEvent::StorageReadCompleted`]). + pub storage_read_calls: RelaxedCounter, + /// Total individual files read via `StorageHandler::read_files`. + pub storage_read_files: RelaxedCounter, + /// Total bytes consumed via `StorageHandler::read_files`. + pub storage_bytes_read: RelaxedCounter, + /// Number of `copy_atomic` calls (one per [`MetricEvent::StorageCopyCompleted`]). + pub copy_calls: RelaxedCounter, + + // JSON handler IO counters (DefaultJsonHandler::read_json_files) + /// Number of `read_json_files` calls (one per [`MetricEvent::JsonReadCompleted`]). + pub json_read_calls: RelaxedCounter, + /// Total JSON files requested across all `read_json_files` calls. + pub json_files_read: RelaxedCounter, + /// Total on-disk bytes of JSON files requested. + pub json_bytes_read: RelaxedCounter, + + // Parquet handler IO counters (DefaultParquetHandler::read_parquet_files) + /// Number of `read_parquet_files` calls (one per [`MetricEvent::ParquetReadCompleted`]). + pub parquet_read_calls: RelaxedCounter, + /// Total Parquet files requested across all `read_parquet_files` calls. + pub parquet_files_read: RelaxedCounter, + /// Total on-disk bytes of Parquet files requested. + pub parquet_bytes_read: RelaxedCounter, + + // Operation-level counters + /// Number of completed snapshot constructions. + pub snapshot_completions: RelaxedCounter, + /// Number of full (non-incremental) log segment loads. Each fresh snapshot construction + /// from a table root contributes one load; incremental snapshot updates do not. + pub log_segment_loads: RelaxedCounter, + /// Total commit (JSON delta) files in the commit tail across all log segment loads. + /// These are the commits between the last checkpoint and the snapshot version — not + /// all historical commits in the table. Commits older than the selected checkpoint + /// are not included. + pub commit_files: RelaxedCounter, + /// Total checkpoint part files read across all log segment loads. For a single-part + /// checkpoint this is 1; for a multi-part checkpoint it equals the number of parts + /// that make up the selected checkpoint. + pub checkpoint_files: RelaxedCounter, + /// Total log compaction files in the commit tail across all log segment loads. + pub compaction_files: RelaxedCounter, + /// Total number of times a latest CRC file was found in the log segment, across all + /// log segment loads. + pub latest_crc_files_found: RelaxedCounter, + + // CRC reader IO counters + /// Number of CRC read calls (one per [`MetricEvent::CrcReadSuccess`]). + pub crc_read_calls: RelaxedCounter, + /// Total number of bytes read from CRC files, across all CRC read calls. + pub crc_bytes_read: RelaxedCounter, + + // Domain metadata load counters (Snapshot::get_domain_metadatas_internal) + pub domain_metadata_loads: RelaxedCounter, + pub domain_metadata_load_failures: RelaxedCounter, + pub domain_metadata_cache_hits: RelaxedCounter, + pub domain_metadata_domains_returned: RelaxedCounter, + + // SetTransaction load counters (Snapshot::get_app_id_version) + pub set_transaction_loads: RelaxedCounter, + pub set_transaction_load_failures: RelaxedCounter, + pub set_transaction_cache_hits: RelaxedCounter, + pub set_transaction_found: RelaxedCounter, + + // Transaction commit counters (Transaction::commit) + pub transaction_commits: RelaxedCounter, + pub commit_conflicts: RelaxedCounter, + pub commit_retryable_failures: RelaxedCounter, + pub commit_errors: RelaxedCounter, + pub commit_add_files: RelaxedCounter, + pub commit_remove_files: RelaxedCounter, + pub commit_add_bytes: RelaxedCounter, + pub commit_remove_bytes: RelaxedCounter, +} + +impl CountingReporter { + /// Create a new reporter with all counters at zero. + pub fn new() -> Self { + Self::default() + } + + /// Reset all counters to zero. + /// + /// Useful before a single profiling iteration to get per-call counts. + pub fn reset(&self) { + self.list_calls.reset(); + self.list_files_seen.reset(); + self.storage_read_calls.reset(); + self.storage_read_files.reset(); + self.storage_bytes_read.reset(); + self.copy_calls.reset(); + self.json_read_calls.reset(); + self.json_files_read.reset(); + self.json_bytes_read.reset(); + self.parquet_read_calls.reset(); + self.parquet_files_read.reset(); + self.parquet_bytes_read.reset(); + self.snapshot_completions.reset(); + self.log_segment_loads.reset(); + self.commit_files.reset(); + self.checkpoint_files.reset(); + self.compaction_files.reset(); + self.latest_crc_files_found.reset(); + self.crc_read_calls.reset(); + self.crc_bytes_read.reset(); + self.domain_metadata_loads.reset(); + self.domain_metadata_load_failures.reset(); + self.domain_metadata_cache_hits.reset(); + self.domain_metadata_domains_returned.reset(); + self.set_transaction_loads.reset(); + self.set_transaction_load_failures.reset(); + self.set_transaction_cache_hits.reset(); + self.set_transaction_found.reset(); + self.transaction_commits.reset(); + self.commit_conflicts.reset(); + self.commit_retryable_failures.reset(); + self.commit_errors.reset(); + self.commit_add_files.reset(); + self.commit_remove_files.reset(); + self.commit_add_bytes.reset(); + self.commit_remove_bytes.reset(); + } + + /// Print a human-readable IO and operation summary. + /// + /// Intended to be called after [`reset`][Self::reset] and one operation so values + /// reflect a single call's cost. Output is visible with `cargo test -- --nocapture` + /// or `cargo nextest run -- --no-capture`. + pub fn print_summary(&self, label: &str) { + let list_calls = self.list_calls.get(); + let list_files = self.list_files_seen.get(); + let storage_reads = self.storage_read_calls.get(); + let storage_files = self.storage_read_files.get(); + let storage_kib = self.storage_bytes_read.get() / 1024; + let copy_calls = self.copy_calls.get(); + let json_calls = self.json_read_calls.get(); + let json_files = self.json_files_read.get(); + let json_kib = self.json_bytes_read.get() / 1024; + let parquet_calls = self.parquet_read_calls.get(); + let parquet_files = self.parquet_files_read.get(); + let parquet_kib = self.parquet_bytes_read.get() / 1024; + let crc_calls = self.crc_read_calls.get(); + let crc_kib = self.crc_bytes_read.get() / 1024; + let log_loads = self.log_segment_loads.get(); + let commits = self.commit_files.get(); + let checkpoints = self.checkpoint_files.get(); + let compactions = self.compaction_files.get(); + let crc_files_found = self.latest_crc_files_found.get(); + + println!(" [io] {label}"); + println!(" storage : {list_calls} list ({list_files} files seen) {storage_reads} raw read ({storage_files} files, {storage_kib} KiB) {copy_calls} copy"); + println!(" json : {json_calls} call(s) {json_files} files {json_kib} KiB"); + println!(" parquet : {parquet_calls} call(s) {parquet_files} files {parquet_kib} KiB"); + println!(" crc : {crc_calls} call(s) {crc_kib} KiB"); + println!(" log : {log_loads} segment load(s) -- {commits} commits {checkpoints} checkpoints {compactions} compactions {crc_files_found} latest_crc_files"); + } +} + +impl MetricsReporter for CountingReporter { + fn report(&self, event: MetricEvent) { + match event { + MetricEvent::StorageListCompleted(e) => { + self.list_calls.inc(); + self.list_files_seen.add(e.num_files); + } + MetricEvent::StorageReadCompleted(e) => { + self.storage_read_calls.inc(); + self.storage_read_files.add(e.num_files); + self.storage_bytes_read.add(e.bytes_read); + } + MetricEvent::StorageCopyCompleted(_) => { + self.copy_calls.inc(); + } + MetricEvent::JsonReadCompleted(e) => { + self.json_read_calls.inc(); + self.json_files_read.add(e.num_files); + self.json_bytes_read.add(e.bytes_read); + } + MetricEvent::ParquetReadCompleted(e) => { + self.parquet_read_calls.inc(); + self.parquet_files_read.add(e.num_files); + self.parquet_bytes_read.add(e.bytes_read); + } + MetricEvent::SnapshotBuildSuccess(_) => { + self.snapshot_completions.inc(); + } + MetricEvent::LogSegmentLoadSuccess(e) => { + self.log_segment_loads.inc(); + self.commit_files.add(e.num_commit_files); + self.checkpoint_files.add(e.num_checkpoint_files); + self.compaction_files.add(e.num_compaction_files); + self.latest_crc_files_found + .add(e.crc_versions_behind.is_some() as u64); + } + MetricEvent::CrcReadSuccess(e) => { + self.crc_read_calls.inc(); + self.crc_bytes_read.add(e.bytes_read); + } + MetricEvent::DomainMetadataLoadSuccess(e) => { + self.domain_metadata_loads.inc(); + if e.from_cache { + self.domain_metadata_cache_hits.inc(); + } + self.domain_metadata_domains_returned + .add(e.num_domains_returned); + } + MetricEvent::DomainMetadataLoadFailure => { + self.domain_metadata_load_failures.inc(); + } + MetricEvent::SetTransactionLoadSuccess(e) => { + self.set_transaction_loads.inc(); + if e.from_cache { + self.set_transaction_cache_hits.inc(); + } + if e.found { + self.set_transaction_found.inc(); + } + } + MetricEvent::SetTransactionLoadFailure => { + self.set_transaction_load_failures.inc(); + } + MetricEvent::TransactionCommitSuccess(e) => { + self.transaction_commits.inc(); + self.commit_add_files.add(e.num_add_files); + self.commit_remove_files.add(e.num_remove_files); + self.commit_add_bytes.add(e.add_files_bytes); + self.commit_remove_bytes.add(e.remove_files_bytes); + } + MetricEvent::TransactionCommitFailure(e) => match e.reason { + CommitFailureReason::Conflict => self.commit_conflicts.inc(), + CommitFailureReason::RetryableIo => self.commit_retryable_failures.inc(), + CommitFailureReason::Error => self.commit_errors.inc(), + }, + // Intentionally not tracked. Add counters if needed. + MetricEvent::ProtocolMetadataLoadSuccess(_) + | MetricEvent::ProtocolMetadataLoadFailure(_) + | MetricEvent::LogSegmentLoadFailure(_) + | MetricEvent::SnapshotBuildFailure(_) + | MetricEvent::CrcReadFailure + | MetricEvent::ScanMetadataCompleted(_) => {} + } + } +} + +/// A [`MetricsReporter`] that retains every [`MetricEvent`] for inspection, for tests that need +/// to assert on event payloads (not just counts, which [`CountingReporter`] handles). +#[derive(Debug, Default)] +pub struct CapturingReporter { + events: Mutex>, +} + +impl MetricsReporter for CapturingReporter { + fn report(&self, event: MetricEvent) { + self.events.lock().unwrap().push(event); + } +} + +impl CapturingReporter { + /// All events captured so far, in report order. + pub fn events(&self) -> Vec { + self.events.lock().unwrap().clone() + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + use std::time::Duration; + + use delta_kernel::metrics::{ + CrcReadSuccess, DomainMetadataLoadSuccess, LogSegmentLoadSuccess, LogSegmentLoadType, + MetricId, ProtocolMetadataLoadSuccess, ProtocolMetadataSource, SetTransactionLoadSuccess, + SnapshotBuildFailure, SnapshotBuildSuccess, StorageCopyCompleted, StorageListCompleted, + StorageReadCompleted, TableType, TransactionCommitFailure, TransactionCommitSuccess, + }; + + use super::*; + + fn dur() -> Duration { + Duration::from_millis(1) + } + + #[test] + fn report_storage_list_completed_increments_list_counters() { + let reporter = CountingReporter::new(); + reporter.report(MetricEvent::StorageListCompleted(StorageListCompleted { + duration: dur(), + num_files: 10, + })); + reporter.report(MetricEvent::StorageListCompleted(StorageListCompleted { + duration: dur(), + num_files: 5, + })); + assert_eq!(reporter.list_calls.get(), 2); + assert_eq!(reporter.list_files_seen.get(), 15); + } + + #[test] + fn report_storage_read_completed_increments_read_counters() { + let reporter = CountingReporter::new(); + reporter.report(MetricEvent::StorageReadCompleted(StorageReadCompleted { + duration: dur(), + num_files: 3, + bytes_read: 1024, + })); + assert_eq!(reporter.storage_read_calls.get(), 1); + assert_eq!(reporter.storage_read_files.get(), 3); + assert_eq!(reporter.storage_bytes_read.get(), 1024); + } + + #[test] + fn report_storage_copy_completed_increments_copy_counter() { + let reporter = CountingReporter::new(); + reporter.report(MetricEvent::StorageCopyCompleted(StorageCopyCompleted { + duration: dur(), + })); + assert_eq!(reporter.copy_calls.get(), 1); + } + + #[test] + fn report_snapshot_completed_increments_snapshot_counter() { + let reporter = CountingReporter::new(); + reporter.report(MetricEvent::SnapshotBuildSuccess(SnapshotBuildSuccess { + operation_id: MetricId::new(), + table_type: TableType::PathBased, + correlation_id: None, + load_type: LogSegmentLoadType::Full, + version: 0, + duration: dur(), + })); + assert_eq!(reporter.snapshot_completions.get(), 1); + } + + #[test] + fn report_log_segment_loaded_increments_log_replay_counters() { + let reporter = CountingReporter::new(); + reporter.report(MetricEvent::LogSegmentLoadSuccess(LogSegmentLoadSuccess { + operation_id: MetricId::new(), + table_type: TableType::PathBased, + correlation_id: None, + load_type: LogSegmentLoadType::Full, + duration: dur(), + num_commit_files: 7, + num_checkpoint_files: 2, + num_compaction_files: 1, + crc_versions_behind: Some(0), + })); + assert_eq!(reporter.log_segment_loads.get(), 1); + assert_eq!(reporter.commit_files.get(), 7); + assert_eq!(reporter.checkpoint_files.get(), 2); + assert_eq!(reporter.compaction_files.get(), 1); + assert_eq!(reporter.latest_crc_files_found.get(), 1); + } + + #[test] + fn report_log_segment_loaded_without_crc_does_not_increment_crc_counter() { + let reporter = CountingReporter::new(); + reporter.report(MetricEvent::LogSegmentLoadSuccess(LogSegmentLoadSuccess { + operation_id: MetricId::new(), + table_type: TableType::PathBased, + correlation_id: None, + load_type: LogSegmentLoadType::Full, + duration: dur(), + num_commit_files: 3, + num_checkpoint_files: 1, + num_compaction_files: 0, + crc_versions_behind: None, + })); + assert_eq!(reporter.log_segment_loads.get(), 1); + assert_eq!(reporter.latest_crc_files_found.get(), 0); + } + + #[test] + fn report_crc_read_completed_increments_crc_counters() { + let reporter = CountingReporter::new(); + reporter.report(MetricEvent::CrcReadSuccess(CrcReadSuccess { + duration: dur(), + bytes_read: 512, + })); + reporter.report(MetricEvent::CrcReadSuccess(CrcReadSuccess { + duration: dur(), + bytes_read: 256, + })); + assert_eq!(reporter.crc_read_calls.get(), 2); + assert_eq!(reporter.crc_bytes_read.get(), 768); + } + + #[test] + fn report_domain_metadata_loaded_increments_domain_metadata_counters() { + let reporter = CountingReporter::new(); + reporter.report(MetricEvent::DomainMetadataLoadSuccess( + DomainMetadataLoadSuccess { + from_cache: true, + num_domains_returned: 3, + duration: dur(), + }, + )); + reporter.report(MetricEvent::DomainMetadataLoadSuccess( + DomainMetadataLoadSuccess { + from_cache: false, + num_domains_returned: 1, + duration: dur(), + }, + )); + assert_eq!(reporter.domain_metadata_loads.get(), 2); + assert_eq!(reporter.domain_metadata_cache_hits.get(), 1); + assert_eq!(reporter.domain_metadata_domains_returned.get(), 4); + } + + #[test] + fn report_set_transaction_loaded_increments_set_transaction_counters() { + let reporter = CountingReporter::new(); + reporter.report(MetricEvent::SetTransactionLoadSuccess( + SetTransactionLoadSuccess { + from_cache: true, + found: true, + duration: dur(), + }, + )); + reporter.report(MetricEvent::SetTransactionLoadSuccess( + SetTransactionLoadSuccess { + from_cache: false, + found: false, + duration: dur(), + }, + )); + assert_eq!(reporter.set_transaction_loads.get(), 2); + assert_eq!(reporter.set_transaction_cache_hits.get(), 1); + assert_eq!(reporter.set_transaction_found.get(), 1); + } + + #[test] + fn report_transaction_commit_success_accumulates_files_and_bytes() { + let reporter = CountingReporter::new(); + reporter.report(MetricEvent::TransactionCommitSuccess( + TransactionCommitSuccess { + operation_id: MetricId::new(), + table_type: TableType::PathBased, + correlation_id: None, + commit_version: 1, + num_add_files: 3, + num_remove_files: 2, + num_dv_updates: 0, + add_files_bytes: 600, + remove_files_bytes: 150, + is_blind_append: false, + data_change: true, + operation: Some("WRITE".to_string()), + prepare_duration: dur(), + committer_duration: dur(), + total_duration: dur(), + }, + )); + assert_eq!(reporter.transaction_commits.get(), 1); + assert_eq!(reporter.commit_add_files.get(), 3); + assert_eq!(reporter.commit_remove_files.get(), 2); + assert_eq!(reporter.commit_add_bytes.get(), 600); + assert_eq!(reporter.commit_remove_bytes.get(), 150); + } + + #[test] + fn report_transaction_commit_failure_increments_matching_reason_counter() { + let reporter = CountingReporter::new(); + for reason in [ + CommitFailureReason::Conflict, + CommitFailureReason::RetryableIo, + CommitFailureReason::Error, + ] { + reporter.report(MetricEvent::TransactionCommitFailure( + TransactionCommitFailure { + operation_id: MetricId::new(), + table_type: TableType::PathBased, + correlation_id: None, + reason, + }, + )); + } + assert_eq!(reporter.commit_conflicts.get(), 1); + assert_eq!(reporter.commit_retryable_failures.get(), 1); + assert_eq!(reporter.commit_errors.get(), 1); + assert_eq!(reporter.transaction_commits.get(), 0); + } + + #[test] + fn report_untracked_events_does_not_panic() { + let reporter = CountingReporter::new(); + reporter.report(MetricEvent::ProtocolMetadataLoadSuccess( + ProtocolMetadataLoadSuccess { + operation_id: MetricId::new(), + table_type: TableType::PathBased, + correlation_id: None, + load_type: LogSegmentLoadType::Full, + source: ProtocolMetadataSource::FullReplay, + duration: dur(), + }, + )); + reporter.report(MetricEvent::SnapshotBuildFailure(SnapshotBuildFailure { + operation_id: MetricId::new(), + table_type: TableType::PathBased, + correlation_id: None, + load_type: LogSegmentLoadType::Full, + })); + assert_eq!(reporter.snapshot_completions.get(), 0); + } + + #[test] + fn reset_zeros_all_counters() { + let reporter = Arc::new(CountingReporter::new()); + reporter.report(MetricEvent::StorageListCompleted(StorageListCompleted { + duration: dur(), + num_files: 10, + })); + reporter.report(MetricEvent::StorageReadCompleted(StorageReadCompleted { + duration: dur(), + num_files: 3, + bytes_read: 1024, + })); + reporter.report(MetricEvent::StorageCopyCompleted(StorageCopyCompleted { + duration: dur(), + })); + reporter.report(MetricEvent::LogSegmentLoadSuccess(LogSegmentLoadSuccess { + operation_id: MetricId::new(), + table_type: TableType::PathBased, + correlation_id: None, + load_type: LogSegmentLoadType::Full, + duration: dur(), + num_commit_files: 7, + num_checkpoint_files: 2, + num_compaction_files: 1, + crc_versions_behind: Some(0), + })); + reporter.report(MetricEvent::CrcReadSuccess(CrcReadSuccess { + duration: dur(), + bytes_read: 512, + })); + reporter.report(MetricEvent::DomainMetadataLoadSuccess( + DomainMetadataLoadSuccess { + from_cache: true, + num_domains_returned: 2, + duration: dur(), + }, + )); + reporter.report(MetricEvent::SetTransactionLoadSuccess( + SetTransactionLoadSuccess { + from_cache: true, + found: true, + duration: dur(), + }, + )); + + reporter.reset(); + + assert_eq!(reporter.list_calls.get(), 0); + assert_eq!(reporter.list_files_seen.get(), 0); + assert_eq!(reporter.storage_read_calls.get(), 0); + assert_eq!(reporter.storage_read_files.get(), 0); + assert_eq!(reporter.storage_bytes_read.get(), 0); + assert_eq!(reporter.copy_calls.get(), 0); + assert_eq!(reporter.json_read_calls.get(), 0); + assert_eq!(reporter.json_files_read.get(), 0); + assert_eq!(reporter.json_bytes_read.get(), 0); + assert_eq!(reporter.parquet_read_calls.get(), 0); + assert_eq!(reporter.parquet_files_read.get(), 0); + assert_eq!(reporter.parquet_bytes_read.get(), 0); + assert_eq!(reporter.snapshot_completions.get(), 0); + assert_eq!(reporter.log_segment_loads.get(), 0); + assert_eq!(reporter.commit_files.get(), 0); + assert_eq!(reporter.checkpoint_files.get(), 0); + assert_eq!(reporter.compaction_files.get(), 0); + assert_eq!(reporter.latest_crc_files_found.get(), 0); + assert_eq!(reporter.crc_read_calls.get(), 0); + assert_eq!(reporter.crc_bytes_read.get(), 0); + assert_eq!(reporter.domain_metadata_loads.get(), 0); + assert_eq!(reporter.domain_metadata_load_failures.get(), 0); + assert_eq!(reporter.domain_metadata_cache_hits.get(), 0); + assert_eq!(reporter.domain_metadata_domains_returned.get(), 0); + assert_eq!(reporter.set_transaction_loads.get(), 0); + assert_eq!(reporter.set_transaction_load_failures.get(), 0); + assert_eq!(reporter.set_transaction_cache_hits.get(), 0); + assert_eq!(reporter.set_transaction_found.get(), 0); + } + + #[test] + fn report_load_failures_increment_failure_counters() { + let reporter = CountingReporter::new(); + reporter.report(MetricEvent::DomainMetadataLoadFailure); + reporter.report(MetricEvent::SetTransactionLoadFailure); + assert_eq!(reporter.domain_metadata_load_failures.get(), 1); + assert_eq!(reporter.set_transaction_load_failures.get(), 1); + assert_eq!(reporter.domain_metadata_loads.get(), 0); + assert_eq!(reporter.set_transaction_loads.get(), 0); + } +} diff --git a/kernel/src/engine/tests.rs b/test-utils/src/engine_contract.rs similarity index 55% rename from kernel/src/engine/tests.rs rename to test-utils/src/engine_contract.rs index b534608c90..aff9b31845 100644 --- a/kernel/src/engine/tests.rs +++ b/test-utils/src/engine_contract.rs @@ -1,42 +1,43 @@ //! Shared contract tests for engine handler traits ([`JsonHandler`], [`ParquetHandler`]). //! //! Each function here tests a specific piece of the handler contract using only the public -//! trait API plus [`GetData`]/[`RowVisitor`] for result inspection — no engine-specific +//! trait API plus [`GetData`]/[`RowVisitor`] for result inspection -- no engine-specific //! downcasting. Individual engine test modules call these to verify their implementation //! satisfies the contract, then add any engine-specific assertions (e.g. Arrow encoding //! details) in their own tests. use std::collections::HashMap; -use std::path::PathBuf; use std::sync::{Arc, LazyLock}; -use tempfile::{tempdir, NamedTempFile}; -use url::Url; - -use crate::arrow::array::{Array, Int64Array, RecordBatch, StringArray}; -use crate::arrow::datatypes::{DataType as ArrowDataType, Field, Schema as ArrowSchema}; -use crate::engine::arrow_data::ArrowEngineData; -use crate::engine_data::{FilteredEngineData, GetData, RowVisitor, TypedGetData as _}; -use crate::object_store::path::Path; -use crate::parquet::arrow::arrow_writer::ArrowWriter; -use crate::parquet::arrow::PARQUET_FIELD_ID_META_KEY; -use crate::schema::{ - column_name, ColumnMetadataKey, ColumnName, ColumnNamesAndTypes, DataType, MetadataColumnSpec, - StructField, StructType, +use delta_kernel::arrow::array::{Array, Int64Array, RecordBatch, StringArray}; +use delta_kernel::arrow::datatypes::{DataType as ArrowDataType, Field, Schema as ArrowSchema}; +use delta_kernel::engine::arrow_conversion::TryIntoKernel as _; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::engine_data::{FilteredEngineData, GetData, RowVisitor, TypedGetData as _}; +use delta_kernel::expressions::{column_name, ColumnName}; +use delta_kernel::object_store::path::Path; +use delta_kernel::parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder; +use delta_kernel::parquet::arrow::arrow_writer::ArrowWriter; +use delta_kernel::parquet::arrow::{ARROW_SCHEMA_META_KEY, PARQUET_FIELD_ID_META_KEY}; +use delta_kernel::schema::{ + ColumnMetadataKey, DataType, MetadataColumnSpec, PrimitiveType, SchemaRef, StructField, + StructType, }; -use crate::{DeltaResult, Engine, EngineData, FileMeta, JsonHandler, ParquetHandler}; +use delta_kernel::{DeltaResult, Engine, EngineData, FileMeta, JsonHandler, ParquetHandler}; use itertools::Itertools; +use tempfile::{tempdir, NamedTempFile}; +use url::Url; -use test_utils::delta_path_for_version; +use crate::delta_path_for_version; -// --------------------------------------------------------------------------- +// =========================================================================== // Shared file-setup helpers -// --------------------------------------------------------------------------- +// =========================================================================== /// Writes `lines` as newline-delimited JSON to a [`NamedTempFile`] and returns the file /// together with a [`FileMeta`] pointing at it. The temp file must be kept alive for as /// long as the `FileMeta` is in use. -pub(crate) fn make_temp_json_file(lines: &[&str]) -> (NamedTempFile, FileMeta) { +pub fn make_temp_json_file(lines: &[&str]) -> (NamedTempFile, FileMeta) { let mut temp_file = NamedTempFile::new().expect("Failed to create temp file"); for line in lines { use std::io::Write as _; @@ -56,7 +57,7 @@ pub(crate) fn make_temp_json_file(lines: &[&str]) -> (NamedTempFile, FileMeta) { } /// Builds a [`FileMeta`] for a local file path, reading the actual size from the filesystem. -pub(crate) fn file_meta_for(path: &std::path::Path) -> FileMeta { +pub fn file_meta_for(path: &std::path::Path) -> FileMeta { let url = Url::from_file_path(path).unwrap(); let size = std::fs::metadata(path).unwrap().len(); FileMeta { @@ -66,14 +67,14 @@ pub(crate) fn file_meta_for(path: &std::path::Path) -> FileMeta { } } -// --------------------------------------------------------------------------- +// =========================================================================== // JsonHandler contract tests -// --------------------------------------------------------------------------- +// =========================================================================== /// Contract: any [`JsonHandler`] that receives a schema with a [`MetadataColumnSpec::FilePath`] /// column must populate it with the file URL for every row, readable via [`GetData`] without /// any Arrow downcasting. -pub(crate) fn test_json_handler_file_path_contract(handler: &dyn JsonHandler) { +pub fn test_json_handler_file_path_contract(handler: &dyn JsonHandler) { let (_temp, file_meta) = make_temp_json_file(&[r#"{"x": 1}"#, r#"{"x": 2}"#]); let expected_url = file_meta.location.to_string(); @@ -97,9 +98,9 @@ pub(crate) fn test_json_handler_file_path_contract(handler: &dyn JsonHandler) { } impl RowVisitor for FilePathCollector { fn selected_column_names_and_types(&self) -> (&'static [ColumnName], &'static [DataType]) { - static NAT: LazyLock = - LazyLock::new(|| (vec![column_name!("_file")], vec![DataType::STRING]).into()); - NAT.as_ref() + static NAMES: LazyLock> = LazyLock::new(|| vec![column_name!("_file")]); + static TYPES: LazyLock> = LazyLock::new(|| vec![DataType::STRING]); + (&NAMES, &TYPES) } fn visit<'a>( &mut self, @@ -123,25 +124,96 @@ pub(crate) fn test_json_handler_file_path_contract(handler: &dyn JsonHandler) { ); } -// --------------------------------------------------------------------------- +// =========================================================================== // ParquetHandler contract tests -// --------------------------------------------------------------------------- - -/// Contract: [`ParquetHandler::read_parquet_footer`] must correctly parse the schema -/// from a real Delta checkpoint file. -pub(crate) fn test_parquet_handler_reads_footer(handler: &dyn ParquetHandler) { - let path = std::fs::canonicalize(PathBuf::from( - "./tests/data/with_checkpoint_no_last_checkpoint/_delta_log/00000000000000000002.checkpoint.parquet", - )) - .unwrap(); +// =========================================================================== + +/// Contract: [`ParquetHandler::read_parquet_footer`] must correctly parse the schema from a +/// real Delta checkpoint file. `checkpoint_path` should point at a V1 checkpoint parquet file +/// (kernel ships a fixture under `kernel/tests/data/`). +pub fn test_parquet_handler_reads_footer( + handler: &dyn ParquetHandler, + checkpoint_path: &std::path::Path, +) { + let path = std::fs::canonicalize(checkpoint_path).unwrap(); let file_meta = file_meta_for(&path); let footer = handler.read_parquet_footer(&file_meta).unwrap(); - crate::utils::test_utils::validate_checkpoint_schema(&footer.schema); + validate_checkpoint_schema(&footer.schema); +} + +/// Validates that a schema has the expected checkpoint structure with top-level action fields +/// and proper nested types for add, metaData, and protocol actions. +fn validate_checkpoint_schema(schema: &SchemaRef) { + fn get_field(struct_type: &StructType, name: &str) -> StructField { + struct_type + .fields() + .find(|f| f.name() == name) + .unwrap_or_else(|| panic!("Field '{name}' not found")) + .clone() + } + + for field_name in ["txn", "add", "remove", "metaData", "protocol"] { + let field = get_field(schema, field_name); + assert!( + matches!(field.data_type(), DataType::Struct(_)), + "Field '{field_name}' should be a struct type" + ); + } + + let add_field = get_field(schema, "add"); + let add_struct = match add_field.data_type() { + DataType::Struct(s) => s, + _ => panic!("'add' should be a struct"), + }; + assert_eq!( + get_field(add_struct, "path").data_type(), + &DataType::Primitive(PrimitiveType::String) + ); + assert_eq!( + get_field(add_struct, "size").data_type(), + &DataType::Primitive(PrimitiveType::Long) + ); + assert!( + matches!( + get_field(add_struct, "partitionValues").data_type(), + DataType::Map(_) + ), + "'partitionValues' should be a map type" + ); + + let metadata_field = get_field(schema, "metaData"); + let metadata_struct = match metadata_field.data_type() { + DataType::Struct(s) => s, + _ => panic!("'metaData' should be a struct"), + }; + let format_field = get_field(metadata_struct, "format"); + let format_struct = match format_field.data_type() { + DataType::Struct(s) => s, + _ => panic!("'format' should be a struct"), + }; + assert_eq!( + get_field(format_struct, "provider").data_type(), + &DataType::Primitive(PrimitiveType::String) + ); + + let protocol_field = get_field(schema, "protocol"); + let protocol_struct = match protocol_field.data_type() { + DataType::Struct(s) => s, + _ => panic!("'protocol' should be a struct"), + }; + assert_eq!( + get_field(protocol_struct, "minReaderVersion").data_type(), + &DataType::Primitive(PrimitiveType::Integer) + ); + assert_eq!( + get_field(protocol_struct, "minWriterVersion").data_type(), + &DataType::Primitive(PrimitiveType::Integer) + ); } /// Contract: [`ParquetHandler::read_parquet_footer`] must return an error for a /// non-existent file. -pub(crate) fn test_parquet_handler_footer_errors_on_missing_file(handler: &dyn ParquetHandler) { +pub fn test_parquet_handler_footer_errors_on_missing_file(handler: &dyn ParquetHandler) { let mut temp_path = std::env::temp_dir(); temp_path.push("non_existent_kernel_test_file.parquet"); let file_meta = FileMeta { @@ -157,7 +229,7 @@ pub(crate) fn test_parquet_handler_footer_errors_on_missing_file(handler: &dyn P /// Contract: [`ParquetHandler::read_parquet_footer`] must preserve Arrow field IDs, /// accessible via [`ColumnMetadataKey::ParquetFieldId`]. -pub(crate) fn test_parquet_handler_footer_preserves_field_ids(handler: &dyn ParquetHandler) { +pub fn test_parquet_handler_footer_preserves_field_ids(handler: &dyn ParquetHandler) { let make_field_with_id = |name: &str, ty: ArrowDataType, nullable: bool, id: &str| { Field::new(name, ty, nullable).with_metadata(HashMap::from([( PARQUET_FIELD_ID_META_KEY.to_string(), @@ -204,7 +276,7 @@ pub(crate) fn test_parquet_handler_footer_preserves_field_ids(handler: &dyn Parq /// Contract: [`ParquetHandler::write_parquet_file`] always overwrites an existing file. /// Writes `[1, 2, 3]`, then overwrites with `[10, 20]`, and verifies only `[10, 20]` /// is present. -pub(crate) fn test_parquet_handler_write_always_overwrites(handler: &dyn ParquetHandler) { +pub fn test_parquet_handler_write_always_overwrites(handler: &dyn ParquetHandler) { let temp_dir = tempdir().unwrap(); let file_path = temp_dir.path().join("overwrite_test.parquet"); let url = Url::from_file_path(&file_path).unwrap(); @@ -264,11 +336,87 @@ pub(crate) fn test_parquet_handler_write_always_overwrites(handler: &dyn Parquet ); } -// --------------------------------------------------------------------------- +/// Contract: [`ParquetHandler::write_parquet_file`] must not embed the Arrow IPC schema +/// (`ARROW:schema`) in the file's key/value metadata. +pub fn test_parquet_handler_write_omits_arrow_schema(handler: &dyn ParquetHandler) { + let temp_dir = tempdir().unwrap(); + let file_path = temp_dir.path().join("no_arrow_schema.parquet"); + let url = Url::from_file_path(&file_path).unwrap(); + + let data: Box = Box::new(ArrowEngineData::new( + RecordBatch::try_from_iter(vec![( + "id", + Arc::new(Int64Array::from(vec![1, 2])) as Arc, + )]) + .unwrap(), + )); + handler + .write_parquet_file(url, Box::new(std::iter::once(Ok(data)))) + .unwrap(); + + let builder = + ParquetRecordBatchReaderBuilder::try_new(std::fs::File::open(&file_path).unwrap()).unwrap(); + let kv = builder.metadata().file_metadata().key_value_metadata(); + let has = kv + .map(|kv| kv.iter().any(|e| e.key == ARROW_SCHEMA_META_KEY)) + .unwrap_or(false); + assert!( + !has, + "Parquet file should not contain embedded Arrow schema metadata" + ); +} + +/// Contract: [`ParquetHandler::read_parquet_files`] must successfully read a file whose +/// metadata contains an embedded Arrow IPC schema (e.g. one produced by [`ArrowWriter`] +/// without explicit options to skip it). +pub fn test_parquet_handler_reads_file_with_arrow_schema(handler: &dyn ParquetHandler) { + let temp_dir = tempdir().unwrap(); + let file_path = temp_dir.path().join("with_arrow_schema.parquet"); + + let batch = RecordBatch::try_from_iter(vec![( + "value", + Arc::new(Int64Array::from(vec![10, 20, 30])) as Arc, + )]) + .unwrap(); + let mut writer = ArrowWriter::try_new( + std::fs::File::create(&file_path).unwrap(), + batch.schema(), + None, + ) + .unwrap(); + writer.write(&batch).unwrap(); + writer.close().unwrap(); + + let file_meta = file_meta_for(&file_path); + let schema = Arc::new(batch.schema().as_ref().try_into_kernel().unwrap()); + let batches: Vec = handler + .read_parquet_files(&[file_meta], schema, None) + .unwrap() + .map(|r| { + ArrowEngineData::try_from_engine_data(r.unwrap()) + .unwrap() + .into() + }) + .collect(); + + assert_eq!(batches.len(), 1); + assert_eq!(batches[0].num_rows(), 3); + assert_eq!( + batches[0] + .column(0) + .as_any() + .downcast_ref::() + .unwrap() + .values(), + &[10, 20, 30] + ); +} + +// =========================================================================== // Storage / Engine helpers (used by the engine-level tests) -// --------------------------------------------------------------------------- +// =========================================================================== -pub(crate) fn test_arrow_engine(engine: &dyn Engine, base_url: &Url) { +pub fn test_arrow_engine(engine: &dyn Engine, base_url: &Url) { test_list_from_should_sort_and_filter(engine, base_url, get_arrow_data); } diff --git a/test-utils/src/lib.rs b/test-utils/src/lib.rs index 734d4cc234..1c72f53142 100644 --- a/test-utils/src/lib.rs +++ b/test-utils/src/lib.rs @@ -1,39 +1,220 @@ //! A number of utilities useful for testing that we want to use in multiple crates -use std::collections::HashMap; -use std::sync::Arc; +pub mod column_mapping_fixtures; +pub mod counting_reporter; +pub mod engine_contract; +pub mod table_builder; + +/// Helper macro to extract a typed column from a RecordBatch or StructArray. +#[macro_export] +macro_rules! get_column { + ($source:expr, $name:expr, $ty:ty) => { + $source + .column_by_name($name) + .unwrap_or_else(|| panic!("should have column '{}'", $name)) + .as_any() + .downcast_ref::<$ty>() + .unwrap_or_else(|| panic!("column '{}' should be {}", $name, stringify!($ty))) + }; +} + +// `rstest` and the `table_builder` factories appear inside the `define_sweeps!` +// invocation below. Macro bodies are token streams that are only resolved when +// consumer crates apply the emitted templates, so rustc doesn't see these uses. +#[allow(unused_imports)] +use rstest::rstest; +pub use rstest_reuse; +use rstest_reuse::template; +#[allow(unused_imports)] +use table_builder::*; + +/// Emits canonical sweep templates from a single source of truth. +/// +/// Generates one `default_sweep` template (full Cartesian product of all five axes) +/// plus one per-axis template (`log_state_sweep`, `feature_set_sweep`, +/// `data_layout_sweep`, `table_config_sweep`, `version_target_sweep`). Each per-axis +/// template only supplies its own axis; the consumer test fills in the others with +/// inline `#[values(...)]` to control combination count. +/// +/// Templates are emitted at crate root so the generated `#[macro_export]` macros +/// are reachable cross-crate as `test_utils::` via `#[rstest_reuse::apply]`. +/// +/// ```ignore +/// // Iterate only the LogState axis; fix everything else. +/// #[apply(log_state_sweep)] +/// fn my_test( +/// log_state: LogState, +/// #[values(no_features())] feature_set: FeatureSet, +/// #[values(unpartitioned())] data_layout: DataLayoutConfig, +/// #[values(checkpoint_json_stats())] table_config: TableConfig, +/// #[values(version_latest())] version_target: VersionTarget, +/// ) { /* ... */ } +/// ``` +macro_rules! define_sweeps { + ( + log_state_values = $ls:tt, + feature_set_values = $fs:tt, + data_layout_values = $dl:tt, + table_config_values = $tc:tt, + layout_config_values = $lc:tt, + version_target_values = $vt:tt $(,)? + ) => { + #[template] + #[export] + #[rstest] + pub fn default_sweep( + #[values $ls] log_state: LogState, + #[values $fs] feature_set: FeatureSet, + #[values $lc] layout_config: (DataLayoutConfig, TableConfig), + #[values $vt] version_target: VersionTarget, + ) { + } + + #[template] + #[export] + #[rstest] + pub fn log_state_sweep(#[values $ls] log_state: LogState) {} + + #[template] + #[export] + #[rstest] + pub fn feature_set_sweep(#[values $fs] feature_set: FeatureSet) {} + + #[template] + #[export] + #[rstest] + pub fn data_layout_sweep(#[values $dl] data_layout: DataLayoutConfig) {} + + #[template] + #[export] + #[rstest] + pub fn table_config_sweep(#[values $tc] table_config: TableConfig) {} + + #[template] + #[export] + #[rstest] + pub fn version_target_sweep(#[values $vt] version_target: VersionTarget) {} + }; +} -use delta_kernel::actions::get_log_add_schema; +define_sweeps! { + // TODO: Log compaction (needs LogState::with_compaction_at, #2337). + // TODO: Schema history (add/drop/rename) (needs schema-evolution support). + log_state_values = ( + commits_only(), + checkpoint_at_end(), + checkpoint_at_end_no_hint(), + checkpoint_mid(), + checkpoint_mid_no_hint(), + two_checkpoints_stale_hint(), + crc_at_end(), + crc_at_mid(), + checkpoint_at_end_crc_at_end(), + checkpoint_at_end_post_cleanup(), + checkpoint_at_end_no_hint_post_cleanup(), + checkpoint_mid_post_cleanup(), + checkpoint_mid_no_hint_post_cleanup(), + two_checkpoints_stale_hint_post_cleanup(), + checkpoint_mid_crc_at_mid_post_cleanup(), + checkpoint_mid_crc_above_mid_post_cleanup(), + checkpoint_mid_crc_at_end_post_cleanup() + ), + // TODO: max-CM=id / max-CM=name full set (needs checkpointProtection, clustering, + // materializePartitionColumns, invariants, checkConstraints, generatedColumns, + // allowColumnDefaults, identityColumns, NTZ/variant (schema-driven), + // catalogManaged, collations for CM=name, typeWidening write support). + // TODO: iceV2+writer (needs icebergCompatV2 + icebergWriterCompatV1). + // TODO: iceV3 (needs icebergCompatV3). + feature_set_values = (no_features(), all_features_cm_id(), all_features_cm_name()), + // TODO: null-distribution and partition-by-timestamp-with-CM rows. + data_layout_values = (unpartitioned(), partitioned(), clustered()), + table_config_values = ( + checkpoint_json_stats(), + checkpoint_struct_stats(), + no_checkpoint_stats() + ), + // Data layout and stats config are bundled into one axis rather than crossed: this + // sweep round-trips each pairing with no predicate, so the stats config can't affect + // the version or row-count assertions, and crossing the two would add cases without + // adding coverage. Skipping behavior is asserted in predicate-bearing unit tests. + layout_config_values = ( + (unpartitioned(), no_checkpoint_stats()), + (partitioned(), with_json_stats(num_indexed_cols_zero())), + (clustered(), with_struct_stats(num_indexed_cols_zero())), + (unpartitioned(), with_json_stats(num_indexed_cols_narrow())), + (partitioned(), with_struct_stats(num_indexed_cols_narrow())), + (clustered(), with_json_stats(num_indexed_cols_all())), + (unpartitioned(), with_struct_stats(num_indexed_cols_all())), + (partitioned(), with_json_stats(stats_columns_empty())), + (clustered(), with_struct_stats(stats_columns_empty())), + (unpartitioned(), with_json_stats(stats_columns_reordered())), + (partitioned(), with_struct_stats(stats_columns_reordered())) + ), + // `version_at_timestamp_max()` is the only timestamp row; see its docs for why + // intermediate-version resolution lives in a dedicated test instead. + version_target_values = ( + version_latest(), + version_at_mid(), + version_incremental_from_mid_to_latest(), + version_incremental_from_mid_to_pre_latest(), + version_at_timestamp_max() + ), +} +use std::collections::{HashMap, HashSet}; +use std::num::NonZero; +use std::sync::{Arc, Mutex}; + +pub use counting_reporter::{ + ensure_metrics_compatible_global_subscriber, install_thread_local_metrics_reporter, + CapturingReporter, CountingReporter, RelaxedCounter, +}; +use delta_kernel::actions::{ + LOG_ADD_SCHEMA, MAX_VALUES, MIN_VALUES, NULL_COUNT, NUM_RECORDS, TIGHT_BOUNDS, +}; use delta_kernel::arrow::array::{ - Array, ArrayRef, BooleanArray, Float64Array, Int32Array, Int64Array, MapArray, RecordBatch, - StringArray, StructArray, + Array, ArrayRef, AsArray, BooleanArray, Float64Array, Int32Array, Int64Array, MapArray, + MapBuilder, RecordBatch, StringArray, StringBuilder, StructArray, }; use delta_kernel::arrow::buffer::OffsetBuffer; -use delta_kernel::arrow::datatypes::{DataType as ArrowDataType, Field, Schema as ArrowSchema}; +use delta_kernel::arrow::compute::concat; +use delta_kernel::arrow::datatypes::{ + DataType as ArrowDataType, Field, Int64Type, Schema as ArrowSchema, +}; use delta_kernel::arrow::error::ArrowError; use delta_kernel::arrow::util::pretty::pretty_format_batches; -use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::committer::{ + CommitMetadata, CommitResponse, Committer, FileSystemCommitter, PublishMetadata, +}; use delta_kernel::engine::arrow_conversion::TryFromKernel; use delta_kernel::engine::arrow_data::{ArrowEngineData, EngineDataArrowExt}; -use delta_kernel::engine::default::executor::tokio::{ - TokioBackgroundExecutor, TokioMultiThreadExecutor, -}; -use delta_kernel::engine::default::executor::TaskExecutor; -use delta_kernel::engine::default::storage::store_from_url; -use delta_kernel::engine::default::{DefaultEngine, DefaultEngineBuilder}; +use delta_kernel::expressions::Scalar; use delta_kernel::object_store::local::LocalFileSystem; use delta_kernel::object_store::memory::InMemory; -use delta_kernel::object_store::{path::Path, DynObjectStore}; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::{DynObjectStore, ObjectStoreExt as _}; use delta_kernel::parquet::arrow::arrow_writer::ArrowWriter; use delta_kernel::parquet::file::properties::WriterProperties; use delta_kernel::scan::Scan; -use delta_kernel::schema::{DataType, SchemaRef, StructField, StructType}; -use delta_kernel::transaction::CommitResult; -use delta_kernel::{try_parse_uri, DeltaResult, Engine, EngineData, Snapshot}; - +use delta_kernel::schema::{ + ColumnMetadataKey, DataType, MetadataValue, SchemaRef, StructField, StructType, +}; +use delta_kernel::table_features::{assign_column_mapping_metadata, find_max_column_id_in_schema}; +use delta_kernel::transaction::{CommitResult, Transaction}; +use delta_kernel::{ + try_parse_uri, CancellationToken, CancelledFuture, DeltaResult, DeltaResultIterator, Engine, + EngineData, Error, FileMeta, FilteredEngineData, LogPath, Snapshot, +}; +// Re-export `delta_kernel_default_engine` so kernel's integration tests can access it without +// taking a direct dev-dep on the new crate (which would create a cycle via this crate). +pub use delta_kernel_default_engine; +use delta_kernel_default_engine::executor::tokio::{ + TokioBackgroundExecutor, TokioMultiThreadExecutor, +}; +use delta_kernel_default_engine::executor::TaskExecutor; +use delta_kernel_default_engine::storage::store_from_url; +use delta_kernel_default_engine::{DefaultEngine, DefaultEngineBuilder}; use itertools::Itertools; use serde_json::{json, to_vec, Deserializer}; -use std::sync::Mutex; use tracing::subscriber::DefaultGuard; use tracing_subscriber::layer::SubscriberExt; use url::Url; @@ -100,12 +281,38 @@ pub const METADATA_WITH_PARTITION_COLS: &str = r#"{"commitInfo":{"timestamp":158 {"protocol":{"minReaderVersion":1,"minWriterVersion":2}} {"metaData":{"id":"5fba94ed-9794-4965-ba6e-6ee3c0d22af9","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"val\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["val"],"configuration":{},"createdTime":1587968585495}}"#; +/// Like [`METADATA`] but with non-empty table properties including `delta.appendOnly` and +/// `custom.key`. +pub const METADATA_WITH_TABLE_PROPERTIES: &str = r#"{"commitInfo":{"timestamp":1587968586154,"operation":"WRITE","operationParameters":{"mode":"ErrorIfExists","partitionBy":"[]"},"isBlindAppend":true}} +{"protocol":{"minReaderVersion":1,"minWriterVersion":2}} +{"metaData":{"id":"5fba94ed-9794-4965-ba6e-6ee3c0d22af9","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"val\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.appendOnly":"true","custom.key":"custom_value"},"createdTime":1587968585495}}"#; + +/// Like [`METADATA`] but with table-features protocol (v3/v7) including columnMapping (reader) +/// and columnMapping + rowTracking (writer). Metadata includes a table name and column mapping +/// configuration. +pub const METADATA_WITH_FEATURES: &str = concat!( + r#"{"commitInfo":{"timestamp":1587968586154,"operation":"WRITE","operationParameters":{},"isBlindAppend":true}}"#, + "\n", + r#"{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["columnMapping"],"writerFeatures":["columnMapping","domainMetadata","rowTracking"]}}"#, + "\n", + r#"{"metaData":{"id":"deadbeef-1234-5678-abcd-000000000000","name":"test_table","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[]}","partitionColumns":[],"configuration":{"delta.columnMapping.mode":"name","delta.rowTracking.enabled":"true","delta.rowTracking.materializedRowIdColumnName":"_row_id","delta.rowTracking.materializedRowCommitVersionColumnName":"_row_commit_version"},"createdTime":1234567890000}}"#, +); + +/// Like [`METADATA`] but with protocol v3/7 and the `catalogManaged` table feature enabled. +/// Per the Delta protocol, `catalogManaged` depends on `inCommitTimestamp`, and commitInfo must +/// include a `txnId`. +pub const CATALOG_MANAGED_METADATA: &str = r#"{"commitInfo":{"timestamp":1587968586154,"operation":"WRITE","operationParameters":{"mode":"ErrorIfExists","partitionBy":"[]"},"isBlindAppend":true,"txnId":"test-txn-0","inCommitTimestamp":1587968586154}} +{"protocol":{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["catalogManaged"],"writerFeatures":["catalogManaged","inCommitTimestamp"]}} +{"metaData":{"id":"5fba94ed-9794-4965-ba6e-6ee3c0d22af9","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"val\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{"delta.enableInCommitTimestamps":"true"},"createdTime":1587968585495}}"#; + pub enum TestAction { Add(String), Remove(String), Metadata, // TODO: This is a temporary fix to make the test compatible with the file size requirement. - // In the future, we can create an AddCommit/RemoveCommit struct type with DefaultAddCommit/DefaultRemoveCommit value to store the commit info in the enum for Add/Remove. + // In the future, we can create an AddCommit/RemoveCommit struct type with + // DefaultAddCommit/DefaultRemoveCommit value to store the commit info in the enum for + // Add/Remove. AddWithSize(String, u64), RemoveWithSize(String, u64), } @@ -117,7 +324,13 @@ pub fn actions_to_string(actions: Vec) -> String { actions_to_string_with_metadata(actions, METADATA) } -/// Convert a vector of actions into a newline delimited json string, with metadata including a partition column +/// Convert a vector of actions into a newline delimited json string, with catalog-managed metadata +pub fn actions_to_string_catalog_managed(actions: Vec) -> String { + actions_to_string_with_metadata(actions, CATALOG_MANAGED_METADATA) +} + +/// Convert a vector of actions into a newline delimited json string, with metadata including a +/// partition column pub fn actions_to_string_partitioned(actions: Vec) -> String { actions_to_string_with_metadata(actions, METADATA_WITH_PARTITION_COLS) } @@ -157,29 +370,29 @@ pub fn record_batch_to_bytes_with_props( /// Anything that implements `IntoArray` can turn itself into a reference to an arrow array pub trait IntoArray { - fn into_array(self) -> ArrayRef; + fn into_arrow_array(self) -> ArrayRef; } impl IntoArray for Vec { - fn into_array(self) -> ArrayRef { + fn into_arrow_array(self) -> ArrayRef { Arc::new(Int32Array::from(self)) } } impl IntoArray for Vec { - fn into_array(self) -> ArrayRef { + fn into_arrow_array(self) -> ArrayRef { Arc::new(Int64Array::from(self)) } } impl IntoArray for Vec { - fn into_array(self) -> ArrayRef { + fn into_arrow_array(self) -> ArrayRef { Arc::new(BooleanArray::from(self)) } } impl IntoArray for Vec<&'static str> { - fn into_array(self) -> ArrayRef { + fn into_arrow_array(self) -> ArrayRef { Arc::new(StringArray::from(self)) } } @@ -198,8 +411,8 @@ where /// respectively pub fn generate_simple_batch() -> Result { generate_batch(vec![ - ("id", vec![1, 2, 3].into_array()), - ("val", vec!["a", "b", "c"].into_array()), + ("id", vec![1, 2, 3].into_arrow_array()), + ("val", vec!["a", "b", "c"].into_arrow_array()), ]) } @@ -209,6 +422,19 @@ pub fn delta_path_for_version(version: u64, suffix: &str) -> Path { Path::from(path.as_str()) } +/// Create a [`LogPath`] from a table root URL string and an object-store commit path. Useful for +/// building log tails in tests. +pub fn create_log_path(table_root: impl AsRef, commit_path: Path) -> LogPath { + let table_url = try_parse_uri(table_root.as_ref()).expect("Failed to parse table root as URL"); + let commit_url = table_url.join(commit_path.as_ref()).unwrap(); + let file_meta = FileMeta { + location: commit_url, + last_modified: 123, + size: 100, + }; + LogPath::try_new(file_meta).expect("Failed to create LogPath") +} + pub fn staged_commit_path_for_version(version: u64) -> Path { let uuid = uuid::Uuid::new_v4(); let path = format!("_delta_log/_staged_commits/{version:020}.{uuid}.json"); @@ -222,7 +448,10 @@ pub fn compacted_log_path_for_versions(start_version: u64, end_version: u64, suf } // Resolve a table from a root and relative path -fn resolve_table_path(table_root: impl AsRef, relative: &Path) -> DeltaResult { +pub(crate) fn resolve_table_path( + table_root: impl AsRef, + relative: &Path, +) -> DeltaResult { let url = try_parse_uri(table_root)?; Ok(Path::from_url_path(url.join(relative.as_ref())?.path())?) } @@ -284,14 +513,127 @@ pub fn into_record_batch(engine_data: Box) -> RecordBatch { .into() } +/// A modification to an add-file batch's `partitionValues` keys. +#[derive(Clone, Copy)] +pub enum AddFilePartitionKeyModify<'a> { + Drop { + key: &'a str, + }, + Insert { + key: &'a str, + value: Option<&'a str>, + }, +} + +/// Applies `modifications` in order to every `partitionValues` row in an add-file batch. +/// +/// `Drop` removes every entry with the given key. `Insert` appends a new entry. +/// +/// # Panics +/// +/// Panics when `batch` does not contain a string-keyed and string-valued `partitionValues` map, or +/// when the modified batch cannot be constructed. +pub fn modify_add_file_partition_keys( + batch: RecordBatch, + modifications: &[AddFilePartitionKeyModify<'_>], +) -> RecordBatch { + if modifications.is_empty() { + return batch; + } + + let index = batch + .schema() + .index_of("partitionValues") + .expect("partitionValues field in add-file batch"); + let map = batch.column(index).as_map(); + let (entry_field, ordered) = match map.data_type() { + ArrowDataType::Map(entry_field, ordered) => (entry_field.clone(), *ordered), + _ => unreachable!("partitionValues column must be a map"), + }; + let (key_field, value_field) = map.entries_fields(); + let mut builder = MapBuilder::new(None, StringBuilder::new(), StringBuilder::new()) + .with_keys_field(key_field.clone()) + .with_values_field(value_field.clone()); + for row in 0..map.len() { + let entries = map.value(row); + let keys = entries.column(0).as_string::(); + let values = entries.column(1).as_string::(); + let mut partition_values: Vec<(&str, Option<&str>)> = (0..keys.len()) + .map(|i| (keys.value(i), values.is_valid(i).then(|| values.value(i)))) + .collect(); + for modification in modifications { + match *modification { + AddFilePartitionKeyModify::Drop { key } => { + partition_values.retain(|(existing_key, _)| *existing_key != key); + } + AddFilePartitionKeyModify::Insert { key, value } => { + partition_values.push((key, value)); + } + } + } + for (key, value) in partition_values { + builder.keys().append_value(key); + match value { + Some(value) => builder.values().append_value(value), + None => builder.values().append_null(), + } + } + builder + .append(true) + .expect("failed to append partition-values map row"); + } + let (_, offsets, entries, nulls, _) = builder.finish().into_parts(); + let new_map: ArrayRef = Arc::new( + MapArray::try_new(entry_field, offsets, entries, nulls, ordered) + .expect("failed to rebuild partition-values map"), + ); + + let mut columns = batch.columns().to_vec(); + columns[index] = new_map; + RecordBatch::try_new(batch.schema(), columns) + .expect("failed to rebuild add-file batch after modifying a partition key") +} + +/// Replaces one row in an Arrow array with a one-row array of the same type. +/// +/// # Panics +/// +/// Panics if `replacement` does not contain exactly one row, `row` is out of bounds, or the arrays +/// cannot be concatenated. +pub fn replace_array_row(column: &ArrayRef, replacement: ArrayRef, row: usize) -> ArrayRef { + assert_eq!( + replacement.len(), + 1, + "replacement must contain exactly one row" + ); + let slices = [ + column.slice(0, row), + replacement, + column.slice(row + 1, column.len() - row - 1), + ]; + let arrays: Vec<&dyn Array> = slices.iter().map(|array| array.as_ref()).collect(); + concat(&arrays).expect("replacement value must match the modified column type") +} + +pub fn create_default_engine( + table_root: &url::Url, +) -> DeltaResult>> { + create_default_engine_with_batch(table_root, None) +} + /// Helper to create a DefaultEngine with the default executor for tests. /// /// Uses `TokioBackgroundExecutor` as the default executor. -pub fn create_default_engine( +pub fn create_default_engine_with_batch( table_root: &url::Url, + batch_size: Option, ) -> DeltaResult>> { let store = store_from_url(table_root)?; - Ok(Arc::new(DefaultEngineBuilder::new(store).build())) + let mut builder = DefaultEngineBuilder::new(store); + if let Some(batch_size) = batch_size { + builder = builder.with_batch_size(NonZero::new(batch_size).unwrap()); + } + Ok(Arc::new(builder.build())) } /// Helper to create a DefaultEngine with the default executor for tests. @@ -361,7 +703,8 @@ pub fn test_table_setup_mt() -> DeltaResult<( Ok((temp_dir, table_path, engine)) } -// setup default engine with in-memory (local_directory=None) or local fs (local_directory=Some(Url)) +// setup default engine with in-memory (local_directory=None) or local fs +// (local_directory=Some(Url)) pub fn engine_store_setup( table_name: &str, local_directory: Option<&Url>, @@ -385,8 +728,14 @@ pub fn engine_store_setup( (storage, engine, url) } +/// Fixed in-commit timestamp (milliseconds since the Unix epoch) written by [`create_table`] when +/// the `inCommitTimestamp` writer feature is enabled. +pub const TEST_ICT_ENABLEMENT_TIMESTAMP: i64 = 1612345678; + // we provide this table creation function since we only do appends to existing tables for now. // this will just create an empty table with the given schema. (just protocol + metadata actions) +// For property-gated writer features, this helper also writes the corresponding enablement +// property when the writer feature is requested. #[allow(clippy::too_many_arguments)] pub async fn create_table( store: Arc, @@ -394,10 +743,36 @@ pub async fn create_table( schema: SchemaRef, partition_columns: &[&str], use_37_protocol: bool, - reader_features: Vec<&str>, - writer_features: Vec<&str>, + mut reader_features: Vec<&str>, + mut writer_features: Vec<&str>, ) -> Result> { let table_id = "test_id"; + + // IcebergCompatV3 requires ColumnMapping, RowTracking, and DomainMetadata. Add them so callers + // can pass just `icebergCompatV3` (plus e.g. `allowColumnDefaults`) and get a loadable table. + let enable_iceberg_compat_v3 = writer_features.contains(&"icebergCompatV3"); + if enable_iceberg_compat_v3 { + if !reader_features.contains(&"columnMapping") { + reader_features.push("columnMapping"); + } + for f in ["columnMapping", "rowTracking", "domainMetadata"] { + if !writer_features.contains(&f) { + writer_features.push(f); + } + } + } + + // Column mapping requires per-field `id`/`physicalName` metadata, without which snapshot load + // fails. Assign it here (with nested ids for iceberg v3); `max_column_id` feeds + // `delta.columnMapping.maxColumnId` below. + let (schema, max_column_id) = if reader_features.contains(&"columnMapping") { + let mut max_id = find_max_column_id_in_schema(&schema).unwrap_or(0); + let schema = + assign_column_mapping_metadata(&schema, &mut max_id, enable_iceberg_compat_v3)?; + (Arc::new(schema), max_id) + } else { + (schema, 0i64) + }; let schema = serde_json::to_string(&schema)?; let protocol = if use_37_protocol { @@ -423,14 +798,22 @@ pub async fn create_table( if reader_features.contains(&"columnMapping") { config.insert("delta.columnMapping.mode".to_string(), json!("name")); + config.insert( + "delta.columnMapping.maxColumnId".to_string(), + json!(max_column_id.to_string()), + ); + } + if writer_features.contains(&"icebergCompatV3") { + config.insert("delta.enableIcebergCompatV3".to_string(), json!("true")); } if writer_features.contains(&"rowTracking") { + config.insert("delta.enableRowTracking".to_string(), json!("true")); config.insert( - "delta.materializedRowIdColumnName".to_string(), + "delta.rowTracking.materializedRowIdColumnName".to_string(), json!("some_dummy_column_name"), ); config.insert( - "delta.materializedRowCommitVersionColumnName".to_string(), + "delta.rowTracking.materializedRowCommitVersionColumnName".to_string(), json!("another_dummy_column_name"), ); } @@ -442,12 +825,18 @@ pub async fn create_table( ); config.insert( "delta.inCommitTimestampEnablementTimestamp".to_string(), - json!("1612345678"), + json!(TEST_ICT_ENABLEMENT_TIMESTAMP.to_string()), ); } if writer_features.contains(&"changeDataFeed") { config.insert("delta.enableChangeDataFeed".to_string(), json!("true")); } + if writer_features.contains(&"deletionVectors") { + config.insert("delta.enableDeletionVectors".to_string(), json!("true")); + } + if reader_features.contains(&"catalogManaged") { + config.insert("io.unitycatalog.tableId".to_string(), json!(table_id)); + } config }; @@ -469,7 +858,7 @@ pub async fn create_table( // Add commitInfo with ICT if ICT is enabled let commit_info = if writer_features.contains(&"inCommitTimestamp") { // When ICT is enabled from version 0, we need to include it in the initial commit - let timestamp = 1612345678i64; // Use a fixed timestamp for testing + let timestamp = TEST_ICT_ENABLEMENT_TIMESTAMP; Some(json!({ "commitInfo": { "timestamp": timestamp, @@ -510,6 +899,39 @@ pub async fn create_table( Ok(table_path) } +/// Returns a copy of `schema` with `CURRENT_DEFAULT` metadata attached to the named top-level +/// fields. +/// +/// `column_defaults` maps `column_name -> default_sql`. The raw SQL is stored verbatim; this +/// helper does not parse or validate it. +/// +/// # Errors +/// +/// Returns an error if `column_defaults` names a column that is not a top-level field of `schema`. +pub fn schema_with_column_defaults( + schema: &StructType, + mut column_defaults: HashMap<&str, &str>, +) -> DeltaResult { + let augmented_fields: Vec<_> = schema + .fields() + .map(|field| match column_defaults.remove(field.name.as_str()) { + Some(sql) => field.clone().add_metadata([( + ColumnMetadataKey::CurrentDefault.as_ref().to_string(), + MetadataValue::String(sql.to_string()), + )]), + None => field.clone(), + }) + .collect(); + if !column_defaults.is_empty() { + return Err(Error::generic(format!( + "column defaults reference unknown top-level columns: {:?}", + column_defaults.into_keys().collect::>() + ))); + } + + Ok(Arc::new(StructType::try_new(augmented_fields)?)) +} + /// Creates two empty test tables, one with 37 protocol and one with 11 protocol. the tables will /// be named {table_base_name}_11 and {table_base_name}_37. The local_directory param can be set to /// write out the tables to the local filesystem, passing in None will create in-memory tables @@ -611,24 +1033,98 @@ pub async fn insert_data( snapshot: Arc, engine: &Arc>, columns: Vec, +) -> DeltaResult { + insert_data_with( + snapshot, + engine, + columns, + Box::new(FileSystemCommitter::new()), + "WRITE", + /* data_change */ true, + /* is_blind_append */ false, + ) + .await +} + +/// Like [`insert_data`] but with the `committer` and the commit's `operation`, `data_change`, and +/// blind-append flag configurable. Pass [`TestCatalogCommitter`] for catalog-managed tables. +#[allow(clippy::too_many_arguments)] +pub async fn insert_data_with( + snapshot: Arc, + engine: &Arc>, + columns: Vec, + committer: Box, + operation: &str, + data_change: bool, + is_blind_append: bool, ) -> DeltaResult { let arrow_schema = TryFromKernel::try_from_kernel(snapshot.schema().as_ref())?; let batch = RecordBatch::try_new(Arc::new(arrow_schema), columns) .map_err(|e| delta_kernel::Error::generic(e.to_string()))?; let mut txn = snapshot - .transaction(Box::new(FileSystemCommitter::new()), engine.as_ref())? - .with_operation("WRITE".to_string()) - .with_data_change(true); + .transaction(committer, engine.as_ref())? + .with_operation(operation.to_string()) + .with_data_change(data_change); + txn.ack_column_defaults(); + if is_blind_append { + txn = txn.with_blind_append(); + } - let write_context = txn.get_write_context(); + let write_context = txn.unpartitioned_write_context()?; let add_files_metadata = engine - .write_parquet(&ArrowEngineData::new(batch), &write_context, HashMap::new()) + .write_parquet(&ArrowEngineData::new(batch), &write_context) .await?; txn.add_files(add_files_metadata); txn.commit(engine.as_ref()) } +/// Starts a transaction using the passed snapshot using a [`FileSystemCommitter`]. +pub fn begin_transaction(snapshot: Arc, engine: &dyn Engine) -> DeltaResult { + snapshot.transaction(Box::new(FileSystemCommitter::new()), engine) +} + +/// A catalog [`Committer`] for tests: writes every commit directly to the published Delta log +/// path, so catalog-managed tables can be created and appended to without a real catalog. +pub struct TestCatalogCommitter; + +impl Committer for TestCatalogCommitter { + fn commit( + &self, + engine: &dyn Engine, + actions: DeltaResultIterator<'_, FilteredEngineData>, + commit_metadata: CommitMetadata, + ) -> DeltaResult { + let path = commit_metadata.published_commit_path()?; + engine + .json_handler() + .write_json_file(&path, Box::new(actions), false)?; + Ok(CommitResponse::Committed { + file_meta: FileMeta::new(path, commit_metadata.in_commit_timestamp(), 0), + }) + } + + fn is_catalog_committer(&self) -> bool { + true + } + + fn publish(&self, _: &dyn Engine, _: PublishMetadata) -> DeltaResult<()> { + Ok(()) + } +} + +/// Load latest snapshot from `table_url` and start a transaction using a [`FileSystemCommitter`]. +/// +/// Convenience for the common test pattern of building a fresh snapshot just to start a +/// transaction, when the snapshot itself is not needed afterward. +pub fn load_and_begin_transaction( + table_url: impl AsRef, + engine: &dyn Engine, +) -> DeltaResult { + let snapshot = Snapshot::builder_for(table_url).build(engine)?; + begin_transaction(snapshot, engine) +} + // Helper function to set json values in a serde_json Values pub fn set_json_value( value: &mut serde_json::Value, @@ -645,16 +1141,17 @@ pub fn set_json_value( } /// Returns a nested schema with 6 top-level fields including a nested struct: -/// `[row_number: long, name: string, score: double, address: {street: string, city: string}, tag: string, value: int]` +/// `[row_number: long, name: string, score: double, address: {street: string, city: string}, tag: +/// string, value: int]` pub fn nested_schema() -> Result> { Ok(Arc::new(StructType::try_new(vec![ - StructField::not_null("row_number", DataType::LONG), + StructField::nullable("row_number", DataType::LONG), StructField::nullable("name", DataType::STRING), StructField::nullable("score", DataType::DOUBLE), StructField::nullable( "address", StructType::try_new(vec![ - StructField::not_null("street", DataType::STRING), + StructField::nullable("street", DataType::STRING), StructField::nullable("city", DataType::STRING), ])?, ), @@ -733,7 +1230,7 @@ pub fn nested_batches() -> Result, Box> /// Schema with one column of the given type: `(id INT, col )`. pub fn schema_with_type(dtype: DataType) -> SchemaRef { Arc::new(StructType::new_unchecked(vec![ - StructField::new("id", DataType::INTEGER, false), + StructField::new("id", DataType::INTEGER, true), StructField::new("col", dtype, true), ])) } @@ -742,12 +1239,10 @@ pub fn schema_with_type(dtype: DataType) -> SchemaRef { /// `(id INT, nested STRUCT>)`. pub fn nested_schema_with_type(dtype: DataType) -> SchemaRef { Arc::new(StructType::new_unchecked(vec![ - StructField::new("id", DataType::INTEGER, false), + StructField::new("id", DataType::INTEGER, true), StructField::new( "nested", - DataType::Struct(Box::new(StructType::new_unchecked(vec![StructField::new( - "inner", dtype, true, - )]))), + StructType::new_unchecked(vec![StructField::new("inner", dtype, true)]), true, ), ])) @@ -756,7 +1251,7 @@ pub fn nested_schema_with_type(dtype: DataType) -> SchemaRef { /// Schema with two columns of the given type: `(id INT, col1 , col2 )`. pub fn multi_schema_with_type(dtype: DataType) -> SchemaRef { Arc::new(StructType::new_unchecked(vec![ - StructField::new("id", DataType::INTEGER, false), + StructField::new("id", DataType::INTEGER, true), StructField::new("col1", dtype.clone(), true), StructField::new("col2", dtype, true), ])) @@ -845,11 +1340,40 @@ pub fn assert_result_error_with_message(res: Result, messa } } +/// Collect `row_id` values from scan batches produced with a `MetadataColumnSpec::RowId` schema. +pub fn collect_row_ids(batches: &[RecordBatch]) -> Vec { + batches + .iter() + .flat_map(|b| { + b.column_by_name("row_id") + .expect("row_id column not found in batch") + .as_primitive::() + .values() + .to_vec() + }) + .collect() +} + +/// Assert every `row_id` across the batches is unique. +pub fn assert_row_ids_unique(batches: &[RecordBatch]) { + let row_ids = collect_row_ids(batches); + let unique: HashSet = row_ids.iter().copied().collect(); + assert_eq!( + unique.len(), + row_ids.len(), + "row IDs must be globally unique: found {} duplicate(s) among {} row(s)", + row_ids.len() - unique.len(), + row_ids.len(), + ); +} + /// Creates add file metadata for one or more files without partition values. -/// Each tuple contains: (file_path, file_size, mod_time, num_records) +/// +/// Each tuple contains `(file_path, file_size, mod_time, num_records)`. `num_records` is +/// `Option` so callers can produce a NULL `stats.numRecords` cell. pub fn create_add_files_metadata( add_files_schema: &SchemaRef, - files: Vec<(&str, i64, i64, i64)>, + files: Vec<(&str, i64, i64, Option)>, ) -> Result, Box> { let num_files = files.len(); @@ -902,12 +1426,12 @@ pub fn create_add_files_metadata( let stats_struct = StructArray::from(vec![ ( - Arc::new(Field::new("numRecords", ArrowDataType::Int64, true)), + Arc::new(Field::new(NUM_RECORDS, ArrowDataType::Int64, true)), Arc::new(num_records_array) as ArrayRef, ), ( Arc::new(Field::new( - "nullCount", + NULL_COUNT, ArrowDataType::Struct(empty_struct_fields.clone()), true, )), @@ -915,7 +1439,7 @@ pub fn create_add_files_metadata( ), ( Arc::new(Field::new( - "minValues", + MIN_VALUES, ArrowDataType::Struct(empty_struct_fields.clone()), true, )), @@ -923,14 +1447,14 @@ pub fn create_add_files_metadata( ), ( Arc::new(Field::new( - "maxValues", + MAX_VALUES, ArrowDataType::Struct(empty_struct_fields), true, )), Arc::new(empty_struct) as ArrayRef, ), ( - Arc::new(Field::new("tightBounds", ArrowDataType::Boolean, true)), + Arc::new(Field::new(TIGHT_BOUNDS, ArrowDataType::Boolean, true)), Arc::new(tight_bounds_array) as ArrayRef, ), ]); @@ -953,22 +1477,27 @@ pub fn create_add_files_metadata( /// snapshot. pub async fn write_batch_to_table( snapshot: &Arc, - engine: &DefaultEngine, + engine: &DefaultEngine, data: RecordBatch, - partition_values: std::collections::HashMap, + partition_values: HashMap, ) -> Result, Box> { let mut txn = snapshot .clone() .transaction(Box::new(FileSystemCommitter::new()), engine)? .with_engine_info("DefaultEngine") .with_data_change(true); - let write_context = txn.get_write_context(); + txn.ack_column_defaults(); + let write_context = if txn.logical_partition_columns().is_empty() { + assert!( + partition_values.is_empty(), + "partition_values should be empty for unpartitioned tables" + ); + txn.unpartitioned_write_context()? + } else { + txn.partitioned_write_context(partition_values)? + }; let add_meta = engine - .write_parquet( - &ArrowEngineData::new(data), - &write_context, - partition_values, - ) + .write_parquet(&ArrowEngineData::new(data), &write_context) .await?; txn.add_files(add_meta); match txn.commit(engine)? { @@ -986,6 +1515,56 @@ pub struct AddInfo { pub stats: Option, } +/// A [`CancellationToken`] for tests. Start uncancelled and flip it with +/// [`cancel`](Self::cancel), or construct one already cancelled with +/// [`cancelled`](Self::cancelled). +/// +/// The [`cancelled_future`](CancellationToken::cancelled_future) future is backed by a +/// [`tokio::sync::Notify`] so it resolves when [`cancel`](Self::cancel) fires even from another +/// thread -- this drives the default engine's mid-read cancellation race, not just the synchronous +/// `is_cancelled` poll. +#[derive(Debug, Default)] +pub struct TestCancellationToken { + cancelled: std::sync::atomic::AtomicBool, + notify: tokio::sync::Notify, +} + +impl TestCancellationToken { + /// A token that is already cancelled. + pub fn cancelled() -> Self { + let token = Self::default(); + token.cancel(); + token + } + + /// Request cancellation, waking any future returned by + /// [`cancelled_future`](CancellationToken::cancelled_future). + pub fn cancel(&self) { + self.cancelled + .store(true, std::sync::atomic::Ordering::SeqCst); + self.notify.notify_waiters(); + } +} + +impl CancellationToken for TestCancellationToken { + fn is_cancelled(&self) -> bool { + self.cancelled.load(std::sync::atomic::Ordering::SeqCst) + } + + fn cancelled_future(&self) -> CancelledFuture<'_> { + Box::pin(async move { + // `notified()` must be registered before the cancellation check to avoid missing a + // `notify_waiters` that races between the two; an already-cancelled token still + // returns immediately via the check. + let notified = self.notify.notified(); + if self.is_cancelled() { + return; + } + notified.await; + }) + } +} + /// Reads all [`AddInfo`]s from a snapshot's log segment. /// /// # Example (conceptual) @@ -1002,7 +1581,7 @@ pub fn read_add_infos( snapshot: &Snapshot, engine: &impl Engine, ) -> Result, Box> { - let schema = get_log_add_schema().clone(); + let schema = LOG_ADD_SCHEMA.clone(); let batches = snapshot.log_segment().read_actions(engine, schema)?; let mut actions = Vec::new(); for batch_result in batches { @@ -1122,6 +1701,92 @@ pub fn read_actions_from_commit( .collect()) } +/// Row tracking fields extracted from a single add action in a commit. +pub struct AddActionRowTracking { + /// The base row ID assigned to the first row in the file. + pub base_row_id: Option, + /// The version of the commit in which this file was first written. + pub default_row_commit_version: Option, +} + +/// Reads all add actions from a commit and returns their row tracking fields, sorted by +/// `baseRowId` for deterministic ordering. +pub fn get_row_tracking_add_actions( + table_url: &Url, + version: u64, +) -> Result, Box> { + let mut actions: Vec = + read_actions_from_commit(table_url, version, "add")? + .into_iter() + .map(|a| AddActionRowTracking { + base_row_id: a["baseRowId"].as_i64(), + default_row_commit_version: a["defaultRowCommitVersion"].as_i64(), + }) + .collect(); + actions.sort_by_key(|a| a.base_row_id); + Ok(actions) +} + +/// Materialized row tracking column name properties extracted from a commit's metadata action. +pub struct MaterializedRowTrackingColumnNames { + /// Value of `delta.rowTracking.materializedRowIdColumnName`, or `None` if not set. + pub row_id_column_name: Option, + /// Value of `delta.rowTracking.materializedRowCommitVersionColumnName`, or `None` if not set. + pub row_commit_version_column_name: Option, +} + +/// Reads the materialized row tracking column name properties from a commit's metadata action. +/// These properties are table properties stored in the metadata `configuration` map. +pub fn get_materialized_row_tracking_column_names( + table_url: &Url, + version: u64, +) -> Result> { + let metadata_actions = read_actions_from_commit(table_url, version, "metaData")?; + let config = metadata_actions + .first() + .and_then(|m| m.get("configuration")) + .cloned() + .unwrap_or(serde_json::Value::Object(Default::default())); + Ok(MaterializedRowTrackingColumnNames { + row_id_column_name: config["delta.rowTracking.materializedRowIdColumnName"] + .as_str() + .map(str::to_owned), + row_commit_version_column_name: config + ["delta.rowTracking.materializedRowCommitVersionColumnName"] + .as_str() + .map(str::to_owned), + }) +} + +/// Reads the `metaData.configuration` map from commit `version` directly through a +/// [`DynObjectStore`]. Works with any backing store (`file://`, `memory://`, etc.) since +/// it bypasses [`read_actions_from_commit`]'s `to_file_path()` requirement. +/// +/// Returns an empty map when the commit contains no `metaData` action, or when the +/// action has an empty/absent `configuration` field. Errors if any configuration value is +/// not a string (the spec requires all configuration entries to be strings). +/// Propagates object-store errors when the commit file is missing or unreadable. +pub async fn read_metadata_configuration_from_store( + store: &DynObjectStore, + version: u64, +) -> Result, Box> { + let path = + delta_kernel::object_store::path::Path::from(format!("_delta_log/{version:020}.json")); + let get_result = store.get(&path).await?; + let bytes = get_result.bytes().await?; + let mut config = HashMap::new(); + for line in std::str::from_utf8(&bytes)?.lines() { + let v: serde_json::Value = serde_json::from_str(line)?; + if let Some(c) = v.get("metaData").and_then(|m| m.get("configuration")) { + if c.is_object() { + let entries: HashMap = serde_json::from_value(c.clone())?; + config.extend(entries); + } + } + } + Ok(config) +} + /// Removes all scan files from the snapshot, commits the transaction, and returns /// the parsed remove actions from the resulting commit log. pub fn remove_all_and_get_remove_actions( @@ -1147,6 +1812,21 @@ pub fn remove_all_and_get_remove_actions( read_actions_from_commit(table_url, committed.commit_version(), "remove") } +/// Build a `serde_json::Value` mapping nested dot-paths to ids. +/// +/// For example, `nested_ids_json(&[("array_in_map.key", 100)])` builds: +/// +/// ```json +/// { "array_in_map.key": 100 } +/// ``` +pub fn nested_ids_json(entries: &[(&str, i64)]) -> serde_json::Value { + let obj: serde_json::Map = entries + .iter() + .map(|(k, v)| (k.to_string(), serde_json::Value::from(*v))) + .collect(); + serde_json::Value::Object(obj) +} + /// Asserts that `action["partitionValues"]` contains the given key with the expected value. pub fn assert_partition_values(action: &serde_json::Value, key: &str, expected_value: &str) { let pv = action["partitionValues"] diff --git a/test-utils/src/table_builder.rs b/test-utils/src/table_builder.rs new file mode 100644 index 0000000000..fe9cf24382 --- /dev/null +++ b/test-utils/src/table_builder.rs @@ -0,0 +1,2690 @@ +// rstest_reuse templates generate macros that appear unused when building the lib target +// (they're only consumed by tests in this and other crates). +#![allow(unused_macros)] + +//! A composable test table builder for delta-kernel-rs. +//! +//! The builder writes data by default (1 parquet file with 10 rows per commit). Override +//! with [`TestTableBuilder::with_data`] when a test needs specific file/row counts. +//! +//! Provides five orthogonal axes for parameterized testing: +//! - [`LogState`]: what log files exist on disk (commits, checkpoints, CRC) +//! - [`FeatureSet`]: which Delta table features are enabled +//! - [`DataLayoutConfig`]: data layout (unpartitioned, partitioned, clustered) +//! - [`TableConfig`]: runtime knobs (e.g. checkpoint stats format) +//! - [`VersionTarget`]: how the snapshot is loaded (latest, time travel, incremental) +//! +//! # Quick start +//! +//! The `test_context!` macro builds a table, engine, and snapshot in one call. +//! Pair it with rstest `#[values]` for cross-product testing: +//! +//! ```ignore +//! use rstest::rstest; +//! use test_utils::table_builder::*; +//! use test_utils::test_context; +//! +//! #[rstest] +//! fn test_scan( +//! #[values(LogState::with_latest_version(2))] +//! log_state: LogState, +//! #[values(FeatureSet::empty())] +//! feature_set: FeatureSet, +//! #[values(unpartitioned())] +//! data_layout: DataLayoutConfig, +//! #[values(checkpoint_json_stats())] +//! table_config: TableConfig, +//! #[values(VersionTarget::Latest, VersionTarget::IncrementalToLatest { from: 0 })] +//! version_target: VersionTarget, +//! ) { +//! let (engine, snap, _table) = test_context!( +//! log_state, feature_set, data_layout, table_config, version_target, +//! ); +//! let scan = snap.scan_builder().build().unwrap(); +//! // ... +//! } +//! ``` +//! +//! Requires `Snapshot` and `DefaultEngineBuilder` to be in scope at the call site. +//! The macros expand there, so types resolve to the caller's kernel crate -- avoiding +//! the type mismatch between `test_utils`'s kernel and `kernel/src/` unit tests. + +use std::collections::{HashMap, HashSet}; +use std::fmt; +use std::sync::Arc; + +use delta_kernel::arrow::array::{ + Array, ArrayRef, BinaryArray, BooleanArray, Date32Array, Decimal128Array, Float32Array, + Float64Array, Int16Array, Int32Array, Int64Array, Int8Array, RecordBatch, StringArray, + StructArray, TimestampMicrosecondArray, +}; +use delta_kernel::arrow::buffer::NullBuffer; +use delta_kernel::arrow::datatypes::{DataType as ArrowDataType, Schema as ArrowSchema, TimeUnit}; +use delta_kernel::checkpoint::{CheckpointSpec, V2CheckpointConfig}; +use delta_kernel::committer::FileSystemCommitter; +use delta_kernel::engine::arrow_conversion::TryFromKernel; +use delta_kernel::engine::arrow_data::ArrowEngineData; +use delta_kernel::expressions::Scalar; +use delta_kernel::object_store::memory::InMemory; +use delta_kernel::object_store::path::Path; +use delta_kernel::object_store::{DynObjectStore, Error as ObjectStoreError, ObjectStoreExt as _}; +use delta_kernel::schema::{DataType, PrimitiveType, SchemaRef, StructField, StructType}; +use delta_kernel::snapshot::ChecksumWriteResult; +use delta_kernel::table_features::TableFeature; +use delta_kernel::transaction::create_table::create_table; +use delta_kernel::transaction::data_layout::DataLayout; +use delta_kernel::{DeltaResult, Engine, Snapshot}; +use delta_kernel_default_engine::executor::tokio::{ + TokioBackgroundExecutor, TokioMultiThreadExecutor, +}; +use delta_kernel_default_engine::executor::TaskExecutor; +use delta_kernel_default_engine::{DefaultEngine, DefaultEngineBuilder}; + +// =========================================================================== +// Sweep constants +// =========================================================================== + +/// Latest commit version used by every log state in the +/// [`default_sweep!`](crate::default_sweep) template. +pub const DEFAULT_SWEEP_LATEST_VERSION: u64 = 10; + +/// Mid version used by the [`default_sweep!`](crate::default_sweep) template for +/// `AtVersion` and `IncrementalToLatest` targets. Must satisfy +/// `mid <= DEFAULT_SWEEP_LATEST_VERSION`. +pub const DEFAULT_SWEEP_MID_VERSION: u64 = 5; + +// =========================================================================== +// Sync/async bridge +// =========================================================================== + +/// Run `make_fut` to completion on a dedicated multi-threaded tokio runtime in a +/// scoped background thread. Safe to call from both sync tests and `#[tokio::test]` +/// bodies -- the scoped thread avoids the nested-runtime panic that occurs when +/// calling `Runtime::block_on` from a thread that already owns a tokio runtime. +/// +/// A multi-threaded runtime is required so kernel operations that call +/// `block_in_place` (e.g. `Snapshot::checkpoint`) do not deadlock, which is why +/// the sync wrappers in this module all route through this helper. +fn block_on_sync(make_fut: F) -> DeltaResult +where + F: FnOnce() -> Fut + Send, + Fut: std::future::Future>, + T: Send, +{ + std::thread::scope(|s| { + s.spawn(|| { + let runtime = tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build() + .map_err(|e| delta_kernel::Error::generic(e.to_string()))?; + runtime.block_on(make_fut()) + }) + .join() + .expect("block_on_sync thread panicked") + }) +} + +// =========================================================================== +// LogState +// =========================================================================== + +/// Format kernel uses when writing each checkpoint in a [`LogState`]. Mirrors +/// kernel's [`CheckpointSpec`], but with a soft-coupling fallback for +/// cross-product tests: requesting V2 sidecars on a table that doesn't have the +/// `v2Checkpoint` feature falls back to kernel's default rather than erroring. +#[derive(Clone, Debug, Default)] +pub enum CheckpointFormat { + /// Kernel picks V1 or V2-NoSidecar based on the `v2Checkpoint` feature flag. + #[default] + Default, + /// V2 with sidecar files. Honored only when `v2Checkpoint` is enabled; + /// silently falls back to [`CheckpointFormat::Default`] otherwise. + V2WithSidecarsIfEnabled { + /// Suggested file actions per sidecar; `None` uses the kernel default. + file_actions_per_sidecar_hint: Option, + }, +} + +/// State of the `_delta_log/_last_checkpoint` hint file on disk. +/// +/// Three states a kernel reader must handle: +/// +/// - [`Present`](Self::Present): hint file exists and points at the highest checkpoint on disk. +/// Default; what `Snapshot::checkpoint` writes. +/// - [`Missing`](Self::Missing): no hint file. Forces the reader's listing fallback to discover the +/// latest checkpoint. +/// - [`Stale`](Self::Stale): hint file exists but points at an OLDER real checkpoint, not the +/// latest. The reader follows the hint, lists from there, and picks up the actual latest +/// checkpoint by listing forward (logging an info on the version mismatch). +/// +/// `Stale` is only meaningful with at least two checkpoints in the [`LogState`] -- the hint +/// rewrites to point at the lowest checkpoint, leaving the highest checkpoint as the actual +/// "latest" the reader should discover. Pairing `Stale` with fewer than two checkpoints +/// panics at build time. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub enum LastCheckpointHintState { + #[default] + Present, + Missing, + Stale, +} + +impl fmt::Display for LastCheckpointHintState { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Present => write!(f, "present"), + Self::Missing => write!(f, "missing"), + Self::Stale => write!(f, "stale"), + } + } +} + +/// Shape of a Delta table's `_delta_log/` directory: commits, checkpoints, CRC +/// files, the `_last_checkpoint` hint, and log cleanup. +#[derive(Clone, Debug)] +pub struct LogState { + /// Latest version on the table. Versions `0..=latest_version` exist as + /// commits on disk (v=0 is the create-table commit); `latest_version + 1` + /// total commits. + latest_version: u64, + /// Sorted ascending, distinct, all `<= latest_version`. + checkpoints_at: Vec, + /// Format applied to every checkpoint in `checkpoints_at`. + checkpoint_format: CheckpointFormat, + /// Sorted ascending, distinct, all `<= latest_version`. A `.crc` file is + /// written at each listed version. + crcs_at: Vec, + /// If `Some(n)`, log files at versions `< n` are deleted after the table is built. + cleanup_before: Option, + /// State of the `_last_checkpoint` hint file. Defaults to `Present`. + last_checkpoint_hint: LastCheckpointHintState, +} + +impl LogState { + /// Build a table whose latest version is `n`. Produces `n + 1` commits + /// (v=0 create + v=1..=n data). No checkpoints by default. + pub fn with_latest_version(n: u64) -> Self { + Self { + latest_version: n, + checkpoints_at: Vec::new(), + checkpoint_format: CheckpointFormat::Default, + crcs_at: Vec::new(), + cleanup_before: None, + last_checkpoint_hint: LastCheckpointHintState::Present, + } + } + + /// Add checkpoints at the given versions. Pass `[v]` for a single + /// checkpoint or `[v1, v2, ...]` for multiple. Each `v` must be + /// `<= latest_version` and not already present. + /// + /// # Panics + /// + /// Panics if any `v > latest_version` or is already present. + pub fn with_checkpoint_at(mut self, vs: impl IntoIterator) -> Self { + for v in vs { + assert!( + v <= self.latest_version, + "checkpoint_at ({v}) must be <= latest_version ({})", + self.latest_version, + ); + assert!( + !self.checkpoints_at.contains(&v), + "checkpoint_at ({v}) already present in {:?}", + self.checkpoints_at, + ); + self.checkpoints_at.push(v); + } + self.checkpoints_at.sort_unstable(); + self + } + + /// Optionally add a checkpoint at the given version. No-op when `None`. + /// Useful for cartesian-product rstest axes parameterized as + /// `#[values(None, Some(v))]`. + pub fn maybe_with_checkpoint_at(self, v: Option) -> Self { + match v { + Some(v) => self.with_checkpoint_at([v]), + None => self, + } + } + + /// Add CRC files at the given versions. Pass `[v]` for a single CRC or + /// `[v1, v2, ...]` for multiple. Each `v` must be `<= latest_version` and + /// not already present. + /// + /// # Panics + /// + /// Panics if any `v > latest_version` or is already present. + pub fn with_crc_at(mut self, vs: impl IntoIterator) -> Self { + for v in vs { + assert!( + v <= self.latest_version, + "crc_at ({v}) must be <= latest_version ({})", + self.latest_version, + ); + assert!( + !self.crcs_at.contains(&v), + "crc_at ({v}) already present in {:?}", + self.crcs_at, + ); + self.crcs_at.push(v); + } + self.crcs_at.sort_unstable(); + self + } + + /// Switch every checkpoint to V2 with sidecars. Pass `None` for the kernel + /// default `file_actions_per_sidecar_hint`, or `Some(n)` to override. + /// Honored only when the paired [`FeatureSet`] enables `v2_checkpoint()`; + /// silently falls back to the default format otherwise so that + /// `#[values]` sweeps mixing both axes don't fail on invalid combinations. + pub fn with_sidecars_if_enabled( + mut self, + file_actions_per_sidecar_hint: Option, + ) -> Self { + self.checkpoint_format = CheckpointFormat::V2WithSidecarsIfEnabled { + file_actions_per_sidecar_hint, + }; + self + } + + /// Simulate log cleanup by deleting files at versions `< n`. Files at + /// `n..=latest_version` (including the JSON and checkpoint at `v=n`) survive. + /// Requires a checkpoint at `v=n` and `n` in `1..=latest_version`. + pub fn with_cleanup_commits_before(mut self, n: u64) -> Self { + self.cleanup_before = Some(n); + self + } + + /// Set the `_delta_log/_last_checkpoint` hint state. + /// + /// `Stale` requires at least two checkpoints in the LogState; the builder + /// asserts on the precondition at `build()` time. + pub fn with_last_checkpoint_hint(mut self, state: LastCheckpointHintState) -> Self { + self.last_checkpoint_hint = state; + self + } + + /// Latest version on the table. The total number of commits on disk is + /// `latest_version + 1` (or fewer if + /// [`with_cleanup_commits_before`](Self::with_cleanup_commits_before) removed earlier + /// versions). + pub fn latest_version(&self) -> u64 { + self.latest_version + } + + /// Versions at which checkpoints are written, in ascending order. + pub(crate) fn checkpoints_at(&self) -> &[u64] { + &self.checkpoints_at + } + + /// Format applied to every checkpoint. + pub(crate) fn checkpoint_format(&self) -> &CheckpointFormat { + &self.checkpoint_format + } + + /// Versions at which CRC files are written, in ascending order. + pub fn crcs_at(&self) -> &[u64] { + &self.crcs_at + } + + /// Version below which commits and checkpoints have been cleaned up, if any. + pub(crate) fn cleanup_before(&self) -> Option { + self.cleanup_before + } + + /// State of the `_last_checkpoint` hint file on the built table. + pub(crate) fn last_checkpoint_hint(&self) -> LastCheckpointHintState { + self.last_checkpoint_hint + } +} + +// Canonical sweep rows for the LogState axis. Test case names derive from these +// function names (e.g. `log_state_1_commits_only__`). + +pub fn commits_only() -> LogState { + LogState::with_latest_version(DEFAULT_SWEEP_LATEST_VERSION) +} + +pub fn checkpoint_at_end() -> LogState { + LogState::with_latest_version(DEFAULT_SWEEP_LATEST_VERSION) + .with_checkpoint_at([DEFAULT_SWEEP_LATEST_VERSION]) +} + +pub fn checkpoint_at_end_no_hint() -> LogState { + checkpoint_at_end().with_last_checkpoint_hint(LastCheckpointHintState::Missing) +} + +pub fn checkpoint_mid() -> LogState { + LogState::with_latest_version(DEFAULT_SWEEP_LATEST_VERSION) + .with_checkpoint_at([DEFAULT_SWEEP_MID_VERSION]) +} + +pub fn checkpoint_mid_no_hint() -> LogState { + checkpoint_mid().with_last_checkpoint_hint(LastCheckpointHintState::Missing) +} + +pub fn two_checkpoints_stale_hint() -> LogState { + LogState::with_latest_version(DEFAULT_SWEEP_LATEST_VERSION) + .with_checkpoint_at([DEFAULT_SWEEP_MID_VERSION, DEFAULT_SWEEP_LATEST_VERSION]) + .with_last_checkpoint_hint(LastCheckpointHintState::Stale) +} + +pub fn crc_at_end() -> LogState { + LogState::with_latest_version(DEFAULT_SWEEP_LATEST_VERSION) + .with_crc_at([DEFAULT_SWEEP_LATEST_VERSION]) +} + +pub fn crc_at_mid() -> LogState { + LogState::with_latest_version(DEFAULT_SWEEP_LATEST_VERSION) + .with_crc_at([DEFAULT_SWEEP_MID_VERSION]) +} + +pub fn checkpoint_at_end_crc_at_end() -> LogState { + LogState::with_latest_version(DEFAULT_SWEEP_LATEST_VERSION) + .with_checkpoint_at([DEFAULT_SWEEP_LATEST_VERSION]) + .with_crc_at([DEFAULT_SWEEP_LATEST_VERSION]) +} + +// Post-cleanup variants: same shapes as above but with log cleanup applied at MID. +// Cleanup at MID (not at LATEST) keeps commits MID..=LATEST reachable, so the canonical +// `at_version(MID)` and `incremental_to_latest { from: MID }` targets still resolve. + +pub fn checkpoint_at_end_post_cleanup() -> LogState { + LogState::with_latest_version(DEFAULT_SWEEP_LATEST_VERSION) + .with_checkpoint_at([DEFAULT_SWEEP_MID_VERSION, DEFAULT_SWEEP_LATEST_VERSION]) + .with_cleanup_commits_before(DEFAULT_SWEEP_MID_VERSION) +} + +pub fn checkpoint_at_end_no_hint_post_cleanup() -> LogState { + checkpoint_at_end_post_cleanup().with_last_checkpoint_hint(LastCheckpointHintState::Missing) +} + +pub fn checkpoint_mid_post_cleanup() -> LogState { + checkpoint_mid().with_cleanup_commits_before(DEFAULT_SWEEP_MID_VERSION) +} + +pub fn checkpoint_mid_no_hint_post_cleanup() -> LogState { + checkpoint_mid_no_hint().with_cleanup_commits_before(DEFAULT_SWEEP_MID_VERSION) +} + +pub fn two_checkpoints_stale_hint_post_cleanup() -> LogState { + two_checkpoints_stale_hint().with_cleanup_commits_before(DEFAULT_SWEEP_MID_VERSION) +} + +pub fn checkpoint_mid_crc_at_mid_post_cleanup() -> LogState { + checkpoint_mid_post_cleanup().with_crc_at([DEFAULT_SWEEP_MID_VERSION]) +} + +pub fn checkpoint_mid_crc_above_mid_post_cleanup() -> LogState { + // Version 8 lies strictly between MID=5 and LATEST=10. + checkpoint_mid_post_cleanup().with_crc_at([8]) +} + +pub fn checkpoint_mid_crc_at_end_post_cleanup() -> LogState { + checkpoint_mid_post_cleanup().with_crc_at([DEFAULT_SWEEP_LATEST_VERSION]) +} + +/// Extract the version from a versioned log file. Returns `None` for unversioned +/// files like `_last_checkpoint`. +fn log_file_version(location: &Path) -> Option { + let filename = location.filename()?; + let (prefix, _) = filename.split_once('.')?; + prefix.parse::().ok() +} + +async fn read_hint_bytes(store: &Arc, path: &Path) -> DeltaResult> { + let result = store.get(path).await.map_err(delta_kernel::Error::from)?; + let bytes = result.bytes().await.map_err(delta_kernel::Error::from)?; + Ok(bytes.to_vec()) +} + +/// Up-front validation so panics surface on the caller's thread, not through +/// `block_on_sync`'s worker-thread join boundary. +fn validate_log_state(log_state: &LogState) { + if let Some(n) = log_state.cleanup_before() { + assert!( + n >= 1, + "with_cleanup_commits_before(n) requires n >= 1 (cleanup_before(0) is a no-op)", + ); + assert!( + n <= log_state.latest_version(), + "with_cleanup_commits_before({n}) exceeds latest_version ({})", + log_state.latest_version(), + ); + assert!( + log_state.checkpoints_at().contains(&n), + "with_cleanup_commits_before({n}) requires a checkpoint at v={n}; \ + pair with `with_checkpoint_at([{n}, ...])`", + ); + } + if log_state.last_checkpoint_hint() == LastCheckpointHintState::Stale { + assert!( + log_state.checkpoints_at().len() >= 2, + "Stale hint requires at least 2 checkpoints (one to be stale relative to); \ + pair with `with_checkpoint_at` at two distinct versions", + ); + } +} + +impl fmt::Display for LogState { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "v={}", self.latest_version)?; + for v in &self.checkpoints_at { + write!(f, "+checkpoint_at({v})")?; + } + for v in &self.crcs_at { + write!(f, "+crc_at({v})")?; + } + if let Some(n) = self.cleanup_before { + write!(f, "+cleanup_before({n})")?; + } + if self.last_checkpoint_hint != LastCheckpointHintState::Present { + write!(f, "+hint({})", self.last_checkpoint_hint)?; + } + match self.checkpoint_format { + CheckpointFormat::V2WithSidecarsIfEnabled { + file_actions_per_sidecar_hint: Some(n), + } => write!(f, "+sidecars({n})")?, + CheckpointFormat::V2WithSidecarsIfEnabled { + file_actions_per_sidecar_hint: None, + } => write!(f, "+sidecars")?, + CheckpointFormat::Default => {} + } + Ok(()) + } +} + +// =========================================================================== +// FeatureSet +// =========================================================================== + +/// Which Delta table features to enable. Methods chain for composability. +/// +/// Stores table properties passed to the `CreateTable` API, which handles protocol +/// derivation, schema annotations (column mapping), and feature auto-enablement. +#[derive(Clone, Debug, Default)] +pub struct FeatureSet { + pub(crate) table_properties: Vec<(String, String)>, +} + +impl FeatureSet { + /// No features enabled. + pub fn new() -> Self { + Self::default() + } + + /// Alias for `new()`. Reads better when no features are intended. + pub fn empty() -> Self { + Self::new() + } + + // === Feature methods === + // + // Each method enables one table feature via its natural property: + // - Enablement property (e.g. delta.enableInCommitTimestamps=true) for features that have one + // - Feature signal (delta.feature.X=supported) for protocol-only features + // + // TimestampNtz and clustering are NOT FeatureSet methods -- they are driven by + // the schema (TimestampNtz columns auto-enable the feature) and DataLayoutConfig + // (clustering via with_data_layout) respectively. + + /// Set column mapping mode ("name" or "id"). Passing "none" is a no-op. + pub fn column_mapping(mut self, mode: &str) -> Self { + if mode != "none" { + self.table_properties + .push(("delta.columnMapping.mode".into(), mode.into())); + } + self + } + + pub fn ict(mut self) -> Self { + self.table_properties + .push(("delta.enableInCommitTimestamps".into(), "true".into())); + self + } + + pub fn v2_checkpoint(mut self) -> Self { + self.table_properties + .push(("delta.feature.v2Checkpoint".into(), "supported".into())); + self + } + + pub fn deletion_vectors(mut self) -> Self { + self.table_properties + .push(("delta.enableDeletionVectors".into(), "true".into())); + self + } + + pub fn append_only(mut self) -> Self { + self.table_properties + .push(("delta.appendOnly".into(), "true".into())); + self + } + + pub fn change_data_feed(mut self) -> Self { + self.table_properties + .push(("delta.enableChangeDataFeed".into(), "true".into())); + self + } + + pub fn type_widening(mut self) -> Self { + self.table_properties + .push(("delta.enableTypeWidening".into(), "true".into())); + self + } + + pub fn domain_metadata(mut self) -> Self { + self.table_properties + .push(("delta.feature.domainMetadata".into(), "supported".into())); + self + } + + pub fn vacuum_protocol_check(mut self) -> Self { + self.table_properties.push(( + "delta.feature.vacuumProtocolCheck".into(), + "supported".into(), + )); + self + } + + pub fn row_tracking(mut self) -> Self { + self.table_properties + .push(("delta.enableRowTracking".into(), "true".into())); + self + } + + /// Set an arbitrary table property. Useful for properties that don't have a + /// dedicated method. + pub fn with_property(mut self, key: &str, value: &str) -> Self { + self.table_properties.push((key.into(), value.into())); + self + } + + /// Returns the table features implied by the properties in this set. Used by tests + /// to check that each builder method actually enables the right feature. + pub fn expected_features(&self) -> Vec { + let mut out = Vec::new(); + for (k, _) in &self.table_properties { + match k.as_str() { + "delta.columnMapping.mode" => out.push(TableFeature::ColumnMapping), + "delta.enableInCommitTimestamps" => out.push(TableFeature::InCommitTimestamp), + "delta.feature.v2Checkpoint" => out.push(TableFeature::V2Checkpoint), + "delta.enableDeletionVectors" => out.push(TableFeature::DeletionVectors), + "delta.appendOnly" => out.push(TableFeature::AppendOnly), + "delta.enableChangeDataFeed" => out.push(TableFeature::ChangeDataFeed), + "delta.enableTypeWidening" => out.push(TableFeature::TypeWidening), + "delta.feature.domainMetadata" => out.push(TableFeature::DomainMetadata), + "delta.feature.vacuumProtocolCheck" => out.push(TableFeature::VacuumProtocolCheck), + "delta.enableRowTracking" => { + out.push(TableFeature::RowTracking); + out.push(TableFeature::DomainMetadata); // row tracking depends on DM + } + _ => {} + } + } + out + } +} + +impl fmt::Display for FeatureSet { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + if self.table_properties.is_empty() { + return write!(f, "empty"); + } + let props: Vec<_> = self + .table_properties + .iter() + .map(|(k, v)| { + // Strip common prefixes for readability + let short_key = k + .strip_prefix("delta.feature.") + .or_else(|| k.strip_prefix("delta.")) + .unwrap_or(k); + format!("{short_key}={v}") + }) + .collect(); + write!(f, "{}", props.join(", ")) + } +} + +// Canonical sweep rows for the FeatureSet axis. + +pub fn no_features() -> FeatureSet { + FeatureSet::empty() +} + +pub fn all_features_cm_id() -> FeatureSet { + all_features_base().column_mapping("id") +} + +pub fn all_features_cm_name() -> FeatureSet { + all_features_base().column_mapping("name") +} + +fn all_features_base() -> FeatureSet { + FeatureSet::new() + .deletion_vectors() + .row_tracking() + .domain_metadata() + .ict() + .v2_checkpoint() + .vacuum_protocol_check() + .change_data_feed() + .append_only() +} + +// =========================================================================== +// TableConfig +// =========================================================================== + +/// Table configuration properties that are orthogonal to table features. +/// +/// These are pure runtime knobs (stats format, checkpoint interval, file sizing, +/// etc.) that don't affect the protocol or enable features. Crossed with +/// [`DataLayoutConfig`] in the sweep so layout shape and write-time properties +/// vary independently. +#[derive(Clone, Debug, Default, PartialEq)] +pub struct TableConfig { + pub(crate) table_properties: Vec<(String, String)>, +} + +impl TableConfig { + /// Default table configuration (no properties set). + pub fn new() -> Self { + Self::default() + } + + /// Set `delta.checkpoint.writeStatsAsJson`. + pub fn write_stats_as_json(mut self, enabled: bool) -> Self { + self.table_properties.push(( + "delta.checkpoint.writeStatsAsJson".into(), + enabled.to_string(), + )); + self + } + + /// Set `delta.checkpoint.writeStatsAsStruct`. + pub fn write_stats_as_struct(mut self, enabled: bool) -> Self { + self.table_properties.push(( + "delta.checkpoint.writeStatsAsStruct".into(), + enabled.to_string(), + )); + self + } + + /// Set `delta.dataSkippingNumIndexedCols` to `n`: stats are capped to the first `n` leaf + /// columns. `-1` means all columns; the protocol default when unset is 32. + pub fn num_data_skipping_indexed_cols(mut self, n: i64) -> Self { + self.table_properties + .push(("delta.dataSkippingNumIndexedCols".into(), n.to_string())); + self + } + + /// Set `delta.dataSkippingStatsColumns` to a comma-separated list of column names. + /// Per the Delta spec this takes precedence over `dataSkippingNumIndexedCols`. The + /// builder joins names verbatim and does not validate them; entries that don't + /// resolve in the active schema are silently skipped by kernel (with a warning). + pub fn data_skipping_stats_columns(mut self, columns: I) -> Self + where + I: IntoIterator, + S: Into, + { + let joined = columns + .into_iter() + .map(Into::into) + .collect::>() + .join(","); + self.table_properties + .push(("delta.dataSkippingStatsColumns".into(), joined)); + self + } +} + +impl fmt::Display for TableConfig { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + if self.table_properties.is_empty() { + return write!(f, "default"); + } + let props: Vec<_> = self + .table_properties + .iter() + .map(|(k, v)| { + let short_key = k + .strip_prefix("delta.checkpoint.") + .or_else(|| k.strip_prefix("delta.")) + .unwrap_or(k); + format!("{short_key}={v}") + }) + .collect(); + write!(f, "{}", props.join(", ")) + } +} + +// Canonical sweep rows for the `TableConfig` axis. The sweep covers three checkpoint +// stats encodings plus six data-skipping rows: three count-based (`numIndexedCols`) +// values and three named-columns (`dataSkippingStatsColumns`) shapes, so each knob +// varies across the full LogState x FeatureSet x DataLayout x VersionTarget cross +// product. Per-commit add-file stats are always written and unaffected by the +// checkpoint format flags. + +/// Stats-column names that span the schemas used by [`DataLayoutConfig`]: `int_col` +/// exists in [`default_schema`], `value` in [`partitioned_schema`] and +/// [`clustered_schema`]. Kernel silently skips entries that don't exist in the active +/// schema, so each layout collects stats on whichever entry matches. +pub(crate) const STATS_COLUMNS_BY_NAME: &[&str] = &["int_col", "value"]; + +// === Checkpoint stats encoding rows === + +pub fn checkpoint_json_stats() -> TableConfig { + TableConfig::new() + .write_stats_as_json(true) + .write_stats_as_struct(false) +} + +pub fn checkpoint_struct_stats() -> TableConfig { + TableConfig::new() + .write_stats_as_json(false) + .write_stats_as_struct(true) +} + +/// Disables both checkpoint stats encodings; per-commit add-file stats are still written. +pub fn no_checkpoint_stats() -> TableConfig { + TableConfig::new() + .write_stats_as_json(false) + .write_stats_as_struct(false) +} + +// === numIndexedCols rows (count-based data skipping) === + +/// `numIndexedCols = 0`: zero leaf columns get stats. +pub fn num_indexed_cols_zero() -> TableConfig { + TableConfig::new().num_data_skipping_indexed_cols(0) +} + +/// `numIndexedCols = 2`: cap stats to the first 2 leaf columns. +pub fn num_indexed_cols_narrow() -> TableConfig { + TableConfig::new().num_data_skipping_indexed_cols(2) +} + +/// `numIndexedCols = -1`: stats on every leaf column. +pub fn num_indexed_cols_all() -> TableConfig { + TableConfig::new().num_data_skipping_indexed_cols(-1) +} + +// === dataSkippingStatsColumns rows (named-cols data skipping) === + +/// `dataSkippingStatsColumns = []`: named-cols knob set, no columns listed. +pub fn stats_columns_empty() -> TableConfig { + TableConfig::new().data_skipping_stats_columns(std::iter::empty::<&str>()) +} + +/// `dataSkippingStatsColumns` listing names that match at least one column in every +/// `DataLayoutConfig` schema (kernel silently skips entries that don't match), in reverse +/// schema order to exercise list-order preservation through write + read. +pub fn stats_columns_reordered() -> TableConfig { + TableConfig::new().data_skipping_stats_columns(STATS_COLUMNS_BY_NAME.iter().rev().copied()) +} + +// === Composers: enable a checkpoint stats encoding on a data-skipping config === +// Each enables only its own encoding, so composing both writes stats in both encodings. + +pub fn with_json_stats(cfg: TableConfig) -> TableConfig { + cfg.write_stats_as_json(true) +} + +pub fn with_struct_stats(cfg: TableConfig) -> TableConfig { + cfg.write_stats_as_struct(true) +} + +// =========================================================================== +// rstest_reuse templates +// =========================================================================== + +// Standard `#[values]` lists for cross-product testing. Apply with `#[apply(template_name)]` +// on an rstest test function. Each template injects one parameter; combine multiple templates +// to build a cross-product. Tests that only need a subset should use inline `#[values]` instead. +// +// Example: +// ```ignore +// use rstest::rstest; +// use rstest_reuse::apply; +// use test_utils::table_builder::*; +// +// #[apply(feature_sets)] +// #[apply(table_configs)] +// fn test_scan(feature_set: FeatureSet, table_config: TableConfig) { ... } +// ``` + +/// Empty + one per write-compatible feature + all combined. `type_widening` is +/// excluded because kernel errors when writing tables with that feature enabled. +#[rstest_reuse::template] +#[rstest::rstest] +pub fn feature_sets( + #[values( + FeatureSet::empty(), + FeatureSet::new().column_mapping("name"), + FeatureSet::new().ict(), + FeatureSet::new().v2_checkpoint(), + FeatureSet::new().deletion_vectors(), + FeatureSet::new().append_only(), + FeatureSet::new().change_data_feed(), + FeatureSet::new().domain_metadata(), + FeatureSet::new().vacuum_protocol_check(), + FeatureSet::new().row_tracking(), + FeatureSet::new() + .column_mapping("name") + .ict() + .v2_checkpoint() + .deletion_vectors() + .append_only() + .change_data_feed() + .domain_metadata() + .vacuum_protocol_check() + .row_tracking() + )] + feature_set: FeatureSet, +) { +} + +/// All common table configs: default plus four stats combos. +#[rstest_reuse::template] +#[rstest::rstest] +pub fn table_configs( + #[values( + TableConfig::new(), + TableConfig::new().write_stats_as_json(true).write_stats_as_struct(false), + TableConfig::new().write_stats_as_json(false).write_stats_as_struct(true), + TableConfig::new().write_stats_as_json(true).write_stats_as_struct(true), + TableConfig::new().write_stats_as_json(false).write_stats_as_struct(false) + )] + table_config: TableConfig, +) { +} + +// =========================================================================== +// DataLayoutConfig +// =========================================================================== + +/// Data layout configuration for cross-product testing. +/// +/// Describes only the partitioning/clustering shape of the table. Write-time table +/// properties (stats format, etc.) live in [`TableConfig`], which is crossed with +/// this axis independently. +/// +/// Designed for rstest `#[values]` parameterization alongside [`LogState`], +/// [`FeatureSet`], and [`TableConfig`]. +#[derive(Clone, Debug, PartialEq)] +pub enum DataLayoutConfig { + /// No partitioning or clustering. Uses an all-primitives schema including a + /// nested-struct column. + Unpartitioned, + /// Partition by every valid primitive type. Uses [`partitioned_schema`] with all columns + /// as partition columns. + PartitionedAllTypes, + /// Cluster by every stats-eligible primitive type. Uses [`clustered_schema`] with all + /// clustering-eligible columns. Boolean and Binary are excluded (not stats-eligible). + ClusteredAllTypes, +} + +/// Period for sparse null injection. Data generation emits a null every +/// `NULL_RATE_EVERY_NTH` rows for each nullable, non-partition column, regardless of +/// [`DataLayoutConfig`]. Clustering columns are nulled like any other data column (kernel +/// permits null clustering keys). Partition columns are left fully populated: the protocol +/// permits null partition values, but the generator does not model them, so partition data +/// stays well-defined and matches the declared partition value. +pub const NULL_RATE_EVERY_NTH: usize = 3; + +impl DataLayoutConfig { + /// The layout column names (partition or clustering) for this config. Returns all + /// schema columns except the `"value"` data column. + pub fn columns(&self) -> Vec { + let schema = match self { + DataLayoutConfig::Unpartitioned => return vec![], + DataLayoutConfig::PartitionedAllTypes => partitioned_schema(), + DataLayoutConfig::ClusteredAllTypes => clustered_schema(), + }; + schema + .fields() + .filter(|f| f.name() != "value") + .map(|f| f.name().to_string()) + .collect() + } + + /// The schema for this config. + pub fn schema(&self) -> SchemaRef { + match self { + DataLayoutConfig::Unpartitioned => default_schema(), + DataLayoutConfig::PartitionedAllTypes => partitioned_schema(), + DataLayoutConfig::ClusteredAllTypes => clustered_schema(), + } + } + + /// Whether this config uses partitioning. + pub fn is_partitioned(&self) -> bool { + self == &DataLayoutConfig::PartitionedAllTypes + } + + /// Whether this config uses clustering. + pub fn is_clustered(&self) -> bool { + self == &DataLayoutConfig::ClusteredAllTypes + } +} + +impl fmt::Display for DataLayoutConfig { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + DataLayoutConfig::Unpartitioned => write!(f, "unpartitioned"), + DataLayoutConfig::PartitionedAllTypes => write!(f, "partitioned(all_types)"), + DataLayoutConfig::ClusteredAllTypes => write!(f, "clustered(all_types)"), + } + } +} + +/// Schema with all partition-valid primitive types. Use with [`DataLayoutConfig`] to select +/// which columns are partition columns. Includes `TimestampNtz` which auto-enables the +/// `timestampNtz` table feature. +/// +/// All columns are nullable. Follow-up: add non-nullable variants to test NOT NULL handling. +pub fn partitioned_schema() -> SchemaRef { + Arc::new(StructType::new_unchecked(vec![ + // Partition-candidate columns (all valid partition types, matches write::partitioned) + StructField::new("part_bool", DataType::BOOLEAN, true), + StructField::new("part_byte", DataType::BYTE, true), + StructField::new("part_short", DataType::SHORT, true), + StructField::new("part_int", DataType::INTEGER, true), + StructField::new("part_long", DataType::LONG, true), + StructField::new("part_float", DataType::FLOAT, true), + StructField::new("part_double", DataType::DOUBLE, true), + StructField::new("part_string", DataType::STRING, true), + StructField::new("part_binary", DataType::BINARY, true), + StructField::new("part_date", DataType::DATE, true), + StructField::new("part_ts", DataType::TIMESTAMP, true), + StructField::new("part_ts_ntz", DataType::TIMESTAMP_NTZ, true), + StructField::new("part_decimal", DataType::decimal(10, 2).unwrap(), true), + // Non-partition data column (required: at least one non-partition column) + StructField::new("value", DataType::INTEGER, true), + ])) +} + +/// Schema with all stats-eligible primitive types for clustering. Boolean and Binary are +/// excluded (not stats-eligible). Includes `TimestampNtz` which auto-enables the +/// `timestampNtz` table feature. +/// +/// All columns are nullable. Follow-up: add non-nullable variants to test NOT NULL handling. +pub fn clustered_schema() -> SchemaRef { + Arc::new(StructType::new_unchecked(vec![ + // Clustering-eligible columns (stats-eligible primitive types) + StructField::new("clust_byte", DataType::BYTE, true), + StructField::new("clust_short", DataType::SHORT, true), + StructField::new("clust_int", DataType::INTEGER, true), + StructField::new("clust_long", DataType::LONG, true), + StructField::new("clust_float", DataType::FLOAT, true), + StructField::new("clust_double", DataType::DOUBLE, true), + StructField::new("clust_string", DataType::STRING, true), + StructField::new("clust_date", DataType::DATE, true), + StructField::new("clust_ts", DataType::TIMESTAMP, true), + StructField::new("clust_ts_ntz", DataType::TIMESTAMP_NTZ, true), + StructField::new("clust_decimal", DataType::decimal(10, 2).unwrap(), true), + // Non-clustering data column + StructField::new("value", DataType::INTEGER, true), + ])) +} + +// Canonical sweep rows for the DataLayoutConfig axis. + +pub fn unpartitioned() -> DataLayoutConfig { + DataLayoutConfig::Unpartitioned +} + +pub fn partitioned() -> DataLayoutConfig { + DataLayoutConfig::PartitionedAllTypes +} + +pub fn clustered() -> DataLayoutConfig { + DataLayoutConfig::ClusteredAllTypes +} + +// =========================================================================== +// VersionTarget +// =========================================================================== + +/// How the snapshot should be loaded from the built table. +/// +/// Designed for use as an rstest `#[values]` parameter. Use the `build_snapshot!` macro +/// or `test_context!` macro to load a snapshot according to this target. +#[derive(Clone, Debug)] +pub enum VersionTarget { + /// Load the latest version. + Latest, + /// Time travel to a specific version. + AtVersion(u64), + /// Load at `from`, then incrementally update to latest. + IncrementalToLatest { from: u64 }, + /// Load at `from`, then incrementally update to a specific target `to`. Exercises the + /// `Snapshot::builder_from(base).at_version(to)` path with `to <= latest`. Requires + /// `from <= to`. + IncrementalFrom { from: u64, to: u64 }, + /// Time travel by timestamp: resolves `timestamp` (milliseconds since Unix epoch) to a + /// version via [`history_manager::latest_version_as_of`], then loads that version. + /// + /// [`history_manager::latest_version_as_of`]: delta_kernel::history_manager::latest_version_as_of + AtTimestamp(i64), +} + +impl fmt::Display for VersionTarget { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + VersionTarget::Latest => write!(f, "latest"), + VersionTarget::AtVersion(v) => write!(f, "at_version({v})"), + VersionTarget::IncrementalToLatest { from } => { + write!(f, "incremental({from}->latest)") + } + VersionTarget::IncrementalFrom { from, to } => { + write!(f, "incremental({from}->{to})") + } + VersionTarget::AtTimestamp(ts) => write!(f, "at_timestamp({ts})"), + } + } +} + +// Canonical sweep rows for the VersionTarget axis. + +pub fn version_latest() -> VersionTarget { + VersionTarget::Latest +} +pub fn version_at_mid() -> VersionTarget { + VersionTarget::AtVersion(DEFAULT_SWEEP_MID_VERSION) +} +pub fn version_incremental_from_mid_to_latest() -> VersionTarget { + VersionTarget::IncrementalToLatest { + from: DEFAULT_SWEEP_MID_VERSION, + } +} +/// Incremental update from `mid` to `latest - 1`. The non-latest `to` exercises the +/// partial-replay path that `version_incremental_from_mid_to_latest()` cannot reach, since +/// updating to latest is indistinguishable from a non-incremental load. +pub fn version_incremental_from_mid_to_pre_latest() -> VersionTarget { + VersionTarget::IncrementalFrom { + from: DEFAULT_SWEEP_MID_VERSION, + to: DEFAULT_SWEEP_LATEST_VERSION - 1, + } +} +/// Timestamp travel using `i64::MAX`, which always resolves to the latest version (every +/// commit's timestamp is below `i64::MAX`). This is a smoke test that the timestamp +/// conversion path runs without error across the sweep, not a resolution-correctness +/// check: `InMemory` collapses successive `put` timestamps to a single millisecond, so the +/// sweep can't reach an intermediate version. Resolution to an intermediate version is +/// covered by `test_at_timestamp_resolves_to_intermediate_version`, which sets commit +/// modification times explicitly on the local filesystem. +pub fn version_at_timestamp_max() -> VersionTarget { + VersionTarget::AtTimestamp(i64::MAX) +} + +// =========================================================================== +// TestTableBuilder +// =========================================================================== + +/// Builds an in-memory Delta table with the requested configuration. +/// +/// Uses kernel's full write path (CreateTable, Transaction) to produce correct tables +/// with proper protocol handling. The builder does NOT expose a snapshot -- the test +/// loads one using its own kernel types. +pub struct TestTableBuilder { + log_state: LogState, + features: FeatureSet, + table_config: TableConfig, + schema: SchemaRef, + partition_columns: Vec, + clustering_columns: Vec, + num_data_files: usize, + rows_per_file: usize, +} + +impl Default for TestTableBuilder { + fn default() -> Self { + Self::new() + } +} + +impl TestTableBuilder { + /// Create a builder with sensible defaults: 1 commit, no features, 1 data file + /// with 10 rows per commit. + pub fn new() -> Self { + Self { + log_state: LogState::with_latest_version(0), + features: FeatureSet::empty(), + table_config: TableConfig::new(), + schema: default_schema(), + partition_columns: Vec::new(), + clustering_columns: Vec::new(), + num_data_files: 1, + rows_per_file: 10, + } + } + + /// Set the log state (what files exist on disk). + pub fn with_log_state(mut self, s: LogState) -> Self { + self.log_state = s; + self + } + + /// Set the table features. + pub fn with_features(mut self, f: FeatureSet) -> Self { + self.features = f; + self + } + + /// Set table configuration properties (orthogonal to features). + pub fn with_table_config(mut self, c: TableConfig) -> Self { + self.table_config = c; + self + } + + /// Override the default schema. + pub fn with_schema(mut self, s: SchemaRef) -> Self { + self.schema = s; + self + } + + /// Override the number of parquet data files per commit and rows per file. Defaults + /// are 1 file with 10 rows -- most tests don't need to call this. + pub fn with_data(mut self, files_per_commit: usize, rows_per_file: usize) -> Self { + self.num_data_files = files_per_commit; + self.rows_per_file = rows_per_file; + self + } + + /// Set partition columns by logical name. For standard layouts, prefer + /// [`with_data_layout`](Self::with_data_layout) which sets schema and columns together. + /// Use this directly only when you need a custom schema with specific partition columns. + /// + /// Each data file gets deterministic partition values derived from version and file index. + /// Clears any previously set clustering columns (partitioning and clustering are mutually + /// exclusive). + pub fn with_partition_columns( + mut self, + cols: impl IntoIterator>, + ) -> Self { + self.partition_columns = cols.into_iter().map(Into::into).collect(); + self.clustering_columns.clear(); + self + } + + /// Set clustering columns by logical name. For standard layouts, prefer + /// [`with_data_layout`](Self::with_data_layout) which sets schema and columns together. + /// Use this directly only when you need a custom schema with specific clustering columns. + /// + /// The columns must exist in the schema and have stats-eligible types. Clears any + /// previously set partition columns (partitioning and clustering are mutually exclusive). + pub fn with_clustering_columns( + mut self, + cols: impl IntoIterator>, + ) -> Self { + self.clustering_columns = cols.into_iter().map(Into::into).collect(); + self.partition_columns.clear(); + self + } + + /// Apply a [`DataLayoutConfig`], setting the schema and layout columns. Pair with + /// [`with_table_config`](Self::with_table_config) to also set write-time properties + /// (e.g. stats format); the two axes are independent. + /// + /// For `Unpartitioned`, leaves the schema and columns unchanged. + pub fn with_data_layout(self, config: DataLayoutConfig) -> Self { + let cols = config.columns(); + if cols.is_empty() { + return self; + } + let builder = self.with_schema(config.schema()); + if config.is_clustered() { + builder.with_clustering_columns(cols) + } else { + builder.with_partition_columns(cols) + } + } + + /// Build the table and return a [`TestTable`] handle to the store. + /// + /// Safe to call from both sync tests and `#[tokio::test]` -- uses a dedicated + /// runtime on a background thread to avoid panicking on nested runtimes. + /// Propagates I/O, create-table, commit, checkpoint, and CRC errors from the + /// underlying write path. + /// + /// # Panics + /// Panics if [`LastCheckpointHintState::Stale`] is paired with fewer than two + /// checkpoints. + pub fn build(self) -> DeltaResult { + validate_log_state(&self.log_state); + block_on_sync(|| self.build_async()) + } + + async fn build_async(self) -> DeltaResult { + let store: Arc = Arc::new(InMemory::new()); + let table_root = "memory:///"; + let executor = Arc::new(TokioMultiThreadExecutor::new( + tokio::runtime::Handle::current(), + )); + let engine = Arc::new( + DefaultEngineBuilder::new(store.clone()) + .with_task_executor(executor) + .build(), + ); + let schema = self.schema; + + // Resolve the checkpoint spec once. `V2WithSidecarsIfEnabled` falls back to + // `None` (kernel default: V1 or V2-NoSidecar based on the feature flag) when + // the table doesn't enable `v2Checkpoint`. + let v2_supported = self + .features + .expected_features() + .contains(&TableFeature::V2Checkpoint); + let spec = match self.log_state.checkpoint_format() { + CheckpointFormat::V2WithSidecarsIfEnabled { + file_actions_per_sidecar_hint, + } if v2_supported => Some(CheckpointSpec::V2(V2CheckpointConfig::WithSidecar { + file_actions_per_sidecar_hint: *file_actions_per_sidecar_hint, + })), + _ => None, + }; + let checkpoints_at: HashSet = + self.log_state.checkpoints_at().iter().copied().collect(); + + // Version 0: CreateTable + let mut builder = create_table(table_root, schema, "TestTableBuilder/1.0"); + let all_properties: Vec<_> = self + .features + .table_properties + .iter() + .chain(self.table_config.table_properties.iter()) + .collect(); + if !all_properties.is_empty() { + builder = builder.with_table_properties( + all_properties.iter().map(|(k, v)| (k.as_str(), v.as_str())), + ); + } + if !self.partition_columns.is_empty() { + builder = builder.with_data_layout(DataLayout::partitioned( + self.partition_columns.iter().map(|s| s.as_str()), + )); + } else if !self.clustering_columns.is_empty() { + builder = builder.with_data_layout(DataLayout::clustered( + self.clustering_columns.iter().map(|s| s.as_str()), + )); + } + let hint_state = self.log_state.last_checkpoint_hint(); + let hint_path = Path::from("_delta_log/_last_checkpoint"); + let resolved_hint_path = crate::resolve_table_path(table_root, &hint_path)?; + // For Stale, capture the hint bytes kernel writes after the lowest checkpoint + // and restore them at the end. + let mut stale_hint_bytes: Option> = None; + + let mut snapshot = builder + .build(engine.as_ref(), Box::new(FileSystemCommitter::new()))? + .commit(engine.as_ref())? + .unwrap_post_commit_snapshot(); + + let crcs_at = self.log_state.crcs_at(); + if crcs_at.contains(&0) { + write_crc(&snapshot, engine.as_ref())?; + } + if checkpoints_at.contains(&0) { + snapshot.checkpoint(engine.as_ref(), spec.as_ref())?; + if hint_state == LastCheckpointHintState::Stale && stale_hint_bytes.is_none() { + stale_hint_bytes = Some(read_hint_bytes(&store, &resolved_hint_path).await?); + } + } + + // Data commits (versions 1..=latest). Checkpoint inline using the post-commit snapshot. + let latest = self.log_state.latest_version(); + for v in 1..=latest { + snapshot = write_data_commit( + snapshot.clone(), + &engine, + self.num_data_files, + self.rows_per_file, + &self.partition_columns, + v, + ) + .await? + .unwrap_post_commit_snapshot(); + if crcs_at.contains(&v) { + write_crc(&snapshot, engine.as_ref())?; + } + if checkpoints_at.contains(&v) { + snapshot.checkpoint(engine.as_ref(), spec.as_ref())?; + if hint_state == LastCheckpointHintState::Stale && stale_hint_bytes.is_none() { + stale_hint_bytes = Some(read_hint_bytes(&store, &resolved_hint_path).await?); + } + } + } + + match hint_state { + LastCheckpointHintState::Present => {} + LastCheckpointHintState::Missing => match store.delete(&resolved_hint_path).await { + Ok(()) | Err(ObjectStoreError::NotFound { .. }) => {} + Err(e) => return Err(delta_kernel::Error::from(e)), + }, + LastCheckpointHintState::Stale => { + // Restore the hint bytes captured after the lowest checkpoint write. + let bytes = stale_hint_bytes + .expect("validate_log_state should have caught Stale + 0 checkpoints"); + store + .put(&resolved_hint_path, bytes.into()) + .await + .map_err(delta_kernel::Error::from)?; + } + } + + // Simulate log cleanup by deleting versioned log files at v < n. Unversioned + // files like `_last_checkpoint` are skipped. + if let Some(n) = self.log_state.cleanup_before() { + let log_dir = Path::from("_delta_log"); + let resolved_log_dir = crate::resolve_table_path(table_root, &log_dir)?; + let listing = store + .list_with_delimiter(Some(&resolved_log_dir)) + .await + .map_err(delta_kernel::Error::from)?; + for object in listing.objects { + if matches!(log_file_version(&object.location), Some(v) if v < n) { + store + .delete(&object.location) + .await + .map_err(delta_kernel::Error::from)?; + } + } + } + + Ok(TestTable { + store, + table_root: table_root.to_string(), + description: if self.table_config.table_properties.is_empty() { + format!("{} + {}", self.log_state, self.features) + } else { + format!( + "{} + {} + config({})", + self.log_state, self.features, self.table_config + ) + }, + }) + } +} + +// =========================================================================== +// Data commit via kernel write path +// =========================================================================== + +/// Write a CRC file via kernel's checksum writer. Builder invariant: the snapshot +/// comes from a post-commit handoff on a fresh in-memory table, so the CRC for +/// that version cannot already exist on disk. +fn write_crc(snapshot: &Arc, engine: &dyn Engine) -> DeltaResult<()> { + let (result, _) = snapshot.write_checksum(engine)?; + assert_eq!( + result, + ChecksumWriteResult::Written, + "fresh in-memory table should never have a CRC on disk at this version", + ); + Ok(()) +} + +/// Write a data commit using kernel's transaction + write_parquet path. +/// Produces `num_files` parquet files with `rows_per_file` rows each. For partitioned +/// tables, all rows in a file share the same partition values; for unpartitioned or +/// clustered tables, uses `unpartitioned_write_context`. Non-partition columns get +/// varying data derived from version and file index. Partition columns are never nulled +/// so their data matches the declared partition value; all other nullable columns +/// (including clustering columns, which kernel permits to be null) get sparse nulls. +async fn write_data_commit( + snapshot: Arc, + engine: &DefaultEngine, + num_files: usize, + rows_per_file: usize, + partition_columns: &[String], + version: u64, +) -> DeltaResult { + let logical_schema = snapshot.schema().clone(); + let arrow_schema: ArrowSchema = TryFromKernel::try_from_kernel(logical_schema.as_ref()) + .map_err(|e| delta_kernel::Error::generic(e.to_string()))?; + + let mut txn = snapshot + .transaction(Box::new(FileSystemCommitter::new()), engine)? + .with_operation("WRITE".to_string()) + .with_data_change(true); + + let partition_set: HashSet<&str> = partition_columns.iter().map(String::as_str).collect(); + + for file_idx in 0..num_files { + let base = (version as i32 * 1000) + (file_idx as i32 * 100); + let partition_seed = (version as usize) * 1000 + file_idx * 100; + // Data batches must not contain any partition columns. + let mut data_fields = Vec::new(); + let mut columns: Vec = Vec::new(); + for (arrow_field, kernel_field) in arrow_schema.fields().iter().zip(logical_schema.fields()) + { + if partition_set.contains(kernel_field.name().as_str()) { + continue; + } + let data_type = arrow_field.data_type(); + let values = generate_column(data_type, rows_per_file, base); + let values = if arrow_field.is_nullable() { + with_sparse_nulls(values, NULL_RATE_EVERY_NTH) + } else { + values + }; + columns.push(values); + data_fields.push(arrow_field.clone()); + } + let data_arrow_schema = ArrowSchema::new(data_fields); + let batch = RecordBatch::try_new(Arc::new(data_arrow_schema), columns) + .map_err(|e| delta_kernel::Error::generic(e.to_string()))?; + + let write_context = if partition_columns.is_empty() { + txn.unpartitioned_write_context()? + } else { + let partition_values = generate_partition_values( + logical_schema.as_ref(), + partition_columns, + partition_seed, + ); + txn.partitioned_write_context(partition_values)? + }; + + let add_files = engine + .write_parquet(&ArrowEngineData::new(batch), &write_context) + .await?; + txn.add_files(add_files); + } + + txn.commit(engine) +} + +/// Generate a single column of data based on its Arrow type. +/// Data is deterministic: values are derived from `base` (version * 1000 + file_idx * 100). +fn generate_column(arrow_type: &ArrowDataType, rows: usize, base: i32) -> ArrayRef { + match arrow_type { + #[allow(unknown_lints, clippy::manual_is_multiple_of)] + ArrowDataType::Boolean => { + let values: Vec = (0..rows).map(|i| (base as usize + i) % 2 == 0).collect(); + Arc::new(BooleanArray::from(values)) + } + ArrowDataType::Int8 => { + let values: Vec = (0..rows).map(|i| ((base + i as i32) % 120) as i8).collect(); + Arc::new(Int8Array::from(values)) + } + ArrowDataType::Int16 => { + let values: Vec = (0..rows) + .map(|i| ((base + i as i32) % 30000) as i16) + .collect(); + Arc::new(Int16Array::from(values)) + } + ArrowDataType::Int32 => { + let values: Vec = (0..rows).map(|i| base + i as i32).collect(); + Arc::new(Int32Array::from(values)) + } + ArrowDataType::Int64 => { + let values: Vec = (0..rows).map(|i| (base + i as i32) as i64 * 1000).collect(); + Arc::new(Int64Array::from(values)) + } + ArrowDataType::Float32 => { + let values: Vec = (0..rows).map(|i| base as f32 + i as f32 * 0.5).collect(); + Arc::new(Float32Array::from(values)) + } + ArrowDataType::Float64 => { + let values: Vec = (0..rows).map(|i| base as f64 + i as f64 * 0.25).collect(); + Arc::new(Float64Array::from(values)) + } + ArrowDataType::Utf8 => { + let values: Vec = (0..rows) + .map(|i| format!("val_{}", base + i as i32)) + .collect(); + Arc::new(StringArray::from(values)) + } + ArrowDataType::Binary => { + let values: Vec> = (0..rows) + .map(|i| format!("bin_{}", base + i as i32).into_bytes()) + .collect(); + Arc::new(BinaryArray::from( + values.iter().map(|v| v.as_slice()).collect::>(), + )) + } + ArrowDataType::Date32 => { + let values: Vec = (0..rows).map(|i| 18000 + base + i as i32).collect(); + Arc::new(Date32Array::from(values)) + } + ArrowDataType::Timestamp(TimeUnit::Microsecond, tz) => { + // The sub-millisecond component keeps stats-formatting tests honest: a whole-second + // value cannot distinguish a three-digit fraction from an elided one. + let values: Vec = (0..rows) + .map(|i| (18000 + base + i as i32) as i64 * 86_400_000_000 + 298_677) + .collect(); + let array = TimestampMicrosecondArray::from(values); + match tz { + Some(tz) => Arc::new(array.with_timezone(tz.as_ref())), + None => Arc::new(array), + } + } + ArrowDataType::Decimal128(precision, scale) => { + let scale_factor = 10i128.pow(*scale as u32); + let values: Vec = (0..rows) + .map(|i| (base + i as i32) as i128 * scale_factor) + .collect(); + Arc::new( + Decimal128Array::from(values) + .with_precision_and_scale(*precision, *scale) + .expect("valid decimal"), + ) + } + ArrowDataType::Struct(fields) => { + let child_arrays: Vec = fields + .iter() + .map(|f| { + let child = generate_column(f.data_type(), rows, base); + if f.is_nullable() { + with_sparse_nulls(child, NULL_RATE_EVERY_NTH) + } else { + child + } + }) + .collect(); + Arc::new(StructArray::new(fields.clone(), child_arrays, None)) + } + other => panic!("unsupported Arrow type in test data generation: {other:?}"), + } +} + +/// Wrap `array` so that every `every_nth`-th row reads as null. Used by +/// [`write_data_commit`] to make the default sweep exercise null-aware code paths +/// (stats null counts, predicate evaluation under null, etc.) uniformly. +fn with_sparse_nulls(array: ArrayRef, every_nth: usize) -> ArrayRef { + assert!(every_nth >= 1, "every_nth must be >= 1"); + let n = array.len(); + let valid: Vec = (0..n).map(|i| !i.is_multiple_of(every_nth)).collect(); + let null_buffer = NullBuffer::from(valid); + let data = array + .to_data() + .into_builder() + .nulls(Some(null_buffer)) + .build() + .expect("rebuilding array with null buffer"); + delta_kernel::arrow::array::make_array(data) +} + +// =========================================================================== +// TestTable +// =========================================================================== + +/// A built test table backed by an in-memory object store. +/// +/// Exposes only the store and table root URL. Does NOT expose kernel types like +/// `Snapshot` or `Engine` -- the test creates those from its own kernel imports. +/// This avoids type mismatches between the test crate's kernel and test_utils's kernel. +pub struct TestTable { + store: Arc, + table_root: String, + description: String, +} + +impl TestTable { + /// The object store containing all table files. + pub fn store(&self) -> &Arc { + &self.store + } + + /// The table root URL string (e.g. `"memory:///"`). + pub fn table_root(&self) -> &str { + &self.table_root + } + + /// Human-readable description of this table's configuration (e.g. + /// `"v=3+checkpoint_at(2) + columnMapping.mode=name, enableInCommitTimestamps=true"`). + /// Useful in assert messages to identify which config failed. + pub fn description(&self) -> &str { + &self.description + } + + /// Create a `DefaultEngine` backed by this table's store. + /// + /// Returns the engine from `test_utils`'s `delta_kernel`. For unit tests inside + /// `kernel/src/`, use `DefaultEngineBuilder::new(table.store().clone()).build()` + /// instead to get the correct crate-local engine type. + pub fn engine(&self) -> DefaultEngine { + DefaultEngineBuilder::new(self.store.clone()).build() + } +} + +impl fmt::Display for TestTable { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{}", self.description) + } +} + +// =========================================================================== +// rstest fixtures +// =========================================================================== + +/// Convenience wrapper: build a [`TestTable`] from a `log_state`, `feature_set`, +/// `data_layout`, and `table_config`. Used by the `test_context!` macro and available +/// for direct use in tests. +pub fn test_table( + log_state: LogState, + feature_set: FeatureSet, + data_layout: DataLayoutConfig, + table_config: TableConfig, +) -> TestTable { + TestTableBuilder::new() + .with_log_state(log_state) + .with_features(feature_set) + .with_data_layout(data_layout) + .with_table_config(table_config) + .build() + .expect("failed to build test table") +} + +// =========================================================================== +// Macros +// =========================================================================== + +/// Load a snapshot from a [`TestTable`] according to a [`VersionTarget`]. +/// +/// Expands at the call site so `Snapshot` resolves to the caller's crate. This avoids +/// the type mismatch between `test_utils`'s kernel and `kernel/src/` unit tests' kernel. +/// Requires `Snapshot` to be in scope at the call site. +#[macro_export] +macro_rules! build_snapshot { + ($version_target:expr, $table_root:expr, $engine:expr) => { + match &$version_target { + $crate::table_builder::VersionTarget::Latest => { + Snapshot::builder_for($table_root).build($engine).unwrap() + } + $crate::table_builder::VersionTarget::AtVersion(v) => { + Snapshot::builder_for($table_root) + .at_version(*v) + .build($engine) + .unwrap() + } + $crate::table_builder::VersionTarget::IncrementalToLatest { from } => { + let base = Snapshot::builder_for($table_root) + .at_version(*from) + .build($engine) + .unwrap(); + Snapshot::builder_from(base).build($engine).unwrap() + } + $crate::table_builder::VersionTarget::IncrementalFrom { from, to } => { + let base = Snapshot::builder_for($table_root) + .at_version(*from) + .build($engine) + .unwrap(); + Snapshot::builder_from(base) + .at_version(*to) + .build($engine) + .unwrap() + } + $crate::table_builder::VersionTarget::AtTimestamp(ts) => { + let latest = Snapshot::builder_for($table_root).build($engine).unwrap(); + let commit = ::delta_kernel::history_manager::latest_version_as_of( + &latest, + $engine, + *ts, + ::delta_kernel::history_manager::HistoryCommitType::Recreatable, + ) + .unwrap(); + Snapshot::builder_for($table_root) + .at_version(commit.version) + .build($engine) + .unwrap() + } + } + }; +} + +/// Build a table, engine, and snapshot from rstest parameters in one call. +/// +/// Expands at the call site so `Snapshot` and the engine type resolve to the caller's crate +/// types. Returns `(engine, snapshot, table)`. +/// +/// The 5-argument form defaults to `DefaultEngine` and requires `DefaultEngineBuilder` to be in +/// scope at the call site. The 6-argument form takes an explicit engine factory closure +/// `Fn(Arc) -> Engine`, useful when the caller cannot construct a +/// `DefaultEngine` (e.g. kernel-internal unit tests that depend only on `SyncEngine`). +/// +/// ```ignore +/// let (engine, snap, table) = test_context!( +/// log_state, feature_set, data_layout, table_config, version_target, +/// ); +/// let (engine, snap, table) = test_context!( +/// log_state, feature_set, data_layout, table_config, version_target, +/// |store| SyncEngine::new_with_store(store), +/// ); +/// ``` +#[macro_export] +macro_rules! test_context { + ($log_state:expr, $feature_set:expr, $data_layout:expr, $table_config:expr, $version_target:expr $(,)?) => { + $crate::test_context!( + $log_state, + $feature_set, + $data_layout, + $table_config, + $version_target, + |store| { DefaultEngineBuilder::new(store).build() } + ) + }; + ($log_state:expr, $feature_set:expr, $data_layout:expr, $table_config:expr, $version_target:expr, $engine_factory:expr $(,)?) => {{ + let table = $crate::table_builder::test_table( + $log_state, + $feature_set, + $data_layout, + $table_config, + ); + let engine = ($engine_factory)(table.store().clone()); + let snap = $crate::build_snapshot!($version_target, table.table_root(), &engine); + (engine, snap, table) + }}; +} + +// =========================================================================== +// Partition value generation +// =========================================================================== + +/// Generate deterministic partition values for a given version and file index. +/// Follows the Delta protocol partition value serialization format. +fn generate_partition_values( + schema: &StructType, + partition_columns: &[String], + seed: usize, +) -> HashMap { + partition_columns + .iter() + .map(|col_name| { + let field = schema + .field(col_name) + .unwrap_or_else(|| panic!("partition column '{col_name}' not in schema")); + let value = scalar_for_type(field.data_type(), seed); + (col_name.clone(), value) + }) + .collect() +} + +/// Generate a deterministic [`Scalar`] partition value for the given data type. +fn scalar_for_type(data_type: &DataType, seed: usize) -> Scalar { + match data_type { + DataType::Primitive(p) => { + #[allow(unreachable_patterns)] + match p { + PrimitiveType::Boolean => Scalar::Boolean(seed.is_multiple_of(2)), + PrimitiveType::Byte => Scalar::Byte((seed % 100) as i8), + PrimitiveType::Short => Scalar::Short((seed % 100) as i16), + PrimitiveType::Integer => Scalar::Integer((seed % 100) as i32), + PrimitiveType::Long => Scalar::Long((seed * 1000) as i64), + PrimitiveType::Float => Scalar::Float(seed as f32 * 0.5), + PrimitiveType::Double => Scalar::Double(seed as f64 * 0.25), + PrimitiveType::String => Scalar::String(format!("part_{seed}")), + PrimitiveType::Binary => Scalar::Binary(format!("bin_{seed}").into_bytes()), + PrimitiveType::Date => { + // Days since epoch (1970-01-01) + Scalar::Date(18000 + seed as i32) + } + PrimitiveType::Timestamp => { + // Microseconds since epoch (UTC) + Scalar::Timestamp((18000 + seed as i64) * 86_400_000_000) + } + PrimitiveType::TimestampNtz => { + // Microseconds since epoch (no timezone) + Scalar::TimestampNtz((18000 + seed as i64) * 86_400_000_000) + } + PrimitiveType::Decimal(dt) => { + let scale_factor = 10i128.pow(dt.scale() as u32); + let bits = seed as i128 * scale_factor; + Scalar::decimal(bits, dt.precision(), dt.scale()) + .expect("test seed produced invalid decimal") + } + PrimitiveType::Void => panic!("void type is not a valid partition column"), + // Intervals are physical integers: months (year-month) / microseconds (day-time). + PrimitiveType::IntervalYearMonth => Scalar::IntervalYearMonth((seed % 100) as i32), + PrimitiveType::IntervalDayTime => Scalar::IntervalDayTime((seed * 1000) as i64), + other => panic!("{other:?} is not a valid partition column type"), + } + } + other => panic!("partition columns must be primitive types, got: {other:?}"), + } +} + +// =========================================================================== +// Helpers: schema +// =========================================================================== + +/// Default schema with all Delta primitive types including TimestampNtz +/// and a nested column type. +pub(crate) fn default_schema() -> SchemaRef { + Arc::new(StructType::new_unchecked(vec![ + StructField::new("bool_col", DataType::BOOLEAN, true), + StructField::new("byte_col", DataType::BYTE, true), + StructField::new("short_col", DataType::SHORT, true), + StructField::new("int_col", DataType::INTEGER, true), + StructField::new("long_col", DataType::LONG, true), + StructField::new("float_col", DataType::FLOAT, true), + StructField::new("double_col", DataType::DOUBLE, true), + StructField::new("string_col", DataType::STRING, true), + StructField::new("binary_col", DataType::BINARY, true), + StructField::new("date_col", DataType::DATE, true), + StructField::new("ts_col", DataType::TIMESTAMP, true), + StructField::new("ts_ntz_col", DataType::TIMESTAMP_NTZ, true), + StructField::new("decimal_col", DataType::decimal(10, 2).unwrap(), true), + StructField::new( + "nested_col", + DataType::try_struct_type([ + StructField::nullable("a", DataType::LONG), + StructField::nullable("b", DataType::STRING), + ]) + .unwrap(), + true, + ), + ])) +} + +#[cfg(test)] +mod tests { + use delta_kernel::object_store::path::Path; + use delta_kernel::object_store::ObjectStore; + use rstest::rstest; + + use super::*; + + #[test] + fn test_basic_build() -> DeltaResult<()> { + let table = TestTableBuilder::new().build()?; + let engine = table.engine(); + let snap = Snapshot::builder_for(table.table_root()).build(&engine)?; + assert_eq!(snap.version(), 0); + Ok(()) + } + + #[test] + fn test_commits_only() -> DeltaResult<()> { + let table = TestTableBuilder::new() + .with_log_state(LogState::with_latest_version(2)) + .build()?; + let engine = table.engine(); + let snap = Snapshot::builder_for(table.table_root()).build(&engine)?; + assert_eq!(snap.version(), 2); + Ok(()) + } + + /// Demonstrates the `test_context!` macro with rstest `#[values]`. + #[rstest] + fn test_version_targets( + #[values(LogState::with_latest_version(4))] log_state: LogState, + #[values(FeatureSet::empty())] feature_set: FeatureSet, + #[values(unpartitioned())] data_layout: DataLayoutConfig, + #[values(checkpoint_json_stats())] table_config: TableConfig, + #[values( + VersionTarget::Latest, + VersionTarget::AtVersion(2), + VersionTarget::IncrementalToLatest { from: 1 }, + VersionTarget::IncrementalFrom { from: 1, to: 3 }, + VersionTarget::AtTimestamp(i64::MAX), + )] + version_target: VersionTarget, + ) { + let (_engine, snap, _table) = test_context!( + log_state, + feature_set, + data_layout, + table_config, + version_target + ); + let expected = match &version_target { + VersionTarget::Latest | VersionTarget::IncrementalToLatest { .. } => 4, + VersionTarget::AtTimestamp(ts) if *ts == i64::MAX => 4, + VersionTarget::AtVersion(v) => *v, + VersionTarget::IncrementalFrom { to, .. } => *to, + VersionTarget::AtTimestamp(ts) => { + panic!("test only uses AtTimestamp(i64::MAX), got {ts}") + } + }; + assert_eq!(snap.version(), expected); + } + + #[rstest_reuse::apply(feature_sets)] + fn test_feature_sets_enable_table_features(feature_set: FeatureSet) -> DeltaResult<()> { + let expected_features = feature_set.expected_features(); + let table = TestTableBuilder::new().with_features(feature_set).build()?; + let engine = table.engine(); + let snap = Snapshot::builder_for(table.table_root()).build(&engine)?; + let protocol = snap.table_configuration().protocol(); + let writer_features = protocol + .writer_features() + .expect("should have writer features"); + for feature in &expected_features { + assert!( + writer_features.contains(feature), + "expected {feature:?} in writer_features: {writer_features:?}" + ); + } + Ok(()) + } + + #[test] + fn test_column_mapping_none_is_noop() { + let fs = FeatureSet::new().column_mapping("none"); + assert!(fs.table_properties.is_empty()); + } + + #[test] + fn test_table_config_properties_applied() -> DeltaResult<()> { + let table = TestTableBuilder::new() + .with_table_config( + TableConfig::new() + .write_stats_as_json(false) + .write_stats_as_struct(true), + ) + .build()?; + let engine = table.engine(); + let snap = Snapshot::builder_for(table.table_root()).build(&engine)?; + assert_eq!(snap.version(), 0); + Ok(()) + } + + #[test] + fn test_data_skipping_table_properties_in_metadata() -> DeltaResult<()> { + let table = TestTableBuilder::new() + .with_table_config( + TableConfig::new() + .num_data_skipping_indexed_cols(2) + .data_skipping_stats_columns(["int_col", "long_col"]), + ) + .build()?; + let config = read_metadata_configuration(table.store(), 0)?; + assert_eq!( + config + .get("delta.dataSkippingNumIndexedCols") + .map(|s| s.as_str()), + Some("2") + ); + assert_eq!( + config + .get("delta.dataSkippingStatsColumns") + .map(|s| s.as_str()), + Some("int_col,long_col") + ); + Ok(()) + } + + fn read_metadata_configuration( + store: &Arc, + version: u64, + ) -> DeltaResult> { + let store = store.clone(); + block_on_sync(move || async move { + crate::read_metadata_configuration_from_store(store.as_ref(), version) + .await + .map_err(|e| delta_kernel::Error::generic(e.to_string())) + }) + } + + /// Verifies every common table config builds successfully. + #[rstest_reuse::apply(table_configs)] + fn test_all_table_configs_build(table_config: TableConfig) -> DeltaResult<()> { + TestTableBuilder::new() + .with_table_config(table_config) + .build()?; + Ok(()) + } + + /// V2 checkpoints with sidecars require both the `v2Checkpoint` feature + /// and `with_sidecars_if_enabled(...)` on the `LogState`. The checkpoint writer + /// emits sidecar parquet files into `_delta_log/_sidecars/`. + #[rstest::rstest] + #[case::default_hint(None)] + #[case::explicit_hint(Some(1))] + fn test_v2_sidecars_emit_sidecar_files(#[case] hint: Option) -> DeltaResult<()> { + let log_state = LogState::with_latest_version(2) + .with_checkpoint_at([1]) + .with_sidecars_if_enabled(hint); + let table = TestTableBuilder::new() + .with_log_state(log_state) + .with_features(FeatureSet::new().v2_checkpoint()) + .with_data(2, 5) + .build()?; + let sidecars = list_dir_filenames(table.store(), "_delta_log/_sidecars")?; + assert!( + sidecars.iter().any(|n| n.ends_with(".parquet")), + "expected sidecar files in _delta_log/_sidecars/, got {sidecars:?}", + ); + Ok(()) + } + + /// `with_sidecars_if_enabled` without the `v2Checkpoint` feature is a no-op: + /// the table builds (no error) and no sidecar directory is produced. + #[test] + fn test_v2_sidecars_silently_ignored_without_v2_feature() -> DeltaResult<()> { + let log_state = LogState::with_latest_version(2) + .with_checkpoint_at([1]) + .with_sidecars_if_enabled(None); + let table = TestTableBuilder::new() + .with_log_state(log_state) + .with_data(2, 5) + .build()?; + let sidecars = list_dir_filenames(table.store(), "_delta_log/_sidecars")?; + assert!( + sidecars.is_empty(), + "expected no sidecar files without v2_checkpoint, got {sidecars:?}", + ); + Ok(()) + } + + #[test] + fn test_checkpoint_and_commits() -> DeltaResult<()> { + let log_state = LogState::with_latest_version(4).with_checkpoint_at([2]); + let table = TestTableBuilder::new() + .with_log_state(log_state.clone()) + .build()?; + assert_log_state_files_on_disk(&table, &log_state)?; + let engine = table.engine(); + let snap = Snapshot::builder_for(table.table_root()).build(&engine)?; + assert_eq!(snap.version(), 4); + Ok(()) + } + + /// Cartesian-product check that every checkpoint configuration the builder + /// permits actually lands on disk and rebuilds correctly. Covers single + /// checkpoints at every legal version (None, 0, mid, latest) plus a + /// multi-checkpoint case (mid + latest). + #[rstest::rstest] + #[case::no_checkpoint(LogState::with_latest_version(2))] + #[case::checkpoint_at_v0(LogState::with_latest_version(2).with_checkpoint_at([0]))] + #[case::checkpoint_at_v1(LogState::with_latest_version(2).with_checkpoint_at([1]))] + #[case::checkpoint_at_latest(LogState::with_latest_version(2).with_checkpoint_at([2]))] + #[case::two_checkpoints( + LogState::with_latest_version(2).with_checkpoint_at([1, 2]), + )] + #[case::hint_missing_no_checkpoint( + LogState::with_latest_version(2).with_last_checkpoint_hint(LastCheckpointHintState::Missing), + )] + #[case::hint_missing_with_checkpoint( + LogState::with_latest_version(2) + .with_checkpoint_at([1]) + .with_last_checkpoint_hint(LastCheckpointHintState::Missing), + )] + #[case::hint_stale_two_checkpoints( + LogState::with_latest_version(2) + .with_checkpoint_at([1, 2]) + .with_last_checkpoint_hint(LastCheckpointHintState::Stale), + )] + #[case::cleanup_with_hint_missing( + LogState::with_latest_version(2) + .with_checkpoint_at([1]) + .with_cleanup_commits_before(1) + .with_last_checkpoint_hint(LastCheckpointHintState::Missing), + )] + #[case::cleanup_with_hint_stale( + LogState::with_latest_version(2) + .with_checkpoint_at([1, 2]) + .with_cleanup_commits_before(1) + .with_last_checkpoint_hint(LastCheckpointHintState::Stale), + )] + #[case::stale_hint_points_at_deleted_checkpoint( + LogState::with_latest_version(10) + .with_checkpoint_at([5, 8]) + .with_cleanup_commits_before(8) + .with_last_checkpoint_hint(LastCheckpointHintState::Stale), + )] + #[case::crc_at_v0(LogState::with_latest_version(2).with_crc_at([0]))] + #[case::crc_at_v1(LogState::with_latest_version(2).with_crc_at([1]))] + #[case::crc_at_latest(LogState::with_latest_version(2).with_crc_at([2]))] + #[case::checkpoint_with_later_crc( + LogState::with_latest_version(2).with_checkpoint_at([1]).with_crc_at([2]), + )] + #[case::crc_below_cleanup_is_deleted( + LogState::with_latest_version(3) + .with_checkpoint_at([2]) + .with_crc_at([1]) + .with_cleanup_commits_before(2), + )] + #[case::crc_at_cleanup_boundary_survives( + LogState::with_latest_version(3) + .with_checkpoint_at([2]) + .with_crc_at([2]) + .with_cleanup_commits_before(2), + )] + fn test_log_state_checkpoint_shapes_land_on_disk( + #[case] log_state: LogState, + ) -> DeltaResult<()> { + let expected_version = log_state.latest_version(); + let table = TestTableBuilder::new() + .with_log_state(log_state.clone()) + .build()?; + assert_log_state_files_on_disk(&table, &log_state)?; + let engine = table.engine(); + let snap = Snapshot::builder_for(table.table_root()).build(&engine)?; + assert_eq!( + snap.version(), + expected_version, + "rebuild lost commits for {}", + table.description(), + ); + Ok(()) + } + + /// Stale hint must point at a real older checkpoint (not the latest); pairing + /// `Stale` with fewer than two checkpoints is rejected at build time. + #[rstest::rstest] + #[case::no_checkpoint(LogState::with_latest_version(3))] + #[case::single_checkpoint(LogState::with_latest_version(3).with_checkpoint_at([1]))] + #[should_panic(expected = "Stale hint requires at least 2 checkpoints")] + fn test_stale_hint_requires_two_checkpoints(#[case] base: LogState) { + let log_state = base.with_last_checkpoint_hint(LastCheckpointHintState::Stale); + let _ = TestTableBuilder::new().with_log_state(log_state).build(); + } + + /// Stale hint behavioral check: kernel must follow the stale (older) hint, list + /// forward, and pick up the actual latest checkpoint at the higher version -- + /// not the stale-hinted older one. + #[test] + fn test_stale_hint_recovery_resolves_to_actual_latest_checkpoint() -> DeltaResult<()> { + let log_state = LogState::with_latest_version(5) + .with_checkpoint_at([2, 4]) + .with_last_checkpoint_hint(LastCheckpointHintState::Stale); + let table = TestTableBuilder::new() + .with_log_state(log_state.clone()) + .build()?; + assert_log_state_files_on_disk(&table, &log_state)?; + let engine = table.engine(); + let snap = Snapshot::builder_for(table.table_root()).build(&engine)?; + assert_eq!(snap.version(), 5); + Ok(()) + } + + #[test] + fn test_scan_with_column_mapping() -> DeltaResult<()> { + let table = TestTableBuilder::new() + .with_log_state(LogState::with_latest_version(1)) + .with_features(FeatureSet::new().column_mapping("name")) + .with_data(1, 5) + .build()?; + let engine: Arc = + Arc::new(DefaultEngineBuilder::new(table.store().clone()).build()); + let snap = Snapshot::builder_for(table.table_root()).build(engine.as_ref())?; + let scan = snap.scan_builder().build()?; + let batches = crate::read_scan(&scan, engine)?; + let total: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total, 5); + Ok(()) + } + + #[test] + fn test_scan_with_data() -> DeltaResult<()> { + let table = TestTableBuilder::new() + .with_log_state(LogState::with_latest_version(1)) + .with_data(2, 5) + .build()?; + let engine: Arc = + Arc::new(DefaultEngineBuilder::new(table.store().clone()).build()); + let snap = Snapshot::builder_for(table.table_root()).build(engine.as_ref())?; + let scan = snap.scan_builder().build()?; + let batches = crate::read_scan(&scan, engine)?; + let total: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total, 10); + Ok(()) + } + + /// Verifies the data generator injects sparse nulls into every nullable data column. + /// Uses `rows_per_file = NULL_RATE_EVERY_NTH * 3` so each file has exactly three null + /// rows per column. + #[test] + fn test_sparse_null_injection_in_generated_data() -> DeltaResult<()> { + let rows_per_file = NULL_RATE_EVERY_NTH * 3; + let table = TestTableBuilder::new() + .with_log_state(LogState::with_latest_version(1)) + .with_data(1, rows_per_file) + .build()?; + let engine: Arc = + Arc::new(DefaultEngineBuilder::new(table.store().clone()).build()); + let snap = Snapshot::builder_for(table.table_root()).build(engine.as_ref())?; + let scan = snap.scan_builder().build()?; + let batches = crate::read_scan(&scan, engine)?; + assert!(!batches.is_empty(), "expected at least one batch"); + let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total_rows, rows_per_file); + let expected_nulls_per_col = rows_per_file / NULL_RATE_EVERY_NTH; + for batch in &batches { + for col_idx in 0..batch.num_columns() { + let null_count = batch.column(col_idx).null_count(); + assert_eq!( + null_count, + expected_nulls_per_col, + "column '{}' should have {expected_nulls_per_col} nulls, got {null_count}", + batch.schema().field(col_idx).name(), + ); + } + } + Ok(()) + } + + /// Partition columns are never nulled so their data matches the declared partition + /// value. Clustering columns are ordinary data columns (kernel permits null clustering + /// keys), so they receive the same sparse nulls as any other nullable column -- the + /// clustered case guards against accidentally protecting them. + #[rstest::rstest] + #[case::partitioned(partitioned())] + #[case::clustered(clustered())] + fn test_layout_column_null_injection(#[case] config: DataLayoutConfig) -> DeltaResult<()> { + let rows_per_file = NULL_RATE_EVERY_NTH * 3; + // Only partition columns are protected from nulling; clustering columns are not. + let protected_columns = if config.is_partitioned() { + config.columns() + } else { + Vec::new() + }; + let table = TestTableBuilder::new() + .with_log_state(LogState::with_latest_version(1)) + .with_data_layout(config) + .with_data(1, rows_per_file) + .build()?; + let engine: Arc = + Arc::new(DefaultEngineBuilder::new(table.store().clone()).build()); + let snap = Snapshot::builder_for(table.table_root()).build(engine.as_ref())?; + let scan = snap.scan_builder().build()?; + let batches = crate::read_scan(&scan, engine)?; + assert!(!batches.is_empty(), "expected at least one batch"); + let expected_nulls_per_col = rows_per_file / NULL_RATE_EVERY_NTH; + for batch in &batches { + for col_idx in 0..batch.num_columns() { + let name = batch.schema().field(col_idx).name().to_string(); + let null_count = batch.column(col_idx).null_count(); + if protected_columns.contains(&name) { + assert_eq!( + null_count, 0, + "partition column '{name}' must never be nulled, got {null_count}", + ); + } else { + assert_eq!( + null_count, expected_nulls_per_col, + "data column '{name}' should have {expected_nulls_per_col} nulls, \ + got {null_count}", + ); + } + } + } + Ok(()) + } + + #[rstest::rstest] + #[case::partitioned(partitioned(), partitioned_schema())] + #[case::clustered(clustered(), clustered_schema())] + fn test_data_layout_table( + #[case] config: DataLayoutConfig, + #[case] expected_schema: SchemaRef, + ) -> DeltaResult<()> { + // 2 versions means v0 (create_table) + v1 (1 data commit with 10 rows) + let table = TestTableBuilder::new() + .with_log_state(LogState::with_latest_version(1)) + .with_data_layout(config) + .build()?; + let engine = table.engine(); + let snap = Snapshot::builder_for(table.table_root()).build(&engine)?; + assert_eq!(snap.version(), 1); + assert_eq!(snap.schema(), expected_schema); + let scan = snap.scan_builder().build()?; + let engine_arc: Arc = + Arc::new(DefaultEngineBuilder::new(table.store().clone()).build()); + let batches = crate::read_scan(&scan, engine_arc)?; + let total: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total, 10); + Ok(()) + } + + #[test] + fn test_clustered_table_multiple_versions() -> DeltaResult<()> { + // v0=create, v1-v3=data commits, 10 rows each + let table = TestTableBuilder::new() + .with_log_state(LogState::with_latest_version(3)) + .with_data_layout(clustered()) + .with_table_config(checkpoint_struct_stats()) + .build()?; + let engine = table.engine(); + let snap = Snapshot::builder_for(table.table_root()).build(&engine)?; + assert_eq!(snap.version(), 3); + let scan = snap.scan_builder().build()?; + let engine_arc: Arc = + Arc::new(DefaultEngineBuilder::new(table.store().clone()).build()); + let batches = crate::read_scan(&scan, engine_arc)?; + let total: usize = batches.iter().map(|b| b.num_rows()).sum(); + assert_eq!(total, 30); + Ok(()) + } + + #[test] + fn test_with_clustering_columns_directly() -> DeltaResult<()> { + let table = TestTableBuilder::new() + .with_log_state(LogState::with_latest_version(1)) + .with_schema(clustered_schema()) + .with_clustering_columns(["clust_int", "clust_string"]) + .build()?; + let engine = table.engine(); + let snap = Snapshot::builder_for(table.table_root()).build(&engine)?; + assert_eq!(snap.version(), 1); + assert_eq!(snap.schema(), clustered_schema()); + Ok(()) + } + + #[test] + fn test_nested_struct_schema_round_trip() -> DeltaResult<()> { + let inner = DataType::try_struct_type([StructField::nullable("a", DataType::LONG)])?; + let schema: SchemaRef = Arc::new(StructType::try_new([ + StructField::nullable("id", DataType::LONG), + StructField::nullable("inner", inner), + ])?); + let table = TestTableBuilder::new() + .with_schema(schema.clone()) + .build()?; + let engine = table.engine(); + let snap = Snapshot::builder_for(table.table_root()).build(&engine)?; + assert_eq!(snap.schema(), schema); + Ok(()) + } + + #[test] + fn test_layout_columns_are_mutually_exclusive() { + let builder = TestTableBuilder::new() + .with_partition_columns(["col_a"]) + .with_clustering_columns(["col_b"]); + assert!(builder.partition_columns.is_empty()); + assert_eq!(builder.clustering_columns, vec!["col_b"]); + + let builder = TestTableBuilder::new() + .with_clustering_columns(["col_a"]) + .with_partition_columns(["col_b"]); + assert!(builder.clustering_columns.is_empty()); + assert_eq!(builder.partition_columns, vec!["col_b"]); + } + + #[test] + #[should_panic(expected = "must be <=")] + fn test_with_checkpoint_at_rejects_above_latest_version() { + LogState::with_latest_version(2).with_checkpoint_at([3]); + } + + #[rstest::rstest] + #[case::no_features(FeatureSet::empty(), false)] + #[case::v2_checkpoint_feature(FeatureSet::new().v2_checkpoint(), false)] + #[case::ict(FeatureSet::new().ict(), true)] + fn test_builder_writes_crc_with_correct_content( + #[case] features: FeatureSet, + #[case] expect_ict: bool, + ) -> DeltaResult<()> { + let log_state = LogState::with_latest_version(2) + .with_checkpoint_at([2]) + .with_crc_at([2]); + let table = TestTableBuilder::new() + .with_log_state(log_state.clone()) + .with_features(features) + .build()?; + assert_log_state_files_on_disk(&table, &log_state)?; + + let crc = read_crc_json(table.store(), 2)?; + assert_eq!( + crc["numFiles"].as_u64().unwrap(), + 2, + "numFiles should match the 2 data adds emitted at v=1 and v=2", + ); + assert!( + crc["tableSizeBytes"].as_u64().unwrap() > 0, + "tableSizeBytes should be positive for a table with 2 data files", + ); + assert_eq!( + crc["inCommitTimestampOpt"].is_i64(), + expect_ict, + "inCommitTimestampOpt presence must match the ICT feature flag", + ); + Ok(()) + } + + #[test] + #[should_panic(expected = "must be <= latest_version")] + fn test_with_crc_at_rejects_above_latest_version() { + LogState::with_latest_version(2).with_crc_at([3]); + } + + #[test] + #[should_panic(expected = "already present")] + fn test_with_crc_at_rejects_duplicate_version() { + LogState::with_latest_version(2).with_crc_at([1, 1]); + } + + #[test] + fn test_cleanup_commits_before_deletes_old_files_and_rebuilds() -> DeltaResult<()> { + // Checkpoints at v=1 (below cutoff), v=3 (at cutoff), v=5 (above cutoff) + // exercise both the deletion and preservation paths for checkpoint files. + let log_state = LogState::with_latest_version(5) + .with_checkpoint_at([1, 3, 5]) + .with_cleanup_commits_before(3); + let table = TestTableBuilder::new() + .with_log_state(log_state.clone()) + .build()?; + assert_log_state_files_on_disk(&table, &log_state)?; + let entries = list_log_dir_filenames(table.store())?; + for v in 0..3 { + let json = format!("{v:020}.json"); + assert!( + !entries.iter().any(|name| name == &json), + "expected v={v} JSON to be cleaned up, found in {entries:?}", + ); + } + for v in 3..=5 { + let json = format!("{v:020}.json"); + assert!( + entries.iter().any(|name| name == &json), + "expected v={v} JSON to survive cleanup, missing from {entries:?}", + ); + } + // Checkpoint files: v=1 deleted, v=3 and v=5 preserved. + let v1_prefix = format!("{:020}.checkpoint", 1u64); + assert!( + !entries.iter().any(|name| name.starts_with(&v1_prefix)), + "expected v=1 checkpoint to be cleaned up, found in {entries:?}", + ); + for v in [3u64, 5] { + let prefix = format!("{v:020}.checkpoint"); + assert!( + entries.iter().any(|name| name.starts_with(&prefix)), + "expected v={v} checkpoint to survive cleanup, missing from {entries:?}", + ); + } + let engine = table.engine(); + let snap = Snapshot::builder_for(table.table_root()).build(&engine)?; + assert_eq!(snap.version(), 5); + Ok(()) + } + + #[rstest::rstest] + #[case::no_checkpoint(LogState::with_latest_version(5))] + #[case::checkpoint_at_early_version( + LogState::with_latest_version(5).with_checkpoint_at([2]), + )] + #[should_panic(expected = "requires a checkpoint at v=3")] + fn test_cleanup_commits_before_requires_checkpoint_at_n(#[case] base: LogState) { + let log_state = base.with_cleanup_commits_before(3); + let _ = TestTableBuilder::new().with_log_state(log_state).build(); + } + + #[test] + #[should_panic(expected = "exceeds latest_version")] + fn test_cleanup_commits_before_rejects_above_latest_version() { + let log_state = LogState::with_latest_version(2) + .with_checkpoint_at([2]) + .with_cleanup_commits_before(3); + let _ = TestTableBuilder::new().with_log_state(log_state).build(); + } + + #[rstest::rstest] + #[case(LogState::with_latest_version(3).with_checkpoint_at([2]), 3)] + #[case(LogState::with_latest_version(3).with_checkpoint_at([1]), 3)] + #[case(LogState::with_latest_version(4), 4)] + #[case(LogState::with_latest_version(2).with_checkpoint_at([1, 2]), 2)] + fn test_log_state_latest_version(#[case] log_state: LogState, #[case] expected: u64) { + assert_eq!(log_state.latest_version(), expected); + } + + /// Asserts the on-disk log directory matches `log_state`: declared checkpoints + /// survive iff their version is >= `cleanup_before`, no versioned log file + /// remains below `cleanup_before`, and the `_last_checkpoint` hint reflects + /// the declared hint state. Snapshot rebuilds can silently succeed via JSON + /// replay even when these files are wrong. + fn assert_log_state_files_on_disk(table: &TestTable, log_state: &LogState) -> DeltaResult<()> { + let entries = list_log_dir_filenames(table.store())?; + let cleanup = log_state.cleanup_before().unwrap_or(0); + for &v in log_state.checkpoints_at() { + let prefix = format!("{v:020}.checkpoint"); + let surviving = v >= cleanup; + let found = entries + .iter() + .any(|name| name.starts_with(&prefix) && name.ends_with(".parquet")); + assert_eq!( + found, + surviving, + "checkpoint at v={v} should be {} for {log_state}: {entries:?}", + if surviving { "present" } else { "cleaned up" }, + ); + } + for entry in &entries { + if let Some(v) = log_file_version(&Path::from(entry.as_str())) { + assert!( + v >= cleanup, + "expected v={v} ({entry}) to be cleaned up (cleanup_before={cleanup}): {entries:?}", + ); + } + } + + for &v in log_state.crcs_at() { + let crc_name = format!("{v:020}.crc"); + let surviving = v >= cleanup; + let found = entries.iter().any(|name| name == &crc_name); + assert_eq!( + found, + surviving, + "CRC at v={v} should be {} for {log_state}: {entries:?}", + if surviving { "present" } else { "cleaned up" }, + ); + if surviving { + let crc = read_crc_json(table.store(), v)?; + assert!( + crc["tableSizeBytes"].is_u64(), + "CRC at v={v} missing tableSizeBytes", + ); + assert!(crc["numFiles"].is_u64(), "CRC at v={v} missing numFiles"); + assert_eq!( + crc["numMetadata"].as_u64(), + Some(1), + "CRC at v={v} must have numMetadata == 1 per spec", + ); + assert_eq!( + crc["numProtocol"].as_u64(), + Some(1), + "CRC at v={v} must have numProtocol == 1 per spec", + ); + assert!(crc["protocol"].is_object(), "CRC at v={v} missing protocol"); + assert!(crc["metadata"].is_object(), "CRC at v={v} missing metadata"); + } + } + + let hint = read_last_checkpoint_hint(table.store())?; + match log_state.last_checkpoint_hint() { + LastCheckpointHintState::Present => { + let want = log_state.checkpoints_at().last().copied(); + assert_eq!( + hint.map(|h| h.version), + want, + "Present hint should point at the highest checkpoint for {log_state}", + ); + } + LastCheckpointHintState::Missing => { + assert!( + hint.is_none(), + "Missing hint should leave no _last_checkpoint file for {log_state}, got {hint:?}", + ); + } + LastCheckpointHintState::Stale => { + let want = log_state.checkpoints_at().first().copied(); + assert_eq!( + hint.map(|h| h.version), + want, + "Stale hint should point at the lowest checkpoint for {log_state}", + ); + } + } + Ok(()) + } + + #[derive(Debug)] + struct HintFile { + version: u64, + } + + /// Read and JSON-parse a file from `store`. Returns `Ok(None)` if absent. + async fn try_read_json( + store: &DynObjectStore, + path: &Path, + ) -> DeltaResult> { + let bytes = match store.get(path).await { + Ok(r) => r.bytes().await.map_err(delta_kernel::Error::from)?, + Err(ObjectStoreError::NotFound { .. }) => return Ok(None), + Err(e) => return Err(delta_kernel::Error::from(e)), + }; + serde_json::from_slice(&bytes) + .map(Some) + .map_err(|e| delta_kernel::Error::generic(e.to_string())) + } + + /// Read and parse the `_last_checkpoint` hint, if present. + fn read_last_checkpoint_hint(store: &Arc) -> DeltaResult> { + let store = store.clone(); + block_on_sync(move || async move { + let path = Path::from("_delta_log/_last_checkpoint"); + match try_read_json(&store, &path).await? { + Some(parsed) => { + let version = parsed["version"].as_u64().ok_or_else(|| { + delta_kernel::Error::generic("hint missing `version` field") + })?; + Ok(Some(HintFile { version })) + } + None => Ok(None), + } + }) + } + + /// Read and parse a CRC file at `version`. Errors if the file is absent. + fn read_crc_json(store: &Arc, version: u64) -> DeltaResult { + let store = store.clone(); + block_on_sync(move || async move { + let path = Path::from(format!("_delta_log/{version:020}.crc")); + try_read_json(&store, &path) + .await? + .ok_or_else(|| delta_kernel::Error::generic(format!("CRC at v={version} missing"))) + }) + } + + /// Helper: list filenames (basenames only) directly under `_delta_log/` in + /// `store`. Returns just the leaf name so callers can match by suffix without + /// dealing with `memory:///`-style path quirks. + fn list_log_dir_filenames(store: &Arc) -> DeltaResult> { + list_dir_filenames(store, "_delta_log") + } + + /// Helper: list filenames at an arbitrary prefix. + fn list_dir_filenames(store: &Arc, prefix: &str) -> DeltaResult> { + let store = store.clone(); + let prefix = Path::from(prefix); + block_on_sync(move || async move { + let result = store + .list_with_delimiter(Some(&prefix)) + .await + .map_err(delta_kernel::Error::from)?; + Ok(result + .objects + .into_iter() + .filter_map(|m| m.location.filename().map(|s| s.to_string())) + .collect()) + }) + } +} diff --git a/uc-catalog/Cargo.toml b/uc-catalog/Cargo.toml deleted file mode 100644 index d41150994c..0000000000 --- a/uc-catalog/Cargo.toml +++ /dev/null @@ -1,30 +0,0 @@ -[package] -name = "uc-catalog" -edition.workspace = true -homepage.workspace = true -keywords.workspace = true -license.workspace = true -repository.workspace = true -readme.workspace = true -rust-version.workspace = true -version.workspace = true - -# for cargo-release -[package.metadata.release] -release = false - -[dependencies] -delta_kernel = { path = "../kernel", features = ["catalog-managed"] } -uc-client = { path = "../uc-client" } -itertools = "0.14" -serde = { version = "1.0", features = ["derive"] } -serde_json = "1.0" -tokio = { version = "1", features = ["full"] } -tracing = "0.1" -url = "2" -uuid = { version = "1", features = ["v4"] } - -[dev-dependencies] -delta_kernel = { path = "../kernel", features = ["arrow-56", "default-engine-rustls", "catalog-managed", "test-utils", "internal-api"] } -uc-client = { path = "../uc-client", features = ["test-utils"] } -tempfile = "3" diff --git a/uc-catalog/src/committer.rs b/uc-catalog/src/committer.rs deleted file mode 100644 index 72269d7a82..0000000000 --- a/uc-catalog/src/committer.rs +++ /dev/null @@ -1,221 +0,0 @@ -use std::sync::Arc; - -use delta_kernel::committer::{CommitMetadata, CommitResponse, Committer, PublishMetadata}; -use delta_kernel::{DeltaResult, Engine, Error as DeltaError, FilteredEngineData}; -use uc_client::models::commits::{Commit, CommitRequest}; -use uc_client::UCCommitClient; - -/// A [UCCommitter] is a Unity Catalog [`Committer`] implementation for committing to a specific -/// delta table in UC. -/// -/// NOTE: this [`Committer`] requires a multi-threaded tokio runtime. That is, whatever -/// implementation consumes the Committer to commit to the table, must call `commit` from within a -/// muti-threaded tokio runtime context. Since the default engine uses tokio, this is compatible, -/// but must ensure that the multi-threaded runtime is used. -#[derive(Debug, Clone)] -pub struct UCCommitter { - commits_client: Arc, - table_id: String, -} - -impl UCCommitter { - /// Create a new [UCCommitter] to commit via the `commits_client` to the specific table with the given - /// `table_id`. - pub fn new(commits_client: Arc, table_id: impl Into) -> Self { - UCCommitter { - commits_client, - table_id: table_id.into(), - } - } -} - -impl Committer for UCCommitter { - /// Commit the given `actions` to the delta table in UC. UC's committer elects to write out a - /// staged commit for the actions then call the UC commit API to 'finalize' (ratify) the staged - /// commit. Note that this will accumulate staged commits, and separately clients are expected - /// to periodically publish the staged commits to the delta log. In it's current form, UC - /// expects to be informed of the last known published version during this commit. - fn commit( - &self, - engine: &dyn Engine, - actions: Box> + Send + '_>, - commit_metadata: CommitMetadata, - ) -> DeltaResult { - if commit_metadata.version() == 0 { - return Err(DeltaError::unsupported( - "UCCommitter does not support version 0 (table creation) commits", - )); - } - let staged_commit_path = commit_metadata.staged_commit_path()?; - engine - .json_handler() - .write_json_file(&staged_commit_path, Box::new(actions), false)?; - - let committed = engine.storage_handler().head(&staged_commit_path)?; - tracing::debug!("wrote staged commit file: {:?}", committed); - - let commit_req = CommitRequest::new( - self.table_id.clone(), - commit_metadata.table_root().as_str(), - Commit::new( - commit_metadata.version().try_into().map_err(|_| { - DeltaError::generic("commit version does not fit into i64 for UC commit") - })?, - commit_metadata.in_commit_timestamp(), - staged_commit_path - .path_segments() - .ok_or_else(|| DeltaError::generic("staged commit contained no path segments"))? - .next_back() - .ok_or_else(|| { - DeltaError::generic("staged commit segments next_back was empty") - })?, - committed - .size - .try_into() - .map_err(|_| DeltaError::generic("committed size does not fit into i64"))?, - committed.last_modified, - ), - commit_metadata - .max_published_version() - .map(|v| { - v.try_into().map_err(|_| { - DeltaError::Generic(format!( - "Max published version {v} does not fit into i64 for UC commit" - )) - }) - }) - .transpose()?, - ); - let handle = tokio::runtime::Handle::try_current().map_err(|_| { - DeltaError::generic("UCCommitter may only be used within a tokio runtime") - })?; - tokio::task::block_in_place(|| { - handle.block_on(async move { - self.commits_client - .commit(commit_req) - .await - .map_err(|e| DeltaError::Generic(format!("UC commit error: {e}"))) - }) - })?; - Ok(CommitResponse::Committed { - file_meta: committed, - }) - } - - fn is_catalog_committer(&self) -> bool { - true - } - - fn publish(&self, engine: &dyn Engine, publish_metadata: PublishMetadata) -> DeltaResult<()> { - if publish_metadata.commits_to_publish().is_empty() { - return Ok(()); - } - - for catalog_commit in publish_metadata.commits_to_publish() { - let src = catalog_commit.location(); - let dest = catalog_commit.published_location(); - match engine.storage_handler().copy_atomic(src, dest) { - Ok(_) => (), - Err(DeltaError::FileAlreadyExists(_)) => (), - Err(e) => return Err(e), - } - } - - Ok(()) - } -} - -#[cfg(test)] -mod tests { - use super::*; - use delta_kernel::committer::{CatalogCommit, CommitMetadata}; - use delta_kernel::engine::default::DefaultEngine; - use delta_kernel::object_store::local::LocalFileSystem; - use delta_kernel::Version; - use std::fs; - use uc_client::error::Result; - - struct MockCommitsClient; - - impl UCCommitClient for MockCommitsClient { - async fn commit(&self, _: CommitRequest) -> Result<()> { - unimplemented!() - } - } - - #[test] - fn commit_rejects_version_0() { - let tmp_dir = tempfile::tempdir().unwrap(); - let table_root = url::Url::from_directory_path(tmp_dir.path()).unwrap(); - let commit_metadata = CommitMetadata::new_unchecked(table_root, 0).unwrap(); - let committer = UCCommitter::new(Arc::new(MockCommitsClient), "test-table-id"); - let engine = DefaultEngine::builder(Arc::new(LocalFileSystem::new())).build(); - let result = committer.commit(&engine, Box::new(std::iter::empty()), commit_metadata); - assert!( - matches!(result, Err(DeltaError::Unsupported(_))), - "expected Unsupported error for version 0, got: {result:?}" - ); - } - - fn staged_commit_url(table_root: &url::Url, version: Version) -> url::Url { - table_root - .join(&format!( - "_delta_log/_staged_commits/{version:020}.uuid.json" - )) - .unwrap() - } - - fn published_commit_url(table_root: &url::Url, version: Version) -> url::Url { - table_root - .join(&format!("_delta_log/{version:020}.json")) - .unwrap() - } - - #[tokio::test] - async fn test_publish() { - let tmp_dir = tempfile::tempdir().unwrap(); - let table_root = url::Url::from_directory_path(tmp_dir.path()).unwrap(); - let staged_dir = tmp_dir.path().join("_delta_log/_staged_commits"); - let versions = [10u64, 11, 12]; - - // ===== GIVEN ===== - // Create catalog commits - let catalog_commits: Vec = versions - .into_iter() - .map(|v| { - CatalogCommit::new_unchecked( - v, - staged_commit_url(&table_root, v), - published_commit_url(&table_root, v), - ) - }) - .collect(); - - // Write staged commit files to disk - fs::create_dir_all(&staged_dir).unwrap(); - for commit in &catalog_commits { - let path = commit.location().to_file_path().unwrap(); - fs::write(&path, format!("version: {}", commit.version())).unwrap(); - } - - // Write 10.json file to disk (should be skipped, not error) - let existing_published = published_commit_url(&table_root, 10) - .to_file_path() - .unwrap(); - fs::create_dir_all(existing_published.parent().unwrap()).unwrap(); - fs::write(&existing_published, "version: 10").unwrap(); - - // ===== WHEN ===== - let publish_metadata = PublishMetadata::try_new(12, catalog_commits).unwrap(); - let committer = UCCommitter::new(Arc::new(MockCommitsClient), "testUcTableId"); - let engine = DefaultEngine::builder(Arc::new(LocalFileSystem::new())).build(); - committer.publish(&engine, publish_metadata).unwrap(); - - // ===== THEN ===== - for v in versions { - let path = published_commit_url(&table_root, v).to_file_path().unwrap(); - assert!(path.exists()); - assert_eq!(fs::read_to_string(&path).unwrap(), format!("version: {v}")); - } - } -} diff --git a/uc-catalog/src/lib.rs b/uc-catalog/src/lib.rs deleted file mode 100644 index 366f99b6a1..0000000000 --- a/uc-catalog/src/lib.rs +++ /dev/null @@ -1,284 +0,0 @@ -//! UCCatalog implements a high-level interface for interacting with Delta Tables in Unity Catalog. - -mod committer; -pub use committer::UCCommitter; - -use std::sync::Arc; - -use delta_kernel::{Engine, LogPath, Snapshot, Version}; - -use uc_client::prelude::*; - -use itertools::Itertools; -use tracing::debug; -use url::Url; - -/// The [UCCatalog] provides a high-level interface to interact with Delta Tables stored in Unity -/// Catalog. For now this is a lightweight wrapper around a [UCGetCommitsClient]. -pub struct UCCatalog<'a, C: UCGetCommitsClient> { - client: &'a C, -} - -impl<'a, C: UCGetCommitsClient> UCCatalog<'a, C> { - /// Create a new [UCCatalog] instance with the provided client. - pub fn new(client: &'a C) -> Self { - UCCatalog { client } - } - - /// Load the latest snapshot of a Delta Table identified by `table_id` and `table_uri` in Unity - /// Catalog. Generally, a separate `get_table` call can be used to resolve the table id/uri from - /// the table name. - pub async fn load_snapshot( - &self, - table_id: &str, - table_uri: &str, - engine: &dyn Engine, - ) -> Result, Box> { - self.load_snapshot_inner(table_id, table_uri, None, engine) - .await - } - - /// Load a snapshot of a Delta Table identified by `table_id` and `table_uri` for a specific - /// version. Generally, a separate `get_table` call can be used to resolve the table id/uri from - /// the table name. - pub async fn load_snapshot_at( - &self, - table_id: &str, - table_uri: &str, - version: Version, - engine: &dyn Engine, - ) -> Result, Box> { - self.load_snapshot_inner(table_id, table_uri, Some(version), engine) - .await - } - - pub(crate) async fn load_snapshot_inner( - &self, - table_id: &str, - table_uri: &str, - version: Option, - engine: &dyn Engine, - ) -> Result, Box> { - let table_uri = table_uri.to_string(); - let req = CommitsRequest { - table_id: table_id.to_string(), - table_uri: table_uri.clone(), - start_version: Some(0), - end_version: version.and_then(|v| v.try_into().ok()), - }; - let mut commits = self.client.get_commits(req).await?; - if let Some(commits) = commits.commits.as_mut() { - commits.sort_by_key(|c| c.version) - } - - // if commits are present, we ensure they are sorted+contiguous - if let Some(commits) = &commits.commits { - if !commits.windows(2).all(|w| w[1].version == w[0].version + 1) { - return Err("Received non-contiguous commit versions".into()); - } - } - - // we always get back the latest version from commits response, and pass that in to - // kernel's Snapshot builder. basically, load_table for the latest version always looks - // like a time travel query since we know the latest version ahead of time. - // - // note there is a weird edge case: if the table was just created it will return - // latest_table_version = -1, but the 0.json will exist in the _delta_log. - let version: Version = match version { - Some(v) => v, - None => match commits.latest_table_version { - -1 => 0, - i => i.try_into()?, - }, - }; - - // consume uc-client's Commit and hand back a delta_kernel LogPath - let mut table_url = Url::parse(&table_uri)?; - // add trailing slash - if !table_url.path().ends_with('/') { - // NB: we push an empty segment which effectively adds a trailing slash - table_url - .path_segments_mut() - .map_err(|_| "Cannot modify URL path segments")? - .push(""); - } - let commits: Vec<_> = commits - .commits - .unwrap_or_default() - .into_iter() - .map( - |c| -> Result> { - LogPath::staged_commit( - table_url.clone(), - &c.file_name, - c.file_modification_timestamp, - c.file_size.try_into()?, - ) - .map_err(|e| e.into()) - }, - ) - .try_collect()?; - - debug!("commits for kernel: {:?}\n", commits); - - Snapshot::builder_for(table_url) - .at_version(version) - .with_log_tail(commits) - .build(engine) - .map_err(|e| e.into()) - } -} - -#[cfg(test)] -mod tests { - use std::env; - - use delta_kernel::engine::default::DefaultEngineBuilder; - use delta_kernel::object_store; - use delta_kernel::transaction::CommitResult; - - use tracing::info; - - use super::*; - - // We could just re-export UCClient's get_table to not require consumers to directly import - // uc_client themselves. - async fn get_table( - client: &UCClient, - table_name: &str, - ) -> Result<(String, String), Box> { - let res = client.get_table(table_name).await?; - let table_id = res.table_id; - let table_uri = res.storage_location; - - info!( - "[GET TABLE] got table_id: {}, table_uri: {}\n", - table_id, table_uri - ); - - Ok((table_id, table_uri)) - } - - // ignored test which you can run manually to play around with reading a UC table. run with: - // `ENDPOINT=".." TABLENAME=".." TOKEN=".." cargo t read_uc_table --nocapture -- --ignored` - #[ignore] - #[tokio::test] - async fn read_uc_table() -> Result<(), Box> { - let endpoint = env::var("ENDPOINT").expect("ENDPOINT environment variable not set"); - let token = env::var("TOKEN").expect("TOKEN environment variable not set"); - let table_name = env::var("TABLENAME").expect("TABLENAME environment variable not set"); - - // build shared config - let config = uc_client::ClientConfig::build(&endpoint, &token).build()?; - - // build clients - let uc_client = UCClient::new(config.clone())?; - let uc_commits_client = UCCommitsRestClient::new(config)?; - - let (table_id, table_uri) = get_table(&uc_client, &table_name).await?; - let creds = uc_client - .get_credentials(&table_id, Operation::Read) - .await - .map_err(|e| format!("Failed to get credentials: {e}"))?; - - let catalog = UCCatalog::new(&uc_commits_client); - - // TODO: support non-AWS - let creds = creds - .aws_temp_credentials - .ok_or("No AWS temporary credentials found")?; - - let options = [ - ("region", "us-west-2"), - ("access_key_id", &creds.access_key_id), - ("secret_access_key", &creds.secret_access_key), - ("session_token", &creds.session_token), - ]; - - let table_url = Url::parse(&table_uri)?; - let (store, path) = object_store::parse_url_opts(&table_url, options)?; - - info!("created object store: {:?}\npath: {:?}\n", store, path); - - let engine = DefaultEngineBuilder::new(store.into()).build(); - - // read table - let snapshot = catalog - .load_snapshot(&table_id, &table_uri, &engine) - .await?; - // or time travel - // let snapshot = catalog.load_snapshot_at(&table, 2).await?; - - println!("🎉 loaded snapshot: {snapshot:?}"); - - Ok(()) - } - - // ignored test which you can run manually to play around with writing to a UC table. run with: - // `ENDPOINT=".." TABLENAME=".." TOKEN=".." cargo t write_uc_table --nocapture -- --ignored` - #[ignore] - #[tokio::test(flavor = "multi_thread")] - async fn write_uc_table() -> Result<(), Box> { - let endpoint = env::var("ENDPOINT").expect("ENDPOINT environment variable not set"); - let token = env::var("TOKEN").expect("TOKEN environment variable not set"); - let table_name = env::var("TABLENAME").expect("TABLENAME environment variable not set"); - - // build shared config - let config = uc_client::ClientConfig::build(&endpoint, &token).build()?; - - // build clients - let client = UCClient::new(config.clone())?; - let commits_client = Arc::new(UCCommitsRestClient::new(config)?); - - let (table_id, table_uri) = get_table(&client, &table_name).await?; - let creds = client - .get_credentials(&table_id, Operation::ReadWrite) - .await - .map_err(|e| format!("Failed to get credentials: {e}"))?; - - let catalog = UCCatalog::new(commits_client.as_ref()); - - // TODO: support non-AWS - let creds = creds - .aws_temp_credentials - .ok_or("No AWS temporary credentials found")?; - - let options = [ - ("region", "us-west-2"), - ("access_key_id", &creds.access_key_id), - ("secret_access_key", &creds.secret_access_key), - ("session_token", &creds.session_token), - ]; - - let table_url = Url::parse(&table_uri)?; - let (store, _path) = object_store::parse_url_opts(&table_url, options)?; - let store = Arc::new(store); - - let engine = DefaultEngineBuilder::new(store.clone()).build(); - let committer = Box::new(UCCommitter::new(commits_client.clone(), table_id.clone())); - let snapshot = catalog - .load_snapshot(&table_id, &table_uri, &engine) - .await?; - println!("latest snapshot version: {:?}", snapshot.version()); - let txn = snapshot.clone().transaction(committer, &engine)?; - let _write_context = txn.get_write_context(); - - match txn.commit(&engine)? { - CommitResult::CommittedTransaction(t) => { - println!("🎉 committed version {}", t.commit_version()); - // TODO: should use post-commit snapshot here (plumb through log tail) - let _snapshot = catalog - .load_snapshot_at(&table_id, &table_uri, t.commit_version(), &engine) - .await?; - // then do publish - } - CommitResult::ConflictedTransaction(t) => { - println!("💥 commit conflicted at version {}", t.conflict_version()); - } - CommitResult::RetryableTransaction(_) => { - println!("we should retry..."); - } - } - Ok(()) - } -} diff --git a/uc-catalog/tests/e2e_in_memory.rs b/uc-catalog/tests/e2e_in_memory.rs deleted file mode 100644 index be01ac479b..0000000000 --- a/uc-catalog/tests/e2e_in_memory.rs +++ /dev/null @@ -1,215 +0,0 @@ -use std::path::PathBuf; -use std::sync::Arc; - -use delta_kernel::engine::default::executor::tokio::TokioMultiThreadExecutor; -use delta_kernel::engine::default::DefaultEngine; -use delta_kernel::object_store::local::LocalFileSystem; -use delta_kernel::transaction::CommitResult; -use delta_kernel::Snapshot; -use uc_catalog::{UCCatalog, UCCommitter}; -use uc_client::commits_client::{InMemoryCommitsClient, TableData}; -use uc_client::models::commits::Commit; - -// ============================================================================ -// Test Setup -// ============================================================================ - -type TestError = Box; - -const TABLE_ID: &str = "64dcd182-b3b4-4ee0-88e0-63c159a4121c"; - -/// Test fixtures: commits client, engine, snapshot at v2, and temp directory. -struct TestSetup { - commits_client: Arc, - engine: DefaultEngine, - snapshot: Arc, - table_uri: url::Url, - /// Tests must bind this field (not ignore with `..` or `_`) to prevent the temp directory - /// from being dropped and cleaned up before the test completes. - _tmp_dir: tempfile::TempDir, -} - -/// Copies test data to temp dir and loads snapshot at v2 with in-memory commits client. -async fn setup() -> Result { - let src = PathBuf::from("./tests/data/catalog_managed_0/"); - let tmp_dir = tempfile::tempdir()?; - copy_dir_recursive(&src, tmp_dir.path())?; - - // v0 published, v1/v2 ratified but unpublished - let commits_client = Arc::new(InMemoryCommitsClient::new()); - commits_client.insert_table( - TABLE_ID, - TableData { - max_ratified_version: 2, - catalog_commits: vec![ - Commit::new( - 1, - 1749830871085, - "00000000000000000001.4cb9708e-b478-44de-b203-53f9ba9b2876.json", - 889, - 1749830870833, - ), - Commit::new( - 2, - 1749830881799, - "00000000000000000002.5b9bba4a-0085-430d-a65e-b0d38c1afbe9.json", - 891, - 1749830881779, - ), - ], - }, - ); - - let store = Arc::new(LocalFileSystem::new()); - let executor = Arc::new(TokioMultiThreadExecutor::new( - tokio::runtime::Handle::current(), - )); - let engine = delta_kernel::engine::default::DefaultEngineBuilder::new(store) - .with_task_executor(executor) - .build(); - let table_uri = url::Url::from_directory_path(tmp_dir.path()).map_err(|_| "invalid path")?; - let snapshot = UCCatalog::new(commits_client.as_ref()) - .load_snapshot_at(TABLE_ID, table_uri.as_str(), 2, &engine) - .await?; - - Ok(TestSetup { - commits_client, - engine, - snapshot, - table_uri, - _tmp_dir: tmp_dir, - }) -} - -/// Recursively copies a directory tree. -fn copy_dir_recursive(src: &std::path::Path, dst: &std::path::Path) -> std::io::Result<()> { - std::fs::create_dir_all(dst)?; - for entry in std::fs::read_dir(src)? { - let entry = entry?; - let dst_path = dst.join(entry.file_name()); - if entry.path().is_dir() { - copy_dir_recursive(&entry.path(), &dst_path)?; - } else { - std::fs::copy(entry.path(), &dst_path)?; - } - } - Ok(()) -} - -/// Commits an empty transaction and returns the post-commit snapshot. -fn commit( - snapshot: &Arc, - commits_client: &Arc, - engine: &DefaultEngine, -) -> Result, TestError> { - let committer = Box::new(UCCommitter::new(commits_client.clone(), TABLE_ID)); - match snapshot - .clone() - .transaction(committer, engine)? - .commit(engine)? - { - CommitResult::CommittedTransaction(t) => Ok(t - .post_commit_snapshot() - .ok_or("no post commit snapshot")? - .clone()), - _ => Err("Expected committed transaction".into()), - } -} - -// ============================================================================ -// Tests -// ============================================================================ - -// multi_thread required: UCCommitter uses block_on which panics on single-threaded runtime -#[tokio::test(flavor = "multi_thread")] -async fn test_insert_and_publish() -> Result<(), TestError> { - let TestSetup { - commits_client, - engine, - mut snapshot, - table_uri: _, - _tmp_dir, - } = setup().await?; - assert_eq!(snapshot.version(), 2); - - let beyond_max = TableData::MAX_UNPUBLISHED_COMMITS as u64 + 5; - - for _ in 3..=beyond_max { - snapshot = commit(&snapshot, &commits_client, &engine)?; - - let committer = UCCommitter::new(commits_client.clone(), TABLE_ID); - - snapshot = snapshot.publish(&engine, &committer)?; - } - Ok(()) -} - -#[tokio::test(flavor = "multi_thread")] -async fn test_insert_without_publish_hits_limit() -> Result<(), TestError> { - let TestSetup { - commits_client, - engine, - mut snapshot, - table_uri: _, - _tmp_dir, - } = setup().await?; - - // Start with 2 unpublished (v1, v2). Insert up to MAX, then the next should fail. - let max = TableData::MAX_UNPUBLISHED_COMMITS as u64; - for _ in 3..=max { - snapshot = commit(&snapshot, &commits_client, &engine)?; - } - assert_eq!(snapshot.version(), max); - - // Next insert should fail with MaxUnpublishedCommitsExceeded - let committer = Box::new(UCCommitter::new(commits_client.clone(), TABLE_ID)); - let err = snapshot - .clone() - .transaction(committer, &engine)? - .commit(&engine) - .unwrap_err(); - assert!( - matches!(err, delta_kernel::Error::Generic(msg) if msg.contains("Max unpublished commits")) - ); - Ok(()) -} - -#[tokio::test(flavor = "multi_thread")] -async fn test_checkpoint_after_publish() -> Result<(), TestError> { - let TestSetup { - commits_client, - engine, - snapshot, - table_uri, - _tmp_dir, - } = setup().await?; - - let committer = UCCommitter::new(commits_client.clone(), TABLE_ID); - - commit(&snapshot, &commits_client, &engine)? - .publish(&engine, &committer)? - .checkpoint(&engine)?; - - // Load a fresh snapshot and verify checkpoint was written - let snapshot = Snapshot::builder_for(table_uri).build(&engine)?; - assert_eq!(snapshot.log_segment().checkpoint_version, Some(3)); - - Ok(()) -} - -#[tokio::test(flavor = "multi_thread")] -async fn test_cannot_checkpoint_unpublished_snapshot() -> Result<(), TestError> { - let TestSetup { - commits_client, - engine, - snapshot, - table_uri: _, - _tmp_dir, - } = setup().await?; - - let snapshot = commit(&snapshot, &commits_client, &engine)?; - - let err = snapshot.checkpoint(&engine).unwrap_err(); - assert!(matches!(err, delta_kernel::Error::Generic(msg) if msg.contains("not published"))); - Ok(()) -} diff --git a/uc-client/README.md b/uc-client/README.md deleted file mode 100644 index 09d6026b7d..0000000000 --- a/uc-client/README.md +++ /dev/null @@ -1,37 +0,0 @@ -# uc-client - -An experimental/under-construction rust client for Unity Catalog. This crate is not intended for -production use. - -## Example CLI -This crate provides a command-line interface (CLI) to interact with Unity Catalog APIs, see -`uc-client/examples/uc-cli.rs`. - -```bash -# set environment variables for UC url/token -UC_WORKSPACE_URL=https://some.uc.org -UC_TOKEN=your-token - -# or set them in each command like -cargo run --example uc-cli -- --workspace-url --token table catalog.schema.table - -# there are 3 operations: -# 1. get table metadata -cargo run --example uc-cli -- table catalog.schema.table - -# 2. get commits for a table (automatically resolves table id and storage location) -cargo run --example uc-cli -- commits catalog.schema.table - -# or get commits with version range -cargo run --example uc-cli -- commits catalog.schema.table \ - --start-version 0 \ - --end-version 10 - -# 3. get temporary credentials -cargo run --example uc-cli -- credentials \ - --table-id "table-uuid" \ - --operation READ - -# also can enable verbose logging -cargo run --example uc-cli -- --verbose table catalog.schema.table -``` \ No newline at end of file diff --git a/uc-client/examples/uc-cli.rs b/uc-client/examples/uc-cli.rs deleted file mode 100644 index 9637d3f95d..0000000000 --- a/uc-client/examples/uc-cli.rs +++ /dev/null @@ -1,265 +0,0 @@ -use clap::{Parser, Subcommand}; -use std::time::Duration; -use tracing_subscriber::{fmt, prelude::*, EnvFilter}; -use uc_client::{ - models::{commits::Commit, credentials::Operation, CommitsRequest}, - UCClient, UCCommitsRestClient, UCGetCommitsClient, -}; - -#[derive(Parser)] -#[command(name = "uc-client")] -#[command(about = "Unity Catalog CLI client", long_about = None)] -struct Cli { - /// Unity Catalog URL - #[arg(short, long, env = "UC_WORKSPACE_URL")] - workspace_url: String, - - /// Authentication token - #[arg(short, long, env = "UC_TOKEN", hide_env_values = true)] - token: String, - - /// Enable verbose logging - #[arg(short, long, default_value_t = false)] - verbose: bool, - - #[command(subcommand)] - command: Commands, -} - -#[derive(Subcommand)] -enum Commands { - /// Get table metadata - Table { - /// Full table name (catalog.schema.table) - #[arg(value_name = "TABLE")] - name: String, - }, - - /// Get commits for a table - Commits { - /// Full table name (catalog.schema.table) - #[arg(value_name = "TABLE")] - name: String, - - /// Start version (optional) - #[arg(short, long)] - start_version: Option, - - /// End version (optional) - #[arg(short, long)] - end_version: Option, - }, - - /// Get temporary credentials for a table - Credentials { - /// Table ID - #[arg(short, long)] - table_id: String, - - /// Operation type (READ, WRITE, or READ_WRITE) - #[arg(short, long, value_parser = parse_operation, default_value = "READ")] - operation: Operation, - }, -} - -fn parse_operation(s: &str) -> Result { - match s.to_uppercase().as_str() { - "READ" => Ok(Operation::Read), - "WRITE" => Ok(Operation::Write), - "READ_WRITE" | "READWRITE" => Ok(Operation::ReadWrite), - _ => Err(format!( - "Invalid operation '{s}'. Must be READ, WRITE, or READ_WRITE" - )), - } -} - -#[tokio::main] -async fn main() -> Result<(), Box> { - let cli = Cli::parse(); - - // Setup logging - let filter_level = if cli.verbose { "debug" } else { "info" }; - tracing_subscriber::registry() - .with(fmt::layer()) - .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(filter_level))) - .init(); - - // Create shared config - let config = uc_client::ClientConfig::build(&cli.workspace_url, &cli.token) - .with_timeout(Duration::from_secs(60)) - .with_max_retries(3) - .build()?; - - // Create shared HTTP client and UC clients - let http_client = uc_client::http::build_http_client(&config)?; - let uc_client = UCClient::with_http_client(http_client.clone(), config.clone()); - let uc_commits_client = UCCommitsRestClient::with_http_client(http_client, config); - - // Execute command - match cli.command { - Commands::Table { name } => { - println!("Fetching table metadata for: {name}"); - - match uc_client.get_table(&name).await { - Ok(table) => { - println!("\n✓ Table metadata retrieved successfully\n"); - println!("{table}"); - } - Err(e) => { - eprintln!("✗ Failed to get table: {e}"); - std::process::exit(1); - } - } - } - - Commands::Commits { - name, - start_version, - end_version, - } => { - println!("Resolving table: {name}"); - - // First, get the table metadata to obtain table_id and storage_location - let table = match uc_client.get_table(&name).await { - Ok(table) => { - println!( - "✓ Table resolved: {} (ID: {})", - table.full_name(), - table.table_id - ); - table - } - Err(e) => { - eprintln!("✗ Failed to resolve table: {e}"); - std::process::exit(1); - } - }; - - println!("Fetching commits for table..."); - - let mut request = CommitsRequest::new(&table.table_id, &table.storage_location); - - // Set start_version - default to 0 if not provided - let start = start_version.unwrap_or(0); - request = request.with_start_version(start); - - if let Some(end) = end_version { - request = request.with_end_version(end); - } - - match uc_commits_client.get_commits(request).await { - Ok(response) => { - println!("\n✓ Commits retrieved successfully\n"); - println!("Table: {}", table.full_name()); - println!("Latest Version: {}", response.latest_table_version); - - if let Some(commits) = response.commits { - println!(); - print_commits(&commits); - } else { - println!("No commits found in the specified range"); - } - } - Err(e) => { - eprintln!("✗ Failed to get commits: {e}"); - std::process::exit(1); - } - } - } - - Commands::Credentials { - table_id, - operation, - } => { - println!("Getting {operation} credentials for table_id: {table_id}"); - - match uc_client.get_credentials(&table_id, operation).await { - Ok(creds) => { - println!("\n✓ Credentials retrieved successfully\n"); - println!("URL: {}", creds.url); - let expires_str = creds - .expiration_as_datetime() - .map(|dt| dt.format("%Y-%m-%d %H:%M:%S UTC").to_string()) - .unwrap_or_else(|| format!("Invalid timestamp: {}", creds.expiration_time)); - println!("Expires at: {expires_str}"); - - if let Some(time_left) = creds.time_until_expiry() { - let hours = time_left.num_hours(); - let minutes = time_left.num_minutes() % 60; - println!("Time until expiry: {hours} hours {minutes} minutes"); - } else { - println!("Time until expiry: Unable to calculate"); - } - - if creds.is_expired() { - println!("⚠ WARNING: Credentials are already expired!"); - } - - if let Some(aws_creds) = &creds.aws_temp_credentials { - println!("\nAWS Credentials:"); - println!( - " Access Key ID: {}...", - &aws_creds.access_key_id[..10.min(aws_creds.access_key_id.len())] - ); - println!(" Secret Access Key: ***hidden***"); - println!( - " Session Token: {}...", - &aws_creds.session_token[..10.min(aws_creds.session_token.len())] - ); - } - } - Err(e) => { - eprintln!("✗ Failed to get credentials: {e}"); - std::process::exit(1); - } - } - } - } - - Ok(()) -} - -fn print_commits(commits: &[Commit]) { - println!("Commits ({} total):", commits.len()); - println!( - "{:<8} {:<20} {:<40} {:<15}", - "Version", "Timestamp", "File Name", "Size" - ); - println!("{}", "-".repeat(85)); - - for commit in commits { - let timestamp_str = commit - .timestamp_as_datetime() - .map(|dt| dt.format("%Y-%m-%d %H:%M:%S").to_string()) - .unwrap_or_else(|| format!("Invalid timestamp: {}", commit.timestamp)); - - println!( - "{:<8} {:<20} {:<40} {:<15}", - commit.version, - timestamp_str, - if commit.file_name.len() > 40 { - format!("{}...", &commit.file_name[..37]) - } else { - commit.file_name.clone() - }, - format_bytes(commit.file_size) - ); - } -} - -fn format_bytes(bytes: i64) -> String { - const UNITS: &[&str] = &["B", "KB", "MB", "GB", "TB"]; - let mut size = bytes as f64; - let mut unit_index = 0; - - while size >= 1024.0 && unit_index < UNITS.len() - 1 { - size /= 1024.0; - unit_index += 1; - } - - if unit_index == 0 { - format!("{} {}", size as i64, UNITS[unit_index]) - } else { - format!("{:.2} {}", size, UNITS[unit_index]) - } -} diff --git a/uc-client/src/client.rs b/uc-client/src/client.rs deleted file mode 100644 index 570e9a1a57..0000000000 --- a/uc-client/src/client.rs +++ /dev/null @@ -1,72 +0,0 @@ -use reqwest::StatusCode; -use tracing::instrument; -use url::Url; - -use crate::config::ClientConfig; -use crate::error::{Error, Result}; -use crate::http::{build_http_client, execute_with_retry, handle_response}; -use crate::models::credentials::{CredentialsRequest, Operation, TemporaryTableCredentials}; -use crate::models::tables::TablesResponse; - -/// An HTTP client for interacting with the Unity Catalog API. -#[derive(Debug, Clone)] -pub struct UCClient { - http_client: reqwest::Client, - config: ClientConfig, - base_url: Url, -} - -impl UCClient { - /// Create a new client from [ClientConfig]. - pub fn new(config: ClientConfig) -> Result { - Ok(Self { - http_client: build_http_client(&config)?, - base_url: config.workspace_url.clone(), - config, - }) - } - - /// Create from existing reqwest Client. - pub fn with_http_client(http_client: reqwest::Client, config: ClientConfig) -> Self { - Self { - base_url: config.workspace_url.clone(), - http_client, - config, - } - } - - /// Resolve the table by name. - #[instrument(skip(self))] - pub async fn get_table(&self, table_name: &str) -> Result { - let url = self.base_url.join(&format!("tables/{table_name}"))?; - - let response = - execute_with_retry(&self.config, || self.http_client.get(url.clone()).send()).await?; - - match response.status() { - StatusCode::NOT_FOUND => Err(Error::TableNotFound(table_name.to_string())), - _ => handle_response(response).await, - } - } - - /// Get temporary cloud storage credentials for accessing a table. - #[instrument(skip(self))] - pub async fn get_credentials( - &self, - table_id: &str, - operation: Operation, - ) -> Result { - let url = self.base_url.join("temporary-table-credentials")?; - - let request_body = CredentialsRequest::new(table_id, operation); - let response = execute_with_retry(&self.config, || { - self.http_client - .post(url.clone()) - .json(&request_body) - .send() - }) - .await?; - - handle_response(response).await - } -} diff --git a/uc-client/src/commits_client/in_memory.rs b/uc-client/src/commits_client/in_memory.rs deleted file mode 100644 index 0328270970..0000000000 --- a/uc-client/src/commits_client/in_memory.rs +++ /dev/null @@ -1,293 +0,0 @@ -//! In-memory implementation of [`UCCommitClient`] and [`UCGetCommitsClient`] for testing. - -use std::collections::hash_map::Entry; -use std::collections::HashMap; -use std::sync::RwLock; - -use crate::error::{Error, Result}; -use crate::models::commits::{Commit, CommitRequest, CommitsRequest, CommitsResponse}; - -use super::{UCCommitClient, UCGetCommitsClient}; - -// ============================================================================ -// TableData -// ============================================================================ - -/// In-memory representation of a UC-managed Delta table's commit state. -pub struct TableData { - /// The highest version that has been ratified (committed) to this table. - pub max_ratified_version: i64, - /// Commits that have been registered with UC but not yet published. - pub catalog_commits: Vec, -} - -impl TableData { - pub const MAX_UNPUBLISHED_COMMITS: usize = 20; - - /// Creates a new `TableData` representing a UC Delta table that has just been created. - /// The table starts with no commits and version 0. - fn new_post_table_create() -> Self { - Self { - max_ratified_version: 0, - catalog_commits: vec![], - } - } - - /// Returns commits within the requested version range. - fn get_commits(&self, request: CommitsRequest) -> Result { - let start = request.start_version.unwrap_or(0); - let end = request.end_version.unwrap_or(i64::MAX); - - Ok(CommitsResponse { - commits: Some( - self.catalog_commits - .iter() - .filter(|commit| start <= commit.version && commit.version <= end) - .cloned() - .collect(), - ), - latest_table_version: self.max_ratified_version, - }) - } - - /// Registers a new commit. Returns an error if the version is not the expected next version - /// or if the number of unpublished commits exceeds the maximum. - fn commit(&mut self, request: CommitRequest) -> Result<()> { - let Some(commit) = request.commit_info else { - return Err(Error::UnsupportedOperation( - "commit_info is required".to_string(), - )); - }; - - let expected_version = self.max_ratified_version + 1; - - if commit.version != expected_version { - return Err(Error::UnsupportedOperation(format!( - "Expected commit version {} but got {}", - expected_version, commit.version - ))); - } - - if self.catalog_commits.len() >= Self::MAX_UNPUBLISHED_COMMITS { - return Err(Error::MaxUnpublishedCommitsExceeded( - Self::MAX_UNPUBLISHED_COMMITS as u16, - )); - } - - if let Some(v) = request.latest_backfilled_version { - self.cleanup_published_commits(v); - } - - self.catalog_commits.push(commit); - self.max_ratified_version = expected_version; - - Ok(()) - } - - /// Removes commits that have been published (backfilled) to the Delta log. - fn cleanup_published_commits(&mut self, max_published_version: i64) { - self.catalog_commits - .retain(|commit| max_published_version < commit.version); - } -} - -// ============================================================================ -// InMemoryCommitsClient -// ============================================================================ - -/// An in-memory implementation of [`UCCommitClient`] and [`UCGetCommitsClient`] for testing. -pub struct InMemoryCommitsClient { - // table id -> table data - tables: RwLock>, -} - -impl InMemoryCommitsClient { - pub fn new() -> Self { - Self { - tables: RwLock::new(HashMap::new()), - } - } - - pub fn create_table(&self, table_id: impl Into) -> Result<()> { - let mut tables = self.tables.write().unwrap(); - match tables.entry(table_id.into()) { - Entry::Vacant(e) => { - e.insert(TableData::new_post_table_create()); - Ok(()) - } - Entry::Occupied(e) => Err(Error::UnsupportedOperation(format!( - "Table {} already exists", - e.key() - ))), - } - } - - /// Inserts a table with pre-existing state. Useful for testing. - pub fn insert_table(&self, table_id: impl Into, table_data: TableData) { - self.tables - .write() - .unwrap() - .insert(table_id.into(), table_data); - } -} - -impl Default for InMemoryCommitsClient { - fn default() -> Self { - Self::new() - } -} - -impl UCGetCommitsClient for InMemoryCommitsClient { - async fn get_commits(&self, request: CommitsRequest) -> Result { - let tables = self.tables.read().unwrap(); - let table = tables - .get(&request.table_id) - .ok_or_else(|| Error::TableNotFound(request.table_id.clone()))?; - table.get_commits(request) - } -} - -impl UCCommitClient for InMemoryCommitsClient { - async fn commit(&self, request: CommitRequest) -> Result<()> { - let mut tables = self.tables.write().unwrap(); - let table = tables - .get_mut(&request.table_id) - .ok_or_else(|| Error::TableNotFound(request.table_id.clone()))?; - table.commit(request) - } -} - -// ============================================================================ -// Tests -// ============================================================================ - -#[cfg(test)] -mod tests { - use super::*; - - const TABLE_ID: &str = "test-table-id"; - const TABLE_URI: &str = "s3://bucket/table"; - - fn make_commit(version: i64) -> Commit { - Commit::new( - version, - version * 1000, - format!("{version:020}.json"), - 100, - version * 1000, - ) - } - - fn commit_request(version: i64, latest_backfilled_version: Option) -> CommitRequest { - CommitRequest::new( - TABLE_ID, - TABLE_URI, - make_commit(version), - latest_backfilled_version, - ) - } - - fn get_commits_request() -> CommitsRequest { - CommitsRequest::new(TABLE_ID, TABLE_URI) - } - - fn extract_commit_versions(commits: &[Commit]) -> Vec { - commits.iter().map(|c| c.version).collect() - } - - #[tokio::test] - async fn test_commit_and_get_commits() { - let client = InMemoryCommitsClient::new(); - - // Create table - client.create_table(TABLE_ID).unwrap(); - - // Insert 10 commits (versions 1-10) - for v in 1..=10 { - client.commit(commit_request(v, None)).await.unwrap(); - } - - // Get commits (versions 3-8) - let commits_request = get_commits_request() - .with_start_version(3) - .with_end_version(8); - let response = client.get_commits(commits_request).await.unwrap(); - let commits = response.commits.unwrap(); - assert_eq!(commits.len(), 6); - assert_eq!(extract_commit_versions(&commits), vec![3, 4, 5, 6, 7, 8]); - assert_eq!(response.latest_table_version, 10); - - // Insert commit 11 with latest_backfilled_version = 5 - // This should cleanup commits 1-5 (retain versions >= 6) - client.commit(commit_request(11, Some(5))).await.unwrap(); - - // Get commits again - should return versions 6-11 - let response = client.get_commits(get_commits_request()).await.unwrap(); - let commits = response.commits.unwrap(); - assert_eq!(extract_commit_versions(&commits), vec![6, 7, 8, 9, 10, 11]); - assert_eq!(response.latest_table_version, 11); - } - - #[test] - fn test_create_table_duplicate_throws() { - let client = InMemoryCommitsClient::new(); - client.create_table(TABLE_ID).unwrap(); - assert!(matches!( - client.create_table(TABLE_ID), - Err(Error::UnsupportedOperation(_)) - )); - } - - #[tokio::test] - async fn test_get_commits_table_not_found() { - assert!(matches!( - InMemoryCommitsClient::new() - .get_commits(get_commits_request()) - .await, - Err(Error::TableNotFound(_)) - )); - } - - #[tokio::test] - async fn test_commit_table_not_found() { - assert!(matches!( - InMemoryCommitsClient::new() - .commit(commit_request(1, None)) - .await, - Err(Error::TableNotFound(_)) - )); - } - - #[tokio::test] - async fn test_commit_wrong_version() { - let client = InMemoryCommitsClient::new(); - client.create_table(TABLE_ID).unwrap(); - assert!(matches!( - client.commit(commit_request(5, None)).await, - Err(Error::UnsupportedOperation(_)) - )); - } - - #[tokio::test] - async fn test_get_commits_empty_table() { - let client = InMemoryCommitsClient::new(); - client.create_table(TABLE_ID).unwrap(); - let response = client.get_commits(get_commits_request()).await.unwrap(); - assert!(response.commits.unwrap().is_empty()); - assert_eq!(response.latest_table_version, 0); - } - - #[tokio::test] - async fn test_commit_max_unpublished_commits_exceeded() { - let client = InMemoryCommitsClient::new(); - client.create_table(TABLE_ID).unwrap(); - for v in 1..=TableData::MAX_UNPUBLISHED_COMMITS as i64 { - client.commit(commit_request(v, None)).await.unwrap(); - } - let next_version = TableData::MAX_UNPUBLISHED_COMMITS as i64 + 1; - assert!(matches!( - client.commit(commit_request(next_version, None)).await, - Err(Error::MaxUnpublishedCommitsExceeded(_)) - )); - } -} diff --git a/uc-client/src/commits_client/mod.rs b/uc-client/src/commits_client/mod.rs deleted file mode 100644 index 8562f1199a..0000000000 --- a/uc-client/src/commits_client/mod.rs +++ /dev/null @@ -1,99 +0,0 @@ -use serde::Deserialize; -use tracing::instrument; -use url::Url; - -use crate::config::ClientConfig; -use crate::error::Result; -use crate::http::{build_http_client, execute_with_retry, handle_response}; -use crate::models::commits::{CommitRequest, CommitsRequest, CommitsResponse}; - -#[cfg(any(test, feature = "test-utils"))] -mod in_memory; - -#[cfg(any(test, feature = "test-utils"))] -pub use in_memory::{InMemoryCommitsClient, TableData}; - -/// Trait for committing new versions to a UC-managed Delta table. -/// -/// Implementations of this trait are responsible for performing any necessary retries on transient -/// failures. This trait is designed to be injected into a `uc_catalog::committer::UCCommitter`, -/// which itself does not perform any retries and relies on the underlying client implementation to -/// handle retry logic. -#[allow(async_fn_in_trait)] -pub trait UCCommitClient: Send + Sync { - /// Commit a new version to the table. - async fn commit(&self, request: CommitRequest) -> Result<()>; -} - -/// Trait for retrieving commits from a UC-managed Delta table. -/// -/// Implementations of this trait are responsible for performing any necessary retries on transient -/// failures. -#[allow(async_fn_in_trait)] -pub trait UCGetCommitsClient: Send + Sync { - /// Get the latest commits for the table. - async fn get_commits(&self, request: CommitsRequest) -> Result; -} - -/// REST implementation of [UCCommitClient] and [UCGetCommitsClient]. -#[derive(Debug, Clone)] -pub struct UCCommitsRestClient { - http_client: reqwest::Client, - config: ClientConfig, - base_url: Url, -} - -impl UCCommitsRestClient { - /// Create from config. - pub fn new(config: ClientConfig) -> Result { - Ok(Self { - http_client: build_http_client(&config)?, - base_url: config.workspace_url.clone(), - config, - }) - } - - /// Create from existing reqwest Client. - pub fn with_http_client(http_client: reqwest::Client, config: ClientConfig) -> Self { - Self { - base_url: config.workspace_url.clone(), - http_client, - config, - } - } -} - -impl UCGetCommitsClient for UCCommitsRestClient { - #[instrument(skip(self))] - async fn get_commits(&self, request: CommitsRequest) -> Result { - let url = self.base_url.join("delta/preview/commits")?; - let response = execute_with_retry(&self.config, || { - self.http_client - .request(reqwest::Method::GET, url.clone()) - .json(&request) - .send() - }) - .await?; - - handle_response(response).await - } -} - -impl UCCommitClient for UCCommitsRestClient { - #[instrument(skip(self))] - async fn commit(&self, request: CommitRequest) -> Result<()> { - let url = self.base_url.join("delta/preview/commits")?; - let response = execute_with_retry(&self.config, || { - self.http_client - .request(reqwest::Method::POST, url.clone()) - .json(&request) - .send() - }) - .await?; - - #[derive(Deserialize)] - struct EmptyResponse {} - let _: EmptyResponse = handle_response(response).await?; - Ok(()) - } -} diff --git a/uc-client/src/error.rs b/uc-client/src/error.rs deleted file mode 100644 index b63dc77792..0000000000 --- a/uc-client/src/error.rs +++ /dev/null @@ -1,42 +0,0 @@ -use thiserror::Error; - -#[derive(Error, Debug)] -pub enum Error { - #[error("HTTP request failed: {0}")] - Http(#[from] reqwest::Error), - - #[error("Invalid header value: {0}")] - InvalidHeaderValue(#[from] reqwest::header::InvalidHeaderValue), - - #[error("URL parse error: {0}")] - UrlParse(#[from] url::ParseError), - - #[error("Serialization error: {0}")] - Serialization(#[from] serde_json::Error), - - #[error("API error (status {status}): {message}")] - ApiError { status: u16, message: String }, - - #[error("Table not found: {0}")] - TableNotFound(String), - - #[error("Invalid configuration: {0}")] - InvalidConfiguration(String), - - #[error("Authentication failed")] - AuthenticationFailed, - - #[error("Operation not supported: {0}")] - UnsupportedOperation(String), - - #[error("Max retries exceeded")] - MaxRetriesExceeded, - - #[error("Max unpublished commits exceeded (max: {0})")] - MaxUnpublishedCommitsExceeded(u16), - - #[error("Generic Error: {0}")] - Generic(String), -} - -pub type Result = std::result::Result; diff --git a/uc-client/src/lib.rs b/uc-client/src/lib.rs deleted file mode 100644 index 0a3bdec207..0000000000 --- a/uc-client/src/lib.rs +++ /dev/null @@ -1,49 +0,0 @@ -//! Unity Catalog Client for Rust -//! -//! This crate provides a Rust client for interacting with Unity Catalog APIs. -//! -//! # Example -//! -//! ```no_run -//! use uc_client::{ClientConfig, UCCommitsRestClient, UCGetCommitsClient, models::CommitsRequest}; -//! -//! #[tokio::main] -//! async fn main() -> Result<(), Box> { -//! let config = ClientConfig::build("uc.awesome.org", "your-token").build()?; -//! let client = UCCommitsRestClient::new(config)?; -//! -//! let request = CommitsRequest::new("table-id", "table-uri"); -//! let commits = client.get_commits(request).await?; -//! -//! Ok(()) -//! } -//! ``` - -pub mod client; -pub mod commits_client; -pub mod config; -pub mod error; -pub mod http; -pub mod models; - -#[cfg(test)] -mod tests; - -pub use client::UCClient; -pub use commits_client::{UCCommitClient, UCCommitsRestClient, UCGetCommitsClient}; -pub use config::{ClientConfig, ClientConfigBuilder}; -pub use error::{Error, Result}; - -#[cfg(any(test, feature = "test-utils"))] -pub use commits_client::InMemoryCommitsClient; - -#[doc(hidden)] -pub mod prelude { - pub use crate::client::UCClient; - pub use crate::commits_client::{UCCommitClient, UCCommitsRestClient, UCGetCommitsClient}; - pub use crate::models::{ - commits::{Commit, CommitsRequest, CommitsResponse}, - credentials::{Operation, TemporaryTableCredentials}, - tables::TablesResponse, - }; -} diff --git a/uc-client/src/models/commits.rs b/uc-client/src/models/commits.rs deleted file mode 100644 index 4c1a2b541f..0000000000 --- a/uc-client/src/models/commits.rs +++ /dev/null @@ -1,111 +0,0 @@ -use serde::{Deserialize, Serialize}; - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct CommitsRequest { - pub table_id: String, - pub table_uri: String, - #[serde(skip_serializing_if = "Option::is_none")] - pub start_version: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub end_version: Option, -} - -impl CommitsRequest { - pub fn new(table_id: impl Into, table_uri: impl Into) -> Self { - Self { - table_id: table_id.into(), - table_uri: table_uri.into(), - start_version: None, - end_version: None, - } - } - - pub fn with_start_version(mut self, version: i64) -> Self { - self.start_version = Some(version); - self - } - - pub fn with_end_version(mut self, version: i64) -> Self { - self.end_version = Some(version); - self - } -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct CommitsResponse { - #[serde(skip_serializing_if = "Option::is_none")] - pub commits: Option>, - pub latest_table_version: i64, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct Commit { - pub version: i64, - pub timestamp: i64, - pub file_name: String, - pub file_size: i64, - pub file_modification_timestamp: i64, -} - -impl Commit { - /// Create a new commit to send to UC with the specified version and timestamp. - pub fn new( - version: i64, - timestamp: i64, - file_name: impl Into, - file_size: i64, - file_modification_timestamp: i64, - ) -> Self { - Self { - version, - timestamp, - file_name: file_name.into(), - file_size, - file_modification_timestamp, - } - } - - pub fn timestamp_as_datetime(&self) -> Option> { - chrono::DateTime::from_timestamp_millis(self.timestamp) - } - - pub fn file_modification_as_datetime(&self) -> Option> { - chrono::DateTime::from_timestamp_millis(self.file_modification_timestamp) - } -} - -/// Request to commit a new version to the table. It must include either a `commit_info` or -/// `latest_backfilled_version`. -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct CommitRequest { - pub table_id: String, - pub table_uri: String, - #[serde(skip_serializing_if = "Option::is_none")] - pub commit_info: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub latest_backfilled_version: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub metadata: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub protocol: Option, -} - -impl CommitRequest { - pub fn new( - table_id: impl Into, - table_uri: impl Into, - commit_info: Commit, - latest_backfilled_version: Option, - ) -> Self { - Self { - table_id: table_id.into(), - table_uri: table_uri.into(), - commit_info: Some(commit_info), - latest_backfilled_version, - metadata: None, - protocol: None, - } - } - - // TODO: expose metadata/protocol (with_metadata, with_protocol) -} diff --git a/uc-client/src/models/credentials.rs b/uc-client/src/models/credentials.rs deleted file mode 100644 index 288d6a683f..0000000000 --- a/uc-client/src/models/credentials.rs +++ /dev/null @@ -1,67 +0,0 @@ -use serde::{Deserialize, Serialize}; - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct TemporaryTableCredentials { - #[serde(skip_serializing_if = "Option::is_none")] - pub aws_temp_credentials: Option, - pub expiration_time: i64, - pub url: String, -} - -impl TemporaryTableCredentials { - pub fn expiration_as_datetime(&self) -> Option> { - chrono::DateTime::from_timestamp_millis(self.expiration_time) - } - - pub fn is_expired(&self) -> bool { - // If we can't parse the timestamp, consider it expired for safety - self.expiration_as_datetime() - .is_none_or(|exp| exp < chrono::Utc::now()) - } - - pub fn time_until_expiry(&self) -> Option { - self.expiration_as_datetime() - .map(|exp| exp - chrono::Utc::now()) - } -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct AwsTempCredentials { - pub access_key_id: String, - pub secret_access_key: String, - pub session_token: String, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(rename_all = "UPPERCASE")] -pub enum Operation { - Read, - Write, - #[serde(rename = "READ_WRITE")] - ReadWrite, -} - -impl std::fmt::Display for Operation { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - match self { - Operation::Read => write!(f, "READ"), - Operation::Write => write!(f, "WRITE"), - Operation::ReadWrite => write!(f, "READ_WRITE"), - } - } -} - -#[derive(Debug, Clone, Serialize)] -pub struct CredentialsRequest { - pub table_id: String, - pub operation: Operation, -} - -impl CredentialsRequest { - pub fn new(table_id: impl Into, operation: Operation) -> Self { - Self { - table_id: table_id.into(), - operation, - } - } -} diff --git a/uc-client/src/models/mod.rs b/uc-client/src/models/mod.rs deleted file mode 100644 index c6048a7d72..0000000000 --- a/uc-client/src/models/mod.rs +++ /dev/null @@ -1,7 +0,0 @@ -pub mod commits; -pub mod credentials; -pub mod tables; - -pub use commits::{Commit, CommitRequest, CommitsRequest, CommitsResponse}; -pub use credentials::{AwsTempCredentials, TemporaryTableCredentials}; -pub use tables::TablesResponse; diff --git a/uc-client/src/models/tables.rs b/uc-client/src/models/tables.rs deleted file mode 100644 index fc8c722c3d..0000000000 --- a/uc-client/src/models/tables.rs +++ /dev/null @@ -1,66 +0,0 @@ -use serde::{Deserialize, Serialize}; -use std::collections::HashMap; -use std::fmt::{self, Display}; - -// note this is a subset of actual get tables response -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct TablesResponse { - pub name: String, - pub catalog_name: String, - pub schema_name: String, - pub table_type: String, - pub data_source_format: String, - pub storage_location: String, - pub owner: String, - #[serde(default)] - pub properties: HashMap, - pub securable_kind: String, - pub metastore_id: String, - pub table_id: String, - pub schema_id: String, - pub catalog_id: String, -} - -impl TablesResponse { - pub fn full_name(&self) -> String { - format!("{}.{}.{}", self.catalog_name, self.schema_name, self.name) - } - - pub fn is_delta_table(&self) -> bool { - self.data_source_format.eq_ignore_ascii_case("delta") - } - - pub fn is_managed_table(&self) -> bool { - self.table_type.eq_ignore_ascii_case("managed") - } - - pub fn is_external_table(&self) -> bool { - self.table_type.eq_ignore_ascii_case("external") - } -} - -impl Display for TablesResponse { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - writeln!(f, "Full Name: {}", self.full_name())?; - writeln!(f, "Catalog: {}", self.catalog_name)?; - writeln!(f, "Schema: {}", self.schema_name)?; - writeln!(f, "Table Name: {}", self.name)?; - writeln!(f, "Type: {}", self.table_type)?; - writeln!(f, "Format: {}", self.data_source_format)?; - writeln!(f, "Storage Location: {}", self.storage_location)?; - writeln!(f, "Owner: {}", self.owner)?; - writeln!(f, "Table ID: {}", self.table_id)?; - writeln!(f, "Is Delta Table: {}", self.is_delta_table())?; - writeln!(f, "Is Managed: {}", self.is_managed_table())?; - - if !self.properties.is_empty() { - writeln!(f)?; - writeln!(f, "Properties:")?; - for (key, value) in &self.properties { - writeln!(f, " {key}: {value}")?; - } - } - - Ok(()) - } -} diff --git a/uc-client/src/tests.rs b/uc-client/src/tests.rs deleted file mode 100644 index 2bb7bed44a..0000000000 --- a/uc-client/src/tests.rs +++ /dev/null @@ -1,48 +0,0 @@ -use crate::models::commits::CommitsRequest; -use crate::models::credentials::Operation; - -#[test] -fn test_commits_request_builder() { - let request = CommitsRequest::new("test-id", "test-uri") - .with_start_version(1) - .with_end_version(10); - - assert_eq!(request.table_id, "test-id"); - assert_eq!(request.table_uri, "test-uri"); - assert_eq!(request.start_version, Some(1)); - assert_eq!(request.end_version, Some(10)); -} - -#[test] -fn test_operation_display() { - assert_eq!(Operation::Read.to_string(), "READ"); - assert_eq!(Operation::Write.to_string(), "WRITE"); - assert_eq!(Operation::ReadWrite.to_string(), "READ_WRITE"); -} - -#[test] -fn test_table_response_helpers() { - use crate::models::tables::TablesResponse; - use std::collections::HashMap; - - let table = TablesResponse { - name: "my_table".to_string(), - catalog_name: "catalog".to_string(), - schema_name: "schema".to_string(), - table_type: "MANAGED".to_string(), - data_source_format: "DELTA".to_string(), - storage_location: "/path/to/table".to_string(), - owner: "user".to_string(), - properties: HashMap::new(), - securable_kind: "TABLE".to_string(), - metastore_id: "metastore-id".to_string(), - table_id: "table-id".to_string(), - schema_id: "schema-id".to_string(), - catalog_id: "catalog-id".to_string(), - }; - - assert_eq!(table.full_name(), "catalog.schema.my_table"); - assert!(table.is_delta_table()); - assert!(table.is_managed_table()); - assert!(!table.is_external_table()); -} diff --git a/uc-client/Cargo.toml b/unity-catalog-delta-client-api/Cargo.toml similarity index 57% rename from uc-client/Cargo.toml rename to unity-catalog-delta-client-api/Cargo.toml index f7a1b97f11..dee7ce181d 100644 --- a/uc-client/Cargo.toml +++ b/unity-catalog-delta-client-api/Cargo.toml @@ -1,32 +1,27 @@ [package] -name = "uc-client" +name = "unity-catalog-delta-client-api" +description = "Transport-agnostic client traits and wire models for the Unity Catalog Delta Tables API." edition.workspace = true homepage.workspace = true keywords.workspace = true license.workspace = true repository.workspace = true -readme.workspace = true +readme = "README.md" rust-version.workspace = true -version.workspace = true +version = "0.1.0" # for cargo-release [package.metadata.release] release = false [dependencies] -reqwest = { version = "0.13", features = ["http2", "json"] } serde = { version = "1.0", features = ["derive"] } serde_json = "1.0" -tokio = { version = "1", features = ["full"] } thiserror = "2.0" -tracing = "0.1" -url = "2.5" -chrono = { version = "0.4", features = ["serde"] } [features] default = [] test-utils = [] [dev-dependencies] -tracing-subscriber = { version = "0.3", features = ["env-filter"] } -clap = { version = "4.5", features = ["derive", "env"] } +tokio = { version = "1", features = ["full"] } diff --git a/unity-catalog-delta-client-api/README.md b/unity-catalog-delta-client-api/README.md new file mode 100644 index 0000000000..f77540fee7 --- /dev/null +++ b/unity-catalog-delta-client-api/README.md @@ -0,0 +1,24 @@ +# unity-catalog-delta-client-api + +> [!WARNING] +> This crate is experimental and under construction. It is not intended for production use. + +A crate defining the client API contract for the Unity Catalog Delta Tables API. + +It has no network dependencies, so any transport can implement it. The REST implementation lives in +[`unity-catalog-delta-rest-client`], and [`delta-kernel-unity-catalog`] depends on this crate rather +than on a concrete client. + +It provides: + +- `UpdateTableClient`: the trait for committing a new version to a UC-managed table, which + `delta-kernel-unity-catalog`'s `UCCommitter` dispatches through. +- Serde wire models for the UC Delta Tables endpoints: `load_table`, credential vending, `/config`, + table and staging-table creation, `update_table`, and metrics reporting. + +Only `update_table` (the commit RPC) sits behind a trait. Read and credential-vending flows are +connector-driven: connectors call concrete client methods, or bring their own HTTP plumbing, then +hand the responses to `delta-kernel-unity-catalog` helpers. + +[`unity-catalog-delta-rest-client`]: https://github.com/delta-io/delta-kernel-rs/tree/main/unity-catalog-delta-rest-client +[`delta-kernel-unity-catalog`]: https://github.com/delta-io/delta-kernel-rs/tree/main/delta-kernel-unity-catalog diff --git a/unity-catalog-delta-client-api/src/clients/in_memory.rs b/unity-catalog-delta-client-api/src/clients/in_memory.rs new file mode 100644 index 0000000000..cea51b0cd0 --- /dev/null +++ b/unity-catalog-delta-client-api/src/clients/in_memory.rs @@ -0,0 +1,302 @@ +//! In-memory implementation of [`UpdateTableClient`] for testing. + +use std::collections::hash_map::Entry; +use std::collections::HashMap; +use std::sync::RwLock; + +use super::UpdateTableClient; +use crate::error::{Error, Result}; +use crate::models::{ + Commit, LoadTableResponse, TableIdentifier, TableMetadata, UpdateTableRequest, +}; + +// ============================================================================ +// TableData +// ============================================================================ + +/// In-memory representation of a UC-managed Delta table's commit state. +pub struct TableData { + /// The highest version that has been ratified (committed) to this table. + pub max_ratified_version: i64, + /// Commits that have been registered with UC but not yet published. + pub catalog_commits: Vec, +} + +impl TableData { + pub const MAX_UNPUBLISHED_COMMITS: usize = 20; + + /// Creates a new `TableData` representing a UC Delta table that has just been created. + /// The table starts with no commits and version 0. + fn new_post_table_create() -> Self { + Self { + max_ratified_version: 0, + catalog_commits: vec![], + } + } + + /// Apply the typed `add-commit` and (optional) `set-latest-backfilled-version` + /// updates from an `UpdateTableRequest` against this table's state. + fn update_table(&mut self, request: &UpdateTableRequest) -> Result<()> { + let backfilled = request.latest_backfilled_version(); + + let commit = request.staged_commit().cloned().ok_or_else(|| { + Error::UnsupportedOperation("add-commit update is required".to_string()) + })?; + + let expected_version = self.max_ratified_version + 1; + if commit.version != expected_version { + return Err(Error::UnsupportedOperation(format!( + "Expected commit version {expected_version} but got {}", + commit.version + ))); + } + if self.catalog_commits.len() >= Self::MAX_UNPUBLISHED_COMMITS { + return Err(Error::MaxUnpublishedCommitsExceeded( + Self::MAX_UNPUBLISHED_COMMITS as u16, + )); + } + if let Some(v) = backfilled { + self.cleanup_published_commits(v); + } + + self.catalog_commits.push(commit); + self.max_ratified_version = expected_version; + + Ok(()) + } + + /// Removes commits that have been published (backfilled) to the Delta log. + fn cleanup_published_commits(&mut self, max_published_version: i64) { + self.catalog_commits + .retain(|commit| max_published_version < commit.version); + } +} + +// ============================================================================ +// InMemoryUpdateTableClient +// ============================================================================ + +/// An in-memory implementation of [`UpdateTableClient`] for testing. +pub struct InMemoryUpdateTableClient { + // table id -> table data + tables: RwLock>, +} + +impl InMemoryUpdateTableClient { + pub fn new() -> Self { + Self { + tables: RwLock::new(HashMap::new()), + } + } + + pub fn create_table(&self, table_id: impl Into) -> Result<()> { + let mut tables = self.tables.write().unwrap(); + match tables.entry(table_id.into()) { + Entry::Vacant(e) => { + e.insert(TableData::new_post_table_create()); + Ok(()) + } + Entry::Occupied(e) => Err(Error::UnsupportedOperation(format!( + "Table {} already exists", + e.key() + ))), + } + } + + /// Inserts a table with pre-existing state. Useful for testing. + pub fn insert_table(&self, table_id: impl Into, table_data: TableData) { + self.tables + .write() + .unwrap() + .insert(table_id.into(), table_data); + } + + /// Build a `LoadTableResponse` for the given table id. Lets tests stand in + /// for the connector's `load_table` HTTP call. + pub fn load_table_response( + &self, + table_id: &str, + storage_location: impl Into, + ) -> Result { + let tables = self.tables.read().unwrap(); + let table = tables + .get(table_id) + .ok_or_else(|| Error::TableNotFound(table_id.to_string()))?; + let metadata = TableMetadata { + etag: String::new(), + table_type: "MANAGED".to_string(), + table_uuid: table_id.to_string(), + location: storage_location.into(), + created_time: 0, + updated_time: 0, + columns: serde_json::json!({ "type": "struct", "fields": [] }), + partition_columns: vec![], + properties: std::collections::HashMap::new(), + last_commit_version: Some(table.max_ratified_version), + last_commit_timestamp_ms: Some(0), + }; + Ok(LoadTableResponse { + metadata, + commits: table.catalog_commits.clone(), + uniform: None, + latest_table_version: Some(table.max_ratified_version), + }) + } +} + +impl Default for InMemoryUpdateTableClient { + fn default() -> Self { + Self::new() + } +} + +impl UpdateTableClient for InMemoryUpdateTableClient { + async fn update_table( + &self, + _target: &TableIdentifier, + request: UpdateTableRequest, + ) -> Result<()> { + // Identify the target table from the assert-table-uuid requirement. + let table_id = request + .table_uuid() + .ok_or_else(|| { + Error::UnsupportedOperation( + "InMemoryUpdateTableClient requires an assert-table-uuid requirement" + .to_string(), + ) + })? + .to_string(); + + let mut tables = self.tables.write().unwrap(); + let table = tables + .get_mut(&table_id) + .ok_or_else(|| Error::TableNotFound(table_id.clone()))?; + table.update_table(&request) + } +} + +// ============================================================================ +// Tests +// ============================================================================ + +#[cfg(test)] +mod tests { + use super::*; + use crate::models::{DeltaTableRequirement, DeltaTableUpdate}; + + const TABLE_ID: &str = "test-table-id"; + + fn make_commit(version: i64) -> Commit { + Commit::new( + version, + version * 1000, + format!("{version:020}.json"), + 100, + version * 1000, + ) + } + + fn target() -> TableIdentifier { + TableIdentifier::new("test_catalog", "test_schema", "test_table") + } + + fn commit_request(version: i64, latest_backfilled_version: Option) -> UpdateTableRequest { + let mut updates = vec![DeltaTableUpdate::AddCommit { + commit: make_commit(version), + }]; + if let Some(v) = latest_backfilled_version { + updates.push(DeltaTableUpdate::SetLatestBackfilledVersion { + latest_published_version: v, + }); + } + UpdateTableRequest::new( + vec![DeltaTableRequirement::AssertTableUuid { + uuid: TABLE_ID.to_string(), + }], + updates, + ) + .unwrap() + } + + #[tokio::test] + async fn test_commit_and_load_table_response() { + let client = InMemoryUpdateTableClient::new(); + client.create_table(TABLE_ID).unwrap(); + + for v in 1..=10 { + client + .update_table(&target(), commit_request(v, None)) + .await + .unwrap(); + } + + let resp = client + .load_table_response(TABLE_ID, "memory:///tbl/") + .unwrap(); + assert_eq!(resp.latest_table_version, Some(10)); + assert_eq!(resp.commits.len(), 10); + + // Backfill cleanup: commit 11 with backfilled=5 keeps versions 6..=11 + client + .update_table(&target(), commit_request(11, Some(5))) + .await + .unwrap(); + let resp = client + .load_table_response(TABLE_ID, "memory:///tbl/") + .unwrap(); + let versions: Vec = resp.commits.iter().map(|c| c.version).collect(); + assert_eq!(versions, vec![6, 7, 8, 9, 10, 11]); + assert_eq!(resp.latest_table_version, Some(11)); + } + + #[test] + fn test_create_table_duplicate_throws() { + let client = InMemoryUpdateTableClient::new(); + client.create_table(TABLE_ID).unwrap(); + assert!(matches!( + client.create_table(TABLE_ID), + Err(Error::UnsupportedOperation(_)) + )); + } + + #[tokio::test] + async fn test_commit_table_not_found() { + assert!(matches!( + InMemoryUpdateTableClient::new() + .update_table(&target(), commit_request(1, None)) + .await, + Err(Error::TableNotFound(_)) + )); + } + + #[tokio::test] + async fn test_commit_wrong_version() { + let client = InMemoryUpdateTableClient::new(); + client.create_table(TABLE_ID).unwrap(); + assert!(matches!( + client + .update_table(&target(), commit_request(5, None)) + .await, + Err(Error::UnsupportedOperation(_)) + )); + } + + #[tokio::test] + async fn test_commit_max_unpublished_commits_exceeded() { + let client = InMemoryUpdateTableClient::new(); + client.create_table(TABLE_ID).unwrap(); + for v in 1..=TableData::MAX_UNPUBLISHED_COMMITS as i64 { + client + .update_table(&target(), commit_request(v, None)) + .await + .unwrap(); + } + let next_version = TableData::MAX_UNPUBLISHED_COMMITS as i64 + 1; + assert!(matches!( + client + .update_table(&target(), commit_request(next_version, None)) + .await, + Err(Error::MaxUnpublishedCommitsExceeded(_)) + )); + } +} diff --git a/unity-catalog-delta-client-api/src/clients/mod.rs b/unity-catalog-delta-client-api/src/clients/mod.rs new file mode 100644 index 0000000000..1fde03a3f6 --- /dev/null +++ b/unity-catalog-delta-client-api/src/clients/mod.rs @@ -0,0 +1,22 @@ +use crate::error::Result; +use crate::models::{TableIdentifier, UpdateTableRequest}; + +#[cfg(any(test, feature = "test-utils"))] +mod in_memory; + +#[cfg(any(test, feature = "test-utils"))] +pub use in_memory::{InMemoryUpdateTableClient, TableData}; + +/// Trait for committing new versions to a UC-managed Delta table via the +/// `update_table` endpoint. +/// +/// Implementations are responsible for any retry logic on transient failures. +#[allow(async_fn_in_trait)] +pub trait UpdateTableClient: Send + Sync { + /// Apply the typed `requirements + updates` payload atomically against `target`. + async fn update_table( + &self, + target: &TableIdentifier, + request: UpdateTableRequest, + ) -> Result<()>; +} diff --git a/unity-catalog-delta-client-api/src/credentials.rs b/unity-catalog-delta-client-api/src/credentials.rs new file mode 100644 index 0000000000..5943f54fe7 --- /dev/null +++ b/unity-catalog-delta-client-api/src/credentials.rs @@ -0,0 +1,111 @@ +use std::collections::HashMap; + +use serde::{Deserialize, Serialize}; + +#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "UPPERCASE")] +pub enum Operation { + Read, + #[serde(rename = "READ_WRITE")] + ReadWrite, +} + +impl std::fmt::Display for Operation { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Operation::Read => write!(f, "READ"), + Operation::ReadWrite => write!(f, "READ_WRITE"), + } + } +} + +/// Response from a credential-vending endpoint. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct CredentialsResponse { + #[serde(rename = "storage-credentials")] + pub storage_credentials: Vec, +} + +/// A single temporary cloud-storage credential scoped to a storage prefix. +#[derive(Clone, Serialize, Deserialize)] +pub struct StorageCredential { + /// Storage path prefix this credential applies to, e.g. `"s3://bucket/path/"`. + pub prefix: String, + /// Permission level granted by this credential. + pub operation: Operation, + /// Credential expiration time in epoch milliseconds, or `None` when the server omits it. + #[serde(rename = "expiration-time-ms")] + pub expiration_time_ms: Option, + /// Cloud provider-specific credential configuration (e.g. `s3.access-key-id`, + /// `s3.secret-access-key`, `s3.session-token`, `azure.sas-token`, `gcs.oauth-token`). + pub config: HashMap, +} + +// Manual `Debug` that redacts `config`: it holds live secrets (`s3.secret-access-key`, +// `azure.sas-token`, `gcs.oauth-token`) that must not leak into logs or error output. +impl std::fmt::Debug for StorageCredential { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("StorageCredential") + .field("prefix", &self.prefix) + .field("operation", &self.operation) + .field("expiration_time_ms", &self.expiration_time_ms) + .field( + "config", + &format_args!("<{} redacted entries>", self.config.len()), + ) + .finish() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn storage_credential_decodes_empty_config_and_absent_expiration() { + // Local `file://` storage: the server sends an empty `config` object (no cloud keys + // apply) and omits the nullable `expiration-time-ms`. + let json = r#"{"prefix":"file:///tmp/t/","operation":"READ_WRITE","config":{}}"#; + let cred: StorageCredential = serde_json::from_str(json).unwrap(); + assert_eq!(cred.operation, Operation::ReadWrite); + assert_eq!(cred.expiration_time_ms, None); + assert!(cred.config.is_empty()); + } + + #[test] + fn storage_credential_debug_redacts_config_secrets() { + let cred = StorageCredential { + prefix: "s3://b/t/".to_string(), + operation: Operation::Read, + expiration_time_ms: Some(123), + config: HashMap::from([( + "s3.secret-access-key".to_string(), + "supersecret".to_string(), + )]), + }; + let debug = format!("{cred:?}"); + assert!( + !debug.contains("supersecret"), + "secret leaked in Debug: {debug}" + ); + assert!(debug.contains("redacted")); + } + + #[test] + fn credentials_response_decodes_populated_body() { + let json = r#"{"storage-credentials":[{ + "prefix":"s3://b/t/", + "operation":"READ", + "expiration-time-ms":123, + "config":{"s3.access-key-id":"ak","s3.secret-access-key":"sk"} + }]}"#; + let resp: CredentialsResponse = serde_json::from_str(json).unwrap(); + assert_eq!(resp.storage_credentials.len(), 1); + let cred = &resp.storage_credentials[0]; + assert_eq!(cred.prefix, "s3://b/t/"); + assert_eq!(cred.operation, Operation::Read); + assert_eq!(cred.expiration_time_ms, Some(123)); + assert_eq!(cred.config["s3.access-key-id"], "ak"); + assert_eq!(cred.config["s3.secret-access-key"], "sk"); + } +} diff --git a/unity-catalog-delta-client-api/src/error.rs b/unity-catalog-delta-client-api/src/error.rs new file mode 100644 index 0000000000..6c9140db0a --- /dev/null +++ b/unity-catalog-delta-client-api/src/error.rs @@ -0,0 +1,34 @@ +use thiserror::Error; + +/// Errors for Unity Catalog client API traits. +/// +/// This error type contains no HTTP-specific variants, allowing any backend +/// (REST, gRPC, in-memory, etc.) to implement the traits without pulling in +/// implementation-specific dependencies. +#[derive(Error, Debug)] +pub enum Error { + /// The requested table was not found. + #[error("Table not found: {0}")] + TableNotFound(String), + + /// The number of unpublished commits has exceeded the maximum allowed. + #[error("Max unpublished commits exceeded (max: {0})")] + MaxUnpublishedCommitsExceeded(u16), + + /// The requested operation is not supported. + #[error("Operation not supported: {0}")] + UnsupportedOperation(String), + + /// Authentication with Unity Catalog failed. + #[error("Authentication failed")] + AuthenticationFailed, + + /// A generic error with a descriptive message. + #[error("{0}")] + Generic(String), +} + +/// A type alias for [`Result`] using [`enum@Error`]. +/// +/// [`Result`]: std::result::Result +pub type Result = std::result::Result; diff --git a/unity-catalog-delta-client-api/src/lib.rs b/unity-catalog-delta-client-api/src/lib.rs new file mode 100644 index 0000000000..8bf41ce9f1 --- /dev/null +++ b/unity-catalog-delta-client-api/src/lib.rs @@ -0,0 +1,30 @@ +//! Unity Catalog client API traits and wire models for the UC API +//! surface. +//! +//! This crate defines the transport-agnostic [`UpdateTableClient`] trait that +//! `delta-kernel-unity-catalog`'s `UCCommitter` dispatches through, plus +//! serde-friendly wire models for the connector-driven endpoints +//! (`load_table`, credentials, `/config`). Concrete HTTP implementations live +//! in `unity-catalog-delta-rest-client`. +//! +//! Only `update_table` (the commit RPC) is behind a trait. Read and +//! credential-vending flows are connector-driven: connectors call concrete +//! REST methods (or bring their own HTTP plumbing) and hand the responses to +//! `delta-kernel-unity-catalog` helpers. + +pub mod clients; +pub mod credentials; +pub mod error; +pub mod models; + +pub use clients::UpdateTableClient; +#[cfg(any(test, feature = "test-utils"))] +pub use clients::{InMemoryUpdateTableClient, TableData}; +pub use credentials::{CredentialsResponse, Operation, StorageCredential}; +pub use error::{Error, Result}; +pub use models::{ + CatalogConfig, Commit, CommitReport, CreateStagingTableRequest, CreateStagingTableResponse, + CreateTableRequest, DeltaTableRequirement, DeltaTableUpdate, FileSizeHistogram, + LoadTableResponse, MetricsReport, Protocol, ReportMetricsRequest, TableIdentifier, + TableMetadata, UpdateTableRequest, +}; diff --git a/unity-catalog-delta-client-api/src/models.rs b/unity-catalog-delta-client-api/src/models.rs new file mode 100644 index 0000000000..2b6b16d72c --- /dev/null +++ b/unity-catalog-delta-client-api/src/models.rs @@ -0,0 +1,815 @@ +use std::collections::HashMap; + +use serde::{Deserialize, Serialize}; + +use crate::credentials::StorageCredential; + +// ============================================================================ +// update_table: typed requirements + updates +// ============================================================================ + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "kebab-case")] +pub struct Commit { + pub version: i64, + pub timestamp: i64, + pub file_name: String, + pub file_size: i64, + pub file_modification_timestamp: i64, +} + +impl Commit { + /// Create a new commit to send to UC with the specified version and timestamp. + pub fn new( + version: i64, + timestamp: i64, + file_name: impl Into, + file_size: i64, + file_modification_timestamp: i64, + ) -> Self { + Self { + version, + timestamp, + file_name: file_name.into(), + file_size, + file_modification_timestamp, + } + } +} + +/// A precondition that the catalog server must validate before applying a +/// commit. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(tag = "type", rename_all = "kebab-case")] +pub enum DeltaTableRequirement { + /// Assert that the table being updated has the given UUID. Used to detect + /// the case where a table has been dropped and recreated under the same + /// name between the client's last read and this update. + AssertTableUuid { uuid: String }, + /// Assert that the table's etag matches the expected value. Used for + /// optimistic-concurrency control on metadata-only updates. + AssertEtag { etag: String }, +} + +/// A typed update to apply atomically as part of an `update_table` call. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(tag = "action", rename_all = "kebab-case")] +pub enum DeltaTableUpdate { + /// Register a new staged commit with the catalog. + AddCommit { commit: Commit }, + /// Inform the catalog of the latest published version. + SetLatestBackfilledVersion { + #[serde(rename = "latest-published-version")] + latest_published_version: i64, + }, +} + +/// The three-part identifier for a UC table (catalog, schema, table), used to route requests. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct TableIdentifier { + pub catalog: String, + pub schema: String, + pub table: String, +} + +impl TableIdentifier { + pub fn new( + catalog: impl Into, + schema: impl Into, + table: impl Into, + ) -> Self { + Self { + catalog: catalog.into(), + schema: schema.into(), + table: table.into(), + } + } +} + +impl std::fmt::Display for TableIdentifier { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{}.{}.{}", self.catalog, self.schema, self.table) + } +} + +/// Body of a `POST /delta/v1/catalogs/{c}/schemas/{s}/tables/{t}` +/// (`update_table`) request. The target table is routed separately (see +/// [`crate::UpdateTableClient::update_table`]); this struct is purely the serialized body. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +pub struct UpdateTableRequest { + /// Preconditions the catalog must validate. + pub requirements: Vec, + /// Updates to apply atomically. + pub updates: Vec, +} + +impl UpdateTableRequest { + /// Build a request from its preconditions and updates. + /// + /// # Errors + /// + /// Returns an error if a singleton entry appears more than once. The UC `updateTable` endpoint + /// accepts at most one each of `AssertTableUuid`, `AssertEtag`, `AddCommit`, and + /// `SetLatestBackfilledVersion`. + pub fn new( + requirements: Vec, + updates: Vec, + ) -> crate::error::Result { + let num_requirements = |is_variant: fn(&DeltaTableRequirement) -> bool| { + requirements.iter().filter(|r| is_variant(r)).count() + }; + if num_requirements(|r| matches!(r, DeltaTableRequirement::AssertTableUuid { .. })) > 1 { + return Err(crate::error::Error::Generic( + "update_table request must not contain more than one AssertTableUuid requirement" + .to_string(), + )); + } + if num_requirements(|r| matches!(r, DeltaTableRequirement::AssertEtag { .. })) > 1 { + return Err(crate::error::Error::Generic( + "update_table request must not contain more than one AssertEtag requirement" + .to_string(), + )); + } + let num_updates = |is_variant: fn(&DeltaTableUpdate) -> bool| { + updates.iter().filter(|u| is_variant(u)).count() + }; + if num_updates(|u| matches!(u, DeltaTableUpdate::AddCommit { .. })) > 1 { + return Err(crate::error::Error::Generic( + "update_table request must not contain more than one AddCommit update".to_string(), + )); + } + if num_updates(|u| matches!(u, DeltaTableUpdate::SetLatestBackfilledVersion { .. })) > 1 { + return Err(crate::error::Error::Generic( + "update_table request must not contain more than one SetLatestBackfilledVersion \ + update" + .to_string(), + )); + } + Ok(Self { + requirements, + updates, + }) + } + + /// The table UUID asserted by this request's `AssertTableUuid` requirement, if present. + /// [`new`](Self::new) guarantees at most one such requirement. + pub fn table_uuid(&self) -> Option<&str> { + self.requirements.iter().find_map(|r| match r { + DeltaTableRequirement::AssertTableUuid { uuid } => Some(uuid.as_str()), + _ => None, + }) + } + + /// The `AddCommit` staged-commit reference in this request's updates, if present. + pub fn staged_commit(&self) -> Option<&Commit> { + self.updates.iter().find_map(|u| match u { + DeltaTableUpdate::AddCommit { commit } => Some(commit), + _ => None, + }) + } + + /// The latest backfilled (published) version in this request's updates, if present. + pub fn latest_backfilled_version(&self) -> Option { + self.updates.iter().find_map(|u| match u { + DeltaTableUpdate::SetLatestBackfilledVersion { + latest_published_version, + } => Some(*latest_published_version), + _ => None, + }) + } +} + +// ============================================================================ +// load_table: read-side response +// ============================================================================ + +/// Response from `GET /delta/v1/catalogs/{c}/schemas/{s}/tables/{t}` +/// (`load_table`). +/// +/// The server returns the table's full metadata plus any unpublished commits. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub struct LoadTableResponse { + /// Full table metadata. + pub metadata: TableMetadata, + /// Unpublished commits known to the catalog at this read, in descending + /// version order (newest first). + #[serde(default)] + pub commits: Vec, + /// Optional UniForm conversion metadata. Modeled as opaque JSON. + // TODO: decode into a typed struct once a consumer needs UniForm metadata. + #[serde(default)] + pub uniform: Option, + /// Highest version the catalog knows about, including data-only commits. + /// Compare with `metadata.last_commit_version`, which only tracks + /// metadata-changing commits. + #[serde(default)] + pub latest_table_version: Option, +} + +/// Full table metadata returned by `load_table`. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub struct TableMetadata { + /// Entity tag for optimistic-concurrency control. + pub etag: String, + /// Table type, e.g. `"MANAGED"` or `"EXTERNAL"`. + pub table_type: String, + /// Stable table UUID, matching `Metadata.id` in the Delta log. + pub table_uuid: String, + /// Cloud-storage location of the table's `_delta_log/` parent. + pub location: String, + /// Creation time in epoch milliseconds. + pub created_time: i64, + /// Last update time in epoch milliseconds. + pub updated_time: i64, + /// Schema as a Delta `StructType` JSON value. + pub columns: serde_json::Value, + /// Partition column names, in declaration order. + #[serde(default)] + pub partition_columns: Vec, + /// Raw `metaData.configuration` properties. + pub properties: HashMap, + /// Version of the last commit that changed table metadata. + #[serde(default)] + pub last_commit_version: Option, + /// Timestamp of the last metadata-changing commit, in epoch milliseconds. + #[serde(default)] + pub last_commit_timestamp_ms: Option, +} + +// ============================================================================ +// /config: protocol negotiation +// ============================================================================ + +/// Response from `GET /delta/v1/config`. +/// +/// The server returns the list of versioned endpoints the client should use +/// for subsequent calls, plus the negotiated protocol version. +#[derive(Debug, Clone, Serialize, Deserialize, Default)] +#[serde(rename_all = "kebab-case")] +pub struct CatalogConfig { + /// Supported endpoints, e.g. `"POST /delta/v1/catalogs/{catalog}/schemas/{schema}/tables"`. + pub endpoints: Vec, + /// Negotiated protocol version, e.g. `"1.0"`. + pub protocol_version: String, +} + +// ============================================================================ +// Protocol wire type +// ============================================================================ + +/// Typed Delta protocol wire model (mirrors kernel's `Protocol` action). +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)] +#[serde(rename_all = "kebab-case")] +pub struct Protocol { + /// Minimum reader version required to read the table. + pub min_reader_version: i32, + /// Minimum writer version required to write to the table. + pub min_writer_version: i32, + /// Reader features (Delta's `readerFeatures` list). + #[serde(default)] + pub reader_features: Vec, + /// Writer features (Delta's `writerFeatures` list). + #[serde(default)] + pub writer_features: Vec, +} + +// ============================================================================ +// create-table: staging-tables and tables (CREATE flow) +// ============================================================================ + +/// Body of a `POST /delta/v1/catalogs/{c}/schemas/{s}/staging-tables` request. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub struct CreateStagingTableRequest { + /// Table name (catalog and schema are in the URL path). + pub name: String, +} + +/// Response from the `staging-tables` endpoint. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub struct CreateStagingTableResponse { + /// Allocated table UUID. + pub table_id: String, + /// Table type (always `"MANAGED"` for staging tables). + pub table_type: String, + /// Cloud-storage location for the table's `_delta_log/` parent. + pub location: String, + /// Temporary credentials for the initial commit. + pub storage_credentials: Vec, + /// Required protocol that the v0 commit must satisfy. + pub required_protocol: Protocol, + /// Suggested protocol the client should consider. Modeled as opaque JSON. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub suggested_protocol: Option, + /// Required raw configuration entries. Null values mean "any valid value". + pub required_properties: HashMap>, + /// Suggested raw configuration entries. + #[serde(default)] + pub suggested_properties: HashMap>, +} + +/// Body of a `POST /delta/v1/catalogs/{c}/schemas/{s}/tables` request, the +/// typed CREATE finalization payload. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub struct CreateTableRequest { + /// Table name (catalog and schema are in the URL path). + pub name: String, + /// Cloud-storage location of the table root, e.g. `"s3://bucket/path/to/table"`. + pub location: String, + /// Table type, e.g. `"MANAGED"` or `"EXTERNAL"`. + pub table_type: String, + /// Optional table comment. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub comment: Option, + /// Typed Delta schema: a `StructType` JSON value. + pub columns: serde_json::Value, + /// Partition column names, in declaration order. + #[serde(default)] + pub partition_columns: Vec, + /// Typed Delta protocol. + pub protocol: Protocol, + /// Raw `metaData.configuration` entries. The server derives the *protocol* properties + /// (`delta.minReaderVersion`/`minWriterVersion`, `delta.feature.*`) from the typed `protocol`, + /// so those must not appear here. + pub properties: HashMap, + /// Per-domain metadata, keyed by domain name. Each value is opaque JSON. + #[serde(default, skip_serializing_if = "HashMap::is_empty")] + pub domain_metadata: HashMap, + /// Timestamp of version 0 (the commit the client wrote before calling this + /// endpoint), in epoch milliseconds. + pub last_commit_timestamp_ms: i64, +} + +// ============================================================================ +// reportMetrics: post-commit telemetry +// ============================================================================ + +/// Body of a `POST /delta/v1/catalogs/{c}/schemas/{s}/tables/{t}/metrics` request. +/// +/// Commit telemetry reported to the catalog after a commit succeeds. This is best-effort +/// telemetry, not part of the commit's correctness path. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)] +#[serde(rename_all = "kebab-case")] +pub struct ReportMetricsRequest { + /// Table UUID; must match the table identified by the URL path. + pub table_id: String, + /// The metrics being reported. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub report: Option, +} + +/// Wrapper matching the server's `report` envelope. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)] +#[serde(rename_all = "kebab-case")] +pub struct MetricsReport { + /// Per-commit statistics. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub commit_report: Option, +} + +/// Statistics describing a single commit. +/// +/// File and byte counts are derivable from the commit's add/remove actions. Row counts are only +/// known to the connector's write engine, so they are optional. The catalog requires a commit +/// version on every report and reads it from `file_size_histogram.commit_version`, so the histogram +/// is required. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)] +#[serde(rename_all = "kebab-case")] +pub struct CommitReport { + pub num_files_added: i64, + pub num_bytes_added: i64, + pub num_files_removed: i64, + pub num_bytes_removed: i64, + /// `None` when the write engine did not observe a row count (distinct from `0`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub num_rows_inserted: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub num_rows_removed: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub num_rows_updated: Option, + pub file_size_histogram: FileSizeHistogram, +} + +/// File-size histogram as index-aligned arrays mirroring the server's `file-size-histogram` +/// wire schema: `file_counts[i]` and `total_bytes[i]` describe the bin bounded by +/// `sorted_bin_boundaries[i]`. The three arrays have equal length. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)] +#[serde(rename_all = "kebab-case")] +pub struct FileSizeHistogram { + pub sorted_bin_boundaries: Vec, + pub file_counts: Vec, + pub total_bytes: Vec, + /// The commit version this histogram is for. + pub commit_version: i64, +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn table_identifier_displays_as_dotted_three_part_name() { + let id = TableIdentifier::new("main", "default", "my_table"); + assert_eq!(id.to_string(), "main.default.my_table"); + } + + #[test] + fn load_table_response_decodes_full_body() { + // Both load_table and update_table return this full table-state body. Decode a + // representative one and pin the kebab-case field renames on TableMetadata. + let body = r#"{ + "metadata": { + "etag": "e1", + "table-type": "MANAGED", + "table-uuid": "abc", + "location": "s3://bucket/t", + "created-time": 1, + "updated-time": 2, + "columns": {"type": "struct", "fields": []}, + "partition-columns": ["p"], + "properties": {}, + "last-commit-version": 5, + "last-commit-timestamp-ms": 6 + }, + "commits": [], + "latest-table-version": 7 + }"#; + let resp: LoadTableResponse = serde_json::from_str(body).unwrap(); + assert_eq!(resp.metadata.table_type, "MANAGED"); + assert_eq!(resp.metadata.table_uuid, "abc"); + assert_eq!(resp.metadata.last_commit_timestamp_ms, Some(6)); + assert_eq!(resp.latest_table_version, Some(7)); + } + + // === Serde golden tests: pin the wire tag + renamed keys for the tagged enums. === + + fn sample_commit() -> Commit { + Commit::new(7, 1234, "00000000000000000007.uuid.json", 100, 5678) + } + + #[test] + fn update_add_commit_wire_shape() { + let v = serde_json::to_value(DeltaTableUpdate::AddCommit { + commit: sample_commit(), + }) + .unwrap(); + assert_eq!(v["action"], "add-commit"); + assert!(v.get("commit").is_some()); + } + + #[test] + fn update_set_latest_backfilled_version_wire_shape() { + let v = serde_json::to_value(DeltaTableUpdate::SetLatestBackfilledVersion { + latest_published_version: 9, + }) + .unwrap(); + assert_eq!(v["action"], "set-latest-backfilled-version"); + assert_eq!(v["latest-published-version"], 9); + } + + #[test] + fn requirement_assert_table_uuid_wire_shape() { + let v = serde_json::to_value(DeltaTableRequirement::AssertTableUuid { + uuid: "abc".to_string(), + }) + .unwrap(); + assert_eq!(v["type"], "assert-table-uuid"); + assert_eq!(v["uuid"], "abc"); + } + + #[test] + fn requirement_assert_etag_wire_shape() { + let v = serde_json::to_value(DeltaTableRequirement::AssertEtag { + etag: "e1".to_string(), + }) + .unwrap(); + assert_eq!(v["type"], "assert-etag"); + assert_eq!(v["etag"], "e1"); + } + + #[test] + fn commit_wire_shape() { + let v = serde_json::to_value(sample_commit()).unwrap(); + assert_eq!(v["version"], 7); + assert_eq!(v["timestamp"], 1234); + assert_eq!(v["file-name"], "00000000000000000007.uuid.json"); + assert_eq!(v["file-size"], 100); + assert_eq!(v["file-modification-timestamp"], 5678); + } + + #[test] + fn update_table_request_serializes_only_body_fields() { + let req = UpdateTableRequest::new(vec![], vec![]).unwrap(); + let v = serde_json::to_value(&req).unwrap(); + let obj = v.as_object().unwrap(); + assert_eq!( + obj.len(), + 2, + "body should have exactly requirements + updates" + ); + assert!(obj.contains_key("requirements")); + assert!(obj.contains_key("updates")); + } + + #[test] + fn table_uuid_returns_none_when_only_etag_requirement_present() { + let req = UpdateTableRequest::new( + vec![DeltaTableRequirement::AssertEtag { etag: "e1".into() }], + vec![], + ) + .unwrap(); + assert_eq!(req.table_uuid(), None); + } + + #[test] + fn table_uuid_returns_uuid_when_assert_table_uuid_present() { + let req = UpdateTableRequest::new( + vec![DeltaTableRequirement::AssertTableUuid { uuid: "abc".into() }], + vec![], + ) + .unwrap(); + assert_eq!(req.table_uuid(), Some("abc")); + } + + #[test] + fn new_rejects_duplicate_requirements() { + for dup in [ + vec![ + DeltaTableRequirement::AssertTableUuid { uuid: "a".into() }, + DeltaTableRequirement::AssertTableUuid { uuid: "b".into() }, + ], + vec![ + DeltaTableRequirement::AssertEtag { etag: "e1".into() }, + DeltaTableRequirement::AssertEtag { etag: "e2".into() }, + ], + ] { + assert!( + UpdateTableRequest::new(dup, vec![]).is_err(), + "duplicate requirement of the same type must be rejected" + ); + } + + // One of each type is allowed. + assert!(UpdateTableRequest::new( + vec![ + DeltaTableRequirement::AssertTableUuid { uuid: "a".into() }, + DeltaTableRequirement::AssertEtag { etag: "e1".into() }, + ], + vec![], + ) + .is_ok()); + } + + #[test] + fn new_rejects_duplicate_set_latest_backfilled_version() { + assert!( + UpdateTableRequest::new( + vec![], + vec![ + DeltaTableUpdate::SetLatestBackfilledVersion { + latest_published_version: 1, + }, + DeltaTableUpdate::SetLatestBackfilledVersion { + latest_published_version: 2, + }, + ], + ) + .is_err(), + "more than one SetLatestBackfilledVersion must be rejected" + ); + + assert!( + UpdateTableRequest::new( + vec![], + vec![ + DeltaTableUpdate::AddCommit { + commit: sample_commit(), + }, + DeltaTableUpdate::AddCommit { + commit: sample_commit(), + }, + ], + ) + .is_err(), + "more than one AddCommit must be rejected" + ); + + // A single AddCommit alongside the other singletons is accepted. + assert!(UpdateTableRequest::new( + vec![], + vec![ + DeltaTableUpdate::AddCommit { + commit: sample_commit(), + }, + DeltaTableUpdate::SetLatestBackfilledVersion { + latest_published_version: 3, + }, + ], + ) + .is_ok()); + } + + #[test] + fn staged_commit_returns_commit_when_add_commit_present() { + let req = UpdateTableRequest::new( + vec![], + vec![DeltaTableUpdate::AddCommit { + commit: sample_commit(), + }], + ) + .unwrap(); + assert_eq!(req.staged_commit(), Some(&sample_commit())); + } + + #[test] + fn staged_commit_returns_none_when_no_add_commit_present() { + let req = UpdateTableRequest::new( + vec![], + vec![DeltaTableUpdate::SetLatestBackfilledVersion { + latest_published_version: 9, + }], + ) + .unwrap(); + assert_eq!(req.staged_commit(), None); + } + + #[test] + fn catalog_config_decodes_kebab_protocol_version() { + let config: CatalogConfig = serde_json::from_str( + r#"{"endpoints": ["POST /delta/v1/catalogs/{catalog}/schemas/{schema}/tables"], "protocol-version": "1.0"}"#, + ) + .unwrap(); + assert_eq!(config.protocol_version, "1.0"); + assert_eq!(config.endpoints.len(), 1); + } + + #[test] + fn create_staging_table_request_wire_shape() { + let v = serde_json::to_value(CreateStagingTableRequest { + name: "t".to_string(), + }) + .unwrap(); + assert_eq!(v["name"], "t"); + } + + #[test] + fn create_staging_table_response_decodes_renamed_fields() { + // Pin the `table-id` rename and that storage-credentials / optional advertisement + // fields decode. + let body = r#"{ + "table-id": "abc", + "table-type": "MANAGED", + "location": "s3://bucket/t", + "storage-credentials": [ + {"prefix": "s3://bucket/t/", "operation": "READ_WRITE", + "expiration-time-ms": 123, "config": {"s3.access-key-id": "ak"}} + ], + "required-protocol": {"min-reader-version": 3, "min-writer-version": 7}, + "required-properties": {} + }"#; + let resp: CreateStagingTableResponse = serde_json::from_str(body).unwrap(); + assert_eq!(resp.table_id, "abc"); + assert_eq!(resp.location, "s3://bucket/t"); + assert_eq!(resp.storage_credentials.len(), 1); + assert_eq!(resp.required_protocol.min_reader_version, 3); + assert!(resp.suggested_protocol.is_none()); + assert!(resp.required_properties.is_empty()); + } + + #[test] + fn create_staging_table_response_decodes_property_maps_and_suggested_protocol() { + let body = r#"{ + "table-id": "abc", + "table-type": "MANAGED", + "location": "s3://bucket/t", + "storage-credentials": [], + "required-protocol": {"min-reader-version": 3, "min-writer-version": 7}, + "suggested-protocol": {"min-reader-version": 1, "min-writer-version": 2}, + "required-properties": {"delta.appendOnly": "true", "delta.columnMapping.mode": null}, + "suggested-properties": {"delta.checkpointInterval": null} + }"#; + let resp: CreateStagingTableResponse = serde_json::from_str(body).unwrap(); + assert_eq!( + resp.required_properties["delta.appendOnly"], + Some("true".to_string()) + ); + assert_eq!(resp.required_properties["delta.columnMapping.mode"], None); + assert_eq!(resp.suggested_properties["delta.checkpointInterval"], None); + assert!(resp.suggested_protocol.is_some()); + } + + #[test] + fn create_table_request_wire_shape() { + let req = CreateTableRequest { + name: "t".to_string(), + location: "s3://bucket/t".to_string(), + table_type: "MANAGED".to_string(), + comment: None, + columns: serde_json::json!({"type": "struct", "fields": []}), + partition_columns: vec![], + protocol: Protocol { + min_reader_version: 3, + min_writer_version: 7, + reader_features: vec!["catalogManaged".to_string()], + writer_features: vec!["catalogManaged".to_string()], + }, + properties: HashMap::from([("delta.appendOnly".to_string(), "true".to_string())]), + domain_metadata: HashMap::new(), + last_commit_timestamp_ms: 42, + }; + let v = serde_json::to_value(&req).unwrap(); + // Field renames. + assert_eq!(v["location"], "s3://bucket/t"); + assert_eq!(v["table-type"], "MANAGED"); + assert_eq!(v["last-commit-timestamp-ms"], 42); + // Protocol is sent typed and separate; properties carry only raw configuration (the server + // derives `delta.feature.*` / versions itself), so no derived keys leak into `properties`. + assert_eq!(v["protocol"]["min-reader-version"], 3); + assert_eq!(v["properties"]["delta.appendOnly"], "true"); + assert!(v["properties"].get("delta.minReaderVersion").is_none()); + assert!(v["properties"] + .get("delta.feature.catalogManaged") + .is_none()); + // Empty domain metadata is omitted from the wire body. + assert!(v.get("domain-metadata").is_none()); + } + + #[test] + fn create_table_request_serializes_optional_fields_when_present() { + let req = CreateTableRequest { + name: "t".to_string(), + location: "s3://bucket/t".to_string(), + table_type: "MANAGED".to_string(), + comment: Some("hello".to_string()), + columns: serde_json::json!({"type": "struct", "fields": []}), + partition_columns: vec!["p1".to_string(), "p2".to_string()], + protocol: Protocol::default(), + properties: HashMap::new(), + domain_metadata: HashMap::from([( + "delta.clustering".to_string(), + serde_json::json!({"clusteringColumns": [["c1"]]}), + )]), + last_commit_timestamp_ms: 42, + }; + let v = serde_json::to_value(&req).unwrap(); + assert_eq!(v["comment"], "hello"); + assert_eq!(v["partition-columns"], serde_json::json!(["p1", "p2"])); + assert_eq!( + v["domain-metadata"]["delta.clustering"]["clusteringColumns"][0], + serde_json::json!(["c1"]) + ); + } + + #[test] + fn report_metrics_request_wire_shape() { + let req = ReportMetricsRequest { + table_id: "abc".to_string(), + report: Some(MetricsReport { + commit_report: Some(CommitReport { + num_files_added: 3, + num_bytes_added: 300, + num_files_removed: 1, + num_bytes_removed: 100, + num_rows_inserted: Some(42), + num_rows_removed: None, + num_rows_updated: None, + file_size_histogram: FileSizeHistogram { + sorted_bin_boundaries: vec![0, 1000], + file_counts: vec![2, 1], + total_bytes: vec![150, 150], + commit_version: 7, + }, + }), + }), + }; + let v = serde_json::to_value(&req).unwrap(); + assert_eq!(v["table-id"], "abc"); + let cr = &v["report"]["commit-report"]; + assert_eq!(cr["num-files-added"], 3); + assert_eq!(cr["num-bytes-removed"], 100); + assert_eq!(cr["num-rows-inserted"], 42); + // Absent optional row counts are omitted from the wire body. + assert!(cr.get("num-rows-removed").is_none()); + assert!(cr.get("num-rows-updated").is_none()); + assert_eq!(cr["file-size-histogram"]["commit-version"], 7); + assert_eq!( + cr["file-size-histogram"]["sorted-bin-boundaries"], + serde_json::json!([0, 1000]) + ); + } + + #[test] + fn report_metrics_request_omits_absent_report() { + let req = ReportMetricsRequest { + table_id: "abc".to_string(), + report: None, + }; + let v = serde_json::to_value(&req).unwrap(); + assert_eq!(v["table-id"], "abc"); + assert!(v.get("report").is_none()); + } +} diff --git a/unity-catalog-delta-rest-client/Cargo.toml b/unity-catalog-delta-rest-client/Cargo.toml new file mode 100644 index 0000000000..5304603e63 --- /dev/null +++ b/unity-catalog-delta-rest-client/Cargo.toml @@ -0,0 +1,35 @@ +[package] +name = "unity-catalog-delta-rest-client" +description = "REST/HTTP client for the Unity Catalog Delta Tables API." +edition.workspace = true +homepage.workspace = true +keywords.workspace = true +license.workspace = true +repository.workspace = true +readme = "README.md" +rust-version.workspace = true +version = "0.1.0" + +# for cargo-release +[package.metadata.release] +release = false + +[dependencies] +unity-catalog-delta-client-api = { path = "../unity-catalog-delta-client-api", version = "0.1.0" } +# Hardcodes rustls (always used alongside default-engine-rustls in practice). +reqwest = { version = "0.13", default-features = false, features = ["charset", "http2", "json", "rustls", "system-proxy"] } +percent-encoding = "2" +serde = { version = "1.0", features = ["derive"] } +serde_json = "1.0" +tokio = { version = "1", features = ["full"] } +thiserror = "2.0" +tracing = "0.1" +url = "2.5" + +[features] +default = [] +test-utils = ["unity-catalog-delta-client-api/test-utils"] +# Gates live integration tests that hit a real Unity Catalog server. Off by default so normal +# `cargo test` never compiles or runs them; the dedicated CI job enables it. See +# tests/integration_live_server.rs. +integration-test = [] diff --git a/unity-catalog-delta-rest-client/README.md b/unity-catalog-delta-rest-client/README.md new file mode 100644 index 0000000000..39aa0aecc4 --- /dev/null +++ b/unity-catalog-delta-rest-client/README.md @@ -0,0 +1,46 @@ +# unity-catalog-delta-rest-client + +> [!WARNING] +> This crate is experimental and under construction. It is not intended for production use. + +A Rust client for the Unity Catalog Delta APIs. + +It provides two REST structs: + +- `UCClient`: concrete HTTP methods for the connector-driven endpoints: `load_table`, credential + vending, `/config`, table and staging-table creation, and metrics reporting. +- `UCUpdateTableRestClient`: an implementation of the `UpdateTableClient` trait (from + `unity-catalog-delta-client-api`) against the `update_table` commit endpoint. + +## Example + +```rust,no_run +use unity_catalog_delta_client_api::Operation; +use unity_catalog_delta_rest_client::{ClientConfig, UCClient}; + +#[tokio::main] +async fn main() -> Result<(), Box> { + // Configure against a UC workspace (URL + token + identity). + let config = ClientConfig::build("https://some.uc.org", "your-token") + .with_additional_user_agent([("MyEngine", "1.0.0"), ("MyConnector", "1.0.0")]) + .build()?; + let client = UCClient::new(config)?; + + // Load a table by its three-part name. The response carries the table + // metadata and any inline (unpublished) commits. + let table = client.load_table("catalog", "schema", "table").await?; + println!("table id: {}", table.metadata.table_uuid); + println!("location: {}", table.metadata.location); + + // Vend temporary storage credentials for reads. + let creds = client + .get_table_credentials("catalog", "schema", "table", Operation::Read) + .await?; + println!("vended {} credential(s)", creds.storage_credentials.len()); + + Ok(()) +} +``` + +To coordinate commits (version >= 1), construct a `UCUpdateTableRestClient` and use it as the +`UpdateTableClient` trait object; see the crate-level documentation for details. diff --git a/unity-catalog-delta-rest-client/src/clients/commits.rs b/unity-catalog-delta-rest-client/src/clients/commits.rs new file mode 100644 index 0000000000..59f31ee227 --- /dev/null +++ b/unity-catalog-delta-rest-client/src/clients/commits.rs @@ -0,0 +1,55 @@ +use tracing::instrument; +use unity_catalog_delta_client_api::{TableIdentifier, UpdateTableClient, UpdateTableRequest}; +use url::Url; + +use crate::config::ClientConfig; +use crate::http::{build_http_client, handle_empty_response}; + +/// REST implementation of [`UpdateTableClient`]. +#[derive(Debug)] +pub struct UCUpdateTableRestClient { + http_client: reqwest::Client, + base_url: Url, +} + +impl UCUpdateTableRestClient { + /// Create from config. + pub fn new(config: ClientConfig) -> crate::error::Result { + Ok(Self { + http_client: build_http_client(&config)?, + base_url: config.workspace_url.clone(), + }) + } + + /// Create from an existing reqwest client and config. + pub fn with_http_client(http_client: reqwest::Client, config: ClientConfig) -> Self { + Self { + base_url: config.workspace_url.clone(), + http_client, + } + } +} + +impl UpdateTableClient for UCUpdateTableRestClient { + #[instrument(skip(self))] + async fn update_table( + &self, + target: &TableIdentifier, + request: UpdateTableRequest, + ) -> unity_catalog_delta_client_api::Result<()> { + let result: crate::error::Result<()> = async { + let path = format!( + "delta/v1/catalogs/{}/schemas/{}/tables/{}", + target.catalog, target.schema, target.table + ); + let url = self.base_url.join(&path)?; + + // Single attempt. UC rejects a resubmit of an already-ratified version with a version + // conflict. Retries belong in the transaction layer. + let response = self.http_client.post(url).json(&request).send().await?; + handle_empty_response(response).await + } + .await; + result.map_err(Into::into) + } +} diff --git a/unity-catalog-delta-rest-client/src/clients/mod.rs b/unity-catalog-delta-rest-client/src/clients/mod.rs new file mode 100644 index 0000000000..863784dd26 --- /dev/null +++ b/unity-catalog-delta-rest-client/src/clients/mod.rs @@ -0,0 +1,5 @@ +mod commits; +mod uc_client; + +pub use commits::UCUpdateTableRestClient; +pub use uc_client::UCClient; diff --git a/unity-catalog-delta-rest-client/src/clients/uc_client.rs b/unity-catalog-delta-rest-client/src/clients/uc_client.rs new file mode 100644 index 0000000000..f66115a0dd --- /dev/null +++ b/unity-catalog-delta-rest-client/src/clients/uc_client.rs @@ -0,0 +1,196 @@ +use percent_encoding::{utf8_percent_encode, NON_ALPHANUMERIC}; +use reqwest::StatusCode; +use tracing::instrument; +use unity_catalog_delta_client_api::{ + CatalogConfig, CommitReport, CreateStagingTableRequest, CreateStagingTableResponse, + CreateTableRequest, CredentialsResponse, LoadTableResponse, MetricsReport, Operation, + ReportMetricsRequest, +}; +use url::Url; + +use crate::config::ClientConfig; +use crate::error::Result; +use crate::http::{ + build_http_client, execute_with_retry, execute_without_retry, handle_empty_response, + handle_response, +}; + +/// Percent-encodes a name for use as a single URL path segment. +fn encode_segment(name: &str) -> impl std::fmt::Display + '_ { + utf8_percent_encode(name, NON_ALPHANUMERIC) +} + +/// Builds the Delta-Tables per-table resource path +/// (`delta/v1/catalogs/{catalog}/schemas/{schema}/tables/{table}`) that the `load_table` and +/// credential-vending endpoints share. +fn table_path(catalog: &str, schema: &str, table: &str) -> String { + format!( + "delta/v1/catalogs/{}/schemas/{}/tables/{}", + encode_segment(catalog), + encode_segment(schema), + encode_segment(table) + ) +} + +/// An HTTP client for interacting with the Unity Catalog API. +#[derive(Debug, Clone)] +pub struct UCClient { + http_client: reqwest::Client, + config: ClientConfig, + base_url: Url, +} + +impl UCClient { + /// Create a new client from [ClientConfig]. + pub fn new(config: ClientConfig) -> Result { + Ok(Self { + http_client: build_http_client(&config)?, + base_url: config.workspace_url.clone(), + config, + }) + } + + /// Create from existing reqwest Client. + pub fn with_http_client(http_client: reqwest::Client, config: ClientConfig) -> Self { + Self { + base_url: config.workspace_url.clone(), + http_client, + config, + } + } + + /// `GET /delta/v1/catalogs/{catalog}/schemas/{schema}/tables/{table}`: + /// fetch the table's metadata plus inline unpublished commits. + #[instrument(skip(self))] + pub async fn load_table( + &self, + catalog: &str, + schema: &str, + table: &str, + ) -> Result { + let url = self.base_url.join(&table_path(catalog, schema, table))?; + + let response = + execute_with_retry(&self.config, || self.http_client.get(url.clone()).send()).await?; + match response.status() { + StatusCode::NOT_FOUND => Err(unity_catalog_delta_client_api::Error::TableNotFound( + format!("{catalog}.{schema}.{table}"), + ) + .into()), + _ => handle_response(response).await, + } + } + + /// Vend temporary cloud-storage credentials for the table via the Delta-Tables + /// `GET .../catalogs/{catalog}/schemas/{schema}/tables/{table}/credentials?operation=...` + /// endpoint. + #[instrument(skip(self))] + pub async fn get_table_credentials( + &self, + catalog: &str, + schema: &str, + table: &str, + operation: Operation, + ) -> Result { + let path = format!("{}/credentials", table_path(catalog, schema, table)); + let mut url = self.base_url.join(&path)?; + url.query_pairs_mut() + .append_pair("operation", &operation.to_string()); + + let response = + execute_with_retry(&self.config, || self.http_client.get(url.clone()).send()).await?; + handle_response(response).await + } + + /// `GET /delta/v1/config?catalog={catalog}&protocol-versions={csv}`: session-start handshake. + /// `protocol_versions` is a list of version strings such as `["1.1", "2.3"]` indicating the + /// highest version per major version the client supports. + #[instrument(skip(self))] + pub async fn get_config( + &self, + catalog: &str, + protocol_versions: &[&str], + ) -> Result { + let mut url = self.base_url.join("delta/v1/config")?; + url.query_pairs_mut().append_pair("catalog", catalog); + url.query_pairs_mut() + .append_pair("protocol-versions", &protocol_versions.join(",")); + + let response = + execute_with_retry(&self.config, || self.http_client.get(url.clone()).send()).await?; + handle_response(response).await + } + + /// `POST /delta/v1/catalogs/{catalog}/schemas/{schema}/tables/{table}/metrics`: report + /// best-effort commit telemetry to the catalog after a commit succeeds. + /// + /// Supply the row counts and histogram that only the write engine knows. + #[instrument(skip(self, report))] + pub async fn report_metrics( + &self, + catalog: &str, + schema: &str, + table: &str, + table_id: &str, + report: CommitReport, + ) -> Result<()> { + let path = format!("{}/metrics", table_path(catalog, schema, table)); + let url = self.base_url.join(&path)?; + let body = ReportMetricsRequest { + table_id: table_id.to_string(), + report: Some(MetricsReport { + commit_report: Some(report), + }), + }; + + let response = execute_with_retry(&self.config, || { + self.http_client.post(url.clone()).json(&body).send() + }) + .await?; + handle_empty_response(response).await + } + + /// `POST /delta/v1/catalogs/{catalog}/schemas/{schema}/staging-tables`: reserve a staging + /// table, allocating its UUID and storage location and returning temporary credentials for + /// the version 0 commit. + #[instrument(skip(self, request))] + pub async fn create_staging_table( + &self, + catalog: &str, + schema: &str, + request: CreateStagingTableRequest, + ) -> Result { + let path = format!( + "delta/v1/catalogs/{}/schemas/{}/staging-tables", + encode_segment(catalog), + encode_segment(schema) + ); + let url = self.base_url.join(&path)?; + let response = + execute_without_retry(|| self.http_client.post(url.clone()).json(&request).send()) + .await?; + handle_response(response).await + } + + /// `POST /delta/v1/catalogs/{catalog}/schemas/{schema}/tables`: register a table with the + /// catalog after its version 0 commit, promoting the staging table. Returns the registered + /// table as a `LoadTableResponse`. + #[instrument(skip(self, request))] + pub async fn create_table( + &self, + catalog: &str, + schema: &str, + request: CreateTableRequest, + ) -> Result { + let path = format!( + "delta/v1/catalogs/{}/schemas/{}/tables", + encode_segment(catalog), + encode_segment(schema) + ); + let url = self.base_url.join(&path)?; + let response = + execute_without_retry(|| self.http_client.post(url.clone()).json(&request).send()) + .await?; + handle_response(response).await + } +} diff --git a/uc-client/src/config.rs b/unity-catalog-delta-rest-client/src/config.rs similarity index 52% rename from uc-client/src/config.rs rename to unity-catalog-delta-rest-client/src/config.rs index 0759451486..6bf2f38aee 100644 --- a/uc-client/src/config.rs +++ b/unity-catalog-delta-rest-client/src/config.rs @@ -4,10 +4,30 @@ use url::Url; use crate::error::Result; -#[derive(Debug, Clone)] +/// The client's own `User-Agent` token, always sent first. Connectors append their engine, +/// connector, and kernel versions via [`ClientConfigBuilder::with_additional_user_agent`]. +fn client_user_agent() -> String { + format!( + "Unity-Catalog-Delta-Rest-Rust-Client/{}", + env!("CARGO_PKG_VERSION") + ) +} + +/// Compose the full `User-Agent`: the client's own token followed by each caller-supplied +/// `name/version` pair, space-separated. +fn compose_user_agent(additional_versions: &[(String, String)]) -> String { + let mut ua = client_user_agent(); + for (name, version) in additional_versions { + ua.push_str(&format!(" {name}/{version}")); + } + ua +} + +#[derive(Clone)] pub struct ClientConfig { pub workspace_url: Url, pub token: String, + user_agent: String, pub timeout: Duration, pub connect_timeout: Duration, pub max_retries: u32, @@ -15,8 +35,36 @@ pub struct ClientConfig { pub retry_max_delay: Duration, } +// Manual Debug to avoid leaking the bearer token. +impl std::fmt::Debug for ClientConfig { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("ClientConfig") + .field("workspace_url", &self.workspace_url) + .field("token", &"***") + .field("user_agent", &self.user_agent) + .field("timeout", &self.timeout) + .field("connect_timeout", &self.connect_timeout) + .field("max_retries", &self.max_retries) + .field("retry_base_delay", &self.retry_base_delay) + .field("retry_max_delay", &self.retry_max_delay) + .finish() + } +} + impl ClientConfig { - fn new(workspace: impl Into, token: impl Into) -> Result { + pub fn user_agent(&self) -> &str { + &self.user_agent + } + + pub fn build(workspace: impl Into, token: impl Into) -> ClientConfigBuilder { + ClientConfigBuilder::new(workspace, token) + } + + fn new( + workspace: impl Into, + token: impl Into, + user_agent: String, + ) -> Result { let workspace_str = workspace.into(); // add http(s) prefix if not present let base_url = @@ -36,6 +84,7 @@ impl ClientConfig { Ok(Self { workspace_url, token: token.into(), + user_agent, timeout: Duration::from_secs(30), connect_timeout: Duration::from_secs(10), max_retries: 3, @@ -43,15 +92,12 @@ impl ClientConfig { retry_max_delay: Duration::from_secs(10), }) } - - pub fn build(workspace: impl Into, token: impl Into) -> ClientConfigBuilder { - ClientConfigBuilder::new(workspace, token) - } } pub struct ClientConfigBuilder { workspace: String, token: String, + additional_versions: Vec<(String, String)>, timeout: Duration, connect_timeout: Duration, max_retries: u32, @@ -64,6 +110,7 @@ impl ClientConfigBuilder { Self { workspace: workspace.into(), token: token.into(), + additional_versions: Vec::new(), timeout: Duration::from_secs(30), connect_timeout: Duration::from_secs(10), max_retries: 3, @@ -72,6 +119,21 @@ impl ClientConfigBuilder { } } + /// Append `name/version` pairs to the `User-Agent`, after the client's own token. Some catalogs + /// allowlist agents, so connectors should identify their compute engine, connector, and (when + /// using kernel) kernel version. + pub fn with_additional_user_agent( + mut self, + versions: impl IntoIterator, impl Into)>, + ) -> Self { + self.additional_versions.extend( + versions + .into_iter() + .map(|(name, version)| (name.into(), version.into())), + ); + self + } + pub fn with_timeout(mut self, timeout: Duration) -> Self { self.timeout = timeout; self @@ -94,7 +156,8 @@ impl ClientConfigBuilder { } pub fn build(self) -> Result { - let mut config = ClientConfig::new(self.workspace, self.token)?; + let user_agent = compose_user_agent(&self.additional_versions); + let mut config = ClientConfig::new(self.workspace, self.token, user_agent)?; config.timeout = self.timeout; config.connect_timeout = self.connect_timeout; config.max_retries = self.max_retries; @@ -108,6 +171,13 @@ impl ClientConfigBuilder { mod tests { use super::*; + fn client_token() -> String { + format!( + "Unity-Catalog-Delta-Rest-Rust-Client/{}", + env!("CARGO_PKG_VERSION") + ) + } + #[test] fn test_client_config_builder() { let config = ClientConfig::build("example.com", "token123") @@ -132,11 +202,51 @@ mod tests { #[test] fn test_client_config() { - let config = ClientConfig::new("some-workspace.something.com", "token").unwrap(); + let config = + ClientConfig::new("some-workspace.something.com", "token", client_user_agent()) + .unwrap(); assert!(config .workspace_url .as_str() .contains("api/2.1/unity-catalog")); assert_eq!(config.token, "token"); } + + #[test] + fn user_agent_defaults_to_client_token() { + let config = ClientConfig::build("example.com", "t").build().unwrap(); + assert_eq!(config.user_agent(), client_token()); + } + + #[test] + fn additional_user_agent_versions_append_in_order() { + let config = ClientConfig::build("example.com", "t") + .with_additional_user_agent([ + ("MyEngine", "1.0.0"), + ("MyConnector", "1.0.0"), + ("Delta-Kernel-Rust", "0.26.0"), + ]) + .build() + .unwrap(); + assert_eq!( + config.user_agent(), + format!( + "{} MyEngine/1.0.0 MyConnector/1.0.0 Delta-Kernel-Rust/0.26.0", + client_token() + ) + ); + } + + #[test] + fn debug_redacts_bearer_token() { + let config = ClientConfig::build("example.com", "super-secret-token") + .build() + .unwrap(); + let debug = format!("{config:?}"); + assert!( + !debug.contains("super-secret-token"), + "token leaked: {debug}" + ); + assert!(debug.contains("***"), "redaction marker missing: {debug}"); + } } diff --git a/unity-catalog-delta-rest-client/src/error.rs b/unity-catalog-delta-rest-client/src/error.rs new file mode 100644 index 0000000000..3416673489 --- /dev/null +++ b/unity-catalog-delta-rest-client/src/error.rs @@ -0,0 +1,99 @@ +use thiserror::Error; + +/// REST-specific errors for the Unity Catalog client. +/// +/// API-level errors (table not found, auth failures, etc.) are wrapped in the +/// [`Api`](Error::Api) variant, keeping them in sync with `unity_catalog_delta_client_api::Error` +/// without duplicating variants. +#[derive(Error, Debug)] +pub enum Error { + /// A transport-agnostic API error. + #[error(transparent)] + Api(#[from] unity_catalog_delta_client_api::Error), + + /// An HTTP request failed. + #[error("HTTP request failed: {0}")] + Http(#[from] reqwest::Error), + + /// An invalid HTTP header value was provided. + #[error("Invalid header value: {0}")] + InvalidHeaderValue(#[from] reqwest::header::InvalidHeaderValue), + + /// A URL could not be parsed. + #[error("URL parse error: {0}")] + UrlParse(#[from] url::ParseError), + + /// JSON serialization or deserialization failed. + #[error("Serialization error: {0}")] + Serialization(#[from] serde_json::Error), + + /// The server returned a non-success HTTP status code. + #[error("HTTP error (status {status}): {message}")] + HttpStatusError { status: u16, message: String }, + + /// The client configuration is invalid. + #[error("Invalid configuration: {0}")] + InvalidConfiguration(String), + + /// All retry attempts have been exhausted. + #[error("Max retries exceeded")] + MaxRetriesExceeded, +} + +/// A type alias for [`Result`] using [`enum@Error`]. +/// +/// [`Result`]: std::result::Result +pub type Result = std::result::Result; + +impl From for unity_catalog_delta_client_api::Error { + fn from(e: Error) -> Self { + match e { + Error::Api(api_err) => api_err, + e => unity_catalog_delta_client_api::Error::Generic(e.to_string()), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn from_error_unwraps_api_variant() { + let api_err = unity_catalog_delta_client_api::Error::TableNotFound("t1".into()); + let rest_err = Error::Api(api_err); + assert!(matches!( + unity_catalog_delta_client_api::Error::from(rest_err), + unity_catalog_delta_client_api::Error::TableNotFound(msg) if msg == "t1" + )); + + let rest_err = Error::Api(unity_catalog_delta_client_api::Error::AuthenticationFailed); + assert!(matches!( + unity_catalog_delta_client_api::Error::from(rest_err), + unity_catalog_delta_client_api::Error::AuthenticationFailed + )); + + let rest_err = + Error::Api(unity_catalog_delta_client_api::Error::MaxUnpublishedCommitsExceeded(5)); + assert!(matches!( + unity_catalog_delta_client_api::Error::from(rest_err), + unity_catalog_delta_client_api::Error::MaxUnpublishedCommitsExceeded(5) + )); + } + + #[test] + fn from_error_maps_rest_only_variants_to_generic() { + let api_err = unity_catalog_delta_client_api::Error::from(Error::MaxRetriesExceeded); + assert!( + matches!(api_err, unity_catalog_delta_client_api::Error::Generic(ref msg) if msg == "Max retries exceeded"), + "unexpected: {api_err:?}" + ); + + let api_err = + unity_catalog_delta_client_api::Error::from(Error::InvalidConfiguration("bad".into())); + assert!( + matches!(api_err, unity_catalog_delta_client_api::Error::Generic(ref msg) if msg == "Invalid configuration: bad"), + "unexpected: {api_err:?}" + ); + } +} diff --git a/uc-client/src/http.rs b/unity-catalog-delta-rest-client/src/http.rs similarity index 51% rename from uc-client/src/http.rs rename to unity-catalog-delta-rest-client/src/http.rs index e815393ccc..1a536a4094 100644 --- a/uc-client/src/http.rs +++ b/unity-catalog-delta-rest-client/src/http.rs @@ -17,6 +17,11 @@ pub fn build_http_client(config: &ClientConfig) -> Result { header::CONTENT_TYPE, header::HeaderValue::from_static("application/json"), ), + // Identifies the calling client to UC. + ( + header::USER_AGENT, + header::HeaderValue::from_str(config.user_agent())?, + ), ]); let client = Client::builder() @@ -47,7 +52,7 @@ where ); } Ok(response) => { - return Err(Error::ApiError { + return Err(Error::HttpStatusError { status: response.status().as_u16(), message: "Server error".to_string(), }) @@ -70,31 +75,83 @@ where Err(Error::MaxRetriesExceeded) } +pub async fn execute_without_retry(f: F) -> Result +where + F: Fn() -> Fut, + Fut: Future>, +{ + f().await.map_err(Error::from) +} + /// Handle HTTP response and deserialize. pub async fn handle_response(response: Response) -> Result where T: serde::de::DeserializeOwned, { - let status = response.status(); - - if status.is_success() { + if response.status().is_success() { response.json::().await.map_err(Error::from) } else { - let error_body = response - .text() - .await - .unwrap_or_else(|_| "Unknown error".to_string()); - - match status { - StatusCode::UNAUTHORIZED => Err(Error::AuthenticationFailed), - StatusCode::NOT_FOUND => Err(Error::ApiError { - status: status.as_u16(), - message: format!("Resource not found: {error_body}"), - }), - _ => Err(Error::ApiError { - status: status.as_u16(), - message: error_body, - }), + Err(error_from_response(response).await) + } +} + +/// Handle a response that carries no body (or a body the caller ignores). Preserves the server's +/// error message on failure without decoding the success body. +pub async fn handle_empty_response(response: Response) -> Result<()> { + if response.status().is_success() { + Ok(()) + } else { + Err(error_from_response(response).await) + } +} + +/// Build an error from a non-success response, preserving the server's body and mapping +/// authentication / not-found statuses. +async fn error_from_response(response: Response) -> Error { + let status = response.status(); + let error_body = response + .text() + .await + .unwrap_or_else(|_| "Unknown error".to_string()); + + match status { + StatusCode::UNAUTHORIZED => { + unity_catalog_delta_client_api::Error::AuthenticationFailed.into() } + StatusCode::NOT_FOUND => Error::HttpStatusError { + status: status.as_u16(), + message: format!("Resource not found: {error_body}"), + }, + _ => Error::HttpStatusError { + status: status.as_u16(), + message: error_body, + }, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::config::ClientConfig; + + #[test] + fn build_http_client_accepts_composed_user_agent() { + let config = ClientConfig::build("example.com", "t") + .with_additional_user_agent([("Spark", "3.5.0")]) + .build() + .unwrap(); + build_http_client(&config).expect("composed user_agent must be a valid header value"); + } + + #[test] + fn build_http_client_rejects_invalid_additional_user_agent_chars() { + let config = ClientConfig::build("example.com", "t") + .with_additional_user_agent([("bad\nname", "1.0")]) + .build() + .unwrap(); + assert!(matches!( + build_http_client(&config), + Err(Error::InvalidHeaderValue(_)) + )); } } diff --git a/unity-catalog-delta-rest-client/src/lib.rs b/unity-catalog-delta-rest-client/src/lib.rs new file mode 100644 index 0000000000..3d106ce44b --- /dev/null +++ b/unity-catalog-delta-rest-client/src/lib.rs @@ -0,0 +1,34 @@ +//! REST client implementation for Unity Catalog Delta APIs. +//! +//! This crate provides HTTP-based implementations of the traits defined in +//! [`unity_catalog_delta_client_api`]. +//! +//! # Example +//! +//! ```no_run +//! use unity_catalog_delta_rest_client::{ClientConfig, UCClient}; +//! +//! #[tokio::main] +//! async fn main() -> Result<(), Box> { +//! let config = ClientConfig::build("uc.awesome.org", "your-token") +//! .with_additional_user_agent([("MyEngine", "1.0.0"), ("MyConnector", "1.0.0")]) +//! .build()?; +//! let client = UCClient::new(config)?; +//! +//! let resp = client.load_table("main", "default", "my_table").await?; +//! println!("table id: {}", resp.metadata.table_uuid); +//! Ok(()) +//! } +//! ``` + +pub mod clients; +pub mod config; +pub mod error; +pub mod http; + +#[cfg(test)] +mod tests; + +pub use clients::{UCClient, UCUpdateTableRestClient}; +pub use config::{ClientConfig, ClientConfigBuilder}; +pub use error::{Error, Result}; diff --git a/unity-catalog-delta-rest-client/src/tests.rs b/unity-catalog-delta-rest-client/src/tests.rs new file mode 100644 index 0000000000..329e3db960 --- /dev/null +++ b/unity-catalog-delta-rest-client/src/tests.rs @@ -0,0 +1,7 @@ +use unity_catalog_delta_client_api::Operation; + +#[test] +fn test_operation_display() { + assert_eq!(Operation::Read.to_string(), "READ"); + assert_eq!(Operation::ReadWrite.to_string(), "READ_WRITE"); +} diff --git a/unity-catalog-delta-rest-client/tests/integration_live_server.rs b/unity-catalog-delta-rest-client/tests/integration_live_server.rs new file mode 100644 index 0000000000..bde48b5afe --- /dev/null +++ b/unity-catalog-delta-rest-client/tests/integration_live_server.rs @@ -0,0 +1,185 @@ +//! Live integration tests that exercise the UC REST client against a real Unity Catalog +//! server. +//! +//! These are gated behind the `integration-test` feature so they are never compiled or run by a +//! normal `cargo test`. The dedicated CI workflow (`.github/workflows/unitycatalog_oss_test.yml`) +//! builds + starts a UC server and runs them: +//! +//! cargo nextest run -p unity-catalog-delta-rest-client --features integration-test -E +//! 'test(live_)' +#![cfg(feature = "integration-test")] + +use unity_catalog_delta_client_api::{CreateStagingTableRequest, Operation}; +use unity_catalog_delta_rest_client::{ClientConfig, UCClient}; + +/// Reads the server URL + token from the environment, or `None` to skip the test. +fn server_env() -> Option<(String, String)> { + let url = std::env::var("UC_SERVER_URL").ok()?; + let token = std::env::var("UC_TOKEN").unwrap_or_else(|_| "not-used".to_string()); + Some((url, token)) +} + +fn client(url: &str, token: &str) -> UCClient { + let config = ClientConfig::build(url, token) + .build() + .expect("failed to build ClientConfig"); + UCClient::new(config).expect("failed to build UCClient") +} + +#[tokio::test(flavor = "multi_thread")] +#[ignore = "requires a running UC server; run with --run-ignored (UC CI job or manual)"] +async fn live_get_config_round_trips() { + let Some((url, token)) = server_env() else { + eprintln!("UC_SERVER_URL unset; skipping live_get_config_round_trips"); + return; + }; + let catalog = std::env::var("UC_TEST_CATALOG").unwrap_or_else(|_| "unity".to_string()); + + let resp = client(&url, &token) + .get_config(&catalog, &["1.0"]) + .await + .expect("get_config failed"); + + assert!( + !resp.protocol_version.is_empty(), + "expected a protocol version from the server" + ); + assert!( + !resp.endpoints.is_empty(), + "expected the server to advertise at least one endpoint" + ); +} + +#[tokio::test(flavor = "multi_thread")] +#[ignore = "requires a running UC server; run with --run-ignored (UC CI job or manual)"] +async fn live_load_table_reads_metadata() { + let Some((url, token)) = server_env() else { + eprintln!("UC_SERVER_URL unset; skipping live_load_table_reads_metadata"); + return; + }; + let Some(table) = std::env::var("UC_TEST_TABLE").ok() else { + eprintln!("UC_TEST_TABLE unset; skipping live_load_table_reads_metadata"); + return; + }; + let catalog = std::env::var("UC_TEST_CATALOG").unwrap_or_else(|_| "unity".to_string()); + let schema = std::env::var("UC_TEST_SCHEMA").unwrap_or_else(|_| "default".to_string()); + + let resp = client(&url, &token) + .load_table(&catalog, &schema, &table) + .await + .expect("load_table failed"); + let meta = &resp.metadata; + + assert!( + !meta.table_uuid.is_empty(), + "expected a table_uuid in the load_table response" + ); + assert_eq!(meta.table_type, "MANAGED", "expected a managed table"); + assert!( + meta.location.starts_with("file:") || meta.location.starts_with("s3"), + "expected an absolute storage location, got {:?}", + meta.location + ); + let columns = meta + .columns + .get("fields") + .and_then(|f| f.as_array()) + .expect("expected columns to decode as a StructType with a `fields` array"); + let names: Vec<&str> = columns + .iter() + .filter_map(|f| f.get("name").and_then(|n| n.as_str())) + .collect(); + assert_eq!(names, ["id", "name"], "expected the seeded columns"); + + assert_eq!( + resp.latest_table_version, + Some(0), + "expected latest_table_version 0 for a freshly created table" + ); +} + +#[tokio::test(flavor = "multi_thread")] +#[ignore = "requires a running UC server; run with --run-ignored (UC CI job or manual)"] +async fn live_get_table_credentials() { + let Some((url, token)) = server_env() else { + eprintln!("UC_SERVER_URL unset; skipping live_get_table_credentials"); + return; + }; + let Some(table) = std::env::var("UC_TEST_TABLE").ok() else { + eprintln!("UC_TEST_TABLE unset; skipping live_get_table_credentials"); + return; + }; + let catalog = std::env::var("UC_TEST_CATALOG").unwrap_or_else(|_| "unity".to_string()); + let schema = std::env::var("UC_TEST_SCHEMA").unwrap_or_else(|_| "default".to_string()); + + let client = client(&url, &token); + + let location = client + .load_table(&catalog, &schema, &table) + .await + .expect("load_table failed") + .metadata + .location; + + let resp = client + .get_table_credentials(&catalog, &schema, &table, Operation::Read) + .await + .expect("get_table_credentials failed"); + + assert!( + !resp.storage_credentials.is_empty(), + "expected the server to vend at least one credential" + ); + for cred in &resp.storage_credentials { + assert!( + cred.prefix.contains("://"), + "expected a storage-URL prefix, got {:?}", + cred.prefix + ); + assert_eq!( + cred.operation, + Operation::Read, + "expected the vended credential to echo the requested operation" + ); + let prefix = cred.prefix.trim_end_matches('/'); + let table_location = location.trim_end_matches('/'); + assert!( + table_location == prefix || table_location.starts_with(&format!("{prefix}/")), + "vended prefix {:?} should scope the table location {:?}", + cred.prefix, + location + ); + } +} + +/// Validates the `CreateStagingTableRequest` / `CreateStagingTableResponse` wire types against a +/// real server. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "requires a running UC server; run with --run-ignored (UC CI job or manual)"] +async fn live_create_staging_table() { + let Some((url, token)) = server_env() else { + eprintln!("UC_SERVER_URL unset; skipping live_create_staging_table"); + return; + }; + let catalog = std::env::var("UC_TEST_CATALOG").unwrap_or_else(|_| "unity".to_string()); + let schema = std::env::var("UC_TEST_SCHEMA").unwrap_or_else(|_| "default".to_string()); + let table = "delta_rest_client_staging_test"; + + let req = CreateStagingTableRequest { + name: table.to_string(), + }; + let staging = client(&url, &token) + .create_staging_table(&catalog, &schema, req) + .await + .expect("create_staging_table failed"); + assert!( + !staging.table_id.is_empty(), + "expected an allocated table id" + ); + assert!( + staging.location.starts_with("file:") || staging.location.starts_with("s3"), + "expected an absolute storage location, got {:?}", + staging.location + ); + assert_eq!(staging.table_type, "MANAGED", "staging tables are managed"); +} diff --git a/workloads/Cargo.toml b/workloads/Cargo.toml new file mode 100644 index 0000000000..148709e7fb --- /dev/null +++ b/workloads/Cargo.toml @@ -0,0 +1,31 @@ +[package] +name = "delta_kernel_workloads" +publish = false +edition.workspace = true +homepage.workspace = true +license.workspace = true +repository.workspace = true +version.workspace = true +rust-version.workspace = true + +# for cargo-release +[package.metadata.release] +release = false + +[features] +default = ["arrow"] +arrow = ["arrow-59"] +arrow-59 = ["delta_kernel/arrow-59"] +arrow-58 = ["delta_kernel/arrow-58"] + +[dependencies] +delta_kernel = { path = "../kernel", features = ["internal-api"] } +itertools = "0.14" +serde = { version = "1", features = ["derive"] } +serde_json = "1" +sqlparser = "0.55" +url = { version = "2", features = ["serde"] } + +[dev-dependencies] +rstest = "0.23" +tempfile = "3" diff --git a/workloads/src/lib.rs b/workloads/src/lib.rs new file mode 100644 index 0000000000..020fdcf7ed --- /dev/null +++ b/workloads/src/lib.rs @@ -0,0 +1,5 @@ +//! Shared workload specification types and predicate parsing for Delta Kernel +//! correctness testing (`acceptance`) and benchmarking (`delta_kernel_benchmarks`). + +pub mod models; +pub mod predicate_parser; diff --git a/workloads/src/models.rs b/workloads/src/models.rs new file mode 100644 index 0000000000..5b9d63a954 --- /dev/null +++ b/workloads/src/models.rs @@ -0,0 +1,560 @@ +//! Data models for workload specifications + +use std::collections::HashMap; +use std::path::{Path, PathBuf}; + +use delta_kernel::actions::{Metadata, Protocol}; +use delta_kernel::schema::Schema; +use serde::Deserialize; +use url::Url; + +/// Info needed to access a UC-managed table via credential vending. +/// This covers both catalog-managed and non-catalog-managed UC tables. +#[derive(Clone, Debug, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct CatalogInfo { + /// Fully-qualified table name: "catalog.schema.table" + pub table_name: String, +} + +/// Table info JSON files are located at the root of each table directory +#[derive(Clone, Debug, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct TableInfo { + /// Human-readable label used in benchmark output. + pub name: String, + /// Human-readable description of the table. Use this to capture context that the name alone + /// doesn't convey (e.g. "A table with 1 commit with 1M add actions. This includes a commit + /// file in the delta log, but no actual Parquet data files and no CRC files"). + /// The description is a free-form more verbose description for human readers. + pub description: String, + /// URL to the table. Used for remote tables (e.g. `s3://my-bucket/my-table`) or (rarely) + /// absolute local paths. If `None`, the table is assumed to be in the `delta/` subdirectory + /// next to `tableInfo.json`. Mutually exclusive with `catalog_info`. + pub table_path: Option, + /// Info needed to access a catalog-managed table via credential vending. + /// When present, the engine is set up with catalog-vended credentials instead of local/S3 + /// access. + /// Mutually exclusive with `table_path`. + /// TODO(#2303): Create an enum type that ensures table_path and catalog_info are mutually + /// exclusive + pub catalog_info: Option, + /// Schema at the latest version of the table, in Delta protocol JSON format + /// e.g. `{"type": "struct", "fields": [...]}` + pub schema: Schema, + /// Delta protocol requirements at the latest version of the table + /// e.g. `{"minReaderVersion": 3, "minWriterVersion": 7, "readerFeatures": [], + /// "writerFeatures": []}` + pub protocol: Protocol, + /// Log-level statistics giving a quick overview of the table without requiring a full log + /// replay. See [`LogInfo`] for field details. + pub log_info: LogInfo, + /// Table properties from the Delta metadata action (string key-value pairs). Use `{}` if + /// none. e.g. `{"delta.enableDeletionVector": "true", "delta.columnMapping.mode": "none"}` + pub properties: HashMap, + /// Physical data layout of the table. Use `{}` for unpartitioned/unclustered tables. + /// See [`DataLayout`] for partitioned and clustered variants. + pub data_layout: DataLayout, + /// Tags for filtering which tables are benchmarked via `BENCH_TAGS`. Use `[]` if none. + /// Built-in tag: `base` (run in CI). e.g. `["base", "my-feature"]` + #[serde(default)] + pub tags: Vec, + /// Path to the directory containing the `tableInfo.json` file + #[serde(skip, default)] + pub table_info_dir: PathBuf, +} + +impl TableInfo { + /// Returns true if the table has at least one tag in common with `required` + /// Tag matching uses union semantics; any single matching tag is sufficient + pub fn matches_tags(&self, required: &[String]) -> bool { + required.iter().any(|r| self.tags.contains(r)) + } + + pub fn resolved_table_root(&self) -> Url { + self.table_path.clone().unwrap_or_else(|| { + // If table path is not provided, assume that the Delta table is in a delta/ + // subdirectory at the same level as tableInfo.json + Url::from_file_path(self.table_info_dir.join("delta")) + .expect("table_info_dir must be an absolute path") + }) + } + + /// Returns the basename of [`Self::table_info_dir`] used as the benchmark registry table key. + /// + /// Returns `None` when the table info has no directory context or the basename is not valid + /// UTF-8. + pub fn registry_table_key(&self) -> Option<&str> { + self.table_info_dir.file_name()?.to_str() + } + + pub fn from_json_path>(path: P) -> Result { + let content = std::fs::read_to_string(path.as_ref()).map_err(serde_json::Error::io)?; + let mut table_info: TableInfo = serde_json::from_str(&content)?; + if table_info.catalog_info.is_some() && table_info.table_path.is_some() { + return Err(serde_json::Error::io(std::io::Error::new( + std::io::ErrorKind::InvalidData, + "catalog_info and table_path are mutually exclusive", + ))); + } + if let Some(parent) = path.as_ref().parent() { + table_info.table_info_dir = parent.to_path_buf(); + } + Ok(table_info) + } +} + +/// Log-level information describing the history and structure of a Delta table +#[derive(Clone, Debug, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct LogInfo { + /// Number of active Add file actions in the table + pub num_add_files: u64, + /// Number of Remove file actions in the table + pub num_remove_files: u64, + /// Total on-disk size of all data files in bytes + pub size_in_bytes: Option, + /// Number of commits (JSON log files) in the table history + pub num_commits: u64, + /// Total number of actions across all commits + pub num_actions: u64, + /// Version of the most recent checkpoint, if any + pub last_checkpoint_version: Option, + /// Version of the most recent CRC file, if any + pub last_crc_version: Option, + /// Number of part files in the most recent multi-part checkpoint, if any. + /// For classic multi-part checkpoints this is the number of parquet parts; for V2 checkpoints + /// this is the number of sidecar files For workloads that don't have multi-part + /// checkpoints/sidecars, this is `None` + pub num_checkpoint_files: Option, +} + +/// Physical data layout of a Delta table +#[derive(Clone, Debug, Deserialize)] +#[serde(untagged)] +pub enum DataLayout { + /// Partitioned table. Two tables with the same number of partition columns can differ + /// significantly in cardinality (e.g. 1 column with 100 distinct values vs. 10000), so both + /// are tracked. e.g. `{"numPartitionColumns": 2, "numDistinctPartitions": 100}` + Partitioned { + /// Number of partition columns + #[serde(rename = "numPartitionColumns")] + num_partition_columns: u32, + /// Number of distinct partition values observed in the table + #[serde(rename = "numDistinctPartitions")] + num_distinct_partitions: u64, + }, + /// Clustered table, e.g. `{"numClusteringColumns": 1}` + Clustered { + /// Number of clustering columns + #[serde(rename = "numClusteringColumns")] + num_clustering_columns: u32, + }, + /// No special data organization (unpartitioned, unclustered). Serializes as `{}` + None {}, +} + +/// Time travel parameter. Either version or timestamp. +#[derive(Clone, Debug, Deserialize)] +#[serde(untagged)] +pub enum TimeTravel { + /// Timetravel to a specific snapshot/read version specified by the workload JSON. + /// It is i64 so that negative values from the test spec can be deserialized correctly. + /// [`TimeTravel::as_version`] will reject values below zero. + Version { version: i64 }, + /// Timetravel to a specific timestamp specified by the workload JSON. + /// (not yet supported) + Timestamp { timestamp: String }, +} + +impl TimeTravel { + /// Returns the version if this is version-based time travel and the version is non-negative. + /// + /// Returns an error for negative versions or for timestamp-based time travel (which is not + /// yet supported). + pub fn as_version(&self) -> Result { + match self { + TimeTravel::Version { version } => u64::try_from(*version) + .map_err(|_| "Only non-negative snapshot versions are supported"), + TimeTravel::Timestamp { .. } => Err("Timestamp-based time travel is not yet supported"), + } + } +} + +/// Spec defines the operation performed on a table - defines what operation at what version (e.g. +/// read at version 0) There will be multiple specs for a given table +#[derive(Clone, Debug, Deserialize)] +#[serde(tag = "type", rename_all = "camelCase")] +pub enum Spec { + Read(ReadSpec), + #[serde(alias = "snapshot_construction", alias = "snapshot")] + SnapshotConstruction(Box), +} + +/// Specification for a read workload. +#[derive(Clone, Debug, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct ReadSpec { + // Time travel version or timestamp for the read + #[serde(flatten)] + pub time_travel: Option, + /// SQL WHERE clause expression (e.g. "id < 500"). Parsed into a kernel `Predicate` + /// and passed to the scan builder for data skipping. + pub predicate: Option, + // Column projections to read + pub columns: Option>, + /// Expected outcome - either success with row count or error with code. + #[serde(flatten)] + pub expected: Option, +} + +impl ReadSpec { + pub fn as_str(&self) -> &str { + "read" + } +} + +/// Specification for a snapshot construction workload. +#[derive(Clone, Debug, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct SnapshotConstructionSpec { + // Time travel version or timestamp for the read + #[serde(flatten)] + pub time_travel: Option, + /// Expected outcome - either success with protocol/metadata or error with code. + #[serde(flatten)] + pub expected: Option, +} + +impl SnapshotConstructionSpec { + pub fn as_str(&self) -> &str { + "snapshotConstruction" + } +} + +impl Spec { + pub fn as_str(&self) -> &str { + match self { + Spec::Read(read_spec) => read_spec.as_str(), + Spec::SnapshotConstruction(snapshot_construction_spec) => { + snapshot_construction_spec.as_str() + } + } + } + + pub fn from_json_path>(path: P) -> Result { + let content = std::fs::read_to_string(path.as_ref()).map_err(serde_json::Error::io)?; + let spec: Spec = serde_json::from_str(&content)?; + Ok(spec) + } +} + +/// Expected error outcome for a workload. +#[derive(Debug, Clone, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct ExpectedError { + pub error_code: String, + pub error_message: Option, +} + +/// Expected success outcome for a read workload. +#[derive(Debug, Clone, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct ReadExpectedSuccess { + pub row_count: u64, + pub file_count: Option, + pub files_skipped: Option, +} + +/// Expected result for read operations - either success or error. +#[derive(Debug, Clone, Deserialize)] +#[serde(untagged)] +pub enum ReadExpected { + Success { expected: ReadExpectedSuccess }, + Error { error: ExpectedError }, +} + +/// Expected success outcome for a snapshot construction workload. +#[derive(Debug, Clone, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct SnapshotExpectedSuccess { + pub protocol: Box, + pub metadata: Box, +} + +/// Expected result for snapshot operations - either success or error. +#[derive(Debug, Clone, Deserialize)] +#[serde(untagged)] +pub enum SnapshotExpected { + Success { expected: SnapshotExpectedSuccess }, + Error { error: ExpectedError }, +} + +/// For Read specs, we will either run a read data operation or a read metadata operation +#[derive(Clone, Copy, Debug)] +pub enum ReadOperation { + ReadData, + ReadMetadata, +} + +impl ReadOperation { + pub fn as_str(&self) -> &str { + match self { + ReadOperation::ReadData => "readData", + ReadOperation::ReadMetadata => "readMetadata", + } + } +} + +/// Partial workload specification loaded from JSON - table, case name, and spec only +#[derive(Clone, Debug)] +pub struct Workload { + pub table_info: TableInfo, + /// Spec filename without extension; used as the benchmark case label + pub case_name: String, + pub spec: Spec, +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + + fn make_table_info(tags: &[&str]) -> TableInfo { + let tags_json = serde_json::to_string(tags).unwrap(); + let json = format!( + r#"{{"name":"t","description":"d","schema":{{"type":"struct","fields":[]}},"protocol":{{"minReaderVersion":1,"minWriterVersion":2}},"logInfo":{{"numAddFiles":0,"numRemoveFiles":0,"sizeInBytes":0,"numCommits":1,"numActions":1}},"properties":{{}},"dataLayout":{{}},"tags":{}}}"#, + tags_json + ); + serde_json::from_str(&json).unwrap() + } + + #[rstest] + #[case(&["ci", "checkpoints"], &["ci"], true)] + #[case(&["ci", "checkpoints"], &["ci", "large"], true)] + #[case(&["large"], &["ci", "checkpoints"], false)] + #[case(&[], &["ci"], false)] + fn test_matches_tags( + #[case] table_tags: &[&str], + #[case] required: &[&str], + #[case] expected: bool, + ) { + let info = make_table_info(table_tags); + let required: Vec = required.iter().map(|s| s.to_string()).collect(); + assert_eq!(info.matches_tags(&required), expected); + } + + #[rstest] + #[case( + r#"{ + "name": "test_table", + "description": "A test table", + "tablePath": "s3://bucket/test_table", + "schema": {"type": "struct", "fields": [{"name": "id", "type": "long", "nullable": true, "metadata": {}}]}, + "protocol": {"minReaderVersion": 1, "minWriterVersion": 2}, + "logInfo": {"numAddFiles": 3, "numRemoveFiles": 1, "sizeInBytes": 1535, "numCommits": 100, "numActions": 10000}, + "properties": {}, + "dataLayout": {"numPartitionColumns": 2, "numDistinctPartitions": 4}, + "tags": ["base"] + }"#, + "test_table", + "A test table", + Some(Url::parse("s3://bucket/test_table").unwrap()), + &["base"] + )] + #[case( + r#"{ + "name": "no_path_table", + "description": "No path specified", + "schema": {"type": "struct", "fields": []}, + "protocol": {"minReaderVersion": 1, "minWriterVersion": 2}, + "logInfo": {"numAddFiles": 0, "numRemoveFiles": 0, "sizeInBytes": 0, "numCommits": 1, "numActions": 1}, + "properties": {}, + "dataLayout": {"numClusteringColumns": 2}, + "tags": ["base"] + }"#, + "no_path_table", + "No path specified", + None, + &["base"] + )] + #[case( + r#"{ + "name": "extra_fields_table", + "description": "Has extra fields", + "extraField": "should be ignored", + "schema": {"type": "struct", "fields": []}, + "protocol": {"minReaderVersion": 1, "minWriterVersion": 2}, + "logInfo": {"numAddFiles": 0, "numRemoveFiles": 0, "sizeInBytes": 0, "numCommits": 1, "numActions": 1}, + "properties": {}, + "dataLayout": {"numClusteringColumns": 1}, + "tags": [] + }"#, + "extra_fields_table", + "Has extra fields", + None, + &[] + )] + fn test_deserialize_table_info( + #[case] json: &str, + #[case] expected_name: &str, + #[case] expected_description: &str, + #[case] expected_table_path: Option, + #[case] expected_tags: &[&str], + ) { + let table_info: TableInfo = + serde_json::from_str(json).expect("Failed to deserialize table info"); + + assert_eq!(table_info.name, expected_name); + assert_eq!(table_info.description, expected_description); + assert_eq!(table_info.table_path, expected_table_path); + let expected_tags: Vec = expected_tags.iter().map(|s| s.to_string()).collect(); + assert_eq!(table_info.tags, expected_tags); + } + + #[rstest] + #[case( + r#"{ + "name": "catalog_table", "description": "A catalog-managed table", + "catalogInfo": {"tableName": "main.schema.table"}, + "schema": {"type": "struct", "fields": []}, + "protocol": {"minReaderVersion": 1, "minWriterVersion": 2}, + "logInfo": {"numAddFiles": 0, "numRemoveFiles": 0, "sizeInBytes": 0, "numCommits": 1, "numActions": 1}, + "properties": {}, "dataLayout": {}, "tags": [] + }"#, + true, + "main.schema.table" + )] + #[case( + r#"{ + "name": "local_table", "description": "A local table", + "schema": {"type": "struct", "fields": []}, + "protocol": {"minReaderVersion": 1, "minWriterVersion": 2}, + "logInfo": {"numAddFiles": 0, "numRemoveFiles": 0, "sizeInBytes": 0, "numCommits": 1, "numActions": 1}, + "properties": {}, "dataLayout": {}, "tags": [] + }"#, + false, + "" + )] + fn test_deserialize_catalog_info_field( + #[case] json: &str, + #[case] expect_present: bool, + #[case] expected_table_name: &str, + ) { + let table_info: TableInfo = + serde_json::from_str(json).expect("Failed to deserialize table info"); + assert_eq!(table_info.catalog_info.is_some(), expect_present); + if expect_present { + assert_eq!( + table_info.catalog_info.unwrap().table_name, + expected_table_name + ); + } + } + + #[rstest] + #[case(r#"{"description": "missing name"}"#, "missing field")] + #[case( + r#"{"name": "missing_schema", "description": "d", + "protocol": {"minReaderVersion": 1, "minWriterVersion": 2}, + "logInfo": {"numAddFiles": 0, "numRemoveFiles": 0, "sizeInBytes": 0, "numCommits": 1, "numActions": 1}, + "properties": {}, "dataLayout": {"numClusteringColumns": 1}, "tags": []}"#, + "missing field" + )] + fn test_deserialize_table_info_errors(#[case] json: &str, #[case] expected_msg: &str) { + let error = serde_json::from_str::(json).unwrap_err(); + assert!(error.to_string().contains(expected_msg)); + } + + #[rstest] + #[case(r#"{"type": "read", "version": 5}"#, "read", Some(5))] + #[case(r#"{"type": "read"}"#, "read", None)] + #[case( + r#"{"type": "read", "version": 7, "extraField": "should be ignored"}"#, + "read", + Some(7) + )] + #[case( + r#"{"type": "snapshotConstruction", "version": 5}"#, + "snapshotConstruction", + Some(5) + )] + #[case(r#"{"type": "snapshotConstruction"}"#, "snapshotConstruction", None)] + #[case( + r#"{"type": "snapshotConstruction", "version": 7, "extraField": "should be ignored"}"#, + "snapshotConstruction", + Some(7) + )] + #[case( + r#"{"type": "snapshotConstruction", "version": -1}"#, + "snapshotConstruction", + Some(-1) + )] + #[case(r#"{"type": "read", "version": -1}"#, "read", Some(-1))] + fn test_deserialize_spec( + #[case] json: &str, + #[case] expected_type: &str, + #[case] expected_version: Option, + ) { + let spec: Spec = serde_json::from_str(json).expect("Failed to deserialize spec"); + assert_eq!(spec.as_str(), expected_type); + let version = match &spec { + Spec::Read(read_spec) => match &read_spec.time_travel { + Some(TimeTravel::Version { version }) => Some(*version), + _ => None, + }, + Spec::SnapshotConstruction(snapshot_construction_spec) => { + match &snapshot_construction_spec.time_travel { + Some(TimeTravel::Version { version }) => Some(*version), + _ => None, + } + } + }; + + assert_eq!(version, expected_version); + } + + #[test] + fn test_time_travel_as_version_rejects_negative() { + let tt = TimeTravel::Version { version: -1 }; + assert_eq!( + tt.as_version(), + Err("Only non-negative snapshot versions are supported") + ); + } + + #[rstest] + #[case(r#"{"version": 10}"#, "missing field")] + #[case(r#"{"type": "write", "version": 3}"#, "unknown variant")] + fn test_deserialize_spec_errors(#[case] json: &str, #[case] expected_msg: &str) { + let error = serde_json::from_str::(json).unwrap_err(); + assert!(error.to_string().contains(expected_msg)); + } + + #[test] + fn registry_table_key_uses_directory_basename_not_name_field() { + // The registry key is the directory basename, not the JSON `name` field. + let dir = tempfile::tempdir().unwrap(); + let table_dir = dir.path().join("dirBasename"); + std::fs::create_dir(&table_dir).unwrap(); + let json = r#"{ + "name": "humanLabel", "description": "d", "tablePath": "s3://b/t", + "schema": {"type": "struct", "fields": []}, + "protocol": {"minReaderVersion": 1, "minWriterVersion": 2}, + "logInfo": {"numAddFiles": 0, "numRemoveFiles": 0, "sizeInBytes": 0, "numCommits": 1, "numActions": 1}, + "properties": {}, "dataLayout": {}, "tags": [] + }"#; + std::fs::write(table_dir.join("tableInfo.json"), json).unwrap(); + + let table_info = TableInfo::from_json_path(table_dir.join("tableInfo.json")).unwrap(); + assert_eq!(table_info.registry_table_key(), Some("dirBasename")); + assert_eq!(table_info.name, "humanLabel"); + } + + #[test] + fn registry_table_key_is_none_without_directory_context() { + let info = make_table_info(&[]); + assert_eq!(info.registry_table_key(), None); + } +} diff --git a/workloads/src/predicate_parser.rs b/workloads/src/predicate_parser.rs new file mode 100644 index 0000000000..ba84b964b0 --- /dev/null +++ b/workloads/src/predicate_parser.rs @@ -0,0 +1,955 @@ +//! Parses SQL WHERE clause expressions into kernel [`KPred`] types. +//! +//! Supports a subset of SQL sufficient for workload predicates: +//! - Binary comparisons: `=`, `!=`, `<>`, `<`, `>`, `<=`, `>=` +//! - Null-safe equals: `<=>` +//! - Logical operators: `AND`, `OR`, `NOT` +//! - Arithmetic operators: `+`, `-`, `*`, `/`, `%` +//! - `IS NULL`, `IS NOT NULL` +//! - `IN (...)`, `NOT IN (...)` +//! - `BETWEEN ... AND ...` +//! - Column references and literal values (integers, floats, strings, booleans) +//! - Typed literals: `DATE'...'`, `TIMESTAMP'...'`, `TIMESTAMP_NTZ'...'` +//! +//! Unsupported (returns error): `LIKE`, function calls (`HEX`, `size`, `length`), +//! `TIME '...'` typed literals. +//! +//! # Type resolution +//! +//! This module uses bidirectional type checking to infer the correct types for literals and to +//! verify that the resulting predicate is well-typed. Bidirectional type checking has two modes: +//! +//! 1) **Type synthesis**: Given an expression, try to infer its type. For example, given a schema +//! where `int_col` has type `Integer`, the column reference `int_col` synthesizes to `Integer`. +//! Synthesis returns `Some(type)` for unambiguous types, or `None` when the type is ambiguous. +//! +//! 2) **Type checking**: Given an expression and a target type, check that the expression can +//! resolve to that type. For example, the literal `5` will successfully type check against +//! `Integer`, `Short`, or `Long`, but will fail against `String`. +//! +//! The predicate parser uses synthesis and checking to correctly resolve the types of literals so +//! that predicates are well-formed. Consider the predicate `double_col > 3.14`: +//! +//! 1) Synthesize the LHS `double_col`. The schema lookup determines the type is `Double`. +//! 2) Synthesize the RHS `3.14`. The type is ambiguous (`Float` or `Double`), so synthesis returns +//! `None`. +//! 3) Since LHS has a concrete type (`Double`) and RHS is ambiguous, use `Double` as the target. +//! 4) Type check both sides against `Double`, producing `Column("double_col")` on the LHS and +//! `Scalar::Double(3.14)` on the RHS. +//! 5) Return the resolved predicate: `Predicate::gt(Column("double_col"), Scalar::Double(3.14))` + +use std::borrow::Cow; + +// Imports use a naming convention to distinguish kernel types from sqlparser types: +// - K-prefix: Kernel types (KExpr, KPred, KBinOp, KPredOp) +// - P-prefix: Parser/sqlparser types (PExpr, PBinOp, PUnaryOp, PVal) +use delta_kernel::expressions::{ + ArrayData, BinaryExpressionOp as KBinOp, BinaryPredicateOp as KPredOp, ColumnName, + Expression as KExpr, Predicate as KPred, Scalar, +}; +use delta_kernel::schema::{ArrayType, DataType, PrimitiveType, Schema}; +use itertools::Itertools as _; +use sqlparser::ast::{ + BinaryOperator as PBinOp, Expr as PExpr, UnaryOperator as PUnaryOp, Value as PVal, +}; +use sqlparser::dialect::GenericDialect; +use sqlparser::parser::Parser; + +/// Preprocesses SQL to replace `TIMESTAMP_NTZ` with `TIMESTAMP` since sqlparser doesn't +/// recognize `TIMESTAMP_NTZ`. +/// +/// The actual type (Timestamp vs TimestampNtz) is determined by schema-based type inference +/// (the column's type determines parsing). +/// +/// Note: This may incorrectly match `TIMESTAMP_NTZ` inside string literals +/// (e.g., `str_col = 'TIMESTAMP_NTZ is cool'`), but this is acceptable for benchmark +/// predicates which don't contain such edge cases. +fn preprocess_timestamp_ntz(input: &str) -> Cow<'_, str> { + if input.contains("TIMESTAMP_NTZ") { + Cow::Owned(input.replace("TIMESTAMP_NTZ", "TIMESTAMP")) + } else { + Cow::Borrowed(input) + } +} + +/// Parses a SQL WHERE clause expression string into a kernel [`KPred`] with type checking. +/// +/// Performs bidirectional type checking using the provided schema: +/// - Column types are looked up in the schema +/// - Literals are checked against expected types from columns +/// - Type mismatches produce detailed error messages +/// +/// # Note +/// +/// `TIMESTAMP_NTZ` literals are preprocessed by stripping the prefix, since sqlparser doesn't +/// recognize this keyword. This preprocessing uses simple string matching and may incorrectly +/// match `TIMESTAMP_NTZ` inside string literals (e.g., `str_col = 'TIMESTAMP_NTZ is cool'`). +/// This is acceptable for benchmark predicates which don't contain such edge cases. +/// +/// # Example +/// ```ignore +/// let schema = Schema::try_new(vec![ +/// StructField::new("id", DataType::LONG, false), +/// StructField::new("name", DataType::STRING, false), +/// ])?; +/// let pred = parse_predicate("id < 500 AND name = 'alice'", &schema).unwrap(); +/// ``` +pub fn parse_predicate(sql: &str, schema: &Schema) -> Result> { + let sql = preprocess_timestamp_ntz(sql); + let dialect = GenericDialect {}; + let expr = Parser::new(&dialect) + .try_with_sql(&sql)? + .parse_expr() + .map_err(|e| format!("Failed to parse predicate: {e}"))?; + expr_to_predicate(schema, &expr) +} + +//////////////////////////////////////////////////////////////////////// +// Typed (bidirectional) conversion functions +//////////////////////////////////////////////////////////////////////// + +/// Tries to infer an expression's type, returning both the KExpr and its DataType. +/// +/// Returns `Some((KExpr, DataType))` for: +/// - Column references: look up type in schema, return column expression +/// - Boolean literals: return literal expression with BOOLEAN type +/// +/// Returns `None` if the expression needs type context (numeric literals, strings, NULL). +fn synthesize_expr(schema: &Schema, expr: &PExpr) -> Option<(KExpr, DataType)> { + match expr { + PExpr::Identifier(ident) => { + let ty = schema.field(ident.value.as_str())?.data_type().clone(); + let expr = KExpr::column([ident.value.clone()]); + Some((expr, ty)) + } + PExpr::CompoundIdentifier(parts) => { + let col_name = ColumnName::new(parts.iter().map(|p| p.value.clone())); + let ty = schema.field_at(&col_name).ok()?.data_type().clone(); + let expr = KExpr::column(parts.iter().map(|p| p.value.clone())); + Some((expr, ty)) + } + PExpr::Value(v) => match &v.value { + PVal::Boolean(b) => Some((Scalar::Boolean(*b).into(), DataType::BOOLEAN)), + _ => None, // Numeric, string, null need type context + }, + // Typed literals need type context from the other side of comparison + // because TIMESTAMP and TIMESTAMP_NTZ use the same syntax after preprocessing + PExpr::TypedString { .. } => None, + PExpr::Nested(inner) => synthesize_expr(schema, inner), + PExpr::BinaryOp { left, op, right } => { + let (l, r, ty) = synthesize_binary_exprs(schema, left, right).ok()?; + let expr = match op { + PBinOp::Plus => KExpr::binary(KBinOp::Plus, l, r), + PBinOp::Minus => KExpr::binary(KBinOp::Minus, l, r), + PBinOp::Multiply => KExpr::binary(KBinOp::Multiply, l, r), + PBinOp::Divide => KExpr::binary(KBinOp::Divide, l, r), + // Modulo: a % b = a - (b * (a / b)) + // This relies on kernel's Divide producing integer division for integer types, + // which truncates toward zero (like Rust/C, matching Spark behavior). + // Examples: + // 7 % 3 = 7 - (3 * (7 / 3)) = 7 - (3 * 2) = 1 + // -7 % 3 = -7 - (3 * (-7 / 3)) = -7 - (3 * -2) = -1 + // 7 % -3 = 7 - (-3 * (7 / -3)) = 7 - (-3 * -2) = 1 + PBinOp::Modulo => { + let a_div_b = KExpr::binary(KBinOp::Divide, l.clone(), r.clone()); + let b_times_quotient = KExpr::binary(KBinOp::Multiply, r, a_div_b); + KExpr::binary(KBinOp::Minus, l, b_times_quotient) + } + _ => return None, + }; + Some((expr, ty)) + } + _ => None, + } +} + +/// Checks that an expression has the expected type, returning the typed KExpr. +fn check_expr( + schema: &Schema, + expected_ty: &DataType, + expr: &PExpr, +) -> Result> { + match expr { + PExpr::Value(v) => check_literal(expected_ty, &v.value, false), + PExpr::UnaryOp { + op: PUnaryOp::Minus, + expr: inner, + } => match inner.as_ref() { + PExpr::Value(v) => check_literal(expected_ty, &v.value, true), + _ => Err(format!("Unsupported unary minus on: {expr}").into()), + }, + // Typed literals like DATE'2024-01-01' and TIMESTAMP '2024-01-01 00:00:00' + // Use expected_ty from column to parse with correct type (Timestamp vs TimestampNtz) + PExpr::TypedString { value, .. } => { + let (PVal::SingleQuotedString(s) | PVal::DoubleQuotedString(s)) = value else { + return Err(format!("Invalid typed string value: {value}").into()); + }; + let DataType::Primitive(prim) = expected_ty else { + return Err( + format!("Typed literal cannot be used for type: {expected_ty:?}").into(), + ); + }; + Ok(prim.parse_scalar(s).map_err(|e| e.to_string())?.into()) + } + PExpr::Nested(inner) => check_expr(schema, expected_ty, inner), + _ => { + // For non-literals, synthesize and verify compatibility + let (e, actual_ty) = synthesize_expr(schema, expr) + .ok_or_else(|| format!("Cannot determine type for: {expr}"))?; + match can_coerce(&actual_ty, expected_ty) { + true => Ok(e), + false => Err( + format!("Type mismatch: expected {expected_ty:?}, got {actual_ty:?}").into(), + ), + } + } + } +} + +/// Checks a literal value against an expected type and converts it to that type. +/// Uses kernel's `PrimitiveType::parse_scalar` for parsing. +fn check_literal( + expected_ty: &DataType, + value: &PVal, + is_negative: bool, +) -> Result> { + // Local import to avoid conflict with Scalar::* variants (Long, Integer, etc.) used in tests + use PrimitiveType::*; + match (expected_ty, value) { + // Numeric literals - only for numeric primitive types + ( + DataType::Primitive( + prim @ (Byte | Short | Integer | Long | Float | Double | Decimal(_)), + ), + PVal::Number(n, _), + ) => { + let raw = format!("{}{n}", if is_negative { "-" } else { "" }); + let scalar = prim.parse_scalar(&raw).map_err(|e| e.to_string())?; + Ok(scalar.into()) + } + + // String literals - use parse_scalar (handles String, Date, Timestamp, etc.) + (DataType::Primitive(prim), PVal::SingleQuotedString(s) | PVal::DoubleQuotedString(s)) => { + let scalar = prim.parse_scalar(s).map_err(|e| e.to_string())?; + Ok(scalar.into()) + } + + // Boolean literals + (DataType::Primitive(Boolean), PVal::Boolean(b)) => Ok(Scalar::Boolean(*b).into()), + + // NULL can be any type + (ty, PVal::Null) => Ok(Scalar::Null(ty.clone()).into()), + + // Type mismatches + (expected, actual) => Err(format!( + "Type mismatch: cannot use {:?} for column of type {:?}", + actual, expected + ) + .into()), + } +} + +/// Checks if one type can be coerced to another using kernel's type widening rules. +fn can_coerce(from: &DataType, to: &DataType) -> bool { + match (from, to) { + (DataType::Primitive(f), DataType::Primitive(t)) => f == t || f.can_widen_to(t), + _ => from == to, + } +} + +/// Finds the widest common type between two types, if one exists. +fn find_common_type(l_ty: &DataType, r_ty: &DataType) -> Option { + if l_ty == r_ty { + Some(l_ty.clone()) + } else if can_coerce(l_ty, r_ty) { + Some(r_ty.clone()) + } else if can_coerce(r_ty, l_ty) { + Some(l_ty.clone()) + } else { + None + } +} + +/// Synthesizes both operands of a binary operation and checks them against the widest common type. +/// Returns `Ok((left_expr, right_expr, common_type))` on success. +fn synthesize_binary_exprs( + schema: &Schema, + left: &PExpr, + right: &PExpr, +) -> Result<(KExpr, KExpr, DataType), Box> { + let l_syn = synthesize_expr(schema, left); + let r_syn = synthesize_expr(schema, right); + + let common_ty = match (&l_syn, &r_syn) { + (Some((_, l_ty)), Some((_, r_ty))) => find_common_type(l_ty, r_ty) + .ok_or_else(|| format!("Type mismatch: cannot compare {l_ty:?} with {r_ty:?}"))?, + (Some((_, ty)), None) | (None, Some((_, ty))) => ty.clone(), + (None, None) => { + return Err(format!("Cannot determine types for: {left:?} and {right:?}").into()) + } + }; + + // Type check each side to the common type, ensuring that the expression is coerced to the + // common type. + let l = check_expr(schema, &common_ty, left)?; + let r = check_expr(schema, &common_ty, right)?; + + Ok((l, r, common_ty)) +} + +/// Converts a sqlparser AST expression into a kernel [`KPred`] with type checking. +fn expr_to_predicate(schema: &Schema, expr: &PExpr) -> Result> { + match expr { + PExpr::BinaryOp { left, op, right } => binary_op_to_predicate(schema, left, op, right), + PExpr::UnaryOp { + op: PUnaryOp::Not, + expr, + } => Ok(KPred::not(expr_to_predicate(schema, expr)?)), + PExpr::IsNull(e) | PExpr::IsNotNull(e) => { + let (col_expr, _) = synthesize_expr(schema, e) + .ok_or_else(|| format!("Cannot synthesize type for IS NULL operand: {e}"))?; + match expr { + PExpr::IsNull(_) => Ok(KPred::is_null(col_expr)), + _ => Ok(KPred::is_not_null(col_expr)), + } + } + PExpr::Nested(inner) => expr_to_predicate(schema, inner), + PExpr::Value(v) => match &v.value { + PVal::Boolean(b) => Ok(KPred::literal(*b)), + _ => Err(format!("Unsupported literal in predicate position: {v}").into()), + }, + PExpr::Identifier(_) | PExpr::CompoundIdentifier(_) => { + let (col_expr, _) = + synthesize_expr(schema, expr).ok_or_else(|| format!("Unknown column: {expr}"))?; + Ok(KPred::from_expr(col_expr)) + } + PExpr::InList { + expr, + list, + negated, + } => in_list_to_pred(schema, expr, list, *negated), + PExpr::Between { + expr, + negated, + low, + high, + } => between_to_pred(schema, expr, *negated, low, high), + _ => Err(format!("Unsupported expression: {expr}").into()), + } +} + +/// Converts a binary comparison into a kernel [`KPred`] with bidirectional type checking. +fn binary_op_to_predicate( + schema: &Schema, + left: &PExpr, + op: &PBinOp, + right: &PExpr, +) -> Result> { + match op { + PBinOp::Eq + | PBinOp::NotEq + | PBinOp::Lt + | PBinOp::LtEq + | PBinOp::Gt + | PBinOp::GtEq + | PBinOp::Spaceship => { + let (l, r, _) = synthesize_binary_exprs(schema, left, right)?; + Ok(match op { + PBinOp::Eq => KPred::eq(l, r), + PBinOp::NotEq => KPred::ne(l, r), + PBinOp::Lt => KPred::lt(l, r), + PBinOp::LtEq => KPred::le(l, r), + PBinOp::Gt => KPred::gt(l, r), + PBinOp::GtEq => KPred::ge(l, r), + PBinOp::Spaceship => KPred::not(KPred::distinct(l, r)), + _ => unreachable!(), + }) + } + PBinOp::And => { + let l = expr_to_predicate(schema, left)?; + let r = expr_to_predicate(schema, right)?; + Ok(KPred::and(l, r)) + } + PBinOp::Or => { + let l = expr_to_predicate(schema, left)?; + let r = expr_to_predicate(schema, right)?; + Ok(KPred::or(l, r)) + } + _ => Err(format!("Unsupported binary operator: {op}").into()), + } +} + +/// Converts an IN/NOT IN list into a kernel [`KPred`] with type checking. +fn in_list_to_pred( + schema: &Schema, + expr: &PExpr, + list: &[PExpr], + negated: bool, +) -> Result> { + let (col_expr, col_ty) = synthesize_expr(schema, expr) + .ok_or_else(|| format!("IN requires a column, got: {expr}"))?; + + let scalars: Vec = list + .iter() + .map(|e| -> Result<_, Box> { + match check_expr(schema, &col_ty, e)? { + KExpr::Literal(s) => Ok(s), + _ => Err(format!("IN list elements must be literals, got: {e}").into()), + } + }) + .try_collect()?; + + // Check if any scalar is null to determine array nullability + let contains_null = scalars.iter().any(|s| s.is_null()); + let array_data = ArrayData::try_new(ArrayType::new(col_ty, contains_null), scalars)?; + let pred = KPred::binary( + KPredOp::In, + col_expr, + KExpr::literal(Scalar::Array(array_data)), + ); + Ok(if negated { KPred::not(pred) } else { pred }) +} + +/// Converts BETWEEN into `col >= low AND col <= high` with type checking. +fn between_to_pred( + schema: &Schema, + expr: &PExpr, + negated: bool, + low: &PExpr, + high: &PExpr, +) -> Result> { + let (expr_checked, low_checked, common_ty) = synthesize_binary_exprs(schema, expr, low)?; + let high_checked = check_expr(schema, &common_ty, high)?; + + let pred = KPred::and( + KPred::ge(expr_checked.clone(), low_checked), + KPred::le(expr_checked, high_checked), + ); + Ok(if negated { KPred::not(pred) } else { pred }) +} + +#[cfg(test)] +mod tests { + use delta_kernel::expressions::col; + use delta_kernel::expressions::Scalar::*; + use delta_kernel::schema::{MapType, Schema, StructField, StructType}; + use rstest::rstest; + + use super::*; + + /// Test schema with columns for all test cases. + fn test_schema() -> Schema { + Schema::new_unchecked(vec![ + // Columns matching upstream tests + StructField::new("id", DataType::LONG, true), + StructField::new("a", DataType::LONG, true), + StructField::new("b", DataType::LONG, true), + StructField::new("c", DataType::LONG, true), + StructField::new("name", DataType::STRING, true), + StructField::new("value", DataType::LONG, true), + StructField::new("flag", DataType::BOOLEAN, true), + StructField::new("partCol", DataType::LONG, true), + StructField::new("version_tag", DataType::STRING, true), + // Double columns for literal_types tests + StructField::new("c3", DataType::DOUBLE, true), + StructField::new("c4", DataType::DOUBLE, true), + StructField::new("val", DataType::DOUBLE, true), + StructField::new("cc9", DataType::BOOLEAN, true), + StructField::new("long_val", DataType::LONG, true), + // Columns for unsupported predicate tests (these trigger other errors first) + StructField::new("fruit", DataType::STRING, true), + StructField::new("cc8", DataType::STRING, true), + StructField::new("s", DataType::STRING, true), + StructField::new("time_col", DataType::TIMESTAMP, true), + StructField::new("items", ArrayType::new(DataType::LONG, true), true), + // Nested struct for upstream tests + StructField::new( + "null_v_struct", + StructType::new_unchecked(vec![StructField::new("v", DataType::LONG, true)]), + true, + ), + // Nested structs for nested_columns tests (a.b, a.b.c, b.c.f.i, data.value) + StructField::new( + "data", + StructType::new_unchecked(vec![StructField::new("value", DataType::LONG, true)]), + true, + ), + // Additional typed columns for type-checking tests + StructField::new("int_col", DataType::INTEGER, true), + StructField::new("str_col", DataType::STRING, true), + StructField::new("long_col", DataType::LONG, true), + StructField::new("double_col", DataType::DOUBLE, true), + StructField::new("short_col", DataType::SHORT, true), + StructField::new("byte_col", DataType::BYTE, true), + StructField::new("float_col", DataType::FLOAT, true), + StructField::new("bool_col", DataType::BOOLEAN, true), + StructField::new("date_col", DataType::DATE, true), + StructField::new("ts_col", DataType::TIMESTAMP, true), + StructField::new("ts_ntz_col", DataType::TIMESTAMP_NTZ, true), + // Struct type for nested tests + StructField::new( + "struct_col", + StructType::new_unchecked(vec![ + StructField::new("inner_int", DataType::INTEGER, true), + StructField::new("inner_str", DataType::STRING, true), + ]), + true, + ), + // Array type + StructField::new("array_col", ArrayType::new(DataType::LONG, true), true), + // Map type + StructField::new( + "map_col", + MapType::new(DataType::STRING, DataType::LONG, true), + true, + ), + ]) + } + + // Helper to build an IN predicate: `col IN (scalars...)` + fn in_list(col: impl Into, scalars: Vec) -> KPred { + let col = col.into(); + let element_type = scalars + .first() + .map(|s| s.data_type()) + .unwrap_or(DataType::LONG); + let array = ArrayData::try_new(ArrayType::new(element_type, false), scalars).unwrap(); + KPred::binary(KPredOp::In, col, KExpr::literal(Array(array))) + } + + // -- Comparisons -- + #[rstest] + #[case("id < 5", KPred::lt(col!("id"), Long(5)))] + #[case("id = 999", KPred::eq(col!("id"), Long(999)))] + #[case("id > 250", KPred::gt(col!("id"), Long(250)))] + #[case("id <= 2", KPred::le(col!("id"), Long(2)))] + #[case("id >= 100", KPred::ge(col!("id"), Long(100)))] + #[case("a <> 1", KPred::ne(col!("a"), Long(1)))] + #[case("a != 1", KPred::ne(col!("a"), Long(1)))] + // Literal on the left + #[case("1 < a", KPred::lt(Long(1), col!("a")))] + #[case("1 = a", KPred::eq(Long(1), col!("a")))] + #[case("1 != a", KPred::ne(Long(1), col!("a")))] + fn comparison(#[case] sql: &str, #[case] expected: KPred) { + let schema = test_schema(); + assert_eq!(parse_predicate(sql, &schema).unwrap(), expected); + } + + // -- Literal types -- + #[rstest] + #[case("name = 'bob'", KPred::eq(col!("name"), String("bob".to_string())))] + #[case("c3 < 1.5", KPred::lt(col!("c3"), Double(1.5)))] + #[case("val < -2.5", KPred::lt(col!("val"), Double(-2.5)))] + #[case("c4 > 5.0", KPred::gt(col!("c4"), Double(5.0)))] + #[case("flag = true", KPred::eq(col!("flag"), Boolean(true)))] + #[case("cc9 = false", KPred::eq(col!("cc9"), Boolean(false)))] + #[case("a = NULL", KPred::eq(col!("a"), Null(DataType::LONG)))] + #[case("id > -100", KPred::gt(col!("id"), Long(-100)))] + #[case("val > 1.0E300", KPred::gt(col!("val"), Double(1.0E300)))] + #[case("a > 2147483647", KPred::gt(col!("a"), Long(2147483647)))] + #[case("long_val < 50000000000", KPred::lt(col!("long_val"), Long(50000000000)))] + fn literal_types(#[case] sql: &str, #[case] expected: KPred) { + let schema = test_schema(); + assert_eq!(parse_predicate(sql, &schema).unwrap(), expected); + } + + // -- Compound identifiers (nested columns) -- + #[rstest] + #[case("data.value > 150", KPred::gt(col!("data.value"), Long(150)))] + #[case("null_v_struct.v > 1", KPred::gt(col!("null_v_struct.v"), Long(1)))] + #[case( + "struct_col.inner_int = 2", + KPred::eq(col!("struct_col.inner_int"), Integer(2)) + )] + fn nested_columns(#[case] sql: &str, #[case] expected: KPred) { + let schema = test_schema(); + assert_eq!(parse_predicate(sql, &schema).unwrap(), expected); + } + + // -- Boolean literals -- + #[rstest] + #[case("TRUE", KPred::TRUE)] + #[case("FALSE", KPred::FALSE)] + fn boolean_literals(#[case] sql: &str, #[case] expected: KPred) { + let schema = test_schema(); + assert_eq!(parse_predicate(sql, &schema).unwrap(), expected); + } + + // -- IS NULL / IS NOT NULL -- + #[rstest] + #[case("a IS NULL", KPred::is_null(col!("a")))] + #[case("a IS NOT NULL", KPred::is_not_null(col!("a")))] + #[case("null_v_struct.v IS NULL", KPred::is_null(col!("null_v_struct.v")))] + fn is_null(#[case] sql: &str, #[case] expected: KPred) { + let schema = test_schema(); + assert_eq!(parse_predicate(sql, &schema).unwrap(), expected); + } + + // -- NOT -- + #[rstest] + #[case("NOT a = 1", KPred::not(KPred::eq(col!("a"), Long(1))))] + #[case("NOT a = NULL", KPred::not(KPred::eq(col!("a"), Null(DataType::LONG))))] + #[case( + "NOT(a < 1 OR b > 20)", + KPred::not(KPred::or( + KPred::lt(col!("a"), Long(1)), + KPred::gt(col!("b"), Long(20)), + )) + )] + fn not_predicate(#[case] sql: &str, #[case] expected: KPred) { + let schema = test_schema(); + assert_eq!(parse_predicate(sql, &schema).unwrap(), expected); + } + + // -- AND / OR -- + #[rstest] + #[case( + "id < 500 AND value > 10", + KPred::and( + KPred::lt(col!("id"), Long(500)), + KPred::gt(col!("value"), Long(10)), + ) + )] + #[case( + "id = 1 OR id = 2", + KPred::or( + KPred::eq(col!("id"), Long(1)), + KPred::eq(col!("id"), Long(2)), + ) + )] + #[case( + "a = 0 AND b = 0 AND c = 0", + KPred::and( + KPred::and( + KPred::eq(col!("a"), Long(0)), + KPred::eq(col!("b"), Long(0)), + ), + KPred::eq(col!("c"), Long(0)), + ) + )] + #[case( + "(a < 3 AND b < 3) OR (a > 7 AND b > 7)", + KPred::or( + KPred::and( + KPred::lt(col!("a"), Long(3)), + KPred::lt(col!("b"), Long(3)), + ), + KPred::and( + KPred::gt(col!("a"), Long(7)), + KPred::gt(col!("b"), Long(7)), + ), + ) + )] + #[case( + "(a = 5 OR a = 7) AND b < 5", + KPred::and( + KPred::or( + KPred::eq(col!("a"), Long(5)), + KPred::eq(col!("a"), Long(7)), + ), + KPred::lt(col!("b"), Long(5)), + ) + )] + fn and_or(#[case] sql: &str, #[case] expected: KPred) { + let schema = test_schema(); + assert_eq!(parse_predicate(sql, &schema).unwrap(), expected); + } + + // -- Null-safe equals (<=>) -> NOT DISTINCT -- + #[rstest] + #[case("a <=> 1", KPred::not(KPred::distinct(col!("a"), Long(1))))] + #[case("a <=> NULL", KPred::not(KPred::distinct(col!("a"), Null(DataType::LONG))))] + #[case("1 <=> a", KPred::not(KPred::distinct(Long(1), col!("a"))))] + #[case( + "NOT a <=> 1", + KPred::not(KPred::not(KPred::distinct(col!("a"), Long(1)))) + )] + fn null_safe_equals(#[case] sql: &str, #[case] expected: KPred) { + let schema = test_schema(); + assert_eq!(parse_predicate(sql, &schema).unwrap(), expected); + } + + // -- IN / NOT IN -- + #[rstest] + #[case( + "a in (1, 2, 3)", + in_list(col!("a"), vec![Long(1), Long(2), Long(3)]) + )] + #[case( + "a in (1)", + in_list(col!("a"), vec![Long(1)]) + )] + #[case( + "value in (300, 787, 239)", + in_list(col!("value"), vec![Long(300), Long(787), Long(239)]) + )] + #[case( + "name in ('alice', 'bob')", + in_list(col!("name"), vec![String("alice".to_string()), String("bob".to_string())]) + )] + fn in_predicate(#[case] sql: &str, #[case] expected: KPred) { + let schema = test_schema(); + assert_eq!(parse_predicate(sql, &schema).unwrap(), expected); + } + + #[rstest] + #[case( + "a NOT IN (1, 2)", + KPred::not(in_list(col!("a"), vec![Long(1), Long(2)])) + )] + #[case( + "a NOT IN (10, 20, 30)", + KPred::not(in_list(col!("a"), vec![Long(10), Long(20), Long(30)])) + )] + fn not_in_predicate(#[case] sql: &str, #[case] expected: KPred) { + let schema = test_schema(); + assert_eq!(parse_predicate(sql, &schema).unwrap(), expected); + } + + // -- BETWEEN -> col >= low AND col <= high -- + #[rstest] + #[case( + "id BETWEEN 10 AND 20", + KPred::and( + KPred::ge(col!("id"), Long(10)), + KPred::le(col!("id"), Long(20)), + ) + )] + #[case( + "id BETWEEN -10 AND -1", + KPred::and( + KPred::ge(col!("id"), Long(-10)), + KPred::le(col!("id"), Long(-1)), + ) + )] + #[case( + "id NOT BETWEEN 10 AND 20", + KPred::not(KPred::and( + KPred::ge(col!("id"), Long(10)), + KPred::le(col!("id"), Long(20)), + )) + )] + fn between(#[case] sql: &str, #[case] expected: KPred) { + let schema = test_schema(); + assert_eq!(parse_predicate(sql, &schema).unwrap(), expected); + } + + // -- Complex predicates (multi-feature) -- + #[rstest] + #[case( + "id >= 0 AND id < 1000 AND version_tag = 'v0'", + KPred::and( + KPred::and( + KPred::ge(col!("id"), Long(0)), + KPred::lt(col!("id"), Long(1000)), + ), + KPred::eq(col!("version_tag"), String("v0".to_string())), + ) + )] + #[case( + "int_col IS NOT NULL AND str_col IS NOT NULL", + KPred::and( + KPred::is_not_null(col!("int_col")), + KPred::is_not_null(col!("str_col")), + ) + )] + #[case( + "NOT (a >= 5 AND NOT (b < 5))", + KPred::not(KPred::and( + KPred::ge(col!("a"), Long(5)), + KPred::not(KPred::lt(col!("b"), Long(5))), + )) + )] + #[case( + "partCol = 3 and id > 25", + KPred::and( + KPred::eq(col!("partCol"), Long(3)), + KPred::gt(col!("id"), Long(25)), + ) + )] + fn complex(#[case] sql: &str, #[case] expected: KPred) { + let schema = test_schema(); + assert_eq!(parse_predicate(sql, &schema).unwrap(), expected); + } + + #[rstest] + // LIKE (no kernel support) + #[case("a like 'C%'")] + #[case("fruit like 'b%'")] + #[case("a > 0 AND b like '2016-%'")] + // Function calls + #[case("cc8 = HEX('1111')")] + #[case("size(items) > 2")] + #[case("length(s) < 4")] + // Typed literals (TIME keyword) + #[case("time_col >= TIME '00:00:00'")] + #[case("time_col < TIME '12:00:00'")] + // IS NULL on non-column expressions + #[case("(a > 0) IS NULL")] + #[case("(a > 0 AND b > 1) IS NULL")] + fn unsupported_predicates_fail_gracefully(#[case] sql: &str) { + let schema = test_schema(); + let result = parse_predicate(sql, &schema); + assert!( + result.is_err(), + "Expected {sql:?} to fail, but it parsed as: {:?}", + result.unwrap() + ); + } + + // ========== New features: type checking, arithmetic, timestamps ========== + + // Type errors (schema-based type checking) + #[rstest] + #[case("long_col > 'CD'", "Failed to parse")] // String literal for Long column + #[case("long_col = false", "Type mismatch")] // Boolean vs Long + #[case("str_col = 123", "Type mismatch")] // Number literal for String column + #[case("short_col < 100000", "Failed to parse")] // Overflow: 100000 > i16::MAX + fn type_error_rejected(#[case] sql: &str, #[case] error_contains: &str) { + let schema = test_schema(); + let result = parse_predicate(sql, &schema); + assert!(result.is_err(), "Expected error for: {}", sql); + assert!( + result.unwrap_err().to_string().contains(error_contains), + "Error should contain '{}'", + error_contains + ); + } + + #[test] + fn unknown_column_produces_error() { + let schema = test_schema(); + let result = parse_predicate("unknown_col < 500", &schema); + assert!(result.is_err(), "Unknown columns should produce an error"); + assert!(result + .unwrap_err() + .to_string() + .contains("Cannot determine type")); + } + + #[test] + fn literal_only_comparison_fails() { + // When both sides are literals (no column context), types cannot be determined + let schema = test_schema(); + let result = parse_predicate("5 > 3", &schema); + assert!(result.is_err(), "Literal-only comparisons should fail"); + assert!(result + .unwrap_err() + .to_string() + .contains("Cannot determine types")); + } + + // IN list with NULL values (tests ArrayType nullability) + #[test] + fn in_list_with_null_values() { + let schema = test_schema(); + let pred = parse_predicate("a IN (1, 2, NULL)", &schema).unwrap(); + // The array should be nullable since it contains NULL + let expected_array = ArrayData::try_new( + ArrayType::new(DataType::LONG, true), // nullable = true + vec![Long(1), Long(2), Null(DataType::LONG)], + ) + .unwrap(); + let expected = KPred::binary( + KPredOp::In, + col!("a"), + KExpr::literal(Scalar::Array(expected_array)), + ); + assert_eq!(pred, expected); + } + + // Arithmetic expressions (now supported) + #[rstest] + #[case( + "a % 100 < 10 OR b > 20", + KPred::or( + KPred::lt( + KExpr::binary( + KBinOp::Minus, + col!("a"), + KExpr::binary( + KBinOp::Multiply, + Long(100), + KExpr::binary(KBinOp::Divide, col!("a"), Long(100)) + ) + ), + Long(10) + ), + KPred::gt(col!("b"), Long(20)) + ) + )] + #[case( + "long_col + 10 > 100", + KPred::gt(KExpr::binary(KBinOp::Plus, col!("long_col"), Long(10)), Long(100)) + )] + #[case( + "long_col * 2 = 10", + KPred::eq(KExpr::binary(KBinOp::Multiply, col!("long_col"), Long(2)), Long(10)) + )] + fn arithmetic(#[case] sql: &str, #[case] expected: KPred) { + let schema = test_schema(); + assert_eq!(parse_predicate(sql, &schema).unwrap(), expected); + } + + // Typed literals (DATE, TIMESTAMP, TIMESTAMP_NTZ) + #[test] + fn parse_date_typed_literal() { + let schema = test_schema(); + let pred = parse_predicate("date_col = DATE'2024-01-15'", &schema).unwrap(); + let expected = KPred::eq(col!("date_col"), Date(19737)); + assert_eq!(pred, expected); + } + + #[test] + fn parse_timestamp_typed_literal() { + let schema = test_schema(); + let pred = parse_predicate("ts_col = TIMESTAMP'2024-01-15 12:30:00'", &schema).unwrap(); + let expected = KPred::eq(col!("ts_col"), Timestamp(1705321800000000)); + assert_eq!(pred, expected); + } + + #[test] + fn parse_timestamp_ntz_typed_literal() { + let schema = test_schema(); + let pred = + parse_predicate("ts_ntz_col = TIMESTAMP_NTZ'2024-01-15 12:30:00'", &schema).unwrap(); + let expected = KPred::eq(col!("ts_ntz_col"), TimestampNtz(1705321800000000)); + assert_eq!(pred, expected); + } + + #[rstest] + #[case("TIMESTAMP_NTZ'2024-01-01'", "TIMESTAMP'2024-01-01'")] + #[case("TIMESTAMP_NTZ '2024-01-01'", "TIMESTAMP '2024-01-01'")] + #[case("col = TIMESTAMP_NTZ'2024-01-01'", "col = TIMESTAMP'2024-01-01'")] + #[case("TIMESTAMP'2024-01-01'", "TIMESTAMP'2024-01-01'")] + #[case("col = 'test'", "col = 'test'")] + fn preprocess_timestamp_ntz_replaces_keyword(#[case] input: &str, #[case] expected: &str) { + assert_eq!(preprocess_timestamp_ntz(input), expected); + } + + // Typed scalar inference from schema + #[rstest] + #[case("int_col > 100", KPred::gt(col!("int_col"), Integer(100)))] + #[case("short_col < 50", KPred::lt(col!("short_col"), Short(50)))] + #[case("byte_col <= 127", KPred::le(col!("byte_col"), Byte(127)))] + #[case("float_col < 1.5", KPred::lt(col!("float_col"), Float(1.5)))] + fn typed_scalar_inference(#[case] sql: &str, #[case] expected: KPred) { + let schema = test_schema(); + assert_eq!(parse_predicate(sql, &schema).unwrap(), expected); + } + + // Nested struct column tests + #[rstest] + #[case( + "struct_col.inner_int > 25", + KPred::gt(col!("struct_col.inner_int"), Integer(25)) + )] + #[case( + "struct_col.inner_str = 'alice'", + KPred::eq(col!("struct_col.inner_str"), String("alice".to_string())) + )] + fn nested_struct_columns(#[case] sql: &str, #[case] expected: KPred) { + let schema = test_schema(); + assert_eq!(parse_predicate(sql, &schema).unwrap(), expected); + } +}